Solution of the quantum mechanical potential barrier

Boundary conditions for
(Subscript[ψ, A])(x) = (e^(i k x))+ a (e^(-i k x))
(Subscript[ψ, B])(x) = b1 (e^(i k1 x))+b2 (e^(-i k1 x))
(Subscript[ψ, C])(x) = c (e^(i k x))

eqs= Solve[{
1+a==b1+b2,
k(1-a)==k1 (b1-b2),
b1 (E^(I k1 L))+b2 (E^(-I k1 L))== c (E^(I k L)),
k1(b1 (E^(I k1 L))-b2 (E^(-I k1 L)))== k c (E^(I k L))
},{a,b1,b2,c}]

sol = FullSimplify[eqs]

The reflection coefficient is R=|a((|)^2)= a*a
Notice how we have to use assumptions in order to tell Mathematica that k, k1 and L are reals. 
(Complex conjugation is typed as a*-> a[Esc]conj[Esc]) or one can use a*a= Conjugate[a] a )

R = Assuming[k ∈ Reals && k1 ∈ Reals && L∈ Reals,FullSimplify[(a\[Conjugate] a)/.sol[[1]]]]

To massage Expression we use  Expand[(k^4)+6 (k^2) (k1^2)+(k1^4)-((((k^2)-(k1^2)))^2)]=8 (k^2) (k1^2)

cc=Simplify[Assuming[k ∈ Reals && k1 ∈ Reals && L∈ Reals,FullSimplify[((c\[Conjugate] c))/.sol[[1]]]]/.Cos[2 k1 L]->1-2((Sin[k1 L])^2)/.(k^4)+6 (k^2) (k1^2)+(k1^4)->((((k^2)-(k1^2)))^2)+8 (k^2) (k1^2)]

aa = Simplify[Assuming[k ∈ Reals && k1 ∈ Reals && L∈ Reals,FullSimplify[((a\[Conjugate] a))/.sol[[1]]]]/.((Cos[k1 L])^2)-> 1- ((Sin[k1 L])^2)]

Here we express |a((|)^(2 ))and |c((|)^2) in terms of ϵ=(k^2)and (k1^2)= ϵ-U

cc1 = FullSimplify[cc/.{(k^2)-> ϵ, (k1^2)-> ϵ-U,(k^4)-> (ϵ^2),(k1^4)-> (((ϵ-U))^4)}]/.Cos[2 k1 L]->1-2((Sin[k1 L])^2)

aa1 = FullSimplify[aa/.{(k^2)-> ϵ, (k1^2)-> ϵ-U}]/.Cos[2 k1 L]->1-2((Sin[k1 L])^2)

The transmission coefficient T=|c((|)^2)

T = cc1

The reflection coefficient R=|a((|)^2)

R = aa1

We finally check the expected relation R+T=1

FullSimplify[R+T]