Isomorphic elliptic curves

To make things concrete, suppose that we're working in characteristic $ne 2,3$ (we can do something similar in those cases, though it gets messier), and the equation of the curve is

$E : y^2 = x^3 + a x + b$

and the equation of the twist is:

$E^{(d)} : d y^2 = x^3 + a x + b$.

Multiplying this by $d^3$:

$(d^2 y)^2 = (d x)^3 + a d^2 (d x) + d^3 b$

for this to be isomorphic to $E$ over $k$ it is necessary and sufficient that
there is a $lambda in k^*$ such that (see Silverman, or any other book on elliptic curves)
such that $a d^2 = a lambda^4$, $b d^3 = b lambda^6$. If $a,b ne 0$ then we must have
$lambda^6 = d^3$ and $lambda^4 = d^2$, which shows that $lambda^2 = d$ which you've ruled out. If $b=0$ you have, by taking square roots, $lambda^2 = pm d$, and only the minus sign is possible. If $a=0$ (and thus $b ne 0$) then $lambda^6 = d^3$. But then $(lambda^3/d)^2 = d^3/d^2 = d$, which you've also ruled out. Thus, only $b=0$ is possible, which is $j=1728$.