Oh, going back to my formula, it sometimes gives a negative answer. Getting the absolute value of the numerator/the whole answer at the end works, but I'm not sure what caused it...
Now, let's simplify that a bit, taking the x part:
(ay + x - ab)/(a2 + 1) - x
ay + x - ab - a2x - x
(a2x + ay - ab)/(a2 + 1)
And the y part:
(a2y + ax + b)/(a2 + 1) - y
a2y + ax + b - a2y - y
(ax + b - y)/(a2 + 1)
In the top set of simplifications, you drop a (-), you should have:
(-a
2x + ay - ab)/(a
2 + 1)
But the main problem is you went through a ton of extra factoring steps... what I just wrote there is equal to:
= a
(y - b - ax)/(a
2 + 1)
Call that m (what you call the x part). Then,
(ax + b - y)/(a
2 + 1)
"The y part"; call this n. Note that the bolded portions are additive opposites.
To simplify writing things, I'm going to call the distance (what we're looking for) d, and so
d
2 = m
2 + n
2OK, so let
k = ax + b - y
z = a
2 + 1
Now, m = -ak/z, and n = k/z
m
2 = a
2k
2/z
2n
2 = k
2/z
2d
2 = m
2 + n
2 = a
2k
2/z
2 + k
2/z
2So
d
2z
2 = a
2k
2 + k
2d
2z
2 = (a
2 + 1)k
2d
2z
2 = zk
2d
2 = k
2/z
d = abs(k/sqrt(z)) = abs((ax + b - y)/sqrt(a
2 + 1))
----
Dropping that negative sign gives the same answer via squaring, I suspect, but it takes a lot more work.