The line is infinite (defined by two anchor points). The mouse is the external point; the red marker is the foot of the perpendicular.
See also: Closest point on a line segment — same projection, clamped to the segment.
ab = b - a
ap = p - a
t = (ap . ab) / (ab . ab) // not clamped: line is infinite
foot = a + ab * t
distance = |p - foot|
const ax = 40, ay = 100
const bx = 260, by = 200
const abx = bx - ax
const aby = by - ay
const apx = mouseX - ax
const apy = mouseY - ay
const t = (apx * abx + apy * aby) / (abx * abx + aby * aby)
const footX = ax + abx * t
const footY = ay + aby * t
const dx = mouseX - footX
const dy = mouseY - footY
const distance = Math.sqrt(dx * dx + dy * dy)