Half the 2D cross product of the two edge vectors gives the signed area of a triangle. The sign tells you which way the vertices wind. Two vertices are fixed; the third follows the mouse. Note: on a canvas the y-axis points down, so the sign is inverted compared to the maths-textbook convention.
See also: Line-segment intersection — the same 2D cross product appears as the denominator.
signedArea = 0.5 * ((b.x - a.x) * (c.y - a.y) - (c.x - a.x) * (b.y - a.y))
// signedArea > 0: one winding direction
// signedArea < 0: the other
// signedArea == 0: collinear
const ax = 80, ay = 80
const bx = 240, by = 220
const cx = mouseX, cy = mouseY
const signedArea = 0.5 * ((bx - ax) * (cy - ay) - (cx - ax) * (by - ay))
const winding = signedArea > 0 ? 'clockwise (on screen)'
: signedArea < 0 ? 'counter-clockwise (on screen)'
: 'collinear'