regexp: Improved error message

Detect and produce an error for missing closing bracket ]
Consider a trailing backslash as an invalid escape

Signed-off-by: Steve Bennett <steveb@workware.net.au>