Fix potential buffer overflow in utf8 mode

Fix potential buffer overflow in utf8 mode

Define and use MAX_UTF8_LEN for the longest utf8 sequence
that can be returned.

In any case reverse incremental search was using + 3 rather
than - 3 to check for buffer overflow.

Also:
- support 4 byte utf8 sequences.
- better handling of invalid utf8 sequences

Signed-off-by: Steve Bennett <steveb@workware.net.au>