When `highlighter:insert_notify` was called, a hole in the array was
created.
If another call to `highlighter:insert_notify` happened before the hole
was filled, a `Position out of bounds` error could have been raised.
The syntax highlighter keep a cache of the documents like tokenization.
In order to minimize the amount of tokenize re-computations we insert some
emtty lines or remove some lines in the highlither lines corresponding to
the lines added or removed to the document.
In the highlither thread We should accept a previously generated line tokenization
past first_invalid_line only if the text is the same. The text can change because of
insert or remove operations.
Close#573.
* Only one highlighter state is kept per-document as opposed
to one per-docview
* Fixes a bug with retaining older highlighter state as a
DocView wasn't able to detect lines changing above it's viewport
* Renames `highlighter` module to more descriptive `tokenizer`
Guldoman
Adam
Francesco Abbate
rxi