* Introduces a flag that syntax writers can turn off named
space_handling, turning it off means that your syntax will take care
of handling the excessive amount of spaces that can slow down the
tokenizer.
* Adds another pattern at the end of every single table that also
improves tokenizer performance by matching words that weren't match by
any of the synxtax patterns.
* Modifies language_md to turn off the provided space_handling and do its
own since it has rules that require a space at the beginning, also
handles long consecutives amount of dashes used in tables that degrade
performance.
* This changes where discussed in collaboration with @Guldoman and
@takase1121 thanks to all!
* mainly the language_md got affected which has some exotic rules
* some other languages are also using spaces at start of pattern
and even if not affected this change tackles that
jgmdev
Guldoman
rxi