Commit Graph

89 Commits

Author SHA1 Message Date
Behdad Esfahbod 769c2b19a3 [indic/khmer/myanmar/use] Minor shuffling of found_syllable() macros 2021-02-01 11:30:39 -08:00
Behdad Esfahbod c417e0d2f9 [indic/khmer/myanmar/use] Move enum category around
Such that the generated -machine.hh headers are independent.
2021-02-01 11:30:39 -08:00
Behdad Esfahbod 880fdb7460 [indic/khmer/myanmar/use] Export categories from Ragel source
Not hooked up yet.

Part of alternative approach to
https://github.com/harfbuzz/harfbuzz/pull/2726
2021-02-01 11:30:39 -08:00
Behdad Esfahbod 3c5b7886eb Revert "Avoid category duplication between C++ and Ragel"
This reverts commit 67ea8133d3.

Merged accidentally on https://github.com/harfbuzz/harfbuzz/pull/2726
2021-01-15 19:26:04 -07:00
David Corbett 67ea8133d3 Avoid category duplication between C++ and Ragel 2021-01-15 18:24:52 -08:00
Ebrahim Byagowi ce114d6b27 minor, tweak spaces 2019-12-31 15:53:02 +03:30
Behdad Esfahbod d115a9e022 [amalgam] Fix most duplicate-id instances in Indic-like shapers
Part of https://github.com/harfbuzz/harfbuzz/issues/1809
2019-07-02 14:58:52 -07:00
Behdad Esfahbod b1d3e54bd3 [indic] Don't constrain how many C, M, ... occur
Fixes https://github.com/harfbuzz/harfbuzz/issues/1709

Part of https://github.com/harfbuzz/harfbuzz/issues/1652
2019-05-13 14:35:04 -07:00
Behdad Esfahbod 21bb80ebf2 [indic] Add back medial-consonant to grammar
Fixes https://github.com/harfbuzz/harfbuzz/issues/1592
2019-03-28 20:50:04 -07:00
Behdad Esfahbod 5101abd42f [indic/use] Factor common expressions in ragel machine
No machine change.
2018-10-02 17:49:06 +02:00
Behdad Esfahbod 6f457f8370 [indic] Minor flip grammar around
No behavior change.
2018-10-02 17:46:27 +02:00
Behdad Esfahbod 40d5d19d5b [ragel] Use ts/te (token-start / token-end) instead of manual tracking 2018-10-02 17:04:05 +02:00
Behdad Esfahbod 8bf4027d23 [indic] Remove Consonant_Medial from grammar and code
The only Indic CM is U+0A75 GURMUKHI SIGN YAKASH, which Uniscribe doesn't treat
specially, and font designers design for that.  So, do the same.
2018-10-01 20:05:00 +02:00
Behdad Esfahbod e46c51f1e9 [indic] Do NOT allow matra after Halant,ZWJ
Fixes https://github.com/harfbuzz/harfbuzz/issues/556

Devanagari regresses 12 tests, and Gujarati 2.  See:

  https://github.com/harfbuzz/harfbuzz/issues/556#issuecomment-419957472

New numbers:

BENGALI: 353725 out of 354188 tests passed. 463 failed (0.130722%)
DEVANAGARI: 707299 out of 707394 tests passed. 95 failed (0.0134296%)
GUJARATI: 366353 out of 366457 tests passed. 104 failed (0.0283799%)
GURMUKHI: 60729 out of 60747 tests passed. 18 failed (0.0296311%)
KANNADA: 951300 out of 951913 tests passed. 613 failed (0.0643966%)
MALAYALAM: 1048136 out of 1048334 tests passed. 198 failed (0.0188871%)
ORIYA: 42327 out of 42329 tests passed. 2 failed (0.00472489%)
SINHALA: 271596 out of 271847 tests passed. 251 failed (0.0923313%)
TAMIL: 1091754 out of 1091754 tests passed. 0 failed (0%)
TELUGU: 970555 out of 970573 tests passed. 18 failed (0.00185457%)
2018-09-10 17:38:19 +02:00
Behdad Esfahbod c77ae40852 Rename hb-*private.hh to hb-*.hh
Sorry for the noise, downstream custom builders.  Please adjust.
2018-08-25 22:36:36 -07:00
Ebrahim Byagowi 2eaba80cbd
Minor, re-add unused mark to some of ragel defined state machines (#1005) 2018-04-24 09:04:15 +04:30
Ebrahim Byagowi 24b8b9b227 Resolve clang's used-but-marked-unused warnings 2018-04-23 11:38:45 -07:00
Behdad Esfahbod bbfdbbdc5e Minor 2018-01-05 17:34:26 +00:00
Behdad Esfahbod f6df251521 [indic] Remove unused OT_VD category 2018-01-05 17:27:48 +00:00
Behdad Esfahbod bbac984b93 Minor 2018-01-05 17:15:31 +00:00
Behdad Esfahbod 9761f9d782 [indic] Remove more Khmer-specific stuff from Indic shaper
No numbers changed.
2018-01-05 15:33:11 +00:00
Behdad Esfahbod f559c63307 [indic] Implement Consonent_With_Stacker
Fixes https://github.com/behdad/harfbuzz/issues/528
2017-10-03 15:20:07 +02:00
Behdad Esfahbod 8b5d6e755b [indic] Remove unused Javanese bits 2016-05-06 15:59:27 +01:00
Behdad Esfahbod e6f80fa104 [indic] Allow ZWJ/ZWNJ before SM
In Oriya, a ZWJ/ZWNJ might be added before candrabindu to encourage
or stop ligation of the candrabindu.  This is clearly specified in
the Unicode section on Oriya.  Allow it there.  Note that Uniscribe
doesn't allow this.

Micro tests added using Noto Sans Oriya draft.

No changes in numbers.  Currently at:

BENGALI: 353725 out of 354188 tests passed. 463 failed (0.130722%)
DEVANAGARI: 707307 out of 707394 tests passed. 87 failed (0.0122987%)
GUJARATI: 366349 out of 366457 tests passed. 108 failed (0.0294714%)
GURMUKHI: 60732 out of 60747 tests passed. 15 failed (0.0246926%)
KANNADA: 951190 out of 951913 tests passed. 723 failed (0.0759523%)
KHMER: 299070 out of 299124 tests passed. 54 failed (0.0180527%)
MALAYALAM: 1048147 out of 1048334 tests passed. 187 failed (0.0178378%)
ORIYA: 42320 out of 42329 tests passed. 9 failed (0.021262%)
SINHALA: 271662 out of 271847 tests passed. 185 failed (0.068053%)
TAMIL: 1091753 out of 1091754 tests passed. 1 failed (9.15957e-05%)
TELUGU: 970555 out of 970573 tests passed. 18 failed (0.00185457%)
2014-12-10 12:05:24 -08:00
Behdad Esfahbod e8b5d64039 [indic] Do NOT allow reph formation on placeholders
Only allow it on DOTTED CIRCLE.  No effect on test numbers.

Test: U+0930,094D,00A0
2014-05-29 15:20:15 -04:00
Behdad Esfahbod cf78dd483c [indic/myanmar] Rename OT_NBSP to OT_PLACEHOLDER 2014-05-27 17:53:37 -04:00
Behdad Esfahbod ddbdfcbf1c [indic] Simplify grammar
No functional change.
2014-05-23 15:39:55 -04:00
Behdad Esfahbod 4e9b1f662b [indic] Always start new syllable for Avagraha
In fact, the previous grammar was ambigious.  No functional
change.
2014-05-23 15:38:42 -04:00
Behdad Esfahbod 9f9bd9bf31 [indic] Rename avagraha cluster to symbol cluster
In anticipation of adding more characters to that class of clusters.
2014-05-23 15:35:38 -04:00
Behdad Esfahbod e848bfae7c [indic] Recategorize U+A8E0..A8F1 as OT_VD
Up to two of them come after all OT_A characters.
2014-05-22 18:50:34 -04:00
Behdad Esfahbod c519536c34 [indic] Allow up to three tone marks
According to Roozbeh, there are valid combinations in Unicode
proposals for up to three.  Previously we were allowing up to two.
2014-05-22 18:43:14 -04:00
Behdad Esfahbod a74f0de225 [indic] Fix CM2, really
Followup from 6e613f3365.
2013-10-24 11:46:33 +02:00
Behdad Esfahbod c16012e901 [indic] Add Javanese support!
Seems to be working just fine!
2013-10-18 18:17:29 +02:00
Behdad Esfahbod 1a7de1ba98 [indic] Improve Avagraha support in machine 2013-10-16 19:55:06 +02:00
Behdad Esfahbod 3756efaf4e [indic] Misc harmless fixes!
First, we were abusing OT_VD instead of OT_A.  Fix that
but moving OT_A in the grammar where it belongs (which
is different from what the spec says).

Also, only allow medial consonants after all other
consonants.  This doesn't affect any current character.

Finally, fix Halant attachment in presence of medial
consonants.  Again, this currently doesn't affect any
sequence.

I lied.  There's Gurmukhi U+0A75 which is Consonant_Medial.
Uniscribe allows one of those in each of these positions:
before matras, after matras and before syllable modifiers,
and after syllable modifiers!  We currently just allow
unlimited numbers of it, before matras.
2013-10-16 19:06:29 +02:00
Behdad Esfahbod 5c558877da [indic] Allow up to two syllable modifiers
Bug 70509 - Candrabindu+Visarga doesn't work in Devanagari
https://bugs.freedesktop.org/show_bug.cgi?id=70509

We categorize both bindus and visarga as syllable-modifiers.
OT spec doesn't actually say what characters go in the syllable
modifier category, and allows one.  We just allow up to two now.

Test case: U+0930,U+0941,U+0901,U+0903

Uniscribe currently doesn't support that and produces a
dotted circle.
2013-10-16 11:18:09 +02:00
Behdad Esfahbod 3c7b3641cf [indic] Handle Avagraha
It can come either at the end(ish!) of the syllable, or independently.
When independent, it accepts a few bits and pieces.
2013-10-15 13:14:31 +02:00
Behdad Esfahbod e7562f53fe Fix compile warnings for ragel-generated machines 2013-02-14 11:58:13 -05:00
Behdad Esfahbod 596740db04 [Indic] Insert dottedcircle after a lone Malayalam dot-reph 2012-12-21 19:41:04 -05:00
Behdad Esfahbod 0beb66e3a6 Fix warnings 2012-12-05 19:14:28 -05:00
Behdad Esfahbod 9cac1338c4 [Indic] Allow Consonant_Medial's after Consonant's
Mostly affects Myanmar, but also Tai Tham, Javanese, and Cham.  The
latter three are untested (no fonts!).
2012-11-12 18:41:22 -08:00
Behdad Esfahbod 166b5cf7ec [Indic] Find syllables before any features are applied
With FreeSerif, it seems that the 'ccmp' feature does ligature
substituttions.  That was then causing syllable match failures.  We now
find syllables before any features have been applied.

Test sequence: U+0D9A,U+0DCA,U+200D,U+0DBB,U+0DCF
2012-09-07 14:56:01 -04:00
Behdad Esfahbod f0b8ed1b6d [Indic] Allow "H,ZWJ,M"
Uniscribe accepts a Halant,ZWJ before matras.  Allow that.

BENGALI down from 295 to 291
DEVANAGARI down from 69 to 57
GUJARATI down from 19 to 17
KANNADA down from 871 to 867
MALAYALAM down from 340 to 337
TELUGU down from 20 to 16

Currently at:

BENGALI: 353897 out of 354188 tests passed. 291 failed (0.0821598%)
DEVANAGARI: 707337 out of 707394 tests passed. 57 failed (0.00805774%)
GUJARATI: 366440 out of 366457 tests passed. 17 failed (0.00463902%)
GURMUKHI: 60704 out of 60747 tests passed. 43 failed (0.0707854%)
KANNADA: 951046 out of 951913 tests passed. 867 failed (0.0910798%)
KHMER: 299077 out of 299124 tests passed. 47 failed (0.0157125%)
LAO: 53611 out of 53644 tests passed. 33 failed (0.0615167%)
MALAYALAM: 1047997 out of 1048334 tests passed. 337 failed (0.0321462%)
ORIYA: 42320 out of 42329 tests passed. 9 failed (0.021262%)
SINHALA: 271666 out of 271847 tests passed. 181 failed (0.0665816%)
TAMIL: 1091754 out of 1091754 tests passed. 0 failed (0%)
TELUGU: 970557 out of 970573 tests passed. 16 failed (0.00164851%)
TIBETAN: 208469 out of 208469 tests passed. 0 failed (0%)
2012-09-05 17:41:08 -04:00
Behdad Esfahbod 4ed717ef61 [Indic] Relax grammar
Now that we insert dotted-circle, tests break more easily when our indic
machine breaks.

In particular, a few Devanagari tests were having sequences like
"C,H,ZWJ,N", and because of the ZWJ the Nukta does NOT get reordered to
before the Halant as the grammar used to expect...  Fixup.

Another case is as simple as "C,ZWJ,SM".

Fixes 10 out of 79 failures:

DEVANAGARI: 707325 out of 707394 tests passed. 69 failed (0.00975411%)
2012-09-05 17:21:17 -04:00
Behdad Esfahbod aa7141efe4 [Indic] Fix Khmer syllable-final coeng-consonant
Brings down Khmer failures from 162 to 47.

KHMER: 299077 out of 299124 tests passed. 47 failed (0.0157125%)

Also rebaselined some of the test files that had only-inherited lines.
Removing those, the stats are:

BENGALI: 353893 out of 354188 tests passed. 295 failed (0.0832891%)
DEVANAGARI: 707315 out of 707394 tests passed. 79 failed (0.0111678%)
GUJARATI: 366438 out of 366457 tests passed. 19 failed (0.00518478%)
GURMUKHI: 60704 out of 60747 tests passed. 43 failed (0.0707854%)
KANNADA: 951042 out of 951913 tests passed. 871 failed (0.0915%)
KHMER: 299077 out of 299124 tests passed. 47 failed (0.0157125%)
LAO: 53611 out of 53644 tests passed. 33 failed (0.0615167%)
MALAYALAM: 1047994 out of 1048334 tests passed. 340 failed (0.0324324%)
ORIYA: 42320 out of 42329 tests passed. 9 failed (0.021262%)
SINHALA: 271666 out of 271847 tests passed. 181 failed (0.0665816%)
TAMIL: 1091754 out of 1091754 tests passed. 0 failed (0%)
TELUGU: 970553 out of 970573 tests passed. 20 failed (0.00206064%)
TIBETAN: 208469 out of 208469 tests passed. 0 failed (0%)

Still some regressions, but some of the more egregious cases are
addressed.
2012-09-05 17:14:52 -04:00
Behdad Esfahbod b85800f9de [Indic] Implement dotted-circle insertion for broken clusters
No panic, we reeally insert dotted circle when it's absolutely broken.

Fixes most of the dotted-circle cases against Uniscribe. (for Devanagari
fixes 80% of them, for Khmer 70%; the rest look like Uniscribe being
really bogus...)

I had to make a decision.  Apparently Uniscribe adds one dotted circle
to each broken character.  I tried that, but that goes wrong easily with
split matras.  So I made it add only one dotted circle to an entire
broken syllable tail.  As in: "if there was a dotted circle here, this
would have formed a correct cluster."  That works better for split
stuff, and I like it more.
2012-08-31 19:18:20 -04:00
Behdad Esfahbod 327d14ef18 [Indic] Start adding dotted-circle instrastructure 2012-08-31 16:49:34 -04:00
Behdad Esfahbod 85fc6c483f [Indic] Move more stuff to the shape_plan
Almost done.  Need to add per-script static tables.
2012-08-02 12:21:44 -04:00
Behdad Esfahbod 8bb5deba96 [OT] Pipe shape_plan down to pause_callbacks 2012-08-02 10:07:58 -04:00
Behdad Esfahbod dff0ece11d [Indic] Limit matras to 4 per syllable
Also limit joiners.

This limits our syllable length to a constant, and is
closer to what Uniscribe does anyway.

Two Devanagari tests regressed, but who cares about tests with 20
joiners in a row?!  Devanagari at 57 (0.00821766%) now.
2012-07-24 02:37:42 -04:00