Sinhala and Telugu use "explicit" reph. That is, the reph is formed by
a Ra,H,ZWJ sequence. Previously, upon detecting this sequence, we were
checking checking whether the 'rphf' feature applies to the first two
glyphs of the sequence. This is how the Microsoft fonts are designed.
However, testing with Noto shows that apparently Uniscribe also forms
the reph if the lookup ligates all three glyphs. So, try both
sequences.
Doesn't affect test results for Sinhala or Telugu.
https://code.google.com/a/google.com/p/noto-alpha/issues/detail?id=232
The grammar in the OT spec, and the existing Windows implementation
seem to be confused around where to allow Asat around the medial
consonants.
The previous grammar for medial group was allowing an Asat after
the medial group only if there was a medial Wa or Ha, but not if
there was only a medial Ya. This doesn't make sense to me and
sounds reversed, as both medial Wa and Ha are below marks while
Asat is an above mark. An Asat can come before the medial group
already (in fact, multiple ones can. Why?!). The medial Ya
however is a spacing mark and according to Roozbeh it's valid
to want an Asat on the medial Ya instead of the base, so it looks
to me like we want to allow an Asat after the medial group if
there *was* a Ya but not if there wasn't any. Not wanting to
produce dotted-circle where Windows is not, this commit changes
the grammar to allow one Asat after the medial group no matter
what comes in the group.
Test: U+1002,103A,103B vs U+1002,103B,103A
Before we were just relying on the compiler inlining them and not
leaving a trace in our public API. Try to fix. Hopefully not
breaking anyone's build.
commit b5a0f69e47
Author: Behdad Esfahbod <behdad@behdad.org>
Date: Thu Oct 17 18:04:23 2013 +0200
[indic] Pass zero-context=false to would_substitute for newer scripts
For scripts without an old/new spec distinction, use zero-context=false.
This changes behavior in Sinhala / Khmer, but doesn't seem to regress.
This will be useful and used in Javanese.
The *intention* was to change zero-context from true to false for scripts that
don't have old-vs-new specs. However, checking the code, looks like we
essentially change zero-context to always be true; ie. we only changed things
for old-spec, and we broke them. That's what causes this bug:
https://bugs.freedesktop.org/show_bug.cgi?id=76705
The root of the bug is here:
/* Use zero-context would_substitute() matching for new-spec of the main
* Indic scripts, but not for old-spec or scripts with one spec only. */
bool zero_context = indic_plan->config->has_old_spec || !indic_plan->is_old_spec;
Note that is_old_spec itself is:
indic_plan->is_old_spec = indic_plan->config->has_old_spec && ((plan->map.chosen_script[0] & 0x000000FF) != '2');
It's easy to show that zero_context is now always true. What we really meant was:
bool zero_context = indic_plan->config->has_old_spec && !indic_plan->is_old_spec;
Ie, "&&" instead of "||". We made this change supposedly to make Javanese
work. But apparently we got it working regardless! So I'm going to fix this
to only change the logic for old-spec and not touch other cases.
This is a higher-priority shaper than default shaper ("ot"), but
only picks up fonts that have AAT "morx"/"mort" table.
Note that for this to work the font face's get_table() implementation
should know how to return the full font blob.
Based on patch from Konstantin Ritt.
Not exhaustively tested, but I think I got the intended logic
right.
The logic can perhaps be simplified. Maybe we should disabled
normalization with this shaper. Then again, for now focusing on
correctness.
When seeing U+2044 FRACTION SLASH in the text, find decimal
digits (Unicode General Category Decimal_Number) around it,
and mark the pre-slash digits with 'numr' feature, the post-slash
digits with 'dnom' feature, and the whole sequence with 'frac'
feature.
This beautifully renders fractions with major Windows fonts,
and any other font that implements those features (numr/dnom is
enough for most fonts.)
Not the fastest way to do this, but good enough for a start.
CoreText does automatic font fallback (AKA "cascading") for characters
not supported by the requested font, and provides no way to turn it off,
so detect if the returned run uses a font other than the requested one
and fill in the buffer with .notdef glyphs instead of random indices
glyph from a different font.
The spec and Uniscribe don't allow these, but UTN#11
specifically says the sequence U+104B,U+1038 is valid.
As such, allow all "P V" sequences. There's about
eight sequences that match that structure, but Roozbeh
thinks it's fine to allow all of them.
Test case: U+104B, U+1038
https://bugs.freedesktop.org/show_bug.cgi?id=71947
The spec and Uniscribe treat it as consonant in the grammar, but
it's not in IndicSyllableCategory.txt, so fix up.
Test sequence: U+1004,U+103A,U+1039,U+104E
https://bugs.freedesktop.org/show_bug.cgi?id=71948
This is broken sequence according to OpenType spec, Uniscribe,
and current HarfBuzz implementation. But Roozbeh says this
is a valid sequence, so allow it. There are multiple
"(DB As?)?" constructs in the grammar, but Roozbeh thinks only
this one needs changing.
Test case: 1014,1063,103A
Fixes https://bugs.freedesktop.org/show_bug.cgi?id=71949
Based on research into latest SIL and Windows fonts, pulling in
the latest OpenType language tag proposal from Microsoft, and updating
to latest language tags and names from ISO 639.
This reverts commit d5bd0590ae.
The reasoning behind that logic was flawed and made under
a misunderstanding of the original problem, and caused
regressions as reported by Jonathan Kew in thread titled
"tibetan marks" in Oct 2013. Apparently I have had fixed
the original problem with this commit:
7e08f1258d
So, revert the faulty commit and everything seems to be in good
shape.
For Javanese (pref_len == 1) only reorder if it didn't ligate. That's
sensible, and what the spec says. For other Indic (pref_len > 1)
only reorder if ligated.
Doesn't change any test numbers.
Bug 58714 - Kannada u+0cb0 u+200d u+0ccd u+0c95 u+0cbe does not provide
same results as Windows8
https://bugs.freedesktop.org/show_bug.cgi?id=58714
Test with U+0CB0,U+200D,U+0CCD,U+0C95,U+0CBF and tunga.ttf.
Improves some scripts. Improves Bengali too, but numbers
are up because we produce better results than Uniscribe for some
sequences now.
New numbers:
BENGALI: 353724 out of 354188 tests passed. 464 failed (0.131004%)
DEVANAGARI: 707307 out of 707394 tests passed. 87 failed (0.0122987%)
GUJARATI: 366349 out of 366457 tests passed. 108 failed (0.0294714%)
GURMUKHI: 60732 out of 60747 tests passed. 15 failed (0.0246926%)
KANNADA: 951190 out of 951913 tests passed. 723 failed (0.0759523%)
KHMER: 299070 out of 299124 tests passed. 54 failed (0.0180527%)
MALAYALAM: 1048140 out of 1048334 tests passed. 194 failed (0.0185056%)
ORIYA: 42320 out of 42329 tests passed. 9 failed (0.021262%)
SINHALA: 271662 out of 271847 tests passed. 185 failed (0.068053%)
TAMIL: 1091753 out of 1091754 tests passed. 1 failed (9.15957e-05%)
TELUGU: 970555 out of 970573 tests passed. 18 failed (0.00185457%)
Lohit-Punjabi has a upem of 769! We were losing one unit in our
code, and FreeType is losing another one... Test with U+0A06.
Has an advance of 854 in the font. We were producing 852.
Now we do 853, which is what FreeType is telling us.
See comments from caveat! Seems to work fine.
This is useful for Javanese which has an atomically encoded pre-base
reordering Ra which should only be reordered if it was substituted
by the pref feature.
For scripts without an old/new spec distinction, use zero-context=false.
This changes behavior in Sinhala / Khmer, but doesn't seem to regress.
This will be useful and used in Javanese.
Whic means these twp are applied per-syllable now. Apparently
in some Khmer fonts the clig interacts with presentation features.
Test case: U+1781,U+17D2,U+1789,U+17BB,U+17C6 with Mondulkiri-R.ttf
should produce one big ligature.
Commit 6b65a76b40. "end" was becoming
negative. Was trigerred by Lohit-Kannada 2.5.3 and the sequence:
U+0CB0,U+200D,U+0CBE,U+0CB7,U+0CCD,U+0C9F,U+0CCD,U+0CB0,U+0C97,U+0CB3
Two glyphs were being duplicated.
First, we were abusing OT_VD instead of OT_A. Fix that
but moving OT_A in the grammar where it belongs (which
is different from what the spec says).
Also, only allow medial consonants after all other
consonants. This doesn't affect any current character.
Finally, fix Halant attachment in presence of medial
consonants. Again, this currently doesn't affect any
sequence.
I lied. There's Gurmukhi U+0A75 which is Consonant_Medial.
Uniscribe allows one of those in each of these positions:
before matras, after matras and before syllable modifiers,
and after syllable modifiers! We currently just allow
unlimited numbers of it, before matras.
Unicode 6.2.0 Section 16.2 / Figure 16.3 says:
"For backward compatibility, between Arabic characters a ZWJ acts just
like the sequence <ZWJ, ZWNJ, ZWJ>, preventing a ligature from forming
instead of requesting the use of a ligature that would not normally be
used. As a result, there is no plain text mechanism for requesting the
use of a ligature in Arabic text."
As such, we flip internal zwj to zwnj flags for GSUB matching, which
means it will block ligation in all features, unless the font
explicitly matches U+200D glyph. This doesn't affect joining behavior.
Bug 70509 - Candrabindu+Visarga doesn't work in Devanagari
https://bugs.freedesktop.org/show_bug.cgi?id=70509
We categorize both bindus and visarga as syllable-modifiers.
OT spec doesn't actually say what characters go in the syllable
modifier category, and allows one. We just allow up to two now.
Test case: U+0930,U+0941,U+0901,U+0903
Uniscribe currently doesn't support that and produces a
dotted circle.
Seems to better match Uniscribe.
Note: NotoSansTelugu-Regular has kern feature, so this fixes most of the
positioning failures there, except for the kern pairs blocked by a
(non-)joiner, in which case we (correctly) kern, but Uniscribe doesn't.
More like Uniscribe... We still allow user-defined features to
work across syllables, but not pres,blws,abs,psts,etc.
This "regressed" Sinhala numbers by 11. These are cases were
there's Consonant followed by Ra,Halant,ZWJ at the of text.
The Ra,Halant,ZWJ ends up forming reph, which is wrong...
But before we were also ligating that reph with the previous
consonant. That's even more wrong. That's also what Uniscribe
does.
Current numbers:
BENGALI: 353732 out of 354188 tests passed. 456 failed (0.128745%)
DEVANAGARI: 707307 out of 707394 tests passed. 87 failed (0.0122987%)
GUJARATI: 366349 out of 366457 tests passed. 108 failed (0.0294714%)
GURMUKHI: 60732 out of 60747 tests passed. 15 failed (0.0246926%)
KANNADA: 951030 out of 951913 tests passed. 883 failed (0.0927606%)
KHMER: 299070 out of 299124 tests passed. 54 failed (0.0180527%)
MALAYALAM: 1048140 out of 1048334 tests passed. 194 failed (0.0185056%)
ORIYA: 42320 out of 42329 tests passed. 9 failed (0.021262%)
SINHALA: 271655 out of 271847 tests passed. 192 failed (0.070628%)
TAMIL: 1091753 out of 1091754 tests passed. 1 failed (9.15957e-05%)
TELUGU: 970555 out of 970573 tests passed. 18 failed (0.00185457%)
Previously we only supported recursive sublookups with
ascending indices. We were also not correctly handling
non-1-to-1 recursed lookups.
Fix all that!
Fixes the three tests in test/shaping/tests/context-matching.tests,
which were derived from NotoSansBengali and NotoSansDevanagari
among others.
Should move these out of the public header...
We're "clean" of introspection warnings now. Remaining ones are about
graphite2 / freetype types not being introspectable.
This reverts commit 10f964623f.
See discussion with Khaled Hosny on mailing list. In short, since
integers here can be negative, and int division is "round towards
zero", proper rounding should take sign into account. Just skip
doing it again, has been serving us well before.
They've been disabled for a while and no one cared. We're past
the point to need them for testing, and if we ever need to
resurrect them again, well, they're in git graveyard somewhere.
Initial setup of gtk-doc. Straight forward setup following the gtk-doc
instructions. Ignore some troublesome types in src/hb-gobject.h. To
build use "./autogen.sh --enable-gtk-doc" then "make". Docs are in
harfbuzz/docs/reference/html/index.html.
Based on patch from Jonathan Kew and data from Apple.
It's not working correctly though, and I suspect I'm hitting a bug in
CoreText. When I do this:
hb-shape /Library/Fonts/Zapfino.ttf ZapfinoZapfino --shaper coretext \
--features=-liga
I expect both ligatures to turn off, but only the second one does:
[Z_a_p_f_i_n_o=0+2333|Z=7+395|a=8+285|p_f=9+433|i=11+181|n=12+261|o=13+250]
whereas if I disable 'dlig' instead of 'liga', both are turned off.
Smells...
Doesn't resolve conflicting feature settings.
When installing per-process fonts using AddFontMemResourceEx(),
if a font with the same family name is already installed, sometimes
that one gets used. Which is problematic for us. As such, we
now mangle the font to install a new 'name' table with a unique
name, which we then use to choose the font.
Patch from Jonathan Kew.
During GSUB, if a ligation happens, subsequence context input matching
matches the new indexing. During GPOS however, the indices never
change. So just go one by one.
Fixes 'dist' positioning with mmrtext.ttf and the following sequence:
U+1014,U+1039,U+1011,U+1014,U+1039,U+1011,U+1014,U+1039,U+1011
Reported by Jonathan Kew.
Email from Jonathan Kew:
My cygwin build kept aborting on certain test words when run with the
uniscribe backend. Turned out this was caused by a bug in the allocation
of scratch buffers in hb-uniscribe.cc.
Commit 2a17f9568d introduced a new line
ALLOCATE_ARRAY (SCRIPT_VISATTR, vis_attr, glyphs_size);
but it failed to account for this in the computation of glyphs_size
(the number of glyphs for which scratch buffer space is available),
with the result that the vis_clusters array ends up overrunning the
end of the scratch buffer and clobbering the beginning of the buffer's
info[].
AFAICS, the vis_attr array is not actually used, so the simple fix is
to remove the line that allocates it. (If/when we -do- need to use
vis_attr for something, we'll need to add another term to the earlier
calculation of glyphs_size.)
With this patch, the uniscribe backend runs reliably again.
JK
This changes the semantics of get_glyph() callback and expect that
callbacks return false if the requested variant is not available, and
then we will call them back with variation_selector=0 and will retain
the glyph for the selector in the glyph stream.
Apparently most Mongolian fonts implement the Mongolian Variation
Selectors using GSUB, not cmap.
https://bugs.freedesktop.org/show_bug.cgi?id=65258
Note that this doesn't fix the Mongolian shaping yet, because the way
that's implemented is that the, say, 'init' feature ligates the letter
and the variation-selector. However, since currently the variation
selector doesn't have the 'init' mask on, it will not be matched...
If there's a mark ligating forward with non-mark, they were
inheriting the GC of the mark and later get advance-zeroed.
Don't do that if there's any non-mark glyph in the ligature.
Sample test: U+1780,U+17D2,U+179F with Kh-Metal-Chrieng.ttf
Also:
Bug 58922 - Issue with mark advance zeroing in generic shaper
We were not initializing the digests properly and as a result they were
being initialized to zero, making digest1 to never do any useful work.
Speeds up Amiri shaping significantly.
See thread "an issue regarding discrepancy between Korean and Unicode
standards" on the mailing list for the rationale. In short: Uniscribe
doesn't, so fonts are designed to work without it.
Testing shows that this is closer to what Uniscribe does.
Reported by Khaled Hosny:
"""
commit 568000274c
...
This commit is causing a regression with Amiri, the string “هَٰذ” with
Uniscribe and HarfBuzz before this commit, gives:
[uni0630.fina=3+965|uni0670.medi=0+600|uni064E=0@-256,0+0|uni0647.init=0+926]
But now it gives:
[uni0630.fina=3+965|uni0670.medi=0+0|uni064E=0@-256,0+0|uni0647.init=0+926]
i.e. uni0670.medi is zeroed though it has a base glyph GDEF class.
"""
The test case is U+0647,U+064E,U+0670,U+0630 with Amiri.
After the Ngapi hackfest work, we were assuming that fonts
won't use presentation features to choose specific forms
(eg. conjuncts). As such, we were using auto-joiner behavior
for such features. It proved to be troublesome as many fonts
used presentation forms ('pres') for example to form conjuncts,
which need to be disabled when a ZWJ is inserted.
Two examples:
U+0D2F,U+200D,U+0D4D,U+0D2F with kartika.ttf
U+0995,U+09CD,U+200D,U+09B7 with vrinda.ttf
What we do now is to never do magic to ZWJ during GSUB's main input
match for Indic-style shapers. Note that backtrack/lookahead are still
matched liberally, as is GPOS. This seems to be an acceptable
compromise.
As to the bug that initially started this work, that one needs to
be fixed differently:
Bug 58714 - Kannada u+0cb0 u+200d u+0ccd u+0c95 u+0cbe does not
provide same results as Windows8
https://bugs.freedesktop.org/show_bug.cgi?id=58714
New numbers:
BENGALI: 353689 out of 354188 tests passed. 499 failed (0.140886%)
DEVANAGARI: 707305 out of 707394 tests passed. 89 failed (0.0125814%)
GUJARATI: 366349 out of 366457 tests passed. 108 failed (0.0294714%)
GURMUKHI: 60706 out of 60747 tests passed. 41 failed (0.067493%)
KANNADA: 951030 out of 951913 tests passed. 883 failed (0.0927606%)
KHMER: 299070 out of 299124 tests passed. 54 failed (0.0180527%)
LAO: 53611 out of 53644 tests passed. 33 failed (0.0615167%)
MALAYALAM: 1048102 out of 1048334 tests passed. 232 failed (0.0221304%)
ORIYA: 42320 out of 42329 tests passed. 9 failed (0.021262%)
SINHALA: 271666 out of 271847 tests passed. 181 failed (0.0665816%)
TAMIL: 1091753 out of 1091754 tests passed. 1 failed (9.15957e-05%)
TELUGU: 970555 out of 970573 tests passed. 18 failed (0.00185457%)
TIBETAN: 208469 out of 208469 tests passed. 0 failed (0%)
That flag is redundant, deprecated, and ignored since April 2011.
From FreeType git log:
commit 8c82ec5b17d0cfc9b0876a2d848acc207a62a25a
Author: Behdad Esfahbod <behdad@behdad.org>
Date: Thu Apr 21 08:21:37 2011 +0200
Always ignore global advance.
This makes FT_LOAD_IGNORE_GLOBAL_ADVANCE_WIDTH redundant,
deprecated, and ignored. The new behavior is what every major user
of FreeType has been requesting. Global advance is broken in many
CJK fonts. Just ignoring it by default makes most sense.
* src/truetype/ttdriver.c (tt_get_advances),
src/truetype/ttgload.c (TT_Get_HMetrics, TT_Get_VMetrics,
tt_get_metrics, compute_glyph_metrics, TT_Load_Glyph),
src/truetype/ttgload.h: Implement it.
* docs/CHANGES: Updated.
I added these because the older mingw32 toolchain didn't have
MemoryBarrier(). The newer mingw-w64 toolchain however has.
As reported by John Emmas this was causing build failure with
MSVC (on glib) because of inline issues. But that reminded me
that we may be taking this path even if the system implements
MemoryBarrier as a function, which is a waste. So, just remove
it.
Behdad Esfahbod
Oleg Oshmyan
Primiano Tucci
Konstantin Ritt
jfkthame
Werner Lemberg
Luis de Bethencourt
Khaled Hosny
Roozbeh Pournader
Anthony Carrico
Bradley Grainger
Chun-wei Fan