walkero
/
harfbuzz
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

[Docs] Usermanual; fill out Buffers chapter.

This commit is contained in:
Nathan Willis 2019-04-30 17:21:33 +01:00
parent 6d9a86ae75
commit 3b301c5ac6
1 changed files with 335 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

344
docs/usermanual-buffers-language-script-and-direction.xml
View File

@ -7,29 +7,37 @@
<chapter id="buffers-language-script-and-direction">
<title>Buffers, language, script and direction</title>
<para>
The input to HarfBuzz is a series of Unicode characters, stored in a
The input to the HarfBuzz shaper is a series of Unicode characters, stored in a
buffer. In this chapter, we'll look at how to set up a buffer with
the text that we want and then customize the properties of the
buffer.
the text that we want and how to customize the properties of the
buffer. We'll also look at a piece of lower-level machinery that
you will need to understand before proceeding: the functions that
HarfBuzz uses to retrieve Unicode information.
</para>
<para>
After shaping is complete, HarfBuzz puts its output back
into the buffer. But getting that output requires setting up a
face and a font first, so we will look at that in the next chapter
instead of here.
</para>
<section id="creating-and-destroying-buffers">
<title>Creating and destroying buffers</title>
<para>
As we saw in our <emphasis>Getting Started</emphasis> example, a
buffer is created and
initialized with <literal>hb_buffer_create()</literal>. This
initialized with <function>hb_buffer_create()</function>. This
produces a new, empty buffer object, instantiated with some
default values and ready to accept your Unicode strings.
</para>
<para>
HarfBuzz manages the memory of objects (such as buffers) that it
creates, so you don't have to. When you have finished working on
a buffer, you can call <literal>hb_buffer_destroy()</literal>:
a buffer, you can call <function>hb_buffer_destroy()</function>:
</para>
<programlisting language="C">
hb_buffer_t *buffer = hb_buffer_create();
hb_buffer_t *buf = hb_buffer_create();
...
hb_buffer_destroy(buffer);
hb_buffer_destroy(buf);
</programlisting>
<para>
This will destroy the object and free its associated memory -
@ -39,46 +47,350 @@
else destroying it, you should increase its reference count:
</para>
<programlisting language="C">
void somefunc(hb_buffer_t *buffer) {
buffer = hb_buffer_reference(buffer);
void somefunc(hb_buffer_t *buf) {
buf = hb_buffer_reference(buf);
...
</programlisting>
<para>
And then decrease it once you're done with it:
</para>
<programlisting language="C">
hb_buffer_destroy(buffer);
hb_buffer_destroy(buf);
}
</programlisting>
<para>
While we are on the subject of reference-counting buffers, it is
worth noting that an individual buffer can only meaningfully be
used by one thread at a time.
</para>
<para>
To throw away all the data in your buffer and start from scratch,
call <literal>hb_buffer_reset(buffer)</literal>. If you want to
call <function>hb_buffer_reset(buf)</function>. If you want to
throw away the string in the buffer but keep the options, you can
instead call <literal>hb_buffer_clear_contents(buffer)</literal>.
instead call <function>hb_buffer_clear_contents(buf)</function>.
</para>
</section>
<section id="adding-text-to-the-buffer">
<title>Adding text to the buffer</title>
<para>
Now we have a brand new HarfBuzz buffer. Let's start filling it
with text! From HarfBuzz's perspective, a buffer is just a stream
of Unicode code points, but your input string is probably in one of
the standard Unicode character encodings (UTF-8, UTF-16, UTF-32)
the standard Unicode character encodings (UTF-8, UTF-16, or
UTF-32). HarfBuzz provides convenience functions that accept
each of these encodings:
<function>hb_buffer_add_utf8()</function>,
<function>hb_buffer_add_utf16()</function>, and
<function>hb_buffer_add_utf32()</function>. Other than the
character encoding they accept, they function identically.
</para>
<para>
You can add UTF-8 text to a buffer by passing in the text array,
the array's length, an offset into the array for the first
character to add, and the length of the segment to add:
</para>
<programlisting language="C">
hb_buffer_add_utf8 (hb_buffer_t *buf,
const char *text,
int text_length,
unsigned int item_offset,
int item_length)
</programlisting>
<para>
So, in practice, you can say:
</para>
<programlisting language="C">
hb_buffer_add_utf8(buf, text, strlen(text), 0, strlen(text));
</programlisting>
<para>
This will append your new characters to
<parameter>buf</parameter>, not replace its existing
contents. Also, note that you can use <literal>-1</literal> in
place of the first instance of <function>strlen(text)</function>
if your text array is NULL-terminated. Similarly, you can also use
<literal>-1</literal> as the final argument want to add its full
contents.
</para>
<para>
Whatever start <parameter>item_offset</parameter> and
<parameter>item_length</parameter> you provide, HarfBuzz will also
attempt to grab the five characters <emphasis>before</emphasis>
the offset point and the five characters
<emphasis>after</emphasis> the designated end. These are the
before and after "context" segments, which are used internally
for HarfBuzz to make shaping decisions. They will not be part of
the final output, but they ensure that HarfBuzz's
script-specific shaping operations are correct. If there are
fewer than five characters available for the before or after
contexts, HarfBuzz will just grab what is there.
</para>
<para>
For longer text runs, such as full paragraphs, it might be
tempting to only add smaller sub-segments to a buffer and
shape them in piecemeal fashion. Generally, this is not a good
idea, however, because a lot of shaping decisions are
dependent on this context information. For example, in Arabic
and other connected scripts, HarfBuzz needs to know the code
points before and after each character in order to correctly
determine which glyph to return.
</para>
<para>
The safest approach is to add all of the text available, then
use <parameter>item_offset</parameter> and
<parameter>item_length</parameter> to indicate which characters you
want shaped, so that HarfBuzz has access to any context.
</para>
<para>
You can also add Unicode code points directly with
<function>hb_buffer_add_codepoints()</function>. The arguments
to this function are the same as those for the UTF
encodings. But it is particularly important to note that
HarfBuzz does not do validity checking on the text that is added
to a buffer. Invalid code points will be replaced, but it is up
to you to do any deep-sanity checking necessary.
</para>
</section>
<section id="setting-buffer-properties">
<title>Setting buffer properties</title>
<para>
Buffers containing input characters still need several
properties set before HarfBuzz can shape their text correctly.
</para>
</section>
<section id="what-about-the-other-scripts">
<title>What about the other scripts?</title>
<para>
Initially, all buffers are set to the
<literal>HB_BUFFER_CONTENT_TYPE_INVALID</literal> content
type. After adding text, the buffer should be set to
<literal>HB_BUFFER_CONTENT_TYPE_UNICODE</literal> instead, which
indicates that it contains un-shaped input
characters. After shaping, the buffer will have the
<literal>HB_BUFFER_CONTENT_TYPE_GLYPHS</literal> content type.
</para>
<para>
<function>hb_buffer_add_utf8()</function> and the
other UTF functions set the content type of their buffer
automatically. But if you are reusing a buffer you may want to
check its state with
<function>hb_buffer_get_content_type(buffer)</function>. If
necessary you can set the content type with
</para>
<programlisting language="C">
hb_buffer_set_content_type(buf, HB_BUFFER_CONTENT_TYPE_UNICODE);
</programlisting>
<para>
to prepare for shaping.
</para>
<para>
Buffers also need to carry information about the script,
language, and text direction of their contents. You can set
these properties individually:
</para>
<programlisting language="C">
hb_buffer_set_direction(buf, HB_DIRECTION_LTR);
hb_buffer_set_script(buf, HB_SCRIPT_LATIN);
hb_buffer_set_language(buf, hb_language_from_string("en", -1));
</programlisting>
<para>
However, since these properties are often the repeated for
multiple text runs, you can also save them in a
<literal>hb_segment_properties_t</literal> for reuse:
</para>
<programlisting language="C">
hb_segment_properties_t *savedprops;
hb_buffer_get_segment_properties (buf, savedprops);
...
hb_buffer_set_segment_properties (buf2, savedprops);
</programlisting>
<para>
HarfBuzz also provides getter functions to retrieve a buffer's
direction, script, and language properties individually.
</para>
<para>
HarfBuzz recognizes four text directions in
<type>hb_direction_t</type>: left-to-right
(<literal>HB_DIRECTION_LTR</literal>), right-to-left (<literal>HB_DIRECTION_RTL</literal>),
top-to-bottom (<literal>HB_DIRECTION_TTB</literal>), and
bottom-to-top (<literal>HB_DIRECTION_BTT</literal>). For the
script property, HarfBuzz uses identifiers based on the
<ulink
url="https://unicode.org/iso15924/">ISO-15924
standard</ulink>. For languages, HarfBuzz uses tags based on the
<ulink url="https://tools.ietf.org/html/bcp47">IETF BCP 47</ulink> standard.
</para>
<para>
Helper functions are provided to convert character strings into
the necessary script and language tag types.
</para>
<para>
Two additional buffer properties to be aware of are the
"invisible glyph" and the replacement code point. The
replacement code point is inserted into buffer output in place of
any invalid code points encountered in the input. By default, it
is the Unicode <literal>REPLACEMENT CHARACTER</literal> code
point, <literal>U+FFFD</literal> "&#xFFFD;". You can change this with
</para>
<programlisting language="C">
hb_buffer_set_replacement_codepoint(buf, replacement);
</programlisting>
<para>
The invisible glyph is used to replace all output glyphs that
are invisible. By default, the standard space character
<literal>U+0020</literal> is used; you can replace this (for
example, when using a font that provides script-specific
spaces) with
</para>
<programlisting language="C">
hb_buffer_set_invisible_glyph(buf, replacement);
</programlisting>
<para>
HarfBuzz supports a few additional flags you might want to set
on your buffer under certain circumstances. The
<literal>HB_BUFFER_FLAG_BOT</literal> and
<literal>HB_BUFFER_FLAG_EOT</literal> flags tell HarfBuzz
that the buffer represents the beginning or end (respectively)
of a text element (such as a paragraph or other block). Knowing
this allows HarfBuzz to apply certain contextual font features
when shaping, such as initial or final variants in connected
scripts.
</para>
<para>
<literal>HB_BUFFER_FLAG_PRESERVE_DEFAULT_IGNORABLES</literal>
tells HarfBuzz not to hide glyphs with the
<literal>Default_Ignorable</literal> property in Unicode. This
property designates control characters and other non-printing
code points, such as joiners and variation selectors. Normally
HarfBuzz replaces them in the output buffer with zero-width
space glyphs; setting this flag causes them to be printed,
which can be helpful for troubleshooting.
</para>
<para>
Conversely, setting the
<literal>HB_BUFFER_FLAG_REMOVE_DEFAULT_IGNORABLES</literal> flag
tells HarfBuzz to remove <literal>Default_Ignorable</literal>
glyphs from the output buffer entirely. Finally, setting the
<literal>HB_BUFFER_FLAG_DO_NOT_INSERT_DOTTED_CIRCLE</literal>
flag tells HarfBuzz not to insert the dotted-circle glyph
(<literal>U+25CC</literal>, "&#x25CC;"), which is normally
inserted into buffer output when broken character sequences are
encountered (such as combining marks that are not attached to a
base character).
</para>
</section>
<section id="customizing-unicode-functions">
<title>Customizing Unicode functions</title>
<para>
HarfBuzz requires some simple functions for accessing
information from the Unicode Character Database (such as the
<literal>General_Category</literal> (gc) and
<literal>Script</literal> (sc) properties) that is useful
for shaping, as well as some useful operations like composing and
decomposing code points.
</para>
<para>
At build time, HarfBuzz looks first for the GLib library. If
it is found, HarfBuzz will use GLib's Unicode functions by
default. If there is no GLib, HarfBuzz will fall back to its own
internal, lightweight set of Unicode functions instead.
</para>
<para>
If your program has access to other Unicode functions, however,
such as through a system library or application framework, you
might prefer to use those instead of the built-in
options. HarfBuzz supports this by implementing its Unicode
functions as a set of virtual methods that you can replace —
without otherwise affecting HarfBuzz's functionality.
</para>
<para>
The Unicode functions are specified in a structure called
<literal>unicode_funcs</literal> which is attached to each
buffer. But even though <literal>unicode_funcs</literal> is
associated with a <type>hb_buffer_t</type>, the functions
themselves are called by other HarfBuzz APIs that access
buffers, so it would be unwise for you to hook different
functions into different buffers.
</para>
<para>
In addition, you can mark your <literal>unicode_funcs</literal>
as immutable by calling
<function>hb_unicode_funcs_make_immutable (ufuncs)</function>. This is especially useful if your code is a
library or framework that will have its own client programs. By
marking your Unicode function choices as immutable, you prevent
your own client programs from changing the
<literal>unicode_funcs</literal> configuration and introducing
inconsistencies and errors downstream.
</para>
<para>
You can retrieve the Unicode-functions configuration for
your buffer by calling <function>hb_buffer_get_unicode_funcs()</function>:
</para>
<programlisting language="C">
hb_unicode_funcs_t *ufunctions;
ufunctions = hb_buffer_get_unicode_funcs(buf);
</programlisting>
<para>
The current version of <literal>unicode_funcs</literal> uses six functions:
</para>
<itemizedlist>
<listitem>
<para>
<function>hb_unicode_combining_class_func_t</function>:
returns the Canonical Combining Class of a code point.
</para>
</listitem>
<listitem>
<para>
<function>hb_unicode_general_category_func_t</function>:
returns the General Category (gc) of a code point.
</para>
</listitem>
<listitem>
<para>
<function>hb_unicode_mirroring_func_t</function>: returns
the Mirroring Glyph code point (for bi-directional
replacement) of a code point.
</para>
</listitem>
<listitem>
<para>
<function>hb_unicode_script_func_t</function>: returns the
Script (sc) property of a code point.
</para>
</listitem>
<listitem>
<para>
<function>hb_unicode_compose_func_t</function>: returns the
canonical composition of a sequence of two code points.
</para>
</listitem>
<listitem>
<para>
<function>hb_unicode_decompose_func_t</function>: returns
the canonical decomposition of a code point.
</para>
</listitem>
</itemizedlist>
<para>
Note, however, that future HarfBuzz releases may alter this set.
</para>
<para>
Each Unicode function has a corresponding setter, with which you
can assign a callback to your replacement function. For example,
to replace
<function>hb_unicode_general_category_func_t</function>, you can call
</para>
<programlisting language="C">
hb_unicode_funcs_set_general_category_func (*ufuncs, func, *user_data, destroy)
</programlisting>
<para>
Virtualizing this set of Unicode functions is primarily intended
to improve portability. There is no need for every client
program to make the effort to replace the default options, so if
you are unsure, do not feel any pressure to customize
<literal>unicode_funcs</literal>.
</para>
</section>
</chapter>