pcre2/README

README file for PCRE2 (Perl-compatible regular expression library)
------------------------------------------------------------------

PCRE2 is a re-implementation of the original PCRE library with an entirely new 
API. The latest release of PCRE2 is always available in three alternative
formats from:

FIXME: THIS WILL NOT BE THE CASE UNTIL THERE IS A FORMAL RELEASE.

  ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre2/pcre2-xxx.tar.gz
  ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre2/pcre2-xxx.tar.bz2
  ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre2/pcre2-xxx.zip

There is a mailing list for discussion about the development of PCRE (both the 
original and new APIs) at pcre-dev@exim.org. You can access the archives and
subscribe or manage your subscription here:

   https://lists.exim.org/mailman/listinfo/pcre-dev

Please read the NEWS file if you are upgrading from a previous release.
The contents of this README file are:

  The PCRE2 APIs
  Documentation for PCRE2
  Contributions by users of PCRE2
  Building PCRE2 on non-Unix-like systems
  Building PCRE2 without using autotools
  Building PCRE2 using autotools
  Retrieving configuration information
  Shared libraries
  Cross-compiling using autotools
  Making new tarballs
  Testing PCRE2
  Character tables
  File manifest


The PCRE2 APIs
--------------

PCRE2 is written in C, and it has its own API. There are three sets of
functions, one for the 8-bit library, which processes strings of bytes, one for
the 16-bit library, which processes strings of 16-bit values, and one for the
32-bit library, which processes strings of 32-bit values. As this is a new API, 
there as yet no C++ wrappers.

The distribution does contain a set of C wrapper functions for the 8-bit
library that are based on the POSIX regular expression API (see the pcre2posix
man page). These end up in the library called libpcre2posix. Note that this
just provides a POSIX calling interface to PCRE2; the regular expressions
themselves still follow Perl syntax and semantics. The POSIX API is restricted,
and does not give full access to all of PCRE2's facilities.

The header file for the POSIX-style functions is called pcre2posix.h. The
official POSIX name is regex.h, but I did not want to risk possible problems
with existing files of that name by distributing it that way. To use PCRE2 with
an existing program that uses the POSIX API, pcre2posix.h will have to be
renamed or pointed at by a link.

If you are using the POSIX interface to PCRE2 and there is already a POSIX
regex library installed on your system, as well as worrying about the regex.h
header file (as mentioned above), you must also take care when linking programs
to ensure that they link with PCRE2's libpcre2posix library. Otherwise they may
pick up the POSIX functions of the same name from the other library.

One way of avoiding this confusion is to compile PCRE2 with the addition of
-Dregcomp=PCRE2regcomp (and similarly for the other POSIX functions) to the
compiler flags (CFLAGS if you are using "configure" -- see below). This has the
effect of renaming the functions so that the names no longer clash. Of course,
you have to do the same thing for your applications, or write them using the
new names.


Documentation for PCRE2
----------------------

If you install PCRE2 in the normal way on a Unix-like system, you will end up
with a set of man pages whose names all start with "pcre2". The one that is
just called "pcre2" lists all the others. In addition to these man pages, the
PCRE2 documentation is supplied in two other forms:

  1. There are files called doc/pcre2.txt, doc/pcre2grep.txt, and
     doc/pcre2test.txt in the source distribution. The first of these is a
     concatenation of the text forms of all the section 3 man pages except the
     listing of pcre2demo.c and those that summarize individual functions. The
     other two are the text forms of the section 1 man pages for the pcre2grep
     and pcre2test commands. These text forms are provided for ease of scanning
     with text editors or similar tools. They are installed in
     <prefix>/share/doc/pcre2, where <prefix> is the installation prefix
     (defaulting to /usr/local).

  2. A set of files containing all the documentation in HTML form, hyperlinked
     in various ways, and rooted in a file called index.html, is distributed in
     doc/html and installed in <prefix>/share/doc/pcre2/html.


Building PCRE2 on non-Unix-like systems
--------------------------------------

For a non-Unix-like system, please read the comments in the file
NON-AUTOTOOLS-BUILD, though if your system supports the use of "configure" and
"make" you may be able to build PCRE2 using autotools in the same way as for
many Unix-like systems.

PCRE2 can also be configured using CMake, which can be run in various ways 
(command line, GUI, etc). This creates Makefiles, solution files, etc. The file
NON-AUTOTOOLS-BUILD has information about CMake.

PCRE2 has been compiled on many different operating systems. It should be
straightforward to build PCRE2 on any system that has a Standard C compiler and
library, because it uses only Standard C functions.


Building PCRE2 without using autotools
-------------------------------------

The use of autotools (in particular, libtool) is problematic in some
environments, even some that are Unix or Unix-like. See the NON-AUTOTOOLS-BUILD
file for ways of building PCRE2 without using autotools.


Building PCRE2 using autotools
-----------------------------

The following instructions assume the use of the widely used "configure; make;
make install" (autotools) process.

To build PCRE2 on system that supports autotools, first run the "configure"
command from the PCRE2 distribution directory, with your current directory set
to the directory where you want the files to be created. This command is a
standard GNU "autoconf" configuration script, for which generic instructions
are supplied in the file INSTALL.

Most commonly, people build PCRE2 within its own distribution directory, and in
this case, on many systems, just running "./configure" is sufficient. However,
the usual methods of changing standard defaults are available. For example:

CFLAGS='-O2 -Wall' ./configure --prefix=/opt/local

This command specifies that the C compiler should be run with the flags '-O2
-Wall' instead of the default, and that "make install" should install PCRE2
under /opt/local instead of the default /usr/local.

If you want to build in a different directory, just run "configure" with that
directory as current. For example, suppose you have unpacked the PCRE2 source
into /source/pcre2/pcre2-xxx, but you want to build it in
/build/pcre2/pcre2-xxx:

cd /build/pcre2/pcre2-xxx
/source/pcre2/pcre2-xxx/configure

PCRE2 is written in C and is normally compiled as a C library. However, it is
possible to build it as a C++ library, though the provided building apparatus
does not have any features to support this.

There are some optional features that can be included or omitted from the PCRE2
library. They are also documented in the pcre2build man page.

. By default, both shared and static libraries are built. You can change this
  by adding one of these options to the "configure" command:

  --disable-shared
  --disable-static

  (See also "Shared libraries on Unix-like systems" below.)

. By default, only the 8-bit library is built. If you add --enable-pcre2-16 to
  the "configure" command, the 16-bit library is also built. If you add
  --enable-pcre2-32 to the "configure" command, the 32-bit library is also built.
  If you want only the 16-bit or 32-bit library, use --disable-pcre2-8 to disable
  building the 8-bit library.

. If you want to include support for just-in-time compiling, which can give
  large performance improvements on certain platforms, add --enable-jit to the
  "configure" command. This support is available only for certain hardware
  architectures. If you try to enable it on an unsupported architecture, there
  will be a compile time error.  FIXME: NOT YET IMPLEMENTED.

. When JIT support is enabled, pcre2grep automatically makes use of it, unless
  you add --disable-pcre2grep-jit to the "configure" command.

. If you want to make use of the support for UTF-8 Unicode character strings in
  the 8-bit library, UTF-16 Unicode character strings in the 16-bit library,
  and UTF-32 Unicode character strings in the 32-bit library, you must add
  --enable-unicode to the "configure" command. Without it, the code for
  handling UTF-8, UTF-16 and UTF-8 is not included. It is not possible to
  configure one library with UTF support and the other without in the same
  configuration.
 
  Even when --enable-unicode is included, the use of a UTF encoding still has
  to be enabled by an option at run time. When PCRE2 is compiled with this
  option, its input can only either be ASCII or UTF-8/16/32, even when running
  on EBCDIC platforms. It is not possible to use both --enable-unicode and
  --enable-ebcdic at the same time.
  
  When --enable-unicode is specified, as well as supporting UTF strings, PCRE2
  includes support for the \P, \p, and \X sequences that recognize Unicode
  character properties. However, only the basic two-letter properties such as
  Lu are supported.

. You can build PCRE2 to recognize either CR or LF or the sequence CRLF or any
  of the preceding, or any of the Unicode newline sequences as indicating the
  end of a line. Whatever you specify at build time is the default; the caller
  of PCRE2 can change the selection at run time. The default newline indicator
  is a single LF character (the Unix standard). You can specify the default
  newline indicator by adding --enable-newline-is-cr or --enable-newline-is-lf
  or --enable-newline-is-crlf or --enable-newline-is-anycrlf or
  --enable-newline-is-any to the "configure" command, respectively.

  If you specify --enable-newline-is-cr or --enable-newline-is-crlf, some of
  the standard tests will fail, because the lines in the test files end with
  LF. Even if the files are edited to change the line endings, there are likely
  to be some failures. With --enable-newline-is-anycrlf or
  --enable-newline-is-any, many tests should succeed, but there may be some
  failures.

. By default, the sequence \R in a pattern matches any Unicode line ending
  sequence. This is independent of the option specifying what PCRE2 considers
  to be the end of a line (see above). However, the caller of PCRE2 can
  restrict \R to match only CR, LF, or CRLF. You can make this the default by
  adding --enable-bsr-anycrlf to the "configure" command (bsr = "backslash R").

. PCRE2 has a counter that limits the depth of nesting of parentheses in a
  pattern. This limits the amount of system stack that a pattern uses when it
  is compiled. The default is 250, but you can change it by setting, for
  example,

  --with-parens-nest-limit=500

. PCRE2 has a counter that can be set to limit the amount of resources it uses
  when matching a pattern. If the limit is exceeded during a match, the match
  fails. The default is ten million. You can change the default by setting, for
  example,

  --with-match-limit=500000

  on the "configure" command. This is just the default; individual calls to
  pcre2_match() can supply their own value. There is more discussion on the
  pcre2api man page.

. There is a separate counter that limits the depth of recursive function calls
  during a matching process. This also has a default of ten million, which is
  essentially "unlimited". You can change the default by setting, for example,

  --with-match-limit-recursion=500000

  Recursive function calls use up the runtime stack; running out of stack can
  cause programs to crash in strange ways. There is a discussion about stack
  sizes in the pcre2stack man page.

. In the 8-bit library, the default maximum compiled pattern size is around 
  64K. You can increase this by adding --with-link-size=3 to the "configure"
  command. PCRE2 then uses three bytes instead of two for offsets to different
  parts of the compiled pattern. In the 16-bit library, --with-link-size=3 is
  the same as --with-link-size=4, which (in both libraries) uses four-byte
  offsets. Increasing the internal link size reduces performance. In the 32-bit
  library, the link size setting is ignored, as 4-byte offsets are always used.

. You can build PCRE2 so that its internal match() function that is called from
  pcre2_match() does not call itself recursively. Instead, it uses memory
  blocks obtained from the heap to save data that would otherwise be saved on
  the stack. To build PCRE2 like this, use

  --disable-stack-for-recursion

  on the "configure" command. PCRE2 runs more slowly in this mode, but it may
  be necessary in environments with limited stack sizes. This applies only to
  the normal execution of the pcre2_match() function; if JIT support is being
  successfully used, it is not relevant. Equally, it does not apply to
  pcre2_dfa_match(), which does not use deeply nested recursion. There is a
  discussion about stack sizes in the pcre2stack man page.

. For speed, PCRE2 uses four tables for manipulating and identifying characters
  whose code point values are less than 256. By default, it uses a set of
  tables for ASCII encoding that is part of the distribution. If you specify

  --enable-rebuild-chartables

  a program called dftables is compiled and run in the default C locale when
  you obey "make". It builds a source file called pcre2_chartables.c. If you do
  not specify this option, pcre2_chartables.c is created as a copy of
  pcre2_chartables.c.dist. See "Character tables" below for further
  information.

. It is possible to compile PCRE2 for use on systems that use EBCDIC as their
  character code (as opposed to ASCII/Unicode) by specifying

  --enable-ebcdic

  This automatically implies --enable-rebuild-chartables (see above). However,
  when PCRE2 is built this way, it always operates in EBCDIC. It cannot support
  both EBCDIC and UTF-8/16/32. There is a second option, --enable-ebcdic-nl25,
  which specifies that the code value for the EBCDIC NL character is 0x25
  instead of the default 0x15.

. In environments where valgrind is installed, if you specify

  --enable-valgrind

  PCRE2 will use valgrind annotations to mark certain memory regions as
  unaddressable. This allows it to detect invalid memory accesses, and is
  mostly useful for debugging PCRE2 itself.

. In environments where the gcc compiler is used and lcov version 1.6 or above
  is installed, if you specify

  --enable-coverage

  the build process implements a code coverage report for the test suite. The
  report is generated by running "make coverage". If ccache is installed on
  your system, it must be disabled when building PCRE2 for coverage reporting.
  You can do this by setting the environment variable CCACHE_DISABLE=1 before
  running "make" to build PCRE2. There is more information about coverage
  reporting in the "pcre2build" documentation.

. The pcre2grep program currently supports only 8-bit data files, and so
  requires the 8-bit PCRE2 library. It is possible to compile pcre2grep to use
  libz and/or libbz2, in order to read .gz and .bz2 files (respectively), by
  specifying one or both of

  --enable-pcre2grep-libz
  --enable-pcre2grep-libbz2

  Of course, the relevant libraries must be installed on your system.

. The default size (in bytes) of the internal buffer used by pcre2grep can be
  set by, for example:

  --with-pcre2grep-bufsize=51200

  The value must be a plain integer. The default is 20480.

. It is possible to compile pcre2test so that it links with the libreadline
  or libedit libraries, by specifying, respectively,

  --enable-pcre2test-libreadline or --enable-pcre2test-libedit

  If this is done, when pcre2test's input is from a terminal, it reads it using
  the readline() function. This provides line-editing and history facilities.
  Note that libreadline is GPL-licenced, so if you distribute a binary of
  pcre2test linked in this way, there may be licensing issues. These can be
  avoided by linking with libedit (which has a BSD licence) instead.

  Enabling libreadline causes the -lreadline option to be added to the
  pcre2test build. In many operating environments with a sytem-installed
  readline library this is sufficient. However, in some environments (e.g. if
  an unmodified distribution version of readline is in use), it may be
  necessary to specify something like LIBS="-lncurses" as well. This is
  because, to quote the readline INSTALL, "Readline uses the termcap functions,
  but does not link with the termcap or curses library itself, allowing
  applications which link with readline the to choose an appropriate library."
  If you get error messages about missing functions tgetstr, tgetent, tputs,
  tgetflag, or tgoto, this is the problem, and linking with the ncurses library
  should fix it.

The "configure" script builds the following files for the basic C library:

. Makefile             the makefile that builds the library
. src/config.h         build-time configuration options for the library
. src/pcre2.h          the public PCRE2 header file
. pcre2-config          script that shows the building settings such as CFLAGS
                         that were set for "configure"
. libpcre2-8.pc        ) 
. libpcre2-16.pc       ) data for the pkg-config command
. libpcre2-32.pc       )
. libpcre2-posix.pc    )
. libtool              script that builds shared and/or static libraries

Versions of config.h and pcre2.h are distributed in the src directory of PCRE2
tarballs under the names config.h.generic and pcre2.h.generic. These are
provided for those who have to build PCRE2 without using "configure" or CMake.
If you use "configure" or CMake, the .generic versions are not used.

The "configure" script also creates config.status, which is an executable
script that can be run to recreate the configuration, and config.log, which
contains compiler output from tests that "configure" runs.

Once "configure" has run, you can run "make". This builds whichever of the
libraries libpcre2-8, libpcre2-16 and libpcre2-32 are configured, and a test
program called pcre2test. If you enabled JIT support with --enable-jit, another
test program called pcre2_jit_test is built as well. FIXME: still to be
implemented. If the 8-bit library is built, libpcre2-posix and the pcre2grep
command are also built.

The command "make check" runs all the appropriate tests. Details of the PCRE2
tests are given below in a separate section of this document.

You can use "make install" to install PCRE2 into live directories on your
system. The following are installed (file names are all relative to the
<prefix> that is set when "configure" is run):

  Commands (bin):
    pcre2test
    pcre2grep (if 8-bit support is enabled)
    pcre2-config

  Libraries (lib):
    libpcre2-8      (if 8-bit support is enabled)
    libpcre2-16     (if 16-bit support is enabled)
    libpcre2-32     (if 32-bit support is enabled)
    libpcre2-posix  (if 8-bit support is enabled)

  Configuration information (lib/pkgconfig):
    libpcre2-8.pc
    libpcre2-16.pc
    libpcre2-32.pc
    libpcre2-posix.pc

  Header files (include):
    pcre2.h
    pcre2posix.h

  Man pages (share/man/man{1,3}):
    pcre2grep.1
    pcre2test.1
    pcre2-config.1
    pcre2.3
    pcre2*.3 (lots more pages, all starting "pcre2")

  HTML documentation (share/doc/pcre2/html):
    index.html
    *.html (lots more pages, hyperlinked from index.html)

  Text file documentation (share/doc/pcre2):
    AUTHORS
    COPYING
    ChangeLog
    LICENCE
    NEWS
    README
    pcre2.txt         (a concatenation of the man(3) pages)
    pcre2test.txt     the pcre2test man page
    pcre2grep.txt     the pcre2grep man page
    pcre2-config.txt  the pcre2-config man page

If you want to remove PCRE2 from your system, you can run "make uninstall".
This removes all the files that "make install" installed. However, it does not
remove any directories, because these are often shared with other programs.


Retrieving configuration information
------------------------------------

Running "make install" installs the command pcre2-config, which can be used to
recall information about the PCRE2 configuration and installation. For example:

  pcre2-config --version

prints the version number, and

  pcre2-config --libs8

outputs information about where the 8-bit library is installed. This command
can be included in makefiles for programs that use PCRE2, saving the programmer
from having to remember too many details. Run pcre2-config with no arguments to 
obtain a list of possible arguments.

The pkg-config command is another system for saving and retrieving information
about installed libraries. Instead of separate commands for each library, a
single command is used. For example:

  pkg-config --libs libpcre2-16

The data is held in *.pc files that are installed in a directory called
<prefix>/lib/pkgconfig.


Shared libraries
----------------

The default distribution builds PCRE2 as shared libraries and static libraries,
as long as the operating system supports shared libraries. Shared library
support relies on the "libtool" script which is built as part of the
"configure" process.

The libtool script is used to compile and link both shared and static
libraries. They are placed in a subdirectory called .libs when they are newly
built. The programs pcre2test and pcre2grep are built to use these uninstalled
libraries (by means of wrapper scripts in the case of shared libraries). When
you use "make install" to install shared libraries, pcre2grep and pcre2test are
automatically re-built to use the newly installed shared libraries before being
installed themselves. However, the versions left in the build directory still
use the uninstalled libraries.

To build PCRE2 using static libraries only you must use --disable-shared when
configuring it. For example:

./configure --prefix=/usr/gnu --disable-shared

Then run "make" in the usual way. Similarly, you can use --disable-static to
build only shared libraries.


Cross-compiling using autotools
-------------------------------

You can specify CC and CFLAGS in the normal way to the "configure" command, in
order to cross-compile PCRE2 for some other host. However, you should NOT
specify --enable-rebuild-chartables, because if you do, the dftables.c source
file is compiled and run on the local host, in order to generate the inbuilt
character tables (the pcre2_chartables.c file). This will probably not work,
because dftables.c needs to be compiled with the local compiler, not the cross
compiler.

When --enable-rebuild-chartables is not specified, pcre2_chartables.c is
created by making a copy of pcre2_chartables.c.dist, which is a default set of
tables that assumes ASCII code. Cross-compiling with the default tables should
not be a problem.

If you need to modify the character tables when cross-compiling, you should
move pcre2_chartables.c.dist out of the way, then compile dftables.c by hand
and run it on the local host to make a new version of pcre2_chartables.c.dist.
Then when you cross-compile PCRE2 this new version of the tables will be used.


Making new tarballs
-------------------

The command "make dist" creates three PCRE2 tarballs, in tar.gz, tar.bz2, and
zip formats. The command "make distcheck" does the same, but then does a trial
build of the new distribution to ensure that it works.

If you have modified any of the man page sources in the doc directory, you
should first run the PrepareRelease script before making a distribution. This
script creates the .txt and HTML forms of the documentation from the man pages.


Testing PCRE2
------------

To test the basic PCRE2 library on a Unix-like system, run the RunTest script.
There is another script called RunGrepTest that tests the options of the
pcre2grep command. When JIT support is enabled, another test program called
pcre2_jit_test is built. Both the scripts and all the program tests are run if
you obey "make check". For other environments, see the instructions in
NON-AUTOTOOLS-BUILD.

The RunTest script runs the pcre2test test program (which is documented in its
own man page) on each of the relevant testinput files in the testdata
directory, and compares the output with the contents of the corresponding
testoutput files. RunTest uses a file called testtry to hold the main output
from pcre2test. Other files whose names begin with "test" are used as working
files in some tests.

Some tests are relevant only when certain build-time options were selected. For
example, the tests for UTF-8/16/32 support are run only if --enable-unicode was
used. RunTest outputs a comment when it skips a test.

Many of the tests that are not skipped are run twice if JIT support is
available. On the second run, JIT compilation is forced. This testing can be
suppressed by putting "nojit" on the RunTest command line.

The entire set of tests is run once for each of the 8-bit, 16-bit and 32-bit
libraries that are enabled. If you want to run just one set of tests, call
RunTest with either the -8, -16 or -32 option.

If valgrind is installed, you can run the tests under it by putting "valgrind"
on the RunTest command line. To run pcre2test on just one or more specific test
files, give their numbers as arguments to RunTest, for example:

  RunTest 2 7 11

You can also specify ranges of tests such as 3-6 or 3- (meaning 3 to the
end), or a number preceded by ~ to exclude a test. For example:

  Runtest 3-15 ~10

This runs tests 3 to 15, excluding test 10, and just ~13 runs all the tests
except test 13. Whatever order the arguments are in, the tests are always run
in numerical order.

You can also call RunTest with the single argument "list" to cause it to output
a list of tests.

The first two tests can always be run, as they expect only plain text strings
(not UTF) and make no use of Unicode properties. The first test file can be fed
directly into the perltest.pl script to check that Perl gives the same results.
The only difference you should see is in the first few lines, where the Perl
version is given instead of the PCRE2 version. The second set of tests check
auxiliary functions, error detection, and run-time flags that are specific to
PCRE2, as well as the POSIX wrapper API. It also uses the debugging flags to
check some of the internals of pcre2_compile().

If you build PCRE2 with a locale setting that is not the standard C locale, the
character tables may be different (see next paragraph). In some cases, this may
cause failures in the second set of tests. For example, in a locale where the
isprint() function yields TRUE for characters in the range 128-255, the use of
[:isascii:] inside a character class defines a different set of characters, and
this shows up in this test as a difference in the compiled code, which is being
listed for checking. Where the comparison test output contains [\x00-\x7f] the
test will contain [\x00-\xff], and similarly in some other cases. This is not a
bug in PCRE2.

The third set of tests checks pcre2_maketables(), the facility for building a
set of character tables for a specific locale and using them instead of the
default tables. The script uses the "locale" command to check for the 
availability of the "fr_FR", "french", or "fr" locale, and uses the first one
that it finds. If the "locale" command fails, or if its output doesn't include
"fr_FR", "french", or "fr" in the list of available locales, the third test
cannot be run, and a comment is output to say why. If running this test
produces an error like this

  ** Failed to set locale "fr_FR"

it means that the given locale is not available on your system, despite being
listed by "locale". This does not mean that PCRE2 is broken. There are three
alternative output files for the third test, because three different versions
of the French locale have been encountered. The test passes if its output
matches any one of them.

The fourth and fifth tests check UTF and Unicode property support, the fourth
being compatible with the perltest.pl script, and the fifth checking 
PCRE2-specific things.

The sixth and seventh tests check the pcre2_dfa_match() alternative matching
function, in non-UTF mode and UTF-mode with Unicode property support,
respectively.

The eighth test checks some internal offsets and code size features; it is
run only when the default "link size" of 2 is set (in other cases the sizes
change) and when Unicode support is enabled.

The ninth and tenth tests are run only in 8-bit mode, and the eleventh and
twelfth tests are run only in 16-bit and 32-bit modes. These are tests that
generate different output in 8-bit mode. Each pair are for general cases and
Unicode support, respectively. The thirteenth test checks the handling of 
non-UTF characters greater than 255 by pcre2_dfa_match() in 16-bit and 32-bit 
modes.

The fourteenth test is run only when JIT support is not available, and the
fifteenth test is run only when JIT support is available. They test some
JIT-specific features such as information output from pcre2test about JIT
compilation.

The sixteenth and seventeenth tests are run only in 8-bit mode. They check the 
POSIX interface to the 8-bit library, withouth and with Unicode support, 
respectively.


Character tables
----------------

For speed, PCRE2 uses four tables for manipulating and identifying characters
whose code point values are less than 256. By default, a set of tables that is
built into the library is used. The pcre2_maketables() function can be called
by an application to create a new set of tables in the current locale. This are
passed to PCRE2 by calling pcre2_set_character_tables() to put a pointer into a
compile context.

The source file called pcre2_chartables.c contains the default set of tables.
By default, this is created as a copy of pcre2_chartables.c.dist, which
contains tables for ASCII coding. However, if --enable-rebuild-chartables is
specified for ./configure, a different version of pcre2_chartables.c is built
by the program dftables (compiled from dftables.c), which uses the ANSI C
character handling functions such as isalnum(), isalpha(), isupper(),
islower(), etc. to build the table sources. This means that the default C
locale which is set for your system will control the contents of these default
tables. You can change the default tables by editing pcre2_chartables.c and
then re-building PCRE2. If you do this, you should take care to ensure that the
file does not get automatically re-generated. The best way to do this is to
move pcre2_chartables.c.dist out of the way and replace it with your customized
tables.

When the dftables program is run as a result of --enable-rebuild-chartables,
it uses the default C locale that is set on your system. It does not pay
attention to the LC_xxx environment variables. In other words, it uses the
system's default locale rather than whatever the compiling user happens to have
set. If you really do want to build a source set of character tables in a
locale that is specified by the LC_xxx variables, you can run the dftables
program by hand with the -L option. For example:

  ./dftables -L pcre2_chartables.c.special

The first two 256-byte tables provide lower casing and case flipping functions,
respectively. The next table consists of three 32-byte bit maps which identify
digits, "word" characters, and white space, respectively. These are used when
building 32-byte bit maps that represent character classes for code points less
than 256. The final 256-byte table has bits indicating various character types,
as follows:

    1   white space character
    2   letter
    4   decimal digit
    8   hexadecimal digit
   16   alphanumeric or '_'
  128   regular expression metacharacter or binary zero

You should not alter the set of characters that contain the 128 bit, as that
will cause PCRE2 to malfunction.


File manifest
-------------

The distribution should contain the files listed below. 

(A) Source files for the PCRE2 library functions and their headers are found in 
    the src directory:

  src/dftables.c           auxiliary program for building pcre2_chartables.c
                           when --enable-rebuild-chartables is specified

  src/pcre2_chartables.c.dist  a default set of character tables that assume
                           ASCII coding; unless --enable-rebuild-chartables is
                           specified, used by copying to pcre2_chartables.c

  src/pcre2posix.c         )
  src/pcre2_auto_possess.c ) 
  src/pcre2_compile.c      )
  src/pcre2_config.c       )
  src/pcre2_context.c      ) 
  src/pcre2_dfa_match.c    )
  src/pcre2_error.c        ) 
  src/pcre2_exec.c         )
  src/pcre2_jit_compile.c  )
  src/pcre2_jit_match.c    ) sources for the functions in the library,  
  src/pcre2_jit_misc.c     )   and some internal functions that they use
  src/pcre2_maketables.c   )
  src/pcre2_match.c        )
  src/pcre2_match_data.c   )  
  src/pcre2_newline.c      )
  src/pcre2_ord2utf.c      )
  src/pcre2_pattern_info.c )
  src/pcre2_string_utils.c )
  src/pcre2_study.c        )
  src/pcre2_substring.c    ) 
  src/pcre2_tables.c       )
  src/pcre2_ucd.c          )
  src/pcre2_valid_utf.c    )
  src/pcre2_xclass.c       )

  src/pcre2_printint.c     debugging function that is used by pcre2test,

  src/config.h.in          template for config.h, when built by "configure"
  src/pcre2.h.in           template for pcre2.h when built by "configure"
  src/pcre2posix.h         header for the external POSIX wrapper API
  src/pcre2_internal.h     header for internal use
  src/pcre2_intmodedep.h   a mode-specific internal header
  src/pcre2_ucp.h          header for Unicode property handling

  sljit/*                  16 files that make up the JIT compiler FIXME

(B) Source files for programs that use PCRE2:

  src/pcre2demo.c          simple demonstration of coding calls to PCRE2
  src/pcre2grep.c          source of a grep utility that uses PCRE2
  src/pcre2test.c          comprehensive test program

(C) Auxiliary files:

  132html                  script to turn "man" pages into HTML
  AUTHORS                  information about the author of PCRE2
  ChangeLog                log of changes to the code
  CleanTxt                 script to clean nroff output for txt man pages
  Detrail                  script to remove trailing spaces
  HACKING                  some notes about the internals of PCRE2
  INSTALL                  generic installation instructions
  LICENCE                  conditions for the use of PCRE2
  COPYING                  the same, using GNU's standard name
  Makefile.in              ) template for Unix Makefile, which is built by
                           )   "configure"
  Makefile.am              ) the automake input that was used to create
                           )   Makefile.in
  NEWS                     important changes in this release
  NON-AUTOTOOLS-BUILD      notes on building PCRE2 without using autotools
  PrepareRelease           script to make preparations for "make dist"
  README                   this file
  RunTest                  a Unix shell script for running tests
  RunGrepTest              a Unix shell script for pcre2grep tests
  aclocal.m4               m4 macros (generated by "aclocal")
  config.guess             ) files used by libtool,
  config.sub               )   used only when building a shared library
  configure                a configuring shell script (built by autoconf)
  configure.ac             ) the autoconf input that was used to build
                           )   "configure" and config.h
  depcomp                  ) script to find program dependencies, generated by
                           )   automake
  doc/*.3                  man page sources for PCRE2
  doc/*.1                  man page sources for pcre2grep and pcre2test
  doc/index.html.src       the base HTML page
  doc/html/*               HTML documentation
  doc/pcre2.txt            plain text version of the man pages
  doc/pcre2test.txt        plain text documentation of test program
  doc/perltest.txt         plain text documentation of Perl test program
  install-sh               a shell script for installing files
  libpcre2-8.pc.in         template for libpcre2-8.pc for pkg-config
  libpcre2-16.pc.in        template for libpcre2-16.pc for pkg-config
  libpcre2-32.pc.in        template for libpcre2-32.pc for pkg-config
  libpcre2posix.pc.in      template for libpcre2posix.pc for pkg-config
  ltmain.sh                file used to build a libtool script
  missing                  ) common stub for a few missing GNU programs while
                           )   installing, generated by automake
  mkinstalldirs            script for making install directories
  perltest.pl              Perl test program
  pcre2-config.in          source of script which retains PCRE2 information
  pcre2_jit_test.c         test program for the JIT compiler
  testdata/testinput*      test data for main library tests
  testdata/testoutput*     expected test results
  testdata/grep*           input and output for pcre2grep tests
  testdata/*               other supporting test files

(D) Auxiliary files for cmake support

  cmake/COPYING-CMAKE-SCRIPTS
  cmake/FindPackageHandleStandardArgs.cmake
  cmake/FindEditline.cmake
  cmake/FindReadline.cmake
  CMakeLists.txt
  config-cmake.h.in

(E) Auxiliary files for VPASCAL  FIXME FIXME

  makevp.bat
  makevp_c.txt
  makevp_l.txt
  pcre2gexp.pas

(F) Auxiliary files for building PCRE2 "by hand"

  pcre2.h.generic         ) a version of the public PCRE2 header file
                          )   for use in non-"configure" environments
  config.h.generic        ) a version of config.h for use in non-"configure"
                          )   environments

(F) Miscellaneous

  RunTest.bat            a script for running tests under Windows  FIXME

Philip Hazel
Email local part: ph10
Email domain: cam.ac.uk
Last updated: 27 October 2014