441 lines
21 KiB
Plaintext
441 lines
21 KiB
Plaintext
MAINTENANCE README FOR PCRE2
|
|
============================
|
|
|
|
The files in the "maint" directory of the PCRE2 source contain data, scripts,
|
|
and programs that are used for the maintenance of PCRE2, but which do not form
|
|
part of the PCRE2 distribution tarballs. This document describes these files
|
|
and also contains some notes for maintainers. Its contents are:
|
|
|
|
Files in the maint directory
|
|
Updating to a new Unicode release
|
|
Preparing for a PCRE2 release
|
|
Making a PCRE2 release
|
|
Long-term ideas (wish list)
|
|
|
|
|
|
Files in the maint directory
|
|
============================
|
|
|
|
GenerateUtt.py A Python script to generate part of the pcre2_tables.c file
|
|
that contains Unicode script names in a long string with
|
|
offsets, which is tedious to maintain by hand.
|
|
|
|
ManyConfigTests A shell script that runs "configure, make, test" a number of
|
|
times with different configuration settings.
|
|
|
|
MultiStage2.py A Python script that generates the file pcre2_ucd.c from six
|
|
Unicode data files, which are themselves downloaded from the
|
|
Unicode web site. Run this script in the "maint" directory.
|
|
The generated file is written to stdout. It contains the
|
|
tables for a 2-stage lookup of Unicode properties, along with
|
|
some auxiliary tables.
|
|
|
|
pcre2_chartables.c.non-standard
|
|
This is a set of character tables that came from a Windows
|
|
system. It has characters greater than 128 that are set as
|
|
spaces, amongst other things. I kept it so that it can be
|
|
used for testing from time to time.
|
|
|
|
README This file.
|
|
|
|
Unicode.tables The files in this directory were downloaded from the Unicode
|
|
web site. They contain information about Unicode characters
|
|
and scripts. The ones used by the MultiStage2.py script are
|
|
CaseFolding.txt, DerivedGeneralCategory.txt, Scripts.txt,
|
|
ScriptExtensions.txt, GraphemeBreakProperty.txt, and
|
|
emoji-data.txt. I've kept UnicodeData.txt (which is no longer
|
|
used by the script) because it is useful occasionally for
|
|
manually looking up the details of certain characters.
|
|
However, note that character names in this file such as
|
|
"Arabic sign sanah" do NOT mean that the character is in a
|
|
particular script (in this case, Arabic). Scripts.txt and
|
|
ScriptExtensions.txt are where to look for script information.
|
|
|
|
ucptest.c A short C program for testing the Unicode property macros
|
|
that do lookups in the pcre2_ucd.c data, mainly useful after
|
|
rebuilding the Unicode property table. Compile and run this in
|
|
the "maint" directory (see comments at its head).
|
|
|
|
ucptestdata A directory containing two files, testinput1 and testoutput1,
|
|
to use in conjunction with the ucptest program.
|
|
|
|
utf8.c A short, freestanding C program for converting a Unicode code
|
|
point into a sequence of bytes in the UTF-8 encoding, and vice
|
|
versa. If its argument is a hex number such as 0x1234, it
|
|
outputs a list of the equivalent UTF-8 bytes. If its argument
|
|
is a sequence of concatenated UTF-8 bytes (e.g. e188b4) it
|
|
treats them as a UTF-8 character and outputs the equivalent
|
|
code point in hex.
|
|
|
|
|
|
Updating to a new Unicode release
|
|
=================================
|
|
|
|
When there is a new release of Unicode, the files in Unicode.tables must be
|
|
refreshed from the web site. If the new version of Unicode adds new character
|
|
scripts, the source file pcre2_ucp.h and both the MultiStage2.py and the
|
|
GenerateUtt.py scripts must be edited to add the new names. I have been adding
|
|
each new group at the end of the relevant list, with a comment. Note also that
|
|
both the pcre2syntax.3 and pcre2pattern.3 man pages contain lists of Unicode
|
|
script names.
|
|
|
|
MultiStage2.py has two lists: the full names and the abbreviations that are
|
|
found in the ScriptExtensions.txt file. A list of script names and their
|
|
abbreviations s can be found in the PropertyValueAliases.txt file on the
|
|
Unicode web site. There is also a Wikipedia page that lists them, and notes the
|
|
Unicode version in which they were introduced:
|
|
|
|
http://en.wikipedia.org/wiki/Unicode_scripts#Table_of_Unicode_scripts
|
|
|
|
Once the script name lists have been updated, MultiStage2.py can be run to
|
|
generate a new version of pcre2_ucd.c, and GenerateUtt.py can be run to
|
|
generate the tricky tables for inclusion in pcre2_tables.c (which must be
|
|
hand-edited). If MultiStage2.py gives the error "ValueError: list.index(x): x
|
|
not in list", the cause is usually a missing (or misspelt) name in one of the
|
|
lists of scripts.
|
|
|
|
The ucptest program can be compiled and used to check that the new tables in
|
|
pcre2_ucd.c work properly, using the data files in ucptestdata to check a
|
|
number of test characters. The source file ucptest.c should also be updated
|
|
whenever new Unicode script names are added, and adding a few tests for new
|
|
scripts is a good idea.
|
|
|
|
|
|
Preparing for a PCRE2 release
|
|
=============================
|
|
|
|
This section contains a checklist of things that I consult before building a
|
|
distribution for a new release.
|
|
|
|
. Ensure that the version number and version date are correct in configure.ac.
|
|
|
|
. Update the library version numbers in configure.ac according to the rules
|
|
given below.
|
|
|
|
. If new build options or new source files have been added, ensure that they
|
|
are added to the CMake files as well as to the autoconf files. The relevant
|
|
files are CMakeLists.txt and config-cmake.h.in. After making a release
|
|
tarball, test it out with CMake if there have been changes here.
|
|
|
|
. Run ./autogen.sh to ensure everything is up-to-date.
|
|
|
|
. Compile and test with many different config options, and combinations of
|
|
options. Also, test with valgrind by running "RunTest valgrind" and
|
|
"RunGrepTest valgrind" (which takes quite a long time). The script
|
|
maint/ManyConfigTests now encapsulates this testing. It runs tests with
|
|
different configurations, and it also runs some of them with valgrind, all of
|
|
which can take quite some time.
|
|
|
|
. Run tests in both 32-bit and 64-bit environments if possible.
|
|
|
|
. Run tests with two or more different compilers (e.g. clang and gcc), and
|
|
make use of -fsanitize=address and friends where possible. For gcc,
|
|
-fsanitize=undefined -std=gnu99 picks up undefined behaviour at runtime, but
|
|
needs -fno-sanitize=shift to get rid of warnings for shifts of negative
|
|
numbers in the JIT compiler. For clang, -fsanitize=address,undefined,integer
|
|
can be used but -fno-sanitize=alignment,shift,unsigned-integer-overflow must
|
|
be added when compiling with JIT. Another useful clang option is
|
|
-fsanitize=signed-integer-overflow
|
|
|
|
. Do a test build using CMake.
|
|
|
|
. Run perltest.sh on the test data for tests 1 and 4. The output should match
|
|
the PCRE2 test output, apart from the version identification at the start of
|
|
each test. Sometimes there are other differences in test 4 if PCRE2 and Perl
|
|
are using different Unicode releases. The other tests are not Perl-compatible
|
|
(they use various PCRE2-specific features or options).
|
|
|
|
. It is possible to test with the emulated memmove() function by undefining
|
|
HAVE_MEMMOVE and HAVE_BCOPY in config.h, though I do not do this often.
|
|
|
|
. Documentation: check AUTHORS, ChangeLog (check version and date), LICENCE,
|
|
NEWS (check version and date), NON-AUTOTOOLS-BUILD, and README. Many of these
|
|
won't need changing, but over the long term things do change.
|
|
|
|
. I used to test new releases myself on a number of different operating
|
|
systems. For example, on Solaris it is helpful to test using Sun's cc
|
|
compiler as a change from gcc. Adding -xarch=v9 to the cc options does a
|
|
64-bit test, but it also needs -S 64 for pcre2test to increase the stack size
|
|
for test 2. Since I retired I can no longer do much of this, but instead I
|
|
rely on putting out release candidates for folks on the pcre-dev list to
|
|
test.
|
|
|
|
. The buildbots at http://buildfarm.opencsw.org/ do some automated testing
|
|
of PCRE2 and should be checked before putting out a release.
|
|
|
|
|
|
Updating version info for libtool
|
|
=================================
|
|
|
|
This set of rules for updating library version information came from a web page
|
|
whose URL I have forgotten. The version information consists of three parts:
|
|
(current, revision, age).
|
|
|
|
1. Start with version information of 0:0:0 for each libtool library.
|
|
|
|
2. Update the version information only immediately before a public release of
|
|
your software. More frequent updates are unnecessary, and only guarantee
|
|
that the current interface number gets larger faster.
|
|
|
|
3. If the library source code has changed at all since the last update, then
|
|
increment revision; c:r:a becomes c:r+1:a.
|
|
|
|
4. If any interfaces have been added, removed, or changed since the last
|
|
update, increment current, and set revision to 0.
|
|
|
|
5. If any interfaces have been added since the last public release, then
|
|
increment age.
|
|
|
|
6. If any interfaces have been removed or changed since the last public
|
|
release, then set age to 0.
|
|
|
|
The following explanation may help in understanding the above rules a bit
|
|
better. Consider that there are three possible kinds of reaction from users to
|
|
changes in a shared library:
|
|
|
|
1. Programs using the previous version may use the new version as a drop-in
|
|
replacement, and programs using the new version can also work with the
|
|
previous one. In other words, no recompiling nor relinking is needed. In
|
|
this case, increment revision only, don't touch current or age.
|
|
|
|
2. Programs using the previous version may use the new version as a drop-in
|
|
replacement, but programs using the new version may use APIs not present in
|
|
the previous one. In other words, a program linking against the new version
|
|
may fail if linked against the old version at run time. In this case, set
|
|
revision to 0, increment current and age.
|
|
|
|
3. Programs may need to be changed, recompiled, relinked in order to use the
|
|
new version. Increment current, set revision and age to 0.
|
|
|
|
|
|
Making a PCRE2 release
|
|
======================
|
|
|
|
Run PrepareRelease and commit the files that it changes (by removing trailing
|
|
spaces). The first thing this script does is to run CheckMan on the man pages;
|
|
if it finds any markup errors, it reports them and then aborts.
|
|
|
|
Once PrepareRelease has run clean, run "make distcheck" to create the tarballs
|
|
and the zipball. Double-check with "svn status", then create an SVN tagged
|
|
copy:
|
|
|
|
svn copy svn://vcs.exim.org/pcre2/code/trunk \
|
|
svn://vcs.exim.org/pcre2/code/tags/pcre2-10.xx
|
|
|
|
When the new release is out, don't forget to tell webmaster@pcre.org and the
|
|
mailing list. Also, update the list of version numbers in Bugzilla
|
|
(administration > products > PCRE > Edit versions).
|
|
|
|
|
|
Future ideas (wish list)
|
|
========================
|
|
|
|
This section records a list of ideas so that they do not get forgotten. They
|
|
vary enormously in their usefulness and potential for implementation. Some are
|
|
very sensible; some are rather wacky. Some have been on this list for years.
|
|
|
|
. Optimization
|
|
|
|
There are always ideas for new optimizations so as to speed up pattern
|
|
matching. Most of them try to save work by recognizing a non-match without
|
|
having to scan all the possibilities. These are some that I've recorded:
|
|
|
|
* /((A{0,5}){0,5}){0,5}(something complex)/ on a non-matching string is very
|
|
slow, though Perl is fast. Can we speed up somehow? Convert to {0,125}?
|
|
OTOH, this is pathological - the user could easily fix it.
|
|
|
|
* Turn ={4} into ==== ? (for speed). I once did an experiment, and it seems
|
|
to have little effect, and maybe makes things worse.
|
|
|
|
* "Ends with literal string" - note that a single character doesn't gain much
|
|
over the existing "required code unit" feature that just remembers one code
|
|
unit.
|
|
|
|
* Remember an initial string rather than just 1 code unit.
|
|
|
|
* A required code unit from alternatives - not just the last unit, but an
|
|
earlier one if common to all alternatives.
|
|
|
|
* Friedl contains other ideas.
|
|
|
|
* The code does not set initial code unit flags for Unicode property types
|
|
such as \p; I don't know how much benefit there would be for, for example,
|
|
setting the bits for 0-9 and all values >= xC0 (in 8-bit mode) when a
|
|
pattern starts with \p{N}.
|
|
|
|
. If Perl gets to a consistent state over the settings of capturing sub-
|
|
patterns inside repeats, see if we can match it. One example of the
|
|
difference is the matching of /(main(O)?)+/ against mainOmain, where PCRE2
|
|
leaves $2 set. In Perl, it's unset. Changing this in PCRE2 will be very hard
|
|
because I think it needs much more state to be remembered.
|
|
|
|
. A feature to suspend a match via a callout was once requested.
|
|
|
|
. An option to convert results into character offsets and character lengths.
|
|
|
|
. An option for pcre2grep to scan only the start of a file. I am not keen -
|
|
this is the job of "head".
|
|
|
|
. A (non-Unix) user wanted pcregrep options to (a) list a file name just once,
|
|
preceded by a blank line, instead of adding it to every matched line, and (b)
|
|
support --outputfile=name.
|
|
|
|
. Define a union for the results from pcre2_pattern_info().
|
|
|
|
. Provide a "random access to the subject" facility so that the way in which it
|
|
is stored is independent of PCRE2. For efficiency, it probably isn't possible
|
|
to switch this dynamically. It would have to be specified when PCRE2 was
|
|
compiled. PCRE2 would then call a function every time it wanted a character.
|
|
|
|
. pcre2grep: add -rs for a sorted recurse. Having to store file names and sort
|
|
them will of course slow it down.
|
|
|
|
. Someone suggested --disable-callout to save code space when callouts are
|
|
never wanted. This seems rather marginal.
|
|
|
|
. A user suggested a parameter to limit the length of string matched, for
|
|
example if the parameter is N, the current match should fail if the matched
|
|
substring exceeds N. This could apply to both match functions. The value
|
|
could be a new field in the match context. Compare the offset_limit feature,
|
|
which limits where a match must start.
|
|
|
|
. Write a function that generates random matching strings for a compiled
|
|
pattern.
|
|
|
|
. Pcre2grep: an option to specify the output line separator, either as a string
|
|
or select from a fixed list. This is not straightforward, because at the
|
|
moment it outputs whatever is in the input file.
|
|
|
|
. Improve the code for duplicate checking in pcre2_dfa_match(). An incomplete,
|
|
non-thread-safe patch showed that this can help performance for patterns
|
|
where there are many alternatives. However, a simple thread-safe
|
|
implementation that I tried made things worse in many simple cases, so this
|
|
is not an obviously good thing.
|
|
|
|
. PCRE2 cannot at present distinguish between subpatterns with different names,
|
|
but the same number (created by the use of ?|). In order to do so, a way of
|
|
remembering *which* subpattern numbered n matched is needed. Bugzilla #760.
|
|
(*MARK) can perhaps be used as a way round this problem. However, note that
|
|
Perl does not distinguish: like PCRE2, a name is just an alias for a number
|
|
in Perl.
|
|
|
|
. Instead of having #ifdef HAVE_CONFIG_H in each module, put #include
|
|
"something" and the the #ifdef appears only in one place, in "something".
|
|
|
|
. Implement something like (?(R2+)... to check outer recursions.
|
|
|
|
. If Perl ever supports the POSIX notation [[.something.]] PCRE2 should try
|
|
to follow.
|
|
|
|
. A user wanted a way of ignoring all Unicode "mark" characters so that, for
|
|
example "a" followed by an accent would, together, match "a". This can only
|
|
be done clumsily at present by using a lookahead such as /(?=a)\X/, which
|
|
works for "combining" characters.
|
|
|
|
. Perl supports [\N{x}-\N{y}] as a Unicode range, even in EBCDIC. PCRE2
|
|
supports \N{U+dd..} everywhere, but not in EBCDIC.
|
|
|
|
. Unicode stuff from Perl:
|
|
|
|
\b{gcb} or \b{g} grapheme cluster boundary
|
|
\b{sb} sentence boundary
|
|
\b{wb} word boundary
|
|
|
|
See Unicode TR 29. The last two are very much aimed at natural language.
|
|
|
|
. (?[...]) extended classes: big project.
|
|
|
|
. Bugzilla #1694 requests backwards searching.
|
|
|
|
. Allow a callout to specify a number of characters to skip. This can be done
|
|
compatibly via an extra callout field.
|
|
|
|
. Allow callouts to return *PRUNE, *COMMIT, *THEN, *SKIP, with and without
|
|
continuing (that is, with and without an implied *FAIL). A new option,
|
|
PCRE2_CALLOUT_EXTENDED say, would be needed. This is unlikely ever to be
|
|
implemented by JIT, so this could be an option for pcre2_match().
|
|
|
|
. A limit on substitutions: a user suggested somehow finding a way of making
|
|
match_limit apply to the whole operation instead of each match separately.
|
|
|
|
. Redesign handling of class/nclass/xclass because the compile code logic is
|
|
currently very contorted and obscure.
|
|
|
|
. Some #defines could be replaced with enums to improve robustness.
|
|
|
|
. There was a request for an option for pcre2_match() to return the longest
|
|
match. This would mean searching for all possible matches, of course.
|
|
|
|
. Perl's /a modifier sets Unicode, but restricts \d etc to ASCII characters,
|
|
which is the PCRE2 default for PCRE2_UTF (use PCRE2_UCP to change). However,
|
|
Perl also has /aa, which in addition, disables ASCII/non-ASCII caseless
|
|
matching. Perhaps we need a new option PCRE2_CASELESS_RESTRICT_ASCII. In
|
|
practice, this just means not using the ucd_caseless_sets[] table.
|
|
|
|
. There is more that could be done to the oss-fuzz setup (needs some research).
|
|
A seed corpus could be built. I noted something about $LIB_FUZZING_ENGINE.
|
|
The test function could make use of get_substrings() to cover more code.
|
|
|
|
. A neater way of handling recursion file names in pcre2grep, e.g. a single
|
|
buffer that can grow.
|
|
|
|
. A user suggested that before/after parameters in pcre2grep could have
|
|
negative values, to list lines near to the matched line, but not necessarily
|
|
the line itself. For example, --before-context=-1 would list the line *after*
|
|
each matched line, without showing the matched line. The problem here is what
|
|
to do with matches that are close together. Maybe a simpler way would be a
|
|
flag to disable showing matched lines, only valid with either -A or -B?
|
|
|
|
. There was a suggestiong for a pcre2grep colour default, or possibly a more
|
|
general PCRE2GREP_OPT, but only for some options - not file names or patterns.
|
|
|
|
. Breaking loops that match an empty string: perhaps find a way of continuing
|
|
if *something* has changed, but this might mean remembering additional data.
|
|
"Something" could be a capture value, but then a list of previous values
|
|
would be needed to avoid a cycle of changes. Bugzilla #2182.
|
|
|
|
. The use of \K in assertions is problematic. There was some talk of Perl
|
|
banning this, but it hasn't happened. Some problems could be avoided by
|
|
not allowing it to set a value before the match start; others by not allowing
|
|
it to set a value after the match end. This could be controlled by an option
|
|
such as PCRE2_SANE_BACKSLASH_K, for compatibility (or possibly make the sane
|
|
behaviour the default and implement PCRE2_INSANE_BACKSLASH_K).
|
|
|
|
. If a function could be written to find 3-character (or other length) fixed
|
|
strings, at least one of which must be present for a match, efficient
|
|
pre-searching of large datasets could be implemented.
|
|
|
|
. If pcre2grep had --first-line (match only in the first line) it could be
|
|
efficiently used to find files "starting with xxx". What about --last-line?
|
|
|
|
. A user requested a means of determining whether a failed match was failed by
|
|
the start-of-match optimizations, or by running the match engine. Easy enough
|
|
to define a bit in the match data, but all three matchers would need work.
|
|
|
|
. Would inlining "simple" recursions provide a useful performance boost for the
|
|
interpreters? JIT already does some of this, but it may not be worth it for
|
|
the interpreters.
|
|
|
|
. There was a request for a way of re-defining \w (and therefore \W, \b, and
|
|
\B). An in-pattern sequence such as (?w=[...]) was suggested. Easiest way
|
|
would be simply to inline the class, with lookarounds for \b and \B. Ideally
|
|
the setting should last till the end of the group, which means remembering
|
|
all previous settings; maybe a fixed amount of stack would do - how deep
|
|
would anyone want to nest these things? Bugzilla #2301.
|
|
|
|
. Recognize the short script names. They are already listed in maint/
|
|
Multistage2.py because they are needed for scanning the script extensions
|
|
file.
|
|
|
|
. Use script extensions for \p?
|
|
|
|
. A user suggested something like --with-build-info to set a build information
|
|
string that could be retrieved by pcre2_config(). However, there's no
|
|
facility for a length limit in pcre2_config(), and what would be the
|
|
encoding?
|
|
|
|
Philip Hazel
|
|
Email local part: ph10
|
|
Email domain: cam.ac.uk
|
|
Last updated: 03 June 2019
|