diff --git a/doc/pcre2test.1 b/doc/pcre2test.1 new file mode 100644 index 0000000..5bc7bfd --- /dev/null +++ b/doc/pcre2test.1 @@ -0,0 +1,1179 @@ +.TH PCRE2TEST 1 "13 May 2014" "PCRE 9.00" +.SH NAME +pcre2test - a program for testing Perl-compatible regular expressions. +.SH SYNOPSIS +.rs +.sp +.B pcre2test "[options] [input file [output file]]" +.sp +\fBpcre2test\fP is a test program for the PCRE2 regular expression libraries, +but it can also be used for experimenting with regular expressions. This +document describes the features of the test program; for details of the regular +expressions themselves, see the +.\" HREF +\fBpcre2pattern\fP +.\" +documentation. For details of the PCRE2 library function calls and their +options, see the +.\" HREF +\fBpcre2api\fP +.\" +, +.\" HREF +\fBpcre16\fP +and +.\" HREF +\fBpcre32\fP +.\" +documentation. +.P +The input for \fBpcre2test\fP is a sequence of regular expression patterns and +subject strings to be matched. The output shows the result of each match +attempt. Modifiers on the command line, the patterns, and the subject lines +specify PCRE2 function options, control how the subject is processed, and what +output is produced. +.P +As the original fairly simple PCRE library evolved, it acquired many different +features, and as a result, the original \fBpcretest\fP program ended up with a +lot of options in a messy, arcane syntax, for testing all the features. The +move to the new PCRE2 API provided an opportunity to re-implement the test +program as \fBpcre2test\fP, with a cleaner modifier syntax. Nevertheless, there +are still many obscure modifiers, some of which are specifically designed for +use in conjunction with the test script and data files that are distributed as +part of PCRE2. All the modifiers are documented here, some without much +justification, but many of them are unlikely to be of use except when testing +the libraries. +. +. +.SH "PCRE2's 8-BIT, 16-BIT AND 32-BIT LIBRARIES" +.rs +.sp +Different versions of the PCRE2 library can be built to support character +strings that are encoded in 8-bit, 16-bit, or 32-bit code units. One, two, or +all three of these libraries may be simultaneously installed. The +\fBpcre2test\fP program can be used to test all the libraries. However, its own +input and output are always in 8-bit format. When testing the 16-bit or 32-bit +libraries, patterns and subject strings are converted to 16- or 32-bit format +before being passed to the library functions. Results are converted back to +8-bit code units for output. +.P +In the rest of this document, the names of library functions and structures +are given in generic form, for example, \fBpcre_compile()\fP. The actual +names used in the libraries have a suffix _8, _16, or _32, as appropriate. +. +. +.SH "INPUT ENCODING" +.rs +.sp +Input to \fBpcre2test\fP is processed line by line, either by calling the C +library's \fBfgets()\fP function, or via the \fBlibreadline\fP library (see +below). In Unix-like environments, \fBfgets()\fP treats any bytes other than +newline as data characters. However, in some Windows environments character 26 +(hex 1A) causes an immediate end of file, and no further data is read. For +maximum portability, therefore, it is safest to avoid non-printing characters +in \fBpcre2test\fP input files. +. +. +.SH "COMMAND LINE OPTIONS" +.rs +.TP 10 +\fB-8\fP +If the 8-bit library has been built, this option causes it to be used (this is +the default). If the 8-bit library has not been built, this option causes an +error. +.TP 10 +\fB-16\fP +If the 16-bit library has been built, this option causes it to be used. If only +the 16-bit library has been built, this is the default. If the 16-bit library +has not been built, this option causes an error. +.TP 10 +\fB-32\fP +If the 32-bit library has been built, this option causes it to be used. If only +the 32-bit library has been built, this is the default. If the 32-bit library +has not been built, this option causes an error. +.TP 10 +\fB-b\fP +Behave as if each pattern has the \fB/fullbincode\fP modifier; the full +internal binary form of the pattern is output after compilation. +.TP 10 +\fB-C\fP +Output the version number of the PCRE2 library, and all available information +about the optional features that are included, and then exit with zero exit +code. All other options are ignored. +.TP 10 +\fB-C\fP \fIoption\fP +Output information about a specific build-time option, then exit. This +functionality is intended for use in scripts such as \fBRunTest\fP. The +following options output the value and set the exit code as indicated: +.sp + ebcdic-nl the code for LF (= NL) in an EBCDIC environment: + 0x15 or 0x25 + 0 if used in an ASCII environment + exit code is always 0 + linksize the configured internal link size (2, 3, or 4) + exit code is set to the link size + newline the default newline setting: + CR, LF, CRLF, ANYCRLF, or ANY + exit code is always 0 + bsr the default setting for what \eR matches: + ANYCRLF or ANY + exit code is always 0 +.sp +The following options output 1 for true or 0 for false, and set the exit code +to the same value: +.sp + ebcdic compiled for an EBCDIC environment + jit just-in-time support is available + pcre16 the 16-bit library was built + pcre32 the 32-bit library was built + pcre8 the 8-bit library was built + utf UTF and Unicode property support is available +.sp +If an unknown option is given, an error message is output; the exit code is 0. +.TP 10 +\fB-d\fP +Behave as if each pattern has the \fB/debug\fP modifier; the internal +form and information about the compiled pattern is output after compilation; +\fB-d\fP is equivalent to \fB-b -i\fP. +.TP 10 +\fB-help\fP +Output a brief summary these options and then exit. +.TP 10 +\fB-i\fP +Behave as if each pattern has the \fB/info\fP modifier; information about the +compiled pattern is given after compilation. +.TP 10 +\fB-pattern\fB \fImodifier-list\fP +Behave as if each pattern line contains the given modifiers. +.TP 10 +\fB-q\fP +Do not output the version number of \fBpcre2test\fP at the start of execution. +.TP 10 +\fB-S\fP \fIsize\fP +On Unix-like systems, set the size of the run-time stack to \fIsize\fP +megabytes. +.TP10 +\fB-subject\fP \fImodifier-list\fP +Behave as if each subject line contains the given modifiers. +.TP 10 +\fB-t\fP +Run each compile and match many times with a timer, and output the resulting +times per compile or match. You can control the number of iterations that are +used for timing by following \fB-t\fP with a number (as a separate item on the +command line). For example, "-t 1000" iterates 1000 times. The default is to +iterate 500,000 times. +.TP 10 +\fB-tm\fP +This is like \fB-t\fP except that it times only the matching phase, not the +compile phase. +.TP 10 +\fB-T\fP \fB-TM\fP +These behave like \fB-t\fP and \fB-tm\fP, but in addition, at the end of a run, +the total times for all compiles and matches are output. +. +. +.SH "DESCRIPTION" +.rs +.sp +If \fBpcre2test\fP is given two filename arguments, it reads from the first and +writes to the second. If it is given only one filename argument, it reads from +that file and writes to stdout. Otherwise, it reads from stdin and writes to +stdout, and prompts for each line of input, using "re>" to prompt for regular +expression patterns, and "data>" to prompt for subject lines. +.P +When \fBpcre2test\fP is built, a configuration option can specify that it +should be linked with the \fBlibreadline\fP or \fBlibedit\fP library. When this +is done, if the input is from a terminal, it is read using the \fBreadline()\fP +function. This provides line-editing and history facilities. The output from +the \fB-help\fP option states whether or not \fBreadline()\fP will be used. +.P +The program handles any number of tests, each of which consists of a set of +input lines. Each set starts with a regular expression pattern, followed by any +number of subject lines to be matched against that pattern. In between sets of +test data, command lines that begin with a hash (#) character may appear. +.P +Each subject line is matched separately and independently. If you want to do +multi-line matches, you have to use the \en escape sequence (or \er or \er\en, +etc., depending on the newline setting) in a single line of input to encode the +newline sequences. There is no limit on the length of subject lines; the input +buffer is automatically extended if it is too small. There is a replication +feature that makes it possible to generate long subject lines without having to +supply them explicitly. +.P +An empty line or the end of the file signals the end of the subject lines for a +test, at which point a new pattern or command line is expected if there is +still input to be read. +. +. +.SH "COMMAND LINES" +.rs +.sp +In between sets of test data, a line that begins with a hash (#) character is +interpreted as a command line. If the first character is followed by white +space or an exclamation mark, the line is treated as a comment, and ignored. +Otherwise, the following commands are recognized: +.sp + #load +.sp +Load a pre-compiled pattern that has been saved in a file. This command must be +followed immediately by any subject lines that are to be matched by the +pattern. +.sp + #pattern +.sp +This command sets a default modifier list that applies to all subsequent +patterns. Modifiers on a pattern can change these settings. +.sp + #subject +.sp +This command sets a default modifier list that applies to all subsequent +subject lines. Modifiers on a subject line can change these settings. +. +. +.SH "MODIFIER SYNTAX" +.rs +.sp +Modifier lists are used with both pattern and subject lines. Items in a list +are separated by commas and optional white space. Some modifiers may be given +for both patterns and subject lines, whereas others are valid for one or the +other only. Each modifier has a long name, for example "anchored", and some of +them must be followed by an equals sign and a value, for example, "offset=12". +Modifiers that do not take values may be preceded by a minus sign to turn off a +previous default setting. +.P +A few of the more common modifiers can also be specified as single or double +letters, for example "i" for "caseless". In documentation, following the Perl +convention, these are written with a slash ("the /i modifier") for clarity. +Abbreviated modifiers must all be concatenated in the first item of a modifier +list. If the first item is not recognized as a long modifier name, it is +interpreted as a sequence of these abbreviations. For example: +.sp + /abc/ig,newline=cr,jit=3 +.sp +This is a pattern line whose modifier list starts with two one-letter modifiers +(/i and /g). The lower-case abbreviated modifiers are the same as used in Perl. +. +. +.SH "PATTERN SYNTAX" +.rs +.sp +A pattern line must start with one of the following characters: +.sp + " / ! ' ` - + = : ; . , +.sp +This is interpreted as the pattern's delimiter. A regular expression may be +continued over several input lines, in which case the newline characters are +included within it. It is possible to include the delimiter within the pattern +by escaping it with a backslash, for example +.sp + /abc\e/def/ +.sp +If you do this, the escape and the delimiter form part of the pattern, but +since the delimiters are all non-alphanumeric, this does not affect its +interpretation. If the terminating delimiter is immediately followed by a +backslash, for example, +.sp + /abc/\e +.sp +then a backslash is added to the end of the pattern. This is done to provide a +way of testing the error condition that arises if a pattern finishes with a +backslash, because +.sp + /abc\e/ +.sp +is interpreted as the first line of a pattern that starts with "abc/", causing +pcre2test to read the next line as a continuation of the regular expression. +.P +A pattern can be followed by a modifier list (details below). +. +. +.SH "SUBJECT LINE SYNTAX" +.rs +.sp +Before each subject line is passed to \fBpcre2_match()\fP or +\fBpcre2_dfa_match()\fP, leading and trailing white space is removed, and the +line is scanned for backslash escapes. The following provide a means of +encoding non-printing characters in a visible way: +.sp + \ea alarm (BEL, \ex07) + \eb backspace (\ex08) + \ee escape (\ex27) + \ef form feed (\ex0c) + \en newline (\ex0a) + \er carriage return (\ex0d) + \et tab (\ex09) + \ev vertical tab (\ex0b) + \ennn octal character (up to 3 octal digits); always + a byte unless > 255 in UTF-8 or 16-bit or 32-bit mode + \eo{dd...} octal character (any number of octal digits} + \exhh hexadecimal byte (up to 2 hex digits) + \ex{hh...} hexadecimal character (any number of hex digits) +.sp +The use of \ex{hh...} is not dependent on the use of the utf modifier on +the pattern. It is recognized always. There may be any number of hexadecimal +digits inside the braces; invalid values provoke error messages. +.P +Note that \exhh specifies one byte rather than one character in UTF-8 mode; +this makes it possible to construct invalid UTF-8 sequences for testing +purposes. On the other hand, \ex{hh} is interpreted as a UTF-8 character in +UTF-8 mode, generating more than one byte if the value is greater than 127. +When testing the 8-bit library not in UTF-8 mode, \ex{hh} generates one byte +for values less than 256, and causes an error for greater values. +.P +In UTF-16 mode, all 4-digit \ex{hhhh} values are accepted. This makes it +possible to construct invalid UTF-16 sequences for testing purposes. +.P +In UTF-32 mode, all 4- to 8-digit \ex{...} values are accepted. This makes it +possible to construct invalid UTF-32 sequences for testing purposes. +.P +There is a special backslash sequence that specifies replication of one or more +characters: +.sp + \e[]{} +.sp +This makes it possible to test long strings without having to provide them as +part of the file. For example: +.sp + \e[abc]{4} +.sp +is converted to "abcabcabcabc". This feature does not support nesting. To +include a closing square bracket in the characters, code it as \ex5D. +.P +A backslash followed by an equals sign marke the end of the subject string and +the start of a modifier list. For example: +.sp + abc\=notbol,notempty +.sp +A backslash followed by anything else causes an error. However, if the very +last character in the line is a backslash (and there is no modifier list), it +is ignored. This gives a way of passing an empty line as data, since a real +empty line terminates the data input. +. +. +.SH "PATTERN MODIFIERS" +.rs +.sp +There are three types of modifier that can appear in pattern lines, two of +which may also be used in a \fB#pattern\fP command. A pattern's modifier list +can add to or override default modifiers that were set by a previous +\fB#pattern\fP command. +. +.SS "Setting compilation options" +.rs +.sp +The following modifiers set options for \fBpcre2_compile()\fP. The most common +ones have single-letter abbreviations. See +.\" HREF +\fBpcreapi\fP +.\" +for a description of their effects. +.sp + allow_empty_class set PCRE2_ALLOW_EMPTY_CLASS + alt_bsux set PCRE2_ALT_BSUX + anchored set PCRE2_ANCHORED + auto_callout set PCRE2_AUTO_CALLOUT + /i caseless set PCRE2_CASELESS + dollar_endonly set PCRE2_DOLLAR_ENDONLY + /s dotall set PCRE2_DOTALL + dupnames set PCRE2_DUPNAMES + /x extended set PCRE2_EXTENDED + firstline set PCRE2_FIRSTLINE + match_unset_backref set PCRE2_MATCH_UNSET_BACKREF + /m multiline set PCRE2_MULTILINE + never_ucp set PCRE2_NEVER_UCP + never_utf set PCRE2_NEVER_UTF + no_auto_capture set PCRE2_NO_AUTO_CAPTURE + no_auto_possess set PCRE2_NO_AUTO_POSSESS + no_start_optimize set PCRE2_NO_START_OPTIMIZE + no_utf_check set PCRE2_NO_UTF_CHECK + ucp set PCRE2_UCP + ungreedy set PCRE2_UNGREEDY + utf set PCRE2_UTF +.sp +As well as turning on the PCRE2_UTF option, the \fButf\fP modifier causes all +non-printing characters in output strings to be printed using the \ex{hh...} +notation. Otherwise, those less than 0x100 are output in hex without the curly +brackets. +. +.SS "Setting compilation controls" +.rs +.sp +The following modifiers affect the compilation process or request information +about the pattern: +.sp + bsr=[anycrlf|unicode] specify \eR handling + /B bincode show binary code without lengths + /D debug same as /DBB + flipbytes flip endianness + /BB fullbincode show binary code with lengths + /I info show info about compiled pattern + jit[=] use JIT + locale= use this locale + memory show memory used + newline= set newline type + parens_nest_limit= set maximum parentheses depth + perlcompat lock out non-Perl modifiers + posix use the POSIX API + save= save compiled pattern + stackguard= test the stackguard feature + tables=[0|1] select internal tables +.sp +The effects of these modifiers are described in the following sections. +FIXME: Give more examples. +. +. +.SS "Newline and \eR handling" +.rs +.sp +The \fBbsr\fP modifier specifies what \eR in a pattern should match. If it is +set to "anycrlf", \eR matches CR, LF, or CRLF only. If it is set to "unicode", +\eR matches any Unicode newline sequence. The default is specified when PCRE2 +is built, with the default default being Unicode. +.P +The \fBnewline\fP modifier specifies which characters are to be interpreted as +newlines, both in the pattern and in subject lines. The type must be one of +CR, LF, CRLF, ANYCRLF, or ANY. +. +. +.SS "Information about a pattern" +.rs +.sp +The \fBdebug\fP modifier is a shorthand for \fBinfo,fullbincode\fP, requesting +all available information. +.P +The \fBbincode\fP modifier causes a representation of the compiled code to be +output after compilation. This information does not contain length and offset +values, which ensures that the same output is generated for different internal +link sizes and different code unit widths. By using \fBbincode\fP, the same +regression tests can be used in different environments. +.P +The \fBfullbincode\fP modifier, by contrast, \fIdoes\fP include length and +offset values. This is used in a few special tests and is also useful for +one-off tests. +.P +The \fBinfo\fP modifier requests information about the compiled pattern +(whether it is anchored, has a fixed first character, and so on). The +information is obtained from the \fBpcre2_pattern_info()\fP function. +. +. +.SS "Changing byte order" +.rs +.sp +The \fBflipbytes\fP modifier causes \fBpcre2test\fP to flip the byte order of +the 2-byte and 4-byte fields in the compiled pattern. This facility is for +testing the feature that allows PCRE2 to use patterns that were compiled on a +host with a different endianness. This feature is not available when the POSIX +interface is being used, that is, when the \fBposix\fP pattern modifier is +specified. See also the section about saving and reloading compiled patterns +below. +. +. +.SS "JIT compilation" +.rs +.sp +The \fB/jit\fP modifier may optionally be followed by a number in the range 0 +to 7: +.sp + 0 disable JIT + 1 normal match only + 2 soft partial match only + 3 normal match and soft partial match + 4 hard partial match only + 6 soft and hard partial match + 7 all three modes (default) +.sp +If JIT compilation is successful, the compiled JIT code will automatically be +used when \fBpcre2_match()\fP is run, except when incompatible run-time +options are specified. For more details, see the +.\" HREF +\fBpcre2jit\fP +.\" +documentation. See also the \fBjitstack\fP modifier below for a way of +setting the size of the JIT stack. +.P +If the \fBjitverify\fP modifier is specified, the text "(JIT)" is added to the +first output line after a match or non match when JIT-compiled code was +actually used. This modifier can also be set on a subject line. +. +. +.SS "Setting a locale" +.rs +.sp +The \fB/locale\fP modifier must specify the name of a locale, for example: +.sp + /pattern/locale=fr_FR +.sp +The given locale is set, +\fBpcre2_maketables()\fP is called to build a set of character tables for +the locale, and this is then passed to \fBpcre2_compile()\fP when compiling the +regular expression. The same tables are used when matching the following +subject lines. The \fB/locale\fP modifier applies only to the pattern on which +it appears, but can be given in a \fB#pattern\fP command if a default is +needed. +. +. +.SS "Showing pattern memory" +.rs +.sp +The \fB/memory\fP modifier causes the size in bytes of the memory block used to +hold the compiled pattern to be output. This does not include the size of the +\fBpcre2_code\fP block; it is just the actual compiled data. If the pattern is +subsequently passed to the JIT compiler, the size of the JIT compiled code is +also output. +. +. +.SS "Limiting nested parentheses" +.rs +.sp +The \fBparens_nest_limit\fP modifier sets a limit on the depth of nested +parentheses in a pattern. Breaching the limit causes a compilation error. +. +. +.SS "Using the POSIX wrapper API" +.rs +.sp +The \fB/posix\fP modifier causes \fBpcre2test\fP to call PCRE2 via the POSIX +wrapper API rather than its native API. This supports only the 8-bit library. +When the POSIX API is being used, the following pattern modifiers set options +for the \fBregcomp()\fP function: +.sp + caseless REG_ICASE + multiline REG_NEWLINE + no_auto_capture REG_NOSUB + dotall REG_DOTALL ) + ungreedy REG_UNGREEDY ) These options are not part of + ucp REG_UCP ) the POSIX standard + utf REG_UTF8 ) +.sp +The \fBaftertext\fP and \fBallaftertext\fP subject modifiers work as described +below. All other modifiers cause an error. +. +. +.SS "Testing the stack guard feature" +.rs +.sp +The \fB/stackguard\fP modifier is used to test the use of +\fBpcre2_stack_guard\fP. It must be followed by '0' or '1', specifying the +return code to be given from an external function that is passed to PCRE2 and +used for stack checking during compilation (see the +.\" HREF +\fBpcre2api\fP +.\" +documentation for details). FIXME: this needs doing properly once the test is +implemented. Mention nested parens limit. +. +. +.SS "Using alternative character tables" +.rs +.sp +The \fB/tables\fP modifier must be followed by a single digit. It causes a +specific set of built-in character tables to be passed to +\fBpcre2_compile()\fP. This is used in the PCRE2 tests to check behaviour with +different character tables. The digit specifies the tables as follows: +.sp + 0 the default ASCII tables, as distributed in + pcre2_chartables.c.dist + 1 a set of tables defining ISO 8859 characters +.sp +In table 1, some characters whose codes are greater than 128 are identified as +letters, digits, spaces, etc. +. +. +.SS "Locking out certain modifiers" +.rs +.sp +FIXME FIXME +PCRE can be compiled with or without support for certain features such as +UTF-8/16/32 or Unicode properties. Accordingly, the standard tests are split up +into a number of different files that are selected for running depending on +which features are available. When updating the tests, it is all too easy to +put a new test into the wrong file by mistake; for example, to put a test that +requires UTF support into a file that is used when it is not available. To help +detect such mistakes as early as possible, there is a facility for locking out +specific modifiers. If an input line for \fBpcre2test\fP starts with the string +"< forbid " the following sequence of characters is taken as a list of +forbidden modifiers. For example, in the test files that must not use UTF or +Unicode property support, this line appears: +.sp + < forbid 8W +.sp +This locks out the /8 and /W modifiers. An immediate error is given if they are +subsequently encountered. If the character string contains < but not >, all the +multi-character modifiers that begin with < are locked out. Otherwise, such +modifiers must be explicitly listed, for example: +.sp + < forbid +.sp +There must be a single space between < and "forbid" for this feature to be +recognised. If there is not, the line is interpreted either as a request to +re-load a pre-compiled pattern (see "SAVING AND RELOADING COMPILED PATTERNS" +below) or, if there is a another < character, as a pattern that uses < as its +delimiter. +. +. +.SS "Setting certain match controls" +.rs +.sp +The following modifiers are really subject modifiers, and are described below. +However, they may be included in a pattern's modifier list, in which case they +are applied to every subject line that is processed with that pattern. They do +not affect the compilation process. +.sp + aftertext show text after match + allaftertext show text after captures + allcaptures show all captures + /gg altglobal alternative global matching + /g global global matching + jitverify verify JIT usage + mark show mark values +.sp +These modifiers may not appear in a \fB#pattern\fP command. +. +. +.SH "SUBJECT MODIFIERS" +.rs +.sp +The modifiers that can appear in subject lines and the \fB#subject\fP +command are of two types. +. +.SS "Setting match options" +.rs +.sp +The following modifiers set options for \fBpcre2_match()\fP or +\fBpcre2_dfa_match()\fP. See +.\" HREF +\fBpcreapi\fP +.\" +for a description of their effects. +.sp + anchored set PCRE2_ANCHORED + dfa_restart set PCRE2_DFA_RESTART + dfa_shortest set PCRE2_DFA_SHORTEST + no_start_optimize set PCRE2_NO_START_OPTIMIZE + no_utf_check set PCRE2_NO_UTF_CHECK + notbol set PCRE2_NOTBOL + notempty set PCRE2_NOTEMPTY + notempty_atstart set PCRE2_NOTEMPTY_ATSTART + noteol set PCRE2_NOTEOL + /PP partial_hard set PCRE2_PARTIAL_HARD + /P partial_soft set PCRE2_PARTIAL_SOFT +.sp +If the \fB/posix\fP modifier was present on the pattern, causing the POSIX +wrapper API to be used, the only option-setting modifiers that have any effect +are \fBnotbol\fP, \fBnotempty\fP, and \fBnoteol\fP, causing REG_NOTBOL, +REG_NOTEMPTY, and REG_NOTEOL, respectively, to be passed to \fBregexec()\fP. +Any other modifiers cause an error. +. +.SS "Setting match controls" +.rs +.sp +The following modifiers affect the matching process or request additional +information. Some of them may also be specified on a pattern line (see above), +in which case they apply to every subject line that is matched against that +pattern. +.sp + aftertext show text after match + allaftertext show text after captures + allcaptures show all captures + /gg altglobal alternative global matching + callout_capture show captures at callout time + callout_fail=[,] control callout failure + callout_none do not supply a callout function + copy= copy captured substring + dfa use \fBpcre2_dfa_match()\fP + get= extract captured substring + getall extract all captured substrings + /g global global matching + jitstack= set size of JIT stack + jitverify verify JIT usage + limits find match and recursion limits + mark show mark values + match_limit=>n> set a match limit + memory show memory usage + offset= set starting offset + ovector= set size of output vector + recursion_limit= set a recursion limit +.sp +The effects of these modifiers are described in the following sections. +FIXME: Give more examples. +. +. +.SS "Showing more text" +.rs +.sp +The \fBaftertext\fP modifier requests that as well as outputting the substring +that matched the entire pattern, \fBpcre2test\fP should in addition output the +remainder of the subject string. This is useful for tests where the subject +contains multiple copies of the same substring. The \fBallaftertext\fP modifier +requests the same action for captured substrings as well as the main matched +substring. In each case the remainder is output on the following line with a +plus character following the capture number. +. +. +.SS "Showing the value of all capture groups" +.rs +.sp +The \fBallcaptures\fP modifier requests that the values of all potential +captured parentheses be output after a match. By default, only those up to the +highest one actually used in the match are output (corresponding to the return +code from \fBpcre2_match()\fP). Groups that did not take part in the match +are output as "". +. +. +.SS "Testing callouts" +.rs +.sp +A callout function is supplied when \fBpcre2test\fP calls the library matching +functions, unless \fBcallout_none\fP is specified. If \fBcallout_capture\fP is +set, the current captured groups are output when a callout occurs. +.P +The \fBcallout_fail\fP modifier can be given one or two numbers. If there is +only one number, 1 is returned instead of 0 when a callout of that number is +reached. If two numbers are given, 1 is returned when callout is reached +for the th time. FIXME: this needs checking. +. +. +.SS "Testing substring extraction functions" +.rs +.sp +The \fBcopy\fP and \fBget\fP modifiers can be used to test the +\fBpcre2_substring_copy_xxx()\fP and \fBpcre2_substring_get_xxx()\fP functions. +They can be given more than once, and each can specify a group name or number. +The \fBgetall\fP modifier tests \fBpcre2_substring_list_get()\fP, which +extracts all captured substrings. +.P +If the subject line is successfully matched, the substrings extracted by the +convenience functions are output with C, G, or L after the string number +instead of a colon. This is in addition to the normal full list. The string +length (that is, the return from the extraction function) is given in +parentheses after each substring. +. +. +.SS "Finding all matches in a string" +.rs +.sp +Searching for all possible matches within a subject can be requested by the +\fBglobal\fP or \fB/altglobal\fP modifier. After finding a match, the matching +function is called again to search the remainder of the subject. The difference +between \fBglobal\fP and \fBaltglobal\fP is that the former uses the +\fIstart_offset\fP argument to \fBpcre2_match()\fP or \fBpcre2_dfa_match()\fP +to start searching at a new point within the entire string (which is what Perl +does), whereas the latter passes over a shortened substring. This makes a +difference to the matching process if the pattern begins with a lookbehind +assertion (including \eb or \eB). +.P +If an empty string is matched, the next match is done with the +PCRE2_NOTEMPTY_ATSTART and PCRE2_ANCHORED flags set, in order to search for +another, non-empty, match at the same point in the subject. If this match +fails, the start offset is advanced, and the normal match is retried. This +imitates the way Perl handles such cases when using the \fB/g\fP modifier or +the \fBsplit()\fP function. Normally, the start offset is advanced by one +character, but if the newline convention recognizes CRLF as a newline, and the +current character is CR followed by LF, an advance of two is used. +. +. +.SS "Setting the JIT stack size" +.rs +.sp +The \fBjitstack\fP modifier provides a way of setting the maximum stack size +that is used by the just-in-time optimization code. It is ignored if JIT +optimization is not being used. Providing a stack that is larger than the +default 32K is necessary only for very complicated patterns. +. +. +.SS "Setting match and recursion limits" +.rs +.sp +The \fBmatch_limit\fP and \fBrecursion_limit\fP modifiers set the appropriate +limits in the match context. These values are ignored when the \fBlimits\fP +modifier is specified. +. +. +.SS "Finding minimum limits" +.rs +.sp +If the \fBlimits\fP modifier is present, \fBpcre2test\fP calls +\fBpcre2_match()\fP several times, setting different values in the match +context via \fBpcre2_set_match_limit()\fP and \fBpcre2_set_recursion_limit()\fP +until it finds the minimum values for each parameter that allow +\fBpcre2_match()\fP to complete without error. +.P +The \fImatch_limit\fP number is a measure of the amount of backtracking +that takes place, and learning the minimum value can be instructive. For most +simple matches, the number is quite small, but for patterns with very large +numbers of matching possibilities, it can become large very quickly with +increasing length of subject string. The \fImatch_limit_recursion\fP number is +a measure of how much stack (or, if PCRE2 is compiled with NO_RECURSE, how much +heap) memory is needed to complete the match attempt. +. +. +.SS "Showing MARK names" +.rs +.sp +.P +The \fBmark\fP modifier causes the names from backtracking control verbs that +are returned from calls to \fBpcre2_match()\fP to be displayed. If a mark is +returned for a match, non-match, or partial match, \fBpcre2test\fP shows it. +For a match, it is on a line by itself, tagged with "MK:". Otherwise, it +is added to the non-match message. +. +. +.SS "Showing memory usage" +.rs +.sp +The \fBmemory\fP modifier causes \fBpcre2test\fP to log all memory allocation +and freeing calls that occur during a match operation. +. +. +.SS "Setting a starting offset" +.rs +.sp +The \fBoffset\fP modifier sets an offset in the subject string at which +matching starts. Its value is a number of code units, not characters. +. +. +.SS "Setting the size of the output vector" +.rs +.sp +The \fBovector\fP modifier applies only to the subject line in which it +appears, though of course it can also be used to set a default in a +\fB#subject\fP command. It specifies the number of pairs of offsets that are +available for storing matching information. The default is 15. +. +. +.SH "THE ALTERNATIVE MATCHING FUNCTION" +.rs +.sp +By default, \fBpcre2test\fP uses the standard PCRE2 matching function, +\fBpcre2_match()\fP to match each subject line. PCRE2 also supports an +alternative matching function, \fBpcre2_dfa_match()\fP, which operates in a +different way, and has some restrictions. The differences between the two +functions are described in the +.\" HREF +\fBpcre2matching\fP +.\" +documentation. +.P +If the \fBdfa\fP modifier is set, the alternative matching function is used. +This function finds all possible matches at a given point in the subject. If, +however, the \fBdfa_shortest\fP modifier is set, processing stops after the +first match is found. This is always the shortest possible match. +. +. +.SH "DEFAULT OUTPUT FROM pcre2test" +.rs +.sp +This section describes the output when the normal matching function, +\fBpcre2_match()\fP, is being used. +.P +When a match succeeds, \fBpcre2test\fP outputs the list of captured substrings, +starting with number 0 for the string that matched the whole pattern. +Otherwise, it outputs "No match" when the return is PCRE2_ERROR_NOMATCH, or +"Partial match:" followed by the partially matching substring when the +return is PCRE2_ERROR_PARTIAL. (Note that this is the +entire substring that was inspected during the partial match; it may include +characters before the actual match start if a lookbehind assertion, \eK, \eb, +or \eB was involved.) +.P +For any other return, \fBpcre2test\fP outputs the PCRE2 +negative error number and a short descriptive phrase. If the error is a failed +UTF string check, the offset of the start of the failing character and the +reason code are also output. Here is an example of an interactive +\fBpcre2test\fP run. +.sp + $ pcre2test + PCRE2 version 9.00 2014-05-10 +.sp + re> /^abc(\ed+)/ + data> abc123 + 0: abc123 + 1: 123 + data> xyz + No match +.sp +Unset capturing substrings that are not followed by one that is set are not +returned by \fBpcre2_match()\fP, and are not shown by \fBpcre2test\fP. In the +following example, there are two capturing substrings, but when the first data +line is matched, the second, unset substring is not shown. An "internal" unset +substring is shown as "", as for the second data line. +.sp + re> /(a)|(b)/ + data> a + 0: a + 1: a + data> b + 0: b + 1: + 2: b +.sp +If the strings contain any non-printing characters, they are output as \exhh +escapes if the value is less than 256 and UTF mode is not set. Otherwise they +are output as \ex{hh...} escapes. See below for the definition of non-printing +characters. If the \fB/aftertext\fP modifier is set, the output for substring +0 is followed by the the rest of the subject string, identified by "0+" like +this: +.sp + re> /cat/aftertext + data> cataract + 0: cat + 0+ aract +.sp +If global matching is requested, the results of successive matching attempts +are output in sequence, like this: +.sp + re> /\eBi(\ew\ew)/g + data> Mississippi + 0: iss + 1: ss + 0: iss + 1: ss + 0: ipp + 1: pp +.sp +"No match" is output only if the first match attempt fails. Here is an example +of a failure message (the offset 4 that is specified by \e>4 is past the end of +the subject string): +.sp + re> /xyz/ + data> xyz\e=offset=4 + Error -24 (bad offset value) +.P +Note that whereas patterns can be continued over several lines (a plain ">" +prompt is used for continuations), subject lines may not. However newlines can +be included in a subject by means of the \en escape (or \er, \er\en, etc., +depending on the newline sequence setting). +. +. +. +.SH "OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION" +.rs +.sp +When the alternative matching function, \fBpcre2_dfa_match()\fP, is used, the +output consists of a list of all the matches that start at the first point in +the subject where there is at least one match. For example: +.sp + re> /(tang|tangerine|tan)/ + data> yellow tangerine\e=dfa + 0: tangerine + 1: tang + 2: tan +.sp +(Using the normal matching function on this data finds only "tang".) The +longest matching string is always given first (and numbered zero). After a +PCRE2_ERROR_PARTIAL return, the output is "Partial match:", followed by the +partially matching substring. (Note that this is the entire substring that was +inspected during the partial match; it may include characters before the actual +match start if a lookbehind assertion, \eK, \eb, or \eB was involved.) +.P +If global matching is requested, the search for further matches resumes +at the end of the longest match. For example: +.sp + re> /(tang|tangerine|tan)/g + data> yellow tangerine and tangy sultana\e=dfa + 0: tangerine + 1: tang + 2: tan + 0: tang + 1: tan + 0: tan +.sp +The alternative matching function does not support substring capture, so the +modifiers that are concerned with captured substrings are not relevant. +. +. +.SH "RESTARTING AFTER A PARTIAL MATCH" +.rs +.sp +When the alternative matching function has given the PCRE2_ERROR_PARTIAL +return, indicating that the subject partially matched the pattern, you can +restart the match with additional subject data by means of the +\fBdfa_restart\fP modifier. For example: +.sp + re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/ + data> 23ja\e=P,dfa + Partial match: 23ja + data> n05\e=dfa,dfa_restart + 0: n05 +.sp +For further information about partial matching, see the +.\" HREF +\fBpcre2partial\fP +.\" +documentation. +. +. +.SH CALLOUTS +.rs +.sp +If the pattern contains any callout requests, \fBpcre2test\fP's callout function +is called during matching. This works with both matching functions. By default, +the called function displays the callout number, the start and current +positions in the text at the callout time, and the next pattern item to be +tested. For example: +.sp + --->pqrabcdef + 0 ^ ^ \ed +.sp +This output indicates that callout number 0 occurred for a match attempt +starting at the fourth character of the subject string, when the pointer was at +the seventh character, and when the next pattern item was \ed. Just +one circumflex is output if the start and current positions are the same. +.P +Callouts numbered 255 are assumed to be automatic callouts, inserted as a +result of the \fB/auto_callout\fP pattern modifier. In this case, instead of +showing the callout number, the offset in the pattern, preceded by a plus, is +output. For example: +.sp + re> /\ed?[A-E]\e*/auto_callout + data> E* + --->E* + +0 ^ \ed? + +3 ^ [A-E] + +8 ^^ \e* + +10 ^ ^ + 0: E* +.sp +If a pattern contains (*MARK) items, an additional line is output whenever +a change of latest mark is passed to the callout function. For example: +.sp + re> /a(*MARK:X)bc/auto_callout + data> abc + --->abc + +0 ^ a + +1 ^^ (*MARK:X) + +10 ^^ b + Latest Mark: X + +11 ^ ^ c + +12 ^ ^ + 0: abc +.sp +The mark changes between matching "a" and "b", but stays the same for the rest +of the match, so nothing more is output. If, as a result of backtracking, the +mark reverts to being unset, the text "" is output. +.P +The callout function in \fBpcre2test\fP returns zero (carry on matching) by +default, but you can use a \fBcallout_fail\fP modifier in a subject line (as +described above) to change this and other parameters of the callout. +.P +Inserting callouts can be helpful when using \fBpcre2test\fP to check +complicated regular expressions. For further information about callouts, see +the +.\" HREF +\fBpcre2callout\fP +.\" +documentation. +. +. +. +.SH "NON-PRINTING CHARACTERS" +.rs +.sp +When \fBpcre2test\fP is outputting text in the compiled version of a pattern, +bytes other than 32-126 are always treated as non-printing characters and are +therefore shown as hex escapes. +.P +When \fBpcre2test\fP is outputting text that is a matched part of a subject +string, it behaves in the same way, unless a different locale has been set for +the pattern (using the \fB/locale\fP modifier). In this case, the +\fBisprint()\fP function is used to distinguish printing and non-printing +characters. +. +. +. +.SH "SAVING AND RELOADING COMPILED PATTERNS" +.rs +.sp +FIXME FIXME +The facilities described in this section are not available when the POSIX +interface to PCRE is being used, that is, when the \fB/P\fP pattern modifier is +specified. +.P +When the POSIX interface is not in use, you can cause \fBpcre2test\fP to write a +compiled pattern to a file, by following the modifiers with > and a file name. +For example: +.sp + /pattern/im >/some/file +.sp +See the +.\" HREF +\fBpcreprecompile\fP +.\" +documentation for a discussion about saving and re-using compiled patterns. +Note that if the pattern was successfully studied with JIT optimization, the +JIT data cannot be saved. +.P +The data that is written is binary. The first eight bytes are the length of the +compiled pattern data followed by the length of the optional study data, each +written as four bytes in big-endian order (most significant byte first). If +there is no study data (either the pattern was not studied, or studying did not +return any data), the second length is zero. The lengths are followed by an +exact copy of the compiled pattern. If there is additional study data, this +(excluding any JIT data) follows immediately after the compiled pattern. After +writing the file, \fBpcre2test\fP expects to read a new pattern. +.P +A saved pattern can be reloaded into \fBpcre2test\fP by specifying < and a file +name instead of a pattern. There must be no space between < and the file name, +which must not contain a < character, as otherwise \fBpcre2test\fP will +interpret the line as a pattern delimited by < characters. For example: +.sp + re>