From 7c32d955a178c27bf83fd9c9fea640525cee461e Mon Sep 17 00:00:00 2001 From: "Philip.Hazel" Date: Mon, 3 Apr 2017 18:01:07 +0000 Subject: [PATCH] Documentation update --- doc/pcre2.txt | 679 +++++++++++++++++++++++++------------------------- 1 file changed, 346 insertions(+), 333 deletions(-) diff --git a/doc/pcre2.txt b/doc/pcre2.txt index 1ca720b..41f39d6 100644 --- a/doc/pcre2.txt +++ b/doc/pcre2.txt @@ -4112,8 +4112,8 @@ DIFFERENCES BETWEEN PCRE2 AND PERL 3. Capturing subpatterns that occur inside negative lookaround asser- tions are counted, but their entries in the offsets vector are set only - if the assertion is a condition. Perl has changed its behaviour in this - regard from time to time. + when a negative assertion is a condition that has a matching branch + (that is, the condition is false). 4. The following Perl escape sequences are not supported: \l, \u, \L, \U, and \N when followed by a character name or Unicode value. (\N on @@ -4266,7 +4266,7 @@ AUTHOR REVISION - Last updated: 29 March 2017 + Last updated: 03 April 2017 Copyright (c) 1997-2017 University of Cambridge. ------------------------------------------------------------------------------ @@ -7319,16 +7319,26 @@ ASSERTIONS More complicated assertions are coded as subpatterns. There are two kinds: those that look ahead of the current position in the subject - string, and those that look behind it. An assertion subpattern is - matched in the normal way, except that it does not cause the current - matching position to be changed. + string, and those that look behind it, and in each case an assertion + may be positive (must succeed for matching to continue) or negative + (must not succeed for matching to continue). An assertion subpattern is + matched in the normal way, except that, when matching continues after- + wards, the matching position in the subject string is as it was at the + start of the assertion. - Assertion subpatterns are not capturing subpatterns. If such an asser- - tion contains capturing subpatterns within it, these are counted for - the purposes of numbering the capturing subpatterns in the whole pat- - tern. However, substring capturing is normally carried out only for - positive assertions (but see the discussion of conditional subpatterns - below). + Assertion subpatterns are not capturing subpatterns. If an assertion + contains capturing subpatterns within it, these are counted for the + purposes of numbering the capturing subpatterns in the whole pattern. + However, substring capturing is carried out only for positive asser- + tions that succeed, that is, one of their branches matches, so matching + continues after the assertion. If all branches of a positive assertion + fail to match, nothing is captured, and control is passed to the previ- + ous backtracking point. + + No capturing is done for a negative assertion unless it is being used + as a condition in a conditional subpattern (see the discussion below). + Matching continues after a non-conditional negative assertion only if + all its branches fail to match. For compatibility with Perl, most assertion subpatterns may be repeated; though it makes no sense to assert the same thing several @@ -7677,54 +7687,55 @@ CONDITIONAL SUBPATTERNS strings in one of the two forms dd-aaa-dd or dd-dd-dd, where aaa are letters and dd are digits. - For Perl compatibility, if an assertion that is a condition contains - capturing subpatterns, any capturing that occurs is retained after- - wards, for both positive and negative assertions. (Compare non-condi- - tional assertions, when captures are retained only for positive asser- - tions.) + When an assertion that is a condition contains capturing subpatterns, + any capturing that occurs in a matching branch is retained afterwards, + for both positive and negative assertions, because matching always con- + tinues after the assertion, whether it succeeds or fails. (Compare non- + conditional assertions, when captures are retained only for positive + assertions that succeed.) COMMENTS There are two ways of including comments in patterns that are processed - by PCRE2. In both cases, the start of the comment must not be in a - character class, nor in the middle of any other sequence of related - characters such as (?: or a subpattern name or number. The characters + by PCRE2. In both cases, the start of the comment must not be in a + character class, nor in the middle of any other sequence of related + characters such as (?: or a subpattern name or number. The characters that make up a comment play no part in the pattern matching. - The sequence (?# marks the start of a comment that continues up to the - next closing parenthesis. Nested parentheses are not permitted. If the - PCRE2_EXTENDED option is set, an unescaped # character also introduces - a comment, which in this case continues to immediately after the next - newline character or character sequence in the pattern. Which charac- - ters are interpreted as newlines is controlled by an option passed to - the compiling function or by a special sequence at the start of the - pattern, as described in the section entitled "Newline conventions" - above. Note that the end of this type of comment is a literal newline - sequence in the pattern; escape sequences that happen to represent a - newline do not count. For example, consider this pattern when - PCRE2_EXTENDED is set, and the default newline convention (a single + The sequence (?# marks the start of a comment that continues up to the + next closing parenthesis. Nested parentheses are not permitted. If the + PCRE2_EXTENDED option is set, an unescaped # character also introduces + a comment, which in this case continues to immediately after the next + newline character or character sequence in the pattern. Which charac- + ters are interpreted as newlines is controlled by an option passed to + the compiling function or by a special sequence at the start of the + pattern, as described in the section entitled "Newline conventions" + above. Note that the end of this type of comment is a literal newline + sequence in the pattern; escape sequences that happen to represent a + newline do not count. For example, consider this pattern when + PCRE2_EXTENDED is set, and the default newline convention (a single linefeed character) is in force: abc #comment \n still comment - On encountering the # character, pcre2_compile() skips along, looking - for a newline in the pattern. The sequence \n is still literal at this - stage, so it does not terminate the comment. Only an actual character + On encountering the # character, pcre2_compile() skips along, looking + for a newline in the pattern. The sequence \n is still literal at this + stage, so it does not terminate the comment. Only an actual character with the code value 0x0a (the default newline) does so. RECURSIVE PATTERNS - Consider the problem of matching a string in parentheses, allowing for - unlimited nested parentheses. Without the use of recursion, the best - that can be done is to use a pattern that matches up to some fixed - depth of nesting. It is not possible to handle an arbitrary nesting + Consider the problem of matching a string in parentheses, allowing for + unlimited nested parentheses. Without the use of recursion, the best + that can be done is to use a pattern that matches up to some fixed + depth of nesting. It is not possible to handle an arbitrary nesting depth. For some time, Perl has provided a facility that allows regular expres- - sions to recurse (amongst other things). It does this by interpolating - Perl code in the expression at run time, and the code can refer to the + sions to recurse (amongst other things). It does this by interpolating + Perl code in the expression at run time, and the code can refer to the expression itself. A Perl pattern using code interpolation to solve the parentheses problem can be created like this: @@ -7734,179 +7745,179 @@ RECURSIVE PATTERNS refers recursively to the pattern in which it appears. Obviously, PCRE2 cannot support the interpolation of Perl code. - Instead, it supports special syntax for recursion of the entire pat- + Instead, it supports special syntax for recursion of the entire pat- tern, and also for individual subpattern recursion. After its introduc- - tion in PCRE1 and Python, this kind of recursion was subsequently + tion in PCRE1 and Python, this kind of recursion was subsequently introduced into Perl at release 5.10. - A special item that consists of (? followed by a number greater than - zero and a closing parenthesis is a recursive subroutine call of the - subpattern of the given number, provided that it occurs inside that - subpattern. (If not, it is a non-recursive subroutine call, which is - described in the next section.) The special item (?R) or (?0) is a + A special item that consists of (? followed by a number greater than + zero and a closing parenthesis is a recursive subroutine call of the + subpattern of the given number, provided that it occurs inside that + subpattern. (If not, it is a non-recursive subroutine call, which is + described in the next section.) The special item (?R) or (?0) is a recursive call of the entire regular expression. - This PCRE2 pattern solves the nested parentheses problem (assume the + This PCRE2 pattern solves the nested parentheses problem (assume the PCRE2_EXTENDED option is set so that white space is ignored): \( ( [^()]++ | (?R) )* \) - First it matches an opening parenthesis. Then it matches any number of - substrings which can either be a sequence of non-parentheses, or a - recursive match of the pattern itself (that is, a correctly parenthe- + First it matches an opening parenthesis. Then it matches any number of + substrings which can either be a sequence of non-parentheses, or a + recursive match of the pattern itself (that is, a correctly parenthe- sized substring). Finally there is a closing parenthesis. Note the use of a possessive quantifier to avoid backtracking into sequences of non- parentheses. - If this were part of a larger pattern, you would not want to recurse + If this were part of a larger pattern, you would not want to recurse the entire pattern, so instead you could use this: ( \( ( [^()]++ | (?1) )* \) ) - We have put the pattern into parentheses, and caused the recursion to + We have put the pattern into parentheses, and caused the recursion to refer to them instead of the whole pattern. - In a larger pattern, keeping track of parenthesis numbers can be - tricky. This is made easier by the use of relative references. Instead + In a larger pattern, keeping track of parenthesis numbers can be + tricky. This is made easier by the use of relative references. Instead of (?1) in the pattern above you can write (?-2) to refer to the second - most recently opened parentheses preceding the recursion. In other - words, a negative number counts capturing parentheses leftwards from + most recently opened parentheses preceding the recursion. In other + words, a negative number counts capturing parentheses leftwards from the point at which it is encountered. Be aware however, that if duplicate subpattern numbers are in use, rel- - ative references refer to the earliest subpattern with the appropriate + ative references refer to the earliest subpattern with the appropriate number. Consider, for example: (?|(a)|(b)) (c) (?-2) - The first two capturing groups (a) and (b) are both numbered 1, and - group (c) is number 2. When the reference (?-2) is encountered, the + The first two capturing groups (a) and (b) are both numbered 1, and + group (c) is number 2. When the reference (?-2) is encountered, the second most recently opened parentheses has the number 1, but it is the - first such group (the (a) group) to which the recursion refers. This - would be the same if an absolute reference (?1) was used. In other - words, relative references are just a shorthand for computing a group + first such group (the (a) group) to which the recursion refers. This + would be the same if an absolute reference (?1) was used. In other + words, relative references are just a shorthand for computing a group number. - It is also possible to refer to subsequently opened parentheses, by - writing references such as (?+2). However, these cannot be recursive - because the reference is not inside the parentheses that are refer- - enced. They are always non-recursive subroutine calls, as described in + It is also possible to refer to subsequently opened parentheses, by + writing references such as (?+2). However, these cannot be recursive + because the reference is not inside the parentheses that are refer- + enced. They are always non-recursive subroutine calls, as described in the next section. - An alternative approach is to use named parentheses. The Perl syntax - for this is (?&name); PCRE1's earlier syntax (?P>name) is also sup- + An alternative approach is to use named parentheses. The Perl syntax + for this is (?&name); PCRE1's earlier syntax (?P>name) is also sup- ported. We could rewrite the above example as follows: (? \( ( [^()]++ | (?&pn) )* \) ) - If there is more than one subpattern with the same name, the earliest + If there is more than one subpattern with the same name, the earliest one is used. The example pattern that we have been looking at contains nested unlim- - ited repeats, and so the use of a possessive quantifier for matching - strings of non-parentheses is important when applying the pattern to + ited repeats, and so the use of a possessive quantifier for matching + strings of non-parentheses is important when applying the pattern to strings that do not match. For example, when this pattern is applied to (aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa() - it yields "no match" quickly. However, if a possessive quantifier is - not used, the match runs for a very long time indeed because there are - so many different ways the + and * repeats can carve up the subject, + it yields "no match" quickly. However, if a possessive quantifier is + not used, the match runs for a very long time indeed because there are + so many different ways the + and * repeats can carve up the subject, and all have to be tested before failure can be reported. - At the end of a match, the values of capturing parentheses are those - from the outermost level. If you want to obtain intermediate values, a + At the end of a match, the values of capturing parentheses are those + from the outermost level. If you want to obtain intermediate values, a callout function can be used (see below and the pcre2callout documenta- tion). If the pattern above is matched against (ab(cd)ef) - the value for the inner capturing parentheses (numbered 2) is "ef", - which is the last value taken on at the top level. If a capturing sub- - pattern is not matched at the top level, its final captured value is - unset, even if it was (temporarily) set at a deeper level during the + the value for the inner capturing parentheses (numbered 2) is "ef", + which is the last value taken on at the top level. If a capturing sub- + pattern is not matched at the top level, its final captured value is + unset, even if it was (temporarily) set at a deeper level during the matching process. If there are more than 15 capturing parentheses in a pattern, PCRE2 has - to obtain extra memory from the heap to store data during a recursion. - If no memory can be obtained, the match fails with the + to obtain extra memory from the heap to store data during a recursion. + If no memory can be obtained, the match fails with the PCRE2_ERROR_NOMEMORY error. - Do not confuse the (?R) item with the condition (R), which tests for - recursion. Consider this pattern, which matches text in angle brack- - ets, allowing for arbitrary nesting. Only digits are allowed in nested - brackets (that is, when recursing), whereas any characters are permit- + Do not confuse the (?R) item with the condition (R), which tests for + recursion. Consider this pattern, which matches text in angle brack- + ets, allowing for arbitrary nesting. Only digits are allowed in nested + brackets (that is, when recursing), whereas any characters are permit- ted at the outer level. < (?: (?(R) \d++ | [^<>]*+) | (?R)) * > - In this pattern, (?(R) is the start of a conditional subpattern, with - two different alternatives for the recursive and non-recursive cases. + In this pattern, (?(R) is the start of a conditional subpattern, with + two different alternatives for the recursive and non-recursive cases. The (?R) item is the actual recursive call. Differences in recursion processing between PCRE2 and Perl Some former differences between PCRE2 and Perl no longer exist. - Before release 10.30, recursion processing in PCRE2 differed from Perl - in that a recursive subpattern call was always treated as an atomic - group. That is, once it had matched some of the subject string, it was - never re-entered, even if it contained untried alternatives and there - was a subsequent matching failure. (Historical note: PCRE implemented + Before release 10.30, recursion processing in PCRE2 differed from Perl + in that a recursive subpattern call was always treated as an atomic + group. That is, once it had matched some of the subject string, it was + never re-entered, even if it contained untried alternatives and there + was a subsequent matching failure. (Historical note: PCRE implemented recursion before Perl did.) - Starting with release 10.30, recursive subroutine calls are no longer + Starting with release 10.30, recursive subroutine calls are no longer treated as atomic. That is, they can be re-entered to try unused alter- - natives if there is a matching failure later in the pattern. This is - now compatible with the way Perl works. If you want a subroutine call + natives if there is a matching failure later in the pattern. This is + now compatible with the way Perl works. If you want a subroutine call to be atomic, you must explicitly enclose it in an atomic group. - Supporting backtracking into recursions simplifies certain types of + Supporting backtracking into recursions simplifies certain types of recursive pattern. For example, this pattern matches palindromic strings: ^((.)(?1)\2|.?)$ - The second branch in the group matches a single central character in - the palindrome when there are an odd number of characters, or nothing - when there are an even number of characters, but in order to work it - has to be able to try the second case when the rest of the pattern + The second branch in the group matches a single central character in + the palindrome when there are an odd number of characters, or nothing + when there are an even number of characters, but in order to work it + has to be able to try the second case when the rest of the pattern match fails. If you want to match typical palindromic phrases, the pat- - tern has to ignore all non-word characters, which can be done like + tern has to ignore all non-word characters, which can be done like this: ^\W*+((.)\W*+(?1)\W*+\2|\W*+.?)\W*+$ - If run with the PCRE2_CASELESS option, this pattern matches phrases - such as "A man, a plan, a canal: Panama!". Note the use of the posses- - sive quantifier *+ to avoid backtracking into sequences of non-word + If run with the PCRE2_CASELESS option, this pattern matches phrases + such as "A man, a plan, a canal: Panama!". Note the use of the posses- + sive quantifier *+ to avoid backtracking into sequences of non-word characters. Without this, PCRE2 takes a great deal longer (ten times or - more) to match typical phrases, and Perl takes so long that you think + more) to match typical phrases, and Perl takes so long that you think it has gone into a loop. - Another way in which PCRE2 and Perl used to differ in their recursion - processing is in the handling of captured values. Formerly in Perl, - when a subpattern was called recursively or as a subpattern (see the - next section), it had no access to any values that were captured out- - side the recursion, whereas in PCRE2 these values can be referenced. + Another way in which PCRE2 and Perl used to differ in their recursion + processing is in the handling of captured values. Formerly in Perl, + when a subpattern was called recursively or as a subpattern (see the + next section), it had no access to any values that were captured out- + side the recursion, whereas in PCRE2 these values can be referenced. Consider this pattern: ^(.)(\1|a(?2)) - This pattern matches "bab". The first capturing parentheses match "b", - then in the second group, when the back reference \1 fails to match - "b", the second alternative matches "a" and then recurses. In the - recursion, \1 does now match "b" and so the whole match succeeds. This - match used to fail in Perl, but in later versions (I tried 5.024) it + This pattern matches "bab". The first capturing parentheses match "b", + then in the second group, when the back reference \1 fails to match + "b", the second alternative matches "a" and then recurses. In the + recursion, \1 does now match "b" and so the whole match succeeds. This + match used to fail in Perl, but in later versions (I tried 5.024) it now works. SUBPATTERNS AS SUBROUTINES - If the syntax for a recursive subpattern call (either by number or by - name) is used outside the parentheses to which it refers, it operates - like a subroutine in a programming language. The called subpattern may - be defined before or after the reference. A numbered reference can be + If the syntax for a recursive subpattern call (either by number or by + name) is used outside the parentheses to which it refers, it operates + like a subroutine in a programming language. The called subpattern may + be defined before or after the reference. A numbered reference can be absolute or relative, as in these examples: (...(absolute)...)...(?2)... @@ -7917,102 +7928,102 @@ SUBPATTERNS AS SUBROUTINES (sens|respons)e and \1ibility - matches "sense and sensibility" and "response and responsibility", but + matches "sense and sensibility" and "response and responsibility", but not "sense and responsibility". If instead the pattern (sens|respons)e and (?1)ibility - is used, it does match "sense and responsibility" as well as the other - two strings. Another example is given in the discussion of DEFINE + is used, it does match "sense and responsibility" as well as the other + two strings. Another example is given in the discussion of DEFINE above. - Like recursions, subroutine calls used to be treated as atomic, but - this changed at PCRE2 release 10.30, so backtracking into subroutine - calls can now occur. However, any capturing parentheses that are set + Like recursions, subroutine calls used to be treated as atomic, but + this changed at PCRE2 release 10.30, so backtracking into subroutine + calls can now occur. However, any capturing parentheses that are set during the subroutine call revert to their previous values afterwards. - Processing options such as case-independence are fixed when a subpat- - tern is defined, so if it is used as a subroutine, such options cannot + Processing options such as case-independence are fixed when a subpat- + tern is defined, so if it is used as a subroutine, such options cannot be changed for different calls. For example, consider this pattern: (abc)(?i:(?-1)) - It matches "abcabc". It does not match "abcABC" because the change of + It matches "abcabc". It does not match "abcABC" because the change of processing option does not affect the called subpattern. ONIGURUMA SUBROUTINE SYNTAX - For compatibility with Oniguruma, the non-Perl syntax \g followed by a + For compatibility with Oniguruma, the non-Perl syntax \g followed by a name or a number enclosed either in angle brackets or single quotes, is - an alternative syntax for referencing a subpattern as a subroutine, - possibly recursively. Here are two of the examples used above, rewrit- + an alternative syntax for referencing a subpattern as a subroutine, + possibly recursively. Here are two of the examples used above, rewrit- ten using this syntax: (? \( ( (?>[^()]+) | \g )* \) ) (sens|respons)e and \g'1'ibility - PCRE2 supports an extension to Oniguruma: if a number is preceded by a + PCRE2 supports an extension to Oniguruma: if a number is preceded by a plus or a minus sign it is taken as a relative reference. For example: (abc)(?i:\g<-1>) - Note that \g{...} (Perl syntax) and \g<...> (Oniguruma syntax) are not - synonymous. The former is a back reference; the latter is a subroutine + Note that \g{...} (Perl syntax) and \g<...> (Oniguruma syntax) are not + synonymous. The former is a back reference; the latter is a subroutine call. CALLOUTS Perl has a feature whereby using the sequence (?{...}) causes arbitrary - Perl code to be obeyed in the middle of matching a regular expression. + Perl code to be obeyed in the middle of matching a regular expression. This makes it possible, amongst other things, to extract different sub- strings that match the same pair of parentheses when there is a repeti- tion. - PCRE2 provides a similar feature, but of course it cannot obey arbi- - trary Perl code. The feature is called "callout". The caller of PCRE2 - provides an external function by putting its entry point in a match - context using the function pcre2_set_callout(), and then passing that - context to pcre2_match() or pcre2_dfa_match(). If no match context is + PCRE2 provides a similar feature, but of course it cannot obey arbi- + trary Perl code. The feature is called "callout". The caller of PCRE2 + provides an external function by putting its entry point in a match + context using the function pcre2_set_callout(), and then passing that + context to pcre2_match() or pcre2_dfa_match(). If no match context is passed, or if the callout entry point is set to NULL, callouts are dis- abled. - Within a regular expression, (?C) indicates a point at which the - external function is to be called. There are two kinds of callout: - those with a numerical argument and those with a string argument. (?C) - on its own with no argument is treated as (?C0). A numerical argument - allows the application to distinguish between different callouts. - String arguments were added for release 10.20 to make it possible for - script languages that use PCRE2 to embed short scripts within patterns + Within a regular expression, (?C) indicates a point at which the + external function is to be called. There are two kinds of callout: + those with a numerical argument and those with a string argument. (?C) + on its own with no argument is treated as (?C0). A numerical argument + allows the application to distinguish between different callouts. + String arguments were added for release 10.20 to make it possible for + script languages that use PCRE2 to embed short scripts within patterns in a similar way to Perl. During matching, when PCRE2 reaches a callout point, the external func- - tion is called. It is provided with the number or string argument of - the callout, the position in the pattern, and one item of data that is + tion is called. It is provided with the number or string argument of + the callout, the position in the pattern, and one item of data that is also set in the match block. The callout function may cause matching to proceed, to backtrack, or to fail. - By default, PCRE2 implements a number of optimizations at matching - time, and one side-effect is that sometimes callouts are skipped. If - you need all possible callouts to happen, you need to set options that - disable the relevant optimizations. More details, including a complete - description of the programming interface to the callout function, are + By default, PCRE2 implements a number of optimizations at matching + time, and one side-effect is that sometimes callouts are skipped. If + you need all possible callouts to happen, you need to set options that + disable the relevant optimizations. More details, including a complete + description of the programming interface to the callout function, are given in the pcre2callout documentation. Callouts with numerical arguments - If you just want to have a means of identifying different callout - points, put a number less than 256 after the letter C. For example, + If you just want to have a means of identifying different callout + points, put a number less than 256 after the letter C. For example, this pattern has two callout points: (?C1)abc(?C2)def - If the PCRE2_AUTO_CALLOUT flag is passed to pcre2_compile(), numerical - callouts are automatically installed before each item in the pattern. - They are all numbered 255. If there is a conditional group in the pat- + If the PCRE2_AUTO_CALLOUT flag is passed to pcre2_compile(), numerical + callouts are automatically installed before each item in the pattern. + They are all numbered 255. If there is a conditional group in the pat- tern whose condition is an assertion, an additional callout is inserted - just before the condition. An explicit callout may also be set at this + just before the condition. An explicit callout may also be set at this position, as in this example: (?(?C9)(?=a)abc|def) @@ -8022,60 +8033,60 @@ CALLOUTS Callouts with string arguments - A delimited string may be used instead of a number as a callout argu- - ment. The starting delimiter must be one of ` ' " ^ % # $ { and the + A delimited string may be used instead of a number as a callout argu- + ment. The starting delimiter must be one of ` ' " ^ % # $ { and the ending delimiter is the same as the start, except for {, where the end- - ing delimiter is }. If the ending delimiter is needed within the + ing delimiter is }. If the ending delimiter is needed within the string, it must be doubled. For example: (?C'ab ''c'' d')xyz(?C{any text})pqr - The doubling is removed before the string is passed to the callout + The doubling is removed before the string is passed to the callout function. BACKTRACKING CONTROL - There are a number of special "Backtracking Control Verbs" (to use - Perl's terminology) that modify the behaviour of backtracking during - matching. They are generally of the form (*VERB) or (*VERB:NAME). Some - verbs take either form, possibly behaving differently depending on + There are a number of special "Backtracking Control Verbs" (to use + Perl's terminology) that modify the behaviour of backtracking during + matching. They are generally of the form (*VERB) or (*VERB:NAME). Some + verbs take either form, possibly behaving differently depending on whether or not a name is present. - By default, for compatibility with Perl, a name is any sequence of + By default, for compatibility with Perl, a name is any sequence of characters that does not include a closing parenthesis. The name is not - processed in any way, and it is not possible to include a closing - parenthesis in the name. This can be changed by setting the - PCRE2_ALT_VERBNAMES option, but the result is no longer Perl-compati- + processed in any way, and it is not possible to include a closing + parenthesis in the name. This can be changed by setting the + PCRE2_ALT_VERBNAMES option, but the result is no longer Perl-compati- ble. - When PCRE2_ALT_VERBNAMES is set, backslash processing is applied to - verb names and only an unescaped closing parenthesis terminates the - name. However, the only backslash items that are permitted are \Q, \E, - and sequences such as \x{100} that define character code points. Char- + When PCRE2_ALT_VERBNAMES is set, backslash processing is applied to + verb names and only an unescaped closing parenthesis terminates the + name. However, the only backslash items that are permitted are \Q, \E, + and sequences such as \x{100} that define character code points. Char- acter type escapes such as \d are faulted. A closing parenthesis can be included in a name either as \) or between - \Q and \E. In addition to backslash processing, if the PCRE2_EXTENDED - option is also set, unescaped whitespace in verb names is skipped, and - #-comments are recognized, exactly as in the rest of the pattern. + \Q and \E. In addition to backslash processing, if the PCRE2_EXTENDED + option is also set, unescaped whitespace in verb names is skipped, and + #-comments are recognized, exactly as in the rest of the pattern. PCRE2_EXTENDED does not affect verb names unless PCRE2_ALT_VERBNAMES is also set. - The maximum length of a name is 255 in the 8-bit library and 65535 in - the 16-bit and 32-bit libraries. If the name is empty, that is, if the - closing parenthesis immediately follows the colon, the effect is as if + The maximum length of a name is 255 in the 8-bit library and 65535 in + the 16-bit and 32-bit libraries. If the name is empty, that is, if the + closing parenthesis immediately follows the colon, the effect is as if the colon were not there. Any number of these verbs may occur in a pat- tern. - Since these verbs are specifically related to backtracking, most of - them can be used only when the pattern is to be matched using the tra- + Since these verbs are specifically related to backtracking, most of + them can be used only when the pattern is to be matched using the tra- ditional matching function, because that uses a backtracking algorithm. - With the exception of (*FAIL), which behaves like a failing negative + With the exception of (*FAIL), which behaves like a failing negative assertion, the backtracking control verbs cause an error if encountered by the DFA matching function. - The behaviour of these verbs in repeated groups, assertions, and in + The behaviour of these verbs in repeated groups, assertions, and in subpatterns called as subroutines (whether or not recursively) is docu- mented below. @@ -8083,71 +8094,71 @@ BACKTRACKING CONTROL PCRE2 contains some optimizations that are used to speed up matching by running some checks at the start of each match attempt. For example, it - may know the minimum length of matching subject, or that a particular + may know the minimum length of matching subject, or that a particular character must be present. When one of these optimizations bypasses the - running of a match, any included backtracking verbs will not, of + running of a match, any included backtracking verbs will not, of course, be processed. You can suppress the start-of-match optimizations - by setting the PCRE2_NO_START_OPTIMIZE option when calling pcre2_com- - pile(), or by starting the pattern with (*NO_START_OPT). There is more + by setting the PCRE2_NO_START_OPTIMIZE option when calling pcre2_com- + pile(), or by starting the pattern with (*NO_START_OPT). There is more discussion of this option in the section entitled "Compiling a pattern" in the pcre2api documentation. - Experiments with Perl suggest that it too has similar optimizations, + Experiments with Perl suggest that it too has similar optimizations, sometimes leading to anomalous results. Verbs that act immediately - The following verbs act as soon as they are encountered. They may not + The following verbs act as soon as they are encountered. They may not be followed by a name. (*ACCEPT) - This verb causes the match to end successfully, skipping the remainder - of the pattern. However, when it is inside a subpattern that is called - as a subroutine, only that subpattern is ended successfully. Matching + This verb causes the match to end successfully, skipping the remainder + of the pattern. However, when it is inside a subpattern that is called + as a subroutine, only that subpattern is ended successfully. Matching then continues at the outer level. If (*ACCEPT) in triggered in a posi- - tive assertion, the assertion succeeds; in a negative assertion, the + tive assertion, the assertion succeeds; in a negative assertion, the assertion fails. - If (*ACCEPT) is inside capturing parentheses, the data so far is cap- + If (*ACCEPT) is inside capturing parentheses, the data so far is cap- tured. For example: A((?:A|B(*ACCEPT)|C)D) - This matches "AB", "AAD", or "ACD"; when it matches "AB", "B" is cap- + This matches "AB", "AAD", or "ACD"; when it matches "AB", "B" is cap- tured by the outer parentheses. (*FAIL) or (*F) - This verb causes a matching failure, forcing backtracking to occur. It - is equivalent to (?!) but easier to read. The Perl documentation notes - that it is probably useful only when combined with (?{}) or (??{}). - Those are, of course, Perl features that are not present in PCRE2. The - nearest equivalent is the callout feature, as for example in this pat- + This verb causes a matching failure, forcing backtracking to occur. It + is equivalent to (?!) but easier to read. The Perl documentation notes + that it is probably useful only when combined with (?{}) or (??{}). + Those are, of course, Perl features that are not present in PCRE2. The + nearest equivalent is the callout feature, as for example in this pat- tern: a+(?C)(*FAIL) - A match with the string "aaaa" always fails, but the callout is taken + A match with the string "aaaa" always fails, but the callout is taken before each backtrack happens (in this example, 10 times). Recording which path was taken - There is one verb whose main purpose is to track how a match was - arrived at, though it also has a secondary use in conjunction with + There is one verb whose main purpose is to track how a match was + arrived at, though it also has a secondary use in conjunction with advancing the match starting point (see (*SKIP) below). (*MARK:NAME) or (*:NAME) - A name is always required with this verb. There may be as many - instances of (*MARK) as you like in a pattern, and their names do not + A name is always required with this verb. There may be as many + instances of (*MARK) as you like in a pattern, and their names do not have to be unique. - When a match succeeds, the name of the last-encountered (*MARK:NAME), - (*PRUNE:NAME), or (*THEN:NAME) on the matching path is passed back to - the caller as described in the section entitled "Other information - about the match" in the pcre2api documentation. Here is an example of - pcre2test output, where the "mark" modifier requests the retrieval and + When a match succeeds, the name of the last-encountered (*MARK:NAME), + (*PRUNE:NAME), or (*THEN:NAME) on the matching path is passed back to + the caller as described in the section entitled "Other information + about the match" in the pcre2api documentation. Here is an example of + pcre2test output, where the "mark" modifier requests the retrieval and outputting of (*MARK) data: re> /X(*MARK:A)Y|X(*MARK:B)Z/mark @@ -8159,72 +8170,72 @@ BACKTRACKING CONTROL MK: B The (*MARK) name is tagged with "MK:" in this output, and in this exam- - ple it indicates which of the two alternatives matched. This is a more - efficient way of obtaining this information than putting each alterna- + ple it indicates which of the two alternatives matched. This is a more + efficient way of obtaining this information than putting each alterna- tive in its own capturing parentheses. - If a verb with a name is encountered in a positive assertion that is - true, the name is recorded and passed back if it is the last-encoun- + If a verb with a name is encountered in a positive assertion that is + true, the name is recorded and passed back if it is the last-encoun- tered. This does not happen for negative assertions or failing positive assertions. - After a partial match or a failed match, the last encountered name in + After a partial match or a failed match, the last encountered name in the entire match process is returned. For example: re> /X(*MARK:A)Y|X(*MARK:B)Z/mark data> XP No match, mark = B - Note that in this unanchored example the mark is retained from the + Note that in this unanchored example the mark is retained from the match attempt that started at the letter "X" in the subject. Subsequent match attempts starting at "P" and then with an empty string do not get as far as the (*MARK) item, but nevertheless do not reset it. - If you are interested in (*MARK) values after failed matches, you - should probably set the PCRE2_NO_START_OPTIMIZE option (see above) to + If you are interested in (*MARK) values after failed matches, you + should probably set the PCRE2_NO_START_OPTIMIZE option (see above) to ensure that the match is always attempted. Verbs that act after backtracking The following verbs do nothing when they are encountered. Matching con- - tinues with what follows, but if there is no subsequent match, causing - a backtrack to the verb, a failure is forced. That is, backtracking - cannot pass to the left of the verb. However, when one of these verbs - appears inside an atomic group or in an assertion that is true, its - effect is confined to that group, because once the group has been - matched, there is never any backtracking into it. In this situation, - backtracking has to jump to the left of the entire atomic group or + tinues with what follows, but if there is no subsequent match, causing + a backtrack to the verb, a failure is forced. That is, backtracking + cannot pass to the left of the verb. However, when one of these verbs + appears inside an atomic group or in an assertion that is true, its + effect is confined to that group, because once the group has been + matched, there is never any backtracking into it. In this situation, + backtracking has to jump to the left of the entire atomic group or assertion. - These verbs differ in exactly what kind of failure occurs when back- - tracking reaches them. The behaviour described below is what happens - when the verb is not in a subroutine or an assertion. Subsequent sec- + These verbs differ in exactly what kind of failure occurs when back- + tracking reaches them. The behaviour described below is what happens + when the verb is not in a subroutine or an assertion. Subsequent sec- tions cover these special cases. (*COMMIT) - This verb, which may not be followed by a name, causes the whole match + This verb, which may not be followed by a name, causes the whole match to fail outright if there is a later matching failure that causes back- - tracking to reach it. Even if the pattern is unanchored, no further + tracking to reach it. Even if the pattern is unanchored, no further attempts to find a match by advancing the starting point take place. If - (*COMMIT) is the only backtracking verb that is encountered, once it - has been passed pcre2_match() is committed to finding a match at the + (*COMMIT) is the only backtracking verb that is encountered, once it + has been passed pcre2_match() is committed to finding a match at the current starting point, or not at all. For example: a+(*COMMIT)b - This matches "xxaab" but not "aacaab". It can be thought of as a kind + This matches "xxaab" but not "aacaab". It can be thought of as a kind of dynamic anchor, or "I've started, so I must finish." The name of the - most recently passed (*MARK) in the path is passed back when (*COMMIT) + most recently passed (*MARK) in the path is passed back when (*COMMIT) forces a match failure. - If there is more than one backtracking verb in a pattern, a different - one that follows (*COMMIT) may be triggered first, so merely passing + If there is more than one backtracking verb in a pattern, a different + one that follows (*COMMIT) may be triggered first, so merely passing (*COMMIT) during a match does not always guarantee that a match must be at this starting point. - Note that (*COMMIT) at the start of a pattern is not the same as an - anchor, unless PCRE2's start-of-match optimizations are turned off, as + Note that (*COMMIT) at the start of a pattern is not the same as an + anchor, unless PCRE2's start-of-match optimizations are turned off, as shown in this output from pcre2test: re> /(*COMMIT)abc/ @@ -8235,184 +8246,189 @@ BACKTRACKING CONTROL data> xyzabc No match - For the first pattern, PCRE2 knows that any match must start with "a", - so the optimization skips along the subject to "a" before applying the - pattern to the first set of data. The match attempt then succeeds. The - second pattern disables the optimization that skips along to the first - character. The pattern is now applied starting at "x", and so the - (*COMMIT) causes the match to fail without trying any other starting + For the first pattern, PCRE2 knows that any match must start with "a", + so the optimization skips along the subject to "a" before applying the + pattern to the first set of data. The match attempt then succeeds. The + second pattern disables the optimization that skips along to the first + character. The pattern is now applied starting at "x", and so the + (*COMMIT) causes the match to fail without trying any other starting points. (*PRUNE) or (*PRUNE:NAME) - This verb causes the match to fail at the current starting position in + This verb causes the match to fail at the current starting position in the subject if there is a later matching failure that causes backtrack- - ing to reach it. If the pattern is unanchored, the normal "bumpalong" - advance to the next starting character then happens. Backtracking can - occur as usual to the left of (*PRUNE), before it is reached, or when - matching to the right of (*PRUNE), but if there is no match to the - right, backtracking cannot cross (*PRUNE). In simple cases, the use of - (*PRUNE) is just an alternative to an atomic group or possessive quan- + ing to reach it. If the pattern is unanchored, the normal "bumpalong" + advance to the next starting character then happens. Backtracking can + occur as usual to the left of (*PRUNE), before it is reached, or when + matching to the right of (*PRUNE), but if there is no match to the + right, backtracking cannot cross (*PRUNE). In simple cases, the use of + (*PRUNE) is just an alternative to an atomic group or possessive quan- tifier, but there are some uses of (*PRUNE) that cannot be expressed in - any other way. In an anchored pattern (*PRUNE) has the same effect as + any other way. In an anchored pattern (*PRUNE) has the same effect as (*COMMIT). The behaviour of (*PRUNE:NAME) is not the same as (*MARK:NAME)(*PRUNE). It is like (*MARK:NAME) in that the name is remembered for passing back - to the caller. However, (*SKIP:NAME) searches only for names set with + to the caller. However, (*SKIP:NAME) searches only for names set with (*MARK), ignoring those set by (*PRUNE) or (*THEN). (*SKIP) - This verb, when given without a name, is like (*PRUNE), except that if - the pattern is unanchored, the "bumpalong" advance is not to the next + This verb, when given without a name, is like (*PRUNE), except that if + the pattern is unanchored, the "bumpalong" advance is not to the next character, but to the position in the subject where (*SKIP) was encoun- - tered. (*SKIP) signifies that whatever text was matched leading up to + tered. (*SKIP) signifies that whatever text was matched leading up to it cannot be part of a successful match. Consider: a+(*SKIP)b - If the subject is "aaaac...", after the first match attempt fails - (starting at the first character in the string), the starting point + If the subject is "aaaac...", after the first match attempt fails + (starting at the first character in the string), the starting point skips on to start the next attempt at "c". Note that a possessive quan- - tifer does not have the same effect as this example; although it would - suppress backtracking during the first match attempt, the second - attempt would start at the second character instead of skipping on to + tifer does not have the same effect as this example; although it would + suppress backtracking during the first match attempt, the second + attempt would start at the second character instead of skipping on to "c". (*SKIP:NAME) When (*SKIP) has an associated name, its behaviour is modified. When it is triggered, the previous path through the pattern is searched for the - most recent (*MARK) that has the same name. If one is found, the + most recent (*MARK) that has the same name. If one is found, the "bumpalong" advance is to the subject position that corresponds to that (*MARK) instead of to where (*SKIP) was encountered. If no (*MARK) with a matching name is found, the (*SKIP) is ignored. - Note that (*SKIP:NAME) searches only for names set by (*MARK:NAME). It + Note that (*SKIP:NAME) searches only for names set by (*MARK:NAME). It ignores names that are set by (*PRUNE:NAME) or (*THEN:NAME). (*THEN) or (*THEN:NAME) - This verb causes a skip to the next innermost alternative when back- - tracking reaches it. That is, it cancels any further backtracking - within the current alternative. Its name comes from the observation + This verb causes a skip to the next innermost alternative when back- + tracking reaches it. That is, it cancels any further backtracking + within the current alternative. Its name comes from the observation that it can be used for a pattern-based if-then-else block: ( COND1 (*THEN) FOO | COND2 (*THEN) BAR | COND3 (*THEN) BAZ ) ... - If the COND1 pattern matches, FOO is tried (and possibly further items - after the end of the group if FOO succeeds); on failure, the matcher - skips to the second alternative and tries COND2, without backtracking - into COND1. If that succeeds and BAR fails, COND3 is tried. If subse- - quently BAZ fails, there are no more alternatives, so there is a back- - track to whatever came before the entire group. If (*THEN) is not + If the COND1 pattern matches, FOO is tried (and possibly further items + after the end of the group if FOO succeeds); on failure, the matcher + skips to the second alternative and tries COND2, without backtracking + into COND1. If that succeeds and BAR fails, COND3 is tried. If subse- + quently BAZ fails, there are no more alternatives, so there is a back- + track to whatever came before the entire group. If (*THEN) is not inside an alternation, it acts like (*PRUNE). - The behaviour of (*THEN:NAME) is the not the same as - (*MARK:NAME)(*THEN). It is like (*MARK:NAME) in that the name is - remembered for passing back to the caller. However, (*SKIP:NAME) - searches only for names set with (*MARK), ignoring those set by + The behaviour of (*THEN:NAME) is the not the same as + (*MARK:NAME)(*THEN). It is like (*MARK:NAME) in that the name is + remembered for passing back to the caller. However, (*SKIP:NAME) + searches only for names set with (*MARK), ignoring those set by (*PRUNE) and (*THEN). - A subpattern that does not contain a | character is just a part of the - enclosing alternative; it is not a nested alternation with only one - alternative. The effect of (*THEN) extends beyond such a subpattern to - the enclosing alternative. Consider this pattern, where A, B, etc. are - complex pattern fragments that do not contain any | characters at this + A subpattern that does not contain a | character is just a part of the + enclosing alternative; it is not a nested alternation with only one + alternative. The effect of (*THEN) extends beyond such a subpattern to + the enclosing alternative. Consider this pattern, where A, B, etc. are + complex pattern fragments that do not contain any | characters at this level: A (B(*THEN)C) | D - If A and B are matched, but there is a failure in C, matching does not + If A and B are matched, but there is a failure in C, matching does not backtrack into A; instead it moves to the next alternative, that is, D. - However, if the subpattern containing (*THEN) is given an alternative, + However, if the subpattern containing (*THEN) is given an alternative, it behaves differently: A (B(*THEN)C | (*FAIL)) | D - The effect of (*THEN) is now confined to the inner subpattern. After a + The effect of (*THEN) is now confined to the inner subpattern. After a failure in C, matching moves to (*FAIL), which causes the whole subpat- - tern to fail because there are no more alternatives to try. In this + tern to fail because there are no more alternatives to try. In this case, matching does now backtrack into A. - Note that a conditional subpattern is not considered as having two - alternatives, because only one is ever used. In other words, the | + Note that a conditional subpattern is not considered as having two + alternatives, because only one is ever used. In other words, the | character in a conditional subpattern has a different meaning. Ignoring white space, consider: ^.*? (?(?=a) a | b(*THEN)c ) - If the subject is "ba", this pattern does not match. Because .*? is - ungreedy, it initially matches zero characters. The condition (?=a) - then fails, the character "b" is matched, but "c" is not. At this - point, matching does not backtrack to .*? as might perhaps be expected - from the presence of the | character. The conditional subpattern is + If the subject is "ba", this pattern does not match. Because .*? is + ungreedy, it initially matches zero characters. The condition (?=a) + then fails, the character "b" is matched, but "c" is not. At this + point, matching does not backtrack to .*? as might perhaps be expected + from the presence of the | character. The conditional subpattern is part of the single alternative that comprises the whole pattern, and so - the match fails. (If there was a backtrack into .*?, allowing it to + the match fails. (If there was a backtrack into .*?, allowing it to match "b", the match would succeed.) - The verbs just described provide four different "strengths" of control + The verbs just described provide four different "strengths" of control when subsequent matching fails. (*THEN) is the weakest, carrying on the - match at the next alternative. (*PRUNE) comes next, failing the match - at the current starting position, but allowing an advance to the next - character (for an unanchored pattern). (*SKIP) is similar, except that + match at the next alternative. (*PRUNE) comes next, failing the match + at the current starting position, but allowing an advance to the next + character (for an unanchored pattern). (*SKIP) is similar, except that the advance may be more than one character. (*COMMIT) is the strongest, causing the entire match to fail. More than one backtracking verb - If more than one backtracking verb is present in a pattern, the one - that is backtracked onto first acts. For example, consider this pat- + If more than one backtracking verb is present in a pattern, the one + that is backtracked onto first acts. For example, consider this pat- tern, where A, B, etc. are complex pattern fragments: (A(*COMMIT)B(*THEN)C|ABD) - If A matches but B fails, the backtrack to (*COMMIT) causes the entire + If A matches but B fails, the backtrack to (*COMMIT) causes the entire match to fail. However, if A and B match, but C fails, the backtrack to - (*THEN) causes the next alternative (ABD) to be tried. This behaviour - is consistent, but is not always the same as Perl's. It means that if - two or more backtracking verbs appear in succession, all the the last + (*THEN) causes the next alternative (ABD) to be tried. This behaviour + is consistent, but is not always the same as Perl's. It means that if + two or more backtracking verbs appear in succession, all the the last of them has no effect. Consider this example: ...(*COMMIT)(*PRUNE)... If there is a matching failure to the right, backtracking onto (*PRUNE) - causes it to be triggered, and its action is taken. There can never be + causes it to be triggered, and its action is taken. There can never be a backtrack onto (*COMMIT). Backtracking verbs in repeated groups - PCRE2 differs from Perl in its handling of backtracking verbs in + PCRE2 differs from Perl in its handling of backtracking verbs in repeated groups. For example, consider: /(a(*COMMIT)b)+ac/ - If the subject is "abac", Perl matches, but PCRE2 fails because the + If the subject is "abac", Perl matches, but PCRE2 fails because the (*COMMIT) in the second repeat of the group acts. Backtracking verbs in assertions - (*FAIL) in an assertion has its normal effect: it forces an immediate - backtrack. + (*FAIL) in any assertion has its normal effect: it forces an immediate + backtrack. The behaviour of the other backtracking verbs depends on + whether or not the assertion is standalone or acting as the condition + in a conditional subpattern. - (*ACCEPT) in a positive assertion causes the assertion to succeed with- - out any further processing. In a negative assertion, (*ACCEPT) causes - the assertion to fail without any further processing. + (*ACCEPT) in a standalone positive assertion causes the assertion to + succeed without any further processing; captured strings are retained. + In a standalone negative assertion, (*ACCEPT) causes the assertion to + fail without any further processing; captured substrings are discarded. + + If the assertion is a condition, (*ACCEPT) causes the condition to be + true for a positive assertion and false for a negative one; captured + substrings are retained in both cases. + + The effect of (*THEN) is not allowed to escape beyond an assertion. If + there are no more branches to try, (*THEN) causes a positive assertion + to be false, and a negative assertion to be true. The other backtracking verbs are not treated specially if they appear - in a positive assertion. In particular, (*THEN) skips to the next - alternative in the innermost enclosing group that has alternations, - whether or not this is within the assertion. - - Negative assertions are, however, different, in order to ensure that - changing a positive assertion into a negative assertion changes its - result. Backtracking into (*COMMIT), (*SKIP), or (*PRUNE) causes a neg- - ative assertion to be true, without considering any further alternative - branches in the assertion. Backtracking into (*THEN) causes it to skip - to the next enclosing alternative within the assertion (the normal be- - haviour), but if the assertion does not have such an alternative, - (*THEN) behaves like (*PRUNE). + in a standalone positive assertion. In a conditional positive asser- + tion, backtracking into (*COMMIT), (*SKIP), or (*PRUNE) causes the con- + dition to be false. However, for both standalone and conditional nega- + tive assertions, backtracking into (*COMMIT), (*SKIP), or (*PRUNE) + causes the assertion to be true, without considering any further alter- + native branches. Backtracking verbs in subroutines @@ -8449,7 +8465,7 @@ AUTHOR REVISION - Last updated: 18 March 2017 + Last updated: 03 April 2017 Copyright (c) 1997-2017 University of Cambridge. ------------------------------------------------------------------------------ @@ -9225,9 +9241,6 @@ REVISION ------------------------------------------------------------------------------ ------------------------------------------------------------------------------- - - PCRE2SYNTAX(3) Library Functions Manual PCRE2SYNTAX(3)