-PCRE2_SIZE pcre2_get_leftchar(pcre2_match_data *match_data);
-
-
PCRE2_SPTR pcre2_get_mark(pcre2_match_data *match_data);
@@ -102,9 +99,6 @@ document for an overview of all the PCRE2 documentation.
PCRE2_SIZE *pcre2_get_ovector_pointer(pcre2_match_data *match_data);
-PCRE2_SIZE pcre2_get_rightchar(pcre2_match_data *match_data);
-
-
PCRE2_SIZE pcre2_get_startchar(pcre2_match_data *match_data);
For those options that can be different in different parts of the pattern, the contents of the options argument specifies their settings at the start of -compilation. The PCRE2_ANCHORED, PCRE2_NO_UTF_CHECK, and -PCRE2_NO_START_OPTIMIZE options can be set at the time of matching as well as -at compile time. +compilation. The PCRE2_ANCHORED and PCRE2_NO_UTF_CHECK options can be set at +the time of matching as well as at compile time.
Other, less frequently required compile-time parameters (for example, the @@ -1176,14 +1140,55 @@ purposes.
PCRE2_NO_START_OPTIMIZE-This is an option that acts at matching time; that is, it is really an option -for pcre2_match() or pcre_dfa_match(). If it is set at compile -time, it is remembered with the compiled pattern and assumed at matching time. -This is necessary if you want to use JIT execution, because the JIT compiler -needs to know whether or not this option is set. For details, see the -discussion of PCRE2_NO_START_OPTIMIZE in the section on pcre2_match() -options -below. +This is an option whose main effect is at matching time. It does not change +what pcre2_compile() generates, but it does affect the output of the JIT +compiler. + +
+There are a number of optimizations that may occur at the start of a match, in +order to speed up the process. For example, if it is known that an unanchored +match must start with a specific character, the matching code searches the +subject for that character, and fails immediately if it cannot find it, without +actually running the main matching function. This means that a special item +such as (*COMMIT) at the start of a pattern is not considered until after a +suitable starting point for the match has been found. Also, when callouts or +(*MARK) items are in use, these "start-up" optimizations can cause them to be +skipped if the pattern is never actually used. The start-up optimizations are +in effect a pre-scan of the subject that takes place before the pattern is run. +
++The PCRE2_NO_START_OPTIMIZE option disables the start-up optimizations, +possibly causing performance to suffer, but ensuring that in cases where the +result is "no match", the callouts do occur, and that items such as (*COMMIT) +and (*MARK) are considered at every possible starting position in the subject +string. +
++Setting PCRE2_NO_START_OPTIMIZE may change the outcome of a matching operation. +Consider the pattern +
+ (*COMMIT)ABC ++When this is compiled, PCRE2 records the fact that a match must start with the +character "A". Suppose the subject string is "DEFABC". The start-up +optimization scans along the subject, finds "A" and runs the first match +attempt from there. The (*COMMIT) item means that the pattern must match the +current starting position, which in this case, it does. However, if the same +match is run with PCRE2_NO_START_OPTIMIZE set, the initial scan along the +subject string does not happen. The first match attempt is run starting from +"D" and when this fails, (*COMMIT) prevents any further matches being tried, so +the overall result is "no match". There are also other start-up optimizations. +For example, a minimum length for the subject may be recorded. Consider the +pattern +
+ (*MARK:A)(X|Y) ++The minimum length for a match is one character. If the subject is "ABC", there +will be attempts to match "ABC", "BC", and "C". An attempt to match an empty +string at the end of the subject does not take place, because PCRE2 knows that +the subject is now too short, and so the (*MARK) is never encountered. In this +case, the optimization does not affect the overall match result, which is still +"no match", but it does affect the auxiliary information that is returned.
PCRE2_NO_UTF_CHECK@@ -1648,13 +1653,15 @@ string that define the matched part of the subject and any substrings that were capured. This is know as the ovector.
-Before calling pcre2_match() or pcre2_dfa_match() you must create a +Before calling pcre2_match() or pcre2_dfa_match() you must create a match data block by calling one of the creation functions above. For pcre2_match_data_create(), the first argument is the number of pairs of offsets in the ovector. One pair of offsets is required to identify the string that matched the whole pattern, with another pair for each captured -substring. For example, a value of 4 creates enough space to record the -matched portion of the subject plus three captured substrings. +substring. For example, a value of 4 creates enough space to record the matched +portion of the subject plus three captured substrings. A minimum of at least 1 +pair is imposed by pcre2_match_data_create(), so it is always possible to +return the overall matched string.
For pcre2_match_data_create_from_pattern(), the first argument is a
@@ -1779,10 +1786,9 @@ Option bits for pcre2_match()
The unused bits of the options argument for pcre2_match() must be -zero. The only bits that may be set are PCRE2_ANCHORED, -PCRE2_NOTBOL, PCRE2_NOTEOL, PCRE2_NOTEMPTY, PCRE2_NOTEMPTY_ATSTART, -PCRE2_NO_START_OPTIMIZE, PCRE2_NO_UTF_CHECK, PCRE2_PARTIAL_HARD, and -PCRE2_PARTIAL_SOFT. Their action is described below. +zero. The only bits that may be set are PCRE2_ANCHORED, PCRE2_NOTBOL, +PCRE2_NOTEOL, PCRE2_NOTEMPTY, PCRE2_NOTEMPTY_ATSTART, PCRE2_NO_UTF_CHECK, +PCRE2_PARTIAL_HARD, and PCRE2_PARTIAL_SOFT. Their action is described below.
If the pattern was successfully processed by the just-in-time (JIT) compiler, @@ -1833,56 +1839,6 @@ valid, so PCRE2 searches further into the string for occurrences of "a" or "b". This is like PCRE2_NOTEMPTY, except that an empty string match that is not at the start of the subject is permitted. If the pattern is anchored, such a match can occur only if the pattern contains \K. -
- PCRE2_NO_START_OPTIMIZE --There are a number of optimizations that pcre2_match() uses at the start -of a match, in order to speed up the process. For example, if it is known that -an unanchored match must start with a specific character, it searches the -subject for that character, and fails immediately if it cannot find it, without -actually running the main matching function. This means that a special item -such as (*COMMIT) at the start of a pattern is not considered until after a -suitable starting point for the match has been found. Also, when callouts or -(*MARK) items are in use, these "start-up" optimizations can cause them to be -skipped if the pattern is never actually used. The start-up optimizations are -in effect a pre-scan of the subject that takes place before the pattern is run. - -
-The PCRE2_NO_START_OPTIMIZE option disables the start-up optimizations, -possibly causing performance to suffer, but ensuring that in cases where the -result is "no match", the callouts do occur, and that items such as (*COMMIT) -and (*MARK) are considered at every possible starting position in the subject -string. If PCRE2_NO_START_OPTIMIZE is set at compile time, it cannot be unset -at matching time. The use of PCRE2_NO_START_OPTIMIZE at matching time (that is, -passing it to pcre2_match()) disables JIT execution; in this situation, -matching is always done using interpretively. -
--Setting PCRE2_NO_START_OPTIMIZE can change the outcome of a matching operation. -Consider the pattern -
- (*COMMIT)ABC --When this is compiled, PCRE2 records the fact that a match must start with the -character "A". Suppose the subject string is "DEFABC". The start-up -optimization scans along the subject, finds "A" and runs the first match -attempt from there. The (*COMMIT) item means that the pattern must match the -current starting position, which in this case, it does. However, if the same -match is run with PCRE2_NO_START_OPTIMIZE set, the initial scan along the -subject string does not happen. The first match attempt is run starting from -"D" and when this fails, (*COMMIT) prevents any further matches being tried, so -the overall result is "no match". There are also other start-up optimizations. -For example, a minimum length for the subject may be recorded. Consider the -pattern -
- (*MARK:A)(X|Y) --The minimum length for a match is one character. If the subject is "ABC", there -will be attempts to match "ABC", "BC", and "C". An attempt to match an empty -string at the end of the subject does not take place, because PCRE2 knows that -the subject is now too short, and so the (*MARK) is never encountered. In this -case, the optimization does not affect the overall match result, which is still -"no match", but it does affect the auxiliary information that is returned.
PCRE2_NO_UTF_CHECK@@ -2035,13 +1991,13 @@ returned.
If the ovector is too small to hold all the captured substring offsets, as much -as possible is filled in, and the function returns a value of zero. If neither -the actual string matched nor any captured substrings are of interest, -pcre2_match() may be called with a match data block whose ovector is of -zero length. However, if the pattern contains back references and the -ovector is not big enough to remember the related substrings, PCRE2 has -to get additional memory for use during matching. Thus it is usually advisable -to set up a match data block containing an ovector of reasonable size. +as possible is filled in, and the function returns a value of zero. If captured +substrings are not of interest, pcre2_match() may be called with a match +data block whose ovector is of minimum length (that is, one pair). However, if +the pattern contains back references and the ovector is not big enough to +remember the related substrings, PCRE2 has to get additional memory for use +during matching. Thus it is usually advisable to set up a match data block +containing an ovector of reasonable size.
It is possible for capturing subpattern number n+1 to match some part of
@@ -2074,12 +2030,6 @@ Other information about the match
PCRE2_SPTR pcre2_get_mark(pcre2_match_data *match_data);
-PCRE2_SIZE pcre2_get_leftchar(pcre2_match_data *match_data);
-
-
-PCRE2_SIZE pcre2_get_rightchar(pcre2_match_data *match_data);
-
-
PCRE2_SIZE pcre2_get_startchar(pcre2_match_data *match_data);
@@ -2093,39 +2043,10 @@ Otherwise NULL is returned. A (*MARK) name may be available after a failed match or a partial match, as well as after a successful one.
-The other three functions yield values that give information about the part of
-the subject string that was inspected during a successful match or a partial
-match. Their results are undefined after a failed match. They return the
-following values, respectively:
-
-
-(1) The offset of the leftmost character that was inspected during the match.
-This can be earlier than the point at which the match started if the pattern
-contains lookbehind assertions or \b or \B at the start.
-
-
-(2) The offset of the character that follows the rightmost character that was
-inspected during the match. This can be after the end of the match if the
-pattern contains lookahead assertions.
-
-
-(3) The offset of the character at which the successful or partial match
-started. This can be different to the value of ovector[0] if the pattern
-contains the \K escape sequence.
-
-For example, if the pattern (?<=abc)xx\Kyy(?=def) is matched against the -string "123abcxxyydef123", the resulting offsets are: -
- ovector[0] 8 - ovector[1] 10 - leftchar 3 - rightchar 13 - startchar 6 --The allusedtext modifier in pcre2test can be used to display a -longer string that shows the leftmost and rightmost characters in a match -instead of just the matched string. +The offset of the character at which the successful match started is +returned by pcre2_get_startchar(). This can be different to the value of +ovector[0] if the pattern contains the \K escape sequence. Note, +however, the \K has no effect for a partial match.
PCRE2_PARTIAL_HARD PCRE2_PARTIAL_SOFT @@ -2650,7 +2570,7 @@ Cambridge CB2 3QH, England.
REVISION
-Last updated: 16 September 2014 +Last updated: 14 October 2014
Copyright © 1997-2014 University of Cambridge.
diff --git a/doc/html/pcre2callout.html b/doc/html/pcre2callout.html index c742f90..6d2c3c6 100644 --- a/doc/html/pcre2callout.html +++ b/doc/html/pcre2callout.html @@ -131,7 +131,7 @@ long enough, or, for unanchored patterns, if it has been scanned far enough.You can disable these optimizations by passing the PCRE2_NO_START_OPTIMIZE -option to the matching function, or by starting the pattern with +option to pcre2_compile(), or by starting the pattern with (*NO_START_OPT). This slows down the matching process, but does ensure that callouts such as the example above are obeyed.
diff --git a/doc/html/pcre2jit.html b/doc/html/pcre2jit.html index 18158dc..79c99dc 100644 --- a/doc/html/pcre2jit.html +++ b/doc/html/pcre2jit.html @@ -128,9 +128,8 @@ or the JIT compiler was not able to handle the pattern.The pcre2_match() options that are supported for JIT matching are PCRE2_NOTBOL, PCRE2_NOTEOL, PCRE2_NOTEMPTY, PCRE2_NOTEMPTY_ATSTART, -PCRE2_NO_UTF_CHECK, PCRE2_PARTIAL_HARD, and PCRE2_PARTIAL_SOFT. The options -that are not supported at match time are PCRE2_ANCHORED and -PCRE2_NO_START_OPTIMIZE, though they are supported if given at compile time. +PCRE2_NO_UTF_CHECK, PCRE2_PARTIAL_HARD, and PCRE2_PARTIAL_SOFT. The +PCRE2_ANCHORED option is not supported at match time.
The only unsupported pattern items are \C (match a single data unit) when diff --git a/doc/html/pcre2partial.html b/doc/html/pcre2partial.html new file mode 100644 index 0000000..463e6df --- /dev/null +++ b/doc/html/pcre2partial.html @@ -0,0 +1,464 @@ + +
+pcre2partial specification + + +pcre2partial man page
++Return to the PCRE2 index page. +
++This page is part of the PCRE2 HTML documentation. It was generated +automatically from the original man page. If there is any nonsense in it, +please consult the man page, in case the conversion went wrong. +
+
+In normal use of PCRE2, if the subject string that is passed to a matching +function matches as far as it goes, but is too short to match the entire +pattern, PCRE2_ERROR_NOMATCH is returned. There are circumstances where it +might be helpful to distinguish this case from other cases in which there is no +match. +
++Consider, for example, an application where a human is required to type in data +for a field with specific formatting requirements. An example might be a date +in the form ddmmmyy, defined by this pattern: +
+ ^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$ ++If the application sees the user's keystrokes one by one, and can check that +what has been typed so far is potentially valid, it is able to raise an error +as soon as a mistake is made, by beeping and not reflecting the character that +has been typed, for example. This immediate feedback is likely to be a better +user interface than a check that is delayed until the entire string has been +entered. Partial matching can also be useful when the subject string is very +long and is not all available at once. + +
+PCRE2 supports partial matching by means of the PCRE2_PARTIAL_SOFT and +PCRE2_PARTIAL_HARD options, which can be set when calling a matching function. +The difference between the two options is whether or not a partial match is +preferred to an alternative complete match, though the details differ between +the two types of matching function. If both options are set, PCRE2_PARTIAL_HARD +takes precedence. +
++If you want to use partial matching with just-in-time optimized code, you must +call pcre2_jit_compile() with one or both of these options: +
+ PCRE2_JIT_PARTIAL_SOFT + PCRE2_JIT_PARTIAL_HARD ++PCRE2_JIT_COMPLETE should also be set if you are going to run non-partial +matches on the same pattern. If the appropriate JIT mode has not been compiled, +interpretive matching code is used. + +
+Setting a partial matching option disables two of PCRE2's standard +optimizations. PCRE2 remembers the last literal code unit in a pattern, and +abandons matching immediately if it is not present in the subject string. This +optimization cannot be used for a subject string that might match only +partially. PCRE2 also knows the minimum length of a matching string, and does +not bother to run the matching function on shorter strings. This optimization +is also disabled for partial matching. +
++A partial match occurs during a call to pcre2_match() when the end of the +subject string is reached successfully, but matching cannot continue because +more characters are needed. However, at least one character in the subject must +have been inspected. This character need not form part of the final matched +string; lookbehind assertions and the \K escape sequence provide ways of +inspecting characters before the start of a matched string. The requirement for +inspecting at least one character exists because an empty string can always be +matched; without such a restriction there would always be a partial match of an +empty string at the end of the subject. +
++When a partial match is returned, the first two elements in the ovector point +to the portion of the subject that was matched. The appearance of \K in the +pattern has no effect for a partial match. Consider this pattern: +
+ /abc\K123/ ++If it is matched against "456abc123xyz" the result is a complete match, and the +ovector defines the matched string as "123", because \K resets the "start of +match" point. However, if a partial match is requested and the subject string +is "456abc12", a partial match is found for the string "abc12", because all +these characters are needed for a subsequent re-match with additional +characters. + +
+What happens when a partial match is identified depends on which of the two +partial matching options are set. +
++If PCRE2_PARTIAL_SOFT is set when pcre2_match() identifies a partial +match, the partial match is remembered, but matching continues as normal, and +other alternatives in the pattern are tried. If no complete match can be found, +PCRE2_ERROR_PARTIAL is returned instead of PCRE2_ERROR_NOMATCH. +
++This option is "soft" because it prefers a complete match over a partial match. +All the various matching items in a pattern behave as if the subject string is +potentially complete. For example, \z, \Z, and $ match at the end of the +subject, as normal, and for \b and \B the end of the subject is treated as a +non-alphanumeric. +
++If there is more than one partial match, the first one that was found provides +the data that is returned. Consider this pattern: +
+ /123\w+X|dogY/ ++If this is matched against the subject string "abc123dog", both +alternatives fail to match, but the end of the subject is reached during +matching, so PCRE2_ERROR_PARTIAL is returned. The offsets are set to 3 and 9, +identifying "123dog" as the first partial match that was found. (In this +example, there are two partial matches, because "dog" on its own partially +matches the second alternative.) + +
+If PCRE2_PARTIAL_HARD is set for pcre2_match(), PCRE2_ERROR_PARTIAL is +returned as soon as a partial match is found, without continuing to search for +possible complete matches. This option is "hard" because it prefers an earlier +partial match over a later complete match. For this reason, the assumption is +made that the end of the supplied subject string may not be the true end of the +available data, and so, if \z, \Z, \b, \B, or $ are encountered at the end +of the subject, the result is PCRE2_ERROR_PARTIAL, provided that at least one +character in the subject has been inspected. +
++The difference between the two partial matching options can be illustrated by a +pattern such as: +
+ /dog(sbody)?/ ++This matches either "dog" or "dogsbody", greedily (that is, it prefers the +longer string if possible). If it is matched against the string "dog" with +PCRE2_PARTIAL_SOFT, it yields a complete match for "dog". However, if +PCRE2_PARTIAL_HARD is set, the result is PCRE2_ERROR_PARTIAL. On the other +hand, if the pattern is made ungreedy the result is different: +
+ /dog(sbody)??/ ++In this case the result is always a complete match because that is found first, +and matching never continues after finding a complete match. It might be easier +to follow this explanation by thinking of the two patterns like this: +
+ /dog(sbody)?/ is the same as /dogsbody|dog/ + /dog(sbody)??/ is the same as /dog|dogsbody/ ++The second pattern will never match "dogsbody", because it will always find the +shorter match first. + +
+The DFA functions move along the subject string character by character, without +backtracking, searching for all possible matches simultaneously. If the end of +the subject is reached before the end of the pattern, there is the possibility +of a partial match, again provided that at least one character has been +inspected. +
++When PCRE2_PARTIAL_SOFT is set, PCRE2_ERROR_PARTIAL is returned only if there +have been no complete matches. Otherwise, the complete matches are returned. +However, if PCRE2_PARTIAL_HARD is set, a partial match takes precedence over +any complete matches. The portion of the string that was matched when the +longest partial match was found is set as the first matching string. +
++Because the DFA functions always search for all possible matches, and there is +no difference between greedy and ungreedy repetition, their behaviour is +different from the standard functions when PCRE2_PARTIAL_HARD is set. Consider +the string "dog" matched against the ungreedy pattern shown above: +
+ /dog(sbody)??/ ++Whereas the standard functions stop as soon as they find the complete match for +"dog", the DFA functions also find the partial match for "dogsbody", and so +return that when PCRE2_PARTIAL_HARD is set. + +
+If a pattern ends with one of sequences \b or \B, which test for word +boundaries, partial matching with PCRE2_PARTIAL_SOFT can give counter-intuitive +results. Consider this pattern: +
+ /\bcat\b/ ++This matches "cat", provided there is a word boundary at either end. If the +subject string is "the cat", the comparison of the final "t" with a following +character cannot take place, so a partial match is found. However, normal +matching carries on, and \b matches at the end of the subject when the last +character is a letter, so a complete match is found. The result, therefore, is +not PCRE2_ERROR_PARTIAL. Using PCRE2_PARTIAL_HARD in this case does yield +PCRE2_ERROR_PARTIAL, because then the partial match takes precedence. + +
+If the partial_soft (or ps) modifier is present on a +pcre2test data line, the PCRE2_PARTIAL_SOFT option is used for the match. +Here is a run of pcre2test that uses the date example quoted above: +
+ re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ + data> 25jun04\=ps + 0: 25jun04 + 1: jun + data> 25dec3\=ps + Partial match: 23dec3 + data> 3ju\=ps + Partial match: 3ju + data> 3juj\=ps + No match + data> j\=ps + No match ++The first data string is matched completely, so pcre2test shows the +matched substrings. The remaining four strings do not match the complete +pattern, but the first two are partial matches. Similar output is obtained +if DFA matching is used. + +
+If the partial_hard (or ph) modifier is present on a +pcre2test data line, the PCRE2_PARTIAL_HARD option is set for the match. +
++When a partial match has been found using a DFA matching function, it is +possible to continue the match by providing additional subject data and calling +the function again with the same compiled regular expression, this time setting +the PCRE2_DFA_RESTART option. You must pass the same working space as before, +because this is where details of the previous partial match are stored. Here is +an example using pcre2test: +
+ re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ + data> 23ja\=dfa,ps + Partial match: 23ja + data> n05\=dfa,dfa_restart + 0: n05 ++The first call has "23ja" as the subject, and requests partial matching; the +second call has "n05" as the subject for the continued (restarted) match. +Notice that when the match is complete, only the last part is shown; PCRE2 does +not retain the previously partially-matched string. It is up to the calling +program to do that if it needs to. + +
+That means that, for an unanchored pattern, if a continued match fails, it is +not possible to try again at a new starting point. All this facility is capable +of doing is continuing with the previous match attempt. In the previous +example, if the second set of data is "ug23" the result is no match, even +though there would be a match for "aug23" if the entire string were given at +once. Depending on the application, this may or may not be what you want. +The only way to allow for starting again at the next character is to retain the +matched part of the subject and try a new complete match. +
++You can set the PCRE2_PARTIAL_SOFT or PCRE2_PARTIAL_HARD options with +PCRE2_DFA_RESTART to continue partial matching over multiple segments. This +facility can be used to pass very long subject strings to the DFA matching +functions. +
++Unlike the DFA function, it is not possible to restart the previous match with +a new segment of data when using pcre2_match(). Instead, new data must be +added to the previous subject string, and the entire match re-run, starting +from the point where the partial match occurred. Earlier data can be discarded. +
++It is best to use PCRE2_PARTIAL_HARD in this situation, because it does not +treat the end of a segment as the end of the subject when matching \z, \Z, +\b, \B, and $. Consider an unanchored pattern that matches dates: +
+ re> /\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d/ + data> The date is 23ja\=ph + Partial match: 23ja ++At this stage, an application could discard the text preceding "23ja", add on +text from the next segment, and call the matching function again. Unlike the +DFA matching function, the entire matching string must always be available, +and the complete matching process occurs for each call, so more memory and more +processing time is needed. + +
+Certain types of pattern may give problems with multi-segment matching, +whichever matching function is used. +
++1. If the pattern contains a test for the beginning of a line, you need to pass +the PCRE2_NOTBOL option when the subject string for any call does start at the +beginning of a line. There is also a PCRE2_NOTEOL option, but in practice when +doing multi-segment matching you should be using PCRE2_PARTIAL_HARD, which +includes the effect of PCRE2_NOTEOL. +
++2. If a pattern contains a lookbehind assertion, characters that precede the +start of the partial match may have been inspected during the matching process. +When using pcre2_match(), sufficient characters must be retained for the +next match attempt. You can ensure that enough characters are retained by doing +the following: +
++Before doing any matching, find the length of the longest lookbehind in the +pattern by calling pcre2_pattern_info() with the PCRE2_INFO_MAXLOOKBEHIND +option. Note that the resulting count is in characters, not code units. After a +partial match, moving back from the ovector[0] offset in the subject by the +number of characters given for the maximum lookbehind gets you to the earliest +character that must be retained. In a non-UTF or a 32-bit situation, moving +back is just a subtraction, but in UTF-8 or UTF-16 you have to count characters +while moving back through the code units. +
++Characters before the point you have now reached can be discarded, and after +the next segment has been added to what is retained, you should run the next +match with the startoffset argument set so that the match begins at the +same point as before. +
++For example, if the pattern "(?<=123)abc" is partially matched against the +string "xx123ab", the ovector offsets are 5 and 7 ("ab"). The maximum +lookbehind count is 3, so all characters before offset 2 can be discarded. The +value of startoffset for the next match should be 3. When pcre2test +displays a partial match, it indicates the lookbehind characters with '<' +characters: +
+ re> "(?<=123)abc" + data> xx123ab\=ph + Partial match: 123ab + <<< ++ +
+3. Because a partial match must always contain at least one character, what +might be considered a partial match of an empty string actually gives a "no +match" result. For example: +
+ re> /c(?<=abc)x/ + data> ab\=ps + No match ++If the next segment begins "cx", a match should be found, but this will only +happen if characters from the previous segment are retained. For this reason, a +"no match" result should be interpreted as "partial match of an empty string" +when the pattern contains lookbehinds. + +
+4. Matching a subject string that is split into multiple segments may not +always produce exactly the same result as matching over one single long string, +especially when PCRE2_PARTIAL_SOFT is used. The section "Partial Matching and +Word Boundaries" above describes an issue that arises if the pattern ends with +\b or \B. Another kind of difference may occur when there are multiple +matching possibilities, because (for PCRE2_PARTIAL_SOFT) a partial match result +is given only when there are no completed matches. This means that as soon as +the shortest match has been found, continuation to a new subject segment is no +longer possible. Consider this pcre2test example: +
+ re> /dog(sbody)?/ + data> dogsb\=ps + 0: dog + data> do\=ps,dfa + Partial match: do + data> gsb\=ps,dfa,dfa_restart + 0: g + data> dogsbody\=dfa + 0: dogsbody + 1: dog ++The first data line passes the string "dogsb" to a standard matching function, +setting the PCRE2_PARTIAL_SOFT option. Although the string is a partial match +for "dogsbody", the result is not PCRE2_ERROR_PARTIAL, because the shorter +string "dog" is a complete match. Similarly, when the subject is presented to +a DFA matching function in several parts ("do" and "gsb" being the first two) +the match stops when "dog" has been found, and it is not possible to continue. +On the other hand, if "dogsbody" is presented as a single string, a DFA +matching function finds both matches. + +
+Because of these problems, it is best to use PCRE2_PARTIAL_HARD when matching +multi-segment data. The example above then behaves differently: +
+ re> /dog(sbody)?/ + data> dogsb\=ph + Partial match: dogsb + data> do\=ps,dfa + Partial match: do + data> gsb\=ph,dfa,dfa_restart + Partial match: gsb ++5. Patterns that contain alternatives at the top level which do not all start +with the same pattern item may not work as expected when PCRE2_DFA_RESTART is +used. For example, consider this pattern: +
+ 1234|3789 ++If the first part of the subject is "ABC123", a partial match of the first +alternative is found at offset 3. There is no partial match for the second +alternative, because such a match does not start at the same point in the +subject string. Attempting to continue with the string "7890" does not yield a +match because only those alternatives that match at one point in the subject +are remembered. The problem arises because the start of the second alternative +matches within the first alternative. There is no problem with anchored +patterns or patterns such as: +
+ 1234|ABCD ++where no string can be a partial match for both alternatives. This is not a +problem if a standard matching function is used, because the entire match has +to be rerun each time: +
+ re> /1234|3789/ + data> ABC123\=ph + Partial match: 123 + data> 1237890 + 0: 3789 ++Of course, instead of using PCRE2_DFA_RESTART, the same technique of re-running +the entire match can also be used with the DFA matching function. Another +possibility is to work with two buffers. If a partial match at offset n +in the first buffer is followed by "no match" when PCRE2_DFA_RESTART is used on +the second buffer, you can then try a new match starting at offset n+1 in +the first buffer. + +
+Philip Hazel
+
+University Computing Service
+
+Cambridge CB2 3QH, England.
+
+
+Last updated: 14 October 2014
+
+Copyright © 1997-2014 University of Cambridge.
+
+
+Return to the PCRE2 index page. +
diff --git a/doc/html/pcre2test.html b/doc/html/pcre2test.html index 30b527d..617c582 100644 --- a/doc/html/pcre2test.html +++ b/doc/html/pcre2test.html @@ -476,6 +476,7 @@ about the pattern: /I info show info about compiled pattern hex pattern is coded in hexadecimal jit[=<number>] use JIT + jitverify verify JIT use locale=<name> use this locale memory show memory used newline=<type> set newline type @@ -503,10 +504,6 @@ The newline modifier specifies which characters are to be interpreted as newlines, both in the pattern and (by default) in subject lines. The type must be one of CR, LF, CRLF, ANYCRLF, or ANY. --Both the \R and newline settings can be changed at match time, but if this is -done, JIT matching is disabled. -
-The /jit modifier may optionally be followed by a number in the range 0 -to 7: +The /jit modifier may optionally be followed by and equals sign and a +number in the range 0 to 7:
0 disable JIT - 1 normal match only - 2 soft partial match only - 3 normal match and soft partial match - 4 hard partial match only - 6 soft and hard partial match + 1 use JIT for normal match only + 2 use JIT for soft partial match only + 3 use JIT for normal match and soft partial match + 4 use JIT for hard partial match only + 6 use JIT for soft and hard partial match 7 all three modesIf no number is given, 7 is assumed. If JIT compilation is successful, the -compiled JIT code will automatically be used when pcre2_match() is run, -except when incompatible run-time options are specified. For more details, see -the +compiled JIT code will automatically be used when pcre2_match() is run +for the appropriate type of match, except when incompatible run-time options +are specified. For more details, see the pcre2jit documentation. See also the jitstack modifier below for a way of setting the size of the JIT stack.
-If the jitverify modifier is specified, the text "(JIT)" is added to the -first output line after a match or non match when JIT-compiled code was -actually used. This modifier can also be set on a subject line. +If the jitverify modifier is specified, information about the compiled +pattern shows whether JIT compilation was or was not successful. If +jitverify is specified without jit, jit=7 is assumed. If JIT +compilation is successful when jitverify is set, the text "(JIT)" is +added to the first output line after a match or non match when JIT-compiled +code was actually used.
-These modifiers set the newline and \R processing conventions for the subject -line, overriding any values that were set at compile time (as described above). -JIT matching is disabled if these settings are changed at match time. -
-@@ -781,11 +768,13 @@ substring. In each case the remainder is output on the following line with a plus character following the capture number.
-The allusedtext modifier requests that all the text that was consulted -during a successful pattern match be shown. This affects the output if there -is a lookbehind at the start of a match, or a lookahead at the end, or if \K -is used in the pattern. Characters that precede or follow the start and end of -the actual match are indicated in the output by '<' or '>' characters +The allusedtext modifier requests that all the text that was consulted +during a successful pattern match by the interpreter should be shown. This +feature is not supported for JIT matching, and if requested with JIT it is +ignored (with a warning message). Setting this modifier affects the output if +there is a lookbehind at the start of a match, or a lookahead at the end, or if +\K is used in the pattern. Characters that precede or follow the start and end +of the actual match are indicated in the output by '<' or '>' characters underneath them. Here is an example:
/(?<=pqr)abc(?=xyz)/ @@ -903,6 +892,11 @@ until it finds the minimum values for each parameter that allow pcre2_match() to complete without error.+If JIT is being used, only the match limit is relevant. If DFA matching is +being used, neither limit is relevant, and this modifier is ignored (with a +warning message). +
+The match_limit number is a measure of the amount of backtracking that takes place, and learning the minimum value can be instructive. For most simple matches, the number is quite small, but for patterns with very large @@ -944,6 +938,13 @@ appears, though of course it can also be used to set a default in a #subject command. It specifies the number of pairs of offsets that are available for storing matching information. The default is 15.
++At least one pair of offsets is always created by +pcre2_match_data_create(), for matching with PCRE2's native API, so a +value of 0 is the same as 1. However a value of 0 is useful when testing the +POSIX API because it causes regexec() to be called with a NULL capture +vector. +
THE ALTERNATIVE MATCHING FUNCTION
By default, pcre2test uses the standard PCRE2 matching function, @@ -1190,7 +1191,7 @@ Cambridge CB2 3QH, England.
REVISION
-Last updated: 19 August 2014 +Last updated: 11 October 2014
Copyright © 1997-2014 University of Cambridge.
diff --git a/doc/pcre2.txt b/doc/pcre2.txt index 52b7406..18e5cc1 100644 --- a/doc/pcre2.txt +++ b/doc/pcre2.txt @@ -52,16 +52,12 @@ PCRE2 NATIVE API BASIC FUNCTIONS PCRE2 NATIVE API AUXILIARY MATCH FUNCTIONS - PCRE2_SIZE pcre2_get_leftchar(pcre2_match_data *match_data); - PCRE2_SPTR pcre2_get_mark(pcre2_match_data *match_data); uint32_t pcre2_get_ovector_count(pcre2_match_data *match_data); PCRE2_SIZE *pcre2_get_ovector_pointer(pcre2_match_data *match_data); - PCRE2_SIZE pcre2_get_rightchar(pcre2_match_data *match_data); - PCRE2_SIZE pcre2_get_startchar(pcre2_match_data *match_data); @@ -87,13 +83,13 @@ PCRE2 NATIVE API COMPILE CONTEXT FUNCTIONS void pcre2_compile_context_free(pcre2_compile_context *ccontext); - int pcre2_set_bsr_compile(pcre2_compile_context *ccontext, + int pcre2_set_bsr(pcre2_compile_context *ccontext, uint32_t value); int pcre2_set_character_tables(pcre2_compile_context *ccontext, const unsigned char *tables); - int pcre2_set_newline_compile(pcre2_compile_context *ccontext, + int pcre2_set_newline(pcre2_compile_context *ccontext, uint32_t value); int pcre2_set_parens_nest_limit(pcre2_compile_context *ccontext, @@ -113,9 +109,6 @@ PCRE2 NATIVE API MATCH CONTEXT FUNCTIONS void pcre2_match_context_free(pcre2_match_context *mcontext); - int pcre2_set_bsr_match(pcre2_match_context *mcontext, - uint32_t value); - int pcre2_set_callout(pcre2_match_context *mcontext, int (*callout_function)(pcre2_callout_block *), void *callout_data); @@ -123,9 +116,6 @@ PCRE2 NATIVE API MATCH CONTEXT FUNCTIONS int pcre2_set_match_limit(pcre2_match_context *mcontext, uint32_t value); - int pcre2_set_newline_match(pcre2_match_context *mcontext, - uint32_t value); - int pcre2_set_recursion_limit(pcre2_match_context *mcontext, uint32_t value); @@ -501,7 +491,7 @@ PCRE2 CONTEXTS These can be changed by calling the following functions, which return 0 on success, or PCRE2_ERROR_BADDATA if invalid data is detected. - int pcre2_set_bsr_compile(pcre2_compile_context *ccontext, + int pcre2_set_bsr(pcre2_compile_context *ccontext, uint32_t value); The value must be PCRE2_BSR_ANYCRLF, to specify that \R matches only @@ -509,9 +499,7 @@ PCRE2 CONTEXTS Unicode line ending sequence. The value of this parameter does not affect what is compiled; it is just saved with the compiled pattern. The value is used by the JIT compiler and by the two interpreted match- - ing functions, pcre2_match() and pcre2_dfa_match(). You can change the - value when calling these functions, but doing so disables the use of - JIT. + ing functions, pcre2_match() and pcre2_dfa_match(). int pcre2_set_character_tables(pcre2_compile_context *ccontext, const unsigned char *tables); @@ -520,7 +508,7 @@ PCRE2 CONTEXTS only argument is a general context. This function builds a set of char- acter tables in the current locale. - int pcre2_set_newline_compile(pcre2_compile_context *ccontext, + int pcre2_set_newline(pcre2_compile_context *ccontext, uint32_t value); This specifies which characters or character sequences are to be recog- @@ -533,9 +521,7 @@ PCRE2 CONTEXTS this parameter affects the recognition of white space and the end of internal comments starting with #. The value is saved with the compiled pattern for subsequent use by the JIT compiler and by the two inter- - preted matching functions, pcre2_match() and pcre2_dfa_match(). You can - change the value when calling these functions, but doing so disables - the use of JIT. + preted matching functions, pcre2_match() and pcre2_dfa_match(). int pcre2_set_parens_nest_limit(pcre2_compile_context *ccontext, uint32_t value); @@ -588,15 +574,6 @@ PCRE2 CONTEXTS These can be changed by calling the following functions, which return 0 on success, or PCRE2_ERROR_BADDATA if invalid data is detected. - int pcre2_set_bsr_match(pcre2_match_context *mcontext, - uint32_t value); - - The value must be PCRE2_BSR_ANYCRLF, to specify that \R matches only - CR, LF, or CRLF, or PCRE2_BSR_UNICODE, to specify that \R matches any - Unicode line ending sequence. If you want to make use of JIT matching, - you should not use this function, but instead set the value in a com- - pile context. - int pcre2_set_callout(pcre2_match_context *mcontext, int (*callout_function)(pcre2_callout_block *), void *callout_data); @@ -668,17 +645,6 @@ PCRE2 CONTEXTS unless ddd is less than the limit set by the caller of pcre2_match() or, if no such limit is set, less than the default. - int pcre2_set_newline_match(pcre2_match_context *mcontext, - uint32_t value); - - This specifies which characters or character sequences are to be recog- - nized as newlines. The value must be one of PCRE2_NEWLINE_CR (carriage - return only), PCRE2_NEWLINE_LF (linefeed only), PCRE2_NEWLINE_CRLF (the - two-character sequence CR followed by LF), PCRE2_NEWLINE_ANYCRLF (any - of the above), or PCRE2_NEWLINE_ANY (any Unicode newline sequence). If - you want to make use of JIT matching, you should not use this function, - but instead set the value in a compile context. - int pcre2_set_recursion_memory_management( pcre2_match_context *mcontext, void *(*private_malloc)(PCRE2_SIZE, void *), @@ -852,30 +818,29 @@ COMPILING A PATTERN For those options that can be different in different parts of the pat- tern, the contents of the options argument specifies their settings at - the start of compilation. The PCRE2_ANCHORED, PCRE2_NO_UTF_CHECK, and - PCRE2_NO_START_OPTIMIZE options can be set at the time of matching as - well as at compile time. + the start of compilation. The PCRE2_ANCHORED and PCRE2_NO_UTF_CHECK + options can be set at the time of matching as well as at compile time. - Other, less frequently required compile-time parameters (for example, + Other, less frequently required compile-time parameters (for example, the newline setting) can be provided in a compile context (as described above). If errorcode or erroroffset is NULL, pcre2_compile() returns NULL imme- - diately. Otherwise, if compilation of a pattern fails, pcre2_compile() + diately. Otherwise, if compilation of a pattern fails, pcre2_compile() returns NULL, having set these variables to an error code and an offset - (number of code units) within the pattern, respectively. The - pcre2_get_error_message() function provides a textual message for each + (number of code units) within the pattern, respectively. The + pcre2_get_error_message() function provides a textual message for each error code. Compilation errors are positive numbers, but UTF formatting errors are negative numbers. For an invalid UTF-8 or UTF-16 string, the offset is that of the first code unit of the failing character. - Some errors are not detected until the whole pattern has been scanned; - in these cases, the offset passed back is the length of the pattern. - Note that the offset is in code units, not characters, even in a UTF + Some errors are not detected until the whole pattern has been scanned; + in these cases, the offset passed back is the length of the pattern. + Note that the offset is in code units, not characters, even in a UTF mode. It may sometimes point into the middle of a UTF-8 or UTF-16 char- acter. - This code fragment shows a typical straightforward call to pcre2_com- + This code fragment shows a typical straightforward call to pcre2_com- pile(): pcre2_code *re; @@ -889,158 +854,158 @@ COMPILING A PATTERN &erroffset, /* for error offset */ NULL); /* no compile context */ - The following names for option bits are defined in the pcre2.h header + The following names for option bits are defined in the pcre2.h header file: PCRE2_ANCHORED If this bit is set, the pattern is forced to be "anchored", that is, it - is constrained to match only at the first matching point in the string - that is being searched (the "subject string"). This effect can also be - achieved by appropriate constructs in the pattern itself, which is the + is constrained to match only at the first matching point in the string + that is being searched (the "subject string"). This effect can also be + achieved by appropriate constructs in the pattern itself, which is the only way to do it in Perl. PCRE2_ALLOW_EMPTY_CLASS - By default, for compatibility with Perl, a closing square bracket that - immediately follows an opening one is treated as a data character for - the class. When PCRE2_ALLOW_EMPTY_CLASS is set, it terminates the + By default, for compatibility with Perl, a closing square bracket that + immediately follows an opening one is treated as a data character for + the class. When PCRE2_ALLOW_EMPTY_CLASS is set, it terminates the class, which therefore contains no characters and so can never match. PCRE2_ALT_BSUX - This option request alternative handling of three escape sequences, - which makes PCRE2's behaviour more like ECMAscript (aka JavaScript). + This option request alternative handling of three escape sequences, + which makes PCRE2's behaviour more like ECMAscript (aka JavaScript). When it is set: (1) \U matches an upper case "U" character; by default \U causes a com- pile time error (Perl uses \U to upper case subsequent characters). (2) \u matches a lower case "u" character unless it is followed by four - hexadecimal digits, in which case the hexadecimal number defines the - code point to match. By default, \u causes a compile time error (Perl + hexadecimal digits, in which case the hexadecimal number defines the + code point to match. By default, \u causes a compile time error (Perl uses it to upper case the following character). - (3) \x matches a lower case "x" character unless it is followed by two - hexadecimal digits, in which case the hexadecimal number defines the - code point to match. By default, as in Perl, a hexadecimal number is + (3) \x matches a lower case "x" character unless it is followed by two + hexadecimal digits, in which case the hexadecimal number defines the + code point to match. By default, as in Perl, a hexadecimal number is always expected after \x, but it may have zero, one, or two digits (so, for example, \xz matches a binary zero character followed by z). PCRE2_AUTO_CALLOUT - If this bit is set, pcre2_compile() automatically inserts callout + If this bit is set, pcre2_compile() automatically inserts callout items, all with number 255, before each pattern item. For discussion of the callout facility, see the pcre2callout documentation. PCRE2_CASELESS - If this bit is set, letters in the pattern match both upper and lower - case letters in the subject. It is equivalent to Perl's /i option, and + If this bit is set, letters in the pattern match both upper and lower + case letters in the subject. It is equivalent to Perl's /i option, and it can be changed within a pattern by a (?i) option setting. PCRE2_DOLLAR_ENDONLY - If this bit is set, a dollar metacharacter in the pattern matches only - at the end of the subject string. Without this option, a dollar also - matches immediately before a newline at the end of the string (but not - before any other newlines). The PCRE2_DOLLAR_ENDONLY option is ignored - if PCRE2_MULTILINE is set. There is no equivalent to this option in + If this bit is set, a dollar metacharacter in the pattern matches only + at the end of the subject string. Without this option, a dollar also + matches immediately before a newline at the end of the string (but not + before any other newlines). The PCRE2_DOLLAR_ENDONLY option is ignored + if PCRE2_MULTILINE is set. There is no equivalent to this option in Perl, and no way to set it within a pattern. PCRE2_DOTALL - If this bit is set, a dot metacharacter in the pattern matches any - character, including one that indicates a newline. However, it only + If this bit is set, a dot metacharacter in the pattern matches any + character, including one that indicates a newline. However, it only ever matches one character, even if newlines are coded as CRLF. Without this option, a dot does not match when the current position in the sub- - ject is at a newline. This option is equivalent to Perl's /s option, + ject is at a newline. This option is equivalent to Perl's /s option, and it can be changed within a pattern by a (?s) option setting. A neg- ative class such as [^a] always matches newline characters, independent of the setting of this option. PCRE2_DUPNAMES - If this bit is set, names used to identify capturing subpatterns need + If this bit is set, names used to identify capturing subpatterns need not be unique. This can be helpful for certain types of pattern when it - is known that only one instance of the named subpattern can ever be - matched. There are more details of named subpatterns below; see also + is known that only one instance of the named subpattern can ever be + matched. There are more details of named subpatterns below; see also the pcre2pattern documentation. PCRE2_EXTENDED - If this bit is set, most white space characters in the pattern are - totally ignored except when escaped or inside a character class. How- - ever, white space is not allowed within sequences such as (?> that + If this bit is set, most white space characters in the pattern are + totally ignored except when escaped or inside a character class. How- + ever, white space is not allowed within sequences such as (?> that introduce various parenthesized subpatterns, nor within numerical quan- - tifiers such as {1,3}. Ignorable white space is permitted between an - item and a following quantifier and between a quantifier and a follow- + tifiers such as {1,3}. Ignorable white space is permitted between an + item and a following quantifier and between a quantifier and a follow- ing + that indicates possessiveness. - PCRE2_EXTENDED also causes characters between an unescaped # outside a - character class and the next newline, inclusive, to be ignored, which + PCRE2_EXTENDED also causes characters between an unescaped # outside a + character class and the next newline, inclusive, to be ignored, which makes it possible to include comments inside complicated patterns. Note - that the end of this type of comment is a literal newline sequence in + that the end of this type of comment is a literal newline sequence in the pattern; escape sequences that happen to represent a newline do not - count. PCRE2_EXTENDED is equivalent to Perl's /x option, and it can be + count. PCRE2_EXTENDED is equivalent to Perl's /x option, and it can be changed within a pattern by a (?x) option setting. Which characters are interpreted as newlines can be specified by a set- - ting in the compile context that is passed to pcre2_compile() or by a - special sequence at the start of the pattern, as described in the sec- - tion entitled "Newline conventions" in the pcre2pattern documentation. + ting in the compile context that is passed to pcre2_compile() or by a + special sequence at the start of the pattern, as described in the sec- + tion entitled "Newline conventions" in the pcre2pattern documentation. A default is defined when PCRE2 is built. PCRE2_FIRSTLINE - If this option is set, an unanchored pattern is required to match - before or at the first newline in the subject string, though the + If this option is set, an unanchored pattern is required to match + before or at the first newline in the subject string, though the matched text may continue over the newline. PCRE2_MATCH_UNSET_BACKREF - If this option is set, a back reference to an unset subpattern group - matches an empty string (by default this causes the current matching - alternative to fail). A pattern such as (\1)(a) succeeds when this - option is set (assuming it can find an "a" in the subject), whereas it - fails by default, for Perl compatibility. Setting this option makes + If this option is set, a back reference to an unset subpattern group + matches an empty string (by default this causes the current matching + alternative to fail). A pattern such as (\1)(a) succeeds when this + option is set (assuming it can find an "a" in the subject), whereas it + fails by default, for Perl compatibility. Setting this option makes PCRE2 behave more like ECMAscript (aka JavaScript). PCRE2_MULTILINE - By default, for the purposes of matching "start of line" and "end of - line", PCRE2 treats the subject string as consisting of a single line - of characters, even if it actually contains newlines. The "start of - line" metacharacter (^) matches only at the start of the string, and - the "end of line" metacharacter ($) matches only at the end of the + By default, for the purposes of matching "start of line" and "end of + line", PCRE2 treats the subject string as consisting of a single line + of characters, even if it actually contains newlines. The "start of + line" metacharacter (^) matches only at the start of the string, and + the "end of line" metacharacter ($) matches only at the end of the string, or before a terminating newline (except when PCRE2_DOL- - LAR_ENDONLY is set). Note, however, that unless PCRE2_DOTALL is set, + LAR_ENDONLY is set). Note, however, that unless PCRE2_DOTALL is set, the "any character" metacharacter (.) does not match at a newline. This behaviour (for ^, $, and dot) is the same as Perl. - When PCRE2_MULTILINE it is set, the "start of line" and "end of line" - constructs match immediately following or immediately before internal - newlines in the subject string, respectively, as well as at the very - start and end. This is equivalent to Perl's /m option, and it can be + When PCRE2_MULTILINE it is set, the "start of line" and "end of line" + constructs match immediately following or immediately before internal + newlines in the subject string, respectively, as well as at the very + start and end. This is equivalent to Perl's /m option, and it can be changed within a pattern by a (?m) option setting. If there are no new- - lines in a subject string, or no occurrences of ^ or $ in a pattern, + lines in a subject string, or no occurrences of ^ or $ in a pattern, setting PCRE2_MULTILINE has no effect. PCRE2_NEVER_UCP - This option locks out the use of Unicode properties for handling \B, + This option locks out the use of Unicode properties for handling \B, \b, \D, \d, \S, \s, \W, \w, and some of the POSIX character classes, as - described for the PCRE2_UCP option below. In particular, it prevents - the creator of the pattern from enabling this facility by starting the - pattern with (*UCP). This may be useful in applications that process - patterns from external sources. The option combination PCRE_UCP and + described for the PCRE2_UCP option below. In particular, it prevents + the creator of the pattern from enabling this facility by starting the + pattern with (*UCP). This may be useful in applications that process + patterns from external sources. The option combination PCRE_UCP and PCRE_NEVER_UCP causes an error. PCRE2_NEVER_UTF - This option locks out interpretation of the pattern as UTF-8, UTF-16, + This option locks out interpretation of the pattern as UTF-8, UTF-16, or UTF-32, depending on which library is in use. In particular, it pre- - vents the creator of the pattern from switching to UTF interpretation + vents the creator of the pattern from switching to UTF interpretation by starting the pattern with (*UTF). This may be useful in applications that process patterns from external sources. The combination of PCRE2_UTF and PCRE2_NEVER_UTF causes an error. @@ -1048,30 +1013,73 @@ COMPILING A PATTERN PCRE2_NO_AUTO_CAPTURE If this option is set, it disables the use of numbered capturing paren- - theses in the pattern. Any opening parenthesis that is not followed by - ? behaves as if it were followed by ?: but named parentheses can still - be used for capturing (and they acquire numbers in the usual way). + theses in the pattern. Any opening parenthesis that is not followed by + ? behaves as if it were followed by ?: but named parentheses can still + be used for capturing (and they acquire numbers in the usual way). There is no equivalent of this option in Perl. PCRE2_NO_AUTO_POSSESS If this option is set, it disables "auto-possessification", which is an - optimization that, for example, turns a+b into a++b in order to avoid - backtracks into a+ that can never be successful. However, if callouts - are in use, auto-possessification means that some callouts are never + optimization that, for example, turns a+b into a++b in order to avoid + backtracks into a+ that can never be successful. However, if callouts + are in use, auto-possessification means that some callouts are never taken. You can set this option if you want the matching functions to do - a full unoptimized search and run all the callouts, but it is mainly + a full unoptimized search and run all the callouts, but it is mainly provided for testing purposes. PCRE2_NO_START_OPTIMIZE - This is an option that acts at matching time; that is, it is really an - option for pcre2_match() or pcre_dfa_match(). If it is set at compile - time, it is remembered with the compiled pattern and assumed at match- - ing time. This is necessary if you want to use JIT execution, because - the JIT compiler needs to know whether or not this option is set. For - details, see the discussion of PCRE2_NO_START_OPTIMIZE in the section - on pcre2_match() options below. + This is an option whose main effect is at matching time. It does not + change what pcre2_compile() generates, but it does affect the output of + the JIT compiler. + + There are a number of optimizations that may occur at the start of a + match, in order to speed up the process. For example, if it is known + that an unanchored match must start with a specific character, the + matching code searches the subject for that character, and fails imme- + diately if it cannot find it, without actually running the main match- + ing function. This means that a special item such as (*COMMIT) at the + start of a pattern is not considered until after a suitable starting + point for the match has been found. Also, when callouts or (*MARK) + items are in use, these "start-up" optimizations can cause them to be + skipped if the pattern is never actually used. The start-up optimiza- + tions are in effect a pre-scan of the subject that takes place before + the pattern is run. + + The PCRE2_NO_START_OPTIMIZE option disables the start-up optimizations, + possibly causing performance to suffer, but ensuring that in cases + where the result is "no match", the callouts do occur, and that items + such as (*COMMIT) and (*MARK) are considered at every possible starting + position in the subject string. + + Setting PCRE2_NO_START_OPTIMIZE may change the outcome of a matching + operation. Consider the pattern + + (*COMMIT)ABC + + When this is compiled, PCRE2 records the fact that a match must start + with the character "A". Suppose the subject string is "DEFABC". The + start-up optimization scans along the subject, finds "A" and runs the + first match attempt from there. The (*COMMIT) item means that the pat- + tern must match the current starting position, which in this case, it + does. However, if the same match is run with PCRE2_NO_START_OPTIMIZE + set, the initial scan along the subject string does not happen. The + first match attempt is run starting from "D" and when this fails, + (*COMMIT) prevents any further matches being tried, so the overall + result is "no match". There are also other start-up optimizations. For + example, a minimum length for the subject may be recorded. Consider the + pattern + + (*MARK:A)(X|Y) + + The minimum length for a match is one character. If the subject is + "ABC", there will be attempts to match "ABC", "BC", and "C". An attempt + to match an empty string at the end of the subject does not take place, + because PCRE2 knows that the subject is now too short, and so the + (*MARK) is never encountered. In this case, the optimization does not + affect the overall match result, which is still "no match", but it does + affect the auxiliary information that is returned. PCRE2_NO_UTF_CHECK @@ -1524,7 +1532,9 @@ THE MATCH DATA BLOCK string that matched the whole pattern, with another pair for each cap- tured substring. For example, a value of 4 creates enough space to record the matched portion of the subject plus three captured sub- - strings. + strings. A minimum of at least 1 pair is imposed by + pcre2_match_data_create(), so it is always possible to return the over- + all matched string. For pcre2_match_data_create_from_pattern(), the first argument is a pointer to a compiled pattern. In this case the ovector is created to @@ -1636,8 +1646,8 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION The unused bits of the options argument for pcre2_match() must be zero. The only bits that may be set are PCRE2_ANCHORED, PCRE2_NOTBOL, PCRE2_NOTEOL, PCRE2_NOTEMPTY, PCRE2_NOTEMPTY_ATSTART, - PCRE2_NO_START_OPTIMIZE, PCRE2_NO_UTF_CHECK, PCRE2_PARTIAL_HARD, and - PCRE2_PARTIAL_SOFT. Their action is described below. + PCRE2_NO_UTF_CHECK, PCRE2_PARTIAL_HARD, and PCRE2_PARTIAL_SOFT. Their + action is described below. If the pattern was successfully processed by the just-in-time (JIT) compiler, the only supported options for matching using the JIT code @@ -1691,58 +1701,6 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION not at the start of the subject is permitted. If the pattern is anchored, such a match can occur only if the pattern contains \K. - PCRE2_NO_START_OPTIMIZE - - There are a number of optimizations that pcre2_match() uses at the - start of a match, in order to speed up the process. For example, if it - is known that an unanchored match must start with a specific character, - it searches the subject for that character, and fails immediately if it - cannot find it, without actually running the main matching function. - This means that a special item such as (*COMMIT) at the start of a pat- - tern is not considered until after a suitable starting point for the - match has been found. Also, when callouts or (*MARK) items are in use, - these "start-up" optimizations can cause them to be skipped if the pat- - tern is never actually used. The start-up optimizations are in effect a - pre-scan of the subject that takes place before the pattern is run. - - The PCRE2_NO_START_OPTIMIZE option disables the start-up optimizations, - possibly causing performance to suffer, but ensuring that in cases - where the result is "no match", the callouts do occur, and that items - such as (*COMMIT) and (*MARK) are considered at every possible starting - position in the subject string. If PCRE2_NO_START_OPTIMIZE is set at - compile time, it cannot be unset at matching time. The use of - PCRE2_NO_START_OPTIMIZE at matching time (that is, passing it to - pcre2_match()) disables JIT execution; in this situation, matching is - always done using interpretively. - - Setting PCRE2_NO_START_OPTIMIZE can change the outcome of a matching - operation. Consider the pattern - - (*COMMIT)ABC - - When this is compiled, PCRE2 records the fact that a match must start - with the character "A". Suppose the subject string is "DEFABC". The - start-up optimization scans along the subject, finds "A" and runs the - first match attempt from there. The (*COMMIT) item means that the pat- - tern must match the current starting position, which in this case, it - does. However, if the same match is run with PCRE2_NO_START_OPTIMIZE - set, the initial scan along the subject string does not happen. The - first match attempt is run starting from "D" and when this fails, - (*COMMIT) prevents any further matches being tried, so the overall - result is "no match". There are also other start-up optimizations. For - example, a minimum length for the subject may be recorded. Consider the - pattern - - (*MARK:A)(X|Y) - - The minimum length for a match is one character. If the subject is - "ABC", there will be attempts to match "ABC", "BC", and "C". An attempt - to match an empty string at the end of the subject does not take place, - because PCRE2 knows that the subject is now too short, and so the - (*MARK) is never encountered. In this case, the optimization does not - affect the overall match result, which is still "no match", but it does - affect the auxiliary information that is returned. - PCRE2_NO_UTF_CHECK When PCRE2_UTF is set at compile time, the validity of the subject as a @@ -1871,13 +1829,13 @@ HOW PCRE2_MATCH() RETURNS A STRING AND CAPTURED SUBSTRINGS If the ovector is too small to hold all the captured substring offsets, as much as possible is filled in, and the function returns a value of - zero. If neither the actual string matched nor any captured substrings - are of interest, pcre2_match() may be called with a match data block - whose ovector is of zero length. However, if the pattern contains back - references and the ovector is not big enough to remember the related - substrings, PCRE2 has to get additional memory for use during matching. - Thus it is usually advisable to set up a match data block containing an - ovector of reasonable size. + zero. If captured substrings are not of interest, pcre2_match() may be + called with a match data block whose ovector is of minimum length (that + is, one pair). However, if the pattern contains back references and the + ovector is not big enough to remember the related substrings, PCRE2 has + to get additional memory for use during matching. Thus it is usually + advisable to set up a match data block containing an ovector of reason- + able size. It is possible for capturing subpattern number n+1 to match some part of the subject when subpattern n has not been used at all. For example, @@ -1904,10 +1862,6 @@ HOW PCRE2_MATCH() RETURNS A STRING AND CAPTURED SUBSTRINGS PCRE2_SPTR pcre2_get_mark(pcre2_match_data *match_data); - PCRE2_SIZE pcre2_get_leftchar(pcre2_match_data *match_data); - - PCRE2_SIZE pcre2_get_rightchar(pcre2_match_data *match_data); - PCRE2_SIZE pcre2_get_startchar(pcre2_match_data *match_data); In addition to the offsets in the ovector, other information about a @@ -1920,44 +1874,19 @@ HOW PCRE2_MATCH() RETURNS A STRING AND CAPTURED SUBSTRINGS after a failed match or a partial match, as well as after a successful one. - The other three functions yield values that give information about the - part of the subject string that was inspected during a successful match - or a partial match. Their results are undefined after a failed match. - They return the following values, respectively: - - (1) The offset of the leftmost character that was inspected during the - match. This can be earlier than the point at which the match started - if the pattern contains lookbehind assertions or \b or \B at the start. - - (2) The offset of the character that follows the rightmost character - that was inspected during the match. This can be after the end of the - match if the pattern contains lookahead assertions. - - (3) The offset of the character at which the successful or partial - match started. This can be different to the value of ovector[0] if the - pattern contains the \K escape sequence. - - For example, if the pattern (?<=abc)xx\Kyy(?=def) is matched against - the string "123abcxxyydef123", the resulting offsets are: - - ovector[0] 8 - ovector[1] 10 - leftchar 3 - rightchar 13 - startchar 6 - - The allusedtext modifier in pcre2test can be used to display a longer - string that shows the leftmost and rightmost characters in a match - instead of just the matched string. + The offset of the character at which the successful match started is + returned by pcre2_get_startchar(). This can be different to the value + of ovector[0] if the pattern contains the \K escape sequence. Note, + however, the \K has no effect for a partial match. Error return values from pcre2_match() - If pcre2_match() fails, it returns a negative number. This can be con- - verted to a text string by calling pcre2_get_error_message(). Negative - error codes are also returned by other functions, and are documented + If pcre2_match() fails, it returns a negative number. This can be con- + verted to a text string by calling pcre2_get_error_message(). Negative + error codes are also returned by other functions, and are documented with them. The codes are given names in the header file. If UTF check- ing is in force and an invalid UTF subject string is detected, one of a - number of UTF-specific negative error codes is returned. Details are + number of UTF-specific negative error codes is returned. Details are given in the pcre2unicode page. The following are the other errors that may be returned by pcre2_match(): @@ -1967,19 +1896,19 @@ HOW PCRE2_MATCH() RETURNS A STRING AND CAPTURED SUBSTRINGS PCRE2_ERROR_PARTIAL - The subject string did not match, but it did match partially. See the + The subject string did not match, but it did match partially. See the pcre2partial documentation for details of partial matching. PCRE2_ERROR_BADMAGIC PCRE2 stores a 4-byte "magic number" at the start of the compiled code, - to catch the case when it is passed a junk pointer. This is the error + to catch the case when it is passed a junk pointer. This is the error that is returned when the magic number is not present. PCRE2_ERROR_BADMODE - This error is given when a pattern that was compiled by the 8-bit - library is passed to a 16-bit or 32-bit library function, or vice + This error is given when a pattern that was compiled by the 8-bit + library is passed to a 16-bit or 32-bit library function, or vice versa. PCRE2_ERROR_BADOFFSET @@ -1993,35 +1922,35 @@ HOW PCRE2_MATCH() RETURNS A STRING AND CAPTURED SUBSTRINGS PCRE2_ERROR_BADUTFOFFSET The UTF code unit sequence that was passed as a subject was checked and - found to be valid (the PCRE2_NO_UTF_CHECK option was not set), but the - value of startoffset did not point to the beginning of a UTF character + found to be valid (the PCRE2_NO_UTF_CHECK option was not set), but the + value of startoffset did not point to the beginning of a UTF character or the end of the subject. PCRE2_ERROR_CALLOUT - This error is never generated by pcre2_match() itself. It is provided + This error is never generated by pcre2_match() itself. It is provided for use by callout functions that want to cause pcre2_match() to return - a distinctive error code. See the pcre2callout documentation for + a distinctive error code. See the pcre2callout documentation for details. PCRE2_ERROR_INTERNAL - An unexpected internal error has occurred. This error could be caused + An unexpected internal error has occurred. This error could be caused by a bug in PCRE2 or by overwriting of the compiled pattern. PCRE2_ERROR_JIT_BADOPTION - This error is returned when a pattern that was successfully studied - using JIT is being matched, but the matching mode (partial or complete - match) does not correspond to any JIT compilation mode. When the JIT - fast path function is used, this error may be also given for invalid + This error is returned when a pattern that was successfully studied + using JIT is being matched, but the matching mode (partial or complete + match) does not correspond to any JIT compilation mode. When the JIT + fast path function is used, this error may be also given for invalid options. See the pcre2jit documentation for more details. PCRE2_ERROR_JIT_STACKLIMIT - This error is returned when a pattern that was successfully studied - using JIT is being matched, but the memory available for the just-in- - time processing stack is not large enough. See the pcre2jit documenta- + This error is returned when a pattern that was successfully studied + using JIT is being matched, but the memory available for the just-in- + time processing stack is not large enough. See the pcre2jit documenta- tion for more details. PCRE2_ERROR_MATCHLIMIT @@ -2030,10 +1959,10 @@ HOW PCRE2_MATCH() RETURNS A STRING AND CAPTURED SUBSTRINGS PCRE2_ERROR_NOMEMORY - If a pattern contains back references, but the ovector is not big - enough to remember the referenced substrings, PCRE2 gets a block of + If a pattern contains back references, but the ovector is not big + enough to remember the referenced substrings, PCRE2 gets a block of memory at the start of matching to use for this purpose. There are some - other special cases where extra memory is needed during matching. This + other special cases where extra memory is needed during matching. This error is given when memory cannot be obtained. PCRE2_ERROR_NULL @@ -2042,12 +1971,12 @@ HOW PCRE2_MATCH() RETURNS A STRING AND CAPTURED SUBSTRINGS PCRE2_ERROR_RECURSELOOP - This error is returned when pcre2_match() detects a recursion loop - within the pattern. Specifically, it means that either the whole pat- + This error is returned when pcre2_match() detects a recursion loop + within the pattern. Specifically, it means that either the whole pat- tern or a subpattern has been called recursively for the second time at - the same position in the subject string. Some simple patterns that - might do this are detected and faulted at compile time, but more com- - plicated cases, in particular mutual recursions between two different + the same position in the subject string. Some simple patterns that + might do this are detected and faulted at compile time, but more com- + plicated cases, in particular mutual recursions between two different subpatterns, cannot be detected until run time. PCRE2_ERROR_RECURSIONLIMIT @@ -2070,52 +1999,52 @@ EXTRACTING CAPTURED SUBSTRINGS BY NUMBER void pcre2_substring_free(PCRE2_UCHAR *buffer); - Captured substrings can be accessed directly by using the ovector as + Captured substrings can be accessed directly by using the ovector as described above. For convenience, auxiliary functions are provided for - extracting captured substrings as new, separate, zero-terminated - strings. The functions in this section identify substrings by number. - The next section describes similar functions for extracting substrings + extracting captured substrings as new, separate, zero-terminated + strings. The functions in this section identify substrings by number. + The next section describes similar functions for extracting substrings by name. A substring that contains a binary zero is correctly extracted - and has a further zero added on the end, but the result is not, of + and has a further zero added on the end, but the result is not, of course, a C string. - You can find the length in code units of a captured substring without - extracting it by calling pcre2_substring_length_bynumber(). The first - argument is a pointer to the match data block, the second is the group - number, and the third is a pointer to a variable into which the length + You can find the length in code units of a captured substring without + extracting it by calling pcre2_substring_length_bynumber(). The first + argument is a pointer to the match data block, the second is the group + number, and the third is a pointer to a variable into which the length is placed. - The pcre2_substring_copy_bynumber() function copies one string into a - supplied buffer, whereas pcre2_substring_get_bynumber() copies it into + The pcre2_substring_copy_bynumber() function copies one string into a + supplied buffer, whereas pcre2_substring_get_bynumber() copies it into new memory, obtained using the same memory allocation function that was - used for the match data block. The first two arguments of these func- - tions are a pointer to the match data block and a capturing group num- - ber. A group number of zero extracts the substring that matched the + used for the match data block. The first two arguments of these func- + tions are a pointer to the match data block and a capturing group num- + ber. A group number of zero extracts the substring that matched the entire pattern, and higher values extract the captured substrings. The final arguments of pcre2_substring_copy_bynumber() are a pointer to the buffer and a pointer to a variable that contains its length in code - units. This is updated to contain the actual number of code units + units. This is updated to contain the actual number of code units used, excluding the terminating zero. For pcre2_substring_get_bynumber() the third and fourth arguments point - to variables that are updated with a pointer to the new memory and the - number of code units that comprise the substring, again excluding the - terminating zero. When the substring is no longer needed, the memory + to variables that are updated with a pointer to the new memory and the + number of code units that comprise the substring, again excluding the + terminating zero. When the substring is no longer needed, the memory should be freed by calling pcre2_substring_free(). - The return value from these functions is zero for success, or one of + The return value from these functions is zero for success, or one of these error codes: PCRE2_ERROR_NOMEMORY - The buffer was too small for pcre2_substring_copy_bynumber(), or the + The buffer was too small for pcre2_substring_copy_bynumber(), or the attempt to get memory failed for pcre2_substring_get_bynumber(). PCRE2_ERROR_NOSUBSTRING - No substring with the given number was captured. This could be because - there is no capturing group of that number in the pattern, or because + No substring with the given number was captured. This could be because + there is no capturing group of that number in the pattern, or because the group with that number did not participate in the match, or because the ovector was too small to capture that group. @@ -2127,28 +2056,28 @@ EXTRACTING A LIST OF ALL CAPTURED SUBSTRINGS void pcre2_substring_list_free(PCRE2_SPTR *list); - The pcre2_substring_list_get() function extracts all available sub- - strings and builds a list of pointers to them, and a second list that - contains their lengths (in code units), excluding a terminating zero - that is added to each of them. All this is done in a single block of - memory that is obtained using the same memory allocation function that + The pcre2_substring_list_get() function extracts all available sub- + strings and builds a list of pointers to them, and a second list that + contains their lengths (in code units), excluding a terminating zero + that is added to each of them. All this is done in a single block of + memory that is obtained using the same memory allocation function that was used to get the match data block. - The address of the memory block is returned via listptr, which is also + The address of the memory block is returned via listptr, which is also the start of the list of string pointers. The end of the list is marked - by a NULL pointer. The address of the list of lengths is returned via - lengthsptr. If your strings do not contain binary zeros and you do not + by a NULL pointer. The address of the list of lengths is returned via + lengthsptr. If your strings do not contain binary zeros and you do not therefore need the lengths, you may supply NULL as the lengthsptr argu- - ment to disable the creation of a list of lengths. The yield of the - function is zero if all went well, or PCRE2_ERROR_NOMEMORY if the mem- - ory block could not be obtained. When the list is no longer needed, it + ment to disable the creation of a list of lengths. The yield of the + function is zero if all went well, or PCRE2_ERROR_NOMEMORY if the mem- + ory block could not be obtained. When the list is no longer needed, it should be freed by calling pcre2_substring_list_free(). If this function encounters a substring that is unset, which can happen - when capturing subpattern number n+1 matches some part of the subject, - but subpattern n has not been used at all, it returns an empty string. - This can be distinguished from a genuine zero-length substring by - inspecting the appropriate offset in the ovector, which contains + when capturing subpattern number n+1 matches some part of the subject, + but subpattern n has not been used at all, it returns an empty string. + This can be distinguished from a genuine zero-length substring by + inspecting the appropriate offset in the ovector, which contains PCRE2_UNSET for unset substrings. @@ -2168,32 +2097,32 @@ EXTRACTING CAPTURED SUBSTRINGS BY NAME void pcre2_substring_free(PCRE2_UCHAR *buffer); - To extract a substring by name, you first have to find associated num- + To extract a substring by name, you first have to find associated num- ber. For example, for this pattern: (a+)b(?\d+)... the number of the subpattern called "xxx" is 2. If the name is known to - be unique (PCRE2_DUPNAMES was not set), you can find the number from + be unique (PCRE2_DUPNAMES was not set), you can find the number from the name by calling pcre2_substring_number_from_name(). The first argu- - ment is the compiled pattern, and the second is the name. The yield of - the function is the subpattern number, or PCRE2_ERROR_NOSUBSTRING if + ment is the compiled pattern, and the second is the name. The yield of + the function is the subpattern number, or PCRE2_ERROR_NOSUBSTRING if there is no subpattern of that name. Given the number, you can extract the substring directly, or use one of the functions described in the previous section. For convenience, there - are also "byname" functions that correspond to the "bynumber" func- - tions, the only difference being that the second argument is a name - instead of a number. However, if PCRE2_DUPNAMES is set and there are - duplicate names, the behaviour may not be what you want (see the next + are also "byname" functions that correspond to the "bynumber" func- + tions, the only difference being that the second argument is a name + instead of a number. However, if PCRE2_DUPNAMES is set and there are + duplicate names, the behaviour may not be what you want (see the next section). Warning: If the pattern uses the (?| feature to set up multiple subpat- - terns with the same number, as described in the section on duplicate - subpattern numbers in the pcre2pattern page, you cannot use names to - distinguish the different subpatterns, because names are not included - in the compiled code. The matching process uses only numbers. For this - reason, the use of different names for subpatterns of the same number + terns with the same number, as described in the section on duplicate + subpattern numbers in the pcre2pattern page, you cannot use names to + distinguish the different subpatterns, because names are not included + in the compiled code. The matching process uses only numbers. For this + reason, the use of different names for subpatterns of the same number causes an error at compile time. @@ -2202,54 +2131,54 @@ DUPLICATE SUBPATTERN NAMES int pcre2_substring_nametable_scan(const pcre2_code *code, PCRE2_SPTR name, PCRE2_SPTR *first, PCRE2_SPTR *last); - When a pattern is compiled with the PCRE2_DUPNAMES option, names for - subpatterns are not required to be unique. Duplicate names are always - allowed for subpatterns with the same number, created by using the (?| - feature. Indeed, if such subpatterns are named, they are required to + When a pattern is compiled with the PCRE2_DUPNAMES option, names for + subpatterns are not required to be unique. Duplicate names are always + allowed for subpatterns with the same number, created by using the (?| + feature. Indeed, if such subpatterns are named, they are required to use the same names. Normally, patterns with duplicate names are such that in any one match, - only one of the named subpatterns participates. An example is shown in + only one of the named subpatterns participates. An example is shown in the pcre2pattern documentation. - When duplicates are present, pcre2_substring_copy_byname() and - pcre2_substring_get_byname() return the first substring corresponding + When duplicates are present, pcre2_substring_copy_byname() and + pcre2_substring_get_byname() return the first substring corresponding to the given name that is set. If none are set, PCRE2_ERROR_NOSUBSTRING - is returned. The pcre2_substring_number_from_name() function returns - one of the numbers that are associated with the name, but it is not + is returned. The pcre2_substring_number_from_name() function returns + one of the numbers that are associated with the name, but it is not defined which it is. - If you want to get full details of all captured substrings for a given - name, you must use the pcre2_substring_nametable_scan() function. The - first argument is the compiled pattern, and the second is the name. If - the third and fourth arguments are NULL, the function returns a group + If you want to get full details of all captured substrings for a given + name, you must use the pcre2_substring_nametable_scan() function. The + first argument is the compiled pattern, and the second is the name. If + the third and fourth arguments are NULL, the function returns a group number (it is not defined which). Otherwise, the third and fourth argu- - ments must be pointers to variables that are updated by the function. + ments must be pointers to variables that are updated by the function. After it has run, they point to the first and last entries in the name- to-number table for the given name, and the function returns the length - of each entry. In both cases, PCRE2_ERROR_NOSUBSTRING is returned if + of each entry. In both cases, PCRE2_ERROR_NOSUBSTRING is returned if there are no entries for the given name. The format of the name table is described above in the section entitled - Information about a pattern above. Given all the relevant entries for + Information about a pattern above. Given all the relevant entries for the name, you can extract each of their numbers, and hence the captured data. FINDING ALL POSSIBLE MATCHES - The traditional matching function uses a similar algorithm to Perl, + The traditional matching function uses a similar algorithm to Perl, which stops when it finds the first match, starting at a given point in - the subject. If you want to find all possible matches, or the longest - possible match at a given position, consider using the alternative - matching function (see below) instead. If you cannot use the alterna- + the subject. If you want to find all possible matches, or the longest + possible match at a given position, consider using the alternative + matching function (see below) instead. If you cannot use the alterna- tive function, you can kludge it up by making use of the callout facil- ity, which is described in the pcre2callout documentation. What you have to do is to insert a callout right at the end of the pat- - tern. When your callout function is called, extract and save the cur- - rent matched substring. Then return 1, which forces pcre2_match() to - backtrack and try other alternatives. Ultimately, when it runs out of + tern. When your callout function is called, extract and save the cur- + rent matched substring. Then return 1, which forces pcre2_match() to + backtrack and try other alternatives. Ultimately, when it runs out of matches, pcre2_match() will yield PCRE2_ERROR_NOMATCH. @@ -2261,26 +2190,26 @@ MATCHING A PATTERN: THE ALTERNATIVE FUNCTION pcre2_match_context *mcontext, int *workspace, PCRE2_SIZE wscount); - The function pcre2_dfa_match() is called to match a subject string - against a compiled pattern, using a matching algorithm that scans the - subject string just once, and does not backtrack. This has different - characteristics to the normal algorithm, and is not compatible with - Perl. Some of the features of PCRE2 patterns are not supported. Never- - theless, there are times when this kind of matching can be useful. For - a discussion of the two matching algorithms, and a list of features + The function pcre2_dfa_match() is called to match a subject string + against a compiled pattern, using a matching algorithm that scans the + subject string just once, and does not backtrack. This has different + characteristics to the normal algorithm, and is not compatible with + Perl. Some of the features of PCRE2 patterns are not supported. Never- + theless, there are times when this kind of matching can be useful. For + a discussion of the two matching algorithms, and a list of features that pcre2_dfa_match() does not support, see the pcre2matching documen- tation. - The arguments for the pcre2_dfa_match() function are the same as for + The arguments for the pcre2_dfa_match() function are the same as for pcre2_match(), plus two extras. The ovector within the match data block is used in a different way, and this is described below. The other com- - mon arguments are used in the same way as for pcre2_match(), so their + mon arguments are used in the same way as for pcre2_match(), so their description is not repeated here. - The two additional arguments provide workspace for the function. The - workspace vector should contain at least 20 elements. It is used for + The two additional arguments provide workspace for the function. The + workspace vector should contain at least 20 elements. It is used for keeping track of multiple paths through the pattern tree. More - workspace is needed for patterns and subjects where there are a lot of + workspace is needed for patterns and subjects where there are a lot of potential matches. Here is an example of a simple call to pcre2_dfa_match(): @@ -2300,45 +2229,45 @@ MATCHING A PATTERN: THE ALTERNATIVE FUNCTION Option bits for pcre_dfa_match() - The unused bits of the options argument for pcre2_dfa_match() must be - zero. The only bits that may be set are PCRE2_ANCHORED, PCRE2_NOTBOL, + The unused bits of the options argument for pcre2_dfa_match() must be + zero. The only bits that may be set are PCRE2_ANCHORED, PCRE2_NOTBOL, PCRE2_NOTEOL, PCRE2_NOTEMPTY, PCRE2_NOTEMPTY_ATSTART, - PCRE2_NO_UTF_CHECK, PCRE2_NO_START_OPTIMIZE, PCRE2_PARTIAL_HARD, - PCRE2_PARTIAL_SOFT, PCRE2_DFA_SHORTEST, and PCRE2_DFA_RESTART. All but - the last four of these are exactly the same as for pcre2_match(), so - their description is not repeated here. + PCRE2_NO_UTF_CHECK, PCRE2_PARTIAL_HARD, PCRE2_PARTIAL_SOFT, + PCRE2_DFA_SHORTEST, and PCRE2_DFA_RESTART. All but the last four of + these are exactly the same as for pcre2_match(), so their description + is not repeated here. PCRE2_PARTIAL_HARD PCRE2_PARTIAL_SOFT - These have the same general effect as they do for pcre2_match(), but - the details are slightly different. When PCRE2_PARTIAL_HARD is set for - pcre2_dfa_match(), it returns PCRE2_ERROR_PARTIAL if the end of the + These have the same general effect as they do for pcre2_match(), but + the details are slightly different. When PCRE2_PARTIAL_HARD is set for + pcre2_dfa_match(), it returns PCRE2_ERROR_PARTIAL if the end of the subject is reached and there is still at least one matching possibility that requires additional characters. This happens even if some complete - matches have already been found. When PCRE2_PARTIAL_SOFT is set, the - return code PCRE2_ERROR_NOMATCH is converted into PCRE2_ERROR_PARTIAL - if the end of the subject is reached, there have been no complete + matches have already been found. When PCRE2_PARTIAL_SOFT is set, the + return code PCRE2_ERROR_NOMATCH is converted into PCRE2_ERROR_PARTIAL + if the end of the subject is reached, there have been no complete matches, but there is still at least one matching possibility. The por- - tion of the string that was inspected when the longest partial match + tion of the string that was inspected when the longest partial match was found is set as the first matching string in both cases. There is a - more detailed discussion of partial and multi-segment matching, with + more detailed discussion of partial and multi-segment matching, with examples, in the pcre2partial documentation. PCRE2_DFA_SHORTEST - Setting the PCRE2_DFA_SHORTEST option causes the matching algorithm to + Setting the PCRE2_DFA_SHORTEST option causes the matching algorithm to stop as soon as it has found one match. Because of the way the alterna- - tive algorithm works, this is necessarily the shortest possible match + tive algorithm works, this is necessarily the shortest possible match at the first possible matching point in the subject string. PCRE2_DFA_RESTART - When pcre2_dfa_match() returns a partial match, it is possible to call + When pcre2_dfa_match() returns a partial match, it is possible to call it again, with additional subject characters, and have it continue with the same match. The PCRE2_DFA_RESTART option requests this action; when - it is set, the workspace and wscount options must reference the same - vector as before because data about the match so far is left in them + it is set, the workspace and wscount options must reference the same + vector as before because data about the match so far is left in them after a partial match. There is more discussion of this facility in the pcre2partial documentation. @@ -2346,8 +2275,8 @@ MATCHING A PATTERN: THE ALTERNATIVE FUNCTION When pcre2_dfa_match() succeeds, it may have matched more than one sub- string in the subject. Note, however, that all the matches from one run - of the function start at the same point in the subject. The shorter - matches are all initial substrings of the longer matches. For example, + of the function start at the same point in the subject. The shorter + matches are all initial substrings of the longer matches. For example, if the pattern <.*> @@ -2362,66 +2291,66 @@ MATCHING A PATTERN: THE ALTERNATIVE FUNCTION - On success, the yield of the function is a number greater than zero, - which is the number of matched substrings. The offsets of the sub- - strings are returned in the ovector, and can be extracted in the same - way as for pcre2_match(). They are returned in reverse order of - length; that is, the longest matching string is given first. If there - were too many matches to fit into the ovector, the yield of the func- + On success, the yield of the function is a number greater than zero, + which is the number of matched substrings. The offsets of the sub- + strings are returned in the ovector, and can be extracted in the same + way as for pcre2_match(). They are returned in reverse order of + length; that is, the longest matching string is given first. If there + were too many matches to fit into the ovector, the yield of the func- tion is zero, and the vector is filled with the longest matches. - NOTE: PCRE2's "auto-possessification" optimization usually applies to - character repeats at the end of a pattern (as well as internally). For - example, the pattern "a\d+" is compiled as if it were "a\d++" because - there is no point in backtracking into the repeated digits. For DFA - matching, this means that only one possible match is found. If you - really do want multiple matches in such cases, either use an ungreedy - repeat ("a\d+?") or set the PCRE2_NO_AUTO_POSSESS option when compil- + NOTE: PCRE2's "auto-possessification" optimization usually applies to + character repeats at the end of a pattern (as well as internally). For + example, the pattern "a\d+" is compiled as if it were "a\d++" because + there is no point in backtracking into the repeated digits. For DFA + matching, this means that only one possible match is found. If you + really do want multiple matches in such cases, either use an ungreedy + repeat ("a\d+?") or set the PCRE2_NO_AUTO_POSSESS option when compil- ing. Error returns from pcre2_dfa_match() The pcre2_dfa_match() function returns a negative number when it fails. - Many of the errors are the same as for pcre2_match(), as described + Many of the errors are the same as for pcre2_match(), as described above. There are in addition the following errors that are specific to pcre2_dfa_match(): PCRE2_ERROR_DFA_UITEM - This return is given if pcre2_dfa_match() encounters an item in the + This return is given if pcre2_dfa_match() encounters an item in the pattern that it does not support, for instance, the use of \C or a back reference. PCRE2_ERROR_DFA_UCOND - This return is given if pcre2_dfa_match() encounters a condition item - that uses a back reference for the condition, or a test for recursion + This return is given if pcre2_dfa_match() encounters a condition item + that uses a back reference for the condition, or a test for recursion in a specific group. These are not supported. PCRE2_ERROR_DFA_WSSIZE - This return is given if pcre2_dfa_match() runs out of space in the + This return is given if pcre2_dfa_match() runs out of space in the workspace vector. PCRE2_ERROR_DFA_RECURSE - When a recursive subpattern is processed, the matching function calls + When a recursive subpattern is processed, the matching function calls itself recursively, using private memory for the ovector and workspace. - This error is given if the internal ovector is not large enough. This + This error is given if the internal ovector is not large enough. This should be extremely rare, as a vector of size 1000 is used. PCRE2_ERROR_DFA_BADRESTART - When pcre2_dfa_match() is called with the pcre2_dfa_RESTART option, - some plausibility checks are made on the contents of the workspace, - which should contain data about the previous partial match. If any of + When pcre2_dfa_match() is called with the pcre2_dfa_RESTART option, + some plausibility checks are made on the contents of the workspace, + which should contain data about the previous partial match. If any of these checks fail, this error is given. SEE ALSO - pcre2build(3), pcre2libs(3), pcre2callout(3), pcre2matching(3), - pcre2partial(3), pcre2posix(3), pcre2demo(3), pcre2sample(3), + pcre2build(3), pcre2libs(3), pcre2callout(3), pcre2matching(3), + pcre2partial(3), pcre2posix(3), pcre2demo(3), pcre2sample(3), pcre2stack(3). @@ -2434,7 +2363,7 @@ AUTHOR REVISION - Last updated: 16 September 2014 + Last updated: 14 October 2014 Copyright (c) 1997-2014 University of Cambridge. ------------------------------------------------------------------------------ @@ -2551,7 +2480,7 @@ MISSING CALLOUTS has been scanned far enough. You can disable these optimizations by passing the PCRE2_NO_START_OPTI- - MIZE option to the matching function, or by starting the pattern with + MIZE option to pcre2_compile(), or by starting the pattern with (*NO_START_OPT). This slows down the matching process, but does ensure that callouts such as the example above are obeyed. diff --git a/doc/pcre2api.3 b/doc/pcre2api.3 index 988b169..f492475 100644 --- a/doc/pcre2api.3 +++ b/doc/pcre2api.3 @@ -1,4 +1,4 @@ -.TH PCRE2API 3 "10 October 2014" "PCRE2 10.00" +.TH PCRE2API 3 "14 October 2014" "PCRE2 10.00" .SH NAME PCRE2 - Perl-compatible regular expressions (revised API) .sp @@ -2061,15 +2061,10 @@ pointer to the zero-terminated name, which is within the compiled pattern. Otherwise NULL is returned. A (*MARK) name may be available after a failed match or a partial match, as well as after a successful one. .P -The offset of the character at which the successful or partial match started is +The offset of the character at which the successful match started is returned by \fBpcre2_get_startchar()\fP. This can be different to the value of -\fIovector[0]\fP if the pattern contains the \eK escape sequence. This -information is needed when doing partial matching over multiple data segments -(see the -.\" HREF -\fBpcre2partial\fP -.\" -documentation). +\fIovector[0]\fP if the pattern contains the \eK escape sequence. Note, +however, the \eK has no effect for a partial match. . . .\" HTML @@ -2626,6 +2621,6 @@ Cambridge CB2 3QH, England. .rs .sp .nf -Last updated: 10 October 2014 +Last updated: 14 October 2014 Copyright (c) 1997-2014 University of Cambridge. .fi diff --git a/doc/pcre2partial.3 b/doc/pcre2partial.3 new file mode 100644 index 0000000..faad43c --- /dev/null +++ b/doc/pcre2partial.3 @@ -0,0 +1,433 @@ +.TH PCRE2PARTIAL 3 "14 October 2014" "PCRE2 10.00" +.SH NAME +PCRE2 - Perl-compatible regular expressions +.SH "PARTIAL MATCHING IN PCRE2" +.rs +.sp +In normal use of PCRE2, if the subject string that is passed to a matching +function matches as far as it goes, but is too short to match the entire +pattern, PCRE2_ERROR_NOMATCH is returned. There are circumstances where it +might be helpful to distinguish this case from other cases in which there is no +match. +.P +Consider, for example, an application where a human is required to type in data +for a field with specific formatting requirements. An example might be a date +in the form \fIddmmmyy\fP, defined by this pattern: +.sp + ^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$ +.sp +If the application sees the user's keystrokes one by one, and can check that +what has been typed so far is potentially valid, it is able to raise an error +as soon as a mistake is made, by beeping and not reflecting the character that +has been typed, for example. This immediate feedback is likely to be a better +user interface than a check that is delayed until the entire string has been +entered. Partial matching can also be useful when the subject string is very +long and is not all available at once. +.P +PCRE2 supports partial matching by means of the PCRE2_PARTIAL_SOFT and +PCRE2_PARTIAL_HARD options, which can be set when calling a matching function. +The difference between the two options is whether or not a partial match is +preferred to an alternative complete match, though the details differ between +the two types of matching function. If both options are set, PCRE2_PARTIAL_HARD +takes precedence. +.P +If you want to use partial matching with just-in-time optimized code, you must +call \fBpcre2_jit_compile()\fP with one or both of these options: +.sp + PCRE2_JIT_PARTIAL_SOFT + PCRE2_JIT_PARTIAL_HARD +.sp +PCRE2_JIT_COMPLETE should also be set if you are going to run non-partial +matches on the same pattern. If the appropriate JIT mode has not been compiled, +interpretive matching code is used. +.P +Setting a partial matching option disables two of PCRE2's standard +optimizations. PCRE2 remembers the last literal code unit in a pattern, and +abandons matching immediately if it is not present in the subject string. This +optimization cannot be used for a subject string that might match only +partially. PCRE2 also knows the minimum length of a matching string, and does +not bother to run the matching function on shorter strings. This optimization +is also disabled for partial matching. +. +. +.SH "PARTIAL MATCHING USING pcre2_match()" +.rs +.sp +A partial match occurs during a call to \fBpcre2_match()\fP when the end of the +subject string is reached successfully, but matching cannot continue because +more characters are needed. However, at least one character in the subject must +have been inspected. This character need not form part of the final matched +string; lookbehind assertions and the \eK escape sequence provide ways of +inspecting characters before the start of a matched string. The requirement for +inspecting at least one character exists because an empty string can always be +matched; without such a restriction there would always be a partial match of an +empty string at the end of the subject. +.P +When a partial match is returned, the first two elements in the ovector point +to the portion of the subject that was matched. The appearance of \eK in the +pattern has no effect for a partial match. Consider this pattern: +.sp + /abc\eK123/ +.sp +If it is matched against "456abc123xyz" the result is a complete match, and the +ovector defines the matched string as "123", because \eK resets the "start of +match" point. However, if a partial match is requested and the subject string +is "456abc12", a partial match is found for the string "abc12", because all +these characters are needed for a subsequent re-match with additional +characters. +.P +What happens when a partial match is identified depends on which of the two +partial matching options are set. +. +. +.SS "PCRE2_PARTIAL_SOFT WITH pcre2_match()" +.rs +.sp +If PCRE2_PARTIAL_SOFT is set when \fBpcre2_match()\fP identifies a partial +match, the partial match is remembered, but matching continues as normal, and +other alternatives in the pattern are tried. If no complete match can be found, +PCRE2_ERROR_PARTIAL is returned instead of PCRE2_ERROR_NOMATCH. +.P +This option is "soft" because it prefers a complete match over a partial match. +All the various matching items in a pattern behave as if the subject string is +potentially complete. For example, \ez, \eZ, and $ match at the end of the +subject, as normal, and for \eb and \eB the end of the subject is treated as a +non-alphanumeric. +.P +If there is more than one partial match, the first one that was found provides +the data that is returned. Consider this pattern: +.sp + /123\ew+X|dogY/ +.sp +If this is matched against the subject string "abc123dog", both +alternatives fail to match, but the end of the subject is reached during +matching, so PCRE2_ERROR_PARTIAL is returned. The offsets are set to 3 and 9, +identifying "123dog" as the first partial match that was found. (In this +example, there are two partial matches, because "dog" on its own partially +matches the second alternative.) +. +. +.SS "PCRE2_PARTIAL_HARD WITH pcre2_match()" +.rs +.sp +If PCRE2_PARTIAL_HARD is set for \fBpcre2_match()\fP, PCRE2_ERROR_PARTIAL is +returned as soon as a partial match is found, without continuing to search for +possible complete matches. This option is "hard" because it prefers an earlier +partial match over a later complete match. For this reason, the assumption is +made that the end of the supplied subject string may not be the true end of the +available data, and so, if \ez, \eZ, \eb, \eB, or $ are encountered at the end +of the subject, the result is PCRE2_ERROR_PARTIAL, provided that at least one +character in the subject has been inspected. +. +. +.SS "Comparing hard and soft partial matching" +.rs +.sp +The difference between the two partial matching options can be illustrated by a +pattern such as: +.sp + /dog(sbody)?/ +.sp +This matches either "dog" or "dogsbody", greedily (that is, it prefers the +longer string if possible). If it is matched against the string "dog" with +PCRE2_PARTIAL_SOFT, it yields a complete match for "dog". However, if +PCRE2_PARTIAL_HARD is set, the result is PCRE2_ERROR_PARTIAL. On the other +hand, if the pattern is made ungreedy the result is different: +.sp + /dog(sbody)??/ +.sp +In this case the result is always a complete match because that is found first, +and matching never continues after finding a complete match. It might be easier +to follow this explanation by thinking of the two patterns like this: +.sp + /dog(sbody)?/ is the same as /dogsbody|dog/ + /dog(sbody)??/ is the same as /dog|dogsbody/ +.sp +The second pattern will never match "dogsbody", because it will always find the +shorter match first. +. +. +.SH "PARTIAL MATCHING USING pcre2_dfa_match()" +.rs +.sp +The DFA functions move along the subject string character by character, without +backtracking, searching for all possible matches simultaneously. If the end of +the subject is reached before the end of the pattern, there is the possibility +of a partial match, again provided that at least one character has been +inspected. +.P +When PCRE2_PARTIAL_SOFT is set, PCRE2_ERROR_PARTIAL is returned only if there +have been no complete matches. Otherwise, the complete matches are returned. +However, if PCRE2_PARTIAL_HARD is set, a partial match takes precedence over +any complete matches. The portion of the string that was matched when the +longest partial match was found is set as the first matching string. +.P +Because the DFA functions always search for all possible matches, and there is +no difference between greedy and ungreedy repetition, their behaviour is +different from the standard functions when PCRE2_PARTIAL_HARD is set. Consider +the string "dog" matched against the ungreedy pattern shown above: +.sp + /dog(sbody)??/ +.sp +Whereas the standard functions stop as soon as they find the complete match for +"dog", the DFA functions also find the partial match for "dogsbody", and so +return that when PCRE2_PARTIAL_HARD is set. +. +. +.SH "PARTIAL MATCHING AND WORD BOUNDARIES" +.rs +.sp +If a pattern ends with one of sequences \eb or \eB, which test for word +boundaries, partial matching with PCRE2_PARTIAL_SOFT can give counter-intuitive +results. Consider this pattern: +.sp + /\ebcat\eb/ +.sp +This matches "cat", provided there is a word boundary at either end. If the +subject string is "the cat", the comparison of the final "t" with a following +character cannot take place, so a partial match is found. However, normal +matching carries on, and \eb matches at the end of the subject when the last +character is a letter, so a complete match is found. The result, therefore, is +\fInot\fP PCRE2_ERROR_PARTIAL. Using PCRE2_PARTIAL_HARD in this case does yield +PCRE2_ERROR_PARTIAL, because then the partial match takes precedence. +. +. +.SH "EXAMPLE OF PARTIAL MATCHING USING PCRE2TEST" +.rs +.sp +If the \fBpartial_soft\fP (or \fBps\fP) modifier is present on a +\fBpcre2test\fP data line, the PCRE2_PARTIAL_SOFT option is used for the match. +Here is a run of \fBpcre2test\fP that uses the date example quoted above: +.sp + re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/ + data> 25jun04\e=ps + 0: 25jun04 + 1: jun + data> 25dec3\e=ps + Partial match: 23dec3 + data> 3ju\e=ps + Partial match: 3ju + data> 3juj\e=ps + No match + data> j\e=ps + No match +.sp +The first data string is matched completely, so \fBpcre2test\fP shows the +matched substrings. The remaining four strings do not match the complete +pattern, but the first two are partial matches. Similar output is obtained +if DFA matching is used. +.P +If the \fBpartial_hard\fP (or \fBph\fP) modifier is present on a +\fBpcre2test\fP data line, the PCRE2_PARTIAL_HARD option is set for the match. +. +. +.SH "MULTI-SEGMENT MATCHING WITH pcre2_dfa_match()" +.rs +.sp +When a partial match has been found using a DFA matching function, it is +possible to continue the match by providing additional subject data and calling +the function again with the same compiled regular expression, this time setting +the PCRE2_DFA_RESTART option. You must pass the same working space as before, +because this is where details of the previous partial match are stored. Here is +an example using \fBpcre2test\fP: +.sp + re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/ + data> 23ja\e=dfa,ps + Partial match: 23ja + data> n05\e=dfa,dfa_restart + 0: n05 +.sp +The first call has "23ja" as the subject, and requests partial matching; the +second call has "n05" as the subject for the continued (restarted) match. +Notice that when the match is complete, only the last part is shown; PCRE2 does +not retain the previously partially-matched string. It is up to the calling +program to do that if it needs to. +.P +That means that, for an unanchored pattern, if a continued match fails, it is +not possible to try again at a new starting point. All this facility is capable +of doing is continuing with the previous match attempt. In the previous +example, if the second set of data is "ug23" the result is no match, even +though there would be a match for "aug23" if the entire string were given at +once. Depending on the application, this may or may not be what you want. +The only way to allow for starting again at the next character is to retain the +matched part of the subject and try a new complete match. +.P +You can set the PCRE2_PARTIAL_SOFT or PCRE2_PARTIAL_HARD options with +PCRE2_DFA_RESTART to continue partial matching over multiple segments. This +facility can be used to pass very long subject strings to the DFA matching +functions. +. +. +.SH "MULTI-SEGMENT MATCHING WITH pcre2_match()" +.rs +.sp +Unlike the DFA function, it is not possible to restart the previous match with +a new segment of data when using \fBpcre2_match()\fP. Instead, new data must be +added to the previous subject string, and the entire match re-run, starting +from the point where the partial match occurred. Earlier data can be discarded. +.P +It is best to use PCRE2_PARTIAL_HARD in this situation, because it does not +treat the end of a segment as the end of the subject when matching \ez, \eZ, +\eb, \eB, and $. Consider an unanchored pattern that matches dates: +.sp + re> /\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed/ + data> The date is 23ja\e=ph + Partial match: 23ja +.sp +At this stage, an application could discard the text preceding "23ja", add on +text from the next segment, and call the matching function again. Unlike the +DFA matching function, the entire matching string must always be available, +and the complete matching process occurs for each call, so more memory and more +processing time is needed. +. +. +.SH "ISSUES WITH MULTI-SEGMENT MATCHING" +.rs +.sp +Certain types of pattern may give problems with multi-segment matching, +whichever matching function is used. +.P +1. If the pattern contains a test for the beginning of a line, you need to pass +the PCRE2_NOTBOL option when the subject string for any call does start at the +beginning of a line. There is also a PCRE2_NOTEOL option, but in practice when +doing multi-segment matching you should be using PCRE2_PARTIAL_HARD, which +includes the effect of PCRE2_NOTEOL. +.P +2. If a pattern contains a lookbehind assertion, characters that precede the +start of the partial match may have been inspected during the matching process. +When using \fBpcre2_match()\fP, sufficient characters must be retained for the +next match attempt. You can ensure that enough characters are retained by doing +the following: +.P +Before doing any matching, find the length of the longest lookbehind in the +pattern by calling \fBpcre2_pattern_info()\fP with the PCRE2_INFO_MAXLOOKBEHIND +option. Note that the resulting count is in characters, not code units. After a +partial match, moving back from the ovector[0] offset in the subject by the +number of characters given for the maximum lookbehind gets you to the earliest +character that must be retained. In a non-UTF or a 32-bit situation, moving +back is just a subtraction, but in UTF-8 or UTF-16 you have to count characters +while moving back through the code units. +.P +Characters before the point you have now reached can be discarded, and after +the next segment has been added to what is retained, you should run the next +match with the \fBstartoffset\fP argument set so that the match begins at the +same point as before. +.P +For example, if the pattern "(?<=123)abc" is partially matched against the +string "xx123ab", the ovector offsets are 5 and 7 ("ab"). The maximum +lookbehind count is 3, so all characters before offset 2 can be discarded. The +value of \fBstartoffset\fP for the next match should be 3. When \fBpcre2test\fP +displays a partial match, it indicates the lookbehind characters with '<' +characters: +.sp + re> "(?<=123)abc" + data> xx123ab\e=ph + Partial match: 123ab + <<< +.P +3. Because a partial match must always contain at least one character, what +might be considered a partial match of an empty string actually gives a "no +match" result. For example: +.sp + re> /c(?<=abc)x/ + data> ab\e=ps + No match +.sp +If the next segment begins "cx", a match should be found, but this will only +happen if characters from the previous segment are retained. For this reason, a +"no match" result should be interpreted as "partial match of an empty string" +when the pattern contains lookbehinds. +.P +4. Matching a subject string that is split into multiple segments may not +always produce exactly the same result as matching over one single long string, +especially when PCRE2_PARTIAL_SOFT is used. The section "Partial Matching and +Word Boundaries" above describes an issue that arises if the pattern ends with +\eb or \eB. Another kind of difference may occur when there are multiple +matching possibilities, because (for PCRE2_PARTIAL_SOFT) a partial match result +is given only when there are no completed matches. This means that as soon as +the shortest match has been found, continuation to a new subject segment is no +longer possible. Consider this \fBpcre2test\fP example: +.sp + re> /dog(sbody)?/ + data> dogsb\e=ps + 0: dog + data> do\e=ps,dfa + Partial match: do + data> gsb\e=ps,dfa,dfa_restart + 0: g + data> dogsbody\e=dfa + 0: dogsbody + 1: dog +.sp +The first data line passes the string "dogsb" to a standard matching function, +setting the PCRE2_PARTIAL_SOFT option. Although the string is a partial match +for "dogsbody", the result is not PCRE2_ERROR_PARTIAL, because the shorter +string "dog" is a complete match. Similarly, when the subject is presented to +a DFA matching function in several parts ("do" and "gsb" being the first two) +the match stops when "dog" has been found, and it is not possible to continue. +On the other hand, if "dogsbody" is presented as a single string, a DFA +matching function finds both matches. +.P +Because of these problems, it is best to use PCRE2_PARTIAL_HARD when matching +multi-segment data. The example above then behaves differently: +.sp + re> /dog(sbody)?/ + data> dogsb\e=ph + Partial match: dogsb + data> do\e=ps,dfa + Partial match: do + data> gsb\e=ph,dfa,dfa_restart + Partial match: gsb +.sp +5. Patterns that contain alternatives at the top level which do not all start +with the same pattern item may not work as expected when PCRE2_DFA_RESTART is +used. For example, consider this pattern: +.sp + 1234|3789 +.sp +If the first part of the subject is "ABC123", a partial match of the first +alternative is found at offset 3. There is no partial match for the second +alternative, because such a match does not start at the same point in the +subject string. Attempting to continue with the string "7890" does not yield a +match because only those alternatives that match at one point in the subject +are remembered. The problem arises because the start of the second alternative +matches within the first alternative. There is no problem with anchored +patterns or patterns such as: +.sp + 1234|ABCD +.sp +where no string can be a partial match for both alternatives. This is not a +problem if a standard matching function is used, because the entire match has +to be rerun each time: +.sp + re> /1234|3789/ + data> ABC123\e=ph + Partial match: 123 + data> 1237890 + 0: 3789 +.sp +Of course, instead of using PCRE2_DFA_RESTART, the same technique of re-running +the entire match can also be used with the DFA matching function. Another +possibility is to work with two buffers. If a partial match at offset \fIn\fP +in the first buffer is followed by "no match" when PCRE2_DFA_RESTART is used on +the second buffer, you can then try a new match starting at offset \fIn+1\fP in +the first buffer. +. +. +.SH AUTHOR +.rs +.sp +.nf +Philip Hazel +University Computing Service +Cambridge CB2 3QH, England. +.fi +. +. +.SH REVISION +.rs +.sp +.nf +Last updated: 14 October 2014 +Copyright (c) 1997-2014 University of Cambridge. +.fi diff --git a/doc/pcre2test.txt b/doc/pcre2test.txt index 34c1a14..3312257 100644 --- a/doc/pcre2test.txt +++ b/doc/pcre2test.txt @@ -424,6 +424,7 @@ PATTERN MODIFIERS /I info show info about compiled pattern hex pattern is coded in hexadecimal jit[= ] use JIT + jitverify verify JIT use locale= use this locale memory show memory used newline= set newline type @@ -448,68 +449,69 @@ PATTERN MODIFIERS as newlines, both in the pattern and (by default) in subject lines. The type must be one of CR, LF, CRLF, ANYCRLF, or ANY. - Both the \R and newline settings can be changed at match time, but if - this is done, JIT matching is disabled. - Information about a pattern - The debug modifier is a shorthand for info,fullbincode, requesting all + The debug modifier is a shorthand for info,fullbincode, requesting all available information. The bincode modifier causes a representation of the compiled code to be - output after compilation. This information does not contain length and + output after compilation. This information does not contain length and offset values, which ensures that the same output is generated for dif- - ferent internal link sizes and different code unit widths. By using - bincode, the same regression tests can be used in different environ- + ferent internal link sizes and different code unit widths. By using + bincode, the same regression tests can be used in different environ- ments. - The fullbincode modifier, by contrast, does include length and offset + The fullbincode modifier, by contrast, does include length and offset values. This is used in a few special tests and is also useful for one- off tests. - The info modifier requests information about the compiled pattern - (whether it is anchored, has a fixed first character, and so on). The + The info modifier requests information about the compiled pattern + (whether it is anchored, has a fixed first character, and so on). The information is obtained from the pcre2_pattern_info() function. Specifying a pattern in hex The hex modifier specifies that the characters of the pattern are to be - interpreted as pairs of hexadecimal digits. White space is permitted + interpreted as pairs of hexadecimal digits. White space is permitted between pairs. For example: /ab 32 59/hex - This feature is provided as a way of creating patterns that contain + This feature is provided as a way of creating patterns that contain binary zero characters. When hex is set, it implies use_length. Using the pattern's length - By default, pcre2test passes patterns as zero-terminated strings to - pcre2_compile(), giving the length as -1. If use_length is set, the + By default, pcre2test passes patterns as zero-terminated strings to + pcre2_compile(), giving the length as -1. If use_length is set, the length of the pattern is passed. This is implied if hex is set. JIT compilation - The /jit modifier may optionally be followed by a number in the range 0 - to 7: + The /jit modifier may optionally be followed by and equals sign and a + number in the range 0 to 7: 0 disable JIT - 1 normal match only - 2 soft partial match only - 3 normal match and soft partial match - 4 hard partial match only - 6 soft and hard partial match + 1 use JIT for normal match only + 2 use JIT for soft partial match only + 3 use JIT for normal match and soft partial match + 4 use JIT for hard partial match only + 6 use JIT for soft and hard partial match 7 all three modes - If no number is given, 7 is assumed. If JIT compilation is successful, - the compiled JIT code will automatically be used when pcre2_match() is - run, except when incompatible run-time options are specified. For more - details, see the pcre2jit documentation. See also the jitstack modifier - below for a way of setting the size of the JIT stack. + If no number is given, 7 is assumed. If JIT compilation is successful, + the compiled JIT code will automatically be used when pcre2_match() is + run for the appropriate type of match, except when incompatible run- + time options are specified. For more details, see the pcre2jit documen- + tation. See also the jitstack modifier below for a way of setting the + size of the JIT stack. - If the jitverify modifier is specified, the text "(JIT)" is added to + If the jitverify modifier is specified, information about the compiled + pattern shows whether JIT compilation was or was not successful. If + jitverify is specified without jit, jit=7 is assumed. If JIT compila- + tion is successful when jitverify is set, the text "(JIT)" is added to the first output line after a match or non match when JIT-compiled code - was actually used. This modifier can also be set on a subject line. + was actually used. Setting a locale @@ -518,31 +520,31 @@ PATTERN MODIFIERS /pattern/locale=fr_FR The given locale is set, pcre2_maketables() is called to build a set of - character tables for the locale, and this is then passed to pcre2_com- - pile() when compiling the regular expression. The same tables are used + character tables for the locale, and this is then passed to pcre2_com- + pile() when compiling the regular expression. The same tables are used when matching the following subject lines. The /locale modifier applies only to the pattern on which it appears, but can be given in a #pattern - command if a default is needed. Setting a locale and alternate charac- + command if a default is needed. Setting a locale and alternate charac- ter tables are mutually exclusive. Showing pattern memory - The /memory modifier causes the size in bytes of the memory block used - to hold the compiled pattern to be output. This does not include the - size of the pcre2_code block; it is just the actual compiled data. If + The /memory modifier causes the size in bytes of the memory block used + to hold the compiled pattern to be output. This does not include the + size of the pcre2_code block; it is just the actual compiled data. If the pattern is subsequently passed to the JIT compiler, the size of the JIT compiled code is also output. Limiting nested parentheses - The parens_nest_limit modifier sets a limit on the depth of nested - parentheses in a pattern. Breaching the limit causes a compilation + The parens_nest_limit modifier sets a limit on the depth of nested + parentheses in a pattern. Breaching the limit causes a compilation error. Using the POSIX wrapper API - The /posix modifier causes pcre2test to call PCRE2 via the POSIX wrap- - per API rather than its native API. This supports only the 8-bit + The /posix modifier causes pcre2test to call PCRE2 via the POSIX wrap- + per API rather than its native API. This supports only the 8-bit library. When the POSIX API is being used, the following pattern modi- fiers set options for the regcomp() function: @@ -554,25 +556,25 @@ PATTERN MODIFIERS ucp REG_UCP ) the POSIX standard utf REG_UTF8 ) - The aftertext and allaftertext subject modifiers work as described + The aftertext and allaftertext subject modifiers work as described below. All other modifiers cause an error. Testing the stack guard feature - The /stackguard modifier is used to test the use of pcre2_set_com- - pile_recursion_guard(), a function that is provided to enable stack - availability to be checked during compilation (see the pcre2api docu- - mentation for details). If the number specified by the modifier is + The /stackguard modifier is used to test the use of pcre2_set_com- + pile_recursion_guard(), a function that is provided to enable stack + availability to be checked during compilation (see the pcre2api docu- + mentation for details). If the number specified by the modifier is greater than zero, pcre2_set_compile_recursion_guard() is called to set - up callback from pcre2_compile() to a local function. The argument it - is passed is the current nesting parenthesis depth; if this is greater + up callback from pcre2_compile() to a local function. The argument it + is passed is the current nesting parenthesis depth; if this is greater than the value given by the modifier, non-zero is returned, causing the compilation to be aborted. Using alternative character tables - The /tables modifier must be followed by a single digit. It causes a - specific set of built-in character tables to be passed to pcre2_com- + The /tables modifier must be followed by a single digit. It causes a + specific set of built-in character tables to be passed to pcre2_com- pile(). This is used in the PCRE2 tests to check behaviour with differ- ent character tables. The digit specifies the tables as follows: @@ -581,15 +583,15 @@ PATTERN MODIFIERS pcre2_chartables.c.dist 2 a set of tables defining ISO 8859 characters - In table 2, some characters whose codes are greater than 128 are iden- - tified as letters, digits, spaces, etc. Setting alternate character + In table 2, some characters whose codes are greater than 128 are iden- + tified as letters, digits, spaces, etc. Setting alternate character tables and a locale are mutually exclusive. Setting certain match controls The following modifiers are really subject modifiers, and are described - below. However, they may be included in a pattern's modifier list, in - which case they are applied to every subject line that is processed + below. However, they may be included in a pattern's modifier list, in + which case they are applied to every subject line that is processed with that pattern. They do not affect the compilation process. aftertext show text after match @@ -597,10 +599,9 @@ PATTERN MODIFIERS allcaptures show all captures allusedtext show all consulted text /g global global matching - jitverify verify JIT usage mark show mark values - These modifiers may not appear in a #pattern command. If you want them + These modifiers may not appear in a #pattern command. If you want them as defaults, set them in a #subject command. @@ -611,13 +612,12 @@ SUBJECT MODIFIERS Setting match options - The following modifiers set options for pcre2_match() or + The following modifiers set options for pcre2_match() or pcre2_dfa_match(). See pcreapi for a description of their effects. anchored set PCRE2_ANCHORED dfa_restart set PCRE2_DFA_RESTART dfa_shortest set PCRE2_DFA_SHORTEST - no_start_optimize set PCRE2_NO_START_OPTIMIZE no_utf_check set PCRE2_NO_UTF_CHECK notbol set PCRE2_NOTBOL notempty set PCRE2_NOTEMPTY @@ -626,28 +626,27 @@ SUBJECT MODIFIERS partial_hard (or ph) set PCRE2_PARTIAL_HARD partial_soft (or ps) set PCRE2_PARTIAL_SOFT - The partial matching modifiers are provided with abbreviations because + The partial matching modifiers are provided with abbreviations because they appear frequently in tests. - If the /posix modifier was present on the pattern, causing the POSIX + If the /posix modifier was present on the pattern, causing the POSIX wrapper API to be used, the only option-setting modifiers that have any - effect are notbol, notempty, and noteol, causing REG_NOTBOL, - REG_NOTEMPTY, and REG_NOTEOL, respectively, to be passed to regexec(). + effect are notbol, notempty, and noteol, causing REG_NOTBOL, + REG_NOTEMPTY, and REG_NOTEOL, respectively, to be passed to regexec(). Any other modifiers cause an error. Setting match controls - The following modifiers affect the matching process or request addi- - tional information. Some of them may also be specified on a pattern - line (see above), in which case they apply to every subject line that + The following modifiers affect the matching process or request addi- + tional information. Some of them may also be specified on a pattern + line (see above), in which case they apply to every subject line that is matched against that pattern. aftertext show text after match allaftertext show text after captures allcaptures show all captures - allusedtext show all consulted text + allusedtext show all consulted text (non-JIT only) altglobal alternative global matching - bsr=[anycrlf|unicode] specify \R handling callout_capture show captures at callout time callout_data= set a value to pass via callouts callout_fail= [: ] control callout failure @@ -659,11 +658,9 @@ SUBJECT MODIFIERS getall extract all captured substrings /g global global matching jitstack= set size of JIT stack - jitverify verify JIT usage mark show mark values match_limit=>n> set a match limit memory show memory usage - newline= set newline type offset= set starting offset ovector= set size of output vector recursion_limit= set a recursion limit @@ -671,13 +668,6 @@ SUBJECT MODIFIERS The effects of these modifiers are described in the following sections. FIXME: Give more examples. - Newline and \R handling - - These modifiers set the newline and \R processing conventions for the - subject line, overriding any values that were set at compile time (as - described above). JIT matching is disabled if these settings are - changed at match time. - Showing more text The aftertext modifier requests that as well as outputting the sub- @@ -690,18 +680,21 @@ SUBJECT MODIFIERS ture number. The allusedtext modifier requests that all the text that was consulted - during a successful pattern match be shown. This affects the output if - there is a lookbehind at the start of a match, or a lookahead at the - end, or if \K is used in the pattern. Characters that precede or follow - the start and end of the actual match are indicated in the output by - '<' or '>' characters underneath them. Here is an example: + during a successful pattern match by the interpreter should be shown. + This feature is not supported for JIT matching, and if requested with + JIT it is ignored (with a warning message). Setting this modifier + affects the output if there is a lookbehind at the start of a match, or + a lookahead at the end, or if \K is used in the pattern. Characters + that precede or follow the start and end of the actual match are indi- + cated in the output by '<' or '>' characters underneath them. Here is + an example: /(?<=pqr)abc(?=xyz)/ 123pqrabcxyz456\=allusedtext 0: pqrabcxyz <<< >>> - This shows that the matched string is "abc", with the preceding and + This shows that the matched string is "abc", with the preceding and following strings "pqr" and "xyz" also consulted during the match. Showing the value of all capture groups @@ -709,124 +702,133 @@ SUBJECT MODIFIERS The allcaptures modifier requests that the values of all potential cap- tured parentheses be output after a match. By default, only those up to the highest one actually used in the match are output (corresponding to - the return code from pcre2_match()). Groups that did not take part in + the return code from pcre2_match()). Groups that did not take part in the match are output as " ". Testing callouts - A callout function is supplied when pcre2test calls the library match- - ing functions, unless callout_none is specified. If callout_capture is + A callout function is supplied when pcre2test calls the library match- + ing functions, unless callout_none is specified. If callout_capture is set, the current captured groups are output when a callout occurs. - The callout_fail modifier can be given one or two numbers. If there is + The callout_fail modifier can be given one or two numbers. If there is only one number, 1 is returned instead of 0 when a callout of that num- - ber is reached. If two numbers are given, 1 is returned when callout + ber is reached. If two numbers are given, 1 is returned when callout is reached for the th time. - The callout_data modifier can be given an unsigned or a negative num- - ber. Any value other than zero is used as a return from pcre2test's + The callout_data modifier can be given an unsigned or a negative num- + ber. Any value other than zero is used as a return from pcre2test's callout function. Testing substring extraction functions - The copy and get modifiers can be used to test the pcre2_sub- + The copy and get modifiers can be used to test the pcre2_sub- string_copy_xxx() and pcre2_substring_get_xxx() functions. They can be - given more than once, and each can specify a group name or number, for + given more than once, and each can specify a group name or number, for example: abcd\=copy=1,copy=3,get=G1 - If the #subject command is used to set default copy and get lists, - these can be unset by specifying a negative number for numbered groups + If the #subject command is used to set default copy and get lists, + these can be unset by specifying a negative number for numbered groups and an empty name for named groups. - The getall modifier tests pcre2_substring_list_get(), which extracts + The getall modifier tests pcre2_substring_list_get(), which extracts all captured substrings. - If the subject line is successfully matched, the substrings extracted - by the convenience functions are output with C, G, or L after the - string number instead of a colon. This is in addition to the normal - full list. The string length (that is, the return from the extraction + If the subject line is successfully matched, the substrings extracted + by the convenience functions are output with C, G, or L after the + string number instead of a colon. This is in addition to the normal + full list. The string length (that is, the return from the extraction function) is given in parentheses after each substring. Finding all matches in a string Searching for all possible matches within a subject can be requested by - the global or /altglobal modifier. After finding a match, the matching - function is called again to search the remainder of the subject. The - difference between global and altglobal is that the former uses the - start_offset argument to pcre2_match() or pcre2_dfa_match() to start - searching at a new point within the entire string (which is what Perl + the global or /altglobal modifier. After finding a match, the matching + function is called again to search the remainder of the subject. The + difference between global and altglobal is that the former uses the + start_offset argument to pcre2_match() or pcre2_dfa_match() to start + searching at a new point within the entire string (which is what Perl does), whereas the latter passes over a shortened substring. This makes a difference to the matching process if the pattern begins with a look- behind assertion (including \b or \B). - If an empty string is matched, the next match is done with the + If an empty string is matched, the next match is done with the PCRE2_NOTEMPTY_ATSTART and PCRE2_ANCHORED flags set, in order to search for another, non-empty, match at the same point in the subject. If this - match fails, the start offset is advanced, and the normal match is - retried. This imitates the way Perl handles such cases when using the - /g modifier or the split() function. Normally, the start offset is - advanced by one character, but if the newline convention recognizes - CRLF as a newline, and the current character is CR followed by LF, an + match fails, the start offset is advanced, and the normal match is + retried. This imitates the way Perl handles such cases when using the + /g modifier or the split() function. Normally, the start offset is + advanced by one character, but if the newline convention recognizes + CRLF as a newline, and the current character is CR followed by LF, an advance of two is used. Setting the JIT stack size - The jitstack modifier provides a way of setting the maximum stack size - that is used by the just-in-time optimization code. It is ignored if - JIT optimization is not being used. Providing a stack that is larger + The jitstack modifier provides a way of setting the maximum stack size + that is used by the just-in-time optimization code. It is ignored if + JIT optimization is not being used. Providing a stack that is larger than the default 32K is necessary only for very complicated patterns. Setting match and recursion limits - The match_limit and recursion_limit modifiers set the appropriate lim- + The match_limit and recursion_limit modifiers set the appropriate lim- its in the match context. These values are ignored when the find_limits modifier is specified. Finding minimum limits - If the find_limits modifier is present, pcre2test calls pcre2_match() - several times, setting different values in the match context via - pcre2_set_match_limit() and pcre2_set_recursion_limit() until it finds - the minimum values for each parameter that allow pcre2_match() to com- + If the find_limits modifier is present, pcre2test calls pcre2_match() + several times, setting different values in the match context via + pcre2_set_match_limit() and pcre2_set_recursion_limit() until it finds + the minimum values for each parameter that allow pcre2_match() to com- plete without error. - The match_limit number is a measure of the amount of backtracking that - takes place, and learning the minimum value can be instructive. For - most simple matches, the number is quite small, but for patterns with - very large numbers of matching possibilities, it can become large very - quickly with increasing length of subject string. The - match_limit_recursion number is a measure of how much stack (or, if - PCRE2 is compiled with NO_RECURSE, how much heap) memory is needed to + If JIT is being used, only the match limit is relevant. If DFA matching + is being used, neither limit is relevant, and this modifier is ignored + (with a warning message). + + The match_limit number is a measure of the amount of backtracking that + takes place, and learning the minimum value can be instructive. For + most simple matches, the number is quite small, but for patterns with + very large numbers of matching possibilities, it can become large very + quickly with increasing length of subject string. The + match_limit_recursion number is a measure of how much stack (or, if + PCRE2 is compiled with NO_RECURSE, how much heap) memory is needed to complete the match attempt. Showing MARK names The mark modifier causes the names from backtracking control verbs that - are returned from calls to pcre2_match() to be displayed. If a mark is - returned for a match, non-match, or partial match, pcre2test shows it. - For a match, it is on a line by itself, tagged with "MK:". Otherwise, + are returned from calls to pcre2_match() to be displayed. If a mark is + returned for a match, non-match, or partial match, pcre2test shows it. + For a match, it is on a line by itself, tagged with "MK:". Otherwise, it is added to the non-match message. Showing memory usage - The memory modifier causes pcre2test to log all memory allocation and + The memory modifier causes pcre2test to log all memory allocation and freeing calls that occur during a match operation. Setting a starting offset - The offset modifier sets an offset in the subject string at which + The offset modifier sets an offset in the subject string at which matching starts. Its value is a number of code units, not characters. Setting the size of the output vector - The ovector modifier applies only to the subject line in which it - appears, though of course it can also be used to set a default in a - #subject command. It specifies the number of pairs of offsets that are + The ovector modifier applies only to the subject line in which it + appears, though of course it can also be used to set a default in a + #subject command. It specifies the number of pairs of offsets that are available for storing matching information. The default is 15. + At least one pair of offsets is always created by pcre2_match_data_cre- + ate(), for matching with PCRE2's native API, so a value of 0 is the + same as 1. However a value of 0 is useful when testing the POSIX API + because it causes regexec() to be called with a NULL capture vector. + THE ALTERNATIVE MATCHING FUNCTION @@ -1069,5 +1071,5 @@ AUTHOR REVISION - Last updated: 19 August 2014 + Last updated: 11 October 2014 Copyright (c) 1997-2014 University of Cambridge. diff --git a/src/pcre2test.c b/src/pcre2test.c index 97b6de3..e2b73bf 100644 --- a/src/pcre2test.c +++ b/src/pcre2test.c @@ -612,6 +612,7 @@ clock_t total_match_time = 0; static uint32_t dfa_matched; static uint32_t forbid_utf = 0; +static uint32_t maxlookbehind; static uint32_t max_oveccount; static uint32_t callout_count; @@ -2293,6 +2294,55 @@ return 0; +/************************************************* +* Move back by so many characters * +*************************************************/ + +/* Given a code unit offset in a subject string, move backwards by a number of +characters, and return the resulting offset. + +Arguments: + subject pointer to the string + offset start offset + count count to move back by + utf TRUE if in UTF mode + +Returns: a possibly changed offset +*/ + +static PCRE2_SIZE +backchars(uint8_t *subject, PCRE2_SIZE offset, uint32_t count, BOOL utf) +{ +long int yield; + +if (!utf || test_mode == PCRE32_MODE) yield = offset - count; + +else if (test_mode == PCRE8_MODE) + { + PCRE2_SPTR8 pp = (PCRE2_SPTR8)subject + offset; + for (; count > 0; count--) + { + pp--; + while ((*pp & 0xc0) == 0x80) pp--; + } + yield = pp - (PCRE2_SPTR8)subject; + } + +else /* 16-bit mode */ + { + PCRE2_SPTR16 pp = (PCRE2_SPTR16)subject + offset; + for (; count > 0; count--) + { + pp--; + if ((*pp & 0xfc00) == 0xdc00) pp--; + } + yield = pp - (PCRE2_SPTR16)subject; + } + +return (yield >= 0)? yield : 0; +} + + /************************************************* * Read or extend an input line * *************************************************/ @@ -3099,8 +3149,7 @@ if ((pat_patctl.control & CTL_INFO) != 0) BOOL match_limit_set, recursion_limit_set; uint32_t backrefmax, bsr_convention, capture_count, first_ctype, first_cunit, hascrorlf, jchanged, last_ctype, last_cunit, match_empty, match_limit, - maxlookbehind, minlength, nameentrysize, namecount, newline_convention, - recursion_limit; + minlength, nameentrysize, namecount, newline_convention, recursion_limit; /* These info requests may return PCRE2_ERROR_UNSET. */ @@ -3145,7 +3194,6 @@ if ((pat_patctl.control & CTL_INFO) != 0) pattern_info(PCRE2_INFO_LASTCODEUNIT, &last_cunit, FALSE) + pattern_info(PCRE2_INFO_LASTCODETYPE, &last_ctype, FALSE) + pattern_info(PCRE2_INFO_MATCHEMPTY, &match_empty, FALSE) + - pattern_info(PCRE2_INFO_MAXLOOKBEHIND, &maxlookbehind, FALSE) + pattern_info(PCRE2_INFO_MINLENGTH, &minlength, FALSE) + pattern_info(PCRE2_INFO_NAMECOUNT, &namecount, FALSE) + pattern_info(PCRE2_INFO_NAMEENTRYSIZE, &nameentrysize, FALSE) + @@ -3700,6 +3748,11 @@ if (TEST(compiled_code, ==, NULL)) fprintf(outfile, "\n"); return PR_SKIP; } + +/* Remember the maximum lookbehind, for partial matching. */ + +if (pattern_info(PCRE2_INFO_MAXLOOKBEHIND, &maxlookbehind, FALSE) != 0) + return PR_ABEND; /* Call the JIT compiler if requested. */ @@ -4875,22 +4928,41 @@ for (gmatched = 0;; gmatched++) } /* End of handling a successful match */ /* There was a partial match. The value of ovector[0] is the bumpalong point, - not any \K point that might exist. */ + that is, startchar, not any \K point that might have been passed. */ else if (capcount == PCRE2_ERROR_PARTIAL) { + PCRE2_SIZE poffset; + int backlength; + int rubriclength = 0; + fprintf(outfile, "Partial match"); if ((dat_datctl.control & CTL_MARK) != 0 && TESTFLD(match_data, mark, !=, NULL)) { fprintf(outfile, ", mark="); - PCHARSV(CASTFLD(void *, match_data, mark), 0, -1, utf, outfile); + PCHARS(rubriclength, CASTFLD(void *, match_data, mark), 0, -1, utf, outfile); + rubriclength += 7; } fprintf(outfile, ": "); + rubriclength += 15; + + poffset = backchars(pp, ovector[0], maxlookbehind, utf); + PCHARS(backlength, pp, poffset, ovector[0] - poffset, utf, outfile); PCHARSV(pp, ovector[0], ulen - ovector[0], utf, outfile); + if ((pat_patctl.control & CTL_JITVERIFY) != 0 && jit_was_used) fprintf(outfile, " (JIT)"); fprintf(outfile, "\n"); + + if (backlength != 0) + { + int i; + for (i = 0; i < rubriclength; i++) fprintf(outfile, " "); + for (i = 0; i < backlength; i++) fprintf(outfile, "<"); + fprintf(outfile, "\n"); + } + break; /* Out of the /g loop */ } /* End of handling partial match */ diff --git a/testdata/testoutput2 b/testdata/testoutput2 index a8a4df6..177c810 100644 --- a/testdata/testoutput2 +++ b/testdata/testoutput2 @@ -9286,17 +9286,21 @@ Partial match: abc12 xyzabc123pqr 0: 123 xyzabc12\=ps -Partial match: 12 +Partial match: abc12 + <<< xyzabc12\=ph -Partial match: 12 +Partial match: abc12 + <<< /\babc\b/ +++abc+++ 0: abc +++ab\=ps -Partial match: ab +Partial match: +ab + < +++ab\=ph -Partial match: ab +Partial match: +ab + < /(?&word)(?&element)(?(DEFINE)(? <[^m][^>]>[^<])(? \w*+))/B ------------------------------------------------------------------ @@ -10324,7 +10328,8 @@ No match /(?<=abc)def/ abc\=ph -Partial match: +Partial match: abc + <<< /abc$/ abc @@ -11877,9 +11882,11 @@ Callout 2: last capture = 0 /(?<=123)(*MARK:xx)abc/mark xxxx123a\=ph -Partial match, mark=xx: a +Partial match, mark=xx: 123a + <<< xxxx123a\=ps -Partial match, mark=xx: a +Partial match, mark=xx: 123a + <<< /123\Kabc/ xxxx123a\=ph diff --git a/testdata/testoutput6 b/testdata/testoutput6 index c629985..e5199ee 100644 --- a/testdata/testoutput6 +++ b/testdata/testoutput6 @@ -947,7 +947,8 @@ Partial match: abc xyzfo\=ps No match foob\=ps,offset=2 -Partial match: b +Partial match: foob + <<< foobar...\=ps,dfa_restart,offset=4 0: ar xyzfo\=ps @@ -7092,17 +7093,21 @@ Failed: error -40: item unsupported for DFA matching xyzabc123pqr 0: 123 xyzabc12\=ps -Partial match: 12 +Partial match: abc12 + <<< xyzabc12\=ph -Partial match: 12 +Partial match: abc12 + <<< /\babc\b/ +++abc+++ 0: abc +++ab\=ps -Partial match: ab +Partial match: +ab + < +++ab\=ph -Partial match: ab +Partial match: +ab + < /(?=C)/g,aftertext ABCDECBA @@ -7226,7 +7231,8 @@ Failed: error -40: item unsupported for DFA matching /(?<=abc)def/ abc\=ph -Partial match: +Partial match: abc + <<< /abc$/ abc