top level:

        Reviewed by: Richard Williamson

        * pcre/doc/*: Added.
        * pcre/testdata/*: Added.

pcre:

   (i)   Added long-form option names like gnu grep.
   (ii)  Added --help to list all options with an explanatory phrase.
   (iii) Added -r, --recursive to recurse into sub-directories.
   (iv)  Added -f, --file to read patterns from a file.


git-svn-id: http://svn.webkit.org/repository/webkit/trunk@2927 268f45cc-cd09-0410-ab3c-d52691b4dbfc

JavaScriptCore/ChangeLog

+2002-12-04  Maciej Stachowiak  <mjs@apple.com>
+
+        Reviewed by: Richard Williamson
+
+        * pcre/doc/*: Added.
+        * pcre/testdata/*: Added.
+
 2002-12-03  Maciej Stachowiak  <mjs@apple.com>
 
         Reviewed by: Darin Adler

JavaScriptCore/ChangeLog-2003-10-25

+2002-12-04  Maciej Stachowiak  <mjs@apple.com>
+
+        Reviewed by: Richard Williamson
+
+        * pcre/doc/*: Added.
+        * pcre/testdata/*: Added.
+
 2002-12-03  Maciej Stachowiak  <mjs@apple.com>
 
         Reviewed by: Darin Adler

JavaScriptCore/pcre/doc/Tech.Notes

+Technical Notes about PCRE
+--------------------------
+
+Many years ago I implemented some regular expression functions to an algorithm
+suggested by Martin Richards. These were not Unix-like in form, and were quite
+restricted in what they could do by comparison with Perl. The interesting part
+about the algorithm was that the amount of space required to hold the compiled
+form of an expression was known in advance. The code to apply an expression did
+not operate by backtracking, as the Henry Spencer and Perl code does, but
+instead checked all possibilities simultaneously by keeping a list of current
+states and checking all of them as it advanced through the subject string. (In
+the terminology of Jeffrey Friedl's book, it was a "DFA algorithm".) When the
+pattern was all used up, all remaining states were possible matches, and the
+one matching the longest subset of the subject string was chosen. This did not
+necessarily maximize the individual wild portions of the pattern, as is
+expected in Unix and Perl-style regular expressions.
+
+By contrast, the code originally written by Henry Spencer and subsequently
+heavily modified for Perl actually compiles the expression twice: once in a
+dummy mode in order to find out how much store will be needed, and then for
+real. The execution function operates by backtracking and maximizing (or,
+optionally, minimizing in Perl) the amount of the subject that matches
+individual wild portions of the pattern. This is an "NFA algorithm" in Friedl's
+terminology.
+
+For the set of functions that forms PCRE (which are unrelated to those
+mentioned above), I tried at first to invent an algorithm that used an amount
+of store bounded by a multiple of the number of characters in the pattern, to
+save on compiling time. However, because of the greater complexity in Perl
+regular expressions, I couldn't do this. In any case, a first pass through the
+pattern is needed, in order to find internal flag settings like (?i) at top
+level. So PCRE works by running a very degenerate first pass to calculate a
+maximum store size, and then a second pass to do the real compile - which may
+use a bit less than the predicted amount of store. The idea is that this is
+going to turn out faster because the first pass is degenerate and the second
+pass can just store stuff straight into the vector. It does make the compiling
+functions bigger, of course, but they have got quite big anyway to handle all
+the Perl stuff.
+
+The compiled form of a pattern is a vector of bytes, containing items of
+variable length. The first byte in an item is an opcode, and the length of the
+item is either implicit in the opcode or contained in the data bytes which
+follow it. A list of all the opcodes follows:
+
+Opcodes with no following data
+------------------------------
+
+These items are all just one byte long
+
+  OP_END                 end of pattern
+  OP_ANY                 match any character
+  OP_SOD                 match start of data: \A
+  OP_CIRC                ^ (start of data, or after \n in multiline)
+  OP_NOT_WORD_BOUNDARY   \W
+  OP_WORD_BOUNDARY       \w
+  OP_NOT_DIGIT           \D
+  OP_DIGIT               \d
+  OP_NOT_WHITESPACE      \S
+  OP_WHITESPACE          \s
+  OP_NOT_WORDCHAR        \W
+  OP_WORDCHAR            \w
+  OP_EODN                match end of data or \n at end: \Z
+  OP_EOD                 match end of data: \z
+  OP_DOLL                $ (end of data, or before \n in multiline)
+  OP_RECURSE             match the pattern recursively
+
+
+Repeating single characters
+---------------------------
+
+The common repeats (*, +, ?) when applied to a single character appear as
+two-byte items using the following opcodes:
+
+  OP_STAR
+  OP_MINSTAR
+  OP_PLUS
+  OP_MINPLUS
+  OP_QUERY
+  OP_MINQUERY
+
+Those with "MIN" in their name are the minimizing versions. Each is followed by
+the character that is to be repeated. Other repeats make use of
+
+  OP_UPTO
+  OP_MINUPTO
+  OP_EXACT
+
+which are followed by a two-byte count (most significant first) and the
+repeated character. OP_UPTO matches from 0 to the given number. A repeat with a
+non-zero minimum and a fixed maximum is coded as an OP_EXACT followed by an
+OP_UPTO (or OP_MINUPTO).
+
+
+Repeating character types
+-------------------------
+
+Repeats of things like \d are done exactly as for single characters, except
+that instead of a character, the opcode for the type is stored in the data
+byte. The opcodes are:
+
+  OP_TYPESTAR
+  OP_TYPEMINSTAR
+  OP_TYPEPLUS
+  OP_TYPEMINPLUS
+  OP_TYPEQUERY
+  OP_TYPEMINQUERY
+  OP_TYPEUPTO
+  OP_TYPEMINUPTO
+  OP_TYPEEXACT
+
+
+Matching a character string
+---------------------------
+
+The OP_CHARS opcode is followed by a one-byte count and then that number of
+characters. If there are more than 255 characters in sequence, successive
+instances of OP_CHARS are used.
+
+
+Character classes
+-----------------
+
+OP_CLASS is used for a character class, provided there are at least two
+characters in the class. If there is only one character, OP_CHARS is used for a
+positive class, and OP_NOT for a negative one (that is, for something like
+[^a]). Another set of repeating opcodes (OP_NOTSTAR etc.) are used for a
+repeated, negated, single-character class. The normal ones (OP_STAR etc.) are
+used for a repeated positive single-character class.
+
+OP_CLASS is followed by a 32-byte bit map containing a 1 bit for every
+character that is acceptable. The bits are counted from the least significant
+end of each byte.
+
+
+Back references
+---------------
+
+OP_REF is followed by two bytes containing the reference number.
+
+
+Repeating character classes and back references
+-----------------------------------------------
+
+Single-character classes are handled specially (see above). This applies to
+OP_CLASS and OP_REF. In both cases, the repeat information follows the base
+item. The matching code looks at the following opcode to see if it is one of
+
+  OP_CRSTAR
+  OP_CRMINSTAR
+  OP_CRPLUS
+  OP_CRMINPLUS
+  OP_CRQUERY
+  OP_CRMINQUERY
+  OP_CRRANGE
+  OP_CRMINRANGE
+
+All but the last two are just single-byte items. The others are followed by
+four bytes of data, comprising the minimum and maximum repeat counts.
+
+
+Brackets and alternation
+------------------------
+
+A pair of non-capturing (round) brackets is wrapped round each expression at
+compile time, so alternation always happens in the context of brackets.
+
+Non-capturing brackets use the opcode OP_BRA, while capturing brackets use
+OP_BRA+1, OP_BRA+2, etc. [Note for North Americans: "bracket" to some English
+speakers, including myself, can be round, square, curly, or pointy. Hence this
+usage.]
+
+Originally PCRE was limited to 99 capturing brackets (so as not to use up all
+the opcodes). From release 3.5, there is no limit. What happens is that the
+first ones, up to EXTRACT_BASIC_MAX are handled with separate opcodes, as
+above. If there are more, the opcode is set to EXTRACT_BASIC_MAX+1, and the
+first operation in the bracket is OP_BRANUMBER, followed by a 2-byte bracket
+number. This opcode is ignored while matching, but is fished out when handling
+the bracket itself. (They could have all been done like this, but I was making
+minimal changes.)
+
+A bracket opcode is followed by two bytes which give the offset to the next
+alternative OP_ALT or, if there aren't any branches, to the matching KET
+opcode. Each OP_ALT is followed by two bytes giving the offset to the next one,
+or to the KET opcode.
+
+OP_KET is used for subpatterns that do not repeat indefinitely, while
+OP_KETRMIN and OP_KETRMAX are used for indefinite repetitions, minimally or
+maximally respectively. All three are followed by two bytes giving (as a
+positive number) the offset back to the matching BRA opcode.
+
+If a subpattern is quantified such that it is permitted to match zero times, it
+is preceded by one of OP_BRAZERO or OP_BRAMINZERO. These are single-byte
+opcodes which tell the matcher that skipping this subpattern entirely is a
+valid branch.
+
+A subpattern with an indefinite maximum repetition is replicated in the
+compiled data its minimum number of times (or once with a BRAZERO if the
+minimum is zero), with the final copy terminating with a KETRMIN or KETRMAX as
+appropriate.
+
+A subpattern with a bounded maximum repetition is replicated in a nested
+fashion up to the maximum number of times, with BRAZERO or BRAMINZERO before
+each replication after the minimum, so that, for example, (abc){2,5} is
+compiled as (abc)(abc)((abc)((abc)(abc)?)?)?. The 99 and 200 bracket limits do
+not apply to these internally generated brackets.
+
+
+Assertions
+----------
+
+Forward assertions are just like other subpatterns, but starting with one of
+the opcodes OP_ASSERT or OP_ASSERT_NOT. Backward assertions use the opcodes
+OP_ASSERTBACK and OP_ASSERTBACK_NOT, and the first opcode inside the assertion
+is OP_REVERSE, followed by a two byte count of the number of characters to move
+back the pointer in the subject string. When operating in UTF-8 mode, the count
+is a character count rather than a byte count. A separate count is present in
+each alternative of a lookbehind assertion, allowing them to have different
+fixed lengths.
+
+
+Once-only subpatterns
+---------------------
+
+These are also just like other subpatterns, but they start with the opcode
+OP_ONCE.
+
+
+Conditional subpatterns
+-----------------------
+
+These are like other subpatterns, but they start with the opcode OP_COND. If
+the condition is a back reference, this is stored at the start of the
+subpattern using the opcode OP_CREF followed by two bytes containing the
+reference number. Otherwise, a conditional subpattern will always start with
+one of the assertions.
+
+
+Changing options
+----------------
+
+If any of the /i, /m, or /s options are changed within a parenthesized group,
+an OP_OPT opcode is compiled, followed by one byte containing the new settings
+of these flags. If there are several alternatives in a group, there is an
+occurrence of OP_OPT at the start of all those following the first options
+change, to set appropriate options for the start of the alternative.
+Immediately after the end of the group there is another such item to reset the
+flags to their previous values. Other changes of flag within the pattern can be
+handled entirely at compile time, and so do not cause anything to be put into
+the compiled data.
+
+
+Philip Hazel
+August 2001

JavaScriptCore/pcre/doc/perltest.txt

+The perltest program
+--------------------
+
+The perltest program tests Perl's regular expressions; it has the same
+specification as pcretest, and so can be given identical input, except that
+input patterns can be followed only by Perl's lower case modifiers and /+ (as
+used by pcretest), which is recognized and handled by the program.
+
+The data lines are processed as Perl double-quoted strings, so if they contain
+" \ $ or @ characters, these have to be escaped. For this reason, all such
+characters in testinput1 and testinput3 are escaped so that they can be used
+for perltest as well as for pcretest, and the special upper case modifiers such
+as /A that pcretest recognizes are not used in these files. The output should
+be identical, apart from the initial identifying banner.
+
+For testing UTF-8 features, an alternative form of perltest, called perltest8,
+is supplied. This requires Perl 5.6 or higher. It recognizes the special
+modifier /8 that pcretest uses to invoke UTF-8 functionality. The testinput5
+file can be fed to perltest8.
+
+The testinput2 and testinput4 files are not suitable for feeding to perltest,
+since they do make use of the special upper case modifiers and escapes that
+pcretest uses to test some features of PCRE. The first of these files also
+contains malformed regular expressions, in order to check that PCRE diagnoses
+them correctly. Similarly, testinput6 tests UTF-8 features that do not relate
+to Perl.
+
+Philip Hazel <ph10@cam.ac.uk>
+August 2000