- •Introduction
- •Who This Book Is For
- •What This Book Covers
- •How This Book Is Structured
- •What You Need to Use This Book
- •Conventions
- •Source Code
- •Errata
- •p2p.wrox.com
- •What Are Regular Expressions?
- •What Can Regular Expressions Be Used For?
- •Finding Doubled Words
- •Checking Input from Web Forms
- •Changing Date Formats
- •Finding Incorrect Case
- •Adding Links to URLs
- •Regular Expressions You Already Use
- •Search and Replace in Word Processors
- •Directory Listings
- •Online Searching
- •Why Regular Expressions Seem Intimidating
- •Compact, Cryptic Syntax
- •Whitespace Can Significantly Alter the Meaning
- •No Standards Body
- •Differences between Implementations
- •Characters Change Meaning in Different Contexts
- •Regular Expressions Can Be Case Sensitive
- •Case-Sensitive and Case-Insensitive Matching
- •Case and Metacharacters
- •Continual Evolution in Techniques Supported
- •Multiple Solutions for a Single Problem
- •What You Want to Do with a Regular Expression
- •Replacing Text in Quantity
- •Regular Expression Tools
- •findstr
- •Microsoft Word
- •StarOffice Writer/OpenOffice.org Writer
- •Komodo Rx Package
- •PowerGrep
- •Microsoft Excel
- •JavaScript and JScript
- •VBScript
- •Visual Basic.NET
- •Java
- •Perl
- •MySQL
- •SQL Server 2000
- •W3C XML Schema
- •An Analytical Approach to Using Regular Expressions
- •Express and Document What You Want to Do in English
- •Consider the Regular Expression Options Available
- •Consider Sensitivity and Specificity
- •Create Appropriate Regular Expressions
- •Document All but Simple Regular Expressions
- •Document What You Expect the Regular Expression to Do
- •Document What You Want to Match
- •Test the Results of a Regular Expression
- •Matching Single Characters
- •Matching Sequences of Characters That Each Occur Once
- •Introducing Metacharacters
- •Matching Sequences of Different Characters
- •Matching Optional Characters
- •Matching Multiple Optional Characters
- •Other Cardinality Operators
- •The * Quantifier
- •The + Quantifier
- •The Curly-Brace Syntax
- •The {n} Syntax
- •The {n,m} Syntax
- •Exercises
- •Regular Expression Metacharacters
- •Thinking about Characters and Positions
- •The Period (.) Metacharacter
- •Matching Variably Structured Part Numbers
- •Matching a Literal Period
- •The \w Metacharacter
- •The \W Metacharacter
- •Digits and Nondigits
- •The \d Metacharacter
- •Canadian Postal Code Example
- •The \D Metacharacter
- •Alternatives to \d and \D
- •The \s Metacharacter
- •Handling Optional Whitespace
- •The \S Metacharacter
- •The \t Metacharacter
- •The \n Metacharacter
- •Escaped Characters
- •Finding the Backslash
- •Modifiers
- •Global Search
- •Case-Insensitive Search
- •Exercises
- •Introduction to Character Classes
- •Choice between Two Characters
- •Using Quantifiers with Character Classes
- •Using the \b Metacharacter in Character Classes
- •Selecting Literal Square Brackets
- •Using Ranges in Character Classes
- •Alphabetic Ranges
- •Use [A-z] With Care
- •Digit Ranges in Character Classes
- •Hexadecimal Numbers
- •IP Addresses
- •Reverse Ranges in Character Classes
- •A Potential Range Trap
- •Finding HTML Heading Elements
- •Metacharacter Meaning within Character Classes
- •The ^ metacharacter
- •How to Use the - Metacharacter
- •Negated Character Classes
- •Combining Positive and Negative Character Classes
- •POSIX Character Classes
- •The [:alnum:] Character Class
- •Exercises
- •String, Line, and Word Boundaries
- •The ^ Metacharacter
- •The ^ Metacharacter and Multiline Mode
- •The $ Metacharacter
- •The $ Metacharacter in Multiline Mode
- •Using the ^ and $ Metacharacters Together
- •Matching Blank Lines
- •Working with Dollar Amounts
- •Revisiting the IP Address Example
- •What Is a Word?
- •Identifying Word Boundaries
- •The \< Syntax
- •The \>Syntax
- •The \b Syntax
- •The \B Metacharacter
- •Less-Common Word-Boundary Metacharacters
- •Exercises
- •Grouping Using Parentheses
- •Parentheses and Quantifiers
- •Matching Literal Parentheses
- •U.S. Telephone Number Example
- •Alternation
- •Choosing among Multiple Options
- •Unexpected Alternation Behavior
- •Capturing Parentheses
- •Numbering of Captured Groups
- •Numbering When Using Nested Parentheses
- •Named Groups
- •Non-Capturing Parentheses
- •Back References
- •Exercises
- •Why You Need Lookahead and Lookbehind
- •The (? metacharacters
- •Lookahead
- •Positive Lookahead
- •Negative Lookahead
- •Positive Lookahead Examples
- •Positive Lookahead in the Same Document
- •Inserting an Apostrophe
- •Lookbehind
- •Positive Lookbehind
- •Negative Lookbehind
- •How to Match Positions
- •Adding Commas to Large Numbers
- •Exercises
- •What Are Sensitivity and Specificity?
- •Extreme Sensitivity, Awful Specificity
- •Email Addresses Example
- •Replacing Hyphens Example
- •The Sensitivity/Specificity Trade-Off
- •Sensitivity, Specificity, and Positional Characters
- •Sensitivity, Specificity, and Modes
- •Sensitivity, Specificity, and Lookahead and Lookbehind
- •How Much Should the Regular Expressions Do?
- •Abbreviations
- •Characters from Other Languages
- •Names
- •Sensitivity and How to Achieve It
- •Specificity and How to Maximize It
- •Exercises
- •Documenting Regular Expressions
- •Document the Problem Definition
- •Add Comments to Your Code
- •Making Use of Extended Mode
- •Know Your Data
- •Abbreviations
- •Proper Names
- •Incorrect Spelling
- •Creating Test Cases
- •Debugging Regular Expressions
- •Treacherous Whitespace
- •Backslashes Causing Problems
- •Considering Other Causes
- •The User Interface
- •Metacharacters Available
- •Quantifiers
- •The @ Quantifier
- •The {n,m} Syntax
- •Modes
- •Character Classes
- •Back References
- •Lookahead and Lookbehind
- •Lazy Matching versus Greedy Matching
- •Examples
- •Character Class Examples, Including Ranges
- •Whole Word Searches
- •Search-and-Replace Examples
- •Changing Name Structure Using Back References
- •Manipulating Dates
- •The Star Training Company Example
- •Regular Expressions in Visual Basic for Applications
- •Exercises
- •The User Interface
- •Metacharacters Available
- •Quantifiers
- •Modes
- •Character Classes
- •Alternation
- •Back References
- •Lookahead and Lookbehind
- •Search Example
- •Search-and-Replace Example
- •Online Chats
- •POSIX Character Classes
- •Matching Numeric Digits
- •Exercises
- •Introducing findstr
- •Finding Literal Text
- •Quantifiers
- •Character Classes
- •Command-Line Switch Examples
- •The /v Switch
- •The /a Switch
- •Single File Examples
- •Simple Character Class Example
- •Find Protocols Example
- •Multiple File Example
- •A Filelist Example
- •Exercises
- •The PowerGREP Interface
- •A Simple Find Example
- •The Replace Tab
- •The File Finder Tab
- •Syntax Coloring
- •Other Tabs
- •Numeric Digits and Alphabetic Characters
- •Quantifiers
- •Back References
- •Alternation
- •Line Position Metacharacters
- •Word-Boundary Metacharacters
- •Lookahead and Lookbehind
- •Longer Examples
- •Finding HTML Horizontal Rule Elements
- •Matching Time Example
- •Exercises
- •The Excel Find Interface
- •Escaping Wildcard Characters
- •Using Wildcards in Data Forms
- •Using Wildcards in Filters
- •Exercises
- •Using LIKE with Regular Expressions
- •The % Metacharacter
- •The _ Metacharacter
- •Character Classes
- •Negated Character Classes
- •Using Full-Text Search
- •Using The CONTAINS Predicate
- •Document Filters on Image Columns
- •Exercises
- •Using the _ and % Metacharacters
- •Testing Matching of Literals: _ and % Metacharacters
- •Using Positional Metacharacters
- •Using Character Classes
- •Quantifiers
- •Social Security Number Example
- •Exercises
- •The Interface to Metacharacters in Microsoft Access
- •Creating a Hard-Wired Query
- •Creating a Parameter Query
- •Using the ? Metacharacter
- •Using the * Metacharacter
- •Using the # Metacharacter
- •Using the # Character with Date/Time Data
- •Using Character Classes in Access
- •Exercises
- •The RegExp Object
- •Attributes of the RegExp Object
- •The Other Properties of the RegExp Object
- •The test() Method of the RegExp Object
- •The exec() Method of the RegExp Object
- •The String Object
- •Metacharacters in JavaScript and JScript
- •SSN Validation Example
- •Exercises
- •The RegExp Object and How to Use It
- •Quantifiers
- •Positional Metacharacters
- •Character Classes
- •Word Boundaries
- •Lookahead
- •Grouping and Nongrouping Parentheses
- •Exercises
- •The System.Text.RegularExpressions namespace
- •A Simple Visual Basic .NET Example
- •The Classes of System.Text.RegularExpressions
- •The Regex Object
- •Using the Match Object and Matches Collection
- •Using the Match.Success Property and Match.NextMatch Method
- •The GroupCollection and Group Classes
- •The CaptureCollection and Capture Class
- •The RegexOptions Enumeration
- •Case-Insensitive Matching: The IgnoreCase Option
- •Multiline Matching: The Effect on the ^ and $ Metacharacters
- •Right to Left Matching: The RightToLeft Option
- •Lookahead and Lookbehind
- •Exercises
- •An Introductory Example
- •The Classes of System.Text.RegularExpressions
- •The Regex Class
- •The Options Property of the Regex Class
- •Regex Class Methods
- •The CompileToAssembly() Method
- •The GetGroupNames() Method
- •The GetGroupNumbers() Method
- •GroupNumberFromName() and GroupNameFromNumber() Methods
- •The IsMatch() Method
- •The Match() Method
- •The Matches() Method
- •The Replace() Method
- •The Split() Method
- •Using the Static Methods of the Regex Class
- •The IsMatch() Method as a Static
- •The Match() Method as a Static
- •The Matches() Method as a Static
- •The Replace() Method as a Static
- •The Split() Method as a Static
- •The Match and Matches Classes
- •The Match Class
- •The GroupCollection and Group Classes
- •The RegexOptions Class
- •The IgnorePatternWhitespace Option
- •Metacharacters Supported in Visual C# .NET
- •Using Named Groups
- •Using Back References
- •Exercise
- •The ereg() Set of Functions
- •The ereg() Function
- •The ereg() Function with Three Arguments
- •The eregi() Function
- •The ereg_replace() Function
- •The eregi_replace() Function
- •The split() Function
- •The spliti() Function
- •The sql_regcase() Function
- •Perl Compatible Regular Expressions
- •Pattern Delimiters in PCRE
- •Escaping Pattern Delimiters
- •Matching Modifiers in PCRE
- •Using the preg_match() Function
- •Using the preg_match_all() Function
- •Using the preg_grep() Function
- •Using the preg_quote() Function
- •Using the preg_replace() Function
- •Using the preg_replace_callback() Function
- •Using the preg_split() Function
- •Supported Metacharacters with ereg()
- •Using POSIX Character Classes with PHP
- •Supported Metacharacters with PCRE
- •Positional Metacharacters
- •Character Classes in PHP
- •Documenting PHP Regular Expressions
- •Exercises
- •W3C XML Schema Basics
- •Tools for Using W3C XML Schema
- •Comparing XML Schema and DTDs
- •How Constraints Are Expressed in W3C XML Schema
- •W3C XML Schema Datatypes
- •Derivation by Restriction
- •Unicode and W3C XML Schema
- •Unicode Overview
- •Using Unicode Character Classes
- •Matching Decimal Numbers
- •Mixing Unicode Character Classes with Other Metacharacters
- •Unicode Character Blocks
- •Using Unicode Character Blocks
- •Metacharacters Supported in W3C XML Schema
- •Positional Metacharacters
- •Matching Numeric Digits
- •Alternation
- •Using the \w and \s Metacharacters
- •Escaping Metacharacters
- •Exercises
- •Introduction to the java.util.regex Package
- •Obtaining and Installing Java
- •The Pattern Class
- •Using the matches() Method Statically
- •Two Simple Java Examples
- •The Properties (Fields) of the Pattern Class
- •The CASE_INSENSITIVE Flag
- •Using the COMMENTS Flag
- •The DOTALL Flag
- •The MULTILINE Flag
- •The UNICODE_CASE Flag
- •The UNIX_LINES Flag
- •The Methods of the Pattern Class
- •The compile() Method
- •The flags() Method
- •The matcher() Method
- •The matches() Method
- •The pattern() Method
- •The split() Method
- •The Matcher Class
- •The appendReplacement() Method
- •The appendTail() Method
- •The end() Method
- •The find() Method
- •The group() Method
- •The groupCount() Method
- •The lookingAt() Method
- •The matches() Method
- •The pattern() Method
- •The replaceAll() Method
- •The replaceFirst() Method
- •The reset() Method
- •The start() Method
- •The PatternSyntaxException Class
- •Using the \d Metacharacter
- •Character Classes
- •The POSIX Character Classes in the java.util.regex Package
- •Unicode Character Classes and Character Blocks
- •Using Escaped Characters
- •Using Methods of the String Class
- •Using the matches() Method
- •Using the replaceFirst() Method
- •Using the replaceAll() Method
- •Using the split() Method
- •Exercises
- •Obtaining and Installing Perl
- •Creating a Simple Perl Program
- •Basics of Perl Regular Expression Usage
- •Using the m// Operator
- •Using Other Regular Expression Delimiters
- •Matching Using Variable Substitution
- •Using the s/// Operator
- •Using s/// with the Global Modifier
- •Using s/// with the Default Variable
- •Using the split Operator
- •Using Quantifiers in Perl
- •Using Positional Metacharacters
- •Captured Groups in Perl
- •Using Back References in Perl
- •Using Alternation
- •Using Character Classes in Perl
- •Using Lookahead
- •Using Lookbehind
- •Escaping Metacharacters
- •A Simple Perl Regex Tester
- •Exercises
- •Index
Regular Expression Tools and an Approach to Using Them
SQL Server 2000
Microsoft’s SQL Server 2000 is, at the time of this writing, the flagship database management system product from Microsoft. SQL Server 2005 is scheduled to be released a few months after this book is published.
The use of regular expressions in SQL Server is described in more detail in Chapter 16.
W3C XML Schema
W3C XML Schema is a language that specifies the permitted structure of classes of XML documents. The W3C XML Schema element, xs:pattern, allows the use of regular expressions to constrain the values allowed in XML documents.
The use of regular expressions in W3C XML Schema is described in more detail in Chapter 24.
An Analytical Approach to Using Regular Expressions
This section explains in some detail an approach that I think can help you considerably in thinking about, designing, and maintaining regular expressions in your code.
The approach shouldn’t be applied mechanically. When you are creating simple regular expressions, such a full-featured approach is pretty evidently overkill. In other situations, you won’t be able to apply all the suggested components because a language or tool may not, for example, support the desired approach to documentation. However, for anything but pretty simple use of regular expressions, making use of an approach like the one I am about to describe will help you create regular expression patterns that achieve exactly what you want and help make maintenance of those regular expressions significantly easier.
The approach is described in several parts. First, here is a list of the components of the approach that I suggest that you take:
Express and document what you want to do in English.
Consider the data source and its likely contents.
Consider the regular expression options available to you.
Consider sensitivity and specificity.
Create appropriate regular expressions.
Document all but simple regular expressions.
Use whitespace to aid in clear documentation of the regular expression.
Test the results of a regular expression.
31
Chapter 2
If you are planning to use a simple regular expression to carry out a search and replace in Microsoft Word, you almost certainly won’t want to bother with such a detailed approach. However, as the complexity of what you are attempting to achieve increases, a systematic approach such as the one just listed becomes increasingly useful.
Let’s look at each of the components of the suggested approach.
Express and Document What You Want to Do in English
In almost any type of programming that goes beyond the trivial, a fairly formal expression of the problem to be solved is of great importance. The same principle applies to the use of any but the simplest regular expressions.
Just as planning and developing an application may take several iterations, during which you refine the detail of the application’s design, you may need several iterations to get the description of what you want to do correct.
In the Star Training Company example in Chapter 1, the new recruit might have made the following first attempt at defining in English what he wanted to achieve:
Replace every occurrence of Star with Moon.
As you saw in Chapter 1, that simple approach, particularly when the default behavior is that the pattern Star is applied in a case-insensitive way, leads to a mess in the document where the blanket replacement has been carried out. The problem with that simple approach is that there are many undesired matches.
The inappropriate replacements result because the definition of what is to be matched isn’t sufficiently precise. The result is a regular expression of low specificity, a problem that is discussed in more detail in Chapter 8.
What is needed is to make the specification of the problem more precise. A second attempt to specify what is desired might be the following:
Replace every occurrence of Star that has an initial uppercase S with Moon.
That removes some of the undesired matches, such as starfish and stare, in the file
StarOriginal.doc.
An alternative approach to the problem might be this:
Replace every occurrence of Star, when Star is a word, with Moon.
That is an improvement over the initial problem definition, because undesired matches such as startling and stare are averted.
You can make the problem definition even more precise by combining the last two definitions:
Replace every occurrence of Star with Moon when Star is a word and Star also has an initial letter that is an uppercase S.
32
Regular Expression Tools and an Approach to Using Them
You can attempt to make the problem definition even more precise, as follows:
Replace every occurrence of Star with Moon when Star is a word and Star also has an initial letter that is an uppercase S and when Star is also followed by a single space character and the word Training (with an initial uppercase T).
But a definition as precise as this will cause you to miss desired matches. For example, in the file StarOriginal.doc, the match in the sentences.
Start now with Star.
and
Think Star.
should be matched but won’t be.
The definition is now too precise. You are losing matches to the regular expression pattern that you want to identify. In that situation, the sensitivity of the regular expression isn’t what you want.
The issues surrounding sensitivity of a regular expression pattern are discussed in more detail in Chapter 8.
One approach to that inadequate sensitivity is to widen the problem definition a little, as follows:
Replace every occurrence of Star with Moon when Star is a word and Star also has an initial letter that is an uppercase S and when Star is also followed by a period character (a full stop) or is followed by a single space character and the word Training (with an initial uppercase T).
That problem definition, when correctly translated into a regular expression pattern, will match Star pretty precisely and will also cut out many of the undesired matches mentioned earlier. Chapter 8 will look at this problem of balancing specificity and sensitivity in a regular expression. In addition, using the various regular expression techniques that you will learn in Chapters 3 to 7 will allow you to translate the various versions of the problem definition into regular expression patterns that you can test against the sample text.
I hope you have noticed that the basis on which the problem definition was revised was an understanding of the data sources that you want to use and their likely contents.
Consider the Data Source and Its Likely Contents
In the Star Training Company example in Chapter 1, you saw how a failure to consider the data source led to many problems in the document created by using a simplistic search and replace. It was, for example, undesirable to replace startling so that the pseudoword Moontling appeared in the middle of a sentence after the search and replace had been carried out. One of the causes of such inappropriate results is a failure to seriously consider the data source to which a regular expression is to be applied.
For example, when the data source is a structured comma-delimited file you may find that the data has very few variations in its structure. On the other hand, when you have word-processed documents as the data source you will need to think carefully about issues such as different word forms.
33
Chapter 2
A regular expression that matches the sequence of characters ball may or may not match the plural form balls or the possessive form ball’s, depending on how the pattern is constructed. Similarly, it may or may not match related words such as balloon, balloons, and ballooned.
In addition, you may have similar words, some of which you want to match while avoiding matching others. In real life, you may also have to consider the issue of whether it is appropriate to spend time trying to match words that are incorrectly spelled.
When validating data entered by a user in a form on a Web page, you will need to consider possible ways the user might enter data. For example, will a user enter the 16 digits of a credit card number with or without spaces? In that situation and others, it makes sense to allow for all reasonable variants that a user might enter and provide additional code to carry out any necessary conversions into the type you desire.
Consider the Regular Expression Options Available
As mentioned in Chapter 1, not every language or application has the same set of regular expression tools available to users or developers. So when faced with a practical problem for which regular expressions might be a suitable solution, you need to consider carefully what regular expression tools are available.
One of the situations where use of regular expressions is limited is when you use regular expressions in Microsoft Word. OpenOffice.org Writer is significantly less limited in regular expression support. Programmer’s editors such as ActiveState Komodo, Microsoft Visual Studio, and the Microsoft Visual Basic for Applications Editor all have different levels of regular expression support.
Be careful to distinguish the regular expression support in a programmer’s editor from the regular expression support in the languages that a programmer’s editor may support.
Frequently, you will be in a situation where the language to be used for a programming project has already been decided, probably for reasons unrelated to the regular expression functionality supported by the language. In that situation, you will quite possibly need to make the best use of the regular expression functionality of the language for the project. However, in some situations you will have more flexibility. For example, it may be possible or desirable to manipulate input files using a language different from the primary language used in the project.
If your task is to manipulate static text files, it is likely that you won’t be constrained by the language(s) used by others on the same project. However, be aware of issues such as differences in newline characters (sometimes called line terminators), which differ between operating systems. If your regular expression patterns depend on a particular type of newline character, and you are using text files from a different operating system, you may run into unexpected and undesired behavior.
Consider Sensitivity and Specificity
The concepts of sensitivity and specificity are explained in detail in Chapter 8. However, they will be briefly described here.
34