Regular Expressions – A Quick Guide

Read it out loud: “Regular Expressions.” It chimes like the name of a horror movie Alfred Hitchcock would be proud to tag his name to. But it shouldn’t. Don’t be frightened by the name. It is, after all, only a name. Familiarize it. Shorten it to Regex or Regexp and give it a little hug as you tell it “We’re going to be lifelong buddies…”

What follows in this guide is an introduction to regular expressions and a reference table of regular expression metacharacters with examples for their usage. Unless you really need an introduction, jump straight ahead to the table because that is where you will learn most.

Getting to Know Regex

Regex is a Language

Like English, Regex makes it possible to convey instructions. In English, we can ask people to look left, to look right, to move objects around or to only carry out our instructions when the circumstances are right. We can do the same with computers using Regular Expressions.

Whereas English has hundreds of thousands of words, Regex has about 30. And they are all very easy to learn.

Like any language, Regex has ways to indicate where something is and where something isn’t.

When searching for a word such as “look” we can tell the regex engine to look behind the word and check that it has another word such as “forward” in front of it before it decides it has found a match for the specific word it is being asked to find.

Regex also has ways to specify whether a search is for a number, a letter, a whitespace character like a space or carriage return, a word or any grouping or combination of the aforementioned. It even has pronouns – it lets us specify groups that we can refer to in both the search and the replacement string.

Using regular expressions, we can search for the string “look” in the phrase “I am a forward looking guy who’s always on the lookout for fun.” and replace “look” with “fac” only when it is preceded by the string “forward ” to make the phrase read “I am a forward facing guy who’s always on the lookout for fun.” Try doing the same find & replace in a text editor without regular expressions.

The regex for performing our hypothetical exchange in would search for #(?<=forward )look(ing)# and replace it with “fac\1”. You can check it here.

The Regex Alphabet and Words

Regex is a written language that uses its own alphabet of special characters to form its verbs and nouns. These verbs and nouns are properly called metacharacters or operators and are written with the following characters:

Western numerals (0-9) and letters of the Latin alphabet (a-z) are also used in conjunction with those characters.

A typical regex consists of a search pattern and a replacement pattern expressed with a combination of metacharacters and non metacharacters.

Metacharacters tell a regex engine what to look for, where to look, when to turn a blind eye to what it is looking at and what to do once it’s matched its search expression to something.

Non metacharacters are the symbols, digits and words being searched for. They have no special meaning to the regex engine and represent exactly what they are – regular text strings. Non metacharacters are usually called literal characters or literals.

We can tell a regex engine to treat a metacharacter as a literal character by placing a backslash (\) in front of it. This act is known as “commenting out”.

Compare the following two regex search patterns:

looking (?=good)

looking \(\?\=good\)

There is no need to understand them. They only serve to illustrate how metacharacters are switched off.

The first regex contains the literal strings “looking ” and “good”. The characters “(, ?, = and )” all have special significance to a regex engine. The parentheses “()” create a grouping and the “?=” immediately after the opening parenthesis instructs the regex engine to look ahead of the string “looking ” and check it is followed by the string “good” before the regex engine determines it has discovered a match.

The second regex tells the regex engine to look for the literal string “looking (?=good)”. The backslashes comment out the special metacharacters “(?=)” to reduce them to literals. To search for a literal backslash, you would need to comment it out too e.g \\.

I mentioned earlier that different regex engines operate slightly differently. The Regular Expression examples and metacharacters shown in the table below are for Perl Compatible Regular Expressions (PCRE’s) as used by PHP’s preg_match () function.

I recommend you use a text editor or an online testing tool for testing your regular expressions.

I recommend the following Regex capable text editors:

• Kate (Linux)
• PowerGrep (Windows)
• EditPad Pro (Windows)
• RegEx Buddy (Windows)
• Notepad++ (Windows)

These three online regex testing tools are also suitable for most needs:

• Regex Tester 2.0
• PHP Regular Expression Tester
• Regex Tool

I prefer to use Kate because it handles most of what I need a regex engine to do within a text editor. You can read more about Kate’s regular expression features here.

PowerGrep (free download) and EditPadPro (free download) are two Windows programs that handle lookbehinds and lookaheads very nicely. EditPadPro is ideal for simple RegEx tasks. PowerGrep is a little overwhelming to begin with but it comes with a very good tutorial to regular expressions.

Tips for Writing Regular Expressions

There are four basic steps to writing a regular expression:

1. know what you want to change
2. know how you want to change it
3. write the regex that finds it
4. write the regex that changes it

Let’s look at those steps in more detail.

Know what you want to change

Regexes don’t just replace strings. They let us play with them. They allow us to be (almost) as precise or as general as circumstances permit and warrant about what we are looking for. We can find a letter, number, symbol, word, sentence, paragraph or any other type of string. In many cases, we only need to know the characteristics of what we’re looking for.

Questions to look at when deciding what to replace or change include:

• Are you looking for something specific like a word or phrase?
• Are you looking for something that changes like a URL with variable anchor texts?
• Will changing your target string suit all contexts surrounding its every occurrence?
• Are there common markers surrounding the string you want to change?
• Can any common markers be used to ensure the string to be changed is only changed when those markers are present?
• Are you looking for more than one string?

Look for anything common to either the string being changed or its surroundings where it needs to be changed. Unintended consequences are easy to create but not always easy to fix so, when possible, try to be precise about what you want the regex engine find.

Know what your replacement is

Regular expressions are not just about finding and replacing text. They can also be used to re-organize text as-well-as to do things to text surrounding the search string.

Questions to consider include:

• Do you want to overwrite the search string with the replacement?
• If the search pattern looks for more than one string, do you want to swap them round?
• If the search pattern looks for more than one string, do you want to edit what lies between them?

The key consideration is whether the search string, replacement string or both strings are start and end markers for what you want to achieve or whether one is to replace the other.

Write the regular expression

A typical regular expression consists of a search pattern and a replacement pattern.

The search pattern can be a literal string or a mixture of metacharacters and literal characters. The replacement pattern usually consists of literals and/or references to groupings specified within the search string.

In writing either the search or replacement pattern it helps to know that they are processed from left to right by regex engines.

When you intend to search for one string and replace it with another string, simply state the search string as the search pattern and the replacement string as the replacement pattern. Take care to comment out (i.e backslash) metacharacters to convert them to literals.

There are two types of parentheses: capturing and non-capturing.

Capturing parentheses create a grouping that stores the matched pattern in the regex engine’s memory. They consist of a string encased in parentheses. For example (one string) (two strings).

When you intend to place a string contained within a “replacement” pattern next to the string contained within a search pattern, put the search pattern in Capturing Parentheses.

Each capturing parentheses pair is given a name. That name is a number. That number is the count of opening parentheses from the search pattern’s left side to its right side. To refer to that grouping (to recall its content), place a backslash in front of its number. For example If the search pattern is #(one string)# then the text “one string” may be recalled by writing “\1”.

Non-capturing parentheses do not create backreferences – the strings mentioned within them are not recorded unless capturing parentheses are used around the strings to be recorded. For example, #(?=one)string# does not create a backreference i.e “\1” does not recall the value “one”; #(?=(one))string# does create a backreference i.e “\1” does recall the value “one”.

Non-capturing parentheses have a question mark (?) immediately after the opening parentheses. That is to say that capturing parentheses never begin with a question mark.

Here’s a longer example

If a search pattern was written “(one)(two) then “\1” would refer to a first set of parentheses; “\2” would refer to a second set of parentheses; and a replacement pattern of “\2 times \1” would return “Two times One”.

That example also demonstrates how to swap strings around: create groupings in the search pattern and change their name order within the replacement pattern.

More complicated example

Do not worry about it looking complex. It is complex. The reference table below here uses less complex examples for each of the metacharacters used in this example. Glide over this example if you need to and return to it once you have reviewed the reference table. It will then make more sense.

Let’s look at the characters in the following regex which matches variations of “This is my [number] apple pie [time]. It is yummy!”.

Here’s the regex:

#[Tt]his is my (\d*(st|nd|rd|th)?) apple (pie)? (today|this ((mor|eve)ning|afternoon))\. It is ([Yy]ummy.?.?)+#

Here’s what those characters do:

The opening and closing delimiter is an hash sign (#). It could easily be a forward slash or any other symbol that does not occur within the regex.

[Tt] is a character class that specifies that either “T” or “t” are acceptable matches at the point where the character class occurs.

All the characters “Tt his is my st nd rd th apple pie today this mor eve ning afternoon It is Yyummy” are literals

The metacharacter “\d” represents a digit from 0 to 9. It is also the only token, or argument, that defines the value of the asterisk to its right

The asterisk (*) is a metacharacter that says the digit represented by “\d” is optional and can be any length from zero digits to an infinite number of digits.

The grouping “(st|nd|rd|th)” says that any of “st” or “nd” or “rd” or “th” are valid matches. The curly brackets denote the literals are to be considered a group token and any metacharacter that follows it applies to the complete group. The pipe symbol, |, is an “or” operator; it says that whatever to the left of it and whatever is to its right are both valid matches at the point where it occurs.

The question mark (?) to the right of the grouping “(st|nd|rd|th)?” tells the regex engine to match any of the grouped literals either never or only once i.e they are optional.

Jumping ahead to the backslash (\) and the dot (.). The dot has a special meaning to regex engines: it represents any single character except a newline. It may be a a full-stop, a letter of the alphabet, a digit of a number series or a non-alphanumeric symbol. The backslash switches off the dot’s metacharacter status and instructs the regex engine to treat it as a literal dot. “\.” tells the regex engine to match a full-stop.

The “.?.?” in the grouping “([Yy]ummy.?.?)” instructs that the token “[Yy]ummy” may be followed by any two singular characters or by nothing at all for the regex to match.

The plus in “([Yy]ummy.?.?)+” specifies that the token “[Yy]ummy.?.?” must be in the statement at least once and it may even be repeated several times for the regex to find a match.

Any of the following statements will be matched by the regex:

• This is my 1st apple pie today. It is yummy!
• This is my 1st apple pie today. It is yummy! Yummy! Yummy!
• This is my 21st apple this evening. It is yummy.
• This is my 2nd apple pie this morning. It is yummy..

The regex engine stores any matched grouping encased in “capturing” parentheses. The match may be referred to in the replacement string by a backslash and a number e.g “\1” or “\3”. To find the back reference’s group number, count the number of opening parentheses within the expression from left to right. Ignore non capturing parentheses. See the table below to learn which parentheses are capturing and non capturing.

The first grouping in our example regex is “(\d*(st|nd|rd|th)?)”; The second grouping is (st|nd|rd|th); and the third grouping is (pie). Each of these groupings may be referred to in either the search or replacement pattern by the metacharacters “\1”, “\2” and “\3”.

Example replacement patterns for the statement “This is my 1st apple pie this evening. It is yummy.” are

“Mmmmmm!!!” would replace the matched string with “Mmmmmm!!!”

“Mmmmmm!!! My \1 \3.” would replace it with “Mmmmmm!!! My 1st pie”

“Mmmmmm!!! My \1 \3 \4.” would replace it with “Mmmmmm!!! My 1st pie this evening”

Notice that the dot at the end of the replacement pattern is treated as a literal. Fewer characters are treated as metacharacters in the replacement patterns than in search patterns.

Also notice that the reference to grouping 4, “\4”, treats the nested fifth group within group 4 as though it were not a separate group. If the nested group was referred to in the replacement pattern with “\5” then it would return the value stored when group 5 is matched, in this case, it is the string “evening”.

If you need a more complete guide to regular expressions, one that is easy to understand and easy to read, then I recommend the tutorial that comes with PowerGrep.

Please post suggestions, questions and supportive corrections at the bottom of this post.

Quick Reference: Metacharacters and Examples

The left hand column of the table lists metacharacters. Related metacharacters are listed in successive rows. The examples shown in the right hand column use an hash symbol (#) as the delimiter for the search pattern.

Sharing is caring!