So What's A $#!%% Regular Expression, Anyway?!

Introduction

Ask any relatively-experienced *NIX user to list his top ten favorite things about the operating system, and you’re almost certain to hear him mutter, somewhere between “99% uptime” and “remote system reboots”, the phrase “regular expressions”.

Ask any relatively-experienced *NIX user to list the ten things he hates most about the operating system, and somewhere between “zombie processes” and “installation”, he’s sure to spit out the phrase “regular expressions”.

It’s precisely this complex love-hate equation that spawned the idea for a tutorial on regular expressions - surely, went the reasoning, something that induced such strong emotions in normally hard-headed NIX administrators was worthy of investigation. And so, regardless of whether you’re new to regular expressions, or an old hand at putting them together, the next few pages should help you resolve your conflicted feelings on the subject. Hey - it *is cheaper than therapy…

And First There Was Love…

Regular expressions, also known as “regex” by the geek community, are a powerful tool used in pattern-matching and substitution. They are commonly associated with almost all *NIX-based tools, including editors like vi, scripting languages like Perl and PHP, and shell programs like awk and sed. You’ll even find them in client-side scripting languages like JavaScript - kinda like Madonna, their popularity cuts across languages and territorial boundaries…

A regular expression lets you build patterns using a set of special characters; these patterns can then be compared with text in a file, data entered into an application, or input from a form filled up by users on a Web site. Depending on whether or not there’s a match, appropriate action can be taken, and appropriate program code executed.

For example, one of the most common applications of regular expressions is to check whether or not a user’s email address, as entered into an online form, is in the correct format; if it is, the form is processed, whereas if it’s not, a warning message pops up asking the user to correct the error. Regular expressions thus play an important role in the decision-making routines of Web applications - although, as you’ll see, they can also be used to great effect in complex find-and-replace operations.

A regular expression usually looks something like this:

/love/

All this does is match the pattern “love” in the text it’s applied to. Like many other things in life, it’s simpler to get your mind around the pattern than the concept - but then, that’s neither here nor there…

How about something a little more complex? Try this:

/fo+/

This would match the words “fool”, “footsie” and “four-seater”. And although it’s a pretty silly example, you have to admit that there’s truth to it - after all, who but fools in love would play footsie in a four-seater?

The “+” that you see above is the first of what are called “meta-characters” - these are characters that have a special meaning when used within a pattern. The “+” metacharacter is used to match one or more occurrence of the preceding character - in the example above, the letter “f” followed by one or more occurrence of the letter “o”.

Similar to the “+” meta-character, we have “*” and “?” - these are used to match zero or more occurrences of the preceding character, and zero or one occurrence of the preceding character, respectively. So,

/eg*/

would match “easy”, “egocentric” and “egg”

while

/Wil?/

would match “Winnie”, “Wimpy” “Wilson” and “William”, though not “Wendy” or “Wolf”.

In case all this seems a little too imprecise, you can also specify a range for the number of matches. For example, the regular expression

/jim{2,6}/

would match “jimmy” and “jimmmmmy!”, but not “jim”. The numbers in the curly braces represent the lower and upper values of the range to match; you can leave out the upper limit for an open-ended range match.

Of Carrots, Bombshells And Four-Figure Incomes

Now that you’ve got the basics down, how about taking it to the next level? It’s also possible to search for white space, numbers and alphabetic characters with a regular expression - and here’s the merry gang of meta-characters that will help you do just that:

\s = used to match a single white space character, including tabs and newline characters

\S = used to match everything that is not a white space character

\d = used to match numbers from 0 to 9

\w = used to match letters, numbers and underscores

\W = used to match anything that does not match with \w

. = used to match everything except the newline character

Now, you’re probably thinking, “Hey, that’s great - but what does it all mean?!”. Well, suppose you wanted to find all the white space in a document…

/\s+/

Easy, isn’t it? If you’re looking only for numbers, try

/\d/

So, if you had a complex financial spreadsheet in front of you, and you wanted to quickly find all amounts of a thousand dollars or more, you could use

/\d000/

How about limiting your search to the beginning or end of a string? Well, that’s why we have “pattern anchors” - these simply tie your regular expression to either the first or last character of the string, and come in very useful when you’re looking for a way to filter through a mass of matches.

There are two basic pattern anchors - the first one is represented by a caret [^], and is used to indicate that the expression should be matched only at the beginning of the string that it is applied to. For example, the expression

/^hell/

will return a match only if it finds a word beginning with “hell” - “hello” and “hellhound”, but not “shell”.

And similarly, to match the end of a string, there’s the “$” pattern anchor. So

/ar$/

would match “scar”, “car” and “bar”, though not “art”, “army” or “arrow”.

There’s also a simpler way to add pattern anchors to your expression - the \b meta-character. This is used to check that the regex matches the boundary of a string, and it can be placed either at the beginning or end of the pattern to be matched - like this:

/\bbom/

This would match both “bombay” and “bombshell”, while

/man\b/

would match “human”, “woman” and “man”, though not “manitou” or “mannequin”. And the converse of this is \B, which matches everywhere but at the boundaries of a string.

Ranging Far And Wide…

Just as you can specify a range for the number of characters to be matched, you can also specify a range of characters. For example, the range

/[A-Z]/

would match a single instance of all upper-case alphabetic characters, while

/[a-z]/

would match all lowercase letters, and

/[0-9]/

would match all numbers between 0 and 9.

Using these three ranges, it’s pretty easy to create a regular expression to match an alphanumeric field.

/([a-z][A-Z][0-9])+/

would match a string that was purely alphanumeric in nature, like “aB0” – although not “abc”. Note the parentheses around the patterns - contrary to what you might think, these are not there purely to confuse you; they come in handy when grouping sections of a regular expression together.

/to|too|2/

would match any one of the three strings “to”, “too” and “2”. As you can imagine, this comes in pretty useful when building expressions that have many possible variants.

You can also invert the regular sense of a regular expression with the negation operator, represented by a caret [^] - so the pattern

/[^A-C]/

would match everything but that which appears in the expression - namely, everything except the letters “A”, “B” and “C”. Note how the caret, when used in a bracketed expression, is used to invert the match; this behaviour is different from when it is used outside a bracketed expression, where it serves as a pattern anchor.

And finally, one important thing to remember - should you decide, for reasons best known to you and your mental health specialist, to add any of the meta-characters described above to your pattern and explicitly match them, you need to “escape” then with a back slash []. So, the pattern

/Th\*/

would match “Th*” but not “The” - the * ensures that the asterisk is matched as a literal character, not a meta-character.

How To Say “Ummmm….” In Three Different Languages

Now that we’ve got all that out of the way, let’s take a closer look at some examples of how regular expressions are used in Perl, PHP and JavaScript. In Perl, for example, you can perform some pretty advanced pattern matching using both the rules you’ve already learnt, and some Perl-specific additions.

A pattern-matching command in Perl usually looks like this:

operator / regular-expression / string-to-replace / modifiers

Let’s take a closer look at each of these components.

The operator can either be an “m” or an “s”, depending on the purpose of the regular expression -“m” is used for “match” operations only, while “s” is used for “substitution” operations.

The regular expression is the pattern that is to be matched. This pattern can be constructed using a variety of characters, meta-characters and pattern anchors.

The string to replace is…well, the string to be replaced in a find-and-replace operation. Yeah, every once in a while, we slip you an easy one.

Finally, the modifiers are used to control the manner in which a particular regex is applied. There are a whole bunch of modifiers, some of them with pretty exotic names; unfortunately, none of them are single, or interested in going out to dinner with you.

So, the statement

s/love/lust/

would replace the first occurrence of the word “love” with “lust”. And if you wanted to perform a global search-and-replace operation, you’d use the “g” modifier, like this

s/love/lust/g

And they say romance is dead!

You can also use case-insensitive pattern matching - simply add the “i” modifier, as in the following example, and watch in awe as Perl matches “jewel”, “Jewel” and “JEWEL”.

m/JewEL/i

In Perl, all interaction with regular expressions takes place via an equality operator, represented by =~; this is used as follows.

$flag =~ m/abc/

$flag returns true if $flag contains “abc”

$flag =~ s/abc/ABC/

replaces abc in the variable $flag with ABC

And here’s an example of a simple Perl program which asks for your email address, and compares it with a regex to verify whether or not it’s in the correct format.

#!/usr/bin/perl

# get input
print "So what's your email address, anyway?\n";
$email = <STDIN>;
chomp($email);

# match and display result
if($email =~ /^([a-zA-Z0-9_-])+@([a-zA-Z0-9_-])+(\.[a-zA-Z0-9_-])+/)
        {
     print("Ummmmm....that sounds good!\n");
        }
else
        {
     print("Hey - who do you think you're kidding?\n");
        }

As you can see, the most important part of this program is the regular expression – it’s been dissected below:

^([a-zA-Z0-9_-])+@([a-zA-Z0-9_-])+(\.[a-zA-Z0-9_-])+

The first part

^([a-zA-Z0-9_-])

matches the username part of the email address – this could be either a number, a character, or a combination of both.

This is followed by an @ symbol, which is followed by the domain part of the address; this could again include letters or numbers, and uses a period as a delimiter - not our usage of an “escaped” period and the “+” meta-character to represent these conditions in the second half of the expression

([a-zA-Z0-9_-])+(\.[a-zA-Z0-9_-])+

Obviously, this is simply an illustrative example – if you’re planning to use it on your Web site, you need to refine it a bit. For example, the script above won’t accept email addresses of the form [email protected] – although such addresses are also pretty common on the Web. You have been warned!

If you prefer PHP to Perl, you need to use the ereg() function for all pattern matching operations – this usually takes the format

ereg(pattern, string)

where “pattern” is the pattern to be matched, and “string” is the character string to be searched for the pattern. The next example should illustrate this a little more clearly:

<?php

if (ereg("^([a-zA-Z0-9_-])+@([a-zA-Z0-9_-])+(\.[a-zA-Z0-9_-])+", $email)) {
    echo "Ummmmm....that sounds good!";
} else {
        echo "Hey - who do you think you're kidding?";
    }

?>

And finally, JavaScript. JavaScript 1.2 comes with a powerful RegExp() object, which can be used to match patterns in strings and variables. The important thing here is the test() method, which searches for a pattern in a string or variable, and returns either true or false – it’s illustrated in the example below.

<html>
<head>
<script language="Javascript1.2">
<!-- start hiding

function verifyAddress(obj) {
// obtain form value into variable
var email = obj.email.value;
// define regex
var pattern = /^([a-zA-Z0-9_-])+@([a-zA-Z0-9_-])+(\.[a-zA-Z0-9_-])+/;

// test for pattern
flag = pattern.test(email);

	if(flag)
	{
	alert("Ummmmm....that sounds good!");
	return true;
	}
	else
	{
	alert("Hey - who do you think you're kidding?");
	return false;
	}

}
// stop hiding -->
</script>

</head>

<body>

<form onSubmit="return verifyAddress(this)&quot&gt;

<input name="email" type="text">

<input type="submit">

</form>

</body>

 </html>

Obviously, there’s a whole lot more that you can do with regular expressions – checking email addresses is just the tip of the iceberg. You can use regular expressions to validate phone numbers, currency figures, Web site URLs, and a whole lot more - all you need is a little bit of creativity and patience, a few slices of leftover pizza…and a therapist who cares.

Note: All program code and examples in this article have been tested on Linux 2.2.13/i386 with Perl 5.004, PHP 3.0.9 and Javascript 1.2. YMMV.