One of the most important new features in Perl 5 was the capability to manage complicated data structures like multidimensional arrays and nested hashes. To enable these, Perl 5 introduced a feature called references, and using references is the key to managing complicated, structured data in Perl. Unfortunately, there's a lot of funny syntax to learn, and the manual is not as clear in this area as it usually is. The manual is quite complete, and a lot of people find that a problem, because it can be hard to tell what is important and what isn't.
Fortunately, you only need to know 10% of what's in the manual to get 90% of the benefit. This article is going to show you that 10%.
WHO NEEDS COMPLICATED DATA STRUCTURES?
One problem that came up all the time in Perl 4 was how to represent a hash whose values were lists. Perl 4 had hashes, of course, but the values had to be scalars; they couldn't be lists.
Why would you want a hash of lists? Let's take a simple example: You have a file of city and state names, like this:
Chicago, Illinois New York, New York Albany, New York Springfield, Illinois Trenton, New Jersey Evanston, Illinois
and you want to produce an output like this, with each state mentioned once, and then an alphabetical list of the cities in that state:
Illinois: Chicago, Evanston, Springfield. New Jersey: Trenton New York: Albany, New York.
The natural way to do this is with a hash whose keys are state names. Associated with each state name is a list of the cities in that state. Whenever you read a line of input, you split it into a state and a city, look up the list of cities already known to be in that state, and append the new city to the list. When you're done reading the input, you can iterate over the hash with keys or each, sorting each list of cities before you print it out.
If hash values can't be lists, you lose. In Perl 4, hash values can't be lists; they can only be numbers or strings. You lose. You'd probably have to combine all the cities into a single string somehow, and then when time came to write the output, you'd have to break the string into a list, sort the list, and turn it back into a string. This is messy and error-prone. And it's frustrating, because Perl already has perfectly good lists that would solve the problem - if only you could use them.
THE SOLUTION
Unfortunately, by the time Perl 5 rolled around, we were already stuck with this design: Hash values must be scalars. The solution is references.
A reference is a scalar value that refers to an entire array or an entire hash (or just about anything else). You're already familiar with English names, which are a kind of reference. Think of the President: a messy, inconvenient bag of blood and bones. But to talk about him, or to represent him in a computer program, all you need is the easy, convenient scalar string "Bill Clinton".
References in Perl are like names for arrays and hashes. They're Perl's private, internal names, so you can be sure they're unambiguous. Unlike common names like "Bill Clinton", a reference refers to only one thing, and you always know what it refers to. If you have a reference to an array, you can recover the entire array from it. If you have a reference to a hash, you can recover the entire hash. But the reference is still an easy, compact scalar value.
You can't have a hash whose values are arrays; hash values can only be scalars. We're stuck with that. But a single reference can refer to an entire array, and references are scalars. So you can have a hash of references to arrays, and it'll act a lot like a hash of arrays, and it'll be just as useful as a hash of arrays.
We'll come back to this city-state problem later, after we've seen some syntax for managing references. There are only two ways to make a reference, and only two ways to use it once you do.
MAKING REFERENCES
Make Rule 1. If you put a \ in front of a variable, you get a reference to that variable.
$aref = \@array; # $aref now holds a reference to @array $href = \%hash; # $href now holds a reference to %hash
Once the reference is stored in a variable like $aref or $href, you can copy it or store it just the same as any other scalar value:
$xy = $aref; # $xy now holds a reference to @array $p[3] = $href; # $p[3] now holds a reference to %hash $z = $p[3]; # $z now holds a reference to %hash
These examples show how to make references to variables with names. Sometimes you want to make an array or a hash that doesn't have a name. This is analogous to the way you like to be able to print the string "\n" or the number 80 without having to store it in a variable first.
Make Rule 2. [ ITEMS ] makes a new, anonymous array, and returns a reference to that array. { ITEMS } makes a new, anonymous hash. and returns a reference to that hash. Remember: Square brackets are for arrays, curly brackets for hashes. The kind of bracket you use is the same as you'd use if you were subscripting the object.
$aref = [ 1, "foo", undef, 13 ]; # $aref now holds a reference to an array $href = { APR => 4, AUG => 8 }; # $href now holds a reference to a hash
The references you get from rule 2 are the same kind of references that you get from rule 1:
$aref = [ 1, 2, 3 ]; @array = (1, 2, 3); $aref = \@array;
The first line is an abbreviation for the following two lines, except that it doesn't create the superfluous array variable @array.
USING REFERENCES
What can you do with a reference once you have it? It's a scalar value, and we've seen that you can store it as a scalar and get it back again just like any scalar. There are just two more ways to use it:
Use Rule 1. If $aref contains a reference to an array, then you can put {$aref} anywhere you would normally put the name of an array. In the example above, you can say @{$aref} instead of @array.
Let's see some examples of that:
@a @{$aref} # An array reverse @a reverse @{$aref} # Reverse the array $a[3] ${$aref}[3] # An element of the array $a[3] = 17; ${$aref}[3] = 17 # Assigning an element
On each line are two expressions that do the same thing. The left-hand versions operate on the array @a, and the right-hand versions operate on the array that is referred to by $aref, but once they find the array they're operating on, they do the same things to the arrays.
Using a hash reference is exactly the same:
%h %{$href} # A hash
keys %h keys %{$href} # Get the hash keys
$h{red} ${$href}{red} # A hash element
$h{red} = 17 ${$href}{red} = 17 # Assigning an element
Most often, when you have an array or a hash, you want to set or get a single element. ${$aref}[3] and ${$href}{red} have too much punctuation, so Perl lets you abbreviate.
Use Rule 2. ${$aref}[3] is too hard to read, so you can write $aref->[3] instead. ${$href}{red} is also too hard to read, so you can write $href->{red} instead.
If $aref holds a reference to an array, then $aref->[3] is the fourth element of that array. Don't confuse this with $aref[3], which is the fourth element of a totally different array, one deceptively named @aref. $aref and @aref are unrelated in the same way that $item and @item are unrelated.
Similarly, $href->{red} is part of the hash referred to by the scalar variable $href, perhaps even one with no name. $href{red} is part of the deceptively named %href hash. It's easy to forget to put in the ->, and if you do forget, you'll get bizarre results when your program gets array and hash elements out of totally unexpected hashes and arrays that weren't the ones you wanted to use. -w and use strict find these errors for you.
AN EXAMPLE
Let's see a quick example of how all this is useful. First, remember that [1, 2, 3] makes an anonymous array containing (1, 2, 3), and gives you a reference to that array.
Now think about
@a = ( [1, 2, 3],
[4, 5, 6],
[7, 8, 9] );
@a is an array with three elements, and each one is a reference to another array.
$a[1] is one of these references. It refers to an array, the array containing (4, 5, 6), and because it is a reference to an array, Use Rule 2 says that we can write $a[1]->[2] to retrieve the third element from that array. $a[1]->[2] is the 6. Similarly, $a[0]->[1] is the 2. What we have here is like a two-dimensional array; you can write $a[ROW]->[COLUMN] to get or set the element in any row and any column of the array.
The notation still looks a little cumbersome, so there is one more abbreviation:
The Arrow Rule. Between two subscripts, the arrow is optional.
Instead of $a[1]->[2], we can write $a[1][2]. Instead of $a[0]->[1], we can write $a[0][1].
Now it really looks like a two-dimensional array!
You can see why the arrows are important. Without them, we would have had to write ${$a[1]}[2] instead of $a[1][2]. For three-dimensional arrays, they let us write $x[2][3][5] instead of the unreadable ${${$x[2]}[3]}[5].
SOLUTION
Here's the answer to the problem I posed the the beginning of the article of reformatting a file of city and state names:
1 while (<>) {
2 chomp;
3 my ($city, $state) = split /, /;
4 push @{$table{$state}}, $city;
5 }
6
7 foreach $state (sort keys %table) {
8 print "$state: ";
9 my @cities = @{$table{$state}};
10 print join ', ', sort @cities;
11 print ".\n";
12 }
The program has two pieces: Lines 1–5 read the input and build a data structure, and lines 7–12 analyze the data and print out the report.
In the first piece, line 4 is the important one. We're going to have a hash, %table, whose keys are state names, and whose values are (references to) arrays of city names. After acquiring a city and state name, the program looks up $table{$state}, which holds (a reference to) the list of cities seen in that state so far. Line 4 is analogous to
push @array, $city;except that '@array' has been replaced by @{$table{$state}} . The push adds a city name to the end of the referred-to array.
In the second part, line 9 is the important one. Again, $table{$state} is (a reference to) the list of cities in the state, so we can recover the original list, and copy it into the array @cities, by using @{$table{$state}}. Line 9 is analogous to
@cities = @array;except that the name 'array' has been replaced by the reference {$table{$state}} . The '@' tells Perl to get the entire array.
The rest of the program is just familiar uses of chomp, split, sort, and print, and doesn't involve references at all.
There's one fine point I skipped. Suppose the program has just read the first line in its input that mentions Ohio. It's on line 4, $state is 'Ohio', and $city is 'Cleveland'. Since this is the first city in Ohio, $table{$state} is undefined - in fact there isn't an 'Ohio' key in %table at all. What does line 4 do here?
4 push @{$table{$state}}, $city;
This is Perl, so it does exactly the right thing. It sees that you want to push 'Cleveland' onto an array that doesn't exist, so it helpfully makes a new, empty, anonymous array for you, installs it in the table, and then pushes 'Cleveland' onto it.
THE REST
I promised to give you 90% of the benefit with 10% of the details, and that means I left out 90% of the details. Now that you have an overview of the important parts, it should be easier to read the perlref manual page, which discusses 100% of the details.
Some highlights of the perlref documentation bundled with Perl: You can create references to anything, including scalars, functions, and other references.
- In Use Rule 1, you can often omit the curly braces. For example, @$aref is the same as @{$aref}, and $$aref[1] is the same as ${$aref}[1]. I usually leave the curly braces in.
- To see if a variable contains a reference, use the ref function. It returns true if its argument is a reference. Actually it's a little better than that: It returns 'HASH' for hash references and 'ARRAY' for array references.
- If you try to use a reference like a string, you get strings like ARRAY(0x80f5dec) or HASH(0x826afc0). If you ever see a string that looks like this, you'll know you printed a reference by mistake.
A side effect of this representation is that you can use eq to see whether two references refer to the same thing. (You can also do that with ==, which is faster.)
- You can use a string as a kind of reference. If you use the string 'foo' as an array reference, it's taken to be a reference to @foo. This is called a "soft reference" or "symbolic reference".
You might prefer to proceed to the perllol documentation instead of the perlref documentation; perllol discusses lists of lists and multidimensional arrays in detail. After that, you should move on to the perldsc documentation; it's a Data Structure Cookbook that shows recipes for using and printing out arrays of hashes, hashes of arrays, and other kinds of data.
IN SUMMARY
Everyone needs compound data structures, and in Perl the way you get them is with references. There are four important rules for managing references: Two for making references and two for using them. Once you know these rules you can do most of the important things you need to do with references.
Mark-Jason Dominus has been programming in Perl since 1993. He is the chief programmer for the Plover Systems company in Philadelphia. He was recently invited to give a talk at the 1998 O'Reilly Perl Conference, on 'Perl Hardware Store: Great Tools You Didn't Know You Needed', or maybe the title is 'Stop Programming in FORTRAN, Already!'. He likes to get email, so send him some at mjd-tpj@plover.com.