Background of the Study
Perl stands for Practical Extraction and Report Language, or Pathologically Eclectic Rubbish Lister. Perl is a very useful programming language that is standard on almost all Unix systems. Its semantics are a lot like C, but its text processing and regular expression matching features are far more powerful. Perl is the ideal tool for the lazy programmer; almost any simple task can be performed using a Perl script that can be written in under 10 minutes. Perl is particularly powerful for scanning large quantities of text and doing pattern matching. Perl is also very useful as a scripting language for web pages (as CGI). Invocation and #! To create a Perl script, simply put your program into a text file. You can invoke it several ways:
1) Execute it from the command prompt with perl -w
2) Put #!/usr/local/bin/perl –w at the top of your script, make the script executable, and invoke it simply by typing the script name at the command prompt. (You still have to type the path to the script, or if the script is in the current directory, type ./scriptname.
Tip: You’ll need to make the permissions on the script executable to run it this way; to do so type: chmod 700 scriptname
A Perl script is a sequence of statements.
Perl recognizes two different types of values: integers and strings.1 Integer literals are written as one expects, 1, 2, -1, . . .. A string is a sequence of characters. String literals can be written in two distinct ways
Significance of the Study
1.1 File system I/O
standard files
Perl provides access to the standard files: STDIN, STDOUT, and STDERR.
STDIN is accessed through the angle brackets (<>) operator. When placed in a scalar context, the operator returns the next line; when place in an array context, it returns the entire file, one line per item in the array.
Examples:
$a = <STDIN>;# returns next line in file
@a = <STDIN>; # returns entire file
STDOUT is the default file accessed through a print statement.
STDERR is the file used by the system to which it writes error messages; it is usually mapped to the terminal display.
open file
Files are accessed within a Perl program through filehandles which are bound to filenames within the UNIX file system through an open statement. By convention, Perl filehandle names are written in all uppercase, to differentiate them from keywords and function names.
Form:
open (FILEHANDLE, “filename”);
Example:
open (INPUT, “index.html”);
In the above, the file is opened for read access. It may also be opened for write access and for update. The difference between the two is that write replaces the file contents, whereas update appends new data to the end of the current contents. These two options are indicated by appending either a single or a double greater than (>) symbol to the file name as a prefix:
Form:
open (FILEHANDLE, “>filename”); # write access
open (FILEHANDLE, “>>filename”); # update
Examples:
open (INPUT, “>index.html”);
open (INPUT, “>>index.html”);
Since Perl will continue operating regardless of whether the open was successful or not, you need to test the open statement. Like other Perl constructs, the open statement returns a true or false value, indicating success or failure. One convenient construct in which this value can be tested and appropriate response taken is with the logical or and die operators. die can be used to deliver a message to STDERR and terminate the Perl program. The following construct can be paraphrased: “open or die.”
Form:
open (FILEHANDLE, “filename”) || die “Message written to STDERR”;
Example:
open (INPUT, “index.html”) || die “Error opening file index.html “;
close file
Files are closed implicitly when another open is encountered. they may also be closed explicitly.
Form:
close (FILEHANDLE);
Example:
close (INPUT);
read file
The file, once opened and associated with a filehandle, can be read with the angle brackets operator (<>), which can be used in a variety of constructs.
Form:
<FILEHANDLE>
Example:
while (<INPUT>) { # read one line at a time until EOF
chop; # remove newline
print line = $_ \n”; # print line read using default scalar variable
}
write file
Once a file has been opened for either write or update access, data can be sent to that file through the print operator.
Form:
print FILEHANDLE (content);
Example:
print OUTPUT “$next \n”; # outputs contents of $next followed by newline char.
file tests
There are a number of circumstances where the actions taken by the Perl program should take into account attributes of the file, such as whether or not the file currently exists, whether or not it has content, etc. A number of tests can be performed on files through the dash (-) operator.
Form:
-SYMBOL # where SYMBOL is a single character designator
See a Perl text for the complete list; some of the more useful ones include the following:
Examples:
-r # readable
-w # writeable
-x # executable
-o # owned by user
-e # exists
-z # zero content
-s # nonzero content (size)
-f # plain file
-d # directory
-l # symbolic link
-T # text file
-B # binary file
-M # modification age
-A # access age
1.2 Database
The are several different DBM libraries, including two open source packages: Gnu‘s GDBM package and Sleepycat‘s Berkeley DB package). GDBM and DB each can emulate the interfaces of the traditional packages, Old DBM and NDBM. It’s important to keep straight which interface to which version of which library you are talking about.
Virtually all Unix systems have some form of Old DBM and NDBM library installed. GDBM and DB can be installed on nearly all of them.
Generally the disk files produced by any two libraries are going to be incompatible with each other. The formats of many packages are not architecture independent either. The formats of these files also change pretty regularly from version to version. In some cases, using different interfaces to the same version of the same package will create slightly incompatible database on disk (e.g., the files will be named differently depending on which interface to GDBM you use.) Ideally, all programs that access the same database file should be running on the same architecture using the same interface to the same version of the same library.
Note that full installs of Redhat 6.1 include three different DBM libraries: GDBM, Berkeley DB Version 1, and Berkeley DB Version 2. Each of these three has different native interfaces and incompatible file format, all three of them can emulate Old DBM and NDBM, and Berkeley DB Version 2 can emulate Version 1. With the introduction of Berkeley DB Version 3, this is likely to get more complex. Figuring out which header file goes with which library is enough to stun an ox.
Using hash files for data can be just a bit perilous. I’ve had hashed databases suddenly become unreadable when an ISP upgraded the operating system on a server, inadvertantly bringing in a new, non-compatible version of a dbm package. This kind of thing can make maintaining a dbm file something of a headache in the long term.
Because of this you should be very careful about using DBM files for storing important data. They just aren’t very maintainable. One workaround for this is to use them only for indexing, and provide a tool that can regenerate the indexes if they are lost. Another workaround for databases that are read often and written rarely is to do something like what is done with the Unix password file, where the master file is a plain text file and the hashed file is generated from that. If all else fails, you should at least provide a tool to dump the hash file into an easily readable plain text file, and a matching tool to read the plain text file to recreate the hash file. A pair of such tools allows you to manually edit the database, or port it to a new system, by dumping it on the old system and loading it on the new.
Old DBM Library
Old DBM is often called just“DBM.” We call it Old DBM to distinguish it from NDBM (which is also sometimes just called “DBM” as in Apache’s mod_auth_dbm). It was developed at UC Berkeley.
Old DBM should probably be considered obsolete, and you should probably avoid using it.
Old DBM allows only one DBM database to be open at a time. Each DBM database consists of two files, one file has a .dir suffix and contains an index, the other holds all the data and has a .pag suffix. Data files are sparse and should not be copied carelessly (there are also problems putting such databases on filesystems that don’t support sparse files).
There are restrictions on the total size of a key/content pair (the maximum size is typically around 1018 bytes, but ranges from 512 to 4096 in different installations). The total size of all keys that hash to the same value must also be within that limit.
There is no automatic locking, so concurrently updating and reading is risky.
The header file dbm.h should be included.
The dbminit() call is used to open a database. It will not create a new database if none exists. You must manually create empty .dir and .pag files.
Data is saved with store() and retrieved with fetch(). Fetch() returns a pointer to static storage which is overwritten with each successive call. The dptr fields in the returned datum types do not necessarily point to word-aligned addresses, so it is unsafe to cast them to arbitrary data types and dereference them.
Some of the manual pages for Old DBM omit to mention the existance of a dbmclose() call. However, it always seems to exist.
Under IRIX, there is an alternate version with function names like dbminit64() that allows you to reference databases bigger than 2 gigabytes (only possible on machines with more than 32 bits).
To compile, you usually need to link with -ldbm. I don’t know if this is always needed.
Old DBM Library (GDBM Emulation)
The Gnu GDBM library provides a historic interface emulating Old DBM.
Old DBM’s key and content size limitations disappear in GDBM’s emulation.
The on-disk file format is incompatible with genuine Old DBM files. To initialize a database, you still need to manually create the .pag file. When an empty .pag file is opened it is initialized, any .dir file is removed, and a new .dir file is created which is just a link to the .pag. The .dir file is thereafter ignored, and the .pag file is a normal GDBM file. This trickery make the files look similar on disk, and ensures that the time-stamp on the .dir file is kept current. If you want progams using the Old DBM interface to the GDBM library to share files with progarms using the native interface to GDBM, then the programs using the native interface should pass gdbm_open() the filename including the .pag suffix, while the programs using the Old DBM interface should omit the .pag suffix.
The Old DBM interface to GDBM will always try to open files as a writer, meaning that they will be locked so no two processes can simultaneously access the same file. The exception is that if some process already has the file open as a reader (through a different interface) then processes using the Old DBM interface will open it as readers too (and be unable to write!).
Data returned by fetch() will actually be in dynamically allocated storage, but later fetch() calls will take care of freeing it for you, so you should mostly just pretend it is in static storage. The last fetched value will not be freed.
To use GDBM’s Old DBM interface, you need GDBM’s version of the dbm.h header file. Gnu’s install scripts don’t install this header file by default, so it is missing from many sites. By default the header is installed as dbm.h, but many glibc-2.1 systems, including Redhat 6.1, install it as gdbm/dbm.h.
To compile, you need to link with -lgdbm and sometimes -lgdbm_compat.
Old DBM Library (DB Emulation)
All versions of Berkeley DB include an interface emulating Old DBM.
Old DBM’s key and content size limitations disappear in DB’s emulation.
The on-disk file format is completely different from genuine Old DBM files and from GDBM format. The dbminit() call will create the database if it doesn’t exist. It will be a single file with a .db suffix added to the name and it be permitted with mode 600 (modified by umask).
The dbmclose() call seems to deallocate the “static” memory that values returned by fetch() are saved in, so you need to be sure to save those elsewhere before closing the database.
The DB emulation of Old DBM buffers writes to the database. You must call dbmclose() to ensure that all data is written out.
In Berkeley DB Version 2 and 3, the dbm.h file is eliminated. Instead you do
#define DB_DBM_HSEARCH 1
#include <db.h>
When compiling, you need to link with the DB library, usually -ldb.
In Berkeley DB Version 1, there is a dbm.h header file, just like the real thing, but in glibc-2.1 systems, including Redhat 6.1, it is installed as db1/dbm.h. It used to be normal to install Berkeley DB so you’d link with -ldb, but in glib-2.1 systems, where both version 1 and version 2 are installed, you need to link with -ldb1 to get the version 1 library.
NDBM Library
To maximize confusion with Old DBM, NDBM is sometimes refered to as “DBM”. Apache’s mod_auth_dbm module is an NDBM interface, not an Old DBM interface. I think NDBM first appeared in 2.10 BSD.
NDBM allows multiple databases to be open simultaneously, and handles very large database files respectably. However, there is still a limit on the total size of the key/content pairs that can be stored (this ranges from 1018 bytes to 4096 bytes). Like Old DBM, each database consists of two files, with .dir and .pag suffixes. Data files are sparse and should not be copied carelessly.
Under IRIX, there is an alternate version with function names like dbm_open64() that allows you to reference databases bigger than 2 gigabytes.
There is no automatic locking, so concurrently updating and reading is risky.
The header file ndbm.h should be included.
The dbm_open() call is used to open a database. Unlike Old DBM, NDBM can be told to automatically create the files if they don’t already exist. On Solaris (at least) opening the database write-only is not supported. If the O_WRONLY flag is given, it actually opens it read-write. I recommend never using O_WRONLY (see below for other wierdnesses with this).
Data is saved with dbm_store() and retrieved with dbm_fetch(). DBM_fetch() returns a pointer to static storage which is overwritten with each successive call. The dptr fields in the returned datum types do not necessarily point to word-aligned addresses, so it is unsafe to cast them to arbitrary data types and dereference them.
To compile, you usually need to link with -lndbm.
NDBM Library (GDBM Emulation)
The Gnu GDBM library provides a historic interface emulating NDBM. NDBM’s size limitations disappear in GDBM’s emulation.
The on-disk file format is incompatible with genuine NDBM files, but the files have the same suffixes. When an empty .pag file is open it is initialized, any .dir file is removed, and a new .dir file is created which is just a link to the .pag. The .dir file is thereafter ignored, and the .pag file is a normal GDBM file.
When opening a database, the mode flags passed to dbm_open() are mapped to gdbm_open() flags as follows:
| dbm_open() | gdbm_open() |
| O_RDONLY | GDBM_READER |
| O_RDWR | O_CREAT | GDBM_WRCREAT |
| anything | O_TRUNC | GDBM_NEWDB |
| anything else | GDBM_WRITER |
Warning: calling GDBM’s emulation of dbm_open() with O_WRONLY|O_CREAT falls into the “anything else” catagory and thus will not create the database. You need to do O_RDWR|O_CREAT to create a database.
Data returned by dbm_fetch() will actually be in dynamically allocated storage, but later dbm_fetch() calls will take care of freeing it for you, so you should mostly just pretend it is in static storage. The last fetched value will not be freed.
To use GDBM’s NDBM interface, you need GDBM’s version of the ndbm.h header file. Gnu’s install scripts don’t install this header file by default, so it is missing from many sites. By default the header is installed as ndbm.h, but many glibc-2.1 systems, including Redhat 6.1, install it as gdbm/ndbm.h.
To compile, you need to link with -lgdbm and sometimes -lgdbm_compat.
NDBM Library (DB Emulation)
All versions of Berkeley DB include an interface emulating NDBM. NDBM’s size limitations disappear in DB’s emulation.
The on-disk file format is completely different from genuine NDBM files. It will a single file with a .db suffix added to the name, instead of two files with .dir and .pag suffixes.
The dbm_close() call seems to deallocate the “static” memory that values returned by fetch() are saved in, so you need to be sure to save those elsewhere before closing the database.
The DB emulation of NDBM buffers writes to the database. You must call dbm_close() to ensure that all data is written out.
In Berkeley DB Version 2 and later, the ndbm.h header file is eliminated. Instead you do
#define DB_DBM_HSEARCH 1
#include <db.h>
When compiling, you need to link with the DB library, usually -ldb.
In Berkeley DB Version 1, there is a ndbm.h header file, but in glibc-2.1 systems, including Redhat 6.1, it is installed as db1/ndbm.h. It used to be normal to install Berkeley DB so you’d link with -ldb, but in glib-2.1 systems, where both version 1 and version 2 are installed, you need to link with -ldb1 to get the version 1 library.
GDBM Library
Gnu’s GDBM can emulate Old DBM or NDBM, but it is best used through it’s own native interface.
GDBM allows multiple databases to be open simultaneously, and handles very large database files respectably. There are no size limits on keys or content. By default GDBM does locking, so that either one writer has the database open, or any number of readers have it open. It also buffers writes by default.
The header file gdbm.h should be included.
Database are opened or created with the gdbm_open() call. No suffixes are automatically appended to the filename.
Data is saved with gdbm_store() and retrieved with gdbm_fetch(). Gdbm_fetch() returns a pointer to dynamically allocated storage. It is the caller’s responsibility to free() this when it is no longer needed.
To compile, you need to link with -lgdbm.
Berkeley DB Library
Sleepycat’s “Berkeley DB” has been steadily evolving into an extremely sophisticated package that supports many kinds of database files beyond mere hash files, and lots of fancy locking and transaction control. It is probably wonderful, but I’ve used it only for simple hash files and that is all that is discussed here.
It has gone through four major versions, the first three with substantially different API’s and file formats. Small changes to the C API appear in most releases. It is normal to have more than one version installed on your system. It’s good to make sure you get the right header files aligned with the right link libraries. All versions can emulate the Old DBM and NDBM interfaces and Version 2 and later can emulate the Version 1.85 interface, but the file formats are all incompatible.
Apache’s mod_auth_db uses any version of Berkeley DB.
The header file db.h should be included and you should link with -ldb. If that header defines DB_VERSION_MAJOR, DB_VERSION_MINOR and DB_VERSION_PATCH then they give the version number. If these are not defined by db.h then you can assume that the major version number is one.
If you have a glibc-2.1 system (like Redhat 6.1), which installs both Version 1 and Version 2, then the header file and library described above are for Version 2. The header file for Version one is db1/db.h and the library is -ldb1.
Databases are always single files, with a .db suffix on their names. How to open them is different for each major release. Version 1 uses dbopen(), Version 2 uses db_open(), and Version 3 uses first db_create() to make a database handle, and then handle->open() to open the database.
Version 1 is known to have some bugs, and certain kinds of operations should be avoided (if you can’t avoid version 1 entirely). See http://www.sleepycat.com/historic.html.
Data is fetched and stored with db->get() and db->put() functions. It is important (and non-intuitive) that before doing any of these calls, you must zero out the key and content data structures that you will be passing in, like:
memset(&key, 0, sizeof(key));
memset(&content, 0, sizeof(content));
You even have to zero out the content structure before you fetch into it.
The syntax of the cursor() command, which creates a cursor for a sequential walk through the database, changed with version 2.6. Version 2.6 added a fourth flag argument (which can be set to integer zero to get the same behavior as older versions).
The syntax of the db->open() command changed in version 4.1, adding a new argument before the filename argument.
—Files
Perl provides basic I/O for both the standard input (keyboard) and output (display) devices and for files in the UNIX file system. More sophisticated I/O is provided through the UNIX DBM library. These various I/O capabilites are discussed.
UNIX DBM library I/O
Many UNIX systems include as a standard library a database management utility called DBM. Perl provides an interface to this library.
The DBM provides a transparent interface between associative arras internal to a Perl program and a pair of files that are managed by DBM in which the keys and corresponding values of the array are stored. Thus, when one inserts, changes, or deletes a key and/or value, the system makes the appropriate update to the persistent file version of the array. When accessing, the each ( ) operator, which returns both key and value, is more efficient than the foreach ( ) operator.
There are only two operators associated with DBM associate arrays: dbmopen and dmbclose. Once such as array has been opened, all interaction with it is conventional.
dbmopen
Opens the persistent array, given a file name and a access mode.
Form:
dbmopen(%ASSOC_ARRAY, “dbmfile”, $mode);
Example:
dbmopen(%AN_ASSOC_ARRAY, “name_address”, $mode);
In this example, the array, %AN_ASSOC_ARRAY can be created and manipulated within the Perl program. It’s actual values will be maintained, however, in two files managed by DBM: name_address.dir and name_address.pag. The $mode includes a standard UNIX access mode, such as “0755”. For details, see the discussion under chmod in the section on System Operators, below.
dbmclose
Closes the files.
Form:
dbmclose(%ASSOC_ARRAY);
Example:
dbmclose(%AN_ASSOC_ARRAY);
1.3 Data Types
Unlike C, Perl does not have separate data types for numbers, characters, strings, and so on. Instead, Perl has three primary data types: scalars, lists, and hashes. Scalars can hold one of several kinds of data: Type Example(s) Integers 3 -9 0377 # octal 0x15 # hexadecimal Floats 1.25 6.55e12 -2e-15 Strings ‘Perl’ “This is a sentence with a newline at the end.\n” Also unlike C, you do not need to declare your variables ahead of time. When you want a variable, you simply use it and it springs into being. Likewise, Perl deals with allocating the appropriate amount of memory for the data you want a scalar (or anything else) to hold. $@%&* Perl is a very punctuation-heavy language. Regular expressions are the worst offenders. In addition, all variable names start with an item of punctuation: $ Scalar @ Array or List % Hash & Function (aka Subroutine) * Typeglob (we won’t cover this) Numerical Data and Operators Arithmetic operators and their associated assignment operators are very much like C’s. A notable exception is /, which will create a float from an integer if appropriate.
Use int ($a / 2) for integer arithmetic. $a = 4; $b = 2.5; $c = $a + $b;
# now $c is 6.5 $b += 2;
# $b is 4.5 $a %= 3; # $a is 1 $a++; #
$a is 2 (same as C) Strings Strings are also held in scalars. They are not represented as arrays of characters (well, they are internally, but you never have to worry about that). Anything you want to do, you can do: $a = “Harvard”; $b = ’Computer’; $c = $a . ” ” .
$b; # $c is Harvard Computer $c .= ” Society”; # $c is Harvard Computer Society $d = $a x 4; # $a is HarvardHarvardHarvardHarvard $b x= 2; # $b is ComputerComputer
Perl has three built-in data types: scalars, arrays of scalars, and associative arrays of scalars, known as “hashes”. A scalar is a single string (of any size, limited only by the available memory), number, or a reference to something (which will be discussed in perlref). Normal arrays are ordered lists of scalars indexed by number, starting with 0. Hashes are unordered collections of scalar values indexed by their associated string key.
Values are usually referred to by name, or through a named reference. The first character of the name tells you to what sort of data structure it refers. This character is called a “sigil”. The rest of the name tells you the particular value to which it refers. Usually this name is a single identifier, that is, a string beginning with a letter or underscore, and containing letters, underscores, and digits. In some cases, it may be a chain of identifiers, separated by :: (or by the deprecated ‘); all but the last are interpreted as names of packages, to locate the namespace in which to look up the final identifier (see “Packages” in perlmod for details). For a more in-depth discussion on identifiers, see “Identifier parsing”. It’s possible to substitute for a simple identifier, an expression that produces a reference to the value at runtime. This is described in more detail below and in perlref. It is legal, but not recommended, to separate a variable’s sigil from its name by space and/or tab characters.
Perl also has its own built-in variables whose names don’t follow these rules. They have strange names so they don’t accidentally collide with one of your normal variables. Strings that match parenthesized parts of a regular expression are saved under names containing only digits after the $ (see perlop and perlre). In addition, several special variables that provide windows into the inner working of Perl have names containing punctuation characters. These are documented in perlvar.
Scalar values are always named with the sigil ‘$’, even when referring to a scalar that is part of an array or a hash. The ‘$’ symbol works semantically like the English word “the” in that it indicates a single value is expected.
If a scalar was the “singular” in Perl, as we described them at the beginning of “Scalar Data”, the “plural” in Perl is represented by lists and arrays. A list is an ordered collection of scalars. An array is a variable that contains a list. In Perl, the two terms are often used as if they’re interchangeable. But, to be accurate, the list is the data, and the array is the variable. You can have a list value that isn’t in an array, but every array variable holds a list (although that list may be empty).
Represents a list, whether it’s stored in an array or not.
A list with five elements Each element of an array or list is a separate scalar variable with an independent scalar value. These values are ordered — that is, they have a particular sequence from the first to the last element. The elements of an array or list are indexed by small integers starting at zero[67] and counting by ones, so the first element of any array or list is always element zero. [67]Array and list indices always start at zero in Perl, unlike in some other languages. In early Perl, it was possible to change the starting number of array and list indexing (not for just one array or list, but for all of them at once!). Larry later realized that this was a misfeature, and its (ab)use is now strongly discouraged. But, if you’re terminally curious, look up the $[variable in the perlvarmanpage. Since each element is an independent scalar value, a list or array may hold numbers, strings, undef values, or any mixture of different scalar values. Nevertheless, it’s most common to have all elements of the same type, such as a list of book titles (all strings) or a list of cosines (all numbers). Arrays and lists can have any number of elements. The smallest one has no elements, while the largest can fill all of available memory. Once again, this is in keeping with Perl’s philosophy of “no unnecessary limits.”
3.1. Accessing Elements of an Array If you’ve used arrays in another language, you won’t be surprised to find that Perl provides a way to subscript an array in order to refer to an element by a numeric index. The array elements are numbered using sequential integers, beginning at zero and increasing by one for each element, like this:
$fred[0] = “yabba”;
$fred[1] = “dabba”;
$fred[2] = “doo”; The array name itself (in this case, “fred”) is from a completely separate namespace than scalars use; you could have a scalar variable named $fred in the same program, and Perl will treat them as different things, and wouldn’t be confused.[68] (Your maintenance programmer might be confused, though, so don’t capriciously make all of your variable names the same!) [68]The syntax is always unambiguous — tricky perhaps, but unambiguous. You can use an array element like $fred[2] in every place[69] where you could use any other scalar variable like $fred. For example, you can get the value from an array element or change that value by the same sorts of expressions we used in the previous chapter: [69]Well, almost. The most notable exception is that the control variable of a foreach loop, which we’ll see later in this chapter, must be a simple scalar. And there are others, like the “indirect object slot” and “indirect filehandle slot” for print and printf. print
$fred[0];
$fred[2] = “diddley”;
$fred[1] .= “whatsis”; Of course, the subscript may be any expression that gives a numeric value. If it’s not an integer already, it’ll automatically be truncated to the next lower integer:
$number = 2.71828; print
$fred[$number – 1]; # Same as printing
$fred[1] If the subscript indicates an element that would be beyond the end of the array, the corresponding value will be undef. This is just as with ordinary scalars; if you’ve never stored a value into the variable, it’s undef.
$blank = $fred[ 142_857 ]; # unused array element gives undef $blanc = $mel; # unused scalar $mel also gives undef docstore.mik.ua/orelly/perl4/lperl/ch03_01.htm