Storing Data Structures to Disk (Perl Cookbook, 2nd Edition)

11.13.1. Problem

You want to save your large, complex data structure to disk so you don't have to reconstruct it from scratch each time your program runs.

11.13.2. Solution

Use the Storable module's store and retrieve functions:

use Storable;
store(\%hash, "filename");

# later on...
$href = retrieve("filename");        # by ref
%hash = %{ retrieve("filename") };   # direct to hash

11.13.3. Discussion

The Storable module uses C functions and a binary format to walk Perl's internal data structures and lay out its data. It's more efficient than a pure Perl and string-based approach, but it's also more fragile.

The store and retrieve functions expect binary data using the machine's own byte-ordering. This means files created with these functions cannot be shared across different architectures. nstore does the same job store does, but keeps data in canonical (network) byte order, at a slight speed cost:

use Storable qw(nstore);
nstore(\%hash, "filename");
# later ...
$href = retrieve("filename");

No matter whether store or nstore was used, you need to call the same retrieve routine to restore the objects in memory. The producer must commit to portability, but the consumer doesn't have to. Code need only be changed in one place when the producer has a change of heart; the code thus offers a consistent interface for the consumer, who does not need to know or care.

The store and nstore functions don't lock the files they work on. If you're worried about concurrent access, open the file yourself, lock it using Recipe 7.18, and then use store_fd or its slower but machine-independent version nstore_fd.

Here's code to save a hash to a file, with locking. We don't open with the O_TRUNC flag because we have to wait to get the lock before we can clobber the file.

use Storable qw(nstore_fd);
use Fcntl qw(:DEFAULT :flock);
sysopen(DF, "/tmp/datafile", O_RDWR|O_CREAT, 0666)
        or die "can't open /tmp/datafile: $!";
flock(DF, LOCK_EX)                   or die "can't lock /tmp/datafile: $!";
nstore_fd(\%hash, *DF)
    or die "can't store hash\n";
truncate(DF, tell(DF));
close(DF);

Here's code to restore that hash from a file, with locking:

use Storable qw(retrieve_fd);
use Fcntl qw(:DEFAULT :flock);
open(DF, " < /tmp/datafile")      or die "can't open /tmp/datafile: $!";
flock(DF, LOCK_SH)                or die "can't lock /tmp/datafile: $!";
$href = retrieve_fd(*DF);
close(DF);

With care, you can pass large data objects efficiently between processes using this strategy, since a filehandle connected to a pipe or socket is still a byte stream, just like a plain file.

Unlike the various DBM bindings, Storable does not restrict you to using only hashes (or arrays, with DB_File). Arbitrary data structures, including objects, can be stored to disk. The whole structure must be read in or written out in its entirety.

11.13.4. See Also

The section on "Remote Procedure Calls (RPC)" in Chapter 13 of Advanced Perl Programming, by Sriram Srinivasan (O'Reilly); Recipe 11.14


11.12. Copying Data Structures		11.14. Transparently Persistent Data Structures

11.13. Storing Data Structures to Disk

11.13.1. Problem

11.13.2. Solution

11.13.3. Discussion

11.13.4. See Also