ResourceSpace has changed the way the DEC uses content, making it much easier for us to quickly make assets available both internally and externally during our emergency appeals.
Managing digital archives is an increasingly important part of a museum archivists role, but preserving digital archives securely presents unique challenges.
In this blog we’re going to look at what can go wrong, the importance of fixity checking and metadata, and provide a practical framework for long-term archiving.
READ MORE: Four common DAM challenges faced by... Museum Archivists
What can go wrong with digital archives over time?
Your digital archives might appear secure, but they might quietly become damaged, inaccessible or vulnerable over time. Here are some of the common risks you need to be aware of.
‘Bit Rot’, corruption and silent file damage
‘Bit rot’ refers to the gradual corruption of digital data, including visual and audio assets. Over time, individual bits (the 1s and 0s that constitute an asset) change or become unreadable because of physical storage degradation, software error or other hardware or software failure.
This damage might not be obvious to people accessing the file, but when assets required for digital preservation suffer bit rot, the quality and accuracy of those files can be impacted, making it hard for archivists or curators to closely inspect them.
Note that bit rot and hardware failure are different. The key distinctions are what fails and how it affects the data. While bit rot is corruption of the data itself, hardware failure is a problem with the physical storage equipment, for example a hard drive, SSD or server.
Format obsolescence: When your files outlive their software
Format obsolescence occurs when the software or system needed to open a file is no longer supported, rendering a file difficult or impossible to access.
For example, a museum might have a collection of digital images stored in a file format that was once widely supported, but is no longer recognised by current software. Older video, audio and proprietary document formats can be particularly vulnerable because they may depend on specific software or operating systems.
Storing multiple copies of a file doesn’t solve the problem either if every copy replies on the same technology. Archivists need to consider which formats are likely to remain accessible and put in place a process for monitoring format developments and migrating to newer formats if necessary.
Fixity Checking: How integrity verification protects your collection
Fixity checking is commonly used in digital preservation for verifying that a digital file hasn’t changed or become corrupted over time, like a fingerprint comparison for your files.
When a file is created or uploaded to a digital preservation system, algorithms such as MD5, SHA-256 and SHA-1 produce a string of characters that represent that specific version of the file. If even a small part of that file changes, recalculating the checksum will identify that change, allowing archivists to compare the current checksum with the original. If the checksums don’t match, the archivist needs to investigate further to find out what’s changed.
Fixity checks can be automated, making it easier to verify large asset libraries without needing someone to open each one individually.
ResourceSpace can handle this for you and verify the file integrity of primary resources, notifying system admins if the file is corrupted and allowing them to take any remedial action necessary.
How often should you run fixity checks?
There’s no set schedule that’s best for every archive, and it really depends on how many assets the collection holds and the risk-level of the assets. For example, a small collection stored on a stable system might only need checking annually, while a large, high-risk collection needs to be reviewed at least monthly.
Archivists should consider:
- the value and uniqueness of the material they’re storing;
- the reliability of the storage environment;
- how often files are accessed or moved;
- whether fixity checks can be automated.
What to do when a fixity check fails
A file isn’t lost just because it ‘failed’ a fixity check, but it does mean that something has changed, and archivists should investigate to find out exactly what that change is. Here’s a simple four step process:
- Identify the affected file, reviewing its location, checksum history and recent activity.
- Compare the file with a ‘good’ copy (if one exists) to establish whether the file has genuinely been corrupted.
- Restore the file to a verified backup version before running another fixity check confirming its integrity.
- Log what happened, including when the failure was detected, which file was affected and how it was resolved. This will help to identify repeated failures that might indicate a wider storage problem that needs to be resolved.
Cloud storage vs cloud archiving
Arts and heritage institutions sometimes assume that uploading digital collections to the cloud means they’re automatically archived, but backups and archiving serve different purposes.
A backup is primarily about recovery in the event that a file is accidentally deleted or corrupted. By contrast, an archive is for keeping digital assets safely for the long term, while ensuring they remain discoverable and accessible when they’re needed.
A useful standard to follow for preserving important digital collections is the 3-2-1 rule. This involves keeping three copies of your data, two of which are stored on different types of storage media, while at least one copy is held offsite. This reduces the risk that a single hardware failure, incident or location-wide event will destroy every copy.
The OAIS Model: A practical framework for long-term archiving
Developed for facilitating a broad consensus on the requirements for an archive to provide long-term preservation of digital information, the Open Archival Information System (OAIS) is a fundamental standard in digital preservation. It defines a set of responsibilities that an OAIS archive must fulfil, and distinguishes OAIS archives from other types of archive.
For museum archivists, the value of OAIS is that it provides a structured way to think about the entire lifecycle of a digital object, from the point it enters the archive through to its long-term preservation and access. It’s not a specific software system or storage platform, instead providing a reference model that organisations can use when developing their own preservation processes.
The OAIS model identifies six key functional areas:
Ingest — receiving digital material and preparing it for preservation.
Archival storage — securely storing preserved information and maintaining its integrity.
Data management — managing descriptive, technical and preservation metadata.
Preservation planning — monitoring changes in technology and planning responses to risks.
Access — making preserved information available.
Administration — managing the archive’s policies, resources and responsibilities.
Metadata, file formats and keeping collections discoverable
Long-term preservation isn’t just about keeping files somewhere safe, but about ensuring they can be discovered and used years after they were first uploaded. After all, if an archivist can’t find a file, it’s effectively lost even if the data integrity is still intact.
READ MORE: The six core metadata schemas explained
This is where structured metadata comes in, preventing this problem by recording the information needed to identify, understand and manage each digital asset. Standards such as Dublin Core can provide consistent descriptive information, while PREMIS is designed for preservation metadata, helping record information needed to support the long-term management of digital materials.
Choice of file format should be part of the preservation strategy too, taking into account obsolescence and proliferation. Not all digital formats are suitable for archiving and preservation, and we examined some of the most common file formats used by archival teams back in September.
