File format policy for born digital collections

Document history

Goal of this document

• This file format policy forms the focus of the preservation planning of the IISH workflow for born digital material. It also an important element of the broader IISH preservation policy. 
• Concretely it forms the basis for the normalisation rules that are applied - in the so called File Format Registry (FPR) - when collections are ingested following the Archivematica archiving workflow. 
• The file format policy is also a supporting document to help reach a TDR (Trusted Digital Repository) status for the IISH.

Status of this document

This is working document - as the need arises new categories or formats might be added. The content may also change with new insights. 

Terminology

• Normalisation: migrating a file to a more common or easier to manage file format. 
• Original format: file format of the digital object when it arrives at the IISH
• Preservation format: a file format that is preferably:
  • Open: technical specification as an ISO standard, or otherwise freely available standard
  • Stable: not frequently changed or is backward compatible
  • Supported: widely adopted by creators and consumers, with readily available software tools
• Preservation copy: when the original format comes under threat of being obscure and/or obsolete, a copy is made in another file format. This preservation copy preserves - as much as is possible- the information value of the original object (see under 'preservation intent' below). 
• Intermediate access copy: in the context of the IISH infrastructure the intermediate access copy is not a file that is directly meant for end users. Instead this file is placed within the DIP as "raw" material from which the "presentation layer" creates one or more access copies for end-use. Therefore, in many cases, the preservation and intermediate access copy will be same. 
• Access copy: The access copy is the file that is meant for end-use. This copy is made from the intermediate access copy. The creation of the access copy is not part of Archivematica functionality.  

Sources

The main sources for this policy are:

• Sustainability of Digital Formats: Planning for Library of Congress Collections (https://www.loc.gov/preservation/digital/formats/). 
• "DE BASIS richtlijnen" (http://www.den.nl/debasis) of the DEN foundation. 
• The file format strategy by the Dutch National Archives presented in Voorkeursformaten Nationaal Archief. Met het oog op duurzame toegankelijkheid, versie 1.0 november 2016 (http://www.nationaalarchief.nl/home/onderwerpen/informatiebeheer-archiefvorming/digitaal/voorkeursformaten) and Bestandsstrategieën Nationaal Archief, versie 1.0 (https://informatie2020.pleio.nl/blog/view/46539532/bestandsstrategieen-nationaal-archief). 
• The Archivematica file format policy wiki (https://wiki.archivematica.org/Format_policies). See especially the pages on preservation (https://wiki.archivematica.org/Preservation_formats) and access formats. (https://wiki.archivematica.org/Access_formats) 
• The DANS preferred and accepted file formats (https://dans.knaw.nl/en/deposit/information-about-depositing-data/file-formats).

Preservation intent

The preservation intent of the IISH is the rationale behind the file format policy (and the IISH preservation policy as whole). In the preservation policy this is described as follows:

• All digital objects will be kept and (as much as possible) preserved in their original form(at).
• But the IISH will, ultimately, give priority to the informational value within these objects. This means that objects may be migrated to other file formats - as the original format is obscure and/or obsolete - as long as can be guaranteed that the information within these files is still authentic.
• These migrated files are called preservation copies.
• As the proof of authenticity will in some cases be a challenge the original object will always be available as a fall-back file.
• The digital object is always shown within the right context and is findable through the correct contextual information.

Normalisation 

Normalisation of file formats are performed for two reasons:

1. Normalisation for preservation: when the original file format is, or might be, in risk of becoming obsolete (that is to say = can no longer be rendered by current software and/or no longer be converted to another format) a preservation copy is made. This copy is as true to an authentic copy as possible. 
2. Normalisation for access: if necessary an intermediate access copy (see explanation above) is made from the original file or the preservation copy. For digital object categories that are subject to a lot of change - like video - new access copies will have to be made on a regular basis (e.g. every 5 years). In many cases the access copy is the same as the preservation copy.
   
   

Because the IISH has little to no influence on file formats coming into the archive, and is confronted with a wide selection of formats, we have limited the amount of file formats in the normalisation for preservation process. This decision was made (May 2017) after a careful deliberation of different scenario's for normalization.  

File format policy rules

Following the above these rules apply:

• The IISH (nearly) always preserves the original file in the file format in which it was received. The original file format is part of the AIP.
• If the original format is in danger of becoming obsolete a preservation copy is made in a different file format (normalisation for preservation). The preservation copy is part of the AIP and DIP.
• If the original file format or preservation copy is not easily accessible the IISH creates an (intermediate) access copy in a different file format (normalisation for access). If this is not the case the original format and the access copy are the same. The access copy is part of the DIP.
• From the intermediate access copy, if necessary, extra derivatives are made for end-use. These extra derivatives are preferably created on an adhoc basis - however this may not always be possible (e.g. image stills of video files). The creation of these extra derivatives is not part of the Archivematica workflow. This process is executed by a separate (to be developed) "presentation layer" into which all information concerning access is brought together. I.e. the DIP only offers the raw material for the presentation layer from which these materials are further refined for end-use.