Ross Harvey reflects on the July monthly meetup for Australasia Preserves.
At the monthly meetup for July we were treated to a presentation from Andrew Martin (from DAMsmart) about an LTO (Linear Tape-Open) data migration, from an earlier version of LTO to LTO 6. Tom Eccles from AIATSIS (The Australian Institute of Aboriginal and Torres Strait Islander Studies) was the client for the data migration work, and he also dialled in to give us more context and perspective for this project.
Andrew began with an introduction to LTO, noting it was an open standard, and that its developers had provided a roadmap for versions up to LTO 12 which will be capable of storing up to 480TB of compressed data. Currently LTO 8 is the latest available.
Counteracting these positive features are that LTO 8 drives can only read LTO 7 and LTO 8 cartridges because of a change in recording heads, and the number of manufacturers of LTO equipment and tapes is reducing. LTO cartridges have been unavailable because of patent lawsuits but are likely to be available again in the third quarter of 2019.
Next Andrew described the case study: the AIATSIS ATSIC digitised video collection, 36 TB of data on 50 LTO 4 cartridges. He provided technical details of how the data were transferred over three weeks onto 5 Western Digital drives each of 12 TB. The data verification process was described, using custom software to search for filenames, extract checksums, and compare them. Files were renamed then were migrated to LTO 6 and verified.
For anyone considering a similar data migration project, some useful advice was given about what to watch out for when preparing:
1. What format / vendor has the source LTO been written under?
2. Is there a database of the source data?
3. Does the data need specific proprietary software to be restored?
4. Are there existing checksums that can be utilised to ensure correct restore?
5. Is there any additional processing required while files are online in the staging process? (e.g. metadata creation, QC checks, transcoding)
6. Would you write tape to tape, or tape to disk to tape?
7. Is LTO the right format to be migrating to? What are the preservation and business requirements?
8. Would migrating to the cloud be appropriate?
The final three questions are about storage mediums and raise important issues, such as whether tape storage is still viable for preservation. And a good question is raised that could prompt further discussion: at what point do business requirements override preservation requirements? The issue of the validity of cloud storage was raised again, a question always worth revisiting.
There was a bonus for me: I learnt about the Amazon Snowball device: ‘a petabyte-scale data transport solution that uses devices designed to be secure to transfer large amounts of data into and out of the AWS Cloud’ – I think I want one!
Andrew has also shared his slides with us.
