The Ånge Videomigration Project
In late 2005, Sveriges Radio Förvaltnings AB (SRF) became the project owner for a massive preservation job involving analog video from Swedish Television (SVT), the Swedish National Sound and Video archive (SLBA) and Educational Radio (UR). A total of 120,000 hours needed to be migrated into video files by the end of 2009. It was also necessary to find storage for around 3PB (petabytes, equivalent to 3,072 terabytes) in a large scale HSM-system in Stockholm. In addition, SRF had to devise the methods by which clients could access migrated video for use in their production chains. This included establishing a new factory for video migration in a small town called Ånge, about 400 Km north Stockholm. EU and regional funds were made available, both to generate employment in the region and because it was hoped that by providing the structure, this new service company would eventually become autonomous and global, providing migration services for audiovisual archives worldwide. In fact the approach was successful as the outstanding technology project of NOA, SVT and SRF has been sold later to the Belgium service provider memnon.
Procedures which had to be achieved:
- To meet this challenge, new workflows and techniques were needed, the most important being:
- Defining all necessary steps for a smooth system operation
- Creating the structures in Ånge
- Sourcing the right suppliers
- Developing and integrating new components for specific tasks
- On a more detailed level this required:
The introduction of parallel partly-attended migration, where one operator would simultaneously run 4 video players and connected ingestion stations together with quality control and metadata management.
Completely consistent flowhandling using MD5 checksums for all content files during the process.
The transfer over around 3-6 TB (terabytes, equivalent to 3,072-6,144 gigabytes) of data per day over IP to SRF’s central storage (datahousing) facility.
Retranscoding into Hi- and LoRes files (both had been requested) with ‘top and tail’ editing.
Acceptance of the service quality in SRF’s existing NOA mediARC Radio Archive system, using the same infrastructure for file ingestion and radio production file delivery.
The volume of data generated for video migration would, when running at full capacity, reach 6TB per day, but additional internal or temporarily generated volumes may amount to as much as 10TB per 24 hours. In total, over the 3 year project timeline, some 3PB (3,072TB, or more than 3 million GB) needed to be generated and put onto a dedicated long-term archive storage.
The workflow requirements for each and every format were demanding. Digibeta, BetaCam SP, 1-inch SP and U-matic formats had to be migrated, necessitating a total of 4x4 lines for manual migration and 6 additional lines for unattended migration handled by a robotic system.
Back in 2006, the technical project managers, Björn Blomberg and Anders Hjelm, investigated the possibility of using and expanding the existing NOA mediARC system with its sophisticated workflow engine as well as a very fast and proven failsafe metadata structure.
To achieve that, the NOA side also had to accomplish a series of tasks:
Introduction of new application servers handling the file management transparently to the user but with dedicated security layers while being compatible with the OAIS (Open Archival Information System) reference model.
Introduction of parallelised “gridlike” server components for failover mechanisms.
Introduction of a freely configurable workflow, which could also take different production domains into account.
Development of a standard open and well documented API (application programming interface) for generic ingestion machines (in this case, video).
A universal API for the NOA workflow engine – an extension of the already existing NOACOMM interface – was provided to address the need for multiple active processors. NOACOMM is a simple API which communicates over the TCP and XML protocols and allows for the attachment of any kind of third party ingest system by simple sample applications. Using these extensions, SRF developed video ingestion and retranscoding engines, wrapping existing ADs and Codecs (available in the market as standard products) with quality control. Codecs were connected to the NOACOM API interface as well as to the newly delivered CLIP (Command Line Interface Processor) interface. A total of 30 ingestion and retranscoding processors were attached over the NOA processor host as well as around 30 generic processors for the management and transportation of files towards the mediARC workflow engine (running in Stockholm over WAN).
Those processors/workstations cover:
- Top Tail Cutting
- Format Conversion
With the flow engine needing to be able to respond to changing situations in the production chain, SRF chose the NOA mediARC platform because it could easily be expanded to handle these newly created workflows and massive throughput.
JPEG2000 was considered but because the technology was not yet completely ready for use, SVT, SLBA and UR decided to go for existing technologies that had already been approved for use in their own production workflows.
Encoding: VC1, MPEG2, MPEG1, DVCPRO50, BWF/Stereo.