How to Create an Intelligent Cloud Media Archive with Dalet Flex

One of our large customer’s media engineering group provides a vivid illustration of how to migrate a major cable TV network’s legacy asset storage system to a highly scalable, automated, cloud media archive designed for continual growth in content utility.

Let’s take a closer look at their approach!

Identifying the business challenges…

The scope of the archive migration included over two decades of the cable TV Network programming. Our customer’s engineers were tasked to quickly architect and deploy an automated migration, archive and distribution pipeline following a recent corporate acquisition. A key objective was to transfer this continually expanding archive to a highly secure, but broadly accessible, storehouse that could make assets searchable and readily available internally and to a global array of satellite, cable TV, and other affiliates.

One of the key challenges to the project was the limited metadata associated to the assets in the archive. Most of the assets were stored with minimal metadata and no supporting framework to augment it for downstream repurposing and enrichment. This was a huge challenge considering the assets were stored on on-premises edit storage under control of a production asset management (PAM) system originally intended to handle just the core A/V content and promotional graphics with no metadata.

The legacy system had no means of supporting ancillary metadata; it instead used a limited asset identification system based on a non-standard formulation of Industry Standard Coding Identification (ISCI) code interleaved with file names.

The team needed a plan for an efficient approach to storage management suited to long-term archive preservation. The new cloud media archive system also had to dynamically assign storage tiers to content based on metadata attributes like: 

• End consumer department or organizational unit 
• Video type or category
• Source / origination point
• Planned or unplanned business event
• And potentially much more…

The solution: media logistics, orchestration, cloud and scalability

As a fully integrated component of our customer’s M&E portfolio, the cable TV Network uses workflow orchestration enabled by Dalet Flex to operate a hyper-scale, cloud-based media logistics platform, instantiated on the Amazon S3 Glacier service.

The archive includes over 8,000 30-minute to 240-minute long-form assets with 3-5 more added daily, in addition to a collection of over 120,000 1-minute to 10-minute clips that grows by 300-500 per day.

Solution architecture, describing systems integrated within Dalet Flex

Focus on the technical implementation

Our customer’s engineers were able to create a highly scalable, interactive library supporting enterprise archive access, security, and usability using the Dalet Flex media logistics platform. Dalet Flex provides the abstraction layer essential to enabling workflow automation and metadata enrichment for the new library. At the outset, this included support for file migration from on-premises production storage, first to Amazon Simple Storage Service (Amazon S3) for primary storage and then to the cloud media archive, Amazon S3 Glacier for data archiving and long-term backup.

Once content was moved into AWS storage, Dalet Flex-enabled workflow automation and metadata enrichment continued to work seamlessly across Amazon S3 and Amazon S3 Glacier, supporting dynamic utilization of each as required. The Dalet Flex platform intelligently archives and restores content files based on both the dynamic archival rules set by end users, and ongoing daily user interactions with FlexMAM, one of the platform’s intuitive toolsets. 

Wherever video files are stored in Amazon S3 or Amazon S3 Glacier, they are surfaced to users through FlexMAM based on FX business and operational scenarios. Regardless of storage tier (class), the user asset management experience remains the same.

At a high-level, the solution operates as follows:

1. Ingest Content
   • An operator monitors an Amazon S3 bucket (hot folder) for available new media. The Assumption is that Aspera will deposit assets into this bucket as they become available on the NAS.
   • Program (long form) and promo (short form) content will be ingested via separate hot folders.
2. Create proxy resolution asset from source material
   • Dalet Flex will use its Flex Stream Processor (FSP) node cluster to create proxies using the FlexMAM standard proxy format.
   • This action includes audio downmixing for multidevice reviewing.
3. Process Metadata
   • Upon ingest the media file name will be parsed to extract metadata.
   • Dalet Flex will auto-populate some of the metadata fields, extract technical metadata from source media file and generate keyframes.
4. Content Archival
   • Content that is scheduled for use in a prescribed timeframe will be stored in the cloud media archive, Amazon S3 Glacier.
   • The operator can modify that time period using the Dalet Flex admin portal.
5. Restore Assets
   • The operator can select one or more assets in FlexMAM and request to send these back to upstream systems.
   • Dalet Flex will call its transfer service to copy the selected assets from Amazon S3/Amazon S3 Glacier to the downstream systems.

The Dalet Flex solution highlights include support for:

• Fully cloud-based content archive and distribution;
• Real-time collaboration among teams organized internally or with affiliate participation;
• Orchestrated archive and restore workflows based on automatic rules, including the ability to define primary storage days based on asset type;
• Access to content from anywhere, with intuitive browser-based user interfaces.
• Fast search and discovery capabilities across sophisticated metadata models
• Frame-accurate proxy playback and clipping

Access management flexibility

Dalet Flex also supports dynamic automatic asset tagging based on content type to drive archival policy. The tagging also defines how long media stays on primary Amazon S3, and when assets should be moved to Amazon S3 Glacier. The ingest of each video file applies object tags based on the just-in-time (JIT) setup of the primary storage duration policies defined by the user.

Dalet FlexMAM: content discovery

All assets are accessible across our customer’s enterprise by authorized end-users. Personnel are provided an improved search experience based on fully extensible metadata schemas, play back frame-accurate proxies, and can activate pre-configured actions such as restoring single or multiple assets (bulk mode) from Amazon S3 Glacier as needed. Users have the option to expedite or to use the standard approach to asset restoration, based on the required speed.

Dalet FlexMAM: asset proxy playout and metadata

The ability to control content lifecycle per storage tier through the FlexMAM portal provides end users dynamic, easy-to-deploy decision-making options that add to the benefits of operating in a cost-effective and resource-optimized working environment. For example, if managers know that a movie will be heavily accessed during the next 30 days, they can simply update the tag that allows that asset to remain in Amazon S3 instead of following the normal aging protocol of moving the asset to the cloud media archive on Amazon S3 Glacier.

Dalet FlexMAM: manual trigger for archive workflows

This added control over their content lifecycle allows our customer to work more efficiently, avoiding the cost and time associated with multiple restores from Amazon S3 Glacier. This manual override lowers overall storage costs.

The results: optimized workloads and significant savings (costs + time)

Our customer’s engineering group overcame the challenges associated with moving assets from a rigid on-premises archive to a dynamically manageable, massively scalable library hosted in the AWS Cloud.

Leveraging the Dalet Flex media logistics platform to transfer assets and manage them in the new cloud media archive, the engineers are now able to efficiently orchestrate asset aging and exchanges between Amazon S3 and Amazon S3 Glacier services. By combining automation and manual intervention, they can optimally balance workloads and virtualized infrastructures to achieve significant savings in costs and time.

With support for smoother operations and greater workload adoption flexibility, Dalet Flex empowers operational staff to realize the expected cost savings envisioned for cloud/virtualized technologies. This allows them to quickly compensate for the variabilities that are a natural part of the television and feature film industries.