When 100,000 Media Files Become an Engineering Problem

Last Updated 1 hour ago by Kenya Engineer

For most organisations, a digital media library becomes a problem long before anyone starts talking about engineering architecture.

At first, there are only a few photographs, some videos and perhaps a collection of documents. WordPress makes uploading and managing them relatively straightforward. As the collection grows, however, familiar problems begin to appear: duplicate files, inconsistent metadata, difficulty finding older material, slower interfaces and uncertainty about which copy of a file is actually the authoritative one.

The problem becomes more complicated when the same media must be used across several websites or assembled into different stories over time.

This is the situation the Trott Bailey Family found itself facing after almost two decades of accumulating photographs, video, audio, articles and other digital records. Their WordPress network eventually approached 100,000 media files.

At that scale, the challenge was no longer simply where to store the files.

It was how to preserve the identity, context, relationships and editability of the underlying media while continuing to distribute it through different websites and platforms.

That problem led to the development of Big King Media, a WordPress-based media operating layer developed within the family’s own production environment.

The experience offers a useful lesson for anyone managing a large digital archive: a media library is not merely a collection of files. It is a system of relationships.

The archive and the distribution platform are not the same thing

The Trott Bailey Family made an early decision that influenced much of the architecture that followed.

Social platforms would be used to reach audiences, but they would not be treated as the permanent home of the family’s digital record.

Facebook, Instagram, TikTok and YouTube could all change their formats, algorithms, interfaces or commercial models. The family website, by contrast, could remain under their control.

The distinction is important.

A social platform is fundamentally a distribution channel. The underlying photograph, video or document is the source material.

When those two roles become confused, organisations can end up building their archives around the requirements of the platform that happens to be dominant at a particular moment.

The Trott Bailey approach was different: maintain the archive as the controlled source and allow different platforms to act as roads leading back to it.

This creates what the developers describe as technical optionality. If a distribution platform changes, the underlying archive does not necessarily have to be rebuilt around that change.

For a family archive, that may sound like a relatively specialised concern. The same principle applies, however, to engineering firms, universities, research organisations, government agencies and media organisations whose digital records may need to survive several generations of software and distribution platforms.

The question becomes simple:

If the platform changes tomorrow, how much of the underlying record has to change with it?

Sharing media can create a provenance problem

WordPress Multisite introduces another challenge.

A network with several sites may benefit enormously from being able to reuse media across those sites. The obvious solution is to make the media available everywhere.

But access and ownership are not necessarily the same thing.

Consider a photograph that originates on Site A and is subsequently used by Sites B, C, D and E. If the photograph is copied into each site’s media library, the network may now have five separate records representing what is essentially the same source.

The image itself may be identical, but its associated information may not be.

One site could have one caption, another could have different alt text and another could contain a different description. Over time, those records can drift apart.

The result is a form of technical debt that is difficult to see because the files still appear to work.

The Trott Bailey Family approached this as a provenance problem.

Big King Media is designed around the idea that reused media should retain an authoritative source attachment. A receiving site can use the media without effectively claiming that it originated there.

The architecture therefore needs to preserve several pieces of information:

• where the media originated;
• which attachment is authoritative;
• what metadata belongs to the source; and
• which other contexts are using or referencing it.

This changes the problem from simply asking, “Can another site access this image?” to asking, “Can another site access this image without destroying our understanding of where it came from?”

That distinction becomes increasingly important as an archive grows.

A large archive turns metadata into infrastructure

In a small media library, poor metadata can be irritating.

In a library approaching 100,000 files, it becomes an infrastructure problem.

WordPress treats uploaded media as attachment records rather than simply as anonymous files. Those records can contain information such as dimensions, file information, alternative text, captions and descriptions, and can be exposed through WordPress’s APIs.

For a large archive, this information becomes central to search, accessibility, duplicate control and contextual discovery.

The Trott Bailey Family has consequently treated automated assistance, including automatic alt-text generation, as a descriptive aid rather than a substitute for human judgement.

That distinction matters.

A system may be able to describe what appears in a photograph. It does not necessarily know why the photograph matters.

A human curator may understand that an apparently ordinary photograph is part of a particular period in a family’s history, a project, a technical study or another larger story.

This leads to another important architectural idea behind the family’s approach: a media object should be able to retain its own identity while participating in multiple contexts.

A photograph can remain an individual source record while simultaneously becoming part of a project, a historical period, a technical study or a visual story.

The alternative is either a collection of isolated files with very little context or a large shared repository in which individual provenance becomes difficult to establish.

Neither is particularly useful in the long term.

The problem with treating a finished video as the story

The same question of source versus distribution appears in video.

The conventional workflow is familiar. Photographs and video clips are assembled on a timeline, text and audio are added, the sequence is rendered and the result is exported as an MP4.

There is nothing inherently wrong with this approach. MP4 remains a useful and widely supported distribution format.

The problem arises when the exported video becomes the primary representation of a story that is still changing.

Every render involves computation. It produces another file and introduces another step between making a change and publishing the result.

More importantly, it can make the story less reversible.

Imagine a family movie assembled from photographs taken over an entire year. After publication, another photograph is discovered. A caption needs correction. A different photograph would make a better opening. A new event needs to be inserted into the middle of the sequence.

If the MP4 is treated as the authoritative version, each significant change can push the workflow back towards editing and re-rendering the movie.

The Trott Bailey Family’s answer to this problem is Kezideki Dynamic Movie.

Instead of treating the finished video file as the only representation of the story, the system represents the movie as an ordered set of relationships to its underlying source media.

A sequence can therefore refer to a photograph, a video clip, a title or an audio element while those underlying objects remain individually addressable.

A change to one element does not necessarily mean that every underlying asset has to be re-encoded.

A new scene can be inserted into the sequence. A title can be changed. A source photograph can be replaced.

The important idea is not that MP4 becomes unnecessary.

It is that MP4 becomes a distribution output rather than the only durable representation of the story.

Source versus output is a familiar engineering distinction

The concept is not unique to media.

An engineer working with software already understands that source code is not the same thing as the compiled executable.

In design, a CAD model is not the same thing as a PDF drawing exported from it.

In information systems, a database is not the same thing as a report generated from the database.

The source contains information and relationships that may be lost when it is converted into a fixed output.

The same principle can be applied to digital storytelling.

A finished video may be exactly what a particular audience or platform requires. But if the organisation expects the story to evolve, retaining the underlying relationships can provide much greater flexibility.

This is the central idea connecting Big King Media’s approach to shared media and Kezideki Dynamic Movie:

Do not destroy source intelligence merely to make distribution convenient.

Building for change rather than for one release

Large software systems rarely fail because one individual feature does not work.

More often, a new feature changes the assumptions on which another feature depends.

A search improvement can affect another query. A change to shared-media handling can affect an existing gallery. A modification to video functionality can have consequences elsewhere in the system.

The Trott Bailey Family therefore treats regression control as part of the engineering process rather than something left until the end.

Their development process separates feature work from stabilisation work. After a feature or fix is implemented, the surrounding system is tested for regressions before further development proceeds.

The philosophy is straightforward:

A feature is not complete simply because its own button works. It is complete when the surrounding system continues to behave correctly.

That distinction becomes particularly important when software is being developed inside a live production environment rather than as a standalone experiment.

The family archive itself becomes the test environment.

Kimroy Bailey’s role focuses heavily on plugin architecture, implementation, debugging, performance, automation and release stabilisation. Sherika Trott Bailey provides a different but equally important form of testing: whether people can actually find the media they need, understand its context, reuse it without losing meaning and present it effectively.

The archive is therefore being tested not only against software requirements but against human use.

That is significant because digital archives do not remain static.

The software changes. The users change. New material arrives. New contexts emerge.

A system that works today but becomes progressively harder to understand may not be a successful long-term archive.

The real engineering objective is optionality

There is a broader lesson in the experience.

When choosing engineering software, one of the important questions is not simply whether a tool can perform today’s task. It is whether the overall system remains sustainable as requirements change.

The same applies to digital media infrastructure.

Can the data move?

Can the source be revised?

Can another website reuse the material without effectively claiming ownership of it?

Can a new distribution platform be introduced without rebuilding the archive?

Can an organisation distinguish between an editable source and a fixed delivery format?

These are architectural questions.

They also become increasingly important as organisations accumulate more digital information.

An engineering firm may have drawings and photographs from projects completed decades ago. A university may have research datasets, photographs and recordings that remain useful long after the original project has ended. A government agency may need to preserve records through multiple generations of information systems.

In all these cases, the cost of poor architecture may not become apparent when the system is first built.

It appears years later, when someone needs to find, reuse, modify or migrate something that was designed only for yesterday’s requirements.

Keeping the house while the roads change

The Trott Bailey Family’s experience began with something deceptively ordinary: a family documenting its life.

Almost two decades later, that record had become a substantial digital archive approaching 100,000 media files.

The number itself is not the most interesting part.

The more important question is what happens when an archive reaches a scale at which the traditional assumptions about media management begin to break down.

Big King Media emerged from that problem.

Its approach is based on keeping the source identifiable, allowing media to be reused without unnecessarily duplicating it, treating metadata as an important part of the archive and retaining the relationships that allow stories to evolve.

Kezideki Dynamic Movie extends the same thinking to video by separating the living structure of a story from the fixed format used to distribute it.

The underlying lesson is relevant far beyond a family media archive.

Digital infrastructure should preserve the information and relationships that make data useful, rather than optimising only for the easiest output today.

The platforms will continue to change. File formats will change. Publishing systems will change. New distribution channels will appear.

The engineering challenge is therefore not to predict which platform will dominate next.

It is to build the archive so that it does not have to care.

About the authors

Kimroy Bailey is a Jamaican systems developer working in Brazil and the principal engineer behind Big King Media. His work includes WordPress Multisite media architecture, shared-media relationships, dynamic visual publishing, archive retrieval, performance, automation and production stabilisation.

Sherika Trott Bailey is a Jamaican family-systems author and visual-systems designer working alongside Kimroy in Brazil. She has created and curated a large part of the Trott Bailey Family’s photographic and video record and contributes the editorial, visual and human-use requirements that shape the software.

Editorial note

Big King Media is developed and used by the Trott Bailey Family and is provided free. The authors are the creators of the software discussed in this article. This article is therefore presented as a first-party technical case study and not as an independent product review.

References

1. WordPress Developer Resources, wp_get_attachment_metadata, WordPress.org.
2. WordPress Developer Resources, Media REST API Reference, WordPress.org.
3. Meta Engineering, How Facebook Encodes Your Videos, 5 April 2021.
4. Kenya Engineer, The Engineer’s Software Stack: How to Choose the Right Tools Without Paying for the Wrong Ones, August 2026.