Metadata-Driven Ingestion Without One-Off Pipelines

Draft. An outline of something I worked out in practice and intend to write up properly.

What this will cover

The pitch for metadata-driven ingestion is easy: stop writing a pipeline per source. The hard part is designing it so it does not become a framework that only its author can operate, or a configuration language that has quietly reinvented programming.

The intended shape of the article:

• What genuinely belongs in configuration: connections, source objects, load behavior, transformations, validation, target tables
• What does not, and the warning sign that you have gone too far — configuration containing conditionals, then expressions, then something Turing-complete
• Designing for the tenth source rather than the first, and why uniform failure behavior is a bigger win than the time saved onboarding
• Schema evolution as a first-class concern rather than an exception handler
• Structured logging that answers what ran, over what, and what happened without reading the Spark UI
• Where a genuinely bespoke pipeline is still the right answer, and giving yourself permission to write one

The honest trade

A framework moves cost from per-source work into one shared thing that must always be right. That is usually the better trade, and it is not free — it concentrates risk, and it needs an owner.