Data Engineering Platform Selection and Architecture

This training helps data engineers and architects select and design data engineering platforms, covering architecture patterns and key tradeoffs for informed decisions.

A digital instructor presents all 14 pages. Hold “Ask” at any point and ask out loud — the answer comes from this course. No sign-up needed.

28 minFree to watchDownloads

What you’ll learn

01Data Engineering Platform: Selection, Architecture, and TradeoffsWelcome. Over the next fourteen slides, we are going to build something practical: a working decision framework for selecting a data engineering platform and defending the tradeoffs you accept along the way. This is not a product tour. It is a peer-level conversation aimed at data engineering leads, architects, platform teams, analytics leads, and our security partners. Here is the core tension. Every choice you make balances five things at once: governance, speed, cost, reliability, and your team's operating capability. You rarely optimize all five. You decide, explicitly, which one absorbs the compromise. By the end, you should be able to compare platform archetypes, design a reference architecture, and run a disciplined build, buy, or assemble decision. We will work through the landscape, the reference architecture, a selection scorecard, total cost of ownership, security, the operating model, and migration. We will also settle a shared vocabulary, terms like archetype, open table format, medallion, federation, catalog, lock-in, and gold path. One framing note for 2026. Composable ecosystems are replacing single-architecture bets, and AI workloads are the forcing function. Let's start with the landscape and buyer context in 2026.thoughtworks.com forrester.com forrester.com+22 min
02Platform Landscape and Buyer Context in 2026Let's set the landscape before we get into selection mechanics. In 2026, the platform categories we compare are warehouses, lakehouses, data lakes, streaming systems, query engines, and integration layers. No single vendor wins all of that. Buyers assemble capabilities into a layered stack. Forrester's Q3 2026 lakehouse Wave is explicit: storage and query performance are now table stakes, and differentiation shifts to governance, openness, and AI readiness. Dresner's 2026 data supports the same conclusion. Fifty-four percent of organizations still run a warehouse, the lakehouse sits at twenty-nine percent adoption, and the lake is accelerating with AI maturity. That is a three-layer stack, not a migration race. So as you evaluate, watch four things: total cost of ownership across compute, storage, egress, and engineering time; lock-in, especially at the catalog layer; operational load, like table maintenance and governance work; and the temptation of feature-checklist or demo-driven buying. Set hard gates for security, residency, and exit before you score anyone. Next, we go deeper into each pattern. Platform Archetypes: Warehouse, Lakehouse, Lake, Mesh, Fabric.thoughtworks.com forrester.com forrester.com+22 min
03Platform Archetypes: Warehouse, Lakehouse, Lake, Mesh, FabricLet's walk through the platform archetypes you'll actually choose between. First, the cloud warehouse. Storage and compute are coupled, SQL concurrency is strong, and governance is on by default. That coupling is a real tradeoff. You pay for predictable performance with less flexibility and tighter vendor lock-in. Next, the lakehouse. Open table formats on object storage, ACID transactions, multi-engine access. It's the fastest-growing pattern, and it decouples storage from compute. The cost is operational complexity. You now own compaction, catalog choice, and table maintenance. Then the raw lake. Cheapest per byte, schema-on-read, but no transactions and no governance. It's a landing zone, not a platform. The mesh is different. It's an ownership model, not a technology. Domain ownership, data as a product, federated governance. Expect twelve to twenty-four months and higher headcount. Fabric is the metadata and integration layer, weaving disparate systems together with active metadata, lineage, and policy enforcement. Now the 2026 synthesis. These are not competing options. Leading implementations combine lakehouse storage, fabric metadata, and mesh ownership. The real decision isn't which architecture. It's whether you can govern, cost-control, and AI-enable whatever you build. Next, we'll look at the reference architecture, layers, ingestion, and storage decisions.thoughtworks.com forrester.com forrester.com+22 min
04Reference Architecture: Layers, Ingestion, and Storage DecisionsLet's move into the reference architecture. Eight functional layers: ingestion, storage, transformation, query, semantics, governance, orchestration, and A I. Two principles anchor everything else. First, federate by default, ingest by exception. If a source can be queried in place, ingesting it just creates another copy to govern, secure, and keep current. Only ingest when there's a specific need: query performance, history the source cannot retain, or protecting a fragile system. Second, table format is the most consequential decision here, and only theoretically reversible. Treat it as deliberate, not a default. Then partition for your query patterns, and schedule compaction from day one, because small files degrade performance quietly. On build order: query and federation first, over what already exists. Then catalog and governance, before new workloads land. Storage comes third. That sequence delivers value in weeks and makes governance a precondition, not a retrofit. Next, we'll look at processing patterns: batch, streaming, and the medallion model.docs.databricks.com yukon.az bigdataboutique.com+21 min
05Processing Patterns: Batch, Streaming, and the Medallion ModelLet's move to processing patterns, where most architectural mistakes get made. The medallion model gives each layer one job. Bronze is append-only raw landing, faithful to source, and it's your replay source of truth. Never clean in bronze. Silver is where you clean, conform, deduplicate, and type. Gold holds business-ready aggregates and dimensional models for BI, ML, and dashboards. Now the key design principle: one transformation graph, streaming and batch differ only by trigger. Same code, same tables. Use MERGE for upserts and slowly changing dimensions, and always guard the update with an event-time predicate, or a late-arriving stale row silently overwrites a newer one. Maintain gold incrementally through change feeds, not nightly full rebuilds. And avoid Lambda's dual paths, because they drift and produce two different revenue numbers. Finally, plan compaction and set explicit latency and cost targets per tier. Next, federated and centralized architecture, choosing where truth lives.docs.databricks.com yukon.az bigdataboutique.com+22 min
06Federated and Centralized Architecture: Choosing Where Truth LivesNow let's decide where truth actually lives. Centralized architecture gives you uniform policy, one audit trail, and simpler compliance validation. It also wins on optimization for known, repeated, schema-stable queries. A morning sales dashboard hitting the same twelve queries will run faster on a tuned central warehouse than on a federated system querying five live sources. Federation gives you live operational state, no copies to synchronize, and residency by construction. But be clear about its limits: joins are network-bound, self-hosted engines hit concurrency ceilings, and access is generally read-only. The workable answer is usually tiered hybrid. Centralize shared foundations like master data and policy. Materialize datasets that prove their value through real usage. Federate the rest. Watch for two failure modes. The central swamp, where everything is copied in and ownership dissolves on arrival. And distributed chaos, where the CFO asks why revenue differs across reports. So as we move on, keep this question in mind: which datasets justify a synchronized copy, and which should be queried in place? Let's turn to the Selection Criteria and an Evidence-Based Evaluation Scorecard.digiqt.com promethium.ai blog.everpuredata.com+22 min
07Selection Criteria and an Evidence-Based Evaluation ScorecardLet's talk about how to evaluate platforms without getting steered by a good demo. Start with your use cases and workload profile: latency, concurrency, volume, retention, and growth. Those numbers set the bar before any vendor conversation, and they should come from your own production data, not a feature matrix. Next, split hard gates from weighted scores. Residency, identity and access, connectivity, recovery, budget boundary, and exit clauses are pass or fail. A failed gate cannot be offset by a higher total. Then set weights before vendor workshops. If you weight after seeing scores, you are reverse-engineering a justification for a decision you already made. Score functional, governance, cost, and portability criteria on evidence, not promises. Capabilities on a slide prove nothing about your environment. Methods that hold up: a real-workload proof of concept, P95 and P99 latency, failure replay and recovery exercises, and an operating-effort log across engineering, governance, and support. Keep the evidence beside every score, so the shortlist is auditable and defensible. Build versus Buy versus Assemble: A Layered Decision.elvenite.se moderndata101.com googlesql.com+22 min
08Build vs. Buy vs. Assemble: A Layered DecisionLet's move from general criteria to a concrete decision model. The first fix here is mental: stop treating build versus buy as one binary bet. A data platform is a stack of layers, and each layer has its own economics. Buy the commodity layers. Storage, compute, orchestration, and governance are solved problems, so buy them and skip the undifferentiated heavy lifting. Build the opinionated layer yourself: your semantic definitions, your domain models, your metrics. No vendor sells your business. Be clear-eyed about managed services. They win on speed, often first value in weeks. But egress costs and consumption pricing surprise buyers around month eighteen, so model that before you sign. Build becomes defensible when the platform is the product, or when regulatory control is genuinely unusual. Assembly is the third path, and it is not free: you trade vendor control for owning integration and upgrades. Whichever path you take, plan your exit early. Open formats like Iceberg and Parquet, metadata export, and portability clauses in the contract. Negotiate exit terms before you sign, not when you are trying to leave. Next, we will look at Total Cost of Ownership and Lock-In Economics.elvenite.se moderndata101.com googlesql.com+22 min
09Total Cost of Ownership and Lock-In EconomicsLet's turn to total cost of ownership and lock-in economics. The core rule: price the same thirty to ninety day production workload across every candidate, never advertised rates. Model the full picture. Storage, compute, ingestion, egress, governance, table maintenance, staffing, and exit. Then measure idle compute and autoscaling explicitly, because paying for capacity with no useful work running is one of the largest hidden line items. Watch for the bundled versus unbundled trap. Raw storage rates pretend service work is free, when in reality your engineers absorb it. Here is the crossover that matters. Below a certain volume and concurrency, self-managed lakehouse engineering costs more than the compute markup you avoid. Above it, the markup dominates and the lakehouse wins. Lock-in has also moved. It now lives in the catalog, not just the storage format, because open files nothing else can find are not portable. Finally, model commitment discounts separately and validate them against expected utilization. A discount only lowers total cost when the committed capacity is consistently used. Tradeoff Matrix: Performance, Cost, Governance, and Speed.2 min
10Tradeoff Matrix: Performance, Cost, Governance, and SpeedSo let's put the tradeoffs side by side, because this is where architecture decisions get made. Doubling capacity costs more than it delivers in performance. In Snowflake or Databricks, doubling warehouse size more than doubles hourly cost for less than double the gain. Batch, micro-batch, and streaming trade freshness against complexity and cost. Central governance gives control; domain autonomy gives speed and accountability. Open formats buy optionality, but you inherit compaction and snapshot-expiry work. Managed services speed time to value, while portability decides your leverage later. And every one of these choices should live in a decision record with a named owner. That record is what lets you revisit a call when costs shift or requirements change. Now, before we look at the security implications of these choices, let's move to: Security, Privacy, and Compliance Architecture.digiqt.com promethium.ai blog.everpuredata.com+21 min
11Security, Privacy, and Compliance ArchitectureLet's turn to security, privacy, and compliance architecture. Start with the shared responsibility line. The vendor secures the service itself. Identity, policy, and evidence remain yours. So on day one, map who owns what. Second, tag-based access control scales. Per-table grants do not survive dataset growth. Tag data by domain, region, sensitivity, and layer, and let policies inherit automatically. Third, row filters, column masks, and tokenization must be enforced across every query engine, not just your primary one. Fourth, watch the bypass. If users can read the object store directly, they can route around fine-grained controls, so lock that path down. Fifth, tie purpose-based access, retention, and erasure workflows to lineage. Finally, treat auditability as evidence. Lineage, access logs, and policy version stamps are what auditors actually want. Next, we look at the operating model, team topology, and governance.elvenite.se moderndata101.com googlesql.com+22 min
12Operating Model, Team Topology, and GovernanceNow let's talk about the operating model, team topology, and governance, because technology alone will not save an unclear operating model. First, choose your topology. Centralized, embedded, or hybrid platform plus embedded. Most scaling organizations converge on hybrid. A central platform team owns the shared substrate, while embedded engineers build on a paved road. Treat the platform as a product, not a service desk. Publish golden paths, service level objectives, and run substrate on-call so the platform team gets paged, not every domain. Before any tooling decision, publish the ownership map. Name who owns the top tables, who approves schema changes, and who accepts the two a.m. page. Governance should be federated. Run an architecture review forum, record decisions as architecture decision records, stand up a metrics council, and integrate security review. Data product owners own freshness, quality, availability targets, and deprecation. On funding, separate a shared platform baseline from chargeback or showback, and map cost to named owners. Finally, track platform health through adoption, cost per query, and time to onboard a new source. Next, we'll look at the implementation roadmap and migration strategy.digiqt.com promethium.ai blog.everpuredata.com+22 min
13Implementation Roadmap and Migration StrategyLet's walk through the implementation roadmap. The sequence is deliberately conservative: assess, architect, migrate in waves, then operate. The one step teams are most tempted to skip is the pilot, and skipping it is almost always a mistake. Start with a ninety day usage audit. In most estates, thirty to fifty percent of tables and reports have no consumers at all. Removing that dead weight is the single largest cost saving in the whole program, and it only costs a few weeks. Then re-platform by business domain, end to end, not by technical layer. When you validate, run source and target in parallel and compare at three levels: row counts, aggregates, and record level samples. Explain every variance, because unexplained differences are blockers, not warnings. Cut over reads first, then writes, and keep the legacy platform read only as a bounded rollback window, measured in days, not until we feel confident. Optimize after go live: partitioning, incremental loads, and workload isolation, with cost reviews at thirty, sixty, and ninety days. Next, we will apply this framework to a live scenario in our decision workshop.2 min
14Decision Workshop: Applying the Framework to a Live ScenarioLet's close by putting the framework to work on a live scenario. Picture a multi-domain enterprise with business intelligence, streaming, and AI workloads, plus hard residency limits. Before anyone scores a candidate, agree the hard gates and the weights. A failed security, residency, or recovery requirement cannot be offset by a higher total. Weight before evaluation, so you are not bending the rules to favor a good demo. Every score needs documented evidence, not vendor claims. Keep the evidence beside the score, or it is not auditable. Then assign owners: who runs the catalog, who takes the on-call page, who funds the shared platform. Name those people early, or you inherit orphaned pipelines. Record the decision as an architecture decision record, including rejected options and the conditions that would trigger a reversal. Finally, avoid three common pitfalls: feature-checklist buying, skipping the pilot, and letting a parallel run become permanent. Run the workshop, decide plainly, and defend the call with evidence.elvenite.se moderndata101.com googlesql.com+22 min

Take the deck with you

Download this course as a file — free, no sign-up needed.

PDF handoutEvery slide page, ready to print or share.15 pages · 3.4 MBDownload
Narrated PowerPointThe deck that presents itself — every slide carries the digital human's narration video.15 pages · 28.6 MBDownload
PowerPoint slidesThe full deck as a .pptx — open it in PowerPoint, Keynote, or Google Slides.15 pages · 3.3 MBDownload

Free to use in your own training — please keep the PersonWise credit page at the end.

Have your own deck? Turn it into a course

Sources consulted

Web sources consulted while building this course.