Data Engineering Fundamentals

Foundational course covering data engineering concepts including pipelines, ETL processes, storage systems, and data architecture for aspiring data engineers.

My workspace28 minFree to watchDownloads

What you’ll learn

01Data Engineering FundamentalsWelcome to Data Engineering Fundamentals. This course gives you the core knowledge to design, build, and operate systems that deliver data reliably. If you work as an analyst, developer, or technical student, you already know that timely and correct data matters. Here, we go a step further and make that data available at scale. Data engineering is the foundation underneath analytics, machine learning, and everyday decision-making. The job is to ensure that data shows up on time, in good shape, and is ready to use. When a pipeline breaks, dashboards go dark and models become unreliable. That is why correctness and reliability come before cleverness. Across the course, we will cover the data landscape, architecture choices, storage systems, SQL, data modeling, and data quality. You will also complete a hands-on end-to-end pipeline project. By the end, you should understand the full data lifecycle, be able to compare architectural approaches, and have a working pipeline you can actually run. Next, we will map that landscape and look at the core roles involved.indeed.com databricks.com coursera.org+21 min
02The Data Engineering Landscape and Core RolesLet's map the landscape. You already know data roles overlap, so think about them by their primary output. The data engineer builds and operates the reliable data supply. The analyst interprets that data to explain what happened. The data scientist builds models to predict what happens next. And the ML engineer serves those models into production. These roles depend on each other, and data engineering is the foundation. From a systems view, that supply chain runs through a lifecycle. Data is generated by applications and sensors. It is ingested in batch or by streaming. It is stored, processed, transformed, and then served to downstream consumers. Governance wraps every stage, covering security, quality, and compliance. In smaller firms, you may own that entire lifecycle as a generalist. In larger organizations, you will likely specialize, focusing on pipelines, warehouse modeling, platform infrastructure, or streaming systems. But regardless of specialization, you are accountable for concrete outcomes. Pipeline SLAs, correctness, freshness, cost, and data quality are your metrics. A pipeline that fails silently is worse than one that fails loudly. Finally, AI-native tooling is shifting the work. Some glue code for dashboards is becoming unnecessary. That refocuses your effort on reliability, governance, and transformations that genuinely need engineering. Next, we will look at the storage layer itself: warehouses, lakes, and lakehouses.indeed.com databricks.com coursera.org+22 min
03Warehouses, Lakes, and LakehousesLet's look at the three main architectural patterns you'll encounter: data warehouses, data lakes, and lakehouses. A data warehouse is built for governed analytics. It uses schema-on-write, meaning data is structured and cleaned before it's stored, which delivers fast, reliable SQL and BI performance. A data lake takes the opposite approach. It stores raw data in its native format on low-cost object storage, and applies the schema only when the data is read. This gives you enormous flexibility for machine learning and exploratory work. The lakehouse is where these two converge. It pairs the economics of lake storage with warehouse-grade features like ACID transactions, schema enforcement, and strong query performance. In practice, warehouses struggle with ML flexibility, while lakes often become hard to govern. That's why many teams now choose a lakehouse for unified BI and ML, or intentionally combine patterns to serve different workloads. Next, we'll dig into the flow of data itself with batch, stream, and micro-batch processing.ibm.com datavidhya.com askantech.com+22 min
04Batch, Stream, and Micro-Batch ProcessingNow let's talk about the core processing patterns you will choose between: batch, stream, and micro-batch. Batch processing operates on bounded datasets. It runs on a schedule to produce reports, train machine learning models, or handle backfills. It is simpler to operate and generally costs less, but there is always a delay between when data arrives and when results are ready. Stream processing, on the other hand, handles unbounded events as they arrive. This is what powers fraud detection, IoT monitoring, and live dashboards where decisions must happen in seconds. The trade-off is real: you take on more complexity around state management, cost, and failure recovery. Micro-batch sits in the middle. It processes small groups of events every few seconds, giving you near-real-time results without the full weight of a continuous streaming system. The decision rule is straightforward. Start from the business deadline, not from the tool. If a daily report meets the need, batch is the responsible choice. If a delayed decision loses value or creates risk, then streaming earns its complexity. That principle will guide us as we move into core data storage concepts and file formats.risingwave.com fivetran.com modal.com+22 min
05Core Data Storage Concepts and File FormatsNow let's bring these ideas together around storage. The first decision is straightforward: relational databases give you transactional integrity, while NoSQL systems handle flexible, high-volume workloads. But when we talk about analytical file formats, the layout matters more than anything else. Columnar stores like Parquet and ORC physically group each column together. That means your queries can scan only the relevant fields and apply aggressive pruning, which is why they dominate analytical workloads. Avro, by contrast, is row-based. It shines when you are writing events quickly and when schema evolution across producers and consumers is a real concern. So the recommended pattern is simple: use Avro for event streams, then convert to Parquet for your analytical storage layer. Next, we'll apply these concepts to how data moves through the pipeline with ETL and ELT fundamentals.dataarchitect.studio amirulislamalmamun.com community.ibm.com+21 min
06ETL and ELT FundamentalsNow let’s look at the two core patterns for moving data: ETL and ELT. With ETL, you transform the data before it lands in the target system. With ELT, you load raw data first, and transform later inside the warehouse or lakehouse. Today, ELT is the standard for cloud warehouses and lakehouses, because it keeps raw data available for new use cases. But ETL still matters when privacy, compliance, or a destination system forces you to clean or restrict data before it leaves the source. In either pattern, the transformation work itself is similar. You clean bad values, cast types, join related tables, filter rows, and aggregate metrics. And in production, these pipelines rarely run on their own. Orchestrators like Airflow, Dagster, and Prefect manage scheduling, retries, dependencies, and backfills. So when you design a pipeline, the first decision is usually ETL versus ELT. The second is how you will operate it reliably. Next, we’ll move into SQL for data engineering.indeed.com databricks.com coursera.org+22 min
07SQL for Data EngineeringSQL is the primary interface you will use to query, transform, and validate data. It is the skill most likely to be tested in a data engineering interview, so treat it as a core working language rather than a reporting tool. At the baseline, you need real fluency with joins, common table expressions, window functions, and date logic. The engineering difference shows up in how you write that SQL. It should be reproducible, tested, incremental, and modular. That means version-controlled queries, clear naming, and transformations that can run repeatedly without side effects. Cleaning work sits within the same discipline. You will routinely handle deduplication, null replacement rules, explicit type casts, and integrity checks that catch broken joins or unexpected cardinality. If you think of SQL as the control surface for pipeline logic, not just a way to pull a report, you will be in the right frame. Next, we will look at data modeling basics, where these query patterns become reusable structures.indeed.com databricks.com coursera.org+21 min
08Data Modeling BasicsNow let's talk about data modeling basics. The core idea is to separate facts from dimensions. Facts are the numbers you measure, like revenue or quantity. Dimensions are the context you filter by, like date, product, or customer. This separation leads to the star schema, which has a central fact table linked to denormalized dimension tables. A key design decision in any star schema is grain. Grain defines exactly what one fact row represents, such as one row per order line item. Star schemas intentionally trade redundancy for simpler, faster analytical queries. The extra storage cost is worth it when queries run against millions of rows. To build a robust model, you also need surrogate keys and slowly changing dimensions. These techniques ensure you can accurately track history, even when source systems change. Next we'll move on to data quality essentials.1 min
09Data Quality EssentialsNow let's look at the essentials of data quality. The core dimensions you need to evaluate are completeness, uniqueness, consistency, timeliness, validity, and accuracy. Validity checks that your data uses expected formats, while completeness checks that required fields are actually present. Before you publish data, run those pre-publication checks. Look for nulls, duplicates, out-of-range values, stale data, and broken referential integrity. Remember that quality monitoring is different from observability. Monitoring checks the actual values against your rules. Observability checks the health of the pipeline itself, including things like volume changes, latency, and schema drift. One practical point is to match your thresholds to your primary consumers and the business purpose. Not every dimension matters equally for every dataset, and tighter checks may not be worth the trade-off. Next, we'll move into governance, metadata, and access control.1 min
10Governance, Metadata, and Access ControlNow let's look at how governance makes data trustworthy across its lifecycle. Governance secures data, documents it, and ensures it can be relied on by analysts and downstream systems. A central part of this is metadata, which describes structure, meaning, origin, and usage. Lineage goes a step further by tracking how pipelines change data over time, so you can trace any field back to its source. Access control then enforces permissions at multiple levels: the platform, a dataset, a column, or even a specific row. This lets you protect sensitive information without blocking all analysis. Data catalogs build on this by helping consumers discover, understand, and trust available datasets. You do not need a heavy framework to start. Define owners for critical data, document the most important datasets, and apply basic access rules first. That lightweight foundation often prevents the biggest risks. Next, we will explore modern data engineering tooling.indeed.com databricks.com coursera.org+21 min
11Modern Data Engineering ToolingLet's zoom out and look at the modern tooling layer as a whole. The key idea is that no single tool covers everything. You're working with a modular stack, typically split into ingestion, storage, transformation, orchestration, quality, and serving. For ingestion, Fivetran and Airbyte handle connectors, Kafka streams events, and dlt gives you a Python-first loading option. Storage usually means Snowflake, BigQuery, or Databricks, plus open table formats like Iceberg and Delta Lake. Transformation is overwhelmingly dbt. Orchestration comes down to Airflow, Dagster, or Prefect. When you evaluate these tools, think by category, but also weigh cloud fit, operational cost, pricing structure, and your team's actual skills. Next, we'll look at how to put these layers together when choosing the right architecture and tools.1 min
12Choosing the Right Architecture and ToolsSelecting an architecture is a decision problem, not a technology preference. Start with the business question, the data variety, the required latency, and what your team can actually operate. If your workload is BI-heavy, with governed and mostly structured data, a warehouse-first approach gives you the fastest path to reliable reporting. If you are ingesting mixed unstructured data and supporting data science, a lake paired with a warehouse avoids forcing raw data into a rigid schema too early. When business intelligence and machine learning need to share open data at scale, typically above one hundred terabytes, a lakehouse-first design reduces duplication and keeps both workloads on the same governed foundation. The key is to think decision-first. Identify the questions, the sources, and the skills you have, and resist the idea of a universal solution. Next, we will put these choices into motion by building your first data pipeline.ibm.com datavidhya.com askantech.com+21 min
13Building Your First Data PipelineNow let's talk about building your first pipeline, because this is where the concepts become real work. Start with extraction. Pull from a public API or a file, and land that raw data into a staging area. Do not modify it there. The next step is transformation. Clean it, shape it, and produce meaningful results using Python and SQL, then store that processed output separately. Two rules will save you from a lot of headaches later. Keep raw data immutable, and make your tasks idempotent. That means running a task twice should produce the same result as running it once, which matters when retries happen. For scheduling and handling failures, use Airflow or Prefect so the pipeline runs on its own. For a portfolio project, connect all of this end to end, and include a clear README plus Docker so someone can run it with a single command. This demonstrates you understand the full architecture, not just a single script. Coming up next, we'll explore ways to continue your learning journey.2 min
14Continue Your Learning JourneyWe have covered a lot of ground, so let's talk about where you should go from here. First, consolidate the fundamentals. That means SQL, Python, data modeling, and core ETL and ELT concepts. These are the skills that get tested in interviews and used every single day on the job. Next, build things. Start with one manual ETL process so you understand the mechanics. Then, wrap it in an orchestrated batch pipeline using a tool like Airflow. Once that is solid, challenge yourself with a streaming project or a lakehouse architecture. For deeper reference, keep Kimball's Toolkit, the DAMA DMBOK, and the official documentation for dbt, Airflow, and Spark close at hand. Do not just read them. Use them. Practice SQL under time pressure. Whiteboard data models. Walk through system design decisions out loud. These habits build the confidence you need for technical interviews. Finally, think about your path beyond the core role. Analytics engineering, platform engineering, MLOps, and streaming-first positions are all natural directions to explore. This course gave you the map. The next step is yours to take. Stay curious, keep building, and thank you for learning with us.indeed.com databricks.com coursera.org+22 min

Take the deck with you

Download this course as a file — free, no sign-up needed.

PDF handoutEvery slide page, ready to print or share.15 pages · 4.0 MBDownload
Narrated PowerPointThe deck that presents itself — every slide carries the digital human's narration video.15 pages · 15.2 MBDownload
PowerPoint slidesThe full deck as a .pptx — open it in PowerPoint, Keynote, or Google Slides.15 pages · 4.0 MBDownload

Free to use in your own training — please keep the PersonWise credit page at the end.

Have your own deck? Turn it into a course

Sources consulted

Web sources consulted while building this course.