Data Science Anti-Patterns: Better Alternatives

This training helps data scientists and analysts recognize common anti-patterns in their workflows and apply more effective, reliable alternatives.

A digital instructor presents all 14 pages. Hold “Ask” at any point and ask out loud — the answer comes from this course. No sign-up needed.

28 minFree to watchDownloads

What you’ll learn

01Data Science Anti-Patterns and Better AlternativesWelcome. This session is about data science anti-patterns, and the better alternatives that replace them. Here's our working definition. An anti-pattern is a practice that looks plausible, even professional, but reliably produces poor outcomes. We'll take a lifecycle-wide lens, from problem framing through governance. That means you'll see the same discipline applied to analysis, modeling, and product work. This is built for data scientists, analysts, machine learning teams, educators, and technical managers, so we'll keep the vocabulary shared and explain terms as we go. The good news is that anti-patterns are diagnosable. For each one, we'll trace four steps. The symptom you can observe. The root cause underneath. The alternative to adopt instead. And the measurement that tells you the change worked. Our roadmap covers framing, data, experimentation, modeling, deployment, tooling, and governance, and ends with a practical playbook. We'll also acknowledge real constraints, like deadlines, unclear requirements, and legacy pipelines, so every fix is proportionate. The goal is simple. Recognize the pattern early, choose a fix that fits your situation, and make the change measurable. Next, let's define what counts as an anti-pattern, and why they persist.scikit-learn.org proceedings.scipy.org developers.google.com+22 min
02What Counts as an Anti-Pattern, and Why They PersistLet's start by defining our terms, so we share a common vocabulary. An anti-pattern is a defective but plausible practice that keeps showing up across teams. It looks reasonable in the moment, yet it reliably produces weak results. The label helps because it gives teams a fast, shared way to name weak practice instead of debating it from scratch. Some anti-patterns are technical errors, like data leakage, where information unavailable at prediction time sneaks into training. Others come from missing context about how model results are actually used. So why do they persist? Often it's metric pressure, unclear decision ownership, or tooling defaults that make the easy path the wrong one. A RAND study interviewing sixty-five practitioners found the top failure cause was misunderstood problems. The practical takeaway is a diagnostic habit for practitioners and a review vocabulary for managers. Next, let's walk through the diagnosis loop, from symptom to root cause to alternative.scikit-learn.org proceedings.scipy.org developers.google.com+21 min
03The Diagnosis Loop: From Symptom to Root Cause to AlternativeLet's turn that pattern recognition into a repeatable method, the diagnosis loop. It has four steps. Observe the symptom, trace it to a root cause, choose an alternative, then measure whether it actually helped. Common symptoms sound familiar: brittle pipelines, analyses nobody can reproduce, models that degrade after launch, dashboards nobody opens, and gains that vanish before they reach a business metric. Before touching the model, check these causes first: misaligned incentives, unclear decision ownership, weak data contracts, and premature optimization. Here's a useful fact. In ninety six production ML postmortems at Google, most outages traced back to distributed systems, deployment, and data join failures, not to ML itself. So fixing the model often fixes nothing. And remember the difference between a deliberate tradeoff and an anti-pattern. A deliberate tradeoff has a known cost, an owner, and an expiry date. An unexamined default is just an anti-pattern waiting to bite. Next, we'll apply this lens to problem framing and metric anti-patterns.scikit-learn.org proceedings.scipy.org developers.google.com+22 min
04Problem Framing and Metric Anti-PatternsNext, let's look at how projects get framed, and where metrics go wrong. Three anti-patterns to name. First, starting from the data or the model you already have, instead of from the decision it should inform. Second, optimizing a proxy metric, an indirect stand-in that is only weakly tied to the real outcome. Third, vague success criteria, which become moving goalposts when results arrive. The better alternative is decision-first framing. Ask what decision this work changes, then separate business metrics from model metrics. Business metrics measure outcomes like revenue or task completion; model metrics measure quality like precision or recall. And treat proxies as provisional evidence. Document your assumptions, and add guardrail metrics so progress in one place doesn't hide harm elsewhere. That sets up the next slide, Why Proxy Metrics Betray Us: Goodhart in Practice.developers.google.com dl.acm.org ones.com+21 min
05Why Proxy Metrics Betray Us: Goodhart in PracticeLet's talk about why proxy metrics betray us, or Goodhart's law in practice. A proxy metric is an indirect stand-in for the outcome you actually care about. Goodhart's law says that once a measure becomes a target, it stops being a good measure. Once a proxy becomes the target, experiments get selected for the proxy, not for the product outcome. There are three common failure shapes. Incomplete coverage, where a high average accuracy hides costly false positives. Component-level metrics, where lower model latency does not reduce user-visible response time because another stage is the bottleneck. And model-as-judge, where a separate model rewards verbosity without adding usefulness. Correlation traps catch teams too. User-level correlation and naive treatment-effect correlation both mislead, because of confounders and correlated measurement error. Proxy assumptions like unconfoundedness, exclusivity, and comparability are almost certainly violated. The real question is how badly. So what do we do instead? Map each metric to value, write a success contract before experimenting, compare proxy and product metrics after deployment, and add guardrail metrics to catch harm. Next, we turn to data collection and preparation anti-patterns.developers.google.com dl.acm.org ones.com+22 min
06Data Collection and Preparation Anti-PatternsLet's turn to data collection and preparation. This is where many projects quietly go wrong. The first anti-pattern is treating raw data as ground truth. When provenance, that is the record of where data came from and how it was processed, is missing, teams make assumptions that lead to costly wrong conclusions. The second is silent schema drift, undocumented joins, and unchecked quality issues that sink projects early. So name and triage the issues explicitly: missing values, duplicates, inconsistencies, wrong or outdated records, malformed entries, schema violations, and dangling references. And remember the big data paradox: volume never guarantees validity. Detailed capture metadata is a prerequisite. Instead, adopt data contracts, track provenance, run automated tests on ingest and on transform, and keep producers and users collaborating closely. Next, we look at data leakage, the silent inflation of model quality.scikit-learn.org proceedings.scipy.org developers.google.com+21 min
07Data Leakage: The Silent Inflation of Model QualityNext, let's talk about data leakage, the silent inflation of model quality. Data leakage is when information that won't be available at prediction time shapes your training. The symptom is predictable. Validation looks great, and production fails. One of the top routes is fitting scaling, imputation, feature selection, or augmentation before the split. The fix is simple. Split first, never fit on test data, and put preprocessing inside a pipeline. Then check the whole leakage family explicitly. That means similarity leakage, where near-duplicates land in different splits. Group leakage, where rows from the same user appear on both sides. And temporal leakage, where future information leaks into the past. Use grouped and time-aware splits. For guardrails, add automated leakage checks, provenance-based screening, and a documented preprocessing contract per model. I know deadlines and legacy pipelines make this hard, so start with one check on your most important model. That brings us to EDA and experimentation anti-patterns.docs.aws.amazon.com doi.org doi.org+22 min
08EDA and Experimentation Anti-PatternsLet's look at anti-patterns in exploratory data analysis and experimentation. First, hypothesis fishing. You run many tests, report the attractive findings, and forget how many you ran. The symptom is a result nobody can reproduce. Instead, pre-register the hypotheses before you look at the data. Second, peeking. Checking a p-value every day and stopping the moment it clears zero point zero five can inflate false positives five to ten times. Field evidence backs this up. In one study of over two thousand commercial experiments, about seventy-five percent of effects in p-hacked tests were truly null. Third, underpowered tests, and treating observational correlation as a causal effect. The fix is to run a power analysis up front, and pre-agree on sequential stopping rules. And keep operational checks, like logging and sample ratio mismatch, separate from success monitoring. Deadlines are real, so start with one small habit: write your hypothesis and stopping rule before launch. Next, let's turn to modeling and evaluation anti-patterns.dl.acm.org ubc-stat.github.io gwern.net+22 min
09Modeling and Evaluation Anti-PatternsNext, let's look at modeling and evaluation anti-patterns. The first one is leaderboard chasing. It's tempting to keep pushing a model to beat the top score. But without a simple baseline, like predicting the majority class, a weak model can look good by accident. So start with a baseline first. Then compare against it honestly. The second trap is accuracy. When classes are imbalanced, accuracy misleads. A model that always predicts the majority class can score ninety nine percent. Instead, use MCC, the Matthews correlation coefficient, or PR AUC, the area under the precision recall curve. These give a clearer picture of real performance. The third anti-pattern is repeated test set use. Every time you tune on the test set, information leaks into your evaluation. Do the tuning on a validation set. Save the test set for one final, honest check. To put this into practice, use three way splits, report results by slice, and check calibration curves. A calibration curve shows whether predicted probabilities match observed frequencies. For overconfidence, proper scoring rules like log loss and the Brier score help. These are called proper because they are minimized when you report your true probabilities. So keep baselines honest, choose metrics that fit your data, and protect your test set. Now, let's move on to deployment and monitoring anti-patterns.docs.aws.amazon.com doi.org doi.org+22 min
10Deployment and Monitoring Anti-PatternsNext, let's talk about what happens after a model ships. The first anti-pattern is deploy and forget. You release the model, and nobody owns it. No monitoring, no alert thresholds. The first sign of trouble is a business stakeholder noticing bad predictions. The second is having no rollback path and no incident playbook, so the first failure becomes a crisis. Now, drift comes in three flavors. Data drift means the inputs change. Concept drift means the relationship between inputs and the target changes. Prediction drift means the output distribution shifts. Here is the better alternative. When you deploy, capture baselines: training data distributions and validation predictions. Then monitor in layers, starting with data quality, then input drift, then prediction drift. Use shadow or canary releases so a bad model never reaches everyone at once. And retrain on drift triggers, not fixed schedules. Calibrate alert thresholds with domain knowledge, because a threshold that fires every day just trains people to ignore it. Deadlines and legacy pipelines are real, so start with a baseline and a named owner. Now let's move on to tooling, automation, and reproducibility anti-patterns.2 min
11Tooling, Automation, and Reproducibility Anti-PatternsNext, let's talk about tooling and reproducibility. A common anti-pattern is the notebook-only workflow with hidden state. You run cells out of order, and the results depend on what happened earlier in your session. Anything done by hand should be done by syntax instead, so the steps live in code, not in memory. Another trap is the one-person environment that never reproduces. Reproducibility needs three things: workflow logic, a pinned environment, and a durable code identity, meaning a persistent record of the exact code version. And the opposite failure is over-engineering, buying a heavy platform for a small analytical question. The better path is modest and practical: version control tied to commits, a parameterized pipeline you can run with one command, lockfiles or containers for the environment, and continuous integration tests. Apply rigor proportionally to scale and impact, and test by rebuilding from scratch on a clean machine. That test is the honest measure. As we move on, let's look at team process, governance, and communication anti-patterns.2 min
12Team Process, Governance, and Communication Anti-PatternsNow let's look at the people side: process, governance, and communication. Three anti-patterns show up again and again. First, hero culture. Critical pipelines depend on one person, and no deputy is recorded. When that person is on vacation, nobody can ship. Second, no decision owner. If nobody decides with a dashboard, it goes unused. Third, governance bolted on as a final gate, instead of designed into the workflow. The fix is straightforward, though not always easy. Name one accountable owner per data domain or model, and document decision rights, meaning who decides, who is consulted, and who is informed. Then set a cadence: always-on alerts, weekly triage, monthly owner review, quarterly risk review. And measure behavior, not activity. Track owned products, overdue quality issues, and repeated incident causes. This keeps ownership real while deadlines, ambiguous requirements, and legacy pipelines still exist. Next, we move on to the Better Alternatives Playbook: One Reference Table.1 min
13Better Alternatives Playbook: One Reference TableNow here is the playbook on one page. Think of it as a reference table you can pin to your team wiki. Five rows, each naming a better alternative. Framing first. Shift to decision-first intake. Before any modeling, write down the decision, the owner, and a success contract. What outcome counts as success, and who decides. Data next. Use data contracts, automated quality tests, and leakage-aware preprocessing. Leakage means information that would not be available at prediction time sneaks into training, so fit your preprocessing on the training data only. Then experimentation. Pre-register your hypotheses, and agree on sequential stopping rules before you look at results, so you do not stop early just because a result looks good. Modeling. Always build a baseline first. Evaluate by slice, not just overall averages, and run calibration checks, meaning the predicted probabilities match observed rates. Operations. Monitor against baselines, keep a rollback path, and name an owner and a deputy for every model. Notice these are cheap habits, not extra projects. Pick one row this week and make it standard practice. Next, we make this personal with a thirty, sixty, ninety day plan.scikit-learn.org proceedings.scipy.org developers.google.com+22 min
14Your 30-60-90 Day Plan, for Individuals, Teams, and EducatorsLet's close with a practical plan. Prioritize upgrades by impact, effort, and reversibility, meaning how easily you can undo the change if it goes wrong. In the first thirty days, pick three anti-patterns, give each one a named owner, and define the evidence that will show success. For example, one owner takes the feature leakage check, with a written test and a pass result as proof. In days thirty-one to ninety, add one data contract, which is an agreed schema and freshness promise for a key dataset, plus one leakage check and one monitoring baseline. Managers and educators, run case reviews on real projects and build a small pattern library your team can reuse. The goal is deliberate, measurable, repeated improvement, not perfection. Let the constraints be real, whether that is a deadline or a legacy pipeline, and take one honest step this week. Thank you for joining me, and I hope the next upgrade you ship is a little easier and a lot more deliberate.2 min

Take the deck with you

Download this course as a file — free, no sign-up needed.

PDF handoutEvery slide page, ready to print or share.15 pages · 3.2 MBDownload
Narrated PowerPointThe deck that presents itself — every slide carries the digital human's narration video.15 pages · 12.9 MBDownload
PowerPoint slidesThe full deck as a .pptx — open it in PowerPoint, Keynote, or Google Slides.15 pages · 3.2 MBDownload

Free to use in your own training — please keep the PersonWise credit page at the end.

Have your own deck? Turn it into a course

Sources consulted

Web sources consulted while building this course.