Data Science Troubleshooting Essentials

This training helps data scientists and analysts diagnose and resolve common data science issues, building practical troubleshooting skills for real-world projects.

A digital instructor presents all 15 pages. Hold “Ask” at any point and ask out loud — the answer comes from this course. No sign-up needed.

30 minFree to watchDownloads

What you’ll learn

01Troubleshooting Common Data Science ProblemsWelcome. Over the next fifteen slides, we will build a shared approach to troubleshooting common data science problems. The goal is simple. When something looks wrong, you should know what to check first, and why. Let's start with the big picture. Troubleshooting is a core competency, not an afterthought. In practice, most of your effort goes into diagnosing and fixing silent failures. The model is rarely the blocker. In 2025 and 2026, real failures cluster in data, metrics, workflow, and monitoring, not in the algorithm itself. So we will use a shared vocabulary: symptom, root cause, mitigation, and prevention. And we will organize the material around six failure zones: data, leakage, training, reproducibility, production, and process. For managers and learners, the impact is straightforward. Fixing these zones earlier reduces rework and protects trust in your results. Next, we look at the data science lifecycle and where it breaks.onecdn.io thinklytics.com exa.ai+21 min
02The Data Science Lifecycle and Where It BreaksLet's look at the lifecycle as it actually runs. Problem, data, cleaning, exploratory analysis, features, training, evaluation, deployment, and monitoring. Here is the key point. Failures rarely cluster at model training. They cluster in data understanding, data preparation, and post-deployment. And failure propagates. A cleaning fault can resurface weeks later as an unexplained accuracy drop. Faults come from data, from frameworks and tooling, and from conceptual or resource constraints. So practitioners need fault localization, the ability to trace a symptom back to its source. Managers need clear ownership and process gates, so problems are caught before they reach production. That distinction drives everything that follows. Next, we focus on data quality and integrity issues.learn.microsoft.com docs.cloud.google.com ppml.dev+21 min
03Data Quality and Integrity IssuesLet's turn to data quality and integrity, the first place most training problems actually originate. Start with missing values. Before you impute, drop, or flag, identify the mechanism: missing completely at random, missing at random, or missing not at random. That distinction determines whether a fix is safe or silently biases the model. Next, outliers. Combine rule-based checks with statistical or machine learning detection, then decide to cap, correct, or keep them. For duplicates and entity resolution, one customer appearing under many IDs distorts both training and evaluation. Label noise is equally serious. Mislabeled targets and inconsistent annotation guidelines can look exactly like an architecture failure, so check labels before you touch the model. Across the workflow, a taxonomy of twenty-three quality problems maps to four detection strategies: rules, statistics, reference data, and human review. The impact is direct: cleaner data means more trustworthy metrics and fewer wasted training cycles. Next, we look at treating data quality as first-class engineering.standards.iteh.ai cell.com cdn.standards.iteh.ai+22 min
04Treating Data Quality as First-Class EngineeringLet's treat data quality as first-class engineering, not cleanup work. Data tests are the unit tests of the pipeline. Run them on every ingest and every transform. If a table arrives with a broken schema or a shifted range, you want that caught before training, not after. The ISO and IEC 5259 series gives multidisciplinary teams a shared quality vocabulary and governance model, so engineers, analysts, and auditors describe the same problem the same way. Beyond accuracy and completeness, machine learning adds its own dimensions: annotation quality, fairness, privacy, version control, and synthetic data provenance. Temporal data carries extra risk. Watch for gaps, irregular sampling, timezone and daylight-saving errors, and duplicated rows. For managers, each check you add converts a silent model failure into an early, cheap signal. Start small. Ship a versioned validation suite covering schema, null, range, and distribution checks. Then extend it as you learn. Next, we look at data leakage, the silent accuracy inflator.standards.iteh.ai cell.com cdn.standards.iteh.ai+22 min
05Data Leakage: The Silent Accuracy InflatorLet's move to data leakage. It's the silent accuracy inflator. Leakage means the model learns from information it won't have at prediction time, so your scores look strong but fail in deployment. There are four common forms: train and test overlap, preprocessing leakage, multi-test leakage, and target leakage. Classic offenders include engineering features before splitting, target encoding on the full dataset, and outcome features like days until payment. Static tools catch this early. NBLyzer found twenty-five true leakages across two thousand one hundred eleven Kaggle notebooks, at roughly ninety-three percent precision. LeakageDetector plugins in your IDE flag overlap, preprocessing, and multi-test issues and offer automatic quick fixes. For managers, that means fewer surprises after launch. For learners, it's a habit worth building now. The prevention rule is simple: split first, fit transformers on training data only, and drop any feature unavailable at prediction time. Next, we'll look at splitting, cross-validation, and metric pitfalls.arxiv.org caterinaurban.github.io arxiv.org+22 min
06Splitting, Cross-Validation, and Metric PitfallsLet's talk about how you split and evaluate data, because this is where reliable results are won or lost. First, match the split to deployment. Use random splits for independent rows, temporal splits for time series, and group-aware splits when records share an entity like a customer. Never shuffle time series. A random split leaks the future into training and inflates your scores. Reserve a validation set for tuning, and touch the test set once. If you reuse it, you are no longer measuring generalization. On metrics, accuracy misleads under class imbalance. A fraud model can score ninety-nine percent accuracy by predicting nothing. Use precision, recall, precision-recall AUC, and calibration instead. Finally, version your datasets, record the split strategy, and flag benchmark contamination in large language models, since overlapping evaluation data inflates scores. Keep splits honest. Next, diagnosing model performance problems.arxiv.org caterinaurban.github.io arxiv.org+21 min
07Diagnosing Model Performance ProblemsLet's move on to diagnosing model performance problems. Start with learning curves. If training error keeps dropping while validation error rises, you have overfitting. If both curves stay high and flat, you have underfitting, so add capacity or better features. If results swing between runs, that is normal for stochastic models. Average over multiple random seeds before you draw conclusions. Now, class imbalance. Resampling, class weights, and threshold tuning are tools, not defaults. Pick them based on the metric that matters to the business. For example, in fraud detection, recall usually matters more than raw accuracy. But before any of that, do error analysis. Inspect the misclassified and high loss examples, and check the labels themselves. Mislabeled data is a very common root cause. That step saves time and points you to the right fix. Next, we look at Debugging Training Loops and Silent Training Errors.learn.microsoft.com docs.cloud.google.com ppml.dev+22 min
08Debugging Training Loops and Silent Training ErrorsLet's turn to debugging training loops and silent training errors. Loss and accuracy are noisy and periodic, so they miss silent errors that never throw an exception. TrainCheck caught eighteen of twenty real silent errors within a single iteration. Root causes split about thirty-two percent user code and thirty-two percent framework bugs. The classic offenders are familiar. A missing zero_grad call, poorly configured gradient clipping, and passing softmax outputs into a loss that expects logits. So monitor per-layer gradient norms, and use adaptive thresholds to catch vanishing or exploding gradients. For discipline, use the smallest config, one GPU, ten steps, and overfit a single batch before scaling up. That habit alone saves teams days of wasted compute. Next, we look at numerical stability, out-of-memory errors, and large-model failures.learn.microsoft.com docs.cloud.google.com ppml.dev+21 min
09Numerical Stability, OOM, and Large-Model FailuresNow let's look at numerical stability, out of memory errors, and large-model failures. NaN and Inf values almost always begin in an exponential, logarithm, or division operation. Once they appear, they spread through the entire graph. So instrument, don't guess. Log gradient norms, activation statistics, and weight update ratios. For out of memory errors, the usual causes are oversized batches, duplicated operations, memory leaks from repeated model creation, or loading the full dataset into memory. At large scale, watch for loss stalls, scheduler failures, communication overhead, and mixed-precision errors, often from float sixteen overflow. The impact is real. A silent fault on a sixty-four GPU run can waste days of compute and delay delivery. So never debug a sixty-four GPU run before a ten-step, one-GPU sanity check. That small test usually catches the root cause in minutes. Next, we move on to reproducibility and environment failures.learn.microsoft.com docs.cloud.google.com ppml.dev+21 min
10Reproducibility and Environment FailuresLet's talk about reproducibility, and the environment failures behind that familiar phrase, it works on my machine. The usual causes are dependency drift, differences in operating system or architecture, unseeded randomness, and undeclared data versions. The practical fix is to version everything, not just code. Version your datasets, your models in a registry, and your experiment tracking. The pattern that holds up today is lockfiles, containers, GPU-aware environment managers, and layered virtual environments such as venvstacks, which let application layers share one copy of PyTorch in a framework layer instead of duplicating it in every archive. Expect a tradeoff. Determinism costs some speed. Caching and per-run receipts buy back verifiability, and for a manager, that means results you can defend in a review. Before any handoff, run this checklist. Seed everything, pin dependencies, snapshot the data, and record the environment for each release. Next, we look at production failures, specifically skew, drift, and silent decay.1 min
11Production Failures: Skew, Drift, and Silent DecayNow let's look at production failures, where the real trouble is usually skew, drift, and silent decay. Training-serving skew is the first culprit. Schema skew means the format or distribution of serving data no longer matches training. Feature skew means the engineering logic itself differs between the two paths. Either one can cause damage equal to a production bug, and both are easy to miss. Drift is not one thing. Covariate shift, concept shift, and benign drift each call for a different response. Retraining is not always the answer, and treating every shift as deterioration just adds noise. The dangerous pattern is silent failure. The model stays confident while quality quietly decays, and up to twenty percent of failures reach customers before anyone detects them internally. Without labels, you still have signals. Track population stability, MMD joint-distribution tests, output drift, conformal coverage, and CUSUM change-points. One caution: confidence degradation does not reliably precede drift. Track it, but do not treat it as your early warning. The takeaway is simple. Detect drift, distinguish its type, then act. Next, we move into the diagnostic workflow and root cause analysis.docs.cloud.google.com onecdn.io thinklytics.com+22 min
12Diagnostic Workflow and Root Cause AnalysisLet's walk through a diagnostic workflow you can reuse for root cause analysis. Start with a repeatable triage loop. Capture the symptom, form a hypothesis, gather evidence, test it, then confirm. Localize before you theorize. Isolate the failing stage and reproduce it on a small sample. That narrows the search without burning compute. Then binary-search across your pipeline steps. Check the midpoint. If it's clean, the fault is downstream. If not, it's upstream. This shortens the path to the fault fast. Now, one caution. Machine learning root cause analysis differs from service RCA, because the causal graph is often unknown. You may not know which feature or stage truly drives the outcome. So lean on logs, traces, and data or model tests as your evidence base. When you escalate, bring the symptom, your best hypothesis, the causes you ruled out, and the business or learning impact. That gives managers and peers what they need to act. You can cut mean time to resolution well below forty minutes with this loop. Next, we move into case studies and hands-on scenarios.learn.microsoft.com docs.cloud.google.com ppml.dev+22 min
13Case Studies and Hands-On ScenariosLet's move into case studies and hands-on scenarios. Case one, a churn model scores too good. The cause is a target leak: total charges and preprocessing applied before the split feed the label directly. Fix: split first, then engineer. Case two, a silent pipeline failure. An upstream field changed meaning, so you see single-feature drift but no parent drift. That pattern points to a pipeline bug, not a real population shift, so check the source before retraining. Case three, a handoff crisis. No data snapshots, no seeds, no locked environment, so retraining is impossible. Reproducibility is the fix, not more compute. Case four, training divergence from a gradient-clipping logic bug, with no exception and no metric anomaly. Here you need invariant checks, not just loss curves. In the lab, sixty to ninety minutes, you diagnose a broken notebook, fix the issues, and present an evidence-backed root cause. Now let's look at the prevention checklist and team practices.arxiv.org caterinaurban.github.io arxiv.org+22 min
14Prevention Checklist and Team PracticesLet's close the troubleshooting loop with prevention. Think of this as the practices that keep you out of the debugging cycle. The first gate is a pre-flight check before any model ships. Validate your data, confirm your train and test splits are correct, run leakage tests, verify reproducibility, and document a baseline you can compare against. This is the same logic as a model production checklist: versioned datasets, seeded training runs, and integration tests on the pipeline. Second, monitor with a baseline. Data quality checks, drift thresholds, pipeline service level agreements, and a named on-call owner. Without an owner, alerts go nowhere and silent degradation goes unnoticed. Third, document for audit. Model cards, datasheets, evaluation reports, deployment specs, and lifecycle plans. This protects you under the EU AI Act and speeds up onboarding and incident response. Fourth, assign one owner per concern: data quality, model performance, and production reliability. Clear ownership is the single strongest predictor of projects that reach production. Fifth, run a learning loop. Blameless postmortems, shared playbooks, and scheduled threshold reviews. Pick one model in production and apply this checklist this week. Next, the wrap-up and your next steps.onecdn.io thinklytics.com exa.ai+22 min
15Wrap-Up, Resources, and Your Next StepsLet's land this. Four habits carry the most weight: debug the data before the model, verify your splits, reproduce problems on a small sample, and instrument for silent failures. Those four cover most of what we walked through today. One more thing worth saying plainly: mindset beats tooling. More tools will not fix unclear metrics or an unowned pipeline. So pick one project this week. Run it through the pre-flight checklist, and log your top three gaps. That list is your roadmap. For managers, that log tells you where to invest. For learners, it tells you what to practice next. We will close with a short quiz, a reflection prompt, and role-specific follow-ups. Thank you for working through this with me. You already have the habits. Now use them.1 min

Take the deck with you

Download this course as a file — free, no sign-up needed.

PDF handoutEvery slide page, ready to print or share.16 pages · 3.8 MBDownload
Narrated PowerPointThe deck that presents itself — every slide carries the digital human's narration video.16 pages · 14.4 MBDownload
PowerPoint slidesThe full deck as a .pptx — open it in PowerPoint, Keynote, or Google Slides.16 pages · 3.7 MBDownload

Free to use in your own training — please keep the PersonWise credit page at the end.

Have your own deck? Turn it into a course

Sources consulted

Web sources consulted while building this course.