Assessing Data Science Quality

This training teaches data scientists and analysts how to evaluate data science quality, covering key methods and metrics to assess models, data, and results.

A digital instructor presents all 14 pages. Hold “Ask” at any point and ask out loud — the answer comes from this course. No sign-up needed.

28 minFree to watchDownloads

What you’ll learn

01Evaluating the Quality of Data Science: An OverviewWelcome. Over the next several slides, we'll treat quality evaluation as a first-class discipline in data science, not a checklist you run the night before release. Think of quality as a layered stack: data, method, model, code, impact, and governance. Each layer carries its own evidence, and each can fail on its own. A single upstream schema change, with no error message, can silently degrade a model weeks later. The cost is real. Poor data quality drives rework, erodes trust, and creates regulatory exposure, roughly twelve point nine million dollars per year across organizations. That figure comes from Gartner. Every role here owns a distinct layer of evidence. Data scientists own method and model, analysts and reviewers own validation, managers and instructors own the decisions that follow. By the end, you should be able to judge quality with concrete criteria, not instinct. Let's build a shared vocabulary for quality and failure archetypes.aicodeinvest.com qaskills.sh datus.ai+21 min
02A Shared Vocabulary for Quality and Failure ArchetypesLet's build a shared vocabulary for quality and failure. Validation checks that requirements are met. Verification checks correctness against a trusted record. Reproducibility means the same code and the same data produce the same result. Replicability means a new dataset confirms the finding. Robustness means different methods reach the same conclusion. Above all, fitness for purpose: the method must answer the actual decision question, not a convenient proxy. Now, beware leakage. It dominates real failures. One survey found three hundred twenty-nine papers across seventeen fields whose results collapsed after correction. Leakage has three categories. Level one is a broken train and test boundary. Level two is illegitimate features that would not exist at prediction time. Level three is an unrepresentative test set. Here is the hard part: checklists and model cards routinely miss leakage. What catches it is provenance records that track where each sample and feature came from, plus adversarial review that actively tries to break the evaluation. Next, we turn to Data Quality Dimensions and Profiling.osti.gov cell.com arxiv.org+21 min
03Data Quality Dimensions and ProfilingLet's move on to data quality dimensions and profiling. Quality here decomposes into six measurable dimensions: completeness, uniqueness, validity, accuracy, consistency, and timeliness. Referential integrity is often added as a seventh. Two of these resist mechanical checks. Accuracy and consistency require a trusted system of record and cross-system reconciliation. A record can be complete, unique, and valid, and still be wrong. So profiling matters. Use column statistics, distribution checks, cross-field constraints, and anomaly detection. When you find a defect, separate source-system errors from pipeline-introduced errors, and account for sampling and measurement error before declaring it real. Next, we look at data contracts and expectation testing.aicodeinvest.com qaskills.sh datus.ai+21 min
04Data Contracts and Expectation TestingLet's move on to a practical mechanism for enforcing quality at the source: data contracts and expectation testing. A data contract is a versioned, machine-readable agreement between the team that produces a dataset and the teams that consume it. It goes beyond schema. It also covers semantics, quality rules, service-level agreements, ownership, and a change policy. The guiding principle is prevention at the boundary, with observability catching whatever slips through. In practice, you enforce contracts with tools like Pydantic, Pandera, dbt, Great Expectations, or a schema registry. And for a vendor-neutral format, standardize on the Open Data Contract Standard, version three point one point zero. Here is the operational test. When a violation occurs, does something happen automatically? If not, it is documentation, not a contract. Start small. Pick the three to five datasets that cause the most incidents. Next, we will examine methodological soundness and statistical validity.aicodeinvest.com qaskills.sh datus.ai+21 min
05Methodological Soundness and Statistical ValidityLet's turn now to methodological soundness and statistical validity. The first question is fitness for purpose. A sound method answers the real decision question, not a convenient proxy that happens to be easy to measure. Next, trace where bias enters. Common sources include sampling, survivorship, confounding, feature selection, and target construction. Each one can quietly reshape your results. Before trusting any result, check the assumptions behind the method, and run a sensitivity analysis to see how much conclusions move when those assumptions shift. Also guard against multiple comparisons, p-hacking, and overfitting to the analysis set. These inflate apparent effects and rarely survive validation on new data. Be precise about causality. Claim it only when the design supports it. Otherwise, stay correlational and say so. And here is a practical reviewer heuristic. Take every headline claim, and trace it back to its evidence. If that path is unclear, the claim is not yet ready. That discipline is what separates results you can defend from results that merely look good. Next, we look at leakage, the silent killer.link.springer.com intechopen.com aclanthology.org+22 min
06Leakage Deep Dive: The Silent KillerLet's take a deep dive into data leakage, the silent killer of model quality. Research across 129 temporal datasets and thousands of tabular ones identifies four classes of leakage. Class one, estimation leakage, where you fit a scaler on the full dataset, is far less dangerous than many people assume, with changes in area under the curve below zero point zero zero five. Class two, selection leakage, is the biggest threat at realistic dataset sizes, with peeking at the test set adding about zero point zero four zero A U C and cherry-picking random seeds adding about zero point zero four five. Class three, memorization, grows with model capacity, from a standardized effect of zero point three seven for naive Bayes up to one point one one for a decision tree at ten percent duplication. Class four, boundary leakage, hides under random cross-validation, averaging about zero point zero two three A U C on genuinely temporal data. The numbers get worse when you look at intervals. Naive z-based confidence intervals cover only about fifty-five percent at a nominal ninety-five percent, while conservative-z reaches roughly eighty-seven percent. The guards are structured: split before you transform, use grouped and temporal splits, audit feature importance, and keep one untouched holdout. Next, we look at model evaluation metrics, validation, and robustness.osti.gov cell.com arxiv.org+22 min
07Model Evaluation: Metrics, Validation, and RobustnessLet's talk about how we evaluate models, because the choice of evaluation design often matters more than the choice of algorithm. Start with metrics. Pick them from the business objective and the cost of each error type, not from convenience or habit. A false negative in cancer screening costs far more than a false positive, so accuracy alone hides what you actually care about. Next, treat your splits as structural decisions. Use temporal splits when you expect drift over time, and grouped splits when the same patient, customer, or device appears repeatedly. Otherwise leakage inflates results. Also remember that discrimination is not reliability. A high area under the curve tells you ranking is good, but it does not mean the probabilities are trustworthy. Calibrate your scores, and justify thresholds against real decision costs. Then stress-test. Check behavior under distribution shift, selective labels where outcomes are only observed for past decisions, adversarial inputs, and edge cases. Stress results reveal fragility that average performance conceals. Finally, always compare against baselines and run ablations. This proves your model adds value beyond simple rules or the previous model. And when you disaggregate results across subgroups, state the causal assumptions behind your interpretation, because unequal data collection can make performance gaps misleading. In short, good evaluation is a design problem, not just a metric. Next, we move to code, pipelines, and reproducibility.huggingface.co arxiv.org proceedings.neurips.cc+22 min
08Code, Pipelines, and ReproducibilityLet's turn to the code and pipeline side of reproducibility. A reproducible run rests on three ingredients: the data, the code with its dependencies, and the runtime environment. None of these is sufficient alone. Version code, data, model, and config together, and tag datasets by content hash so you can prove which bytes were used. Pin container images by digest, never by a floating latest tag, and pin seeds in Python, NumPy, and your framework. Here is the practical test. If two runs on identical data disagree, your evaluation gate is measuring noise, not model quality. So wire your CI gates in order: data validation before training, model evaluation before packaging. Start with those two controls; they prevent most incidents. Next, we will look at the review process and technical peer critique.link.springer.com intechopen.com aclanthology.org+22 min
09Review Process and Technical Peer CritiqueNow let's walk through the review process and technical peer critique. A good review checklist spans six areas: data, method, model, code, claims, and governance. Then classify severity, so blocking issues stay separate from suggestions. And ask clarifying questions before you pass judgment. Structure the conversation instead of trading opinions. For high-stakes work, red-team it, for example feature blackout tests and prompt injection tests. Here is a practical standard: the analyst-inspector test. Can an independent reviewer regenerate the numbers from the workflow alone? If not, the documentation is incomplete. When something fails, the message should name the broken field, the owner, and the approval path. Finally, keep review records, because they support audit and let future teams reuse what you learned. Next, we turn to business impact, risk tiering, and governance.link.springer.com intechopen.com aclanthology.org+21 min
10Business Impact, Risk Tiering, and GovernanceLet's talk about how validation connects to business impact, risk tiering, and governance. A validation is not a permanent property of a model. It is a bounded permission: a stated use, for a stated period, with a named approver. So approval memos need an expiry date, compensating controls, and a rollback plan. Otherwise, a release decision quietly becomes permanent permission. Now, regulation raises the bar. EU AI Act Article 72 has applied since the second of August, 2026. In practice, post-market monitoring becomes a rolling conformity audit. You continuously monitor Articles nine through fifteen: risk management, data governance, documentation, logging, transparency, human oversight, and robustness. Intensity should track risk tier. A startup may run a ten to fifteen item checklist, while a regulated enterprise review can exceed fifty items. State residual risk honestly. Non-compliance fines can reach thirty-five million euros, or seven percent of global turnover. The takeaway is simple: treat approval as scoped, time-bound, and reviewable. Next, we move to monitoring, drift detection, and retraining triggers.2 min
11Monitoring, Drift Detection, and Retraining TriggersLet's turn to monitoring, drift detection, and retraining triggers. The defining risk here is that models degrade silently. Latency looks normal, error rates sit at zero, and the predictions are still wrong. So monitor four layers together: input quality, prediction distribution, live performance, and operational metrics. Drift triggers are leading indicators. Performance triggers lag, because labels arrive late. Time-based retraining is a failsafe, not a strategy. For Population Stability Index, below zero point one is stable, zero point one to zero point two five is watch, and at or above zero point two five, alert. But a drift alert starts an investigation, not an automatic retrain. First check pipelines, schemas, and labels. Every promotion also needs champion-challenger gating, canary rollout, and a tested rollback. Next, teaching and assessing quality evaluation skills.1 min
12Teaching and Assessing Quality Evaluation SkillsNext, let us look at how we teach and assess quality evaluation itself. The starting point is alignment: each learning objective should map to the task that measures it. If you want reviewers to judge interpretation, do not assess them only on code. A useful rubric keeps three things separate: implementation, solution correctness, and interpretation. That separation is what lets you give partial credit, and it is also how you diagnose where a student or reviewer is actually struggling. Then calibrate. Have reviewers score low, medium, and high samples, compare their scores to reference scores, and discuss the gaps openly. Over time, that practice builds shared standards. For practice, use case critiques, seeded defects, code review, and the I like, I wish, What if format. And when you give feedback, separate technical defects from reasoning gaps. Be specific, be meta where it matters, and stay respectful. Finally, assess evaluators, not just artifacts, using inter-rater agreement and reviewer calibration over time. That turns review quality into a measurable skill. That leads us into the next topic: Tiered Evaluation Workflow: Lightweight Review to Full Audit.2 min
13Tiered Evaluation Workflow: Lightweight Review to Full AuditLet's turn evaluation into a workflow. One integrated checklist covers seven areas: data, method, model, code, impact, monitoring, and governance. Then you match intensity to stakes. A pull request review is light. A release review is deeper. An independent structural audit is the heaviest option, and you reserve it for high-consequence systems. Next, define roles explicitly. Who decides, who produces evidence, who can block. The MESA model risk framework makes this concrete. A validation is a bounded permission, over a stated use, for a stated period. One accountable approving executive, separate from the head of model risk. The framework forbids the same person holding both. Artifacts make that permission auditable: model cards, data documentation, control-to-evidence matrices, and approval records with expiration dates. Finally, a maturity ladder. Manual checks first, then pipeline automation, then full continuous integration and delivery with continuous training. Move up only when the lower level is reliable. Next, let's make this personal in the action plan: what you change in your own workflow.link.springer.com intechopen.com aclanthology.org+22 min
14Action Plan: What You Change in Your Own WorkflowAs we close, let's turn evaluation into five concrete changes you can make in your own workflow. Data first. Put a first contract on the one dataset whose changes burned you most. Just a schema and one or two quality assertions, enforced automatically. A contract nobody validates is documentation, not a gate. Method. Audit your validation split for temporal, grouped, or duplicate leakage. If rows share a patient, a user, or a time window, random splitting can inflate your scores, and the inflation only disappears in production. Process. Add an evaluation gate with two checks: an absolute floor no model may ship below, and a regression guard against the current champion. This stops a harmless-looking refresh from quietly degrading production. Review. Classify finding severity. Blockers and suggestions are treated differently, so teams stop removing gates because every warning blocked a pull request. And people. Set a review cadence, name an owner for each contract, and schedule reviewer calibration so severity judgments stay consistent. Start with one of these this week, not all five. Thank you for your attention, and good luck applying this in your own work.aicodeinvest.com qaskills.sh datus.ai+22 min

Take the deck with you

Download this course as a file — free, no sign-up needed.

PDF handoutEvery slide page, ready to print or share.15 pages · 3.4 MBDownload
Narrated PowerPointThe deck that presents itself — every slide carries the digital human's narration video.15 pages · 14.7 MBDownload
PowerPoint slidesThe full deck as a .pptx — open it in PowerPoint, Keynote, or Google Slides.15 pages · 3.3 MBDownload

Free to use in your own training — please keep the PersonWise credit page at the end.

Have your own deck? Turn it into a course

Sources consulted

Web sources consulted while building this course.