Chapter 3 — Data Validation, Cleaning, and Quality

Data quality is not a single binary property. It decomposes into five orthogonal dimensions, each with its own failure signature in ML:

• Completeness — fraction of expected values present. A null-rate spike in total_charges indicates upstream pipeline bugs (renamed column, dropped join key, partial backfill).
• Accuracy — whether values reflect ground truth. age = -7 and country = "XX" are obvious; wrong timezone or currency unit failures hide for months.
• Consistency — same fact represented identically across sources. Dangerous across the train/serve boundary because the model sees mutually incompatible encodings.
• Timeliness — lag between event and availability. Stale features produce a fraud model that under-reacts to regime change.
• Uniqueness — each entity appears exactly as often as expected. Duplicates inflate certain classes during training and bias loss estimates.

Quality Scoring

Mature platforms persist a per-dimension score and dashboard it next to model metrics. A typical scheme:

• Completeness: 1 − mean(null_rate) across critical features
• Accuracy: 1 − fraction failing range/domain checks
• Timeliness: 1 − min(1, lag / SLA_lag)
• Uniqueness: 1 − duplicate_rate

Cost of Poor Quality

The "1-10-100" rule scales precisely to ML: $1 to prevent at source, $10 to remediate downstream, $100 once it hits a customer. A bad row costs a millisecond at ingestion, hours at training debugging, weeks of business impact when the model misclassifies in production.

Two complementary categories of tools dominate: schema-based validators (enforce types, presence, domains) and statistical anomaly detectors (compare distributions to a baseline). Modern frameworks blend both.

TFDV (TensorFlow Data Validation)

The TFX data-quality component, built on Apache Beam — scales to terabytes on Dataflow, Spark, or Flink. Three-step workflow: generate stats, infer a schema, validate every subsequent batch.

import tensorflow_data_validation as tfdv

stats = tfdv.generate_statistics_from_csv('train.csv')
schema = tfdv.infer_schema(stats)
eval_stats = tfdv.generate_statistics_from_csv('batch.csv')
anomalies = tfdv.validate_statistics(eval_stats, schema)
tfdv.display_anomalies(anomalies)

Schemas are protobuf describing types, presence, domains, and nested structure. TFDV's killer feature is built-in training-serving skew and drift detection: hand it two stats artifacts and it emits per-feature comparisons.

Figure 3.1 — Validation pipeline

Great Expectations

GE inverts the philosophy: humans write Expectation Suites — assertions about the data — and every failed expectation is an anomaly.

import great_expectations as ge
import pandas as pd

df = pd.read_csv("train.csv")
ge_df = ge.from_pandas(df)
ge_df.expect_column_values_to_not_be_null("user_id")
ge_df.expect_column_values_to_be_between("age", min_value=0, max_value=120)
ge_df.expect_column_mean_to_be_between("click_count", min_value=0, max_value=10)
result = ge_df.validate()

Suites can be auto-profiled then human-refined — like generating tests via coverage then editing by hand. Data Docs render as browsable HTML reports, doubling as living documentation. First-class Airflow integration via GreatExpectationsOperator.

Pandera and pydantic

Pandera validates Pandas/Polars DataFrames inline; pydantic validates request payloads at FastAPI service boundaries. Both are the last mile.

import pandera as pa
from pandera.typing import Series

class TransactionSchema(pa.DataFrameModel):
    user_id: Series[int] = pa.Field(ge=1)
    amount: Series[float] = pa.Field(ge=0, le=1_000_000)
    country: Series[str] = pa.Field(isin=["US", "CA", "UK", "DE"])

TransactionSchema.validate(df)

Expectations as Code

The unifying idea: version-controlled, PR-reviewed, CI-executed data contracts — the data-quality analog of infrastructure-as-code.

The mature hybrid: GE upstream in the warehouse; TFDV downstream for ML-specific skew; pandera/pydantic at service boundaries.

Validation finds problems; cleaning resolves them. The three perennial issues: missing values, outliers, and label noise.

Missing Values — Mechanism Matters

• MCAR — missingness unrelated to data. Dropping rows safe but wasteful.
• MAR — depends on other observed variables. Conditional imputation works.
• MNAR — depends on the unobserved value itself. Missingness is informative; add a missingness indicator feature.

Figure 3.2 — Imputation decision tree

Imputation strategies in increasing complexity: drop → mean/median/mode → constant+indicator → KNN → MICE → model-based. Non-negotiable rule: fit imputers only on training data, then apply the fitted artifact to validation, test, and production. Fitting on the full dataset leaks information.

numeric_transformer = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="median", add_indicator=True)),
    ("scaler", StandardScaler())
])
preprocess = ColumnTransformer([
    ("num", numeric_transformer, numeric_features),
    ("cat", categorical_transformer, categorical_features),
])
clf = Pipeline([("preprocess", preprocess), ("model", RandomForestClassifier(...))])
clf.fit(X_train, y_train)

The serialized pipeline applies identical imputation to both train and inference — eliminating an entire class of training-serving skew.

Outliers

Three flavors: data errors, rare-but-valid cases, distribution shifts. They require different responses.

• IQR rule — outside [Q1 − 1.5·IQR, Q3 + 1.5·IQR]. Univariate, robust.
• Z-score — |z| > 3. Affected by the outliers it measures; prefer median + MAD.
• Isolation Forest — randomly partitions; scales well; key knob is contamination.
• DBSCAN — density-based; noise points are unassigned. Sensitive to eps.
• LOF — local density vs. neighbors. Good for heterogeneous data with local anomalies.

Treatment: fix, winsorize (cap at 1st/99th percentile), remove (only when confident), down-weight, or use robust models (trees, Huber loss). Critical: in fraud or rare-disease prediction, the "outliers" are the positive class — never auto-delete.

Deduplication and Normalization

Exact dedup is trivial; near-dup (Jaro-Winkler, MinHash, embedding similarity) is harder. Duplicates inflate training-set size, bias the model toward the duplicated examples, and produce optimistic CV scores. Normalization (lowercase, ISO 8601 dates, canonical country codes) is one of the most common sources of train-serve skew.

Label Noise

The most damaging form of corruption — corrupts the learning signal directly. The dominant modern technique is confident learning (Cleanlab):

1. Train a baseline model and obtain out-of-sample predicted probabilities via 5-fold CV.
2. Pass labels and probabilities to find_label_issues; it estimates the joint distribution of noisy vs. true labels and flags confident disagreements.
3. Route flagged examples for review, drop them, or use CleanLearning for noise-aware reweighting.

from cleanlab.filter import find_label_issues
probs = cross_val_predict(base_clf, X, y, cv=5, method="predict_proba")
label_issues = find_label_issues(labels=y, probs=probs)
X_clean, y_clean = X[~label_issues], y[~label_issues]

Complementary: label smoothing, co-teaching, per-example loss tracking. In production, the highest-leverage action is a human-in-the-loop review queue for high-confidence model-vs-label disagreements.

A clean snapshot still degrades in production because the world moves. Drift detection compares live distributions against a training reference and alarms when divergence matters.

Training-Serving Skew

A systematic difference between training and serving inputs caused by an engineering issue, not natural shift. Examples: SQL vs. Python feature implementations, a categorical encoder fit on training silently mapping unseen serving categories to 0, normalization re-fit per batch. Fixable by engineering: share a single serialized preprocessing pipeline, validate serving against the training schema, continuously compare distributions. Feature stores exist largely to make this easy.

Concept Drift vs. Data Drift

• Data drift (covariate): P(X) changes — new geography, new demographic. Detectable from inputs alone.
• Concept drift: P(Y | X) changes — spammers adopt new tactics, the relationship moves. Needs labels (often delayed) or proxies: prediction-distribution shift, performance estimation, confidence shifts.

KS, PSI, JS, and Other Divergence Measures

PSI = Σ (qᵢ − pᵢ) · ln(qᵢ / pᵢ) — the de-facto credit-risk standard. JS = ½KL(P||M) + ½KL(Q||M), bounded and finite, safer than raw KL for dashboards. MMD is kernel-based in an RKHS — the heavy artillery for multivariate / embedding drift, as in Alibi Detect's MMDDrift.

Two practical heuristics:

1. Statistical vs. practical significance: p-values answer "real?", effect sizes (PSI, Wasserstein) answer "big enough?" — use both.
2. Calibrate thresholds on healthy data: compute each metric over historical no-drift windows; alert at the 95th or 99th percentile.

Alerting and SLAs

Drift detection without alerting is theater. A production monitor should:

1. Run continuously, latest window vs. fixed training reference.
2. Compute multiple metrics per feature (at least PSI + one significance test).
3. Aggregate into a drift share across features.
4. Route by severity: warn on moderate, page on significant.
5. Tie alerts to operational playbooks: retrain, fallback, kill-switch.

Figure 3.4 — Drift detection and SLA monitoring loop

A data-quality SLA template:

The point is that the response is automatic. Humans should be informed, not interrupted, when the system behaves as designed.