KubeCon EU 2026

The Sanity Check

Observability for AI Data Pipelines

David Aronchick Expanso Monday March 23 | 10:40 CET

Opening

Your AI just gave a completely wrong answer.

The model looks fine. Confidence is high. Latency is normal. But the answer is dangerously wrong — and nobody in the stack flagged it.

Act I

The Data Problem

Scale

Data is exploding. Most of it goes unused.

Traditional centralized approaches were already struggling. AI pipelines make it worse — they need more data, from more sources, refreshed more often.

175 ZB global data this year IDC
60%+ generated outside data centers IDC
73% can’t keep up with processing demand Gartner
85% of AI projects never reach production Gartner

Why AI Is Different

AI pipelines multiply the data path problem.

  • RAG pulls from CRM, docs, tickets, policies — simultaneously.
  • Each source has its own freshness, schema, and authority model.
  • Embeddings silently compress meaning — drift is invisible.
  • Retrieval ranks evidence with no visibility into why.
  • Every link in the chain is a place for silent corruption.

Thesis

The model is not the mystery. The data path is the mystery.

When an AI system gives a wrong answer, the investigation should start with the data — not the weights.

Act II

Where We Look vs. Where Things Break

The Gap

Teams monitor the model. The pipeline is unobserved.

What teams watch today

  • Model latency and throughput
  • Token counts and cost
  • Accuracy on evaluation benchmarks
  • Error rates at the API boundary

What actually causes incidents

  • Stale or drifted source data
  • Schema changes in upstream connectors
  • Embedding clusters that shifted silently
  • Retrieval ranking the wrong evidence
  • Authority conflicts between sources

Architecture

The full data path from source to answer.

Sources

CRM, docs, tickets, billing

Connectors

Ingest & schema mapping

Normalize

Authority & validation

Embed

Chunk & vectorize

Retrieve

Rank & select evidence

Answer

Generate response

Observability layer — freshness • contracts • drift • lineage

Failure Modes

Four ways AI data pipelines break silently.

Freshness lag

A canonical source falls behind its freshness SLO. Stale data loses ranking to newer, less trustworthy records.

Schema drift

A connector emits a new field path or drops an existing one. Downstream transforms map garbage silently.

Semantic drift

Embedding clusters shift after an upstream data change. The meaning of “refund policy” quietly changes.

Authority conflict

A non-canonical source overrides policy data. Retrieval ranks it above the approved source of truth.

Cost of Blindness

Without pipeline observability, you cannot debug AI.

  • Wrong answers ship with high confidence.
  • Incident responders blame the model because the data path is invisible.
  • Remediation is guesswork — retrain, re-prompt, hope.
  • The same class of failure happens again next quarter.

Act III

The Sanity Check

Framework

Four observability pillars for AI data pipelines.

Freshness

19h

Is the data current? Set SLOs for canonical sources. Alert when freshness budgets are exceeded.

Contract Health

3

Did the shape of data change? Track schema mutations, new field paths, and connector changes.

Semantic Drift

+42%

Did the meaning change? Monitor embedding cluster centroids against historical baselines.

Lineage

→

Where did this answer come from? Trace every answer back to the source record and pipeline run.

Platform

Expanso is the edge pipeline foundation.

Expanso provides the distributed runtime — nodes, jobs, executions, selectors, and a unified control plane. The observability layer is what you build on top to make AI pipelines operable.

4 edge jobs
4 nodes
1 poisoned source
1 replay path

Failure Story

A rogue Salesforce update changes one policy from 30 to 365.

  • The EU sales edge job ingests it.
  • Normalization misses the authority violation.
  • Embeddings shift.
  • Retrieval ranks the poisoned chunk first.
  • The copilot answers confidently from bad context.
edge-eu-sales
edge-us-support
edge-finance-canonical
edge-docs-sync

Live Demo

Investigation flow

  1. Show the wrong answer.
  2. Compare retrieved evidence.
  3. Isolate the bad job in the fleet.
  4. Trace answer to source record.
  5. Replay with a guardrail.
  6. Show the corrected answer.

Demo Commands

Operator flow during the incident

python cli/democtl.py fleet status
python cli/democtl.py jobs inspect edge-eu-sales
python cli/democtl.py trace answer ans_9f2c
python cli/democtl.py alerts list --job edge-eu-sales
python cli/democtl.py replay job edge-eu-sales \
  --guardrail policy_authority_guard

Act IV

Live Demo

Act V

Operating Model

Practice

Treat AI data systems like production infrastructure.

  • Set freshness objectives for every canonical source.
  • Track contract changes in every connector.
  • Alert on semantic drift before it reaches users.
  • Make answer-to-source tracing routine, not heroic.
freshness SLO <3h
drift threshold <15%
contract violations 0

Takeaways

Three things to remember.

1

Treat AI pipelines as production distributed systems. The same SRE disciplines that fixed microservices will fix AI reliability.

2

If you cannot trace an answer to a source record, you cannot trust it. Lineage is not optional.

3

Observability is the prerequisite for reliable AI. Without it, every incident is a mystery and every fix is a guess.

Close

The model is not the mystery. The data path is the mystery.

Expanso gives us the edge pipeline and control-plane foundation. The rest is making that system observable enough to operate with confidence.

David Aronchick aronchick@expanso.io expanso.io
All talks