Pipeline Monitoring Is Not One Thing
Ask a data engineering team whether their pipelines are monitored and most will say yes. Ask what that means in practice and the answers start to diverge: some point to dashboards that are rarely checked, some point to an alert that fires when a job fails, and some point to a person whose job it is to look at Synapse Studio every morning before the daily standup.
None of these are wrong but none are the whole answer either. Pipeline monitoring isn't binary, it's a maturity ladder, and most teams are standing on a lower rung than they think.
The Four Rungs
We're going to start with the basics and then discuss the increasingly levels of maturity.
-
Ad hoc. Someone opens the portal, checks the run history, and moves on. This works until it doesn't. It depends entirely on a person remembering to look, knowing where to look, and having time to do it that day. There is no record that the check even happened.
-
Reactive. An alert fires when a pipeline fails. Instead of relying on a human to notice the issues, you have an automated alert that tells you about failures. But it doesn't tell you anything about degraded performance, pipelines runtimes, or the fact that nobody looked at yesterday's results either.
-
Systematic. Monitoring runs on a schedule regardless of whether anything went wrong. A report gets generated and delivered whether the news is good or bad. This is the point where monitoring goes from a task that someone does to a fully automated process.
-
Governed. The systematic reports become part of a compliance or service level agreement (SLA) framework. There is a persistent, queryable record of pipeline health over time, tied to the business' standards, and someone could reconstruct exactly what was known and when if they ever needed to.
Most teams live somewhere between ad hoc and reactive. Very few make it to systematic, and fewer still to governed. This isn't because the later rungs are technically difficult, but because data teams often have poor self-awareness about which rung they actually need to be on.
Three Interpretations of "Monitoring"
Part of the confusion is that "monitoring" is doing the work of three separate concepts, and most tooling only solves one of them.
Observability is whether you can see what happened, think logs, metrics, and traces. Azure Monitor and Application Insights are built for exactly this. Most teams that invest in monitoring stop here, because it feels like the job is done. You can query anything you want, if you know to ask.
Alerting is the layer that tells you when something's wrong, so you don't have to go looking yourself. This is not a replacement for observability but instead builds on top of it. Alerting has a massive blind spot though: between alerts, nobody's actually watching anything.
Audit and evidence is whether you can prove, after the fact, that monitoring was actually happening. This is the layer that few teams actually implement, and it's the one that matters most in regulated environments. A dashboard that existed is not evidence. A log retention window that expired six months before the audit is not evidence. What's needed is a persistent, timestamped record that survives independently of whether anyone was watching the dashboard that day.
Teams that feel like their monitoring is "good enough" have usually solved observability, sometimes solved alerting, but almost never solved the third one, because it doesn't show up as a gap until someone external asks for it.
The Fragmentation Problem
One of the main reasons that the audit layer gets skipped is because Azure's native tooling covers the other two rather well, just spread across several places. Synapse Studio shows you run status. Azure Monitor handles alert rules. Application Insights and Log Analytics hold the structured logs. Each one does its job.
None of them, though, hand you a single, consolidated, human-readable answer to "what happened last night", without you having to go and ask three separate tools. That's the gap where monitoring quietly reverts to a manual process, even for teams with decent tooling in place. Checking three systems every morning is exactly the kind of thing that gets skipped on a busy day, and a skipped day never shows up anywhere.
What "Good" Actually Looks Like
Stripped of any particular vendor or platform, a monitoring setup that closes the gap has a few consistent characteristics:
- A single consolidated summary. The report itself should answer "what happened last night", rather than leaving you to check run status, alerts, and logs separately and stitch the picture together yourself.
- Actionable failure detail in the first place you look. Which pipeline, which trigger, how long it ran before it failed. If you need to open a second tool to find the run ID, the first tool hasn't done its job.
- A persistent, queryable history, not a 30 or 90-day retention window that quietly erases the evidence before you actually need it.
- Least-privilege access for whatever is doing the checking. Monitoring tools that read production pipeline metadata do not need write access, and scoping them tightly means a leaked credential can't do any real damage.
- Cost proportionate to the signal. A lot of monitoring spend goes toward platforms built for much larger, more varied environments than a single Synapse workspace actually needs.
None of these require a particular product. They're a checklist for evaluating whatever you already have, or whatever you're about to build.
Build, Buy, or Borrow
Once a team recognises the gap, the usual question is what to do about it, and there are really three paths.
Native tooling alone is the right call for smaller teams that don't have a compliance framework to satisfy. Azure Monitor plus a well-set alert rule covers the job when nobody's asking for evidence, just results.
Commercial observability platforms solve the problem thoroughly, and they're usually the right call for organizations running many different systems and platforms at once. For a team running pipelines on a single Synapse workspace, they're often the wrong tool for the job.
Lightweight custom tooling sits in between: a scheduled function that fetches run history, summarises it, emails it, and archives a snapshot somewhere durable. This is a deliberately small piece of infrastructure, built to answer one question well rather than cover every observability question a larger platform is designed for.
None of the three options is universally right. The mistake is picking one without first working out which rung of the maturity ladder you're actually trying to reach.
Compliance Is It's Own Problem
Most writing on pipeline monitoring is aimed at SaaS companies optimising for uptime and customer experience. That's a legitimate problem, but it's a different problem to the one facing teams in defence, aviation, healthcare, or financial services, where the question isn't just "did it work" but "can you show us it was being watched."
That distinction changes what "solved" means. A team that can answer failures quickly but can't produce a dated record of pipeline health for the last six months has solved the engineering problem and left the compliance problem untouched. In regulated environments, the second problem is often the one that actually gets asked about, and it's the one native tooling and commercial platforms alike tend to treat as an afterthought rather than a first-class requirement.
Where This Leaves Most Teams
The most pressing question to ask isn't "what platform should we use"; it's where you currently sit on the maturity ladder, and where you actually need to get to. Most teams have decent observability already. Fewer have alerting that covers what actually matters and even fewer have anything that would hold up to a rigorous auditor.
Closing that gap doesn't usually mean removing what's already in place. It just means adding one more layer, deliberately scoped to produce a persistent, systematic record, that sits on top of the tooling you already have.
If you're working through where your own pipeline monitoring sits on that ladder, or what a governed layer would look like for your environment, I'm always happy to talk it through.