Alerting on Third-Party API Degradation in Production

Third-party APIs power most of what your production stack does, and most teams have no real plan for what happens when one of them quietly breaks. Postman's 2024 State of the API Report puts the average application at 26 to 50 dependent APIs, and over 80% of digital transactions now run through some kind of API call. That means external API health basically is user-experience health, whether your dashboards admit it or not. When an internal service fails, you SSH in, read logs, and roll back a bad deploy. When a third-party service fails, you watch a graph move and wait for someone else to fix it. That asymmetry changes which signals you can collect, how you set thresholds, and what resolving an alert even means when you don't control the system.

Three failure modes are worth understanding because they look nothing like internal failures. Silent degradation shows up as slower responses in one region with zero error codes and no status page update. Correctness failures are sneakier: the API returns a crisp HTTP 200 while the payload is empty, partial, or structurally wrong, a pattern that commonly precedes hard 500s in payment API degradations. And schema changes, authentication rotations, and tightened rate limits arrive without warning because the vendor's changelog is not your operational concern until it causes an incident. A 2025 EMA report found 68% of enterprises lack full API visibility, and 40% only catch issues after customers already felt them. For third-party dependencies, that visibility gap is structural. You cannot see inside the box, so everything you build has to come from what you can measure at the edge of the connection.

Signals you can actually observe externally

Start with latency, and don't stop at the average. Averages obscure problems by smoothing variance; P95 and P99 spikes appear well before error rates climb, which makes them your actual early warning system. Treat P50 as your baseline, P95 as an early indicator, and P99 as a signal that something structural has broken. Alert on P95 and P99 independently, because an alert that only fires on averages fires late or not at all.

Break the response down by phase: DNS resolution, TCP handshake, TLS negotiation, time-to-first-byte, and total transfer. Each phase has a different owner and a different fix. A two-second response that is entirely delayed in TLS negotiation is a different problem than one delayed in server processing, and combining them produces a signal that points nowhere useful.

Error rate matters, but only if you split it by endpoint and by error class. A 429 rate-limit response is not the same as a 500, and it often appears before a full outage, so it warrants its own alert logic rather than being buried in a generic error count. Mixing authentication errors with server errors with client errors produces a number that carries no actionable information.

The most commonly skipped signal is payload correctness. Run content-match assertions or JSONPath checks on every critical monitor: $.status == "active", $.data.length > 0, whatever your contract actually requires. A 200 that does not return what it is supposed to return is a failure, and your alerting needs to treat it as one.

Location matters more than most teams assume. An API can be healthy in one region and timing out in another, and a single monitoring point will report everything as normal the entire time. Postman's 2024 data shows 67% of organizations serve traffic from three or more regions, so your monitoring footprint should match that. BGP route leaks and regional network failures are real operational risks, and single-origin monitoring cannot detect them. For calibration: simple reads should fall under 100ms, complex queries under 300ms, and anything involving chained external calls under a second. Google's research puts 200ms as the threshold where users begin noticing lag in interactive applications.

Why static thresholds fail for third-party APIs

Static thresholds fail in exactly two directions: too noisy on things that don't matter, too quiet on things that do. Teams can accumulate over 2,000 alerts a week with only about 3% requiring real action, which trains engineers to ignore the alert queue entirely. Most teams find that 60 to 80% of their alerts are noise, and for third-party monitors that ratio tends to be worse because external services carry their own traffic-driven variance that has nothing to do with your application.

A fixed 500ms threshold fires on every legitimate traffic spike, every CDN hiccup, every maintenance window the vendor did not announce. None of it is actionable, so engineers learn to ignore the alert. Meanwhile the real risk arrives gradually: P99 creeping from 400ms to 490ms over six hours never crosses your threshold, so nothing fires while your error budget burns down.

Check intervals are a threshold in disguise, and this one is consistently overlooked. A five-minute check interval means a five-minute blind spot. For a payment API, that is five minutes of silently failing charges before anyone knows to investigate. For anything revenue-critical, 30 to 60 second intervals are appropriate. The interval is the floor on how quickly you can detect a problem, regardless of how sophisticated your alerting logic is above it. IT outages cost roughly $5,600 a minute on average, and 41% of firms report costs that can exceed $1 million an hour, so detection delays have direct financial consequences.

SLO burn-rate alerting for third-party APIs

SLO-based alerting reframes the core question. Instead of asking whether a number crossed a line, you ask whether you are spending your error budget faster than you can afford to. That question maps directly to what your users experience.

Set the SLO in concrete terms: "99.9% of requests to the payment API return a valid response in under 500ms, measured on a rolling 30-day window." The remaining 0.1% is your error budget, the amount of degradation you can absorb before customers are meaningfully affected.

The Google SRE Workbook's multi-burn-rate framework gives you three speeds to watch. Fast burn, 2% of budget consumed in an hour, is a P0: page on-call within five minutes because something is actively failing. Medium burn, 5% of budget consumed in six hours, is a P1: notify the team and expect a response within 30 minutes because this is real degradation even if it has not reached a critical threshold yet. Slow burn, 10% of budget consumed across three days, is a low-urgency signal: address it without escalating and begin a vendor conversation during normal business hours. Burn rates catch both sudden failures and gradual degradation within the same framework, which static thresholds cannot do simultaneously.

Keep third-party SLOs separate from your internal ones. A vendor outage that breaches your SLO is still a breach on paper, but the response looks nothing like a code patch. Build a dedicated error budget per critical vendor and track it independently, so your SLO reporting accurately distinguishes vendor-caused breaches from engineering-caused ones. Circuit breakers support this separation: they prevent a vendor's degradation from consuming your internal error budget and create a clear boundary between what your team owns and what the vendor owns.

Synthetic monitoring that catches degradation early

Synthetic monitoring is valuable here because it sits outside your infrastructure entirely and mimics what a real user's request experiences, independent of whether your own servers are performing well.

A synthetic monitor worth trusting closely resembles your real traffic. Send the actual calls your application sends: real authentication headers, realistic payloads, and the response shape you actually expect. Validate that payload on every run, because a 200 with an empty data array is a business logic failure regardless of what the status line says. Capture timing at the phase level so when something slows you can identify which layer is responsible. Probe from every region your users are in, because an API degraded in one region is still a real incident for users there even if monitoring in another region shows no problems.

Interval sizing should be an explicit decision. Payment, authentication, and anything on the revenue path should run every 30 to 60 seconds. Lower-stakes integrations like enrichment or analytics feeds can run less frequently, but that should be a deliberate choice based on acceptable detection time, not whatever the platform defaults to.

Suppress noise before it obscures root causes. When a third-party monitor fires, every downstream dependent monitor is likely to fire as well, and if that cascade runs unfiltered the root cause alert gets buried. Give each critical third-party API its own dedicated monitor with its own thresholds and escalation policy so suppression logic has a clean target.

Vendor status pages are useful as confirmation, not as detection. IncidentHub tracked more than 48,000 outages in 2025 across hundreds of SaaS and cloud services, and status pages are consistently a lagging indicator. Your synthetic monitors should already know about a problem before you check the vendor's status page.

Escalation paths when you can't fix it

Third-party incident escalation operates on different rules because your on-call engineer cannot deploy a fix to someone else's API. The escalation path needs to route to whoever actually has a lever to pull.

Map severity to available action. A P0, fast burn or full outage, means activate the circuit breaker or fallback immediately, open a vendor incident channel, and begin customer communication. The on-call engineer's role is containment, not repair, because there is nothing to repair on their end. A P1, medium burn or sustained degradation, means engage the vendor account team, begin documenting the degradation window for a potential SLA credit, and decide whether to activate graceful-degradation mode. A P2, slow burn or a trend, means schedule a vendor conversation and determine whether the observed behavior actually violates your SLA terms.

Every third-party alert runbook should answer three questions quickly. Is this the vendor or us? Check by comparing your independent synthetic monitor data against your own application error logs. What does our service do right now? This should be defined before any incident occurs: circuit breaker activation, cached response, degraded-mode response, or a user-facing status flag. Who communicates with customers? That is a support or account-team decision, not an engineering one, and it should not wait on a technical resolution.

If you want an SLA credit, you need documentation. Synthetic monitor data with timestamps and phase-level timing is the evidence that holds up in that conversation. Route alerts by who can take action: engineering on-call handles containment, the vendor relationship owner handles the external escalation, and product or customer support handles the public-facing message. These workstreams run in parallel because sequencing them just adds delay.

A starting config for new API integrations

Classify the dependency before writing a single alert rule. Revenue-path dependencies, including payment processors, identity providers, and core data APIs, require tight check intervals, full burn-rate alerting, and a fallback that is built and tested before launch. Supporting dependencies like email delivery, enrichment services, and analytics pipelines can run on longer intervals with slower burn-rate thresholds, since degradation there typically does not require an immediate response. That classification drives everything downstream: interval, threshold, escalation depth, and runbook detail.

For anything revenue-critical, the minimum viable setup includes a synthetic monitor running every 30 to 60 seconds from every region your users are in, using real authentication headers, with payload validation on every pass. Layer burn-rate alerting on top of that, tied to an SLO that is written down and shared with the team. Add a circuit breaker with a defined fallback state, and a runbook that answers the vendor-or-us question, specifies what the service does during degradation, and identifies who handles customer communication, all before the first incident rather than during it.

Omitting any of these pieces is not choosing a leaner setup. It is choosing to discover the gap during an incident, at the worst possible time, when the vendor's status page is still showing green.