Service Level Agreements (SLAs) for Stochastic Software: How to Promise 99.9% SLA Uptime on Probabilistic Models

Abstract:

We are currently witnessing a collision between two fundamental disciplines: Site Reliability Engineering (SRE), which demands deterministic guarantees (99.9% uptime), and Generative AI, which operates on probabilistic outputs. How do you sign a contract promising "accuracy" when the underlying engine is non-deterministic? The industry's current approach - treating "Model Uptime" as "Service Uptime"—is a category error. To build enterprise-grade AI, we must decouple the Availability of the Model from the Reliability of the Outcome. This requires a new architectural primitive: The Synthetic SLA.

I. The Stochastic Paradox

To a Principal Engineer at a cloud provider or a major SaaS platform, an SLA is a binary contract.

• Latency SLA: 99% of requests < 200ms.
• Availability SLA: 99.9% of requests return 200 OK.

For an LLM-based agent, these metrics are insufficient. An agent can return 200 OK in 150ms and still fail the customer's request by hallucinating a database column that doesn't exist.

The HTTP Status Code has been decoupled from the Semantic Status.

The Paradox:

We are selling Outcomes (e.g., "Book a Meeting"), but we are measuring API Calls.

If we want to sell "AI Employees" to the Fortune 500, we cannot just promise that the server is up. We must promise that the work is correct.

This requires us to invent a new metric: Reliability @ k (The probability of convergence on a valid state within $k$ attempts).

II. Architectural Primitive: The "Synthetic" SLA

You cannot guarantee that a single inference call ($k=1$) will be correct. The entropy of the model prevents this.

However, you can guarantee that a system of inference calls ($k=n$) will converge on a correct answer with $99.9$ probability.

This is the Law of Large Numbers applied to Software Engineering.

To sign a 99.9% SLA on a stochastic system, you must engineer a Synthetic SLA layer. This layer sits between the User and the Model, absorbing the variance of the underlying engine.

The Formula for Synthetic Reliability:

P(Success)=1−(1−p)nP(Success) = 1 - (1 - p)^n

P(Success)=1−(1−p)n

• $p$ = The success rate of a single model call (e.g., 80%).
• $n$ = The number of independent attempts (retries/paths) allowed.

If your base model has an accuracy of only 80% ($0.8$), running a "Best of 3" voting ensemble increases your system reliability to 99.2%.

Running a "Best of 5" increases it to 99.96%.

The Engineering Implication:

We are trading Compute for Reliability.

The SLA is no longer a function of code quality; it is a function of the Compute Budget allocated to verification loops.

III. The Triad of Verification: Implementing the Guarantee

To make this mathematical guarantee a reality, we must implement three specific architectural patterns.

1. The Deterministic Guardrail (Syntactic SLAs)

Before we check if the answer is smart, we must check if it is valid.

• The Component: A constrained decoding layer (using libraries like Instructor or Outlines) or a post-hoc Pydantic validator.
• The Guarantee: "We promise 100% adherence to the JSON Schema."
• Mechanism: If the model outputs a string instead of an integer, the system catches it, patches the prompt ("You output a string, I need an integer"), and retries. The user never sees the failure.
• SLA Impact: Eliminates 100% of "Format Hallucinations."

2. The Semantic Verifier (Logic SLAs)

This is the hardest layer. How do you promise the SQL query is correct without running it?

• The Component: A "Dry Run" Environment or a "Critic" Model.
• The Guarantee: "We promise 99.9% executable code."
• Mechanism:
• Agent: Generates SQL.
• Verifier: Runs EXPLAIN QUERY PLAN on a read-only replica.
• Observation: "Column usr_id does not exist."
• Reflector: "Ah, I meant user_id." -> Retry.
• SLA Impact: The "Service" does not return a response until the internal loop (Agent <-> Verifier) has converged.

3. The Human-in-the-Loop Circuit Breaker (The "9s" of Last Resort)

For mission-critical workflows (e.g., transferring funds), the "99.9%" comes from a fallback protocol.

• The Component: A Confidence Score Threshold.
• The Guarantee: "We promise we will not take an autonomous action unless confidence > 98%."
• Mechanism: If the Agent loops 3 times and still has low confidence (high entropy in the token distribution), the system escalates to a human review queue.
• The Trick: The SLA is not "We will do it automatically." The SLA is "We will get it done." If the AI fails, the SLA is preserved by routing the task to a human operator (the "Human Fallback").

IV. Latency Budgeting: The Cost of Reliability

The tradeoff for Synthetic SLAs is Latency.

If you need 3 retries to guarantee 99.9% accuracy, your P99 latency will triple.

The "Thinking Time" Agreement:

We must renegotiate the user's expectation of speed.

• Old SLA: "Response in 200ms."
• New SLA: "Acknowledgement in 200ms. Resolution in 2 minutes."

Architectural Pattern: The "Async-Ack"

• Request: User sends "Generate Report."
• Ack: System returns 202 Accepted + job_id.
• Process: The Agent enters the Verification Loop. It tries, fails, reflects, and succeeds.
• Webhook: System pushes the result to the user.

This allows us to hide the Volatility of Inference behind a stable Queue Interface. The user sees a reliable system; the backend sees a chaotic warzone of retries and error handling.

V. The Portfolio of Models: Hedging Risk

Finally, to guarantee uptime, we cannot rely on a single provider.

If OpenAI has an outage, your SLA is breached.

The "Router" Pattern:

Reliable Agentic Systems must be Model Agnostic.

• Primary: GPT-4 (Highest Intelligence).
• Secondary: Claude 3.5 (High Intelligence, different failure modes).
• Fallback: Llama 3 (Hosted on Groq/proprietary infra).

The Circuit Breaker:

If the Semantic Verifier detects that GPT-4 is stuck in a loop (hallucinating the same error twice), the Router automatically hot-swaps the backend to Claude for the next retry.

This is Cognitive Diversity. Different models have different blind spots. By ensembling them dynamically, we reduce the uncorrelated error rate to near zero.

Conclusion: From Uptime to "Outcome Reliability"

The era of "Five Nines of Availability" is ending. The era of "Five Nines of Correctness" is beginning.

For Staff Engineers and AI Scientists, the mandate is clear:

• Stop optimizing the model for perfection (it will never be perfect).
• Start optimizing the System for correction.

An SLA is not a promise that the machine won't break.

It is a promise that the System is robust enough to fix the machine before the user notices.

We are not building software that never fails. We are building software that refuses to stay broken.