Business outcomes
Seven distinct cases: warranty support, dispatch planning, approved purchasing, incident remediation, customer preferences, invoice reconciliation and price comparison.
Benchmarks
Evaluating outcomes and responsibility boundaries
Task completion and respected authority answer complementary questions. We evaluate business outcomes, governance contracts and their representation separately.
Common conditions
The same business requests, service operations and isolated synthetic records. Controlled instructions, requests and records are in English. Authorized operations can be reused.
Multi-agent orchestration
The incident and invoice cases each involve three sequential agents, with corresponding responsibilities and operation subsets in each system.
17 October 2026 · 7 models · 7 cases · 5 repetitions
Business-case completion
The study includes 1,225 executions. Each bar shows completions out of 35 trials: seven business cases repeated five times. The five methods appear in the same order in every figure.
OpenAI
completions out of 35GPT-4o miniGPT-4.1 miniGPT-6 AstraAnthropic
completions out of 35Claude Haiku 5.5Claude Sonnet 5.5Completions out of 35 observations per model and method. All three figures use the same 0–35 scale.
Native Touba reaches 35/35 with six models and 30/35 with GPT-4o mini. It matches the highest observed count for each of the seven models. Astra, Sonnet and Argon bring all five methods to 35/35.
Across all five methods, 1,158 executions succeed and 67 fail: 31 produce incorrect or incomplete business results, and 36 reach the limit of 12 service operations before completing the task.
View all exact counts
| Model | Touba structured | Agents SDK | LangChain | LangGraph | Touba native |
|---|---|---|---|---|---|
| GPT-4o minigpt-4o-mini-2024-07-18 | 21/35 | 21/35 | 25/35 | 30/35 | 30/35 |
| GPT-4.1 minigpt-4.1-mini-2025-04-14 | 34/35 | 35/35 | 35/35 | 31/35 | 35/35 |
| GPT-6 Astragpt-6-astra | 35/35 | 35/35 | 35/35 | 35/35 | 35/35 |
| Claude Haiku 5.5claude-haiku-5-5 | 31/35 | 34/35 | 33/35 | 35/35 | 35/35 |
| Claude Sonnet 5.5claude-sonnet-5-5 | 35/35 | 35/35 | 35/35 | 35/35 | 35/35 |
| Gemini 3.8 Flashgemini-3.8-flash | 35/35 | 33/35 | 30/35 | 35/35 | 35/35 |
| Gemini 4 Argongemini-4-argon | 35/35 | 35/35 | 35/35 | 35/35 | 35/35 |
Paired differences between the Touba interfaces
Difference = native success rate − structured success rate, in percentage points. A positive difference favors native; a negative difference favors structured; zero means equal observed rates.
To estimate variation from the choice of cases, the analysis recalculates the difference on 10,000 samples drawn with replacement from the seven cases. Each sampled case retains its five repetitions. These calculations produce the descriptive 95% intervals below. All intervals include zero: the data remain compatible with a zero difference.
| Model | Difference (points) | Favors | 95% interval |
|---|---|---|---|
| GPT-4o mini | +25.7 | Native | [0; 57.1] |
| GPT-4.1 mini | +2.9 | Native | [0; 8.6] |
| GPT-6 Astra | 0 | Equal | [0; 0] |
| Claude Haiku 5.5 | +11.4 | Native | [0; 28.6] |
| Claude Sonnet 5.5 | 0 | Equal | [0; 0] |
| Gemini 3.8 Flash | 0 | Equal | [0; 0] |
| Gemini 4 Argon | 0 | Equal | [0; 0] |
Conditions and interpretation
- Study design
Each model runs all seven cases with each of the five methods, five times: 7 × 5 × 7 × 5 = 1,225 executions. Method order varies across cases and repetitions. Exact model API identifiers appear in the results table.
- Limits per execution
Each execution allows 24 model calls and 12 service operations, shared across its agents. Reads count as service operations too. These counters reset for each new trial. Each provider request has a 120-second timeout; automatic retries are disabled.
For a given model, settings stay the same across all five methods. The two OpenAI mini models use temperature 0 and a maximum of 1,000 output tokens. The other models use low reasoning effort and model-specific output limits.
- Checking the result
An automated checker examines saved data, completed operations and the requested outcome. For price comparison, it checks the final answer and its expected format. Purchase approval is simulated, and customer preferences are stored in a service shared by all five methods. Touba’s own approval and memory mechanisms have dedicated tests.
Observed example: four items at €189 cost €756. An agent used the €1,000 purchase ceiling as the order amount. The simulated approver accepted that amount; the checker detected the incorrect amount. The business rule to check here is: amount = quantity × unit price.
- History and context
For each agent invocation, Touba retains original messages in memory, along with provider information needed to continue the conversation. History has a size limit and belongs to that invocation. Automatic summarization is disabled in this study. Both Touba modes support a final answer in the format defined by the program.
All five methods check input size before each model call. The token count is increased by 20%, then by 512 tokens, and compared with a 100,000-token ceiling. The financial budget also reserves the maximum cost of the response.
- Reading the results
The five repetitions show how consistently each method handles the seven cases. The comparison between Touba interfaces keeps each case’s repetitions together. These findings cover the seven cases studied; testing other task families would broaden their scope.
Two ways to communicate with the model
In structured mode, the model describes its choice and parameters in a response format defined by Touba. This is the default mode. Native mode uses the provider’s tool-calling format, enabled in the SDK configuration. Touba applies the same permissions and retains the conversation history in both modes.
Across the cases studied, native mode completes more executions with three models. The other four achieve the same number of completions in both modes.
Structured: default
import { createAiSdkDecisionProvider } from "@touba/sdk";
export const decideAgentCapability =
createAiSdkDecisionProvider({
model,
decisionInterface: "structured",
instructions: "Use observed data and authorized capabilities.",
});
Native: opt-in
import { createAiSdkDecisionProvider } from "@touba/sdk";
export const decideAgentCapability =
createAiSdkDecisionProvider({
model,
decisionInterface: "native",
instructions: "Use observed data and authorized capabilities.",
});
Implementation invariant coverage
Each check pairs a refused proposal with a permitted path. Evidence combines the contract, admission decision and observed effects: blocking an operation at the required boundary demonstrates enforcement.
Trace forensicsReconstruct dispatch, refusal and human pause from events.
Trigger identityAccept a valid event and refuse replay or source mismatch.
Memory provenancePreserve metadata without exposing raw content or secrets in evidence.
Memory contaminationKeep injected content out of public traces across reads and writes.
BudgetsRefuse excess tool calls, agent invocations and dispatch after timeout.
Supply-chain integrityRefuse a missing lock or altered handler and manifest artifacts.
Controlled stopApply cancellation and kill switch at tested dispatch boundaries.
Delegation and reviewCheck caller authority and preserve continuations across human pauses.
Language and SDK tests
Compilation, declaration metadata, types, imports, loops, serialization and typed completion are checked separately from authorization contracts. V1 construct coverage is documented with the language and will evolve with its releases.
What formal evidence establishes
Probabilistic Agentic Programming (PAP) defines responsibility obligations. The chosen kernel establishes properties under its mediation, trust and sequential-execution assumptions. A separate theorem relates a defined Touba fragment to its execution graph. Compiled tests examine implementation correspondences.
Finite exploration covers 4,608 kernel configurations: eight Boolean choices, one two-valued human mode and two three-valued limits, giving 2⁸ × 2 × 3 × 3. The 203,712 states and 314,904 transitions are reachable situations and their changes, counted per configuration.