Evaluations and Observability
AIF separates evaluation outcomes from operational metrics while attaching both to the governed execution lifecycle.
Evaluation Pipeline
Evaluators can run before or after execution. Each result uses one of three outcomes:
| Outcome | Behavior |
|---|---|
| block | Stop the operation and retain evidence. |
| warn | Allow the operation while marking a reviewable concern. |
| annotate | Add informational evidence without changing execution. |
Pre-execution checks can examine request and context. Post-execution checks can examine normalized provider output. Evaluation evidence is attached to the terminal audit record.
Metrics
MetricsCollectorInterface accepts vendor-neutral metric events for:
- execution count and latency
- token and cost metadata
- policy denials and review requirements
- retry and provider-health signals
- audit persistence failures
Metric collectors should not become a second audit store. Use metrics for operations and alerting; use audit records for accountable execution evidence.
Production Guidance
Choose bounded dimensions for metric labels, keep sensitive content out of telemetry, and correlate metrics with trace or request identifiers rather than copying prompts or responses.