OpenClaw Atlas

Overview

OpenClaw Atlas is a comprehensive evaluation framework built at Outlier.ai to benchmark frontier AI models on agentic, multi-server reasoning tasks. The system exposes LLMs to deliberately contradictory information spread across six synthetic data servers — mimicking real-world knowledge fragmentation — and measures how well each model synthesises, reconciles, and reasons across those sources.

Architecture

The framework consists of three layers:

1. ▸Synthetic Data Layer — Six independently curated datasets across logistics and healthcare domains, each containing 12-field Story Drafts with deliberate cross-server contradictions injected at known positions.
2. ▸Evaluation Harness — A Python orchestration layer that feeds identical prompt sets to four frontier AI models, isolating model capability from prompt variance.
3. ▸Failure Taxonomy — A structured rubric for categorising model failures: instruction-following gaps, hallucination under ambiguity, safety enforcement edge-cases, coherence breakdowns, and context-persistence failures.

Key Outcomes

The framework produced a reusable evaluation benchmark applicable to future model releases, and surfaced model-specific failure modes that informed annotation guidelines for subsequent RLHF batches.