AI Agent Skills Benchmark
Klaat bundles official AI formatting skills (such as klaat-markdown-author, klaat-cli, and klaat-gdocs-author) that install into 55+ coding agent environments via klaat install-skills. To ensure these guidelines genuinely improve agent output and prevent hallucination, we evaluate them against both live LLM agents and deterministic compiler validation.
A/B Benchmark Scorecard
We compare an agent with only standard programming knowledge (Baseline) against an agent with access to Klaat's skills catalog (With Klaat Skill).
Evaluation Model: gemini-3.5-flash-lite • Run Mode: Live API Evaluation • Last Evaluated: Oct 22, 2026 05:58 UTC
| Evaluation Criteria | Baseline (No Skill) | With Klaat Skill | Net Improvement |
|---|---|---|---|
Klaat Codelab Compilation |
0% | 50% | +50% |
Duration Tag Presence |
17% | 50% | +33% |
Klaat Callout Markers |
0% | 50% | +50% |
Test Suite Prompt Breakdown
The prompt suite includes target codelab authoring requests as well as distractor prompts (general documentation, standard READMEs, blog posts) to verify that skills do not hijack non-codelab markdown tasks.
| Prompt ID | Difficulty | Category | Autonomous Trigger | Klaat Compiler Result | Prompt Preview |
|---|---|---|---|---|---|
explicit-coroutine-workshop |
Easy | explicit-tutorial | Triggered | Compiled OK | Create a 3-step Klaat codelab on Kotlin Coroutines for our team workshop. I... |
explicit-git-conversion |
Easy | explicit-conversion | Triggered | Compiled OK | Format this technical guide on Git branching strategies into a Klaat codela... |
implicit-ktor-tutorial |
Medium | implicit-tutorial | Triggered | Compile Error | Help me write a step-by-step hands-on tutorial for building a REST API in K... |
implicit-compose-state |
Medium | implicit-tutorial | Triggered | Compiled OK | I need an interactive multi-step coding guide on Jetpack Compose state hois... |
multi-step-duration-consistency |
Hard | edge-case-adherence | Triggered | Compile Error | Write a 5-step interactive workshop tutorial on SQL migrations with Flyway.... |
rich-elements-survey |
Hard | edge-case-adherence | Triggered | Compile Error | Create a Klaat codelab on unit testing with MockK. Include code blocks with... |
distractor-contributing-guide |
Medium | hard-distractor | Ignored (Correct) | - | Write a step-by-step guide in a CONTRIBUTING.md file on how to contribute t... |
distractor-blog-post |
Medium | hard-distractor | Ignored (Correct) | - | Write a technical blog post walking through how garbage collection works in... |
distractor-deployment-runbook |
Hard | hard-distractor | False Trigger | - | Create a step-by-step production deployment checklist and runbook in markdo... |
distractor-readme |
Easy | easy-distractor | Ignored (Correct) | - | Write a standard README.md file for a Kotlin Multiplatform networking libra... |
Evaluation Methodology
Our eval suite uses a two-stage approach combining autonomous agent simulation and compiler-backed deterministic validation:
psychology 1. Autonomous Invocation Test
We do not hardcode SKILL.md into the prompt. Instead, we present the agent with a catalog of skill descriptions and a read_skill tool. We check whether the model independently chooses to invoke read_skill("klaat-markdown-author") when handed a codelab prompt, and whether it correctly leaves the tool untouched for general markdown requests.
fact_check 2. Deterministic Compiler Verification
Rather than relying only on subjective LLM judges, we feed the agent's raw output directly into Klaat's official MarkdownParser. A test only passes if the output extracts valid YAML metadata (id, summary, categories), valid per-step durations (Duration: > 0), and produces compilable HTML with zero parser errors.
./gradlew evalSkills -PupdateDocs=true