LLM evals · observability · red team
Scorith is a core AI/ML product: an evaluation and observability platform for LLM products. Teams turn live traces into datasets, run graders in CI, and red-team agents so every release shares one quality bar. Scorith grades the agents you already ship — it is not itself an agent.
AI features
Scorith is in the business of building a core AI/ML product for LLM quality. Named jobs: traces, datasets from live misses, graders in CI, and red-team packs. It observes and grades agents (LangChain, LangGraph, homebrew). It does not act as one.
AI feature · Live traces to datasets
For teams drowning in production misses with no labeled set
Drop the SDK. Inspect agent trees, tool calls, cost, and latency on a single span. Pull live misses into a dataset. Product can add rows from the studio without waiting on a dump.
SDK span → POST /v1/traces → labeled dataset
AI feature · Graders in CI
For engineers who refuse a silent quality drop
Metric families you version in git. Scorith comments on the PR with the delta against main. The suite holds, or the merge waits. Same suite on a nightly sample.
PR → POST /v1/suites/{id}/run → GET /v1/suites/{id}/delta
AI feature · Red-team packs
For regulated desks that need a named owner and a calendar
Red-team agents so every release shares one quality bar. Adversarial chats and PDF-ready risk notes. A calendar, a pack, a named owner.
Pack → adversarial runs → risk notes
AI feature · Model-backed graders
For platform teams scoring LangChain, LangGraph, or homebrew agents
Planned model families: Claude 3.7 Sonnet, GPT-4o, Gemini 2.5, Llama 3 70B, Mistral Large. Access: Bedrock, Anthropic, OpenAI Direct, Hugging Face. Scorith is not fine-tuning those models.
Span sample → selected grader model → score on the hall
AI development tools / APIs. Machine learning, compliance, cybersecurity. MVP stage — no invented customers, funding, or uptime claims on this page. Scorith grades LLM products; it does not replace them.
Method
Instrument the tree. Collect gold from live misses. Gate the merge against the suite on main.
Drop the SDK. Every LLM and tool call becomes a span you can inspect, label, and replay.
Pull live misses into a set. Product can add rows from the dataset studio without waiting on a dump.
Scorith comments on the PR with the delta against main. The suite holds, or the merge waits.
Stack
A score hall. Trace, dataset, grade, red team — not a trained foundation model.
Inspect agent trees, tool calls, cost, and latency on a single span.
Auto-curate failures from production. Label once. Reuse in every PR.
Metric families you version in git. Merge only when the suite holds.
Adversarial chats and PDF-ready risk notes for regulated desks.
Start
Write founder@scorith.fun. You get a test token, a sample dataset, and a 30-minute walkthrough on the metrics you want to freeze.
Write founder@scorith.fun. This form stores the request on the page so you can copy it into that mail.
Thanks. We saved this request on the page. Mail founder@scorith.fun if you want a live reply.