LLM evaluation 2026: practical approaches to measuring model performance

[Aior]

Evaluation neden zor?​

Klasik yazılımda test "doğru cevap" var: 2+2 = 4. LLM çıktısında "doğru cevap" çoğu zaman birden fazla geçerli formda gelebilir — "Paris is the capital of France" ile "France's capital city is Paris" semantically eşit. AIOR projelerinde LLM evaluation klasik unit test'ten farklı yaklaşım gerektiriyor.

Evaluation kategorileri​

LLM uygulamalarında ölçtüğümüz boyutlar:

• Accuracy — model doğru cevap veriyor mu?
• Format compliance — output beklenen yapıda mı?
• Relevance — cevap sorunla ilgili mi?
• Coherence — cevap mantıksal akışta mı?
• Safety — model zararlı veya bias'lı içerik üretmiyor mu?
• Latency — yanıt ne kadar sürede geliyor?
• Cost — token bazında maliyet ne?

Gold set — manuel onaylanmış referans​

Her LLM uygulamasının kalbinde 50-200 input/expected-output pair'i olan bir gold set bulunmalı. AIOR'da gold set yaratım disiplini:

• Real production logs'tan örnek seçim.
• Edge case'ler dahil edilir (yaygın değil ama önemli).
• Domain expert tarafından expected output onaylanır.
• Her örnek tagged (intent, complexity, language).
• Düzenli refresh — domain değiştikçe gold set güncellenir.

Otomatik metric'ler​

Bazı kalite boyutları otomatik ölçülebilir:

• Exact match — output kesinlikle aynı mı (rare, sadece yapılandırılmış output).
• F1 / BLEU / ROUGE — text similarity (NLP klasik).
• Semantic similarity — embedding cosine similarity.
• JSON schema validation — yapısal doğruluk.
• Regex pattern match — belirli formatın varlığı.

AIOR'da semantic similarity LLM evaluation'da en sık kullandığımız teknik.

LLM-as-a-judge​

Otomatik metric'lerin yakalayamadığı subjective kalite (helpfulness, fluency) için başka bir LLM judge olarak çağrılır:

Sen bir uzman değerlendirici. İki cevabı karşılaştır:
Sorgu: [user question]
Cevap A: [model output 1]
Cevap B: [model output 2]
Hangisi daha iyi ve neden? (1-5 puan ölçeğinde her birine puan ver)

AIOR'da Claude'u GPT outputlarını değerlendirirken (veya tam tersi) judge olarak kullanıyoruz. Avantaj: scale (binlerce eval otomatik). Risk: judge bias'ları aktarır.

Human evaluation — gold standard​

LLM-as-a-judge yeterli olmadığı durumlarda human evaluator gerekli. AIOR projelerinde:

• Sample selection — production log'lardan random örnek (haftalık 50-100 örnek).
• Blind evaluation — değerlendirici hangi version'ın output'u olduğunu bilmemeli.
• Multiple rater — inter-rater agreement ölçülür (Cohen's Kappa).
• Rubric — net değerlendirme kriterleri.

Production metrics — gerçek dünyada​

Eval suite controlled environment iyi ama production farklı:

• CSAT (Customer Satisfaction) — kullanıcı son response'a reaction ile.
• Task completion rate — kullanıcı amacına ulaştı mı?
• Escalation rate — human agent'a aktarma oranı.
• Session length — kısa olması iyi olabilir (hızlı çözüm) veya kötü (kullanıcı vazgeçti).
• Retry rate — kullanıcı sorusunu tekrar sorma sıklığı.

A/B testing​

Yeni prompt veya model değişikliğini production'da deneme. AIOR'ın A/B test disiplin:

• Kontrollü ramp — %5 → %25 → %50 → %100.
• Statistical significance — yeterli sample size.
• Metrics dashboard real-time.
• Auto-rollback kuralı — metric belirli eşiği aşarsa otomatik geri dön.

Safety ve bias evaluation​

Model toksik veya bias'lı içerik üretiyor mu? AIOR'ın test bataryası:

• Toxicity classifier (Perspective API, Detoxify).
• Bias benchmark suites (BBQ, BOLD, StereoSet).
• Adversarial prompt'lar — jailbreak denemeleri.
• Demographic parity — farklı kullanıcı gruplarında benzer performans?

Cost tracking​

Modern LLM'ler maliyetli. Evaluation cost-aware olmalı:

• Token usage per session.
• Cost per 1000 successful task.
• Cache hit ratio (Anthropic prompt caching).
• Model selection — pahalı model her zaman daha iyi mi?

Sonuç​

LLM evaluation klasik test'ten farklı, multi-dimensional bir disiplin. Gold set + otomatik metric + LLM-as-judge + human evaluation + production metric kombinasyonu ile model performansı objektif olarak ölçülebilir. AIOR olarak müşteri LLM projelerinde evaluation pipeline'ını standart paket halinde teslim ediyoruz. Sizin tarafınızda LLM evaluation için en sık kullandığınız framework hangisi — LangSmith, custom internal tool, yoksa manuel review mi?

Why is evaluation hard?​

In classic software, tests have "the right answer": 2+2 = 4. With LLM output, "the right answer" can often come in multiple valid forms — "Paris is the capital of France" is semantically equivalent to "France's capital city is Paris." LLM evaluation on AIOR projects requires a different approach than classic unit testing.

Evaluation categories​

Dimensions we measure on LLM applications:

• Accuracy — does the model give correct answers?
• Format compliance — is the output in the expected shape?
• Relevance — is the answer related to the question?
• Coherence — does the answer follow logical flow?
• Safety — is the model producing harmful or biased content?
• Latency — how long does the response take?
• Cost — what does it cost in tokens?

Gold set — manually approved reference​

At the heart of every LLM application should be a gold set of 50-200 input/expected-output pairs. Gold-set creation discipline at AIOR:

• Selection from real production logs.
• Edge cases included (uncommon but important).
• Expected output approved by domain expert.
• Each example tagged (intent, complexity, language).
• Regular refresh — gold set updates as domain evolves.

Automatic metrics​

Some quality dimensions can be measured automatically:

• Exact match — is output strictly identical (rare, only for structured output).
• F1 / BLEU / ROUGE — text similarity (NLP classics).
• Semantic similarity — embedding cosine similarity.
• JSON schema validation — structural correctness.
• Regex pattern match — presence of a specific format.

Semantic similarity is the technique we use most in LLM evaluation at AIOR.

LLM-as-a-judge​

For subjective quality (helpfulness, fluency) that automatic metrics can't catch, another LLM is called as a judge:

You are an expert evaluator. Compare two answers:
Query: [user question]
Answer A: [model output 1]
Answer B: [model output 2]
Which is better and why? (Rate each on a 1-5 scale)

AIOR uses Claude as judge evaluating GPT outputs (or vice versa). Advantage: scale (thousands of evals automated). Risk: the judge's biases carry over.

Human evaluation — the gold standard​

When LLM-as-a-judge isn't enough, human evaluators are needed. On AIOR projects:

• Sample selection — random samples from production logs (50-100 per week).
• Blind evaluation — the rater shouldn't know which version produced the output.
• Multiple raters — inter-rater agreement measured (Cohen's Kappa).
• Rubric — explicit evaluation criteria.

Production metrics — in the real world​

The eval suite is good in a controlled environment but production is different:

• CSAT (Customer Satisfaction) — user reaction to the final response.
• Task completion rate — did the user reach their goal?
• Escalation rate — handoff to a human agent rate.
• Session length — short can be good (fast resolution) or bad (user gave up).
• Retry rate — how often users re-ask the same question.

A/B testing​

Try a new prompt or model change in production. AIOR's A/B test discipline:

• Controlled ramp — 5% → 25% → 50% → 100%.
• Statistical significance — adequate sample size.
• Real-time metrics dashboard.
• Auto-rollback rule — if a metric crosses a threshold, automatic rollback.

Safety and bias evaluation​

Is the model producing toxic or biased content? AIOR's test battery:

• Toxicity classifier (Perspective API, Detoxify).
• Bias benchmark suites (BBQ, BOLD, StereoSet).
• Adversarial prompts — jailbreak attempts.
• Demographic parity — similar performance across user groups?

Cost tracking​

Modern LLMs are costly. Evaluation must be cost-aware:

• Token usage per session.
• Cost per 1000 successful tasks.
• Cache hit ratio (Anthropic prompt caching).
• Model selection — is the more expensive model always better?

Bottom line​

LLM evaluation is different from classic testing — a multi-dimensional discipline. The combination of gold set + automatic metrics + LLM-as-judge + human evaluation + production metrics measures model performance objectively. AIOR delivers an evaluation pipeline as a standard package on customer LLM projects. What's your most-used framework for LLM evaluation — LangSmith, custom internal tool, or manual review?