LalaEval: A Holistic Human Evaluation Framework for Domain-Specific Large Language Models

2 min read

LalaEval is a procedure for human evaluation of LLMs inside one company and domain, worked through for intracity freight at the logistics firm Huolala. It has five parts. A hierarchical domain specification runs from intracity freight up to logistics, ranked P0 to P3. Capability criteria split into general skills (semantic understanding, context, completeness, factuality, creativity, logic) and domain skills (terminology, companies, law and policy, industry insight, company-specific knowledge, creative domain writing). A question bank records the source of every answer and passes a quality check. A 0 to 3 rubric gives 0 to any answer containing incorrect information. Answers are anonymised, shuffled and graded single-blind by at least three trained evaluators.

Findings

  • Dispute analysis flags a grader when their grade is non-zero and every other grader gave zero, or the reverse. It flags a question when graders split roughly in half. High-dispute graders are retrained and their grades may be voided; high-dispute questions are reviewed and dropped if faulty.
  • Grade changes between rounds are attributed to four causes: the question changed, the model’s answer changed, the same grader was inconsistent, or the graders changed.
  • Accuracy (share of non-zero grades), overall: Ernie Bot 88.0%, the in-house PLLM3 77.5%, PLLM2 74.5%, PLLM1 71.9%, GPT-4 67.1%.
  • Domain factuality: GPT-4 without web access scores 45.0%, PLLM3 (a ChatGLM2-6B fine-tune with web access and retrieval) 93.2%. On general capability GPT-4 scores 80.0% and PLLM3 68.0%.

Limits

  • The authors list grader subjectivity, data selection bias, domain knowledge that goes stale, and scale.
  • The comparison mixes models with and without web access, so “domain factuality” partly measures retrieval.
  • One company, Chinese-language logistics, proprietary questions and rubrics.
  • The example dispute rules only look at zero versus non-zero, so a 1 against a 3 is never a dispute.

Relevance

The dispute analysis uses disagreement among graders to find bad graders and bad questions before it gets averaged away. The same move applies when robots grade each other’s reports, an open question for robust robotics and byzantine-fault-tolerance: a lone dissenter can be a faulty grader or the only one who saw what happened. See llm-evaluation and the LLM-as-a-Judge survey.