AI / LLM Evaluation QA interview questions and answers

9 of 9 questions shown

1. juniorBasicsWhy are exact-match assertions not enough for an LLM feature?

   Model answer. Outputs are non-deterministic and many phrasings are valid. Combine constraint checks (format, length, required fields), rubric-based scoring and semantic comparison, and judge results statistically over a dataset instead of one run. Low temperature reduces variance but does not remove it.

   Tools: Python

   I can answer this
2. juniorDatasetsWhat makes a good evaluation dataset?

   Model answer. It reflects real traffic and covers edge and adversarial cases. It is labelled against written criteria and versioned. It is kept separate from data used to tune prompts, and it includes known production failures.

   I can answer this
3. midMetricsExplain precision and recall for a toxic-content classifier and which you prioritise.

   Model answer. Precision is TP/(TP+FP) and recall is TP/(TP+FN). Prioritise by the cost of errors: for safety filtering favour recall with human review of flags, for automatic account bans favour precision. Report the confusion matrix and confidence intervals, not a single number.

   I can answer this
4. midRAGHow do you measure hallucination in a retrieval-augmented assistant?

   Model answer. Evaluate retrieval separately (recall@k, MRR) from generation. For generation, check faithfulness: each claim must be supported by the retrieved passages, judged by humans or a calibrated LLM judge. Include unanswerable questions where the correct behaviour is to abstain.

   I can answer this
5. midJudgingWhat are the risks of using an LLM as a judge?

   Model answer. Position bias, verbosity bias, self-preference and inconsistency. Calibrate against human labels with an agreement measure such as Cohen's kappa, randomise answer order, use clear rubrics and spot-check regularly.

   I can answer this
6. midSafetyWhich categories do you cover when red-teaming a chatbot?

   Model answer. Direct and indirect prompt injection (through retrieved documents), jailbreaks, system-prompt and PII leakage, harmful content, bias, tool misuse and excessive agency. Record attack success rate per category and add every fixed case to a regression set.

   I can answer this
7. seniorRegressionHow do you detect regressions when you change a prompt or model version?

   Model answer. Run a frozen evaluation set on both versions, compare per-slice metrics with a paired significance test such as bootstrap, review the individual flipped cases and canary online metrics. Pin model versions and keep prompts in version control with CI gating.

   Tools: Python

   I can answer this
8. seniorLabellingHow do you ensure labelling quality across annotators?

   Model answer. Write guidelines with examples and run calibration rounds. Measure inter-annotator agreement (Cohen's kappa or Krippendorff's alpha) and insert gold questions. Adjudicate disagreements and monitor drift over time.

   I can answer this
9. seniorEngineeringHow would you build a minimal automated evaluation harness in Python?

   Model answer. A JSONL dataset feeds a function that calls the model with fixed parameters and records the model version, then scorers run (exact, regex, JSON-schema validation, LLM judge). Aggregate by slice into a report and store runs with metadata so they can be diffed. Handle rate limits with retries and cache identical calls.

   Tools: Python

   I can answer this

No question matches those filters. Clear the search or choose All.