LLM Evaluator – Model Response Analyst (Remote)

About the role

We are looking for a detail‑oriented LLM Evaluator (Model Response Analyst) to join our remote team. In this position you will assess large language model outputs for factual accuracy, coherence, safety, bias, and alignment with our guidelines, helping to continuously improve model performance.

Key responsibilities

• Evaluate and rank model‑generated text using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple responses to the same prompt and justify the preferred output from a human perspective.
• Provide concise feedback to modeling and training teams about recurring failure patterns.
• Design adversarial prompts to expose biased, harmful, or insecure model behavior.
• Collaborate with QA to refine evaluation guidelines and resolve ambiguous edge cases.
• Participate in cross‑checking sessions to calibrate scoring and ensure inter‑rater reliability.
• Analyze underperforming outputs, hypothesize root causes, and flag novel model behaviors for research.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP quality assurance, or cognitive science.
• Bachelor’s degree in computer science or a related field.
• Strong ability to explain why a model output is good or bad based on logic, tone, factuality, and instruction adherence.
• Experience with RLHF data collection and evaluation consistency across teams.

Required skills

• Prompt engineering
• Reinforcement Learning from Human Feedback (RLHF) data collection
• Dataset sourcing, cleaning, and annotation for LLM fine‑tuning or evaluation
• A/B testing applied to AI models
• Evaluation rubric development and statistical analysis of IAA scores