Machine Learning Engineer – Evals

The Machine Learning Engineer for Evals will own end-to-end evaluation for an AI memory and identity system, shaping how quality is measured and ensuring evaluation infrastructure supports rapid iteration across models, prompts, and representations.

Responsibilities

• Define how to determine whether evolving entity representations are improving.
• Build infrastructure needed to measure improvements and track evaluation outcomes over time.
• Use evaluation results to identify failures and drive fixes through engineering or research changes.
• Define what “better” means for representations that evolve across updates.
• Design novel evaluation methodologies when traditional ground truth is unavailable or insufficient.
• Develop scoring frameworks, metrics, rubrics, judge models, and diagnostic methodologies.
• Continuously evolve evaluation definitions as models and product capabilities change.
• Build production-quality Python pipelines and evaluation harnesses.
• Manage datasets, labels, experiment versions, model versions, judges, reruns, and regression testing.
• Create systems that help researchers and engineers answer new evaluation questions quickly.
• Integrate experiment tracking and observability into evaluation workflows.
• Analyze model outputs, agent trajectories, and execution traces.
• Identify meaningful failure modes instead of optimizing for superficial metrics.
• Run experiments, compare results, document findings, and execute the resulting rerun and recommendations.
• Develop and operate simulation environments capable of modeling large numbers of entities.
• Increase throughput and reliability of evaluation runs.
• Build systems that enable rapid iteration across models, prompts, representations, and evaluation strategies.
• Own the evaluation process end to end: research question → evaluation design → data → implementation → execution → analysis → diagnosis → rerun → recommendation.

Required Qualifications

• 3+ years of professional experience designing and owning LLM evaluations.
• Demonstrated experience designing, building, and running LLM evaluations end to end.
• Production-quality Python experience, including testing, maintainability, debugging, and release practices.
• Experience creating evaluation methodologies, not only executing existing benchmarks.
• Strong ability to investigate model or agent failures using traces, outputs, datasets, and experimental results.
• Ability to work onsite in New York City five days per week.
• Unrestricted U.S. work authorization; visa sponsorship is not available.

Technologies

• Python
• PyTorch
• Hugging Face / Transformers
• Weights & Biases
• Hydra
• SQL
• Kafka

Preferred Technical Skills

• Weights & Biases
• Hydra
• SQL
• Kafka or other streaming or event-driven infrastructure
• Evaluation and observability frameworks
• Agent simulation and multi-agent systems
• Large-scale data pipelines

Nice to Have

• Experience with AI memory, identity, personalization, agent evaluation, or agentic optimization.
• Research experience in LLMs, NLP, machine learning, evaluation, or related areas.
• First-author or co-author research at NeurIPS, ICML, ICLR, ACL, EMNLP, or comparable venues.
• Alternatively, a high-quality open-source project, benchmark, framework, or technical publication demonstrating comparable research ability.
• Degree in Computer Science, Mathematics, Statistics, Machine Learning, or another quantitative discipline.

Benefits

• Employee assistance program

Compensation

USD 220,000 - 300,000 base salary per year plus competitive equity. Final compensation depends on experience, technical depth, research contributions, and interview performance.

Hiring Pro