Evals Lead

1 week ago

San Francisco, California, United States Cartesia Full time

About Cartesia
Our mission is to build the next generation of AI: ubiquitous, interactive intelligence that runs wherever you are. Today, not even the best models can continuously process and reason over a year-long stream of audio, video and text—1B text tokens, 10B audio tokens and 1T video tokens—let alone do this on-device.

We're pioneering the model architectures that will make this possible. Our founding team met as PhDs at the Stanford AI Lab, where we invented State Space Models or SSMs, a new primitive for training efficient, large-scale foundation models. Our team combines deep expertise in model innovation and systems engineering paired with a design-minded product engineering team to build and ship cutting edge models and experiences.

We're funded by leading investors at Index Ventures and Lightspeed Venture Partners, along with Factory, Conviction, A Star, General Catalyst, SV Angel, Databricks and others. We're fortunate to have the support of many amazing advisors, and 90+ angels across many industries, including the world's foremost experts in AI.

About The Role
The New Horizons Evaluations team is reimagining how we measure progress in interactive machine intelligence. As
Evaluations Lead
, you will design evaluation frameworks that capture not just what models know — but how they reason, remember, and interact over time. You'll work at the intersection of research, product, and infrastructure to develop metrics, systems, and studies that define what "intelligence" means in the next generation of AI. This role is ideal for someone who combines scientific rigor with technical execution, and who's deeply curious about how people use — and want to use — intelligent systems. Your work will shape how Cartesia builds and evaluates frontier models, ensuring that progress isn't measured solely by static benchmarks, but by deeper qualities like understanding, naturalness, and adaptability in real-world interaction.

Your Impact

  • Identify and define key model capabilities and behaviors that matter for next-generation model evals
  • Develop and implement new evaluation pipelines with robust statistical analysis and clear reporting
  • Partner closely with model training and research teams to embed evaluation systems directly into model development loops
  • Prototype new user studies and behavioral experiments to ground evaluations in real-world use.

What You Bring

  • Experience designing or implementing evaluation frameworks for generative models (audio, text, or multimodal)
  • Strong technical and analytical skills, ability to take open-ended research ideas and translate them into production-ready systems
  • Creativity in defining novel quantitative metrics for subjective or behavioral qualities
  • Excitement for building evaluation systems that bridge research and real-world use
  • Curiosity and rigor in equal measure; motivation driven by discovering how to measure meaningful progress in intelligent behavior

What Sets You Apart

  • Understanding of model alignment concepts and evaluation approaches

Our culture
We're an in-person team based out of San Francisco. We love being in the office, hanging out together and learning from each other everyday.

We ship fast. All of our work is novel and cutting edge, and execution speed is paramount. We have a high bar, and we don't sacrifice quality and design along the way.

We support each other. We have an open and inclusive culture that's focused on giving everyone the resources they need to succeed.

  • Strategic Product Manager

    4 days ago

    San Francisco, California, United States Scale AI Full time $201,600 - $241,920 per year

    About The RoleWe're hiring a Strategic Product Manager to sit at the intersection of frontier AI research, product, and go-to-market. You'll partner closely with account executives in high-stakes meetings, scope and pitch solutions to top AI labs, and translate research needs (post-training, evals, alignment) into clear product roadmaps and measurable...

  • Applied Research

    1 week ago

    San Francisco, California, United States Prime Intellect Full time $120,000 - $180,000 per year

    Building Open Superintelligence InfrastructurePrime Intellect is building the open superintelligence stack - from frontier agentic models to the infra that enables anyone to create, train, and deploy them. We aggregate and orchestrate global compute into a single control plane and pair it with the full RL post-training stack: environments, secure sandboxes,...

  • Applied AI Software Engineer

    5 days ago

    San Francisco, California, United States Canvas Medical Full time $180,000 - $300,000 per year

    Canvas Medical is the electronic medical records (EMR) and payments development platform for healthcare. We build modern, elegant front- and back-end tooling to enable new ways for developers and clinicians to collaborate to solve healthcare's toughest challenges. Canvas is institutionally backed by some of the greatest technology investors in the world...

  • Senior Full Stack Engineer, Observability

    3 days ago

    San Francisco, California, United States LangChain Full time

    About LangChainAt LangChain, our mission is to make intelligent agents ubiquitous. We provide the agent engineering platform and open source frameworks developers need to ship reliable agents fast.Our open source frameworks, LangChain and LangGraph, see over 90+ million downloads per month and help developers build agents with speed and granular control....

  • AI Product Engineer

    7 days ago

    San Francisco, California, United States Ramp Full time $200,000 - $250,000 per year

    About RampAt Ramp, we're rethinking how modern finance teams function in the age of AI. We believe AI isn't just the next big wave. It's the new foundation for how business gets done. We're investing in that future — and in the people bold enough to build it.Ramp is a financial operations platform designed to save companies time and money. Our all-in-one...

  • Engineering Manager, Internal Tools, AGI Autonomy

    6 days ago

    San Francisco, California, United States Amazon Full time $200,000 - $400,000 per year

    DESCRIPTIONAmazon has launched a new research lab in San Francisco to develop foundational capabilities for useful AI agents. We're enabling practical AI to make our customers more productive, empowered, and fulfilled. In particular, our work combines large language models (LLMs) with reinforcement learning (RL) to solve reasoning, planning, and world...

  • Cloud Infrastructure Engineer

    1 week ago

    San Francisco, California, United States Braintrust Full time $140,000 - $160,000 per year

    About the companyBraintrust is the AI observability platform. By connecting evals and observability in one workflow, Braintrust gives builders the visibility to understand how AI behaves in production and the tools to improve it.Teams at Notion, Stripe, Zapier, Vercel, and Ramp use Braintrust to compare models, test prompts, and catch regressions — turning...

  • AI Engineer

    3 hours ago

    San Francisco, California, United States Sphere, Inc. Full time

    About the roleMission:Sphere has built the world's first AI-native tax engine that helps companies like Lovable, Eleven Labs, Replit, Runway AI, Windsurf (+ many more) automate their sales tax, VAT and GST obligations:We're growing rapidly (34%+ MoM revenue growth) and helping the fastest growing tech companies around the world including Lovable, Eleven...

  • Head of Growth

    13 hours ago

    San Francisco, California, United States Prime Intellect Full time

    Building Open Superintelligence InfrastructurePrime Intellect is building the open superintelligence stack - from frontier agentic models to the infra that enables anyone to create, train, and deploy them. We aggregate and orchestrate global compute into a single control plane and pair it with the full rl post-training stack: environments, secure sandboxes,...

  • Sr. Staff Applied AI Engineer

    1 week ago

    San Francisco, California, United States Quizlet Full time $282,000 - $344,000

    About Quizlet:At Quizlet, our mission is to help every learner achieve their outcomes in the most effective and delightful way. Our $1B+ learning platform serves tens of millions of students every month,  including two-thirds of U.S. high schoolers and half of U.S. college students, powering over 2 billion learning interactions monthly.We blend cognitive...