Research Engineer, ML Systems

2 days ago


Menlo Park CA United States Character.AI Full time

Joining us as a Research Engineer on the ML Systems team, you’ll be working on cutting-edge ML training and inference systems, optimizing the performance and efficiency of our GPU clusters, and developing new technologies that fine-tune leading consumer AI models with a data flywheel, and serve 20K+ QPS in production with LLMs. Your work will directly contribute to our groundbreaking advancements in AI, helping shape an era where technology is not just a tool, but a companion in our daily lives. At Character.AI , your talent, creativity, and expertise will not just be valued—they will be the catalyst for change in an AI-driven future.

About the role

The ML Systems team is responsible for the research and deployment of systems that efficiently utilize GPU for AI-enabled products.

As a research engineer, you will work across teams and our technical stack to improve our training performance and inference runtime. You will get to shape the conversational experience of millions of users per day.

Example projects:

  • Write efficient Triton kernels and tune them for our specific models and hardware

  • Develop prefix-aware routing algorithms to improve serving cache hit rate

  • Train and distill LLMs to improve latency while preserving accuracy and engagements

  • Build an efficient and scalable distributed RLHF stack powering the model innovations

  • Develop systems for efficient multimodal (image gen/video gen) model training & inference

Job Requirements
  • "All Industry Levels": at least PhD (or equivalent) research experience

  • Write clear and clean production system code

  • Strong understanding of modern machine learning techniques (reinforcement learning, transformers, etc)

  • Track record of exceptional research or creative ML systems projects

  • Comfortable writing model development code (PyTorch) for either training or inference

Nice to Have
  • Experience training large models in a distributed setting utilizing PyTorch distributed, DeepSpeed, Megatron.

  • Experience working with GPUs & collectives (training, serving, debugging) and writing kernels (Triton, CUDA, CUTLASS).

  • Experience with LLM inference systems and literature such as vLLM and FlashAttention.

  • Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)

  • Publications in relevant academic journals or conferences in the field of machine learning and systems

About Character.AI

Founded in 2021, Character is a leading AI company offering personalized experiences through customizable AI 'Characters.' As one of the most widely used AI platforms worldwide, Character enables users to interact with AI tailored to their unique needs and preferences.

In just two years, we achieved unicorn status and were named Google Play's AI App of the Year – a testament to our groundbreaking technology and vision.

Ready to shape the future of Consumer AI?

At Character, we value diversity and welcome applicants from all backgrounds. As an equal opportunity employer, we firmly uphold a non-discrimination policy based on race, religion, national origin, gender, sexual orientation, age, veteran status, or disability. Your unique perspectives are vital to our success.

#J-18808-Ljbffr
  • ML Research Engineer

    2 weeks ago


    Menlo Park, United States Diffuse Bio Full time

    The role: Extend and scale Diffuse's in-house deep generative modeling toolkit for    downstream applications in molecular design. Thoughtfully execute deep learning experiments to improve performance    of models or develop new functionality (e.g. loop engineering,    structure prediction of protein-protein complexes). Work closely with software...


  • Menlo Park, United States META Full time

    Summary: The PyTorch Compiler team is dedicated to making PyTorch run faster and more resource-efficient without sacrificing its flexibility and ease of use. The team is the driving force behind PT2, a step function change in PyTorch’s history that brought compiler technologies to the core of PyTorch. PT2 technologies have gained industry-wide recognition...


  • Menlo Park, United States META Full time

    Software Engineer, Systems ML - PyTorch Compiler / ML Framework / Performance Apply to this job Location pin icon Menlo Park, CA Apply to this job The PyTorch Compiler team is dedicated to making PyTorch run faster and more resource-efficient without sacrificing its flexibility and ease of use. The team is the driving force behind PT2, a step function...


  • Menlo Park, United States META Full time

    Software Engineer, Systems ML - PyTorch Compiler / ML Framework / Performance Apply to this job Location pin icon Menlo Park, CA Apply to this job The PyTorch Compiler team is dedicated to making PyTorch run faster and more resource-efficient without sacrificing its flexibility and ease of use. The team is the driving force behind PT2, a step function...


  • Menlo Park, United States META Full time

    Summary: The PyTorch Compiler team is dedicated to making PyTorch run faster and more resource-efficient without sacrificing its flexibility and ease of use. The team is the driving force behind PT2, a step function change in PyTorch’s history that brought compiler technologies to the core of PyTorch. PT2 technologies have gained industry-wide recognition...


  • Menlo Park, CA, United States Instagram Full time

    Applied Research Scientist, Instagram Core ML (PhD) Meta is seeking a Research Scientist to join our Instagram Core ML team. This team is an applied research organization focused on improving the quality of IG recommendation models through AI. Advances in AI are key to our mission, spanning some of the most pressing research challenges of our generation...


  • San Francisco, CA, United States ats.rippling.com- ATS Full time

    Bronco is an applied AI lab helping chipmakers keep Moore’s law going. Our mission is to build AI silicon engineers that can automate chip design and verification from initial spec to final tape-out. We are starting with the first AI Design Verification Engineer to help close the verification gap, which is where the bottleneck currently is and where we...


  • Cupertino, CA, United States Apple Inc. Full time

    AIML - ML Research Engineer or Scientist Our team is looking for a roboticist to work with a collaborative team of researchers within Apple’s AIML organization. This team is working to leverage and build upon groundbreaking ML robotics research, thereby enabling development of generalizable and reliable robot systems. Description Duties associated with...


  • Menlo Park, United States Character Full time

    About the Role As a Software Engineer you will work cross functionally with product, data science, data platform and ML researchers to identify, design and implement the applied ML infra, ranging from the data pipeline to model training and serving infrastructure, on one of the fastest growing consumer applications on the market today. You will lead...


  • Menlo Park, United States META Full time

    Summary: In this role, you will be a member of the MTIA (Meta Training & Inference Accelerator) Software team and part of the bigger industry-leading PyTorch AI framework organization. MTIA Software Team has been developing a comprehensive AI Compiler strategy that delivers a highly flexible platform to train & serve new DL/ML model architectures, combined...

  • Research Engineer

    3 days ago


    San Francisco, CA, United States RI Research Instruments GmbH Full time

    You want to build large scale ML systems from the ground up. You care about making safe, steerable, trustworthy systems. As a Research Engineer, you'll touch all parts of our code and infrastructure, whether that's making the cluster more reliable for our big jobs, improving throughput and efficiency, running and designing scientific experiments, or...


  • Mountain View, CA, United States Thetalkingmachines Full time

    Send me a weekly email with the latest jobs. Research Engineer, Applied ML (Mountain View) Job Type: Full Time Location: Mountain View, CA The DeepMind Applied group collaborates with teams at Google and elsewhere to apply our cutting-edge research to products and infrastructure used by many millions of people across the world. We’re very proud to already...


  • Menlo Park, CA, United States Lamini Full time

    Lamini enables every enterprise to safely, quickly, and cost-effectively build their own Expert AI. Our customers own their own models, trained on their data. Lamini optimizes for Expert AI workloads with minimal hallucination, enterprise-grade security, and enterprise flexibility, running on any infrastructure. Our team is made up of highly committed...

  • Core ML Researcher

    3 days ago


    San Francisco, CA, United States Vast.ai Full time

    About Us: At Vast.ai , we value creativity, energy, drive and teamwork. We are looking for talented people who share these values to join as we grow our team. Our vision is to widely distribute AI computing to reshape our future for the good of humanity. Come join us. If witnessing the birth of AGI excites you, we can’t wait to hear from you! Vast.ai is...


  • Menlo Park, United States Lamini Full time

    Lamini enables every enterprise to safely, quickly, and cost-effectively build their own Expert AI. Our customers own their own models, trained on their data. Lamini optimizes for Expert AI workloads with minimal hallucination, enterprise-grade security, and enterprise flexibility, running on any infrastructure. Our team is made up of highly committed...

  • Research Scientist

    1 week ago


    Menlo Park, California, United States Instagram Full time

    Company OverviewAbout InstagramWe're a community of people who are passionate about helping others share their moments and connect with what matters most to them.Salary$165,000 - $250,000 per yearJob DescriptionDevelop Scalable Deep Learning ModelsWe're looking for a Research Scientist to join our Core ML team and help us improve our recommendation models...


  • Olmos Park, United States Southwest Research Institute Full time

    Who We Are:We develop and apply advanced computational methods to investigate the performance of a variety of engineering materials that are used in a wide range of applications, especially estimating the life and reliability of high-consequence engineering systems.Objectives of this Role:Perform fundamental and applied research through the development of...


  • San Jose, CA, United States Cisco Systems, Inc. Full time

    Tech Lead – AI/ML Infrastructure Engineer Location: San Jose, California, US Compensation Range: 173100 USD - 241700 USD Job Type: Professional Job Id: 1426055 Application window has been extended and expected to close 11/15/2024. Who We Are: The Cisco Distributed System Engineering (DSE) group is at the forefront of developing products that power the...


  • Cupertino, CA, United States Apple Inc. Full time

    AIML - ML Research Engineer, Forecasting Foundation Models We are a group of innovative researchers and engineers dedicated to fundamentally increase Apple’s business success across Finance, Sales, and Operations with Deep Learning technologies and innovations. We build foundation models to forecast critical business and financial metrics that drive...

  • AIML Resident

    2 days ago


    Cupertino, CA, United States Apple Full time

    Summary Imagine what you could do here! At Apple, great ideas have a way of becoming great products, services, and customer experiences very quickly. Combining groundbreaking machine learning research with next-generation hardware, our teams take user experiences to the next level! Description Apple’s AIML residency is a year-long program inviting...