Research Engineer, Collective Alignment

1 month ago


San Francisco, United States OpenAI Full time
About the team

OpenAI's mission is to ensure that artificial general intelligence benefits all of humanity. OpenAI's Collective Alignment Team works on technical approaches to ensure that AGI is shaped democratically. We plan to do so by implementing a system for collecting and encoding public input on model behavior into our systems. While OpenAI has existing efforts to solve technical alignment, we also need systems to help us determine the answer to the question "What do we align to?"

Regulatory bodies are building up capacity to manage such systems, but may not be up-to-speed on the latest technology. Our best bet is to find more ways to ensure that we are appropriately managing risk and harms that our technologies may pose, including by increasing our ability to make good decisions about system development and deployment.

One way to increase our ability to make good decisions is to have input from the broader public. We are encouraged by the phenomenal work done by the recipients of our Democratic Inputs to AI grant that we launched in May, which was aimed at exploring and building in the space of governing AI behavior democratically. We selected 10 teams (out of nearly 1000 applicants) who were each awarded $100,000 to build, test, and share learnings from projects that explore how we can use democratic methods to decide the rules that govern AI systems.

We're excited to combine our research with ideas and prototypes developed by these teams in the coming months. We will implement a system that can find representative participants to provide their perspectives on a given topic, enable discussion and deliberation, and aggregate inputs into policies that are used to change our model behavior. And we're looking for excellent, highly motivated Research Engineers to help us.

About the role

We are seeking Research Engineers to help design and implement experiments for collective alignment research. Responsibilities may include:
  • Writing performant and clean code for ML training.
  • Independently running and analyzing ML experiments to diagnose problems and understand which changes are real improvements.
  • Writing clean non-ML code, for example when building interfaces to let workers and participants interact with our models or pipelines for managing human data.
  • Collaborating closely with a small team to balance the need for flexibility and iteration speed in research with the need for stability and reliability in a complex long-lived project.
  • Understanding our high-level research roadmap to help plan and prioritize future experiments.
  • Implement experiments to measure the effectiveness of different preference learning techniques, including RLHF
  • Examine the impact of different aggregation methodologies on model behavior
  • Curate large datasets of prompts and investigate coverage of boundary cases
  • Exploring methods to understand and predict model behaviors, such as finding inputs causing anomalous circuits or catastrophic outputs
  • Designing novel approaches for using LLMs in democratic inputs to AI research
You might thrive in this role if you
  • Are excited about OpenAI's mission of building safe, universally beneficial AGI and are aligned with OpenAI's charter
  • Want to use your engineering skills to push the frontiers of what state-of-the-art language models can accomplish
  • Possess a strong curiosity about the sociotechnical challenges around aligning and understanding ML models, and are motivated to use your career to address this challenge
  • Enjoy fast-paced, collaborative, and cutting-edge research environments
  • Have experience implementing ML algorithms (e.g., PyTorch)
  • Can develop data visualization or data collection interfaces (e.g., JavaScript, Python)
  • Want to ensure that humanity can shape future powerful AI systems


About OpenAI

OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity. We push the boundaries of the capabilities of AI systems and seek to safely deploy them to the world through our products. AI is an extremely powerful tool that must be created with safety and human needs at its core, and to achieve our mission, we must encompass and value the many different perspectives, voices, and experiences that form the full spectrum of humanity.

We are an equal opportunity employer and do not discriminate on the basis of race, religion, national origin, gender, sexual orientation, age, veteran status, disability or any other legally protected status.

For US Based Candidates: Pursuant to the San Francisco Fair Chance Ordinance, we will consider qualified applicants with arrest and conviction records.

We are committed to providing reasonable accommodations to applicants with disabilities, and requests can be made via this link.

OpenAI Global Applicant Privacy Policy

At OpenAI, we believe artificial intelligence has the potential to help people solve immense global challenges, and we want the upside of AI to be widely shared. Join us in shaping the future of technology.

  • San Francisco, United States OpenAI Full time

    About the team OpenAI's mission is to ensure that artificial general intelligence benefits all of humanity. OpenAI's Collective Alignment Team works on technical approaches to ensure that AGI is shaped democratically. We plan to do so by implementing a system for collecting and encoding public input on model behavior into our systems. While OpenAI has...


  • San Francisco, United States Karkidi Full time

    You want to build and run elegant and thorough machine learning experiments to help us understand and steer the behavior of powerful AI systems. You care about making AI helpful, honest, and harmless, and are interested in the ways that this could be challenging in the context of human-level capabilities. You could describe yourself as both a scientist and...


  • San Francisco, CA, United States Anthropic Limited Full time

    About the role: You want to build and run elegant and thorough machine learning experiments to help us understand and steer the behavior of powerful AI systems. You care about making AI helpful, honest, and harmless, and are interested in the ways that this could be challenging in the context of human-level capabilities. You could describe yourself as both...


  • San Francisco, CA, United States Genai Works Full time

    The Safety Systems team is responsible for various safety work to ensure our best models can be safely deployed to the real world to benefit the society and is at the forefront of OpenAI's mission to build and deploy safe AGI, driving our commitment to AI safety and fostering a culture of trust and transparency. The Human-AI Interaction (HAI) team is...

  • Research Engineer

    24 hours ago


    San Francisco, CA, United States Anthropic Limited Full time

    About the role: You want to help construct and rapidly iterate on machine learning experiments to help us improve the behavior of powerful AI systems through finetuning. You care about making AI helpful, honest, and harmless, and are interested in shaping model behavior to be more aligned with human values and goals. You could describe yourself as both a...

  • Research Engineer

    2 days ago


    San Francisco, United States Genmo Full time

    Job Description Job Description Our mission Genmo makes it easy for anyone to create movies, as if it were magic. Using our web application, any user can create cinematic video using a simple text prompt. We imagine a world where high-quality cinematic video content is as plentiful as water. Our mission is to empower the next billion video creators to tell...

  • Research Engineer

    3 days ago


    San Francisco, United States Genmo Full time

    Job DescriptionJob DescriptionOur missionGenmo makes it easy for anyone to create movies, as if it were magic. Using our web application, any user can create cinematic video using a simple text prompt.We imagine a world where high-quality cinematic video content is as plentiful as water. Our mission is to empower the next billion video creators to tell their...


  • San Francisco, California, United States Wispr AI Full time

    Wispr is building a more natural way to interact with technology with neural interfaces. We have an elite team of engineers, product designers, and research scientists building magic.About Wispr: We've raised $25M from top-tier VCs like NEA and 8VC. Our angels and advisors include Chester Chipperfield (product lead for the first Apple Watch), Ben Jones (COO,...


  • San Francisco, CA, United States Anthropic Limited Full time

    About the role: You want to help construct and rapidly iterate on machine learning experiments to help us improve the behavior of powerful AI systems through finetuning. You care about making AI helpful, honest, and harmless, and are interested in shaping model behavior to be more aligned with human values and goals. You could describe yourself as both a...


  • San Francisco, United States Veterans Health Research Institute Full time

    Position Definition: The Staff Research Associate I is responsible for participating in the day-to-day operations of routine, minimally complex clinical research programs and/or studies conducted by Principal Investigators (PIs) at the San Francisco VA. Performs a variety of administrative and operational duties involved in the collection, compilation,...

  • Research Assistant I

    4 weeks ago


    San Francisco, United States Veterans Health Research Institute Full time

    Under the general supervision of the Principal Investigator, perform specialized and routine procedures and techniques in controlled experiments in the Brain Imaging and EEG Laboratory. Position will aid in an ongoing research projects investigating brain structure and function in psychosis and depression. Essential Functions: * Collect and analyze EEG...

  • RESEARCH ENGINEER

    1 month ago


    San Antonio, United States Southwest Research Institute Full time

    Who We Are:The Mechanics & Materials team in the Materials Engineering department performs advanced materials research to solve our clients’ most challenging technical problems related to aerospace and manufacturing applications. This position will support senior staff members and ongoing programs t...


  • San Francisco, United States Veterans Health Research Institute Full time

    Position Definition: The incumbent will serve as the Study Coordinator for an investigator-initiated research study which uses ambulatory technology to understand sleep disturbance in PTSD. The SRA II will collaborate on other studies, including the STAR study, as needed. This research is being conducted by researchers from the San Francisco VA Medical...


  • San Francisco, CA, United States Genai Works Full time

    About the Team The Safety Systems team is responsible for various safety work to ensure our best models can be safely deployed to the real world to benefit the society and is at the forefront of OpenAI's mission to build and deploy safe AGI, driving our commitment to AI safety and fostering a culture of trust and transparency. The Human-AI...

  • RESEARCH ENGINEER

    2 days ago


    San Antonio, United States Southwest Research Institute Full time

    Who We Are: The Liquid Propulsion Team performs R&D to solve challenges and advance technologies in machinery and power systems with a focus on liquid propulsion systems. We support actual launch, landing, lunar, and orbital vehicles and ground equipment, ranging from high-g to 0-g and supercritical to cryogenic conditions. Our technologies are advancing...


  • San Francisco, United States CytoVale Full time

    Company Description Headquartered in San Francisco, CA, Cytovale is a dynamic, late-stage startup focused on developing a faster, more insightful way to diagnose fast-moving and immune-mediated diseases. Cytovale's IntelliSep test is pioneering a new way of quickly and accurately analyzing white blood cells to stratify a patient's risk of sepsis. The test...


  • San Francisco, United States Primeintellect Full time

    At Prime Intellect, we are on a mission to accelerate open and decentralized AI progress by enabling anyone to contribute compute, code or capital to train powerful, open models. Our ultimate goal? Openly accessible AGI that benefits everyone. But we can't do it alone and we want to do this together with you. We are building the infrastructure for...


  • San Francisco, United States SF Research Institute Inc. Full time

    Research experience is a plus NOT a remote position. Must be able work on site. *Position Overview:* The administrative staff is the go-between for management, staff members, study participants, vendors, consultants, partnerships and other outside organizations. They play a large role in supporting the daily operations. Besides support, the administrative...


  • San Francisco, United States CytoVale Full time

    Company Description Headquartered in San Francisco, CA, Cytovale is a dynamic, late-stage startup focused on developing a faster, more insightful way to diagnose fast-moving and immune-mediated diseases. Cytovale's IntelliSep test is pioneering a new way of quickly and accurately analyzing white blood cells to stratify a patient's risk of sepsis. The test...

  • Research Engineer

    3 days ago


    San Francisco, United States Autodesk Full time

    Job Requisition ID # 23WD74103 Position OverviewWe are looking for a Research Engineer, Mechatronics & Software Research to develop and implement research prototypes for the Autodesk Research organization.Our dedicated team, Autodesk Research, seeks to explore the "new possible" what's possible in the future which means we're looking as far as 10 years out,...