Software Engineer, Analytics Platform

3 weeks ago


San Francisco, United States OpenAI Full time

About the Team

The Research Platform Analytics team designs, builds, and operates the critical foundational data and analytics infrastructure that enables research at OpenAI.

Our goal is one, and one only: accelerate the progress of research towards AGI. We do this by owning critical components of the research training stack, starting from key data processing pipelines and complex libraries that directly feed into our distributed training, followed by a variety of observability and analytics systems aimed at providing quality signals about our research, as well as an important series of components to handle data lifecycle at scale.

About the Role

As we scale up with more researchers and engineers joining OpenAI, we seek a pragmatic and passionate engineer with a strong focus on the experience for both engineers and scientists that work in our large data sets. Some examples of the work we do include making sure we have robust data processing pipelines that carry observability metrics and other important signals, highly effective observability systems for our researchers to reason about the quality of our research, and various data lifecycle management projects centered around efficiency, security and scale.

You will find yourself at home if you are comfortable with work such as scaling Kubernetes services, debugging Kafka consumer lag, diagnosing various distributed systems failures, and developing a new end-to-end data processing pipeline, all the way from raw data capture to bespoke analytics leveraging a data warehouse or a stream processing framework. If you have previous experience with dealing with data processing and transformations during pre-training, this is also a role for you.

This role is based in San Francisco, CA or open to being remote within the US. We use a hybrid work model of 3 days in the office per week and offer relocation assistance to new employees.

In this role, you will:

Participate in architecture and engineering decisions, bringing your strong experience and knowledge to bear.

Ensure the security, integrity, and compliance of data according to industry and company standards.

Ensure our analytics and data platforms can scale reliably to the next several orders of magnitude

Accelerate company productivity by empowering your fellow engineers, researchers, and teammates with excellent data tooling and systems, providing a best in case experience

Bring new features and capabilities to the world by partnering with product engineers, trust & safety and other teams to build the technical foundations

Like all other teams, we are responsible for the reliability of the systems we build. This includes an on-call rotation to respond to critical incidents as needed 

You might thrive in this role if you have:

Experience in building stream and batch data processing pipelines, using technologies such as, or equivalent to, Kafka, Spark, Flink.

Proficient with modern infrastructure management tools, such as Kubernetes and Terraform.

Have a passion for observability systems (bonus if for ML training). You are excited by the idea of building bespoke analytics systems that provide answers to key ML research questions.

Have worked in a ML training organization and have experience with the problem of data transformation during pre-training.

Are a proficient software engineer, ideally in Python, and are used to working with large monorepo codebases.

Have worked on data lifecycle management systems for large organizations, and dealt with the problems of access control, provenance, auditing, data movement at scale, metadata management, etc.

Are a self-starter and are comfortable operating in a fast-paced environment.



  • San Francisco, CA, United States OpenAI Full time

    About The Team The Research Platform Analytics team designs, builds, and operates the critical foundational data and analytics infrastructure that enables research at OpenAI. Our goal is one, and one only: accelerate the progress of research towards AGI. We do this by owning critical components of the research training stack, starting from key data...


  • San Francisco, CA, United States Advent Software, Inc. Full time

    Associate Software Engineer page is loaded Associate Software Engineer Apply locations San Francisco, CA time type Full time posted on Posted 2 Days Ago job requisition id R16507 SS&C is a global provider of investment and financial services and software for the financial services and healthcare industries. S. company based on revenue, SS&C is...

  • Software Engineer

    6 days ago


    San Francisco, United States Factory Full time

    Factory is on a mission to bring autonomy to software engineering. To do that, we are building AI systems called Droids that accelerate the engineering output of enterprise software organizations. Droids automate tedious but essential processes in software development like code review, documentation, test creation, as well as end-to-end coding tasks. Some...

  • Software Engineer

    2 weeks ago


    San Ramon, United States CXApp US, Inc. Full time

    Job DescriptionJob DescriptionSalary: Who we are At CXApp, we are the innovators of Indoor Intelligence, delivering actionable insights for people, places and things. Our flagship product the “CXApp” is a workplace experience platform for the enterprise. Our technologies and solutions help enterprise customers deliver a comprehensive business journey in...

  • Software Engineer

    4 days ago


    San Francisco, CA, United States Factory Full time

    Factory is on a mission to bring autonomy to software engineering. To do that, we are building AI systems called Droids that accelerate the engineering output of enterprise software organizations. Droids automate tedious but essential processes in software development like code review, documentation, test creation, as well as end-to-end coding tasks. Some...


  • San Francisco, United States Clipboard Health Full time

    Here are some frequently asked questions about our engineering team. you can also read about our work on our engineering careers page . About the Platform Team The Platform team at Clipboard Health builds the tools and infrastructure to allow us to ship high quality software incredibly fast. Our team tackles some of the hardest engineering challenges we have...


  • San Francisco, United States Yurts Full time

    Company Overview: At Yurts, we are on a mission to revolutionize the world of artificial intelligence and machine learning. We are passionate about pushing the boundaries of technology to build innovative platforms that empower enterprises to leverage Generative AI (LLMs) successfully. We are seeking an exceptional Senior Software Engineer for our Platform...


  • San Francisco, United States Yurts Full time

    Principal Software Engineer - Core Platform Company Overview: At Yurts, we are at the forefront of driving innovation in the field of artificial intelligence and Generative AI. Our mission is to create cutting-edge enterprise platforms that leverage the power of Language Model (LLMs) to transform document management, knowledge management, and information...


  • San Francisco, United States Abnormal Security Full time

    Job DescriptionJob DescriptionAbout the RoleAbnormal Security is looking for a Senior Software Engineer who is passionate about building and operating microservices at large scale. The Core Platform team owns foundational platform services including but is not limited to the Dynamic Configuration system which is responsible for delivery of runtime...


  • San Francisco, United States Nova Credit Full time

    At Nova Credit, our mission is to power a more fair and inclusive financial system for the world. We are on our way to accomplishing this mission by rewiring the financial industry with better credit infrastructure, analytics, and workflows, enabling more people to access credit opportunities. Our cross-border credit product, Credit Passport®, cash flow...


  • San Francisco, United States Agtonomy Full time

    About Us Agtonomy is a hybrid autonomy and tele-assist service platform that turns tractors and other equipment into autonomous machines. A robust sensor suite and custom software stack enable remote modes of operation with a higher margin of safety than conventional equipment. Agtonomy, through its OEM partners, will address both local agriculture skilled...


  • San Francisco, United States Chef Robotics Full time

    Chef Robotics is on a mission to accelerate the advent of intelligent machines in the physical world. As the rise of LLMs like ChatGPT has shown, AI has the potential to drive immense change. However, we believe AI’s biggest impact will be in the physical world, given industries like manufacturing, food, and transportation represent 90% of global GDP....


  • San Francisco, United States Clipboard Health Full time

    Job DescriptionJob DescriptionWhy Clipboard Health Exists:We exist to lift as many people up the socioeconomic ladder as possible. We dramatically improve lives, by letting healthcare professionals turn extra time and ambition into career growth and financial opportunity. We achieve this with our app-based marketplace that connects healthcare facilities and...

  • Software Engineer

    4 days ago


    San Francisco, CA, United States X Corp. Full time

    Are you prepared to join the X team and help build the ultimate real-time information-sharing app, revolutionizing how people connect? At X, we're on a mission to become a trusted global digital public square, committed to minimal censorship within legal boundaries. Our goal is to empower every user to freely create and share ideas, fostering open public...


  • San Francisco, California, United States Chef Robotics Full time

    Chef Robotics is on a mission to accelerate the advent of intelligent machines in the physical world. As the rise of LLMs like ChatGPT has shown, AI has the potential to drive immense change. However, we believe AI's biggest impact will be in the physical world, given industries like manufacturing, food, and transportation represent 90% of global GDP. We're...


  • San Francisco, United States Agtonomy Full time

    About Us Agtonomy is a hybrid autonomy and tele-assist service platform that turns tractors and other equipment into autonomous machines. A robust sensor suite and custom software stack enable remote modes of operation with a higher margin of safety than conventional equipment. Agtonomy, through its OEM partners, will address both local agriculture skilled...

  • Software Engineer

    1 month ago


    San Francisco, California, United States Platform Venture Studio Full time

    PlatformOS is our proprietary software platform for running Platform Venture Studio. Think of it as "Github for building startups". It captures all of our best practice, allows us to build in the open, and engages the collective intelligence of our network.PlatformOS comprises various responsive web applications, a browser extension, API integrations, and...

  • Software Engineer

    1 month ago


    San Francisco, United States Platform Venture Studio Full time

    PlatformOS is our proprietary software platform for running Platform Venture Studio. Think of it as "Github for building startups". It captures all of our best practice, allows us to build in the open, and engages the collective intelligence of our network. PlatformOS comprises various responsive web applications, a browser extension, API integrations, and...

  • Software Engineer

    4 days ago


    San Francisco, CA, United States Trov Full time

    Team The Platform team sits at the epicenter of Pave Engineering. As a member of the Developer Platform team, you will be tasked with designing and implementing scalable, secure cloud infrastructure and creating the development tooling and processes that enable Pave engineers to ship enterprise quality software with high velocity. You will play a technical...


  • San Mateo, United States ICONMA Full time

    Platform Software Engineer Location: Foster City, CA/Hybrid Duration: 1 year with possible contract to hire/ possible extension Description: In this role, you will be responsible for designing and building an effective and scalable CI/CD solution that addresses complex problems and impacts the end-to-end development and release lifecycle. You will be...