Highly Scalable Distributed Systems Architect

2 days ago


San Francisco, California, United States OpenAI Full time
About the Role

We are seeking a highly skilled Software Engineer to join our Data Acquisition team at OpenAI. The ideal candidate will have 4+ years of industry experience in software development, with a strong background in large stateful distributed systems and data processing.

The successful candidate will have expertise in Kubernetes, Infrastructure-as-Code concepts, and proficiency in working with large web crawlers. Additionally, they should be able to handle multiple tasks and adapt to changing priorities.

This is an excellent opportunity for someone who is passionate about AI research and deployment, and wants to contribute to the development of safe and beneficial AI systems.

Job Requirements

Key Responsibilities:

  • Own and lead engineering projects in the area of data acquisition, including web crawling, data ingestion, and search.
  • Collaborate with other sub-teams to ensure smooth data flow and system operability.
  • Work closely with the legal team to handle compliance or data privacy-related matters.
  • Develop and deploy highly scalable distributed systems capable of handling petabytes of data.
  • Architect and implement algorithms for data indexing and search capabilities.
  • Build and maintain backend services for data storage, including work with key-value databases and synchronization.
  • Deploy solutions in a Kubernetes Infrastructure-as-Code environment and perform routine system checks.
  • Conduct and analyze experiments on data to provide insights into system performance.

Qualifications:

  • BS/MS/PhD in Computer Science or a related field.
  • 4+ years of industry experience in software development.
  • Experience with large web crawlers a plus.
  • Strong expertise in large stateful distributed systems and data processing.
  • Proficiency in Kubernetes, and Infrastructure-as-Code concepts.
  • Willingness and enthusiasm for trying new approaches and technologies.
  • Ability to handle multiple tasks and adapt to changing priorities.
  • Strong communication skills, both written and verbal.

Benefits:

  • A competitive salary range: $220,000 - $320,000 per year.
  • A comprehensive benefits package, including health insurance, retirement savings plan, and paid time off.
  • An opportunity to work on cutting-edge AI research and deployment projects.
  • A collaborative and dynamic work environment.


  • San Francisco, California, United States Intelliswift Software Full time

    We are looking for a talented Distributed System Architect to design and implement our Kafka infrastructure at Intelliswift Software. The ideal candidate will have extensive experience with Confluent Kafka and be able to architect and implement scalable, high-performance distributed systems.Responsibilities include designing and implementing scalable Kafka...


  • San Francisco, California, United States Caldera Full time

    We're seeking a highly skilled Scalable Blockchain Infrastructure Architect to join our team at Caldera, located in San Francisco. As an expert in distributed systems and cloud technologies, you'll be responsible for architecting and developing backend systems to ensure the smooth operation of Caldera chains.About the RoleArchitect and develop...


  • San Francisco, California, United States Succinct Full time

    Innovate with Succinct, a pioneer in blockchain scaling, interoperability, and privacy solutions. As a Senior Software Engineer, you'll play a crucial role in developing our distributed proving cluster for SP1 and prover network in our San Francisco office.About the RoleThis position requires expertise in architecting and maintaining a highly available...


  • San Francisco, California, United States Anyscale Full time

    We're seeking a skilled Distributed Systems Architect to join our team at Anyscale. Our mission is to democratize distributed computing and make it accessible to software developers of all skill levels.As a Distributed Systems Architect, you'll play a key role in building the best place to run Ray, a popular open-source project that's creating an ecosystem...


  • San Francisco, California, United States Arbitrum Full time

    Senior Distributed Systems EngineerWe're looking for a seasoned professional to design and develop scalable, reliable, and high-performance distributed systems.This is an ideal opportunity for an engineer who is passionate about tackling complex problems in blockchain scalability, or looking to break into the field of blockchain engineering. If you're...


  • San Jose, California, United States Mindlance Full time

    Distributed Systems Architect PositionWe are seeking an experienced Distributed Systems Architect to join our team. As a Distributed Systems Architect, you will be responsible for designing and building large-scale distributed systems. You will work closely with our engineering team to ensure that our systems are secure, scalable, and highly available.Key...


  • San Francisco, California, United States Discord Full time

    As a Highly Available System Architect at Discord, you'll design and implement scalable and fault-tolerant systems that support our real-time features and services. With a strong focus on observability and monitoring, you'll work closely with our engineering teams to ensure the high availability of our systems and identify areas for...


  • San Diego, California, United States Apple Full time

    Job Description:Distributed Systems Architect - Cloud EngineeringDesign novel distributed architectures to accelerate software build, test, and deployment.Combine problem domain expertise with established techniques to achieve high performance, reliability, and long-term maintainability.Analyze requirements, existing solutions, and systems to inform...


  • San Jose, California, United States Adobe Full time

    Role OverviewThis Scalable System Architect position at Adobe offers an exciting opportunity for engineers to tackle complex challenges of scale and build core services used by all Adobe Digital Experience solutions.Key ResponsibilitiesDesigning and implementing high-performance distributed systems to support scalable services.Collaborating with...


  • San Francisco, California, United States Nextdoor Full time

    Job DescriptionWelcome to the Core Services team at Nextdoor, where we operate critical high throughput services that power communities worldwide. As a Distributed Systems Architect, you will work in a large-scale distributed system environment, identifying high leverage opportunities to increase performance, scalability, and resilience.The primary focus of...


  • San Francisco, California, United States Tools for Humanity Full time

    **Job Description**Design and build highly scalable systems to support the World App.Be responsible for key components within the App, including designing, building, maintaining, and scaling them, while delivering on business metrics.Cross-coordinate with mobile, design, and web teammates to deliver backend systems for high scalability.Collaborate with...

  • Cloud Architect

    3 days ago


    San Jose, California, United States Tik Tok Full time

    Job OverviewWe are seeking a highly skilled Cloud Architect - Distributed Systems to join our team at TikTok. As a key member of our infrastructure team, you will be responsible for designing and implementing scalable microservices architectures for our complex online systems.


  • San Jose, California, United States WeRide Full time

    As a Senior Software Engineer at WeRide, you will have the opportunity to work on challenging projects that require innovation and technical expertise. Your primary responsibilities will include:Building and maintaining highly scalable and reliable PaaS and IaaS platforms for our autonomous driving solutions.Developing high-performance and scalable...

  • Scalable AI Architect

    3 weeks ago


    San Francisco, California, United States Anyscale Full time

    About Anyscale:Anyscale is a pioneering technology company that empowers software developers to harness the full potential of distributed computing. By commercializing Ray, an open-source project, we're creating an ecosystem of libraries for scalable machine learning. Companies like OpenAI, Uber, Spotify, Instacart, and Cruise trust Ray as a critical...


  • San Francisco, California, United States salesforce Full time

    About the RoleWe are seeking a highly experienced Senior Software Architect to lead our distributed systems team at Salesforce. As a key member of our Cloud Economics and Capacity Management (CECM) team, you will be responsible for designing, implementing, and delivering highly scalable products that enable strategic decision-making.The ideal candidate will...


  • San Francisco, California, United States Anyscale Full time

    About the CompanyAnyscale is a cutting-edge technology company dedicated to democratizing distributed computing and making it accessible to developers of all skill levels. We're a leader in the field, commercializing Ray, a popular open-source project that's creating an ecosystem of libraries for scalable machine learning.With Anyscale, you'll have the...


  • San Francisco, California, United States Nextdoor Full time

    Your Future Role:We are seeking a Distributed Datastore Architect to join our Core Services team. As a key member of this team, you will be responsible for designing and implementing a distributed, globally replicated graph store with high availability tiered caching. This datastore will provide predictable performance and enable product engineers to move...


  • San Francisco, California, United States Tools for Humanity Full time

    About the CompanyTools for Humanity is a pioneering organization that has created a network of real humans built on privacy-preserving proof-of-human technology. Our goal is to empower everyone through a globally inclusive financial network that enables the free flow of digital assets.About the OpportunityWe are seeking an experienced Senior Software...


  • San Francisco, California, United States MongoDB Full time

    MongoDB's Atlas Search Query team is looking for a highly skilled Distributed System Architect to join our ranks. As a key member of our team, you'll design and implement a cloud-based search service that allows users to execute complex search queries using the MongoDB Query Language.You'll work closely with our team to develop new features, optimize...


  • San Francisco, California, United States Replo (YC S21) Full time

    About the RoleAt Replo, we're seeking an exceptional Senior Backend Engineer to join our team and focus on designing and implementing backend systems that power our most popular products. With a strong background in distributed systems and service-oriented architecture, you'll be responsible for architecting and building scalable systems that meet the...