Director of Data Engineering, Data and Systems

1 month ago


San Francisco, California, United States Pendulum Full time

Pendulum is revolutionizing the way we understand and interact with the human microbiome. As a Director of Data Engineering, you will play a pivotal role in building and maintaining our data platform and infrastructure, driving data-driven decision-making and AI/ML initiatives.

Key Responsibilities:

  • Lead and mentor a high-performing data engineering team, fostering collaboration, innovation, and continuous learning.
  • Define and drive the data infrastructure strategy, ensuring alignment with business goals, scalability, and adaptability.
  • Own the management and optimization of data warehouses and lakehouse, ensuring data reliability, performance, and accessibility.
  • Collaborate with cross-functional leaders to build data foundations that support machine learning, analytics, and business intelligence efforts.
  • Ensure best practices in data governance, security, and compliance, managing data quality, lineage, and security frameworks.

Requirements:

  • MSc/PhD in Computer Science or a related field.
  • 10+ years of experience building and managing data infrastructure, with expertise in data pipelines, cloud platforms, and big data technologies.
  • Expert in Python, SQL, and big data tools (Kafka, Spark, Hive/Iceberg), with significant experience in cloud platforms (AWS, GCP, Azure).
  • Proven experience with containerization (Docker, Kubernetes) and orchestration tools like Airflow to scale and optimize workflows.
  • Strong understanding of data quality management, governance, and regulatory compliance, with experience in GDPR and CCPA requirements.

What We Offer:

  • $175,000-$225,000 salary range.
  • Medical, Dental, and Vision benefits.
  • Commuter Benefits.
  • Life & STD Insurance.
  • Company match on 401(k).
  • Flexible Time Off (FTO).
  • Equity.


  • San Francisco, California, United States Pendulum Full time

    Pendulum is a revolutionary company leading a global movement to improve physical and mental health by understanding, restoring, and enhancing the human microbiome.The company has made significant breakthroughs in the field, creating proprietary probiotic pipelines and a unique discovery platform to identify key bacterial strains and prebiotics.As a Director...


  • San Francisco, California, United States Amazon Data Services, Inc. Full time

    About the RoleWe are seeking a highly skilled Data Center Operations Specialist to join our team at Amazon Data Services, Inc. As a key member of our Data Center Engineering Operations team, you will play a critical role in maintaining and operating our critical infrastructure systems to ensure 99.999% uptime of our cloud computing portfolio.Key...


  • San Francisco, California, United States GoFundMe Full time

    Lead Data Engineering and Analytics at GoFundMeWe're seeking an experienced Senior Director of Data Engineering to lead our data engineering and analytics team at GoFundMe. As a key member of our organization, you'll be responsible for driving data-driven decision-making and innovation across the company.Key Responsibilities:Collaborate with stakeholders to...

  • Senior Data Engineer

    4 weeks ago


    San Francisco, California, United States Plato Systems Full time

    Transforming Manufacturing Operations with DataAt Plato Systems, we're revolutionizing the way manufacturers approach operations by harnessing the power of spatial activity data and AI. As a Solution Engineer, you'll play a critical role in helping our clients unlock new levels of capacity, productivity, and safety.Our intelligent analytics platform, fueled...


  • San Diego, California, United States Live Nation Entertainment Full time

    Job Title: Director of Data EngineeringLive Nation Entertainment is seeking a highly experienced and skilled Director of Data Engineering to lead our team in developing and optimizing data pipelines, solving complex problems, and driving technical innovation.About the RoleThe Director of Data Engineering will be responsible for providing hands-on leadership...


  • San Francisco, California, United States Unreal Gigs Full time

    Unlock the Power of Data-Driven InnovationWelcome to Unreal Gigs, where we harness the transformative potential of data to drive meaningful change across industries. As a Lead Data Systems Engineer, you will play a pivotal role in shaping our cutting-edge data systems infrastructure.Company OverviewWe are dedicated to building and delivering high-quality,...


  • San Francisco, California, United States Pendulum Full time

    About PendulumPendulum is a pioneering company that is revolutionizing the field of microbiome research and development. Our mission is to improve physical and mental health by understanding, restoring, and enhancing the human microbiome.Our team is dedicated to developing innovative solutions that address the complex relationships between the microbiome and...

  • Senior Data Engineer

    4 weeks ago


    San Francisco, California, United States Plato Systems Full time

    Plato Systems is a pioneering company in the field of spatial AI, aiming to boost capacity, productivity, and safety across physical operations, starting with manufacturing.We are seeking a Senior Data Engineer with over 7 years of relevant experience to join our growing team.The ideal candidate will have a deep understanding of ETL pipeline design,...

  • Senior Data Engineer

    4 weeks ago


    San Francisco, California, United States Plato Systems Full time

    At Plato Systems, we are pioneering the use of spatial AI to boost capacity, productivity, and safety across physical operations, starting with manufacturing.Our Operations Digital Twin and AI copilot, fueled by proprietary hardware, uses machine perception and sensor fusion on the edge to digitize patterns of activity.Advanced manufacturing companies use...

  • Senior Data Engineer

    4 weeks ago


    San Francisco, California, United States Crunchyroll Full time

    About the RoleCrunchyroll is growing and changing, presenting unique challenges and opportunities to support millions of anime fans around the world. The Data Engineering team provides seamless help to our internal stakeholders, ensuring an exceptional experience for all Crunchyroll fans.As a Senior Data Engineer, you will play a pivotal role in designing,...

  • Data Systems Manager

    4 weeks ago


    San Francisco, California, United States University of California Full time

    Job SummaryUCSF Health is seeking a highly skilled Data Systems Manager to join our Enterprise Information & Analytics - Health Quality team. As a key member of our team, you will be responsible for developing and implementing a data strategy that aligns with our business and technology goals for quality improvement and reporting.Key ResponsibilitiesDefine...


  • San Francisco, California, United States Amazon Data Services, Inc. Full time

    About the RoleAmazon Web Services (AWS) is seeking a highly skilled Data Center Operations Manager to join our expanding Infrastructure Operations team. As a key member of our team, you will be responsible for ensuring the overall availability and reliability of our data center operations, meeting or exceeding defined service levels.The ideal candidate will...


  • San Francisco, California, United States California Water Data Consortium Full time

    The California Water Data Consortium is seeking a visionary leader to serve as its next Chief Executive Officer.The ideal candidate will have a proven track record of strategic planning, programmatic execution, and financial management.They will oversee and help expand a talented team of professionals, which will manage regular meetings of the Board of...


  • San Francisco, California, United States Checkr Full time

    About CheckrCheckr is a leading provider of background check services, empowering businesses to make informed hiring decisions. As the Senior Director of Data, you will play a critical role in shaping the company's data strategy and driving business growth.Job SummaryWe are seeking an experienced Senior Director of Data to lead our data engineering, product...


  • San Francisco, California, United States California Water Data Consortium Full time

    The California Water Data Consortium is seeking a visionary leader to serve as its next Chief Executive Officer.The ideal candidate will have a proven track record of strategic planning, program development, and team management.They will oversee and help expand a talented small team of professionals, which will manage regular meetings of the Board of...


  • San Francisco, California, United States California Water Data Consortium Full time

    The California Water Data Consortium is seeking a visionary leader to serve as its next Chief Executive Officer.The ideal candidate will have a proven track record of strategic planning, programmatic execution, and financial management.They will oversee and help expand a talented team of professionals, which will manage regular meetings of the Board of...

  • Data Engineer

    1 month ago


    San Francisco, California, United States REQ Solutions Full time

    Job Title: Data EngineerWe are seeking a highly skilled Data Engineer to join our team at REQ Solutions. As a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data systems that support our business operations.Key Responsibilities:Design and implement data pipelines using cloud-based technologies such as AWS.Develop...


  • San Francisco, California, United States California Water Data Consortium Full time

    The California Water Data Consortium is seeking a visionary leader to serve as its next Chief Executive Officer.The ideal candidate will be an excellent communicator and proven leader who demonstrates a spirit of entrepreneurship and can provide thought leadership on California's highest impact water data opportunities.The successful candidate will have at...

  • Data Engineer

    4 weeks ago


    San Francisco, California, United States Bayone Full time

    Job Title: Data EngineerCompany Overview:At Bayone, we strive to create an environment where people are valued, and differences are celebrated. Our purpose is to empower our teams and inspire our customers to become the best versions of themselves.The Opportunity:Our technology team works fast and smart, developing the latest in mobile technologies, scalable...


  • San Francisco, California, United States Unreal Gigs Full time

    Unreal Gigs is seeking a highly skilled Data Systems Architect to join our team. As a key member of our data engineering team, you will be responsible for designing, building, and optimizing our data systems infrastructure.### Key Responsibilities:* Design and architect scalable and resilient data systems infrastructure to meet business requirements and...