Data Engineering Leader for Microbiome Research

7 days ago

San Francisco, California, United States Pendulum Full time
About Pendulum
Pendulum is pioneering a revolution in human health by understanding, restoring, and enhancing the microbiome. Our mission is to improve physical and mental well-being through groundbreaking research and innovative technologies.
Studies have shown that our microbiome is linked to various aspects of health, including metabolism, diabetes, longevity, weight loss, immune systems, cancer prevention, feelings of well-being, and more. We're just beginning to understand the impact of our microbiome on our lives. A diverse and balanced microbiome developed during early life plays a critical role in shaping long-term health. However, factors like diet, lifestyle, antibiotics, and aging can decrease its effectiveness over time.
Pendulum recognized the enormous potential to transform lives by addressing microbiome imbalances. To achieve this, we've created proprietary probiotic pipelines and a unique discovery platform to identify key bacterial strains and prebiotics. Our manufacturing technology produces bacteria in an anaerobic environment at scale, enabling the development of medical-grade probiotics. These offerings have transformed the consumer probiotics market into a new category of therapeutic products, combining pharmaceutical efficacy with natural safety and accessibility.
Pendulum's explosive revenue growth has earned us a spot on Forbes Magazine's exclusive 'Next Billion Dollar Startups' list. If you're passionate about improving global health and thrive in a collaborative, inspiring environment, please continue reading.
Position Summary:
We seek a Director of Data Engineering to lead the development, optimization, and scaling of our data platform and infrastructure. This role will be instrumental in building and maintaining robust data pipelines, managing cloud-based data environments, and driving data platform innovations to support AI/ML initiatives. As a key leader, you'll oversee scalable, real-time data architectures, ensuring data reliability, accessibility, and compliance with data governance standards. The ideal candidate will have a strong technical foundation, proven leadership in managing data engineering teams, and the ability to align data strategies with business goals. The role will be hands-on, with a small team, and plans to expand over the years.
Key Responsibilities:
  • Lead, mentor, and grow a high-performing data engineering team, fostering collaboration, innovation, and continuous learning to meet evolving business needs.
  • Define and drive the data infrastructure strategy, ensuring alignment with business goals, scalability, and adaptability to support data-driven decision-making and AI/ML initiatives.
  • Own the management and optimization of data warehouses and lakehouse (e.g., Snowflake), ensuring data is reliable, performant, and accessible for stakeholders.
  • Collaborate with cross-functional leaders (e.g., Data Science, Product, Analytics) to build data foundations that support machine learning, analytics, and business intelligence efforts.
  • Ensure best practices in data governance, security, and compliance (GDPR, CCPA), managing data quality, lineage, and security frameworks across all platforms.
  • Develop efficient data models and schema designs to support data integration and querying, ensuring scalability and performance across business functions.
  • Design and maintain scalable ETL pipelines using tools like dbt, Fivetran, and Airflow, with a focus on real-time processing and automation.
  • Leverage Docker, Kubernetes, and automation tools to streamline data processing workflows and ensure efficient resource utilization across the data platform.
  • Establish a robust operational framework, including monitoring, alerting, and incident management, to ensure the stability and performance of data platforms in production environments.
Requirements:
  • MSc/PhD in Computer Science or a related field.
  • 10+ years of experience building and managing data infrastructure, with expertise in data pipelines, cloud platforms, and big data technologies.
  • Expert in Python, SQL, and big data tools (Kafka, Spark, Hive/Iceberg), with significant experience in cloud platforms (AWS, GCP, Azure).
  • Proven experience with containerization (Docker, Kubernetes) and orchestration tools like Airflow to scale and optimize workflows.
  • Strong understanding of data quality management, governance, and regulatory compliance, with experience in GDPR and CCPA requirements.
  • Demonstrated ability to collaborate with cross-functional teams, including Data Science and ML teams, to deliver data solutions aligned with AI/ML and business objectives.
  • Leadership experience managing data engineering teams, driving strategic decisions, and adapting to the latest advancements in data technologies.
What We Offer:
  • $175,000-$225,000 salary range.
  • Medical, Dental, and Vision insurance.
  • Commuter Benefits.
  • Life & STD Insurance.
  • Company match on 401(k).
  • Flexible Time Off (FTO).
  • Equity.

  • Postdoctoral Researcher in Nutrition Science

    7 days ago

    San Francisco, California, United States University of California Full time

    OverviewThe University of California seeks a highly motivated Postdoctoral Researcher to join our interdisciplinary team in the Department of Nutrition. This two-year postdoctoral fellowship offers a unique opportunity to conduct clinical research in hospitalized patients with malnutrition, receive leadership training in adolescent health, and develop...

  • Microbiome Operations Specialist

    5 days ago

    San Francisco, California, United States University of California Full time

    Estimated Salary: $86,400 - $143,200About the University of CaliforniaThe University of California is a world-renowned institution dedicated to advancing health and biomedical research. With a strong commitment to excellence in patient care, graduate-level education, and innovative research, we strive to make a positive impact on global health.We are proud...

  • Senior Product Designer

    6 days ago

    San Francisco, California, United States Zing Data, Inc. Full time

    Drive Data-Driven Growth">We are seeking an experienced Senior Product Designer to lead the design of our data-driven products. As a key member of our team, you will play a crucial role in shaping the user experience and driving business growth through innovative product design.">About the Role">">Design and develop new features: Collaborate with...

  • Air Quality Engineering Leader

    6 days ago

    San Francisco, California, United States Yorke Engineering Full time

    Job OverviewYorke Engineering is seeking a highly experienced Air Quality Engineer/Scientist to lead our environmental compliance and permitting functions. This challenging role involves reviewing processes and equipment, analyzing technical and regulatory issues, and communicating with clients, government agencies, and the public.About the RoleThis...

  • Data Engineer

    6 days ago

    San Francisco, California, United States Scale AI, Inc. Full time

    Scale AI, Inc.We're seeking an experienced Data Engineer to lead our AI research and development efforts in Generative AI. As a Machine Learning Research Engineer, you will design next-generation data pipelines and supervision strategies in collaboration with our customers to accelerate progress in machine learning research.You will:Collaborate closely with...

  • Data Engineering Leader for Scalable Data Infrastructure

    1 week ago

    San Francisco, California, United States DoorDash USA Full time

    DoorDash is a leading food delivery and logistics company, and our Data Engineering team plays a crucial role in building database solutions to support various use cases. As a Staff Software Engineer, Data, you will be responsible for architecting and scaling our data reliability, infrastructure, automation, and tools to meet growing business needs.About the...

  • Data Engineering Leader

    1 week ago

    San Francisco, California, United States Early Warning® Full time

    Early Warning® has been a trusted name in payments for over thirty years, powering and protecting the U.S. financial system with innovative solutions like Zelle, PazeSM, and more. We partner with thousands of institutions to increase access to financial services and safeguard transactions for hundreds of millions of consumers and small businesses.As a...

  • Senior Data Insights Expert for Commercial Growth

    6 days ago

    San Francisco, California, United States Pendulum Full time

    About PendulumPendulum is pioneering a revolution to enhance human health by understanding, restoring, and improving the human microbiome. Our mission is to unlock the full potential of our microbiome, which plays a vital role in shaping our long-term health.During early life, we develop a diverse and balanced microbiome that sets the stage for our lifelong...

  • Data Engineering Innovation Leader

    6 days ago

    San Francisco, California, United States Unreal Gigs Full time

    Job OverviewWe are seeking a seasoned Data Engineering Innovation Leader to spearhead our data engineering efforts and drive the design, development, and maintenance of our data infrastructure and pipelines.The ideal candidate will have a strong background in data engineering technologies and leadership experience, with a proven track record of leading...

  • Data Engineer for Industrial Research and Development

    7 days ago

    San Francisco, California, United States Uncountable Inc Full time

    We're revolutionizing industrial research and development at Uncountable Inc, a technology startup that's pushing the boundaries of what's possible. As a Data Engineer, you'll play a critical role in helping us achieve our goal of accelerating the development of new chemicals and materials.What You'll DoWe're looking for a skilled Data Engineer to join our...

  • Data Scientist Manager II, Research Leader

    7 days ago

    San Francisco, California, United States Google Full time

    About the Role:At Google, we're committed to fostering a culture of innovation and collaboration. As a Data Scientist Manager II, Research, you'll have the opportunity to drive scientific rigor and statistical methods to the challenges of product creation, development, and improvement. You'll work closely with a multi-disciplinary team of engineers and...

  • Data Research Strategist

    7 days ago

    San Francisco, California, United States Databricks Full time

    At Databricks, our mission is to empower users to make data-driven decisions. We are seeking a Data Research Strategist to join our team.About the RoleWe are looking for an expert researcher who can develop business-level strategy and vision in close collaboration with other disciplines. The ideal candidate will have experience in user research, mixed...

  • Senior Data Science Engineer

    7 days ago

    San Francisco, California, United States Future House USA Full time

    About FutureHouseFutureHouse is a non-profit AI-for-science lab that uses AI to automate research in biology and other complex sciences. We are backed by Eric Schmidt, and our mission is to build AI systems that can scale scientific research and accelerate breakthrough discoveries.We are seeking talented AI Research Engineers with experience designing and...

  • Senior Engineering Leader for Distributed Data Systems

    1 week ago

    San Francisco, California, United States Databricks Full time

    Job Title: Senior Engineering Leader for Distributed Data SystemsJob Summary:At Databricks, we are passionate about empowering data teams to tackle the world's most complex challenges. We achieve this by building and operating the world's leading data and AI infrastructure platform, enabling our customers to derive deep insights that drive their business...

  • Data Engineer Leader

    7 days ago

    San Diego, California, United States Catapult Solutions Group Full time

    At Catapult Solutions Group, we're looking for a skilled Data Engineer to lead our data engineering efforts. This role focuses on designing and implementing scalable ETL pipelines, ensuring data quality, and supporting data analytics across the organization.The ideal candidate will collaborate with cross-functional teams to drive business decisions and...

  • Advanced Data Researcher

    4 days ago

    San Francisco, California, United States ZipRecruiter Full time

    Job DescriptionWe are seeking a highly skilled Advanced Data Researcher to join our team at ZipRecruiter. As a key member of our data science group, you will play a critical role in developing and implementing innovative data-driven solutions that drive business growth.The ideal candidate will have a strong background in machine learning and AI, with...

  • Data Analytics Engineer

    7 days ago

    San Francisco, California, United States Advanced Engineering Group LLC Full time

    Company Overview: Advanced Engineering Group LLC is a leading technology firm that seeks highly motivated and enthusiastic engineers to join our product development team. Our technology stack primarily includes Java/Javascript, and while experience with these languages is beneficial, it's not essential.Job Description: We're looking for an exceptional Data...

  • Data Analyst

    3 weeks ago

    San Francisco, California, United States Bitwise Investments Full time

    Crypto Research Expert OpportunityWe are seeking a highly skilled Data Analyst to join our Research team at Bitwise Investments. As a crypto research expert, you will play a crucial role in building and spearheading the team's data pipelines, infrastructure, and strategy.Job SummaryThe ideal candidate will have 5+ years of experience in research,...

  • Data Scientist for Pharmaceutical Research

    4 days ago

    San Francisco, California, United States Software Guidance & Assistance, Inc. (SGA, Inc.) Full time

    About Software Guidance & Assistance, Inc.Software Guidance & Assistance, Inc. (SGA, Inc.) is a women-owned business that provides technology and resource solutions to tackle big IT problems with a personal approach. Our mission is to stand out by solving complex challenges through a boutique service.About the JobWe are seeking a highly motivated Data...

  • Data Quality Engineer Leader

    1 week ago

    San Francisco, California, United States Nextdata Technologies Inc Full time

    We are seeking a seasoned Data Quality Engineer to lead the excellence of Nextdata OS across various deployments. The successful candidate will have a knack for enhancing CI/CD pipelines and a love for data and analytics tools.About the RoleThe Data Quality Engineer will oversee the quality of Nextdata OS in Kubernetes-based stateful systems with multiple...