Current jobs related to Lead Data Engineer for Data and Systems Development - San Francisco, California - Pendulum™

  • Lead Data Engineer, Data and Systems

    3 weeks ago

    San Francisco, California, United States Pendulum Full time

    About PendulumPendulum is a pioneering company that is revolutionizing the field of microbiome research and development. Our mission is to improve physical and mental health by understanding, restoring, and enhancing the human microbiome.Our team is dedicated to developing innovative solutions that address the complex relationships between the microbiome and...

  • Director of Data Engineering, Data and Systems

    2 weeks ago

    San Francisco, California, United States Pendulum Full time

    Pendulum is a revolutionary company leading a global movement to improve physical and mental health by understanding, restoring, and enhancing the human microbiome.The company has made significant breakthroughs in the field, creating proprietary probiotic pipelines and a unique discovery platform to identify key bacterial strains and prebiotics.As a Director...

  • Data Center Operations Specialist

    2 weeks ago

    San Francisco, California, United States Amazon Data Services, Inc. Full time

    About the RoleWe are seeking a highly skilled Data Center Operations Specialist to join our team at Amazon Data Services, Inc. As a key member of our Data Center Engineering Operations team, you will play a critical role in maintaining and operating our critical infrastructure systems to ensure 99.999% uptime of our cloud computing portfolio.Key...

  • Senior Data Engineer

    2 weeks ago

    San Francisco, California, United States Plato Systems Full time

    Transforming Manufacturing Operations with DataAt Plato Systems, we're revolutionizing the way manufacturers approach operations by harnessing the power of spatial activity data and AI. As a Solution Engineer, you'll play a critical role in helping our clients unlock new levels of capacity, productivity, and safety.Our intelligent analytics platform, fueled...

  • Senior Data Engineer

    2 weeks ago

    San Francisco, California, United States Plato Systems Full time

    Plato Systems is a pioneering company in the field of spatial AI, aiming to boost capacity, productivity, and safety across physical operations, starting with manufacturing.We are seeking a Senior Data Engineer with over 7 years of relevant experience to join our growing team.The ideal candidate will have a deep understanding of ETL pipeline design,...

  • Data Engineering Lead

    1 week ago

    San Francisco, California, United States Windfall Full time

    About the Role:Windfall is seeking a highly skilled Data Engineer to join our core data team. As a Sr. Data Engineer, you will be responsible for designing and building the pipelines for massive datasets, taking them from inception to exploration to production and customer use.Key Responsibilities:Construct data pipelines to ingest and merge billions of...

  • Lead Data Scientist, Data and Systems

    3 weeks ago

    San Francisco, California, United States Pendulum Full time

    Pendulum is a revolutionary company that is leading a global movement to improve physical and mental health by understanding, restoring, and enhancing the human microbiome.The company has developed proprietary probiotic pipelines and a unique discovery platform to identify key, novel bacterial strains and the prebiotics that feed them.Pendulum has also built...

  • Senior Data Engineer

    2 weeks ago

    San Francisco, California, United States Plato Systems Full time

    At Plato Systems, we are pioneering the use of spatial AI to boost capacity, productivity, and safety across physical operations, starting with manufacturing.Our Operations Digital Twin and AI copilot, fueled by proprietary hardware, uses machine perception and sensor fusion on the edge to digitize patterns of activity.Advanced manufacturing companies use...

  • Data Center Operations Manager

    2 weeks ago

    San Francisco, California, United States Amazon Data Services, Inc. Full time

    About the RoleAmazon Web Services (AWS) is seeking a highly skilled Data Center Operations Manager to join our expanding Infrastructure Operations team. As a key member of our team, you will be responsible for ensuring the overall availability and reliability of our data center operations, meeting or exceeding defined service levels.The ideal candidate will...

  • Data Center Operations Manager

    4 weeks ago

    San Francisco, California, United States Amazon Data Services, Inc. Full time

    About the RoleWe are seeking a highly skilled Data Center Manager to join our expanding Infrastructure Operations team at Amazon Web Services (AWS). As a Data Center Manager, you will play a critical role in ensuring the overall availability and reliability of our data center operations, meeting or exceeding defined service levels.Key ResponsibilitiesLead...

  • Senior Data Systems Engineer

    4 days ago

    San Francisco, California, United States Harnahm Full time

    Harnahm is seeking a Senior Data Systems Engineer to lead the development of scalable and efficient data processing workflows. The ideal candidate will have 5+ years of experience with a degree in a related field and a strong understanding of Python and SQL.Responsibilities:Design and maintain effective ELT pipelines.Collaborate with Data Science and...

  • Lead Data Architect

    2 weeks ago

    San Francisco, California, United States Unreal Gigs Full time

    Job OverviewWelcome to Unreal Gigs, where we're pushing the boundaries of data-driven innovation. As a key member of our team, you'll play a vital role in shaping the future of data engineering.Job SummaryWe're seeking an experienced Lead Data Engineer to lead our data engineering efforts and drive the design, development, and maintenance of our data...

  • Data Engineering Lead

    2 weeks ago

    San Jose, California, United States Intelliswift Full time

    Data Engineering LeadAt Intelliswift, we are seeking a skilled Data Engineering Lead to join our team. As a Data Engineering Lead, you will be responsible for translating business requirements into technical specifications using logical and critical thinking skills. You will profile data, cleanse them, and build data products that meet the business...

  • Data Engineer

    2 weeks ago

    San Francisco, California, United States Saxon Global Full time

    Data Engineer Role at Saxon GlobalSaxon Global is seeking a highly skilled Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, implementing, and operating next-generation data lakes containing complex Genentech data from various business areas. You will work in a global environment, conceptualizing and owning the...

  • Lead Data Architect

    3 weeks ago

    San Francisco, California, United States Unreal Gigs Full time

    Unreal Gigs is seeking a seasoned Lead Data Architect to lead our data architecture initiatives and drive the design, implementation, and optimization of our data architecture solutions. As a key member of our team, you will provide technical leadership, guidance, and mentorship to a team of data architects and engineers, fostering a culture of excellence...

  • Director of Data Engineering, Data and Systems

    3 weeks ago

    San Francisco, California, United States Pendulum Full time

    Pendulum is revolutionizing the way we understand and interact with the human microbiome. As a Director of Data Engineering, you will play a pivotal role in building and maintaining our data platform and infrastructure, driving data-driven decision-making and AI/ML initiatives.Key Responsibilities:Lead and mentor a high-performing data engineering team,...

  • Product Design Lead

    1 week ago

    San Francisco, California, United States Acryl Data Full time

    About the RoleWe are seeking a highly skilled Product Design Lead to join our team at Acryl Data. As a key member of our product team, you will be responsible for designing and developing cutting-edge products that empower businesses to develop and maintain a reliable and compliant data ecosystem.Key Responsibilities:Collaborate with the founding engineer...

  • Cloud Data Engineer

    1 month ago

    San Francisco, California, United States Stefanini North America and APAC Full time

    Job Title: Cloud Data EngineerJob Summary:We are seeking a highly skilled Cloud Data Engineer to join our team at Stefanini North America and APAC. As a Cloud Data Engineer, you will be responsible for designing, developing, and implementing complex data warehouse and business intelligence solutions using cloud-based technologies.Responsibilities:Design and...

  • Data Systems Architect

    4 weeks ago

    San Francisco, California, United States Unreal Gigs Full time

    Unreal Gigs Data Systems EngineerCompany Overview:Welcome to Unreal Gigs, a cutting-edge company that harnesses the potential of data to drive innovation and transformation. Our mission is to develop robust and scalable data systems that empower organizations to extract insights and make data-driven decisions.We're passionate about data engineering and...

  • Data Systems Architect

    1 month ago

    San Francisco, California, United States Unreal Gigs Full time

    Unreal GigsCompany Overview:Welcome to Unreal Gigs, a cutting-edge company that harnesses the potential of data to drive innovation and transformation. Our mission is to develop robust and scalable data systems that empower organizations to extract insights and make data-driven decisions.We're passionate about data engineering and committed to shaping the...

Lead Data Engineer for Data and Systems Development

2 months ago

San Francisco, California, United States Pendulum™ Full time
About Pendulum

Pendulum is a pioneering company that is revolutionizing the way we understand and improve physical and mental health by harnessing the power of the human microbiome.

Our research has shown that the microbiome plays a critical role in various aspects of our lives, from metabolism and diabetes to longevity, weight loss, and healthy immune systems. We are committed to addressing the imbalances in the microbiome and improving people's lives globally.

Position Summary:

We are seeking a highly skilled Lead Data Engineer to join our team and drive the development and optimization of our data infrastructure. This role is crucial for building and maintaining robust data pipelines, managing our data warehouse, and ensuring the reliability and scalability of our data systems.

Key Responsibilities:
  • Design, build, and maintain efficient and scalable ETL pipelines using tools like dbt, Fivetran, and orchestration frameworks such as Airflow.
  • Develop and implement robust schema designs and data models that support efficient querying and data integration across the organization.
  • Manage and optimize our data warehouse and lakehouse environments on platforms like Snowflake, ensuring data is accessible, reliable, and performant.
  • Implement data validation, cleansing, and anomaly detection processes to ensure the integrity and quality of our data.
  • Collaborate with Data Science and Analytics teams to support the deployment of ML models in production, including training, inference, and evaluation processes.
  • Implement monitoring and observability solutions to maintain the reliability and performance of data pipelines and models in production.
  • Leverage Docker, Kubernetes, and workflow management systems to ensure scalable and automated data processing workflows.
  • Ensure compliance with data governance standards and regulations, including GDPR and CCPA, through proper data lineage tracking, metadata management, and secure data handling practices.
Requirements:
  • MSc/PhD in Computer Science, or a related field.
  • 5+ years of experience in data engineering, with a strong focus on building and managing data pipelines, data warehouses, and big data platforms.
  • Expert in Python and SQL, and experience with data technologies (Kafka, Spark, Hive/Iceberg, Postgres, Redis) and cloud infrastructure (GCP, AWS) is required.
  • Proficiency in orchestration and workflow management tools like Airflow, with experience in container orchestration such as Kubernetes being a plus.
  • Strong understanding of data quality management practices, data lineage, metadata management, and compliance with regulations like GDPR and CCPA.
  • Proven ability to work closely with Data Science and Analytics, and other cross-functional teams to deliver data solutions aligned with business needs.
  • Ability to lead data projects independently, make strategic infrastructure decisions, and stay updated with the latest data engineering technologies and practices.
  • Ability to work in a fast-paced, dynamic environment where adaptability is imperative.
What We Offer:
  • $170,000-$225,000
  • Medical, Dental, and Vision
  • Commuter Benefits
  • Life & STD Insurance
  • Company match on 401(k)
  • Flexible Time Off (FTO)
  • Equity