Data Engineer

3 days ago

San Francisco, United States ZipRecruiter Full time

Job Description

Are you passionate about designing, building, and maintaining data pipelines that support robust data architectures and facilitate seamless data flow? Do you excel in creating scalable solutions that empower data-driven decision-making? If you’re ready to develop and optimize data systems that drive impactful analytics, our client has the perfect role for you. We’re seeking a Data Engineer (aka The Data Pipeline Architect) to build and manage cloud-based data infrastructures that support analytical needs and operational efficiencies.

As a Data Engineer at our client, you’ll collaborate with data scientists, analysts, and software engineers to construct data pipelines and storage solutions that are both efficient and secure. Your role will be critical in ensuring data systems are optimized for performance, reliability, and scalability.

Key Responsibilities:

  1. Design and Implement Scalable Data Pipelines: Develop and maintain data pipelines that support data ingestion, transformation, and integration using cloud technologies. You’ll automate data workflows and ensure the seamless movement of data between various systems.
  2. Manage and Optimize Data Storage Solutions: Architect and maintain data lakes and data warehouses using platforms like BigQuery, Redshift, Snowflake, or similar cloud-based solutions. You’ll ensure data structures are built for performance and scalability.
  3. Collaborate with Data Teams for Strategy Development: Work closely with data scientists, analysts, and business stakeholders to understand data requirements and align data solutions with business goals. You’ll provide input on data models and storage strategies.
  4. Ensure Data Quality and Reliability: Implement and manage processes for data validation, error handling, and consistency checks. You’ll ensure the quality of data is maintained through robust testing and monitoring practices.
  5. Develop and Automate ETL Processes: Build ETL (Extract, Transform, Load) workflows to handle complex data transformations. You’ll automate data extraction and transformation to support efficient data integration and reporting.
  6. Monitor and Maintain Data Infrastructure: Use monitoring tools to track the performance and reliability of data systems. You’ll proactively identify and resolve potential issues to maintain system health and performance.
  7. Optimize Data Processing and Resource Management: Implement strategies for efficient resource allocation and cost-effective data processing. You’ll leverage parallel processing and cloud capabilities to enhance performance.

Requirements

Required Skills:

  • Cloud Data Platform Expertise: Experience with cloud data platforms such as AWS (Redshift, S3, Glue), GCP (BigQuery, Dataflow), or Azure (Azure Data Lake, Synapse). You’re proficient in handling cloud-based data solutions.
  • Programming and Scripting Knowledge: Proficiency in Python, Java, or Scala for building data pipelines and data processing tasks. You can write clean, efficient code for automation.
  • ETL and Data Pipeline Management: Proven ability to develop, maintain, and optimize ETL processes that handle large volumes of data. You’re experienced with orchestration tools like Apache Airflow or Luigi.
  • SQL and Database Management: Strong ability to write complex SQL queries and work with relational and NoSQL databases.
  • Problem-Solving and Critical Thinking: Excellent problem-solving skills with a proactive approach to identifying and resolving data-related challenges.

Educational Requirements:

  • Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field. Equivalent experience in data engineering and cloud technologies may be considered.
  • Certifications in cloud data engineering (e.g., Google Professional Data Engineer, AWS Certified Big Data – Specialty, Microsoft Certified: Azure Data Engineer Associate) are a plus.

Experience Requirements:

  • 3+ years of experience in data engineering, with a proven track record of building and managing cloud-based data systems.
  • Experience with real-time data processing frameworks like Apache Kafka or AWS Kinesis is advantageous.
  • Familiarity with containerization and microservices architecture is a plus.

Benefits

  • Health and Wellness: Comprehensive medical, dental, and vision insurance plans with low co-pays and premiums.
  • Paid Time Off: Competitive vacation, sick leave, and 20 paid holidays per year.
  • Work-Life Balance: Flexible work schedules and telecommuting options.
  • Professional Development: Opportunities for training, certification reimbursement, and career advancement programs.
  • Wellness Programs: Access to wellness programs, including gym memberships, health screenings, and mental health resources.
  • Life and Insurance: Life insurance and short-term/long-term coverage.
  • Employee Assistance Program (EAP): Confidential counseling and support services for personal and professional challenges.
  • Tuition Reimbursement: Financial assistance for continuing education and professional development.
  • Community Engagement: Opportunities to participate in community service and volunteer activities.
  • Recognition Programs: Employee recognition programs to celebrate achievements and milestones.
#J-18808-Ljbffr

  • DC Design Manager for Region, Data Center Engineering

    1 week ago

    San Francisco, United States Amazon Data Services, Inc. Full time

    This unique vantage point allows the Design Managers to maintain a close connection with project details and drive project outcomes.You will have an impact on the design direction and ability to improve design delivery for an entire region, while establishing design procedures and protocols for the development of future systems.Responsibilities...

  • Data Engineer

    1 month ago

    San Francisco, California, United States Syntricate Technologies Full time

    Data EngineerJob Summary: We are seeking a highly skilled Data Engineer to join our team at Syntricate Technologies. As a Data Engineer, you will be responsible for designing, implementing, and maintaining large-scale data pipelines using Snowflake and other data engineering tools. You will work closely with our data team to ensure the accuracy and...

  • Data Engineer

    2 weeks ago

    San Francisco, United States Harnham Full time

    Data Engineer1 Year ContractHybrid (2 Days Onsite)$75-85/hrWe are partnered with a leading beauty retailer, delivering innovative and personalized beauty experiences to their customers. We are looking for a highly skilled Data Engineer to join the team and help enhance our data platforms and analytics capabilities. You will work with cutting-edge...

  • Data Engineer

    2 weeks ago

    san francisco, United States Harnham Full time

    Data Engineer1 Year ContractHybrid (2 Days Onsite)$75-85/hrWe are partnered with a leading beauty retailer, delivering innovative and personalized beauty experiences to their customers. We are looking for a highly skilled Data Engineer to join the team and help enhance our data platforms and analytics capabilities. You will work with cutting-edge...

  • Data Engineer

    2 weeks ago

    san francisco, United States Harnham Full time

    Data Engineer1 Year ContractHybrid (2 Days Onsite)$75-85/hrWe are partnered with a leading beauty retailer, delivering innovative and personalized beauty experiences to their customers. We are looking for a highly skilled Data Engineer to join the team and help enhance our data platforms and analytics capabilities. You will work with cutting-edge...

  • Data Engineer

    1 month ago

    San Francisco, California, United States Bayone Full time

    Job Title: Data EngineerCompany Overview:At Bayone, we strive to create an environment where people are valued, and differences are celebrated. Our purpose is to empower our teams and inspire our customers to become the best versions of themselves.The Opportunity:Our technology team works fast and smart, developing the latest in mobile technologies, scalable...

  • Data Engineer

    2 weeks ago

    San Francisco, United States Legalist Full time

    Job DescriptionJob DescriptionIntro description:Legalist is an institutional alternative asset management firm. Founded in 2016 and incubated at Y Combinator, the firm uses data-driven technology to invest in credit assets at scale. We are always looking for talented people to join our team.As a highly collaborative organization, our data engineers work...

  • Data Center Operations Specialist

    3 days ago

    San Francisco, California, United States Amazon Data Services, Inc. Full time

    About Amazon Data Services, Inc.At Amazon Data Services, Inc., we're passionate about delivering exceptional customer experiences. As the world's most comprehensive and broadly adopted cloud platform, we're committed to innovation and excellence in all that we do.We're seeking a highly skilled Data Center Operations Specialist to join our team. In this...

  • Data Engineer

    1 month ago

    San Francisco, California, United States Airbyte Full time

    We are seeking a highly skilled and experienced Analytics Engineer to join the Data Team at Airbyte. In this role, you will work closely with product managers, engineers, and functional leads to build and maintain robust data pipelines and analytics infrastructure.Data Pipeline Development:Design, build, and maintain scalable and efficient data pipelines...

  • Data Engineer

    4 weeks ago

    San Francisco, California, United States Factory Full time

    Job Description:Factory is seeking a highly skilled Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, building, and maintaining optimal data pipelines that support a variety of data consumers.Key Responsibilities:Design and implement robust data warehousing solutions to ensure secure and efficient data storage and...

  • Lead Data Engineer

    1 week ago

    san francisco, United States Unreal Gigs Full time

    DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to harnessing the power of data to drive transformative change and solve complex problems across industries. We're committed to building scalable and reliable data infrastructure that enables advanced analytics, machine learning, and business...

  • Data Engineer

    1 month ago

    san francisco, United States Replicate Full time

    You’re a generalist data and analytics expert who builds data infrastructure at scale. You act like an owner and have a desire to lead. You’ve likely been a data engineer at traditional companies but you’re ready to be the first data hire at a startup.Replicate is a complex business and we need solid data infrastructure to guide it. You’ll own this...

  • Data Engineer

    2 months ago

    san francisco, United States Replicate Full time

    You’re a generalist data and analytics expert who builds data infrastructure at scale. You act like an owner and have a desire to lead. You’ve likely been a data engineer at traditional companies but you’re ready to be the first data hire at a startup.Replicate is a complex business and we need solid data infrastructure to guide it. You’ll own this...

  • Data Engineer

    2 months ago

    San Francisco, United States Replicate Full time

    You’re a generalist data and analytics expert who builds data infrastructure at scale. You act like an owner and have a desire to lead. You’ve likely been a data engineer at traditional companies but you’re ready to be the first data hire at a startup.Replicate is a complex business and we need solid data infrastructure to guide it. You’ll own this...

  • Software Engineer

    3 weeks ago

    San Francisco, United States Factory Full time

    Factory is looking for highly capable engineers focused on harnessing and managing data that powers both our internal AI-driven systems and external customer-facing features. What you will do and achieve Design, build, and maintain optimal data pipelines that support a variety of data consumers, from internal teams focused on Droid optimization to external...

  • Senior Data Engineer

    1 month ago

    San Francisco, California, United States Syntricate Technologies Full time

    Job Title: Senior Data EngineerAbout the Role:We are seeking a highly skilled Senior Data Engineer to join our team at Syntricate Technologies. As a Senior Data Engineer, you will be responsible for designing, implementing, and maintaining large-scale data pipelines using SQL and Snowflake. You will work closely with our data team to ensure the smooth...

  • AWS Data Engineer

    1 month ago

    san francisco, United States Norconsulting Global Recruitment Full time

    In Norconsulting we are currently looking for AWS Data Engineer to join one of our clients in San FranciscoAWS Data EngineerLocation: San Francisco ( 3 days onsite 2 days remote)look for nearby Candidates only ( Only w2 Required)Rate: 70 USD per hourMust Have Skills : Spark, AWS, data lake, data pipelining, pythonJob Description:As an AWS Data Engineer, you...

  • AWS Data Engineer

    1 month ago

    San Francisco, United States Norconsulting Global Recruitment Full time

    In Norconsulting we are currently looking for AWS Data Engineer to join one of our clients in San FranciscoAWS Data EngineerLocation: San Francisco ( 3 days onsite 2 days remote)look for nearby Candidates only ( Only w2 Required)Rate: 70 USD per hourMust Have Skills : Spark, AWS, data lake, data pipelining, pythonJob Description:As an AWS Data Engineer, you...

  • Director of Data Engineering, Data and Systems

    4 weeks ago

    San Francisco, California, United States Pendulum Full time

    Pendulum is a revolutionary company leading a global movement to improve physical and mental health by understanding, restoring, and enhancing the human microbiome.The company has made significant breakthroughs in the field, creating proprietary probiotic pipelines and a unique discovery platform to identify key bacterial strains and prebiotics.As a Director...

  • Data Scientist/Engineer

    1 month ago

    San Francisco, California, United States SynergisticIT Full time

    We are seeking a highly motivated and skilled Data Scientist/Engineer to join our team at SynergisticIT. As a Data Scientist/Engineer, you will be responsible for designing, developing, and implementing data-driven solutions to drive business growth and improve operational efficiency.Key Responsibilities:Design and develop data models and algorithms to solve...