Sr. Data Engineer

3 days ago


San Francisco, United States eTeam Full time

Job Title: Sr. Data Engineer

Job Duration: 6 months contract

Job Location: San Francisco, CA, Hybrid (San Fran)/2 days a week between Tues through Thurs


Our Client is looking for an experienced Data Engineer to support the Marketing Solutions execution of complex data development and database efficiency, with a deep focus on improving the data quality, accuracy and timeliness for marketing related products. This role will interact directly with Engineering teams to optimize data operations for product metrics to surface insights at terabyte scale, globally. As our data and business needs increase, we continue to iterate and evolve our processes, best-practice documentation, and deep-dive retrospectives to inform our go/no go decision making. Ultimately to be successful, the individual should feel comfortable advising a team of data professionals, navigating our tech stack, and recommending solutions to complex data problems. Our team is highly collaborative, open and honest, and brings a low-ego, high impact mindset to work

Responsibilities:

Design, develop, and manage data pipelines and workflows to enable efficient and accurate data processing using Trino SQL/Spark SQL warehoused in HDFS datasets.

Effectively performs code designs and reviews/approves test cases.

Implement data quality checks and audits to maintain high data accuracy and integrity.

Produces elegant and efficient designs, high performance, and scalable code that allows for easy extension to future needs.

Collaborate with cross-functional teams, especially data engineering, to understand data requirements and implement robust data solutions.

Work closely with data domain experts to gather data requirements, translate business needs into technical specifications, and communicate data insights effectively for sales representative workflow efficiency.

Optimize data storage for performance and scalability, ensuring efficient data Extraction, Transformation and Load (ETL).

Develop and maintain documentation related to data pipelines, QA, metrics, and data policy as it relates to best practice, compliance and GDPR.

Stay up to date with industry best practices and emerging trends in data engineering and analytics, including Generative AI as it impacts our data operations.


Qualifications:

2+ years in using SQL and experience optimizing SQL databases for performance (Trino SQL, or Spark).

Demonstrated experience in managing data pipelines (like HDFS), data repository (like GitHub), workflows (like Apache Airflow), and ETL (best practice coding).

Ability to communicate complex technical concepts to both technical and non-technical individuals.

Experience working with multiple stakeholders, setting project priorities and delivering on Objectives and Key Results (OKRs).

Experience automating script changes in Python.


Preferred Qualifications:

BA/BS in engineering, computer science, or related technical field (such as statistics, or data science).

Excellent analytical skills, designing data workflows and analyzing data for anomalies, or setting data quality thresholds via automated solutions.

Familiarity with data governance principles

Program Manager experience

Demonstrated experience in managing data pipelines in HDFS

Experience running a scrum team and using Jira.

Spark SQL


Suggested Skills:

Data Analysis

Project Management

Data Engineering


  • Sr. Data Engineer

    4 weeks ago


    San Francisco, United States SGF Global Full time

    Job Title: Sr. Data Engineer Job Title: Sr. Data Engineer (Remote for California based team) California -USA SGF Global is looking for a Sr. Data Engineer for our client based in California USA. Requisitions: You're a talented, creative, and motivated engineer who loves developing powerful, stable, and intuitive apps - and you're excited to...

  • Sr. Data Engineer

    5 days ago


    San Francisco, United States SGF Global Full time

    Job Title: Sr. Data Engineer Job Title: Sr. Data Engineer (Remote for California based team) California -US SGF Global is looking for a Sr. Data Engineer for our client based in California USA. Requisitions: You're a talented, creative, and motivated engineer who loves developing powerful, stable, and intuitive apps - and you're excited to...

  • Sr. Data Engineer

    1 month ago


    San Francisco, United States Windfall Full time

    Windfall is seeking a Sr. Data Engineer to join our core data team. As a Sr. Data Engineer on our core data team, you will be building out the core data asset that everything else at Windfall is built on top of. Communication and collaboration are at the heart of Windfall, and you will work closely with our product, data science, and other engineering teams....

  • Sr. Data Engineer

    1 day ago


    San Francisco, United States Smart IT Frame LLC Full time

    Job Title: Sr. Data EngineerLocation: San Francisco, CA (Hybrid)Work Type: Full TimeJob Description:-Create and maintain optimal data pipeline architecture Build data pipelines that transform raw, unstructured data into formats that data analyst can use to for analysis Assemble large, complex data sets that meet functional / non-functional business...

  • Sr. Data Engineer

    1 day ago


    San Francisco, United States Smart IT Frame LLC Full time

    Job Title: Sr. Data EngineerLocation: San Francisco, CA (Hybrid)Work Type: Full TimeJob Description:-Create and maintain optimal data pipeline architecture Build data pipelines that transform raw, unstructured data into formats that data analyst can use to for analysis Assemble large, complex data sets that meet functional / non-functional business...


  • san francisco, United States eTeam Full time

    Job Title: Sr. Data Engineer Job Duration: 6 months contractJob Location: San Francisco, CA, Hybrid (San Fran)/2 days a week between Tues through ThursOur Client is looking for an experienced Data Engineer to support the Marketing Solutions execution of complex data development and database efficiency, with a deep focus on improving the data quality,...

  • Sr. Backend Engineer

    14 hours ago


    San Francisco, United States Hashicorp Full time

    About Engineering at HashiCorpOn the HashiCorp engineering team, we build the Infrastructure Cloud which allows enterprises to take a unified approach to Infrastructure and Security Lifecycle Management:Infrastructure Lifecycle Management: Build / Deploy / ManageTerraform allows you to use infrastructure as code to provision and manage any infrastructure...

  • Sr. Data Scientist

    2 days ago


    San Francisco, United States Abacus Full time

    Title Data Analyst-III Location HQ, USA, CA, San Francisco, 2nd St Description Job Title: Sr. Data Scientist Job Description: Utilize quantitative analysis to understand business drivers and conduct analytics experiments for hypothesis testing. Leverage statistical techniques to provide business recommendations and insights into marketplace dynamics and...


  • San Francisco, California, United States YO HR CONSULTANCY Full time

    We are seeking a highly skilled Sr Data Engineer to join our team in Glendale, CA. The ideal candidate will have 7+ years of experience in data engineering and proficiency in at least one major programming language (e.g. Python, Java, Scala).The successful candidate will work with technologies such as Airflow, Spark, Databricks, and Snowflake, and contribute...


  • San Francisco, California, United States Windfall Full time

    About the JobWe're looking for a talented Sr. Data Engineer to build out the core data asset that everything else at Windfall is built on top of. As a key member of our core data team, you'll design and implement scalable data pipelines, collaborate with cross-functional teams, and leverage cutting-edge technologies like GCP, Java, Python, and Kotlin.Job...


  • San Francisco, United States Robert Half Full time

    Job DescriptionJob DescriptionWe are offering a short term contract employment opportunity for a Sr Data Entry Specialist in San Francisco, California. As a Sr Data Entry Specialist, you will be primarily focused on data entry and processing, maintaining both electronic and physical filing systems, troubleshooting data-related issues, and upholding the...


  • San Francisco, United States Amazon Full time

    Sr. Software Engineer, Data Plane, NPD Forwarding Stack, Data Plane Job ID: 2776438 | Amazon Data Services, Inc. AWS Infrastructure Services owns the design, planning, delivery, and operation of all AWS global infrastructure. In other words, we’re the people who keep the cloud running. We support all AWS data centers and all of the servers, storage,...

  • Senior QA Engineer

    2 weeks ago


    San Francisco, California, United States Altruist Full time

    />We're in the midst of an exceptional growth phase and we're excited to hire a Sr. QA Engineer to join our growing Data team.


  • San Francisco, United States Amazon Full time

    Sr. Software Engineer, Data Plane, NPD Forwarding Stack, Data PlaneJob ID: 2776438 | Amazon Data Services, Inc.AWS Infrastructure Services owns the design, planning, delivery, and operation of all AWS global infrastructure. In other words, we’re the people who keep the cloud running. We support all AWS data centers and all of the servers, storage,...


  • San Francisco, California, United States Braintrust Data Full time

    About Braintrust DataWe are a cutting-edge developer platform for building world-class AI products. Our innovative approach combines code and datasets, incrementally refining both using frequent evaluations. We provide a rich set of tools to visualize changes and interrogate failures, empowering developers to integrate our platform into their continuous...


  • San Francisco, United States Robert Half Full time

    Job DescriptionJob DescriptionWe are on the lookout for a Sr Data Entry Specialist to become a part of our team. This role is based in San Francisco, California, and offers a short term contract employment opportunity. As a Sr Data Entry Specialist, you will play a crucial role in maintaining and managing data across various database systems, ensuring the...


  • San Francisco, California, United States VAST Data Full time

    About VAST DataVAST Data is a pioneering company in the field of data management, revolutionizing the way enterprises capture, catalog, refine, enrich, and protect massive datasets. As a leader in enterprise software infrastructure, we empower real-time data analysis and AI training and inference.We are driven by innovation, customer-centricity, and a...


  • San Francisco, United States Robert Half Full time

    Job DescriptionJob DescriptionWe are in search of a Sr Data Entry Specialist to join our team in San Francisco, California. In this role, you will be tasked with managing data entry tasks, maintaining the integrity of our databases, and resolving any data-related issues. Your key responsibilities will also include maintaining an organized filing system and...


  • San Francisco, United States Robert Half Full time

    Job DescriptionJob DescriptionWe are offering a short term contract employment opportunity in San Francisco, California, for a Sr Data Entry Specialist to join our dynamic team. This role primarily involves managing and processing customer data in a variety of database systems.Responsibilities:• Accurately input and upkeep data in several database...

  • Sr. Data Engineer

    2 weeks ago


    San Francisco Bay Area, United States Flexton Inc. Full time

    Pay Rate: $80/hr to $90/hr on W2Type: ContractMust Haves:JavaPythonShellResponsibilities: -Design, develop, and manage data pipelines and workflows to enable efficient and accurate data processing using Trino SQL/Spark SQL warehoused in HDFS datasets. -Effectively performs code designs and reviews/approves test cases. -Implement data quality checks and...