Senior Data Engineer

1 month ago

San Francisco, United States People Data Labs Full time
Job DescriptionJob Description

About Us

At People Data Labs, we're committed to democratizing access to high-quality B2B data and leading the emerging DaaS economy. We empower developers, engineers, and data scientists to create innovative, compliant data products at scale with our clean, easy-to-use datasets of resume, company, location, and education data consumed through our suite of APIs.

PDL is an innovative, fast-growing, global team backed by world-class investors, including Craft Ventures, Flex Capital, and Founders Fund. We scour the world for people hungry to improve, curious about how things work, and willing to challenge the status quo to build something new and better.

Roles & Responsibilities:

  • Build infrastructure for ingestion, transformation, and loading an exponentially increasing volume of data from a variety of sources using Spark, SQL, AWS, and Databricks
  • Building an organic entity resolution framework capable of correctly merging hundreds of billions of individual entities into a number of clean, consumable datasets.
  • Developing CI/CD pipelines and anomaly detection systems capable of continuously improving the quality of data we're pushing into production.
  • Devising solutions to largely-undefined data engineering and data science problems.
  • Work with stakeholders in Engineering and Product to assist with data-related technical issues and support their infrastructure needs

Technical Requirements

  • 5-7+ years industry experience with clear examples of strategic technical problem solving and implementation
  • Strong software development fundamentals.
  • Experience with Python
  • Expertise with Apache Spark (Java, Scala, and/or Python-based)
  • Experience with SQL
  • Experience building scalable data processing systems (e.g., cleaning, transformation) from the ground up.
  • Experience using developer-oriented data pipeline and workflow orchestration (e.g., Airflow (preferred), dbt, dagster or similar)
  • Knowledge of modern data design and storage patterns (e.g., incremental updating, partitioning and segmentation, rebuilds and backfills)
  • Experience working in Databricks (including delta live tables, data lakehouse patterns, etc.)
  • Experience with cloud computing services (AWS (preferred), GCP, Azure or similar)
  • Experience with data warehousing (e.g., Databricks, Snowflake, Redshift, BigQuery, or similar)
  • Understanding of modern data storage formats and tools (e.g., parquet, ORC, Avro, Delta Lake)

Professional Requirements

  • Must thrive in a fast paced environment and be able to work independently
  • Can work effectively remotely (able to be proactive about managing blockers, proactive on reaching out and asking questions, and participating in team activities)
  • Strong written communication skills on Slack/Chat and in documents
  • You are experienced in writing data design docs (pipeline design, dataflow, schema design)
  • You can scope and breakdown projects, communicate and collaborate progress and blockers effectively with your manager, team, and stakeholders

Nice To Haves:

  • Degree in a quantitative discipline such as computer science, mathematics, statistics, or engineering
  • Experience working with entity data (entity resolution / record linkage)
  • Experience working with data acquisition / data integration
  • Expertise with Python and the Python data stack (e.g., numpy, pandas)
  • Experience with streaming platforms (e.g., Kafka)
  • Experience evaluating data quality and maintaining consistently high data standards across new feature releases (e.g., consistency, accuracy, validity, completeness)

Our Benefits

  • Stock
  • Competitive Salaries
  • Unlimited paid time off
  • Medical, dental, & vision insurance
  • Health, fitness, and office stipends
  • The permanent ability to work wherever and however you want

Salary: $170K - $220K

No C2C, 1099, or Contract-to-Hire. Recruiters need not apply.

People Data Labs does not discriminate on the basis of race, sex, color, religion, age, national origin, marital status, disability, veteran status, genetic information, sexual orientation, gender identity or any other reason prohibited by law in provision of employment opportunities and benefits.

  • Senior Data Engineer

    2 months ago

    San Francisco, California, United States Innovaccer Full time

    Position:Senior Data Engineer (multiple openings)Job Location:InnovAccer, Inc. 101 Mission Street, Suite 1950, San Francisco, CA allows for telecommuting)Job Duties:With a high level of independent decision-making capability and minimum supervision, the Senior Data Engineer will be responsible for performing the following duties:Defining the end-to-end data...

  • Senior Data Engineer

    3 weeks ago

    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to harnessing the power of data to drive transformative change and solve complex problems across industries. We're committed to building scalable and reliable data infrastructure that enables advanced analytics, machine learning,...

  • Senior Data Engineer

    3 months ago

    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to harnessing the power of data to drive transformative change and solve complex problems across industries. We're committed to building scalable and reliable data infrastructure that enables advanced analytics, machine learning,...

  • Senior Data Engineer

    3 weeks ago

    San Francisco, United States StyleSeat Full time

    Job DescriptionJob DescriptionSenior Data Engineer100% Remote (U.S. Based Only, Select States - See Below)About the roleStyleSeat is looking to add a Senior Data Engineer to its cross-functional Search product team. This team of data scientists, analysts, data engineers, software engineers and SDETs is focused on improving our search capability and customer...

  • Senior Data Engineer

    2 months ago

    San Francisco, United States SmithRx Full time

    Job DescriptionJob DescriptionWho We Are:SmithRx is a rapidly growing, venture-backed Health-Tech company. Our mission is to disrupt the expensive and inefficient Pharmacy Benefit Management (PBM) sector by building a next-generation drug acquisition platform driven by cutting edge technology, innovative cost saving tools, and best-in-class customer service....

  • Senior Data Warehouse Engineer

    3 weeks ago

    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to leveraging the power of data to drive transformative change and solve complex problems across industries. We're committed to building scalable and efficient data warehousing solutions that enable advanced analytics, reporting,...

  • Senior Data Warehouse Engineer

    3 months ago

    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to leveraging the power of data to drive transformative change and solve complex problems across industries. We're committed to building scalable and efficient data warehousing solutions that enable advanced analytics, reporting,...

  • Senior Data Systems Engineer

    3 weeks ago

    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to harnessing the power of data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge data systems that enable efficient data management, processing, and analysis....

  • Senior Data Systems Engineer

    3 months ago

    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to harnessing the power of data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge data systems that enable efficient data management, processing, and analysis....

  • Senior Data Systems Engineer

    3 months ago

    San Francisco, United States Unreal Gigs Full time

    Company Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to harnessing the power of data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge data systems that enable efficient data management, processing, and analysis. Join us and be part of a dynamic team...

  • Senior Cloud Data Engineer

    3 weeks ago

    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to harnessing the power of data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge cloud data solutions that leverage the scalability, flexibility, and power of...

  • Senior Cloud Data Engineer

    3 months ago

    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to harnessing the power of data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge cloud data solutions that leverage the scalability, flexibility, and power of...

  • Senior Big Data Engineer

    3 weeks ago

    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is committed to harnessing the power of big data to drive transformative change and solve complex problems across industries. We're dedicated to building scalable and efficient big data solutions that enable advanced analytics, machine...

  • Senior Big Data Engineer

    3 months ago

    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is committed to harnessing the power of big data to drive transformative change and solve complex problems across industries. We're dedicated to building scalable and efficient big data solutions that enable advanced analytics, machine...

  • Senior Data Engineer

    3 months ago

    San Francisco, United States Nightfall AI Full time

    Nightfall AI (www.nightfall.ai) is the unified platform that prevents data leaks and enables secure collaboration by protecting sensitive data and controlling how it's shared. For decades, legacy data leak prevention (DLP) solutions have failed to adequately protect sensitive information. Traditional DLP is outdated, intrusive, and complex - it...

  • Senior Data Engineer

    3 months ago

    San Francisco, California, United States Nightfall AI Full time

    Nightfall AI ) is the unified platform that prevents data leaks and enables secure collaboration by protecting sensitive data and controlling how it's shared. For decades, legacy data leak prevention (DLP) solutions have failed to adequately protect sensitive information. Traditional DLP is outdated, intrusive, and complex - it wasn't designed for today's...

  • Senior Data Engineer

    1 day ago

    San Francisco, California, United States Capital One Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Capital One. As a pioneer in technology, you will have the opportunity to drive a major transformation within our organization.Key ResponsibilitiesCollaborate with Agile teams to design, develop, test, implement, and support technical solutions in full-stack development...

  • Senior Engineering Manager, Data Platform

    1 day ago

    San Francisco, United States DoorDash USA Full time

    About the TeamDoorDash is a data-driven organization and relies on timely, accurate, and reliable data to drive many business and product decisions. The Data Platform owns all the infrastructure necessary to run an operationally efficient analytical data stack. The Core Data part of this includes data ingestion (batch and real-time), data compute &...

  • Senior Real-Time Data Engineer

    3 weeks ago

    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to harnessing the power of real-time data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge real-time data solutions that enable timely insights and actions. Join...

  • Senior Real-Time Data Engineer

    3 months ago

    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to harnessing the power of real-time data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge real-time data solutions that enable timely insights and actions. Join...