Senior Data Engineer

7 days ago


San Francisco, United States People Data Labs Full time
Job DescriptionJob Description

About Us

At People Data Labs, we're committed to democratizing access to high-quality B2B data and leading the emerging DaaS economy. We empower developers, engineers, and data scientists to create innovative, compliant data products at scale with our clean, easy-to-use datasets of resume, company, location, and education data consumed through our suite of APIs.

PDL is an innovative, fast-growing, global team backed by world-class investors, including Craft Ventures, Flex Capital, and Founders Fund. We scour the world for people hungry to improve, curious about how things work, and willing to challenge the status quo to build something new and better.

Roles & Responsibilities:

  • Build infrastructure for ingestion, transformation, and loading an exponentially increasing volume of data from a variety of sources using Spark, SQL, AWS, and Databricks
  • Building an organic entity resolution framework capable of correctly merging hundreds of billions of individual entities into a number of clean, consumable datasets.
  • Developing CI/CD pipelines and anomaly detection systems capable of continuously improving the quality of data we're pushing into production.
  • Devising solutions to largely-undefined data engineering and data science problems.
  • Work with stakeholders in Engineering and Product to assist with data-related technical issues and support their infrastructure needs

Technical Requirements

  • 5-7+ years industry experience with clear examples of strategic technical problem solving and implementation
  • Strong software development fundamentals.
  • Experience with Python
  • Expertise with Apache Spark (Java, Scala, and/or Python-based)
  • Experience with SQL
  • Experience building scalable data processing systems (e.g., cleaning, transformation) from the ground up.
  • Experience using developer-oriented data pipeline and workflow orchestration (e.g., Airflow (preferred), dbt, dagster or similar)
  • Knowledge of modern data design and storage patterns (e.g., incremental updating, partitioning and segmentation, rebuilds and backfills)
  • Experience working in Databricks (including delta live tables, data lakehouse patterns, etc.)
  • Experience with cloud computing services (AWS (preferred), GCP, Azure or similar)
  • Experience with data warehousing (e.g., Databricks, Snowflake, Redshift, BigQuery, or similar)
  • Understanding of modern data storage formats and tools (e.g., parquet, ORC, Avro, Delta Lake)

Professional Requirements

  • Must thrive in a fast paced environment and be able to work independently
  • Can work effectively remotely (able to be proactive about managing blockers, proactive on reaching out and asking questions, and participating in team activities)
  • Strong written communication skills on Slack/Chat and in documents
  • You are experienced in writing data design docs (pipeline design, dataflow, schema design)
  • You can scope and breakdown projects, communicate and collaborate progress and blockers effectively with your manager, team, and stakeholders

Nice To Haves:

  • Degree in a quantitative discipline such as computer science, mathematics, statistics, or engineering
  • Experience working with entity data (entity resolution / record linkage)
  • Experience working with data acquisition / data integration
  • Expertise with Python and the Python data stack (e.g., numpy, pandas)
  • Experience with streaming platforms (e.g., Kafka)
  • Experience evaluating data quality and maintaining consistently high data standards across new feature releases (e.g., consistency, accuracy, validity, completeness)

Our Benefits

  • Stock
  • Competitive Salaries
  • Unlimited paid time off
  • Medical, dental, & vision insurance
  • Health, fitness, and office stipends
  • The permanent ability to work wherever and however you want

No C2C, 1099, or Contract-to-Hire. Recruiters need not apply.

People Data Labs does not discriminate on the basis of race, sex, color, religion, age, national origin, marital status, disability, veteran status, genetic information, sexual orientation, gender identity or any other reason prohibited by law in provision of employment opportunities and benefits.


  • Senior Data Scientist

    3 weeks ago


    San Francisco, CA, United States Data Masked Full time

    Neighbors around the world turn to Nextdoor daily to receive trusted information, give and get help, get things done, and build real-world connections with those nearby — neighbors, businesses, and public services. At Nextdoor, our Growth Marketing team is charged with driving member growth and revenues from businesses advertising on the platform. This...


  • San Francisco, United States VAST Data Full time

    VAST Data is looking for a Senior Systems Engineer to join our growing team! This is a great opportunity to be part of the fastest-growing infrastructure company in history. In just a few short years, we’ve shaken up the industry by challenging traditional architecture models and introduced a revolutionary set of storage possibilities through our...


  • San Francisco, United States VAST Data Full time

    VAST Data is looking for a Senior Systems Engineer to join our growing team! This is a great opportunity to be part of the fastest-growing infrastructure company in history. In just a few short years, we’ve shaken up the industry by challenging traditional architecture models and introduced a revolutionary set of storage possibilities through our...

  • Lead Data Engineer

    3 hours ago


    San Francisco, United States StreetLight Data Full time

    StreetLight pioneered the use of Big Data analytics to shed light on how people, goods, and services move, empowering smarter, data-driven transportation decisions. The company applies proprietary machine-learning algorithms and data processing resources to measure travel patterns of vehicles, bicycles and pedestrians that enable complex transportation...

  • Senior Data Engineer

    3 hours ago


    San Francisco, United States Apollo.io Full time

    Your Role & Mission As a Senior Data Engineer, you will be responsible for maintaining and operating the data warehouse and connecting in Apollo’s data sources. Daily Adventures and Responsibilities • Develop and maintain scalable data pipelines and build new integrations to support continuing increases in data volume and complexity. • Implement...

  • Senior Data Engineer

    1 month ago


    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to harnessing the power of data to drive transformative change and solve complex problems across industries. We're committed to building scalable and reliable data infrastructure that enables advanced analytics, machine learning,...

  • Senior Data Engineer

    2 weeks ago


    San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to harnessing the power of data to drive transformative change and solve complex problems across industries. We're committed to building scalable and reliable data infrastructure that enables advanced analytics, machine learning,...


  • San Francisco, United States UpRecruit Full time

    Job Details: Title: Sr. Data Engineer Salary: $180-$230K Requirements: 8+ years of software dev exp (Large Data Sets or Big Data) Location: 100% onsite (San Francisco Bay Area) Our client, a gaming startup, is seeking a Sr. Data Engineer to join their passionate team. The ideal candidate is an experienced engineer who will play a vital role in...

  • Senior Data Engineer

    2 months ago


    San Francisco, United States UpRecruit Full time

    Job DescriptionJob DescriptionJob Details: Title: Sr. Data EngineerSalary: $180-$230K Requirements: 8+ years of software dev exp (Large Data Sets or Big Data)Location: 100% onsite (San Francisco Bay Area) Our client, a gaming startup, is seeking a Sr. Data Engineer to join their passionate team. The ideal candidate is an experienced engineer who will play a...

  • Senior Data Engineer

    2 weeks ago


    San Francisco, United States UpRecruit Full time

    Job DescriptionJob DescriptionJob Details: Title: Sr. Data EngineerSalary: $180-$230K Requirements: 8+ years of software dev exp (Large Data Sets or Big Data)Location: 100% onsite (San Francisco Bay Area) Our client, a gaming startup, is seeking a Sr. Data Engineer to join their passionate team. The ideal candidate is an experienced engineer who will play a...


  • San Francisco, United States People Data Labs Full time

    Job DescriptionJob DescriptionAbout UsAt People Data Labs, we're committed to democratizing access to high-quality B2B data and leading the emerging DaaS economy. We empower developers, engineers, and data scientists to create innovative, compliant data products at scale with our clean, easy-to-use datasets of resume, company, location, and education...

  • Senior Data Engineer

    2 months ago


    San Francisco, United States Sephora Full time

    At Sephora we inspire our customers, empower our teams, and help them become the best versions of themselves.  We create an environment where people are valued, and differences are celebrated. Every day, our teams across the world bring to life our purpose: to expand the way the world sees beauty by empowering the ExtraOrdinary in each of us. We are...


  • San Francisco, United States Premise Data Corporation Full time

    We all know every decision should be driven by data. But what about the data you dont know? For years, the status quo in data aggregation has lacked visibility, moved slowly, and cost too muchleaving organizations to make critical decisions, day after day, without the whole picture. Premise changes that. Across 138 countries and counting, our technology...

  • Senior Data Engineer

    12 hours ago


    San Francisco, California, United States Airwallex Full time

    Airwallex is a global payments fintech company transforming the way businesses move and manage money globally. We have built a global financial infrastructure platform to help businesses transact, collect and pay across 130+ countries and 50+ currencies, without the constraints of the traditional global financial system. We've grown to 13 global locations...

  • Senior Data Engineer

    4 hours ago


    San Francisco, United States SmithRx Full time

    Job DescriptionJob DescriptionWho We Are:SmithRx is a rapidly growing, venture-backed Health-Tech company. Our mission is to disrupt the expensive and inefficient Pharmacy Benefit Management (PBM) sector by building a next-generation drug acquisition platform driven by cutting edge technology, innovative cost saving tools, and best-in-class customer service....

  • Senior Data Engineer

    4 weeks ago


    San Francisco, United States Siri InfoSolutions Inc Full time

    Job DescriptionJob DescriptionSenior Data Engineer 12 MonthsSFO, CA( Onsite)Need to be strong with object oriented programming languages Java, Scala, PythonNeed to have some frontend skills JavaScript or similar.Skill set Big Data: Scala, Databricks, Spark SQL, Spark Streaming ( Nice to Have ), Python ( Nice to Have ), PySpark ( Nice to Have )Cloud: Azure ,...


  • San Francisco, United States Social Finance Ltd Full time

    Who we are: Shape a brighter financial future with us. Together with our members, we're changing the way people think about and interact with personal finance. We're a next-generation fintech company using innovative, mobile-first technology to help our millions of members reach their goals. The industry is going through an unprecedented transformation, and...


  • San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to leveraging the power of data to drive transformative change and solve complex problems across industries. We're committed to building scalable and efficient data warehousing solutions that enable advanced analytics, reporting,...


  • San Francisco, United States Unreal Gigs Full time

    Job DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to leveraging the power of data to drive transformative change and solve complex problems across industries. We're committed to building scalable and efficient data warehousing solutions that enable advanced analytics, reporting,...

  • Senior Data Engineer

    15 hours ago


    San Francisco, California, United States FOCUSKPI INC Full time

    Job DescriptionFocusKPI is seeking a talented Senior Data Engineer to join our client who believes Ethereum holds the key to solving crucial coordination issues. From its beginnings as a research group, our client is dedicated to expanding its technology and values. Currently, the main obstacle to Ethereum's growth is performance and scalability, and we are...