Senior Data Engineer

1 week ago

Santa Rosa, CA, United States Sigmaways Full time

We are seeking an experienced Data Engineer to design and optimize scalable data pipelines that drive our global data and analytics initiatives.

In this role, you will leverage technologies such as Apache Spark, Airflow, and Python to build high performance data processing systems and ensure data quality, reliability, and lineage across Mastercard's data ecosystem.

The ideal candidate combines strong technical expertise with hands-on experience in distributed data systems, workflow automation, and performance tuning to deliver impactful, data-driven solutions at enterprise scale.

Responsibilities:

  • Design and optimize Spark-based ETL pipelines for large-scale data processing.
  • Build and manage Airflow DAGs for scheduling, orchestration, and checkpointing.
  • Implement partitioning and shuffling strategies to improve Spark performance.
  • Ensure data lineage, quality, and traceability across systems.
  • Develop Python scripts for data transformation, aggregation, and validation.
  • Execute and tune Spark jobs using spark-submit.
  • Perform DataFrame joins and aggregations for analytical insights.
  • Automate multi-step processes through shell scripting and variable management.
  • Collaborate with data, DevOps, and analytics teams to deliver scalable data solutions.

Qualifications:

  • Bachelor's degree in Computer Science, Data Engineering, or related field (or equivalent experience).
  • At least 7 years of experience in data engineering or big data development.
  • Strong expertise in Apache Spark architecture, optimization, and job configuration.
  • Proven experience with Airflow DAGs using authoring, scheduling, checkpointing, monitoring.
  • Skilled in data shuffling, partitioning strategies, and performance tuning in distributed systems.
  • Expertise in Python programming including data structures and algorithmic problem-solving.
  • Hands-on with Spark DataFrames and PySpark transformations using joins, aggregations, filters.
  • Proficient in shell scripting, including managing and passing variables between scripts.
  • Experienced with spark submit for deployment and tuning.
  • Solid understanding of ETL design, workflow automation, and distributed data systems.
  • Excellent debugging and problem-solving skills in large-scale environments.
  • Experience with AWS Glue, EMR, Databricks, or similar Spark platforms.
  • Knowledge of data lineage and data quality frameworks like Apache Atlas.
  • Familiarity with CI/CD pipelines, Docker/Kubernetes, and data governance tools.

  • Senior Data Engineer

    3 days ago

    Santa Monica, CA, United States Software Resources Full time

    Software Resources has an immediate job opportunity for a Senior Data Engineer (Graph) with a major corporation in Santa Monica, CA. 4 days per week on-site. Duration: 12 month(s) Pay Rate: $90 -$93/hr DOE Must Haves: Focus on graph databases (e.g., Neo4j), but candidates with broader data engineering skills (including Snowflake) are preferred for...

  • Senior Data Engineer/Snowflake Architect

    3 days ago

    Santa Clarita, CA, United States Inficare Full time

    Title: Senior Data Engineer (Job Location: Santa Clara, CA -95054 (Onsite role & need only local canddiate) Job Type: ContractJob Duration: 12 Months + Senior Data Engineer - Key Requirements: Strong hands-on experience in data engineering using Snowflake and Databricks, with proven ability to build and optimize large-scale data pipelines. Deep...

  • Senior Software Engineer

    1 week ago

    Santa Rosa, CA, United States Raydar Inc Full time

    Senior Software Engineer - Product San Francisco, CA (In-Person, 5 Days/Week) $200K-$320K base + 0.1-0.3% equityAbout the Role:We're partnering with a fast-growing AI startup that's redefining how the insurance broking industry operates. The company builds automation tools that turn hours of manual work into minutes, transforming how brokers process data,...

  • Founding Data Engineer

    5 days ago

    Santa Rosa, CA, United States Strativ Group Full time

    Our client is an elite applied AI research and product lab focused on building AI-native systems for finance and deploying cutting-edge models into real production environments. Their work navigates the critical interface of data, research, and high-stakes financial decision-making. As a Founding Data Engineer, you will take charge of the data platform that...

  • Founding Data Engineer

    2 weeks ago

    Santa Rosa, CA, United States Strativ Group Full time

    Our client is an elite applied AI research and product lab focused on building AI-native systems for finance and deploying cutting-edge models into real production environments. Their work navigates the critical interface of data, research, and high-stakes financial decision-making. As a Founding Data Engineer, you will take charge of the data platform that...

  • Founding Data Engineer

    7 days ago

    Santa Rosa, CA, United States Strativ Group Full time

    Our client is an elite applied AI research and product lab focused on building AI-native systems for finance and deploying cutting-edge models into real production environments. Their work navigates the critical interface of data, research, and high-stakes financial decision-making. As a Founding Data Engineer, you will take charge of the data platform that...

  • Staff Data Engineer

    2 weeks ago

    Santa Rosa, CA, United States Strativ Group Full time

    San Francisco (Hybrid) Founding/Staff Data Engineer $200-300k baseOur client is an elite applied AI research and product lab building AI-native systems for finance-and pushing frontier models into real production environments. Their work sits at the intersection of data, research, and high-stakes financial decision-making.As the Founding Data Engineer, you...

  • Staff Data Engineer

    1 week ago

    Santa Rosa, CA, United States Strativ Group Full time

    San Francisco (Hybrid) Founding/Staff Data Engineer $200-300k baseOur client is an elite applied AI research and product lab building AI-native systems for finance-and pushing frontier models into real production environments. Their work sits at the intersection of data, research, and high-stakes financial decision-making.As the Founding Data Engineer, you...

  • Senior/Staff Engineer

    1 week ago

    Santa Rosa, CA, United States Pagos consultants Full time

    Senior / Staff Software Engineer Location: San Francisco or New York City (Hybrid) About the Role Join our client, a trailblazer in the enterprise Model Context Protocol (MCP) space, as they develop the AI enablement layer that empowers companies to securely integrate AI agents with their internal tools. This dynamic platform prioritizes strong controls...

  • Senior Backend Engineer

    1 week ago

    Santa Rosa, CA, United States UMATR Full time

    Senior Software Engineer (Backend / Platform)UMATR are partnered with an early-stage technology company building an AI-driven internal platform designed to help fast-growing organisations maintain clarity, alignment, and operational efficiency as they scale. Their product acts as an intelligent internal system that understands company context, objectives,...