Senior Data Ingestion Engineer

1 month ago


dallas, United States Tiva's Client Full time

Senior Data Ingestion Engineer

Most important for this role (key skills)

  • Kafka (data streaming)
  • Python/PySparkAWS
  • Databricks Cloud Platform
  • Unity Catalog


Role Overview:

This role will be responsible for conducting data ingestion and platform management on the Databricks platform. This role requires a deep understanding of:

  • Data Lake ingestion processes and best practices
  • Data Streaming (Kafka)
  • ETL/ELT implementation
  • CI/CD
  • System integration tools
  • Data pipeline management



About the role:

  • Ingest data from a variety of source systems and tailor ingestion approaches on a per-system basis
  • Manage, maintain, and oversee ETL/ELT pipelines on the Databricks platformOptimize data pipelines for scalability and speed
  • Document ingestion and integration flows and pipelines
  • Use Airflow to schedule and automate ingestion jobs
  • Manage metadata and master data in technical data catalog
  • Ensure ELT/ETL design meets required security and compliance guidelines, and ensure PII management, flagging and risk assessment during ingestion
  • Maintain ETL/ELT pipeline infrastructure and implement automated monitoring strategies
  • Ensure adherence to SDLC best practices


About you:

  • Bring 5+ years of experience in data engineering, ingestion pipelining, Data Streaming using Kafka and ETL/ELTHold a bachelor’s degree in computer science, engineering, statistics, or related field
  • Have hands-on experience with and understanding of the following:
  • Kafka ( minimum 2 years)AWS (minimum 2 years)Python/PySpark (minimum 3 years)Spark/ScalaDatabricksUnity CatalogSQL
  • ETL/ELT development, monitoring and pipelining using tools such as Apache Airflow
  • Ingestion tools such as Dell Boomi
  • Data quality guidelines
  • CI/CD pipelines
  • Agile
  • Git and version control


"PySpark" AND "kafka" AND "aws" AND "databricks" AND "data ingestion" AND "python" AND "unity catalog" AND "data streaming"



  • Dallas, Texas, United States Tiva's Client Full time

    Data Ingestion Platform SpecialistAbout the Role:This role will be responsible for designing and implementing data ingestion solutions on the Databricks platform. The ideal candidate will have a deep understanding of data streaming, ETL/ELT implementation, and CI/CD.Key Responsibilities:Ingest data from various source systems and tailor ingestion approaches...


  • Dallas, United States InfoVision Inc. Full time

    Senior Data EngineerDallas, TXLong term contractJob Description:We are looking for a Data Engineer to join our Advanced Analytics team under the Wholesale organization. This engineer will design and deliver data solutions for the analytics and data science teams in the consumer organization. The engineer will be able to work with all types of interesting...


  • Dallas, Texas, United States Data Axle Full time

    Job SummaryData Axle is seeking a highly experienced Principal Data Engineer to lead the design, implementation, and optimization of our modern data architecture.This role requires a strong background in data engineering, expertise in Python and SQL programming, and a passion for creating scalable solutions to support business needs.The ideal candidate will...


  • Dallas, Texas, United States Abode Techzone LLC Full time

    Job Title:Senior Cloud Data Engineer - Air flowLocation: RemoteProject Duration: 12 months ContractInterview: Three rounds of Video interviewJob Description :Primary Skills:Air Flow, Big Query, Data Proc , Spark), Hadoop. Lead the Team. E2E DeliverySecondary Skills:Java, Unix Scripts, Oozie, Google Cloud Big Query & Big Table, Google Data Proc, Google Data...

  • Senior Data Engineer

    2 months ago


    dallas, United States TopShelf Talent Group Full time

    Our client is a well-known investment management firm in the Dallas area and they're seeking a Senior Data Engineer to join their team.They need a Senior Data Engineer with 8+ years of hands-on expertise in data engineering, with a focus on creating reliable, efficient, and scalable systems. This role is pivotal in shaping the data architecture and pipelines...

  • Senior Data Engineer

    1 month ago


    dallas, United States TopShelf Talent Group Full time

    Our client is a well-known investment management firm in the Dallas area and they're seeking a Senior Data Engineer to join their team.They need a Senior Data Engineer with 8+ years of hands-on expertise in data engineering, with a focus on creating reliable, efficient, and scalable systems. This role is pivotal in shaping the data architecture and pipelines...

  • Senior Data Engineer

    2 months ago


    Dallas, United States TopShelf Talent Group Full time

    Our client is a well-known investment management firm in the Dallas area and they're seeking a Senior Data Engineer to join their team.They need a Senior Data Engineer with 8+ years of hands-on expertise in data engineering, with a focus on creating reliable, efficient, and scalable systems. This role is pivotal in shaping the data architecture and pipelines...

  • Senior Scrum Master

    3 weeks ago


    dallas, United States Tiva's Client Full time

    Senior Scrum Master (Data Analytics and Data Engineering Teams)We are looking for a Senior Scrum Master who has driven projects/programs that are data heavy projects around Data Analytics, Data Engineering, Data Curation, Data Ingestion, Cloud Data etc because they are supporting Data Engineering and Data Analytics Team.ResponsibilitiesFacilitating daily...

  • Senior Scrum Master

    3 weeks ago


    Dallas, United States Tiva's Client Full time

    Senior Scrum Master (Data Analytics and Data Engineering Teams)We are looking for a Senior Scrum Master who has driven projects/programs that are data heavy projects around Data Analytics, Data Engineering, Data Curation, Data Ingestion, Cloud Data etc because they are supporting Data Engineering and Data Analytics Team.ResponsibilitiesFacilitating daily...

  • Senior Scrum Master

    2 weeks ago


    dallas, United States Tiva's Client Full time

    Senior Scrum Master (Data Analytics and Data Engineering Teams)We are looking for a Senior Scrum Master who has driven projects/programs that are data heavy projects around Data Analytics, Data Engineering, Data Curation, Data Ingestion, Cloud Data etc because they are supporting Data Engineering and Data Analytics Team.ResponsibilitiesFacilitating daily...

  • Senior Data Engineer

    1 month ago


    Dallas, Texas, United States Global Channel Management Full time

    About the JobGlobal Channel Management is seeking a highly skilled Senior Data Engineer - Cloud Architecture to lead our data engineering efforts in consumer finance.The ideal candidate will have 7+ years of experience as a data engineer in consumer finance or equivalent industry, with a strong background in math, statistics, computer science, data science,...

  • Senior Data Architect

    4 weeks ago


    Dallas, Texas, United States Optimize Search Group Full time

    Job Title: Senior Data ArchitectAt Optimize Search Group, we are seeking a highly skilled and experienced Senior Data Architect to lead the design and development of our data platforms. The ideal candidate will have a strong background in data engineering, data warehousing, and cloud platforms. This role will involve leading the design and development of...

  • Lead Data Engineer

    1 month ago


    Dallas, Texas, United States Alcority Full time

    Job Title: Lead Data EngineerWe are seeking a highly skilled and experienced Lead Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, data warehousing, and cloud platforms.This role will involve leading the design and development of robust data pipelines, streaming ingestion frameworks, and scalable...


  • dallas, United States InfoVision Inc. Full time

    Senior Data EngineerDallas, TXLong term contractJob Description:We are looking for a Data Engineer to join our Advanced Analytics team under the Wholesale organization. This engineer will design and deliver data solutions for the analytics and data science teams in the consumer organization. The engineer will be able to work with all types of interesting...


  • dallas, United States Tiva's Client Full time

    Senior Technical Data StewardRemote Contract RoleRole Overview:This role will be responsible for translating business needs into technical requirements, managing logical data models, and working closely with ingestion and curation teams. This role requires a deep understanding of:Logical data modelingComplex metadata managementDeveloping, managing, and...


  • dallas, United States Tiva's Client Full time

    Senior Technical Data StewardRemote Contract RoleRole Overview:This role will be responsible for translating business needs into technical requirements, managing logical data models, and working closely with ingestion and curation teams. This role requires a deep understanding of:Logical data modelingComplex metadata managementDeveloping, managing, and...

  • Lead Data Engineer

    2 weeks ago


    Dallas, United States INSPYR Solutions Full time

    Title: Lead Data Engineer Location: Irving, TX 75039 (Hybrid) Duration: Direct Hire Compensation: $175K - $200K Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S. Skillset / Experience: We are seeking a highly skilled and experienced Lead Data Engineer to join our dynamic team. The ideal candidate will have a strong background in...


  • Dallas, United States Tiva's Client Full time

    Senior Technical Data StewardRemote Contract RoleRole Overview:This role will be responsible for translating business needs into technical requirements, managing logical data models, and working closely with ingestion and curation teams. This role requires a deep understanding of:Logical data modelingComplex metadata managementDeveloping, managing, and...

  • Senior Data Engineer

    2 months ago


    Dallas, United States Anblicks Full time

    The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data solutions....

  • Senior Data Engineer

    1 month ago


    dallas, United States Anblicks Full time

    The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data solutions....