Senior Data Ingestion Engineer
1 month ago
Senior Data Ingestion Engineer
Most important for this role (key skills)
- Kafka (data streaming)
- Python/PySparkAWS
- Databricks Cloud Platform
- Unity Catalog
Role Overview:
This role will be responsible for conducting data ingestion and platform management on the Databricks platform. This role requires a deep understanding of:
- Data Lake ingestion processes and best practices
- Data Streaming (Kafka)
- ETL/ELT implementation
- CI/CD
- System integration tools
- Data pipeline management
About the role:
- Ingest data from a variety of source systems and tailor ingestion approaches on a per-system basis
- Manage, maintain, and oversee ETL/ELT pipelines on the Databricks platformOptimize data pipelines for scalability and speed
- Document ingestion and integration flows and pipelines
- Use Airflow to schedule and automate ingestion jobs
- Manage metadata and master data in technical data catalog
- Ensure ELT/ETL design meets required security and compliance guidelines, and ensure PII management, flagging and risk assessment during ingestion
- Maintain ETL/ELT pipeline infrastructure and implement automated monitoring strategies
- Ensure adherence to SDLC best practices
About you:
- Bring 5+ years of experience in data engineering, ingestion pipelining, Data Streaming using Kafka and ETL/ELTHold a bachelor’s degree in computer science, engineering, statistics, or related field
- Have hands-on experience with and understanding of the following:
- Kafka ( minimum 2 years)AWS (minimum 2 years)Python/PySpark (minimum 3 years)Spark/ScalaDatabricksUnity CatalogSQL
- ETL/ELT development, monitoring and pipelining using tools such as Apache Airflow
- Ingestion tools such as Dell Boomi
- Data quality guidelines
- CI/CD pipelines
- Agile
- Git and version control
"PySpark" AND "kafka" AND "aws" AND "databricks" AND "data ingestion" AND "python" AND "unity catalog" AND "data streaming"
-
Data Ingestion Platform Specialist
4 weeks ago
Dallas, Texas, United States Tiva's Client Full timeData Ingestion Platform SpecialistAbout the Role:This role will be responsible for designing and implementing data ingestion solutions on the Databricks platform. The ideal candidate will have a deep understanding of data streaming, ETL/ELT implementation, and CI/CD.Key Responsibilities:Ingest data from various source systems and tailor ingestion approaches...
-
Senior Data Engineer
2 days ago
Dallas, United States InfoVision Inc. Full timeSenior Data EngineerDallas, TXLong term contractJob Description:We are looking for a Data Engineer to join our Advanced Analytics team under the Wholesale organization. This engineer will design and deliver data solutions for the analytics and data science teams in the consumer organization. The engineer will be able to work with all types of interesting...
-
Data Science Engineering Lead
4 weeks ago
Dallas, Texas, United States Data Axle Full timeJob SummaryData Axle is seeking a highly experienced Principal Data Engineer to lead the design, implementation, and optimization of our modern data architecture.This role requires a strong background in data engineering, expertise in Python and SQL programming, and a passion for creating scalable solutions to support business needs.The ideal candidate will...
-
Senior Cloud Data Engineer
4 weeks ago
Dallas, Texas, United States Abode Techzone LLC Full timeJob Title:Senior Cloud Data Engineer - Air flowLocation: RemoteProject Duration: 12 months ContractInterview: Three rounds of Video interviewJob Description :Primary Skills:Air Flow, Big Query, Data Proc , Spark), Hadoop. Lead the Team. E2E DeliverySecondary Skills:Java, Unix Scripts, Oozie, Google Cloud Big Query & Big Table, Google Data Proc, Google Data...
-
Senior Data Engineer
2 months ago
dallas, United States TopShelf Talent Group Full timeOur client is a well-known investment management firm in the Dallas area and they're seeking a Senior Data Engineer to join their team.They need a Senior Data Engineer with 8+ years of hands-on expertise in data engineering, with a focus on creating reliable, efficient, and scalable systems. This role is pivotal in shaping the data architecture and pipelines...
-
Senior Data Engineer
1 month ago
dallas, United States TopShelf Talent Group Full timeOur client is a well-known investment management firm in the Dallas area and they're seeking a Senior Data Engineer to join their team.They need a Senior Data Engineer with 8+ years of hands-on expertise in data engineering, with a focus on creating reliable, efficient, and scalable systems. This role is pivotal in shaping the data architecture and pipelines...
-
Senior Data Engineer
2 months ago
Dallas, United States TopShelf Talent Group Full timeOur client is a well-known investment management firm in the Dallas area and they're seeking a Senior Data Engineer to join their team.They need a Senior Data Engineer with 8+ years of hands-on expertise in data engineering, with a focus on creating reliable, efficient, and scalable systems. This role is pivotal in shaping the data architecture and pipelines...
-
Senior Scrum Master
3 weeks ago
dallas, United States Tiva's Client Full timeSenior Scrum Master (Data Analytics and Data Engineering Teams)We are looking for a Senior Scrum Master who has driven projects/programs that are data heavy projects around Data Analytics, Data Engineering, Data Curation, Data Ingestion, Cloud Data etc because they are supporting Data Engineering and Data Analytics Team.ResponsibilitiesFacilitating daily...
-
Senior Scrum Master
3 weeks ago
Dallas, United States Tiva's Client Full timeSenior Scrum Master (Data Analytics and Data Engineering Teams)We are looking for a Senior Scrum Master who has driven projects/programs that are data heavy projects around Data Analytics, Data Engineering, Data Curation, Data Ingestion, Cloud Data etc because they are supporting Data Engineering and Data Analytics Team.ResponsibilitiesFacilitating daily...
-
Senior Scrum Master
2 weeks ago
dallas, United States Tiva's Client Full timeSenior Scrum Master (Data Analytics and Data Engineering Teams)We are looking for a Senior Scrum Master who has driven projects/programs that are data heavy projects around Data Analytics, Data Engineering, Data Curation, Data Ingestion, Cloud Data etc because they are supporting Data Engineering and Data Analytics Team.ResponsibilitiesFacilitating daily...
-
Senior Data Engineer
1 month ago
Dallas, Texas, United States Global Channel Management Full timeAbout the JobGlobal Channel Management is seeking a highly skilled Senior Data Engineer - Cloud Architecture to lead our data engineering efforts in consumer finance.The ideal candidate will have 7+ years of experience as a data engineer in consumer finance or equivalent industry, with a strong background in math, statistics, computer science, data science,...
-
Senior Data Architect
4 weeks ago
Dallas, Texas, United States Optimize Search Group Full timeJob Title: Senior Data ArchitectAt Optimize Search Group, we are seeking a highly skilled and experienced Senior Data Architect to lead the design and development of our data platforms. The ideal candidate will have a strong background in data engineering, data warehousing, and cloud platforms. This role will involve leading the design and development of...
-
Lead Data Engineer
1 month ago
Dallas, Texas, United States Alcority Full timeJob Title: Lead Data EngineerWe are seeking a highly skilled and experienced Lead Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, data warehousing, and cloud platforms.This role will involve leading the design and development of robust data pipelines, streaming ingestion frameworks, and scalable...
-
dallas, United States InfoVision Inc. Full timeSenior Data EngineerDallas, TXLong term contractJob Description:We are looking for a Data Engineer to join our Advanced Analytics team under the Wholesale organization. This engineer will design and deliver data solutions for the analytics and data science teams in the consumer organization. The engineer will be able to work with all types of interesting...
-
Senior Technical Data Steward
1 month ago
dallas, United States Tiva's Client Full timeSenior Technical Data StewardRemote Contract RoleRole Overview:This role will be responsible for translating business needs into technical requirements, managing logical data models, and working closely with ingestion and curation teams. This role requires a deep understanding of:Logical data modelingComplex metadata managementDeveloping, managing, and...
-
Senior Technical Data Steward
1 month ago
dallas, United States Tiva's Client Full timeSenior Technical Data StewardRemote Contract RoleRole Overview:This role will be responsible for translating business needs into technical requirements, managing logical data models, and working closely with ingestion and curation teams. This role requires a deep understanding of:Logical data modelingComplex metadata managementDeveloping, managing, and...
-
Lead Data Engineer
2 weeks ago
Dallas, United States INSPYR Solutions Full timeTitle: Lead Data Engineer Location: Irving, TX 75039 (Hybrid) Duration: Direct Hire Compensation: $175K - $200K Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S. Skillset / Experience: We are seeking a highly skilled and experienced Lead Data Engineer to join our dynamic team. The ideal candidate will have a strong background in...
-
Senior Technical Data Steward
2 months ago
Dallas, United States Tiva's Client Full timeSenior Technical Data StewardRemote Contract RoleRole Overview:This role will be responsible for translating business needs into technical requirements, managing logical data models, and working closely with ingestion and curation teams. This role requires a deep understanding of:Logical data modelingComplex metadata managementDeveloping, managing, and...
-
Senior Data Engineer
2 months ago
Dallas, United States Anblicks Full timeThe Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data solutions....
-
Senior Data Engineer
1 month ago
dallas, United States Anblicks Full timeThe Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data solutions....