Data Engineer

10 hours ago

Irving, United States Talent Groups Full time

Role: Cloud Data Engineer (Python, Spark, Synapse, Azure storage) for ETL and Data Processing in AI/ML

Location: Minneapolis, MN or Irving, TX (3+ days in office/Week)


Yrs. of experience: 10+ Yrs.



Job Description:

Position Overview

We are seeking a Lead Developer with advanced skills in Python, Apache Spark, Azure Synapse, and Azure Data Engineering services to develop and

manage ETL pipelines and data processing solutions that support AI/ML initiatives. The ideal candidate will have strong expertise in Azure Cloud,

including SQL, Data Factory, SQL Pools, Spark Pools, and Data Warehousing. A background in CI/CD processes, cross-functional collaboration, and

methodologies is essential. In addition, knowledge of Data Science, Java, Kubernetes, Azure Data Lake Storage (ADLS) Gen2, and API development is required.


Key Responsibilities


1. ETL and Data Pipeline Development

- Design, develop, and optimize scalable ETL processes using Python, Apache Spark, and Azure Synapse.

- Build and manage Azure Data Factory pipelines to orchestrate complex data workflows.

- Use SQL Pools and Spark Pools within Synapse to manage and process large datasets efficiently.

- Implement Data Warehousing solutions using Azure Synapse Analytics to provide structured and queryable data layers.

- Ensure the data platform supports real-time and batch AI/ML data requirements.


2. Azure Cloud Development & CI/CD Deployment

- Build, configure, and manage CI/CD pipelines on Azure DevOps for ETL and data processing tasks.

- Automate infrastructure provisioning, testing, and deployment using Infrastructure-as-Code (IaC) tools like ARM templates or Terraform.

- Optimize Azure Data Lake Storage (ADLS Gen2) to store and manage raw and processed data efficiently, ensuring proper access control and data security.


3. Cross-Functional Collaboration

- Collaborate with Data Scientists, Data Engineers, ML Engineers, and Business Analysts to translate business requirements into data solutions.

- Work with the DevOps and Security teams to ensure smooth and secure deployment of applications and pipelines.

- Act as the technical lead in designing, developing, and implementing data solutions, mentoring junior team members.


4. Data Engineering and API Development

- Develop and integrate with external and internal APIs for data ingestion and data exchange.

- Build, test, and deploy RESTful APIs for secure data access.

- Use Kubernetes for containerizing and deploying data processing applications.

- Manage data storage and transformation to support advanced Data Science and AI/ML models.


5. Agile Project Management

- Participate in and lead Agile ceremonies, such as sprint planning, daily stand-ups, and retrospectives.

- Collaborate with cross-functional teams in iterative development to ensure high-quality and timely feature delivery.

- Adapt to changing project priorities and business needs in an Agile environment.


Required Skills and Qualifications


1. Technical Skills:

- Expertise in Python and Apache Spark for large-scale data processing.

- Strong experience in Azure Synapse Analytics, including SQL Pools and Spark Pools.

- Advanced proficiency in Azure Data Factory for ETL pipeline orchestration and management.

- Knowledge of Data Warehousing principles, with hands-on experience building solutions on Azure.

- Experience with SQL, including complex queries, optimization, and performance tuning.

- Familiarity with CI/CD tools like Azure DevOps and managing infrastructure in Azure Cloud.

- Experience in Java for API integration and microservices architecture.

- Hands-on knowledge of Kubernetes for containerized data processing environments.

- Proficiency in working with Azure Data Lake Storage (ADLS) Gen2 for data storage and management.

- Experience working with APIs (REST, SOAP) and building API-based data integrations.


2. Agile and Cross-Functional Skills:

- Experience working in an Agile environment, using Scrum or Kanban.

- Ability to lead, mentor, and coach junior developers in the team.

- Strong collaboration skills to work with data scientists, analysts, and cross-functional teams to deliver end-to-end data solutions.


3. Behavioral Skills:

- Strong analytical and problem-solving skills with a passion for data-driven solutions.

- Excellent communication and presentation skills, able to explain complex technical concepts to non-technical stakeholders.

- Ability to work in a fast-paced, dynamic environment with changing priorities.

- Self-motivated and results-oriented with attention to detail.


Preferred Qualifications

- Azure certifications in data engineering or cloud architecture.

- Experience deploying AI/ML models on cloud platforms.

- Familiarity with Data Governance best practices, ensuring compliance with data privacy regulations

  • Data engineer

    7 months ago

    Irving, United States CyberTec Full time

    Client: Client, project implemented for Sonepar (Reminder- Do not post client info online) Data Engineer US Citizen ONLY Rate options: $75/hr w2 + benefits + $7/hr referral fee to you OR $ 8 0/hr c2c AIC Expected Start Date ASAP End Date (if applicable): 12/31/24 Location: Irving, on-site 3x per week - locals to DFW only at this time. Interview Process:...

  • Data Engineer

    3 weeks ago

    Irving, Texas, United States Diverse Lynx Full time

    We are seeking a highly skilled Data Engineer to join our team at Diverse Lynx LLC. As a key member of our technical staff, you will be responsible for designing, implementing, and maintaining large-scale data processing systems using Apache Spark.About the Role:The successful candidate will have a minimum of 6-8 years of experience in big data engineering...

  • Data Engineer

    3 weeks ago

    Irving, United States Synechron Full time

    We areAt Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm combines creativity and innovative technology to deliver industry-leading digital solutions. Synechron's progressive technologies and optimization strategies span end-to-end Artificial Intelligence, Consulting, Digital, Cloud & DevOps,...

  • Data Engineer

    1 month ago

    Irving, United States Synechron Full time

    We areAt Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm combines creativity and innovative technology to deliver industry-leading digital solutions. Synechron’s progressive technologies and optimization strategies span end-to-end Artificial Intelligence, Consulting, Digital, Cloud & DevOps,...

  • Data Engineer

    12 hours ago

    Irving, United States ExecRecruitment Full time

    Job DescriptionJob DescriptionExecRecruitment is a global professional services provider and contingency staffing company. Our main objective is to source top talent and support professional growth.One of our direct clients is actively seeking a Data Engineer to join their team.Job Title: Data EngineerLocation: RemoteDuration: 6 months+ extensionThe ideal...

  • Data Engineer Position

    2 weeks ago

    Irving, Texas, United States Resource Informatics Group Full time

    Job Title: Data Engineer PositionAbout Us: Resource Informatics Group is a leading provider of data engineering services. We work with clients to design and implement data pipelines that meet their business needs.Estimated Salary: $140,000 - $180,000 per year depending on experience.Description: We are seeking an experienced Data Engineer to join our team....

  • Data Engineer

    4 weeks ago

    Irving, United States TEKsystems Full time

    Description One of TEKsystems's customers, a Fortune 500 organization and the nation's leader in their respective industry, is looking for a Data Engineer. A successful candidate must have a minimum of 3-5 years of industry experience in design, development, and implementation of cloud and hybrid data solutions that meets the client's business needs. A Data...

  • Data Architect and Engineer

    4 weeks ago

    Irving, Texas, United States Synechron Full time

    About the Role">This Senior Technical Lead position is responsible for leading the engineering team for data cloud technologies and designing solutions for AWS, Snowflake, and other cloud platforms. The ideal candidate will have a strong background in cloud computing, data engineering, and data architecture, with experience in Snowflake, GCP, and Cloudera...

  • Data Engineer

    5 months ago

    Irving, United States FusionHealth Full time

    Job Description Job Description The Data Engineer will work cross-functionally with application development, data science, and product management to support growth, strategy, and development of innovative solutions internally and for our external clients. This position will work with operational and business leaders to deliver business intelligence and...

  • data engineer

    5 months ago

    Irving, United States Resource Informatics Group Full time

    Role: Data Engineer Location: Hybrid onsite (Mclean, VA) Duration: 6-12 Months - (With possible extension) Must Haves Overall exp of 5 years SQL - 3 years Python (building data pipeline) - 3 years Spark - 3 years Big Data/Hadoop Unix/Shell Scripting Autosys

  • Big Data Engineer

    3 months ago

    Irving, United States Newt Global Full time

    Greeting from Newt Global LLC!!!We are currently looking for Big Data Engineer to join our team for long term project with our Banking client. Interested with the below job description, please share your resume with expected rate/hr. Big Data Engineer Irving, TX (Hybrid – 2 Days a Week)Contract Type: W2 or SCorp (No Third Party Resume)Long TermExp: 8+ yrs...

  • Big Data Engineer

    2 weeks ago

    Irving, United States Newt Global Full time

    Greeting from Newt Global LLC!!!We are currently looking for Big Data Engineer to join our team for long term project with our Banking client. Interested with the below job description, please share your resume with expected rate/hr. Big Data Engineer Irving, TX (Hybrid – 2 Days a Week)Contract Type: W2 or SCorp (No Third Party Resume)Long TermExp: 8+ yrs...

  • Big Data Engineer

    7 days ago

    Irving, United States Newt Global Full time

    Greeting from Newt Global LLC!!!We are currently looking for Big Data Engineer to join our team for long term project with our Banking client. Interested with the below job description, please share your resume with expected rate/hr. Big Data Engineer Irving, TX (Hybrid – 2 Days a Week)Contract Type: W2 or SCorp (No Third Party Resume)Long TermExp: 8+ yrs...

  • Data Engineering Specialist

    2 weeks ago

    Irving, Texas, United States Resource Informatics Group Full time

    Job Title: Data Engineering SpecialistWe are looking for a skilled Data Engineer to join our team at the Resource Informatics Group in Boston MA, Durham NC, or work remotely in a hybrid environment. The ideal candidate will have experience with cloud-based data solutions and a strong background in software development.Key Responsibilities:- Develop...

  • Big Data Engineer

    2 weeks ago

    Irving, Texas, United States Citigroup Inc Full time

    Citigroup Inc. is a leading global bank with operations in over 160 countries and jurisdictions, serving more than 200 million customer accounts.The company's core activities include safeguarding assets, lending money, making payments, and accessing the capital markets on behalf of its clients.This Senior Data Engineer position is responsible for building...

  • Data Engineering Lead

    3 weeks ago

    Irving, Texas, United States ShiftCode Analytics Full time

    Job Description:At ShiftCode Analytics, we're looking for a seasoned Senior Data Engineer to join our team. As a key member of our data engineering team, you'll be responsible for designing, developing, and delivering large-scale data solutions.Key Responsibilities:Design and implement complex data pipelines using various technologies.Develop and maintain...

  • Data Engineer

    7 months ago

    Irving, United States Photon Full time

    Job DescriptionData Engineer - Graph Database Specialist (Neo4j) Design, develop, and maintain graph databases using Neo4j. Collaborate with stakeholders to understand data requirements and translate them into database schemas. Strong understanding of graph database concepts, data modeling, and query optimization. Optimize database performance for efficient...

  • Cloud Data Engineer

    1 month ago

    Irving, United States Diverse Lynx Full time

    Role: Azure Data Engineer Location: Downtown Minneapolis (Day 1 Onsite) Full-Time Key Responsibilities: Position Overview We are seeking a Lead Developer with advanced skills in Python, Apache Spark, Azure Synapse, and Azure Data Engineering services to develop and manage ETL pipelines and data processing solutions that support AI/ML initiatives. The ideal...

  • Senior Data Engineer

    7 months ago

    Irving, United States OTSI Full time

    Job Description Object Technology Solutions, Inc (OTSI) has an immediate opening for a Senior Data Engineer. Senior Data Engineer (Irving, Tx) JOB DESCRIPTION: • Analyze and understand data sources & APIs • Design and Develop methods to connect & collect data from different data sources • Design and Develop methods to filter/cleanse the data • Design...

  • Data Engineering Expert

    1 month ago

    Irving, Texas, United States OTSI Full time

    Senior Data Engineer JobAt OTSI, we are seeking a skilled Senior Data Engineer to join our team. The ideal candidate will have experience in designing and developing methods to connect and collect data from various sources.Key Responsibilities:Analyze and understand data sources and APIsDesign and develop methods to connect and collect data from different...