Cloud Data Engineer

3 weeks ago

Irving, Texas, United States Diverse Lynx Full time
Role: Cloud Data Engineer

Location: Minneapolis, MN or Irving, TX (3+ days in office/Week)

Mode of employment: Full-Time

Job Description:

Position Overview

We are seeking a Lead Developer with advanced skills in Python, Apache Spark, Azure Synapse, and Azure Data Engineering services to develop and manage ETL pipelines and data processing solutions that support AI/ML initiatives. The ideal candidate will have strong expertise in Azure Cloud, including SQL, Data Factory, SQL Pools, Spark Pools, and Data Warehousing. A background in CI/CD processes, cross-functional collaboration, and methodologies is essential. In addition, knowledge of Data Science, Java, Kubernetes, Azure Data Lake Storage (ADLS) Gen2, and API development is required.

Key Responsibilities

1. ETL and Data Pipeline Development
  • Design, develop, and optimize scalable ETL processes using Python, Apache Spark, and Azure Synapse.
  • Build and manage Azure Data Factory pipelines to orchestrate complex data workflows.
  • Use SQL Pools and Spark Pools within Synapse to manage and process large datasets efficiently.
  • Implement Data Warehousing solutions using Azure Synapse Analytics to provide structured and queryable data layers.
  • Ensure the data platform supports real-time and batch AI/ML data requirements.
2. Azure Cloud Development & CI/CD Deployment
  • Build, configure, and manage CI/CD pipelines on Azure DevOps for ETL and data processing tasks.
  • Automate infrastructure provisioning, testing, and deployment using Infrastructure-as-Code (IaC) tools like ARM templates or Terraform.
  • Optimize Azure Data Lake Storage (ADLS Gen2) to store and manage raw and processed data efficiently, ensuring proper access control and data security.
3. Cross-Functional Collaboration
  • Collaborate with Data Scientists, Data Engineers, ML Engineers, and Business Analysts to translate business requirements into data solutions.
  • Work with the DevOps and Security teams to ensure smooth and secure deployment of applications and pipelines.
  • Act as the technical lead in designing, developing, and implementing data solutions, mentoring junior team members.
4. Data Engineering and API Development
  • Develop and integrate with external and internal APIs for data ingestion and data exchange.
  • Build, test, and deploy RESTful APIs for secure data access.
  • Use Kubernetes for containerizing and deploying data processing applications.
  • Manage data storage and transformation to support advanced Data Science and AI/ML models.
5. Agile Project Management
  • Participate in and lead Agile ceremonies, such as sprint planning, daily stand-ups, and retrospectives.
  • Collaborate with cross-functional teams in iterative development to ensure high-quality and timely feature delivery.
  • Adapt to changing project priorities and business needs in an Agile environment.
Required Skills and Qualifications

1. Technical Skills:
  • Expertise in Python and Apache Spark for large-scale data processing.
  • Strong experience in Azure Synapse Analytics, including SQL Pools and Spark Pools.
  • Advanced proficiency in Azure Data Factory for ETL pipeline orchestration and management.
  • Knowledge of Data Warehousing principles, with hands-on experience building solutions on Azure.
  • Experience with SQL, including complex queries, optimization, and performance tuning.
  • Familiarity with CI/CD tools like Azure DevOps and managing infrastructure in Azure Cloud.
  • Experience in Java for API integration and microservices architecture.
  • Hands-on knowledge of Kubernetes for containerized data processing environments.
  • Proficiency in working with Azure Data Lake Storage (ADLS) Gen2 for data storage and management.
  • Experience working with APIs (REST, SOAP) and building API-based data integrations.
2. Agile and Cross-Functional Skills:
  • Experience working in an Agile environment, using Scrum or Kanban.
  • Ability to lead, mentor, and coach junior developers in the team.
  • Strong collaboration skills to work with data scientists, analysts, and cross-functional teams to deliver end-to-end data solutions.
3. Behavioral Skills:
  • Strong analytical and problem-solving skills with a passion for data-driven solutions.
  • Excellent communication and presentation skills, able to explain complex technical concepts to non-technical stakeholders.
  • Ability to work in a fast-paced, dynamic environment with changing priorities.
  • Self-motivated and results-oriented with attention to detail.
Preferred Qualifications
  • Azure certifications in data engineering or cloud architecture.
  • Experience deploying AI/ML models on cloud platforms.
  • Familiarity with Data Governance best practices, ensuring compliance with data privacy regulations
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.

  • Cloud Data Engineer

    10 hours ago

    Irving, Texas, United States Experis Full time

    Job SummaryWe are looking for a Cloud Data Engineer to join our team. As a Cloud Data Engineer, you will be responsible for designing and implementing data pipelines for reporting and other downstream applications using open-source software and cloud technologies.Key ResponsibilitiesDesign and architect data pipelines for reporting and other downstream...

  • Cloud Data Engineer

    15 hours ago

    Irving, Texas, United States Motion Recruitment Partners LLC Full time

    Job OverviewMotion Recruitment Partners LLC is seeking a skilled Cloud Data Engineer to join our team. This is a fantastic opportunity for an experienced professional to work on complex data projects and contribute to the growth of our organization.We are looking for a highly motivated individual who has a strong background in data engineering, cloud...

  • Cloud Data Engineering Specialist

    3 days ago

    Irving, Texas, United States Abode Techzone LLC Full time

    Job Title: Cloud Data Engineering SpecialistWe are seeking an experienced Cloud Data Engineering Specialist to join our team at Abode Techzone, LLC. As a key member of our team, you will be responsible for designing and developing large-scale data processing systems on the Google Cloud Platform.The ideal candidate will have a strong background in cloud-based...

  • Cloud Data Engineer

    3 days ago

    Irving, Texas, United States TechnoGen Full time

    At TechnoGen, we're looking for a talented Cloud Data Engineer to join our team. With 10+ years of experience in data engineering and a strong background in Java, you'll play a crucial role in designing and implementing scalable data pipelines using Google Cloud Platform (GCP).About the Role:Design, build, and maintain large-scale data processing systems on...

  • Cloud Data Engineer

    2 days ago

    Irving, Texas, United States Experis Full time

    **About the Role:**We are seeking a skilled Data Engineer to design and develop data pipelines for reporting and other downstream applications using open-source software and cloud technologies.The ideal candidate will have expertise in SQL, Python/Spark, and data pipelining tools, with proficiency in ETL/ELT and well-versed in handling data structures...

  • Cloud Data Engineer

    8 hours ago

    Irving, Texas, United States Manpower Group Inc. Full time

    Manpower Group Inc. is seeking a highly skilled Cloud Data Engineer to join their team.The ideal candidate will have expertise in designing and implementing cloud-based data solutions.Key Responsibilities:Design and architect data pipelines for reporting and other downstream applications using open-source software and cloud technologies.Develop and maintain...

  • Cloud Architect

    3 days ago

    Irving, Texas, United States Technogen International Company Full time

    Cloud Architect - Data EngineeringWe are seeking a highly skilled Cloud Architect to lead our Data Engineering team in the development and maintenance of scalable data pipelines on AWS.The ideal candidate will have a deep understanding of cloud computing and data engineering principles, as well as hands-on experience with AWS services.About the JobLead a...

  • Cloud Data Engineering Specialist

    3 days ago

    Irving, Texas, United States Resource Informatics Group Full time

    Job OverviewThe Resource Informatics Group is seeking a skilled Data Engineer to join our team. As a Data Engineer, you will be responsible for building data systems and pipelines on cloud providers, specifically Google Cloud Platform (GCP).Your primary focus will be on designing and implementing efficient ETL/ELT pipelines using tools such as Big Query,...

  • Data Cloud Engineer

    13 hours ago

    Irving, Texas, United States Motion Recruitment Full time

    Company OverviewMotion Recruitment Partners is a well-established recruitment agency that specializes in connecting top talent with leading financial services companies. Our client is one of the largest and most respected institutions in the world, known for its commitment to innovation and customer satisfaction.Job DescriptionWe are seeking an experienced...

  • ETL / ELT Data Engineer / Cloud / Big Data

    3 days ago

    Irving, Texas, United States Motion Recruitment Partners, LLC Full time

    Outstanding long-term contract opportunity A well-known Financial Services Company is looking for a ETL / ELT Data Engineer - Cloud & Big Data in Irving, TX (Hybrid). Work with the brightest minds at one of the largest financial institutions in the world. This is a long-term contract opportunity that includes a competitive benefit package Our client has...

  • Data Engineer

    7 days ago

    Irving, Texas, United States S Tekwizards LLC Full time

    Job OverviewS Tekwizards LLC is seeking a highly skilled Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining scalable data pipelines using Snowflake, dbt, and other data transformation tools.The ideal candidate will have a strong understanding of data warehousing...

  • Cloud Data Engineering Manager

    1 hour ago

    Irving, Texas, United States Motion Recruitment Partners, LLC Full time

    Our Ideal Candidate: To excel in this role, you will require 6+ years of experience in data engineering & analytics, with 4+ years of experience working with both Azure and Azure Data Factory (ADF). A strong fundamental knowledge of SQL is essential, along with relevant certifications or a Bachelor's degree in CS or analytics. Desired skills include Azure...

  • Cloud and Big Data Solutions Engineer

    6 hours ago

    Irving, Texas, United States Motion Recruitment Full time

    Job SummaryWe are seeking an experienced Cloud and Big Data Solutions Engineer to join our client's team in Irving, TX. As a key member of the software engineering team, you will be responsible for designing, architecting, and implementing data pipelines for reporting and downstream applications using open-source software and cloud platforms.The ideal...

  • Senior Software Engineer

    9 hours ago

    Irving, Texas, United States Motion Recruitment Full time

    About the RoleWe are looking for a skilled Senior Software Engineer - Cloud Data to join our client's team in Irving, TX. As a key member of the software engineering team, you will be responsible for designing, architecting, and implementing data pipelines for reporting and downstream applications using open-source software and cloud platforms.The ideal...

  • ETL / ELT Data Engineer / Cloud / Big Data

    17 hours ago

    Irving, Texas, United States Motion Recruitment Partners LLC Full time

    ETL / ELT Data Engineer / Cloud / Big DataIrving, TXHybridContract$53.56/hr - $60.35/hrOutstanding long-term contract opportunity A well-known Financial Services Company is looking for a ETL / ELT Data Engineer – Cloud & Big Data in Irving, TX (Hybrid).Work with the brightest minds at one of the largest financial institutions in the world. This is a...

  • Data Engineer

    1 week ago

    Irving, Texas, United States NTT DATA Full time

    Req ID: 317685 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer - Irving, Texas to join our team in Irving, Texas (US-TX), United States (US). NTT Data Americas is...

  • ETL / ELT Data Engineer / Cloud & Big Data

    16 hours ago

    Irving, Texas, United States Motion Recruitment Full time

    Outstanding long-term contract opportunity A well-known Financial Services Company is looking for a ETL / ELT Data Engineer - Cloud & Big Data in Irving, TX (Hybrid). Work with the brightest minds at one of the largest financial institutions in the world. This is a long-term contract opportunity that includes a competitive benefit package Our client has...

  • Cloud Data Specialist

    2 days ago

    Irving, Texas, United States Resource Informatics Group Full time

    **Role Summary**We are looking for a skilled Cloud Data Specialist to join our team at Resource Informatics Group. As a key member of our engineering team, you will play a crucial role in designing and implementing cloud-based data solutions.**Key Responsibilities:**Provide support for and own enterprise integrations (i.e., Salesforce, AWS Cognito,...

  • Data Pipeline Architect in the Cloud

    2 days ago

    Irving, Texas, United States Motion Recruitment Partners, LLC Full time

    **Company Overview**Our client is a well-established financial institution that has been around for over 150 years. They continuously innovate in today's digital age and offer a long-term contract opportunity for an experienced ETL/ELT Data Engineer - Cloud & Big Data.About the RoleThe successful candidate will be responsible for designing, architecting data...

  • ETL ELT Data Engineer

    2 hours ago

    Irving, Texas, United States Motion Recruitment Full time

    About the JobWe are looking for a highly skilled ETL ELT Data Engineer - Cloud & Big Data Specialist to join our client's team in Irving, TX. As a key member of the software engineering team, you will be responsible for designing, architecting, and implementing data pipelines for reporting and downstream applications using open-source software and cloud...