Azure Data Engineer

4 weeks ago

Minneapolis, United States Compunnel Inc. Full time

Role: Azure Data Engineer

Location: Downtown Minneapolis (Day-1 Onsite)

Duration: W2 Contract (12+Months)

Job responsibilities: Position Overview

We are seeking a Lead Developer with advanced skills in Python, Apache Spark, Azure Synapse, and Azure Data Engineering services to develop and manage ETL pipelines and data processing solutions that support AI/ML initiatives. The ideal candidate will have strong expertise in Azure Cloud, including SQL, Data Factory, SQL Pools, Spark Pools, and Data Warehousing. A background in CI/CD processes, cross-functional collaboration, and methodologies is essential. In addition, knowledge of Data Science, Java, Kubernetes, Azure Data Lake Storage (ADLS) Gen2, and API development is required.

Key Responsibilities

1. ETL and Data Pipeline Development

- Design, develop, and optimize scalable ETL processes using Python, Apache Spark, and Azure Synapse.

- Build and manage Azure Data Factory pipelines to orchestrate complex data workflows.

- Use SQL Pools and Spark Pools within Synapse to manage and process large datasets efficiently.

- Implement Data Warehousing solutions using Azure Synapse Analytics to provide structured and queryable data layers.

- Ensure the data platform supports real-time and batch AI/ML data requirements.

2. Azure Cloud Development & CI/CD Deployment

- Build, configure, and manage CI/CD pipelines on Azure DevOps for ETL and data processing tasks.

- Automate infrastructure provisioning, testing, and deployment using Infrastructure-as-Code (IaC) tools like ARM templates or Terraform.

- Optimize Azure Data Lake Storage (ADLS Gen2) to store and manage raw and processed data efficiently, ensuring proper access control and data security.

3. Cross-Functional Collaboration

- Collaborate with Data Scientists, Data Engineers, ML Engineers, and Business Analysts to translate business requirements into data solutions.

- Work with the DevOps and Security teams to ensure smooth and secure deployment of applications and pipelines.

- Act as the technical lead in designing, developing, and implementing data solutions, mentoring junior team members.

4. Data Engineering and API Development

- Develop and integrate with external and internal APIs for data ingestion and data exchange.

- Build, test, and deploy RESTful APIs for secure data access.

- Use Kubernetes for containerizing and deploying data processing applications.

- Manage data storage and transformation to support advanced Data Science and AI/ML models.

5. Agile Project Management

- Participate in and lead Agile ceremonies, such as sprint planning, daily stand-ups, and retrospectives.

- Collaborate with cross-functional teams in iterative development to ensure high-quality and timely feature delivery.

- Adapt to changing project priorities and business needs in an Agile environment.

Required Skills and Qualifications

1. Technical Skills:

- Expertise in Python and Apache Spark for large-scale data processing.

- Strong experience in Azure Synapse Analytics, including SQL Pools and Spark Pools.

- Advanced proficiency in Azure Data Factory for ETL pipeline orchestration and management.

- Knowledge of Data Warehousing principles, with hands-on experience building solutions on Azure.

- Experience with SQL, including complex queries, optimization, and performance tuning.

- Familiarity with CI/CD tools like Azure DevOps and managing infrastructure in Azure Cloud.

- Experience in Java for API integration and microservices architecture.

- Hands-on knowledge of Kubernetes for containerized data processing environments.

- Proficiency in working with Azure Data Lake Storage (ADLS) Gen2 for data storage and management.

- Experience working with APIs (REST, SOAP) and building API-based data integrations.

2. Agile and Cross-Functional Skills:

- Experience working in an Agile environment, using Scrum or Kanban.

- Ability to lead, mentor, and coach junior developers in the team.

- Strong collaboration skills to work with data scientists, analysts, and cross-functional teams to deliver end-to-end data solutions.

3. Behavioral Skills:

- Strong analytical and problem-solving skills with a passion for data-driven solutions.

- Excellent communication and presentation skills, able to explain complex technical concepts to non-technical stakeholders.

- Ability to work in a fast-paced, dynamic environment with changing priorities.

- Self-motivated and results-oriented with attention to detail.

Preferred Qualifications

- Azure certifications in data engineering or cloud architecture.

- Experience deploying AI/ML models on cloud platforms.

- Familiarity with Data Governance best practices, ensuring compliance with data privacy regulations

  • Azure Data Engineer

    1 month ago

    Minneapolis, United States Diverse Lynx Full time

    Role: Azure Data Engineer Location: Downtown Minneapolis,MN (Day 1 Onsite) Contract Position Overview We are seeking a Lead Developer with advanced skills in Python, Apache Spark, Azure Synapse, and Azure Data Engineering services to develop and manage ETL pipelines and data processing solutions that support AI/ML initiatives. The ideal candidate will have...

  • Azure Data Engineer

    2 weeks ago

    Minneapolis, United States Compunnel Full time

    Project Summary: This project involves building AC360 data pipelines in Azure using Spark technologies. As part of this initiative, the development of Zafin data pipelines in Azure is also included. Key Responsibilities: 1. Identify Requirements and Develop Solutions · Identify requirements for RMS replacement and develop solutions in Azure. ·...

  • Azure Data Engineer Specialist

    2 weeks ago

    Minneapolis, Minnesota, United States Diverse Lynx Full time

    Job OverviewDiverse Lynx LLC is a leading company in the field of data engineering, and we are currently seeking a highly skilled Azure Data Engineer Specialist to join our team. As a key member of our data engineering team, you will be responsible for designing, developing, and optimizing scalable ETL processes using Python, Apache Spark, and Azure Synapse....

  • Lead Azure Data Engineer

    1 month ago

    Minneapolis, United States Aloden, Inc. Full time

    Job Details: Lead Azure Data Engineer Duration: 12+ Month (s) Contract - 12m (possibility for extension or conversion) Location: - Hybrid Roles (3 Days Onsite/2 Days WFH) Note: ENTIRE TEAM IS OOO LAST 2 WEEKS OF THE YEAR INCLUDING CONTRACTORS - PLEASE MAKE SURE THEY ARE AWARE PRIOR TO SUBMISSION Job Descriptions: In this contingent resource assignment,...

  • Azure Data Scientist

    2 weeks ago

    Minneapolis, Minnesota, United States STIAOS Technologies Full time

    Job Summary: We are seeking an Azure Data Scientist to design, develop, and implement data engineering solutions using Azure cloud services. The estimated salary for this role is $160,000 per year.Key Responsibilities:• Develop and maintain data pipelines using Databricks (PySpark) for data ingestion, processing, and analysis.• Configure and manage Azure...

  • Senior Azure Data Engineer Lead

    3 weeks ago

    Minneapolis, Minnesota, United States Aloden, Inc. Full time

    Job DescriptionWe are seeking an experienced Azure Data Engineer Lead to join our team at Aloden, Inc. As a key member of our engineering team, you will be responsible for leading the design, development, and implementation of large-scale data solutions on Microsoft Azure.The ideal candidate will have a strong understanding of relational database concepts,...

  • Azure Data Engineer

    1 month ago

    Minneapolis, United States Futran Tech Solutions Pvt. Ltd. Full time

    Azure Data Engineer Minneapolis, MN ADF CICD Airflow SQL Cloud Databases Understanding of Agile methodologies 4-6 years of working experience in primary skills Overall 8-10 years in ETL/Data Engineering Understand business requirements. Understand source systems, source data and source data formats that are available on-prem / cloud. Design and build data...

  • Azure Data Architect

    2 weeks ago

    Minneapolis, Minnesota, United States Omni Inclusive Full time

    We are looking for a skilled Senior Data Engineer to join our team at Omni Inclusive. This role offers a competitive salary range of $220,000 - $280,000 per year and comes with various benefits.About the RoleThe successful candidate will have extensive experience in designing and implementing large-scale data engineering solutions using Azure services.Main...

  • Senior Azure Data Engineer Position

    3 weeks ago

    Minneapolis, Minnesota, United States Eateam Full time

    Job Description: We are looking for an experienced Data Engineer to join our team at Eateam. As a Sr. Data Engineer, you will play a key role in designing and implementing data pipelines that drive business growth. Your primary responsibilities will include developing scalable data pipelines using Azure Data Factory and Azure Databricks, implementing data...

  • Azure Engineer

    4 weeks ago

    Minneapolis, United States Saxon Global Full time

    This is a 12 month contract to hire role with CHS. 100% remote. USC/GC. Linkedin Must. Salary on conversion is 110-125K. A growing industry is looking for a contract or likely contract to hire Azure Engineer. The primary responsibilities will be to build and enable an Azure cloud environment including include responsibility for the design/planning,...

  • Senior Data Engineer

    1 month ago

    Minneapolis, United States Allianz Full time

    At Allianz Life, we are guided by our mission - we secure your future - and by our common goal of caring for tomorrow. We care for our employees by promoting an inclusive culture where everyone has the opportunity to grow and be rewarded for their success. We care for our customers by creating new products that help them protect their families and pursue...

  • Azure Data Solutions Director

    3 weeks ago

    Minneapolis, Minnesota, United States Aloden, Inc. Full time

    Key ResponsibilitiesThe Senior Azure Data Engineer Lead will be responsible for leading the design, development, and implementation of large-scale data solutions on Microsoft Azure, as well as collaborating with cross-functional teams to ensure smooth project delivery.The successful candidate will have a strong understanding of relational database concepts,...

  • Senior Azure Data Scientist

    7 days ago

    Minneapolis, Minnesota, United States Compunnel Full time

    About the RoleWe're looking for an exceptional Senior Azure Data Scientist to join our team at Compunnel. As a key member of our data science team, you'll be responsible for designing, developing, and deploying cutting-edge data analytics solutions using Azure services and machine learning algorithms.The ideal candidate will have a Ph.D. in Computer Science,...

  • Data Engineer

    2 weeks ago

    Minneapolis, United States STIAOS Technologies Full time

    Required : 8+ Years of hands on experience on below skillset 1. Snowflake 2. Azure Databricks 3. Azure Data Factory 4. Pyspark 5. Python 6. SQL Role and Responsibilities: Design, implement, and manage Azure cloud solutions for various projects. • Develop and maintain data pipelines using Databricks (PySpark) for data ingestion, processing, and analysis....

  • Azure Spark Engineering Specialist

    2 weeks ago

    Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    We are hiring a seasoned Azure Spark Engineering Specialist to join our team at Futran Tech Solutions Pvt. Ltd.The successful candidate will have extensive experience in designing, developing, and deploying scalable cloud-based data engineering solutions using Pyspark and Spark SQL programming.This role involves working closely with the development team to...

  • Compunnel Inc. | Azure Data Engineer

    4 weeks ago

    minneapolis, United States Compunnel Inc. Full time

    Role: Azure Data Engineer Location: Downtown Minneapolis (Day-1 Onsite)Duration: W2 Contract (12+Months) Job responsibilities: Position OverviewWe are seeking a Lead Developer with advanced skills in Python, Apache Spark, Azure Synapse, and Azure Data Engineering services to develop and manage ETL pipelines and data processing solutions that support AI/ML...

  • Data Engineer

    1 month ago

    Minneapolis, United States Eateam Full time

    Primary : SQL, Azure, Azure Data Factory, Azure DevOps, Azure DataBricks Snowflake, SQL Server, GitHub Secondary: Python, Pyspark Sr Data Engineer with expertise to design and create Data Pipelines in Azure Cloudusing Azure Data factory, ADLS, Azure Databricks, Log Analytics , Application Insight, Azure MonitorExpertise in writing python and SQL...

  • Data Engineering Leader

    2 weeks ago

    Minneapolis, Minnesota, United States Omni Inclusive Full time

    Omni Inclusive is offering a highly competitive salary of $250,000 - $300,000 per year for this Senior Data Engineer role. The position requires a strong background in developing and delivering scalable data solutions using Azure Databricks, Azure Data Lake, and Azure Cosmos DB.Job DescriptionWe are seeking an experienced Senior Data Engineer to join our...

  • Senior Data Engineer

    1 month ago

    Minneapolis, United States Omni Inclusive Full time

    Job Summary: We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. The ideal candidate will have a strong background in developing, and delivering scalable data solutions using Azure Databricks, Azure Data Lake, and Azure Cosmos DB. This role will involve coaching data engineers technically on topics like Azure...

  • Data Engineer Expert

    2 weeks ago

    Minneapolis, Minnesota, United States SysMind Tech Full time

    At SysMind Tech, we are seeking a skilled Data Engineer Expert to join our team. As a key member of our engineering department, you will be responsible for designing and implementing scalable data systems that meet the needs of our organization.The ideal candidate will have a strong background in data engineering with experience in structured databases...