Compunnel Inc. | Azure Data Engineer

4 weeks ago

minneapolis, United States Compunnel Inc. Full time

Role: Azure Data Engineer

Location: Downtown Minneapolis (Day-1 Onsite)

Duration: W2 Contract (12+Months)

Job responsibilities: Position Overview

We are seeking a Lead Developer with advanced skills in Python, Apache Spark, Azure Synapse, and Azure Data Engineering services to develop and manage ETL pipelines and data processing solutions that support AI/ML initiatives. The ideal candidate will have strong expertise in Azure Cloud, including SQL, Data Factory, SQL Pools, Spark Pools, and Data Warehousing. A background in CI/CD processes, cross-functional collaboration, and methodologies is essential. In addition, knowledge of Data Science, Java, Kubernetes, Azure Data Lake Storage (ADLS) Gen2, and API development is required.

Key Responsibilities

1. ETL and Data Pipeline Development

- Design, develop, and optimize scalable ETL processes using Python, Apache Spark, and Azure Synapse.

- Build and manage Azure Data Factory pipelines to orchestrate complex data workflows.

- Use SQL Pools and Spark Pools within Synapse to manage and process large datasets efficiently.

- Implement Data Warehousing solutions using Azure Synapse Analytics to provide structured and queryable data layers.

- Ensure the data platform supports real-time and batch AI/ML data requirements.

2. Azure Cloud Development & CI/CD Deployment

- Build, configure, and manage CI/CD pipelines on Azure DevOps for ETL and data processing tasks.

- Automate infrastructure provisioning, testing, and deployment using Infrastructure-as-Code (IaC) tools like ARM templates or Terraform.

- Optimize Azure Data Lake Storage (ADLS Gen2) to store and manage raw and processed data efficiently, ensuring proper access control and data security.

3. Cross-Functional Collaboration

- Collaborate with Data Scientists, Data Engineers, ML Engineers, and Business Analysts to translate business requirements into data solutions.

- Work with the DevOps and Security teams to ensure smooth and secure deployment of applications and pipelines.

- Act as the technical lead in designing, developing, and implementing data solutions, mentoring junior team members.

4. Data Engineering and API Development

- Develop and integrate with external and internal APIs for data ingestion and data exchange.

- Build, test, and deploy RESTful APIs for secure data access.

- Use Kubernetes for containerizing and deploying data processing applications.

- Manage data storage and transformation to support advanced Data Science and AI/ML models.

5. Agile Project Management

- Participate in and lead Agile ceremonies, such as sprint planning, daily stand-ups, and retrospectives.

- Collaborate with cross-functional teams in iterative development to ensure high-quality and timely feature delivery.

- Adapt to changing project priorities and business needs in an Agile environment.

Required Skills and Qualifications

1. Technical Skills:

- Expertise in Python and Apache Spark for large-scale data processing.

- Strong experience in Azure Synapse Analytics, including SQL Pools and Spark Pools.

- Advanced proficiency in Azure Data Factory for ETL pipeline orchestration and management.

- Knowledge of Data Warehousing principles, with hands-on experience building solutions on Azure.

- Experience with SQL, including complex queries, optimization, and performance tuning.

- Familiarity with CI/CD tools like Azure DevOps and managing infrastructure in Azure Cloud.

- Experience in Java for API integration and microservices architecture.

- Hands-on knowledge of Kubernetes for containerized data processing environments.

- Proficiency in working with Azure Data Lake Storage (ADLS) Gen2 for data storage and management.

- Experience working with APIs (REST, SOAP) and building API-based data integrations.

2. Agile and Cross-Functional Skills:

- Experience working in an Agile environment, using Scrum or Kanban.

- Ability to lead, mentor, and coach junior developers in the team.

- Strong collaboration skills to work with data scientists, analysts, and cross-functional teams to deliver end-to-end data solutions.

3. Behavioral Skills:

- Strong analytical and problem-solving skills with a passion for data-driven solutions.

- Excellent communication and presentation skills, able to explain complex technical concepts to non-technical stakeholders.

- Ability to work in a fast-paced, dynamic environment with changing priorities.

- Self-motivated and results-oriented with attention to detail.

Preferred Qualifications

- Azure certifications in data engineering or cloud architecture.

- Experience deploying AI/ML models on cloud platforms.

- Familiarity with Data Governance best practices, ensuring compliance with data privacy regulations

  • Senior Azure Data Scientist

    7 days ago

    Minneapolis, Minnesota, United States Compunnel Full time

    About the RoleWe're looking for an exceptional Senior Azure Data Scientist to join our team at Compunnel. As a key member of our data science team, you'll be responsible for designing, developing, and deploying cutting-edge data analytics solutions using Azure services and machine learning algorithms.The ideal candidate will have a Ph.D. in Computer Science,...

  • Azure Data Engineer

    2 weeks ago

    Minneapolis, United States Compunnel Full time

    Project Summary: This project involves building AC360 data pipelines in Azure using Spark technologies. As part of this initiative, the development of Zafin data pipelines in Azure is also included. Key Responsibilities: 1. Identify Requirements and Develop Solutions · Identify requirements for RMS replacement and develop solutions in Azure. ·...

  • Azure Data Engineer

    1 month ago

    Minneapolis, United States Compunnel Inc. Full time

    Role: Azure Data Engineer Location: Downtown Minneapolis (Day-1 Onsite)Duration: W2 Contract (12+Months) Job responsibilities: Position OverviewWe are seeking a Lead Developer with advanced skills in Python, Apache Spark, Azure Synapse, and Azure Data Engineering services to develop and manage ETL pipelines and data processing solutions that support AI/ML...

  • Senior Azure Data Engineer Lead

    3 weeks ago

    Minneapolis, Minnesota, United States Aloden, Inc. Full time

    Job DescriptionWe are seeking an experienced Azure Data Engineer Lead to join our team at Aloden, Inc. As a key member of our engineering team, you will be responsible for leading the design, development, and implementation of large-scale data solutions on Microsoft Azure.The ideal candidate will have a strong understanding of relational database concepts,...

  • Lead Azure Data Engineer

    1 month ago

    Minneapolis, United States Aloden, Inc. Full time

    Job Details: Lead Azure Data Engineer Duration: 12+ Month (s) Contract - 12m (possibility for extension or conversion) Location: - Hybrid Roles (3 Days Onsite/2 Days WFH) Note: ENTIRE TEAM IS OOO LAST 2 WEEKS OF THE YEAR INCLUDING CONTRACTORS - PLEASE MAKE SURE THEY ARE AWARE PRIOR TO SUBMISSION Job Descriptions: In this contingent resource assignment,...

  • Azure Data Engineer

    1 month ago

    Minneapolis, United States Diverse Lynx Full time

    Role: Azure Data Engineer Location: Downtown Minneapolis,MN (Day 1 Onsite) Contract Position Overview We are seeking a Lead Developer with advanced skills in Python, Apache Spark, Azure Synapse, and Azure Data Engineering services to develop and manage ETL pipelines and data processing solutions that support AI/ML initiatives. The ideal candidate will have...

  • Azure Data Engineer Specialist

    2 weeks ago

    Minneapolis, Minnesota, United States Diverse Lynx Full time

    Job OverviewDiverse Lynx LLC is a leading company in the field of data engineering, and we are currently seeking a highly skilled Azure Data Engineer Specialist to join our team. As a key member of our data engineering team, you will be responsible for designing, developing, and optimizing scalable ETL processes using Python, Apache Spark, and Azure Synapse....

  • Azure Data Scientist

    2 weeks ago

    Minneapolis, Minnesota, United States STIAOS Technologies Full time

    Job Summary: We are seeking an Azure Data Scientist to design, develop, and implement data engineering solutions using Azure cloud services. The estimated salary for this role is $160,000 per year.Key Responsibilities:• Develop and maintain data pipelines using Databricks (PySpark) for data ingestion, processing, and analysis.• Configure and manage Azure...

  • Azure Data Solutions Director

    3 weeks ago

    Minneapolis, Minnesota, United States Aloden, Inc. Full time

    Key ResponsibilitiesThe Senior Azure Data Engineer Lead will be responsible for leading the design, development, and implementation of large-scale data solutions on Microsoft Azure, as well as collaborating with cross-functional teams to ensure smooth project delivery.The successful candidate will have a strong understanding of relational database concepts,...

  • Azure Data Engineer

    1 month ago

    Minneapolis, United States Futran Tech Solutions Pvt. Ltd. Full time

    Azure Data Engineer Minneapolis, MN ADF CICD Airflow SQL Cloud Databases Understanding of Agile methodologies 4-6 years of working experience in primary skills Overall 8-10 years in ETL/Data Engineering Understand business requirements. Understand source systems, source data and source data formats that are available on-prem / cloud. Design and build data...

  • Senior Configuration Manager

    4 weeks ago

    Minneapolis, Minnesota, United States Compunnel Inc. Full time

    Job DescriptionWe are seeking a highly skilled Senior Configuration Manager to join our team at Compunnel Inc. in Minneapolis, MN.Job Summary:The Senior Configuration Manager will be responsible for performing configuration management activities related to the design, development, and production of complex systems in a defense environment.Key...

  • Azure Data Architect

    2 weeks ago

    Minneapolis, Minnesota, United States Omni Inclusive Full time

    We are looking for a skilled Senior Data Engineer to join our team at Omni Inclusive. This role offers a competitive salary range of $220,000 - $280,000 per year and comes with various benefits.About the RoleThe successful candidate will have extensive experience in designing and implementing large-scale data engineering solutions using Azure services.Main...

  • Senior Azure Data Engineer Position

    3 weeks ago

    Minneapolis, Minnesota, United States Eateam Full time

    Job Description: We are looking for an experienced Data Engineer to join our team at Eateam. As a Sr. Data Engineer, you will play a key role in designing and implementing data pipelines that drive business growth. Your primary responsibilities will include developing scalable data pipelines using Azure Data Factory and Azure Databricks, implementing data...

  • Data Management Specialist

    2 weeks ago

    Minneapolis, Minnesota, United States Compunnel Full time

    About the RoleWe are looking for a highly skilled Data Management Specialist to join our team at Compunnel. As a key member of our IT department, you will be responsible for creating table space areas, tables, and views, as well as logical and physical database design.The ideal candidate will have at least 3 years of experience in database administration and...

  • Azure Engineer

    4 weeks ago

    Minneapolis, United States Saxon Global Full time

    This is a 12 month contract to hire role with CHS. 100% remote. USC/GC. Linkedin Must. Salary on conversion is 110-125K. A growing industry is looking for a contract or likely contract to hire Azure Engineer. The primary responsibilities will be to build and enable an Azure cloud environment including include responsibility for the design/planning,...

  • Senior Data Engineer

    1 month ago

    Minneapolis, United States Allianz Full time

    At Allianz Life, we are guided by our mission - we secure your future - and by our common goal of caring for tomorrow. We care for our employees by promoting an inclusive culture where everyone has the opportunity to grow and be rewarded for their success. We care for our customers by creating new products that help them protect their families and pursue...

  • Lead Data Architect

    1 month ago

    Minneapolis, Minnesota, United States Aloden, Inc. Full time

    Job DescriptionAloden, Inc. seeks a seasoned Lead Data Architect to spearhead the design and implementation of our cloud-based data infrastructure. As a key member of our team, you will be responsible for architecting and implementing scalable, secure, and high-performance data solutions using Microsoft Azure technologies.Key Responsibilities:Design and...

  • Inventory Management Coordinator

    1 month ago

    Minneapolis, Minnesota, United States Compunnel Inc. Full time

    Job OverviewWe are seeking a highly skilled Inventory Control Technician to join our team at Compunnel Inc.

  • Data Analysis Specialist

    2 weeks ago

    Minneapolis, Minnesota, United States Compunnel Full time

    Job Title: Data Analysis SpecialistWe are seeking a skilled Data Analysis Specialist to join our team at Compunnel. As a key member of our data analytics team, you will be responsible for working on a program consolidating all Consumer Communications into a central organization and platform.Key Responsibilities:Evaluate Business Requirements: Understand and...

  • Database Systems Engineer

    2 weeks ago

    Minneapolis, Minnesota, United States Compunnel Full time

    Compunnel Seeks Database ExpertWe are seeking an experienced Database Administrator to join our team at Compunnel. The successful candidate will be responsible for setting up and maintaining one or more database systems in a client/server operating system environment.The ideal candidate will have experience with Oracle RAC, proficiency in SQL, and knowledge...