Cloud Data Engineering Lead

3 weeks ago


Minneapolis, Minnesota, United States Diverse Lynx Full time
Job Description

We are seeking a highly skilled Cloud Data Engineering Lead to join our team at Diverse Lynx LLC. As a key member of our data engineering team, you will be responsible for designing, developing, and optimizing scalable ETL processes using Python, Apache Spark, and Azure Synapse. Your expertise in Azure Cloud, including SQL Pools and Spark Pools, will be essential in managing and processing large datasets efficiently.

The ideal candidate will have a strong background in data engineering, including experience with Azure Synapse Analytics, Azure Data Factory, and SQL. You will also have excellent communication and leadership skills, with the ability to lead and mentor junior developers.

We offer a competitive salary of $190,000 per annum, commensurate with experience. Our benefits package includes health insurance, 401(k) matching, and paid time off.

Key Responsibilities

- Design, develop, and optimize scalable ETL processes using Python, Apache Spark, and Azure Synapse.
- Build and manage Azure Data Factory pipelines to orchestrate complex data workflows.
- Use SQL Pools and Spark Pools within Synapse to manage and process large datasets efficiently.
- Implement Data Warehousing solutions using Azure Synapse Analytics to provide structured and queryable data layers.
- Ensure the data platform supports real-time and batch AI/ML data requirements.

Requirements

- Bachelor's degree in Computer Science or related field.
- At least 5 years of experience in data engineering, with a focus on Azure Cloud services.
- Expertise in Python and Apache Spark for large-scale data processing.
- Strong experience in Azure Synapse Analytics, including SQL Pools and Spark Pools.
- Advanced proficiency in Azure Data Factory for ETL pipeline orchestration and management.
- Knowledge of Data Warehousing principles, with hands-on experience building solutions on Azure.
- Experience with SQL, including complex queries, optimization, and performance tuning.
- Familiarity with CI/CD tools like Azure DevOps and managing infrastructure in Azure Cloud.
- Experience in Java for API integration and microservices architecture.
- Hands-on knowledge of Kubernetes for containerized data processing environments.
- Proficiency in working with Azure Data Lake Storage (ADLS) Gen2 for data storage and management.
- Experience working with APIs (REST, SOAP) and building API-based data integrations.

Salary: $190,000 per annum



  • Minneapolis, Minnesota, United States Diverse Lynx Full time

    Job Overview: We are seeking a highly skilled Cloud Data Engineer Lead to join our team at Diverse Lynx LLC. This is an exciting opportunity to work with cutting-edge technology and be part of a dynamic and innovative company.About the Role: As a Cloud Data Engineer Lead, you will design, implement, and maintain scalable and secure data architectures using...


  • Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    Job Title: Cloud Engineering Lead for Big Data IntegrationWe are seeking an experienced Cloud Engineering Lead to join our team at Futran Tech Solutions Pvt. Ltd. The ideal candidate will have hands-on experience in designing, implementing, and managing big data integration solutions using Azure and Apache Spark.The estimated salary for this role is...


  • Minneapolis, Minnesota, United States Tata Consultancy Services Full time

    The ideal candidate for this Senior Technical Lead for Data Platforms role at Tata Consultancy Services will possess a unique blend of technical expertise and leadership skills. As a seasoned data engineer, you will be responsible for driving technical excellence in data engineering and leading our data platform efforts.About YouWe are looking for someone...

  • Cloud Engineer Lead

    3 weeks ago


    Minneapolis, Minnesota, United States Omni Inclusive Full time

    Job DescriptionWe are looking for a Cloud Engineer Lead to oversee the design and implementation of our cloud infrastructure on AWS. The ideal candidate will have extensive experience in cloud computing and be able to lead a team of engineers to deliver high-quality solutions.Responsibilities:Lead the design and implementation of cloud infrastructure on...


  • Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    Futran Tech Solutions Pvt. Ltd. is seeking a skilled Cloud Big Data Tech Lead to lead our team in designing, implementing, and maintaining large-scale cloud-based data engineering solutions.The ideal candidate will have hands-on experience with Pyspark, Spark SQL programming, and expertise in Azure services such as ADLS Gen2, AZ copy, Synapse, Kubernetes,...


  • Minneapolis, Minnesota, United States SmartThings Full time

    At SmartThings, we're pushing the boundaries of smart home technology. As a Cloud Engineering Lead, you'll play a key role in shaping our cloud infrastructure and ensuring seamless experiences for our users.About the RoleWe're seeking an experienced professional to join our team and lead the development of our cloud engineering practices. This is an exciting...


  • Minneapolis, Minnesota, United States Snowflake Computing Full time

    Build the Future of Cloud-Based Data PlatformsSnowflake Computing is a leading provider of cloud-based data platforms. We are seeking an experienced Data Cloud Solutions Specialist to join our team. As a key member of our implementation and strategy experts, you will engage with customers at all phases of their journey to optimize, accelerate, and achieve...

  • Senior Data Engineer

    3 weeks ago


    Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    Futran Tech Solutions Pvt. Ltd. is seeking a highly skilled Senior Data Engineer - Cloud Computing to lead our team in designing, implementing, and maintaining large-scale cloud-based data engineering solutions.The ideal candidate will have extensive experience with Pyspark, Spark SQL programming, and expertise in Azure services such as ADLS Gen2, AZ copy,...

  • Data Engineer Lead

    3 weeks ago


    Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    Futran Tech Solutions Pvt. Ltd. is looking for a skilled Data Engineer Lead to join their team in Minneapolis, MN. As a key member of the data engineering team, you will design and build complex data processing pipelines using ADF and Airflow. You will work closely with cross-functional teams to ensure seamless integration with cloud databases like SQL. The...


  • Minneapolis, Minnesota, United States Levi Ray & Shoup Inc Full time

    Job Title: Senior Data Engineer - LeadLocation: Minneapolis, MNEstimated Salary: $195,000 per yearJob Summary: As the Senior Data Engineer Lead, you will oversee the design, development, and maintenance of our data infrastructure. You will lead a team of data engineers, driving best practices in data integration, modeling, and reporting.Key...


  • Minneapolis, Minnesota, United States Chewy Full time

    About Chewy, we're committed to providing the best possible experience for our customers and their pets. Our mission is to become the most trusted and convenient online destination for pet parents and our partners – vets and service providers – alike. We believe in thinking big and exploring opportunities to differentiate ourselves in the industry.About...


  • Minneapolis, Minnesota, United States Tata Consultancy Services Full time

    Tata Consultancy Services is seeking a seasoned data engineering professional to lead our data platform efforts. As the Data Engineering Team Lead, you will be responsible for overseeing the design, development, and deployment of our data engineering capabilities.About This RoleIn this critical position, you will drive technical excellence in data...

  • Data Engineering Lead

    3 weeks ago


    Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    Futran Tech Solutions Pvt. Ltd. is seeking a highly skilled Data Engineer to join their team in Minneapolis, MN. This role requires at least 10 years of experience in big data skills and expertise in Apache Spark Scala development. The ideal candidate will have strong SQL concepts, Unix skills, and experience with ETL tools like Data Stage and Spark/Scala....


  • Minneapolis, Minnesota, United States Diverse Lynx Full time

    Job Description:Data Engineering Team Lead plays a crucial role in leading and mentoring teams to design, develop, and maintain data platforms. This individual will be responsible for providing technical expertise and direction to scrum teams on GCP migration efforts, identifying best practices for data platforms, and ensuring seamless collaboration with...

  • Data Architect 2

    3 weeks ago


    Minneapolis, Minnesota, United States American Cybersystems Full time

    Job SummaryWe are seeking a highly skilled Data Architect 2 to join our team as a Cloud Data Engineering Specialist. This role involves designing and implementing scalable data pipelines using AWS services.Key ResponsibilitiesDesign and develop data pipelines using AWS services such as Glue, Redshift, and Lambda.Collaborate with cross-functional teams to...


  • Minneapolis, Minnesota, United States Improving Full time

    About the Team:Improving's Data Engineering team is passionate about delivering innovative data solutions to our clients. We believe in fostering a collaborative environment where engineers can grow professionally and personally. As a Sr. Data Engineer, you will be part of a dynamic team that is dedicated to staying ahead of the curve in terms of technology,...

  • Cloud Data Architect

    3 weeks ago


    Minneapolis, Minnesota, United States Emergent Software Full time

    Job Title: Cloud Data ArchitectThe estimated salary for this role is $120,000 - $180,000 per year. We are seeking an experienced Cloud Data Architect to join our team at Emergent Software. In this position, you will be responsible for building and managing production data systems using cloud infrastructure.Key Responsibilities:Collaborate with a team of...


  • Minneapolis, Minnesota, United States Federal Reserve Bank Full time

    Company OverviewThe Federal Reserve Bank of Minneapolis is seeking an experienced Data Engineering Manager Lead to join our TRIM Team. This team is responsible for building and maintaining components, as well as providing design services for subscribing applications.Job DescriptionWe are looking for a skilled professional with a deep technical background in...


  • Minneapolis, Minnesota, United States Aloden, Inc. Full time

    Job DescriptionWe are seeking an experienced Azure Data Engineer Lead to join our team at Aloden, Inc. As a key member of our engineering team, you will be responsible for leading the design, development, and implementation of large-scale data solutions on Microsoft Azure.The ideal candidate will have a strong understanding of relational database concepts,...


  • Minneapolis, Minnesota, United States Wells Fargo Full time

    About this role:We are seeking an experienced Cloud Systems Engineer Lead to join our Consumer Technology Platform Management team. As a key member of the team, you will provide platform support for critical applications and infrastructure based on Windows/Linux/Azure technologies used throughout the Wells Fargo retail branch network.In this role, you will...