Data Integration Lead with Spark

5 days ago


Minneapolis, Minnesota, United States Accuro Group Full time

We are seeking a highly skilled Data Integration Lead to join our team at Accuro Group in Minneapolis, MN. The ideal candidate will have 10+ years of experience in ETL development and a strong background in PySpark.

Job Description

Role: Data Integration Lead with Spark

Location: Minneapolis, MN (Onsite)

Experience: 10+ Years

Duration: Long-term contract

Job Summary:

  • Lead the design, development, and maintenance of ETL processes using AWS Glue and PySpark to support data integration and business intelligence initiatives.
  • Collaborate with stakeholders to understand data requirements and ensure efficient data flow and transformation using ETL tools and PySpark.
  • Develop and implement ETL processes using PySpark to extract, transform, and load data into AWS services such as S3, RDS, and Redshift.
  • Maintain and optimize existing ETL pipelines to improve performance and reduce costs.
  • Closely work with the development team to deliver high-quality solutions that meet business needs.

Responsibilities:

  • Design, develop, and maintain ETL processes using AWS Glue and PySpark.
  • Collaborate with stakeholders to understand data requirements and ensure efficient data flow and transformation using ETL tools and PySpark.
  • Develop and implement ETL processes using PySpark to extract, transform, and load data.
  • Maintain and optimize existing ETL pipelines to improve performance and reduce costs.
  • Closely work with the development team to deliver high-quality solutions that meet business needs.

Requirements:

  • 4+ years of experience in ETL development with knowledge on PySpark.
  • 5+ years as an ETL Developer SQL expert AWS Glue WITH Python (PySpark).
  • PySpark Dataframe API Spark SQL Knowledge in AWS services (e.g. DMS, S3, RDS, Redshift, Step Function).
  • Nice to Haves: Etl development experience with tools e.g. SAP BODS, Informatica.
  • Good understanding of version control tools like Git, GitHub, TortoiseHg.
  • Financial services experience Agile

Estimated Salary: $120,000 - $150,000 per year based on location and experience.

Benefits:

  • Competitive salary and benefits package.
  • Opportunities for professional growth and development.
  • A dynamic and supportive work environment.


  • Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    Job Title: Cloud Engineering Lead for Big Data IntegrationWe are seeking an experienced Cloud Engineering Lead to join our team at Futran Tech Solutions Pvt. Ltd. The ideal candidate will have hands-on experience in designing, implementing, and managing big data integration solutions using Azure and Apache Spark.The estimated salary for this role is...


  • Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    Futran Tech Solutions Pvt. Ltd. is seeking a highly skilled Data Engineer to join their team in Minneapolis, MN. This role requires at least 10 years of experience in big data skills and expertise in Apache Spark Scala development. The ideal candidate will have strong SQL concepts, Unix skills, and experience with ETL tools like Data Stage and Spark/Scala....


  • Minneapolis, Minnesota, United States Diverse Lynx Full time

    Job DescriptionWe are seeking a highly skilled Cloud Data Engineering Lead to join our team at Diverse Lynx LLC. As a key member of our data engineering team, you will be responsible for designing, developing, and optimizing scalable ETL processes using Python, Apache Spark, and Azure Synapse. Your expertise in Azure Cloud, including SQL Pools and Spark...

  • Data Engineer Lead

    4 weeks ago


    Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    Futran Tech Solutions Pvt. Ltd. is seeking a highly skilled Data Engineer Lead to join our team.Job Responsibilities:Lead a team of data engineers in designing, developing, and maintaining large-scale data processing systems.Develop and implement efficient ETL pipelines using Apache Spark and Scala.Collaborate with cross-functional teams to integrate data...


  • Minneapolis, Minnesota, United States Diverse Lynx Full time

    Job Description:Diverse Lynx LLC is seeking a highly skilled Senior Data Integration Specialist to lead our data pipelining efforts using PySpark. As a key member of our team, you will design, develop, and maintain efficient ETL processes to support data integration and business intelligence initiatives.Responsibilities:Work closely with stakeholders to...


  • Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    Futran Tech Solutions Pvt. Ltd. is seeking a skilled Cloud Big Data Tech Lead to lead our team in designing, implementing, and maintaining large-scale cloud-based data engineering solutions.The ideal candidate will have hands-on experience with Pyspark, Spark SQL programming, and expertise in Azure services such as ADLS Gen2, AZ copy, Synapse, Kubernetes,...

  • Lead Data Architect

    4 weeks ago


    Minneapolis, Minnesota, United States Aloden, Inc. Full time

    Job DescriptionAloden, Inc. seeks a seasoned Lead Data Architect to spearhead the design and implementation of our cloud-based data infrastructure. As a key member of our team, you will be responsible for architecting and implementing scalable, secure, and high-performance data solutions using Microsoft Azure technologies.Key Responsibilities:Design and...


  • Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    We are hiring a seasoned Azure Spark Engineering Specialist to join our team at Futran Tech Solutions Pvt. Ltd.The successful candidate will have extensive experience in designing, developing, and deploying scalable cloud-based data engineering solutions using Pyspark and Spark SQL programming.This role involves working closely with the development team to...


  • Minneapolis, Minnesota, United States Scout Exchange Full time

    About Scout ExchangeScout Exchange is a leading organization in the industry, committed to delivering exceptional results.Salary and BenefitsWe offer a competitive salary range of $120,000 - $160,000 per year, depending on experience. Our benefits package includes comprehensive health insurance, retirement plan matching, and generous paid time off.Job...


  • Minneapolis, Minnesota, United States Diverse Lynx Full time

    Job OverviewDiverse Lynx LLC is a leading company in the field of data engineering, and we are currently seeking a highly skilled Azure Data Engineer Specialist to join our team. As a key member of our data engineering team, you will be responsible for designing, developing, and optimizing scalable ETL processes using Python, Apache Spark, and Azure Synapse....


  • Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    Job Description: We are looking for a skilled Azure Tech Lead with expertise in Pyspark to join our team at Futran Tech Solutions Pvt. Ltd. The ideal candidate will have hands-on experience in designing, implementing, and managing big data integration solutions using Azure and Apache Spark.The estimated salary for this role is $130-135K, which includes a...


  • Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    About the JobFutran Tech Solutions Pvt. Ltd. is seeking a highly skilled Data Scientist for Cloud and Big Data to join our team. As a Data Scientist, you will work on designing, implementing, and deploying big data solutions for clients across various industries. You should have strong knowledge of big data technologies, including Hadoop, Spark, and NoSQL...


  • Minneapolis, Minnesota, United States K&K Global Talent Solutions Inc. Full time

    Job OverviewWe are seeking a highly skilled Lead Data Integration Engineer to join our team at K&K Global Talent Solutions Inc. in this key role, you will be responsible for designing, developing, and maintaining ETL processes to support data integration and business intelligence initiatives.About the RoleThis is a full-time position with an estimated salary...

  • Senior Consultant

    8 hours ago


    Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    About the CompanyFutran Tech Solutions Pvt. Ltd. is a leading provider of big data solutions for clients across various industries. We are seeking a highly skilled Senior Consultant - Big Data Solutions to join our team. As a Senior Consultant, you will work with our team to deliver cutting-edge big data solutions for clients. You should have good knowledge...


  • Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    Futran Tech Solutions Pvt. Ltd. is seeking a highly skilled Senior Data Engineer - Cloud Computing to lead our team in designing, implementing, and maintaining large-scale cloud-based data engineering solutions.The ideal candidate will have extensive experience with Pyspark, Spark SQL programming, and expertise in Azure services such as ADLS Gen2, AZ copy,...


  • Minneapolis, Minnesota, United States Futran Tech Solutions Pvt. Ltd. Full time

    About the RoleAs a Senior Consultant - Big Data Solutions at Futran Tech Solutions Pvt. Ltd., you will work with our team to deliver cutting-edge big data solutions for clients across various industries. The ideal candidate should have good knowledge of basic big data concepts, including Hive and SQL, as well as hands-on experience with Spark, Scala, or...


  • Minneapolis, Minnesota, United States Spark Packaging Full time

    Job OverviewSpark Packaging, a leading recruiting and staffing company based in Michigan, is seeking an experienced Maintenance Manager to join our team.

  • Master Data Architect

    4 weeks ago


    Minneapolis, Minnesota, United States Omni Inclusive Full time

    Job Summary:We're seeking an accomplished Senior Data Engineer to enhance our dynamic team. The ideal candidate will possess a strong background in developing and delivering scalable data solutions using Azure Databricks, Azure Data Lake, and Azure Cosmos DB. This role involves coaching data engineers on topics like Azure Databricks, optimizing data...


  • Minneapolis, Minnesota, United States Omni Inclusive Full time

    We are looking for a skilled Senior Data Engineer to join our team at Omni Inclusive. This role offers a competitive salary range of $220,000 - $280,000 per year and comes with various benefits.About the RoleThe successful candidate will have extensive experience in designing and implementing large-scale data engineering solutions using Azure services.Main...


  • Minneapolis, Minnesota, United States SysMind Tech Full time

    At SysMind Tech, we are seeking a skilled Data Engineer Expert to join our team. As a key member of our engineering department, you will be responsible for designing and implementing scalable data systems that meet the needs of our organization.The ideal candidate will have a strong background in data engineering with experience in structured databases...