Big Data Engineer

3 weeks ago


Louisville, Kentucky, United States Diverse Lynx Full time

Diverse Lynx LLC is an Equal Employment Opportunity employer.

">">

We are committed to fostering a diverse and inclusive workplace culture.

">">Salary:">

The estimated annual salary for this position is approximately $140,000, taking into account the location and industry standards.

">">Job Description:">

Title: Data Engineer - Azure Databricks

">

Location: Louisville, KY

">

Job Type: Full-time/Contract

">

Job Description:

">

As a seasoned Data Engineer - Azure Databricks, you will play a pivotal role in designing, building, and maintaining large-scale data processing systems.

">

Your primary responsibilities will include developing efficient data pipelines in Azure ADF, ensuring high-quality data ingestion from various sources using Azure ADF, and leveraging Azure Databricks & Delta lake and Pyspark for data transformation.

">

Furthermore, you will implement robust integration solutions in Microsoft Azure to drive business value and enhance customer experiences.

">

You will need to possess exceptional problem-solving skills to troubleshoot complex data-related issues and provide actionable insights to stakeholders.

">

A collaborative mindset, paired with strong communication skills, is essential for success in this role.

">

You will work closely with cross-functional teams to architect effective data pipelines and ensure seamless data flow.

">

Maintaining data quality, integrity, and security will be top priorities in this position.

">

The selected candidate will be responsible for evolving existing scripts and notebooks to meet client needs, monitoring, troubleshooting, and optimizing data pipelines.

">

Documenting technical specifications, best practices, and documenting programming efforts is crucial.

">

Key Responsibilities:

">

• Design and develop large-scale data processing systems

">

• Develop efficient data pipelines in Azure ADF

">

• Ensure high-quality data ingestion from various sources using Azure ADF

">

• Leverage Azure Databricks & Delta lake and Pyspark for data transformation

">

• Implement robust integration solutions in Microsoft Azure

">

• Troubleshoot complex data-related issues and provide actionable insights

">

• Collaborate with cross-functional teams to architect effective data pipelines

">

• Maintain data quality, integrity, and security

">

• Evolve existing scripts and notebooks to meet client needs

">

• Monitor, troubleshoot, and optimize data pipelines

">

Requirements:

">

• Bachelor's degree in Computer Science or related field

">

• 3+ years of experience in data engineering

">

• Proficiency in Azure ADF, Azure Databricks & Delta lake and Pyspark

">

• Excellent problem-solving skills

">

• Strong analytical and problem-solving skills

">

• Experience with data transformation and cleansing operations

">

• Strong understanding of data architecture and design

">

• Familiarity with cloud-based data platforms

">

What We Offer:

">

• Competitive salary and benefits package

">

• Opportunities for professional growth and development

">

• Collaborative and dynamic work environment

">

• Recognition and rewards for outstanding performance

">

Others:

">

Diverse Lynx LLC is an equal opportunity employer committed to diversity and inclusion in the workplace.

">

We welcome applications from qualified candidates from diverse backgrounds.

">

Please note that all applicants will be evaluated solely on their ability, competence, and proven capability to perform the functions outlined in the corresponding role.

">

We look forward to receiving your application.

">

BEST OF LUCK WITH YOUR APPLICATION



  • Louisville, Kentucky, United States JobRialto Full time

    About the JobWe need a highly skilled QA Engineer to join our team. As a Big Data/ETL QA Engineer, you will be responsible for testing our ETL pipelines, APIs, and big data applications. You should have expertise in developing and maintaining automated test scripts for API and ETL using Selenium or other automation tools.The ideal candidate will have over 10...


  • Louisville, Kentucky, United States Futran Tech Solutions Pvt. Ltd. Full time

    Job Summary: We are looking for a highly skilled Azure Databricks Engineer to lead the design and development of large-scale data systems using Big Data technologies.">Key Responsibilities:">">Design and develop data pipelines and infrastructure using Databricks, Hadoop, Cloudera, and other Big Data technologies">Collaborate with cross-functional teams to...

  • Big Data Developer

    3 weeks ago


    Louisville, Kentucky, United States Diverse Lynx Full time

    Job Overview:Diverse Lynx LLC is an Equal Employment Opportunity employer. We promote a diverse workforce across all levels in the company and welcome applications from qualified candidates. As a Data Engineer - Cloud Architect, you will play a critical role in shaping our cloud infrastructure and driving business growth through data-driven insights.Key...

  • Cloud Data Engineer

    3 weeks ago


    Louisville, Kentucky, United States Futran Tech Solutions Pvt. Ltd. Full time

    Company Overview: Futran Tech Solutions Pvt. Ltd. is a leading technology firm providing innovative solutions to businesses worldwide.">Job Title: Azure Databricks Engineer">Location: Louisville, KY (Onsite)">Duration: Contract">Estimated Salary: $140,000 - $160,000 per annum">Job Description:">We are seeking an experienced Azure Databricks Engineer to join...


  • Louisville, Kentucky, United States Marsh LLC Full time

    About the CompanyMMA National is an award-winning company with a Top Workplace culture that values collaboration and innovation.We are seeking a highly skilled Senior Data Engineer to join our IT Data and Analytics Team, where you will be responsible for designing and implementing scalable data pipelines, enhancing data models, and providing subject matter...


  • Louisville, Kentucky, United States CHEMICAL & INDUSTRIAL ENGINEER Full time

    Career OpportunityC&I Engineering is currently looking for a talented Civil Structural Engineer to help us achieve our company mission. This person will be part of the Civil/Structural team responsible for structural designs.Salary: Estimated salary range: $80,000 - $120,000 per year, depending on experience.Develop models, drawings, and design sketches...


  • Louisville, Kentucky, United States CHEMICAL & INDUSTRIAL ENGINEER Full time

    We are seeking a highly skilled Mechanical Engineer to join our team as a Staff Mechanical Engineer in the food and beverage industry.The ideal candidate will have experience designing production facilities and working closely with other disciplines to ensure customer needs are fulfilled.This role requires strong technical skills, including piping stress...

  • Process Systems Lead

    2 weeks ago


    Louisville, Kentucky, United States CHEMICAL & INDUSTRIAL ENGINEER Full time

    We are seeking an experienced Process Systems Lead to join our Chemical & Industrial Engineer team as a Staff Mechanical Engineer in the food and beverage industry.The ideal candidate will have a strong background in designing production facilities and working with other disciplines to meet customer needs.This role requires a deep understanding of technical...


  • Louisville, Kentucky, United States Diverse Lynx Full time

    Job Description:In this role, you will design, develop, and maintain data pipelines in Azure ADF and Databricks to support business growth and decision-making. You will collaborate with cross-functional teams to implement Integration solutions in Microsoft Azure and troubleshoot data-related issues. Additionally, you will participate in Agile development...

  • Cloud Data Engineer

    3 weeks ago


    Louisville, Kentucky, United States Marsh LLC Full time

    OverviewMMA National is a leading provider of business insurance, employee health & benefits, retirement, and private client insurance solutions. Our company culture values hard work, diversity, and innovation.Job SummaryWe are seeking a Senior Data Engineer to join our IT Data and Analytics Team. As a key member of the team, you will develop and maintain...


  • Louisville, Kentucky, United States Marsh LLC Full time

    Job SummaryWe are seeking a highly skilled Senior Data Engineer to join our MMA National IT Data and Analytics Team. The successful candidate will develop and maintain scalable data pipelines, enhance data models, and provide subject matter expertise in data acquisition and consumption pipelines on Azure, Databricks, and Confluent.This role involves...

  • Data Engineer

    3 weeks ago


    Louisville, Kentucky, United States SysMind Tech Full time

    Job DescriptionWe are seeking a highly skilled Data Engineer to join our team at SysMind Tech.The successful candidate will design and develop SQL Server stored procedures, functions, views, and triggers to be used during the ETL process. Additionally, they will build data transformations with SSIS, including importing data from files and moving data from...

  • Data Engineer Lead

    3 weeks ago


    Louisville, Kentucky, United States Marsh LLC Full time

    Job OverviewMMA National is seeking a highly skilled Senior Data Engineer to join its IT Data and Analytics Team. As a key member of the team, you will be responsible for designing and implementing scalable data pipelines, enhancing data models, and providing subject matter expertise in data acquisition and consumption pipelines on Azure, Databricks, and...

  • Azure Data Engineer

    3 weeks ago


    Louisville, Kentucky, United States Diverse Lynx Full time

    **Job Description:**We are seeking an experienced Azure Data Engineer to join our team at Diverse Lynx LLC. This role involves designing, implementing, and maintaining large-scale data pipelines in Azure ADF. The ideal candidate will have a strong background in data engineering, with experience in ingesting data from various sources using Azure ADF.The...


  • Louisville, Kentucky, United States Marsh LLC Full time

    About the RoleWe are looking for a talented Data Engineering Specialist to join our team at Marsh McLennan Agency (MMA). As a key member of our IT Data and Analytics Team, you will play a critical role in developing and maintaining scalable data pipelines, enhancing data models, and providing subject matter expertise in data acquisition and consumption...


  • Louisville, Kentucky, United States Marsh LLC Full time

    About the RoleWe are seeking a seasoned Senior Data Engineer to join our IT Data and Analytics Team at MMA National. The successful candidate will be responsible for developing and maintaining scalable data pipelines, enhancing data models, and providing subject matter expertise in data acquisition and consumption pipelines on Azure, Databricks, and...


  • Louisville, Kentucky, United States CHEMICAL & INDUSTRIAL ENGINEER Full time

    Job Title: Staff Piping DesignerCompany OverviewCHEMICAL & INDUSTRIAL ENGINEER is a full-service engineering design firm providing engineering/design/project management and procurement services to various clients in oil & gas, food & beverage, and distilled spirits.About the OpportunityWe are seeking a skilled Staff Piping Designer to help us achieve our...

  • Cloud Data Engineer

    2 weeks ago


    Louisville, Kentucky, United States Cognizant North America Full time

    Role OverviewCognizant North America is looking for a skilled AEP Data Engineer to join our team of IT professionals in a permanent role. The successful candidate will be responsible for designing and implementing data pipelines, creating and modifying experience data models, and monitoring and troubleshooting data pipelines. You will also work closely with...


  • Louisville, Kentucky, United States Diverse Lynx Full time

    About Us\Diverse Lynx LLC is an Equal Employment Opportunity employer committed to diversity and inclusion. We promote a workplace culture that values innovation, creativity, and teamwork.\Why Join Us?\This role offers the opportunity to work with a talented team of professionals who share a passion for technology and innovation. You will have the chance to...


  • Louisville, Kentucky, United States CHEMICAL & INDUSTRIAL ENGINEER Full time

    Job Title: Cost Estimator/SchedulerLocation: [Your Location]Job Type: Full-timeAbout Us: CHEMICAL & INDUSTRIAL ENGINEER is an employee-owned company providing engineering design and project management services to various industries. We value teamwork, innovation, and customer satisfaction.Job Summary: We are seeking a highly skilled Cost Estimator/Scheduler...