Databricks Expert

Diverse Lynx LLC is an Equal Employment Opportunity employer.

We strive to create a workplace culture that values diversity, equity, and inclusion.

The estimated annual salary for this position is approximately $125,000, considering the location and industry standards.

Title: Azure Databricks Data Engineer

Location: Louisville, KY

Job Type: Full-time/Contract

Job Description:

As a skilled Azure Databricks Data Engineer, you will play a critical role in designing, building, and maintaining large-scale data processing systems in Azure ADF.

Your primary responsibilities will include developing efficient data pipelines in Azure ADF, ensuring high-quality data ingestion from various sources using Azure ADF, and leveraging Azure Databricks & Delta lake and Pyspark for data transformation.

Furthermore, you will implement robust integration solutions in Microsoft Azure to drive business value and enhance customer experiences.

You will need to possess exceptional problem-solving skills to troubleshoot complex data-related issues and provide actionable insights to stakeholders.

A collaborative mindset, paired with strong communication skills, is essential for success in this role.

You will work closely with cross-functional teams to architect effective data pipelines and ensure seamless data flow.

Maintaining data quality, integrity, and security will be top priorities in this position.

The selected candidate will be responsible for evolving existing scripts and notebooks to meet client needs, monitoring, troubleshooting, and optimizing data pipelines.

Documenting technical specifications, best practices, and documenting programming efforts is crucial.

Responsibilities:

• Design and develop large-scale data processing systems in Azure ADF

• Develop efficient data pipelines in Azure ADF

• Ensure high-quality data ingestion from various sources using Azure ADF

• Leverage Azure Databricks & Delta lake and Pyspark for data transformation

• Implement robust integration solutions in Microsoft Azure

• Troubleshoot complex data-related issues and provide actionable insights

• Collaborate with cross-functional teams to architect effective data pipelines

• Maintain data quality, integrity, and security

• Evolve existing scripts and notebooks to meet client needs

• Monitor, troubleshoot, and optimize data pipelines

Requirements:

• Bachelor's degree in Computer Science or related field

• 3+ years of experience in data engineering

• Proficiency in Azure ADF, Azure Databricks & Delta lake and Pyspark

• Excellent problem-solving skills

• Strong analytical and problem-solving skills

• Experience with data transformation and cleansing operations

• Strong understanding of data architecture and design

• Familiarity with cloud-based data platforms

What We Offer:

• Competitive salary and benefits package

• Opportunities for professional growth and development

• Collaborative and dynamic work environment

• Recognition and rewards for outstanding performance

Others:

Diverse Lynx LLC is committed to creating a workplace culture that promotes diversity, equity, and inclusion.

We welcome applications from qualified candidates from diverse backgrounds.

Please note that all applicants will be evaluated solely on their ability, competence, and proven capability to perform the functions outlined in the corresponding role.

We look forward to receiving your application.

BEST OF LUCK WITH YOUR APPLICATION