Databricks DLT Lead Engineer

2 weeks ago

Plano, United States CrackaJack Digital Solutions Full time
Databricks DLT Lead Engineer

Plano, TX (Hybrid, 3 Days a week)

12 Months

*** Local Candidates Highly Preferred**

Job Summary:

We are seeking an experienced Senior Databricks Engineer to join our team of data professionals. In this role, you will collaborate closely with Data Scientists to prepare and transform large-scale datasets using PySpark, a popular open-source Python library for Apache Spark. You will play a crucial role in enabling effective data analysis and modeling by ensuring the availability of high-quality, feature-rich datasets.

Responsibilities:

1. Data Preparation and Transformation:

Leverage PySpark to efficiently process and transform large-scale datasets from various sources.

Develop robust and scalable PySpark code to handle data cleaning, munging, and feature engineering tasks.

Collaborate with Data Scientists to understand their data requirements and translate them into efficient PySpark workflows.

2. Feature Engineering:

Work closely with Data Scientists to identify and implement relevant feature engineering techniques.

Employ advanced feature engineering methods, such as one-hot encoding, scaling, binning, and feature creation/selection, to enhance the predictive power of machine learning models.

Stay up-to-date with the latest feature engineering techniques and best practices in the industry.

3. Distributed Computing:

Leverage Apache Spark's distributed computing capabilities to process and analyze large-scale datasets efficiently.

Optimize PySpark code for performance, scalability, and fault tolerance.

Implement and maintain data pipelines using PySpark to automate data preparation and transformation processes.

4. Code Quality and Documentation:

Write clean, maintainable, and well-documented PySpark code following best practices and coding standards.

Collaborate with team members through code reviews and knowledge sharing sessions.

Contribute to the development and maintenance of PySpark-related documentation and best practices within the organization.

5. Continuous Learning and Improvement:

Stay current with the latest developments in PySpark, Apache Spark, and related big data technologies.

Actively participate in professional development opportunities, such as attending conferences, workshops, or online training.

Identify areas for improvement in existing data preparation and transformation processes and propose solutions.

Databricks DLT Lead Engineer

  • Databricks Engineer

    2 days ago

    Plano, United States Accord Technologies Inc. Full time

    Job DescriptionJob DescriptionDatabricks Engineer Location: Plano, TX Hybrid ( need candidates within TX)Experience : 10 + years Must Have: 1. Databricks 2. Apache Spark 3. Python(and/or Scala) 4. AWS 5. SQL 6. Java 7. CI/CD Pipeline toolsMinimum Industry experience: 10 years Job Description: 1. Bachelor's degree or equivalent work experience 2. Must...

  • Databricks Engineer

    2 days ago

    Plano, United States Artech Full time

    Job title: Databricks Engineer Locations: Plano, TX Duration: 12 Months Job Family Description: Expert in in Amazon Web Services and Client Azure technology and design techniques as well as experience working across large environments with multiple operating systems/infrastructure for large-scale programs (e.g., Expert Engineers) starting to be...

  • Databricks Architect

    3 days ago

    Plano, United States KTek Resourcing Full time

    Job Summary: The Databricks Architect will be responsible for designing, implementing, and managing Databricks-based data solutions. The ideal candidate will have a deep understanding of data engineering, architecture, and analytics, as well as proficiency with monitoring tools to ensure optimal performance and reliability of our data platforms.Key...

  • SR. DATA ENGINEER with DATABRICKS, AWS, PYSPARK/ SPARK, PYTHON

    3 days ago

    Plano, United States Excelon Solutions Full time

    Job DescriptionJob DescriptionMust have skills: DATABRICKS, AWS, PYSPARK/ SPARK, PYTHONDatabricks should be strong. Minimum Industry experience: 10 yearsJob Description:1. Bachelor's degree or equivalent work experience2. Must have experience implementing atleast 2 end to end projects with Databricks which includes components like- Delta lake- Unity...

  • Senior Databricks Engineer/Data Engineer

    2 weeks ago

    Plano, United States Elite Mente LLC Full time

    Job DescriptionJob DescriptionSenior Databricks Engineer/Data Engineer Plano, TX (Onsite) (open to all Visa W2 only)Long term ContractJob Description:They need to hire an expert someone who can direct the team and provide the guidance in how to modernize/re-write/re-build/etc.This person must be a Databricks expertMust be very strong in Python (java is a...

  • Sr Data Engineer

    2 weeks ago

    Plano, United States Wizcom Consulting Full time

    Job DescriptionJob DescriptionSr Data Engineer (Pyspark/AWS/Databricks)Plano TX (Hybrid Role)Relocation can be considered W2 only (No C2C) Pre-screen must be on Pyspark & AWS, Databricks. I need someone with React, AWS & Pyspark with Databricks experience - THESE ARE MUST HAVE SKILLS Looking for 10+ years experience Formal training or certification on...

  • Sr Data Engineer

    2 weeks ago

    Plano, United States Accord Technologies Inc. Full time

    Job DescriptionJob DescriptionSr Data Engineer (Pyspark/AWS/Databricks)Plano TX (Hybrid Role)Relocation can be considered Pre-screen must be on Pyspark & AWS, Databricks. I need someone with React, AWS & Pyspark with Databricks experience - THESE ARE MUST HAVE SKILLS Looking for 10+ years experience Formal training or certification on software engineering...

  • Lead Software Engineer

    4 days ago

    Plano, United States JPMorgan Chase Full time

    Job Description We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible. As a Lead Software Engineer at JPMorgan Chase within the Corporate Technology, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure,...

  • Data Engineer

    2 months ago

    Plano, United States Promantus Inc Full time

    Data Engineer Location Plano Texas (Remote until COVID) Duration Long-term Job Description Develops Python/PySpark HQL queries. Develops new data models as necessary with Lead Backend Developer and Architect. Performs data visualization and analysis. Produces data samples for UI/UX Designer. Communicates insights to Architect Lead Backend Developer ...

  • Data Engineer

    2 months ago

    Plano, Texas, United States Promantus Inc Full time

    Data Engineer Location Plano Texas (Remote until COVID) Duration Long-term Job Description Develops Python/PySpark HQL queries. Develops new data models as necessary with Lead Backend Developer and Architect. Performs data visualization and analysis. Produces data samples for UI/UX Designer. Communicates insights to Architect Lead Backend Developer Lead...

  • Data Engineer

    3 days ago

    Plano, United States AllSTEM Connections Full time

    DATA ENGINEERON W2 ONLY,POSITIONHYBRID REPORTING:3DAYS ONSITELOCATION; PLANO,TXTechnology Stack Needed:- 1. Databricks 2. Apache Spark 3. Python(and/or Scala) 4. AWS 5. SQL 6. Java 7. CI/CD Pipeline tools Minimum Industry experience: 10 years Job Description: 1. Bachelor’s degree or equivalent work experience 2. Must have experience implementing atleast 2...

  • Data Engineer

    2 hours ago

    Plano, United States AllSTEM Connections Full time

    DATA ENGINEERON W2 ONLY,POSITIONHYBRID REPORTING:3DAYS ONSITELOCATION; PLANO,TXTechnology Stack Needed:- 1. Databricks 2. Apache Spark 3. Python(and/or Scala) 4. AWS 5. SQL 6. Java 7. CI/CD Pipeline tools Minimum Industry experience: 10 years Job Description: 1. Bachelor’s degree or equivalent work experience 2. Must have experience implementing atleast 2...

  • Triage Software Engineer

    1 week ago

    Plano, United States ClifyX Full time

    Triage Software Engineer Plano, TX office 3 times a week Infosys/Toyot Bill rate: $60 openings - 1• As an IAT engineer, you will be responsible for initial analysis and triage of defects before they are assigned to individual development teams for root cause and fix.• You will perform high level defect analysis to filter out environmental/ setup issues,...

  • Big Data Engineer

    1 day ago

    Plano, United States The AES Group Full time

    Job title: Big Data EngineerLocations: Plano, TXDuration: 12 MonthsNeed only USC & GC as this is C2HJob Family Description: Expert in AWS and Client Azure technology and design techniques as well as experience working across large environments with multiple operating systems/infrastructure for large-scale programs (e.g., Expert Engineers) starting to be...

  • Big Data Engineer

    20 hours ago

    Plano, United States The AES Group Full time

    Job title: Big Data EngineerLocations: Plano, TXDuration: 12 MonthsNeed only USC & GC as this is C2HJob Family Description: Expert in AWS and Client Azure technology and design techniques as well as experience working across large environments with multiple operating systems/infrastructure for large-scale programs (e.g., Expert Engineers) starting to be...

  • Triage Software Engineer

    6 days ago

    Plano, United States Diverse Lynx Full time

    Job Title: Triage Software Engineer Job location: Plano, TX (Onsite) Job description:As an IAT engineer, you will be responsible for initial analysis and triage of defects before they are assigned to individual development teams for root cause and fix.You will perform high level defect analysis to filter out environmental/ setup issues, IVI vs server-side...

  • Senior Lead Engineer

    7 days ago

    Plano, United States NTT DATA Full time

    Company Overview Req ID: 268435 NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Senior Lead Engineer to join our team in Plano, Texas (US-TX), United States (US). Job...

  • Lead Software Engineer

    4 days ago

    Plano, United States JPMorgan Chase Bank, N.A. Full time

    We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible. As a Lead Software Engineer at JPMorgan Chase within the Corporate Technology - Data Services team, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure,...

  • Software Engineer

    2 weeks ago

    Plano, United States The Judge Group Full time

    Title: React JS – Advanced Location : Plano, TX Hybrid need someone with AWS & Pyspark with Databricks experience Description: Expert in React JS and design technique as well as experience working across large environments with multiple operating systems/infrastructure for large-scale programs (e.g., Expert Engineers) starting to be firm-wide resources...

  • Software Engineer

    2 weeks ago

    Plano, United States The Judge Group Full time

    Title: React JS – AdvancedLocation : Plano, TX Hybrid need someone with AWS & Pyspark with Databricks experienceDescription:Expert in React JS and design technique as well as experience working across large environments with multiple operating systems/infrastructure for large-scale programs (e.g., Expert Engineers) starting to be firm-wide resourcesFormal...