Senior Data Engineer-Databricks, Python

2 weeks ago


Jersey City, United States Zortech Solutions Full time
Job DescriptionJob Description

Role: Senior Data Engineer-Databricks, Python

Location: New Jersey(Hybrid)

Duration: 6+ Months

Job Description:

Must have: Databricks, Spark, Python, ADF, SQL

Responsibilities

Design, develop, and maintain data pipelines using Databricks and Spark, and other cloud technologies as needed

Optimize data pipelines for performance, scalability, and reliability

Ensure data quality and integrity throughout the data lifecycle

Collaborate with data scientists, analysts, and other stakeholders to understand and meet their data needs

Troubleshoot and resolve data-related issues, and provide root cause analysis and recommendations

Document data pipeline specifications, requirements, and enhancements, and communicate them effectively to the team and management

Create new data validation methods and data analysis tools, and share best practices and learnings with the data engineering community

Implement ETL processes and data warehouse solutions, and ensure compliance with data governance and security policies

Requirements

Qualifications

Bachelor's degree in Computer Science, Engineering, or related field, or equivalent work experience

5+ years of experience in data engineering with Databricks and Spark

Proficient in SQL and Python andPyspark

Experience with Azure Databricks Medallion Architecture with DLT, Iceberg

Financial/Corporate Banking context would be a plus

Experience with data integration and ETL tools, such as Azure Data Factory

Experience with Azure cloud platform and services

Experience with data warehouse and data lake concepts and architectures

Good to have experience with big data technologies, such as Kafka, Hadoop, Hive, etc

Strong analytical and problem-solving skills

Excellent communication and teamwork skills
  • Databricks Engineer

    3 weeks ago


    Jersey City, United States CHUBB Full time

    Job Description: We are searching for a seasoned Databricks Engineer to pioneer advanced Databricks solutions. The Databricks Engineer will serve as a central figure in our data engineering team, delivering highly sophisticated technological solutions using Databricks, Delta Live Tables, Snowflake, and Medallion Architecture. Key Responsibilities: ...


  • Jersey City, United States KTek Resourcing Full time

    Responsibilities:Evaluate and understand the latest features and capabilities of the Databricks platform and assess their potential impact on our organization's data ecosystem.Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions using Databricks.Help to design, develop, and implement scalable...


  • Jersey City, United States KTek Resourcing Full time

    Responsibilities:Evaluate and understand the latest features and capabilities of the Databricks platform and assess their potential impact on our organization's data ecosystem.Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions using Databricks.Help to design, develop, and implement scalable...


  • Jersey City, United States Diverse Lynx Full time

    Senior Data Engineer with SAS/Databricks/PySpark Locations Mt Laurel, NJ & Charlotte, NC (Hybrid Work: 3 Days Onsite Per Week) Required Experience: 10-12+ Years Primary Tech skills SAS, Databricks, Pyspark Secondary Tech skills Azure, Synapse Experience in building end to end architecture for Data Lakes, Data Warehouses and Data Marts Experience...

  • Data Engineer

    2 weeks ago


    Jersey City, United States ClifyX Full time

    Job title : Data Engineer - Databricks Location : New Jersey (Remote) Duration : Contract / Fulltime Health Care Company $75/Hr (Contract) salary 120-135K/Annum (Fulltime) Job Description: • Experience with PySpark or Python • DataBricks experience is mandatory • Experience with big data tools Spark, Hadoop Kafka would be good to have • Hands on...

  • Databricks Architect

    3 weeks ago


    Jersey City, United States Inizio Partners Full time

    Role & Responsibilities Overview:Develop and optimize ETL pipelines from various data sources using Databricks on cloud (AWS, Azure, etc.)Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD, Terraform for infrastructure as code, and Splunk for monitoringContinuously improve systems through...


  • Jersey City, United States Inizio Partners Corp Full time

    Job DescriptionJob DescriptionRole & Responsibilities Overview:Develop and optimize ETL pipelines from various data sources using Databricks on cloud (AWS, Azure, etc.)Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD, Terraform for infrastructure as code, and Splunk for...

  • Databricks Architect

    4 weeks ago


    Jersey City, United States Inizio Partners Corp Full time

    Job DescriptionJob DescriptionRole & Responsibilities Overview:Develop and optimize ETL pipelines from various data sources using Databricks on cloud (AWS, Azure, etc.)Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD, Terraform for infrastructure as code, and Splunk for...

  • Cloud Data Engineer

    2 days ago


    Jersey City, United States ShiftCode Analytics Full time

    Title: Cloud Data Engineer Location: Charlotte, NC This role is 100% remote But Local Candidate is highly Preferred Sector: Banking Duration: 8 month (C2H) (USC or GC candidates Only***) Job Summary: Project Details: Job Description: Be the subject matter expert for data, pipeline design, architecture, and other data engineering technologies for the...

  • Cloud Data Engineer

    2 weeks ago


    Jersey City, United States ShiftCode Analytics Full time

    Title: Cloud Data Engineer Location: Charlotte, NC This role is 100% remote But Local Candidate is highly Preferred Sector: Banking Duration: 8 month (C2H) (USC or GC candidates Only***) Job Summary: Project Details: Job Description: Be the subject matter expert for data, pipeline design, architecture, and other data engineering technologies for the...

  • Senior Data Engineer

    2 weeks ago


    Arizona City, United States Saxon Global Full time

    UnitedHealth Group, Inc. (UHG) is looking for a Senior Data Engineer to work 100% remote (will work 9am-5pm either AZ time zone or EST time zone) to design, develop, test, and maintain pipelines ingesting critical real-time claims data sets into OnePaaS (Cloud). Project Initiative: There is need to ingest critical claims datasets (real time) into the cloud...


  • Jersey City, New Jersey, United States Tiger Analytics, LLC Full time

    As a Principal Data Engineer (Azure), you would have hands on experience working on Azure as cloud, Databricks and some exposure/experience on Data Modelling.using different Open Source, Big Data, and Cloud technologies on Microsoft Azure.Experience in implementing Data Lake with technologies like Azure Data Factory (ADF), PySpark, Databricks,...

  • Data Engineer

    2 weeks ago


    Jersey City, United States A.M. Best Full time

    Overview The Data Engineer will build data pipelines / data orchestrator and data structures for existing operational data stores, Azure data lake & data warehouse. They will develop proof-of-concepts to evaluate new technologies and suggest improved and better ways to re-architect existing data loading processes and data structures. They will work...


  • Kansas City, United States ClifyX Full time

    RR-344784.1 zure Databricks Architect Kansas City,Missouri Day 1 onsite Burns & McDonnell Pay Rate $95/hr or $150 K customer was explicitly looking for a profile with design experience in Databricks Medallion architecture, and hands-on development as well. • Must have minimum of 3 Years of Data & Analytics Solution Experience as an Architect • Overall...

  • Data Engineer

    2 weeks ago


    Jersey City, United States Diverse Lynx Full time

    Job Title: Data Engineer ( Python + AWS) Location : Jersey City , NJ (Onsite, Local s only) Job responsibilities : Executes creative software solutions, design, development, and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problems Develops secure high-quality...


  • Jersey City, United States Kavyos Consulting Full time

    As a leader of our data engineering team, you'll help define the foundation that is critical for analyzing everything from heavy, cross-channel analysis of media interaction logs to optimizing Paid Media to distilling purchase data down to understanding our customers better to evaluating our warehouse and sourcing data to improve our operational efficiency....


  • Jersey City, United States Kavyos Consulting Full time

    As a leader of our data engineering team, you'll help define the foundation that is critical for analyzing everything from heavy, cross-channel analysis of media interaction logs to optimizing Paid Media to distilling purchase data down to understanding our customers better to evaluating our warehouse and sourcing data to improve our operational efficiency....


  • Foster City, United States Royalty Staffing Full time

    Royalty Staffing is helping an on-demand, autonomous ride-hailing company hire an experienced Business Intelligence Administrator to manage and maintain company data stack tools.In this role, you will be responsible for various duties that will include user management, data modeling/ETL, building pipelines, system configuration, security and providing...


  • Foster City, United States Royalty Staffing Full time

    Royalty Staffing is helping an on-demand, autonomous ride-hailing company hire an experienced Business Intelligence Administrator to manage and maintain company data stack tools.In this role, you will be responsible for various duties that will include user management, data modeling/ETL, building pipelines, system configuration, security and providing...


  • Foster City, United States Royalty Staffing Full time

    Royalty Staffing is helping an on-demand, autonomous ride-hailing company hire an experienced Business Intelligence Administrator to manage and maintain company data stack tools.In this role, you will be responsible for various duties that will include user management, data modeling/ETL, building pipelines, system configuration, security and providing...