Databricks Engineer with Pyspark

2 days ago

Iselin, NJ, United States Zortech Solutions Full time
Role: Databricks Engineer with Pyspark & Python Exp

Location: NYC, NY OR Iselin, NJ (Hybrid - 3 days' work from office)

Duration: 6+ Months

Job Description:

Job Description:

This position is for a Cloud Data / Reporting engineer with a background in SQL and data warehousing for enterprise level systems. The position calls for someone that is comfortable working with business users along with business analyst expertise.

Major Responsibilities:

Design, develop, and deploy Databricks jobs to process and analyze large volumes of data.

Collaborate with data engineers and data scientists to understand data requirements and implement appropriate data processing pipelines.

Optimize Databricks jobs for performance and scalability to handle big data workloads.

Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks.

Implement best practices for data management, security, and governance within the Databricks environment. Experience designing and developing Enterprise Data Warehouse solutions.

Demonstrated proficiency with Data Analytics, Data Insights

Proficient writing SQL queries and programming including stored procedures and reverse engineering existing process

Leverage SQL, programming language (Python or similar) and/or ETL Tools (Azure Data Factory, Data Bricks, Talend and SnowSQL) to develop data pipeline solutions to ingest and exploit new and existing data sources.

Perform code reviews to ensure fit to requirements, optimal execution patterns and adherence to established standards.

Skills:

• 10+ years - Enterprise Data Management

• 10+ years - SQL Server based development of large datasets

• 5+ years with Data Warehouse Architecture, hands-on experience with Databricks platform. Extensive experience in PySpark coding. Snowflake experience is good to have

• 3+ years Python (numpy, pandas) coding experience

• 3+ years' experience in Finance / Banking industry - some understanding of Securities and Banking products and their data footprints.

• Experience with Snowflake utilities such as SnowSQL and SnowPipe - good to have

• Experience in Data warehousing - OLTP, OLAP, Dimensions, Facts, and Data modeling.

• Previous experience leading an enterprise-wide Cloud Data Platform migration with strong architectural and design skills

• Capable of discussing enterprise level services independent of technology stack

• Experience with Cloud based data architectures, messaging, and analytics

• Superior communication skills

• Cloud certification(s)

• Any experience with Regulatory Reporting is a Plus

Education:

• Minimally a BA degree within an engineering and/or computer science discipline

• Master's degree strongly preferred.

  • Databricks Engineer with Pyspark

    9 hours ago

    Iselin, NJ, United States Zortech Solutions Full time

    Hello Professionals, We have urgent requirement for Databricks Engineer with Pyspark & Python Exp Please have a look at the below job description, if interested please share your updated resume, email me, pavan@zortechsolutions.ca or share any References. Role: Databricks Engineer with Pyspark & Python Exp Location: NYC, NY OR Iselin, NJ (Hybrid - 3 days'...

  • Lead Databricks Engineer with Pyspark

    4 days ago

    Iselin, NJ, United States Zortech Solutions Full time

    Role: Lead Databricks Engineer with Pyspark & Python Exp Location: NYC, NY OR Iselin, NJ (Hybrid - 3 days' work from office) Duration: 6+ Months uery/Redshift experience is good to have 3+ years Python(numpy, pandas) coding experience Experience with Snowflake utilities such as SnowSQL and SnowPipe - good to have Experience in Data warehousing - OLTP, OLAP,...

  • Lead Databricks Engineer with Pyspark

    4 days ago

    Iselin, NJ, United States Zortech Solutions Full time

    Hello Professionals, We have urgent requirement for Lead Databricks Engineer with Pyspark & Python Exp Please have a look at the below job description, if interested please share your updated resume, email me, pavan@zortechsolutions.ca or share any References Role: Lead Databricks Engineer with Pyspark & Python Exp Location: NYC, NY OR Iselin, NJ (Hybrid - 3...

  • Lead Databricks Engineer with Pyspark

    2 days ago

    Iselin, NJ, United States Zortech Solutions Full time

    Hello Professionals, We have urgent requirement for Lead Databricks Engineer with Pyspark & Python Exp Please have a look at the below job description, if interested please share your updated resume, email me, pavan@zortechsolutions.ca or share any References Role: Lead Databricks Engineer with Pyspark & Python Exp Location: NYC, NY OR Iselin, NJ (Hybrid - 3...

  • Lead Databricks Engineer with Pyspark

    10 minutes ago

    Iselin, NJ, United States Zortech Solutions Full time

    Hello Professionals, We have urgent requirement for Lead Databricks Engineer with Pyspark & Python Exp Please have a look at the below job description, if interested please share your updated resume, email me, pavan@zortechsolutions.ca or share any References Role: Lead Databricks Engineer with Pyspark & Python Exp Location: NYC, NY OR Iselin, NJ (Hybrid - 3...

  • Senior DevOps Engineer

    2 weeks ago

    Iselin, NJ, United States ApTask Full time

    About Client: The client is a global consulting, technology, and digital firm specializing in the financial services industry. Established in 2001, the company provides innovative solutions to help clients navigate digital transformation, enhance operational efficiency, and achieve strategic objectives. With a focus on banking, capital markets, and...

  • Data Engineer

    2 days ago

    Iselin, NJ, United States RIT Solutions, Inc. Full time

    Title: Data Engineer Location: Hybrid 3 times a week too Iselin, NJ May Need to come onsite for final or an Apex Local office for final IV feedback: Quick update on this we have had two interviews go off. First candidate was not hand on enough Second candidate was good, but didn't have java/spark that was needed at the moment. So I am looking for hands on...

  • Data Engineer

    2 days ago

    Iselin, NJ, United States RIT Solutions, Inc. Full time

    Title: Data Engineer Location: Hybrid 3 times a week too Iselin, NJ May Need to come onsite for final or an Apex Local office for final IV feedback: Quick update on this we have had two interviews go off. First candidate was not hand on enough Second candidate was good, but didn't have java/spark that was needed at the moment. So I am looking for hands on...

  • Data Engineer

    8 hours ago

    Iselin, NJ, United States RIT Solutions, Inc. Full time

    Title: Data Engineer Location: Hybrid 3 times a week too Iselin, NJ May Need to come onsite for final or an Apex Local office for final IV feedback: Quick update on this we have had two interviews go off. First candidate was not hand on enough Second candidate was good, but didn't have java/spark that was needed at the moment. So I am looking for hands on...

  • Azure Databricks Admin- US

    4 days ago

    Iselin, NJ, United States Zortech Solutions Full time

    Role: Azure Databricks Admin Location: Iselin NJ / New York (Hybrid Mode) Duration: 6+ Months Job Description: The perfect candidate has to have a mix of Azure Administration skills and Databricks setup/configuration/integration skills. Note: This is a hybrid role and candidate can work from either Iselin NJ or New York office 2-3 days/week, so its...