Databricks Engineer with Pyspark
13 hours ago
Location: NYC, NY OR Iselin, NJ (Hybrid - 3 days' work from office)
Duration: 6+ Months
Job Description:
Inperson interview is must, Expenses will be paid for in-person if not local
Job Description:
This position is for a Cloud Data / Reporting engineer with a background in SQL and data warehousing for enterprise level systems. The position calls for someone that is comfortable working with business users along with business analyst expertise.
Major Responsibilities:
Design, develop, and deploy Databricks jobs to process and analyze large volumes of data.
Collaborate with data engineers and data scientists to understand data requirements and implement appropriate data processing pipelines.
Optimize Databricks jobs for performance and scalability to handle big data workloads.
Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks.
Implement best practices for data management, security, and governance within the Databricks environment. Experience designing and developing Enterprise Data Warehouse solutions.
Demonstrated proficiency with Data Analytics, Data Insights
Proficient writing SQL queries and programming including stored procedures and reverse engineering existing process
Leverage SQL, programming language (Python or similar) and/or ETL Tools (Azure Data Factory, Data Bricks, Talend and SnowSQL) to develop data pipeline solutions to ingest and exploit new and existing data sources.
Perform code reviews to ensure fit to requirements, optimal execution patterns and adherence to established standards.
Skills:
• 10+ years - Enterprise Data Management
• 10+ years - SQL Server based development of large datasets
• 5+ years with Data Warehouse Architecture, hands-on experience with Databricks platform. Extensive experience in PySpark coding. Snowflake experience is good to have
• 3+ years Python (numpy, pandas) coding experience
• 3+ years' experience in Finance / Banking industry - some understanding of Securities and Banking products and their data footprints.
• Experience with Snowflake utilities such as SnowSQL and SnowPipe - good to have
• Experience in Data warehousing - OLTP, OLAP, Dimensions, Facts, and Data modeling.
• Previous experience leading an enterprise-wide Cloud Data Platform migration with strong architectural and design skills
• Capable of discussing enterprise level services independent of technology stack
• Experience with Cloud based data architectures, messaging, and analytics
• Superior communication skills
• Cloud certification(s)
• Any experience with Regulatory Reporting is a Plus
Education:
• Minimally a BA degree within an engineering and/or computer science discipline
• Master's degree strongly preferred
-
Azure Data Engineer
1 week ago
Jersey City, NJ, United States André Global, Inc. Full timeAzure Data Engineer: * 10+ years of strong experience in a large & complex application development environment, last 2 projects in investment banking or financial services industry and on data-heavy applications and systems * Strong programming experience in PySpark & Python or similar server-side languages * Proven experience as an Azure Data Engineer or in...
-
Azure Data Engineer
1 week ago
Jersey City, NJ, United States Andr Global, Inc. Full timeAzure Data Engineer: * 10+ years of strong experience in a large & complex application development environment, last 2 projects in investment banking or financial services industry and on data-heavy applications and systems * Strong programming experience in PySpark & Python or similar server-side languages * Proven experience as an Azure Data Engineer or in...
-
Azure Databricks Architect
4 days ago
Jersey City, NJ, United States NR Consulting Full timeRole: zure Databricks Architect Location: Raleigh, NC/Jersey City, NJ Type: Contract Duration: Long Term Mandatory Skill zure Data Architect Experience Synapse to Databricks implement Databricks Hand's on Experience DF & Synapse Data modeling, designing strategy & data architecture concepts Cloud technologies Experience Synapse...
-
Azure Databricks Architect
2 days ago
Jersey City, NJ, United States NR Consulting Full timeRole: zure Databricks Architect Location: Raleigh, NC/Jersey City, NJ Type: Contract Duration: Long Term Mandatory Skill zure Data Architect Experience Synapse to Databricks implement Databricks Hand's on Experience DF & Synapse Data modeling, designing strategy & data architecture concepts Cloud technologies Experience Synapse...
-
Jersey City, NJ, United States Axiom Software Solutions Limited Full timeA senior technical expert responsible for designing, building, and optimizing data solutions using Databricks and Azure Data technologies. This principal engineer oversees the architecture, integration, design, development and deployment of reusable patterns and solutions ensuring that they are scalable, secure and efficient. Also, responsible for managing...
-
Senior Data Engineer in
4 days ago
Jersey City, NJ, United States Lorven Technologies Full timeSenior Data Engineer specialising in Master Data Management (MDM) Position: W2 Location: Iselin, NJ (Hybrid) Duration: 12+ months Job description: Required Qualifications: • 12+ years of experience in data engineering, with a proven track record of MDM implementations, preferably in the financial services industry. • Extensive hands-on experience...
-
Senior Data Engineer in
6 days ago
Jersey City, NJ, United States Lorven Technologies Full timeSenior Data Engineer specialising in Master Data Management (MDM) Position: W2 Location: Iselin, NJ (Hybrid) Duration: 12+ months Job description: Required Qualifications: • 12+ years of experience in data engineering, with a proven track record of MDM implementations, preferably in the financial services industry. • Extensive hands-on experience...
-
Senior Data Engineer in
2 weeks ago
Jersey City, NJ, United States Lorven Technologies Full timeSenior Data Engineer specialising in Master Data Management (MDM) Position: W2 Location: Iselin, NJ (Hybrid) Duration: 12+ months Job description: Required Qualifications: • 12+ years of experience in data engineering, with a proven track record of MDM implementations, preferably in the financial services industry. • Extensive hands-on experience...
-
Senior Data Engineer in
2 weeks ago
Jersey City, NJ, United States Lorven Technologies Full timeSenior Data Engineer specialising in Master Data Management (MDM) Position: W2 Location: Iselin, NJ (Hybrid) Duration: 12+ months Job description: Required Qualifications: • 12+ years of experience in data engineering, with a proven track record of MDM implementations, preferably in the financial services industry. • Extensive hands-on experience...
-
Associate Principal
3 days ago
Jersey City, NJ, United States LTIMindtree Full timeRole description Azure Databricks Architect Must Have 12 years of experience of working on Azure and Databricks Lakehouse implementation with grasp of following o Translate business requirement into technical solution for data curation and consumptions o Have experience in building Databricks Lakehouse from a modelling and transformation ELT perspective...
