Databricks Architect

2 weeks ago

Dublin, United States Smart IT Frame LLC Full time

Role & Responsibilities Overview:

 Develop and optimize ETL pipelines from various data sources using Databricks on cloud (AWS, Azure, etc.)

 Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD, Terraform for infrastructure as code, and Splunk for monitoring

 Continuously improve systems through performance enhancements and cost reductions in compute and storage

 Data Processing and API Integration: Utilize Spark Structured Streaming for real-time data processing and integrate data outputs with REST APIs

 Lead Data Engineering Projects to manage and implement data-driven communication systems

 Experienced with Scrum and Agile Methodologies to coordinate global delivery teams, run scrum ceremonies, manage backlog items, and handle escalations

 Integrate data across different systems and platforms

 Strong verbal and written communication skills to manage client discussions


Candidate Profile:

 8+ years experience in developing and implementing ETL pipelines from various data sources using Databricks on cloud

 Based out of US

 Some experience in insurance domain/ data is must

 Programming Languages – SQL, Python

 Technologies - IaaS (AWS or Azure or GCP), Databricks platform, Delta Lake storage, Spark (PySpark, Spark SQL).

o Good to have - Airflow, Splunk, Kubernetes, Power BI, Git, Azure Devops

 Project Management using Agile, Scrum

 B.S. Degree in a data-centric field (Mathematics, Economics, Computer Science, Engineering or other science field), Information Systems, Information Processing or engineering.

 Excellent communication & leadership skills, with the ability to lead and motivate team members

  • Databricks Architect _US

    2 days ago

    Dublin, United States Zortech Solutions Full time

    Job DescriptionJob DescriptionRole: Databricks ArchitectLocation: Dublin, OHExperience: 8 to 15Must Have:Azure Databricks. Pyspark, SQL, CI/CD Pipelines for Databricks, ETL Processes, Project Management, Delta Lake StorageNice to Have: Airflow, Splunk, Kubernetes, PowerBI, Git, Azure Dev Ops

  • Databricks Architect

    2 weeks ago

    Dublin, United States Inizio Partners Full time

    About the job Databricks Architect OUR CLIENT Our client provides data-driven, action-oriented solutions to business problems through statistical data mining, cutting-edge analytics techniques, and a consultative approach. Leveraging proprietary methodology and best-of-breed technology, our client's analytics team takes an industry-specific approach to...

  • Databricks Engineer Lead

    1 day ago

    Dublin, United States Prodware Solutions Full time

    Job Details:Position: Sr. Databricks Architect / Engineer LeadLocation: Columbus Metro area, OHDuration: Full Time OpportunityNeed to travel every other week (twice a month) 3-4 days each trip --- Columbus Metro area, OHTravel expenses will be paidRequired Skills: Azure Databricks, Pyspark, SQL, CI/CD Pipelines for Databricks, ETL Processes, Project...

  • Databricks Engineer Lead

    1 day ago

    Dublin, United States Prodware Solutions Full time

    Job Details:Position: Sr. Databricks Architect / Engineer LeadLocation: Columbus Metro area, OHDuration: Full Time OpportunityNeed to travel every other week (twice a month) 3-4 days each trip --- Columbus Metro area, OHTravel expenses will be paidRequired Skills: Azure Databricks, Pyspark, SQL, CI/CD Pipelines for Databricks, ETL Processes, Project...

  • Databricks Engineer Lead

    6 hours ago

    Dublin, United States Prodware Solutions Full time

    Job Details:Position: Sr. Databricks Architect / Engineer LeadLocation: Columbus Metro area, OHDuration: Full Time OpportunityNeed to travel every other week (twice a month) 3-4 days each trip --- Columbus Metro area, OHTravel expenses will be paidRequired Skills: Azure Databricks, Pyspark, SQL, CI/CD Pipelines for Databricks, ETL Processes, Project...

  • Azure Data Architect

    4 weeks ago

    Dublin, Ohio, United States Oxenham Group LLC Full time

    5+ years experience in architecture, design and implementation using DatabricksDemonstrated experience with the deployment of Databricks on cloud platforms , including advanced configurationsIn depth knowledge of Spark internals, catalyst optimization, and Databricks runtime environmentMust have experience in implementing solution using DatabricksAirflow,...