Databricks Architect
2 months ago
Role: Decision Analytics/Databricks Architect
Location: Dublin, Ohio, New Jersey-Travel once in a month
Duration: Fulltime
Job Description:
Experience Level: 7 to 12
Must Have:
Strong in Data engineering
Strong developer profile
Strong Coding skills
Hand-on experience on:- Azure / AWS / Databricks (Good to have) / Analytics skills.
Experience- 7-12 years.
Role & Responsibilities Overview:
· Develop and optimize ETL pipelines from various data sources using Databricks on cloud (AWS, Azure, etc.)
· Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD, Terraform for infrastructure as code, and Splunk for monitoring
· Continuously improve systems through performance enhancements and cost reductions in compute and storage
· Data Processing and API Integration: Utilize Spark Structured Streaming for real-time data processing and integrate data outputs with REST APIs
· Lead Data Engineering Projects to manage and implement data-driven communication systems
· Experienced with Scrum and Agile Methodologies to coordinate global delivery teams, run scrum ceremonies, manage backlog items, and handle escalations
· Integrate data across different systems and platforms
· Strong verbal and written communication skills to manage client discussions
Candidate Profile:
· 8+ years experience in developing and implementing ETL pipelines from various data sources using Databricks on cloud
· Based out of US
· Some experience in insurance domain/ data is must
· Programming Languages – SQL, Python
· Technologies - IaaS (AWS or Azure or GCP), Databricks platform, Delta Lake storage, Spark (PySpark, Spark SQL).
o Good to have - Airflow, Splunk, Kubernetes, Power BI, Git, Azure Devops
· Project Management using Agile, Scrum
· B.S. Degree in a data-centric field (Mathematics, Economics, Computer Science, Engineering or other science field), Information Systems, Information Processing or engineering.
#J-18808-Ljbffr