Senior Databricks Engineer

Senior Databricks Engineer

Associate Data Architect I

Who We Are:

Born digital, UST transforms lives through the power of technology. We walk alongside our clients and partners, embedding innovation and agility into everything they do. We help them create transformative experiences and human-centered solutions for a better world.

UST is a mission-driven group of 29,000+ practical problem solvers and creative thinkers in more than 30 countries. Our entrepreneurial teams are empowered to innovate, act nimbly, and create a lasting and sustainable impact for our clients, their customers, and the communities in which we live.

With us, you’ll create a boundless impact that transforms your career—and the lives of people across the world.

Visit us at UST.com.

You Are:

UST is searching for a Senior Databricks Engineer with deep hands-on experience designing, developing, securing, and deploying complex enterprise data pipelines on AWS Databricks within the US healthcare industry. The engineer will build scalable batch and streaming data pipelines supporting clinical, claims, analytics, and machine learning use cases.

The opportunity:

· Design, develop, test, deploy, and support complex batch and streaming data pipelines using AWS Databricks

· Design scalable lakehouse solutions using Delta Lake and Apache Iceberg table formats

· Implement enterprise data governance using Databricks Unity Catalog, including catalogs, schemas, tables, views, volumes, external locations, and governed data access

· Design and implement complex Databricks RBAC and access-control models across users, groups, service principals, workspaces, catalogs, schemas, tables, views, external locations, and other governed resources

· Implement secure data-access patterns following least privilege, separation of duties, and role-based access control principles

· Design and implement appropriate data tokenization, masking, encryption, and de-identification techniques for sensitive healthcare data, including PHI/PII

· Ensure encryption is appropriately implemented for data at rest and data in transit, including integration with enterprise key-management and security controls where applicable

· Develop high-volume data ingestion and transformation pipelines using PySpark, Spark SQL and Databricks-native capabilities

· Develop and optimize both batch and real-time/streaming pipelines

· Work with SMEs to understand healthcare data structures, business rules, data lineage, quality requirements, and transformation logic

· Partner with Machine Learning Engineers to create secure, governed, high-quality datasets for ML/AI workloads

· Collaborate with AWS DevOps engineers on CI/CD, infrastructure automation, deployment, monitoring, logging, and production operations

This position description identifies the responsibilities and tasks typically associated with the performance of the position. Other relevant essential functions may be required .

What you need :

· 5+ years of professional data engineering experience, with substantial recent hands-on experience developing production solutions on Databricks

· Strong hands-on expertise with Apache Spark / PySpark and advanced SQL

· Strong experience designing and implementing enterprise-scale batch and streaming data pipelines

· Deep understanding of distributed data processing, partitioning, schema evolution, fault tolerance, performance optimization, and scalability

· Strong experience with Delta Lake and lakehouse architecture

· Hands-on experience with Apache Iceberg table format, including table design, partitioning, schema evolution, interoperability, and production implementation

· Experience working with multiple enterprise source systems and downstream analytical, reporting, and ML platforms

· Strong hands-on experience with Databricks Unity Catalog in an enterprise environment

· Strong understanding of Databricks privilege hierarchies and permission inheritance

· Experience implementing least-privilege access, segregation of duties, and environment-specific access controls across development, test/UAT, and production

· Experience designing secure access patterns for both human users and machine/service identities

· Ability to troubleshoot complex authorization and access-control issues in Databricks

· Hands-on experience implementing data tokenization, masking, anonymization, or de-identification techniques for sensitive enterprise data

· Strong understanding of protecting PHI, PII, and other sensitive healthcare information

· Experience implementing encryption for data exchanged betw