Databricks Architect

Job ID

Location
REMOTE WORK, DC, US

Date Posted

Category
Engineering and Sciences

Subcategory
Solutions Archt

Schedule
Full-time

Shift
Day Job

Travel
No

Minimum Clearance Required None
Clearance Level Must Be Able to Obtain
Public Trust

Potential for Remote Work
Yes

Description
We are seeking an experienced and versatile
Databricks Architect
to lead the architectural design and implementation of Databricks-based solutions within a broader AI/ML platform environment. This role is part of the core platform team enabling scalable, secure, and reusable data science and GenAI workflows across federal mission use cases. The ideal candidate will have strong data architecture skills, a deep understanding of the AI/ML lifecycle, and familiarity with both traditional and emerging GenAI technologies.

This architect plays a vital role in enabling the
full lifecycle of AI/ML model development
for IRS data scientists—from data ingestion and feature engineering to model training, deployment, and monitoring. As the platform team serves data science users directly, the architect must ensure Databricks capabilities are accessible, extensible, and aligned with user workflows.

You will collaborate with infrastructure, security, compliance, and AI engineering teams to ensure Databricks capabilities are integrated seamlessly with supporting services such as S3, Amazon SageMaker, Bedrock, and shared pipeline frameworks. You will also work closely with the
Product Manager and Chief Architect
to inform roadmap evolution and ensure technical feasibility, performance, and alignment with broader architectural direction. As a hands-on architect, you will guide the design of workspaces, cluster policies, CI/CD integration, and governance structures that enable secure and efficient AI/ML model development and deployment.

Key Responsibilities

• Lead the architecture and design of Databricks environments (multi-tenant, secure, scalable) to support diverse AI/ML and GenAI workloads.
• Define and maintain workspace patterns, access policies, cluster configurations, and notebook orchestration aligned to platform governance.
• Collaborate with AI/ML engineers, data scientists, and platform SMEs to optimize data pipelines, feature stores, and model training processes.
• Drive integration of Databricks with enterprise data sources and services including AWS S3, IAM, SageMaker, Bedrock, and Terraform-managed infrastructure.
• Support development of reusable accelerators, templates, and onboarding patterns to reduce friction for mission teams.
• Provide architectural input into CI/CD pipelines for notebook deployment, model packaging, and automation of data workflows.
• Work with architects from adjacent platform areas (infrastructure, security, LLM, CSx) to ensure end-to-end solution alignment.
• Participate in planning and estimation activities with service delivery and product leadership.

Qualifications
Required Qualifications

• Bachelor's or master's degree in computer science, Data Engineering, or a related technical field.
• 8+ years of experience in data architecture, platform engineering, or advanced analytics.
• 5+ years of hands-on experience designing and implementing Databricks environments in AWS.
• Must be a U.S. Citizen with the ability to obtain and maintain a Public Trust security clearance.
• Strong working knowledge of Spark, Delta Lake, MLflow, Unity Catalog, and Databricks Runtime for ML.
• Solid foundation in data modeling, ETL pipeline design, and modern data lake architecture.
• Familiarity with MLOps practices, model lifecycle management, and compliance-aligned deployment patterns.

Desired Skills

• Active IRS Security Clearance is highly desired.
• Awareness of GenAI integration patterns and experience supporting LLM experimentation or prompt-based workflows.
• Experience working with infrastructure-as-code and CI/CD tools (Terraform, Bitbucket, Bamboo).
• Understanding of government compliance standards (FedRAMP, NIST, SOC2) and secure boundary design.
• Ability to collaborate across multiple engineering teams to support reusable, well-governed platform architecture.
• Excellent communication skills and technical documentation ability.

Target salary range $200,001 - $240,000. The estimate displayed represents the typical salary range for this position based on experience and other factors.