Director Data Engineering

Core Specialty's Data Environment is the organizational data ecosystem that supports enterprise requirements for data management, consolidation, insights, and reporting. The Data & Analytics organization ingests and manages data from over 200 distinct sources and growing, which serves business intelligence, AI/ML models, and enterprise decision-making. Maintaining the integrity, accuracy, and completeness of this extensive data environment requires a scalable and automated quality assurance strategy. The Director Of Engineering Qa Automation will be responsible for designing and implementing this strategy, integrating quality processes throughout the data platformincluding ingestion pipelines, semantic models, and machine learning workflows. This role will require 40% hands on involvement to start in order to build out the architecture, blueprints, and coding standards required for QA Engineers to adhere to. It may also include development efforts as needed.

Key Accountabilities/Deliverables:

• Define and lead the QA automation strategy for data engineering, analytics, and machine learning initiatives.
• Architect and implement scalable, reusable automation frameworks for data pipelines, ingestion models, semantic layers, and ML workflows.
• Collaborate with Data Engineering, Data Science, and DevOps teams to integrate automated testing into CI/CD pipelines.
• Establish coding standards, architectural patterns, and best practices for test automation across the organization.
• Develop automated validation for data quality, schema enforcement, lineage tracking, and model performance.
• Mentor and grow a high-performing QA team with deep domain knowledge in data and analytics.
• Drive continuous improvement in testing tools, processes, and observability.
• In addition to the above key responsibilities, you may be required to undertake other duties from time to time as the Company may reasonably require.

Technical Knowledge and Understanding:

• Programming Languages: Python (preferred), SQL, Java/Scala (for Spark-based pipelines)
• Automation Frameworks: PyTest, Great Expectations, dbt tests, Selenium, Airflow test suites
• Data Technologies: Spark, Kafka, Airflow, dbt, Delta Lake, Parquet, Avro
• Cloud Platforms: AWS (Glue, S3, Redshift), Azure, GCP (BigQuery, Dataflow), Databricks
• Data Warehouses: Snowflake, BigQuery, Redshift
• CI/CD & DevOps: GitHub Actions, Jenkins, Docker, Kubernetes
• ML Testing: MLflow, TensorFlow Model Analysis, custom validation pipelines
• Observability & Monitoring: Monte Carlo, Databand, OpenLineage
• SDLC: Agile/SCRUM Frameworks, Kanban, JIRA
• Data Quality Management: MS Purview

Preferred Skills:

• Experience with synthetic data generation and data fuzzing tools
• Experience with Data Contracts and the Producer vs. Consumer model
• Familiarity with data governance and compliance frameworks (e.g., GDPR, HIPAA)
• Exposure to AI-assisted testing platforms

Experience:

• Bachelor's degree in computer science, data engineering, or related field required
• Master's degree in computer science, data engineering, or related field preferred
• Minimum of 15 years of experience in Data Engineering (hands-on roles)
• Minimum of 5 years of experience in a leadership position
• Prior QA and automation experience within data engineering space required
• Proven experience in testing large-scale data systems and ML workflows
• Strong architectural thinking and ability to define scalable testing patterns
• Excellent leadership, communication, and cross-functional collaboration skills

The expected pay range for the role is $150,000 - $180,000. The specific offer will depend on an applicant's skills and experiences. The disclosed pay range estimate may also be adjusted for the applicable geographic differential for the location in which the position is filled.

Applicants must be authorized to work for any employer in the U.S. We are unable to sponsor or take over sponsorship of an employment Visa for this position. #LI-Hybrid

At Core Specialty, you will receive a competitive salary and opportunities for professional development and advancement. We offer medical, dental, vision, and life insurances; short and long-term disability; a Company-match of 100% of a 6% contribution 401(k) plan; an Employee Assistance Plan; Health Savings Account, Flexible Spending Account, Health Reimbursement Account, and a wellness program