Engineering & Data Science Specialist, Applied AI Evaluations

About The Team
BMO's Applied AI team sets the standard for safe, high-performing AI systems across the bank, building BMO's leading-edge AI products and capabilities. Our work enables BMO's strategic outcomes, bridging business needs with scalable and responsible AI. If you thrive on technical challenges and want to make a strategic impact, this role is for you.

About The Role
As an Engineering & Data Science Specialist in Applied AI Evaluations, you will design, implement, and optimize evaluation frameworks and pipelines that measure the quality, safety, and performance of advanced AI systems at BMO. You'll work closely with the Product Owner and collaborate with engineering, research, and business teams to translate evaluation standards into robust, scalable solutions.

Your work will directly impact model development, deployment, and continuous improvement cycles—ensuring that evaluation signals are deeply integrated into experimental and production systems. You'll contribute to the development of reusable tools, agents, and harnesses that support responsible AI practices and regulatory compliance.

Key Responsibilities

• Develop, implement, and maintain evaluation pipelines and harnesses for AI models, ensuring reliability, reproducibility, and scalability for banking applications.
• Collaborate with the Product Owner to translate evaluation standards and business requirements into technical solutions.
• Prototype and validate evaluation methods using real banking workflows, integrating feedback into model training and deployment.
• Integrate evaluation metrics and signals into research, training, and production systems to drive measurable improvements in model performance and customer outcomes.
• Partner with engineering, research, and product teams to shape model interaction paradigms and deployment strategies.
• Build reusable tools and systems that enable contributions from across BMO and raise the quality bar for AI solutions.
• Ensure all evaluation frameworks adhere to regulatory, compliance, and Responsible AI requirements.
• Document processes, share knowledge, and contribute to a culture of continuous learning and responsible innovation.

Qualifications

• 3+ years of experience in engineering or data science roles, preferably in AI, machine learning, or data-driven product environments.
• Strong programming skills (e.g., Python, Java, or similar) and experience with ML/AI frameworks (e.g., PyTorch, TensorFlow).
• Experience building and maintaining evaluation pipelines for large language models (LLMs) and/or other AI systems, including multi-agent workflows, tool use, or long-context scenarios.
• Solid understanding of deep learning concepts, model training, and deployment practices.
• Familiarity with MLOps, CI/CD, and cloud-based AI infrastructure is a plus.
• Strong analytical and problem-solving skills, with the ability to communicate technical concepts to both technical and non-technical audiences.
• Motivated by high-impact collaboration and able to thrive in fast-paced, ambiguous environments.
• Commitment to BMO's values of inclusion, integrity, and responsible innovation.

Salary:
$122, $228,000.00

Pay Type:
Salaried

The above represents BMO Financial Group's pay range and type.

Salaries will vary based on factors such as location, skills, experience, education, and qualifications for the role, and may include a commission structure. Salaries for part-time roles will be pro-rated based on number of hours regularly worked. For commission roles, the salary listed above represents BMO Financial Group's expected target for the first year in this position.

BMO Financial Group's total compensation package will vary based on the pay type of the position and may include performance-based incentives, discretionary bonuses, as well as other perks and rewards. BMO also offers health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans. To view more details of our benefits, please visit:

About Us
At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life. It calls on us to create lasting, positive change for our customers, our communities and our people. By working together, innovating and pushing boundaries, we transform lives and businesses, and power economic growth around the world.

As a member of the BMO team you are valued, respected and heard, and you have more ways to grow and make an impact. We strive to help you make an impact from day one – for yourself and our customers. We'll support you with the tools and resources you need to reach new milestones, as you help our customers reach theirs. From in-depth training and coaching, to manager support and network-building opportunities, we'll help you gain valuable experience, and broaden your skillset.

To find out more visit us at

BMO is proud to be an equal employment opportunity employer. We evaluate applicants without regard to race, religion, color, national origin, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, gender expression, transgender status, sexual stereotypes, age, status as a protected veteran, status as an individual with a disability, or any other legally protected characteristics. We also consider applicants with criminal histories, consistent with applicable federal, state and local law.

BMO is committed to working with and providing reasonable accommodations to individuals with disabilities. If you need a reasonable accommodation because of a disability for any part of the employment process, please send an e-mail to and let us know the nature of your request and your contact information.

Note to Recruiters: BMO does not accept unsolicited resumes from any source other than directly from a candidate. Any unsolicited resumes sent to BMO, directly or indirectly, will be considered BMO property. BMO will not pay a fee for any placement resulting from the receipt of an unsolicited resume. A recruiting agency must first have a valid, written and fully executed agency agreement contract for service to submit resumes.