Principal or Senior AI/ML Test Engineer

3 days ago

Chicago, IL, United States Spruce Infotech Full time

The AI/ML Test Engineer will serve as both a technical expert and consultative leader within their team, responsible for establishing and executing rigorous testing methodologies for software-based AI and machine learning solutions. These responsibilities includes developing best practices, defining test frameworks, validating performance metrics, and implementing and validating human-in-the-loop (HITL) evaluation processes to ensure they are properly functioning within established quality and ethical standards. This team will be the first of its kind within a large enterprise insurance company and will play a key role in establishing foundational AI/ML best practices and procedures, so candidates with expertise working in 'greenfield' environments are preferred.

Key Responsibilities

  • Define and implement testing frameworks and validation strategies for AI and machine learning applications
  • Establish best practices for AI model testing, including performance, bias, accuracy, interpretability, and reliability measures
  • Perform hands-on testing, model validation, and results analysis for LLMs and AI-driven systems
  • Incorporate and validate human-in-the-loop (HITL) methodologies to ensure proper implementation and alignment with ethical and quality standards
  • Develop automated testing workflows for reproducible model evaluation and regression analysis
  • Collaborate with data scientists, AI engineers, and DevOps teams to integrate quality validation throughout the AI development lifecycle
  • Analyze and document test results, generating actionable insights and improvement recommendations
  • Advise leadership and engineering teams on AI testing governance, risk mitigation, and continuous improvement strategies
  • Ensure compliance with internal standards and evolving AI governance and ethics practices
Required Skills & Experience
  • Bachelor's or Master's degree in Computer Science, Data Science, or a related field
  • 8+ years of professional experience in software quality engineering, with at least 3+ years focused on AI or ML systems
  • Deep understanding of AI/ML model development, evaluation metrics, and data validation techniques
  • Hands-on experience with testing tools and frameworks for Python, PyTest, and model evaluation environments
  • Experience testing solutions built on LLMs and generative AI frameworks
  • Knowledge of statistical validation, drift detection, A/B testing, and bias analysis
  • Strong understanding of AI ethics, governance, and human-centered testing principles
  • Excellent analytical and problem-solving skills, with the ability to communicate findings to both technical and business audiences
Preferred Experience
  • Familiarity with LangChain, MLflow, TensorFlow Extended (TFX), or MLOps testing pipelines
  • Demonstrated success establishing AI/ML testing best practices in enterprise environments

  • Principal AI Engineer

    4 days ago

    Chicago, IL, United States TAG - The Aspen Group Full time

    The Aspen Group (TAG) is one of the largest and most trusted retail healthcare business support organizations in the U.S. and has supported over 20,000 healthcare professionals and team members with close to 1,500 health and wellness offices across 48 states in four distinct categories: dental care, urgent care, medical aesthetics, and animal health. Working...

  • Principal AI Engineer

    2 days ago

    Chicago, IL, United States TAG - The Aspen Group Full time

    The Aspen Group (TAG) is one of the largest and most trusted retail healthcare business support organizations in the U.S. and has supported over 20,000 healthcare professionals and team members with close to 1,500 health and wellness offices across 48 states in four distinct categories: dental care, urgent care, medical aesthetics, and animal health. Working...

  • Principal AI Engineer

    2 days ago

    Chicago, IL, United States The Aspen Group Full time

    The Aspen Group (TAG) is one of the largest and most trusted retail healthcare business support organizations in the U.S. and has supported over 20,000 healthcare professionals and team members with close to 1,500 health and wellness offices across 48 states in four distinct categories: dental care, urgent care, medical aesthetics, and animal health. Working...

  • AI/ML Engineer

    7 hours ago

    Chicago, IL, United States MillerKnoll, Inc. Full time

    Why join us? Our purpose is to design for the good of humankind. It's the ideal we strive toward each day in everything we do. Being a part of MillerKnoll means being a part of something larger than your work team, or even your brand. We are redefining modern for the 21st century. And our success allows MillerKnoll to support causes that align with our...

  • Senior AI Engineer

    7 days ago

    Chicago, IL, United States Motorola Solutions Full time

    Company Overview At Motorola Solutions, we believe that everything starts with our people. We're a global close-knit community, united by the relentless pursuit to help keep people safer everywhere. Our critical communications, video security and command center technologies support public safety agencies and enterprises alike, enabling the coordination...

  • Senior AI Engineer

    13 hours ago

    Chicago, IL, United States Motorola Solutions Full time

    Company Overview At Motorola Solutions, we believe that everything starts with our people. We're a global close-knit community, united by the relentless pursuit to help keep people safer everywhere. Our critical communications, video security and command center technologies support public safety agencies and enterprises alike, enabling the coordination...

  • Senior AI Engineer

    1 week ago

    Chicago, IL, United States Motorola Solutions Full time

    Company Overview At Motorola Solutions, we believe that everything starts with our people. We're a global close-knit community, united by the relentless pursuit to help keep people safer everywhere. Our critical communications, video security and command center technologies support public safety agencies and enterprises alike, enabling the coordination...

  • Senior AI Engineer

    3 days ago

    Chicago, IL, United States Motorola Solutions Full time

    Company Overview At Motorola Solutions, we believe that everything starts with our people. We're a global close-knit community, united by the relentless pursuit to help keep people safer everywhere. Our critical communications, video security and command center technologies support public safety agencies and enterprises alike, enabling the coordination...

  • AI Security Senior Engineer

    14 hours ago

    Chicago, IL, United States TOP TALENT, INC. Full time

    AI Security Senior EngineerCompany: Award-Winning Fortune 500 Company Location: Chicago, IL Work Model: Hybrid (3 days onsite / 2 remote) About the Role The AI Security Senior Engineer will play a critical role in securing enterprise-wide AI platforms, applications, and cloud environments. This position is responsible for designing, implementing, and...

  • AI Security Senior Engineer

    2 weeks ago

    Chicago, IL, United States TOP TALENT, INC. Full time

    AI Security Senior EngineerCompany: Award-Winning Fortune 500 Company Location: Chicago, IL Work Model: Hybrid (3 days onsite / 2 remote) About the Role The AI Security Senior Engineer will play a critical role in securing enterprise-wide AI platforms, applications, and cloud environments. This position is responsible for designing, implementing, and...