AI Test Architect

6 days ago

Santa Clara, United States QualiTest Group Full time

Are you interested in working with the World's leading AI-powered Quality Engineering Company? Ready to advance your career, team up with global thought leaders across industries and make a difference every day? Join us at Qualitest

We are looking for an AI Test Architect to join our growing team in United States

Role - AI Test Architect

Location - Remote (US)

Job Description:

- Architect and build scalable test automation frameworks for generative AI models, leveraging tools like DeepEval, Langsmith, PyTest, or custom Python scripts.

- Implement automated evaluation workflows for model validation, including functionality, accuracy, performance, and reliability testing.

- Design continuous testing pipelines integrated with CI/CD for model deployments.

- Proven expertise in test automation frameworks for AI/ML systems, with a focus on generative models and NLP applications.

- Hands-on experience with DeepEval, Langsmith, PyTest, LangChain, or similar evaluation libraries.

- Proficiency in programming languages: Python (primary), JavaScript, or TypeScript.

- Solid understanding of AI evaluation metrics (BLEU, ROUGE, embedding-based metrics) and adversarial testing techniques.

Role Overview:

The Automation Architect will design and implement a robust automation framework for testing generative AI models, chatbots, and related AI-powered systems. The role involves optimizing testing pipelines for large-scale, high-throughput models, ensuring quality, scalability, and performance.

Roles and Responsibilities:
• Architect and build scalable test automation frameworks for generative AI models, leveraging tools like DeepEval, PyTest, or custom Python scripts.
• Implement automated evaluation workflows for model validation, including functionality, accuracy, performance, and reliability testing.
• Design continuous testing pipelines integrated with CI/CD for model deployments.
• Develop modular frameworks to automate evaluation of NLP-based outputs, including coherence, relevance, toxicity, bias, and safety checks.
• Optimize test execution for large-scale LLM deployments, minimizing latency and maximizing throughput.
• Automate performance and load testing for high-token transactions and API-driven AI endpoints.
• Leverage vector database testing to ensure quality of data retrieval and seamless model integration.
• Introduce monitoring mechanisms to validate models post-deployment and detect model drift.
• Collaborate with DevOps and AI/ML teams to integrate AI model testing within MLOps workflows.
• Drive adversarial and stress testing automation to assess AI robustness and edge-case handling.

Skills and Qualifications:
• Proven expertise in test automation frameworks for AI/ML systems, with a focus on generative models and NLP applications.
• Hands-on experience with DeepEval, PyTest, LangChain, or similar evaluation libraries.
• Proficiency in programming languages: Python, JavaScript, or TypeScript.
• Knowledge of vector databases (e.g., Pinecone, Milvus) and their role in AI model pipelines.
• Experience with CI/CD pipelines, version control (Git), and MLOps platforms.
• Familiarity with tools for performance testing, such as JMeter, Locust, or custom scripts.
• Solid understanding of AI evaluation metrics (BLEU, ROUGE, embedding-based metrics) and adversarial testing techniques.
• Strong architectural skills to build reusable, scalable automation solutions.

Why QualiTest?

  • Be a part of a company who strives to support for diversity and inclusion in the workplace - we are one, we are many at Qualitest. Celebrate culture, share knowledge with engineers from around the globe, and inspire each other through our differences.
  • Local and global opportunities - we offer you internal rotation and international mobility opportunities to grow your career.
  • Clear view of your career and progression with the company - Qualitest is growing massively and giving you the opportunity to grow with us.
  • Work hard and play harder with our flexible and casual culture. Take a break from work and join an employee event, or enjoy the amenities and games provided from one of our Employees Centers.
  • Save your earnings and prepare for your future by enrolling in our 401k plan where Qualitest will match your contributions accelerating your savings plan.
  • Take care of health with enrollment into one of our competitive healthcare benefits. Qualitest will match towards your HSA if you choose to participate.
  • Never stop experimenting and learning with Qualitest Tech academy: 3000+ training courses, mentorship programs, technical tribes, sponsored certifications, leadership programs and much more
  • Stay active and get rewarded with our Corporate Wellness Program. We pay your Gym membership and giving you opportunities to Earn additional vacation times for attendance the gym
  • Competitive Salary of $170k-180k per year
  • Earn bonuses via our Client Referral and Employee Referral Program's. Refer and earn - tap your network for net-worth.
  • Planning a vacation? Looking for car insurance? Get access to Qualitest Employee Perks for discounts on anything from travel to electronics. With so many offerings the savings are endless
Intrigued to find more about us?
  • Visit our website at Careers - Qualitest Group

If you like what you have read, send us your resume and let's start talking

  • Gen AI Architect

    3 months ago

    Santa Clara, United States HCLTech Full time

    Experience Requirements: 8-10 years of experience in developing Data Science, AI, and ML solutions, with a specific focus on generative AI and LLMs in the Finance/Telecomm/LSH/Manufacturing/Retail domain.Prior experience in identifying new opportunities to optimize the business through analytics, AI/ML and use case prioritization.The individual should be a...

  • AI/ML Solutions Architect

    1 week ago

    Santa Clara, California, United States Proximity Works Full time

    Job Description:We are seeking a highly skilled Ai/ml solutions architect to lead the design, development, and implementation of cutting-edge AI/ML solutions. As a key technical leader, you will be responsible for architecting scalable, low-latency distributed systems optimized for AI/ML workloads.The ideal candidate will have 8+ years of experience in...

  • Principal AI Architect/GenAI Architect/Technical AI Architect

    5 hours ago

    Santa Clara, United States E-Solutions Full time

    Title: Principal AI Architect/GenAI Architect (Technical)Location: New Jersey & Santa Clara CAEmployment:- Full-Time (Permanent)Position SummaryThe Principal AI Architect/GenAI Architect will lead the design and implementation of cutting-edge AI solutions, acting as a senior Trusted Advisor for clients’ AI-enabled transformation journeys. This role...

  • Senior AI Solutions Architect

    4 weeks ago

    Santa Clara, California, United States HCLTech Full time

    Job Summary: HCLTech is looking for a Senior AI Solutions Architect to join our team. In this role, you will be responsible for designing and implementing advanced AI solutions that drive business value and transform industries.**Key Responsibilities:**• Architect innovative AI solutions from ideation to MVP, rapidly enabling genuine business value.•...

  • Senior Solutions Architect

    2 weeks ago

    Santa Clara, United States Proximity Works Full time

    We are looking for a Senior Solutions Architect to design, develop, and scale innovative AI/ML-driven solutions. You will be responsible for architecting highly scalable, low-latency distributed systems optimized for AI/ML workloads. As a key technical leader, you will solve complex challenges, influence next-generation AI/ML infrastructures, and guide...

  • Generative AI Solutions Architect

    4 weeks ago

    Santa Clara, California, United States HCLTech Full time

    Job Description: Generative AI Solutions ArchitectCompany Overview:HCLTech is a leading technology company that provides innovative solutions to businesses worldwide. We are committed to delivering high-quality services and products that meet the evolving needs of our customers.About the Job:We are seeking a highly skilled Generative AI Solutions Architect...

  • AI Security Architect

    1 week ago

    Santa Clara, California, United States ZipRecruiter Full time

    We're on a mission to revolutionize cybersecurity at Palo Alto Networks, and we need your expertise as a Senior Principal Software Engineer to help us achieve our goal. We're looking for someone with a strong background in cloud computing, particularly in the Google Cloud Platform (GCP), and a passion for developing scalable and efficient cloud services.Your...

  • AI Solutions Architect with Generative AI Expertise

    5 days ago

    Santa Clara, California, United States Promote Project Full time

    Unlock Innovative AI Solutions as a Solutions Architect at Promote ProjectWe are seeking an experienced Ai Solutions Architect Engineer to join our team, specializing in Generative AI software development and deployment.This is an exciting opportunity to build your career in the rapidly growing field of AI, working closely with our team to drive...

  • Principal AI Architect

    4 months ago

    Santa Clara, United States HCLTech Full time

    Position Summary The Principal AI Architect is responsible for leading the design and implementation of advanced AI solutions and strategic architecture. Working closely with technology leaders from across our global client community, you will be their senior Trusted Advisor for their AI-enabled transformation journey This role demands deep understanding of...

  • Cloud AI/ML Solutions Architect

    4 weeks ago

    Santa Clara, California, United States Amazon Full time

    About the JobWe are seeking a highly skilled Cloud AI/ML Solutions Architect to join our team at Amazon. As a key member of our Prototyping Solutions Architect team, you will be responsible for designing and developing working prototypes with customers using AWS services.This is an excellent opportunity for individuals who have a passion for technology and...

  • AI and Machine Learning Solutions Architect

    1 week ago

    Santa Clara, California, United States Resource Informatics Group Full time

    Job DescriptionWe are seeking a highly skilled Senior ML Engineer to join our team at Resource Informatics Group. The ideal candidate will have expertise in integrating Client/AI capabilities into the ServiceNow platform, building Client/AI enabled applications, and evolving the Client Ops process.The successful candidate will be responsible for designing...

  • AI Governance Architect

    4 weeks ago

    Santa Clara, California, United States ServiceNow Full time

    Transforming Data into a Compliant AssetWe are seeking an experienced AI Governance Architect to oversee the entire data and development lifecycle, transforming data into a compliant, actionable, and secure asset for our AI practitioners.This is a unique opportunity to establish data and governance frameworks for the AI Software Development Lifecycle (SDLC),...

  • AI Implementation Solutions Architect

    4 weeks ago

    Santa Clara, California, United States ServiceNow Full time

    **About ServiceNow**We're a global leader in AI-enhanced technology, transforming the way organizations work.Our intelligent cloud-based platform connects people, systems, and processes to empower businesses to find smarter, faster, and better ways to work.We're on a mission to make the world work better for everyone.**Job Summary**We're seeking an...

  • AI Systems Solutions Architect

    4 weeks ago

    Santa Clara, United States d-Matrix Full time

    d-Matrix has fundamentally changed the physics of memory-compute integration with our digital in-memory compute (DIMC) engine. The "holy grail" of AI compute has been to break through the memory wall to minimize data movements. We've achieved this with a first-of-its-kind DIMC engine. Having secured over $154M, $110M in our Series B offering, d-Matrix is...

  • AI Systems Solutions Architect

    4 weeks ago

    Santa Clara, United States d-Matrix Full time

    d-Matrix has fundamentally changed the physics of memory-compute integration with our digital in-memory compute (DIMC) engine. The "holy grail" of AI compute has been to break through the memory wall to minimize data movements. We've achieved this with a first-of-its-kind DIMC engine. Having secured over $154M, $110M in our Series B offering, d-Matrix is...

  • Solutions Architect Generative AI

    2 months ago

    Santa Clara, United States Promote Project Full time

    We are looking for an AI Solution Architect Engineer with experience in Generative AI software development and deployment. As part of the Solution Architect organization, we work with the most exciting computing hardware and software, driving the latest breakthroughs in deep learning and AI with NVIDIA’s key customers. This role offers an excellent...

  • AI Solutions Architect

    3 days ago

    Santa Clara, California, United States ServiceNow Full time

    Unlock Your Career Potential with ServiceNowServe as a key member of our Customer Outcomes team at ServiceNow, where you'll work closely with customers to drive adoption and satisfaction on our platform. This is an exceptional opportunity to leverage your technical expertise in AI solutions and contribute to the growth of our customers' businesses.As a GenAI...

  • AI Solutions Architect and Engineer

    4 weeks ago

    Santa Clara, California, United States Bronze Technologies, Inc. Full time

    We are looking for a highly skilled AI Solution Architect and Engineer to join our team at Bronze Technologies, Inc. This role offers an excellent opportunity to build your career in the rapidly growing field of AI while working with world-class technology companies.Primary Responsibilities:Lead software customer technical engagements with NVIDIA products...

  • AI Solutions Architect

    7 days ago

    Santa Clara, California, United States Amazon Full time

    About the Role: As a Senior GenAI Specialist Solutions Architect at Amazon, you will be responsible for designing and implementing scalable Generative AI solutions for our customers. With deep technical expertise in large language models, distributed training, and inference, you will work closely with engineering teams to develop cloud-based solutions that...

  • Cloud AI Solutions Architect

    7 days ago

    Santa Clara, California, United States Amazon Full time

    You will have the opportunity to be part of a diverse and inclusive workplace where you can make a meaningful impact. As a Worldwide Specialist Solutions Architect, you will work collaboratively with cross-functional teams to deliver exceptional customer experiences. Your primary focus will be on helping customers navigate the complexities of cloud computing...