QA/AI Automation Tester

1 day ago

Tampa, FL, United States Purple Drive Full time

AI Automation Tester

Job Summary:

As a Functional Tester specializing in AI Agents, you will be responsible for designing, executing, and maintaining

test cases that validate the functionality, reliability, and performance of AI-driven agent systems.

You will work closely with development teams, AI specialists, and product owners to ensure that AI agents

perform accurately and efficiently in real-world scenarios.

Key Responsibilities:

• AI/LLM-Specific QA: Designing test frameworks for LLM outputs (hallucination checks, factuality tests,

toxicity/bias detection).

• Automation Frameworks: PyTest, Robot Framework, Playwright, Cypress, or custom LLM test harnesses.

• Evaluation Metrics: BLEU, ROUGE, BERTScore, GPT-based evaluators, human-in-the-loop validation.

• Data QA: Ensuring high-quality training/evaluation datasets, data validation (Great Expectations, Deequ).

• Python Automation: Scripting test cases, API test automation.

• CI/CD Integration: Automated test pipelines in Jenkins/GitHub Actions/GitLab CI.

• Performance Testing: Latency, throughput, stress testing of LLM-powered services.

• Leadership (Manager level): Define QA strategies for AI systems, lead test automation teams, enforce quality gates.

  • AI Automation Tester

    5 days ago

    Tampa, FL, United States Artech Full time

    Job Title: AI Automation Tester Location: Tampa, FL - A vibrant city known for its cultural diversity, beautiful beaches, and thriving tech community. An ideal place for professionals seeking a dynamic work-life balance. Salary Range: Competitive salary based on experience and qualifications. Introduction Join our innovative team as a Functional Tester...

  • AI/LLM Agent QA Tester

    1 week ago

    Tampa, FL, United States Diverse Lynx Full time

    Position: AI/LLM Agent QA Tester Location : Tampa, FL - Onsite Type of Hire :- Long Term Contract Summary: s a Functional Tester specializing in AI Agents , you will be responsible for designing, executing, and maintaining test cases that validate the functionality, reliability, and performance of AI-driven agent systems. You will work closely...

  • AI/LLM Agent QA Tester

    2 weeks ago

    Tampa, FL, United States Diverse Lynx Full time

    Position: AI/LLM Agent QA Tester Location : Tampa, FL - Onsite Type of Hire :- Long Term Contract Summary: s a Functional Tester specializing in AI Agents , you will be responsible for designing, executing, and maintaining test cases that validate the functionality, reliability, and performance of AI-driven agent systems. You will work closely...

  • AI/LLM Agent QA Tester

    2 weeks ago

    Tampa, FL, United States Diverse Lynx Full time

    Position: AI/LLM Agent QA Tester Location : Tampa, FL - Onsite Type of Hire :- Long Term Contract Summary: s a Functional Tester specializing in AI Agents , you will be responsible for designing, executing, and maintaining test cases that validate the functionality, reliability, and performance of AI-driven agent systems. You will work closely...

  • AI/LLM Agent QA Tester

    5 days ago

    Tampa, FL, United States Diverse Lynx Full time

    Position: AI/LLM Agent QA Tester Location : Tampa, FL - Onsite Type of Hire :- Long Term Contract Summary: s a Functional Tester specializing in AI Agents , you will be responsible for designing, executing, and maintaining test cases that validate the functionality, reliability, and performance of AI-driven agent systems. You will work closely...

  • QA Tester

    5 days ago

    Tampa, FL, United States RIT Solutions, Inc. Full time

    Title: QA Tester Location: 100% remote - Must be willing to work EST. Client prefers people who currently live in an EST state so those candidates will have priority Duration: 6 - 12 month contract to potential hire after 12 months Must Haves & Responsibilities: Hands on Software QA testing experience Ability to write and execute manual testcases for...

  • QA Tester

    19 hours ago

    Tampa, FL, United States RIT Solutions, Inc. Full time

    Title: QA Tester Location: 100% remote - Must be willing to work EST. Client prefers people who currently live in an EST state so those candidates will have priority Duration: 6 - 12 month contract to potential hire after 12 months Must Haves & Responsibilities: Hands on Software QA testing experience Ability to write and execute manual testcases for...

  • Automation QA

    1 week ago

    Tampa, FL, United States RIT Solutions, Inc. Full time

    Automation QA Remote 6+ Months We're specifically looking for a Senior QA Automation SDET with strong experience in test automation. Ideal candidates should have hands-on skills with Java as the primary programming language, Scala experience is a plus. We're also looking for solid experience in Postman, REST APIs, GRPC, microservices testing, and CI/CD...

  • Automation QA

    5 days ago

    Tampa, FL, United States RIT Solutions, Inc. Full time

    Automation QA Remote 6+ Months We're specifically looking for a Senior QA Automation SDET with strong experience in test automation. Ideal candidates should have hands-on skills with Java as the primary programming language, Scala experience is a plus. We're also looking for solid experience in Postman, REST APIs, GRPC, microservices testing, and CI/CD...

  • Automation QA

    2 weeks ago

    Tampa, FL, United States RIT Solutions, Inc. Full time

    Automation QA Remote 6+ Months We're specifically looking for a Senior QA Automation SDET with strong experience in test automation. Ideal candidates should have hands-on skills with Java as the primary programming language, Scala experience is a plus. We're also looking for solid experience in Postman, REST APIs, GRPC, microservices testing, and CI/CD...