QA Analyst Lead

6 days ago


Remote, United States Codeworks L.L.C Full time
Job DescriptionJob DescriptionCodeworks is an IT Services firm in SE Wisconsin, known for our strong commitment to quality and for our direct client relationships. Who We’re Looking For...A Software Quality Assurance Lead for a 6-12 month contract-to-hire role with largest and fastest growing on-site radiology practice in the US. Applicants must be authorized to work in the United States without the need for sponsorship now or in the future.We are seeking an excellent Quality Assurance Lead to spearhead the quality assurance efforts for our cutting-edge AI-powered radiology reporting solution within our product line. This role involves working closely with a development team leveraging Amazon AWS and Large Language Models (LLM) to automate the interpretation of radiology reports, and explicitly summarizing the radiologist’s findings to streamline the review process; thus, providing the most accurate and relevant information at-a-glance.As the Quality Assurance Lead, you will be responsible for ensuring the accuracy and reliability of AI models through rigorous testing, automation, and data validation. The ideal candidate will have a strong background in preferably in automation, scripting, and data integrity, with a focus on AWS-based systems and machine learning models. Any experience with evaluating LLMs against BERTscores, ROUGE (Recall-Oriented Understudy for Gisting Evaluation), BLEU (Bilingual Evaluation Understudy), METEOR (Metric for Evaluation of Translation with Explicit ORdering) would also be very helpful.Key Responsibilities:Lead Automation of System Verifications
  1. Validation of Report accuracy in RecoMD RPI against Intelerad (Radiologist dictation tool).
  2. Validation of Patient matching schema based on development specifications.
  3. Validation of Logs (CloudWatch) and Reporting Dashboards based on specifications.
  4. Validation of data integrity and synchronization in Intelligence DB (Aurora PostgreSQL).
  5. Validation of ingestion and transmission of HL7 messages.
Fine-tune LLM Models
  1. Collaborate with AI/ML engineers to test and improve large language models for radiology reporting.
  2. Drive the design and implementation of automation frameworks to validate implemented AI and LLM-based solutions in RecoMD RPI.
Scripting and Automation
  1. Develop and maintain test scripts using Python or C# for automating QA processes, with tools like Selenium or similar.
Data Integrity and Validation
  1. Ensure the accuracy of generated reports by performing data integrity checks and validating AI outputs (aka study relevancy and report summarization) against test datasets like original report content.
Cloud-based QA
  1. Design test scenarios within the Amazon AWS cloud environment to validate functionality, performance, and scalability.
Writing Queries
  1. Create and execute SQL queries to validate data and ensure accuracy in database-related operations.
Required Skills
  1. Proven experience in QA Automation and technical leadership.
  2. Experience with Large Language Models (LLM) and machine learning-based testing.
  3. Strong expertise in AWS cloud services. (Lambda, S3, Bedrock, etc.)
  4. Proficiency in scripting languages such as Python or C#.
  5. Hands-on experience with Automation tools (e.g., Selenium, or similar).
  6. Ability to perform Data Integrity checking and strong SQL skills for writing queries.
Nice to Have
  1. Familiarity with foundational and custom models like Bedrock, Sagemaker, etc.
  2. Knowledge of microservices architecture.
  3. Familiarity with healthcare or radiology domains.
About our Client—Our client is the largest physician-led and physician-owned radiology practice in the United States.If you feel that you meet the qualifications listed above, please forward your resume in Word format to kristy.harmann@codeworks-inc.com.About CODEWORKS:Headquartered in Milwaukee, WI with an office in Madison, WI—Codeworks has over 23+ years of experience serving Fortune 1000 companies in Wisconsin as well as our client's national locations. Our recruiting team is extremely good at evaluating, advising, and connecting IT professionals with new opportunities that will satisfy their expectations both in salary and opportunity for growth.For more information, please visit our website at: www.codeworks-inc.com.For priority career/job posting updates, please follow us on Twitter: @CodeworksITAt Codeworks, we're committed to diversity, equity, and inclusion in our workforce and beyond. We believe in equal opportunities and value the unique perspectives that every individual brings to our team. Join us in creating an inclusive, innovative, and collaborative workplace where your talents can thrive.#LI-KH1
  • Qa Engineer

    3 months ago


    Remote, United States Veracity Software Inc. Full time

    We're Looking for Someone With Minimum of 3 years of experience as a QA Analyst preferably at a consumer-oriented Website Proficient in SQL query inserts and updates Proficient in QA methodology and process (Agile testing experience preferred) Strong communication skills must be able to communicate with other functional teams to coordinate systems...

  • Qa Engineer

    3 months ago


    Remote, Oregon, United States Veracity Software Inc. Full time

    We're Looking for Someone With Minimum of 3 years of experience as a QA Analyst preferably at a consumer-oriented Website Proficient in SQL query inserts and updates Proficient in QA methodology and process (Agile testing experience preferred) Strong communication skills must be able to communicate with other functional teams to coordinate systems...

  • Sr IT QA Analyst

    4 weeks ago


    Remote, United States United BioSource Full time

    As a pharmaceutical support industry leader , UBC is devoted to empowering health solutions for a better tomorrow . We take pride in improving  patient outcomes and advancing healthcare.  At UBC, we provide services to enhance the entire drug development process and commercialization lifecycle - From clinical trial support to real-world evidence...

  • QA Engineer

    5 months ago


    Remote, United States SparkzIT Full time

    Successful candidates have a passion for testing software a strong track record of growth and good demonstrated experience. Individuals in this role are supported by a fantastic team and leave an impact by helping to ensure project success - Not only testing to make sure that the solution meets our clients’ requirements but that the solution helps to...

  • QA Engineer

    5 months ago


    Remote, Oregon, United States SparkzIT Full time

    Successful candidates have a passion for testing software a strong track record of growth and good demonstrated experience. Individuals in this role are supported by a fantastic team and leave an impact by helping to ensure project success Not only testing to make sure that the solution meets our clients' requirements but that the solution helps to solve our...

  • QA Engineer

    1 month ago


    Remote, Oregon, United States Intradiem Full time

    Intradiem's intelligent automation solution for customer service teams is reinventing customer service for everyone.Who We AreIntradiem is a technology company on a mission to reinvent customer service through automation. What We DoWe develop innovative, AI-powered Intelligent Automation solutions for contact center and back-office teams. Our solutions...


  • Remote, Oregon, United States PlanetArt Full time

    DescriptionCompany and Vision PlanetArt's vision is to be the leading seller of personalized and make-on-demand products worldwide. We provide consumers with unmatched tools and content and an unparalleled end-to-end customer experience that result in high-quality, meaningful finished products and memorable celebrations of live events. The company's brands...

  • Sr Systems Analyst

    3 months ago


    Remote, United States Maximus Full time

    General information Job Posting Title Sr Systems Analyst Date Wednesday, June 5, 2024 City Remote Country United States Working time Full-time Description & Requirements This position can be worked 100% from a home office. Essential Duties and Responsibilities: - Serve as primary point of contact with MAXIMUS project staff and/or clients to...

  • QA Engineer

    1 month ago


    Remote, Oregon, United States Gradient AI Full time

    (Senior) QA Engineer Gradient AI: Gradient AI is a leading provider of AI solutions for the Group Health and P&C insurance industries. Our solutions improve loss ratios and profitability by predicting underwriting and claim risks with greater accuracy, as well as reducing quote turnaround times and claim expenses through intelligent automation. Gradient...

  • QA Engineer

    2 months ago


    Remote, Oregon, United States BizLibrary Full time

    BizLibrary is seeking a QA Engineer to work alongside our Product and Development teams. A qualified candidate is experienced applying quality assurance concepts and leading quality control efforts utilizing both manual testing and automated testing on complex, data-driven, web-based software applications. This individual will ensure that the developed...

  • Lead Data Analyst

    2 months ago


    Remote, Oregon, United States EvoPlay Full time

    Who we are:Evoplay is one of the leading iGaming providers, recognized for its innovative approach to game development and technological solutions. The key to this success is the teamwork of like-minded people who are passionate about a common goal and love games.Our mission is to surprise not only the iGaming market but the entire entertainment industry...

  • Contract Lead Analyst

    2 months ago


    Remote, Oregon, United States The Cigna Group Full time

    The job profile for this position is Contract Lead Analyst, which is a Band 3 Senior Contributor Career Track Role.Excited to grow your career?We value our talented employees, and whenever possible strive to help one of our associates grow professionally before recruiting new talent to our open positions. If you think the open position you see is right for...


  • Remote, Oregon, United States Bozeman Health Full time

    Position Summary: The Application Lead Analyst spends approximately 85% of time performing analyst work and 15% supporting Manager by monitoring team's workload and work distribution, facilitating meetings, and assisting with scheduling and on call coverage. They are the primary provider of customer service for application related issues and opportunities....


  • Remote, Oregon, United States Maximus Full time

    Senior Business Analyst at MaximusGeneral InformationJob Title: Senior Business Analyst – RPA/UiPathLocation: RemoteJob Type: Full-timeOverview & QualificationsMaximus is seeking skilled Senior Business Analysts to enhance our collaborative efforts in supporting a key client. This role is primarily remote, with occasional requirements for in-person...


  • Remote, Oregon, United States Maximus Full time

    Senior Business Analyst at MaximusGeneral InformationJob Title: Senior Business Analyst – RPA/UiPathLocation: RemoteJob Type: Full-timeRole OverviewMaximus is seeking experienced Senior Business Analysts to support a key client in the public sector. This position is primarily remote, with occasional requirements for in-person meetings. If you possess...


  • Remote, Oregon, United States Maximus Full time

    Senior Business Analyst at MaximusGeneral InformationPosition: Senior Business Analyst – RPA/UiPathWork Arrangement: RemoteEmployment Type: Full-timeRole OverviewMaximus is seeking experienced Senior Business Analysts to contribute to a significant project for a key client. This position offers the flexibility of remote work while requiring occasional...

  • Senior QA Engineer

    2 months ago


    Remote, Oregon, United States Yassir Full time

    Yassir is the leading super App in the Maghreb region set to changing the way daily services are provided. It currently operates in 45 cities across Algeria, Morocco and Tunisia with recent expansions into France, Canada and Sub-Saharan Africa. It is backed (~$200M in funding) by VCs from Silicon Valley, Europe and other parts of the world. We offer...


  • Remote, Oregon, United States The Cigna Group Full time

    The job profile for this position is Operations Lead Analyst, which is a Band 3 Senior Contributor Career Track Role.Excited to grow your career?We value our talented employees, and whenever possible strive to help one of our associates grow professionally before recruiting new talent to our open positions. If you think the open position you see is right for...


  • Remote, Oregon, United States Maximus Full time

    Senior Systems AnalystMaximus is seeking a Senior Systems Analyst to enhance our team dynamics.This position requires effective collaboration with project personnel and clients to ascertain both business and technical needs. You will take charge of defining data file extracts, outlining process flows, and establishing report frameworks to facilitate...


  • Remote, Oregon, United States The Cigna Group Full time

    The job profile for this position is Operations Lead Analyst, which is a Band 3 Senior Contributor Career Track Role.Excited to grow your career?We value our talented employees, and whenever possible strive to help one of our associates grow professionally before recruiting new talent to our open positions. If you think the open position you see is right for...