QA Tester II

QA Tester, AI Development 

About This Role 

The QA Tester is the quality gatekeeper for Insight's AI Development team, responsible for defining and executing test strategies across AI-powered applications. This role goes beyond traditional functional testing — you will validate the reasoning, accuracy, and safety of AI agents and LLM-powered features, while also maintaining robust automation frameworks for the full application stack. You will lead QA efforts across multiple projects and mentor team members on testing best practices for AI applications. 

Key Responsibilities 

• Define and implement test strategies for AI-powered applications, including functional, integration, regression, and AI-specific testing (prompt evaluation, hallucination detection, guardrail validation). 

• Build and maintain automated test frameworks using Python (Pytest) and UI automation tools (Playwright, Cypress, or Selenium) for React front-ends and API back-ends. 

• Validate AI model outputs for accuracy, safety, and adherence to expected behaviors across agentic workflows and conversational interfaces. 

• Integrate automated test suites into CI/CD pipelines to ensure rapid feedback on code changes. 

• Lead defect management: identify, document, reproduce, and track bugs; work closely with developers to verify fixes. 

• Collaborate with AI Developers, Full Stack Developers, and DevOps Engineers to ensure quality is embedded throughout the development lifecycle. 

• Establish QA metrics and reporting to communicate quality status and risk to stakeholders. 

Qualifications 

• Experience: Minimum 3 years of experience as a QA Engineer or SDET, with experience testing complex web applications and API integrations. 

• Automation: Proficient in Python for writing automated test scripts (Pytest preferred); experience with UI automation tools (Playwright, Cypress, or Selenium). 

• API Testing: Strong understanding of REST APIs; experience validating API responses and JSON schemas. 

• AI/ML Testing: Familiarity with testing LLM-powered applications, including non-deterministic output validation. Experience with evaluation frameworks (RAGAS, LangSmith, or similar) is a plus. 

• Cloud: Working knowledge of Microsoft Azure; ability to navigate cloud logs and debug deployed services. 

• CI/CD: Experience integrating test suites into CI/CD pipelines (Azure DevOps, GitHub Actions). 

• Communication: Strong written and verbal English communication skills; ability to articulate risk and quality metrics to stakeholders. 

• Nice to Have: Experience with performance testing, security testing, or accessibility testing.