🧪 QA Roadmap Manual · Automation · AI — an open curriculum by Arifuzzaman Antor

🤖 The AI QA Tools Directory (prioritized & auto-updated)

Goal: every AI tool a QA should know — what it is, why it matters, how it works, and what to do to start — ranked by priority so you don’t miss anything. Pairs with Stage 16: AI-Powered QA (the strategy) and Stage 11: AI × QA (hands-on).

🔄 This page auto-updates. The latest versions of the open-source tools below are refreshed weekly by a GitHub Action — last refreshed: 2026-08-17. (See “How the auto-update works” at the bottom.)

Priority legend: 🥇 learn first (high-leverage, mostly free) · 🥈 strong / next · 🥉 situational / enterprise.

⭐ Start here — the 5 to learn first

If you do nothing else, learn these — they’re free or accessible and cover 80% of the value:

  1. Claude Code or Cursor — your AI cockpit (runs agents, MCP, generates & reviews tests)
  2. Playwright MCP v0.0.79 — lets the agent drive a real browser
  3. Playwright + Test Agents v1.62.1 — the automation engine, with planner/generator/healer
  4. promptfoo 0.122.0 — evals + red-teaming for AI features
  5. GitHub Copilot — inline test generation in your editor

1. 🥇 AI cockpit & agentic browser control (the foundation)

The layer everything else plugs into: an AI client + MCP + a browser engine.

Tool Tier What & why How it works Get started
Claude Code 🥇 Terminal-native AI agent; runs MCP servers, drives tests, reviews code You prompt; it plans, edits, runs, and calls tools npm i -g @anthropic-ai/claude-code, then claude
Cursor 🥇 AI-first IDE; great for generating & refactoring test code Chat + inline edits + MCP support Install, open your test repo, enable MCP
Playwright MCP v0.0.79 🥇 Exposes a real browser to any AI agent via MCP — the key primitive Agent reads the accessibility tree, clicks/types, runs tests claude mcp add playwright -- npx @playwright/mcp@latest
ChatGPT / Claude (web) 🥇 Test-design, data, bug-report drafting, log triage Prompt with role + context + constraints Free tiers; see the prompt library

2. 🥇 Automation engine with built-in AI

Tool Tier What & why How it works Get started
Playwright + Test Agents v1.62.1 🥇 Open-source E2E with first-party planner → generator → healer agents; free Agents author and auto-repair tests in plain Playwright code playwright.dev · npm init playwright@latest
Playwright Codegen 🥇 Record clicks → test code (then you harden it) Records actions into a spec npx playwright codegen <url>

3. 🥈 Autonomous / natural-language test platforms (commercial)

Write tests in plain English or from recordings; AI builds and maintains them. Great for teams; most are paid with trials/demos.

Tool Tier What & why How it works
Mabl 🥈 AI-native, low-code; agentic tester builds E2E from user stories NL/record → adaptive healing + computer vision
testRigor 🥈 Plain-English test authoring, very low maintenance Executable specs in English
Tricentis Testim / AI Workspace 🥈🥉 Codeless authoring + self-healing; 2026 agentic “AI Workspace” (test creation/automation/perf agents) AI locators, auto-heal, agentic execution
Functionize 🥉 ML-driven enterprise testing; NL test creation Specialized agents + ML models
Katalon 🥈 Broad platform (web/API/mobile) for mixed-skill teams Low-code + AI assists
QA Wolf 🥈 Fully-managed coverage (they write & maintain your tests) Service + Playwright under the hood
ACCELQ 🥉 Codeless cross-platform, enterprise AI-assisted, business-process focused
LambdaTest KaneAI 🥈 NL test authoring on a real device/browser cloud AI agent + cloud grid
BrowserStack Low Code 🥈 NL authoring + self-healing on real devices Low-code + device cloud
Shiplight 🥈 AI-coding-agent workflow; YAML tests in git, heals as reviewable PRs; on Playwright Intent re-derivation + cached locators
Rainforest QA 🥉 No-code with AI test generation Visual authoring

4. 🩹 Self-healing (reduce locator maintenance)

Overlaps with above — the key mechanism to know. Playwright healer (free, 🥇), plus Testim, Mabl, Functionize, Shiplight. Rule: healing is a suggestion — you approve the fix.

5. 👁️ Visual AI testing

Tool Tier What & why How it works
Applitools 🥈 Leading Visual AI; “eye-like” diffing that ignores noise Visual AI baselines + smart diff ($$$)
Percy (BrowserStack) 🥈 Visual regression in CI Snapshot + diff
Meticulous 🥉 Auto-generates visual/functional tests from real usage Records sessions, replays

6. 🧑‍💻 AI test / unit-code generation (dev-side)

Tool Tier What & why How it works Get started
GitHub Copilot 🥇 Inline test scaffolding across languages; dedicated “tests” agent Prompt/inline in editor Editor extension
Qodo (CodiumAI) 🥈 Behavior-first, context-aware tests tied to PRs Analyzes intent, generates + reviews IDE plugin
Diffblue Cover 🥉 Autonomous JUnit generation for enterprise Java at scale Reinforcement-learning search CLI/IDE
Amazon Q Developer 🥈 Test scaffolding in AWS-centric stacks Inline suggestions Editor extension

7. 🧠 Testing AI systems — LLM evals & red-team (the 2026 edge)

If your product has AI features, this is your job. Start with promptfoo + DeepEval.

Tool Tier What & why How it works Get started
promptfoo 0.122.0 🥇 Evals + red-teaming/security; compare models, harden prompts; most-adopted OSS YAML config, CLI, CI-friendly npx promptfoo@latest init
DeepEval v4.1.7 🥇 “PyTest for LLMs” — unit-test-style evals Python assertions on LLM output pip install deepeval
RAGAS v0.4.3 🥈 Reference-free RAG metrics (faithfulness, relevancy, context) Metrics library pip install ragas
Langfuse v4.11.0 🥈 Self-hostable tracing + evals + dashboards Observability SDK langfuse.com
Arize Phoenix arize-phoenix-v20.2.1 🥈 OpenTelemetry-native, self-hostable eval/tracing Traces + evals pip install arize-phoenix
OpenAI Evals v0.1.1 🥉 Reference eval framework + registry Python eval specs github.com/openai/evals
Braintrust / LangSmith 🥈 Hosted eval platforms: human annotation, regression tracking, dashboards SaaS free tiers

AI security: OWASP Top 10 for LLM Apps (the checklist) · garak (LLM vuln scanner) · promptfoo red-team (prompt-injection).

8. 📚 The living list (bookmark it)

  • awesome-ai-testing — a community-curated, continuously-updated list of AI testing tools (test generation, self-healing, MCP, LLM eval). The best single place to catch new entrants.

🧭 How to choose (decision guide)

  • Solo / open-source / want control? → Claude Code or Cursor + Playwright MCP + Playwright Test Agents. (All free.)
  • Team wants low-code + managed healing? → Mabl, testRigor, or Tricentis.
  • Don’t want to write/maintain tests at all? → QA Wolf (managed) or Rainforest.
  • Enterprise Java legacy? → Diffblue Cover for unit tests.
  • Heavy visual UI? → Applitools or Percy.
  • Your product has LLM/AI features? → promptfoo + DeepEval (add Langfuse/Phoenix for tracing).
  • Just want faster daily QA? → an AI client + the prompt library.

Don’t buy ten tools. Master one agentic workflow (MCP + Playwright) and one eval tool (promptfoo). Depth beats breadth.

🔄 How the auto-update works

You asked for a system that stays current with releases — here it is, living in the repo:

  1. _data/tool_versions.yml holds the latest version of each tracked open-source tool.
  2. .github/workflows/update-tool-versions.yml runs every Monday (and on-demand from the Actions tab). It queries each tool’s GitHub Releases API, rewrites the data file, and commits only if a version changed — which triggers a site rebuild.
  3. This page renders those versions live, so the badges above are always current — last refreshed 2026-08-17.

To add a tool to the tracker: add its key|owner/repo line to the workflow’s tools list. To run it now: open the repo’s Actions → “Update AI-QA tool versions” → Run workflow.

🎓 Learn to actually use these in guided courses on AZADEMY.

Sources (2026): awesome-ai-testing · QA Wolf — 12 best AI testing tools · Shiplight — best AI QA tools · TestDino — Playwright AI ecosystem · DeepEval — top eval frameworks · Braintrust — DeepEval alternatives

← Back to the roadmap · See also: Stage 16: AI-Powered QA · Cheatsheets