AI-Driven QA Automation Framework

Web · Android · iOS · API — framework, workflow, tools and skills

1. Purpose and outcomes

This framework moves our web and mobile testing from mostly manual to mostly automated without lowering any quality gate. AI agents write tests, investigate failures and draft documents. Deterministic test runs decide pass or fail. People approve.

OutcomeHow we know it is met
Much less manual testingTargets: 70–80% of execution effort automated; smoke and retest fully automated; at least 90% of regression automated. Measured once automation is running
On-time deliveryTesting runs in parallel with development, not as a phase after it
Results we can trustEvery result is backed by evidence; flaky tests never count as passed
Clear output for everyoneOne report pack per run, in our SOP templates, with a calculated confidence score
Fewer escaped bugsEscaped defects tracked per release and must not rise as speed increases

The framework replaces the tool-neutral parts of QA-SOP-001 with a concrete stack. The SOP's roles, templates, approvals and exit criteria still apply.

SOP: https://panthos-qa-sop.vercel.app

2. Scope

The framework covers every test type in SOP section 5.3 for web, Android, iOS and API. Each type is automated where a machine can judge the result reliably, and stays human where judgement, real hardware or business acceptance is needed.

Test typeAutomatedStays human
FunctionalHappy paths, negative and boundary casesApproving new and AI-written tests
SmokeTarget: all smoke cases, on every buildNone
RegressionTarget: at least 90% of regression casesTriage of failures
RetestingRerun of the test tagged with the defect IDConfirming flaky results
APIFunctional, schema, fuzzing, contractManual exploration in Postman
Integration and systemFull journeys on staging and emulatorsNone
AcceptanceEach acceptance criterion maps to a testProduct Owner acceptance (UAT)
ExploratoryAgent sessions that report findingsConfirming every finding; human sessions on new features
UI/UXScreenshot baselines, Figma comparisonJudging whether a mismatch matters
UsabilityNoneFully human, with agent notes
Responsive (cut-off text and elements)Layout checks at every screen width and device sizeApproving screenshot baseline changes
CompatibilityLocal browsers (Chrome, Edge, Firefox, WebKit, Safari), emulators, simulators, team phonesFinal look on 1–2 real devices
Mobile-specificOrientation, install/upgrade, permissions, background/resume, network, emulated calls and SMS (Android), simulated push (iOS)Real calls, camera and biometrics on team phones
PerformanceLoad, stress, soak; page-speed budgetsSetting the limits
SecurityScans of the running app, code, dependencies, app binaryReviewing every finding
AccessibilityWCAG scans in every web testScreen-reader pass (TalkBack, VoiceOver)

Out of scope for now: paid device clouds (BrowserStack, LambdaTest).

3. Architecture

The framework has five layers, and only the execution layer decides pass or fail. Agents write tests and explain results; they never give the verdict.

Agents write tests, test runs decide, people sign off. Inputs (stories and PRDs, Figma designs, OpenAPI spec, code changes, app builds) feed the agent layer (Claude Code + our skills), which writes test suites in Git (Playwright, Maestro / Appium, k6, ZAP, MobSF, axe-core). Execution on GitHub Actions decides pass or fail, then results: evidence, HTML summary, SOP documents and defects in ClickUp or Linear. People: QA reviews new tests, QA confirms findings, developers fix defects, QA Lead signs off, Product Owner accepts.
framework architecture · 5 layers + human approvals

Everything the agent layer produces lands in Git as a reviewed PR, so every test that counts has been read by a person and run by CI.

4. Automation workflow

Automation runs at six points from feature to production, plus a weekly health check, and each point has a gate that can stop the change. Testing starts when a feature is ready for development, not after it is built.

Automation runs at six points, and each one can stop a change. 1 Feature ready: agent drafts cases and tests; gate: QA approves cases. 2 Pull request: tests for changed areas; gate: blocks the merge. 3 Deploy to QA: smoke + API health checks; gate: rejects bad builds. 4 Nightly: full regression, all devices; gate: triaged next morning. 5 Release candidate: full suite on all devices; gate: Go / No-go. 6 After release: production smoke + monitors; gate: rollback signal. Weekly and quarterly: flaky-test review, planted-fault check, suite clean-up, metrics.
automation workflow · 6 triggers, each with a gate

A failed gate sends the change back to its developer with the evidence; it re-enters the flow at the same point after the fix.

#TriggerWhat runsWhereReport
1Feature ready (on demand)spec-to-tests drafts cases and tests; exploratory agent session; new tests pass test-quality-gateLocal, CIFeature test report
2Every pull requestTests for changed areas on Chromium + one phone width; API contract; accessibility and visual checks on touched pagesCI, preview environmentShort PR comment
3Deploy to QA/stagingWeb smoke + API health; mobile smoke on one Android emulator and one iOS simulatorCIBuild check report
4NightlyFull regression on all browsers, widths and emulators; API fuzzing; ZAP scan; Lighthouse; flaky-test detectionCI, team MacNightly report
5Release candidateFull regression on all browsers, emulators, simulators and team phones; k6 performance; full accessibility and security checks; mobile internal testing trackCI, team Mac, team phonesRelease report (SOP Test Summary Report)
6After releaseRead-only production smoke; synthetic monitors every 15–30 min; Sentry crash and error watch for 48 hoursCheckly, SentryPost-release report
7Weekly and quarterlyFlaky-test review; planted-fault check; suite clean-up; metricsCISuite health report

Retest flow: the developer moves a defect to Ready for Retest in ClickUp or Linear; CI reruns the test tagged with that defect ID plus nearby tests on the same device; a pass closes the defect with evidence, a fail reopens it with new evidence.

5. Toolset by test type

The core is eight free tools: Playwright, Maestro, k6, OWASP ZAP, MobSF, axe-core, Schemathesis and GitHub Actions. All test code we write is TypeScript/JavaScript (Playwright, k6) or YAML (Maestro flows, CI workflows). The Python- and Java-based tools (Schemathesis, ZAP, MobSF) run as ready-made command-line tools or Docker containers, so the team never writes or maintains Python or Java. Every tool is free except real cross-device and cross-browser testing, which needs a paid device cloud (BrowserStack or LambdaTest). Until we start that, we test on local browsers, emulators, simulators and the team's own phones. The only other costs are Claude seats and a Figma Dev Mode seat (only if needed).

Human review levels: None = fully automated pass/fail, a person looks only at failures. Low = a person approves changes or spot-checks. Medium = a person reviews every output. High = led by a person, the tool assists.

Test typeWebMobileAPI / backendCostHuman review
FunctionalPlaywrightMaestro; Appium + WebdriverIO as fallbackPlaywright API testsFreeLow: review new and AI-written tests once
SmokePlaywright @smokeMaestro smoke flowsHealth-check testsFreeNone
RegressionPlaywright @regressionMaestro regression flowsPlaywright API + contractFreeNone: triage failures only
RetestingRerun test tagged @DEF-xxxSame, in MaestroSameFreeNone; Low if flaky
APIPlaywright API + Schemathesis fuzzing; Pact for contracts; Bruno or Postman for manualFreeLow: confirm fuzzing findings
IntegrationPlaywright on staging + Testcontainers + MSWMaestro on stagingTestcontainers, WireMockFreeLow
SystemFull Playwright journeysFull Maestro journeysSameFreeNone
AcceptanceTests named after each acceptance criterionSameSameFreeMedium: Product Owner accepts
ExploratoryAgent via Playwright CLI/MCPAgent via mobile-mcp / Maestro MCPAgent with BrunoFree + Claude seatsMedium: confirm every finding
UI/UX design matchScreenshots + Figma MCPScreenshots + Figma MCPFigma paid seat with Dev ModeMedium
UsabilityHuman sessionHuman sessionFreeHigh
UI validation and responsivePlaywright layout checks + screenshot baselinesMaestro + element-position checks + baselinesFreeNone for checks; Low for baselines
Visual review (optional)ArgosArgosFree tier (5,000 screenshots a month)Low
Cross-browserPlaywright: Chromium, Firefox, WebKit, real Chrome and real Edge; real Safari on MacMobile Safari in iOS simulators (Appium); Chrome in Android emulatorsFreeNone
Cross-deviceAndroid emulators and iOS simulators across the device matrix (section 6)FreeNone
Real devicesTeam phones over USB: Maestro for Android, Appium for iPhone (Maestro has no official real-iPhone support); Firebase Test Lab free quota, 10 virtual + 5 physical runs a dayFreeLow: final look and feel
Mobile-specificMaestro + emulator controls: orientation, network speed, emulated calls and SMS (Android), simulated push (iOS); Network Link Conditioner on MacFreeNone; High for real calls, camera, biometrics
PerformanceLighthouse CIPerfetto + Macrobenchmark (Android), Xcode Instruments (iOS)k6FreeLow: a person sets the limits
SecurityOWASP ZAP + role/permission testsMobSF (APK/IPA scan)ZAP API scan, Semgrep, Trivy, GitleaksFreeMedium: review every finding
Accessibilityaxe-core inside PlaywrightAccessibility Scanner (Android), Accessibility Inspector (iOS)FreeNone for web scans; High for screen readers
Cross-device and cross-browser on real devicesBrowserStack or LambdaTest: real browsers and OS versionsBrowserStack or LambdaTest: real phones and tabletsPaid; the only paid testing tool, added when we start real-device testingNone

Supporting tools

PurposeToolCost
CI and schedulingGitHub ActionsFree minutes; team Mac runner is free
Raw reports and flaky-test historyAllure + Playwright HTMLFree
SOP-template reportsqa-run-report skillFree
DefectsClickUp or Linear (set per project)Existing plan
Production monitoring and errorsCheckly, SentryFree tiers
Agent layerClaude Code + our skillsClaude seat per QA engineer

6. Devices and environments

We test on what we have: local browsers, emulators and simulators on team Windows and Mac machines, and the team's own phones. Real cross-device and cross-browser testing on BrowserStack or LambdaTest comes later. iOS simulators and real desktop Safari need a Mac; everything else runs on both.

WorkWindowsMac
Web tests, all screen widthsYesYes
Real desktop SafariNoYes
Android emulatorsYesYes (ARM system images on Apple Silicon)
iOS simulatorsNoYes
API, performance, security, accessibilityYesYes

Device and browser matrix

PlatformWhat we test onLater (paid device cloud)
WebChromium, Firefox and WebKit; real Chrome and Edge (Windows) and real Safari (Mac); screen widths from mobile to large desktopReal browsers and OS versions on BrowserStack or LambdaTest
AndroidAndroid emulators and team Android phones; oldest supported and latest Android version; normal and large fontWider range of real devices
iOSiOS simulators (Mac) and team iPhones; oldest supported and latest iOS version; normal and large textWider range of real devices

The exact devices and screen sizes are finalized per project and recorded in qa-context.

Environments

EnvironmentUsed for
Preview environment per PRTests for changed areas before merge
QA / stagingSmoke on deploy, nightly regression, release candidate runs
ProductionRead-only smoke tests and monitoring with safe test accounts

CI runners: Linux GitHub-hosted runners for web, Android, API, security and performance. A self-hosted team Mac for nightly iOS runs; GitHub-hosted macOS runners only as a backup, because they cost about ten times more per minute.

7. Stack detection and tool selection

The qa-context skill detects each project's stack from evidence and picks the tools; nobody fills this in by hand. It reads the source code when we have it, the APK/IPA when we only have a build, or the live URL for web.

Fingerprint foundStackPrimary toolFallback
react-native in package.jsonReact NativeMaestroDetox
expo in package.jsonExpo (React Native)Maestro on dev buildsDetox
flutter in pubspec.yaml; libflutter.so in the APKFlutterMaestroPatrol
capacitor.config.* or Cordova config.xml; www/ in the APKHybridAppium (switches between web and native views)Maestro for native screens
Only Gradle/Kotlin or Xcode/SwiftNativeMaestroAppium
next, nuxt, angular, vue, react in package.json; __NEXT_DATA__ on the pageWebPlaywrightReal Chrome, Edge and Safari, run locally
OpenAPI file, express, nestjs, django, laravelBackendPlaywright API tests + SchemathesisBruno for manual work

The skill outputs four things: the detected stack with the file and line it came from, the chosen tools and why, a readiness list of what developers must add (section 12), and a confidence level. When signals conflict, it asks QA to confirm instead of guessing.

8. UI validation and responsive testing

UI validation checks that every screen looks and works as designed on every supported screen size, on web and mobile. Layout checks and screenshot comparison can fail a build; the AI review only raises findings for a person to confirm.

What we validate

CheckWebMobile
LayoutNo horizontal scroll; nothing off-screen or overlapping; grids and columns reflow at each widthEverything inside the screen; nothing hidden under the notch, status bar or keyboard
TextNo text cut off, truncated or overflowing its boxSame, also with large system text (1.3× and 2× on Android, larger Dynamic Type on iOS)
NavigationMenus adapt correctly, for example the desktop menu becoming a hamburger menuTabs, drawers and back navigation work in portrait and landscape
Images and mediaScale without stretching or cropping key contentSame, on phone and tablet sizes
Forms and touchInputs usable at every width; tap targets at least 44 pxThe keyboard never hides the active field; tap targets at least 44 pt
Design matchScreenshots compared with FigmaSame

Screen sizes: the exact web widths and mobile devices are finalized per project and recorded in qa-context; mobile is tested in portrait and landscape. Test data includes long names, long translations, empty states and maximum-length inputs, where most layout bugs appear.

How it is checked

LayerWhat it doesFails the build?
1. Layout checksPlaywright (web) and Maestro or Appium (mobile) measure element positions and text overflow at every sizeYes
2. Screenshot comparisonEach size is compared with its approved baselineYes; a person approves baseline changes
3. AI visual reviewAn agent reviews the screenshots for clipping, misalignment and design mismatchesNo: findings only

9. Recommended skills

We build our own skills, taking the best patterns from three public QA skill repos: short skill files, one shared project context, tests for every skill, and evidence before any conclusion. Twenty-three skills cover the whole QA cycle; the nine in phase 1 deliver most of the saving. Every skill below states when it runs, what it does, what it produces and who checks it; section 5 lists the tools.

PhaseSkillWhen it runsWhat it doesOutputHuman gate
1qa-contextProject start; when the stack changesDetects the stack from code, APK/IPA or URL; picks tools; records device matrix, environments, thresholds, test IDs and bug tracker; checks developer prerequisitesProject config + readiness listQA confirms
1spec-to-testsFeature ready for developmentReads story, PRD and Figma; lists gaps and conflicts; writes prioritised positive, negative and boundary cases and test outlinesSOP test cases, traceability matrix, question listQA approves cases
1web-e2e-authorAfter cases are approvedWrites Playwright tests with page objects, screen-width projects, layout checks, visual checks and axe-coreTest code in a PRPR review
1mobile-flow-authorAfter cases are approvedWrites Maestro flows across devices and orientations; Appium for real iPhones and hybrid appsFlow files in a PRPR review
1ci-pipelineProject setup; when triggers changeSets up the six triggers, sharding, evidence capture, quarantine and the team Mac runnerWorkflow files in a PRPR review
1failure-triageAfter any failed runReads traces, videos and logs; classifies each failure as product bug, test problem or environment, with evidenceTriage listQA confirms
1defect-reporterWhen triage confirms a bugChecks ClickUp or Linear for duplicates; writes the SOP defect with steps, expected vs actual, severity and evidence; links the test caseDefect with evidence in the trackerQA confirms severity
1test-quality-gateEvery PR that adds or changes testsRuns new tests against a planted fault; reruns them 5 times; blocks fixed sleeps, forced clicks, XPath and tests with no assertionPass/fail on the PRNone (automatic)
1qa-run-reportAfter every runBuilds the HTML summary and SOP-template documents; gives every test a confidence level; calculates the run confidence score; lists what was not testedHTML summary + SOP documentsQA reads the summary
2api-spec-to-testsWhen the OpenAPI spec changesGenerates API tests, response-schema checks, Schemathesis setup and contract checksTest code in a PRPR review
2responsive-layout-auditNightly; on demandRuns layout checks, screenshot comparison and AI visual review at every width, device and font scaleFindings with screenshotsQA confirms
2exploratory-agentFeature ready; before releaseExplores the app against a charter for a fixed time; records steps and evidenceFindings + candidate testsQA confirms each finding
2findings-conflictsFeature ready; after each runCompares requirements, Figma, API spec and actual behaviour; lists every mismatchConflict list with evidenceProduct Owner decides
2fix-verifierDefect moved to Ready for RetestReruns the defect's test and nearby tests on the same device; closes or reopens the defect with evidenceRetest result in the trackerNone; QA if flaky
2impact-selectorEvery PRMaps changed files or the ticket to the tests they affectTest list for the runNone
2test-data-factoryWhen tests need dataBuilds data factories and API seeding, with clean-up after each testData helpers in a PRPR review
2edge-case-mockerWhen tests need error statesSets up mocked errors, empty data, slow and offline responsesMock handlers in a PRPR review
2flake-managerNightly; weekly reviewFinds flaky tests from run history; quarantines them with owner and expiry; finds the root causeQuarantine list + fixesQA reviews weekly
2guarded-healerWhen locators break after a UI changeProposes new locators only; any assertion change is blockedFix in a PRPR review
3perf-baselineRelease candidate; monthlyWrites k6 load, stress and soak scripts and Lighthouse budgets; compares with the last baselinePerformance reportQA sets the limits
3security-baselineNightly; release candidateRuns the security scanners; removes duplicate findings; ranks by riskSecurity findingsReview every finding
3a11y-auditNightly (web); release candidate (mobile)Runs automated accessibility scans; prepares the screen-reader checklistAccessibility reportManual screen-reader pass
3post-release-watchAfter each releaseSets up production smoke tests and monitors; summarises the first 48 hoursMonitoring + 48-hour reportOn-call reviews alerts

Packaging and distribution

The skills, templates and presets live in one GitHub repository and ship as two versioned packages, so every project installs the same framework in minutes.

PackageContainsInstalled withUsed by
Claude Code pluginThe 23 skills, agent settings and MCP server config (Playwright, mobile-mcp, Maestro, Figma, ClickUp or Linear)/plugin marketplace add <org>/qa-framework, then /plugin install qa-frameworkQA engineers' Claude Code
npm package @<org>/qa-kitReport templates (HTML summary, execution sheet, defect, Test Summary Report), the report generator, Playwright and Maestro presets, layout-check helpers, CI workflow templatesnpm install -D @<org>/qa-kitEach project's test code and CI

The skills follow the open Agent Skills format, so the same folders also work in Codex, Cursor and other agents (npx skills add <org>/qa-framework).

Skill maintenance

10. Reporting and evidence

Every run produces the same report pack on every project: a one-page HTML summary, SOP-template documents, and defects with evidence attached in the project's bug tracker, ClickUp or Linear. qa-context records each project's tracker and devices, and qa-run-report fills the same templates every time, so nothing is rebuilt per project.

What every run delivers

DeliverableFormatReaderContents
Run summaryOne HTML file; opens in any browser, no loginEveryoneVerdict, run confidence, counts, top three risks, findings and conflicts, what was not tested, links to everything else
Execution sheetSOP test case template (XLSX)QAEvery test run: status, test confidence, environment, build, device, evidence link
Defect reportsSOP defect template, filed in ClickUp or LinearDevelopersSteps, expected vs actual, severity, priority, evidence attached
Test Summary ReportSOP template (DOCX/PDF/HTML)Sign-offFilled from run data; produced for release candidate runs only

Evidence on every failure

Each failure carries the items below. They are attached to the defect in the tracker, linked from the HTML summary and kept with the CI run for 90 days.

Confidence on every test

Each test result gets a confidence level in the execution sheet. It says how far the result can be trusted, whether the test passed or failed.

LevelRule
HighRan on every planned device and browser; same result every time with no retry; full evidence; proven to catch a planted fault; linked to an acceptance criterion
MediumRan on only some planned devices, needed one retry, or not yet proven against a planted fault
LowBlocked or skipped on planned devices, flaky, or written by AI and not yet reviewed

Confidence on every run

The run score is calculated from run data, never self-assessed by the agent. The weights can be tuned after the first measured releases.

FactorWeightMeasured as
Requirement coverage35%Acceptance criteria with at least one executed test ÷ all criteria
Test case execution20%Executed cases ÷ planned cases
Device and browser coverage20%Combinations run ÷ combinations in the plan
Stability15%Tests passing with no flaky retry ÷ all tests
Evidence completeness10%Failures with full evidence ÷ all failures

Target: 90% or higher for release. Two rules override the score: any open Critical or High defect means Not approved, and every area not tested is listed with its reason. The summary explains every point lost, so the score is never a black box.

Example run summary

One HTML page answers go or no-go, with every detail a click away

Rent Portal · Build 2.4.1 (RC) · Staging · 19 Nov 2026

NOT APPROVED
Run confidence 82% · target 90%
214 / 230Tests executed
201Passed
9Failed
2Flaky, quarantined

Test confidence: High 188 · Medium 20 · Low 6 (details in the execution sheet)


Top risks
  1. DEF-031 Critical · payment confirmation missing on iOS Safari · video · trace
  2. DEF-034 High · tenant list overflows at 360 px · screenshot
  3. Conflict · story allows 5 units, API accepts 50 · request log
Not tested

Push notifications on a real iPhone (no device) · load test (deferred to R2.5)

Why 82%

iPad landscape not run (−8) · 16 cases blocked by DEF-031 (−6) · 2 flaky tests (−4)

Links: execution sheet · defects in tracker · findings and conflicts · Test Summary Report · evidence

example run summary (HTML) · made-up figures

One screen answers go or no-go; every detail is one click away, so the summary never grows past a page.

11. Quality gates, reliability and data privacy

Agents write tests, test runs decide pass or fail, and people sign off. No rule below may be relaxed to go faster without QA approval.

#GateRuleEnforced by
1Prove it failsEvery new or AI-written test must fail against a planted fault or a reverted fix before it joins regressiontest-quality-gate
2Prove it is stableA new test must pass 5 reruns in a rowtest-quality-gate
3Healer limitsAutomatic repair may change locators and waits only; any assertion change is blocked; every fix goes through PR reviewguarded-healer + CI check
4No hidden flakinessA retry only detects flakiness. A flaky test is quarantined with an owner and expiry date, and reports count it as not passedflake-manager
5Evidence requiredEvery failure carries a screenshot, video or trace, logs and device details; a defect without evidence is not fileddefect-reporter
6Strength checkEach quarter, planted faults in the app must be caught by the suite; Stryker mutation score for unit leveltest-quality-gate
7Exit criteriaSOP exit criteria apply unchanged: 0 open Critical/High, pass rate ≥ 95%, plus run confidence ≥ 90%qa-run-report

Test design standards (adapted from the UKHSA automation design guide)

Data privacy

12. Developer prerequisites

Reliable automation depends on five things from developers, agreed at every project kickoff. Without them tests become flaky, and the framework cannot deliver.

NeedWhyOwner
Stable test IDs: data-testid (web), testID / accessibilityIdentifier (mobile)Tests survive UI changes; no fragile locatorsDevelopers
A way to create test data through the API or a seed scriptEach test starts from known data; no shared accountsDevelopers
A preview environment per PR, or a stable QA environmentTests run before merge, not afterDevOps
An up-to-date OpenAPI specAPI tests, fuzzing and contract checks are generated from itBackend developers
CI builds of the Android APK and iOS simulator app on every mergeMobile tests always run on the latest buildMobile developers

qa-context checks each item and lists what is missing as a developer to-do.

13. Metrics

Five numbers, tracked per release, show whether the framework works. They are targets, not results: we start measuring them only once automation is running. If escaped defects rise while time falls, the framework is failing, and we fix it before going faster.

MetricTargetSource
Regression automated≥ 90% of regression casesExecution sheet
Manual execution effortDown 70–80% from the first measured baselineTime logs
Flaky rate< 1% of test runsAllure history
Escaped defectsNo increase release on releaseBug tracker, production bugs
Feature-ready to sign-offShorter every quarterBug tracker dates

Measure one real feature before the rollout starts so we have a baseline.

↑ Top