1. Purpose and outcomes
This framework moves our web and mobile testing from mostly manual to mostly automated without lowering any quality gate. AI agents write tests, investigate failures and draft documents. Deterministic test runs decide pass or fail. People approve.
| Outcome | How we know it is met |
|---|---|
| Much less manual testing | Targets: 70–80% of execution effort automated; smoke and retest fully automated; at least 90% of regression automated. Measured once automation is running |
| On-time delivery | Testing runs in parallel with development, not as a phase after it |
| Results we can trust | Every result is backed by evidence; flaky tests never count as passed |
| Clear output for everyone | One report pack per run, in our SOP templates, with a calculated confidence score |
| Fewer escaped bugs | Escaped defects tracked per release and must not rise as speed increases |
The framework replaces the tool-neutral parts of QA-SOP-001 with a concrete stack. The SOP's roles, templates, approvals and exit criteria still apply.
SOP: https://panthos-qa-sop.vercel.app
2. Scope
The framework covers every test type in SOP section 5.3 for web, Android, iOS and API. Each type is automated where a machine can judge the result reliably, and stays human where judgement, real hardware or business acceptance is needed.
| Test type | Automated | Stays human |
|---|---|---|
| Functional | Happy paths, negative and boundary cases | Approving new and AI-written tests |
| Smoke | Target: all smoke cases, on every build | None |
| Regression | Target: at least 90% of regression cases | Triage of failures |
| Retesting | Rerun of the test tagged with the defect ID | Confirming flaky results |
| API | Functional, schema, fuzzing, contract | Manual exploration in Postman |
| Integration and system | Full journeys on staging and emulators | None |
| Acceptance | Each acceptance criterion maps to a test | Product Owner acceptance (UAT) |
| Exploratory | Agent sessions that report findings | Confirming every finding; human sessions on new features |
| UI/UX | Screenshot baselines, Figma comparison | Judging whether a mismatch matters |
| Usability | None | Fully human, with agent notes |
| Responsive (cut-off text and elements) | Layout checks at every screen width and device size | Approving screenshot baseline changes |
| Compatibility | Local browsers (Chrome, Edge, Firefox, WebKit, Safari), emulators, simulators, team phones | Final look on 1–2 real devices |
| Mobile-specific | Orientation, install/upgrade, permissions, background/resume, network, emulated calls and SMS (Android), simulated push (iOS) | Real calls, camera and biometrics on team phones |
| Performance | Load, stress, soak; page-speed budgets | Setting the limits |
| Security | Scans of the running app, code, dependencies, app binary | Reviewing every finding |
| Accessibility | WCAG scans in every web test | Screen-reader pass (TalkBack, VoiceOver) |
Out of scope for now: paid device clouds (BrowserStack, LambdaTest).
3. Architecture
The framework has five layers, and only the execution layer decides pass or fail. Agents write tests and explain results; they never give the verdict.
Everything the agent layer produces lands in Git as a reviewed PR, so every test that counts has been read by a person and run by CI.
4. Automation workflow
Automation runs at six points from feature to production, plus a weekly health check, and each point has a gate that can stop the change. Testing starts when a feature is ready for development, not after it is built.
A failed gate sends the change back to its developer with the evidence; it re-enters the flow at the same point after the fix.
| # | Trigger | What runs | Where | Report |
|---|---|---|---|---|
| 1 | Feature ready (on demand) | spec-to-tests drafts cases and tests; exploratory agent session; new tests pass test-quality-gate | Local, CI | Feature test report |
| 2 | Every pull request | Tests for changed areas on Chromium + one phone width; API contract; accessibility and visual checks on touched pages | CI, preview environment | Short PR comment |
| 3 | Deploy to QA/staging | Web smoke + API health; mobile smoke on one Android emulator and one iOS simulator | CI | Build check report |
| 4 | Nightly | Full regression on all browsers, widths and emulators; API fuzzing; ZAP scan; Lighthouse; flaky-test detection | CI, team Mac | Nightly report |
| 5 | Release candidate | Full regression on all browsers, emulators, simulators and team phones; k6 performance; full accessibility and security checks; mobile internal testing track | CI, team Mac, team phones | Release report (SOP Test Summary Report) |
| 6 | After release | Read-only production smoke; synthetic monitors every 15–30 min; Sentry crash and error watch for 48 hours | Checkly, Sentry | Post-release report |
| 7 | Weekly and quarterly | Flaky-test review; planted-fault check; suite clean-up; metrics | CI | Suite health report |
Retest flow: the developer moves a defect to Ready for Retest in ClickUp or Linear; CI reruns the test tagged with that defect ID plus nearby tests on the same device; a pass closes the defect with evidence, a fail reopens it with new evidence.
5. Toolset by test type
The core is eight free tools: Playwright, Maestro, k6, OWASP ZAP, MobSF, axe-core, Schemathesis and GitHub Actions. All test code we write is TypeScript/JavaScript (Playwright, k6) or YAML (Maestro flows, CI workflows). The Python- and Java-based tools (Schemathesis, ZAP, MobSF) run as ready-made command-line tools or Docker containers, so the team never writes or maintains Python or Java. Every tool is free except real cross-device and cross-browser testing, which needs a paid device cloud (BrowserStack or LambdaTest). Until we start that, we test on local browsers, emulators, simulators and the team's own phones. The only other costs are Claude seats and a Figma Dev Mode seat (only if needed).
Human review levels: None = fully automated pass/fail, a person looks only at failures. Low = a person approves changes or spot-checks. Medium = a person reviews every output. High = led by a person, the tool assists.
| Test type | Web | Mobile | API / backend | Cost | Human review |
|---|---|---|---|---|---|
| Functional | Playwright | Maestro; Appium + WebdriverIO as fallback | Playwright API tests | Free | Low: review new and AI-written tests once |
| Smoke | Playwright @smoke | Maestro smoke flows | Health-check tests | Free | None |
| Regression | Playwright @regression | Maestro regression flows | Playwright API + contract | Free | None: triage failures only |
| Retesting | Rerun test tagged @DEF-xxx | Same, in Maestro | Same | Free | None; Low if flaky |
| API | Playwright API + Schemathesis fuzzing; Pact for contracts; Bruno or Postman for manual | Free | Low: confirm fuzzing findings | ||
| Integration | Playwright on staging + Testcontainers + MSW | Maestro on staging | Testcontainers, WireMock | Free | Low |
| System | Full Playwright journeys | Full Maestro journeys | Same | Free | None |
| Acceptance | Tests named after each acceptance criterion | Same | Same | Free | Medium: Product Owner accepts |
| Exploratory | Agent via Playwright CLI/MCP | Agent via mobile-mcp / Maestro MCP | Agent with Bruno | Free + Claude seats | Medium: confirm every finding |
| UI/UX design match | Screenshots + Figma MCP | Screenshots + Figma MCP | Figma paid seat with Dev Mode | Medium | |
| Usability | Human session | Human session | Free | High | |
| UI validation and responsive | Playwright layout checks + screenshot baselines | Maestro + element-position checks + baselines | Free | None for checks; Low for baselines | |
| Visual review (optional) | Argos | Argos | Free tier (5,000 screenshots a month) | Low | |
| Cross-browser | Playwright: Chromium, Firefox, WebKit, real Chrome and real Edge; real Safari on Mac | Mobile Safari in iOS simulators (Appium); Chrome in Android emulators | Free | None | |
| Cross-device | Android emulators and iOS simulators across the device matrix (section 6) | Free | None | ||
| Real devices | Team phones over USB: Maestro for Android, Appium for iPhone (Maestro has no official real-iPhone support); Firebase Test Lab free quota, 10 virtual + 5 physical runs a day | Free | Low: final look and feel | ||
| Mobile-specific | Maestro + emulator controls: orientation, network speed, emulated calls and SMS (Android), simulated push (iOS); Network Link Conditioner on Mac | Free | None; High for real calls, camera, biometrics | ||
| Performance | Lighthouse CI | Perfetto + Macrobenchmark (Android), Xcode Instruments (iOS) | k6 | Free | Low: a person sets the limits |
| Security | OWASP ZAP + role/permission tests | MobSF (APK/IPA scan) | ZAP API scan, Semgrep, Trivy, Gitleaks | Free | Medium: review every finding |
| Accessibility | axe-core inside Playwright | Accessibility Scanner (Android), Accessibility Inspector (iOS) | Free | None for web scans; High for screen readers | |
| Cross-device and cross-browser on real devices | BrowserStack or LambdaTest: real browsers and OS versions | BrowserStack or LambdaTest: real phones and tablets | Paid; the only paid testing tool, added when we start real-device testing | None |
Supporting tools
| Purpose | Tool | Cost |
|---|---|---|
| CI and scheduling | GitHub Actions | Free minutes; team Mac runner is free |
| Raw reports and flaky-test history | Allure + Playwright HTML | Free |
| SOP-template reports | qa-run-report skill | Free |
| Defects | ClickUp or Linear (set per project) | Existing plan |
| Production monitoring and errors | Checkly, Sentry | Free tiers |
| Agent layer | Claude Code + our skills | Claude seat per QA engineer |
6. Devices and environments
We test on what we have: local browsers, emulators and simulators on team Windows and Mac machines, and the team's own phones. Real cross-device and cross-browser testing on BrowserStack or LambdaTest comes later. iOS simulators and real desktop Safari need a Mac; everything else runs on both.
| Work | Windows | Mac |
|---|---|---|
| Web tests, all screen widths | Yes | Yes |
| Real desktop Safari | No | Yes |
| Android emulators | Yes | Yes (ARM system images on Apple Silicon) |
| iOS simulators | No | Yes |
| API, performance, security, accessibility | Yes | Yes |
Device and browser matrix
| Platform | What we test on | Later (paid device cloud) |
|---|---|---|
| Web | Chromium, Firefox and WebKit; real Chrome and Edge (Windows) and real Safari (Mac); screen widths from mobile to large desktop | Real browsers and OS versions on BrowserStack or LambdaTest |
| Android | Android emulators and team Android phones; oldest supported and latest Android version; normal and large font | Wider range of real devices |
| iOS | iOS simulators (Mac) and team iPhones; oldest supported and latest iOS version; normal and large text | Wider range of real devices |
The exact devices and screen sizes are finalized per project and recorded in qa-context.
Environments
| Environment | Used for |
|---|---|
| Preview environment per PR | Tests for changed areas before merge |
| QA / staging | Smoke on deploy, nightly regression, release candidate runs |
| Production | Read-only smoke tests and monitoring with safe test accounts |
CI runners: Linux GitHub-hosted runners for web, Android, API, security and performance. A self-hosted team Mac for nightly iOS runs; GitHub-hosted macOS runners only as a backup, because they cost about ten times more per minute.
7. Stack detection and tool selection
The qa-context skill detects each project's stack from evidence and picks the tools; nobody fills this in by hand. It reads the source code when we have it, the APK/IPA when we only have a build, or the live URL for web.
| Fingerprint found | Stack | Primary tool | Fallback |
|---|---|---|---|
react-native in package.json | React Native | Maestro | Detox |
expo in package.json | Expo (React Native) | Maestro on dev builds | Detox |
flutter in pubspec.yaml; libflutter.so in the APK | Flutter | Maestro | Patrol |
capacitor.config.* or Cordova config.xml; www/ in the APK | Hybrid | Appium (switches between web and native views) | Maestro for native screens |
| Only Gradle/Kotlin or Xcode/Swift | Native | Maestro | Appium |
next, nuxt, angular, vue, react in package.json; __NEXT_DATA__ on the page | Web | Playwright | Real Chrome, Edge and Safari, run locally |
| OpenAPI file, express, nestjs, django, laravel | Backend | Playwright API tests + Schemathesis | Bruno for manual work |
The skill outputs four things: the detected stack with the file and line it came from, the chosen tools and why, a readiness list of what developers must add (section 12), and a confidence level. When signals conflict, it asks QA to confirm instead of guessing.
8. UI validation and responsive testing
UI validation checks that every screen looks and works as designed on every supported screen size, on web and mobile. Layout checks and screenshot comparison can fail a build; the AI review only raises findings for a person to confirm.
What we validate
| Check | Web | Mobile |
|---|---|---|
| Layout | No horizontal scroll; nothing off-screen or overlapping; grids and columns reflow at each width | Everything inside the screen; nothing hidden under the notch, status bar or keyboard |
| Text | No text cut off, truncated or overflowing its box | Same, also with large system text (1.3× and 2× on Android, larger Dynamic Type on iOS) |
| Navigation | Menus adapt correctly, for example the desktop menu becoming a hamburger menu | Tabs, drawers and back navigation work in portrait and landscape |
| Images and media | Scale without stretching or cropping key content | Same, on phone and tablet sizes |
| Forms and touch | Inputs usable at every width; tap targets at least 44 px | The keyboard never hides the active field; tap targets at least 44 pt |
| Design match | Screenshots compared with Figma | Same |
Screen sizes: the exact web widths and mobile devices are finalized per project and recorded in qa-context; mobile is tested in portrait and landscape. Test data includes long names, long translations, empty states and maximum-length inputs, where most layout bugs appear.
How it is checked
| Layer | What it does | Fails the build? |
|---|---|---|
| 1. Layout checks | Playwright (web) and Maestro or Appium (mobile) measure element positions and text overflow at every size | Yes |
| 2. Screenshot comparison | Each size is compared with its approved baseline | Yes; a person approves baseline changes |
| 3. AI visual review | An agent reviews the screenshots for clipping, misalignment and design mismatches | No: findings only |
9. Recommended skills
We build our own skills, taking the best patterns from three public QA skill repos: short skill files, one shared project context, tests for every skill, and evidence before any conclusion. Twenty-three skills cover the whole QA cycle; the nine in phase 1 deliver most of the saving. Every skill below states when it runs, what it does, what it produces and who checks it; section 5 lists the tools.
| Phase | Skill | When it runs | What it does | Output | Human gate |
|---|---|---|---|---|---|
| 1 | qa-context | Project start; when the stack changes | Detects the stack from code, APK/IPA or URL; picks tools; records device matrix, environments, thresholds, test IDs and bug tracker; checks developer prerequisites | Project config + readiness list | QA confirms |
| 1 | spec-to-tests | Feature ready for development | Reads story, PRD and Figma; lists gaps and conflicts; writes prioritised positive, negative and boundary cases and test outlines | SOP test cases, traceability matrix, question list | QA approves cases |
| 1 | web-e2e-author | After cases are approved | Writes Playwright tests with page objects, screen-width projects, layout checks, visual checks and axe-core | Test code in a PR | PR review |
| 1 | mobile-flow-author | After cases are approved | Writes Maestro flows across devices and orientations; Appium for real iPhones and hybrid apps | Flow files in a PR | PR review |
| 1 | ci-pipeline | Project setup; when triggers change | Sets up the six triggers, sharding, evidence capture, quarantine and the team Mac runner | Workflow files in a PR | PR review |
| 1 | failure-triage | After any failed run | Reads traces, videos and logs; classifies each failure as product bug, test problem or environment, with evidence | Triage list | QA confirms |
| 1 | defect-reporter | When triage confirms a bug | Checks ClickUp or Linear for duplicates; writes the SOP defect with steps, expected vs actual, severity and evidence; links the test case | Defect with evidence in the tracker | QA confirms severity |
| 1 | test-quality-gate | Every PR that adds or changes tests | Runs new tests against a planted fault; reruns them 5 times; blocks fixed sleeps, forced clicks, XPath and tests with no assertion | Pass/fail on the PR | None (automatic) |
| 1 | qa-run-report | After every run | Builds the HTML summary and SOP-template documents; gives every test a confidence level; calculates the run confidence score; lists what was not tested | HTML summary + SOP documents | QA reads the summary |
| 2 | api-spec-to-tests | When the OpenAPI spec changes | Generates API tests, response-schema checks, Schemathesis setup and contract checks | Test code in a PR | PR review |
| 2 | responsive-layout-audit | Nightly; on demand | Runs layout checks, screenshot comparison and AI visual review at every width, device and font scale | Findings with screenshots | QA confirms |
| 2 | exploratory-agent | Feature ready; before release | Explores the app against a charter for a fixed time; records steps and evidence | Findings + candidate tests | QA confirms each finding |
| 2 | findings-conflicts | Feature ready; after each run | Compares requirements, Figma, API spec and actual behaviour; lists every mismatch | Conflict list with evidence | Product Owner decides |
| 2 | fix-verifier | Defect moved to Ready for Retest | Reruns the defect's test and nearby tests on the same device; closes or reopens the defect with evidence | Retest result in the tracker | None; QA if flaky |
| 2 | impact-selector | Every PR | Maps changed files or the ticket to the tests they affect | Test list for the run | None |
| 2 | test-data-factory | When tests need data | Builds data factories and API seeding, with clean-up after each test | Data helpers in a PR | PR review |
| 2 | edge-case-mocker | When tests need error states | Sets up mocked errors, empty data, slow and offline responses | Mock handlers in a PR | PR review |
| 2 | flake-manager | Nightly; weekly review | Finds flaky tests from run history; quarantines them with owner and expiry; finds the root cause | Quarantine list + fixes | QA reviews weekly |
| 2 | guarded-healer | When locators break after a UI change | Proposes new locators only; any assertion change is blocked | Fix in a PR | PR review |
| 3 | perf-baseline | Release candidate; monthly | Writes k6 load, stress and soak scripts and Lighthouse budgets; compares with the last baseline | Performance report | QA sets the limits |
| 3 | security-baseline | Nightly; release candidate | Runs the security scanners; removes duplicate findings; ranks by risk | Security findings | Review every finding |
| 3 | a11y-audit | Nightly (web); release candidate (mobile) | Runs automated accessibility scans; prepares the screen-reader checklist | Accessibility report | Manual screen-reader pass |
| 3 | post-release-watch | After each release | Sets up production smoke tests and monitors; summarises the first 48 hours | Monitoring + 48-hour report | On-call reviews alerts |
Packaging and distribution
The skills, templates and presets live in one GitHub repository and ship as two versioned packages, so every project installs the same framework in minutes.
| Package | Contains | Installed with | Used by |
|---|---|---|---|
| Claude Code plugin | The 23 skills, agent settings and MCP server config (Playwright, mobile-mcp, Maestro, Figma, ClickUp or Linear) | /plugin marketplace add <org>/qa-framework, then /plugin install qa-framework | QA engineers' Claude Code |
npm package @<org>/qa-kit | Report templates (HTML summary, execution sheet, defect, Test Summary Report), the report generator, Playwright and Maestro presets, layout-check helpers, CI workflow templates | npm install -D @<org>/qa-kit | Each project's test code and CI |
The skills follow the open Agent Skills format, so the same folders also work in Codex, Cursor and other agents (npx skills add <org>/qa-framework).
Skill maintenance
- Each skill has a named owner who approves every change to it
- Every PR runs the skill tests and planted-fault fixtures; a release needs all of them green, a version number and a changelog entry
- Projects pin a framework version and upgrade on purpose; tool and Claude model versions are checked monthly and upgraded only when all skill tests pass
- When a skill gives a wrong result, the example becomes a new skill test; we track how often QA rewrites each skill's output
- Third-party skills and MCP servers are read in full before install and pinned; no secrets or client data in the repo, which starts private
- Unused skills are retired each quarter; SOP or template changes trigger updates to the skills that fill them
10. Reporting and evidence
Every run produces the same report pack on every project: a one-page HTML summary, SOP-template documents, and defects with evidence attached in the project's bug tracker, ClickUp or Linear. qa-context records each project's tracker and devices, and qa-run-report fills the same templates every time, so nothing is rebuilt per project.
What every run delivers
| Deliverable | Format | Reader | Contents |
|---|---|---|---|
| Run summary | One HTML file; opens in any browser, no login | Everyone | Verdict, run confidence, counts, top three risks, findings and conflicts, what was not tested, links to everything else |
| Execution sheet | SOP test case template (XLSX) | QA | Every test run: status, test confidence, environment, build, device, evidence link |
| Defect reports | SOP defect template, filed in ClickUp or Linear | Developers | Steps, expected vs actual, severity, priority, evidence attached |
| Test Summary Report | SOP template (DOCX/PDF/HTML) | Sign-off | Filled from run data; produced for release candidate runs only |
Evidence on every failure
Each failure carries the items below. They are attached to the defect in the tracker, linked from the HTML summary and kept with the CI run for 90 days.
- Screenshot at the moment of failure, and a video of the run
- Step trace: Playwright trace (web) or Maestro command log (mobile)
- Logs: browser console, Android logcat or iOS device log
- Network: request and response for each failing API call
- Context: build, environment, browser or device, OS, screen size, orientation and font scale
- Exact steps to reproduce, written for a developer who has not seen the test
Confidence on every test
Each test result gets a confidence level in the execution sheet. It says how far the result can be trusted, whether the test passed or failed.
| Level | Rule |
|---|---|
| High | Ran on every planned device and browser; same result every time with no retry; full evidence; proven to catch a planted fault; linked to an acceptance criterion |
| Medium | Ran on only some planned devices, needed one retry, or not yet proven against a planted fault |
| Low | Blocked or skipped on planned devices, flaky, or written by AI and not yet reviewed |
Confidence on every run
The run score is calculated from run data, never self-assessed by the agent. The weights can be tuned after the first measured releases.
| Factor | Weight | Measured as |
|---|---|---|
| Requirement coverage | 35% | Acceptance criteria with at least one executed test ÷ all criteria |
| Test case execution | 20% | Executed cases ÷ planned cases |
| Device and browser coverage | 20% | Combinations run ÷ combinations in the plan |
| Stability | 15% | Tests passing with no flaky retry ÷ all tests |
| Evidence completeness | 10% | Failures with full evidence ÷ all failures |
Target: 90% or higher for release. Two rules override the score: any open Critical or High defect means Not approved, and every area not tested is listed with its reason. The summary explains every point lost, so the score is never a black box.
Example run summary
Rent Portal · Build 2.4.1 (RC) · Staging · 19 Nov 2026
Test confidence: High 188 · Medium 20 · Low 6 (details in the execution sheet)
Top risks
- DEF-031 Critical · payment confirmation missing on iOS Safari · video · trace
- DEF-034 High · tenant list overflows at 360 px · screenshot
- Conflict · story allows 5 units, API accepts 50 · request log
Not tested
Push notifications on a real iPhone (no device) · load test (deferred to R2.5)
Why 82%
iPad landscape not run (−8) · 16 cases blocked by DEF-031 (−6) · 2 flaky tests (−4)
Links: execution sheet · defects in tracker · findings and conflicts · Test Summary Report · evidence
One screen answers go or no-go; every detail is one click away, so the summary never grows past a page.
11. Quality gates, reliability and data privacy
Agents write tests, test runs decide pass or fail, and people sign off. No rule below may be relaxed to go faster without QA approval.
| # | Gate | Rule | Enforced by |
|---|---|---|---|
| 1 | Prove it fails | Every new or AI-written test must fail against a planted fault or a reverted fix before it joins regression | test-quality-gate |
| 2 | Prove it is stable | A new test must pass 5 reruns in a row | test-quality-gate |
| 3 | Healer limits | Automatic repair may change locators and waits only; any assertion change is blocked; every fix goes through PR review | guarded-healer + CI check |
| 4 | No hidden flakiness | A retry only detects flakiness. A flaky test is quarantined with an owner and expiry date, and reports count it as not passed | flake-manager |
| 5 | Evidence required | Every failure carries a screenshot, video or trace, logs and device details; a defect without evidence is not filed | defect-reporter |
| 6 | Strength check | Each quarter, planted faults in the app must be caught by the suite; Stryker mutation score for unit level | test-quality-gate |
| 7 | Exit criteria | SOP exit criteria apply unchanged: 0 open Critical/High, pass rate ≥ 95%, plus run confidence ≥ 90% | qa-run-report |
Test design standards (adapted from the UKHSA automation design guide)
- One scenario per test, with a clear descriptive name; no dependence on other tests
- Page Object Model for web; shared flows (login, navigation) as reusable steps on mobile
- Locators by role or test ID (
data-testid,testID,accessibilityIdentifier); no XPath, no long CSS chains - No fixed sleeps; rely on the tool's automatic waiting and explicit conditions
- Test data separate from test logic, created per test and cleaned up afterwards; anonymised, never real customer data
- URLs, credentials and environment values in config files and CI secrets, never in test code
- Assertions state the expected result in plain words
- Clean-up runs even when a test fails
Data privacy
- Synthetic or anonymised test data only; never production data or real customer accounts
- Sensitive fields are masked in screenshots; tokens, passwords, emails, phone and card numbers are removed from logs before evidence is attached
- Agents work on QA/staging with test accounts only; no production data, secrets or confidential client documents go into prompts; we use a Claude business plan whose terms exclude training on our data
- Production checks are read-only and use test accounts
- Evidence needs a login to view; CI evidence is deleted after 90 days
- Each client's data rules are recorded in qa-context; the stricter rule always wins
12. Developer prerequisites
Reliable automation depends on five things from developers, agreed at every project kickoff. Without them tests become flaky, and the framework cannot deliver.
| Need | Why | Owner |
|---|---|---|
Stable test IDs: data-testid (web), testID / accessibilityIdentifier (mobile) | Tests survive UI changes; no fragile locators | Developers |
| A way to create test data through the API or a seed script | Each test starts from known data; no shared accounts | Developers |
| A preview environment per PR, or a stable QA environment | Tests run before merge, not after | DevOps |
| An up-to-date OpenAPI spec | API tests, fuzzing and contract checks are generated from it | Backend developers |
| CI builds of the Android APK and iOS simulator app on every merge | Mobile tests always run on the latest build | Mobile developers |
qa-context checks each item and lists what is missing as a developer to-do.
13. Metrics
Five numbers, tracked per release, show whether the framework works. They are targets, not results: we start measuring them only once automation is running. If escaped defects rise while time falls, the framework is failing, and we fix it before going faster.
| Metric | Target | Source |
|---|---|---|
| Regression automated | ≥ 90% of regression cases | Execution sheet |
| Manual execution effort | Down 70–80% from the first measured baseline | Time logs |
| Flaky rate | < 1% of test runs | Allure history |
| Escaped defects | No increase release on release | Bug tracker, production bugs |
| Feature-ready to sign-off | Shorter every quarter | Bug tracker dates |
Measure one real feature before the rollout starts so we have a baseline.