FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat(a11y): add mobile automation support by morning4coffe-dev · Pull Request #23935 · unoplatform/uno · GitHub

feat(a11y): add mobile automation support - #23935

Open
morning4coffe-dev wants to merge 30 commits into
dev/doti/a11y-parity-remediation-implfrom
005-mobile-a11y-automation
Open

feat(a11y): add mobile automation support#23935
morning4coffe-dev wants to merge 30 commits into
dev/doti/a11y-parity-remediation-implfrom
005-mobile-a11y-automation

Conversation

morning4coffe-dev commented Jul 31, 2026
edited
Loading

Copy link
Copy Markdown
Member

GitHub Issue: closes #23934

PR Type:

✨ Feature

What changed? 🚀

This adds first-class accessibility and automation support for the Skia Android and iOS renderers:

  • Projects the WinUI automation-peer tree into native Android accessibility nodes and iOS accessibility elements.
  • Routes native actions, focus, property changes, structure changes, announcements, and role descriptions through live automation peers.
  • Exposes stable automation identifiers for UIAutomator, XCUITest, and Appium-compatible tooling while redacting secure text.
  • Adds native-node, lifecycle, performance, capability-matrix, and SamplesApp automation coverage.
  • Extends the Android and iOS Skia CI stages with mobile accessibility tests.

The latest revision closes parity gaps found by auditing against the MUX peer/provider sources and the native AccessibilityNodeInfo / UIAccessibility contracts: one ItemAutomationPeer per item across the children tree and the pattern providers, real word/line/paragraph granularity in TextRangeAdapter, XAML focus mirrored into Android input focus, app-private custom-action IDs, the UIKit pageScrolled notification, iOS 17+ assertive announcements and the updatesFrequently trait, plus removal of unreachable action arms.

This PR is stacked on #23722, which supplies the shared Skia accessibility remediation consumed by the mobile renderers.

Validation

  • Analyzer-enabled Release builds (ContinuousIntegrationBuild=true) succeed for SamplesApp.Skia.Generic (net10.0), Uno.UI.Runtime.Skia.Android (net10.0-android) and Uno.UI.Runtime.Skia.AppleUIKit (net9.0-ios18.0), after fixing 11 CI-blocking analyzer violations in the new code.
  • 138 accessibility runtime tests pass on Skia Desktop, 0 failed. The three new parity tests were verified fail-before / pass-after.
  • Previous revision: 372 Android runtime tests and 14 UIAutomator tests passed on the emulator; a clean APK published, installed and ran.

Note: the Azure Uno.UI - CI pipeline does not trigger here because .vsts-ci.yml only lists master, release/*, feature/* and legacy/* in its pr: filter and this PR targets the stacked branch. It runs once the stack is retargeted to master.

PR Checklist ✅

github-actions Bot added platform/wasm 🌐 Categorizes an issue or PR as relevant to the WebAssembly platform platform/android 🤖 Categorizes an issue or PR as relevant to the Android platform platform/macos 🍏 Categorizes an issue or PR as relevant to the macOS platform platform/ios 🍎 Categorizes an issue or PR as relevant to the iOS platform area/skia ✏️ Categorizes an issue or PR as relevant to Skia area/build Categorizes an issue or PR as relevant to build infrastructure area/automation Categorizes an issue or PR as relevant to project automation kind/documentation labels Jul 31, 2026

github-actions Bot commented Jul 31, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude finished @morning4coffe-dev's task in 6m 12s —— View job


Code Review

Tasks:

  • Gather context and understand the PR
  • Review core accessibility implementation files (Android + iOS)
  • Review shared/cross-platform accessibility files
  • Post inline review comments
  • Post summary

Verdict: fix-first — one confirmed compilation bug blocks the PR; two code-style issues should be resolved before merge.

Issues (3 inline comments posted)

🔴 Critical — compilation error on Android and iOS (×2)

SkiaAccessibilityBase.UpdateRoleDescription is declared abstract. Every other concrete subclass provides an override (Win32Accessibility, MacOSAccessibility, WebAssemblyAccessibility), but AndroidSkiaAccessibility and AppleUIKitAccessibility do not. Both sealed classes will fail to compile on their respective targets. The fix is a one-liner => ScheduleInvalidation(handle) / => InvalidateElement(handle) modelled after the other Update* overrides in the same file (see inline comments on line 498 and line 1669).

🟡 Moderate — brace-style violations in ExecuteAction (iOS, ×9)

Nine if (peer is null || !peer.IsEnabled()) return false; guards in AppleUIKitAccessibility.ExecuteAction have no braces. The repo's code-style rule (Allman, always-braces) is enforced by CI analyzers when UnoFastDevBuild=false. These will become analyzer warnings on the PR CI run.

🟡 Moderate — dead code in InvokeMatchingCustomAction

ScrollForward → "Scroll Forward" and ScrollBackward → "Scroll Backward" in the switch expression inside InvokeMatchingCustomAction can never be reached: the outer ExecuteAction switch handles those two enum values with explicit case arms before the default: that delegates here.


Cross-cutting observations (no inline comment needed)

  • Design is sound. The adapter/router split (per-window SkiaAccessibilityBase subclasses, process-wide AccessibilityRouter fan-out) is clean. The Android swap-buffer invalidation pattern and iOS live-pull approach are appropriate to their respective native models.
  • Event contract is respected. No new event Action / event Action<T> declarations; all hook points use typed Func<> / Action<> delegate properties (not event declarations), which is correct.
  • Thread discipline. The Android coalesced-flush and iOS main-thread marshal patterns are consistent and correct. AnnounceOnPlatform is properly gated by thread checks on both platforms.
  • AccessibilityPeerHelper accumulates static Func/Action hook slots. 37 static nullable delegates is a wide surface. Consider whether some of these can be collapsed into an interface per platform to make the coupling explicit and the registration lifecycle clearer.
  • FindFirstFocusableChild / EnumerateFocusableChildren (pre-existing): These use unbounded recursion without the MaxTreeDepth = 1000 guard that GetPeerTree applies. Not introduced by this PR, but worth tracking.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude finished @morning4coffe-dev's task in 12m 10s —— View job


Claude Code Review — Round 3 Final

PR #23935 · feat(a11y): add mobile automation support


Verdict: ✅ SHIP

All round-1 and round-2 blockers are resolved. The updated diff (rounds of commits since the last review) introduced no new significant issues.


Previously Flagged Issues — Status

Finding Round Status
Missing UpdateRoleDescription override in both adapters (compile blocker) 1 ✅ Fixed
15 × lambda guard if (!weakSelf.TryGetTarget) without Allman braces 2 ✅ Fixed
ExecuteAction case bodies missing braces 2 ✅ Fixed
Dead ScrollForward/ScrollBackward arms in InvokeMatchingCustomAction 2 ✅ Fixed (removed)
Fragile BindingFlags.NonPublic reflection behind dead #if __SKIA__ in test 2 ✅ Fixed — now direct textBox.IsBackwardSelection (commit fc6c0c55)

Round 3 Scan — No New Issues

The following areas were reviewed against the new commits (fix(ios), fix(android), fix(a11y) × several, test(a11y) × two):

UnoExploreByTouchHelper.cs (4 132 lines)

  • Virtual-ID registry cleanup in PruneRegistryToCurrentTree and ClearAdapter — correct; _orderedIds is cleared on the next EnsureVisibleTreeBuilt call triggered by MarkAccessibilityTreeDirty.
  • _settingNativeFocus re-entry guard correctly scoped around RequestNativeFocusById and SyncNativeKeyboardFocus; no XAML→native→XAML loop.
  • fix(android): only mirror focus when host owns it — guard if (!_host.IsFocused) return; in SyncNativeKeyboardFocus is correct; prevents fighting the invisible text-input view for focus.
  • OnPerformActionForVirtualView action dispatch — no dead arms; all branches have proper braces.
  • Item peer / realized-container identity: GetOrCreatePeerVirtualId uses ConditionalWeakTable<AutomationPeer, VirtualIdBox> keyed by peer instance — recycled containers' new ItemAutomationPeer gets a fresh ID; stable across re-realization.

AndroidSkiaAccessibility.cs (772 lines)

  • Coalesced invalidation (FlushPendingInvalidations) — handle-to-ID deduplication via _flushingInvalidationTargets is correct.
  • AnnounceOnPlatform off-thread dispatch — stale-helper guard if (IsDisposed || !ReferenceEquals(_helper, helper)) is correct.
  • Detach() — all batched state is cleared before reference-equality guards fire.

AppleUIKitAccessibility.cs (2 924 lines)

  • Formatter pass (fix(ios): apply formatter layout) verified — all 15 lambda guards now have proper Allman braces; InvokeMatchingCustomAction switch has only the correct arms (Expand, Collapse, ScrollIntoView, Realize).
  • Scroll handling (fix(ios): honour UIKit scroll, announcement and trait contracts) — PostScrollPositionAnnouncement correctly satisfies the PageScrolled VoiceOver contract after a successful scroll.

AccessibilityNodeInfoCompatJni.cs (152 lines)

  • JNI version-detection logic for setChecked(int) vs setChecked(bool) — correct; double-checked locking for one-time init; Indeterminate falls back gracefully on old bindings and overrides via native API on Android ≥ 36.
  • Return value semantics (false = indeterminate not fully represented) are consumed correctly by the caller (node.Extras?.PutInt fallback path).

SkiaAccessibilityBase.cs — landmark owner resolution

  • fix(a11y): resolve landmark owner through events source — peer = peer.ResolveProviderPeer(resolveEventsSource: true) is applied before the LandmarkTypeProperty branch; ListItem/TabItem/TreeItem landmark updates now attribute to the correct data peer.

Test file (Given_SkiaAndroidAccessibilityNode.skia.cs, 1 818 lines)

  • All [TestMethod] methods are async Task; no async void.
  • Reflection removed (commit fc6c0c55); direct textBox.IsBackwardSelection access.
  • When_EventsSource_Changes_Then_New_Peer_Gets_New_Id_And_Stale_Id_Is_Rejected — correctly verifies ID isolation after EventsSource reassignment.
  • Text-traversal tests (MoveTextNext / MoveTextPrevious) — verify the fix(a11y): move and expand text ranges by real units behavior at character, word, line, paragraph, and page granularities.

Validation Evidence

  • Code review (by inspection): all previously flagged issues confirmed resolved; no new correctness or style issues found in the reviewed diff.
  • Compile validation: not run locally (CI covers this; UnoFastDevBuild=false on every PR build enforces the Allman-brace analyzer).
  • Runtime validation: not run locally; the new test suite in Given_SkiaAndroidAccessibilityNode.skia.cs is designed to cover these paths end-to-end on a Skia-Android target.

Round 3 of 3 · all blockers resolved · ready to merge

morning4coffe-dev marked this pull request as ready for review August 25, 2026 03:17

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 24s —— View job


I'll analyze this and get back to you.

mergify Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 18s —— View job


I'll analyze this and get back to you.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 19s —— View job


I'll analyze this and get back to you.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 21s —— View job


I'll analyze this and get back to you.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 15s —— View job


I'll analyze this and get back to you.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 28s —— View job


I'll analyze this and get back to you.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 17s —— View job


I'll analyze this and get back to you.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 19s —— View job


I'll analyze this and get back to you.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 18s —— View job


I'll analyze this and get back to you.

morning4coffe-dev marked this pull request as draft August 25, 2026 04:44
morning4coffe-dev marked this pull request as ready for review August 25, 2026 04:44

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 22s —— View job


I'll analyze this and get back to you.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 21s —— View job


I'll analyze this and get back to you.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 26s —— View job


I'll analyze this and get back to you.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 22s —— View job


I'll analyze this and get back to you.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 15s —— View job


I'll analyze this and get back to you.

morning4coffe-dev commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Member Author

review check is red for an action-side reason — requesting a maintainer/action-owner decision

Status: everything else on this PR is green (Check Docs links, Codacy, WinAppSDK Sync Generator Check, XAML Style Check, triage, WIP, Summary), all 7 review threads are resolved, and the base PR #23722 head 4636e911 is fully merged in (0 behind / 30 ahead). Head is unchanged at acdc10b8a; no no-op commits have been pushed and no code was altered to work around this.

Symptom. 18 attempts — replays, a push-triggered run, and two fresh ready_for_review runs — all die identically before the agent reviews anything:

{"type":"result","subtype":"success","is_error":true,"num_turns":2,"duration_ms":15000-30000}
##[error]Claude execution failed: result is_error:true
No buffered inline comments

No gh pr diff, no create_inline_comment, no assistant output — so no finding is being suppressed. Retries were spaced ≥45 min, then hourly (05:34Z, 06:37Z, 08:35Z, 09:29Z, 10:22Z, 11:16Z on 25 Aug), then a single fresh attempt after a ~26 h cooldown: run 32975329111 at 13:38Z on 26 Aug — same failure.

Nothing changed across the cooldown. The workflow is byte-identical on both master and this PR's base branch, still pinned to anthropics/claude-code-action@459ad358 (v1.0.194) with --model claude-sonnet-4-6 and track_progress: true; REVIEW.md is unchanged. Upstream has since published v1.0.203–v1.0.206, none adopted here yet.

Likely cause. With track_progress: true the PR context (including the diff) is assembled up front and turn 1 is spent seeding the tracking comment; the failure is on turn 2. The billing fingerprint is the strongest signal — total_cost_usd per failure is essentially constant across independent attempts and days: 0.9145, 0.9221, 0.9283, 0.9309, 0.9320, 0.9398. Throttling or a rejected request would not bill this consistently; this is the same oversized request being sent, charged, and erroring every time. At cache-write pricing that corresponds to roughly a quarter-million input tokens against claude-sonnet-4-6's 200K context window. This PR is 96 files, +22 595 / −853, the largest input the action handles in this repo.

Consistent with that: the check passed on this exact content in run 32803722005 (verdict SHIP, 12 m 50 s) when the triggering event carried a smaller context; the action succeeds continuously on other PRs; and other large PRs (dev/doti/appium-a11y-validation, dev/doti/port-richeditbox, dev/doti/app-notifications) fail the same way but recover on retry — this one, being the largest, never does.

Two hypotheses were tested and falsified, so they can be ruled out: shortening the PR description had no effect (the workflow header confirms PR title/body are never interpolated into the prompt), and consolidating my four earlier CI notes into this single comment — removing context I had added myself — also had no effect. The diff is the dominant input.

Local validation standing in for the skipped Azure pipeline (analyzer-enabled, -c Release -p:ContinuousIntegrationBuild=true):

Target Result
SamplesApp.Skia.Generic (net10.0) 0 errors, 0 warnings
Uno.UI.Runtime.Skia.Android (net10.0-android) 0 errors, 0 warnings
Uno.UI.Runtime.Skia.AppleUIKit (net9.0-ios18.0) 0 errors, 0 warnings
Skia Desktop runtime tests, whole Windows_UI_Xaml_Automation namespace 233 passed, 0 failed

(Uno.UI - CI / Uno.UI - docs report skipping by design: .vsts-ci.yml's pr: filter excludes the stacked base branch. They run once #23722 merges and this PR is retargeted to master.)

Asking for one of:

  1. Raise the review context budget — a 1M-context model, or diff truncation/chunking in the workflow — so oversized PRs can be reviewed (bumping the pinned action to ≥ v1.0.206 may also help), or
  2. Accept run 32803722005 on this content as satisfying the check.

I have stopped retrying, since 18 identical attempts across two days establish that this will not clear on its own. Happy to re-trigger immediately once either change lands.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 27s —— View job


I'll analyze this and get back to you.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 19s —— View job


I'll analyze this and get back to you.

github-actions Bot commented Aug 25, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 21s —— View job


I'll analyze this and get back to you.

morning4coffe-dev marked this pull request as draft August 26, 2026 13:37
morning4coffe-dev marked this pull request as ready for review August 26, 2026 13:37

github-actions Bot commented Aug 26, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Claude encountered an error after 30s —— View job


I'll analyze this and get back to you.

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/automation Categorizes an issue or PR as relevant to project automation area/build Categorizes an issue or PR as relevant to build infrastructure area/skia ✏️ Categorizes an issue or PR as relevant to Skia kind/documentation platform/android 🤖 Categorizes an issue or PR as relevant to the Android platform platform/ios 🍎 Categorizes an issue or PR as relevant to the iOS platform platform/macos 🍏 Categorizes an issue or PR as relevant to the macOS platform platform/wasm 🌐 Categorizes an issue or PR as relevant to the WebAssembly platform

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant


Back | FazBrowse Home | New Git URL