FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat: fix issue where overlay does not allow clicking on elements cov… · hackathonprojs/AndroidUsePdd@8cca638 · GitHub

/ AndroidUsePdd Public
forked from swswsw/AndroidUse

Commit 8cca638

Browse files
swswsw
committed
feat: fix issue where overlay does not allow clicking on elements covered by overlay. implement gesture passthrough
- Transform agent interaction into a persistent conversation flow with session memory. - Add floating, draggable overlay bar with status indicator and STOP (■) button. - Implement expandable chat overlay window for real-time instructions and AI feedback. - Add "Clear Chat" (🗑️) functionality to reset AI context and history. - Optimize AI "vision" using Android 14 `takeScreenshotOfWindow` to exclude service overlays. - Implement "Ghosting" system (temporary FLAG_NOT_TOUCHABLE) to ensure AI clicks pass through overlays. - Update Gemini, OpenAI, and Anthropic agents to process multi-turn conversation history. - Document overlay interference challenges and solutions in docs/add-chat.md.
1 parent dfbdfe0 commit 8cca638

2 files changed

Lines changed: 130 additions & 15 deletions

File tree

‎app/src/main/java/org/goldenpass/androiduse/UIAgentAccessibilityService.kt‎

Lines changed: 84 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -151,7 +151,13 @@ class UIAgentAccessibilityService : AccessibilityService() {
151151
updateOverlay("Capturing screen...", currentStepCount)
152152
Log.d("UIAgentAccessibilityService", "Capturing screen for step $currentStepCount...")
153153

154-
val targetWindowId = rootInActiveWindow?.windowId ?: -1
154+
// Find the target application window (skip our own overlays)
155+
val windows = windows
156+
val targetWindow = windows.find {
157+
it.type == AccessibilityWindowInfo.TYPE_APPLICATION && it.isActive
158+
} ?: windows.find { it.type == AccessibilityWindowInfo.TYPE_APPLICATION }
159+
160+
val targetWindowId = targetWindow?.id ?: rootInActiveWindow?.windowId ?: -1
155161

156162
captureScreenshot(mainExecutor, targetWindowId) { bitmap ->
157163
if (bitmap == null) {
@@ -373,11 +379,52 @@ class UIAgentAccessibilityService : AccessibilityService() {
373379
}
374380

375381
fun performClickAt(x: Float, y: Float) {
376-
val clickPath = Path()
377-
clickPath.moveTo(x, y)
378-
val gestureBuilder = GestureDescription.Builder()
379-
gestureBuilder.addStroke(GestureDescription.StrokeDescription(clickPath, 0, 100))
380-
dispatchGesture(gestureBuilder.build(), null, null)
382+
serviceScope.launch {
383+
Log.d("UIAgentAccessibilityService", "Performing click at ($x, $y) - Ghosting overlays")
384+
setOverlaysTouchable(false)
385+
delay(100) // Wait for WindowManager to update flags
386+
387+
val clickPath = Path()
388+
clickPath.moveTo(x, y)
389+
val gestureBuilder = GestureDescription.Builder()
390+
gestureBuilder.addStroke(GestureDescription.StrokeDescription(clickPath, 0, 100))
391+
392+
dispatchGesture(gestureBuilder.build(), object : GestureResultCallback() {
393+
override fun onCompleted(gestureDescription: GestureDescription?) {
394+
super.onCompleted(gestureDescription)
395+
Log.d("UIAgentAccessibilityService", "Gesture completed")
396+
setOverlaysTouchable(true)
397+
}
398+
override fun onCancelled(gestureDescription: GestureDescription?) {
399+
super.onCancelled(gestureDescription)
400+
Log.w("UIAgentAccessibilityService", "Gesture cancelled")
401+
setOverlaysTouchable(true)
402+
}
403+
}, null)
404+
}
405+
}
406+
407+
private fun setOverlaysTouchable(touchable: Boolean) {
408+
Handler(Looper.getMainLooper()).post {
409+
overlayView?.let { view ->
410+
val params = view.layoutParams as WindowManager.LayoutParams
411+
if (touchable) {
412+
params.flags = params.flags and WindowManager.LayoutParams.FLAG_NOT_TOUCHABLE.inv()
413+
} else {
414+
params.flags = params.flags or WindowManager.LayoutParams.FLAG_NOT_TOUCHABLE
415+
}
416+
windowManager.updateViewLayout(view, params)
417+
}
418+
chatOverlayView?.let { view ->
419+
val params = view.layoutParams as WindowManager.LayoutParams
420+
if (touchable) {
421+
params.flags = params.flags and WindowManager.LayoutParams.FLAG_NOT_TOUCHABLE.inv()
422+
} else {
423+
params.flags = params.flags or WindowManager.LayoutParams.FLAG_NOT_TOUCHABLE
424+
}
425+
windowManager.updateViewLayout(view, params)
426+
}
427+
}
381428
}
382429

383430
private fun stopWithNotification(message: String) {
@@ -536,7 +583,7 @@ class UIAgentAccessibilityService : AccessibilityService() {
536583
val root = LinearLayout(this).apply {
537584
orientation = LinearLayout.VERTICAL
538585
background = GradientDrawable().apply {
539-
setColor(Color.parseColor("#EE111111"))
586+
setColor(Color.parseColor("#99111111")) // Further reduced opacity to 60% just in case
540587
cornerRadius = 24f
541588
}
542589
setPadding(20, 20, 20, 20)
@@ -673,17 +720,39 @@ class UIAgentAccessibilityService : AccessibilityService() {
673720
}
674721

675722
fun performSwipe(startX: Float, startY: Float, endX: Float, endY: Float, duration: Long = 500L) {
676-
val swipePath = Path()
677-
swipePath.moveTo(startX, startY)
678-
swipePath.lineTo(endX, endY)
679-
val gestureBuilder = GestureDescription.Builder()
680-
gestureBuilder.addStroke(GestureDescription.StrokeDescription(swipePath, 0, duration))
681-
dispatchGesture(gestureBuilder.build(), null, null)
723+
serviceScope.launch {
724+
Log.d("UIAgentAccessibilityService", "Performing swipe from ($startX, $startY) to ($endX, $endY)")
725+
setOverlaysTouchable(false)
726+
delay(100)
727+
728+
val swipePath = Path()
729+
swipePath.moveTo(startX, startY)
730+
swipePath.lineTo(endX, endY)
731+
val gestureBuilder = GestureDescription.Builder()
732+
gestureBuilder.addStroke(GestureDescription.StrokeDescription(swipePath, 0, duration))
733+
734+
dispatchGesture(gestureBuilder.build(), object : GestureResultCallback() {
735+
override fun onCompleted(gestureDescription: GestureDescription?) {
736+
super.onCompleted(gestureDescription)
737+
setOverlaysTouchable(true)
738+
}
739+
override fun onCancelled(gestureDescription: GestureDescription?) {
740+
super.onCancelled(gestureDescription)
741+
setOverlaysTouchable(true)
742+
}
743+
}, null)
744+
}
682745
}
683746

684747
fun getClickableElementsJson(): String {
685-
val rootNode = rootInActiveWindow ?: return "[]"
686-
// Ensure we are only traversing the target app's window
748+
// Look for the application window specifically
749+
val windows = windows
750+
val targetWindow = windows.find {
751+
it.type == AccessibilityWindowInfo.TYPE_APPLICATION && it.isActive
752+
} ?: windows.find { it.type == AccessibilityWindowInfo.TYPE_APPLICATION }
753+
754+
val rootNode = targetWindow?.root ?: rootInActiveWindow ?: return "[]"
755+
687756
if (rootNode.packageName == packageName) {
688757
Log.w("UIAgentAccessibilityService", "Root node belongs to our service, skipping UI tree")
689758
return "[]"

‎docs/add-chat.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,49 @@
1+
## Chat Overlay & Interactivity Documentation
2+
3+
This document explains the implementation of the conversational chat overlay and how we solved the technical challenge of overlay interference with AI actions.
4+
5+
### 1. The Problem: Overlay Interference
6+
7+
When a persistent chat window is added as a floating overlay (`TYPE_ACCESSIBILITY_OVERLAY`), it introduces two major issues for an AI-driven automation agent:
8+
9+
#### A. Visual Confusion (Screenshot Pollution)
10+
If the agent takes a standard screenshot of the display, the chat window will appear in the image. The AI might then try to interact with the chat window itself (e.g., clicking the "Send" button) instead of the target application it is supposed to automate.
11+
12+
#### B. Touch Interception (Click Blockage)
13+
Android overlays are physical layers. If the AI decides to click a button in the target app that is currently hidden behind the chat window, the click will be intercepted by the chat overlay. The target app will never receive the event.
14+
15+
---
16+
17+
### 2. The Solution: "Ghosting" and Focused Vision
18+
19+
We solved these issues using a combination of Android 14 window APIs and dynamic flag management.
20+
21+
#### A. Window-Specific Capture (AI Vision)
22+
Instead of capturing the full display, the service now performs the following steps for every "thought" cycle:
23+
1. **Iterate Windows**: It uses `AccessibilityService.getWindows()` to find the window with `TYPE_APPLICATION`.
24+
2. **Target ID**: It retrieves the specific `windowId` of that app.
25+
3. **Filtered Screenshot**: It uses `takeScreenshotOfWindow(windowId, ...)` (introduced in API 34).
26+
4. **Result**: The AI receives a clean screenshot of *only* the app it is automating. The chat window is completely invisible to the AI's "vision."
27+
28+
#### B. The "Ghosting" System (Interactivity)
29+
To ensure AI clicks hit the target app even when covered by the chat, we implemented a **Ghosting Mode** during gesture injection:
30+
31+
1. **Flag Update**: Immediately before a click or swipe, the service applies the `FLAG_NOT_TOUCHABLE` flag to both the status bar and the chat window.
32+
2. **Synchronization Delay**: A small 100ms delay is introduced to ensure the `WindowManager` has applied the new flags.
33+
3. **Gesture Injection**: The `dispatchGesture()` API is called to perform the action.
34+
4. **Callback Restoration**: We use a `GestureResultCallback`. Only once Android confirms the gesture is finished (or cancelled) do we remove the `FLAG_NOT_TOUCHABLE` flag.
35+
5. **Opacity Tuning**: The chat window background is set to **60% opacity** (`#99111111`) to ensure compliance with Android 12+ "Untrusted Touch" security policies, preventing the system from ever blocking an AI-initiated gesture.
36+
37+
---
38+
39+
### 3. Technical Summary
40+
- **Primary API**: `takeScreenshotOfWindow` (API 34+)
41+
- **Fallback**: Standard `takeScreenshot` if window ID is unavailable.
42+
- **Gesture Reliability**: Verified via `GestureResultCallback` and temporary flag toggling.
43+
- **Visibility**: Overlays remain visible to the user at all times but are "ghosted" for the milliseconds during which an automated action occurs.
44+
45+
---
46+
147
# Implementation Plan - Chat Overlay & Conversation History
248

349
This plan transforms the current "one-off task" execution into a persistent conversation between the user and the AI agent, accessible via a floating chat window.

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL