I'm trying to create an AccessibilityService which can read text from the user screen, it works well and I'm able to grab most of the texts from native Android applications. Still, when I tried it with my example Flutter application, I'm getting all text as null.
This is the current code to read text from my service:
private void listAllTextsInActiveWindow() {
AccessibilityNodeInfo rootNode = getRootInActiveWindow();
if (rootNode != null) {
List<HashMap<String, String>> allTexts = new ArrayList<>();
traverseNodesForText(rootNode, allTexts);
rootNode.recycle();
// Now 'allTexts' contains a list of all texts in the active window
Gson gson = new Gson();
String json = gson.toJson(allTexts);
Log.d("OUTPUT", json);
} else {
Log.d("OUTPUT", "NULL");
}
}
private void traverseNodesForText(AccessibilityNodeInfo node, List<HashMap<String, String>> allTexts) {
if (node == null) return;
if (node.getText() != null && !node.getText().toString().isEmpty()) {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) {
HashMap data = new HashMap();
data.put("view_class", node.getClassName());
data.put("view_text", node.getText());
data.put("view_description", node.getContentDescription());
data.put("view_complete_info", node.toString());
allTexts.add(data);
}
}
for (int i = 0; i < node.getChildCount(); i++) {
AccessibilityNodeInfo childNode = node.getChild(i);
traverseNodesForText(childNode, allTexts);
}
}
I'm getting an empty array as an output.
While I was trying to get it to work, there was an app named 'Automate', and when I tried that application with its Inspect layout in flow, I was able to get a hierarchy XML with all my texts.
Its output was:
<hierarchy rotation="0">
<node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
<node index="0" text="" resource-id="" class="android.widget.LinearLayout" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
<node index="0" text="" resource-id="android:id/content" class="android.widget.FrameLayout" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
<node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
<node index="0" text="" resource-id="" class="android.view.View" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
<node index="0" text="" resource-id="" class="android.view.View" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
<node index="0" text="" resource-id="" class="android.view.View" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
<node index="0" text="" resource-id="" class="android.view.View" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
<node index="0" text="" resource-id="" class="android.view.View" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,154]">
<node index="0" text="" resource-id="" class="android.view.View" package="com.example.apk_installer_demo" content-desc="Download & Install Test APK" checkable="false" checked="false" clickable="false" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[30,75][549,128]"/></node>
<node index="1" text="" resource-id="" class="android.widget.Button" package="com.example.apk_installer_demo" content-desc="Download & Install New Version" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[126,787][594,877]"/></node>
</node>
</node>
</node>
</node>
</node>
</node>
<node index="1" text="" resource-id="android:id/navigationBarBackground" class="android.view.View" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][0,0]"/></node>
</hierarchy>
Now this output is readable, and I'm trying to achieve like this.
Let me know if there's anyway I can do this.
Good call using the uiautomator dump! Did you notice all your node's text is empty? I am not sure how Flutter is handling the text visual text on the buttons, but you need to check your content descriptions (content-desc) in the line
if (node.getText() != null && !node.getText().toString().isEmpty())
otherwise you'll miss it. Try
if (node.getText() != null && (!node.getText().toString().isEmpty() || node.getContentDescription().toString().isEmpty()))
This might not be the end to your problems though, as this will also catch images with alt text, so you might need to make the matching more complex.