androidandroidviewclient

Why is Android View Client returning the view above the view I'm trying to select?


(Edit: the following problem only seems to occur on my Samsung S10e. I just got home and tried Android View Client on a Pixel phone and I'm not having any problem.)

I'm automating the input process for a jetlag calculator android app Entrain. Using the culebra tool, I can find each of the views on the screen. But when I use the ViewClient.touch() method on a view, the program touches the view immediately above it on the screen.

For example, culebra lists these 2 views as the last views on the main menu of the app:

edu_umich_entrain___id_mainTableItem = vc.findViewWithTextOrRaise(u'Update Settings')
edu_umich_entrain___id_mainTableItem = vc.findViewWithTextOrRaise(u'Extras')

So I wrote this bit of code to touch the 'Extras' item:

vc = ViewClient(device, serialno)
vc.findViewWithTextOrRaise(u"Extras", root = 'ROOT').touch()

But it touches the 'Update Settings' view. (The same is true for any other view I touch.)

I've tried finding views with the findViewByIdOrRaise() method but I get the same result. I've tried the culebra GUI tool but it freezes when I click anywhere on the GUI.

I've written similar scripts to automate other android apps (Lutron, IAquaLink, Ankidroid) and they all work fine.

The coordinates for the 2 views are not overlapping, as revealed by getCoords:

print(vc.findViewWithTextOrRaise(u'Update Settings').getCoords())
print(vc.findViewWithTextOrRaise(u'Extras').getCoords())

((195, 1380), (945, 1566))
((195, 1575), (945, 1761))

If I calculate and touch the center of the 'Extras' view:

coords = vc.findViewWithTextOrRaise(u'Extras').getCoords()
x = (coords[1][0] + coords[0][0]) / 2
y = (coords[1][1] + coords[0][1]) / 2
vc.touch(x, y)

it still touches the higher 'Update Settings' view.

If I add 100 to the y axis of the center point, it finally touches the correct view. For now, I'll do that as a workaround.

coords = vc.findViewWithTextOrRaise(u'Extras').getCoords()
x = (coords[1][0] + coords[0][0]) / 2
y = (coords[1][1] + coords[0][1]) / 2 + 100
vc.touch(x, y)

Solution

  • Unfortunately, I don't have a Samsung S10e to test.

    Nonetheless, you may find these tricks useful (I used a Pixel).

    The bounds of the views can be obtained running

    $ dump --bounds
    

    for example

          android.widget.LinearLayout   ((0, 1210), (1080, 1381))
             android.widget.ImageView edu.umich.entrain:id/img  ((0, 1210), (171, 1381))
             android.widget.TextView edu.umich.entrain:id/mainTableItem Update Settings ((171, 1210), (827, 1362))
             android.widget.ImageView edu.umich.entrain:id/img2  ((827, 1210), (958, 1341))
          android.widget.LinearLayout   ((0, 1381), (1080, 1552))
             android.widget.ImageView edu.umich.entrain:id/img  ((0, 1381), (171, 1552))
             android.widget.TextView edu.umich.entrain:id/mainTableItem Extras ((171, 1381), (827, 1533))
             android.widget.ImageView edu.umich.entrain:id/img2  ((827, 1381), (958, 1512))
    

    If you want to actually see every view individually you can use

    $ mkdir imgs
    $ dump --save-view-screenshots=$PWD/imgs/
    

    and when finished, the directory contains a screenshot of each view, for example

    enter image description here

    and

    enter image description here

    this way you can determine if there's an offset or something else in the bounds obtained by AVC/culebra.

    Lastly, if for some reason you have to apply a workaround that depends on your device model or vendor, you can do something like

    ...
    device, serialno = ViewClient.connectToDeviceOrExit(**kwargs1)
    ...
    if device.getProperty('ro.product.vendor.model') == 'Pixel':
       # do something for pixel
       ...
    
    

    Hope this helps.