user-interfaceapplescriptui-automation

Use AppleScript to list the names of all UI elements in a window (GUI scripting)


One line summary - I'm looking for a way to get AppleScript itself to reveal the name it expects a specific piece of window content (UI element) to be referred to as in a "tell" statement.

How do I get AppleScript to list the name it wants me to use to refer to a window's contents?

for example I can say tell current application to tell its front window's list 1 to ...

I'm trying to find out the term like "list 1" for all of the window's contents so I can cross-reference it with the list from Accessibility Inspector..

I tried this code but the first line generates an error saying "error "Can’t make names of «class ects» of window 1 of «class prcs» \"iTunes\" of application \"System Events\" into type string." number -1700 from names of «class ects» of window 1 of «class prcs» "iTunes" to string"

tell application "System Events" to tell process "iTunes" to set elementNames to the names of the entire contents of its front window as string
tell application "TextEdit"
    activate
    make new document at the front
    set the text of the front document to elementNames
    set WrapToWindow to text 2 thru -1 of (localized string "&Wrap to Window")
end tell

Solution

  • In GUI scripting, using entire contents [of] on a process's window in the context of the System Events application returns a list of all the UI elements (GUI controls) of the frontmost window in the active application (the entire hierarchy flattened to a list):

    tell application "System Events"
      tell front window of (first application process whose frontmost is true)
        set uiElems to entire contents
      end tell
    end tell
    

    If you run the above in Script Editor, the Result pane will show a list of object specifiers - e.g., button 1 of window "Main" of application process "Music" of application "System Events" - which do not directly reveal specific information, but from which you can glean least the type (class) of UI element (button, in this example) and the position among its siblings of the same type (1)

    To target Music, for instance, substitute application process "Music" for (first application process whose frontmost is true) .

    Note that if you only want the immediate child elements, you can use UI elements [of] instead of entire contents [of], and you can apply it not just to a window (to get the top-level UI elements), but to any of the UI elements it contains to get their children.


    You cannot extract properties of these elements by converting the enumeration to a string with as string as you've tried, but you can extract properties in a repeat loop:

    Assuming that by names you mean the class names (such as table and button), try this:

    set classNames to {}
    tell application "System Events"
        tell front window of (first application process whose frontmost is true)
            repeat with uiElem in entire contents as list
                set classNames to classNames & (class of uiElem as string)
            end repeat
        end tell
    end tell
    

    Note that as list is inexplicably needed (as of macOS 10.12.3) to make this work (not necessary with UI elements [of]).

    This will return a list of class names such as {"splitter group", "text field", "button", "scroll area", ... }, which in itself is not enough to target a specific element, however, because the hierarchy has been flattened.