NAME

Win32::ActAcc - Active Accessibility client in Perl

SYNOPSIS

Note: We assume you're already familiar with Microsoft's Active Accessibility.

Tools

Explore your Active Accessibility world with Win32::ActAcc's utilities:

 C:\> aadigger    # Explore the window hierarchy (details: perldoc aadigger.pl)
 C:\> aaWhereAmI  # Mouse around and see the path to each accessible Object.
 C:\> aaEvents    # Watch WinEvents through Active Accessibility.

Script essentials

Now let's write a Perl script. You need this stuff at the top:

 use Win32::OLE;
 use Win32::ActAcc qw(:all);
 Win32::OLE->Initialize();
 use Win32::GuiTest; # recommended

Get Accessible Object from Desktop, Point, or HWND

Get the ``root'' Accessible Object:

 $ao = Desktop();
Note
A Perl Win32::ActAcc::AO object contains an IAccessible* and childID.

The object's precise subclass of AO reflects its role (Window, Client, Pushbutton, etc.).

Other ways to get an Accessible Object:

 $ao = AccessibleObjectFromPoint($x, $y); # pixels from upper left
 $ao = AccessibleObjectFromWindow($hwnd);

You can also get an Accessible Object from an event, such as when an application opens..

Invoke an app and watch for its window to appear

Here's how to invoke an app with system and latch onto its main window by listening to events.

 # Install event hook with first call to "clearEvents":
 clearEvents();
 
 # Give event hook time to permeate your computer:
 # ...
 
 # Start Notepad, but first discard the event backlog:
 clearEvents();
 system("start notepad.exe");
 
 # Wait for Notepad to appear, as signaled by
 # an EVENT_OBJECT_SHOW event associated with an
 # Accessible Object whose name matches qr/Notepad$/,
 # and make a note of that useful Accessible Object.
 my $aoNotepad = waitForEvent
   (
    +{ 
      'event'=>EVENT_OBJECT_SHOW(),
      'ao_profile'=>qr/Notepad$/ # a 'window test': see below
     }, 
    # options:
    +{
      'timeout'=>30, # seconds
                     # (an hourglass buys a few more seconds)
      'trace'=>1 # display events as troubleshooting aid
      # Tip: Don't turn off 'trace' until your script works!
     }
   );
 # The sentinel event might not be the last in the flurry of events.
 # Wait for steady state before proceeding.
 awaitCalm();

For jobs too elaborate for waitForEvent, see Events below.

Accessible Object properties

Having found an Accessible Object, examine it:

 my $hwnd = $ao->WindowFromAccessibleObject();
 my $roleNumber = $ao->get_accRole();
 my $roleText = GetRoleText( $roleNumber );
 my $stateBits = $ao->get_accState();
 my $name = $ao->get_accName();
 my $value = $ao->get_accValue();
 my $description = $ao->get_accDescription();
 my $default_action = $ao->get_accDefaultAction();
 my $help = $ao->get_accHelp();
 my $f = $ao->get_accFocus();
 my ($left, $top, $width, $height) = $ao->accLocation();
 my $ks = $ao->get_accKeyboardShortcut();
 my $id = $ao->get_itemID();
 my $bp = $ao->get_nativeOM();
 my @selections = $ao->get_accSelection();

visible considers the STATE_SYSTEM_INVISIBLE bit from get_accState, among other factors - see visible.

 my $might_be_visible = $ao->visible();

Troubleshoot your script by printing out the Accessible Objects.

 # Display class, name, state, location, ID, HWND, default action:
 print "badger/limpid: " . $ao->describe() . "\n";
 print "pilfer/bugle:  $ao\n"; # same thing
 
 # display summary of $ao and all its descendants
 $ao->debug_tree();

GUI Manipulation

Active Accessibility alone is feeble, so be sure to see also Using Active Accessibility with Win32::GuiTest.

 # Selection and focus
 $ao->accSelect(SELFLAG_TAKEFOCUS());
 
 # doable action at this moment
 my $action = $ao->get_accDefaultAction();
 $ao->accDoDefaultAction();

If accDoDefaultAction will do, then perhaps there is a particular action that the script would like to assert is default before executing it.

 # Perl shortcut: Do named action iff it's the default -- otherwise die.
 $ao->doActionIfDefault('Press'); # do-or-die
 
 # Shorthand for the shortcut (for English-language Windows):
 $ao->dda_Check();
 $ao->dda_Click();
 $ao->dda_Close();
 $ao->dda_Collapse();
 $ao->dda_DoubleClick();
 $ao->dda_Execute();
 $ao->dda_Expand();
 $ao->dda_Press();
 $ao->dda_Switch();
 $ao->dda_Uncheck();

AO can simulate a click using the Windows API.

 # Simulate click at center of an Accessible Object:
 $ao->click(); # there's also $ao->rightclick()

Parents and Children

Find an Accessible Object's relatives:

 my $p = $ao->get_accParent(); # query Active Accessibility
 $p = $ao->parent(); # prefer cached weak-ref from iterator, if present
 
 # Get child-count, then one child at a time:
 my $kk = $ao->get_accChildCount();
 my $ak = $ao->get_accChild(0); # etc.
 
 # Get children in a list:
 my @likely_visible_children = $ao->AccessibleChildren();
 my @all = $ao->AccessibleChildren(0,0); # state-bits to compare, bit values
 
 # Navigate turtle-style:
 my $np1 = $ao->accNavigate(NAVDIR_FIRSTCHILD()); # etc. etc.
Note
Win32::ActAcc's AccessibleChildren, with no arguments, screens out `invisible' and `offscreen' results by assuming the default arguments (STATE_SYSTEM_INVISIBLE()|STATE_SYSTEM_OFFSCREEN(), 0).

Buggy apps may respond inconsistently to one or another technique of enumerating children. Unfortunately, you must program the script differently for each technique, so experimenting with more than one is tedious.

So why not use an Iterator instead?

Iterators

Here's how to visit an Accessible Object's children using an iterator:

 my $iter = $ao->iterator();
 $iter->open();
 while ( my $aoi = $iter->nextAO() )
   {
     print "$aoi\n";
   }
 $iter->close();

Accessible Objects from iterators keep a weak reference to the ``parent'' that enumerated them, and can infer some state information from the parent's state.

 my $p = $ao->iparent(); # parent as noted by iterator...
 $p = $ao->parent();     # ... or get_accParent() if iparent=undef
 
 # get state bits, including states implicit from parent
 # (readonly, offscreen, invisible, unavailable):
 my $allstate = $ao->istate();

The iterator for most windows uses a slow, but thorough, combination of AccessibleChildren and accNavigate. The iterator for menus and outlines can click through them and treat sub-items like children. You can select the best child-enumeration technique for each occasion. See details at More on Iterators below.

Win32::ActAcc's power tools -- dig, tree, menuPick -- use iterators so as not to couple their mission to any specific child-enumeration technique.

Tree of Accessible Objects

Use tree to traverse a hierarchy of Accessible Objects depth-first, calling a code-ref once for each Accessible Object (including the starting object). The code can control iteration using level, prune, stop and pin (see sample).

 $ao->tree
   (
    sub
    {
      my ($ao, $monkey) = @_;
      # $monkey->level() returns count of levels from root.
      # $monkey->prune() skips this AO's children.
      # $monkey->stop() visits NO more Accessible Objects.
      # $monkey->pin() prevents closing any menus and outlines
      #                that tree() opened
      #                (applies only if flag 'active'=>1)
      print ' 'x($monkey->level())."$ao\n";
    },
    #+{ 'active'=>1 } # optional iterator flags-hash
   );

When tree obtains an iterator for each Accessible Object it visits, tree passes its second argument (an optional hash) to the iterator's constructor. (See More on Iterators.)

Referring to an object by name or role

Supposing $ao is a client area containing a Close button, here's how to find and press Close:

 $aoBtnClose = $ao->dig
   ( 
    +[
      "{push button}Close" # {role}name
     ] 
   ); 
 $aoBtnClose->dda_Press();

If $ao is a window, containing a client area, containing a Close button, just set forth both steps in the path to reach Close from $ao:

 $aoBtnClose = $ao->dig
   ( 
    +[
      "{client}",  # {role} only, name doesn't matter
      "{push button}Close"
     ] 
   );

In a word, dig follows a path of Window Tests, and returns what it finds. See details at Finding Accessible Objects using 'dig'.

You can run aadigger or aaWhereAmI interactively to reconnoiter and figure out a path to the interesting Accessible Object.

Menus

menuPick uses Active Accessibility and Win32::GuiTest to manipulate standard Windows menu-bars and context-menus. Your mileage may vary with apps that use custom menus with cockeyed support for Active Accessibility.

 # menuPick takes a ref to a list of window-tests,
 # tracing a path from menubar to menu to submenu etc.,
 # plus an optional flags hash.
 $ao->menuPick(+[ 'Edit', qr/^Undo/ ], +{'trace'=>1} );

menuPick can summon and operate a right-clicky context menu:

 $ao->context_menu()->menuPick(+[ 'Paste' ]);

If Win32::GuiTest has been loaded (as by use Win32::GuiTest;), the active menu iterator closes menus when it's done with them.

Some menus contain items marked as invisible. Use the HASH form of the window-test to pick such an invisible item; the string and regex window-tests match only visible items.

Using Active Accessibility with Win32::GuiTest

Get an HWND or location from an Accessible Object and manipulate it with the Windows API:

 use Win32::GuiTest;
 
 # use a HWND
 my $hwnd = $ao->WindowFromAccessibleObject();
 my $name = Win32::GuiTest::GetWindowText($hwnd);
 
 # use an (x,y) location
 my ($left, $top, $width, $height) = $ao->accLocation();
 Win32::GuiTest::MouseMoveAbsPix($left, $top);

DETAILS

Window Tests

A window-test examines an Accessible Object and returns a true or false value -- like Perl's file tests (-e, -f, etc.). Window-tests are used in waitForEvent, dig, menuPick, and match.

A window-test can take the form of a string, a regex, or a hash.

String
The string must completely match the object's name, {role}, or {role}name. Matches visible objects only. You can't use the string form of window-test if you need to include a literal brace in the name. For the role, use whatever notation is convenient: 
 window                               # role text
 ROLE_SYSTEM_WINDOW                   # constant name
 WINDOW                               # last leg of constant name
 Win32::ActAcc::Window                # Perl package for the role
 Window                               # last leg of package name
 value of ROLE_SYSTEM_WINDOW()        # the role number

Regular expression (qr/blabla/)
The regex matches the object's name, as in $name=~qr/regex/. Matches visible objects only.

Hash
Specifying a window-test as a hash is the most flexible way, as it can test not only the name and role, but also the state and other attributes of the Accessible Object, and even run a custom code-ref.

Hash members (all are optional; all present members must match the Accessible Object):

get_accRole hash member
role number

get_accName hash member
string (match entire) or regex

get_accValue hash member
string (match entire) or regex

get_accDescription hash member
string (match entire) or regex

get_accHelp hash member
string (match entire) or regex

get_accDefaultAction hash member
string (match entire) or regex

WindowFromAccessibleObject hash member
match an HWND number

visible hash member
a true value to match only visible objects. (Use a false value to match only invisible objects. Omit the 'visible' key if you don't care whether the object is visible.)

state_has and state_lacks hash members
or'd state bits

role_in or role_not_in hash member
LIST of roles (each item in the list uses any ``{role}'' notation (above), but without the braces)

code hash member
a code-ref to call if the other hash keys match. Return a true value to indicate a match.

Sample window-tests:

 $b = $ao->match('Close'); # Is AO's name exactly Close?
 $b = $ao->match( +{'get_accName'=>'Close'} ); # ... using a hash.
 
 $b = $ao->match(qr/Close/); # Does AO's name match that regexp?
 $b = $ao->match( +{'get_accName'=>qr/Close/} ); # ... using a hash.
 
 $b = $ao->match('{ROLE_SYSTEM_PUSHBUTTON}Close'); # Is AO a pushbutton named Close?
 $b = $ao->match('{push button}Close'); # ... using localized 'role text'
 $b = $ao->match('{Pushbutton}Close'); # ... using ActAcc package name
 $b = $ao->match( +{'get_accRole'=>ROLE_SYSTEM_PUSHBUTTON(), 'name'=>'Close'} ); # ...
 $b = $ao->match( +{'rolename'=>'Pushbutton', 'get_accName'=>'Close'} ); # ...
 
 $b = $ao->match
   ( 
    +{'code'=>
      sub
      { 
        my $ao = shift; 
        return $ao->match( qr/Bankruptcy in progress/ );
      } 
     } 
   ); 
 
 $b = $ao->match( +{'visible'=>1} );

There is more to the 'visible'=>1 test than meets the eye..

visible

 my $might_be_visible = $ao->visible();
 my $same_thing       = $ao->match( +{'visible'=>1} );

The visible function returns a true value if none of these reasons-for-being-invisible applies.

The algorithm overlooks other reasons-for-being-invisible, such as occlusion and inconspicuousness.

Finding Accessible Objects using 'dig'

dig follows a path of Window Tests, and returns what it finds.

Depending on its scalar or list context, and min and max options, dig can perform various searches:

Find all matching Accessible Objects (die if none)
Use dig in array context without specifying options min or max.

Find all matching Accessible Objects (if any)
Use dig in array context, specifying option min=0.

Find first matching Accessible Object (die if none)
Use dig in scalar context, specifying neither option min nor max, or specifying them both as 1.

Find first matching Accessible Object (if any)
Use dig in scalar context, specifying min=0. If it finds no matching Accessible Object, dig returns undef.

The optional second parameter is a hash of options:

``min'' option
Find this many objects, or die.

``max'' option
Stop looking after finding this many objects.

``trace'' option
If true, display the objects being examined.

``active'', ``nav'', and ``perfunctory'' options
dig passes these along when it obtains an iterator for each Accessible Object it traverses. dig sets the ``active'' flag unless the options hash specifies it as a non-true value.

Samples using dig:

 # Find one immediate child of $ao with role "client"; die if not found.
 my $aoClient = $ao->dig( +[ '{client}' ] );
 
 # Find one untitled Notepad within the Desktop's client area; die if not found.
 my $someNewNotepad = Desktop()->
   dig(+[ 
         '{client}',                     # step 1
         '{window}Untitled - Notepad'    # step 2
        ]);
 
 # Get results into a list: find *all* untitled Notepad
 # windows in the Desktop's client area. Die if none found.
 my @allNewNotepads1 = 
   Desktop()->
     dig(+[ 
           '{client}',                   # step 12
           '{window}Untitled - Notepad'  # step 2
          ]);
 
 # Find all untitled Notepads, using a regex to match their name.
 my @allNewNotepads2 = 
   Desktop()->
     dig(+[ '{client}',                  # step 1
            +{                           # step 2:
              'get_accRole'=>ROLE_SYSTEM_WINDOW(),
              'get_accName'=>qr/^Untitled - Notepad$/
             }
          ]);
 
 # Find all untitled Notepads that contain an Application menubar.
 my @allNewNotepads3 = 
   Desktop()->
     dig(+[ '{client}',                  # step 1
            +{                           # step 2:
              'get_accRole'=>ROLE_SYSTEM_WINDOW(),
              'get_accName'=>qr/^Untitled - Notepad$/
             },
            +{                           # step 3:
              'get_accRole'=>ROLE_SYSTEM_MENUBAR(),
              'get_accName'=>'Application'
             },
            +{                           # step 4: back up!
              'axis'=>'parent'
             },
          ]);
 
 # Find windows on desktop. Die if fewer than 2. Return at most 42.
 my @upTo42Windows =
   Desktop()->dig( +[
                     '{client}',         # step 1
                     '{window}'          # step 2
                    ], 
                   +{                    # options
                     'min'=>2,           #  -die unless at least 2
                     'max'=>42,          #  -shortcut after 42
 
                     'trace'=>1          #  -for troubleshooting
                    } 
                 );

The active, nav, and perfunctory options configure the iterator with which dig enumerates each Accessible Object's children in its quest for potential matches..

More on Iterators

The default iterator uses both AccessibleChildren and accNavigate, which is slow but works with many applications.

 my $iter = $ao->iterator();

Optional hints convey a preference for an iterator type, if it applies to the Accessible Object:

 # operate menus and outlines, treating consequences as children
 my $iter = $ao->iterator( +{ 'active'=>1 } );
 
 # use AccessibleChildren
 my $iter = $ao->iterator( +{ 'perfunctory'=>1 } );
 
 # use accNavigate
 my $iter = $ao->iterator( +{ 'nav'=>1 } );

For completeness, there is an iterator that uses get_accChildCount and get_accChild:

 my $iter = new Win32::ActAcc::get_accChildIterator($ao);

Events

Win32::ActAcc installs a system-wide in-process event hook upon the first call to clearEvents. Thereafter, events stampede through a circular buffer. You can watch by running aaEvents.

All Perl processes share one event hook and one circular buffer, but each Perl process keeps its own independent pointer into the buffer.

getEvent retrieves one event from the circular buffer and advances the pointer:

 # Retrieve one event from circular buffer (if any there be).
 my $anEvent = getEvent();
 if (defined($anEvent))
   {
     print "Event: $anEvent\n";
   }

Scripts may getEvent in a loop to watch for a specific sentinel event. Such a loop is included: waitForEvent consumes events until one satisfies a hash-of-criteria (sample in the Synopsis) or a code-ref:

 waitForEvent
   (
    sub
    {
      my $e = shift;
      if ($$e{'event'} == EVENT_SYSTEM_FOREGROUND())
        {
          my $ao = $e->AccessibleObjectFromEvent(); # or getAO() for short
          my $name = $ao->get_accName();
          return $ao if ($name =~ qr/Notepad$/);
        }
      return undef;
    },
    # options:
    +{
      'timeout'=>30, # seconds
                     # (an hourglass buys a few more seconds)
      'trace'=>1 # display events as troubleshooting aid
      # Tip: Don't turn off 'trace' until your script works!
     }
   )
     or die("Notepad didn't come to foreground in the allotted time.");

To prevent a stale event from triggering the exit condition, call clearEvents before taking the action whose consequences the script will be looping in wait for.

SAMPLE

``eg\playpen.pl'' demonstrates using Active Accessibility to inspect and manipulate a menu, a popup menu, a text-entry blank, a checkbox, a radio button, a spin button, tabs, a list box, a tree-list, a two-column list view, and suchlike.

Of course, playpen.pl depends on an application that presents such widgets. The applets that come with Windows change too often, so playpen.pl uses a simple C# app whose source code is in eg\playpen.

playpen.pl also depends on Win32::GuiTest.

Build the playpen C# app, then invoke playpen.pl to explore it:

 > vcvars32 || rem ember to put 'csc' on the path
 > cd eg\playpen
 > build.cmd
 > cd ..
 > perl playpen.pl

TROUBLESHOOTING

If an Active Accessibility function unexpectedly returns undef, check Perl's Extended OS Error special variable $^E for clues.

Run your script with ``perl -w''.

If your script doesn't work, see whether the aadigger sample works.

If you see Windows error 0x800401f0 (``CoInitialize has not been called''), make sure your script starts off with Win32::OLE->Initialize().

If you get a Windows error number and want to know what it means, try using Win32::FormatMessage.

If your script sometimes misses noticing an event that occurs very soon after your script calls clearEvents() for the first time, insert a sleep after that first clearEvents(). Installing a WinEvent handler seems to take effect ``soon'', but not synchronously.

If you are desperate enough to insert ``print'' statements:

 print "The Accessible Object is: $ao\n"; # shows several attributes
 print "same as: ". $ao->describe() . "\n";
 $ao->debug_tree(); # display $ao and all its descendants

If you are struggling to find the right event or window-test for use with waitForEvent, dig, tree, or menuPick, try using the trace flag to evoke a lot of progress messages. Or, embed the interactive aadigger feature into your script:

 # invoke interactive explorer feature starting at $ao:
 use Win32::ActAcc::aaExplorer;
 Win32::ActAcc::aaExplorer::aaExplore($ao);

If menuPick doesn't work because your computer is too slow, increase the value of $Win32::ActAcc::MENU_SLOWNESS. menuPick relies on a hover delay to give the app a chance to update a menu-item object's default action.

If your script displays gibberish instead of Unicode text on the console, try writing to a file instead.

BUGS

It doesn't implement get_accHelpTopic and accHitTest.

menuPick doesn't know how to choose commands that hid because you seldom use them.

You can't use a Win32::ActAcc ``Accessible Object'' with Win32::OLE.

It probably doesn't work multi-threaded.

Apps with a buggy IAccessible implementation may cause the Perl process to crash.

COPYRIGHT

Copyright 2000-2004, Phill Wolf.

pbwolf@cpan.org