Win32::ActAcc - Active Accessibility client in Perl
Note: We assume you're already familiar with Microsoft's Active Accessibility.
Explore your Active Accessibility world with Win32::ActAcc's utilities:
C:\> aadigger # Explore the window hierarchy (details: perldoc aadigger.pl) C:\> aaWhereAmI # Mouse around and see the path to each accessible Object. C:\> aaEvents # Watch WinEvents through Active Accessibility.
Now let's write a Perl script. You need this stuff at the top:
use Win32::OLE; use Win32::ActAcc qw(:all); Win32::OLE->Initialize(); use Win32::GuiTest; # recommended
Get the ``root'' Accessible Object:
$ao = Desktop();
Win32::ActAcc::AO
object contains an IAccessible*
and
childID
.
The object's precise subclass of AO reflects its role (Window, Client, Pushbutton, etc.).
Other ways to get an Accessible Object:
$ao = AccessibleObjectFromPoint($x, $y); # pixels from upper left $ao = AccessibleObjectFromWindow($hwnd);
You can also get an Accessible Object from an event, such as when an application opens..
Here's how to invoke an app with system
and latch onto its main
window by listening to events.
# Install event hook with first call to "clearEvents": clearEvents(); # Give event hook time to permeate your computer: # ... # Start Notepad, but first discard the event backlog: clearEvents(); system("start notepad.exe"); # Wait for Notepad to appear, as signaled by # an EVENT_OBJECT_SHOW event associated with an # Accessible Object whose name matches qr/Notepad$/, # and make a note of that useful Accessible Object. my $aoNotepad = waitForEvent ( +{ 'event'=>EVENT_OBJECT_SHOW(), 'ao_profile'=>qr/Notepad$/ # a 'window test': see below }, # options: +{ 'timeout'=>30, # seconds # (an hourglass buys a few more seconds) 'trace'=>1 # display events as troubleshooting aid # Tip: Don't turn off 'trace' until your script works! } ); # The sentinel event might not be the last in the flurry of events. # Wait for steady state before proceeding. awaitCalm();
For jobs too elaborate for waitForEvent
, see Events below.
Having found an Accessible Object, examine it:
my $hwnd = $ao->WindowFromAccessibleObject(); my $roleNumber = $ao->get_accRole(); my $roleText = GetRoleText( $roleNumber ); my $stateBits = $ao->get_accState(); my $name = $ao->get_accName(); my $value = $ao->get_accValue(); my $description = $ao->get_accDescription(); my $default_action = $ao->get_accDefaultAction(); my $help = $ao->get_accHelp(); my $f = $ao->get_accFocus(); my ($left, $top, $width, $height) = $ao->accLocation(); my $ks = $ao->get_accKeyboardShortcut(); my $id = $ao->get_itemID(); my $bp = $ao->get_nativeOM(); my @selections = $ao->get_accSelection();
visible
considers the STATE_SYSTEM_INVISIBLE
bit from
get_accState
, among other factors - see visible.
my $might_be_visible = $ao->visible();
Troubleshoot your script by printing out the Accessible Objects.
# Display class, name, state, location, ID, HWND, default action: print "badger/limpid: " . $ao->describe() . "\n"; print "pilfer/bugle: $ao\n"; # same thing # display summary of $ao and all its descendants $ao->debug_tree();
Active Accessibility alone is feeble, so be sure to see also Using Active Accessibility with Win32::GuiTest.
# Selection and focus $ao->accSelect(SELFLAG_TAKEFOCUS()); # doable action at this moment my $action = $ao->get_accDefaultAction(); $ao->accDoDefaultAction();
If accDoDefaultAction
will do, then perhaps there is a
particular action that the script would like to assert is default
before executing it.
# Perl shortcut: Do named action iff it's the default -- otherwise die. $ao->doActionIfDefault('Press'); # do-or-die # Shorthand for the shortcut (for English-language Windows): $ao->dda_Check(); $ao->dda_Click(); $ao->dda_Close(); $ao->dda_Collapse(); $ao->dda_DoubleClick(); $ao->dda_Execute(); $ao->dda_Expand(); $ao->dda_Press(); $ao->dda_Switch(); $ao->dda_Uncheck();
AO can simulate a click using the Windows API.
# Simulate click at center of an Accessible Object: $ao->click(); # there's also $ao->rightclick()
Find an Accessible Object's relatives:
my $p = $ao->get_accParent(); # query Active Accessibility $p = $ao->parent(); # prefer cached weak-ref from iterator, if present # Get child-count, then one child at a time: my $kk = $ao->get_accChildCount(); my $ak = $ao->get_accChild(0); # etc. # Get children in a list: my @likely_visible_children = $ao->AccessibleChildren(); my @all = $ao->AccessibleChildren(0,0); # state-bits to compare, bit values # Navigate turtle-style: my $np1 = $ao->accNavigate(NAVDIR_FIRSTCHILD()); # etc. etc.
AccessibleChildren
, with no arguments, screens out
`invisible' and `offscreen' results by assuming the default arguments
(STATE_SYSTEM_INVISIBLE()|STATE_SYSTEM_OFFSCREEN(), 0
).
Buggy apps may respond inconsistently to one or another technique of enumerating children. Unfortunately, you must program the script differently for each technique, so experimenting with more than one is tedious.
So why not use an Iterator instead?
Here's how to visit an Accessible Object's children using an iterator:
my $iter = $ao->iterator(); $iter->open(); while ( my $aoi = $iter->nextAO() ) { print "$aoi\n"; } $iter->close();
Accessible Objects from iterators keep a weak reference to the ``parent'' that enumerated them, and can infer some state information from the parent's state.
my $p = $ao->iparent(); # parent as noted by iterator... $p = $ao->parent(); # ... or get_accParent() if iparent=undef # get state bits, including states implicit from parent # (readonly, offscreen, invisible, unavailable): my $allstate = $ao->istate();
The iterator for most windows uses a slow, but thorough, combination of
AccessibleChildren
and accNavigate
. The iterator for menus and
outlines can click through them and treat sub-items like children.
You can select the best child-enumeration technique for each occasion.
See details at More on Iterators below.
Win32::ActAcc's power tools -- dig
, tree
, menuPick
-- use
iterators so as not to couple their mission to any specific
child-enumeration technique.
Use tree
to traverse a hierarchy of Accessible Objects
depth-first, calling a code-ref once for each Accessible Object
(including the starting object). The code can control iteration using
level
, prune
, stop
and pin
(see sample).
$ao->tree ( sub { my ($ao, $monkey) = @_; # $monkey->level() returns count of levels from root. # $monkey->prune() skips this AO's children. # $monkey->stop() visits NO more Accessible Objects. # $monkey->pin() prevents closing any menus and outlines # that tree() opened # (applies only if flag 'active'=>1) print ' 'x($monkey->level())."$ao\n"; }, #+{ 'active'=>1 } # optional iterator flags-hash );
When tree
obtains an iterator for each Accessible Object it visits,
tree
passes its second argument (an optional hash) to the
iterator's constructor. (See More on Iterators.)
Supposing $ao
is a client area containing a Close button, here's
how to find and press Close:
$aoBtnClose = $ao->dig ( +[ "{push button}Close" # {role}name ] ); $aoBtnClose->dda_Press();
If $ao
is a window, containing a client area, containing a Close button,
just set forth both steps in the path to reach Close from $ao
:
$aoBtnClose = $ao->dig ( +[ "{client}", # {role} only, name doesn't matter "{push button}Close" ] );
In a word, dig
follows a path of Window Tests,
and returns what it finds. See details at Finding Accessible Objects using 'dig'.
You can run aadigger or aaWhereAmI interactively to reconnoiter and figure out a path to the interesting Accessible Object.
menuPick
uses Active Accessibility and Win32::GuiTest to manipulate
standard Windows menu-bars and context-menus. Your mileage may vary
with apps that use custom menus with cockeyed support for Active
Accessibility.
# menuPick takes a ref to a list of window-tests, # tracing a path from menubar to menu to submenu etc., # plus an optional flags hash. $ao->menuPick(+[ 'Edit', qr/^Undo/ ], +{'trace'=>1} );
menuPick
can summon and operate a right-clicky context menu:
$ao->context_menu()->menuPick(+[ 'Paste' ]);
If Win32::GuiTest
has been loaded (as by use Win32::GuiTest;
),
the active menu iterator closes menus when it's done with them.
Some menus contain items marked as invisible. Use the HASH form of the window-test to pick such an invisible item; the string and regex window-tests match only visible items.
Get an HWND or location from an Accessible Object and manipulate it with the Windows API:
use Win32::GuiTest; # use a HWND my $hwnd = $ao->WindowFromAccessibleObject(); my $name = Win32::GuiTest::GetWindowText($hwnd); # use an (x,y) location my ($left, $top, $width, $height) = $ao->accLocation(); Win32::GuiTest::MouseMoveAbsPix($left, $top);
A window-test examines an Accessible Object and returns a true or
false value -- like Perl's file tests (-e, -f, etc.). Window-tests are used in waitForEvent
,
dig
, menuPick
, and match
.
A window-test can take the form of a string, a regex, or a hash.
window # role text ROLE_SYSTEM_WINDOW # constant name WINDOW # last leg of constant name Win32::ActAcc::Window # Perl package for the role Window # last leg of package name value of ROLE_SYSTEM_WINDOW() # the role number
$name=~qr/regex/
.
Matches visible objects only.
Hash members (all are optional; all present members must match the Accessible Object):
get_accRole
hash memberget_accName
hash memberget_accValue
hash memberget_accDescription
hash memberget_accHelp
hash memberget_accDefaultAction
hash memberWindowFromAccessibleObject
hash membervisible
hash memberstate_has
and state_lacks
hash membersrole_in
or role_not_in
hash membercode
hash memberSample window-tests:
$b = $ao->match('Close'); # Is AO's name exactly Close? $b = $ao->match( +{'get_accName'=>'Close'} ); # ... using a hash. $b = $ao->match(qr/Close/); # Does AO's name match that regexp? $b = $ao->match( +{'get_accName'=>qr/Close/} ); # ... using a hash. $b = $ao->match('{ROLE_SYSTEM_PUSHBUTTON}Close'); # Is AO a pushbutton named Close? $b = $ao->match('{push button}Close'); # ... using localized 'role text' $b = $ao->match('{Pushbutton}Close'); # ... using ActAcc package name $b = $ao->match( +{'get_accRole'=>ROLE_SYSTEM_PUSHBUTTON(), 'name'=>'Close'} ); # ... $b = $ao->match( +{'rolename'=>'Pushbutton', 'get_accName'=>'Close'} ); # ... $b = $ao->match ( +{'code'=> sub { my $ao = shift; return $ao->match( qr/Bankruptcy in progress/ ); } } ); $b = $ao->match( +{'visible'=>1} );
There is more to the 'visible'=>1 test than meets the eye..
my $might_be_visible = $ao->visible(); my $same_thing = $ao->match( +{'visible'=>1} );
The visible
function returns a true value if none of these
reasons-for-being-invisible applies.
visible
does not call get_accParent
, which may lead to a cycle
in a buggy app, but instead relies on the cached weak-ref from the
iterator that found this Accessible Object.
Location is undefined (unless state includes 'focusable')
Location is entirely negative
Height or width is zero
The algorithm overlooks other reasons-for-being-invisible, such as occlusion and inconspicuousness.
dig
follows a path of Window Tests, and returns what it
finds.
Depending on its scalar or list context, and min and max options,
dig
can perform various searches:
dig
in array context without specifying options min
or max
.
dig
in array context, specifying option min=0
.
dig
in scalar context, specifying neither option min
nor max
, or specifying them both as 1.
dig
in scalar context, specifying min=0
. If it finds no
matching Accessible Object, dig
returns undef
.
The optional second parameter is a hash of options:
dig
passes these along when it obtains an iterator for each
Accessible Object it traverses. dig
sets the ``active'' flag unless
the options hash specifies it as a non-true value.
Samples using dig
:
# Find one immediate child of $ao with role "client"; die if not found. my $aoClient = $ao->dig( +[ '{client}' ] ); # Find one untitled Notepad within the Desktop's client area; die if not found. my $someNewNotepad = Desktop()-> dig(+[ '{client}', # step 1 '{window}Untitled - Notepad' # step 2 ]); # Get results into a list: find *all* untitled Notepad # windows in the Desktop's client area. Die if none found. my @allNewNotepads1 = Desktop()-> dig(+[ '{client}', # step 12 '{window}Untitled - Notepad' # step 2 ]); # Find all untitled Notepads, using a regex to match their name. my @allNewNotepads2 = Desktop()-> dig(+[ '{client}', # step 1 +{ # step 2: 'get_accRole'=>ROLE_SYSTEM_WINDOW(), 'get_accName'=>qr/^Untitled - Notepad$/ } ]); # Find all untitled Notepads that contain an Application menubar. my @allNewNotepads3 = Desktop()-> dig(+[ '{client}', # step 1 +{ # step 2: 'get_accRole'=>ROLE_SYSTEM_WINDOW(), 'get_accName'=>qr/^Untitled - Notepad$/ }, +{ # step 3: 'get_accRole'=>ROLE_SYSTEM_MENUBAR(), 'get_accName'=>'Application' }, +{ # step 4: back up! 'axis'=>'parent' }, ]); # Find windows on desktop. Die if fewer than 2. Return at most 42. my @upTo42Windows = Desktop()->dig( +[ '{client}', # step 1 '{window}' # step 2 ], +{ # options 'min'=>2, # -die unless at least 2 'max'=>42, # -shortcut after 42 'trace'=>1 # -for troubleshooting } );
The active
, nav
, and perfunctory
options configure the
iterator with which dig
enumerates each Accessible Object's
children in its quest for potential matches..
The default iterator uses both AccessibleChildren
and accNavigate
,
which is slow but works with many applications.
my $iter = $ao->iterator();
Optional hints convey a preference for an iterator type, if it applies to the Accessible Object:
# operate menus and outlines, treating consequences as children my $iter = $ao->iterator( +{ 'active'=>1 } ); # use AccessibleChildren my $iter = $ao->iterator( +{ 'perfunctory'=>1 } ); # use accNavigate my $iter = $ao->iterator( +{ 'nav'=>1 } );
For completeness, there is an iterator that uses get_accChildCount
and get_accChild
:
my $iter = new Win32::ActAcc::get_accChildIterator($ao);
Win32::ActAcc installs a system-wide in-process event hook upon the
first call to clearEvents
. Thereafter, events stampede through
a circular buffer. You can watch by running aaEvents.
All Perl processes share one event hook and one circular buffer, but each Perl process keeps its own independent pointer into the buffer.
getEvent
retrieves one event from the circular buffer and
advances the pointer:
# Retrieve one event from circular buffer (if any there be). my $anEvent = getEvent(); if (defined($anEvent)) { print "Event: $anEvent\n"; }
Scripts may getEvent
in a loop to watch for a specific sentinel
event. Such a loop is included: waitForEvent
consumes events
until one satisfies a hash-of-criteria (sample in the Synopsis) or a
code-ref:
waitForEvent ( sub { my $e = shift; if ($$e{'event'} == EVENT_SYSTEM_FOREGROUND()) { my $ao = $e->AccessibleObjectFromEvent(); # or getAO() for short my $name = $ao->get_accName(); return $ao if ($name =~ qr/Notepad$/); } return undef; }, # options: +{ 'timeout'=>30, # seconds # (an hourglass buys a few more seconds) 'trace'=>1 # display events as troubleshooting aid # Tip: Don't turn off 'trace' until your script works! } ) or die("Notepad didn't come to foreground in the allotted time.");
To prevent a stale event from triggering the exit condition, call
clearEvents
before taking the action whose consequences the script
will be looping in wait for.
``eg\playpen.pl'' demonstrates using Active Accessibility to inspect and manipulate a menu, a popup menu, a text-entry blank, a checkbox, a radio button, a spin button, tabs, a list box, a tree-list, a two-column list view, and suchlike.
Of course, playpen.pl depends on an application that presents such widgets. The applets that come with Windows change too often, so playpen.pl uses a simple C# app whose source code is in eg\playpen.
playpen.pl also depends on Win32::GuiTest.
Build the playpen C# app, then invoke playpen.pl to explore it:
> vcvars32 || rem ember to put 'csc' on the path > cd eg\playpen > build.cmd > cd .. > perl playpen.pl
If an Active Accessibility function unexpectedly returns undef
,
check Perl's Extended OS Error special variable $^E
for clues.
Run your script with ``perl -w''.
If your script doesn't work, see whether the aadigger sample works.
If you see Windows error 0x800401f0 (``CoInitialize has not been called''), make sure your script starts off with Win32::OLE->Initialize().
If you get a Windows error number and want to know what it means,
try using Win32::FormatMessage
.
If your script sometimes misses noticing an event that occurs very
soon after your script calls clearEvents()
for the first time,
insert a sleep
after that first clearEvents()
. Installing a
WinEvent handler seems to take effect ``soon'', but not synchronously.
If you are desperate enough to insert ``print'' statements:
print "The Accessible Object is: $ao\n"; # shows several attributes print "same as: ". $ao->describe() . "\n";
$ao->debug_tree(); # display $ao and all its descendants
If you are struggling to find the right event or window-test for
use with waitForEvent
, dig
, tree
, or menuPick
,
try using the trace
flag to evoke a lot of progress messages.
Or, embed the interactive aadigger feature into your script:
# invoke interactive explorer feature starting at $ao: use Win32::ActAcc::aaExplorer; Win32::ActAcc::aaExplorer::aaExplore($ao);
If menuPick
doesn't work because your computer is too slow,
increase the value of $Win32::ActAcc::MENU_SLOWNESS
. menuPick
relies on a hover delay to give the app a chance to update a menu-item
object's default action.
If your script displays gibberish instead of Unicode text on the console, try writing to a file instead.
It doesn't implement get_accHelpTopic
and accHitTest
.
menuPick
doesn't know how to choose commands that hid
because you seldom use them.
You can't use a Win32::ActAcc ``Accessible Object'' with Win32::OLE.
It probably doesn't work multi-threaded.
Apps with a buggy IAccessible
implementation may cause the Perl
process to crash.
Copyright 2000-2004, Phill Wolf.