libcvautomation/writing_app_tests.html

350 lines
22 KiB
HTML

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<title>libcvautomation: Writing Application Tests</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="search/search.js"></script>
<script type="text/javascript">
$(document).ready(function() { searchBox.OnSelectItem(0); });
</script>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
extensions: ["tex2jax.js"],
jax: ["input/TeX","output/HTML-CSS"],
});
</script><script src="http://www.mathjax.org/mathjax/MathJax.js"></script>
</head>
<body>
<div id="top"><!-- do not remove this div! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<td style="padding-left: 0.5em;">
<div id="projectname">libcvautomation
&#160;<span id="projectnumber">2.0</span>
</div>
</td>
</tr>
</tbody>
</table>
</div>
<!-- Generated by Doxygen 1.7.6.1 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "search",false,'Search');
</script>
<div id="navrow1" class="tabs">
<ul class="tablist">
<li><a href="index.html"><span>Main&#160;Page</span></a></li>
<li class="current"><a href="pages.html"><span>Related&#160;Pages</span></a></li>
<li><a href="namespaces.html"><span>Namespaces</span></a></li>
<li><a href="annotated.html"><span>Classes</span></a></li>
<li><a href="files.html"><span>Files</span></a></li>
<li>
<div id="MSearchBox" class="MSearchBoxInactive">
<span class="left">
<img id="MSearchSelect" src="search/mag_sel.png"
onmouseover="return searchBox.OnSearchSelectShow()"
onmouseout="return searchBox.OnSearchSelectHide()"
alt=""/>
<input type="text" id="MSearchField" value="Search" accesskey="S"
onfocus="searchBox.OnSearchFieldFocus(true)"
onblur="searchBox.OnSearchFieldFocus(false)"
onkeyup="searchBox.OnSearchFieldChange(event)"/>
</span><span class="right">
<a id="MSearchClose" href="javascript:searchBox.CloseResultsWindow()"><img id="MSearchCloseImg" border="0" src="search/close.png" alt=""/></a>
</span>
</div>
</li>
</ul>
</div>
</div>
<div class="header">
<div class="headertitle">
<div class="title">Writing Application Tests </div> </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><dl class="author"><dt><b>Author:</b></dt><dd>Bradlee Speice &lt;<a href="mailto:bspeice@uncc.edu">bspeice@uncc.edu</a>&gt; </dd></dl>
<dl class="date"><dt><b>Date:</b></dt><dd>7/30/2012 </dd></dl>
<h2><a class="anchor" id="audience"></a>
Audience</h2>
<ul>
<li>
This document was written for people with some intermediate knowledge of BASH. </li>
<li>
Screenshot experience is required - being able to create screenshots of windows using <a href="http://www.gimp.org/">GIMP</a>, the <code>import</code> command from <a href="http://www.imagemagick.org/script/index.php">ImageMagick</a>, or something similar. <br/>
</li>
<li>
Required for working with libcvautomation internals (not needed strictly for writing application testing): <ul>
<li>
Intermediate-level C knowledge required for interfacing with libcvautomation. There isn't much complicated going on with libcvautomation itself, but you need to know to use <a class="el" href="libcvautomation-xlib_8h.html#a575e80fc9cf076026e127a57888423d4" title="Simple wrapper for XOpenDisplay.">cvaOpenDisplay()</a> for grabbing a display and then <a class="el" href="libcvautomation-xlib_8h.html#a65ab3bc56b956970a6b3d31c624d464a" title="Simple wrapper for XCloseDisplay.">cvaCloseDisplay()</a> for closing it later for example. </li>
<li>
C++ is available, but currently only as <code>extern "C"</code> style bindings. </li>
<li>
Python bindings are in progress as of time-of-writing </li>
</ul>
</li>
</ul>
<h2><a class="anchor" id="purpose"></a>
Purpose</h2>
<ul>
<li>
This document is intended to outline the libcvautomation testing library for GUI applications and its two reference programs <code>cva-match</code> and <code>cva-input</code> </li>
<li>
These programs allow you to automate mouse and keyboard events in response to what appears on screen - for example, clicking a button based on an image of that button on screen. </li>
</ul>
<h2><a class="anchor" id="using"></a>
Using Libcvautomation</h2>
<ul>
<li>
Since libcvautomation is a shared-object library intended to bundle a lot of functionality in one area, we must use external programs to agin access to the functions of libcvautomation. </li>
<li>
Two reference programs have been included to make this easy - <code>cva-match</code> and <code>cva-input</code> </li>
<li>
The reference programs are fairly full-featured, and expose most of libcvautomation: <ul>
<li>
<code>cva-match</code> allows you to match multiple image files against a root image providing very fine control over how specific the match is </li>
<li>
<code>cva-input</code> allows you to drive the X11 server using the XTest extension - for example, clicking on a button from image, clicking a key on the keyboard, and more. </li>
</ul>
</li>
<li>
These two programs should implement all functionality needed for GUI automation. Please contact <a href="mailto:bspeice@uncc.edu">bspeice@uncc.edu</a> if you have suggestions for extra functionality, patches, comments, etc. </li>
<li>
Finally, if you want to write your own programs using libcvautomation, the headers are located on your system. Use: <div class="fragment"><pre class="fragment"><span class="preprocessor"> #include &lt;<a class="code" href="libcvautomation_8h.html" title="The top-level include for all projects involving libcvautomation.">libcvautomation/libcvautomation.h</a>&gt;</span>
</pre></div> to include all necessary header files. See the "Files" tab above to get an idea of what functionality exists. <dl class="note"><dt><b>Note:</b></dt><dd>Intermediate C or C++ knowledge is required for programming with libcvautomation. C++ is currently only supported through <code>extern "C"</code>, and full C++ bindings are not currently planned. Libcvautomation itself is incredibly simple, but interfaces with a few API's (Xlib, libcv) that it is helpful to have an idea of how to use.</dd></dl>
</li>
</ul>
<h2><a class="anchor" id="testing"></a>
Application Testing</h2>
<p>Now we get into the good stuff. The basic process for application testing is as follows: </p>
<ul>
<li>
Create a sequence of screenshots for all mouse-clicks </li>
<li>
Create a testing wrapper in BASH for automating the application </li>
<li>
Test the wrapper and tune any necessary options </li>
</ul>
<h3><a class="anchor" id="screenshot_sequence"></a>
Creating the Screenshot Sequence</h3>
<ul>
<li>
This process creates a set of screenshots by which libcvautomation can drive the X11 server </li>
<li>
The process is simple - create a screenshot of all buttons you would like to click in an application test, or location where you would like to move the mouse, etc. </li>
<li>
This can be accomplished a number of different ways - using GIMP, the <code>import</code> command, or any other screenshot program. Additionally, libcvautomation will support any image format that OpenCV does. At the time of writing, these are: <ul>
<li>
Windows bitmaps - <code>*.bmp</code>, <code>*.dib</code> </li>
<li>
JPEG files - <code>*.jpeg</code>, <code>*.jpg</code>, <code>*.jpe</code> </li>
<li>
JPEG 2000 files - <code>*.jp2</code> </li>
<li>
Portable Network Graphics - <code>*.png</code> </li>
<li>
Portable image format - <code>*.pbm</code>, <code>*.pgm</code>, <code>*.ppm</code> </li>
<li>
Sun rasters - <code>*.sr</code>, <code>*.ras</code> </li>
<li>
TIFF files - <code>*.tiff</code>, <code>*.tif</code> </li>
</ul>
<br/>
</li>
<li>
<p class="startli">Some tips on creating screenshots: </p>
<ul>
<li>
Use as distinct an image as possible in a screenshot. For example, the following images look very similar, but can do very different things: <div class="image">
<img src="print-printbutton1.png" alt="print-printbutton1.png"/>
</div>
<br/>
<div class="image">
<img src="print-helpbutton.png" alt="print-helpbutton.png"/>
</div>
<ul>
<li>
The large amount of grey-space in each of the images can be very confusing. While the text itself is different, the overwhelming amount of grey space can result in a false positive. Limiting the space of the screenshot can be useful: <div class="image">
<img src="print-printbutton2.png" alt="print-printbutton2.png"/>
</div>
<br/>
<div class="image">
<img src="print-helpbutton.png" alt="print-helpbutton.png"/>
</div>
</li>
<li>
Limiting the button size in this case helps the program identify what button you want to click, but in general you want to include as much detail as possible. </li>
</ul>
<br/>
</li>
<li>
Be careful of where an image may appear multiple times in a screenshot. For example, a "Help" button on a dialog may get confused with the "Help" menu at the top of the screen. </li>
</ul>
<p class="endli"></p>
</li>
</ul>
<h3><a class="anchor" id="testing_wrapper"></a>
Create a Testing Wrapper</h3>
<ul>
<li>
This is where the BASH knowledge comes into play. We are going to write a script that will run your application test, to make sure that the GUI is functioning correctly. </li>
<li>
The following is the (strongly) recommended process, but is not strictly necessary to follow this. The way I'm going to explain this is by giving an example test I wrote, and explain what is going on: <div class="fragment"><pre class="fragment"><span class="preprocessor"> #!/bin/bash</span>
<span class="preprocessor"></span><span class="preprocessor"> #This is an application test involving libcvautomation and libreoffice</span>
<span class="preprocessor"></span> . /etc/libcvautomation_funcs
<span class="keyword">set</span> -o errexit
<span class="preprocessor"> #Changing any wrapper parameters should go here</span>
<span class="preprocessor"></span> TIMEOUT=30
start_libreoffice_writer ()
{
<a class="code" href="namespacelibcvautomation__funcs.html#a1a05342b0159044fae6bdb9969d488d6" title="Click a mouse button on an image inside the root X11 window.">mouse_click_image</a> <span class="stringliteral">&quot;screens/gnome-menu.png&quot;</span> <span class="stringliteral">&quot;screens/kde-menu.png&quot;</span>
<a class="code" href="namespacelibcvautomation__funcs.html#a1a05342b0159044fae6bdb9969d488d6" title="Click a mouse button on an image inside the root X11 window.">mouse_click_image</a> <span class="stringliteral">&quot;screens/gnome-officeMenu.png&quot;</span> <span class="stringliteral">&quot;screens/kde-officeMenu.png&quot;</span>
<a class="code" href="namespacelibcvautomation__funcs.html#ac952f6f97c0bc2acdf1ad5b4d1d5acc1" title="Move the mouse to an image inside the root X11 window.">mouse_hover_image</a> <span class="stringliteral">&quot;screens/gnome-LibreOfficeWriter.png&quot;</span> <span class="stringliteral">&quot;screens/kde-LibreOfficeWriter.png&quot;</span>
<a class="code" href="namespacelibcvautomation__funcs.html#a7c2ecbf863dcd5e4af933f38752e81e5" title="Jiggle the mouse in place.">mouse_jiggle</a>
<a class="code" href="namespacelibcvautomation__funcs.html#af4d9b2f324c23336b83399f7fd71852e" title="Press and release a mouse button.">mouse_click</a>
}
close_libreoffice_writer()
{
<a class="code" href="namespacelibcvautomation__funcs.html#a1a05342b0159044fae6bdb9969d488d6" title="Click a mouse button on an image inside the root X11 window.">mouse_click_image</a> <span class="stringliteral">&quot;screens/gnome-fileMenu.png&quot;</span> <span class="stringliteral">&quot;screens/kde-fileMenu.png&quot;</span>
<a class="code" href="namespacelibcvautomation__funcs.html#a1a05342b0159044fae6bdb9969d488d6" title="Click a mouse button on an image inside the root X11 window.">mouse_click_image</a> <span class="stringliteral">&quot;screens/gnome-fileExit.png&quot;</span> <span class="stringliteral">&quot;screens/kde-fileExit.png&quot;</span>
<a class="code" href="namespacelibcvautomation__funcs.html#a1a05342b0159044fae6bdb9969d488d6" title="Click a mouse button on an image inside the root X11 window.">mouse_click_image</a> <span class="stringliteral">&quot;screens/gnome-discard.png&quot;</span> <span class="stringliteral">&quot;screens/kde-discard.png&quot;</span>
}
start_libreoffice_writer
close_libreoffice_writer
</pre></div> <br/>
<div class="fragment"><pre class="fragment"><span class="preprocessor"> #!/bin/bash</span>
<span class="preprocessor"></span><span class="preprocessor"> #This is an application test involving libcvautomation and libreoffice</span>
<span class="preprocessor"></span> . /etc/libcvautomation_funcs
<span class="keyword">set</span> -o errexit
<span class="preprocessor"> #Changing any wrapper parameters should go here</span>
<span class="preprocessor"> TIMEOUT=30</span>
</pre></div> </li>
<li>
The purpose of these lines is just the standard BASH header. Additionally, we import a wrapper created for libcvautomation to make our job easier. Please note that this is the default directory for the wrapper, your installation may be different. Use the command <code>locate <a class="el" href="namespacelibcvautomation__funcs.html" title="Namespace to contain all of the features needed to use the libcvautomation_funcs wrapper.">libcvautomation_funcs</a></code> to find it on your computer. The wrapper itself is a handful of macros used to make our job easy. </li>
<li>
The <code>set</code> line will abort the test if an error is ever encountered - for example, no images are found. </li>
<li>
Changing any wrapper parameters should go after sourcing the wrapper functions. See <a class="el" href="wrapper_functions.html#appendix_variables">Environment Variables</a> for more information. We set the TIMEOUT to 30 seconds here, so that the wrapper will wait 30 seconds (max) for an image to appear before giving up. This way if LibreOffice takes 30 seconds to load, we will wait 30 seconds. If it takes only 5 seconds to load, we will click after those 5 seconds. <br/>
<br/>
<div class="fragment"><pre class="fragment"> start_libreoffice_writer ()
{
<a class="code" href="namespacelibcvautomation__funcs.html#a1a05342b0159044fae6bdb9969d488d6" title="Click a mouse button on an image inside the root X11 window.">mouse_click_image</a> <span class="stringliteral">&quot;screens/gnome-menu.png&quot;</span> <span class="stringliteral">&quot;screens/kde-menu.png&quot;</span>
<a class="code" href="namespacelibcvautomation__funcs.html#a1a05342b0159044fae6bdb9969d488d6" title="Click a mouse button on an image inside the root X11 window.">mouse_click_image</a> <span class="stringliteral">&quot;screens/gnome-officeMenu.png&quot;</span> <span class="stringliteral">&quot;screens/kde-officeMenu.png&quot;</span>
<a class="code" href="namespacelibcvautomation__funcs.html#ac952f6f97c0bc2acdf1ad5b4d1d5acc1" title="Move the mouse to an image inside the root X11 window.">mouse_hover_image</a> <span class="stringliteral">&quot;screens/gnome-LibreOfficeWriter.png&quot;</span> <span class="stringliteral">&quot;screens/kde-LibreOfficeWriter.png&quot;</span>
<a class="code" href="namespacelibcvautomation__funcs.html#a7c2ecbf863dcd5e4af933f38752e81e5" title="Jiggle the mouse in place.">mouse_jiggle</a>
<a class="code" href="namespacelibcvautomation__funcs.html#af4d9b2f324c23336b83399f7fd71852e" title="Press and release a mouse button.">mouse_click</a>
}
</pre></div> </li>
<li>
This is the actual body of work done by libcvautomation </li>
<li>
<code>click_i</code> is a function to click the mouse at an image - in this case, the gnome or kde menu. <ul>
<li>
Because of how the cva-input program is designed, you can give it multiple images, and it will only select the one currently available. See the <a class="el" href="wrapper_functions.html">Appendix of Wrapper Functions and Environment Variables</a> for more information on how to use this (<code>TOLERANCE</code> specifically) </li>
<li>
Additionally, the wrapper (by default) will wait for an image to appear before clicking on it. This way, you can string together click_i commands even when the program may need to wait a while on processing. Make sure to read up on the <code>TIMEOUT</code> option to learn how to use this. </li>
<li>
By using the function <code>click_i</code>, we make things a bit more readable - the full command line is <code>cva-input -s 'icmouseclick &lt;filename&gt;'</code> </li>
<li>
See the <a class="el" href="wrapper_functions.html">Appendix of Wrapper Functions and Environment Variables</a> for a list of all functions available in the wrapper. </li>
</ul>
</li>
<li>
<code>hover_i</code> is a function to move the mouse to an image - in this case, move it over the LibreOffice menu item. </li>
<li>
Then we jiggle the mouse to make sure that the item activates, click, and wait for the program to start up. <br/>
<br/>
<div class="fragment"><pre class="fragment"> close_libreoffice_writer()
{
<a class="code" href="namespacelibcvautomation__funcs.html#a1a05342b0159044fae6bdb9969d488d6" title="Click a mouse button on an image inside the root X11 window.">mouse_click_image</a> <span class="stringliteral">&quot;screens/gnome-fileMenu.png&quot;</span> <span class="stringliteral">&quot;screens/kde-fileMenu.png&quot;</span>
<a class="code" href="namespacelibcvautomation__funcs.html#a1a05342b0159044fae6bdb9969d488d6" title="Click a mouse button on an image inside the root X11 window.">mouse_click_image</a> <span class="stringliteral">&quot;screens/gnome-fileExit.png&quot;</span> <span class="stringliteral">&quot;screens/kde-fileExit.png&quot;</span>
<a class="code" href="namespacelibcvautomation__funcs.html#a1a05342b0159044fae6bdb9969d488d6" title="Click a mouse button on an image inside the root X11 window.">mouse_click_image</a> <span class="stringliteral">&quot;screens/gnome-discard.png&quot;</span> <span class="stringliteral">&quot;screens/kde-discard.png&quot;</span>
}
</pre></div> </li>
<li>
Use the <code>click_i</code> function to close down LibreOffice writer - Find the "File" menu, click "Exit", and then make sure to discard all changes. <br/>
<br/>
<div class="fragment"><pre class="fragment"> start_libreoffice_writer
close_libreoffice_writer
</pre></div> </li>
<li>
Actually run the functions given. <br/>
<br/>
</li>
<li>
Please note that this is a fairly trivial example. The full list of commands available in the wrapper is given in <a class="el" href="wrapper_functions.html">Appendix of Wrapper Functions and Environment Variables</a> </li>
</ul>
<h3><a class="anchor" id="testing_test_wrapper"></a>
Testing the Testing Wrapper</h3>
<ul>
<li>
First things first, run through the testing wrapper to make sure that everything is O.K. </li>
<li>
If you need to, there are a few environment variables you can set to change how the wrapper works. See <a class="el" href="wrapper_functions.html#appendix_variables">Environment Variables</a> for more information on how these work. </li>
<li>
A full list of commands provided by the wrapper is available at <a class="el" href="wrapper_functions.html#appendix_functions">Functions</a> </li>
</ul>
<dl class="note"><dt><b>Note:</b></dt><dd>These options are controlled using the testing script as demonstrated above. Any modifications to the following values should be done at the line: <div class="fragment"><pre class="fragment"><span class="preprocessor"> #Changing any wrapper parameters should go here </span>
</pre></div> </dd></dl>
<h2><a class="anchor" id="wrapping_up"></a>
Wrapping Up</h2>
<ul>
<li>
At this point you should have all the information you need to write your own application tests. The libcvautomation library and reference programs were designed to be simple and powerful, but if you invest the time to learn them and some expert BASH scripting, you can do some very complex things. <ul>
<li>
For example, integrating a <a href="http://testanything.org/wiki/index.php/Tap-functions">Test Anything Protocol</a> wrapper into your scripts as well. </li>
</ul>
</li>
<li>
If you have questions, comments, concerns, suggestions, or feedback in general, feel free to let me know at <a href="mailto:bspeice@uncc.edu">bspeice@uncc.edu</a>. </li>
</ul>
</div></div><!-- contents -->
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
onmouseover="return searchBox.OnSearchSelectShow()"
onmouseout="return searchBox.OnSearchSelectHide()"
onkeydown="return searchBox.OnSearchSelectKey(event)">
<a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(0)"><span class="SelectionMark">&#160;</span>All</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(1)"><span class="SelectionMark">&#160;</span>Classes</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(2)"><span class="SelectionMark">&#160;</span>Namespaces</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(3)"><span class="SelectionMark">&#160;</span>Files</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(4)"><span class="SelectionMark">&#160;</span>Functions</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(5)"><span class="SelectionMark">&#160;</span>Defines</a></div>
<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0"
name="MSearchResults" id="MSearchResults">
</iframe>
</div>
<hr class="footer"/><address class="footer"><small>
Generated on Thu Oct 11 2012 17:42:49 for libcvautomation by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/>
</a> 1.7.6.1
</small></address>
</body>
</html>