MiroTestAutomation
From Develop
Contents |
Create or Run Miro's automated test suites
Using Sikuli,a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots) we have a cross platform suite of Automated tests.
Test Organization
The automated tests suites are created as a set of python unittests that are driven by the sikuli-script.jar file. Tests are organized into test suites the mimic the functionality tested in the manual regression test cases in litmus regression tests.
For example:
litmus subgroup 6 displays the list of tests in Subgroup 6 feeds. In Miro Sikuli test suite there is the directory: sg6_feeds.sikuli, that contains the file sg6_feeds.py.
sg6_feeds.py is a set of python unittests that contain the following test modules: test_123, test_138, test_338, test_339...
Test Execution
To get started executing the automated tests:
- Install the version of miro you want to test. Start it once before testing to get rid of warning messages from running a downloaded program.
- Install and setup Sikuli according to the: TestAutomation#Sikuli_Testing_Setup_Instructions
- Download or clone the Miro test suite from the github repository
- Execute the test suite
Additional Test Run information
It is possible to run the complete test suite, just 1 subgroup, or 1 or more in a particular subgroup. In this case, a subgroup is a suite of tests related to a particular functional area, as defined in the Litmus manual regression tests.
To run all the tests:
python run_suite.py
To run a particular subgroup, using subgroup 6 as the example:
java -jar $SIKULI_HOME/sikuli-script.jar sg6_feeds.sikuli
on Windows, you need to quote the file to account for spaces in the path:
java -jar "$SIKULI_HOME/sikuli-script.jar" sg6_feeds.sikuli
To run 1 or many testcases from a subgroup, using subgroup 6 as an example:
java -jar "$SIKULI_HOME/sikuli-script.jar" sg6_feeds.sikuli test_123 test_338
Things to consider when running tests
- After you start the test suite, minimize the terminal window. Having a terminal window with "Miro" in the title can confuse Sikuli
- Use a clean sidebar - since these are just basic functional tests, the tests will run more reliably if you clear out extra podcasts, sources and stores, so there is less to look through when finding matching text.
- Hands Off - you need to leave the machine to run the tests, if you open other windows or applications, then Sikuli won't be able to find the images at the correct time.
- Make sure your computer won't go to sleep during the tests.
- Make sure your keyboard layout is set to US.
Results Reporting
As each subgroup is run, a log file Log_current.xml is written to the main PROJECT_DIR. At the completion of the test, it is copied into PROJECT_DIR/last_run directory and given a date-time stamp to identify it uniquely. The log file is an xml file formatted to be compliant with the Litmus Web Services standard.
Upon completion of the test run, examine the results files in the last_run directory to determine if the results accurately represent the functionality of the Miro test. If you are satisfied with the test results, and any failures are Not the the result of errors in the test itself, you can submit the results directory to litmus.
To submit the results:
python just_send_result.py
Note: If you encounter any test failures - please manually test the functionality, and file a bug for any Miro regressions.
If you think test failures are a result of issues with the test themselves, please zip up the last_run directory and send it to me, jed at pculture dot org.
You may want to clear out the old results before starting a new test run.
Creating more tests
Want to make some? See testautomation-howto - coming soon.
