UniversalSubtitlesQA

From Develop

Contents 1 QA information for Universal Subtitles 1.1 Exploratory Testing 1.2 Bug Verification 1.3 Manual Regression Testing 1.4 Automated Testing 1.4.1 Javascript Unittest Execution 1.4.2 Functional Test Execution 1.4.3 Test Organization: 1.5 Dev -> Testing -> Release Guidelines 1.6 Complete Release Process 1.6.1 Start of release process 1.6.2 Steps 1.6.3 Final Release Testing 1.6.4 Push to Production 1.6.5 Follow-up 1.7 Cherrypick Release Process 1.7.1 Start of Cherrypick Release Process 1.7.2 Steps 1.7.3 Push to Production 1.7.4 Follow-up

QA information for Universal Subtitles

If you would like to contribute or just keep up with what's going on, please join our google testing group

We are on irc.freendode.org #universalsubtitles

Exploratory Testing

See UniSubs-Current-Release-Testing to see the list of new features that need testing.

Bug Verification

Work through the list of FIXED bugs in bugzilla, marking as verified if fixed, or reopen if not fixed. See UniSubs-Current-Release-Testing#Bugzilla_information_.28Fixed_.2F_Open_.2F_Create.29 for a link to fixed bugs that need verification.

Manual Regression Testing

Regression tests are found on litmus

To execute the manual regression tests:

1. Login (or create an account) on litmus
2. Execute the tests in the Universal Subtitles Test Run

New users can get more information from the getting started guide

Automated Testing

We are using Selenium-RC to automate testing for universal subtitles functional tests and the javascript unittests.

Javascript Unittest Execution

To run the tests:

1. Install and setup Selenium according to the: TestAutomation#Selenium-RC_Testing_Setup_Instructions
2. Download or clone the Universal Subtitles test suite from the github repository
3. Execute the test suite

python js_testrunner.py

The default run command will run all the tests on Firefox and output the test results as an HTML formatted file into the Results directory.

Additional Test Run information

There are some commandline arguments available to control the test run:

python js_testrunner.py --help
Usage: js_testrunner.py [options]

Options:
  -h, --help            show this help message and exit
  -s, --sauce           Runs the test on saucelabs.com runs tests on all browsers and outputs results to Result dir
  -u SITE, --siteurl=SITE
                        dev for: http://dev.universalsubtitles.org,
                        staging for: http://staging.universalsubtitles.org
  -p PORT, --port=PORT  

Functional Test Execution

To run the tests:

1. Install and setup Selenium according to the: TestAutomation#Selenium-RC_Testing_Setup_Instructions
2. Download or clone the Universal Subtitles test suite from the github repository
3. Execute the test suite

python test_withHTMLoutput.py

The default run command will output the test results as an HTML formatted file into the Results directory.

Additional Test Run information

There are some commandline arguments available to control the test run:

python test_withHTMLoutput.py --help
Usage: test_withHTMLoutput.py [options]

Options:
  -h, --help            show this help message and exit
  
  -b BROWSER, --browser=BROWSER
                        Possible browser choices: firefox,chrome,opera,
                        safari, iexplore, googlechrome
  -p PORT, --port=PORT  
  -u SITE, --siteurl=SITE
                        '''dev''' for: http://dev.universalsubtitles.org,
                        '''staging''' for: http://staging.universalsubtitles.org
  -l, --litmus          Sends test output directly to litmus.pculture.org
  -i BUILDID, --buildid=BUILDID
                        specify the build id of the litmus testrun results to
                        display there

Test Organization:

The selenium tests use the selenium-rc python client library and are structered as python unit tests. Tests are organized by litmus subgroup and should match the functionality defined by the manual test cases.

Automated Subgroups:

• Subgroup_64 - Submit videos from various sources
• Subgroup 65 - Login tests
• Subgroup 69 - Demo Wiget UI tests
• Subgroup 70 - Revisions tests
• Subgroup 78 - Pagedemo tests
• Subgroup 78 - Offsite widget tests
• Subgroup 80 - Comments tests
• Subgroup_81 - Subtitles Upload / Download
• Subgroup_88 - Teams tests
• Subgroup_94 - widgetizer tests
• Subgroup_95 - watch page tests

To run a specific subgroup:

python sg_69_demoUI.py

Browser configuration is specified in selvars.py and can be manually modified, default is firefox.

Helper Fuctions and variables:

• widget.py - performs actions in the widget
• website.py - performs actions on the universalsubtitles test website.
• offsite.py - performs actions on other sites, ex: gets a blip or youtube video url
• mslib.py - general helper functions.
• testvars.py - variables for the UI elements.
• selvars.py - variables for selenium runtime config for local or sauce (ex. host, port, browser)

Dev -> Testing -> Release Guidelines

See ReleaseProcess

A. Milestones should be small and cohesive.

Milestones should have the following properties:

1. Fit into a maximum of one week of work.
2. Be focused for one particular feature, e.g. altering the widget in a particular way, or adding teams.
3. Include any bugs deemed critical
4. Aim for weekly consistency - (ie dev work through Thurs afternoon, final QA on Fri.) Could be Dev week ends on a Tues and Wed is final QA day.

B. Each milestone goes through the Complete Release Process

C. Certain bug fixes can be pushed independently of the milestone

1. Some bugs are critical - either correcting major issues, or designed to prevent undesired behavior.
2. QA team should strive for 1-day turnaround time for verifying these fixes.
3. Bug can be pushed to production following the CherryPick Release Process.

D. Hudson Integration We also decided to integrate Hudson in a 2-step process:

1. Hudson 0.1: Run all unit tests on every commit.
2. Hudson 0.2: Run all Selenium tests, possibly with Sauce plugin. It's not yet clear how often we should do this.

Complete Release Process

Start of release process

When the last ticket for the Milestone is complete and nightly automated regression tests (run against dev site are successful (any errors are explainable test errors, not site errors) the release process begins.

Steps

1. Install infrastructure changes to all staging hosts and/or pcf-us-admin (Rob)
2. Staging is updated to the current milestone and the database migration is completed (by Adam)
3. QA team is notified that staging is ready for testing against current milestone (by Adam, to Dev and Testing google groups)
4. New Litmus test run is setup to capture manual and automated test results (Janet)
5. Some type of Release Notes are generated and distributed that list the high-priority changes in the Milestone (hopefully something can be pulled from Pivotal). (Holmes?)
6. Notes are posted on UniversubtitlesQA wiki for Current Release Testing (Janet)
7. QA team executes complete automated test suite on: Firefox (windows, linux, os x) (Margarita and Janet)
8. QA team executes additional automated tests (sauce subset) or alternatively manual quicktest in litmus on: (Margarita, Janet, additional interns) * GoogleChrome (if it cooperates) on Windows, Linux, OSX * Safari on os x * IE8 (9 when release) on Windows * Opera if we feel like it.
9. Blocker bugs found are added to the current milestone.

Final Release Testing

Begins when the final additional tickets for the milestone are completed, accepted and cherry-picked to milestone (staging site).

1. Add this to languages: http://www.transifex.net/projects/p/universalsubtitles/resource/ocale-en-LC_MESSAGES-django-po_0/
2. Depending on magnitude of final bugs - staging is again updated and migrated (Adam)
3. Set of Automated tests are run (sauce subset on FF and IE and Safari on Windows) (Margarita, Janet[backup])
4. Manual visual verification of partner demo pages, /pagedemo/index. (Margarita, Janet[backup])

Push to Production

1. QA gives acceptance of release on staging (email to Dev / Testing list) (Margarita, Janet[backup])
2. Adam and Rob agree on timeframe
3. Install infrastructure changes to all production hosts and/or pcf-us-admin (Rob)
4. Push to production (Adam, Rob)
5. Notification that production site is updated is sent to Dev / Testing lists) (Adam)
6. Create new production scaling AMI and run awhile in production (Rob)

Follow-up

1. High-Priority live partner pages are inspected for any issues (QA / Dean / Holmes) (we should add a list of links here.
2. High-Priority teams pages are inspected for any issues (QA / Dean / Holmes) (we should add a list of pages here).
3. Automated tests are branched and tagged to match Milestone, to preserve functionality in case of any cherry-pick releases. Any broken tests are removed or commented out of the test run, to prevent confusion later.

Cherrypick Release Process

Start of Cherrypick Release Process

Begins when a high-priority ticket is completed, accepted, and shouldn't wait for the next release milestone.

Steps

1. Ticket is completed, accepted and cherry-picked to milestone.
2. Depending on magnitude of the bug - staging is again updated and migrated (Adam)
3. Set of Automated tests are run (sauce subset on FF and IE and Safari on Windows) (Hudson, Margarita[backup], Janet[backup])
4. Manual visual verification of partner demo pages, /pagedemo/index. (Margarita, Janet[backup])

Push to Production

1. QA gives acceptance of release on staging (email to Dev / Testing list) (Margarita, Janet[backup])
2. Adam or other Developer pushes to production (Adam / Dev)
3. Notification that production site is updated is sent to Dev / Testing lists with link to piv ticket) (Adam / Dev)

Follow-up

1. High-Priority live partner pages are inspected for any issues (QA / Dean / Holmes) (we should add a list of links here.
2. High-Priority teams pages are inspected for any issues (QA / Dean / Holmes) (we should add a list of pages here).