Test Automation Architecture

Following three repos have been used to facilitate test automation for WSO2 products. Each component will describe in this post later.

1. Test Automation Framework (git repo link - carbon-platform-integration)
2. Carbon Automation Test Utils (git repo link - carbon-platform-integration-utils)
3. Collection of all test suites and a test runner (git repo link - carbon-platform-automated-test-suite)

Test Automation Framework components

Components related to TestNG is marked in dark-red colour

Automation framework engine

• Automation Context builder - Process automation.xml given though test module and make all configurations available through Automation Context.

• TestNG extension Executor -  Responsible for execution of internal/external extension classes in various states of the TestNG listeners.

Pluggable utilities to the test execution (TestNG)

There are several interfaces that allow you to modify TestNG's behaviour. These interfaces are broadly called "TestNG Listeners".  Test Automation Framework supports the execution of internal/external extension classes in various states of the TestNG listeners. Users can define the class paths in the automation.xml file under the desired TestNG listener states

Automation Framework uses Java reflection to execute classes. Users are expected to use the interface provided by the framework to develop external extension classes to be executed in the test. There are 5 interfaces provided by TAF for different TestNG listeners. Those interfaces have specific methods defined explicitly to be inline with the corresponding TestNG listeners. The interfaces are:

• ExecutionListenerExtension
• ReportListenerExtension
• SuiteListenerExtension
• TestListenerExtension
• TransformListenerExtension

Automation Framework Common Extensions (Java)

This module consists of a set of default pluggable modules common to the platform. These pluggable modules can be executed in the test execution registering those in the test configuration file (automation.xml). TAF provides common modules such as the

• CarbonServerExtension -  For server startup, shutdown operations of carbon server. And coverage    generation is also a part of this class.  
• Axis2ServerExtension -  For axis2 server startup and shutdown. Facilitate backend for integration tests.

Also extensions module contain classes facilitate third party framework integration

• SeleniumBrowserManager - Creates Selenium WebDriver instance based on the given browser configuration at automation.xml
• JmeterTestRunner - Executes jmeter test scripts in headless mode and inject result to surefire test result reports.

Users can also add any platform wide common modules into the Automation Framework Extensions. Test specific pluggable modules should be kept in the test module side and those modules can also be used for the tests by registering it in the automation configuration file.

Automation Framework Utils (Java)

Utility components that provide frequently used functionality, such as sending SOAP and REST requests to a known endpoint or monitoring a port to determine whether it's available. You can add the Utils module to your test case to get the functionality you need. Some sample utilities are:

• TCPMon Listener
• FTP Server Manager
• SFTP Server Manager
• SimpleHTTPServer
• ActiveMQ Server
• Axis2 clients
• JsonClients
• MutualSSLClient
• HTTPClient
• HTTPURLConnectionClient
• Wire message monitor
• Proxy Server
• Tomcat Server
• Simple Web Server (To facilitate content type testing)
• FileReader, XMLFileReader
• WireMonitor
• Concurrent request Generator
• Database Manager (DAO)

Test Framework Unit Tests - Unit test classes to verify context builder and utility classes, depends on TestNG for unit testing.

Carbon Automation Test Utils

Common Framework Tests

There are integration test scenarios common to all WSO2 products. Implementing tests for those common scenarios in each product might introduce test duplication and test management difficulties. Thus, a set of common tests are introduced under the framework utilities module which can be used directly by extending the test classes.

e.g

1. DistributionValidationTest
2. JaggeryServerTest
3. OSGIServerBundleStatusTest
4. ServerStartupBaseTest

Common Admin Clients

This module consists of set of admin clients to invoke backend admin services together with supportive utility methods which are common for WSO2 product platform.

e.g

• ServerAminClient
• LogViewerAdminClient
• UserManagementClient
• SecurityAdminServiceClient

Common Test Utils

Frequently used test utility classes which depends on platform dependencies.

e.g

• SecureAxis2ServiceClient
• TestCoverageReportUtil - (To merge coverage reports)
• ServerConfiurationManager (To change carbon configuration files)
• LoginLogoutClient

Common Extensions

Consists of Pluggable classes allowing to plug additional extensions to the test execution flow. Frequently used test framework extension classes which are available by default, these classes use platform dependencies.

e.g UserPopulatorExtension - For user population/deletion operations

Platform Automated Test Suite

Builds a distribution containing all integration and platform test jars released with each product. It contains an ant based test executor to run the test cases in each test jar file. This ant script is based on TestNG ant task which can be used as a test case runner. The ant script contains a mail task which can be configured to send out a notification mail upon test execution. The PATS distribution can be used to run the set of tests on a pre-configured product cluster/distribution.

How to capture popup windows using selenium.

I am going to discuss on selenium popup handling specifically with selenium RC. I was searching though internet for a while to find a solution for processing of popups though selenium.But I was unable to find best fit answer for my test scenario. I wanted to capture pop-up window do some processing then close the popup and get back to parent window again.Anyway by following the reference guides and other available materials I was able to implement the the above mentioned test scenario.

As per my understanding selenium IDE supports capturing popup windows but get back to parent window is not supported. I tried several times and failed to set focus to parent window again. Selenium API doc says, in order to select parent window again, use selectWindow() methods with null parameter. But that didn't work for me. selenium RC was fail to select the parent window again. There are several API methods available in selenium to detect required references for popup windows. Those are

selenium.selectWindow();
selenium.getAllWindowIds();
selenium.getAllWindowNames();
selenium.windowFocus();
selenium.waitForPopUp();
selenium.close();

In my case,I have followed implementation give below to write the test scenarios in junit.

click the popup window link

selenium.click("link=Feed");

Get the pop up window ID.

String feedWinId = selenium.getEval("{var windowId; for(var x in selenium.browserbot.openedWindows ) {windowId=x;} }");

Select the popup window.In order to select it, you must somehow identify it. Various identification methods are available with selectWindow() function see http://release.seleniumhq.org/selenium-remote-control/0.9.0/doc/java/ for more details

selenium.selectWindow(feedWinId);

Set focus to popup window.

selenium.windowFocus();

//Then do some proccssing

close the popup

selenium.close();

select parent window again. You need to know the window title or name

String[] winFocus;
String winTitle;
winFocus = selenium.getAllWindowTitles();

If there is more than one open windows you have to iterate though winFocus array and find the correct window to set the focus.

winTitle = winFocus[0];

if (winTitle.equalsIgnoreCase("WSO2 Management Console")) {
selenium.selectWindow(winTitle);
}

If you need to wait for the popup then use

selenium.waitForPopUp(winID, time-ms);

I hope above instructions will help you in dealing with pop ups. Please feel free to comment and ask questions

Source code of the test : https://wso2.org/repos/wso2/branches/commons/qa/web-test-framework/2.0.2/registry/src/test/java/org/wso2/carbon/web/test/registry/FeedsTest.java