Test Automation Architecture

Following three repos have been used to facilitate test automation for WSO2 products. Each component will describe in this post later.

1. Test Automation Framework (git repo link - carbon-platform-integration)
2. Carbon Automation Test Utils (git repo link - carbon-platform-integration-utils)
3. Collection of all test suites and a test runner (git repo link - carbon-platform-automated-test-suite)

Test Automation Framework components

Components related to TestNG is marked in dark-red colour

Automation framework engine

• Automation Context builder - Process automation.xml given though test module and make all configurations available through Automation Context.

• TestNG extension Executor -  Responsible for execution of internal/external extension classes in various states of the TestNG listeners.

Pluggable utilities to the test execution (TestNG)

There are several interfaces that allow you to modify TestNG's behaviour. These interfaces are broadly called "TestNG Listeners".  Test Automation Framework supports the execution of internal/external extension classes in various states of the TestNG listeners. Users can define the class paths in the automation.xml file under the desired TestNG listener states

Automation Framework uses Java reflection to execute classes. Users are expected to use the interface provided by the framework to develop external extension classes to be executed in the test. There are 5 interfaces provided by TAF for different TestNG listeners. Those interfaces have specific methods defined explicitly to be inline with the corresponding TestNG listeners. The interfaces are:

• ExecutionListenerExtension
• ReportListenerExtension
• SuiteListenerExtension
• TestListenerExtension
• TransformListenerExtension

Automation Framework Common Extensions (Java)

This module consists of a set of default pluggable modules common to the platform. These pluggable modules can be executed in the test execution registering those in the test configuration file (automation.xml). TAF provides common modules such as the

• CarbonServerExtension -  For server startup, shutdown operations of carbon server. And coverage    generation is also a part of this class.  
• Axis2ServerExtension -  For axis2 server startup and shutdown. Facilitate backend for integration tests.

Also extensions module contain classes facilitate third party framework integration

• SeleniumBrowserManager - Creates Selenium WebDriver instance based on the given browser configuration at automation.xml
• JmeterTestRunner - Executes jmeter test scripts in headless mode and inject result to surefire test result reports.

Users can also add any platform wide common modules into the Automation Framework Extensions. Test specific pluggable modules should be kept in the test module side and those modules can also be used for the tests by registering it in the automation configuration file.

Automation Framework Utils (Java)

Utility components that provide frequently used functionality, such as sending SOAP and REST requests to a known endpoint or monitoring a port to determine whether it's available. You can add the Utils module to your test case to get the functionality you need. Some sample utilities are:

• TCPMon Listener
• FTP Server Manager
• SFTP Server Manager
• SimpleHTTPServer
• ActiveMQ Server
• Axis2 clients
• JsonClients
• MutualSSLClient
• HTTPClient
• HTTPURLConnectionClient
• Wire message monitor
• Proxy Server
• Tomcat Server
• Simple Web Server (To facilitate content type testing)
• FileReader, XMLFileReader
• WireMonitor
• Concurrent request Generator
• Database Manager (DAO)

Test Framework Unit Tests - Unit test classes to verify context builder and utility classes, depends on TestNG for unit testing.

Carbon Automation Test Utils

Common Framework Tests

There are integration test scenarios common to all WSO2 products. Implementing tests for those common scenarios in each product might introduce test duplication and test management difficulties. Thus, a set of common tests are introduced under the framework utilities module which can be used directly by extending the test classes.

e.g

1. DistributionValidationTest
2. JaggeryServerTest
3. OSGIServerBundleStatusTest
4. ServerStartupBaseTest

Common Admin Clients

This module consists of set of admin clients to invoke backend admin services together with supportive utility methods which are common for WSO2 product platform.

e.g

• ServerAminClient
• LogViewerAdminClient
• UserManagementClient
• SecurityAdminServiceClient

Common Test Utils

Frequently used test utility classes which depends on platform dependencies.

e.g

• SecureAxis2ServiceClient
• TestCoverageReportUtil - (To merge coverage reports)
• ServerConfiurationManager (To change carbon configuration files)
• LoginLogoutClient

Common Extensions

Consists of Pluggable classes allowing to plug additional extensions to the test execution flow. Frequently used test framework extension classes which are available by default, these classes use platform dependencies.

e.g UserPopulatorExtension - For user population/deletion operations

Platform Automated Test Suite

Builds a distribution containing all integration and platform test jars released with each product. It contains an ant based test executor to run the test cases in each test jar file. This ant script is based on TestNG ant task which can be used as a test case runner. The ant script contains a mail task which can be configured to send out a notification mail upon test execution. The PATS distribution can be used to run the set of tests on a pre-configured product cluster/distribution.