Make the Most of Your Test Report

Taras Lytvyn
Automation QA Competence Expert

Thanks to techniques like automated testing and continuous integration, software can be developed at a high standard, easily packaged and deployed to test environments. Clearly, automated testing clings to an integral part of the software development process. But are we satisfied with our automation test results? Are they informative enough? Do we have to rerun failed tests to be sure the failure occurred by a test that is stuck, not a functionality bug itself?

You don’t have to worry about the accuracy of your test results if you have descriptive test reports with each test step documented and even screened, with logs attached to it and other helpful information, like exceptions during execution, present in your report.

With modern automation tools and 3rd party libraries, we can automate testing of any application type and even web services. Talking about web testing – I definitely prefer Selenium, Appium for mobile devices and some 3rd party libraries like Rest Assured or Http Client for web services. In this article I’ll dig into a list of tools for test reporting and figure out the best choices.

xUnit Frameworks Reports

This is the kind of results we receive if we use simple xUnit frameworks for test reporting in automation tests

JUnit:
Simple report example:
Automated_Testing_Reports_ELEKSlabs1

TestNg:

Emailable report example:
Automated_Testing_Reports_ELEKSlabs2_1

Simple report example:
Automated_Testing_Reports_ELEKSlabs3_1

Looks unclear and boring, huh? Yes, you can use the ReportNG plugin and have a more appealing UI for test results, but it is still rather primitive and not worth a damn.

ReportNg example:
Automated_Testing_Reports_ELEKSlabs4_1

Of course you can simply fine-tune the TestNg report while creating your own custom version with some HTML injections using Reporter.log():

/**
  * Reports the detailed test step with step summary and details
  *
  * @param stepText the test step summary text
  * @param stepOutput the step output text
  */
 public static void report(String stepText, String stepDetails) {
  Reporter.log("<b>" + stepText + "</b>" + "&nbsp" + stepDetails + "<p>", true);
 }

And use it inside your test DSL methods:

/**
  * performs search of a text on ask search page.
  *
  * @param searchText the search text
  * @return the ask search result Page Object
  */
 public AskSearchResult searchInAsk(String searchText){
  searchTextBox.sendKeys(searchText);
  findAnswersButton.click();
  TestStepReporter.report("search for text:", searchText);
  webDriver.manage().timeouts().pageLoadTimeout(10, TimeUnit.SECONDS);
  return PageFactory.initElements(webDriver, AskSearchResult.class);
 }

You can also adjust the reporter with the screenshot functionality in case you tests are UI-based (web, mobile, desktop).

Here is a customized TestNg report example with steps and screenshot for each failed test:
Automated_Testing_Reports_ELEKSlabs5

Still looks dull? To make your reporter more flexible and enhance it with useful and valuable extensions, you can lean toward TestNG listener classes and interfaces.

Reporting alternatives: ATU

Alternatively, for web tests using Selenium, WebDriver ATU reporter works fine. This reporting tool seems to be quite trivial in its integration and uses similar approaches as the TestNg custom reporter.

ATU report example:
Automated_Testing_Reports_ELEKSlabs6

First, you need to add some already created listeners to your test classes and then report your steps and screenshots with special methods provided by ATU.

Reporting alternatives: Thucydides

The Thucydides framework has become a new buzzword in automated acceptance testing. It uses acceptance tests to generate detailed/comprehensive documentation and reports about your product and project. It works with both JUnit and BDD tools like JBehave and Cucumber-JVM, providing strong support for automated web testing through tight WebDriver integration. The great thing is that Thucydides can be applied within any approach described above.

If you use the approach offered by the Thucydides creators, you’ll get more descriptive and informative reports while using one of ATDD-based approaches during the tests creation process.

Thucydides suggested approach:
Automated_Testing_Reports_ELEKSlabs7

Here are some examples of Thucydides reports:
Automated_Testing_Reports_ELEKSlabs8

Automated_Testing_Reports_ELEKSlabs9

As seen from the images, you can easily navigate between your features, monitor test coverage for each feature, navigate between separate tests and their steps and many more.

WebDriver can be effortlessly integrated with Thucydides, providing you with screenshots for each test step. To make the most of this powerful tool, use it with the BDD framework and Gherkin Given-When-Then syntax.

Exhaustive reporting is possible. Allure Framework

But what if you want to add more attachments to your test report including customized screenshots, log files, text files with predefined test data, etc.? The right choice inevitably goes to Allure. The Allure framework is a flexible, lightweight multi-language test report tool with the possibility to add:

  • Steps
  • Attachments
  • Features and Stories
  • Parameters
  • Environment
  • Issues

Here are some Allure report examples:
Automated_Testing_Reports_ELEKSlabs10

Automated_Testing_Reports_ELEKSlabs11

Automated_Testing_Reports_ELEKSlabs12_2

With Allure all the features and stories, test scenarios and their steps can be highlighted. The great thing that distinguishes Allure from other report solutions is its attachment functionality. You can attach String or byte array data and review it inside the reporter. This data can be customized depending on your needs.

For example, you would like to have a screenshot of each step with a highlighted element that was interacted during a test step. You can write your own screenshot method and attach the results of its execution to your report as a byte array while pointing the attachment type image/png.

/**
  * attaches separate web element screenshot image
  * @param driver driver
  * @param element element
  * @param screenShotStepFilter screenshot step filter (redline or blur)
  * @return byte array of element's screenshot image
  */
 @Attachment(value = "A step screenshot with highlighted element: ")
 static byte[] takeWebElementScreenshotImage(WebDriver driver,
   WebElement element, String screenShotStepFilter) {
  return TestListenerUtil
    .getByteArrayFromImage(
      takeWebElementScreenshot(driver, element,
        screenShotStepFilter), "png");
 }

Automated_Testing_Reports_ELEKSlabs13_1

You can also set the attachment content to the String and attach it as text/plain. This can be used for test logs, as in the example below.

/**
  * attach log execution file to test allure report
  * @param testClassName test class name
  * @param testName test method name
  * @return String content to attach
  */
 @Attachment(value = "Test log during execution for \"{1}\"", type = "text/plain")
 static String attachLogToReporter(String testClassName, String testName) {

Automated_Testing_Reports_ELEKSlabs14_1

Conclusion

No matter what test types you are utilizing – unit, system or functional, or how you are automating them – through automation tools or 3rd party libraries, the point is in comprehensive, flexible reporting that should help increase the quality of an application under testing conditions. While following the ATDD best practices, we can simply switch to ready-made tools such as Thucydides. You never know when a defect occurs, a test sticks or an unhandled exception appears. With Allure attachment and step functionalities, you will always be in the loop.

You can learn more about our expertise in Automated Testing and delivering solutions with Ranorex on top.

tags

Comments