Testing is one of the most important parts of software development. And Android applications are not an exception. When writing code, you should review it and think how it will be tested afterwards.
Let’s imagine a situation where you need to cover your fully completed project with tests. It’s not that simple. Probably, your code wasn’t implemented to be easily testable. This means that you need to change it and make it testable… without damaging any functionality (actually, nowadays it’s called refactoring). But can you confidently say that these changes didn’t harm the code if it didn’t undergo thorough testing? Doubtful.
I think tests, that is to say unit tests, have to be written by the developer himself, because only the developer knows how and what he has coded.
As far as Android unit testing is concerned, Google offers pretty nice tools for testing, but there is not much you can do with them. Often the JUnit framework is used for testing, but, of course, it has its pros and cons.
JUnit is used for unit testing, which allows you to check the correctness of individual modules of the application source code. The advantage of this approach is in the isolation of an individual module from the others. In this case, the purpose of such method allows the developer to ensure that the module itself works correctly. JUnit is a class library.
Unfortunately, anyone who worked with this framework at least once noticed that it is completely unsuitable for GUI testing. A sign of good manners is code that was covered with tests and has a test coverage report with the percentage of code covered.
At my latest projects, my task included writing different kinds of tests – from load to GUI, and I wanted to share my experience of working with different frameworks and tools, their features and peculiarities, and to discuss how to create these test reports. But first, let’s talk about basic frameworks.
Above you can see an approximate picture of using basic frameworks. Which one to choose is up to you: some use Robotium because they don’t want to meddle with the source code, some use Espresso due to its intuitive interface. I use Spoon.
Spoon
Android Spoon is a framework that allows making screenshots from a device or an emulator during test execution and uses them to generate reports. Besides adding screenshots to reports, Spoon also adds test runner log and, in case a test failed, displays the full stack-trace (really handy). To create a test report:
- Download Spoon-client and Spoon-runner.
- Copy Spoon-client into the libs folder of the test project.
- In that very project, create a folder (for example, spoon).
- Copy Spoon-runner into the created folder.
- Create a .bat file with the following script:
call java -jar spoon-runner-1.1.1-jar-with-dependencies.jar
-- apk Path\to\your\project \bin\project.apk
-- test-apk Path\to\your\test-project \bin\tests.apk
In sub-parametres, you can indicate a filter for tests, for example, to run only tests marked @Medium, add — size medium to the script.
Now the script will look like this:
call java -jar spoon-runner-1.1.1-jar-with-dependencies.jar
-- apk Path\to\your\project \bin\project.apk
-- test-apk Path\to\your\test-project \bin\tests.apk
-- size medium
You can view the full list of sub-parameters on the official Github page of Spoon.
Thant’s it. Now every time you need to write tests, just insert where needed
Spoon.screenshot(activity, "state_changed"),
where the second argument is the text that will be highlighted above the screenshot. One more feature of Spoon is that it uses regular expressions, and when you use space in signatures, it throws an exception. When you run an emulator or plug in a device and run the bat file, this is the report you will see if everything works correctly:
You can find an official example of a report from the developers of Spoon here.
Looks good, doesn’t it? Another advantage of Spoon is that it runs tests simultaneously on all connected devices, i.e. it will gather results from all devices into a single report. The one but significant drawback of Spoon is that it doesn’t make screenshots of dialogs. And it doesn’t create reports about the test coverage of code. Well, let’s fix this.
Emma
You have to admit, this report looks at least decent
An example of code, covered with tests:
An example of code partially covered with tests:
EMMA is an open-source toolkit for measuring code coverage for Java code and reporting about it. This tool is built into Android SDK. Its key features are:
- Classes inspection support for both offline mode (before running) and “on the fly”.
- Types of code coverage supported: package, class, method, line and the main unit. It is also possible to detect when a source line was covered only partially.
- Report formats supported: txt, HTML, XML.
Test Building Using Ant
Apache Ant is a tool for converting development structures in a deployment construction. It is declarative, and all the command-line instructions that are used to deploy the application are presented in simple XML-elements. Find more details here.
To describe the procedure of project building, you will need a main project (MyProject) and a project with tests for it (MyProjectTests). You can read test creation rules here.
First of all, when building a project using Ant, you need to build projects used in the application as libraries. If there are no – skip this step. For example, if the project uses libraries like «google_play_service_lib», you should do the following:
in the command line, go to the folder with installed sdk\tools (for example, D:\android\adt-bundle\sdk\tools) and execute:
android update lib-project -p MyLibProject
(MyLibProject is the path to the lib, used in the project)
As a result, build.xml should appear in the project root, and the console should show the following message:
Updated local.properties
Updated file D:\Workspace\MyProject\build.xml
Updated file D:\Workspace\MyProjectTests\proguard-project.txt
After building all libraries, you need to build the main project. For this, execute the following in the same folder:
android update project -p MyProject
(MyProject is the path to the working branch of the project)
Of course, AndroidManifest.xml should be in this folder. The script will again generate build.xml and will build the main project. You can enter the name of the project using sub-parameters to make it easier to use in the future:
android update project -p MyProject -n NameForProject
How does building a project with tests work? Looks the same and pretty simple – here is the script for that:
android update test-project -m ..\MyProject -p \MyProjectTests
(MyProject is the path to the main project, and MyProjectTests is the path to the project with tests)
Everything is ready!
By the way, at this step developers may encounter issues due to the use of libraries. For example, in a project different jar-libraries can be built on the basis of libraries used in other libraries. Ant doesn’t know what to do with them and, as a result, an error occurs during building process. So, if you use the same library in the project, it can cause an error.
To start calculating how much code was covered with tests, add a sub-parameter “emma” in the script. Before start, you need to run an emulator or connect a device. In the top branch of the project with tests, run this script in command line:
ant clean emma debug install test
During test execution, Emma generates a coverage.em file in the bin folder of the main project (metadata). On test completion, it sets necessary permissions, creates the coverage.ec file in the folder of the installed project, copies these two files into the bin folder of the test project and, based on them, generates a report in the same folder.
Summary
Look at the test, even if they are written very well. Can you answer by looking at them how much code they cover? Is it enough for you to see only the green strip that shows the results of your tests, as a visual report about all of them? I think not. Using reports after tests execution indicates the competence of the specialist, and when it comes to test automation, even more. Writing scripts can take some time, but, believe me, it is worth it!
