JMockit An automated testing toolkit for Java

Measuring code coverage

  1. The coverage metric
  2. Types of coverage output
    1. Call points
  3. Configuring the coverage tool
  4. Aggregated reports for multiple test runs
    1. Generating an aggregate report from multiple data files
    2. Generating an aggregate report from a single data file appended after each test run
  1. Checking minimum coverage
  2. Activating coverage in a Maven project
    1. Including the HTML report in a Maven site
  3. Turning coverage off

Code coverage consists of a set of software metrics that can tell you how much of the production code is covered by a given test suite. It's purely quantitative, and does not say anything about the quality of either the production code or the test code. That said, the examination of code coverage reports will sometimes lead to the discovery of unreachable code which can be eliminated. But more importantly, such reports can be used as a guide for the discovery of missing tests. This is not only useful when creating tests for existing production code, but also when writing tests first, such as in the practice of TDD (Test Driven Development).

1 The coverage metric

The coverage metric produced by the tool tells us how much of the executable code in a source file has been exercised by tests, and also how many of the instance and static non-final fields were fully exercised by the test run.

Each executable line of code can be uncovered, covered, or partially covered. The third case can happen, for example, with lines of code containing multiple logical conditions in a complex boolean expression. The tool identifies all three cases, computing the coverage percentage for each executable line of code accordingly: 0% for an uncovered line, 100% for a covered line, or some value in between for a partially covered line.

Regarding fields, to be fully exercised, each must have the last value assigned to it read by at least one test.

The coverage percentage for a source file is calculated as 100 * (NE + NFE) / (NS + NF), where NS is the total number of line segments, NF the number of non-final fields, NE the number of executed segments, and NFE the number of fully exercised fields.

The percentage for a package, in turn, is calculated from the total and covered numbers of segments and fields in the whole set of source files belonging to the package. Finally, the total code coverage percentage is computed by the same formula on the totals for all packages.

2 Types of coverage output

The JMockit Coverage tool can generate the following types of output:

  1. HTML reports: a multi-page HTML report is written in the "coverage-report" directory, under the current working directory (a different output directory can be specified if needed). The directory is created if it doesn't yet exist; its contents are overwritten if previously generated. The report will include pages containing all Java source files covered by the test suite. By default, the tool looks for ".java" source files inside all directories of name "src" found directly or indirectly under the current working directory; any intermediate sub-directories between "src" and the top-level package directory, such as "src/java" for example, are also searched.
  2. Coverage data files: a single serialized file of name "coverage.ser" is written under the current working directory or a specified output directory. If the file already exists, its contents are either overwritten or appended with the in-memory results of the current test run, as specified.
    These files can be read and processed by external tools. The method will create a new CoverageData instance with all the coverage data available in a given serialized file.

2.1 Call points

When running a test suite with the coverage tool, there is optional "call point" information which can be gathered, as selected by the user. A call point is the point in the source test code from which a specific line of production code was exercised.

Generating coverage with this extra information takes more time and produces significantly larger output; on the other hand, it can be useful to know which lines of test code caused a given line of production code to be executed during the test run. When included in the HTML report, the list of call points appears hidden at first but can be easily viewed by clicking on each executable line of code.

3 Configuring the coverage tool

To activate the Coverage tool in a JUnit/TestNG test run, use the -javaagent JVM initialization parameter, and specify at least one of the "coverage-output" and "coverage-classes" system properties. (See Running tests with JMockit for more details.) All aspects of the tool's behavior can be configured by setting one or more of the following properties.

  1. coverage-output: one or more comma-separated values between html, html-cp ("cp" = "call points"), html-nocp, serial, and serial-append, which select the kind of output to be generated at the end of the test run. The default is to generate the HTML report with no call points (html = html-nocp).
    The "html", "html-cp" and "html-nocp" values are mutually exclusive, just like "serial" and "serial-append". However, it is valid to have one of each pair specified at the same time. In such a case, at the end of the test run both kinds of output will be written.
    The presence of "serial" or "serial-append" causes a serialized data file of name "coverage.ser" to be generated; with "serial-append", coverage data gathered by the current test run will be appended to the contents of a previously existing data file (if said file doesn't exist, it has the same effect as "serial").
  2. coverage-outputDir: absolute or relative path to the output directory, to be used for writing any "coverage.ser" or "index.html" files (plus the remaining ".html" files of the HTML report, in automatically created sub-directories). By default, the current working directory of the running JVM is used, with all HTML pages written into a "coverage-report" sub-directory.
  3. coverage-srcDirs: comma-separated list of Java source directories to be searched when generating an HTML report. (This is not relevant for the serialized data file.) Each directory is specified by an absolute or relative path. If no such directory is specified, all "src" directories under the current working directory are searched. If, however, the property is set to the empty string, then no source files are searched, and only the index.html file is generated.
  4. coverage-classes: Either an OS-like regular expression (with the typical "*" and "?" wildcards), or a java.util.regex-conformable regular expression. The given expression will be used to select the classes (by fully qualified name) from production code which should be considered for coverage. By default, all classes in production code loaded during the test run and which are not inside jar files are considered.
    For example, "some.package.*" selects all classes under some.package or any sub-package.
    As a special case, if the property is specified as "loaded", then all classes will be considered, but only those which get loaded by the JVM during the test run; classes that are part of the codebase but never get loaded are left out. This is very useful when the test run includes only a few tests, targeting only a subset of the codebase.
  5. coverage-excludes: The same as the previous property, but for class names which should be excluded from consideration when instrumenting classes for coverage. This property can be used together with coverage-classes or on its own. By default, no classes between those selected for coverage are excluded from consideration.
  6. coverage-check: one or more semicolon-separated rules specifying minimum coverage checks to be performed at the end of a test run. By default, no such checks are performed. For details, see the Checking minimum coverage section.

Note that you should be able to easily specify these properties inside a Maven surefire plugin configuration, or a test run configuration for your Java IDE of choice, using either JUnit or TestNG; no JMockit-specific plugin is needed.

4 Aggregated reports for multiple test runs

When the coverage tool generates a report at the end of a test run, it always overwrites any previous report. Normally, the coverage data from which the report is generated reflects only what was gathered during the current test run. Now suppose you have multiple test suites or test run configurations, and you want to generate a single aggregated HTML report for the code covered by the full set of tests. Here is where the "coverage.ser" serialized data files come in.

To activate the generation of these files, we simply set the coverage-output system property to a value containing "serial" or "serial-append". As these two values suggest, there are different ways to combine multiple coverage data files. The following sub-sections provide the details for each case.

4.1 Generating an aggregate report from multiple data files

Suppose we want to gather coverage data from multiple test runs and later generate an aggregate HTML report merging together the results from all test runs. Each test run needs to generate its own coverage.ser file, so that later they can be merged together in a final step which produces the report; therefore, each test run should be configured with "coverage-output=serial". Note that, in order to preserve the original coverage.ser output files generated by each test run, they will need to be written or copied into different output directories.

Assuming that two or more coverage.ser files are available in separate directories, an aggregate report can be generated from them by executing the mockit.coverage.CodeCoverage.main method (a regular Java "main" method). To facilitate this, the jmockit-1.x.jar file is executable. As an example, the following Ant task could be used:

<java fork="yes" dir="myBaseDir" jar="jmockit-1.x.jar">
   <jvmarg line="-Dcoverage-output=html"/>
   <arg line="module1-outDir anotherOutDir"/>

The example above uses "myBaseDir" as the base directory where a separate JVM instance will run. Two output directories containing "coverage.ser" data files are specified, as command line arguments. Other configuration parameters can be specified through the "coverage-xyz" system properties. This separate JVM instance will read each of the "coverage.ser" data files, merge the coverage data in memory, and then generate the aggregate HTML report before exiting.

4.2 Generating an aggregate report from a single data file appended after each test run

Another way to obtain an aggregate coverage report from the execution of multiple test runs is to accumulate coverage data from all tests into a single data file. This can be achieved by using the same working directory for all test runs or by pointing coverage-outputDir to a shared directory, while having coverage-output=serial-append for each test run. Additionally, the last test run in the sequence should also specify html or html-nocp for the coverage-output property, together with serial-append. Naturally, the first test run must not read data from this file; therefore, either the file should be deleted before the first test run, or ignored by having the first test run use coverage-output=serial.

So, the difference between output modes "serial" and "serial-append" is that with the first we have multiple "coverage.ser" files (each in a different directory used by a separate test run), while with the second we share a single data file between all test runs.

5 Checking minimum coverage

If desired, JMockit Coverage can check that the final coverage percentages at the end of a test run satisfy arbitrary minimum values. Such checks can be specified through one or more checking rules assigned to the "coverage-check" system property (when more than one, they must be separated by ";" characters).

Each checking rule must be in the form "[scope:]min percentage". There are three types of scopes:

All checks (if any) are performed at the end of the test run (at JVM shutdown, actually). Other forms of output (HTML report, serialized file) are not affected. When an individual check fails, a descriptive message is printed to standard output. If one or more checks have failed, two final actions are taken to have the fact reported: first, an empty file of name "coverage.check.failed" is created in the current working directory; second, an error (specifically, an AssertionError) is thrown. When checks are performed but they all pass, the "coverage.check.failed" file, if present in the current directory, is deleted.

The use of a file to mark the success or failure of coverage checks is meant to allow build tools to react accordingly, typically by failing the build when the file is present. For example, we can do the following in a Maven pom.xml file:


6 Activating coverage in a Maven project

With Maven, the surefire plugin is the one usually responsible for running tests. To activate and configure the coverage tool, specify values for the appropriate "coverage-xyz" system properties.


         <!-- Coverage properties -->
         <!-- At least one of the following needs to be set: -->
         -Dcoverage-output=html             <!-- or html-cp, serial, serial-append; if not set, defaults to "html" -->
         -Dcoverage-classes=loaded          <!-- or a "*" expression for class names; if not set, measures all production code classes -->

         <!-- Other properties, if needed: -->
         -Dcoverage-outputDir=my-dir        <!-- default: target/coverage-report -->
         -Dcoverage-srcDirs=sources         <!-- default: all "src" directories -->
         -Dcoverage-excludes=some.package.* <!-- default: empty -->
         -Dcoverage-check=80                <!-- default: no checks -->

6.1 Including the HTML report in a Maven site

To have the JMockit Coverage HTML report included in the generated Maven site documentation, the src/site/site.xml descriptor file needs to be provided, with contents similar to what's shown below.

<?xml version="1.0" encoding="UTF-8"?>
      <menu ref="reports"/>
         <item name="Code Coverage Report" href="../../coverage-report/index.html"/>

7 Turning coverage off

To temporarily turn coverage off for a particular test run, we can set the coverage-output system property to an unknown output format, such as "-Dcoverage-output=none". The same effect will be achieved by setting "-Dcoverage-classes=none".

Another, more interactive, way is to manipulate the read-only attribute of the relevant output file, when one has already been generated. The particular file to be manipulated, always in the working directory, is "coverage.ser" for serialized output or "coverage-report/index.html" for HTML output. The file attribute is checked by JMockit at startup; when marked as read-only it cannot be overwritten, so JMockit avoids the attempt entirely. Note that the working directory can usually be selected separately for each test run configuration in the Java IDE. Also, a Java IDE usually provides an easy mechanism to toggle the read-only status of a file in the project: in IntelliJ IDEA it is done by double clicking the status bar, with the desired file opened in the editor; in Eclipse there is a "Read only" check box in the "Properties" screen (which can be opened by typing "Alt+Enter") for the text file selected in the editor.