Skip to main content

EJUnit

EJUnit is a JUnit 4 (4.12) runner for unit tests in Java. It builds on ideas from several existing projects (JUnitXmlFormatter, Schmant) while adding the following features:

  • Generation of result reports in the TREX format
    • Distinction between failures (asserts) and errors (runtime exceptions)
    • Handling of timeouts caused by infinite loops, allowing a complete report to be generated even when individual tests do not terminate
  • Isolated execution of JUnit test cases in separate JVMs, allowing the remaining tests to continue even if one test case enters an infinite loop (similar to this Maven plugin)
  • Per-test bytecode instrumentation with JaCoCo
  • Generation of per-test coverage reports in the CEXF format for JaCoCo and OpenClover

Build​

Build the runner with ./gradlew build. The generated jar files are then located under build/libs.

Test Execution​

Tests can be executed in two different modes, which will be described in the following sections. Both modes support source code instrumentation. When using JaCoCo, .exec files are generated for each test case in the working directory during test execution. The JaCoCo agent must be configured separately and is available as jacocoagent.jar in build/libs after the build. When using OpenClover, the source code must be instrumented before test execution using the ejunit-openclover.jar tool. Examples on how to use the runner and these tools can be found in evaluators/shared/java.

Mode: Mono​

For running all test cases in a single JVM.

usage: ejunit-mono [OPTIONS] [selector1 [selector2 [selector3] ...]]
Executes all JUnit test methods in the test classes identified by the
given selectors in the SAME JVM AS THE TEST RUNNER. A selector is either a
FQCN of a test class (all test methods) or an identifier in the format
<testclass >#<testmethod> (single test method). Test classes need to be on
the class path with the code under test.

OPTIONS:
 -jacoco,--with-jacoco        Enable per-test coverage instrumentation
                              with JaCoCo (online). The jacocoagent.jar
                              needs to be configured using the -javaagent
                              JVM flag. Per-test coverage data will be
                              saved in the working directory after each
                              test in the following format:
                              <testclass>#<testmethod>.exec
 -o,--out <arg>               Output path for the resulting TREX report.
 -s,--max-output-size <arg>   Maximum number of characters to include in
                              stdout/stderr outputs from testcases before
                              truncation. If the output is larger, only
                              the last ARG characters are kept and a
                              truncation notice is added to the output.
                              Defaults to 4096.

Mode: Isolated Execution​

info

This tool was a prototype and is no longer used in its current form. There are various issues when the process for coordinating test execution runs in the same container as the test execution itself (resource starvation, a largely redundant Docker image, inconsistent CPU and memory usage, ...). The same functionality is now provided by the JavaIE evaluator, which runs outside the Docker container.

For running each test case in its own JVM.

usage: ejunit-ie-master [OPTIONS] [class1 [class2 [class3] ...]]
Executes all JUnit test methods in the given test classes identified by
their FQCNs in SEPARATE JVMs. Test classes need to be on the class path
with the code under test.

OPTIONS:
 -agent,--jacoco-agent <arg>   Path to the jacocoagent jar file for
                               enabling per-test coverage instrumentation
                               with JaCoCo (online). Per-test coverage
                               data will be saved in the working directory
                               after each test in the following format:
                               <testclass>#<testmethod>.exec
 -o,--out <arg>                Output path for the resulting TREX report.
 -t,--timeout <arg>            Execution timeout (per testcase) in
                               seconds.
 -w,--worker <arg>             Path to the IE worker jar file.

If all test methods in the given test classes have finished, exit with
status 0. This is independent from the result of the test case, meaning
that even if test failures or errors occur, the status is 0. If an
unexpected exception occurs in the test runner itself, return status 1.

In contrast to the mono runner, test methods are executed in their own
JVM. Test methods are executed sequentially (not in parellel). For each
test method, a JVM processes is started and waited for a result. If the
JVM does not terminate after the configured timeout, the JVM process is
killed. This will ensure that test methods running indefinitely will not
prevent other test methods from being executed. Thus, this process may run
for a total time = timeout * number of test methods in the test classes.

There is a significant overhead due to the start of a separate JVM for
each test method, which slows down test execution in comparison to the
mono runner.

JVM processes are started with the -XX:MaxRAMPercentage=50 flag (to enable
comparison with the mono runner) and inherit some flags from the spawning
JVM (-ea, --enable-preview).

Extracting Test Methods from Test Classes​

usage: ejunit-ls [OPTIONS] [class1 [class2 [class3] ...]]
Extracts all test methods from the given test classes as testcase
identifiers. Test classes need to be on the class path with the code
under test.

OPTIONS:
 -o,--out <arg>   Output path for writing the testcase identifiers.
                  Testcase identifiers are written in the format
                  <testclass>#<testmethod> into a text file with one
                  identifier per line.

Generating Coverage Reports​

After the tests have been run on the instrumented code with ejunit-mono or ejunit-ie-master, the CEXF coverage report can be generated. For JaCoCo, the report generator is located under cexf/jacoco:

usage: ejunit-jacoco [OPTIONS] directory
Parses all *.exec files generated by JaCoCo in the given directory and
                     generates a CEXF report.
OPTIONS:
 -ff,--file <arg>    FQCN of a class file for filtering code coverage. The
                     class needs to be on the class path. If no class with
                     the specified FQCN is found on the class path, the
                     report will be empty.
 -lt,--ltype <arg>   Code coverage granularity to include in the report.
                     One of the following: instr, cond, line. If not
                     specified, all types will be included in the report.
 -o,--out <arg>      Output path for the resulting CEXF report.

For OpenClover, the generator can be found at cexf/openclover:

  USAGE: ejunit-openclover [OPTIONS] PARAMS

  PARAMS:
    -i, --initstring <string> clover initstring

    -ff,  --file <string> FQCN of a class file for filtering code coverage.
    In contrast to the JaCoCo report generator, the class is NOT
    required to be on the class path.

  OPTIONS:
    -o,  --outfile <dir> the file to write report to.

    -lt,  --ltype <string> Code coverage granularity to include in the report.
    One of the following: instr, cond, line, stmt, method, class.
    If not specified, all types will be included in the report.

Debugging with Gradle​

The :mono:runSample and :ie:master:runSample tasks can be used to run the tests for a sample solution in the respective mode with JaCoCo instrumentation being enabled. The test classes (and the classes under test) must be located in the samples subproject and are automatically added to the classpath when the task is run with Gradle. After execution, a CEXF report can be generated with the :cexf:jacoco:generate task.

Example:

./gradlew :mono:runSample -PappArgs='sample.PingTest'
./gradlew :cexf:jacoco:generate # parses all *.exec files in the current directory

Test execution with OpenClover cannot (yet) be debugged. However, it can be tested with the test.sh bash script:

cli/openclover/test.sh -ff sample.Ping sample.PingTest

Reports in various formats, including the CEXF report, are then located under cli/openclover/build/report. The generation of the CEXF report can be run with Gradle as follows:

./gradlew cexf:openclover:report -PappArgs='--file sample/Ping.java -i cli/openclover/build/cli/db/clover.db'

You can also create an IntelliJ run configuration (Gradle) for the task to enable debugging. Make sure to select ejunit as the Gradle project so that the working directory is set accordingly.

IntelliJ

Without Gradle, IntelliJ cannot run the project correctly because the classpath cannot be extended due to a known issue.

Additional Notes on the Architecture​

The source code of all components is organized in a multi-project structure with Gradle to avoid redundancy.

  • The samples project contains sample code files for testing, which include different JUnit features and also cover special cases, such as infinite loops. This samples project is included in many of the other projects and enables convenient manual testing with runSample. In addition, there are a few automated tests that use these sample files.
  • The trex project contains all classes for generating the TREX report.
  • The common project contains helper classes for use in all projects.