Skip to main content

Test Result Exchange XML (TREX)

TREX is an exchange format for representing test results and is based on the Apache Ant JUnit XML/JUnit XML format. It addresses two limitations of existing formats: The lack of a standard for uniformly representing language- and technology specific test result representations and the need to represent additional information required for automated analysis of programming process data.

The format design follows these principles:

  1. Compatibility with JUnit XML: JUnit XML is widely adopted, and libraries for generating and processing this format are available in many programming languages. Thus, TREX should be designed to be compatible with JUnit XML. Thus, it uses a subset of the available attributes and elements and introduces additional custom elements where needed.
  2. Support for incremental generation: Information should be allowed to appear at any point in the document to enable stream-based generation without requiring the writer to seek back and modify previously written data.
  3. Avoiding redundant information: Denormalized data, such as manually maintained summaries which include the total number of executed tests, should be avoided whenever possible. Redundant information can easily become inconsistent with the actual test results.

TREX introduces the following additional custom elements:

  • storage captures execution-related information that may only become available during or after test execution.
  • system-out and system-err within testcase allow to capture standard output and error streams for individual test cases (also used by other tools).
  • pending and started represent additional test case states that are not available in the original JUnit XML format.

Limitations:

  • Test cases can be skipped in theory, yet the format does not currently allow representing skipped test cases.

Structure​

trex.xml
<?xml version="1.0" encoding="UTF-8" ?>
<testsuite timestamp="2023-07-14T16:11:12">
    <testcase name="test01" classname="de.hsrm.sls.eva.junit.samples.simple.PingTest" time="0.005">
        <system-out><![CDATA[JAA]]></system-out>
        <system-err><![CDATA[BUUUUH]]></system-err>
        <failure message="expected: <43> but was: <42>">java.lang.AssertionError: expected:&lt;43&gt; but was:&lt;42&gt;
at de.hsrm.sls.eva.junit.samples.simple.PingTest.test02(PingTest.java:23)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at de.hsrm.sls.eva.junit.Runner.runTests(Runner.java:28)
at de.hsrm.sls.eva.junit.Runner.main(Runner.java:18)
        </failure>
    </testcase>
    <testcase name="test02" classname="de.hsrm.sls.eva.junit.samples.simple.PingTest" time="0.005">
        <error message="Moin">java.lang.RuntimeException: Moin
at de.hsrm.sls.eva.junit.samples.simple.Ping.faulty(Ping.java:8)
at de.hsrm.sls.eva.junit.samples.simple.PingTest.test03(PingTest.java:28)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at de.hsrm.sls.eva.junit.Runner.runTests(Runner.java:28)
at de.hsrm.sls.eva.junit.Runner.main(Runner.java:18)
        </error>
    </testcase>
    <testcase name="test03" classname="de.hsrm.sls.eva.junit.samples.simple.PingTest" time="0.005" />
    <testcase name="test04" classname="de.hsrm.sls.eva.junit.samples.simple.PingTest">
        <timeout timeout="3"/>
    </testcase>
    <system-out><![CDATA[JAA]]></system-out>
    <system-err><![CDATA[BUUUUH]]></system-err>
    <!-- makes little sense here (Java test cases are shown in the example) - only for illustrating the structure-->
    <storage allocated="2" free="1"/>
</testsuite>

testsuite​

AttributeRequiredTypeDescription
timestampnoISO8601Start (UTC)
timenodoubleExecution time in seconds of the whole test suite. If test execution is aborted, this attribute is not set. Note: The value does not necessarily correspond to the sum of the time attributes of all test cases. The overhead for running the test cases can be significantly higher than the execution itself.

testcase​

trex.xml
<testcase name="test01" classname="de.hsrm.sls.eva.junit.samples.simple.PingTest" time="0.005">
    <system-out><![CDATA[JAA]]></system-out>
    <system-err><![CDATA[BUUUUH]]></system-err>
    <failure message="expected: <43> but was: <42>">java.lang.AssertionError: expected:&lt;43&gt; but was:&lt;42&gt;
at de.hsrm.sls.eva.junit.samples.simple.PingTest.test02(PingTest.java:23)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at de.hsrm.sls.eva.junit.Runner.runTests(Runner.java:28)
at de.hsrm.sls.eva.junit.Runner.main(Runner.java:18)
    </failure>
</testcase>
AttributeRequiredTypeDescription
nameyesstringName of the test case
classnamenostringName of the test class (or test file, if there is no class concept)
timenodoubleExecution time of the test case in seconds. If a timeout occurs, this attribute is not set.

The content of a testcase element represents the test result. It may be empty (indicating a successful test case) or contain one of the following elements:

  • error: Indicates that the test case failed due to an assertion error. The message attribute should contain the message that was provided in the assert. If no such message was provided, it should contain a generated message based on the expected and actual values.
  • failure: Indicates that the test case failed due to an unexpected exception. The message attribute should contain a short description of the exception.
  • timeout: Indicates that the test case did not complete within the expected time limit. The timeout attribute should specify the duration (in seconds) for which the result was awaited.
  • pending: Indicates that the test case has not been executed yet.
  • (started): Indicates that the test case has started but has not completed yet. This element is only used internally by Eva and is replaced with timeout before results are sent to the client. Background: Due to technical limitations, the timeout element cannot always be set directly. If the test execution process is terminated unexpectedly, Eva identifies the most recently started test case based on the started element and automatically marks it as timeout.

These result elements may contain additional details in their body. TREX does not define any requirements for these details. For example, a Java implementation may include the stack trace of the exception that caused the test result. Moreover, each test case may contain system-out and system-err elements with the corresponding output for that specific test case. The system-err element should not be used to store error details if the test case already contains an error or failure result. The content of system-out and system-err must be limited in size to prevent excessive document growth caused by scenarios such as infinite loops. Very large documents can exceed available memory and are difficult to transfer over the network. We recommend limiting each element to 4096 characters. If the output is truncated, the last N characters should be retained, and the content should include an indication that truncation occurred.

storage​

In some programming languages, memory must be managed explicitly by allocating and releasing memory manually. The correct usage of such operations within the tested code can be represented using the storage element:

trex.xml
<storage allocated="2", free="1"/>
AttributeRequiredTypeDescription
allocatedyesintNumber of executed operations that allocate memory (e.g. calls to malloc())
freeyesintNumber of executed operations that release memory

system-out/system-err​

In addition to output that was captured per test cases, aggregated standard output and error output across all test cases can be represented using these elements.