Skip to main content

Dispatcher

To process code analysis requests, a dispatcher instance connects to the RabbitMQ broker configured through the EVA_DISPATCHER_RABBITMQ_HOST environment variable on startup. Then, it creates a queue in the amq.topic exchange if it does not exist already. If the connection cannot be established or is lost, the dispatcher automatically retries at regular intervals. During temporary connection failures, the queue remains available for some time instead of being removed immediately, allowing pending requests to be processed once the connection is restored. Requests published before a dispatcher instance starts are discarded unless clients create the queue themselves using the same name and TTL configuration.

Request​

Clients must send requests for code analysis to the broker as an AMQP message with the routing key eva.request. Every request must include the VERSION header in the message properties, where the value must match the major component of the Eva version (4.0.2). This allows clients and dispatchers to detect incompatible protocol changes early.

Before sending a request, the client should create an exclusive queue in which the dispatcher can put the response. The reply_to header of the request must then be set accordingly, so that the response later arrives at this queue and can be processed by clients.

The message body must contain a JSON object with the following structure:

EvalRequest: {
  stack_ref: string,
  solution_files: EvaFile[],
  include_files: EvaFile[] = [],
  test_files: EvaFile[] = [],
  capability: RequestedCapability {
    compile: bool = False,
    test: bool = False,
    style: bool = False
  },
  params: { [key: string]: str }
}
  • stack_ref: The selector for the evaluation stack that should be used for analysis.
  • solution_files: Code files to analyze.
  • test_files: Test files whose test cases are to be run. (only when test = true)
  • include_files: Additional files required for compiling or running the code.
  • capability:
    • compile: Whether the code should be compiled.
    • style: Whether a style check should be performed.
    • test: Whether the tests should be run.
  • params: Additional evaluator-specific parameters. Refer to the documentation of the evaluation stack that should be used.
EvaFile: {
    path: string
    content: string
}
  • path: File name or relative path. For Java, the path must match the fully qualified class name. For example, de/hsrm/subato/Date.java if the class has the FQCN de.hsrm.subato.Date.java. If the class has no package, the file name is sufficient. For most of the other languages, specifying the file name is sufficient as well.
  • content: File content.

Code Analysis​

The following diagram describes the analysis process with the java evaluator and a java19 environment that is provided through a Docker image.

After an incoming request has been validated and the metadata of the evaluation stack has been queried through the Registry, a new container is started through the Docker Engine API. The files from the request are then copied into the container. Execution then takes place through the evaluator, which runs bash scripts or commands in the containers through the Docker Engine API and processes their output.

Handling Non-Termination​

If the code under test exceeds the configured execution timeout, the evaluator aborts execution instead of waiting indefinitely. This minimizes the waste of compute time and reduces the risk of a failure due to resource exhaustion.

Timeouts are typically caused by infinite loops or deadlocks during dynamic analysis. Evaluators should terminate only the affected test case and continue with the remaining ones whenever possible. Even if test execution cannot be completed, evaluators should always return a test result. This ensures that results remain consistent and allows the test case to be identified that caused the timeout.

The maximum execution time for a script or command can be configured with the following environment variables:

  • EVA_DISPATCHER_EV_COMPILE_TIMEOUT
  • EVA_DISPATCHER_EV_STYLE_TIMEOUT
  • EVA_DISPATCHER_EV_TEST_TIMEOUT
  • EVA_DISPATCHER_EV_CODECOV_TIMEOUT

Parallelization and Resource Limits​

The dispatcher can run up to EVA_DISPATCHER_DOCKER_MAX_INSTANCES containers in parallel. Additional requests remain queued until a running container finishes. A dispatcher instance can therefore effectively process up to EVA_DISPATCHER_DOCKER_MAX_INSTANCES requests in parallel.

The CPU and memory of containers are limited so that infinite loops or endless memory consumption do not affect the compute time and memory of the other containers (and do not crash the host system). These resource limits also improve determinism. Since test execution relies on timeouts, heavy CPU contention could otherwise cause correct code to exceed their timeout simply because other containers consume a significant share of the available CPU time.

Resource limits are configured through the environment variables EVA_DISPATCHER_DOCKER_CPU_QUOTA and EVA_DISPATCHER_DOCKER_MEM_LIMIT. The CPU quota indicates the maximum number of CPUs available to a container and can be specified with both integer and fractional values. Memory limits are specified in the format <limit>k|m|g, e.g. 512m.

Network Access​

By default, containers are started with the network_mode=host option, allowing them to access the network. When issuing many requests at once, the network_mode=none option should be used if network access is not required for analysis. Studies have shown that disabling network access can significantly improve container performance. The network mode can be configured through the EVA_DISPATCHER_DOCKER_NETWORK_MODE environment variable.

Caching of Docker Images​

As Docker images can be rebuild or updated in the Docker image registry, the dispatcher needs to check regularly if there is a new version available. Yet, this may require HTTP requests to the registry, which can slow down request processing if this is performed before every analysis request. To reduce this overhead, the dispatcher caches image metadata. The cache lifetime can be configured with EVA_DISPATCHER_DOCKER_IMG_CACHE_TTL, where the value refers to the number of seconds. A value of -1 indicates an infinite TTL, and the cache can be disabled by setting the value to 0.

Output Limiting​

The code to analyze may produce hundreds of megabytes of output during testing. To prevent excessive memory usage, the output can be limited with EVA_DISPATCHER_EV_MAX_OUTPUT_SIZE. This limit is also passed to the evaluator, which is responsible for enforcing it during execution.

Orphaned Containers​

If a dispatcher instance terminates unexpectedly, running containers cannot be stopped immediately due to technical limitations. To prevent orphaned containers from running indefinitely, every container is configured to shut down automatically after 10 minutes.

Response​

On successful analysis, the dispatcher sends a response in JSON format back to the RabbitMQ broker. The response is then delivered to the reply queue specified by the client. The response has the following structure:

EvalResponse: {
  success: bool
  message: string
  result: EvalResult
  version: string
  sysconf: SysConfig {
    rt: {
      type: string, # e.g. docker

      # if type = 'docker', then the following fields are included:
      mem_limit: string,
      cpu_quota: float,
      auto_shutdown: int,
      network_mode: string
    }
    evaluator_opts: EvaluatorOptions {
      max_output_size: int
      compile_timeout: int
      style_timeout: int
      test_timeout: int
      codecov_timeout: int
    }
  }
}
  • success: true if the analysis completed successfully, false otherwise. This indicates whether the analysis completed successfully, not whether the code passed all tests.
  • message: Error message, only when success = false.
  • result: Result of the analysis, only when success = true.
  • version: Eva version of the dispatcher instance that processed the request.
  • sysconf: Runtime configuration of the dispatcher instance that processed the request.

The result field has the following structure:

EvalResult: {
  compilation: CompilationResult
  test: TestResult
  style: StyleResult
}

Whether a field is null depends on the requested capabilities and their dependencies. For example:

  • If only compilation is requested, test and style are null.
  • If testing is requested but compilation fails, test is null because the tests cannot be executed

Compilation​

CompilationResult: {
   compiling: bool
   output: string
}
  • compiling: Indicates whether the code compiled successfully.
  • output: Combined stdout and stderr produced by the compiler.

Test​

TestResult: {
    testcases: int
    failed: int
    passed: int
    stats: TestcaseStats {
      failures: int
      errors: int
      pending: int
      timeouts: int
    }
    correctness: float
    leaks: int
    testsuite: Testsuite
    output: str
    coverage: Coverage
}
  • testsuite: Test results in TREX format. Since test case names are not necessarily unique across test files, each Testcase includes an additional id field. The ID is composed of the name and class_name attributes in the format <name>#<class_name>. If class_name is not present (e.g. in programming languages without a class concept), the ID is identical to name.
  • testcases: Number of test cases.
  • passed: Number of passed test cases.
  • failed: Number of failed test cases (testcases - passed)
  • stats: Number of test cases for each outcome
  • correctness: Value between 0 and 1, computed as (testcases - failed) / testcases. If testcases = 0, then the value is also 0.
  • leaks: Difference between the number of memory allocations and deallocations.
  • output: Output produced by the test process itself, such as messages from the test runner. Unlike stdout and stderr, which contain output generated by the analyzed code, this field contains output from the testing infrastructure.
  • coverage: Code coverage results in CEXF format.

Style​

StyleResult: {
  errors: int
  report: StyleReport {
    errors: StyleError[]
    tool: string
    version: string
  }
}
  • errors: Number of reported style violations.
  • report: Style analysis results in STEX format