Evaluators

Each evaluator can declare support for one or more of the following capabilities: compile, test, style.

Implementing Evaluators

To implement a new evaluator, create a subclass of eva.pltform.evaluation.core.eval.Evaluator and override the setup method:

class JavaEvaluator(Evaluator):
    # ...

    def setup(self, ctx: EvaluationContext):
        # Evaluator-specific validation logic: reject certain requests
        check_dynamic_asserts(ctx)

        # Process custom parameters
        java_args, javac_args = configure_jvm(ctx)

        # Define handlers to be called for each capability:
        return ExecutionHandlerConfig(
            compile=JavaCompileHandler(javac_args=javac_args),
            style=JavaStyleHandler(java_args=java_args),
            test=JavaTestHandler(java_args=java_args),
            coverage=JavaCoverageHandler(javac_args=javac_args, java_args=java_args)
        )

Then, each handler can implement the analysis logic for the given capability. For example:

class JavaTestHandler(TestHandler):

    def __init__(self, java_args):
        self.java_args = java_args

    def run(self, evaluator: Evaluator, ctx: EvaluationContext) -> TestResult:
        test_fqcns = [to_fqcn(f) for f in ctx.payload.test_files]
        r = ctx.env.bash(
            [
                f'java {self.java_args} -cp $EJUNIT_RUNNER_PATH:$CLASSPATH:classes',
                'de.hsrm.sls.ejunit.mono.Runner -o trex_report.xml',
                f'-s {ctx.opts.max_output_size}',
                *test_fqcns,
            ],
            max_output_size=ctx.opts.max_output_size,
            timeout=ctx.opts.test_timeout,
        )
        r.expect(status=[CODE_SUCCESS, CODE_TIMEOUT])

        testsuite = self.trex.parse(ctx.env.read_file('trex_report.xml')).fix_timeouts(
            ctx.opts.test_timeout
        )
        return self.mapper.map(testsuite=testsuite, output=r.out())

The evaluation context (ctx) provides access to the environment (ctx.env), the request (ctx.payload), and the evaluator configuration (ctx.opts). ctx.env can be used to run commands, create directories and read files. The result returned by the ctx.env.bash method allows to use asserts such as expect() for verifying that a command completed with one of the expected exit codes before processing its output.

For each evaluator, the following metadata must be declared:

class JavaEvaluator(Evaluator):
    id = 'java'
    supported_envs = 'java*'
    default_env = 'java19'
    capability = EvaluatorCapability(compile=True, style=True, test=True)

• id: Unique ID of the evaluator.
• supported_envs: Unix glob pattern that matches all environments for which the evaluator can be used.
• default_env: In requests, an evaluator can be specified without an environment. In that case, the environment specified here is used by default.
• capability: Capabilities supported by the evaluator.

For the evaluator to become available, it must be registered in EvaluatorRegistry as follows:

class EvaluatorRegistry:
    # ...

    def _load_evaluators(self):
        if self._evaluators is None:
            self._evaluators = [
                # ...
                JavaEvaluator,
            ]

        return self._evaluators

Writing Integration Tests

Each evaluator should include integration tests to simplify debugging and provide automatic regression coverage. Create one or more test modules for the evaluator:

@pytest.mark.evaluator('java') # 'java' is the ID previously assigned to the evaluator
class JavaTest:
    def test_passing(self, ctx: TestContext):
        payload = EvalPayload(
            solution_files=[
                EvaFile(path='Heron.java', content=ctx.fixtures.load('java/passing/Heron.java'))
            ],
            test_files=[
                EvaFile(
                    path='HeronTest.java', content=ctx.fixtures.load('java/passing/HeronTest.java')
                )
            ],
            include_files=[],
            compile=True,
            test=True,
        )

        result = ctx.eval(payload)
        assert result.compilation.compiling
        assert pytest.approx(result.test.correctness) == 1.0
        # ...

Fixtures should be organized in eva.pltform/tests/integration/fixtures and can be accessed by passing a path relative to the fixtures directory into ctx.fixtures.load. Each fixture should contain the minimal code required for a test case. When writing integration tests, treat evaluators as black boxes. Avoid testing every internal detail separately, as this increases complexity and makes the tests harder to maintain.

Finally, the evaluator must be registered for testing by defining all environments that should be used during testing:

env_map = {
        # ...
        'java': ['java19', 'java20', 'java21'],
}

In this configuration, all test cases in classes annotated with @pytest.mark.evaluator('java') are run once for the environments java19, java20 and java21. See Integration Tests for more details on how the test suite can be run.