Tests
The following sections describe how Eva can be tested manually (Playground, Jupyter notebooks) and how the automated tests can be run.
Playground
There is a web-based demo system for testing Eva (eva.playground), built with
Streamlit. It allows selecting tasks in the Subato Task Exchange Format
(STEF) as specified by Subato, with features for temporarily modifying the
task files. The playground can be started locally with python -m streamlit run app.py in the
eva.playground/eva/playground directory.
Streamlit offers an auto-reload feature that reloads source files on changes when running the
playground locally. However, clicking the Reload button may cause errors, such as:
pydantic_core._pydantic_core.ValidationError: 6 validation errors for EvalRequest
solution_files.0
Input should be a valid dictionary or instance of EvaFile [type=model_type, input_value=EvaFile(path='Zeit.java',...nute+min)%60);\n
Streamlit uses
pickle
for session serialization, which causes these errors. When they occur, simply reload the page. See
this question on
Stackoverflow
for more context.
Jupyter Notebooks
Alternatively, Jupyter notebooks can be used for manual testing. For an example, see
exceptions.ipynb.
To be able to run notebooks, run cd eva.pltform && poetry install --with test. Then run:
jupyter notebook /home/fischer/Repositories/SLS/eva2/eva.pltform/tests/integration/coverage/exceptions.ipynb
Automated Tests
For each system component (e.g., Dispatcher,
Registry), unit tests are located in the tests directory of the
corresponding project. Integration and system tests are located under
eva.platform/tests/integration. Refer to the docs on writing integration
tests for more details.
For running the tests, set up the Poetry project with:
cd eva.pltform && poetry install --with test
Then, the tests can be started with:
poetry run pytest tests/integration/evaluator -v
Under "Settings" -> "Python Integrated Tools", you may need to switch the test runner to pytest.
Currently, a bug in PyCharm may prevent failure messages from being displayed when assertions fail
(PY-50396). As a workaround, add --tb=short as an
argument to pytest in the run configuration.
Running the integration tests requires the environments to be built first. To automatically build
the environments located in the environments directory, pass the --build option. The logging
level can be configured using --log_level. Setting it to DEBUG displays all commands executed by
evaluators in the environments, along with their output.
You can also run specific test cases for a subset of the configured environments. For example:
poetry run pytest --env java19 tests/integration/evaluator/java/compile.py -v