feat ; make systemtest runs portable and re-runnable from CI artifacts (closes #387)

[PranjalManhgaye]

This PR addresses #387 by making
systemtests run directories portable and easily re-runnable after downloading the
system_tests_run_<run_id>_<run_attempt> artifact from GitHub Actions.

Summary of changes

• Systemtest portability

  • In tools/tests/systemtests/Systemtest.py, Docker Compose generation now uses
    relative paths instead of CI-host-specific absolute paths:
    • run_directory is now "." in the templates.
    • dockerfile_context is set to tools/tests/dockerfiles/<PLATFORM> relative to
      the systemtest directory.
  • This makes each runs/<systemtest_dir>/ folder self-contained and portable
    to any path on a Linux machine with Docker.

• Replay helper script

  • During _build_docker, we now create rerun_systemtest.sh inside each systemtest
    directory. It:
    • Builds tutorial images:
      docker compose --file docker-compose.tutorial.yaml build
    • Runs the tutorial:
      docker compose --file docker-compose.tutorial.yaml up --abort-on-container-exit
    • If docker-compose.field_compare.yaml exists, it also runs fieldcompare with:
      docker compose --file docker-compose.field_compare.yaml up --abort-on-container-exit.
  • Users can re-run a failing test locally by:
    1. Downloading system_tests_run_<run_id>_<run_attempt>.zip
    2. cd system_tests_run_<run_id>_<run_attempt>/runs/<systemtest_dir>
    3. Running ./rerun_systemtest.sh (or sh rerun_systemtest.sh)

• Defaults for local runs

  • Updated tools/tests/components.yaml so TUTORIALS_REF defaults to develop
    instead of master, which is more consistent with the current branch naming.

• Documentation

  • Extended tools/tests/README.md with a section “Re-running system tests from CI artifacts”
    describing step-by-step how to replay a systemtest locally from a downloaded artifact.

How to use with a failing CI artifact

1. From the failed CI run, download system_tests_run_<run_id>_<run_attempt>.zip.
2. Extract it:
   unzip system_tests_run_<run_id><run_attempt>.zip
   cd system_tests_run<run_id>_<run_attempt>/runs
   cd <systemtest_dir>
3. Re-run:
   ./rerun_systemtest.sh # or sh rerun_systemtest.sh
   All paths in the generated compose files are relative to this directory, so it
   does not depend on the original GitHub runner paths.

[Copilot]

Pull request overview

This PR aims to make system test run directories portable and re-runnable from downloaded CI artifacts by switching generated Docker Compose files to relative paths, adding a local rerun helper script, updating default tutorial refs, and documenting the workflow.

Changes:

• Generate Docker Compose files using relative paths (intended to avoid CI-runner-specific absolute paths).
• Create a rerun_systemtest.sh helper script inside each system test run directory.
• Update TUTORIALS_REF defaults to develop and document how to rerun tests from CI artifacts.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
tools/tests/systemtests/Systemtest.py	Switches compose generation to relative paths and adds a rerun helper script.
tools/tests/components.yaml	Updates default TUTORIALS_REF from master to develop.
tools/tests/README.md	Adds documentation for rerunning system tests from CI artifacts.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

dockerfile_context_relative is set to tools/tests/dockerfiles/<PLATFORM>, but __copy_tools() copies tools/ into the run directory (parent of self.system_test_dir), not into self.system_test_dir. When running from inside the system test directory, this relative path won’t exist (it would need to be ../tools/... unless the copy location changes). Either copy tools/ into each system_test_dir or update the relative path accordingly.

Suggested change

	dockerfile_context_relative = Path("tools") / "tests" / "dockerfiles" / Path(plaform_requested)
	dockerfile_context_relative = Path("..") / "tools" / "tests" / "dockerfiles" / Path(plaform_requested)

[PranjalManhgaye]

Good catch — the initial version was using paths relative to self.system_test_dir but tools/ is actually copied into the parent run directory (<run_directory>/tools).

I’ve updated the code so that:

• run_directory is now "..", pointing to the parent run directory, matching the expectations of the compose templates ({{ run_directory }}:/runs and cd '/runs/{{ tutorial_folder }}').
• dockerfile_context is now ../tools/tests/dockerfiles/, which matches the actual copy location from __copy_tools.

This keeps the original volume layout intact while still avoiding CI-specific absolute paths.
For the rerun_systemtest.sh comment:
Agreed — to faithfully reproduce CI behavior, the helper script should use the same compose invocations as _run_tutorial() and _run_field_compare().
I’ve updated rerun_systemtest.sh generation to:

• Use docker compose -f docker-compose.tutorial.yaml up for the tutorial step.
• If docker-compose.field_compare.yaml exists, run fieldcompare with docker compose -f docker-compose.field_compare.yaml up --exit-code-from field-compare.

This way the replay script matches the Python runner’s semantics, including the exit code behavior from fieldcompare.

[Copilot]

__get_docker_compose_file() also sets run_directory to "." and dockerfile_context to tools/tests/dockerfiles/<PLATFORM>, which has the same directory-layout mismatch as in __get_docker_services(). As written, the generated docker-compose.tutorial.yaml will likely fail to mount the expected /runs/<tutorial_folder> and fail to find the build context unless tools/ is present inside the system test directory.

Suggested change

	'dockerfile_context': Path("tools") / "tests" / "dockerfiles" / Path(self.params_to_use.get("PLATFORM")),
	'dockerfile_context': Path(os.path.relpath(
	PRECICE_TOOLS_DIR / "tests" / "dockerfiles" / Path(self.params_to_use.get("PLATFORM")),
	start=PRECICE_TESTS_DIR,
	)),

[PranjalManhgaye]

done

[Copilot]

The generated rerun_systemtest.sh uses docker compose ... up --abort-on-container-exit for both tutorial and fieldcompare, but the Python runner uses plain up for the tutorial and up --exit-code-from field-compare for fieldcompare. Without matching flags (especially --exit-code-from), the script may not reproduce CI behavior and may return a success exit code even when fieldcompare fails. Consider mirroring the exact compose invocations used in _run_tutorial() / _run_field_compare().

Suggested change

	"docker compose --file docker-compose.tutorial.yaml up --abort-on-container-exit\n"
	"\n"
	"if [ -f docker-compose.field_compare.yaml ]; then\n"
	" echo \"[systemtests] Running fieldcompare...\"\n"
	" docker compose --file docker-compose.field_compare.yaml up --abort-on-container-exit\n"
	"docker compose --file docker-compose.tutorial.yaml up\n"
	"\n"
	"if [ -f docker-compose.field_compare.yaml ]; then\n"
	" echo \"[systemtests] Running fieldcompare...\"\n"
	" docker compose --file docker-compose.field_compare.yaml up --exit-code-from field-compare\n"

[PranjalManhgaye]

resolved

[PranjalManhgaye]

@MakisH i have resolved the suggested changes by copilot , review it once , thanks