Authoring Guide

How to structure and publish a Divekit Evaluation Pipeline plugin repository.

Authoring contracts are still experimental and may evolve.

This page is now only a stable pointer.

Source of Truth

Plugin authoring and CI conventions should be maintained in the plugin template repository:

That repository should define:

  • repository layout: manifest.json, job.yaml, Dockerfile, code under src/
  • the plugin image, derived from the suite’s build image
  • optional reusable feedback assets under feedback/
  • release-template usage
  • plugin artifact-writing conventions
  • FAQ artifact conventions
  • copyable examples for new plugin repositories

Stable Summary

A plugin is one repository inside its suite’s GitLab group, named after the plugin id. Every plugin repository provides:

  • manifest.json with "kind": "plugin"; the plugin’s version is its Git tag
  • job.yaml: image, script, artifact paths, and resource hints of the one job the plugin contributes
  • a Dockerfile when the plugin ships code, and a release-oriented .gitlab-ci.yml that includes the shared release template
  • a maintainer-readable README.md

The CLI generates the pipeline job from job.yaml, names it by runtime id (<suite>:<plugin>), and runs it after the suite’s build job with the materialized submission and the build stage’s artifacts. The spec carries no job name, stage, rules, needs, variables, or cache; the CLI rejects a spec that sets them.

version: "1.0"
image: registry.git.nrw/divekit/plugins/maven/example:__DIVEKIT_PLUGIN_VERSION__
script:
  - sh /opt/divekit/src/run.sh
artifacts:
  paths:
    - plugin-results/

The shared image-and-release template builds the image for each SemVer tag, replaces __DIVEKIT_PLUGIN_VERSION__ with the release tag, and packages manifest.json and job.yaml (plus feedback/** when present) into the release. Courses lock the resolved tag and the image digest. Keep dependency installation in the image build instead of running apt, pip, or equivalent installers in every evaluation pipeline.

Plugin implementation code lives under src/ and ships in the image, commonly below /opt/divekit/src/; it is never copied into evaluation repositories:

FROM registry.git.nrw/divekit/plugins/maven/builder:v1.0.2

COPY src/ /opt/divekit/src/
COPY feedback/ /opt/divekit/feedback/

The job receives its namespace from the CLI and writes everything it publishes below the result directory:

DIVEKIT_SUITE_ID            the suite half of the runtime id, e.g. maven
DIVEKIT_PLUGIN_ID           the plugin half of the runtime id, e.g. pmd
DIVEKIT_PLUGIN_RESULT_DIR   plugin-results/<suite>/<plugin>

Optional feedback/ files may be packaged in the release asset and materialized into the evaluation repository. If the plugin wants to show feedback on the generated GitLab Page, write it to the plugin result artifacts:

plugin-results/<suite>/<plugin>/feedback.md
plugin-results/<suite>/<plugin>/feedback/index.json
plugin-results/<suite>/<plugin>/feedback/**/*.md

Static Markdown resources can live in the plugin repository under feedback/ and be copied or referenced by the plugin job when it builds run-specific feedback. Feedback is published through the eval repository’s GitLab Page; it is not written back to student repositories. Pages a plugin renders under plugin-results/<suite>/<plugin>/html/ are published into the root of that Page.

Items declared in feedback/index.json can set defaultOpen to control whether the generated Page expands them initially. If the field is omitted, test-specific feedback is expanded by default and global feedback stays collapsed by default.

Use report notices when an annotation belongs to one concrete report result instead of the broader feedback section. Notices are plain text and render above the report body on the generated Page.

For execution problems that are not test results, publish issues instead of fake tests:

plugin-results/<suite>/<plugin>/issues/index.json
plugin-results/<suite>/<plugin>/issues/**/*.md
plugin-results/<suite>/<plugin>/logs/**

Issue items can reference a Markdown explanation and an optional log file. Use affectsResult: true when the problem means the evaluation could not run successfully.

A plugin that needs settings declares a wizard in manifest.json; see Configuration Wizards. For the CLI-consumed semantic contract, see Contract Reference.