Reverse PolEval: Diagnostic Benchmark Design for Polish LLMs

PolEval 2026 shared task. Participants design the benchmark. Organisers run a fixed panel of local open-weight models and score how diagnostic the submitted set is.

A useful item is not merely hard. An item on which every model fails, or every model succeeds, does not compare systems. The target is an automatically checkable Polish example on which some models are correct and others are not, for a reason that depends on Polish.

Organiser: Krzysztof Wróbel, Jagiellonian University.