Study. How to evaluate a translation engine on Swiss law

By Neur.on's legal and language team · 15 September 2026 · Updated on 15 September 2026

The question ‘which is the best translation engine for Swiss law?’ has no answer until you have said on what texts, in which direction, by which criteria and with what margin of error. This document proposes a method that any legal department can apply without a laboratory, and that anyone can challenge point by point.

1. Why automatic scores are not enough

Automatic translation metrics compare an output with a reference translation and measure how close they are. They are useful for tracking how a system evolves from one version to the next. They are misleading for a legal choice, for a structural reason: they do not distinguish a nuance of wording from an error of law.

A sentence that renders every word correctly but cites an article that does not exist, or gives jurisdiction to a foreign court, stays close to the reference and gets a good score. A perfectly accurate sentence worded differently from the reference is penalised. For a legal expert, this is exactly the opposite of what matters.

The evaluation of an engine on legal texts must therefore be done by people, on binary criteria defined in advance. Binary, because a mark out of ten mostly measures the evaluator's mood, whereas a closed question such as ‘is the statutory reference cited correct?’ gives the same verdict from two different people.

2. Six measurable criteria

Each criterion is marked compliant or non-compliant, segment by segment. When a criterion does not apply to a segment, it is marked not applicable and leaves the denominator for that segment.

These six criteria are not equally serious, and above all they must not be averaged. Separate disqualifying errors from differences of preference. A wrong statutory reference or an omission makes a segment unusable; a correct term that is absent from the official texts needs a correction, nothing more. So publish two separate rates: the rate of segments with at least one disqualifying error, and the rate of segments with at least one deviation to correct.

3. Building the corpus

The corpus decides the result more than the engines do. Four requirements.

Representativeness

The corpus must reflect what you actually handle, not what is easy to collect. Cross three dimensions: the area of law (contracts, litigation, company law, financial regulation, administrative law), the type of document (deed, written submission, ruling, correspondence, model clause), and the translation direction. An engine that performs well from German into French may not do so from French into German: the two directions must be counted separately, never aggregated.

Size

There is no magic number of segments: the useful size depends on the number of criteria, the rarity of the phenomena observed and the precision you want. The practical rule is to reason cell by cell: each combination of area of law and translation direction must contain enough segments for a difference not to rest on one or two sentences. A small, balanced corpus is better than a large one concentrated on a single area.

Density of phenomena

Deliberately build part of the corpus from segments that contain what you want to measure: statutory references, names of courts, amounts, dates, regulatory terms. A corpus of neutral sentences measures nothing useful, because every engine succeeds on it.

Traceability and confidentiality

Each segment must carry its origin, so that its accuracy can be checked against the official source. If you use internal documents, pseudonymise them before any transmission to a third-party engine, and document this processing: an evaluation corpus is data processing like any other. This point is developed in our article on where data is processed.

4. The evaluation protocol

1. Fix the parameters. Same language pair, same settings, same confidentiality mode, and the same use or non-use of a glossary for all engines. A difference in configuration invalidates the entire comparison.
2. Generate within a single time window and record the date and time of each output. Engines change; an output without a date is not data.
3. Anonymise the outputs. Remove any mention of the engine, including formatting clues that give it away, and assign a neutral identifier.
4. Randomise the order of presentation, differently for each evaluator, so that fatigue and order effects do not fall on the same engine.
5. Have two independent people evaluate, with no communication between them, using the same grid.
6. Measure agreement between evaluators before looking at the results. Low agreement shows that the grid is ambiguous, not that the evaluators are poor: clarify the definitions and start again.
7. Resolve disagreements through a third person acting as arbiter, who does not know where the outputs come from either, and record the rule that emerges for later cases.
8. Replicate. Run a subset of the corpus again a few weeks later, with the same parameters, to measure variance over time.

This last point is not theoretical. During an observation session, regenerating certain segments produced strictly identical outputs, while other segments varied from one generation to the next. A campaign that does not measure this variance cannot tell whether a gap between two engines is a difference in quality or a random effect of generation.

5. Seven methodological pitfalls

1. Judging on a single sentence. This is the most common and most costly pitfall. One well-translated sentence says nothing about how an engine behaves on a 60-page document, where consistency becomes the first criterion.
2. Confusing fluency with accuracy. Recent engines produce pleasant sentences. A fluent output that cites a non-existent text is more dangerous than a clumsy one, because it does not prompt anyone to check. In the same observation session, one output rendered the name of Switzerland's highest court by that of a foreign court, in an otherwise perfectly constructed sentence.
3. Forgetting the variance between generations. Comparing one output from each engine means comparing two draws. Without replication, the measured gap may be due entirely to chance in generation.
4. Aggregating translation directions. An average score across German into French and French into German hides exactly the information you are looking for.
5. Letting the engine's name show. An evaluator who knows what they are marking no longer marks the output; they mark their opinion of the engine.
6. Mixing seriousness and preference. Counting a wrong statutory reference and a debatable stylistic choice in the same total produces a figure that no one can interpret.
7. Evaluating on widely circulated public texts. The most frequently cited legislative texts are abundantly present in the training data of general-purpose models. Success on them does not predict performance on your documents. Keep some as a control, but build the core of the corpus on less exposed texts.

6. Reporting the results

A credible evaluation report publishes everything needed to challenge it.

• The composition of the corpus: number of segments per area of law and per translation direction, origin, date compiled, version.
• The exact parameters of each generation, and their date.
• The grid of criteria, with the definitions of non-compliance used.
• The rate of agreement between evaluators, before arbitration.
• Results by criterion and by cell, never a single overall score.
• The variance measured during replication.
• The limits, spelled out: what the corpus does not cover, what the method does not measure.

And one word of caution: never use a ranking of engines as a sales argument, including your own. A ranking holds for one corpus, one date and one configuration. What can honestly be passed on is the method and the data, not the podium.

7. What will be published on this page

We are currently applying this method to a first corpus of Swiss law, built according to the requirements set out in point 3 and covering the national language pairs. The results will be published on this page, with the composition of the corpus, the grid used, the rates of agreement between evaluators and the limits of the exercise.

Until then, no figure is put forward here: a method proves nothing by itself, and publishing a score before publishing the protocol would be exactly what this document advises against.