Benchmark

How we measured

A short, plain account of how OTS-HOMR and HOMR were compared, fixed before the results were read.

What was compared

The OTS-HOMR engine the scanner runs, and the HOMR engine the scanner offers (HOMR’s released model). Both read exactly the same picture of each staff, prepared the same way, through the same software, so the only difference is the recognition model.

The test sets

  • String quartet scans: 792 staves from 9 scores, scanned pages paired with the OpenScore quartet editions.
  • Mixed repertoire: all 3,422 staves of the validation split of a large collection of public scores, from 632 scores.
  • Voice and piano scans: 300 staves from 13 scores, scanned pages paired with the OpenScore Lieder editions.

Each set is reported separately. We did not read any test split reserved for the end, and no model was chosen or retrained using these results.

What was counted

Each staff is turned into a sequence of symbols, each described by six things: pitch, rhythm, accidental, articulation, slur and position. The main figure is the number of symbols that must be inserted, deleted or changed to turn the engine’s reading into the correct one, divided by the number of symbols in the correct reading.

Time-signature numbers are removed from both readings for the main figure, because HOMR’s released model cannot output some of them; the error including those numbers is shown alongside, and the time signatures OTS-HOMR adds are reported separately. A staff an engine could not read counts as an empty reading, not as an exclusion; none failed.

We also report the errors for each kind of symbol, pitch and accidental errors on notes alone, the share of staves read perfectly, and a simple position accuracy.

Confidence intervals

The intervals come from resampling whole scores 2,000 times (with a fixed random seed), not individual staves, because staves from one score are not independent: they share a composer, an engraving style and often a typo. An improvement is reported only when the interval supports it; neutral or worse results are reported the same way.

What this cannot tell you

  • It measures reading single staves. It says nothing about page layout, whole-page correctness, or the lyrics and text features.
  • The test staves had been looked at in earlier work, so this describes benchmark performance and is not evidence of how the engine does on completely new material.
  • The correct answers were built by us from the editions; where the edition or our pairing is wrong, both engines are marked wrong alike.
  • Run times were taken while other jobs were using the same machine, so we make no speed claims from this benchmark.
← Back to the results