OTS-HOMR

OTS-HOMR against HOMR, staff by staff

Both engines read the same staves from three test sets, and each reading was compared with the correct one. Lower error is better. Every staff counts, including the ones an engine could not read (none failed). Read how we measured before drawing conclusions from the numbers.

Summary

Test setStaves / scoresHOMR errorOTS-HOMR errorChange
String quartet scans792 / 914.47%5.25%−9.23 points (95% interval −12.37 to −7.00)
Mixed repertoire3,422 / 6327.44%4.52%−2.92 points (95% interval −4.05 to −2.03)
Voice and piano scans300 / 135.76%2.59%−3.16 points (95% interval −5.10 to −1.54)

The error rate counts the symbols — notes, rests, accidentals, articulations, slurs, positions — that had to be added, removed or changed to turn a reading into the correct one, divided by the number of symbols in the correct reading. Time-signature numbers are left out of this main figure so the two engines are compared on what both can read.

String quartet scans

792 staves from 9 scores

EngineSymbol errorError incl. time-signature numbersNote pitch or accidental errorStaves read perfectlyPosition accuracyFailed
HOMR14.47%14.64%9.59%146 / 79291.68%0
OTS-HOMR5.25%5.26%0.91%469 / 79295.04%0

Change in symbol error: −9.23 points (95% interval −12.37 to −7.00). The interval comes from resampling whole scores, so related staves are not counted as independent evidence.

Error by kind of symbol

Kind of symbolHOMROTS-HOMR
Pitch2.85%0.61%
Rhythm7.06%2.93%
Accidentals5.52%0.90%
Articulation3.28%2.12%
Slurs1.88%0.74%
Position1.22%0.46%

No five- or twelve-beat time signatures in this set.

Mixed repertoire

3,422 staves from 632 scores

EngineSymbol errorError incl. time-signature numbersNote pitch or accidental errorStaves read perfectlyPosition accuracyFailed
HOMR7.44%8.44%7.32%1,460 / 3,42284.27%0
OTS-HOMR4.52%4.57%4.07%2,132 / 3,42288.62%0

Change in symbol error: −2.92 points (95% interval −4.05 to −2.03). The interval comes from resampling whole scores, so related staves are not counted as independent evidence.

Error by kind of symbol

Kind of symbolHOMROTS-HOMR
Pitch4.58%2.96%
Rhythm4.46%2.70%
Accidentals4.88%2.39%
Articulation3.18%1.81%
Slurs3.53%2.09%
Position3.15%1.71%

Five-beat time signatures read correctly: HOMR 0 of 84, OTS-HOMR 48 of 84. Twelve-beat: HOMR 0 of 115, OTS-HOMR 70 of 115.

Voice and piano scans

300 staves from 13 scores

EngineSymbol errorError incl. time-signature numbersNote pitch or accidental errorStaves read perfectlyPosition accuracyFailed
HOMR5.76%5.86%5.93%97 / 30087.31%0
OTS-HOMR2.59%2.62%1.26%162 / 30094.10%0

Change in symbol error: −3.16 points (95% interval −5.10 to −1.54). The interval comes from resampling whole scores, so related staves are not counted as independent evidence.

Error by kind of symbol

Kind of symbolHOMROTS-HOMR
Pitch2.17%0.78%
Rhythm2.04%1.23%
Accidentals3.53%0.75%
Articulation1.41%0.63%
Slurs1.59%0.84%
Position1.08%0.35%

Twelve-beat time signatures: HOMR 0 of 2, OTS-HOMR 1 of 2 (too few to say anything).

What these numbers do and do not show

  • They measure how well single staves are read. They say nothing directly about whole-page results, layout, or the lyrics and text features.
  • The test staves were looked at during development, so these are descriptive benchmark figures, not a fresh test.
  • The quartet figure comes from nine scores. The correct answers for all three sets were built by us and can contain errors.
  • Both engines were run through the same software, not through each engine’s own full pipeline, so staff reading is compared like for like.
← Back to OTS-HOMR