OTS-HOMR against HOMR, staff by staff
Both engines read the same staves from three test sets, and each reading was compared with the correct one. Lower error is better. Every staff counts, including the ones an engine could not read (none failed). Read how we measured before drawing conclusions from the numbers.
Summary
| Test set | Staves / scores | HOMR error | OTS-HOMR error | Change |
|---|---|---|---|---|
| String quartet scans | 792 / 9 | 14.47% | 5.25% | −9.23 points (95% interval −12.37 to −7.00) |
| Mixed repertoire | 3,422 / 632 | 7.44% | 4.52% | −2.92 points (95% interval −4.05 to −2.03) |
| Voice and piano scans | 300 / 13 | 5.76% | 2.59% | −3.16 points (95% interval −5.10 to −1.54) |
The error rate counts the symbols — notes, rests, accidentals, articulations, slurs, positions — that had to be added, removed or changed to turn a reading into the correct one, divided by the number of symbols in the correct reading. Time-signature numbers are left out of this main figure so the two engines are compared on what both can read.
String quartet scans
792 staves from 9 scores
| Engine | Symbol error | Error incl. time-signature numbers | Note pitch or accidental error | Staves read perfectly | Position accuracy | Failed |
|---|---|---|---|---|---|---|
| HOMR | 14.47% | 14.64% | 9.59% | 146 / 792 | 91.68% | 0 |
| OTS-HOMR | 5.25% | 5.26% | 0.91% | 469 / 792 | 95.04% | 0 |
Change in symbol error: −9.23 points (95% interval −12.37 to −7.00). The interval comes from resampling whole scores, so related staves are not counted as independent evidence.
Error by kind of symbol
| Kind of symbol | HOMR | OTS-HOMR |
|---|---|---|
| Pitch | 2.85% | 0.61% |
| Rhythm | 7.06% | 2.93% |
| Accidentals | 5.52% | 0.90% |
| Articulation | 3.28% | 2.12% |
| Slurs | 1.88% | 0.74% |
| Position | 1.22% | 0.46% |
No five- or twelve-beat time signatures in this set.
Mixed repertoire
3,422 staves from 632 scores
| Engine | Symbol error | Error incl. time-signature numbers | Note pitch or accidental error | Staves read perfectly | Position accuracy | Failed |
|---|---|---|---|---|---|---|
| HOMR | 7.44% | 8.44% | 7.32% | 1,460 / 3,422 | 84.27% | 0 |
| OTS-HOMR | 4.52% | 4.57% | 4.07% | 2,132 / 3,422 | 88.62% | 0 |
Change in symbol error: −2.92 points (95% interval −4.05 to −2.03). The interval comes from resampling whole scores, so related staves are not counted as independent evidence.
Error by kind of symbol
| Kind of symbol | HOMR | OTS-HOMR |
|---|---|---|
| Pitch | 4.58% | 2.96% |
| Rhythm | 4.46% | 2.70% |
| Accidentals | 4.88% | 2.39% |
| Articulation | 3.18% | 1.81% |
| Slurs | 3.53% | 2.09% |
| Position | 3.15% | 1.71% |
Five-beat time signatures read correctly: HOMR 0 of 84, OTS-HOMR 48 of 84. Twelve-beat: HOMR 0 of 115, OTS-HOMR 70 of 115.
Voice and piano scans
300 staves from 13 scores
| Engine | Symbol error | Error incl. time-signature numbers | Note pitch or accidental error | Staves read perfectly | Position accuracy | Failed |
|---|---|---|---|---|---|---|
| HOMR | 5.76% | 5.86% | 5.93% | 97 / 300 | 87.31% | 0 |
| OTS-HOMR | 2.59% | 2.62% | 1.26% | 162 / 300 | 94.10% | 0 |
Change in symbol error: −3.16 points (95% interval −5.10 to −1.54). The interval comes from resampling whole scores, so related staves are not counted as independent evidence.
Error by kind of symbol
| Kind of symbol | HOMR | OTS-HOMR |
|---|---|---|
| Pitch | 2.17% | 0.78% |
| Rhythm | 2.04% | 1.23% |
| Accidentals | 3.53% | 0.75% |
| Articulation | 1.41% | 0.63% |
| Slurs | 1.59% | 0.84% |
| Position | 1.08% | 0.35% |
Twelve-beat time signatures: HOMR 0 of 2, OTS-HOMR 1 of 2 (too few to say anything).
What these numbers do and do not show
- They measure how well single staves are read. They say nothing directly about whole-page results, layout, or the lyrics and text features.
- The test staves were looked at during development, so these are descriptive benchmark figures, not a fresh test.
- The quartet figure comes from nine scores. The correct answers for all three sets were built by us and can contain errors.
- Both engines were run through the same software, not through each engine’s own full pipeline, so staff reading is compared like for like.