score from video

#what it is

a lot of drum and piano covers on youtube show the score on screen while the song plays. a playhead or a highlight moves across it, and the page changes every few bars. this app takes a list of those links and gives back one a4 pdf per video with just the notation, black on white.

it's a small korean windows gui: paste links, pick a folder, press start.

the windows app, captured on a real desktop by the build script

#how it works

the video is decoded twice. the first pass looks at about 140 frames and finds the score. the second samples at 4 fps and crops only that area. frames showing the same line of music get grouped, and each group becomes one strip by taking the per-pixel median, which removes the playhead and anything else that crosses the score for a moment. then the strip is turned into black on white and laid out on a4.

there are no per-video settings. the one thing every score in every video has is a staff line: a long, thin, perfectly horizontal ridge. so the app finds ridges in every frame, in both polarities, and lets the dark-on-light and light-on-dark readings vote.

#the hard parts

the vote counted rows, not lines. two videos from one channel came back with "no score found". their staff spacing was 10 px. every video i had tested sat between 14 and 24. the ridge detector used a 9 px tall kernel, and at 10 px spacing the 8 px white gaps read as bright ridges. four fat white gaps (32 rows) outvoted five thin black lines (10 rows), and the score got read inside out. voting with a 5 px kernel instead got 20 of 20 videos right.

median is right until it isn't. taking the minimum keeps anything dark that passes through, and whitening the playhead erases the notes under it. the median survives both. then one video painted a translucent blue box over the current bar and left it parked there. every frame in the group had the box in the same spot, so the median itself was white there and whole bars vanished. my first two fixes also fired on other videos and changed their output. what worked was measuring the overlay's shape in real runs: the parked highlight is one solid rectangle (fill 0.96 to 1.0), while coloured notation on other videos breaks into fragments (fill 0.63 to 0.94). so the restore only runs when the mask is one connected block filling at least 0.95 of its box.

repeated bars look identical. a new strip starts when a frame stops looking like the current one, but songs repeat. on one video a group held 46 frames where every other group held 22 or 23. two different systems with nearly the same notes got merged, and the median printed bar 110 and bar 114 on top of each other, so the last digit came out as a 0 and a 4 stacked into one glyph. the fix was to stop trusting the picture. the app reads the measure number in the leftmost strip of the header, and it tracks the playhead itself, which sweeps left to right and jumps back about 0.9 of the width when a new system starts. that jump doesn't care whether two systems look the same.

a defect from the test set: a fade-in copy of the intro (top) printed above the real one

counts lie. after one fix the system count stayed exactly 34, but one deleted bar had come back and one duplicate had gone away. so i wrote a check that ocrs the bar number off every printed strip and verifies the sequence steps the way the score does. it reads the actual page instead of a similarity score.

#what i did about regressions

most fixes moved a threshold that other videos depended on, so every change went through an 18-video gate where strips had to stay byte-identical and pages pixel-identical, unless the diff was the intended one. a wider set of 108 videos the client had converted was checked against the counts from the client's own runs.

that gate meant saying no to some fixes. one clipped articulation mark traced back to the top trim, but the trim was doing real work on 458 strips across 69 videos. loosening it would have let video imagery bleed onto lots of pages to save one mark, so i left it.

#where it is now

it went from 1.0.0 to 1.7.16 in three weeks. each release is built by a script on a windows vm, installed there and run on a real video before the client sees it. a 5 minute video converts in about 6 minutes, and most of that is decoding 1080p av1 twice. the known limit is physical: white ink on a near-white background carries almost no signal, so translucent overlays never come out fully clean.