Tracktailor - ML experiment
An exploratory music-editing experiment
I listen to somewhere between three and seven hours of music a day. No joke.
Every time I listen to a track, I wonder whether it would work in a showreel or an ad. If it does, I start editing it in my head: finding the climax, picking a good opening and ending, and figuring out where to cut so it fits into a minute without breaking the rhythm. Don’t ask me why. That’s just how my brain works.
That habit gave me an idea: could I get a program to find those cuts for me?
That’s what Tracktailor is built around: an intro, a climax and an ending. Find those moments, then work out how to connect them so the track fits into a minute and still goes somewhere.
What I’m really curious about is how to turn something I do by ear into code. Why does one cut work while another sounds wrong? Could I start with a few rules, then train a model on transitions I think work or don’t? It’s a chance to get into machine learning with something I actually want to listen to.
The obvious approach is to find a minute that works on its own and leave it as it is. Sometimes that’s enough. But the intro I want is at the beginning, the climax comes two minutes later, and the ending is somewhere else again. To bring them together, I need to remove what’s in between and find joins that sound right. So I start by looking at what I can measure in the audio to find passages that might fit together.
I start in Python with librosa, an audio analysis library. It lets me estimate the tempo, locate beats and measure how the sound changes throughout a track. I can also use Beat This to locate beats and the first beat of each bar. These give me a grid to work with: if I cut out on the second beat of a bar, I look for a way back in on another second beat so the rhythm stays in place.
Next, I need to describe what’s happening around those points. Energy gives me a sense of the signal level: a large difference between two passages can mean a sudden drop or jump in volume at the cut. Chroma groups harmonic content into twelve pitch classes, combining notes across octaves. It lets me compare the harmony without having to identify every chord. MFCCs and spectral contrast describe other aspects of the sound’s texture. Two passages can have a similar volume but sound very different; these measurements help me tell them apart.
With Matplotlib, I put this together in a graph showing changes in energy and texture, along with the estimated sections of the track. It helps me connect what I’m measuring with what I’m hearing.
I also explore Demucs, a pretrained model that can separate vocals from the backing track. Measuring the energy of the isolated vocal track gives me an estimate of where someone is singing, so I can flag cuts that might interrupt them. It doesn’t tell me where a word or phrase ends: it detects vocal activity, not what’s being sung.
Using those reference points, I generate candidate transitions.
To avoid testing every possible pair, I start by looking for cuts on the beats. I try picking the track back up four bars later, then eight, twelve, and so on. In a track with four beats per bar, that means removing 16, 32 or 48 beats: if I leave on a second beat, I come back on a second beat. I also rule out jumps that are too short, too long or too close to the start or end of the track.
I then compare the sound around both positions. I take four beats at each one: two before the cut point and two starting there. I keep them in order so I’m comparing a short musical sequence rather than a single average. I measure how similar the sequences are using chroma, MFCCs and spectral contrast, giving more weight to chroma. I discard candidates that are too different.
Then I score the remaining candidates. Similarity between the sequences is the starting point: I lower the score for a large energy difference and add a bonus when both points fall on the first beat of a bar. I also slightly favour cuts that leave near the start of a section. If vocals are detected around either end of the jump, I penalise the transition, or exclude it when vocal protection is enabled.
I don’t really know where I’m going with this. I’m just assuming, a bit naively, that if I keep listening and give it enough time, I’ll end up with something. I listen to the result, notice what sounds off and try to understand why. One cut breaks the rhythm, another interrupts a vocal, another makes the energy drop all at once. I adjust the criteria as these problems come up, then listen again to see whether it actually helps.
The top ten candidate transitions, shown as arcs at the bottom of the graph, are the ten highest-scoring results.
I still need to choose cuts that work together and get me to the right duration. I draw on Music Rearrangement Using Hierarchical Segmentation, which connects comparisons between beat sequences with the structure of a track and the search for a complete arrangement. I use some of those ideas in a simpler implementation. Scalable Music also helps me think about the edit as a whole: choosing cuts that reach a given duration while keeping the important passages.
By default, I focus the search on how much time I need to remove. To go from three minutes to one, I look for a combination of jumps that removes two minutes without overlapping, while leaving enough music between cuts. I favour higher-scoring transitions and avoid piling on too many cuts, aiming to land within one second of the requested duration. I can also protect the highest-energy section so it doesn’t get cut out.
I build several possible edits in parallel. At each step, I try adding cuts, compare the options and keep a small number of promising ones to explore further. I take into account the estimated quality of the transitions, the resulting duration and the passages I want to keep. A highly rated cut can still get left out if another combination makes more sense as a whole.
Once I’ve chosen the passages, I join them with crossfades: for a brief moment, one sound fades out as the other takes over. That helps smooth the join, but it won’t fix a phrase cut in the wrong place. Across twelve edits of four tracks, the results are all within one second of the target duration. Whether they work musically still comes down to listening.
Here’s an example saved from the early tests: a version of “Hold me” shortened from about 2:46 to 1:19.
Listening makes it clear that these scores aren’t enough. Two identical chords can belong to completely different phrases. A transition can keep the beat but interrupt a melody or a word. And several cuts that sound fine on their own can leave a track going nowhere. To take this further, I need examples of what actually sounds good, not just what scores well against my criteria.
To look into this, I build an annotation interface with Flask. It plays short extracts around a proposed cut and lets me rate them from one to five. One means a distracting break; five means a transition I barely notice. I mix the algorithm’s highest-ranked candidates, its lowest-ranked candidates and random examples.
I’ve annotated 450 cuts across nine tracks. Honestly, listening to and rating all those transitions nearly drove me mad. Annotating music is exhausting: hearing the same passages over and over, judging tiny differences and trying to stay consistent across hundreds of examples is really difficult.
I then try predicting my ratings from the audio features, using LightGBM and simpler models, with pandas and scikit-learn for data preparation and evaluation. To see how well this carries over to other music, I leave one track out: train on eight, evaluate on the ninth, then repeat for each track.
With LightGBM, I use shallow trees and regularization to limit overfitting: fitting the training examples closely but doing poorly on other tracks. I also include measurements of changes in volume, attacks and harmonic content. Even so, the predictions often stay close to an average rating and miss many of the transitions I’m trying to find.
To see whether the model adds anything, I compare it with a very simple approach: always predict the median rating from the training examples. Here’s the average error on the tracks left out of training:
| Method | Mean absolute error |
|---|---|
| Predict the median training rating | 0.840 |
| Ridge regression | 0.929 |
| Regularized LightGBM | 0.853 |
Error is measured in points on the 1–5 rating scale; lower is better.
LightGBM doesn’t beat the median on this test. So this experiment gives me no measurable improvement in predicting the ratings. The cuts you can listen to here are selected using the criteria described earlier; they don’t show that the model has learned to make better edits.
Looking back, I can see a few limits in how I’ve approached this. I have 450 annotations, but they come from just nine tracks. Many examples are similar, which doesn’t give the model much variety to learn from before judging unfamiliar music. I also have relatively few really good transitions to show it. More varied examples, with more successful cuts among them, would be one thing to try.
My ratings are another limitation. I’m the only person rating the cuts. After listening to the same passages over and over, it gets hard to judge each one by the same standard. I’m asking the model to reproduce a judgment I struggle to make consistently myself. Having several people listen to the same cuts would help show which ones we actually agree on.
Finally, I describe each transition through measurements of rhythm, energy and texture. But when I listen, I’m also judging the melody, where a phrase ends and what I expect to hear next. The information I give the model may not be enough to explain my ratings. And predicting a rating with a small error doesn’t quite answer my original question: out of all the possible cuts, which ones should I keep? These are possible explanations for the result, not causes I’ve tested and confirmed individually.
I spent three days on this experiment, and that’s enough for me. I wanted to try an idea and see how far I could get from something I hear in my head to an automatic edit. I’ve come away with a few cuts I like, and a much clearer idea of the work involved in collecting useful data and checking whether a model actually helps. Taking it further would mean spending more time on the annotations and how I evaluate the results. I’m leaving it there for now.
Here’s a short extract with a cut five seconds in.
It’s satisfying when the cut works.
I’m curious: what do you think of this one?
Loading a cut…