Evaluating Shazam-Style Fingerprinting and CLAP Embeddings Under Audio Degradation¶
This notebook presents results from evaluating two fundamentally different audio retrieval methods — deterministic fingerprinting (Shazam-style) and neural audio embeddings (LAION-CLAP) — against the GTZAN dataset under a battery of real-world degradations.
Structure¶
| Section | Topic |
|---|---|
| 1 | Dataset Overview |
| 2 | Shazam Fingerprinting Results |
| 3 | CLAP Genre Classification |
| 4 | CLAP Exact-Song Retrieval |
| 5 | Head-to-Head Comparison |
Evaluation Setup¶
Dataset: GTZAN — 1,000 audio tracks (10 genres × 100 tracks × 30 s).
Augmentations — 17,982 files generated in two categories:
| Category | Types | Levels |
|---|---|---|
| Additive Noise | White, Crowd, Street | SNR 0 / 10 / 20 dB |
| Musical Transforms | Pitch Shift Up, Pitch Shift Down, Lo-Fi Filter | 1 / 2 / 3 (semitones or severity) |
Models evaluated:
- Shazam: constellation-map fingerprinting, time-coherence matching. Full tracks indexed; 10 s random snippets queried.
- CLAP General: LAION-CLAP
630k-audioset-best.pt— broad multi-domain training. - CLAP Music: LAION-CLAP
music_audioset_epoch_15_esc_90.14.pt— music-specialist fine-tune.
Evaluation tasks:
- Genre classification: logistic regression trained on clean CLAP embeddings, tested on augmented.
- Exact-song retrieval: cosine similarity search — does the augmented embedding rank its clean original first?
- Shazam identification: Top-1 match; reported as correct / false-positive / reject.
1. Dataset Overview¶
The GTZAN dataset is perfectly balanced: 100 tracks per genre, 1,000 tracks total. Each augmentation type × level cell contains 999 files (one track per genre missing a clean source is excluded consistently).
Discussion: The dataset is clean and balanced — no genre is over-represented, and every augmentation cell has equal coverage. This means accuracy numbers are directly comparable across conditions without correction for class imbalance.
2. Shazam Fingerprinting¶
Shazam's landmark hashing relies on the exact time-frequency coordinates of spectral peaks. A match requires enough (hash, time-offset) pairs to align in the time-coherence scorer. This architecture is highly optimised for the specific failure mode it was designed for — the "noisy bar" scenario — but has zero tolerance for pitch shift.
2.1 Overall outcome breakdown¶
Discussion: Across all 17,982 queries, the bulk of failures are rejects (matcher returns None) rather than false positives. This is a safety property of the design: when the fingerprinter is uncertain, it withholds a guess. The 0.x% false-positive rate is essentially noise.
2.2 Noise vs transform split¶
Discussion: The noise/transform split reveals two completely different systems in one. Under additive noise the system is robust. Under musical transforms it collapses — nearly every query is rejected. The asymmetry is stark.
2.3 Accuracy vs condition severity¶
Discussion: Additive noise degrades accuracy very little, even at 0 dB SNR (equal signal and noise power). The lo-fi filter produces near-zero accuracy at all severity levels — the bandpass discards the high-frequency peaks that fingerprints depend on. Pitch shift is instant death at 1 semitone: every single query is rejected. Shazam's hashes encode exact frequency bins; a 1-semitone shift (~6%) moves every peak out of its bin.
2.4 Outcome composition per augmentation type¶
2.5 Per-genre accuracy¶
Discussion: Genre accuracy is remarkably uniform across noise conditions — no genre is consistently harder to identify than another. This suggests the fingerprinting performance is driven by the augmentation type, not by musical style.
2.6 Per-genre accuracy by severity¶
2.7 Genre × condition heatmap¶
Discussion: The heatmap shows the noise columns uniformly green and the pitch/lo-fi columns uniformly red regardless of genre. There are no outlier genres that resist transforms better than others.
2.8 Identification latency¶
Discussion: Rejects are fast — the matcher exhausts its hash-lookup table quickly and bails when there are no coherent candidates. Correct matches take longer on average (more candidate pairs must be assembled and scored). The median latency of ~2–3 s for a 10 s snippet is consistent with the DSP pipeline processing the full query before lookup.
2.9 Score / confidence distributions¶
Discussion: Correct matches cluster at high score/confidence values; false positives appear at lower scores. This distribution supports threshold-based filtering: raising ID_MIN_SCORE would convert most false positives into (safe) rejects at the cost of a small true-positive rate.
2.10 False-positive genre confusion¶
Discussion: When Shazam does produce a wrong answer (false positive), the wrongly-picked song is spread roughly uniformly across genres rather than concentrating within-genre. This suggests false positives are spurious hash collisions rather than musically-motivated confusions.
2.11 Summary table¶
| Condition | Level | Accuracy % | False Positive % | Reject % |
|---|---|---|---|---|
| Crowd Noise | 0 dB | 90.8 | 0.7 | 8.5 |
| Crowd Noise | 10 dB | 91.3 | 0.9 | 7.8 |
| Crowd Noise | 20 dB | 91.5 | 1.1 | 7.4 |
| Street Noise | 0 dB | 89.8 | 0.5 | 9.7 |
| Street Noise | 10 dB | 91.2 | 0.8 | 8.0 |
| Street Noise | 20 dB | 91.5 | 0.9 | 7.6 |
| White Noise | 0 dB | 91.3 | 0.9 | 7.8 |
| White Noise | 10 dB | 91.5 | 1.0 | 7.5 |
| White Noise | 20 dB | 91.5 | 1.2 | 7.3 |
| Lo-Fi Filter | L1 | 6.5 | 0.2 | 93.3 |
| Lo-Fi Filter | L2 | 7.7 | 0.2 | 92.1 |
| Lo-Fi Filter | L3 | 8.4 | 0.3 | 91.3 |
| Pitch Up | 1 st | 0.0 | 0.0 | 100.0 |
| Pitch Up | 2 st | 0.0 | 0.0 | 100.0 |
| Pitch Up | 3 st | 0.0 | 0.0 | 100.0 |
| Pitch Down | 1 st | 0.0 | 0.0 | 100.0 |
| Pitch Down | 2 st | 0.0 | 0.0 | 100.0 |
| Pitch Down | 3 st | 0.0 | 0.0 | 100.0 |
3. CLAP Genre Classification¶
A logistic regression classifier is trained on the clean CLAP embeddings of all 1,000 GTZAN originals and then evaluated on augmented queries. This tests whether the embedding space preserves genre identity under degradation, regardless of whether the specific track can be retrieved. CLAP General and CLAP Music are evaluated separately.
3.1 Overall accuracy — noise¶
Discussion: Both models perform well at 20 dB SNR. CLAP General degrades more gracefully at 0 dB. CLAP Music is competitive on noise — its failure mode is specific to transforms.
3.2 Overall accuracy — transforms¶
Discussion: CLAP Music collapses on transforms — pitch shift drives accuracy to approximately random chance (10% across 10 genres). CLAP General holds at 52–62% on pitch shift and 64–80% on lo-fi. The two models are not just quantitatively different; they are qualitatively different systems.
3.3 Per-genre accuracy by noise — CLAP General¶
3.4 Per-genre accuracy by noise — CLAP Music¶
Discussion: CLAP General shows consistent per-genre accuracy across noise levels. CLAP Music shows a tighter cluster — slightly better at high SNR but narrower margin before collapse.
3.5 Per-genre accuracy by transform — CLAP General¶
3.6 Per-genre accuracy by transform — CLAP Music¶
Discussion: CLAP Music genre accuracy under transforms is essentially uniform around 10% — the embedding has collapsed into a single region of the space (metal, in practice) regardless of the input genre. This is not random noise; it is a structured failure.
3.7 Accuracy by noise SNR and genre (heatmap)¶
Discussion: CLAP General's per-genre noise heatmap is uniformly green. Some genres (metal, rock) are harder at 0 dB SNR — likely because crowd/street noise masks the distorted-guitar texture that distinguishes them.
3.8 Confusion matrices¶
Discussion: CLAP General's confusion matrix shows genre-specific errors (rock ↔ metal, country ↔ pop) consistent with genuine musical similarity. CLAP Music's confusion matrix under transforms concentrates all predictions into a single genre column — the embedding collapse signature.
4. CLAP Exact-Song Retrieval¶
For each augmented query, we compute cosine similarity against all clean originals and ask: does the correct track rank first (Top-1)? Also reports Top-5 accuracy, genre-correct rate (right genre, wrong song), and wrong_song_right_genre rate (the fraction of retrievals that land in the right genre even when the exact song is wrong).
4.1 Top-1 exact retrieval — noise¶
Discussion: At 20 dB SNR, CLAP General achieves 61–77% Top-1 — strong for exact retrieval in a 1,000-track database with no metadata. CLAP Music is somewhat higher at clean/light noise but starts catching up at 20 dB where noise is mild enough not to corrupt tonal features.
4.2 Top-1 exact retrieval — transforms¶
Discussion: CLAP Music drops to ~0% exact retrieval on pitch shift and ~0–5% on lo-fi. CLAP General holds at 22–33% on pitch shift and 28–60% on lo-fi. The gap here is larger than in the genre task — exact retrieval is harder, so CLAP Music's collapse is even more pronounced.
4.3 Top-1 by genre — noise — CLAP General¶
4.4 Top-1 by genre — noise — CLAP Music¶
Discussion: CLAP General's per-genre noise retrieval is consistently high with moderate variation by genre. Classical and jazz tracks tend to retrieve well — they occupy distinctive embedding regions. Rock and metal overlap more.
4.5 Top-1 by genre — transforms — CLAP General¶
4.6 Top-1 by genre — transforms — CLAP Music¶
Discussion: CLAP General's transform retrieval by genre shows that pitch shift hurts all genres roughly equally — the model does not have a genre-specific pitch invariance. Lo-fi hurts genres with heavy high-frequency content (metal, rock) more than genres dominated by low-mid frequencies (blues, country). CLAP Music's transform panels are uniformly near-zero.
5. Head-to-Head Comparison¶
5.1 Summary¶
| System | Additive Noise (0 dB) | Pitch Shift (1 st) | Lo-Fi (L1) | Key failure mode |
|---|---|---|---|---|
| Shazam | ~90% correct | 0% (100% reject) | 6–8% | Destroys hash coordinates |
| CLAP General (genre) | ~86–92% | ~52–62% | ~64–80% | Graceful degradation |
| CLAP General (retrieval) | ~61–77% Top-1 | ~22–33% | ~28–60% | Graceful degradation |
| CLAP Music (genre) | ~88–92% | ~10% (chance) | ~10–20% | Embedding collapse |
| CLAP Music (retrieval) | ~65–80% Top-1 | ~0% | ~0–5% | Embedding collapse |
5.2 Key findings¶
Shazam is brittle to exact acoustic changes, robust to additive interference. The landmark hashing algorithm was engineered for a specific use case — the noisy-bar problem — and it excels there, achieving ~91% accuracy even at 0 dB SNR. But it has zero tolerance for frequency shifts: a 1-semitone pitch shift moves every spectral peak out of its hash bin, destroying every fingerprint. The lo-fi bandpass filter removes the high-frequency peaks entirely. In both cases the matcher cannot assemble a coherent time-coherence window and rejects (fails safe). The false-positive rate remains under 1% even under failure conditions.
CLAP General is the most robust system overall — not because it was designed to be, but because of breadth.
The 630k-audio-set general checkpoint was trained on environmental sounds, speech, and music together. This breadth means the embedding space is not tightly fitted to exact tonal relationships within music. When pitch shifts, the embedding drifts but does not collapse — it still lands in the right neighbourhood. The wrong_song_right_genre_rate of 38–46% on transforms confirms the model is doing semantic retrieval rather than exact matching: the wrong song, but the right genre.
CLAP Music's specialist fine-tuning is catastrophically counterproductive for robustness. The music-specialist checkpoint was fine-tuned to discriminate between musical characteristics — exactly the fine-grained tonal relationships that pitch shift destroys. When a query is pitch-shifted, its embedding moves to a region of the space that no training example occupies, and the model consistently predicts "metal" regardless of genre (a classic embedding collapse symptom). The cosine similarity between a pitch-shifted query and its clean original falls from ~0.99 to ~0.048 — the embedding is not just slightly wrong, it has moved to the opposite side of the space.
Shared fatal weakness: Shazam and CLAP Music both fail because they over-fit exact acoustic features. The paradox: the model purpose-built for music recognition is the worst at handling music-specific transforms.
5.3 Limitations and experimental bias¶
The augmented audio files are processed offline without codec re-encoding (no MP3/AAC artifacts) and without room impulse response. This gives Shazam a slight advantage in absolute accuracy versus real-world deployment. Since both methods are evaluated against the same augmented files, relative claims hold. See notes.md for mitigations if absolute accuracy numbers are needed.
CLAP evaluation is inherently easier than Shazam's because CLAP embeds the full augmented file (30 s), while Shazam queries with a random 10 s snippet. A fairer comparison would embed only a 10 s window for CLAP too — CLAP's numbers are therefore upper bounds relative to the Shazam evaluation protocol.