Setting up a listening test to compare various types of files is not easy. You need a base of equipment and a trusted set of files to start. Set aside several hours and have a few people involved. We rarely do a test with fewer than three people. Here we'll explore some of the basics steps to compare files.
Keep in mind, the objective is to determine differences in formats, cables, devices, conversion tools, etc. You don't need to like the music. The question is "do you hear a difference". That difference might be insignificant, but if it repeatedly appears, then there is a difference. Differences may be covered and only appear in certain sections of the music, so it's important to do the test with different music and repeat the test. Again, if there are repeatable differences, regardless of how small or how few times it occurs, then that constitutes a difference.
Does it matter that these differences exist? Only to the listener. Our job is to determine either there is a difference or there is not.
Here's What You Need:
Two people to do the listening test. We'll call them the "Listeners"
Optional – a separate room where the devices are located and where playback occurs. If you don't have this, the second choice is to setup the playback device in a position where the Listeners can not see the Operator.
Pencils and paper to track answers (computers don't work for us because the typing noise and chances for movement from Listeners)
A high end speaker system capable of full frequency and dynamic results. Professional audio headphones (Beyerdynamic, Sony, and Sennheiser) work well. Avoid consumer headphones due to manipulation of phase response.
We suggest solid-state over vacuum tube amplification for testing (has to do with dynamic and frequency response, not whether the sound is pleasing -- we're looking for ways to hear very small differences).
Chairs positioned optimally for speaker response and where Listeners can hold their position without movement.
Acoustic music works best for testing because electronically produced music often has frequency range cutoffs from sampling or limitations of the electronics.
Look for music that is not over compressed and has full dynamic range. If you can find music that is "pre mastering" with full dynamic response, wonderful. Pop, rock and other music are often over-compressed for commercial release.
Piano can work well; orchestral is wonderful, acoustic jazz, folk and singer-songwriter, world music, etc. You don't have to like the music, just need something with full dynamics and frequency.
Stereo is a must for these tests – large hall sizes with reverb helps. Reverb trails are one of the easiest places to spot differences.
You'll only be listening to 15 to 30 seconds for the test. Start with 3 pieces of music or one with variety. You'll want to repeat the 15 to 30 second test several times.
Stillness And Hand
When changing devices or cables, there is a longer period of time involved between playbacks. These tests are best to setup where the Listeners cannot see the devices being switched. And it's important that the Operator add time to mask when doing a series of playback that might be AABABBA for example.
Compare Two Things At A Time –
As you get more sophisticated in your listening you may want to alter this test. For instance, when first listening to a clip you might find a spot where a reverb trail extends through a long silence or a high piano note appears in a specific area of the stereo image. Listen to that spot again – maybe as little as five seconds. Listen to the B clip in the same spot. Does the reverb trail die before the silence ends? Does the high note of the piano move to an outer part of the image? These are great giveaways to something changing.
Headphones Versus Speakers And What To Listen
Here Are A Few Things We Listen For To Dear
Frequency response... do each clip reach the lowest lows and highest highs?
Room changes... does the reverb of the room cut off at different times (shorter or longer) when a silent passage occurs.
Wideness of the stereo image... especially helpful to hear differences in formats. Listen for the wideness of an orchestral passage or other acoustic ensemble. Piano can work well for this too. As sampling rate increase, generally we've found wider imaging and more clarity with the widening.
Phasing issues... this is tricky to hear, but when comparing first and second generations of digital transfers at the DSD256 range, comparing FLAC conversion levels, etc, we have found that using an orchestral or multiple mic recording of live jazz can expose shifting phase issue (seemingly instruments move differently in various clips).
Noise floor changes... Funny enough, listening to the noise at the beginning, end or silent section of a music clip can offer an easy way to spot differences.
I'm sure there are many methods for comparison tests. If you have suggestions for other tests, please submit them.
Remember this is not about putting down anyone's listening style or beliefs. These are tests for us to determine "is there a difference". Listening to noises may not be your cup of tea, but if there is a difference, then there is a difference. That's all we're looking for.
When we sell our files at Blue Coast Music we want to stand behind what we sell with our ears. It's not a musical judgment. It is a technical judgment. If you prefer the sound of lower sampling rates because they sound "thicker" – wonderful! If you prefer the wider image with full frequency and dynamic response that DSD offers – wonderful!