October 2017

If You've Got Ears You Don't Need To Be Blind
Roger Skoff writes about a strange request for testing.
Article By Roger Skoff

  A genuinely strange thing happened to me just very recently. As part of a series of articles about phono playback gear that I'm writing for another publication, someone responded to the most recent installment by saying "Double blind test needed."

Normally, I might have ignored this or simply dismissed it as the work of one of the many trolls who lurk the internet, but this time, I was baffled: I had said nothing at all in the article that could in any way have been regarded as either subjective or judgmental and, therefore, require blind testing as proof.

In fact, although I did refer in one paragraph to some basic beliefs about Hi-Fi that we had all held to be true half a century ago, there was no modern opinion given at all. All I had done to that point in the series was to describe some early phono cartridges (crystal and ceramic), tell about how they worked (bending a piezoelectric element produced a voltage), and say that their very high output (hundreds or even thousands of time greater than the best modern cartridges); their comparatively very low price; and their naturally skewed frequency response (boosted bass; rolled-off treble) – all working together to need only the simplest, low-gain-no equalization electronics – made them the perfect products – even today – for very low-cost or high risk (cats walking on the turntable, for example) phono systems where the very best sound wasn't the primary concern. 

So what could the reader who wrote that comment have possibly been referring to when he demanded double-blind testing? Frankly, I don't know. But I do know that the issue of testing seems never to either be resolved or to go away.

Actually, it's not even testing, itself, that's the issue: Just ordinary testing – the kind of thing that we all do every day when, for example, we order something at a restaurant and decide whether or not we like it, or that manufacturers do when they test a prototype to determine if the product it could become is good enough to go to market – has, according to the people who present themselves as experts on this sort of thing, an inherent possibility of bias: Mothers love their own children, they reason, and may give them – even, perhaps, without knowing it – certain extra advantages, or may overlook certain shortcomings in them that they might find problematic in the children of others. Similarly, inventors may prefer their own products, or the person who pays money to buy something – a Hi-Fi "Tweak", for example – may have an incentive to believe that it's good so that he won't have to feel that his money was wasted.

Everything, they say, in our background, our circumstances, and even in our mood can contribute to bias and can mean that whatever testing we do can either be misinterpreted or may, even inadvertently, actually have been set-up to pre-determine a desired outcome. If that's the case, then what good is the testing? Why bother to test anything at all if we can't be certain we can rely on the results?

Obviously there is at least the possibility that these concerns are real. We might certainly have prejudices (pre-judgments) and biases (directing tendencies) that will affect the tests, themselves, or our ability to properly interpret their results. And if we do have such hindrances to our judgment, how can we get rid of them? The obvious way would be to discover them, see where they are and what they do, and then work to get rid of them. Unfortunately, simply the fact of those prejudices and biases might work to prevent us from either perceiving or remedying them. For it is far better to simply make it so that they can have no effect.

That's what "double blind" testing is supposed to do: In formal research, "blind" or "single blind" testing" is where the testee has no idea of which of two or more things is being tested. (In Hi-Fi, that would mean which product or product combination the testee is listening to.) Because of that, the testee can't have a favorite other than as determined by what he hears, because he doesn't know which of the things under test he's actually hearing. Even under single-blind testing, though, the person administering the test (the "tester") does know which product or test item or combination is being tested and he might, therefore, consciously or otherwise, communicate that to the testee, allowing the testee's bias to come into effect. The solution is "double-blind" testing, where neither the tester nor the testee knows which thing is being tested and neither finds out until after the testing is over.

That's all well and good, except for just two considerations: The first is that testing of any kind is of no value unless only one single variable can be isolated for testing, and the second is that when the thing to be tested is a matter of taste, the only meaningful test is whether you like it or not.

Where double-blind testing really shines is in testing things like pharmaceuticals: There, you use two groups; a "test" group and a "control" group, with neither the testers nor the testees knowing which group is which. All of the members of both groups will have been selected to have whatever condition the drug to be tested is hoped to cure, and to have it to, as much as possible, the same degree. Then the one group is given the medication to be tested and the other group is given a placebo, again, with neither the testers nor the testees knowing which group gets which. Both groups are then closely monitored for whatever period of time and, from whether or not the test group (once it's finally identified at the end of the testing), gets better than the control group, and, if so, to what degree, the testers can determine the efficacy of what they're testing.

That's it: Two groups of very similar people with only one difference between them – one group gets the drug and the other doesn't. That's what double blind testing works best for and, where it works, it's of immense value.

But what about Hi-Fi? Is it even possible to do something similar? I don't think so. Let's suppose you were to set up any system in any room to do the test in; is it really possible to build/find/treat a room so that it will have no effect on the sound? If not, will the same test still apply if done in a different room? What about the listening position within that room? Is it even possible for different people sitting in different places in the same room to hear the same thing? Does that mean that double-blind testing can only involve one person at a time? What about the entire AES test sessions that had as many as hundreds of people listening all at once? Were they valid? What about the test signal? If you use a 1 kHz tone, it will certainly be clearly audible, will be one single isolated factor, and will be constant for the entire test period, no matter how long, but what will it prove? Even using equipment or full systems that we know to sound different, will we be able to hear significant difference between 1 kHz played on one system, as compared to another? What if we use music? Most music is a series of changes – in level, in tone, in kind or number of instrument playing. What if the portion of it that we listen to doesn't include the one characteristic sound or section that would be most revealing of the real nature of what we're listening to? Is the test still viable? What about the people doing the listening? I personally am a great fan of imaging and soundstaging, but what if some other listener isn't? Will he even comment on it in his test report?

The fact is that to make the test simple – as with just a single tone – is to make it potentially meaningless and to make it make it complex and changing, with a real music signal, may violate the "single variable" rule and also make it meaningless.

I was talking just yesterday with John Curl, one of our industry's leading electronics designers, about testing. He, like I, with my own creations, believes strongly in the importance of testing a commercial product to make sure that it will be the best possible within its limiting parameters and do for its buyers all that it promises to do. For that, he initially uses the very best electronic test equipment available to tell him if he's getting close. Then he switches over to the very best test equipment of all, his ears, for the final adjustments and detail tuning. In at least that kind of thing, the ears of an expert are far better, quicker, and more revealing than any "blind" test.

As to that other issue that I mentioned earlier – whether we "like" something or not – did you ever own a pair of "specially"-tinted sunglasses? They're wonderful, and make everything you look at seem somehow "warmer" or "nicer" Certainly, I 'm aware that that's not how those things may actually look, but I "like" them better that way and, even knowing that what I'm seeing isn't reality, it pleases me and I prefer it.

So what good would a "scientific" test be that proves that my preferred vision isn't accurate? None at all; I'd still prefer it. It's the same thing with many Hi-Fi tweaks and goodies. If I like what they do, or even if I (as a result of the dread "placebo effect" or whatever else) just think that I like what they do, that's all that matters: Even though some guy may tell me that I'm wrong and that "double-blind" testing will prove it; what I like is what I'm going to listen to, every time I sit down to relax, close my eyes, and...

Enjoy the music.