July 2016

A Meta-Analysis Of High Resolution Audio Perceptual Evaluation
Article By Joshua Reiss

4. Conclusions
4.1 Implications for practice
The meta-analysis herein was focused on discrimination studies concerning high resolution audio. Overall, there was a small but statistically significant ability to discriminate between standard quality audio (44.1 or 48 kHz, 16 bit) and high resolution audio (beyond standard quality). When subjects were trained, the ability to discriminate was far more significant. The analysis also suggested that careful selection of stimuli, including their duration, may play an important role in the ability to discriminate between high resolution and standard resolution audio. Sensitivity analysis, where different selection criteria and different analysis approaches were applied, confirmed these results. Potential biases in the studies leaned towards Type II errors, suggesting that the ability to discriminate high resolution audio may possibly be stronger than the statistical analysis indicates. Several important practical aspects of high resolution audio perception could neither be confirmed nor denied. Most studies focused on the sample rate, so the ability to discriminate high bit depth, e.g., 24 bit versus 16 bit, remains an open question. None of the studies subjected to meta-analysis used headphones, so questions regarding how presentation over headphones affects perception also remain open. The meta-analysis also did not pursue questions regarding specific implementations of audio systems, such as the choice of filtering applied, the specific high resolution audio format that was chosen, or the influence of the various hardware components in the audio recording and reproduction chain (other than assessing potential biases that might be introduced by poor choices).

In summary, these results imply that, though the effect is perhaps small and difficult to detect, the perceived fidelity of an audio recording and playback chain is affected by operating beyond conventional consumer oriented levels. Furthermore, though the causes are still unknown, this perceived effect can be confirmed with a variety of statistical approaches and it can be greatly improved through training.

4.2 Implications for experimental design
Evaluation of high resolution audio discrimination involves testing the limits of perception, and it is clear from the presented meta-analysis that it is difficult to detect. It is thus important that good test procedures are carefully followed. In addition, the work herein suggests several recommendations for future experimental design in this field;

1. Training - Test subjects should be trained in how to discriminate, given examples and informed of their results in practice sessions before the test.
2. Experimental design – There are several issues in the experimental set-up that may lead to Type I or Type II errors. In all stages, the recording and playback system for high resolution audio needs to have sufficient bandwidth to reproduce the full range of frequency content. There should be no level imbalance or differences in processing between the signal paths for high resolution and normal resolution content. Distortion levels and dynamic range should be measured, and tweeters (if used) should be aimed at the listener. Where possible, this should be confirmed by measuring the end-to-end response of the playback system. In general, any potential artifacts, confounding factors or additional variables should be measured and accounted for.
3. Stimuli - The study authors should ensure that the stimuli contain high resolution content. Ideally, the signal received at the listener position should be measured to ensure that this is the case. Since little has been established about the causes of high resolution perception, a wide range of stimuli should be considered. Test signals should be used with care since they may lack whatever features are needed for perception. Also, long duration stimuli are preferred, with (where this is an option for the methodology) a sufficient interval between stimuli.
4. Methodology - In several studies, test subjects may have had multiple interpretations of the research question. Preference or quality questions may be clouded by the participants' prior assumptions, leading to Type II errors. The task given to subjects should be unambiguous, and all participants should have a similar understanding of that task.
5. Analysis – Analysis methods should be established prior to the experiment, and any post-hoc approaches should be clearly identified. An over-reliance on individual p values should be avoided, especially when there are a finite number of trials with dichotomous outcomes. Where possible, multiple comparisons should be corrected.
6. Reporting – A full description of the experimental set-up should be provided, including data sheets of the used equipment. The listening level at the listener position should be provided. Full data should be made available, including each participant's answers, the stimuli and their presentation (duration, ordering) in each trial.

4.3 Implications for meta-analysis
The work presented herein is one of a very few, if any, papers that have applied rigorous and formal meta- analysis techniques to studies in the field of perceptual audio evaluation, or more generally, psychophysics. It has shown that techniques designed for studies involving intervention and control groups can be applied to experiments involving repeated trials with dichotomous outcomes, typically lacking a control. Measures of risk difference or mean difference, and their standard errors, can be adapted to situations where the mean value of the control (in this case, correct discrimination by pure guessing) is determined by probability theory, rather This paper also uncovered interesting phenomena that needed to be considered in the analysis. Several studies, such as Oohashi 1991 and King 2012, showed evidence of Simpson's paradox, where opposite trends in the data may have led to little effect being observed. Others (Nishiguchi 2003 and Hamasaki 2004) may have employed an equivalent of the Martingale betting system, where an experiment was repeated with a participant until a lack of effect was observed (though this may also be considered a method of verifying an initial observation). And several studies had conclusions that may have suffered from the multiple comparisons problem (Yoshikawa 1995, Nishiguchi 2003, Hamasaki 2004, Pras 2010). Interestingly, several studies reported results suggesting that for some trials, participants had an uncanny ability to discriminate far worse than guessing (Oohashi 1991, Meyer 2007, Woszcyk 2007, Pras 2010).

We also uncovered an issue with the use of standard statistical hypothesis testing applied to multiple trials with dichotomous outcomes. This issue, which occurred in many studies, may lead to Type II errors, and to our knowledge has not been widely addressed elsewhere in the literature.

4.4 Future research directions
As previously mentioned, many proposed causes or factors in perception of high resolution audio could not be confirmed nor denied, and warrant further investigation. Some of these questions are particularly intriguing, such as differences in perception over headphones versus loudspeakers, the effect of spatial audio rendering, the effect of quantization, the effect of duration (e.g., the trade-off between short-term auditory memory and the persistent effect of exposure to high frequency content), and the identification of critical stimuli where differences between high and standard resolution are most easily perceived.

There is a strong need for several listening tests. First, it is important that all test results be published. Notably, there is still a potential for reporting bias. That is, smaller studies that did not show an ability to discriminate high resolution content may not have been published. Second, it would be interesting to perform a subjective evaluation incorporating all of the design choices that, while not yielding Type I errors, were taken in those studies with the strongest discrimination results, e.g., Theiss 1997 had test subjects blindfolded to eliminate any visual distraction. If these procedures are followed, one might find that the ability to discriminate high resolution content is even higher than any reported study. Finally, no research group has mirrored the test design of another team, so there is need for an experiment that would provide independent verification of some of the more high profile or interesting reported results.

Many studies, reviewed in Section 1, involved indirect discrimination of high resolution audio, or focused on the limits of perceptual resolution. These studies were not included in the meta-analysis in order to limit our investigation to those studies focused on related questions of high interest, and amenable to systematic analysis. Further analysis should consider these additional listening tests. Such tests might offer insight both on causes of high resolution audio perception and on good test design, and might allow us to provide stronger results in some aspects of the meta-analysis.

However, many of these additional studies resulted in data that do not fit any of the standard forms for meta-analysis. Research is required for the development of statistical techniques that either transform the data into a more standard form, or establish a means of meta-analysis based on the acquired data. Finally, further research into statistical hypothesis testing of (multiple comparisons of) multiple trials with dichotomous outcomes would be useful for interpreting the results of many studies described herein, and widely applicable to other research.

Additional data and analysis is available from https://code.soundsoftware.ac.uk/projects/hi-res-meta-analysis.

5. Acknowledgements
The author would like to express the deepest gratitude to all authors who provided additional data or insights regarding their experiments, including Helen Jackson, Michael Capp, Bob Stuart, Mitsunori Mizumachi, Amandine Pras, Brad Meyer, David Moran, Brett Leonard, Richard King, Wieslaw Woszczyk and Richard Repp. The author is also very grateful for the advice and support from Vicki Melchior, George Massenburg, Bob Katz, Bob Schulein and Juan Adriano.

References
[1] A. Flexer, et al., "A MIREX meta‐analysis of hubness in audio music similarity," in Int. Soc. Music Inf. Retrieval Conf. (ISMIR), Porto, Portugal, 2012.

[2] J. B. L. Smith and E. Chew, "A meta‐analysis of the MIREX structure segmentation task," in Int. Soc. Music Inf. Retrieval Conf. (ISMIR), Curitiba, Brazil, 2013.

[3] M. McVicar, et al., "Automatic Chord Estimation from Audio: A Review of the State of the Art " IEEE/ACM Trans. Audio, Speech Lang. Proc., vol. 22, 2014.

[4] E. Hemery and J.‐J. Aucouturier, "One hundred ways to process time, frequency, rate and scale in the central auditory system: a pattern‐recognition meta‐analysis," Frontiers in Computational Neuroscience, 03 July 2015.

[5] R. J. Wilson, "Special issue: High‐Resolution Audio," J. Audio Eng. Soc., vol. 52, p. 116, 2004.

[6] J. R. Stuart and P. G. Craven, "A Hierarchical Approach to Archiving and Distribution," in 137th AES Conv., 2014.

[7] H. van Maanen, "Requirements for Loudspeakers and Headphones in the "High Resolution Audio" Era," in 51st AES Int. Conf., 2013.

[8] J. R. Stuart, "Coding for high‐resolution audio systems," J. Audio Eng. Soc., vol. 52, pp. 117‐144, 2004.

[9] J. R. Stuart, "High‐Resolution Audio: A perspective," J. Audio Eng. Soc., vol. 63, October 2015.

[10] W. Woszczyk, "Physical and perceptual considerations for high‐resolution audio," in 115th AES Conv., New York, 2003.

[11] H. M. Jackson, et al., "The audibility of typical digital audio filters in a high‐fidelity playback system," in

137th AES Conv., Los Angeles, 2014.

[12] J. Vanderkooy, "A digital‐domain listening test for high‐resolution," in AES 129th Conv., San Francisco, 2010.

[13] B. Smagowska and M. Pawlaczyk‐Łuszczyńska, "Effects of Ultrasonic Noise on the Human Body—A Bibliographic Review," Int. J. Occupational Safety and Ergonomics (JOSE), vol. 19, pp. 195‐202, 2013.

[14] W. B. Snow, "Audible Frequency Ranges of Music, Speech, and Noise," J. Acoust. Soc. Am., vol. 3, pp. 155‐166, 1931.

[15] D. Gannett and I. Kerney, "The Discernibility of Changes in Program Band Width," Bell Systems Technical Journal, vol. 23, pp. 1‐10, 1944.

[16] H. Fletcher, Speech and hearing in communication. Princeton, New Jersey: Van Nostrand, 1953.

[17] T. Zislis and J. L. Fletcher, "Relation of high frequency thresholds to age and sex," J. Aud. Res., vol. 6, pp. 189‐198, 1966.

[18] J. D. Harris and C. K. Meyers, "Tentative audiometric threshold level standards from 8 to 18 kHz," J. Acoust. Soc. Am., vol. 49, pp. 600‐608, 1971.

[19] J. L. Northern, et al., "Recommended high‐frequency audiometric threshold levels (8000‐18 000 Hz)," J. Acoust. Soc. Am., vol. 52, pp. 585‐595, 1972.

[20] D. R. Cunningham and C. P. Goetzinger, "Extra‐high frequency hearing loss and hyperlipidemia,"Audiology, vol. 13, pp. 470‐484, 1974.

[21] S. A. Fausti, et al., "A system for evaluating auditory function from 8000–20000 Hz," J. Acoust. Soc. Am., vol. 66, pp. 1713‐, 1979.

[22] T. Oohashi, et al., "Multidisciplinary study on the hypersonic effect," in Interareal Coupling of Human Brain Function,Int. Congress Series vol. 1226, ed: Elsevier, 2002, pp. 27–42.

[23] B. Theiss and M. O. J. Hawksford, "Phantom Source Perception in 24 Bit @ 96 kHz Digital Audio," in 103rd

AES Conv., New York, 1997.

[24] N. Kanetada, et al., "Evaluation of Sound Quality of High Resolution Audio," in 1st IEEE/IIAE Int. Conf.

Intelligent Systems and Image Processing, 2013.

[25] K. Ashihara and S. Kiryu, "Audibility of components above 22 kHz in a harmonic complex tones," Acustica‐ Acta Acustica, vol. 89, pp. 540‐546, 2003.

[26] W. Bulla, "Detection of High‐Frequency Harmonics in a Complex Tone," in 139th AES Conv., New York, 2015.

[27] J.‐E. Mortberg, "Is dithered truncation preferred over pure truncation at a bit depth of 16 bits when a digital requantization has been performed on a 24 bit sound file?," Bachelor, Lulea University of Technology, 2007.

[28] M. L. Lenhardt, et al., "Human Ultrasonic Speech Perception," Science, vol. 253, pp. 82‐85, 1991.

[29] S. Nakagawa and S. Kawamura, "Temporary threshold shift in audition induced by exposure to ultrasound via bone conduction," in 27th Annual Meeting Int. Soc. Psychophysics, Herzliya, Israel, 2011.

[30] T. Hotehama and S. Nakagawaa, "Modulation detection for amplitude‐modulated bone‐conducted sounds with sinusoidal carriers in the high‐ and ultrasonic‐frequency range," J. Acoust. Soc. Am., vol.

128, November 2010.

[31] K. Krumbholz, et al., "Microsecond temporal resolution in monaural hearing without spectral cues?," J. Acoust. Soc. Am., vol. 113, pp. 2790‐2800, 2003.

[32] M. N. Kunchur, "Audibility of temporal smearing and time misalignment of acoustic signals," Technical

Acoustics, vol. 17, 2007.

[33] M. N. Kunchur, "Temporal resolution of hearing probed by bandwidth restriction," Acta Acustica united with Acustica, vol. 94, pp. 594‐603, 2008.

[34] M. Kunchur, "Probing the temporal resolution and bandwidth of human hearing," in Proceedings of Meetings on Acoustics, New Orleans, 2007.

[35] T. Muraoka, et al., "Examination of audio bandwidth requirements for optimum sound signal transmission," J. Audio Eng. Soc., vol. 29, pp. 2‐9, 1981.

[36] K. Ashihara, et al., "Hearing Thresholds in Free‐Field for Pure Tone above 20 kHz," in Int. Cong. Acoustics (ICA), 2004.

[37] K. Ashihara, et al., "Hearing threshold for pure tones above 20 kHz," Acoust. Sci. & Technology, vol. 27,

2006.

[38] K. Ashihara, "Hearing thresholds for pure tones above 16 kHz," JASA Express Letters, August 2007.

[39] M. Omata, et al., "A Psycho‐acoustic Measurement and ABR for the Sound Signals in the Frequency Range between 10 kHz and 24 kHz," in 125th AES Conv., San Francisco, 2008.

[40] M. Koubori, et al., "Psycho‐acoustic Measurement and Auditory Brainstem Response in the Frequency Range between 10 kHz and 30 kHz," in 129th AES Conv., San Francisco, CA, USA, 2010.

[41] J. N. Oppenheim and M. O. Magnasco, "Human Time‐Frequency Acuity Beats the Fourier Uncertainty Principle," Physical Review Letters, vol. 110, Jan. 25 2013.

[42] M. Majka, et al., "Hearing overcome uncertainty relation and measure duration of ultrashort pulses," Europhysics News, vol. 46, pp. 27 ‐ 31, 2015.

[43] T. Oohashi, et al., "High‐Frequency Sound Above the Audible Range Affects Brain Electric Activity and Sound Perception," in 91st AES Conv., 1991.

[44] T. Oohashi, et al., "Inaudible high‐frequency sounds affect brain activity: hypersonic effect," J. Neurophysiology, vol. 83, pp. 3548‐3558, 2000.

[45] R. Yagi, et al., "Auditory display for deep brain activation: Hypersonic effect," in Int. Conf. Auditory Display, Kyoto, Japan, 2002.

[46] R. Yagi, et al., "Modulatory effect of inaudible high‐frequency sounds on human acoustic perception," Neuroscience Letters, vol. 351, pp. 191–195, 2003.

[47] A. Fukushima, et al., "Frequencies of Inaudible High‐Frequency Sounds Differentially Affect Brain Activity: Positive and Negative Hypersonic Effects," PLOS One, 2014.

[48] R. Kuribayashi, et al., "High‐resolution music with inaudible high‐frequency components produces a

lagged effect on human electroencephalographic activities," Clinical Neuroscience, vol. 25, pp. 651‐ 655, June 2014.

[49] S. Han‐Moi, et al., "Inaudible High‐Frequency Sound Affects Frontlobe Brain Activity," Contemporary Engineering Sciences, vol. 23, pp. 1189 ‐ 1196, 2014.

[50] M. Honda, et al., "Functional neuronal network subserving the hypersonic effect," in Int. Cong. Acoustics (ICA), Kyoto, 2004.

[51] T. Oohashi, et al., "The role of biological system other than auditory air‐conduction in the emergence of the hypersonic effect," Brain Research, vol. 1073‐1074, pp. 339‐347, 2006.

[52] M. Higuchi, et al., "Ultrasound Influence on Impression Evaluation of Music," IEEE Pacific Rim Conf. Comm., Comp. and Sig. Proc. (PacRim), Aug. 2009.

[53] M. Kamada and K. Toraichi, "Effects of ultrasonic components on perceived tone quality," in IEEE Int. Conf. Acoustics, Speech, and Signal Processing, 1989.

[54] S. Yoshikawa, et al., "Does high sampling frequency improve perceptual time‐axis resolution of digital audio signal?," in 103rd AES Conv., New York, 1997.

[55] R. Yagi, et al., "A method for behavioral evaluation of the "hypersonic effect"," Acoust. Sci. & Tech., vol. 24, 2003.

[56] D. Blech and M. Yang, "DVD‐Audio versus SACD: perceptual discrimination of digital audio coding formats," in AES 116th Conv., Berlin, 2004.

[57] A. Marui, et al., "Subjective evaluation of high resolution recordings in PCM and DSD audio formats," in AES 136th Conv., Berlin, 2014.

[58] T. Nishiguchi and K. Hamasaki, "Differences of Hearing Impressions among Several High Sampling Digital Recording Formats," in 118th AES Conv., Barcelona, 2005.

[59] G. Plenge, et al., "Which Bandwidth is Necessary for Optimal Sound Transmission?," J. Audio Eng. Soc., vol. 28, pp. 114‐119, 1980.

[60] K. Ashihara, "Audibility of complex tones above 20 kHz," 29th Int. Cong. and Exhibition on Noise Control

Engineering (InterNoise), 2000.

[61] K. Ashihara and S. Kiryu, "Detection threshold for tones above 22 kHz," in 110th AES Conv., Amsterdam, 2001.

[62] K. Hamasaki, et al., "Perceptual Discrimination of Very High Frequency Components in Musical Sound Recorded with a Newly Developed Wide Frequency Range Microphone," in 117th AES Conv., San Francisco, 2004.

[63] E. B. Meyer and D. R. Moran, "Audibility of a CD‐standard A/D/A loop inserted into high‐resolution audio playback," J. Audio Eng. Soc., vol. 55, pp. 775‐779, 2007.

[64] T. Nishiguchi, et al., "Perceptual discrimination of very high frequency components in wide frequency range musical sound," Applied Acoustics, vol. 70, pp. 921–934, 2009.

[65] H. M. Jackson, et al., "Further investigations of the audibility of digital audio filters in a high‐fidelity playback system," J. Audio Eng. Soc. (to appear), 2016.

[66] A. Pras and C. Guastavino, "Sampling rate discrimination: 44.1 kHz vs. 88.2 kHz," in AES 128th Conv., London, 2010.

[67] S. Yoshikawa, et al., "Sound Quality Evaluation of 96kHz Sampling Digital Audio," in 99th AES Conv., New

York, 1995.

[68] T. Nishiguchi, et al., "Perceptual discrimination between musical sounds with and without very high frequency components," in 115th AES Conv., New York, 2003.

[69] W. Woszczyk, et al., "Which of the two digital audio systems best matches the quality of the analog system?," in 31st Int. AES Conf., London, 2007.

[70] M. Mizumachi, et al., "Subjective Evaluation of High Resolution Audio Under In‐car Listening

Environments," in 138th AES Conv., Warsaw, 2015.

[71] R. Repp, "Recording Quality Ratings by Music Professionals," in Int. Comp. Music Conf. (ICMC), New

Orleans, 2006.

[72] R. King, et al., "How Can Sample Rates be Properly Compared in Terms of Audio Quality?," in 133rd AES Conv., 2012.

[73] B. C. J. Moore, " The role of temporal fine structure processing in pitch perception, masking, and speech perception for normal‐hearing and hearing‐impaired people," J. Assoc. Res. Otolaryngology, vol. 9, pp. 399‐ 406, 2008.

[74] P. G. Craven, "Antialias filters and system transient response at high sample rates," J. Audio Eng. Soc., vol. 52, pp. 216‐242, 2004.

[75] G. S. Thekkadath and M. Spanner, "Comment on "Human Time‐Frequency Acuity Beats the Fourier Uncertainty Principle"," Physical Review Letters, vol. 114, 2015.

[76] D. Dranove, "Comments on "Audibility of a CD‐standard A/D/A loop inserted into high‐resolution audio playback"," J. Audio Eng. Soc., vol. 58, p. 3, March 2010.

[77] J. P. T. Higgins and S. Green, Eds., Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. The Cochrane Collaboration, 2011, p.^pp. Pages.

[78] ITU, "Subjective assessment of sound quality, Recommendation BS.562‐3," 1990.

[79] F. A. A. Kingdom and N. Prins, Psychophysics: a practical introduction: Academic Press, 2009.

[80] L. R. Rabiner and J. A. Johnson, "Perceptual Evaluation of the Effects of Dither on Low Bit Rate PCM Systems," The Bell System Technical Journal, vol. 51, September 1972.

[81] P. Kvist, et al., "A Listening Test of Dither in Audio Systems," in 118th AES Conv., Barcelona, 2005.

[82] M. Egger, et al., "Bias in meta‐analysis detected by a simple, graphical test," BMJ vol. 315, pp. 629‐634, 1997.

[83] J. Lau, et al., "The case of the misleading funnel plot," BMJ, vol. 333, pp. 597–600, 2006.

[84] J. J. Deeks and J. P. T. Higgins, "Statistical algorithms in Review Manager 5," Statistical Methods Group of the Cochrane Collaboration, August 2010.

[85] N. Mantel and W. Haenszel, "Statistical aspects of the analysis of data from retrospective studies of disease," J. National Cancer Institute, vol. 22, pp. 719‐748, 1959.

[86] R. DerSimonian and L. N., "Meta‐analysis in clinical trials," Controlled Clinical Trials, vol. 7, pp. 177‐188, 1986.

---> Back to first page.