Proceedings of the Institute of Acoustics

 

Challenges in the subjective evaluation of spatial audio in automotive environments

 

B. I. Bacila, Institute of Sound and Vibration Research, University of Southampton, UK
M. Orita, Institute of Sound and Vibration Research, University of Southampton, UK
F. M. Fazi, Institute of Sound and Vibration Research, University of Southampton, UK

 

1 INTRODUCTION

 

1.1 The car cabin as an environment for audio

 

In professional music production, the ultimate test for the final mix is often known as the “car test”¹. That is, when the mixing engineer takes the finished product out of the studio for the first time and plays it in a real environment; better said, in the environment in which it is expected it will be most often reproduced. Of course, a car is not a replacement for a set of studio monitors and a well-treated listening room. But this test is specifically conducted as a “trial by fire” due to the unfriendly nature of an automotive environment. It is no surprise that a well-balanced mix can suffer from significant degradation when reproduced through loudspeakers that are not positioned optimally in a car, or when very early reflections caused by near surfaces such as windows cause a harsh spectral distortion. However, as mentioned by Rumsey², a particular advantage of the automotive environment is that the positions of the listeners are known, along with detailed information about the audio system and the environment. Increasing efforts have been made in an attempt to quantify the general particularities of such an environment, and a white paper has been published by the AES Technical Committee on Automotive Audio (TC-AA)³, in which a set of initial metrics are proposed. These are, however, at the current time, not including any metrics describing spatial aspects of the sound field.

 

Even though the audio experience is increasingly important, the position of the loudspeakers in a vehicle is not always ideal, due to space and safety restrictions. The most common locations for the drivers are in the lower part of the doors for the woofers, and higher up for mid-frequency drivers and tweeters. Nonetheless, these positions are rarely ideal in terms of relative angles and symmetry to the seated listener. Whilst this may not be desirable for stereo reproduction, there are a few advantages that this placement may bring. For example, spatial audio can be reproduced more easily, since many loudspeakers are positioned all around a listener. With this, of course, the need for accurate spatial audio reproduction becomes important. The first step in trying to quantify the quality of spatial audio reproduction is to conduct some subjective evaluation. This step will highlight the limitations of the receiving end of the signal chain - the listener. Once the perceptual aspects are identified, a natural next step would be to measure certain physical properties of the signal chain, from environment alterations to properties of the signal itself. Ultimately, it is desirable to know which parameters could be adjusted in order to provide the listener with an enhanced spatial audio experience.

 

1.2 Listening tests in automotive environments

 

Subjective evaluation of reproduced audio is a well-researched field, with well-documented procedures and methodologies. For instance, a number of standards^4,5 provide detailed descriptions of method ologies, environments, test protocols and statistical analysis which may be used to ensure consistency between experiments.

 

Multiple studies have previously investigated the perceptual aspects of spatial audio reproduced in a vehicle. Due to the physical limitations of an automotive environment it is common that a virtual assessment is conducted. This can be done via different methods: auralization^6–10, Cross-talk Cancellation (CTC )¹¹, Ambisonics¹², Spatial Decomposition Method (SDM)¹³ and combinations of the previous¹⁴. It is indicated in the literature that certain aspects of the reproduced sound field are preserved in the virtual tests^7,8,10,15, although not much emphasis was placed on spatial attributes. Kaplanis et al.¹³ hypothesised that aspects such as externalisation and the ease of assessing spatial attributes may be improved by reproducing the SDM-aquired sound field through a large spherical loudspeaker array. Camilleri et al.¹⁴ noted that spatial attributes may not be preserved well enough in the binaural reproduction of the aforementioned experimental setup. Similarly, Hegarty et al.⁸ identified a number of differences in the preservation of attributes in a binaural reproduction system and recommended that head tracking should be used in virtual listening tests to unveil some dynamic cues involved in localisation.

 

Specific experiments, for instance those that do not require changing of the audio system or drivers, could be conducted in-situ. Whilst they are logistically more complex, they may be chosen over their virtual counterparts if the ecological validity of the environment is of greater importance, while also addressing some shortcomings. Virtual assessments may present a particular difficulty in reproducing the ecological validity of the actual environment, from the specific non-linearities, to the position of the driver seat and the confined space of the vehicle. These elements are integrated into the perceptual assessment of the overall experience. Furthermore, since previous literature is not entirely conclusive on how well the perception of different spatial attributes is preserved in a virtual environment, it is advantageous to still conduct in-situ experiments.

 

The decision to conduct an in-situ listening test in an automotive environment is not without particular challenges. Some of these challenges are identified and discussed in this paper. A number of practical solutions are proposed to contribute to the development of a methodology framework and of a set of good practices to be followed in the design and execution of in-situ automotive listening tests. The authors believe that sharing these findings with the scientific community will help in the preparation of subjective evaluation strategies and encourage colleagues to conduct in-situ experiments.

 

2 HARDWARE

 

2.1 Vehicle and test environment

 

The main piece of hardware involved in an in-situ automotive listening test will be an adequate vehicle that suits the experiment requirements. An important consideration in this instance is the number and location of the loudspeakers. More recent vehicles are equipped with loudspeakers in the ceiling as well as in the doors and on shelves, such that the former facilitate the reproduction of the elevated virtual sound sources. However, this is not currently common. It is desirable that all the loudspeakers needed for the test are part of the original vehicle design because of the constrained space inside the cabin. Mounting additional loudspeakers in the cabin is not a trivial task due to the constrained space and to what mounting surfaces are available. Furthermore, the acoustical properties of the original drivers will differ significantly from their added counterparts, as a result of the effects on sound radiation when a loudspeaker is mounted on the trim panels inside the car door. These effects comprise of baffling and vibration, which are governed by the elastic properties of the panel.

 

Whilst not critical, another consideration is the location of the audio amplifier and the ease of accessing it. An amplifier that is easily accessible through the car boot will significantly improve the preparation time of the experiment compared to an amplifier which is hidden under the driver seat and is not easily accessible.

 

During the test, the vehicle must be stored according to the needs of the experiment. The size of a car imposes limitations on the types of storage space that can be used for experiments. The typical listening rooms for audio experiments are designed around the size of a human and cannot accommodate for the placement of car. The latter also applies in the case of most anechoic chambers. One alternative is to store the vehicle in a garage. It is desirable for this environment to be acoustically dead, similar to a listening room, in order to guarantee minimal influence from the storing space on the acoustics inside the cabin. However, in most cases, this is not possible or too difficult to achieve.

 

When using a garage, it is important to maintain the ambient temperature, ambient air pressure, ambient humidity, and the CO₂ concentration of the air inside as constant as possible to ensure consistency across experimental results. This is because all these physical quantities affect the speed of sound in air¹⁶. Measuring and logging values of these quantities during experiments is also recommended, especially for long sessions. It is good practice not to treat the car cabin as a sealed enclosure that cannot undergo changes in atmospheric conditions. During a listening experiment, the doors of the car may be frequently opened and closed to allow subject and researcher to communicate. Furthermore, a human subject present inside the closed cabin for long periods of time may affect the temperature and CO₂ concentration.

 

Another alternative is to conduct the listening test with the car placed in an open field to simulate semi-anechoic conditions, for example, on the runway for driving training. However, such a choice further complicates the logistics of the experiment as it requires subjects to travel to an atypical location. Also, in this scenario, the car cabin may be more susceptible to changes in atmospheric conditions. In both alternatives mentioned above it is also important to take into account the potential for any external noise clashing with the listening experiments. For example, if a garage is close to a busy road, or if an open field is near train tracks or a farmland.

 

2.2 Loudspeakers

 

As mentioned previously, the original vehicle audio system should ideally be used for an in situ listening test. Thus, the ability to control each loudspeaker individually is fundamental. However, this may not be trivial since the majority of automotive entertainment systems do not offer multichannel interfacing capabilities to the end-user. To this end, a custom system must be implemented in order to gain access to the individual drivers. The most straightforward approach, is to bypass the default processing unit and amplifier and connect a custom multichannel amplifier directly to the loudspeakers. For instance, the Innosonix MA32-LP amplifier¹⁷, which enables connections via a Dante network¹⁸, is a compact, lightweight solution to the aforementioned issue. Notable advantages to this system are the flexibility of networked audio, resulting thus in only requiring one network cable for the connection between the amplifier and a computer. Furthermore, more complex audio processing can be conducted on a standard computer, offering a flexible development environment, which greatly facilitates the assessment of different processing algorithms. An example block diagram of the system described previously can be seen in figure 1.

 

Figure 1: Block diagram of the connections and processes used in a typical listening test in automotive environments.

 

A custom wiring harness as shown in figure 2 can be manufactured and connected directly to the audio harness, thus ensuring the reversibility of the modifications to the original state. Certain amplifiers may use connectors that are not readily available for purchase or which may be difficult to identify. In order to overcome this issue, one may decide to purchase a donor amplifier (which does not have to be in working condition, but the output connector must match the original one), remove the connector and manufacture the adapter cable.

 

In specific circumstances, multiple loudspeakers with similar positions may be combined into one single channel. In this way, content with standard channel configurations (e.g., 5.1, 7.1, 7.1.4) may be easily reproduced, with time re-alignment where necessary. It is to be noted that, unless already implemented in the loudspeakers, crossover filters must be designed and applied to the drivers used in each channel, in order to ensure that each loudspeaker operates in the frequency range for which it was designed for. It is important to highlight that an appropriate crossover filter must be designed for each loudspeaker driver that is part of a crossover network. This filter would be implemented at the audio playback stage, on the processing computer. Some loudspeaker drivers, typically tweeters, may already have a built-in electrical filter in their design. In these cases, the filters for the rest of the crossover drivers will need to be designed around this existing one.

 

Figure 2: Custom wiring adapter used for connecting the Innosonix amplifier directly to the vehicle’s loudspeaker harness.

 

When it comes to designing crossover filters, in general, it is important to take into consideration the magnitude and phase distortion that these filters have on the electrical signal fed to the respective drivers. The popular 4th-order Linkwitz-Riley filters¹⁹ can achieve constant magnitude in the overall crossover frequency response and steep slopes in the transition region from pass-band to stop-band. However, being causal IIR filters, these do not exhibit a flat group delay. In contrast, FIR filters, which are typically implemented as an equi-ripple design, can be forced to produce an overall flat magnitude response in the cross-over²⁰ while also achieving a flat group delay. However, FIR filters often require an impractically high order to achieve the same steep slope in the transition region and the same level of stop-band rejection as IIR filters.

 

Individual filters in a crossover are commonly designed by taking into consideration the resulting overall frequency-dependent directivity of the sound radiator²¹. This is because phase artefacts from individual filters can create phase differences between drivers in a cross-over at a given frequency, thus altering how the two drivers interact constructively or destructively. However, in a car, the directivity of a loudspeaker driver mounted in the door is complicated by the effects of baffling and panel vibrations while also being difficult to measure in the confined space of the cabin.

 

2.3 Determining the loudspeaker positions

 

One of the most important aspects necessary in the design of listening tests in a physical car cabin is the knowledge of the physical positions of the loudspeakers. Unless this information is provided by the manufacturer, this task needs to be performed as precisely as possible. A few different methods, with varying degrees of accuracy, can be adopted.

 

A crude and straightforward method is to use a laser level mounted on a short tripod via a base with a level and angle gradations. A tripod can be placed on one of the car seats such that the laser resides at the required location in space. The laser is rotated toward each loudspeaker, and the horizontal angle is recorded. However, once the laser is removed, there is not an option to accurately maintain a global origin point at which to place a listener.

 

A more involved, yet still straightforward method is to measure the distances between loudspeakers and reference points (anchors) with a string or a similar device, then apply triangulation²². For 3D space, at least 4 anchors are required in order to establish the coordinates of the loudspeakers relative to an arbitrary origin, but can be reduced to 3 if some simple assumptions are made on the loudspeaker location. The anchors must be chosen such that no two pairs of anchors have x, y or z coordinates that are very close to each other, otherwise high errors can occur from triangulation²². In the car, this may be difficult to achieve as the points that are both accessible and have line of sight to all loudspeakers are on the ceiling of the cabin and close to being on the same plane. The origin is typically chosen in a location relative to the interior of the cabin that is relevant to the listening experiment. With the global origin set, the Cartesian coordinates of each loudspeaker can be converted to spherical coordinates, and can be visualized in CAD or other software.

 

The distance-based procedure described above relies on representing the loudspeaker as a point in space. The determination of such a point has been the research of the literature on the acoustic center for a number of years^23,24. Such methods, however, require measurement of the directivity of the sound radiator, which is complicated to perform in the car cabin, as previously described.

 

More complicated methods exist to establish the positions of the loudspeakers relative to a coordinate system. For example, given an arrangement of sound sources and sound receivers, acoustic impulse response measurements can be used to establish the distances between items by analysing the position of the peaks in the responses. Then, from these few distances, the rest of the distances, and hence an arbitrary coordinate system, can be derived via algorithms based on the Euclidean distance matrix^25,26. However, such algorithms have not been tested in a car, where impulse responses are dominated by early reflections, making peak analysis impractical for distance inference. Another example is using photogrammetry to extract a 3D mesh of the interior surfaces of the cabin, as done by Fonseca et al.⁷, and then calculating all the required coordinates in a CAD software.

 

2.4 Vehicle battery

 

Ideally, when the vehicle is stored and listening tests are performed for longer periods of time, it is recommended that the main battery of the vehicle is disconnected if it is not absolutely necessary for the running of the experiment. Depending on the vehicle type, certain features may require the use of the internal battery. For instance, ensuring that all subjects are in the same position relative to the coordinate system for the loudspeakers will necessitate the use of the car seat adjustments. The majority of modern vehicles, however, use electric seat adjustments, which require the ignition to be on. It is to be noted that having the ignition on or even keeping the vehicle unlocked may drain the battery. Consequently, it is strongly recommended that a battery maintainer (i.e., a trickle charger) is used to avoid the premature discharge of the battery. Of course, the use of such a device must be conducted in accordance with all the health and safety measures that are in place for the specific vehicle, charger and battery.

 

3 LISTENING TEST DESIGN

 

When conducting a listening test in an environment that is already set, there are a few limitations which need to be considered. For instance, one would not be able to easily evaluate different audio systems, loudspeaker placements, or different environments, due to the physicality of the in-situ test. However, such an environment would be suitable for tests concerning different processing algorithms, where switching between stimuli can be instantaneous.

 

3.1 Subject position

 

Conventional listening tests require that the subject’s ears are at the same positions relative to the loudspeaker system, for each subject. Through the years, this has been achieved by adjusting the position and the height of the chair on which the subject is seated, by using lasers which are aligned with the loudspeakers.

 

In the case of an in-situ automotive listening test, a number of additional aspects must be considered. The loudspeaker system can not be used as a reference point for the listener position, since they are not at ear height or symmetrical to the centre axis of the head. Thus, a global origin at the centre of the listener head must be chosen with consideration to the height of the subjects. In an ideal scenario a few subjects of differing heights would be invited into the vehicle and asked to adjust the seat to a comfortable driving position. An average of these positions could be recorded and considered as the global reference listening position. Alternatively, the median human height of 168.5 cm can also be used for determining this position³. The AES Technical Committee on Automotive Audio (TC-AA)³ reminds of a specific designated user space called the “eye ellipse”, which, if known, can be used to determine a suitable global listening position, by translating it to an “ear ellipse”. This global origin should be used as the origin of the coordinate system in the previously described methods to establish the locations of the loudspeakers in the car. Once this position has been established, it can be marked by using a cross-line laser level. The positions of the horizontal and vertical beams can be marked on both left and right windows using white tape and a permanent marker. Since both the horizontal and vertical lines are marked on both windows, the plane will be fixed, and the lasers can be adjusted to these marks before each listening session. Finally, in the subject setup stage of the session, the experimenter would adjust the position of the driver seat such that the ears are located at the intersection of the planes indicated by the laser levels. This setup ensures that a consistent position between subjects is maintained, which is highly important in critical listening tests. The application of the aforementioned method is, however, usually limited to the front two seats of a vehicle, since the majority of the cars do not have adjustments for the rear seats.

 

In the case of experiments where the subjects’ head position must be maintained throughout the session, a head tracking device can be used. Such a device can monitor the movement of the head and notify the subject to reposition their head to the original position. A number of implementations can be used, however, a hardware tracker may not be suitable in situations where headphones are not used and there is therefore no physical mounting point for the tracker. Alternatively, a webcam-based head tracker²⁷ is an elegant solution, since the camera can be mounted directly on top of the steering wheel (as can be seen in figure 3), thus providing a secure mounting point. Such a tracker may be implemented on a Raspberry Pi and a small screen can be mounted in front of the instrument cluster for displaying instructions to the subject. For instance, if the position of the head is out of a certain tolerance range, instructions could read “Rotate left/right” or “Move left/right” until the position is within tolerance.

 

3.2 Interfaces

 

A wide range of user interfaces can be used, depending on the nature of the test. A simple tablet can be mounted to the steering wheel using detachable 3M Command™ Strips (figure 3). This method ensures that the tablet can be easily removed when not in use, and since it is not a permanent fixture, the vehicle can be restored to the original state after the experiment has concluded. Current iPads can natively be used for extending the screen of an Apple computer via Bluetooth and WiFi or via USB cable. This enables the experimenter to use the iPad as an input interface and even to monitor the responses of the subject in real-time. With such portable devices it is important to keep them charged, especially when long listening sessions are required. Thus, a safe approach is to leave the device plugged in during the experiments.

 

3.3 Visual bias

 

Different experimental methodologies require that the subjects do not see the positions of the loud speakers, in order to reduce the visual bias. In a sound localisation test, for example, a subject may be more inclined to rate the location of a stimulus close to the position of a physical loudspeaker if the latter can be seen, due to an extensively studied effect commonly known as the “Ventriloquism effect”²⁸. The interior of a car cabin may also be distracting and may be biasing due to its non-symmetric nature. Furthermore, the brand and model of the car may also bias subjects into thinking that certain features should sound better on more premium vehicle models. It is generally recommended that the environment of a listening test is rather neutral. To this end, acoustically transparent curtains may be added around the driver seat (seen in figure 4) to isolate any undesired visual biases. Since this visual alteration to the car cabin leads to a more confined space, caution must be taken, and the risks associated with claustrophobia should be assessed and mitigated accordingly.

 

Figure 3: iPad and webcam mounted on the steering wheel. The screen for the Raspberry Pi head tracker is fixed in front of the instrument cluster and can be seen through the steering wheel

 

Figure 4: Curtain surrounding the listener to avoid visual bias

 

4 CONCLUSION

 

This paper reviewed a number of challenges associated with in-situ automotive audio listening tests. It was discussed that whilst certain experiments may not allow for the use of a real environment, and virtual experiments are logistically more inviting, in-situ tests present a few advantages and should be considered whenever possible. Previous studies suggested that the ecological validity of the environment may be better preserved in-situ than in a virtual experiment. This is especially important in the case of investigations into the perception of spatial attributes in the automotive environment, which may not translate well in virtual representations. The obstacles associated with the vehicle storage and maintenance, audio reproduction through the original sound system, listening test design and environment setup, have been discussed in this work. It was, however, argued that whilst seemingly complex, these aspects can be mitigated through careful planning, thus providing a suitable listening test infrastructure. Finally, a number of good practices were proposed, with the outlook that they contribute to the development of a methodological framework which could be followed for the development of in-situ automotive listening experiments.

 

5 REFERENCES

 

1. G. Childs. The Final Mix: Using Your Car. http://ask.audio/articles/the-final-mix-using-your-car, 2017.

2. F. Rumsey. Automotive Audio: They Know Where You Sit. Journal of the Audio Engineering Society, 64(9):705–708, September 2016.

3. AES Technical Committee on Automotive Audio (TC-AA). In-car Acoustic Measurements, October 2023.

4. International Telecommunication Union. Recommendation ITU-R BS.1116-3: Methods for the subjective assessment of small impairments in audio systems, February 2015.

5. International Telecommunication Union. Method for the subjective assessment of intermediate quality level of audio systems, 2014.

6. D. Koya, R. Mason, M. Dewhirst, and S. Bech. A Perceptual Model of Spatial Quality for Automotive Audio Systems. Journal of the Audio Engineering Society, 71(10):689–706, October 2023.

7. W. D. Fonseca, F. R. de Mello, D. R. Carvalho, P. H. Mareze, and O. M. Silva. Measurement of car cabin binaural impulse responses and auralization via convolution. In 2021 Immersive and 3D Audio: From Architecture to Automotive (I3DA), pages 1–13, September 2021.

8. P. Hegarty, S. Choisel, and S. Bech. A Listening Test System for Automotive Audio – Part 3: Comparison of Attribute Ratings Made in a Vehicle with Those Made Using an Auralization System. In Audio Engineering Society Convention 123. Audio Engineering Society, October 2007.

9. F. Christensen, M. Lydolf, G. Martin, P. Minnaar, B. Pedersen, and W.-K. Song. A listening test system for automotive audio – Part 1: System description. In Audio Engineering Society Convention 118. Audio Engineering Society, 2005.

10. J. Reimes, A. Fiebig, T. Deutsch, and M. Oehler. Comparison of auditory testing environments for car audio systems. Fortschritte der Akustik-DAGA, 2017.

11. A. Azzali, G. Boreanaz, A. Farina, G. Irato, and G. Rovai. Construction of a car stereo audio quality index. In Audio Engineering Society Convention 117. Audio Engineering Society, October 2004.

12. T. Deutsch, J. Reimes, A. Fiebig, and M. Oehler. Relevance of stimuli and test environment in the perceptual evaluation of car audio systems. Applied Acoustics, 156:404–415, December 2019.

13. N. Kaplanis, S. Bech, S. Tervo, J. Pätynen, T. Lokki, T. van Waterschoot, and S. H. Jensen. Perceptual aspects of reproduced sound in car cabin acoustics. The Journal of the Acoustical Society of America, 141(3):1459–1469, March 2017.

14. J. Camilleri, N. Kaplanis, and E. De Sena. Evaluation of car cabin acoustics using auralisation over headphones. In Audio Engineering Society Conference: 2019 AES International Conference on Immersive and Interactive Audio. Audio Engineering Society, March 2019.

15. S. Bech, W. Ellermeier, J. Ghani, M.-A. Gulbol, and G. Martin. A listening test system for automotive audio – Part 2: Initial verification. In Audio Engineering Society Convention 118. Audio Engineering Society, May 2005.

16. O. Cramer. The variation of the specific heat ratio and the speed of sound in air with temperature, pressure, humidity, and CO₂ concentration. The Journal of the Acoustical Society of America, 93(5):2510–2516, May 1993.

17. Innosonix GmbH. Innosonix MA32/LP. https://www.innosonix.de/discontinued-products, 2024.

18. Audinate. Dante – One Connection. Endless Possibilities. https://www.getdante.com/, 2024.

19. S. Linkwitz. Active Filters. https://www.linkwitzlab.com/filters.htm#3.

20. R. Wilson, G. Adams, and J. Scott. Application of digital filters to loudspeaker crossover networks. Journal of the Audio Engineering Society, 37(6):455–464, 1989.

21. S. Cecchi, V. Bruschi, S. Nobili, A. Terenzi, and V. Välimäki. Crossover networks: A review. Journal of the Audio Engineering Society, 71(9):526–551, 2023.

22. Z. Parisek, Z. Ruzsa, and G. Gordos. Mathematical algorithms of an indoor ultrasonic localization system. Infocommunications Journal, 64(4):30–36, 2009.

23. J. Vanderkooy. Applications of the acoustic centre. In Audio Engineering Society Convention 122. Audio Engineering Society, 2007.

24. J. Vanderkooy. The low-frequency acoustic center: Measurement, theory, and application. In Audio Engineering Society Convention 128. Audio Engineering Society, 2010.

25. I. Dokmanic, R. Parhizkar, J. Ranieri, and M. Vetterli. Euclidean distance matrices: Essential theory, algorithms, and applications. IEEE Signal Processing Magazine, 32(6):12–30, November 2015.

26. S. Bouley, C. Vanwynsberghe, T. L. Magueresse, J. Antoni, and A. Outrequin. Microphone array positioning technique with Euclidean distance geometry. Applied Acoustics, 167:107377, October 2020.

27. D. Rocha Carvalho, W. D’Andrea Fonseca, J. Hollebon, P. H. Mareze, and F. M. Fazi. Head tracker using webcam for auralization. INTER-NOISE and NOISE-CON Congress and Conference Proceedings, 263(1):5071–5082, August 2021.

28. W. R. Thurlow and C. E. Jack. Certain determinants of the “ventriloquism effect”. Perceptual and Motor Skills, 36(3_suppl):1171–1184, June 1973.