Proceedings of the Institute of Acoustics

Beyond the frame: textural realisation in cinema sounds

S. Coleclough, University of Staffordshire, Stoke on Trent, UK

1 INTRODUCTION

Worldbuilding using audio is a key aspect of cinema, particularly in expanding the mise-en-scène and the narrative beyond the frame (Bresson¹; O’Brien²; Skjerseth³). At the same time, audio is still underrepresented as a key contextualising component in the transfer of production design to the conceptualised paracosm of the film’s world (Silvey and McKeith⁴; Shone⁵). The texturing of the seen and unseen, that which lies outside of the frame, becomes a focus of the development of the role and importance of sound when aligned to visual works.

This work proposes that technical development in sound production and delivery, and the increased capacity of game engines to utilise sound creatively means that audio can begin to become more central in the delivery of immersion so sought by screen entertainments. Therefore, the textural qualities (Garcia⁶) of cinema sound will be considered and the ways in which audio concerns can be used to develop upon and extend the fictional work. Through such application a consideration of the potential to invoke a sonic mirroring and augmentation of the visual production design and in so doing support or further the concept of immersion when considering the experience of an audience member in relation to a moving image product.

In this vein we ask how we might challenge and extend the concept of the place of sound as an established reflector and enable it, at times, to become instigator and active motivator within both cinema and in turn game audio design. With this ambition in mind, we look to position audio within this work as a sensory contributor and reinforcer beyond the obvious use of volume and high dBs used in cinema exhibitions. This work therefore proposes if we embrace the textural ambition and potential of sound, we unlock another opportunity for immersion within screen entertainment. By engaging with the psycho-acoustic properties of audio we broaden the creative brush with which the sound designer can paint. In encouraging those learning the discipline to push a little further than representation we as educators can encourage the extension of this craft into the sensory contribution to a film or game and enable the immersion to become that little bit deeper for viewer or player. Where cinema has led so game has followed albeit with a slightly altered application of sound, using primarily non-linear approaches as opposed to linear found in older visual media. How audio is used in cinematics, however, provides a bridge between the two and opens the opportunities for an extension of the use of sound and sound design within the game industry.

These approaches and conceptualisations are then extended to consider how they can be communicated to those beginning careers in sound design and encourage them to think creatively as well as causally about sound within the narrative. The work specifies a set of bespoke activities which can be used to assist students in developing their understanding of audio and its potential relationship to sensory experiences. The exercises move the student past the initial craft application of Foley, mixing and layering and encourage experimentation and considered creativity which extends the moving image work.

2 OVERVIEW OF APPROACHES TO AUDIO IN FILM AND GAME

2.1 Where the Visual Leads and Audio Follows – As It Is As It Ever Will be?

The traditional approach to audio where it is aligned with a moving image product is that the sound is informed by the visual. In this way it is the reflection of the picture rather than the instigator of content or meaning. Indeed, the primacy of the image to screen products follows a tradition found originally within theatre which has in turn transferred to cinema and so to game, as we shall explore later. The image leading the aural and controlling the presentation of narrative information or ambiance creation is one which is accepted as rote. As sound designer Walter Murch confirms audio is “...considered the inevitable (and therefore mostly ignored) accompaniment of the visual — stuck like an insubstantial, submissive shadow to the object that "caused" them”⁷. When we teach the concepts of audio design, we establish the importance of synchronisation of the sound to the image, Murch’s shadowing, and establish audience expectation as central to their, the viewer’s, acceptance of the sounds created for picture. We confirm the submission of sound to the visual and in addition, we teach the importance of fidelity of the audio to the image, see Bordwell and Thompson⁸. In this case fidelity corresponds to the visual connection of a known sound being shown to emanate from that source. With this connection in mind the audience then looks, albeit unknowingly, to apply Chion’s causal link to the sound and image relationship⁹. Matching from expectation and experience the validity of the connection offered the spectator then accepts the linking of image and audio. Skjerseth identifies the tradition in animation as one connected strongly to musical, where instruments and sound techniques can extend the established use of mickey mousing and leitmotifs¹⁰. Such a linkage reinforces the concept of synchresis offered by Chion¹¹. The ability to connect and accept that a visual creates an audio element when synchronised on screen, even if that audio aspect is not actually realistic or real, is the assistive method of sound design. Where creatively attached yet not always literally created sounds are tied by an audience through the visual synchronisation to the sound. As can be seen in the willing linking of musical cue and gestural move to the visual, the audience is ready and disposed to make connections.

As we shall see later in this work, this approach of synchronicity and fidelity is not to curb the creative drive of the students learning their craft, but to ensure that the foundation of the practice is critically present and that the audience will accept the sound which is delivered. Via this application of audio to image we are confirming their relationship and solidifying the created world on screen for the viewer. An audience is capable of many leaps of the imagination and temporal understanding of the way in which sound is applied in a film or a game for example the use of meta-diegetic sound where the viewer hears what a character hears sharing a special connection and experience, but the key to such stretches is that the solid foundation of image connected to audio is present.

Another established precept when working with sound and picture is that the audio considerations and needs of a moving image product are usually added when the picture, in the case of cinema, is locked. As locking the picture is a final process at the close of the post-production cycle this means that the positioning of the audio editing work is traditionally one of the last points in the completion of a film. With such a workflow in mind it is easy to see why the audio has continued to “follow” the visual as an approach. It is simple to see the reasonability of the connection – picture leads as the audience have knowledge of how sound works, audio and visual work in synch therefore the sound is led by the image and the tangibility of the visual actions in the world on-screen.

In cinema or television, we rely on sound generally to fill in the spaces. It is the indicator of space outside of the frame and as such tends not to drive the narrative space but support it. A reminder to the viewer that there is a world beyond which carries on around the important visual information being delivered via the edit. Such an approach has been carried through to game worlds and their narratives via cinematics but also using ambiance during in-play portions of the game. In a game the audio therefore also assists in the delivery of the game play, ostensibly building an ambiance which fills in the sonic space or which offers elements of spatial placement or gameplay guidance for the player. In each case the impetus for the application of the sonic to the playable space tends to rely on the visual indicators rather than solely auditory ones. Where sound is applied for game play, so as a narrative assistance or confusion, the positioning of the audio suggests checking or exploration for the player rather than direct identification of a mechanic or specific playable aspect. These statements are not to lessen the use of sound within moving image products but to identify where the traditional approaches to the audio/visual relationship stem from. It is via this knowledge that we can then suggest that there are approaches to audio which can strengthen the opportunities of immersion for a viewer or player, and which, with consideration and experimentation, can drive the overall sensory experience offered via visual or interactive narrative.

Therefore, we challenge the status quo of the sound and picture connection, not at its root but at its reach. In answering the question posed in the heading of this section we find that this does not have to be a continuation of approach but an augmenting and developing as we find the opportunities of immersion furthered by the expansion of sound delivery within cinema and the game engine. If the audience, as earlier indicated, can make leaps of faith with the audio and visual connection this can be further developed and built upon. This work would suggest that such a change in approach requires a shift in the way in which we work with those learning the craft of audio design to encourage the challenging and extending of the status quo.

2.1.1 Worldbuilding – Sonic Texture and World Efficacy

To extend the potential of sound as an instigative force within screen entertainment we must consider the formation of approaches to promoting the sensory possibilities that audio can bring to picture. Alongside this is the confirmatory and constructive contribution that sound can make to the establishment and widening of the world as presented within the production design. In this case the widening of the created place shall be referred to as worldbuilding. Worldbuilding as used here refers to the construction of a sonic landscape which facilitates the understanding and acceptance of the genre-based mise-en-scène offered by the content on-screen. In this case the audio supports the production design, a furtherance of role as yet only lightly applied in cinema but more strongly applied in game through the consideration of Juicy Audio¹², and so the paracosm which is being created for the audience within the film’s world. There is also, as we shall see, the opportunity to extend and embellish the textures of the visual by application of textured audio and so by implication the audio’s juiciness which indicates a contribution to the overall coherence of the game. In this case juiciness is a useful concept as we think about the potential for audio in film and game to contribute more fully to the experience and the immersion of the product. In this case the application extends Hicks et al’s initial consideration of the way in which audio contributes physically and emotionally to game spaces and asks that sound be identified as a psycho-acoustic means of embellishing upon the visual and extending the feelings of the space being portrayed. When thinking about the effect of sound, there is a tendency to simply consider volume. As identified in the introduction, the increasing of volume within a cinema sound system, the pulse of the sub moving through the audiences' body has been an approach to garnering a physical link to the audio. This work suggests that there is a much subtler psycho-acoustic link to the way in which audio can connect to the senses beyond the aural. “...[S]ound can evoke touch through timbre and sonic texture, conveying something about haptic experience without being routed through representation¹³. Although writing in response to musical evocations of sensory interaction between audio and listener Garcia offers that there can be a physical relationship to sound. In the case of music only this is the literal haptic experience, a physical reaction which can be through the physicality of the sound played, waves striking flesh. But we can extend this physicality through to a perceived corporeal interplay if we experiment with the spectator or player’s psycho-acoustic relationship to sound. This is an effect which occurs within cinema and game audio as an experience presented in addition to rather than as the focus of the application of effect within the audio mix. Focusing on the application of audio for sensory effect extends its potential within the creating of a feeling for the viewer. A useful example of this extension to mimic a physical or olfactory sensation or reaction/linking can be seen in The Texas Chainsaw Massacre¹⁴ where the application of audio reinforced the feeling of heat, decay, and misery which pervaded the visuals and indeed the shoot itself¹⁵, to apply the term again a textural or juicy application of audio. Although this approach cannot be undertaken for every sound within a film or game, the consideration and use of this creative opportunity and approach is a way to extend the contribution of the audio for a viewer or a player.

The use of the term paracosm leads us to the concept of world efficacy within a fictional screen narrative. World efficacy relates to the wider conceptualisation of a created world termed the paracosm. Regardless of the ways in which the narrative world is constructed the elements held within it must make sense, must hold verisimilitude for the viewer, a realism within the fabrication. At an extreme this creates the paracosm, the created and minutely detailed construction which can be seen in the invention (at the furthermost point of its application of languages for other races – see Lord of the Rings¹⁶). This work posits however that we need not go to such extremes to apply the term, although Silvey and McKeith¹⁷ presented the paracosm as private childhood worlds created by and for the owner, it is the potential to specific detail which attracts interest. Primarily such application relates to the term verisimilitude that the created rules must make sense for the constructed reality but also for the established rules of the place. As such this means that in space sound can be heard, intergalactic engines make sounds of the mechanical, technical, and physical. Such sounds, although not essentially of this actuality, are grounded in the concepts we hold and extended to the newly constructed reality – the paracosm of starships and light-speed jumps. To an extent such an application mixes the constructs of animated audio with the realities of the world we know. The trick being the combining of the real and the fantasy to ground the obviously created visuals, especially when we are dealing with science fiction or horror genres. So, efficacy becomes our watch word as we build sonics to sell the mise-en-scène.

Because worlds have been built over and over and so in turn have used those examples which have gone before there are established expectations held by audiences based not on personal experience but on constructed aural experience. Constructed aural experience, as it will be applied within this work, is that which we as viewers have learned through the medium of film or television in connection to an action or application of a sound within a circumstance, we have not experienced firsthand. Many of us therefore “know” what a gun sounds like despite having not fired a gun or been near one being discharged. We are aware that the wind “always” blows in a desert even if we have never been. How best to ascribe desolation to a space than to hear the wind blow through a place? It is a trope which is used through many genres and in many narrative establishments of the feel of a setting. As an audience we are educated, through exposure, to continuously be aware of the meaning of such mechanical, technological, and natural sounds as applied to screen entertainment. We understand the causal and semantic allocation¹⁸ of these constructed auditory experiences and continue to expect them reinforced in each new piece of media we encounter. Bresson offers a conceptualisation of this relationship, identifying “[i]mages and sounds like people who make acquaintance on a journey and afterwards cannot separate”¹⁹. However, such constructs are so powerful through their repetition that they become the thing they purport to be. Chion’s and Bresson’s linking of image and audio becoming indelible in the minds of the audience beyond the establishment of the power of this association merely to be believed by the viewer. Such is the learned association of created audio to action, or application, reality is rejected and augmentation or imagination become the only acceptable representations of substantiality. At its root such acceptance of the created as the real or actuality proves the power of sound to push the boundaries of literal representation, or Murch’s shadowing of the visual, and encourages us to consider the opportunity to develop this reception and extend it into a more complex and integrated audio contribution to the sensory experience that a moving image product can offer. If Bresson defined that “Cinema seeks immediate and definitive expression through mimicry, gestures, intonations of voice. This system inevitably excludes expression through contacts and exchanges of images and of sounds and the transformations that result from them”²⁰, then this work proposes that the power relationship between visual and audio becomes open to Bresson’s disallowed exchange. In approaching sound as a more flexible and demonstrative tool than its traditional application would suggest we can push audio into avenues which many would bar. Although the visual still leads cinema and game, it is now viable for audio to become a more integral storytelling and textural tool, confirming sensation, setting, and action alongside the images offered.

2.1.2 Approaches to Creative Implementation of Sound Design Approaches in HE

Beyond the technical and so foundational understanding of the recording and mixing of audio (mic placement, room treatment, Digital Audio Workstation use) is the craft and the creation of the sonic world of the film or game. As such both areas are key elements in inspiring new audio designers. This section, as the title suggests, will consider the approaches to supporting and instructing students in Higher Education (HE) as they develop their practice as creators of sound and importantly in the remit of this work – designed sound. With approaches to moving creativity away from only being screen led and to embrace the potential which sound carries to assist in producing a stronger sensory experience. Additionally, through encouraging creativity students are readied for the non-linear approach to sound application within game engine where sounds must be short but interesting within their deployment.

Focusing on the ways in which we can develop sonic spaces which are not directly visually led can be a useful first step in encouraging students to think sonically and to break away from the requirement for an image as inspiration. This is not just using audio as a storytelling tool, however it is a useful approach to encourage consideration of the potential power and opportunity of sound, but also as one which creates a version of the environment for the listener that involves more detail than would be afforded were sound left as ambiance or reactive support. As the title of this work identifies, the textural qualities of audio tend to be overlooked and as such provide room for a more detailed exploration and foregrounding as useful sensory contributor.

Such a perspective on audio encourages the sound design student to interrogate the extended textural (actual and figurative) applications of sound alongside the confirming of the created world as real and solid no matter how fantasy based the mise-en-scène. As we can have a hyper-realistic visual, sound design students should also be encouraged to identify the hyper-realistic audio, the extension and layering of a sound which as with many audio pursuits seeks to go unnoticed by the audience present as hyperbolic yet accepted augmentation to the verisimilitude of the work.

Suggested approaches to extending the power of audio and its creative opportunities for HE students to explore the textural contributions of sound.

1. On and off-screen space Exercise - Using a still image to create an initial visual input to the creation of a soundscape. This exercise offers a tangible visual guide for the student yet asks them to think creatively and “outside” of the frame to consider a wider landscape. It is helpful if students source a copyright free image which is concept led in nature, primarily so they are not in a position where they are tempted to mimic rather than design the soundscape. This exercise also accommodates those students already familiar with spatialisation and can encourage them to think about the illusion of movement via sound placement within the mix creating a more dynamic piece that reflects a more natural ebb and flow of sound elements.
2. The above exercise can then be built out to a more complex piece which alludes to La Jettee²⁰ building a “moving image” piece only with kinetic sound and still images. Again, such an approach encourages the student to think about the power of the audio they are producing, in this case to expand the still visual into a wider and more colourful extended world.
3. “Make me a monster” is an exercise that again uses a visual as a jumping off point for the design, but again also is a still image which then offers the student the opportunity to consider the information given to them to instigate a sound realisation of the selected creature. As with the prior tasks identified the selected image should be copyright free and preferably from a concept artist source so that the produced audio design is for that creature and not for a reproduction of an already existing design for a well-known figure. This exercise asks the student to think about texture and the living breathing creature they are designing the audio for. The student should look at the creature and consider indicators of size (decisions can be made if there are no direct indicators), weight, construction, conceptualised movement, the texture of skin etc. The environment the creature is found in is also an important indicator for the design of the voice and physicality of our monster. Students should be asked to offer versions of the creature in differing emotional states to bring a wider breadth to the vocalisations and to indicate that there are a variety of aspects beyond simple attack sounds for a “monster”.
4. The Limitation Exercise presents an approach where one source for all sounds created exists. In this case a can of fizzy drink. The students should be given a simple sci-fi loop which is short and so manageable for the application of a limited sound set. In this case the selected genre works well with the hisses, fizz, clicks and metallic hits and tearing which can be gathered from the source. Such an approach encourages experimentation with sound and underscores the value of listening and gathering sound for an evolving sound library. Other interesting sources can be washing machines (so all sounds gathered must be from the machine), steam irons or cars.
5. The Creation Exercise takes inspiration from the approach of Wayne Bell sound designer for The Texas Chainsaw Massacre²². Students are given an overview of a sequence and associated genre and asked to create a sound bank for that premise. When the sounds have been gathered, they should be fitted to the sequence. Again, such an approach although unorthodox, in terms of traditional approaches to sound design, assists in widening the students’ experimentation with sound creation. Without picture immediately influencing there is scope to play and create with fewer conceptual restrictions.
6. Augmentation via Opposition Exercise takes inspiration from the work undertaken in The Jungle²³ an episode of the cult classic science fiction television series The Twilight Zone. The episode uses meta-diegetic sounds to reinforce the haunting/cursing of the episode's central character audio of a jungle locale are played over the in-frame world of an American city. The audience did not see the diegetic cause of the audio until the final payoff which confirmed that however fantastical the sound was, it emanated from the characters' world. Although an exercise which requires some connection to a film department to set-up the action over which to place the meta-diegetic sounds such an approach can show the sound design student the power of sound to create added meaning and narrative drive. A scene can be as complicated as the scared man roaming the city streets then haunted by the sounds of a far off but intimidating location as in The Jungle, or can be as simple as a character in a single shot reacting to direction to be aware of unseen forces in the room they are in. If we have a simple basis which might initially drive the audio, we are then creatively free, within this exercise to play. Such a creative opportunity also enables the application of skills and approaches developed in the Creation Exercise to augment and add sensory elements to the sound design for the sequence.

3 CONCLUSION

Sound in relation to moving image products has been positioned for many decades as follower and not lead. Walter Murch likens audio in film to a shadow, Skerjeth presents sound as a mimic. It is the contention of this work that the potential of sound has not been fully realised and that as immersion becomes a more central part of screen entertainments, the time for audio to take a more central and responsive and indeed instigative role is on the horizon. The examples of audio being a truly exploratory and creative centrally utilised craft are still few enough that when sound does take the lead or a more integrated position such instances are used as indicators of opportunity but not actuality for those studying sound design. It is fully accepted that not every instance where sound accompanies image is suitable for a stronger or more creative sound design than the one used. It is a foundational aspect of cinema and game cinematics that the dialogue is privileged and if being offered should be audible. However, as we can see the potential for sound is one which carries more weight than it presently does, making a textual contribution, creating a feeling or sense of the production design – enhancing the juiciness on offer to the player or audience.

Audio in film and game differs in its application and experience by viewers and players respectively. However, the potential response to audio, its ability to effect us in a bodily way, to suggest sensory feelings which go beyond the auditory is clear. In approaching audio as a contributor to the textural opportunities that images offer, we can broaden the experienced world. Extending it beyond the edges of the frame but also more strongly engaging with the feelings of the audience or player, creating physical responses, additional narrative or production design information as we build upon the mise-en-scène and into the consciousness of those engaging with the material.

The approaches and exercises identified here are designed to extend the foundational work which all sound design students engage with. The stretching of the creative impulse is, in the opinion of this work, central to the movement of the potential of sound in film and game to take a more central role within the building of the narrative, the play and the realisation of the production design. Why just see? When at certain moments can we, the audience or player, feel, be aware of the world the characters are in and become, even if just momentarily, a part of it through physical sensation?

4 REFERENCES

1. R. Bresson. Notes on Cinematography. Translated by Jonathan Griffin. Urizen Books, 1977.

2. S. O’Brien. The Art of Sound: guest lecture. In Cornwall Film Festival, Falmouth, 12–15 November, unpublished, 2009.

3. A. Skjerseth. Grains of Sound: Visual and Sonic Textures in Sand or Peter and the Wolf. Animation: An Interdisciplinary Journal, vol. 16(3), pp. 110–125, 2021. doi: 10.1177/17468477211049353.

4. T. Shone. Storytelling has become the art of world building: Avatar and the rise of the paracosm. The Guardian, 2022. [Online]. Available: https://www.theguardian.com/film [Accessed 15/10/2024].

5. R. Silvey & S. MacKeith. The paracosm: A special form of fantasy. In D. C. Morrison (Ed.), Organizing Early Experience: Imagination and Cognition in Childhood (pp. 173–197). Baywood Publishing Co., 1988.

6. L. M. Garcia. Beats, flesh, and grain: sonic tactility and affect in electronic dance music. Sound Studies, 1(1), pp. 59–76, 2015. doi: 10.1080/20551940.2015.1079072.

7. W. Murch. Stretching Sound to Help the Mind See. [Online] FilmSound.org. Available: www.filmsound.org/murch/stretching.htm [Accessed 21/10/2024], 2000.

8. D. Bordwell and Thompson. Film as Art. Seventh Edition. McGraw-Hill, 2003.

9. M. Chion. Audio-Vision: Sound on Screen. 2nd Edition. Columbia University Press, 2019.

10. A. Skjerseth. Op. cit.

11. M. Chion. Op. cit.

12. K. Hicks, K. Rogers, K. Girling and L. E. Nacke. Juicy Audio: Audio Designers’ Conceptualization of the Term in Video Games. Proc. ACM Hum.-Comput. Interact., vol. 8, no. CHI PLAY, article 319, 2024.

13. L. M. Garcia. Op. cit., p. 60.

14. The Texas Chainsaw Massacre. T. Hooper (Dir). Vortex Inc., 1974.

15. W. Bell in K. Periera. Wayne Bell: The man behind the sounds of The Texas Chain Saw Massacre. [Online] Available: https://designingsound.org/2018/10/25/wayne-bell-the-man-behind-the-sounds-of-the-texas-chain-saw-massacre/ [Accessed 22/10/2024].

16. J. R. R. Tolkien. The Lord of the Rings. Allen & Unwin: London, 1954.

17. R. Silvey & S. MacKeith. Op. cit.

18. M. Chion. Op. cit.

19. R. Bresson. Op. cit., p. 13.

20. R. Bresson. Op. cit., p. 19.

21. La Jetée. C. Marker (Dir). Argos Films, 1962.

22. The Texas Chainsaw Massacre. Op. cit.

23. The Jungle. Ep. 12, Season 3 of The Twilight Zone. W. F. Claxton (Dir), 1961.