Augmented Reality in museums

By Seb Chan and Jini Maxwell

Museums have always been a type of virtual, or augmented, reality. The classical dioramas of natural history museums transported museum visitors to ‘virtual worlds’ as early as the nineteenth century while light and slide projections, and on-glass images and text have been used to augment display cabinets with information and interpretation for many years.

Seventy years ago, museums and galleries began experimenting with augmented audio guides to define and share the institution’s ‘voice’ with its audience. The first example of this was in 1952, when Stedelijk Musem’s director Willem Sandberg pioneered the world’s first museum audio tour, which broadcast pre-recorded lectures via closed-circuit shortwave radio.¹ Like so many innovators, Sandberg’s expertise came from outside of the institution. With a background as an artist and designer himself, Sandberg has a rebellious reputation; his interventions in the museum space foregrounded experimentation and play to find new ways to communicate with diverse audiences. His conception of the museum as a space that existed to serve and engage its visitors, rather than as an austere and academic space, was considered iconoclastic – but his commitment to experimentation and play resulted in frameworks that are considered more than half a century later. Now it is common practice for museum visitors to receive a recorded audio guide, either through a device provided by the institution or on their own smartphone.

Following the commercial release of virtual reality (VR) headsets in the 1990s, followed later by artificial intelligence (AI) and smartphone-based augmented reality (AR) in the 2010s, these enhancements and augmentations now provide individualised computational environments for a visitor to interact with on their own. Through these new technologies, the way museums speak for themselves is adapting, but so is who can speak back.

Since the development of AR, the technology has been used in galleries, museums and archives to broaden the ways audiences engage with their collections, allowing audiences to interact with works in AR that in the physical world are usually kept strictly under glass. In 2010 the Museum of London and in 2009 the Powerhouse Museum in Sydney geolocated parts of their historical photograph collections, allowing the public to ‘see the past’ through their smartphone screen as they wandered those cities. In 2018, the Cleveland Museum of Art introduced the ArtLens Studio, offering creative prompts that allowed audience members to play games, draw, sculpt and paint in AR using 3D scans of objects in the museum’s collection.

In 2009, a new wave of innovation driven by Bluetooth Low Energy (a form of Bluetooth that consumes less power) led to the introduction of audio-augmented museum tours that dynamically responded to the visitor’s location in a gallery. Initially intended to replace, or at least supplement, a human guide, audio tours continue to evolve with advancements in technology. In 2017, IBM Brazil worked with the Pinacoteca de São Paulo on a project called ‘A Voz da Arte’ in which an AI chatbot was trained on seven artworks in the collection and programmed to answer questions from visitors about the works. Rather than rely on a pre-recorded script, the bot developed answers in real time. The museum’s ‘voice’ was here outsourced to a machine whose answers were shaped not by the gallery’s expertise but on data pulled from the IBM cloud.

Innovation in the field of 3D sensing in 2020 amplified the smartphone’s ability to recognise objects and faces, leading to another wave of AR experimentation. Technologies like stereoscopic vision and structured light patterning allow the smartphone to detect and retain distance, geometry and orientation through the camera lens, making it easy for software to position virtual objects in ‘real’ space. 3D sensing technologies have also been used by social media platforms – first Snap (formerly Snapchat) and now Instagram both offer spatially aware camera ‘filters’ as part of their integrated advertising systems. The technology used to develop AR Instagram filters is free to download, allowing users to create filters and frames that automatically orient around a face on camera. These offer novelty to the user and their friends while communicating valuable location and content data back to the platform and its advertisers. This opportunity was quickly seized on by marketing teams in institutions as a low-cost way to reach new audiences.

Artistic approaches to augmenting reality can be traced back to at least 1981, when new-media artist Jeffrey Shaw created AR sculptures using a combination of video and theatrical illusion techniques. In his pioneering work Virtual sculpture, 1981, Shaw mounted a Fresnel lens and semi-transparent mirror on top of a monitor, allowing a viewer to rotate and tilt the monitor, viewing various computer-generated objects through the mirror as if they were floating in real space in front of them.

Locally, Australian new media artist Troy Innocent has been working in virtual worlds since 1989, creating work that explores the limits of digital space. His latest work, 64 Ways of Being, 2021, is a location-based AR experience developed as a smartphone app. It is designed to be experienced as the user walks around Melbourne, with different pieces of participatory art inviting audiences to view specific locations throughout the city through a curious, creative lens.

The advent of the World Wide Web in 1993 heralded an explosion of digital philosophies and artistic interventions, many of which characterised the internet as an ‘empire of the mind’ that offered a new world completely divorced from the constraints of analogue reality. Australian new-media art collective VNS Matrix recuperated the emerging digital sphere from the ‘technocowboys’ in order to ‘remap cyberculture with a feminist bent’, exploring the way our digital lives were informed by the physical, particularly sex and the body and how they are informed and augmented by the internet.² Through public art projects like Cyberfeminist Manifesto for the Future (1991) and E Sensual Fragments (1994), the collective explored these messy, embodied interdependencies of digital and physical realms, coining a phrase that would come to embody their philosophy: ‘the clitoris is the direct line to the matrix’.³ Their cutting-edge work provides an early framework to understand both the opportunities and the limitations of AR: the digital experience is not separate from the physical world, but borne out in it.

In the following decades, particularly with the stabilisation of the smartphone market and the creation of the Android and iOS operating systems, the opposite has also become increasingly true – the physical world is ineluctable from the digital one. Through the venture capital–led tech boom of the 2010s, significant money and media interest flowed into the field of AR. The development of an array of increasingly precise orientation sensors and cameras, and the connection to high-speed wireless networks and geo-location technologies led to an explosion of smartphone products, projects and apps. The uses for these technologies are broad ranging, from commercial to creative, and cultural institutions have always taken part – though arguably the most successful deployments of AR in museums have come from outside institutions.

After the release of Niantic’s Pokemon GO in 2016, the Museum of London began placing in-game ‘lures’ throughout the museum that summoned rare Pokémon in the AR game. This drew younger players into the building and visitors were encouraged to share screenshots within the museum that were re-shared across social media. Niantic has since partnered with several British museums, including the Science Museum of London and the Historic Royal Palaces on similar projects, developing a toolkit called Lightship which museums can use to develop AR products of their own.

For all the success stories, AR products also introduce new concerns for institutions. Firstly, augmented reality is not an easily shared experience. The most popular AR product, Pokemon GO, for example, exposes a technical limitation that extends across a range of AR products. Using a smartphone camera and GPS, Pokemon GO superimposes digital objects onto a physical space. But this experience only exists between the audience member and their phone – it’s not easily shared with another person. Even if two audience members use the same AR app on their individual devices, they can’t easily share an AR experience. GPS is too imprecise a system to reliably replicate a digital object in the exact same place on two different devices. Unlike the public space of a gallery or museum, which is designed to be shared, AR currently exists exclusively for an audience of one.

Despite these issues, artists and activists have used AR technologies both in and around institutions, using new methods to speak back to institutional power. The ability to place location-aware virtual objects within the geographical confines of the gallery or museum quickly led to projects like MoMAR, an activist-run AR gallery concept that ‘hangs’ AR artworks inside the Museum of Modern Art in New York and in other major galleries. Since their first exhibition in 2018, MoMAR’s app has been fully open-source on their website, encouraging artists around the world to develop their own version of the app and continue the project’s goal of ‘democratizing physical exhibition spaces, museums, and the curation of art within them’. MoMAR’s instructions are designed to be beginner friendly and even detail the costs associated with creating an app on the Apple Store.

The Unfiltered History Tour is another activist-led AR intervention within a musuem. In 2021, Vice extended the work they began with their anti-colonial documentary series Empires of Dirt (2020) by creating an unofficial AR guide to the British Museum. The Unfiltered History Tour brings to AR the de-colonial work pioneered by art historian Alice Procter, whose Uncomfortable Art Tours provide in-person decolonial tours of some of Britain’s biggest institutions, including the British Museum, as well as a podcast exploring the colonial roots of their collections. Developed by Vice and Indian creative agency Dentsu Webchutney, the Unfiltered History Tour uses Instagram filters tailored to specific objects in the collection to reveal and challenge their colonial provenance, centring the voices of Indigenous peoples from whom these artefacts were stolen. The Tour provides an excoriating critique of the colonial image of the museum as a beacon of civilisation, making a space for artists to speak back against the institution, all within the walls of the institution itself.

The popular culture view of AI and AR has been largely shaped by science fiction, particularly film. As much as Steven Spielberg’s 2002 visualisation of Philip K. Dick’s novella The Minority Report set the stage for interactive gestural interfaces, it was probably John Carpenter’s cult film They Live (1988) – in which the protagonist finds a pair of sunglasses that allows him to see covert messages in advertising ¬– that first outlined a dystopian vision of AR. A video work by Keiichi Matsuda, Hyper-Reality (2016), extends this dystopian vision of augmented advertising by presenting the viewer with a first-person view of a frenetic mental interface where ads, instant messages and gamified apps impose over the real world, not just augmenting it but almost obscuring it completely.

Science fiction provides a lens through which people understand evolving technologies. For AR, the expectations it sets are almost always dystopian, proposing AR as a symptom of a lack of humanity, as in the calculated visual interfaces seen in The Terminator films, or as being only useful for selling you something. Museums and artists are now attempting to push back against this cultural impression. In a 2022 exhibition that took place in the videogame Fortnite (2017), artist KAWS developed and sold wearable ‘skins’ of his work, coinciding with an AR exhibition at Serpentine Gallery in 2022. KAWS described the exhibition within a videogame as a method to ‘throw bridges to a new generation’, helping them engage with artworks by placing them in a familiar context.⁴

Like many AR works by artists who do not primarily work with the technology, KAWS’s exhibitions at Serpentine and within Fortnite treat AR as a vessel for the artwork, rather than an essential feature of the work itself. Unlike the probing inquiries into physical and digital space proposed by artists like Innocent and Shaw, these AR works function more as a way of ‘re-skinning’ an existing product or technology than as distinct, vital works that stand on their own. In fact, AR and extended-reality artworks and exhibitions are often criticised for not being distinct enough from their for-profit counterparts. In a critique of Olafur Eliasson’s Wunderkammer, 2020, an AR filter work that allows audience members to virtually impose weather conditions, American artist Gretchen Andrew described the work as equivalent to ‘a giftshop postcard of the artwork’, noting that while the experience is enjoyable, it is ‘ultimately indistinguishable from “non-art” augmented reality applications like AR Make it Rain (2018) [a commercial weather filter app], which freezes less often’.⁵

Like many AR art works, Eliasson’s Wunderkammer includes premium content that can be purchased for a fee, further muddying the distinction between commercial apps and AR art. Artist and curator Eron Rauch has been more frank: ‘If I wanted to run around and see ladybugs and rainbows and cartoon characters, I’d just play Animal Crossing like everyone else’.⁶

The ubiquity of smart devices means that many citizens are now augmenting their own realities, experiencing privately curated versions of the physical world. We move through the city to the beat of personalised soundtracks on our headphones as the navigation tools built into our phones direct us to our destinations. And yet, AR technology is often met with distrust, revealing the tension between the private and shared versions of reality that are in contention in public spaces, including museums and galleries.

The digital sphere that VNS Matrix foresaw was one fundamentally shaped by analogue. But for digitally augmented realities to truly cross the threshold back into the public, physical sphere, there needs to be infrastructural change, not just technological innovation. There are early signs of such change: in the Melbourne CBD, illuminated dots mimicking the colours of pedestrian traffic lights are embedded into the pavement, anticipating that pedestrians will be looking down at their phone, rather than up and out. Here, the slimy, embodied interdependencies of VNS Matrix’s conception of the internet begins to seep across to the physical plane – but these examples remain relatively rare.

For now, AR technologies – whether implemented by institutions, artists, activists or businesses – belong to our private, personal realities. These interventions exist only as digital interactions between a unique user and their unique experience, filtered intimately through a smartphone. For a public institution that serves a diverse citizenry, this continues to pose a barrier. Artists, companies and creative technologists continue to push AR technologies forward, but the ‘real’ world operates at a different cadence.

Seb Chan is Director and CEO at the Australian Centre for the Moving Image (ACMI) in Melbourne. He is the senior executive responsible for the Experience & Engagement division of the museum, guiding teams responsible for visitor experience, marketing, brand and communication design, digital products, technology, and the museum’s collections, digitisation and digital preservation programs. Prior to ACMI, Seb led the digital renewal and transformation of the Cooper Hewitt Smithsonian Design Museum in New York (2011–15) and the Powerhouse Museum’s pioneering work in open access, mass collaboration and digital experience during the 2000s. His work has won awards from American Alliance of Museums, One Club, D&AD, Fast Company and Core77. He is an Adjunct Professor, School of Media and Communications, in the College of Design and Social Context at RMIT, and is an international advisory board member of Art Science Museum (Singapore), and board of the National Communications Museum (Melbourne), and National President of the Australian Museums and Galleries Association. He is an alumni of the Getty Leadership Institute, Salzburg Global Seminar, and UNSW. He also leads a parallel life in digital art, writing, and music.

Jini Maxwell is a curator and award-winning videogames writer. Jini is currently an assistant curator at ACMI Australia’s museum of screen culture, and also runs ACMI’s Women & Non-Binary Gamers’ club. Jini is also an award-winner writer: in 2021, they were awarded Best Gaming Journalist at the Samsung IT Journalism Awards. In September 2020, Jini wrote the The Saturday Paper‘s first videogame review, and has since contributed a regular games criticism column to the publication, as well as appearing on The Saturday Paper’s podcast, The Culture, discussing the videogame industry. Jini has also contributed critical writing to a wide array of publications, including Screen Education, ACCA online, The Wheeler Centre online, Overland Journal, Gameshub, Artshub, and the Melbourne Recital Centre’s soundescapes. In 2018, Jini was one of three Emerging Critics in Residence at the Melbourne Recital Centre.