Museum Virtual Worlds

MIND THE FISH Augmented Reality exhibit at CineKid Festival, Amsterdam, Oct 2011. Image by Sander Veenhof

Ever wondered what your goldfish are thinking about? A playful exhibit called MIND THE FISH at the Cinekid festival in Amsterdam this month allows visitors to peer into a fishbowl via AR and read discussions the fish are having with themselves, with other fish, and with visitors. The amusing texts, like “I feel like everyone is looking at me … or am I crazy?’ appear as thought bubbles projecting from fish as they swim around their aquatic habitat as seen by a movable screen with an augmented view of the bowl. The interactive work was made by Arthur van Beek, Sander Veenhof, Edith Kuyvenhoven, and Tijmes Woudenberg, collaborators from the Netherlands who each contributed different skills to bring the inner thoughts of the fish to light.

I asked AR artist Sander Veenhof some questions about creating the work and the reactions of visitors.

What are the challenges of designing an AR exhibit that incorporates a physical interface beyond a mobile device?

The subject matter we were trying to augment caused the foremost challenge. Since no off-the-shelve and open source system for tracking and tracing goldfish exists, we created one ourselves. Fish are quite challenging to track, because of their shininess and moving and turning in front of the camera. To our distress, when the lighting of the festival was installed, it appeared that orange was the a very popular light color, changing the whole surrounding into orange, so our system detected numerous invisible fish all the time.

The physical interface is actually not very physical. There’s no movement sensor or angle detection involved. Just by turing the device and the webcam, new fish enter the visible screen, and the software handles the placement of text call-outs.

What are the fish saying?

The fish stories are written by an expert in the field of children stories, and she even had experience writing stories in which animals play the main role. A couple of stories are about fish wondering what to do. Playing computer games, seeing a movie. And of course: playing hide and seek. Lucky for them, there’s one plant in the fish bowl. Once in a while, a virtual fish appears. Its remarks indicate that it has no clue how things in the real world work. Not even knowing what a dictionary is, when being suggested to look into such a book. Furthermore, one of the goldfish acts as a #twittervis relaying recent tweets including both #twittervis and #cinekid

What were the roles of the different collaborators on the project?

This was collaboration to the extreme. Every part was done by someone else, and all the components came together in the last few days before the opening. The system had an optimal modular set-up. The hardware was designed by Arthur van Beek, based on an original concept developed by me and him. He reserved space for a laptop on which Tijmen Woudenberg put his ‘orange tracking’ software. The tracking software kept track of fish, numbering them uniquely. His software did requests to an online dialogue server I created, which analyzed which fish were available, if a monologue or dialogue was going on, and if the involved fish were still present. If not, a new one-liner or story was started. All monologues, dialogues and even trialogues were written by Edith Kuyvenhoven, who could keep contributing new texts until half an hour before the opening, fine tuning the texts based on the experience of seeing the installation working as a whole for the first time.

There must have been a great sense of wonderment and whimsy for people who engaged with the exhibit. Were any observed reactions surprising?

I found it fascinating to keep getting the question: “How do you know what they are saying? Is this real? Or not?”. The fact that it is a question, means it did look convincing to the children. Many wanted to install such a system at home, pointed at their own fish bowl.

Besides the AR aspect, the installation functioned in a very almost analog way. Kids are of course checking how they look, when being viewed at from within a fishbowl. Actually, it was something I was curious about too. How does it look when looking to the outside world from inside? We’ve a lot of plans for variations and new versions of the installation. One of them is to check the effects of installing a fish-eye webcam. I’m very curious how the world in the bowl will look like then.

MIND THE FISH at CineKid Festival, Amsterdam, Oct 2011

Over 700 museum professionals convening last week in Philadelphia for the Museums and the Web 2011 conference saw applications and content expanding from the desktops of museum visitors onto networked exhibits inside of museums and onto a growing collection of mobile devices. The annual gathering, which brings together museum exhibit developers, producers, user-experience designers, artists, curators, software developers, digital media directors, and vendors of content creation tools and services, showcases the latest digital media work in the international museums community. It provides a great opportunity for people to share work, ideas, and strategies for using digital media in it’s many evolving forms to engage visitors both in the museum through networked exhibits and online through websites and mobile applications.

As part of a conference session on augmented reality, I presented a paper which discusses the initial investigations of AR that I’ve been doing here at the Exploratorium. This includes a science inquiry activity about weather in the San Francisco bay which will be part of our Science in the City video program series, and the playful Step Into a Virtual View art installations at February’s After Dark: Get Surreal event. The paper presents details about how these exhibit prototypes were developed, considerations for preparing 3D content for use in mobile AR applications, interface design challenges, and some initial observations of how museum visitors interact with mobile AR exhibits. Read the full paper, “Mixing Realities to Connect People, Places, and Exhibits Using Mobile Augmented-Reality Applications” here.

Planning an Exploratorium AR exhibit prototype- Golden Gate Bridge Fog Altimeter

Also presenting about their work with AR in this session were two museums from the Netherlands. Margriet Schavemaker, from the Stedelijk Museum in Amsterdam, discussed that museum’s work with mobile AR using the Layar platform. Their ARTours project, currently one year into a two year project, investigates how their museum visitors interact with AR art and architecture points of interest and exhibits in Amsterdam. Read the paper, “Augmented Reality and the Museum Experience”, co-written with her colleagues Hein Wils from the Stedelijk and with Paul Stork and Ebelien Pondaag from Amsterdam-based design studio Fabrique.

Cutting-edge work was also shown by Ingeborg Veldman and Tanja van der Woude from Science LinX, the science center of the Faculty of Mathematics and Natural Sciences of the University of Groningen in the Netherlands. Science LinX is focused on engaging teenagers in STEM disciplines and they experiment with exhibit development methods and tools that can be used to communicate hard to explain phenomena. I was captivated by Ingeborg’s presentation about their project, MIGHT-y, an exhibit and game which uses 2D markers on the faces of cubes to let visitors explore concepts about scale from the film by Charles and Ray Eames, Powers of Ten. It’s a mixed-reality exhibit in which visitors don AR glasses to see “into” the cubes and manipulate 3D animated objects. Use of AR glasses seem new for museum exhibits and still in an early stage of application. Also, it’s expensive. The glasses used in MIGHT-y, Wrap920 video eyewear from Vuzix, are a consumer product geared toward gamers that can be adapted for use in exhibits to provide an immersive virtual reality experience. I tried a portable version of MIGHT-y with the eyeware, and although it was a bit jarring in terms of head tracking lag, it does provide a compelling augmented display. It’s encouraging to see experimentation with different forms of AR in museums. Read their paper, “Science LinX: the neXt level in augmenting science center Xperiences” which was co-written with Bart van de Laar, also of the University of Groningen

MW2011 was a great conference and it was a special honor for the Exploratorium’s website to be recognized with the Best Long-Lived Site Award by the Museums and the Web community!

Visitors to the Exploratorium’s monthly evening program series for adults, After Dark, experienced bizarre interactions with their sense of reality at After Dark: Get Surreal on Feb 3, 2010–an event infused with surrealist themes in art, music, and science.

Takuji Narumi, a Ph.D. candidate in the Graduate School of Engineering at the University of Tokyo, and Takashi Kajinami, a master course student in the Graduate School of Information Science and Technology at the University of Tokyo tantalized visitors with their interactive experience, “Meta Cookie.” The installation allows participants to don a head-mounted display while holding a cookie that has a 2D marker image burned onto it. A webcam attached to the headgear detects the marker on the cookie while the AR magicians change settings which cause the image of the cookie the person sees through the display to change to a different flavored cookie. Before your eyes, the cookie morphs visually from a butter cookie to a chocolate cookie to a strawberry cookie to a maple cookie to a lemon cookie. Did I mention that tubes attached to the headgear aimed at the adventurous person’s nose deliver scents of the different cookie flavors as the display changes? This remarkable exhibit really pulls at your sense of reality as you nibble on the cookie when it appears to be one flavor and then again as the image and smells change, sensing the flavor as entirely different!

Meta Cookie at the Exploratorium’s After Dark: Get Surreal event on Feb 3, 2010

A few months ago, I was introduced to Helen Papagiannis, an artist, designer, and researcher working with the emerging technology Augmented Reality (AR). I was captivated by the way her playful AR exhibits and installations drew people in through touch, video, and sound. She’d recently exhibited her work at the Ontario Science Center and was exploring ways to engage museum audiences with their sense of discovery and wonder. I caught up with her recently and asked her some questions about the ideas behind her work.

The Amazing Cinemagician: New Media Meets Victorian Wonder” exhibition by Helen Papagiannis at the Ontario Science Center, May-Sept., 2010, Toronto, Canada. Photos: Pippin Lee

When did you start experimenting with augmented reality?

I began experimenting with AR in September 2005. When I saw AR for the first time, I was so entranced I think I entered a permanent state of wonder with the technology. And it was all very simple: a bare bone 3D virtual cube seemingly appearing in my physical space. It was completely astonishing! I went into mad scientist mode from there tinkering, prototyping, and dreaming of the creative possibilities for AR. Five and a half years later, and I’m still riveted.

What are some of the challenges that you’re exploring in your AR work?

When I began working with AR, the challenges were largely around the technical constraints. It has been important for me to work with what is at hand, right now, not tomorrow, or 6 months from now. I always ask, ‘How can we realize this now and make it compelling within the parameters?’ My process has entailed allowing the constraints to guide the work, then finding ways to push beyond those boundaries to create something new.

I strongly believe AR is emerging as a new medium and it will come to play a large role in entertainment and information sharing. The challenge at hand is to continue to investigate how best to apply the medium and really elevate it creatively, and to do this as a community of artists, engineers and industry. We need to identify what is truly unique about this new form, and how we can best leverage these characteristics to tell new stories and create engaging experiences that are unlike anything we’ve ever seen before.

How can museum audiences experience AR?

We’re beginning to see more AR applications in museums, which is very exciting. AR is becoming more accessible and affordable, including the use of personal devices that museum visitors may already have at hand, such as smart phones. AR can be used to provide additional information about objects in a museum’s collection and to enable experiential learning through discovery and play. Wonderment, as discussed in my TEDx talk, is an important part of my work in AR. For me, AR fosters a great sense of wonder as a looking glass into another world and can be used to further ignite curiosity and inquiry in a museum setting.

Museum audiences can experience AR through various viewing devices including tablets equipped with cameras, large wall-mounted screens or kiosks, and mobile devices. Here are a few contemporary examples:

Tablets:
Natural History Museum’s interactive film “Who Do You Think You Really Are?” as well as Metaio and Louvre-DNP collaboration

Screens:
Examples include my Wonder Turner installation at the Ontario Science Centre, and Total Immersion’s environmental kiosk

Mobile devices:
An unofficial exhibit at the MoMA using Layar and one of my recent projects “AR Hanging Mobile” using ARToolkit for iPhone.

What limitations of current ways that people can experience AR art works and installations would you like to get beyond?

I’d like to see more work move beyond the single viewer experience in AR and engage larger audiences in a simultaneous viewing and even collaborative interactive experience. I think this is particularly relevant for museums in designing and producing AR experiences. My early work in AR started out with books and other small hand-held objects creating an intimate experience for a single user at a given time and then expanded to a more collaborative exploratory setting with multiple users engaging in an act of play and discovery together. There’s a great opportunity for enabling a larger group dynamic at work in AR with multi-users. This can combine visitors both on and off-site.

Let’s also make the work so wondrous people forget they are looking at a screen or using a device! I’m continually exploring ways to heighten “presence”, a term used in AR to describe the illusion of non-mediation.

***
Helen Papagiannis is an artist, designer, and researcher specializing in Augmented Reality (AR). She is presently completing her Ph.D. at York University in Toronto, Canada and is a Senior Research Associate at the Augmented Reality Lab (Department of Film, Faculty of Fine Arts). Helen’s mixed reality art installations were recently featured in a solo exhibition at the Ontario Science Centre, and at TEDx, where she was also an invited speaker. Prior to her augmented life, Helen was a member of the internationally renowned Bruce Mau Design studio, where she was project lead on “Massive Change: The Future of Global Design”, a touring exhibition and book published by Phaidon Press.

I haven’t posted about any new developments with virtual worlds that the Exploratorium is developing in a few months. Fortunately, my lack of writing about it doesn’t equate to nothing going on in that realm. Exploratorium Island and ‘Sploland island in Second Life continue to thrive with several hundred visitors per week each, guided tours of exhibits, and occasional building parties and building tutorials. We’re thinking about what events would be good to stage there in the coming year. Many new exhibits have been added to Exploratorium Island recently and the space is undergoing a needed make-over. More on these developments soon!

Now I’m going to begin a little detour from immersive 3D virtual worlds and introduce some new experiments with mobile augmented reality (AR) –an emerging technology and practice that allows you to overlay a view of a physical environment or object with virtual content. We did an experiment here at the Exploratorium last year with AR, as part of our kick-off for the celebration of our 40th anniversary. It presented a real-time rendered view of an overlaid interactive 3D model onto a 2D marker that was embedded in the cover design of our quarterly publication, Explore. This was inspired by others beginning to use the print medium as an entre to the technology. I first became fascinated with the idea of location-based augmented reality when I read descriptions of pieces that “locative” artists made in William Gibson’s book Pattern Recognition. In his post-cyberpunk story, Gibson describes several large-scale 3D virtual objects that are placed at specific locations meaningful to the story (via geo-referencing). A giant octopus, a field of moving flowers, a re-created scene, etc. Viewers needed to don cumbersome headgear to see these things, which were rendered in high resolution 3D stereo and composited compellingly into a view of a physical environment. It reminded me of things I’d tried previously with slide and video projections at specific locations. In a digitally-enhanced world, with the ubiquity of the Global Positioning System and the mass-market access to it through hand-held devices including smartphones, augmented reality is now a technology that many can experience, and one that museums are experimenting with.

For me, augmented reality represents a slice of a virtual world that people can experience on the go, in their day-to-day lives. It seems like a fluid extension of immersive 3D virtual worlds in that virtual and physical world realities are mixed. There are possibilities for blending communication (social aspects) and user-generation of content in the augemented views. AR is also being heavily influenced by the commercial gaming space, with haptics and full-body gestural interfaces promising more natural-feeling interaction with augmented views. Products like the Wii and Kinect are being used as interfaces to non-gaming content. Open-source toolkits to use those devices are now available. More on that topic soon.

AR technology is in an early stage. Hardware and software platforms are rapidly emerging, open and proprietary systems are vying for developer and consumer attention, business models are being invented, content developers are exploring different uses. Museums are beginning to develop models for use for AR, designed to engage visitors in discovery of “extra” information about objects, exhibits, and places. The New Media Consortium 2010 Horizon Report: Museum Edition, tracks the time-to-adoption for AR as two to three years. So, it’s an exciting time to see this new technology grow and to work with it yourself to see how you can use it to engage your audiences.