Museum Virtual Worlds

MIND THE FISH Augmented Reality exhibit at CineKid Festival, Amsterdam, Oct 2011. Image by Sander Veenhof

Ever wondered what your goldfish are thinking about? A playful exhibit called MIND THE FISH at the Cinekid festival in Amsterdam this month allows visitors to peer into a fishbowl via AR and read discussions the fish are having with themselves, with other fish, and with visitors. The amusing texts, like “I feel like everyone is looking at me … or am I crazy?’ appear as thought bubbles projecting from fish as they swim around their aquatic habitat as seen by a movable screen with an augmented view of the bowl. The interactive work was made by Arthur van Beek, Sander Veenhof, Edith Kuyvenhoven, and Tijmes Woudenberg, collaborators from the Netherlands who each contributed different skills to bring the inner thoughts of the fish to light.

I asked AR artist Sander Veenhof some questions about creating the work and the reactions of visitors.

What are the challenges of designing an AR exhibit that incorporates a physical interface beyond a mobile device?

The subject matter we were trying to augment caused the foremost challenge. Since no off-the-shelve and open source system for tracking and tracing goldfish exists, we created one ourselves. Fish are quite challenging to track, because of their shininess and moving and turning in front of the camera. To our distress, when the lighting of the festival was installed, it appeared that orange was the a very popular light color, changing the whole surrounding into orange, so our system detected numerous invisible fish all the time.

The physical interface is actually not very physical. There’s no movement sensor or angle detection involved. Just by turing the device and the webcam, new fish enter the visible screen, and the software handles the placement of text call-outs.

What are the fish saying?

The fish stories are written by an expert in the field of children stories, and she even had experience writing stories in which animals play the main role. A couple of stories are about fish wondering what to do. Playing computer games, seeing a movie. And of course: playing hide and seek. Lucky for them, there’s one plant in the fish bowl. Once in a while, a virtual fish appears. Its remarks indicate that it has no clue how things in the real world work. Not even knowing what a dictionary is, when being suggested to look into such a book. Furthermore, one of the goldfish acts as a #twittervis relaying recent tweets including both #twittervis and #cinekid

What were the roles of the different collaborators on the project?

This was collaboration to the extreme. Every part was done by someone else, and all the components came together in the last few days before the opening. The system had an optimal modular set-up. The hardware was designed by Arthur van Beek, based on an original concept developed by me and him. He reserved space for a laptop on which Tijmen Woudenberg put his ‘orange tracking’ software. The tracking software kept track of fish, numbering them uniquely. His software did requests to an online dialogue server I created, which analyzed which fish were available, if a monologue or dialogue was going on, and if the involved fish were still present. If not, a new one-liner or story was started. All monologues, dialogues and even trialogues were written by Edith Kuyvenhoven, who could keep contributing new texts until half an hour before the opening, fine tuning the texts based on the experience of seeing the installation working as a whole for the first time.

There must have been a great sense of wonderment and whimsy for people who engaged with the exhibit. Were any observed reactions surprising?

I found it fascinating to keep getting the question: “How do you know what they are saying? Is this real? Or not?”. The fact that it is a question, means it did look convincing to the children. Many wanted to install such a system at home, pointed at their own fish bowl.

Besides the AR aspect, the installation functioned in a very almost analog way. Kids are of course checking how they look, when being viewed at from within a fishbowl. Actually, it was something I was curious about too. How does it look when looking to the outside world from inside? We’ve a lot of plans for variations and new versions of the installation. One of them is to check the effects of installing a fish-eye webcam. I’m very curious how the world in the bowl will look like then.

MIND THE FISH at CineKid Festival, Amsterdam, Oct 2011

To create immersive experiences in 3D virtual worlds, developers craft objects and scenes that offer compelling visuals and interactions. These elements, along with sounds, lighting design, animated objects, and avatars can all work together to stimulate senses and allow people to control aspects of the virtual environment. We naturally want to use more of our senses to experience and manipulate these digital worlds though, which is where stereoscopic displays, motion tracking, and haptics comes in. Interfaces like these are important to exhibit designers in museums. They allow us to to create physically engaging experiences with minimal technical interface barriers for visitors to use them.

From time to time, we see some great prototype technologies in each of these areas as they apply to virtual worlds (remember the data glove?) and occasionally new research emerges that seems like we can begin to distance ourselves from the keyboards, mice, and flat screen devices that are not the best interfaces for being seamlessly immersed.

With their Hands Free 3D r&d project, veteran technologist Mitch Kapor and Phillippe Bossut are developing a new way to interact with worlds like Second Life. Using a 3D camera designed by 3DV Systems, they’ve developed a custom SL client that allows users to operate SL without using a mouse or a keyboard. The system uses a webcam-like camera that captures the depth of objects in front of it (including a person) and using the modified client translates the person’s actual motion to the motion of an avatar to allow the person to walk or fly through SL by literally walking in front of the camera. Their goal is to make the interface to virtual worlds more natural and they expect this ease of use to make virtual worlds more appealing. Check out this video.