Museum Virtual Worlds

Bringing Real and Virtual Together

MIND THE FISH–Augmented reality exhibit reveals aquatic creatures’ inner thoughts


MIND THE FISH: Augmented Reality exhibit

MIND THE FISH Augmented Reality exhibit at CineKid Festival, Amsterdam, Oct 2011. Image by Sander Veenhof

Ever wondered what your goldfish are thinking about? A playful exhibit called MIND THE FISH at the Cinekid festival in Amsterdam this month allows visitors to peer into a fishbowl via AR and read discussions the fish are having with themselves, with other fish, and with visitors. The amusing texts, like “I feel like everyone is looking at me … or am I crazy?’ appear as thought bubbles projecting from fish as they swim around their aquatic habitat as seen by a movable screen with an augmented view of the bowl. The interactive work was made by Arthur van Beek, Sander Veenhof, Edith Kuyvenhoven, and Tijmes Woudenberg, collaborators from the Netherlands who each contributed different skills to bring the inner thoughts of the fish to light.

I asked AR artist Sander Veenhof some questions about creating the work and the reactions of visitors.

What are the challenges of designing an AR exhibit that incorporates a physical interface beyond a mobile device?

The subject matter we were trying to augment caused the foremost challenge. Since no off-the-shelve and open source system for tracking and tracing goldfish exists, we created one ourselves. Fish are quite challenging to track, because of their shininess and moving and turning in front of the camera. To our distress, when the lighting of the festival was installed, it appeared that orange was the a very popular light color, changing the whole surrounding into orange, so our system detected numerous invisible fish all the time.

The physical interface is actually not very physical. There’s no movement sensor or angle detection involved. Just by turing the device and the webcam, new fish enter the visible screen, and the software handles the placement of text call-outs.

What are the fish saying?

The fish stories are written by an expert in the field of children stories, and she even had experience writing stories in which animals play the main role. A couple of stories are about fish wondering what to do. Playing computer games, seeing a movie. And of course: playing hide and seek. Lucky for them, there’s one plant in the fish bowl. Once in a while, a virtual fish appears. Its remarks indicate that it has no clue how things in the real world work. Not even knowing what a dictionary is, when being suggested to look into such a book. Furthermore, one of the goldfish acts as a #twittervis relaying recent tweets including both #twittervis and #cinekid

What were the roles of the different collaborators on the project?

This was collaboration to the extreme. Every part was done by someone else, and all the components came together in the last few days before the opening. The system had an optimal modular set-up. The hardware was designed by Arthur van Beek, based on an original concept developed by me and him. He reserved space for a laptop on which Tijmen Woudenberg put his ‘orange tracking’ software. The tracking software kept track of fish, numbering them uniquely. His software did requests to an online dialogue server I created, which analyzed which fish were available, if a monologue or dialogue was going on, and if the involved fish were still present. If not, a new one-liner or story was started. All monologues, dialogues and even trialogues were written by Edith Kuyvenhoven, who could keep contributing new texts until half an hour before the opening, fine tuning the texts based on the experience of seeing the installation working as a whole for the first time.

There must have been a great sense of wonderment and whimsy for people who engaged with the exhibit. Were any observed reactions surprising?

I found it fascinating to keep getting the question: “How do you know what they are saying? Is this real? Or not?”. The fact that it is a question, means it did look convincing to the children. Many wanted to install such a system at home, pointed at their own fish bowl.

Besides the AR aspect, the installation functioned in a very almost analog way. Kids are of course checking how they look, when being viewed at from within a fishbowl. Actually, it was something I was curious about too. How does it look when looking to the outside world from inside? We’ve a lot of plans for variations and new versions of the installation. One of them is to check the effects of installing a fish-eye webcam. I’m very curious how the world in the bowl will look like then.

You need to a flashplayer enabled browser to view this YouTube video
MIND THE FISH at CineKid Festival, Amsterdam, Oct 2011

Author: Rob Rothfarb

Rob Rothfarb directs Web development at the Exploratorium in San Francisco, a museum of science, art, and human perception. He develops online exhibits, educational resources, and technical infrastructure for the museum. He’s interested in the application of interactive media technologies to the development of dynamic, networked content and media infrastructure to support exploration, communication, and learning. His background includes experience in software development, computer graphics, interaction design, and digital video. Rob’s interest in community and interactive 3D technologies led to work with several pioneering virtual world software companies and to interactive multimedia projects for museums including the National Gallery of Art and the National Constitution Center.

Leave a Reply

Required fields are marked *.