I’ve been spending quite a few hours researching and thinking about Virtual Reality input interfaces, and I’m developing a perspective I haven’t seen written about yet, so I’ll try to communicate here.

I’m now working on a big-vision, non-game project: a Virtual Reality activity that could consume hours a day, day after day. So I’m asking, what are the right practical interaction frameworks that would be both powerful and functional?

The number of options for alternative and gestural computer inputs in development is vast, with Leap Motion gaining the most public interest, but also devices like the  Nod ring, Microsoft Kinect, arm-bands, and a whole slew of wireless handheld controllers.

And of course, the artificial user interfaces (AUIs) seen in movies are such a gorgeous vision of what is possible: Iron Man and Minority Report are my two favorites.

This Iron Man interface was a key inspiration for me as I started on this VR journey, I like the way he can navigate and manipulate information:

But, as I’ll try to communicate, there are some serious practical issues with these futuristic visions.

Gesture-based inputs, such as Leap Motion, seem like they would be such a powerful, natural addition to the Virtual Reality experience, so I spent time learning to program the Leap Motion with LeapJS in a browser. Actually, a lot of time. I built a number of prototypes.

I came across this rather innovative interaction solution yesterday, more successful than most: Thomas Street’s UI Exploration with the Oculus Rift and Leap Motion, and this graphic showing the interaction modes.

The idea is that a series of menus (in this case they are spherical icons) are visible across the bottom of the view, and when you ‘gesture over’ each one by moving your hand back and forth, the menuitems (more spherical icons) appear going into the distance: move your hand forward a varying distance to gesture over your intended target, then rotate your hand to select.

The goal is to work around what is a pretty problematic device at this point. The lack of precise control is mitigated by mapping broad gestural strokes into a spatial grid to make a selection.

I gained another valuable lesson from this brilliant article by JODY MEDICH, also on LeapMotion’s blog: What Would a Truly 3D Operating System Look Like? She makes a convincing argument, which I agree with, that spatialization of information and content is a huge win: it is time to get past the limits of our 2D computer universe, with it’s ubiquitous frames (all screens have edges) around everything.

Information just wants to be FREE… with relationships visible everywhere: everything in its spatial place and always viewed within the myriad of relationships that screen edges inevitably obfuscate.

YET, for all that goodness, how are we to manipulate this new world? The mouse is pretty good for dealing with a 2D world, but unable to handle the third dimension.

Shall we wave our hands (literally above our pumping hearts) like Minority Report? For how many hours (actually minutes) will we last before we are tired and irritated? Rumor has it that Tom Cruise had to rest often during filming these scenes.

A mouse click, for example is one of the most physically efficient things I can think of.

Let’s look at the keyboard and mouse to understand their key benefit: most of us can use them all day every day without exhaustion or too much physical repetitive stress damage. A mouse click, for example is one of the most physically efficient things I can think of: we don’t even have to lift a finger, merely apply a bit more pressure, and we can move mountains of atoms or worlds of ideas. And we can do it thousands of times per day.

The table the keyboard rests upon is always there to support our weak arms. We rarely think about it…

Along with the mouse, we have a keyboard. Both these devices have a hugely important interface element: a palm (or wrist) rest. Well, the keyboard may or may not have it, but the table the keyboard rests upon is always there: it provides that solid structure upon which we can rest our palms and arms. We rarely think about it: but it is vital for all but the most athletic computer users.

Should we be able to wiggle our fingers in our laps? Right now, Leap Motion can’t see our laps.

This unfortunate practical reality changes the conversation. At a very good talk by Jody Medich in San Francisco (Metaverse Scholar’s Club) last week, she discussed that the future of this field is in “multi-modal” interfaces. Speech is great, but too limited by physical environmental concerns. Keyboards are the really best way to enter text, but they are problematic when visually blocked by a VR headset. Gestural interfaces hold the greatest promise but they suffer from inefficiency of motion. I wonder: is it possible that there is no great solution? Brainwave tech, maybe?

So the solutions are going to be cobbled together: maybe some voice, some gamepad, some keyboard, some wearable miniature tech for precision, and some video-based gesture recognition for large-motion occasional context-switching. At least for now.

I don’t have the answer: sorry to disappoint. But I do look forward to the challenge. I’ll be designing a system that will try to elegantly solve this problem, in a practical way, using current hardware (over which I have no design control). I won’t be alone.