Are we rewriting our emotion-memories with fact-memories?

Jason Marsh

Some of our most personal moments are being captured for history, and without our conscious thought. An event in my household this week struck me as a particularly poignant case.

Yesterday, my wife’s mother passed away, in her own home, with a nurse at her side. She lived 500 miles away, but she and my wife had been together a few days earlier, and her mother passing relatively soon after her visit was not wholly unexpected. When the nurse noticed a change in her mother’s condition, she called my wife and rested the phone by her mother’s ear. My wife spoke to her about spirituality, personal memories, and love for about half an hour. After some silence, the nurse came on the phone to say that she had passed peacefully.

In this personal and intimate moment, technology played an interesting role.

Unknown to my wife at the time, her cell phone recorded her final conversation with her mother.

Earlier in the week, my wife was experimenting with a recording app on her phone. When she received the nurse’s call, she forgot it was active. When the call ended, the recording app let her know that the last 33 minutes with mother had been digitized and saved.

One of life’s most personal moments is captured for posterity. A key point is that it was completely non-self-conscious: the technology was just ‘on’, always recording, and didn’t influence the real life that was happening.

Now she has a 33-minute recording that includes the moment of passing. I can’t imagine anything more personal, more real, more profound. And what does one do with it? Share it with her family? Yes. Keep it for the grand-kids? Maybe. Ever listen to it again? No… maybe…?

I’m struck by the comparison to the Samsung commercial for Gear VR which showed a real-time feed of a birth between a mother on one side of Australia to a father across the continent. It was amazing, but different in that the tech in the room had to be significant and invasive.

samsungBirth

What happens when Augmented Reality glasses, or body-cams, or dash-cams, or who-knows-what-cams capture more of these significant, most-personal moments, non-intrusively and continuously? So I wonder:

How will it affect our memories of the event?

These memories are so influenced by emotion: reality is often blurred through our emotional experience. But now these emotions are likely to be re-written by the facts, just the facts. They say that the brain re-writes the memory each time it is replayed in mind. Do we want only facts to remain in mind from these emotional moments, losing these first-person emotion-memories?

memories

In speaking with a friend involved in the venture capital side of Japanese VR, I learned that there is an expectation there that cinematic VR capture will be huge. Weddings, vacations, and daily life will be captured. In a wedding, the bride creates an emotion-memory while standing beside her fiancé, looking into his eyes from her unique perspective. But when she sees the VR video created from a camera off to the side, her first-person perspective will be changed. She will be removed from the truly special first-person experience with all its unique emotions. And when she looks at the video a few times, her first-person emotion-memory will be weakened, replaced with the memory of the VR experience and its unyielding factual detail, perhaps drilled deeper through repetition.

The implications of always-on recording are huge. There are many forces moving us that direction.

For example, a while ago I was reading about a problem with determining the facts related to disputed sexual encounters on campuses. A huge increase in sexual assault allegations has led to campus adjudicators deciding between conflicting he-said/she-said testimony; testimony often impaired by alcohol. The resulting decision implications are drastic: a crime may be ignored leaving a perpetrator at large, or an innocent young man’s college education, and therefore career, may be ruined. Since consent can be withdrawn at any time, no prior agreements can hold. There is only one way to be sure: video recording. So another intensely personal moment, perhaps even the loss of one’s virginity, may be captured on tape in order to remove the ambiguity. The emotion-memory of the event could well be replaced by the fact-memory on tape.

Technology, no doubt, will enhance our fact-memories. If kept hidden away, ten years from now, that VR wedding recording will flood us with emotion. But viewings well before that period will happen.

At what cost to our emotion-memories?

This is not to bemoan the value of facts, which have their place, such as the use of police body-cams. But VR feels different than the traditional 2D media. We are not as likely to easily replace our real memories with flat photos, although that also does happen over time. The emotional depth of VR, once again, may surprise us.

My immediate question is: what should we do with a recording of the last earthly moments of a beloved mother? We’re not sure.

[8/22/2015 Edits: my wife corrected a few minor details, length of call, etc…. and a few phrases for clarity.]

A redesign for the boring resume

Jason Marsh

,

When I started my adventure with VR a year ago, I thought it would be cool to create a VR resume. As with all things VR, there is so much room for disruption, and when I started showing my interactive 3D resume to folks, I got great responses. In fact, Tony Parisi saw it and within 90 seconds said “we’ve got to talk.” And soon he invited me to co-found a company. So I can say it worked for me.

I started with a 2D version. I’m an information designer, so I designed it to reflect the way resumes should be. Using a few lines and words, it re-defines the boring old resume into a timeline, statements of value, and interests. I graphed out my career based on key principles that I care about: creativity, learning, level of responsibility for organizational vision and organizational impact.

Jason Marsh resume

This is the ‘leave-behind’ version. The in-person interactive version has the color rectangles without the lines. On paper, since I always travel with multi-colored pens, I pull out the right pen and draw in the value lines as I discuss the areas relevant to the conversation. On my MS Surface, I draw onto it on-screen with the Surface pan.

I’m hoping that my audience will see that this gives him/her a way to quickly build a mental model of what I’ve done and how I feel about it. My experience with it has been very successful. In fact, it is surprising how little I get to describe my past: people get it, and want to launch into brainstorming what we can do together. As described in my post about information architecture, they get the ‘why’ of the big picture and the credibility of the micro picture, and the information transfer is deep and efficient.

And then, using WebGL in the browser, I pull up the 3D version and they can watch it lift off out of flat land. Here is a screen shot of an early prototype of the 3D version:

VR resume

Really this is about a conversation, not just the data. To be effective when flying through the timeline in VR, I think it would need to include a master-slave shared experience, and I’ve yet to build that out.

There are many places I could take this; perhaps you have ideas. Let me know @jmarshworks.

VR as a new way to see the world

Jason Marsh

My friend Issac Cohen (also goes by cabbi.bo), the brilliant digital interactive artist, spoke at a WebVR meetup this Spring, and while he waved his arms (with a LeapMotion) to interactively ripple a gorgeous metallic face-like blob, he said something like:

“I want people to have an amazing experience in VR, and come out, look around at the real world, and be astounded at the Ultra-HiDef  detail and wonderfulness of being in real reality. I want VR to change the way they see the world in real life and notice things they always used to overlook.”

So I gotta say he succeeded, at least with my family, and he did it in a browser, without VR.

I spent an hour going through his new storybook, Enough, first with my wife, and then the next day with my son and his girlfriend (college-aged). We were all stunned.

And then we went camping and hiking in the high Sierra for the weekend. All weekend long, the kids would say: look at how those ripply clouds are like the sandy ocean floor in Enough.enough

And, look at the color of the California Fuschia, and how it reminds me of the color of “Sol” a creature from the story.

enough2

 

These still shots do not do it justice. And watching it as a video doesn’t do it justice either. The interactive motion of the screen elements is such an important part of the experience: each individual letter is animated based on mouse movement.

I think that Issac is establishing a new art form. He calls it a storybook, and it is that. But it is unlike any storybook.

Instead of NetFlix tonight, put your family in front of a big monitor, or hook your laptop up to your TV, and simply go to the website: http://cabbi.bo/enough/ (it’s all WebGL and runs in the browser). We swapped off reading the text aloud and playing with the mouse.

What he’ll be able to do with translating it to VR is crazy… I hope he’s working on it.

And it changed our weekend in the mountains.

Human interaction for Virtual Reality: any practical solutions?

Jason Marsh

,

I’ve been spending quite a few hours researching and thinking about Virtual Reality input interfaces, and I’m developing a perspective I haven’t seen written about yet, so I’ll try to communicate here.

I’m now working on a big-vision, non-game project: a Virtual Reality activity that could consume hours a day, day after day. So I’m asking, what are the right practical interaction frameworks that would be both powerful and functional?

The number of options for alternative and gestural computer inputs in development is vast, with Leap Motion gaining the most public interest, but also devices like the  Nod ring, Microsoft Kinect, arm-bands, and a whole slew of wireless handheld controllers.

And of course, the artificial user interfaces (AUIs) seen in movies are such a gorgeous vision of what is possible: Iron Man and Minority Report are my two favorites.

This Iron Man interface was a key inspiration for me as I started on this VR journey, I like the way he can navigate and manipulate information:

But, as I’ll try to communicate, there are some serious practical issues with these futuristic visions.

Gesture-based inputs, such as Leap Motion, seem like they would be such a powerful, natural addition to the Virtual Reality experience, so I spent time learning to program the Leap Motion with LeapJS in a browser. Actually, a lot of time. I built a number of prototypes.

I came across this rather innovative interaction solution yesterday, more successful than most: Thomas Street’s UI Exploration with the Oculus Rift and Leap Motion, and this graphic showing the interaction modes.

The idea is that a series of menus (in this case they are spherical icons) are visible across the bottom of the view, and when you ‘gesture over’ each one by moving your hand back and forth, the menuitems (more spherical icons) appear going into the distance: move your hand forward a varying distance to gesture over your intended target, then rotate your hand to select.

The goal is to work around what is a pretty problematic device at this point. The lack of precise control is mitigated by mapping broad gestural strokes into a spatial grid to make a selection.

I gained another valuable lesson from this brilliant article by , also on LeapMotion’s blog: What Would a Truly 3D Operating System Look Like? She makes a convincing argument, which I agree with, that spatialization of information and content is a huge win: it is time to get past the limits of our 2D computer universe, with it’s ubiquitous frames (all screens have edges) around everything.

Information just wants to be FREE… with relationships visible everywhere: everything in its spatial place and always viewed within the myriad of relationships that screen edges inevitably obfuscate.

YET, for all that goodness, how are we to manipulate this new world? The mouse is pretty good for dealing with a 2D world, but unable to handle the third dimension.

Shall we wave our hands (literally above our pumping hearts) like Minority Report? For how many hours (actually minutes) will we last before we are tired and irritated? Rumor has it that Tom Cruise had to rest often during filming these scenes.

A mouse click, for example is one of the most physically efficient things I can think of.

Let’s look at the keyboard and mouse to understand their key benefit: most of us can use them all day every day without exhaustion or too much physical repetitive stress damage. A mouse click, for example is one of the most physically efficient things I can think of: we don’t even have to lift a finger, merely apply a bit more pressure, and we can move mountains of atoms or worlds of ideas. And we can do it thousands of times per day.

The table the keyboard rests upon is always there to support our weak arms. We rarely think about it…

Along with the mouse, we have a keyboard. Both these devices have a hugely important interface element: a palm (or wrist) rest. Well, the keyboard may or may not have it, but the table the keyboard rests upon is always there: it provides that solid structure upon which we can rest our palms and arms. We rarely think about it: but it is vital for all but the most athletic computer users.

Should we be able to wiggle our fingers in our laps? Right now, Leap Motion can’t see our laps.

This unfortunate practical reality changes the conversation. At a very good talk by Jody Medich in San Francisco (Metaverse Scholar’s Club) last week, she discussed that the future of this field is in “multi-modal” interfaces. Speech is great, but too limited by physical environmental concerns. Keyboards are the really best way to enter text, but they are problematic when visually blocked by a VR headset. Gestural interfaces hold the greatest promise but they suffer from inefficiency of motion. I wonder: is it possible that there is no great solution? Brainwave tech, maybe?

So the solutions are going to be cobbled together: maybe some voice, some gamepad, some keyboard, some wearable miniature tech for precision, and some video-based gesture recognition for large-motion occasional context-switching. At least for now.

I don’t have the answer: sorry to disappoint. But I do look forward to the challenge. I’ll be designing a system that will try to elegantly solve this problem, in a practical way, using current hardware (over which I have no design control). I won’t be alone.

What is an information architect?

Jason Marsh

I call myself an information architect. I don’t use it exactly the same way as Wikipedia defines it, but since there are similarities, I’ll start there:

Information architecture (IA) is the structural design of shared information environments; the art and science of organizing and labelling websites, intranets, online communities and software to support usability and findability; and an emerging community of practice focused on bringing principles of design and architecture to the digital landscape. Typically, it involves a model or concept of information which is used and applied to activities that require explicit details of complex information systems. These activities include library systems and database development. –Wikipedia

I read this definition as more of a software systems designer, kind of similar to a software architect.

This doesn’t quite fit my definition, because I’m not a hard-core programmer (although I did start programming 45 years ago at the age of 6 on a 6502 Jolt kit computer: pre-Apple I.)  I didn’t study Computer Science in school. I’ve been programming my entire career, but I’ve always focused on the user experience, and have focused exclusively on client-side software. This has guided me more toward script-based platforms, and since the Internet, I’ve focused on JavaScript, Flash, Adobe Flex, CSS, XML, XSLT… you get the idea. I just believe that focusing on the user experience is my unique value-add.

But my definition of “information architecture” transcends the realm of software.

Here is my definition:

Ideas have shape. A mental model of an idea can be created in a single consciousness, and transferred to others, though a process of discovery, diagramming, analyzing, and discussing. Sometimes an idea feels like it can be fully documented, and sometimes big areas of the idea are left to other parties to figure out. But architected ideas include the ‘why’: the big picture of where this concept fits into the next larger system of thought. And they include the concrete: the details of that make this realizable in the world, and therefore give it credibility. For years, I’ve described ideas like a blob, floating above the table we share when we discuss and deliberate, listen and argue. We can spin the blob, seeing it from different perspectives, flipping the axis, or discovering the N-dimensions that may be required to see it in adequate clarity.

I would agree with the Wikipedia definition in these aspects:

  • These ideas are usually complex (and real-world.)
  • Labeling is key: the process of categorizing, organizing aspects of an idea into mental buckets, is vitally important.

Ideas that need architecting can be seen as ‘problems’ needing solutions, in which case the both the problem and the solution may need to be analyzed and ‘architected’, and then the solution needs to fit the problem like a 3D Tetris.

Ideas that need architecting transcend a single consciousness. The first goal is to build a shared mental model, and that involves the concretizing of the idea into documentation.

Ideas that need architecting can’t be merely described in prose. Diagrams are key. Ideally diagrams should be able to ‘zoom’ from the macro view to the micro view, the why and the concrete, and then back to the macro view quickly to re-affirm the big picture. I use Prezi for this: not as a slideshow replacement, but as an interactive, sharable whiteboard with infinite zoom capabilities (I should do another post about this!) I also use colored pens to draw words and lines on pure white sheets of loose paper. The colors convey multiple dimensions, as well as provide a more memorable visualization. Words are key, even more important than little stick-figures or other visual representation. I say that a word is worth a thousand pictures. Put a word on a piece of paper in the middle of the table, and soon everyone is pointing to it and saying “this,” and everyone sees how “this” is connected to “that.” A new shorthand language appears spontaneously, and the team is actively developing a shared mental model.

Every discussion of consequence that uses language can be a candidate for information architecture, including politics, religion, history, and the arts. As an anti-example, pure art transcends language: for example, words only hint at the emotive expression of dance, and Taoism explicitly transcends words. Love transcends language so it is also not a good candidate, although that is an interesting idea: maybe if more of us had a clear mutual understanding our deepest relationship, many practical long-term benefits are possible.

Sidebar: I’ve realized I’ve violated my rule of information architecture! I described it in prose without diagrams and examples! Most of my best info design has been with medical customers with proprietary information, so I can’t share them as-is. I’ll have to rectify this in my next post!

Here is one trivial example of the micro/macro view: a photograph I took this morning:

Lupine on Table Mountain

Lupine on Table Mountain

What do I like about this photo? The crisp detail of a single flower is a micro view, very real and concrete. But it has its place in a vast field of flowers, which would merely be a wash of color without the micro view. And it has a relationship to the macro view (not as in “macro photographic lens”, but as in big-picture view) which is the moment of sunrise, which gives a sense of place and meaning to the flower. Your eye naturally flows around the photo: back and forth between the flower in focus, and the sun, and the tree, and the other close-up flowers, and the sun, circling around and around.

The micro view gives credibility to all the other flowers: you trust that you can understand each one of them without needing to zoom to each element. A manager will only look at one detail in a big picture: when she finds that the detail is adequate, then she assumes that all the other details are in good order and leaves them to her subordinates because she can trust them on the basis of the single well-documented detail.

The macro view becomes the story within which I can mentally file away the detail. In a Montessori classroom, a timeline rings the room’s walls right up against the ceiling. And every history lesson starts with students finding its time period on the timeline. This gives a mental filing-place for the lesson, and relationships are seen and remembered. An abstract history lesson without a proper context would be confused and forgotten, similar to what happens in many less-inspired classrooms.

 

By this point, I’ve spent many years doing information architecture. I do aspects of this at every meeting: if something is worth discussing, it is worth documenting and understanding. I try to see every meeting as an opportunity to create a shared, concrete, meaningful mental model. And that is how I define Information Architecture.

 

 

 

I’ve made the plunge into Virtual Reality

Jason Marsh

After nine years where I’ve been deep the world of medical software at Acesis, I’m finally free to pursue my next start-up.

I am a entrepreneurial guy, after leaving Apple in ’93, I’ve co-founded three companies/organizations:

  • ESCAtech Media – CD-ROM and computer-based media development, ad agency model
  • MarshWorks – interactive instructional media design and programming
  • Sierra Montessori Academy – K-8 public charter school based on Montessori and Project-Based Learning
  • Acesis – medical software for hospitals, multiple performance improvement applications

And after many months researching the VR space, and developing fun prototypes, I’ve successfully extracted myself from Acesis (although I’ll continue contracting with them for possibly many years.)

Stay tuned!

Pivoting this blog from music to Virtual Reality and Information Architecture

Jason Marsh

,

It’s been a while since I’ve updated this blog. I’ve been preferring to spread my online presence elsewhere, mostly within corporate internal mechanisms.

But that’s changing. So if you want to see and hear my musical endeavors from a few years ago, you’ll see them documented in older posts, and if you are here to see my current professional and creative endeavors and not interested in the music, don’t scroll down too far!

 

A House Built on Sand – new song

Jason Marsh

Now for something completely different!

My son inspired me with a DeadMou5 song Strobe, which is electronica dance music… typically called “house” music (which is why I needed the word ‘house’ to appear in my song’s name). It builds to that heavy 4/4 bass drum and has little or no melody. So… I did a piece that satisfies my musical architectural form, with some interesting sounds, all synthetic.

You can hear or download it on my bandcamp site:

http://jasonmarsh.bandcamp.com/track/a-house-built-on-sand

I know not all of my violin fans with think it is great. But bear with me while I do some experimentation, and if you listen to its energetic sonic soundscapes, you might be surprised how similar it is to my violin compositions. And the younger folks might dig it.

AirPhone at Palace of Fine Arts

Jason Marsh

I’ve posted a new YouTube!

I was looking for a creative project during a stay near the Marina in San Francisco, and we decided to do a Geek Dad project where we floated my iPhone 4s up amidst the columns of the Palace of Fine Arts.

It took 20 helium balloons to get it off the ground, and yes, at the end the wind came up just enough to get the balloons caught in a giant vase, so I needed to climb up and untangle it.

I composed a peaceful, ascending piece of music to enhance the gentle, undulating motion of the balloons.

1 2 3