Why Do We Need NUIs? Assistive Technology

If I had to pick one reason why we need natural user interfaces (NUIs) it would be for use in assistive technology. Technology for increased convenience is all well and good. Technology for improving our ability to multi-task may or may not be good. A growing body of literature demonstrates the more we multi-task the less productive and efficient we actually are. Disturbing to say the least.

However, there are people for whom NUIs have the potential to be life transforming. Jonathan Josephson of Quantum Interface (introduced in my last post) told me about an incredibly moving experience he had observing a quadriplegic trying out a motion based NUI prototype. Apparently, many partially paralyzed people have limited range of motion in their forearm but not their upper arm. After studying what the natural motions were in this situation, QI designed an interface that enabled this individual to interact with his computer using his arm even though he could not lift it. Consider: prior to that moment, this man had been forced to use a straw held in his mouth. Put yourself in his shoes and imagine what that would be like. 

 

How would you feel?

Jonathan's voice choked up as he told me how the user and his family members were virtually in tears. To them, this interface represented freedom and autonomy.

What is holding back full scale development and deployment of this type of NUI? Sensor technology for one thing. We need highly precise 3D pinpointing sensors to locate and track motions, and to enable fine tuned feedback. Fortunately, says Jonathan, these are on the horizon.

So when people get into discussions about what is "natural", there is no single, simple, answer. As pointed out by a LinkedIn reader in response to my last post, there will be cultural differences. There will be differences based upon ability and impaired ability. As we become increasingly sophisticated, and some inventors move into the realm of AI, cognitive issues will become increasingly important. For example, what is "natural" will be different for victims of head trauma.

Our society is undeniably digital and the more people who can access technology easily and naturally, the better. 

There is a ways to go before the assistive technology Jonathan envisions can be produced and marketed widely and affordably, but when it happens the societal impact has enormous potential. At the end of the day, this potential is what drives Jonathan in his NUI work.

Do you want to have a fight with your car's software?

Perhaps because it is Sunday and I had promised myself not to work today, yet find myself on the computer (again :) a little bit of irreverence and levity. Besides missing out on the sunny skies to plop myself in this ergonomically correct chair, I am working with 9 fingers, having accidentally smashed one of them into a steel pipe last weekend. Typing has been a bit challenging. But not nearly as much as flossing. Try flossing without both pinkies. It can be done but it takes creativity that would make my dentist proud.

I just listened to the podcast from National Public Radio's Science Friday. The host interviewed the high school pair who won the grand prize in the Siemens Foundation Math, Science and Technology contest  in.... yes... computer science. (The grand prize also went to someone working in astrophysics. NPR did not mention this, but it is on the Siemens Foundation contest web site. See above link) The prize is $100,000.  The host noted afterwards with some regret that he forgot to ask the teenagers what they would do with the money. I don't know...when you were 16, what would you have done with $100,000? That's a lot of [fill in your own blank].

The students developed code to identify emotions from your voice. The NPR host started off by describing how he would sometimes tell his automobile GPS to "shut up". And wondered if this technology would allow the car to adjust if it noted that he was pissed off. My first thought (remember, it is Sunday) was - so turn off the annoying GPS. It is bad enough when Google Maps takes me 5 miles out of my way and dumps me in the far end of a dead end parking lot when my desired destination was half a mile from where I originally got into my car in the first place. I have been in cars with people who cannot seem to drive without their GPS - brain, out the window. Annoying computer voice at the helm. ick....

So I'm not sure that I'd call that particular application of emotional recognition software socially beneficial. I might be more likely to get into an escalating argument with my car as it tried to be soothing and I became more and more annoyed at its cloying fakeness. Then I might find myself and my car in a ditch. Definitely not socially beneficial. Guess I should be clear that I'm pretty sure the car GPS application was the invention of the NPR host, not the students.

The students actually had something much more interesting in mind (assuming that your idea of fun and excitement does not involve fisticuffs with your dashboard). They are now considering creating a wristwatch that can be worn by Autistic children to help them in social situations. The watch will identify the emotion of the person being spoken to and show a picture (smiley face, frown, etc) to help the wearer correctly judge the mood of the other person. Now that is a bright idea and much more interesting I think. I hope they do it.

Meanwhile, I shall attempt to go back to not working....happy Sunday to all!

Designing for Inclusiveness

As part of some background research into the use of mobile devices in education, this morning I was reading a book called "design meets disability". The lack of capitalization is how the title is written. The author is Graham Pullin, publisher is MIT Press (2009).

There is a chapter that I found disturbing and I'm processing it aloud here. The chapter is called "expression meets information". The author makes several claims, which I paraphrase (hopefully I have not inadvertently mangled his intent - if so the fault is mine):

• The design of assistive devices for disabled people is often focused on the technology and coming up with a "one size fits all" device.
• Mainstream culture on the other hand now demands flexibility and customizability in electronic devices.
• Notions of the individual self, culture, and other factors are not sufficiently addressed in the design of assistive devices.
• Using assistive devices to communicate as *individuals* - which the mainstream community takes for granted (facial expression, tone of voice, body posture, choice of word and gesture) is not integrated into design as a primary requirement. Perhaps at all.
• The creation of assistive devices is or should be a very interdisciplinary activity: intercultural communications and diversity, attitudes, qualitative design methods, ethnography, computer science, engineering, and others.

One of Pullin's main points seems to be that while we as a culture value diversity, and design for it in many technological devices (sometimes rising to the level of becoming Bloatware), we toss all thought of encouraging individuality out the window when it comes to devices for the disabled. 

And to muddy the waters, he points out that the notion of individuality is very personal (hopefully that phrase is clearly redundant) and sometimes people don't want the latest and greatest "improvement". He uses Stephen Hawking as an example, claiming that the ultra famous physicist has declined an update to his voice synthesizing software, stating that the robotic voice has become "his voice". 

Pullin supports a paradigm shift, that comes back around to the role that computing professionals can play. The author suggests that there is no clear divide between the disabled and not disabled, but that we all exist on a continuum of ability - and that that understanding is what should drive our designs. It isn't just HCI Expanded. (HCI = Human Computer Interaction)

I'm still not sure why I was so disturbed reading this chapter. Perhaps it is the idea that as we work harder to create the "best" assistive devices, we may be doing more damage than good? The author doesn't say that, but that is something I read out of his discussion. I'm going to have to wrestle with this one. There are a lot of things that Pullin does not say outright, but dangles in front of you to consider.

However, and Meanwhile, there is a huge opportunity for interdisciplinary minded computing professionals who want to work in the field of device creation, whether for fun, work or daily living. Software engineering Reqs and Specs would screech to a stop and reconsider traditional approaches to data gathering. It means a whole new way of thinking and working across disciplinary boundaries for inclusiveness. 

The author provides an interesting example to demonstrate that the desired end result is reasonable and doable. He points out that audio books were originally created for the vision impaired, but have now become wildly popular with the mainstream community. Many people like the medium of listening to books. This application crossed over. Whether by accident or intent I do not know. But, what if we always created our work with this idea in mind?