Musings about NUI, Perceptive Pixel and Microsoft, Rapid Creative Prototyping (Lots of video and links) Revised

It just might be the right time for everyone to brush up on 21st century tech skills. iPads and touch-phones are ubiquitous. Touch-enabled interactive whiteboards and displays are in schools and boardrooms.  With Microsoft's Windows 8 and the news that the company recently acquired Jeff Han's company, Perspective Pixel, I think that there will be good support - and more opportunities- for designers and developers interested in moving from GUI to NUI.    


In the video below, from CES 2012, Jeff Han provides a good overview of where things are moving in the future.  We are in a post-WIMP world and there is a lot of catching up to do!

CES 2012  Perceptive Pixel and the Future of Multitouch (IEEE Spectrum YouTube Channel)



During the video clip, Jeff explains how far things have come during the past few years:

 "Five and 1/2 years ago I had to explain to everybody what multi-touch was and meant. And then, frankly, we've seen some great products from folks like Apple, and really have executed so brilliantly, that everyone really sees what a good implementation can be, and have come to expect it.  I also think though, that the explosion of NUI is less about just multi-touch, but an awareness that finally people have that you don't have to use a keyboard and mouse, you can demand something else beside that.  People are now willing to say, "Oh, this is something I can try, you know, touch is something I can try as my friendlier interface"."

Who wouldn't want to interact with a friendlier interface?  Steve Ballmer doesn't curb his enthusiasm about Windows 8 and Perceptive Pixel.  Jeff Han is happy how designs created in Windows 8 scales for use on screens large and small. He explains how Windows 8 can support collaboration. The Story Board application (7:58) on the large touchscreen display looks interesting.

I continue to be frustrated by the poor usability of many web-based and desk-top applications.  I like my iPad, but only because so many dedicated souls have given some thought to the user experience when creating their apps.  I often meet with disappointment when I encounter interactive displays when I'm out and about during the day.  It is 2012, and it seems that there are a lot of application designers and developers who have never read Don Norman's The Design of Everyday Things!

I enjoy making working prototypes and demo apps, but my skill set is stuck in 2008, the last year I took a graduate-level computer course.  I was thinking about taking a class next semester, something hands-on, creative, and also practical, to move me forward. I can only do so much when I'm in the DIY mode alone in my "lab" at home.  I need to explore new tools, alongside like-minded others.  


There ARE many more tools available to designers and developers than there were just four years ago.  Some of them are available online, free, or for a modest fee.  I was inspired by a link posted by my former HCI professor, Celine Latulipe, to her updated webpage devoted to Rapid Prototyping tools. The resources on her website look like a good place to start for people who are interested in creating applications for the "NUI" era.  (Celine has worked many interesting projects that explore how technology can support new and creative interaction, such as Dance.Draw.) Below is her description of her updated HCI resources:

"New HCI resource to share: I have created a few pages on my web site devoted to Rapid Prototyping tools, books, and methods. These pages contain reviews of various digital tools, including 7 different desktop prototyping apps, and including 8 different iPad apps for wireframing/prototyping. I hope it's useful to others. Feel free to share... and please send me comments and suggestions if you find anything inaccurate, or if you think there is stuff that I should be adding. I will be continuing to update this resource." -http://www.celinelatulipe.com (click on the rapid prototyping link at the top)

IDEAS
Below are just a few of my ideas that I'd like to implement in some way. I can't claim ownership to these ideas- they are mash-ups of what comes to me in my dreams, usually after reading scholarly publications from ACM or IEEE, or attending tech conferences. 

• An interactive timeline, (multi-dimensional, multi-modal, multimedia) for off-the-desktop interaction, collaboration, data/info analysis exploration.  It might be useful for medical researchers, historians, genealogists, or people who are into the "history of ideas".  Big Data folks would love it, too. It would handle data from a variety of sources, including sensor networks. It would be beautiful to use.
• A web-based system of delivering seamless interactive, multi-modal, immersive experiences, across devices, displays, and surfaces. The system would support multi-user, collaborative interaction.  The system would provide an option for tangible interaction.
• A visual/auditory display interface that presents network activity, including potential intrusions, malfunctions, or anything that needs immediate attention that would be likely to be missed under present monitoring methods. 
• Interactive video tools for creation, collaboration, storytelling.  (No bad remote controllers needed.)
• A "wearable" that provides new ways for people to express and communicate creatively, through art, music, dance, with wireless capability. (It can interact with wireless sensor networks.)*
• An public health application designed to provide information useful in understanding and sepsis prevention efforts. This application would utilize the timeline concept describe at the top of this list. This concept could also be useful in analyzing other medical puzzles, such as autism.

Most of these ideas could translate nicely to educational settings, and the focus on natural user interaction and multi-modal i/o aligns with the principles of Universal Design for Learning, something that is important to consider, given the number of "at-risk" learners and young people who have disabilities.

I welcome comments from readers who are working on similar projects, or who know of similar projects.  I also encourage graduate students and researchers who are interested in natural user interfaces to and move forward with an off-the-desktop NUI project.  I hope that my efforts can play a part in helping people make the move from GUI to NUI!  



Below are a few videos of some interesting projects, along with a list of a few references and links.

SMALLab (Multi-modal embodied immersive learning)

PUPPET PARADE: Interactive Kinect Puppets(CineKid 2011)



MEDIA FACADES: When Buildings Start to Twitter

HUMANAQUARIUM (CHI 2012)

 

NANOSCIENCE NRC Cambridge (Nokia's Morph project)






 
Examples: YouTube Playlists
POST WIMP EXPLORERS' CLUB
POST-WIMP EXPLORER'S CLUB II

Web Resources
Celine Latulipe's Rapid Prototyping Resources 
Creative Applications
NUI Group: Natural User Interface Group
OpenFrameworks and Interactive Multimedia: Funky Forest Installation for CineKid
SMALLab Learning

Microsoft Research Natural User Interface 

OpenExhibits: Free multi-touch + multiuser software initiative for museums, education, nonprofits, and students.
OpenSense Wiki 
CINEKID 2012 Website 
Multitouch Systems I Have Known and Loved (Bill Buxton)
Windows 8
Perceptive Pixel
Books
Natural User Interfaces in .NET  WPF 4, Surface2, and Kinect (Josh Blake, Manning Publications)
Chapter 1 pdf (Free)
Brave NUI World: Designing Natural User Interfaces for Touch and Gesture (Daniel Wigdor and Dennis Wixon)
Designing Gestural Interfaces (Dan Saffer)
Posts

Microsoft's"Magic Wall" Tech on Windows 8 Laptops and Tablets by Next Year

Bill Snyder, ReadWrite Web, 7/20/12

I noticed some interesting tools on the Chrome web store - I plan to devote a few more posts to NUI tools in the future.

Looking forward to CHI 2012: Child Computer Interaction, Educational Interfaces, Health, Special Needs, and Much More!

I am looking forward to attending the ACM-CHI 2012 conference!


On Saturday and Sunday I'll be at the Educational Interfaces, Software, and Technology (EIST) workshop, which is part of the Child Computer Interaction/HCI and Kids community.  I've selected a few papers/presentations/panels that I hope to attend during the remainder of the conference below.


If you are curious about other topics that will be presented at CHI 2012, take a look at the short video previews on the CHIMadness2012 YouTube channel.  

VocSyl


VocSyl: Designing Visualizations to Facilitate Multisyllabic Speech with Children with Autism and Speech Delay (pdf)
Joshua Hailpern, Andrew Harris, Reed LaBotz, Brianna Birman, Karrie Karahalios, Laura DeThorne, Jim Halle (Social Spaces Research Group, University of Illinois at Urbana Champaign)

Social Scripts
 
Fatima Bourjarwah: Facilitating the Authoring of Multimedia Social Skills Instructional Modules for Adolescents with High-Functioning Autism

Autism and Maths Tutor
Research Showcase - Autism & Maths tutor - CHI 2012 from George Chan on Vimeo.

ShoeSense


Giles Bailly, Joerg Mueller, Michael Rohs, Daniel Wigdor, Sven Kratz
ShoeSense: A New Perspective on Hand Gestures and Wearable Applications


Escobedo, L., Nguyen, D.H., Boyd, L., Hirano, S.H., Rangel, A., Garcia, D., Tentori, M., & Hayes, G.R. (2012). MOSOCO: A Mobile Assistive Tool to Support Children with Autism Practicing Social Skills in Real-Life Situations. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2012). Austin, Texas (May 5-10, 2012). New York: ACM Press. To appear

Design of an Exergaming Station for Children with Cerebral Palsy

(Research team from Queen's University, Canada, Holland Bloorview Kids Rehabilitation Hospital)


Illumishare


Paper: IllumiShare: Sharing Any Surface

Sasa Junuzovic, Kori Inkpen, Tom Blank, Anoop Gupta, Microsoft Research

Panel: Tangible Interfaces for Children: Cognitive, Social & Physical Benefits and Challenges
Shuili Gilutz, Sandra Calvert, Kathleen Kremer, Barbara Chamberline, Geri Gay

Paper: Improving Literacy in Developing Countries Using Speech-Recognition-Supported Games on Mobile Devices 
Anuj Kumar, Pooja Reddy, Anuj Tewari, Rajat Agrawal, Matthew Kam


Tap & Play: And End-User Toolkit for Authoring Interactive Pen and Paper Language Activities (pdf) Ann Marie Piper, Nadir Weibel, James D. Hollan


RELATED
About the CHI Conference (from the CHI 2012 Conference Preview)
"Originally a small conference for psychologists interested in user interface design, the annual CHI conference has grown to include a very diverse participant group (such as interaction designers, computer scientists, engineering psychologists, developers, performing artists and more). It has also grown to deal with larger problems such as the organizational integration of technology and the use of technology in the home rather than only office settings. This year's conference marks 30 years of research, innovation and development in the field of Human-Computer Interaction and is expected to draw more than 2500 professionals from over 40 countries. The experience at CHI 2012 offers innovative opportunities for interacting with future technologies. The following areas represent a small portion of the total conference. For complete information about this year's conference, consult the Advance Program."


"Featuring over 900 works, the CHI conference is the premier worldwide forum for the exchange of information on all aspects of human-computer interaction. Typically the works presented address the concerns of design, engineering, management and user experience professionals. This year's conference also features works that focus on: Digital Arts, Games and Entertainment, Human-Computer Interaction for Kids, Health and Sustainability. Works are presented in several different venues. After two days of intimate pre-conference workshops, the main conference includes panel discussions, courses that provide leading-edge HCI knowledge, paper sessions, case studies, works-in-progress, student competitions (involving design, research and games), interactive demonstrations, special interest groups, and a Tuesday evening video night (popcorn included)." 



SIG-CHI is the Special Interest Group on Computer-Human Interaction, and is part of ACM, the Association for Computing Machinery


Child Computer Interaction/HCI for Kids Chairs:
Janet C Read, University of Central Lancashire, UK
Panos Markopoulos, Eindhoven University of Technology
Allison Druin, University of Maryland


Educational Interfaces, Software, and Technology workshop organizers
CHI 2012 Facebook Page
EIST 2012 Facebook Page


SOMEWHAT RELATED 
The Autism Experience in Ubiquitous Computing (pdf)
Monica Tenority, Ph.D., UZBC, Ensenada, BC


Gillian Hayes:  Assistant Professor in Informatics, School of Information and Computer Sciences; Department of Education, University of California, Irvine.
Social and Technological Action Research (STAR), 

Interactive Technologies for Children with Special NeedsIDC 2012 
(11th International Conference on Interaction Design and Children)

As special themes, IDC 2012 would like to discuss children's needs under the perspectives of "pedagogical aspects in theory and practise" as well as "children from diverse cultural backgrounds".


Grawemeyer, B., Johnson, H., Brosnan, M., Ashwin, E., Benton, L. (2012) Developing an Embodied Pedagogical Agent With and For Young People with Autism Spectrum Disorder. Accepted for presentation at the 11th International Conference on Intelligent Tutoring Systems, ITS 2012.

Seamless Collaborative Computing Between Tables and Tablets: nSquared Presenter -Video

I came across the following video about nsquare's presenter application. It looks like it has potential.  It supports "seamless" collaboration between people, multimedia content, interactive tables, interactive touch screens, and tablet devices.






RELATED
nsquared website
Video presentation about "Seamless Computing", by Neil Roodyn


More blog posts on the topic of natural user interaction, interfaces, and examples of seamless computing are planned for the future!


This was shared by one of my FB friends.

Link to a great rant about interaction design. It is all about the hands, screens and surfaces, not so much!?

Brief Rant on the Future of Interaction DesignBret Victor, 11/10/11 Via Putting People First

GarageBand on the iPad2 is Awesome!

For just $4.99, this app has given me hours of joy - and I've only had it for a week!  The first time I played around with it, I exclaimed that I no longer have a need for my Motif 8 keyboard.

(Seriously, my husband now thinks I'll be unloading my keyboard on eBay or Craigslist soon.)

Below are a couple of videos to give you an idea of what the iPad version of GarageBand is all about:

GarageBand for iPad



Be sure to visit soon for an update - I'll include videos of my own.

Vision-Based Hand-Gesture Applications: Video from Communications of the ACM

The latest edition of Communications of the ACM, via "snail mail", was the inspiration for this post:

Vision-Based Hand-Gesture Applications
Juan Pablo Wachs, Mathias Kolsch, Helman Stern and Yael Edan

"Body posture and finger pointing are a natural modality for human-machine interaction, but first the system must know what it's seeing."


More to come!

Video: DaVinci Surface Physics Illustrator Interface on Xbox Kinect, with gesture interaction, by Razorfish

DaVinci prototype on Xbox Kinect from Razorfish - Emerging Experiences on Vimeo.

RELATED
Razorfish ports DaVinci interface to Kinect, makes physics cool (video)
Time Stevens, Engaget, 12/5/10
Razorfish
(I love this website.)

For the iPad: Reactable Multi-touch Mobile

More details later!

RELATED
History of the Reactable


How it works (Info from the Reactable website)
"The way the Reactable works is deeply inspired by modular analogue synthesizers such as those developed by Bob Moog in the early 60s and enhances them with new concepts of sampling and digital effects processing."

"While with modular synthesizers one typically had to connect these different modules with patch cables in a complex and error-prone process, with the Reactable this is attained in a much easier and more intuitive way, since connections between the blocks are managed automatically based on their types and affinities and on their proximity."


Martin Kaltenbrunner (Co-founder of Reactable)
"His research concentrates on tangible user interfaces and the development of novel human computer interaction concepts within open tools for creative production. As co-founder of Reactable Systems he had been mainly working on the interaction design of the Reactable - an electronic musical instrument with a tangible user interface. He is author of the open source tangible interaction framework reacTIVision and the related TUIO protocol, which have been widely adopted for the realization of tangible tabletop applications."


SOMEWHAT RELATED
Soundythingie for the iPad
iPad Apps for Making Music: What's Coming Later
SurfaceDJ iPhone App (Vectorform)