Jessica Vascellaro, of the Wall Street Journal, reports about gesture, motion. and even object control for computers, highlighting the work of Leap Motion and Flutter.
Apparently the Leap Motion sensor is less expensive than Microsoft's Kinect. It can track movements down to 1/100 of a millimeter and can track fingers and movement. It handles interaction with 8 cubic feet of space.
Although I'm currently exploring the world of interactive HTML5, interactive video, etc., I think I just might make "kinecteducation" the focus of my tech-hobbies. I have some experience with game programming-one of my computer courses required a project using XNA- and I know quite a bit about gesture and multitouch, multi-user interactio, so it would'nt be too much of a stretch.
My motivation?
As a school psychologist, my main assignment is a school/program for students with disabilities, including about 40 or so who have autism spectrum disorders. Yesterday, the principal of the school attended a demonstration of the Kinect and requested that our school be considered for piloting it. One of my other assignments is a magnet high school for technology and the arts, and rumor has it that it will be offering a game programming curriculum. I'd love to co-sponsor an after-school game club and encourage the students to program educational apps for the Kinect sometime in the near future!
I'm also working as a client, in collaboration with come of my educator colleagues, with a team of university students who are creating a communication/social skills game suite geared for students with autism and related disabilities....
I'm inspired by the possibilities!
We have large SMARTboards in each classroom and in other locations around the building, and we have a Wii set up in the large therapy room adjacent to my office. The Wii has proven to be very useful in helping the students develop social and leisure skills that they can use in and outside of the school settings, but some of the students have difficulty manipulating the buttons on the controllers.
The video below features an interactive window created by the Marcel agency for a Repetto store in France. The display allows passers-by the ability to interact with the content in the display through arm gestures via motion sensors.
Nicolas Loeillot, of LM3LABS, has been ahead of the natural user interaction/ interface game for many years as his company has expanded. He's done quite a bit of deep thinking about the work of his company, and has used this wisdom to create a nice concept map that describes how LM3LABS' solutions fit into the world of gesture-based control and interaction:
In my opinion, this chart would make a great template for mapping out other natural interaction applications and products!
Here is the description of the concepts outlined in the chart:
"If all of them belong to the “gesture control” world, the best segmentation is made from 4 categories:
Finger tracking: precise finger tracking, it can be single touch or multi-touch (this latest not always being a plus). Finger tracking also encompasses hand tracking which comes, for LM3LABS products, as a gestures.
Body tracking: using one’s body as a pointing device. Body tracking can be associated to “passive” interactivity (users are engaged without their decision to be) or “active” interactivity like 3D Feel where “players” use their body to interact with content.
Face tracking: using user face as a pointing device. It can be mono user or multiple users. Face tracking is a “passive” interactivity tool for engaging user in an interactive relationship with digital content.
Image Tracking: Augmented Reality (AR) lets users use images (flyers, real products, t-shirts, faces,…) to interact with digital content. AR can be markerless or marker-based. Markerless technology has advantages but marker-based AR is easier for users to understand. (Please note here that Markerless AR is made in close collaboration with AR leader Total Immersion)." -LM3LABS
If you are interested in this subject and want to view some good examples of off-the-desktop interfaces and interactions, take a look at the LM3LABS blog, as well as Nicolas Loeillot's Vimeo channel. Also take a look at the sample of posts I've written about LM3LABS over the last few years - the links are at the end of this post.
I first discovered LM3LABS when I was taking a VR class and researching interactive, immersive large displays in 2005 or 2006. Back then, there wasn't much information about this sort of technology. A lot has changed since then!
I've learned quite a bit from watching LM3LABS (and others) grow, given my passion for postWIMP interactive technology and my commitment to blogging about this subject. Nicolas has really worked hard in this arena. As early as 2005, LM3LABS was working with Scala to provide "smart" interactive displays, and his company's applications have been supported by computer vision technologies for many years, allowing for gesture-based, or "touch-less" interaction, as demonstrated by the Catchyoo Interactive Table. This application caught my eye back in early 2007, when I was working on projects for large interactive displays for my HCI and Ubicomp classes, and was thinking about creating a table-top application.
My hunch is that LM3LABS has set the foundation for further growth in the future, given the lessons they've learned by taking risks with postWIMP technologies over the past few years!
"Founded in 2003 by a team of passionate researchers, engineers, designers, and marketers from various international backgrounds, focused on fast transformation of innovation into unique products, LM3LABS is a recognized pioneer in computer vision-based interactivity solutions. Keeping a strong customer focus, LM3LABS' team of unique people pioneers new directions, explores new concepts, new technologies and new interactions. Engaging, playful and magic, LM3LABS' products and solutions are always scalable and reliable"
info@lm3labs.com
Note to readers:
Over the past couple of years there has been an explosion of postWIMP technologies and applications, and with this pace, it has been difficult for me to keep abreast of it all. There is quite a bit I miss, given my full time job and daily life!
I welcome information about postWIMP interactive technologies and applications from my readers. Due to time constraints, not interest, I am not always able to post about a topic as soon as I'd like. That is OK, as my intention is not to be the first blogger to spread the latest tech news. I like to dig in deep when I can and make connections between innovative, interesting technologies and the people and ideas behind them.
As soon as Kinect was released by Microsoft, there was a flurry of app development. Evoluce and So Touch partnered to create a presentation application for the Kinect that could be used in work settings. Take a look!
Information about Air Presenter Plus, from the So touch's YouTube channel:
"So touch, the leading creative software company for new digital technologies, in partnership with Evoluce, the leading provider of advanced multi-touch screen technologies, present: So touch Air Presenter for Kinect. The world's first presentation software optimized for Kinect.
Turn your corporate presentations, welcome areas, trade show booths and point of sales into mind boggling experiences, controlling your presentation with multi-touch gestures leveraging So touch Air Presenter gestures software and Evoluce Kinect Windows 7 software.
Integrate your usual PDF, Power point, JPG and video materials into So touch multi-touch minority report's style interface and control it with gestures in the air.
So touch Air Presenter is delivered with a very graphic player, featuring a multi-touch zoom mode and an integrated video player as well as a very easy to use content manager.
So touch Air Presenter content, sourced locally or from the network, can be played on multiple screens at the same time. So touch Air Presenter content manager can deliver customize or generic content to each player.
So touch Air Presenter packaged with Evoluce Kinect Windows 7 software will be released soon. So touch Air Presenter is already available for TUIO based gestures devices. To know more and download a free trial version, visit http://www.so-touch.com/air-presenter"
Martin Kaltenbrunner's description of TuioKinect: "TuioKinect tracks simple hand gestures using the Kinect controller and sends control data based on the TUIO protocol. This allows the rapid creation of gesture enabled applications with any platform or environment that supports TUIO tuio.org/ You can download the application from: code.google.com/p/tuiokinect/ Music: Jabon Jabon by El Club de los Astronautas (Institut Fatima)"
I've played around with Tuio and OpenFrameworks, but it has been a while. I can't wait until I have time to dig into this with a Kinect. I think this has great potential for supporting learning and communication among students with special needs.
RELATED The following video is from the Canesta3D YouTube channel. It demonstrates the 3D input sensor in action, with four people moving around in a living room. The chip used in the system depicted in the video was the precursor to the current chip, called the "Cobra 320x200".
Below is a demo of gesture interaction using Canesta3D technology to control and select information and content on a large display. In my opinion, this will change the way we interact with our TV's, at least for those of us who hate using bad remotes! Microsoft's acquisition of Canesta is good news, especially if they allow this technology to be used by the masses. I'm pretty sure it has the capability of supporting interaction with HD TV's are internet-ready, and can support GoogleTV, LeanBack, and Vimeo's Couch Mode.
About Canesta (From the Canesta website) "Canesta (www.canesta.com) is the inventor of revolutionary, low cost electronic perception technology and leading provider of single chip CMOS 3-D sensors that fundamentally change the relationship between devices and their users. This capability makes possible true 3-D perception as input to everyday devices, rather than the widely understood 3-D representational technologies as output. Canesta’s 3-D input technology, based upon tiny, CMOS 3-D imaging chips or “sensors”, enables fine-grained, 3-dimensional depth-perception in a wide range of applications. Products based on this capability can then react on sight to the actions or motions of individuals and objects in their field of view, gaining levels of functionality and ease of use that were simply not possible in an era when such devices were blind. Canesta’s focus is on mass market consumer electronics, but many applications exist in other markets as well. Canesta is located in Sunnyvale, CA. The company has filedin excess of fifty patents, 44 of which have been granted so far."
I posted some videos about Canesta's technologies on the following post. There are two videos that show Canesta's 3D depth camera works on a Hitachi flat-panel display: Interactive Displays 2009 Conference
Dan Saffer is one of my "important influences". When I was taking HCI and Ubiquitous Computing courses, I bought the first edition of his book, Designing for Interaction: Creating Innovative Applications and Devices.In today's world of technical convergence, it is an important read, as Saffer's content crosses a number of disciplines.
Thoughts: It doesn't surprise me to learn that the #1 book on Saffer's Essential Interaction Design Books list is list is Don Norman's The Design of Everyday Things. According to Saffer, "there’s no getting around it: this isthebook. Affordances, mental models, and other bits that have all become part of the general lexicon all started with The Don’s book. A must read." Don Norman's book was required reading in the Human-Computer Interaction class I took a few years ago. As I read through the book, I sensed a familiar tone. I later learned that Don Norman was the co-author of a required textbook for one of the psychology courses I took when I was a university student the first time around.
Don Norman's thinking has influenced me for decades - he continues to be an influence, because he writes articles for one of my favorite publications, Interactions Magazine:
It brightens up my day when I open up my mailbox- the one at the end of my real-life driveway- and find my Interactions magazine, in all of its well-designed, well-written, semi-glossy-paged glory, waiting for me to open up and read. The September/October, 2010 issue includes articles on topics related to authenticity in new media, the complexity of "advancement", design and usability, and the politics of development.
Here is an excerpt from the article, which highlights some of the problems of rushing to get products with natural-user interfaces out to market:
"Why are we having trouble? Several reasons:
The lack of established guidelines for gestural control
The misguided insistence by companies (e.g., Apple and Google) to ignore established conventions and establish ill-conceived new ones.
The developer community’s apparent ignorance of the long history and many findings of HCI research, which results in their feeling empowered to unleash untested and unproven creative efforts upon the unwitting public"
(Interactions Magazine is a publication of ACM CHI -Association of Computing Machinery, Computer-Human Interaction interest group).
Here are a list of books/articles, suggested by Dan Saffer's readers:
Designing for Interaction – Saffer, D. (2nd Edition; 2009) Thoughts on Interaction Design – Kolko, J. (2009) The Humane Interface – Raskin, J. Digital Ground – McCullough, M. Inmates are running the Asylum – Cooper, A Designing Interactions – Moggridge, B (ed.) Everyware – Greenfeild, A. Designing Social Interfaces – Malone & Crumlisch Emotional Design – Norman, D. Invisible Computer – Norman, D. Persuasion Technology – Fogg, BJ Thoughtful Interaction Design: A Design Perspective on Information Technology by Jonas Lowgren and Erik Stolterman (Paperback – Mar 30, 2007) Designing Visual Interfaces by Mullet/San Steve Krug – Don’t Make Me Think: A Common Sense Approach to Web Usability Design Research: Methods and Perspectives edited by Brenda Laurel Information Architecture (“The Polar Bear Book”) by Peter Morville.
The video below gives a demo of Sony's RayModeler, "A 360-Degree Display that doesn't require glasses". The video shows how the auto-stereoscopic 3D content is filmed. It also shows how items within the display respond to gesture interaction. The first prototype was introduced in 2009 and then brought out at the SIGGRAPH conference this summer.
According to an article written by Richard Lawler, Core77 created "Breakout" for the RayModeler, a game similar to Pong. I'll have to think more about this technology before I form an opinion!
Project Natal was the code name for the Kinect Sensor for Xbox 360. For $149.99 you can pre-order your very-own system from the Microsoft Store that will allow you to interact with video games with your body alone. No need for controllers or 'motes!
Presentation about the fitness benefits of the Kinect Sensor for Xbox 360:
This video is a preview of a dance game for the Xbox using the Kinect Sensor:
It would be great if I could do my Zumba moves with Kinect Sensor system and a great Xbox application!
Here's another video that explains the system in more detail, with brief interviews of innovators from Microsoft:
Here is a copy of my previous post about Project Natal:
How It Works: Microsoft's Project Natal for the Xbox 360 video from Scientific American
Microsoft gathered a wealth of biometric data to recognize the range of human movement in order to develop an algorithm for the next generation of controller-less gaming. "Natal will consist of a depth sensor that uses infrared signals to create a digital 3-D model of a player's body as it moves, a video camera that can pick up fine details such as facial expressions, and a microphone that can identify and locate individual voices."
The technology behind Natal has the potential for a range of uses beyond gaming.
