Multi-touch and Gesture Interaction News and Updates You Might Have Missed (Part I)

Over the past couple of months, I've come across many interesting links related to multi-touch and gesture interaction, but I haven't had time to devote a thoughtful post to each one.  "Part I",  is a nice collection of experimental, commercial, and non-commercial efforts by a variety of creative technologists, with a smattering of industry news that might be of interest to IMT readers. 


Ideum's MT55 HD Multitouch Table 4/19/11

New MT55 HD Multitouch Table Now Shipping,  Jim Spadaccini, Ideum Blog 4/11/11

Smithsonian American Art Museum to Open Education Center  Sara Beladi, NBC Washington News, 4/4/11 (Rumor has it that the Smithsonian American Art might include touch and multi-touch displays in it's plans for a new education center.  The center was funded by an anonymous $8 million dollar gift.)

Some Milestones in Computer Input Devices: An Informal Timeline

Bill Buxton, Microsoft Research, 4/7/11 - Includes lots of pictures, links to videos, and more information of what might be the first touch-screen.  Also see Bill Buxton's companion website, Multi-Touch Systems that I have Known and Loved, updated on 3/21/11.  Bill Buxton knows all (almost!)

How to make a Multitouch Biggie (Seth Sandler)

"The MTbiggie uses the "Front Diffused Illumination" multitouch technique, with ambient infrared light and a DIY infrared webcam. The MTbiggie is similar to the MTmini, but includes a projected image and infrared webcam (rather than a normal webcam)...The MTbiggie isn’t the most stable and robust setup, but it is the easiest to build. To see other methods of building more stable multitouch displays, view the full multitouch display list." -Seth Sandler

(Also check out NodeBeat, a multi-touch music/audio sequencer/generator app by Seth Sandler and Justin Windle)

IntuiLab announces support for Microsoft Kinect interactive software solutions

Intuilab, 4/13/11

"IntuiLab, a global leader in surface computing software applications, today announced support for the revolutionary Microsoft Kinect device across its full line of IntuiFace products and solutions including IntuiFace Presentation and IntuiFace Commerce...Microsoft Kinect brings distant gesture control to interactive solutions. These gesture controls allow users to interact with displayed digital assets from a distance at their own pace and path – for example, browsing through a large quantity of products in a store catalog or manipulating 3D models (such as a mobile phone) – all without having to actually touch the screen..."  -IntuiLab (Take a look at the IntuiLab team- an interactive page!)

EON Reality Launches EON Interact - Interact with 3D Data by Using Your Bare Hands Animation xPress, 3/31/11

Sparkon:  Videos and links related to multi-touch and gesture-based applications

Official Kinect SDK to be Open Source, Josh Blake, Deconstructing the NUI, 4/18/11  

9 This bit of news excited me, but don't get your hopes up. If anyone knows what will happen with the Kinect SDK, please leave a comment.)

"Update 4/18 7:34pm: Mary Jo Foley picks up this story, but the Microsoft spokesperson she talked to denied that the Kinect SDK will be open source. As she notes, Microsoft has pulled 180’s before regarding Kinect. After spokespeople initially were hostile to the idea of Kinect hacking, Xbox executives later embraced the idea that people are using Kinect for non-gaming purposes on the PC. Let’s hope Microsoft stays open to this idea." -Josh Blake

New Channel 9 Video:  Josh Blake on NaturalShow Multi-touch in WPF, Pete Brown, 10rem.net, 4/22/11

Eden: A Professional Multitouch Tool for Constructing Virtual Environments (pdf)

Kenrick Kin, Tom Miller, Bjoern Bollensdorff, Tony DeRose, Bjoern Hartmann, Manees Agrawala (Pixar Online Library)

Flight Race Game on 3DFeel lm3Labs, 4/18/11


JazzMutant Lemure Version 2 : "The only multi-touch and modular controller for sequencers, synthesizers, virtual instruments, vjing and lights, now even better."


Harry van der Veen's Multitouch Blog (NUITEQ)


Stantum "Unlimited Multi-Touch" Latest News

At Immersive Labs, Ads Watch Who Looks at Them Amy Lee, Huffington Post, 4/26/11 

Immersive Labs

Hard Rock Cafe International Using NextWindow Touch Screens:  "Rock Wall Solo displays enhance music lovers' experience in Seattle, Dallas, Detroit and Berlin" 4/12/11 (Full press release pdf)
Music on Touch Screens (NextWindow)

Razorfish: Thoughts on MIX 11 ,James Ashley, Razorfish Blog, 4/20/11  Also see: Razorfish Lab's Prototypes

"The multitouch microscope brings new dimensions into teaching and research. Researchers at the Institute for Molecular Medicine Finland (FIMM) and Multitouch Ltd have created a hand and finger gesture controlled microscope. The method is a combination between two technologies: web-based virtual microscopy and a giant size multitouch display."
"The result is an entirely new way of performing microscopy: by touching a table- or wall-sized screen the user can navigate and zoom within a microscope sample in the same way as in a conventional microscope. Using the touch control it is possible to move from the natural size of the sample to a 1000-fold magnification, at which cells and even subcellular details can be seen."  -Multitouchfi  Also see the Multitouch website.


Big Size Multitouch Display Turned into a Microscope, Microscopy-News, 3/28/11

Mac OX 10.7 Lion: new multi-touch gestures, Dock integration for Expose, Launchpad, Mission Control Appleinsider, 4/14/11


Vectorform App featured in Royal Caribbean's Video Promotion: James Brolin, Dean Cain get hands-on with Vectorform app Alison Weber, Vectorform Blog, 3/3/11


3M Touch Systems's YouTube Channel

Social Mirror 3D Gestural Display, Now Using Kinnect:  SnibbeInteractive

Why bother switching from GUI to NUI? - Asked and answered by Josh Blake; My 2-cents; Stevie B’s Microsoft Research Video; Marco Silva’s NUI-HCI presentation -and links

In Chapter 1 of Natural User Interfaces in .NET,  Josh Blake asks and answers a question posed by many people who have been under the spell of keyboard input and GUI/ WIMP interaction: 


Why bother switching from GUI to NUI?  The answer?  Read Chapter 1 (pdf) of the book - the chapter is free.


Here are a few of my personal reasons:  
1.  I want to buy the next version of the iPad or something like it.
2.  I want to buy a new large-screen Internet HD TV.

3.  I want to buy a Kinect.

4.  I do NOT want to interact with my new TV with a Sony remote.  Too many tiny buttons!

5. I do NOT want to interact with my new TV with a keyboard,  because it reminds me of...work.

6.  Most importantly: 

I want to design apps for the people I care about, and others with similar needs:

    My mom.  

    My grandson.

Moms and dads with kids in tow.
People with special needs and/or health concerns, and the people who care and guide them.
Knowledge sharers and (life-long) learners....

RELATED

"Smart" Interactive Display Research

How Natural User Interfaces are changing Human Computer Interaction

 

View more presentations from Marco Silva

My YouTube Playlist:

Post-WIMP Explorers' Club Videos

"Natural user interfaces, gesture interaction, multi-touch, natural interaction, post WIMP examples and more... "

Interactive Multimedia Technology Post:
Interactive Touch-Screen Technology, Participatory Design, and "Getting It": Revisited

RELATED - and somewhat related   

Moms with Apps

Design for Care

SparkOn

NUI Group

UX Matters      "Why don't usability problems get fixed"

Encyclopedia:  Human Computer Interaction, Interaction Design, User Experience, Information Architecture, Usability and More (Interaction-Design.org)

Josh Blake's Blog: Deconstructing the NUI    Book: Chapter 1 (pdf)  Free!
Blake.NUI

"Blake.NUI is a collection of helpful controls, utilities, and samples useful for multi-touch and NUI development with WPF, Surface, and Silverlight."

OpenKinect Wiki

OpenKinect Community Founder Joshua Blake Comes Up With His Own Version of Win 7 Multitouch on Kinect   (Madhav K, KinectHacks 1/14/11)

Kinect SDK Could Open New Worlds for Motion-Controlled Interfaces (Mike Martin, TechNews World, 2/22/11)

Seth Sandler

Seth Sandler's Blog

Seth Sandler's  Ultimate List of Interactive Tables

Dan Saffer

Bill Buxton

Johnny Chung Lee's Procrastineering Blog
Adam Greenfield's Speedbird Blog

 (This is not an inclusive list.)


GUI to NUI Post-WIMP Manifesto:  TBA

New Microsoft Surface 2.0 and InfoStrat's Surface 2.0 Information Visualization Controls

Microsoft Surface 2.0 was unveiled at CES 2011 a few days ago,  the result of a collaboration between Microsoft and Samsung.  Surface 2.0 is a step up!  The 40 inch 1080p high-definition LCD display no longer requires a projection/camera system, which clears the area below the screen  of bulky hardware.  The best part about Surface 2.0, in my opinion, is that it doesn't have to be used as a table.  It can be configured in a variety of ways, even mounted on walls.   For this reason, it will be useful in a variety of settings and situations.


Below is a quote about Surface 2.0 from Steve Ballmer's recent keynote address at CES 2011 that outlines the new technology that is embedded in the Surface 2.0 display:

"But what's really amazing about this technology, what really makes it magical, is the sensor itself. So, those first-generation Surface PCs needed cameras underneath that would look up to try to see what was going on. But what we have here is called PixelSense. PixelSense is new technology we've invented where there's infrared sensors all across this screen. Every single pixel is actually acting as a camera. The PC, the Surface here, can actually see." -Steve Ballmer:  My Keynote Address at the 2011 International Consumer Electronics Show" (Huffington Post, 1/6/2011)

The good news is that developers have been busy at work to create applications for Surface 2.0. Below is a video demonstration of what the folks at InfoStrat have recently created to support collaborative information visualization activities:







Here's the information about the controls from the Infostratcville YouTube channel:

"This is a sneak preview of a suite of data visualization controls developed by InfoStrat for Microsoft Surface 2.0. The controls will be made available as open source software at no charge on CodePlex.com in the first half of 2011."


"This data visualization control suite provides multi-touch versions of the following controls:
- DeepZoom multi-resolution image control that allows high performance display of very high-resolution imagery
- PowerPoint Viewer which enables slide decks to be arranged and presented using multi-touch
- PivotViewer chart control that allows dynamic sorting and categorization of data
- PhysicsCanvas which provides an infinite, dynamic canvas for viewing and organizing content"




RELATED
Josh Blake's post:  "Microsoft Surface 2.0 Data Visualization Controls by InfoStrat" 
Microsoft Surface Blog: "Microsoft and Samsung Unveil the Next Generation of Surface"


PRESS RELEASE
For Immediate Release

9 a.m. PST

January 6, 2010

InfoStrat Releases Next-Generation Data Visualization Controls for Microsoft Surface 2.0

Washington DC – January 6, 2010 – InfoStrat today announced plans to support Microsoft Surface 2.0 by releasing a control suite that accelerates the development of next-generation multi-touch data visualizations. The controls will be made available as open source software at no charge on CodePlex.com in the first half of 2011.

This data visualization control suite provides multi-touch versions of the following controls:

§ Deep Zoom multi-resolution image control that allows high performance display of very high-resolution imagery

§ PowerPoint Viewer which enables slide decks to be arranged and presented using multi-touch

§ Pivot Viewer chart control that allows dynamic sorting and categorization of data

§ Physics Canvas which provides an infinite, dynamic canvas for viewing and organizing content

Other features of the controls:

§ Works on both Microsoft Surface and Microsoft Windows 7 with touch

§ A single application built with the data visualization framework can support multiple hardware form factors including: horizontal multi-touch tables, tablets, and large format vertical touch screens

§ Innovative object recognition to enable rapid data manipulations (only on Microsoft Surface)

Watch a sneak preview of the control suite on YouTube:http://www.youtube.com/watch?v=lEVtjHlrf4I

InfoStrat is a member of Microsoft’s Technology Adoption Program (TAP) for Microsoft Surface. As a Microsoft Surface 2.0 TAP member, InfoStrat receives early access to hardware and software, allowing InfoStrat to gain expertise and influence the development of the product before it was released to the public.

In 2008, InfoStrat solved the problem of using Bing Maps 3D on Microsoft Surface in a way that performed well and was WPF-friendly. InfoStrat open-sourced the solution as a reusable control for the WPF and Surface community. Since then, the control has received over 120,000 page views and has over 8200 downloads, and has also been featured in many of our own applications. This control, known as InfoStrat.VE, has become one of the most popular controls for building mapping applications on Microsoft Surface: http://bingmapswpf.codeplex.com

“We are proud to be part of the Microsoft Surface development community,” according to Jim Townsend, president of InfoStrat, “and excited about the possibilities of Microsoft’s new version of Surface.”

Microsoft Surface provides a new way to experience and use information and digital content, engaging the senses, improving collaboration and empowering people to interact. Microsoft Surface is at the forefront of developing software and hardware that uses vision-based technology to fundamentally change the way people use computing devices. More information can be found at http://www.surface.com.

Information Strategies ("InfoStrat") is an award-winning Microsoft Gold Certified Partner and a Microsoft Surface Strategic Partner and member of the Technology Adopter Program.

For more information, press only:

Josh Wall, InfoStrat, (202) 364-8822 ext. 202, joshw@infostrat.com

Interested in the OpenNI Initiative? OpenKinect? To learn more, read Josh Blake's Interview of Tamir Berliner of PrimeSense

Interview: Tamir Berliner of PrimeSense

Josh Blake, Deconstructing the NUI, 12/9/10

Josh Blake recently interviewed Tamir Berliner, one of the founders of PrimeSense.  If you haven't heard, Microsoft's Kinect was based on work by PrimeSense, and licensed their technology. PrimeSense provides consumer electronics with natural user interaction capabilities. The good news is that the company recently released open-sourced middleware for natural interaction and depth-camera drivers. It will be interesting to see how this will play in the near future!

In the interview, Tamir discussed a number of topics related to postWIMP technologies.  He also announced the newly created  OpenNI, "an industry-led, not-for-profit organization formed to certify compatibility and interoperability of Natural Interaction (NI) devices, applications, and middleware."   It is good to see this level of support for the cause!

Here is a quote from the interview that I especially liked:

"I believe that till today the devices we’ve been using, made us learn greatly lot about them before we could use them and gain their value. I’m pretty sure everyone who is reading this has got at least 3 remotes sitting on his living room table, and at least once a week needs to help someone use their computer/media center/phone/etc. It’s time for that to change and it’s up to us, the technologists to make this revolution happen, it’s time for the devices to take the step of understanding what we want and making sure we get that, even without asking if it’s a trivial task as opening a door when we approach, closing the lights when we leave the room, even making sure we have hot water to shower with when we return from work or wake up in the morning, depends on what we normally do." -Tamir


RELATED
Here are a couple of videos from the OpenNI website that demonstrate OpenNI-compliant applications:

OpenNI-compliant real time skelton tracking by PrimeSense


OpenNI-compliant real time SceneAnalyzer by PrimeSense

FYI: 

Josh Blake is the author of the Deconstructing the NUI blog. Over the past couple of years, he's explored natural user interfaces and interactions through his work on applications designed for Microsoft Surface and Win7 with Windows Presentation Foundation.

About a month ago, Josh organized OpenKinect, an on-line community to support collaboration among people interested in exploring ways to use Kinect with PCs and other devices.  An example of this effort is the open source code, libfreenect, which includes drivers and libraries for Windows, Linux, and OS X. 

The Natural User Interface Revolution
Josh Blake, 1/5/09


Kinect for Xbox 360: The inside story of Microsft's secret 'Project Natal'  (long, but worth reading) David Rowan, Wired UK, 10/29/10


People of libreenect

OpenNI User Guide (pdf)

Microsoft Surface Light and Physics App for Kids at the Smithsonian

Microsoft Surface at the Smithsonian


The Surface is located in the Smithsonian's Castle,  and is part of "The Wonder of Light: Touch and Learn!" exhibit, which opened on Tuesday, November 9th (2010).  Microsoft donated the Surface unit to the Smithsonian.


Below is  slideshow of the interactive exhibit:



The video below provides a closer look at the applications created by Infostrat for the Smithsonian exhibit:


RELATED
New Interactive Exhibit Opens in Smithsonian's Castle, Bringing Light To Life
Smithsonian News Release, 11/9/20

Josh Blake's post, Microsoft Surface and Magical Object Interaction.

Update on Josh Blake, newly designated Microsoft Surface MVP

Josh Blake is the Tech Lead of the InfoStrat Advance Technology Group in DC.  He has been creating multi-touch applications Microsoft's Surface multi-user table-tops for a while. Recently, his team built a suite of applications designed for use by young children at a museum.  Below is a video demonstration of some of this work. It really looks exciting!


Microsoft Surface and Magical Object Interaction

Josh Blake's blog is called Deconstructing the NUI- for those of you new to this blog, NUI stands for Natural User Interface (also known as Natural User Interaction).  See his post, Microsoft Surface and Magical Object Interaction, for more information!

RELATED
Here is a plug for Josh Blake's book, "Multitouch on Windows"

Book Ordering Information

FYI:  InfoStrat  is hiring  WPF experts as well as Microsoft CRM and Microsoft SharePoint experts.


Microsoft Surface MVPs
Dr. Neil Roodyn
Dennis Vroegop
Rick Barraza
Joshua Blake

For Multi-Touch Techies and the Tech Curious- Touch and Retouch article, with code samples, by Charles Petzold

Thanks to Josh Blake, I came across a good article by Charles Petzold  in the September 2010 issue of MSDN Magazine:
Touch and Response


In this article,  Charles Petzold continues his discussion of the multi-touch support in version 4 of Windows Presentation Foundation (WPF).  I like the introduction to his article:
"Programming is an engineering discipline rather than a science or a branch of mathematics, so rarely does there exist a single correct solution to a problem. Varieties and variations are the norm, and often it’s illuminating to explore these alternatives rather than focus on one particular approach." -Charles Petzold
I also appreciate Petzold's discussion about smooth Z transitions, his thoughtful code samples and related links.

RELATED
Multi-touch Manipulation in WPF 
Charles Petzold (August, 2010, MSDN Magazine, UI Frontiers)
"Just within the past few years, multi-touch has progressed from a futuristic sci-fi film prop to a mainstream UI. Multi-touch displays are now standard on new models of smartphones and tablet computers. Multi-touch is also likely to become ubiquitous on computers in public spaces, such as kiosks or the table computer pioneered by Microsoft Surface."

"The only real uncertainly is the popularity of multi-touch on the conventional desktop computer. Perhaps the greatest impediment is the fatigue known as “gorilla arm” associated with moving fingers on vertical screens for long periods of time. My personal hope is that the power of multi-touch will actually provoke a redesign of the desktop display. I can envision a desktop computer with a display resembling the configuration of a drafting table, and perhaps almost as large." -Charles Petzold

Exploring Multi-Touch Support in Silverlight
Charles Petzold (March 2010, MSDN Magazine, Finger Style)


Comment:
I agree with Charles about the need for a re-design of desktop displays.  I like the drafting table as a form factor.  Here are a couple of my previous posts related to this topic:
Emerging Interactive Technologies, Emerging Interactions, and Emerging Integrated Form Factors (2008)
DigiBoard Multi-Touch Mixed Reality Game; Ideas for future design of a flexible, adjustable multi-touch surface (June 2008)