Musings about NUI, Perceptive Pixel and Microsoft, Rapid Creative Prototyping (Lots of video and links) Revised

It just might be the right time for everyone to brush up on 21st century tech skills. iPads and touch-phones are ubiquitous. Touch-enabled interactive whiteboards and displays are in schools and boardrooms.  With Microsoft's Windows 8 and the news that the company recently acquired Jeff Han's company, Perspective Pixel, I think that there will be good support - and more opportunities- for designers and developers interested in moving from GUI to NUI.    


In the video below, from CES 2012, Jeff Han provides a good overview of where things are moving in the future.  We are in a post-WIMP world and there is a lot of catching up to do!

CES 2012  Perceptive Pixel and the Future of Multitouch (IEEE Spectrum YouTube Channel)



During the video clip, Jeff explains how far things have come during the past few years:

 "Five and 1/2 years ago I had to explain to everybody what multi-touch was and meant. And then, frankly, we've seen some great products from folks like Apple, and really have executed so brilliantly, that everyone really sees what a good implementation can be, and have come to expect it.  I also think though, that the explosion of NUI is less about just multi-touch, but an awareness that finally people have that you don't have to use a keyboard and mouse, you can demand something else beside that.  People are now willing to say, "Oh, this is something I can try, you know, touch is something I can try as my friendlier interface"."

Who wouldn't want to interact with a friendlier interface?  Steve Ballmer doesn't curb his enthusiasm about Windows 8 and Perceptive Pixel.  Jeff Han is happy how designs created in Windows 8 scales for use on screens large and small. He explains how Windows 8 can support collaboration. The Story Board application (7:58) on the large touchscreen display looks interesting.

I continue to be frustrated by the poor usability of many web-based and desk-top applications.  I like my iPad, but only because so many dedicated souls have given some thought to the user experience when creating their apps.  I often meet with disappointment when I encounter interactive displays when I'm out and about during the day.  It is 2012, and it seems that there are a lot of application designers and developers who have never read Don Norman's The Design of Everyday Things!

I enjoy making working prototypes and demo apps, but my skill set is stuck in 2008, the last year I took a graduate-level computer course.  I was thinking about taking a class next semester, something hands-on, creative, and also practical, to move me forward. I can only do so much when I'm in the DIY mode alone in my "lab" at home.  I need to explore new tools, alongside like-minded others.  


There ARE many more tools available to designers and developers than there were just four years ago.  Some of them are available online, free, or for a modest fee.  I was inspired by a link posted by my former HCI professor, Celine Latulipe, to her updated webpage devoted to Rapid Prototyping tools. The resources on her website look like a good place to start for people who are interested in creating applications for the "NUI" era.  (Celine has worked many interesting projects that explore how technology can support new and creative interaction, such as Dance.Draw.) Below is her description of her updated HCI resources:

"New HCI resource to share: I have created a few pages on my web site devoted to Rapid Prototyping tools, books, and methods. These pages contain reviews of various digital tools, including 7 different desktop prototyping apps, and including 8 different iPad apps for wireframing/prototyping. I hope it's useful to others. Feel free to share... and please send me comments and suggestions if you find anything inaccurate, or if you think there is stuff that I should be adding. I will be continuing to update this resource." -http://www.celinelatulipe.com (click on the rapid prototyping link at the top)

IDEAS
Below are just a few of my ideas that I'd like to implement in some way. I can't claim ownership to these ideas- they are mash-ups of what comes to me in my dreams, usually after reading scholarly publications from ACM or IEEE, or attending tech conferences. 

• An interactive timeline, (multi-dimensional, multi-modal, multimedia) for off-the-desktop interaction, collaboration, data/info analysis exploration.  It might be useful for medical researchers, historians, genealogists, or people who are into the "history of ideas".  Big Data folks would love it, too. It would handle data from a variety of sources, including sensor networks. It would be beautiful to use.
• A web-based system of delivering seamless interactive, multi-modal, immersive experiences, across devices, displays, and surfaces. The system would support multi-user, collaborative interaction.  The system would provide an option for tangible interaction.
• A visual/auditory display interface that presents network activity, including potential intrusions, malfunctions, or anything that needs immediate attention that would be likely to be missed under present monitoring methods. 
• Interactive video tools for creation, collaboration, storytelling.  (No bad remote controllers needed.)
• A "wearable" that provides new ways for people to express and communicate creatively, through art, music, dance, with wireless capability. (It can interact with wireless sensor networks.)*
• An public health application designed to provide information useful in understanding and sepsis prevention efforts. This application would utilize the timeline concept describe at the top of this list. This concept could also be useful in analyzing other medical puzzles, such as autism.

Most of these ideas could translate nicely to educational settings, and the focus on natural user interaction and multi-modal i/o aligns with the principles of Universal Design for Learning, something that is important to consider, given the number of "at-risk" learners and young people who have disabilities.

I welcome comments from readers who are working on similar projects, or who know of similar projects.  I also encourage graduate students and researchers who are interested in natural user interfaces to and move forward with an off-the-desktop NUI project.  I hope that my efforts can play a part in helping people make the move from GUI to NUI!  



Below are a few videos of some interesting projects, along with a list of a few references and links.

SMALLab (Multi-modal embodied immersive learning)

PUPPET PARADE: Interactive Kinect Puppets(CineKid 2011)



MEDIA FACADES: When Buildings Start to Twitter

HUMANAQUARIUM (CHI 2012)

 

NANOSCIENCE NRC Cambridge (Nokia's Morph project)






 
Examples: YouTube Playlists
POST WIMP EXPLORERS' CLUB
POST-WIMP EXPLORER'S CLUB II

Web Resources
Celine Latulipe's Rapid Prototyping Resources 
Creative Applications
NUI Group: Natural User Interface Group
OpenFrameworks and Interactive Multimedia: Funky Forest Installation for CineKid
SMALLab Learning

Microsoft Research Natural User Interface 

OpenExhibits: Free multi-touch + multiuser software initiative for museums, education, nonprofits, and students.
OpenSense Wiki 
CINEKID 2012 Website 
Multitouch Systems I Have Known and Loved (Bill Buxton)
Windows 8
Perceptive Pixel
Books
Natural User Interfaces in .NET  WPF 4, Surface2, and Kinect (Josh Blake, Manning Publications)
Chapter 1 pdf (Free)
Brave NUI World: Designing Natural User Interfaces for Touch and Gesture (Daniel Wigdor and Dennis Wixon)
Designing Gestural Interfaces (Dan Saffer)
Posts

Microsoft's"Magic Wall" Tech on Windows 8 Laptops and Tablets by Next Year

Bill Snyder, ReadWrite Web, 7/20/12

I noticed some interesting tools on the Chrome web store - I plan to devote a few more posts to NUI tools in the future.

NUITEQ's Latest Multitouch Showreel: Snowflake Suite

I've been following a number of people that have been working in the area of natural user interfaces and interaction for many years.  An example of this work is NUITEQ, a company lead by Harry van deer Veen.  Below is NUITEQ's most recent show reel of Snowflake Suite, an off-the-shelf multitouch SDK.


Here is the description of the software from the naturaluserinterface YouTube channel:


"NUITEQ's award-winning multitouch software product Snowflake Suite comes off the shelf with 30+ apps, a free SDK to develop your own multitouch software apps and its content is easy to customize. The solution is offers high performance, stability, quality and comes with dedicated support. Apps includes presentation, productivity and creativity tools as well as games. The software can be used in different scenarios such as corporate presentations, exhibitions, entertainment, education, public spaces, consumer electronics, retail and digital signage."

FYI: Tutorials about the user of Snowflake Suite can be found on the naturaluserinterface YouTube channel. 


Harry van der Veen has been sharing his NUI journey journey since 2007 on his Multitouch blog.

iBooks Textbooks for iPads, iTunesU App -and links to info about 1:1 iPad initiatives in public schools

There is a lot of buzz about iPads in education. 


I'm convinced that tablet technology will be adopted by schools at a faster rate than anticipated a year or so ago, now that Apple has launched iBooks Textbooks for iPad and the iTunesU app.


Parents, teachers, students, administrators, and school IT specialists, hold on to your hats!

RELATED
Learning Together: The Evolution of a 1:1 iPad Program
Carolyn Foote, Internet@Schools, 1/1/12
The Swiss-Army Knife of Education
Mr. Hooker, The WIFI--Eanes ISD iPad Pilot Project, 10/3/11
iPads in Education
John Evans, Curator
iBooks Textbooks
iTunesU App

iPad for Art: Sketchbook Express, via the Montlieu Academy of Technology, a public K-5 magnet school that adopted a 1:1 iPad initiative in August, 2011.


COMMENT
I have been planning on devoting more posts in the future about the iPad and related interactive technologies for learning- and for fun.  When I have a chance to write my mega-post, you won't be disappointed!

Christian Bannister's Interactive Multi-touch and Gesture-based Subcycle Project. Use your hands to shape sound, create, and manipulate music. Wow!!

I came across a link to Christian Bannister's Subcycle Labs website when I was taking a look at the Creative Applications Network website.


If you have an interest in music technology and innovative gesture/multitouch applications, you'll appreciate the details that are shared on the Subcycle website.  In the meantime, take the time to watch a few of Christian's videos.  Enjoy!


Blip Shaper Walkthrough

Blip Shaper Walkthrough from christian bannister on Vimeo.
"a) creating percussive patterns with monome b) shaping the individual sounds that make up the patterns with multitouch gestures c) recording touchscreen gestures as automation d) storing, duplicating and navigation patterns e) recording the resulting audio to a dynamic buffer f) manipulating the buffer with a multitouch cut-up approach g) visualizing everything with dual screens"
Subcycle Walkthrough

Subcycle Blip Shaper from christian bannister on Vimeo.

"a) creating percussive patterns with monome b) shaping the individual sounds that make up the patterns with multitouch gestures c) recording touchscreen gestures as automation d) storing, duplicating and navigation patterns e) recording the resulting audio to a dynamic buffer f) manipulating the buffer with a multitouch cut-up approach g) visualizing everything with dual screens"


The following information describing the Blip Shaper is from Christian's Subcycle website/blog:
"For the drum sounds I have Drumaxx running for synthesized sounds and Battery running for sampled sounds.  These are running in parallel so for each voice there is a separate patch running in each VST.  The Parameters are modified with the touchscreen independently but in all cases a single touch gesture on the X-Axis will cross fade between the sampled version of the sound and the synthesized version of the sound.  I love this because I have never seen this before and I can never decide which technique I like better.  The synthesized drums are more malleable and have more interesting parameters to play with but the sampled sounds seem more substantial.  I will post a detailed list of parameters and gestures in the future."   


Christian currently uses Max for Live (ableton), and codes with Processing/Java, using Eclipse for organization.

Autopilot - Subcycle
autopilot - subcycle labs from christian bannister on Vimeo.
"sound visualization, multi-touch interface, break beat performance engine, autopilot, spacialized sound, dsp, max/MSP, fm synthesis, sonic navigation, sound storm visualization, time machine, granular, interactive sound sculpture, joystick array, more at subcycle.org"

SOMEWHAT RELATED 
Community Core Vision
NUI Group

BTW,I'm wondering if Christian or other NUI group members would like to do something with some of my ideas that incorporate interactive music. To learn more, see my post, "It must be  Voronoi: Looking for ideas for my music+art+dance+tech dream...."
...


....

Camera-less Tabletop Computing with Samsung SUR40 for Microsoft® Surface® with PixelSense™

Here is the press release: 
Next Generation of Microsoft Surface Available for Pre-Order in 23 Countries
"Software developers interested in creating solutions for the Samsung SUR40 can get started immediately by visiting the Surface Developer Center. The site provides free and easy access to the Surface 2.0 software developer kit, featuring the Input Simulator, which enables developers to write Surface applications on any Windows 7 machine, as well as other helpful developer-related resources. There are already hundreds of highly skilled Surface software development partners that can be found at http://www.surface.com."

Photos: Samsung SUR40 for Microsoft® Surface®; PopSci, Graham Murdoch

(I've listed Microsoft Surface partners at the end of this post and plan to share more about the latest applications for surface computing in the near future.)


The following product information was taken from the Samsung website:

40" Surface Experience 
"Samsung SUR40 is the new generation of Microsoft® Surface® experience featuring PixelSense™ technology, which gives LCD panels the power to see without the use of cameras. Building from the innovation of the first version of Microsoft® Surface® and Samsung’s leading display technology, it is now possible for people to share, collaborate and explore together using a large, thin display that recognizes fingers, hands and other objects placed on the screen." 


PixelSense™ 
"PixelSense™ allows an LCD display to recognize fingers, hands, and objects placed on the screen, including more than 50 simultaneous touch points. With PixelSense™, pixels in the display see what’s touching the screen and that information is immediately processed and interpreted."


Resolution:             1920 x 1080 
Viewing Angle:       (H/V) 178 / 178° (CR ≥ 10) 
CPU:                     Athlon X2 Dual-Core 245e (2.9GHz) 
Operating System: Windows 7 Professional  x64 


GPU:        AMD HD6750M 
N/B:         AMD RS780E 
S/B:         AMD SB710 


Storage:                  SATA2 320 GB 
Memory:                 DDR3 4 GB
USB:                      4 USB 2.0 
VGA:                     Out HDMI Ethernet: 100 / 1000 
Audio Codec:         Realtek ALC262 Azalia CODEC


Product Dimensions (With Stand):            1,095 x 728 x 707.4 mm 
Product Dimensions (Without Stand):       1,095 x 102.5 x 707.4 mm 
Shipment Dimensions:                            1,214 x 299 x 832 mm 


Product Weight:     35 Kg 
Shipment Weight:  45.4.Kg


RELATED
Samsung
Microsoft Surface
Microsoft Surface "What's New"
Microsoft Surface Partners:
Aftermous.com
AKT
AM Production
Black Marble
ETT
Headcandy
IdentityMine
Information Strategies
Infusion
Inhance Digital
Interknowlogy
Intuilab
nSquared
Object Consulting
Onwijs
Razorfish
Sevensteps
Stimulant
Touchtech
T-Systems
MultiMedia
UID
Vectorform
XFace

Building Interfaces for the New Decade: Conference at the Tech Museum, San Jose, California

Here is information about a day-long conference that I'm sure will be well worth attending:

November 15, 2011 (Tuesday)

Building Interfaces for the New Decade Conference

The Tech Museum

San Jose, California

If you can't make it to San Jose, the conference can be attended "virtually".  You can register for the conference via the EventBright website.


Here is the list of speakers and presentations for the conference:


Tamara Schwarz, Senior Manager of Experience Design, Chabot Space & Science Center
"RFID-enabled Interactive Exhibits, or... What's a warehouse inventory management technology doing in an interactive science center?"
Christopher Stapleton, Creative Venture Catalyst, Simiosys
"The Phydgital InterSpace -The Next Generation of Museum User Experience and Workshop: Experiential Makeover for Learning"
Jim Spadaccini, Director, Ideum; Principal Investigator, Open Exhibits
"Open Exhibits - Community Software for Multi-touch and Motion Recognition"
Rick Ernst, Lead Designer, Ogmento
"How to get People off the Couch and Interact with the World with Augmented Reality Games"
Bob Ketner, Curator, The Tech Test Zone
"The Tech Virtual and The Tech Test Zone Open Platforms"
John MacDuffie Woodburn, Co-Developer, Pixel, and Graduate Student, University of the Arts, PA, & Matt Miller, Co-Developer, Pixel
"Pixel - Using Eye Tracking to Engage and Educate Your Audience"
Ross Smith, Ph.D, Deputy Director, Wearable Computer Laboratory, University of South Australia
"Bringing Fingertip Sculpting Techniques to the Digital Artist"

Below is an excerpt from the conference website:
"Join us on Tuesday, November 15, 2011 for a gathering of the most innovative developers of new interactive technologies at The Tech Museum's Test Zone 2011...Connect with entrepreneurs, developers, and researchers working on cutting-edge interfaces such as:
Augmented Reality
Context Aware 
Deformable Surfaces
Eye Tracking
Feedback
Gestural Interfaces
Gigapixel Images
Motion Sensing
Multi-touch
Natural User Interfaces
RFID
Smart Museum Systems
Tablet Computers
Ubiquitous Media 


Find out how museums and other public-space designers can use these technologies to enhance visitor learning and experience. Experience the interactives firsthand at the opening of The Tech Test Zone Gallery, a new gallery dedicated to temporary exhibitions of prototypes from corporate, academic, and nonprofit partners."

RELATED

About the Tech Museum:

"The Tech Museum is a hands-on technology and science museum for people of all ages and backgrounds. The museum-located in the Capital of Silicon Valley -is a non-profit learning resource established to engage people in exploring and experiencing technologies affecting their lives. Through programs such as The Tech Challenge presented by Cisco, our annual team design competition for youth, and internationally renowned programs such as The Tech Awards presented by Applied Materials, Inc., The Tech Museum celebrates the present and encourages the development of innovative ideas for a more promising future." -Tech Museum YouTube Channel

Hack the Future Day at the Tech Museum

T(ether): Next-Gen Data Gloves + iPad - just what I need for the colder weather! (quick post)

"T(ether) is a novel spatial aware display that supports intuitive interaction with volumetric data. The display acts as a window affording users a perspective view of three- dimensional data through tracking of head position and orientation. T(ether) creates a 1:1 mapping between real and virtual coordinate space allowing immersive exploration of the joint domain. Our system creates a shared workspace in which co-located or remote users can collaborate in both the real and virtual worlds. The system allows input through capacitive touch on the display and a motion-tracked glove. When placed behind the display, the user’s hand extends into the virtual world, enabling the user to interact with objects directly." -Vimeo

For more pictures and information, see the following post on the Creative Applications Network website: 
T(ether) [Cinder]
Filip Visnjic

Stantum's Multi-Modal, Multi-Touch Technology: Looks Great for 21st Century Learners, Supports Creative, Collaborative Knowledge-Sharing

Not long a go I had the opportunity to participate in a phone conference with Guillaume Largillier, of Stantum,  to learn more about the company's work in developing multi-touch technology, and how this technology can support a wide range of uses.  

What is exciting is that the tablet technology was designed from the beginning with content-creation in mind, unlike the iPad and similar Android-based tablets.  Although Stantum's tablet technology supports  passive "entertainment" and Internet content viewing, it offers so much more.  

I agree with Guillaume that this technology will be ideal for use in education. It is ideal for 21st Century teaching and learning.   In the very near future,  I'm sure we'll see fewer students lugging heavy  backpacks as this technology transforms textbooks to multi-modal, interactive experience platforms!

Stantum' IVSM multi-touch technology (IVSM = Interpolated Voltage Sensing Matrix) enables 10 finger touches with stylus input.  The goal is to enable content creation via tablets, for educational and office settings. This technology provides reliable palm rejection and contact discrimination,  a useful feature for students or anyone else who might have the need for a versatile tablet.  A multi-touch tablet is demoed in the video below:
 



Intel-Quanta Redvale Tablet featuring Stantum's Digital Resistive Technology
RELATED
Add Ambidexterity and Multi-modality to your Touch-enabled Devices
Guillaume Largillier, Electronic Design, 7/14/11
 "In South Korea, the government recently confirmed its plan to replace all printed textbooks nationwide by electronic counterparts by 2015, a move that will benefit from now available multi-touch technologies with precision and stylus input that enable handwriting recognition, making them especially suitable for products sold into Asian markets." 
About Guillaume Largillier:
"Guillaume Largillier’s background with the company includes seven years of entrepreneurship – co-founding JazzMutant and serving as CEO as it evolved into Stantum. Largillier has a broad knowledge of human-machine interface technologies, with strong expertise in multi-modal user interfaces. He has participated in major industry conferences (including SIGGRAPH, Microsoft Summit, SID Mobile Display Conference, Microsoft TechDays, and Interactive Displays Conference). Largillier earned an MA degree in audio-visual studies at the University of Bordeaux." -Stantum


Post: Multi-touch Update from Stantum
2010 Demo: MeeGo Tablet on Intel Moorestown 
Stantum's TouchPoints Newsletter (July 2011)
Stantum Whitepapers:
How to Evaluate Multi-Touch While Standing in a Store (pdf) - a great source of information.
Jim Meador, Pascal Auriel, Gauthier Chastan, Stantum
Specifying and Characterizing Tactile Performances for Multi-touch Panels: Toward a User-Centric Metrology (pdf) - outlines some important points!
Guillaume Largillier, Pascal Joguet, Cyril Recoquillon, Pascal Auriel, Axel Balley, Jim Meador, Julien Olivier, Gauthier Chastan
Stantum: info@stantum.com

MULTITOUCH NEWS: NUITEQ's interactive display at a business park; Innovation Now's multitouch anatomy application

Multitouch Display for Business Science Park Aurorum, by NUITEQ

"NUITEQ developed a customized multi-touch software solution for Corporate Reception / Lounge Areas for Business Science Park Aurorum in Luleå, based on the award-winning Snowflake Suite framework. In addition to the software, NUITEQ delivered and installed a 32 touch points multitouch dreaMTouch LCD from Germany based Elektrosil."-NUITEQ

Multitouch Solution for Corporate Reception/Lounge areas based on Snowflake Framework

Harry van der Veen, Multitouch Blog, 8/19/11

Multitouch Anatomy by Innovation Now

"Virtual anatomy surface computer in the shape of autopsy table that show and enable the user to use hand gestures to do the anatomy process virtually." -Innovation Now

Homam Alghorani, CEO, Innovation Now Sdn Bhd

Usability of Touchable-Moveable Multimedia Content. Part I (carousels and more)

I've been revisiting the topic of interaction design/usability for "touchable/moveable" multimedia content for the past couple of weeks.    I recently got an iPad2, so I'm in the "app" exploration phase -  a new world, since I never had an iPhone, iPod touch, or iPad "1".  


I plan to share  experiences, thoughts, opinions, and suggestions related to this topic during the upcoming months.   I welcome input from IMT readers!


Getting back to my iPad2....In addition to using the device to run apps, like most people, I also use it to visit websites.  Many sites now appear to be optimized for mobile devices, which is great for my Android-based smartphone, but for my iPad, not so much.   I thought by now we'd have more touchy-feely content to play with.


I've noticed that  basic features that work OK with touch interaction, such as the carousel, 
seem to miss the target, as discussed in the following article:



5 Big Usability Designers Make on Carousels
Anthony, UX Movement, 8/2/11


Although the above article was written from a "click" rather than "touch" perspective, many of the comments Anthony brings up ring true for websites and apps that are accessed through tablets and other touch enable screens.



I'm disappointed.  Here are a few of my grumbles:

• Many designers/developers have been slow to look at the increasing number of people who are accessing websites and applications via touch-enabled screens - of any size.   
• Many designers/developers have been slow to catch on that a website or app might be viewed/used by more than one person at a time.  Websites for house-hunting, home furnishings, vacation/travel, and education are a few examples that come to mind.
• Too many "multimedia" apps/websites are flat. Many still have a power-pointy feel.
• Websites optimized for mobile devices often leave important features out.
• Designers/developers sometimes don't seem to think about the various scenarios in which their applications/websites might be used - while balancing a baby on one hip, preparing a meal,  working out,  walking the dog, trotting around the mall with a friends/kids/spouse,  eating, waiting at a stop light, etc.

Too many iPad apps are flat and power-pointy. Too many apps don't take full advantage multi-touch features.  Very few apps allow for efficient interaction between two people on the screen.  




SOMEWHAT RELATED
Carousel Design Patterns   I will add more related links in the future - not limited to carousels : )
Carousel Interaction #21 (Mike Highfield's Rapid eLearning Blog
Carousel (Interaction Patterns Wiki)
Carousel (Welie)
Mobile Implementation of a Web 3D Carousel with Touch Input
Bjorkskog, C., Jacucci, G., Lorentin, B., Gamberinit, L.



Photo Credit:  Ohad

OPEN SOURCE MULTI-TOUCH SOFTWARE FOR YOUNG PEOPLE WITH AUTISM SPECTRUM DISORDERS- Featured in the WSJ! (Video and links)

Well before the iPad entered the market, researchers such as Dr. Juan Hourcade at the University of Iowa, began to develop multi-touch software to  support young people with autism spectrum disorders and other disabilities.  I've been following this research for a while, and was happy to discover that Dr. Hourcade's work was  recently featured in a recent Wall Street Journal article and video.  


If you are a parent or work with a young person with special needs, you might be interested in learning more about Windows-7 based touch-screen software.  The software shown in the video below, as well as related resources, are available for free on the Open Autism Software website.




RELATED

Article:

Playing on a Tablet as Therapy

Shalini Ramachandran, Wall Street Journal, 7/26/11

Previous Post: Interactive Multi-Touch for Children with Autism Spectrum Disorders:  Research and Apps by Juan Pablo Hourcade, Thomas Hanson, and Natasha Bullock-Rest, University of Iowa 4/26/10

Interaction Design and Children Remixed, CHI 2011

CHI 2011 Child Computer Interaction: 2nd Workshop on UI Technologies and Impact on Educational Pedagogy

Why is this important to me?   

I work with young people with a range of disabilities, including autism, as a school psychologist.  One of my schools is a program that is technology-rich.  Although we don't have iPads, there is a SMARTBoard in every classroom and in other rooms around the school. We have a Wii,  a few multi-touch all-in-one PC's,  and recently received two Dell multi-touch tablet PCs. 

I noticed that Dr. Hourcade is looking for people who'd like to create similar apps for the iPad.  I'd be interested in knowing anyone who might be working on this.  I'd like to help in some way, but I don't have a lot of free time to do this on my own. 

I have a new iPad2, and after playing around with a ton of apps, I was inspired to downloaded XCode4 and think about what I could create. I'm a Mac person at heart, but I learned to program within a Windows environment (C#).    I need a way to maintain my programming skills and gradually learn a few new ones. This sort of project would be useful to me in my "day job", too!

SOMEWHAT RELATED
3D Interactive Heart, by 22Miles, on a Dell Latitude Xt2

Multi-touch Update from Stantum

The people at Stantum have been working hard to improve multi-touch technology, focusing on smaller tablet-sized systems.  Stantum is a company I've been following for several years, from the time it was known as Jazz Mutant.  I have been impressed by Stantum's focus on the needs of people as well as the company's careful attention to important details.


I'm pleased to see that the company has an idea of how its multi-modal technology can support multi-touch in education:   "Ambidexterity and multi-modality are the two pillars of Stantum's core project – making the use of touch-enabled devices more creative and productive. Amongst others, there is one field of application where we truly see a soaring need for ambidexterity and multi-modality – augmented textbooks." -Guillaume Largillier


At the Society for Information Display's Display Week exhibition this past May, Stantum introduced a new palm rejection feature for its Interpolated Voltage Sensitivity technology. This technology provides users with a more natural way to interact with the interface and application content on tablets.   The technology supports Android's multi-touch framework and is also Windows 7 certified.  The palm rejection feature will be a welcome improvement for future multi-touch applications designed for education settings, where it is likely that  more than one hand - or person, might be interacting with content on the screen at the same time.


Below are two videos that provide a glimpse of Stantum's innovations:




Stantum's technology can enable ten simultaneous touches, is highly responsive, and supports high-resolution content. According to a May press release, "Palm rejection is available as an API (application programming interface) to Windows and Android operating systems on x86 and ARM platforms. IVSM touch modules are offered to OEMs through the company’s Qualified Manufacturers Partners, comprising tier-one touch-screen manufacturers with high-volume production capabilities. More information is available at info@stantum.com"


RELATED
Stantum's TouchPoints Newsletter (July 2011)


Stantum Whitepapers:
How to Evaluate Multi-Touch While Standing in a Store (pdf) - a great source of information.
Jim Meador, Pascal Auriel, Gauthier Chastan, Stantum
Specifying and Characterizing Tactile Performances for Multi-touch Panels: Toward a User-Centric Metrology (pdf) - outlines some important points!
Guillaume Largillier, Pascal Joguet, Cyril Recoquillon, Pascal Auriel, Axel Balley, Jim Meador, Julien Olivier, Gauthier Chastan

Summer Break: Music Apps, Multimedia, Kinect, My New iPad2, Tech-reading, Google+, Dancing...

I'm on summer break, which for me, means that I spend an increased amount playing/creating music and doing all of the other fun stuff I don't have much time for during the school year. I'm still exploring what I can do with my new iPad2 - there are so many music apps!  My favorite at this moment is Garage Band. It keeps me engaged for hours, and I can take it with me anywhere I go.  I'm also exploring iPad apps for education and students with special needs, since many of the young people I work with have autism spectrum disorders.  They all really love music.


Today, I came across turntable.fm, a "social-djing" website, from a link shared by Dimitri Diakopoulos.  I think it would be fun to play with.

Turntable.FM, The Fastest-Growing Music Service You're Not Using
William Fenton, PC Magazine, 6/23/11
Social DJing with Turntable.FM
Andrew Mager, 5/28/11

I'm still plowing through technology journals and zines from previous months - I had to skip over my stack to read the cover article of the most recent Communications of the ACM:

Algorithmic Composition: Computational Thinking in Music

Michael Edwards, University of Edinburgh, 2011

I LOVE the design of this cover. It would make for a nice interactive interface for an iPad music app. Or a larger touch-screen display. Or even a SMARTBoard! (BTW, My first computer-related course was Computer Music Technology, in 2003. My undergraduate honors research (psychology), years ago, focused on constructive cognition and music recognition/memory. This topic is dear to my heart.)

I've spent some quality time with my first grand-baby this summer.  Although his "screen time" is limited, given his age of 7 1/2 months, he enjoys playing with music on my iPad.  He likes the drums found in the iPad GarageBand application.  Here he is playing with NodeBeat, an app created by Seth Sandler and Justin Windle:

Most of his time is spent off-screen:





Over the last few months, there has been a surge of interactive touch-enabled apps for education, including some for young people with special needs.  This will be the topic of a few of my future posts.

CHI 2011 Workshop Program and Related Links: UI Technologies and Impact on Educational Pedagogy, Related Child-Computer Interaction Papers and Courses

WORKSHOP PROGRAM
CHI 2011 will be held at the Vancouver Convention Centre in Vancouver. The UI Technologies and Pedagogy workshop will be held on Saturday, May 7th,  in the West Building, Level 1, Room 119/120.  PDF versions of the following presentations can be found on the workshop's website.


9:00am – 09:05am Introduction
Edward Tse and Johannes Schöning
9:05am - 10:05am Paper Session I (8 min presentation, 4 min Discussion)
Chair: Yvonne Rogers
Evaluating Pervasive Classrooms
• Son Do-Lenh et al.: Classroom-experience evaluation: An ecological approach to evaluating pervasive technologies in classrooms 
• Kathryn Rounding et al.: Evaluating Interfaces with Children 
• Dan Morris et al.: Using Machine Learning to Scaffold Artistic Creativity in Education
• Tamara Polajnar et al.: Enabling Picture-based Querying and Learning with the JuSe Interface
• Hamed Alavi et al.: Lantern 2.0: A Pervasive Scripting Tool
•Quincy Brown et al.: Mobile Natives: Unlocking the Potential of Educational Technology
10:15am - 11:00am  Coffee Break and Preparation time for short talks
11:00am - 12:10pm  Paper Session II (8min presentation, 4min Discussion)
Chair: Max Mühlhäuser
Collaboration in Math: Fears, Myths, and Insights.
• Nicola Yull et al.: Pass the iPad: Comparing collaboration on paper and screen
• Veronika Irvine et al.: Math Education: A Creative Approach
• Group talk: Math and Embodied Agents
a.Kristina Richter et al.: Bridging Diagnosis and Learning for Mathematics Education in Classroom Setting
b.Lisa Anthony et al.: Handwriting Interaction for Math Tutors: Lessons for HCI in Education
c.Andrew Jensen et al.: Using Embodied Pedagogical Agents and Direct Instruction to Augment Learning for Young Children with Special Needs
• Group talk: Dispelling Myths About the Next Generation Classroom
a. Alex Thayer et al.: The Myth of the Paperless School: Replacing Printed Texts with E-readers
b.Sharon Oviatt: Designing Digital Tools for Thinking, Adaptive Learning and Cognitive Evolution
c.Alexandra Dunn et al.: Designing Classroom Technology to Meet the Needs of All
• Group talk: Games, Wearables and Fun Learning.
a.Christiane Moser et al.: Child-Centered Game Development
b.Lizbeth Escobedo et al.: Blue’s Clues: An Augmented Reality Positioning System
c.Si Jung Kim et al.: Wearable Story: A Story Telling Jacket for Young Children to Improve Their Independent Physical and Learning Activities
12:10pm - 1:30pm   Lunch
1:30pm - 2:30pm   Paper Session III (8 min presentation, 4 min Discussion)
Chair: Richard Beckwith
Tangibles and Tabletops
• Izdihar Jamil et al.:    Communication Patterns in Collaborative Peer Learning around Interactive Table
• Aura Pon et al.:   Graspable Music and Vuzik: Music Learning and Creativity using an Interactive Surface
• Ahmed Kharrufa et al.: Learning at interactive surfaces and designing for reflection
• Cristina Sylla et al.: TOK – a Tangible Interface for Storytelling
• Group talk: Tables and Tangibles
a.Robert Sheehan:    Constructionism, Programming and Touch Interfaces
b.Orit Shaer et al.:     Making the Invisible Tangible: Learning Biological Engineering in Kindergarten
c.Sebastien Kubicki: Evaluation of an interactive table with tangible objects: Application with children in a classroom
d.Cristina Emilia Costa et al.:  I-Theatre: developing narratives skills in kindergarten children
e.Wooi Boon Goh et al.:  Potential Challenges in Collaborative Game Design for Inclusive Settings
2:30pm - 3:00pm       Coffee Break
3:00pm - 4:00pm       Breakout Discussion (What are future challenges? Next years workshop?)
4:00pm - 4:45pm   Closing Keynote by Allison Druin: Mining the Imagination from Time Travel to Anti-gravity: The Future of Child-Computer Interaction
"If you’ve ever sat with a child imagining the future of new technologies, you will undoubtedly hear about “anti-gravity machines,” “peace-technologies that work by tickling people,” “backpacks of the future with ice cream makers,” and “time-travel-mobile-phones that when you call someone it takes you back in time.” The question is how can these child-like imagined technologies lead us to real answers for the future?


In my talk I will propose that “data mining” is not enough, that “mining the imagination” is critical to understanding the future. I will describe the “iChild” and the interactive, independent, international, imaginative child we need to consider designing for in the future. If we look for real change, we need to listen to children, not because they know more, but because they can help us question, explore, and push us in true directions for change."


Allison Druin is an Associate Dean for Research in the iSchool at the University of Maryland and is Director of the Human-Computer Interaction Lab. For almost 14 years, she has led an intergenerational research team, where children, ages 7-11 years old partner with an interdisciplinary group of adult researchers to develop new technologies for children. With this team, she has helped to developed new digital library and storytelling tools with such partners as the U.S. National Park Service, Sesame Workshop, Nickelodeon, Nokia, UNICEF, and many others. In 2010, she received with Ben Bederson the SIGCHI Social Impact Award for her work leading the International Children’s Digital Library (www.childrenslibrary.org).
4:45pm - 5:00pm Closing Remarks (All)
7:00pm Optional Workshop Dinner


RELATED CHILD-COMPUTER INTERACTION SESSIONS AT CHI 2011
CHI 2011 will be offering a variety of opportunities for people interested in interaction design and technologies for children.  Detailed information regarding the researchers/ presenters can be found on the IDC-SIG News/Events page.


CHI2011 COURSE - Evaluating Children's Interactive Technology
Monday 9th May 2011 - 16:00
Janet C Read - University of Central Lancashire
Panos Markopoulos - Technical University of Eindhoven
CHI2011 COURSE - New Methods for Designing for and with the iChild: Strategies for Today’s Mobile, Social, and Internet Technologies
Thursday 12th May 2011 - 09:00
Allison Druin - University of Maryland
Jerry Fails - Montclair State University
Mona Leigh Gubler - University of Maryland
CHI2011 Paper Session - Engaging Youth  Thursday 12th May 2011 - 11:00
Exploratory Evaluations of a Computer Game Supporting Cognitive Behavioural Therapy for Adolescents
In the Mood: Engaging Teenagers in Psychotherapy Using Mobile Phones
Breaking Boundaries: Strategies for Mentoring through Textile Computing Workshops
Brick by Brick: Iterating Interventions to Bridge the Achievement Gap with Virtual Peers
CHI2011 - Other IDC Related Papers
Paper Session: Art, Music and  Movement: Monday 9th May 2011
MOGCLASS: Evaluation of a Collaborative System of Mobile Devices for Classroom Music Education of Young Children
MoBoogie: Creative Expression Through Whole Body Musical Interaction
Paper Session: Low-cost ICT4D Tuesday 10th May 2011
Utilizing Multimedia Capabilities of Mobile Phones to Support Teaching in Schools in Rural Panama
Infrastructures for low-cost laptop use in Mexican schools
Paper Session: Storytelling and Perceptual Crossing Wednesday 11th May 2011
ShadowStory: Creative and Collaborative Digital Storytelling Inspired by Cultural Heritage
Paper Session: Learning
Practical, Appropriate, Empirically-Validated Guidelines for Designing Educational Games

CHI2011 Paper Session - Families Tuesday 10th May 2011 - 11:00

Learning Patterns of Pick-ups and Drop-offs to Support Busy Family Coordination

Mediated Parent-Child Contact in Work-Separated Families

Hello, is Grandma there? Let’s Read! StoryVisit: Family Video Chat and Connected E-Books

Family Portals: Connecting Families Through A Multifamily Media Space

If these topics interest you, consider making plans to attend IDC 2011!
IDC 2011:  The 10th International Conference on Interaction Design and Children
June 19-23,  University of Michigan (My alma mater!)