I’ve been following their work all summer, and an really excited that they are finally presenting it at ISEA 2012 (International Symposium on Electronic Art) this week!

Nettrice Gaskins and Laurie Marion (who are variously artists, designers and digital media researchers) spent part of the summer running a workshop with students in Albuquerque, NM, creating an interactive AR mural along the train corridor at the Wells Park Community Center.  (Nettrice and Laurie are students in GT’s excellent Digital Media program).

The big idea was to have the kids work on a design project that is culturally relevant to them (see http://csdt.rip.edu), to situate their learning of some underlying STEM concepts.

The focus of the project was creating an AR-interactive mural, using our Argon browser. They documented the project and process on their AROS blog, and if you are interested in the process of using AR and outdoor art to working with kids, I highly recommend checking out the whole blog. The final mural and AR overlay can be seen in this recent blog post.   You can also see pictures documenting the project on Pinterest.

Nettrice is presenting the project this week, and sent me this picture of the brochure for the Wells Park Rail Runner Mural Project that created by a group of local people in Albuquerque:

Laurie designed an insert for the brochure, to explain how to load the AR content:

As an AR researcher, I’m always excited when people do cool things with the tools we build; the idea that our browser could be used for something so important as engaging kids with art and technology is awesome. Of course, it’s also a reminder to us researchers not to get hung on the technology too much:  any technologist reading this might be “horrified” to see human-sized “AR markers” as part of a mural on the side of a building (and I will say that the upcoming version of Argon will have Qualcomm’s Vuforia image-based tracking software in it, for those who just can’t bear the thought of it).  But, for me, the idea that the technology is approachable enough that kids could build with it is what’s important, and is motivating us to make the tech better, and more approachable and usable for these sorts of projects!

Great job, Nettrice and Laurie;  I hope ISEA goes well!

I’ve been contemplating Project Glass recently, imagining what the user experience of wearing one might be.  While I’m increasingly looking forward to trying one out, one problem that keeps coming to mind is the social acceptability of walking around with a camera pointing at everything I look at.   The more I ponder it, the more concerned I am that if Google isn’t careful, Project Glass may crash and burn, and set back public acceptance of head-worn displays quite a bit, because of that darn camera.

It may seem counter-intuitive to assert that its camera could be the downfall of Google’s Project Glass.  After all, much of what gets shown off during Glass demos is it’s ability to grab pictures and movies with little to no effort.  Movies while swinging a kid in the backyard, or while falling through the sky.  Pictures of talk show hosts and beautiful people at fashion shows.  Much of the devices ability to sense a wearer’s context and proactively provide serendipitous information and opportunistic interaction depends on the camera.

Theres a huge problem with the trajectory of their current design, however:  the people being looked at can’t tell if the camera is taking pictures.  Privacy and security aren’t about me (the wearer), they’re about you (the targets). As a huge enthusiast for mobile augmented reality, and  someone who’s looking forward to trying one out, I’m uncomfortable that Google seems not to”get” this or be taking it seriously.

During the demos, you see Sergey and others make a cute hand gesture on the arm of the Glass to trigger a picture;  some might naively say that’s enough to signal to someone else that a picture has been taken.  But, it’s obviously not:  while in that one instance we know a picture is being taken,  how do you/me/we (the targets, again) know that this is the only time a picture is being taken?  And there lies the problem.

Early iPhone developers who wanted to leverage the camera on an iOS device will remember Apples tight control over access to the camera.  For years, it was impossible to grab images from the camera without using the built in image capture widget; and even then, it was impossible to capture images without the loud faux-camera-shutter sound!  When we created Arf, the iOS virtual pet years ago, we couldn’t release it because of the hackery involved with accessing the camera in real time.  One of the reasons apple was so careful with camera access was a desire to make both the user of the phone and those near the phone comfortable knowing if and when pictures were taken.

And while the restrictions on developers have eased, Apples careful approach was largely successful.  As a society, we’ve adapted to camera phones.  We know that if someone is holding up their phone, they might be taking pictures.  Locker rooms at athletic clubs now regularly ban camera phones, for obvious reasons.  Some high security government agencies entirely disallow phones on the premises.

But (practically speaking) it’s extremely hard to take pictures with a phone without your target knowing.  Try facing someone you don’t know, holding up your phone and pointing it them;  it’s easy for them to see you do this.  Now imagine pointing it at a couple with their arms around each other on a park bench;  at children you don’t know on a playground;  at an attractive person of the appropriate gender in a coffee shop or bar.  It’s easy to imagine, shall we say, a “diversity” of reactions.

Now consider what will happen when people realize (or imagine) that folks wearing Glass are pointing cameras at them, all the time, every minute they are nearby, and could be recording everything they see and hear.

Consider the bizzarre recent altercation in a Paris McDonalds between Steve Mann and the employees.  We may never know what really happened, if Mann was at fault (perhaps provoking the employees in some way) or if the employees were at fault (over-reacting to something that made them uncomfortable).  What is clear is that during this episode Mann was taking pictures of the whole thing, and by extension, likely to be taking picture of anyone and anything he looks at, all the time.  While he hid the employees faces in the pictures he posted, the point (that is largely being ignored by the media and wearable enthusiasts) is that he was taking pictures continuously, and  could later choose a set to post, a set that supported his point.  It’s not hard to imagine what will happen when thousands of people are “wearing” and able to mine the images/video/audio they’ve been capturing as they go about their day.

A first step toward heading off possible problems is obvious:  Google really needs to take this seriously.  At a minimum, they need to ensure that any time the camera or microphone is operational, there is visible (and perhaps auditory) feedback available to those near the camera. There should be one kind of feedback when the camera or mic is operational at all (even if just in preview mode), different feedback when an image is captured and passed to an app on the device or stored in the camera roll, and perhaps a third sort of feedback when video or audio clip is being captured. And this really has to be done at the hardware level, not at the OS level (since the OS will likely be open like Android, to support research) and certainly not left up to app developers.

Doing this right is a serious challenge.  A simple LED on the front could easily be covered with tape;  so  something might need to be integrated into the camera lens assembly so it can’t be covered without destroying the cameras ability to capture images.

Whatever the solution, it would be a shame if the camera, the thing that gives Glass much of its power and appeal, might also be its undoing.

So, in response to a short article by the ACM CHI Program chair titled “A Cry for More Tech at CHI!.”, I wrote a blog post (titled “What? You want us to be your Dancing Bears?“) saying why I thought this request was off the mark.

After a lively discussion in the comments, parts of my article and some of the followup commentary, were posted in ACM Interactions last month.  Perhaps inevitably, a number of us were not satisfied with the outcome of that post.

So, we’d like to collect some more comments and reflections, and author a longer, more well thought out discussion in an upcoming issue.

So, please, if you have thoughts about how (or how not) to include interactive demos at CHI, chime in!

Over the years, many have dreamed about using mobile technology to create a kind of “augmented reality translation system” that translates whatever you are looking at into a language you can understand, and displays it in place.  Back in 2008, Intel’s Paul Otellini demonstrated a prototype Chinese-to-English translator as part of his CES Keynote.   More recently, Word Lens for the iPhone does a beautiful job of cleanly translating text in place (supporting english, spanish and french), but the translation (in my experience) tends to be a bit spotty.

Now, Microsoft has demonstrated a far more comprehensive approach as part of Bing Translate for Windows Mobile, and it looks amazing.

[msn c0b91059-9ce0-477f-a391-53fbb53c061e 534 300]

I’ll be trying it out on one of our Nokia 800′s, and it along would probably make it worth carrying the phone on international trips, even though I’m still firmly an iPhone user.  If they keep putting amazing things like this out, and decide to add in better non-MS-centric mail and calendar to WP8, and perhaps that will change.

Ok, I know I’m either going to love or hate a blog post about an internal research project on mobile AR, when the first line is

It all started with Snow Crash.

Michael Abrash, game and tech industry veteran, and now at Valve, recently wrote a blog post about his journey to Valve and the current internal research project on what he calls “wearable computing.”   For the rest of us, though, the description of what they are doing sounds exactly what researchers like me would call fully immersive AR. (Ironically, the folks at Google are doing something that the research community would traditionally call wearable computing, and folks want to call it AR, but that’s a different conversation.)

Suffice to say, I loved this post.  Now, he doesn’t really provide much information on what they are doing, so it’s not that it’s the details that are what’s compelling.  But, I find myself really excited about the idea of a top tier game company doing an internal project that is focused on what is arguably the distant future of AR gaming.

Sure, companies like Sony and Nintendo and Microsoft have built AR games for recently, and the core feature of the PS Vita, 3DS and Kinect is the ability to do augmented reality style games.  But, all of these products and projects are focused on the near future, on games and experiences that will be possible with this hardware. In my research group, especially our Augmented Reality Game Studio, we’ve been working on similar kinds of games for many years, and so I obviously see the value in trying to figure out how to create compelling AR game experiences with current hardware and software (heck, we will hopefully be releasing one of our games, Nerdherder, into the app stores soon!)

But, lately I’ve been thinking that it’s time to refocus on the future.  When we started the AR Game Studio, it still wasn’t easy to build AR games on mobile devices, and we were lucky to team up with Qualcomm to see what would be possible with what has since become Vuforia, their free AR SDK (that is probably one of the best solutions for doing AR on mobiles right now).  Now that Vuforia is out there, and is available for the Unity game engine as a free plugin, everyone can get in on the action!  If you can build games in Unity, you can build AR games!    I will keep teaching AR game design classes, and working with students in my lab to build these games.  And my students and I will keep investigating how to create compelling experiences of this sort (with the advantage that we can study what others make, too, not just what we’ve built).

But, my first love is working with technology and concepts on the cutting edge that are (for all intents and purposes) impossible on a large scale, rather than studying and understanding what people can do right now.

Which brings me back to Michael Abrash.  I have no idea what they are really doing.  But, the thing I’ve learned about game design over the past half dozen years, building games in my lab and at my company, is that game design is different than research, even if your research needs you to build a game.  Unlike researchers or product engineers, game designers are trained and focused on creating compelling experiences.  Things only matter insofar as they are perceived by the player and impact the experience of the player.  If the experience sucks, the experience sucks:  there is no “oh, it only works if you do this” or “imagine what it would be like if this was better or that was better”.  There is no making it work well enough for the paper, or the video, or the evaluation.  There is only the experience.   As Miyagi said in The Karate Kid:

Miyagi: Now, ready?
Daniel: Yeah, I guess so.
Miyagi: Daniel-san, must talk. Walk on road, hm? Walk left side, safe. Walk right side, safe. Walk middle, sooner or later, (makes squish gesture) get squish just like grape. Here, karate, same thing. Either you karate do “yes”, or karate do “no”. You karate do “guess so”, (makes squish gesture) just like grape. Understand?

AR (or wearable computing, as Michael calls it) will not really be possible any time soon.  To safely and unobtrusively integrate content with your view of the world around you, you must be able to achieve the kind of integration we see in live sports on TV, and that simply isn’t going to be feasible soon.  But if you want to work towards that dream, you need to decide:  am I building something to take me one step in that direction (e.g., my games in the AR Game Studio, or release of the Argon AR-enabled web browser, Google’s Project Glass) or are we going to just try to create a compelling experience of what it might be like when we get there.

I don’t know if the later is what Valve really wants to do, but I think a game studio that is saying up front

To be clear, this is R&D – it doesn’t in any way involve a product at this point, and won’t for a long while, if ever – so please, no rumors about Steam glasses being announced at E3.

is likely to do something interesting and exciting and different than the other projects you see around the web.

Too bad Valve isn’t in Atlanta.

In light of all the talk of Google’s project glass, there are a bunch of folks suggesting that the other displays on the market can already do what Project Glass is trying to do.  And, I will agree, some of these displays are getting closer.

Brother’s Airscouter (apparently due out this summer in Japan) is indeed a see-through virtual retinal display, and is “relatively small”.

YouTube Preview Image

But, it’s a LOT bigger and bulkier, and it’s “just the display.”   No camera.  No computer.  No sensors.  So, in practice, it would be much bigger, and a lot uglier.  As a researcher, of course, I’m totally fine with this, since it will let us do the research we want with the platforms we want (from desktop to laptop to tablet to handheld) and not be bound to a specific OS or set of sensors.  But, for practical use, that’s a huge problem.  So, we’ll likely be buying some (there’s nothing else quite like it coming out), just as we’ll be buying other displays!

Epson has an actual see-through display on the market, and it’s relatively cheap!

Like Project Glass, the Moverio BT-100 includes an Android device and is see-though.  But, it’s also stereo!   Unfortunately, it doesn’t include sensors on the display (either a camera or orientation sensors to tell which direction the head is looking), so it can’t really be used for augmented reality.  I am not even sure if it includes GPS (to let it at least be used for a limited collection of location aware applications, like Project Glass does).  So, again,  this isn’t really useful for the kinds of things we want to do in AR or even wearable computing.  The integration and reliance on an Android device (running a relatively old version of Android) is also a huge problem for folks like me, who want to try new things.  We are porting our AR-enabled web browser (Argon) to Android right now, but will only support ICS (Android 4.0) and above; in fact, since there are bugs in the Android webview that won’t be fixed till at least the next version (I hope) we probably can’t even support 4.0.  Having to use 2.2 or other old revs of Android isn’t that useful.

So, the world keeps moving forward, things keep getting better … but we aren’t quite there yet!

Almost immediately after posting their video on the Glasses project, creative folks around the world began posting knock-off videos, expressing their thoughts on the projects through pictures rather than words.  Because of my rather negative comments on the video, I’ve had a few people send me links to them.  I wanted to reply to a few here, since I think both of these videos (and ones like them) miss the point.

First, there are a number of the “oh, you covered my vision and now I bump into things” videos.  The one below was the first I saw, and also pokes fun at the seemingly perfect and appropriate contextual speech recognition capabilities in the original video.  The later is valid observation, and would hopefully be handled via a physical switch, like on Siri or Android’s speech recognition (or have more “confirm?” multi-step interaction).  But, the first problem, blocking the user’s field of view, is a side-effect of my major issue with the Google video, namely their unwise presentation of the heads-up virtual content as covering the full field of view of the wearer of the display (in contrast to the hardware images, which show the display as tiny, located up to the side).  The virtual content shown in the video is actually a pretty nice design for a small, peripheral heads-up display, but is terrible for an immersive AR display.  I do think people should reflect on both the “automated natural interaction” presented in the video, and re-think their view of the video overlay if it was really out of their focal view.

YouTube Preview Image

The second group of videos are on the theme of “google is really just going to use this to put ads everywhere”.  The one below is also the first one of these I saw, and while it’s cute for a laugh, I think these videos miss the point.  While I can believe some folks deep inside Google would love to monetize your view of the world this, it’s pretty clear that an obnoxious cluttering of automatic ads (or even a less obnoxious one) would kill such a product.  The Glasses are a novelty, and would be pretty easy to take off and sell on eBay if they subjected their users to an intruisive barrage of ads.  My suspicion, actually, is that the Glasses would run apps (much like other Android devices) and individual app manufacturers would choose their monetization strategy as the desire (much as they do now).  There would be more options available to them, obviously, but just as ad-supported internet access (you all remember those days, right?) was an abysmal failure, so would an ad-supported wearable display.

YouTube Preview Image

Despite all of these issues, I’m still pretty excited about what these glasses represent.  If Google can truly figure out how to make a usable, always-connected, stand-alone wearable heads-up display, for approximately the price of a phone or a tablet, we’ll finally have a sandbox in which to see if there truly is a use for always-available access to information.  It’s not an ideal device, but it looks pretty good.  The array of sensors and capabilities seems like a nice balance, and if they can crack the “display quality” issues (the Big If[tm]), it’ll be interesting to see where it goes.

Too bad the video wasn’t better.

So, Google has finally released some pics and a video showing off their “Project Glass” head-worn display concept.  I have many reactions to the ideas and concepts presented in it, some good and some bad.  I think the glasses exhibit some nice industrial design, for example (although they’re still to geeky for broad adoption).  And the idea of them being a stand-alone device is really cool (complete with Android phone functionality and a variety of sensors for understanding and interacting with the world);  it’s something I’ve mocked up in my group, as have others around the world, and have been proposing to research sponsors for years (but, most of us don’t do hardware, so it’s not like we could have ever done this pretty a job!).  So, like many people, I’ve been waiting for more information on the project!

Alas, though, my main reaction to the video is “Oh no!”

Read the rest of this entry »

In the most recent issue of ACM Interactions, there was a short article by the ACM CHI Program chair titled “A Cry for More Tech at CHI!”. Since Interaction is not a blog, where you can’t reply, I’m replying here, because the article struck a nerve.

While the column starts well (Let’s get more tech at CHI!), the direction the article heads in at the end isn’t that fruitful.  Here’s my view.

People don’t bring tech to CHI because CHI doesn’t respect tech (in pretty much any form:  novel tech, systems that test new tech, etc)

As a tech-oriented researcher, I can’t easily get papers published at CHI.  I’ve had a few, but those were outliers, and only “squeaked” in.  The metrics CHI reviewers use for tech papers are not appropriate for technology-focused research (where’s the study?  Where’s the killer app for this tech, validated by ethno-whatever studies of some important domain?) so “real” tech papers have such a low chance of acceptance that nobody in the right mind would submit them.

I think the CHI community know and largely accepts this, which is why conferences like UIST were created. Fine.

But, the flip side, the idea that the same folks should bring their demos and show them off, is absurd.

A CHI demo is not refereed, so its not respected by our peers.

Doing a demo at a conference, when you have a paper about the work, is great.  You have a respected contribution, and you are letting folks play with it and try it out.

But just doing a demo at CHI is like paying a ton of money to be a “dancing bear” in a circus.  If the CHI “research” community does not want the work in the papers track (i.e., it’s not good research!), why would technology researchers go to the extreme trouble and significant expense of doing a demo at CHI?  The conference is absurdly expensive, and doing a demo requires 2 or 3 people to be there.  If you are going to be there anyway, doing a demo is fine.

It’s much more economical and impactful to go to the conferences that will publish the work, and demo there.    Then post the video to youtube for the folks who don’t  go!

So there you have it.  If you want want technology at CHI, start accepting the work into the papers track, and stop trying to create new venues for those folks to be dancing bears.

I’ve been thinking about board games and card games recently (ok, I’ve been thinking about them for years, but over the past few days, I’ve been thinking about them more than I have all semester).  Part of the motivation I have for creating “AR board games” (AR games that combine computer games and board/card games) is to recreate the social play experience of these games.  The experience I imagine is that of sitting around a table, playing a game with friends, where you are looking at each other and playing in the same physical space.  These experiences seem qualitatively different than the experience of multiplayer computer/console games, even games on the Wii.

Read the rest of this entry »