Archives for category: commercial

I’ve been contemplating Project Glass recently, imagining what the user experience of wearing one might be.  While I’m increasingly looking forward to trying one out, one problem that keeps coming to mind is the social acceptability of walking around with a camera pointing at everything I look at.   The more I ponder it, the more concerned I am that if Google isn’t careful, Project Glass may crash and burn, and set back public acceptance of head-worn displays quite a bit, because of that darn camera.

It may seem counter-intuitive to assert that its camera could be the downfall of Google’s Project Glass.  After all, much of what gets shown off during Glass demos is it’s ability to grab pictures and movies with little to no effort.  Movies while swinging a kid in the backyard, or while falling through the sky.  Pictures of talk show hosts and beautiful people at fashion shows.  Much of the devices ability to sense a wearer’s context and proactively provide serendipitous information and opportunistic interaction depends on the camera.

Theres a huge problem with the trajectory of their current design, however:  the people being looked at can’t tell if the camera is taking pictures.  Privacy and security aren’t about me (the wearer), they’re about you (the targets). As a huge enthusiast for mobile augmented reality, and  someone who’s looking forward to trying one out, I’m uncomfortable that Google seems not to”get” this or be taking it seriously.

During the demos, you see Sergey and others make a cute hand gesture on the arm of the Glass to trigger a picture;  some might naively say that’s enough to signal to someone else that a picture has been taken.  But, it’s obviously not:  while in that one instance we know a picture is being taken,  how do you/me/we (the targets, again) know that this is the only time a picture is being taken?  And there lies the problem.

Early iPhone developers who wanted to leverage the camera on an iOS device will remember Apples tight control over access to the camera.  For years, it was impossible to grab images from the camera without using the built in image capture widget; and even then, it was impossible to capture images without the loud faux-camera-shutter sound!  When we created Arf, the iOS virtual pet years ago, we couldn’t release it because of the hackery involved with accessing the camera in real time.  One of the reasons apple was so careful with camera access was a desire to make both the user of the phone and those near the phone comfortable knowing if and when pictures were taken.

And while the restrictions on developers have eased, Apples careful approach was largely successful.  As a society, we’ve adapted to camera phones.  We know that if someone is holding up their phone, they might be taking pictures.  Locker rooms at athletic clubs now regularly ban camera phones, for obvious reasons.  Some high security government agencies entirely disallow phones on the premises.

But (practically speaking) it’s extremely hard to take pictures with a phone without your target knowing.  Try facing someone you don’t know, holding up your phone and pointing it them;  it’s easy for them to see you do this.  Now imagine pointing it at a couple with their arms around each other on a park bench;  at children you don’t know on a playground;  at an attractive person of the appropriate gender in a coffee shop or bar.  It’s easy to imagine, shall we say, a “diversity” of reactions.

Now consider what will happen when people realize (or imagine) that folks wearing Glass are pointing cameras at them, all the time, every minute they are nearby, and could be recording everything they see and hear.

Consider the bizzarre recent altercation in a Paris McDonalds between Steve Mann and the employees.  We may never know what really happened, if Mann was at fault (perhaps provoking the employees in some way) or if the employees were at fault (over-reacting to something that made them uncomfortable).  What is clear is that during this episode Mann was taking pictures of the whole thing, and by extension, likely to be taking picture of anyone and anything he looks at, all the time.  While he hid the employees faces in the pictures he posted, the point (that is largely being ignored by the media and wearable enthusiasts) is that he was taking pictures continuously, and  could later choose a set to post, a set that supported his point.  It’s not hard to imagine what will happen when thousands of people are “wearing” and able to mine the images/video/audio they’ve been capturing as they go about their day.

A first step toward heading off possible problems is obvious:  Google really needs to take this seriously.  At a minimum, they need to ensure that any time the camera or microphone is operational, there is visible (and perhaps auditory) feedback available to those near the camera. There should be one kind of feedback when the camera or mic is operational at all (even if just in preview mode), different feedback when an image is captured and passed to an app on the device or stored in the camera roll, and perhaps a third sort of feedback when video or audio clip is being captured. And this really has to be done at the hardware level, not at the OS level (since the OS will likely be open like Android, to support research) and certainly not left up to app developers.

Doing this right is a serious challenge.  A simple LED on the front could easily be covered with tape;  so  something might need to be integrated into the camera lens assembly so it can’t be covered without destroying the cameras ability to capture images.

Whatever the solution, it would be a shame if the camera, the thing that gives Glass much of its power and appeal, might also be its undoing.

Over the years, many have dreamed about using mobile technology to create a kind of “augmented reality translation system” that translates whatever you are looking at into a language you can understand, and displays it in place.  Back in 2008, Intel’s Paul Otellini demonstrated a prototype Chinese-to-English translator as part of his CES Keynote.   More recently, Word Lens for the iPhone does a beautiful job of cleanly translating text in place (supporting english, spanish and french), but the translation (in my experience) tends to be a bit spotty.

Now, Microsoft has demonstrated a far more comprehensive approach as part of Bing Translate for Windows Mobile, and it looks amazing.

[msn c0b91059-9ce0-477f-a391-53fbb53c061e 534 300]

I’ll be trying it out on one of our Nokia 800′s, and it along would probably make it worth carrying the phone on international trips, even though I’m still firmly an iPhone user.  If they keep putting amazing things like this out, and decide to add in better non-MS-centric mail and calendar to WP8, and perhaps that will change.

Ok, I know I’m either going to love or hate a blog post about an internal research project on mobile AR, when the first line is

It all started with Snow Crash.

Michael Abrash, game and tech industry veteran, and now at Valve, recently wrote a blog post about his journey to Valve and the current internal research project on what he calls “wearable computing.”   For the rest of us, though, the description of what they are doing sounds exactly what researchers like me would call fully immersive AR. (Ironically, the folks at Google are doing something that the research community would traditionally call wearable computing, and folks want to call it AR, but that’s a different conversation.)

Suffice to say, I loved this post.  Now, he doesn’t really provide much information on what they are doing, so it’s not that it’s the details that are what’s compelling.  But, I find myself really excited about the idea of a top tier game company doing an internal project that is focused on what is arguably the distant future of AR gaming.

Sure, companies like Sony and Nintendo and Microsoft have built AR games for recently, and the core feature of the PS Vita, 3DS and Kinect is the ability to do augmented reality style games.  But, all of these products and projects are focused on the near future, on games and experiences that will be possible with this hardware. In my research group, especially our Augmented Reality Game Studio, we’ve been working on similar kinds of games for many years, and so I obviously see the value in trying to figure out how to create compelling AR game experiences with current hardware and software (heck, we will hopefully be releasing one of our games, Nerdherder, into the app stores soon!)

But, lately I’ve been thinking that it’s time to refocus on the future.  When we started the AR Game Studio, it still wasn’t easy to build AR games on mobile devices, and we were lucky to team up with Qualcomm to see what would be possible with what has since become Vuforia, their free AR SDK (that is probably one of the best solutions for doing AR on mobiles right now).  Now that Vuforia is out there, and is available for the Unity game engine as a free plugin, everyone can get in on the action!  If you can build games in Unity, you can build AR games!    I will keep teaching AR game design classes, and working with students in my lab to build these games.  And my students and I will keep investigating how to create compelling experiences of this sort (with the advantage that we can study what others make, too, not just what we’ve built).

But, my first love is working with technology and concepts on the cutting edge that are (for all intents and purposes) impossible on a large scale, rather than studying and understanding what people can do right now.

Which brings me back to Michael Abrash.  I have no idea what they are really doing.  But, the thing I’ve learned about game design over the past half dozen years, building games in my lab and at my company, is that game design is different than research, even if your research needs you to build a game.  Unlike researchers or product engineers, game designers are trained and focused on creating compelling experiences.  Things only matter insofar as they are perceived by the player and impact the experience of the player.  If the experience sucks, the experience sucks:  there is no “oh, it only works if you do this” or “imagine what it would be like if this was better or that was better”.  There is no making it work well enough for the paper, or the video, or the evaluation.  There is only the experience.   As Miyagi said in The Karate Kid:

Miyagi: Now, ready?
Daniel: Yeah, I guess so.
Miyagi: Daniel-san, must talk. Walk on road, hm? Walk left side, safe. Walk right side, safe. Walk middle, sooner or later, (makes squish gesture) get squish just like grape. Here, karate, same thing. Either you karate do “yes”, or karate do “no”. You karate do “guess so”, (makes squish gesture) just like grape. Understand?

AR (or wearable computing, as Michael calls it) will not really be possible any time soon.  To safely and unobtrusively integrate content with your view of the world around you, you must be able to achieve the kind of integration we see in live sports on TV, and that simply isn’t going to be feasible soon.  But if you want to work towards that dream, you need to decide:  am I building something to take me one step in that direction (e.g., my games in the AR Game Studio, or release of the Argon AR-enabled web browser, Google’s Project Glass) or are we going to just try to create a compelling experience of what it might be like when we get there.

I don’t know if the later is what Valve really wants to do, but I think a game studio that is saying up front

To be clear, this is R&D – it doesn’t in any way involve a product at this point, and won’t for a long while, if ever – so please, no rumors about Steam glasses being announced at E3.

is likely to do something interesting and exciting and different than the other projects you see around the web.

Too bad Valve isn’t in Atlanta.

In light of all the talk of Google’s project glass, there are a bunch of folks suggesting that the other displays on the market can already do what Project Glass is trying to do.  And, I will agree, some of these displays are getting closer.

Brother’s Airscouter (apparently due out this summer in Japan) is indeed a see-through virtual retinal display, and is “relatively small”.

YouTube Preview Image

But, it’s a LOT bigger and bulkier, and it’s “just the display.”   No camera.  No computer.  No sensors.  So, in practice, it would be much bigger, and a lot uglier.  As a researcher, of course, I’m totally fine with this, since it will let us do the research we want with the platforms we want (from desktop to laptop to tablet to handheld) and not be bound to a specific OS or set of sensors.  But, for practical use, that’s a huge problem.  So, we’ll likely be buying some (there’s nothing else quite like it coming out), just as we’ll be buying other displays!

Epson has an actual see-through display on the market, and it’s relatively cheap!

Like Project Glass, the Moverio BT-100 includes an Android device and is see-though.  But, it’s also stereo!   Unfortunately, it doesn’t include sensors on the display (either a camera or orientation sensors to tell which direction the head is looking), so it can’t really be used for augmented reality.  I am not even sure if it includes GPS (to let it at least be used for a limited collection of location aware applications, like Project Glass does).  So, again,  this isn’t really useful for the kinds of things we want to do in AR or even wearable computing.  The integration and reliance on an Android device (running a relatively old version of Android) is also a huge problem for folks like me, who want to try new things.  We are porting our AR-enabled web browser (Argon) to Android right now, but will only support ICS (Android 4.0) and above; in fact, since there are bugs in the Android webview that won’t be fixed till at least the next version (I hope) we probably can’t even support 4.0.  Having to use 2.2 or other old revs of Android isn’t that useful.

So, the world keeps moving forward, things keep getting better … but we aren’t quite there yet!

Almost immediately after posting their video on the Glasses project, creative folks around the world began posting knock-off videos, expressing their thoughts on the projects through pictures rather than words.  Because of my rather negative comments on the video, I’ve had a few people send me links to them.  I wanted to reply to a few here, since I think both of these videos (and ones like them) miss the point.

First, there are a number of the “oh, you covered my vision and now I bump into things” videos.  The one below was the first I saw, and also pokes fun at the seemingly perfect and appropriate contextual speech recognition capabilities in the original video.  The later is valid observation, and would hopefully be handled via a physical switch, like on Siri or Android’s speech recognition (or have more “confirm?” multi-step interaction).  But, the first problem, blocking the user’s field of view, is a side-effect of my major issue with the Google video, namely their unwise presentation of the heads-up virtual content as covering the full field of view of the wearer of the display (in contrast to the hardware images, which show the display as tiny, located up to the side).  The virtual content shown in the video is actually a pretty nice design for a small, peripheral heads-up display, but is terrible for an immersive AR display.  I do think people should reflect on both the “automated natural interaction” presented in the video, and re-think their view of the video overlay if it was really out of their focal view.

YouTube Preview Image

The second group of videos are on the theme of “google is really just going to use this to put ads everywhere”.  The one below is also the first one of these I saw, and while it’s cute for a laugh, I think these videos miss the point.  While I can believe some folks deep inside Google would love to monetize your view of the world this, it’s pretty clear that an obnoxious cluttering of automatic ads (or even a less obnoxious one) would kill such a product.  The Glasses are a novelty, and would be pretty easy to take off and sell on eBay if they subjected their users to an intruisive barrage of ads.  My suspicion, actually, is that the Glasses would run apps (much like other Android devices) and individual app manufacturers would choose their monetization strategy as the desire (much as they do now).  There would be more options available to them, obviously, but just as ad-supported internet access (you all remember those days, right?) was an abysmal failure, so would an ad-supported wearable display.

YouTube Preview Image

Despite all of these issues, I’m still pretty excited about what these glasses represent.  If Google can truly figure out how to make a usable, always-connected, stand-alone wearable heads-up display, for approximately the price of a phone or a tablet, we’ll finally have a sandbox in which to see if there truly is a use for always-available access to information.  It’s not an ideal device, but it looks pretty good.  The array of sensors and capabilities seems like a nice balance, and if they can crack the “display quality” issues (the Big If[tm]), it’ll be interesting to see where it goes.

Too bad the video wasn’t better.

So, Google has finally released some pics and a video showing off their “Project Glass” head-worn display concept.  I have many reactions to the ideas and concepts presented in it, some good and some bad.  I think the glasses exhibit some nice industrial design, for example (although they’re still to geeky for broad adoption).  And the idea of them being a stand-alone device is really cool (complete with Android phone functionality and a variety of sensors for understanding and interacting with the world);  it’s something I’ve mocked up in my group, as have others around the world, and have been proposing to research sponsors for years (but, most of us don’t do hardware, so it’s not like we could have ever done this pretty a job!).  So, like many people, I’ve been waiting for more information on the project!

Alas, though, my main reaction to the video is “Oh no!”

Read the rest of this entry »

I was looking at the video of this app, “Jigsaw Live,” as I look at most potentially interesting AR apps.  On the surface, it sounds interesting:  ” an AR jigsaw puzzle.”   I’ve been interested in making AR jigsaw puzzles for a while (heck, I even went to the extreme of sending mail to my favorite puzzle company, Stave, to see if I could interest them, to no avail).  But, to me, the concept is about augmenting a puzzle.

After all, AR apps really need to have two components: reality + augmenting.  If reality doesn’t really matter, than why bother with making it an AR app (for example, holding up the phone in this app to solve the “puzzle” is tiring, as it is with any app that requires you to hold the phone up for a period of time).  And, you need to augment the reality in some way … right?  Bring these two things together, and then add something fun, exciting, engaging or useful.

So, let’s consider this app, then.

They have reality, you might say. It’s live video on the puzzle pieces, so that’s reality, right … or, is it?  Does reality matter? Not really, actually.  It’s just video.  Looking at something different doesn’t change the puzzle, nor affect the program.  Reality is “just a texture” … it could be a video or image from you camera roll, and it wouldn’t really change things.  The live video is cool at first;  but, in the end, it’s just a gimmick.

Going further, do they “augment” it?  No.  Their “reality” (the video texture) augments their puzzle, but reality isn’t augmented, nothing is moved out into the world, nothing about the world around you is enhanced.

Perhaps the title “Jigsaw Live:  Augmented Puzzle” would be a more accurate title?  I think so.   Unfortunately, even the puzzle doesn’t look that interesting.  Square pieces?  Up to 100?  For $2.99?  Yikes!  I would have bought this and played with it if it was $0.99, since I’ll pretty much buy any potentially interesting app for that.  But, for $3, you have to do better!    The “ultimate in puzzle fun” (to quote their page)?  Really?  One wonders if the authors even played a puzzle (a real one, not one of those 4×4 square slider puzzles, or the cornucopia of lame flash puzzles).

Ah well, another good app name taken.

I’m including their video here, so you can look at it now, and draw your own conclusions.  What do folks think … am I being too critical?

YouTube Preview Image

 

While I usually reserve this spot to comment on the AR stuff other folks are working on, I thought I’d take this opportunity to comment on something we just released.

My colleague Maribeth Gandy and I create a small startup this year called Aura Interactive;  while we are doing a variety of things with the company, ranging from consulting to product development, our main goal is to create some fun AR games and toys.

And so I’m happy to share that we’ve just had our first iPhone application (done in collaboration with our colleague Kim Spreen) go live in the iTunes App Store, a little toy we call SantaVision (click here to go directly to the app in the iTunes Store).

Like many people, we’ve struggled with the inability to “touch” the live video on the iPhone, and the corresponding inability to do accurate, fast 3D tracking, of the sort we leverage in our research prototypes (e.g., the Zombie shooter ARhrrrr!).  At the same time, we’ve want to avoid creating yet-another shoot-the-<xxxxx>-around-you compass-based game, a couple more of which appeared today.  We think we have come up with a fun middle ground, leveraging the accelerometer and compass to let people immerse themselves inside small AR Christmas scenes they create themselves, using the “stickers” (static and animated images), sound effects and music loaded into the application.  And, using the GPS, you can share your geo-located scenes with the world.

If you try it, let me know what you think.  It’s not perfect, but we’re having fun with it!

I’m not quite sure what this means, but it is probably a good thing:

Zemoga, a pioneering digital marketing and innovation firm, today announced the launch of its augmented reality (AR) practice, designed to further the company’s work in this groundbreaking technology that has taken the marketing world by storm.

[...]

“We built the prototype of the human heart and could visually demonstrate in a dynamic 3D environment the effect a new drug would have on the heart,” explained Zemoga CEO and co-founder, DJ Edgerton.

I wonder if they are teaming up with an more experienced AR folks (if they are in New York, Steve Feiner’s group at Columbia seems like a good choice), to more rapidly leverage the research that’s been done over the years?  Regardless, though, it’s interesting.

Looking back over the summer, it’s almost comical to re-read the sometimes-frenzied excitement at the possibility that the iPhone will support AR with OS 3.1. The possibility was exciting, even to those of us not trying to keep a small company afloat;  the idea that there would finally be a platform that would let us get our ideas, games and products to millions of eager customers was an unfamiliar feeling for those of us who have been doing AR for a (very long) while. And for the dozen or so small AR companies, surviving on contract work for movies web sites and eye-candy web advertising, it must have been intoxicating.
Read the rest of this entry »