Now, Microsoft has demonstrated a far more comprehensive approach as part of Bing Translate for Windows Mobile, and it looks amazing.

I’ll be trying it out on one of our Nokia 800′s, and it along would probably make it worth carrying the phone on international trips, even though I’m still firmly an iPhone user.  If they keep putting amazing things like this out, and decide to add in better non-MS-centric mail and calendar to WP8, and perhaps that will change.

It all started with Snow Crash.

Michael Abrash, game and tech industry veteran, and now at Valve, recently wrote a blog post about his journey to Valve and the current internal research project on what he calls “wearable computing.”   For the rest of us, though, the description of what they are doing sounds exactly what researchers like me would call fully immersive AR. (Ironically, the folks at Google are doing something that the research community would traditionally call wearable computing, and folks want to call it AR, but that’s a different conversation.)

Suffice to say, I loved this post.  Now, he doesn’t really provide much information on what they are doing, so it’s not that it’s the details that are what’s compelling.  But, I find myself really excited about the idea of a top tier game company doing an internal project that is focused on what is arguably the distant future of AR gaming.

Sure, companies like Sony and Nintendo and Microsoft have built AR games for recently, and the core feature of the PS Vita, 3DS and Kinect is the ability to do augmented reality style games.  But, all of these products and projects are focused on the near future, on games and experiences that will be possible with this hardware. In my research group, especially our Augmented Reality Game Studio, we’ve been working on similar kinds of games for many years, and so I obviously see the value in trying to figure out how to create compelling AR game experiences with current hardware and software (heck, we will hopefully be releasing one of our games, Nerdherder, into the app stores soon!)

But, lately I’ve been thinking that it’s time to refocus on the future.  When we started the AR Game Studio, it still wasn’t easy to build AR games on mobile devices, and we were lucky to team up with Qualcomm to see what would be possible with what has since become Vuforia, their free AR SDK (that is probably one of the best solutions for doing AR on mobiles right now).  Now that Vuforia is out there, and is available for the Unity game engine as a free plugin, everyone can get in on the action!  If you can build games in Unity, you can build AR games!    I will keep teaching AR game design classes, and working with students in my lab to build these games.  And my students and I will keep investigating how to create compelling experiences of this sort (with the advantage that we can study what others make, too, not just what we’ve built).

But, my first love is working with technology and concepts on the cutting edge that are (for all intents and purposes) impossible on a large scale, rather than studying and understanding what people can do right now.

Which brings me back to Michael Abrash.  I have no idea what they are really doing.  But, the thing I’ve learned about game design over the past half dozen years, building games in my lab and at my company, is that game design is different than research, even if your research needs you to build a game.  Unlike researchers or product engineers, game designers are trained and focused on creating compelling experiences.  Things only matter insofar as they are perceived by the player and impact the experience of the player.  If the experience sucks, the experience sucks:  there is no “oh, it only works if you do this” or “imagine what it would be like if this was better or that was better”.  There is no making it work well enough for the paper, or the video, or the evaluation.  There is only the experience.   As Miyagi said in The Karate Kid:

Miyagi: Now, ready?
Daniel: Yeah, I guess so.
Miyagi: Daniel-san, must talk. Walk on road, hm? Walk left side, safe. Walk right side, safe. Walk middle, sooner or later, (makes squish gesture) get squish just like grape. Here, karate, same thing. Either you karate do “yes”, or karate do “no”. You karate do “guess so”, (makes squish gesture) just like grape. Understand?

AR (or wearable computing, as Michael calls it) will not really be possible any time soon.  To safely and unobtrusively integrate content with your view of the world around you, you must be able to achieve the kind of integration we see in live sports on TV, and that simply isn’t going to be feasible soon.  But if you want to work towards that dream, you need to decide:  am I building something to take me one step in that direction (e.g., my games in the AR Game Studio, or release of the Argon AR-enabled web browser, Google’s Project Glass) or are we going to just try to create a compelling experience of what it might be like when we get there.

I don’t know if the later is what Valve really wants to do, but I think a game studio that is saying up front

To be clear, this is R&D – it doesn’t in any way involve a product at this point, and won’t for a long while, if ever – so please, no rumors about Steam glasses being announced at E3.

is likely to do something interesting and exciting and different than the other projects you see around the web.

Too bad Valve isn’t in Atlanta.

Brother’s Airscouter (apparently due out this summer in Japan) is indeed a see-through virtual retinal display, and is “relatively small”.

Click here to view the embedded video.

But, it’s a LOT bigger and bulkier, and it’s “just the display.”   No camera.  No computer.  No sensors.  So, in practice, it would be much bigger, and a lot uglier.  As a researcher, of course, I’m totally fine with this, since it will let us do the research we want with the platforms we want (from desktop to laptop to tablet to handheld) and not be bound to a specific OS or set of sensors.  But, for practical use, that’s a huge problem.  So, we’ll likely be buying some (there’s nothing else quite like it coming out), just as we’ll be buying other displays!

Epson has an actual see-through display on the market, and it’s relatively cheap!

Like Project Glass, the Moverio BT-100 includes an Android device and is see-though.  But, it’s also stereo!   Unfortunately, it doesn’t include sensors on the display (either a camera or orientation sensors to tell which direction the head is looking), so it can’t really be used for augmented reality.  I am not even sure if it includes GPS (to let it at least be used for a limited collection of location aware applications, like Project Glass does).  So, again,  this isn’t really useful for the kinds of things we want to do in AR or even wearable computing.  The integration and reliance on an Android device (running a relatively old version of Android) is also a huge problem for folks like me, who want to try new things.  We are porting our AR-enabled web browser (Argon) to Android right now, but will only support ICS (Android 4.0) and above; in fact, since there are bugs in the Android webview that won’t be fixed till at least the next version (I hope) we probably can’t even support 4.0.  Having to use 2.2 or other old revs of Android isn’t that useful.

So, the world keeps moving forward, things keep getting better … but we aren’t quite there yet!

First, there are a number of the “oh, you covered my vision and now I bump into things” videos.  The one below was the first I saw, and also pokes fun at the seemingly perfect and appropriate contextual speech recognition capabilities in the original video.  The later is valid observation, and would hopefully be handled via a physical switch, like on Siri or Android’s speech recognition (or have more “confirm?” multi-step interaction).  But, the first problem, blocking the user’s field of view, is a side-effect of my major issue with the Google video, namely their unwise presentation of the heads-up virtual content as covering the full field of view of the wearer of the display (in contrast to the hardware images, which show the display as tiny, located up to the side).  The virtual content shown in the video is actually a pretty nice design for a small, peripheral heads-up display, but is terrible for an immersive AR display.  I do think people should reflect on both the “automated natural interaction” presented in the video, and re-think their view of the video overlay if it was really out of their focal view.

Click here to view the embedded video.

The second group of videos are on the theme of “google is really just going to use this to put ads everywhere”.  The one below is also the first one of these I saw, and while it’s cute for a laugh, I think these videos miss the point.  While I can believe some folks deep inside Google would love to monetize your view of the world this, it’s pretty clear that an obnoxious cluttering of automatic ads (or even a less obnoxious one) would kill such a product.  The Glasses are a novelty, and would be pretty easy to take off and sell on eBay if they subjected their users to an intruisive barrage of ads.  My suspicion, actually, is that the Glasses would run apps (much like other Android devices) and individual app manufacturers would choose their monetization strategy as the desire (much as they do now).  There would be more options available to them, obviously, but just as ad-supported internet access (you all remember those days, right?) was an abysmal failure, so would an ad-supported wearable display.

Click here to view the embedded video.

Despite all of these issues, I’m still pretty excited about what these glasses represent.  If Google can truly figure out how to make a usable, always-connected, stand-alone wearable heads-up display, for approximately the price of a phone or a tablet, we’ll finally have a sandbox in which to see if there truly is a use for always-available access to information.  It’s not an ideal device, but it looks pretty good.  The array of sensors and capabilities seems like a nice balance, and if they can crack the “display quality” issues (the Big If[tm]), it’ll be interesting to see where it goes.

Too bad the video wasn’t better.

Alas, though, my main reaction to the video is “Oh no!”

Why oh why, Google, did you feel the need to release a video that your project cannot live up to?  In one simple fake video, you have created a level of over-hype and over-expectation that your hardware cannot possibly live up to.  I care for two reasons.  First, because the hardware does look nice, and I think there is some interesting potential here.  Second (and more personally), I work in this area (broadly speaking) and in the mid-term this kind of fakery will harm the research prospects of the rest of us.

Why do I say the video is “fake” and that the product can’t live up to it?

• Field of view.  I don’t know what the field of view of the video camera used to shoot the video, and I don’t even know what the field of view of the Glasses are.  But the glasses (from the pictures) look to have a relatively limited field of view that is up and to the right of the wearer’s right eye;  the video, on the other hand, covers the entire video frame with the supposed display content, giving the impression of complete immersion.  Which hardware like this cannot possibly achieve.  The image I included above is looking directly ahead, at the camera, and you can see his eyes around the display … which means he cannot see anything on the display when he looks straight ahead.
• Stability.  Video-guy is walking around, going about his life.  And the images on the display are rock solid, easy to focus on.
• Depth of field.  Everything is in focus, all the time.  Ignoring the glasses, the world around us is not all in focus all the time.  The glasses will likely have a fixed focus distance from the wearer, so the wearer will NOT see the contents of the glasses overlaid on all of these different contexts and scenarios where the virtual display and the world are both in focus.  This matters, because when you refocus on a virtual object some distance in front of you, everything in the physical world (you know, the stuff that matters!) will go out of focus.
• Image quality.  Amazingly, as display-guy goes from inside to outside, bright daylight to dusk, the contents of the display are uniformly visible.  All while the clear part of the display is perfectly clear.  This isn’t possible, using any technology I’m aware of, at least not for full color.  Now, this is the one that I’d love to be wrong on, since companies have been trying this for years.  Microvision’s Virtual Retinal Displays where able to achieve this with red-only graphics and half-silvered mirrors that reflected the appropriate wavelength of red.

I’m not going to comment deeply on the actual application scenario.  Some are cute, some seem highly dubious.  None of it is novel, pretty much a collection of research ideas going back to Mark Weiser’s early Ubicomp vision and the work the wearable computing has been doing for years.  That’s great;  it’s nice to see the ideas being taken one step forward!

One closing comment, btw.  To all the press:  this is a heads-up display, it’s not “augmented reality”.  AR is about putting content out in the world, virtually attaching it to the objects, people and places around you.  You could not do AR with a display like this (the small field of view, and placement off the side, would result in an experience where the content is rarely on the display and hard to discover and interact with), but its a fine size and structure for a small HUD.  The video application concepts are all screen-fixed (“heads up” instead of “in the world”) for this reason.  This is not a criticism, but we still have a long way to go before someone creates a cheap, potentially usable set of “augmented reality glasses”.

In case you missed it, here’s the video.

Click here to view the embedded video.

While the column starts well (Let’s get more tech at CHI!), the direction the article heads in at the end isn’t that fruitful.  Here’s my view.

People don’t bring tech to CHI because CHI doesn’t respect tech (in pretty much any form:  novel tech, systems that test new tech, etc)

As a tech-oriented researcher, I can’t easily get papers published at CHI.  I’ve had a few, but those were outliers, and only “squeaked” in.  The metrics CHI reviewers use for tech papers are not appropriate for technology-focused research (where’s the study?  Where’s the killer app for this tech, validated by ethno-whatever studies of some important domain?) so “real” tech papers have such a low chance of acceptance that nobody in the right mind would submit them.

I think the CHI community know and largely accepts this, which is why conferences like UIST were created. Fine.

But, the flip side, the idea that the same folks should bring their demos and show them off, is absurd.

A CHI demo is not refereed, so its not respected by our peers.

Doing a demo at a conference, when you have a paper about the work, is great.  You have a respected contribution, and you are letting folks play with it and try it out.

But just doing a demo at CHI is like paying a ton of money to be a “dancing bear” in a circus.  If the CHI “research” community does not want the work in the papers track (i.e., it’s not good research!), why would technology researchers go to the extreme trouble and significant expense of doing a demo at CHI?  The conference is absurdly expensive, and doing a demo requires 2 or 3 people to be there.  If you are going to be there anyway, doing a demo is fine.

It’s much more economical and impactful to go to the conferences that will publish the work, and demo there.    Then post the video to youtube for the folks who don’t  go!

So there you have it.  If you want want technology at CHI, start accepting the work into the papers track, and stop trying to create new venues for those folks to be dancing bears.

But, I wonder about some of the non-obvious ways ways these games are different.  Today, as I was walking to the coffee shop to get beans for our espresso machine, I was thinking about the ways games enforce rules, and how board games and card games are fundamentally different than computer games.  With board and card games, rule enforcement is left up to the players;  they know the rules, and they abide by them.  Even when there is nobody around to “catch” them people naturally follow the rules;  to cheat removes the fun.  My dad played solitaire for hours to unwind after working shifts in an auto plant, and as far as I know, he never cheated.  What would be the point?

Computer games, on the other hand, encourage players to do anything the game allows to win.  Because the system is closed and the rules are enforced by the computer, finding ways to get around the system is part of the fun for many players.  If the game let’s you do it, it must be ok!

But, this presents a problem, which I’ll put this way:  will players treat hybrid computer/board games (or computer/card games) as board/card games or as computer games?  Will they play along and follow the rules, even if nobody is there to “call them on it”, or will they do what they can to win?  When there is a high score board, and achievements to unlock,  will players be content to take what the deck gives, or will they stack the cards?

This question is more than academic, because it impacts the kinds of games we can create.  Consider Sony’s Eye of Judgement, which is designed assuming players will treat the game as a computer game, and thus is structured to allow the computer to enforce the rules.

Click here to view the embedded video.

For those of you who have played the game, you know how awkward it is.  After playing for a while, you realize that you are playing a relatively complex card game (ala Magic), but one where the computer’s main job is to enforce the rules. The computer is used for little else, beyond adding eye-candy to justify playing the game in awkward space of the PS Eye camera.

But beyond the awkwardness, one can’t help but wonder what the game might have been like if the players were trusted to follow the rules and enlisted in the gameplay.  What kind of games could we create if the player was allowed to move cards fluidly, was allowed to manipulate their environment naturally, and not treated as a hostile, untrustworthy interloper?  What if the rules were presented and it was assumed people would follow them?

Imagine a single player augmented card game that starts with the player shuffling a deck and laying out 9 cards in a grid.  No checking by the computer, no proactive rule enforcement.  BUT, like traditional solitaire, the placement of the cards determines much about how the game unfolds.  Would such a game work?  It’s hard to say.  If a player approaches it like a card game, where the goal is to enjoy a pleasant diversion for a few minutes, and challenge yourself to solve a puzzle, perhaps such a game would work.

But, will players even consider that approach?  After years of being trained by computer games to take any advantage offered by the computer, will they “cheat” and chose a card layout designed to give them the best start to the game?  Will they even consider this “cheating”?

I’m not sure, but it certainly impacts the kinds of games we can create.

After all, AR apps really need to have two components: reality + augmenting.  If reality doesn’t really matter, than why bother with making it an AR app (for example, holding up the phone in this app to solve the “puzzle” is tiring, as it is with any app that requires you to hold the phone up for a period of time).  And, you need to augment the reality in some way … right?  Bring these two things together, and then add something fun, exciting, engaging or useful.

So, let’s consider this app, then.

They have reality, you might say. It’s live video on the puzzle pieces, so that’s reality, right … or, is it?  Does reality matter? Not really, actually.  It’s just video.  Looking at something different doesn’t change the puzzle, nor affect the program.  Reality is “just a texture” … it could be a video or image from you camera roll, and it wouldn’t really change things.  The live video is cool at first;  but, in the end, it’s just a gimmick.

Going further, do they “augment” it?  No.  Their “reality” (the video texture) augments their puzzle, but reality isn’t augmented, nothing is moved out into the world, nothing about the world around you is enhanced.

Perhaps the title “Jigsaw Live:  Augmented Puzzle” would be a more accurate title?  I think so.   Unfortunately, even the puzzle doesn’t look that interesting.  Square pieces?  Up to 100?  For $2.99?  Yikes!  I would have bought this and played with it if it was $0.99, since I’ll pretty much buy any potentially interesting app for that.  But, for $3, you have to do better!    The “ultimate in puzzle fun” (to quote their page)?  Really?  One wonders if the authors even played a puzzle (a real one, not one of those 4×4 square slider puzzles, or the cornucopia of lame flash puzzles).

Ah well, another good app name taken.

I’m including their video here, so you can look at it now, and draw your own conclusions.  What do folks think … am I being too critical?

Click here to view the embedded video.

Click here to view the embedded video.

When I think about the discussions we have in our experience design class about how we need to design media experiences that take into account the media culture of the time, I can’t help but think that this is a great example of that.  I wonder how many people are fascinated by the technology, and watch because it’s novel and exciting (judging by the faces of the folks in the video, many do), but then find themselves slightly uneasy about being captivated and entertained by what is otherwise an upsetting thing to watch.  The conflict (however minor) this sets up in someone would be the ideal vehicle for having them reflect on the message and remember it.  Not as in-your-face upsetting as some of the bloody public-service announcements we’ve seen recently (such as the gory anti-texting-while-driving ad that was put out in England this year), but something that causes people to be reflective without needing the shock value.

I wonder what it would be like to experience it;  if anyone reading this has seen it, please add a comment and tell us what it’s like.

Yet, whenever I read about someone’s technical game degree program, I’m always left wondering about the jobs we’re pointing these students at, and the “unwritten pack” we make with students we accept into our programs.  I’ve thought a lot about this, because the topic of a “game degree” comes up occasionally.  I’m also teaching 3 game classes this semester (a game-oriented capstone, a game prototyping lab, and an augmented reality game design class), so I’m very focused on the issue of game education.  Yet, with plenty of course and interest in games here at Georgia Tech, we haven’t created a full-blown game degree;  our game education activities are folded into a number of other degrees that offer a broader education beyond games.

This article isn’t about our choices at GT, though.  Rather, I interested in the opinions of others who might read this.  I believe there is an implicit suggestion that, if we have a focused degree program in a technical area, it’s educating the students in preparation for actually doing something.  With a liberal arts education, the goal is to give a broad education, and the students understand that there aren’t “liberal arts jobs” per se.  But, I suspect that students don’t generally get a degree in Mechanical Engineering, Information Security, or Pre-Med just because they want to broaden their horizons and open their minds;  they get these sorts of degrees (presumably) because they want to work in these areas after they graduate (or move on to other degrees, in the case of pre-Med or pre-Law).

So it is, I think, with a computer science or technical game-oriented degree.  Which brings me back to the topic of the post.  I wonder how many “game degrees” are being created because the school honestly believes that there is an industry need they are fulfilling (industry has unfilled job positions and needs to have more folks educated to fill them) or because the university has a need they are fulfilling (the student enrollments are dropping and they need students to fill up the classes).  For a number of years, CS enrollments have been down at many schools (we’ve been doing OK at Georgia Tech, and if you include our cross-over degrees like Computational Media, we’re actually doing better than OK, both at attracting students, but also at attracting students who aren’t young, white boys).  Over the past few years, a number of schools have created game degrees that have attracted a lot of attention, not least because it appears that they attract a lot of students and because their graduates get to go and work in the game industry.  That’s all well and good;  a few schools (USC’s MS program in the School of Cinema TV, CMU’s MS in Educational Technology, our combined undergrad and grad degrees across CS, CM and Digital Media, for example) have a great reputation with the game industry, and the students coming out of the program have generally had good success in getting the jobs they want.

But, how many schools are just putting lipstick (and new name) on a pig (their dying CS degree programs) to attract new students?

Overall, my sense is that there aren’t that many great jobs in the game industry, at least not in the numbers that are needed to employ an increasing number of game program graduates.  Especially the jobs many students seem to dream about (game design, game engine programming, etc.); here, I’m talking about technical and design jobs, I’m not talking about testing and QA, or level design, or content-oriented jobs (a few schools, like SCAD and RISD and so on, are doing a great job educating those students).

Unfortunately, when I read articles like Mike’s, I’m reminded of the increasing trend of CS departments to offer Game Degrees (note:  I’m not saying this is what USC is doing;  Mike has a long history of game education, going back to the Naval PostGraduate School, and actually came to USC to create a game program in the CS department).  A game degree is a great fit … for the school.  There is virtually no area of CS that isn’t applicable to the technical side of a modern game, especially the blockbuster console games;  these games require everything a major CS degree offers, and then some.  Taking your existing CS courses, faculty, labs and infrastructure, putting some new makeup on it, and calling it a game degree can be a great way to attract students.  And (based on anecdotal evidence) for many schools, it seems to be helping.  Helping them, that is, attract students.

My worry, though, is that as more and more schools offer game degrees, we’re going to turn out a generation of pseudo computer science students who can’t get the jobs they want.  After all, how many jobs are there?  I was chatting with a game industry exec at a recent conference, and he joked that, given the low turn-over in the good jobs, the good opportunities number in the hundreds … not in the thousands or more that will be needed to place these students.

So, I’d love to hear other folks thoughts on this, or get pointers to hard data about jobs and graduates.