CINeSPACE presentation in Barcelona

Just got back from three days at 5th International Conference of the Virtual City and Territory hosted by the CTV in Barcelona. The conference was full of gems with presentations spanning many domains. I presented "New Approaches for Mixed Reality in Urban Environments: The CINeSPACE Project" and discussed the way the CINeSPACE device uses Mixed Reality to let the user discover things about the cultural heritage of a city.

The Long Journey to Mixed Reality, Photosynth with the Wiimote (Wii Remote)

The year was 1995. Ace of Base was in the top 10 and Sweden was the place to be. I was just finish my graduate studies in Plasma Physics and my cousin and I were working on a secret project. I was working as a consultant with VR and my cousin was working at a large Swedish company specializing in radio and telephones.

MIT Media Lab was all the rage with the wearable computing initiative which was one of the first groups to widely publicize the use of solid state accelerometers. Or plan was simple: to build an inertial tracker based on solid state accelerometers and filter the output to reduce the error and use this device information combined with GPS to make what is now called a Mixed Reality game.

All this when we were not busy having too much fun in the fantastic Stockholm nightlife.

Fast forward billions of dollars of investment and 12 years....

I now have in my hand the final product. Its done!!

The good news is that the billions of dollars weren't mine and that the device which costs $39 has bluetooth to integrate to any computer.

That right it is the Wii Remote and you don't need the game console you can just buy the remote separately.

The bad news is I don't have time to play games (never mind th Stockholm nightlife) anymore since I am busy most of the time working on project schedules or PowerPoint's.

Mixed Reality is a reality and almost every new computer, PDA, digital camera or mobile phone now sports at least an accelerometer.

To get convinced that the Wii Remote is an incredible thing watch this TED video.

And for the real hackers out there check out Johnny Chung Lee's wii page where he tells you how to do various wild and crazy things with the Wii Remote. The software for download (evidently more than 500,000 people have thought this was the cool thing to do) is there and it allows you to read the accelerometer values directly.

Want to work on your Tennis or golf swing? Connect the Wii to racket or golf club!

But what am going to do with my Wii Remote right now?

Use it to demo my favorite application: Photosynth and pretend I am in the movie Minority Report.

Johnny Chung Lee says that the arms get tired after waving them around for a while. A small price to pay for a dream come true.

Computer Vision Gurus and a Cool Technique to Use PhotoSynth as a 3D Scanner of Any Object

(3D scan of a Sculpture by René Shapshak created using PhotoSynth)

Well after a few days of horrible weather this weekend the sun finally came out but as much as I wanted to get out, for reasons known only to some strange and alien creatures from another planet, I was stuck deep in review of the fantastic world of SIFT, affine transformations, Homogeneous coordinates and other such chin stroking stuff.

It was as if Drs. Zhang and Lowe had jumped out of their personal web pages and where prodding at me to figure out how it could be possible to solve the insolvable.

I mean how could it be possible to take a few pictures and from those pictures make a 3D model. No way I thought. Impossible. Maybe if you measure at least a few of the distances involve or tell the program what kind of lens and what kind of zoom you are using.

Zhang and Lowe just rolled their eyes and told me to keep trying. And try I did.

First thing you have to do, the ghost of Lowe told me, was to use computer vision algorithms to find the feature points with the SIFT method. This method uses all sorts of fancy math like Gaussian, Gaussian pyramids and a whole bunch of other stuff to find features in the images. These features are special, very special in that they don't change as you move away or rotate the image. This makes them great points to look for if you are stitching together photos to make a panorama.

OK so now we have these feature points but how do we know where they are in 3D space. Well this is when the ghost of Dr. Zhang floated over and told me, "for that you need to find the intrinsic and extrinsic camera matrices."

That did sound very cool so I decided to do just that.

But how to do it. I remembered an episode of "24" where Cloe had used a library from Intel called CV to do something like this but the episode of CSI where they use Photosynth from Microsoft to do something similar seemed even better.

Cool I thought, I won't have to program at all. I'd just plug the photos into Photosynth and off I go.

And off I went to install the Photosynth plugin in my exiled Explorer browser. The rest must have been good for Ms. IE. though because as she came up everything went fantastically smooth. The plugin installed and I found my self mesmerized by the Pyramids of -not Gauss- but of Giza. Boy would I like to have a 3D model of that. Wouldn't it be cool to have a digital copy millimeter by millimeter of the egyptian pyramids on my hard disk. That's what I wanted. For sure. Cool!

So now I had a plan. I was going to use Photosynth and my cameras to make a 3D digitizer of everything I could get my hands on or visit. Would this possible? I couldnt see why not, after all they where right there on my screen. Just holding down the CRTL button in Photosynth makes those litten feature points appear like Christmas lights on 5 ave in December.

What to start with. I looked around. A little sculpture would do nicely. I took some pictures from different angles and then created my Photosynth. Its here if you want to see it.

And sure enough as I held the CTRL button down those magical feature points in 3D showed up. So I just hit left click and then save --- what no save *?¿?. This was a problem-my plans where shot. Photosynth would not be my 3D digital scanner. The guys at Photosynth said they would releasing an API etc etc. but with the world economy crashing I thought it would be better sooner than later to get my 3D digital scan of the sculpture.

So a little digging got me to these guys who have written a script to parse the files that Photosynth uses and makes guess what...a file with the 3D points of my sculpture that I can store on my harddisk and they are even color coded!!

Mission accomplished!

Here is the recap for those of you are still trying to solve the Gaussian pyramid from the first part of the lecture ... woops I mean explanation ;-)

Step by Step approach to make a 3D digital scan of anything

1. Install Photosynth
   
2. Make a Synth
   
3. Look at the page source of your synth when you look at it and find the synth file which has a URL like: http://mslabs-771.vo.llnwd.net/d8/photosynth/m6/collections/c0/01/16/c00116f9-3342-4c33-91aa-d37cbeb0fba9.dzc
   
4. Then change the ending .dzc to .points_0_0.bin
   
5. Download this bin file
   
6. Run the python script on the file (Oh forgot-you will need Python, but its a good thing to have anyway :-) )
   
7. Pipe the output (that's slang for putting > temp.csv after the name of the python script) to a file like temp.csv
   
8. The same guy who made the python script has a nice little Processing program (Processing is a really cool Java Graphics Tool) for looking at the data. I used this to make the image above and it works great.
   
9. If you would like to see other information about your synth these guys tell you how to look at the associated XML which has the image information.

For more information about Dr. Lowe´s work see:

http://en.wikipedia.org/wiki/Scale-invariant_feature_transform

And for the work of Dr. Zhang see this link:

http://research.microsoft.com/en-us/um/people/zhang/Papers/TR98-71.pdf

Using the Mixed Reality device in Disaster Areas

Dr. Robin Murphy 

The use of Photosynth and CINeSPACE like tools could have a large impact in disaster areas.  The MIT Technology Review reports that Dr. Robin Murphy, Raytheon Professor of Computer Science and Engineering and fellow researchers from Texas A&M University (TAMU) lab have created an application which uses several small unmanned air vehicles (SUAVs) or drones to take images to assist in the evaluation of the most important areas for search and rescue teams to concentrate on.  

What is clearly interesting is to use this input in Photosynth software with in situ functionality like that of the augmented reality binocular device CINeSPACE to allow the search planners to morph into the setting.

Mixed Reality Contact Lens

"Looking through a completed lens, you would see what the display is generating superimposed on the world outside," said Babak Parviz, a UW assistant professor of electrical engineering about a micro display they have inbeded in a contact lens.  "This is a very small step toward that goal, but I think it's extremely promising." reports the The University of Washington News.

But as the Guardian points out the real problem is how to focus on the image so close to the eye. After all you can see dust particles or fancy mini LEDs on the surface of the eye since that much too close for the eye's lens to focus on.

"To create the focused image we have to manipulate the light rays," says Parviz. "You can create a focused image if you use laser instead of LEDs."

The Guardian reports that, Parviz might integrate an array of individual micro-lenses into the contact lens. "If the pixel [the microLED] is close enough to the micro-lens, it will generate a virtual image that could be 30cm or more away from the surface. Our eyes can focus on this now."

Nikons Media Port Up

Nikon's Media Port Up for around $600 adds WiFi to the single near the eye screen. The specs arent all that impressive with a 640 x 480 0.44-in. transmissive LCOS microdisplay and only a 2 hour battery life watching video.

I found using this kind of device problematic as my other eye very much disturbed me, but in some situations where hands free information is important, the guys from Trivisio say they get a fair amount of orders and that it can be an excellent tool.

Here is a picture of Andrea from the CINeSPACE team using a similar Trivisio device with a much higher resolution microdisplay.

From: ISONTECH

As a user of Mathematica in Graduate school I remember how cool Mathematica was in doing some pretty complex Plasma Physics equation work.  So when Wolfram says that he is about to do something new I am one of the first to listen.

What have got here:

• Natural Lanuguage (Check)
  
• Symbolic Language (Check)
  
• Choas, Strange Atractors and Basics of Phsyical Reality (Check )
  

So I guess in just about 45 days time Wolfram will fire up the engine.  I just hope the answer to every query wont be 42.