Natural interactions for medical image analysis

I just created a page for all the material I used in the 3D tracking prototype. You can download the vvvv-patch and the 3D files from there if you want to play around with it.

Last night, 3 days before the final presentation, I have reached a state in my prototype that I am really happy with. The tracking stability is even better than expected with the new LED pen. Also I added a basic graphical user interface and some cool features. But look yourself, I am explaining it in this 5 minute video:

I found this nice plugin for vvvv called “roentgen” by Sebastian Gregor and implemented it into my 3D tracking prototype. I looks great and gives the prototype a whole new level. I was not sure if I’ll be able to prototype this idea, so I am really happy that it worked in less than a day!

Here are some screenshots… I’ll shoot some videos later when I’ve implemented some more features.

I experimented with video-3D-tracking before here(1, 2, 3).

This is my latest 3D tracking prototype… I added 2 colored leds to the pen to track its position and rotation in space. Also I’ve implemented a 3D model of an actual CT scan… The pen has two different functions at this moment. You can point at the 3D model. And you can switch into a viewing mode in which the pen behaves like a camera, so you can easily zoom and view the (virtual) skull from all directions.

close up photo of the pen…. assembled with electrical tape. You can also see the digital projection of the pen on the lower screen (that is supposed to be the touchscreen)

This is the second iteration of my multitouch prototype. I added a big screen to the setup as it is in my design concept. The big screen shows only the images without any grpahical interface. All GUI and detailed information about the scan is showed on the touch surface.

I still don’t have a projection on the touch surface – but you can see the projection on my laptop screen in this video.

I decided not to build a full working multitouch-screen-table because it would cost to much time and also I could not travel with it between my two workspaces in Eindhoven, the Netherlands and Umea, Sweden. But I still think this projects needs something to evaluate and test my ideas, so I’ve built this very simple multitouch prototype. I haven’t implemented the projection yet, so its still a multitouch-trackpad and not a full touch screen. But I might add projection to it later on.

I used reacTIvision for finger and fiducial detection. Reactivision sends the data via OSC and I can use vvvv or flash to interpret the data (thats where the fun starts)

full resolution pictures and descriptions after the click….

Read the rest of this entry »

This is a prototype for pointing and drawing in a 3D space. I think this could be helpful for annotating and marking 3D body scans.

I used a simple color tracker to detect the pink top of the pen. 1 webcam is tracking x and y position, the 2. webcam is tracking the z position of the pink-pen-top. But you could also imagine this working with tracking a finger or even multiple fingers.

I think it is a very interesting to make these 3D interface explorations… but how reasonable are they for the radiologists work? At this stage of the 3D concepts I still see a lot of problems.

Dicom image viewers have the ability to show scanned images with different filters applied. It helps in situations where the radiologist is interested in only some of the body parts. For example there could be a filter that shows only the bones and another filter that shows only the blood vessels. I was thinking about how to combine these different “layers” of information and came up with this concept of “viewing-lenses”. You can drag these lenses over the picture and see different layers of information.
Imagine a touch table with tangible-sheets like in the Microsoft Second Light project… or maybe is better to be able to change the shape and size of the layers. Another idea that came to my mind was combining the lenses – one that shows the bones, another that shows the blood-vessels, put them over each other to see bones and blood-vessel at the same time.

Here is a quick flash prototype to illustrate the idea:

Taking the whole idea and mixing it with my cube interface concept could also be interesting…

In my last meeting with my mentor Mine we discussed the idea of the physical-cube-interface. Mine had a good point by mentioning the difference between looking at a 3D model on your own to the situation in a video conference. I think the augmented reality approach (in which you see the actual video stream with your own hands and the cube, augmented with the digital 3D model) is great for discussing a case online in a video conference, but doesn’t make much sense while analyzing a case on your own. If you are on your own, you don’t really need to see your own hands on the screen.

So I think there should be 2 concepts – one for working alone and another one for discussing and showing a case online.

In the second version of my physical-cube-interface prototype you can not see the video stream. I tried to make the tracking more stable and also implemented some of my initial ideas for viewing actual radiology data. This is still a very early prototype but I have the feeling that I am on the right track here. While playing around with the prototype I become more familiar with the advantages and problems. It already feels really natural to view a 3D model. But I think the other ideas I’ve implemented (pressure sensor, move cube left/right to change transparency) need some more consideration. At this stage there is too much things happening at the same time.