Stereo camera

Are you talking about in-game rendered stereo or physical stereo cameras mounted on the headset?

If you are just talking about in-game stereo, then of course this works with Unity (and Unreal).

Real life stereo. (Also I'm referring to openGL with GLUT toolkit)

I haven't tried it myself, but it seems very possible to me.

Unity apparently supports a WebCamTexture that lets you feed data from a camera into the scene. However, in order to do this effectively you need precise knowledge of the latency of the camera.

A naive approach would be to just take the webcam texture and place it in the scene at a fixed position relative to the Unity camera. However this doesn't produce very good results because in the delay between when the image was captured by the webcam and the time when it's rendered in the scene, your head may have moved.

But, if you track the head pose for when images are captured, then you can compensate for the change in head pose by the time they're displayed. Like this:

And here is naive approach as explained by jherico (simpler to implement but the latency of the 3D camera might be uncomfortable (the ZED camera seems to provide 120fps with VGA resolution)): https://github.com/stereolabs/zed-oculus
It depends of what you would like.

Thanks both. I'll be rendering videos from two GoPros 4 black @90 FPS and 1080p resolution. Since GoPro has fisheye distortion lenses can we simply use openCV to render videos on to HMD? I understand the latency part. If we have a good transmitter we can cut down the latency. Again if latency wasn't an issue can this be done?

"saimouli" wrote:
Since GoPro has fisheye distortion lenses can we simply use openCV to render videos on to HMD?

You cannot use OpenCV alone to render to the rift, if yoiu have the rift runtime installed. You can only render to it using an oepngl or Direct3D texture via the SDK. If your doing have the runtime or drivers. Installed, then its just another monitor.


Regardless,just because your camera has a fisheye lens doesn't mean it would look correct in the rift unless the lens distortion happens to be the exact inverse of the Rift lens distortion, which is extremely unlikely.