Head offset

Seems like the only sure-fire way (short of modifying the engine or duplicating the code inside it, which is unsafe because it could change later) to get the world space center eye position is like this.

			FMinimalViewInfo ViewInfo;
			camera->GetCameraView(0, /*out*/ ViewInfo);

			FVector StereoViewLocationLeft = ViewInfo.Location;
			FVector StereoViewLocationRight = ViewInfo.Location;

			GEngine->StereoRenderingDevice->CalculateStereoViewOffset(eSSP_LEFT_EYE, ViewInfo.Rotation, GetWorld()->GetWorldSettings()->WorldToMeters, StereoViewLocationLeft);
			GEngine->StereoRenderingDevice->CalculateStereoViewOffset(eSSP_RIGHT_EYE, ViewInfo.Rotation, GetWorld()->GetWorldSettings()->WorldToMeters, StereoViewLocationRight);
			
			FVector eyePos = 0.5f * (StereoViewLocationLeft + StereoViewLocationRight);
			FQuat eyeRot = ViewInfo.Rotation.Quaternion();

Here what I have prototyped so far:

1) Adding bFollowHmdOrientation and bFollowHmdPosition to Camera Component class (accessible r/w via Camera Settings property sheet or via blueprint). Once these properties are set (or one of them) and once this particular Camera Component is set as a view target, any rotational / positional change of HMD will be applied to the camera rotation / position. If these properties are not enabled then nothing will happen. Note, at the moment it works only with absolute transform on camera (doesn't work correctly with relative transforms).
Note, there is no notion of two camera (per eye), I don't think it is useful. To change perceptible scale you should change World2Meters scale. Hmm.... Maybe, make this parameters of the Camera Component? So, each camera could have its own W2M scale?

2) PlayerCameraManager also has bFollowHmdOrientation / bFollowHmdPosition, enabling HMD tracking for any Camera Component which is used as view target. Just a bit easier way to manage HMD tracking (no need to enable it for each camera component).

3) PlayerController should have a bFollowHmd for compatibility reasons (for apps which already rely on its current behavior), but it could be disabled by setting bFollowHmd to false.

4) GetCurrentOrientationAndPosition(FRotator, FVector) gets two bool params:
GetCurrentOrientationAndPosition(FRotator orient, FVector pos, bool bUseOrientForPlayerCamera, bool bUsePosForPlayerCamera). If you update your camera yourself then set these params to true, so the plugin will know how to calculate deltas for updating orientation / position on a render thread and for timewarp (for lower latency). Theoretically, one more param is necessary (PosScale3D), to set proper scale for positional delta to be applied on a render thread.

What do you guys think? Let me know. Once everything works, I could share these changes via github through an experimental branch.

@artyom17

I think this is a good start and agree that W2M is probably a better factor than having two separate cameras. This should reduce the complexity and incidence of common questions such as these.

One of the reasons I was thinking of having the 2 camera analog is that right now to send separate images to each eye it requires using a post process material which splits using screenposition. Is there a cleaner way to render opaque or transparent images for each eye with a chosen render order? Perhaps there can be a textured2d or material parameter we can set for each eye, or do you think this belongs as a post process material? Is there a way we can get separate post process materials for each eye (from profiling it looks like the renderer does a PP pass for each eye anyway)?

Additionally I was wondering if we can get at least a C++ function that exposes the raw accelerometer data from the rift for gesture recognition (otherwise using position will delay actions by 2 frames and may have some of the high frequency peaks filtered[are these filtered?]).

Regarding, GetCurrentOrientationAndPosition(FRotator orient, FVector pos, bool bUseOrientForPlayerCamera, bool bUsePosForPlayerCamera) what would the default behavior be? Can this be inferred from the current camera settings in use?

Rendering: I am not sure what you are trying to achieve by that, honestly. In any case, the significant changes in stereo rendering should be discussed with Epic engineers (but first, would be nice to understand why exactly do you want this).

As to raw data: theoretically, it is possible. The only issue I foresee: this data is gonna be Oculus Rift specific. Unlike Unity, UE4 must work with multiple different HMDs (Rift, GearVR, SteamVR, Morpheus, etc). So, it is possible to meet some resistance from Epic guys.... Unless, we re-design IHeadMountedDisplay interface and split it on several smaller interfaces, including HMD-specific ones.... For example, we can have IOculusRiftHMD public intf, that would have that method implemented, and there will be a method IHeadMountedDisplay::QueryInterface("OculusRift") that returns a ptr on it (or nullptr if non-Oculus HMD is connected). Need to discuss with Epic engineers, since again, unlike Unity, these changes are going to be integrated into the main UE4 (sooner or later).

GetCurrentOrientationAndPosition default behavior: by default those bool params would be false and it won't affect any camera/rendering (pretty much what's happening right now). Moreover, those parameters would matter only if you set bFollowHmdOrientation/Position properties of CameraComponent or PlayerCameraManager classes to false and change the camera's orientation/position by yourself.

@artyom17

I was thinking about the rendering for a bit and as long as there is a reliable way to feed custom images to each eye, the specific implementation doesn't matter. The use case is mostly AR (e.g. Leap Motion camera passthrough) or custom 3d effects (faking render layers/portals or producing otherworldly effects). Currently this can be achieved via a post-process material with ScreenPosition variable used to split image sources based on left/right of the screen. In unity I believe you simply specify the image to the camera and the render order it should be applied, which seems like a cleaner form. But I agree with you that it is prudent to be conservative on this and if I find another more compelling reason I will revisit the issue.

I don't believe there is a single HMD that wouldn't have an accelerometer included, seeing as this is the primary way orientation is deduced, but even HMD specific api as you suggested would be great to have.

Thanks for clearing up GetCurrentOrientationAndPosition, seems like it would work as intended.

With any api changes, good documentation is key so we can follow the best practices and minimize confusion.

Cheers artyom17, keep up the good work!

"artyom17" wrote:
To change perceptible scale you should change World2Meters scale. Hmm.... Maybe, make this parameters of the Camera Component? So, each camera could have its own W2M scale?

I think that would be extremely useful, and it would resolve my unanswered question on the UE4 VR forum: https://forums.unrealengine.com/showthread.php?45567-Scale-Stereo-Camera-Separation

Thanks!

- Dave

Perhaps just make the camera respect the transform scale, like in the Unity plugin.

Then, if scale is 1, it uses the global W2M as you would expect, and we can scale up the whole actor and get the expected change in IPD.

A good point, I already thought about this.

"owenwp" wrote:
Perhaps just make the camera respect the transform scale, like in the Unity plugin.

Then, if scale is 1, it uses the global W2M as you would expect, and we can scale up the whole actor and get the expected change in IPD.

So long as you mean camera scale rather than actor scale, then that would work. Assuming that scaling the camera doesn't cause other side-effects in UE4, of course.

I wouldn't want it to be actor scale as my giant would still be a scale of 1.0 in-game, just its mesh and bounds would be larger than a normal human.

- Dave

"Gnometech" wrote:

"owenwp" wrote:
Perhaps just make the camera respect the transform scale, like in the Unity plugin.

Then, if scale is 1, it uses the global W2M as you would expect, and we can scale up the whole actor and get the expected change in IPD.

So long as you mean camera scale rather than actor scale, then that would work. Assuming that scaling the camera doesn't cause other side-effects in UE4, of course.

I wouldn't want it to be actor scale as my giant would still be a scale of 1.0 in-game, just its mesh and bounds would be larger than a normal human.

- Dave

Yes, totally we are talking about the camera scale rather than actor scale. This is a good question about side effects of changing camera scale. I tried it and it doesn't seem to create any issues, but probably should verify with epic guys.