Spherical Panoramic Stereo Image Viewer

I am attempting to author a light field viewer using pre-rendered images from my own physically-based renderer. My pre-rendered images are spherical panoramic images of the 3D scene taken from different viewpoints. I will be interpolating 2 images depending on the user position, one for the left eye and one of the right eye however I am unsure of how to go about the viewing of the final image itself using Unity.

Can I map each image to a sphere one for the left and one for the right eye? and if I do so how do I prevent the right eye from rendering the left eye sphere?