HandRayInteractor not working for Left Hand

I am working on Interaction SDK in Unity. I am using Synthetic Hands with HandRayInteractor to use ray interaction to interact with UI.

Using the right hand, I am able to interact with the UI as expected (using ray interaction). But the ray in the left hand is not working as expected. I am not able to select and scroll UI elements using ray interaction. Both (Left and Right) ray interactions are setup in the same way. 

Please help me understand if I am doing any mistake here.