Restrict the movement of the object attached to the controller in Oculus Go

Hey,
I have written a script to pick an object with a rigidbody component by the ray casted from the controller and move it around. Now, I want to limit the movement of the object. After parenting the object to the controller, i want it to move in just one direction, e.g. along the x-axis only. I tried doing it by freezing the object's position in its rigid body component, however, it as no effect, when the controller is attached to the object.

If anyone could help me with this, I would greatly appreciate it!
Thanks in advance.