Connecting Glasses to Two phones.

I have the Oakley HTSN glasses, and I need them to work with two phones: one for work calls and one personal one for video, photos, and music when I'm off the clock.

My problem is this: I can't easily switch between the two. If I turn off Bluetooth on the first phone and pair the glasses with the second, they refuse to connect back to the first one later.

How do I get these things to reliably switch between both phones?