Multiplayer lags when host is not wearing headset

We are making a multiplayer game title. One player will host and the other player join as clients. 
When the host takes off the headset, the other players seem to experience lags badly. 
I am thinking that this is due to the game being paused at the host pc end, is there a way to prevent the game from being paused by default when the headset is not worn?
Thanks.