From within Recroom, open your watch menu, press the "people" button, select the friends tab, select the person you want to remove, then press "unfriend".
This will remove them from your Recroom friends list, not your Oculus friends list (if they are on both). To remove friends from Oculus is a similar process, but from the Quest home menu.