Entitlement check - what does it check?

We're trying to implement the entitlement check in a way that we can quit the app if the test fails. The game is on a beta channel and I've invited a user to this channel (not registered as a dev for the organisation).
The script used (Oculus sample code) returns "success" if this player runs the game. If I remove the player from the channel the check still returns success which isn't what I expect from the check. What are the rules and conditions for the entitlement check?

Simon