I have over 6.25 GB of something n my blank world and it will crash a headset.

The Meta devs are aware of this memory issue. What seems to happen is that the memory accumulates overtime, and exploring other worlds before visiting yours might affect the memory readings. 

Also, running other apps in the background (like if you have the browser open) can affect the memory metrics.