Unity 2019.3.7f1 build crashes

I have a project with 8 scenes in it. When I build the each scene individually, it works fine. But, when I connect the scenes and build, the first 3 work fine. When I get to the fourth scene, it loads but freezes on the frame it's on. Then I eventually get kicked back to the menu.
The scenes with less things in it load without any problem but when it gets to the bigger scenes, it crashes. 

The FPS drops quite low on the heavier scenes, could that be the problem?
Or is it possible the memory isn't being cleared when transitioning between scenes?

Does anyone know what the problem might be?