Judder in my Unity project

Been banging my head against the wall with unity trying to optimize the scene. It seems very hard to get the scene to look decent and at the same time get 75 fps. Deferred rendering seems to be unusable at the moment. I used a fix to get shadows to work with oculus cameras, but there is still some weird stuff happening with the lights. Right now I have setup where I use realtime lights with forward rendering and lightmaps that only have bounce lighting. I can achieve 75 fps in most places, but the bigger open room drops below if I add a Dalek model in there. The Dalek model in not very optimal for realtime stuff. I suppose I could optimize it quite a bit, but I have no idea how much it would help just to reduce the polycount. The problem is that I have very little knowledge on what helps and what doesn't when it comes to real time graphics in Unity. I use offline renderers for work stuff and I understand those rules a lot better than these.

I wish I had Unity 5 beta version so I could test the new stuff in there. It's kinda weird since I see many people testing it, but when I asked Unity support they couldn't tell me when the beta would be available. I'm a Unity subscriber so I should have access to the newest version when they become available.