performance hit with multiple shadow sources

It sounds like the extra shadow pass is making you miss the 13.3ms deadline. It should show up if you attach the profiler. Can you mark the directional light as "not important" and/or simplify your shadow quality settings?

I have run into the same issues recently and I wanted to keep all my light sources with shadows. So the solution that worked for me was, Bake almost everything to light maps except dynamic assets. After bake, turn off all lights except one, choose one main directional light to maintain your shadows (Hard or soft). Create a layer called character lighting or something like that. Set your light, your characters and dynamic assets to that layer.
My scene now looked identical but I went from 45 frames back up to 80+ . Also look into your audio settings and compress what you can, I gained another 10 frames there.

thanks vrdaveb, will have a look and report back.

jmorris - sadly I can not bake my lights since none of them are static. but I can see where I cut my passes shorter :)

setting the shadow quality to low solved the issue for me!

Are you using the Forward or Deferred renderer? Using deferred I've had good luck with several dynamic shadow casting point/spot lights. You can tell how much of a hit you're taking by looking at the stats in the game view and looking at your tri/vert counts. They will go up dramatically as you add lights.