I've driven the Toyota badged chassis in the UK not so long ago, and it is very very good, but the whole package is slightly under-powered. The handling is very intuitive and direct, the narrow standard tires and loose ESP allow good amounts of slide before the electronics safety net takes over, but the engine is just a tad power-shy to keep the tail out for long and forget about smokey doughnuts. I've heard a rumor that Toyota and Subaru are developing a more powerful engine, but I could be wrong. This chassis should be a hit with aftermarket tuning. I would say it's a good overall package that should prove easy to live with.
I can't speak for the E36 M3 as I have never driven one but I expect the M3 to be a few levels above the BRZ. My heart would go with an unabused M3, if you can find one. Good luck.