This all boils doing to personal preference and opinion.
For me, the E36 M3 "stirred" me the most. American model aside, when the E36 M3 was launched it brought real super car performance to the man in the street. It was a significant step up compared to everything else in the price range and certainly moved the proverbial goal posts.
Hence, even though we could argue that the E46 was an improved model and its engine won many awards, from a market perspective it simply did not have the same impact, as great a car as it is. By this time, competitors were all offering other models and the M3 lost its sparkle. However, take a look at its 2001 competitors (when the E46 m3 was launched). In the day, journalists argued about which was better. BUT in true M tradition, it's still the M that truly won at the end of the day - how: Go ask any Merc fan if he still wants the C32 or Audi fan if he wants the (old) S4. Now ask a BMW fan if he wants an E46 M?
So I'm sure each BMW fan has their preference, but at the end of the day every generation of M had something special to offer. It may have not realized at the time, but in future it might... Lots of people look "down" at the E90 M, but in a few years we may hail it as the last true NA M. Who knows.