It seems that the criteria you mention at the start, and then with your last post, a bigger M car would be better. Put the F10 M5 on your list to view as well, and if budget allows it, the F90.
They are super powerfull cars, and super versatile, that you can take to a wedding, drop the kids of at school, or even a race day. They have the most creature comforts and extremely comfortable.
Going on a trip with buddies with an M5, would be way more enjoyable than an M3. More power, bigger space for suitcases, self closing doors, 360 cameras, full infotainment etc. Always look for a model wich was specced correctly from the factory.
Last thing to consider as well. With an M5 you will technically never need to modify anything (since you don't want to), the power is more than enough. Cosmetic you could do small tastefull changes.
Note: on the M5 everything is changeable from inside, including suspension stiffness, steering feel, pedal sensitivity etc.
I actually just forgot to put the F10 M5 on the list - you are right.
E90/92 M3 (added as per advice)
F80 M3
F82 M4 (added as per advice)
M140i
M135i (new to the list) - interior is a yes. Drive to be confirmed.
M240i (added as per advice)
F10 M5 (added as per advice)
I can though see this is also becoming a decision of will the new car be a weekend car (E90/92 M3, F80 M3/4, M240i, Porsche) and I keep the 320d OR become a replace the 320d car and be a new daily (M140i, M135i, F10 M5). Guys I know everyone is not going to agree with how I categorize the cars between weekend vs daily car - but please just bare with me and how I see things - I am taking heed of the advice as you can see how I've added cars from your suggestions.
Perhaps to clarify that on how I look at it:
E90/92 M3 - this would not be a car I would want to daily mostly based on age and running costs I would think. Remember this car was never on my list and I've not done much research on this one. And a door car simply cannot be a daily for me.
F80/82 M3/4 - for me personally, just too much of a nice car for me to drive everyday - and if it turns out to be an M4 the 2 doors stops is from being a daily for me period.
M140i - as per the feedback here and some consideration I think this can easily be a daily.
M135i - after looking at it this weekend I think it falls in the same place as the M140i - can easily be a daily and not a car I would want to keep in the garage.
M240i - 2 doors = weekend car.
F10 M5 - I am not so sold on this one - although I can see how really nice the interior is - will try.
I am sure you can see how my initial brief is evolving - but I think that is the point of this thread.