I feel there is some good advice in this thread. End of the day, you are the one who will be parting with money and driving the car so you need to be happy with the choice you make.
The newer M cars may be special when compared to a non M BMW, however they're still full of creature comforts and as Llew said, they are built to be driven daily and one can do so. An M car and 320d I feel can make sense, when you just need to do a boring commute in traffic go for the 320d, or when the roads are worse off you get in the 320d. When you want to have more fun on your commute and it is outside of traffic or on the weekends, get in the M car. However you can still do the boring, stuck in traffic commute with an M car very comfortably and you can still do running around to the shops and back in an M car without issues. If you go the route of an M car, you may find yourself driving it more than the 320d as time goes on - and that is perfectly fine.
Owning a nice car which is fun to drive, you likely want to drive it more and enjoy it more often. It is a fair bit of money to spend on a car, might as well enjoy it and get your moneys worth out of it! As you can see, I am not a collector. I want to enjoy what I have and not just look at it. lol
An M lite car, I would find it hard to justify owning a 320d and a 35i / 40i and paying for both and maintaining both... Unless the one car was for the wife and one for me. They're a bit too similar, one just has more power. If you are leaning in this direction, to me and in my mind, it makes more sense to sell the 320d and just get a 35i/40i (whichever you desire, seems to be the 140i). I do not feel it is that much more special over a 320d, pretty much the same as your 320d just sounds a bit better and feels better when you put your foot down.
In my mind, if you want a weekend warrior from BMW then an M car makes sense - or something else, perhaps a Porsche. Not sure what else tickles your fancy.
You may find you end up driving the M car more often as well, if you are worried about damage to the vehicle such as stone chips, put on some protective films.
I think what TPB88 said is true, you have the 320d for the boring life stuff and it is a sensible choice in terms of cars and having a family - fuel efficient, cheap to maintain, not excessive in anyway. Great all rounder.
You want something else to compliment that and I do not feel a 35i / 40i does that.... It does all of what the 320d does, but with more power and a higher fuel consumption. Hence why an M car makes more sense.
... or a sportier looking 2 door such as an M2 / M2 comp, BMW Supra, M4