It depends on quite a few factors. If you're only looking at it from a fuel consumption perspective only it may not be worth it.
If you example you go to a 320d out of MP, your maintenance would negate any possible fuel saving. Also, don't compare fuel consumption figures on face value. If for example your 328i is averaging 15l/100km due to your current driving conditions, don't expect 5l/100km from a 320d just because someone gets that. It would however be lighter, but then again, your costs to get into a 320d/330d could negate the fuel savings.
Maybe provide a bit more info on what you are looking for and we can advise accordingly, as a 328i is a pretty decent motor for a petrol and it's fuel consumption is decent ... if you go to a E46 320d you may find it sluggish - a 330d is a different story though. Also, are you looking for manual or auto ....