Consider expanding your shopping list. If you want a 6 pot look at the 330i as well but if you're concerned about fuel consumption, add the 320d and 330d to your list. The power figures on the diesels are incredible.
Anyway, I suppose it depends on what you want.
320i - comfortable, luxury vehicle but lacks power :nono: :bin:
323i - great engine note but not fuel efficient and lacks power for a 6 cylinder

fft:
325i - satisfactory power and great engine note, not fuel efficient :argh:
330i - powerful and great engine/exhaust note, not fuel efficient :excited:
320d - satisfactory power and great fuel efficiency, but that tractor sound :shakehead: (tuning potential)
330d - powerful and fuel efficient :thumbsup: (great tuning potential)
If it was my money, I would definitely get a 330d.