I think that you are correct. If I remember correctly, the 2004 / 2005 E46 320D included a turbo change from previous 320d's which resulted in better reliability.
however things are not as gloomy as it might seem. Maintain the car properly, and perhaps plan on a turbo replacement (not reconditioned), as a preventive maintenance project, which should set you back around R10k (whilst not cheap when considering the effect on the wallet, it is relatively cheap considering the costs that a damaged turbo can cost you)... and you can have many years of trouble free motoring.