Hotter intake charge temperature does make a difference, as Andy has noted... less dense air makes less power...
But that doesn't tell the whole story as to why turbo engines (and especially Diesel ones) suffer in hot conditions.
Diesel engine management doesn't work quite the same as petrol because of the different technologies employed... important differences between the two for dealing with heat are as follows:
Petrol engines have knock sensors, while diesels don't (as diesels "knock" all the time during normal running).
Petrol engines have adjustable ignition timing, which can be altered on the fly, IE - retarded when the knock sensor gets tickled.
If a petrol engine runs hot, and starts knocking, it's likely running lean - so once the knock sensor is triggered, not only is timing retarded, but fueling is enriched. Richer means it's running cooler, IE More petrol makes for a cooler burn, so the fuel actually cools down the cylinders and thus the engine. Both these factors rob a bit of power, but help the engine survive longer.
Diesels on the other hand, don't have ignition timing control per se - some will have heat chambers in the intake for starting the burn, but for the most part they're reliant on compression to initiate the burn. This means it can't manipulate timing as well.
Diesel has a lower burn rate, but is more compressible, which means you can get more of it in there.
The more diesel you add, the hotter the burn cycle (the opposite of petrol) - so theoretically, you could melt your motor if you add enough diesel.
So, with these factors in mind, a turbo diesel engine becomes a slightly more volatile creature than your average turbo petrol engine.
Their maps are tuned more conservatively, for very high boost, and moderate fueling, and have a load of safety parameters built in for reigning things in when ambient temperatures go hot.
Instead of using a knock sensor, the ECU looks at MAF airflow rate + relative intake temperature, MAP sensor pressures + charged air temperature, and most importantly, the EGT's (exhaust gas temperatures) coming out of the engine.
The most important scenarios based on the above:
For starters, obviously, the base map is selected based on the relative temperature and barometric pressure seen by the MAF.
If EGT's spike at any point, fueling and boost get cut down until EGT's resume within a safe range.
If there's a huge discrepancy between MAF temperature and charge temperature, boost gets dropped until that temperature differential falls within parameters.
So that means, while viscous fans and air conditioners do sap some power in summertime, the biggest factor in the event of sluggish performance would be a non-optimum running state in any of sensors or the charge tract.
Here are some things to look at to ensure a happy & nippy diesel in summer:
1. The MAF needs to be running 100% perfectly - if yours is older than a few years, then consider replacing it. They're not too expensive from Bosch directly. It measures the flow of air, and the ambient temperature / barometric pressure / humidity, thus controlling all of your compensation factors for altitude and atmospheric conditions.
2. The MAP sensor similarly needs to be working accurately - they don't really degrade like MAFs, but the occasional cleaning doesn't hurt, as being covered in caked-on oil and muck will throw off their readings somewhat.
3. EGT's don't really die either - but it won't hurt to do a check / calibration test.
4. As Andy has said, clean the intercooler - or fit a bigger one, like the stock N54 unit or the likes - everything you can do to cool your charge temperatures as effectively as possible, means that both the MAP sensor and EGT sensor will see cooler temperatures, which means more performance before they call a halt to your fun.
5. Methanol/water injection is handy for making more power, as well as keeping your EGT's down due to the inherent cooling effects... use it wisely though, as too much of it will overspin and prematurely kill your turbo.
Hope that helps.