It could be some oil seeping out the turbo's bearings into the turbine (exhaust) area. As things heat up they expand and the leak stops (that's the theory anyway). Valve guides are similar although they usually provoke smoke after idling.
One culprit it might be is the blow-by oil, condensing in the intercooler and drops are getting blasted into the intake manifold as the car is cold. Once the engine (and everything else in the bay) warms up there's less oil in the intercooler because it's remaining as a vapour for longer and just being burned normally.
First thing I'd do is clean the intercooler, then check the turbo's main shaft for any wobble (it shouldn't). You might also check the flexible hose near the baffle (it's a sound-reducing device between the turbocharger and the intercooler, mounted on the side of the engine). Remove one hose clamp, get the hose off and wipe a clean finger on the inside of the hose then examine the oily residue under a strong light. Any sign of metal flakes means the turbocharger is suspect.
