Well in the case of the VVT-i controller (if this is the same issue you have) it is indeed caused by a lack of oil pressure in the overhead gear. Nothing wrong as it does require oil pressure to function and this is not present when the engine is off.
In the words of Toyota with regards to the VVT-i mechanism:
From reports I have heard, it is this lock pin that that doesn't engage. Now whether this is because the "advanced angle assist spring" isn't doing it's job or the lock pin, I'm not sure. But at the end of the day, the controller can only advance or retard within set limits so the only wear that would be present is the contact between the grey and yellow parts shown in the attached diagram. For the brief amount of contact, I would imagine there would be no harm.
And here is what is actually looks like: