Friday, 8 February 2008

Un-Scheduled Tasks

Just had a weird problem with one of my servers running scheduled tasks twice.

Sometimes, the backup job would run twice, and sometimes the job that emails me the backup report would run twice. When ever a job was duplicated, the logs showed that it ran several minutes early.

Most of the time though, they both only run once, as they should.

I had no idea what might be causing this.

I checked the scheduled task logs, even ran the AT command in case something had been put in there, but nothing.

It was only once I got round to looking in the obvious place of the event logs that I saw loads of w32time errors....

What was happening was that the server was failing to synchronise with the domain controller for long periods and the clock would drift until it managed to reconnect. If this happened while the backup ran then the clock would get shifted backwards and the scheduled task would run again.

Why the server is having trouble connecting to the NTP server, and why the clock is drifting so much and why it seems to resynchronise during the backups, are entirely different questions of course.

I suspect it is because this server is a virtual machine running on a heavily loaded physical server, but further work is happening to rectify this so we'll see then if the time problem goes away too.