The corresponding concept in gears is to avoid meshing two gears unless the number of teeth are co-prime, because this will cause uneven wear patterns. If the tooth counts are coprime, then each tooth will contact every tooth on the other gear the same number of times, effectively "wear leveling" the teeth.
In this instance, one fix is to fire the timer every 59 seconds, so it will naturally drift out of phase with the minute hand of the clock. Or, if work needs to be done exactly every 60 seconds, then WAIT(RAND(60)) seconds for the timer initialization, so if 10,000 units start simultaneously they don't try all at once.
This is why many systems will do automated updates every 17 hours: 17 is coprime with 24.
Classic "thundering herd" mitigation tactics.
Since this problem is news, I don't think anyone even knew this was something to avoid! We do have fixes, however.
I know my older APC UPSes do weekly self tests, but I guess newer equipment does more frequent self testing? And of course they're all internet-connected meaning clocks are within a few hundred ms of each other.
It also made me think about how many GitHub Actions I have that kick off on Sunday morning at a specific time. I try to vary the time/day for those jobs but I'm guessing there are enough things that kick off at specific time intervals to make an impact at the top of an hour, or at :15, :30, :45...
This is why systemd gives you the RandomizedDelaySec option. (Other cron-like facilities may have similar features, I'm not discussing them here). Set that to N and each time the timer reaches its trigger (say, once per day), its actual start will be up to N seconds after the trigger. E.g. if a once-per-day trigger is at midnight but you have RandomizedDelaySec=300 set, then it will run sometime between 0:00:00 and 0:05:00, with a different actual start time each day.
If enough people used that, with a high enough value of N (say 1800 for jobs that really don't have to start at any particular time), it would greatly spread out the spikes in accidentally-coordinated usage.
In this instance, one fix is to fire the timer every 59 seconds, so it will naturally drift out of phase with the minute hand of the clock. Or, if work needs to be done exactly every 60 seconds, then WAIT(RAND(60)) seconds for the timer initialization, so if 10,000 units start simultaneously they don't try all at once.
This is why many systems will do automated updates every 17 hours: 17 is coprime with 24.
Classic "thundering herd" mitigation tactics.
Since this problem is news, I don't think anyone even knew this was something to avoid! We do have fixes, however.
It also made me think about how many GitHub Actions I have that kick off on Sunday morning at a specific time. I try to vary the time/day for those jobs but I'm guessing there are enough things that kick off at specific time intervals to make an impact at the top of an hour, or at :15, :30, :45...
If enough people used that, with a high enough value of N (say 1800 for jobs that really don't have to start at any particular time), it would greatly spread out the spikes in accidentally-coordinated usage.