Major lesson from when I worked on Google Search indexing is that queues have a lot of hidden complexity and can make your outages much longer than they need to be. We had a big project to get rid of a bunch of queues by just scaling up our synchronous backends and making them faster.
I haven't modeled it, but I wonder how far you'd get on randomizing the policy choice for concurrency limit 1. Maybe weighted by past results, but bounded to allow it to shift instead of falling permanently into a basin.
6 comments