Over the last month I contacted Support for the first time in many years.
This was for a question about how billing works.
It went like this;
1. Case created.
2. Unassigned for seven days.
3. Open real-time chat, talk for 25 or so minutes where I guide a first-line Indian chap who plainly doesn't know about the subject in hand and who is as we talk reading the AWS docs I've already read. At the end, just as I couldn't find an answer, he couldn't - which is good, he didn't try to give me the wrong answer - he escalates. That's fine - a lot of questions are simple and even silly, and first line support is there to handle them - but they could have done all this without me, if they'd opened the ticket themselves rather than me having to chase.
4. Eleven days later, comes back with exactly the wrong answer. In the meantime, I had figured out the correct answer, and reply, explaining it to him.
5. Next day, I get a wall of plainly AI generated text telling me my answer is correct.
It seems to me a key issue here relating to AI generated text, is a misunderstanding on the part of AWS that I as a consumer will value that answer exactly (or indeed, even remotely) as I would value the answer from a human.
I do not. I almost ignore AI generated text, as I think it as unvalidated response.
AI "support" bots that just attempt to read the published documentation for you are possibly the most annoying thing to have come out of the current AI plague.
Even Stripe - once legendary for the quality of its support - has apparently given up now. I had to deal with it recently over a case where the merchant was seeing an unexpected change in the way it was collecting payments and the AI bot was worse than useless - it actively suggested incorrect explanations and resulted in several days of trying to change the wrong things while the problem persisted.
For my own businesses we give this issue a heavy weight when choosing which services to use. We have even seriously considered moving existing integrations to different services over this one issue recently. If we're integrating with a service then we want to know there's a real person who can actually help if we have questions or anything goes wrong. Failing to provide that because it's cheaper to push everyone through the AI bot is a statement of intent about how much you value your customers.
> I guide a first-line Indian chap who plainly doesn't know about the subject in hand
Out of curiosity, why is it relevant that this person is Indian? AWS employs a lot of Indians on their actual product and engineering teams, who have built AWS, and surely do know their own work well. Isn’t this just an issue of support mostly being lesser-paid people who work off scripts, rather than of race?
Just happens that these very-lesser-paid workers are usually Indian, Filipino and Moroccan, I guess. And if a company goes this route, they're most likely significantly cutting costs at the expense of the provided service
Indeed, why is it relevant? We should ask companies like Amazon, because they're the ones who choose to outsource customer support to India specifically 99% of the time.
westerners are loaded with the context that much work is outsourced to india for profit purposes. is indian a race? if it isnt a race, why are you making this about race?
I used to see AI generated images with lots of unintelligible writing or misspelled words in slides, but the speaker left them in anyway. “Good enough” is not customer obsession.
This enforced adoption of immature GenAI reminds me of Milo Minderbinder trying to make people eat cotton in Catch 22, because he had inadvertently obtained a huge amount of it.
I don't know if there is another industry that behaves this childishly. There might be. But good grief, how much more juvenile can ours possibly get? AI-generated images with obviously nonsensical text is something I never thought I'd see in professional meetings. But it is becoming more and more common.
Long before GenAI, I saw people using meme generators a lot in corporate presentations. I found that equally jarring. Replacing that with GenAI stuff is probably an improvement. At least it's reducing legal risk. It seems more understandable to a global audience, too.
I still don't have an explanation why people are doing this. Is it part of leadership training? Or do presenters have their own theory that including this stuff makes the presentation more memorable and enjoyable?
I think a key goal of senior management at any big company in the last 6 months is to make rank and file fungible or obsolete. It’s one big experiment. There are precedents like the Industrial Revolution. Things get worse for the workers for a generation or so.
> There are precedents like the Industrial Revolution. Things get worse for the workers for a generation or so.
And things only got better post-Industrial Revolution when labor organized and forced the issue.
There's no guarantee that will work again if labor has reduced leverage due to AI reducing their value.
I think in one way or another this all works itself out, but I'm not convinced it won't be a very painful (and possibly violent) transition to whatever comes next.
I am also increasingly worried by the potential for violence here. This is a social experiment that is harming the daily lives of millions of people in very obvious ways already. The environmental costs for the data centres are not insignificant. The economic damage from allowing AI to have so much funny money when it doesn't make much real money to justify it could be disastrous on a generational scale. Governments aren't making any serious attempt to regulate and if anything are drinking the Kool-Aid. We might be on a path that literally collapses the established Western capitalist order within a generation but historically societal change of that scale usually has a body count and I have no idea what comes afterwards.
The actual Industrial Revolution labor wars happened because workers were being maimed, killed, and disposed of with zero legal recourse. The Ludlow Massacre in 1914 ended with the Colorado National Guard machine-gunning a tent colony and burning women and children alive. The Battle of Blair Mountain in 1921 had the United States Army bombing American coal miners from biplanes. Pinkertons routinely shot organizers. The Triangle Shirtwaist Fire killed 146 garment workers because management locked the exit doors to prevent unauthorized breaks. Coal miners were paid in company scrip redeemable only at company stores in towns the company also owned and policed. Black workers attempting to organize in the South were lynched. Children were maimed in textile mills.
A software engineer getting four months severance after a layoff exists in a different universe from this so no. There is no precedent. Don't you dare talk about the industrial revolution because its not even in the parking lot of the ballpark.
I'm also an AWS alumni from many years back now, and truthfully, the organizational problems really took off when Jassy moved to being CEO of amazon as a whole and major leaders left the company (Charlie Bell, et al.).
There were always other problems too, pressure on the company in both directions across many different product lines on both cost (any number of cheaper baremetal providers who are much faster at providing customers instances than they were a decade ago), and product quality (any number of startups to now bigger companies, databricks probably being the biggest success) along with a number of expensive bets that were made that didn't work out especially as interest rates began to rise (there were numbers of of different services ranging from IoT, AI, business support, robotics, groundstation, that essentially all failed).
AI infra being their latest bet, along with doubling down on custom hardware is smart, but these roles don't require the same number of SWEs and instead require a different type of high skilled professional.
Unrelated to your main point, but it's "alumnus" in the singular form. For bonus language nerd points, you would use "alumna" to refer to a woman, or "alumnae" to refer to multiple women. Not sure how Latin handles mixed gender groups, though I would guess it's "alumni".
I also joined in 2022, and it aligns so much with my experience. Good manager that moves on, then a gradual erosion of "insist in the highest standards" towards a dreaded "good enough", GenAI only accelerated it IMO.
AWS lost its way. S3, SQS, EC2 and VPC were great innovations and those services were done by a bunch of engineers who wanted to have a reliable elastically scalable system. This was coincidentally cost effective at the same time. What came after especially the data stack and now the AI services were done by a MBA heavy management team who does not understand innovation and treats engineering like a bank does: putting it in the cost category. Recent financial results show the impact: Google grew almost twice as much as AWS did. Maybe it is just coincidence.
The story about recovering the account rings very close to me. At least they had coworkers cheering for him, I feel teams are shrinking so much that we'll end up with just the LLM of choice to pat our back with "good work" and "you're absolutely right"
Co-workers cheered while managers were sharpening their axes. One doesn’t do such “heroics” without approval, making the system look incompetent and broken and then apologizing for it without being decapitated in the public square for everyone to learn the implicit lesson. Anyone cheering for him publicly should watch their back, too.
The economy sucks, and they do pay decently for software engineers. Especially now that the rest of FAANG aren't massively over-paying for college hires, I doubt the supply of bright young minds will ever entirely dry up.
I have worked there for 8 years now and we ARE running out of bodies.
It has become extremely difficult to hire at any level, we have had an open position on my team for a senior data scientist for a year and a half now, with barely any candidate applying, and none of them being competent.
Similarly the average level of new employees has dropped dramatically. The famous "hiring bar" is now below ground.
There are now more highly competent devs ready to work for cheap available now than ever before and all of them are boosted with state of the art coding agents…
It‘s the golden age for software engineer employers.
Absolutely. With his name in the public and apologizing to the customer for sheer internal incompetence. Then also cheered on internally.
I bet as the managers publicly nodded in praise for his heroic act, their hands were already typing his name to be sent to HR for “get this guy out of here on any excuse you can” note. (In reality it would be a nonverbal hint of sorts. Nothing to leave any trace discoverable by lawsuit)
Meh you always read one side of the story. I have seen a lot of people getting PIPed out of Amazon and only one or two didn't deserve it, yet all claimed they didn't.
All the AI hype aside, I wonder if there is a way to avoid becoming one of these faceless corporations where customers are just numbers. For years Amazon has been fantastically customer centered, but at some point they just lost it. I could compile a list where Amazon is actually way more customer unfriendly than in the past now, but I guess everybody already got their own anecdotes about that. So what exactly went wrong and how could that be avoided at other companies?
The account recovery story says a lot. At some size, companies start handling people as tickets. Sometimes it only gets fixed because one person inside still cares.
Our company also requires everyone to use more AI-related tools, and I don't think there's anything wrong with that. But the quality of work produced using these tools really depends on the individual's ability. Some people don't put in much effort, and the results they produce are really sloppy, which bothers me a lot.
I think there is actually something wrong with that. What should matter is the work produced, not the tools used to produce it. If AI tools really are all they are cracked up to be, then people using them will get ahead, and the company can justifiably point out "your peer gets twice the work done as you" to the other employees. But mandating tool use in and of itself is senseless and counterproductive.
This genai is going to bring about huge quality drop in software across the stack and across the domains. I already see orgs that had reasonable software processes transform into orgs where the only metric is how much generated code you can slap and slop together and how fast. There's no success here for anyone.
And this is not a dink on the ai tooling itself but on the organizationan processes that provide the context in which the AI code generation is being used.
Bad processes will always produce bad low quality outcomes regardless of tbe technology.
> Amazon has a really odd viewpoint when it comes to the people who work there. They view almost all employees as “fungible”.
Hardly an Amazon-only thing. In fact, enterprises need this mindset, because people moves on, retires, or just suddenly die. With that said, due to its late-stage capitalistic ethos, Amazon is just too overly gleeful about this tasteless reality of life. It's the equivalent of a nephew coming to an aunt's funeral and shouting "A week ago, I told her everybody dies! And now she did! Wasn't I right??? Everybody dies!"
> Also, last year the focus at AWS turned fully and almost desperately toward GenAI.
I wonder if I'm being too cynical, but late-stage capitalism companies also love profiteering, and the mere prospect of firing all those pesky workers and not having to pay their salaries is like cocaine to those organizations. Which is why I think Amazon fulfillment centers will at some point rent robots at a price point between 2x and 3x their current human labor costs, in the hope that it will eventually make economic sense.
> In this whole pivot to GenAI, AWS has lost its focus on the customer. Instead of working backwards from a genuine customer need, the goal seems to be to create as many things as fast as possible, throw them into the world and see which ones gain traction, whether or not they serve a real need.
AWS has been this way for a lot longer than GenAI, since the basic infrastructure products were built out early on. But when I read this line about throwing things out there quickly, I also think of Google and even Anthropic. Google has a long list of products that got created and killed, as part of their internal politics and promotion culture. Anthropic is currently rushing vibe coded slop all the time to try and win over OpenAI and set up their IPO.
Maybe all the rich high funding companies can afford to this and maybe it is the right thing for them to do. They can afford to make big mistakes without hurting their stability. A true startup or smaller company can’t - they would shutdown because one big investment that fails is enough to destroy the whole company.
to be fair, even though they have "working backwards" and "customer obsession", amazon has always been about making lots of different experimental bets. Bezos:
> To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there. Outsized returns often come from betting against conventional wisdom, and conventional wisdom is usually right. Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten. We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs. The difference between baseball and business, however, is that baseball has a truncated outcome distribution. When you swing, no matter how well you connect with the ball, the most runs you can get is four. In business, every once in a while, when you step up to the plate, you can score 1,000 runs. This long-tailed distribution of returns is why it’s important to be bold. Big winners pay for so many experiments.”
> Maybe all the rich high funding companies can afford to this and maybe it is the right thing for them to do. They can afford to make big mistakes without hurting their stability. A true startup or smaller company can’t - they would shutdown because one big investment that fails is enough to destroy the whole company.
Both are following the same strategy. Amazon has a $2.86 trillion market cap. That's the equivalent of 143,000 $20 million Series A startups. Companies like Amazon and Google are basically an integrated herd of cash cows plus a VC portfolio.
“I have to say being fired from AWS is actually a relief. There have been a lot of changes to the company since I joined in 2022, and the company I wanted to work for is no longer the same company.”
Many storied companies can be described this way. It’s a shame. Have any companies hit such scale and kept the ethos and magic of before? Is it inevitable for companies to enshitify themselves in the pursuit of their shareholder’s goals?
Not possible once big parts of the company start not knowing other big parts of the company and the company also has a board of directors that must increase shareholder value at all costs.
> Long story short, I was able to get his resources restored. All I did was manage to poke the right bear and the support team did the rest of the work (and they were amazing).
No they utterly failed and needed a special non fungible employee to get them to do their job.
I was enjoying the article and then he makes some of the most bizarre claims about what cloud did and how we had to provision servers
If any of you young'uns read this, that is not how we had to do provisioning before cloud.
VMs already existed before AWS came out. You could already provision a new server usually in minutes and rent it month to month.
In fact, all the existing VM server companies had to start calling themselves cloud companies because pointy haired bosses couldn't understand what cloud really was.
And they were cheaper than renting AWS. MUCH cheaper. They still are.
The original point of AWS is that could scale according to demand. Have 10 VMs running at lunchtime and 1 VM running at midnight.
But using a cloud VM also required less server admin experience. It was a bit easier and came.pre-configured with things like firewalls.
And THAT is what ended up being the USP of cloud hosting. Especially when they started rolling out all the SQL as a service, redis as a service, etc.
You didn't need to really understand servers to run a server, and it turned out almost all developers really didn't want to understand servers. TBH, I don't, server admin sucks. Right now I'm working somewhere where I have to think about SSL certs occasionally and I consider it a complete waste of my life.
Digital Ocean came out like 5 years after AWS, what was revolutionary about that wasn't that you could spin up VMs quickly, it was the price. VMs went from $20-30 p/m to $5.
For developers who weren't SV rich, that meant you could run a side project without it being a significant cost.
I'm glad to see that one core amazon principle has endured the 10 years since I worked there, even if none of the actual leadership principles have survived /s
This was for a question about how billing works.
It went like this;
1. Case created.
2. Unassigned for seven days.
3. Open real-time chat, talk for 25 or so minutes where I guide a first-line Indian chap who plainly doesn't know about the subject in hand and who is as we talk reading the AWS docs I've already read. At the end, just as I couldn't find an answer, he couldn't - which is good, he didn't try to give me the wrong answer - he escalates. That's fine - a lot of questions are simple and even silly, and first line support is there to handle them - but they could have done all this without me, if they'd opened the ticket themselves rather than me having to chase.
4. Eleven days later, comes back with exactly the wrong answer. In the meantime, I had figured out the correct answer, and reply, explaining it to him.
5. Next day, I get a wall of plainly AI generated text telling me my answer is correct.
It seems to me a key issue here relating to AI generated text, is a misunderstanding on the part of AWS that I as a consumer will value that answer exactly (or indeed, even remotely) as I would value the answer from a human.
I do not. I almost ignore AI generated text, as I think it as unvalidated response.
Even Stripe - once legendary for the quality of its support - has apparently given up now. I had to deal with it recently over a case where the merchant was seeing an unexpected change in the way it was collecting payments and the AI bot was worse than useless - it actively suggested incorrect explanations and resulted in several days of trying to change the wrong things while the problem persisted.
For my own businesses we give this issue a heavy weight when choosing which services to use. We have even seriously considered moving existing integrations to different services over this one issue recently. If we're integrating with a service then we want to know there's a real person who can actually help if we have questions or anything goes wrong. Failing to provide that because it's cheaper to push everyone through the AI bot is a statement of intent about how much you value your customers.
Out of curiosity, why is it relevant that this person is Indian? AWS employs a lot of Indians on their actual product and engineering teams, who have built AWS, and surely do know their own work well. Isn’t this just an issue of support mostly being lesser-paid people who work off scripts, rather than of race?
This enforced adoption of immature GenAI reminds me of Milo Minderbinder trying to make people eat cotton in Catch 22, because he had inadvertently obtained a huge amount of it.
I still don't have an explanation why people are doing this. Is it part of leadership training? Or do presenters have their own theory that including this stuff makes the presentation more memorable and enjoyable?
And things only got better post-Industrial Revolution when labor organized and forced the issue.
There's no guarantee that will work again if labor has reduced leverage due to AI reducing their value.
I think in one way or another this all works itself out, but I'm not convinced it won't be a very painful (and possibly violent) transition to whatever comes next.
A software engineer getting four months severance after a layoff exists in a different universe from this so no. There is no precedent. Don't you dare talk about the industrial revolution because its not even in the parking lot of the ballpark.
There were always other problems too, pressure on the company in both directions across many different product lines on both cost (any number of cheaper baremetal providers who are much faster at providing customers instances than they were a decade ago), and product quality (any number of startups to now bigger companies, databricks probably being the biggest success) along with a number of expensive bets that were made that didn't work out especially as interest rates began to rise (there were numbers of of different services ranging from IoT, AI, business support, robotics, groundstation, that essentially all failed).
AI infra being their latest bet, along with doubling down on custom hardware is smart, but these roles don't require the same number of SWEs and instead require a different type of high skilled professional.
Unrelated to your main point, but it's "alumnus" in the singular form. For bonus language nerd points, you would use "alumna" to refer to a woman, or "alumnae" to refer to multiple women. Not sure how Latin handles mixed gender groups, though I would guess it's "alumni".
I personally think it’s not worth it.
See https://en.wikipedia.org/wiki/Latin_declension
I also joined in 2022, and it aligns so much with my experience. Good manager that moves on, then a gradual erosion of "insist in the highest standards" towards a dreaded "good enough", GenAI only accelerated it IMO.
It has become extremely difficult to hire at any level, we have had an open position on my team for a senior data scientist for a year and a half now, with barely any candidate applying, and none of them being competent.
Similarly the average level of new employees has dropped dramatically. The famous "hiring bar" is now below ground.
It‘s the golden age for software engineer employers.
I bet as the managers publicly nodded in praise for his heroic act, their hands were already typing his name to be sent to HR for “get this guy out of here on any excuse you can” note. (In reality it would be a nonverbal hint of sorts. Nothing to leave any trace discoverable by lawsuit)
And this is not a dink on the ai tooling itself but on the organizationan processes that provide the context in which the AI code generation is being used.
Bad processes will always produce bad low quality outcomes regardless of tbe technology.
Has that changed, or is it the non-AWS part of Amazon?
Hardly an Amazon-only thing. In fact, enterprises need this mindset, because people moves on, retires, or just suddenly die. With that said, due to its late-stage capitalistic ethos, Amazon is just too overly gleeful about this tasteless reality of life. It's the equivalent of a nephew coming to an aunt's funeral and shouting "A week ago, I told her everybody dies! And now she did! Wasn't I right??? Everybody dies!"
> Also, last year the focus at AWS turned fully and almost desperately toward GenAI.
I wonder if I'm being too cynical, but late-stage capitalism companies also love profiteering, and the mere prospect of firing all those pesky workers and not having to pay their salaries is like cocaine to those organizations. Which is why I think Amazon fulfillment centers will at some point rent robots at a price point between 2x and 3x their current human labor costs, in the hope that it will eventually make economic sense.
AWS has been this way for a lot longer than GenAI, since the basic infrastructure products were built out early on. But when I read this line about throwing things out there quickly, I also think of Google and even Anthropic. Google has a long list of products that got created and killed, as part of their internal politics and promotion culture. Anthropic is currently rushing vibe coded slop all the time to try and win over OpenAI and set up their IPO.
Maybe all the rich high funding companies can afford to this and maybe it is the right thing for them to do. They can afford to make big mistakes without hurting their stability. A true startup or smaller company can’t - they would shutdown because one big investment that fails is enough to destroy the whole company.
> To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there. Outsized returns often come from betting against conventional wisdom, and conventional wisdom is usually right. Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten. We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs. The difference between baseball and business, however, is that baseball has a truncated outcome distribution. When you swing, no matter how well you connect with the ball, the most runs you can get is four. In business, every once in a while, when you step up to the plate, you can score 1,000 runs. This long-tailed distribution of returns is why it’s important to be bold. Big winners pay for so many experiments.”
Both are following the same strategy. Amazon has a $2.86 trillion market cap. That's the equivalent of 143,000 $20 million Series A startups. Companies like Amazon and Google are basically an integrated herd of cash cows plus a VC portfolio.
Many storied companies can be described this way. It’s a shame. Have any companies hit such scale and kept the ethos and magic of before? Is it inevitable for companies to enshitify themselves in the pursuit of their shareholder’s goals?
No they utterly failed and needed a special non fungible employee to get them to do their job.
If any of you young'uns read this, that is not how we had to do provisioning before cloud.
VMs already existed before AWS came out. You could already provision a new server usually in minutes and rent it month to month.
In fact, all the existing VM server companies had to start calling themselves cloud companies because pointy haired bosses couldn't understand what cloud really was.
Where could you rent VMs in 2006?
IIRC there were two ways to run stuff, get your own server or get an account on a big shared computer.
Linode definitely had something along those lines.
Amazon won on APIs and overall integration but VMs were around already.
I remember the story really well as this is when i joined the workforce as a young GNU/Linux fan.
And they were cheaper than renting AWS. MUCH cheaper. They still are.
The original point of AWS is that could scale according to demand. Have 10 VMs running at lunchtime and 1 VM running at midnight.
But using a cloud VM also required less server admin experience. It was a bit easier and came.pre-configured with things like firewalls.
And THAT is what ended up being the USP of cloud hosting. Especially when they started rolling out all the SQL as a service, redis as a service, etc.
You didn't need to really understand servers to run a server, and it turned out almost all developers really didn't want to understand servers. TBH, I don't, server admin sucks. Right now I'm working somewhere where I have to think about SSL certs occasionally and I consider it a complete waste of my life.
Digital Ocean came out like 5 years after AWS, what was revolutionary about that wasn't that you could spin up VMs quickly, it was the price. VMs went from $20-30 p/m to $5.
For developers who weren't SV rich, that meant you could run a side project without it being a significant cost.
I'm glad to see that one core amazon principle has endured the 10 years since I worked there, even if none of the actual leadership principles have survived /s