> scaling to 1T parameters significantly enhances sample efficiency and performance ceilings;
Man, I find SOTA deep learning somewhat hilarious. We scale models to absurd proportions, burning through a shitload of resources just to achieve (slightly above) human intelligence.
The human brain has a few billion neurons and uses as much power as a light bulb.
Man, I find SOTA deep learning somewhat hilarious. We scale models to absurd proportions, burning through a shitload of resources just to achieve (slightly above) human intelligence. The human brain has a few billion neurons and uses as much power as a light bulb.
However, a neuron is much more than a single parameter. The brain is estimated to have from 10^14 to 5x10^14 synapses.