Microgpt in pure C hits 10M tps on Apple m5

 (github.com)

43 points | by dhorthy 1 day ago

3 comments

  • ilaksh 42 minutes ago
    This is not an LLM obviously , it's just for generating random names. But interesting to think of the possibilities of truly tiny language models if there were connected together.
    [-]
    • api 8 minutes ago
      Isn't a MoE model basically a cascading tree of smaller models or some variation of that?
    • dcow 32 minutes ago
      Is token rate a function of parameter size?
  • throwa356262 1 hour ago
    And the 5 years old AMD Ryzen 5 5600H is doing 7M?

    Am I reading this right? Then I need to try this on Strix Halo

  • fwip 48 minutes ago
    Model is 4K parameters - I don't know enough about that size of model to know if this impressive or not.
    [-]
    • altcognito 29 minutes ago
      It's a trivial example. This won't be useful outside of a VERY specific domain without more parameters. Many people need to know about the bitter lesson.

      https://en.wikipedia.org/wiki/Bitter_lesson

      Over time, I'm sure we'll be able to filter information better and get parameter counts down, but I wouldn't count on that within the next 6 months.