Show HN: Reame – a CPU inference server that gets faster as it runs

 (github.com)

22 points | by targetbridge 4 hours ago

5 comments

  • K0IN 22 minutes ago
    Sick that you can get 2 arm cores and 12 GB ram for free at Oracle cloud, did not know that
  • JPLeRouzic 1 hour ago
    Thanks for sharing your work!

    Please how to select the model? I downloaded tinyLlama, put it in ./models, changed reame.conf but I get:

    (No such file or directory)

    Otherwise putting the model in /opt does not please me much, I fear to forget a model is there, if it is in reame folder its much easier to notice and manage.

  • ahepp 1 hour ago
    If a prefix cache is good, why not keep it in primary mem?
  • ohadkr 2 hours ago
    The persistent KV cache is interesting; I’d love to see how much of the speedup remains when requests share less structure.
  • tyzoid 2 hours ago
    Why qwen 2.5 everywhere? Why not 3.5?
    [-]
    • TheGoddessInari 34 minutes ago
      Because llms reflect qwen 2.5 heavily in training data, and that's who is giving the recommendations.