I’ve been playing around with mapping steam games + their reviews based on Qwen embeddings. I revisit this project every year around this time to see what improvements I can make to make a “better” map than last year (According to my own preferences). [0/1]
One thing I learned this year is that naive text mapping (embed a corpus of reviews with a Qwen embedding model, use umap to plot, then compare where things fall) primarily compares embedded text length more than anything else. What a review says affects the position less than how long the review is.
This was a little surprising to me. For whatever reason, I didn’t think the embedding would so strongly describe the length of the embedded passage. So most of this year’s time was spent fixing that artifact.
was one of very few useful results of the first decade of TREC research... And the main thing it addresses is balancing various factors which could cause a full text search engine to privilege either long or short documents.
This is an interesting idea, do you find it has more interesting properties than doing cosine similarly by reviewers? Have you found things adjacent to games you enjoyed that were also fun to play?
Yes, I’ve actually found a few interesting games I had never heard of before! So the project has been a success for me already.
But there’s still plenty of places where games are placed strangely on the map, and I like to investigate why that’s happening (it’s usually because Steam reviews are pretty noisy signal).
The first version of this idea was actually based on cosine similarity. The user would enter arbitrary text or an existing steam app id, and would get a list of games back.
The arbitrary text results were never very good because to get related results you needed to write a fake review of the kind of game you were looking for, which is a difficult concept to explain to users. Most users just entered text like "beautiful game", and that lead to bad results. "Not a beautiful game" winds up very close to that.
The results based on steam app id were much better, because it was comparing like-for-like. That lead to me considering different ways of presenting the games based on their review similarity. And so this map was made.
The umap projection does introduce a fair amount of distortion, so it's not a strictly better result than cosine similarity between specific Steam games. I'm still playing with the umap parameters to try and find something which causes the least problems.
The methodology feels either heavily undercooked to the point of being misleading, or heavily under explained. Why are we in 3D? At which point did dimensionality reduction hit? And if we're doing dimensionality reduction, does the "plane" really mean anything?
It's under explained. It's in 3D because of PCA. It mentions PCA all over the place, but never even says what it is, less alone briefly describe what it does.
There are a lot of more fancy ways to reduce dimensions and create visualizations like these that tend to preserve more of the higher dimensional structure.
I'm very familiar with PCA. There are so, so many ways you can apply it to this problem.
Assuming the simplest interpretation of what the analysis did, then it's saying "the variance of the biggest variance direction stays constant". Which in itself is an interesting thread to tug on, but the charts aren't how you'd present such a result.
There’s no meat here. It’s just someone throwing data into a plotting tool with no meaningful results or conclusions. I was expecting a meaningful analysis. Nope.
About a year ago, I made a very simple language detection model from the token embedding tensor of gemma 3. I used UMAP to show the projections of how languages separated into a 2D space. It's very interesting to look at IMO.
I wish there had been a third broad language style such as arabic although perhaps Qwen doesnt use arabic.
It made me think what if totally different knowledge bases could exist in a model where as my mental model was similar to OPs translation from pure knowledge source.
It also explains why rewriring prompts to another language sometimes lets you bypass restrictions.
This is visualizing end states, right? That’s why there are no ‘paths’ between the orange initial cloud and the ‘result’ clouds?
Wouldn’t we expect the languages to be ‘separate’ by that point? The output is necessarily different after all - and ‘more different’ for Chinese and English than for other Latin script languages and English.
It feels like it would be more interesting to see the intermediate states. Do questions about the same topics in Chinese and English ‘traverse’ similar areas in the vector space?
I think the raw tokens embeddings shouldn't have been included in the PCA (past the first or second graph). Since they're disjoint from the conversation embeddings, you're essentially losing a dimension and compressing all the conversations into a plane. There may well have been much more obvious structure in the language embeddings that you'd be able to see.
One thing I learned this year is that naive text mapping (embed a corpus of reviews with a Qwen embedding model, use umap to plot, then compare where things fall) primarily compares embedded text length more than anything else. What a review says affects the position less than how long the review is.
This was a little surprising to me. For whatever reason, I didn’t think the embedding would so strongly describe the length of the embedded passage. So most of this year’s time was spent fixing that artifact.
[0]: 2025’s map: https://storage.danieltperry.me/share/steam-map/
[1]: 2026’s map (still a work in progress, both the ui and the review processing are incomplete): https://storage.danieltperry.me/share/steam-map-refresh/
https://en.wikipedia.org/wiki/Okapi_BM25
was one of very few useful results of the first decade of TREC research... And the main thing it addresses is balancing various factors which could cause a full text search engine to privilege either long or short documents.
(Edit: They're live now, you'll need to hard refresh/clear your cache to get the updated HTML and JS)
But there’s still plenty of places where games are placed strangely on the map, and I like to investigate why that’s happening (it’s usually because Steam reviews are pretty noisy signal).
The first version of this idea was actually based on cosine similarity. The user would enter arbitrary text or an existing steam app id, and would get a list of games back.
The arbitrary text results were never very good because to get related results you needed to write a fake review of the kind of game you were looking for, which is a difficult concept to explain to users. Most users just entered text like "beautiful game", and that lead to bad results. "Not a beautiful game" winds up very close to that.
The results based on steam app id were much better, because it was comparing like-for-like. That lead to me considering different ways of presenting the games based on their review similarity. And so this map was made.
The umap projection does introduce a fair amount of distortion, so it's not a strictly better result than cosine similarity between specific Steam games. I'm still playing with the umap parameters to try and find something which causes the least problems.
https://en.wikipedia.org/wiki/Principal_component_analysis
There are a lot of more fancy ways to reduce dimensions and create visualizations like these that tend to preserve more of the higher dimensional structure.
Here's a semi recent paper discussing and comparing some of them, I didn't dig too deep to try to find a more recent or better one : https://pmc.ncbi.nlm.nih.gov/articles/PMC12453773/
Assuming the simplest interpretation of what the analysis did, then it's saying "the variance of the biggest variance direction stays constant". Which in itself is an interesting thread to tug on, but the charts aren't how you'd present such a result.
https://huggingface.co/blog/dleemiller/wordllama-detect
If you just want to see the plot directly: https://cdn-uploads.huggingface.co/production/uploads/65ff92...
It made me think what if totally different knowledge bases could exist in a model where as my mental model was similar to OPs translation from pure knowledge source.
It also explains why rewriring prompts to another language sometimes lets you bypass restrictions.
This is visualizing end states, right? That’s why there are no ‘paths’ between the orange initial cloud and the ‘result’ clouds?
Wouldn’t we expect the languages to be ‘separate’ by that point? The output is necessarily different after all - and ‘more different’ for Chinese and English than for other Latin script languages and English.
It feels like it would be more interesting to see the intermediate states. Do questions about the same topics in Chinese and English ‘traverse’ similar areas in the vector space?
Am I completely misunderstanding?…