My one-shot memory worsened with years but grasping concepts from examples got much better due to experience. So I find walls of text much less useful than a continuous “synopsis” of snippets and best practices. One usage example is worth a hundred of words and few pages of it may be worth a whole manual, especially when “at hand” through editor means. For bash idk if that exists on the internet, so I maintain my own in my one big text file. I wish every manual had the synopsis-only section (maybe I should just LLM these).
While I agree with the larger sentiment I think you’re making (I also make it a habit to peruse the manual just to see what is available), how often do you reread the manual, especially for your shell? I knew about associative arrays in bash, but not by reading the manual as those were introduced in v4 and I’ve been using bash for longer than that.
You should probably read the release notes for new major versions. Also, anytime you think “I wish there was <feature> available in this program”, you should probably double-check if such a feature has appeared since you last learned the program.
Do you mean PCRE’s “(?&name)” feature? That is not portable; for instance, Python calls it “(?P=name)”. Or do you mean the common feature of “&” in the replacement string meaning “the whole matched string”? While this feature is common in languages which use regexps (including ed, sed, awk, etc.), that’s not actually a regexp feature, since the & character has that effect in the replacement string, not in the regular expression. It is also not always portable; for instance, Emacs requires escaping as \& for the character to have the special feature.
&pattern
Display only lines which match the pattern; lines which do not
match the pattern are not displayed. If pattern is empty (if
you type & immediately followed by ENTER), any filtering is
turned off, and all lines are displayed. While filtering is in
effect, an ampersand is displayed at the beginning of the
prompt, as a reminder that some lines in the file may be hidden.
$ less --version
less 487 (GNU regular expressions)
Copyright (C) 1984-2016 Mark Nudelman
Too many. I still write /bin/sh syntax (I know it's a symlink to bash now but I mean the old school sh). Anything that requires bash-that-isnt-sh is usually better written in perl or something else.
I love shell, I think it's killer feature are pipes and whoever figured out how to design so you can pipe in and out of control structures(Doug Mcilroy?) is a goddamn genius. however, after writing one to many overly clever shell scripts, I have a very clearly delineated point in which the script has become too complex and it is time to rewrite in a language better suited for the task. and that point is when I need associative arrays.
A lot of the sins of shell are due to it's primary focus as an interactive language. Many of those features that make it so nice interactively really hurt it as a scripting language.
Pipe-related concepts in various restricted forms were floating around for years. Doug McIlroy indeed proposed them in 1964 and was heading the Bell Labs team when they were implemented in the Third Research Edition of Unix (1973).
Picky and probably pointless question: are they actually hashmaps? If I understand correctly, a hashmap isn’t the only way to implement an associative array.
I think many devs don't know the difference and simply call any dictionary/associative array a hash map. It might be one of the concepts that "the internet" promotes: it sounds fancy, more technical, makes you seem more knowledgeable, so it gets repeated. Then newcomers think this is the way it's always been called, and that gives it enough visibility to become the preferred name.
No one else seems to have mentioned this, but POSIX sh does not include this feature.
Until it does, and major POSIX shs have shipped with it for a decade, then the feature will actually exist in a way that the average shell coder cares about. You're better off just shipping Rust/Go/etc binaries and leave sh for your simple glue tasks.
Even I've eventually switched to this, and I've written 10k+ long Bash scripts that followed best practices and passed shellcheck's more paranoid optional tests.
I discover stuff like this every day, and it’s delightful. Sure, reading the manual would’ve saved me from the surprise, but it’s incredibly difficult for me to read them unless I have a specific goal in hand.
I found out about hashmaps in Bash a year ago[1], and it came as a surprise. This morning, I came across dynamic shell variables and the indirection syntax[2]. Each time I wrote about them and learned more than I would have if I had just passively grokked the manual.
A couple of weeks ago I learnt that Bash on Mac does not have associative arrays. We worked around the issue by changing the script to run under Zsh, but beware.
To elaborate on this, macOS default bash is still stuck (assuming due to licensing) in v3.2.x (released in 2007). Bash got associative arrays in v4 (released in 2009).
I'm guilty of this. I knew zsh had them but since I can never remember the exact things zsh has that bash doesn't, I just assume anything remotely useful isn't compatible.
This policy comes from a six hour debugging session involving (somewhere) a script that manipulated a binary file - bash can't store zero bytes in variables and zsh can, but it's not like it'll tell you that it's pointlessly truncating your data and I never thought to look. So now every step in that script gets converted back and forth through xxd, and I don't trust Bash anymore.
bash associative arrays are fantastic, I've made heavy use of them, but be warned that there have been several memory leaks in the implementation. (Sorry, no version notes, once we realized this, we rewrote a key component in C.)
IIRC, the latest bash addresses all of these, but that doesn't help too much if you are stuck with an older, stable OS, e.g., RHEL 7 or 8; even 9 likely has a few remaining.
These leaks become an issue if you have a long running bash script with frequent adds and updates. The update leak can be mitigated somewhat by calling unset, e.g., unset a[b], before updating, but only partially (again, apologies, no notes, just the memory of the discovery and the need to drop bash).
I'd agree with the idea that bash wasn't the best choice for this purpose in the first place, but there was history and other technical debt involved.
Notice that other shells also has associative arrays, or at least zsh. I've found hyperpolyglot [0] to be a nice Rosetta stone for translating syntax between e.g. bash and zsh.
To me, this is a development in the wrong direction. Shell is great precisely because it's so minimal. Everything is string rule is one that calms all of your type-induced fears.
Having to implement hash-tables, while still keeping the appearances of everything being a string is the anti-pattern known as "string programming" (i.e. when application develops a convention about storing type / structure information inside unstructured data, often strings).
I would love for there to be a different system shell with a small number of built-in types that include some type for constructing complex types. There are many languages that have that kind of type system: Erlang, for example. But, extending Unix Shell in this way is not going to accomplish this.
<https://www.gnu.org/software/bash/manual/bash.html#Arrays>
Totally agree with that - I maintain a big txt file too.
Maybe this bash compendium is a bit similar:
https://www.commandlinefu.com/commands/browse
• <https://learnxinyminutes.com/>
And some code search engines:
• Debian: <https://codesearch.debian.net/>
• Python: <https://www.programcreek.com/python/>
• Linux: <https://livegrep.com/search/linux>
• GitHub: <https://grep.app/>
• HTML, JS and CSS: <https://publicwww.com/>
Otherwise wouldn't be getting the full shell experience.
A lot of the sins of shell are due to it's primary focus as an interactive language. Many of those features that make it so nice interactively really hurt it as a scripting language.
See https://youtu.be/FbDebSinSQo?si=xbWIqES80hI2S802&t=901
# Write a shell script:
* Heavy lifting done by powerful tool (sort, grep, curl, git, sed, find, …)
* Script will glue diverse tools
* Workflow resembles a pipeline
* Steps can be interactively developed as shell commands
* Portability
* Avoid dependency hell
* One-off job
# Avoid shell scripting:
* Difficult to see the preceding patterns
* Hot loops
* Complex arithmetic / data structures / parameters / error handling
* Mostly binary data
* Large code body (> 500 LoC)
* Need a user interface
A need for associative arrays (implemented in Bash as via hashmaps) moves the task to the second category (avoid shell scripting).
Seems to suggest it's indeed a hashmap in implementation, but I can't be bothered to look any closer.
Until it does, and major POSIX shs have shipped with it for a decade, then the feature will actually exist in a way that the average shell coder cares about. You're better off just shipping Rust/Go/etc binaries and leave sh for your simple glue tasks.
Even I've eventually switched to this, and I've written 10k+ long Bash scripts that followed best practices and passed shellcheck's more paranoid optional tests.
Use the right language for the right job.
I found out about hashmaps in Bash a year ago[1], and it came as a surprise. This morning, I came across dynamic shell variables and the indirection syntax[2]. Each time I wrote about them and learned more than I would have if I had just passively grokked the manual.
[1]: https://rednafi.com/misc/associative_arrays_in_bash/
[2]: https://rednafi.com/misc/dynamic_shell_variables/
<https://web.archive.org/web/20240810094701/https://meta.ath0...>
$ bash --version
GNU bash, version 5.2.32(1)-release (aarch64-apple-darwin23.4.0)
$ declare -A aaa; aaa[a]=a; aaa[b]=bb; for i in ${!aaa[@]}; do echo "$i --> ${aaa[$i]}"; done
b --> bb
a --> a
This policy comes from a six hour debugging session involving (somewhere) a script that manipulated a binary file - bash can't store zero bytes in variables and zsh can, but it's not like it'll tell you that it's pointlessly truncating your data and I never thought to look. So now every step in that script gets converted back and forth through xxd, and I don't trust Bash anymore.
For example: `${!myvar[@]}` would list all of the keys.
I've written about associative arrays in Bash a bit here: https://nickjanetakis.com/blog/associative-arrays-in-bash-ak...
IIRC, the latest bash addresses all of these, but that doesn't help too much if you are stuck with an older, stable OS, e.g., RHEL 7 or 8; even 9 likely has a few remaining.
These leaks become an issue if you have a long running bash script with frequent adds and updates. The update leak can be mitigated somewhat by calling unset, e.g., unset a[b], before updating, but only partially (again, apologies, no notes, just the memory of the discovery and the need to drop bash).
I'd agree with the idea that bash wasn't the best choice for this purpose in the first place, but there was history and other technical debt involved.
[0]: https://hyperpolyglot.org/unix-shells#associative-arrays
> A: You use the command `declare -A HASHMAP_NAME`
This is why I think Bash is a horrible language
Now to add hashtables to that.
Having to implement hash-tables, while still keeping the appearances of everything being a string is the anti-pattern known as "string programming" (i.e. when application develops a convention about storing type / structure information inside unstructured data, often strings).
I would love for there to be a different system shell with a small number of built-in types that include some type for constructing complex types. There are many languages that have that kind of type system: Erlang, for example. But, extending Unix Shell in this way is not going to accomplish this.
That ship has sailed. These were introduced in Bash 4, released in 2009, and Bash is already on version 5.