This is so cool. I spent years at Pachyderm (YC...something) banging our head against:
- we're "Bazel for data"
- but...we have no Starlark. We're trying to port Airflow but it's not playing well with the way we cache partial results.
Amazing to me that I'm only now seeing the architecture pattern of "pure coordination language connecting sandboxed imperative components", which is used by Nix and Bazel, applied to distributed workflows. If somebody applied this to batch data processing you might have a Pachyderm that actually works better than ours did.
- we're "Bazel for data"
- but...we have no Starlark. We're trying to port Airflow but it's not playing well with the way we cache partial results.
Amazing to me that I'm only now seeing the architecture pattern of "pure coordination language connecting sandboxed imperative components", which is used by Nix and Bazel, applied to distributed workflows. If somebody applied this to batch data processing you might have a Pachyderm that actually works better than ours did.