Create a 'download' language for fetching pre-built binaries #1453
Description
I'm aware of the requirement that additional dependencies for system hooks - the 'escape hatch' for hooks that don't fit into the python/node/go etc. managed environments list, such as shellcheck, must be installed outside of pre-commit in order to work.
Also aware of the python module that can be used to 'mirror' shellcheck in particular, though that feels a little hacky.
I'm sure you've given this thought before, and there's a very good reason why this hasn't been done, but, is it conceivable, with a PR I'd be willing to write, that additional dependencies could be supported for non-python etc. hooks?
I'm thinking along the lines of being able to specify a URL to a ZIP, which pre-commit would wget into its cache directory, storing a checksum so it could 'manage' when to re-download it, and then adding the directory to the front of $PATH when running the hook.
I'm sure it's not possible, but would be keen to know why, as there are a few hooks I use that this applies too and would feel like a useful addition for others.
Activity
asottile commented on May 12, 2020
I don't think that would be a feature to language: system -- I think that would be a separate language itself (maybe language: download or something). in the ideal world, pre-commit would learn how to install environments of all major programming languages and then you'd be able to install the tool you're looking for in that manner. there's directions for adding a new language here and several languages have been contributed by external contributors already!
also python is not the only language which supports additional_dependencies, go, ruby, node, and a few others also support the setting (and presumably languages which are added can also support that as well).
for shellcheck in particular, shellcheck-py is the suggested tool for pre-commit
note that a url alone does not help you here since you need to work about platform and architecture and likely checksumming, etc.
for system / script hooks in particular, they do not get any "environment" to install things into (they're run directly from configuration-only)
marcjay commented on May 12, 2020
That's a good point, yup
Is always installing from source ideal? The language a hook is written in feels like it should be an implementation detail as much as possible. We don't use Go at all for instance, but if we want to use a hook written in Go, using language: golang we need to have a working Go toolchain installed locally. (I did see your comment about it being a second-class language at the moment, and that it's intended to become a first class one - #1449 (comment)). Shellcheck, amongst some other tools, is written in Haskell - I'm not sure I like the idea of needing a Haskell toolchain installed locally. Again, I know about shellcheck-py, but needing to use a wrapper that relies on a third party keeping it up to date with new releases of shellcheck seems sub-optimal.
Appreciate the need to handle platforms, architectures and checksums, but being able to point at a pre-built binary, pull it and run it seems useful?
asottile commented on May 12, 2020
yeah I'm not saying language: download is off the table, just that it's not as simple as you're making it out to be
marcjay commented on May 12, 2020
I don't think it's that simple - I don't think I could have added any more caveats up top 😄
I guess I just wanted to know if
a) there's a known reason why it's not possible/sensible
b) if a PR was put out for it, would it be rejected as not something you'd want in the project
If either are true, I wasn't going to expend any energy having a deeper look at it. On the basis that it seems to get a luke warm reception, I'll have a proper look. And hey, worst case, I'll run into a blocker and leave a comment here to save the next person from wasting time on it :)
asottile commented on May 12, 2020
yeah nothing has been attempted there
asottile commented on May 13, 2020
would you like to design / work on such a thing?
marcjay commented on May 13, 2020
Yes I would like to. I'm aiming to look at it in the next day or so
marcjay commented on Jul 8, 2020
Timeline was ambitious but not forgotten about. Made a start at the time but need to find some more time to pick it up again. If you'd rather have a cleaner issues page, by all means close this and I can come back when/if there's a PR to be had
marcjay commented on Aug 21, 2020
I've been working on this further and noticed a general case which might be simpler, and cover most of what I was looking for in the first place. It's 50/50 on whether you'll hate the idea, if you don't, I'll raise a separate issue to track it and work on it.
Premise 1: A lot of tools that aren't pip installable are on GitHub, and when they are released, artifacts are available from the releases page, with artifacts built for most OS + arch combinations
Premise 2: These artifacts have names that include the OS + arch in some way that can be pattern-matched (like ...linux_x86_68..., ...darwin-amd64...)
Packaging aside (zip vs tar.xz vs bare), this is true for a number of repos - some examples:
https://github.com/koalaman/shellcheck/releases
https://github.com/mvdan/sh/releases
https://github.com/terraform-docs/terraform-docs/releases
https://github.com/terraform-linters/tflint/releases
What do you think to the following rough proposal for config?:
.pre-commit-config.yaml:
repos: - repo: https://github.com/terraform-linters/tflint rev: v0.19.0 hooks: - id: tflint.pre-commit-hooks.yaml:
language: github_release would work by:
With the benefits as before, that you don't need to have the toolchain installed for whatever language the hook was written in, and there's a good chance it can download and unpack an asset faster than it can build from source on that initial environment setup.
Is that a terrible idea?
marcjay commented on Aug 22, 2020
Some quick experimentation on the above worked pretty well. Only framework change I had to make was to add rev to the Hook class, so the tag was available as well as src, and to pass the hook.src and hook.rev to install_environment(), else run_hook() has to handle checking if it needs to download the asset or not.
.pre-commit-config.yaml (the example-shellcheck repo is private):
repos: - repo: git@github.com:marcjay/example-shellcheck rev: v0.7.1 hooks: - id: shellcheckTiming pre-commit:
If you hate it @asottile I'll abandon it, else I can tidy it up and put out a PR
asottile commented on Aug 22, 2020
it kinda doesn't gel nicely with the framework which expects the cloned repository to have all the necessary things (and would only work with tags and not frozen revs, and is a bit too github-specific). I was hoping for something a little more generic 🤔
the description of your changes wrt src / rev also concerns me -- the installed state is supposed to dictate whether something is installed or not and it sounds like parts of that are missing
marcjay commented on Aug 22, 2020
Appreciate the speedy response
I see what you mean, though tempted to argue this already isn't the case - if the user needs to have the matching runtime installed to build the source. If the maintainer has gone to the 'effort' of building binaries, it seems a shame not to use them?
Yup, that's what I started on and I thought this could be additional to a generic approach, to optimise this seemingly common scenario and greatly reduce the config needed. I can stick to just the one language addition though.
I think I worded it badly, the src/rev are just needed when installing the environment purely to build the API URL. Everything works normally wrt to state
marcjay commented on Sep 2, 2020
Before I tidy this up and figure out how to write tests for it all, could I run the (working) proposal past you for language: download @asottile?
(I noted the : separator in cli:{package_name}:{package_version} for rust but this gets messy here given the : in URIs, so opted for pipes)
syntax: <os>|<arch>|<url>('->'<name>)(|<checksum>)
Example (checksum is optional - provided in the darwin case):
shfmt is an example of an unpackaged artifact, so an optional ->NAME can be used to transform the downloaded binary to a sensible version-independent name:
Keen to know how close this is to what you're expecting/what needs to change. Cheers!
51 remaining items
Add a 'download' language for fetching pre-built binaries
fangfufu commented on Oct 6, 2023
Pull request generated at
#3020
Add a 'download' language for fetching pre-built binaries
scop commented on Nov 27, 2023
Looks like there's quite a bit of movement in this area currently. Anyway I filed pre-commit/pre-commit.com#904 which wasn't accepted but was asked to chime in here so here goes:
https://github.com/scop/wrun implements a generic "download launcher" with caching and archive support, not specific to pre-commit. It does work with pre-commit too though, just not with pre-commit.ci as it stands, and perhaps not fully the way pre-commit's languages are intended to work.
It might contain some ideas that a pre-commit specific implementation could make use of.
One comment about the subresource identity idea here: I considered that for wrun as well, but stuck with the "regular" hex digests with algorithm prefix. This is because these downloads are not really "subresources" in my opinion, and because IME hex digests are vastly more common than base64 ones in this problem space. Using base64 would mean one would practically need to generate or convert the digests separately instead of being able to use upstream provided ones as is. It might require some custom / not that common tooling as well or a few more hops, as the standard sha256sum and friends family of tools would not be applicable.
Timmmm commented on Jan 31, 2024
I just created a simple Rust hook and wondered how pre-commit was setting up Rust if a user doesn't have it. Turns out it installs it to $HOME in the normal way. Probably the only sensible option but I can guarantee if I introduce this into my company people are going to complain "why has something installed Rust when I didn't want it to"?
Anyway, a workaround I've used previously is to distribute binaries via Pip. Yes it sounds mad, but Pip is pretty much universally available and works quite well for distributing Rust programs. Here's an example - you can pip install maxtime and nobody is any the wiser that it's using Rust (some people are weirdly sensitive about that).
So, for Rust programs at least, can I:
Is there any reason that wouldn't work? Will that all be cached?
asottile commented on Jan 31, 2024
they would never know, it is not exposed outside of the pre-commit cache. it's not installed to their homedir at all you must be confused
Timmmm commented on Jan 31, 2024
Ah even better. Looks like I misread the code! I guess my suggestion would still be good from a setup speed & disk usage point of view, but I don't think I care enough about that to bother seeing as it is one-time.
asottile commented on Jul 18, 2024
@gytis-ivaskevicius this is not an advertising space for off topic tools
awoimbee commented on Oct 10, 2025
Honestly I don't see how shipping binary dependencies can work without a hook postinstall script.
A simple 'download' language would not support setting permissions, unzipping, etc. For example, here is one of my Makefile target:
c0m1c5an5 commented on Nov 15, 2025
wrun is now archived and suggests using mise or aqua instead. I found this thread during a research on previous issues before creating a new issue for adding the aqua language and I am willing to work on a PR for this language.
Edit: After spending some time with aqua both locally and in CI, I no longer see a reason for adding support as a language specifically, as regular aqua install + sysyem/python language achieves most requirements. I guess the only advantage of having it as a language would be pre-commit.ci support.
c0m1c5an5 commented on Jul 3, 2026
asottile commented on Jul 3, 2026
that's not really relevant for this discussion
c0m1c5an5 commented on Jul 3, 2026
asottile commented on Jul 3, 2026
there's already conda if you want that. no it's not relevant here