Repository navigation
Please drop fat binary gems for Windows #173
Description
Activity
@larskanis are you recommending that we just don't ship binary gems at all? I'm fine with that, I just want to make sure I understand what you're asking us to do. 😀
@tenderlove Exactly, that's it!
RubyInstaller comes with builtin DevKit nowadays, so that bcrypt builds and runs fine out of the box. There's no benefit from binary gems other than somewhat faster gem installation. They do more harm than good.
Even for gems with external dependencies to C libraries, it's often enough to install the corresponding package from the MSYS2-MINGW repository to get it installed. The pacman tool (known from arch linux) is also built into RubyInstaller. The package installation can even be automated by a hint in the gemspec like here for sqlite3-ruby.
Reacted by Aaron PattersonSounds good to me! @tjschuck lets not build binary gems for windows anymore. It seems like we don't need them and Windows folks would be better off if we didn't.
@larskanis This is music to my ears — building the Windows fat binaries has always been the primary challenge in releasing new versions of this gem.
A couple of questions:
-
Most of the popular C-extension gems I quickly searched for all still seem to include Windows binaries — are there popular C-extension gems you can point me to that have already moved to this "you must have a compiler on Windows" system? It seems like having the DevKit installed is both popular and easy enough now, but it'd be nice to have some confirmation that this isn't a big problem.
-
You might be the wrong person to ask, but do we need to keep providing the compiled Java JAR file, or can we stop packaging that as well? Can we just assume/require a compiler across the board and just ship source without any binaries?
I'm going to start ripping out the infrastructure for the Windows binaries now. I think I'd like to jump to a new major version (4.0.0) for the next release and make a couple of major upgrades and breaking changes, so we'll also clean-slate that into the world of "no provided binaries".
@larskanis Thanks a million for clarifying that dropping these is probably okay! ❤️
-
are there popular C-extension gems you can point me to that have already moved to this "you must have a compiler on Windows" system?
I know about the pkcs11 gem only, where I dropped the binary gem (so far without any user complaints). However rails implicit made Devkit a requirement with rails-4.0 which required atomic gem to be compiled. Since then most of the gems with C-extensions need to be compiled on Windows as well: nio4r, websocket-driver, bindex, bootsnap, byebug, duktape and puma for rails-5.2. So working with rails without the Devkit isn't possible since then.
Precompiled are nokogiri, msgpack, ffi, pg, mysql2 and sqlite3. They depend on particular C libraries, which are shipped with the binary gems. This makes installation somewhat easier on RubyInstaller-2.4+ and would make it really hard on earlier Ruby versions otherwise. So for some of them binary gems still make sense. Anyway I think this it's becoming a thing of the past and that's one of the reasons why I wrote RubyInstaller2 with it's pacman support.
do we need to keep providing the compiled Java JAR file, or can we stop packaging that as well?
I'm not familiar enough with JRuby to answer this comprehensive. But since JRuby is pretty cross platform, releasing a gem is usually just switching to JRuby and doing rake gem and gem push, without all the cross compile efforts.
I really think that fat-binary gems should continue to be supported. First of all, like it or not, there a lot of users still using Ruby 2.2/2.3. There are also some rather tricky gems to build.
Given that testing on Appveyor performs almost all the work needed to construct a fat-binary gem, I've created a PowerShell framework to build the fat-binaries, then test. It can also be run locally if one has all the Ruby versions & build tools (there is also a PowerShell script available that will install all of them).
I've done PR's to Puma, EventMachine, and SQLite3-Ruby for it. It even builds the 64 bit version with trunk support (only for testing). There's also the issues with Bundler...
Thoughts?
Greg
First of all, like it or not, there a lot of users still using Ruby 2.2/2.3.
Why would using Ruby 2.2/2.3 particularly necessitate pre-compiled binaries? As long as you have the DevKit installed, everything compiles just fine seemingly all the way back to 1.9.3. The gem even compiles fine under 1.8.7 as long as you jump back to DevKit 4.5.2.
There are also some rather tricky gems to build.
That's a good argument for some gems to offer precompiled binaries, like the ones @larskanis mentioned, particularly if they link to external libraries. But bcrypt-ruby is just compiling some simple standalone C files — nothing tricky at all. You can build the gem using rake compile (or more likely rake compile -rdevkit on Windows) and it should just work for everyone as long as their compiler isn't broken.
Based on how easy it was for me to get the gem compiled and tests running on every version of ruby from head all the way back to 1.8.7 on Appveyor in #174, it seems like dropping the precompiled binaries won't really be an issue as long as you're fine with having a compiler via the DevKit. The *nix based version of this gem (including macOS) has never provided a precompiled binary, so "having a compiler" has always been a requirement for that, and it's never caused an issue for anyone. I know "having a compiler" might have been a bigger hurdle historically in Windows, but via the DevKit, it seems like it's pretty much the default for any Ruby programmer on Windows these days.
Thanks for the response. I don't disagree with anything you've said. But, you're fluent in C, and everyone else here is either the same, or outliers like me that have become familiar with building. So what is easy for us may not be for others.
Hence, the whole reason I think pre-compiled gems should be supported is for users not like us. Maybe someone who is trying to learn how to program, who may also have very basic computer knowledge (or knowledge of platforms other than Windows, but not Windows).
All they want is for bundle install to work, and they don't even have a cue as to how to troubleshoot issues with a build system. At one time, we were all like that...
Thanks, Greg
@MSP-Greg Haha, oh how I wish I was fluent in C! I'm a Ruby developer through and through. When nokogiri has trouble installing on my system, I Google the error messages like anyone else.
I think what we're getting at here is that for bcrypt-ruby, bundle install DOES just work. Because it has no dependencies or linked libs in the C extension code, as long as you have a compiler installed (which at this point, you probably do without even knowing it — this seems to have not been the case with Ruby on Windows historically, but now it is), everything will just work. We're not suggesting all precompiled binaries for all gems be removed here; just for this specific gem.
Thank you all for doing this. I can finally delete this from my Gemfile:
if Bundler::WINDOWS gem 'bcrypt', '~> 3.1.11', git: 'https://github.com/codahale/bcrypt-ruby.git', ref: 'fbbece54c6cb8b53db01132c7eeb58955944547d', require: 'bcrypt' end
This, naturally, results in a Gemfile.lock diff anytime I bundle on Windows.
Because they do more harm than good. Every year we get the same issues, when the new Ruby version is released on Christmas: All the fat binary gems are incompatible.
This is although I release a new version of rake-compiler-dock at the same time as well as several core libraries (ffi, nokogiri, pg) with support for the new Ruby release. It still takes months or even years until other important libraries get updated and the new Ruby version gets usable on Windows.
Years ago I added gem version constraints for fat binary gems to rake-compiler, so that rubygems or bundler could solve this issue by switching to the source gem automatically. Unfortunately this doesn't work, since bundler still doesn't support this use case. Therefore Windows users have to use the various workarounds mentioned in all the issues (#142, #149, #139, etc).
Another issue is that fat binary gems hinder to run and test applications or gems on Ruby head versions, since they can't support ruby-trunk by design.
bcrypt-ruby doesn't depend on any external libraries to be available, so that it builds fine with any older or newer RubyInstaller version. It's easier than ever to install the Devkit on Windows and the Ruby+Devkit flavor is by far the most downloaded version. So there's no need to provide fat binary gems for Windows.
Please make Windows a first class citizen by dropping the x64-mingw32 and x86-mingw32 gem versions!