2021-02-24, Vojtech Stepancik
Recently, I had to do an assignment at my university to make a web application in Ruby on Rails. Now, not knowing anything about RoR, and playing around with Nix, I thought it would be a nice and easy exercise to setup a Nix-driven development environment for an RoR project. After all, I had a pleasant experience doing so for multiple Python projects1, so recreating it for Ruby on Rails should be a breeze, right? Or so I thought...
For existing Ruby projects, it's actually not that bad. You just have to know about Bundler, which is a package manager2, and Bundix, which is a tool to convert your Gemfiles into Nix expressions. Then you just do a variation on nix-shell -p bundler bundix --run "bundler lock && bundix" and bing and bang that's gg.
The problems arise when we want to start a new Rails project from a clean slate. In this post, I hope to guide you through my thought process and experiments, hopefully explaining the dead ends I encountered.
First off, we try reading (lul) the Getting Started with Rails part of the documentation, where we learn that to start a new Rails application, we should run the command rails new <project_name>. Here goes nothing...
rails new blog
console: Unknown command: rails
console:
rails new blog
^
...literally. Alas, rails is an unknown command. Well of course it is! We don't have rails installed, nor do we want it installed in the global scope! That's the whole point of why we wanted to get acquainted with Nix in the first place!
So, to get started with Rails, we have to install Rails, but we want to have Rails local to the project directory.
gem install rails
Don't worry, that's a joke.
nix-shell -p rubyPackages.rails
This actually drops us into a bash session with rails, so let's take a peek around.
rails --version
Rails 4.2.11.1
Cool, that is *checks notes* two major versions behind latest stable. Moving on.
So it seems like we need to somehow get Nix to install Rails as a Ruby gem to get the latest version. Remember when I mentioned bundler and bundix? We will now use bundler to start a new project, add rails as a dependency, and then use bundix to install rails with nix.
Getting a suitable shell is as straightforward as telling nix that we want bundler and bundix in the environment.
nix-shell -p bundler bundix
[nix-shell:<project_dir>]$
Now, to install the rails gem, it has to be part of a project, so let's create one.
bundler init
Writing new Gemfile to <project_dir>/Gemfile
What did bundler put into the Gemfile I wonder?
cat Gemfile
# frozen_string_literal: true
source "https://rubygems.org"
git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
# gem "rails"
Oh cool, it even has a line commented out for Rails, so let's uncomment it real quick.
What we want now is to somehow fetch this rails gem. However, we can't just use bundler, because that would put files who knows where and for sure touch the global state of the system, and that's Big Bad™.
Instead, we'll only use the dependency locking feature of bundler. What it does is resolve the correct specific versions of the dependencies as per the project's specification, and save these version descriptions in a Gemfile.lock file. Once we have this lockfile, we can use bundix to convert it into a nix expression that can be used as a build input for a nix shell.
bundler lock
Fetching gem metadata from https://rubygems.org/.............
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Writing lockfile to <project_dir>/Gemfile.lock
bundix
[...]
This results in a file called gemset.nix being created. This is at its core just a list of ruby gems and which versions the projects transitively depends on.
We can then use this gemset to make a shell containing the project-local ruby and gems by creating a shell.nix file with the following contents:
{ pkgs ? import <nixpkgs> {} }:
let
rubyEnv = pkgs.bundlerEnv {
# 👆 built-in function to create a ruby environment from a gemset
name = "env-proto";
gemdir = ./.;
# 👆 The directory that has a gemset.nix file
};
in pkgs.mkShell {
buildInputs = [
rubyEnv
rubyEnv.wrappedRuby
# 👆 without wrappedRuby, the shell wouldn't contain the ruby binary
];
}
After a quick Ctrl-D and nix-shell, we should get a fresh bash session with everything we asked for.
which ruby
/nix/store/pgalih75ki228d7nd2cilsaia9zplid5-wrapped-ruby-env-proto/bin/ruby
rails --version
Rails 6.1.0
Looks about right. With a rails binary at out disposal, we try to pick up where we left off, specifically with creating a new rails project.
(disclaimer: if you are following along, every unsuccessful attempt to initialize a project in the current directory makes further execution of the rails command fail, so I would recommend using something like rails new ../<alternative_project_name> instead for testing).
rails new .
[...]
An error occurred while installing nokogiri (1.10.10), and Bundler cannot continue.
Make sure that `gem install nokogiri -v '1.10.10' --source 'https://rubygems.org/'` succeeds
before bundling.
In Gemfile:
rails was resolved to 6.1.0, which depends on
actioncable was resolved to 6.1.0, which depends on
actionpack was resolved to 6.1.0, which depends on
actionview was resolved to 6.1.0, which depends on
rails-dom-testing was resolved to 2.0.3, which depends on
nokogiri
run bundle binstubs bundler
Could not find nokogiri-1.10.10 in any of the sources
Well, that failed spectacularly, let's understand why. It looks like the rails generator not only creates some files, but runs bundler to install its dependencies, which is obviously not what we want (since we're working on this sweet pure setup).
So first of all, we delete the ~/.gem directory that bundler created, and then we try to look through some docs on ways to prevent rails from running bundler. Fortunately, we don't need to look too far, because it's a flag we can pass to rails.
rails new --help
[...]
-B, [--skip-bundle], [--no-skip-bundle] # Don't run bundle install
[...]
Let's try it out then.
rails new . --skip-bundle
[...]
rails webpacker:install
Traceback (most recent call last):
3: from bin/rails:4:in `<main>'
2: from bin/rails:4:in `require_relative'
1: from <project_dir>/config/boot.rb:4:in `<top (required)>'
<project_dir>/config/boot.rb:4:in `require': cannot load such file -- bootsnap/setup (LoadError)
So I'm no Ruby expert, but that seems like a flop. Apparently, the bootstrapping code tries to load a file that doesn't exist. Looking into it further, bootsnap is a ruby gem that apparently rails just assumes is installed, which is not the case for us, since we don't have any gems apart from rails installed at this stage yet.
Once again, the rails CLI has a flag to deal with this, so let's try again.
rails new . --skip-bundle --skip-bootsnap
[...]
rails aborted!
Don't know how to build task 'webpacker:install' (See the list of available tasks with `rails --tasks`)
Uh huh. So, webpacker is another gem and we're in the same situation as above. Because webpacker isn't installed, rails doesn't know how to run one of its tasks, and we need to add one more flag.
rails new . --skip-bundle --skip-bootsnap --skip-webpack-install
Look ma, no errors! And as a nice bonus, the --skip-bootsnap flag isn't even required, because it was triggered by the webpacker:install step (you can check the log above).
Because rails changed the Gemfile, we need to rerun bundler and bundix and reenter the nix-shell, so that nix can build and install all out gems.
bundler lock && bundix && exit
Enter nix-shell again and wait a few minutes, because nix needs to download all the gems and presumably also build some native dependencies.
With all that out of the way, we can run the rails server and call it a day.
rails server
[...]
/nix/store/iwyych5hbs0kd46k7z01qhsb9c8di1n2-ruby2.6.6-webpacker-5.2.1/lib/ruby/gems/2.6.0/gems/webpacker-5.2.1/lib/webpacker/configuration.rb:99:in `rescue in load': Webpacker configuration file not found <project_dir>/config/webpacker.yml. Please run rails webpacker:install Error: No such file or directory @ rb_sysopen - <project_dir>/config/webpacker.yml (RuntimeError)
Well, almost. Remember all the flags we passed to rails new? We compensated for --skip-bundler by installing the gems with bundix, but we haven't done anything to address webpacker.
The fix is easy enough though, the instructions are even printed in the console (although they may get lost in the wall of text that ruby prints along with it).
rails webpacker:install
sh: line 1: node: command not found
sh: line 1: nodejs: command not found
Node.js not installed. Please download and install Node.js https://nodejs.org/en/download/
Exiting!
For the final touches, webpacker depends on Node.js and Yarn, so let's just add these two shell.nix.
{ pkgs ? import <nixpkgs> {} }:
let
rubyEnv = pkgs.bundlerEnv {
name = "env-proto";
gemdir = ./.;
};
in pkgs.mkShell {
buildInputs = [
rubyEnv
rubyEnv.wrappedRuby
pkgs.nodejs
pkgs.yarn
# 👆 These two are new
];
}
Then, we can run the webpacker task as intended.
rails webpacker:install
[...]
Webpacker successfully installed 🎉 🍰
At this point, we should have a working Ruby on Rails project.
rails server
=> Booting Puma
=> Rails 6.1.0 application starting in development
=> Run `bin/rails server --help` for more startup options
Puma starting in single mode...
* Puma version: 5.1.1 (ruby 2.6.6-p146) ("At Your Service")
* Min threads: 5
* Max threads: 5
* Environment: development
* PID: 16657
* Listening on http://127.0.0.1:3000
Use Ctrl-C to stop
And indeed we do 🎉.
We can go to http://localhost:3000/ to observe the beauty that is a default rails webapp.
Even with a working project directory, there are some steps that we could improve upon.
In short, there are 3 things we can change to make the setup cleaner:
Even though I call this "cleanup", it's more of a "proper setup", so instead of iterating on the existing project directory, this section will show all the necessary steps to get started from scratch.
First off, make an empty directory and enter into it. Then, initialize the niv boilerplate.
niv init -b release-20.09
You can choose versions of nixpkgs other than release-20.09, but that's what I use as of the publishing of this post. Also note that niv pins the source even if you use a branch like nixpkgs-unstable, so to get the newest version you need to manually update it with niv update nixpkgs.
To make working with bundler and bundix easier, let's create a nix shell environment that contains those two dependencies in nix/bundler.nix. It uses the pinned nixpkgs provided by niv and instructs bundler to put its gem index in the project directory, specifically nix/cache/bundle.
# ./nix/bundler.nix
{ sources ? import ./sources.nix }:
let
pkgs = import sources.nixpkgs {};
# 👆 use the pinned packages
in pkgs.mkShell {
buildInputs = with pkgs;[
bundler
bundix
];
BUNDLE_USER_HOME = toString ./cache/bundle;
# 👆 prevent bundler from putting stuff into ~/.bundle
}
Then, using the environment we just defined, initialize an empty bundler project.
nix-shell ./nix/bundler.nix --run "bundler init"
Don't forget to uncomment the line in the Gemfile that tells bundler to install rails.
Let's also add a short shell script that we can run to lock and install the dependencies specified in the Gemfile, and keep the gemset.nix file in the nix directory.
# ./nix/bundle
#!/bin/sh
nix-shell ./nix/bundler.nix --run "bundler lock $* && bundix --gemset=./nix/gemset.nix"
And then run the script right away, because we need the gemset.nix file for the next step.
./nix/bundle
I prefer keeping the ruby environment expression in a separate file, so let's define it in nix/rubyEnv.nix.
# ./nix/rubyEnv.nix
{ sources ? import ./sources.nix }:
let
pkgs = import sources.nixpkgs {};
in pkgs.bundlerEnv {
name = "project-ruby";
# 👆 Either `name` or `pname` is required
gemdir = ./..;
# 👆 The project root is one level above relative to this file
gemset = ./gemset.nix;
# 👆 But the gemset is right next to it
}
With the ruby parts ready, we can put it all together into a shell environment.
# ./shell.nix
{ sources ? import ./nix/sources.nix }:
let
nivOverlay = super: self: {
niv = (import sources.niv {}).niv;
};
# 👆 The overlay isn't strictly necessary, but I get a fuzzy feeling knowing that even the niv version is pinned
pkgs = import sources.nixpkgs {
overlays = [ nivOverlay ];
};
rubyEnv = import ./nix/rubyEnv.nix {
inherit sources;
};
# 👆 Import the environment we defined above
in pkgs.mkShell {
buildInputs = [
rubyEnv
rubyEnv.wrappedRuby
pkgs.nodejs
pkgs.yarn
pkgs.niv
];
}
That should do it for the initial setup, so that now we can bootstrap the rails project as before. The --force flag makes it so that rails doesn't ask whether to overwrite the existing Gemfile and just does it.
nix-shell --run "rails new . --force --skip-bundle --skip-webpack-install"
With the Gemfile changed, we need to regenerate the gemset.nix file.
./nix/bundle
And to finish the project initialization, we run the install step from webpacker.
nix-shell --run "rails webpacker:install"
And we check that everything's working properly.
nix-shell --run "rails server"
At this point, we have once again a working project directory, so we can talk about lorri a little.
I won't go into details, but basically what lorri does is act as a build daemon. It runs as a background service and watches for relevant changes in allowed directories, and when the environment declaration changes, it builds the new version in the background, and it keeps the used packages from being collected by nix-collect-garbage.
It also exposes a command that allows direnv to update environment variables when operating in the project directory. That means that if you setup your shell or editor to respect .envrc files, you don't even have to use nix-shell like, ever!
The setup is pretty simple. lorri init initializes the .envrc file (it would normally also create a sample shell.nix, but we already have one), and direnv allow adds the project on its internal whitelist (direnv won't run where you haven't told it to, to prevent some attacks).
lorri init && direnv allow
And that's pretty much the setup I'm currently using.
Unfortunately, this setup still has some minor issues regarding isolation.
For example, I couldn't figure out how to prevent yarn from creating the ~/.yarn directory and ~/.yarnrc file. It should respect the --use-yarnrc CLI flag, but for some reason it always touches those things anyway.
Also, bundix puts stuff in ~/.cache/bundix, but that's expected I guess.
During my investigation for this article, these two repos were of huge help, because I didn't really know what I was doing.