Erlang can load and run C libraries. This can be useful if you want to speed up parts of your code, or if you want to interact with some OS-specific features. These special C libraries are written specially to work with the Erlang VM, and can’t easily be run in isolation without extra code. This makes these libraries difficult to debug. In this post, I introduce Erlang debug VMs, explain how to build them, and show why they can be tremendously useful for debugging linked-in drivers and NIFs.
Once you load and run arbitrary C code, the Erlang VM becomes very easy to kill.
$ erl
Erlang/OTP 19 [erts-8.2.2] [source] [64-bit] [smp:4:4]
[async-threads:10] [hipe] [kernel-poll:false]
Eshell V8.2.2 (abort with ^G)
1> test_nif:do_something_cool().
Segmentation fault
$ :(There are so many great tools for debugging Erlang code (dbg, redbug, erlyberly, recon) as well as excellent tools for debugging C code (gdb, valgind, afl-fuzz). However, there is a bit of a shortage of good tooling for debugging C code that gets executed by the BEAM. There’s niffy, which is a neat tool that acts like a reasonable subset of the Erlang VM, letting you load and call NIFs. Niffy and your NIF can then be run through something like afl-fuzz or your own test suite to try and uncover bugs. As far as I am aware, no such tool exists for Port Drivers.
I’ve gotten into the habit of building Erlang/OTP from source. This affords me more flexibility than using a package manager. One particularly nice thing about building Erlang from source is that you can compile debug VMs, modified BEAM binaries intended to be run inside of tools like GDB and Valgrind. This lets us debug any C code we’ve written that gets executed by BEAM. In addition, there is also a bonus set of GDB scripts called ETP which can pretty-print Erlang terms from inside of gdb! Debug VM binaries and ETP make the ordeal of digging through core dumps of crashed BEAM processes much easier, as you can poke around various parts of memory, and print the Erlang terms they correspond to.
If you want to build some debug VMs for yourself, the first thing you need to do is clone the Erlang/OTP git repository. (For managing software compiled from source, I highly recommend GNU Stow).
Here’s the general workflow of building and installing Erlang from source, using the git repository. Please note that these commands are somewhat symbolic; if you copy and paste them into your terminal, they may not work.
export ERL_TOP="$(pwd)/otp"; cd $ERL_TOP git checkout OTP-X.Y.Z $PAGER HOWTO/INSTALL.md ./otp_build autoconf
./configure --help | $PAGER
./configure --prefix=/usr/local/stow/erlang-X.Y.Z --and-other-opts make && make docs The following steps are optional:
sudo make install install-docs cd /usr/local/stow readlink $(which erl) to figure out the current version in use sudo stow -D erlang-X.(Y-1).W to unlink the old version sudo stow erlang-X.Y.Z to link the new version
We haven’t built any debug VMs yet. Let’s do that now. There are quite a couple
of different VMs you can build. The ones I use the most are the debug and
valgrind variants.
$ cd $ERL_TOP/erts/emulator
$ make debug valgrind gcov gprof lcnt icount
After this, you should have a shell script called cerl in $ERL_TOP/bin.
This is the script to invoke when you want to run a special BEAM VM. Adding
that script to your path will let you invoke a debug VM whenever you like!
cerl
Now that you have built debug VMs, and have cerl somewhere in your
path, we can start the BEAM inside of gdb or valgrind using cerl -rgdb or
cerl -valgrind. After that, you can pass arguments to the script as if you
were passing them to erl.
NOTE: If you type cerl -gdb, the script will
start Emacs to control your gdb session, which can be surprising!
I hope this inspires some of you to play around with Erlang debug VMs! There’s a lot of great stuff to be discovered! In the near future, I hope to publish another post showing off some use cases of debug VMs!