A Trip into FreeBSD
Published on , 1867 words, 7 minutes to read
I normally deal with Linux machines. Linux is what I know and it's what I've been using since I was in college. A friend of mine has been coaxing me into trying out FreeBSD, and I decided to try it out and see what it's like. Here's some details about my experience and what I've learned.
I've tried out FreeBSD on the following hardware:
- qemu/KVM on amd64
- Raspberry Pi 4 (4 GB)
- Raspberry Pi 3B (1 GB)
I've had the most luck with the Raspberry Pi 3 though. The KVM machine would hang infinitely after the install process waiting for the mail service to do a DNS probe of its own hostname (I do not host automagic FQDNS for my vms). I'm pretty sure I was doing something wrong there but I wasn't able to figure out what I needed to do in order to disable the DNS probe blocking startup.
After waiting for about 5 minutes I gave up and decided to try out the Raspberry Pi 4. The Raspberry Pi 4 is the most powerful arm board I own. It has 4 GB of ram and a quad core processor that is way more than sufficient for my needs. I was hoping to use FreeBSD on that machine so I could benefit from the hardware the most. Following the instructions on the wiki page, I downloaded the 12.2 RPI image and flashed it to an SD card using Etcher. I put the SD card in, turned the raspi on and then waited for it to show up on the network.
Except it never showed up on the network. I ran scans with nmap (specifically
with the command
sudo nmap -sS -p 22 192.168.0.0/24) and the IP address never
showed up. I also didn't see any new MAC addresses on the network, so that lead
me to believe that the pi was failing to boot. I downloaded an image for 13-BETA
and followed this
that claims to make it work on the pi 4, but I got the same issue. The Raspberry
Pi 4 unfortunately has a micro-HDMI port on it, so I was unable to attach it to
my monitor to see any error messages. After trying for a while to see if I could
set up a serial port to get the serial log messages (spoiler: I couldn't), I dug
up my Pi 3 and stuck the same SD card into it, hooked it up to my monitor,
attached a spare keyboard to it and booted into FreeBSD first try.
FreeBSD is a very down to earth operating system. It also has a
handbook that legitimately
includes all of the information you need to get up and running. Following the
handbook, I set a new password, installed the
pkg tool, set up
fish and then also installed the Go compiler toolchain
for the hell of it.
pkg is a very minimal looking package manager. It doesn't have very many
frills and it is integrated into the system pretty darn well. It looks like it
prefers putting everything into
/usr/local, including init scripts and other
This interestingly lets you separate out the concerns of the base system from
individual machine-local configuration. I am not sure if this also works with
/etc/resolv.conf or other system configuration files, but it does
/usr/local a reason to exist beyond being a legacy location for
yolo-installed software that may or may not be able to be upgraded separately.
Speaking of services, I wanted to see how hard it would be to get a custom service running on a FreeBSD box. At the minimum I would need the following:
- The binary built for freebsd/aarch64 and installed to
- A user account for that service
- An init script for that service
- To enable the init script in
I decided to do this with a service I made years ago called whatsmyip.
Building a Binary
Building the service is easy, I just go into the directory and run
Then I get a binary. Running it in another tmux tab, we can see it in action:
$ curl http://[::1]:9090
I can also run the curl command from my macbook:
$ curl http://pai:9090
Cool, I've got a working service! Let's install it to
$ doas cp ./whatismyip /usr/local/bin
doas? What is
doas? It looks like it's doing something close to what
doas is a program that does most of the
same things that sudo does, but it's a much smaller codebase. I decided to try
out doas for this install for no other reason than I thought it would be a cool
thing to learn. It's actually pretty simple, and I'm going to look at using it
elsewhere (with an alias for
The handbook says that we use the
command to add users to the system. So, let's run
adduser to create a
Full name: github.com/Xe/whatsmyip
Uid (Leave empty for default): 666
Login group [whatsmyip]:
Login group is whatsmyip. Invite whatsmyip into other groups? :
Login class [default]:
Shell (sh csh tcsh bash rbash git-shell fish nologin) [sh]: sh
Home directory [/home/whatsmyip]: /var/db/whatsmyip
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]: no
Lock out the account after creation? [no]: yes
Username : whatsmyip
Password : <disabled>
Full Name : github.com/Xe/whatsmyip
Uid : 666
Groups : whatsmyip
Home : /var/db/whatsmyip
Home Mode :
Shell : /bin/sh
Locked : yes
OK? (yes/no): yes
adduser: INFO: Successfully added (whatsmyip) to the user database.
adduser: INFO: Account (whatsmyip) is locked.
Add another user? (yes/no): no
It's a bit weird that there's not a flow for creating a "system user" that automatically sets the flags that I expect from Linux system administration, but I was able to specify the values manually without too much effort.
Something interesting is that when I set the user account to
actually was unable to log in as the user. Usually in Linux you can hack around
su flags but FreeBSD doesn't have this escape hatch. Neat.
Now that I had the service account set up, I need to write an init service that will start this program on boot. Following other parts of the handbook I was able to get a base script that looks like this:
# PROVIDE: whatsmyip
# REQUIRE: DAEMON
# KEYWORD: shutdown
command_args="-S -u whatsmyip -r -f -p /var/run/whatsmyip.pid /usr/local/bin/whatsmyip"
# DO NOT CHANGE THESE DEFAULT VALUES HERE
# SET THEM IN THE /etc/rc.conf FILE
Now I can copy this file to
/usr/local/etc/rc.d/whatsmyip and then make sure
it's set to the permissions
0555 with something like:
$ chmod 0555 ./whatsmyip.rc
$ doas cp ./whatsmyip.rc /usr/local/etc/rc.d/whatsmyip
Enabling The Service
Once I had the file in the right place, I enabled the service in
Then I started the service with
service whatsmyip start, and I was unable to
start the service. I got this error:
Feb 13 20:40:00 pai freebsd: /usr/local/etc/rc.d/whatsmyip: WARNING: failed to start whatsmyip
And no other useful information to help me actually fix the problem. I assume
there's some weirdness going on with permissions, so let's sidestep the user
account for now and just run the service as root directly by changing the
command_args="-S -r -p /var/run/whatsmyip.pid /usr/local/bin/whatsmyip"
Restarting the service, everything works! I can hit that service all I want and I get back the IP address that I used to hit that service.
What I Learned
FreeBSD has excellent documentation. The people on the documentation team really care about making the handbook useful. I wish it went into more detail about best practices for making your own services (I had to crib from some other service files as well as googling for a minimal template), but overall it gives you enough information to get off the ground.
FreeBSD is also fairly weird. It's familiar-ish, but it's a very different
experience. It's also super-minimal. Looking at the output of
ps x, there's
only 45 processes running on the system, including kernel threads.
root@pai ~# ps x | wc -l
The only processes are
dhclient, a device manager,
fish and a few instances of
to allow me to log in with an HDMI monitor and keyboard should I need to. That's
it. That's all that's running. It's only using 96 MB of ram and most of the
machine's power is left over to me.
It's just a shame that FreeBSD support for programming languages is so poor in general. Go works fine on it, but Rust doesn't have any pre-built binaries for the compiled (and using ports/pkg isn't an option because aarch64 is a tier-2 architecture in FreeBSD land which means that it's not guaranteed to have prebuilt binaries for everything). Compiling Rust from source also really isn't an option because I don't have enough ram on my raspi to do that. Go works though.
I really wonder how this kind of network effect will boil down with more and more security libraries like pyca integrating Rust deeper into core security components. It probably means that people are going to have to step up and actually do the legwork required to get Rust working on more platforms, however it definitely is going to leave some older hardware or less commonly used configurations (like aarch64 FreeBSD) in the dust if we aren't careful. Maybe this isn't a technical problem, but it is definitely something interesting to think about.
Overall, FreeBSD is an interesting tool and if I ever have a good use for it in
my server infrastructure I will definitely give it a solid look. I just wish it
was as easy to manage a FreeBSD system as it is to manage a NixOS system. A lot
of my faffing about with
rc.conf and rc scripts wouldn't have needed to
happen if that was the case.
Facts and circumstances may have changed since publication. Please contact me before jumping to conclusions if something seems wrong or unclear.