Installing FreeBSD into a live ZFS filesystem from f.ex. Linux

I have been using ZFS for a number of years, and have had my live filesystem moving between different Operating system. Hence my ZFS filesystem lives on, even if the operating systems get updated/changed or even replaced. In other words: my ZFS file-system is one day booted up by a Linux-kernel, some time later the same box could be running FreeBSD or Illumos.

And who cares what the name of Operating system is, those day are over, we just want to get our own job done, and your data is what is relevant in end. — With ZFS we can finally keep our filesystem even when we change our Operating system.

A little history

My first experience with ZFS was on FreeNAS v0.7 around 2009. One day while adding a disk to a zpool I made a grave mistake, I wanted to make a mirror, instead I added it as a concatenation. I immediately realized my mistake and removed the disk. That happened to be an even worse mistake. I now had a defunct ZFS system with 2 Terabytes of inaccessible files. At that time there was no way of fixing that problem under FreeNAS, which used an already an older version of ZFS. But it helped me learn a lot about the internal structure of ZFS. Basicly the way I got the ZFS filesystem back to life was to move the disks to a Solaris box, and scrolling back through the uberblocks to a time just before I made the fatal mistake.

I kept Solaris as the O.S. for some time, until ZFSonLinux became stable enough to use. Then I just installed Ubuntu onto one of the datasets, and booted Linux on the exact same ZFS-filesystem. Since then I have been using ZFS-on-linux as the main”bootloader” for my ZFS filesystem. By now my ZFS-filesystem had been:

  • created under FreeNAS which is based on FreeBSD ,
  • brought into an unusable and unrepairable state by a systems administrator error
  • brought back to life under Illumos, and used there for years until
  • ubuntu and zfs-on-linux was installed on a dataset
  • used daily for years under Ubuntu.

Without copying or making changes to the main ZFS datasets and home-dirs.

Lately I stumbled into a bug in ZFS-on-Linux github.com/zfsonlinux/zfs/issues/4122, which I am happy to say is fixed, but that made me adopt a back-up-plan – NAMELY – give myself the option to boot into FreeBSD. Which is what this post is all about.

To Day

My current system is running Ubuntu-14.04.3 after this: the system will run FreeBSD-10.2. The only thing tricky with standard FreeBSD is that they use their own boot-loader. We will avoid that and just use GRUB2, the ZFS enabled version from ubuntu in this case, but the one from PC-BSD and the version from Illumos should work too.

I would like to have the option to boot the system into either Linux or FreeBSD. To do that, we just have to add that functionality to /etc/grub.d/09-zfs which will generate the magic grub-lines that can boot FreeBSD too.

The plan for todays work is:

  • Install FreeBSD into a dataset
  • Create a /boot/zfs/zpool.cache so that FreeBSD will recognize the ZFS pool
  • Remove ZFS internal mount-points
  • Teach GRUB2 how to boot into a FreeBSD ZFS-data set
  • Use the same SSH credentials to avoid a lot of fuss

Most of the work is done on a live system system with very little down-time.

Install FreeBSD into a dataset

This little shell-script will do the majority of the work.

Create a /boot/zfs/zpool.cache so that FreeBSD will recognize the ZFS pool

Most likely the zpool.cache from Linux is not compatible with FreeBSD (linux/bsd, gcc/clang, device-naming). Regarding the device-naming, tweaking udev on Linux might be able to fix it part of the way, like we discuss in github.com/zfsonlinux/grub/issues/5 (I am storepeter).

The easiest way to fix this though is to boot FreeBSD from an USB-stick, start a shell, and create a zpool.cache manually this way

Remove ZFS internal mount-points

One thing I really don’t like about ZFS, is the facility to set mount-points inside ZFS. Hence if you import ZFS filesystem, have also imported the mount-points. and mounting Linux-root-fileystem on / for FreeBSD, is BAD.

To get rid of all the explicit mount-points I make the following changes to /etc/rc.d/zfs on FreeBSD (Recursively inherit mount-point for all datasets under /ROOT)

Teach GRUB2 how to boot into a FreeBSD ZFS-data set

I have been using ZFS-on-Linux for a long time and have made my own grub configuration so I can boot into various Linux versions. Kind of my own home-grown boot-environments, but much simpler than the Solaris/Illumos/PCBSD beadm

GRUB2 uses a number of shell-script to generate the grub.cfg that the bootloader uses when booting. Below is /etc/grub.d/09-zfs which makes it possible to boot into all the root-filesystems available on the zpool

That was quite a few number of script lines, but straight forward, and it is easy to test just run the script and see if it generates what you expect.

Install grub2 from FreeBSD

I already have GRUB2 installed on the harddrives, This one was installed from Ubuntu.
And it will happily boot both Ubuntu and FreeBSD. But if I have to update the Grub menus I have to reboot into Ubuntu and run update-grub.

I am normally running my ZFS-systems as mirrors, so it would be handy to have one disk boot with a Linux-based GRUB2, and the other with a FreeBSD-based GRUB2. So that we we will do. Assuming that we have booted the system into FreeBSD.

    • install the GRUB2 package

    • install the GRUB2 bootloader to the disk

grub-install –modules=”part_gpt zfs” /dev/ada0

    • add script to create menus /usr/local/etc/grub.d/09-zfs

The 09-zfs script above, will also work from within FreeBSD

  • update /boot/grub/grub.cfg

Keep ssh credentials

I copy over the ssh credential, so ssh-wise it looks like the same host.

That was all for now

Print Friendly, PDF & Email
This entry was posted in FreeBSD, Linux. Bookmark the permalink.