Table of Contents

For over 2 years now I’ve been running OpenWrt on my main router. It’s a reasonably powerful Qotom box, meaning it uses the x86/64 image which is fairly atypical for OpenWrt. And the upgrade process is not exactly straight forward. So I decided to document the procedure, and create a script to automate it!

Overview

This is a lengthy article describing my motivation for this project and the steps I took to create a script that automates the upgrade process for my OpenWrt router. If you’re just interested getting straight to the script, head to the TLDR or simply find the most updated version of the script on my GitHub page.

Requirements

• An x86 machine running OpenWrt
• Main drive > 256 Mb
• Main drive split into at least 3 partitions

The 256 Mb drive requirement is a bit ridiculous, I know, but it’s to make my next point. You’ll need a /boot partition (16 Mb by default), and two others > 100 Mb each. For example, I have a 128 Gb drive, split like this:

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0 119.2G  0 disk 
├─sda1   8:1    0    16M  0 part 
├─sda2   8:2    0    10G  0 part 
├─sda3   8:3    0    10G  0 part /
└─sda4   8:4    0  99.2G  0 part /opt

Here, sda1 is the boot drive (containing the kernels), sda2 and sda3 are my OpenWrt root filesystem partitions, and sda4 (optional) is simply the ‘rest’ of the drive which I mount to /opt to keep certain files between upgrades - more on this later.

Background

Why am I not using a proper firewall software like pfsense, OPNsense, or otherwise? Well to be honest I probably should move my main router over to one of these purpose-built softwares and just leave my WiFi routers on OpenWrt. But there’s one (well 2) piece(s) of software that keeps me on OpenWrt - cake (and cake-autorate). This isn’t a post praising cake QoS, so I’ll just say that it is amazing for those of us still on fluctuating bandwidth connections like 4G/5G.

Because of this, I am running OpenWrt on an x86/64 machine which isn’t very well supported by the OpenWrt project. There are directions on getting it to work, but as soon as you get to the “Upgrading” section, things look a lot more challenging. Normally, on my WiFi routers, there’s a “sysupgrade” option that makes upgrading between versions effortless.

So I’ve followed the instructions a handful of times to upgrade my box between both minor and major versions. It’s worked well, but was often a frustrating process. That was until I came across a post by user “frolic” in the OpenWrt forums. The idea is straight forward, and upon seeing the steps, I decided to jump in and try it myself.

This process worked well enough, so why not create a script to do it automatically? ;)

Basic idea

The idea behind this upgrade is to have at least 3 partitions on your main drive. For example:

/dev/sda1 = Boot partition
/dev/sda2 = OpenWrt root filesystem partition 1
/dev/sda3 = OpenWrt root filesystem partition 2
/dev/sdaX = Whatever else you'd like (more on this later)

With the above in mind, we can imagine that we have our current version of OpenWrt (let’s say 23.05.2 for example) on /dev/sda2, and /dev/sda3 will be the old version (let’s say 23.05.0). Assuming we are happily running 23.05.2 with no issues, we can completely overwrite /dev/sda3 to upgrade it to the newest available version (let’s say 23.05.3).

We will be able to copy the new version’s root filesystem in its entirety onto /dev/sda3, followed by the working configuration from your current working installation.

We will also need to get the newest kernel, and copy that onto the /boot partition, and update our GRUB configuration to ensure we boot with the correct kernel and root filesystem

Caveats

This all sounds great, but how do we deal with things like installed packages? Considering we are using an x86 machine, chances are we are going to be running other, memory or cpu intensive applications that we’ll want running on the new version directly.

Some examples:

• Separate DNS server (Adblock, AdGuard Home, Pi-Hole, etc.)
• An IDS/IPS (Snort or otherwise)
• A DPI engine (Netify)
• Containers (Docker or Podman)
• Etc. - This is just a Linux machine after all!

There are a couple ways we can deal with this:

1. Use the standard installation, get a list of user-installed packages, and put a small helper script in /etc/uci-default which is run at first boot. More here.
2. Use the Attended Sysupgrade build server to install the packages ahead, and use that image to have everything installed ahead.

The first option is clearly the easiest. I did this manually (without said helper script), the first handful of times I upgraded my router. But there is one issue… If you’re using a custom DNS software (like I am with AdGuard Home), you won’t have internet access (or at least be able to do DNS lookups). This isn’t impossible to overcome, but comes with other challenges as well (AdGuard Home is a number of versions behind in the repos!).

Thus, I’ve decided to lean on the second version with a small caveat. This leads me to another option:

3. Use the Attended Sysupgrade API to build the image, but keep custom-installed applications on a 4th partition - sda4.

Plan

Let’s look a little deeper at the full plan required to accomplish this upgrade:

1. Install script dependencies
2. Do a few checks to see what version we’re on, what’s the newest release, and which partitions contain which.
3. Get a list of user-installed packages for the new version
4. Build the custom image, download, and unzip to the target partition
5. Remove old kernel(s), install new kernel
6. Update GRUB with new kernel and partition ID’s
7. Copy config and files from current working version to new version

Wow, well this is a lot… But it reminds me of the idiom: “How does one eat an elephant?”…“One bite at a time.”

Breakdown of the script

This is a long and verbose description of each section of the script at the time of writing it. Firstly, it is set of personal notes for me to refer to for future projects. Secondly, installing and running scripts from random people on the internet is generally not a great idea - this adds some transparency for those interested. For the rest, head to the TLDR section.

1) Dependencies

Uh… how do we know what dependencies to install when we haven’t even started! Frankly, I usually do this step “last”, or at least add to the list as I go.

I ended up needing 2 packages that aren’t installed by default: lsblk and curl. The first is necessary to get partition names, the second is because the built-in wget is not actually a full version of wget and cannot send custom headers in a request. This is a problem later on…

opkg update && opkg install lsblk curl

2) Checks and Variables

We want to check which version we are on, which updated version is available, and which version is on the ‘old’ partition. We will use all of these as variables throughout the script to keep things straight.

# Set mount point for second OpenWrt installation
mount_pt=/tmp/mnt
mkdir ${mount_pt}

# Check current release vs new release
current_dist=$(grep DISTRIB_RELEASE /etc/openwrt_release | cut -d "'" -f 2)
new_release=$(wget -qO- https://downloads.openwrt.org | grep releases | awk -F'/' '{print $2}' | tr '\n' ' ' | awk '{print $1}')

# If the running release is the same as the newest, exit
if [[ $current_dist == $new_release ]]
  then echo -e "Already on newest release: /n/t Current: ${current_dist} = Newest: ${new_release}"; exit 1
fi

# Which device/partition is the boot, which is the currently mounted as rootfs, which is the target
boot_dev=$(lsblk -pPo LABEL,PATH | grep kernel | sed -E 's/.*PATH="(.*)".*/\1/')
current_dev=$(lsblk -pPo MOUNTPOINTS,PATH | grep 'MOUNTPOINTS="/"' | sed -E 's/.*PATH="(.*)".*/\1/')
if [[ $current_dev =~ 2 ]]
  then target_dev=$(lsblk -pPo PATH | grep '3"' | sed -E 's/.*PATH="(.*)".*/\1/')
else target_dev=$(lsblk -pPo PATH | grep '2"' | sed -E 's/.*PATH="(.*)".*/\1/')
fi

# Mount the target device, check old version
mount ${target_dev} ${mount_pt}
old_dist=`grep DISTRIB_RELEASE ${mount_pt}/etc/openwrt_release | cut -d "'" -f 2`

echo "Current OpenWRT release: ${current_dist} on ${current_dev}"
echo "New OpenWRT release: ${new_release} to replace ${old_dist} on target ${target_dev}"

# Ask user to confirm continuation of upgrade process
read -n1 -p "Continue with upgrade? (WARNING: THIS WILL OVERWRITE ${target_dev}) [y/N]: " doit

Each section is commented to give an idea of what it does. You’ll notice I make heavy use of the lsblk command, especially with the -pPo flags. This combination gives me full paths, key=value pairs, and only outputs selected info (columns). The key=value pairs are critical to easily find information by the provided key (like MOUNTPOINTS or PATH). Pipe (|) these outputs into grep and sed and we get an easy way to grab the relevant information.

One thing to note is that I limited (hard coded) the target_dev to either your /dev/sda2 or /dev/sda3.

3) Installed Packages

Most of us have probably something other than a 100% stock image installed on our x86 machines, thus we will want to ensure the new version of OpenWrt has the same applications . The forums are littered with multiple ways to get this list.

Potential Solutions

One would think there should be an easy way to do this with a opkg list-installed command, but this lists every package (including those installed by default with the OS). We could simply take this list and install it over the base image, but there’s one potential breaking issue: Software package replacements.

For example, the release of 22.03.0 moved the firewall backend from iptables to nftables. Thus, installing the “currently installed” firewall package would potentially conflict with or rollback the updated firewall. So we truly need to filter out all system-installed packages, and only get a list of user-installed ones.

User-Installed package lists

The two solutions I found are from users “spence” and “efahl”.

Both of their solutions work well. The first compares the installed packages with the package manifest posted with version released, and compiles a list of user-installed packages. This works well, but takes some time to create and involves two separate scripts.

The second solution is simplified by using files already existing within the opkg system in combination with grep and awk commands. This significantly cuts down on the run time and is a much smaller addition to the overall script. Once again, shoutout to “efahl” for the solution. I simplified their original idea a little:

package_list=./installed-packages
rm -f $package_list

examined=0
for pkg in $(opkg list-installed | awk '{print $1}') ; do
    examined=$((examined + 1))
    printf '%5d - %-40s\r' "$examined" "$pkg"
    #deps=$(opkg whatdepends "$pkg" | awk '/^\t/{printf $1" "}')
    deps=$(
        cd /usr/lib/opkg/info/ &&
        grep -lE "Depends:.* ${pkg}([, ].*|)$" -- *.control | awk -F'\.control' '{printf $1" "}'
    )
    count=$(echo "$deps" | wc -w)
    if [ "$count" -eq 0 ] ; then
        printf '%s\t%s\n' "$pkg" "$deps" >> $package_list
    fi
done

n_logged=$(wc -l < $package_list)
printf 'Done, logged %d of %d entries\n' "$n_logged" "$examined"

4) Getting the System Image

As stated in the “Caveats” section above, I decided to utilize the “attended sysupgrade” (ASU) API to build the a custom image that included my user-installed packages on top of the base images. This is a very cool concept and reduces the issues that could arise by booting first into the base environment without the user packages.

Note: If one is so inclined, you can even run this ASU API on your own server, thus reducing the reliance on a third-party software. I haven’t gotten to that point yet, but would be relatively easy to implement. More here.

The concept here is relatively simple:

• Send a (JSON-formatted) build request to the ASU API
• Wait for the build to complete
• Download the resulting root filesystem image
• Unzip said image to the target device

Build the build-request

The request must be formatted correctly, which is described in the API’s docs. It is a little dense, but with the help of awk, we can pass in our installed-packages list, and the desired version:

# Build json for Image Builder request
awk -v new_release=${new_release} '{
    items[NR] = $1
}
END {
    printf "{\n"
    printf "  \"packages\": [\n"
    for (i = 1; i <= NR; i++) {
        printf "    \"%s\"", items[i]
        if (i < NR) {
            printf ","
        }
        printf "\n"
    }
    printf "  ],\n"
    printf "  \"filesystem\": \"ext4\",\n"
    printf "  \"profile\": \"generic\",\n"
    printf "  \"target\": \"x86/64\",\n"
    printf "  \"version\": \"%s\"\n", new_release
    printf "}\n"
}' installed-packages > json_data

This script creates a minimal json file that we will send with our request to tell the image builder which packages and which version of OpenWrt to build it with. I have hard coded everything into x86 and ext4, but those could easily be put into variables if needed. There are also loads more options that you can find in the above-mentioned docs.

Send the request

Since the built-in version of wget does not allow us to customize the headers we send with the request, we will need curl. This is why it is a dependency needed at the start of the script.

curl -H 'accept: application/json' -H 'Content-Type: application/json' --data-binary '@json_data' 'https://sysupgrade.openwrt.org/api/v1/build' > build_reply
build_status=$(cat build_reply | jsonfilter -e '@.status')
if [ $build_status == 202 ] || [ $build_status == 200 ]; then
  build_hash=$(cat build_reply | jsonfilter -e '@.request_hash')
  printf "Request OK. Request hash: %s\n" "${build_hash}"
else
  echo "Error requesting Image build:"
  cat build_reply
  umount ${mount_pt}
  exit 1
fi

This is a pretty standard curl POST. Afterwards, you’ll notice I do a check to see the status code:

• 200 = Build request received, build is complete
• 202 = Build request received, build is in the queue/in-progress
• All else = Problem with the request

Check the build status

There’s a different API to check the status of our build, also well documented in the API docs. The docs ask us to not hit the API more than once every 5 seconds. I have implemented a little visual spinner while we are waiting for the build request to change from code 202 to 200:

i=0
spin='-\|/'
build_time=0
while [ true ]; do 
  # Sleep to comply with API rules
  sleep 6
  building=$(curl -s "https://sysupgrade.openwrt.org/api/v1/build/${build_hash}")
  build_status=$(echo $building | jsonfilter -e '@.status')
  # 202 = in-progress
  if [ $build_status == 202 ]; then
    i=$(( (i+1) %4 ))
    build_time=$(( build_time + 6 ))
    printf "\rWaiting for build to complete ${spin:$i:1}"
    continue
  # 200 = build complete
  elif [ $build_status == 200 ]; then
    image=$(echo $building | jsonfilter -e '@.images[@.filesystem="ext4" && @.type="rootfs"].name')
    hash=$(echo $building | jsonfilter -e '@.images[@.filesystem="ext4" && @.type="rootfs"].sha256')
    image_hash="${hash} ${image}"
    printf "\nBuild finished in %d seconds\n" "${build_time}"
    break
  else 
    # Sleep an extra 5 seconds to not hit the API back-to-back just to report an error
    sleep 5
    printf "\nError with Image Builder:\n"
      curl "https://sysupgrade.openwrt.org/api/v1/build/${build_hash}"
    exit 1
  fi
done

Download resulting image

Pretty straightforward, using the build hash given to us at the time of the initial request:

cd /tmp
# Download new release
wget "https://sysupgrade.openwrt.org/store/${build_hash}/${image}"
# Check sha256 hash against file downloaded
csum=$(echo $image_hash | sha256sum -c | awk '{ print $2 }')
printf "Checksum %s!\n" "${csum}"
if [ $csum != "OK" ]; then
  # If hash doesn't match, exit
  printf "Downloaded image doesn't match sha256sum! Exiting...\n\n"
  umount ${mount_pt}
  exit 1
else
  printf "---Image downloaded and hash OK. Installing---\n"
fi

The build output also gives us a sha256 checksum, so I do a quick check of the downloaded image before actually installing it.

Unzip and write the image

Also a relatively straightforward procedure.

# Unzip and write directly to partition
gzip -d -c ${image} | dd of=${target_dev}
# Unmount partition to resize without error
umount ${target_dev}
# Check filesystem for errors
e2fsck -fp ${target_dev}
# Resize filesystem to partition size
resize2fs ${target_dev}
# Check partition for errors
fsck.ext4 ${target_dev}
# Remount target device
mount ${target_dev} ${mount_pt}

After unzipping the root filesystem, there tends to be an error with the partition table? The e2fsck command repairs the error. We resize the filesystem to the partition size (otherwise you’ll only have ~ 128 MB drive), and finally do a filesystem check. I don’t believe the final check is necessary, but it can’t hurt right?

Alright, so now we have the new version of OpenWrt installed with our desired packages. Next we’ll need to update the boot environment to ensure we load the new OS with the updated kernel.

5) Remove old kernel(s), install new kernel

Removal

This section was a bit tricky, but relies on grabbing the currently running version’s kernel name from your grub.conf, while deleting the rest.

mkdir -p /tmp/boot
# Mount /boot into a tmp directory
mount ${boot_dev} /tmp/boot
# Check if more that one kernel exists
num_kernels=$(find /tmp/boot/boot/ -name *vmlinuz* | wc -l)
if [[ $num_kernels > 1 ]]; then
  current_root_partuuid=$(lsblk -pPo MOUNTPOINTS,PARTUUID | grep 'MOUNTPOINTS="/"' | sed -E 's/.*PARTUUID="(.*)".*/\1/')
  current_kernel=$(grep ${current_root_partuuid} /tmp/boot/boot/grub/grub.cfg | sed -E 's/.*linux \/boot\/(.*) .*/\1/g' | cut -d " " -f 1 | uniq)
  find /tmp/boot/boot -name *vmlinuz* ! -name ${current_kernel} -exec mv {} /tmp \;
else
  echo "One existing kernel found, not deletion required"
fi

If there is currently only a single kernel installed, nothing will get deleted. One potential point that could break this script, would be if there is not enough room on your /boot partition to install an additional kernel.

Installation of new kernel

Once again pretty simple. We will be downloading the kernel that matches the new version of OpenWrt we installed previously, and naming the new kernel with the release version.

new_kernel=vmlinuz-${new_release}
wget https://downloads.openwrt.org/releases/${new_release}/targets/x86/64/openwrt-${new_release}-x86-64-generic-kernel.bin -O /tmp/boot/boot/${new_kernel}

6) Update GRUB with new kernel and partition ID’s

This is the section that took me the longest to accomplish. There are many “operations” we do quite easily and quickly in text editors, but becomes a lot more challenging when doing it via an automated script.

While I could simply stick to the basic ID’s like /dev/sda2, I was always taught that it’s better to use partition ID’s in case you have other drives attached, etc.

# Get new partition UUID
new_partuuid=`lsblk -pPo PATH,PARTUUID | grep ${target_dev} | sed -E 's/.*PARTUUID="(.*)".*/\1/'`

# Create a backup copy of grub in case something fails
cp /tmp/boot/boot/grub/grub.cfg /tmp/boot/boot/grub/grub.cfg.bak

# Copy the first menu entry to create an additional entry
sed -i '1,/menuentry/{/menuentry/{N;N;p;N}}' /tmp/boot/boot/grub/grub.cfg
## Update the first menu entry
# Name 
sed -i "1,/menuentry/s/\"OpenWrt.*\"/\"OpenWrt-${new_release}\"/" /tmp/boot/boot/grub/grub.cfg
# Kernel
sed -i "1,/linux/s/vmlinuz[-0-9.]*/${new_kernel}/" /tmp/boot/boot/grub/grub.cfg
# Partition
sed -i "1,/linux/s/PARTUUID=[-0-9a-f]*/PARTUUID=${new_partuuid}/" /tmp/boot/boot/grub/grub.cfg

# Leave the second entry as is - the current working (old) version

Before I move on to the second half of the section, I wanted to point out a couple of things I’m doing above. I start by getting the new partition’s UUID. Next, I’m creating a backup of grub in case we really messed up.

From here, I am using a trick with sed to find the first GRUB menu entry, and copying it below. This ensures that the currently-running version is preserved. The technique uses sed’s “pattern” space. Once the word “menuentry” is found, we are appending the ‘found’ pattern with lines above and after the pattern itself - that’s the {N;N;p;N}.

The next 3 sed commands are more standard “substitutions” that update the new GRUB entry name, kernel, and partition uuid.

Next, we’ll need to trim any old grub entries.

# If there are now 4 menu entries, delete the 3rd (oldest version)
grub_entries=`grep menuentry /tmp/boot/boot/grub/grub.cfg | wc -l`
if [[ grub_entries == 4 ]]; then
  awk 'BEGIN {count=0} /menuentry/ {count++} count!=3' /tmp/boot/boot/grub/grub.cfg > tmp && mv tmp /tmp/boot/boot/grub/grub.cfg
fi

## Update failsafe entry
# Copy (new) first entry to the end of grub, add failsafe
sed -n '1,/menuentry/{/menuentry/{N;N;p}}' /tmp/boot/boot/grub/grub.cfg | sed -E 's/(\"OpenWrt-.*)\"/\1 \(failsafe\)"/' | sed -E 's/(^.*)(root=PARTUUID=.*$)/\1failsafe=true \2/' >> /tmp/boot/boot/grub/grub.cfg
# Delete the old failsafe entry
sed -i '1,/failsafe/{/failsafe/{N;N;d}}' /tmp/boot/boot/grub/grub.cfg

# Since we used awk to replace the file, restore original permissions
chmod 755 /tmp/boot/boot/grub/grub.cfg

Ugh, this was another challenging one. Since sed (my preferred editor) doesn’t do well at only editing say the third entry, I switched to awk. I am not as proficient in awk, but slowly learning! In this case, if we have 4 entries (new, current, old, and failsafe), we want to delete the third entry. If your GRUB config file is “non-standard”, there’s a chance this could break and you’d end up with too many entries, or non-working entries!

# If there are now 4 menu entries, delete the 3rd (oldest version)
grub_entries=`grep menuentry /tmp/boot/boot/grub/grub.cfg | wc -l`
if [[ grub_entries == 4 ]]; then
  awk 'BEGIN {count=0} /menuentry/ {count++} count!=3' /tmp/boot/boot/grub/grub.cfg > tmp && mv tmp /tmp/boot/boot/grub/grub.cfg
fi

## Update failsafe entry
# Copy (new) first entry to the end of grub, add failsafe
sed -n '1,/menuentry/{/menuentry/{N;N;p}}' /tmp/boot/boot/grub/grub.cfg | sed -E 's/(\"OpenWrt-.*)\"/\1 \(failsafe\)"/' | sed -E 's/(^.*)(root=PARTUUID=.*$)/\1failsafe=true \2/' >> /tmp/boot/boot/grub/grub.cfg
# Delete the old failsafe entry
sed -i '1,/failsafe/{/failsafe/{N;N;d}}' /tmp/boot/boot/grub/grub.cfg

# Since we used awk to replace the file, restore original permissions
chmod 755 /tmp/boot/boot/grub/grub.cfg

The first part of the awk command uses a named variable that we count. We then begin coping each line when the count does not equal 3. Each time the word “menuentry” is found, the count is increased by 1. This will effectively skip the third entry and copy them into a new file.

The final steps create a new “failsafe” entry and delete the old one. Lastly, we want to ensure the grub file has the correct permissions since we “recreated” the file using the awk command.

7) Copy config and files from current to new version

This step ensures any of our current working config(s) make it across to the new installation.

cp -au /etc/. /tmp/mnt/etc

I learned that the . after /etc/ is important for the command to copy the contents of /etc and not the /etc directory itself (otherwise we end up with /etc/etc!).

Other files

As per the normal “backup”, we want to keep whatever you’ve got in your sysupgrade.conf file. This is the file containing anything you’ve put into your System –> Backup –> Configuration section of the UI. This section is a little longer in case the containing directory doesn’t exist (like /opt).

for file in $(awk '!/^[ \t]*#/&&NF' /etc/sysupgrade.conf); do 
  directory=$(dirname ${file})
  if [ ! -d $directory ]; then
    mkdir -p "/tmp/mnt${directory}"
  fi 
  cp -a $file /tmp/mnt$file
done

TLDR

Link to the script.

The installation and usage of the script is pretty simple. Download it to your router, ensure it’s executable (chmod +x), and then run it. If you have lots of user-installed applications, it might take a while for the build system to complete. Patience.

What about the other /dev/sdX devices?

Let’s give a concrete example of this to start:

Personally, I use AdGuard Home as my DNS resolver and ad blocker. But of note, the version in the OpenWrt repositories is 10+ releases behind the current release. Thus, I run it from /dev/sda4 which I have mounted to /opt.

I originally tried having it auto-mount using /etc/fstab like it other Linux distributions, but for OpenWrt, it’s a little different. We must use the block-mount package and /etc/config/fstab as described in the docs.

This allows for non-OpenWrt software to easily migrate between upgrades.

Conclusion

Upgrading my (non-x86) OpenWrt devices has nearly always been a breeze. But the process for the x86 boxes is unnecessarily frustrating. I wrote this script as a way to mitigate some of the frustration and share it with others who might be doing the same thing. I hope you have found the article (and more importantly the script itself!) helpful.

Comments, constructive criticism, and ideas are always welcome.