pvr allow you to create squash file systems used by pantavisor as a read-only file system for the running containers, but sometimes if you have a big container, as could be the case for a full fetch ubuntu container, the resulting squashfs could be 900mb+, and therefore every time you deploy a new revision of your container, even though you only change 1 file inside the container, the latest update will download the whole new squashfs, that means will use more storage on your pantacor hub account, will be slower because it needs to download a bigger object and will use more disk on the board to maintain a previous version to rollback.
That is why we have a new feature in our development branch of pvr, that allows you two this:
- Create patch updates to your containers: That means it will create overlay squashfs that only have the files that change against the base squashfs in the container.
- Create an application container as an extension of another application: This will create a new application that only has the overlay squashfs with the diff between the base container and the new one. This will be useful if you have several containers that have the same base dependencies like libraries, binaries, and more. but they differ in a couple of configuration files or new packages installed.
Requirements
To test this new feature you will need to have pvr installed (how to install pvr) and you will need to switch to the development branch and do a self-upgrade of your pvr cli.
pvr global-config set DistributionTag=develop
pvr self-upgrade
How to create patches for your existing applications.
This new feature is early and easy to use you only need to add --patch argument when you run the app update command.
Example:
pvr app update --patch mycroft
Case of study
To see the difference that can make only sending the overlay patch instead of the whole squashfs less take the mycroft example.
Mycroft is an Open Source project for a speaker assistant called Mark II. The Mark II is a state-of-the-art, privacy-respecting smart speaker designed for you and your family, from kids to grandparents. The Mark II provides a great voice experience without sacrificing privacy.
In this product repository, where you can find out how they build their container to manage the whole Mark II device.
Then lest take this device as an example:
https://hub.pantacor.com/u/mycroft-ci/devices/62cc35779340a8000a387632/step/163
We can clone using:
pvr clone https://pvr.pantahub.com/mycroft-ci/myc200_dev/163 myc200_dev
cd myc200_dev
This will generate a folder structure like this:
[4.0K] .
├── [4.0K] awconnect
│ ├── [ 695] lxc.container.conf
│ ├── [108M] root.squashfs
│ ├── [ 126] root.squashfs.docker-digest
│ ├── [ 510] run.json
│ └── [1.1K] src.json
├── [4.0K] bsp
│ ├── [ 348] build.json
│ ├── [ 28M] firmware.squashfs
│ ├── [ 22M] kernel.img
│ ├── [ 15M] modules.squashfs
│ ├── [3.3M] pantavisor
│ ├── [ 172] run.json
│ └── [ 106] src.json
├── [4.0K] _config
│ └── [4.0K] awconnect
│ ├── [4.0K] etc
│ │ ├── [4.0K] modules-load.d
│ │ │ └── [ 85] mark-ii.conf
│ │ └── [4.0K] wifi-connect
│ │ └── [ 54] config.json
│ └── [4.0K] opt
│ └── [4.0K] wifi-connect
│ └── [4.0K] ui
│ └── [4.0K] build
│ └── [4.0K] static
│ ├── [ 17K] favicon.ico
│ ├── [4.8K] favicon.png
│ └── [7.5K] theme.css
├── [4.0K] mycroft
│ ├── [ 903] lxc.container.conf
│ ├── [ 417] mdev.json
│ ├── [978M] root.squashfs
│ ├── [ 88] root.squashfs.docker-digest
│ ├── [ 388] run.json
│ ├── [1.1K] src.json
│ └── [ 788] v0alpha.app.manifest.json
├── [ 2] network-mapping.json
└── [ 2] storage-mapping.json
13 directories, 26 files
You can see the mycroft/root.squashfs with 978M and the root.squashfs.docker-digest that save the docker digest of the source container. When we run the pvr app update the cli will check what is the digest of the latest container in the same tag, if is different is going to download the new container and create the squashfs, but when we use the --patch instead of creating a full root.squashfs it will create a root.ovl.squashfs a lot smaller.
[4.0K] mycroft/
├── [ 938] lxc.container.conf
├── [ 417] mdev.json
├── [ 32K] root.ovl.squashfs
├── [ 88] root.ovl.squashfs.docker-digest
├── [978M] root.squashfs
├── [ 88] root.squashfs.docker-digest
├── [ 322] run.json
├── [1.1K] src.json
└── [ 788] v0alpha.app.manifest.json
Only 32K of root.ovl.squashfs will be the only part that will be sent to the device as an update, making the process of updating a lot more faster and efficient. An using the unsquashfs command we can see the files that changed between the two container versions.
$ unsquashfs -l mycroft/root.ovl.squashfs
squashfs-root
squashfs-root/etc
squashfs-root/etc/mycroft
squashfs-root/etc/mycroft/build-info.json
squashfs-root/usr
squashfs-root/usr/share
squashfs-root/usr/share/fonts
squashfs-root/usr/share/fonts/truetype
squashfs-root/usr/share/fonts/truetype/noto-sans
squashfs-root/usr/share/fonts/truetype/noto-sans/.uuid
squashfs-root/var
squashfs-root/var/cache
squashfs-root/var/cache/fontconfig
squashfs-root/var/cache/fontconfig/1055c6cb-ce21-4f17-83fb-04817aec2d90-le64.cache-7
squashfs-root/var/cache/fontconfig/80708cb0-69bf-42ac-80c2-199c7c9ba90a-le64.cache-7
squashfs-root/var/cache/fontconfig/cafc8e5e-8030-4ab8-ac42-6431b99e4ad7-le64.cache-7
Making this method a great option to update your application inside a pantavisor device. If after several updates have an overlay that is growing to a point that is almost the same size as the root.squashfs we can reset the system to create a new root.squashfs that will be used in the future to create the new patches. In order to do this we need to run:
pvr app update --newbase mycroft
How to create extensions applications
Creating applications that extend from another container is pretty easy. You need to follow two steps:
- Create the container that will be used as the base for the other applications.
pvr app add --group=data --status-goal=MOUNTED --from=nginx:stable-alpine nginx
- Create the extension application using the argument
--baseto refer to the application that we will extend.
You need to create this container as a normal extension in your Dockerfile
FROM nginx:stable-alpine
COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY index.html /usr/share/nginx/html/index.html
And then you will create a new app inside your device description using pvr with the --base argument.
pvr app add --base nginx --from=highercomve/hello_world_nginx:latest hello_world_nginx
This will create two folders inside the device definition structure.
├── [4.0K] nginx
│ ├── [7.0M] root.squashfs
│ ├── [ 77] root.squashfs.docker-digest
│ ├── [ 288] run.json
│ └── [1.2K] src.json
├── [4.0K] hello_world_nginx
│ ├── [ 697] lxc.container.conf
│ ├── [4.0K] root.ovl.squashfs
│ ├── [ 101] root.ovl.squashfs.docker-digest
│ ├── [ 324] run.json
│ └── [ 962] src.json
As you can see the second container is only 4.0K and the named root.ovl.squashfs if we run a
unsquashfs -l hello_world_nginx/root.ovl.squashfs
We can see the only two files that are inside the root.ovl.squashfs are the ones we copied inside the Dockerfile of the second container.
squashfs-root
squashfs-root/etc
squashfs-root/etc/nginx
squashfs-root/etc/nginx/conf.d
squashfs-root/etc/nginx/conf.d/default.conf
squashfs-root/usr
squashfs-root/usr/share
squashfs-root/usr/share/nginx
squashfs-root/usr/share/nginx/html
squashfs-root/usr/share/nginx/html/index.html
Final thoughts
We hope these two new features will allow all the developers inside the pantavisor community to manage better their devices resources and update process.