I’ve been trying for 2 weeks, unsuccessfully, to get meta-pantavisor to work with my imx8 based system.
Initially I had issues with my kernel not having the requisite modules built in for LXC (most of them are built as loadable modules in the BSP package), but even now I’m struggling. After including bridge support and a few others into the kernel itself, things are progressing a bit further, but still not starting.
the initramfs with pantavisor doesnt boot as is. I’ve made my own initramfs which gives me a debug shell to run pantavisor myself and observe the output and logs.
a few questions:
Does pantavisor ever try to load modules form the bsp squashfs’s?
Should the network interface be up before pantavisor starts? I’m assuming eth0 being down is fine as pantavisor is supposed to be pid 1. I notice it starts dropbear on lo, (i’m guessing this is for the container debug interface?) but it seems the SSH server which ‘pvr device scan’ searches for is mounted from the pvr-sdk container, right?
I have an issue with the console tty. I suppose this is a imx8 quirk rather than anything else. normally, the imx8 seems to use getty and systemd to organise the tty’s, but starting a new initramfs, ‘tty’ gives me ‘not a tty’. However, /dev/console and /dev/ttymxc0 (the device) work fine. I can get around this by running pv as ‘exec pantavisor </dev/console >/dev/console 2>/dev/console’, after mdev has done its thing and the network is up.
I’ve tried building the initramfs with udev instead of mdev, but no difference.
Beyond that, i still seem to have a mounting issue.
The container log shows: “/lib/pv/hooks_lxc-mount.d/remount.sh” for container “connman” … ERROR … Script exited with status 1"
And the console output shows “overlayfs: failed to set xattr on upper”. I find this strange as I tried making a overlayfs and that works fine. Any idea what this could be?
Sorry for the incovenience. Things are still in flight. The layer was originally developed for an imx8 var platform so it cant be that far away, but I moved on since to a different platform which might have caused some damage … let me check how things are and get back to you to avoid more time sink for you…
okay, i’ve learned some things.
I got things working by disbabling the ‘set -e’ on the remount hook, and disabling the mdev remount hook.
The problem with the mdev seems to be:
busybox mktemp freaks out if there’s no /tmp
for some reason evaluating the ENV in the mdev.sh hook overrides the PATH so the script cant find JSON.sh.
OK i have finished refreshing the layer for the imx8 var platform… please pull the latest from refactor1 branch please.
In general most things are probably follow ups from some early problem or misconfiguration during build.
Key things to check:
in local.conf you need PANTAVISOR_SYSTEM_MODE to enable the Xrecipes-core recipes. I have these:
### BEGIN META PANTAVISOR
IMAGE_CLASSES += "bspimage-pvrexport"
KERNEL_CLASSES += "pvbsp"
GOVERSION = "1.20%"
PANTAVISOR_SYSTEM_DISTRO = "yes"
### END META PANTAVISOR
Those would bring all the drivers as built-in needed for lxc and pantavisor through linux…bbappend. Also it will bring a patched busybox from Xrecipes-core that is needed to get a fully working experience.
Also you need to boot things with our boot script which works well with distro_bootcmd. So if you can your CONFIG_BOOTCMD please use “run distro_bootcmd” there. You might still have issues as there is no real standard about variables, so I try to add fallbacks etc. to the single script rather then creating a new one for each variant. If you still have issues maybe post a log from your early boot so i can see whats missing.
Also if you want to not run pantavisor as init, but rather try from shell, using our boot.scr, you can setenv localargs rdinit=/bin/sh and then from shell run /usr/bin/pantavisor pv_standalone to run pantavisor system mode from shell.
we built-in all essentials for us and lxc if you have PANTAVISOR_SYSTEM_MODE in your local.conf
we have in our official builds now an early mdev option that runs before rootfs is found. This feature is not yet in yocto tree, but most likely you don’t need it if your backing store can be mounted without modules.
We offer an mdev.json that a container can ship that will allow devices to be mapped and hotplugged into the container. I will add an example to our yocto layer soon to show that. This could probably also be used to do late auto driver hotplug loading…
further we offer a declarative drivers framework where bsps can offer aliases that containers can ask for and then pantavisor loads the drivers on demand as needed etc… Managed Drivers - Docs
Just seeing that this is not working with the pantavisor tree in yocto right now (will have to rebase against master i guess) … but rdinit=/usr/bin/pantavisor should boot with our bootscript (you can press D during countdown and get to shell )…
Ah, thanks. I somehow didn’t notice the kernel configs in the x-recipes dir. Not that it would matter much as linux-imx ignores the cfg segments… I’ll try and let you know.
OK, i just dry-tried spinning linux-imx 5.15 here and that seemed to have picked up the the .cfg judging from the .config that popped out … keep me posted. thx.
Hi Asac, my build is now mostly working, but I’m trying to add an image as pvr container and its failing.
I get an error message saying: Failed to parse config file "/storage/trails/0/pv-avahi/lxc.container.conf" at line "lxc.tty.min = 0"
How do I modify the contents of the lxc.container.conf file from the recipe?
Also, the system is wanting to reboot the moment there is an issue with any of the containers. Is there a way to tell pantavisor to run all the images that are fine and ignore the broken ones so I can at least get to the pvr-sdk to modify them?
Also, although I have my device running with a pvr-sdk image, I still can’t connect it to pantahub, and I’m unsure how to install/update containers to a local device, locally or through pantahub. Could you shed some light on this?
Failed to parse config file "/storage/trails/0/pv-avahi/lxc.container.conf" at line "lxc.tty.min = 0"
hmm i pushed pvr too early. sorry. the matching meta-pantavisor are still local. fixed pvr by making the feature optional in latest meta-pantavisor refactor1 branch
Also, the system is wanting to reboot the moment there is an issue with any of the containers. Is there a way to tell pantavisor to run all the images that are fine and ignore the broken ones so I can at least get to the pvr-sdk to modify them?
There is room for improvement from an app lifecycle feature perspective, but you can enter debug shell by pressing [d] during bootup when you see the countdown. This will prevent the system for doing the reboot due to container or other upgrade/bootup failures until you exit that shell. This is then a good place to investigate and learn how to fix things.
Also, although I have my device running with a pvr-sdk image, I still can’t connect it to pantahub, and I’m unsure how to install/update containers to a local device, locally or through pantahub. Could you shed some light on this?
I worked with local experience on yocto front so far, just checked and yes remote to pantahub is broken right now, most likely due to lack of certs… will try to fix it…
to allow you to pvr clone YOUR.IP.ADD.RRESS you would have to open the pvr-sdk httpd to the outside world like this:
Environment size: 5989/16380 bytes
u-boot=> setenv localargs pvrsdk_httpd_listen=0.0.0.0
u-boot=> saveenv
Saving Environment to MMC... Writing to MMC(2)... OK
u-boot=> run bootcmd
With that you can pvr clone and post directly to and from the IP … you can also use the pvtx web UI from local device available on IP:12368/app … this allows you to upload pvrexport.tgz tarballs rather than the pvr protocol, but also look at logs etc. (also offered through REST API documented in the web UI).
Thanks.
I was able to enable the web-ui and do some API transactions, but adding an image still failed. When I try to commit the image.tgz transaction, I just get { "Error": "State verification has failed" }
Also, regarding ‘claiming’ the device, I see that both /pv/device-id and /pv/challenge are blank. I’m guessing that’s one reason why it can’t be added to pantahub…
I think for now I’m giving up on pantavisor.
Hi Kobus, in case you still have notifications enabled. The meta layers don’t bring our generic development certificates at the moment [1], so you are required to run with disabled secureboot [2].
Passing pv_secureboot.mode=0 in the command line to the kernel should be enough for your system to come up. Let us know if you have any other questions.
@anibal Hi Anibal, could we give some thought to making the state verification errors more verbose to help pinpoint the cause? The final error should list which components failed to verify and why, the why can be signature mismatch or non-existent certs, or a non-descript error.
We are working right now on more explanatory status messages for these types of signature errors. Other types of update errors that are also poorly presented will follow too!
Those will make installation of containers more quick, as the user will have more detailed feedback in case of update failure, but the best way to fully understand what is happening on the device is always to read the Pantavisor logs
