| *** iurygregory_ is now known as iurygregory | 11:38 | |
| JayF | [proxied from kubajj on Metal3 slack] is anybody running diskimage-builder inside containers? I am facing create_inode permission issues in unsquashfs run, so I wonder what extra requirements might apply | 13:44 |
|---|---|---|
| JayF | I assume we've run DIB in a container before so you all might know the answer | 13:45 |
| JayF | kubajj: I proxied over your question already | 13:50 |
| JayF | Hopefully someone knows :) | 13:50 |
| JayF | kukacz: ^ I meant you, I proxied your question over already | 13:51 |
| kukacz | ah, thanks a lot JayF! | 13:51 |
| kukacz | I should add that I'm running podman inside WSL, specifically. | 13:53 |
| JayF | yeah, my hunch was it might be a podman thing | 13:54 |
| JayF | because podman does try to run with reduced permissions | 13:54 |
| kukacz | I have used --privileged parameter. the whole command like `podman run --name dib --rm -it --env-host -v .:/build --privileged dib/dib`, where dib/dib is my custom built ubuntu jammy image with all the build prerequirements | 14:00 |
| JayF | yeah but podman is non-root, right? | 14:22 |
| JayF | this may be something you need system-root to be able to do, but I'm not sure | 14:23 |
| JayF | clarkb: Have you heard any success stories of dib use in containers? | 14:23 |
| kukacz | yes, the podman I run is non-root. I suspect the issue might be around devicemapper access to some device nodes | 14:31 |
| clarkb | JayF: we (opendev) run all of our dib inside of containers | 14:46 |
| clarkb | we run the zuul/nodepool-builder container with docker and docker-compose and that container executes dib to build images | 14:47 |
| clarkb | also the dib functional test jobs all do ^ that too | 14:47 |
| clarkb | I can't speak to wsl but I think we've also had iterations that ran inside of podman that were rolled back for unrelated reasons | 14:47 |
| JayF | yeah I would not be surprised if it's not possible :/ | 14:48 |
| JayF | (in rootless podman) | 14:48 |
| clarkb | I think you do need elevated permissions because dib needs stuff like /dev access | 14:48 |
| kukacz | clarkb: great to hear it works somewhere, thanks! could you point me at any env details or the container run configuration? | 14:52 |
| kukacz | I am running privileged, but that's not enough. so any inspiration of the privilege parameters or mappings used would help me a lot | 14:52 |
| kukacz | docker and podman parameters are more or less same, they're pretty standardized across all runtimes I believe | 14:53 |
| clarkb | kukacz: https://opendev.org/zuul/nodepool/src/branch/master/playbooks/nodepool-functional-container-openstack/run.yaml#L89-L123 this is the functional test I was talking about. Looks like it does use podman | 14:57 |
| clarkb | that runs a tool called nodepool-builder which uses dib to build images and upload them to cloud providers. So it is a layer removed but dib is still running inside of a container | 14:57 |
| kukacz | thank you guys. it's aparently good direction. I've chunged podman to run as a system service (rooted) and now I'm one error further ;-) | 15:12 |
| kukacz | as WSL identifies itself in uname using some -WSL specific kernel version, dib is trying to find the kernel headers. I'll try to satisfy it | 15:14 |
| opendevreview | Merged openstack/diskimage-builder master: Reduce LVM extent usage https://review.opendev.org/c/openstack/diskimage-builder/+/930950 | 16:27 |
| shermanm | hey, I wanted to ask about the reasons behind having `nofb nomodeset gfxpayload=text` as the defaults for kernel commandline in DIB | 16:46 |
| shermanm | I recently found that unsetting `nomodeset` resolved issues we were seeing in amdgpu drivers on ubuntu, but wasn't sure if there were other side-effects | 16:46 |
| clarkb | shermanm: I suspect that is an artifact of a couple of things. First up nomodeset increases compatibility with various hardware at the expense of functionality. So that ensures things are more likely to work. But also I think a lot of us use dib with VMs which doesn't relaly have hardware backed graphics (though this is slowly changing) | 16:50 |
| clarkb | "Apply kernel command line parameters to force Linux to remain in text mode. Typical distro kernels quickly switch into a graphical screenmode and this is incompatible with common Lights Out Management hardware which is able to intercept text mode consoles to display them remotely." | 16:52 |
| clarkb | so ya compatibility reasons though with ilo I didn't expect that | 16:52 |
| shermanm | yeah, I did see that the default that Ironic uses for pxe booting recently changed to remove it https://opendev.org/openstack/ironic/commit/f2605e928136f88032ef9b4977a076d5c535987b | 16:53 |
| shermanm | our specific issue was that with amd MI series datacenter GPUs, sometimes installing the ROCM software stack would render the image unbootable. more recent versions of it simply fail to detect the GPUs with nomodeset enabled | 16:54 |
| shermanm | and I'm not advocating for the default to be changed or anything, just kind of gathering context while I put together new defaults for our (primarily baremetal) iamges | 16:55 |
| clarkb | thats interesting. nomodeset is like graphical safe mode and it should be more compatible just without functionality. I had to run my laptop recently that way when the gpu on it was dying until I could get it RMA'd | 16:55 |
| clarkb | I'm not an expert on how that is supposed to work but I think there is some base set of functionality GPUs are expected to always have and then when you boot nomodeset you use a general driver to drive that functionality | 16:56 |
| clarkb | if you boot ubuntu in "safe mode" you get nomodeset on the command line for this reason | 16:56 |
| clarkb | I suspect that booting in text mode is still a good idea for ilo compatibility but with newer hardware maybe dropping nomodeset will work? | 16:58 |
| shermanm | yeah, I think we'll end up keeping `gfxpayload=text`, but dropping nomodeset and possibly nofb. I've had some issues with nofb reported to me in the past, but I can't seem to find the details at the moment | 17:10 |
| JayF | clarkb: not just ilo, basically anything with serial console redirection, aka any ipmi | 17:17 |
| kukacz | getting back to my podman in WSL build - I don't understand why dib is searching for build host kernel headers. since it is inside WSL, the headers are specific and not provided by Microsoft. is it really necessary or I can apply some parameter to point the build at different kernel for example? | 17:35 |
| JayF | the build itself happens in a chroot iirc, so I suspect what may be happening is more like: - something inside the chroot needs kernel headers, a package being installed or whatever -- it runs `uname -a`, sees the WSL kernel, and looks for those headers | 17:39 |
| JayF | This is a hypothesis/conjecture | 17:39 |
| clarkb | yes dib itself shouldn't care I don't think | 17:40 |
| clarkb | its likely something in your build that you need to look at. | 17:40 |
| kukacz | hmm, just thinking how to track it down. perhaps I should disable the additional elements and try with plain ubuntu image? | 17:45 |
| JayF | If you can share logs, we might be able to point at the output, but I'd generally expect it to be a package install or similar | 17:46 |
| kukacz | I'm running the metal3 projects ci build script: https://github.com/metal3-io/project-infra/blob/main/jenkins/image_building/build-image.sh | 17:46 |
| clarkb | dib should log fairly verbosely | 17:47 |
| clarkb | kukacz: I think you can add the -x flag to dib to have it trace further | 17:50 |
| kukacz | clarkb: ok, I'll add that. can I make it save the output to logfile? | 17:53 |
| clarkb | yes if you redirect sterr/stdout to the file | 17:53 |
| kukacz | found a --logfile param, it doesn't output to screen though, so I'm faiting for the failure now :-) | 18:01 |
| kukacz | here: https://gist.githubusercontent.com/kukacz/9591aad27d737a0ad00dc3ef2c27fbfc/raw/2a3951d92bf2cf989b15986668e91ea150e93825/gistfile1.txt | 18:04 |
| clarkb | from this line: 2024-10-22 18:01:18.328 | dib-run-parts Running /tmp/in_target.d/install.d/55-install we see that dib is running an install.d/55-install script from one of your elements | 18:07 |
| clarkb | that then tries to install inux-headers-5.15.153.1-microsoft-standard-WSL2 which fails | 18:07 |
| clarkb | that appears to come from here: https://github.com/metal3-io/project-infra/blob/main/jenkins/image_building/dib_elements/ubuntu-node/install.d/55-install#L23 | 18:07 |
| clarkb | I think this is broken with or without WSL since the package is unlikely to be inux-* but instead linux-* | 18:07 |
| clarkb | using uname -r there is probably not the best idea since you're building an image for something that may not be on the same platform | 18:11 |
| clarkb | instead you want the headers package that corresponds to the kernel package within the image | 18:11 |
| kukacz | clarkb: perfect! thank you for the instant debug. now I have even more questions, just for different audience | 18:11 |
| clarkb | which I thought was installed by default | 18:11 |
| kukacz | yes, this makes no sense and since the file was updated 6 months ago, I don't get how it could have worked to anyone since then | 18:12 |
| kukacz | great that I can now tune this element file for my need to finish the build | 18:13 |
| JayF | I'll also note there are other ways to install packages, using package-installs.yaml there | 18:15 |
| JayF | but I guess the hold, I don't know if install-packages element does that | 18:15 |
| kukacz | now I am not sure whether I am using right image type ("node") for the build. it was my first choice without further inspecting it. need to check | 18:19 |
| kukacz | clarkb: and JayF: build completed. thanks a lot for your great help! | 18:24 |
| clarkb | you're welcome | 18:25 |
| JayF | o/ np | 18:48 |
| TheJulia | clarkb: the root issue behind nomodeset is a reliance on more of a textual console which gets driven by interrupts, some BMC graphics interfaces can have degraded performance impacting overall CPU time and even generating packet loss. Just best to avoid as the framebuffer seems to be the safest, highest performance, lowest cpu impact option | 19:23 |
| shermanm | would that impact the `nofb` flag as well? | 19:25 |
| JayF | nofb is the old-old-old equivalent of nomodeset I think? | 19:28 |
| JayF | but I'm not 100% sure | 19:28 |
| TheJulia | yeah | 20:08 |
| TheJulia | it is | 20:08 |
| TheJulia | AFAIK it does nothing in modern linux | 20:08 |
| TheJulia | silly thing is, you actually want the frame buffer if you have a real graphics card | 20:08 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!