| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [dnm] testing ubuntu builds https://review.opendev.org/c/openstack/diskimage-builder/+/806660 | 01:56 |
|---|---|---|
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [dnm] testing ubuntu builds https://review.opendev.org/c/openstack/diskimage-builder/+/806660 | 03:16 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [dnm] testing ubuntu builds https://review.opendev.org/c/openstack/diskimage-builder/+/806660 | 06:04 |
| ianw | mazzy: package-installs.yaml is described @ https://docs.openstack.org/diskimage-builder/2.7.0/elements/package-installs/README.html | 06:07 |
| ianw | it's a way to get packages installed without having to manually call apt/yum/dnf/zypper, etc | 06:07 |
| ianw | it works with pkg-map; see https://docs.openstack.org/diskimage-builder/latest/elements/pkg-map/README.html | 06:08 |
| ianw | basically the installer will look up the package name in pkg-map, where you can install the right packages for each distro | 06:09 |
| ianw | it batches them and de-duplicates | 06:11 |
| ianw | if you want to use jq/git in your element, they should be defined in a package-install to make sure they are there | 06:11 |
| ianw | as for assuming the base system -- there's no one rule; it depends on what the element is trying to do. some things are quite generic and work on basically any build environment, others are specific | 06:13 |
| mazzy | ianw: thanks for the explanation. | 06:33 |
| mazzy | During the testing of an element locally, how do we specify the base env? | 06:35 |
| ianw | mazzy: i'm not sure what you mean by the base env? | 08:58 |
| mazzy | ianw: maybe it's me that I still did not grasp fully how dib works | 09:02 |
| mazzy | i meant to say when i would run diskimage builder to create an image using my element that i'm designing which will be the OS where this element will run? | 09:03 |
| mazzy | also the testing of dib could be done only in linux right? | 09:51 |
| mazzy | because when I run under MacOS i get | 09:52 |
| mazzy | ❯ disk-image-create ubuntu vm 11:39:27 | 09:52 |
| mazzy | 2021-08-31 09:40:02.102 | diskimage-builder version 3.13.1 | 09:52 |
| mazzy | 2021-08-31 09:40:02.103 | Building elements: base a:ho:t:xucnp: -l checksum,no-tmpfs,offline,help,version,min-tmpfs:,image-size:,image-cache:,max-online-resize:,mkfs-options:,qemu-img-options:,ramdisk-element:,root-label:,install-type:,docker-target:,logfile: -n disk-image-create -- ubuntu vm block-device-efi | 09:52 |
| mazzy | 2021-08-31 09:40:02.142 | usage: element-info.py [-h] [--env] elements [elements ...] | 09:52 |
| mazzy | 2021-08-31 09:40:02.142 | element-info.py: error: unrecognized arguments: -l checksum,no-tmpfs,offline,help,version,min-tmpfs:,image-size:,image-cache:,max-online-resize:,mkfs-options:,qemu-img-options:,ramdisk-element:,root-label:,install-type:,docker-target:,logfile: -n disk-image-create -- ubuntu vm block-device-efi | 09:52 |
| mazzy | i'm now running on ubuntu and the elements starts to run | 10:49 |
| mazzy | ianw: does the package install work during root.d? | 10:57 |
| ecsantos | mazzy: just to address some of your questions: 1. You need to have one "distro element", such as ubuntu, ubuntu-minimal, centos, centos-minimal, and so on, that's what'll define the image's OS (if I understood your question correctly) | 15:29 |
| ecsantos | 2. Yes, supported host distros are listed here: https://docs.openstack.org/diskimage-builder/latest/user_guide/supported_distros.html#supported-distributions | 15:29 |
| mazzy | ecsantos: thanks | 15:30 |
| ecsantos | Maybe it's possible to run it on macOS, but it's not officially supported | 15:30 |
| mazzy | i gave up actually. i'm running it now on ubuntu and i made some progress | 15:30 |
| mazzy | but i have so many blockers :-D | 15:31 |
| mazzy | let's see if you might help to solve them | 15:31 |
| ecsantos | 3. I believe the elements are applied to the image in a sequential manner, so I don't think the package-installs element is applied during the root.d phase of the distro element (say, ubuntu) | 15:32 |
| mazzy | ah make sense. this is the reason why it does not work | 15:32 |
| ecsantos | I just started with dib too so take my words with a grain of salt :D | 15:32 |
| mazzy | another issue i'm having is the size of | 15:32 |
| mazzy | tmpfs 3.9G 3.2G 786M 81% /tmp/dib_build.m4YD3Jcv | 15:32 |
| mazzy | i'm trying to enlarge that but w/o luck. - according the doc --image-size should work but it does not work for me | 15:33 |
| ecsantos | about package-installs: I had this doubt too, where should I declare the packages I need to install? What I did was create a custom element with just a package-installs.yaml file listing the packages I wanted, and it worked | 15:34 |
| ecsantos | maybe try the growroot element | 15:34 |
| mazzy | but package-installs does not install package in the OS where you run the command disk-image-create | 15:34 |
| mazzy | it installs from what i understood inside the chroot. | 15:34 |
| mazzy | did i get correctly? | 15:35 |
| ecsantos | yes exactly, it's intended to install packages in the target image | 15:35 |
| mazzy | ah ok now it makes sense | 15:35 |
| mazzy | also the other problem is how dib understand which is the target image? | 15:36 |
| mazzy | because I have my target image inside the chroot env dib get in | 15:36 |
| mazzy | I get the following error while running dib cerate https://pastebin.com/quAtmkfp ) | 17:25 |
| mazzy | I assume python is an hard requirement in order to complete the build process. can anyone conferm? | 17:25 |
| ecsantos | mazzy: dib builds an image called "image.qcow2" in the directory you run the disk-image-create command by default. you can change this by passing -o /path/to/image.qcow2 | 17:48 |
| ecsantos | regarding your error, if you're going to run dib directly from git master, I recommend following this: https://docs.openstack.org/diskimage-builder/latest/developer/index.html#quickstart | 17:49 |
| mazzy | ecsantos: yes I'm aware of that. i'm running everything on virtualenv | 18:07 |
| mazzy | the line | 18:08 |
| mazzy | > 2021-08-31 17:23:30.344 | /usr/bin/env: 'python3': No such file or directory | 18:08 |
| mazzy | is referred to the OS where i run the build or the target image? | 18:08 |
| ecsantos | to the host OS, where you run the build | 18:11 |
| mazzy | ah ok. so in my case despite python3 is in place i have issue with the path. let's find out to solve them | 18:11 |
| ecsantos | how did you install dib? | 18:24 |
| mazzy | nothing. even running in a proper virtualenv i still get that | 18:24 |
| mazzy | same as the dev guide | 18:24 |
| mazzy | (env) ubuntu@ubuntu:~/dib/diskimage-builder$ pip list | 18:24 |
| mazzy | Package Version Location | 18:24 |
| mazzy | ----------------- ----------- ---------------------------------- | 18:24 |
| mazzy | diskimage-builder 3.13.1.dev5 /home/ubuntu/dib/diskimage-builder | 18:24 |
| mazzy | (env) ubuntu@ubuntu:~/dib/diskimage-builder$ which python3 | 18:25 |
| mazzy | dir /home/ubuntu/dib/env/bin/python3 | 18:25 |
| mazzy | The issue is in running ~/dib/diskimage-builder/diskimage_builder/elements/package-installs/bin/package-installs-v2 which is a python and for some reasons does not find python | 18:37 |
| mazzy | ecsantos: i think the reason is because the python exec is not inside the target image (and not the OS where i run the build) see here https://bugs.launchpad.net/diskimage-builder/+bug/1577105 | 18:41 |
| ecsantos | you're right. that's weird, I couldn't reproduce the bug, tried building xenial and it worked fine | 18:55 |
| ianw | mazzy: running with "-x" will help put more info in the logs | 19:07 |
| mazzy | ok maybe i understand but i need someone to confirm my thoughts | 19:11 |
| mazzy | ianw: so flatcar OS does not come out with python. | 19:11 |
| mazzy | package-installsv2 script needs python. so when package-installs2 runs it fails | 19:11 |
| mazzy | the problem is that even if I try to install python on the flatcar image, package-installs script run before my pre-install.d or install.d | 19:12 |
| ianw | yes, that would be right. python in the chroot is probably assumed | 19:13 |
| ianw | that should be done in a root.d phase | 19:14 |
| mazzy | yeah i think that is the only way to solve unless some dib experts can say otherwise | 19:14 |
| mazzy | but in the root.d i'm not inside the chroot so i need to get to it manually | 19:14 |
| ianw | yes most distros do some manual work like that in early phases | 19:15 |
| ianw | it seems you're mostly doing like what the containerfile element does | 19:15 |
| ianw | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/containerfile/root.d/08-containerfile | 19:15 |
| mazzy | ianw: i will check that. thanks for the direction | 19:17 |
| ianw | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/yum-minimal/root.d/08-yum-chroot#L173 | 19:18 |
| ianw | is an example of how yum-minimal installs things in the chroot in root.d | 19:19 |
| ianw | i don't think we have exact requirements of tools in the chroot listed | 19:19 |
| ianw | i imagine it's python3, python-yaml, bash (not ash) | 19:20 |
| mazzy | ianw: that is really a nice example | 19:27 |
| ianw | package-installs is just an element, so it must be coming from element-deps | 19:52 |
| mazzy | but i have nothing there | 19:52 |
| mazzy | it comes from dib-init-system | 19:53 |
| ianw | that doens't have an elements-deps either, it must be coming from somehwere else? | 19:53 |
| ianw | i feel like with "-x" it dumps info on the dependency chain bringing in elements. maybe it's "-x -x" | 19:54 |
| ianw | there's a line anyway "Expanded element dependencies to: " | 19:56 |
| mazzy | i will try and post the result | 19:56 |
| ianw | it comes in via base | 19:57 |
| ianw | diskimage_builder/elements/base/pkg-map | 19:57 |
| ianw | that is a list of default things we're installing in images | 19:57 |
| mazzy | yeah exactly it's base | 19:58 |
| mazzy | but flatar is a different os that makes different assumptions | 19:58 |
| ianw | hrm, still, none of those bring in package-installs | 19:59 |
| mazzy | ok give some mins so we'll ahve the logs | 20:00 |
| mazzy | ianw: https://pastebin.com/kEZQsfGj | 20:06 |
| ianw | oh, cache-url will bring it in | 20:07 |
| ianw | which i would say is more-or-less right; it needs curl in the image to download things and that's what the package-install says | 20:09 |
| mazzy | oh right | 20:17 |
| mazzy | i have it | 20:17 |
| ianw | are you trying to make a bootable image? | 20:19 |
| mazzy | yes right | 20:19 |
| ianw | i mean, you're going to have to install grub, etc. then? how can that happen without a package manager? | 20:20 |
| mazzy | the idea is to add the support for a new repo to have it in kolla | 20:20 |
| mazzy | the thing is quite convoluted | 20:21 |
| mazzy | let me point you to the documentation | 20:21 |
| mazzy | https://kinvolk.io/docs/flatcar-container-linux/latest/reference/developer-guides/sdk-modifying-flatcar/ | 20:21 |
| ianw | hrm | 20:27 |
| ianw | a *very* rough analogy seems like "cork" is like dib ... a thing that gives you a chroot and a bunch of scripts to modify it | 20:28 |
| ianw | prometheanfire might have some more idea of how things plug together, since it's based on gentoo | 20:30 |
| mazzy | yeah exactly. dib is. what cork does | 20:31 |
| ianw | if you have a root.d element that makes a flatcar .tar.gz with a package manager, python, bash that would seem to look like a pretty standard gentoo environment? | 20:31 |
| ianw | but it does seem the point of this is to *not* make generic images | 20:32 |
| ianw | it is probably worth stepping back to make sure dib is the right tool here... | 20:33 |
| mazzy | yeah i thought about this several times | 20:34 |
| mazzy | but how to make flatacar available to nodepools? | 20:34 |
| mazzy | the point is that the guy from kolla redirected me to dib beause in order to add support to a distro in kolla, it must be tested the distro ofc | 20:35 |
| mazzy | and they want to standardize that with dib. because tbh kinvolk people already distribute openstack images | 20:35 |
| mazzy | see here https://kinvolk.io/docs/flatcar-container-linux/latest/installing/community-platforms/openstack/ | 20:36 |
| ianw | nothing is impossible, but i'm not sure that flatcar fits into what we think of as a host that fits into opendev nodepool in any easy way | 20:43 |
| ianw | a more practical solution might be to figure out something for testing purposes is where flatcar runs virtualised | 20:44 |
| mazzy | which is your idea? | 20:48 |
| ianw | i would say this is probably something for openstack-discuss list, because it is the type of thing more eyes will benefit from, and also probably time to think | 20:51 |
| ianw | but in theory, you could have flatcar running in a nested VM on one of our hosts and deploy kolla to it? | 20:52 |
| mazzy | yes that could be doable | 20:54 |
| mazzy | actually i'm already running kolla in flatcar on my premises | 20:54 |
| ianw | as in kolla makes a bunch of flatcar containers that run openstack services? | 21:08 |
| mazzy | the containers would not be flatcar | 21:10 |
| mazzy | the containers would be still ubuntu/centos/etc... | 21:11 |
| mazzy | only the OS where the containers run would be flatcar | 21:11 |
| ianw | ohh, ok | 21:12 |
| opendevreview | Shnaidman Sagi (Sergey) proposed openstack/diskimage-builder master: Correct path for CentOS 9 stream base image https://review.opendev.org/c/openstack/diskimage-builder/+/806819 | 23:24 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!