| *** pmannidi has joined #openstack-dib | 00:03 | |
| *** mjturek has quit IRC | 00:36 | |
| openstackgerrit | Merged openstack/diskimage-builder master: delete unused module https://review.openstack.org/558167 | 01:04 |
|---|---|---|
| ianw | prometheanfire: why's the gentoo job timing out? http://logs.openstack.org/58/558358/1/check/nodepool-functional-py35-gentoo-src/c7f5612/ | 01:21 |
| prometheanfire | probably updating something | 01:24 |
| ianw | echo 'invalid checksum on downloaded tarball: /opt/dib/cache/gentoo-stage4-systemd.tar.bz2' | 01:27 |
| ianw | the problem is, even with non-voting, if it takes 1 1/2 hours to timeout, that holds everything up | 01:28 |
| ianw | i think we need to put a retry limit on things, in general | 01:28 |
| prometheanfire | gah... | 01:28 |
| prometheanfire | let me verify that again | 01:29 |
| prometheanfire | blueness: where are you seeing the invalid checksum? looking at the job output | 01:35 |
| prometheanfire | not blueness... | 01:36 |
| prometheanfire | ianw: where are you seeing the invalid checksum? looking at the job output now | 01:36 |
| ianw | http://logs.openstack.org/58/558358/1/check/nodepool-functional-py35-gentoo-src/c7f5612/controller/logs/builds/gentoo-17-0-systemd-0000000174_log.txt.gz#_2018-04-03_08_53_29_320 | 01:38 |
| ianw | was one | 01:38 |
| ianw | there's a loop of failed builds in http://logs.openstack.org/58/558358/1/check/nodepool-functional-py35-gentoo-src/c7f5612/controller/logs/builds/ | 01:38 |
| prometheanfire | thanks | 01:38 |
| prometheanfire | ok, need to bug our infra | 01:41 |
| *** phuongnh has joined #openstack-dib | 01:52 | |
| *** Sukhdev has joined #openstack-dib | 03:50 | |
| *** Sukhdev has quit IRC | 03:56 | |
| *** tonyb has quit IRC | 04:45 | |
| *** chhagarw has joined #openstack-dib | 04:47 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Formalise saving of /etc/resolv.conf https://review.openstack.org/558362 | 05:19 |
| *** pmannidi has quit IRC | 05:49 | |
| *** greghaynes has quit IRC | 05:52 | |
| *** hashar has joined #openstack-dib | 06:12 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Restore tracing on exit points of block_device_create_config_file https://review.openstack.org/558358 | 06:13 |
| *** prometheanfire has quit IRC | 07:12 | |
| *** andreas_s has joined #openstack-dib | 07:12 | |
| *** prometheanfire has joined #openstack-dib | 07:13 | |
| *** prometheanfire has quit IRC | 07:52 | |
| *** prometheanfire has joined #openstack-dib | 07:54 | |
| *** pmannidi has joined #openstack-dib | 09:25 | |
| olivierbourdon38 | Hello everyone. ianw and others, I have a situation where ALL my image creation attempts fails due to some umount errors (32) but neither lsof nor fuser shows anything. even dmsetup remove loopX fails for the same reason on the same device. Any advice on this issue ? | 09:43 |
| olivierbourdon38 | note that I am using xfs on both the building machine and the build image | 09:48 |
| olivierbourdon38 | build system is ubuntu xenial amd64 | 09:49 |
| *** phuongnh has quit IRC | 10:08 | |
| *** olivierb- has joined #openstack-dib | 10:14 | |
| *** olivierbourdon38 has quit IRC | 10:15 | |
| ianw | oliverbourdon38: it is likely the actual error is before that ... the exit path often has some issues that throw up umount errors, but the real problem happens before that and causes things to abort | 10:19 |
| ianw | basically scroll up through the logs about 50 lines and check | 10:19 |
| ianw | feel free to file a bug with the full log, i can take a look in my morning | 10:19 |
| olivierb- | ianw thanks for the info and I understand however the same mechanism was working quite well end of last week so I am currently trying to dig a little bit more to see what has changed from my perspective | 10:32 |
| *** pmannidi has quit IRC | 10:40 | |
| olivierb- | ok after looking into my problem for quite a bit of (unproductive) time I was wonderning why the following is now failing: | 12:42 |
| olivierb- | disk-image-create -o /httpboot/deployment_image.qcow2 -t qcow2 --image-size 60 -p python,python3-yaml,dracut,dnsutils,net-tools,netcat,dnsutils,iputils-ping,traceroute,curl ubuntu-minimal bootloader enable-serial-console dhcp-all-interfaces dracut-regenerate devuser cloud-init-nocloud openssh-server | 12:42 |
| olivierb- | it was working perfectly end of last week | 12:43 |
| olivierb- | the error seems to be: | 12:43 |
| olivierb- | 2018-04-04 12:42:32.924 | /tmp/in_target.d/finalise.d/80-update-initramfs: line 30: update-initramfs: command not found | 12:43 |
| olivierb- | putting an absolute path /usr/sbin/update-initramfs in /usr/local/lib/python2.7/dist-packages/diskimage_builder/elements/debootstrap/finalise.d/80-update-initramfs seems to work a little bit better but still fails | 12:44 |
| olivierb- | 2018-04-04 12:46:49.812 | subprocess.CalledProcessError: Command 'sudo vgcreate vg /dev/mapper/loop1p1 --force' returned non-zero exit status 3 | 12:48 |
| *** vmlinuz has joined #openstack-dib | 12:49 | |
| *** vmlinuz has joined #openstack-dib | 12:49 | |
| olivierb- | any clue ? | 12:55 |
| olivierb- | andreas_s seems to be related to http://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=09e5b2d35786373e77e75c9b6793d1685918a88b which does not seem to work with ubuntu-minimal | 13:15 |
| olivierb- | thanks yolanda for digging this out for me | 13:15 |
| yolanda | no problem :) | 13:15 |
| olivierb- | removing 80-update-initramfs file just make everything work back as expected | 13:17 |
| *** yolanda_ has joined #openstack-dib | 13:22 | |
| *** yolanda has quit IRC | 13:24 | |
| *** hashar is now known as hasharAway | 13:25 | |
| andreas_s | olivierb-: interesting, for me it was working fine with ubuntu-minimal distro... | 13:32 |
| *** mjturek has joined #openstack-dib | 13:33 | |
| olivierb- | andreas_s may be some of my other options for building the images make this different to your case | 13:39 |
| andreas_s | let me check your command on my system... | 13:39 |
| andreas_s | I wonder if the dracut-regenerate element works with ubuntu-minimal... cause the comment in the commit says, it does not (but that was 6 month ago) | 13:41 |
| olivierb- | it has been working for me for several weeks now | 13:41 |
| olivierb- | and my images are fully functional | 13:41 |
| olivierb- | seems like at least | 13:42 |
| olivierb- | andreas_s just let me know the result of the command on your system | 13:47 |
| andreas_s | olivierb-: also running into the "command not found" issue...hhmm | 13:51 |
| olivierb- | as mentioned to yolanda_ privately trying to add package initramfs-tools does not help because it conflicts | 13:52 |
| olivierb- | 2018-04-04 13:11:30.982 | dracut : Conflicts: initramfs-tools but 0.122ubuntu8.11 is to be installed | 13:53 |
| olivierb- | 2018-04-04 13:11:30.982 | initramfs-tools : Conflicts: linux-initramfs-tool | 13:53 |
| olivierb- | 2018-04-04 13:11:30.988 | E: Unable to correct problems, you have held broken packages. | 13:53 |
| andreas_s | this simple command still works disk-image-create -t qcow2 ubuntu-minimal zipl vm (where zipl is the s390x replacment for bootloader) | 13:55 |
| andreas_s | it's probably one of the additional elements or packages that causes this issue... | 13:56 |
| andreas_s | now trying without dracut... | 13:56 |
| andreas_s | olivierb-: removed dracut package and dracut-regenerate element and it succeeded | 13:58 |
| andreas_s | removed it from your long command | 13:58 |
| olivierb- | however from what yolanda_ told me, I need dracut+dracut-regenerate to have my LVM partitioning to work properly | 13:59 |
| olivierb- | which does | 13:59 |
| andreas_s | can you share your blockdevice json config? | 14:00 |
| yolanda_ | he needs to add lvm module to make it work | 14:00 |
| olivierb- | it's yaml but will pastebin it | 14:00 |
| andreas_s | exaclty, add lvm as package and it should work | 14:00 |
| yolanda_ | andreas_s, i remember you told me dracut was not intended for ubuntu? | 14:00 |
| andreas_s | -p lvm2 | 14:01 |
| olivierb- | https://pastebin.com/s7CkJQdT | 14:01 |
| olivierb- | + some other env variables | 14:02 |
| andreas_s | yolanda_, olivierb-: https://github.com/openstack/diskimage-builder/blob/43bc352c5977272ceadc1ca44d96c086cad59f5c/diskimage_builder/elements/debootstrap/finalise.d/80-update-initramfs#L32,L38 | 14:03 |
| andreas_s | andreas florath figrued that out... | 14:03 |
| olivierb- | https://pastebin.com/1eBq73jF | 14:04 |
| andreas_s | olivierb-: so my experience with building lvm images with ubunt16 was, that it does not work when the boot volume is located on lvm | 14:05 |
| andreas_s | the image build will succeed, but on first boot it will not be able to lookup the boot disk by label | 14:06 |
| andreas_s | this is probably the root cause for this: https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/1573982 | 14:07 |
| openstack | Launchpad bug 1573982 in lvm2 (Ubuntu) "LVM boot problem - volumes not activated after upgrade to Xenial" [Undecided,Confirmed] | 14:07 |
| andreas_s | but it might work if you put the boot volume on a non lvm partition | 14:07 |
| andreas_s | (not sure if I'm mixing up boot and root now - it has been a while since I did some experiments with it) | 14:08 |
| olivierb- | from my experience the image produced with the lvm I pasted seems to boot ok | 14:08 |
| andreas_s | ok, then maybe I was taking the wrong approach... | 14:09 |
| olivierb- | I am currently bootstrapping a new env with the latest generated image and will confirm this | 14:09 |
| yolanda_ | if you want to boot from lvm, you need that module to be loaded on kernel | 14:10 |
| olivierb- | yes it is | 14:10 |
| andreas_s | olivierb-: if you know a way that works, then maybe let's roll back the patch and add some piece of documentation that describes how to create such a lvm image | 14:10 |
| andreas_s | I tried it with adding lvm2 as package and rebuilding the initrd using update-initramfs - inspired by https://bugs.launchpad.net/diskimage-builder/+bug/1715686 | 14:11 |
| openstack | Launchpad bug 1715686 in diskimage-builder "using LVM does not create bootable systems" [Undecided,New] | 14:11 |
| andreas_s | or even better, let's add a gate job for it... | 14:15 |
| olivierb- | andreas_s just to confirm that my deployed image generated by dib using command sent earlier in this channel works perfectly booting from LVM however /boot is not a separate partition but resides within / which is a LVM http://www.olivierbourdon/BIFROST/lvm.png snapshot of vncviewer console | 15:33 |
| andreas_s | olivierb-: so you're saying it's booting fine when having the dracut stuff in place and the update-initramfs file removed? | 15:35 |
| olivierb- | yes | 15:35 |
| andreas_s | cool. So then maybe the solution is to document your approach and undo the update-initramfs patch... | 15:36 |
| olivierb- | were do you want me to document this please ? | 15:36 |
| olivierb- | where sorry ;-) | 15:36 |
| andreas_s | maybe somewhere here: https://docs.openstack.org/diskimage-builder/latest/user_guide/building_an_image.html | 15:37 |
| andreas_s | under the lvm module | 15:38 |
| andreas_s | I'm not a disk-image-builder guy, so there might be better places... | 15:38 |
| olivierb- | ok will do but most probably tomorrow as I need to wrap up several other stuff todat | 15:38 |
| andreas_s | but this is the piece of documentation I was looking at when playing around with it... | 15:38 |
| andreas_s | no problem | 15:38 |
| andreas_s | maybe it is already documented? At least there's a sentence about dracut regenerate... | 15:40 |
| *** andreas_s has quit IRC | 15:41 | |
| olivierb- | https://github.com/obourdon/diskimage-builder/commit/1dc945fd84a600d342e76ed61b4ba0d3eb7fc686 | 16:50 |
| *** greghaynes has joined #openstack-dib | 16:55 | |
| *** Sukhdev has joined #openstack-dib | 18:36 | |
| *** Sukhdev has quit IRC | 18:41 | |
| *** hasharAway is now known as hashar | 19:07 | |
| *** Sukhdev has joined #openstack-dib | 20:29 | |
| *** hashar has quit IRC | 20:48 | |
| ianw | olivierb-: glad you were able to figure it out :) are you filing something on the dracut issue? | 20:53 |
| olivierb- | ianw right now I am discussing with Andre Florath to see implications of reverting his change, however doc is ready and will most probably be submitted tomorrow for review as a separate patchset | 20:55 |
| ianw | ok, great, thanks | 20:56 |
| *** Sukhdev has quit IRC | 21:04 | |
| *** vmlinuz has quit IRC | 21:44 | |
| *** chhagarw has quit IRC | 22:00 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
