| *** rfolco has joined #openstack-dib | 00:49 | |
| *** rfolco has quit IRC | 00:49 | |
| *** rfolco has joined #openstack-dib | 01:03 | |
| *** rfolco has quit IRC | 01:03 | |
| *** isaacb has joined #openstack-dib | 03:22 | |
| *** isaacb has quit IRC | 03:24 | |
| *** brault has joined #openstack-dib | 05:18 | |
| *** brault has quit IRC | 05:19 | |
| *** andreas-f has joined #openstack-dib | 05:22 | |
| *** makowals has joined #openstack-dib | 05:59 | |
| *** makowals has quit IRC | 06:03 | |
| *** hashar has joined #openstack-dib | 06:39 | |
| *** makowals has joined #openstack-dib | 06:40 | |
| *** andreas-f has quit IRC | 07:07 | |
| *** aparnav has joined #openstack-dib | 08:44 | |
| *** yolanda_ has joined #openstack-dib | 08:59 | |
| lennyb | Hi Team, what could be a reason for not passing 'compat=10.0' in nodepool-builder ? I am bringing up new CI and I see that qemu-img fails due to a missing compat flag | 10:26 |
|---|---|---|
| lennyb | #link http://paste.openstack.org/show/612668/ | 10:26 |
| ianw | lennyb: did you set it in nodepool.yaml? | 11:02 |
| ianw | see https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml | 11:03 |
| *** yolanda__ has joined #openstack-dib | 11:30 | |
| *** yolanda_ has quit IRC | 11:30 | |
| *** yolanda__ is now known as yolanda | 12:05 | |
| lennyb | thanks, ianw. I've missed this var | 12:12 |
| *** appu103 has joined #openstack-dib | 12:54 | |
| *** aparnav has quit IRC | 12:57 | |
| *** appu103 has quit IRC | 12:58 | |
| *** hashar has quit IRC | 13:47 | |
| *** hashar has joined #openstack-dib | 13:54 | |
| *** hashar has quit IRC | 14:04 | |
| *** hashar has joined #openstack-dib | 14:12 | |
| *** makowals has quit IRC | 14:47 | |
| *** makowals has joined #openstack-dib | 14:53 | |
| *** hashar has quit IRC | 15:31 | |
| *** andreas-f has joined #openstack-dib | 16:12 | |
| *** greghaynes is now known as greghayn1 | 17:18 | |
| *** greghaynes has joined #openstack-dib | 17:18 | |
| *** greghaynes has quit IRC | 17:22 | |
| *** greghaynes has joined #openstack-dib | 17:23 | |
| *** greghayn1 has quit IRC | 17:25 | |
| johnsom | Hmm, seeing a few: | 17:46 |
| johnsom | subprocess.CalledProcessError: Command 'sudo mkfs -t ext4 -i 4096 -J size=64 -L cloudimg-rootfs -U f9dc428c-394e-427a-a6d6-18c49a822fa0 -q /dev/mapper/loop0p1' returned non-zero exit status 1 | 17:46 |
| johnsom | http://logs.openstack.org/74/474674/1/check/gate-octavia-v1-dsvm-py3x-scenario-multinode/6c43279/logs/devstacklog.txt.gz#_2017-06-15_16_59_05_414 | 17:46 |
| johnsom | Not sure what is causing DIB to fail there. Two out of six gate test failed with that | 17:47 |
| johnsom | All run DIB | 17:47 |
| greghaynes | I wonder if out of disk space? | 18:31 |
| *** Sukhdev has joined #openstack-dib | 19:26 | |
| *** yolanda has quit IRC | 21:45 | |
| ianw | johnsom: hmm, 2.6.0 did move to using kpartx, it's using /dev/mapper ... but for only 2 out of several to fail? | 21:50 |
| johnsom | Yeah, it is really strange, same patch, six jobs that build images, just two failed with this error. | 21:50 |
| johnsom | On a global requirements update no less | 21:51 |
| johnsom | Though we have seen this recently on a few other random patches | 21:52 |
| ianw | hmm, are we tracing the output? i don't think so | 22:06 |
| ianw | oh yeah, we are ... that's right | 22:07 |
| ianw | 2017-06-15 16:59:05.349 | 2017-06-15 16:59:05.349 INFO diskimage_builder.block_device.utils [-] Calling [sudo kpartx -avs /dev/loop0] | 22:08 |
| ianw | 2017-06-15 16:59:05.358 | 2017-06-15 16:59:05.357 INFO diskimage_builder.block_device.utils [-] Calling [sudo mkfs -t ext4 -i 4096 -J size=64 -L cloudimg-rootfs -U f9dc428c-394e-427a-a6d6-18c49a822fa0 -q /dev/mapper/loop0p1] | 22:08 |
| ianw | that's not very long between the kpartx call and the mkfs call | 22:08 |
| ianw | maybe we should put a sleep in | 22:08 |
| ianw | https://bugs.launchpad.net/nova/+bug/1024586 -> reports that kpartx is not synchronous | 22:10 |
| openstack | Launchpad bug 1024586 in OpenStack Compute (nova) "avoid the use of kpartx in file injection" [Medium,Invalid] - Assigned to Pádraig Brady (p-draigbrady) | 22:10 |
| ianw | first hit for "kpartx" and synchronous :) | 22:10 |
| ianw | second comment > I just noticed a mail to say in kernel Loop device partition scanning is unreliable | 22:11 |
| ianw | haha, everything is terrible :) | 22:11 |
| johnsom | Ha, yeah, that actually brings up some memories for a bug we had early in DIB | 22:15 |
| ianw | i wonder if "udevadm settle" is actually useful here, or it's just a placebo and a sleep(2) would be just as effective? | 22:17 |
| johnsom | Maybe a partprobe or kpartx -u? | 22:20 |
| johnsom | Ugh, it's been a while since I dealt with this sort of thing, it is fuzzy | 22:21 |
| ianw | we deliberately removed the partprobe so we didn't have to cleanup two paths | 22:22 |
| ianw | but, we did used to have a udevadm settle after it, looking at old logs | 22:22 |
| ianw | http://logs.openstack.org/91/470791/3/check/gate-dib-dsvm-functests-python2-ubuntu-trusty/9a08409/console.html#_2017-06-05_07_26_32_789660 | 22:22 |
| ianw | and we are running in "sync" mode (-s) which should not return until the partitions are made | 22:23 |
| johnsom | I'm not super familiar with udevadm settle, but it sounds reasonable from the man page. | 22:25 |
| johnsom | Sleeps bother me as someone always invents a slower/busier host | 22:29 |
| *** rm_work has joined #openstack-dib | 22:29 | |
| johnsom | ianw Should I open an actual bug and throw up a patch? | 22:30 |
| ianw | the other thing is, if it wasn't there, why didn't mkfs output anything | 22:30 |
| ianw | The file /dev/mapper/loop0p3 does not exist and no size was specified. | 22:30 |
| ianw | ^ mkfs output on a non-existent device for me using that command line | 22:30 |
| johnsom | Yeah, I was wondering what happened to the output too. I just thought DIB was eating it or since we don't have -x tracing on it wasn't printed. | 22:32 |
| ianw | ohhhh ... no "-x", yeah it's debug level | 22:32 |
| ianw | is there a reason not run with "-x"? i've wanted to make that default level | 22:32 |
| johnsom | Yeah, too chatty for us in day-to-day dev | 22:32 |
| johnsom | Yeah, we tried it for a bit, but it is super chatty | 22:33 |
| ianw | it has got a *lot* better ... | 22:33 |
| rm_work | i LIKE -x | 22:33 |
| rm_work | for the record | 22:33 |
| rm_work | i keep trying to add it | 22:33 |
| ianw | i've done stuff like turn off things like logging loop internals | 22:33 |
| rm_work | but can't find the right spot (and my care level was admittedly low) | 22:34 |
| johnsom | Fine, fine, we can make that 300k log file 600k... grin. I won't block it. | 22:34 |
| ianw | we could turn the tracing of commands run under exec_sudo up to info level | 22:34 |
| ianw | it would be really great to see why that mkfs fails ... if it is a missing device, well clearly there's a bug in the "-s" mode of kpartx | 22:35 |
| ianw | i'm happy to work around it, but would like to understand why we are | 22:35 |
| rm_work | I will +2 adding "-x" | 22:35 |
| rm_work | :) | 22:35 |
| ianw | i do not believe we've seen this in the dib gate. but maybe our hosts aren't as busy | 22:35 |
| ianw | if there are parts of "-x" spewing out useless boilerplate, i'm happy to turn that down | 22:36 |
| johnsom | We probably turn a lot more patchsets too | 22:36 |
| ianw | true | 22:36 |
| ianw | so my first preference would be to catch one of these failures so we know what we're dealing with | 22:37 |
| ianw | either up the log level in the exec_sudo, or add "-x" to the call | 22:37 |
| johnsom | rm_work It's OCTAVIA_DIB_TRACING in gate_hook | 22:39 |
| johnsom | https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L62 | 22:39 |
| rm_work | k | 22:40 |
| rm_work | OCTAVIA_DIB_TRACING=True ? | 22:40 |
| rm_work | OCTAVIA_DIB_TRACING=1 | 22:40 |
| rm_work | OCTAVIA_DIB_TRACING=-x | 22:40 |
| *** openstackgerrit has joined #openstack-dib | 22:40 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder master: Updated from global requirements https://review.openstack.org/474798 | 22:40 |
| johnsom | 1 | 22:40 |
| rm_work | k i'll set that in ... one of these | 22:41 |
| rm_work | err whatever i'll make it a thing | 22:41 |
| ianw | ok, hopefully (?) it's a missing partition ... | 23:40 |
| ianw | now we're driving the build from python, i wonder if it would be more practical to capture logs to a file | 23:41 |
| ianw | if we dup()'d stdout/stderr before exec() maybe... | 23:44 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!