| *** rlandy has quit IRC | 00:02 | |
| *** panda has quit IRC | 00:02 | |
| *** panda has joined #openstack-infra | 00:05 | |
| *** jamesmcarthur has quit IRC | 00:15 | |
| *** tkajinam has joined #openstack-infra | 00:16 | |
| *** weifan has joined #openstack-infra | 00:26 | |
| *** jamesmcarthur has joined #openstack-infra | 00:29 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Clean up review comments https://review.opendev.org/692003 | 00:29 |
|---|---|---|
| *** weifan has quit IRC | 00:30 | |
| *** goldyfruit___ has quit IRC | 00:42 | |
| *** michael-beaver has quit IRC | 00:43 | |
| *** jamesmcarthur has quit IRC | 00:44 | |
| *** ricolin has quit IRC | 00:45 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Revert "Drop vhdutil dependency, use qemu-img" https://review.opendev.org/691998 | 00:52 |
| *** pots has quit IRC | 00:54 | |
| *** pots has joined #openstack-infra | 00:55 | |
| ianw | i'll release dib with ^ | 00:55 |
| ianw | 2.29.1 out ... will check in on deployment | 00:57 |
| *** ricolin has joined #openstack-infra | 00:57 | |
| *** jamesmcarthur has joined #openstack-infra | 01:01 | |
| mordred | :( | 01:02 |
| ianw | i'm trying having qemu-img set the tag to "tap\0" | 01:05 |
| *** weifan has joined #openstack-infra | 01:06 | |
| ianw | not that carrying a patched qemu-img is really any easier ... well maybe a little | 01:06 |
| mordred | ianw: if only rackspace would accept uploads in a realistic format | 01:07 |
| *** jamesmcarthur has quit IRC | 01:13 | |
| *** weifan has quit IRC | 01:14 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Plumb through secure.config contents https://review.opendev.org/691800 | 01:53 |
| *** yamamoto has joined #openstack-infra | 01:55 | |
| *** yamamoto has quit IRC | 01:56 | |
| *** yamamoto has joined #openstack-infra | 01:56 | |
| *** slaweq has joined #openstack-infra | 02:13 | |
| *** slaweq has quit IRC | 02:17 | |
| *** slaweq has joined #openstack-infra | 02:22 | |
| *** diablo_rojo has quit IRC | 02:24 | |
| *** slaweq has quit IRC | 02:27 | |
| *** uberjay has quit IRC | 02:27 | |
| *** uberjay has joined #openstack-infra | 02:29 | |
| ianw | well with a patched qemu-img i can upload an image that becomes active ... | 02:32 |
| mordred | ianw: that's cool. it's just the tag change? | 02:32 |
| ianw | you also have to fiddle the version to fool https://fossies.org/dox/xen-4.12.1/libvhd_8c_source.html#l01193 | 02:33 |
| mordred | ianw: but it *didn't* work with editing the file, right? was there maybe just an additional byte that needed changing or something perhaps? | 02:33 |
| mordred | ah | 02:33 |
| mordred | so it's conceivable that we could just edit the header afterwards perhaps? | 02:33 |
| ianw | not really as it fiddles the checksum | 02:36 |
| ianw | but, possibly maintaining a patch for qemu which is alive, rather than removed bits of xen, might be easier at least | 02:36 |
| *** slaweq has joined #openstack-infra | 02:37 | |
| mordred | yeah | 02:45 |
| mordred | we *might* even be able to convince someone in ubuntu to accept the patch | 02:46 |
| mordred | or even upstream | 02:46 |
| *** slaweq has quit IRC | 02:52 | |
| ianw | i think i can make a compelling case for including a flag upstream ... if this thing boots ... | 02:53 |
| *** slaweq has joined #openstack-infra | 02:57 | |
| ianw | 23.253.245.130 ... it has booted | 02:57 |
| *** yamamoto has quit IRC | 02:58 | |
| *** slaweq has quit IRC | 03:12 | |
| *** slaweq has joined #openstack-infra | 03:13 | |
| mordred | \o/ | 03:17 |
| mordred | that's SO EXCITING | 03:17 |
| mordred | and it resized appropriately? | 03:17 |
| mordred | looks like it: /dev/xvda1 15G 9.6G 4.4G 69% / | 03:18 |
| * mordred hands ianw a box of celebration chickens | 03:19 | |
| ianw | no, i don't think it has, i think that should be 40gb | 03:20 |
| ianw | the images are close ... http://paste.openstack.org/show/785645/ | 03:22 |
| ianw | but the one using vhd-util has this batmap ... my test one was giving an error about missing batmaps prior to the version change | 03:22 |
| *** slaweq has quit IRC | 03:22 | |
| *** slaweq_ has joined #openstack-infra | 03:23 | |
| ianw | it appears to be a xen feature of vhd-util, it's not described in the offiical spec | 03:25 |
| *** slaweq_ has quit IRC | 03:27 | |
| *** yamamoto has joined #openstack-infra | 03:29 | |
| *** slaweq_ has joined #openstack-infra | 03:32 | |
| mordred | oh. well - piddle. I guess those were premature celebration chickens | 03:33 |
| paladox | :D | 03:34 |
| mordred | ianw: so really supporting it in qemu-img would be like making another output format, like "xen" or something, that implements the xen extensions to vhd | 03:34 |
| ianw | mordred: upon further investigation, someone has sort of tried to implement the batmap with https://patchwork.ozlabs.org/project/qemu-devel/list/?submitter=64750 | 03:35 |
| ianw | ... patchwork ... my opinions best left unsaid | 03:36 |
| *** slaweq_ has quit IRC | 03:37 | |
| *** psachin has joined #openstack-infra | 03:37 | |
| mordred | yeah. that's *definitely* better than gerrit. I can *totally* follow what's going on there | 03:37 |
| ianw | i used vhd-utils to resize the test disk, made with (patched) qemu and it seemed to work -> Current disk size : 33554432 MB (35184372088832 Bytes) | 03:38 |
| *** rh-jelabarre has quit IRC | 03:38 | |
| ianw | i feel like that is what xen/nova/something is doing in the background | 03:38 |
| *** yamamoto has quit IRC | 03:39 | |
| mordred | ianw: when you say you used vhd-utils - you mean you did that on the image file? or on the booted vm? | 03:39 |
| ianw | mordred: on the image file ... vhd-util resize --debug -n ./test.vhd -s $((32 * 1024 * 1024 )) -j resize.log | 03:39 |
| mordred | ianw: if only we could upload in raw format and have the cloud transform that into vhd and do the resize for us ... | 03:40 |
| ianw | i wonder though, if i reupload ^^ if that will then boot with a 32gb root disk | 03:40 |
| mordred | (or really, if only we could upload in qcow2 everywhere, and have the clouds auto-reformat to their desired format as part of the uopload) | 03:41 |
| ianw | -rw-r--r-- 1 ianw ianw 16G Oct 30 03:33 test.vhd | 03:41 |
| mordred | ianw: I'd be willing to bet it would | 03:41 |
| mordred | oh. it still takes 16G? | 03:41 |
| mordred | on disk? | 03:41 |
| mordred | is that unpatched vhd-util you used for the resize? | 03:41 |
| ianw | yes still 16gb, but whatever vhd-util is on the nb's | 03:42 |
| *** slaweq_ has joined #openstack-infra | 03:42 | |
| ianw | i will re-upload the image and see if we get a bigger host | 03:43 |
| *** slaweq_ has quit IRC | 03:47 | |
| mordred | ianw: if it works - we should try again with the vhd-util that's not from the ppa | 03:49 |
| mordred | I'm pretty sure nb has the ppa version | 03:50 |
| *** slaweq_ has joined #openstack-infra | 03:52 | |
| *** slaweq_ has quit IRC | 03:57 | |
| mordred | ianw: actually - it seems like there is no version of vhd-util in ubuntu bionic without our ppa | 03:58 |
| mordred | just to make this as complex as possible | 03:59 |
| *** ykarel|pto has joined #openstack-infra | 04:01 | |
| *** ykarel|pto is now known as ykarel | 04:01 | |
| *** dave-mccowan has quit IRC | 04:02 | |
| *** jerryz has quit IRC | 04:07 | |
| *** todun has joined #openstack-infra | 04:08 | |
| ianw | "message": "Image cannot be imported. Error code: '413'", | 04:17 |
| *** udesale has joined #openstack-infra | 04:35 | |
| *** soniya29 has joined #openstack-infra | 04:59 | |
| *** ykarel is now known as ykarel|afk | 05:02 | |
| *** ykarel|afk has quit IRC | 05:02 | |
| *** janki has joined #openstack-infra | 05:17 | |
| *** sgw has quit IRC | 05:17 | |
| *** todun has quit IRC | 05:23 | |
| *** ykarel has joined #openstack-infra | 05:25 | |
| *** todun has joined #openstack-infra | 05:31 | |
| *** todun has quit IRC | 05:34 | |
| *** larainema has joined #openstack-infra | 05:39 | |
| *** yamamoto has joined #openstack-infra | 05:41 | |
| *** yamamoto has quit IRC | 05:55 | |
| *** kjackal has joined #openstack-infra | 05:56 | |
| *** kjackal has quit IRC | 06:08 | |
| AJaeger | config-core, please review https://review.opendev.org/691986 to remove obsolete jobs | 06:21 |
| *** lmiccini has joined #openstack-infra | 06:22 | |
| *** yamamoto has joined #openstack-infra | 06:23 | |
| *** igordc has quit IRC | 06:37 | |
| *** kopecmartin|off is now known as kopecmartin | 06:40 | |
| *** georgk has quit IRC | 06:40 | |
| *** fdegir has quit IRC | 06:40 | |
| *** georgk has joined #openstack-infra | 06:41 | |
| *** fdegir has joined #openstack-infra | 06:41 | |
| *** dpawlik has joined #openstack-infra | 06:58 | |
| *** ykarel is now known as ykarel|afk | 06:59 | |
| *** dpawlik has quit IRC | 07:03 | |
| *** pcaruana has joined #openstack-infra | 07:09 | |
| *** yamamoto has quit IRC | 07:09 | |
| *** dpawlik has joined #openstack-infra | 07:18 | |
| *** pgaxatte has joined #openstack-infra | 07:23 | |
| *** dciabrin has quit IRC | 07:24 | |
| *** rascasoft has quit IRC | 07:29 | |
| *** lpetrut has joined #openstack-infra | 07:35 | |
| *** kjackal has joined #openstack-infra | 07:36 | |
| *** yamamoto has joined #openstack-infra | 07:47 | |
| *** aedc has quit IRC | 07:49 | |
| *** pkopec has joined #openstack-infra | 07:55 | |
| *** florianf has joined #openstack-infra | 08:02 | |
| *** tkajinam has quit IRC | 08:03 | |
| *** ykarel|afk is now known as ykarel | 08:03 | |
| *** dchen has quit IRC | 08:07 | |
| *** yamamoto has quit IRC | 08:17 | |
| *** florianf has left #openstack-infra | 08:17 | |
| *** yamamoto has joined #openstack-infra | 08:18 | |
| *** chandankumar has quit IRC | 08:19 | |
| *** chandankumar has joined #openstack-infra | 08:20 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove openstack-tox-py27-with-oslo-master https://review.opendev.org/691986 | 08:22 |
| *** hashar has joined #openstack-infra | 08:27 | |
| *** slaweq_ has joined #openstack-infra | 08:30 | |
| *** gfidente has joined #openstack-infra | 08:34 | |
| *** yamamoto has quit IRC | 08:34 | |
| *** prometheanfire has quit IRC | 08:35 | |
| *** jpena|off is now known as jpena | 08:37 | |
| *** prometheanfire has joined #openstack-infra | 08:37 | |
| *** amoralej|off is now known as amoralej | 08:46 | |
| *** janki has quit IRC | 08:46 | |
| *** jchhatbar has joined #openstack-infra | 08:47 | |
| *** yamamoto has joined #openstack-infra | 08:47 | |
| *** lucasagomes has joined #openstack-infra | 08:51 | |
| *** lmiccini has quit IRC | 08:57 | |
| *** rpittau|afk is now known as rpittau | 08:58 | |
| *** ralonsoh has joined #openstack-infra | 08:58 | |
| *** dtantsur|afk is now known as dtantsur | 09:01 | |
| *** ykarel is now known as ykarel|lunch | 09:03 | |
| *** lmiccini has joined #openstack-infra | 09:06 | |
| *** vesper has joined #openstack-infra | 09:08 | |
| *** vesper11 has quit IRC | 09:09 | |
| *** iurygregory has joined #openstack-infra | 09:10 | |
| *** slaweq_ is now known as slaweq | 09:18 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:21 | |
| *** yamamoto has quit IRC | 09:25 | |
| *** trident has quit IRC | 09:26 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - add support for git.tag.creation event https://review.opendev.org/679938 | 09:30 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - Support for branch creation/deletion https://review.opendev.org/685116 | 09:30 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - add support for git.tag.creation event https://review.opendev.org/679938 | 09:32 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - Support for branch creation/deletion https://review.opendev.org/685116 | 09:32 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - add the enqueue_ref unit test https://review.opendev.org/687351 | 09:32 |
| *** dpawlik has quit IRC | 09:33 | |
| *** trident has joined #openstack-infra | 09:34 | |
| *** yamamoto has joined #openstack-infra | 09:34 | |
| *** elod has quit IRC | 09:37 | |
| *** tesseract has joined #openstack-infra | 09:38 | |
| *** dciabrin has joined #openstack-infra | 09:42 | |
| *** aedc has joined #openstack-infra | 09:46 | |
| *** dpawlik has joined #openstack-infra | 09:48 | |
| *** cgoncalves has quit IRC | 09:49 | |
| *** yamamoto has quit IRC | 09:50 | |
| *** elod has joined #openstack-infra | 09:57 | |
| *** cgoncalves has joined #openstack-infra | 09:58 | |
| *** ykarel|lunch is now known as ykarel | 10:13 | |
| AJaeger | corvus, regarding the /opt/git changes on our images: We have lots of "/usr/zuul-env/bin/zuul-cloner --cache-dir /opt/git" - did we just break that? | 10:17 |
| *** Tengu has quit IRC | 10:18 | |
| *** kjackal has quit IRC | 10:18 | |
| *** Tengu has joined #openstack-infra | 10:19 | |
| *** kjackal has joined #openstack-infra | 10:22 | |
| *** yamamoto has joined #openstack-infra | 10:25 | |
| *** rcernin has quit IRC | 10:27 | |
| *** pcaruana has quit IRC | 10:35 | |
| *** yamamoto has quit IRC | 10:37 | |
| openstackgerrit | Fabien Boucher proposed zuul/nodepool master: Remove uneeded shebang and exec bit on some files https://review.opendev.org/692100 | 10:39 |
| *** jtomasek has quit IRC | 10:44 | |
| *** openstackstatus has quit IRC | 10:44 | |
| *** mgoddard has quit IRC | 10:46 | |
| *** mgoddard has joined #openstack-infra | 10:47 | |
| *** jtomasek has joined #openstack-infra | 10:47 | |
| *** rfolco|off has joined #openstack-infra | 10:54 | |
| *** sulo has joined #openstack-infra | 10:54 | |
| *** panda is now known as panda|pto | 11:00 | |
| *** arxcruz is now known as arxcruz|lunch | 11:10 | |
| *** jchhatbar has quit IRC | 11:14 | |
| *** rh-jelabarre has joined #openstack-infra | 11:16 | |
| *** aedc has quit IRC | 11:17 | |
| *** sshnaidm has quit IRC | 11:23 | |
| *** goldyfruit___ has joined #openstack-infra | 11:29 | |
| *** rfolco|off has quit IRC | 11:35 | |
| openstackgerrit | Harald Jensås proposed openstack/diskimage-builder master: WIP: Add IPv6 support in dhcp-all-interfaces https://review.opendev.org/692110 | 11:40 |
| *** dpawlik has quit IRC | 11:41 | |
| openstackgerrit | Harald Jensås proposed openstack/diskimage-builder master: WIP: Add IPv6 support in dhcp-all-interfaces https://review.opendev.org/692110 | 11:43 |
| *** sshnaidm has joined #openstack-infra | 11:43 | |
| *** ramishra has quit IRC | 11:46 | |
| *** ramishra has joined #openstack-infra | 11:47 | |
| *** goldyfruit___ has quit IRC | 11:54 | |
| *** dciabrin has quit IRC | 11:55 | |
| *** jpena is now known as jpena|lunch | 11:59 | |
| *** pcaruana has joined #openstack-infra | 12:00 | |
| *** ykarel is now known as ykarel|afk | 12:02 | |
| dtantsur | hi folks! what's the unbound service installed in the CI? it conflicts with dnsmasq from bifrost, apparently.. | 12:02 |
| *** dpawlik has joined #openstack-infra | 12:08 | |
| *** larainema has quit IRC | 12:09 | |
| *** dpawlik has quit IRC | 12:12 | |
| *** rlandy has joined #openstack-infra | 12:13 | |
| openstackgerrit | Harald Jensås proposed openstack/diskimage-builder master: WIP: Add IPv6 support in dhcp-all-interfaces https://review.opendev.org/692110 | 12:17 |
| frickler | dtantsur: iiuc we install a local resolver mainly in order to be able to direct requests via ipv6 when that is natively available, to avoid drops in cloud providers' overloaded ipv4 nat devices | 12:18 |
| *** arxcruz|lunch is now known as arxcruz | 12:18 | |
| dtantsur | ack | 12:18 |
| dtantsur | I think I've figured why the CI is failing, it's not related | 12:19 |
| *** markvoelker has quit IRC | 12:25 | |
| *** hashar is now known as hasharAway | 12:25 | |
| *** markvoelker has joined #openstack-infra | 12:25 | |
| *** udesale has quit IRC | 12:26 | |
| *** hasharAway has quit IRC | 12:32 | |
| *** hashar has joined #openstack-infra | 12:34 | |
| *** hashar is now known as hasharAway | 12:35 | |
| *** aedc has joined #openstack-infra | 12:39 | |
| *** sulo has quit IRC | 12:41 | |
| *** Goneri has joined #openstack-infra | 12:49 | |
| fungi | frickler: the main reason we install unbound is to have a local dns cache on the node so that it doesn't query the same records from an external resolver over and over, as the latter increases chances of failure from intermittent network issues/packet loss | 12:56 |
| *** eharney has quit IRC | 12:57 | |
| fungi | if we didn't already want that, there would have been other ways to set preferred resolver addresses on different providers | 12:57 |
| *** rfolco has joined #openstack-infra | 13:01 | |
| *** jpena|lunch is now known as jpena | 13:02 | |
| *** psachin has quit IRC | 13:02 | |
| *** sgw has joined #openstack-infra | 13:03 | |
| *** goldyfruit___ has joined #openstack-infra | 13:04 | |
| *** ramishra has quit IRC | 13:07 | |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| *** soniya29 has quit IRC | 13:11 | |
| *** amoralej is now known as amoralej|lunch | 13:12 | |
| *** hasharAway has quit IRC | 13:13 | |
| *** xek has joined #openstack-infra | 13:13 | |
| *** yamamoto has quit IRC | 13:13 | |
| *** hashar has joined #openstack-infra | 13:14 | |
| *** nicholas has joined #openstack-infra | 13:14 | |
| *** hashar_ has joined #openstack-infra | 13:15 | |
| *** pkopec has quit IRC | 13:19 | |
| *** pkopec has joined #openstack-infra | 13:19 | |
| *** mriedem has joined #openstack-infra | 13:22 | |
| *** yamamoto has joined #openstack-infra | 13:36 | |
| *** hashar_ has quit IRC | 13:37 | |
| *** hashar has quit IRC | 13:38 | |
| *** hashar has joined #openstack-infra | 13:38 | |
| *** yamamoto has quit IRC | 13:41 | |
| *** ramishra has joined #openstack-infra | 13:41 | |
| *** yamamoto has joined #openstack-infra | 13:41 | |
| *** ykarel|afk is now known as ykarel | 13:45 | |
| *** kopecmartin is now known as kopecmartin|scho | 13:48 | |
| *** dciabrin has joined #openstack-infra | 13:50 | |
| *** dpawlik has joined #openstack-infra | 13:54 | |
| *** amoralej|lunch is now known as amoralej | 13:58 | |
| *** dave-mccowan has joined #openstack-infra | 14:00 | |
| *** eharney has joined #openstack-infra | 14:03 | |
| Shrews | infra-root: Most of mordred's topic:gerrit-images changes have 3 +2's now. Do we want to begin landing those? | 14:03 |
| *** ykarel is now known as ykarel|meeting | 14:04 | |
| *** mattw4 has joined #openstack-infra | 14:04 | |
| clarkb | Shrews: I was planning on lettibg mordred land what he had tested on -dev but wasnt sure what those were so left off +As | 14:04 |
| Shrews | clarkb: k. probably best to let mordred control the landing | 14:04 |
| AJaeger | clarkb: good morning. Good progress on openSUSE 150 changes but stein is causing problems. osa is fixing their branches right now. | 14:05 |
| AJaeger | clarkb: I wonder whether we should force-merge https://review.opendev.org/677181 for bifrost? | 14:06 |
| *** mattw4 has quit IRC | 14:16 | |
| *** diablo_rojo has joined #openstack-infra | 14:16 | |
| *** jaosorior has joined #openstack-infra | 14:21 | |
| *** jamesmcarthur has joined #openstack-infra | 14:24 | |
| *** goldyfruit_ has joined #openstack-infra | 14:28 | |
| *** goldyfruit___ has quit IRC | 14:31 | |
| *** jamesmcarthur has quit IRC | 14:31 | |
| *** anteaya has joined #openstack-infra | 14:37 | |
| openstackgerrit | Harald Jensås proposed openstack/diskimage-builder master: WIP: Add IPv6 support in dhcp-all-interfaces https://review.opendev.org/692110 | 14:43 |
| *** rkukura has quit IRC | 14:47 | |
| *** Goneri has quit IRC | 14:49 | |
| *** ykarel|meeting is now known as ykarel | 15:01 | |
| *** xek_ has joined #openstack-infra | 15:03 | |
| *** amoralej is now known as amoralej|off | 15:03 | |
| *** jpena is now known as jpena|off | 15:04 | |
| *** yamamoto has quit IRC | 15:04 | |
| *** xek has quit IRC | 15:05 | |
| *** eharney has quit IRC | 15:05 | |
| *** jaosorior has quit IRC | 15:06 | |
| *** efried1 has joined #openstack-infra | 15:07 | |
| *** yamamoto has joined #openstack-infra | 15:08 | |
| *** weshay is now known as weshay|rover | 15:08 | |
| *** efried has quit IRC | 15:08 | |
| *** efried1 is now known as efried | 15:08 | |
| *** rfolco is now known as rfolco|ruck | 15:08 | |
| *** yamamoto has quit IRC | 15:12 | |
| *** rfolco|ruck is now known as rfolco|rucker | 15:13 | |
| openstackgerrit | David Shrewsbury proposed opendev/base-jobs master: Remove buildset_proxy reference https://review.opendev.org/692167 | 15:16 |
| *** dpawlik has quit IRC | 15:16 | |
| Shrews | clarkb: ^^ should fix us | 15:16 |
| fungi | gonna go grab lunch and run some pre-travel errands but will return in a couple hours | 15:17 |
| *** eharney has joined #openstack-infra | 15:19 | |
| *** rkukura has joined #openstack-infra | 15:19 | |
| *** elod has quit IRC | 15:19 | |
| *** elod has joined #openstack-infra | 15:19 | |
| *** lpetrut has quit IRC | 15:22 | |
| AJaeger | infra-root, anything wrong with nodepool? Looking at http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1 I see a max of 1064 nodes but only 500 in use. and there are requests for more. | 15:23 |
| AJaeger | infra-root, looks like Rackspace is down, see http://grafana.openstack.org/d/8wFIHcSiz/nodepool-rackspace?orgId=1 | 15:24 |
| *** gyee has joined #openstack-infra | 15:24 | |
| *** rfolco|rucker has quit IRC | 15:26 | |
| corvus | Detailed error for node 936cf8e3-3dfc-4dab-8725-dfdf94d9371d: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 936cf8e3-3dfc-4dab-8725-dfdf94d9371d. Last exception: ['SR_BACKEND_FAILURE_110', '', "VDI resize failed [opterr=Command ['/usr/bin/vhd-util', 'revert', '--debug', '-n', '/ | 15:26 |
| corvus | clarkb, Shrews, AJaeger, fungi: did i see people talking about images yesterday? | 15:27 |
| AJaeger | corvus: yes, saw something in backlog, let's check that... | 15:27 |
| Shrews | not sure | 15:28 |
| AJaeger | mordred, ianw ^ | 15:28 |
| AJaeger | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/latest.log.html#t2019-10-30T00:55:36 | 15:28 |
| AJaeger | ianw wanted to release a new diskimage-builder with a fix | 15:28 |
| *** jamesmcarthur has joined #openstack-infra | 15:29 | |
| AJaeger | which he did - https://review.opendev.org/691998 is in 2.29.1 | 15:29 |
| AJaeger | where those images build with older dib and we need to rebuild with current? | 15:30 |
| Shrews | corvus: is https://review.opendev.org/692167 the right way to handle that, or should we have a check for a running buildset_proxy before attempting to gather the logs? The separation between starting containers and gathering logs for them made that difficult to catch. | 15:32 |
| corvus | Shrews: yep that's right, thanks. it's gone and not coming back. | 15:32 |
| Shrews | cool | 15:33 |
| corvus | Shrews, AJaeger: there are a lot of old instances in rax-dfw | 15:35 |
| corvus | like this: http://paste.openstack.org/show/785659/ | 15:36 |
| Shrews | empty properties? shouldn't that have nodepool specific info? | 15:37 |
| corvus | yeah, maybe they leaked because of that? | 15:37 |
| Shrews | nothing in nodepool changed around that. | 15:37 |
| Shrews | that would definitely cause a leak though | 15:37 |
| *** dciabrin has quit IRC | 15:37 | |
| corvus | anyone have any idea how i can ask for the quota via openstack cli? | 15:39 |
| Shrews | i can't find that server id in launcher logs | 15:39 |
| corvus | it's ancient look at the dates | 15:39 |
| nicholas | in novaclient it used to be quota-show , not sure if it's same command in openstack cli | 15:39 |
| *** michael-beaver has joined #openstack-infra | 15:39 | |
| corvus | nicholas: Service 'network' is disabled because its configuration could not be loaded. | 15:40 |
| Shrews | corvus: oh, i was looking at updated date | 15:40 |
| Shrews | which is still older than our logs | 15:40 |
| *** jamesmcarthur has quit IRC | 15:41 | |
| openstackgerrit | Merged opendev/base-jobs master: Remove buildset_proxy reference https://review.opendev.org/692167 | 15:42 |
| corvus | AJaeger, Shrews: i have to go catch a plane. i'm sorry i couldn't help more. i still have no idea what our quota is currently set to, so i can't even investigate whether it has changed or we are exceeding it. i think that's still something to pursue. | 15:43 |
| corvus | if anyone figures out how to check quota in openstack, please update https://docs.openstack.org/infra/system-config/nodepool.html | 15:43 |
| AJaeger | corvus: safe travels! | 15:47 |
| nicholas | talking about rax quota ? | 15:47 |
| *** armax has joined #openstack-infra | 15:48 | |
| AJaeger | yes | 15:48 |
| *** Goneri has joined #openstack-infra | 15:49 | |
| nicholas | someone set it to 0 | 15:49 |
| *** goldyfruit___ has joined #openstack-infra | 15:49 | |
| *** jaosorior has joined #openstack-infra | 15:51 | |
| *** goldyfruit_ has quit IRC | 15:52 | |
| AJaeger | any idea why? are you from rackspace? | 15:52 |
| nicholas | i'm not sure why, but i am from rackspace | 15:53 |
| nicholas | something about using a bad base image | 15:54 |
| AJaeger | that should have been solved with new diskimage-builder. Seems we need an infra-root to check that we have working images up again. | 15:55 |
| AJaeger | (and some are just on the way to Shanghai for the summmit) | 15:55 |
| *** ykarel is now known as ykarel|away | 15:55 | |
| AJaeger | nicholas: thanks! | 15:55 |
| * AJaeger cannot do anything. | 15:55 | |
| nicholas | i see there's a ticket as well | 15:55 |
| AJaeger | #status log Rackspace quota is at 0 apparently due to broken base images. | 15:56 |
| *** armax has quit IRC | 15:56 | |
| clarkb | the issue is related to qemu-img built images from dib | 15:56 |
| clarkb | dib was updated to revert that change | 15:56 |
| clarkb | new images should be built correctly | 15:56 |
| AJaeger | clarkb: do we have new images now? | 15:57 |
| clarkb | I dont know. Slow start today. But I can check soon | 15:57 |
| AJaeger | infra-root, we lost our boot - #status is not working | 15:57 |
| *** xarses_ has joined #openstack-infra | 15:57 | |
| AJaeger | it's one of those days ;/ | 15:57 |
| corvus | infra-root: i see the ticket in the openstackjenkins account; we should probably verify that we only have correct images uploaded, delete all instances, then ask on the ticket for quota to be reinstated. there's more info with problematic image ids on the ticket. | 15:57 |
| * corvus really afks again | 15:58 | |
| clarkb | nicholas: the issur is vhd utils checks if it made the image and if it didnt it fails even though the image qemu-img created is fine (tested by updating qemu-img to claim it is vhd-utils) | 15:59 |
| clarkb | nicholas: I believe that ianw found all that tap codehadbeen removed from xen evebtually too | 15:59 |
| *** xarses has quit IRC | 16:00 | |
| clarkb | this is a case where nodepool's behavior is actually really annoying | 16:04 |
| clarkb | I want to delete all the centos-8 images from rax | 16:04 |
| clarkb | but as soon as I do nodepool will reupload the bad image iirc | 16:05 |
| clarkb | but I don't want to delete the bad image because only the vhd is bad and not the other formats | 16:05 |
| *** kjackal has quit IRC | 16:05 | |
| clarkb | ya its uploading the bad centos-8 image now :( | 16:07 |
| Shrews | clarkb: i believe if you pause the image, it will neither be rebuilt or uploaded | 16:07 |
| clarkb | Shrews: we want it to be built | 16:08 |
| clarkb | which is the problem | 16:08 |
| *** xarses_ has quit IRC | 16:08 | |
| *** armax has joined #openstack-infra | 16:09 | |
| Shrews | clarkb: in dfw, i see several active instances without anything in 'properties'. from the april/may time period, looks like. should i begin manually deleting those? | 16:11 |
| *** kjackal has joined #openstack-infra | 16:11 | |
| Shrews | (no idea how they end up with no metadata) | 16:11 |
| clarkb | Shrews: what are their instance names? I would expect them to be not booted by nodepool if they don't have properties | 16:12 |
| clarkb | but if the instance names match nodepool instance names then I would double check for holds and if not held then delete | 16:12 |
| Shrews | ubuntu-xenial-rax-dfw-*, centos-7-rax-dfw-*, ubuntu-bionic-rax-dfw-* | 16:12 |
| clarkb | probably fine then | 16:13 |
| Shrews | we don't have any holds that old | 16:13 |
| *** mattw4 has joined #openstack-infra | 16:14 | |
| *** jamesmcarthur has joined #openstack-infra | 16:18 | |
| clarkb | ok this is actually really awkward. I keep thinking I can delete images that are bad and then they start getting reuploaded | 16:19 |
| clarkb | I'm going to stop deleting images and focus on queuing up image builds | 16:19 |
| Shrews | starting to delete old instances, from oldest to newest | 16:21 |
| *** igordc has joined #openstack-infra | 16:21 | |
| *** jamesmcarthur has quit IRC | 16:22 | |
| clarkb | only nb02 is building an image according to dib-image-list but nb01 has a dib process running | 16:22 |
| *** jamesmcarthur has joined #openstack-infra | 16:22 | |
| *** kopecmartin|scho is now known as kopecmartin|off | 16:24 | |
| clarkb | I'm going to clean up nb01 and reboot and bring it back up again so that it communicates to zk hopefully | 16:24 |
| clarkb | Shrews: I think we need a way to represent a state where we want remove an image from a particular provider (and not all providers) while allowing rebuilds | 16:25 |
| clarkb | a different kind of pause basically | 16:25 |
| *** pgaxatte has quit IRC | 16:26 | |
| *** pgaxatte has joined #openstack-infra | 16:27 | |
| clarkb | based on image build times I expect it will be towards the end of the day before we can turn rax back on | 16:34 |
| clarkb | if anyone has better ideas on how to convince nodepool to use older images after deleting newer images while still building replacement images that may speed it up | 16:34 |
| *** ricolin has quit IRC | 16:41 | |
| *** hashar has quit IRC | 16:49 | |
| *** cjohnston has joined #openstack-infra | 16:53 | |
| Shrews | infra-root: ok, rax ORD and DFW are both cleaned up of nodepool created instances that had an empty properties value. I have no idea how that could have happened, but most seemed to be from last year, and one as recently as Jan 2019. Probably something we may want to keep an eye on. I left a few that had some values in properties. Will have to look at those later after lunching. | 16:54 |
| *** lucasagomes has quit IRC | 16:54 | |
| clarkb | ok both builders are building images now | 16:55 |
| clarkb | we have ~8 images to build I think | 16:56 |
| clarkb | I have issued explicit built requests for all of the images that I believe need updating | 16:56 |
| *** diablo_rojo has quit IRC | 16:57 | |
| *** rpittau is now known as rpittau|afk | 16:59 | |
| clarkb | AJaeger: nicholas ^ fyi at this point we are just waiting for all the new images to be built using vhd-util so that rax xen doesn't complain | 16:59 |
| clarkb | when that is done we can update the ticket and request a revert on the quota change | 16:59 |
| *** hashar has joined #openstack-infra | 16:59 | |
| AJaeger | clarkb: thanks! | 17:01 |
| *** openstackstatus has joined #openstack-infra | 17:03 | |
| *** ChanServ sets mode: +v openstackstatus | 17:03 | |
| clarkb | AJaeger: statusbot is back now | 17:03 |
| clarkb | (I restarted it | 17:03 |
| clarkb | #status log Rax quota set to 0 while we rebuild images with vhd-util instead of qemu-img so that they can be resized in rax. Once images are updated we should request quote be reset back to normal | 17:04 |
| openstackstatus | clarkb: finished logging | 17:04 |
| *** rfolco has joined #openstack-infra | 17:04 | |
| AJaeger | welcome back, openstackstatus ;) | 17:05 |
| *** dayou has quit IRC | 17:12 | |
| clarkb | these are the images I think we need updates on after the current point in time. debian-stretch, debian-buster, fedora-29, gentoo-17, opensuse-150, opensuse-tumbleweed, ubuntu-bionic | 17:14 |
| clarkb | the other images have been updated already | 17:14 |
| clarkb | gentoo and fedora-29 are building now and that should take just over an hour. figure ~4.5 hours or so probably for all of them? | 17:15 |
| *** dayou has joined #openstack-infra | 17:15 | |
| clarkb | I'm going to pop out for breakfast since we are waiting anyway, nodepool should build all of those images as it finishes others | 17:16 |
| *** dpawlik has joined #openstack-infra | 17:23 | |
| *** Garyx_ has quit IRC | 17:25 | |
| fungi | okay, back from lunch/errands and seeing lots of pings. will try to follow up in order | 17:26 |
| *** pgaxatte has quit IRC | 17:28 | |
| *** weifan has joined #openstack-infra | 17:29 | |
| *** xek_ has quit IRC | 17:29 | |
| *** weifan has quit IRC | 17:31 | |
| fungi | ahh, looks like things have settled down and we're just watching for images to rebuild now | 17:31 |
| *** weifan has joined #openstack-infra | 17:31 | |
| clarkb | looks like nb02 disconnected from zk too? | 17:37 |
| clarkb | it is still building the gentoo iamge but dib-image-list has no record of it | 17:37 |
| clarkb | we may want to clean up that server too | 17:37 |
| *** jamesmcarthur has quit IRC | 17:38 | |
| *** jamesmcarthur has joined #openstack-infra | 17:39 | |
| clarkb | ya I'm going to get that started | 17:39 |
| *** jamesmcarthur has quit IRC | 17:44 | |
| *** hashar has quit IRC | 17:44 | |
| *** dtantsur is now known as dtantsur|afk | 17:49 | |
| *** jamesmcarthur has joined #openstack-infra | 17:51 | |
| *** Garyx has joined #openstack-infra | 17:51 | |
| clarkb | fedora-29 is uploading now | 17:52 |
| clarkb | bionic is building | 17:52 |
| *** jamesmcarthur has quit IRC | 17:58 | |
| clarkb | nb02 is back in the rotation | 18:00 |
| *** jamesmcarthur has joined #openstack-infra | 18:00 | |
| *** hashar has joined #openstack-infra | 18:01 | |
| *** jamesmcarthur has quit IRC | 18:05 | |
| *** jerryz has joined #openstack-infra | 18:06 | |
| *** pcaruana has quit IRC | 18:10 | |
| *** jamesmcarthur has joined #openstack-infra | 18:13 | |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for nova functional test fail bug 1850682 https://review.opendev.org/692203 | 18:13 |
| openstack | bug 1850682 in OpenStack Compute (nova) "functional tests in rocky randomly fail with "Build of instance was re-scheduled: Cannot modify readonly field uuid"" [Undecided,Confirmed] https://launchpad.net/bugs/1850682 | 18:13 |
| *** eharney has quit IRC | 18:14 | |
| *** jamesmcarthur has quit IRC | 18:15 | |
| *** jamesmcarthur has joined #openstack-infra | 18:16 | |
| *** chandankumar is now known as raukadah | 18:17 | |
| *** dciabrin has joined #openstack-infra | 18:18 | |
| *** hwoarang has quit IRC | 18:21 | |
| *** hwoarang has joined #openstack-infra | 18:22 | |
| *** jamesmcarthur has quit IRC | 18:23 | |
| *** jamesmcarthur has joined #openstack-infra | 18:24 | |
| *** dpawlik has quit IRC | 18:26 | |
| *** ykarel|away has quit IRC | 18:31 | |
| *** aedc has quit IRC | 18:36 | |
| *** dpawlik has joined #openstack-infra | 18:37 | |
| nicholas | clarkb: let me know if there's anything i can do to help out | 18:40 |
| *** dciabrin has quit IRC | 18:42 | |
| *** Goneri has quit IRC | 18:46 | |
| *** jamesmcarthur has quit IRC | 18:58 | |
| *** jamesmcarthur has joined #openstack-infra | 18:58 | |
| *** tesseract has quit IRC | 19:03 | |
| *** jamesmcarthur has quit IRC | 19:03 | |
| *** eharney has joined #openstack-infra | 19:03 | |
| *** pcaruana has joined #openstack-infra | 19:05 | |
| *** dpawlik has quit IRC | 19:16 | |
| *** jamesmcarthur has joined #openstack-infra | 19:16 | |
| *** jaosorior has quit IRC | 19:21 | |
| *** prometheanfire has quit IRC | 19:21 | |
| *** prometheanfire has joined #openstack-infra | 19:21 | |
| *** hashar has quit IRC | 19:24 | |
| *** gfidente is now known as gfidente|afk | 19:28 | |
| *** pcaruana has quit IRC | 19:29 | |
| *** jamesmcarthur has quit IRC | 19:34 | |
| *** Goneri has joined #openstack-infra | 19:36 | |
| *** kjackal has quit IRC | 19:39 | |
| *** pcaruana has joined #openstack-infra | 19:39 | |
| *** kjackal_v2 has joined #openstack-infra | 19:39 | |
| efried | o/ infra | 19:40 |
| efried | Yesterday mordred helpfully held the node for one of the failing jobs here https://review.opendev.org/#/c/691980/ (not sure which one) and set up my ssh key. | 19:40 |
| efried | But I'm having trouble connecting, and could use some help... | 19:40 |
| efried | The problem is possibly that I can't IPv6 from here, but I'm not even sure how to figure that out (never IPv6ed in my life). | 19:41 |
| clarkb | efried: if on linux you want to run `ip addr` or `ifconfig` in a terminal to get interface and address listings you are looking for an ipv6 address that is marked global | 19:42 |
| clarkb | if you don't have one of those then no ipv6 global routing from your host | 19:42 |
| clarkb | if you do and it isn't working then we'll have to dig in more | 19:42 |
| clarkb | if on windows I think you can run ipconfig | 19:42 |
| clarkb | not sure on osx | 19:43 |
| efried | I'm on bionic | 19:43 |
| *** aedc has joined #openstack-infra | 19:43 | |
| efried | ip addr | grep global only shows a v4 addr | 19:43 |
| clarkb | there is a good chance that you don't have ipv6 wherever you currently are then. I have this problem too and bounce through a cheap ovh instance. You can also set up a hurricane electric tunnel | 19:44 |
| clarkb | also most cellphone networks are ipv6 enabled hwoever when you tether on them they tend to be ipv4 only :/ | 19:44 |
| efried | egads, all of those things sound complicated. Chances of me doing a lot of this are pretty slim; is it worth it for a one-off? | 19:44 |
| clarkb | if you want access to an instance in an ipv6 only cloud its your only option | 19:45 |
| efried | I'm on home network (wifi to my cable modem) | 19:45 |
| clarkb | we can't add ipv4 to those instances | 19:45 |
| * clarkb double checks the hold list | 19:45 | |
| efried | okay, which thing is easiest? | 19:45 |
| clarkb | yes that is a fortnebula instance which is an ipv6 only cloud | 19:46 |
| efried | oh, donnyd has been able to set up an ipv4 jump station for me in the past... | 19:46 |
| efried | seemed to be able to do it by snapping his fingers | 19:46 |
| donnyd | efried: what do you need | 19:46 |
| efried | but it's all magic from where I sit. | 19:46 |
| efried | donnyd: a way to ipv4 to... | 19:47 |
| efried | [pm'd] | 19:47 |
| efried | clarkb: how much longer is that hold good for? Like another 2.5h or so? | 19:48 |
| clarkb | I don't think they expire by default | 19:48 |
| clarkb | should be good until you tell us it isn't | 19:48 |
| efried | okay, mordred said something about 24h | 19:49 |
| clarkb | oh hrm let me double check on the zuul side then | 19:49 |
| efried | FYI I'm unbreaking python-openstackclient builds, which went pear-shaped when something or other switched them to py3 (oops, they hadn't been testing under py3 prior to that) | 19:49 |
| efried | and need the node because I can't run that thing locally | 19:50 |
| clarkb | autohold list doesn't show any timeout | 19:50 |
| clarkb | Shrews: ^ how do we check that? | 19:50 |
| clarkb | infra-root we are down to opensuse-tumbleweed, gentoo-17, debian-buster and debian-stretch images needing to go to rax | 19:51 |
| clarkb | I've been trying to delete the older images as the new ones update too to make it clear which are good | 19:51 |
| efried | clarkb, Shrews: it may have just been speculation on mordred's part http://eavesdrop.openstack.org/irclogs/%23openstack-sdks/%23openstack-sdks.2019-10-29.log.html#t2019-10-29T22:22:58 -- this could be just a wild goose chase. | 19:51 |
| donnyd | efried: finger snap complete | 19:58 |
| efried | kablam | 19:58 |
| sshnaidm | cores, please review fix for os_image module in ansible: https://github.com/ansible/ansible/pull/63959 | 20:05 |
| ianw | clarkb / everyone : sorry for the mess on that, i thought they'd clear faster | 20:07 |
| clarkb | ianw: no worries. I think this is only weird/painful due to how nodepool works | 20:08 |
| donnyd | lmk if there is anything else I can do to help :) | 20:08 |
| clarkb | maybe we can brainstorm how to do that better at the summit/forum/ptg | 20:08 |
| clarkb | basically we can't force nodepool to use an older image now and also build new images | 20:08 |
| clarkb | which would've soled the problem quickly | 20:08 |
| ianw | mordred: re our conversation, i have tried every permutation i can come up with creating vhd's with patched qemu and while i can get them to boot, they don't grow | 20:08 |
| *** pcaruana has quit IRC | 20:09 | |
| ianw | the "only" difference is the vhd-utils based images have the "batmap" table (the xen extension not mentioned in the microsoft vpc spec) | 20:09 |
| ianw | although it looks like xen's resize utils should handle "legacy" images without it, it seems like they don't | 20:09 |
| ianw | i'm going to write this all up in a note so that we don't have to debug this far again | 20:10 |
| * fungi imagines a note saying something to the effect of "this is here while we wait for xen to finally die" | 20:10 | |
| *** gfidente|afk has quit IRC | 20:11 | |
| fungi | it's unfortunate that they would extend a specification in undocumented ways and not work to get support for that extension into mainstream tooling | 20:11 |
| *** rfolco has quit IRC | 20:12 | |
| fungi | but given the dwindling interest in xen, i have doubts anyone cares enough to solve that properly now | 20:12 |
| *** jamesmcarthur has joined #openstack-infra | 20:13 | |
| *** aedc has quit IRC | 20:14 | |
| *** Goneri has quit IRC | 20:22 | |
| ianw | fungi: yeah, the problem is that qemu-img creates a tempting close replica. there's patches out there that add support for the batmap *and* overlay disks | 20:23 |
| ianw | it's all stuffed in random mailing list posts as is the development model | 20:24 |
| ianw | I also found blktap sort of exists ... but has this rather comforting INSTALL note from 7+ years ago ... https://github.com/xapi-project/blktap/blob/master/INSTALL | 20:26 |
| *** hashar has joined #openstack-infra | 20:28 | |
| *** jamesmcarthur has quit IRC | 20:29 | |
| *** adriant has quit IRC | 20:31 | |
| *** iokiwi has quit IRC | 20:31 | |
| fungi | indeed, that really makes me want to use it | 20:34 |
| *** diablo_rojo has joined #openstack-infra | 20:35 | |
| ianw | nicholas: so can you see logs of instances booting on the rax side? | 20:39 |
| *** goldyfruit___ has quit IRC | 20:42 | |
| *** ralonsoh has quit IRC | 20:46 | |
| *** iokiwi has joined #openstack-infra | 20:49 | |
| fungi | oh, also, mordred suggested there was no vhd-util in ubuntu, looks like it was dropped from debian/unstable in january: https://bugs.debian.org/917907 | 20:49 |
| openstack | Debian bug 917907 in ftp.debian.org "RM: blktap -- RoQA; obsolete, FTBFS" [Normal,Open] | 20:49 |
| *** jamesmcarthur has joined #openstack-infra | 20:50 | |
| ianw | yeah, fair enough as upstream xen rf -rf'd the whole thing | 20:50 |
| fungi | as a result the last ubuntu lts release to include a blktap-utils package was xenial: https://packages.ubuntu.com/blktap-utils | 20:51 |
| *** dciabrin has joined #openstack-infra | 20:54 | |
| *** weifan has quit IRC | 20:54 | |
| *** markvoelker has quit IRC | 20:55 | |
| *** markvoelker has joined #openstack-infra | 20:55 | |
| *** hashar has quit IRC | 20:55 | |
| *** dciabrin_ has joined #openstack-infra | 20:58 | |
| ianw | yeah, this whole thing is a yak shave from me thinking i want to have a bionic builder (and pabelanger note on the same) | 20:58 |
| *** markvoelker has quit IRC | 20:59 | |
| *** jamesmcarthur has quit IRC | 21:00 | |
| *** weifan has joined #openstack-infra | 21:00 | |
| *** dciabrin has quit IRC | 21:02 | |
| *** hashar has joined #openstack-infra | 21:03 | |
| *** ociuhandu has joined #openstack-infra | 21:05 | |
| *** jamesmcarthur has joined #openstack-infra | 21:07 | |
| nicholas | ianw: yes | 21:08 |
| *** jamesmcarthur_ has joined #openstack-infra | 21:09 | |
| ianw | nicholas: so i'd be interested to see why the instance i've created with a patched qemu-img hasn't managed to grow it's disk, or if it tried | 21:09 |
| ianw | it's running @ 104.239.143.79 now | 21:10 |
| ianw | id is 1f535e76-3e8f-4d6b-80b4-06313f1a150c | 21:11 |
| *** jamesmcarthur has quit IRC | 21:12 | |
| *** panda|pto has quit IRC | 21:14 | |
| nicholas | which region is that | 21:14 |
| clarkb | we are now waiting on buster and stretch. buster is uploading now and stretch is building | 21:14 |
| clarkb | ianw: ^ fyi | 21:14 |
| clarkb | we should be good to go in about 1.5 hours I think | 21:15 |
| ianw | nicholas: DFW | 21:15 |
| clarkb | I've tried to delete the bad images for all the other images so far as well | 21:15 |
| *** panda has joined #openstack-infra | 21:18 | |
| *** jamesmcarthur_ has quit IRC | 21:24 | |
| nicholas | ianw: it looks like there are a few vbd's attached... grepping the UUIDs generates a ton of logs.. | 21:24 |
| nicholas | which device? might help me target better | 21:24 |
| ianw | nicholas: hrm, that should just have one main disk | 21:25 |
| ianw | i has booted from the image 95e9b40b-4161-45e8-bbdb-02f0dc29bb9a | 21:25 |
| ianw | "ubuntu-bionic-qemu" ... it's one i generated with a patched qemu-img to set the creator to 'tap\0' with a low version number | 21:26 |
| *** jamesmcarthur has joined #openstack-infra | 21:27 | |
| nicholas | yeah i see the instance, it's showing 3 vbds: hda, xvde, hdd | 21:27 |
| *** ociuhandu has quit IRC | 21:27 | |
| nicholas | i'm not a xen guru fwiw :) i used to be on the ops/eng team back in the day but i've been on a different team for a few years now | 21:27 |
| ianw | umm, so xvde would be the ephemeral storage ... not that one. hda i guess? | 21:27 |
| ianw | /dev/xvdd: UUID="2019-10-30-08-05-30-00" LABEL="config-2" TYPE="iso9660" | 21:28 |
| ianw | would be configdrive | 21:28 |
| *** jamesmcarthur has quit IRC | 21:29 | |
| *** ociuhandu has joined #openstack-infra | 21:29 | |
| jrosser | perhaps some console streams may have gone awol again | 21:30 |
| jrosser | i believe this is running right now http://zuul.openstack.org/stream/8f6bfb46f4234d0ea5c02bb32207bcfb?logfile=console.log | 21:30 |
| *** jamesmcarthur has joined #openstack-infra | 21:32 | |
| nicholas | ianw: my xen fu is rusty, but it appears to have successfully attempted it: [1739] ['/usr/bin/vhd-util', 'resize', '--debug', '-s', '40960', '-n', '/var/run/sr-mount/da64c4c1-e5b8-2ac7-b24a-ccad9e4cfdf7/600e1f41-d11c-4e45-b5f5-6acbdd24cb16.vhd', '-j', '.journal-600e1f41-d11c-4e45-b5f5-6acbdd24cb16'] | 21:34 |
| ianw | xvda 202:0 0 16.3G 0 disk | 21:35 |
| ianw | └─xvda1 202:1 0 16.3G 0 part / | 21:35 |
| ianw | yeah, unfortunately the host doesn't seem to see that :/ | 21:35 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: vhd-util : note on Xen/RAX images https://review.opendev.org/692234 | 21:36 |
| ianw | nicholas: ^ that's pretty much the summary at this point | 21:37 |
| ianw | clearly other customers are not banging down rax's door to create custom vhd images from raw files ... | 21:38 |
| ianw | i'm guessing this all works better from a windows environment | 21:38 |
| *** iurygregory has quit IRC | 21:40 | |
| *** ociuhandu has quit IRC | 21:43 | |
| *** jamesmcarthur has quit IRC | 21:44 | |
| *** jamesmcarthur has joined #openstack-infra | 21:44 | |
| *** jmccrory has joined #openstack-infra | 21:46 | |
| *** jamesmcarthur has quit IRC | 21:49 | |
| *** jamesmcarthur has joined #openstack-infra | 21:50 | |
| nicholas | ianw: i need to head afk for the evening, but if you need any logs or anything, you can update the ticket and the ops folks can pull it for you | 21:51 |
| ianw | nicholas: np, thanks for the help | 21:53 |
| clarkb | jrosser: I don't knwo that we restarted any of the executors after we caught hte last failed set | 21:55 |
| clarkb | fungi: ^ | 21:55 |
| *** ociuhandu has joined #openstack-infra | 21:55 | |
| *** jamesmcarthur has quit IRC | 21:55 | |
| jrosser | clarkb: ah i guess thats quite invasive? | 21:55 |
| *** jamesmcarthur has joined #openstack-infra | 21:56 | |
| clarkb | jrosser: it stops all running jobs and restarts them | 21:56 |
| fungi | i thought they got restarted again after that for a new zuul version | 21:57 |
| fungi | but checking | 21:57 |
| openstackgerrit | Harald Jensås proposed openstack/diskimage-builder master: WIP: Add IPv6 support in dhcp-all-interfaces https://review.opendev.org/692110 | 21:59 |
| fungi | yeah, they did, but since the last restart an oom incident has killed the finger listener process on ze09 and ze12 | 22:00 |
| *** jamesmcarthur has quit IRC | 22:00 | |
| fungi | they were all restarted around 16:00z on 2019-10-24 | 22:01 |
| fungi | so a little less than a week ago | 22:01 |
| clarkb | now down to just stretch | 22:03 |
| clarkb | the build for stretch should complete soon then we wait for uploads and I think we are good | 22:03 |
| *** ociuhandu has quit IRC | 22:04 | |
| clarkb | and now uploading stretch | 22:09 |
| *** ociuhandu has joined #openstack-infra | 22:10 | |
| *** hashar has quit IRC | 22:15 | |
| *** ociuhandu has quit IRC | 22:15 | |
| *** kjackal_v2 has quit IRC | 22:16 | |
| *** hashar has joined #openstack-infra | 22:17 | |
| *** ociuhandu has joined #openstack-infra | 22:20 | |
| efried | mordred, clarkb, donnyd: I'm done with that node and the v4 jumper. Thank you for the help! | 22:23 |
| donnyd | Any time efried | 22:23 |
| *** igordc has quit IRC | 22:25 | |
| *** weifan has quit IRC | 22:26 | |
| *** goldyfruit___ has joined #openstack-infra | 22:27 | |
| *** ociuhandu has quit IRC | 22:31 | |
| *** ociuhandu has joined #openstack-infra | 22:33 | |
| *** pkopec has quit IRC | 22:33 | |
| *** ociuhandu has quit IRC | 22:37 | |
| *** ociuhandu has joined #openstack-infra | 22:38 | |
| *** mriedem has quit IRC | 22:40 | |
| *** hashar has quit IRC | 22:42 | |
| *** ociuhandu has quit IRC | 22:43 | |
| *** jklare has quit IRC | 22:45 | |
| *** jklare has joined #openstack-infra | 22:48 | |
| *** dchen has joined #openstack-infra | 22:48 | |
| clarkb | ianw: nicholas fungi AJaeger I think we are good to go in rax now | 22:48 |
| clarkb | stretch miages uploaded and older ones deleted | 22:49 |
| clarkb | I'm not in a great spot to go update the ticket, maybe ianw can do that? but I think all our images should boot now | 22:49 |
| ianw | clarkb: will do | 22:50 |
| *** adriant has joined #openstack-infra | 22:51 | |
| clarkb | ianw: note the fedora-30 images are ~20 days old so are unaffected (I'm guessing that is pre dnf stopped working) | 22:51 |
| *** goldyfruit___ has quit IRC | 22:51 | |
| clarkb | other than that all the images should be from within the last 12 hours or so | 22:51 |
| ianw | clarkb: yep, we can turn that back on now, the dnf fixes have made it to mainline | 22:52 |
| *** pkopec has joined #openstack-infra | 22:52 | |
| *** markvoelker has joined #openstack-infra | 22:56 | |
| *** fresta has quit IRC | 22:59 | |
| *** fresta has joined #openstack-infra | 22:59 | |
| *** rlandy has quit IRC | 23:00 | |
| *** markvoelker has quit IRC | 23:00 | |
| *** tkajinam has joined #openstack-infra | 23:01 | |
| *** tkajinam has quit IRC | 23:01 | |
| ianw | i updated the ticket, with links to https://review.opendev.org/692234 ... maybe rax have an interest in fixing this, maybe not ... happy to work with them if they do, we might be able to get qemu-img making compatible images | 23:03 |
| *** tkajinam has joined #openstack-infra | 23:04 | |
| *** slaweq has quit IRC | 23:05 | |
| *** slaweq has joined #openstack-infra | 23:10 | |
| *** weifan has joined #openstack-infra | 23:11 | |
| *** mattw4 has quit IRC | 23:14 | |
| *** slaweq has quit IRC | 23:16 | |
| mordred | clarkb: if you have a sec, feel like reviewing https://review.opendev.org/#/c/692003/ - which is the patch containing the fixes from your review of the first patch? | 23:18 |
| clarkb | done | 23:19 |
| mordred | clarkb: woot. thanis | 23:21 |
| mordred | tahnks | 23:21 |
| mordred | zomg | 23:21 |
| mordred | T H A N K S | 23:21 |
| clarkb | don't worry I can't type either :) | 23:21 |
| clarkb | ianw: nicholas If I understand correctly we are waiting for rax to revert the quotas back to what they were now? | 23:32 |
| fungi | that sounds right based on my reading of scrollback from earlier | 23:36 |
| *** ociuhandu has joined #openstack-infra | 23:43 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed zuul/zuul-jobs master: Remove argument to ssh-keygen for key size https://review.opendev.org/692244 | 23:44 |
| *** EmilienM has quit IRC | 23:47 | |
| *** ociuhandu has quit IRC | 23:48 | |
| *** EmilienM has joined #openstack-infra | 23:48 | |
| *** weifan has quit IRC | 23:59 | |
| *** jklare has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!