| *** mattw4 has quit IRC | 00:00 | |
| clarkb | that one is 5d31e134-3abf-4802-b7d5-df4824b79e95 | 00:00 |
|---|---|---|
| clarkb | also available | 00:00 |
| fungi | wacky | 00:00 |
| *** eernst has quit IRC | 00:00 | |
| clarkb | is it racing maybe? | 00:01 |
| clarkb | the volume isn't ready for nova when noav tries to use it? | 00:01 |
| fungi | ~clarkb/.bash_history doesn't seem to have any bfv examples in it | 00:01 |
| *** eernst has joined #openstack-infra | 00:02 | |
| clarkb | fungi: `sudo ./launch-node.py $FQDN --flavor "$FLAVOR" --cloud=$OS_CLOUD --region=$OS_REGION_NAME --image $IMAGE --boot-from-volume --volume-size 80 --config-drive --network public` | 00:02 |
| fungi | oh, wait, theres one | 00:02 |
| fungi | --config-drive --network public | 00:02 |
| fungi | differs from what i used | 00:02 |
| fungi | do we need to specify --config-drive? | 00:03 |
| clarkb | oh ya we need config drive (I wish that was just default standard nova behavior) | 00:03 |
| clarkb | because the minimal images use glean | 00:03 |
| clarkb | however that shouldn't affect the volume thing unless it is a race and that slows down nova enough | 00:03 |
| clarkb | fungi: you should delete the leaked volumes too or should I go ahead and do that? | 00:03 |
| fungi | trying but yeah i have a feeling it's not getting that far | 00:03 |
| fungi | i can clean them up in a sec | 00:04 |
| fungi | how were you identifying them since they don't mention the server? | 00:04 |
| clarkb | k. The two 40GB volumes that don't appear attached are for gitea01 and gitea06 I think | 00:04 |
| clarkb | fungi: timestamp and size | 00:04 |
| clarkb | fungi: and the first one was in your error message | 00:04 |
| fungi | 40gb? but i specified 80 | 00:04 |
| clarkb | fungi: there are two leaked 40GB volumes (old 06 and 01) and two leaked 80GB volumes (your recent failures) | 00:04 |
| clarkb | they are not attached to anything | 00:05 |
| fungi | oh neat it's getting farther this time | 00:05 |
| clarkb | and then using image name, timestamps and the error above you can kinda infer stuff | 00:05 |
| fungi | not sure if this is random race-winning or the options really made a difference | 00:05 |
| *** jamesmcarthur has joined #openstack-infra | 00:05 | |
| clarkb | I think its gonna be race winning | 00:05 |
| clarkb | because all you chaged was use public network instead of public network uuid and add a config drive | 00:06 |
| clarkb | the add a config drive step is likely making nova take long enough that cinder is ready with a volume in time | 00:06 |
| openstackgerrit | Merged opendev/system-config master: Increate gerrit user connection limit by 50% https://review.opendev.org/672188 | 00:06 |
| *** eernst has quit IRC | 00:06 | |
| fungi | oh, yep that makes some sense re: nova lag | 00:06 |
| *** gyee has quit IRC | 00:07 | |
| *** eernst has joined #openstack-infra | 00:08 | |
| clarkb | we should consider setting config drive to true in launch node by default | 00:08 |
| clarkb | I always forget that one | 00:08 |
| *** tkajinam has quit IRC | 00:10 | |
| ianw | corvus: just looping back to 669780 which sets up debug logging for nodepool; using "-d" is going to foreground the process. i'm not sure on running " &" but it seems we should rely on nodepool's own daemonizing imo | 00:11 |
| ianw | i wonder if it used to run in a systemd service using run_process or if the old way just called it too | 00:12 |
| clarkb | fungi: there is a message to the infra list about opendev.org closing a connection unexpectedly. You deleted gitea01 right? | 00:12 |
| clarkb | I'm now remembering that haproxy doesn't auto reload its config because docker | 00:13 |
| *** eernst has quit IRC | 00:13 | |
| clarkb | it will notice the server is down after it goes away but any connections up at the time would be toast I think | 00:13 |
| clarkb | would probably explain that | 00:13 |
| clarkb | (did we have to delete the old one due to quota?) | 00:13 |
| ianw | to answer above : yes it used to run via devstack which did it in a service : run_process nodepool-builder "$NODEPOOL_INSTALL/bin/nodepool-builder -c $NODEPOOL_CONFIG -l $NODEPOOL_LOGGING -d" | 00:14 |
| *** eernst has joined #openstack-infra | 00:14 | |
| *** jamesmcarthur has quit IRC | 00:15 | |
| fungi | clarkb: --config-drive should only be an issue on static.o.o where every last block device assignment counts (and the configdrive uses one we could otherwise attach a cinder volume at) | 00:16 |
| fungi | so in general i agree | 00:16 |
| fungi | maybe we can invert the option at least | 00:17 |
| clarkb | fungi: ya | 00:17 |
| clarkb | well that and live migration potentially not working with config drive depending on the age of the cloud | 00:17 |
| clarkb | but maybe we want that | 00:17 |
| clarkb | I've responded to the opendev connection question with my theory and asked for more info if it happens again | 00:17 |
| fungi | and yes, if haproxy says gitea01 is still in the table but disabled due to failures, that was probably me circa 23:25z | 00:17 |
| clarkb | I think we either want toteach ansible and docker how to gracefully restart an haproxy or stick to manually removing those backends, waiting for connections to drop, then making these changes in the future | 00:18 |
| clarkb | (it would've been fine if my change had been applied) | 00:18 |
| fungi | okay, new gitea01 is 84a24c36-b8fa-4e5f-8e1a-6d80767e52fc/38.108.68.172/2604:e100:3:0:f816:3eff:fe16:274 | 00:19 |
| fungi | time to clean up stray cinder volumes | 00:19 |
| *** eernst has quit IRC | 00:19 | |
| clarkb | I've been asked to start dinner so going to go do that now | 00:20 |
| clarkb | remember to exclude the server in the remote_puppet_git.yaml playbook so that we don't configure projects on it automatically (instead want that to happen via db backup) | 00:20 |
| fungi | i may be handing the rest of gitea01 off to others or picking it up in the morning. getting weird looks from guests wondering why i'm working past 8pm | 00:20 |
| *** eernst has joined #openstack-infra | 00:21 | |
| fungi | there are currently 4 volumes showing as "available" for that tenant in sjc1 | 00:21 |
| fungi | they should all be safe to delete, in theory | 00:22 |
| fungi | the two showing as 80gb were created at times corresponding with my prior failed launch attempts | 00:23 |
| fungi | the rest are the two 40gb volumes you mentioned | 00:23 |
| clarkb | the two 80gb are definitely safe | 00:23 |
| clarkb | the 40gb ones maybe we want to double check more? | 00:23 |
| fungi | will do | 00:23 |
| fungi | okay, the two leaked 80gb volumes have been deleted now | 00:24 |
| *** eernst has quit IRC | 00:25 | |
| fungi | i see a bit of a timeline with the two available 40gb volumes | 00:26 |
| fungi | b9eb27fe-3dfe-49a8-85f4-c142b60aa06b was created 2019-02-28T13:41:45.000000 and updated 2019-02-28T13:41:50.000000 based on an image named "ubuntu-bionic-minimal" | 00:27 |
| *** eernst has joined #openstack-infra | 00:27 | |
| fungi | 7cd5e56d-8d16-4a5e-b0d6-a4b982c61e80 was created 2019-02-28T15:58:19.000000 and updated 2019-06-27T22:06:47.000000 based on the same image | 00:29 |
| fungi | (comparing image uuid, not just name) | 00:29 |
| fungi | was huh, that's weird that the second image was created the day after it was updated | 00:30 |
| fungi | anyway, https://wiki.openstack.org/wiki/Infrastructure_Status says "2019-06-27 22:12:16 UTC Gitea06 had a corrupted root disk around the time of the Denver summit. It has been replaced with a new server and added back to the haproxy config." | 00:31 |
| fungi | i'm willing to bet you tried to launch it more than once and leaked an extra image | 00:31 |
| *** eernst has quit IRC | 00:32 | |
| *** eernst has joined #openstack-infra | 00:39 | |
| *** betherly has joined #openstack-infra | 00:42 | |
| *** eernst has quit IRC | 00:43 | |
| *** eernst has joined #openstack-infra | 00:45 | |
| *** betherly has quit IRC | 00:46 | |
| *** eernst has quit IRC | 00:50 | |
| *** eernst has joined #openstack-infra | 00:51 | |
| *** hongbin has joined #openstack-infra | 00:55 | |
| *** igordc has quit IRC | 00:55 | |
| *** eernst has quit IRC | 00:57 | |
| *** eharney has quit IRC | 00:57 | |
| *** eernst has joined #openstack-infra | 00:58 | |
| fungi | i was able to delete one of the two ubuntu-bionic-minimal images (d090dd9a-cd77-46d1-afa6-97a99f54dea8) since no volumes were using it | 01:00 |
| fungi | if i delete those two "available" 40gb volumes i should be able to delete the other ubuntu-bionic-minimal images (d0edcf7a-0779-476e-8285-bcab9043b616) which their metadata says they're based on | 01:01 |
| *** eernst has quit IRC | 01:02 | |
| *** eernst has joined #openstack-infra | 01:04 | |
| *** eernst has quit IRC | 01:09 | |
| *** eernst has joined #openstack-infra | 01:11 | |
| *** eernst has quit IRC | 01:15 | |
| *** imacdonn has quit IRC | 01:18 | |
| *** imacdonn has joined #openstack-infra | 01:18 | |
| *** ricolin has joined #openstack-infra | 01:19 | |
| *** jamesmcarthur has joined #openstack-infra | 01:24 | |
| *** eernst has joined #openstack-infra | 01:25 | |
| *** eernst has quit IRC | 01:30 | |
| *** eernst has joined #openstack-infra | 01:32 | |
| *** jamesmcarthur_ has joined #openstack-infra | 01:32 | |
| *** betherly has joined #openstack-infra | 01:34 | |
| *** jamesmcarthur has quit IRC | 01:34 | |
| *** yamamoto has quit IRC | 01:34 | |
| *** eernst has quit IRC | 01:36 | |
| *** betherly has quit IRC | 01:38 | |
| *** eernst has joined #openstack-infra | 01:41 | |
| *** whoami-rajat has joined #openstack-infra | 01:43 | |
| *** rchurch has joined #openstack-infra | 01:43 | |
| *** eernst has quit IRC | 01:46 | |
| *** _erlon_ has quit IRC | 01:46 | |
| *** eernst has joined #openstack-infra | 01:48 | |
| *** jcoufal has joined #openstack-infra | 01:52 | |
| *** eernst has quit IRC | 01:52 | |
| *** rchurch has quit IRC | 01:53 | |
| *** eernst has joined #openstack-infra | 01:54 | |
| *** jcoufal has quit IRC | 01:56 | |
| *** yamamoto has joined #openstack-infra | 01:58 | |
| *** eernst has quit IRC | 01:58 | |
| *** eernst has joined #openstack-infra | 02:00 | |
| *** rchurch has joined #openstack-infra | 02:01 | |
| *** eernst has quit IRC | 02:05 | |
| *** eernst has joined #openstack-infra | 02:07 | |
| *** eernst has quit IRC | 02:11 | |
| *** eernst has joined #openstack-infra | 02:13 | |
| *** betherly has joined #openstack-infra | 02:15 | |
| *** yamamoto has quit IRC | 02:17 | |
| *** eernst has quit IRC | 02:17 | |
| *** eernst has joined #openstack-infra | 02:19 | |
| *** betherly has quit IRC | 02:20 | |
| *** yamamoto has joined #openstack-infra | 02:21 | |
| *** eernst has quit IRC | 02:25 | |
| *** eernst has joined #openstack-infra | 02:26 | |
| *** eernst has quit IRC | 02:31 | |
| *** eernst has joined #openstack-infra | 02:32 | |
| *** jamesmcarthur_ has quit IRC | 02:35 | |
| *** eernst has quit IRC | 02:37 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:38 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:40 | |
| *** bhavikdbavishi has quit IRC | 02:42 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:42 | |
| *** eernst has joined #openstack-infra | 02:43 | |
| *** eernst has quit IRC | 02:48 | |
| *** eernst has joined #openstack-infra | 02:50 | |
| *** rchurch has left #openstack-infra | 02:51 | |
| *** michael-beaver has quit IRC | 02:54 | |
| *** eernst has quit IRC | 02:54 | |
| *** eernst has joined #openstack-infra | 02:56 | |
| *** eernst has quit IRC | 03:01 | |
| *** ykarel has joined #openstack-infra | 03:23 | |
| *** psachin has joined #openstack-infra | 03:28 | |
| *** betherly has joined #openstack-infra | 03:29 | |
| *** betherly has quit IRC | 03:33 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] dib_cmd https://review.opendev.org/672196 | 03:33 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] dib_cmd https://review.opendev.org/672196 | 03:42 |
| *** hongbin has quit IRC | 03:45 | |
| *** betherly has joined #openstack-infra | 03:49 | |
| *** hongbin has joined #openstack-infra | 03:50 | |
| *** hongbin has quit IRC | 03:50 | |
| ianw | i feel like the job status for ^ is not showing up in http://zuul.openstack.org/status | 03:50 |
| *** betherly has quit IRC | 03:54 | |
| clarkb | ianw zuul and nodepool arein their own tenant now | 03:58 |
| clarkb | so you have to go to zuul.opendev.org and follow zuul tenant links | 03:58 |
| ianw | ahhh, got it thanks :) | 03:59 |
| ianw | yeah, i was a bit confused as it still appears under projects @ http://zuul.openstack.org/projects so wasn't sure if it was moved | 04:00 |
| *** udesale has joined #openstack-infra | 04:01 | |
| clarkb | I think we have it there to load the integration job config for glean and dib | 04:02 |
| *** kjackal has joined #openstack-infra | 04:05 | |
| *** rcernin has quit IRC | 04:13 | |
| *** rcernin has joined #openstack-infra | 04:14 | |
| *** rcernin has quit IRC | 04:20 | |
| *** betherly has joined #openstack-infra | 04:31 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] dib_cmd https://review.opendev.org/672196 | 04:34 |
| *** betherly has quit IRC | 04:36 | |
| *** gfidente has quit IRC | 04:43 | |
| *** pcaruana has joined #openstack-infra | 04:43 | |
| *** raukadah is now known as chandankumar | 04:44 | |
| *** betherly has joined #openstack-infra | 04:47 | |
| *** betherly has quit IRC | 04:51 | |
| *** ramishra has quit IRC | 04:54 | |
| *** ykarel is now known as ykarel|afk | 04:55 | |
| *** AJaeger is now known as AJaeger_ | 05:02 | |
| *** ykarel|afk has quit IRC | 05:02 | |
| *** ykarel|afk has joined #openstack-infra | 05:18 | |
| *** ykarel|afk is now known as ykarel | 05:18 | |
| *** bhavikdbavishi has quit IRC | 05:19 | |
| *** bhavikdbavishi has joined #openstack-infra | 05:25 | |
| *** ramishra has joined #openstack-infra | 05:29 | |
| AJaeger_ | I see "Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='opendev.org', port=443): Max retries exceeded with url: /openstack/requirements/raw/branch/master/upper-constraints.txt (Caused by ProtocolError('Connection aborted.', OSError(0, 'Error')))" | 05:30 |
| AJaeger_ | example http://logs.openstack.org/31/672131/1/check/openstacksdk-functional-devstack-senlin/ebf3a80/job-output.txt.gz#_2019-07-23_04_38_46_144415 | 05:30 |
| *** Lucas_Gray has joined #openstack-infra | 05:31 | |
| AJaeger_ | That just happened 1 hour ago - is that still to be expected? | 05:31 |
| *** armax has quit IRC | 05:34 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] dib_cmd https://review.opendev.org/672196 | 05:35 |
| *** notmyname has quit IRC | 05:41 | |
| *** notmyname has joined #openstack-infra | 05:42 | |
| *** dpawlik has joined #openstack-infra | 05:44 | |
| *** eernst has joined #openstack-infra | 05:46 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] dib_cmd https://review.opendev.org/672196 | 05:46 |
| *** eernst has quit IRC | 05:51 | |
| *** jamesmcarthur has joined #openstack-infra | 05:55 | |
| *** jamesmcarthur has quit IRC | 05:56 | |
| *** kjackal has quit IRC | 06:01 | |
| *** kjackal has joined #openstack-infra | 06:05 | |
| *** igordc has joined #openstack-infra | 06:08 | |
| *** Lucas_Gray has quit IRC | 06:09 | |
| *** jamesmcarthur has joined #openstack-infra | 06:09 | |
| *** Lucas_Gray has joined #openstack-infra | 06:10 | |
| *** jhesketh has quit IRC | 06:11 | |
| *** jhesketh has joined #openstack-infra | 06:11 | |
| *** jamesmcarthur has quit IRC | 06:13 | |
| *** igordc has quit IRC | 06:15 | |
| *** jamesmcarthur has joined #openstack-infra | 06:16 | |
| *** apetrich has quit IRC | 06:20 | |
| *** diablo_rojo has joined #openstack-infra | 06:27 | |
| *** jamesmcarthur_ has joined #openstack-infra | 06:28 | |
| *** jamesmca_ has joined #openstack-infra | 06:31 | |
| *** diablo_rojo has quit IRC | 06:32 | |
| *** jamesmcarthur has quit IRC | 06:32 | |
| *** jamesmcarthur_ has quit IRC | 06:35 | |
| *** jamesmca_ has quit IRC | 06:38 | |
| *** Lucas_Gray has quit IRC | 06:39 | |
| *** e0ne has joined #openstack-infra | 06:40 | |
| *** e0ne has quit IRC | 06:41 | |
| *** Lucas_Gray has joined #openstack-infra | 06:43 | |
| *** jamesmcarthur has joined #openstack-infra | 06:43 | |
| *** odicha has joined #openstack-infra | 06:44 | |
| *** piotrowskim has joined #openstack-infra | 06:47 | |
| *** dchen has quit IRC | 06:47 | |
| *** pgaxatte has joined #openstack-infra | 06:50 | |
| *** jamesmcarthur has quit IRC | 06:51 | |
| *** jamesmcarthur has joined #openstack-infra | 06:53 | |
| *** gfidente has joined #openstack-infra | 06:59 | |
| *** jtomasek has joined #openstack-infra | 07:04 | |
| *** yamamoto has quit IRC | 07:05 | |
| *** bhavikdbavishi has quit IRC | 07:05 | |
| *** yamamoto has joined #openstack-infra | 07:06 | |
| *** Goneri has joined #openstack-infra | 07:07 | |
| *** slaweq has joined #openstack-infra | 07:07 | |
| *** tesseract has joined #openstack-infra | 07:09 | |
| *** rpittau|afk is now known as rpittau | 07:09 | |
| *** iurygregory has joined #openstack-infra | 07:14 | |
| *** lucasagomes has joined #openstack-infra | 07:18 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Add a dib_cmd option for diskimages https://review.opendev.org/672196 | 07:18 |
| *** zbr_ has joined #openstack-infra | 07:20 | |
| *** joeguo_ has joined #openstack-infra | 07:20 | |
| *** irclogbot_2 has quit IRC | 07:20 | |
| *** kaisers has quit IRC | 07:20 | |
| *** openstackstatus has quit IRC | 07:20 | |
| *** kaisers has joined #openstack-infra | 07:21 | |
| *** irclogbot_3 has joined #openstack-infra | 07:21 | |
| *** dansmith has quit IRC | 07:23 | |
| *** zbr has quit IRC | 07:23 | |
| *** joeguo has quit IRC | 07:23 | |
| *** dansmith has joined #openstack-infra | 07:24 | |
| *** Anticimex has quit IRC | 07:24 | |
| *** ginopc has joined #openstack-infra | 07:24 | |
| *** jamesmcarthur has quit IRC | 07:24 | |
| *** beagles has quit IRC | 07:26 | |
| *** tosky has joined #openstack-infra | 07:28 | |
| *** Anticimex has joined #openstack-infra | 07:29 | |
| noonedeadpunk | I see strange thing while connecting to opendev.org (probably it's due t my VPN but 12 hours before everything was ok): http://paste.openstack.org/show/754747/ | 07:30 |
| noonedeadpunk | Does anyone have any ideas why this may happen? | 07:31 |
| gmann | AJaeger_: clarkb fungi ianw can any of you remove the stable/stein branch for patrole project - https://opendev.org/openstack/patrole/src/branch/stable/stein. | 07:32 |
| gmann | release patch to remove the branch is merged now: https://review.opendev.org/#/c/670942/ | 07:32 |
| gmann | it was created by mistake | 07:33 |
| *** ykarel is now known as ykarel|lunch | 07:50 | |
| *** apetrich has joined #openstack-infra | 07:57 | |
| *** dtantsur|afk is now known as dtantsur | 07:59 | |
| *** priteau has joined #openstack-infra | 07:59 | |
| noonedeadpunk | And we're eventually catching common things in CI http://logs.openstack.org/26/670126/1/gate/openstack-ansible-functional-centos-7/a2fda1c/job-output.txt.gz#_2019-07-23_07_55_36_288751 | 08:06 |
| *** Goneri has quit IRC | 08:09 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:10 | |
| *** Goneri has joined #openstack-infra | 08:11 | |
| *** betherly has joined #openstack-infra | 08:20 | |
| *** yamamoto has quit IRC | 08:20 | |
| *** pkopec has joined #openstack-infra | 08:23 | |
| *** ralonsoh has joined #openstack-infra | 08:26 | |
| *** panda has quit IRC | 08:38 | |
| *** panda has joined #openstack-infra | 08:38 | |
| noonedeadpunk | infra-root ^ | 08:40 |
| *** e0ne has joined #openstack-infra | 08:46 | |
| *** yamamoto has joined #openstack-infra | 08:50 | |
| *** ykarel|lunch is now known as ykarel | 08:57 | |
| *** priteau has quit IRC | 09:00 | |
| *** Lucas_Gray has quit IRC | 09:01 | |
| *** yamamoto has quit IRC | 09:02 | |
| *** priteau has joined #openstack-infra | 09:03 | |
| *** pgaxatte has quit IRC | 09:11 | |
| *** Goneri has quit IRC | 09:11 | |
| *** betherly has quit IRC | 09:11 | |
| *** Lucas_Gray has joined #openstack-infra | 09:12 | |
| *** pgaxatte has joined #openstack-infra | 09:12 | |
| *** Goneri has joined #openstack-infra | 09:16 | |
| *** Lucas_Gray has quit IRC | 09:18 | |
| *** Lucas_Gray has joined #openstack-infra | 09:19 | |
| *** psachin has quit IRC | 09:20 | |
| *** ociuhandu has joined #openstack-infra | 09:31 | |
| *** psachin has joined #openstack-infra | 09:35 | |
| *** apetrich has quit IRC | 09:36 | |
| *** pgaxatte has quit IRC | 09:41 | |
| *** pgaxatte has joined #openstack-infra | 09:43 | |
| *** betherly has joined #openstack-infra | 09:46 | |
| *** ociuhandu has quit IRC | 09:48 | |
| *** ociuhandu has joined #openstack-infra | 09:50 | |
| *** jaosorior has joined #openstack-infra | 09:52 | |
| *** bhavikdbavishi has quit IRC | 09:59 | |
| *** dpawlik has quit IRC | 10:02 | |
| *** dpawlik has joined #openstack-infra | 10:03 | |
| jrosser | we have a bunch of jobs failing due to errors fetching files from git http://logs.openstack.org/89/667789/3/gate/openstack-ansible-deploy-aio_metal-debian-stable/14b42ef/job-output.txt.gz#_2019-07-23_07_50_24_824613 | 10:10 |
| *** kopecmartin|off is now known as kopecmartin | 10:12 | |
| noonedeadpunk | that's what I've reported 2 hours ago:) And I have similar thing when I'm connected trough my VPN: http://paste.openstack.org/show/754747/ | 10:13 |
| *** traskat has quit IRC | 10:13 | |
| jrosser | noonedeadpunk: interesting, changing curl to curl -4 and it works | 10:15 |
| jrosser | so looks like the ipv6 gremlins are at work | 10:15 |
| *** lpetrut has joined #openstack-infra | 10:21 | |
| *** tdasilva has joined #openstack-infra | 10:21 | |
| *** yamamoto has joined #openstack-infra | 10:27 | |
| *** yamamoto has quit IRC | 10:28 | |
| *** yamamoto has joined #openstack-infra | 10:28 | |
| *** shachar has joined #openstack-infra | 10:29 | |
| *** snapiri has quit IRC | 10:32 | |
| *** ricolin_ has joined #openstack-infra | 10:34 | |
| *** ricolin has quit IRC | 10:37 | |
| *** yamamoto has quit IRC | 10:48 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Don't barf in dashboard on CORS violations for 404s https://review.opendev.org/672262 | 10:50 |
| *** apetrich has joined #openstack-infra | 10:53 | |
| *** Lucas_Gray has quit IRC | 10:54 | |
| *** yamamoto has joined #openstack-infra | 10:57 | |
| *** yamamoto has quit IRC | 11:02 | |
| *** yamamoto has joined #openstack-infra | 11:05 | |
| *** b3nt_pin has joined #openstack-infra | 11:36 | |
| *** ykarel is now known as ykarel|afk | 11:40 | |
| *** ginopc has quit IRC | 11:40 | |
| *** markvoelker has quit IRC | 11:58 | |
| *** rh-jelabarre has joined #openstack-infra | 12:00 | |
| *** eharney has joined #openstack-infra | 12:01 | |
| *** yamamoto has quit IRC | 12:03 | |
| *** udesale has quit IRC | 12:04 | |
| *** udesale has joined #openstack-infra | 12:04 | |
| *** ricolin_ is now known as ricolin | 12:05 | |
| *** Goneri has quit IRC | 12:06 | |
| *** yamamoto has joined #openstack-infra | 12:08 | |
| *** pfallenop has joined #openstack-infra | 12:12 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build gerrit images for 2.16 and 3.0 as well https://review.opendev.org/672273 | 12:15 |
| *** markvoelker has joined #openstack-infra | 12:16 | |
| *** Goneri has joined #openstack-infra | 12:18 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Trim some bazel flags https://review.opendev.org/672274 | 12:18 |
| *** pfallenop has quit IRC | 12:20 | |
| *** ccamacho has joined #openstack-infra | 12:22 | |
| *** goldyfruit has quit IRC | 12:26 | |
| *** ykarel|afk is now known as ykarel | 12:34 | |
| *** electrofelix has joined #openstack-infra | 12:36 | |
| *** mriedem has joined #openstack-infra | 12:38 | |
| *** joeguo_ has quit IRC | 12:42 | |
| *** siqbal has joined #openstack-infra | 12:48 | |
| *** pkopec has quit IRC | 12:57 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:59 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 13:02 | |
| *** sshnaidm has quit IRC | 13:03 | |
| *** bhavikdbavishi has quit IRC | 13:04 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 13:04 | |
| *** pkopec has joined #openstack-infra | 13:04 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Use cherrypy_cors to set cors headers https://review.opendev.org/672285 | 13:05 |
| *** apetrich has quit IRC | 13:07 | |
| *** pkopec has quit IRC | 13:08 | |
| *** pkopec has joined #openstack-infra | 13:08 | |
| *** jamesmcarthur has joined #openstack-infra | 13:08 | |
| *** pkopec has quit IRC | 13:09 | |
| *** sshnaidm has joined #openstack-infra | 13:12 | |
| *** jamesmcarthur has quit IRC | 13:18 | |
| fungi | AJaeger_: noonedeadpunk: jrosser: i have a feeling haproxy isn't taking the dead gitea01 backend out of the pools. i've manually disabled its backends for http and https now | 13:18 |
| fungi | #status log manually disabled http and https backends for missing gitea01 in haproxy | 13:19 |
| noonedeadpunk | fungi: worked for me | 13:19 |
| fungi | yeah, you have a 1/8 chance of being balanced to the dead backend (based on client address hash) | 13:19 |
| fungi | statusbot is apparently dead too. restarting it now | 13:20 |
| noonedeadpunk | I guess I had 100% for some reason, since today gitea never worked for me (from VPN) | 13:20 |
| noonedeadpunk | so my IP wasn't in lucky list:( | 13:21 |
| *** openstackstatus has joined #openstack-infra | 13:21 | |
| *** ChanServ sets mode: +v openstackstatus | 13:21 | |
| fungi | well, the backend is chosen based on a hash of your client address, so you'll be consistently sent to the same backend, yeah | 13:21 |
| fungi | #status log restarted statusbot after a 07:20z ctcp ping timeout | 13:22 |
| openstackstatus | fungi: finished logging | 13:22 |
| fungi | #status log manually disabled http and https backends for missing gitea01 in haproxy | 13:22 |
| openstackstatus | fungi: finished logging | 13:22 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Use cherrypy_cors to set cors headers https://review.opendev.org/672285 | 13:24 |
| *** pkopec has joined #openstack-infra | 13:26 | |
| AJaeger_ | thanks, fungi | 13:30 |
| ykarel | fungi, AJaeger_ is below issue related to ^^ discussion:- ERROR: Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='opendev.org', port=443): Max retries exceeded with url: /openstack/requirements/raw/branch/master/upper-constraints.txt (Caused by SSLError(SSLEOFError(8, u'EOF occurred in violation of protocol (_ssl.c:618)'),)) | 13:31 |
| ykarel | seeing in one of the job http://logs.openstack.org/97/672197/2/check/tripleo-ci-centos-7-containers-multinode/0dd1fe6/job-output.txt.gz#_2019-07-23_10_24_26_400171 | 13:31 |
| fungi | ykarel: probably, yes | 13:33 |
| fungi | as to why that job doesn't have zuul provide the source code for it, i guess that's a bigger debate | 13:33 |
| ykarel | fungi, ack, so it should not be seen again from now, right? | 13:33 |
| fungi | ykarel: as far as i know, yes. i'm not sure why haproxy didn't mark those backends down when they became unreachable, but am still catching up for the morning before i can dig deeper into that | 13:34 |
| *** goldyfruit has joined #openstack-infra | 13:35 | |
| ykarel | fungi, ack Thanks for the info, i will keep watching, if i see again will share with u | 13:35 |
| fungi | appreciated! | 13:35 |
| *** bhavikdbavishi has quit IRC | 13:35 | |
| *** mriedem has quit IRC | 13:38 | |
| *** ykarel is now known as ykarel|away | 13:39 | |
| *** jamesmcarthur has joined #openstack-infra | 13:39 | |
| *** tosky_ has joined #openstack-infra | 13:40 | |
| *** tosky has quit IRC | 13:42 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/project-config master: Rename "tripleo-ci-centos-7-scenario007-standalone" in Neutron https://review.opendev.org/672290 | 13:43 |
| *** ykarel|away has quit IRC | 13:44 | |
| *** sreejithp has joined #openstack-infra | 13:45 | |
| petevg | Good morning/afternoon/evening! Does anybody have any handy examples of adding artifacts to test output after a test failure? I want to gather and tar up some logs for inspection, and I'm not certain what the best practice for that sort of thing is in zuul ... | 13:47 |
| *** AJaeger_ is now known as AJaeger | 13:50 | |
| *** aaronsheffield has joined #openstack-infra | 13:51 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: WIP Use cherrypy_cors to set cors headers https://review.opendev.org/672285 | 13:52 |
| *** yamamoto has quit IRC | 13:53 | |
| *** yamamoto has joined #openstack-infra | 13:54 | |
| *** iurygregory has quit IRC | 13:55 | |
| *** iurygregory has joined #openstack-infra | 13:55 | |
| *** yamamoto has quit IRC | 13:58 | |
| *** jamesmcarthur has quit IRC | 13:58 | |
| *** rpittau is now known as rpittau|afk | 14:03 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/project-config master: Rename "tripleo-ci-centos-7-scenario007-standalone" in Neutron https://review.opendev.org/672290 | 14:07 |
| *** michael-beaver has joined #openstack-infra | 14:07 | |
| *** tosky_ is now known as tosky | 14:08 | |
| *** ykarel|away has joined #openstack-infra | 14:11 | |
| *** jcoufal has joined #openstack-infra | 14:16 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build gerrit images for 2.16 and 3.0 as well https://review.opendev.org/672273 | 14:21 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Trim some bazel flags https://review.opendev.org/672274 | 14:21 |
| mordred | corvus: there's a change to build 2.16 and 3.0 too - maybe we'll get lucky and they'll build with no issues | 14:22 |
| *** gyee has joined #openstack-infra | 14:23 | |
| *** apetrich has joined #openstack-infra | 14:26 | |
| fungi | petevg: usually the simplest thing to do is if some action you expected to succeed fails, copy the additional logs/artifacts for it into the place where zuul expects to find them so it will slurp them up at the end of the job | 14:29 |
| *** jeremy_houser has joined #openstack-infra | 14:31 | |
| petevg | fungi: aha. So by default, if I just copy things into work/logs, it'll make them available to me? | 14:34 |
| *** ykarel|away has quit IRC | 14:34 | |
| fungi | petevg: i'll check the log collection role, but i believe so yes | 14:34 |
| petevg | fungi: cool. Thx. | 14:35 |
| *** mriedem has joined #openstack-infra | 14:36 | |
| *** ykarel|away has joined #openstack-infra | 14:37 | |
| *** yamamoto has joined #openstack-infra | 14:37 | |
| *** kjackal has quit IRC | 14:38 | |
| *** kjackal has joined #openstack-infra | 14:40 | |
| *** dosaboy has joined #openstack-infra | 14:41 | |
| fungi | petevg: so it's a combination of two roles usually... this one collects files from the job nodes and pulls them back to the executor workspace: https://zuul-ci.org/docs/zuul-jobs/log-roles.html#role-fetch-output | 14:41 |
| fungi | petevg: and then this one copies those to a site for publication: https://zuul-ci.org/docs/zuul-jobs/log-roles.html#role-upload-logs | 14:41 |
| *** yamamoto has quit IRC | 14:43 | |
| fungi | and yeah, the collect log output task in the fetch-output role looks in {{ zuul_output_dir }}/logs/ generally, unless overridden | 14:43 |
| fungi | whatever's in there at the time the post playbook runs should get archived | 14:44 |
| petevg | fungi: got it. I'll give that a try. Thank you again! | 14:45 |
| fungi | any time! | 14:45 |
| *** iurygregory has quit IRC | 14:45 | |
| *** iurygregory has joined #openstack-infra | 14:47 | |
| *** gfidente has quit IRC | 14:49 | |
| fungi | okay, so i've figured out why gitea01 didn't get taken out of the haproxy pools when it became unreachable... none of the backends have any sort of checking enabled (status column is "no check" for all of them) | 14:55 |
| fungi | at least i think that's the reason | 14:57 |
| fungi | still trying to wrap my head around haproxy's socket forwarding model. we do have defaults set to redispatch with a variety of conditional timeouts and retrying | 14:57 |
| fungi | is the expectation that forwarding failures are used in lieu of service checks? | 14:58 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: WIP Use cherrypy_cors to set cors headers https://review.opendev.org/672285 | 15:01 |
| *** armax has joined #openstack-infra | 15:05 | |
| *** kjackal has quit IRC | 15:06 | |
| *** lseki has joined #openstack-infra | 15:06 | |
| *** kjackal has joined #openstack-infra | 15:06 | |
| *** odicha has quit IRC | 15:08 | |
| fungi | so far i'm not finding any mention in the haproxy docs of an implicit checking feature | 15:09 |
| *** pfallenop has joined #openstack-infra | 15:09 | |
| *** bhavikdbavishi has joined #openstack-infra | 15:10 | |
| clarkb | fungi: I think this configuration was lost in the conversion to ansible + docker | 15:12 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: WIP Do public cors without cherrypy_cors https://review.opendev.org/672313 | 15:12 |
| clarkb | fungi: you have to explicitly configure checks | 15:12 |
| fungi | thanks, that's what i was getting from the haproxy docs as well | 15:12 |
| fungi | presumably we can get by with just a tcp socket check on the ports | 15:13 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Actually check backends are alive in haproxy https://review.opendev.org/672314 | 15:13 |
| clarkb | fungi: something like ^ | 15:14 |
| *** eernst has joined #openstack-infra | 15:14 | |
| mordred | clarkb: ++ | 15:15 |
| clarkb | also a "listen" block is a combined frontend and backend block | 15:16 |
| *** pfallenop has quit IRC | 15:16 | |
| clarkb | so I'm 99% sure the backend directives are valid in that listen block | 15:16 |
| *** trident has quit IRC | 15:18 | |
| *** trident has joined #openstack-infra | 15:20 | |
| fungi | the examples i found looked exactly like that | 15:21 |
| *** ccamacho has quit IRC | 15:21 | |
| *** rh-jelabarre has quit IRC | 15:23 | |
| *** rh-jelabarre has joined #openstack-infra | 15:25 | |
| *** dpawlik has quit IRC | 15:30 | |
| *** pgaxatte has quit IRC | 15:30 | |
| clarkb | fungi: is there a change up to add new gitea01 back into the inventory yet? | 15:33 |
| clarkb | mordred: fungi re haproxy config changes the way you gracefully stop haproxy is to send some signal to it that says exit when all connections are closed, then you start a new daemon that will listen for new connections. I'm not really sure how to coordinate that with docker-compose, do you all have any idea | 15:34 |
| clarkb | `docker kill -s HUP my-running-haproxy` is what the image docs say to do | 15:36 |
| clarkb | which addresses half of the problem (when only config updates and not the image) | 15:36 |
| fungi | clarkb: hah, i was just researching how to do that. i guess https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/haproxy/tasks/main.yaml#L18-L21 is where we need to trigger a configuration reload? | 15:36 |
| clarkb | fungi: that is one place where we need to the other is when we do a docker-compose up and the haproxy image has updated | 15:37 |
| clarkb | fungi: I suppose we can start with the config updated case first and image updates are likely to be less frequent | 15:37 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add additional gerrit plugin repos https://review.opendev.org/672320 | 15:38 |
| clarkb | fungi: the chagnes we've made in the last day or so would all be the first case anyway so that is likely good enough | 15:39 |
| *** e0ne has quit IRC | 15:39 | |
| clarkb | fungi: are you writing that change (adding a handler to the role to trigger that command?) | 15:39 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build gerrit images for 2.16 and 3.0 as well https://review.opendev.org/672273 | 15:40 |
| fungi | i'm learning ansible enough to find the pidfile and use that to template out the kill command | 15:40 |
| *** david-lyle is now known as dklyle | 15:40 | |
| clarkb | fungi: why not use docker or docker-compose for that? | 15:40 |
| fungi | since i guess we don't have normal systemd service management hooked up to containerized haproxy | 15:40 |
| clarkb | sorry let me link you to the docs I've got | 15:41 |
| fungi | oh, because i didn't know docker-compose could do it ;) | 15:41 |
| clarkb | fungi: https://hub.docker.com/_/haproxy/ | 15:41 |
| mordred | clarkb: I'm not sure what the answer is | 15:41 |
| fungi | (or really the first thing about docker compose, to be honest) | 15:41 |
| mordred | clarkb: oh - cool - that seems neat | 15:42 |
| clarkb | fungi: the command you want is likely `docker-compose -f /etc/haproxy-docker/docker-compose.yaml kill -s HUP haproxy` | 15:42 |
| fungi | neat-o | 15:42 |
| mordred | clarkb, fungi: if you have a sec, 672320 is easy and needed for the gerrit image work | 15:42 |
| fungi | worst case, /var/haproxy/run/haproxy.pid does seem to contain the pid for the haproxy daemon | 15:43 |
| clarkb | fungi: their docs document the docker command equivalent but beacuse docker-compose is managing our containers we'd have to lookup whatever it called the haproxy container. Instead if we have docker compose run the command it knows how to map the logical thing we called 'haproxy' in our config to the running container for us | 15:43 |
| fungi | oh, i take that back | 15:43 |
| fungi | ot | 15:43 |
| fungi | it's the pid of the haproxy daemon within the process namespace | 15:43 |
| fungi | (so... 1) | 15:43 |
| clarkb | ya that is why the docker tools exists | 15:44 |
| clarkb | fungi: you can also test that command by running it on opendev.org as root. Shoudl result in a new pid for the haproxy process | 15:45 |
| fungi | yep, that command works (as root, not as an unprivileged user) | 15:45 |
| portdirect | hey - im wondering if we can 'choose' which pool of nodes some ci jobs run on | 15:45 |
| clarkb | if it somehow causes haproxy to stop functioning then docker-compose -f /that/same/file restart should get it back up and running | 15:45 |
| clarkb | portdirect: no | 15:45 |
| fungi | and caused it to finally remove the gitea01 pool entries we commented out of the config | 15:46 |
| clarkb | portdirect: there is a bit more to it than that (like arm64 resources only come from one location currently so thats an implicit choosing) but in general we try to have a generic pool of resources beacuse resources come and go over time | 15:47 |
| clarkb | portdirect: is there something more specific you are trying to achieve? | 15:47 |
| clarkb | eg what is the goal with that? | 15:47 |
| portdirect | yeah - we'd like to do some dpdk checks | 15:48 |
| portdirect | i'll paste a snippet that may help/provide context | 15:48 |
| portdirect | https://www.irccloud.com/pastebin/J4JXDe1k/ | 15:48 |
| fungi | we have waaaay more than "two kinds" of virtual machines managed by nodepool | 15:49 |
| portdirect | yeah ;) | 15:49 |
| clarkb | the two that seem to be detected there are rackspace (no nested virt + two interfaces) and not rackspace (always one interface sometimes nested virt) | 15:49 |
| fungi | portdirect: https://docs.openstack.org/infra/manual/testing.html tries to cover the variances you can expect | 15:49 |
| clarkb | portdirect: what are the requirements to test dpdk ? | 15:49 |
| portdirect | interface with pci addr, and nested virt | 15:50 |
| clarkb | officially we don't support nested virt because it has never worked reliably | 15:50 |
| portdirect | thats fair | 15:50 |
| clarkb | even in clouds where it works today we've found that the next round of kernel updates to our images tend to break things | 15:50 |
| clarkb | then we have to wait for the cloud to update all their hypervisor kernels | 15:50 |
| fungi | especially in environments where we don't control the underlying hardware, host kernel or hypervisor | 15:51 |
| fungi | (which is basically all of them) | 15:51 |
| portdirect | though here is not for real nested virt, its just vmx we need i think | 15:51 |
| clarkb | as for interface with pci addr does that require directed io or whatever it is called today (pci passthrough?) | 15:51 |
| clarkb | I don't think we've got that in any clouds | 15:51 |
| fungi | and we don't even have guarantees that all server instances booted in a particular provider/region will use the same sort of hypervisor hosts with consistent features | 15:52 |
| *** trident has quit IRC | 15:52 | |
| *** jaosorior has quit IRC | 15:53 | |
| portdirect | ok - thanks clarkb and fungi | 15:53 |
| clarkb | At one point we had a reasonably good setup for pushing the boundaries a bit on this with logan- (limestone) and mnaser (vexxhost) but I think we are still experiencing networking problems on limestone so are back down to one cloud region for that again | 15:53 |
| clarkb | its possible fn may be able to help support some of that. | 15:54 |
| clarkb | I'd be happy for people to experiment more but I think the goal is likely to be "improve reliability of nested virt" and not "test dpdk" at least initially | 15:54 |
| *** trident has joined #openstack-infra | 15:55 | |
| mnaser | indeed. i had mentioned at the time that i was more than happy to work with whoever would volunteer to figure out what needs to be done on the host level | 15:56 |
| mnaser | (i.e. run this kernel or whatever) | 15:56 |
| mnaser | but i cant really put time into digging that out myself unfortunately | 15:56 |
| clarkb | portdirect: if intereting in helping to improve that base layer the people I know that have owrked on it in the past are mnaser logan- johnsom rm_work and sean mooney | 15:57 |
| *** kjackal has quit IRC | 15:58 | |
| clarkb | I think the next step given where we are at now is to get a second cloud back up again that can support/assist direct debugging: either address networking issues in limestone and reenable there or see if donnyd thinks fortnebula can support it | 15:58 |
| johnsom | Yeah, let me know if I can help with getting that enabled. | 15:59 |
| clarkb | Then add a flavor that allows us to run testing directed at exercising nested virt (boot cirros, ubuntu, centos smoke test? maybe more complicated then that) and that will allow us to track that directly | 15:59 |
| portdirect | I'll reach out to cheng1 and see if we can assit there | 15:59 |
| portdirect | thanks so much :) | 15:59 |
| johnsom | The last issue that caused our project to turn it off was a nodepool instance kernel bug that came and went with the kernel releases. Probably resolved now, but we still haven't turned it back on. | 16:00 |
| clarkb | johnsom: it seems like the way those end up working out is guest kernel upadtes and breaks nested virt, then cloud updates their hypervisor kernel and it works again | 16:00 |
| clarkb | which is why the involvement from the cloud side has been so valuable | 16:00 |
| johnsom | We narrowed it to not be related to the guest kernel or the host. It was the nodepool kernel. (working with limestone team) | 16:01 |
| *** e0ne has joined #openstack-infra | 16:01 | |
| johnsom | I should check if my kernel bug report is still open for that or not. | 16:02 |
| *** iurygregory has quit IRC | 16:02 | |
| johnsom | https://bugzilla.kernel.org/show_bug.cgi?id=192521 | 16:03 |
| openstack | bugzilla.kernel.org bug 192521 in kvm "KVM: entry failed, hardware error 0x0" [High,New] - Assigned to virtualization_kvm | 16:03 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Reload haproxy configuration when config changes https://review.opendev.org/672323 | 16:03 |
| johnsom | Still open, but who knows.... | 16:03 |
| clarkb | johnsom: yes in my context nodepool is the gueast | 16:04 |
| fungi | clarkb: mordred: i feel like i've sort of cargo-culted 672323... don't really understand enough about ansible still to be confident that i'm understanding how file management works | 16:04 |
| clarkb | you have cloud <- nodepool image <- nested image | 16:04 |
| fungi | (i'm assuming the template task only returns success when the file content changes) | 16:04 |
| clarkb | and we know that updating the middle kernel does break things and usually updating the first kernel fixes it | 16:04 |
| clarkb | fungi: look up ansible handlers that is the "correct" way to do the association between tasks iirc | 16:05 |
| clarkb | fungi: I believe config_update will always be successful whether it writes bytes or not | 16:05 |
| fungi | ahh | 16:05 |
| fungi | still mentally mapping puppet concepts onto ansible. sorry! | 16:05 |
| fungi | will read more | 16:05 |
| mordred | fungi: you'll like handlers - they're nice and clean | 16:06 |
| clarkb | fungi: opendev/system-config/playbooks/roles/nameserver/handlers/main.yaml is a good example likely | 16:06 |
| clarkb | mordred: when they work | 16:06 |
| *** mattw4 has joined #openstack-infra | 16:06 | |
| fungi | heh | 16:06 |
| clarkb | mordred: when they don't work they cause ansible to exit 0 without running any subsequent tasks and you wonder why | 16:07 |
| mordred | clarkb: of course :) | 16:07 |
| * clarkb is a bit grumpy about how unreliable ansible has been with handlers | 16:07 | |
| mordred | clarkb: remember when we tried salt and it returned 0 on every invocation regardless of success or failure? | 16:07 |
| clarkb | ya and puppet returns 2 on success | 16:07 |
| mordred | ya | 16:07 |
| *** tesseract has quit IRC | 16:07 | |
| donnyd | I should be able to help with that | 16:08 |
| clarkb | johnsom: also we may need to set up some consistent terminology if we start to dig into this more. Nodepool doesn't run on these VMs nor does it have a special kernel | 16:09 |
| donnyd | My gear should be able to do dpdk, and I'm happy to enable nested virt | 16:09 |
| clarkb | so calling it the "nodepool kernel" implies nodepool is at fault when really it is normal upstream kernels being used as regular old VMs | 16:09 |
| *** eernst has quit IRC | 16:09 | |
| johnsom | Agreed, consistent terms would help. Yep, it's distro kernels for sure. | 16:10 |
| *** lucasagomes has quit IRC | 16:10 | |
| mordred | clarkb, johnsom: terminology is always the hardest part | 16:10 |
| *** pfallenop has joined #openstack-infra | 16:10 | |
| johnsom | Even the "levels" terminology is troubled with obi-wan errors. lol | 16:11 |
| clarkb | Good news is that the linux kernel in 4.19 (I think) enabled nested virt by default on intel cpus | 16:11 |
| clarkb | which means that in a year or two maybe this will all just work in the wild | 16:11 |
| johnsom | Yeah, it's been on by default for quiet some time. | 16:12 |
| clarkb | it has been on for amd for forever but kashyap mentioned that was likely an oversight | 16:12 |
| *** eernst_ has joined #openstack-infra | 16:12 | |
| johnsom | We had a good few years run with this stuff turned on without any issues. It was just this bug that stopped us. | 16:12 |
| clarkb | johnsom: on the specific hardware we have and so on | 16:13 |
| clarkb | there are a lot of variables at play and the kernel itself is looking at it more globally | 16:13 |
| clarkb | (so their stamp of approval implies that it is likely way more stable than we've seen it previously) | 16:13 |
| clarkb | oh also kashyap is another person that is likely willing to help if people start pushing on this more | 16:14 |
| logan- | o/ | 16:14 |
| logan- | johnsom: yep typically it breaks in limestone when guest kernels update but the host kernel does not. i've found the host kernel needs to be updated in lock step with nodepool for it to keep working. | 16:15 |
| johnsom | Joy | 16:15 |
| donnyd | Be back later | 16:16 |
| *** eernst_ has quit IRC | 16:18 | |
| *** lpetrut has quit IRC | 16:19 | |
| *** eernst has joined #openstack-infra | 16:19 | |
| openstackgerrit | Merged opendev/system-config master: Actually check backends are alive in haproxy https://review.opendev.org/672314 | 16:21 |
| clarkb | logan-: fwiw I believe the network problems came back as soon as we put workload on limestone. iirc fungi pulled it back out again | 16:23 |
| clarkb | logan-: that probably does imply it is somethign to do with our network traffic that triggers that card problems | 16:23 |
| clarkb | (not sure if you were caught up on that) | 16:23 |
| fungi | it didn't come back right away | 16:23 |
| *** eernst has quit IRC | 16:23 | |
| fungi | or at least not that i saw | 16:23 |
| fungi | though this last time it was harder to identify since it didn't impact the part of the network between cacti and the mirror instance | 16:24 |
| logan- | it looked like a different problem. multinode job having problems SSHing between hosts, so it would not have been going thru the neutron gateways where we were seeing problems previously. | 16:24 |
| clarkb | ah | 16:25 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: WIP Use cherrypy_cors to set cors headers https://review.opendev.org/672285 | 16:33 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: WIP Do public cors without cherrypy_cors https://review.opendev.org/672313 | 16:33 |
| logan- | the issue with the multinode job failure might have been a one off. we have continued running our local nodepool jobs on that cloud and they have not been impacted. we previously saw our jobs affected when the network node issues were occurring. | 16:34 |
| *** ricolin has quit IRC | 16:34 | |
| fungi | clarkb: i'm still not sure how to go about making it so that the template task only notifies the handler if it changed file content. it looks like you can do that with normal tasks instead using register and the .changed attribute though in examples i'm seeing | 16:35 |
| fungi | also i suppose i need to have it no-op if the service isn't running (maybe only when the pidfile exists?) | 16:36 |
| *** jamesdenton has quit IRC | 16:36 | |
| *** jamesdenton has joined #openstack-infra | 16:36 | |
| *** dtantsur is now known as dtantsur|afk | 16:38 | |
| clarkb | fungi: I think you just do a notify like the nameserver handler does | 16:42 |
| clarkb | then ansible knows to notify only if the file was updated | 16:42 |
| clarkb | (that was my example above) | 16:43 |
| *** yamamoto has joined #openstack-infra | 16:43 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add generate-zuul-manifest role https://review.opendev.org/671874 | 16:44 |
| *** Goneri has quit IRC | 16:44 | |
| clarkb | fungi: I think notify only fires if a task has the changed attribute set to true | 16:45 |
| *** yamamoto has quit IRC | 16:48 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Reload haproxy configuration when config changes https://review.opendev.org/672323 | 16:48 |
| fungi | clarkb: ahh, in that case ^ | 16:48 |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Reduce yum-minimal based OS install size footprint https://review.opendev.org/672329 | 16:48 |
| fungi | it was what i had already cribbed together trying to reconcile documentation against the other examples | 16:49 |
| *** yamamoto has joined #openstack-infra | 16:49 | |
| fungi | i just wasn't sure how to make certain it only did it on content changes | 16:49 |
| *** betherly has quit IRC | 16:49 | |
| clarkb | fungi: +2 | 16:50 |
| *** armax has quit IRC | 16:50 | |
| clarkb | fungi: might be worth manually running that command just to confirm that the container does the right thing | 16:51 |
| fungi | i already did earlier | 16:51 |
| fungi | and it properly removed gitea01 from the pools | 16:51 |
| clarkb | huh the processes are still from July 17 though | 16:51 |
| clarkb | maybe the way it does restarts is different than I thought | 16:51 |
| fungi | it doesn't restart anything, just tells the daemon to reread its config | 16:52 |
| mordred | yeah | 16:52 |
| clarkb | ok I distinctly remember reading a thing that said it doesn't reload the config but instead stops accepting new connections and you have to start a new process with the nwe config | 16:53 |
| clarkb | but maybe that is an alternative method which ubuntu packaging employs or something | 16:53 |
| clarkb | ya it definitely should make new processes | 16:54 |
| clarkb | based on http://www.haproxy.org/download/1.7/doc/management.txt which the image docs link to | 16:55 |
| openstackgerrit | Merged openstack/project-config master: Add additional gerrit plugin repos https://review.opendev.org/672320 | 16:55 |
| fungi | clarkb: i think you're referring to what happens when you send sigusr1? | 16:56 |
| clarkb | ya ok we have three haproxy processes | 16:56 |
| clarkb | two from the 17th and one from an hour ago | 16:56 |
| mordred | do the two from the 17th ever go away? | 16:56 |
| clarkb | mordred: I kind of expect the one with the -sf 6 to have been replaced by the new one that is -sf 6 | 16:57 |
| clarkb | but maybe not | 16:57 |
| fungi | ahh, so it does start a new daemon when rereading its config? that's strange | 16:57 |
| clarkb | fungi: yes | 16:57 |
| clarkb | that is how you get the new config it is a new process | 16:57 |
| clarkb | can we run the command again and see if we get a fourth process? | 16:57 |
| *** siqbal90 has joined #openstack-infra | 16:57 | |
| fungi | sure, just a sec | 16:57 |
| mordred | maybe it just tells the live one to stop accepting connections, and starts a new one that accepts - and getting rid of the old stale one is an exercise for the listener? | 16:57 |
| fungi | well, documentation suggested you should be able to gracefully stop the old processes when doing socket takeover | 16:58 |
| fungi | but i assumed that was only for no-downtime restarts, not also for config changes | 16:58 |
| mordred | clarkb, fungi: since you're both enjoying docker at the moment, https://review.opendev.org/#/c/671457 is also ready for your enjoyment | 16:58 |
| clarkb | fungi: they are the asme thing to haproxy | 16:58 |
| *** siqbal has quit IRC | 16:58 | |
| fungi | indeed, i find that a strange design choice on their part | 16:58 |
| *** dancek has quit IRC | 16:59 | |
| mordred | in particular, https://review.opendev.org/#/c/671457/13/docker/gerrit/2.13/Dockerfile "should" result in a container image that resembles what puppet would do on a server - minus the data and config files ... that's likely the most 'interesting' bit to look at | 17:00 |
| *** siqbal90 has quit IRC | 17:02 | |
| fungi | clarkb: okay, ran it as close as i can to how the handler will (cwd into /etc/haproxy-docker/ and not specifying a compose file) | 17:02 |
| clarkb | mordred: fungi ok that leaked another process | 17:02 |
| fungi | and it does seem to have started yet another haproxy process, yes | 17:02 |
| clarkb | I guess we have to check if eventually those older processes go away. It is possible those older processes are still handling clients | 17:02 |
| clarkb | (and so have not exited yet) | 17:02 |
| clarkb | haproxy[25482]: proxy balance_git_https has no server available! | 17:03 |
| clarkb | that seemed to have picked up the check? | 17:03 |
| fungi | also i find it interesting that 26683 is running as root but the others are running as uid 1000 | 17:03 |
| *** roman_g has quit IRC | 17:03 | |
| *** udesale has quit IRC | 17:03 | |
| clarkb | heres hoping it discovered quickly that tcp works | 17:03 |
| johnsom | Any idea why I can't open this page on opendev.org? https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/package-installs/post-install.d | 17:03 |
| clarkb | johnsom: see the paste a couple lines above yours | 17:04 |
| fungi | Connection refused at initial connection step of tcp-check | 17:04 |
| fungi | i'll undo the service checks | 17:04 |
| fungi | manually | 17:04 |
| clarkb | fungi: ok | 17:04 |
| fungi | okay, the backends are all back to no check now | 17:05 |
| *** roman_g has joined #openstack-infra | 17:05 | |
| fungi | fungi@gitea-lb01:/etc/haproxy-docker$ telnet 38.108.68.122 3080 | 17:05 |
| fungi | Trying 38.108.68.122... | 17:05 |
| fungi | telnet: Unable to connect to remote host: Connection refused | 17:05 |
| clarkb | is it actually the backends that are sad? | 17:06 |
| clarkb | ya ok | 17:06 |
| clarkb | gitea web restarted ~3 minutes ago | 17:06 |
| fungi | now i can reach them from the lb | 17:07 |
| fungi | okay, that's... strange timing? | 17:07 |
| clarkb | fungi: so the checks shoudl be fine | 17:07 |
| fungi | also i guess we don't have rolling restarts set up for gitea yet | 17:07 |
| clarkb | fungi: we don't but they should only restart if the images update | 17:07 |
| *** trident has quit IRC | 17:07 | |
| fungi | so did we just get new gitea images? | 17:08 |
| *** chandankumar is now known as raukadah | 17:08 | |
| clarkb | mariadb updated | 17:08 |
| clarkb | according to sudo docker image ls | 17:08 |
| fungi | neat | 17:09 |
| clarkb | fungi: we should put the checks back | 17:10 |
| *** trident has joined #openstack-infra | 17:10 | |
| clarkb | (if they have been successfully removed) | 17:10 |
| fungi | done | 17:10 |
| clarkb | and our older haproxy process from about an hour ago has gone away | 17:10 |
| fungi | Layer4 check passed | 17:10 |
| clarkb | so that may just be delay waiting for connections to die | 17:10 |
| fungi | looking okay so far | 17:11 |
| *** yamamoto has quit IRC | 17:11 | |
| *** priteau has quit IRC | 17:11 | |
| clarkb | ya its working for me now | 17:11 |
| clarkb | that was highly coincidental | 17:11 |
| fungi | confusingly so, yes | 17:11 |
| clarkb | the worst kind of wtf did it stop working :) | 17:11 |
| clarkb | mordred: I think https://review.opendev.org/#/c/672323/2 is good to go | 17:12 |
| clarkb | as the older process did go away | 17:12 |
| johnsom | If you use the haproxy reload, it will spawn a new process for new connections and keep the old one around to finish out any active connections. Once they are all closed the old process will exit. | 17:12 |
| clarkb | johnsom: yup | 17:12 |
| clarkb | johnsom: we were just confirming that the docker image supervisor process actually ensures that happens | 17:12 |
| *** yamamoto has joined #openstack-infra | 17:13 | |
| *** ociuhandu has quit IRC | 17:13 | |
| johnsom | FYI, if you are using the HAProxy docker image, 2.0.3 and 1.9.9 haproxy is out today with a CVE fix. | 17:16 |
| *** lseki has quit IRC | 17:16 | |
| mordred | clarkb: I hopped onto a call right as things were unhappy - tldr was that it was just bad timing? | 17:16 |
| clarkb | johnsom: doesn't appear to be on dockerhub yet | 17:16 |
| fungi | i'm going to go get lunch, and then once i'm back i'll finish getting the gitea01 replacement into the mix | 17:16 |
| fungi | (managed to get the replacement server launched last night) | 17:17 |
| johnsom | Here is the announce e-mail for 2.0.3: https://www.mail-archive.com/haproxy@formilux.org/msg34586.html | 17:17 |
| clarkb | mordred: yes mariadb image updated which restarted all of the giteas and they need a few minutes to start up. At the same time fungi HUP'd haproxy to pick up the health checks nad they reported no tcp connections | 17:17 |
| clarkb | mordred: so was coincidence aftera couple minutes gitea was back and checks worked fine | 17:17 |
| clarkb | mordred: and we should be safe to approve the change to add the graceful restart on config updates | 17:17 |
| fungi | clarkb: also see scrollback from last night about the two available 40gb volumes in sjc1 control plane tenant and see if the timeline there makes sense for how we wound up with them. if you concur i'll delete them when i get back | 17:18 |
| *** yamamoto has quit IRC | 17:18 | |
| mordred | clarkb: awesome. out of curiosity - isn't an image update like that supposed to do one backend at a time? | 17:18 |
| clarkb | mordred: no our ansible does not serialize them | 17:18 |
| clarkb | (that would be the easy fix but would slowdown ansible runtime) | 17:18 |
| mordred | ah - maybe we should serialize them, now that we have haproxy doing health checks | 17:18 |
| fungi | but sounds like a great enhancement | 17:18 |
| fungi | yes | 17:18 |
| mordred | and have it at least wait on the port being up | 17:18 |
| clarkb | johnsom: I don't think we are affected we only forward tcp so there is no cookie parsing | 17:18 |
| corvus | ++ can't think of a reason not to | 17:18 |
| fungi | bbiaw | 17:19 |
| mordred | although - while we're talking about it - am I remembering right that gitea isn't fully up as soon as its port is up? | 17:19 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Trim some bazel flags https://review.opendev.org/672274 | 17:20 |
| mordred | corvus: this is happening in remote_puppet_git right? it's the docker-compose pull in the gitea role that also causes the update? | 17:22 |
| corvus | yeah | 17:22 |
| openstackgerrit | Merged zuul/nodepool master: static: add host-key-checking toggle https://review.opendev.org/653679 | 17:23 |
| mordred | hrm. well - that will make initial spinup for integration testing annoying | 17:23 |
| mordred | (serializing the gitea role) | 17:23 |
| corvus | we only have one gitea during testing though | 17:23 |
| mordred | oh - duh | 17:23 |
| clarkb | mordred: corvus its also done separately from the create projects play | 17:23 |
| clarkb | so I think we should be able to do it fairly isolated | 17:23 |
| clarkb | mordred: are you working on that? | 17:24 |
| corvus | even better | 17:24 |
| mordred | cool - then I'm less worried about it - because we also don't actually expect to run the playbook from scratch against 8 giteas too | 17:24 |
| mordred | clarkb: yes | 17:24 |
| corvus | agreed, that time is hopefully past | 17:24 |
| *** ramishra has quit IRC | 17:24 | |
| *** _erlon_ has joined #openstack-infra | 17:25 | |
| clarkb | mordred: k I'll stop looking too hard at it then and will await change to review | 17:25 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Serialize the gitea role https://review.opendev.org/672335 | 17:26 |
| mordred | clarkb: I believe it's easy like that | 17:26 |
| clarkb | mordred: should we add a wait until gitea accepts connection on 443 task too? | 17:27 |
| mordred | clarkb: well, there's already a "make sure root user exists" in the role | 17:27 |
| clarkb | aha | 17:27 |
| clarkb | ok | 17:27 |
| mordred | clarkb: so I think we're covered | 17:27 |
| clarkb | +2 | 17:28 |
| mordred | \o/ | 17:28 |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Reduce yum-minimal based OS install size footprint https://review.opendev.org/672329 | 17:30 |
| clarkb | I have rechecked https://review.opendev.org/#/c/672323/2 as it failed on vcsrepo failing | 17:32 |
| clarkb | I think due to the short gitea outage | 17:32 |
| *** armax has joined #openstack-infra | 17:32 | |
| clarkb | infra-root https://review.opendev.org/#/c/672335/1 is the fix for that outage if we can get a second reviewer on it | 17:32 |
| *** e0ne has quit IRC | 17:33 | |
| clarkb | once we haev those two changes the only outstanding issue is how to gracefully restart when haproxy image updates I think | 17:33 |
| clarkb | there is a good chance that will happen soon too given the cve | 17:35 |
| *** ykarel|away has quit IRC | 17:35 | |
| *** ralonsoh has quit IRC | 17:36 | |
| *** dancek has joined #openstack-infra | 17:38 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add generate-zuul-manifest role https://review.opendev.org/671874 | 17:39 |
| *** panda is now known as panda|off | 17:45 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:50 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Add build ID to failure message https://review.opendev.org/672337 | 17:52 |
| *** Lucas_Gray has joined #openstack-infra | 17:58 | |
| *** jcoufal_ has joined #openstack-infra | 17:59 | |
| *** armax has quit IRC | 18:01 | |
| clarkb | mordred: left comments on the gerrit dockerfile change | 18:02 |
| clarkb | mordred: mostly didn't review the 2.15 stuff since I Have less context for that but tried to point out where there are deltas with the existing 2.13 things | 18:02 |
| mordred | clarkb: awesome - thanks! the 2.15 is fairly different so that's ok - the approach there is "build the plugins together with the war file in the first place" | 18:03 |
| *** jcoufal has quit IRC | 18:03 | |
| *** Lucas_Gray has quit IRC | 18:03 | |
| *** armax has joined #openstack-infra | 18:03 | |
| mordred | clarkb: (and if that's not sufficient, I expect we'll figure that out as we try doing upgrade testing) | 18:03 |
| clarkb | ya | 18:03 |
| corvus | mordred, clarkb: can you +3 https://review.opendev.org/671893 to add the new manifest role to base-test? | 18:04 |
| clarkb | corvus: done | 18:04 |
| clarkb | (its base-test so I just single core approved) | 18:04 |
| corvus | sounds good, thx | 18:04 |
| *** Lucas_Gray has joined #openstack-infra | 18:05 | |
| *** factor has joined #openstack-infra | 18:14 | |
| openstackgerrit | Merged opendev/base-jobs master: Test generate-zuul-manifest role https://review.opendev.org/671893 | 18:14 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: DNM: Test base-jobs https://review.opendev.org/671894 | 18:15 |
| *** ociuhandu has joined #openstack-infra | 18:16 | |
| *** psachin has quit IRC | 18:19 | |
| *** bhavikdbavishi has quit IRC | 18:19 | |
| *** bhavikdbavishi has joined #openstack-infra | 18:19 | |
| *** Lucas_Gray has quit IRC | 18:20 | |
| *** Lucas_Gray has joined #openstack-infra | 18:22 | |
| *** igordc has joined #openstack-infra | 18:27 | |
| openstackgerrit | Merged opendev/system-config master: Serialize the gitea role https://review.opendev.org/672335 | 18:29 |
| mordred | clarkb: jeez /etc/init.d/gerrit | 18:33 |
| clarkb | mordred: ya its a big one | 18:33 |
| clarkb | I tried to pull out the highlights :) | 18:33 |
| mordred | clarkb: the best part is that it does a TON of things that are all elided by the container | 18:33 |
| mordred | clarkb: but it also does a ton of things that are not | 18:34 |
| mordred | and they're all intermingled | 18:34 |
| *** ociuhandu has quit IRC | 18:34 | |
| mordred | ulimit, for instance, is a docker-level setting | 18:34 |
| *** Lucas_Gray has quit IRC | 18:36 | |
| *** Lucas_Gray has joined #openstack-infra | 18:36 | |
| *** e0ne has joined #openstack-infra | 18:42 | |
| *** mriedem has quit IRC | 18:42 | |
| *** tdasilva has quit IRC | 18:51 | |
| *** jcoufal_ has quit IRC | 18:52 | |
| fungi | okay, back and catching up before the meeting | 18:54 |
| openstackgerrit | Merged opendev/system-config master: Reload haproxy configuration when config changes https://review.opendev.org/672323 | 18:55 |
| *** bhavikdbavishi has quit IRC | 18:56 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Fix typo in generate-zuul-manifest role https://review.opendev.org/672343 | 18:58 |
| clarkb | oh right the meeting | 18:59 |
| *** igordc has quit IRC | 19:00 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build a docker images of gerrit https://review.opendev.org/671457 | 19:01 |
| mordred | clarkb, corvus: ^^ I think that addresses clarkb's comments - we'll need to translate some gerrit.config settings into docker-compose.yaml when we get there (most notably heap size - which we'll also clearly want to set differently for test) | 19:01 |
| *** goldyfruit has quit IRC | 19:02 | |
| *** ociuhandu has joined #openstack-infra | 19:04 | |
| *** goldyfruit has joined #openstack-infra | 19:06 | |
| *** factor has quit IRC | 19:07 | |
| *** factor has joined #openstack-infra | 19:08 | |
| *** ociuhandu_ has joined #openstack-infra | 19:09 | |
| *** ociuhandu has quit IRC | 19:09 | |
| *** factor has quit IRC | 19:10 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix typo in generate-zuul-manifest role https://review.opendev.org/672343 | 19:11 |
| *** factor has joined #openstack-infra | 19:11 | |
| donnyd | Ok I'm back | 19:11 |
| *** factor has quit IRC | 19:13 | |
| *** factor has joined #openstack-infra | 19:13 | |
| *** kopecmartin is now known as kopecmartin|off | 19:16 | |
| *** Lucas_Gray has quit IRC | 19:22 | |
| *** whoami-rajat has quit IRC | 19:22 | |
| *** Wryhder has joined #openstack-infra | 19:22 | |
| *** panda|off has quit IRC | 19:23 | |
| *** Wryhder is now known as Lucas_Gray | 19:23 | |
| *** panda has joined #openstack-infra | 19:25 | |
| openstackgerrit | Merged zuul/nodepool master: Add build ID to failure message https://review.opendev.org/672337 | 19:29 |
| *** Lucas_Gray has quit IRC | 19:34 | |
| *** Lucas_Gray has joined #openstack-infra | 19:35 | |
| *** joeguo has joined #openstack-infra | 19:44 | |
| *** kjackal has joined #openstack-infra | 19:45 | |
| *** tosky has quit IRC | 19:47 | |
| *** Lucas_Gray has quit IRC | 19:49 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Promote generate-zuul-manifest role to base https://review.opendev.org/672348 | 19:51 |
| *** Lucas_Gray has joined #openstack-infra | 19:52 | |
| *** factor has quit IRC | 19:52 | |
| *** factor has joined #openstack-infra | 19:53 | |
| *** factor has quit IRC | 19:55 | |
| *** factor has joined #openstack-infra | 19:55 | |
| *** factor has quit IRC | 19:55 | |
| *** mriedem has joined #openstack-infra | 20:00 | |
| clarkb | fungi: I've commented on the linaro flavor change. Thanks for pointing that out | 20:06 |
| *** e0ne has quit IRC | 20:06 | |
| *** slaweq has quit IRC | 20:08 | |
| clarkb | fungi: for gitea01 what are we ready to add it to the inventory and get it ansibled? | 20:09 |
| clarkb | fungi: I was going to ask if it got a 8GB swapfile setup properly too | 20:09 |
| fungi | good point, i'll investigate | 20:10 |
| fungi | i also haven't followed the steps to import data yet | 20:10 |
| clarkb | fungi: that happens after we add it to the inventory | 20:10 |
| clarkb | we add it back to ansible to get gitea and everything installed but not configured, then we import the data to configure it, then we fully add it to the git playbook | 20:11 |
| corvus | clarkb, mordred, fungi: https://review.opendev.org/672348 is ready; then i can stop pestering for a while :) | 20:11 |
| clarkb | looking | 20:11 |
| clarkb | I wont single core approve that one :) | 20:11 |
| fungi | clarkb: what was your take on the two available 40gb volumes in the sjc1 control plane tenant, after my investigation last night? | 20:12 |
| fungi | you okay with them being cleaned up? | 20:12 |
| clarkb | fungi: I think they are for old gitea06 and old gitea01. I seem to recall trying to delete the old gitea06 volume and it refused to so I filed that away for later | 20:12 |
| clarkb | might have ended up in my #status ntoes /me looks | 20:12 |
| *** roman_g has quit IRC | 20:12 | |
| openstackgerrit | Monty Taylor proposed zuul/nodepool master: Install libffi6 on dpkg platforms https://review.opendev.org/672352 | 20:13 |
| clarkb | looks like no :/ | 20:13 |
| clarkb | fungi: I think they likely can be deleted | 20:14 |
| *** raissa has joined #openstack-infra | 20:14 | |
| fungi | Swap: 8191 0 8191 | 20:14 |
| fungi | that's from `free -m` on the replacement gitea01 | 20:14 |
| clarkb | yay the make swap updates worked then (I mean I tested them but still) | 20:14 |
| fungi | so i think we're all set on virtual memory | 20:14 |
| fungi | right, this was just me bumbling around trying to get a server to boot, so of they worked there then i think we can be pretty certain they're foolproof now | 20:15 |
| fungi | i've deleted the two available 40gb volumes and the accompanying ubuntu-bionic-minimal base image they were blocking | 20:17 |
| clarkb | were they blocking that image? | 20:17 |
| fungi | no, wait, i haven't deleted that image | 20:17 |
| clarkb | I'm betting the other gitea servers are still on that image | 20:17 |
| fungi | yeah, i concur | 20:18 |
| fungi | Failed to delete image with name or ID 'd0edcf7a-0779-476e-8285-bcab9043b616': 409 Conflict: Image d0edcf7a-0779-476e-8285-bcab9043b616 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance. (HTTP 409) | 20:18 |
| mordred | corvus: done | 20:18 |
| fungi | i bet you're right, it's also in use by the other gitea instance root filesystems | 20:18 |
| fungi | once those are all replaced we can clean it up | 20:18 |
| clarkb | yup | 20:19 |
| *** jtomasek has quit IRC | 20:21 | |
| fungi | #status log openstack/doc8 in github has been transferred to the PyCQA organization | 20:23 |
| openstackstatus | fungi: finished logging | 20:23 |
| fungi | stephenfin: ^ | 20:23 |
| openstackgerrit | Merged opendev/base-jobs master: Promote generate-zuul-manifest role to base https://review.opendev.org/672348 | 20:26 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add log browsing to build page https://review.opendev.org/671906 | 20:27 |
| clarkb | fungi: I'm going to have to pop out here soon for a bit. Did you want to get the inventory addition up soon if so I'll wait for that and review it | 20:28 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Re-add gitea01 replacement to inventory https://review.opendev.org/672354 | 20:32 |
| fungi | clarkb: ^ sorry, was working on it | 20:32 |
| *** ociuhandu_ has quit IRC | 20:32 | |
| clarkb | fungi: need to exclude it from remote_puppet_git.yaml too | 20:32 |
| *** ociuhandu has joined #openstack-infra | 20:33 | |
| clarkb | fungi: https://review.opendev.org/#/c/667474/ is the example (I didn't capture the xample in the docs but did make note of it, maybe we should make the docs more explicit | 20:34 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Re-add gitea01 replacement to inventory https://review.opendev.org/672354 | 20:34 |
| fungi | clarkb: like that? ^ | 20:34 |
| clarkb | +2 that should do it | 20:35 |
| *** Lucas_Gray has quit IRC | 20:36 | |
| corvus | i bet we could use hostvars to get the ip addrs there | 20:40 |
| clarkb | corvus: for the haproxy config you mean? | 20:42 |
| corvus | clarkb: yep | 20:46 |
| *** pcaruana has quit IRC | 20:50 | |
| clarkb | and now I must pop out. Back later | 20:52 |
| *** kjackal has quit IRC | 20:55 | |
| *** slaweq has joined #openstack-infra | 20:56 | |
| *** slaweq has quit IRC | 21:01 | |
| *** ociuhandu has quit IRC | 21:02 | |
| *** sreejithp has quit IRC | 21:12 | |
| openstackgerrit | Merged zuul/nodepool master: Install libffi6 on dpkg platforms https://review.opendev.org/672352 | 21:15 |
| *** lpetrut has joined #openstack-infra | 21:22 | |
| *** e0ne has joined #openstack-infra | 21:22 | |
| *** lpetrut has quit IRC | 21:22 | |
| *** lpetrut has joined #openstack-infra | 21:23 | |
| *** lpetrut has quit IRC | 21:30 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add log browsing to build page https://review.opendev.org/671906 | 21:31 |
| *** altlogbot_2 has quit IRC | 21:33 | |
| *** irclogbot_3 has quit IRC | 21:33 | |
| *** altlogbot_0 has joined #openstack-infra | 21:33 | |
| *** irclogbot_1 has joined #openstack-infra | 21:34 | |
| *** rosmaita has left #openstack-infra | 21:39 | |
| fungi | i'm confused by the linters error on 672354 as it seems unrelated to the proposed change but i'm also unsure why it would have started spontaneously breaking | 21:42 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix sphinx error https://review.opendev.org/672372 | 21:44 |
| *** yamamoto has joined #openstack-infra | 21:50 | |
| *** e0ne has quit IRC | 21:50 | |
| *** yamamoto has quit IRC | 21:55 | |
| *** irclogbot_1 has quit IRC | 21:59 | |
| *** altlogbot_0 has quit IRC | 22:01 | |
| dmsimard | btw, ara static report generation will land in the next release of 1.x -- need to iterate a bit on it but it works: http://logs.openstack.org/76/672376/2/check/ansible-role-ara-api-ubuntu-postgresql/2e6a610/logs/static/ | 22:07 |
| fungi | dmsimard: that's awesome news! | 22:08 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Move artifacts to their own section https://review.opendev.org/672379 | 22:09 |
| dmsimard | it turns out that I'm much more productive in pure html/css than in javascript :p | 22:09 |
| clarkb | fungi it is mad about the remote puppet git change but I dont understand it | 22:10 |
| clarkb | also I'm only about at my halfway point on today's ride so phone debugging | 22:10 |
| *** armax has quit IRC | 22:11 | |
| *** adriant has quit IRC | 22:11 | |
| *** iokiwi has quit IRC | 22:11 | |
| clarkb | fungi it is the whitespacing | 22:12 |
| clarkb | you need to dedent | 22:12 |
| fungi | hrm, fun | 22:16 |
| fungi | that's new though? not something being changed there | 22:17 |
| fungi | and it's not the duplicate keys it's complaining about which are the problem? | 22:18 |
| fungi | found duplicate key "name" with value "Create repos on gitea servers" (original value: "Puppet-git: Collect the project-config ref") | 22:18 |
| fungi | http://logs.openstack.org/54/672354/2/check/tox-linters/c4e534c/job-output.txt.gz#_2019-07-23_20_46_23_932882 | 22:18 |
| fungi | ahh, yeah, that's in playbooks/remote_puppet_git.yaml | 22:20 |
| *** altlogbot_2 has joined #openstack-infra | 22:22 | |
| fungi | okay, yeah i see where we have two "name" keys defined there but it's not clear what needs to happen with them. should they be in individual list elements? | 22:22 |
| clarkb | fungi the new bit is over indented | 22:22 |
| fungi | oh! it's the hosts line | 22:22 |
| *** gyee has quit IRC | 22:23 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Re-add gitea01 replacement to inventory https://review.opendev.org/672354 | 22:23 |
| * fungi swears audibly at his editor | 22:23 | |
| *** jeremy_houser has quit IRC | 22:24 | |
| fungi | autoindent is a blight | 22:24 |
| fungi | yet another reason to turn it off | 22:24 |
| clarkb | You have my +2 from a phone if you want to reapprove | 22:25 |
| clarkb | (not a +2 in gerrit because ehard | 22:25 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Download-artifact: use the artifact type rather than name https://review.opendev.org/672381 | 22:26 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Use human-readable names for artifact returns https://review.opendev.org/672382 | 22:26 |
| *** altlogbot_2 has quit IRC | 22:27 | |
| *** iokiwi has joined #openstack-infra | 22:28 | |
| *** goldyfruit has quit IRC | 22:30 | |
| *** goldyfruit has joined #openstack-infra | 22:30 | |
| fungi | thanks clarkb, my eyesight is suffering tonight | 22:32 |
| corvus | dmsimard: sweet! i've picked up work on the zuul log display stuff, so we're getting closer to a place where i think we'll feel comfortable switching to swift (where we're going to want to use static generation) | 22:32 |
| corvus | fungi, clarkb +3 | 22:33 |
| *** goldyfruit has quit IRC | 22:38 | |
| *** diablo_rojo has joined #openstack-infra | 22:41 | |
| *** tkajinam has joined #openstack-infra | 22:51 | |
| *** mriedem has quit IRC | 22:55 | |
| *** armax has joined #openstack-infra | 23:04 | |
| *** gyee has joined #openstack-infra | 23:10 | |
| clarkb | oh hah 672354 fails now for a different reason | 23:13 |
| clarkb | it is because gitea01 is the node we test in CI | 23:13 |
| clarkb | and well that one is being untested with this change | 23:13 |
| *** altlogbot_3 has joined #openstack-infra | 23:14 | |
| fungi | oh, likely so | 23:15 |
| *** rcernin has joined #openstack-infra | 23:16 | |
| clarkb | I've got a patch one sec | 23:17 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Re-add gitea01 replacement to inventory https://review.opendev.org/672354 | 23:18 |
| clarkb | fungi: corvus ^ I think that might fix it | 23:18 |
| fungi | http://logs.openstack.org/54/672354/3/check/system-config-run-gitea/0d26df1/job-output.txt.gz#_2019-07-23_22_39_50_318361 | 23:18 |
| fungi | i guess that's the error? | 23:18 |
| *** altlogbot_3 has quit IRC | 23:19 | |
| fungi | hah, yep | 23:20 |
| clarkb | ya we basically don't configure gitea in that change (intentionally) | 23:20 |
| clarkb | switching to a host that we should never actually use in production should give us flexibility to test and also rotate these out like this | 23:20 |
| *** altlogbot_1 has joined #openstack-infra | 23:28 | |
| dmsimard | corvus: ack | 23:28 |
| clarkb | fungi: I +2'd the latest ps I'll let you decide if we should keep trying to work on it today | 23:31 |
| clarkb | (if you want to approve it I mean) | 23:31 |
| fungi | i saw and approved it | 23:31 |
| fungi | still hacking on it some | 23:31 |
| fungi | also winding down for the evening but will see how far i get | 23:31 |
| clarkb | it should be safe as that host isn't in the load balancer | 23:31 |
| fungi | yup | 23:31 |
| clarkb | so worst case we continue to have an unconfigured gitea | 23:31 |
| *** irclogbot_0 has joined #openstack-infra | 23:32 | |
| *** igordc has joined #openstack-infra | 23:35 | |
| *** jamesmcarthur has joined #openstack-infra | 23:36 | |
| *** dchen has joined #openstack-infra | 23:47 | |
| *** aaronsheffield has quit IRC | 23:50 | |
| *** jamesmcarthur has quit IRC | 23:52 | |
| clarkb | seems that it made it to the gate with those changes | 23:54 |
| clarkb | \o/ | 23:54 |
| *** diablo_rojo has quit IRC | 23:54 | |
| *** eernst has joined #openstack-infra | 23:57 | |
| fungi | <mr_burns>exxxxxcellent</mr_burns> | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!