| *** yamamoto has joined #openstack-infra | 00:01 | |
| *** yamamoto has quit IRC | 00:06 | |
| *** ianychoi has quit IRC | 00:07 | |
| *** Goneri has quit IRC | 00:08 | |
| *** ianychoi has joined #openstack-infra | 00:08 | |
| openstackgerrit | Merged opendev/system-config master: Simplify goaccess command https://review.opendev.org/712201 | 00:10 |
|---|---|---|
| openstackgerrit | Merged opendev/system-config master: Pull goaccess report into zuul logs https://review.opendev.org/712202 | 00:10 |
| clarkb | ok I'm going to trigger the periodic job manually now | 00:10 |
| *** igordc has joined #openstack-infra | 00:12 | |
| *** gyee has quit IRC | 00:13 | |
| clarkb | looks like it ran hpapily and the upload worked | 00:15 |
| clarkb | https://de57d96d98408b3dfcc4-c8d4c771caa7bac99c72129a7fbb96d3.ssl.cf1.rackcdn.com/c0b59083df27e26dc03adf2a2b18df46899e3f21/periodic/zuul-ci-goaccess-report/ec88a14/zuul-ci.org_goaccess_report.html | 00:15 |
| clarkb | I'll look into turning that into an artifact tomorrow, but then I think we can treat that as ready for applying to other sites on static | 00:16 |
| clarkb | AJaeger: ^ fyi we can apply that to things to get 404 data | 00:16 |
| *** armax has joined #openstack-infra | 00:16 | |
| *** tosky has quit IRC | 00:28 | |
| openstackgerrit | Merged opendev/system-config master: nodepool-builder: Add webserver https://review.opendev.org/711574 | 00:31 |
| clarkb | and now I'm calling it a day. Thanks allfor helping geting that going | 00:36 |
| ianw | thanks for reviews, ttyl! | 00:36 |
| *** aedc_ has joined #openstack-infra | 00:50 | |
| *** ondrejburian has quit IRC | 00:50 | |
| *** ondrejburian has joined #openstack-infra | 00:51 | |
| ianw | hrm thinking more on the dib cache volume issue; some of it should be ephemeral (build areas) but some of it should probably last across container upgrades (git repo cache) | 00:51 |
| ianw | something to think about | 00:51 |
| *** aedc has quit IRC | 00:52 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add nb01.opendev.org https://review.opendev.org/693118 | 00:57 |
| *** yamamoto has joined #openstack-infra | 00:58 | |
| donnyd | OE just got hit with a power outage - good news is the control plane side UPS is in good shape - bad news is I still haven' | 01:12 |
| donnyd | haven't replaced the compute side UPS | 01:13 |
| *** iurygregory has quit IRC | 01:17 | |
| *** dmellado has quit IRC | 01:22 | |
| *** ianychoi has quit IRC | 01:30 | |
| *** ianychoi has joined #openstack-infra | 01:32 | |
| *** dmellado has joined #openstack-infra | 01:46 | |
| *** larainema has joined #openstack-infra | 01:53 | |
| *** lbragstad_ has quit IRC | 01:56 | |
| *** lbragstad_ has joined #openstack-infra | 01:58 | |
| *** happyhemant has quit IRC | 02:09 | |
| *** ianychoi has quit IRC | 02:10 | |
| *** ianychoi has joined #openstack-infra | 02:18 | |
| *** hongbin has joined #openstack-infra | 02:20 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: nodepool-elements: skip tox install with no pip-and-virtualenv https://review.opendev.org/712225 | 02:29 |
| ianw | i'm going to quick merge that, because it only affects the "plain" builds we're trying out, which are currently in a broken loop | 02:29 |
| *** diablo_rojo has quit IRC | 02:34 | |
| *** yamamoto has quit IRC | 02:35 | |
| *** hongbin has quit IRC | 02:39 | |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: nb01 acme challenge: add missing trailing period https://review.opendev.org/712227 | 02:43 |
| *** yamamoto has joined #openstack-infra | 02:43 | |
| openstackgerrit | Merged openstack/project-config master: nodepool-elements: skip tox install with no pip-and-virtualenv https://review.opendev.org/712225 | 02:47 |
| openstackgerrit | Merged opendev/zone-opendev.org master: nb01 acme challenge: add missing trailing period https://review.opendev.org/712227 | 02:49 |
| *** lbragstad_ has quit IRC | 02:57 | |
| *** ricolin_ has joined #openstack-infra | 03:01 | |
| *** rkukura has quit IRC | 03:06 | |
| *** rkukura has joined #openstack-infra | 03:07 | |
| *** psachin has joined #openstack-infra | 03:26 | |
| *** igordc has quit IRC | 03:30 | |
| *** weifan has joined #openstack-infra | 03:33 | |
| *** weifan has quit IRC | 03:35 | |
| *** ianychoi has quit IRC | 03:35 | |
| *** ianychoi has joined #openstack-infra | 03:37 | |
| *** ricolin_ has quit IRC | 03:38 | |
| *** ykarel|away is now known as ykarel | 03:44 | |
| *** prometheanfire has quit IRC | 03:47 | |
| *** prometheanfire has joined #openstack-infra | 04:10 | |
| *** armax has quit IRC | 04:13 | |
| *** ricolin has quit IRC | 04:44 | |
| *** admcleod has quit IRC | 04:49 | |
| *** ianychoi has quit IRC | 05:00 | |
| *** ianychoi has joined #openstack-infra | 05:03 | |
| *** udesale has joined #openstack-infra | 05:07 | |
| *** Liang__ has joined #openstack-infra | 05:09 | |
| *** raukadah is now known as chandankumar | 05:18 | |
| ianw | yay, "plain" nodes now ready in rax, can start testing out some pip/virtualenv/tox roles | 05:23 |
| *** admcleod has joined #openstack-infra | 05:31 | |
| *** yamamoto has quit IRC | 05:34 | |
| *** evrardjp has quit IRC | 05:35 | |
| *** evrardjp has joined #openstack-infra | 05:35 | |
| *** yamamoto has joined #openstack-infra | 05:39 | |
| *** yamamoto has quit IRC | 05:43 | |
| *** ricolin has joined #openstack-infra | 05:51 | |
| *** tetsuro has joined #openstack-infra | 05:52 | |
| *** rcernin has quit IRC | 05:58 | |
| *** aedc_ has quit IRC | 06:01 | |
| *** ianychoi has quit IRC | 06:01 | |
| *** aedc has joined #openstack-infra | 06:02 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/712235 | 06:03 |
| *** ianychoi has joined #openstack-infra | 06:03 | |
| *** ianychoi has quit IRC | 06:11 | |
| *** ianychoi has joined #openstack-infra | 06:13 | |
| *** yamamoto has joined #openstack-infra | 06:25 | |
| *** ianychoi has quit IRC | 06:29 | |
| *** ianychoi has joined #openstack-infra | 06:31 | |
| *** ianychoi has quit IRC | 06:39 | |
| *** ianychoi has joined #openstack-infra | 06:41 | |
| *** lmiccini has joined #openstack-infra | 06:46 | |
| *** ianychoi has quit IRC | 06:58 | |
| *** pgaxatte has joined #openstack-infra | 07:01 | |
| *** ianychoi has joined #openstack-infra | 07:05 | |
| *** pgaxatte has quit IRC | 07:06 | |
| *** pgaxatte has joined #openstack-infra | 07:06 | |
| AJaeger | clarkb: nice, thanks! | 07:10 |
| *** sshnaidm|afk is now known as sshnaidm | 07:16 | |
| *** dpawlik has joined #openstack-infra | 07:17 | |
| *** iurygregory has joined #openstack-infra | 07:29 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/712235 | 07:29 |
| *** matt_kosut has joined #openstack-infra | 07:33 | |
| *** matt_kosut has quit IRC | 07:35 | |
| *** matt_kosut has joined #openstack-infra | 07:35 | |
| *** hashar has joined #openstack-infra | 07:45 | |
| *** jcapitao has joined #openstack-infra | 07:46 | |
| *** aedc has quit IRC | 07:47 | |
| *** ccamacho has joined #openstack-infra | 07:57 | |
| *** yamamoto has quit IRC | 07:59 | |
| *** yamamoto has joined #openstack-infra | 08:00 | |
| *** tesseract has joined #openstack-infra | 08:02 | |
| *** slaweq has joined #openstack-infra | 08:03 | |
| *** yamamoto has quit IRC | 08:04 | |
| *** yamamoto has joined #openstack-infra | 08:05 | |
| yoctozepto | infra-root: did openstack-discuss ml hang? | 08:08 |
| yoctozepto | we are missing mails we have sent | 08:08 |
| yoctozepto | me and hrw at least | 08:08 |
| *** hrw has joined #openstack-infra | 08:08 | |
| hrw | hi | 08:09 |
| hrw | can someone tell me is there a problem with openstack-discuss ML? | 08:09 |
| hrw | I sent mail "[horizon][kolla] pyscss failure on newest setuptools" and it did not arrived there | 08:09 |
| hrw | yesterday and again today | 08:09 |
| hrw | Message-ID: <6295fd26-d983-75c5-4ead-a36823034d2c@linaro.org> | 08:09 |
| yoctozepto | and mine was "osc version" from yesterday | 08:10 |
| yoctozepto | I don't see them in the queue | 08:10 |
| yoctozepto | ianw, fungi, AJaeger, clarkb: sorry for mentioning you all but it seems pretty urgent and you were here recently ^ | 08:11 |
| yoctozepto | (with "queue" I meant "moderation queue" of course) | 08:13 |
| hrw | last mail was yesterday 15:36 UTC. I mailed later | 08:13 |
| yoctozepto | it's probably not hitting the mailman, the queue is empty, there is always some spam ;-) | 08:14 |
| *** tkajinam has quit IRC | 08:14 | |
| *** psachin has quit IRC | 08:15 | |
| *** yamamoto_ has joined #openstack-infra | 08:18 | |
| *** yamamoto has quit IRC | 08:22 | |
| *** tosky has joined #openstack-infra | 08:28 | |
| *** amoralej|off is now known as amoralej | 08:31 | |
| *** yamamoto_ has quit IRC | 08:35 | |
| *** yamamoto has joined #openstack-infra | 08:37 | |
| *** ykarel is now known as ykarel|lunch | 08:38 | |
| *** ralonsoh has joined #openstack-infra | 08:47 | |
| *** tetsuro has quit IRC | 08:49 | |
| *** gouthamr has quit IRC | 08:49 | |
| *** mgoddard has quit IRC | 08:49 | |
| *** dulek has quit IRC | 08:49 | |
| *** gouthamr has joined #openstack-infra | 08:50 | |
| *** mgoddard has joined #openstack-infra | 08:50 | |
| *** jpena|off is now known as jpena | 08:52 | |
| *** dchen has quit IRC | 08:52 | |
| *** dtantsur|afk is now known as dtantsur | 08:56 | |
| *** ykarel has joined #openstack-infra | 08:56 | |
| *** ykarel|lunch has quit IRC | 08:57 | |
| *** rpittau|afk is now known as rpittau | 08:58 | |
| *** gfidente|afk is now known as gfidente | 09:01 | |
| *** xek has joined #openstack-infra | 09:01 | |
| *** dulek has joined #openstack-infra | 09:03 | |
| *** lmiccini has quit IRC | 09:08 | |
| *** ociuhandu has joined #openstack-infra | 09:16 | |
| *** udesale_ has joined #openstack-infra | 09:27 | |
| *** udesale has quit IRC | 09:30 | |
| *** ociuhandu has quit IRC | 09:38 | |
| *** derekh has joined #openstack-infra | 09:42 | |
| *** ccamacho has quit IRC | 09:44 | |
| *** roman_g has joined #openstack-infra | 09:46 | |
| *** ociuhandu has joined #openstack-infra | 09:50 | |
| *** lmiccini has joined #openstack-infra | 09:50 | |
| AJaeger | yoctozepto: Sorry, cannot help here - and the rest you poinged is not around now. | 09:53 |
| AJaeger | yoctozepto, hrw, we might need to wait for fungi to wake up in a few hours... | 09:53 |
| hrw | AJaeger: ok. | 09:53 |
| hrw | AJaeger: I prefer to report, add all details and then wait rather then report, wait, add details | 09:54 |
| *** AJaeger has quit IRC | 09:54 | |
| yoctozepto | hrw: you scared off Andreas ;-( | 09:55 |
| *** yamamoto has quit IRC | 09:56 | |
| *** yamamoto has joined #openstack-infra | 09:57 | |
| *** yamamoto has quit IRC | 09:59 | |
| *** pkopec has joined #openstack-infra | 10:07 | |
| openstackgerrit | Merged opendev/irc-meetings master: Reschedule Nova Team meeting https://review.opendev.org/712052 | 10:10 |
| *** AJaeger has joined #openstack-infra | 10:14 | |
| yoctozepto | AJaeger: could you send an alert to all channels though? :-) | 10:16 |
| yoctozepto | or something visible | 10:16 |
| yoctozepto | there is currently no indication that ml is non-worky :-( | 10:16 |
| AJaeger | yoctozepto: something like #status notice The mail server for lists.openstack.org is currently not delivering emails. | 10:20 |
| *** yamamoto has joined #openstack-infra | 10:20 | |
| yoctozepto | AJaeger: "not handling" maybe? unsure whether it drops them or forgets to process | 10:21 |
| yoctozepto | AJaeger: I would be clear that mailing lists are simply down | 10:21 |
| AJaeger | #status notice The mail server for lists.openstack.org is currently not handling emails. The infra team will investigate and fix during US morning. | 10:24 |
| openstackstatus | AJaeger: sending notice | 10:24 |
| AJaeger | yoctozepto: done | 10:24 |
| -openstackstatus- NOTICE: The mail server for lists.openstack.org is currently not handling emails. The infra team will investigate and fix during US morning. | 10:26 | |
| yoctozepto | AJaeger: thanks! :D | 10:26 |
| hrw | AJaeger: thank you | 10:26 |
| *** yamamoto has quit IRC | 10:27 | |
| openstackstatus | AJaeger: finished sending notice | 10:28 |
| *** hashar has quit IRC | 10:31 | |
| frickler | infra-root: couple of OOMs on lists.o.o since yesterday, but I'm not sure how to proceed, waiting for fungi's feedback | 10:40 |
| AJaeger | thanks, frickler ! | 10:40 |
| yoctozepto | frickler: and thanks from me too | 10:49 |
| yoctozepto | it probably killed some process responsible for processing, there is no smtp-level rejection response | 10:49 |
| *** rh-jelabarre has joined #openstack-infra | 10:50 | |
| *** Lucas_Gray has joined #openstack-infra | 10:52 | |
| frickler | yoctozepto: likely all the mails are queued locally, waiting to be processed by mailman | 10:59 |
| yoctozepto | frickler: indeed | 10:59 |
| *** ccamacho has joined #openstack-infra | 11:00 | |
| *** roman_g has quit IRC | 11:00 | |
| frickler | infra-root: could you review https://review.opendev.org/711245 in order to get statusbot into #opendev? I think that would be a good thing to have before further IRC migration happens | 11:01 |
| *** priteau has joined #openstack-infra | 11:02 | |
| *** Lucas_Gray has quit IRC | 11:03 | |
| *** yamamoto has joined #openstack-infra | 11:03 | |
| *** verdurin has joined #openstack-infra | 11:05 | |
| *** Lucas_Gray has joined #openstack-infra | 11:08 | |
| *** yamamoto has quit IRC | 11:09 | |
| *** Lucas_Gray has quit IRC | 11:13 | |
| *** Lucas_Gray has joined #openstack-infra | 11:15 | |
| *** udesale_ has quit IRC | 11:17 | |
| *** ttx has quit IRC | 11:17 | |
| *** rh-jelabarre has quit IRC | 11:18 | |
| *** ttx has joined #openstack-infra | 11:19 | |
| *** ttx has quit IRC | 11:19 | |
| *** ianychoi has quit IRC | 11:24 | |
| *** ttx has joined #openstack-infra | 11:28 | |
| *** Wryhder has joined #openstack-infra | 11:33 | |
| *** Lucas_Gray has quit IRC | 11:34 | |
| *** aedc has joined #openstack-infra | 11:34 | |
| *** Wryhder is now known as Lucas_Gray | 11:34 | |
| *** nicolasbock has joined #openstack-infra | 11:38 | |
| *** rlandy has joined #openstack-infra | 11:54 | |
| *** rpittau is now known as rpittau|bbl | 11:58 | |
| *** Goneri has joined #openstack-infra | 11:58 | |
| *** jcapitao is now known as jcapitao_lunch | 11:58 | |
| *** aedc has quit IRC | 11:59 | |
| *** udesale has joined #openstack-infra | 12:02 | |
| *** jcapitao_lunch has quit IRC | 12:03 | |
| *** jcapitao_lunch has joined #openstack-infra | 12:05 | |
| openstackgerrit | Merged openstack/project-config master: Add rss link for cyborg-specs https://review.opendev.org/711875 | 12:08 |
| *** sshnaidm is now known as sshnaidm|afk | 12:10 | |
| *** ociuhandu has quit IRC | 12:15 | |
| *** jpena is now known as jpena|lunch | 12:22 | |
| *** ociuhandu has joined #openstack-infra | 12:27 | |
| *** rh-jelabarre has joined #openstack-infra | 12:28 | |
| *** ociuhandu has quit IRC | 12:29 | |
| *** ociuhandu has joined #openstack-infra | 12:30 | |
| *** ociuhandu_ has joined #openstack-infra | 12:32 | |
| *** gfidente has quit IRC | 12:33 | |
| *** ociuhandu has quit IRC | 12:35 | |
| *** gfidente has joined #openstack-infra | 12:38 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Enforce sql connections for scheduler and web https://review.opendev.org/630472 | 12:41 |
| *** Lucas_Gray has quit IRC | 12:47 | |
| *** Lucas_Gray has joined #openstack-infra | 12:48 | |
| *** aedc has joined #openstack-infra | 12:52 | |
| *** Lucas_Gray has quit IRC | 12:54 | |
| zbr | can we add a periodic-weekly pipeline? | 12:55 |
| *** roman_g has joined #openstack-infra | 12:57 | |
| *** jpena|lunch is now known as jpena | 13:01 | |
| *** Lucas_Gray has joined #openstack-infra | 13:02 | |
| *** aedc has quit IRC | 13:14 | |
| *** ociuhandu_ has quit IRC | 13:15 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add ensure-snap role https://review.opendev.org/712414 | 13:17 |
| *** ociuhandu has joined #openstack-infra | 13:17 | |
| *** lbragstad_ has joined #openstack-infra | 13:18 | |
| *** matt_kosut has quit IRC | 13:19 | |
| yoctozepto | zbr: we considered that for kolla as well, but then again the most daily load is usually from check/gate, except on weekends | 13:19 |
| zbr | yoctozepto: i recently faced few jobs that are not triggered often enough, and they got broken during last 11 days. daily changes do not help because the files triggering them are very narrow. | 13:21 |
| zbr | there are 78 jobs defined in tripleo-ansible, and you do not want to trigger all on any change. | 13:21 |
| zbr | still you may want to build them weekly to know their status. | 13:21 |
| *** irclogbot_0 has quit IRC | 13:22 | |
| zbr | we have this on rdo, with two additonal pipelines weekly and twice-a-week ones but if I enable jobs there, I would not be able to make their runs visible in opendev zuul interface, so basically killing the benefits of running them. | 13:23 |
| yoctozepto | zbr: :O all right, your case is worse | 13:23 |
| *** irclogbot_1 has joined #openstack-infra | 13:23 | |
| zbr | using the daily pipeline is overkill | 13:24 |
| yoctozepto | zbr: agreed | 13:24 |
| zbr | (even if these jobs are quick, ~10mins each) | 13:24 |
| yoctozepto | zbr: well, multiplied by 78 it still is 13 hours of compute time somewhere, even if done concurrently | 13:25 |
| zbr | travis has a brilliant feature to enable rebuilding on schedule if no recent builds are made, quite a nice way to keep build "fresh". | 13:26 |
| *** jamesmcarthur has joined #openstack-infra | 13:27 | |
| yoctozepto | zbr: agreed, only periodics we really need periodic are those like kolla publishing jobs | 13:28 |
| yoctozepto | zbr: others should try to stay "fresh" and that's it :-) | 13:28 |
| *** Lucas_Gray has quit IRC | 13:30 | |
| *** jcapitao_lunch has quit IRC | 13:30 | |
| yoctozepto | zbr: did you try proposing this to zuul? | 13:31 |
| *** Lucas_Gray has joined #openstack-infra | 13:32 | |
| zbr | i am waiting for some feedback from infra-cores before proposing a new pipelines | 13:33 |
| *** jcapitao has joined #openstack-infra | 13:33 | |
| *** yamamoto has joined #openstack-infra | 13:36 | |
| *** amoralej is now known as amoralej|lunch | 13:40 | |
| *** ociuhandu has quit IRC | 13:41 | |
| *** dave-mccowan has joined #openstack-infra | 13:42 | |
| *** ociuhandu has joined #openstack-infra | 13:43 | |
| *** zxiiro has joined #openstack-infra | 13:43 | |
| *** rh-jelabarre has quit IRC | 13:43 | |
| *** rh-jelabarre has joined #openstack-infra | 13:44 | |
| *** dave-mccowan has quit IRC | 13:46 | |
| *** ccamacho has quit IRC | 13:46 | |
| *** ociuhandu has quit IRC | 13:48 | |
| *** yamamoto has quit IRC | 13:49 | |
| *** zigo has quit IRC | 13:49 | |
| *** lbragstad_ is now known as lbragstad | 13:52 | |
| *** matt_kosut has joined #openstack-infra | 13:53 | |
| *** dave-mccowan has joined #openstack-infra | 13:55 | |
| fungi | i'm taking a look at lists.o.o, probably an oom incident | 13:56 |
| *** Liang__ is now known as LiangFang | 13:56 | |
| fungi | [Wed Mar 11 10:02:56 2020] Out of memory: Kill process 14406 (python) score 7 or sacrifice child | 13:57 |
| fungi | yup | 13:57 |
| fungi | i'll get the queue handler restarted | 13:57 |
| *** matt_kosut has quit IRC | 13:58 | |
| *** ccamacho has joined #openstack-infra | 13:59 | |
| yoctozepto | fungi: yay, thank you very much | 13:59 |
| fungi | okay, i've restarted all the handlers for the 5 ml sites we operate from that server | 14:00 |
| yoctozepto | zbr: I meant also the "refresh" thingy from travis ;-) | 14:00 |
| fungi | hopefully messages start arriving momentarily | 14:00 |
| yoctozepto | fungi: spam assassin engaged | 14:00 |
| fungi | i'm going to take a look in cacti and see what was going on with memory utilization there | 14:00 |
| fungi | looks like some sort of daily process, might be log rotation, spiking memory utilization: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=219&rra_id=all | 14:02 |
| fungi | oh, though the spike in used (not just buffers/cache) kicked in at 09:45z | 14:04 |
| fungi | no obvious cronjobs around that time | 14:06 |
| fungi | and no shell logins | 14:06 |
| fungi | frickler ssh'd into it at 10:30 but that was well after | 14:07 |
| fungi | so i don't think it was anything we did manually | 14:07 |
| yoctozepto | very weird | 14:08 |
| * fungi has sometimes accidentally oom'd the server looking at very large logfiles | 14:08 | |
| * yoctozepto understands | 14:08 | |
| yoctozepto | anyways, messages don't seem like coming | 14:09 |
| fungi | i'll check to see if it's still all held | 14:09 |
| yoctozepto | fungi: nothing in pipermail nor moderation queue | 14:10 |
| fungi | yeah, service restart didn't properly restart them | 14:12 |
| yoctozepto | bbl | 14:12 |
| fungi | i had to service stop and then service start | 14:12 |
| fungi | now they're running | 14:12 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Remove Networking VPP team meeting https://review.opendev.org/712448 | 14:14 |
| *** amoralej|lunch is now known as amoralej | 14:14 | |
| artom | o/ If I'd like to request a specific flavor from some of our nodepool providers, who do I talk to? Specifically, vexxhost and limestone | 14:15 |
| artom | (This isn't out of the blue, sean-k-mooney's been talking about with them before) | 14:17 |
| artom | I'm just here to finish the job ;) | 14:17 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Remove Senlin team meeting https://review.opendev.org/712450 | 14:17 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Remove Trove team meeting https://review.opendev.org/712451 | 14:18 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Remove Upgrades SIG meeting https://review.opendev.org/712452 | 14:18 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Remove Doc team meeting https://review.opendev.org/712453 | 14:20 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Remove FEMDC SIG meeting https://review.opendev.org/712454 | 14:21 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Remove Glance Bug Squad meeting https://review.opendev.org/712455 | 14:22 |
| *** ociuhandu has joined #openstack-infra | 14:22 | |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Remove Group-Based policy team meeting https://review.opendev.org/712457 | 14:25 |
| *** yamamoto has joined #openstack-infra | 14:25 | |
| *** ociuhandu has quit IRC | 14:27 | |
| *** matt_kosut has joined #openstack-infra | 14:29 | |
| *** yamamoto has quit IRC | 14:31 | |
| *** ociuhandu has joined #openstack-infra | 14:37 | |
| *** sshnaidm|afk is now known as sshnaidm | 14:37 | |
| *** armax has joined #openstack-infra | 14:38 | |
| openstackgerrit | Merged openstack/infra-manual master: Fix Peer Review section formatting https://review.opendev.org/711819 | 14:45 |
| openstackgerrit | Merged openstack/infra-manual master: Remove section "Updating devstack-vm-gate-wrap.sh" https://review.opendev.org/711820 | 14:45 |
| zbr | fungi: did you see my question re having a weekly pipeline? what do you think? | 14:54 |
| corvus | artom: is there any documentation of the effort (a ml post or something about what the new flavors would be for and why they are needed?) | 14:54 |
| artom | corvus, ah, sorry for not following up, but I was told (by Sean, internally), that what I thought we had to ask for already existed | 14:55 |
| artom | corvus, specifically, nested-virt and nested-virt-centos-7 | 14:55 |
| * fungi doesn't understand "internally" | 14:55 | |
| artom | fungi, Red Hat internal IRC :) | 14:55 |
| artom | He didn't, like, get all up in my giblets | 14:55 |
| fungi | ahh, just remember we're an open collaboration and aren't going to be aware or have access to internal conversation details so would need them reexplained in public | 14:56 |
| *** ahosam has joined #openstack-infra | 14:57 | |
| fungi | zbr: i don't have any objections to periodic pipelines with different periods. you mainly want them for a bunch of the tripleo daily jobs? | 14:57 |
| corvus | zbr, yoctozepto: i think a weekly pipeline would be fine. i was also thinking that if there are a lot of jobs which don't run often, but they share some commonality that might break, you could consider running one of them on all changes as a representative sample. if they're all too different, then that won't help of coures. | 14:57 |
| artom | fungi, yeah, totally aware of that - just didn't think it was super relevant since it was a misunderstanding between Sean and I, and what we needed already exists | 14:57 |
| artom | So nothing actionable on anyone besides ourselves | 14:57 |
| fungi | oh, i misread what you were saying before | 14:58 |
| openstackgerrit | Merged openstack/cookiecutter master: Autofill bug tracker link in contributor docs https://review.opendev.org/708703 | 14:58 |
| zbr | super, I will propose one new pipeline. | 14:58 |
| fungi | artom: yeah, makes sense | 14:58 |
| fungi | glad to hear we have what you need ;) | 14:58 |
| artom | You always do ^_^ | 14:58 |
| *** lbragstad_ has joined #openstack-infra | 14:59 | |
| *** Lucas_Gray has quit IRC | 14:59 | |
| *** Wryhder has joined #openstack-infra | 14:59 | |
| *** Wryhder is now known as Lucas_Gray | 15:00 | |
| *** hrw has left #openstack-infra | 15:01 | |
| *** lbragstad has quit IRC | 15:01 | |
| *** bdodd has joined #openstack-infra | 15:03 | |
| *** hashar has joined #openstack-infra | 15:03 | |
| *** Lucas_Gray has quit IRC | 15:04 | |
| clarkb | fungi: re lists OOM, I seem to recall that mailamn processes each queue serially. That makes me wonder if its single bad inputs that cause it to do that | 15:06 |
| *** Lucas_Gray has joined #openstack-infra | 15:08 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Add periodic-1w pipeline https://review.opendev.org/712473 | 15:09 |
| yoctozepto | fungi: thanks for handling lists, I guess we can reset the irc status now? :_) | 15:09 |
| corvus | well, we didn't change the status, it was just a notice | 15:10 |
| corvus | i guess we could send another? if folks think that's necessary | 15:10 |
| *** hashar has quit IRC | 15:11 | |
| clarkb | fungi: the oom message may record if a single mailman process is very large then we can work back from that to its logs to check input messages? | 15:12 |
| *** ykarel is now known as ykarel|away | 15:13 | |
| *** AJaeger has quit IRC | 15:14 | |
| *** yamamoto has joined #openstack-infra | 15:17 | |
| *** eharney has quit IRC | 15:20 | |
| yoctozepto | corvus: true, my bad | 15:21 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Return goaccess html as zuul artifact https://review.opendev.org/712478 | 15:24 |
| clarkb | I think I got ^ right, but someone that understands that better than me should probably double check (corvus maybe?) | 15:24 |
| *** sreejithp has joined #openstack-infra | 15:24 | |
| *** yamamoto has quit IRC | 15:25 | |
| *** mattw4 has joined #openstack-infra | 15:26 | |
| *** pkopec has quit IRC | 15:26 | |
| fungi | #status log restarted all mailman queue runners on lists.o.o at 14:12z following an oom killer incident | 15:26 |
| openstackstatus | fungi: finished logging | 15:26 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Add tox-py38 in check https://review.opendev.org/712480 | 15:27 |
| clarkb | fungi: looks like all the oom logs show us is `python` and a pid | 15:30 |
| fungi | and sizes | 15:31 |
| fungi | we might be able to correlate pids to logs | 15:31 |
| clarkb | I'm digging around to see if I can map a pid to ya that | 15:31 |
| clarkb | not having much luck yet. Looking at mailman error logs for the various vhosts and trying to match up pids with those that were killed by oomkiller | 15:39 |
| clarkb | now I'm going to go the other way and see what pids I can map to vhosts then check against the oom logs as that might be more productive | 15:41 |
| clarkb | I think zuul's mailman smtp queue processor was the first process chosen to be killed | 15:45 |
| clarkb | pid 32137 | 15:45 |
| clarkb | it had a relatively small rss according to the logs | 15:46 |
| clarkb | I don't think it was the instigator, just an innocent bystander | 15:46 |
| *** gyee has joined #openstack-infra | 15:47 | |
| clarkb | aha the qrunner log is what I need to be looking at | 15:47 |
| *** mattw4 has quit IRC | 15:51 | |
| *** zigo has joined #openstack-infra | 15:52 | |
| *** dtantsur is now known as dtantsur|brb | 15:53 | |
| clarkb | ya grepping for '(14[1234]' in all the qrunner logs doesn't show anything | 15:54 |
| clarkb | perhaps these are subprocesses that don't get their pids logged? There are also significantly more python processes in the oom process listing than are running on the server now | 15:55 |
| clarkb | I'm quickly running out of thread to pull on | 15:55 |
| *** AJaeger has joined #openstack-infra | 15:56 | |
| *** yamamoto has joined #openstack-infra | 15:57 | |
| *** jcapitao is now known as jcapitao_afk | 15:57 | |
| *** eharney has joined #openstack-infra | 15:58 | |
| *** diablo_rojo has joined #openstack-infra | 15:59 | |
| AJaeger | infra-root, can we have https://review.opendev.org/#/c/711245/ merge to get statusbot to #opendev, please? | 16:03 |
| *** yamamoto has quit IRC | 16:03 | |
| openstackgerrit | Thierry Carrez proposed opendev/infra-specs master: POC Xwiki farm at wiki.opendev.org https://review.opendev.org/710057 | 16:04 |
| clarkb | fungi: its almost like we leaked processes. Or we received a bunch of spam all at once maybe and that forked a lot of processes that are short lived | 16:10 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Declare support for Python3.8 https://review.opendev.org/712489 | 16:16 |
| clarkb | mordred: my zuul_return change failed because ansible lint doesn't know about zuul_return https://zuul.opendev.org/t/openstack/build/ceb5142782e74bd493535386366ec78f/log/job-output.txt#752 what is the magic for having it learn that is a thing? | 16:19 |
| *** pkopec has joined #openstack-infra | 16:20 | |
| AJaeger | clarkb, is Zuul installed? | 16:22 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Return goaccess html as zuul artifact https://review.opendev.org/712478 | 16:23 |
| clarkb | AJaeger: it wasn't I think ^ may do it based on ozj setup | 16:23 |
| *** ricolin has quit IRC | 16:25 | |
| openstackgerrit | Jeremy Stanley proposed zuul/nodepool master: Declare support for Python3.8 https://review.opendev.org/712494 | 16:25 |
| AJaeger | clarkb: those changes in tox.ini look fine | 16:25 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Remove fix-tox workaround for python3.5 https://review.opendev.org/712495 | 16:26 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Remove duplicate variables for tox jobs https://review.opendev.org/712496 | 16:26 |
| mordred | clarkb: oh right. this is a general problem | 16:27 |
| mordred | clarkb: and yes - that might work - although we also run in to bindep issues - so it still might not work | 16:28 |
| clarkb | ah yup ozj has a very zuul looking bindep file | 16:28 |
| mordred | I really wish transitive bindep was a thing | 16:28 |
| clarkb | is there a way to tell the linter "foo" exists without actually defining foo? | 16:29 |
| mordred | clarkb: I don't know - but I would love to figure out a way to do so | 16:29 |
| mordred | because the "install zuul everywhere we want to lint jobs and use zuul_return" thing gets old | 16:29 |
| clarkb | ansible-lint's docs don't really hint at that functionality | 16:30 |
| clarkb | (its actually super simple config wise) | 16:30 |
| zbr | clarkb: you refer to foo module? | 16:32 |
| zbr | nope, not possible now, but there is a feature request I added for it, never had time to work on it. | 16:32 |
| *** jcapitao_afk is now known as jcapitao | 16:32 | |
| zbr | there is a real need for that, as zuul is very FAT | 16:32 |
| clarkb | zbr: zuul defines zuul_return as an action plugin (I think) to return dictionairy data from zuul jobs into zuul itself | 16:33 |
| clarkb | zbr: we then want to lint the zuul jobs with ansible-lint and have to forklift zuul in to get zuul_return as valid name | 16:33 |
| zbr | yep, it is prefered to install the module in order to properly lint it. ignoring would prevent detecting wrong arguments. | 16:34 |
| zbr | easiest way to do this is to publish this module as a galaxy collection, so we can install it and consume it without having install zuul | 16:35 |
| zbr | never did this myself, but it should work. | 16:35 |
| clarkb | does ansible-lint actually check things like arguments? it seems it mostly complains about "you shouldn't use shell for this" and the actual bug it catches is unmatched ' in comments | 16:36 |
| zbr | it uses ansible a lot, so it needs the module. | 16:36 |
| zbr | in previous versions it was very stupid, doing mostly only syntax checks, but as it grew older, it started to rely more on ansible. | 16:37 |
| zbr | obviosuly this has pros and cons. | 16:37 |
| *** jcoufal has joined #openstack-infra | 16:37 | |
| mordred | zbr: I agree - publishing as a collection could be a solution ... but I think that infrastructure is still too new, and also only works on latest ansible and we still support several versions back | 16:38 |
| mordred | so - I think once collection support is solid and in all the versions of ansible we support, that will likely be a really good choice | 16:38 |
| clarkb | my firefox process has suddenly gotten very sad | 16:40 |
| clarkb | (trying to confirm that the latest ps failed on the lack of bindep rules) | 16:40 |
| *** udesale has quit IRC | 16:41 | |
| mordred | clarkb: for system-config - we could just add zuul to required-projects and set a path to ../../zuul/zuul/zuul/ansible | 16:41 |
| mordred | we don't ACTUALLY need to install anything | 16:41 |
| mordred | it makes locally running slightly more challenging - but not extremely so | 16:41 |
| *** jcapitao has quit IRC | 16:42 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Return goaccess html as zuul artifact https://review.opendev.org/712478 | 16:43 |
| *** rpittau|bbl is now known as rpittau | 16:43 | |
| clarkb | mordred: I think consistency with ozj and friends is probably worthwhile | 16:43 |
| mordred | clarkb: nod | 16:44 |
| clarkb | also I'm not sure zuul is in the openstack (or opendev) tenants? | 16:45 |
| fungi | it does seem like there might be an opportunity for zuul to provide some of those tools natively/transparently without having to explicitly request zuul source code or install its package | 16:46 |
| mordred | fungi: yeah - I agree. it's just mainly that the packaging mechanism that would allow that to work _well_ is still pre-release | 16:47 |
| fungi | zuul is already installed on the executors... can we leverage that somehow? | 16:47 |
| mordred | fungi: don't think so - also - we need to be able to solve this for local tox invocation as well | 16:47 |
| *** jamesmcarthur has quit IRC | 16:49 | |
| fungi | ahh | 16:49 |
| fungi | yeah that gets tough | 16:49 |
| fungi | though i'll admit i'm not sure why a local user needs to run whatever the zuul-return bits are outside a zuul invoked context | 16:49 |
| clarkb | fungi: because that is how ansible-lint determines zuul_module is valid ansible | 16:50 |
| clarkb | er zuul_return not zuul_module | 16:50 |
| *** jamesmcarthur has joined #openstack-infra | 16:50 | |
| fungi | oh, this is all so the local ansible in the consuming project can be linted even though it's ansible callouts to a module which makes no sense to run outside a zuul context? | 16:51 |
| clarkb | fungi: well we ansible lint both the actual ansible and the ansible for zuul, but ya | 16:51 |
| fungi | would it make sense to just tell ansible-lint to ignore the bits which call zuul_return (if that's possible)? | 16:52 |
| clarkb | fungi: we can exclude them yes. We'd miss out on checking for unbalanced ' (the We'd at the beginning of the sentence would trigger that) | 16:53 |
| zbr | excluding files using zuul_return is what I done so far, in 2-3 places, ugly but worked. | 16:56 |
| fungi | and i guess the files which use zuul_return can be trimmed down to the bare minimum and other stuff can go in separate files, to reduce what's not covered through skipping | 16:58 |
| *** ricolin has joined #openstack-infra | 17:01 | |
| *** jamesmcarthur has quit IRC | 17:01 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split zuul_return into its own file and skip linting https://review.opendev.org/712507 | 17:05 |
| *** rh-jelabarre has quit IRC | 17:05 | |
| mordred | fungi, zbr, clarkb: ^^ maybe something like that | 17:05 |
| clarkb | mordred: can you do multiple run playbooks like that? I don't think you can but in this case its fine as that can be a post playbook anyway | 17:06 |
| mordred | clarkb: you can | 17:06 |
| clarkb | TIL | 17:06 |
| mordred | clarkb: but we could also make it a post | 17:06 |
| mordred | clarkb: it was a feature pabelanger needed - but turns out to be handy from time to time | 17:06 |
| mordred | clarkb: the tl;dr was when using zuul to test things that in the real world already existed as a sequence of playbooks not just one playbook | 17:07 |
| clarkb | gotcha makes sense. | 17:07 |
| *** ociuhandu has quit IRC | 17:07 | |
| clarkb | In that case I'll leave a +2 | 17:07 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Split zuul_return into its own file and skip linting https://review.opendev.org/712507 | 17:07 |
| mordred | clarkb: actually - I like it being post now that you mention it | 17:07 |
| mordred | I think it's more appropriate | 17:07 |
| clarkb | ++ | 17:08 |
| *** rh-jelabarre has joined #openstack-infra | 17:08 | |
| clarkb | if we land those two changes I'll happily rerun periodic via zuul-enqueue | 17:08 |
| mordred | so - this report made me think - it would be nice if zuul-web had a "show me the latest build of job X" url/api | 17:08 |
| clarkb | mordred: I think that would need js to parse out zuul builds api responses but should be doable | 17:09 |
| mordred | yeah | 17:09 |
| clarkb | Also once those changes land I can push up a set of jobs for the other vhosts on static (that should give ajaeger his 404 reporting in all the places) | 17:10 |
| mordred | that way you could have something like /t/openstack/build/latest/system-config-goaccess-report | 17:10 |
| clarkb | I wanted to have one working well before we applied it more broadly though | 17:10 |
| mordred | clarkb: ++ | 17:10 |
| fungi | seems like that would be pretty easy to add to the dashboard | 17:12 |
| zbr | ansible-lint can take care of its own config file, w/ excludes, i am not a big supporter of using bash contraptions to call it. | 17:12 |
| zbr | also the 4.2.0 is able to detect ansible files, so we are not forced to compile a list of files to feed it | 17:13 |
| openstackgerrit | sebastian marcet proposed opendev/puppet-openstackid master: Added python packages install support https://review.opendev.org/712511 | 17:13 |
| *** nicolasbock has quit IRC | 17:14 | |
| clarkb | zbr: the bash contraption exists because it tried to read all yaml files | 17:14 |
| clarkb | and all yaml files are not ansible files | 17:14 |
| clarkb | if 4.2.0 fixes that great. A change to switch the bash out will be self testing | 17:14 |
| zbr | clarkb: 4.2.0 is not silver bullet, but has detection and we can also add exclude patterns, which I prefer instead bash/find. | 17:15 |
| zbr | new version can be run like flake8, without any arguments. | 17:15 |
| *** jamesmcarthur has joined #openstack-infra | 17:15 | |
| openstackgerrit | Merged zuul/zuul master: Store build.error_detail in SQL https://review.opendev.org/709857 | 17:16 |
| *** chandankumar is now known as raukadah | 17:18 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Update Feature branches, releases sections for OpenStack https://review.opendev.org/711817 | 17:20 |
| *** nicolasbock has joined #openstack-infra | 17:20 | |
| *** jamesmcarthur has quit IRC | 17:23 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Update Feature branches, releases sections for OpenStack https://review.opendev.org/711817 | 17:24 |
| *** pgaxatte has quit IRC | 17:26 | |
| *** jamesmcarthur has joined #openstack-infra | 17:28 | |
| *** Lucas_Gray has quit IRC | 17:33 | |
| corvus | clarkb, mordred: hang on. i'm really unhappy with the amount of contortion we're doing just to use zuul_return. let's look at other options. | 17:34 |
| corvus | clarkb, mordred: first, do we actually want ansible-lint? we've frozen at an old version and turned off all kinds of stuff | 17:34 |
| corvus | can we just get rid of it because it's clearly wrong? | 17:34 |
| clarkb | corvus: that playbook I am modifying has had ansible-lint catch the unparied ' issue | 17:35 |
| clarkb | I am struggling to think of other valid bug catches | 17:35 |
| *** evrardjp has quit IRC | 17:35 | |
| corvus | clarkb: what was that issue? | 17:35 |
| *** evrardjp has joined #openstack-infra | 17:35 | |
| clarkb | corvus: if you put an unmatched ' in an ansible playbook it is an ansible error. Even if that ' is in a comment or in a literal string quote | 17:36 |
| corvus | clarkb: it was in a comment? | 17:36 |
| mordred | you can't use contractions in ansible comments | 17:36 |
| mordred | yeah | 17:36 |
| clarkb | corvus: in this case it was in the shell command block | 17:36 |
| clarkb | (as a shell comment) | 17:36 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move ansible-lint config into config file https://review.opendev.org/712519 | 17:36 |
| mordred | also - we do not have ansible-lint pinned | 17:36 |
| corvus | http://paste.openstack.org/show/790554/ | 17:36 |
| corvus | that works | 17:36 |
| mordred | sorry - was in a comment in a shell block | 17:37 |
| corvus | mordred: oh, i thought we did? because of that conversation about how it was growing all sorts of opinions about what style you should use | 17:37 |
| *** jamesmcarthur has quit IRC | 17:37 | |
| mordred | system-config is ansible-lint>=4.1.0 currently | 17:37 |
| *** jamesmcarthur has joined #openstack-infra | 17:37 | |
| clarkb | corvus: possible they fixed it in the yaml comment case but not the literal string case | 17:37 |
| fungi | we previously pinned ansible-lint | 17:38 |
| clarkb | corvus: I'll get a link to where it broke | 17:38 |
| corvus | clarkb: so like this? http://paste.openstack.org/show/790555/ | 17:38 |
| clarkb | corvus: yup | 17:38 |
| corvus | clarkb: that fails in ansible | 17:38 |
| fungi | i proposed some straw-man changes for disabling a variety of checks, and some others for what was needed to reenable them | 17:38 |
| fungi | some of which merged | 17:38 |
| *** bnemec has quit IRC | 17:38 | |
| fungi | though i was really using those changes to argue for dropping ansible-lint jobs | 17:38 |
| corvus | clarkb: so i guess you're saying that ansible-lint saved a round-trip fix because this is a periodic job? | 17:39 |
| clarkb | corvus: https://review.opendev.org/#/c/712121/2/playbooks/periodic/goaccess.yaml line 51 | 17:39 |
| clarkb | corvus: yup | 17:39 |
| *** bnemec has joined #openstack-infra | 17:39 | |
| clarkb | (I'm just trying to call out the good things that ansible-lint has done for us) | 17:39 |
| *** hashar has joined #openstack-infra | 17:39 | |
| corvus | yeah. i'd like to keep ansible-lint, but if the choice is "rewrite our playbooks so we can manually exclude the ones with zuul_return" or "drop ansible-lint" i would go with door #2. | 17:40 |
| * mordred does not have a strong opinion one way or the other - but if we are going to use it, I do like the ability to configure it in a file rather than with a super-long command line (those outputs are always hard to read) | 17:40 | |
| *** jamesmcarthur has quit IRC | 17:40 | |
| corvus | i feel like a linter which is actively causing us to rewrite our code in a sub-optimal way is having the opposite of the intended effect | 17:40 |
| mordred | yah | 17:41 |
| zbr | i personally find it quite good at promoting good coding practices, skips rules make it easy to pick only those rules we want the adopt. | 17:41 |
| *** pkopec has quit IRC | 17:41 | |
| mordred | zbr: the problem is it keeps adding rules which we keep having to exclude | 17:41 |
| mordred | like its antipathy towards shell commands or telling us we shouldin't use "latest", etc | 17:41 |
| zbr | yeah, same applies to any linter, and TBH, I think it did not add more than two rules in the last year. | 17:42 |
| mordred | but - otoh - I do find it to have an amount of value and it does catch some bugd | 17:42 |
| corvus | maybe it's growth spurt is at an end :) | 17:42 |
| mordred | yeah - which is a good thing for a linter :) | 17:42 |
| corvus | so are there any other ways we can trick it into thinking zuul_return is real? | 17:42 |
| mordred | corvus: we could put a fake zuul_return module in system-config | 17:43 |
| zbr | my plan was to add rule-profiles to it, so you can pick between more or less conservative sets of rules (but never got time) | 17:43 |
| fungi | need to go grab lunch and run some errands but i should be back in a couple of hours | 17:43 |
| corvus | mordred: yeah, maybe that's worth looking into? something that only takes effect when run under tox? | 17:43 |
| mordred | yeah | 17:44 |
| mordred | corvus: let me push up a sake-of-argument patch | 17:44 |
| clarkb | a just valid enough shim to trick the parser? | 17:51 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move ansible-lint config into config file https://review.opendev.org/712519 | 17:52 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a fake zuul_return module https://review.opendev.org/712525 | 17:52 |
| mordred | clarkb, corvus: ^^ something liek that | 17:52 |
| zbr | btw, can we do something to avoid sending notification to two irc channels? | 17:53 |
| corvus | zbr: like https://review.opendev.org/711106 ? | 17:54 |
| clarkb | I've asked for voting quorum on ^ then we can schedule a split day and hold users to it at that point | 17:54 |
| corvus | mordred: you win the zuul error message lottery | 17:54 |
| mordred | corvus: yay! | 17:54 |
| *** jamesmcarthur has joined #openstack-infra | 17:54 | |
| mordred | corvus: wow | 17:55 |
| corvus | mordred: i like 525 | 17:55 |
| openstackgerrit | Merged zuul/zuul master: Cache getUser in Github connection https://review.opendev.org/710985 | 17:55 |
| *** eharney has quit IRC | 17:55 | |
| mordred | corvus: I'm excited to learn what the error was | 17:55 |
| corvus | mordred: http://paste.openstack.org/show/790558/ | 17:56 |
| openstackgerrit | Merged opendev/irc-meetings master: Remove FEMDC SIG meeting https://review.opendev.org/712454 | 17:56 |
| mordred | corvus: wow. that's even more clear | 17:57 |
| *** jpena is now known as jpena|off | 17:57 | |
| clarkb | there are None LoadingErrors therefore you have an UnknownError :) | 17:57 |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Add periodic-1w pipeline https://review.opendev.org/712473 | 17:58 |
| openstackgerrit | Merged opendev/irc-meetings master: Remove Senlin team meeting https://review.opendev.org/712450 | 17:59 |
| mordred | zbr: how about if we call it "weekly" | 17:59 |
| zbr | mordred: no real problem, two reasons for 1w: shorter (less wrap), avoid future confusion if we add one to run every other week, biweekly is known to a confusing term, as it can mean 0.5x or 2.x based on who you ask. | 18:01 |
| mordred | zbr: no it can't | 18:02 |
| zbr | https://en.wikipedia.org/wiki/Biweekly interesting read. | 18:02 |
| mordred | oh wow really? | 18:02 |
| zbr | i worked as i18n eng for few years, got enough Unicode & ..... for an entire lifetime :D | 18:02 |
| mordred | jeez. well - I consider that usage to be epically wrong. semi-weekly twice a week | 18:03 |
| mordred | zbr: but - we could use fortnightly if we wanted to have a biweekly job - it's not unclear | 18:03 |
| mordred | and semiweekly is also not unclear :) | 18:03 |
| hashar | biweekly and fortnightly are always a source of confusion when you have international folks ;) | 18:03 |
| *** priteau has quit IRC | 18:03 | |
| hashar | we changed to "twice per week" or "once every two weeks" which is a bit clearer in simple english | 18:04 |
| *** sshnaidm is now known as sshnaidm|afk | 18:04 | |
| mordred | https://en.wikipedia.org/wiki/Bimonthly | 18:04 |
| zbr | maybe we should pick relative terms, like: "periodic-rare" so we can change the timing, without touching the pipeline name. | 18:04 |
| mordred | that biweekly does not have a matching meaning in british is absurd, and I typically like the fun idiosyncracies of british vs american | 18:04 |
| mordred | I thnk this isn one time when we unambiguously got the difference correct | 18:05 |
| mordred | anyway | 18:05 |
| mordred | I agree that because british english has a bug we should avoid biweekly *headdesk* | 18:06 |
| hashar | ;]]] | 18:06 |
| mordred | zbr: I mostly don't like the less-readable-english quality of periodic-1w - I agree it is clear | 18:06 |
| *** derekh has quit IRC | 18:07 | |
| mordred | zbr: what about "once-per-week" | 18:07 |
| zbr | periodic-weekend ? | 18:07 |
| mordred | I think we shoudl rename periodic | 18:08 |
| *** eharney has joined #openstack-infra | 18:08 | |
| zbr | in fact there is nothing wrong with periodic-weekly, the question is what to do when/if we want to scale/split periodic(s) | 18:08 |
| mordred | but - I mean - this is just me talking | 18:08 |
| zbr | from the maintenance point of view, I kinda like using vague/relative terms in pipeline name, as it allows infra to tune their recurrence as needed, like lowering the load. | 18:10 |
| zbr | maybe at some point we want to run every 10 days instead of 7, or maybe I am trying to over-engineer it. | 18:10 |
| mordred | yeah - our way of tuning most things it to try to reduce the overall need for periodics anyway :) | 18:11 |
| mordred | but - I mean - it's a decent point | 18:11 |
| jrosser | periodic jobs would be hugely more valuable to me if I had a view like this of a related set https://grafana.com/grafana/plugins/flant-statusmap-panel | 18:20 |
| jrosser | currently it’s too hard to infer stuff from the results | 18:21 |
| *** hashar_ has joined #openstack-infra | 18:21 | |
| *** hashar has quit IRC | 18:21 | |
| AJaeger | corvus: want to remove your WIP from https://review.opendev.org/#/c/711257/ and approve it? That's the limestone re-enable change | 18:22 |
| *** hashar_ has quit IRC | 18:22 | |
| *** hashar has joined #openstack-infra | 18:22 | |
| *** diablo_rojo has quit IRC | 18:23 | |
| mordred | jrosser: you mean like http://zuul.opendev.org/t/openstack/buildset/93300b6263c64f1cb9f3e15466e3e4ef ? | 18:24 |
| *** diablo_rojo has joined #openstack-infra | 18:24 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Add periodic-1w pipeline https://review.opendev.org/712473 | 18:24 |
| *** dtantsur|brb is now known as dtantsur | 18:24 | |
| AJaeger | jrosser: or like http://zuul.opendev.org/t/openstack/builds?job_name=propose-translation-update&result=FAILURE ? | 18:24 |
| jrosser | mordred: if everything were in one repo yes that would be fine | 18:24 |
| mordred | jrosser: oh - you mean a map of the status of the periodic thing over time | 18:24 |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Add periodic-weekly pipeline https://review.opendev.org/712473 | 18:25 |
| jrosser | yes, across like all the osa roles plus the parent repo, which would be ~40+ rows | 18:25 |
| clarkb | openstack health provides that for oprnstack projects | 18:27 |
| AJaeger | jrosser: http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1 shows some failure rates, so we have data - if you want to propose a change, go ahead | 18:28 |
| *** lmiccini has quit IRC | 18:29 | |
| AJaeger | clarkb: for https://review.opendev.org/711106 (#opendev cleanup) what do you consider quorum? | 18:30 |
| AJaeger | Do you want mordred, fungi, ianw, Shrews, to review it as well? ^ | 18:31 |
| clarkb | ya the other active reviewers basically. That list is probably complete | 18:32 |
| AJaeger | mordred, fungi, ianw, Shrews , please review 711106 so that we can move forward with #opendev. Notice there's a followup change as well. | 18:33 |
| *** amoralej is now known as amoralej|off | 18:33 | |
| AJaeger | mordred done (thanks), three more to go ;) | 18:33 |
| *** AJaeger has quit IRC | 18:34 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: ZK TLS https://review.opendev.org/712531 | 18:34 |
| *** AJaeger has joined #openstack-infra | 18:37 | |
| *** gfidente has quit IRC | 18:38 | |
| *** tesseract has quit IRC | 18:43 | |
| mnaser | yay https://hub.docker.com/r/opendevorg/lodgeit | 19:03 |
| mnaser | clarkb: i remember you pinging about some stale vols but i cant find that in my scrollback :\ | 19:03 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Remove OpenStack specific steps from project retirement https://review.opendev.org/711818 | 19:06 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Move Translation setup to i18n document https://review.opendev.org/711827 | 19:07 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Add options to CLI info command https://review.opendev.org/712539 | 19:08 |
| AJaeger | infra-root, https://review.opendev.org/#/q/project:openstack/infra-manual+topic:opendev+is:open has three changes for OpenDev manual as discussed on #opendev - is that going in the right direction now? | 19:08 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Add options to CLI info command https://review.opendev.org/712539 | 19:09 |
| zbr | clarkb: mordred are e ok to proceed with https://review.opendev.org/#/c/712473/ ? | 19:10 |
| clarkb | zbr: I'm ok wiht it, but didn't approve as I hadn't followed the discussion about terminology too closely | 19:13 |
| clarkb | mnaser: ya I can get a link one sec | 19:13 |
| clarkb | mnaser: http://paste.openstack.org/show/790497/ lines 7 - 9 are volumes and servers that refuse to delete | 19:14 |
| clarkb | mordred: for https://review.opendev.org/#/c/712525/ shoudl we squash that down into a single chnage maybe? rather than iterating through a bunch of the options? | 19:15 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Remove OpenStack specific steps from project retirement/rename https://review.opendev.org/711818 | 19:16 |
| *** eharney has quit IRC | 19:16 | |
| *** ociuhandu has joined #openstack-infra | 19:17 | |
| clarkb | mordred: my parsing of corvus' comments are that we should do the squash. Also see note on https://review.opendev.org/#/c/712525/1 | 19:17 |
| *** adriant has quit IRC | 19:20 | |
| *** iokiwi has quit IRC | 19:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Remove OpenStack specific steps from project retirement/rename https://review.opendev.org/711818 | 19:20 |
| *** ociuhandu has quit IRC | 19:21 | |
| mordred | clarkb: yeah. let's squash | 19:26 |
| mordred | clarkb: I'll push up a squash | 19:26 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Return goaccess html as zuul artifact https://review.opendev.org/712478 | 19:28 |
| openstackgerrit | Merged openstack/project-config master: Add periodic-weekly pipeline https://review.opendev.org/712473 | 19:28 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Don't access parent layout errors if there is no parent layout https://review.opendev.org/712544 | 19:32 |
| clarkb | mordred: +2 thanks | 19:33 |
| *** adriant has joined #openstack-infra | 19:39 | |
| *** rpittau is now known as rpittau|afk | 19:40 | |
| *** dtantsur is now known as dtantsur|afk | 19:40 | |
| clarkb | mordred: AJaeger we may even want to use periodic-weekly for these goaccess jobs because we typically produce a report with at least a week of data | 19:41 |
| AJaeger | clarkb: fine with me | 19:41 |
| openstackgerrit | Monty Taylor proposed openstack/openstack-zuul-jobs master: Use a fake zuul_return and an .ansible-lint file https://review.opendev.org/712546 | 19:42 |
| mordred | AJaeger: ^^ | 19:42 |
| *** ralonsoh has quit IRC | 19:44 | |
| AJaeger | cool | 19:45 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Use a fake zuul_return and an .ansible-lint file https://review.opendev.org/712547 | 19:46 |
| openstackgerrit | Monty Taylor proposed openstack/openstack-zuul-jobs master: Use a fake zuul_return and an .ansible-lint file https://review.opendev.org/712546 | 19:46 |
| *** jamesmcarthur has quit IRC | 19:48 | |
| *** hashar has quit IRC | 19:48 | |
| *** jamesmcarthur has joined #openstack-infra | 19:49 | |
| *** jamesmcarthur has quit IRC | 19:49 | |
| AJaeger | mordred: https://review.opendev.org/712547 is in merge-conflict | 19:49 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Use a fake zuul_return and an .ansible-lint file https://review.opendev.org/712547 | 19:51 |
| *** ociuhandu has joined #openstack-infra | 19:52 | |
| *** slaweq has quit IRC | 19:53 | |
| mordred | AJaeger: let's give that a shot - I'm not sure about the bindep contents - but I'm now not convinced we need any of them? | 19:53 |
| AJaeger | mordred: I agree, we should not need it - let's see... | 19:54 |
| AJaeger | but what's up with 547? | 19:54 |
| mordred | what do you mean? | 19:55 |
| mordred | oh - 546 | 19:56 |
| AJaeger | it fails to freeze the job -graph. Found the problem: The job has base-minimal as parent | 19:56 |
| AJaeger | sorry, 546 indeed | 19:56 |
| *** ociuhandu has quit IRC | 19:56 | |
| mordred | hrm | 19:56 |
| AJaeger | and base-minimal has a allowed-projects: zuul/zuul-jobs | 19:56 |
| AJaeger | defined in opendev/base-jobs | 19:56 |
| *** ociuhandu has joined #openstack-infra | 19:56 | |
| AJaeger | So, what's wrong - the parent base-minimal or the allowed-projects? | 19:56 |
| mordred | that's a GREAT question | 19:57 |
| mordred | corvus, clarkb: ^^ ? I do not know the answer to that question | 19:57 |
| *** jamesmcarthur has joined #openstack-infra | 19:59 | |
| clarkb | mordred: AJaeger the change doesn't update the job in question though? | 20:00 |
| clarkb | I feel like I'm missing an additional piece of info here | 20:00 |
| mordred | clarkb: we made a change to opendev/base-jobs | 20:00 |
| mordred | which in turn made new changes to openstack-zuul-jobs break - but we didn't notice | 20:00 |
| mordred | this change, which is unrelated, is hitting that | 20:00 |
| clarkb | gotcha | 20:01 |
| clarkb | "We can set allowed-projects so that only zuul-jobs can use it in order to avoid accidental use and confusion." | 20:02 |
| clarkb | we aren't protecting secrets or the like at least with that | 20:02 |
| mordred | nod. so it seems like ozj might also be reasonable (because we want to be able to test that mirror-info thing) | 20:03 |
| clarkb | ya | 20:03 |
| clarkb | I think we can update the allowed-projects list to include ozj | 20:03 |
| mordred | k. patch coming | 20:03 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Allow ozj to use base-minimal https://review.opendev.org/712548 | 20:05 |
| *** ociuhandu has quit IRC | 20:06 | |
| *** ociuhandu has joined #openstack-infra | 20:09 | |
| corvus | mordred: actually, i think we need to remove mirror-info from ozj | 20:10 |
| corvus | it's in opendev/base-jobs | 20:10 |
| corvus | (though we don't have a test for it there... probably because, well, it's base-jobs) | 20:11 |
| clarkb | corvus: yup I think we just got to that conclusion our selves in #opendev | 20:11 |
| corvus | (but that's the one we're using, so meh) | 20:11 |
| clarkb | (not sure why the discussion shifted) | 20:11 |
| corvus | it's um hard to follow | 20:12 |
| openstackgerrit | Monty Taylor proposed openstack/openstack-zuul-jobs master: Use a fake zuul_return and an .ansible-lint file https://review.opendev.org/712546 | 20:13 |
| openstackgerrit | Monty Taylor proposed openstack/openstack-zuul-jobs master: Remove mirror-info role and tests https://review.opendev.org/712550 | 20:13 |
| AJaeger | corvus: could you remove your WIP from https://review.opendev.org/#/c/711257/ , please? | 20:13 |
| clarkb | infra-root airship is apparently still struggling to boot nodes in citycloud. I've pulled out a subset of logs showing its the same cloud side no valid host found error and forwarded that on. I half expect the next suggestion to be using a different cloud region, but want them to decide that rather than me/us | 20:13 |
| corvus | AJaeger: +3 thx | 20:13 |
| AJaeger | thanks | 20:14 |
| *** ociuhandu has quit IRC | 20:14 | |
| clarkb | corvus: mordred squashed the zuul_return simplification down into my change to use zuul_return in system-config if you want to rereview that https://review.opendev.org/#/c/712478/4 | 20:15 |
| fungi | you folks were sure talkative while i was slacking off with lunch errands | 20:15 |
| fungi | but i think i'm caught up now | 20:15 |
| corvus | clarkb: +2 | 20:16 |
| corvus | clarkb: is that ready to +W? | 20:16 |
| clarkb | yup I +A'd it. Thank you! | 20:16 |
| clarkb | I'll manually trigger it again via enqueue-ref just to double check it produces the expected result | 20:16 |
| openstackgerrit | Jeremy Stanley proposed zuul/nodepool master: Declare support for Python3.8 https://review.opendev.org/712494 | 20:22 |
| AJaeger | clarkb: we now have +2 on 711106 from everybody besides ianw, shall I merge if he's fine? | 20:22 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Declare support for Python3.8 https://review.opendev.org/712489 | 20:22 |
| AJaeger | clarkb: or want to schedule merging that one - and until one? | 20:22 |
| clarkb | AJaeger: ya I think maybe we can schedule it now that we have most everyone on board. Then merge it at that scheduled time | 20:23 |
| clarkb | (I dunno whats good, say starting monday we'll try to split conversations properly?) | 20:23 |
| AJaeger | Or tuesday after infra meeting? - either works for me... | 20:23 |
| clarkb | AJaeger: actually I like that because its a good point in time where we are all largely around and can be reminded | 20:24 |
| clarkb | rather than monday which is a bit arbitrary depending on timezones :) | 20:24 |
| openstackgerrit | Merged openstack/project-config master: Revert "Disable limestone" https://review.opendev.org/711257 | 20:25 |
| openstackgerrit | Merged openstack/infra-manual master: Update Feature branches, releases sections for OpenStack https://review.opendev.org/711817 | 20:26 |
| AJaeger | clarkb: I'll put it in the agenda ;) | 20:26 |
| openstackgerrit | Merged openstack/infra-manual master: Remove OpenStack specific steps from project retirement/rename https://review.opendev.org/711818 | 20:29 |
| openstackgerrit | Merged openstack/infra-manual master: Move Translation setup to i18n document https://review.opendev.org/711827 | 20:29 |
| fungi | i for one welcome our new multi-channel overloads | 20:29 |
| *** yamamoto has joined #openstack-infra | 20:33 | |
| *** eharney has joined #openstack-infra | 20:33 | |
| *** yamamoto has quit IRC | 20:38 | |
| *** jcoufal has quit IRC | 20:39 | |
| clarkb | I'm going to pop out for a bike ride, but will rerun the goaccess job when back to ensure it is happy before it runs at 0600UTC tomorrow | 20:39 |
| clarkb | assuming it works I'll get a change up to add similar reporting for the other static hosted sites | 20:40 |
| openstackgerrit | Merged opendev/system-config master: Return goaccess html as zuul artifact https://review.opendev.org/712478 | 20:40 |
| ianw | clarkb: i see in that artifact change you switched to the config file; are we sure that it's actually checking anything? https://zuul.opendev.org/t/openstack/build/9ac1e62b035b4651be8e099351d3fce4/log/job-output.txt#619 | 20:44 |
| ianw | clarkb: i'm reminded of https://review.opendev.org/#/c/702575/ where the matching wasn't quite working | 20:45 |
| clarkb | ianw: hrm that came from mordred stack that he squashed down into my change | 20:46 |
| clarkb | it does spend non zero time there but is definitely quiet | 20:46 |
| clarkb | mordred: ^ fyi | 20:46 |
| mordred | we could maybe remove the quiet: line? | 20:50 |
| * mordred plays around some | 20:51 | |
| ianw | mordred: yeah just running locally with that now | 20:51 |
| mordred | ianw: cool | 20:51 |
| mordred | also - I that that our ansible-lint excludes are so different between system-config, zj and ozj | 20:51 |
| mordred | ianw: I think we should remove the playbooks roles arguments | 20:53 |
| mordred | ianw: I just tried both ways - and it is obviously doing stuff (and properly fails) without them | 20:53 |
| ianw | mordred: hrm, without quiet i just got a weird nothingness : http://paste.openstack.org/show/790565/ | 20:54 |
| ianw | ansible-lint 4.2.0 ? | 20:55 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove ansible-lint arguments https://review.opendev.org/712554 | 20:55 |
| mordred | ianw: yeah | 20:55 |
| ianw | ok yeah without the arguments it's doing stuff | 20:56 |
| ianw | fan is running on laptop, that's more what i expected :) | 20:56 |
| ianw | Examining playbooks/roles/letsencrypt-create-certs/tasks/roles/letsencrypt-create-certs/handlers/touch_file.yaml of type playbook | 20:58 |
| ianw | WARNING: Couldn't open /home/iwienand/programs/openstack-infra/system-config/playbooks/roles/letsencrypt-create-certs/tasks/roles/letsencrypt-create-certs/handlers/touch_file.yaml - No such file or directory | 20:58 |
| ianw | that path looks a bit mashed up | 20:58 |
| mordred | yeah | 21:03 |
| *** bdodd has quit IRC | 21:07 | |
| ianw | looks like it does the same in the gate | 21:10 |
| ianw | mordred: in fact, i think it's still doing the same things as that other change -> 2020-03-11 21:06:35.207335 | ubuntu-bionic | Unknown file type: roles/exim/tasks/main.yaml | 21:11 |
| ianw | that should really be linted | 21:12 |
| *** ijw has quit IRC | 21:18 | |
| *** ijw has joined #openstack-infra | 21:21 | |
| mordred | ianw: so maybe we should go back to find - but we could still potentially keep the config for exclusions in there | 21:21 |
| *** ijw_ has joined #openstack-infra | 21:24 | |
| *** ijw has quit IRC | 21:25 | |
| *** rcernin has joined #openstack-infra | 21:28 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Go back to using find with ansible-lint https://review.opendev.org/712554 | 21:30 |
| mordred | ianw: ^^ I think that should be better | 21:31 |
| ianw | cool, thanks. this was pretty much why bashate got out of the file matching business -- better to just let the user feed in a list of files than come up with increasing levels of magic that never really works trying to detect everything | 21:33 |
| openstackgerrit | Monty Taylor proposed openstack/openstack-zuul-jobs master: Use a fake zuul_return and an .ansible-lint file https://review.opendev.org/712546 | 21:38 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Use a fake zuul_return and an .ansible-lint file https://review.opendev.org/712547 | 21:40 |
| openstackgerrit | Monty Taylor proposed openstack/openstack-zuul-jobs master: Use a fake zuul_return and an .ansible-lint file https://review.opendev.org/712546 | 21:40 |
| *** xek has quit IRC | 21:42 | |
| *** sreejithp has quit IRC | 22:01 | |
| *** ociuhandu has joined #openstack-infra | 22:04 | |
| *** dave-mccowan has quit IRC | 22:05 | |
| clarkb | I've triggered the periodic job for zuul website stats manually | 22:08 |
| clarkb | hopefully that gives us a shiny artifact link and then we can apply this more globally | 22:08 |
| *** dave-mccowan has joined #openstack-infra | 22:08 | |
| *** dpawlik has quit IRC | 22:11 | |
| *** rlandy is now known as rlandy|bbl | 22:12 | |
| clarkb | http://zuul.openstack.org/build/18c85ad49ca5491291e7a8c2dc703fc5 has a goaccess report under artifacts | 22:12 |
| mordred | clarkb: I agree! | 22:14 |
| ianw | nice! :) | 22:19 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove old 404 checker job https://review.opendev.org/712575 | 22:23 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add goaccess reporting to more static.o.o sites https://review.opendev.org/712576 | 22:23 |
| clarkb | That cleans up the now replaced 404 checker and then adds a bunch of additional sites including docs.openstack.org so AJaeger can fix 404s :) | 22:24 |
| ianw | clarkb: did you forget to add it to the periodic: section? | 22:29 |
| clarkb | ianw: I did! | 22:29 |
| clarkb | one moment | 22:29 |
| ianw | clarkb: you might also like to update the file referenced in https://review.opendev.org/#/c/709952/ ... that was afaik the only place we talked about 404 lists. although adding something somewhere else might help too | 22:31 |
| *** jdelaros1 has quit IRC | 22:31 | |
| *** Shrews has quit IRC | 22:31 | |
| *** nickv1985 has quit IRC | 22:31 | |
| *** sdoran has quit IRC | 22:31 | |
| *** adriancz has quit IRC | 22:31 | |
| *** jrosser has quit IRC | 22:31 | |
| *** dustinc has quit IRC | 22:31 | |
| *** portdirect has quit IRC | 22:31 | |
| *** larainema has quit IRC | 22:31 | |
| *** mwhahaha has quit IRC | 22:31 | |
| *** philroche has quit IRC | 22:31 | |
| *** mrmartin has quit IRC | 22:31 | |
| *** jbryce has quit IRC | 22:31 | |
| *** rm_work has quit IRC | 22:31 | |
| *** jberg-dev has quit IRC | 22:31 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add goaccess reporting to more static.o.o sites https://review.opendev.org/712576 | 22:33 |
| clarkb | ianw: thanks, I'll get a change up for that too | 22:33 |
| *** rm_work has joined #openstack-infra | 22:33 | |
| *** philroche has joined #openstack-infra | 22:33 | |
| *** mwhahaha has joined #openstack-infra | 22:33 | |
| *** jbryce has joined #openstack-infra | 22:34 | |
| *** portdirect has joined #openstack-infra | 22:34 | |
| *** jamesmcarthur has quit IRC | 22:34 | |
| *** mrmartin has joined #openstack-infra | 22:34 | |
| *** sdoran has joined #openstack-infra | 22:34 | |
| *** jberg-dev has joined #openstack-infra | 22:35 | |
| *** nickv1985 has joined #openstack-infra | 22:36 | |
| *** jrosser has joined #openstack-infra | 22:36 | |
| *** guilhermesp has quit IRC | 22:36 | |
| *** clayg has quit IRC | 22:36 | |
| *** jungleboyj has quit IRC | 22:36 | |
| *** diablo_rojo_phon has quit IRC | 22:36 | |
| *** npochet has quit IRC | 22:36 | |
| *** thedac has quit IRC | 22:36 | |
| *** lsell has quit IRC | 22:36 | |
| *** mnaser has quit IRC | 22:36 | |
| *** gmann has quit IRC | 22:36 | |
| *** lseki has quit IRC | 22:36 | |
| *** donnyd has quit IRC | 22:36 | |
| *** beisner has quit IRC | 22:36 | |
| *** dougwig has quit IRC | 22:36 | |
| *** davidlenwell has quit IRC | 22:36 | |
| *** crodriguez has quit IRC | 22:36 | |
| *** hrybacki has quit IRC | 22:36 | |
| *** cyberpear has quit IRC | 22:36 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: ZK TLS https://review.opendev.org/712531 | 22:37 |
| *** larainema has joined #openstack-infra | 22:38 | |
| *** dougwig has joined #openstack-infra | 22:38 | |
| clarkb | ianw: AJaeger remote: https://review.opendev.org/712579 Link to goaccess report for docs.o.o 404s | 22:39 |
| *** davidlenwell has joined #openstack-infra | 22:39 | |
| *** adriancz has joined #openstack-infra | 22:39 | |
| *** hrybacki has joined #openstack-infra | 22:39 | |
| *** mnaser has joined #openstack-infra | 22:39 | |
| *** jungleboyj has joined #openstack-infra | 22:40 | |
| *** dustinc has joined #openstack-infra | 22:40 | |
| *** Shrews has joined #openstack-infra | 22:40 | |
| *** jdelaros1 has joined #openstack-infra | 22:40 | |
| *** lsell has joined #openstack-infra | 22:40 | |
| *** clayg has joined #openstack-infra | 22:40 | |
| *** beisner has joined #openstack-infra | 22:40 | |
| *** gmann has joined #openstack-infra | 22:41 | |
| *** guilhermesp has joined #openstack-infra | 22:41 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: ZK TLS https://review.opendev.org/712531 | 22:41 |
| *** thedac has joined #openstack-infra | 22:41 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: ZK TLS https://review.opendev.org/712531 | 22:41 |
| *** ociuhandu has quit IRC | 22:42 | |
| openstackgerrit | Merged opendev/system-config master: Go back to using find with ansible-lint https://review.opendev.org/712554 | 22:42 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: ZK TLS https://review.opendev.org/712531 | 22:43 |
| *** lseki has joined #openstack-infra | 22:43 | |
| *** ociuhandu has joined #openstack-infra | 22:43 | |
| *** npochet has joined #openstack-infra | 22:44 | |
| *** donnyd has joined #openstack-infra | 22:44 | |
| *** crodriguez has joined #openstack-infra | 22:44 | |
| *** cyberpear has joined #openstack-infra | 22:45 | |
| *** tdasilva has quit IRC | 22:46 | |
| *** rajinir has quit IRC | 22:46 | |
| *** evgenyl has quit IRC | 22:46 | |
| *** andrewbonney has quit IRC | 22:46 | |
| *** arne_wiebalck has quit IRC | 22:46 | |
| *** irclogbot_1 has quit IRC | 22:47 | |
| *** evgenyl has joined #openstack-infra | 22:47 | |
| *** irclogbot_2 has joined #openstack-infra | 22:47 | |
| *** ociuhandu has quit IRC | 22:48 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: ZK TLS https://review.opendev.org/712531 | 22:48 |
| *** openstackstatus has quit IRC | 22:48 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: ZK TLS https://review.opendev.org/712531 | 22:49 |
| *** zbr has quit IRC | 22:50 | |
| *** gagehugo has quit IRC | 22:50 | |
| *** seongsoocho has quit IRC | 22:50 | |
| *** Ng has quit IRC | 22:50 | |
| *** mnasiadka has quit IRC | 22:50 | |
| *** petevg has quit IRC | 22:50 | |
| *** coreycb has quit IRC | 22:50 | |
| *** rajinir has joined #openstack-infra | 22:50 | |
| *** arne_wiebalck has joined #openstack-infra | 22:51 | |
| *** tdasilva has joined #openstack-infra | 22:51 | |
| *** Ng has joined #openstack-infra | 22:52 | |
| *** andrewbonney has joined #openstack-infra | 22:52 | |
| *** rh-jelabarre has quit IRC | 22:53 | |
| *** petevg has joined #openstack-infra | 22:53 | |
| *** seongsoocho has joined #openstack-infra | 22:54 | |
| *** coreycb has joined #openstack-infra | 22:54 | |
| *** mnasiadka has joined #openstack-infra | 22:54 | |
| *** zbr has joined #openstack-infra | 22:55 | |
| *** gagehugo has joined #openstack-infra | 22:55 | |
| *** ildikov has quit IRC | 22:56 | |
| *** tonyb[m] has quit IRC | 22:56 | |
| *** howell has quit IRC | 22:56 | |
| *** dannins has quit IRC | 22:56 | |
| *** srwilkers has quit IRC | 22:56 | |
| *** rpioso has quit IRC | 22:56 | |
| *** davecore has quit IRC | 22:56 | |
| *** knikolla has quit IRC | 22:56 | |
| *** lxkong has quit IRC | 22:56 | |
| *** csatari has quit IRC | 22:56 | |
| *** cjohnston has quit IRC | 22:56 | |
| clarkb | ianw: now that I've flushed my queue on goaccess stuff is there anything I can help with the image modifications you are trying to do? I know you mentioned it boots now? (that implies glean is happy in a venv?) and next its a matter of testing the transition stuff? | 22:58 |
| *** ildikov has joined #openstack-infra | 22:59 | |
| *** cjohnston has joined #openstack-infra | 22:59 | |
| *** davecore has joined #openstack-infra | 23:00 | |
| *** csatari has joined #openstack-infra | 23:00 | |
| *** srwilkers has joined #openstack-infra | 23:01 | |
| *** knikolla has joined #openstack-infra | 23:01 | |
| *** tonyb[m] has joined #openstack-infra | 23:01 | |
| *** tkajinam has joined #openstack-infra | 23:01 | |
| *** howell has joined #openstack-infra | 23:02 | |
| *** lxkong has joined #openstack-infra | 23:02 | |
| *** rpioso has joined #openstack-infra | 23:03 | |
| *** ianychoi has joined #openstack-infra | 23:06 | |
| ianw | clarkb: yeah just starting to look at the base job setup | 23:06 |
| ianw | problem 0 : https://zuul.opendev.org/t/openstack/build/636f2e9212714c908dfba44dc1a7bb6b/log/job-output.txt#1032 | 23:07 |
| *** dave-mccowan has quit IRC | 23:08 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Configure statusbot to join #opendev https://review.opendev.org/712582 | 23:09 |
| ianw | starting to rewrite that to prefer venv | 23:09 |
| clarkb | ianw: https://review.opendev.org/#/c/712107/ is another change we'll want to land before switching | 23:09 |
| clarkb | ianw: that removes the global cleanup/fix of virtualenv | 23:09 |
| clarkb | (and if virtualenv is gone globally it will just be noise) | 23:09 |
| *** dave-mccowan has joined #openstack-infra | 23:10 | |
| clarkb | ianw: you might be able to reuse the ansible venv since ansible deps on pyyaml | 23:10 |
| clarkb | ianw: will reduce the number of time swe install pyyaml in the d-g jobs | 23:10 |
| ianw | clarkb: what's a bit weird though is that i don't understand why set get that pkg_resources error | 23:13 |
| ianw | what exactly did it try and run? | 23:14 |
| clarkb | ianw: based on the traceback some ansible pip module code | 23:14 |
| clarkb | I don't think it got far enough to run pip itself, but the ansible code is assuming pkg_resources (setuptools) exists | 23:14 |
| *** Goneri has quit IRC | 23:15 | |
| ianw | hrm, yeah i guess it could be in the pip module -- i'm worndring if that came from inside a virtualenv call or something though | 23:15 |
| ianw | from pkg_resources import Requirement | 23:17 |
| ianw | that's in the ansible pip role, so yeah | 23:17 |
| ianw | "The interpreter used by Ansible (see ansible_python_interpreter) requires the setuptools package, regardless of the version of pip set with the executable option." | 23:18 |
| clarkb | hrm | 23:18 |
| ianw | i guess we dont' have that package installed | 23:18 |
| clarkb | I think that means part of the migration is $job will need a step that install setuptools to their preferred version before using the pip module | 23:19 |
| clarkb | ianw: ya I think we are intentionally avoiding instaling it because its one of the things that people may want to control | 23:19 |
| clarkb | (however considering our integration with ansible we may have to rethink setuptools as a specific exception) | 23:19 |
| lifeless | highly recommend rethinking setuptools | 23:21 |
| clarkb | lifeless: I mean we don't really control ansible's pip module | 23:21 |
| lifeless | ... but not in the way you mean ;) | 23:21 |
| clarkb | lifeless: and yes, the current thing we are testing is removing the assumption that tools like setuptools exist systemwide | 23:23 |
| clarkb | then let jobs bootstrap that themselves as necessary (if that is what they want) | 23:23 |
| lifeless | folk in pip land have this idyllic view again and again that perhaps setuptools won't be needed ;) | 23:24 |
| mordred | lifeless: folk in pip land have many different idyllic views again and again | 23:25 |
| ianw | clarkb: the thing is, https://opendev.org/openstack/devstack-gate/src/branch/master/roles/test-matrix/tasks/main.yaml#L6 the check for virtualenv passed, but then the pip install failed. how did that happen i wonder | 23:26 |
| fungi | lifeless: yeah, i brought up flit and pep 516/517/518 again yesterday the situation down in the trenches dealing with existing packages hasn't changed much (yet at least) | 23:26 |
| clarkb | ianw: hrm something else pulling it in at image build time maybe (as a dep?) | 23:27 |
| *** ahosam has quit IRC | 23:27 | |
| ianw | clarkb: hrm, not as far as i can seee : https://nb02.openstack.org/ubuntu-bionic-plain-0000000007.log | 23:28 |
| clarkb | ianw: I agree a quick search for virtualenv only returns a git repo name, not package installation | 23:29 |
| clarkb | (and apt is good about tleling you all the things it iwll install) | 23:29 |
| ianw | got a node building, will login and see | 23:31 |
| *** dchen has joined #openstack-infra | 23:31 | |
| ianw | i guess this doesn't depend on setuptools? https://packages.ubuntu.com/bionic/python3-virtualenv | 23:32 |
| clarkb | looks like distutils not setuptools | 23:33 |
| *** ijw_ has quit IRC | 23:36 | |
| ianw | so, virtualenv wasn't on the node to start, something must have brought it in | 23:36 |
| clarkb | oh do we pull it in for the ansible virtualenv? | 23:37 |
| clarkb | I thought that happened later because its in the d-g bash scripts, but I oculd be wrong becuse this is nested ansible possibly? | 23:37 |
| ianw | hang on .... i think the test is borked @ https://zuul.opendev.org/t/openstack/build/636f2e9212714c908dfba44dc1a7bb6b/console#2/0/4/controller | 23:40 |
| ianw | failed_when: false ? | 23:40 |
| clarkb | ya maybe virtualenv_available | failed checks the failed_when attribute and not rc value | 23:41 |
| clarkb | ianw: perhaps that check should be virtualenv_available.rc != 0 | 23:41 |
| *** Martouf has joined #openstack-infra | 23:43 | |
| *** tosky has quit IRC | 23:44 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Add OpenInfra Labs IRC channels to bots https://review.opendev.org/712586 | 23:45 |
| openstackgerrit | Ian Wienand proposed openstack/devstack-gate master: Fix virtualenv detection https://review.opendev.org/712587 | 23:45 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add IRC bots to OpenInfra Labs channels https://review.opendev.org/712588 | 23:45 |
| ianw | clarkb: ^^ that would make some sense; i guess really if you want to use "pip:" you have to manually ensure you have dependencies | 23:47 |
| *** eharney has quit IRC | 23:47 | |
| *** yamamoto has joined #openstack-infra | 23:47 | |
| ianw | making sure that bails with "no virtualenv" error now | 23:48 |
| openstackgerrit | Merged opendev/system-config master: Add statusbot to #opendev https://review.opendev.org/711245 | 23:50 |
| *** KeithMnemonic has quit IRC | 23:51 | |
| *** KeithMnemonic has joined #openstack-infra | 23:51 | |
| openstackgerrit | Merged zuul/nodepool master: Install zypper on the nodepool-builder image https://review.opendev.org/712177 | 23:53 |
| *** yamamoto has quit IRC | 23:54 | |
| *** ijw has joined #openstack-infra | 23:54 | |
| ianw | hrm, since d-g is branchless i guess it needs to cover python2 still | 23:56 |
| *** prometheanfire has quit IRC | 23:57 | |
| clarkb | ianw: yes, but python3 virtualenv can install a python2 venv | 23:57 |
| clarkb | I think that means on the 'infra' side we can be python3 only if it is easier | 23:57 |
| ianw | yeah, trying to keep virtualenv out of it if possible and use venv | 23:59 |
| clarkb | ah ya that can't make a python2 venv | 23:59 |
| *** ijw has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!