| *** agopi|out has quit IRC | 00:00 | |
| clarkb | journalctl -u kube-apiserver has no entries | 00:01 |
|---|---|---|
| clarkb | -u kubelet has one entry | 00:01 |
| *** jamesmcarthur has quit IRC | 00:03 | |
| corvus | okay, i guess we don't get api request logs | 00:04 |
| *** markvoelker has quit IRC | 00:05 | |
| fungi | servers with logs are for grey-bearded old fogies. the hipster way to manage services is to just redeploy your containers over and over until some fogey with logs fixes things upstream | 00:06 |
| *** smarcet has joined #openstack-infra | 00:07 | |
| *** rh-jelabarre has quit IRC | 00:07 | |
| corvus | i'm giving serious thought to going back to the shared-nothing gitea idea... | 00:08 |
| corvus | when setting this up, i ran a lot of kubectl commands by hand which should be equivalent to this | 00:08 |
| corvus | and they never failed | 00:08 |
| openstackgerrit | Merged openstack-infra/zuul master: web: remove build and job_name filter from the buildset route https://review.openstack.org/636504 | 00:09 |
| corvus | i ran lsof on the ansible process, it has no open network connections. | 00:09 |
| *** rascasoft has joined #openstack-infra | 00:09 | |
| clarkb | fwiw my reading of the module on the version of ansible we run is that it won't wait for anything | 00:09 |
| clarkb | and wait is off by default in 2.8 (and we don't enable it) | 00:09 |
| clarkb | corvus: fwiw the kubectl commands are written in one implementation in one language and the k8s ansible module an entirely different language by a third party | 00:10 |
| *** jamesmcarthur_ has joined #openstack-infra | 00:10 | |
| corvus | yep -- my implication is that i lean toward the problem being ansible | 00:10 |
| *** hwoarang has quit IRC | 00:11 | |
| *** weshay|ruck has quit IRC | 00:11 | |
| clarkb | corvus: maybe ansible -vvvvv will help narrow down where the problem is? | 00:12 |
| *** jamesmcarthur_ has quit IRC | 00:12 | |
| corvus | i'd love to get a stacktrace now :/ | 00:13 |
| *** rlandy is now known as rlandy|bbl | 00:13 | |
| *** hwoarang has joined #openstack-infra | 00:13 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Add Fake Github Review object to test suite https://review.openstack.org/636788 | 00:15 |
| *** rascasoft has quit IRC | 00:16 | |
| clarkb | corvus: another approach might be to run an out of band k8s module change? though if it is specific to the resource types being created that may be tricky to narrow down from | 00:17 |
| *** jamesmcarthur has joined #openstack-infra | 00:17 | |
| clarkb | rereadign that select the first arg is 0 and the other args are NULL so it isn't really selecting anything | 00:18 |
| corvus | clarkb: http://paste.openstack.org/show/745046/ these are our stack traces | 00:18 |
| clarkb | whcih lines up with your no network connections observation | 00:18 |
| corvus | proc 13560 has 2 threads | 00:18 |
| *** jamesmcarthur has quit IRC | 00:19 | |
| corvus | oh wait should i be looking at 13759 ? | 00:19 |
| *** jamesmcarthur has joined #openstack-infra | 00:20 | |
| corvus | it also has no network connections | 00:20 |
| clarkb | 13759 is the "remote" side of the local connection I think | 00:20 |
| corvus | strace says restart_syscall(<... resuming interrupted poll ...> | 00:21 |
| corvus | it has 1 thread: http://paste.openstack.org/show/745047/ | 00:21 |
| *** jamesmcarthur has quit IRC | 00:22 | |
| corvus | clarkb: oh, note these stacktraces are backwards (most recent call first) | 00:22 |
| corvus | oh *that* process has 13762 as a child | 00:24 |
| corvus | aha! python3 13762 root 8u IPv4 4241073484 0t0 TCP bridge.openstack.org:42764->38.108.68.20:6443 (ESTABLISHED) | 00:24 |
| fungi | 6443 is the api? | 00:24 |
| *** mattw4 has quit IRC | 00:25 | |
| clarkb | ah ok 13762 is the remote local side | 00:25 |
| corvus | fungi: i don't know, but i assume so for now | 00:25 |
| corvus | strace says futex(0x7f4764000e70, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff | 00:25 |
| corvus | 1 thread, traceback is: http://paste.openstack.org/show/745048/ | 00:26 |
| *** markvoelker has joined #openstack-infra | 00:26 | |
| clarkb | tcp6 0 0 :::6443 :::* LISTEN 1680/kube-apiserver | 00:27 |
| corvus | it is... disheartening... to see a deadlock on a threadpool in a process with a single thread. | 00:29 |
| clarkb | I can curl that from my desktop (I get a json resposne saying I am not authenticating) | 00:29 |
| *** mriedem has quit IRC | 00:29 | |
| clarkb | corvus: https://github.com/kubernetes-client/python/blob/v8.0.1/kubernetes/client/api_client.py#L76 is the code that is blocking | 00:30 |
| *** jamesmcarthur has joined #openstack-infra | 00:31 | |
| corvus | yep | 00:31 |
| clarkb | so its made its requests and is now trying to clean up the request thread(s) | 00:31 |
| clarkb | and that is deadlocked. Fun | 00:31 |
| corvus | but there's only one thread running according to gdb | 00:31 |
| clarkb | and the client itself is generated code so there isn't a useful history in the log | 00:32 |
| clarkb | https://github.com/kubernetes-client/python/blob/master/kubernetes/client/api_client.py#L78 new version is slightly different though | 00:32 |
| *** jamesmcarthur has quit IRC | 00:33 | |
| clarkb | corvus: maybe try upgrading the kubernetes lib? | 00:34 |
| clarkb | there is only a beta for 9.0.0 though | 00:34 |
| corvus | clarkb: i'm trying to figure out if ansible k8s calls call_api with async true or false | 00:34 |
| corvus | i haven't found the call path yet | 00:35 |
| clarkb | ++ | 00:35 |
| corvus | i agree, if it calls it with false, upgrading it may help | 00:35 |
| clarkb | corvus: https://github.com/openshift/openshift-restclient-python/blob/master/openshift/dynamic/client.py#L268 is where that happens via openshift | 00:35 |
| clarkb | and I don't see anywhere on the ansible side setting async_req (though my search is via github search which may not be very exact) | 00:37 |
| corvus | yeah, i'm not finding it either | 00:38 |
| corvus | so maybe the upgrade will help | 00:38 |
| corvus | #status log manually ran "pip3 install kubernetes==9.0.0b1" on bridge to see if newer version avoids deadlock on k8s api calls | 00:41 |
| openstackstatus | corvus: finished logging | 00:41 |
| corvus | i'm going to kill that process now | 00:41 |
| corvus | clarkb: thanks for your help! i'm pleasantly surprised how far we were able to actually chase that down | 00:42 |
| corvus | and if this doesn't work, we'll just figure out how to run 'kubectl' for all of these | 00:43 |
| corvus | (the nice thing about the k8s module is the free jinja2 templating) | 00:43 |
| corvus | but i'm sure we can do something with stdin | 00:43 |
| clarkb | ya should be workable to figure out kubectl commands | 00:43 |
| mordred | corvus: if we have to kubectl, clint has a setup for templating he's fond of | 00:43 |
| mordred | but fingers crossed that the upgrade just works | 00:44 |
| fungi | i bet in involves gearman ;) | 00:44 |
| * mordred waves from the supercharger in alexandria | 00:44 | |
| corvus | mordred: do not lick the supercharger | 00:44 |
| fungi | electrifying! | 00:44 |
| mordred | corvus: too late | 00:44 |
| fungi | you're just around the corner from me | 00:45 |
| mordred | fungi: LA not VA | 00:45 |
| fungi | unless you mean egypt | 00:45 |
| fungi | oh, or there | 00:45 |
| mordred | :) | 00:45 |
| mordred | egypt would also be cool | 00:45 |
| fungi | i bet it still has a decent library? | 00:45 |
| mordred | fungi: I'm still closer to you than if I was in egypt though | 00:45 |
| corvus | you never know with mordred | 00:45 |
| fungi | indeed | 00:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Update gitreview file with correct project name https://review.openstack.org/636791 | 00:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Add perf testing framework https://review.openstack.org/636792 | 00:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Reimplement in Rust https://review.openstack.org/636793 | 00:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Remove C++ version https://review.openstack.org/636794 | 00:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Use rust:slim base image https://review.openstack.org/636795 | 00:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Use slice matching for hostname unpacking https://review.openstack.org/636796 | 00:46 |
| fungi | he is in all alexandrias simultaneously, until you collapse his wave function | 00:47 |
| * mordred bombs people with patches | 00:47 | |
| mordred | fungi: ++ | 00:47 |
| mordred | corvus: feel free to do whatever is useful with any, all or none of those | 00:48 |
| clarkb | I was going to say virginia is wrong direction | 00:48 |
| *** jamesmcarthur has joined #openstack-infra | 00:49 | |
| mordred | clarkb: virginia was the right direction when duke beat them this weekend :) | 00:49 |
| *** jamesmcarthur has quit IRC | 00:51 | |
| *** betherly has joined #openstack-infra | 00:53 | |
| pabelanger | clarkb: mgagne: looks like inap might be having some problems: http://grafana.openstack.org/dashboard/db/nodepool-inap | 00:54 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Add Fake Github Review object to test suite https://review.openstack.org/636788 | 00:54 |
| pabelanger | happen to notice it when looking at grafana | 00:54 |
| mgagne | pabelanger: thanks for the info, I'm currently busy debugging something else. I will look into it asap. | 00:54 |
| pabelanger | np! mostly an FYI | 00:55 |
| *** betherly has quit IRC | 00:58 | |
| *** wolverineav has quit IRC | 01:00 | |
| *** hwoarang has quit IRC | 01:11 | |
| *** jamesmcarthur has joined #openstack-infra | 01:12 | |
| *** hwoarang has joined #openstack-infra | 01:13 | |
| *** smarcet has quit IRC | 01:13 | |
| *** gyee has quit IRC | 01:15 | |
| *** smarcet has joined #openstack-infra | 01:15 | |
| *** jamesmcarthur has quit IRC | 01:17 | |
| *** sthussey has quit IRC | 01:17 | |
| *** eumel8 has quit IRC | 01:19 | |
| *** whoami-rajat has joined #openstack-infra | 01:19 | |
| *** jamesmcarthur has joined #openstack-infra | 01:20 | |
| *** ekultails has quit IRC | 01:20 | |
| *** jamesmcarthur has quit IRC | 01:24 | |
| *** wolverineav has joined #openstack-infra | 01:28 | |
| *** rascasoft has joined #openstack-infra | 01:30 | |
| *** bhavikdbavishi has joined #openstack-infra | 01:38 | |
| *** rascasoft has quit IRC | 01:39 | |
| *** jamesmcarthur has joined #openstack-infra | 01:40 | |
| *** jamesmcarthur has quit IRC | 01:46 | |
| *** jamesmcarthur has joined #openstack-infra | 01:52 | |
| *** jamesmcarthur has quit IRC | 02:00 | |
| *** hongbin has joined #openstack-infra | 02:07 | |
| *** jamesmcarthur has joined #openstack-infra | 02:15 | |
| *** jamesmcarthur has quit IRC | 02:15 | |
| *** jamesmcarthur has joined #openstack-infra | 02:16 | |
| *** jamesmcarthur has quit IRC | 02:21 | |
| *** wolverineav has quit IRC | 02:34 | |
| *** jamesmcarthur has joined #openstack-infra | 02:42 | |
| *** jamesmcarthur has quit IRC | 02:46 | |
| *** jamesmcarthur has joined #openstack-infra | 02:52 | |
| *** psachin has joined #openstack-infra | 02:54 | |
| *** jamesmcarthur has quit IRC | 03:00 | |
| *** betherly has joined #openstack-infra | 03:00 | |
| *** jamesmcarthur has joined #openstack-infra | 03:02 | |
| *** betherly has quit IRC | 03:04 | |
| *** rlandy|bbl is now known as rlandy | 03:10 | |
| *** rlandy has quit IRC | 03:13 | |
| *** rascasoft has joined #openstack-infra | 03:23 | |
| *** armax has quit IRC | 03:24 | |
| *** markvoelker has quit IRC | 03:27 | |
| *** markvoelker has joined #openstack-infra | 03:27 | |
| *** rascasoft has quit IRC | 03:30 | |
| *** markvoelker has quit IRC | 03:32 | |
| *** jamesmcarthur has quit IRC | 03:33 | |
| *** ykarel|away has joined #openstack-infra | 03:40 | |
| *** agopi|out has joined #openstack-infra | 03:40 | |
| *** ykarel|away is now known as ykarel | 03:48 | |
| *** diablo_rojo has quit IRC | 03:53 | |
| *** jamesmcarthur has joined #openstack-infra | 03:54 | |
| *** jamesmcarthur has quit IRC | 03:59 | |
| *** eernst has joined #openstack-infra | 03:59 | |
| *** jamesmcarthur has joined #openstack-infra | 04:04 | |
| *** smarcet has quit IRC | 04:05 | |
| *** ramishra has joined #openstack-infra | 04:09 | |
| *** jamesmcarthur has quit IRC | 04:10 | |
| *** armax has joined #openstack-infra | 04:13 | |
| *** wolverineav has joined #openstack-infra | 04:17 | |
| *** wolverineav has quit IRC | 04:21 | |
| *** jamesmcarthur has joined #openstack-infra | 04:27 | |
| *** markvoelker has joined #openstack-infra | 04:28 | |
| *** jamesmcarthur has quit IRC | 04:31 | |
| *** udesale has joined #openstack-infra | 04:36 | |
| *** wolverineav has joined #openstack-infra | 04:39 | |
| *** eernst has quit IRC | 04:42 | |
| *** hwoarang has quit IRC | 04:47 | |
| *** jamesmcarthur has joined #openstack-infra | 04:48 | |
| *** owalsh_ has joined #openstack-infra | 04:49 | |
| *** hwoarang has joined #openstack-infra | 04:50 | |
| *** owalsh has quit IRC | 04:52 | |
| *** jamesmcarthur has quit IRC | 04:52 | |
| *** hwoarang has quit IRC | 04:56 | |
| *** hwoarang has joined #openstack-infra | 04:56 | |
| *** ykarel has quit IRC | 04:57 | |
| *** markvoelker has quit IRC | 05:02 | |
| *** jamesmcarthur has joined #openstack-infra | 05:09 | |
| *** ykarel has joined #openstack-infra | 05:11 | |
| *** jamesmcarthur has quit IRC | 05:13 | |
| *** wolverineav has quit IRC | 05:18 | |
| *** jamesmcarthur has joined #openstack-infra | 05:30 | |
| *** jamesmcarthur has quit IRC | 05:35 | |
| *** hongbin has quit IRC | 05:44 | |
| *** jamesmcarthur has joined #openstack-infra | 05:51 | |
| *** jamesmcarthur has quit IRC | 05:56 | |
| *** yboaron_ has joined #openstack-infra | 05:58 | |
| *** markvoelker has joined #openstack-infra | 05:58 | |
| *** ramishra_ has joined #openstack-infra | 06:00 | |
| *** ramishra has quit IRC | 06:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/636831 | 06:06 |
| *** jamesmcarthur has joined #openstack-infra | 06:12 | |
| *** ramishra_ is now known as ramishra | 06:17 | |
| *** jamesmcarthur has quit IRC | 06:17 | |
| *** snapiri has joined #openstack-infra | 06:29 | |
| *** e0ne has joined #openstack-infra | 06:29 | |
| *** dpawlik has joined #openstack-infra | 06:31 | |
| *** markvoelker has quit IRC | 06:32 | |
| *** jamesmcarthur has joined #openstack-infra | 06:33 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 06:35 |
| *** jamesmcarthur has quit IRC | 06:38 | |
| *** ccamacho has quit IRC | 06:41 | |
| *** e0ne has quit IRC | 06:46 | |
| *** quiquell|off is now known as quiquell|rover | 06:47 | |
| *** jamesmcarthur has joined #openstack-infra | 06:54 | |
| *** dpawlik has quit IRC | 06:55 | |
| *** jamesmcarthur has quit IRC | 07:00 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 07:00 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 07:04 |
| *** jamesmcarthur has joined #openstack-infra | 07:05 | |
| *** wolverineav has joined #openstack-infra | 07:05 | |
| *** dpawlik has joined #openstack-infra | 07:05 | |
| *** slaweq has joined #openstack-infra | 07:08 | |
| *** wolverineav has quit IRC | 07:09 | |
| *** jamesmcarthur has quit IRC | 07:09 | |
| *** janki has joined #openstack-infra | 07:11 | |
| *** janki has quit IRC | 07:13 | |
| *** janki has joined #openstack-infra | 07:13 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Optionally silence git push in mirror-workspace-git-repos https://review.openstack.org/635166 | 07:16 |
| *** bhavikdbavishi has quit IRC | 07:17 | |
| *** pgaxatte has joined #openstack-infra | 07:18 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Optionally silence git in mirror-workspace-git-repos https://review.openstack.org/635166 | 07:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/636831 | 07:21 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Optionally silence git in mirror-workspace-git-repos https://review.openstack.org/635166 | 07:21 |
| *** aojea has joined #openstack-infra | 07:21 | |
| *** jamesmcarthur has joined #openstack-infra | 07:25 | |
| *** markvoelker has joined #openstack-infra | 07:28 | |
| *** Adri2000 has quit IRC | 07:28 | |
| *** Adri2000 has joined #openstack-infra | 07:29 | |
| *** jamesmcarthur has quit IRC | 07:30 | |
| *** quiquell|rover is now known as quique|rover|brb | 07:31 | |
| *** lujinluo has joined #openstack-infra | 07:32 | |
| *** apetrich has joined #openstack-infra | 07:35 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 07:35 |
| *** jtomasek has joined #openstack-infra | 07:35 | |
| *** jtomasek has quit IRC | 07:39 | |
| *** ykarel is now known as ykarel|lunch | 07:42 | |
| *** jtomasek has joined #openstack-infra | 07:44 | |
| *** jamesmcarthur has joined #openstack-infra | 07:46 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 07:46 |
| *** jamesmcarthur has quit IRC | 07:51 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Optionally silence git in mirror-workspace-git-repos https://review.openstack.org/635166 | 07:52 |
| *** kjackal has joined #openstack-infra | 07:56 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 07:57 |
| *** markvoelker has quit IRC | 08:01 | |
| *** e0ne has joined #openstack-infra | 08:05 | |
| *** jamesmcarthur has joined #openstack-infra | 08:07 | |
| *** ramishra has quit IRC | 08:08 | |
| *** e0ne has quit IRC | 08:08 | |
| *** tkajinam has quit IRC | 08:09 | |
| *** memoussati has joined #openstack-infra | 08:10 | |
| *** jamesmcarthur has quit IRC | 08:12 | |
| *** quique|rover|brb is now known as quiquell|rover | 08:15 | |
| *** rpittau has joined #openstack-infra | 08:17 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 08:17 |
| *** e0ne has joined #openstack-infra | 08:18 | |
| *** e0ne has quit IRC | 08:18 | |
| *** ramishra has joined #openstack-infra | 08:18 | |
| *** ccamacho has joined #openstack-infra | 08:22 | |
| *** jamesmcarthur has joined #openstack-infra | 08:28 | |
| *** e0ne has joined #openstack-infra | 08:29 | |
| *** ykarel|lunch is now known as ykarel | 08:29 | |
| *** memoussati has quit IRC | 08:31 | |
| *** jamesmcarthur has quit IRC | 08:32 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 08:33 |
| *** e0ne has quit IRC | 08:35 | |
| *** jpena|off is now known as jpena | 08:38 | |
| *** xek has joined #openstack-infra | 08:40 | |
| *** electrofelix has joined #openstack-infra | 08:43 | |
| *** tosky has joined #openstack-infra | 08:47 | |
| *** jamesmcarthur has joined #openstack-infra | 08:48 | |
| *** memoussati has joined #openstack-infra | 08:49 | |
| *** jpich has joined #openstack-infra | 08:53 | |
| *** jamesmcarthur has quit IRC | 08:53 | |
| tobias-urdin | infra-root: these two are merged now https://review.openstack.org/#/c/635941/ https://review.openstack.org/#/c/635965/ but i need some help requeuing a old release to test it | 08:53 |
| tobias-urdin | 19:40 < fungi> sudo zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=release --project=openstack/puppet-aodh --ref=refs/tags/14.2.0 --newrev=617ffad84b633618490ca1023f8a31d9694b31a9 | 08:53 |
| *** wolverineav has joined #openstack-infra | 08:57 | |
| *** kopecmartin|off is now known as kopecmartin | 08:57 | |
| *** markvoelker has joined #openstack-infra | 08:58 | |
| *** panda|off is now known as panda | 09:00 | |
| frickler | tobias-urdin: enqueued | 09:00 |
| *** wolverineav has quit IRC | 09:01 | |
| *** memoussati has quit IRC | 09:03 | |
| *** jamesmcarthur has joined #openstack-infra | 09:06 | |
| frickler | tobias-urdin: still no luck it seems "Forge API auth failed with code: 400" | 09:09 |
| *** memoussati has joined #openstack-infra | 09:10 | |
| *** jamesmcarthur has quit IRC | 09:12 | |
| *** dtantsur|afk is now known as dtantsur | 09:20 | |
| *** memoussati has quit IRC | 09:21 | |
| *** jamesmcarthur has joined #openstack-infra | 09:27 | |
| *** markvoelker has quit IRC | 09:31 | |
| *** jamesmcarthur has quit IRC | 09:32 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul-jobs master: Use zuul_workspace_root variable for Git workspace prep https://review.openstack.org/636870 | 09:32 |
| *** memoussati has joined #openstack-infra | 09:33 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul-jobs master: Use zuul_workspace_root variable for Git workspace prep https://review.openstack.org/636870 | 09:34 |
| *** derekh has joined #openstack-infra | 09:37 | |
| *** luizbag has joined #openstack-infra | 09:38 | |
| *** stakeda has quit IRC | 09:40 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 09:42 |
| *** memoussati has quit IRC | 09:43 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 09:43 |
| *** jamesmcarthur has joined #openstack-infra | 09:49 | |
| *** jamesmcarthur has quit IRC | 09:53 | |
| *** rosmaita has quit IRC | 09:55 | |
| tobias-urdin | frickler: hm, i tried that ansible module with my own credentials without any issues, perhaps it's actually something wrong with credentials | 09:55 |
| tobias-urdin | or maybe i should add more debug output, let me check what the reponse is on code 400 | 09:55 |
| *** ociuhandu has joined #openstack-infra | 09:56 | |
| *** AJaeger has quit IRC | 10:01 | |
| *** memoussati has joined #openstack-infra | 10:03 | |
| *** fdegir has joined #openstack-infra | 10:04 | |
| *** AJaeger has joined #openstack-infra | 10:04 | |
| tobias-urdin | frickler: i would prefer we do not send the password out, but can somebody that has access to that password try http://paste.openstack.org/show/745081/ | 10:06 |
| *** jamesmcarthur has joined #openstack-infra | 10:10 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:12 | |
| *** jamesmcarthur has quit IRC | 10:15 | |
| *** walshh__ has quit IRC | 10:16 | |
| tobias-urdin | if that works its something with the encrypted secret or the usage of the secret that is faulty in project-config/zuul.d/jobs.yaml or project-config/playbooks/publish/puppetforge.yaml | 10:17 |
| *** whoami-rajat has quit IRC | 10:19 | |
| openstackgerrit | sahid proposed openstack-dev/pbr master: Change python3.5 job to python3.7 job on Stein+ https://review.openstack.org/610659 | 10:24 |
| *** markvoelker has joined #openstack-infra | 10:28 | |
| frickler | tobias-urdin: that works with the password we have on record. so probably something wrong with the encrypted secret or the way it is used. maybe fungi can look into that later | 10:31 |
| *** jamesmcarthur has joined #openstack-infra | 10:31 | |
| frickler | infra-root: FYI, running "gpg-agent --daemon emacs" failed on bridge because there was an active gpg-agent. please remember to use this incantation to avoid leaving secrets active after you exit emacs | 10:32 |
| *** bhavikdbavishi has quit IRC | 10:34 | |
| tobias-urdin | frickler: thanks! good then we know what is blocking | 10:34 |
| *** jamesmcarthur has quit IRC | 10:36 | |
| *** yamamoto has quit IRC | 10:40 | |
| *** jbadiapa has quit IRC | 10:42 | |
| frickler | tobias-urdin: hmm, just browsing things it seems that the api also returns 400 for things like bad module names, probably it would be good to have more logging available in that module anyway. https://tickets.puppetlabs.com/projects/FORGE/issues/FORGE-228?filter=allopenissues | 10:43 |
| *** betherly has joined #openstack-infra | 10:44 | |
| *** betherly has quit IRC | 10:44 | |
| *** jamesmcarthur has joined #openstack-infra | 10:52 | |
| tobias-urdin | frickler: should be fine, the auth and actual upload call is separated so it will show all errors regarding bad module etc | 10:54 |
| tobias-urdin | https://github.com/openstack-infra/zuul-jobs/blob/master/roles/upload-forge/library/forge_upload.py#L168 | 10:54 |
| tobias-urdin | when response code != 201 | 10:54 |
| tobias-urdin | on 409 (exact module version already exists) it fails with "module already exists" | 10:55 |
| *** jamesmcarthur has quit IRC | 10:57 | |
| *** udesale has quit IRC | 10:58 | |
| *** markvoelker has quit IRC | 11:01 | |
| frickler | tobias-urdin: ah, right, that would give a different error msg. so probably someone will have to crosscheck that decrypting the encrypted secret gives the correct result. or otherwise just try to refresh with a newly encrypted version. | 11:03 |
| tobias-urdin | yeah, it's also entirely possible that i messed up the "secrets" passed on the "job", set the "name" on that and are doing it wrong in publish/puppetforge.yaml | 11:04 |
| tobias-urdin | but i don't see any errors when staring at it | 11:04 |
| *** roman_g has joined #openstack-infra | 11:12 | |
| *** jamesmcarthur has joined #openstack-infra | 11:14 | |
| frickler | tobias-urdin: did some staring myself and that didn't help either ;) | 11:16 |
| tobias-urdin | hehe :) | 11:16 |
| *** jamesmcarthur has quit IRC | 11:18 | |
| *** yamamoto has joined #openstack-infra | 11:21 | |
| *** yamamoto has quit IRC | 11:28 | |
| *** jamesmcarthur has joined #openstack-infra | 11:35 | |
| *** jamesmcarthur has quit IRC | 11:40 | |
| *** priteau has joined #openstack-infra | 11:43 | |
| *** vdrok_ has quit IRC | 11:44 | |
| *** vdrok has joined #openstack-infra | 11:47 | |
| *** janki has quit IRC | 11:48 | |
| *** janki has joined #openstack-infra | 11:48 | |
| *** quiquell|rover is now known as quique|rover|r-- | 11:53 | |
| *** janki has quit IRC | 11:55 | |
| *** janki has joined #openstack-infra | 11:55 | |
| *** jamesmcarthur has joined #openstack-infra | 11:57 | |
| *** yamamoto has joined #openstack-infra | 11:57 | |
| *** rpittau has quit IRC | 11:58 | |
| *** jpena is now known as jpena|lunch | 11:58 | |
| *** markvoelker has joined #openstack-infra | 11:58 | |
| *** jamesmcarthur has quit IRC | 12:01 | |
| *** yamamoto has quit IRC | 12:10 | |
| *** armstrong has joined #openstack-infra | 12:11 | |
| *** yamamoto has joined #openstack-infra | 12:14 | |
| *** jamesmcarthur has joined #openstack-infra | 12:18 | |
| *** memoussati has quit IRC | 12:19 | |
| *** yamamoto has quit IRC | 12:22 | |
| *** jamesmcarthur has quit IRC | 12:23 | |
| *** markvoelker has quit IRC | 12:26 | |
| *** wolverineav has joined #openstack-infra | 12:33 | |
| *** smarcet has joined #openstack-infra | 12:34 | |
| *** priteau has quit IRC | 12:36 | |
| *** priteau has joined #openstack-infra | 12:38 | |
| *** wolverineav has quit IRC | 12:38 | |
| *** jamesmcarthur has joined #openstack-infra | 12:39 | |
| *** memoussati has joined #openstack-infra | 12:41 | |
| *** janki has quit IRC | 12:41 | |
| *** yamamoto has joined #openstack-infra | 12:42 | |
| *** smarcet has quit IRC | 12:43 | |
| *** jamesmcarthur has quit IRC | 12:44 | |
| *** udesale has joined #openstack-infra | 12:45 | |
| *** owalsh_ is now known as owalsh_afk | 12:51 | |
| *** rh-jelabarre has joined #openstack-infra | 12:53 | |
| *** rpittau has joined #openstack-infra | 12:56 | |
| *** yamamoto has quit IRC | 13:00 | |
| *** jamesmcarthur has joined #openstack-infra | 13:01 | |
| *** ekultails has joined #openstack-infra | 13:01 | |
| *** rosmaita has joined #openstack-infra | 13:03 | |
| *** yamamoto has joined #openstack-infra | 13:03 | |
| *** jamesmcarthur has quit IRC | 13:05 | |
| *** yamamoto has quit IRC | 13:06 | |
| *** yamamoto has joined #openstack-infra | 13:06 | |
| *** yamamoto has quit IRC | 13:07 | |
| *** yamamoto has joined #openstack-infra | 13:13 | |
| *** priteau has quit IRC | 13:13 | |
| *** trown|outtypewww is now known as trown | 13:16 | |
| *** jbadiapa has joined #openstack-infra | 13:17 | |
| *** yamamoto has quit IRC | 13:19 | |
| *** jamesmcarthur has joined #openstack-infra | 13:21 | |
| *** jamesmcarthur has quit IRC | 13:21 | |
| *** jamesmcarthur_ has joined #openstack-infra | 13:22 | |
| *** armstrong has quit IRC | 13:22 | |
| *** armstrong has joined #openstack-infra | 13:23 | |
| *** jpena|lunch is now known as jpena | 13:24 | |
| *** smarcet has joined #openstack-infra | 13:25 | |
| *** agopi|out is now known as agopi|brb | 13:28 | |
| *** jamesmcarthur_ has quit IRC | 13:29 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config master: Update puppet config for openstackid-dev node https://review.openstack.org/636952 | 13:30 |
| *** agopi|brb has quit IRC | 13:32 | |
| *** weshay has joined #openstack-infra | 13:34 | |
| *** yboaron_ has quit IRC | 13:36 | |
| *** yboaron_ has joined #openstack-infra | 13:36 | |
| *** mriedem has joined #openstack-infra | 13:39 | |
| *** rlandy has joined #openstack-infra | 13:41 | |
| *** yamamoto has joined #openstack-infra | 13:46 | |
| *** yamamoto has quit IRC | 13:46 | |
| *** yamamoto has joined #openstack-infra | 13:46 | |
| *** yamamoto has quit IRC | 13:47 | |
| *** weshay is now known as weshay|ruck | 13:47 | |
| *** yamamoto has joined #openstack-infra | 13:47 | |
| *** jaosorior has quit IRC | 13:47 | |
| *** jamesmcarthur has joined #openstack-infra | 13:49 | |
| *** jaosorior has joined #openstack-infra | 13:51 | |
| *** jamesmcarthur has quit IRC | 13:52 | |
| *** jamesmcarthur has joined #openstack-infra | 13:52 | |
| *** quique|rover|r-- is now known as quiquell|rover | 13:53 | |
| *** memoussati has quit IRC | 13:54 | |
| *** priteau has joined #openstack-infra | 13:55 | |
| sshnaidm | clarkb, pabelanger fungi do you know if there is a way to prevent merge of patch if 3d party CI failed? | 13:58 |
| *** agopi|brb has joined #openstack-infra | 14:00 | |
| *** jamesmcarthur has quit IRC | 14:01 | |
| *** rfolco is now known as rfolco|off | 14:01 | |
| *** owalsh_afk is now known as owalsh | 14:01 | |
| *** rosmaita has quit IRC | 14:02 | |
| *** agopi_ has joined #openstack-infra | 14:04 | |
| *** agopi|brb has quit IRC | 14:06 | |
| fungi | sshnaidm: yes, don't approve it | 14:08 |
| *** yamamoto has quit IRC | 14:09 | |
| sshnaidm | fungi, and something more rough and tyranic? :) | 14:09 |
| fungi | sshnaidm: short of that, you might be able to write a zuul job which checked the vote details on the change under test and then return a failure result under specific conditions, though i haven't thought through what race conditions an implementation like that might imply | 14:09 |
| sshnaidm | fungi, cool, will think how to do it.. | 14:10 |
| fungi | sshnaidm: though to answer the question you may have been trying to ask, zuul doesn't have extensible features for depending on the results of nor deferring to other ci systems | 14:11 |
| sshnaidm | fungi, yeah, seems like good feature to have in the future. | 14:12 |
| *** memoussati has joined #openstack-infra | 14:12 | |
| fungi | i disagree, but it's worthy of debating | 14:13 |
| sshnaidm | especially if the another ci is also zuul based | 14:13 |
| fungi | teams like neutron, cinder, ironic and nova who have potentially dozens of third-party ci systems manage to pay attention to the votes those cast and take them into account when deciding whether or not to approve a change | 14:13 |
| fungi | also, ci testing and gating aren't a substitute for reviewers' attention to detail | 14:15 |
| *** agopi_ is now known as agopi | 14:18 | |
| *** ykarel is now known as ykarel|away | 14:18 | |
| *** yboaron_ has quit IRC | 14:20 | |
| *** jamesmcarthur has joined #openstack-infra | 14:22 | |
| *** psachin has quit IRC | 14:22 | |
| *** kjackal has quit IRC | 14:23 | |
| *** ykarel|away has quit IRC | 14:23 | |
| *** jamesmcarthur has quit IRC | 14:26 | |
| *** kjackal has joined #openstack-infra | 14:27 | |
| sshnaidm | can not disagree | 14:30 |
| sshnaidm | but things happen | 14:31 |
| openstackgerrit | Nir Magnezi proposed openstack/diskimage-builder master: [wip] rhel8 beta support https://review.openstack.org/623137 | 14:32 |
| tobias-urdin | fungi: could you check the conversation i had with frickler a bit up, thanks! so summarize there's something wrong the the secret or how the secret is passed/configured in jobs->playbook | 14:32 |
| *** ekultails has quit IRC | 14:34 | |
| *** yboaron_ has joined #openstack-infra | 14:38 | |
| fungi | tobias-urdin: yep, saw it. i'll double-check that the secret decrypts to the same value we have on record, but it may be a bit | 14:38 |
| *** ykarel|away has joined #openstack-infra | 14:40 | |
| *** yamamoto has joined #openstack-infra | 14:41 | |
| *** gfidente has joined #openstack-infra | 14:41 | |
| frickler | fungi: I had the same idea, but gave up after looking into the code to see how that would be done. do you happen to have a tool for it? | 14:41 |
| *** yboaron_ has quit IRC | 14:42 | |
| *** yboaron_ has joined #openstack-infra | 14:42 | |
| fungi | frickler: yeah, the openssl command line ought to be able to do it | 14:42 |
| fungi | it uses a standard protocol | 14:42 |
| *** jamesmcarthur has joined #openstack-infra | 14:43 | |
| *** ykarel|away is now known as ykarel | 14:45 | |
| *** yamamoto has quit IRC | 14:46 | |
| *** jamesmcarthur has quit IRC | 14:47 | |
| *** nhicher has quit IRC | 14:48 | |
| *** nhicher has joined #openstack-infra | 14:50 | |
| smcginnis | sshnaidm, fungi: I believe if you give a CI voting rights then it can block patches from merging. | 14:51 |
| sshnaidm | smcginnis, it can vote, but not block.. | 14:51 |
| smcginnis | We don't go that far in Cinder though since most CIs are not super reliable, so there's always manual evaluation needed. | 14:51 |
| smcginnis | I could have sworn we accidentally added a third party CI account to a voting group once and had to quickly change it back. And I thought Nova would let the VMware NSX CI vote for that reason. | 14:52 |
| fungi | smcginnis: we grant third-party ci systems at most -1..+1 voting rights on the verified label. that doesn't block anything | 14:52 |
| smcginnis | Perhaps I'm wrong, or perhaps things have changed. | 14:52 |
| smcginnis | fungi: Ah, probably my misunderstanding then. | 14:52 |
| *** electrofelix has quit IRC | 14:53 | |
| *** ekultails has joined #openstack-infra | 14:53 | |
| fungi | the argument used by the cinder team a while back to not have their third-party ci systems vote is that a lot of those ci systems were unreliable and reviewers were skipping patches which had a verified -1 | 14:53 |
| smcginnis | Maybe I'm thinking of it showing up in the list of reviews as Verified-1, causing reviewers to skip it thinking it failed gate. | 14:53 |
| smcginnis | fungi: Yeah | 14:53 |
| smcginnis | Context switch: anyone know where things ended up with the LOCI post queue issue from yesterday? | 14:54 |
| fungi | smcginnis: i think we were hoping to get up with hogepodge but the short-term solution is you can delete that job from your post pipeline in your project. longer term, if the job is valuable, is to move it to a trusted/config repository so other projects can resume running it | 14:55 |
| smcginnis | fungi: I would think it would be very valuable to the loci team, but I hadn't even realized we had it in the cinder job config. | 14:56 |
| smcginnis | I will remove from there for now and leave it to them to decide how they want to handle things I guess. | 14:56 |
| smcginnis | THanks | 14:56 |
| fungi | it's in the git history, so easy enough to add back once it's working again | 15:00 |
| smcginnis | ++ | 15:00 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 15:00 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 15:02 |
| *** jamesmcarthur has joined #openstack-infra | 15:04 | |
| *** jamesmcarthur has quit IRC | 15:08 | |
| *** rosmaita has joined #openstack-infra | 15:14 | |
| *** wolverineav has joined #openstack-infra | 15:15 | |
| *** roman_g has quit IRC | 15:16 | |
| *** wolverineav has quit IRC | 15:19 | |
| *** udesale has quit IRC | 15:22 | |
| *** jamesmcarthur has joined #openstack-infra | 15:24 | |
| *** jamesmcarthur has quit IRC | 15:29 | |
| *** eernst has joined #openstack-infra | 15:34 | |
| *** memoussati has quit IRC | 15:35 | |
| *** jamesmcarthur has joined #openstack-infra | 15:45 | |
| *** jamesmcarthur has quit IRC | 15:50 | |
| *** jamesmcarthur has joined #openstack-infra | 15:52 | |
| *** jamesmcarthur has quit IRC | 15:54 | |
| *** jamesmcarthur has joined #openstack-infra | 15:54 | |
| clarkb | frickler: sorry that was probably me | 15:55 |
| *** gfidente has quit IRC | 15:56 | |
| *** memoussati has joined #openstack-infra | 15:56 | |
| *** yboaron_ has quit IRC | 15:57 | |
| clarkb | re gpg agent | 15:58 |
| *** pgaxatte has quit IRC | 16:01 | |
| *** diablo_rojo has joined #openstack-infra | 16:01 | |
| *** eernst has quit IRC | 16:02 | |
| *** kaisers has quit IRC | 16:07 | |
| *** ramishra has quit IRC | 16:07 | |
| *** kaisers has joined #openstack-infra | 16:15 | |
| *** dtantsur is now known as dtantsur|afk | 16:16 | |
| zbr | clarkb: fungi: few days ago I was asked to by pabelanger to stop using bindep-fallback template due to deprecation. Does this look ok? https://review.openstack.org/#/c/636163/ | 16:18 |
| *** smarcet has quit IRC | 16:24 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Add Fake Github Review object to test suite https://review.openstack.org/636788 | 16:24 |
| *** smarcet has joined #openstack-infra | 16:25 | |
| clarkb | zbr: lgtm. I made a couple suggestions inline if you have to make a new patchset but I don't think they are critical | 16:27 |
| *** smarcet has quit IRC | 16:28 | |
| *** smarcet has joined #openstack-infra | 16:30 | |
| *** memoussati has quit IRC | 16:32 | |
| clarkb | corvus: I think we are still experiencing the slow ansible runs due to broken k8s module (at least I don't expect these tasks to run this long | 16:36 |
| clarkb | so the upgrade and changes to how thread pools were handled didn't help | 16:36 |
| tobias-urdin | fungi: thanks :) | 16:36 |
| clarkb | I wonder if this is a multiprocessing interaction between ansible use of multiprocessing and k8s module use of it | 16:36 |
| clarkb | anyone else seen this problem? dmsimard perhaps? | 16:36 |
| *** markvoelker has joined #openstack-infra | 16:38 | |
| clarkb | fungi: https://review.openstack.org/#/c/636681/ is a quick and easy one if you have a moment. I'll start figuring out the replacement pbx as soon as that merges and is on bridge | 16:39 |
| dmsimard | clarkb: it doesn't ring me a bell but if you are reproducing this with Zuul's Ansible, I would try to see if we can reproduce it with the latest 2.7 | 16:40 |
| clarkb | dmsimard: no this is with ansible 2.7.3 on bridge.openstack.org. The k8s module ends up deadlocking around thread pool cleanup | 16:40 |
| clarkb | this also doesn't put much faith in swagger client generation :/ | 16:41 |
| dmsimard | clarkb: have a link to the playbook/role where the tasks in question are used ? | 16:41 |
| clarkb | dmsimard: https://git.openstack.org/cgit/openstack-infra/system-config/tree/kubernetes/gitea/gitea-playbook.yaml the k8s tasks there | 16:42 |
| *** ociuhand_ has joined #openstack-infra | 16:42 | |
| clarkb | oh now thats curious | 16:43 |
| clarkb | double checking which version of the python kubernetes lib we have isntalled I still see 8.0.1 | 16:43 |
| clarkb | something undid the 9.0.0 beta install maybe? we should rerun with that and double check if it fixes or not | 16:43 |
| *** agopi is now known as agopi|FOOD | 16:44 | |
| *** ociuhandu has quit IRC | 16:45 | |
| *** trident has quit IRC | 16:46 | |
| *** ccamacho has quit IRC | 16:48 | |
| *** ociuhand_ has quit IRC | 16:48 | |
| *** dpawlik has quit IRC | 16:52 | |
| dmsimard | clarkb: I see kubernetes==8.0.1 and openshift==0.8.2. I guess kubernetes is pulled by openshift: https://github.com/openshift/openshift-restclient-python/blob/master/requirements.txt | 16:53 |
| *** ijw has joined #openstack-infra | 16:53 | |
| dmsimard | clarkb: openshift python lib 0.8.5 was released yesterday fwiw: https://pypi.org/project/openshift/#history | 16:53 |
| clarkb | dmsimard: yes it is, and that is where corvus' debugging seemed to pinpoint the deadlock. Reviewing its code (whcih is generated) there were chagnes around thread pool management in 9.0.0 beta | 16:53 |
| clarkb | so we thought maybe an upgrade would fix things but we seem to have downgraded after the upgrade so we should try that again | 16:54 |
| clarkb | possibly because of that openshift release yesterday causing a reinstall | 16:54 |
| *** gyee has joined #openstack-infra | 16:55 | |
| dmsimard | there's a 9.0.0b1 of kubernetes that was released yesterday as well.. is that a coincidence ? | 16:55 |
| clarkb | ya thats the one I thought we had installed so we'll just need to redo that again | 16:56 |
| *** quiquell|rover is now known as quiquell|off | 16:57 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: URLTrigger driver time based - artifact change jobs triggering driver https://review.openstack.org/635567 | 17:00 |
| *** ociuhandu has joined #openstack-infra | 17:02 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: URLTrigger driver time based - artifact change jobs triggering driver https://review.openstack.org/635567 | 17:02 |
| *** sreejithp has joined #openstack-infra | 17:05 | |
| *** sthussey has joined #openstack-infra | 17:05 | |
| *** ociuhandu has quit IRC | 17:06 | |
| *** ianychoi has joined #openstack-infra | 17:06 | |
| *** ijw_ has joined #openstack-infra | 17:06 | |
| *** jpich has quit IRC | 17:08 | |
| *** ijw has quit IRC | 17:10 | |
| *** markvoelker has quit IRC | 17:12 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Update API code to work with Presentation Moderators collection (+N) https://review.openstack.org/636190 | 17:13 |
| *** ociuhandu has joined #openstack-infra | 17:15 | |
| *** kopecmartin is now known as kopecmartin|off | 17:16 | |
| *** wolverineav has joined #openstack-infra | 17:17 | |
| *** wolverineav has quit IRC | 17:17 | |
| *** wolverineav has joined #openstack-infra | 17:17 | |
| *** armstrong has quit IRC | 17:21 | |
| *** mattw4 has joined #openstack-infra | 17:22 | |
| *** rpittau has quit IRC | 17:22 | |
| *** betherly has joined #openstack-infra | 17:23 | |
| *** ociuhandu has quit IRC | 17:24 | |
| *** memoussati has joined #openstack-infra | 17:27 | |
| *** aojea has quit IRC | 17:31 | |
| *** wolverineav has quit IRC | 17:32 | |
| *** wolverineav has joined #openstack-infra | 17:32 | |
| *** ijw_ has quit IRC | 17:34 | |
| *** wolverineav has quit IRC | 17:36 | |
| *** wolverineav has joined #openstack-infra | 17:36 | |
| *** NCLanceman has joined #openstack-infra | 17:43 | |
| *** Jason_Lee has joined #openstack-infra | 17:46 | |
| *** luizbag has quit IRC | 17:47 | |
| *** NCLanceman has quit IRC | 17:47 | |
| *** Jason_Lee has quit IRC | 17:48 | |
| *** NCLanceman has joined #openstack-infra | 17:48 | |
| *** trown is now known as trown|lunch | 17:49 | |
| fungi | okay, i need to go out for lunch and some errands. on my return i'll try to decrypt the puppetforge creds we have in job config to make sure they match what's on record (tobias-urdin), and get the openstackid-dev mysql ssl keys into private hiera (smarcet) | 17:49 |
| smarcet | fungi: thx u! :) | 17:49 |
| corvus | clarkb, dmsimard: wow, what an unlucky coincidence. did you re-install 9.0.0? | 17:50 |
| fungi | those are currently the top items to be popped off my (lengthy) to do list anyway | 17:50 |
| clarkb | corvus: I have not reinstalled yet in case you had other ideas for why that may have happened | 17:52 |
| *** ykarel is now known as ykarel|away | 17:54 | |
| corvus | clarkb: what does ~= mean? | 17:56 |
| clarkb | corvus: in what context? did I type that? usually if I type that I mean approximately | 17:56 |
| corvus | clarkb: sorry -- https://github.com/openshift/openshift-restclient-python/blob/master/requirements.txt#L3 | 17:57 |
| clarkb | hrm I think that means any 8.x version | 17:57 |
| clarkb | it won't upgrade them by default either | 17:57 |
| clarkb | except when you have 9.0 :/ | 17:57 |
| corvus | clarkb: meh. maybe we should just upgrade and check back in a few hours :) | 17:58 |
| corvus | i'll do it | 17:58 |
| clarkb | wfm | 17:58 |
| corvus | (i still have the shell open from last time) | 17:58 |
| corvus | looks like we currently have a stuck task. | 17:58 |
| corvus | i will kill it | 17:58 |
| mgagne | currently checking for an issue with orphan/zombie neutron ports in inap-mtl01. new instances fail to get an IP address since there are none free. | 17:59 |
| *** ykarel|away has quit IRC | 18:00 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Re-use the github PR object when fetching reviews https://review.openstack.org/636705 | 18:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add comment about extra issues request https://review.openstack.org/636706 | 18:03 |
| *** Jason_Lee has joined #openstack-infra | 18:07 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/bindep master: Replace deprecated bindep-fallback testing https://review.openstack.org/636163 | 18:07 |
| *** jpena is now known as jpena|off | 18:08 | |
| *** derekh has quit IRC | 18:09 | |
| *** NCLanceman has quit IRC | 18:09 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/bindep master: Replace deprecated bindep-fallback testing https://review.openstack.org/636163 | 18:09 |
| *** markvoelker has joined #openstack-infra | 18:09 | |
| *** priteau has quit IRC | 18:13 | |
| *** wolverineav has quit IRC | 18:14 | |
| clarkb | corvus: next run_all.sh just started | 18:15 |
| corvus | still at 9.0.0b1 | 18:15 |
| *** Jason_Lee has quit IRC | 18:16 | |
| *** wolverineav has joined #openstack-infra | 18:16 | |
| clarkb | just downgreaded looks like | 18:16 |
| clarkb | TASK [install-ansible : Install openshift client] ****************************** and friends must do it | 18:16 |
| corvus | yeah :( | 18:16 |
| clarkb | maybe we just delete that task for now? | 18:17 |
| corvus | wfm | 18:17 |
| corvus | clarkb: i'll patch | 18:18 |
| *** wolverineav has quit IRC | 18:18 | |
| corvus | i'm reinstalling manually | 18:18 |
| *** wolverineav has joined #openstack-infra | 18:18 | |
| corvus | it'll get used on this run | 18:18 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Stop install the openshift client https://review.openstack.org/637020 | 18:19 |
| clarkb | is the chagne if we want to do it longer term | 18:19 |
| *** agopi|FOOD is now known as agopi | 18:19 | |
| clarkb | corvus: oh thats a good idea, we'll be able to confirm if it works and if it helps that way | 18:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Temporarily stop installing openshift https://review.openstack.org/637021 | 18:20 |
| corvus | clarkb: ^ can we do that instead so we don't forget? | 18:20 |
| clarkb | wfm | 18:20 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Make UnsafeTag self registering https://review.openstack.org/637023 | 18:21 |
| *** NCLanceman has joined #openstack-infra | 18:24 | |
| *** smarcet has quit IRC | 18:24 | |
| *** Jason_Lee has joined #openstack-infra | 18:28 | |
| *** NCLanceman has quit IRC | 18:30 | |
| openstackgerrit | Merged openstack-infra/zuul master: Mark as unsafe commit message at inventory https://review.openstack.org/633930 | 18:32 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade all dev servers to puppet 4 https://review.openstack.org/630391 | 18:34 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade some servers to puppet 4 https://review.openstack.org/634726 | 18:34 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade git01.openstack.org to puppet 4 https://review.openstack.org/634727 | 18:34 |
| cmurphy | clarkb: ^ i didn't realize that was in merge conflict | 18:34 |
| cmurphy | clarkb: also noticing that we didn't do https://review.openstack.org/616001 i think there was a good reason for that? is puppet turned back on for those? | 18:35 |
| *** Jason_Lee has quit IRC | 18:35 | |
| clarkb | cmurphy: it isn't turned on yet. smarcet is working on upgrades now (and is almost done on the dev server, next prod, then we can puppet 4 I think) | 18:35 |
| cmurphy | ok | 18:36 |
| cmurphy | and ask.o.o? | 18:36 |
| clarkb | I don't know about ask | 18:36 |
| clarkb | fungi: ^ do you? | 18:36 |
| cmurphy | also refstack https://review.openstack.org/628153 i think i took that out of the list for a reason, i think i found some channel history about refstack going to containers? | 18:37 |
| clarkb | ya refstack needs some care/feeding | 18:37 |
| cmurphy | should we puppet4 it or ignore it? | 18:37 |
| clarkb | I know hogepodge wanted to move it to a docker image based deployment (which we can now do) but unsure if progress was made on that yet | 18:37 |
| clarkb | cmurphy: I would probably ignore it for now | 18:38 |
| cmurphy | kk | 18:38 |
| cmurphy | hoping to get all this done before the end of the cycle | 18:38 |
| clarkb | corvus: ansible is running puppet afs playbook now so either newer kubernetes lib fixed things or it broke it faster :) | 18:39 |
| *** shardy has quit IRC | 18:40 | |
| clarkb | corvus: I think it worked | 18:40 |
| clarkb | http://paste.openstack.org/show/745115/ | 18:41 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: URLTrigger driver time based - artifact change jobs triggering driver https://review.openstack.org/635567 | 18:42 |
| *** markvoelker has quit IRC | 18:42 | |
| *** sshnaidm is now known as sshnaidm|off | 18:47 | |
| *** dims has quit IRC | 18:47 | |
| *** ccamacho has joined #openstack-infra | 18:47 | |
| tosky | uh, a "REMOTE HOST IDENTIFICATION HAS CHANGED" error led to a POST_FAILURE in a sahara job: http://logs.openstack.org/57/634757/4/check/python-saharaclient-tempest/f64602b/ara-report/ | 18:48 |
| tosky | there are also other failures in the devstack run | 18:48 |
| clarkb | corvus: if you confirm that it seems to have worked too then I think you should go ahead and approve https://review.openstack.org/#/c/637021/ | 18:48 |
| tosky | and that job was not touched in a while - a transient error? | 18:48 |
| clarkb | tosky: that happens when clouds reuse IP addresses | 18:48 |
| clarkb | tosky: then the instance fight over the IP via ARP | 18:48 |
| clarkb | and if the "wrong" host has the IP when we try to ssh bad things happen | 18:49 |
| clarkb | the joys of dogfooding :) | 18:49 |
| tosky | I was lucky enough so far, it's the first time I see this | 18:49 |
| clarkb | ya its not a super frequent thing but >0 | 18:49 |
| tosky | is this classified in some way, so that I can add the proper label after recheck? | 18:50 |
| clarkb | I think it is /me double checks. However there is no need to label things with rechecks. | 18:50 |
| clarkb | we stopped doing that when we compared human generated data to elastic-recheck data and found humans are very error prone | 18:50 |
| tosky | I know, but at least this time it seems to be a known issue | 18:50 |
| tosky | ah | 18:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Cache github PR shas https://review.openstack.org/636764 | 18:51 |
| clarkb | I'm not seeing it in e-r currently so maybe we need to add it or revert a removal if it was there | 18:51 |
| corvus | clarkb: lgtm +3 | 18:52 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/bindep master: Fix tox python3 overrides https://review.openstack.org/605613 | 18:59 |
| clarkb | corvus: I wonder if we should file bugs against openshift client and ansible (seems k8s client is already fixing it, but the others need to change their deps) | 19:00 |
| corvus | clarkb: for some reason i don't understand, the system-config updates take 2 pulses to take effect | 19:02 |
| *** wolverineav has quit IRC | 19:02 | |
| corvus | oh, and the change hasn't landed anyway | 19:03 |
| corvus | i re-installed 9.0.0 | 19:03 |
| clarkb | corvus: we only update system-config at the very beginning of the run | 19:03 |
| clarkb | so you have to get in before that for most things | 19:03 |
| corvus | ah, i must have had bad timing | 19:03 |
| corvus | hopefully the openshift change will land between now and the next run | 19:03 |
| *** wolverineav has joined #openstack-infra | 19:03 | |
| *** jamesmcarthur has quit IRC | 19:03 | |
| *** wolverineav has quit IRC | 19:06 | |
| *** wolverineav has joined #openstack-infra | 19:06 | |
| pabelanger | we seem to have a large backlog in executor queue at the moment, does anybody know why? | 19:08 |
| clarkb | looks like executors have been nnot accepting new jobs since about 1800UTC (small brief periods where they do) | 19:11 |
| clarkb | load averages have been high according to the grafana graphs | 19:12 |
| clarkb | and the running builds spiked, but they haven't persisted at those levels so you'd expect jobs take up to happen again | 19:12 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Cache github PR shas https://review.openstack.org/636764 | 19:13 |
| pabelanger | I am guessing the large stack at 636253 has something to do with it | 19:13 |
| clarkb | pabelanger: if you look at node usage graph we continue to use just as many or more nodes through that time period | 19:14 |
| clarkb | so I think its demand | 19:14 |
| *** NCLanceman has joined #openstack-infra | 19:14 | |
| clarkb | I don't think we're falling behind so much as going full steam ahead | 19:14 |
| clarkb | (and maybe falling behind when the current is stronger) | 19:15 |
| pabelanger | clarkb: yah, I think we might be capped at how fast we can launch jobs on the executors, I haven't looked at logs, but graphs to be seem to show we have the capacity (RAM / HDD / CPU) on current executors to launch more jobs at once | 19:16 |
| clarkb | pabelanger: it could also be the 4 jobs per second or whatever that number/period is for throttling | 19:17 |
| clarkb | pabelanger: but if you look at the node usage we are using on average more nodes over that time period | 19:17 |
| clarkb | so zuul si going as fast as it can I think | 19:17 |
| *** ccamacho has quit IRC | 19:17 | |
| pabelanger | clarkb: well, i think the nodes are ready, but we haven't started ansible-playbook process yet | 19:18 |
| pabelanger | due to 4 jobs per sec limit | 19:18 |
| clarkb | they say they are in use, do we flip that bit over before starting ansible? | 19:18 |
| pabelanger | so, nodes are idle not working in used state | 19:18 |
| pabelanger | clarkb: yah, I think so | 19:18 |
| clarkb | gotcha | 19:18 |
| *** jamesmcarthur has joined #openstack-infra | 19:18 | |
| clarkb | some of the dip is due to the merger queue backlog I think (which would be explained by that large stack | 19:19 |
| clarkb | that roughly coincides with the dip in running jobs | 19:19 |
| pabelanger | ah, I missed thata | 19:20 |
| clarkb | and the scheduler itself is consuming quite a bit of cpu. Those two things may end up being our bottlenecks? | 19:20 |
| *** NCLanceman has quit IRC | 19:20 | |
| clarkb | merger to say here is the new configs, scheduelr to parse them all and take action on them. Then once it does there is a lot of work for executors to pick up and we hit the 4 per second limit? | 19:20 |
| clarkb | (I don't have hard data for that other than teh zuul-status page graphs) | 19:21 |
| *** jamesmcarthur has quit IRC | 19:21 | |
| clarkb | but ya the biggest dip coincides with merger backlog. And scheduler tends to have to take action on post merger activity | 19:21 |
| *** sreejithp_ has joined #openstack-infra | 19:22 | |
| *** wolverineav has quit IRC | 19:22 | |
| clarkb | and ya that nova stack rebase conincides with all that | 19:22 |
| *** wolverineav has joined #openstack-infra | 19:23 | |
| *** NCLanceman has joined #openstack-infra | 19:23 | |
| pabelanger | yah, starting builds governor likely could be allowed to open up a little more, to then let memory / cpu / hdd govenor be our limit | 19:23 |
| *** sreejithp has quit IRC | 19:24 | |
| *** ociuhandu has joined #openstack-infra | 19:24 | |
| *** ociuhandu has quit IRC | 19:30 | |
| cmurphy | huh i hit the remote host key change too http://logs.openstack.org/91/630391/5/check/tox-linters/9d053bf/job-output.txt.gz#_2019-02-14_19_08_09_416162 | 19:30 |
| *** jamesmcarthur has joined #openstack-infra | 19:30 | |
| *** Jason_Lee has joined #openstack-infra | 19:30 | |
| tobiash | clarkb: are your executors on local disks or shared storage? | 19:32 |
| clarkb | tobiash: they are cinder block volumes, which are remote disks | 19:32 |
| *** NCLanceman has quit IRC | 19:32 | |
| tobiash | clarkb: so if the disks would become slower due to something somthing this would also count into the load and that spikes up leading to a deregister of the executors | 19:33 |
| pabelanger | tobiash: loadavg / memory / swap all look okay in grafana | 19:35 |
| tobiash | pabelanger: how about io of the disks? | 19:35 |
| pabelanger | s/swap/hdd | 19:35 |
| tobiash | I mean not disk space but iops | 19:36 |
| tobiash | we see that our executors are mostly io limited | 19:36 |
| tobiash | (they're on ceph) | 19:36 |
| pabelanger | tobiash: not sure, haven't looked at cacti | 19:36 |
| clarkb | tobiash: http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1&from=now-3h&to=now the starting builds graph seems to show its the main jobs per $timeframe limit I thin | 19:36 |
| pabelanger | we should log the data there | 19:36 |
| clarkb | starting builds should trend lower than 4 if it is another limit | 19:37 |
| clarkb | I think | 19:37 |
| corvus | has anyone compared ze08 to the rest? | 19:38 |
| clarkb | corvus: I looked the other day comparing swap and its swap usage is way down compared to the others | 19:38 |
| clarkb | haven't checked today | 19:39 |
| corvus | reduced mem/swap should allow it to accept more jobs when we hit capacity | 19:39 |
| corvus | our executors should be able to handle more than they are, the reason they aren't is swapping activity | 19:39 |
| openstackgerrit | Merged openstack-infra/system-config master: Trigger deployment with gitea 1.6.3 https://review.openstack.org/635016 | 19:39 |
| *** markvoelker has joined #openstack-infra | 19:39 | |
| clarkb | ya but I don't think the governing happens due to cpu or memory or disk, its just starting jobs limit? or maybe I misunderstand how that limit works? | 19:40 |
| *** weshay|ruck has quit IRC | 19:40 | |
| corvus | clarkb: that's correct -- but here's the thing -- if the system has enough headroom, then a starting job becomes a running job quickly | 19:41 |
| corvus | clarkb: think about how the system behaves when we restart the executors | 19:41 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64537&rra_id=all ze08 swap http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64465&rra_id=all ze05 swap | 19:41 |
| corvus | it goes from 0 to capacity in a very short time | 19:41 |
| *** jamesmcarthur has quit IRC | 19:41 | |
| clarkb | 08 still looking much better on swap | 19:41 |
| corvus | the starting jobs governor is still in place, but it doesn't unduly inhibit things | 19:41 |
| *** wolverineav has quit IRC | 19:41 | |
| *** jamesmcarthur has joined #openstack-infra | 19:41 | |
| corvus | but after running a long time, the executors swap more and spend more system cpu time, and things just slow down. | 19:42 |
| clarkb | gotcha its the turnover on those 4 jobs that the resource contention effects not the raw limit | 19:42 |
| corvus | yep | 19:42 |
| clarkb | ze08 isn't running many more jobs than the other executors. It seems to fall right in the middle | 19:42 |
| corvus | bummer | 19:43 |
| corvus | was there a really big reset i wonder? | 19:44 |
| clarkb | corvus: yes huge nova stack rebase | 19:44 |
| clarkb | and then some smaller gate resets due to the ssh key change problems :/ | 19:45 |
| clarkb | cloudnull: re ssh key change problems ^ we still think those likely to be due to duplicate IP/ IP reuse in rax regions | 19:45 |
| corvus | it could be that a high enough percentage of jobs had to stop at once, and the system was big enough to keep the starting governor at the min | 19:46 |
| corvus | rather, system was 'busy' enough | 19:46 |
| *** rlandy is now known as rlandy|afk | 19:47 | |
| clarkb | mgagne: fwiw those IPs seem to be from the inap cloud this time around example are 198.72.124.136 and 198.72.124.191 | 19:47 |
| openstackgerrit | Christian Berendt proposed openstack/diskimage-builder master: bootloader: add support for GRUB_CMDLINE_LINUX https://review.openstack.org/637036 | 19:47 |
| clarkb | mgagne: perhaps related to the ports issue you mentioned previously | 19:47 |
| mgagne | clarkb: I think Zuul looped on node creation because it couldn't find any free IPs. I'm not sure if there are any left zombie instances with "free" IPs. are there any issues atm since the cleanup? | 19:48 |
| cloudnull | clarkb on it | 19:49 |
| corvus | clarkb: there's still a lot of swap activity on ze08 | 19:49 |
| clarkb | cloudnull: sorry I assumed it was still rax this time | 19:49 |
| cloudnull | oh , not rax? | 19:49 |
| clarkb | cloudnull: at least the two examples I just pulled were inap | 19:49 |
| *** ssusteve has joined #openstack-infra | 19:50 | |
| cloudnull | ah, i see. | 19:50 |
| * cloudnull off it :) | 19:50 | |
| * cloudnull reading back | 19:50 | |
| clarkb | mgagne: http://logs.openstack.org/96/636696/1/gate/openstack-tox-py27/43988b0/job-output.txt.gz#_2019-02-14_19_39_38_475279 is a case from about 10 minutes ago | 19:50 |
| *** rosmaita has left #openstack-infra | 19:50 | |
| mgagne | let me check | 19:50 |
| clarkb | cloudnull: that said if you know what might be causing that, that migh be helpful info for mgagne | 19:50 |
| clarkb | cloudnull: I'm not sure how far your debugging got | 19:51 |
| corvus | installed 9.0.0 again | 19:51 |
| cloudnull | i just saw the ping and assumed a duplicate IP was a RAX problem | 19:51 |
| clarkb | corvus: ya I'm guessing that total memory use is less because jemalloc packs better, but the hot spots are still there and we still swap them | 19:51 |
| cloudnull | I'd not really looked into it just yet | 19:51 |
| clarkb | cloudnull: ya no worries | 19:52 |
| *** Jason_Lee has quit IRC | 19:52 | |
| cloudnull | when we see those, in our case, its normally some zombie instance | 19:52 |
| clarkb | cloudnull: sometimes you discover $thing is a known problem and so and so fixed it and there is some change over here that needs backporting :) | 19:52 |
| cloudnull | in our case thats normally caused by a delete, while recorded as successful, fails to actually shutdown / remove the instance. Not that it happens often, but it happens. | 19:54 |
| clarkb | I've got to step out for a few, but the thrashing from the IP issues seems like it could be a big part of the busy system load | 19:54 |
| clarkb | since that results in gate resets | 19:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: Update gitreview file with correct project name https://review.openstack.org/636791 | 19:54 |
| mgagne | 198.72.124.191 is no longer responding to ping and I can't find it on any compute nodes. 198.72.124.136 responds to ping but I can't find it with my usual method. | 19:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: Update gitreview file with correct project name https://review.openstack.org/636791 | 19:55 |
| openstackgerrit | Merged openstack-infra/zuul-preview master: Update gitreview file with correct project name https://review.openstack.org/636791 | 19:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: WIP: test docker registry https://review.openstack.org/637037 | 20:02 |
| fungi | clarkb: cmurphy: i don't know anything about ask.o.o really. not sure anyone does. we puppet it, but it's mostly unmaintained at this point (from the underlying software point of view anyway) | 20:04 |
| clarkb | I think it was ianw that had started to look at it | 20:06 |
| clarkb | can ask when awake | 20:06 |
| mgagne | clarkb: found it for 198.72.124.136 | 20:07 |
| *** yboaron_ has joined #openstack-infra | 20:07 | |
| fungi | corvus: clarkb: also, the primary playbooks i think do require to rounds of updated because they're already read in at the start of the pulse and responsible for updating their source code thereafter? | 20:07 |
| clarkb | only the playbool that updates systemd config iirc. we use different processesas wego along to mostly address that | 20:07 |
| fungi | cmurphy: tosky: those reused ip addresses generally happen because nova has lost track of an instance which remained running. usually if we track the specific ip addresses we'll see the same ones crop up over and over for that error | 20:08 |
| corvus | ok, yeah, i think i just had a run of bad timing then -- possibly combined with the fact that runs were timing out due to k8s | 20:08 |
| mgagne | found 198.72.124.191 too | 20:09 |
| mgagne | so far, Neutron Queens has been a huge bag of hurts for us... =( | 20:09 |
| *** markvoelker has quit IRC | 20:12 | |
| fungi | and now i'm caught up on scrollback and will dig into my to do list | 20:12 |
| clarkb | mgagne: thats we should keep an eye out for others | 20:13 |
| clarkb | fungi ^ re ssh failures | 20:13 |
| fungi | neutron queens, got it | 20:13 |
| clarkb | er thanks | 20:17 |
| *** trown|lunch is now known as trown | 20:17 | |
| *** jamesmcarthur has quit IRC | 20:19 | |
| *** e0ne has joined #openstack-infra | 20:21 | |
| *** jamesmcarthur has joined #openstack-infra | 20:21 | |
| ianw | clarkb / cmurphy / fungi : i have looked as ask.o.o in the past, but no current work | 20:22 |
| ianw | i think i probably got to the point of "wow this has bitrotted so far it's like starting new" | 20:22 |
| fungi | sounds like a fair assessment | 20:23 |
| *** wolverineav has joined #openstack-infra | 20:23 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: WIP: test docker registry https://review.openstack.org/637037 | 20:23 |
| *** jamesmcarthur has quit IRC | 20:24 | |
| *** jamesmcarthur has joined #openstack-infra | 20:25 | |
| cmurphy | ianw: i think this stack https://review.openstack.org/559178 and https://review.openstack.org/585196 were doing pretty good they just need some love | 20:29 |
| *** wolverineav has quit IRC | 20:30 | |
| cmurphy | but if it's not going to happen any time soon let's turn puppet back on for it so we can puppet4 it? | 20:31 |
| ianw | cmurphy: yeah, sorry i can't remember if it was that not getting reviews that stalled it, or i stopped pushing because I found even more problems | 20:32 |
| ianw | looking back at my notes, i think i was looking at getting it into a virtualenv https://review.openstack.org/#/c/560696/ | 20:34 |
| *** memoussati has quit IRC | 20:34 | |
| *** dave-mccowan has joined #openstack-infra | 20:34 | |
| ianw | given the intervening year, i guess docker is now the new virtualenv | 20:35 |
| *** memoussati has joined #openstack-infra | 20:37 | |
| *** wolverineav has joined #openstack-infra | 20:42 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade some servers to puppet 4 https://review.openstack.org/634726 | 20:44 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade git01.openstack.org to puppet 4 https://review.openstack.org/634727 | 20:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Switch gitea to TLS https://review.openstack.org/637045 | 20:47 |
| corvus | infra-root: \o/ http://38.108.68.64/ is running 1.6.3 -- our automatic upgrade driven by git and ansible worked! | 20:48 |
| fungi | tobiash: frickler: if i cut out the list item for the password variable of the openstack_puppetforge_credentials secret and pipe it into `| base64 -d | sudo openssl rsautl -inkey /var/lib/zuul/k | 20:48 |
| fungi | eys/secrets/project/gerrit/openstack-infra/project-config/0.pem -decrypt -oaep` on the scheduler, i get back the password corresponding to the "openstack" account we have on record | 20:48 |
| fungi | er, sorry, that was for tobias-urdin not tobiash! | 20:48 |
| corvus | clarkb, fungi: can you look at https://review.openstack.org/637045 as an option for gitea ssl termination? | 20:48 |
| fungi | (also pardon the stray newline there) | 20:48 |
| fungi | corvus: checking now | 20:49 |
| corvus | fungi, ianw, clarkb: i found this interesting -- i don't think it's relevant for us, since it's just one service, but gitea has built-in support for letsencrypt. https://docs.gitea.io/en-us/https-setup/ | 20:50 |
| fungi | clarkb: 636681 seems to have run afoul of a post failure in one of the rspec jobs | 20:50 |
| corvus | fungi, clarkb: ^ another inap ssh error | 20:50 |
| corvus | some zuul changes hit it recently too | 20:50 |
| fungi | noted | 20:50 |
| fungi | corvus: regarding gitea and letsencrypt, that's interesting... so it has some feature which just directly does acme negotiation (or vendors its own copy of certbot or something anyway)> | 20:52 |
| corvus | fungi: that's what i'm imagining. | 20:53 |
| openstackgerrit | Merged openstack-infra/system-config master: Update to gitea 1.7.1 https://review.openstack.org/634565 | 20:54 |
| mgagne | corvus, fungi, clarkb: deleted orphan instance | 20:54 |
| *** auristor has joined #openstack-infra | 20:57 | |
| *** memoussati has quit IRC | 21:00 | |
| *** memoussati has joined #openstack-infra | 21:01 | |
| *** memoussati has quit IRC | 21:01 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Enable logging on registry/push/pull jobs https://review.openstack.org/637049 | 21:01 |
| clarkb | corvus: you can reuse the opendev.org cert I already sorted out right? | 21:07 |
| corvus | clarkb: yep | 21:07 |
| *** markvoelker has joined #openstack-infra | 21:09 | |
| *** kjackal has quit IRC | 21:09 | |
| clarkb | mgagne: unsure of where you ended up with cleanupg but returning from lunch and notice 198.72.124.117 198.72.124.131 198.72.124.72 198.72.124.183 all being sad recently | 21:12 |
| mgagne | clarkb: I just checked the IPs mentioned above, I'm context switching between inap-mtl01 and a forest fire atm. | 21:13 |
| mgagne | I will check those | 21:14 |
| clarkb | 198.72.124.59 198.72.124.33 too | 21:16 |
| clarkb | mgagne: do you know if this is something we can clean up on our end? | 21:16 |
| clarkb | or are those rogue instances not going to be exposes to us? | 21:16 |
| mgagne | clarkb: if the instance is unmanaged by Nova, there is little you can do, you need to virsh destroy the instance on the compute node. | 21:16 |
| clarkb | roger | 21:16 |
| clarkb | I wonder if we need to do a whole audit of that /24 or whatever it is | 21:16 |
| tosky | clarkb, mgagne: I have two additional failures: 198.72.124.60 and 198.72.124.70 | 21:17 |
| mgagne | we usually have a tool to find those but so far I'm not getting luck into running it. And since inap-mtl01 is a very busy region, there is a race between listing the instances and actually checking on the compute nodes. | 21:18 |
| *** jamesmcarthur has quit IRC | 21:19 | |
| *** test_weshay has joined #openstack-infra | 21:20 | |
| *** jtomasek has quit IRC | 21:23 | |
| mgagne | done: 198.72.124.117 198.72.124.131 198.72.124.72 198.72.124.183 198.72.124.60 | 21:23 |
| *** eharney has quit IRC | 21:23 | |
| mgagne | couldn't find orphan 198.72.124.70, only one found is legit. | 21:24 |
| clarkb | mgagne: I am finding more :/ 198.72.124.184 198.72.124.151 198.72.124.171 | 21:24 |
| clarkb | I wonder if we shouldn't disable that region so that it can be cleaned up without affecting jobs | 21:25 |
| mgagne | I think it's the best solution for now | 21:25 |
| clarkb | I'll get that patch up and we can decide since the impact seems to be pretty widespread | 21:25 |
| mgagne | ok, better be slower than faster but failing. | 21:25 |
| mgagne | and it's gonna be easier to cleanup if the region isn't used that much. | 21:25 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Disable inap region due to duplicate IPs https://review.openstack.org/637054 | 21:26 |
| clarkb | infra-root ^ fyi | 21:26 |
| *** xek has quit IRC | 21:27 | |
| *** jamesmcarthur has joined #openstack-infra | 21:27 | |
| clarkb | also something like #status notice Jobs are failing due to ssh host key mismatches caused by duplicate IPs in test cloud region. We are disabling the region and will let you know when jobs can be rechecked. | 21:28 |
| clarkb | How does that look for a notice? | 21:28 |
| corvus | clarkb: lgtm. should we force-merge 637054? | 21:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Fix gitea repository root https://review.openstack.org/637055 | 21:29 |
| clarkb | corvus: we probably should | 21:29 |
| corvus | clarkb, fungi: ^ got another small gitea patch -- our reward for getting the playbook working is that it broke the config since i left something out | 21:30 |
| corvus | clarkb: i'll do it | 21:30 |
| clarkb | corvus: would you like to do that? I'll send the notice | 21:30 |
| clarkb | thanks! | 21:30 |
| clarkb | #status notice Jobs are failing due to ssh host key mismatches caused by duplicate IPs in a test cloud region. We are disabling the region and will let you know when jobs can be rechecked. | 21:30 |
| openstackstatus | clarkb: sending notice | 21:30 |
| *** iurygregory has quit IRC | 21:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Disable inap region due to duplicate IPs https://review.openstack.org/637054 | 21:30 |
| clarkb | I'll go edit the nodepool launchers config by hand too to speed that up | 21:31 |
| *** jamesmcarthur has quit IRC | 21:31 | |
| -openstackstatus- NOTICE: Jobs are failing due to ssh host key mismatches caused by duplicate IPs in a test cloud region. We are disabling the region and will let you know when jobs can be rechecked. | 21:31 | |
| clarkb | thats done | 21:32 |
| *** jtomasek has joined #openstack-infra | 21:32 | |
| openstackstatus | clarkb: finished sending notice | 21:33 |
| clarkb | thank you openstackstatus | 21:34 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 21:35 |
| *** agopi_ has joined #openstack-infra | 21:39 | |
| openstackgerrit | Merged openstack-infra/system-config master: Trigger deployment with gitea 1.7.1 https://review.openstack.org/635017 | 21:39 |
| *** whoami-rajat has joined #openstack-infra | 21:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Switch gitea to TLS https://review.openstack.org/637045 | 21:41 |
| corvus | i'm going to squash those 2 changes | 21:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Switch gitea to TLS https://review.openstack.org/637045 | 21:42 |
| *** agopi has quit IRC | 21:42 | |
| *** markvoelker has quit IRC | 21:42 | |
| corvus | clarkb, fungi: can you re-review that? i squashed the app.ini change into it, and also fixed the gitea-init docker build (which i probably broke when i changed how the docker jobs work) | 21:42 |
| fungi | sure | 21:43 |
| clarkb | yup looking | 21:43 |
| fungi | lgtm, thanks! | 21:43 |
| clarkb | corvus: why is the image alias important ther eif we aren't doing a multi stage build in that dockerfile? | 21:43 |
| corvus | clarkb: because if i change the .zuul.yaml, a bunch more jobs are going to run :( | 21:44 |
| corvus | that may not have been your question, let me say more | 21:44 |
| *** test_weshay has quit IRC | 21:44 | |
| clarkb | corvus: more curious how that would have an appreciable change on anything since that alias of gitea-init really only applies to the build there | 21:44 |
| corvus | because .zuul.yaml tells docker to build the 'gitea-init' target, there must be a named target in the dockerfile. so we either need to remove that config from zuul, or add the target name. | 21:45 |
| clarkb | it isn't externally visible? | 21:45 |
| corvus | correct, not externally visible, it's just that the build fails with "failed to reach build target gitea-init in Dockerfile" | 21:45 |
| clarkb | gotcha it is due to asking for gitea-init and we pretend that is it there | 21:45 |
| tobias-urdin | fungi: hm, so maybe there is some spacing in the password or something? | 21:46 |
| corvus | basically, we're running "make foo" and there's no makefile target for that :) so we either define "foo", or change the command to "make" | 21:46 |
| tobias-urdin | or i'm doing it wrong in the playbook where i use the secret | 21:46 |
| clarkb | corvus: ya I just always read FROM lines as take this image from over there and we'll modify it | 21:46 |
| tobias-urdin | since frickler tried this http://paste.openstack.org/show/745081/ manually with the password, i'm not sure whats wrong | 21:46 |
| clarkb | but then we never refer to that source again so the alias shouldn't matter. But I guess that impacts the output name as well? | 21:47 |
| tobias-urdin | fungi: did you have a minute to check the playbook as well to verify it looks correct where i use the secret? | 21:47 |
| corvus | clarkb: yeah, it's a little weird that also means "at the end of this, the result will be called 'foo" | 21:47 |
| corvus | tobias-urdin: can you point at the playbook? | 21:47 |
| clarkb | corvus: yup reading multistage docs again and the as names the end result not the source | 21:48 |
| clarkb | so that does do as you intend. Just slightly confusing as a language | 21:48 |
| corvus | clarkb: yeah, that's not what those words mean in english. | 21:48 |
| corvus | clarkb: the "AS" is ironic. | 21:48 |
| tobias-urdin | sure, secret: https://github.com/openstack-infra/project-config/blob/master/zuul.d/secrets.yaml#L683 | 21:48 |
| tobias-urdin | job: https://github.com/openstack-infra/project-config/blob/master/zuul.d/jobs.yaml#L91 | 21:48 |
| *** e0ne has quit IRC | 21:48 | |
| *** dave-mccowan has quit IRC | 21:49 | |
| tobias-urdin | playbook https://github.com/openstack-infra/project-config/blob/master/playbooks/publish/puppetforge.yaml | 21:49 |
| fungi | tobias-urdin: sure, looks like the secret is passed in via the release-openstack-puppet job with the name "puppetforge" and has a "user" key (which matches the username we have on record) and a "password" key (which decrypts to the corresponding password we have on record). i assume the way forge_username and forge_password are set to strings via substitution of "{{ puppetforge.user }}" and "{{ | 21:52 |
| fungi | puppetforge.password }}" works but i'll check some similar working examples | 21:52 |
| corvus | tobias-urdin: i don't see anything wrong with that :/ | 21:53 |
| *** jtomasek has quit IRC | 21:54 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 21:57 |
| *** ekultails has quit IRC | 21:57 | |
| tobias-urdin | yeah been staring at it as well, it must be something about that though | 21:58 |
| fungi | https://git.zuul-ci.org/cgit/zuul-jobs/tree/roles/upload-forge/tasks/main.yaml seems to in turn use similar substitution to set the forge_username and forge_password contents as username and password variables to the forge_upload library module | 21:58 |
| tobias-urdin | since we've tried using http://paste.openstack.org/show/745081/ with that account | 21:58 |
| *** jamesmcarthur has joined #openstack-infra | 22:00 | |
| pabelanger | did you pass --strip to encrypt_secret? | 22:00 |
| fungi | and the library modle does use username and password keys from the module.params dict and pass them into _forge_auth() in the anticipated order | 22:01 |
| clarkb | encrypt_secret strips input on the cli automatically | 22:01 |
| pabelanger | I thought you needed to use --strip | 22:02 |
| clarkb | pabelanger: but ya that might be a good thing to check, that the secret comes out with expected whitespace | 22:02 |
| fungi | when i decrypt the secret it does end in a newline | 22:02 |
| clarkb | pabelanger: you might if doing it with an input file, but inptu on command line its fine | 22:02 |
| *** trown is now known as trown|outtypewww | 22:02 | |
| pabelanger | Ah, that might be it | 22:02 |
| pabelanger | I was encrypting some clouds.yaml files for ansible-network, and pretty sure had whitespace issue until I used --strip | 22:02 |
| fungi | i have no idea if any of the layers of string substitution in ansible/jinja2/yaml strips it | 22:02 |
| clarkb | corvus: bridge has downgraded kubernetes lib again :/ | 22:03 |
| fungi | but yeah, possible we're passing the trailing \n in as part of the password | 22:03 |
| corvus | fungi: they don't. if you say it has a newline, it will send it. | 22:03 |
| corvus | clarkb: yep. i've been re-upgrading it all day | 22:03 |
| fungi | likely a mistake on my part encrypting it in that case | 22:03 |
| corvus | i would really like that change to land. | 22:03 |
| fungi | i'll upload a replacement secret in moments | 22:03 |
| fungi | corvus: which was the change to stop it upgrading? | 22:04 |
| pabelanger | I think you could use | trim(), but would need to call filter in playbook | 22:04 |
| fungi | er, downgrading i suppose | 22:04 |
| corvus | no, just encrypt the right secret :) | 22:04 |
| pabelanger | +1 | 22:04 |
| clarkb | fungi: if you use the cli input it will strip for you | 22:04 |
| fungi | indeed | 22:04 |
| fungi | i probably put it in a file | 22:04 |
| clarkb | you do have to ^C^D or ^D^D though | 22:04 |
| clarkb | to close the input | 22:04 |
| *** wolverineav has quit IRC | 22:05 | |
| corvus | fungi: https://review.openstack.org/637021 is the change | 22:05 |
| *** tosky has quit IRC | 22:05 | |
| clarkb | I think its safe to recheck now | 22:05 |
| corvus | it keeps failing due to the inap stuff | 22:05 |
| *** tosky has joined #openstack-infra | 22:05 | |
| clarkb | I should send that notice, but still trying to be extra sure | 22:06 |
| clarkb | oh it could be old inap servers we haven't rotated out | 22:06 |
| * clarkb does a listing | 22:06 | |
| clarkb | all inap servers are in use or deleting and max-servers is set to 0 | 22:07 |
| clarkb | so I think it is "safe" now | 22:07 |
| fungi | sounds safe to clear then | 22:07 |
| corvus | clarkb, fungi, pabelanger: there is no difference between cli and stdin. in both cases, encrypt_secret will encrypt *exactly* what you give it. so if you don't want a newline, don't give it one. ^D^D after the value will cause it to terminate reading input without a newline. again, in either case, you can add --strip to the command line and it will strip leading/trailing whitespace. | 22:07 |
| clarkb | how about #status notice The test cloud region using duplicate IPs has been removed from nodepool. Jobs can be rechecked now. | 22:08 |
| *** jamesmcarthur has quit IRC | 22:08 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Enable logging on registry/push/pull jobs https://review.openstack.org/637049 | 22:08 |
| corvus | clarkb: wfm | 22:08 |
| clarkb | corvus: oh hrm I thought it had stripped the one secret I had to replace with opendev. Maybe the ^C^D thing did what I wanted and I assumed another thing | 22:09 |
| *** auristor has quit IRC | 22:09 | |
| *** rlandy|afk is now known as rlandy | 22:09 | |
| fungi | clarkb: corvus: hrm, yeah when i piped it through stdin i did indeed also get a trailing newline in there | 22:11 |
| openstackgerrit | Merged openstack-infra/system-config master: Prep for pbx upgrade to xenial https://review.openstack.org/636681 | 22:12 |
| clarkb | #status notice The test cloud region using duplicate IPs has been removed from nodepool. Jobs can be rechecked now. | 22:12 |
| openstackstatus | clarkb: sending notice | 22:12 |
| corvus | fungi: piped? from what? | 22:13 |
| fungi | corvus: echo | 22:13 |
| corvus | fungi: echo -n or echo? | 22:13 |
| fungi | just echo with no -n, so yeah unsurprising i suppose | 22:13 |
| corvus | i would expect echo -n to work | 22:13 |
| -openstackstatus- NOTICE: The test cloud region using duplicate IPs has been removed from nodepool. Jobs can be rechecked now. | 22:14 | |
| *** jamesmcarthur has joined #openstack-infra | 22:14 | |
| fungi | i'll try that too, but expect --strip will do the trick | 22:14 |
| *** ijw has joined #openstack-infra | 22:15 | |
| fungi | yep, echo -n did what we want too | 22:15 |
| fungi | without --strip in the tool | 22:15 |
| openstackstatus | clarkb: finished sending notice | 22:15 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Update Puppetforge secret https://review.openstack.org/637067 | 22:17 |
| fungi | tobias-urdin: clarkb: pabelanger: corvus: ^ there we go | 22:17 |
| tobias-urdin | \o/ | 22:18 |
| *** jamesmcarthur has quit IRC | 22:18 | |
| *** auristor has joined #openstack-infra | 22:18 | |
| *** jamesmcarthur has joined #openstack-infra | 22:18 | |
| fungi | once it merges (hopefully in just a few minutes) i can retrigger the reenqueue that tag yet agani | 22:21 |
| fungi | again too | 22:21 |
| *** yboaron_ has quit IRC | 22:21 | |
| *** tjgresha has joined #openstack-infra | 22:21 | |
| *** dave-mccowan has joined #openstack-infra | 22:22 | |
| *** ijw has quit IRC | 22:24 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 22:27 |
| clarkb | oh wow storyboard upgrade is tomorrow. fungi anything need to be done before then? | 22:28 |
| fungi | clarkb: not that i know of. as a reminder, the plan is at https://etherpad.openstack.org/p/gCj4NfcnbW | 22:29 |
| fungi | it's straightforward and should only be a few minutes downtime | 22:29 |
| fungi | the db dump/load ought to take less time than dns propagation, based on my evaluations with the (larger) storyboard-dev db on a smaller machine | 22:30 |
| clarkb | thats a fun size difference due to how we use them differently. Sounds good | 22:30 |
| *** yamamoto has joined #openstack-infra | 22:30 | |
| *** wolverineav has joined #openstack-infra | 22:30 | |
| *** armax has quit IRC | 22:30 | |
| fungi | yeah, the massive projects diablo_rojo has done import testing for is what leads sb-dev to be so much larger of a db | 22:30 |
| *** wolverineav has quit IRC | 22:35 | |
| fungi | i've added the new hiera keys for https://review.openstack.org/636952 on bridge.o.o if another infra-root wants to review that. the osf is relocating the foundation member databases to vexxhost and adding ssl/tls client cert auth for mysql | 22:36 |
| fungi | that change is to basically do a dry run for openstackid-dev | 22:36 |
| clarkb | fungi: approved | 22:37 |
| fungi | that way they can be confident it's working as intended before they do the same for the production db a week from tuesday | 22:38 |
| clarkb | __ | 22:38 |
| fungi | thanks clarkb! | 22:38 |
| clarkb | er | 22:38 |
| clarkb | ++ | 22:38 |
| *** sreejithp_ has quit IRC | 22:39 | |
| *** markvoelker has joined #openstack-infra | 22:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update Puppetforge secret https://review.openstack.org/637067 | 22:42 |
| openstackgerrit | Merged openstack-infra/zuul master: Cache github PR shas https://review.openstack.org/636764 | 22:42 |
| clarkb | corvus: things are moving again ^ | 22:43 |
| *** agopi_ has quit IRC | 22:43 | |
| clarkb | now do we want to try and restart now since things are backlogged/slow anyway or let them clear out | 22:43 |
| openstackgerrit | Merged openstack-infra/system-config master: Switch gitea to TLS https://review.openstack.org/637045 | 22:45 |
| fungi | happy to help with a restart in the next 10 minutes, then i'm on a conference call for a while | 22:48 |
| clarkb | let me check when puppet will do the install for us | 22:48 |
| *** wolverineav has joined #openstack-infra | 22:49 | |
| clarkb | I think we are about 25 minutes away from a puppet install of zuul on zuul01 | 22:49 |
| fungi | ahh, happy to half-watch a restart in progress while on my conference call in that case | 22:49 |
| clarkb | I can probably do a restart after that as long as we don't think it will eat into backlog too much (which is my next thing to check) | 22:49 |
| clarkb | we are headed in the right direction now with backlog so a restart probably isn't too terrible | 22:50 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 22:50 |
| *** jamesmcarthur has quit IRC | 22:51 | |
| *** wolverineav has quit IRC | 22:53 | |
| fungi | tobias-urdin: i've reenqueued the puppet-aodh 14.2.0 tag now | 22:55 |
| fungi | that should complete before puppet updates zuul01.o.o anyway | 22:56 |
| fungi | so ought not get in the way of the restart | 22:56 |
| *** wolverineav has joined #openstack-infra | 22:56 | |
| openstackgerrit | Merged openstack-infra/system-config master: Temporarily stop installing openshift https://review.openstack.org/637021 | 23:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix pull-from-intermediate-registry artifacts error https://review.openstack.org/637072 | 23:00 |
| corvus | yay, maybe my sysiphean task will end soon! | 23:01 |
| *** jamesmcarthur has joined #openstack-infra | 23:01 | |
| fungi | there can't be *that* much of a boulder left now | 23:01 |
| *** eernst has joined #openstack-infra | 23:01 | |
| fungi | it's been worn down by all the rolling | 23:01 |
| *** tkajinam has joined #openstack-infra | 23:02 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix undefined attrs in registry push/pull roles https://review.openstack.org/637072 | 23:03 |
| tobias-urdin | fungi: yay successful upload | 23:03 |
| fungi | mic drop! | 23:04 |
| fungi | excellent work | 23:04 |
| corvus | yay! | 23:04 |
| fungi | sorry my stray newline gummed up the last of it | 23:04 |
| openstackgerrit | Merged openstack-infra/system-config master: Update puppet config for openstackid-dev node https://review.openstack.org/636952 | 23:05 |
| tobias-urdin | here comes the annoying part though, i'm sorry to drop you all this crap but there is a lot of failed releases | 23:05 |
| *** jamesmcarthur has quit IRC | 23:05 | |
| tobias-urdin | the last release was a bump of pretty much all modules | 23:05 |
| *** jamesmcarthur has joined #openstack-infra | 23:06 | |
| tobias-urdin | here is the commit, it was the stein-2 milestone https://github.com/openstack/releases/commit/d815ac8ed2b4043aaa2cb3be2a4d2c3b398fc4df | 23:06 |
| *** markvoelker has quit IRC | 23:12 | |
| openstackgerrit | Jan Kundrát proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 23:12 |
| clarkb | there should be a rule we have with the release team where we release a single thing whenever there are job changes :) | 23:15 |
| clarkb | tobias-urdin: so all of those releases failed and need to be reeunqueud except for the on that fungi just enqueued? | 23:15 |
| tobias-urdin | yeah, all of them except for the first one (puppet-aodh) that he reenqueued | 23:16 |
| clarkb | tobias-urdin: ok we need to gather some data (probably in an etherpad?) we need the project, tag, and tag sha1 (not the sha1 the tag points to but the sha1 of the tag itself) | 23:16 |
| clarkb | corvus: fungi zuul==3.5.1.dev68 # git sha 275cbc9 is installed on zuul01 now which looks correct to me for the PR caching change | 23:18 |
| clarkb | do we want to go ahead and do a restart now? | 23:18 |
| clarkb | I can do it just have to remember the steps | 23:18 |
| clarkb | fungi: you did have to restart the web process too ? | 23:18 |
| clarkb | the way github renders that diff you can't just copy paste the text because you lose the +'s | 23:20 |
| * clarkb does a git show locally | 23:20 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 23:22 |
| tobias-urdin | ok, i'll compile a list | 23:23 |
| openstackgerrit | Merged openstack-infra/system-config master: Update DNS documentation https://review.openstack.org/633569 | 23:23 |
| *** wolverineav has quit IRC | 23:23 | |
| fungi | clarkb: yeah, i ended up restarting zuul-web to get status content | 23:24 |
| *** wolverineav has joined #openstack-infra | 23:26 | |
| tobias-urdin | so sha1 of tag would be $(git show-ref -s 14.2.0) for example | 23:26 |
| clarkb | tobias-urdin: ya that looks right | 23:27 |
| *** rh-jelabarre has quit IRC | 23:29 | |
| clarkb | ok I think I'm ina good spot to restart zuul scheduler | 23:30 |
| clarkb | process would be store queues, stop scheduler, start scheduler, wait for config to be loaded, reload queues. And restart zuul-web at some point | 23:31 |
| clarkb | fungi: ^ is that how you did it? corvus ^ any reason to not do that nowish? | 23:31 |
| fungi | wfm | 23:32 |
| openstackgerrit | Hunter Werkman proposed openstack-infra/project-config master: Add puppet-gitrepo project to puppet-OpenStack https://review.openstack.org/637076 | 23:32 |
| fungi | let #openstack-release know first? | 23:32 |
| corvus | clarkb: sgtm | 23:32 |
| *** eernst has quit IRC | 23:33 | |
| clarkb | ok saving queues now | 23:33 |
| fungi | config-core reviewers: 637076 there is some university student interns, be gentle! ;) | 23:33 |
| *** yamamoto has quit IRC | 23:33 | |
| clarkb | I have asked the zuul-scheduler to stop | 23:34 |
| corvus | clarkb: re ^ maybe that should be non-infra? | 23:34 |
| corvus | sorry, was re 076 | 23:34 |
| clarkb | and it is starting now | 23:34 |
| clarkb | corvus: ya we can always consume a generic module if we like, doesn't have to be infra (and we are moving away from puppet) | 23:35 |
| corvus | yes -- i mostly suggest that because i don't expect us to use it | 23:35 |
| clarkb | indeed | 23:36 |
| clarkb | re zuul-web, I stopped and started it but the start doesn't seem to have worked. I'm going to assume it will fix itself if I stop start again once shceduler has configs loaded | 23:36 |
| clarkb | but if that doesn't happen then we may need eyeballs | 23:37 |
| corvus | clarkb: remove the pidfile | 23:37 |
| clarkb | corvus: will do | 23:37 |
| clarkb | that was it | 23:38 |
| clarkb | enqueuing changes now | 23:39 |
| tobias-urdin | corvus: raw data http://paste.openstack.org/show/745127/ | 23:44 |
| tobias-urdin | here is commands http://paste.openstack.org/show/745128/ | 23:44 |
| tobias-urdin | :) | 23:44 |
| *** rlandy is now known as rlandy|bbl | 23:44 | |
| clarkb | I assume that was for me | 23:45 |
| clarkb | fungi: do you have your puppet-aodh enqueue command nearby? I can apply it to the above pastes and we can run a mass enqueue assuming we are happy with the resutls of the aodh enqueu | 23:46 |
| fungi | clarkb: just a sec, sure | 23:46 |
| fungi | sudo zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=release --project=openstack/puppet-aodh --ref=refs/tags/14.2.0 --newrev=617ffad84b633618490ca1023f8a31d9694b31a9 | 23:47 |
| clarkb | I think I found it | 23:47 |
| clarkb | ya thanks | 23:47 |
| fungi | you need to show-ref the tag to get the newrev unless you have the failure links handy | 23:47 |
| clarkb | fungi: ya tobias-urdin got all that data for me in the paste above | 23:48 |
| clarkb | I'll do puppet-barbican and double check things then do a larger chunk | 23:49 |
| fungi | the script in https://review.openstack.org/613676 can just be fed the url of a log for a previous run | 23:50 |
| tobias-urdin | clarkb: oh yeah | 23:50 |
| fungi | and will output the reenqueue command | 23:50 |
| clarkb | its pretty easy with visual mode vim and the columnar data too | 23:50 |
| *** tosky has quit IRC | 23:51 | |
| *** agopi_ has joined #openstack-infra | 23:51 | |
| tobias-urdin | fungi: clean utility right there | 23:52 |
| *** mattw4 has quit IRC | 23:53 | |
| tobias-urdin | i'll see if can work with the release team to propose some pre-release check that queries puppetforge api to prevent failed release jobs if somebody duplicates versions | 23:54 |
| tobias-urdin | or maybe it fails on that already, then it's probably fine | 23:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Fix gitea k8s files https://review.openstack.org/637080 | 23:58 |
| corvus | clarkb, fungi: ^ i will be very happy when we figure out testing of k8s stuff ^ | 23:59 |
| clarkb | tobias-urdin: zuul says puppet-barbican job ran successfully, want to double check things in puppetforce and if it looks good I'll enqueue the others | 23:59 |
| tobias-urdin | sure, give me a sec | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!