| *** lifeless_ has joined #openstack-infra | 00:01 | |
| *** lifeless has quit IRC | 00:02 | |
| *** AJaeger has joined #openstack-infra | 00:03 | |
| *** rlandy|rover is now known as rlandy|rover|bbl | 00:11 | |
| *** masuberu has joined #openstack-infra | 00:15 | |
| *** pahuang has quit IRC | 00:18 | |
| *** masber has quit IRC | 00:19 | |
| *** dingyichen has joined #openstack-infra | 00:20 | |
| *** vaidy has quit IRC | 00:24 | |
| *** isviridov_away has quit IRC | 00:24 | |
| *** jamesmcarthur has joined #openstack-infra | 00:31 | |
| *** pahuang has joined #openstack-infra | 00:32 | |
| *** jamesmcarthur has quit IRC | 00:36 | |
| *** rcernin_ has joined #openstack-infra | 00:37 | |
| *** rcernin has quit IRC | 00:37 | |
| *** jamesmcarthur has joined #openstack-infra | 00:39 | |
| *** eernst has quit IRC | 00:41 | |
| *** eernst has joined #openstack-infra | 00:42 | |
| *** jamesmcarthur has quit IRC | 00:44 | |
| *** jamesmcarthur has joined #openstack-infra | 00:46 | |
| *** eernst has quit IRC | 00:47 | |
| *** eernst has joined #openstack-infra | 00:49 | |
| *** eernst has quit IRC | 00:54 | |
| *** hongbin has joined #openstack-infra | 00:57 | |
| *** jamesmcarthur has quit IRC | 01:04 | |
| *** jamesmcarthur has joined #openstack-infra | 01:06 | |
| *** harlowja has quit IRC | 01:07 | |
| *** dpawlik has joined #openstack-infra | 01:10 | |
| *** kiennt26 has joined #openstack-infra | 01:10 | |
| *** jamesmcarthur has quit IRC | 01:11 | |
| *** kjackal has quit IRC | 01:12 | |
| *** jamesmcarthur has joined #openstack-infra | 01:13 | |
| *** dpawlik has quit IRC | 01:14 | |
| *** jamesmcarthur has quit IRC | 01:17 | |
| *** zhangfei has joined #openstack-infra | 01:21 | |
| *** caphrim007 has joined #openstack-infra | 01:24 | |
| *** caphrim007 has quit IRC | 01:29 | |
| *** rlandy|rover|bbl is now known as rlandy|rover | 01:35 | |
| *** jamesmcarthur has joined #openstack-infra | 01:40 | |
| *** r-daneel has joined #openstack-infra | 01:43 | |
| *** ramishra has joined #openstack-infra | 01:50 | |
| *** namnh has joined #openstack-infra | 01:53 | |
| *** jamesmcarthur has quit IRC | 01:53 | |
| *** jesslampe has quit IRC | 01:54 | |
| *** rwsu has joined #openstack-infra | 01:59 | |
| *** jamesmcarthur has joined #openstack-infra | 02:01 | |
| *** namnh_ has joined #openstack-infra | 02:02 | |
| *** namnh has quit IRC | 02:02 | |
| *** namnh_ has quit IRC | 02:02 | |
| *** namnh has joined #openstack-infra | 02:03 | |
| *** armaan_ has joined #openstack-infra | 02:05 | |
| *** jamesmcarthur has quit IRC | 02:05 | |
| *** armaan has quit IRC | 02:06 | |
| *** ihar has quit IRC | 02:06 | |
| *** ihar has joined #openstack-infra | 02:07 | |
| *** yamamoto has joined #openstack-infra | 02:15 | |
| *** caphrim007 has joined #openstack-infra | 02:22 | |
| *** jamesmcarthur has joined #openstack-infra | 02:24 | |
| *** caphrim007 has quit IRC | 02:26 | |
| *** caphrim007 has joined #openstack-infra | 02:26 | |
| *** gongysh has joined #openstack-infra | 02:27 | |
| *** jamesmcarthur has quit IRC | 02:28 | |
| *** kiennt26 has quit IRC | 02:28 | |
| *** rosmaita has quit IRC | 02:32 | |
| *** lihi has quit IRC | 02:36 | |
| *** lihi has joined #openstack-infra | 02:37 | |
| *** lifeless_ has quit IRC | 02:38 | |
| *** lifeless has joined #openstack-infra | 02:39 | |
| *** jamesmcarthur has joined #openstack-infra | 02:40 | |
| *** kien-ha has quit IRC | 02:40 | |
| *** kiennt26 has joined #openstack-infra | 02:43 | |
| *** jamesmcarthur has quit IRC | 02:44 | |
| *** jamesmcarthur has joined #openstack-infra | 02:45 | |
| *** rossella_s has quit IRC | 02:47 | |
| *** rossella_s has joined #openstack-infra | 02:47 | |
| *** vaidy has joined #openstack-infra | 02:49 | |
| *** jamesmcarthur has quit IRC | 02:50 | |
| *** isviridov_away has joined #openstack-infra | 02:50 | |
| *** rossella_s has quit IRC | 02:53 | |
| *** dave-mccowan has quit IRC | 02:54 | |
| *** rossella_s has joined #openstack-infra | 02:56 | |
| *** vivsoni has quit IRC | 02:57 | |
| *** vivsoni has joined #openstack-infra | 02:57 | |
| *** alex_xu has quit IRC | 03:00 | |
| *** alex_xu has joined #openstack-infra | 03:01 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Initial grafana AFS dashboard https://review.openstack.org/572083 | 03:02 |
|---|---|---|
| *** rossella_s has quit IRC | 03:05 | |
| *** myoung|off is now known as myoung | 03:08 | |
| *** rossella_s has joined #openstack-infra | 03:08 | |
| *** jamesmcarthur has joined #openstack-infra | 03:12 | |
| *** StaceyF has joined #openstack-infra | 03:16 | |
| StaceyF | Hello, I am in the process of putting in 3rd party gating for airship-deckhand, the project has added the CI user to the group and given the ability of Verified +1 -1, but it looks like the result is not being published to gerrit. I'm not sure if I have something set up incorrectly or not. I went through the documentation for 3rd party gating. | 03:18 |
| *** jamesmcarthur has quit IRC | 03:18 | |
| StaceyF | Apparently all I had to do was post here, just saw it finally post a result. | 03:18 |
| StaceyF | I'm back I was wrong :) | 03:20 |
| *** ramishra has quit IRC | 03:22 | |
| *** rossella_s has quit IRC | 03:23 | |
| *** ramishra has joined #openstack-infra | 03:24 | |
| *** rossella_s has joined #openstack-infra | 03:26 | |
| *** rh-jelabarre has quit IRC | 03:27 | |
| *** rwsu has quit IRC | 03:30 | |
| *** kjackal has joined #openstack-infra | 03:32 | |
| *** eernst has joined #openstack-infra | 03:34 | |
| *** rh-jelabarre has joined #openstack-infra | 03:36 | |
| *** nhandler has quit IRC | 03:36 | |
| *** nhandler_ has joined #openstack-infra | 03:36 | |
| *** rossella_s has quit IRC | 03:37 | |
| *** esbergl__ has quit IRC | 03:38 | |
| *** anteaya has quit IRC | 03:38 | |
| *** rossella_s has joined #openstack-infra | 03:39 | |
| *** yamahata has quit IRC | 03:41 | |
| clarkb | StaceyF: is it leaving any comments? they labels are case sensitive in gerrit too | 03:42 |
| clarkb | also what tool is talking ti gerrit? | 03:43 |
| *** udesale has joined #openstack-infra | 03:44 | |
| StaceyF | just Jenkins, I'm not sure what you mean by the labels are case sensitive? It's not leaving any comments at all, it is triggering the job in Jenkins though | 03:44 |
| *** jamesmcarthur has joined #openstack-infra | 03:45 | |
| *** lpetrut has joined #openstack-infra | 03:48 | |
| clarkb | If you leave a commwnt for verified -1 it wont work because it expects Verified -1 | 03:54 |
| *** rossella_s has quit IRC | 03:55 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: project-config-grafyaml: pass source to tox https://review.openstack.org/572251 | 03:55 |
| *** rossella_s has joined #openstack-infra | 03:57 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Initial grafana AFS dashboard https://review.openstack.org/572083 | 03:58 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Use tox for grafyaml validation https://review.openstack.org/572252 | 03:58 |
| *** jesslampe has joined #openstack-infra | 03:59 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: project-config-grafyaml: pass source to tox https://review.openstack.org/572251 | 03:59 |
| *** rh-jelabarre has quit IRC | 03:59 | |
| *** germs has quit IRC | 04:01 | |
| *** germs has joined #openstack-infra | 04:02 | |
| *** germs has quit IRC | 04:02 | |
| *** germs has joined #openstack-infra | 04:02 | |
| *** germs has quit IRC | 04:06 | |
| *** jamesmcarthur has quit IRC | 04:08 | |
| openstackgerrit | Samuel Cassiba proposed openstack-infra/project-config master: Import openstackclient to Chef OpenStack https://review.openstack.org/571496 | 04:09 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: project-config-grafyaml: pass source to tox https://review.openstack.org/572251 | 04:12 |
| *** kjackal has quit IRC | 04:12 | |
| *** lpetrut has quit IRC | 04:20 | |
| *** links has joined #openstack-infra | 04:22 | |
| *** kiennt26 has quit IRC | 04:29 | |
| *** gongysh has quit IRC | 04:31 | |
| *** vivsoni has quit IRC | 04:38 | |
| StaceyF | going to take a break til tomorrow, thanks for the help | 04:39 |
| *** vivsoni has joined #openstack-infra | 04:42 | |
| *** jamesmcarthur has joined #openstack-infra | 04:50 | |
| *** dhajare has joined #openstack-infra | 04:50 | |
| *** aeng_ has quit IRC | 04:53 | |
| *** rwsu has joined #openstack-infra | 04:54 | |
| *** jamesmcarthur has quit IRC | 04:55 | |
| *** CrayZee has joined #openstack-infra | 04:57 | |
| *** gyee has quit IRC | 05:00 | |
| *** lpetrut has joined #openstack-infra | 05:04 | |
| *** hongbin has quit IRC | 05:06 | |
| *** ccamacho has quit IRC | 05:07 | |
| *** psachin has joined #openstack-infra | 05:14 | |
| *** lpetrut has quit IRC | 05:15 | |
| *** lpetrut has joined #openstack-infra | 05:16 | |
| *** salv-orlando has joined #openstack-infra | 05:17 | |
| *** salv-orlando has quit IRC | 05:17 | |
| *** rlandy|rover has quit IRC | 05:22 | |
| *** psachin has quit IRC | 05:22 | |
| *** lpetrut has quit IRC | 05:25 | |
| *** cshastri has joined #openstack-infra | 05:26 | |
| *** jamesmcarthur has joined #openstack-infra | 05:33 | |
| *** eernst has quit IRC | 05:34 | |
| *** jamesmcarthur has quit IRC | 05:38 | |
| *** lifeless has quit IRC | 05:40 | |
| *** lifeless has joined #openstack-infra | 05:41 | |
| *** gongysh has joined #openstack-infra | 05:43 | |
| *** ccamacho has joined #openstack-infra | 05:46 | |
| *** yolanda_ is now known as yolanda | 05:48 | |
| *** myoung is now known as myoung|zzz | 05:49 | |
| *** ccamacho has quit IRC | 05:49 | |
| *** pgadiya has joined #openstack-infra | 05:50 | |
| *** pgadiya has quit IRC | 05:50 | |
| *** jamesmcarthur has joined #openstack-infra | 05:54 | |
| *** dhajare has quit IRC | 05:54 | |
| *** vickyc has quit IRC | 05:55 | |
| *** jamesmcarthur has quit IRC | 05:59 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: project-config-grafyaml: pass source to tox https://review.openstack.org/572251 | 06:01 |
| *** germs has joined #openstack-infra | 06:03 | |
| *** germs has quit IRC | 06:03 | |
| *** germs has joined #openstack-infra | 06:03 | |
| *** germs has quit IRC | 06:08 | |
| *** iranzo has joined #openstack-infra | 06:08 | |
| *** links has quit IRC | 06:08 | |
| *** ispp has joined #openstack-infra | 06:12 | |
| *** jaosorior has joined #openstack-infra | 06:14 | |
| *** isssp has quit IRC | 06:14 | |
| *** jamesmcarthur has joined #openstack-infra | 06:15 | |
| *** ccamacho has joined #openstack-infra | 06:19 | |
| *** jamesmcarthur has quit IRC | 06:20 | |
| *** links has joined #openstack-infra | 06:26 | |
| *** dhajare has joined #openstack-infra | 06:26 | |
| *** salv-orlando has joined #openstack-infra | 06:32 | |
| *** salv-orlando has quit IRC | 06:32 | |
| *** cshastri has quit IRC | 06:32 | |
| openstackgerrit | Merged openstack-infra/project-config master: nodepool: Set min-ready to '1' for opensuse-150 https://review.openstack.org/572030 | 06:33 |
| *** salv-orlando has joined #openstack-infra | 06:33 | |
| *** andreas_s has joined #openstack-infra | 06:33 | |
| *** jamesmcarthur has joined #openstack-infra | 06:36 | |
| *** cshastri has joined #openstack-infra | 06:37 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: project-config-grafyaml: pass source to tox https://review.openstack.org/572251 | 06:37 |
| AJaeger | ianw: hope this works better ^ | 06:37 |
| *** jamesmcarthur has quit IRC | 06:41 | |
| *** jesslampe has quit IRC | 06:41 | |
| *** armaan_ has quit IRC | 06:44 | |
| *** armaan has joined #openstack-infra | 06:44 | |
| *** aojea has joined #openstack-infra | 06:46 | |
| *** salv-orlando has quit IRC | 06:52 | |
| hwoarang | ianw: thank you for merging the opensuse-150. However zuul complains 'The nodeset "opensuse-150" was not found.' when I try to add a job. Do you know what's missing? or do I need to wait sometime until zuul picks up the changes? | 06:53 |
| *** sshnaidm has joined #openstack-infra | 06:53 | |
| *** pcaruana has joined #openstack-infra | 06:54 | |
| *** dpawlik has joined #openstack-infra | 06:56 | |
| *** yamamoto_ has joined #openstack-infra | 06:57 | |
| *** jamesmcarthur has joined #openstack-infra | 06:57 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: project-config-grafyaml: pass source to tox https://review.openstack.org/572251 | 06:57 |
| *** slaweq has joined #openstack-infra | 06:59 | |
| *** lpetrut has joined #openstack-infra | 06:59 | |
| *** yamamoto has quit IRC | 07:00 | |
| *** jbadiapa has joined #openstack-infra | 07:01 | |
| AJaeger | hwoarang: where does zuul complain? | 07:01 |
| hwoarang | AJaeger: this one https://review.openstack.org/#/c/572109/ | 07:01 |
| * hwoarang grabs a coffee | 07:01 | |
| AJaeger | hwoarang: link for you - while I check your review - http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/nodesets.yaml | 07:02 |
| hwoarang | aha | 07:02 |
| *** jamesmcarthur has quit IRC | 07:02 | |
| hwoarang | missed that :/ | 07:02 |
| *** alexchadin has joined #openstack-infra | 07:03 | |
| AJaeger | hwoarang: yeah, send a change for that one, please | 07:03 |
| hwoarang | on it | 07:03 |
| *** pcaruana is now known as pcaruana|worksho | 07:03 | |
| AJaeger | hwoarang: grab your coffee first ;) | 07:03 |
| hwoarang | :) | 07:04 |
| hwoarang | i'm guessing i should not add any legacy definitions | 07:04 |
| *** sshnaidm is now known as sshnaidm|brq | 07:05 | |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/568209 , https://review.openstack.org/#/c/572016/ , and https://review.openstack.org/#/c/572017/ , please? | 07:11 |
| AJaeger | hwoarang: correct | 07:11 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove unused legacy jobs https://review.openstack.org/572276 | 07:12 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/openstack-zuul-jobs master: zuul.d: nodesets: Add openSUSE Leap 15 definition https://review.openstack.org/572277 | 07:15 |
| *** janki has joined #openstack-infra | 07:15 | |
| *** ykarel has joined #openstack-infra | 07:16 | |
| *** dpawlik has quit IRC | 07:16 | |
| *** hashar has joined #openstack-infra | 07:17 | |
| *** ykarel is now known as ykarel|training | 07:17 | |
| *** rajinir has quit IRC | 07:18 | |
| *** jamesmcarthur has joined #openstack-infra | 07:18 | |
| *** dpawlik has joined #openstack-infra | 07:21 | |
| *** jamesmcarthur has quit IRC | 07:22 | |
| *** jbadiapa has quit IRC | 07:32 | |
| *** jbadiapa has joined #openstack-infra | 07:33 | |
| *** gfidente has joined #openstack-infra | 07:37 | |
| *** gfidente has joined #openstack-infra | 07:37 | |
| *** amoralej|off is now known as amoralej | 07:37 | |
| *** jamesmcarthur has joined #openstack-infra | 07:39 | |
| *** lifeless has quit IRC | 07:40 | |
| *** lifeless has joined #openstack-infra | 07:42 | |
| *** jamesmcarthur has quit IRC | 07:44 | |
| *** StaceyF has quit IRC | 07:46 | |
| *** anteaya has joined #openstack-infra | 07:49 | |
| *** pcaruana|worksho is now known as pcaruana | 07:50 | |
| *** jpena|off is now known as jpena | 07:50 | |
| *** shardy has joined #openstack-infra | 07:58 | |
| ianw | hwoarang: you'll need to add that in zuul-jobs | 07:59 |
| hwoarang | yep ^^ :) | 07:59 |
| ianw | oh haha, sorry, started reading at the top | 07:59 |
| hwoarang | hopefully the last missing bit | 07:59 |
| *** zhangfei has quit IRC | 08:00 | |
| *** jamesmcarthur has joined #openstack-infra | 08:00 | |
| openstackgerrit | Artem Goncharov proposed openstack-infra/nodepool master: Add support for specifying security_group in nodepool https://review.openstack.org/570857 | 08:01 |
| *** jpich has joined #openstack-infra | 08:03 | |
| *** germs has joined #openstack-infra | 08:04 | |
| *** germs has quit IRC | 08:04 | |
| *** germs has joined #openstack-infra | 08:04 | |
| *** jamesmcarthur has quit IRC | 08:05 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: zuul.d: nodesets: Add openSUSE Leap 15 definition https://review.openstack.org/572277 | 08:08 |
| *** germs has quit IRC | 08:08 | |
| *** zhangfei has joined #openstack-infra | 08:12 | |
| *** pblaho has joined #openstack-infra | 08:14 | |
| *** tosky has joined #openstack-infra | 08:15 | |
| *** priteau has joined #openstack-infra | 08:15 | |
| *** jamesmcarthur has joined #openstack-infra | 08:21 | |
| *** priteau has quit IRC | 08:25 | |
| *** jamesmcarthur has quit IRC | 08:26 | |
| *** sambetts_ is now known as sambetts|afk | 08:28 | |
| *** roman_g has joined #openstack-infra | 08:28 | |
| *** andreas_s has quit IRC | 08:33 | |
| *** ralonsoh has joined #openstack-infra | 08:33 | |
| *** andreas_s has joined #openstack-infra | 08:34 | |
| *** e0ne has joined #openstack-infra | 08:37 | |
| *** namnh has quit IRC | 08:37 | |
| *** jamesmcarthur has joined #openstack-infra | 08:42 | |
| *** ralonsoh_ has joined #openstack-infra | 08:42 | |
| *** ralonsoh has quit IRC | 08:44 | |
| *** derekh has joined #openstack-infra | 08:46 | |
| *** jamesmcarthur has quit IRC | 08:47 | |
| *** priteau has joined #openstack-infra | 08:48 | |
| *** salv-orlando has joined #openstack-infra | 08:49 | |
| *** salv-orlando has quit IRC | 08:55 | |
| *** tdasilva has quit IRC | 09:02 | |
| *** jamesmcarthur has joined #openstack-infra | 09:03 | |
| *** electrofelix has joined #openstack-infra | 09:03 | |
| *** jamesmcarthur has quit IRC | 09:08 | |
| *** owalsh_ is now known as owalsh | 09:08 | |
| *** lxkong has quit IRC | 09:09 | |
| *** alexchadin has quit IRC | 09:13 | |
| *** gongysh has quit IRC | 09:14 | |
| *** alexchadin has joined #openstack-infra | 09:22 | |
| *** zhangfei has quit IRC | 09:22 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: DNM Fix etc/hosts https://review.openstack.org/572313 | 09:24 |
| *** jamesmcarthur has joined #openstack-infra | 09:24 | |
| *** lifeless has quit IRC | 09:25 | |
| *** lifeless has joined #openstack-infra | 09:28 | |
| *** jamesmcarthur has quit IRC | 09:29 | |
| openstackgerrit | Merged openstack-dev/cookiecutter master: Fixed two issues regarding cookiecutter template https://review.openstack.org/571924 | 09:30 |
| *** lxkong has joined #openstack-infra | 09:32 | |
| *** zhangfei has joined #openstack-infra | 09:38 | |
| *** sambetts|afk is now known as sambetts] | 09:41 | |
| *** sambetts] is now known as sambetts | 09:41 | |
| *** pbourke has quit IRC | 09:41 | |
| *** pbourke has joined #openstack-infra | 09:42 | |
| *** andreas_s has quit IRC | 09:44 | |
| *** andreas_s has joined #openstack-infra | 09:45 | |
| *** jamesmcarthur has joined #openstack-infra | 09:45 | |
| *** stakeda has quit IRC | 09:46 | |
| *** jamesmcarthur has quit IRC | 09:50 | |
| *** eandersson has quit IRC | 09:50 | |
| *** salv-orlando has joined #openstack-infra | 09:51 | |
| *** salv-orlando has quit IRC | 09:55 | |
| *** dhajare has quit IRC | 09:57 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/infra-manual master: Mention keywords to highlight tech support https://review.openstack.org/572323 | 09:58 |
| *** andreas_s has quit IRC | 09:59 | |
| *** germs has joined #openstack-infra | 10:04 | |
| *** jamesmcarthur has joined #openstack-infra | 10:06 | |
| *** zoli is now known as zoli|lunch | 10:07 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Retire kolla-kubernetes project - step 2 end project gating https://review.openstack.org/568209 | 10:09 |
| *** germs has quit IRC | 10:09 | |
| *** jamesmcarthur has quit IRC | 10:11 | |
| *** dhajare has joined #openstack-infra | 10:11 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Include hostnames in /etc/hosts https://review.openstack.org/572313 | 10:14 |
| openstackgerrit | Merged openstack-infra/project-config master: Move planet unittests in-tree https://review.openstack.org/572016 | 10:23 |
| *** bhavik1 has joined #openstack-infra | 10:23 | |
| *** jamesmcarthur has joined #openstack-infra | 10:27 | |
| *** dbecker has joined #openstack-infra | 10:31 | |
| *** jamesmcarthur has quit IRC | 10:32 | |
| *** Bhujay has joined #openstack-infra | 10:38 | |
| *** rcernin_ has quit IRC | 10:42 | |
| *** alexchadin has quit IRC | 10:45 | |
| *** dbecker has quit IRC | 10:45 | |
| *** alexchadin has joined #openstack-infra | 10:46 | |
| *** andreas_s has joined #openstack-infra | 10:48 | |
| *** jamesmcarthur has joined #openstack-infra | 10:48 | |
| *** jamesmcarthur has quit IRC | 10:49 | |
| *** dhajare has quit IRC | 10:49 | |
| *** bhavik1 has quit IRC | 10:50 | |
| *** alexchadin has quit IRC | 10:50 | |
| *** _ari_ has quit IRC | 10:50 | |
| *** tosky__ has joined #openstack-infra | 10:51 | |
| *** _ari_ has joined #openstack-infra | 10:51 | |
| *** tosky has quit IRC | 10:52 | |
| openstackgerrit | Artem Goncharov proposed openstack-infra/nodepool master: Use openstacksdk instead of os-client-config https://review.openstack.org/566158 | 10:54 |
| *** alexchadin has joined #openstack-infra | 10:56 | |
| *** myoung|zzz has quit IRC | 10:57 | |
| *** rnoriega has quit IRC | 10:57 | |
| *** weshay has quit IRC | 10:57 | |
| *** rasca has quit IRC | 10:58 | |
| *** myoung has joined #openstack-infra | 10:58 | |
| *** weshay has joined #openstack-infra | 10:58 | |
| *** rnoriega has joined #openstack-infra | 10:58 | |
| *** rasca has joined #openstack-infra | 11:01 | |
| *** aojea has quit IRC | 11:02 | |
| *** dhajare has joined #openstack-infra | 11:02 | |
| *** jpena is now known as jpena|lunch | 11:04 | |
| *** links has quit IRC | 11:09 | |
| *** jamesmcarthur has joined #openstack-infra | 11:09 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Temporary fix static node assign race https://review.openstack.org/572342 | 11:11 |
| *** lifeless has quit IRC | 11:12 | |
| *** aojea has joined #openstack-infra | 11:12 | |
| *** lifeless has joined #openstack-infra | 11:13 | |
| *** ianw has quit IRC | 11:13 | |
| *** jamesmcarthur has quit IRC | 11:14 | |
| *** udesale has quit IRC | 11:17 | |
| *** udesale has joined #openstack-infra | 11:17 | |
| *** edmondsw has joined #openstack-infra | 11:19 | |
| *** udesale has quit IRC | 11:23 | |
| *** links has joined #openstack-infra | 11:25 | |
| *** zoli|lunch is now known as zoli | 11:27 | |
| *** yamamoto_ has quit IRC | 11:29 | |
| *** zhangfei has quit IRC | 11:30 | |
| *** jamesmcarthur has joined #openstack-infra | 11:30 | |
| *** ianw has joined #openstack-infra | 11:32 | |
| *** jamesmcarthur has quit IRC | 11:35 | |
| *** yamamoto has joined #openstack-infra | 11:35 | |
| *** alexchadin has quit IRC | 11:36 | |
| *** dtantsur is now known as dtantsur|brb | 11:41 | |
| *** ldnunes has joined #openstack-infra | 11:41 | |
| *** dbecker has joined #openstack-infra | 11:45 | |
| *** snapiri- has joined #openstack-infra | 11:48 | |
| *** CrayZee has quit IRC | 11:50 | |
| *** jamesmcarthur has joined #openstack-infra | 11:51 | |
| *** tpsilva has joined #openstack-infra | 11:52 | |
| *** snapiri- is now known as CrayZee | 11:53 | |
| *** CrayZee is now known as Guest91145 | 11:53 | |
| *** Guest91145 has quit IRC | 11:53 | |
| *** snapiri- has joined #openstack-infra | 11:54 | |
| *** snapiri- has quit IRC | 11:54 | |
| *** snapiri- has joined #openstack-infra | 11:55 | |
| *** snapiri- has quit IRC | 11:55 | |
| *** amoralej is now known as amoralej|lunch | 11:56 | |
| *** jamesmcarthur has quit IRC | 11:56 | |
| *** yamamoto has quit IRC | 11:58 | |
| *** jpena|lunch is now known as jpena | 11:59 | |
| *** rh-jelabarre has joined #openstack-infra | 11:59 | |
| *** anteaya has quit IRC | 12:02 | |
| *** anteaya has joined #openstack-infra | 12:04 | |
| *** rosmaita has joined #openstack-infra | 12:04 | |
| *** germs has joined #openstack-infra | 12:05 | |
| *** germs has quit IRC | 12:10 | |
| andreaf | clarkb corvus https://review.openstack.org/#/c/572313/ - live migration test pass with that patch - and this is what /etc/hosts looks like http://logs.openstack.org/84/572084/2/check/tempest-multinode-full/68dec30/job-output.txt.gz#_2018-06-05_10_58_12_870444 | 12:11 |
| *** jcoufal has joined #openstack-infra | 12:12 | |
| *** jamesmcarthur has joined #openstack-infra | 12:12 | |
| *** katkapilatova has joined #openstack-infra | 12:16 | |
| *** eharney has joined #openstack-infra | 12:16 | |
| *** trown|outtypewww is now known as trown | 12:16 | |
| *** jamesmcarthur has quit IRC | 12:17 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/devstack-gate master: Stop creating reproduce.sh https://review.openstack.org/572367 | 12:19 |
| *** eandersson has joined #openstack-infra | 12:19 | |
| *** tosky__ is now known as tosky | 12:23 | |
| *** andreas_s has quit IRC | 12:29 | |
| *** jcoufal has quit IRC | 12:29 | |
| *** andreas_s has joined #openstack-infra | 12:30 | |
| *** andreas_s has quit IRC | 12:30 | |
| *** andreas_s has joined #openstack-infra | 12:30 | |
| *** roman_g has quit IRC | 12:30 | |
| *** roman_g has joined #openstack-infra | 12:31 | |
| *** rlandy has joined #openstack-infra | 12:31 | |
| *** armaan has quit IRC | 12:32 | |
| *** rlandy is now known as rlandy|rover | 12:32 | |
| *** jamesmcarthur has joined #openstack-infra | 12:33 | |
| *** jamesmcarthur has quit IRC | 12:39 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add afsmon project https://review.openstack.org/572371 | 12:43 |
| *** finucannot is now known as stephenfin | 12:44 | |
| *** yamamoto has joined #openstack-infra | 12:44 | |
| *** nmathew has joined #openstack-infra | 12:46 | |
| *** mriedem has joined #openstack-infra | 12:46 | |
| *** hemna_ has joined #openstack-infra | 12:47 | |
| *** camunoz has joined #openstack-infra | 12:48 | |
| *** amoralej|lunch is now known as amoralej | 12:51 | |
| *** rh-jelabarre has quit IRC | 12:52 | |
| *** dave-mccowan has joined #openstack-infra | 12:54 | |
| *** jamesmcarthur has joined #openstack-infra | 12:54 | |
| *** kgiusti has joined #openstack-infra | 12:56 | |
| *** rh-jelabarre has joined #openstack-infra | 12:58 | |
| *** jamesmcarthur has quit IRC | 12:59 | |
| *** andreas_s has quit IRC | 13:00 | |
| *** andreas_s has joined #openstack-infra | 13:01 | |
| *** andreas_s has quit IRC | 13:01 | |
| *** andreas_s has joined #openstack-infra | 13:01 | |
| *** andreas_s has quit IRC | 13:02 | |
| *** andreas_s has joined #openstack-infra | 13:03 | |
| *** andreas_s has quit IRC | 13:05 | |
| *** andreas_s has joined #openstack-infra | 13:09 | |
| *** dtantsur|brb is now known as dtantsur | 13:10 | |
| *** r-daneel has quit IRC | 13:12 | |
| *** andreas_s has quit IRC | 13:13 | |
| *** CrayZee has joined #openstack-infra | 13:15 | |
| *** jamesmcarthur has joined #openstack-infra | 13:16 | |
| *** andreas_s has joined #openstack-infra | 13:18 | |
| *** jamesmcarthur has quit IRC | 13:21 | |
| *** armaan has joined #openstack-infra | 13:21 | |
| zul | im seeing some weirdness, dont if its me or something else https://pastebin.com/r8Ysnms2 | 13:22 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: Replace neutron ovsfw job with iptables_hybrid https://review.openstack.org/572386 | 13:24 |
| *** andreas_s has quit IRC | 13:26 | |
| *** esberglu has joined #openstack-infra | 13:26 | |
| *** andreas_s has joined #openstack-infra | 13:27 | |
| mnaser | zul: i know we're struggling with the openstack brand, but we havent done the switch to opentack | 13:27 |
| mnaser | you're missing an s ;) enjoy the rest of your morning :p | 13:28 |
| zul | shit, i need more caffine apparently ;) | 13:28 |
| mnaser | i was gonna say opentack is a bit tacky but too early for that | 13:28 |
| zul | hehe | 13:28 |
| zul | oh look it works now | 13:29 |
| zul | ;) | 13:29 |
| *** jbadiapa has quit IRC | 13:29 | |
| *** jesslampe has joined #openstack-infra | 13:31 | |
| *** andreas_s has quit IRC | 13:32 | |
| fungi | but... great idea if we decide to rebrand the project. thanks for the idea! ;) | 13:38 |
| hwoarang | hello infra. zuul jobs for 572292 are stuck waiting for opensuse-150 node to become available for about 1h. since opensuse-150 is a new addition i was wondering if something is missing to have better availabitliy for these nodes | 13:40 |
| *** eharney has quit IRC | 13:40 | |
| *** andreas_s has joined #openstack-infra | 13:41 | |
| *** jbadiapa has joined #openstack-infra | 13:41 | |
| *** jesslampe has quit IRC | 13:43 | |
| *** andreas_s has quit IRC | 13:44 | |
| *** yamamoto has quit IRC | 13:44 | |
| *** andreas_s has joined #openstack-infra | 13:45 | |
| *** VW has joined #openstack-infra | 13:46 | |
| *** andreas_s has quit IRC | 13:49 | |
| *** mihalis68 has joined #openstack-infra | 13:50 | |
| *** andreas_s has joined #openstack-infra | 13:51 | |
| *** andreas_s has quit IRC | 13:51 | |
| *** jesslampe has joined #openstack-infra | 13:52 | |
| *** caphrim007 has quit IRC | 13:52 | |
| *** andreas_s_ has joined #openstack-infra | 13:54 | |
| *** rpioso|afk is now known as rpioso | 13:56 | |
| *** andreas_s_ has quit IRC | 13:58 | |
| *** r-daneel has joined #openstack-infra | 13:59 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Make Zuul able to start with a broken config https://review.openstack.org/535511 | 14:00 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Make Zuul able to start with a broken config https://review.openstack.org/535511 | 14:01 |
| *** caphrim007 has joined #openstack-infra | 14:01 | |
| frickler | hwoarang: seems building the nodes times out because ssh access fails. hopefully some other infra-root can debug further | 14:02 |
| hwoarang | frickler: oh ... ok then | 14:02 |
| pabelanger | last I looked, before summit, glean was failing on opensuse. Not sure anybody ever fixed it | 14:02 |
| *** armaan has quit IRC | 14:02 | |
| *** armaan has joined #openstack-infra | 14:03 | |
| hwoarang | pabelanger: do you have more info on that? | 14:04 |
| *** neiloy has joined #openstack-infra | 14:04 | |
| hwoarang | any logs from glean or from the node building i can look at? | 14:04 |
| frickler | pabelanger: the opensuse-423 job does seem to work. but maybe only on a specific provider and we need to restrict the 15.0 to that same provider | 14:04 |
| *** neiloy has quit IRC | 14:05 | |
| *** neiloy has joined #openstack-infra | 14:05 | |
| *** caphrim007 has quit IRC | 14:05 | |
| hwoarang | yeah 423 has been working fine for all i know | 14:06 |
| openstackgerrit | Michel Peterson proposed openstack-infra/project-config master: Remove networking-odl carbon jobs from stats https://review.openstack.org/567317 | 14:06 |
| openstackgerrit | Michel Peterson proposed openstack-infra/project-config master: Remove nitrogen from n-odl grafana https://review.openstack.org/572406 | 14:06 |
| pabelanger | right, it was just tumbleweed that was failing | 14:06 |
| pabelanger | IIRC, you could see the failure in nodepool dsvm jobs on nodepool project | 14:06 |
| pabelanger | for opensuse | 14:06 |
| *** germs has joined #openstack-infra | 14:06 | |
| pabelanger | but, I haven't looked recently | 14:06 |
| hwoarang | that's for tumbleweed right? | 14:06 |
| pabelanger | yes | 14:06 |
| hwoarang | ok ignore that ;p | 14:06 |
| pabelanger | did you add tumblweed to nodepool dsvm job? | 14:07 |
| hwoarang | though i can see why glean would fail there | 14:07 |
| pabelanger | tumbleweed-150 | 14:07 |
| hwoarang | no it should be opensuse-150 | 14:07 |
| hwoarang | like opensuse-423 | 14:07 |
| hwoarang | tumbleweed is a different thing | 14:07 |
| pabelanger | k, that is a straightforward way to validate it builds properly with nodepool / DIB / glean | 14:07 |
| pabelanger | for example: https://review.openstack.org//559211/ | 14:08 |
| pabelanger | that is for fedora-28, should do the same for opensuse-150 | 14:08 |
| pabelanger | have to run now, good luck | 14:08 |
| hwoarang | ok thanks. lets try that | 14:09 |
| *** germs has quit IRC | 14:11 | |
| *** andreas_s has joined #openstack-infra | 14:12 | |
| *** andreas_s has quit IRC | 14:12 | |
| *** ccamacho has quit IRC | 14:12 | |
| *** ccamacho1 has joined #openstack-infra | 14:12 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Add support for specifying security_group in nodepool https://review.openstack.org/570857 | 14:12 |
| *** andreas_s has joined #openstack-infra | 14:12 | |
| *** andreas_s has quit IRC | 14:13 | |
| *** zhangfei has joined #openstack-infra | 14:13 | |
| *** andreas_s has joined #openstack-infra | 14:13 | |
| *** jamesmcarthur has joined #openstack-infra | 14:15 | |
| *** ramishra has quit IRC | 14:16 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/nodepool master: Add opensuse-150 to nodepool dsvm https://review.openstack.org/572416 | 14:16 |
| hwoarang | could be glean related in the end... | 14:20 |
| *** hongbin has joined #openstack-infra | 14:21 | |
| *** cshastri has quit IRC | 14:22 | |
| hwoarang | yep it's glean | 14:23 |
| *** esarault has joined #openstack-infra | 14:23 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/glean master: glean: cmd: Add new identifiers for openSUSE Leap 15 and Tumbleweed https://review.openstack.org/572424 | 14:25 |
| hwoarang | pabelanger: ^ (that would also require a new glean release :/) | 14:25 |
| hwoarang | thank you for the pointers | 14:26 |
| *** jcoufal has joined #openstack-infra | 14:26 | |
| *** dklyle has joined #openstack-infra | 14:30 | |
| *** kjackal has joined #openstack-infra | 14:41 | |
| *** derekh has quit IRC | 14:44 | |
| *** yamamoto has joined #openstack-infra | 14:45 | |
| *** eandersson_ has joined #openstack-infra | 14:45 | |
| *** eandersson has quit IRC | 14:45 | |
| *** lpetrut has quit IRC | 14:49 | |
| *** yamamoto has quit IRC | 14:51 | |
| *** mriedem has quit IRC | 14:53 | |
| openstackgerrit | Michael Johnson proposed openstack-infra/zuul-jobs master: Collect the coverage report for npm test jobs https://review.openstack.org/570260 | 14:53 |
| clarkb | hwoarang: if you set up https://review.openstack.org/#/c/572416/ to depends on https://review.openstack.org/572424 I think it will test that everything works together properly | 14:53 |
| *** mriedem has joined #openstack-infra | 14:53 | |
| hwoarang | clarkb: ah awesome didn't know you can test glean/master like that | 14:54 |
| clarkb | hwoarang: ya the src nodepool jobs should use glean from source | 14:54 |
| clarkb | the non src will use from release | 14:54 |
| hwoarang | ok lets see | 14:54 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/nodepool master: Add opensuse-150 to nodepool dsvm https://review.openstack.org/572416 | 14:55 |
| *** yamamoto has joined #openstack-infra | 14:56 | |
| *** slaweq_ has joined #openstack-infra | 14:58 | |
| *** slaweq has quit IRC | 14:58 | |
| *** yamamoto has quit IRC | 15:01 | |
| *** dpawlik has quit IRC | 15:03 | |
| *** pcaruana has quit IRC | 15:05 | |
| *** zhangfei has quit IRC | 15:09 | |
| openstackgerrit | Michel Peterson proposed openstack-infra/project-config master: Remove nitrogen from n-odl grafana https://review.openstack.org/572406 | 15:12 |
| *** iranzo has quit IRC | 15:16 | |
| openstackgerrit | Spyros Trigazis proposed openstack-infra/project-config master: WIP: Add magnum to storyboard https://review.openstack.org/572440 | 15:17 |
| corvus | i'm going to be afk for most of the day; i may miss the infra meeting (cc clarkb) | 15:19 |
| *** ccamacho1 has quit IRC | 15:19 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Add publish-to-pypi jobs to ansible-role-container-registry https://review.openstack.org/572444 | 15:20 |
| fungi | thanks for the heads up, corvus! | 15:21 |
| *** andreas_s has quit IRC | 15:21 | |
| *** andreas_s has joined #openstack-infra | 15:22 | |
| *** udesale has joined #openstack-infra | 15:24 | |
| *** Bhujay has quit IRC | 15:26 | |
| *** udesale has quit IRC | 15:26 | |
| *** udesale has joined #openstack-infra | 15:27 | |
| *** jcoufal has quit IRC | 15:28 | |
| *** udesale has quit IRC | 15:28 | |
| *** slaweq has joined #openstack-infra | 15:29 | |
| *** udesale has joined #openstack-infra | 15:29 | |
| *** jbadiapa has quit IRC | 15:30 | |
| *** udesale has quit IRC | 15:30 | |
| *** andreas_s has quit IRC | 15:31 | |
| *** slaweq_ has quit IRC | 15:33 | |
| *** udesale has joined #openstack-infra | 15:33 | |
| *** sshnaidm|brq has quit IRC | 15:33 | |
| *** udesale has quit IRC | 15:34 | |
| *** eernst has joined #openstack-infra | 15:34 | |
| *** katkapilatova has quit IRC | 15:35 | |
| *** lifeless_ has joined #openstack-infra | 15:35 | |
| *** eharney has joined #openstack-infra | 15:35 | |
| *** lifeless has quit IRC | 15:36 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Add kata-containers/proxy to zuul https://review.openstack.org/572451 | 15:37 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Add kata-run job to kata-containers/proxy https://review.openstack.org/572452 | 15:37 |
| clarkb | corvus: noted, if you get a chance ^ that is my trying to get the ball rolling on the kata stuff we talked about at the summit | 15:37 |
| *** janki has quit IRC | 15:37 | |
| clarkb | I'm sure I missed something obvious there that will need fixing | 15:37 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Add publish-to-pypi jobs to ansible-role-container-registry https://review.openstack.org/572444 | 15:40 |
| *** ykarel|training has quit IRC | 15:41 | |
| hwoarang | mordred: do you have 2 min to discuss a zuul problem? | 15:46 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Add kata-run job to kata-containers/proxy https://review.openstack.org/572452 | 15:47 |
| *** yolanda has quit IRC | 15:50 | |
| *** caphrim007 has joined #openstack-infra | 15:50 | |
| evrardjp | hwoarang: don't hesitate to also ask corvus :D | 15:50 |
| evrardjp | or anyone here :) | 15:50 |
| hwoarang | i am trying to find proper english to explain | 15:51 |
| anteaya | evrardjp, except corvus is offline today | 15:51 |
| anteaya | hwoarang, just give it a try someone will ask a question if they need more information | 15:51 |
| *** nmathew has quit IRC | 15:52 | |
| *** e0ne has quit IRC | 15:52 | |
| *** caphrim007 has quit IRC | 15:53 | |
| *** caphrim007 has joined #openstack-infra | 15:53 | |
| *** caphrim007 has quit IRC | 15:53 | |
| *** dpawlik has joined #openstack-infra | 15:53 | |
| *** caphrim007 has joined #openstack-infra | 15:54 | |
| mordred | hwoarang: yah - don't wait on me - I'm in transit today so I'll be in and out - but I'll certainly try to help as best I can | 15:54 |
| mordred | hwoarang: there is also the #zuul channel where there are many people who might be ale to help | 15:54 |
| hwoarang | no problem. i am trying some thing locally and then i will write a brief description of the problem | 15:54 |
| hwoarang | ok will ask there | 15:54 |
| *** yamamoto has joined #openstack-infra | 15:58 | |
| *** dpawlik has quit IRC | 15:58 | |
| *** gyee has joined #openstack-infra | 16:00 | |
| *** neiloy has quit IRC | 16:00 | |
| *** neiloy has joined #openstack-infra | 16:01 | |
| *** yamamoto has quit IRC | 16:03 | |
| *** links has quit IRC | 16:03 | |
| *** eernst_ has joined #openstack-infra | 16:03 | |
| *** eernst has quit IRC | 16:03 | |
| *** jpich has quit IRC | 16:04 | |
| *** dizquierdo has joined #openstack-infra | 16:05 | |
| *** yolanda has joined #openstack-infra | 16:05 | |
| *** slaweq_ has joined #openstack-infra | 16:05 | |
| *** germs has joined #openstack-infra | 16:07 | |
| *** germs has quit IRC | 16:07 | |
| *** germs has joined #openstack-infra | 16:07 | |
| *** germs has quit IRC | 16:09 | |
| *** slaweq has quit IRC | 16:09 | |
| *** slaweq_ is now known as slaweq | 16:09 | |
| *** germs has joined #openstack-infra | 16:09 | |
| *** germs has quit IRC | 16:09 | |
| *** germs has joined #openstack-infra | 16:09 | |
| *** germs has quit IRC | 16:09 | |
| *** germs has joined #openstack-infra | 16:10 | |
| *** germs has joined #openstack-infra | 16:10 | |
| *** zoli is now known as zoli|gone | 16:13 | |
| hwoarang | mordred: so this is more like an openstack+zuul thing so i will ask here. take this for example http://logs.openstack.org/99/496199/8/check/openstack-ansible-functional-centos-7/ea7fb1d/job-output.txt.gz#_2018-06-05_16_03_01_173363 Looks like tox -e functional is executed but the actual command from the tox.ini is not. only the deps are installed. This is from https://review.openstack.org/#/c/496199/ | 16:16 |
| hwoarang | evrardjp ^^^ gonna leave in a bit so I wont be able to follow up but logs suggest that the commands from tox.ini are being ignored | 16:16 |
| hwoarang | no clue why | 16:16 |
| hwoarang | and i can't see why it would be a zuul issue but who knows... | 16:16 |
| *** neiloy has quit IRC | 16:17 | |
| mordred | hwoarang: WELL - I haven't dug in too deeply yet - but it's worth pointing out that the tox role in the base tox job does run tox twice - once with a flag to make only deps get installed, and then a second time to actually run the command (this is to support the siblings functions) | 16:19 |
| clarkb | this isn't our tox role it is running an osa script | 16:20 |
| mordred | hwoarang: although it doesn't look like that job is using the tox role | 16:20 |
| mordred | yeah | 16:20 |
| hwoarang | very strange problem | 16:21 |
| rlandy|rover | pabelanger: hello - I was looking at the difference in the verbosity of the outputs in these two job logs - was there some change in the infra that could have impacted this change ... | 16:21 |
| clarkb | the sed is what has me concerned there | 16:21 |
| rlandy|rover | http://logs.openstack.org/42/572042/3/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/10947cd/job-output.txt.gz#_2018-06-04_14_20_33_713356 | 16:21 |
| rlandy|rover | vs | 16:21 |
| clarkb | I'm trying to find where that sed and tox call actually are, might be in the job playbook itself /me looks | 16:22 |
| rlandy|rover | http://logs.openstack.org/86/506186/23/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/8b16fc9/job-output.txt.gz#_2018-06-05_00_26_13_920855 | 16:22 |
| rlandy|rover | weshay: ^^ | 16:22 |
| clarkb | hwoarang: mordred http://logs.openstack.org/99/496199/8/check/openstack-ansible-functional-centos-7/ea7fb1d/ara-report/result/4359dced-5935-4552-b7a9-f7a6c196c885/ that log has the full results which are different than your log above | 16:27 |
| hwoarang | aha | 16:27 |
| hwoarang | that log looks correct | 16:27 |
| clarkb | why this happened I don't know and this is confusing | 16:27 |
| clarkb | but the post logs playbooks weren't able to complete | 16:27 |
| clarkb | I'm guessing this is related | 16:28 |
| *** harlowja has joined #openstack-infra | 16:28 | |
| clarkb | or possibly a file redirection issue? | 16:28 |
| clarkb | rlandy|rover: that may be related to hwoarang's issue too | 16:29 |
| clarkb | rlandy|rover: lets see if ara has the full data | 16:29 |
| clarkb | there is a non zero chance this is related to the recent zuul updates | 16:29 |
| rlandy|rover | clarkb: thanks - you can tell from those two logs when we started to pick up the difference ... we have the ara | 16:30 |
| clarkb | rlandy|rover: http://logs.openstack.org/86/506186/23/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/8b16fc9/ara-report/result/7023b3dc-582c-4aec-be42-cf3c564796fc/ ya seems to be | 16:30 |
| rlandy|rover | right - ok | 16:30 |
| clarkb | mordred: corvus tobiash fyi ^ possible bug in log streaming | 16:31 |
| clarkb | I'll look to see if I can find tracebacks related to these logs | 16:31 |
| hwoarang | might be related but the streaming logs in zuul.openstack.org do not currently work for me | 16:31 |
| hwoarang | the live logs i mean | 16:31 |
| EmilienM | hey, can someone approve this project-config patch please? it's blocking our m2 release, thanks :) https://review.openstack.org/#/c/572444/ | 16:32 |
| rlandy|rover | great - thanks | 16:32 |
| zxiiro | pabelanger: Looks like Dan's doing that ApartCI presentation again today at the Open Networking meetup. | 16:33 |
| clarkb | AJaeger: re EmilienM's change thoughts on https://review.openstack.org/#/c/571520/ ? | 16:33 |
| zxiiro | I'm on the fence about attending heh | 16:33 |
| EmilienM | clarkb: oh ok so I could manage this job in tree | 16:34 |
| EmilienM | clarkb: should I abandon this patch then? | 16:34 |
| clarkb | EmilienM: well I know AJaeger has concerns with that, but I'd prefer that projects stop telling the infra team we are blocking them especially now that zuul means we don't have to :) | 16:35 |
| clarkb | EmilienM: hoping that AJaeger will weigh in as he does a lot of these reviews | 16:35 |
| EmilienM | :-o | 16:35 |
| EmilienM | ok | 16:35 |
| *** jamesmcarthur has quit IRC | 16:35 | |
| *** jesslampe has quit IRC | 16:36 | |
| clarkb | on the log streaming front it looks like the logs are stopping being streamed but the fingerfw can get what was streamed | 16:36 |
| clarkb | that path is zuul-web -> fingergw -> executor -> job node right? | 16:37 |
| clarkb | actually no fingergw is only used when running finger directly. Should be web -> executor I think | 16:38 |
| *** dbecker has quit IRC | 16:40 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add publish-to-pypi jobs to ansible-role-container-registry https://review.openstack.org/572444 | 16:42 |
| Shrews | clarkb: zuul-web should still contact the fingergw | 16:42 |
| clarkb | ok the /var/lib/zuul/builds/91db88b32a5a4d7a9a1d4d498c2ae3ff/work/logs/job-output.txt file is truncated implying the problem is between executor and job node | 16:42 |
| *** myoung is now known as myoung|lunch | 16:43 | |
| clarkb | Shrews: ah ok, in any case I don't think web or finger gw are at fault because the job-output.txt file itself is truncated in the work dir of a running job | 16:43 |
| * mordred apologizes for having dropped the ball on the rework of streaming from job node to executor | 16:43 | |
| Shrews | clarkb: ooh, that's not good | 16:43 |
| clarkb | I'm slowly working my way up that stack in debuggin the above job | 16:45 |
| *** harlowja has quit IRC | 16:45 | |
| clarkb | /tmp/console-3e16920a3c2d4024ba8f62fce166cd1c.log on the test node does not appear to be truncated | 16:45 |
| *** jbadiapa has joined #openstack-infra | 16:46 | |
| clarkb | job side: [pid 2033] sendto(5, "2018-06-05 15:54:35.817754 | cha"..., 4096, 0, NULL, 0 | 16:47 |
| clarkb | executor side epoll_wait(14, | 16:47 |
| *** e0ne has joined #openstack-infra | 16:48 | |
| clarkb | seems like the executor side isn't reading what the job side is writing | 16:49 |
| clarkb | and this is running in ansible-playbook | 16:49 |
| *** yamamoto has joined #openstack-infra | 16:50 | |
| *** e0ne_ has joined #openstack-infra | 16:50 | |
| tobiash | clarkb: I think there were no recent changes related to log streaming in ansible | 16:51 |
| *** dizquierdo has quit IRC | 16:51 | |
| tobiash | strange | 16:53 |
| clarkb | tobiash: I4e395b650e83a78d66aa362a420ef0d836c80d35 is the only one I think | 16:53 |
| clarkb | but that shouldn't affect this since the connection is being made and data is moving then not | 16:53 |
| *** e0ne has quit IRC | 16:54 | |
| mordred | clarkb: did the zuul-console process stop/die? | 16:54 |
| *** jamesmcarthur has joined #openstack-infra | 16:54 | |
| mordred | (on the job node) | 16:54 |
| tobiash | oom on the node maybe? | 16:55 |
| clarkb | mordred: no I straced it above. it is blocking on the send | 16:55 |
| clarkb | mordred: and the read side is block on an epoll | 16:55 |
| clarkb | mordred: both sides show the tcp connection as established | 16:56 |
| clarkb | my reading of it is the tcp connection is there and job is sending data but executor is not reading it | 16:56 |
| clarkb | unfortunately the epoll doesn't map directly to our python code so I'm trying to wrap my head around that | 16:57 |
| *** rpittau has quit IRC | 16:57 | |
| *** kaiokmo has quit IRC | 16:58 | |
| Shrews | i don't see any active streaming. finger works | 16:58 |
| *** jaosorior has quit IRC | 16:58 | |
| Shrews | this seems zuul-web related | 16:58 |
| clarkb | Shrews: web works too, the problem is you get a truncated file which is in the workd ir | 16:59 |
| *** kgiusti has left #openstack-infra | 16:59 | |
| clarkb | https://zuul.openstack.org/stream.html?uuid=91db88b32a5a4d7a9a1d4d498c2ae3ff&logfile=console.log for example | 16:59 |
| *** kaiokmo has joined #openstack-infra | 16:59 | |
| clarkb | a lot has happened since 15:45:18UTC | 16:59 |
| openstackgerrit | Elod Illes proposed openstack-infra/project-config master: WIP: Add networking-odl for bgpvpn stable jobs https://review.openstack.org/572495 | 16:59 |
| *** yamamoto has quit IRC | 17:00 | |
| *** masuberu has quit IRC | 17:02 | |
| *** masuberu has joined #openstack-infra | 17:02 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Make Zuul able to start with a broken config https://review.openstack.org/535511 | 17:03 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Make Zuul able to start with a broken config https://review.openstack.org/535511 | 17:03 |
| *** slaweq has quit IRC | 17:04 | |
| *** slaweq has joined #openstack-infra | 17:04 | |
| *** tosky__ has joined #openstack-infra | 17:05 | |
| *** tosky has quit IRC | 17:05 | |
| *** tosky__ is now known as tosky | 17:05 | |
| clarkb | its stopping after ~27k bytes which is more than one time through the read loop | 17:07 |
| *** esarault_ has joined #openstack-infra | 17:07 | |
| clarkb | there is a 16 second gap in logging after what we did get and it starting again on the job side | 17:07 |
| *** e0ne_ has quit IRC | 17:08 | |
| Shrews | clarkb: mordred: tobiash: suspicious traceback in the executor: http://paste.openstack.org/show/722744/ | 17:09 |
| clarkb | the send that is blocking job side is several minutes after where it stops executor side and many bytes later | 17:09 |
| clarkb | Shrews: ah ya I bet that would've killed it | 17:10 |
| *** ralonsoh_ has quit IRC | 17:10 | |
| clarkb | however that is long after the problem seems to have manifested so may be a side effect confirming it is an underlying tcp problem | 17:10 |
| *** jpena is now known as jpena|off | 17:10 | |
| *** esarault has quit IRC | 17:11 | |
| Shrews | lots of these errors in the executor.log: ConnectionResetError: [Errno 104] Connection reset by peer | 17:11 |
| Shrews | so yeah, something funky with the network | 17:12 |
| clarkb | Shrews: what cloud did that job run in? | 17:13 |
| clarkb | Shrews: the one I was looking at was limestone | 17:13 |
| *** lpetrut has joined #openstack-infra | 17:13 | |
| Shrews | clarkb: not sure | 17:13 |
| *** rpioso is now known as rpioso|eat | 17:14 | |
| *** jamesmcarthur has quit IRC | 17:14 | |
| *** e0ne has joined #openstack-infra | 17:14 | |
| clarkb | that one was rax | 17:15 |
| *** armaan has quit IRC | 17:15 | |
| AJaeger | clarkb: thanks for bringing this up. Looking at EmilienM : I think the release team has scripts that demand that either of these jobs exist. So, this specific job cannot move in-tree unless the release team changes their policy. Same for requirements team... | 17:15 |
| *** armaan has joined #openstack-infra | 17:15 | |
| AJaeger | clarkb: but looking at your infra-manual change, I'm fine with it! will +2 in a bit. I think we have the jobs stable so that we are fine moving to projects. | 17:16 |
| clarkb | Shrews: 2018-06-05 17:09:48,592 DEBUG zuul.AnsibleJob: [build: 91db88b32a5a4d7a9a1d4d498c2ae3ff] Ansible output: b' more = s.recv(4096)' found that timeout for my job on a different executor and it ran on a different cloud | 17:16 |
| clarkb | Shrews: I think that means either rax-dfw (where we host our executors) is having trouble with networking or this is related to the updates of the executors | 17:17 |
| *** eernst_ has quit IRC | 17:17 | |
| AJaeger | clarkb: it's for me more a policy/trust question - do we want a central config or do we trust projects to do the right thing? Let's try it... | 17:17 |
| AJaeger | clarkb: was there any discussion in the last two weeks about this or what triggered your change? | 17:17 |
| *** eernst has joined #openstack-infra | 17:17 | |
| clarkb | AJaeger: ya, I share some of the same trust concerns but I don't think we will be able to keep up otherwise. | 17:17 |
| clarkb | AJaeger: jsut the number of "infra we need you to review this now because it is blocking us" requests over the last little while and realizing that they could just make the job config updates in their own repos | 17:18 |
| *** neiloy has joined #openstack-infra | 17:18 | |
| AJaeger | clarkb: we didn't enforce PTI either so far - so no change on that front | 17:18 |
| AJaeger | clarkb: yeah, the number of reviewers does not grow significantly but the number of projects and pings does ;( | 17:19 |
| *** jcoufal has joined #openstack-infra | 17:19 | |
| *** eernst_ has joined #openstack-infra | 17:19 | |
| *** eernst_ has quit IRC | 17:20 | |
| *** eernst has quit IRC | 17:20 | |
| AJaeger | clarkb: what to do about release jobs and requirements - and publish ones? Those access secrets or teams mandate them in project-config. | 17:20 |
| *** eernst has joined #openstack-infra | 17:20 | |
| *** neiloy has quit IRC | 17:21 | |
| *** neiloy has joined #openstack-infra | 17:21 | |
| clarkb | AJaeger: the jobs themselves will have to continue to be defined in the infra repos with the secrets but other projects can consume them from their managed config (at least zuul allows for this) we should sync up with the other teams (requirements/release) and see if they would be comfortable with this change | 17:22 |
| *** dbecker has joined #openstack-infra | 17:22 | |
| *** anteaya has quit IRC | 17:22 | |
| AJaeger | clarkb: left a comment for these | 17:22 |
| *** anteaya has joined #openstack-infra | 17:23 | |
| AJaeger | clarkb: translations falls in the same one - and those are tricky since we update the translation jobs with each release. Having them in a single place makes those changes much easier | 17:23 |
| AJaeger | clarkb: so, what about: Adding my suggested note - talk with those teams - and then remove it again/change it? | 17:23 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/nodepool master: [DNM] Add opensuse-150 to nodepool dsvm https://review.openstack.org/572416 | 17:24 |
| clarkb | AJaeger: wfm | 17:24 |
| *** dtantsur is now known as dtantsur|afk | 17:24 | |
| *** eernst has quit IRC | 17:25 | |
| AJaeger | clarkb: might be worth bringing this up at infra-meeting to point rest of config-core and infra-root at https://review.openstack.org/#/c/571520 - so that we all speak with one voice. | 17:26 |
| *** rajinir has joined #openstack-infra | 17:26 | |
| fungi | yeah, i think for a bunch of those jobs, keeping the job definitions in an infra-managed repo but letting teams add/remove variants in their own repos makes a lot of sense from a workload standpoint | 17:26 |
| *** eernst has joined #openstack-infra | 17:26 | |
| clarkb | AJaeger: ++ | 17:26 |
| clarkb | fungi: see above discussion re zuul job logs. We are having tcp troubles. | 17:26 |
| *** lifeless has joined #openstack-infra | 17:26 | |
| fungi | yeah, i read it. https://rackspace.service-now.com/system_status/ doesn't claim anything known disruptive in rax | 17:27 |
| clarkb | however it does seem constrained to the streaming? ssh isn't breaking and ansible is completing the jobs and the ara logs are complete | 17:27 |
| *** yamahata has joined #openstack-infra | 17:27 | |
| clarkb | this implies to me that the bug is in zuul somewhere | 17:27 |
| *** eernst has quit IRC | 17:27 | |
| fungi | and i haven't seen any incident notifications from them to infra-root@o.o either | 17:27 |
| *** jesslampe has joined #openstack-infra | 17:27 | |
| *** eernst has joined #openstack-infra | 17:27 | |
| AJaeger | clarkb: note I probably will not be around for the meeting - but will read eavesdrop later | 17:27 |
| clarkb | actually I think I know what it is | 17:27 |
| *** electrofelix has quit IRC | 17:27 | |
| clarkb | AJaeger: no problem, thank you for discussing it now, this was helpful | 17:27 |
| * clarkb checks then will write a patch | 17:28 | |
| AJaeger | thanks clarkb for bringing it up! | 17:28 |
| *** lifeless_ has quit IRC | 17:28 | |
| *** amoralej is now known as amoralej|off | 17:28 | |
| *** isq has joined #openstack-infra | 17:29 | |
| *** apetrich has quit IRC | 17:29 | |
| *** dpawlik has joined #openstack-infra | 17:30 | |
| *** apetrich has joined #openstack-infra | 17:30 | |
| clarkb | tobiash: the issue is the commit I posted earlier I think. It sets a 5 second timeout on the scoket object which will affect the initial connection and subsequent use. THe problem is there may be many seconds of quiet time on that socket if we aren't logging anything | 17:31 |
| *** lpetrut has quit IRC | 17:31 | |
| fungi | do we set tcp keepalives? | 17:32 |
| clarkb | I don't think so but even then they tend to be much less frequent then our timeout of 5 seconds | 17:33 |
| clarkb | the problem here is the overlap in timing out the initial connection and then timing out the socket once connected. I think we want a timeout for the initial connection then rely on keepalives after that? | 17:35 |
| *** isssp has joined #openstack-infra | 17:35 | |
| *** sambetts is now known as sambetts|afk | 17:36 | |
| *** dpawlik has quit IRC | 17:36 | |
| tobiash | clarkb: so that connection timout is not just for creating the connection? | 17:36 |
| clarkb | tobiash: correct | 17:36 |
| *** dpawlik has joined #openstack-infra | 17:36 | |
| clarkb | that is why it happens when we have our first gap in logging that lasts for more than 5 seconds | 17:36 |
| tobiash | Oh, that's bad | 17:36 |
| clarkb | I'll work on a fix that uses keepalives to keep things chatting | 17:37 |
| *** trown is now known as trown|lunch | 17:38 | |
| *** dpawlik has quit IRC | 17:39 | |
| *** ispp has quit IRC | 17:40 | |
| clarkb | also timed out sockets are implicitly in non blocking mode in python internals. This is why I saw the epoll in strace I think | 17:42 |
| *** eernst has joined #openstack-infra | 17:44 | |
| *** myoung|lunch is now known as myoung | 17:45 | |
| *** eernst has quit IRC | 17:46 | |
| *** dbecker has quit IRC | 17:46 | |
| *** ccamacho has joined #openstack-infra | 17:46 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Send keepalives on the log callback connection https://review.openstack.org/572503 | 17:47 |
| *** eernst has joined #openstack-infra | 17:47 | |
| tobiash | clarkb: I don't think it's the keepalive | 17:48 |
| clarkb | tobiash: its the lack of a keepalive | 17:48 |
| *** eernst has quit IRC | 17:48 | |
| *** eernst has joined #openstack-infra | 17:48 | |
| *** jcoufal has quit IRC | 17:49 | |
| clarkb | tobiash: reading the log that was truncated there is no gap of longer than a second while it is working then we have a 16 second gap and it stops working. Then later the executor side catches up and does http://paste.openstack.org/show/722744/ | 17:49 |
| *** hashar is now known as LeftForDead | 17:49 | |
| *** jcoufal has joined #openstack-infra | 17:49 | |
| tobiash | I think it's more the not handling of the timeout exception during reading | 17:49 |
| clarkb | that would be another way to handle it but the connection is never in an error state so I think it is wrong to put it in one and then hadle it | 17:50 |
| tobiash | clarkb: what about just resetting the timeout after the connection has been established? | 17:50 |
| clarkb | from a system and tcp perspective the connection is fine and established on both ends | 17:50 |
| clarkb | tobiash: ya that might work too. Depends if python lets you update the timeout value | 17:51 |
| tobiash | Because we don't handle the timout and thus stopping reading | 17:51 |
| clarkb | right but the timeout is purely self imposed. Tcp itself doesn't care and is happy si my point | 17:51 |
| clarkb | so it seems weird to handle an error that enver really even happens | 17:52 |
| clarkb | "If a non-zero value is given, subsequent socket operations will raise a timeout exception if the timeout period value has elapsed before the operation has completed." that implies you can update the value and it will just apply to subsequent actions | 17:53 |
| *** armaan has quit IRC | 17:53 | |
| *** armaan has joined #openstack-infra | 17:53 | |
| *** lpetrut has joined #openstack-infra | 17:55 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Disable timeouts on the log streaming connection https://review.openstack.org/572503 | 17:56 |
| *** yamamoto has joined #openstack-infra | 17:56 | |
| *** armaan has quit IRC | 17:58 | |
| *** esarault has joined #openstack-infra | 17:59 | |
| *** myoung is now known as myoung|biaf | 17:59 | |
| *** eernst has quit IRC | 17:59 | |
| *** eernst has joined #openstack-infra | 18:00 | |
| *** ccamacho has quit IRC | 18:00 | |
| *** kien-ha has joined #openstack-infra | 18:00 | |
| *** eernst has quit IRC | 18:01 | |
| *** eernst has joined #openstack-infra | 18:01 | |
| *** yamamoto has quit IRC | 18:02 | |
| *** esarault_ has quit IRC | 18:03 | |
| AJaeger | clarkb: regarding https://review.openstack.org/#/c/572451/1/zuul/main.yaml - do you mind if I sort the list? that's your change adding kata containers... With the number of entries, I like some order ;) | 18:04 |
| clarkb | AJaeger: thats fine. Maybe we have a separate section just for third party check projects? | 18:05 |
| *** aojea has quit IRC | 18:06 | |
| AJaeger | isn't that the complete list of untrusted projects? | 18:07 |
| clarkb | AJaeger: the ones we don't load configs for yes, its like ~6 projects | 18:07 |
| AJaeger | exactly | 18:08 |
| clarkb | but I'm fine adding order to it | 18:09 |
| * AJaeger will propose a followup... | 18:09 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Sort zuul/main.yaml https://review.openstack.org/572505 | 18:11 |
| AJaeger | clarkb: ^ | 18:12 |
| *** CrayZee has quit IRC | 18:15 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Cleanup log streaming error handling https://review.openstack.org/572513 | 18:17 |
| clarkb | AJaeger: thanks | 18:18 |
| *** Adri2000 has quit IRC | 18:22 | |
| *** shardy has quit IRC | 18:23 | |
| AJaeger | clarkb: could you check https://review.openstack.org/#/c/572371/ , please? See my comment on usage of namespace... | 18:23 |
| *** Adri2000 has joined #openstack-infra | 18:24 | |
| *** shardy has joined #openstack-infra | 18:24 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Fix bwrap leak test https://review.openstack.org/572514 | 18:25 |
| clarkb | AJaeger: done. I'm happy for it to be in -infra if ianw is | 18:26 |
| clarkb | should probably update goverancne too | 18:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Run openstacksdk ansible tests on stable-2.6 of ansible https://review.openstack.org/571238 | 18:27 |
| AJaeger | clarkb: for me it is openstack-infra = under governance and part of infra ;) | 18:27 |
| clarkb | ya I mean if ianw wants that we should make sure to update governance too | 18:27 |
| clarkb | but shouldn't do it if ianw doesn't want that | 18:27 |
| AJaeger | clarkb: so, you suggest that my "=" is more an approximiation ;) | 18:29 |
| AJaeger | let's see what ianw wants... | 18:29 |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/irc-meetings master: I18n: convert weekly meeting to office hours https://review.openstack.org/572517 | 18:30 |
| *** myoung|biaf is now known as myoung | 18:30 | |
| *** esarault has quit IRC | 18:32 | |
| clarkb | I need to prep for infra meeting | 18:32 |
| *** rpioso|eat is now known as rpioso | 18:37 | |
| *** trown|lunch is now known as trown | 18:38 | |
| *** edmondsw has quit IRC | 18:41 | |
| *** EmilienM is now known as EmilienM|PTO | 18:43 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Make streaming test run longer than connection timeout https://review.openstack.org/572522 | 18:44 |
| *** yolanda has quit IRC | 18:49 | |
| *** dbecker has joined #openstack-infra | 18:58 | |
| *** eharney has quit IRC | 18:58 | |
| *** yamamoto has joined #openstack-infra | 18:58 | |
| *** links has joined #openstack-infra | 18:59 | |
| clarkb | meeting time | 19:00 |
| *** yamamoto has quit IRC | 19:03 | |
| *** eernst has quit IRC | 19:04 | |
| *** edmondsw has joined #openstack-infra | 19:07 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix bwrap leak test https://review.openstack.org/572514 | 19:08 |
| *** pkesavar has joined #openstack-infra | 19:08 | |
| *** jbadiapa has quit IRC | 19:10 | |
| *** mihalis68 has quit IRC | 19:10 | |
| *** edmondsw_ has joined #openstack-infra | 19:12 | |
| *** jamesmcarthur has joined #openstack-infra | 19:15 | |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/zuul-jobs master: WIP Support building pdfs of sphinx docs https://review.openstack.org/570957 | 19:15 |
| *** edmondsw has quit IRC | 19:15 | |
| mtreinish | mordred, clarkb: I just pushed the new stestr release: https://pypi.org/project/stestr/ which has the suppress attachments flag | 19:18 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add afsmon project https://review.openstack.org/572371 | 19:18 |
| openstackgerrit | Merged openstack-infra/system-config master: Add opendaylight connection to zuul https://review.openstack.org/570087 | 19:21 |
| *** anteaya has quit IRC | 19:23 | |
| *** anteaya has joined #openstack-infra | 19:24 | |
| *** armaan has joined #openstack-infra | 19:26 | |
| *** armaan has quit IRC | 19:26 | |
| *** armaan has joined #openstack-infra | 19:27 | |
| *** yolanda has joined #openstack-infra | 19:29 | |
| *** kzaitsev_pi has quit IRC | 19:29 | |
| *** kzaitsev_pi has joined #openstack-infra | 19:30 | |
| *** janki has joined #openstack-infra | 19:32 | |
| openstackgerrit | Zane Bitter proposed openstack-infra/openstack-zuul-jobs master: Run a non-voting py36 job for all py35 tox jobs https://review.openstack.org/572535 | 19:44 |
| *** links has quit IRC | 19:46 | |
| *** eharney has joined #openstack-infra | 19:49 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: askbot: remove duplicated templates https://review.openstack.org/572537 | 19:49 |
| openstackgerrit | Merged openstack-infra/zuul master: Disable timeouts on the log streaming connection https://review.openstack.org/572503 | 19:49 |
| *** kjackal has quit IRC | 19:50 | |
| ianw | cmurphy: ^^ based on your point about the config files | 19:50 |
| *** timothyb89_ is now known as timothyb89 | 19:51 | |
| anteaya | spotz, thanks for the review :) | 19:53 |
| spotz | anteaya: My pleasure:) | 19:53 |
| anteaya | :) | 19:53 |
| *** sshnaidm|brq has joined #openstack-infra | 19:53 | |
| anteaya | spotz, if you do some playing around with it, I'm interested in hearing your feedback | 19:54 |
| anteaya | there is a lot I don't know | 19:54 |
| *** e0ne has quit IRC | 19:54 | |
| spotz | anteaya: Looks like fungi gave it a whirl for sure from his comments, so I suspect your doc might be a bit of a work inprogress but it's a very nice start | 19:55 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/zuul master: github: Optimize getPullReviews() a bit https://review.openstack.org/570077 | 19:55 |
| anteaya | spotz, very much a work in progress yes, thank you | 19:55 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Add clean_session.log for rotation https://review.openstack.org/558977 | 19:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Remove old apache logs https://review.openstack.org/558985 | 19:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Update settings https://review.openstack.org/559694 | 19:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Pin captcha version https://review.openstack.org/559704 | 19:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Add systemd support https://review.openstack.org/559601 | 19:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Don't require virtualenv package https://review.openstack.org/559622 | 19:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Ensure more correct ordering https://review.openstack.org/559623 | 19:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Use earlier six release https://review.openstack.org/559624 | 19:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Add rspec test https://review.openstack.org/558991 | 19:57 |
| *** yamamoto has joined #openstack-infra | 19:59 | |
| openstackgerrit | Dean Troyer proposed openstack-infra/project-config master: Add StarlingX repos https://review.openstack.org/569562 | 20:00 |
| openstackgerrit | Dean Troyer proposed openstack-infra/project-config master: Add zuul jobs for StarlingX https://review.openstack.org/569605 | 20:00 |
| mordred | clarkb: I tried adding some symlinks to the multi-tenant zuul dashoard job here: https://review.openstack.org/#/c/551989/36/playbooks/dashboard/multi.yaml - it timed out in the rsync, which makes me think the rsync is following symlinks rather tahn copying them as symlinks ... any thoughts on what a better way to achieve that is? (this is because logs.o.o doesn't do rewrite rules, but if we make a | 20:01 |
| mordred | couple of symlinks, then t/local/status.html will work | 20:01 |
| clarkb | mordred: I think it is an rsync option to not follow them | 20:01 |
| clarkb | mordred: you may be able to change that in the ansible task | 20:01 |
| mordred | clarkb: it's in the base publication/copy job - is that a flag we would want? | 20:02 |
| clarkb | mordred: I could go either way on it. copying links as links is more truthful to the original we are trying to preserve but may result in "data loss" if we don't copy the remote end | 20:03 |
| *** anteaya has quit IRC | 20:03 | |
| clarkb | with https://review.openstack.org/572503 merged I would like to restart the executor process on one executor | 20:03 |
| mordred | yah. maybe add a flag that could be set in a job if the person knows what they're doing? | 20:03 |
| clarkb | if that one is happy and stops showing timeouts in the logs I will restart the other executors. infra-root you good with that? | 20:04 |
| *** anteaya has joined #openstack-infra | 20:04 | |
| mordred | clarkb: ++ | 20:04 |
| clarkb | mordred: that could work, just be careful to not allow other args to be injected | 20:04 |
| fungi | yes, sounds great | 20:04 |
| clarkb | ok I'll do ze01 | 20:04 |
| *** yamamoto has quit IRC | 20:05 | |
| *** r-daneel has quit IRC | 20:08 | |
| clarkb | prometheanfire: smcginnis dhellmann I'ev proposed https://review.openstack.org/#/c/571520/1 and quick survey in infra meeting seemed to be that infra isn't opposed to it but we did want buy in from release, requirements, and TC for making that change | 20:09 |
| clarkb | can you all take a look and leave your thoughts? | 20:09 |
| *** kien-ha has quit IRC | 20:10 | |
| smcginnis | clarkb: Will do, thanks for the heads up. | 20:10 |
| *** e0ne has joined #openstack-infra | 20:10 | |
| *** edmondsw_ is now known as edmondsw | 20:10 | |
| dhellmann | clarkb : as it happens, I'm making note about how to update project testing to use some of the new python 3 versions of doc and release notes jobs | 20:10 |
| dhellmann | I wonder if we can combine the efforts to update to those jobs with moving the PTI settings out of project-config | 20:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add a job variable to control javascript link copying https://review.openstack.org/572542 | 20:11 |
| clarkb | fwiw I think we can keep the definition portion of the configs in a central place and project-config is largely that today (that and ozj) I just don't want to force people to consume it there too | 20:11 |
| mordred | clarkb: ^^ I think that should fix the rsync issue | 20:11 |
| prometheanfire | clarkb: nice | 20:11 |
| dhellmann | clarkb : yeah, the consumption part is what I mean | 20:12 |
| clarkb | dhellmann: smcginnis prometheanfire one specific concern is that requirements and release teams and maybe tc have tools to check project-config for compliance which would need updating | 20:12 |
| dhellmann | yeah, that's a good point | 20:12 |
| *** lpetrut has quit IRC | 20:12 | |
| clarkb | ze01 executor starting back up again with patched log streaming | 20:12 |
| prometheanfire | clarkb: ya, that's the main reason I like the central config | 20:12 |
| prometheanfire | only reason really | 20:12 |
| prometheanfire | ensure they had a reqs sync, it looked a certian way, and was enforcing | 20:13 |
| dhellmann | we don't do that one any more, right? :-) | 20:13 |
| prometheanfire | now that we don't have reqs sync that's easier | 20:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v6) https://review.openstack.org/551989 | 20:14 |
| prometheanfire | I do suspect that projects don't have valid lower-constraints, but that's ever the case (no change) | 20:14 |
| dhellmann | looking for the release jobs for release validation is going to be a bit painful because of the variations in local configuration file names | 20:14 |
| *** e0ne has quit IRC | 20:14 | |
| *** VW_ has joined #openstack-infra | 20:14 | |
| dhellmann | smcginnis : ^^ | 20:15 |
| smcginnis | Yeah, that may take some time to work through. | 20:15 |
| dhellmann | clarkb : is there a tool/library to point at a directory/repo and ask for all of the zuul settings as a blob of some sort, instead of all of these validation tools reinventing a way to read arbitrary yaml files? | 20:15 |
| clarkb | https://zuul.openstack.org/stream.html?uuid=fcc4fadd9f9a498cad86919b43656c61&logfile=console.log is running on ze01 | 20:16 |
| clarkb | dhellmann: the only one I am aware of is zuul itself | 20:16 |
| mordred | dhellmann: we've talked about the need for local tools for being ale to do things like that for various reasons | 20:17 |
| dhellmann | I know corvus showed me some code at one point for making mechanical updates to a repo but it was sort of standalone "fork this to make the tool you need" code | 20:17 |
| prometheanfire | dhellmann: don't we still have a job to make sure that a given projects requirements are a superset of the requirements project's global-requirements? | 20:17 |
| prometheanfire | I thought we still had a minimal test, or something | 20:17 |
| *** VW has quit IRC | 20:17 | |
| mordred | dhellmann: it gets tricky because of config coming from other repos too - but ultimately I think the use case is something we're going to need to do something aout | 20:17 |
| *** hemna_ has quit IRC | 20:18 | |
| *** VW_ has quit IRC | 20:18 | |
| dhellmann | prometheanfire : we do, but that doesn't look at any of the zuul settings, iirc | 20:18 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Don't require virtualenv package https://review.openstack.org/559622 | 20:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Ensure more correct ordering https://review.openstack.org/559623 | 20:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Use earlier six release https://review.openstack.org/559624 | 20:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Add rspec test https://review.openstack.org/558991 | 20:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Pin stopforumspam https://review.openstack.org/572549 | 20:19 |
| dhellmann | mordred : for my purposes I would be happy with a function that I could give a root directory and it would return a dictionary with all of the various yaml namespaces merged | 20:19 |
| prometheanfire | dhellmann: k | 20:19 |
| prometheanfire | was thinking the same thing about merged yaml | 20:20 |
| dhellmann | mordred : I'm OK with having to call it for a few separate directories | 20:20 |
| prometheanfire | deep_merge :D | 20:20 |
| dhellmann | although if I could just call the zuul service and ask it for that stuff, that would make me happy, too | 20:20 |
| dhellmann | "what do you know about openstack/oslo.config?" | 20:20 |
| clarkb | dhellmann: ya that might be the easiest way to do it | 20:20 |
| clarkb | is make that part of the zuul api | 20:20 |
| mordred | yup. the hardest part right now will be that notinng in repo A tells you additional stuff comes from repo B - I think asking the zuul service for the config info would e the most consistent way to do such a thing | 20:21 |
| dhellmann | it avoids having 2 implementations, for sure | 20:21 |
| *** e0ne has joined #openstack-infra | 20:21 | |
| mordred | "hi, I've got this .zuul config in this local copy of oslo.config, if you were to apply it what would you think the config for oslo.config is?" | 20:21 |
| dhellmann | as long as there's nothing "secret" in the response it should be OK | 20:21 |
| clarkb | assuming we hand wave that as a thing somehow are we ok with starting to push config into projects or do we need to have the config checker first? | 20:21 |
| prometheanfire | ya, much more clear | 20:21 |
| dhellmann | mordred : for the validation I'm doing, I don't even need that, although that would be useful | 20:22 |
| *** dpawlik has joined #openstack-infra | 20:22 | |
| dhellmann | I just need to know "does oslo.config have a release job defined" | 20:22 |
| prometheanfire | how much do we trust projects | 20:22 |
| dhellmann | I don't expect zuul to understand the semantics of how to answer that, though | 20:22 |
| prometheanfire | I trust no one | 20:22 |
| mordred | dhellmann: yah - that's the 'full' case - and would be needed for a tool to run zuul jobs locally | 20:22 |
| dhellmann | clarkb : I'd like to have the config checker first, because otherwise we open ourselves up to broken release jobs | 20:22 |
| clarkb | ok | 20:22 |
| dhellmann | how big of a request is that, though? | 20:22 |
| dhellmann | in terms of work time | 20:23 |
| clarkb | dhellmann: I think adding it as an api to zuul shouldn't be too bad | 20:23 |
| clarkb | zuul has all that data in memory at all times to run the jobs anyways | 20:23 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Import new repos to openstack-ansible https://review.openstack.org/572551 | 20:23 |
| clarkb | should be just a matter of adding the http request and returning reasonable json | 20:23 |
| dhellmann | is any part of that in-memory data structure stuff that would have to be purged before returning it? or can the API endpoint just return json.dumps()? | 20:23 |
| clarkb | I think for these datastructures it is safe to just return them as secrets are stored encrypted until used | 20:24 |
| clarkb | iirc | 20:24 |
| clarkb | you'll get the encrypted secret back which should be fine they are already in the git repos | 20:24 |
| prometheanfire | good design working for you | 20:24 |
| *** pcichy has joined #openstack-infra | 20:24 | |
| *** e0ne has quit IRC | 20:25 | |
| dhellmann | that all seems OK then | 20:25 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Add rspec test https://review.openstack.org/558991 | 20:26 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Add idempotentcy rspec test https://review.openstack.org/572553 | 20:26 |
| prometheanfire | even if the verification job lives with reqs, if a repo is 'failing' we fail, then complain/remove them from projects.txt | 20:27 |
| prometheanfire | win/win lol | 20:27 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Import new repos to openstack-ansible https://review.openstack.org/572551 | 20:29 |
| dhellmann | clarkb : as far as the transition goes, does it confuse zuul to have the same list of projects or job templates attached to a repo multiple times? or does it remove the duplicates? | 20:29 |
| clarkb | dhellmann: iirc corvus said it will remove duplicates | 20:29 |
| clarkb | mechanically teh transition should be simple | 20:30 |
| dhellmann | ok, so we can -- right | 20:30 |
| dhellmann | assuming we can figure out which file to edit in each repo :-) | 20:30 |
| clarkb | I'm going to let ze01 run for another half hour and check the logs for timeouts, the job I linked above ran successfully. If there are no new timouts I will restart all the executors | 20:30 |
| clarkb | http://logs.openstack.org/16/569916/7/gate/openstack-tox-py35/fcc4fad/job-output.txt.gz#_2018-06-05_20_18_54_718263 shows a gap longer than 5 seconds too | 20:32 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Add new OSA project base jobs https://review.openstack.org/572558 | 20:35 |
| openstackgerrit | Merged openstack-infra/puppet-askbot master: Add clean_session.log for rotation https://review.openstack.org/558977 | 20:35 |
| clarkb | thinking about this more we may have to scope the data to a project simply to avoid the requests being massive | 20:37 |
| clarkb | we'll have to see how big the complete running config for a pipeline is | 20:37 |
| fungi | mordred: clarkb: dhellmann: prometheanfire: catching up, but i think we had talked about the zuul status dashboard growing a config browser, so a corresponding json interface is likely to be part of that anyway right? there may even be a story for it already | 20:38 |
| dhellmann | clarkb : oh, yeah, I don't want an API that gives me everything zuul knows, just what it knows about a specific repository | 20:39 |
| dhellmann | fungi : nice! | 20:39 |
| evrardjp | that's cool. | 20:40 |
| fungi | this was supposed to be the solution to the "where are all the bits of this job defined?" dilemma people regularly face with zuulv3's configuration model | 20:41 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Don't require virtualenv package https://review.openstack.org/559622 | 20:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Ensure more correct ordering https://review.openstack.org/559623 | 20:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Pin stopforumspam https://review.openstack.org/572549 | 20:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Use earlier six release https://review.openstack.org/559624 | 20:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Add rspec test https://review.openstack.org/558991 | 20:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Add idempotentcy rspec test https://review.openstack.org/572553 | 20:42 |
| clarkb | ok lunch then executor updates, back in a bit | 20:42 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/glean master: glean: cmd: Add new identifiers for openSUSE Leap 15 and Tumbleweed https://review.openstack.org/572424 | 20:45 |
| *** r-daneel has joined #openstack-infra | 20:45 | |
| *** trown is now known as trown|outtypewww | 20:46 | |
| *** pcichy has quit IRC | 20:47 | |
| *** lifeless has quit IRC | 20:51 | |
| *** lifeless has joined #openstack-infra | 20:51 | |
| *** dhajare has quit IRC | 20:56 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add openSUSE Leap 15 to dib boot tests https://review.openstack.org/572561 | 20:57 |
| ianw | hwoarang: ^ so we can start testing glean changes | 20:57 |
| *** jamesmcarthur has quit IRC | 20:57 | |
| *** dpawlik has quit IRC | 20:58 | |
| *** dpawlik has joined #openstack-infra | 20:58 | |
| *** kjackal has joined #openstack-infra | 20:59 | |
| *** SumitNaiksatam has joined #openstack-infra | 21:00 | |
| *** yamamoto has joined #openstack-infra | 21:01 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add StarlingX repos https://review.openstack.org/569562 | 21:03 |
| *** yamamoto has quit IRC | 21:05 | |
| *** myoung is now known as myoung|off | 21:08 | |
| ianw | i wonder why console logs in http://logs.openstack.org/53/572553/2/check/legacy-puppet-beaker-rspec-infra/eebd976/job-output.txt.gz#_2018-06-05_21_01_23_208461 just suddenly disappear for 6 minutes | 21:09 |
| ianw | it's logged in json/ara | 21:09 |
| *** slaweq has quit IRC | 21:10 | |
| ianw | clarkb: do you remember anything about the bot timeouts in suse jobs discussed in -> https://review.openstack.org/#/c/572424/2 ? | 21:12 |
| *** rh-jelabarre has quit IRC | 21:12 | |
| fungi | ianw: clarkb has some zuul patches to solve that | 21:13 |
| clarkb | ianw: the logging thing is an issue with the update to zuul yesterday | 21:13 |
| *** jcoufal has quit IRC | 21:13 | |
| clarkb | ianw: my fix for it got in and I've been making sure ze01 is still happy with it. Aout to double check again the ndeploy to the other executors | 21:14 |
| clarkb | ianw: https://review.openstack.org/572503 if curious | 21:14 |
| fungi | in short, there's a 5-second idle timeout kicking in as soon as the finger stream gets a lull in activity, and the web streamer drops it | 21:14 |
| clarkb | no new timeouts on ze01 since the restart | 21:15 |
| clarkb | going to restart the other executors now | 21:15 |
| *** esberglu has quit IRC | 21:16 | |
| *** dave-mccowan has quit IRC | 21:16 | |
| zigo | Hi there! | 21:18 |
| zigo | I don't understand why this suddunly fails: http://logs.openstack.org/28/564328/47/check/puppet-openstack-integration-4-scenario002-tempest-debian-stable/8152442/job-output.txt.gz#_2018-06-05_20_38_20_674096 | 21:18 |
| zigo | Has something changed in the Debian image? | 21:18 |
| zigo | This is just a dd call ... | 21:19 |
| clarkb | zigo: I can take a look in a few minutes after I get these zuul-executors restarted | 21:19 |
| zigo | clarkb: Thanks ! | 21:19 |
| *** SumitNaiksatam has quit IRC | 21:19 | |
| *** janki has quit IRC | 21:23 | |
| clarkb | fungi: ianw ok all 10 executors should be running the fix now | 21:25 |
| *** lifeless has quit IRC | 21:25 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Enable doc and release notes for python-tempestconf https://review.openstack.org/572564 | 21:27 |
| *** priteau has quit IRC | 21:27 | |
| fungi | zigo: looks like it dies when trying to create a 4gb swapfile... maybe the rootfs is filling up? i wonder if the growroot bit isn't working at boot | 21:27 |
| clarkb | zigo: I think you ran into the same problem I just deployed a fix for. It won't fix your job but will make the logs complete. For jobs that ran when it was broken you can look at ara for the data http://logs.openstack.org/28/564328/47/check/puppet-openstack-integration-4-scenario002-tempest-debian-stable/8152442/ara-report/result/b54ed3bf-3390-43e4-8b35-392733b93c45/ | 21:28 |
| clarkb | fungi: no I think same thing as ianw | 21:28 |
| fungi | aha! | 21:28 |
| fungi | so that was simply where the stream went silent | 21:28 |
| clarkb | protip on ext4 that should be fast using fallocate | 21:28 |
| clarkb | ya | 21:28 |
| fungi | yeah, i thought it was odd that there was no actual error message | 21:28 |
| clarkb | I don't see an obvious isse in the log above though | 21:29 |
| clarkb | hopefully it makes sense to zigp | 21:29 |
| clarkb | *zigo | 21:29 |
| *** neiloy has quit IRC | 21:29 | |
| zigo | clarkb: The log you've just pasted is fixed by this commit: https://review.openstack.org/#/c/564328 | 21:30 |
| zigo | clarkb: Though currently, it fails with the swapfile stuff ... :/ | 21:30 |
| clarkb | zigo: the swapfile stuff isn't a real issue, its just the log stops there because it takes more than 5 seconds | 21:30 |
| zigo | ? | 21:30 |
| clarkb | zigo: you'll need to look in the ara logs for complete logs or rerun the jobs as the zuul executors have been fixed | 21:30 |
| zigo | clarkb: What too more than 5 seconds? | 21:31 |
| clarkb | zigo: there was a bug where if log output delayed for 5 or more seconds we stopped collecting the logs at this point | 21:31 |
| clarkb | zigo: the dd | 21:31 |
| clarkb | separately you can use fallocate on ext4 if you want it to go quicker | 21:31 |
| zigo | clarkb: Oh, so it's fixed now? | 21:31 |
| clarkb | zigo: yes, but your jobs will still be broken | 21:31 |
| clarkb | it was just hiding the errors not the source of the error | 21:31 |
| zigo | Why will my jobs still fail? | 21:32 |
| zigo | I don't get it, sorry. | 21:32 |
| clarkb | zigo: the problem I fixed was only in collecting the logs, we were not collecting the complete log. This has since been fixed but this issue is why you see your logs end with the swap setup | 21:32 |
| clarkb | zigo: if yourun your jobs now you will see the complete log and it does get past the swap setup | 21:33 |
| clarkb | if they were failing before then my fix to zuul won't fix those failures | 21:33 |
| clarkb | but you will be able to see the errors more clearly | 21:33 |
| fungi | zigo: it's failing on the problem you say the proposed change is intended to fix, so presumably the fix is inadequate | 21:33 |
| zigo | fungi: My patch is fixing *one* problem, and it's supposed to fail later, when then functional tests are running. | 21:34 |
| zigo | I'm currently working on that part... | 21:34 |
| *** edmondsw has quit IRC | 21:34 | |
| fungi | well, anyway, as clarkb says recheck and you should see a complete log to better confirm what the current issue is | 21:34 |
| *** edmondsw has joined #openstack-infra | 21:34 | |
| *** camunoz has quit IRC | 21:35 | |
| zigo | ok, thanks | 21:35 |
| * zigo rebased, so it will stop the logs and restart | 21:35 | |
| *** dpawlik has quit IRC | 21:35 | |
| *** pkesavar has quit IRC | 21:36 | |
| clarkb | #status log Zuul job-output.txt files were incomplete if at any point the job stopped producing logs for more than 5 seconds. This happened due to a timeout in the log streaming daemon. This has been fixed and the zuul executors have been restarted. Jobs running after now should have complete logs. | 21:36 |
| openstackstatus | clarkb: finished logging | 21:37 |
| clarkb | separately I think we've broken the links to the log streaming if you try to click on them directly | 21:38 |
| clarkb | they work if you open in a new tab but not in the existing window. Any chance this is fallout from mnaser's updates to the angular and jquery stuff? | 21:38 |
| clarkb | mordred: ^ you may know | 21:38 |
| *** edmondsw has quit IRC | 21:38 | |
| mnaser | let me see | 21:39 |
| clarkb | mnaser: with the network debugging window open in ff I don't see it making any new network requests | 21:39 |
| clarkb | it just continues to run the status.json GET | 21:39 |
| mnaser | i may have broken it, yes, sorry | 21:39 |
| mnaser | let me see how to fix | 21:39 |
| *** r-daneel_ has joined #openstack-infra | 21:39 | |
| *** r-daneel has quit IRC | 21:39 | |
| *** r-daneel_ is now known as r-daneel | 21:39 | |
| mnaser | sorry, i didnt test that | 21:40 |
| clarkb | no problem, not the first issue related to updating zuul yesterday :) | 21:40 |
| clarkb | also has a not crazy workaround of opening in a new tab | 21:40 |
| fungi | yeah, i generally ctrl-click those anyway | 21:41 |
| mnaser | ok fixed | 21:42 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul master: Disable rewriting links in status page with HTML5 mode https://review.openstack.org/572566 | 21:44 |
| clarkb | ianw: hwoarang as soon as I review ^ I'll look into the suse thing | 21:44 |
| *** jesslampe has quit IRC | 21:44 | |
| ianw | clarkb: thanks ... sorry that log timeout thing must have passed me by | 21:45 |
| ianw | i'm sure we've talked about glean+suse issues, but not sure where it got to | 21:46 |
| clarkb | I think hwoarang thought the glean fix that you've seen is the fix but it may not be complete based on test data/ | 21:47 |
| clarkb | looks like ara doesn't load things properly if they timeout so we may need to rerun those jobs :/ | 21:47 |
| *** lifeless has joined #openstack-infra | 21:47 | |
| clarkb | ya I think we may just need new runs now that zuul executors are happier | 21:48 |
| *** camunoz has joined #openstack-infra | 21:50 | |
| openstackgerrit | Linda Marquez proposed openstack-infra/system-config master: Adding mail list for OpenStack Mentoring; amending for list placement. https://review.openstack.org/571553 | 21:50 |
| andreaf | I meant this here clarkb corvus to complete the tempest multinode job: https://review.openstack.org/563099 https://review.openstack.org/572313 https://review.openstack.org/#/c/564220/ :) | 21:52 |
| *** sshnaidm|brq has quit IRC | 21:58 | |
| *** yamamoto has joined #openstack-infra | 22:02 | |
| clarkb | as a heads up I just approved 3 of the changes cmurphy is waiting on for puppet4 test work | 22:06 |
| *** yamamoto has quit IRC | 22:06 | |
| clarkb | they modify the template defs for zuul.o.o and logs.o.o apache configs. They should be noops | 22:07 |
| clarkb | but if yu notice anything funny there these are the breadcrumbs to follow | 22:07 |
| fungi | keeping an eye out--thanks! | 22:07 |
| clarkb | fungi: if you've got a moment https://review.openstack.org/#/c/564610/ is the last one that needs review before we make the puppet4 jobs voting | 22:08 |
| fungi | you bet. just wrapped up dinner so in that quiet transitional time before wednesday tc office hour starts | 22:09 |
| *** shardy has quit IRC | 22:10 | |
| *** dbecker has quit IRC | 22:11 | |
| *** aeng has quit IRC | 22:13 | |
| *** aeng has joined #openstack-infra | 22:14 | |
| *** gfidente has quit IRC | 22:15 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Make streaming test run longer than connection timeout https://review.openstack.org/572522 | 22:19 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Fix remote stream log test https://review.openstack.org/572574 | 22:19 |
| *** r-daneel has quit IRC | 22:21 | |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Fix operatingsystem major release fact name https://review.openstack.org/564577 | 22:22 |
| *** LeftForDead has quit IRC | 22:24 | |
| *** jesslampe has joined #openstack-infra | 22:25 | |
| *** rcernin has joined #openstack-infra | 22:25 | |
| ianw | http://logs.openstack.org/24/572424/1/check/nodepool-functional-py35-suse-src/5b80a83/controller/logs/screen-nodepool-launcher.txt.gz#_Jun_05_17_42_37_705414 | 22:26 |
| ianw | Jun 05 17:42:37.705414 ubuntu-xenial-inap-mtl01-0004300030 nodepool-launcher[28090]: Manager devstack running task ComputeGetServersDetail (queue: 0) | 22:26 |
| ianw | i've never noticed that before, but it ends up dumping the entire console log into the debug logs | 22:26 |
| ianw | the console log being something that would be helpful, but in another form ideally | 22:26 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Fix scope of vhost template variables https://review.openstack.org/564844 | 22:27 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci master: Fix scope of vhost template variables https://review.openstack.org/564600 | 22:27 |
| clarkb | ianw: http://logs.openstack.org/24/572424/1/check/nodepool-functional-py35-suse-src/5b80a83/controller/logs/screen-nodepool-launcher.txt.gz#_Jun_05_17_43_22_858211 its fairly readable there | 22:28 |
| clarkb | looks like glean isn't completing based on that | 22:29 |
| *** VW has joined #openstack-infra | 22:29 | |
| clarkb | http://logs.openstack.org/24/572424/1/check/nodepool-functional-py35-suse-src/5b80a83/controller/logs/screen-nodepool-launcher.txt.gz#_Jun_05_17_43_22_974116 | 22:29 |
| clarkb | probably need someone to build an image locally and inspect it | 22:29 |
| ianw | http://logs.openstack.org/24/572424/1/check/nodepool-functional-py35-suse-src/5b80a83/controller/logs/screen-nodepool-launcher.txt.gz#_Jun_05_17_43_22_975636 | 22:29 |
| ianw | glean hung about there | 22:30 |
| ianw | heh, yeah | 22:30 |
| ianw | 423 works, and tumbleweed doesn't? maybe that's where we were at with that | 22:30 |
| clarkb | ya | 22:31 |
| clarkb | and then 15 (which is newer than 423 for reasons) seems to have picked up the tumbleweed brokeness | 22:31 |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite master: Fix defined type namespacing https://review.openstack.org/564610 | 22:32 |
| openstackgerrit | Merged openstack-infra/system-config master: Adding mail list for OpenStack Mentoring; amending for list placement. https://review.openstack.org/571553 | 22:34 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Make puppet 4 puppet-apply jobs voting https://review.openstack.org/571022 | 22:37 |
| clarkb | cmurphy: ^ there you go | 22:37 |
| ianw | hmm, i have no idea how the general stats http://codesearch.openstack.org/?q=ComputeGetServersDetail&i=nope&files=&repos= are produced | 22:38 |
| cmurphy | clarkb: tyty | 22:39 |
| hwoarang | clarkb: the glean fix is needed because leap 15 is being identified as opensuse-leap which is not covered by existing code | 22:40 |
| hwoarang | however, the dsvm timeouts even on leap 423 so i feel like i am looking at two distinct issues again | 22:40 |
| clarkb | hwoarang: ianw once we get 15 working maybe we update it to see 'suse' in $string as good enough? | 22:40 |
| hwoarang | yeah quite possibly | 22:41 |
| clarkb | ianw: nodepool writes those to graphite then graphana is just rendering it or do you mean something different? | 22:41 |
| hwoarang | the glean fix might be enough to get leap 15 working on real nodepool. no idea about the dsvm tests at the moment | 22:41 |
| clarkb | hwoarang: based on the logs above it appears its still not configuring the interface | 22:42 |
| clarkb | hwoarang: not sure production nodepool would be any happier with it | 22:42 |
| ianw | clarkb: oh i mean how it actually decides on the string "ComputeGetServersDetail" to put in. seems to be the task managed | 22:42 |
| *** VW has quit IRC | 22:44 | |
| *** VW has joined #openstack-infra | 22:44 | |
| hwoarang | hmm | 22:45 |
| hwoarang | ok i will investigate | 22:45 |
| *** boris_42_ has joined #openstack-infra | 22:47 | |
| *** VW has quit IRC | 22:49 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/nodepool master: [DNM] Add opensuse-150 to nodepool dsvm https://review.openstack.org/572416 | 22:52 |
| *** tpsilva has quit IRC | 22:56 | |
| mnaser | is there a problem with zuul logging or is my browser being bleh | 23:02 |
| *** yamamoto has joined #openstack-infra | 23:03 | |
| mnaser | ok, reopening in a new tab fixed it, it might just be the thing that i broke | 23:03 |
| *** edmondsw has joined #openstack-infra | 23:03 | |
| hwoarang | clarkb: is there a place where i can get the config drives used in the nodepool jobs? | 23:04 |
| hwoarang | i prob need to feed it locally to my dib image to find out what glean does | 23:05 |
| *** caphrim007 has quit IRC | 23:06 | |
| *** edmondsw has quit IRC | 23:07 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/glean master: glean: cmd: Add new identifiers for openSUSE Leap 15 and Tumbleweed https://review.openstack.org/572424 | 23:07 |
| *** yamamoto has quit IRC | 23:07 | |
| *** threestrands has joined #openstack-infra | 23:08 | |
| *** threestrands has quit IRC | 23:08 | |
| *** threestrands has joined #openstack-infra | 23:08 | |
| * hwoarang has high hopes for ^^ but he needs to go to bed | 23:08 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/nodepool master: Add opensuse-150 to nodepool dsvm https://review.openstack.org/572416 | 23:09 |
| *** harlowja has joined #openstack-infra | 23:10 | |
| ianw | hwoarang: i can get you a metadata | 23:11 |
| hwoarang | ianw: that would be helpful so i can look at it tomorrow if tests are still failing | 23:12 |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: Survey Documentation https://review.openstack.org/571536 | 23:16 |
| *** ldnunes has quit IRC | 23:19 | |
| *** rlandy|rover is now known as rlandy|rover|bbl | 23:22 | |
| *** mriedem is now known as mriedem_afk | 23:25 | |
| *** hongbin has quit IRC | 23:27 | |
| *** camunoz has quit IRC | 23:27 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Fix race in slow start test https://review.openstack.org/572584 | 23:30 |
| *** tosky has quit IRC | 23:30 | |
| *** annp has quit IRC | 23:38 | |
| *** annp has joined #openstack-infra | 23:38 | |
| anteaya | my survey docs patch failed: http://logs.openstack.org/36/571536/5/check/infra-puppet-apply-4-ubuntu-xenial/fac40eb/applytest/puppetapplytest38.final.out.FAILED | 23:38 |
| anteaya | I have no idea what I did wrong | 23:38 |
| clarkb | anteaya: that is a brand new job | 23:39 |
| clarkb | well not brand new but newly voting | 23:39 |
| openstackgerrit | Merged openstack-infra/zuul master: Disable rewriting links in status page with HTML5 mode https://review.openstack.org/572566 | 23:40 |
| clarkb | I'm guessing we regressed while trying to get up to puppet4 speed | 23:40 |
| clarkb | cmurphy: ^ | 23:40 |
| clarkb | ya considering this is new puppet not surprising | 23:40 |
| anteaya | okay great, so not me, thank you | 23:40 |
| anteaya | anything I can do to help? | 23:40 |
| ianw | if !defined(Mod['rewrite']) ... hmm, puppet-httpd would define that? | 23:41 |
| clarkb | anteaya: we need to update the new survey puppetry to be puppet4 compat | 23:41 |
| anteaya | ah ha | 23:41 |
| clarkb | ianw: we need to namespace that iirc so http::mod and similar | 23:41 |
| anteaya | I await direction on how to do that | 23:41 |
| anteaya | commands I can copy paste are my favourite | 23:41 |
| clarkb | need to fix the thing ianw points out above and scope it to http:: | 23:41 |
| clarkb | the others are warnings so should be ok I think | 23:42 |
| anteaya | I need to fix the thing? | 23:42 |
| ianw | anteaya: yeah, Mod -> ::http::Mod | 23:42 |
| ianw | i wonder if you need the leading :: | 23:42 |
| anteaya | I'll offer something and we can watch it break | 23:42 |
| ianw | anteaya: Httpd::Mod probably | 23:43 |
| anteaya | I can try thata | 23:44 |
| anteaya | s/thata/that | 23:44 |
| clarkb | there is an example I can dig up one moment | 23:44 |
| ianw | puppet-zuul does that http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/manifests/init.pp | 23:45 |
| clarkb | anteaya: ianw https://review.openstack.org/#/c/564610/1/manifests/apache.pp | 23:45 |
| ianw | ++ | 23:45 |
| anteaya | thank you | 23:46 |
| ianw | and i now see how those nodepool stats weave their way around taskmanagers, shade and driver plugins | 23:47 |
| *** jamesmcarthur has joined #openstack-infra | 23:50 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: Update survey.pp for Puppet 4 https://review.openstack.org/572587 | 23:52 |
| clarkb | ianw: https://review.openstack.org/#/c/572252/1 that means we can't cross test grafyaml with project-config right? I don't know that is important right now but we lose that functionality I think | 23:52 |
| *** jamesmcarthur has quit IRC | 23:54 | |
| *** lifeless has quit IRC | 23:54 | |
| ianw | clarkb: https://review.openstack.org/#/c/572251/ is setting the grafyaml path to the zuul checked out version | 23:54 |
| clarkb | ah | 23:54 |
| clarkb | derp I see the depends on now | 23:55 |
| ianw | it's the type of thing tox-siblings might handle for a regular project i guess | 23:55 |
| anteaya | my patch passes puppet 4 on xenial, thank you for the help | 23:55 |
| ianw | but we don't really have a requirements.txt (well tox-siblings bails out if the project doesn't have setup.cfg) | 23:56 |
| *** lifeless has joined #openstack-infra | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!