| clarkb | ansipuppet has started | 00:00 |
|---|---|---|
| *** dingyichen has joined #openstack-infra | 00:01 | |
| jeblair | i started the executor on ze01 (since cloning finished... a while ago) | 00:01 |
| *** baoli has joined #openstack-infra | 00:01 | |
| *** xarses has quit IRC | 00:01 | |
| *** jaypipes has quit IRC | 00:02 | |
| clarkb | rnning on afs nodes now | 00:03 |
| *** SumitNaiksatam has quit IRC | 00:05 | |
| *** baoli has quit IRC | 00:06 | |
| *** jascott1_ has quit IRC | 00:06 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Cap ansible<2.4.0 for linters job https://review.openstack.org/507300 | 00:06 |
| *** rlandy is now known as rlandy|bbl | 00:07 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Merge project pipeline definitions https://review.openstack.org/507163 | 00:12 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: It's branches, not branch https://review.openstack.org/507285 | 00:13 |
| clarkb | ok I think we are reverted at this point based on syslog on zuulv3 | 00:14 |
| *** jkilpatr has quit IRC | 00:16 | |
| *** yamamoto has quit IRC | 00:16 | |
| clarkb | and all the changes we listed have merged | 00:17 |
| *** Swami has quit IRC | 00:17 | |
| *** nunchuck has quit IRC | 00:20 | |
| *** yamamoto has joined #openstack-infra | 00:21 | |
| *** srobert has quit IRC | 00:22 | |
| *** jdandrea_ has joined #openstack-infra | 00:27 | |
| *** mikal_ has quit IRC | 00:32 | |
| *** mikal has joined #openstack-infra | 00:34 | |
| fungi | reset and ready for tomorrow's launch window | 00:35 |
| *** gildub has joined #openstack-infra | 00:40 | |
| *** zhurong has joined #openstack-infra | 00:41 | |
| *** aeng has quit IRC | 00:44 | |
| *** LindaWang has joined #openstack-infra | 00:45 | |
| *** jamesmcarthur has joined #openstack-infra | 00:53 | |
| openstackgerrit | Rico Lin proposed openstack-infra/irc-meetings master: Change heat meeting schedule to 1300 UTC Wed. https://review.openstack.org/506901 | 00:56 |
| openstackgerrit | Rico Lin proposed openstack-infra/irc-meetings master: Change heat meeting schedule to 1300 UTC Wed. https://review.openstack.org/506901 | 00:56 |
| *** Apoorva_ has joined #openstack-infra | 00:57 | |
| *** Apoorva has quit IRC | 01:00 | |
| *** aeng has joined #openstack-infra | 01:00 | |
| *** ijw has quit IRC | 01:02 | |
| *** Apoorva_ has quit IRC | 01:02 | |
| *** ijw has joined #openstack-infra | 01:02 | |
| *** ijw has quit IRC | 01:07 | |
| *** aeng has quit IRC | 01:08 | |
| *** kjackal_ has joined #openstack-infra | 01:18 | |
| *** xarses has joined #openstack-infra | 01:18 | |
| *** xarses has quit IRC | 01:19 | |
| *** xarses has joined #openstack-infra | 01:19 | |
| *** cshastri has joined #openstack-infra | 01:20 | |
| *** aeng has joined #openstack-infra | 01:21 | |
| *** dhill_ has quit IRC | 01:21 | |
| *** dhill_ has joined #openstack-infra | 01:21 | |
| *** esberglu has quit IRC | 01:33 | |
| *** rbergeron has joined #openstack-infra | 01:33 | |
| *** hongbin has joined #openstack-infra | 01:36 | |
| *** liujiong has joined #openstack-infra | 01:39 | |
| zhenguo | hi infras, regarding this https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg111088.html , could some core please help us? thanks! | 01:39 |
| *** camunoz has quit IRC | 01:41 | |
| *** aeng has quit IRC | 01:42 | |
| *** tpsilva has quit IRC | 01:42 | |
| *** yamamoto has quit IRC | 01:50 | |
| *** yamamoto has joined #openstack-infra | 01:50 | |
| *** yamamoto_ has joined #openstack-infra | 01:51 | |
| *** gcb has joined #openstack-infra | 01:52 | |
| *** yamamoto has quit IRC | 01:55 | |
| *** pahuang has quit IRC | 02:07 | |
| *** rwsu has joined #openstack-infra | 02:20 | |
| *** pahuang has joined #openstack-infra | 02:23 | |
| *** kjackal_ has quit IRC | 02:26 | |
| *** pblaho has quit IRC | 02:29 | |
| *** xarses_ has joined #openstack-infra | 02:30 | |
| *** xarses has quit IRC | 02:33 | |
| *** mwarad has joined #openstack-infra | 02:35 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 02:39 | |
| *** erlon has quit IRC | 02:46 | |
| *** mwarad has quit IRC | 02:47 | |
| *** thorst has quit IRC | 02:51 | |
| *** thorst has joined #openstack-infra | 02:51 | |
| *** hongbin has quit IRC | 02:51 | |
| *** hongbin has joined #openstack-infra | 02:52 | |
| *** Dinesh_Bhor has joined #openstack-infra | 02:54 | |
| *** thorst has quit IRC | 02:56 | |
| *** SumitNaiksatam has joined #openstack-infra | 02:56 | |
| *** kiennt26 has joined #openstack-infra | 02:56 | |
| *** Dinesh_Bhor has quit IRC | 02:57 | |
| *** Dinesh_Bhor has joined #openstack-infra | 02:57 | |
| *** mikal has quit IRC | 02:59 | |
| *** mikal has joined #openstack-infra | 03:02 | |
| *** rlandy|bbl is now known as rlandy | 03:06 | |
| *** pblaho has joined #openstack-infra | 03:07 | |
| *** dave-mcc_ has quit IRC | 03:09 | |
| *** rlandy has quit IRC | 03:09 | |
| *** ramishra has joined #openstack-infra | 03:18 | |
| *** lewo` has quit IRC | 03:18 | |
| *** lewo` has joined #openstack-infra | 03:19 | |
| *** Dinesh_Bhor has quit IRC | 03:24 | |
| *** Dinesh_Bhor has joined #openstack-infra | 03:28 | |
| *** rwsu has quit IRC | 03:30 | |
| *** Apoorva has joined #openstack-infra | 03:30 | |
| *** jdandrea_ has quit IRC | 03:31 | |
| *** Dinesh_Bhor has quit IRC | 03:34 | |
| *** gouthamr has quit IRC | 03:35 | |
| *** rwsu has joined #openstack-infra | 03:37 | |
| *** hongbin has quit IRC | 03:40 | |
| *** Dinesh_Bhor has joined #openstack-infra | 03:46 | |
| *** aeng has joined #openstack-infra | 03:51 | |
| *** thorst has joined #openstack-infra | 03:53 | |
| *** udesale has joined #openstack-infra | 03:59 | |
| *** rwsu has quit IRC | 04:02 | |
| *** rhallisey has quit IRC | 04:02 | |
| *** bhavik1 has joined #openstack-infra | 04:08 | |
| *** jamesmcarthur has quit IRC | 04:09 | |
| *** rwsu has joined #openstack-infra | 04:15 | |
| *** ykarel_ has joined #openstack-infra | 04:15 | |
| *** aviau has quit IRC | 04:19 | |
| *** aviau has joined #openstack-infra | 04:20 | |
| *** mtreinish has quit IRC | 04:24 | |
| *** psachin has joined #openstack-infra | 04:26 | |
| *** mwarad has joined #openstack-infra | 04:34 | |
| *** mtreinish has joined #openstack-infra | 04:34 | |
| *** Apoorva has quit IRC | 04:36 | |
| *** Apoorva has joined #openstack-infra | 04:37 | |
| *** cuongnv has joined #openstack-infra | 04:38 | |
| *** jamesmcarthur has joined #openstack-infra | 04:40 | |
| *** Apoorva has quit IRC | 04:41 | |
| *** Apoorva has joined #openstack-infra | 04:45 | |
| *** bhavik1 has quit IRC | 04:46 | |
| *** mwarad has quit IRC | 04:47 | |
| *** gcb has quit IRC | 04:48 | |
| *** claudiub has joined #openstack-infra | 04:49 | |
| *** gildub has quit IRC | 04:55 | |
| *** Apoorva has quit IRC | 05:02 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add username to build and upload information https://review.openstack.org/453968 | 05:04 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Rename ssh_port to connection_port https://review.openstack.org/500800 | 05:04 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support username also for unmanaged cloud images https://review.openstack.org/500808 | 05:04 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add connection_type to build and upload information https://review.openstack.org/503148 | 05:04 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Don't gather host keys for non ssh connections https://review.openstack.org/503166 | 05:04 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add connection_port to build and upload information https://review.openstack.org/504112 | 05:04 |
| *** Apoorva has joined #openstack-infra | 05:05 | |
| *** Apoorva has quit IRC | 05:05 | |
| *** dhajare has joined #openstack-infra | 05:05 | |
| *** sree has joined #openstack-infra | 05:06 | |
| *** Apoorva has joined #openstack-infra | 05:08 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Improve test case node_assignment_at_quota https://review.openstack.org/506134 | 05:09 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Honor cloud quotas before launching nodes https://review.openstack.org/503838 | 05:09 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Don't fail on quota exceeded https://review.openstack.org/503051 | 05:09 |
| *** sree has quit IRC | 05:09 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Make max-servers optional https://review.openstack.org/504282 | 05:09 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support cores limit per pool https://review.openstack.org/504283 | 05:09 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support ram limit per pool https://review.openstack.org/504284 | 05:09 |
| *** sree has joined #openstack-infra | 05:09 | |
| *** mriedem has quit IRC | 05:12 | |
| *** adreznec has quit IRC | 05:13 | |
| *** gildub has joined #openstack-infra | 05:13 | |
| *** bnemec has quit IRC | 05:14 | |
| *** mwarad has joined #openstack-infra | 05:14 | |
| *** Apoorva has quit IRC | 05:18 | |
| *** gcb has joined #openstack-infra | 05:24 | |
| *** felipemonteiro has joined #openstack-infra | 05:35 | |
| *** yamamoto_ has quit IRC | 05:37 | |
| *** yamamoto has joined #openstack-infra | 05:37 | |
| *** mat128 has joined #openstack-infra | 05:38 | |
| smcginnis | zhenguo: A think a lot of the infra cores are offline for the next several hours | 05:39 |
| smcginnis | zhenguo: If you can, it might be best to reply to that ML thread and ask there what the next steps would be to move that along. | 05:39 |
| *** mat128 has quit IRC | 05:41 | |
| *** mat128 has joined #openstack-infra | 05:43 | |
| *** mat128 has quit IRC | 05:46 | |
| *** mat128 has joined #openstack-infra | 05:47 | |
| *** armax has joined #openstack-infra | 05:59 | |
| *** felipemonteiro has quit IRC | 06:00 | |
| *** adreznec has joined #openstack-infra | 06:01 | |
| *** thorst has quit IRC | 06:03 | |
| *** pgadiya has joined #openstack-infra | 06:04 | |
| *** ihrachys has quit IRC | 06:05 | |
| *** mat128 has quit IRC | 06:09 | |
| zhenguo | smcginnis: thanks for the information | 06:11 |
| *** jaosorior has quit IRC | 06:12 | |
| *** aeng has quit IRC | 06:17 | |
| *** pcaruana has joined #openstack-infra | 06:20 | |
| *** andreas_s has joined #openstack-infra | 06:21 | |
| *** jascott1 has joined #openstack-infra | 06:22 | |
| *** zhurong has quit IRC | 06:22 | |
| *** iyamahat has joined #openstack-infra | 06:23 | |
| *** iyamahat has quit IRC | 06:24 | |
| *** iyamahat has joined #openstack-infra | 06:24 | |
| *** eranrom has quit IRC | 06:27 | |
| *** eranrom has joined #openstack-infra | 06:32 | |
| *** makowals has joined #openstack-infra | 06:33 | |
| *** jaosorior has joined #openstack-infra | 06:35 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add connection_port to build and upload information https://review.openstack.org/504112 | 06:35 |
| *** kiennt26 has quit IRC | 06:36 | |
| *** rcernin has joined #openstack-infra | 06:47 | |
| *** eranrom has quit IRC | 06:57 | |
| *** thorst has joined #openstack-infra | 06:59 | |
| *** _mwarad_ has joined #openstack-infra | 07:00 | |
| *** sree has quit IRC | 07:02 | |
| *** mwarad has quit IRC | 07:02 | |
| *** sree has joined #openstack-infra | 07:02 | |
| *** eranrom has joined #openstack-infra | 07:06 | |
| *** sree has quit IRC | 07:07 | |
| *** iyamahat has quit IRC | 07:07 | |
| *** florianf has joined #openstack-infra | 07:09 | |
| *** makowals has quit IRC | 07:11 | |
| *** hashar has joined #openstack-infra | 07:11 | |
| *** dtantsur|afk is now known as dtantsur | 07:11 | |
| *** tesseract has joined #openstack-infra | 07:14 | |
| *** eranrom_ has joined #openstack-infra | 07:16 | |
| *** eranrom__ has joined #openstack-infra | 07:19 | |
| *** aviau has quit IRC | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** eranrom has quit IRC | 07:19 | |
| *** e0ne has joined #openstack-infra | 07:22 | |
| *** jtomasek has joined #openstack-infra | 07:22 | |
| *** eranrom_ has quit IRC | 07:23 | |
| *** armax has quit IRC | 07:29 | |
| *** rossella_s has joined #openstack-infra | 07:37 | |
| *** sree has joined #openstack-infra | 07:41 | |
| *** egonzalez has joined #openstack-infra | 07:42 | |
| *** apuimedo has joined #openstack-infra | 07:42 | |
| *** sree has quit IRC | 07:46 | |
| openstackgerrit | Andreas Scheuring proposed openstack/diskimage-builder master: Add zipl element as s390x architecture bootloader https://review.openstack.org/443548 | 07:48 |
| *** rossella_s has quit IRC | 07:49 | |
| *** gildub has quit IRC | 07:53 | |
| *** eranrom__ has quit IRC | 07:54 | |
| *** xinliang has quit IRC | 07:55 | |
| *** ramishra has quit IRC | 07:56 | |
| *** hrubi has quit IRC | 07:57 | |
| *** hrubi has joined #openstack-infra | 07:57 | |
| *** ccamacho has joined #openstack-infra | 07:59 | |
| *** alexchadin has joined #openstack-infra | 08:00 | |
| *** sree has joined #openstack-infra | 08:02 | |
| *** jpich has joined #openstack-infra | 08:04 | |
| *** jascott1 has quit IRC | 08:05 | |
| *** markvoelker has quit IRC | 08:06 | |
| *** timrc has quit IRC | 08:07 | |
| *** eranrom has joined #openstack-infra | 08:07 | |
| *** dizquierdo has joined #openstack-infra | 08:07 | |
| *** timrc has joined #openstack-infra | 08:08 | |
| *** alexchadin has quit IRC | 08:08 | |
| *** xinliang has joined #openstack-infra | 08:08 | |
| *** dhajare has quit IRC | 08:12 | |
| *** iyamahat has joined #openstack-infra | 08:12 | |
| *** iyamahat_ has joined #openstack-infra | 08:14 | |
| *** iyamahat has quit IRC | 08:14 | |
| *** eranrom has quit IRC | 08:18 | |
| *** iyamahat_ has quit IRC | 08:20 | |
| *** zhurong has joined #openstack-infra | 08:21 | |
| *** dhajare has joined #openstack-infra | 08:21 | |
| *** ykarel_ is now known as ykarel|lunch | 08:22 | |
| *** _mwarad_ has quit IRC | 08:22 | |
| *** electrofelix has joined #openstack-infra | 08:26 | |
| *** pgadiya has quit IRC | 08:30 | |
| *** ralonsoh has joined #openstack-infra | 08:31 | |
| *** yamamoto has quit IRC | 08:31 | |
| *** alexchadin has joined #openstack-infra | 08:31 | |
| *** gildub has joined #openstack-infra | 08:34 | |
| *** yamamoto has joined #openstack-infra | 08:34 | |
| *** yamamoto has quit IRC | 08:39 | |
| *** udesale has quit IRC | 08:45 | |
| *** pgadiya has joined #openstack-infra | 08:46 | |
| *** yamamoto has joined #openstack-infra | 08:47 | |
| *** yamamoto has quit IRC | 08:47 | |
| *** udesale has joined #openstack-infra | 08:51 | |
| *** ykarel|lunch is now known as ykarel | 08:54 | |
| *** jpena|off is now known as jpena | 08:57 | |
| *** sree has quit IRC | 09:00 | |
| *** sree has joined #openstack-infra | 09:00 | |
| *** derekh has joined #openstack-infra | 09:00 | |
| *** dingyichen has quit IRC | 09:02 | |
| *** sree has quit IRC | 09:05 | |
| *** yamamoto has joined #openstack-infra | 09:10 | |
| *** udesale has quit IRC | 09:11 | |
| *** udesale has joined #openstack-infra | 09:11 | |
| *** yamamoto has quit IRC | 09:12 | |
| frickler | fungi: there is still more cleanup to do regarding apps.o.o, see https://ask.openstack.org/en/question/110400/is-the-openstack-community-app-catalog-dead/. I added a comment on the wiki, but someone with more background should do a complete update of that page | 09:14 |
| *** yamamoto has joined #openstack-infra | 09:14 | |
| frickler | fungi: also please pull the cname record for storage.apps.o.o, that site still does a js-redirect to apps.o.o | 09:14 |
| *** yamamoto has quit IRC | 09:16 | |
| *** yamamoto has joined #openstack-infra | 09:16 | |
| *** dtantsur is now known as dtantsur|lunch | 09:21 | |
| *** pbourke has quit IRC | 09:23 | |
| *** pbourke has joined #openstack-infra | 09:23 | |
| *** tosky has joined #openstack-infra | 09:29 | |
| *** stakeda has quit IRC | 09:30 | |
| *** yamamoto has quit IRC | 09:31 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci master: DNM: troubleshoot ocata periodic jobs https://review.openstack.org/507448 | 09:31 |
| *** yamamoto has joined #openstack-infra | 09:35 | |
| *** pgadiya has quit IRC | 09:45 | |
| *** zhurong has quit IRC | 09:52 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci master: Add sudo to ping https://review.openstack.org/507463 | 09:58 |
| *** shu-mutou is now known as shu-mutou-AWAY | 10:00 | |
| *** sdague has joined #openstack-infra | 10:00 | |
| *** jascott1 has joined #openstack-infra | 10:06 | |
| *** markvoelker has joined #openstack-infra | 10:07 | |
| *** ijw has joined #openstack-infra | 10:08 | |
| *** jascott1 has quit IRC | 10:10 | |
| *** hrubi has quit IRC | 10:11 | |
| *** rhallisey has joined #openstack-infra | 10:11 | |
| *** sree has joined #openstack-infra | 10:11 | |
| *** ijw has quit IRC | 10:13 | |
| *** armax has joined #openstack-infra | 10:15 | |
| *** LindaWang has quit IRC | 10:15 | |
| *** sree has quit IRC | 10:16 | |
| *** liceoprova has joined #openstack-infra | 10:19 | |
| *** pblaho has quit IRC | 10:22 | |
| *** cshastri has quit IRC | 10:26 | |
| *** gildub has quit IRC | 10:32 | |
| *** zhurong has joined #openstack-infra | 10:35 | |
| *** iyamahat has joined #openstack-infra | 10:35 | |
| *** iyamahat has quit IRC | 10:36 | |
| *** iyamahat has joined #openstack-infra | 10:37 | |
| *** udesale has quit IRC | 10:37 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Set repo setup release in playbook https://review.openstack.org/504939 | 10:40 |
| *** cuongnv has quit IRC | 10:41 | |
| *** markvoelker has quit IRC | 10:41 | |
| *** iyamahat has quit IRC | 10:43 | |
| *** iyamahat has joined #openstack-infra | 10:45 | |
| *** iyamahat has quit IRC | 10:46 | |
| *** iyamahat_ has joined #openstack-infra | 10:46 | |
| *** liceoprova has quit IRC | 10:48 | |
| *** liceoprova has joined #openstack-infra | 10:48 | |
| *** liujiong has quit IRC | 10:51 | |
| kong | hi, is it a right place to ask devstack related questions here? | 10:53 |
| *** udesale has joined #openstack-infra | 10:53 | |
| *** iyamahat_ has quit IRC | 10:54 | |
| *** jkilpatr has joined #openstack-infra | 10:54 | |
| kong | ok, just try my luck. I wonder how to add an additional service in an existing devstack environment? | 10:55 |
| liceoprova | http://echootv.org/live/showsat_PegD8u/2KzUQW6Y/10341.ts | 10:56 |
| *** liceoprova has quit IRC | 10:57 | |
| *** alexchadin has quit IRC | 10:58 | |
| *** alexchadin has joined #openstack-infra | 10:59 | |
| *** jkilpatr has quit IRC | 10:59 | |
| dims | infra-root : please take a look at this git repo swap request when you get a chance - https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg111037.html | 11:01 |
| dims | fungi : ^^ ( cc @zhenguo ) | 11:02 |
| zhenguo | dims: thanks :) | 11:03 |
| *** sulo has left #openstack-infra | 11:04 | |
| *** nicolasbock has joined #openstack-infra | 11:07 | |
| *** armax has quit IRC | 11:12 | |
| frickler | kong: for devstack please use #openstack-qa | 11:13 |
| kong | frickler: ok, thanks | 11:14 |
| *** alexchadin has quit IRC | 11:14 | |
| *** jkilpatr has joined #openstack-infra | 11:15 | |
| *** ralonsoh_ has joined #openstack-infra | 11:21 | |
| *** ykarel is now known as ykarel|afk | 11:21 | |
| *** e0ne has quit IRC | 11:21 | |
| *** rhallisey has quit IRC | 11:22 | |
| *** rossella_s has joined #openstack-infra | 11:22 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/irc-meetings master: Allow for meetings to be hosted in project channel https://review.openstack.org/507489 | 11:23 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/irc-meetings master: Reflect TC's office hours in the meetings calendar https://review.openstack.org/507490 | 11:23 |
| *** rhallisey has joined #openstack-infra | 11:24 | |
| *** ralonsoh has quit IRC | 11:24 | |
| *** iyamahat has joined #openstack-infra | 11:27 | |
| *** e0ne has joined #openstack-infra | 11:28 | |
| *** iyamahat has quit IRC | 11:29 | |
| *** alexchadin has joined #openstack-infra | 11:30 | |
| *** iyamahat has joined #openstack-infra | 11:31 | |
| *** pblaho has joined #openstack-infra | 11:31 | |
| *** thorst has quit IRC | 11:31 | |
| *** rossella_s has quit IRC | 11:33 | |
| *** zhurong has quit IRC | 11:35 | |
| *** markvoelker has joined #openstack-infra | 11:39 | |
| *** zhurong has joined #openstack-infra | 11:39 | |
| *** iyamahat has quit IRC | 11:48 | |
| *** bobh has joined #openstack-infra | 11:55 | |
| *** ykarel|afk is now known as ykarel | 11:57 | |
| *** dprince has joined #openstack-infra | 11:57 | |
| *** tpsilva has joined #openstack-infra | 12:01 | |
| *** jpena is now known as jpena|lunch | 12:03 | |
| *** timrc has quit IRC | 12:05 | |
| *** timrc has joined #openstack-infra | 12:06 | |
| *** dave-mccowan has joined #openstack-infra | 12:07 | |
| *** mat128 has joined #openstack-infra | 12:08 | |
| *** dave-mcc_ has joined #openstack-infra | 12:10 | |
| *** thorst has joined #openstack-infra | 12:11 | |
| *** dave-mccowan has quit IRC | 12:12 | |
| *** markvoelker has quit IRC | 12:12 | |
| *** dtantsur|lunch is now known as dtantsur | 12:15 | |
| *** raissa_ has quit IRC | 12:17 | |
| *** hrubi has joined #openstack-infra | 12:17 | |
| *** raissa_ has joined #openstack-infra | 12:20 | |
| *** sree has joined #openstack-infra | 12:20 | |
| *** rlandy has joined #openstack-infra | 12:22 | |
| *** wolverineav has joined #openstack-infra | 12:27 | |
| *** bobh has quit IRC | 12:27 | |
| *** udesale has quit IRC | 12:29 | |
| *** armax has joined #openstack-infra | 12:30 | |
| *** markvoelker has joined #openstack-infra | 12:32 | |
| *** hemna__ has joined #openstack-infra | 12:34 | |
| *** armax has quit IRC | 12:37 | |
| *** yamamoto has quit IRC | 12:38 | |
| mordred | 163452 | 12:40 |
| *** trown|outtypewww is now known as trown | 12:45 | |
| *** zhurong has quit IRC | 12:46 | |
| *** jaypipes has joined #openstack-infra | 12:49 | |
| *** mat128 has quit IRC | 12:52 | |
| *** bnemec has joined #openstack-infra | 12:52 | |
| fungi | now mordred is a numbers station | 12:52 |
| mordred | I'm always a numbers station | 12:52 |
| *** srobert has joined #openstack-infra | 12:54 | |
| *** yamamoto has joined #openstack-infra | 12:55 | |
| *** mat128 has joined #openstack-infra | 12:55 | |
| *** erlon has joined #openstack-infra | 12:57 | |
| *** esberglu has joined #openstack-infra | 12:57 | |
| *** mat128 has quit IRC | 12:57 | |
| *** sree has quit IRC | 12:58 | |
| *** dizquierdo has quit IRC | 12:58 | |
| *** sree has joined #openstack-infra | 12:58 | |
| *** LindaWang has joined #openstack-infra | 12:58 | |
| *** sree has quit IRC | 12:59 | |
| *** sree has joined #openstack-infra | 12:59 | |
| *** Dinesh_Bhor has quit IRC | 12:59 | |
| fungi | frickler: thanks for the suggestion. i've cleaned up dns records for storage.apps and also apps-beta and apps-dev | 12:59 |
| *** sree has quit IRC | 13:00 | |
| *** sree has joined #openstack-infra | 13:00 | |
| *** jpena|lunch is now known as jpena | 13:01 | |
| frickler | fungi: cool, thx | 13:01 |
| *** alexchadin has quit IRC | 13:01 | |
| *** jcoufal has joined #openstack-infra | 13:04 | |
| fungi | i also tweaked the warning heading on that wiki article to "no longer in service" | 13:05 |
| *** yamamoto has quit IRC | 13:06 | |
| *** mat128 has joined #openstack-infra | 13:07 | |
| *** Dinesh_Bhor has joined #openstack-infra | 13:08 | |
| *** kuromagi has quit IRC | 13:09 | |
| *** kuromagi has joined #openstack-infra | 13:10 | |
| *** mat128 has quit IRC | 13:10 | |
| fungi | frickler: a bit of poking in search engines points out that we don't clean up the published copies on the governance site when we move/remove files in the build: https://governance.openstack.org/tc/reference/projects/community-app-catalog.html | 13:10 |
| *** mriedem has joined #openstack-infra | 13:11 | |
| *** bobh has joined #openstack-infra | 13:11 | |
| fungi | maybe after zuul v3 is behind us, we can start looking at leveraging our docs site mechanisms for a lot of our other static content sites too | 13:11 |
| mordred | fungi: ++ | 13:12 |
| *** mat128 has joined #openstack-infra | 13:12 | |
| *** Goneri has joined #openstack-infra | 13:14 | |
| *** mat128 has quit IRC | 13:15 | |
| *** kgiusti has joined #openstack-infra | 13:17 | |
| *** bh526r has joined #openstack-infra | 13:20 | |
| *** kjackal_ has joined #openstack-infra | 13:20 | |
| *** amoralej is now known as amoralej|lunch | 13:21 | |
| *** mat128 has joined #openstack-infra | 13:22 | |
| *** ykarel has quit IRC | 13:22 | |
| *** baoli has joined #openstack-infra | 13:23 | |
| mordred | infra-root: just pushed up four patches to zuul that didn't report here- but they pick up the timeout thing from yesterday and the file rename - as well as add two sorted calls | 13:25 |
| mordred | https://review.openstack.org/507535 https://review.openstack.org/507536 https://review.openstack.org/507537 https://review.openstack.org/507538 | 13:25 |
| Shrews | already reviewing | 13:26 |
| mordred | infra-root: also - yes, there was some leftover cruft from previous runs in the playbook dir (sorry) - I'm re-running the migration script with everything rebased and clean and whatnot | 13:27 |
| Shrews | mordred: is sorting required-projects a visual thing, or a necessity? | 13:27 |
| mordred | Shrews: yah - just visual | 13:27 |
| *** udesale has joined #openstack-infra | 13:27 | |
| mordred | Shrews: it stood out when I was comparing diffs from a previous run - and python decided to reorder the entries for no reason | 13:27 |
| mordred | (because they go through a set() I believe) | 13:27 |
| *** ihrachys has joined #openstack-infra | 13:27 | |
| Shrews | i like predictability | 13:27 |
| mordred | yah | 13:28 |
| mnaser | https://review.openstack.org/#/c/507538 isnt this technically removing the sorting? | 13:29 |
| mnaser | or does ordereddict not always serialize order properly | 13:29 |
| mordred | mnaser: I have found that passing kwargs to ordereddict constructor does not preserve order of the kwargs | 13:32 |
| mordred | which makes sense, since python **kwargs are a normal unordered dict | 13:32 |
| mnaser | TIL | 13:32 |
| mordred | yah - I've learned many things in this process | 13:32 |
| *** felipemonteiro has joined #openstack-infra | 13:32 | |
| mordred | mnaser: if you wanna see some real crack - check out the http://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/add-projects-to-main.py script :) | 13:33 |
| mordred | mnaser: it uses ruamel.yaml to do yaml things while preserving comments ... and is COMPLETELY insane | 13:33 |
| *** mat128 has quit IRC | 13:33 | |
| Shrews | i was about to ask about the ordereddict thing | 13:34 |
| mnaser | shoulda just used mechnical turk | 13:34 |
| mnaser | :P | 13:34 |
| *** felipemonteiro_ has joined #openstack-infra | 13:34 | |
| mnaser | my brains stops functioning when anything involving tokens is involved | 13:35 |
| Shrews | mordred: oops, need to update the zuul-migrate job | 13:35 |
| Shrews | still uses the -m option | 13:35 |
| mordred | whoops | 13:36 |
| *** felipemonteiro has quit IRC | 13:38 | |
| *** dmsimard|off is now known as dmsimard | 13:39 | |
| *** mat128 has joined #openstack-infra | 13:40 | |
| *** armax has joined #openstack-infra | 13:41 | |
| *** mat128 has quit IRC | 13:41 | |
| *** mat128 has joined #openstack-infra | 13:42 | |
| *** liujiong has joined #openstack-infra | 13:46 | |
| *** srobert has quit IRC | 13:46 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Rename outfiles to zuul- to match rollout https://review.openstack.org/507535 | 13:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Sort list of required projects https://review.openstack.org/507536 | 13:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Set a max on timeout https://review.openstack.org/507537 | 13:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Sort fields in post playbook https://review.openstack.org/507538 | 13:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add local file copying steps https://review.openstack.org/507544 | 13:46 |
| *** srobert has joined #openstack-infra | 13:46 | |
| mordred | Shrews: therewego | 13:46 |
| *** camunoz has joined #openstack-infra | 13:47 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add local file copying steps https://review.openstack.org/507544 | 13:47 |
| *** ykarel has joined #openstack-infra | 13:48 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add local file copying steps https://review.openstack.org/507544 | 13:48 |
| *** camunoz has quit IRC | 13:55 | |
| *** camunoz has joined #openstack-infra | 13:55 | |
| *** eranrom has joined #openstack-infra | 13:56 | |
| *** eharney has joined #openstack-infra | 13:58 | |
| *** armax has quit IRC | 13:58 | |
| *** eranrom has quit IRC | 14:00 | |
| *** eranrom has joined #openstack-infra | 14:05 | |
| *** yamamoto has joined #openstack-infra | 14:07 | |
| *** armax has joined #openstack-infra | 14:07 | |
| *** jascott1 has joined #openstack-infra | 14:08 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: switch whereto to publish docs on docs.o.o https://review.openstack.org/507243 | 14:11 |
| *** amoralej|lunch is now known as amoralej | 14:12 | |
| *** yamamoto has quit IRC | 14:14 | |
| *** jascott1 has quit IRC | 14:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 14:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add jobs for project-config https://review.openstack.org/507551 | 14:16 |
| *** baoli has quit IRC | 14:17 | |
| *** baoli has joined #openstack-infra | 14:17 | |
| *** eranrom has quit IRC | 14:17 | |
| *** mwarad has joined #openstack-infra | 14:19 | |
| *** rossella_s has joined #openstack-infra | 14:20 | |
| mnaser | hmm | 14:20 |
| mnaser | i can consistently get gerrit to give me a 500 | 14:20 |
| mnaser | https://review.openstack.org/#/c/507435/ > rebase with ui > enter "507555" | 14:21 |
| mnaser | and im seeing timeouts to git.openstack.org from ovh | 14:21 |
| *** felipemonteiro_ has quit IRC | 14:22 | |
| mnaser | http://logs.openstack.org/50/507550/1/check/gate-puppet-ceph-puppet-unit-4.8-centos-7/8af24a7/console.html | 14:22 |
| *** rbrndt has joined #openstack-infra | 14:23 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Limit base-integration jobs https://review.openstack.org/507558 | 14:24 |
| *** mwarad has quit IRC | 14:24 | |
| *** sree has quit IRC | 14:24 | |
| *** dizquierdo has joined #openstack-infra | 14:25 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add jobs for project-config https://review.openstack.org/507551 | 14:30 |
| *** rossella_s has quit IRC | 14:31 | |
| *** makowals has joined #openstack-infra | 14:34 | |
| *** xarses has joined #openstack-infra | 14:42 | |
| openstackgerrit | Gage Hugo proposed openstack-infra/project-config master: Skip ansible upgrade job in keystone https://review.openstack.org/505426 | 14:42 |
| *** lbragstad has quit IRC | 14:44 | |
| clarkb | mnaser: I think if the rebase has conflicts you get a 500 | 14:45 |
| clarkb | mnaser: maybe try it locally and see if that is the case? | 14:45 |
| *** srobert has quit IRC | 14:46 | |
| mnaser | if i remember correctly it'll tell me there's a merge conflict in the UI usually | 14:46 |
| mnaser | let me see | 14:46 |
| clarkb | mordred: did you see there was confusion over the source of your migration data? | 14:46 |
| mnaser | http://logs.openstack.org/72/507472/1/gate/gate-puppet-openstack-integration-4-scenario002-tempest-ubuntu-xenial/c73dbf0/console.html | 14:47 |
| mnaser | more timeouts over at ovh | 14:47 |
| mnaser | unable to access git.o.o | 14:47 |
| mordred | clarkb: yes - I'm finishing up getting it cleaned up (and also making a new patch to decouple a section of the migration rollout that does not need to be coupled) | 14:47 |
| clarkb | mordred: in any case we merged the migration fixes from yesterday and expect to generate new migration data today. Also we should make sure that ansible lint syntax check runs over it though that may be slow in CI | 14:47 |
| fungi | mnaser: isolated to one region in ovh, or multiple? | 14:47 |
| *** hongbin has joined #openstack-infra | 14:47 | |
| mnaser | fungi every case i've seen fail is ovh | 14:47 |
| mordred | clarkb: yup. totally agree | 14:47 |
| fungi | mnaser: sure, but more than one ovh region? (gra1, bhs1...) | 14:48 |
| mnaser | http://logs.openstack.org/12/507512/6/check/gate-puppet-octavia-puppet-syntax-4-centos-7/22578c6/console.html - ovh-gra1 failure | 14:48 |
| mnaser | all failure have been gra1 | 14:48 |
| fungi | okay, so coming across the atlantic | 14:48 |
| fungi | not from canada | 14:49 |
| fungi | could be a backbone peering issue, or could be something local to their environment i suppose. either way we can temporarily remove them from nodepool if retries aren't working around it sufficiently | 14:50 |
| *** trown is now known as trown|brb | 14:50 | |
| fungi | though it looks like no retries there. i guess 282099 never actually landed | 14:50 |
| fungi | zuul v3 will make the need for that obsolete anyway | 14:51 |
| *** ralonsoh_ is now known as ralonsoh | 14:52 | |
| *** lbragstad has joined #openstack-infra | 14:54 | |
| *** eharney has quit IRC | 14:56 | |
| *** jamesmcarthur has quit IRC | 14:56 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make project-config gating in zuul v3 https://review.openstack.org/507219 | 14:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 14:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Revert "Revert "Add the rest of the projects to Zuul v3"" https://review.openstack.org/507564 | 14:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 14:56 |
| *** liujiong has quit IRC | 14:59 | |
| jeblair | o/ | 15:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add jobs for project-config https://review.openstack.org/507551 | 15:02 |
| clarkb | mordred: is ^ the split out change you were talking about? | 15:03 |
| *** ganso has joined #openstack-infra | 15:04 | |
| Shrews | mordred: i'm trying to figure out the "no such file or directory" error from 507544, but the script runs perfectly on my system :( | 15:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Rename outfiles to zuul- to match rollout https://review.openstack.org/507535 | 15:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Sort list of required projects https://review.openstack.org/507536 | 15:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Set a max on timeout https://review.openstack.org/507537 | 15:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Sort fields in post playbook https://review.openstack.org/507538 | 15:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add local file copying steps https://review.openstack.org/507544 | 15:05 |
| mordred | clarkb: yes | 15:06 |
| mordred | Shrews: oh - looking | 15:06 |
| *** kgiusti has quit IRC | 15:06 | |
| mordred | Shrews: I see it | 15:07 |
| inc0 | hey guys, how is zuulv3 implementation going? Anything I can help with? | 15:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add local file copying steps https://review.openstack.org/507544 | 15:07 |
| mordred | inc0: it's going well! we hit a couple of small issues yesterday then ran out of time before it was EOD - once we're awake we'll be taking another stab at it | 15:08 |
| ganso | hello guys. I am stacking my devstack again for the first time since a long time. I don't see my logs in a screen anymore. I see that the services are running through systemd now, but I can't seem to find the screens with the logs. Is this a known bug or we are not doing that anymore? is there a doc entry to get the screens back up? | 15:08 |
| mordred | inc0: and I think we're good- but thanks! | 15:08 |
| mordred | ganso: there is a doc about devstack and systemd ... one sec | 15:09 |
| Shrews | gah | 15:09 |
| mordred | ganso: https://docs.openstack.org/devstack/latest/systemd.html | 15:09 |
| ganso | mordred: thanks! :) | 15:09 |
| mordred | ganso: sure thing! | 15:09 |
| fungi | ganso: also you might consider asking future devstack questions on #openstack-qa, devstack is not an infra project | 15:09 |
| mordred | infra-root: I have updated the etherpad with a new list of changes to land before we go through rollout again - and have also updated the rollout and rollback steps | 15:10 |
| fungi | (granted, we do happen to know a fair amount about it and a lot of the devstack authors/maintainers also hang out in this channel) | 15:10 |
| ganso | fungi: oh sorry, will ask devstack-related questions on the QA channel from now on, thanks! | 15:10 |
| *** slaweq_ has joined #openstack-infra | 15:10 | |
| jeblair | mordred: reviewing changes now | 15:10 |
| fungi | ganso: no apology necessary, it's probably a confusing division of labor there | 15:10 |
| mordred | jeblair: thanks! | 15:10 |
| *** eharney has joined #openstack-infra | 15:11 | |
| *** jamesmcarthur has joined #openstack-infra | 15:12 | |
| *** felipemonteiro has joined #openstack-infra | 15:15 | |
| mordred | infra-root: tl;dr on a couple of them - 507551 and 507219 are a manual creation/migration of the project-config jobs so that the steps "make v3 gating on project-config" don't need to depend on landing the larger set of new jobs in OZJ (that needs a better acronym) | 15:15 |
| jeblair | mordred: -1 on 507544 | 15:15 |
| *** rnoriega_ is now known as rnoriega | 15:16 | |
| mordred | jeblair: yah - just saw - fixing | 15:16 |
| *** felipemonteiro_ has joined #openstack-infra | 15:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add local file copying steps https://review.openstack.org/507544 | 15:17 |
| *** srobert has joined #openstack-infra | 15:17 | |
| *** trown|brb is now known as trown | 15:18 | |
| *** andreas_s has quit IRC | 15:18 | |
| openstackgerrit | Gage Hugo proposed openstack-infra/project-config master: Skip non-doc jobs for keystone libraries https://review.openstack.org/504243 | 15:19 |
| *** felipemonteiro has quit IRC | 15:20 | |
| *** felipemonteiro_ has quit IRC | 15:21 | |
| *** tonyb has quit IRC | 15:22 | |
| mordred | Shrews: wanna re: +3 https://review.openstack.org/#/c/507544 ? | 15:23 |
| Shrews | NO | 15:23 |
| Shrews | i mean | 15:23 |
| Shrews | DONE | 15:23 |
| mordred | heh | 15:23 |
| *** tonyb has joined #openstack-infra | 15:24 | |
| jeblair | what does 'build-specs-site' do? | 15:24 |
| jeblair | mordred: ^ | 15:25 |
| fungi | is that the one that regenerates the site index? | 15:25 |
| mordred | jeblair: that runs the tox environment 'specs' to generate the site index - but does not publish it | 15:26 |
| *** ganso has left #openstack-infra | 15:26 | |
| jeblair | that's for the root of the specs site, or the infra root? | 15:26 |
| mordred | publish-openstack-specs-site does the publication - I suppose we should maybe call build-specs-site build-opnestack-specs-site ... | 15:26 |
| jeblair | there is no infra root | 15:27 |
| jeblair | so i guess that's the answer to my question :) | 15:27 |
| mordred | :) | 15:27 |
| mordred | jeblair: shall I rename thatto build-openstack-specs-site real quick? | 15:27 |
| jeblair | mordred: i'm writing descriptions for all those jobs you're adding in 551 | 15:27 |
| jeblair | mordred: that sounds good to me | 15:28 |
| *** e0ne has quit IRC | 15:28 | |
| mordred | jeblair: ok. one sec | 15:28 |
| jeblair | mordred: oh | 15:28 |
| jeblair | mordred: left a -1 on that too; maybe fix in same update | 15:28 |
| mordred | kk | 15:28 |
| *** hashar has quit IRC | 15:30 | |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Add some missing job descriptions https://review.openstack.org/507566 | 15:30 |
| clarkb | mordred: looks like https://review.openstack.org/#/c/507535/3 is still running py35 job and failing (does that imply it does break py35?) | 15:31 |
| jeblair | going forward, let's try not to add jobs to zuul-jobs, openstack-zuul-jobs, or project-config without descriptions | 15:31 |
| jeblair | i think we'll want a gate job to force that eventually | 15:31 |
| *** sree has joined #openstack-infra | 15:32 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add jobs for project-config https://review.openstack.org/507551 | 15:32 |
| *** psachin has quit IRC | 15:32 | |
| mordred | jeblair: ^^ thereyougo | 15:33 |
| mordred | jeblair: and yes - I totally agree | 15:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add some missing job descriptions https://review.openstack.org/507566 | 15:33 |
| mordred | jeblair: ^^ rebased yours | 15:33 |
| jeblair | mordred: double thanxs | 15:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 15:34 |
| *** dhajare has quit IRC | 15:34 | |
| jeblair | clarkb: i don't think those py35 errors are related; i think they're running on a very slow node | 15:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make project-config gating in zuul v3 https://review.openstack.org/507219 | 15:35 |
| mordred | jeblair: yah - but I'd added in a ... oh, there'sa .zuul.yaml change there so irrelevant files didn't exclude the job | 15:36 |
| jeblair | mordred: yep. i think that's probably okay. :) | 15:36 |
| mordred | yah | 15:36 |
| *** sree has quit IRC | 15:37 | |
| clarkb | becuaes we aren't speculatively applying that change? | 15:38 |
| *** armax has quit IRC | 15:38 | |
| jeblair | clarkb: we are speculatively applying it, but the change includes a change to .zuul.yaml, which isn't on the irrelevant files list (which makes it a *relevant* file). so we run the test. | 15:41 |
| clarkb | gotcha | 15:41 |
| jeblair | mordred: zuul -1 on 507219 | 15:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make project-config gating in zuul v3 https://review.openstack.org/507219 | 15:42 |
| mordred | jeblair: blerg | 15:42 |
| *** felipemonteiro has joined #openstack-infra | 15:42 | |
| mordred | clarkb: feel like a +3 on https://review.openstack.org/#/c/507551 ? | 15:43 |
| clarkb | mordred: re changing the afs package location will we have to update puppet where we consume that? | 15:44 |
| mordred | clarkb: ah- yah - we should probably do that, yes | 15:44 |
| clarkb | synchronize pull implies delegate to I guess | 15:45 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-hosts-file https://review.openstack.org/505789 | 15:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-openafs master: Pull centos packages from project-config subdir https://review.openstack.org/507614 | 15:46 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall https://review.openstack.org/504788 | 15:47 |
| mordred | clarkb: https://review.openstack.org/507614 | 15:47 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge role https://review.openstack.org/504789 | 15:47 |
| mordred | clarkb: yah - synchronize is special in general | 15:47 |
| clarkb | mordred: ty | 15:47 |
| *** jcoufal has quit IRC | 15:48 | |
| *** jcoufal_ has joined #openstack-infra | 15:48 | |
| *** ykarel is now known as ykarel|afk | 15:48 | |
| *** LindaWang has quit IRC | 15:49 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add multinode integration jobs and integration tests for known_hosts https://review.openstack.org/504787 | 15:50 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-hosts-file https://review.openstack.org/505789 | 15:50 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall https://review.openstack.org/504788 | 15:50 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge role https://review.openstack.org/504789 | 15:50 |
| dmsimard | sorry for the rebase spam | 15:50 |
| Shrews | FYI, not sure how close we are to pulling the zuulv3 trigger again, but will need to step out in about 30m or so. | 15:51 |
| mordred | Shrews: I think once the pre-req patches all land we'll be ready to pull the trigger (basically, once we've got project-config transitioned to gating in v3, I think we're ready to go) | 15:51 |
| *** ccamacho has quit IRC | 15:51 | |
| mordred | jeblair: mind hitting https://review.openstack.org/#/c/507614/ ? | 15:52 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add multinode integration jobs and integration tests for known_hosts https://review.openstack.org/504787 | 15:52 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-hosts-file https://review.openstack.org/505789 | 15:52 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall https://review.openstack.org/504788 | 15:52 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge role https://review.openstack.org/504789 | 15:52 |
| dmsimard | mordred: I think that should do it for integration tests ^ looking at the actual roles now.. | 15:52 |
| mordred | Shrews: how long you gonna be afk? (no worries either way, just curious) | 15:53 |
| dmsimard | mordred: looks like the roles from zuul-jobs are clear from rebase conflicts (I guess cause it's new stuff.) | 15:53 |
| *** kiennt26 has joined #openstack-infra | 15:54 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 15:54 |
| Shrews | mordred: 1.5h or so? got a chiro appt and meeting someone for lunch | 15:54 |
| mordred | cool | 15:57 |
| jeblair | mordred: done. looks like everything except 219 is approved and gating | 15:57 |
| jeblair | mordred: another zuul -1 on 219 | 15:57 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Rename outfiles to zuul- to match rollout https://review.openstack.org/507535 | 15:58 |
| *** srobert has quit IRC | 15:58 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Sort list of required projects https://review.openstack.org/507536 | 15:58 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Set a max on timeout https://review.openstack.org/507537 | 15:58 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Sort fields in post playbook https://review.openstack.org/507538 | 15:58 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add local file copying steps https://review.openstack.org/507544 | 15:59 |
| *** srobert has joined #openstack-infra | 15:59 | |
| *** egonzalez has quit IRC | 16:00 | |
| *** sbezverk has quit IRC | 16:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make project-config gating in zuul v3 https://review.openstack.org/507219 | 16:00 |
| *** jistr is now known as jistr|mtg | 16:01 | |
| clarkb | my parents are in town and this is the day my brothers have off work, which means I'm getting dragged to the coast with everyone. I expect I'll be able to run the meeting and keep an eye on zuulv3 things (and help out where I can) via a cell phone tether. But in case things don't work out as expected I've asked fungi to run the meeting (assuming we don't cancel due to the transition) | 16:01 |
| pabelanger | clarkb: ack | 16:02 |
| fungi | yeah, also be forewarned my power may go out, so it wouldn't hurt for us to get a backup backup volunteer chair as well | 16:02 |
| *** dprince has quit IRC | 16:02 | |
| * fungi is getting a visit from maria this afternoon | 16:02 | |
| jeblair | mordred: KeyError: 'bindep-fallback' | 16:03 |
| jeblair | mordred: that's the unknown config error. i'll go work on a patch to zuul to make that a known config error. | 16:03 |
| *** ykarel|afk is now known as ykarel | 16:04 | |
| mordred | jeblair: isn't the issue actually that it wants just '- bindep-fallback' and not '- name: bindep-fallback' ? | 16:04 |
| *** dprince has joined #openstack-infra | 16:05 | |
| *** rcernin has quit IRC | 16:05 | |
| jeblair | mordred: you fixed that in ps8. the new error is on ps8. | 16:05 |
| mordred | oh! silly me | 16:06 |
| mordred | jeblair: that's the error we expect at this point becaue the ozj patch hasn't landed and project-config is a config repo yah? | 16:06 |
| *** udesale has quit IRC | 16:08 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Consume project templates correctly in projects https://review.openstack.org/507616 | 16:08 |
| mordred | jeblair, Shrews, clarkb: ^^ that fixes that in the migration script | 16:08 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Enable Debug: True for all OVB jobs https://review.openstack.org/507241 | 16:08 |
| *** Apoorva has joined #openstack-infra | 16:09 | |
| *** dizquierdo has quit IRC | 16:09 | |
| tinwood | asettle, have you got a moment please for me to ask a question about https://review.openstack.org/#/c/507268/1 (Add link to OpenStack Charms Deployment Guide) -- it's tangential, but I can't work out how the charm-deployment-guide has ended up where it is in docs.openstack.org tree !? | 16:11 |
| asettle | tinwood: hellooooo | 16:11 |
| tinwood | hi | 16:11 |
| Shrews | mordred: you sure it isn't project['template'] ??? it's 'template' in the yaml | 16:11 |
| asettle | tinwood: what's confusing you about the placement? | 16:12 |
| tinwood | asettle, I asked over in #openstack-docs but it seemed kind of quiet there. Do you want to move there, or stay here? | 16:12 |
| mordred | Shrews: it's template in v2 - it's templates in v3 | 16:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Handle unknown project-template error https://review.openstack.org/507617 | 16:12 |
| asettle | tinwood: sorry, that's probably my fault. I admittedly am not always doing a great job of monitoring that chan | 16:12 |
| asettle | Let's switch there | 16:12 |
| tinwood | asettle, okay | 16:12 |
| Shrews | mordred: right, but you change it to 'templates' on line 1388 (where it should be the v2 project) | 16:12 |
| Shrews | mordred: or i'm wrong about that... | 16:13 |
| Shrews | oh, no, that's v3 | 16:13 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall https://review.openstack.org/504788 | 16:13 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge role https://review.openstack.org/504789 | 16:13 |
| asettle | tinwood: heh #openstack-doc (singular) that's why it was quiet ;) | 16:13 |
| mordred | Shrews: hrm. no - there's definitely an issue there ... | 16:13 |
| tinwood | asettle, ahhh | 16:13 |
| * tinwood doh | 16:13 | |
| tinwood | I'll go to the *right* one! | 16:14 |
| * Shrews needs to pull up the entire script.. | 16:14 | |
| mgagne | looking at this bug https://bugs.launchpad.net/openstack-gate/+bug/1617309 | 16:15 |
| openstack | Launchpad bug 1617309 in OpenStack-Gate "internap-mtl01 network connectivity issues" [Undecided,New] | 16:15 |
| *** armax has joined #openstack-infra | 16:15 | |
| mgagne | I found that according to logstash, it's not isolated to mtl01 | 16:15 |
| mordred | Shrews: I'm adding a comment about which thing is which thing | 16:16 |
| Shrews | mordred: i think that's correct, since it gets passed the new project | 16:16 |
| Shrews | just had to wrap my mind around what thing is what version | 16:17 |
| *** jistr|mtg is now known as jistr | 16:18 | |
| mordred | Shrews: it's still not fully working - I'm winding up with both templates and template sections :( | 16:20 |
| *** armax has quit IRC | 16:20 | |
| Shrews | mordred: oops | 16:20 |
| jeblair | mordred: i only see templates: in the output | 16:20 |
| jeblair | http://logs.openstack.org/16/507616/1/check/zuul-migrate/d862fec/openstack-zuul-jobs/zuul.d/projects.yaml | 16:20 |
| jeblair | where do you see template? | 16:20 |
| mordred | locally - one sec, lemme push up ... | 16:21 |
| mordred | OH - I know why | 16:21 |
| mordred | sorry - operator error locally | 16:22 |
| *** jpich has quit IRC | 16:22 | |
| Shrews | hah. ok, reapplying my +2 then | 16:22 |
| Shrews | gtg now. bbs | 16:23 |
| mordred | (the git reset in the script was competing with me having checked out the prior version so that it wouldn't merge with itself) | 16:23 |
| *** trown is now known as trown|lunch | 16:23 | |
| mordred | Shrews: cool! have fun | 16:23 |
| fungi | mgagne: that's the problem with trying to bend a system geared for tracking bugs in software to also track short-term incidents in infrastructure | 16:24 |
| fungi | mgagne: note that bug is well over a year old now so almost certainly no longer relevant (at least in its current form) | 16:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Revert "Revert "Add the rest of the projects to Zuul v3"" https://review.openstack.org/507564 | 16:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 16:24 |
| fungi | mgagne: probably just needs to be closed out and any related queries removed from elastic-recheck (if there still are any) | 16:25 |
| fungi | in most intermittent connectivity cases that we never have a clear resolution, the situation just silently solves itself | 16:26 |
| mgagne | there are still occurrences but it looks to be specific to an other provider: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22errno%3DNetwork%20is%20unreachable%5C%22%20AND%20tags:console%20AND%20voting:1&from=864000s | 16:26 |
| fungi | sometimes one of us has the time to traceroute bidirectionally and then go hunting on nanog for discussions of backbone peering problems or the like | 16:27 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add jobs for project-config https://review.openstack.org/507551 | 16:27 |
| *** r-daneel has joined #openstack-infra | 16:27 | |
| fungi | mgagne: yeah, that just looks like the query in e-r was less specific than the bug which was opened | 16:28 |
| jeblair | mordred: you said 507219 depends on something? | 16:28 |
| mordred | jeblair: that thing has now merged | 16:28 |
| clarkb | is 616 good now? | 16:28 |
| mordred | jeblair: https://review.openstack.org/507551 was what it depended on | 16:28 |
| *** jamesmcarthur has quit IRC | 16:28 | |
| clarkb | seems like lots of back and forth on that one | 16:29 |
| mordred | clarkb: yah | 16:29 |
| mordred | clarkb: I had a local derp - the patch was good | 16:29 |
| fungi | mgagne: also, nobody in infra ever looks at openstack-gate bugs as far as i know. that repo exists solely so people who are building elastic-recheck queries can have a point of reference for them | 16:29 |
| clarkb | mordred: gotcha | 16:29 |
| jeblair | mordred: what adds the bindep-fallback template? | 16:29 |
| mgagne | ok, just wanted to make sure there was no pending issue related to inap-mtl01 | 16:29 |
| fungi | mgagne: not that i know of, and certainly not one which has persisted for over a year since the bug was opened | 16:30 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Consume project templates correctly in projects https://review.openstack.org/507616 | 16:31 |
| mordred | jeblair: oh crapbasket. that's in the big job dump | 16:31 |
| *** tosky has quit IRC | 16:35 | |
| *** kiennt26 has quit IRC | 16:35 | |
| mordred | jeblair: well - I was mostly trying to decouple flipping project-config gating from the bigger ozj patch ... but maybe we're gonna spin too many wheels on that at this point | 16:35 |
| mordred | jeblair: I mean, we can make the bindep fallback jobs real quick of course, but that was a better idea to me when I was doing it this morning before people woke up | 16:36 |
| jeblair | mordred: whatever gets us running faster. :) | 16:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Don't emit yaml suffixes in run and post-run lines https://review.openstack.org/507624 | 16:37 |
| mordred | jeblair: how do you feel about cheating just a little bit? | 16:38 |
| jeblair | mordred: not good | 16:39 |
| mordred | the suggestion I was going to make was to put an bindep-fallback project-template into ozj so that we can land the project-config patch - I doubt we're going to land bindep-fallback changes in the next hour - then replace the empty template with the real one in the large jobs patch | 16:40 |
| jeblair | mordred: what does decoupling project-config get us? | 16:40 |
| mordred | jeblair: just let's use have that landed and not part of the roll-forward/roll-back sequencing | 16:40 |
| mordred | not decouping also obviously works | 16:41 |
| mordred | and we just nee to swap the main.yaml patch and the project-config-gating patch | 16:41 |
| jeblair | mordred: i think we should re-couple and move forward. if we roll-back, we'll have another opportunity to try decoupling. | 16:41 |
| *** jamesmcarthur has joined #openstack-infra | 16:41 | |
| *** jamesmcarthur has quit IRC | 16:41 | |
| mordred | ok. cool. lemme change those stacks real quick and we'll be good to go | 16:41 |
| *** jamesmcarthur has joined #openstack-infra | 16:41 | |
| *** kjackal_ has quit IRC | 16:42 | |
| *** dtantsur is now known as dtantsur|afk | 16:42 | |
| *** jascott1 has joined #openstack-infra | 16:43 | |
| openstackgerrit | melanie witt proposed openstack-infra/devstack-gate master: WIP Add mysqladmin -v extended-status processlist https://review.openstack.org/507626 | 16:43 |
| jeblair | i've let the release team know we're about to start work again and to please not make releases | 16:45 |
| jeblair | this job looks really unhappy: http://zuulv3.openstack.org/static/stream.html?uuid=fd6fc295e9764d2d9e9e335481d7c2b6&logfile=console.log | 16:47 |
| jeblair | looks like each playbook takes about 10m to timeout | 16:47 |
| mordred | jeblair: I think it's having ssh connection problems during fact gathering | 16:48 |
| pabelanger | agree | 16:49 |
| jeblair | mordred: ya; that must happen to also return exit code 4 | 16:49 |
| *** SumitNaiksatam has quit IRC | 16:49 | |
| mordred | yah. so our log header of "parse error" may be a bit erroneous | 16:49 |
| jeblair | it's unfortunate it takes so long to error out | 16:50 |
| jeblair | that's been holding up the gate pipeline for an hour. | 16:50 |
| *** ykarel is now known as ykarel|away | 16:50 | |
| mordred | agree | 16:51 |
| pabelanger | SSH connects but handshake is failing it seems | 16:51 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Handle unknown project-template error https://review.openstack.org/507617 | 16:51 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't emit yaml suffixes in run and post-run lines https://review.openstack.org/507624 | 16:51 |
| jeblair | i guess the only way we could detect that is by parsing job-output.json and checknig for unreachable? | 16:52 |
| *** claudiub has quit IRC | 16:52 | |
| *** derekh has quit IRC | 16:54 | |
| jeblair | mordred: i saved job-output.json from that job; anything else i should save before it terminates? | 16:54 |
| mordred | jeblair: no | 16:54 |
| jeblair | (and yes, exit code is 4) | 16:55 |
| mordred | awesome | 16:55 |
| pabelanger | okay, I am in the node | 16:55 |
| pabelanger | I used a different cipher | 16:55 |
| jeblair | pabelanger: ? | 16:56 |
| pabelanger | and I think it was just killed | 16:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Revert "Revert "Add the rest of the projects to Zuul v3"" https://review.openstack.org/507564 | 16:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make project-config gating in zuul v3 https://review.openstack.org/507219 | 16:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 16:56 |
| jeblair | pabelanger: yes, job just finished | 16:56 |
| pabelanger | ssh -v -c aes128-ctr -oConnectTimeout=5 root@46.254.11.126 | 16:56 |
| pabelanger | that worked | 16:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 16:56 |
| pabelanger | so, maybe something was wrong with our hostkeys | 16:56 |
| jeblair | pabelanger: ya, maybe next time we spot this, we'll have time to examine them on host | 16:57 |
| jeblair | pabelanger: also, your use of connecttimeout makes me wonder if we can change an ansible setting to make this fail faster than 15 minutes | 16:57 |
| mordred | ++ | 16:58 |
| pabelanger | jeblair: possible, when I didn't have -c aes128-ctr, it did hang (greater then 5 seconds) | 16:58 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add some missing job descriptions https://review.openstack.org/507566 | 16:58 |
| jeblair | pabelanger: oh, so if there is an option, that's not the one :/ | 16:58 |
| mordred | infra-root: I've updated the etherpad rollout steps and have rebased the patches | 16:59 |
| pabelanger | ya, hopefully it happens again, and we can properly debug | 16:59 |
| pabelanger | Hmm | 17:00 |
| pabelanger | I am seeing a number of failures in citycloud-sto2 around SSH connections | 17:01 |
| pabelanger | I wonder if something is up | 17:01 |
| pabelanger | that is on nodepool.o.o | 17:01 |
| *** baoli has quit IRC | 17:01 | |
| mordred | maybe we should turn down citycloud-sto2 for now so issues there don't make us think there are v3 issue? | 17:02 |
| jeblair | pabelanger: http://zuulv3.openstack.org/static/stream.html?uuid=08282a21fa2b45eead4eeeb5069efdc1&logfile=console.log | 17:02 |
| *** slaweq_ has quit IRC | 17:02 | |
| jeblair | that job is seeing it too, but is also about to finish | 17:02 |
| *** baoli has joined #openstack-infra | 17:02 | |
| jeblair | pabelanger: are those both sto2 nodes? | 17:03 |
| mordred | that is citycloud-sto2 and I was able to ssh in | 17:03 |
| pabelanger | jeblair: yes | 17:03 |
| pabelanger | okay on node again | 17:03 |
| jeblair | i was also able to ssh | 17:03 |
| pabelanger | okay, lost terminal | 17:03 |
| pabelanger | when I ran dmesg | 17:04 |
| jeblair | both errors were on ze08 | 17:04 |
| fungi | connectivity issues in sto2, or do we know? | 17:04 |
| jeblair | i logged in to the node as zuul user from ze08 | 17:05 |
| pabelanger | Ya, I think we are having networking issues in sto2. Every time I try to read a file, I stop getting SSH traffic from the node | 17:05 |
| fungi | we were getting reports of connectivity issues in ovh-gra1 earlier too, so wondering if there's some transatlantic routing problem today | 17:05 |
| jeblair | gathering facts is like reading a big file | 17:05 |
| pabelanger | ya | 17:05 |
| fungi | though pabelanger's symptom sounds like a pmtud blackhole or something | 17:05 |
| pabelanger | so, no surprise it is failing, I am seeing the same issue | 17:05 |
| jeblair | mordred, pabelanger, jlk: is there a simple "ansible ..." cli i can use to just grab facts from a host? | 17:05 |
| mordred | jeblair: yah - ansible -m setup | 17:06 |
| pabelanger | yup | 17:06 |
| mordred | jeblair: or, 'ansible hostname -m setup' - obvs | 17:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Shift nodepool quota from v2 to v3 https://review.openstack.org/507222 | 17:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Move tripleo-test-cloud-rh1 into nl01 https://review.openstack.org/507275 | 17:06 |
| mordred | rebased those two | 17:06 |
| pabelanger | I'd be okay with dropping sto2 down to 0 for now, and keep working on zuulv3 roll out | 17:07 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-city-cloud show an upwards trend in building nodes for sto2 | 17:08 |
| jeblair | zuul@ze08:/var/lib/zuul/builds/08282a21fa2b45eead4eeeb5069efdc1/ansible$ ANSIBLE_CONFIG=playbook_0/ansible.cfg ansible ubuntu-xenial -m setup | 17:08 |
| jeblair | that actually fails really fast | 17:08 |
| jeblair | zuul@ze08:/var/lib/zuul/builds/08282a21fa2b45eead4eeeb5069efdc1/ansible$ ansible -i inventory.yaml ubuntu-xenial -m setup --private-key /var/lib/zuul/ssh/nodepool_id_rsa -vvv | 17:11 |
| jeblair | that is hanging | 17:11 |
| *** rcernin has joined #openstack-infra | 17:11 | |
| *** panda is now known as panda|off | 17:11 | |
| jeblair | i think the controlpersist is having an impact | 17:12 |
| jeblair | yeah | 17:12 |
| mordred | jeblair: oh - so control persist making the failure take longer | 17:13 |
| jeblair | yep | 17:13 |
| jeblair | so maybe the network problem has tanked the controlpersist connection? | 17:13 |
| mordred | yah - leading to things getting weird subsequently | 17:13 |
| jeblair | exactly | 17:13 |
| pabelanger | I'm still surprised we get exit code: 4 | 17:14 |
| pabelanger | I would have expected exit code: 3 | 17:14 |
| pabelanger | or 2 | 17:14 |
| *** dhajare has joined #openstack-infra | 17:15 | |
| pabelanger | oh, https://github.com/ansible/ansible/issues/19720 could be related to exit code 4 | 17:15 |
| mordred | jeblair: ssh -O check 46.254.11.128 | 17:15 |
| mordred | jeblair: should check the status of the control persist if I'm reading correctly | 17:15 |
| jeblair | ssh -o ControlPath=/home/zuul/.ansible/cp/cacf3fef3b -O check 46.254.11.128 | 17:16 |
| jeblair | Master running (pid=26185) | 17:16 |
| mordred | ok. well, that didn't help us :) | 17:17 |
| *** jamesmcarthur has quit IRC | 17:17 | |
| jeblair | pabelanger: i see the same behavior if i run dmesg | 17:18 |
| jeblair | heck, i can't even cat .ssh/authorized_keys without seeing problems | 17:19 |
| pabelanger | so, I am thinking out handling of exit code 4 is not correct in this case, reading https://github.com/ansible/ansible/issues/19720 I don't think we can assume it is a 'parse error' for yaml | 17:20 |
| pabelanger | in the case of now, zuul should be doing what we do for exit code: 3, return self.RESULT_UNREACHABLE and retry the job | 17:21 |
| *** ykarel|away has quit IRC | 17:21 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Print cloud location information in job header https://review.openstack.org/507631 | 17:21 |
| *** SumitNaiksatam has joined #openstack-infra | 17:21 | |
| jeblair | pabelanger: but how do we detect this case? ideally, in less than 15 minutes? | 17:21 |
| jeblair | (but 15m is still preferable to 1hour) | 17:22 |
| *** Swami has joined #openstack-infra | 17:23 | |
| pabelanger | jeblair: I think we need to still wait for exit code from ansible-playbook, then validate if UNREACHABLE is logged. I'm still looking to see where the 15m timeout is happening, possible an ansible.cfg setting | 17:24 |
| mordred | jeblair: we could run "ansible '*' -m setup" - but with a much shorter watchdog timer | 17:24 |
| *** trown|lunch is now known as trown | 17:24 | |
| *** kjackal_ has joined #openstack-infra | 17:24 | |
| mordred | would also like to know where the 15 minute timeout is coming from thoguh | 17:24 |
| jeblair | could be ssh | 17:25 |
| pabelanger | server side? | 17:25 |
| jeblair | doubt it; i was thinking controlpersist | 17:25 |
| jeblair | mordred: i think that's the best idea so far | 17:26 |
| pabelanger | wouldn't that mean we always gather facts? | 17:27 |
| mordred | pabelanger: we already always gather facts - but we have fact caching set up so it only happens once | 17:28 |
| odyssey4me | do you really always want to gather all facts - it's pretty slow and onerous | 17:28 |
| mordred | pabelanger: now - I'm not 100% sure if running ansible foo -m setup will write to the fact cache or not | 17:28 |
| pabelanger | mordred: well, we do set gather_facts: no for logs.o.o | 17:28 |
| odyssey4me | mordred it does, we use that in OSA | 17:28 |
| pabelanger | so we don't hammer it when we run 1000 jobs | 17:28 |
| jeblair | pabelanger: and i believe mordred is suggesting an explicit hard-coded sanity check on the job nodes in the executor before starting the job | 17:28 |
| mordred | pabelanger: right. that's different and wouldn't be affected | 17:28 |
| mordred | odyssey4me: awesome | 17:28 |
| pabelanger | okay cool | 17:28 |
| mordred | so it's not any EXTRA things we're not already doing | 17:29 |
| odyssey4me | mordred https://github.com/openstack/openstack-ansible/blob/master/scripts/gate-check-commit.sh#L184 | 17:29 |
| clarkb | jeblair: something aimilar is what I wish jenkins had when it had ssh trouble | 17:29 |
| clarkb | a simpl ia tcp + ssh passing traffic check | 17:29 |
| jeblair | mordred: and as far as runtime, we'll then have the facts cached before our first "real" pre-playbook, so we're mostly just shifting when that happens, yeah? | 17:29 |
| pabelanger | mordred: also, we add-host for logs.o.o, so that would be ignored at the start | 17:29 |
| mordred | jeblair: yes, that's right | 17:30 |
| odyssey4me | we also limit the facts gathered, 'cos that cuts time taken down considerably | 17:30 |
| mordred | pabelanger: yes | 17:30 |
| pabelanger | odyssey4me: ya, we've been do gather_subset=!all, seems to be a good balance | 17:30 |
| *** eranrom has joined #openstack-infra | 17:31 | |
| jeblair | clarkb: yeah. nodepool is giving us that now in v2. the idea was that the first pre-playbook would give us the same thing in zuul v3, it just seems that we've hit an edge case where we can't detect the issue. | 17:31 |
| odyssey4me | pabelanger yup, that works if that's enough facts... and if you're not doing bash which doesn't like ! | 17:31 |
| pabelanger | odyssey4me: and onetime cost now that we cache | 17:31 |
| odyssey4me | well, it interprets it | 17:31 |
| mordred | jeblair: do you thing we should implement the setup check now? | 17:32 |
| odyssey4me | are you caching on file, or in zookeeper or something? | 17:32 |
| mordred | odyssey4me: file | 17:32 |
| odyssey4me | file gets super slow when you have a really big inventory, and will actually hang | 17:32 |
| mordred | odyssey4me: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py?h=feature/zuulv3#n1441 | 17:32 |
| mordred | odyssey4me: hopefully we won't have jobs with super big inventories :) | 17:32 |
| *** jpena is now known as jpena|off | 17:32 | |
| odyssey4me | ja, I guess the inventory is only the number of nodes in a job - so you should be good | 17:32 |
| jeblair | mordred: i think we need it soon. we can either do it now or shut-down sto2 and put it on the backlog. | 17:33 |
| mordred | jeblair: I think we should shut down sto2 and put it on the backlog - it might take us a few iterations to get it right and also to verify to our satisfaction that it works | 17:33 |
| mordred | jeblair: and I think it's a thing we should get right and not a thing we should feel rushed about implementing | 17:34 |
| odyssey4me | it might also make sense to up the timeout for the cached facts? | 17:34 |
| jeblair | mordred: ack; adding to etherpad | 17:34 |
| mordred | odyssey4me: that's probably a great idea - especially since we know what job timeouts are :) | 17:34 |
| pabelanger | fact_caching_timeout = 86400 is the current default | 17:35 |
| pabelanger | and we run a fact cache per job | 17:35 |
| mordred | well - our top job timeout is 10800 - so we should be covered there | 17:35 |
| odyssey4me | alright, there should also be an ability to pipeline ssh and also to retry on connection failure IIRC, which may be useful | 17:36 |
| odyssey4me | it has been for our CI jobs | 17:36 |
| mordred | odyssey4me: yah - we have controlpersist set up and all that | 17:36 |
| odyssey4me | some of these may be handy: https://github.com/openstack/openstack-ansible/blob/master/group_vars/all/all.yml#L107-L117 | 17:37 |
| odyssey4me | we found those useful in OSIC | 17:37 |
| *** tesseract has quit IRC | 17:37 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Zuul v3: Stop using citycloud-sto2 https://review.openstack.org/507636 | 17:38 |
| jeblair | i'll update the nodepool quota move patch too | 17:39 |
| mordred | jeblair: cool | 17:39 |
| mordred | jeblair, clarkb, pabelanger: ^^ we don't have tcp keepalive or ServerAlive settings currently - maybe we should consider those when we work on the setup task in the backlog? | 17:39 |
| jeblair | mordred: i'm going to rebase the project-config stack on my sto2 change | 17:40 |
| mordred | jeblair: ++ | 17:40 |
| mordred | odyssey4me: I'm curious about you turning compression off - I'm assuming that was too much cpu spent dealing with compression? | 17:40 |
| pabelanger | mordred: sure, or even seeing about exposing the setting some how to make tuning easier | 17:40 |
| odyssey4me | mordred yep, ultimately it just slowed things down | 17:40 |
| odyssey4me | not really about CPU utilisation - more about connectivity speed and reliability | 17:41 |
| mordred | pabelanger: k. I made a note about that | 17:41 |
| mordred | odyssey4me: oh really? that's interesting | 17:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Revert "Revert "Add the rest of the projects to Zuul v3"" https://review.openstack.org/507564 | 17:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Make project-config gating in zuul v3 https://review.openstack.org/507219 | 17:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Shift nodepool quota from v2 to v3 https://review.openstack.org/507222 | 17:42 |
| odyssey4me | mordred IIRC cloudnull spent around a week testing and tuning things to figure out the optimal bits | 17:42 |
| cloudnull | o/ | 17:42 |
| odyssey4me | WB from holiday cloudnull :) | 17:43 |
| cloudnull | thanks! | 17:43 |
| cloudnull | it was a good time away | 17:43 |
| jeblair | mordred, pabelanger, clarkb: https://review.openstack.org/507636 | 17:43 |
| mordred | jeblair: +2 | 17:44 |
| mordred | cloudnull: welcome back just in time for the zuul v3 happy fun times! | 17:44 |
| cloudnull | woot!~ | 17:44 |
| jeblair | mordred, pabelanger: you both +2d. i'll +3 | 17:44 |
| mordred | jeblair, pabelanger, clarkb: I have added a few more notes to the etherpad from the discussion with odyssey4me | 17:44 |
| mordred | pabelanger: +2 race :) | 17:44 |
| cloudnull | I'm looking forward to that | 17:44 |
| jeblair | mordred: i need to afk for 30m; can you continue pushing changes through up to 507176 ? | 17:45 |
| mordred | cloudnull: enjoy https://review.openstack.org/#/c/507176/ - and the fact that you might get to see that gerrit has a patch-loading-progress-bar | 17:45 |
| mordred | jeblair: yup | 17:45 |
| *** jamesmcarthur has joined #openstack-infra | 17:45 | |
| odyssey4me | mordred one more thing that may be handy: https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/connection/ssh.py#L74-L85 | 17:46 |
| mordred | jeblair: as in - you're fine with going ahead and landing the add-projects-to-main.yaml patch? | 17:46 |
| cloudnull | ++ | 17:46 |
| odyssey4me | anyway, enough of that - lemme get out of all y'all's hair | 17:47 |
| *** lbragstad has quit IRC | 17:47 | |
| jeblair | mordred: yes | 17:47 |
| mordred | jeblair: kk | 17:47 |
| mordred | infra-root: y'all ready to get this show re-started? I'm going to land the add-projects patch so that we're ready to rock-and-roll when jeblair gets back | 17:48 |
| pabelanger | mordred: I am ready | 17:49 |
| mordred | woot | 17:49 |
| * fungi puts on böc's "r.u. ready 2 rock" | 17:50 | |
| pabelanger | yes I am | 17:51 |
| *** ralonsoh has quit IRC | 17:51 | |
| *** lbragstad has joined #openstack-infra | 17:51 | |
| mordred | now it's my favorite part - the part where we wait for patches to land | 17:51 |
| fungi | just like any other day | 17:52 |
| *** ijw has joined #openstack-infra | 17:53 | |
| rm_work | hey, getting some WEIRD build issues with octavia pep8 check gate | 17:54 |
| rm_work | http://logs.openstack.org/75/504175/12/check/gate-octavia-pep8-ubuntu-xenial/f9afa68/console.html#_2017-09-26_05_05_47_130885 | 17:54 |
| rm_work | ./octavia/tests/unit/certificates/manager/test_barbican.py:19:1: F401 'six' imported but unused | 17:54 |
| rm_work | except ... https://review.openstack.org/#/c/504175/12/octavia/tests/unit/certificates/manager/test_barbican.py | 17:54 |
| rm_work | it's like it isn't testing the patch | 17:54 |
| *** ijw has quit IRC | 17:55 | |
| rm_work | like it's testing MASTER instead of the CR | 17:55 |
| rm_work | (we merged changes to our pep8 checks and now they fail | 17:55 |
| rm_work | ) | 17:55 |
| rm_work | err sorry, actually not master, but like ... an older patchset | 17:55 |
| rm_work | or something. i'm really not sure wtf (since master had "six" but also used it, and current code doesn't use it, but also removed the import | 17:56 |
| mordred | rm_work: 2017-09-26 05:01:50.340663 | INFO:zuul.Cloner:Prepared openstack/octavia repo with commit 612b309f98f6af8978218ee5c609be60bc953db2 is the line where it's checking out octavia repo | 17:57 |
| rm_work | i'm really unable to figure out what it could possibly be looking at where that line is there in that position | 17:58 |
| rm_work | ummmmmm | 17:58 |
| *** ijw has joined #openstack-infra | 17:58 | |
| rm_work | ok well that isn't the commit of this patchset | 17:58 |
| *** ijw has joined #openstack-infra | 17:58 | |
| mordred | which is the zuul commit: http://logs.openstack.org/75/504175/12/check/gate-octavia-pep8-ubuntu-xenial/f9afa68/_zuul_ansible/vars.yaml | 17:59 |
| rm_work | uhhh wut | 17:59 |
| mordred | (mostly just walking back through to try to figure out what might be happening) | 17:59 |
| rm_work | yeah huh, i see that, how the heck did that get set | 17:59 |
| *** bh526r has quit IRC | 17:59 | |
| mordred | that's how zuul works :) | 18:00 |
| rm_work | i can't even find that commit hash anywhere | 18:00 |
| mordred | it's merging these: openstack/octavia:master:refs/changes/75/504175/12 | 18:00 |
| mordred | right. it's a temporary speculative hash for the purposes of testing that change | 18:00 |
| rm_work | oh, right. k | 18:00 |
| *** baoli has quit IRC | 18:00 | |
| fungi | rm_work: it's a spontaneously created commit merging your change onto master for the purpose of testing that it's viable with the target branch tip | 18:00 |
| mordred | but all of te things look correct- it should have merged that change with master, then zuul-cloner should have cloned it | 18:01 |
| *** baoli has joined #openstack-infra | 18:01 | |
| mordred | and z-c is reporting that it cloned the ref that zuul think it should have cloned -so that's good at least - | 18:01 |
| mordred | then tox is reporting that it installed octavia: | 18:01 |
| mordred | 2017-09-26 05:05:54.990605 | octavia==1.0.0.0rc2.dev60 # git sha 612b309 | 18:01 |
| mordred | at that same commit | 18:02 |
| rm_work | so that's good also i assume | 18:02 |
| *** Apoorva_ has joined #openstack-infra | 18:02 | |
| rm_work | yeah uhhh ok wtf | 18:03 |
| rm_work | so it's not a zuul problem it's a git problem | 18:03 |
| rm_work | i guess? | 18:03 |
| rm_work | i just cherry-picked that commit onto master | 18:03 |
| rm_work | and it ... does not generate the file that i'd expect | 18:03 |
| *** jkilpatr has quit IRC | 18:03 | |
| mordred | yah - git merging that commit on to master winds up with six in that file | 18:03 |
| *** rbrndt has quit IRC | 18:04 | |
| mordred | it may be worth rebasing at this point | 18:04 |
| rm_work | eugh | 18:04 |
| fungi | if there are repetitive sections, git can sometimes get confused when merging | 18:04 |
| *** jkilpatr has joined #openstack-infra | 18:04 | |
| rm_work | this is the first time i've ever seen this much of an issue | 18:04 |
| mordred | yah - I just did a rebase and wound up with six in that file | 18:04 |
| rm_work | or really ANY issue | 18:04 |
| mordred | so you can at least fix it that way :) | 18:04 |
| rm_work | yeah, ugh i wonder what all it did | 18:04 |
| mordred | rm_work: git merging is perfect and magical ... except when it fails horribly | 18:05 |
| fungi | it is definitely magical, and sometimes magic backfires | 18:05 |
| rm_work | lol literally it just added six magically | 18:05 |
| pabelanger | ya | 18:05 |
| pabelanger | I just reproduced the issue | 18:05 |
| mordred | rm_work: maybe it knows something you don't ... | 18:05 |
| mordred | rm_work: and maybe you really _do_ want six in that file :) | 18:06 |
| rm_work | lol | 18:06 |
| rm_work | well i just "fixed" that | 18:06 |
| pabelanger | git-review -d504175; git merge origin/master; tox -epep8; fails | 18:06 |
| fungi | trust the git. the git is your friend | 18:06 |
| rm_work | i wonder if it added any other random lines to my files | 18:06 |
| *** Apoorva has quit IRC | 18:06 | |
| rm_work | ffff now i'm ALWAYS going to wonder that | 18:06 |
| rm_work | great, thanks, now i don't actually trust git anymore T_T | 18:06 |
| *** pvaneck has joined #openstack-infra | 18:06 | |
| fungi | rm_work: i wonder that just about constantly. a healthy dose of paranoia never hurts | 18:06 |
| fungi | this is one of the reasons we couple revision control with tons of testing and review | 18:07 |
| pabelanger | ++ | 18:07 |
| *** leong has joined #openstack-infra | 18:08 | |
| fungi | and even if a new patchset claims to be a trivial rebase, that doesn't stop me skimming it again | 18:08 |
| *** Apoorva_ has quit IRC | 18:08 | |
| *** srobert_ has joined #openstack-infra | 18:08 | |
| *** Apoorva has joined #openstack-infra | 18:08 | |
| *** hashar has joined #openstack-infra | 18:09 | |
| odyssey4me | oh yeah, mordred - one more thing regarding ssh connections - if you're using delegation much, then be aware of the issue described in https://docs.openstack.org/openstack-ansible/pike/admin/advanced-config.html#ansible-forks | 18:09 |
| odyssey4me | well, delegation and forks | 18:10 |
| openstackgerrit | Merged openstack-infra/project-config master: Zuul v3: Stop using citycloud-sto2 https://review.openstack.org/507636 | 18:10 |
| leong | just a small favor if anyone from infra has time: to remove a maillist: https://review.openstack.org/#/c/482251 | 18:11 |
| *** dave-mcc_ is now known as dave-mccowan | 18:12 | |
| *** srobert has quit IRC | 18:12 | |
| amoralej | we are hitting networking issues today in ovh | 18:13 |
| amoralej | pabelanger, ^ has it been reported? | 18:13 |
| amoralej | example http://logs.openstack.org/55/507555/1/check/gate-puppet-openstack-integration-4-scenario003-tempest-centos-7/37b111d/console.html | 18:15 |
| clarkb | amoralej: yes | 18:15 |
| *** kgiusti has joined #openstack-infra | 18:15 | |
| amoralej | ok | 18:15 |
| clarkb | mordred: so we are just waiting for stuff to merge now? | 18:15 |
| openstackgerrit | Merged openstack-infra/puppet-openafs master: Pull centos packages from project-config subdir https://review.openstack.org/507614 | 18:16 |
| mordred | clarkb: yup. our ffavorite game | 18:17 |
| jeblair | mordred: o/ | 18:18 |
| mordred | jeblair: you have missed literally no things | 18:18 |
| jeblair | sto2 backout merged! | 18:18 |
| mordred | well - yah | 18:18 |
| mordred | thta happend | 18:18 |
| jeblair | centos afs thingy merged! | 18:19 |
| mordred | jeblair: ok. you missed a couple of things | 18:19 |
| Shrews | ooh, back just in time to watch merging | 18:19 |
| jeblair | there are no open changes in the changes section of etherpad! | 18:19 |
| clarkb | leong: we dont typically remove mailing lists instead the list admin can disable new membership, that way the archives stay up | 18:19 |
| clarkb | fungi: ^ that sound right to you? | 18:19 |
| jeblair | looks like 564 is on the cusp of merging | 18:20 |
| mordred | yah - it's been on the cusp for a little bit now ... | 18:20 |
| jeblair | hrm | 18:21 |
| mordred | the job looks finished | 18:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Revert "Add the rest of the projects to Zuul v3"" https://review.openstack.org/507564 | 18:21 |
| jeblair | there it goes :) | 18:21 |
| *** amoralej is now known as amoralej|off | 18:21 | |
| mordred | woot | 18:21 |
| leong | clarkb: ok thanks. | 18:21 |
| jeblair | shall i send out a status notice? | 18:21 |
| mordred | yah | 18:22 |
| mordred | I'm going to kick pippet to accelerate v3 picking that up | 18:22 |
| jeblair | status notice The infra team is continuing work to bring Zuul v3 online; expect service disruptions and please see https://docs.openstack.org/infra/manual/zuulv3.html for more information. | 18:22 |
| mordred | it's almost like we know what we're doing with this step | 18:22 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Zun] Remove the tempest plugin testing job https://review.openstack.org/506886 | 18:22 |
| jeblair | how's that ^ | 18:22 |
| mordred | jeblair: ++ wfm | 18:22 |
| jeblair | #status notice The infra team is continuing work to bring Zuul v3 online; expect service disruptions and please see https://docs.openstack.org/infra/manual/zuulv3.html for more information. | 18:23 |
| openstackstatus | jeblair: sending notice | 18:23 |
| *** dprince has quit IRC | 18:23 | |
| pabelanger | here we go | 18:23 |
| mordred | jeblair: v3 should be doing the config dance now | 18:23 |
| mordred | it's nice starting this at 18:23 - we have so many hours in which we can do things! | 18:24 |
| *** xyang1 has joined #openstack-infra | 18:24 | |
| -openstackstatus- NOTICE: The infra team is continuing work to bring Zuul v3 online; expect service disruptions and please see https://docs.openstack.org/infra/manual/zuulv3.html for more information. | 18:24 | |
| andreaf | good luck with the migration! | 18:24 |
| mordred | andreaf: thanks!!! | 18:24 |
| jeblair | 2017-09-26 18:22:58,875 DEBUG zuul.Scheduler: Performing reconfiguration | 18:24 |
| *** electrofelix has quit IRC | 18:25 | |
| fungi | clarkb: leong hah, i was just in the process of adding that comment to the proposed change. cleaning it up from the puppet manifest seems fine, but they | 18:25 |
| openstackstatus | jeblair: finished sending notice | 18:26 |
| fungi | they'll also want to set options to reject new posts and hide the list from the mailman index | 18:26 |
| andreaf | jeblair: it looks like zuul does retry of post plays only in case post failures, right? so we need post jobs to be idempotent always | 18:26 |
| leong | fungi: i just abandon that change | 18:27 |
| leong | fungi: clarkb: what is your suggestion? | 18:27 |
| fungi | leong: yep, i discovered that when it refused to accept my +2 w/ comments a moment ago | 18:27 |
| *** slaweq_ has joined #openstack-infra | 18:27 | |
| *** xyang1 has quit IRC | 18:27 | |
| mordred | andreaf: post playbooks shouldn't get retried- but all of the post playbooks should get run even if one of the previous ones fails | 18:27 |
| leong | btw, also the admin email is no longer valid as carol has left the company | 18:27 |
| mordred | andreaf: so a post playbook shouldn't assume success on the part of other post playbooks | 18:28 |
| jeblair | andreaf: it shouldn't retry a post playbook; it runs all the post playbooks as deep as it has run pre-playbooks. if any one fails, it marks the build as POST_FAILURE, but continues to run subsequent post playbooks. | 18:28 |
| mordred | yah. that | 18:28 |
| leong | fungi: or should i "restore" | 18:28 |
| *** e0ne has joined #openstack-infra | 18:28 | |
| fungi | leong: set the necessary options through the listadmin interface to hide it from the main mailman index, disallow new subscriptions, reject posts and remove all existing subscribers. removing it from the puppet manifest like you did in your proposed change seems fine as additional cleanup, but is not strictly needed | 18:28 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall https://review.openstack.org/504788 | 18:28 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge role https://review.openstack.org/504789 | 18:29 |
| leong | i am not the admin to that maillist now... | 18:29 |
| fungi | leong: i can add you (or someone) as a new listadmin to be able to shut down the list | 18:29 |
| EmilienM | FYI stderr: 'fatal: unable to access 'http://zm01.openstack.org/p/openstack/python-tripleoclient/': Failed to connect to zm01.openstack.org port 80: Connection timed out' | 18:29 |
| EmilienM | just in case | 18:29 |
| jeblair | andreaf: for example, in https://docs.openstack.org/infra/manual/zuulv3.html#job-inheritance if tox pre-run fails, it won't run any tox-py27 playbooks, and it won't run tox post-run, but will run unittests and base post-run. and it will run both of those regardless of any other failures. | 18:30 |
| pabelanger | EmilienM: ovh? | 18:30 |
| jeblair | 2017-09-26 18:28:04,161 DEBUG zuul.Scheduler: Reconfiguration complete | 18:30 |
| pabelanger | Yay | 18:30 |
| EmilienM | pabelanger: yes ubuntu-xenial-ovh-gra1-11147230 | 18:30 |
| mordred | woot! | 18:30 |
| * mordred rechecks ozj patch | 18:30 | |
| pabelanger | EmilienM: expected, we are having networking issues today | 18:30 |
| EmilienM | perfect | 18:30 |
| leong | fungi: that will be great | 18:30 |
| andreaf | mordred, jeblair: that doesn't seem to be what happens in my case - I get RETRY_LIMIT when my post change fails https://review.openstack.org/#/c/506835/ and the rest of post is not executed since I don't get logs | 18:30 |
| andreaf | mordred, jeblair: I could see in the live log that the failure happens during my post-tempest job | 18:31 |
| Shrews | ooh, lots of requests in nodepool | 18:31 |
| jeblair | andreaf: i'll check the logs for that | 18:31 |
| pabelanger | mordred: ansible-playbook --syntax-check is likely going to take some time on the legacy playbooks. Not sure we want to run it long term on legacy things | 18:31 |
| pabelanger | or we setup parallel | 18:32 |
| andreaf | jeblair: heh there are no logs - at least according to the gerrit comment | 18:32 |
| mordred | pabelanger: I agree - in fact, I think running it exactly this one time seems like the right amount of times to run it | 18:32 |
| fungi | leong: you want me to set it to your gmail address or a different one? | 18:32 |
| leong | do you have my intel address? | 18:32 |
| jeblair | andreaf: i mean on the executor; we can sometimes learn a bit about catastrophic failures like that from zuul logs | 18:32 |
| mordred | this might wind up being the longest linter job ever :) | 18:33 |
| fungi | leong: probably, just a sec and i'll double-check | 18:33 |
| *** Apoorva has quit IRC | 18:33 | |
| leong | fungi: thanks | 18:33 |
| pabelanger | indeed | 18:33 |
| andreaf | jeblair: ah ok | 18:33 |
| *** Apoorva has joined #openstack-infra | 18:33 | |
| jeblair | andreaf: http://paste.openstack.org/show/621965/ | 18:33 |
| pabelanger | we are linting now | 18:33 |
| jeblair | andreaf: i think there may be some zuul bugs in there. we should check with mordred and dmsimard on that when we have a break in the migration work | 18:33 |
| mordred | yah. wow. this isn't going to be quick | 18:34 |
| pabelanger | my testing with parallel got it down to about 8mins on my laptop | 18:34 |
| pabelanger | with 8 instances of ansible-playbook running | 18:34 |
| pabelanger | our default timeout on tox-linters is 30mins, so I'm now wondering if we'll not finish in time | 18:35 |
| mordred | pabelanger: if we have to iterate on this one again, we probably want to add that | 18:35 |
| jeblair | andreaf, mordred, dmsimard: in short, it looks like zuul_json may be having issues with something... | 18:35 |
| jeblair | dmsimard: did your include_role fixes get merged? | 18:35 |
| mordred | jeblair: well - I think the real issue is: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: IOError: [Errno 13] Permission denied: '/opt/stack/tempest/etc/tempest_conf.txt.gz'" | 18:35 |
| jeblair | andreaf: are you using include_role? | 18:35 |
| andreaf | jeblair: I am indeed | 18:35 |
| dmsimard | jeblair: nope, I mostly disappeared between last friday and today | 18:35 |
| dmsimard | jeblair: so I didn't get the chance to pair up with a friend :( | 18:36 |
| andreaf | jeblair: it works fine when I run it locally | 18:36 |
| dmsimard | andreaf: there is a known issue that we are trying to resolve | 18:36 |
| fungi | leong: the one listed at https://wiki.openstack.org/wiki/Enterprise_Working_Group is still correct for you? | 18:36 |
| mordred | andreaf: yah - there'sa bug in our log processing related to include_role | 18:36 |
| dmsimard | andreaf: https://review.openstack.org/#/c/504238/ | 18:36 |
| andreaf | mordred: yes that's the real issue in my code, but zuul should handle that a bit more nicely :) | 18:36 |
| *** e0ne has quit IRC | 18:36 | |
| jeblair | mordred: that's an error, but i don't think that's what prevented the log uploads from happening; this is looking a lot like the zuul_json thing | 18:37 |
| jeblair | so hopefully we can get it squared away asap | 18:37 |
| mordred | andreaf: indeed - I believe the include_role issue is the thing that's causing our attempt to tell you about it to fail | 18:37 |
| mordred | jeblair: yah | 18:37 |
| *** srobert_ has quit IRC | 18:37 | |
| mordred | dmsimard: the balls all look green on that patch - were you still working on getting the failure to reproduce in test caes? or is that ready for review? | 18:37 |
| dmsimard | andreaf: making very 'core' bits more resilient to failure is a continuous and iterative process that we're working on | 18:37 |
| dmsimard | mordred: the zuul_json patch ? nope, the balls are not green, see the finger url in the last job run | 18:38 |
| clarkb | ok beach tether is currently working | 18:38 |
| mordred | dmsimard: oh ! yes, you are totally right | 18:38 |
| andreaf | mordred, jeblair, dmsimard: it looks like I'm doing good testing of zuul v3 ;) | 18:38 |
| dmsimard | mordred: I need someone who has access to executor logs to help because I am unable to reproduce the issue using your tools/test-logs.sh script | 18:38 |
| mordred | andreaf: yes indeed! | 18:39 |
| dmsimard | I've been meaning to stand up a local zuul v3 instance but haven't got around to it yet | 18:39 |
| mordred | pabelanger: how hard would it be to put your parallel stuff into the tox.ini? | 18:39 |
| mordred | pabelanger: cause I agree, I doubt we'll finish within 30 minutes - and we get to run the linters a second time once we approve | 18:40 |
| fungi | clarkb: i vote we keep today's meeting short at least | 18:40 |
| clarkb | indeed | 18:40 |
| fungi | (and would not be against cancelling due to inclement zuul) | 18:40 |
| clarkb | ya maybe take the tempurature in 10 minutes then decide if we want a meeting or focus on zuul | 18:41 |
| mordred | I mean - right now focusing on zuul means "watch job run and wait for it to possibly timeout" | 18:41 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall https://review.openstack.org/504788 | 18:42 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge role https://review.openstack.org/504789 | 18:42 |
| andreaf | dmsimard: do you think it makes sense for me to try and rebase my patch on top of yours and see what happens? will zuul be able to actually test a change to itself and to a job live? | 18:43 |
| clarkb | mordred: this is the ansible lint syntax check of doom ya? | 18:44 |
| mordred | yah | 18:44 |
| jeblair | andreaf: no, the component with the issue is very low level and needs the installation of zuul we're running updated in order to change | 18:44 |
| leong | fungi: yes. | 18:44 |
| andreaf | jeblair: right, I thought so | 18:44 |
| dmsimard | andreaf: that wouldn't help because my patch doesn't completely fix the issue. It's missing something and we can only tell what it is by looking at the logs directly on the zuul node. | 18:46 |
| clarkb | mordred: does it make sense to just have someone run it locally on a fast machine and if that goes green call it good? | 18:46 |
| dmsimard | andreaf: we are able to reproduce the issue in-gate with a "special" integration test job | 18:46 |
| clarkb | at this point probably doesn't make sense to stop the existing job though | 18:46 |
| *** slaweq_ has quit IRC | 18:46 | |
| mordred | clarkb, pabelanger: http://paste.openstack.org/show/621967/ how does that look? | 18:47 |
| clarkb | mordred: xargs can do it too | 18:47 |
| *** slaweq_ has joined #openstack-infra | 18:47 | |
| mordred | clarkb: well - considering we hav eto run the check again in the gate - it may be worth considering restarting with a faster version of the job, or having someone check it out of band as you suggested | 18:47 |
| clarkb | -P | 18:47 |
| mordred | clarkb: xargs does parallel operation? | 18:47 |
| mordred | ah-TIL | 18:47 |
| clarkb | mordred: yup | 18:47 |
| pabelanger | didn't know that | 18:47 |
| mordred | so bash -c "cd playbooks; find . -type f -regex '.*.y[a]?ml' -print0 | xargs -t -P -n1 -0 ansible-playbook --syntax-check -i {toxinidir}/tests/inventory > /dev/null" | 18:48 |
| fungi | leong: it should hopefully have e-mailed you the new password i reset for that ml just now | 18:48 |
| clarkb | mordred: you have to pass the number of processes to -P | 18:48 |
| mordred | clarkb: anywho - do you think we should restart with -P ? or just have someone do local testing of playbook syntax and restart this with legacy playbooks excluded? | 18:49 |
| fungi | leong: use that at http://lists.openstack.org/cgi-bin/mailman/admin/enterprise-wg/ and you can get to all the list settings | 18:49 |
| clarkb | -P $(nproc) | 18:49 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Add post step to Tempest base job https://review.openstack.org/506835 | 18:49 |
| clarkb | mordred: I'm thinking might be quicket to have someone run with -P $(nproc) locally and verify the result | 18:49 |
| clarkb | I'm guessing its predominantly cpu bound and local cpus likely to go pretty quick | 18:49 |
| mordred | clarkb: kk. I don't have a fast computer locally available - anybody else got a good one? | 18:50 |
| clarkb | (assuming we have decent local cpus I want to say my laptop cpu is roughly the speed of a gate VM) | 18:50 |
| pabelanger | I have local testing going too with parallel, but -P for xargs is +1 | 18:50 |
| andreaf | jeblair: the permission issue is because I was trying to compress tempest files directly in their folder but the user running post does not have write access there | 18:50 |
| openstackgerrit | Brian Haley proposed openstack-infra/project-config master: Make neutron-multinode job the default https://review.openstack.org/483600 | 18:51 |
| *** slaweq_ has quit IRC | 18:52 | |
| *** gouthamr has joined #openstack-infra | 18:52 | |
| *** dprince has joined #openstack-infra | 18:53 | |
| pabelanger | clarkb: mordred: local testing shows now errors | 18:53 |
| pabelanger | but also testing -P ($nproc) now | 18:53 |
| pabelanger | much faster :) | 18:53 |
| pabelanger | mordred: clarkb: http://paste.openstack.org/show/621969/ | 18:54 |
| pabelanger | we should use that next patch | 18:54 |
| mordred | pabelanger: ++ | 18:56 |
| *** gouthamr has quit IRC | 18:57 | |
| leong | fungi: got it. thanks! | 18:57 |
| mordred | pabelanger: so - you tested locally with that patch and it did not show ansible-playbook syntax errors? | 18:57 |
| *** baoli has quit IRC | 18:57 | |
| mordred | pabelanger: how long did it take to run? | 18:57 |
| pabelanger | mordred: ya, I used parallel first, things worked. Just waiting for 2nd pass using xargs | 18:57 |
| mordred | kk | 18:57 |
| pabelanger | mordred: ~6mins to run first time | 18:58 |
| clarkb | ok we are tminus one minute or so, do we want to bother with a meeting? | 18:58 |
| clarkb | (I'm good to run it if we do want one) | 18:58 |
| *** rbrndt has joined #openstack-infra | 18:58 | |
| clarkb | I do think it will be relatively quick too since the thing to tlak about is what we are deploying | 18:58 |
| mordred | I don't have much to report other than v3 going | 18:59 |
| mordred | so happy to do meeting - or to skip meeting | 18:59 |
| pabelanger | same | 18:59 |
| *** baoli has joined #openstack-infra | 18:59 | |
| fungi | same here. if we do have the meeting, maybe acknowledge that people are probably paying more attention to the cut-over | 18:59 |
| pabelanger | linters: commands succeeded | 19:00 |
| clarkb | I think I'll start it and note we are busy with zuul and go straight ot open discussion if anyone has anything | 19:00 |
| clarkb | rathe rthan a normal structured meeting | 19:00 |
| * clarkb does this | 19:00 | |
| mordred | pabelanger: so you have run it twice locally in the time we're still waiting on the first iteration to run in the gate | 19:00 |
| pabelanger | yup | 19:00 |
| pabelanger | running again with time | 19:00 |
| mordred | pabelanger: I vote you amend that commit with the -P $(nproc) - because even if we're almost done with the current run - we still have the gate run | 19:00 |
| fungi | clarkb: thanks, wfm | 19:01 |
| pabelanger | mordred: sure | 19:01 |
| mordred | clarkb, fungi, jeblair: ^^ y'all have opinion on that ? | 19:01 |
| *** ianw|pto is now known as ianw | 19:01 | |
| clarkb | mordred: sounds reasonable | 19:02 |
| clarkb | but don't we also have t ofix the errors pabelanger ran into? | 19:02 |
| clarkb | or did I misparse? | 19:02 |
| fungi | yeah, and if we're really concerned about duration for running twice we could enqueue to the gate once pushed? | 19:02 |
| mordred | clarkb: pabelanger did not run in to any issues | 19:02 |
| pabelanger | clarkb: no, I don't have errors locally | 19:02 |
| clarkb | ah ok I misparsed then | 19:02 |
| pabelanger | push up? | 19:02 |
| clarkb | fungi: ++ | 19:02 |
| *** Goneri has quit IRC | 19:03 | |
| jeblair | ++ | 19:03 |
| *** ijw has quit IRC | 19:03 | |
| mordred | ah - it just passed | 19:04 |
| pabelanger | kk | 19:04 |
| mordred | nope. it just finished | 19:04 |
| mordred | and timed out | 19:04 |
| mordred | pabelanger: so yes - please push up :) | 19:04 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 19:04 |
| pabelanger | done | 19:04 |
| mordred | woot | 19:04 |
| mordred | finger crossed the gate node is at least as fast as pabelanger's computer | 19:05 |
| *** hashar has quit IRC | 19:06 | |
| pabelanger | real 6m52.422s | 19:07 |
| pabelanger | time to beat | 19:07 |
| mordred | Shrews, pabelanger: I'm getting no log from http://zuulv3.openstack.org/static/stream.html?uuid=2042402c144c44edaf21d52540ff54c0&logfile=console.log | 19:07 |
| pabelanger | same | 19:07 |
| *** dhajare has quit IRC | 19:08 | |
| mordred | socket.gaierror: [Errno -2] Name or service not known | 19:08 |
| mordred | have we had another hostname issue? | 19:08 |
| Shrews | ummm | 19:08 |
| *** jaosorior has quit IRC | 19:09 | |
| jeblair | mordred: where's that error? | 19:09 |
| pabelanger | where did you see that? | 19:09 |
| mordred | jeblair: web-debug.log on zuulv3.o.o | 19:09 |
| jeblair | 2017-09-26 19:08:53,352 DEBUG zuul.web.LogStreamingHandler: Connecting to finger server ze01:79 | 19:09 |
| jeblair | the replacement ze01 does not know its hostname appropriately | 19:09 |
| mordred | I just checked and hostname --fqdn reports the rightthing | 19:10 |
| mordred | but hostname does not | 19:10 |
| jeblair | yeah, it's the only executor without a fqdn for /etc/hostname | 19:10 |
| mordred | I agree | 19:11 |
| mordred | we just built it yesterday - so that makes me think we have a bug in launch-node perhaps? | 19:11 |
| fungi | that's... a strange change in behavior for sure | 19:11 |
| jeblair | i feel like it's always changing :( | 19:11 |
| pabelanger | 2017-09-25 20:05:11,942 - util.py[DEBUG]: Running command ['hostname', 'ze01'] with allowed return codes [0] (shell=False, capture=True) | 19:12 |
| pabelanger | cloud-init again | 19:12 |
| mordred | UGH | 19:12 |
| Shrews | geez, clouds are hard | 19:12 |
| mordred | ok.. seriously. as soon as zuulv3 has settled I would like to go on a vendetta march and erradicate cloud-init from our lives as it brings nothing but sorry and misery | 19:12 |
| mordred | of course, that's hard and there are more important things ... so I'll probably do other things instead | 19:13 |
| fungi | yeah, the difference seems to be in /etc/hostname having only the shortname on ze01 vs an fqdn on the other 7 | 19:13 |
| mordred | yah | 19:13 |
| fungi | oh, jeblair already found that | 19:14 |
| pabelanger | I guess /etc/cloud/cloud-init.disabled didn't work | 19:14 |
| jeblair | launch-node sets the hostname using an ansible playbook | 19:14 |
| fungi | pabelanger: or didn't kick in soon enough? | 19:14 |
| jeblair | anyway, what do we want to do? manually fix and restart? | 19:16 |
| jeblair | i added this to the non-blocking bugs section | 19:16 |
| fungi | yeah, i can edit /etc/hostname by hand and reboot the server... any safety precautions around running jobs there? | 19:16 |
| mordred | well - let's wait until the linters job finished | 19:17 |
| mordred | oh! it finished | 19:17 |
| jeblair | mordred: it's not running on v3? | 19:17 |
| jeblair | i need to get food | 19:17 |
| mordred | jeblair: it's totally running on v3 - but it's finished | 19:17 |
| mordred | (this is the tox-linters job for openstack-zuul-jobs) | 19:17 |
| pabelanger | success in 8m 25s | 19:17 |
| jeblair | well it was running :) | 19:17 |
| jeblair | but it is not running now :) | 19:18 |
| mordred | :) | 19:18 |
| fungi | i don't see any ansible processes on ze01 | 19:18 |
| jeblair | i'm fine with manually fixing and rebooting ze01 | 19:18 |
| mordred | fungi: you can just do service zuul-executor stop | 19:18 |
| jeblair | i'll be back in maybe 20m | 19:18 |
| mordred | fungi: to stop the executor | 19:18 |
| mordred | jeblair: kk | 19:18 |
| pabelanger | if we reboot, won't cloud-init reset hostname? | 19:18 |
| *** xarses has quit IRC | 19:18 | |
| mordred | I am, of course, fine with deleting cloud-init from the servers | 19:19 |
| fungi | mordred: sure, and done now. was more pointing out that there weren't running jobs on it (i don't think) | 19:19 |
| mordred | fungi: ++ | 19:19 |
| mordred | fungi: I agree with your assessment | 19:19 |
| pabelanger | or even disabling cloud-init services | 19:19 |
| mordred | infra-root: https://review.openstack.org/#/c/507176/ is ready for +2/+3 | 19:20 |
| pabelanger | but ya, after zuulv3 I'm totally working on DIB control plane :D | 19:20 |
| fungi | should we be uninstalling cloud-init packages? | 19:20 |
| mordred | fungi: I tested doing that on a server the other day and it worked fine | 19:20 |
| fungi | it seems to be installed on the other executors | 19:20 |
| fungi | and they aren't exhibiting this behavior | 19:20 |
| fungi | which leads me to suspect it won't revert on ze01 once fixed either | 19:21 |
| Shrews | mordred: there's one +2 | 19:21 |
| fungi | i'm testing that theory now, fwiw | 19:21 |
| mordred | fungi: good test | 19:21 |
| fungi | hostname corrected, server rebooted, cloud-init not uninstalled | 19:21 |
| *** srobert has joined #openstack-infra | 19:22 | |
| *** xarses has joined #openstack-infra | 19:22 | |
| fungi | hostname reports correct fqdn now | 19:22 |
| *** srobert has quit IRC | 19:22 | |
| fungi | so we possibly have a race of some sort in our launch tooling | 19:22 |
| mordred | k. that'll be good to sort out | 19:22 |
| *** hashar has joined #openstack-infra | 19:23 | |
| * mordred believes in pabelanger and getting DIB control plane sorted :) | 19:23 | |
| *** srobert has joined #openstack-infra | 19:23 | |
| fungi | but anyway, should be safe to proceed. i've started zuul-executor again | 19:23 |
| mordred | woot | 19:23 |
| fungi | er, except it isn't running | 19:23 |
| mordred | fungi: how you did start it? | 19:24 |
| fungi | sudo service zuul-executor start | 19:25 |
| fungi | there's no stale pid file | 19:25 |
| *** andreww has joined #openstack-infra | 19:25 | |
| fungi | no new entries in its debug log | 19:25 |
| mordred | fungi: that all seems correct to me | 19:25 |
| fungi | but no process in the process table | 19:25 |
| mordred | of course, sometimes systemd gets confused- try systemctl stop zuul-executor | 19:25 |
| *** sree has joined #openstack-infra | 19:25 | |
| mordred | then starting it again | 19:25 |
| fungi | no dice | 19:26 |
| mordred | clarkb: you wanna pull the trigger on https://review.openstack.org/#/c/507176/ ? | 19:26 |
| pabelanger | service zuul-executor status | 19:26 |
| pabelanger | systemd things it is running | 19:26 |
| *** xarses has quit IRC | 19:26 | |
| openstackgerrit | Nick Stones-Havas proposed openstack/python-jenkins master: QuietDown requires a POST request https://review.openstack.org/507656 | 19:27 |
| fungi | could there be something about the hostname changing which is causing it to silently bomb? | 19:27 |
| mordred | fungi: mind if I poke for a sec? | 19:28 |
| clarkb | mordred: we are ready for +W? | 19:28 |
| mordred | clarkb: yup! | 19:28 |
| clarkb | ok here goes | 19:28 |
| pabelanger | Yay | 19:28 |
| *** ijw has joined #openstack-infra | 19:28 | |
| fungi | mordred: please poke, my next step was resorting to starting in fireground | 19:28 |
| fungi | foreground | 19:28 |
| fungi | though the fireground might work better | 19:29 |
| clarkb | not in oregon please | 19:29 |
| fungi | touché | 19:30 |
| mordred | it's filesystem | 19:30 |
| mordred | the /var/lib/zuul volume is mounted on /opt | 19:30 |
| *** sree has quit IRC | 19:30 | |
| *** slaweq_ has joined #openstack-infra | 19:31 | |
| fungi | not that i can tell... did you fix it already? | 19:31 |
| mordred | I don't fully know how to consistently find the traceback - but I happened to see http://paste.openstack.org/show/621975/ when I ran service zuul-executor status | 19:32 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up connectivity with default OVS bridge https://review.openstack.org/504554 | 19:32 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Streamline multi-node-known-hosts https://review.openstack.org/504629 | 19:32 |
| mordred | I ahve fixed it - and have also fixed the /etc/fstab file | 19:32 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Append /sbin and /usr/sbin to $PATH for 'ip' commands https://review.openstack.org/504743 | 19:32 |
| mordred | we should fix that file on all of the executors | 19:32 |
| *** ijw has quit IRC | 19:33 | |
| *** chenying has quit IRC | 19:33 | |
| pabelanger | Oh | 19:33 |
| pabelanger | that would explain it | 19:33 |
| mordred | ansible 'ze0*' -m shell -a "sed -i 's/\/opt/\/var\/lib\/zuul/' /etc/fstab" | 19:34 |
| mordred | I have done that - so fstab should be correct now | 19:34 |
| openstackgerrit | melanie witt proposed openstack-infra/devstack-gate master: WIP Add mysqladmin -v extended-status processlist https://review.openstack.org/507626 | 19:34 |
| *** chenying has joined #openstack-infra | 19:34 | |
| *** rcernin has quit IRC | 19:34 | |
| fungi | yeah, i didn't think to go manually set up filesystems after launching | 19:35 |
| fungi | though i wonder why it was working previously | 19:35 |
| mordred | fungi: we had done the filesystem /opt to /var/lib/zuul swap after booting | 19:35 |
| *** ijw has joined #openstack-infra | 19:35 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Migrate jobs from zuulv2 to zuulv3 https://review.openstack.org/507176 | 19:35 |
| mordred | fungi: we just hadn't persisted that into /etc/fstab | 19:35 |
| pabelanger | \o/ | 19:35 |
| mordred | zomg | 19:36 |
| Shrews | and merged. let's all take a week off starting.... now! | 19:36 |
| mordred | I have rechecked the project-config change to make project-config gate on v3 | 19:36 |
| pabelanger | next up 507219 | 19:36 |
| fungi | mordred: thanks, that 'splains it | 19:36 |
| *** chenying_ has joined #openstack-infra | 19:37 | |
| Shrews | oh right. still have that one | 19:37 |
| fungi | mordred: should i go ahead and start the service on it now? still isn't running | 19:37 |
| jeblair | back | 19:37 |
| mordred | Shrews, tobiash, tristanC: http://paste.openstack.org/show/621978/ is the traceback that happened with ze01 was misconfigured. although it shouldn't happen again - maybe we should trap that error and send something to the websocket console that says like "couldn't connect to ze01" or something | 19:38 |
| mordred | fungi: oh - yah - safe to start now | 19:38 |
| fungi | started | 19:38 |
| fungi | and running correctly this time, it seems | 19:38 |
| mordred | infra-root: ok - journalctl -t zuul-executor would have shown the traceback error (and still does) | 19:39 |
| mordred | welcome back jeblair | 19:39 |
| fungi | but not systemctl status | 19:39 |
| fungi | odd | 19:39 |
| *** leong has quit IRC | 19:39 | |
| *** chenying has quit IRC | 19:40 | |
| *** ijw has quit IRC | 19:40 | |
| *** d0ugal has joined #openstack-infra | 19:41 | |
| *** d0ugal has quit IRC | 19:41 | |
| *** d0ugal has joined #openstack-infra | 19:41 | |
| *** armax has joined #openstack-infra | 19:43 | |
| *** Apoorva has quit IRC | 19:44 | |
| *** jamesmcarthur has quit IRC | 19:46 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Return stream log message on finger client failure https://review.openstack.org/507663 | 19:48 |
| Shrews | mordred: ^^^ | 19:49 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Return stream log message on finger client failure https://review.openstack.org/507663 | 19:49 |
| Shrews | pep8 fail | 19:49 |
| *** jkilpatr has quit IRC | 19:50 | |
| mordred | Shrews: does % e work - I thought you had to do str(e) | 19:50 |
| mordred | Shrews: neat! it does - TIL | 19:50 |
| Shrews | mordred: yeah, the %s forces the conversion | 19:50 |
| mordred | gotcha | 19:51 |
| fungi | gets coerced to a text type by the formatting | 19:51 |
| fungi | though sometimes you want more direct control over what it's coerced to, it usually "just works" for most data types | 19:51 |
| * mordred is enjoying watching the job running that's checked the jjb output | 19:52 | |
| jeblair | ugh, we missed a "dump all the file content into the log" debug line | 19:52 |
| pabelanger | https://review.openstack.org/507219/ +1 from jenkins | 19:53 |
| *** jamesmcarthur has joined #openstack-infra | 19:53 | |
| mordred | woot! | 19:53 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 19:53 |
| mordred | jeblair: you wanna do the honors on https://review.openstack.org/#/c/507219/ ? | 19:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't output cat file contents during reconfig https://review.openstack.org/507665 | 19:54 |
| jeblair | mordred: gimme a sec, i'm trying to determine zuul's current state which is difficult with all that output in the logs | 19:54 |
| mordred | jeblair: kk | 19:55 |
| *** jamesmcarthur has quit IRC | 19:57 | |
| jeblair | not bad | 19:58 |
| jeblair | zuul took 12 seconds to reconfigure after the project-config change with all the content landed | 19:58 |
| *** jamesmcarthur has joined #openstack-infra | 19:58 | |
| fungi | wow! | 19:58 |
| jeblair | that *includes* the rpc call to fetch the project-config data | 19:58 |
| jlk | hells ya | 19:58 |
| clarkb | wow not bad | 19:59 |
| jeblair | and, um, that also includes logging it in its entirety to the debug log. :| | 19:59 |
| fungi | apparently | 19:59 |
| clarkb | all 200k lines? | 20:00 |
| jlk | lines are cheap! | 20:00 |
| jeblair | clarkb: it's efficient! it logged it in a single line! | 20:00 |
| *** baoli has quit IRC | 20:00 | |
| Shrews | no one should be forced to wait 12 whole seconds | 20:00 |
| clarkb | nice | 20:00 |
| mnaser | hi folks | 20:00 |
| jeblair | mordred: +3 507219 | 20:01 |
| * mnaser just got to ottawa for the openstack meetup | 20:01 | |
| jeblair | mnaser: wave to pabelanger for me! | 20:01 |
| pabelanger | nice, I'm making my way over in about 50mins | 20:01 |
| mnaser | came to learn about life of an upstreamer :D | 20:01 |
| pabelanger | going to stream zuulv3 roll out for talk :D | 20:02 |
| mordred | pabelanger: :) | 20:02 |
| fungi | that's be worth seeing! | 20:02 |
| fungi | oh, wait... | 20:02 |
| jeblair | mnaser: wow that sounds like some kind of post-apocalyptic gang | 20:02 |
| mordred | pabelanger: sounds like you'll be in a position to be the first to present zuulv3 as a live thing | 20:02 |
| jeblair | look at all the red! | 20:02 |
| pabelanger | mordred: no pressure | 20:02 |
| mnaser | i have an hour or so before this starts | 20:03 |
| mnaser | so um i can try to unbreak puppet | 20:03 |
| fungi | "and here's the new version of the upstream ci system; it's been live for ... *looks at watch* 20 minutes" | 20:03 |
| mordred | mnaser: dude. the puppet jobs are just going to work | 20:03 |
| mnaser | mordred lets take bets :D | 20:03 |
| mordred | it's odyssey4me's ansible jobs that I'm worried about :) | 20:04 |
| *** baoli has joined #openstack-infra | 20:04 | |
| *** pcaruana has quit IRC | 20:05 | |
| jlk | those crazy people wanting to run Ansible things in CI | 20:07 |
| Shrews | i can't wait to tell folks that we've *finally* replaced jenkins with hudson | 20:08 |
| jlk | ಠ_ಠ| 20:08 |
| mordred | infra-root: so - once this patch lands, I think we should run kick.sh on zuul.o.o since there is a layout removal and we don't really want double-gating lurking in the weeds | 20:08 |
| fungi | sounds sane | 20:09 |
| jeblair | ++ | 20:09 |
| pabelanger | ++ | 20:09 |
| jeblair | eta 3min | 20:09 |
| *** dave-mcc_ has joined #openstack-infra | 20:09 | |
| pabelanger | when we shutdown zuulv2, are we going to abort running jobs? To release nodes back to nodepool.o.o? | 20:10 |
| openstackgerrit | Merged openstack-infra/project-config master: Make project-config gating in zuul v3 https://review.openstack.org/507219 | 20:11 |
| prometheanfire | no way to drain? | 20:11 |
| mordred | \o/ | 20:11 |
| pabelanger | prometheanfire: ya but takes upwards of 3 hours (with tripleo jobs) | 20:11 |
| * mordred runs kick | 20:11 | |
| prometheanfire | heh, true... | 20:11 |
| *** dave-mccowan has quit IRC | 20:12 | |
| jeblair | pabelanger: won't shutting down zuul v2 scheduler stop all running jobs? | 20:12 |
| pabelanger | jeblair: I cannot remember to be honest | 20:12 |
| prometheanfire | just kill (imo), gertty tag all and recheck is easy | 20:12 |
| mordred | ok. zuul v2 kick has run | 20:13 |
| mordred | so the layout change should be applied | 20:13 |
| mordred | rechecking the nodepool patch to make sure | 20:13 |
| fungi | though i suppose the alternative question is whether we could export the check/gate pipelines and then reenqueue on zuulv3 | 20:13 |
| *** rlandy is now known as rlandy|afk | 20:14 | |
| pabelanger | post jobs for project-config running now :) | 20:14 |
| pabelanger | http://zuulv3.openstack.org/ | 20:14 |
| jeblair | fungi: i'll do the export, though we may need some translation to re-enqueue | 20:14 |
| mordred | and check jobs | 20:14 |
| fungi | should we do a #status alert (or notice)? | 20:14 |
| pabelanger | ya | 20:14 |
| fungi | i guess there was the one a while back | 20:15 |
| pabelanger | we have a failure | 20:15 |
| mordred | might still be nice (or just fun) to do a "hold on to your hats, we're flipping the main lever" | 20:15 |
| mordred | BLAST | 20:15 |
| pabelanger | http://paste.openstack.org/show/621982/ | 20:15 |
| fungi | was around two hours ago, so probably recent enough | 20:16 |
| clarkb | we're about to hit 88mph | 20:16 |
| jeblair | mordred: blast for what pabelanger wrote or something else? | 20:17 |
| pabelanger | primary hostvar doesn't work for non multinode jobs | 20:17 |
| mordred | oh. yah. pabelanger thing | 20:17 |
| mordred | yah- we havea bug in that pre playbook | 20:17 |
| *** Apoorva has joined #openstack-infra | 20:18 | |
| mordred | it'll work for devstack jobs | 20:18 |
| mordred | do we should we write that primary file even if there is not a primary node defined if there's only one? | 20:19 |
| jeblair | it seems unlikely that any non-devstack non-multinode job would use it | 20:19 |
| mordred | or should we just skip it and switch people that need it to the devstack-singlenode | 20:20 |
| mordred | yah | 20:20 |
| mordred | just gonna put in a when for now then | 20:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Don't write out nodepool primary file on non-primary node https://review.openstack.org/507667 | 20:20 |
| mordred | jeblair, pabelanger: ^^ | 20:20 |
| mordred | bah | 20:21 |
| pabelanger | mordred: can't we just use {{ nodepool.private_ip }} ? | 20:21 |
| pabelanger | err | 20:22 |
| mordred | oh- you're right | 20:22 |
| pabelanger | nodepool.private_ipv4 | 20:22 |
| *** florianf has quit IRC | 20:22 | |
| mordred | well - no | 20:22 |
| mordred | we can't do that | 20:22 |
| mordred | bceaues that won't work for the devstackjobs | 20:22 |
| mordred | we could do it if primary isn't defined | 20:22 |
| pabelanger | Ah right | 20:22 |
| pabelanger | because multinode | 20:22 |
| mordred | yah | 20:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Don't write out nodepool primary file on non-primary node https://review.openstack.org/507667 | 20:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Write private ip of non-primary node https://review.openstack.org/507669 | 20:23 |
| mordred | there- I did the otherthing as a follow on - so we can add it if we want to | 20:23 |
| *** mat128 has quit IRC | 20:23 | |
| *** dave-mccowan has joined #openstack-infra | 20:24 | |
| mordred | ooh. this one is new: http://logs.openstack.org/22/507222/3/check/legacy-bindep-fallback-fedora-26/c6183a9/ | 20:25 |
| *** dave-mcc_ has quit IRC | 20:25 | |
| clarkb | wait if we write tge same file with the same content in both places why cant we write it wuth one task? | 20:26 |
| *** jkilpatr has joined #openstack-infra | 20:26 | |
| jlk | it's not the same content | 20:26 |
| mordred | clarkb: becaue the source from which we get the data is different | 20:26 |
| mordred | yah. what jlk said | 20:26 |
| mordred | pabelanger, jeblair: potentially larger problem - we're publishing things to logs.o.o with dir permissions of 700 | 20:27 |
| mordred | mordred@static:~$ ls -ltra /srv/static/logs/22/507222/3/check/legacy-bindep-fallback-fedora-26/ | 20:27 |
| mordred | drwx------ 5 jenkins jenkins 4096 Sep 26 20:21 c6183a9 | 20:27 |
| jlk | FORBIDDEN | 20:27 |
| jeblair | mordred: all 3 of runs of those jobs have that | 20:27 |
| jeblair | yeah | 20:27 |
| pabelanger | Hmm | 20:27 |
| *** jamesmcarthur has quit IRC | 20:27 | |
| clarkb | well the filename is the same in both places | 20:27 |
| clarkb | should have the same content | 20:27 |
| mordred | clarkb: nope. it's not the same content | 20:27 |
| mordred | becaue in multi-node jobs it's writing theip of the primary host on all of the hosts whetherthey are primary or not | 20:28 |
| clarkb | ok then why are both primary_node_private? | 20:28 |
| jlk | in multi-node there is a "primary" node, that's the one for multinode setups. But if there is just one node, ther eis no "primary" | 20:28 |
| clarkb | right we dont write primary node if not multinide | 20:28 |
| clarkb | we write nothing | 20:28 |
| mordred | great. so we should abandon that patch | 20:28 |
| jlk | I see what you're suggesting | 20:28 |
| mordred | (that's why I put it up separate) | 20:28 |
| clarkb | gotcha ok | 20:28 |
| mordred | abandoned | 20:29 |
| pabelanger | mordred: I think this is because /home/zuul on fedora is 0700 | 20:29 |
| jeblair | pretty sure nodepool writes it anyway | 20:29 |
| pabelanger | and we preserve the permissions | 20:29 |
| pabelanger | 2017-09-26 20:18:48.655308 | TASK [Copy files from /home/zuul/ on node] | 20:29 |
| jeblair | but like i said earlier, i'm fine attempting the regression | 20:29 |
| mordred | awesome. | 20:29 |
| clarkb | I thought nodepool only wrote thos files in multinide setup | 20:29 |
| clarkb | it writes some other files always | 20:30 |
| jeblair | i wonder why zuul v2.5 is not hitting this? | 20:30 |
| pabelanger | mordred: http://logs.openstack.org/22/507222/3/check/legacy-bindep-fallback-ubuntu-trusty/00eb779/job-output.txt.gz#_2017-09-26_20_23_40_377869 | 20:30 |
| mordred | jeblair: because workspace shifted | 20:30 |
| pabelanger | I'm surprised we are syncing /home/zuul to executor | 20:30 |
| mordred | jeblair: in 2.5 workspace and home are not the same thing | 20:30 |
| mordred | in v3 they are the same thing | 20:30 |
| jeblair | clarkb: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/nodepool.py#n648 | 20:30 |
| jeblair | mordred: gotcha | 20:30 |
| mordred | pabelanger: we're not actually syncing all of /home/zuul - there are rsync filters in place | 20:31 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/bindep-fallback-fedora-26/post.yaml#n4 is the issue | 20:31 |
| mordred | pabelanger: it's part of the migration/translation of the old-style jjb publishers | 20:31 |
| pabelanger | I think we are preserving the permissions on /home/zuul | 20:31 |
| *** ltomasbo has quit IRC | 20:31 | |
| jlk | grabbing all of it would be... amusing. | 20:31 |
| clarkb | ah so it doesnt write subnodes but does the primary always | 20:31 |
| jlk | so whatever creates the logs directory/files should maybe set perms? | 20:31 |
| *** jpena|off has quit IRC | 20:32 | |
| *** Goneri has joined #openstack-infra | 20:32 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: We should await ws.send_str() https://review.openstack.org/507671 | 20:32 |
| *** amoralej|off has quit IRC | 20:32 | |
| mordred | yah - I think having the log publisher do it is more sane - we don't really have any valid use cases for publishing logs that people can't read | 20:32 |
| jeblair | we could 1) fix this in the migration script by adding an extra step after the rsync. or 2) fix it in the v3 post-logs playbook so all logs are always published world-readable | 20:32 |
| jeblair | jlk, mordred: i think you just favored #2 ^? | 20:33 |
| mordred | we could also put a chmod 755 /home/zuul in the pre-playbook before we copy git repos over even | 20:33 |
| mordred | jeblair: yes - I favor #2 | 20:33 |
| pabelanger | #2 seems good step | 20:33 |
| mordred | because something else could cause a similar issue | 20:33 |
| pabelanger | ya | 20:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Return stream log message on finger client failure https://review.openstack.org/507663 | 20:33 |
| jlk | where is this set up? | 20:34 |
| jeblair | error in log: http://paste.openstack.org/show/621984/ | 20:34 |
| jeblair | jlk: project-config/playbooks/base/post-logs | 20:34 |
| jlk | ah | 20:34 |
| *** ltomasbo has joined #openstack-infra | 20:35 | |
| clarkb | feiw this logs publishing perms issue existed in 2.5 as well | 20:35 |
| jeblair | jlk: at least, that's where we would implement #2; you might have been asking something else... | 20:35 |
| *** baoli_ has joined #openstack-infra | 20:35 | |
| mordred | which uses roles/upload-logs/ from zuul-jobs | 20:35 |
| clarkb | we mostly told jobs to chmod first | 20:35 |
| *** jpena|off has joined #openstack-infra | 20:35 | |
| *** amoralej has joined #openstack-infra | 20:35 | |
| clarkb | but handling that in zuul might be nice | 20:35 |
| clarkb | (at least in shared role) | 20:35 |
| mordred | clarkb: yah- the difference here is that we shifted what WORKSPACE is | 20:35 |
| jlk | well | 20:35 |
| pabelanger | I thought we removed afs_publisher_target | 20:35 |
| mordred | clarkb: so ratherthan WORKSPACE eing a dir that was created - it's thezuul user's homedir | 20:35 |
| jeblair | i'm going to track down the final job error; assuming someone else will work on logs | 20:36 |
| jlk | I was thinking that whatever is initially creating the directory and files on the Fedora host where the logs go is what should set the permissions | 20:36 |
| *** ijw has joined #openstack-infra | 20:36 | |
| *** felipemonteiro has quit IRC | 20:36 | |
| *** baoli has quit IRC | 20:36 | |
| pabelanger | jlk: that might not work, because we are pulling from /home/zuul, which is 0700 | 20:36 |
| mordred | yah. although - we could also do a chmod on that very early in the base pre-playbook | 20:37 |
| jlk | are the logs not going into a subdir? just directly in the WORKSPACE? | 20:38 |
| jeblair | jlk: they are in a subdir, but the rsync is rooted at WORKSPACE | 20:38 |
| jlk | sure | 20:38 |
| jlk | I'm not seeing where that rsync is that's pulling it from the node | 20:39 |
| jeblair | jlk: pabelanger said it was here: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/bindep-fallback-fedora-26/post.yaml#n4 | 20:39 |
| jlk | I see it pushing from the executor, from zuul.executor.log_root | 20:39 |
| jlk | oh | 20:40 |
| jlk | that's... odd | 20:40 |
| jeblair | jlk: yeah, if you're looking where i told you earlier, that's not it. i apparently misunderstood your question. | 20:40 |
| mordred | jlk: ansible file module - for making contents of a directory readable - | 20:40 |
| pabelanger | the more I look at http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/bindep-fallback-fedora-26/post.yaml the more I see an issue | 20:40 |
| pabelanger | .tox/*/log will not be in /home/zuul | 20:41 |
| jeblair | mordred: http://git.openstack.org/cgit/openstack-infra/zuul/tree/.zuul.yaml?h=feature/zuulv3#n80 that sure is a modification of a final job | 20:41 |
| pabelanger | it will be in /home/zuul/src/... | 20:41 |
| pabelanger | so, I think we might have an issue on publishers | 20:41 |
| mordred | pabelanger: yah - that rsync invocation will actually do the correct thing | 20:41 |
| mordred | pabelanger: there is a big comment about it in th e2.5 source and also in the migration script source - it's using some advanced rsync magic from jeblair from the 2.5 days | 20:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't output cat file contents during reconfig https://review.openstack.org/507665 | 20:42 |
| pabelanger | okay | 20:42 |
| mordred | jlk: is it smart enough to be able to do files +r and dirs +rx? | 20:42 |
| jlk | mordred: ¯\_(ツ)_/¯ I'd have to read the code. | 20:42 |
| mordred | pabelanger: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/cmd/migrate.py?h=feature/zuulv3#n566 | 20:42 |
| jeblair | pabelanger: http://logs.openstack.org/22/507222/3/check/legacy-bindep-fallback-fedora-26/c6183a9/ | 20:42 |
| mordred | jlk: butyou know everything! | 20:42 |
| pabelanger | I would just expect http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/bindep-fallback-fedora-26/post.yaml#n58 to be src/{{ zuul.project.canonical_name }} | 20:43 |
| jeblair | pabelanger: check that again, i made it world-readable | 20:43 |
| jlk | I just know where to look :) | 20:43 |
| jeblair | pabelanger: you can verify whether it copies what you expect | 20:43 |
| *** ijw has quit IRC | 20:43 | |
| jlk | If i understand this job, does anything explicitly in ansible create these directories we're pulling from? | 20:43 |
| jlk | and also, why is it up to this particular job to gather the logs like this? Wasn't a design consideration of zuul that jobs can just plunk content into a known path and it'll automatically get scooped up and published? | 20:44 |
| mordred | aha! I have learned a thing today: u=rwX,g=rX,o=rX | 20:44 |
| pabelanger | http://logs.openstack.org/22/507222/3/check/legacy-bindep-fallback-fedora-26/c6183a9/ara/result/d45f3f0f-b093-467f-8860-2eef073e89b1/ | 20:44 |
| pabelanger | I don't believe that will get our tox logs | 20:44 |
| prometheanfire | pabelanger: still needed or can be abandoned? https://review.openstack.org/506951 | 20:44 |
| pabelanger | but, happy to wait until we runt he job again | 20:44 |
| mordred | prometheanfire: right. bit you can see that it does | 20:44 |
| mordred | gah | 20:44 |
| mordred | pabelanger: | 20:44 |
| fungi | yeah, X is awesome in newer chmod anyway | 20:44 |
| prometheanfire | lol | 20:44 |
| mordred | oh! I lied - they are not there | 20:45 |
| pabelanger | right | 20:45 |
| jeblair | http://logs.openstack.org/22/507222/3/check/legacy-bindep-fallback-fedora-26/c6183a9/tox/ is empty | 20:45 |
| mordred | well- let's get the perms thing done real quick | 20:45 |
| pabelanger | lets wait until 507667 lands and recheck for tox things | 20:45 |
| jlk | mordred: with that finding, yes file module can handle different perms for files and dirs | 20:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Ensure logs are readable before uploading them https://review.openstack.org/507673 | 20:46 |
| mordred | jlk, jeblair, pabelanger: ^^ | 20:46 |
| jlk | (assuming a new enough chmod on the target) | 20:46 |
| *** jamesmcarthur has joined #openstack-infra | 20:46 | |
| pabelanger | I have to relocation to mnaser location, but will be taking laptop with me | 20:46 |
| mnaser | we have wifi :> | 20:46 |
| pabelanger | yay | 20:47 |
| jlk | Is this legacy job copying content around just a hold over until the job is re-designed to put logs into the expected path? | 20:48 |
| mordred | jlk: yes | 20:48 |
| jeblair | well | 20:48 |
| *** jcoufal_ has quit IRC | 20:49 | |
| jeblair | to be clear, in our use of v3 jobs still need to fetch logs from workers and put them on the executor for them to be archived. | 20:49 |
| jlk | why does it copy things to the executor itself, rather than just to where the expected path is on the worker node so that the standard "scoop and publish" routine does its thing? | 20:49 |
| jeblair | so the "plop it into a location" location is on the executor | 20:49 |
| jeblair | not on the worker | 20:49 |
| jlk | ah, there is no base job that scoops? | 20:49 |
| *** armax has quit IRC | 20:50 | |
| mordred | it does - from the executor | 20:50 |
| jeblair | jlk: nope, only publishes from executor. no automatic scooping from node. | 20:50 |
| jlk | right. | 20:50 |
| *** armax has joined #openstack-infra | 20:50 | |
| jlk | okay | 20:50 |
| jeblair | you could *totally* write a v3 base playbook to do that, but we didn't. | 20:50 |
| jlk | in bonny land, we felt teaching folks about the executor was a bit of a stretch, so we were planning to have known paths for artifact upload | 20:50 |
| *** armax has quit IRC | 20:50 | |
| jlk | and making everybody write their own rsync tasks would have been rude :) | 20:51 |
| jeblair | that would be a really good thing to agree on so that things in zuul-jobs implement the same interface. | 20:51 |
| *** armax has joined #openstack-infra | 20:51 | |
| mordred | we talked in denver a little bit about trying to rework that some - similar to that ^^ - but I'm not sure we got to consensus on that yet | 20:51 |
| mordred | butyes - I agree with jeblair | 20:51 |
| jlk | nod | 20:51 |
| *** armax has quit IRC | 20:51 | |
| mordred | jeblair: I'm concerned about the tox dir being empty in that job though | 20:51 |
| jeblair | i think andreaf and i (or maybe just andreaf because i've been busy and haven't looked!) will end up with a handy role to collect logs. | 20:51 |
| *** armax has joined #openstack-infra | 20:52 | |
| *** armax has quit IRC | 20:52 | |
| *** jamesmcarthur has quit IRC | 20:52 | |
| jeblair | jlk: at the end of the day though "copy this from a to b" vs "copy this from a to c" isn't that different :) | 20:52 |
| *** armax has joined #openstack-infra | 20:52 | |
| jlk | well, it's a lot easier to tell a task to redirect it's output to path/over/there | 20:53 |
| *** armax has quit IRC | 20:53 | |
| jlk | than to rsync things around | 20:53 |
| jlk | If you can document "anything placed in directory X will be published to logs", it makes it easy to design jobs around that | 20:53 |
| rm_work | erg, anyone else run into issues with cert validation for https://bootstrap.pypa.io/ from xenial machines? | 20:54 |
| *** ijw has joined #openstack-infra | 20:54 | |
| jlk | (at least, that's my hypothesis, I have no data to back it up) | 20:54 |
| rm_work | seems the ca-certificates installed do not validate the cert they're using now | 20:54 |
| jeblair | jlk: that is true, but only in some limited cases (like "run this test runner"). much harder with "archive this service log". and then you end up with "copy from a to b" followed by "copy from b to c". | 20:54 |
| *** camunoz has quit IRC | 20:54 | |
| jlk | jeblair: where one is exposed to the end user (copy from A to B on the ephemeral host), and the other is a hidden implementation detail of the service provider | 20:55 |
| *** trown is now known as trown|outtypewww | 20:55 | |
| jlk | an end user can easily use some shell to copy desired files into the standard path. Less easily write a proper rsync task in Ansible | 20:56 |
| mordred | jeblair: so- http://logs.openstack.org/36/507636/1/gate/gate-project-config-layout/9ad4622/_zuul_ansible/post_playbook is an old 2.5 generated post playbook and http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/bindep-fallback-fedora-26/post.yaml is the legacy-bindep-fallback job | 20:56 |
| jlk | (Ansible use is another implementation detail that I guess many consumers would not care about) | 20:56 |
| jeblair | jlk: i will categorically say that evaluating this decision in the light of the auto-migrated jobs is not something we should do. | 20:56 |
| jeblair | jlk: if we reopen this conversation, it should only be in the context of actual native v3 jobs. :) | 20:56 |
| rm_work | nevermind, my VM's clock is off (FML) | 20:56 |
| jlk | 100% agreement | 20:56 |
| jeblair | mordred: looking | 20:57 |
| jeblair | mordred: i guess we're looking at the last one, line 56 in the new playbook? | 20:58 |
| mordred | SOGH | 20:58 |
| mordred | red herring | 20:58 |
| mordred | that job does not use tox | 20:58 |
| mordred | the test-results macro from jjb is used on that job, which attempts to collect a pile of things | 20:59 |
| mordred | jeblair: ^^ | 20:59 |
| jeblair | okay, does anyone know what this job is supposed to output? | 20:59 |
| mordred | it just runs /usr/local/jenkins/slave_scripts/install-distro-packages.sh | 21:00 |
| mordred | so it should just have console output | 21:00 |
| *** hemna__ has quit IRC | 21:00 | |
| jeblair | and it does have console output. | 21:00 |
| mordred | http://logs.openstack.org/22/507222/3/check/legacy-bindep-fallback-fedora-26/c6183a9/job-output.txt.gz | 21:00 |
| jeblair | i mean, it's missing the bulk of the job because of the primary thing, but that's in progress | 21:00 |
| mordred | jeblair: http://logs.openstack.org/22/507222/3/check/project-config-nodepool/4b8c33b/ | 21:01 |
| jeblair | tox-linters timed out on that | 21:01 |
| jeblair | http://logs.openstack.org/67/507667/2/check/tox-linters/773f7e0/ | 21:01 |
| jeblair | mordred: what am i looking at there? | 21:01 |
| mordred | oh- nevermind- I was thining that was oneusing migrated content- please ignore link | 21:02 |
| jeblair | 2017-09-26 20:26:57.628776 | TASK [tox-siblings : Install any sibling python packages] | 21:02 |
| jeblair | 2017-09-26 20:26:58.515040 | ubuntu-xenial | MODULE FAILURE | 21:02 |
| jeblair | mordred: it does include that | 21:02 |
| *** ihrachys has quit IRC | 21:03 | |
| jlk | One other approach we could take with permissions is to use filesystem ACLs so that anything placed in the executor log path gets the ACLs we want (group/world readable), and use the right flag when syncing from executor to fileserver so that ACLs are preserved. Would save needing an explicit chmod -R call prior to sync. | 21:03 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Ensure logs are readable before uploading them https://review.openstack.org/507673 | 21:04 |
| jeblair | jlk: worth looking into | 21:05 |
| jeblair | mordred, clarkb: what's the deal with tox-linters on ozj? | 21:05 |
| jeblair | it looks like it's still taking 30+ minutes to run | 21:05 |
| *** dprince has quit IRC | 21:06 | |
| jeblair | i guess we *are* running in parallel, it's just slow? | 21:07 |
| *** baoli_ has quit IRC | 21:07 | |
| SpamapS | jlk: rsync 3.1 has some stuff to do that in-flight I think | 21:08 |
| *** panda|off has quit IRC | 21:09 | |
| jlvillal | So is there a better place for us to download ipmitool from? Our Ironic job downloads it from sourceforge. But I am seeing this message. | 21:10 |
| mordred | jeblair: ok - I see the bug in the project-config-nodepool job | 21:10 |
| jlvillal | We're sorry -- the Sourceforge site is currently in Disaster Recovery mode, and currently requires | 21:10 |
| jlvillal | the use of javascript to function. Please check back later. | 21:10 |
| *** camunoz has joined #openstack-infra | 21:10 | |
| jeblair | mordred: sorry i didn't know there was a bug | 21:10 |
| jlvillal | Can we store the ipmitool file on some openstack infra site? | 21:10 |
| mordred | jeblair: the module failure you pasted above | 21:10 |
| mordred | jeblair: it's a bug specific to project-config :) | 21:10 |
| jeblair | mordred: ok | 21:10 |
| jeblair | mordred: is there a reason we only do one file at a time with ansible-playbook --syntax check? | 21:11 |
| jeblair | it accepts multiple playbooks at once | 21:11 |
| jlk | jlvillal: why isn't it packaged in a distro? | 21:11 |
| mordred | jeblair: I do not believe there is a reason for that no - also - the earlier run took 6 or 8 minutes iirc- so I do also think we got a very slow node | 21:11 |
| jlvillal | jlk: Our distro is tinycore linux | 21:12 |
| *** srobert has quit IRC | 21:12 | |
| *** armax has joined #openstack-infra | 21:12 | |
| jlk | infra has methods to mirror the package repo content of many distros, which is typically the way it provides resiliency against failures like this. Random file mirrors aren't necessarily a thing it likes to do | 21:13 |
| jlvillal | jlk: Is there a solution in that answer? Is there something we can do to fix this? | 21:14 |
| *** slaweq_ has quit IRC | 21:14 | |
| jlk | sorry I'm not really a part of the infra decision makers, I'm just sharing observations from witnessing a similar request at PTG | 21:14 |
| jlvillal | jlk: Ah, okay. Thanks. | 21:15 |
| jlk | which ended with reluctant mirroring of a file, but only with an explicit promise to get the file into a package for a distro | 21:15 |
| jlvillal | Good to know | 21:16 |
| *** thorst has quit IRC | 21:16 | |
| *** thorst has joined #openstack-infra | 21:16 | |
| *** pabelanger1 has joined #openstack-infra | 21:16 | |
| pabelanger1 | o/ | 21:16 |
| pabelanger1 | unable to connect to my IRC proxy from location :( | 21:17 |
| SpamapS | jlvillal: perhaps you should build it from source | 21:17 |
| *** hashar has quit IRC | 21:17 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Handle projets without setup.py or valid setup.cfg https://review.openstack.org/507678 | 21:17 |
| mordred | jeblair: ok. that ^^ is the module failure issue from above | 21:18 |
| jlvillal | SpamapS: That is what we are doing... https://github.com/openstack/ironic-python-agent/blob/master/imagebuild/tinyipa/build-tinyipa.sh#L68-L70 | 21:18 |
| mordred | jeblair: (project-config is special and does not have a setup.py file) | 21:18 |
| jlvillal | SpamapS: https://github.com/openstack/ironic-python-agent/blob/master/imagebuild/tinyipa/build-tinyipa.sh#L148-L153 | 21:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Check all playbooks simultaneously https://review.openstack.org/507680 | 21:19 |
| mordred | jeblair: also - yay - it seems the readable logs thing helped | 21:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Run ansible-list on all playbooks simultaneously https://review.openstack.org/507681 | 21:20 |
| jlvillal | As an FYI, our failure getting ipmitool tarball: http://logs.openstack.org/72/507272/3/check/gate-tempest-dsvm-ironic-lib-wholedisk-agent_ipmitool-ubuntu-xenial/6d8f17d/logs/devstacklog.txt.gz#_2017-09-26_18_01_22_738 | 21:20 |
| jeblair | mordred, pabelanger1: 507680 works locally. i haven't tested 507681 | 21:20 |
| mordred | jeblair: I believe if we're using -P for parallel we need to use -n or -L or else it'll all just be sent to one of the parallel processes | 21:21 |
| *** thorst has quit IRC | 21:21 | |
| *** tosky has joined #openstack-infra | 21:21 | |
| jeblair | mordred: are you sure it doesn't divide it up appropriately? | 21:21 |
| mordred | jeblair: the man page says you're supposed to use -n or -L ... | 21:22 |
| jeblair | mordred: okay, i'll drop -P | 21:22 |
| mordred | "Use the -n option or the -L option with -P; otherwise chances are that only one exec will be done." | 21:22 |
| *** wolverineav has quit IRC | 21:22 | |
| mordred | jeblair: kk | 21:22 |
| jeblair | mordred: well, i get the same behavior in both cases | 21:24 |
| openstackgerrit | Tin Lam proposed openstack-infra/project-config master: Add nfs test gate to OpenStack-Helm https://review.openstack.org/507682 | 21:24 |
| jeblair | it basically makes -P a noop. so it's not an error, it's just not useful. | 21:24 |
| mordred | yah | 21:24 |
| mordred | jeblair: but if the all-in-one makes it happy - that's great by me! | 21:24 |
| jlk | SpamapS: ah, so you would have an option of --chmod=ugo=rwX in the synchronize task rsync flags | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Check all playbooks simultaneously https://review.openstack.org/507680 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Run ansible-list on all playbooks simultaneously https://review.openstack.org/507681 | 21:25 |
| jeblair | mordred: there are some post failures on 507678 | 21:26 |
| jeblair | mordred: perhaps the chmod thing was broken | 21:26 |
| mordred | the chmod thing worked for other patches - I'll investigate | 21:27 |
| jeblair | had it run yet? i thought it only recently landed | 21:29 |
| mordred | yah - may have been race condition | 21:30 |
| mordred | worth rechecking for sure | 21:30 |
| *** hashar has joined #openstack-infra | 21:30 | |
| jeblair | there are lots of post_ | 21:30 |
| jeblair | failures on the page | 21:30 |
| mordred | jeblair: nope. still getting failure | 21:30 |
| jeblair | 2017-09-26 21:30:28.432104 | logs.openstack.org | "msg": "There was an issue creating /var/lib/zuul as requested: [Errno 13] Permission denied: '/var/lib/zuul'", | 21:30 |
| jeblair | 2017-09-26 21:30:28.432242 | logs.openstack.org | "path": "/var/lib/zuul/builds/7d566c9729284f4b89522bbf8fedb3f5/work/logs/", | 21:30 |
| *** eharney has quit IRC | 21:30 | |
| *** panda has joined #openstack-infra | 21:30 | |
| jeblair | mordred: task running on wrong host | 21:30 |
| mordred | well that certainly doesn't seem right | 21:30 |
| jlk | wait | 21:30 |
| mordred | AH | 21:31 |
| jlk | you want ansible-lint to run on all the playbooks at once, right? | 21:31 |
| jlk | instead of serially one after the other | 21:31 |
| jlk | or do you want multiple runs of ansible-lint to happen, one per playbook found? | 21:31 |
| jeblair | jlk: to be honest, i want the job to finish quickly and don't care. :) | 21:31 |
| jlk | jeblair: your change removed the -n1, but didn't tell xargs to do anything in parallel, so I'm confused | 21:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: chmod the files on the executor https://review.openstack.org/507686 | 21:32 |
| jeblair | jlk: the job was previously serialized, and very very slow, so folks made it run $nproc at once, at which point it was only slow. i'm trying to do all at once and have it be fast. | 21:32 |
| jlk | oh wait. | 21:32 |
| jeblair | jlk: that will cause all the files to be put on the commandline together | 21:32 |
| jlk | I see what the default is, stack them all into a command | 21:32 |
| jeblair | so, very long command. so long, xargs actually has to run it twice | 21:32 |
| jeblair | mordred: we'll need to force-merge that | 21:33 |
| mordred | jeblair: ^^ that patch - and I apologize we will need to force-merge that | 21:33 |
| jlk | nod | 21:33 |
| mordred | jeblair: jinx | 21:33 |
| jeblair | mordred: i'll push the buttons | 21:33 |
| jlk | you can skip xargs all together though | 21:33 |
| mordred | jeblair: cool. thank you | 21:33 |
| jeblair | jlk: even better! | 21:33 |
| jlk | find -exec <whatever> {} + | 21:33 |
| jlk | the + groups all the output, but I wonder if that will fail if it's too many things | 21:34 |
| mordred | jlk: will that run exec more than once if the argument list would be too long? | 21:34 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: chmod the files on the executor https://review.openstack.org/507686 | 21:34 |
| jlk | that's what I"m about to check | 21:34 |
| mordred | neat | 21:34 |
| mordred | we're gonna have this lint job OPTIMIZED | 21:34 |
| jeblair | mordred: well, it's giving us plenty of time in which to do so :| | 21:34 |
| jlk | yes it appears it will | 21:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Handle projects without setup.py or valid setup.cfg https://review.openstack.org/507678 | 21:35 |
| jeblair | jlk: so we need xargs to split the long line? | 21:35 |
| jlk | no, the man page says it builds the command(s) like xarg does, presumably breaking it into multiple commands | 21:35 |
| jlk | if the arg list grows beyond bounds | 21:36 |
| jeblair | jlk: oh, heh, i thought you were saying it will fail. but you're saying it will work. | 21:36 |
| jlk | huh, "This variant of -exec | 21:36 |
| jlk | always returns true." | 21:36 |
| jlk | that may not be what we want :/ | 21:36 |
| jlk | now I'm confused | 21:37 |
| *** aeng has joined #openstack-infra | 21:37 | |
| jlk | that paragraph is confusing. If the exec command fails, you get a non-zero exit, which is what we want | 21:38 |
| mordred | jeblair: 678 is trending green this time | 21:39 |
| mordred | and 680 is WAY quicker | 21:40 |
| mordred | jeblair: woot. 680 finished in 2 minutes | 21:41 |
| mordred | jeblair: I find that an acceptable timeframe :) | 21:41 |
| jeblair | mordred: ++ | 21:41 |
| fungi | that _is_ way quicker | 21:41 |
| jlk | http://paste.openstack.org/show/621989/ | 21:42 |
| jlk | ^^ that should do what you want, avoiding xargs | 21:42 |
| mordred | fungi: mind +Aing https://review.openstack.org/#/c/507680 ? | 21:42 |
| clarkb | what is/was problem with xargs? | 21:42 |
| jlk | it's... not necessary? | 21:42 |
| clarkb | ah | 21:42 |
| *** r-daneel has quit IRC | 21:43 | |
| jlk | originally it's use was improperly limiting things to one execution per file | 21:43 |
| jeblair | jlk: cool, want to convert that into a patch? | 21:43 |
| jlk | sure can! | 21:43 |
| jeblair | jlk: it's self testing ;) | 21:43 |
| *** armax has quit IRC | 21:43 | |
| jeblair | though, obviously, not the bit about always returning 0 | 21:43 |
| jlk | heh | 21:43 |
| jlk | yeah I'll do that once 680 merges | 21:43 |
| jlk | or make mine depend on it | 21:44 |
| *** d0ugal has quit IRC | 21:44 | |
| *** claudiub has joined #openstack-infra | 21:44 | |
| openstackgerrit | Nick Stones-Havas proposed openstack/python-jenkins master: QuietDown requires a POST request https://review.openstack.org/507688 | 21:44 |
| jeblair | mordred: if your stack is clear, should we look at the final problem? (/me evades glare from conan doyle) | 21:45 |
| mordred | fungi, clarkb: https://review.openstack.org/#/c/507681 also when you get a sec (it's trending green and I expect it to pas) | 21:45 |
| mordred | jeblair: yes! what's the final problem? | 21:45 |
| jeblair | mordred: http://paste.openstack.org/show/621984/ | 21:45 |
| jeblair | mordred: caused by this definition: http://git.openstack.org/cgit/openstack-infra/zuul/tree/.zuul.yaml?h=feature/zuulv3#n80 | 21:45 |
| jeblair | pabelanger1: ^ of interest to you | 21:46 |
| mordred | jeblair: ah! that. so - I think we can just remove that var | 21:46 |
| jeblair | mordred: var was for earlier version of job? | 21:47 |
| mordred | jeblair: it is no longer necessary - publish-openstack-python-docs-infra publishes to infra/{short_project_name} already | 21:47 |
| *** hashar has quit IRC | 21:47 | |
| mordred | yah | 21:47 |
| jeblair | mordred: cool, i'll nab it | 21:47 |
| pabelanger1 | reading | 21:47 |
| mordred | pabelanger1: it's the docs-infra job passing afs_publisher_target thing ... we can just remove the var | 21:48 |
| pabelanger1 | k | 21:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove var override for docs publish job https://review.openstack.org/507690 | 21:48 |
| mordred | jeblair: good - I was worried we'd found an deeper issue :) | 21:48 |
| mordred | jeblair: why did that only just now trigger? | 21:48 |
| pabelanger1 | +2 | 21:48 |
| jeblair | mordred: nope, we just exercised a thing that was only exercised in tests | 21:48 |
| *** ijw has quit IRC | 21:49 | |
| jeblair | mordred: probably didn't just trigger now, probably just went unnoticed | 21:49 |
| jeblair | i'm trolling logs for issues | 21:49 |
| mordred | ah- gotcha | 21:49 |
| mordred | *phew* | 21:49 |
| mordred | I'm WAY more ok with that | 21:49 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Check all playbooks simultaneously https://review.openstack.org/507680 | 21:49 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Don't write out nodepool primary file on non-primary node https://review.openstack.org/507667 | 21:49 |
| jeblair | btw, i am keeping the etherpad up to date | 21:50 |
| mordred | cool - thank you | 21:50 |
| *** ijw has joined #openstack-infra | 21:50 | |
| *** pabelanger1 has quit IRC | 21:50 | |
| *** pabelanger1 has joined #openstack-infra | 21:50 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/openstack-zuul-jobs master: Remove spurrious use of xargs https://review.openstack.org/507691 | 21:50 |
| *** pabelanger has quit IRC | 21:50 | |
| *** pabelanger1 is now known as pabelanger | 21:50 | |
| Shrews | mordred: while we wait, this is cleanup from the finger thing: https://review.openstack.org/507671 | 21:51 |
| mordred | jeblair: incidentally- zuul is still running check jobs for 507686 even though it has merged - it's an edge case, since we normally shouldn't be merging things out from under zuul - but thought I'd point it out | 21:51 |
| *** pabelanger_ has joined #openstack-infra | 21:51 | |
| mordred | Shrews: +2 | 21:51 |
| jeblair | mordred: ya, that's something that should be fixed (it likely represents a minor bug with pipeline requirements) | 21:52 |
| mordred | Shrews, clarkb, fungi: can haz +3 on https://review.openstack.org/#/c/507678/ please? | 21:52 |
| *** kjackal_ has quit IRC | 21:52 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Run ansible-list on all playbooks simultaneously https://review.openstack.org/507681 | 21:53 |
| jeblair | mordred: oh, is the module failure blocking the nodepool change? | 21:53 |
| Shrews | mordred: i like the existential comment in that one | 21:53 |
| mordred | jeblair: yes | 21:54 |
| mordred | jeblair: because of the oddness of tox and project-config | 21:54 |
| jeblair | gotcha | 21:54 |
| mordred | jeblair: we got ALL the good edge conditions with this batch | 21:54 |
| jeblair | mordred: promise? | 21:54 |
| fungi | mordred: it's okay that we're skipping that for projects which may have a setup.py but aren't pbr/distutils2-based? | 21:55 |
| *** yamahata has joined #openstack-infra | 21:55 | |
| fungi | (and so no setup.cfg) | 21:55 |
| mordred | fungi: yah - it's a new fancy feature in v3 - if someone in our zuul isn't using pbr, then they don't get the shiny feature | 21:55 |
| pabelanger | Hmm, error with bindep fallback | 21:55 |
| fungi | k | 21:56 |
| fungi | pabelanger: i believe that's what 507678 was trying to address | 21:56 |
| mordred | fungi: skipping just reverts back to the way tox jobs have worked all this time so far | 21:56 |
| pabelanger | fungi: looking | 21:56 |
| mordred | yes. it is | 21:56 |
| fungi | mordred: thanks, makes sense | 21:56 |
| pabelanger | is bindep-fallback even valid in zuulv3 world? | 21:57 |
| openstackgerrit | Nick Stones-Havas proposed openstack/python-jenkins master: QuietDown requires a POST request https://review.openstack.org/507688 | 21:57 |
| mordred | pabelanger: the job? it needs to be rewritten for sure | 21:58 |
| mordred | pabelanger: rather than calling that script, it should call the bindep role pointing to the contents | 21:58 |
| *** Goneri has quit IRC | 21:58 | |
| pabelanger | mordred: right | 21:58 |
| *** esberglu has quit IRC | 21:58 | |
| mordred | pabelanger: but we just went with the auto-migrated version of it for today - which is good - we found some bugs! | 21:59 |
| openstackgerrit | Jesse Keating proposed openstack-infra/openstack-zuul-jobs master: Remove spurrious use of xargs https://review.openstack.org/507691 | 22:00 |
| pabelanger | mordred: ya, I'm trying to see if http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/bindep-fallback-debian-jessie/run.yaml#n46 exists any more | 22:00 |
| pabelanger | mordred: atleast for the zuul user | 22:00 |
| fungi | tricky bit about the bindep fallback job, is that it should probably have been two jobs | 22:01 |
| pabelanger | Oh, I think i see the issue | 22:02 |
| pabelanger | fungi: ya, I think you are right | 22:02 |
| pabelanger | because, project-config path to jenkins/data is not correct now | 22:02 |
| fungi | one to test that proposed changes to the fallback list work with the version of bindep installed on various images, the other to test that changes to bindep work with the version of the fallback list baked into all our images | 22:02 |
| pabelanger | export PACKAGES=~zuul/src/git.openstack.org/openstack-infra/project-conifg/jenkins/data/bindep-fallback.txt | 22:03 |
| *** jklare has quit IRC | 22:03 | |
| pabelanger | that is the fix | 22:03 |
| mordred | pabelanger: no - it's fine | 22:03 |
| pabelanger | okay | 22:03 |
| mordred | pabelanger: it's an auto-migrated job - so it's running zuul-cloner at the top - which is cloning project-config into . | 22:03 |
| mordred | see: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/bindep-fallback-debian-jessie/run.yaml#n26 | 22:04 |
| *** bobh has quit IRC | 22:04 | |
| pabelanger | mordred: oh, so legacy jobs don't have ~/src/git.o.o | 22:04 |
| mordred | they DO | 22:04 |
| mordred | but they don't know anything about that yet | 22:04 |
| fungi | so the one overloaded job optionally either tested by installing the provided bindep source via zuul-cloner and calling it against the default fallback list present on the node, or by not installing bindep and invoking the preinstalled bindep against the zuul-cloner checked out copy of the fallback list | 22:04 |
| pabelanger | okay, I'll stand down for now | 22:04 |
| mordred | so the zuul-cloner script we install clones from ~/src/git.o.o to wherever the job thinks its source repos should be | 22:05 |
| mordred | pabelanger: I mean - keep looking for issues for sure :) | 22:05 |
| mordred | pabelanger: this is a big beastie after all | 22:05 |
| fungi | i'd be fine if we just declare the fallback list and bindep frozen for a little bit while we rework the job | 22:05 |
| fungi | that one at least is on us, since it | 22:06 |
| fungi | 's infra projects on all sides | 22:06 |
| andreaf | jeblair, mordred: I made a handy role called save_file which does something like what save_file would do on d-g https://review.openstack.org/#/c/506835/7/roles/save-file/tasks/main.yaml | 22:06 |
| jlk | wtaf | 22:07 |
| jlk | tox seems to be eating my {} argument | 22:07 |
| mordred | fungi: yah- also - it would be great to make bindep-fallback go away too - so freezing the fallback list is certainly fine by me | 22:07 |
| andreaf | jeblair, mordred: however to use it I need to use include_role e.g. https://review.openstack.org/#/c/506835/7/roles/fetch-tempest-logs/tasks/main.yaml | 22:07 |
| pabelanger | I'm going to autohold a bindep-falback job to see inspect node | 22:07 |
| mordred | UGH | 22:07 |
| jlk | {} is turning into : | 22:07 |
| mordred | hit race condition bug in zuul on 507678 | 22:07 |
| mordred | :( | 22:07 |
| jeblair | mordred: ya; just rechecked it | 22:08 |
| andreaf | jeblair, mordred: which breaks the ansible callback zuul_json | 22:08 |
| jeblair | andreaf: why is include_role needed for that? | 22:08 |
| mordred | andreaf: also - I don't think you can use with_items with include_role either | 22:08 |
| pabelanger | Shrews: how can I list autohold jobs? | 22:09 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove var override for docs publish job https://review.openstack.org/507690 | 22:09 |
| Shrews | pabelanger: you can't right now | 22:09 |
| pabelanger | Shrews: kk | 22:09 |
| jeblair | andreaf: oh, because you want to do that from within a role i guess? (as opposed to within the tempest post playbook directly?) | 22:09 |
| andreaf | mordred: yes you can - it works fine (at least on my local machine) | 22:09 |
| mordred | andreaf: oh neat | 22:10 |
| andreaf | jeblair: yeah I wanted to have a role that encapulates that | 22:10 |
| openstackgerrit | Jesse Keating proposed openstack-infra/openstack-zuul-jobs master: Remove spurrious use of xargs https://review.openstack.org/507691 | 22:10 |
| jeblair | andreaf: makes sense | 22:10 |
| andreaf | jeblair: but I guess I could move it to the play | 22:10 |
| andreaf | jeblair: at least until zuul can handle it | 22:10 |
| jeblair | Shrews: autohold list would be super helpful if you have a sec | 22:10 |
| *** jklare has joined #openstack-infra | 22:11 | |
| andreaf | if I don't use the include role though, I cannot use the with_items form which I though was quite nice and readable... but ok that's just a minor thing | 22:11 |
| *** esberglu has joined #openstack-infra | 22:12 | |
| jeblair | andreaf: yeah, i think what you have there is a good thing once we fix zuul | 22:12 |
| mordred | andreaf: yah- we do want to fix the include_role bug | 22:12 |
| *** sdague has quit IRC | 22:13 | |
| *** camunoz has quit IRC | 22:14 | |
| *** jaypipes has quit IRC | 22:16 | |
| andreaf | jeblair: I wonder if the latest ps on https://review.openstack.org/#/c/506835 still triggers the permission issue on the archive - do you have time to grab the zuul log for that for me again? or is there a way I can see that log otherwise | 22:16 |
| *** esberglu has quit IRC | 22:16 | |
| pabelanger | okay, I have a bindep-fallback node held | 22:17 |
| jlk | haha, my version seems to shave 6 seconds off the run.yaml bit. 49 to 43 seconds. I think that's well within the "noise" range. | 22:17 |
| mordred | pabelanger: ok. I stand corrected - that is definitely failing | 22:17 |
| mordred | pabelanger: which node you have? can I look too? | 22:17 |
| pabelanger | 199.204.45.81 | 22:17 |
| pabelanger | we cloned into /home/zuul/project-config, but script expects /home/zuul | 22:18 |
| mordred | oh goodie | 22:18 |
| pabelanger | http://logs.openstack.org/22/507222/3/check/legacy-bindep-fallback-ubuntu-trusty/0db3741/job-output.txt.gz#_2017-09-26_22_15_30_362760 | 22:18 |
| pabelanger | I think that is why | 22:18 |
| pabelanger | we need training slash? | 22:18 |
| jeblair | andreaf: i only see the zuul_json issues for that build; no permissions errors. | 22:19 |
| pabelanger | mordred: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/bindep-fallback-ubuntu-trusty/run.yaml#n24 is clonemap | 22:19 |
| pabelanger | so I _think_ trailing slash might do it | 22:19 |
| mordred | we can't do that though | 22:19 |
| pabelanger | ya, might get messy | 22:20 |
| mordred | this is auto-migrated content - we need to fix the zuul-cloner shim if it's not doing what the previous macros expect | 22:20 |
| pabelanger | ya | 22:20 |
| mordred | jeblair: ^^ have a sec to turn your eyes to this? | 22:20 |
| pabelanger | where is our zuul-cloner shim? | 22:21 |
| jeblair | Shrews: ^ also | 22:21 |
| jeblair | pabelanger: in zuul | 22:21 |
| Shrews | pabelanger: zuul-jobs | 22:21 |
| jeblair | er that | 22:21 |
| pabelanger | ty | 22:21 |
| mordred | pabelanger: http://logs.openstack.org/03/506803/1/check/gate-bindep-bindep-fallback-ubuntu-trusty/69c2538/_zuul_ansible/scripts/02-e3e05dc31f7d466fbb9f8aa7dd60d582.sh | 22:21 |
| mordred | is the v2.5 script | 22:22 |
| pabelanger | mordred: thanks | 22:22 |
| mordred | which did this: "openstack-infra/bindep -> /home/jenkins/workspace/gate-bindep-bindep-fallback-ubuntu-trusty" | 22:22 |
| mordred | http://logs.openstack.org/03/506803/1/check/gate-bindep-bindep-fallback-ubuntu-trusty/69c2538/console.html#_2017-09-22_23_05_06_031633 | 22:23 |
| Shrews | pabelanger: are you looking at the cloner issue? i can after dinner things | 22:23 |
| pabelanger | wait, /home/jenkins? | 22:23 |
| pabelanger | Shrews: I'm looking, but learning how it works | 22:23 |
| mordred | yah- the v2.5 generated versions of the same thing | 22:24 |
| mordred | run this playbook: http://logs.openstack.org/03/506803/1/check/gate-bindep-bindep-fallback-ubuntu-trusty/69c2538/_zuul_ansible/playbook | 22:24 |
| pabelanger | Oh, ya | 22:24 |
| pabelanger | that makes sense | 22:24 |
| mordred | which sets WORKSPACE to /home/jenkins/workspace/gate-bindep-bindep-fallback-ubuntu-trusty and then chdir's to that before each script | 22:24 |
| pabelanger | 2017-09-26 22:15:30.362760 | ubuntu-trusty | cp -al /home/zuul/src/git.openstack.org/openstack-infra/project-config /home/zuul | 22:24 |
| pabelanger | that is what we did | 22:24 |
| pabelanger | which creates /home/zuul/project-config | 22:24 |
| *** Sukhdev has joined #openstack-infra | 22:24 | |
| mordred | Shrews: so - tl;dr - I believe dest: . needs to clone into the current dir instead of one dir down | 22:24 |
| pabelanger | for zuulv3 | 22:24 |
| mordred | pabelanger: yah | 22:24 |
| *** mat128 has joined #openstack-infra | 22:25 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Handle projects without setup.py or valid setup.cfg https://review.openstack.org/507678 | 22:25 |
| mordred | oh good | 22:25 |
| mordred | infra-root: I believe this is a showstopper bug - and it's late in the day | 22:26 |
| jeblair | i suspect this may be a difference between "cp" and "git clone" | 22:26 |
| mordred | or, it's late in the day for me I suppose | 22:26 |
| jeblair | so the fix may not be in clone mapper expansion, but rather in constructing the cp command | 22:26 |
| mordred | yah | 22:26 |
| pabelanger | ya | 22:27 |
| mnaser | i know this isn't the best time to ask but would we be okay with disabling ovh-gra1 till git.openstack.org timeouts disappear? we've had a gate blocking puppet change fail to pass the gate 4 times today because one of the jobs land in ovh-gra1 | 22:27 |
| jeblair | mordred: whatcha wanna do? rollback? leave it and hope no one starts using it? leave it and shut it down? | 22:28 |
| mordred | jeblair: just get drunk and pass out? | 22:28 |
| andreaf | jeblair: cool thanks - so if I move it to the play it should be fine | 22:29 |
| mordred | jeblair: honestly, I think we should just leave it - other than current inability to respond to mnaser's request | 22:29 |
| *** mat128 has quit IRC | 22:29 | |
| mordred | jeblair: the jobs themselves are landed, so people can't add .zuul.yaml files wiht conflicting job names | 22:29 |
| *** jascott1 has quit IRC | 22:29 | |
| *** kgiusti has left #openstack-infra | 22:30 | |
| jeblair | mordred: ok. do you want to send today's status update email? | 22:30 |
| mordred | jeblair: but - I tihnk we could also rollback the gate-project-config-on-v3 work depending on what people think about the chances that we'll get that zuul-cloner bug fixed in a reasonable enough time to be able to land nodepool quota changes as needed | 22:30 |
| *** tpsilva has quit IRC | 22:31 | |
| mordred | jeblair: you're online for longer than I am today - what do you think about the current project-config state | 22:31 |
| mordred | and sure - I can do the status update email | 22:31 |
| *** jascott1 has joined #openstack-infra | 22:32 | |
| jeblair | mordred: we *might* have zuul-cloner fixed by my eod? | 22:32 |
| Shrews | so, if i read correctly, should z-c just "cp $SRC/* ." when destination is "." instead of "cp $SRC ." ? because that is a very easy fix | 22:32 |
| pabelanger | well, as long as no project has a src directory in tree, otherwise, we already have a src folder in /home/zuul | 22:32 |
| pabelanger | maybe we should create a workspace directory in /home/zuul/workspace for legacy? | 22:33 |
| Shrews | oh, the clonemapper stuff might make that more difficult though | 22:33 |
| jeblair | Shrews: put simply, the "cp" command should do whatever the "git clone" command did before. | 22:34 |
| jeblair | Shrews: i'm not sure that quite matches up with anything else said so far. | 22:35 |
| Shrews | ok, well i'm not sure what the difference is offhand, and i'd like to finish dinner. so maybe best to hold off until tomorrow, unless someone else has a handle on it | 22:35 |
| openstackgerrit | Jesse Keating proposed openstack-infra/openstack-zuul-jobs master: Remove spurious use of xargs https://review.openstack.org/507691 | 22:35 |
| jeblair | Shrews: ya, go finish dinner. i'm trying to get a handle on it. i'll let you know how far i get. :) | 22:36 |
| Shrews | k | 22:36 |
| fungi | there are two patches to revert at this stage if we want to rollback and try again tomorrow? | 22:36 |
| *** dave-mccowan has quit IRC | 22:36 | |
| jeblair | fungi: 3 i think? | 22:36 |
| jeblair | fungi: though mordred is arguing that if we can accept project-config being broken we can leave it. | 22:37 |
| pabelanger | I am dropping for the next 90mins, I'll be able to come back after that | 22:37 |
| *** Guest61942 has joined #openstack-infra | 22:37 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/openstack-zuul-jobs master: Remove gratuitous use of xargs https://review.openstack.org/507691 | 22:37 |
| mordred | infra-root: https://etherpad.openstack.org/p/3ep1A1uNfz there's current draft of status email | 22:39 |
| jeblair | mordred: lgtm | 22:40 |
| mordred | fungi: the biggest issue currently is that it's not possible to land patches to nodepool's config - but the projects being registered with v3 and the jobs landed in openstack-zuul-jobs aren't really destructure and should be able to be left in place | 22:40 |
| *** lbragstad has quit IRC | 22:40 | |
| mordred | fungi: I think if we do choose to roll anything back it would want to be the make-project-config-voting-in-v3 piece | 22:41 |
| *** pabelanger has quit IRC | 22:41 | |
| mordred | which might actually need to be force-merged anyway since the issue is that we can't merge patchesto project-config | 22:41 |
| mordred | oh - WELL | 22:41 |
| mordred | jeblair, fungi: we could force-merge a patch to project-config to remove the legacy-bindep-fallback patches from the projet pipeline config - that way if the decision is made to roll back v3 gating it's just a simple revert and then a kick.sh to let zuulv2 know to take over - and we won't be stuck on landing patches | 22:43 |
| fungi | so can nodepool config adjustments be force-merged at least, if we decide we need to turn off a particular provider/region? | 22:43 |
| mordred | but if we do that we should not roll anything forward without re-adding those jobs and verifying that they work | 22:43 |
| mordred | fungi: yah- totally | 22:43 |
| mordred | so - there's a few options on thetable depending on how people are feeling as they need to EOD | 22:44 |
| fungi | while it's not great, i'm okay with the supposition that manually gerrit-submitting very trivial emergency changes to project-config is the workaround | 22:45 |
| fungi | if it allows us to get further tomorrow without having to catch back up to the bits we unwound otherwise | 22:45 |
| mordred | yah- I thnk it sets us up to be further along in the morning to take the next steps | 22:46 |
| *** threestrands has joined #openstack-infra | 22:54 | |
| *** threestrands has quit IRC | 22:54 | |
| *** threestrands has joined #openstack-infra | 22:54 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Add post step to Tempest base job https://review.openstack.org/506835 | 22:55 |
| andreaf | jeblair: moved save-file invocation to the post-tempest play ^^^ hopefully it will pass CI this time | 22:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 22:56 |
| mordred | that's just rebased - also marked it WIP ^^ | 22:56 |
| *** claudiub has quit IRC | 22:58 | |
| mordred | infra-root: I'm going to send status email now unless anyone has any objections | 22:58 |
| fungi | lgtm | 22:59 |
| *** rbrndt has quit IRC | 23:01 | |
| *** rbrndt has joined #openstack-infra | 23:01 | |
| *** rbrndt has quit IRC | 23:01 | |
| *** thorst has joined #openstack-infra | 23:04 | |
| *** yamamoto has joined #openstack-infra | 23:05 | |
| * mordred afks for a bit - will check back in in a couple of hours | 23:06 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Always create dest dir in zuul-cloner shim https://review.openstack.org/507704 | 23:07 |
| *** jascott1 has quit IRC | 23:07 | |
| jeblair | Shrews: ^ hopefully the commit message explains my state of mind. in the tests i could think of to run locally, that makes z-c and z-c-s do the same things. | 23:08 |
| *** thorst has quit IRC | 23:08 | |
| mordred | jeblair: cool | 23:09 |
| * mordred hasn't QUITE afk'd yet | 23:09 | |
| mordred | jeblair, Shrews: pabelanger_ brought up the possibility of a conflict if someone has a src dir in their repo - I think we should handle those as a great opportunity to get someone migrated off of legacy jobs | 23:10 |
| mordred | the vast majority of openstack python repos should not have src/ dirs | 23:11 |
| mordred | ok. NOW I afk | 23:11 |
| *** pabelanger has joined #openstack-infra | 23:16 | |
| *** hongbin has quit IRC | 23:22 | |
| *** jascott1 has joined #openstack-infra | 23:25 | |
| *** bobh has joined #openstack-infra | 23:26 | |
| *** sree has joined #openstack-infra | 23:26 | |
| *** jamesmcarthur has joined #openstack-infra | 23:27 | |
| *** jascott1 has quit IRC | 23:29 | |
| *** sree has quit IRC | 23:30 | |
| *** jcoufal has joined #openstack-infra | 23:31 | |
| *** jamesmcarthur has quit IRC | 23:32 | |
| *** rlandy|afk is now known as rlandy | 23:33 | |
| *** andreww has quit IRC | 23:33 | |
| *** tosky has quit IRC | 23:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix race in test_in_repo_branch https://review.openstack.org/507708 | 23:36 |
| jeblair | mordred: that *might* fix that race we keep hitting; i'm running it locally in a loop | 23:36 |
| clarkb | is projct config broken due to the zuul cloner thing? | 23:37 |
| clarkb | I'll review the two changes above shortl | 23:39 |
| jeblair | clarkb: yes; current thinking is force-merge patches if necessary | 23:39 |
| *** jcoufal has quit IRC | 23:40 | |
| jeblair | #status log project-config is unable to merge changes due to problems found during zuul v3 migration. for the time being, if any emergency changes are needed (eg, nodepool config), please discuss in #openstack-infra and force-merge them. | 23:40 |
| openstackstatus | jeblair: finished logging | 23:40 |
| jeblair | infra-root: ^ | 23:40 |
| jeblair | clarkb: but yeah, if the zuul-cloner patch is correct, we might be able to just fix that | 23:41 |
| *** bobh has quit IRC | 23:46 | |
| *** Swami has quit IRC | 23:46 | |
| *** jdandrea_ has joined #openstack-infra | 23:50 | |
| clarkb | jeblair: the commit message on the z-c shim fix makes esnse to me | 23:50 |
| fungi | taking a look too | 23:51 |
| *** pvaneck has quit IRC | 23:51 | |
| clarkb | separately concerned that blindly using cp -l might fail across fs boundaries | 23:53 |
| clarkb | (does that gracefully degreade to proper copy or failure?) | 23:53 |
| *** jascott1 has joined #openstack-infra | 23:56 | |
| bkero | mordred: Here you go :) https://www.facebook.com/notes/lcdfans/special-machine-t70-cpu-i7-7700hqwith-ips/1882070528726717/?qid=6433062313884965485&mf_story_key=3616766401621463574 | 23:56 |
| clarkb | cp: cannot create hard link '/tank/data/foo' to 'foo': Invalid cross-device link <- ya its an error | 23:57 |
| clarkb | so we may need to fix that too | 23:57 |
| *** SumitNaiksatam has quit IRC | 23:58 | |
| clarkb | oh right the shim happens after any caching so should be on the same device | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!