| pabelanger | I wouldn't expect that to be on executor, are you sure that isn't on the remote node? | 00:00 |
|---|---|---|
| jeblair | SamYaple: if pabelanger is correct, and /home/zuul/.ansible is not writable, then that's very strange | 00:00 |
| pabelanger | yah | 00:00 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py?h=feature/zuulv3#n1294 | 00:00 |
| pabelanger | that's why i think it wouldn't be executor | 00:01 |
| jeblair | SamYaple: yeah, i think that file is on the remote host | 00:01 |
| SamYaple | hmmm i suppose being on the remote host would make sense here | 00:01 |
| pabelanger | but hard to tell, I think your the first using async | 00:01 |
| jeblair | it is async, after all, so it needs to store its ..right | 00:01 |
| SamYaple | ill run a quick test | 00:01 |
| *** Apoorva has quit IRC | 00:02 | |
| *** Apoorva_ has quit IRC | 00:02 | |
| *** xarses has joined #openstack-infra | 00:02 | |
| *** markvoelker has joined #openstack-infra | 00:03 | |
| jeblair | clarkb: responded; let me know if that clears it up | 00:03 |
| clarkb | oh I see it awsn't using any of that data before | 00:04 |
| clarkb | jeblair: ok I've approved the change | 00:05 |
| *** apetrich has quit IRC | 00:05 | |
| *** apetrich has joined #openstack-infra | 00:06 | |
| ianw | pabelanger: did you figure out the finger ze05 thing? | 00:07 |
| *** markvoelker has quit IRC | 00:07 | |
| pabelanger | ianw: no, I haven't done much work today to look. | 00:08 |
| pabelanger | ianw: did we add it to zuulv3-issues etherpad? | 00:08 |
| jeblair | i'm going to restart the scheduler now | 00:09 |
| pabelanger | ianw: ya, ze06, still down it seem | 00:09 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Remove migrated legacy-octavia-dashboard-* https://review.openstack.org/513566 | 00:09 |
| pabelanger | adding it to zuul3-issues now | 00:09 |
| pabelanger | ianw: okay, added. | 00:12 |
| openstackgerrit | Michael Johnson proposed openstack-infra/openstack-zuul-jobs master: Remove migrated legacy-octavia-dashboard-* https://review.openstack.org/513568 | 00:12 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Remove migrated legacy-octavia-dashboard-* https://review.openstack.org/513566 | 00:12 |
| ianw | pabelanger: cool ... see also my notes above on the rename, you still ok to drive it? | 00:13 |
| jeblair | scheduler is up and re-enqueuing | 00:13 |
| jeblair | if you hard-refresh, you can see the management event counts now | 00:13 |
| pabelanger | ianw: I haven't looked yes, I thought fungi was driving, I should be able to assist, but could be delayed due to flight tomorrow | 00:14 |
| pabelanger | yet* | 00:14 |
| pabelanger | ianw: 20:00 UTC, right? | 00:15 |
| ianw | yep | 00:15 |
| pabelanger | ianw: Yah, I should be home well before that | 00:15 |
| ianw | i plan to be asleep before that :) | 00:16 |
| *** andreas_s has joined #openstack-infra | 00:18 | |
| *** masber has joined #openstack-infra | 00:20 | |
| *** markvoelker has joined #openstack-infra | 00:21 | |
| *** masuberu has quit IRC | 00:22 | |
| *** xarses has quit IRC | 00:22 | |
| *** andreas_s has quit IRC | 00:22 | |
| *** signed8bit has quit IRC | 00:24 | |
| *** xarses has joined #openstack-infra | 00:28 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Stop storing dependent items on buildsets https://review.openstack.org/513441 | 00:28 |
| *** thorst has quit IRC | 00:29 | |
| *** felipemonteiro has joined #openstack-infra | 00:30 | |
| *** smatzek has quit IRC | 00:31 | |
| *** xinliang has quit IRC | 00:33 | |
| *** baoli has joined #openstack-infra | 00:40 | |
| *** esberglu has quit IRC | 00:40 | |
| SamYaple | jeblair: pabelanger http://logs.openstack.org/79/512479/6/check/loci-nova/a66a9ff/ubuntu-xenial/logs/async_logs/ | 00:40 |
| SamYaple | thanks both fo you | 00:41 |
| SamYaple | async logs are on the remote host | 00:41 |
| *** xinliang has joined #openstack-infra | 00:45 | |
| *** xinliang has quit IRC | 00:45 | |
| *** xinliang has joined #openstack-infra | 00:45 | |
| *** andreas_s has joined #openstack-infra | 00:46 | |
| jeblair | SamYaple: \o/ | 00:46 |
| *** xarses has quit IRC | 00:47 | |
| jeblair | SamYaple: we should probably swing back around to that later though -- i'm sure there's a way to get that into the native ansible reporting without poking at those files directly. maybe some specific way to register the async task or something. | 00:48 |
| SamYaple | or just test if the folder exists maybe? | 00:49 |
| *** andreas_s has quit IRC | 00:50 | |
| *** dave-mccowan has quit IRC | 00:52 | |
| *** felipemonteiro__ has joined #openstack-infra | 00:52 | |
| *** felipemonteiro__ has quit IRC | 00:53 | |
| *** felipemonteiro__ has joined #openstack-infra | 00:53 | |
| *** dave-mccowan has joined #openstack-infra | 00:55 | |
| *** baoli has quit IRC | 00:55 | |
| *** markvoelker has quit IRC | 00:55 | |
| *** huanxie has joined #openstack-infra | 00:55 | |
| *** felipemonteiro has quit IRC | 00:56 | |
| *** felipemonteiro__ has quit IRC | 01:02 | |
| *** LindaWang has joined #openstack-infra | 01:02 | |
| *** yamahata has quit IRC | 01:03 | |
| *** cuongnv has joined #openstack-infra | 01:03 | |
| *** iyamahat__ has quit IRC | 01:03 | |
| *** aeng has quit IRC | 01:08 | |
| *** markvoelker has joined #openstack-infra | 01:09 | |
| *** Goneri has quit IRC | 01:09 | |
| *** gyee has quit IRC | 01:09 | |
| dmsimard | Back home from Ottawa \o/ | 01:11 |
| SamYaple | w00t | 01:11 |
| SamYaple | silly canada | 01:11 |
| *** kiennt26 has joined #openstack-infra | 01:11 | |
| *** kiennt26 has quit IRC | 01:13 | |
| *** markvoelker has quit IRC | 01:14 | |
| *** dave-mccowan has quit IRC | 01:15 | |
| *** kiennt26 has joined #openstack-infra | 01:15 | |
| *** kiennt26 has quit IRC | 01:15 | |
| *** kiennt26 has joined #openstack-infra | 01:16 | |
| *** erlon has quit IRC | 01:16 | |
| *** dingyichen has joined #openstack-infra | 01:16 | |
| *** markvoelker has joined #openstack-infra | 01:18 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add project-templates to docs https://review.openstack.org/513185 | 01:19 |
| *** andreas_s has joined #openstack-infra | 01:22 | |
| *** markvoelker has quit IRC | 01:23 | |
| *** dave-mccowan has joined #openstack-infra | 01:23 | |
| dmsimard | SamYaple: dude there was poutine at the OpenStack days Canada, it was awesome :D | 01:24 |
| *** rhallisey has quit IRC | 01:24 | |
| *** Goneri has joined #openstack-infra | 01:24 | |
| jeblair | tonyb: implemented your suggestion https://review.openstack.org/513574 | 01:25 |
| *** andreas_s has quit IRC | 01:26 | |
| *** markvoelker has joined #openstack-infra | 01:27 | |
| tonyb | jeblair: Thanks. I'll go over that series again now | 01:29 |
| *** liujiong has joined #openstack-infra | 01:30 | |
| *** wolverineav has quit IRC | 01:31 | |
| *** wolverineav has joined #openstack-infra | 01:32 | |
| *** markvoelker has quit IRC | 01:32 | |
| *** salv-orlando has joined #openstack-infra | 01:33 | |
| *** salv-orl_ has quit IRC | 01:36 | |
| *** wolverineav has quit IRC | 01:36 | |
| *** markvoelker has joined #openstack-infra | 01:37 | |
| openstackgerrit | Rico Lin proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs for heat https://review.openstack.org/513576 | 01:37 |
| openstackgerrit | Rico Lin proposed openstack-infra/project-config master: Remove legacy jobs from heat https://review.openstack.org/509194 | 01:39 |
| *** andreas_s has joined #openstack-infra | 01:40 | |
| *** markvoelker has quit IRC | 01:41 | |
| *** andreas_s has quit IRC | 01:44 | |
| *** markvoelker has joined #openstack-infra | 01:46 | |
| *** markvoelker has quit IRC | 01:50 | |
| pabelanger | SamYaple: good to hear | 01:50 |
| *** thorst has joined #openstack-infra | 01:53 | |
| *** liujiong has quit IRC | 01:53 | |
| *** liujiong has joined #openstack-infra | 01:54 | |
| *** thorst has quit IRC | 01:54 | |
| *** markvoelker has joined #openstack-infra | 01:55 | |
| *** baoli has joined #openstack-infra | 01:56 | |
| *** hongbin has joined #openstack-infra | 01:56 | |
| *** harlowja has quit IRC | 01:58 | |
| *** andreas_s has joined #openstack-infra | 01:58 | |
| openstackgerrit | Rico Lin proposed openstack-infra/project-config master: Remove legacy job for python-heatclient https://review.openstack.org/513578 | 01:59 |
| openstackgerrit | Rico Lin proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs for python-heatclient https://review.openstack.org/513579 | 01:59 |
| *** baoli has quit IRC | 02:01 | |
| *** panda|rover is now known as panda|rover|off | 02:01 | |
| *** mriedem has quit IRC | 02:02 | |
| *** dave-mccowan has quit IRC | 02:04 | |
| *** markvoelker has quit IRC | 02:16 | |
| *** andreas_s has quit IRC | 02:16 | |
| *** markvoelker has joined #openstack-infra | 02:17 | |
| openstackgerrit | Anup Navare proposed openstack-infra/devstack-gate master: [Test] DNM Checking if tinyIPA builds with py3 https://review.openstack.org/509641 | 02:18 |
| *** ijw has quit IRC | 02:20 | |
| *** andreas_s has joined #openstack-infra | 02:21 | |
| *** ijw has joined #openstack-infra | 02:22 | |
| *** andreas_s has quit IRC | 02:26 | |
| *** ijw has quit IRC | 02:27 | |
| *** markvoelker has quit IRC | 02:29 | |
| *** psachin has joined #openstack-infra | 02:30 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Routes Mapping Bug https://review.openstack.org/513586 | 02:32 |
| *** edmondsw has quit IRC | 02:33 | |
| *** huanxie has quit IRC | 02:33 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Routes Mapping Bug https://review.openstack.org/513586 | 02:33 |
| openstackgerrit | Rico Lin proposed openstack-infra/project-config master: Remove legacy job for heat-templates https://review.openstack.org/513587 | 02:33 |
| openstackgerrit | Rico Lin proposed openstack-infra/openstack-zuul-jobs master: Remove legacy job for heat-templates https://review.openstack.org/513589 | 02:34 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Marketplace API https://review.openstack.org/498102 | 02:36 |
| openstackgerrit | Sam Yaple proposed openstack-infra/system-config master: Add stretch mirror for ceph https://review.openstack.org/513591 | 02:38 |
| mnaser | hi frienz | 02:38 |
| mnaser | how are release jobs coping if someone has any ideas | 02:39 |
| *** markvoelker has joined #openstack-infra | 02:43 | |
| *** ykarel|off has joined #openstack-infra | 02:43 | |
| *** ykarel|away has joined #openstack-infra | 02:43 | |
| *** nicolasbock has quit IRC | 02:45 | |
| *** markvoelker has quit IRC | 02:47 | |
| *** ykarel|off has quit IRC | 02:52 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Don't cleanup TripleO CI Tempest resources https://review.openstack.org/513169 | 02:52 |
| *** ykarel|away has quit IRC | 02:52 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove networking-ovn legacy jobs https://review.openstack.org/513367 | 02:53 |
| openstackgerrit | Merged openstack-infra/project-config master: Rename ironic jobs for nova/neutron/devstack https://review.openstack.org/513410 | 02:54 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove dib legacy playbooks https://review.openstack.org/512166 | 02:54 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: networking-odl: Removing legacy jobs definitions https://review.openstack.org/512644 | 02:54 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: make openstack-tox-pypy run tox pypy https://review.openstack.org/513271 | 02:56 |
| *** markvoelker has joined #openstack-infra | 03:01 | |
| *** ramishra has joined #openstack-infra | 03:03 | |
| *** markvoelker has quit IRC | 03:05 | |
| *** andreas_s has joined #openstack-infra | 03:06 | |
| *** huanxie has joined #openstack-infra | 03:10 | |
| *** markvoelker has joined #openstack-infra | 03:10 | |
| *** baoli has joined #openstack-infra | 03:11 | |
| *** hongbin has quit IRC | 03:13 | |
| *** baoli has quit IRC | 03:14 | |
| *** hongbin has joined #openstack-infra | 03:14 | |
| *** baoli has joined #openstack-infra | 03:14 | |
| *** markvoelker has quit IRC | 03:15 | |
| *** baoli has quit IRC | 03:19 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add configure-swap role https://review.openstack.org/513595 | 03:19 |
| *** andreas_s has quit IRC | 03:20 | |
| *** annp has joined #openstack-infra | 03:22 | |
| *** dhinesh has quit IRC | 03:24 | |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Fix releasenotes job for VPNaaS https://review.openstack.org/513598 | 03:26 |
| *** markvoelker has joined #openstack-infra | 03:28 | |
| clarkb | mnaser: aiui very close to working just have to sort out the announce jobs | 03:29 |
| *** andreas_s has joined #openstack-infra | 03:29 | |
| *** dingyichen has quit IRC | 03:34 | |
| *** andreas_s has quit IRC | 03:34 | |
| ianw | ok, something weird with a devstack job | 03:35 |
| ianw | i just watched the console, this comes out | 03:35 |
| ianw | http://paste.openstack.org/show/624120/ | 03:35 |
| ianw | the job finishes ... now it's back in "queued" status? | 03:35 |
| dmsimard | project-config-core, mordred: https://review.openstack.org/#/c/513509/ should fix the publish api thing | 03:39 |
| *** xinni9e has quit IRC | 03:40 | |
| * dmsimard sleep | 03:40 | |
| *** edmondsw has joined #openstack-infra | 03:44 | |
| *** edmondsw has quit IRC | 03:48 | |
| *** thorst has joined #openstack-infra | 03:55 | |
| *** hongbin has quit IRC | 03:58 | |
| ianw | ok, it's my syntax error, but there wasn't a clear message | 04:00 |
| *** thorst has quit IRC | 04:00 | |
| *** markvoelker has quit IRC | 04:02 | |
| openstackgerrit | Nguyen Van Trung proposed openstack-infra/infra-manual master: Change http to https link https://review.openstack.org/512640 | 04:09 |
| *** claudiub has joined #openstack-infra | 04:10 | |
| *** markvoelker has joined #openstack-infra | 04:16 | |
| *** ykarel|away has joined #openstack-infra | 04:17 | |
| *** ykarel|off has joined #openstack-infra | 04:17 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 04:18 | |
| *** markvoelker has quit IRC | 04:21 | |
| *** huanxie has quit IRC | 04:22 | |
| *** jamesmcarthur has joined #openstack-infra | 04:25 | |
| *** markvoelker has joined #openstack-infra | 04:26 | |
| *** namnh has joined #openstack-infra | 04:26 | |
| *** Kevin_Zheng has quit IRC | 04:28 | |
| *** rosmaita has quit IRC | 04:28 | |
| *** bobh has joined #openstack-infra | 04:29 | |
| *** jamesmcarthur has quit IRC | 04:29 | |
| *** markvoelker has quit IRC | 04:30 | |
| *** gildub has quit IRC | 04:33 | |
| *** markvoelker has joined #openstack-infra | 04:35 | |
| *** bobh has quit IRC | 04:35 | |
| *** markvoelker has quit IRC | 04:39 | |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Fix releasenotes job for VPNaaS https://review.openstack.org/513598 | 04:43 |
| *** markvoelker has joined #openstack-infra | 04:44 | |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Fix releasenotes job for VPNaaS https://review.openstack.org/513598 | 04:47 |
| *** markvoelker has quit IRC | 04:48 | |
| *** ianychoi_ has joined #openstack-infra | 04:48 | |
| *** jaosorior has joined #openstack-infra | 04:49 | |
| *** aeng has joined #openstack-infra | 04:49 | |
| *** ianychoi has quit IRC | 04:50 | |
| *** claudiub has quit IRC | 04:51 | |
| *** markvoelker has joined #openstack-infra | 04:53 | |
| *** ianychoi__ has joined #openstack-infra | 04:54 | |
| *** ykarel|away has quit IRC | 04:56 | |
| *** ykarel|off has quit IRC | 04:56 | |
| *** ianychoi_ has quit IRC | 04:56 | |
| *** markvoelker has quit IRC | 04:57 | |
| AJaeger | ianw, project-config-cores: care to review https://review.openstack.org/513493 and https://review.openstack.org/513497 to fix translatoins and specs publishing, please? | 04:58 |
| *** dimak has quit IRC | 04:59 | |
| *** dimak has joined #openstack-infra | 04:59 | |
| *** markvoelker has joined #openstack-infra | 05:02 | |
| *** huanxie has joined #openstack-infra | 05:05 | |
| *** ianychoi has joined #openstack-infra | 05:07 | |
| *** ianychoi__ has quit IRC | 05:10 | |
| *** baoli has joined #openstack-infra | 05:16 | |
| *** jbadiapa has joined #openstack-infra | 05:16 | |
| *** ijw has joined #openstack-infra | 05:18 | |
| *** baoli has quit IRC | 05:21 | |
| *** armaan has joined #openstack-infra | 05:23 | |
| *** ijw has quit IRC | 05:27 | |
| *** edmondsw has joined #openstack-infra | 05:32 | |
| *** markvoelker has quit IRC | 05:35 | |
| *** edmondsw has quit IRC | 05:36 | |
| *** markvoelker has joined #openstack-infra | 05:41 | |
| Jeffrey4l | when the same nodeset is defined in bother master and pike branch, zuul will complain that the nodeset is already defined. is this a bug? | 05:44 |
| *** apetrich has quit IRC | 05:44 | |
| *** markvoelker has quit IRC | 05:45 | |
| *** nikhil has quit IRC | 05:46 | |
| *** apetrich has joined #openstack-infra | 05:46 | |
| *** spectr has joined #openstack-infra | 05:54 | |
| *** thorst has joined #openstack-infra | 05:57 | |
| *** ggillies has quit IRC | 05:57 | |
| *** aeng has quit IRC | 05:58 | |
| *** thorst has quit IRC | 06:02 | |
| *** armax has quit IRC | 06:03 | |
| dirk | AJaeger: https://review.openstack.org/#/c/512487/ | 06:05 |
| dirk | AJaeger: and https://review.openstack.org/#/c/512901/ pretty please.. | 06:06 |
| openstackgerrit | Merged openstack-dev/pbr master: Discover Distribution through the class hierarchy https://review.openstack.org/399188 | 06:10 |
| AJaeger | morning, dirk - will do... | 06:10 |
| openstackgerrit | Merged openstack-dev/pbr master: Remove unnecessary 'if True' https://review.openstack.org/510806 | 06:10 |
| AJaeger | dirk: LGTM | 06:11 |
| *** salv-orlando has quit IRC | 06:13 | |
| *** salv-orlando has joined #openstack-infra | 06:14 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Requirements propose_updates runs master only https://review.openstack.org/513618 | 06:17 |
| *** andreas_s has joined #openstack-infra | 06:17 | |
| *** salv-orlando has quit IRC | 06:18 | |
| *** mandre is now known as mandre_afk | 06:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Run gitdm periodic job only on master https://review.openstack.org/513621 | 06:22 |
| dirk | AJaeger: good morning, thanks! | 06:23 |
| *** markvoelker has joined #openstack-infra | 06:32 | |
| *** psachin has quit IRC | 06:33 | |
| *** cuongnv has quit IRC | 06:37 | |
| *** claudiub has joined #openstack-infra | 06:37 | |
| *** kukacz has quit IRC | 06:41 | |
| *** kukacz has joined #openstack-infra | 06:43 | |
| *** huanxie has quit IRC | 06:44 | |
| *** florianf has joined #openstack-infra | 06:47 | |
| *** huanxie has joined #openstack-infra | 06:47 | |
| *** armaan has quit IRC | 06:54 | |
| *** armaan has joined #openstack-infra | 06:54 | |
| *** yolanda has joined #openstack-infra | 07:00 | |
| *** armaan has quit IRC | 07:02 | |
| *** tesseract has joined #openstack-infra | 07:03 | |
| *** tmorin has joined #openstack-infra | 07:04 | |
| *** ociuhandu has joined #openstack-infra | 07:07 | |
| *** jpich has joined #openstack-infra | 07:09 | |
| *** stakeda has joined #openstack-infra | 07:11 | |
| *** ociuhandu has quit IRC | 07:12 | |
| *** salv-orlando has joined #openstack-infra | 07:14 | |
| *** e0ne has joined #openstack-infra | 07:15 | |
| *** dizquierdo has joined #openstack-infra | 07:18 | |
| *** huanxie has quit IRC | 07:19 | |
| *** hashar has joined #openstack-infra | 07:19 | |
| *** salv-orlando has quit IRC | 07:19 | |
| *** aviau has quit IRC | 07:20 | |
| *** edmondsw has joined #openstack-infra | 07:20 | |
| *** aviau has joined #openstack-infra | 07:20 | |
| *** huanxie has joined #openstack-infra | 07:20 | |
| frickler | wasn't someone fixing jobs recently that ran on dedicated nodes earlier and are lacking python-yaml now? this is another one http://logs.openstack.org/periodic/git.openstack.org/openstack-infra/project-config/master/propose-project-config-update/3166d06/job-output.txt.gz#_2017-10-20_06_19_51_058686 | 07:20 |
| openstackgerrit | Merged openstack-infra/project-config master: fix specs publishing https://review.openstack.org/513493 | 07:24 |
| frickler | AJaeger: I'd say lets verify ianw's suggestion on https://review.openstack.org/513497 before changing it everywhere | 07:24 |
| *** edmondsw has quit IRC | 07:24 | |
| frickler | ah, no the yaml issue is still on the etherpad | 07:25 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Vitrage https://review.openstack.org/510431 | 07:27 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Run bindep for translation jobs https://review.openstack.org/513497 | 07:27 |
| AJaeger | frickler: updated ^ | 07:28 |
| AJaeger | frickler: haven't found anything to quickly copy for python-yaml - patches welcome ;) | 07:28 |
| *** ccamacho has joined #openstack-infra | 07:31 | |
| *** cuongnv has joined #openstack-infra | 07:33 | |
| frickler | AJaeger: playbooks/proposal/pre is already included in pre-run for the parent in your translation jobs, isn't that cumulative? i.e. wouldn't just adding playbooks/translation/pre be enough? | 07:35 |
| *** jpena|off is now known as jpena | 07:41 | |
| *** amoralej|off is now known as amoralej | 07:45 | |
| *** ralonsoh has joined #openstack-infra | 07:46 | |
| *** Hal has joined #openstack-infra | 07:50 | |
| *** Hal is now known as Guest6618 | 07:50 | |
| openstackgerrit | Nguyen Van Trung proposed openstack-infra/system-config master: Move to Zuulv3 link to check status https://review.openstack.org/513641 | 07:54 |
| openstackgerrit | Nguyen Van Trung proposed openstack-infra/zuul master: Move to Zuulv3 link to check status https://review.openstack.org/513648 | 07:55 |
| *** thorst has joined #openstack-infra | 07:58 | |
| *** martinkopec has joined #openstack-infra | 07:58 | |
| *** salv-orlando has joined #openstack-infra | 08:00 | |
| evrardjp | may I have an expert eye on this review: to give me an insight on why my new in-tree jobs don't appear in the tests? Did I do something wrong with my definition? https://review.openstack.org/#/c/513406 | 08:00 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove migrated legacy-octavia-dashboard-* https://review.openstack.org/513566 | 08:01 |
| *** tosky has joined #openstack-infra | 08:02 | |
| *** thorst has quit IRC | 08:03 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Change http to https link https://review.openstack.org/512640 | 08:03 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy jobs in Karbor https://review.openstack.org/511433 | 08:03 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix releasenotes job for VPNaaS https://review.openstack.org/513598 | 08:03 |
| openstackgerrit | Merged openstack-infra/project-config master: add required-projects for the release-openstack-python jobs https://review.openstack.org/513507 | 08:03 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix contributor guide post run job location https://review.openstack.org/513542 | 08:03 |
| openstackgerrit | Merged openstack-infra/project-config master: sahara-image-elements: remove the migrated jobs https://review.openstack.org/513484 | 08:03 |
| evrardjp | oh never mind. | 08:03 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add the correct branch overrides to the networking-cisco jobs https://review.openstack.org/513339 | 08:06 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Install PyYAML for proposal jobs https://review.openstack.org/513650 | 08:16 |
| frickler | AJaeger: ^^ not very creative, but I think it should work | 08:17 |
| *** martinkopec has quit IRC | 08:19 | |
| *** claudiub|2 has joined #openstack-infra | 08:19 | |
| *** baoli has joined #openstack-infra | 08:19 | |
| *** claudiub has quit IRC | 08:22 | |
| *** dtantsur|afk is now known as dtantsur | 08:23 | |
| *** baoli has quit IRC | 08:23 | |
| *** mandre_afk is now known as mandre | 08:25 | |
| *** jamesmcarthur has joined #openstack-infra | 08:26 | |
| *** makowals has joined #openstack-infra | 08:29 | |
| *** jamesmcarthur has quit IRC | 08:30 | |
| hwoarang | good morning. I have an issue with a proposal and zuulv3 | 08:31 |
| *** lucas-afk is now known as lucasagomes | 08:31 | |
| openstackgerrit | Tetsuro Nakamura proposed openstack-infra/project-config master: Follow up change for networking-spp creation https://review.openstack.org/512536 | 08:31 |
| hwoarang | i am looking at this log http://logs.openstack.org/ac/ac930796d66e7fdd952477df03d75c4024c8f5a2/post/propose-updates/8efb7bb/job-output.txt.gz and it seems that BRANCH ends up being empty because "git branch -a | grep -q "^ remotes/origin/$ZUUL_REFNAME$" returns empty string | 08:31 |
| hwoarang | and as such nothing happens in the propose_update.sh script | 08:32 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs from networking-onos https://review.openstack.org/513657 | 08:33 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs from networking-onos https://review.openstack.org/513658 | 08:34 |
| *** martinkopec has joined #openstack-infra | 08:35 | |
| *** dizquierdo has quit IRC | 08:39 | |
| *** derekh has joined #openstack-infra | 08:42 | |
| *** electrofelix has joined #openstack-infra | 08:42 | |
| AJaeger | frickler: thanks, commented | 08:44 |
| *** mdrabe has quit IRC | 08:44 | |
| evrardjp | hwoarang: oh maybe the $ZUUL_REFNAME was fixed by AJaeger | 08:44 |
| evrardjp | and this job has run before the fix merged | 08:44 |
| evrardjp | https://github.com/openstack-infra/project-config/commit/bf61f85cf83487bb9a65babd38f7fb44a7b68328 | 08:45 |
| evrardjp | oh maybe not | 08:46 |
| *** huanxie has quit IRC | 08:46 | |
| evrardjp | I just remembered refname ... | 08:46 |
| *** huanxie has joined #openstack-infra | 08:48 | |
| *** mdrabe has joined #openstack-infra | 08:51 | |
| *** gus has quit IRC | 08:57 | |
| *** jamielennox has quit IRC | 08:57 | |
| *** claudiub|2 has quit IRC | 08:57 | |
| *** efoley has joined #openstack-infra | 08:58 | |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/openstack-zuul-jobs master: Remove ironic legacy jobs https://review.openstack.org/511264 | 08:59 |
| *** jamielennox has joined #openstack-infra | 09:02 | |
| *** gus has joined #openstack-infra | 09:03 | |
| AJaeger | hwoarang: do you have a more recent log file? Let's look at that one instead, please | 09:03 |
| *** thorst has joined #openstack-infra | 09:03 | |
| *** bhavik1 has joined #openstack-infra | 09:05 | |
| tosky | AJaeger: hi, do you know if status.openstack.org/zuul will be still considered the main URL for zuul or should we use zuulv3.openstack.org? | 09:06 |
| tosky | asking because I received this: https://review.openstack.org/#/c/513643/ | 09:06 |
| tosky | and I'm not sure that it makes change to replace the URL | 09:06 |
| *** bhavik1 has quit IRC | 09:06 | |
| *** bhavik1 has joined #openstack-infra | 09:07 | |
| *** edmondsw has joined #openstack-infra | 09:08 | |
| *** thorst has quit IRC | 09:08 | |
| *** rcernin has joined #openstack-infra | 09:12 | |
| *** edmondsw has quit IRC | 09:13 | |
| *** bhavik1 has quit IRC | 09:13 | |
| frickler | tosky: yes, zuulv3.o.o is only temporary. I think it should get moved back next week when zuul v2 is thrown out completely | 09:15 |
| tosky | frickler: so I guess that all changes can be blocked: https://review.openstack.org/#/q/topic:improve-zuul-link+(status:open+OR+status:merged) | 09:17 |
| *** baoli has joined #openstack-infra | 09:20 | |
| frickler | tosky: I think yes, but maybe wait for independent confirmation | 09:22 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/openstack-zuul-jobs master: Remove networking-ovn legacy jobs https://review.openstack.org/513675 | 09:22 |
| *** s-shiono has quit IRC | 09:22 | |
| tosky | frickler: I asked to hold the patches for now | 09:22 |
| frickler | https://etherpad.openstack.org/p/zuulv3-migration-faq says "http://status.openstack.org/zuul/ will likely be updated to feed from the new Zuul v3 data in the near future." so except the "likely" which makes it a bit vague, it is documented there | 09:23 |
| *** baoli has quit IRC | 09:24 | |
| frickler | AJaeger: hwoarang: assuming that I located the correct version of the script, we seem to need to drop the "remote/" here https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/propose_update.sh#n127 | 09:26 |
| *** shardy has joined #openstack-infra | 09:26 | |
| *** dizquierdo has joined #openstack-infra | 09:28 | |
| *** baoli has joined #openstack-infra | 09:30 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 09:32 | |
| *** kiennt26 has quit IRC | 09:32 | |
| *** kiennt26 has joined #openstack-infra | 09:33 | |
| *** Kevin_Zheng has joined #openstack-infra | 09:33 | |
| hwoarang | frickler: yeah that's the script | 09:34 |
| *** baoli has quit IRC | 09:35 | |
| hwoarang | frickler: i presume this prevents all propose jobs to work properly and not just ours right? | 09:35 |
| *** yolanda has quit IRC | 09:36 | |
| AJaeger | tosky: interesting - that document mentions Jenkins which is really wrong and should be changed... | 09:36 |
| frickler | hwoarang: I'd guess so, yes | 09:36 |
| hwoarang | AJaeger: sorry i have no recent log file right now | 09:37 |
| hwoarang | frickler: ok thank you | 09:37 |
| AJaeger | hwoarang: we work on other proposal jobs right now, so, let's wait for next run... | 09:37 |
| AJaeger | frickler: yes, that's the fix... | 09:37 |
| tosky | AJaeger: right, but apart from that, the question was more about the change of the zuul URL | 09:38 |
| AJaeger | tosky: frickler answered that - I agree | 09:38 |
| tosky | yep :) | 09:39 |
| hwoarang | AJaeger: OK then | 09:39 |
| AJaeger | hwoarang: please keep an eye on it - we know the proposal jobs are broken and fix them step by step... | 09:41 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Fix branch check for proposal script https://review.openstack.org/513695 | 09:45 |
| *** pcaruana has joined #openstack-infra | 09:46 | |
| *** makowals has quit IRC | 09:49 | |
| *** martinkopec has quit IRC | 09:50 | |
| *** LindaWang has quit IRC | 09:54 | |
| *** arxcruz is now known as arxcruz|off | 09:54 | |
| mordred | AJaeger: morning! I'm around for a minute or two - anything I should prioritize reviewing? | 10:02 |
| *** cuongnv has quit IRC | 10:03 | |
| *** erlon has joined #openstack-infra | 10:04 | |
| mordred | andreaf: I've got an interesting failure on my attempt to add new-style devstack jobs to shade ... | 10:05 |
| andreaf | mordred: oh... link? | 10:06 |
| mordred | andreaf: http://logs.openstack.org/65/500365/30/check/shade-functional-devstack/24dd124/ - I'm getting a sudo failure when it's trying to do the systemd log collection (otherwise the job seems to have run fine) | 10:06 |
| mordred | andreaf: but I don't see anywhere that revoke-sudo has been run or anything lke that | 10:06 |
| mordred | andreaf: http://logs.openstack.org/65/500365/30/check/shade-functional-devstack/24dd124/ara/result/ae91138d-7be5-4fa8-98fd-8c600b3cf893/ | 10:07 |
| mordred | a little easier to see there ... | 10:07 |
| dtantsur | folks, could you please clarify https://docs.openstack.org/infra/manual/zuulv3.html#stable-branches: should the job definitions themselves be backported or not? | 10:07 |
| vdrok | morning all! yeah, we have the following in master https://review.openstack.org/511267 | 10:08 |
| dtantsur | I don't quite get how "The Zuul config on stable branches doesn’t need everything on the master branch – the jobs defined in the master branch will be available in any branch." plays with "If you have playbooks or roles included on the master branch, backport these as well." | 10:08 |
| andreaf | mordred: that's strange ... revoke sudo we do for user stack only I believe, and the journal export is done as zuul | 10:08 |
| mordred | andreaf: we do revoke sudo for the stack user? | 10:09 |
| frickler | andreaf: mordred: I do see "revoke sudo for zuul" in the last pre step | 10:09 |
| mordred | frickler: OH! | 10:09 |
| mordred | frickler: well that would explain why sudo is being revoked | 10:09 |
| andreaf | frickler: yeah that must be it | 10:09 |
| hwoarang | I have a question about the upload-logs role. We store all logs in $root/logs but it seems the upload-logs role doesn't upload any of these files to the log server. The 'base' job suggests that logs should be copied by 'us' in logs/ and the post job will upload them. Is that true? | 10:09 |
| frickler | seems to be within shade's devstack/pre.yaml | 10:09 |
| mordred | frickler: yup. there it is | 10:10 |
| mordred | wow. thank you | 10:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 10:10 |
| mordred | frickler, andreaf: I expect that to be green now ^^ | 10:10 |
| mordred | frickler: I spent almost an hour trying to figure out where the revoke sudo was happening :) | 10:11 |
| frickler | mordred: yw ;) | 10:11 |
| * mordred does a little dance - hopes the jobs go green so he can switch to new-style devstack jobs | 10:12 | |
| andreaf | mordred: regarding user stack I believe we do not yet have the revoke sudo for stack in the new style job | 10:12 |
| *** kiennt26 has quit IRC | 10:12 | |
| AJaeger | mordred: https://review.openstack.org/513497 and https://review.openstack.org/513650 | 10:18 |
| AJaeger | mordred: otherwise: the javascript PTI jobs wait for you but that needs more than a minute or two... | 10:18 |
| AJaeger | dtantsur: you don't need to backport the job template. | 10:19 |
| AJaeger | dtantsur: just try it ;) | 10:19 |
| AJaeger | dtantsur: happy to review a change and comment | 10:19 |
| dtantsur | vdrok: ^^^ | 10:19 |
| dtantsur | thanks AJaeger! | 10:19 |
| vdrok | gotcha | 10:19 |
| *** claudiub|2 has joined #openstack-infra | 10:20 | |
| *** liujiong has quit IRC | 10:20 | |
| AJaeger | dtantsur, vdrok : Changes for the manual are always welcome to make it easier for the next one... | 10:20 |
| *** annp has quit IRC | 10:23 | |
| *** gmann is now known as gmann_afk | 10:24 | |
| *** sambetts|afk is now known as sambetts | 10:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix branch check for proposal script https://review.openstack.org/513695 | 10:26 |
| *** stakeda has quit IRC | 10:27 | |
| hwoarang | does anyone know if there is already a proper way to export zuul.executor.* infomration as variables? We need to use the zuul.executor.work_root in a shell script | 10:28 |
| smcginnis | frickler: I think we went with bindep for pyyaml installation because requirements.txt was not processed for the failing jobs, but not sure if there was another, possibly more relevant, reason for doing a package installation. | 10:28 |
| hwoarang | otherwise we can export that in our in-project playbooks i suppose. but if there is a role or something that does that it's better to re-use it | 10:28 |
| *** baoli has joined #openstack-infra | 10:30 | |
| *** pbourke has quit IRC | 10:31 | |
| *** boden has joined #openstack-infra | 10:32 | |
| AJaeger | hwoarang: your playbook can define an environment variable and use that... | 10:32 |
| frickler | smcginnis: I looked at the bindep for project-config and that contains things that seem overkill for this particular set of tools being run, so I didn't want to use that variant. waiting for feedback from infra-root whether they prefer pkg vs. pip | 10:32 |
| *** pbourke has joined #openstack-infra | 10:33 | |
| AJaeger | hwoarang: or pass it in as http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/proposal/propose-updates.yaml#n5 does | 10:33 |
| smcginnis | frickler: I'm curious now too if there is a better approach. | 10:33 |
| *** baoli has quit IRC | 10:35 | |
| *** yamamoto has quit IRC | 10:40 | |
| *** yamamoto has joined #openstack-infra | 10:41 | |
| pabelanger | frickler: smcginnis: adding pyyaml to bindep should be fine for now, until we refactor the proposal job into native ansible playbook. You'd need to update the job to then use the bindep-role for proposal pre playbook | 10:43 |
| *** ldnunes has joined #openstack-infra | 10:43 | |
| smcginnis | pabelanger: Long term, is it better to get it working with pip? | 10:43 |
| smcginnis | e.g. this approach: https://review.openstack.org/513650 | 10:44 |
| *** namnh has quit IRC | 10:44 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: neutron-lbaas-dashboard jobs require horizon https://review.openstack.org/513562 | 10:45 |
| *** yamamoto has quit IRC | 10:45 | |
| pabelanger | smcginnis: well, I'd rather use something like requirements.txt or test-requirements.txt for installing pip things, we already do a good job with other tooling to support that. Having playbooks that just pip install something from a random location, would get confusing over time, at least for me | 10:45 |
| *** yamamoto has joined #openstack-infra | 10:45 | |
| *** yamamoto has quit IRC | 10:46 | |
| pabelanger | smcginnis: then either use tox or virtualenv to install them, since we don't really need to be root | 10:46 |
| smcginnis | That may have been the reason we went went with bindep for the requirements tools, since normal requirements.txt processing doesn't happen in some cases. | 10:46 |
| *** boden_ has joined #openstack-infra | 10:47 | |
| *** boden has quit IRC | 10:48 | |
| *** boden_ is now known as boden | 10:48 | |
| AJaeger | pabelanger, could you review https://review.openstack.org/513497 and https://review.openstack.org/513650 , please? | 10:49 |
| AJaeger | pabelanger, frickler, should we merge 513650 and the pyyaml change? So, do it in pre.rule? I can rework 513650... | 10:50 |
| AJaeger | I meant I can merge 513650 anf 513497 - let me do that now... | 10:51 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Run bindep for proposal jobs https://review.openstack.org/513497 | 10:54 |
| AJaeger | frickler, smcginnis, pabelanger ^ | 10:54 |
| hwoarang | AJaeger: ok thank you | 10:54 |
| vdrok | AJaeger: could you take a look at https://review.openstack.org/513696? zuul still complains about playbooks | 10:59 |
| *** daidv_ has quit IRC | 10:59 | |
| *** yamamoto has joined #openstack-infra | 10:59 | |
| AJaeger | pabelanger: could you check https://review.openstack.org/#/c/513509/ as well, please? | 11:00 |
| AJaeger | vdrok: will do | 11:00 |
| vdrok | thank you! | 11:00 |
| *** yamamoto has quit IRC | 11:01 | |
| AJaeger | vdrok: yes, you need to backport any playbooks. | 11:02 |
| AJaeger | Just not the job definitions itself. So, zuul.yaml is fine (just can be done better, see my comment) | 11:02 |
| *** ociuhandu has joined #openstack-infra | 11:03 | |
| vdrok | AJaeger: aha, so no need for legacy-ironic-jobs.yaml, but everything else should be present. gotcha, thank you | 11:03 |
| *** lucasagomes is now known as lucas-hungry | 11:03 | |
| AJaeger | vdrok: yes | 11:04 |
| *** thorst has joined #openstack-infra | 11:04 | |
| pabelanger | AJaeger: yah, lets get mordred to review that one, since he original wrote it | 11:05 |
| AJaeger | mordred: could you review https://review.openstack.org/#/c/513509/ , please? | 11:06 |
| *** ociuhandu has quit IRC | 11:07 | |
| *** thorst has quit IRC | 11:09 | |
| smcginnis | dhellmann, fungi: Looks like this is the change in behavior: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/release-tools/update_constraints.sh#n102 | 11:09 |
| *** nicolasbock has joined #openstack-infra | 11:09 | |
| *** Qiming_ has quit IRC | 11:11 | |
| *** Qiming has joined #openstack-infra | 11:13 | |
| odyssey4me | is there a variable available to job playbooks which tells us that the job succeeded or failed? I want to use it when determining which logs to collect in the post stage | 11:13 |
| *** baoli has joined #openstack-infra | 11:14 | |
| *** baoli has quit IRC | 11:15 | |
| *** yamamoto has joined #openstack-infra | 11:15 | |
| pabelanger | odyssey4me: zuul_success https://docs.openstack.org/infra/zuul/feature/zuulv3/user/jobs.html?highlight=zuul%20success#var-zuul_success | 11:15 |
| odyssey4me | ah, nice thanks pabelanger | 11:16 |
| pabelanger | np | 11:16 |
| openstackgerrit | Merged openstack-infra/project-config master: Run bindep for proposal jobs https://review.openstack.org/513497 | 11:16 |
| *** edmondsw has joined #openstack-infra | 11:18 | |
| mordred | pabelanger, AJaeger: looking | 11:19 |
| mordred | AJaeger: oh - that looks great! +3 | 11:19 |
| mordred | dmsimard: ^^ good job | 11:19 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Default clone_repo to master if branch not found https://review.openstack.org/513710 | 11:20 |
| AJaeger | mordred: thanks for reviewing... | 11:23 |
| AJaeger | smcginnis: that was always the default with zuul-cloner | 11:24 |
| AJaeger | smcginnis: ah, you see it does not anymore - argh ;( | 11:24 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Refactor fact configuration for service_type_data https://review.openstack.org/513509 | 11:24 |
| smcginnis | AJaeger: Yeah, looks like that's why this is failing now: http://logs.openstack.org/dd/dd0bb2d44053e11d9d4a4775b3c71f2c0889dc2a/release/propose-update-constraints/953c933/job-output.txt.gz#_2017-10-19_23_27_20_561841 | 11:25 |
| *** makowals has joined #openstack-infra | 11:26 | |
| mordred | smcginnis: I think that's still going to fail | 11:28 |
| smcginnis | mordred: Oops, what'd I miss? | 11:28 |
| mordred | smcginnis: hang on - re-reading ... | 11:29 |
| mordred | smcginnis: k. nevermind. I'm wrong | 11:30 |
| smcginnis | K, I don't doubt I'm possibly missing something, so please do let me know if you see any issue. | 11:31 |
| odyssey4me | is there an example of a cross-repo job available somewhere for me to look at? | 11:32 |
| odyssey4me | I'm thinking something along the lines of a job which is initated by one repo, but uses the tests from another | 11:32 |
| mordred | yup. one sec | 11:33 |
| andreas_s | Hi, the build-openstack-sphinx-docs job in our project is failing: Log says I should add neutron to the required projects list. What would be the right approach to fix this? cause the job is a generic openstack job... | 11:33 |
| andreas_s | an example: https://review.openstack.org/#/c/513683/ | 11:34 |
| mordred | odyssey4me: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n200 is a job definition for a job that runs zuul's tox py35 tests - and we trigger it on patches to openstack-infra/zuul-jobs | 11:34 |
| pabelanger | andreas_s: you likely want to switch to the publish-openstack-sphinx-docs-neutron project-template, that will setup neutron properly | 11:35 |
| andreas_s | pabelanger: do you suggest to fix things in project-config first, or should I start moving all job definitions to our repo first? | 11:36 |
| mordred | odyssey4me: the main things to know are that on the job the 'required-projects' tells zuul which projects to clone/prepare when the job is triggered | 11:36 |
| odyssey4me | mordred yep, got that | 11:36 |
| pabelanger | andreas_s: you cand first fix it in project config, it will actually be openstack-python-jobs-neutron, then once working, migrate into your repo | 11:37 |
| AJaeger | andreas_s: you should keep that job in project-config, only convert legacy jobs over but not the standard PTI required ones | 11:37 |
| mordred | cool. so then many of the job types (like in this case the tox job) take an argument of "zuul_work_dir" which tells it which directory to operate in - that variable tends to default to 'zuul.project.src_dir' - which is the src dir of the project that triggered the job | 11:37 |
| andreas_s | AJaeger: understood, thx | 11:38 |
| mordred | odyssey4me: ^^ so if you want to trigger by one project but run tests from another project, overriding zuul_work_dir is likely the main thing you need to do | 11:38 |
| andreas_s | pabelanger: ok, thx. let me have a look at the job you proposed... | 11:38 |
| mordred | odyssey4me: alternately, if the job content isn't using one of the existing base jobs but is custom for your projects - you can just refernece {{ ansible_user_dir }}/src/git.openstack.org/openstack/foo as a location to do things | 11:39 |
| odyssey4me | mordred ok, I'm doing this right now, but it feels a bit clunky: https://review.openstack.org/513457 & https://review.openstack.org/513453 | 11:39 |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config master: increase timeout for ec2-api rally job https://review.openstack.org/513713 | 11:40 |
| odyssey4me | mordred so that's basically your second suggestion | 11:40 |
| openstackgerrit | Merged openstack-infra/project-config master: Default clone_repo to master if branch not found https://review.openstack.org/513710 | 11:43 |
| mordred | odyssey4me: that looks correct ... and I can't immediatley come up with a suggestion for a better structure | 11:43 |
| *** Kevin_Zheng has quit IRC | 11:43 | |
| odyssey4me | mordred alright, thanks for looking into it | 11:44 |
| *** Qiming has quit IRC | 11:44 | |
| odyssey4me | I'm sure that in time we'll get a bit smarter - but for now we're just slowly evolving. | 11:44 |
| mordred | odyssey4me: for having a base job in one repo, tests in another repo and triggering that on a third repo is probably always going to be partially mind-bending :) | 11:44 |
| odyssey4me | haha, yeah | 11:44 |
| mordred | odyssey4me: but I think what you're doing there is cool and stuff | 11:44 |
| mordred | odyssey4me: and yah - it's definitely an iterative learning process for all of us | 11:45 |
| *** Qiming has joined #openstack-infra | 11:45 | |
| odyssey4me | mordred to improve my understanding - what does the 'required-projects' do which is better than us just git cloning in a task? | 11:47 |
| odyssey4me | in this case I'm referring to a situation where there is no need for zuul to work out dependent patches and refs | 11:47 |
| odyssey4me | I literally just need a repo on a disk in it's current committed state | 11:47 |
| mordred | odyssey4me: yah - in that case required-projects does not do anything for you | 11:48 |
| odyssey4me | well, I was wondering whether it uses something like synchronise or something other than git? | 11:48 |
| mordred | odyssey4me: required-projects is better when you need zuul to be able to work out dependent patches and refs | 11:48 |
| odyssey4me | we get quite a few failure when using a normal git clone to get repositories which zuul hasn't put there for us already - this is why I'm asking | 11:49 |
| mordred | odyssey4me: WELL - yah - it does the cloning on the executor (using caches there) and then it uses synchronize to rsync the repos to the build nodes | 11:49 |
| odyssey4me | ah, yes - so that would be more reliable | 11:49 |
| mordred | odyssey4me: at the moment that also involves making use of the cached versions of the repos on the build node too | 11:49 |
| *** rosmaita has joined #openstack-infra | 11:49 | |
| odyssey4me | ok, so now I wonder if I can use a filter to extract the list of required projects from a file in the current repo | 11:50 |
| mordred | BUT ... there's definitely a trade-off in your case potentially if you have to do awkward job construction to get the required-projects and you're not making use of dependencies | 11:50 |
| *** yamamoto has quit IRC | 11:50 | |
| odyssey4me | may as well give it a go :) | 11:50 |
| mordred | odyssey4me: that is not possible - the required-projects is zuul config and happens before ansible | 11:50 |
| odyssey4me | dammit | 11:50 |
| *** thorst has joined #openstack-infra | 11:52 | |
| mordred | odyssey4me: that said - you could probably write a script to generate required-projects from a file in a repo and just run that on all your projects and splat out a pile of generated yaml - then add a check somewhere to make sure if the file is updated that the required-projects list matches | 11:53 |
| odyssey4me | mordred yeah, that sucks a bit - but it's an option | 11:54 |
| *** huanxie has quit IRC | 11:54 | |
| odyssey4me | basically the options are - specify all repositories as required projects for all our jobs... or implement all the git cloning necessary in the pre stage so that it retries a few times | 11:55 |
| mordred | odyssey4me: there's also an idea that has come up of allowing setting required-projects on project instead of a job | 11:55 |
| odyssey4me | the second is arguably not very nice to the git servers, so less good citizen | 11:55 |
| mordred | but that might not help you in this case | 11:55 |
| mordred | odyssey4me: yah - I think tripleo and puppet-openstack each just have a giant list of repos in the required-projects list of their base job | 11:56 |
| *** dizquierdo has quit IRC | 11:56 | |
| * odyssey4me considers adding something strong to his coffee | 11:56 | |
| mordred | odyssey4me: ooh, I could do with some of that | 11:57 |
| odyssey4me | I imagine that required-projects can't take wildcards or regex right? | 11:57 |
| *** jpena is now known as jpena|lunch | 11:57 | |
| chandankumar | clarkb: review issue got fixed | 11:58 |
| chandankumar | clarkb: please update this review group https://review.openstack.org/#/admin/groups/1842,members | 11:59 |
| *** dizquierdo has joined #openstack-infra | 11:59 | |
| *** dprince has joined #openstack-infra | 12:00 | |
| *** rhallisey has joined #openstack-infra | 12:00 | |
| odyssey4me | jeblair it would seem that zuul is behaving far better today than the previous few days - was something done to improve memory usage? | 12:03 |
| odyssey4me | I ask because we're still pushing ~5 job changes - then waiting for them to fully process, and it's quite painful. | 12:04 |
| *** bobh has joined #openstack-infra | 12:04 | |
| *** armaan has joined #openstack-infra | 12:04 | |
| tobiash | odyssey4me: yes: https://review.openstack.org/#/c/513441/ | 12:04 |
| tobiash | memory usage also looks far better now: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all | 12:05 |
| *** bobh has quit IRC | 12:06 | |
| odyssey4me | tobiash oh nice - thanks, it's made a *massive* difference | 12:06 |
| *** andreas_s has quit IRC | 12:08 | |
| *** andreas_s has joined #openstack-infra | 12:08 | |
| *** dbecker has joined #openstack-infra | 12:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Support upper-constraints in tox-siblings https://review.openstack.org/513199 | 12:11 |
| *** bobh has joined #openstack-infra | 12:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Support upper-constraints in tox-siblings https://review.openstack.org/513199 | 12:14 |
| mordred | frickler, AJaeger, fungi: ^^ I think that's in-line with the review comments now | 12:14 |
| *** rcernin has quit IRC | 12:20 | |
| *** andreas_s has quit IRC | 12:22 | |
| *** andreas_s has joined #openstack-infra | 12:27 | |
| *** lucas-hungry is now known as lucasagomes | 12:30 | |
| *** pcaruana has quit IRC | 12:31 | |
| *** andreas_s has quit IRC | 12:32 | |
| *** andreas_s has joined #openstack-infra | 12:32 | |
| openstackgerrit | Adrian Czarnecki proposed openstack-infra/project-config master: Remove legacy job for monasca-api https://review.openstack.org/513722 | 12:34 |
| odyssey4me | hmm, how do I do the equivalent of the devstack log collection now? | 12:37 |
| *** bobh has quit IRC | 12:38 | |
| odyssey4me | our log collection appears to not be happening as it used to with our new jobs, so I'm obviously missing something | 12:38 |
| odyssey4me | we used to be able to collect logs from ${WORKDING_DIR}/logs | 12:38 |
| openstackgerrit | Adrian Czarnecki proposed openstack-infra/project-config master: Remove legacy job for monasca-api https://review.openstack.org/513722 | 12:38 |
| *** mat128 has joined #openstack-infra | 12:39 | |
| *** andreas_s has quit IRC | 12:40 | |
| *** andreas_s has joined #openstack-infra | 12:41 | |
| *** andreas_s has quit IRC | 12:41 | |
| *** ramishra has quit IRC | 12:42 | |
| *** andreas_s has joined #openstack-infra | 12:42 | |
| fungi | infra-root: project-config-core: anyone else... the power company just rang my doorbell to let me know they're replacing the power pole at the street feeding my house, and project that i'll be without electricity for the next ~3 hours | 12:42 |
| fungi | so, er, don't expect me for a while | 12:42 |
| smcginnis | fungi: Wow, that's all the notice you got? | 12:45 |
| fungi | yup | 12:46 |
| fungi | i think they assume nobody lives in these houses during the off-season | 12:47 |
| *** priteau has joined #openstack-infra | 12:47 | |
| fungi | granted, we've seen them working up the street doing the other poles, so it was mostly inevitable | 12:47 |
| *** gcb has quit IRC | 12:47 | |
| *** LindaWang has joined #openstack-infra | 12:48 | |
| *** rossella_s has quit IRC | 12:49 | |
| *** jamesmcarthur has joined #openstack-infra | 12:49 | |
| *** andreas_s has quit IRC | 12:50 | |
| *** yamamoto has joined #openstack-infra | 12:50 | |
| smcginnis | fungi: Not sure what pulls from where, but my clone_repo changed merged in project-config, but there is another one in requirements. Do we need that requirements one landed before we can try releasing again? | 12:51 |
| *** rossella_s has joined #openstack-infra | 12:51 | |
| *** mriedem has joined #openstack-infra | 12:53 | |
| *** gouthamr has joined #openstack-infra | 12:54 | |
| *** huanxie has joined #openstack-infra | 12:54 | |
| *** andreas_s has joined #openstack-infra | 12:55 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Fix Puppet integration jobs to run on master only https://review.openstack.org/513731 | 12:55 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Use playbook from tripleo-quickstart-extras for OVB https://review.openstack.org/513508 | 12:57 |
| *** iyamahat has joined #openstack-infra | 12:57 | |
| *** yamamoto has quit IRC | 12:57 | |
| *** yamamoto has joined #openstack-infra | 12:57 | |
| *** iyamahat_ has joined #openstack-infra | 12:58 | |
| mnaser | anyone ever seen this type of behaviour before.. tempest test failing.. | 13:00 |
| mnaser | Details: {u'message': u'Unable to associate floating IP 172.24.5.12 to fixed IP 10.100.0.5 for instance 40a2d904-1c2e-460e-8084-8e7ac3344a4d. Error: Request to https://127.0.0.1:9696/v2.0/ports?tenant_id=1b890b5880c74e338624e417ce45a3b4&device_id=40a2d904-1c2e-460e-8084-8e7ac3344a4d timed out', u'code': 400} | 13:00 |
| mnaser | asking here first, then maybe ill try -qa .. we had to go non-voting for puppet jobs on xenial (this seems to be a xenial only issue) | 13:00 |
| *** jpena|lunch is now known as jpena | 13:01 | |
| *** rossella_s has quit IRC | 13:01 | |
| *** hashar has quit IRC | 13:01 | |
| *** iyamahat has quit IRC | 13:01 | |
| *** trown|outtypewww is now known as trown | 13:02 | |
| *** markvoelker has quit IRC | 13:03 | |
| *** rossella_s has joined #openstack-infra | 13:03 | |
| *** spectr has quit IRC | 13:03 | |
| *** stephenfin is now known as finucannot | 13:04 | |
| dmsimard | AJaeger: how's publish-api-ref looking now ? | 13:05 |
| *** efoley has quit IRC | 13:06 | |
| *** bobh has joined #openstack-infra | 13:08 | |
| *** andreas_s has quit IRC | 13:08 | |
| *** hashar has joined #openstack-infra | 13:09 | |
| *** panda|rover|off is now known as panda|rover | 13:10 | |
| *** yamahata has joined #openstack-infra | 13:13 | |
| *** andreas_s has joined #openstack-infra | 13:14 | |
| *** esberglu has joined #openstack-infra | 13:14 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: sahara-image-elements: remove the migrated jobs https://review.openstack.org/513485 | 13:15 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Create new repo cinder tempest plugin https://review.openstack.org/486303 | 13:16 |
| *** dizquierdo has quit IRC | 13:17 | |
| *** spectr has joined #openstack-infra | 13:20 | |
| *** tosky has quit IRC | 13:20 | |
| *** tosky has joined #openstack-infra | 13:21 | |
| *** baoli has joined #openstack-infra | 13:22 | |
| *** nicolasbock has quit IRC | 13:24 | |
| *** jcoufal has joined #openstack-infra | 13:25 | |
| *** huanxie has quit IRC | 13:25 | |
| *** migi has quit IRC | 13:25 | |
| openstackgerrit | Tom Barron proposed openstack-infra/openstack-zuul-jobs master: make openstack-tox-cover inherit from tox-cover https://review.openstack.org/513737 | 13:26 |
| *** armaan has quit IRC | 13:26 | |
| *** baoli has quit IRC | 13:27 | |
| *** migi has joined #openstack-infra | 13:27 | |
| *** andreas_s has quit IRC | 13:27 | |
| *** amoralej is now known as amoralej|lunch | 13:27 | |
| *** jcoufal has quit IRC | 13:27 | |
| *** jcoufal has joined #openstack-infra | 13:28 | |
| *** dulek has joined #openstack-infra | 13:28 | |
| *** jcoufal has quit IRC | 13:28 | |
| *** lifeless has quit IRC | 13:28 | |
| *** jcoufal has joined #openstack-infra | 13:29 | |
| *** lifeless has joined #openstack-infra | 13:29 | |
| *** andreas_s has joined #openstack-infra | 13:32 | |
| *** nicolasbock has joined #openstack-infra | 13:32 | |
| pabelanger | fungi: ack | 13:33 |
| pabelanger | I have a couple regional fights myself today, just at airport now | 13:34 |
| *** mrunge has quit IRC | 13:36 | |
| *** mrunge has joined #openstack-infra | 13:37 | |
| *** andreas_s has quit IRC | 13:40 | |
| *** yolanda has joined #openstack-infra | 13:42 | |
| *** dave-mccowan has joined #openstack-infra | 13:42 | |
| *** jaypipes is now known as leakypipes | 13:43 | |
| *** dave-mcc_ has joined #openstack-infra | 13:45 | |
| *** LindaWang has quit IRC | 13:46 | |
| *** LindaWang has joined #openstack-infra | 13:46 | |
| dmsimard | Wow Zuul RAM looks great today | 13:46 |
| *** dave-mccowan has quit IRC | 13:48 | |
| *** andreas_s has joined #openstack-infra | 13:50 | |
| *** dansmith is now known as superdan | 13:53 | |
| *** andreas_s has quit IRC | 13:55 | |
| *** ldnunes has quit IRC | 13:55 | |
| *** jamesmcarthur has quit IRC | 13:57 | |
| AJaeger | dmsimard: haven't seen a run since then yet, will monitor for next one and tell... | 13:57 |
| *** sdague has joined #openstack-infra | 13:58 | |
| dmsimard | AJaeger: I'll consider writing an integration role for that job | 13:58 |
| AJaeger | write one for translation jobs, please - those still have some way to go I fear... | 13:58 |
| *** d0ugal_ has joined #openstack-infra | 13:59 | |
| AJaeger | dmsimard: but please go for it, the better tests we have... | 13:59 |
| *** ethfci has quit IRC | 13:59 | |
| *** d0ugal has quit IRC | 13:59 | |
| dmsimard | AJaeger: yeah, writing the jobs aren't too hard. It seems like it's merging them that is challenging :) | 14:00 |
| * dmsimard been rebasing some of them forever now | 14:00 | |
| *** d0ugal_ has quit IRC | 14:01 | |
| AJaeger | dmsimard: is there any I should review? | 14:01 |
| *** d0ugal has joined #openstack-infra | 14:01 | |
| *** d0ugal has quit IRC | 14:01 | |
| *** d0ugal has joined #openstack-infra | 14:01 | |
| dmsimard | AJaeger: you already +2'd them so it's ok | 14:04 |
| *** baoli has joined #openstack-infra | 14:05 | |
| *** spectr has quit IRC | 14:06 | |
| *** _milan_ has joined #openstack-infra | 14:06 | |
| _milan_ | folks, anyone seeing http://logs.openstack.org/15/513415/1/gate/legacy-grenade-dsvm-ironic-inspector/1bd5274/job-output.txt.gz#_2017-10-20_08_12_18_743064 | 14:06 |
| *** armax has joined #openstack-infra | 14:07 | |
| *** hongbin has joined #openstack-infra | 14:07 | |
| AJaeger | dmsimard: let's ask pabelanger or frickler then for review of https://review.openstack.org/#/c/512927/ and https://review.openstack.org/#/c/512904/ | 14:07 |
| *** ldnunes has joined #openstack-infra | 14:08 | |
| *** efoley has joined #openstack-infra | 14:08 | |
| AJaeger | _milan_: that error - that's harmless, ignore it. pip cannot find the git remote to output it. | 14:08 |
| *** gridinv has quit IRC | 14:09 | |
| _milan_ | AJaeger, ack, thx, /me rechecks | 14:09 |
| dhellmann | smcginnis : yeah, I think we need to make clone_repo.sh not fail if the branch doesn't exist *or* we need to change update_constraints.sh to handle the fallback itself gracefully | 14:09 |
| *** jcoufal has quit IRC | 14:09 | |
| dmsimard | btw I'd vote for something shorter than project-config-core.. like config-core ? just.. quality of life change :) | 14:09 |
| AJaeger | _milan_: so, search further for the real bug. That message you always get, so no reason for recheck | 14:09 |
| smcginnis | dhellmann: Which would you prefer? | 14:10 |
| _milan_ | AJaeger, alright, search I shall | 14:10 |
| dhellmann | thinking | 14:10 |
| *** gridinv has joined #openstack-infra | 14:11 | |
| dhellmann | smcginnis : I wonder how many other places we rely on the fallback behavior. | 14:11 |
| *** armax has quit IRC | 14:11 | |
| _milan_ | AJaeger, how about this ansible parse error http://logs.openstack.org/15/513415/1/gate/legacy-grenade-dsvm-ironic-inspector/1bd5274/job-output.txt.gz#_2017-10-20_10_47_58_633516 | 14:11 |
| smcginnis | dhellmann: Yeah, has potential to pop up in multiple places. | 14:11 |
| *** jcoufal has joined #openstack-infra | 14:11 | |
| *** markvoelker has joined #openstack-infra | 14:11 | |
| smcginnis | dhellmann: project-config change merged: https://review.openstack.org/513710 | 14:11 |
| smcginnis | dhellmann: Releases one is still out there though: https://review.openstack.org/513709 | 14:12 |
| *** wolverineav has joined #openstack-infra | 14:12 | |
| AJaeger | sorry, need to run some errands, hope somebody else can help you further, _milan_ . | 14:12 |
| _milan_ | AJaeger, no worries, thanks anyway! | 14:12 |
| frickler | dmsimard: I'm wondering whether we need that distinction at all. I'm hilighting on infra-root even though I'm not one, and IMO that is a good enough trigger. or are there infra-roots that do not want to be highlighted for project-config reviews? | 14:12 |
| dmsimard | frickler: I think it's the latter. | 14:12 |
| *** jaosorior has quit IRC | 14:12 | |
| dmsimard | I am also highlighted on infra-root fwiw | 14:12 |
| dhellmann | smcginnis: approved; doing it there makes sense for consistency. I'm likely to forget that the clone script *doesn't* behave that way :-) | 14:13 |
| smcginnis | ;) | 14:13 |
| *** sdague has quit IRC | 14:13 | |
| *** jamesmcarthur has joined #openstack-infra | 14:13 | |
| smcginnis | dhellmann: Then we'll just need fungi to get power back or find someone else that can requeue that job. | 14:13 |
| smcginnis | dhellmann: Or are there any other errors you've noticed. I thought there were more, but couldn't find anything else when reviewing this morning. | 14:14 |
| dhellmann | smcginnis : I think any of the infra-core folks can do that? | 14:14 |
| dhellmann | there is a problem with the update constraints job | 14:14 |
| *** ldnunes has quit IRC | 14:14 | |
| dhellmann | oh, wait, that's what this fixes | 14:15 |
| pabelanger | frickler: possible, I would continue to use project-config-core, as we have been doing so for some time. | 14:15 |
| smcginnis | dhellmann: I think (thought?) that was another checkout_branch issue. | 14:15 |
| dhellmann | so no, I'm not aware of any other issues | 14:15 |
| dhellmann | yeah, I forgot what the issue was but remembered the job | 14:15 |
| dmsimard | pabelanger: I was suggesting just shortening it to 'config-core' just because project- is a whole 8 keystrokes :) | 14:15 |
| dhellmann | smcginnis : I have a meeting in a few minutes, so I'm going to have to drop off. I'll be back by our release team meeting. Maybe pabelanger or dmsimard can re-enqueue that job? | 14:16 |
| smcginnis | dhellmann: ack | 14:16 |
| frickler | I must admit I have never seen that used until a couple of days ago. maybe because the only person I knew to belong to one and not the other was AJaeger ;) | 14:16 |
| dmsimard | frickler: it was volounteered as a tag in order to ping new cores and filter down things that might not require root access | 14:17 |
| dmsimard | It wasn't used until this week afaik | 14:17 |
| pabelanger | dmsimard: well, might propose it at next meeting, see what others say. As it will require client side changes | 14:17 |
| dmsimard | sure, I'll add it. | 14:18 |
| *** andreas_s has joined #openstack-infra | 14:18 | |
| pabelanger | frickler: yah, now that we have more project-config-core members, I'll be using it over direct pings to AJaeger | 14:18 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Added nat_source flag for networks. https://review.openstack.org/513751 | 14:19 |
| *** spectr has joined #openstack-infra | 14:19 | |
| *** armax has joined #openstack-infra | 14:19 | |
| *** andreas_s has quit IRC | 14:22 | |
| *** ralonsoh has quit IRC | 14:26 | |
| *** amoralej|lunch is now known as amoralej | 14:26 | |
| *** ldnunes has joined #openstack-infra | 14:27 | |
| *** camunoz has joined #openstack-infra | 14:28 | |
| *** armax has quit IRC | 14:29 | |
| *** jcoufal has quit IRC | 14:31 | |
| dmsimard | is the list of retired repositories available somewhere ? or, perhaps a list of repositories without the retired ones ? | 14:31 |
| dmsimard | like that doesn't include deb-* | 14:31 |
| smcginnis | Any infra-core that can re-enqueue a release-post job like fungi was doing yesterday to help test if we have the release process working again? | 14:32 |
| dmsimard | infra-root ^ | 14:32 |
| dmsimard | smcginnis: sorry don't have access to do that | 14:33 |
| dmsimard | hopefully someone around can :) | 14:33 |
| smcginnis | dmsimard: Thanks, wasn't sure on the right bat signal. :) | 14:33 |
| *** jamesmcarthur has quit IRC | 14:39 | |
| *** tikitavi has joined #openstack-infra | 14:40 | |
| *** tpsilva has joined #openstack-infra | 14:40 | |
| dmsimard | answering my own question about retired repos, it seems the closest I've got would be to load the gerrit project yaml and look for "/home/gerrit2/acls/openstack/retired.config" | 14:40 |
| *** spectr has quit IRC | 14:42 | |
| *** jcoufal has joined #openstack-infra | 14:42 | |
| dmsimard | frickler: please look at https://review.openstack.org/#/c/512927/ and https://review.openstack.org/#/c/512904/ if you can, I've been rebasing those for a long while | 14:42 |
| tikitavi | Hi, I have a question about Zuul.3 jobs, we have rally job in our project which fails because of timeout. Can I just add timeout parameter to zuul.d/projects.yaml / project / job (as it is done in vmware-nsx)? | 14:44 |
| *** e0ne has quit IRC | 14:44 | |
| *** andreas_s has joined #openstack-infra | 14:45 | |
| *** felipemonteiro__ has joined #openstack-infra | 14:45 | |
| tikitavi | Or how can I raise timeout? | 14:46 |
| *** nicolasbock has quit IRC | 14:46 | |
| dmsimard | tikitavi: there is a default timeout, yes, and you can change it accordingly. What is the current timeout you're seeing ? | 14:46 |
| tikitavi | 130 | 14:46 |
| tikitavi | I suppose | 14:46 |
| *** e0ne has joined #openstack-infra | 14:47 | |
| *** felipemonteiro has joined #openstack-infra | 14:47 | |
| *** salv-orlando has quit IRC | 14:47 | |
| dmsimard | tikitavi: most jobs inherit from a default timeout of 1800 (seconds) from the base job which is 30 minutes: https://github.com/openstack-infra/project-config/blob/master/zuul.d/jobs.yaml#L26-L55 | 14:47 |
| dmsimard | tikitavi: you can specify a timeout of your choice directly in your job definition to override the default one. | 14:48 |
| tikitavi | ok, thank you! I'll try | 14:48 |
| *** felipemonteiro__ has quit IRC | 14:49 | |
| *** salv-orlando has joined #openstack-infra | 14:50 | |
| *** jcoufal has quit IRC | 14:50 | |
| *** slaweq has quit IRC | 14:51 | |
| *** slaweq has joined #openstack-infra | 14:52 | |
| *** dave-mcc_ has quit IRC | 14:53 | |
| *** andreas_s has quit IRC | 14:53 | |
| *** andreas_s has joined #openstack-infra | 14:54 | |
| *** jcoufal has joined #openstack-infra | 14:54 | |
| *** dave-mccowan has joined #openstack-infra | 14:54 | |
| *** makowals has quit IRC | 14:56 | |
| *** xarses has joined #openstack-infra | 14:56 | |
| *** slaweq has quit IRC | 14:56 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Migrate legacy jobs for feature/zuulv3 branch https://review.openstack.org/512611 | 14:56 |
| *** baoli has quit IRC | 14:58 | |
| *** andreas_s has quit IRC | 14:58 | |
| *** nicolasbock has joined #openstack-infra | 14:58 | |
| smcginnis | Looks like some sort of zuul job error here: http://logs.openstack.org/2d/2d2efc4f7360b194cd0249924ce37e4210b2fef9/release-post/publish-static/bfa64e0/job-output.txt.gz#_2017-10-20_14_41_40_442654 | 14:58 |
| Shrews | umm, ok, i don't know how this happened, but i seem to have 2 changes with the same Change ID: https://review.openstack.org/512637 and https://review.openstack.org/512611 | 15:00 |
| Shrews | infra-root: ^^^ | 15:00 |
| jeblair | Shrews: different branches | 15:01 |
| dmsimard | Shrews: different branches | 15:01 |
| dmsimard | damn, jeblair is quick on the trigger | 15:01 |
| Shrews | yes | 15:01 |
| Shrews | is change id not unique across branches? | 15:01 |
| dmsimard | Shrews: you can have same change-ids for different changes, it's used typically to keep changes grouped together (i.e, cherry picking backports to stable branches) | 15:01 |
| jeblair | Shrews: nope. intentionally so -- often you need to apply the same "change" to multiple branches. ie, bcakports. | 15:02 |
| jeblair | Shrews: to different projects too | 15:02 |
| jeblair | Shrews: when we make, say, tox.ini changes to all the projects, we usually give them the same change id | 15:02 |
| Shrews | jeblair: but i created those changes independently. just wondering what determined they should share a change id | 15:03 |
| Shrews | jeblair: i could have swore i had depends-on and needed-by on both of these at one point (each pointing to the other), but i'm not seeing that now | 15:04 |
| jeblair | Shrews: oh wow, that's neat. that's like a hash collision. | 15:04 |
| jeblair | Shrews: buy a lottery ticket. | 15:04 |
| *** baoli has joined #openstack-infra | 15:04 | |
| *** dave-mcc_ has joined #openstack-infra | 15:05 | |
| *** ijw has joined #openstack-infra | 15:05 | |
| Shrews | jeblair: this is really weird b/c i don't feel like what i'm being shown in gerrit is the same patch i submitted. the feature/zuulv3 branch was only supposed to define the nodepool-zuul-functional job, but all of a sudden it had all the jobs from the master branch change too. | 15:06 |
| Shrews | maybe i'm losing my mind | 15:06 |
| *** dave-mccowan has quit IRC | 15:06 | |
| *** thiagolib has joined #openstack-infra | 15:06 | |
| *** tikitavi has quit IRC | 15:07 | |
| jeblair | Shrews: maybe a local cherry-pick or rebase went wrong? that might explain both the content and change-id discrepancy | 15:07 |
| *** dizquierdo has joined #openstack-infra | 15:07 | |
| Shrews | jeblair: possibly. i'm sure it's an pebkac error | 15:08 |
| Shrews | going to abandon the one and resubmit | 15:09 |
| *** _milan_ has quit IRC | 15:11 | |
| *** jpich has quit IRC | 15:13 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Migrate legacy jobs for feature/zuulv3 branch https://review.openstack.org/513766 | 15:13 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Use playbook from tripleo-quickstart-extras for OVB https://review.openstack.org/513508 | 15:14 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 15:14 |
| dmsimard | Shrews: I've had that happen to me once but it was like a bad "commit --amend" muscle memory and I didn't originally mean to submit a change-id I had used before | 15:14 |
| *** jamesmcarthur has joined #openstack-infra | 15:15 | |
| *** ihrachys has joined #openstack-infra | 15:16 | |
| finucannot | AJaeger: any chance you could provide some direction on https://review.openstack.org/#/c/396289/ ? | 15:19 |
| finucannot | I'm not even sure where to start. The 'jenkins/jobs/nova.yaml' file stil | 15:20 |
| finucannot | *still exists, but I guess the zuul variant is stored elsewhere now? | 15:20 |
| dmsimard | finucannot: yes, everything under jenkins/ is frozen for eventual deletion: http://lists.openstack.org/pipermail/openstack-dev/2017-October/123768.html | 15:22 |
| dmsimard | finucannot: let me see where the v3 equivalent would be for you | 15:22 |
| tosky | finucannot: I suggest checking https://docs.openstack.org/infra/manual/zuulv3.html too | 15:22 |
| tosky | the migration guide | 15:23 |
| finucannot | tosky: Yup, reading that now but nothing's jumping out at me | 15:23 |
| tosky | finucannot: "Moving Legacy Jobs to Projects" explains where the legacy jobs, automatically migrated from v2 jobs, have been created | 15:24 |
| tosky | and how to create native jobs in-tree | 15:24 |
| *** LindaWang has quit IRC | 15:25 | |
| dmsimard | finucannot: I believe what you are looking for is here: https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/playbooks/legacy/tempest-dsvm-neutron-nova-next-full/run.yaml | 15:25 |
| finucannot | dmsimard: Oh, it's a totally different repo | 15:25 |
| finucannot | tosky: Yeah, not creating a new job or migrating anything. Purely adding something to a job that should already have been migrated | 15:26 |
| dmsimard | finucannot: things are a bit split up between three repositories right now. | 15:26 |
| finucannot | There's no "fix up my already-migrated job" section :P | 15:26 |
| tosky | because we are not supposed to extend them; at most fix them | 15:26 |
| tosky | but the migration section says where to migrate them from, so where to find them | 15:26 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config master: Remove py27-based template for nodepool https://review.openstack.org/513770 | 15:26 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 15:27 |
| dmsimard | finucannot: so that's an experimental job on devstack right ? | 15:27 |
| Shrews | jeblair: so, similar situation we discussed the other day about skipping jobs defined in project-config. Don't want to run the tox-py27 job on the features branch, so submitted the above 2 changes (770 and 637). | 15:28 |
| finucannot | dmsimard: I _think_ so, but I'm really not sure. This wasn't my patch originally 🙈 | 15:29 |
| dmsimard | Is that a... monkey emoji ? I'm half surprised my client rendered that correctly | 15:29 |
| finucannot | Indeed - the Unicode Consortium need to get paid somehow :) | 15:30 |
| finucannot | https://emojipedia.org/see-no-evil-monkey/ | 15:30 |
| finucannot | dmsimard, tosky: So is the expectation here that I migrate the legacy project to the nova repo before I modify it? | 15:31 |
| dmsimard | finucannot: I added a comment in the review you linked. I think ideally we would try to keep modifications to "migrated" (legacy) jobs to a minimum and they should be moved in-tree as soon as possible. Otherwise projects would end up never moving their stuff in-tree. There's already been some work done towards native Zuul v3 jobs for devstack as you can see here: | 15:32 |
| dmsimard | https://github.com/openstack-dev/devstack/blob/master/.zuul.yaml | 15:32 |
| tosky | finucannot: I'm not a core on that repository; this is how I read that documentation, so I may be wrong | 15:32 |
| finucannot | This is going to be fun :D | 15:32 |
| * finucannot really hopes someone has already done this for me, heh | 15:32 | |
| jeblair | Shrews: yeah, we either do that, or we put the branch exclusion in project-config (i could see doing that as a way to at least have an inventory of where projects deviate from the pti) | 15:33 |
| finucannot | dmsimard, tosky: OK, cool. Thanks for the info and agreed that merging features into the legacy projects probably isn't the best idea | 15:34 |
| openstackgerrit | Michael Johnson proposed openstack-infra/infra-manual master: Clarify which jobs belong in the project block https://review.openstack.org/513197 | 15:34 |
| finucannot | I'll see if anything's been migrated and, if not, will do so | 15:34 |
| jeblair | smcginnis, dhellmann, dmsimard: did anyone re-enqueue that ref? | 15:35 |
| finucannot | A recommended Gerrit topic would be a good addition to that doc, as would a huge admonition not to modify the legacy jobs | 15:35 |
| finucannot | I can do both if anyone has topic suggestions | 15:35 |
| dmsimard | jeblair: not that I know of. | 15:35 |
| jeblair | dmsimard: do you know what needs to be done? | 15:35 |
| *** e0ne has quit IRC | 15:36 | |
| dhellmann | jeblair: the last failure we were looking at was from http://logs.openstack.org/dd/dd0bb2d44053e11d9d4a4775b3c71f2c0889dc2a/release/propose-update-constraints/953c933/job-output.txt.gz#_2017-10-19_23_27_20_561841 I think | 15:37 |
| dmsimard | jeblair: "re-enqueue a release-post job like fungi was doing yesterday", I don't have a specific ref -- but it might be that a post job would have already triggered from https://review.openstack.org/#/c/513709/ merging. | 15:37 |
| dhellmann | we merged fixes for that into project-config and release-tools | 15:37 |
| *** Apoorva has joined #openstack-infra | 15:37 | |
| dhellmann | so if you could re-run at least the post jobs for that patch, that would tell us if the job failed again. I think we're using the same patch we were using yesterday to tag a release in the release-test repository | 15:37 |
| smcginnis | jeblair: Yeah, sorry. dhellmann do we need to work out that last issue now before we requeue anything? | 15:38 |
| dhellmann | the *next* failure is in our publish job. that error has to do with the ssh key and I don't understand the error message: http://logs.openstack.org/2d/2d2efc4f7360b194cd0249924ce37e4210b2fef9/release-post/publish-static/bfa64e0/job-output.txt.gz#_2017-10-20_14_41_40_442654 | 15:38 |
| *** hashar is now known as hasharAway | 15:38 | |
| *** leakypipes has quit IRC | 15:38 | |
| smcginnis | Me neither. | 15:38 |
| dhellmann | maybe we're missing some metadata for that job? | 15:38 |
| dmsimard | dhellmann: where are you picking up post/release job failures from ? by email ? | 15:38 |
| dhellmann | I don't know why that job needs an ssh key | 15:38 |
| dhellmann | dmsimard : email, obsessively watching the zuul status page, or using git os-job to find the url | 15:39 |
| AJaeger | dmsimard: required repos have the required ACL group in project-config/gerrit/projects.yaml | 15:39 |
| odyssey4me | dhellmann I'm glad it's not just me ... | 15:39 |
| AJaeger | dmsimard: ah, you found it ;) I'm reading backscroll and catch up right now... | 15:40 |
| odyssey4me | (obsessively watching job progress) | 15:40 |
| dmsimard | dhellmann: oh, I meant to try out your new git os-job tool, guess this is a good opportunity | 15:40 |
| jeblair | dhellmann: looks like that job publishing something to static.o.o (which requires an ssh key) | 15:40 |
| *** yamamoto has quit IRC | 15:40 | |
| smcginnis | dmsimard, jeblair: You can see all the release-test retries we've done here: http://lists.openstack.org/pipermail/release-job-failures/2017-October/thread.html | 15:40 |
| openstackgerrit | Michael Johnson proposed openstack-infra/openstack-zuul-jobs master: Switch openstack-tox-cover to parent tox-cover https://review.openstack.org/513773 | 15:40 |
| dhellmann | dmsimard : in a copy of the release-test repo if you run "git os-job -u 0.8.0" it should give you http://logs.openstack.org/dd/dd0bb2d44053e11d9d4a4775b3c71f2c0889dc2a/ | 15:41 |
| smcginnis | Oh, actually that's missing most since we weren't getting emails for awhile. | 15:41 |
| dhellmann | dmsimard : that will show you the jobs for that tag | 15:41 |
| dhellmann | the jobs for the release request in openstack/releases can be found with git os-job in that repo | 15:42 |
| dhellmann | jeblair : ok. which key should it be using, I can update the job | 15:42 |
| dmsimard | dhellmann, smcginnis: so https://review.openstack.org/#/c/513709/ merged and from looking it up with os-job I find http://logs.openstack.org/2d/2d2efc4f7360b194cd0249924ce37e4210b2fef9/release-post/ which leads me to the error here: http://logs.openstack.org/2d/2d2efc4f7360b194cd0249924ce37e4210b2fef9/release-post/publish-static/bfa64e0/job-output.txt.gz#_2017-10-20_14_41_40_442654 | 15:43 |
| dmsimard | which I suppose is the ssh key problem you speak of | 15:43 |
| smcginnis | dmsimard: Yep, that's the latest one that I'm not sure about. | 15:43 |
| jeblair | dhellmann: it looks like it is already configured to use the 'static_ssh_key' secret | 15:43 |
| dhellmann | well, I could update it if I could find it | 15:43 |
| dhellmann | where is that job defined? | 15:43 |
| jeblair | dhellmann: project-config | 15:44 |
| jeblair | wow | 15:44 |
| jeblair | does any static publishing work? | 15:44 |
| dhellmann | and where is it associated with the releases repo? | 15:44 |
| dmsimard | jeblair: that's what I'm wondering right now | 15:44 |
| dhellmann | I have not yet internalized the rules for finding this stuff, sorry. | 15:45 |
| jeblair | governace, tc, uc, security, etc... | 15:45 |
| jeblair | dhellmann: the association is in openstack/releases (.zuul.yaml) | 15:46 |
| dhellmann | ok, I didn't think we'd moved anything over there yet | 15:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Fix typo in static ssh key secret https://review.openstack.org/513777 | 15:47 |
| jeblair | dhellmann, dmsimard: ^ gimme a minute to futz with a yaml parser to see if that was the problem | 15:47 |
| smcginnis | Oh, no way I would have been able to track that one down any time soon. :) | 15:47 |
| dmsimard | jeblair: I found the error | 15:48 |
| dmsimard | jeblair: a typo, sending a patch | 15:48 |
| dhellmann | I had just managed to find where the secret for the job was named. | 15:48 |
| smcginnis | jeblair: Isn't that secret used elsewhere though? | 15:48 |
| dhellmann | dmsimard : ^^ | 15:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Fix typo in static_ssh_key definition https://review.openstack.org/513779 | 15:48 |
| dmsimard | jeblair, dhellmann, smcginnis ^ | 15:48 |
| jeblair | dhellmann: yes, for many things -- goverance, tc, uc, security publishing | 15:48 |
| smcginnis | Just surprised we haven't seen other errors. | 15:49 |
| tosky | mordred: hi, I was rechecking the list of jobs to migrate for sahara, namely sahara-extras which generates tarballs, and I was wondering if there is some progress in the "Change publication interface to be directories on node" proposal | 15:49 |
| dmsimard | jeblair: I wonder how that typo didn't just crash everything | 15:49 |
| smcginnis | dmsimard: He beat you to it. | 15:49 |
| dhellmann | dmsimard : your patch is the same as jeblair's | 15:49 |
| dmsimard | oh damn | 15:49 |
| dhellmann | https://review.openstack.org/#/c/513777/1 | 15:49 |
| jeblair | it parses like this: | 15:49 |
| jeblair | {'foo': {'bar:': 'baz'}} | 15:49 |
| frickler | from http://logs.openstack.org/2d/2d2efc4f7360b194cd0249924ce37e4210b2fef9/release-post/publish-static/bfa64e0/job-output.txt.gz#_2017-10-20_14_41_40_442654 I'd think that the role add-fileserver in playbooks/publish/static.yaml needs to be given an argument, I do not see a default there | 15:50 |
| jeblair | so the dict had an entry for 'ssh_private_key:' rather than 'ssh_private_key' | 15:50 |
| dmsimard | bleh | 15:50 |
| jeblair | frickler: it picks it up from the global ansible variables, which are set in the job | 15:50 |
| dmsimard | frickler: those are expected to be supplied by secrets | 15:50 |
| jeblair | (which, yeah, is a secret in this case) | 15:50 |
| dmsimard | dhellmann: git os-job is awesome btw | 15:51 |
| dmsimard | dhellmann++ | 15:51 |
| smcginnis | OK, so once 513559 lands, can we get the release-test jobs rerun? | 15:52 |
| *** lucasagomes is now known as lucas-afk | 15:52 | |
| dhellmann | dmsimard : cool, I'm glad you find it useful | 15:52 |
| jeblair | odyssey4me: if you want to try submitting a few more patches simultaneously, i think that'd be fine. they still take some memory, so let's ease into it. but they should not hold onto that memory for as long as before. | 15:52 |
| dhellmann | when this transition settles down I should import git-os-job into infra | 15:53 |
| odyssey4me | jeblair we noticed the better memory usage and upped the pace just a little more, we're still watching the memory usage like a hawk - but you've done a great job of improving it with that patch! | 15:53 |
| jeblair | odyssey4me: great! | 15:53 |
| jeblair | smcginnis: 559 landed yesterday, did you mean 513777? | 15:53 |
| *** jaypipes has joined #openstack-infra | 15:54 | |
| smcginnis | jeblair: Oops, right, 777 I meant. | 15:54 |
| *** jaypipes is now known as leakypipes | 15:54 | |
| openstackgerrit | Tom Barron proposed openstack-infra/openstack-zuul-jobs master: make openstack-tox-cover inherit from tox-cover https://review.openstack.org/513737 | 15:55 |
| jeblair | smcginnis: yeah, what should i run? should i re-run the release-post jobs dhellmann linked? http://logs.openstack.org/2d/2d2efc4f7360b194cd0249924ce37e4210b2fef9/ ? | 15:55 |
| dhellmann | jeblair, smcginnis : I think we need 2, to test both fixes, don't we? | 15:56 |
| dhellmann | 2d tests the secret change jeblair is making | 15:56 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Removes migrated legacy-octavia-* https://review.openstack.org/513781 | 15:56 |
| dhellmann | http://logs.openstack.org/dd/dd0bb2d44053e11d9d4a4775b3c71f2c0889dc2a/release/propose-update-constraints/953c933/job-output.txt.gz also failed | 15:56 |
| smcginnis | Yeah, trying to find the release-test patch. There it is. ^ :) | 15:56 |
| dhellmann | and that's fixed by smcginnis' patch | 15:56 |
| *** gyee has joined #openstack-infra | 15:56 | |
| jeblair | okay, so enqueue 2d and dd | 15:56 |
| dhellmann | yes, please | 15:57 |
| jeblair | oh, and dd goes into release | 15:57 |
| dhellmann | yes, that was the tag event | 15:57 |
| smcginnis | 777 is still queued in gate, but that shouldn't take too much longer. | 15:57 |
| dhellmann | oh, I wonder if update constraints is even going to re-run there | 15:57 |
| dhellmann | well, it won't hurt | 15:57 |
| dhellmann | we may need to test that job with a real library release to get it to do everything it has to do | 15:58 |
| *** Guest6618 has quit IRC | 15:58 | |
| *** efoley has quit IRC | 15:58 | |
| smcginnis | We could queue up a new release-test patch too just to exercise part of it. | 15:58 |
| dhellmann | yeah, openstack-release-test doesn't appear in the global requirements list | 15:58 |
| smcginnis | Or just go for it with a lib release. | 15:58 |
| dhellmann | so the job won't actually try to propose a constraint update | 15:58 |
| dhellmann | it will determine that there's nothing to do | 15:58 |
| jeblair | zuul enqueue-ref --tenant openstack --trigger gerrit --pipeline release-post --project openstack/releases --ref refs/heads/master --newrev 2d2efc4f7360b194cd0249924ce37e4210b2fef9 --oldrev feb0fdabca16208a18c443f41104f0568beaa3dc | 15:59 |
| jeblair | zuul enqueue-ref --tenant openstack --trigger gerrit --pipeline release --project openstack/release-test --ref refs/tags/0.8.0 --newrev dd0bb2d44053e11d9d4a4775b3c71f2c0889dc2a | 15:59 |
| jeblair | that's what i have staged | 16:00 |
| smcginnis | dhellmann: Are you saying we should just skipp dd0bb? | 16:01 |
| *** iyamahat_ has quit IRC | 16:01 | |
| dhellmann | smcginnis : https://review.openstack.org/513784 adds openstack-release-test to the requirements list for the future | 16:01 |
| smcginnis | +2 | 16:01 |
| dhellmann | smcginnis : let's go ahead and run both, just knowing that dd0bb won't be a full test and we need to be careful interpreting success as such | 16:01 |
| *** yamahata has quit IRC | 16:01 | |
| dhellmann | jeblair : ^^ | 16:01 |
| smcginnis | Probably good for it to at least go through as far as it can. | 16:02 |
| dhellmann | right | 16:02 |
| openstackgerrit | Michael Johnson proposed openstack-infra/openstack-zuul-jobs master: Removes migrated legacy-octavia-* https://review.openstack.org/513785 | 16:02 |
| dhellmann | it won't hurt | 16:02 |
| dhellmann | and may tell us something | 16:02 |
| frickler | another release related failure, readthedocs.org cert verification failure: http://logs.openstack.org/1e/1e635bd6c4cb1cea2aa97d08addcf23c084f321f/release/trigger-readthedocs/5f50aea/job-output.txt.gz#_2017-10-20_15_43_08_152279 | 16:02 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Removes migrated legacy-octavia-* https://review.openstack.org/513781 | 16:03 |
| frickler | adding to etherpad | 16:03 |
| dhellmann | added to https://etherpad.openstack.org/p/release-job-failures | 16:03 |
| dhellmann | which etherpad are we using for tracking? | 16:03 |
| frickler | oh, was already there. dhellmann: https://etherpad.openstack.org/p/zuulv3-issues | 16:03 |
| *** annp has joined #openstack-infra | 16:03 | |
| dhellmann | ok | 16:03 |
| smcginnis | dhellmann: The one at the top of the release-job-failures is the real, common one. | 16:04 |
| dhellmann | ah, missed that, thanks smcginnis | 16:04 |
| smcginnis | But I started the release-job-failures to help us keep tabs on what's actually impacting the release activities. | 16:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix typo in static ssh key secret https://review.openstack.org/513777 | 16:04 |
| mnaser | ^ can i help with that? | 16:04 |
| smcginnis | So that one probably doesn't hurt to have in both locations I guess. | 16:04 |
| smcginnis | jeblair: OK, looks like we are ready. | 16:05 |
| jeblair | smcginnis: ok. the first enqueue is the "1 management events" you see on the status page | 16:06 |
| smcginnis | Where is that? | 16:06 |
| jeblair | smcginnis: at the top; you may need to shift-reload (it's new) | 16:06 |
| smcginnis | Oh, lookie there. | 16:07 |
| jeblair | it's currently rebuilding all the dynamic configurations in queues since we merged a config change to project-config, so it'll be a min | 16:07 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy-requirements-cross-* jobs https://review.openstack.org/513275 | 16:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix Puppet integration jobs to run on master only https://review.openstack.org/513731 | 16:08 |
| openstackgerrit | Merged openstack-infra/project-config master: ansible-role-k8s-cookiecutter to zuul.d/projects https://review.openstack.org/512330 | 16:08 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add integration tests for use-cached-repos https://review.openstack.org/512927 | 16:08 |
| jeblair | smcginnis, dhellmann: enqueued | 16:09 |
| *** tmorin has quit IRC | 16:10 | |
| dhellmann | https://www.youtube.com/watch?v=uMyCa35_mOg | 16:11 |
| smcginnis | :) | 16:12 |
| * dhellmann has been listening to a lot of tom petty lately | 16:12 | |
| smcginnis | It seems that song in particular. | 16:12 |
| dhellmann | it has been thematically appropriate this week | 16:12 |
| *** baoli has quit IRC | 16:13 | |
| SamYaple | ive been on a Tom Petty free fall before.... but i didnt back down | 16:13 |
| *** baoli has joined #openstack-infra | 16:13 | |
| AJaeger | jeblair: publishing to specs.openstack.org works | 16:13 |
| pabelanger | AJaeger: yay | 16:13 |
| jeblair | AJaeger: has it been working? | 16:14 |
| jeblair | AJaeger: it seems to use the 'site_logs' secret | 16:14 |
| jeblair | which, erm, well, i'm not going to change that now. | 16:14 |
| openstackgerrit | Merged openstack-infra/project-config master: Add ansible-role-k8s-(keystone|mariadb) https://review.openstack.org/513022 | 16:15 |
| jeblair | AJaeger: so if it has been working, that at least explains why it did but not the other static.o.o sites | 16:15 |
| smcginnis | Here we go. | 16:15 |
| jeblair | smcginnis: and i'm just getting to the end of the tom petty song | 16:16 |
| jeblair | dhellmann: well timed | 16:16 |
| *** tesseract has quit IRC | 16:17 | |
| smcginnis | Got past the fileserver ssh key at least. And done! | 16:18 |
| smcginnis | One good one at least. | 16:18 |
| openstackgerrit | Merged openstack-infra/project-config master: tripleo: index /var/log/*.log.txt files https://review.openstack.org/513469 | 16:18 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy jobs in Vitrage https://review.openstack.org/510432 | 16:18 |
| *** iyamahat has joined #openstack-infra | 16:19 | |
| AJaeger | jeblair: http://zuulv3.openstack.org/static/stream.html?uuid=1c13427583f547b8bc50535128b82131&logfile=console.log is just running - governance publishing | 16:22 |
| *** Apoorva has quit IRC | 16:22 | |
| *** dtantsur is now known as dtantsur|afk | 16:22 | |
| frickler | hmm, the other one seems to be using bindep from releases instead of p-c, so still no yaml I fear :( | 16:22 |
| *** Apoorva has joined #openstack-infra | 16:23 | |
| frickler | on http://zuulv3.openstack.org/static/stream.html?uuid=4f5786992cc443fcbfc574cc5344d3b4&logfile=console.log | 16:23 |
| dhellmann | frickler : which job is that? | 16:23 |
| jeblair | AJaeger: good, that should be fixed by the change i just made | 16:24 |
| dmsimard | jeblair: fyi https://review.openstack.org/#/c/513199/ lgtm but just wanted to get your ack | 16:24 |
| frickler | dhellmann: release-openstack-python on the dd... change | 16:24 |
| dhellmann | ok. I'm not sure that needs yaml. Did it fail because of that before? | 16:24 |
| frickler | dhellmann: hmm, maybe I'm mixing things up now. I thought that this was the one that https://review.openstack.org/513497 was supposed to fix. but maybe I'd better EOD now :-/ | 16:25 |
| AJaeger | jeblair: how quick is a change to project-config or openstack-zuul-job used by the next job? Do we still have puppet run? Or is it immediately? | 16:26 |
| *** vhosakot has joined #openstack-infra | 16:26 | |
| dhellmann | post failure on that one, so something is broken | 16:26 |
| smcginnis | post_failure | 16:26 |
| dhellmann | frickler : ^^ | 16:26 |
| jeblair | AJaeger: near immediate (zuul does it itself, but there's a small delay while events propogate through queues) | 16:26 |
| dhellmann | where do we look for the log for that sort of failure? | 16:26 |
| pabelanger | 2017-10-20 16:24:55.155717 | localhost | HTTPError: 400 Client Error: File already exists. for url: https://upload.pypi.org/legacy/ | 16:26 |
| dhellmann | oh | 16:26 |
| pabelanger | that was on stream | 16:26 |
| smcginnis | Just saw that error. | 16:26 |
| dhellmann | well, that's not a surprise | 16:26 |
| dhellmann | unfortunately it means we didn't get to test the announce job | 16:27 |
| smcginnis | So we're probably OK with that one. Just would need a new patch to try the full path. | 16:27 |
| dhellmann | smcginnis : I'll propose a new release | 16:27 |
| dhellmann | yeah | 16:27 |
| smcginnis | dhellmann: ++ | 16:27 |
| smcginnis | Now interested in the publish-static on the other one. | 16:27 |
| pabelanger | maybe test to testpypi_secret for that job? that will publish to testpypi | 16:28 |
| pabelanger | "bindep_file": "/usr/local/jenkins/common_data/bindep-fallback.txt" | 16:29 |
| pabelanger | that was also from the job, so it didn't use in repo bindep.txt file | 16:30 |
| *** dbecker has quit IRC | 16:30 | |
| dhellmann | smcginnis : https://review.openstack.org/513799 | 16:30 |
| dhellmann | I set that to depend on the requirements update so we can test it all the way through | 16:30 |
| smcginnis | dhellmann: Good plan. | 16:30 |
| dmsimard | pabelanger: logs.o.o is trusty right ? | 16:30 |
| pabelanger | yes | 16:30 |
| dmsimard | thanks. | 16:30 |
| smcginnis | Except when it runs out of space. :) | 16:31 |
| pabelanger | zing | 16:31 |
| * dhellmann slaps knee | 16:31 | |
| smcginnis | Thank you, I'll be here all day. | 16:31 |
| AJaeger | jeblair: cool! | 16:31 |
| *** Goneri has quit IRC | 16:31 | |
| dmsimard | smcginnis: ಠ_ಠ| 16:32 |
| *** trown is now known as trown|lunch | 16:33 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move to dictionary list of projects zuul._projects https://review.openstack.org/513260 | 16:34 |
| dhellmann | now that I think of it, I'll bet this same problem is why the docs specs are out of date. | 16:35 |
| pabelanger | jeblair: Shrews: If you haven't seen, there is an issue on zuulv3-issues about ze06.o.o, no longer listening on port tcp/79, breaking log streaming from that executor. | 16:35 |
| jeblair | AJaeger: does it seem strange that governance job is still running? | 16:35 |
| pabelanger | making sure you are aware, incase we schedule restarts today on executors | 16:35 |
| jeblair | pabelanger: thanks, i was not. if you or Shrews have time to look into it, that'd be great | 16:35 |
| AJaeger | YEAH! We have for the time that I'm aware pushed content to the translatoin server!!!!!!! | 16:36 |
| Shrews | pabelanger: when did it start? do we still have logs for the time it started? | 16:36 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Add some aliases to congress meeting chairs https://review.openstack.org/513470 | 16:37 |
| AJaeger | jeblair: yes, it does - but I'm looking at translations and api-ref right now | 16:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Follow up change for networking-spp creation https://review.openstack.org/512536 | 16:37 |
| pabelanger | Shrews: I want to stay I noticed it on Wednesday evening. Pretty sure we still have logs, and haven't restarted since it stopped listening | 16:37 |
| dhellmann | it is taking quite a while to upload the release site update to the server | 16:37 |
| pabelanger | Shrews: but, due to traveling I haven't looked more into it | 16:38 |
| Shrews | pabelanger: ok. i'll see if i can find something in the logs | 16:38 |
| *** yamahata has joined #openstack-infra | 16:38 | |
| jeblair | dhellmann: sounds like the governance job which is similarly slow | 16:38 |
| dmsimard | dhellmann: don't forget to give us the green light on the outstanding reviews that we put a hold on if the issues have been addressed. | 16:38 |
| jeblair | dhellmann: last output: 2017-10-20 16:24:29.189923 | TASK [Upload docs to static site] | 16:38 |
| dhellmann | jeblair : yeah, I was just peeking at those, too | 16:38 |
| smcginnis | Docs are updated, but just hanging there. | 16:38 |
| dhellmann | jeblair : last output for the release site job: 2017-10-20 16:34:22.246619 | TASK [Upload docs to static site] | 16:39 |
| pabelanger | Shrews: I'm just boarding my last fight here in 15mins, expect to be back to work for 20:00UTC | 16:39 |
| dhellmann | so not as long, but it does appear hung? | 16:39 |
| dhellmann | is there some way to see whether it's actively copying content? | 16:39 |
| jeblair | dhellmann: i've logged into the executor for the governance site | 16:40 |
| dhellmann | dmsimard : which reviews did you have in mind? | 16:40 |
| jeblair | it's ... doing something | 16:40 |
| openstackgerrit | Anastasia Kravets proposed openstack-infra/project-config master: [ec2-api] Increase timeout for rally job https://review.openstack.org/513802 | 16:40 |
| jeblair | i'm seeing multiple ssh processes | 16:40 |
| pabelanger | dhellmann: which job are you looking at? | 16:40 |
| dhellmann | pabelanger : http://zuulv3.openstack.org/static/stream.html?uuid=c7b0bda1c4cd47fcbb5fce1611416915&logfile=console.log | 16:41 |
| *** yamamoto has joined #openstack-infra | 16:41 | |
| jeblair | it seems to be repeatedly spawing short-lived ssh processes | 16:41 |
| dmsimard | dhellmann: let me find them, hang on | 16:41 |
| dhellmann | one per file maybe? | 16:41 |
| jeblair | dhellmann: oh maybe | 16:41 |
| jeblair | i haven't gotten a full commandline out of strace yet | 16:41 |
| dhellmann | that would be terribly inefficient, but unsurprising | 16:42 |
| dmsimard | dhellmann: https://review.openstack.org/#/c/512676/ and https://review.openstack.org/#/c/512788/ | 16:42 |
| pabelanger | are we using with_items? | 16:42 |
| dhellmann | dmsimard : 676 looks ok to go ahead with. I'd like to keep the job moves on hold until we have the q-1 milestone done. What do you think smcginnis ? | 16:43 |
| pabelanger | Oh | 16:43 |
| pabelanger | we are using copy | 16:43 |
| pabelanger | not sync | 16:43 |
| dhellmann | aha | 16:43 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Fix publish location for translations https://review.openstack.org/513803 | 16:43 |
| jeblair | that seems to match the behavior | 16:43 |
| AJaeger | pabelanger: will that copy over the complete directory tree? ^ | 16:43 |
| jeblair | they are sftp processes | 16:43 |
| pabelanger | I wonder if rsync will be more efficient over copy | 16:43 |
| AJaeger | that's the missing bits for translation push! | 16:43 |
| pabelanger | jeblair: yah | 16:44 |
| dhellmann | jeblair : the release publish job did finish and I see the release-test item mentioned on https://releases.openstack.org/queens/index.html as expected | 16:44 |
| jeblair | pabelanger: i'll write a change | 16:44 |
| pabelanger | ++ | 16:44 |
| *** amoralej is now known as amoralej|off | 16:45 | |
| *** kiennt26 has joined #openstack-infra | 16:45 | |
| *** kiennt26 has quit IRC | 16:45 | |
| pabelanger | and boarding, good luck :D | 16:45 |
| dhellmann | pabelanger : safe travels | 16:45 |
| *** kiennt26 has joined #openstack-infra | 16:45 | |
| jeblair | dhellmann: i'd like to change this and test it again, if that's okay? | 16:45 |
| dhellmann | jeblair : yes, let's | 16:46 |
| jeblair | (this seems like too big of an inefficiency to leave as is) | 16:46 |
| dhellmann | I think we're still waiting for that requirements change to merge before we can go end-to-end again | 16:46 |
| AJaeger | dmsimard: publish-api-ref succeeded at http://logs.openstack.org/1b/1b55698fab55d6607aecbec6f37d8074fe9300a2/post/publish-api-ref/5efb542/ - but I do not find the new pages, so wonder where we published to ;/ | 16:46 |
| dmsimard | AJaeger: yay, progress ? | 16:46 |
| jeblair | hehe, check out the first note here: http://docs.ansible.com/ansible/latest/copy_module.html#notes | 16:47 |
| AJaeger | dmsimard: definitely progress! | 16:47 |
| dhellmann | jeblair : nice, at least it's called out | 16:47 |
| AJaeger | dmsimard: /afs/.openstack.org/developer-docs/api-ref/baremetal is the target, so the conversion from ironic to baremtal worked | 16:49 |
| *** caphrim007 has quit IRC | 16:49 | |
| jeblair | so i *think*, from comparing the docs, that we don't need to change any options. by default, copy is doing a recursive copy, and follows the same trailing-slash behavior as synchronize. the synchronize module uses rsync '-a' by default, and does not enable remote delete by default. | 16:49 |
| clarkb | ok slwo start this morning has I got to fight openwrt wireless bridge mode. Turns out it just doesn't work and is likely the cause of my earlier packet loss \o/ | 16:50 |
| clarkb | but I think I have working networking again | 16:50 |
| *** yamamoto has quit IRC | 16:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Use synchronize to upload to static site https://review.openstack.org/513807 | 16:50 |
| dmsimard | clarkb: it looks like your packets made it to the IRC server, yay | 16:51 |
| jeblair | dhellmann, smcginnis, dmsimard: https://review.openstack.org/513807 | 16:51 |
| *** salv-orlando has quit IRC | 16:51 | |
| *** salv-orlando has joined #openstack-infra | 16:52 | |
| dhellmann | jeblair : I'm certainly in favor of testing that to see :-) | 16:52 |
| jeblair | heh, i think the governance site is probably taking so long because of all the badges. | 16:52 |
| *** annp has quit IRC | 16:52 | |
| AJaeger | infra-root, any idea why http://logs.openstack.org/1b/1b55698fab55d6607aecbec6f37d8074fe9300a2/post/publish-api-ref/5efb542/ published to /afs/.openstack.org/developer-docs/api-ref/baremetal but looking at https://developer.openstack.org/api-ref/baremetal/ I see still old content ? | 16:53 |
| dhellmann | the next release test will exercise a couple of things, so I started a list in https://etherpad.openstack.org/p/release-job-failures so we can verify them all | 16:53 |
| dhellmann | jeblair : yeah, there is a *lot* of dynamic content generation for that build | 16:53 |
| jeblair | governance is 76% badges | 16:53 |
| dhellmann | wow | 16:54 |
| AJaeger | oh, we published to https://developer.openstack.org/api-ref/baremetal/html/ ;( | 16:54 |
| AJaeger | Now figuring out what went wrong - but we did publish! | 16:54 |
| jeblair | AJaeger: the job-output.json may be useful in finding exactly which path is wrong (it has all the input and output parameters) | 16:56 |
| dmsimard | AJaeger: let me know if you can't figure it out and I'll poke at it | 16:56 |
| *** salv-orlando has quit IRC | 16:56 | |
| AJaeger | jeblair: yes, that helped me to figure out the /html/ - now trying to see what is wrong in our setting. | 16:56 |
| AJaeger | dmsimard: thanks for the offer, let me stare 5 mins at it first :) | 16:57 |
| *** jpena is now known as jpena|off | 16:57 | |
| AJaeger | missing "/" I guess ;( patch coming... | 17:00 |
| inc0 | hey guys, any chances we can get higher timeout than 5400? I fear that if we do simplest publisher - just build and publish to dockerhub in same job - we won't make it | 17:00 |
| inc0 | either that or we'll need to figure out how to transfer artifact (built images) from one gate to another | 17:01 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix publish jobs https://review.openstack.org/513810 | 17:03 |
| smcginnis | dhellmann: Had stepped away and missed your earlier question. Yes, makes sense to hold on job moves for now. | 17:03 |
| AJaeger | project-config-core, a couple of one char changes at 513810 for review, please | 17:03 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add log streaming logging and exception handling https://review.openstack.org/513811 | 17:04 |
| jeblair | inc0: oh sure, just increase the 'timeout:' setting for your job. we'll probably have a max-timeout for the system, but i expect it to be 2 or 3 hours. | 17:04 |
| jeblair | AJaeger: those are the hardest | 17:04 |
| inc0 | jeblair: 5400 - which means 1.5h | 17:04 |
| *** tosky has quit IRC | 17:04 | |
| inc0 | might not be enough, especially in nodepool uplink scenario | 17:05 |
| jeblair | inc0: are you saying that there is currently a 5400s limit you can't change? i'm confused. | 17:05 |
| inc0 | jeblair: I'm saying that 5400 is maximum timeout zuul allows | 17:06 |
| AJaeger | jeblair: https://governance.openstack.org/tc/ "Last updated on Fri Oct 20 14:52:20 2017" | 17:06 |
| AJaeger | Woot! | 17:06 |
| *** dbecker has joined #openstack-infra | 17:06 | |
| inc0 | and in scenario where we also want to push images up to dockerhub, that might not be enough | 17:07 |
| AJaeger | infra-root, https://review.openstack.org/513621 has retry-limits suddenly | 17:07 |
| jeblair | inc0: can you show me a change where zuul rejected an increase past 5400? | 17:07 |
| *** ihrachys has quit IRC | 17:07 | |
| dmsimard | AJaeger: I know where that is from | 17:08 |
| dmsimard | AJaeger: sending a revert pending we figure it out | 17:08 |
| *** ihrachys has joined #openstack-infra | 17:08 | |
| AJaeger | dmsimard: that comes from rsync not finding files - curious what your sending. | 17:08 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Revert "Move to dictionary list of projects zuul._projects" https://review.openstack.org/513812 | 17:08 |
| dmsimard | AJaeger: http://logs.openstack.org/21/513621/1/gate/tox-linters/a8e4670/ara/result/7e06e5bb-23e8-412b-885a-44ca7933bf68/ | 17:09 |
| inc0 | https://review.openstack.org/#/c/508759/ - patchset 54 | 17:09 |
| *** electrofelix has quit IRC | 17:09 | |
| dmsimard | infra-root: We might need to do a force submit for https://review.openstack.org/513812, it unbreaks use-cached-repos | 17:09 |
| dmsimard | How ironic that the integration tests for use-cached-repos *just* merged https://review.openstack.org/#/c/512927/ | 17:10 |
| inc0 | hold on, I might've failed at dividing numbers | 17:10 |
| dmsimard | oh, the job that includes the use-cached-repos integration tests does not apply on project-config | 17:11 |
| inc0 | yeah, I did, it's 3hrs... math is hard:/ | 17:11 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Move to dictionary list of projects zuul._projects" https://review.openstack.org/513812 | 17:11 |
| inc0 | 3hrs should be enough with healthy margin of error | 17:11 |
| AJaeger | thanks, jeblair for force merging | 17:12 |
| jeblair | inc0: try to not set it too much higher than you think it should take, so that if there's a problem, the timeout is still useful. | 17:12 |
| inc0 | sure, I know, 5400 was me doing this, but for some reason I thought it's limit | 17:13 |
| fungi | okay, i'm back online (needed to find a serial cable so i could manually fsck /var on my openbsd firewall when the power came back on) | 17:13 |
| fungi | i have ~500lines of scrollback in here... should i skim first or just jump in with whatever's broken currently? | 17:13 |
| smcginnis | fungi: Welcome back to the electronic age. | 17:13 |
| fungi | yes, i got tired of bashing rocks together all morning | 17:14 |
| fungi | just about done booting the house back up again | 17:14 |
| smcginnis | On the release front, we may be in good shape. | 17:14 |
| *** harlowja has joined #openstack-infra | 17:14 | |
| clarkb | fungi: you were bashing rocks together too? | 17:14 |
| smcginnis | We had a publish docs job that was hanging or just taking a really long time. | 17:14 |
| fungi | clarkb: i was back in the stone age, so that's what you do i guess | 17:14 |
| clarkb | fungi: I was bashing them together to generate 802.11ac waves to connect my office to my network "closet" | 17:15 |
| AJaeger | fungi: we're making progress! Pushing again to translatoin server, pushing to governance.o.o, pushing api-ref ... A few final fixes needed... | 17:15 |
| smcginnis | Looks like https://review.openstack.org/#/c/513807/ is the fix we needed there. | 17:15 |
| inc0 | "use-cached-repos : Find locally cached git repos" - Failed - you've seen that? | 17:17 |
| *** sambetts is now known as sambetts|afk | 17:17 | |
| frickler | inc0: yeah, broken patch just got reverted | 17:17 |
| inc0 | thank you | 17:17 |
| openstackgerrit | Merged openstack-infra/project-config master: Skip openstack-tox-py35 jobs on release deliverables https://review.openstack.org/512676 | 17:18 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-grenade-dsvm-ceilometer https://review.openstack.org/513162 | 17:18 |
| *** ijw has quit IRC | 17:18 | |
| fungi | so the summary is that this morning's omelets have needed far fewer eggs broken? | 17:18 |
| smcginnis | Pretty much it, yeah. | 17:19 |
| fungi | okay, well i'm around for a bit to help iterate on whatever else is blocking milestone 1 (or general fixes otherwise) for a couple hours, but then unfortunately need to disappear again to run errands i couldn't do this morning while the power trucks had us blocked in | 17:20 |
| mnaser | ^ i am (kinda) a bit familiar with whats going on release and i can help with that as well | 17:21 |
| fungi | at least this morning's outage provided an opportunity to catch up on my languishing yardwork | 17:21 |
| clarkb | chandankumar: you still around? | 17:24 |
| clarkb | chandankumar: is the problem that the user in taht group is your old use? | 17:24 |
| *** gouthamr has quit IRC | 17:24 | |
| chandankumar | clarkb: i think so | 17:24 |
| jeblair | fungi: okay, i'm going to take you up on that offer and afk for a bit now. thanks! :) | 17:25 |
| chandankumar | clarkb: i am not able to see +2 there or add button there | 17:25 |
| fungi | jeblair: you bet! enjoy a nice friday break | 17:25 |
| clarkb | chandankumar: ok let me look at it really quickly to confirm | 17:25 |
| smcginnis | I have some lunch time obligations, but looks like a good time to step away anyway as we wait for a couple patches to work through. | 17:26 |
| clarkb | infra-root can I get reviews on https://review.openstack.org/#/c/513189/ before that becomes a problem? | 17:26 |
| fungi | on it | 17:26 |
| dmsimard | AJaeger: re-looking at https://review.openstack.org/#/c/513260/ I'm embarrassed at how obvious the mistake is :( | 17:26 |
| *** dbecker has quit IRC | 17:27 | |
| * mordred waves to the fine humans - is somewhat around | 17:27 | |
| fungi | mordred: somewhat around seems to be the theme of the day | 17:27 |
| fungi | at least you're in good company | 17:28 |
| AJaeger | dmsimard: that happens ;( No worries ;) | 17:28 |
| clarkb | chandankumar: ya its the old 8944 account | 17:30 |
| * dmsimard sends somewhat of a hello in mordred's direction | 17:30 | |
| clarkb | chandankumar: I'll replace it with the current account if I can remember what that was | 17:31 |
| clarkb | oh I guess I can just use the email addr now since it should be unique | 17:31 |
| clarkb | chandankumar: updated | 17:31 |
| chandankumar | clarkb: let me check | 17:33 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix publish jobs https://review.openstack.org/513810 | 17:33 |
| frickler | mordred: infra-root: shade-functional-devstack-legacy is failing to get a node since 7h, is it just starved by the gate queue or may there be an issue with particular (trusty maybe?) nodes? https://review.openstack.org/500365 | 17:33 |
| chandankumar | clarkb: perfect working fine now | 17:34 |
| chandankumar | clarkb: Thank you :-) | 17:34 |
| openstackgerrit | Daniel Speichert proposed openstack-infra/shade master: Support filtering servers in list_servers using arbitrary parameters https://review.openstack.org/506969 | 17:35 |
| fungi | memory utilization on the scheduler is looking GREAT today. i wonder if that's thanks to the dependent item tree fix from yesterday | 17:35 |
| clarkb | fungi: was it a power outage? | 17:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Use synchronize to upload to static site https://review.openstack.org/513807 | 17:37 |
| *** salv-orlando has joined #openstack-infra | 17:39 | |
| mnaser | there was talks of passing artifacts between jobs in zuul v3 i remember | 17:39 |
| mnaser | did that ever come out to something? | 17:39 |
| *** kiennt26 has quit IRC | 17:39 | |
| *** dave-mcc_ is now known as dave-mccowan | 17:39 | |
| clarkb | mnaser: I think that is on the list of things to work on post rollout | 17:40 |
| mnaser | ah okay | 17:40 |
| clarkb | along with the dashboard and multiple flavors/nodepool drivers etc | 17:40 |
| mnaser | job dependencies made it from what i remember though i think | 17:40 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Run gitdm periodic job only on master https://review.openstack.org/513621 | 17:41 |
| clarkb | yes job dependencies as proper graph made it in | 17:41 |
| mnaser | be awesome for projects to run linters first, run unit tests only if linters pass, run integration/func tests only if unit tests pass, etc | 17:41 |
| *** huanxie has joined #openstack-infra | 17:41 | |
| mnaser | consumes less resource and makes things faster | 17:41 |
| clarkb | mnaser: we actually did that in the long ago | 17:41 |
| *** jamesmcarthur has quit IRC | 17:41 | |
| clarkb | the problem we ran into was it just created more churn as you incrementally made things work. Better to teach devs to use tox -epep8 before pushing :) | 17:42 |
| clarkb | mnaser: basically if you get good feedback on patchset 1 across the jobs hopefully that means patchset 2 passes all jobs rather than having 3 4 5 just to get there | 17:42 |
| mnaser | yep | 17:42 |
| mnaser | oh i see what you mean | 17:42 |
| *** tosky has joined #openstack-infra | 17:42 | |
| *** jcoufal has quit IRC | 17:45 | |
| clarkb | fungi: I think the management event merging may help that too because its fewer iterations of layouts? | 17:45 |
| fungi | oh, i missed that patch | 17:46 |
| fungi | neat idea | 17:46 |
| fungi | clarkb: and yes, extended power outage because the power company is replacing the poles and transformers for local distribution on our street. this morning is when they finally got to the one feeding our house (though all things considered, pretty amazing they can do that in a matter of a few hours) | 17:47 |
| AJaeger | could I get a second review on https://review.openstack.org/#/c/513803/ to fix final bit in translation upload, please? | 17:50 |
| clarkb | AJaeger: looking | 17:50 |
| clarkb | AJaeger: does it need to keep the *.pot suffix? or is everything in that dir a .pot? | 17:51 |
| AJaeger | clarkb: everyting in there is a .pot | 17:52 |
| *** dhinesh has joined #openstack-infra | 17:52 | |
| clarkb | wfm thanks | 17:52 |
| openstackgerrit | Merged openstack-infra/system-config master: Logrotate track upstream logs https://review.openstack.org/513189 | 17:53 |
| AJaeger | thanks, clarkb | 17:53 |
| *** weshay|ruck is now known as weshay|ruck|afk5 | 17:53 | |
| *** claudiub|2 has quit IRC | 17:53 | |
| *** weshay|ruck|afk5 is now known as weshay|afk50min | 17:54 | |
| dhellmann | smcginnis, jeblair: wow, I forgot how long it can take to land a requirements update patch | 17:54 |
| *** kuchi has joined #openstack-infra | 17:54 | |
| *** kuchi has quit IRC | 17:56 | |
| *** kuchi has joined #openstack-infra | 17:56 | |
| *** trown|lunch is now known as trown | 17:58 | |
| *** ldnunes has quit IRC | 18:01 | |
| *** ldnunes has joined #openstack-infra | 18:02 | |
| *** rlandy is now known as rlandy|brb | 18:04 | |
| *** jamesmcarthur has joined #openstack-infra | 18:04 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Improve test coverage of the fetch-zuul-cloner role and the shim https://review.openstack.org/512904 | 18:06 |
| *** baoli has quit IRC | 18:09 | |
| *** baoli has joined #openstack-infra | 18:10 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add build-placement-api-ref https://review.openstack.org/513822 | 18:11 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove now unused legacy placement jobs https://review.openstack.org/513823 | 18:11 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix placement-api jobs https://review.openstack.org/513824 | 18:11 |
| *** jamesmcarthur has quit IRC | 18:11 | |
| *** huanxie has quit IRC | 18:12 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix publish location for translations https://review.openstack.org/513803 | 18:13 |
| mnaser | does anyone recall any recent changes to xenial images? | 18:14 |
| mnaser | puppet integration jobs are failing with a weird timeout where i have no idea why/how they are | 18:15 |
| mnaser | and it happens from time to time as well :( | 18:15 |
| mnaser | doesnt seem to be isolated to a specific cloud provider either | 18:15 |
| dhinesh | when trying to upload images using "nodepool image-upload all dpc", the image shows as queued in the openstack controller, but it eventually fails with error "OpenStackCloudHTTPError: (500) Server Error for url: http://server:9292/v2/images/b751d36b-1acd-46f3-8cb4-c9a3cc2182b7/file Internal Server Error (Inner Exception: Extra data: line 1 column 5 - line 5 column 4 (char 4 - 114))" | 18:15 |
| dhinesh | is this an issue with the server itself? | 18:16 |
| fungi | mnaser: do you think the imtermittency stems from running in specific service providers, or does it appear completely nondeterministic? | 18:17 |
| clarkb | dhinesh: yes, you'll need to check the glance server logs | 18:18 |
| mnaser | fungi: i have two failures here for example, one on rax and one on ovh | 18:18 |
| *** jamesmcarthur has joined #openstack-infra | 18:18 | |
| mnaser | which are pretty.. diverse in terms of config | 18:18 |
| mnaser | http://logs.openstack.org/60/513760/2/check/puppet-openstack-integration-4-scenario002-tempest-ubuntu-xenial/bdcb862/logs/testr_results.html.gz .. its always the same failure, nova timing out while contacting neutron to assign a floating ip | 18:19 |
| fungi | mnaser: do they seem to pause in the same parts of their respective jobs/on the same actions? | 18:19 |
| mnaser | http://logs.openstack.org/60/513760/2/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/dcd2d64/logs/testr_results.html.gz | 18:19 |
| fungi | ahh, so it's timing out on an interaction between openstack services? | 18:19 |
| mnaser | yeah | 18:19 |
| mnaser | and it only happens on xenial | 18:19 |
| fungi | multi-node or aio? | 18:19 |
| clarkb | could you be swapping? | 18:19 |
| mnaser | aio | 18:19 |
| mnaser | hmm, that could be a theory, let me see | 18:20 |
| mnaser | but only xenial is failing (consistently) so maybe it has something weird in it, one sec | 18:20 |
| fungi | okay, so this is all local communication too. at least that narrows it down a smidge | 18:20 |
| mnaser | http://logs.openstack.org/60/513760/2/check/puppet-openstack-integration-4-scenario001-tempest-centos-7/1cbbd68/logs/free.txt.gz <-- successful centos 7 job with 1.8gb of swap and its fine | 18:20 |
| mnaser | http://logs.openstack.org/60/513760/2/check/puppet-openstack-integration-4-scenario002-tempest-ubuntu-xenial/bdcb862/logs/free.txt.gz <-- 5mb swap used and it fails | 18:21 |
| *** jamesmcarthur has quit IRC | 18:21 | |
| mnaser | i'm really at a loss, neutron doesnt show the request arriving (but then why would it, if its timing out) | 18:21 |
| EmilienM | are you aware of multinode issues with some providers? | 18:22 |
| EmilienM | I fail to know which provider is running a job now, in which log is it? | 18:22 |
| clarkb | EmilienM: it is in the inventory file now iirc | 18:22 |
| *** masber has quit IRC | 18:22 | |
| mnaser | EmilienM: zuul-info folder, inventory.yaml | 18:22 |
| EmilienM | oh found it | 18:22 |
| fungi | yeah, you can see it in the inventory | 18:22 |
| EmilienM | I saw a bunch of jobs having issue with multinode setup on rax-ord | 18:22 |
| AJaeger | jeblair: could you review your updated change and +2A if it's fine, please? https://review.openstack.org/#/c/513199/ | 18:23 |
| EmilienM | let me show logs | 18:23 |
| *** felipemonteiro has quit IRC | 18:23 | |
| *** felipemonteiro has joined #openstack-infra | 18:24 | |
| EmilienM | http://logs.openstack.org/08/513808/1/check/legacy-tripleo-ci-centos-7-nonha-multinode-oooq/89738c4/job-output.txt.gz#_2017-10-20_17_44_46_392156 | 18:24 |
| *** yolanda has quit IRC | 18:24 | |
| AJaeger | I have three changes to fix the placement-api-ref jobs, could I get reviews for https://review.openstack.org/513822 and the referenced jobs, please? | 18:25 |
| jeblair | AJaeger, dmsimard: 513199 lgtm, but let's ask dhellmann, fungi, and smcginnis whether it's okay to merge that tox-siblings change now, or if we need to wait until after q-1 | 18:26 |
| clarkb | EmilienM: looking at ara I don't see it running the pre plays for setting up the overlay | 18:26 |
| AJaeger | jeblair: good point. Do you want to WIP? | 18:26 |
| clarkb | oh its al egacy job | 18:26 |
| * clarkb digs in more | 18:26 | |
| dhellmann | jeblair, AJaeger , fungi : I'll let smcginnis make the call, but if it was up to me I'd wait. | 18:27 |
| dhellmann | unless there's something blocked on that related to q-1 | 18:27 |
| *** edmondsw has quit IRC | 18:28 | |
| fungi | i guess the way in which we disabled tox-siblings by turning it into a no-op makes it harder to test out the fixed tox-siblings without reenabling it across all tox-based jobs | 18:28 |
| jeblair | fungi: nah, we can depends-on that change | 18:29 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Renaming collectd-ceilometer-plugin to collectd-openstack-plugins https://review.openstack.org/500768 | 18:29 |
| fungi | oh! yes, i guess we can ;) | 18:30 |
| clarkb | EmilienM: that job doesn't appear to grab the log file for the vxlan setup | 18:31 |
| EmilienM | ok let me look how we can get logs on this one | 18:32 |
| clarkb | EmilienM: http://logs.openstack.org/08/513808/1/check/legacy-tripleo-ci-centos-7-nonha-multinode-oooq/89738c4/job-output.txt.gz#_2017-10-20_17_44_45_047525 we want /home/zuul/vxlan_networking.sh.log according to that | 18:32 |
| EmilienM | ah we have it | 18:33 |
| EmilienM | a sec | 18:33 |
| EmilienM | http://logs.openstack.org/08/513808/1/check/legacy-tripleo-ci-centos-7-nonha-multinode-oooq/89738c4/logs/undercloud/home/zuul/vxlan_networking.sh.log.txt.gz | 18:33 |
| EmilienM | clarkb: this one? | 18:33 |
| clarkb | EmilienM: ya | 18:35 |
| *** slaweq has joined #openstack-infra | 18:35 | |
| AJaeger | fungi, did we fix the readthedocs failure with SSL certification validation error to access to it? | 18:39 |
| clarkb | EmilienM: one thing I notice in that log is that the job is installing ovs from delorean | 18:42 |
| clarkb | EmilienM: could it be possible that ovs is just not working? | 18:42 |
| AJaeger | infra-root, ianw: For the rename: openstack/collectd-ceilometer-plugin has a .zuul.yaml file. We need to update that one as well as part of the rename. | 18:43 |
| EmilienM | clarkb: it would fail everywhere in that case, let me check | 18:43 |
| *** kuchi has quit IRC | 18:43 | |
| *** erlon has quit IRC | 18:44 | |
| *** slaweq has quit IRC | 18:44 | |
| jeblair | AJaeger: ugh. we want to drop the name requirement from in-repo project definitions, but no one has gotten around to that yet. | 18:44 |
| fungi | AJaeger: afaik the rtd cert verification problem was fixed early this week by exposing /etc/ssl/certs read-only into bubblewrap environments on the executors | 18:44 |
| jeblair | AJaeger: we're going to have to force-merge that change to the repo while it is out of the zuul config (or zuul is offline) | 18:44 |
| *** kuchi has joined #openstack-infra | 18:44 | |
| EmilienM | clarkb: no change on OVS, and it works on some other providers, weird. Let me debug - I'll let you know | 18:44 |
| clarkb | EmilienM: reading the rest of that log it seems fine. the remote and local IPs look good and its using a high vxlan id to avoid conflicts with eg neutron | 18:45 |
| EmilienM | on ovh-gra1 as well | 18:45 |
| clarkb | EmilienM: mtus appear to be set as do the IPs | 18:45 |
| EmilienM | so it's not just one provider | 18:45 |
| clarkb | EmilienM: oh one thing to check is that the firewall is opened between the nodes | 18:45 |
| * clarkb looks for that | 18:45 | |
| fungi | jeblair: i suppose the steps could be X. stop zuul, Y. submit change for merge in gerrit, Z. stop gerrit | 18:46 |
| EmilienM | openvswitch-2.7.2-3.1fc27.el7.x86_64 on failing / working jobs | 18:46 |
| fungi | rather than having to do a dance to disable the project in zuul, take everything down, do the rename work, bring everything back up, add the project anew, then add back the jobs | 18:46 |
| clarkb | EmilienM: ya firewall is opened up be the pre playbooks | 18:47 |
| clarkb | s/be/by/ | 18:47 |
| EmilienM | I'm going to create a query | 18:47 |
| AJaeger | fungi: that did not work - see http://logs.openstack.org/fc/fc1fe410ef8b497553adfef76ffefc0a80890503/post/trigger-readthedocs/ffe9b79/ara/ | 18:48 |
| AJaeger | jeblair: or remove the file and readd? | 18:48 |
| *** contra-test-bot has joined #openstack-infra | 18:48 | |
| AJaeger | fungi, it's also on https://etherpad.openstack.org/p/release-job-failures | 18:48 |
| jeblair | dhellmann: are you waiting on any changes currently in gate? | 18:49 |
| dhellmann | jeblair : https://review.openstack.org/#/c/513784/ | 18:50 |
| dhellmann | when that merges, we can approve the new release of release-test, which will trigger all new sets of jobs and test end-to-end, including submitting a patch to the requirements repo | 18:51 |
| dhellmann | jeblair : notes in https://etherpad.openstack.org/p/release-job-failures | 18:51 |
| dhellmann | line 32 | 18:51 |
| fungi | AJaeger: i could'a sworn there was a patch for it. hunting now | 18:51 |
| dhellmann | oops 21 | 18:51 |
| *** weshay|afk50min is now known as weshay|ruck | 18:51 | |
| smcginnis | Back from errands. Any new fires or victories? | 18:52 |
| jeblair | okay. i need to start finding a time to perform a scheduler restart, but don't want to disrupt things. maybe as soon as that lands (it just started some devstack jobs, so it's probably 1-1.5 hours out) i can do it. before we do the next test release. | 18:52 |
| jeblair | dhellmann: ^ | 18:53 |
| clarkb | fungi: AJaeger I bet that doesn't work because /etc/ssl/certs is just full of symlinks to elsewhere | 18:53 |
| *** ijw has joined #openstack-infra | 18:53 | |
| clarkb | so we also have to bind mount the destination of those links | 18:53 |
| smcginnis | jeblair: That's probably a good time for it. | 18:53 |
| clarkb | /usr/share/ca-certificates/mozilla/ | 18:53 |
| *** ldnunes has quit IRC | 18:53 | |
| jeblair | smcginnis: seems like we're waiting on that change. also, i think we want you to make the call on whether we land https://review.openstack.org/513199 now or wait until after q-1. | 18:53 |
| smcginnis | Looking.. | 18:54 |
| fungi | jeblair: we're also 1 hour from the project rename maintenance, wherein we plan to stop/start zuul anyway, right? is that good timing? | 18:54 |
| *** baoli has quit IRC | 18:54 | |
| jeblair | smcginnis: we disabled the tox-siblings role because of errors found in some release jobs; that should fix those errors, but we've also worked around them in the interim. so that's entirely optional at this point. | 18:54 |
| dhellmann | jeblair, smcginnis: that sounds good. We'll probably get one more test of the release jobs in today with that plan, and we can pick up on monday if those fail. | 18:54 |
| jeblair | fungi: yeah, i guess that's the plan then. :) | 18:54 |
| *** baoli has joined #openstack-infra | 18:54 | |
| jeblair | fungi: i do have my doubts as to whether 513784 will actually land before 20:00 though... | 18:55 |
| smcginnis | jeblair: I'd prefer if we could wait until we get the q-1 releases out of the way if it's not time critical. | 18:55 |
| *** baoli has quit IRC | 18:55 | |
| fungi | jeblair: we could probably push out the maintenance a bit | 18:55 |
| clarkb | I think ianw had a thing later in the day but that is 2 hours after we start iirc | 18:55 |
| jeblair | smcginnis: i think that's okay. i'll wip it and ask mordred to let us know if waiting would cause a problem i don't anticipate. | 18:55 |
| fungi | or if ianw prefers, we could reschedule the rename work for after milestone 1 too, and give ourselves a little more time to figure out the rename process | 18:56 |
| smcginnis | jeblair: Good plan. | 18:56 |
| *** eroux has joined #openstack-infra | 18:56 | |
| fungi | clarkb: yeah, by a little i meant like maybe 30 minutes or something | 18:56 |
| *** baoli has joined #openstack-infra | 18:56 | |
| fungi | i know he's got a hard-stop at 22:00z | 18:56 |
| *** rlandy|brb is now known as rlandy | 18:57 | |
| fungi | i'm mildly worried that we'll end up inflicting an unnecessarily lengthy outage while we fumble through our first project rename under zuul v3 and getting teh right changes in to true up zuul's config so it's not insta-broken when we bring it back online | 18:57 |
| smcginnis | Anyone look at the readthedocs failure? | 18:57 |
| fungi | smcginnis: i though we had a fix merged for that, so i'm currently digging to find whether it actually merged | 18:58 |
| jeblair | i know we did not plan on still being in the release window when we scheduled the rename. i think deferring it would be reasonable. | 18:58 |
| jeblair | (still need to do the zuul restart, but that's more predictably disruptive) | 18:59 |
| clarkb | fungi: re the fix did you see my comments about about the symlinks | 18:59 |
| jeblair | i'm going to grab lunch while that change lands | 18:59 |
| clarkb | fungi: I think the fix did merge, but was incomplete | 18:59 |
| clarkb | jeblair: ya | 19:00 |
| *** contra-test-bot has quit IRC | 19:00 | |
| clarkb | I'd be fine with delaying the rename as well. | 19:00 |
| *** eroux has quit IRC | 19:00 | |
| *** contra-test-bot has joined #openstack-infra | 19:00 | |
| fungi | ianw: as soon as you see this, it seems like we're leaning toward giving you back your saturday morning | 19:01 |
| *** contra-test-bot has quit IRC | 19:01 | |
| *** contra-test-bot has joined #openstack-infra | 19:02 | |
| clarkb | fungi: I too am having a hard time finding where we specify the bind mounts for brwrap | 19:02 |
| clarkb | fungi: it is in place on ze01 at least | 19:03 |
| *** contra-test-bot has quit IRC | 19:03 | |
| *** jamesmcarthur has joined #openstack-infra | 19:03 | |
| clarkb | aha system-config/manifests/site.pp | 19:03 |
| clarkb | I think it likely that we need to add that second path to the trusted ro paths | 19:04 |
| clarkb | and then restart the executors | 19:04 |
| clarkb | unless we already mount all of /usr/share anyawys? | 19:04 |
| *** contra-test-bot has joined #openstack-infra | 19:04 | |
| *** contra-test-bot has quit IRC | 19:04 | |
| fungi | wouldn't we see that in the mount output? | 19:05 |
| *** ldnunes has joined #openstack-infra | 19:06 | |
| clarkb | fungi: on the root of the executor? I don't think so you'd need to look from within the bwrap namespace I think | 19:06 |
| fungi | yep, just confirmed it doesn't show them all | 19:07 |
| fungi | or at all | 19:07 |
| clarkb | also is a trusted_ro_path mounted 1:1 within the bwrap container? | 19:09 |
| clarkb | that would probably be the other thing to determine | 19:09 |
| tobiash | clarkb: should be | 19:10 |
| clarkb | tobiash: thanks | 19:11 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/irc-meetings master: Remove a couple of stale nova meetings https://review.openstack.org/513830 | 19:11 |
| tobiash | clarkb: but you'll have to define trusted_ro_path and untrusted_ro_path if you want it in trusted and untrusted jobs | 19:11 |
| AJaeger | fungi: change Ib662afbc0e3375a2d461ef7fc6e7e4f8741a700c | 19:13 |
| AJaeger | fungi, https://review.openstack.org/#/c/512657/ - but that merged 3 days ago and we have failures from yesterday | 19:13 |
| EmilienM | clarkb: sorry i'm on phone, back in 5m | 19:14 |
| EmilienM | but it's pretty consistent | 19:14 |
| *** markvoelker has quit IRC | 19:14 | |
| EmilienM | weshay|ruck: ^ fyi | 19:14 |
| *** slaweq has joined #openstack-infra | 19:14 | |
| fungi | AJaeger: yeah, found it, but as clarkb points out the contents of /etc/ssl/certs/* are symlinks into /usr/share/... and so we need to add that as well | 19:14 |
| AJaeger | fungi: yes, agreed. | 19:15 |
| EmilienM | clarkb: I was wondering if we have a new image pushed on providers lately | 19:16 |
| tobiash | clarkb: it is 1:1 https://github.com/openstack-infra/zuul/blob/feature/zuulv3/zuul/driver/bubblewrap/__init__.py#L113 | 19:17 |
| clarkb | EmilienM: should be daily, but I doubt that matters much if you are installing ovs from delorean | 19:17 |
| EmilienM | right | 19:17 |
| EmilienM | I'm digging | 19:17 |
| dmsimard | clarkb: what version of puppet is in use for logs.o.o ? 3.x ? | 19:18 |
| clarkb | dmsimard: yes | 19:18 |
| dmsimard | thanks | 19:18 |
| clarkb | EmilienM: if there is a job that hits it more than others we could put a hold on that (though currently also distracted by helping release things) but then we could log into the test nodse and poke at it directly | 19:18 |
| EmilienM | yeah it's the new image I think | 19:19 |
| EmilienM | I compared the rpms between a recent working job and a failing | 19:19 |
| EmilienM | https://www.diffchecker.com/huccl3ol | 19:19 |
| EmilienM | ok using the diff isn't helpful since we deployed an undercloud afterward | 19:20 |
| clarkb | EmilienM: thought the kernel and ovs don't change in that diff | 19:20 |
| EmilienM | right | 19:20 |
| clarkb | er *though | 19:20 |
| clarkb | I would expect those two items to be the package related problems if any | 19:20 |
| clarkb | I've got to put dinner in the oven, back ina bit | 19:20 |
| EmilienM | iptables version changed | 19:20 |
| EmilienM | upgrade to iptables-1.4.21-18.2 | 19:21 |
| AJaeger | hwoarang: http://logs.openstack.org/0e/0ec124bb3a184754ea8bf6934695127be9dc8474/post/propose-updates/f3ac211/ - that looks fine, doesn't it? | 19:22 |
| AJaeger | hwoarang: but did not push any changes to other repos | 19:22 |
| hwoarang | AJaeger: yeah it didn't work | 19:22 |
| hwoarang | 2017-10-20 19:20:48.883669 | ubuntu-xenial | + git branch | 19:22 |
| hwoarang | 2017-10-20 19:20:48.884112 | ubuntu-xenial | + grep -q '^ refs/heads/master$' | 19:22 |
| hwoarang | 2017-10-20 19:20:48.887280 | ubuntu-xenial | + '[' -n '' ']' | 19:22 |
| hwoarang | BRANCH is still empty so nothing happens | 19:23 |
| hwoarang | for whatever reaon git branch | grep -q '^ refs/heads/master$' never matches anything | 19:23 |
| hwoarang | ohhhh | 19:23 |
| hwoarang | yeah no idea why | 19:24 |
| hwoarang | this refs/heads/master looks suspicious | 19:25 |
| EmilienM | I don't get it, we shouldn't have this version of iptables, latest is https://centos.pkgs.org/7/centos-x86_64/iptables-1.4.21-18.0.1.el7.centos.x86_64.rpm.html | 19:26 |
| *** salv-orlando has quit IRC | 19:27 | |
| weshay|ruck | EmilienM, I have query if you want me to send it up | 19:27 |
| *** salv-orlando has joined #openstack-infra | 19:27 | |
| EmilienM | ok so we had an import yesterday: https://git.centos.org/summary/?r=rpms/iptables.git | 19:28 |
| EmilienM | changelog is here : (still digging) https://git.centos.org/blob/rpms!iptables.git/4d0bb22ecdc68576e7a4a7afd966207c50b664d2/SPECS!iptables.spec#L278 | 19:29 |
| AJaeger | hwoarang: could document on https://etherpad.openstack.org/p/zuulv3-issues , please? Somebody needs to look into it... | 19:29 |
| EmilienM | weshay|ruck: send me query please | 19:29 |
| EmilienM | weshay|ruck: I want to know since when we have that | 19:29 |
| weshay|ruck | k | 19:29 |
| EmilienM | but i'm pretty sure it's in the new image | 19:30 |
| EmilienM | (in centos I mean) | 19:30 |
| hwoarang | AJaeger: yeah I will | 19:30 |
| AJaeger | fungi, jeblair: the propose-updates job as post job has ZUUL_REFNAME=refs/heads/master - is that really correct? I think this is different to what v2 did... | 19:31 |
| *** salv-orlando has quit IRC | 19:31 | |
| EmilienM | I would be curious to see if it's only tripleo jobs or if it's also other multinode jobs in OpenStack that fail | 19:32 |
| EmilienM | (probably only tripleo) | 19:32 |
| *** felipemonteiro has quit IRC | 19:32 | |
| fungi | AJaeger: should be able to tell by checking the vars list archived with the logs from a v2 run of the equivalent job | 19:33 |
| hwoarang | AJaeger: i documented it. last bullet point in the 'job problems' section | 19:33 |
| EmilienM | weshay|ruck: I created https://bugs.launchpad.net/tripleo/+bug/1725451 | 19:35 |
| openstack | Launchpad bug 1725451 in tripleo "CI: new centos7 image contains a regression that prevents multinode networking setup to work" [Critical,Triaged] | 19:35 |
| AJaeger | fungi: indeed, was "master" before. | 19:35 |
| weshay|ruck | EmilienM, k.. I'll push a query | 19:36 |
| AJaeger | jeblair: is that a bug - or should we handle refs/heads/master now as REFNAME ? | 19:36 |
| *** huanxie has joined #openstack-infra | 19:36 | |
| EmilienM | weshay|ruck: I'm not sure about your query | 19:36 |
| EmilienM | weshay|ruck: we need to catch the ping error | 19:36 |
| weshay|ruck | EmilienM, I think we sould change how qs pings then | 19:36 |
| weshay|ruck | want to create a log for it | 19:36 |
| EmilienM | weshay|ruck: try in http://logstash.openstack.org/#/dashboard/file/logstash.json and you won't have hits today | 19:37 |
| EmilienM | weshay|ruck: so query doesn't work | 19:37 |
| *** salv-orlando has joined #openstack-infra | 19:37 | |
| mordred | AJaeger: I think we should fix that in the script ... one sec | 19:38 |
| AJaeger | mordred: on it... | 19:38 |
| mordred | AJaeger: I think we can replace it with ZUUL_BRANCH | 19:38 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix propose_update REFNAME handling https://review.openstack.org/513835 | 19:38 |
| mordred | AJaeger: or what you did :) | 19:39 |
| weshay|ruck | EmilienM, it has errors from yesterday ( is the latest )http://paste.openstack.org/show/624218/ | 19:39 |
| *** slaweq has quit IRC | 19:39 | |
| AJaeger | mordred: I'm fine either way... | 19:39 |
| AJaeger | hwoarang: ^ | 19:39 |
| *** slaweq has joined #openstack-infra | 19:39 | |
| mordred | AJaeger: +2 from me - I prefer fix existing to not existing | 19:40 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Fix doc typo https://review.openstack.org/513836 | 19:40 |
| hwoarang | AJaeger: thanks | 19:41 |
| mordred | Shrews: there is a shade job in the check queue that has one job waiting on a node for 9 hours ... this makes me think there might be an actual problem and not just a lot of jobs/capacity | 19:41 |
| mordred | Shrews: what's the best way to investigate such a thing? | 19:42 |
| smcginnis | AJaeger: 513835 will affect all requirements updates, right? | 19:42 |
| hwoarang | mordred: we struggle to upload custom logs from the openstack-ansible logs to logs.openstack.org. I notice that the upload-jobs role expects custom logs to be in zuul.executor.log_root | 19:42 |
| AJaeger | smcginnis: it should enable them ;) | 19:42 |
| hwoarang | we move all logs there in our run playbook, but the post-run one doesn't copy them to the log server | 19:42 |
| hwoarang | any clues on what we may be missing? | 19:42 |
| mordred | hwoarang: can you point me to a sample job? | 19:43 |
| Shrews | mordred: you have to prep zuul logs for the request id then use that to see what nodepool is doing with it | 19:44 |
| *** gordc has joined #openstack-infra | 19:44 | |
| Shrews | grep | 19:44 |
| hwoarang | mordred: that's the log http://logs.openstack.org/06/513706/11/check/openstack-ansible-functional-opensuse-423/7156839/job-output.txt.gz and this is the jobhttps://github.com/openstack/openstack-ansible-tests/blob/master/zuul.d/jobs.yaml#L60 | 19:44 |
| hwoarang | mordred: over here http://logs.openstack.org/06/513706/11/check/openstack-ansible-functional-opensuse-423/7156839/job-output.txt.gz#_2017-10-20_19_20_03_197599 you can see that we rsync the logs to the appropriate location but still we can't see them published | 19:44 |
| *** markvoelker has joined #openstack-infra | 19:45 | |
| * hwoarang is fairly sure he is missing something obvious | 19:45 | |
| mordred | hwoarang: looking | 19:45 |
| AJaeger | project-config-core, could you +2A https://review.openstack.org/513835 - to fix propose-update jobs | 19:45 |
| mordred | Shrews: 2017-10-20 10:11:08,546 DEBUG zuul.Pipeline.openstack.check: Adding node request <NodeRequest 200-0000539944 <NodeSet openstack-single-node OrderedDict([('controller', <Node None controller:ubuntu-xenial>)])OrderedDict([('tempest', <Group tempest ['controller']>)])>> for job shade-functional-devstack-legacy to item <QueueItem 0x7fbf98113e80 for <Change 0x7fbe78b3dc50 500365,31> in check> | 19:45 |
| mordred | Shrews: 200-0000539944 is the one yeah? | 19:45 |
| tobiash | dmsimard: comment on 509436 regarding the failed gate tests | 19:45 |
| dmsimard | tobiash: you're correct | 19:46 |
| Shrews | mordred: that is a request id, yes | 19:47 |
| dmsimard | tobiash: thank god for the integration tests I wrote :D | 19:47 |
| mordred | Shrews: cool. thanks | 19:47 |
| tobiash | :) | 19:47 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header https://review.openstack.org/509436 | 19:47 |
| dmsimard | tobiash: fixed ^ | 19:47 |
| dmsimard | AJaeger: looking | 19:47 |
| *** edmondsw has joined #openstack-infra | 19:47 | |
| gordc | hi, just curious, but is there a list of approved projects that you can put in "require-projects" somewhere? | 19:47 |
| gordc | asking because of https://review.openstack.org/#/c/513764/ | 19:48 |
| tobiash | dmsimard: +2 | 19:48 |
| clarkb | and back | 19:48 |
| AJaeger | thanks, dmsimard | 19:48 |
| dmsimard | gordc: openstack/gnocchi does not exist | 19:48 |
| fungi | mordred: fixes in the job aside, having the format of the ZUUL_BRANCH legacy variable change in v3 seems not particularly good? | 19:49 |
| fungi | er, ZUUL_REFNAME i mean | 19:49 |
| AJaeger | dmsimard: I could use your help - we upload translations now to http://tarballs.openstack.org/translation-source/oslo.db/oslo.db/master instead of http://tarballs.openstack.org/translation-source/oslo.db/master - see the double oslo.db | 19:49 |
| gordc | dmsimard: well it did :) | 19:49 |
| dmsimard | gordc: ultimately projects authorized there are https://github.com/openstack-infra/project-config/blob/master/zuul/main.yaml#L7 | 19:49 |
| *** markvoelker has quit IRC | 19:49 | |
| mordred | fungi: it's using ZUUL_REFNAME though - I'm pretty sure thatwas refs/heads/foo before ? | 19:49 |
| dmsimard | gordc: 'required-projects' makes zuul pre-clone/prepare that project inside the job workspace | 19:50 |
| mordred | fungi: maybe we mistakenly changed the scrit from branch to refname? | 19:50 |
| AJaeger | dmsimard: log file http://logs.openstack.org/fa/fa578a91a134fec9212c742205a7b717d4bacd2f/post/upstream-translation-update/92311d3 - do you have an idea where this gets duplicated? | 19:50 |
| dmsimard | AJaeger: looking | 19:50 |
| fungi | mordred: logged "vars" list from a v2 job confirms ZUUL_REFNAME was just "master" and not "refs/heads/master" | 19:50 |
| gordc | dmsimard: kk, so there's a list. i guess i'll need to hack it another way to replace zuulv2 ocata job | 19:51 |
| dmsimard | gordc: you can do a manual git clone or something | 19:51 |
| mordred | hwoarang: SO - zuul.executor.work_root is a directory on the executor, which is the machine on which ansible is running - not the node on which the job is running | 19:51 |
| mordred | hwoarang: what we should do is add a post playbook to that job that does an synchronize: pull task | 19:52 |
| gordc | dmsimard: yeah, we do that for pike/master jobs (we do pip install). | 19:52 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix propose_update REFNAME handling https://review.openstack.org/513835 | 19:52 |
| mordred | hwoarang: one sec - lemme make you a quick patch | 19:52 |
| gordc | our ocata job used teh enable_plugin functionality though so was hoping to reuse it. | 19:52 |
| EmilienM | clarkb: I'm trying to see if other multinode centos jobs fail or if it's just tripleo | 19:52 |
| hwoarang | mordred: oh i thought job and ansible run on the same host | 19:52 |
| gordc | guess not possibel. | 19:52 |
| *** edmondsw has quit IRC | 19:52 | |
| dmsimard | AJaeger: I'm really restraining myself from writing a dirty "role and job finder" because I know it would probably be best to write it cleanly and expose it as a Zuul API or something | 19:52 |
| hwoarang | interesting | 19:52 |
| AJaeger | ;) | 19:52 |
| AJaeger | dmsimard: upstream-translation-update is in project-config | 19:52 |
| dmsimard | AJaeger: I have half the code written, I write a bit each time I have to look for something | 19:52 |
| AJaeger | dmsimard: fetch-translation-output is in zuul-jobs | 19:53 |
| mordred | hwoarang: nope - ansible is executed on one of the zuul executor hosts | 19:53 |
| *** abishop has joined #openstack-infra | 19:53 | |
| AJaeger | dmsimard: Looking forward to it | 19:53 |
| ianw | fungi: ok ... happy to wait on the rename | 19:53 |
| mordred | hwoarang: incidentally, I have a proposal written up and the first patches to change the interface to be more similar to whatyou are doing there ... a dir on the remote node for you to put logs in to | 19:53 |
| jeblair | mordred, fungi, AJaeger: the refname change is actually a gerrit change | 19:54 |
| *** markvoelker has joined #openstack-infra | 19:54 | |
| fungi | ianw: thanks! i guess go enjoy your weekend! | 19:54 |
| fungi | jeblair: ooh, so this changed when we upgraded to 2.13? | 19:54 |
| hwoarang | mordred: i see. ok thank you for the explanation | 19:54 |
| EmilienM | which dsvm job is running on centos7? it's not in the jobname anymore | 19:55 |
| AJaeger | jeblair: on the 13th we used 2.13 and there ZUUL_REFNAME was still "master" | 19:55 |
| clarkb | EmilienM: there should be an explicit centos7 dsvm job but its single node iirc | 19:55 |
| EmilienM | clarkb: ok so we might be the only ones then | 19:55 |
| AJaeger | jeblair: with v2 | 19:55 |
| jeblair | fungi: yes. though at this point, zuul v3 has compatability code that pulls forward | 19:56 |
| dmsimard | AJaeger: looking at https://github.com/openstack-infra/zuul-jobs/tree/master/roles/publish-artifacts-to-fileserver "The remote path. Content will be put into a directory below this path that matches ``zuul.project.short_name``." seems like a likely culprit | 19:56 |
| fungi | jeblair: got it. so v2 worked around the gerrit behavior change and v3 dropped the workaround as far as generating legacy envvars is concerned? | 19:57 |
| jeblair | fungi: i don't know what v2 did; perhaps so, and that's what AJaeger is reporting | 19:58 |
| clarkb | EmilienM: also you run your own overlay stuff | 19:58 |
| clarkb | EmilienM: I'm not sure anyone else is running that script | 19:58 |
| EmilienM | weshay|ruck: why do we run our own overlay scripts in quickstart? | 19:58 |
| EmilienM | weshay|ruck: why don't we use what infra provides? | 19:58 |
| AJaeger | jeblair: understood | 19:58 |
| jeblair | fungi, AJaeger: but if you tell me we should add extra code to the legacy vars, i'll believe you. | 19:59 |
| *** markvoelker has quit IRC | 19:59 | |
| weshay|ruck | EmilienM, what does infra provide in this case, please point me at it | 19:59 |
| *** gordc has left #openstack-infra | 20:00 | |
| clarkb | weshay|ruck: EmilienM there is the flag in devstack-gate for "legacy" jobs and there is the multi node network overlay role for zuulv3 native jobs | 20:00 |
| EmilienM | let's just use that | 20:01 |
| *** makowals has joined #openstack-infra | 20:01 | |
| EmilienM | and stop using our own stuffs | 20:01 |
| weshay|ruck | agree | 20:01 |
| mordred | hwoarang: https://review.openstack.org/513706 [DNM]zuul.d: run.yml: Use new BUILD_DIR variable for log collection | 20:01 |
| mordred | hwoarang: updated it with a post playbook to copy the logs back | 20:01 |
| AJaeger | jeblair: depends in how many problems we ran | 20:01 |
| dmsimard | weshay|ruck, EmilienM: IIRC tripleo used devstack-gate' scripts to create the bridges and when we took it out, it broke and pabelanger copied it in-tree. | 20:01 |
| EmilienM | yeah I remember | 20:02 |
| clarkb | dmsimard: we didn't take it out so much as move it, it is still there | 20:02 |
| dmsimard | weshay|ruck, EmilienM: I don't know if tripleo needs any "custom" bridges, but there is a generic role to set things up now if you want | 20:02 |
| dmsimard | clarkb: we took it out of functions.sh | 20:02 |
| mordred | hwoarang: this job is actually set up very well for the new method - once that's in place, you'll just be able to drop the post playbook | 20:02 |
| fungi | jeblair: i'll try to do some digging to figure out what gerrit value v2 was embedding in that variable and how it was transforming it (if at all), but for stuff running in post pipelines the refname used to just be the name of the branch rather than the full refs/heads/branch | 20:02 |
| AJaeger | dmsimard: I found it - the translation jobs use translation-source/$PROJECT - and the role adds proejct as well. | 20:02 |
| mordred | hwoarang: although since it's rsync it's obviously safe if we run it twice :) | 20:02 |
| AJaeger | Let me remove the $PROJECT from translation job... | 20:02 |
| AJaeger | dmsimard: thanks for your help | 20:03 |
| tbarron | https://review.openstack.org/#/c/513076/ should be ready to go now (the change to define the legacy jobs in manila itself merged) | 20:03 |
| *** markvoelker has joined #openstack-infra | 20:03 | |
| dmsimard | EmilienM, weshay|ruck: https://github.com/openstack-infra/zuul-jobs/tree/master/roles/multi-node-bridge -- see https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/zuul.d/jobs.yaml#L57-L97 and https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/tests/multi-node-bridge.yaml for examples | 20:04 |
| hwoarang | mordred: awesome thank you very much | 20:04 |
| hwoarang | mordred: could you let us know when the new method is in place? :) than kyou | 20:04 |
| dmsimard | AJaeger: bah you beat me to it | 20:04 |
| dmsimard | good job :p | 20:04 |
| mordred | hwoarang: I shall (although I'll be trying to let *everyone* know) | 20:04 |
| hwoarang | :) | 20:04 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove double $PROJECT from translation publishing https://review.openstack.org/513839 | 20:05 |
| AJaeger | dmsimard: ^ | 20:05 |
| AJaeger | mordred: do you have time to work on the javascript PTI jobs? | 20:05 |
| EmilienM | dmsimard: ok so tripleo multinode should just depeend on "multinode-integration" ? | 20:06 |
| mordred | AJaeger: yah - I'll put that on my list next | 20:06 |
| AJaeger | thanks, mordred | 20:06 |
| dmsimard | EmilienM: multinode-integration is a job that tests the various multinode roles | 20:06 |
| EmilienM | oh ok | 20:06 |
| dmsimard | EmilienM: you shouldn't depend on that -- I don't know what you need or what you're after, I'm missing a lot of context here | 20:07 |
| *** DuncanT has quit IRC | 20:07 | |
| dmsimard | EmilienM: what's the problem ? | 20:07 |
| EmilienM | dmsimard: https://bugs.launchpad.net/tripleo/+bug/1725451 | 20:07 |
| openstack | Launchpad bug 1725451 in tripleo "CI: new centos7 image contains a regression that prevents multinode networking setup to work" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 20:07 |
| mordred | Shrews: by "see what nodepool is doing with it" - does that mean zk_shell? | 20:07 |
| *** jtomasek has quit IRC | 20:07 | |
| mordred | Shrews: (it seems I may not have debugged such a thing yet) | 20:07 |
| mordred | ah - request-list seems promising | 20:07 |
| *** markvoelker has quit IRC | 20:07 | |
| *** DuncanT has joined #openstack-infra | 20:08 | |
| clarkb | fungi: AJaeger is there a change for the certs path thing yet? | 20:08 |
| fungi | clarkb: oh, no but i'll push it up in a sec | 20:09 |
| EmilienM | weshay|ruck: also selinux-policy was updated but I don't think we enforce anyway | 20:09 |
| weshay|ruck | EmilienM, ya.. we don't | 20:09 |
| EmilienM | I bet it's iptables | 20:09 |
| *** yolanda has joined #openstack-infra | 20:10 | |
| dmsimard | EmilienM: legacy-tripleo-ci-centos-7-nonha-multinode-oooq already inherits from legacy-tripleo-ci-dsvm-multinode which runs https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/playbooks/legacy/multinode-networking/pre.yaml | 20:10 |
| dmsimard | EmilienM: that configures the firewall on each node part of a multinode job to accept all traffic on all ports from all nodes | 20:10 |
| clarkb | dhellmann: I think http://logs.openstack.org/25/513825/4/check/legacy-tempest-dsvm-py35/c31deb2/logs/devstacklog.txt.gz#_2017-10-20_20_07_54_838 is going to majorly break us with pip 10 | 20:10 |
| mordred | Shrews, tobiash: http://paste.openstack.org/show/624223/ | 20:11 |
| clarkb | the good news is I think that will majorly break everyone so maybe we can get pip to be a little less faily about it | 20:11 |
| * AJaeger waves good night and wishes everybody a great weekend | 20:11 | |
| pabelanger | dmsimard: do you mind using git.openstack.org for your URL to git? We don't want want people assuming github is our source of truth | 20:11 |
| tobiash | mordred: is the nodepool webapp exposed on the openstack infra? | 20:11 |
| dmsimard | pabelanger: I have a significant preference towards github in terms of browsing UX versus cgit :( | 20:12 |
| *** markvoelker has joined #openstack-infra | 20:12 | |
| EmilienM | dmsimard: thx for confirming | 20:12 |
| EmilienM | clarkb: could we hold a node please? | 20:12 |
| mordred | tobiash: that's a great question ... I don't think so? Shrews? | 20:13 |
| *** markvoelker has quit IRC | 20:13 | |
| clarkb | EmilienM: yes we need to know what job to hold for | 20:13 |
| EmilienM | clarkb: I can tell you, a sec | 20:14 |
| tobiash | I fell in love with it as I don't have to login and run nodepool list etc when debugging :) | 20:14 |
| pabelanger | dmsimard: right, but github is a mirror, git.o.o is what we should be directing users towards | 20:14 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add /usr/share/ca-certificates to trusted_ro_paths https://review.openstack.org/513840 | 20:14 |
| *** markvoelker has joined #openstack-infra | 20:14 | |
| fungi | clarkb: AJaeger: smcginnis: pabelanger: ^ | 20:14 |
| clarkb | fungi: +2 | 20:14 |
| *** ijw has quit IRC | 20:14 | |
| EmilienM | clarkb: http://zuulv3.openstack.org/static/stream.html?uuid=a6ce3a3d77c34de1996f2c9d9f7a3848&logfile=console.log | 20:15 |
| clarkb | fungi: and then we will have to restart the executors | 20:15 |
| EmilienM | this one | 20:15 |
| EmilienM | clarkb: 512978,3 | 20:15 |
| dmsimard | pabelanger: I know that, but you're asking me to use a web interface I am less productive with and frankly missing features I use on a regular basis | 20:15 |
| *** andreas_s has joined #openstack-infra | 20:15 | |
| fungi | dmsimard: just remember that gh is missing the biggest feature of all: it's not free software | 20:16 |
| mordred | tobiash: yah - we don't seem to have it enabled in our config | 20:16 |
| dmsimard | fungi: I know :( | 20:16 |
| dhellmann | clarkb : sigh. | 20:16 |
| EmilienM | fungi: heh it reminds me our little game in Ann Arbor ;-) | 20:16 |
| fungi | indeed! | 20:16 |
| EmilienM | fungi: I still have the pictures | 20:16 |
| clarkb | EmilienM: the way we hold now is by telling zuul to keep a failure around for a project, job name pair | 20:16 |
| EmilienM | folks we can migrate to github, we have a plan already :D | 20:17 |
| clarkb | EmilienM: I've got the job name from that but what is the project? | 20:17 |
| *** ldnunes has quit IRC | 20:17 | |
| tobiash | mordred: what problem are you tracking with this paste? | 20:17 |
| mordred | clarkb, dhellmann oh goodie | 20:17 |
| EmilienM | clarkb: openstack/tripleo-heat-templates | 20:17 |
| dhellmann | clarkb : could that be due to a system package version of pyyaml? or is something trying to do an upgrade in a virtualenv? | 20:17 |
| clarkb | EmilienM: thanks | 20:17 |
| mordred | tobiash: there is a patch in the check queue that has been waiting for a node for the last 9.5 hours | 20:17 |
| clarkb | dhellmann: its pip trying to upgrade pyyaml and blowing up because of the system package being distutils installed | 20:17 |
| mordred | tobiash: those are the last two lines related to it in the debug log | 20:17 |
| pabelanger | dmsimard: no, I'm asking you to share links for other users to git.o.o which is our infrastrucure, and like fungi mention, github is not free software. Keep in mind, we made an effort in all our documentation URLs for git repos, to also point to git.o.o. We should be consistant. | 20:17 |
| dhellmann | clarkb : it seems like having the virtualenv not mirror system site-packages would fix that? | 20:18 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove double $PROJECT from translation publishing https://review.openstack.org/513839 | 20:18 |
| mordred | tobiash: so I was thinking there might be an issue hiding there with quotas and locking | 20:18 |
| dhellmann | clarkb : or not installing the system package, but I guess that's needed for ansible? | 20:18 |
| tobiash | mordred: is citycloud-lon1-main full? | 20:18 |
| dhellmann | oh, wait, this is tempest. it's probably not in a virtualenv? | 20:19 |
| *** dprince has quit IRC | 20:19 | |
| tobiash | mordred: are there failed nodes in there blocking the max-servers? | 20:19 |
| clarkb | dhellmann: yse this is devstack | 20:19 |
| mordred | tobiash: lemme check | 20:19 |
| clarkb | dhellmann: so the problem is anyone that does global installs with pip is going to have a bad day with pip 10 | 20:19 |
| clarkb | infra, devstack, devstack-gate, anywone else | 20:19 |
| dhellmann | yep | 20:19 |
| dhellmann | we may have to solve those in different ways | 20:20 |
| jeblair | how long do we have? | 20:20 |
| *** andreas_s has quit IRC | 20:20 | |
| clarkb | jeblair: the email dhellmann included said a couple months | 20:20 |
| dhellmann | either allowing pyyaml from the system package or using a virtualenv for whatever we're installing | 20:20 |
| fungi | "a couple months" apparently | 20:20 |
| clarkb | but also I think its worth pushing back on upstream about this | 20:20 |
| clarkb | this is a major change in behavior that will break basically everyone | 20:20 |
| dhellmann | jeblair : https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html | 20:20 |
| dhellmann | someone who has a good grasp of the impact to us should reply to that thread and explain the upcoming issue | 20:21 |
| dmsimard | pabelanger, fungi: while gitlab is open core (and selling a product), there are alternatives like gogs which is fully open source: https://github.com/gogits/gogs (example here: https://try.gogs.io/unknwon/grafana ). Anyway, I'll try and use cgit more, maybe it will grow on me or something. | 20:21 |
| dhellmann | make sure to point out we're not using internals, but this is a different impact of the 10 release | 20:21 |
| dhellmann | maybe there's a force upgrade option or something | 20:21 |
| clarkb | jeblair: once I haev a zuul hold in place how do I see if it has caught any nodes? is that a nodepool list | grep something? | 20:21 |
| jeblair | clarkb: yep. grep "hold" | 20:22 |
| clarkb | jeblair: thanks | 20:22 |
| fungi | dmsimard: i don't consider cgit anywhere near perfect, and am certainly open to replacements if we find one which fits our needs | 20:22 |
| jeblair | clarkb: you can also zuul autohold-list and see if the counter has decremented or the request has disappeared (because it went to 0) | 20:22 |
| clarkb | dhellmann: on debuntu in theory pip can just ignore the system install (that is what it used to do iirc) | 20:22 |
| mordred | fungi, dmsimard: when we're less busy I've been meaning to suggest looking at notabug | 20:22 |
| clarkb | jeblair: if the counter is still 1 then it hasn't caught anything yet? | 20:22 |
| jeblair | clarkb: right | 20:22 |
| *** vhosakot has quit IRC | 20:23 | |
| dhellmann | clarkb : if the new pyyaml is being installed to overwrite the old one, I'm not sure it's safe to ignore the old one? | 20:23 |
| dmsimard | mordred: never heard of notabug before, looks like another convincing clone like gogs :) | 20:23 |
| mordred | dmsimard: https://notabug.org/mordred/shade | 20:23 |
| hwoarang | mordred: can you explain to me how this syncrhonize thing works? i presume the host which runs ansible and the one who runs the job share the same /home/zuul directory ? | 20:23 |
| clarkb | dhellmann: on debuntu it is because old pyyaml is in /usr and new pyyaml si going into /usr/local and the python path is appropriately configured to make that work the way you want | 20:23 |
| dhellmann | ah | 20:23 |
| clarkb | dhellmann: red hat/suse et al don't do that though (because apparently upstream python said it was wrong for al ong time?) | 20:24 |
| dhellmann | tbh, I thought "debuntu" was a typo or something, I didn't realize that was a distro | 20:24 |
| clarkb | dhellmann: oh debian + ubuntu | 20:24 |
| fungi | infra-root: i need to disappear to run some errands, but will be back to help with executor restarts as soon as i can | 20:24 |
| dhellmann | yeah, I looked it up :-) | 20:24 |
| mordred | hwoarang: no - the /home/zuul directory is on the remote node. the host that runs ansible is one of the executors, and it's running ansible inside of a bubblewrap instance for isolation from the other invocations of ansible on that host for other jobs | 20:24 |
| abishop | hey, is there a tl;dr for what's happening in the tripleo gate queue? | 20:25 |
| mordred | tobiash: interestingly enough - that log line says "citycloud-lon1" - but the nodepool request-list says: | 20:25 |
| dmsimard | mordred: nice! The feature I miss the most when using cgit is proper line *AND* block highlighting | 20:25 |
| *** slaweq has quit IRC | 20:25 | |
| clarkb | EmilienM: is abishop's thing related to your multinode thing? | 20:25 |
| hwoarang | mordred: ok i see. btw your patch didn't work. the logs are still not available | 20:25 |
| mordred | tobiash: | 200-0000539944 | pending | zuulv3 | ubuntu-xenial | | nl02.openstack.org-31729-PoolWorker.citycloud-la1-main,nl02.openstack.org-31729-PoolWorker.citycloud-sto2-main | 20:25 |
| hwoarang | i need to investigate more | 20:25 |
| clarkb | dhellmann: anyways the thread you started is up to date with my findings, we should probably follwoup there | 20:25 |
| EmilienM | clarkb: most probably | 20:25 |
| mordred | hwoarang: darn. | 20:25 |
| tobiash | mordred: some sort of race? | 20:26 |
| abishop | my job (511275) looks like it's been re-running, and I fear the one above me is red which smells like doom | 20:26 |
| jeblair | mordred: what's the implication of what you just said? | 20:26 |
| dhellmann | clarkb : ++ | 20:26 |
| tobiash | mordred: citycloud-lon1 locked it and doesn't seem to do anything about it | 20:26 |
| mordred | jeblair: I do not yet know - trying to figure that out | 20:26 |
| *** slaweq has joined #openstack-infra | 20:27 | |
| jeblair | mordred: okay, i'm just not sure what you found interesting | 20:27 |
| mordred | jeblair: the reason we're looking at it is that there is a job that has been waiting on a node for over nine hours and there doesn't seem to be a good reason for that | 20:27 |
| jeblair | mordred: yeah that i got | 20:27 |
| jeblair | i'm trying to follow this | 20:27 |
| jeblair | mordred: but you said you found something interesting, pasted 2 lines, but i don't know what's interesting | 20:27 |
| jeblair | it's been a long week and maybe i just need you to spell it out for me, sorry. | 20:28 |
| *** jamesmcarthur has quit IRC | 20:28 | |
| mordred | jeblair: assume I'm flailing in the dark - what was interesting to me is that the log lines indicate lon was processing it with la1 and sto2 having already declined it - but the request-list seems to be telling me about the inverse of those hosts | 20:28 |
| hwoarang | maybe it should be hosts: primary instead of 'all' | 20:29 |
| mordred | jeblair: and yes - also long week - I'm very likely missing something obvious and also not doing a good job of making words | 20:29 |
| *** huanxie has quit IRC | 20:29 | |
| *** jamesmcarthur has joined #openstack-infra | 20:29 | |
| *** jamesmcarthur has quit IRC | 20:29 | |
| *** markvoelker_ has joined #openstack-infra | 20:29 | |
| *** jamesmcarthur has joined #openstack-infra | 20:29 | |
| tobiash | mordred: whats the node list grepped by citycloud-lon1? | 20:29 |
| mordred | hwoarang: is that a multinode job? | 20:29 |
| hwoarang | hmm no | 20:30 |
| mordred | tobiash: http://paste.openstack.org/show/624225/ | 20:30 |
| mordred | hwoarang: then all should be fine - that's what we use everywhere unless there is a need to limit it to be specific | 20:30 |
| tobiash | mordred: if that's only these two lines maybe also the provider deadlocked after locking the request | 20:30 |
| hwoarang | actually i doens't matter. i think the playbook wasn't taken into consideration at all | 20:30 |
| * hwoarang digs deeper | 20:31 | |
| *** spzala has joined #openstack-infra | 20:31 | |
| mordred | tobiash: yah - lon seems to be otherwise active and happy | 20:31 |
| tobiash | mordred: | 0000341123 | citycloud-lon1 | nova | ubuntu-xenial | 707f915c-32fb-4c9d-ad3f-ee21caea73e6 | building | 00:10:01:01 | locked | 20:31 |
| clarkb | EmilienM: I think I have caught a test env, can you point me at your public key? | 20:31 |
| jeblair | mordred: the request list says that la1 and sto2 declined it, which matches the log i believe. | 20:31 |
| tobiash | looks like the boot timeout is a bit too high | 20:31 |
| EmilienM | clarkb: a sec | 20:31 |
| EmilienM | clarkb: second one: https://launchpad.net/~emilienm/+sshkeys | 20:32 |
| mordred | jeblair: ah - ok. my bad on that one then | 20:32 |
| tobiash | mordred: can you grep that again with --detail? | 20:32 |
| mordred | tobiash: oh derp - I was looking through nodepool list for the wrong identifier | 20:32 |
| *** markvoelker has quit IRC | 20:32 | |
| *** spzala has quit IRC | 20:33 | |
| *** harlowja has quit IRC | 20:33 | |
| mordred | tobiash: | 0000341123 | citycloud-lon1 | nova | ubuntu-xenial | 707f915c-32fb-4c9d-ad3f-ee21caea73e6 | building | 00:10:03:23 | locked | ubuntu-xenial-citycloud-lon1-0000341123 | 37.153.172.138 | 10.0.1.59 | | 22 | nl02.openstack.org-31729-PoolWorker.citycloud-lon1-main | 200-0000539944 | None | 20:33 |
| jeblair | mordred, tobiash: yeah, we've set shorter build timeouts on other providers... though i thought the default was like 1 hour...? | 20:33 |
| mordred | fwiw - the node is totally up and exists and I can ssh in to it | 20:33 |
| clarkb | EmilienM: root@ 158.69.76.26 and 158.69.76.254 | 20:34 |
| EmilienM | ok | 20:35 |
| clarkb | EmilienM: hrm | 20:35 |
| clarkb | EmilienM: I don't actually knwo if the overlay stuff ran on those nodes | 20:35 |
| tobiash | mordred, jeblair: just checked project-config: the timeouts look much shorter than 10h ;) | 20:35 |
| tobiash | mordred: ssh thread stuck? | 20:35 |
| *** dave-mccowan has quit IRC | 20:35 | |
| clarkb | jeblair: does a hold go into affect before the job is done? or only on failure? | 20:36 |
| jeblair | mordred, tobiash: how about i sigusr2 to try to find out what it's doing? | 20:36 |
| jeblair | clarkb: on failure | 20:36 |
| tobiash | mordred: a thread dump might or might not help | 20:36 |
| mordred | jeblair: ++ | 20:36 |
| jeblair | k i'll do it | 20:36 |
| mordred | jeblair, tobiash: the last (only) launcher debug entry related to that server uuid is: | 20:36 |
| mordred | 2017-10-20 10:29:02,113 DEBUG nodepool.NodeLauncher-0000341123: Waiting for server 707f915c-32fb-4c9d-ad3f-ee21caea73e6 for node id: 0000341123 | 20:36 |
| *** jamesmcarthur has quit IRC | 20:36 | |
| clarkb | ok so the job did run then, but not seeing any ovs stuff | 20:36 |
| clarkb | so maybe it failed for a different rason | 20:36 |
| *** dave-mccowan has joined #openstack-infra | 20:36 | |
| jeblair | mordred: there are a few more lines for the node id | 20:37 |
| jeblair | launcher-debug.log.2017-10-20_04:2017-10-20 10:29:53,826 DEBUG nodepool.NodeLauncher-0000341123: Node 0000341123 is running [region: Lon1, az: nova, ip: 37.153.172.138 ipv4: 37.153.172.138, ipv6: ] | 20:37 |
| jeblair | launcher-debug.log.2017-10-20_04:2017-10-20 10:29:53,826 DEBUG nodepool.NodeLauncher-0000341123: Gathering host keys for node 0000341123 | 20:37 |
| *** jamesmcarthur has joined #openstack-infra | 20:37 | |
| jeblair | mordred, tobiash ^ | 20:37 |
| jeblair | which lends credence to tobiash's stuck ssh thread theory | 20:37 |
| tobiash | jeblair, mordred: that could indicate that ssh is stuck | 20:37 |
| tobiash | jeblair, mordred: we might want to set a timeout for the host key gathering | 20:38 |
| EmilienM | clarkb: right | 20:38 |
| jeblair | yes, though i thought it had one | 20:38 |
| mordred | what a fascinating place to be stuck | 20:38 |
| jeblair | mordred, tobiash: http://paste.openstack.org/show/624226/ | 20:38 |
| hwoarang | mordred: this is also interesting 2017-10-20 20:09:25.487148 | RUN END RESULT_NORMAL: [untrusted : git.openstack.org/openstack/openstack-ansible-tests/zuul.d/playbooks/run@stable/newton] | 20:38 |
| hwoarang | it seem that the stable/newton run playbook is executed but we work on master | 20:39 |
| EmilienM | dmsimard: does it mean we need we can stop doing multinode-setup in quickstart? | 20:39 |
| hwoarang | in stable/newton branch there is no post-run job... | 20:39 |
| *** thorst has quit IRC | 20:39 | |
| hwoarang | no clue why newton/stable is referenced there | 20:39 |
| *** tmorin has joined #openstack-infra | 20:39 | |
| tobiash | jeblair, mordred: so we have a timeout (which doesn't work) | 20:40 |
| dmsimard | EmilienM: does what mean that ? | 20:40 |
| jeblair | tobiash: start_client takes a timeout, but we don't pass it one | 20:40 |
| jeblair | i'll patch | 20:40 |
| EmilienM | dmsimard: you said we alraedy run playbooks that prepare multinode setup, or was it just firewall? | 20:40 |
| EmilienM | clarkb: can we hold http://zuulv3.openstack.org/static/stream.html?uuid=79c2d309fec048408f57234cd2056321&logfile=console.log | 20:40 |
| mordred | jeblair: awesome. that seems like a simple fix :) | 20:40 |
| EmilienM | 511275,2 | 20:40 |
| mordred | hwoarang: http://logs.openstack.org/06/513706/12/check/openstack-ansible-functional-ubuntu-xenial/e0e2e77/ara/result/e8bca89f-5c2d-4da7-84aa-cec74942272c/ | 20:41 |
| clarkb | EmilienM: we cannot hold specific jobs as far as I know | 20:41 |
| EmilienM | clarkb: it's tht again | 20:41 |
| EmilienM | oh ok | 20:41 |
| clarkb | EmilienM: I'm trying to get the logs from the job that ran on the nodes we do have held | 20:41 |
| *** jamesmcarthur has quit IRC | 20:41 | |
| clarkb | beacuse no ovs would explain why this didn't work too :) | 20:41 |
| clarkb | just want to double check | 20:41 |
| *** e0ne has joined #openstack-infra | 20:41 | |
| dmsimard | EmilienM: I know you run firewall setup for sure, I don't know about what else | 20:41 |
| tobiash | jeblair: ah, just looked at the handler (where we set a timeout) | 20:42 |
| clarkb | oh the job was aborted | 20:42 |
| EmilienM | dmsimard: right | 20:42 |
| clarkb | jeblair: I think ^ is a bug in the hold logic | 20:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Add timeout for ssh negotiation on keyscan https://review.openstack.org/513845 | 20:42 |
| jeblair | tobiash, mordred: ^ thanks! | 20:42 |
| clarkb | we shouldn't hold aborted jobs beacuse well this situation. Jobs in gate being aborted due to failures and you want to catch the actual failures | 20:42 |
| clarkb | I'm going to poke at fixing that | 20:42 |
| mordred | hwoarang: and yes - I agree, the post-run does not seem to have fired ... looking further | 20:42 |
| jeblair | clarkb: ++ | 20:42 |
| *** andreas_s has joined #openstack-infra | 20:43 | |
| dmsimard | EmilienM: your legacy jobs probably use the legacy code from devstack-gate to set up the multinode networking ? I mean, how have things been working until now ? | 20:43 |
| dmsimard | EmilienM: the multi-node-bridge is a "native" and generic zuul v3 implementation of what devstack-gate used to run | 20:43 |
| dmsimard | multi-node-bridge role* | 20:43 |
| hwoarang | mordred: what's with the ara output? does it explain the stable/newton thing? i can't see it :/ | 20:43 |
| jeblair | dhellmann, smcginnis: i believe the requirements change merged? | 20:43 |
| dhellmann | jeblair : yep, smcginnis has poked the next release patch and it's in the gate now | 20:44 |
| mordred | hwoarang: no - I'm still just looking for the post-run / logs stuff -where did you get that stable/newton line from? | 20:44 |
| dhellmann | jeblair : https://review.openstack.org/513799 | 20:44 |
| jeblair | dhellmann: oh, er, i wanted to restart the scheduler | 20:44 |
| hwoarang | mordred: from the console log | 20:44 |
| dhellmann | ah, damn, forgot | 20:45 |
| dhellmann | smcginnis : ^^ | 20:45 |
| smcginnis | jeblair: Bah! It just started. Want me to pull approval? | 20:45 |
| hwoarang | right here http://logs.openstack.org/06/513706/12/check/openstack-ansible-functional-ubuntu-xenial/e0e2e77/job-output.txt.gz#_2017-10-20_20_09_25_487148 | 20:45 |
| dhellmann | yeah | 20:45 |
| mordred | hwoarang: thanks | 20:45 |
| tobiash | jeblair: have a comment/question there | 20:45 |
| Shrews | mordred: sorry, back now and can help better | 20:45 |
| Shrews | mordred: where do you stand with this request dealio? | 20:45 |
| smcginnis | jeblair: Removed workflow. Feel free to restart at will. | 20:45 |
| jeblair | ok | 20:46 |
| mordred | Shrews: I believe we have found it (and by we, I mean jeblair and tobiash) | 20:46 |
| jeblair | i'm stopping the scheduler now | 20:46 |
| *** trown is now known as trown|outtypewww | 20:46 | |
| dmsimard | EmilienM: your job runs this: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/tripleo-ci-centos-7-nonha-multinode-oooq/run.yaml | 20:46 |
| smcginnis | jeblair: Sorry about that, guess I got too excited. ;) | 20:46 |
| jeblair | mordred: and you :) | 20:46 |
| mordred | Shrews: https://review.openstack.org/513845 | 20:46 |
| Shrews | mordred: then my timing is perfect | 20:46 |
| dmsimard | EmilienM: so it's using legacy devstack-gate things to set up the bridge afaict | 20:46 |
| mordred | jeblair: you have brainspace for another weird one? | 20:46 |
| jeblair | mordred: after i restart zuul | 20:46 |
| mordred | jeblair: awesome. tl;dr - there's a job that seems to indicate it's using something from a stable branch for reasons I cannot fathom | 20:47 |
| dmsimard | EmilienM: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/playbooks/ovs_vxlan_bridge.yaml | 20:47 |
| EmilienM | clarkb, dmsimard : I'm going to send a patch to test with the previous version of IPtables, I'm pretty sure it's that | 20:47 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header https://review.openstack.org/509436 | 20:48 |
| jeblair | okay, i'm restarting the scheduler | 20:48 |
| mordred | kk | 20:49 |
| jeblair | it's going to take me a few minutes to re-enqueue things | 20:49 |
| *** panda|rover is now known as panda|rover|off | 20:49 | |
| tobiash | end of day now, cya | 20:50 |
| mordred | tobiash: thanks for the help!!! | 20:50 |
| openstackgerrit | Sam Yaple proposed openstack-infra/irc-meetings master: Add LOCI meeting https://review.openstack.org/512471 | 20:50 |
| clarkb | jeblair: it does actually look like we may hold any completed job | 20:52 |
| clarkb | _doBuildCompletedEvent doesn't seem to distinguish | 20:52 |
| mordred | dmsimard: yay! that integration test wound up being useful didn't it? | 20:52 |
| clarkb | anyways I'll get a patch up and you can tell me where I misread it :) | 20:52 |
| clarkb | dmsimard: its only sort of using the legacy devsatck-gate stuff fwiw | 20:52 |
| clarkb | dmsimard: its using delorean and completely different network ranges and so on | 20:53 |
| clarkb | its possible those network ranges conflict with $cloud | 20:53 |
| dmsimard | mordred: yeah, I'm not able to keep up however.. roles are coming out faster than I can write tests for them | 20:53 |
| clarkb | (I don't know but the ranges we picked were somewhat carefully chosen not to do that) | 20:53 |
| dmsimard | mordred: there are different roles that are widely used that still don't have any tests | 20:53 |
| mordred | dmsimard: well - we should get better at writing tests when we write roles | 20:53 |
| *** abishop has quit IRC | 20:54 | |
| mordred | dmsimard: so that it's not you trying to keep up | 20:54 |
| smcginnis | jeblair: All clear for me to approve that release patch again? | 20:55 |
| *** salv-orl_ has joined #openstack-infra | 20:55 | |
| jeblair | smcginnis: y | 20:55 |
| EmilienM | clarkb, dmsimard: right now, debugging https://review.openstack.org/513848 and see if that's iptables | 20:55 |
| smcginnis | jeblair: Thanks | 20:55 |
| *** e0ne has quit IRC | 20:56 | |
| dmsimard | mordred: my brain is a bit sluggish and I don't have any great ideas right now but we have to come up with a good design for keeping roles tested. It's challenging because playbooks and roles are split left and right between three repos. | 20:56 |
| *** bobh has quit IRC | 20:56 | |
| dmsimard | mordred: we broke the gate temporarily earlier by merging a regression for use-cached-repos | 20:56 |
| smcginnis | Hmm, 0 events in queue, but the release patch isn't showing up in the gate queue. | 20:57 |
| mordred | dmsimard: yah. I also don't have a good suggestion for it either | 20:57 |
| dmsimard | because the playbook for it is in project-config so the jobs from o-z-j which test use-cached-repos do not run on project-config | 20:57 |
| *** salv-orlando has quit IRC | 20:58 | |
| mordred | dmsimard: well, we could add ozj integration tests to project-config for that case | 20:58 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add timeout for ssh negotiation on keyscan https://review.openstack.org/513845 | 20:58 |
| Shrews | jeblair: fixed the start_client() call for you ^^^ | 20:58 |
| dmsimard | mordred: it wouldn't work due to project-config being a trusted project | 20:58 |
| Shrews | mordred: +3 that if you wish | 20:58 |
| dmsimard | mordred: (I think) | 20:58 |
| * dmsimard needs vacation | 20:58 | |
| mordred | dmsimard: ah - but it would - because your integration tests are running ansible on a build node | 20:58 |
| mordred | dmsimard: so it would be testing a proposed patch - which would work - it just wouldn't be having the executor run with that content | 20:59 |
| mordred | dmsimard: I think. | 20:59 |
| dmsimard | mordred: so if the content isn't there, then it's not testing the patch so it's useless ? | 20:59 |
| jeblair | dmsimard: honestly, we only need to solve this for some roles which are used in project-config playbooks. we don't need to solve it for everything (since most roles are self-testing). | 21:00 |
| mordred | dmsimard: no - I mean the content would be there - becaue we DO put the speculative project-config content onto the test nodes when we trigger jobs via patches to project-config | 21:00 |
| jeblair | re-enqueing changes now | 21:00 |
| mordred | dmsimard: so the way the integration tests are structured it'll totally work and do the right thing | 21:00 |
| dmsimard | jeblair: most roles are not self-testing because they are not gated against themselves running | 21:00 |
| jeblair | dmsimard: well that's just bonkers :) | 21:00 |
| mordred | dmsimard: they are if they are used anywhere other than playbooks in project-config | 21:00 |
| dmsimard | jeblair: we only test a fraction of the roles we use right now: https://github.com/openstack-infra/openstack-zuul-jobs/tree/master/tests | 21:01 |
| jeblair | all the tox-related roles, for instance, are self testing | 21:01 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul feature/zuulv3: Only autohold failed builds https://review.openstack.org/513850 | 21:01 |
| clarkb | jeblair: ^ | 21:01 |
| clarkb | I don't have a test yet because I want to make sure I'm not completely misreading what is going on there | 21:01 |
| jeblair | dmsimard: right, that's the sort of thing you need to do for a role that's only used in project-config. it's not something we need to do for, say, tox. | 21:02 |
| dmsimard | jeblair: hm, okay I guess we have indirect coverage of some roles | 21:02 |
| mordred | yah | 21:02 |
| smcginnis | clarkb: Is there any instance where you would want to hold and inspect a non-failing job? | 21:02 |
| mordred | well - it's pretty direct coverage -it's like how tox.ini files in projects are self-testing - if you break them - they break :) | 21:02 |
| clarkb | smcginnis: possibly, but I can' think of a good way of representing that nad holding failed jobs in a way that allows you to debug failing jobs (which I think is the more important use case?) | 21:02 |
| smcginnis | clarkb: True, that definitely is more important. | 21:03 |
| mordred | dmsimard: in any case - I think what I'm trying to say is that the intregration test framework you put together is set up well to take care of the things we need - and it might not be as dire as it seems | 21:03 |
| mordred | dmsimard: so - yay! | 21:03 |
| jeblair | mordred, dmsimard ++ | 21:03 |
| dmsimard | jeblair: but anyway there's a lot of roles we don't have coverage for still, I need to start a todo list or something. For example set-service-type-data-fact was outright broken and it was hard to tell because it only runs in post | 21:03 |
| mordred | (and thank you) | 21:03 |
| jeblair | dmsimard: i wouldn't worry about that | 21:03 |
| jeblair | dmsimard: well, maybe | 21:03 |
| jeblair | dmsimard: let me put it this way | 21:04 |
| *** baoli has quit IRC | 21:04 | |
| clarkb | EmilienM: what you can do wiht those held nodes is run the script that sets up an overlay | 21:04 |
| jeblair | dmsimard: i'd worry about that if people are worried about the corresponding post jobs :) | 21:04 |
| mordred | I think that's one where writing a test would have been a nicer way to iterate | 21:04 |
| clarkb | EmilienM: you might want to do that to seeif you can debug why it is failing | 21:04 |
| jeblair | mordred: true | 21:04 |
| clarkb | smcginnis: jeblair I think we may want to continue to have an explicit hold in nodepool so that I can hold any node I want, then also have auto hold to catch failures in zuul | 21:04 |
| clarkb | that should cover all the bases | 21:05 |
| dmsimard | It's no longer possible to hold an explicit node in nodepool v3? | 21:05 |
| clarkb | dmsimard: I don't think so /me double checks | 21:05 |
| jeblair | clarkb: the locking algorithm does not facilitate that. | 21:05 |
| jeblair | it is not | 21:05 |
| jeblair | if you try, zuul will overwrite it as soon as it returns the node- | 21:05 |
| EmilienM | clarkb: I can do it, indeed | 21:05 |
| jeblair | zuul has the lock | 21:05 |
| clarkb | jeblair: then an explicit zuul hold? | 21:06 |
| mordred | yah - if we wanted to do that - we'd need to add a thing to zuul to ask it to set a hold on the nodes for a specific build I think | 21:06 |
| jeblair | yep | 21:06 |
| clarkb | anyways I think if we want to solve this for the case of I need to hold successful job for some reason that is the way to do it | 21:06 |
| mordred | zuul hold <build-id> | 21:06 |
| clarkb | then let autohold be a catch failures when they happen thing | 21:06 |
| clarkb | mordred: ya | 21:06 |
| jeblair | zuul hold <build>; zuul autohold <job>; zuul autohold --success <job> | 21:07 |
| EmilienM | I'm watching http://zuulv3.openstack.org/static/stream.html?uuid=ad33c87d3ae545ddaa61e32422ed10f2&logfile=console.log now with my debug patch | 21:07 |
| jeblair | mordred, clarkb: ^? | 21:07 |
| EmilienM | can we hold this one? | 21:07 |
| mordred | jeblair, clarkb: ++ | 21:07 |
| jeblair | think that covers the bases | 21:07 |
| clarkb | jeblair: well with success you tend to want a specific one | 21:07 |
| jeblair | clarkb: today, but not always? | 21:07 |
| clarkb | jeblair: rather than a random one, but with failure usually its all broken and you just want one to debug | 21:07 |
| dmsimard | EmilienM: your bug is weird, you're downloading what is the latest version of iptables on the centos repo, I don't understand | 21:07 |
| clarkb | jeblair: well today I just want any failing job so my patch I pushed should handle that | 21:07 |
| EmilienM | dmsimard: latest version is -2 | 21:07 |
| clarkb | jeblair: but ya maybe it doesn't matter | 21:08 |
| EmilienM | dmsimard: I'm downloading -1 and downgrading, to see if it works like before | 21:08 |
| clarkb | I guess if you are debugging a specific patch then you want specific hold regardless of success or failure | 21:08 |
| jeblair | mordred: so popping a few things off the stack... | 21:08 |
| mordred | I think both --success and hold are good improvements | 21:08 |
| mordred | jeblair: SO MUCH STACKS | 21:08 |
| clarkb | so ya those three commands should cover the bases | 21:08 |
| EmilienM | I wish it would be "SO MUCH SNACKS" | 21:08 |
| * EmilienM leaves | 21:08 | |
| mordred | EmilienM: ++ | 21:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Ignore duplicate project-template definitions https://review.openstack.org/513853 | 21:08 |
| dmsimard | EmilienM: oh ok I see the updated version in updates | 21:09 |
| jeblair | mordred: that may fix the problem you mentioned ^ | 21:09 |
| jeblair | mordred: that is in production now | 21:09 |
| EmilienM | dmsimard: yeah, they pushed a tag yesterday | 21:09 |
| EmilienM | dmsimard: a new nodepool-centos7 image was built and push in the meantime | 21:09 |
| EmilienM | dmsimard: and now we're broken. I'm pretty sure it's that. | 21:09 |
| EmilienM | dmsimard: but not 100% | 21:09 |
| dmsimard | EmilienM: couple bugzillas come up with https://www.google.ca/search?q=iptables-1.4.21-18.2.el7_4 | 21:10 |
| mordred | jeblair: cool. SO - the thing I'm seeing right now is - job defined in master, job also defined in stable/newton ... patch to master that changes master definition of job is running with stable/newton definition | 21:10 |
| clarkb | ok I've got to step away beacuse I've managed to not eat lunch | 21:10 |
| *** ijw has joined #openstack-infra | 21:10 | |
| EmilienM | clarkb: I skipped breakfast | 21:10 |
| EmilienM | and now lunch | 21:11 |
| jeblair | mordred: since the restart? | 21:11 |
| EmilienM | that's why when mordred wrote that I could read SNACKS lol | 21:11 |
| mordred | jeblair: nope. I'll recheck it | 21:11 |
| mordred | jeblair: just putting in the infos about this while they're paged in | 21:11 |
| mordred | jeblair: patch is http://logs.openstack.org/06/513706/12/check/openstack-ansible-functional-ubuntu-xenial/e0e2e77/job-output.txt.gz#_2017-10-20_20_04_49_272209 is the log showing running the stable/newton version of the job/playbook | 21:12 |
| mordred | jeblair: I'm not sure if project-templates come in to play or not | 21:12 |
| mordred | in fact, no, they do not | 21:12 |
| mordred | no project-template involved | 21:12 |
| *** andreas_s has quit IRC | 21:14 | |
| jeblair | well that's another puzzle :) | 21:17 |
| mordred | yah | 21:17 |
| mordred | it seems to be a behavior we do not desire :) | 21:17 |
| tosky | mordred: hi again, I was wondering if there was any news/WIP review for the proposed change to the publication interface | 21:18 |
| *** masber has joined #openstack-infra | 21:18 | |
| mordred | tosky: there are some patches up - but I got eaten by conference this week so did not make huge progress https://review.openstack.org/#/q/topic:zuulv3-output is the topic | 21:19 |
| tosky | mordred: oh, sure, no rush; thanks! | 21:19 |
| smcginnis | dhellmann, fungi, jeblair: Release job is in release-post now if your interested. | 21:20 |
| dhellmann | watching | 21:20 |
| mordred | spekaing of jobs in queues ... | 21:20 |
| *** thorst has joined #openstack-infra | 21:20 | |
| *** thorst has quit IRC | 21:20 | |
| mordred | jeblair: the shade job (that triggered the nodepool bug earlier) is in check and shold, if the last time is any indication - end up all green | 21:21 |
| mordred | jeblair: this is the 'use the new native devstack job' patch - so if it goes green, it means we can delete a chunkn of legacy devstack-gate jobs woot | 21:21 |
| *** masber has quit IRC | 21:22 | |
| smcginnis | tag-release passed. | 21:23 |
| dhellmann | good, that's consistent :-) | 21:24 |
| smcginnis | At least we haven't gone backwards. ;) | 21:24 |
| smcginnis | release-test patch is also now in the release queue. | 21:25 |
| *** mat128 has quit IRC | 21:27 | |
| dhellmann | for the stable publish job we're watching for that synchronize to run faster (and properly), right? | 21:27 |
| dhellmann | sorry, publish-static not "stable publish" | 21:27 |
| *** andreas_s has joined #openstack-infra | 21:28 | |
| dhellmann | oh, wow, and it did run faster | 21:28 |
| smcginnis | Yes, I believe it should work now. | 21:28 |
| dhellmann | and seems to have worked; I see 0.9.0 on the release site | 21:28 |
| smcginnis | Not too fast I hope. | 21:28 |
| smcginnis | Woot | 21:28 |
| dhellmann | took about 1 second? | 21:29 |
| smcginnis | Not bad. Definitely better than before. :) | 21:29 |
| dhellmann | next we're looking at the propose-update-constraints job? | 21:29 |
| dhellmann | in the release-test queue | 21:29 |
| smcginnis | Yep. Looks like the release-test jobs are picking up now. | 21:29 |
| dmsimard | I'm bikeshedding myself endlessly over the name of a variable, so I think it's time for me to sign off the weekend. Have yourself a nice one ! | 21:30 |
| smcginnis | dmsimard: Have a good one! | 21:30 |
| dhellmann | dmsimard : thanks for your help this week! | 21:30 |
| *** slaweq has quit IRC | 21:31 | |
| *** andreas_s has quit IRC | 21:32 | |
| jeblair | mordred: i've inspected zuul's in-memory data structures related to those jobs and can't see a problem right now. let's see what happens with that recheck. | 21:35 |
| mordred | jeblair: cool | 21:35 |
| *** edmondsw has joined #openstack-infra | 21:36 | |
| *** thiagolib has quit IRC | 21:36 | |
| *** claudiub|2 has joined #openstack-infra | 21:36 | |
| *** priteau has quit IRC | 21:37 | |
| *** andreas_s has joined #openstack-infra | 21:37 | |
| *** priteau has joined #openstack-infra | 21:37 | |
| *** edmondsw has quit IRC | 21:40 | |
| *** derekh has quit IRC | 21:42 | |
| *** priteau has quit IRC | 21:42 | |
| *** shardy has quit IRC | 21:45 | |
| *** baoli has joined #openstack-infra | 21:46 | |
| *** baoli has quit IRC | 21:46 | |
| *** andreas_s has quit IRC | 21:50 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add /usr/share/ca-certificates to trusted_ro_paths https://review.openstack.org/513840 | 21:52 |
| dhellmann | jeblair , fungi , smcginnis : it looks like the propose-update-constraints job is failing with an ssh configuration issue similar to what we dealt with for the tag job: http://logs.openstack.org/09/0946ecfb7ec0ad686aa5d0565c556b212ff5dbc0/release/propose-update-constraints/93aba68/job-output.txt.gz#_2017-10-20_21_49_20_258988 | 21:52 |
| dhellmann | I believe we needed some on-server debugging help to figure out exactly what was going on there, didn't we? | 21:53 |
| dhellmann | oh, I'll bet we're using the wrong ssh key there | 21:54 |
| jeblair | dhellmann: is it setting the wrong username there? | 21:54 |
| jeblair | dhellmann: or, well, username and ssh key don't match | 21:54 |
| jeblair | dhellmann: so, change one or the other, i reckon | 21:55 |
| dhellmann | we configure the user as "release" but -- yeah | 21:55 |
| dhellmann | right | 21:55 |
| dhellmann | can we change the key easily? | 21:55 |
| smcginnis | OpenStack Release Bot with infra-root@openstack.org - I believe that was right. | 21:55 |
| jeblair | we don't want to use the 'proposal bot' user? | 21:55 |
| dhellmann | the stuff to set up the user name is in a script. I could make it do some work to figure out which user to use, I guess. How can I tell what job I'm in from an arbitrary script? | 21:56 |
| jeblair | i guess before this just relied on running on different nodes? | 21:56 |
| dhellmann | I thought they key might be something I could just override on the job definition | 21:56 |
| dhellmann | yes | 21:56 |
| dhellmann | and I guess the users were set up differently, but we just hard-coded the release user into that script yesterday | 21:57 |
| jeblair | dhellmann: yes, the key is something that can be overridden, i'm just wondering if that's what we want to do. the release key is way more sensitive than the proposal key? | 21:57 |
| dhellmann | I guess I could change the way that hard-coding works, and move it up into the job | 21:57 |
| dhellmann | so in the release job before any scripts run I could just force the global settings we need; and then do something similar with different settings in the proposal job | 21:57 |
| dhellmann | would that be icky? | 21:58 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add timeout for ssh negotiation on keyscan https://review.openstack.org/513845 | 21:58 |
| jeblair | dhellmann: i don't think so... i think doing that as a role might be nice actually... | 21:58 |
| jeblair | like a 'configure-gitreview' role | 21:58 |
| dhellmann | ok. that sounds like more than I want to take on at 6 on a friday. I'll look into how to do that monday | 21:59 |
| dhellmann | if no one beats me to it | 21:59 |
| jeblair | oh look: http://docs.ansible.com/ansible/latest/git_config_module.html | 21:59 |
| dhellmann | oh, fun | 21:59 |
| smcginnis | So we could use that in each job to set the right config? | 21:59 |
| jeblair | so we can stick that into the pre playbook, and have it use job variables as necessary | 21:59 |
| jeblair | yep | 22:00 |
| smcginnis | Nice | 22:00 |
| *** andreas_s has joined #openstack-infra | 22:00 | |
| *** tpsilva has quit IRC | 22:01 | |
| *** slaweq has joined #openstack-infra | 22:02 | |
| dhellmann | jeblair : into the pre playbook for each job? or into a new role? | 22:02 |
| jeblair | dhellmann: looking at the structure of the jobs involved, maybe just in the main playbook of each job | 22:03 |
| dhellmann | so into project-config/playbook/release/tag.yaml for example? | 22:04 |
| dhellmann | jeblair : ^^ | 22:04 |
| *** ccamacho has quit IRC | 22:05 | |
| *** tmorin has quit IRC | 22:05 | |
| *** xinliang has quit IRC | 22:05 | |
| jeblair | dhellmann: yeah, we'd either just stick those git_config tasks in there, or make a new role and put the role in there. i'm starting to get friday brain too and am not sure which. maybe start with the tasks and see what it looks like. :) | 22:06 |
| dhellmann | ok | 22:06 |
| smcginnis | We add the origin remote in playbooks/release/pre.yaml. Looks reasonable to put this in there too. | 22:07 |
| *** andreas_s has quit IRC | 22:09 | |
| *** hasharAway has quit IRC | 22:09 | |
| dhellmann | what is the proposal bot's actual user name and email? | 22:09 |
| jeblair | smcginnis: oh, i missed that the proposals had a pre playbook too. so both release and proposal have pre playbooks, so it may make sense to do it there instead of the main playbooks. | 22:10 |
| openstackgerrit | Edgar Magana proposed openstack-infra/irc-meetings master: Include a new time for UC IRC meetings https://review.openstack.org/513863 | 22:10 |
| jeblair | especially since that's where the ssh key gets added | 22:10 |
| *** harlowja has joined #openstack-infra | 22:10 | |
| dhellmann | ++ | 22:10 |
| dhellmann | now I just need to know what values to use for the proposal bot | 22:10 |
| dhellmann | is the user "proposal" or "proposal-bot" | 22:10 |
| dhellmann | and is the email associated with the key right? "proposal-bot@review.openstack.org"? | 22:11 |
| jeblair | looking that up now | 22:11 |
| *** rlandy has quit IRC | 22:13 | |
| jeblair | i am still here and still confused. | 22:15 |
| dhellmann | according to jenkins/scripts/common.sh it is proposal-bot and openstack-infra@lists.openstack.org | 22:15 |
| jeblair | yes! i just found that | 22:16 |
| clarkb | ya I thought it was proposal-bot | 22:16 |
| jeblair | does that not get used by this job? | 22:16 |
| dhellmann | not any more | 22:16 |
| clarkb | (when I lookd at making sure the key was correct) | 22:16 |
| dhellmann | that was the wrong setting for the release job, so we changed the release script to not use that function | 22:16 |
| jeblair | oh, and this job uses that release script? | 22:16 |
| dhellmann | so the clone_repo function in jenkins/scripts/release-tools/functions does not invoke configure_git_review any more | 22:16 |
| jeblair | ok. coool. | 22:17 |
| dhellmann | yeah, it runs update_constraints_for_branch.sh | 22:17 |
| dhellmann | hmm | 22:17 |
| jeblair | if that involves check_already_approved, then we definitely want to keep the proposal key and continue doing what you're doing, since that function is hard-coded to use proposal-bot | 22:18 |
| dhellmann | this job may need more work than this | 22:18 |
| dhellmann | it's calling update_upper_constraints in jenkins/scripts/functions | 22:18 |
| *** andreas_s has joined #openstack-infra | 22:18 | |
| dhellmann | oh, wait, I'm in the wrong place ignore me | 22:19 |
| dhellmann | hang on | 22:19 |
| dhellmann | wrong script; it uses update_constraints.sh | 22:20 |
| dhellmann | that does not seem to use check_already_approved | 22:20 |
| dhellmann | it always proposes a new patch if there's a change | 22:21 |
| *** andreas_s has quit IRC | 22:23 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: move git configuration for release jobs to ansible tasks https://review.openstack.org/513864 | 22:23 |
| dhellmann | jeblair , smcginnis , clarkb : something like ^^ ought to be close | 22:23 |
| smcginnis | Sorry, I know it's even later where you are, but I'm going to have to sign off for today. | 22:25 |
| smcginnis | I'll likely check in a few times over the weekend though. | 22:25 |
| dhellmann | smcginnis : np, have a good weekend | 22:25 |
| smcginnis | dhellmann: You too. | 22:25 |
| smcginnis | Thanks everyone for all the help this week. | 22:25 |
| fungi | okay, i'm back | 22:27 |
| dhellmann | fungi : is your power all fixed up? | 22:27 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-openstackci master: Add support for installing ARA wsgi middleware for sqlite databases https://review.openstack.org/513866 | 22:28 |
| fungi | dhellmann: oh, yeah they were done after a few hours. i just got back from a brief interruption involving some evening errands | 22:28 |
| dhellmann | oh | 22:28 |
| fungi | infra-root: looks like we're still in need of an executor restart. happy to do that, but catching up on the last hour or so of scrollback now | 22:28 |
| mnaser | dhellmann do you mind if i grab your change and abstract them the git configs into a simple role? | 22:29 |
| mnaser | s/them the/the/ | 22:29 |
| mnaser | it'll be a few minutes | 22:29 |
| dhellmann | mnaser : go for it. I started some notes in https://etherpad.openstack.org/p/zuulv3-issues if you want to update that with a different patch (or just update my patch in place, that's fine, too) | 22:30 |
| jeblair | fungi: i think you're clear for executor restart | 22:30 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Enable and configure the ara middleware for logs-dev.o.o https://review.openstack.org/513868 | 22:30 |
| mnaser | dhellmann ill update yours, 2mins | 22:30 |
| jeblair | heh, i guess role is the right answer :) | 22:30 |
| dhellmann | mnaser: I'm going to sign off, too, while I can still enjoy a bit of daylight on the patio | 22:30 |
| dhellmann | I'll check back in Monday morning my time | 22:30 |
| fungi | jeblair: and yes, the release account got used by any v2 jobs running on the signing label node, while the proposal-bot account got used by anything running on the proposal label node | 22:31 |
| mnaser | i can help debug/test releases with other folks if anyone has access to some sort of testing system i guess | 22:31 |
| dhellmann | mnaser : you can test by pushing tags to the release-test repo. I have https://review.openstack.org/513869 lined up for a new release, too | 22:33 |
| *** leakypipes has quit IRC | 22:33 | |
| dhellmann | or I guess you could re-enqueue the post jobs for the failure from http://logs.openstack.org/09/0946ecfb7ec0ad686aa5d0565c556b212ff5dbc0/release/propose-update-constraints/93aba68/job-output.txt.gz#_2017-10-20_21_49_20_258988 | 22:33 |
| dhellmann | whatever is easier :-) | 22:34 |
| *** esberglu has quit IRC | 22:34 | |
| mnaser | dont know if i have access to do those :-p | 22:34 |
| * dhellmann nods | 22:34 | |
| dhellmann | thanks again for all the help today everyone; I'll check in Monday and see where things stand | 22:35 |
| dhellmann | have a good weekend! | 22:35 |
| *** slaweq has quit IRC | 22:36 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: move git configuration for release jobs to ansible tasks https://review.openstack.org/513864 | 22:38 |
| mnaser | ^ updated into a role | 22:38 |
| mordred | mnaser: love it | 22:41 |
| mnaser | :D | 22:42 |
| mnaser | and now we can reuse it, woo | 22:42 |
| *** boden has quit IRC | 22:42 | |
| *** slaweq has joined #openstack-infra | 22:42 | |
| mnaser | though i think it would love nicely in zuul-jobs... maybe when we're not trying to fix releases :) | 22:42 |
| *** dizquierdo has quit IRC | 22:43 | |
| mordred | mnaser: ++ | 22:44 |
| mordred | I actually have a half-baked thought in the back of my head about a way to write a general proposal job that could live in zuul-jobs and safely be used by anyone if they felt like it (as long as they had a secret to pass to it with credentials) | 22:45 |
| mordred | but that's a for-way-later kind of fun game | 22:45 |
| *** claudiub|2 has quit IRC | 22:45 | |
| *** andreas_s has joined #openstack-infra | 22:46 | |
| mnaser | mordred im thinking hack on zuul status page this weekend :p | 22:46 |
| mnaser | i've been thinking of trying to find a way to make it an iterative set of changes till its fully angular but i haven't been successful at thinking of a strategy of getting there | 22:47 |
| mnaser | mixing angular with non angular is meh. unless we use other more lightweight ui stuff like vue.js | 22:47 |
| fungi | hrm... unit test failure on zuul change 513853 | 22:48 |
| fungi | anyway, i've finished reading scrollback, so i'll get the executor restarts underway now | 22:48 |
| *** wolverineav has quit IRC | 22:48 | |
| *** andreas_s has quit IRC | 22:50 | |
| fungi | looks from the initscript like `sudo service zuul-executor restart` should do the trick | 22:51 |
| fungi | i'm guessing that triggers a graceful restart since it's been a few minutes now and it's still running under the original pid with plenty of ansible child processes | 22:53 |
| pabelanger | I've been doing stop, check things stopped, then start lately | 22:54 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-openstackci master: Add support for installing ARA wsgi middleware for sqlite databases https://review.openstack.org/513866 | 22:54 |
| pabelanger | I haven't tried a restart myself | 22:54 |
| fungi | well, consider this a test of the restart process in that case ;) | 22:54 |
| pabelanger | +1 | 22:54 |
| fungi | if it doesn't go as planned on ze01 i'll fall back to that | 22:54 |
| fungi | and then use whatever worked to do the rest | 22:54 |
| fungi | pabelanger: what has the shutdown timeframe for the executor daemon been for you in the recent past? | 22:56 |
| *** yamahata has quit IRC | 22:56 | |
| fungi | just curious how long is long enough to start getting worried | 22:56 |
| *** iyamahat has quit IRC | 22:56 | |
| pabelanger | fungi: usually a few minutes, I've seen upwards to 20mins. But I know jeblair and clarkb might have recently fixed that | 22:58 |
| fungi | okay, i'll refrain from getting anxious just yet, thanks | 22:58 |
| clarkb | ya it as a long long time but there were fixes for that | 22:58 |
| clarkb | I imagine it could still take some time as it waits for the threads to shut down | 22:58 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DEBUG: downgrade iptables https://review.openstack.org/513873 | 23:00 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Save the ARA sqlite database in a specific folder https://review.openstack.org/513874 | 23:03 |
| dmsimard | infra-root: I got a stack of patches to enable loading ARA sqlite databases dynamically: https://review.openstack.org/#/q/topic:ara-sqlite-middleware | 23:04 |
| * dmsimard weekend | 23:04 | |
| *** xarses has quit IRC | 23:05 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DEBUG: downgrade iptables https://review.openstack.org/513873 | 23:06 |
| *** camunoz has quit IRC | 23:07 | |
| *** armax has joined #openstack-infra | 23:07 | |
| fungi | clarkb: pabelanger: well, we're coming up on 20 minutes since i issued a service restart on ze01 and i still see the original daemon process | 23:07 |
| clarkb | fungi: oh did you tell it to restart? | 23:08 |
| clarkb | I think that may not be imlemented in the init script | 23:08 |
| * clarkb looks | 23:08 | |
| fungi | seems to be | 23:08 |
| clarkb | oh it is | 23:08 |
| clarkb | fungi: what I've done to make sure it is progressing is tracked ps -elf | grep zuul | wc -l | 23:09 |
| fungi | good idea, thanks | 23:09 |
| clarkb | it may not go all the way to zero if ssh agents are leaked but it should get close and slowly decreease over time | 23:09 |
| clarkb | right now it seems to be floating between 460 and 500 | 23:10 |
| clarkb | on 01 | 23:10 |
| fungi | yeah, i watched it go up a moment ago | 23:10 |
| pabelanger | fungi: you should see socket event in log for stopped, if it worked | 23:10 |
| pabelanger | then, jobs should be aborting | 23:10 |
| fungi | the actual word "stopped"? | 23:11 |
| pabelanger | zuul.CommandSocket: Received b'stop' from socket | 23:11 |
| pabelanger | something like that | 23:11 |
| clarkb | seems to be between ~420 and 440 now | 23:12 |
| clarkb | so I think it may be falling off | 23:12 |
| *** salv-orl_ has quit IRC | 23:12 | |
| fungi | pabelanger: not finding in the executor log | 23:12 |
| clarkb | 2017-10-20 22:50:00,387 DEBUG zuul.CommandSocket: Received b'_stop' from socket | 23:12 |
| pabelanger | we are aborting jobs on ze01 | 23:12 |
| pabelanger | so, should be stopping | 23:12 |
| *** salv-orlando has joined #openstack-infra | 23:12 | |
| clarkb | ya ps count is now floating around 400 | 23:12 |
| clarkb | so I think it is owrking just slowly | 23:12 |
| fungi | ahh, in the debug log | 23:13 |
| *** andreas_s has joined #openstack-infra | 23:13 | |
| fungi | yeah, i see it now | 23:13 |
| fungi | seems more like an info level event, but okay | 23:13 |
| clarkb | ya I think we can probably incrase the level on that one | 23:14 |
| clarkb | jeblair: did you see that https://review.openstack.org/#/c/513853/ failed test jobs? | 23:15 |
| *** slaweq has quit IRC | 23:15 | |
| *** ijw has quit IRC | 23:15 | |
| fungi | i mentioned that a few minutes before the executor restart, haven't dug into the test failures yet though | 23:16 |
| *** ijw has joined #openstack-infra | 23:16 | |
| *** salv-orlando has quit IRC | 23:17 | |
| *** andreas_s has quit IRC | 23:17 | |
| clarkb | I'm writing a test for my autohold change then will likely have to call it a day | 23:18 |
| *** thorst has joined #openstack-infra | 23:21 | |
| pabelanger | fungi: clarkb: ze01 is in a bad state | 23:23 |
| pabelanger | we have 3 zuul-executor and 1 defuncted | 23:23 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul feature/zuulv3: Only autohold failed builds https://review.openstack.org/513850 | 23:23 |
| pabelanger | so, restart command is likely to blame | 23:23 |
| pabelanger | see http://grafana.openstack.org/dashboard/db/zuul-status | 23:23 |
| pabelanger | for when fungi ran the command | 23:23 |
| *** edmondsw has joined #openstack-infra | 23:23 | |
| clarkb | jeblair: mordred tobiash https://review.openstack.org/513850 now with tests | 23:24 |
| pabelanger | running builds shows the aborts at 22:50, but it didn't completely stop things before zuul-executor launched another process | 23:24 |
| clarkb | ah so the init script isn't waiting properly | 23:24 |
| pabelanger | yah, IIRC, socket commands are non-blocking | 23:25 |
| clarkb | fungi: so it only half implemented restart :P | 23:25 |
| clarkb | in this case I think killing them is probably ok? | 23:25 |
| clarkb | or maybe issue a zuul-executor stop | 23:25 |
| clarkb | to stop the newer processes? | 23:25 |
| pabelanger | 510155 is what I want to test for an ansible-playbook, but need to update | 23:26 |
| pabelanger | might look into that tomorrow and do a little test | 23:26 |
| *** slaweq has joined #openstack-infra | 23:26 | |
| pabelanger | clarkb: yah, we should try to stop | 23:26 |
| *** thorst has quit IRC | 23:26 | |
| fungi | clarkb: pabelanger: so do you think i should kill off the newer executor and wait for the old one to terminate naturally? is it not okay for those to continue in parallel i guess? | 23:28 |
| *** edmondsw has quit IRC | 23:28 | |
| clarkb | fungi: I would run systemctl stop zuul-executor | 23:28 |
| clarkb | and see if that makes the newer processes stop | 23:28 |
| clarkb | I guess it depends who has the socket open? | 23:29 |
| clarkb | but at that point they should both stop assuming the nwer one is reading the socket? | 23:29 |
| fungi | okay, initiated stop | 23:29 |
| clarkb | but ya I think that other preocess is defuct because it couldn't open the finger socket(s)? | 23:30 |
| clarkb | since the old one still hsa those open? | 23:30 |
| pabelanger | yah, I think we'll have to killall -9 zuul-executor, once we get into multiple processes, it never seems to end well :) | 23:33 |
| fungi | likely. once it quiesces i'll do whatever we need to clean up the rest | 23:33 |
| pabelanger | k, newest processes stopped properly | 23:34 |
| pabelanger | but original still running | 23:34 |
| *** jamesmcarthur has joined #openstack-infra | 23:34 | |
| pabelanger | I wonder is we did | 23:34 |
| pabelanger | sudo su zuul; zuul stop | 23:34 |
| *** tosky has quit IRC | 23:34 | |
| clarkb | unrelated but the pip 10 saga gets more interesting, the python2 job ran just fine | 23:34 |
| pabelanger | if it would accept the command over socket | 23:34 |
| clarkb | and as far as I know it also installs pyyaml | 23:35 |
| pabelanger | looks to still be open | 23:35 |
| clarkb | does that imply the python3-pyyaml and python2-pyyaml packages on debuntu are built differently? | 23:35 |
| *** anupn is now known as anup | 23:35 | |
| clarkb | oh wow | 23:36 |
| clarkb | python-yaml is 3.12 what we need to install, but python3-yaml is 3.11 | 23:37 |
| *** priteau has joined #openstack-infra | 23:38 | |
| *** jamesmcarthur has quit IRC | 23:39 | |
| fungi | that does suggest they're built from different source packages at any rate | 23:39 |
| clarkb | I think python-yaml may come out of cloud archive | 23:40 |
| clarkb | so it is ahead of python3-yaml | 23:41 |
| clarkb | but I didn't think we had that enabled early enough for d-g to pick it up | 23:41 |
| clarkb | its 3.11 on my local xenial fileserver which gives weight to the cloud archive theory. In any case I'm going to install from pip to fix the pip 10 problem then we can sort ouf the best way forward | 23:42 |
| *** priteau has quit IRC | 23:43 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Install PyYAML from pypi https://review.openstack.org/513880 | 23:47 |
| clarkb | ok I've pushed up what I think are workarounds until the next system package becomes a problem | 23:48 |
| pabelanger | fungi: did you want to kill off the other zuul-executor and start on ze01? | 23:51 |
| fungi | it's still in the process of shutting down, looks like | 23:52 |
| mnaser | pabelanger congrats :) | 23:52 |
| pabelanger | ? | 23:54 |
| * fungi has a feeling he knows why you're being congratulated | 23:54 | |
| fungi | pabelanger: don't worry about it until after the weekend, trust me | 23:54 |
| mnaser | pabelanger http://civs.cs.cornell.edu/cgi-bin/results.pl?num_winners=6&id=E_ce86063991ef8aae :D | 23:55 |
| fungi | ctrl-?ctr | 23:55 |
| fungi | heh, that's a fun irc client screwup | 23:55 |
| pabelanger | mnaser: wow, thanks for sharing | 23:56 |
| mnaser | :) | 23:58 |
| mnaser | congats to fungi as well but i dont want to highlight everyone else whos here on a friday night, aha | 23:59 |
| mnaser | s/congats/congrats/ | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!