| *** edmondsw has joined #openstack-infra | 00:02 | |
| *** thorst has joined #openstack-infra | 00:03 | |
| *** edmondsw has quit IRC | 00:06 | |
| *** thorst has quit IRC | 00:07 | |
| *** thorst has joined #openstack-infra | 00:07 | |
| *** salv-orlando has quit IRC | 00:07 | |
| *** salv-orlando has joined #openstack-infra | 00:08 | |
| *** thorst has quit IRC | 00:08 | |
| *** gildub has joined #openstack-infra | 00:14 | |
| *** gildub has quit IRC | 00:19 | |
| *** dingyichen has joined #openstack-infra | 00:19 | |
| *** dingyichen has quit IRC | 00:20 | |
| *** dingyichen has joined #openstack-infra | 00:23 | |
| *** gildub has joined #openstack-infra | 00:32 | |
| *** edmondsw has joined #openstack-infra | 00:33 | |
| *** armax has joined #openstack-infra | 00:45 | |
| *** huanxie has joined #openstack-infra | 00:54 | |
| *** edmondsw has quit IRC | 00:56 | |
| *** armax has quit IRC | 01:02 | |
| *** armax has joined #openstack-infra | 01:02 | |
| *** armax has quit IRC | 01:03 | |
| *** armax has joined #openstack-infra | 01:03 | |
| *** armax has quit IRC | 01:03 | |
| *** armax has joined #openstack-infra | 01:04 | |
| *** armax has quit IRC | 01:04 | |
| *** armax has joined #openstack-infra | 01:05 | |
| *** armax has quit IRC | 01:05 | |
| *** armax has joined #openstack-infra | 01:06 | |
| *** armax has quit IRC | 01:06 | |
| *** LindaWang has joined #openstack-infra | 01:06 | |
| *** salv-orl_ has joined #openstack-infra | 01:07 | |
| *** salv-orlando has quit IRC | 01:10 | |
| *** thorst has joined #openstack-infra | 01:11 | |
| *** thorst has quit IRC | 01:11 | |
| *** armax has joined #openstack-infra | 01:18 | |
| *** armax has quit IRC | 01:19 | |
| *** liusheng has quit IRC | 01:21 | |
| *** gildub has quit IRC | 01:23 | |
| *** hongbin has joined #openstack-infra | 01:26 | |
| *** mat128 has joined #openstack-infra | 01:30 | |
| *** cuongnv has joined #openstack-infra | 01:31 | |
| *** d0ugal has quit IRC | 01:36 | |
| *** psachin has joined #openstack-infra | 01:39 | |
| *** liujiong has joined #openstack-infra | 01:46 | |
| *** edmondsw has joined #openstack-infra | 01:55 | |
| *** edmondsw has quit IRC | 01:59 | |
| *** d0ugal has joined #openstack-infra | 01:59 | |
| *** liusheng has joined #openstack-infra | 02:04 | |
| *** jamesmcarthur has joined #openstack-infra | 02:05 | |
| *** jamesmcarthur has quit IRC | 02:07 | |
| *** jamesmcarthur has joined #openstack-infra | 02:07 | |
| *** jamesmcarthur has quit IRC | 02:08 | |
| *** kiennt26 has joined #openstack-infra | 02:08 | |
| *** jamesmcarthur has joined #openstack-infra | 02:08 | |
| *** armaan has quit IRC | 02:09 | |
| *** armaan has joined #openstack-infra | 02:10 | |
| *** armaan has quit IRC | 02:14 | |
| *** gcb has joined #openstack-infra | 02:15 | |
| *** jamesmcarthur has quit IRC | 02:18 | |
| *** jamesmcarthur has joined #openstack-infra | 02:19 | |
| *** namnh has joined #openstack-infra | 02:21 | |
| *** liusheng has quit IRC | 02:22 | |
| *** liusheng has joined #openstack-infra | 02:22 | |
| *** dhinesh has joined #openstack-infra | 02:22 | |
| *** dhinesh has quit IRC | 02:23 | |
| *** dhinesh has joined #openstack-infra | 02:23 | |
| *** xinliang has quit IRC | 02:23 | |
| *** mat128 has quit IRC | 02:23 | |
| *** salv-orl_ has quit IRC | 02:25 | |
| *** salv-orlando has joined #openstack-infra | 02:25 | |
| *** annp has joined #openstack-infra | 02:27 | |
| *** wolverineav has joined #openstack-infra | 02:35 | |
| *** xinliang has joined #openstack-infra | 02:35 | |
| *** xinliang has quit IRC | 02:35 | |
| *** xinliang has joined #openstack-infra | 02:35 | |
| *** wolverineav has quit IRC | 02:40 | |
| *** salv-orlando has quit IRC | 02:43 | |
| *** salv-orlando has joined #openstack-infra | 02:44 | |
| *** masber has quit IRC | 02:47 | |
| *** wolverineav has joined #openstack-infra | 02:58 | |
| *** jamesmcarthur has quit IRC | 02:58 | |
| *** jamesmcarthur has joined #openstack-infra | 02:59 | |
| *** wolverineav has quit IRC | 03:02 | |
| *** thorst has joined #openstack-infra | 03:02 | |
| *** jamesmcarthur has quit IRC | 03:03 | |
| *** dave-mccowan has quit IRC | 03:05 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Test that the project being tested was prepared, not necessarily o-z-j https://review.openstack.org/514113 | 03:07 |
|---|---|---|
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration test coverage for iptables persistence https://review.openstack.org/513934 | 03:07 |
| *** thorst has quit IRC | 03:08 | |
| tonyb | SamYaple: Do you want the LOCI meeting request to merge? | 03:08 |
| tonyb | SamYaple: initally you asked for feedback but I don't see any | 03:09 |
| *** ramishra has joined #openstack-infra | 03:09 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Move to dictionary list of projects zuul._projects https://review.openstack.org/513233 | 03:09 |
| SamYaple | tonyb: yes | 03:12 |
| SamYaple | feedback was offline | 03:12 |
| tonyb | SamYaple: okay | 03:12 |
| SamYaple | tonyb: thanks for checking | 03:12 |
| SamYaple | offline/not in ticket | 03:13 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Remove a couple of stale nova meetings https://review.openstack.org/513830 | 03:13 |
| tonyb | SamYaple: np | 03:19 |
| *** gildub has joined #openstack-infra | 03:20 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Add LOCI meeting https://review.openstack.org/512471 | 03:22 |
| *** yolanda has quit IRC | 03:23 | |
| *** cuongnv has quit IRC | 03:23 | |
| *** wolverineav has joined #openstack-infra | 03:26 | |
| *** nicolasbock has quit IRC | 03:26 | |
| *** mat128 has joined #openstack-infra | 03:27 | |
| *** masber has joined #openstack-infra | 03:33 | |
| *** cuongnv has joined #openstack-infra | 03:33 | |
| *** wolverineav has quit IRC | 03:33 | |
| *** pahuang has quit IRC | 03:37 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Senlin https://review.openstack.org/512995 | 03:37 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove unused playbooks https://review.openstack.org/514042 | 03:37 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove unused oslo.messaging playbooks https://review.openstack.org/514041 | 03:37 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: remove legacy aodh jobs https://review.openstack.org/512412 | 03:37 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Removal of kolla legacy jobs https://review.openstack.org/508950 | 03:37 |
| *** sree has joined #openstack-infra | 03:37 | |
| openstackgerrit | Tetsuro Nakamura proposed openstack-infra/infra-manual master: Update PyPI registration instruction https://review.openstack.org/512596 | 03:41 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Revert "Revert "Move to dictionary list of projects zuul._projects"" https://review.openstack.org/514117 | 03:43 |
| *** edmondsw has joined #openstack-infra | 03:43 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: [DNM] Test _projects https://review.openstack.org/514118 | 03:43 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: [DNM] Test _projects https://review.openstack.org/514118 | 03:46 |
| *** tetsuro has joined #openstack-infra | 03:47 | |
| *** edmondsw has quit IRC | 03:47 | |
| *** dhajare has joined #openstack-infra | 03:49 | |
| *** udesale has joined #openstack-infra | 03:49 | |
| tetsuro | I proposed networking-spp project (https://review.openstack.org/#/c/512536/) and it looks like the project was created. Could you add me into the gerrit groups? | 03:51 |
| tetsuro | Tetsuro Nakamura (nakamura.tetsuro at lab.ntt.co.jp) https://review.openstack.org/#/admin/groups/1845,members https://review.openstack.org/#/admin/groups/1846,members | 03:51 |
| tetsuro | Additionaly, this patch is ready for review. https://review.openstack.org/#/c/512596/ | 03:53 |
| *** pahuang has joined #openstack-infra | 03:54 | |
| *** ykarel has joined #openstack-infra | 03:59 | |
| *** armax has joined #openstack-infra | 04:00 | |
| *** wolverineav has joined #openstack-infra | 04:02 | |
| *** stakeda has quit IRC | 04:04 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Revert "Revert "Move to dictionary list of projects zuul._projects"" https://review.openstack.org/514117 | 04:06 |
| *** wolverineav has quit IRC | 04:07 | |
| ianw | tetsuro: done | 04:08 |
| *** armax has quit IRC | 04:09 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: [WIP] Revert "Revert "Move to dictionary list of projects zuul._projects"" https://review.openstack.org/514117 | 04:10 |
| *** claudiub|2 has joined #openstack-infra | 04:11 | |
| tetsuro | ianw: thank you ! | 04:12 |
| *** hongbin has quit IRC | 04:12 | |
| *** yamamoto has joined #openstack-infra | 04:16 | |
| *** stakeda has joined #openstack-infra | 04:17 | |
| *** Dinesh_Bhor has joined #openstack-infra | 04:19 | |
| *** janki has joined #openstack-infra | 04:22 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: [WIP] Revert "Revert "Move to dictionary list of projects zuul._projects"" https://review.openstack.org/514117 | 04:25 |
| ianw | "with_items:" not supporting a iterator/generator is odd | 04:32 |
| *** wolverineav has joined #openstack-infra | 04:32 | |
| *** wolverineav has quit IRC | 04:37 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Revert "Revert "Move to dictionary list of projects zuul._projects"" https://review.openstack.org/514117 | 04:41 |
| openstackgerrit | Tetsuro Nakamura proposed openstack-infra/infra-manual master: Update PyPI registration instruction https://review.openstack.org/512596 | 04:42 |
| *** bhavik1 has joined #openstack-infra | 04:42 | |
| *** tetsuro has quit IRC | 04:52 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Update PyPI registration instruction https://review.openstack.org/512596 | 04:54 |
| SamYaple | ianw: its because of the ansible rendering, yaml first, then jinaj2 | 04:58 |
| ianw | SamYaple: yeah, it just doesn't give you particularly helpful feedback, that's all :) | 04:59 |
| SamYaple | agreed | 04:59 |
| SamYaple | its a bit silly | 04:59 |
| SamYaple | much bigger fan of jinja2 then yaml personally | 04:59 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Test that the project being tested was prepared, not necessarily o-z-j https://review.openstack.org/514113 | 04:59 |
| SamYaple | but hey i like saltstack, so that shouldnt be a suprise :) | 05:00 |
| *** jaosorior has joined #openstack-infra | 05:02 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Docker matrix build https://review.openstack.org/414347 | 05:02 |
| *** armaan has joined #openstack-infra | 05:02 | |
| *** claudiub|2 has quit IRC | 05:03 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Convert zuul.projects to a dict https://review.openstack.org/514119 | 05:08 |
| *** wolverineav has joined #openstack-infra | 05:11 | |
| *** wolverineav has quit IRC | 05:15 | |
| *** dhinesh has quit IRC | 05:18 | |
| *** claudiub|2 has joined #openstack-infra | 05:25 | |
| *** wolverineav has joined #openstack-infra | 05:28 | |
| *** sree_ has joined #openstack-infra | 05:31 | |
| *** sree_ is now known as Guest38932 | 05:31 | |
| *** edmondsw has joined #openstack-infra | 05:31 | |
| *** cshastri has joined #openstack-infra | 05:32 | |
| *** wolverineav has quit IRC | 05:33 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Convert zuul.projects to a dict https://review.openstack.org/514119 | 05:33 |
| *** sree has quit IRC | 05:34 | |
| *** edmondsw has quit IRC | 05:36 | |
| *** wolverineav has joined #openstack-infra | 05:42 | |
| *** wolverineav has quit IRC | 05:47 | |
| *** gildub has quit IRC | 05:49 | |
| *** spectr has joined #openstack-infra | 05:52 | |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Fix releasenotes job for VPNaaS dashboard https://review.openstack.org/514123 | 05:59 |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Fix releasenotes job for VPNaaS dashboard https://review.openstack.org/514123 | 06:01 |
| *** Goneri has joined #openstack-infra | 06:02 | |
| *** Guest38877 has quit IRC | 06:05 | |
| *** martinkopec has joined #openstack-infra | 06:09 | |
| *** udesale__ has joined #openstack-infra | 06:09 | |
| *** udesale has quit IRC | 06:10 | |
| *** martinkopec_ has joined #openstack-infra | 06:10 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Run gerrit periodic on openstack/2.13 only https://review.openstack.org/514127 | 06:10 |
| *** udesale has joined #openstack-infra | 06:12 | |
| *** martinkopec has quit IRC | 06:13 | |
| *** udesale__ has quit IRC | 06:14 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Run wheel builds only once https://review.openstack.org/514135 | 06:19 |
| *** bhavik1 has quit IRC | 06:21 | |
| *** pgadiya has joined #openstack-infra | 06:23 | |
| *** andreas_s has joined #openstack-infra | 06:26 | |
| *** eumel8 has joined #openstack-infra | 06:27 | |
| dirk | AJaeger: doesn't look like generate constraints runs only for master: https://review.openstack.org/#/c/514132/ | 06:30 |
| *** yamamoto has quit IRC | 06:33 | |
| *** ykarel_ has joined #openstack-infra | 06:35 | |
| *** ykarel has quit IRC | 06:38 | |
| *** ociuhandu has joined #openstack-infra | 06:39 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Add umount_safe function to get better debug info https://review.openstack.org/514081 | 06:43 |
| *** florianf has joined #openstack-infra | 06:48 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: TLS everywhere: Only generate ssh key if it's not there https://review.openstack.org/514152 | 06:48 |
| *** yamamoto has joined #openstack-infra | 06:52 | |
| *** aviau has quit IRC | 06:54 | |
| *** aviau has joined #openstack-infra | 06:54 | |
| *** fabo has joined #openstack-infra | 07:00 | |
| *** pcaruana has joined #openstack-infra | 07:00 | |
| AJaeger | dirk: mmh, let me check why it did that ;( Thanks for telling | 07:01 |
| *** jtomasek has joined #openstack-infra | 07:01 | |
| AJaeger | dirk: indeed, we have not - let me fix... | 07:03 |
| *** slaweq has joined #openstack-infra | 07:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix requirements proposal jobs https://review.openstack.org/514161 | 07:07 |
| AJaeger | dirk, I fixed the wrong one ;( ^ | 07:07 |
| *** andreas_s has quit IRC | 07:09 | |
| *** andreas_s has joined #openstack-infra | 07:09 | |
| *** yamamoto_ has joined #openstack-infra | 07:11 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Use -t devpts for /dev/pts mounts https://review.openstack.org/514043 | 07:13 |
| *** yamamoto has quit IRC | 07:13 | |
| frickler | AJaeger: I'm a bit confused now, when does the "branches:" tag need "^$" around it and when does it not? | 07:13 |
| *** jtomasek has quit IRC | 07:14 | |
| *** andreas_s has quit IRC | 07:14 | |
| AJaeger | frickler: it might be a bug as well ;) | 07:16 |
| openstackgerrit | Tovin Seven proposed openstack-infra/openstack-zuul-jobs master: Remove legacy oslo.versionedobjects job https://review.openstack.org/511784 | 07:16 |
| AJaeger | frickler: I noticed that the job run on branch openstack/stable-2.13.3 (or something like that) and openstack/stable-2.13.4 | 07:17 |
| AJaeger | But it should only run on exactly "openstack/stable-2.13" | 07:17 |
| *** dbecker has joined #openstack-infra | 07:17 | |
| AJaeger | So, adding the regex to catch that | 07:17 |
| AJaeger | frickler: let me update commit message... | 07:18 |
| *** ccamacho has joined #openstack-infra | 07:18 | |
| mpeterson | Good morning, I need help with some jobs failing with 'ERROR No valid playbook found' on a branch but the same jobs working on master, do you know why it might be? (works: https://review.openstack.org/#/c/513938/ | doesn't work: https://review.openstack.org/#/c/513374/ ) | 07:19 |
| *** aviau has quit IRC | 07:19 | |
| *** edmondsw has joined #openstack-infra | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| frickler | mpeterson: without looking in detail: iiuc you need to cherry-pick the playbook into the stable branches | 07:20 |
| *** pahuang has quit IRC | 07:21 | |
| *** witek_ has joined #openstack-infra | 07:22 | |
| *** witek has quit IRC | 07:22 | |
| AJaeger | mpeterson: https://docs.openstack.org/infra/manual/zuulv3.html#stable-branches | 07:22 |
| mordred | AJaeger, frickler: fyi, there is some weirdness around jobs in master vs. stable branches right now that jeblair is looking in to - so some of it may be a little wonkier than it's supposed to be for a week or two ... | 07:22 |
| frickler | AJaeger: ah, o.k., substring matching is a good explanation, but then things like "^stable/ocata$" which we have in some locations really are the same as "stable/ocata" assuming the usual branch naming | 07:22 |
| mordred | tl;dr - there may be some cases where branch matchers need to be explicitly provided where it is not the intent for them to need to be explicitly provided | 07:23 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Run gerrit periodic on openstack/2.13 only https://review.openstack.org/514127 | 07:23 |
| AJaeger | frickler: better? ^ | 07:24 |
| *** edmondsw has quit IRC | 07:24 | |
| mordred | (we don't have a great specific suggestion of action yet - have some concerns about getting workarounds cargo-culted in to too many places) | 07:24 |
| *** ykarel__ has joined #openstack-infra | 07:24 | |
| AJaeger | frickler: yes, as long as we do not have stable/ocata-something, those are the same | 07:24 |
| *** andreas_s has joined #openstack-infra | 07:24 | |
| AJaeger | mordred: let's discuss later. | 07:25 |
| AJaeger | mordred: morning, are you still in Europe? | 07:25 |
| *** andreas_s has quit IRC | 07:25 | |
| *** andreas_s has joined #openstack-infra | 07:25 | |
| mpeterson | frickler, AJaeger: mmm I see, I got confused since it picked the project stanza but not the playbooks. Thanks! | 07:25 |
| AJaeger | mpeterson: no worries, we're all learning together here - one step at a time ;) | 07:26 |
| mordred | AJaeger: I am - doing a talk on zuul v3 at OSSummit Europe in a couple of hours | 07:26 |
| *** jistr is now known as jistr|mtgs | 07:26 | |
| AJaeger | mordred: Enjoy Prague! | 07:27 |
| SamYaple | mordred: you must be out of the country because you are up at some weird times | 07:27 |
| SamYaple | ah prague | 07:27 |
| *** ykarel_ has quit IRC | 07:27 | |
| *** tesseract has joined #openstack-infra | 07:27 | |
| AJaeger | frickler, mordred, would be nice to get a review of https://review.openstack.org/514161 ... | 07:27 |
| kazsh | AJaeger: G'day, could you kindly add myself "kaz_shinohara" to heat-dashboard-core team on gerrit ? my change safely merged in governance. https://review.openstack.org/#/c/504166/ | 07:28 |
| AJaeger | kazsh: I don't have those permissions. Perhaps an infra-roote is around ^ | 07:29 |
| *** yamamoto_ has quit IRC | 07:29 | |
| AJaeger | kazsh: and then the PTL should be added ^ | 07:29 |
| AJaeger | kazsh: and the PTL can add others | 07:29 |
| *** yamamoto has joined #openstack-infra | 07:29 | |
| dirk | AJaeger: blame the reviewers ! ;) | 07:30 |
| AJaeger | mordred: you can tell the audience that v3 is running just fine. I'm really happy with the state of it and the progress made in the last weeks - both on Zuul v3 and on jobs. It feels like v3 runs faster | 07:30 |
| AJaeger | dirk: please review it, so that I don't have to blame anybody ;) | 07:31 |
| AJaeger | dirk: errors happen and can easily get corrected, no need for blaming | 07:32 |
| *** hashar has joined #openstack-infra | 07:32 | |
| AJaeger | fungi, clarkb , readthedocs still fails, see http://logs.openstack.org/f7/f7bbe55504c57f703c5858f7a8b8d5359df3f101/post/trigger-readthedocs/c2b562f/ara/result/872671c1-fd73-4bab-8143-09a4f84d20fa/ - I'll put to etherpad | 07:33 |
| *** pahuang has joined #openstack-infra | 07:33 | |
| openstackgerrit | Adrian Czarnecki proposed openstack-infra/project-config master: Remove legacy job for monasca-api https://review.openstack.org/513722 | 07:35 |
| dirk | AJaeger: looks good | 07:35 |
| *** kjackal_ has joined #openstack-infra | 07:40 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove unused legacy templates https://review.openstack.org/513941 | 07:42 |
| *** jpena|off is now known as jpena | 07:43 | |
| *** yamamoto has quit IRC | 07:43 | |
| *** Hal has joined #openstack-infra | 07:45 | |
| *** rtjure has joined #openstack-infra | 07:45 | |
| *** Hal is now known as Guest97374 | 07:45 | |
| openstackgerrit | Adrian Czarnecki proposed openstack-infra/project-config master: Remove legacy job for monasca-api https://review.openstack.org/513722 | 07:48 |
| *** salv-orlando has quit IRC | 07:50 | |
| *** salv-orlando has joined #openstack-infra | 07:50 | |
| *** dhinesh has joined #openstack-infra | 07:52 | |
| *** jtomasek has joined #openstack-infra | 07:54 | |
| *** salv-orl_ has joined #openstack-infra | 07:55 | |
| *** nephel has joined #openstack-infra | 07:55 | |
| *** amoralej|off is now known as amoralej | 07:55 | |
| *** yamamoto has joined #openstack-infra | 07:57 | |
| *** dhinesh has quit IRC | 07:57 | |
| *** salv-orlando has quit IRC | 07:58 | |
| *** yamamoto has quit IRC | 08:03 | |
| *** vponomaryov1 has quit IRC | 08:05 | |
| *** ykarel__ is now known as ykarel|lunch | 08:10 | |
| *** vponomaryov has joined #openstack-infra | 08:12 | |
| *** e0ne has joined #openstack-infra | 08:13 | |
| openstackgerrit | Adrian Czarnecki proposed openstack-infra/project-config master: Remove legacy job for monasca-api https://review.openstack.org/513722 | 08:13 |
| *** rtjure has quit IRC | 08:14 | |
| *** owalsh_ is now known as owalsh | 08:18 | |
| kazsh | AJaeger: Thanks a lot, could anybody infra-root team help me out ? need to add myself and PTL to heat-dashboard-core so that our project move forward... | 08:19 |
| *** rtjure has joined #openstack-infra | 08:20 | |
| ianw | kazsh: i added you | 08:22 |
| *** ralonsoh has joined #openstack-infra | 08:22 | |
| *** finucannot is now known as stephenfin | 08:23 | |
| *** rtjure has quit IRC | 08:25 | |
| kazsh | ianw: thank you very much ! | 08:26 |
| *** rtjure has joined #openstack-infra | 08:30 | |
| *** derekh has joined #openstack-infra | 08:32 | |
| *** spectr has quit IRC | 08:35 | |
| *** rtjure has quit IRC | 08:35 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Fixed typo in zaqar tempest plugin configuration https://review.openstack.org/514182 | 08:35 |
| *** lucas-afk is now known as lucasagomes | 08:36 | |
| *** yamamoto has joined #openstack-infra | 08:36 | |
| *** wolverineav has joined #openstack-infra | 08:36 | |
| *** yamamoto has quit IRC | 08:38 | |
| *** mpjetta has quit IRC | 08:39 | |
| *** rtjure has joined #openstack-infra | 08:40 | |
| *** wolverineav has quit IRC | 08:40 | |
| *** mpjetta has joined #openstack-infra | 08:41 | |
| openstackgerrit | Adrian Czarnecki proposed openstack-infra/project-config master: Remove legacy job for monasca-api https://review.openstack.org/513722 | 08:41 |
| *** rtjure has quit IRC | 08:44 | |
| *** dtantsur|afk is now known as dtantsur | 08:45 | |
| *** ykarel|lunch is now known as ykarel | 08:46 | |
| *** mpjetta has quit IRC | 08:46 | |
| *** mpjetta has joined #openstack-infra | 08:48 | |
| *** spectr has joined #openstack-infra | 08:49 | |
| *** yamamoto has joined #openstack-infra | 08:50 | |
| *** dingyichen has quit IRC | 08:53 | |
| *** dhinesh has joined #openstack-infra | 08:53 | |
| *** tosky has joined #openstack-infra | 08:54 | |
| *** rtjure has joined #openstack-infra | 08:54 | |
| *** raissa has quit IRC | 08:57 | |
| *** dhinesh has quit IRC | 08:58 | |
| *** raissa has joined #openstack-infra | 08:58 | |
| *** rtjure has quit IRC | 08:59 | |
| flaper87 | hey folks, can someone help me understand why this patch is not being merged? https://review.openstack.org/#/c/513311/ | 09:01 |
| flaper87 | The repo was recently imported from GH and all required patches from the infra side seem to be merged and correct | 09:02 |
| flaper87 | am I missing something? | 09:02 |
| * flaper87 is sure he's missing something | 09:02 | |
| flaper87 | just not sure what that something is | 09:02 |
| tosky | flaper87: do you mean that you fixed project-config with at least the basic template that enables merging? | 09:03 |
| *** yamamoto has quit IRC | 09:03 | |
| *** ankkumar has joined #openstack-infra | 09:04 | |
| flaper87 | tosky: what do you mean I fixed project-config? The patches to project-config were submitted and approved. Lemme get the links | 09:04 |
| tosky | an alternative question would have been then (what is the link to the project-config review)? :) | 09:04 |
| tosky | flaper87: this one? https://review.openstack.org/#/c/512330 | 09:05 |
| flaper87 | https://review.openstack.org/#/c/512330/ and https://review.openstack.org/#/c/512323/ | 09:05 |
| AJaeger | flaper87: I rechecked. | 09:06 |
| tosky | AJaeger always faster :) | 09:06 |
| AJaeger | flaper87: you need Zuul to +1 on the change and I did not see a vote on it. | 09:06 |
| flaper87 | AJaeger: thanks, I had rechecked over the weekend | 09:06 |
| flaper87 | hence me asking | 09:07 |
| AJaeger | flaper87: you did not recheck! | 09:07 |
| AJaeger | flaper87: you toggled the +W | 09:07 |
| flaper87 | wait, what? | 09:07 |
| AJaeger | flaper87: I do not see "recheck" comment | 09:07 |
| *** rtjure has joined #openstack-infra | 09:07 | |
| flaper87 | no, I swear I did | 09:07 |
| flaper87 | fuck this internet connection | 09:07 |
| flaper87 | pardon my language | 09:07 |
| flaper87 | T_T | 09:07 |
| AJaeger | flaper87: and it's merged now ;) | 09:07 |
| *** edmondsw has joined #openstack-infra | 09:08 | |
| flaper87 | I'm in a place with poor internet connection and the recheck comment must have gone nowhere | 09:08 |
| flaper87 | thanks Andreas | 09:08 |
| AJaeger | glad we could sort it out so quickly... | 09:08 |
| * flaper87 starts reading how to add in-repo zuul jobs | 09:09 | |
| *** derekh has quit IRC | 09:11 | |
| *** edmondsw has quit IRC | 09:12 | |
| *** rtjure has quit IRC | 09:12 | |
| *** yamamoto has joined #openstack-infra | 09:13 | |
| *** derekh has joined #openstack-infra | 09:13 | |
| *** sambetts|afk is now known as sambetts | 09:15 | |
| *** rtjure has joined #openstack-infra | 09:17 | |
| AJaeger | flaper87: once you have your first patch up, feel free to ask for review... | 09:19 |
| *** arxcruz|off is now known as arxcruz | 09:19 | |
| flaper87 | AJaeger: I will :) | 09:20 |
| *** janki has quit IRC | 09:21 | |
| *** rtjure has quit IRC | 09:21 | |
| andreas_s | Hi. is there a way to just disable the build-openstack-sphinx-docs for the project networking-dpm? Since the transition to zuulv3 this job is facing some issue. I want to deactivate to unblock our work and then add it again once we define the jobs in our own repo... | 09:22 |
| *** claudiub has joined #openstack-infra | 09:23 | |
| AJaeger | andreas_s: sure, you can remove it. But let's first understand why it fails. Have a link to a failure? | 09:24 |
| andreas_s | AJaeger: http://logs.openstack.org/83/513683/1/check/build-openstack-sphinx-docs/da7caa2/job-output.txt.gz | 09:24 |
| andreas_s | the issue is, that zuul cloner is pulling neutron | 09:24 |
| andreas_s | but neutron is not in the list of dpendend projects | 09:24 |
| AJaeger | andreas_s: so, send a patch to add neutron to required-projects like other repos did already... | 09:25 |
| andreas_s | last week I got the acdvice to use the publish-openstack-sphinx-docs-neutron job instead | 09:25 |
| andreas_s | but we're not allowed to publish to openstack.org, as we're not part of the neutron stadium... | 09:26 |
| *** claudiub|2 has quit IRC | 09:26 | |
| AJaeger | So, just do what that template does ;) | 09:26 |
| AJaeger | andreas_s: shall I fix it for you quickly? | 09:26 |
| andreas_s | AJaeger: that would be great :) | 09:26 |
| andreas_s | I'm still a bit lost in finding the right places... thanks | 09:27 |
| *** rtjure has joined #openstack-infra | 09:29 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix required repos for networking-dpm https://review.openstack.org/514191 | 09:29 |
| AJaeger | andreas_s: this should do the trick ^ | 09:29 |
| AJaeger | andreas_s: that will change soon ;) | 09:29 |
| *** spectr has quit IRC | 09:31 | |
| *** electrofelix has joined #openstack-infra | 09:31 | |
| *** spectr has joined #openstack-infra | 09:31 | |
| andreas_s | AJaeger: ah ok. so the explicit definition is overriding the default definition, right? | 09:31 |
| andreas_s | AJaeger: cause I'm not 100% sure where the sphinx job comes from - It's probably part of some other template... | 09:32 |
| *** yolanda has joined #openstack-infra | 09:34 | |
| *** rtjure has quit IRC | 09:34 | |
| *** salv-orl_ has quit IRC | 09:36 | |
| andreas_s | AJaeger: can you explain in a nutshell how the zuul.d/projects.yaml (jenkins/jobs/projects.yaml) and the zuul/layout.yaml play together? I'm still struggeling a bit with the many places where definitions exist... | 09:37 |
| *** salv-orlando has joined #openstack-infra | 09:37 | |
| AJaeger | andreas_s: the job is part of the docs-on-rtd template. | 09:37 |
| AJaeger | andreas_s: zuul/layout.yaml is dead and ignored | 09:38 |
| AJaeger | We have jobs defined in openstack-zuul-jobs now and in project-config, best to read docs.openstack.org/infra/manual/zuulv3.html for an overview | 09:38 |
| andreas_s | AJaeger: got it. Thanks. | 09:39 |
| *** rtjure has joined #openstack-infra | 09:41 | |
| *** salv-orlando has quit IRC | 09:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update README to reflect current status https://review.openstack.org/514195 | 09:42 |
| *** sdague has joined #openstack-infra | 09:44 | |
| *** LindaWang has quit IRC | 09:45 | |
| *** nephel has quit IRC | 09:45 | |
| *** rtjure has quit IRC | 09:46 | |
| stephenfin | AJaeger: Morning. I've a question for you on the zuulv3 migration | 09:47 |
| stephenfin | I'm looking at the list of jobs for openstack/nova in 'project-config/zuul.d/projects.yaml', and I notice a good deal of them are reused by multiple projects (mostly the tempest-dsvm and grenade-dvsm, of which there are many) | 09:49 |
| stephenfin | Is the expectation that we'd be duplicating these between projects? If so, how do we stop them getting out of sync over time? | 09:49 |
| tosky | if the question is "are there some basic jobs that we can built upon", the answer is: | 09:49 |
| tosky | - there is a work-in-progress base devstack-tempest job, but it's not working properly yet; you can try to build one based on the stack of patches under review | 09:50 |
| stephenfin | tosky: Not quite. I've figured out that 'openstack-zuul-jobs/tree/zuul.d/jobs.yaml' exists | 09:50 |
| tosky | - work on a common grenade test will start when the shared tempest job is done | 09:50 |
| tosky | oh, no, it is exactly that question :) | 09:50 |
| *** rossella_s has joined #openstack-infra | 09:51 | |
| *** andreas_s has quit IRC | 09:51 | |
| stephenfin | tosky: So essentially don't worry about those until the shared jobs are finished? | 09:51 |
| *** andreas_s has joined #openstack-infra | 09:52 | |
| tosky | stephenfin: you can start building over the under-construction jobs and help testing them | 09:52 |
| tosky | at least from tempest job | 09:52 |
| tosky | for* | 09:52 |
| *** rtjure has joined #openstack-infra | 09:52 | |
| tosky | for example, this is my work in progress for Sahara https://review.openstack.org/#/c/512058/ ; ignore the scenario job, just check sahara-tests-tempest | 09:53 |
| stephenfin | Considering that I only just learnt what dsvm stands for, I might not be the best candidate. I can try though :) | 09:53 |
| tosky | this is the tip of the patches for devstack-tempest https://review.openstack.org/#/c/512057/ | 09:53 |
| tosky | oh | 09:53 |
| tosky | well, if you want to go for the slow route and you didn't do it already, you can import the legacy jobs in-tree (in your repositories) for now and wait | 09:54 |
| AJaeger | stephenfin: the idea here is that one repo defines those jobs - and other just use it. | 09:54 |
| AJaeger | stephenfin: work with andreaf and QA to define the jobs in tempest and grenade ;) | 09:55 |
| *** gcb has quit IRC | 09:56 | |
| stephenfin | tosky: That's what I was doing, but I'd no idea what to do with jobs like 'legacy-tempest-dsvm-multinode-live-migration', which are used by both 'openstack/nova' and 'openstack/tempest' | 09:56 |
| stephenfin | I can't simply move them to nova's repo or I'll lose that test coverage for tempest. That's my main question for now | 09:57 |
| *** rtjure has quit IRC | 09:57 | |
| stephenfin | There's a lot more of those too: nova and neutron in particular share a _lot_ of jobs | 09:57 |
| *** yamamoto has quit IRC | 09:58 | |
| *** liujiong has quit IRC | 09:58 | |
| *** liujiong has joined #openstack-infra | 09:59 | |
| *** jamesmcarthur has joined #openstack-infra | 09:59 | |
| *** yamamoto has joined #openstack-infra | 09:59 | |
| *** gcb has joined #openstack-infra | 10:00 | |
| *** andreas_s has quit IRC | 10:00 | |
| *** Guest38932 has quit IRC | 10:00 | |
| tosky | stephenfin: you can still migrate the legacy jobs to the place where the native job will live, and reference them from the other repository | 10:01 |
| tosky | even if we had all "native" jobs right now, we would have only a single definition for each one | 10:01 |
| *** andreas_s has joined #openstack-infra | 10:01 | |
| tosky | for example, I'm going to define the sahara-tempest job into sahara-tests, and it will be used for gating jobs in sahara-tests (where it is defined), in sahara and python-saharaclient (where it will be referred) | 10:02 |
| *** cuongnv has quit IRC | 10:03 | |
| *** rtjure has joined #openstack-infra | 10:03 | |
| *** jamesmcarthur has quit IRC | 10:04 | |
| *** spectr has quit IRC | 10:04 | |
| stephenfin | tosky: How do I ensure that zuul picks up the job? Simply install the dependency (e.g. via 'required-projects')? | 10:05 |
| strigazi | hi AJaeger , how can I recheck this https://review.openstack.org/367012 I get RETRY_LIMIT | 10:05 |
| *** kiennt26 has quit IRC | 10:05 | |
| *** yamamoto has quit IRC | 10:06 | |
| *** udesale has quit IRC | 10:06 | |
| *** pblaho has joined #openstack-infra | 10:06 | |
| *** andreas_s has quit IRC | 10:06 | |
| *** yamamoto has joined #openstack-infra | 10:06 | |
| tosky | stephenfin: good question; required-projects make sure that the content of the repository is copied to the worker nodes | 10:06 |
| *** gildub has joined #openstack-infra | 10:07 | |
| tosky | here you need the job on the executor, and I'm not sure if you really need anything special; the namespace of the job names is global | 10:07 |
| stephenfin | So something like 'legacy-tempest-dsvm-nova-v20-api' should probably live in the nova repo | 10:08 |
| *** rtjure has quit IRC | 10:08 | |
| stephenfin | I want to migrate that (with all the commit metadata like Depends-On), then ensure nova is in the 'required-projects' list for any other project using that job | 10:08 |
| *** annp has quit IRC | 10:08 | |
| stephenfin | Oh, wait - misunderstood you. I only need the first step so | 10:10 |
| tosky | you may have seen it, but in case: https://docs.openstack.org/infra/manual/zuulv3.html | 10:10 |
| tosky | the part about migrating legacy jobs | 10:10 |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config master: [openstack-helm] add job for opencontrail https://review.openstack.org/514210 | 10:10 |
| stephenfin | Sure have. Again though, migrating something like 'legacy-nova-tox-functional-py35' is easy enough. It's the ones that were used by multiple projects that were confusing me | 10:11 |
| *** calbers has joined #openstack-infra | 10:11 | |
| stephenfin | I really didn't want to move stuff to nova, only to find out that broke tempest, neutron, cinder etc. because the job definitions disappeared | 10:11 |
| stephenfin | But if it's a global namespace and things magically get registered, then it seems that's not an issue and I can just move anything that looks to be nova-first | 10:12 |
| tosky | I would maybe draft a list with a plan of changes and contact the interested projects to double check if the solution looks correct | 10:14 |
| tosky | a list of changed jobs* | 10:14 |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config master: [openstack-helm] add job for opencontrail https://review.openstack.org/514210 | 10:15 |
| stephenfin | Not a bad idea. I'll do that | 10:15 |
| *** andreas_s has joined #openstack-infra | 10:16 | |
| *** rtjure has joined #openstack-infra | 10:16 | |
| tosky | AJaeger: do you happen to know how to manage ansible modules which requires shade ('ERROR: shade is required for this module')? Should I just add it to required-projects or do I need to install it explicitly? | 10:16 |
| *** armaan has quit IRC | 10:19 | |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config master: [openstack-helm] add job for opencontrail https://review.openstack.org/514210 | 10:19 |
| *** armaan has joined #openstack-infra | 10:19 | |
| *** liujiong has quit IRC | 10:20 | |
| frickler | strigazi: this looks like a real error, I don't think a recheck will help http://logs.openstack.org/12/367012/21/check/puppet-openstack-syntax-4/719fe90/job-output.txt.gz#_2017-10-23_09_47_26_154905 | 10:20 |
| *** rtjure has quit IRC | 10:21 | |
| *** rossella_s has quit IRC | 10:21 | |
| *** spectr has joined #openstack-infra | 10:21 | |
| openstackgerrit | Rico Lin proposed openstack-infra/project-config master: Remove legacy jobs from heat https://review.openstack.org/509194 | 10:21 |
| *** namnh has quit IRC | 10:22 | |
| AJaeger | stephenfin: convert what you're sure about - and do the rest later after dsicussion - in this case with QA team | 10:24 |
| AJaeger | strigazi: something looks completely broken if you get RETRY_LIMIT everywhere, will check | 10:24 |
| *** andreas_s has quit IRC | 10:25 | |
| *** rtjure has joined #openstack-infra | 10:25 | |
| *** gmann is now known as gmann_afk | 10:26 | |
| AJaeger | strigazi: best wait for mnaser and dmsimard to be awake on that one | 10:26 |
| AJaeger | tosky: I suggest to discuss with mordred | 10:27 |
| tosky | AJaeger: ack, thanks | 10:27 |
| *** wolverineav has joined #openstack-infra | 10:30 | |
| *** Guest97374 has quit IRC | 10:30 | |
| *** andreas_s has joined #openstack-infra | 10:30 | |
| *** rtjure has quit IRC | 10:30 | |
| *** andreas_s has quit IRC | 10:35 | |
| *** wolverineav has quit IRC | 10:35 | |
| *** rtjure has joined #openstack-infra | 10:35 | |
| *** pbourke has quit IRC | 10:37 | |
| *** yamamoto_ has joined #openstack-infra | 10:37 | |
| *** pbourke has joined #openstack-infra | 10:39 | |
| *** andreas_s has joined #openstack-infra | 10:39 | |
| *** rtjure has quit IRC | 10:40 | |
| *** janki has joined #openstack-infra | 10:40 | |
| *** yamamoto has quit IRC | 10:40 | |
| openstackgerrit | Rico Lin proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs for heat https://review.openstack.org/513576 | 10:42 |
| *** andreas_s has quit IRC | 10:42 | |
| *** andreas_s has joined #openstack-infra | 10:42 | |
| *** ldnunes has joined #openstack-infra | 10:43 | |
| openstackgerrit | Adrian Czarnecki proposed openstack-infra/openstack-zuul-jobs master: Remove monasca legacy job https://review.openstack.org/514251 | 10:44 |
| *** spectr has quit IRC | 10:45 | |
| *** spectr has joined #openstack-infra | 10:47 | |
| openstackgerrit | Adrian Czarnecki proposed openstack-infra/openstack-zuul-jobs master: Remove monasca legacy job https://review.openstack.org/514251 | 10:48 |
| *** yamamoto_ has quit IRC | 10:48 | |
| openstackgerrit | Adrian Czarnecki proposed openstack-infra/openstack-zuul-jobs master: Remove monasca legacy job https://review.openstack.org/514251 | 10:49 |
| *** yamamoto has joined #openstack-infra | 10:50 | |
| *** Qiming has quit IRC | 10:50 | |
| *** Qiming has joined #openstack-infra | 10:54 | |
| *** rtjure has joined #openstack-infra | 10:55 | |
| *** yamamoto has quit IRC | 10:55 | |
| *** yamamoto has joined #openstack-infra | 10:55 | |
| *** gcb has quit IRC | 10:58 | |
| *** Qiming has quit IRC | 10:58 | |
| *** vponomaryov has quit IRC | 10:59 | |
| *** vponomaryov has joined #openstack-infra | 10:59 | |
| *** rtjure has quit IRC | 10:59 | |
| *** Qiming has joined #openstack-infra | 11:01 | |
| *** armaan has quit IRC | 11:03 | |
| *** armaan has joined #openstack-infra | 11:03 | |
| *** AJaeger has quit IRC | 11:03 | |
| *** thiagolib has joined #openstack-infra | 11:03 | |
| *** dave-mccowan has joined #openstack-infra | 11:03 | |
| *** rtjure has joined #openstack-infra | 11:04 | |
| flaper87 | can I add a full git url to required-projects in a job definition? | 11:05 |
| * flaper87 should just try this | 11:05 | |
| strigazi | AJaeger ok thanks | 11:06 |
| *** nicolasbock has joined #openstack-infra | 11:07 | |
| *** vponomaryov has quit IRC | 11:09 | |
| *** rtjure has quit IRC | 11:09 | |
| *** yolanda has quit IRC | 11:09 | |
| *** dbecker has quit IRC | 11:09 | |
| *** AJaeger has joined #openstack-infra | 11:11 | |
| openstackgerrit | Adrian Czarnecki proposed openstack-infra/project-config master: Remove legacy job for monasca https://review.openstack.org/513722 | 11:12 |
| *** vponomaryov has joined #openstack-infra | 11:13 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 11:13 |
| openstackgerrit | Adrian Czarnecki proposed openstack-infra/project-config master: Remove legacy job for monasca https://review.openstack.org/513722 | 11:14 |
| *** tpsilva has joined #openstack-infra | 11:14 | |
| *** ykarel is now known as ykarel|afk | 11:15 | |
| *** rtjure has joined #openstack-infra | 11:17 | |
| *** lucasagomes is now known as lucas-hungry | 11:17 | |
| *** jkilpatr has joined #openstack-infra | 11:19 | |
| *** vponomaryov has quit IRC | 11:20 | |
| *** smatzek has joined #openstack-infra | 11:20 | |
| *** vponomaryov has joined #openstack-infra | 11:20 | |
| *** rtjure has quit IRC | 11:22 | |
| *** gildub has quit IRC | 11:24 | |
| *** stakeda has quit IRC | 11:26 | |
| *** rtjure has joined #openstack-infra | 11:28 | |
| *** yamamoto_ has joined #openstack-infra | 11:30 | |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config master: [openstack-helm] add job for opencontrail https://review.openstack.org/514210 | 11:30 |
| *** LindaWang has joined #openstack-infra | 11:31 | |
| *** yamamoto_ has quit IRC | 11:31 | |
| *** rtjure has quit IRC | 11:32 | |
| *** yamamoto has quit IRC | 11:32 | |
| *** yamamoto has joined #openstack-infra | 11:32 | |
| *** jpena is now known as jpena|lunch | 11:36 | |
| *** rtjure has joined #openstack-infra | 11:38 | |
| *** pgadiya has quit IRC | 11:38 | |
| *** nicolasbock has quit IRC | 11:39 | |
| *** jtomasek has quit IRC | 11:39 | |
| *** Hal has joined #openstack-infra | 11:40 | |
| *** Hal is now known as Guest57916 | 11:40 | |
| *** erlon has joined #openstack-infra | 11:42 | |
| *** dtantsur is now known as dtantsur|brb | 11:42 | |
| *** rtjure has quit IRC | 11:43 | |
| *** ykarel|afk is now known as ykarel | 11:43 | |
| *** janki has quit IRC | 11:44 | |
| *** pblaho has quit IRC | 11:44 | |
| *** ldnunes has quit IRC | 11:46 | |
| *** ldnunes has joined #openstack-infra | 11:47 | |
| dmsimard | flaper87: why a full git url ? | 11:47 |
| dmsimard | You mean like github or something ? | 11:48 |
| *** rtjure has joined #openstack-infra | 11:48 | |
| flaper87 | dmsimard: yes | 11:49 |
| dmsimard | flaper87: for a project to be in 'required-projects', it must first be configured to be in Zuul -- that list of projects in main.yaml. | 11:50 |
| dmsimard | It's the equivalent of Zuul Cloner, basically | 11:51 |
| flaper87 | dmsimard: gotcha, thanks. That answers my question | 11:51 |
| *** yamamoto has quit IRC | 11:52 | |
| *** nicolasbock has joined #openstack-infra | 11:52 | |
| *** rosmaita has joined #openstack-infra | 11:52 | |
| dmsimard | flaper87: you would need to setup a job with a 'pre-run' which does a git clone of the stuff outside OpenStack that you need. | 11:52 |
| *** rtjure has quit IRC | 11:53 | |
| flaper87 | dmsimard: that's exactly what I did | 11:53 |
| flaper87 | :) | 11:53 |
| *** yamamoto has joined #openstack-infra | 11:53 | |
| dmsimard | Well there you go | 11:54 |
| *** LindaWang has quit IRC | 11:57 | |
| *** yamamoto has quit IRC | 11:58 | |
| *** rtjure has joined #openstack-infra | 11:58 | |
| *** thorst has joined #openstack-infra | 11:59 | |
| *** thorst_ has joined #openstack-infra | 12:00 | |
| *** thorst_ has quit IRC | 12:01 | |
| *** witek_ is now known as witek | 12:02 | |
| *** rtjure has quit IRC | 12:03 | |
| *** liusheng has quit IRC | 12:03 | |
| *** thorst has quit IRC | 12:04 | |
| *** salv-orlando has joined #openstack-infra | 12:04 | |
| *** thorst has joined #openstack-infra | 12:04 | |
| *** nicolasbock has quit IRC | 12:05 | |
| *** nicolasbock has joined #openstack-infra | 12:06 | |
| *** rlandy has joined #openstack-infra | 12:06 | |
| *** LindaWang has joined #openstack-infra | 12:07 | |
| *** jtomasek has joined #openstack-infra | 12:08 | |
| *** rtjure has joined #openstack-infra | 12:10 | |
| *** rossella_s has joined #openstack-infra | 12:12 | |
| *** rtjure has quit IRC | 12:15 | |
| *** dhellmann has quit IRC | 12:15 | |
| *** jpich has joined #openstack-infra | 12:15 | |
| *** dhellmann has joined #openstack-infra | 12:16 | |
| *** huanxie has quit IRC | 12:16 | |
| *** lucas-hungry is now known as lucasagomes | 12:18 | |
| *** trown|outtypewww is now known as trown | 12:18 | |
| *** weshay|bbiab is now known as weshay|ruck | 12:18 | |
| *** dbecker has joined #openstack-infra | 12:20 | |
| *** rtjure has joined #openstack-infra | 12:21 | |
| *** edmondsw has joined #openstack-infra | 12:21 | |
| *** dprince has joined #openstack-infra | 12:21 | |
| *** edmondsw has quit IRC | 12:21 | |
| *** edmondsw has joined #openstack-infra | 12:22 | |
| *** e0ne has quit IRC | 12:25 | |
| *** rtjure has quit IRC | 12:26 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Remove legacy tempest-tox-plugin-sanity-check jobs https://review.openstack.org/514271 | 12:26 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Remove legacy tempest-tox-plugin-sanity-check jobs https://review.openstack.org/514273 | 12:29 |
| *** pblaho has joined #openstack-infra | 12:30 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Remove legacy tempest-tox-plugin-sanity-check jobs https://review.openstack.org/514271 | 12:30 |
| *** claudiub|2 has joined #openstack-infra | 12:30 | |
| *** claudiub has quit IRC | 12:33 | |
| chandankumar | dmsimard: for in repo tox specific jobs we also need pre-run? | 12:34 |
| dmsimard | chandankumar: I'm not sure I understand your question | 12:34 |
| chandankumar | dmsimard: for this one https://review.openstack.org/#/c/514269/ | 12:35 |
| *** yamamoto has joined #openstack-infra | 12:36 | |
| dmsimard | chandankumar: zuul comments on what the syntax error is, I added a comment | 12:36 |
| chandankumar | dmsimard: thanks :-) | 12:38 |
| ykarel | dmsimard, if using ubuntu-xenial then hosts also has to be taken care off:-https://review.openstack.org/#/c/514269/2/playbooks/tempest-tox-plugin-sanity-check/post.yaml@1 | 12:39 |
| *** nicolasbock has quit IRC | 12:39 | |
| *** jpena|lunch is now known as jpena | 12:40 | |
| *** ociuhandu has quit IRC | 12:40 | |
| *** pcaruana has quit IRC | 12:41 | |
| dmsimard | ykarel: yes | 12:41 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Use playbook from tripleo-quickstart-extras for OVB https://review.openstack.org/513508 | 12:41 |
| ykarel | dmsimard, Ok, https://github.com/openstack-infra/openstack-zuul-jobs/blob/455d621bf575351471e75d24256de95b2b07aa29/zuul.d/nodesets.yaml#L38-L42 | 12:41 |
| ykarel | chandankumar, ^^ | 12:41 |
| chandankumar | sorr fixing it | 12:41 |
| *** ankkumar_ has joined #openstack-infra | 12:45 | |
| *** yamamoto has quit IRC | 12:46 | |
| *** ankkumar has quit IRC | 12:46 | |
| *** nicolasbock has joined #openstack-infra | 12:46 | |
| *** ankkumar_ has quit IRC | 12:46 | |
| *** yolanda has joined #openstack-infra | 12:47 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Create rescue user on ironic agent https://review.openstack.org/506579 | 12:50 |
| *** vponomaryov has quit IRC | 12:52 | |
| *** vponomaryov has joined #openstack-infra | 12:52 | |
| *** huanxie has joined #openstack-infra | 12:56 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config master: Ensure branch overrides are correctly set for networking-cisco https://review.openstack.org/512588 | 12:56 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix releasenotes job for VPNaaS dashboard https://review.openstack.org/514123 | 12:57 |
| *** lennyb has quit IRC | 12:59 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in nova https://review.openstack.org/514310 | 13:01 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/project-config master: Remove legacy jobs in nova https://review.openstack.org/514311 | 13:01 |
| *** vponomaryov has quit IRC | 13:01 | |
| *** vponomaryov has joined #openstack-infra | 13:01 | |
| stephenfin | sdague: Not sure how "up" on zuul v3 you are, but I think you'd be a good person to look at those ^ ? | 13:02 |
| openstackgerrit | Corey Bryant proposed openstack-infra/project-config master: snaps: add snap-gnocchi definitions https://review.openstack.org/500964 | 13:02 |
| stephenfin | The nova change is here https://review.openstack.org/514309 | 13:02 |
| *** yamamoto has joined #openstack-infra | 13:02 | |
| *** mat128 has quit IRC | 13:03 | |
| *** jtomasek has quit IRC | 13:05 | |
| *** lennyb has joined #openstack-infra | 13:05 | |
| *** vponomaryov has quit IRC | 13:07 | |
| *** vponomaryov has joined #openstack-infra | 13:07 | |
| *** bobh has joined #openstack-infra | 13:07 | |
| *** mriedem has joined #openstack-infra | 13:07 | |
| dmsimard | project-config-core: need a +3 on https://review.openstack.org/#/c/513930/ to unblock puppet-openstackci please | 13:09 |
| *** jamesmcarthur has joined #openstack-infra | 13:09 | |
| *** vponomaryov has quit IRC | 13:09 | |
| *** vponomaryov has joined #openstack-infra | 13:09 | |
| dmsimard | clarkb: is it just me or does devstack-gate start to be redundant with v3 ? | 13:10 |
| ykarel | just copying project stanza in stable branch works? i did it in: https://review.openstack.org/#/c/514272/ as per: http://lists.openstack.org/pipermail/openstack-dev/2017-October/123688.html | 13:11 |
| *** jtomasek has joined #openstack-infra | 13:11 | |
| ykarel | i did something wrong? | 13:11 |
| ykarel | jeblair, ^^ | 13:12 |
| dmsimard | ykarel: I'm not fully awake yet but I *think* a zuul.yaml applies to all branches unless you explicitely match branches in your jobs | 13:12 |
| AJaeger | ykarel: you need playbooks as well | 13:12 |
| dmsimard | but yes, like AJaeger mentions you need playbooks | 13:12 |
| AJaeger | https://docs.openstack.org/infra/manual/zuulv3.html#stable-branches | 13:12 |
| dmsimard | AJaeger: ah, so a zuul.yaml is needed for every branch, okay | 13:13 |
| chandankumar | AJaeger: depends on for this is added here https://review.openstack.org/#/c/514271/ but it is not working place have a look | 13:13 |
| ykarel | AJaeger, yes find it in the doc: If you have playbooks or roles included on the master branch, backport these as well., Thanks | 13:13 |
| *** salv-orl_ has joined #openstack-infra | 13:15 | |
| openstackgerrit | Sam Betts proposed openstack-infra/openstack-zuul-jobs master: [DNM] THIS IS A TEST https://review.openstack.org/514314 | 13:16 |
| AJaeger | chandankumar: did you read the message by zuul? | 13:16 |
| *** salv-orlando has quit IRC | 13:16 | |
| AJaeger | chandankumar: it tells you that the dependend change needs to merge first - and then you can recheck | 13:16 |
| AJaeger | chandankumar: so, ignore for now | 13:17 |
| *** salv-orl_ has quit IRC | 13:17 | |
| *** salv-orlando has joined #openstack-infra | 13:18 | |
| mnaser | dmsimard i think we can drop puppet-openstack-integration from legacy-openstackci-base because it was only there because of a bug in the converter | 13:19 |
| mnaser | re: 513930 at least | 13:19 |
| AJaeger | bbl | 13:20 |
| *** vponomaryov has quit IRC | 13:20 | |
| dmsimard | mnaser: sure, I can do it in followup ? just want to unblock it | 13:20 |
| dmsimard | mnaser: was copy pasta :) | 13:20 |
| mnaser | dmsimard im okay with that, one sec | 13:21 |
| *** xinliang has quit IRC | 13:23 | |
| *** lennyb has quit IRC | 13:23 | |
| *** sree has joined #openstack-infra | 13:23 | |
| sdague | stephenfin: I'm not very up on it | 13:24 |
| *** baoli has joined #openstack-infra | 13:25 | |
| stephenfin | sdague: Darn :( | 13:25 |
| *** gcb has joined #openstack-infra | 13:25 | |
| *** smatzek has quit IRC | 13:26 | |
| mnaser | dmsimard parent of jobs should be legacy-openstackci-beaker no? | 13:26 |
| *** smatzek has joined #openstack-infra | 13:27 | |
| mnaser | oh | 13:27 |
| mnaser | nvm | 13:27 |
| *** huanxie has quit IRC | 13:27 | |
| *** lennyb has joined #openstack-infra | 13:27 | |
| *** slaweq has quit IRC | 13:28 | |
| *** slaweq has joined #openstack-infra | 13:28 | |
| *** jamesmcarthur has quit IRC | 13:29 | |
| *** makowals has quit IRC | 13:29 | |
| smcginnis | mnaser: Just checking - is it a mistake that the puppet jobs ended up with publish-to-pypi? http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml#n24415 | 13:30 |
| *** eumel8 has quit IRC | 13:30 | |
| *** smatzek has quit IRC | 13:31 | |
| *** makowals has joined #openstack-infra | 13:31 | |
| *** abishop has joined #openstack-infra | 13:31 | |
| *** slaweq has quit IRC | 13:33 | |
| AJaeger | smcginnis: mordred, looks like a bug in the conversion - comparing with zuul/layout.yaml ^ | 13:33 |
| smcginnis | Yeah, seems like a mistake. | 13:33 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Use playbook from tripleo-quickstart-extras for OVB https://review.openstack.org/513508 | 13:34 |
| *** Goneri has quit IRC | 13:34 | |
| *** abishop has left #openstack-infra | 13:34 | |
| *** jcoufal has joined #openstack-infra | 13:34 | |
| fungi | dmsimard: as far as devsatack-gate becoming redundant under zuul v3, yes. the idea is that its actions become roles/playbooks in our general-purpose library and/or in the devstack repo, and eventually we can probably retire devstack-gate entirely | 13:35 |
| *** Goneri has joined #openstack-infra | 13:35 | |
| dmsimard | fungi: ok, just making sure my assessment was correct | 13:36 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add missing required-projects for puppet-openstackci https://review.openstack.org/513930 | 13:37 |
| *** vponomaryov has joined #openstack-infra | 13:38 | |
| *** ykarel has quit IRC | 13:38 | |
| *** yolanda has quit IRC | 13:43 | |
| mnaser | smcginnis yes it is a mistake, we should disable that most likely | 13:43 |
| *** rpittau_ has joined #openstack-infra | 13:44 | |
| *** salv-orl_ has joined #openstack-infra | 13:44 | |
| *** eharney has joined #openstack-infra | 13:44 | |
| *** salv-orlando has quit IRC | 13:45 | |
| *** mat128 has joined #openstack-infra | 13:45 | |
| AJaeger | mnaser, smcginnis : Will either of you fix this? Or do we need to write it on the etherpad and wait until takes it? | 13:46 |
| smcginnis | mnaser: I'm at a conference the next couple days, so a little distracted. Can you get that? | 13:47 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/project-config master: Remove legacy jobs in nova https://review.openstack.org/514311 | 13:47 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/openstack-zuul-jobs master: Remove legacy nodepool jobs https://review.openstack.org/514321 | 13:51 |
| *** hemna_ has joined #openstack-infra | 13:51 | |
| *** ykarel has joined #openstack-infra | 13:56 | |
| *** hongbin has joined #openstack-infra | 13:56 | |
| *** yolanda has joined #openstack-infra | 13:57 | |
| *** esberglu has joined #openstack-infra | 13:58 | |
| *** tojuvone has joined #openstack-infra | 13:58 | |
| *** gcb has quit IRC | 13:58 | |
| *** gcb has joined #openstack-infra | 13:58 | |
| tosky | fungi: talking about that... in the previous world, when a test was shared between a library and a non-library, we had to define DEVSTACK_PROJECT_FROM_GIT=<library> when using the test on the library | 14:02 |
| tosky | because iirc the library are taken from pip | 14:02 |
| tosky | how does it work in the new world? | 14:02 |
| *** ihrachys has joined #openstack-infra | 14:02 | |
| *** tojuvone has left #openstack-infra | 14:02 | |
| chandankumar | AJaeger: is it necesaary to add openstack/requirements under required-projects in .zuul-yaml? but why | 14:03 |
| AJaeger | chandankumar: if you need it ;) | 14:04 |
| chandankumar | AJaeger: for example here http://logs.openstack.org/69/514269/6/check/tempest-tox-plugin-sanity-check/55bc06a/job-output.txt.gz | 14:04 |
| AJaeger | chandankumar: if you need upper-constraints file, you need requirements | 14:04 |
| chandankumar | AJaeger: ah got it :-) | 14:05 |
| *** sree_ has joined #openstack-infra | 14:05 | |
| *** dtantsur|brb is now known as dtantsur | 14:05 | |
| *** sree has quit IRC | 14:05 | |
| *** sree_ is now known as Guest52014 | 14:05 | |
| *** jamesmcarthur has joined #openstack-infra | 14:06 | |
| *** jaosorior has quit IRC | 14:08 | |
| *** jtomasek has quit IRC | 14:08 | |
| *** ociuhandu has joined #openstack-infra | 14:09 | |
| *** jtomasek has joined #openstack-infra | 14:09 | |
| *** smatzek has joined #openstack-infra | 14:09 | |
| *** smatzek has quit IRC | 14:10 | |
| *** smatzek has joined #openstack-infra | 14:10 | |
| *** ociuhandu has quit IRC | 14:12 | |
| *** rbrndt has joined #openstack-infra | 14:13 | |
| *** slaweq has joined #openstack-infra | 14:13 | |
| fungi | tosky: devstack still needs that variable as far as i know, but you also have to add the same projects to the job's required-projects list so that they'll be made available to it on the filesystem | 14:14 |
| chandankumar | AJaeger: i am confused about the parent https://review.openstack.org/#/c/514269/7/.zuul.yaml currently there is no other job there. what should i put in parent? | 14:14 |
| AJaeger | chandankumar: you migrate a job from openstack-zuul-jobs wehre it has a parent. | 14:14 |
| AJaeger | chandankumar: keep in mind we have a global namespace, you can use jobs from other repos | 14:14 |
| tosky | fungi: ack, the variable part was relevant, thanks; that means that I need to derive a new job though | 14:15 |
| tosky | fungi: also, do you know where should I put shade so that things like os_image works in my job? required-project seems to not be enough (of course shade probably needs to be installed) | 14:16 |
| ykarel | AJaeger, isn't "base" used as default parent if not specified? | 14:16 |
| tosky | context: a job which uses devstack and returns "ERROR: shade is required for this module" when I try to use os_image in a local role | 14:16 |
| *** slaweq has quit IRC | 14:18 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:18 | |
| tosky | AJaeger: but iirc there is an implicit parent | 14:18 |
| tosky | AJaeger: unless that job required a specific parent | 14:18 |
| *** felipemonteiro__ has joined #openstack-infra | 14:19 | |
| AJaeger | tosky: you're right, we have a default parent. I would still set it - the legacy parent might be needed ;) | 14:20 |
| *** gouthamr has joined #openstack-infra | 14:20 | |
| AJaeger | tosky: requirements file? | 14:20 |
| *** salv-orl_ has quit IRC | 14:20 | |
| *** wolverineav has joined #openstack-infra | 14:20 | |
| *** salv-orlando has joined #openstack-infra | 14:20 | |
| *** david-lyle has joined #openstack-infra | 14:21 | |
| AJaeger | tosky: the parent is legacy-base in the change and that does more than base, I would keep it... | 14:21 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: fix puppet release jobs https://review.openstack.org/514334 | 14:21 |
| AJaeger | tosky: last comment is for you and chandankumar | 14:21 |
| tosky | oh, I see | 14:21 |
| tosky | and the "requirements file" is the answer for the question on shade? | 14:22 |
| *** dhinesh has joined #openstack-infra | 14:22 | |
| smcginnis | mnaser: dhellmann got the jobs fixed up. | 14:22 |
| AJaeger | tosky: yes, requiremetns file was one suggestion for shade | 14:23 |
| * AJaeger goes back meeting | 14:23 | |
| *** felipemonteiro_ has quit IRC | 14:23 | |
| tosky | thanks, I will try it | 14:23 |
| Shrews | tosky: AJaeger: i think you actually want the tox-siblings role. shade uses this in some of it's jobs | 14:26 |
| dhellmann | mnaser , smcginnis : after that 514334 merges we should re-queue the tag jobs for puppet-vitgrage so the right ones run | 14:26 |
| tosky | Shrews: tox-siblings? But I'm not using tox | 14:27 |
| *** dhinesh has quit IRC | 14:27 | |
| tosky | or better, I'm not using tox when I'm using os_image | 14:27 |
| *** jtomasek has quit IRC | 14:27 | |
| Shrews | tosky: are you using devstack? | 14:27 |
| tosky | I use tox later in the process, but that's separate | 14:27 |
| tosky | Shrews: I do, yes, this is a job child of "devstack" | 14:28 |
| Shrews | tosky: take a look at http://git.openstack.org/cgit/openstack-infra/shade/tree/playbooks/devstack/pre.yaml | 14:29 |
| Shrews | tosky: i don't have full context, so that may or may not be useful to you. just though i'd mention it | 14:29 |
| *** gcb has quit IRC | 14:29 | |
| tosky | Shrews: thanks for that; I have the feeling that my use case may be different, but good to know nevertheless (for other use cases) | 14:30 |
| * tosky waits for mordred :) | 14:30 | |
| Shrews | but the shade jobs use devstack, then run some ansible playbooks within a tox env to test the os-* modules | 14:30 |
| *** ccamacho has quit IRC | 14:32 | |
| *** makowals has quit IRC | 14:33 | |
| jeblair | tosky: can you describe briefly what the job should do? | 14:33 |
| jeblair | (or point me at a change in progress) | 14:34 |
| *** annp has joined #openstack-infra | 14:34 | |
| *** cshastri has quit IRC | 14:35 | |
| *** yolanda has quit IRC | 14:36 | |
| *** slaweq has joined #openstack-infra | 14:37 | |
| *** slaweq has quit IRC | 14:37 | |
| *** slaweq has joined #openstack-infra | 14:37 | |
| *** makowals has joined #openstack-infra | 14:38 | |
| *** xarses has joined #openstack-infra | 14:41 | |
| *** baoli has quit IRC | 14:41 | |
| *** psachin has quit IRC | 14:42 | |
| *** baoli has joined #openstack-infra | 14:42 | |
| *** camunoz has joined #openstack-infra | 14:43 | |
| EmilienM | pabelanger: how iptables thing has been fixed? | 14:44 |
| dhellmann | jeblair : one thing I kept running into last week with all of the release job debugging was getting confused about which browser tab when with which thing I was debugging. How hard would it be to dump the equivalent of "git show" at the top of each log as a way to differentiate why a log was triggered? Is that something we would have to add to each job separately? or is there a base job we could add that to? | 14:44 |
| dhellmann | s/when/went/ | 14:44 |
| tosky | jeblair: sahara-tests-scenario here: https://review.openstack.org/#/c/512058/ | 14:44 |
| tosky | jeblair: and I apologize, I may disconnect really soon, but I will reconnect soon | 14:45 |
| tosky | well, soon, at most in two hours | 14:45 |
| tosky | jeblair: the last test result shows a failure because os_image, correctly, does not find shade | 14:45 |
| pabelanger | EmilienM: I don't know | 14:45 |
| jeblair | dhellmann: i think dmsimard &co have been working on improving the top-of-job boilerplate (so it has good change links, etc) | 14:45 |
| dhellmann | oh, cool | 14:46 |
| *** tosky has quit IRC | 14:46 | |
| jeblair | dhellmann: the commit message subject of the change (or changes) in test would take some more plumbing in zuul, but i bet we could add that if more context would be helpful (it's a balancing act between not helpful and too helpful :) | 14:47 |
| jeblair | dhellmann: let's check and see if all dmsimard's changes have gone in and what that looks like, and then see if we need more. | 14:48 |
| *** armaan has quit IRC | 14:48 | |
| dhellmann | yeah, I can see it being not that useful 90% of the time but last week it would have been 100% useful, so... | 14:48 |
| dhellmann | k | 14:48 |
| dhellmann | I could also just add some detail to the release jobs themselves | 14:48 |
| dhellmann | if the info I'm thinking of isn't seen as generally useful, that is | 14:48 |
| *** armaan has joined #openstack-infra | 14:48 | |
| jeblair | dhellmann: would the commit msg subject be enough for what you're imagining, or would you need the whole commit message (so you can see the release meta tags)? | 14:49 |
| dhellmann | hmm | 14:50 |
| dhellmann | the release jobs spit out the metadata as they run, so the subject is probably enough | 14:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Document executor/merger stats https://review.openstack.org/514343 | 14:50 |
| jeblair | dhellmann: ok | 14:50 |
| *** jtomasek has joined #openstack-infra | 14:52 | |
| clarkb | AJaeger: I think the rtd thing required we restart all executors to pick up the fix and I don't know that that has happened yet | 14:54 |
| AJaeger | clarkb: can you check the job whether it run on a restarted executor? | 14:56 |
| *** jaosorior has joined #openstack-infra | 14:57 | |
| jeblair | clarkb, AJaeger: i thought fungi did that fridayish? | 14:58 |
| *** yamahata has joined #openstack-infra | 14:59 | |
| *** annp has quit IRC | 14:59 | |
| clarkb | jeblair: ya checking ps output they appear to all have been done | 14:59 |
| jeblair | pabelanger: can you look at https://review.openstack.org/513533 ? | 14:59 |
| clarkb | AJaeger: so I was mistaken t hey were all restarted on the 21st (last friday) | 15:00 |
| clarkb | probably next thing to do is run bwrap locally and see what needs doing for working ssl | 15:01 |
| fungi | clarkb: jeblair: AJaeger: i also #status log'd a note about the restarts once i had completed the last one | 15:01 |
| *** ccamacho has joined #openstack-infra | 15:01 | |
| clarkb | I'll probably poke at that once I get my zuul change for auto hold sorted out this morning if no one beats me to it | 15:01 |
| fungi | executor restarts were complete at 02:33z saturday | 15:02 |
| fungi | according to https://wiki.openstack.org/wiki/Infrastructure_Status | 15:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 15:02 |
| pabelanger | jeblair: +3 | 15:02 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: always make the requirements repo available for release jobs https://review.openstack.org/514348 | 15:02 |
| *** jamesmcarthur has quit IRC | 15:02 | |
| dhellmann | smcginnis , pabelanger , jeblair : we need ^^ to allow us to release python libraries | 15:03 |
| *** jamesmcarthur has joined #openstack-infra | 15:03 | |
| dhellmann | oh, that might not be enough | 15:03 |
| frickler | is broken stream output still an issue? I'm seeing no output for http://zuulv3.openstack.org/static/stream.html?uuid=d59f56fd1fc142b9bab1f97fe5c8f9a5&logfile=console.log but the other two jobs still running for this patch do show proper console output https://review.openstack.org/340763 | 15:04 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: always make the requirements repo available for release jobs https://review.openstack.org/514348 | 15:04 |
| pabelanger | frickler: possible, there is a known issue where zuul-exxecutor stopped listening on tcp/79 | 15:05 |
| jeblair | Shrews: did your extra logging changes land? (see question from frickler) | 15:05 |
| pabelanger | let me check executors | 15:05 |
| Shrews | jeblair: oh, no. i forgot to clean that up this morning. let me do that now | 15:06 |
| dhellmann | pabelanger : thanks for that +2, but I updated the patch. please take another look when you have a sec. | 15:07 |
| fungi | jeblair: in case you haven't seen it yet, there were some gertty questions on the dev ml yesterday: http://lists.openstack.org/pipermail/openstack-dev/2017-October/123862.html | 15:07 |
| dmsimard | dhellmann: hey there -- https://review.openstack.org/#/c/509436/ should hopefully be merging soon, I just did a recheck. | 15:07 |
| dmsimard | dhellmann: that gives a quick summary of what the job is, where it is running and why it is running | 15:08 |
| dhellmann | dmsimard : oh, nice! | 15:08 |
| dhellmann | that will be very useful, thanks! | 15:08 |
| clarkb | heh the man page for bwrap is broken on suse | 15:09 |
| dmsimard | dhellmann: like jeblair mentioned, getting the git commit ref in there would need a bit of plumbing but hopefully that patch improves things in the meanwhile | 15:09 |
| clarkb | (I should file a bug) | 15:09 |
| *** e0ne has joined #openstack-infra | 15:10 | |
| dhellmann | dmsimard : that looks like it will improve things. and I can make the release jobs dump more data in the scripts they run, too | 15:10 |
| dhellmann | smcginnis, pabelanger : I need to step out to run an errand. When https://review.openstack.org/#/c/514348/2 merges, we need to requeue the 2.11.0 tag for ironic-lib (details on line 90 of https://etherpad.openstack.org/p/zuulv3-issues) | 15:12 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add log streaming logging and exception handling https://review.openstack.org/513811 | 15:13 |
| *** baoli has quit IRC | 15:13 | |
| Shrews | jeblair: ^^^ I don't think that solves the listening process going away, but is probably good to have anyway | 15:14 |
| openstackgerrit | Edgar Magana proposed openstack-infra/irc-meetings master: Include a new time for UC IRC meetings https://review.openstack.org/513863 | 15:15 |
| clarkb | fungi: reading zuul's bwrap invocation more closely it was already bind mounting /usr to /usr ... this is very confusing | 15:15 |
| fungi | so something else is causing it to be unable to validate the server cert for the rtd api? | 15:16 |
| *** kiennt26 has joined #openstack-infra | 15:17 | |
| *** andreas_s has quit IRC | 15:18 | |
| clarkb | ya I think so, I'm still trying to get local bwrap to exec bash with a more restricted bind mount than / | 15:18 |
| *** andreas_s has joined #openstack-infra | 15:18 | |
| clarkb | in an effort to try and reproduce | 15:18 |
| *** ykarel is now known as ykarel|afk | 15:19 | |
| *** salv-orlando has quit IRC | 15:19 | |
| jeblair | clarkb: there's a zuul-bwrap command | 15:20 |
| *** salv-orlando has joined #openstack-infra | 15:20 | |
| jeblair | can use it to get yourself a bwrapped shell on an executor | 15:20 |
| clarkb | cool that will likely be helpful too. Though I think I have reproduced the behavior locally at least | 15:21 |
| openstackgerrit | Merged openstack-infra/system-config master: Update signing key rotation instructions for Zuul v3 https://review.openstack.org/513533 | 15:22 |
| *** LindaWang has quit IRC | 15:24 | |
| *** baoli has joined #openstack-infra | 15:24 | |
| EmilienM | dmsimard: how iptables thing has been fixed? | 15:24 |
| *** Apoorva has joined #openstack-infra | 15:24 | |
| *** Apoorva has quit IRC | 15:26 | |
| dmsimard | EmilienM: nothing has merged yet | 15:27 |
| dmsimard | EmilienM: if it's fixed, it's not something I did | 15:27 |
| dmsimard | EmilienM: I sorted out some issues during the weekend | 15:27 |
| EmilienM | weshay|ruck: ^ interesting | 15:28 |
| EmilienM | weshay|ruck: I'm going to check packaging diff again to see if something changed again | 15:28 |
| *** Apoorva has joined #openstack-infra | 15:29 | |
| weshay|ruck | iptables was the same | 15:29 |
| fungi | clarkb: yeah, confirmed by default zuul-bwrap shows a bubblewrap command including "... --ro-bind /usr /usr ..." | 15:29 |
| dmsimard | EmilienM, weshay|ruck: my patches are here: https://review.openstack.org/#/q/topic:iptables-persist | 15:30 |
| smcginnis | Could we get another look at https://review.openstack.org/#/c/514348/ to handle some release issues? | 15:30 |
| clarkb | fungi: `bwrap --ro-bind /usr /usr --ro-bind /lib /lib --ro-bind /sbin /sbin --ro-bind /bin /bin --ro-bind /lib64 /lib64 --ro-bind /etc/resolv.conf /etc/resolv.conf --ro-bind /etc/ssl/certs /etc/ssl/certs --dev /dev bash` is enough to get wget working in that shell. leaving off the /etc/ssl/certs does not work | 15:30 |
| EmilienM | dmsimard, weshay|ruck: I checked, no changed in centos7 this weekend - so something happened in infra probably | 15:31 |
| dmsimard | EmilienM: didn't you end up merging that workaround patch ? | 15:31 |
| EmilienM | no | 15:31 |
| dmsimard | that downgrades iptables | 15:31 |
| EmilienM | we merged nothing | 15:31 |
| EmilienM | and centos7 didn't update any rpm, I checked | 15:31 |
| clarkb | fungi: ps -elf | grep bwrap doesn't show /etc/ssl/certs from what I see on ze01 | 15:32 |
| clarkb | some commands have it and some don't? | 15:33 |
| fungi | clarkb: we've got it in trusted_ro_paths... do we also need it in untrusted_ro_paths? | 15:33 |
| clarkb | ya I'm beginning to wonder if we only applied it to a few locations and this job is hitting the other? | 15:34 |
| fungi | just looking at /etc/zuul/zuul.conf and wondering | 15:34 |
| fungi | i'll propose that and drop the /usr/share/ca-certificates since it's redundant anyway | 15:34 |
| clarkb | sounds good | 15:34 |
| clarkb | but ya I imagine that hitting the rtfd url is an untrusted playbook | 15:35 |
| clarkb | so we need this in both locations | 15:35 |
| clarkb | I've also discovered the version of bwrap I have (0.2.0) is a bit more forgiving in its command args than the version zuul has 0.1.8 | 15:36 |
| *** ccamacho has quit IRC | 15:36 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Revert "Add /usr/share/ca-certificates to trusted_ro_paths" https://review.openstack.org/514358 | 15:37 |
| *** gyee has joined #openstack-infra | 15:37 | |
| fungi | so there's the revert ^ | 15:38 |
| *** jpena is now known as jpena|brb | 15:40 | |
| *** kjackal_ has quit IRC | 15:42 | |
| fungi | and that's the untrusted addition ^ | 15:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Also add /etc/ssl/certs to untrusted_ro_paths https://review.openstack.org/514362 | 15:44 |
| fungi | er, ^ | 15:44 |
| *** dhinesh has joined #openstack-infra | 15:45 | |
| *** andreas_s has quit IRC | 15:46 | |
| *** andreas_s has joined #openstack-infra | 15:46 | |
| *** wolverineav has quit IRC | 15:51 | |
| *** dhajare has quit IRC | 15:51 | |
| openstackgerrit | Merged openstack-infra/project-config master: always make the requirements repo available for release jobs https://review.openstack.org/514348 | 15:52 |
| *** Guest52014 has quit IRC | 15:53 | |
| *** andreas_s has quit IRC | 15:53 | |
| *** andreas_s has joined #openstack-infra | 15:53 | |
| *** sree has joined #openstack-infra | 15:53 | |
| *** jkilpatr has quit IRC | 15:55 | |
| smcginnis | Is there anyone that can re-queue jobs for ironic-lib now that that has landed? ^ | 15:57 |
| smcginnis | Details currently on line 88 of https://etherpad.openstack.org/p/zuulv3-issues | 15:57 |
| *** sree has quit IRC | 15:57 | |
| *** udesale has joined #openstack-infra | 15:58 | |
| fungi | smcginnis: taking a look | 16:00 |
| smcginnis | fungi: Thanks | 16:00 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Use playbook from tripleo-quickstart-extras for OVB https://review.openstack.org/513508 | 16:00 |
| *** ykarel|afk is now known as ykarel | 16:03 | |
| *** andreas_s has quit IRC | 16:03 | |
| *** jamesmcarthur has quit IRC | 16:04 | |
| *** andreas_s has joined #openstack-infra | 16:04 | |
| *** jamesmcarthur has joined #openstack-infra | 16:07 | |
| fungi | smcginnis: enqueued | 16:07 |
| *** kiennt26 has quit IRC | 16:12 | |
| *** armax has joined #openstack-infra | 16:13 | |
| *** andreas_s has quit IRC | 16:13 | |
| jeblair | infra-root: pep8 (er, pycodestyle?) just released a new version which adds E741: do not use variables named 'l', 'O', or 'I'. | 16:14 |
| jeblair | i propose we add that one to our standard ignore list | 16:14 |
| fungi | wha? | 16:14 |
| fungi | yes, kiwf | 16:14 |
| *** lucasagomes is now known as lucas-afk | 16:14 | |
| Shrews | pfft, like anyone would name variables 'l', 'O', or 'I' | 16:14 |
| Shrews | oh wait | 16:14 |
| jeblair | fungi: i believe it's to accomodate people who don't like fonts which can differentiate those characters | 16:15 |
| jeblair | Shrews: right? | 16:15 |
| fungi | jeblair: yeah, i get that... however you could just as well add a slew of other other unicode codepoints which are hard to distinguish from integers | 16:15 |
| * fungi also wonders who actually uses fonts which make 1/l/I and 0/O hard to tell apart | 16:16 | |
| jeblair | i would have assumed they would have put 'j' on the list too... i mean, i still confuse it with 'i' all the time. | 16:17 |
| jeblair | ^ (typographic history joke) | 16:17 |
| fungi | indeed | 16:17 |
| *** andreas_s has joined #openstack-infra | 16:18 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration test coverage for iptables persistence https://review.openstack.org/513934 | 16:18 |
| openstackgerrit | gordon chung proposed openstack-infra/openstack-zuul-jobs master: remove legacy panko jobs https://review.openstack.org/512430 | 16:18 |
| openstackgerrit | gordon chung proposed openstack-infra/openstack-zuul-jobs master: remove legacy panko jobs https://review.openstack.org/512430 | 16:18 |
| *** jpena|brb is now known as jpena | 16:19 | |
| *** ihrachys has quit IRC | 16:19 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Fix for pep8 E722 and ignore E741 https://review.openstack.org/514372 | 16:19 |
| *** ihrachys has joined #openstack-infra | 16:20 | |
| fungi | i remember not long ago getting a nasty surprise when someone had encoded a name with a \u0421 in one place but a \u0043 in another, and my font at the time did not distinguish them visually | 16:21 |
| fungi | or might have been \u0441 vs \u0099 | 16:22 |
| *** andreas_s has quit IRC | 16:22 | |
| dmsimard | does any project actually *need* openstack/openstack ? the first time I ever saw it was post migration in required-projects and it's all over the place | 16:23 |
| fungi | s/99/63/ | 16:23 |
| dmsimard | http://git.openstack.org/cgit/openstack/openstack | 16:23 |
| fungi | dmsimard: no project should be adding openstack/openstack to its required-projects. that's just a (incomplete) meta-project we used at one time for timeline analysis | 16:24 |
| *** apevec has joined #openstack-infra | 16:24 | |
| dmsimard | fungi: ok so we should remove it everywhere then ? | 16:24 |
| *** Goneri has quit IRC | 16:24 | |
| fungi | as far as i know, yes. i can't imagine what purpose it would serve in _any_ ci job ever | 16:24 |
| jeblair | dmsimard: where do you see it used? | 16:25 |
| *** yamahata has quit IRC | 16:25 | |
| * clarkb is happy his font differentiates those characters | 16:27 | |
| *** ykarel_ has joined #openstack-infra | 16:28 | |
| *** tosky has joined #openstack-infra | 16:28 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Remove 'openstack/openstack' from lists of required-projects https://review.openstack.org/514378 | 16:28 |
| dmsimard | fungi, jeblair ^ | 16:29 |
| dmsimard | It was probably a mistake in the migration script or something. | 16:29 |
| *** jkilpatr has joined #openstack-infra | 16:32 | |
| *** vtapia has quit IRC | 16:32 | |
| fungi | clarkb: which characters? 1/l/I and 0/O or C/С and c/с | 16:33 |
| clarkb | fungi: 1/l/I and 0/O | 16:33 |
| clarkb | I'm using source code pro from adobe (one of their open source fonts) | 16:34 |
| * fungi is glad his current font actually distinguishes _all_ of those | 16:34 | |
| *** jaosorior has quit IRC | 16:35 | |
| * fungi is using the built-in commodore 64 font in cool-retro-term | 16:36 | |
| *** jaosorior has joined #openstack-infra | 16:36 | |
| *** kien-ha has joined #openstack-infra | 16:39 | |
| *** kien_ has joined #openstack-infra | 16:39 | |
| *** ykarel_ has quit IRC | 16:39 | |
| *** ykarel has quit IRC | 16:39 | |
| *** Goneri has joined #openstack-infra | 16:41 | |
| *** vtapia has joined #openstack-infra | 16:41 | |
| *** Guest57916 has quit IRC | 16:42 | |
| *** derekh has quit IRC | 16:45 | |
| *** wolverineav has joined #openstack-infra | 16:47 | |
| clarkb | fungi: what is С used for? | 16:47 |
| tosky | jeblair: hi again! About my earlier question on shade usage with commands like os_image (https://review.openstack.org/#/c/512058/) | 16:47 |
| clarkb | and should I change my nick to сlarkb ? | 16:48 |
| *** martinkopec_ has quit IRC | 16:49 | |
| *** andreas_s has joined #openstack-infra | 16:49 | |
| jeblair | tosky: i was just looking at that... | 16:49 |
| *** trown is now known as trown|lunch | 16:49 | |
| jeblair | tosky: it looks like you want to use the openstack ansible modules in the ansible that zuul itself runs | 16:50 |
| jeblair | tosky: which is neat, but i don't think it's something we've explored yet | 16:50 |
| fungi | clarkb: that's the capital cyrillic "es" | 16:50 |
| fungi | which just happens to be written identically to the capital latin "see" | 16:50 |
| jeblair | tosky: so i think the error you are seeing means, basically, that the shade library is not installed in the zuul executor environment where the job is running | 16:51 |
| *** wolverineav has quit IRC | 16:51 | |
| *** rossella_s has quit IRC | 16:51 | |
| clarkb | fungi: oh this is the fun dns ssl issue | 16:51 |
| dhellmann | can anyone help me figure out why a release-openstack-python job that reports an error in the logs didn't trigger an email to the release-job-failures mailing list? http://logs.openstack.org/59/597020817c6021349abe4708171c8f179772ecb6/release/release-openstack-python/9d2d44c/job-output.txt.gz#_2017-10-23_14_57_16_602440 | 16:51 |
| tosky | jeblair: but then what is the magic incantation? In the last execution I added it to required-projects, but that's on the node, right? | 16:52 |
| *** dhinesh has quit IRC | 16:52 | |
| fungi | dhellmann: good question, looks like it _should_ have done: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/pipelines.yaml#n190 | 16:53 |
| *** udesale has quit IRC | 16:53 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul feature/zuulv3: Only autohold failed builds https://review.openstack.org/513850 | 16:53 |
| dhellmann | fungi : we got email for some other failures | 16:54 |
| dhellmann | http://lists.openstack.org/pipermail/release-job-failures/2017-October/thread.html | 16:54 |
| *** andreas_s has quit IRC | 16:54 | |
| dhellmann | although those were last week | 16:54 |
| smcginnis | I think we've had some other ones we should have seen after that one on Friday. | 16:54 |
| fungi | dhellmann: i'll check the exim logs for zuulv3.o.o and lists.o.o | 16:55 |
| dhellmann | ty | 16:55 |
| jeblair | tosky: oh, i was mistaken, you are running this on the controller node, that makes perfect sense. so that error means shade is not installed on the controller node | 16:55 |
| fungi | hrm... the exim logs on zuulv3 don't seem to indicate any attempt to send messages to lists.o.o | 16:56 |
| jeblair | tosky: i think what's missing is something to install shade. just adding it to required-projects isn't enough -- that only causes the git repo to be on the node. | 16:56 |
| *** ralonsoh has quit IRC | 16:57 | |
| tosky | jeblair: yep, that's what I thought - am I the first one that is trying to run some os_<something> module? | 16:57 |
| *** apevec has quit IRC | 16:57 | |
| jeblair | tosky: probably | 16:57 |
| dhellmann | fungi : looking back at the rest of the logs for http://logs.openstack.org/59/597020817c6021349abe4708171c8f179772ecb6/release/release-openstack-python/9d2d44c/job-output.txt.gz#_2017-10-23_14_57_16_602440 | 16:57 |
| tosky | oh joy | 16:57 |
| jeblair | tosky: i think we're missing the shade devstack module | 16:57 |
| jeblair | er plugin | 16:57 |
| dhellmann | fungi : it doesn't appear that the job thinks it failed. It tried to run twine to upload something. | 16:57 |
| jeblair | tosky: try adding shade to the 'devstack_plugins' var | 16:58 |
| tosky | jeblair: oh, that should be easy, let me try | 16:58 |
| jeblair | tosky: i think that will install shade on the controller, and then you should be able to use it | 16:58 |
| jeblair | tosky: and i'm going to ping Shrews here who may also be familiar with some of this | 16:58 |
| tosky | I didn't think that shade had a devstack plugin | 16:58 |
| tosky | but I should have checked | 16:58 |
| smcginnis | fungi, dhellmann: Shouldn't there be something in those logs mentioning emailing? Or is that a task that happens outside of the job run itself? | 16:58 |
| jeblair | tosky: i believe it's in the shade repo | 16:58 |
| dhellmann | smcginnis, fungi: probably? it shouldn't have run the twine step, either, I think? | 16:58 |
| jeblair | tosky: so "shade: shade git://git.openstack.org/openstack-infra/shade | 16:59 |
| jeblair | blah | 16:59 |
| jeblair | tosky: so "shade: 'git://git.openstack.org/openstack-infra/shade'" | 16:59 |
| tosky | jeblair: with shade being branchless, isn't there some risk that it will install higher dependencies (as it can happen with tempest)? Or will it work in a separate virtualenv? | 16:59 |
| fungi | smcginnis: dhellmann: the job definitely failed, but the artifact uploader and the pypi uploader are happening as "post" phase tasks and seem to be set to attempt even if the run phase tasks fail | 17:00 |
| dhellmann | fungi : that seems like a bad idea? what if we build 1/2 of a tarball? | 17:00 |
| dhellmann | maybe there's a reason it's ok that I don't get, though | 17:01 |
| fungi | dhellmann: i am inclined to agree this seems like an oversight in the job definition | 17:01 |
| pabelanger | likey want to use when: zuul_success | 17:01 |
| *** sree has joined #openstack-infra | 17:01 | |
| fungi | as for the e-mail reporter, that's a built-in feature of zuul (similar to the gerrit comment reporting) so won't show up in the logs as it's not implemented in ansible | 17:01 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Do not rely on openstack-puppet-modules in Queens and beyond https://review.openstack.org/512783 | 17:02 |
| pabelanger | maybe have block task in post playbooks, using zuul_success | 17:02 |
| fungi | so i'll need to go spelunking in the scheduler's debug logs for a reason e-mails aren't going out | 17:02 |
| dhellmann | fungi : can a job have more than one playbook listed in the "run" section? can I just move that post playbook to the run list? | 17:03 |
| dhellmann | or do we need to move the tasks into a role that can be included in the existing run playbook? | 17:03 |
| *** sree has quit IRC | 17:03 | |
| fungi | dhellmann: not sure whether there might be a reason artifact and pypi publishing are done in post tasks, given they happen on the executor rather than a build node | 17:04 |
| *** sree has joined #openstack-infra | 17:04 | |
| dhellmann | I'm not sure what the difference is between an executor and a build node. Is the executor where zuul is running? | 17:04 |
| *** jcoufal has quit IRC | 17:05 | |
| pabelanger | we can switch http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/publish/pypi.yaml to use include_role, in block with when: zuul_success. that would then only call roles if run playbook passed | 17:05 |
| fungi | dhellmann: dedicated servers running the zuul-executor daemons, they're basically what runs ansible and then ansible connects over the network to the build nodes where job payloads are (usually) executed | 17:06 |
| openstackgerrit | Merged openstack-infra/system-config master: Revert "Add /usr/share/ca-certificates to trusted_ro_paths" https://review.openstack.org/514358 | 17:06 |
| openstackgerrit | Merged openstack-infra/system-config master: Also add /etc/ssl/certs to untrusted_ro_paths https://review.openstack.org/514362 | 17:06 |
| dhellmann | fungi : so the post-run playbooks run on the executor? | 17:07 |
| *** ldnunes has quit IRC | 17:07 | |
| fungi | dhellmann: by keeping some playbooks which need sensitive credentials but are only going to use them to hit some remote service (e.g., uploading things to places or triggering a remote service api) it's an added layer of safety to not ship those secrets to build nodes but just use them locally (within a very stripped-down bubblewrap namespace/chroot) on the executors themselves | 17:07 |
| *** sree has quit IRC | 17:08 | |
| pabelanger | dhellmann: in this case yes, but you can tell where the playbook runs by looking at hosts: of a playbook: http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/publish/pypi.yaml | 17:08 |
| pabelanger | in that case, localhost == zuul-executor | 17:08 |
| *** yamahata has joined #openstack-infra | 17:09 | |
| jeblair | dhellmann: any of the three phases (pre/run/post) can run on any of the hosts. but in this case, this particular post playbook performs tasks on the executor. | 17:09 |
| dhellmann | ok | 17:09 |
| jeblair | dhellmann: (playbooks aren't limited to one host either -- their real power, is, in fact multi-host orchestration) | 17:09 |
| dhellmann | and I guess we need it to run there because of the pypi credentials? | 17:09 |
| *** salv-orlando has quit IRC | 17:09 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/zuul-jobs master: add more debugging to the upload-pypi role https://review.openstack.org/514394 | 17:10 |
| jeblair | dhellmann: yep. it runs on the executor for safety. | 17:10 |
| *** efried has joined #openstack-infra | 17:10 | |
| pabelanger | yah, we don't trust remote nodes for publishing to pypi, so we pull everything back to executor and run twine there | 17:10 |
| *** salv-orlando has joined #openstack-infra | 17:10 | |
| fungi | we could certainly provide the pypi credentials to remote build nodes, but since we can get away with not doing that we don't | 17:10 |
| dhellmann | jeblair : how do I discover the variables available in a job? I want to add debugging ^^ but I don't know what the data structure looks like. | 17:11 |
| dhellmann | I think running it on a different node is fine, and adding messages indicating that even though it looks like it's going to upload something it *wont'* will be enough | 17:11 |
| *** slaweq has quit IRC | 17:11 | |
| fungi | dhellmann: the entire structure saved in inventory.yaml in the logs we publish should be available to your playbooks as well | 17:11 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/zuul-jobs master: add more debugging to the upload-pypi role https://review.openstack.org/514394 | 17:11 |
| jeblair | dhellmann: well, we should also do pabelanger's suggestion of not attempting on failure | 17:12 |
| *** slaweq has joined #openstack-infra | 17:12 | |
| *** andreas_s has joined #openstack-infra | 17:12 | |
| dhellmann | jeblair : that would be good, too, sure. is there some sort of way for the task to check that? | 17:12 |
| fungi | i.e. http://logs.openstack.org/59/597020817c6021349abe4708171c8f179772ecb6/release/release-openstack-python/9d2d44c/zuul-info/inventory.yaml | 17:12 |
| pabelanger | dhellmann: i don't think that debug is needed, for upload-pypi. I seem to recall we list found files. Looking for example | 17:13 |
| dhellmann | fungi : I specifically meant for printing file names in https://review.openstack.org/#/c/514394/ | 17:13 |
| jeblair | dhellmann: variables can come from a number of places; the ones from zuul itself are documented and will appear in the inventory file. ones that come from other roles, etc, *should* be documented in those roles. but may not be. yet. | 17:13 |
| *** jpich has quit IRC | 17:13 | |
| efried | Hey kids, I noticed that a spec [1] merged but didn't get published [2]. [1] https://review.openstack.org/#/c/510244/ [2] https://specs.openstack.org/openstack/nova-specs/specs/queens/ | 17:13 |
| dhellmann | pabelanger : ok, I was hoping for a more obvious "we didn't find anything to upload" indicator | 17:13 |
| efried | It may be that all specs that merged after the zuulv3 cutover suffered the same fate - I haven't checked. | 17:13 |
| dhellmann | jeblair : ok | 17:14 |
| efried | mriedem says he thinks there's a publish job that's either busted or not running. | 17:14 |
| pabelanger | dhellmann: that will just return OK, if nothing found. It should be changed, if something is found | 17:14 |
| pabelanger | confirming now | 17:14 |
| *** electrofelix has quit IRC | 17:14 | |
| fungi | efried: mriedem: specs publication was broken for a while but should have been solved recently, i thought | 17:14 |
| dhellmann | jeblair : I guess in this case I'm looking for what "find" reports, and maybe that's an ansible built-in | 17:14 |
| efried | fungi Cool - anything I can do to redrive that job? | 17:14 |
| *** salv-orlando has quit IRC | 17:14 | |
| dhellmann | jeblair : ok, found it at http://docs.ansible.com/ansible/latest/find_module.html | 17:15 |
| jeblair | dhellmann: ah, yeah, here are the docs for that: http://docs.ansible.com/ansible/latest/find_module.html#return-values | 17:15 |
| fungi | efried: looks like that one merged on 10/18 which i think is after we fixed publication. have there been any nova-specs changes to merge more recently than that? the site contents should be rebuilt in their entirety by each new change which lands | 17:16 |
| *** jpena is now known as jpena|off | 17:16 | |
| *** katyafervent_awa has quit IRC | 17:16 | |
| efried | fungi Looking... | 17:16 |
| jeblair | dhellmann: which also references http://docs.ansible.com/ansible/latest/stat_module.html#return-values for individual file entries | 17:16 |
| dhellmann | how do I test for an empty list in ansible? "when found_wheels == []"? or "when not found_wheels"? | 17:16 |
| *** slaweq has quit IRC | 17:16 | |
| *** andreas_s has quit IRC | 17:16 | |
| fungi | efried: it's entirely possible that one was caught in a zuul scheduler restart last week and the post pipeline jobs for it were simply lost | 17:16 |
| *** rakhmerov has quit IRC | 17:17 | |
| fungi | we had a bunch of emergency restarts to deal with memory consumption | 17:17 |
| *** dprince has quit IRC | 17:17 | |
| smcginnis | efried: Maybe you or mriedem can push through a trivial update to the nova-specs repo to trigger a publish again. | 17:17 |
| fungi | which seem to be thankfully under control nopw | 17:17 |
| jeblair | dhellmann: "when:" is a jinja2 expression, so i think either would work. | 17:17 |
| dhellmann | ok | 17:17 |
| *** asilenkov has quit IRC | 17:17 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/zuul-jobs master: add more debugging to the upload-pypi role https://review.openstack.org/514394 | 17:18 |
| efried | smcginnis fungi Will do. I've found at least one other that didn't get published. I'll finish compiling that list... | 17:18 |
| fungi | efried: mriedem: worst case, i can reenqueue post jobs for teh most recent merge to that repo if you don't have anything ready to approve now | 17:18 |
| *** tesseract has quit IRC | 17:18 | |
| mriedem | we have something to approve, sec | 17:18 |
| *** ldnunes has joined #openstack-infra | 17:19 | |
| fungi | efried: mriedem: looks like there was a more recent merge than that one and the site was built and uploaded: http://logs.openstack.org/fd/fd0986ccc2391cc21d3ca0fc8ded24913bda5874/post/publish-openstack-specs/01f213f/job-output.txt.gz | 17:20 |
| fungi | i wonder if we uploaded it to the wrong path or something there | 17:21 |
| *** kjackal_ has joined #openstack-infra | 17:21 | |
| efried | fungi I see the specs in question in that build log... | 17:21 |
| *** rakhmerov has joined #openstack-infra | 17:21 | |
| efried | fungi http://logs.openstack.org/fd/fd0986ccc2391cc21d3ca0fc8ded24913bda5874/post/publish-openstack-specs/01f213f/job-output.txt.gz#_2017-10-19_23_31_38_102885 | 17:21 |
| pabelanger | dhellmann: http://logs.openstack.org/59/597020817c6021349abe4708171c8f179772ecb6/release/release-openstack-python/9d2d44c/job-output.json.gz | 17:22 |
| efried | fungi I don't really know what I'm looking at there, but I don't see anything that uploaded those files anywhere. | 17:22 |
| pabelanger | dhellmann: if you look at Find tarballs and wheels in dist folder task | 17:22 |
| pabelanger | you'll see files: [] | 17:22 |
| smcginnis | efried: Looks like here, but no data: http://logs.openstack.org/fd/fd0986ccc2391cc21d3ca0fc8ded24913bda5874/post/publish-openstack-specs/01f213f/job-output.txt.gz#_2017-10-19_23_31_43_294389 | 17:22 |
| pabelanger | which means, we failed to find anything | 17:22 |
| smcginnis | Or rather, no output of the details. | 17:22 |
| pabelanger | dhellmann: same with 'Find wheels to upload' task | 17:23 |
| dhellmann | pabelanger : that's not really obvious from the human-consumable logs, though: http://logs.openstack.org/59/597020817c6021349abe4708171c8f179772ecb6/release/release-openstack-python/9d2d44c/job-output.txt.gz#_2017-10-23_14_57_34_467229 | 17:23 |
| *** rbrndt has quit IRC | 17:23 | |
| *** sambetts is now known as sambetts|afk | 17:23 | |
| dhellmann | pabelanger : is there some way for me to get that message out of the find task results? | 17:23 |
| *** katyafervent_awa has joined #openstack-infra | 17:23 | |
| pabelanger | dhellmann: Right, it would return changed, if it found something. | 17:23 |
| *** sambetts has joined #openstack-infra | 17:24 | |
| pabelanger | localhost: changed (eg) | 17:24 |
| *** asilenkov has joined #openstack-infra | 17:24 | |
| efried | fungi https://review.openstack.org/#/q/project:openstack/nova-specs+status:merged newest one that got published was https://review.openstack.org/509013 (Return Alternate Hosts) | 17:24 |
| pabelanger | dhellmann: sorry, which message | 17:24 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/zuul-jobs master: add more debugging to the upload-pypi role https://review.openstack.org/514394 | 17:25 |
| efried | fungi But this one merged earlier and also didn't make it: https://review.openstack.org/#/c/361140/ | 17:25 |
| dhellmann | pabelanger : I think ^^ will do it? I want to actually have the log say "The next step is not going to do anything" so that I don't have to keep an ansible interpreter in my head when I'm debugging. | 17:26 |
| fungi | efried: yeah, we're either not actually uploading these or they're getting uploaded to the wrong place from what i can tell. the footer for the nova-specs site says " Last updated on Fri Oct 13 16:04:21 2017, commit 3a5d54a" | 17:26 |
| fungi | the "Upload specs to fileserver" task claims to have succeeded, but "Output suppressed because no_log was given" (i guess we're trying not to spam the logs with the list of files uploaded to the remote site) | 17:27 |
| fungi | no, wait, that's for logs.openstack.org | 17:28 |
| fungi | i don't see one for static.openstack.org | 17:28 |
| smcginnis | Oh right, missed that to. | 17:28 |
| smcginnis | o | 17:28 |
| fungi | or specs.openstack.org | 17:28 |
| efried | fungi Here's an interesting note: https://review.openstack.org/#/c/504275/ merged on 10/3, but there's another zuul build entry on 10/18 | 17:29 |
| *** hashar is now known as hasharDinner | 17:29 | |
| *** jamesmcarthur has quit IRC | 17:29 | |
| efried | fungi smcginnis: dansmith just pushed https://review.openstack.org/#/c/456756/ -- let's see what happens there... | 17:29 |
| fungi | efried: well, only the most recent build is of any particular interest because the entire site is rebuilt and uploaded in its entirety on every change which merges | 17:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Upload upload to pypi if pre / run was successful https://review.openstack.org/514398 | 17:30 |
| efried | mm | 17:30 |
| pabelanger | dhellmann: jeblair: fungi: lets start with, ^ that will stop upload-pypi from happening on failed pre / run playbooks | 17:30 |
| *** andreas_s has joined #openstack-infra | 17:31 | |
| fungi | rather the entire subtree for https://specs.openstack.org/openstack/nova-specs anyway | 17:31 |
| dhellmann | pabelanger : is that going to check the pre/run outcome or the ensure-twine role outcome? | 17:31 |
| fungi | efried: mriedem: i think i found it, looks like we have the destination path incorrectly specified: http://specs.openstack.org/nova-specs/specs/queens/approved/granular-resource-requests.html | 17:31 |
| dhellmann | I don't know how zuul_success is scoped | 17:31 |
| efried | fungi Nyaha! | 17:31 |
| pabelanger | dhellmann: pre / run playbooks only | 17:32 |
| fungi | efried: mriedem: should probably be using project.name instead of project.short_name somewhere in the publisher | 17:32 |
| fungi | i'll see if i can find it | 17:32 |
| pabelanger | we pass zuul_status into ansible-playbook, each time we call it on executor | 17:32 |
| dhellmann | pabelanger : ok. I would still like to know when no files are found, because that may indicate that the pre/run steps passed but did the wrong thing. | 17:32 |
| jeblair | dhellmann, pabelanger: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/jobs.html#var-zuul_success though it actually needs a doc update to indicate it's affected by 'pre' as well. | 17:32 |
| jeblair | pabelanger: why do we need the boolean filter? | 17:34 |
| *** jamesmcarthur has joined #openstack-infra | 17:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Upload upload to pypi if pre / run was successful https://review.openstack.org/514398 | 17:34 |
| jeblair | (i realize the docs say we do, i just don't know why) | 17:34 |
| efried | fungi Thanks for tracking that down. Lemme know when I can +1 something :) | 17:34 |
| fungi | efried: mriedem: looks like this may have already been fixed since the last one merged so you should be good: https://review.openstack.org/513493 | 17:34 |
| pabelanger | jeblair: not sure, safely? I actually haven't tried zuul_success yet | 17:34 |
| pabelanger | jeblair: happy to drop it | 17:34 |
| jeblair | pabelanger: don't drop it, the docs say we need it. i just want to know why. :) | 17:35 |
| efried | fungi Noyce, thanks again. | 17:35 |
| *** andreas_s has quit IRC | 17:35 | |
| *** sree has joined #openstack-infra | 17:35 | |
| pabelanger | jeblair: yah, I am not sure myself. Will look into it more | 17:35 |
| jeblair | pabelanger: we don't need to install twine on failure | 17:35 |
| dhellmann | does the filter handle the case where the variable isn't defined? | 17:35 |
| jeblair | pabelanger: maybe a block? | 17:35 |
| pabelanger | sure | 17:36 |
| jeblair | dhellmann: ah maybe... maybe it bombs if it's not defined | 17:36 |
| pabelanger | yah, that might be it | 17:36 |
| jeblair | dhellmann: that makes sense. because we only supply it on post playbooks. | 17:36 |
| *** amoralej is now known as amoralej|off | 17:36 | |
| *** ijw has joined #openstack-infra | 17:36 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove 'openstack/openstack' from lists of required-projects https://review.openstack.org/514378 | 17:37 |
| AJaeger | efried, fungi, specs publishing was fixed on the 20th | 17:37 |
| *** zoli is now known as zoli|gone | 17:37 | |
| efried | AJaeger Thanks. We're pushing a new change, so hopefully everything will publish correctly once that merges. | 17:37 |
| *** dhinesh has joined #openstack-infra | 17:37 | |
| *** zoli|gone is now known as zoli | 17:37 | |
| AJaeger | fungi, efried http://specs.openstack.org/nova-specs/ is the wrong location - any infra-root to delete? | 17:38 |
| *** dbecker has quit IRC | 17:38 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 17:39 |
| * AJaeger finished reading backscroll | 17:39 | |
| *** sree has quit IRC | 17:39 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Upload upload to pypi if pre / run was successful https://review.openstack.org/514398 | 17:39 |
| pabelanger | k, think that is the correct syntax | 17:39 |
| *** dprince has joined #openstack-infra | 17:39 | |
| Shrews | jeblair: tosky: yes, shade does have a devstack plugin, but shade itself doesn't actually use it. but something like "enable_plugin shade git://git.openstack.org/openstack-infra/shade | 17:39 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: do not run pep8 for deliverable file changes https://review.openstack.org/514401 | 17:40 |
| Shrews | " in your local.conf | 17:40 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Upload to pypi if pre / run was successful https://review.openstack.org/514398 | 17:40 |
| efried | fungi That just merged. Assume the publish job comes along after? | 17:41 |
| fungi | yup, in the post pipeline | 17:41 |
| efried | cool, will watch. | 17:41 |
| *** dhinesh has quit IRC | 17:42 | |
| fungi | smcginnis: dhellmann: jeblair: i suspect the smtp reporter is never being reached due to a prior exception raised in the sql reporter: http://paste.openstack.org/show/624397 | 17:42 |
| dhellmann | fungi : that's in zuul itself? | 17:42 |
| jeblair | pabelanger: apparently you can apply 'when:' directly to a role. what do you think about going back to the roles list and adding when to both entries? | 17:42 |
| fungi | dhellmann: correct, from zuul's scheduler log | 17:42 |
| jeblair | fungi: yes that would do it | 17:43 |
| dhellmann | fungi : ok. I don't necessarily think that's a blocker for finishing the milestone releases. What do you think smcginnis ? | 17:43 |
| smcginnis | fungi: Well, at least that would be a good explanation. | 17:43 |
| smcginnis | dhellmann: No, not a blocker. | 17:43 |
| smcginnis | But probably still a hihger priority item long term. | 17:43 |
| dhellmann | oh, yeah, definitely | 17:44 |
| pabelanger | jeblair: yah, using block means 1 when condition, over two. Which do you prefer? | 17:44 |
| fungi | jeblair: looks like i may have truncated the end of that second traceback but seems like some unexpected situation because of tags not having a branch | 17:44 |
| dhellmann | fungi : I added a link to that pastebin in the issue report on line 84 of the etherpad | 17:44 |
| fungi | thanks dhellmann i'll mark myself as digging into it | 17:44 |
| dhellmann | smcginnis : shall we try another? | 17:45 |
| smcginnis | dhellmann: Yeah, sounds good. Should we try any different types? | 17:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Upload to pypi if pre / run was successful https://review.openstack.org/514398 | 17:45 |
| * smcginnis pulls up the dashboard to see what's out there... | 17:45 | |
| pabelanger | jeblair: ^ | 17:46 |
| dhellmann | smcginnis : I can't decide whether it's better to do that, or release the things we think will work then move on to the next type. | 17:46 |
| smcginnis | dhellmann: Good point. | 17:46 |
| dhellmann | smcginnis : I'm still not 100% confident of any of the types :-/ | 17:46 |
| smcginnis | Let's get through what we know will work first I guess. | 17:46 |
| jeblair | pabelanger: i was suggesting going back to roles | 17:46 |
| *** vhosakot has joined #openstack-infra | 17:47 | |
| dhellmann | smcginnis : ack | 17:47 |
| smcginnis | s/know will work/have higher confidence will work/ | 17:47 |
| jeblair | pabelanger: http://paste.openstack.org/show/624398/ | 17:47 |
| pabelanger | jeblair: Oh, really? I didn't know when could be use like that | 17:47 |
| dhellmann | smcginnis : I'll approve the sushy release next: https://review.openstack.org/#/c/512586/1 | 17:47 |
| smcginnis | dhellmann: ++ | 17:47 |
| pabelanger | jeblair: interesting, sure lets try | 17:47 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration test coverage for iptables persistence https://review.openstack.org/513934 | 17:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Upload to pypi if pre / run was successful https://review.openstack.org/514398 | 17:48 |
| dhellmann | smcginnis : I wonder why we're still seeing py35 jobs in the gate when we only have deliverable files in the change? | 17:48 |
| smcginnis | dhellmann: Didn't we block that patch until things settled? | 17:49 |
| smcginnis | Oh, nope. Looks like it went through. | 17:49 |
| dhellmann | smcginnis : it's in my master branch of project-config locally? | 17:49 |
| smcginnis | Hmm | 17:49 |
| dhellmann | I wonder if the regex doesn't match | 17:49 |
| *** ijw has quit IRC | 17:50 | |
| *** dhinesh has joined #openstack-infra | 17:50 | |
| dhellmann | it looks like some other regexes do start with directory names, so that shouldn't be it | 17:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix for pep8 E722 and ignore E741 https://review.openstack.org/514372 | 17:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix undefined sort order when applying parent data https://review.openstack.org/514056 | 17:51 |
| *** jamesmcarthur has quit IRC | 17:51 | |
| AJaeger | project-config-core, requirements post and periodic jobs are broken, could I get a +2A on https://review.openstack.org/#/c/514161/ , please | 17:51 |
| jeblair | AJaeger: why did you move periodic to after post? | 17:52 |
| dmsimard | AJaeger: looking | 17:52 |
| AJaeger | jeblair: cosmetic since that was the order in the past. | 17:53 |
| *** jamesmcarthur has joined #openstack-infra | 17:54 | |
| AJaeger | and is the order for other projects | 17:54 |
| *** camunoz has quit IRC | 17:54 | |
| jeblair | i need to afk for a bit | 17:55 |
| *** trown|lunch is now known as trown | 17:55 | |
| AJaeger | speaking of periodic jobs, please review https://review.openstack.org/514135 and https://review.openstack.org/514127 as well - those jobs run more often than needed currently | 17:56 |
| *** notemerson has joined #openstack-infra | 17:56 | |
| AJaeger | thanks, jeblair and dmsimard | 17:58 |
| dhellmann | AJaeger , mnaser : we just noticed that the regex we have for excluding py35 (which I copied for pep8) isn't actually causing the jobs to be skipped. Do either of you have an idea of why that might be? https://review.openstack.org/#/c/514401/ | 17:58 |
| mnaser | dhellmann do you have a sample job that ran them? | 17:59 |
| dhellmann | mnaser : https://review.openstack.org/512586 ran py35 | 17:59 |
| dhellmann | I guess we already don't have pep8 in the gate queue | 17:59 |
| dhellmann | so ignoring it is superfluous | 17:59 |
| *** andreas_s has joined #openstack-infra | 18:01 | |
| *** rbrndt has joined #openstack-infra | 18:01 | |
| AJaeger | dhellmann: you have a local .zuul.yaml and that seems to override project-config. | 18:02 |
| AJaeger | dhellmann: in project-config you exclude py35, in your .zuul.yaml not... | 18:02 |
| dhellmann | oh, so we need to put those settings in .zuul.yaml? | 18:02 |
| dhellmann | ok | 18:02 |
| AJaeger | dhellmann: WAIT! | 18:02 |
| * dhellmann freezes with his hands over the keyboard | 18:02 | |
| AJaeger | dhellmann: you not have py35 and pep8 in your .zuul.yaml at all, those should stay in project-config by policy | 18:02 |
| AJaeger | dhellmann: https://docs.openstack.org/infra/manual/zuulv3.html#what-not-to-convert | 18:03 |
| dhellmann | so we should remove those entries from the .zuul.yaml file? | 18:03 |
| AJaeger | dhellmann: let me send a patch for you to approve, ok? | 18:03 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix requirements proposal jobs https://review.openstack.org/514161 | 18:03 |
| AJaeger | dhellmann: yes | 18:03 |
| dhellmann | ok | 18:03 |
| smcginnis | AJaeger: We have openstack-python35-jobs under templates in the local .zzul.yaml. Is that part correct? | 18:03 |
| openstackgerrit | Edgar Magana proposed openstack-infra/irc-meetings master: Include a new time for UC IRC meetings https://review.openstack.org/513863 | 18:04 |
| *** ociuhandu has joined #openstack-infra | 18:04 | |
| *** dtantsur is now known as dtantsur|afk | 18:04 | |
| dhellmann | AJaeger, smcginnis : like https://review.openstack.org/514408 ? | 18:05 |
| dhellmann | smcginnis : oh, maybe those need to be added in project-config? | 18:06 |
| AJaeger | dhellmann: yes | 18:06 |
| *** e0ne has quit IRC | 18:06 | |
| *** dhinesh has quit IRC | 18:07 | |
| smcginnis | Added to project-config? Or that the patch is right? | 18:07 |
| AJaeger | dhellmann: and https://review.openstack.org/#/c/514401/1/zuul.d/projects.yaml adds those | 18:07 |
| AJaeger | so, once 514408 is merged, you're good again... | 18:08 |
| *** e0ne has joined #openstack-infra | 18:08 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: fix basic job definitions for releases repo https://review.openstack.org/514401 | 18:08 |
| smcginnis | dhellmann: I'll approve once you un-workflow-1 that. | 18:08 |
| smcginnis | AJaeger: Thanks | 18:08 |
| dhellmann | AJaeger : oh, I just updated that project-config patch. Maybe I don't need the template and the individual jobs for py35? | 18:09 |
| *** camunoz has joined #openstack-infra | 18:09 | |
| * dhellmann is still extremely confused about how job configuration works now | 18:09 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul feature/zuulv3: Only autohold failed builds https://review.openstack.org/513850 | 18:09 |
| *** andreas_s has quit IRC | 18:10 | |
| dhellmann | AJaeger : you asked me to rebase my patch on top of itself? | 18:10 |
| *** e0ne has quit IRC | 18:10 | |
| AJaeger | Now I'm confused ;( | 18:10 |
| dhellmann | I edited the patch to add the python35 template and to remove the pep8 job from the gate queue | 18:10 |
| dhellmann | do I need the template? | 18:11 |
| AJaeger | dhellmann: Ah, that change was approved and I thought you send another one ;) | 18:11 |
| dhellmann | oh, I missed that it was approved | 18:11 |
| AJaeger | dhellmann: no worries..., now +2 from me | 18:11 |
| *** baoli has quit IRC | 18:11 | |
| dhellmann | mnaser : could you put https://review.openstack.org/#/c/514401/2 back into your queue, please? | 18:12 |
| *** baoli has joined #openstack-infra | 18:12 | |
| * AJaeger signs off for today... | 18:12 | |
| *** baoli has quit IRC | 18:12 | |
| dhellmann | smcginnis : I set up https://review.openstack.org/#/c/514408/ as a depends-on for 401 instead of using WIP | 18:12 |
| mnaser | dhellmann done :) | 18:15 |
| dhellmann | mnaser : thank you! | 18:15 |
| smcginnis | dhellmann: Shouldn't we see pep8 running on 514408? | 18:16 |
| dhellmann | smcginnis : maybe not because of the depends-on? | 18:16 |
| dhellmann | although it shouldn't be running py35 either | 18:17 |
| dhellmann | I have no idea | 18:17 |
| mnaser | depends on trusted repos won't work, dhellmann smcginnis | 18:17 |
| smcginnis | Oh, I wonder if depends-on doesn't quite work the same for thi type of change. | 18:17 |
| mnaser | project-config = trusted repo | 18:17 |
| dhellmann | ok, so we need all of those patches to land before they take effect | 18:17 |
| mnaser | correct | 18:17 |
| smcginnis | mnaser: Ah, that! :) | 18:17 |
| dhellmann | it's weird that pep8 is *not* running, though. I would have though we'd see both. | 18:18 |
| *** baoli has joined #openstack-infra | 18:18 | |
| *** jcoufal has joined #openstack-infra | 18:18 | |
| mnaser | i may be wrong on terminology (trusted or config repo) but yeah, project-config depends-on is no go | 18:18 |
| mnaser | dhellmann because that patch removes that job | 18:18 |
| dhellmann | ah, sure, of course | 18:18 |
| dhellmann | this is what I get for trying to watch 2 unrelated changes at the same time | 18:19 |
| mnaser | the change of removing the job is being applied in config, but the change of adding the job that it depends on (from project-config) is not being applied .. hence none show | 18:19 |
| openstackgerrit | Merged openstack-infra/project-config master: fix puppet release jobs https://review.openstack.org/514334 | 18:21 |
| *** jcoufal_ has joined #openstack-infra | 18:21 | |
| dhellmann | smcginnis : at the top of the hour I have to stop and work on some other things today. | 18:21 |
| *** jcoufal has quit IRC | 18:23 | |
| smcginnis | dhellmann: Sounds good. I'll queue up the next one to send through after these few patches complete. | 18:24 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul feature/zuulv3: Default change and patchset to 0 in SQLReporter https://review.openstack.org/514423 | 18:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Run wheel builds only once https://review.openstack.org/514135 | 18:25 |
| smcginnis | fungi: Looks like that's the patch for the issue that caused the lack of email notification? ^ | 18:25 |
| fungi | smcginnis: that's what i believe to be the case anyway | 18:27 |
| dhellmann | fungi , smcginnis : is there a larger change to make somewhere to prevent a failure in one reporter from blocking the next? | 18:27 |
| fungi | dhellmann: that's probably a good idea as well, though will i think need someone with a deeper understanding of the reporter dispatch scheme | 18:29 |
| * dhellmann nods | 18:29 | |
| *** rwsu has quit IRC | 18:29 | |
| fungi | i'll add it to the etherpad | 18:29 |
| smcginnis | Is this job hung? http://zuulv3.openstack.org/static/stream.html?uuid=e78b4ed0ef904c7bbef6d9daa73112e8&logfile=console.log | 18:31 |
| *** camunoz has quit IRC | 18:31 | |
| *** camunoz has joined #openstack-infra | 18:32 | |
| *** ociuhandu has quit IRC | 18:32 | |
| *** dhinesh has joined #openstack-infra | 18:32 | |
| *** wolverineav has joined #openstack-infra | 18:32 | |
| dhellmann | smcginnis : it looks likely | 18:33 |
| *** rkukura has joined #openstack-infra | 18:33 | |
| dhellmann | smcginnis : I started a list of test scenarios at the bottom of https://etherpad.openstack.org/p/release-job-failures | 18:33 |
| fungi | or it may continue after whatever the launchpadlib timeout is for that comment call | 18:33 |
| fungi | but that timeout could very well be longer than the job timeout too | 18:34 |
| *** dbecker has joined #openstack-infra | 18:34 | |
| * fungi checks for comments on bug 1720443 | 18:34 | |
| openstack | bug 1720443 in sushy "Get throws exception when system memory is null" [High,Fix released] https://launchpad.net/bugs/1720443 - Assigned to Nate Potter (ntpttr) | 18:34 |
| openstackgerrit | Merged openstack-infra/project-config master: fix basic job definitions for releases repo https://review.openstack.org/514401 | 18:35 |
| smcginnis | So maybe a transient launchpad communication failure? | 18:35 |
| fungi | smcginnis: dhellmann: yeah, the release job hasn't successfully added any comment there that i can see | 18:35 |
| fungi | smcginnis: dhellmann: have you had any tag-releases build successfully comment on an lp bug yet? | 18:37 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: use unbuffered i/o for script adding launchpad comments https://review.openstack.org/514427 | 18:37 |
| dhellmann | fungi : no, the test jobs we were running didn't try that step :-/ | 18:38 |
| smcginnis | Maybe an auth issue? | 18:38 |
| fungi | that would be my first guess in the face of no output to tell us | 18:38 |
| fungi | and no prior working runs | 18:38 |
| dhellmann | it may be prompting for something; I think the lazr library does that if the auth doesn't work | 18:39 |
| smcginnis | Yeah, I thought it woul fail if in a remote shell, but maybe it's different in this scenario. | 18:40 |
| dhellmann | that library is such a pain, I wouldn't be surprised to learn it doesn't check for that before prompting | 18:40 |
| smcginnis | So... we probably need someting like the git config and gpg setup for launchpad config? | 18:41 |
| dhellmann | fungi : if we can set up a test node, it should be possible to run that command by hand | 18:41 |
| fungi | entirely possible. but if that's the case i have the old persistent node this used to happen on and can hold a failing node | 18:41 |
| dhellmann | smcginnis : yeah, we do have that step. it's probably not working. | 18:41 |
| fungi | i'll set an autohold for tag-releases for the openstack/releases repo | 18:42 |
| dmsimard | infra-root: the ram on zuul v3 is looking great but it looks like there's some large spikes http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all (see just before the 22nd and just now).. do we know what those spikes are ? | 18:42 |
| dhellmann | "add-launchpad-credentials" is running a couple of stepd up at 18:14:46 | 18:42 |
| fungi | autohold set | 18:43 |
| dhellmann | fungi : does that mean we need to run another job to do more investigation? | 18:43 |
| fungi | dhellmann: i think this one will get held once it times out | 18:43 |
| fungi | and if not, i can retrigger the build | 18:43 |
| jeblair | dmsimard: i'm unaware | 18:43 |
| dhellmann | fungi : ok | 18:43 |
| fungi | dhellmann: well, reenqueue the ref anyway | 18:44 |
| *** jamesmcarthur has quit IRC | 18:44 | |
| dmsimard | jeblair: it doesn't look like those spikes subsides so we'll be in dangerous territory if it happens again | 18:44 |
| jeblair | fungi: yeah, autohold is checked at the end of the build; you can add it after it's started | 18:45 |
| jeblair | dmsimard: yeah. python generally doesn't release it's memory once allocated, but it may be able to reuse it. so things could go either way. | 18:46 |
| fungi | dmsimard: the temporary spike from around 30-60 minutes ago is likely me opening scheduler debug logs... they're massive | 18:46 |
| jeblair | i'll look for leaked layouts | 18:46 |
| smcginnis | fungi: Looks like it timed out. | 18:47 |
| dmsimard | fungi: opening.. like, in vi ? | 18:47 |
| fungi | dmsimard: yes, to be able to yank tracebacks out into a separate file for pasting | 18:47 |
| dmsimard | jeblair: true, figured I'd at least point it out in case we're able to correlate that to something | 18:47 |
| *** kien_ has quit IRC | 18:47 | |
| *** kien-ha has quit IRC | 18:47 | |
| *** e0ne has joined #openstack-infra | 18:47 | |
| jeblair | dmsimard: thanks | 18:47 |
| dmsimard | fungi: yikes, I wonder if grep/awk has the same kind of impact | 18:48 |
| jeblair | according to my little debug script, there are 34 layouts in memory, 33 are reachable from the main config | 18:48 |
| clarkb | fungi: I've been using a lot of grep -B and grep -A for that | 18:49 |
| fungi | dmsimard: grep wouldn't, no but it's hard to really analyze context around the traceback that way. but yes it's what i should have been doing it's just easy to forget how massive that logfile is | 18:49 |
| jeblair | i use less | 18:49 |
| fungi | clarkb: yeah, that's also what i do when i remember. should get in the habit of doing that more often | 18:49 |
| dmsimard | clarkb: or -C too (which is an equal number for A and B) | 18:49 |
| dmsimard | like -C5 prints 5 lines before and 5 lines after | 18:50 |
| dmsimard | fungi: should we rotate those logs more often ? | 18:50 |
| *** ldnunes has quit IRC | 18:50 | |
| fungi | dmsimard: we probably can find ways to scale down some of the log volume once we get into smoother operation, but more frequent rotation probably isn't a terrible idea | 18:50 |
| *** jgriffith has quit IRC | 18:51 | |
| *** armaan has quit IRC | 18:52 | |
| *** armaan has joined #openstack-infra | 18:52 | |
| fungi | current scheduler debug log is "only" 3.2gib, which is not as bad as it was getting last week | 18:52 |
| *** e0ne has quit IRC | 18:52 | |
| jeblair | http://paste.openstack.org/show/624403/ | 18:52 |
| dmsimard | fungi: it's on rotate 30 according to puppet-zuul, I'll send a patch to cut that down to... 7 ? | 18:52 |
| *** jgriffith_ has joined #openstack-infra | 18:52 | |
| jeblair | that's the most common types in memory | 18:52 |
| *** jamesmcarthur has joined #openstack-infra | 18:52 | |
| jeblair | dmsimard: please don't | 18:52 |
| dhellmann | fungi : according to the logs after the job timed out it removed the launchpad credentials file | 18:52 |
| *** harlowja has joined #openstack-infra | 18:53 | |
| fungi | dmsimard: i don't think we need to reduce retention, just might make sense to rotate more than once a day | 18:53 |
| dmsimard | jeblair: the fact that it gets rotated just means it's compressed and rotated out, doesn't mean it gets deleted ? | 18:53 |
| jeblair | dmsimard: we have a near 100% success rate in diagnosing and fixing production errors from debug logs alone. i need logs for that. | 18:53 |
| jeblair | dmsimard: you suggested we drop the retention from 30 days to 7. | 18:54 |
| dmsimard | not my understanding, /me reads logrotate man | 18:54 |
| pabelanger | jeblair: when you have time, would be interested in how you generatd that pastebin, looks super interesting | 18:54 |
| jeblair | pabelanger: objgraph.show_most_common_types(100) | 18:54 |
| fungi | dhellmann: ahh, right, we do that for added safety. i have a feeling we'll need to neuter that and reenqueue | 18:54 |
| jeblair | pabelanger: i used the repl backdoor we're running to catch memory leaks, but we've also attached that to sigusr2 so can run it any time | 18:55 |
| pabelanger | cool, thanks for explaining | 18:55 |
| *** e0ne has joined #openstack-infra | 18:55 | |
| clarkb | dmsimard: note that logrotate isn't used to rotate the zuul logs, python logging does it | 18:55 |
| fungi | clarkb: for whatever reason, manually firing logrotate rotated the zuul logs for me on the scheduler a few weeks ago | 18:56 |
| dmsimard | jeblair: oh, you're right, I am confusing 'rotate' with the frequency of the rotation | 18:56 |
| jeblair | (also, i'm mildly opposed to increasing rotation frequency. i find the nodepool logs difficult to follow and we do it there every... 6 hours?) | 18:56 |
| clarkb | jeblair: I think it may be 8 | 18:57 |
| clarkb | but ya its pretty frequent | 18:57 |
| fungi | jeblair: yeah, i'm okay with the current size of the debug logs for the time being. just need to remember not to actually load them into memory | 18:57 |
| pabelanger | yah, I'd love to move nodepool back to 24 hours | 18:57 |
| jeblair | i end up having to do 'grep *' a lot there | 18:57 |
| jeblair | which is kind of the opposite of what i want | 18:57 |
| jeblair | nodepool is a lot quieter now, we probably could. | 18:58 |
| fungi | agreed, it does get annoying | 18:58 |
| jeblair | the ultimate goal is for 'zuul.log' (not 'debug.log') to be able to answer any question like "what happened to this change?" or "why did zuul error?" | 18:58 |
| *** jascott1 has joined #openstack-infra | 18:58 | |
| fungi | infra-root: who set the "tripleo iptables debug" autohold and are the nodes it caught still needed? | 18:58 |
| dmsimard | fungi: I don't believe so. EmilienM ^ | 18:59 |
| jeblair | fungi: i did, sorry i omitted my name | 18:59 |
| jeblair | i thought folks were going to use them over the weekend | 18:59 |
| jeblair | fungi: i think you can drop | 18:59 |
| jeblair | fungi: i'll delete held nodes | 19:00 |
| *** e0ne has quit IRC | 19:00 | |
| fungi | ah, no problem. clarkb also has at least one held for "clarkb helping emilienm debug overlay networking" | 19:00 |
| fungi | which i guess is the same/similar issue | 19:00 |
| dmsimard | pabelanger: is it normal that there are two citycloud nodepool regions with max-servers 0 ? | 19:01 |
| dmsimard | We're really capping our nodepool at ~950 VMs right now I was looking to see if we had any unused capacity | 19:02 |
| *** ldnunes has joined #openstack-infra | 19:02 | |
| clarkb | fungi: that one can be deleted actually | 19:03 |
| fungi | will do, thanks | 19:03 |
| clarkb | fungi: since I think we root caused why it didn't get held as expected | 19:03 |
| clarkb | (I'm working to fix that with 513850) | 19:03 |
| fungi | clarkb: two you held with that comment i guess. deleting both now | 19:04 |
| dmsimard | clarkb, fungi: if you have time to spare, I'm still looking for reviews to set up the ara sqlite middleware on logs-dev.o.o: https://review.openstack.org/#/q/topic:ara-sqlite-middleware | 19:04 |
| pabelanger | dmsimard: yah, both regions have been disabled as citycloud asked us to stop using them | 19:04 |
| dmsimard | clarkb, fungi: I've tested it successfully on a fake logs-dev.o.o server | 19:04 |
| pabelanger | I think both | 19:04 |
| dmsimard | pabelanger: ok, just making sure | 19:04 |
| *** arxcruz- has joined #openstack-infra | 19:05 | |
| inc0 | hey, I don't understand this error.. https://review.openstack.org/#/c/513855/ | 19:05 |
| efried | fungi Can you point me to that job? The TOC hasn't updated yet: https://specs.openstack.org/openstack/nova-specs/specs/queens/index.html Not sure how long it's supposed to take. | 19:06 |
| *** jgriffith_ has quit IRC | 19:06 | |
| jeblair | inc0: does this help explain? https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-pipeline.post-review | 19:06 |
| jeblair | inc0: basically, you can't use secrets in 'check' because people can trivially steal them. | 19:07 |
| dmsimard | pabelanger, mgagne: oops, nodepool v2 has a max-server of 150 for inap-mtl01 http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n558 and 40 in v3: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nl01.openstack.org.yaml#n47 | 19:07 |
| dmsimard | maybe something got lost between inap-mtl01 maintenance/reverse/zuul v3 roll out | 19:07 |
| inc0 | yeah that makes sense | 19:07 |
| dmsimard | pabelanger, mgagne: should we set v2 to 0 and increase v3 to 150 ? | 19:08 |
| pabelanger | clarkb: fungi: Added a few things to https://etherpad.openstack.org/p/infra-sydney-summit-project-update | 19:08 |
| SamYaple | inc0: yea unfortunately the only way to check the patch works is to merge it and wait for the post pipeline | 19:08 |
| pabelanger | dmsimard: no? Last I heard mgagne was working on cloud, not sure if ready to be used again | 19:08 |
| jeblair | SamYaple, inc0: we've talked about adding a new pipeline for situations like this. something like 'check experimental' but it would only run if there's a +2. so it's been reviewed enough to know it's not a trojan. | 19:09 |
| EmilienM | fungi: you can remove autohold. Thank you | 19:09 |
| dmsimard | pabelanger, mgagne: https://review.openstack.org/#/c/511328/ | 19:10 |
| fungi | efried: it's the nova-specs 65dabe0 commit which has been in the post pipeline for the last ~1.5 hours awaiting node assignments. we're running close to a 5 hour backlog in post because it gets lowest priority and there's a lot of gate and check activity today starving it | 19:10 |
| fungi | efried: http://zuulv3.openstack.org/ | 19:10 |
| inc0 | I'll just change password and keep this "fake" one for debug | 19:10 |
| efried | fungi Cool. Just want to be able to watch it so I don't have to keep pestering you :) | 19:11 |
| fungi | thanks EmilienM! | 19:11 |
| EmilienM | no, thanks for your help | 19:11 |
| inc0 | ahh | 19:11 |
| EmilienM | I'm still curious how the iptables issue was fixed | 19:11 |
| inc0 | ofc I can't really override this var so it doesn't change anythign | 19:11 |
| inc0 | hmm | 19:11 |
| *** slaweq has joined #openstack-infra | 19:12 | |
| * inc0 doesn't want to merge stuff just to see if it works.. | 19:12 | |
| *** Goneri has quit IRC | 19:12 | |
| *** slaweq_ has joined #openstack-infra | 19:13 | |
| SamYaple | inc0: well your patch as is doesnt work | 19:13 |
| SamYaple | inc0: i left a pretty extensive review before | 19:13 |
| inc0 | I know | 19:13 |
| SamYaple | we can get pretty close to confident before we merge | 19:14 |
| clarkb | EmilienM: aiui we are using iptables-persistent to save the rules so that restarting the "service" doesn't remove our ruleset | 19:14 |
| *** slaweq_ has quit IRC | 19:14 | |
| SamYaple | i *just* went through the same stuff, doing the exact same thing, so it should be fairly easy | 19:14 |
| pabelanger | dmsimard: guess we missed it on roll out | 19:14 |
| dmsimard | pabelanger: I pinged someone else from inap to confirm, mgagne might be on pto or something if it's someone else that sent the patch | 19:15 |
| dmsimard | I'll let you know when I hear back | 19:15 |
| * jeblair -> lunch | 19:15 | |
| pabelanger | dmsimard: yah, shouldn't be an issue to bump it back up. We are running node there now | 19:18 |
| *** jamesmcarthur has quit IRC | 19:19 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Temporarily leave launchpad creds for release jobs https://review.openstack.org/514436 | 19:19 |
| fungi | dhellmann: smcginnis: ^ | 19:19 |
| *** e0ne has joined #openstack-infra | 19:20 | |
| *** camunoz has quit IRC | 19:21 | |
| dmsimard | dhellmann: a small question before doing a +3 on https://review.openstack.org/#/c/514427/ | 19:21 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Revert "Temporarily leave launchpad creds for release jobs" https://review.openstack.org/514438 | 19:21 |
| fungi | wip ^ | 19:21 |
| dhellmann | dmsimard : we have to use python 2 for that script anyway | 19:22 |
| dmsimard | dhellmann: okay, just making sure, thanks | 19:22 |
| dhellmann | dmsimard : yep, thanks for checking that | 19:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix required repos for networking-dpm https://review.openstack.org/514191 | 19:23 |
| chandankumar | dmsimard: please have a looj on this https://review.openstack.org/#/c/514269/ the job is timing out | 19:23 |
| chandankumar | *look | 19:23 |
| dmsimard | AJaeger: huh, I didn't realize that doing a "Depends-On" on a change-id seems to require all changes of that change-id to merge (like, when doing backports across different branches) | 19:24 |
| fungi | dmsimard: yes | 19:24 |
| fungi | it's a feature | 19:24 |
| dmsimard | chandankumar: what job is timing out ? | 19:24 |
| *** e0ne has quit IRC | 19:25 | |
| fungi | dmsimard: to complicate matters further, you can perpetually block the depending changes by abandoning one of the dependencies using that shared change-id | 19:25 |
| chandankumar | dmsimard: http://logs.openstack.org/69/514269/9/check/tempest-tox-plugin-sanity-check/0636426/ -> created by my zuul v3 changes | 19:25 |
| dmsimard | Should we put RETRY_LIMIT, POST_FAILURE and TIMED_OUT in red rather than black in the gerrit status table ? Would stand out more :/ | 19:26 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Support upper-constraints in tox-siblings https://review.openstack.org/513199 | 19:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Removes migrated legacy-neutron-lbaasv2-dsvm-* https://review.openstack.org/513548 | 19:26 |
| openstackgerrit | Merged openstack-infra/shade master: Turn on voting for functional tips jobs https://review.openstack.org/514066 | 19:26 |
| dmsimard | chandankumar: what do you want me to check ? did you look at what happened ? | 19:26 |
| *** dbecker has quit IRC | 19:26 | |
| chandankumar | dmsimard: http://logs.openstack.org/69/514269/9/check/tempest-tox-plugin-sanity-check/0636426/job-output.txt.gz#_2017-10-23_17_41_35_300211 | 19:27 |
| chandankumar | dmsimard: i am not getting how to fix it? | 19:27 |
| dmsimard | chandankumar: does the job just legitimately need more time ? | 19:27 |
| chandankumar | dmsimard: yup, as it need to clone/install/uninstall more than 40 projects | 19:28 |
| dmsimard | chandankumar: you set a timeout of 40 minutes (2400 seconds) | 19:28 |
| dmsimard | if you need more than that, you can raise that value | 19:28 |
| chandankumar | dmsimard: let me raise it | 19:28 |
| dmsimard | chandankumar: try to keep the amount reasonable so that you still timeout properly if there is a genuine issue | 19:29 |
| fungi | raising the timeout can also be done as a troubleshooting step in a proposed job change without even needing to merge it | 19:29 |
| fungi | if you just want to see what will happen to the job when allowed to run to completion | 19:30 |
| *** jamesmcarthur has joined #openstack-infra | 19:32 | |
| *** e0ne has joined #openstack-infra | 19:33 | |
| openstackgerrit | gordon chung proposed openstack-infra/project-config master: drop legacy panko jobs https://review.openstack.org/512429 | 19:36 |
| openstackgerrit | Merged openstack-infra/project-config master: use unbuffered i/o for script adding launchpad comments https://review.openstack.org/514427 | 19:37 |
| *** e0ne has quit IRC | 19:38 | |
| jeblair | fungi: i thought we fixed the abandoned depends-on thing | 19:38 |
| fungi | oh, did that change? i didn't realize it was something we wanted fixed, but rather a safety measure to prevent accidentally merging a change after someone abandons its dependencies | 19:39 |
| jeblair | fungi: it's basically an unfixable situation if we don't ignore abandoned deps | 19:39 |
| jeblair | like, if you accidentally propose a dep to the wrong branch, if we didn't ignore it being abandoned, you'd have to start all over | 19:40 |
| fungi | yeah, and granted that's less of a concern for actual code dependencies and more a worry with dependencies declared solely for blocking purposes | 19:40 |
| fungi | it's a tough call since gerrit isn't designed to allow you to alter the change-id on a change without making a wholly new change | 19:41 |
| fungi | makes the blocking-dependency use case a little less effective | 19:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix for pep8 E722 and ignore E741 https://review.openstack.org/514372 | 19:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix undefined sort order when applying parent data https://review.openstack.org/514056 | 19:42 |
| jeblair | fungi: what's the blocking dependency use case? | 19:42 |
| fungi | i don't want change b to merge until change a has merged | 19:42 |
| smcginnis | fungi: Did we need to still disable deleting the launchpad credentials and rerun that job? | 19:43 |
| fungi | jeblair: and so abandoning change a allows change b to merge even with a depends-on a declared | 19:43 |
| *** thiagolib has quit IRC | 19:43 | |
| fungi | smcginnis: yes, 514436 needs one more project-config-core review | 19:44 |
| smcginnis | fungi: Cool, missed it. | 19:44 |
| clarkb | I'll review that now. Sorry been distracted by relearning zuuls test suite in order to write this test and run tests locally | 19:45 |
| fungi | jeblair: anyway, it's not a huge concern since it can be done by blocking with wip or cr-2 | 19:45 |
| fungi | just requires a little additional manual attention | 19:45 |
| *** camunoz has joined #openstack-infra | 19:47 | |
| *** e0ne has joined #openstack-infra | 19:47 | |
| fungi | thanks clarkb! smcginnis: once it merges here in a few minutes i'll reenqueue that last releases commit into release-post and see what we catch. i've already reset the spring on our autohold trap | 19:49 |
| *** e0ne has quit IRC | 19:52 | |
| smcginnis | fungi: Awesome, thanks | 19:52 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul feature/zuulv3: Only autohold failed builds https://review.openstack.org/513850 | 19:53 |
| openstackgerrit | Merged openstack-infra/project-config master: drop legacy panko jobs https://review.openstack.org/512429 | 19:53 |
| *** sree has joined #openstack-infra | 19:55 | |
| *** armax has quit IRC | 19:57 | |
| fungi | smcginnis: okay, reenqueued | 19:58 |
| *** e0ne has joined #openstack-infra | 19:59 | |
| *** rtjure has joined #openstack-infra | 19:59 | |
| smcginnis | ++ | 19:59 |
| *** sree has quit IRC | 20:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: Temporarily leave launchpad creds for release jobs https://review.openstack.org/514436 | 20:02 |
| fungi | oh, nuts, that change hadn't merged yet? for some reason i thought it had | 20:04 |
| *** e0ne has quit IRC | 20:04 | |
| fungi | i'll probably have to enqueue again, but maybe we'll get something useful out of the unbuffered stdio | 20:04 |
| *** rtjure has quit IRC | 20:04 | |
| smcginnis | Those release-post jobs are still queued, so it should be OK, right? | 20:05 |
| smcginnis | I thought that had merged already too. | 20:05 |
| tosky | jeblair: talking about debugging; I used "roles:" instead of "tasks:" by mistake in the run: playbook of my job, and I didn't get any special error message; the run phase simply completed in 1 second without other details | 20:05 |
| tosky | and it took a bit to me to realize what was wrong | 20:05 |
| tosky | maybe it was just an ansible issue, but still | 20:06 |
| fungi | smcginnis: not sure but i think as long as they hadn't gotten node assignments we may be in luck | 20:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Document executor/merger stats https://review.openstack.org/514343 | 20:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 20:08 |
| jeblair | tosky: yeah, there are some patches in-flight to make a nice standard ansible-linter job for zuul playbooks. i think we'll be able to suggest adding it to repos soon | 20:09 |
| jeblair | pabelanger, jlk: ^ | 20:10 |
| jlk | Was that a pointer to the WIP one or the docs one? | 20:10 |
| jeblair | jlk: oh sorry, was a pointer to the ansible lint convo | 20:11 |
| tosky | if the ansible-linter jobs is just a call to ansible-lint, I think that it did not help in my case | 20:11 |
| jeblair | tosky: oh bummer | 20:11 |
| *** e0ne has joined #openstack-infra | 20:11 | |
| jlk | ah okay. Yeah, that's WIP, lots of moving parts and I got tied up in some other things | 20:11 |
| jlk | hrm, ansible-lint may have caught that you were calling a role that didn't exist | 20:12 |
| pabelanger | tosky: ansible-playbook --syntax-check ? | 20:12 |
| pabelanger | should have caught it | 20:12 |
| jlk | or hte syntax check | 20:12 |
| tosky | oh | 20:12 |
| pabelanger | which we do on our linters | 20:12 |
| tosky | I always forget that | 20:12 |
| jeblair | cool, so once we get that going, it should help errors like that | 20:12 |
| tosky | so eagerly waiting for the linters :) | 20:12 |
| *** rtjure has joined #openstack-infra | 20:13 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Add Test Migration Directions https://review.openstack.org/502509 | 20:14 |
| *** ldnunes has quit IRC | 20:15 | |
| *** e0ne has quit IRC | 20:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Create openstack-zuul-jobs-linters jobs https://review.openstack.org/512715 | 20:17 |
| pabelanger | tosky: jlk: jeblair: AJaeger: ^should get us started, plan is to iterate on that and maybe drop tox | 20:17 |
| *** Apoorva has quit IRC | 20:17 | |
| *** rtjure has quit IRC | 20:18 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add log streaming logging and exception handling https://review.openstack.org/513811 | 20:18 |
| *** Apoorva has joined #openstack-infra | 20:18 | |
| smcginnis | fungi: Wait, did it work that time? | 20:18 |
| smcginnis | Or... did it not try to update the bugs because it thought it was already done. | 20:19 |
| fungi | smcginnis: both are possibilities here... checking the log now | 20:20 |
| *** jamesmcarthur has quit IRC | 20:20 | |
| smcginnis | Session wrapping up. I'll be back later. | 20:20 |
| fungi | smcginnis: i don't see any mention of bug 1720443 in the log this time, so may not be exercising it now | 20:21 |
| openstack | bug 1720443 in sushy "Get throws exception when system memory is null" [High,Fix released] https://launchpad.net/bugs/1720443 - Assigned to Nate Potter (ntpttr) | 20:21 |
| fungi | no references in the console log to launchpad_add_comment being called either | 20:22 |
| *** e0ne has joined #openstack-infra | 20:23 | |
| *** mat128 has quit IRC | 20:24 | |
| fungi | smcginnis: we may need a bug-fixing commit to the release-test repo to properly exercise this | 20:24 |
| *** rtjure has joined #openstack-infra | 20:25 | |
| mgagne | dmsimard: mtl01 has 150 servers available. benj_ reverted to previous values (them being right or wrong). it's up to infra to allocate those servers as they please. | 20:25 |
| jeblair | fungi: i saw a question earlier about whether a job would run with a change that had just landed. note that zuul decides what refs to use for all of the required-projects (and projects under test) for all of the jobs when the item is enqueued into the pipeline. | 20:27 |
| *** smatzek has quit IRC | 20:27 | |
| mgagne | dmsimard: in fact, we increased to 190 recently: https://review.openstack.org/#/c/491882/ | 20:27 |
| mgagne | so 150 + 40 was fine. | 20:27 |
| fungi | jeblair: thanks, that helps | 20:27 |
| fungi | i know in the past v2 would add and remove jobs to enqueued items when layout updates were applied | 20:28 |
| *** e0ne has quit IRC | 20:28 | |
| fungi | good to know this is a little more deterministic under v3 | 20:28 |
| *** rtjure has quit IRC | 20:29 | |
| mgagne | fungi: should all capacity for inap-mtl01 be moved to zuulv3 ? | 20:31 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Shift inap-mtl01 capacity to full Zuul/Nodepool v3 https://review.openstack.org/514451 | 20:31 |
| jeblair | fungi: heh, it will still do that. but it will use the same git checkouts for all of the content. | 20:31 |
| dmsimard | mgagne: ack, can you review ^ ? | 20:31 |
| *** e0ne has joined #openstack-infra | 20:31 | |
| mgagne | well, we got our answer | 20:31 |
| dmsimard | that'll give us an extra 150 nodes on v3 which will really help, we were capped all day | 20:31 |
| fungi | mgagne: yeah, i expect the temporary removal/revert straddled when we shifted all our v2 quota to v3 | 20:32 |
| mgagne | hehe, as long as you get back the capacity, it's all fine =) | 20:32 |
| fungi | so we ended up adding a lot of quota from there back to (now unused) v2 | 20:32 |
| pabelanger | Yay for more nodes | 20:33 |
| *** salv-orlando has joined #openstack-infra | 20:33 | |
| mgagne | and it looks like the nova upgrade improved requests timing. but it could also be related to zuulv3, who knows ¯\_(ツ)_/¯ | 20:34 |
| pabelanger | https://review.openstack.org/513959/ should be an easy +3 to update zuul-status page in grafana | 20:34 |
| *** rtjure has joined #openstack-infra | 20:34 | |
| *** armax has joined #openstack-infra | 20:34 | |
| dmsimard | mgagne: thanks, much appreciated ! | 20:35 |
| *** e0ne has quit IRC | 20:35 | |
| *** hemna_ has quit IRC | 20:36 | |
| *** salv-orlando has quit IRC | 20:37 | |
| *** salv-orlando has joined #openstack-infra | 20:37 | |
| *** dprince has quit IRC | 20:37 | |
| dhellmann | fungi, smcginnis : I'm getting a timeout error trying to report a bug on launchpad | 20:37 |
| dhellmann | setting up a test case for this may prove challenging | 20:38 |
| *** eharney has quit IRC | 20:38 | |
| fungi | dhellmann: this seems to be much more likely to have been the cause of the hung lp api call in the job earlier | 20:38 |
| dhellmann | fungi : perhaps, though the browser is giving me an error page quite quickly | 20:39 |
| dhellmann | the browser itself responds; it's some database that is reported as timing out | 20:39 |
| *** rtjure has quit IRC | 20:39 | |
| dhellmann | ah, interesting, the pbr repo works. sharding magic. | 20:40 |
| fungi | huh, okay | 20:41 |
| dhellmann | https://review.openstack.org/514453 includes a reference to a test bug | 20:42 |
| dhellmann | when that merges I'll tag another release in release-test | 20:43 |
| *** baoli has quit IRC | 20:44 | |
| dhellmann | fungi : https://review.openstack.org/#/c/514454/ should trigger another test run. let me know when you're ready for me to approve it | 20:45 |
| *** e0ne has joined #openstack-infra | 20:45 | |
| *** armaan has quit IRC | 20:45 | |
| fungi | dhellmann: approve at will | 20:46 |
| *** armaan has joined #openstack-infra | 20:46 | |
| *** rtjure has joined #openstack-infra | 20:46 | |
| fungi | i already have the autohold in place to catch the node in case we need it | 20:46 |
| *** nicolasbock has quit IRC | 20:48 | |
| dhellmann | ok, it's approved and in the check queue | 20:48 |
| *** trown is now known as trown|outtypewww | 20:48 | |
| *** jascott1_ has joined #openstack-infra | 20:49 | |
| *** jascott1 has quit IRC | 20:49 | |
| *** e0ne has quit IRC | 20:50 | |
| *** baoli has joined #openstack-infra | 20:50 | |
| *** aviau has quit IRC | 20:50 | |
| *** baoli has quit IRC | 20:50 | |
| *** aviau has joined #openstack-infra | 20:51 | |
| *** rtjure has quit IRC | 20:51 | |
| *** e0ne has joined #openstack-infra | 20:52 | |
| *** ijw has joined #openstack-infra | 20:54 | |
| *** hemna_ has joined #openstack-infra | 20:54 | |
| *** smatzek has joined #openstack-infra | 20:56 | |
| *** rtjure has joined #openstack-infra | 20:56 | |
| *** e0ne has quit IRC | 20:57 | |
| *** smatzek has quit IRC | 20:57 | |
| *** smatzek has joined #openstack-infra | 20:58 | |
| *** hemna_ has quit IRC | 20:59 | |
| *** jamesmcarthur has joined #openstack-infra | 21:00 | |
| *** baoli has joined #openstack-infra | 21:00 | |
| *** jcoufal_ has quit IRC | 21:01 | |
| *** smatzek has quit IRC | 21:02 | |
| *** rtjure has quit IRC | 21:02 | |
| *** thorst has quit IRC | 21:02 | |
| *** smatzek has joined #openstack-infra | 21:02 | |
| *** andreas_s has joined #openstack-infra | 21:05 | |
| dhellmann | ah, that shortcut did not have the effect I expected | 21:05 |
| dhellmann | I'll have to do that again, because it ran the script as me :-/ | 21:06 |
| fungi | d'oh | 21:06 |
| dhellmann | yeah. I thought it would just apply the tag, and the job would run the script | 21:06 |
| dhellmann | oh well | 21:06 |
| *** smatzek has quit IRC | 21:07 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 21:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Add implied branch matchers on 'master' https://review.openstack.org/514459 | 21:07 |
| *** rtjure has joined #openstack-infra | 21:08 | |
| *** caphrim007 has joined #openstack-infra | 21:10 | |
| dhellmann | fungi : another release is queued up and I will leave it to run | 21:10 |
| tosky | how do I get the full "address" for finger from zuulv3.openstack.org? I see only the link to the web feed | 21:11 |
| *** ijw has quit IRC | 21:12 | |
| openstackgerrit | Merged openstack-infra/project-config master: Shift inap-mtl01 capacity to full Zuul/Nodepool v3 https://review.openstack.org/514451 | 21:12 |
| fungi | dhellmann: thanks, i'll keep an eye out | 21:13 |
| *** rtjure has quit IRC | 21:13 | |
| *** slaweq has quit IRC | 21:14 | |
| jeblair | tosky: i'm not sure that we've exposed that yet. we plan to implement a multiplexer so you can "finger UUID@zuul.openstack.org", but that's not in place yet. the backend executor address might be in status.json? i'm not certain. | 21:15 |
| *** slaweq has joined #openstack-infra | 21:15 | |
| *** jamesmcarthur has quit IRC | 21:17 | |
| *** jamesmcarthur has joined #openstack-infra | 21:18 | |
| *** rtjure has joined #openstack-infra | 21:18 | |
| *** andreas_s has quit IRC | 21:18 | |
| *** edmondsw has quit IRC | 21:20 | |
| *** slaweq has quit IRC | 21:20 | |
| *** ijw has joined #openstack-infra | 21:22 | |
| *** rtjure has quit IRC | 21:23 | |
| tosky | jeblair: when can you use finger right now? Is it just for internal usage? | 21:23 |
| *** andreas_s has joined #openstack-infra | 21:23 | |
| clarkb | tosky: you can use it now but you have to know what executor it is running on to directly address that host as I understand it | 21:24 |
| clarkb | tosky: the multiplexor would figure that out for you when it is in place | 21:24 |
| tosky | I see | 21:25 |
| *** thorst has joined #openstack-infra | 21:27 | |
| *** ijw has quit IRC | 21:27 | |
| tosky | yep, it's relatively easy; the executor is printed when add-build-sshkey is executed (Create Temp SSH key) | 21:27 |
| *** rtjure has joined #openstack-infra | 21:27 | |
| *** dave-mccowan has quit IRC | 21:28 | |
| *** iyamahat has joined #openstack-infra | 21:28 | |
| *** iyamahat_ has joined #openstack-infra | 21:28 | |
| *** sree has joined #openstack-infra | 21:29 | |
| *** thorst has quit IRC | 21:31 | |
| *** hemna_ has joined #openstack-infra | 21:32 | |
| *** rtjure has quit IRC | 21:32 | |
| *** sree has quit IRC | 21:34 | |
| *** MasterOfBugs has joined #openstack-infra | 21:34 | |
| *** pramodrj07 has joined #openstack-infra | 21:34 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add log streaming logging and exception handling https://review.openstack.org/513811 | 21:35 |
| *** rtjure has joined #openstack-infra | 21:40 | |
| fungi | dhellmann: smcginnis: looks like we ran it and i have the node held | 21:41 |
| *** andreas_s has quit IRC | 21:41 | |
| *** ijw has joined #openstack-infra | 21:42 | |
| fungi | http://logs.openstack.org/81/81e38a48b3b9e31c4e2bbce5cd5f63a9d4518421/release-post/tag-releases/7661866/ is the build and looks like it succeeded? | 21:42 |
| fungi | i've set another autohold just in case that was not the one we needed | 21:43 |
| *** tpsilva has quit IRC | 21:44 | |
| *** slaweq has joined #openstack-infra | 21:45 | |
| fungi | ahh, that was a different commit. okay | 21:45 |
| *** rtjure has quit IRC | 21:45 | |
| *** andreas_s has joined #openstack-infra | 21:46 | |
| *** mriedem is now known as mriedem_away | 21:48 | |
| *** ijw has quit IRC | 21:49 | |
| *** esberglu has quit IRC | 21:49 | |
| *** esberglu has joined #openstack-infra | 21:49 | |
| *** esberglu_ has joined #openstack-infra | 21:51 | |
| *** rtjure has joined #openstack-infra | 21:52 | |
| *** florianf has quit IRC | 21:52 | |
| *** sambetts has quit IRC | 21:53 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Publish requirements loci images to DockerHub https://review.openstack.org/512941 | 21:54 |
| *** esberglu has quit IRC | 21:54 | |
| clarkb | pabelanger: jlk SpamapS Shrews willing to review https://review.openstack.org/#/c/513850/6 (just needs another reviewer and will help make autohold more useful for us) | 21:55 |
| *** esberglu_ has quit IRC | 21:56 | |
| *** rtjure has quit IRC | 21:56 | |
| Shrews | clarkb: lgtm | 21:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 21:58 |
| jeblair | it's zuul meeting time in #openstack-meeting-alt | 21:59 |
| jeblair | more or less | 21:59 |
| *** esberglu has joined #openstack-infra | 22:01 | |
| *** rtjure has joined #openstack-infra | 22:02 | |
| *** andreas_s has quit IRC | 22:04 | |
| *** rtjure has quit IRC | 22:07 | |
| *** iyamahat_ has quit IRC | 22:08 | |
| *** iyamahat has quit IRC | 22:08 | |
| *** ijw has joined #openstack-infra | 22:08 | |
| *** andreas_s has joined #openstack-infra | 22:09 | |
| *** ijw has quit IRC | 22:11 | |
| *** claudiub|2 has quit IRC | 22:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove py27-based template for nodepool https://review.openstack.org/513770 | 22:12 |
| *** rtjure has joined #openstack-infra | 22:12 | |
| *** ijw has joined #openstack-infra | 22:13 | |
| *** andreas_s has quit IRC | 22:13 | |
| pabelanger | I'd like to promote up https://review.openstack.org/513506/ to base if possible. | 22:14 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add log streaming logging and exception handling https://review.openstack.org/513811 | 22:14 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Only autohold failed builds https://review.openstack.org/513850 | 22:14 |
| *** rtjure has quit IRC | 22:17 | |
| *** iyamahat has joined #openstack-infra | 22:17 | |
| *** slaweq has quit IRC | 22:18 | |
| *** ijw has quit IRC | 22:19 | |
| *** rtjure has joined #openstack-infra | 22:21 | |
| dhellmann | fungi : looking... | 22:22 |
| *** ijw has joined #openstack-infra | 22:23 | |
| dhellmann | fungi : that job doesn't show any releases. I think that's a different patch that cleaned up the zuul.yaml? | 22:23 |
| dhellmann | fungi : yeah, that wasn't the release | 22:23 |
| *** esberglu has quit IRC | 22:25 | |
| fungi | yup | 22:25 |
| dhellmann | fungi : I do see a comment from another job on https://bugs.launchpad.net/pbr/+bug/1726588 though | 22:26 |
| openstack | Launchpad bug 1726588 in PBR "test bug for release system" [Undecided,New] | 22:26 |
| dhellmann | oh, wait, that's the one I did by hand | 22:26 |
| fungi | got sidetracked by the zuul meeting | 22:26 |
| dhellmann | we want release-test 0.10.2 | 22:26 |
| fungi | we held another node... checking to see what triggered it now | 22:26 |
| *** rtjure has quit IRC | 22:26 | |
| dhellmann | http://logs.openstack.org/4e/4e6d9fe5a89fbb7157fd3f790682fef421d56c2c/ | 22:26 |
| dhellmann | oh, not that one though, the commit where it merged in openstack/releases | 22:27 |
| dhellmann | sorry, I'm also distracted | 22:27 |
| *** slaweq has joined #openstack-infra | 22:28 | |
| dhellmann | fungi : http://zuulv3.openstack.org/static/stream.html?uuid=2bb541549ed34c7c934c3c17719b0128&logfile=console.log is hung waiting for input from the console | 22:28 |
| fungi | yeah, we don't seem to have a tag-releases run for 4b66723dd54f56ab5555e422aa58daf86b98f700 | 22:28 |
| dhellmann | it's still running | 22:28 |
| dhellmann | so no logs yet | 22:28 |
| fungi | but once it ends i'll jump on there | 22:28 |
| dhellmann | ^^ | 22:29 |
| *** hemna_ has quit IRC | 22:29 | |
| fungi | the error in the console stream there does indeed look like it can't find the lp creds | 22:29 |
| dhellmann | it is trying to open a page in a web browser to auth the user | 22:29 |
| fungi | right | 22:29 |
| fungi | that's its default "i don't have my credentials token (yet)" behavior | 22:30 |
| * dhellmann nods | 22:30 | |
| *** baoli has quit IRC | 22:30 | |
| dhellmann | can you get into that node to see what credentials it has? | 22:31 |
| dhellmann | I'm not sure how the hold thing works, if you have to wait for the job to time out | 22:31 |
| dhellmann | or if you can login while it's running | 22:31 |
| *** rtjure has joined #openstack-infra | 22:31 | |
| *** iyamahat has quit IRC | 22:32 | |
| *** iyamahat_ has joined #openstack-infra | 22:32 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove zuul-env from DIB images https://review.openstack.org/514483 | 22:32 |
| *** bobh has quit IRC | 22:33 | |
| *** salv-orlando has quit IRC | 22:34 | |
| *** salv-orlando has joined #openstack-infra | 22:34 | |
| *** rtjure has quit IRC | 22:36 | |
| *** andreas_s has joined #openstack-infra | 22:36 | |
| fungi | dhellmann: it'll be easier for me to get in there after zuul holds the node since i have to go digging in launcher logs for the ip address otherwise | 22:37 |
| dhellmann | wow, the timeout is 30 minutes. I guess the library never actually gives up. | 22:37 |
| fungi | but also, still trying to participate in the zuul meeting for now | 22:37 |
| dhellmann | ok | 22:38 |
| *** salv-orlando has quit IRC | 22:39 | |
| fungi | we did just hold another node a couple minutes ago and i'm logged into it now, but need to refresh my memory on where the lp credential file is supposed to be on the fs | 22:40 |
| clarkb | fungi: its in ~/.config or something like that | 22:41 |
| clarkb | iirc | 22:41 |
| fungi | ara says /home/zuul/.launchpadlib/creds | 22:41 |
| *** rtjure has joined #openstack-infra | 22:42 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/zuul-jobs master: fix the path for the launchpad credentials file https://review.openstack.org/514484 | 22:43 |
| *** ijw has quit IRC | 22:43 | |
| dhellmann | fungi, clarkb : ^^ updates the path to match what I have on my system | 22:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove jenkins from DIB images https://review.openstack.org/514485 | 22:43 |
| *** vhosakot has quit IRC | 22:44 | |
| *** ijw has joined #openstack-infra | 22:44 | |
| fungi | dhellmann: comparing ~/.launchpadlib/creds on signing01.ci and the node where that job just ran, the only difference i can spot is some extra blank lines following the two with the secret values | 22:45 |
| dhellmann | I wouldn't expect some blank lines to matter, but who knows | 22:45 |
| *** andreas_s has quit IRC | 22:45 | |
| *** thorst has joined #openstack-infra | 22:46 | |
| thingee | Hi infra, I've been trying to get the contributor guide post jobs to go. The last review I pushed through https://review.openstack.org/#/c/514461/ showed up on status zuul under post jobs but I never saw anything in the console log http://zuulv3.openstack.org/static/stream.html?uuid=65d05dbee99c42fa889bb4196b563d60&logfile=console.log | 22:46 |
| *** rtjure has quit IRC | 22:46 | |
| fungi | what's the reason for 514484? is that where the script is looking for those files? worth noting zuul-jobs is supposed to be a non-openstack-specific stdlib for zuul, so we likely want to stick to whatever defaults launchpadlib expects | 22:47 |
| fungi | dhellmann: ^ | 22:47 |
| dhellmann | fungi : that's where I found my credentials file. I am trying to figure out if launchpad lib has a search path or something. | 22:47 |
| *** edmondsw has joined #openstack-infra | 22:48 | |
| fungi | the version of launchpadlib we were using on signing01.ci.o.o at least relied on credentials in ~/.launchpadlib/creds | 22:48 |
| dhellmann | oh, we can specify the file using an environment variable | 22:49 |
| dhellmann | to the script we're running | 22:49 |
| dhellmann | or an argument | 22:49 |
| dhellmann | I wonder if we do that | 22:49 |
| *** dave-mccowan has joined #openstack-infra | 22:49 | |
| dhellmann | the default is empty | 22:49 |
| dhellmann | that's likely to be an issue | 22:49 |
| fungi | i can test out theories with this node once the meeting wraps up | 22:50 |
| dhellmann | ah, it is set in the job in tag.yaml | 22:50 |
| fungi | i have some scripts i can use to exercise the lp api pretty trivially to confirm auth is working correctly | 22:50 |
| dhellmann | LP_CREDS_FILE=~/.launchpadlib/creds | 22:51 |
| *** thorst has quit IRC | 22:51 | |
| *** jamesmcarthur has quit IRC | 22:51 | |
| dhellmann | if you run the command to leave a comment, that would be best because it exercises the way we actually set up auth in our script | 22:51 |
| dhellmann | it's not automatic :-/ | 22:51 |
| *** jamesmcarthur has joined #openstack-infra | 22:52 | |
| *** edmondsw has quit IRC | 22:52 | |
| dhellmann | I see it setting the variable properly in http://logs.openstack.org/4b/4b66723dd54f56ab5555e422aa58daf86b98f700/release-post/tag-releases/2bb5415/job-output.txt.gz#_2017-10-23_22_05_55_879886 | 22:53 |
| *** rtjure has joined #openstack-infra | 22:53 | |
| *** ijw has quit IRC | 22:54 | |
| *** ijw has joined #openstack-infra | 22:55 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header https://review.openstack.org/509436 | 22:55 |
| *** jamesmcarthur has quit IRC | 22:56 | |
| *** janki has joined #openstack-infra | 22:57 | |
| *** sdague has quit IRC | 22:58 | |
| *** Apoorva has quit IRC | 22:58 | |
| *** rtjure has quit IRC | 22:58 | |
| *** andreas_s has joined #openstack-infra | 22:59 | |
| *** ijw has quit IRC | 23:00 | |
| *** slaweq has quit IRC | 23:01 | |
| *** janki has quit IRC | 23:02 | |
| *** ijw has joined #openstack-infra | 23:02 | |
| *** aeng has quit IRC | 23:02 | |
| *** iyamahat_ has quit IRC | 23:02 | |
| *** iyamahat has joined #openstack-infra | 23:03 | |
| *** sree has joined #openstack-infra | 23:04 | |
| *** Apoorva has joined #openstack-infra | 23:05 | |
| dhellmann | it's time to call it a night. I'll be back on in the morning to pick up from here. | 23:05 |
| fungi | dhellmann: so... i've confirmed that i can interact with the lp api from the held test node using the credentials provided, and that they match the ones we used on the v2 signing01.ci node | 23:06 |
| *** xarses has quit IRC | 23:06 | |
| dhellmann | fungi : with the script that the job is running? or something else? | 23:07 |
| fungi | dhellmann: yep, seems this one is a bit mysterious... either we're overriding the default location for credentials or something else weird is happening | 23:07 |
| fungi | dhellmann: not with the script from the job yet, no | 23:07 |
| dhellmann | because setting up auth can be done different ways in the lp lib api, I don't trust anything other than the actual script we're trying to use | 23:07 |
| dhellmann | I mean, the credentials are right, but they're not making it into the script | 23:07 |
| dhellmann | maybe a debug print is worthwhile | 23:07 |
| fungi | yeah, i'll fiddle with that, just need to make sure i set up the shell environment as identical as possible to however that script was orignially invoked in the job | 23:08 |
| *** andreas_s has quit IRC | 23:08 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 23:08 |
| *** jgriffith has joined #openstack-infra | 23:08 | |
| *** sree has quit IRC | 23:08 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add more detail to output of launchpad script https://review.openstack.org/514486 | 23:08 |
| fungi | because the credentials as provided to zuul's account seem to be getting found and used successfully for simple queries (and if i move them out of the way, i get the prompt like the job showed instead) | 23:09 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add more detail to output of launchpad script https://review.openstack.org/514486 | 23:09 |
| dhellmann | ok | 23:09 |
| *** rbrndt has quit IRC | 23:10 | |
| *** slaweq has joined #openstack-infra | 23:10 | |
| dhellmann | like I said, there are different ways to do the auth within the script and I don't know if the one you're testing with is doing it the way the one from the job is, or if we're getting the right filename down in the script when it's run from the job | 23:10 |
| fungi | dhellmann: yes, both are distinct possibilities which is why i need to do some more manual testing on here | 23:10 |
| dhellmann | so it's good the credentials work, I guess, because that means the problem is definitely somewhere else in the process | 23:11 |
| * dhellmann nods | 23:11 | |
| dhellmann | ok, time to set up dinner, I'll check in tomorrow morning to see how things are looking and pick up from there -- thanks fungi | 23:11 |
| fungi | dhellmann: you bet. enjoy your evening! | 23:12 |
| *** gildub has joined #openstack-infra | 23:12 | |
| thingee | fungi can I get help with that contributor guide post job? | 23:13 |
| fungi | thingee: probably, but i'm still trying to get release jobs working | 23:13 |
| thingee | oh whoa nevermind :) | 23:13 |
| *** felipemonteiro__ has quit IRC | 23:13 | |
| fungi | and volunteered to work on restarting all the executors once i figure this issue out | 23:13 |
| pabelanger | fungi: jeblair: clarkb: https://review.openstack.org/514483/ removes zuul-env (and zuul-cloner) from DIBs. If you want to review, otherwise happy to WIP until we are ready | 23:14 |
| *** rtjure has joined #openstack-infra | 23:14 | |
| *** hasharDinner has quit IRC | 23:15 | |
| mnaser | hey uh | 23:16 |
| mnaser | "the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: 'nodepool' is undefined" | 23:16 |
| mnaser | emit-job-header broken? | 23:16 |
| mnaser | https://github.com/openstack-infra/zuul-jobs/commit/96bd24134364124d79c1a7ed96f22174bec6bf02 | 23:17 |
| mnaser | this broke it i think | 23:17 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul-jobs master: Revert "Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header" https://review.openstack.org/514488 | 23:18 |
| *** aeng has joined #openstack-infra | 23:19 | |
| *** rtjure has quit IRC | 23:19 | |
| jeblair | i imagine we're going to need to force-merge that revert? | 23:19 |
| dmsimard | jeblair: yes | 23:20 |
| jeblair | on it | 23:20 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Revert "Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header" https://review.openstack.org/514488 | 23:21 |
| pabelanger | we didn't test on localhost, which doesn't have nodepool variables | 23:21 |
| *** ijw has quit IRC | 23:21 | |
| pabelanger | so, likely want to emit that on the node | 23:21 |
| clarkb | jeblair: in https://review.openstack.org/#/c/513766/1/playbooks/nodepool-zuul-functional/post.yaml what does --include=*/ --exlude=* mean when rsync runs? | 23:21 |
| *** gildub has quit IRC | 23:22 | |
| clarkb | is the include=*/ redundant with include=/logs/** ? | 23:22 |
| *** gmann_afk is now known as gmann | 23:22 | |
| jeblair | pabelanger: or go through the hostvars and print all of the values | 23:22 |
| jeblair | pabelanger: i think it'd be best to stay on the executor for this if we can | 23:22 |
| dmsimard | pabelanger: http://logs.openstack.org/36/509436/6/gate/base-integration-ubuntu-trusty/694fc5b/job-output.txt.gz#_2017-10-23_22_44_20_092357 works | 23:22 |
| jeblair | clarkb: that looks like a legacy publisher | 23:23 |
| pabelanger | dmsimard: that is not localhost, you ran it on xenial node | 23:23 |
| dmsimard | pabelanger: the reason why we want it to run on localhost was to avoid printing that 4 times if there's 4 nodes.. maybe we can try run_once | 23:23 |
| jeblair | let's run it on localhost and have it print out the nodepool variables for all the hosts | 23:23 |
| *** rtjure has joined #openstack-infra | 23:24 | |
| jeblair | there's no reason to delegate to a remote node for a debug line. | 23:24 |
| dmsimard | jeblair: I believe we can set hosts: all, and then have the task use "run_once" so that the task is only run on one (remote) host -- debug is an action module which runs on the control (localhost) node anyway | 23:25 |
| dmsimard | http://docs.ansible.com/ansible/latest/playbooks_delegation.html#run-once | 23:25 |
| jeblair | dmsimard: you're saying that 'debug' doesn't run over ssh? | 23:26 |
| jeblair | dmsimard: even if it's told to run on a remote host | 23:26 |
| dmsimard | jeblair: action modules are typically executed on the control node, "debug" is one, "template" has one half on the control host, the other half on the remote host | 23:26 |
| clarkb | jeblair: ya I think this new non legacy job is doing a lot of things in a legacy way? trying to understand if we should be doing something different there | 23:26 |
| *** dave-mccowan has quit IRC | 23:27 | |
| clarkb | Shrews: I left a comment on 513766 too | 23:27 |
| dmsimard | jeblair: the task will still run *for* the remote host, but the actual execution typically happens on the control host | 23:27 |
| dmsimard | that's why on centos you'll need libselinux on the control host, because ansible doesn't do everything on the remote nodes | 23:28 |
| jeblair | dmsimard: okay. well, this is a role that's in a play currently restricted to localhost, so that'll be a bit of reengineering to do that, yeah? | 23:28 |
| *** rtjure has quit IRC | 23:29 | |
| dmsimard | jeblair: sending a patch for it now. | 23:29 |
| jeblair | dmsimard: okay. if you're up for that. :) | 23:29 |
| jeblair | dmsimard: however, it is still possible to do all of this in the current role, just by looping over hostvars.hosts in the template | 23:30 |
| *** wolverineav has quit IRC | 23:31 | |
| *** tosky has quit IRC | 23:34 | |
| fungi | is http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/release/tag.yaml guaranteed to run under bash, or might it actually use sh? | 23:34 |
| *** rtjure has joined #openstack-infra | 23:34 | |
| clarkb | fungi: it will use sh unless you set the executable I think | 23:35 |
| fungi | it's acting an awful lot like the shell is not expanding ~ there | 23:35 |
| fungi | which, if it's plain sh, would be expected | 23:35 |
| clarkb | we seem to explicitly set executable: /bin/bash where we need it | 23:35 |
| *** andreas_s has joined #openstack-infra | 23:36 | |
| *** aeng has quit IRC | 23:36 | |
| fungi | i guess we just don't have any examples of that in project-config playbooks (yet) | 23:37 |
| fungi | though we do use it in roles/build-wheels/tasks/main.yaml: | 23:37 |
| clarkb | fungi: https://review.openstack.org/#/c/513766/1/playbooks/nodepool-zuul-functional/run.yaml has an example | 23:37 |
| fungi | yup | 23:38 |
| *** rtjure has quit IRC | 23:39 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Revert "Revert "Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header"" https://review.openstack.org/514489 | 23:39 |
| *** hongbin has quit IRC | 23:39 | |
| *** andreas_s has quit IRC | 23:40 | |
| *** gildub has joined #openstack-infra | 23:40 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Run emit-job-header on any host but localhost https://review.openstack.org/514490 | 23:40 |
| dmsimard | jeblair: ^ these two patches | 23:40 |
| dmsimard | tested locally so fairly confident but please do review :) | 23:41 |
| dmsimard | mnaser: ^ | 23:41 |
| clarkb | dmsimard: I think you need to flip the order around | 23:42 |
| clarkb | otherwise when the revert merges we'll be broken until the second chagne merges | 23:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Use bash for the release/tag playbook https://review.openstack.org/514491 | 23:42 |
| fungi | dhellmann: smcginnis: ^ pretty sure that's the culprit | 23:42 |
| dmsimard | clarkb: hmm, right, but you can't depend on a project-config patch | 23:42 |
| clarkb | dmsimard: you can it just won't take effect until after it merges | 23:43 |
| clarkb | dmsimard: but it will enforce that merge order which is what we want | 23:43 |
| *** dbecker has joined #openstack-infra | 23:43 | |
| *** slaweq has quit IRC | 23:43 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Removes migrated legacy-neutron-lbaasv2-dsvm-* https://review.openstack.org/513550 | 23:43 |
| *** dbecker has quit IRC | 23:43 | |
| dmsimard | clarkb: so what this means is that some jobs might get the header printed N times until the other patch lands, not a huge dela. | 23:43 |
| dmsimard | switching the order. | 23:44 |
| *** rtjure has joined #openstack-infra | 23:44 | |
| clarkb | ya it should work just maybe eb noisy rather than the other way where things will be broken for a period of time aiui | 23:44 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Run emit-job-header on any host but localhost https://review.openstack.org/514490 | 23:44 |
| *** jascott1_ has quit IRC | 23:44 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Revert "Revert "Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header"" https://review.openstack.org/514489 | 23:44 |
| *** jascott1 has joined #openstack-infra | 23:44 | |
| *** jascott1 has quit IRC | 23:45 | |
| clarkb | fungi: that dict is different than the one in the example I found .I guess it is to the docs to figure out if one or the other or both should work | 23:46 |
| fungi | clarkb: the one i stole was from roles/build-wheels/tasks/main.yaml in project-config | 23:47 |
| *** mat128 has joined #openstack-infra | 23:47 | |
| fungi | maybe that's role-only syntax and doesn't work for tasks in playbooks? | 23:47 |
| *** bobh has joined #openstack-infra | 23:48 | |
| *** rtjure has quit IRC | 23:48 | |
| clarkb | fungi: your version seems closer to the docs | 23:48 |
| clarkb | I'm trying to remember why we use cmd like that as its undocumented too iirc | 23:49 |
| clarkb | there was a reason I just don't remember it | 23:49 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fixed typo in zaqar tempest plugin configuration https://review.openstack.org/514182 | 23:49 |
| *** aeng has joined #openstack-infra | 23:49 | |
| dmsimard | pabelanger: how do you suggest we test that if the integration tests didn't catch it ? | 23:50 |
| *** xarses has joined #openstack-infra | 23:50 | |
| dmsimard | pabelanger: my brain is tired, I'll -W for now and look tomorrow | 23:50 |
| pabelanger | dmsimard: the role should run properly on the remote node, regardress of the changes in 514489. We just want to exersize it did run on the remote node | 23:51 |
| pabelanger | so, landing in base-test will allow that to be tested | 23:52 |
| clarkb | pabelanger: do you remember why we use cmd: | in shell tasks? | 23:52 |
| *** slaweq has joined #openstack-infra | 23:52 | |
| clarkb | pabelanger: rather than just shell: and then args: | 23:52 |
| pabelanger | clarkb: i don't, mordred indicated we needed to do that for a reason, but do not remember | 23:53 |
| fungi | infra-root: i'm preparing to begin cycling through zuul executors restarting them one at a time to pick up the most recent bindmounting addition. if you object please let me know soon | 23:53 |
| *** rtjure has joined #openstack-infra | 23:53 | |
| pabelanger | fungi: wfm | 23:53 |
| clarkb | go for it | 23:54 |
| jeblair | clarkb: cmd is the only way to get a multiline shell script without leading spaces. | 23:55 |
| pabelanger | fungi: I'd love to try and restart 1 executor to test https://review.openstack.org/510155/ | 23:55 |
| ianw | do we know about the summary box not showing up on changes like https://review.openstack.org/#/c/514081/ ? | 23:56 |
| inc0 | hey, http://logs.openstack.org/55/513855/5/check/kolla-build-ubuntu-source-dockerhub/541164d/ara/ <- you've seen that? | 23:56 |
| clarkb | jeblair: aha! thats right its for heredocs | 23:56 |
| pabelanger | inc0: yes, recheck please. broken code landed | 23:56 |
| clarkb | so if no heredoc or similar its fine (so I think fungi's change is fine) | 23:56 |
| inc0 | thanks pabelanger | 23:56 |
| jeblair | clarkb: yes. though i am in favor of using cmd everwhere. | 23:57 |
| *** rkukura has quit IRC | 23:58 | |
| clarkb | fungi: ^ that is for you too I think | 23:58 |
| clarkb | (not sure if you want a new ps) | 23:58 |
| *** rtjure has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!