| *** rbrndt has quit IRC | 00:01 | |
| *** gmann_afk is now known as gmann | 00:01 | |
| *** threestrands has joined #openstack-infra | 00:01 | |
| *** hemna_ has quit IRC | 00:02 | |
| *** yamamoto has quit IRC | 00:03 | |
| *** hongbin has quit IRC | 00:08 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add sphinx_python variable to sphinx role and job https://review.openstack.org/525688 | 00:11 |
|---|---|---|
| *** aeng has joined #openstack-infra | 00:12 | |
| *** tosky has quit IRC | 00:13 | |
| *** baoli has quit IRC | 00:13 | |
| fungi | smcginnis: i don't think we made headway on them today, no. at least not that i'm aware | 00:17 |
| smcginnis | fungi: OK, thanks. I think I'll have some free time tomorrow morning to try to look at it. | 00:18 |
| *** slaweq has joined #openstack-infra | 00:22 | |
| *** matbu has quit IRC | 00:24 | |
| *** rcernin has quit IRC | 00:25 | |
| *** rcernin has joined #openstack-infra | 00:25 | |
| *** gouthamr has quit IRC | 00:25 | |
| *** slaweq has quit IRC | 00:26 | |
| *** sdague has quit IRC | 00:29 | |
| fungi | i'll be around at least by the time the release team meeting was scheduled for, but have morning errands to tackle before then | 00:30 |
| *** matbu has joined #openstack-infra | 00:30 | |
| fungi | need to disappear again for now though | 00:30 |
| *** cody-somerville has quit IRC | 00:39 | |
| *** camunoz has quit IRC | 00:41 | |
| *** masayukig[m] has joined #openstack-infra | 00:46 | |
| *** kiennt26 has joined #openstack-infra | 00:50 | |
| *** SumitNaiksatam has quit IRC | 00:51 | |
| *** slaweq has joined #openstack-infra | 00:58 | |
| *** huanxie has joined #openstack-infra | 00:59 | |
| *** cuongnv has joined #openstack-infra | 00:59 | |
| *** yamamoto has joined #openstack-infra | 00:59 | |
| *** gyee has quit IRC | 01:01 | |
| *** slaweq has quit IRC | 01:02 | |
| *** yamamoto has quit IRC | 01:04 | |
| *** caphrim007 has joined #openstack-infra | 01:07 | |
| *** ilpianista_ has joined #openstack-infra | 01:14 | |
| *** aspiers[m] has joined #openstack-infra | 01:14 | |
| *** esberglu has joined #openstack-infra | 01:16 | |
| *** caphrim007 has quit IRC | 01:16 | |
| *** caphrim007 has joined #openstack-infra | 01:17 | |
| *** david-lyle has joined #openstack-infra | 01:18 | |
| *** esberglu has quit IRC | 01:21 | |
| *** mikal has quit IRC | 01:24 | |
| *** mikal has joined #openstack-infra | 01:30 | |
| *** slaweq has joined #openstack-infra | 01:32 | |
| *** Apoorva_ has joined #openstack-infra | 01:33 | |
| *** Apoorva has quit IRC | 01:36 | |
| *** david-lyle has quit IRC | 01:36 | |
| *** slaweq has quit IRC | 01:37 | |
| *** dhinesh has quit IRC | 01:37 | |
| *** Apoorva_ has quit IRC | 01:38 | |
| *** rhallisey has quit IRC | 01:45 | |
| *** csomerville has quit IRC | 01:48 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/grafyaml master: Add support for influxdb datasource https://review.openstack.org/306050 | 01:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/grafyaml master: Add support for graph's legend https://review.openstack.org/306660 | 01:55 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Improve zanata-cli download https://review.openstack.org/525658 | 01:57 |
| *** yamamoto has joined #openstack-infra | 02:00 | |
| *** harlowja has quit IRC | 02:03 | |
| *** yamamoto has quit IRC | 02:04 | |
| *** hongbin has joined #openstack-infra | 02:05 | |
| *** jascott1 has quit IRC | 02:08 | |
| dmsimard | AJaeger_: fyi ianw was reporting that there might be some issues with the zanata cache we introduced | 02:08 |
| dmsimard | I don't have details, just pointing out since you're working on zanata in parallel | 02:09 |
| *** slaweq has joined #openstack-infra | 02:12 | |
| *** Zara has joined #openstack-infra | 02:12 | |
| AJaeger_ | dmsimard: thanks | 02:15 |
| AJaeger_ | dmsimard: hope ianw has an idea to fix it, I don't know what's wrong but my change above should not change behavior | 02:15 |
| *** slaweq has quit IRC | 02:17 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove freezer-api legacy jobs https://review.openstack.org/525176 | 02:20 |
| *** mat128 has joined #openstack-infra | 02:23 | |
| *** annp has joined #openstack-infra | 02:26 | |
| *** pbourke has quit IRC | 02:29 | |
| *** pbourke has joined #openstack-infra | 02:30 | |
| *** Goneri has quit IRC | 02:36 | |
| *** namnh has joined #openstack-infra | 02:38 | |
| *** jascott1 has joined #openstack-infra | 02:42 | |
| *** zhurong has joined #openstack-infra | 02:47 | |
| *** slaweq has joined #openstack-infra | 02:48 | |
| dmsimard | Duong Ha-Quang has been surprisingly a great help with the migration, been seeing his name all over the place :D | 02:49 |
| dmsimard | I don't know his IRC handle :/ but thanks | 02:49 |
| *** slaweq has quit IRC | 02:52 | |
| dmsimard | fungi, AJaeger_: Did I miss a discussion around zuul-base-jobs from https://review.openstack.org/#/c/526148/ ? | 02:52 |
| ianw | AJaeger_: yes, changes in progress :) | 02:54 |
| ianw | i think we can just merge everything into the new job, though, when it's ready | 02:54 |
| dmsimard | ianw: did you figure out what was the issue ? | 02:54 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: [WIP] Move prepare-zanata-client to o-z-j https://review.openstack.org/525760 | 02:55 |
| ianw | yes the file has no trailing newline, so when it's read with "< file" in bash it silently skips it | 02:55 |
| ianw | mea culpa for not *fully* checking the result. i just watched the logs, saw it finding the file and assumed | 02:56 |
| AJaeger_ | ianw: works for me as well... | 02:59 |
| AJaeger_ | ianw: shall we do it in two steps, merge and fix? | 02:59 |
| AJaeger_ | merge=move | 02:59 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove legacy jobs in storlets https://review.openstack.org/512959 | 03:00 |
| ianw | AJaeger_: yeah, since the name is slightly different, to be more consistent with the other o-z-j names that are called "prepare-" (rather than prep-) i think we can roll out the new job, then remove the old one when it's unused | 03:00 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove legacy jobs in Murano https://review.openstack.org/511439 | 03:00 |
| *** yamamoto has joined #openstack-infra | 03:01 | |
| AJaeger_ | ianw: care to review dmsimard rename dance stack starting at 526532, please? | 03:02 |
| * dmsimard dances | 03:03 | |
| ianw | ok, will do | 03:03 |
| * AJaeger_ just reviewed a couple of other changes that are ready to merge but nothing urgent... | 03:04 | |
| * AJaeger_ waves good night | 03:05 | |
| *** yamamoto has quit IRC | 03:06 | |
| *** jascott1 has quit IRC | 03:06 | |
| *** Wei_Liu has quit IRC | 03:14 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy jobs in Murano https://review.openstack.org/511439 | 03:14 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy jobs in storlets https://review.openstack.org/512959 | 03:14 |
| ianw | dmsimard: huh ; TASK [prepare-zanata-client : Extract Zanata client archive] : Accessing files from outside the working dir /var/lib/zuul/builds/90ab0ca4f33843428cc9c595ceb72559/work is prohibited", | 03:15 |
| *** ykarel has joined #openstack-infra | 03:15 | |
| ianw | that was not what i expected | 03:15 |
| dmsimard | ianw: is that running on localhost (the executor)? | 03:15 |
| ianw | dmsimard: i was not expecting it to (https://review.openstack.org/#/c/525760/10/roles/prepare-zanata-client/tasks/main.yaml) | 03:18 |
| *** slaweq has joined #openstack-infra | 03:19 | |
| pabelanger | ianw: dmsimard: By default, it will copy the source file from the local system to the target before unpacking. | 03:19 |
| pabelanger | it is the task you are using | 03:19 |
| pabelanger | http://docs.ansible.com/ansible/latest/unarchive_module.html | 03:19 |
| ianw | right, this change just added unarchive, rather than doing it manually | 03:20 |
| pabelanger | remote_src: yes | 03:20 |
| pabelanger | should fix it | 03:20 |
| ianw | that's a good trick | 03:20 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: [WIP] Move prepare-zanata-client to o-z-j https://review.openstack.org/525760 | 03:22 |
| ianw | ... and that is why we test :) | 03:22 |
| *** rlandy has quit IRC | 03:23 | |
| *** slaweq has quit IRC | 03:24 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Check source-repository-* files for trailing newline https://review.openstack.org/526583 | 03:25 |
| *** slaweq has joined #openstack-infra | 03:25 | |
| *** dave-mccowan has quit IRC | 03:26 | |
| *** Wei_Liu has joined #openstack-infra | 03:27 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add newline to zanata source-repository file https://review.openstack.org/526586 | 03:29 |
| *** SumitNaiksatam has joined #openstack-infra | 03:29 | |
| *** slaweq has quit IRC | 03:30 | |
| ianw | can we consider turning on ara even for successful runs in o-z-j / z-j integration tests. 99% of the time you make changes there you're going to want to manually examine the output, even when it works | 03:31 |
| *** SumitNaiksatam_ has joined #openstack-infra | 03:32 | |
| *** Apoorva has joined #openstack-infra | 03:33 | |
| *** SumitNaiksatam has quit IRC | 03:33 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 03:33 | |
| dmsimard | ianw: that means we need to land https://review.openstack.org/#/q/topic:ara-sqlite-middleware :D | 03:36 |
| dmsimard | I need to work on those .. | 03:36 |
| *** mriedem has quit IRC | 03:36 | |
| dmsimard | ianw: that's the idea that came from your middleware thingy | 03:37 |
| dmsimard | https://ara.readthedocs.io/en/latest/advanced.html | 03:37 |
| *** bobh has quit IRC | 03:38 | |
| dmsimard | ianw: but, yes, I'd +2 a patch to enable it for z-j and o-z-j pending the middleware running, considering z-j and o-z-j are not exactly super-high volume (in terms of tasks.. OSA has like >2k tasks in some playbooks) and therefore the amount of files is not very high. | 03:38 |
| dmsimard | I'm biased a little bit :P | 03:38 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: [WIP] Move prepare-zanata-client to o-z-j https://review.openstack.org/525760 | 03:41 |
| *** ramishra has joined #openstack-infra | 03:42 | |
| *** links has joined #openstack-infra | 03:42 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Always generate ara reports for integration tests https://review.openstack.org/526590 | 03:45 |
| ianw | we already have it on multinode, i knew i'd seen it before | 03:46 |
| *** bobh has joined #openstack-infra | 03:49 | |
| *** bobh has quit IRC | 03:53 | |
| *** udesale has joined #openstack-infra | 03:54 | |
| *** slaweq has joined #openstack-infra | 03:57 | |
| *** slaweq has quit IRC | 04:02 | |
| *** yamamoto has joined #openstack-infra | 04:03 | |
| *** rosmaita has quit IRC | 04:03 | |
| *** yamamoto has quit IRC | 04:06 | |
| *** armax has quit IRC | 04:06 | |
| *** yamamoto has joined #openstack-infra | 04:06 | |
| *** armax has joined #openstack-infra | 04:07 | |
| *** armax has quit IRC | 04:07 | |
| *** dhajare has joined #openstack-infra | 04:14 | |
| efried | I'll just leave this here: Anyone know why https://review.openstack.org/#/c/385693/ is stuck? Could it be because the Depends-On has changes in two releases? | 04:15 |
| efried | (Even though both are merged) | 04:16 |
| *** andreas_s has joined #openstack-infra | 04:22 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: handler: fix support for handler without launch_manager https://review.openstack.org/524773 | 04:24 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add a plugin interface for drivers https://review.openstack.org/524620 | 04:24 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: builder: do not cleanup image for driver not managing image https://review.openstack.org/516920 | 04:24 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool https://review.openstack.org/468624 | 04:24 |
| *** zhurong has quit IRC | 04:26 | |
| *** andreas_s has quit IRC | 04:27 | |
| dmsimard | ianw: can you rebase https://review.openstack.org/#/c/526590/ on top of https://review.openstack.org/#/q/topic:base-rename ? | 04:27 |
| dmsimard | Otherwise there'll be a conflict | 04:27 |
| *** huanxie has quit IRC | 04:32 | |
| *** pgadiya has joined #openstack-infra | 04:33 | |
| *** adisky_ has joined #openstack-infra | 04:34 | |
| *** slaweq has joined #openstack-infra | 04:36 | |
| frickler | efried: that change is proceeding in gate, sadly we have 16h backlog currently | 04:38 |
| frickler | ianw: for the mistral patch, jeblair triggered a zuul reconfig to make it pick up the new project | 04:39 |
| *** slaweq has quit IRC | 04:41 | |
| *** bobh has joined #openstack-infra | 04:43 | |
| *** mat128 has quit IRC | 04:43 | |
| *** bhavik1 has joined #openstack-infra | 04:48 | |
| frickler | infra-root: seems something is still/again bad with the executors, we have long backlogs in gate. patches seem stuck on simple jobs like py27 here https://review.openstack.org/526008 | 04:51 |
| *** bobh has quit IRC | 04:51 | |
| frickler | timing would seem to indicate relation with th ze04 issue yesterday | 04:53 |
| *** dhajare has quit IRC | 04:53 | |
| *** adreznec has joined #openstack-infra | 04:54 | |
| *** adreznec has quit IRC | 04:55 | |
| *** bhavik1 has quit IRC | 04:56 | |
| *** adreznec has joined #openstack-infra | 04:57 | |
| *** sree has joined #openstack-infra | 05:00 | |
| *** huanxie has joined #openstack-infra | 05:02 | |
| ianw | frickler: interesting, ze04 seems to not have the executor running | 05:03 |
| ianw | service zuul-executor start doesn't actually start it | 05:07 |
| *** slaweq has joined #openstack-infra | 05:07 | |
| *** Apoorva has quit IRC | 05:09 | |
| *** rcernin has quit IRC | 05:11 | |
| *** pgadiya has quit IRC | 05:11 | |
| *** slaweq has quit IRC | 05:11 | |
| ianw | infra-root: we definitely have an issue with the zuul init scripts. service zuul-executor on ze04 says "ok" but doesn't actually start | 05:12 |
| ianw | i have actually started it now with "export _SYSTEMCTL_SKIP_REDIRECT=1" then /etc/init.d/zuul-exector start | 05:12 |
| ianw | i don't have time to debug the details right now unfortunately | 05:13 |
| *** pahuang has quit IRC | 05:13 | |
| *** Apoorva has joined #openstack-infra | 05:13 | |
| *** pahuang has joined #openstack-infra | 05:14 | |
| ianw | Dec 08 05:08:33 ze04.openstack.org systemd[1]: Started LSB: Zuul. | 05:14 |
| ianw | Dec 08 05:09:08 ze04.openstack.org systemd[1]: Started LSB: Zuul. | 05:14 |
| ianw | systemd gets the message, but the init.d script must be failing silently somehow? | 05:14 |
| *** Apoorva has quit IRC | 05:15 | |
| ianw | #status log manually started zuul-executor on ze04 | 05:16 |
| openstackstatus | ianw: finished logging | 05:16 |
| ianw | this same thing happened yesterday with zuul-scheduler on zuulv3.o.o ... i convinced myself at the time it was a left-over .pid file, but i might have been wrong | 05:17 |
| ianw | i checked for a pid file in /var/run/zuul-executor on ze04 and that wasn't present, so that wasn't stopping the load | 05:17 |
| frickler | ianw: I'm still not seeing any movement on the gate queue, how long would you expext that to take? would a reboot of ze04 be an option? | 05:21 |
| *** dhajare has joined #openstack-infra | 05:22 | |
| ianw | frickler: well ze04 wasn't actually running zuul-executor at all ... it is now | 05:22 |
| ianw | i've checked and all the others seem active | 05:22 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove freezer-api legacy jobs https://review.openstack.org/525176 | 05:23 |
| ianw | it's only 1/9th of the executor capacity however | 05:23 |
| *** threestrands has quit IRC | 05:24 | |
| frickler | it looks to me like not an issue with capacity, but something being locked up | 05:24 |
| openstackgerrit | Merged openstack-infra/project-config master: Add newline to zanata source-repository file https://review.openstack.org/526586 | 05:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Add release-notes-jobs for heat-agents and python-heatclient https://review.openstack.org/526461 | 05:26 |
| ianw | yeah, the legacy-tempest-dsvm-neutron-full job (bba5d98bb7b14b99afb539a75ee86a80) as part of https://review.openstack.org/475955 seems to be stuck | 05:27 |
| ianw | and guess where that went ... | 05:29 |
| ianw | 2017-12-07 15:06:20,962 DEBUG zuul.Pipeline.openstack.gate: Build <Build bba5d98bb7b14b99afb539a75ee86a80 of legacy-tempest-dsvm-neutron-full on <Worker ze04.openstack.org>> started | 05:29 |
| ianw | so ze04 has died and come back to life since then, it's currently Fri Dec 8 05:30:17 UTC 2017 | 05:30 |
| ianw | frickler: i'm going to agree that i think giving zuul a kick here is pretty much the option available, it's clearly not moving on from that job | 05:31 |
| *** raissa has joined #openstack-infra | 05:31 | |
| *** harlowja has joined #openstack-infra | 05:37 | |
| ianw | #status log due to stuck jobs seemingly related to ze04, zuul has been restarted. jobs have been requeued | 05:37 |
| openstackstatus | ianw: finished logging | 05:37 |
| ianw | well, jobs are requeueing right now | 05:38 |
| *** hongbin has quit IRC | 05:38 | |
| *** threestrands has joined #openstack-infra | 05:39 | |
| *** raissa has quit IRC | 05:39 | |
| *** zhurong has joined #openstack-infra | 05:40 | |
| *** slaweq has joined #openstack-infra | 05:40 | |
| *** eumel8 has joined #openstack-infra | 05:41 | |
| *** slaweq has quit IRC | 05:45 | |
| *** armax has joined #openstack-infra | 05:45 | |
| *** armax has quit IRC | 05:55 | |
| *** armax has joined #openstack-infra | 05:55 | |
| *** armax has quit IRC | 05:55 | |
| *** armax has joined #openstack-infra | 05:56 | |
| *** armax has quit IRC | 05:56 | |
| *** eumel8 has quit IRC | 05:57 | |
| *** mikal has quit IRC | 05:57 | |
| *** mikal has joined #openstack-infra | 05:59 | |
| *** janki has joined #openstack-infra | 05:59 | |
| frickler | now build-openstack-releasenotes seems to be failing everywhere, iirc there was some change to it last night. "Could not import extension openstackdocstheme" see e.g. http://logs.openstack.org/95/526595/1/check/build-openstack-releasenotes/f38ccb4/job-output.txt.gz | 06:00 |
| *** dingyichen has quit IRC | 06:01 | |
| *** armaan has joined #openstack-infra | 06:01 | |
| *** armaan has quit IRC | 06:02 | |
| *** dingyichen has joined #openstack-infra | 06:02 | |
| *** ykarel_ has joined #openstack-infra | 06:03 | |
| *** dchen has joined #openstack-infra | 06:04 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Always generate ara reports for integration tests https://review.openstack.org/526590 | 06:05 |
| *** ykarel has quit IRC | 06:05 | |
| ianw | dmsimard: ^ rebased | 06:05 |
| *** dingyichen has quit IRC | 06:07 | |
| *** ykarel_ is now known as ykarel | 06:08 | |
| ianw | frickler: why has that not run ensure-reno? | 06:10 |
| *** rcernin has joined #openstack-infra | 06:11 | |
| frickler | AJaeger_: mordred: my guess is that https://review.openstack.org/525688 is causing the release-notes issues | 06:11 |
| frickler | http://logs.openstack.org/95/526595/1/check/build-openstack-releasenotes/f38ccb4/job-output.txt.gz#_2017-12-08_05_40_07_546925 | 06:11 |
| frickler | the initialize-virtualenv task is skipped | 06:11 |
| frickler | thus installation from doc/requirements.txt is missing | 06:11 |
| *** sree has quit IRC | 06:12 | |
| *** sree has joined #openstack-infra | 06:13 | |
| *** cshastri has joined #openstack-infra | 06:14 | |
| *** harlowja has quit IRC | 06:14 | |
| *** dingyichen has joined #openstack-infra | 06:14 | |
| *** dchen has quit IRC | 06:17 | |
| *** slaweq has joined #openstack-infra | 06:17 | |
| *** dingyichen has quit IRC | 06:17 | |
| *** sree has quit IRC | 06:17 | |
| *** slaweq has quit IRC | 06:21 | |
| *** xinliang has quit IRC | 06:22 | |
| frickler | infra-root: something else seems very broken with zuul now, see e.g. https://review.openstack.org/475955 again, hitting retry_limit on various jobs, and the result link on the status page is http://zuulv3.openstack.org/openstack-tox-py35 | 06:23 |
| *** threestrands has quit IRC | 06:28 | |
| *** aeng has quit IRC | 06:31 | |
| *** Wei_Liu has quit IRC | 06:31 | |
| *** Wei_Liu has joined #openstack-infra | 06:31 | |
| *** xinliang has joined #openstack-infra | 06:35 | |
| *** xinliang has quit IRC | 06:35 | |
| *** xinliang has joined #openstack-infra | 06:35 | |
| *** sree has joined #openstack-infra | 06:43 | |
| ianw | frickler: i think this is what's going on http://paste.openstack.org/show/628420/ | 06:46 |
| *** zhurong has quit IRC | 06:47 | |
| ianw | that's coming up over and over on executor jobs | 06:47 |
| ianw | File "/usr/local/lib/python3.5/dist-packages/zuul/executor/server.py", line 500, in make_inventory_dict | 06:48 |
| ianw | for name in node['name']: | 06:48 |
| ianw | TypeError: unhashable type: 'list' | 06:48 |
| *** vivsoni__ has quit IRC | 06:51 | |
| frickler | ianw: jeblair mentioned some patches yesterday, I added an item to zuulv3 issues | 06:54 |
| ianw | frickler: yeah, i'm thinking at this point my restart probably dumped us into some code with issues :/ | 06:55 |
| *** slaweq has joined #openstack-infra | 06:55 | |
| ianw | i *really* have to go now :) i think i'm just going to have to send an alert that things aren't going well | 06:56 |
| *** andreas_s has joined #openstack-infra | 06:57 | |
| *** sree has quit IRC | 06:57 | |
| *** sree has joined #openstack-infra | 06:58 | |
| frickler | that line above seems to point to https://review.openstack.org/521324 Add support for shared ansible_host in inventory by pabelanger | 06:59 |
| ianw | #status alert Due to some unforseen Zuul issues the gate is under very high load and extremely unstable at the moment. This is likely to persist until PST morning | 07:00 |
| openstackstatus | ianw: sending alert | 07:00 |
| *** slaweq has quit IRC | 07:00 | |
| -openstackstatus- NOTICE: Due to some unforseen Zuul issues the gate is under very high load and extremely unstable at the moment. This is likely to persist until PST morning | 07:02 | |
| *** ChanServ changes topic to "Due to some unforseen Zuul issues the gate is under very high load and extremely unstable at the moment. This is likely to persist until PST morning" | 07:02 | |
| *** sree has quit IRC | 07:03 | |
| ianw | frickler: yeah, i noticed that in the logs ... i suspect too. i'm not enough in the loop to try force merging stuff etc as i'll probably just make things worse | 07:03 |
| ianw | and my kids are about to go to war so, yeah, i think that status is about where we're at | 07:03 |
| openstackstatus | ianw: finished sending alert | 07:06 |
| *** amito has quit IRC | 07:06 | |
| *** seongsoocho has quit IRC | 07:07 | |
| *** serverascode has quit IRC | 07:07 | |
| *** Ng has quit IRC | 07:07 | |
| *** dham1 has quit IRC | 07:07 | |
| *** Ng has joined #openstack-infra | 07:07 | |
| *** mugsie has quit IRC | 07:07 | |
| *** bgmccollum has quit IRC | 07:07 | |
| *** seongsoocho has joined #openstack-infra | 07:07 | |
| *** persia has quit IRC | 07:07 | |
| *** serverascode has joined #openstack-infra | 07:07 | |
| *** dham1 has joined #openstack-infra | 07:07 | |
| *** onovy has quit IRC | 07:08 | |
| *** uberjay has quit IRC | 07:08 | |
| *** kencjohnston has quit IRC | 07:08 | |
| *** odyssey4me has quit IRC | 07:08 | |
| *** tdasilva has quit IRC | 07:08 | |
| *** vkmc has quit IRC | 07:08 | |
| *** melwitt has quit IRC | 07:08 | |
| *** mugsie has joined #openstack-infra | 07:08 | |
| *** mugsie has quit IRC | 07:08 | |
| *** mugsie has joined #openstack-infra | 07:08 | |
| *** persia has joined #openstack-infra | 07:08 | |
| *** cmurphy has quit IRC | 07:08 | |
| *** uberjay has joined #openstack-infra | 07:09 | |
| *** cmurphy has joined #openstack-infra | 07:09 | |
| *** sree has joined #openstack-infra | 07:10 | |
| *** melwitt has joined #openstack-infra | 07:10 | |
| *** vkmc has joined #openstack-infra | 07:10 | |
| *** vkmc has quit IRC | 07:10 | |
| *** vkmc has joined #openstack-infra | 07:10 | |
| *** melwitt is now known as Guest9054 | 07:11 | |
| *** cshastri has quit IRC | 07:11 | |
| *** onovy has joined #openstack-infra | 07:11 | |
| *** bgmccollum has joined #openstack-infra | 07:12 | |
| *** odyssey4me has joined #openstack-infra | 07:12 | |
| *** tdasilva has joined #openstack-infra | 07:13 | |
| *** kencjohnston has joined #openstack-infra | 07:13 | |
| *** eumel8 has joined #openstack-infra | 07:13 | |
| *** masayukig[m] has quit IRC | 07:14 | |
| *** aspiers[m] has quit IRC | 07:15 | |
| *** ilpianista_ has quit IRC | 07:15 | |
| *** sree has quit IRC | 07:17 | |
| *** armaan has joined #openstack-infra | 07:17 | |
| *** bandini has quit IRC | 07:17 | |
| *** patriciadomin has quit IRC | 07:17 | |
| *** sree has joined #openstack-infra | 07:17 | |
| *** patriciadomin has joined #openstack-infra | 07:17 | |
| *** jamesdenton has quit IRC | 07:18 | |
| *** bandini has joined #openstack-infra | 07:19 | |
| *** armaan has quit IRC | 07:21 | |
| *** sree has quit IRC | 07:25 | |
| *** isviridov_away has quit IRC | 07:30 | |
| *** vaidy has quit IRC | 07:30 | |
| *** huanxie has quit IRC | 07:31 | |
| *** slaweq has joined #openstack-infra | 07:34 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a generic run_handler https://review.openstack.org/526325 | 07:36 |
| *** slaweq has quit IRC | 07:39 | |
| *** makowals has quit IRC | 07:45 | |
| *** vaidy has joined #openstack-infra | 07:46 | |
| *** slaweq has joined #openstack-infra | 07:47 | |
| *** isviridov_away has joined #openstack-infra | 07:47 | |
| *** makowals has joined #openstack-infra | 07:47 | |
| *** sree has joined #openstack-infra | 07:47 | |
| *** slaweq has quit IRC | 07:50 | |
| *** slaweq has joined #openstack-infra | 07:50 | |
| *** slaweq has quit IRC | 07:51 | |
| *** david-lyle has joined #openstack-infra | 07:52 | |
| *** slaweq has joined #openstack-infra | 07:53 | |
| *** shardy has joined #openstack-infra | 07:58 | |
| *** hashar has joined #openstack-infra | 07:58 | |
| chason | ianw Hi o/ | 08:03 |
| chason | ianw http://paste.openstack.org/show/628420/ | 08:03 |
| chason | ianw Can you tell me where this log from? ^^ | 08:04 |
| *** liujiong has joined #openstack-infra | 08:04 | |
| *** florianf has joined #openstack-infra | 08:12 | |
| *** slaweq has quit IRC | 08:12 | |
| *** slaweq has joined #openstack-infra | 08:12 | |
| *** ykarel has quit IRC | 08:13 | |
| *** ykarel has joined #openstack-infra | 08:14 | |
| *** zhurong has joined #openstack-infra | 08:16 | |
| *** huanxie has joined #openstack-infra | 08:16 | |
| *** tesseract has joined #openstack-infra | 08:20 | |
| *** stakeda has quit IRC | 08:20 | |
| *** Hal has joined #openstack-infra | 08:20 | |
| *** dingyichen has joined #openstack-infra | 08:31 | |
| *** shardy is now known as shardy_afk | 08:31 | |
| *** adreznec has quit IRC | 08:32 | |
| *** adreznec has joined #openstack-infra | 08:32 | |
| *** vivsoni has joined #openstack-infra | 08:34 | |
| ianw | chason: one of our zuul build executors | 08:34 |
| *** jpich has joined #openstack-infra | 08:36 | |
| *** hjensas has joined #openstack-infra | 08:36 | |
| *** armaan has joined #openstack-infra | 08:45 | |
| *** ykarel has quit IRC | 08:47 | |
| *** ykarel has joined #openstack-infra | 08:47 | |
| *** rossella_s has joined #openstack-infra | 08:48 | |
| *** shardy_afk is now known as shardy | 08:48 | |
| *** armaan has quit IRC | 08:53 | |
| *** armaan has joined #openstack-infra | 08:55 | |
| *** lucas-afk is now known as lucasagomes | 08:57 | |
| *** armaan has quit IRC | 08:57 | |
| *** amito has joined #openstack-infra | 09:06 | |
| Jeffrey4l | ifra-root, seem the jobs in zuul are blocked for 3 hours. | 09:06 |
| *** bbc1__ has joined #openstack-infra | 09:07 | |
| *** owalsh has quit IRC | 09:10 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a generic run_handler https://review.openstack.org/526325 | 09:11 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement an OpenContainer driver https://review.openstack.org/468753 | 09:11 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a Kubernetes driver https://review.openstack.org/521356 | 09:11 |
| *** andreas_s has quit IRC | 09:11 | |
| *** owalsh has joined #openstack-infra | 09:12 | |
| *** rossella_s has quit IRC | 09:12 | |
| *** andreas_s has joined #openstack-infra | 09:12 | |
| *** jbadiapa has joined #openstack-infra | 09:14 | |
| chason | ianw OK, thanks! | 09:16 |
| *** andreas_s has quit IRC | 09:17 | |
| *** alexchadin has joined #openstack-infra | 09:17 | |
| *** masayukig[m] has joined #openstack-infra | 09:17 | |
| alexchadin | lbragstad: ping | 09:17 |
| *** jbadiapa has quit IRC | 09:21 | |
| *** ykarel_ has joined #openstack-infra | 09:21 | |
| hwoarang | good moring. Could someone look at https://review.openstack.org/#/c/505657/ to unblock bindep+openSUSE tumbleweed please? it's quite old :( | 09:22 |
| *** andreas_s has joined #openstack-infra | 09:23 | |
| *** ykarel has quit IRC | 09:24 | |
| *** dbecker has joined #openstack-infra | 09:24 | |
| *** cshastri has joined #openstack-infra | 09:26 | |
| *** e0ne has joined #openstack-infra | 09:27 | |
| *** andreas_s has quit IRC | 09:28 | |
| slaweq | hello | 09:29 |
| slaweq | I saw that sometimes job openstack-tox-cover is failing in neutron due to reached 30 minutes timeout | 09:29 |
| slaweq | do You know maybe if it's some more "common" problem and we should maybe increase this timeout? I don't think this is related to changes in tested patch | 09:31 |
| slaweq | for example it was like that on: http://logs.openstack.org/73/519573/11/check/openstack-tox-cover/6aef97f/ | 09:31 |
| *** ykarel_ is now known as ykarel|away | 09:35 | |
| *** andreas_s has joined #openstack-infra | 09:37 | |
| *** ykarel|away has quit IRC | 09:40 | |
| frickler | ianw: infra-root: all the passing jobs seem to have been started on ze04, this would confirm my assumption that the other executors need a restart now, too, to pick up the new patches and match zuul master | 09:40 |
| ianw | frickler: hmm, now that's a good point ... i can try that pretty quickly | 09:41 |
| *** yamamoto has quit IRC | 09:41 | |
| frickler | ianw: also https://review.openstack.org/526463 may be related to trouble starting executor | 09:42 |
| *** andreas_s has quit IRC | 09:42 | |
| *** yamamoto has joined #openstack-infra | 09:43 | |
| ianw | i've done ze01, i'll move on one by one | 09:44 |
| *** ilpianista_ has joined #openstack-infra | 09:44 | |
| *** aspiers[m] has joined #openstack-infra | 09:44 | |
| *** numans has quit IRC | 09:45 | |
| *** gmann is now known as gmann_afk | 09:48 | |
| ianw | ok, all restarted | 09:49 |
| frickler | ianw: that's already starting to look much better | 09:49 |
| ianw | frickler: hmm, think we should restart and re-queue everything? | 09:50 |
| *** numans has joined #openstack-infra | 09:50 | |
| frickler | ianw: not sure, I think there's a lot that still could work fine now | 09:50 |
| frickler | maybe rather wait a bit and then ask folk to recheck if they received retry_limit failures | 09:51 |
| *** andreas_s has joined #openstack-infra | 09:51 | |
| ianw | frickler: yeah, nice call, it's rather obvious the executors needed a restart in hindsight | 09:52 |
| *** andreas_s has quit IRC | 09:56 | |
| *** ganso has joined #openstack-infra | 10:00 | |
| *** andreas_s has joined #openstack-infra | 10:03 | |
| *** cuongnv has quit IRC | 10:06 | |
| *** annp has quit IRC | 10:07 | |
| *** andreas_s has quit IRC | 10:12 | |
| *** dtantsur|afk is now known as dtantsur | 10:16 | |
| *** namnh has quit IRC | 10:19 | |
| *** andreas_s has joined #openstack-infra | 10:19 | |
| *** ociuhandu has joined #openstack-infra | 10:20 | |
| *** liujiong has quit IRC | 10:22 | |
| *** andreas_s has quit IRC | 10:24 | |
| *** sdague has joined #openstack-infra | 10:28 | |
| *** tosky has joined #openstack-infra | 10:28 | |
| *** electrofelix has joined #openstack-infra | 10:30 | |
| *** andreas_s has joined #openstack-infra | 10:33 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add n-g-s gerritbot to ironic IRC channel https://review.openstack.org/526318 | 10:34 |
| *** ldnunes has joined #openstack-infra | 10:37 | |
| *** daidv has quit IRC | 10:38 | |
| *** ociuhandu has quit IRC | 10:40 | |
| *** zhurong has quit IRC | 10:40 | |
| *** derekh has joined #openstack-infra | 10:40 | |
| *** danpawlik has quit IRC | 10:42 | |
| *** wolverineav has joined #openstack-infra | 10:42 | |
| *** danpawlik has joined #openstack-infra | 10:43 | |
| slaweq | hi again | 10:46 |
| slaweq | I have another question, this time about logstash | 10:47 |
| slaweq | I have error like "No results There were no results because no indices were found that match your selected time span" every time when I want to do query from last day | 10:48 |
| *** andreas_s has quit IRC | 10:48 | |
| slaweq | do You know maybe what can be a reason of that? | 10:48 |
| *** andreas_s has joined #openstack-infra | 10:48 | |
| *** kjackal has quit IRC | 10:52 | |
| *** kiennt26 has quit IRC | 10:54 | |
| frickler | slaweq: elastic-recheck says "Delay in Elastic Search: Indexing behind by 38 hours", so it seems we haven't processed that data yet | 10:54 |
| frickler | slaweq: probably some infra-root will have to look into this later | 10:55 |
| frickler | I seem to remember there were some issues with it yesterday already | 10:55 |
| *** gibi is now known as giblet | 10:57 | |
| *** andreas_s has quit IRC | 10:57 | |
| slaweq | frickler: thx, I will wait then | 10:58 |
| *** andreas_s has joined #openstack-infra | 10:58 | |
| *** danpawlik has quit IRC | 10:59 | |
| *** danpawlik has joined #openstack-infra | 11:00 | |
| *** rfolco|off is now known as rfolco | 11:01 | |
| *** danpawlik has quit IRC | 11:01 | |
| *** danpawlik has joined #openstack-infra | 11:03 | |
| *** danpawlik has quit IRC | 11:06 | |
| *** ldesimone has joined #openstack-infra | 11:07 | |
| *** ldesimone has quit IRC | 11:07 | |
| *** ldesimone has joined #openstack-infra | 11:07 | |
| *** danpawlik has joined #openstack-infra | 11:10 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/zuul-jobs master: Revert "Add sphinx_python variable to sphinx role and job" https://review.openstack.org/526657 | 11:10 |
| frickler | infra-root: config-core: ^^ this is causing gate failures and I haven't found a fix so far | 11:12 |
| *** ethfci has quit IRC | 11:13 | |
| *** alexchadin has quit IRC | 11:17 | |
| *** openstackgerrit has quit IRC | 11:17 | |
| *** askb_ has quit IRC | 11:18 | |
| *** yamamoto has quit IRC | 11:21 | |
| *** ccamacho has joined #openstack-infra | 11:22 | |
| *** sree has quit IRC | 11:25 | |
| *** links has quit IRC | 11:28 | |
| *** claudiub has joined #openstack-infra | 11:28 | |
| *** jkilpatr has quit IRC | 11:28 | |
| *** udesale has quit IRC | 11:29 | |
| *** dhajare has quit IRC | 11:36 | |
| *** rraja has joined #openstack-infra | 11:38 | |
| AJaeger_ | frickler: thanks for looking into it. So, if "Initialize virtual environment" fails, we now what to fix... | 11:41 |
| *** links has joined #openstack-infra | 11:42 | |
| * AJaeger_ single-core approves the revert to let us move forward | 11:42 | |
| AJaeger_ | The change is still waiting for jobs, might take another 20 mins ;( | 11:44 |
| *** kjackal has joined #openstack-infra | 11:44 | |
| *** dave-mccowan has joined #openstack-infra | 11:45 | |
| *** openstackgerrit has joined #openstack-infra | 11:47 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove freezer-web-ui legacy jobs https://review.openstack.org/525528 | 11:47 |
| *** yamamoto has joined #openstack-infra | 11:49 | |
| *** tesseract has quit IRC | 11:50 | |
| *** tesseract has joined #openstack-infra | 11:51 | |
| *** adisky_ has quit IRC | 11:53 | |
| andreaf | infra-core: this series enables the test matrix for zuulv3 devstack jobs - it needs a second reviewer (d-g / devstack) https://review.openstack.org/#/q/status:open+branch:master+topic:test_matrix_role | 11:53 |
| andreaf | it's the main bit missing (apart from extra log files) for people to start using zuulv3 native dsvm jobs | 11:54 |
| *** kjackal has quit IRC | 11:55 | |
| AJaeger_ | infra-root, something looks odd with zuul looking at grafana: We have 287 nodes available but those are not used directly desplite a large backlog. But looking for the number of jobs, I don't see that large queue in grafana. Please check the health of the queues | 11:57 |
| AJaeger_ | at least the revert god nodes now... | 11:58 |
| *** andreas_s has quit IRC | 12:04 | |
| *** andreas_s has joined #openstack-infra | 12:05 | |
| *** andreas_s has quit IRC | 12:05 | |
| *** andreas_s has joined #openstack-infra | 12:05 | |
| *** janki has quit IRC | 12:07 | |
| *** martinkopec has joined #openstack-infra | 12:09 | |
| *** rhallisey has joined #openstack-infra | 12:12 | |
| *** claudiub has quit IRC | 12:14 | |
| *** rhallisey has quit IRC | 12:16 | |
| *** andreas_s has quit IRC | 12:17 | |
| *** rhallisey has joined #openstack-infra | 12:17 | |
| *** andreas_s has joined #openstack-infra | 12:17 | |
| *** jkilpatr has joined #openstack-infra | 12:23 | |
| *** andreas_s has quit IRC | 12:27 | |
| *** efried is now known as fried_rice | 12:29 | |
| *** rfolco is now known as rfolco_brb | 12:30 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Revert "Add sphinx_python variable to sphinx role and job" https://review.openstack.org/526657 | 12:31 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: WIP: Revert "Revert "Add sphinx_python variable to sphinx role and job"" https://review.openstack.org/526666 | 12:33 |
| *** bobh has joined #openstack-infra | 12:34 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul feature/zuulv3: Update sphinx jobs to use python3 https://review.openstack.org/525690 | 12:34 |
| *** cshastri has quit IRC | 12:34 | |
| AJaeger_ | mordred, pushed also https://review.openstack.org/526668 for testing of the above change ^ | 12:38 |
| AJaeger_ | mordred, dmsimard, pleaes investigate why "Initialize virtual environment" fails as pointed out by frickler earlier | 12:38 |
| AJaeger_ | dmsimard: you have two +2s on your stack starting https://review.openstack.org/#/c/526532/ - once you're aroudn, feel free to merge them one by one (=add +W) and check that everything works fine - ianw asked for babysit in case of unforeseen problems on these. | 12:41 |
| *** andreas_s has joined #openstack-infra | 12:42 | |
| *** salv-orlando has joined #openstack-infra | 12:42 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove freezer-web-ui legacy jobs https://review.openstack.org/525528 | 12:44 |
| *** andreas__ has joined #openstack-infra | 12:48 | |
| *** andreas_s has quit IRC | 12:49 | |
| *** alexchadin has joined #openstack-infra | 12:50 | |
| *** rosmaita has joined #openstack-infra | 12:51 | |
| *** andreas_s has joined #openstack-infra | 12:51 | |
| *** yamamoto has quit IRC | 12:51 | |
| *** andreas__ has quit IRC | 12:52 | |
| *** alexchadin has quit IRC | 12:54 | |
| *** alexchadin has joined #openstack-infra | 12:55 | |
| *** andreas_s has quit IRC | 12:55 | |
| *** andreas_s has joined #openstack-infra | 12:57 | |
| *** makowals has quit IRC | 12:57 | |
| *** makowals has joined #openstack-infra | 12:59 | |
| *** andreas_s has quit IRC | 13:01 | |
| *** bbc1__ has quit IRC | 13:02 | |
| *** rhallisey has quit IRC | 13:04 | |
| *** rhallisey has joined #openstack-infra | 13:04 | |
| *** stephenfin is now known as finucannot | 13:05 | |
| *** sean-k-mooney has joined #openstack-infra | 13:08 | |
| sean-k-mooney | clarkb: fungi QQ do ye still accept changes to enable legacy jobs in the gate or do projects have to swap to in tree zuul configs before they can add new jobs? | 13:09 |
| *** jaosorior has quit IRC | 13:12 | |
| *** jaosorior has joined #openstack-infra | 13:13 | |
| *** mat128 has joined #openstack-infra | 13:16 | |
| pabelanger | we have a large amount of ready locked nodes | 13:18 |
| pabelanger | I think citycloud-sto2 has been wedged for 2 days | 13:18 |
| pabelanger | 45 nodes in sto2 have ready / lock, which I think is our quota | 13:20 |
| pabelanger | I am checking | 13:20 |
| pabelanger | I also see some nodes that are failed / locked | 13:20 |
| *** claudiub has joined #openstack-infra | 13:20 | |
| pabelanger | okay, 1 is building in sto2, hopefully that is enough to start unwedgeing it | 13:21 |
| pabelanger | Shrews: ^when you are set with coffee in hand | 13:21 |
| pabelanger | rax-iad is also showing a lot of locked / ready nodes | 13:23 |
| *** mat128 has quit IRC | 13:24 | |
| *** eumel8 has quit IRC | 13:27 | |
| *** rlandy has joined #openstack-infra | 13:28 | |
| *** jaypipes has joined #openstack-infra | 13:28 | |
| *** trown|outtypewww is now known as trown | 13:29 | |
| *** salv-orlando has quit IRC | 13:29 | |
| *** jaypipes is now known as leakypipes | 13:29 | |
| *** salv-orlando has joined #openstack-infra | 13:30 | |
| *** mat128 has joined #openstack-infra | 13:30 | |
| pabelanger | okay, so I don't think we restarted zuul executors when we restarted scheduler | 13:32 |
| pabelanger | TypeError: unhashable type: 'list' | 13:32 |
| pabelanger | I see that in executor logs | 13:32 |
| pabelanger | I am going to restart ze01 now to confirm latest zuul fixes issues | 13:32 |
| *** salv-orlando has quit IRC | 13:34 | |
| tosky | pabelanger: at least some executors were restarted, because zuul.projects is now a dict :) | 13:34 |
| pabelanger | tosky: yah, I think you are right. looks like ze01 was restarted, I looked in wrong spot | 13:36 |
| *** Wei_Liu has quit IRC | 13:37 | |
| AJaeger_ | pabelanger: ianw restarted all of them, see also his email on openstack-infra with summaries | 13:38 |
| AJaeger_ | sean-k-mooney: yes, for now | 13:38 |
| pabelanger | AJaeger_: okay, thanks. I haven't see that yet. Will read now | 13:39 |
| *** markvoelker has quit IRC | 13:40 | |
| *** mat128 has quit IRC | 13:40 | |
| dmsimard | AJaeger_: do you have a link for the broken job(s) due to the python3 Sphinx patch ? | 13:41 |
| dmsimard | Wait nevermind, saw backlog | 13:41 |
| * dmsimard was reading emails before backlog | 13:41 | |
| *** markvoelker has joined #openstack-infra | 13:43 | |
| AJaeger_ | dmsimard: use https://review.openstack.org/526668 for reproduction | 13:44 |
| *** kiennt26 has joined #openstack-infra | 13:46 | |
| *** tesseract has quit IRC | 13:47 | |
| *** rfolco_brb has quit IRC | 13:49 | |
| *** mriedem has joined #openstack-infra | 13:49 | |
| *** wolverineav has quit IRC | 13:50 | |
| *** tesseract has joined #openstack-infra | 13:50 | |
| *** yamamoto has joined #openstack-infra | 13:52 | |
| *** jkilpatr has quit IRC | 13:53 | |
| AJaeger_ | I think we can give the #status ok again... | 13:54 |
| *** rfolco has joined #openstack-infra | 13:55 | |
| *** yamahata has joined #openstack-infra | 13:56 | |
| AJaeger_ | #status ok The issues have been fixed, Zuul is operating fine again but has a large backlog. You can recheck jobs that failed. | 13:56 |
| *** AJaeger_ is now known as AJaeger | 13:56 | |
| AJaeger | #status ok The issues have been fixed, Zuul is operating fine again but has a large backlog. You can recheck jobs that failed. | 13:56 |
| openstackstatus | AJaeger: sending ok | 13:56 |
| *** AJaeger is now known as AJaeger_ | 13:56 | |
| *** wolverineav has joined #openstack-infra | 13:57 | |
| *** yamamoto has quit IRC | 13:58 | |
| *** makowals has quit IRC | 13:58 | |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 13:59 | |
| -openstackstatus- NOTICE: The issues have been fixed, Zuul is operating fine again but has a large backlog. You can recheck jobs that failed. | 13:59 | |
| pabelanger | AJaeger_: we still have some issues, but holding off until jeblair or Shrews looks. Just sent an email to ML | 13:59 |
| pabelanger | but, I think we'll need to restart scheduler again | 13:59 |
| pabelanger | to clean up locked ready nodes | 13:59 |
| *** ramishra has quit IRC | 13:59 | |
| *** alexchadin has quit IRC | 14:00 | |
| Shrews | pabelanger: looking, but i have no idea how we got into this state. citycloud-sto2 is paused trying to get a node (which it can't do). might just need to restart the launcher for now | 14:00 |
| AJaeger_ | pabelanger: but zuul is not unstable anymore, so that is fixed - and we can restart the scheduler anytime with another #status. Or do you think we need to still alert? | 14:01 |
| *** yamamoto has joined #openstack-infra | 14:01 | |
| Shrews | pabelanger: looks like nodepool launcher actually has the ready nodes locked. i'm unclear why | 14:01 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: update zuul cloned repo directory for zuulv3 https://review.openstack.org/526546 | 14:02 |
| pabelanger | Shrews: okay, I'll let you restart when you are ready. | 14:02 |
| openstackstatus | AJaeger: finished sending ok | 14:02 |
| pabelanger | AJaeger_: I am not sure, TBH. But need some more coffee :) | 14:02 |
| dmsimard | AJaeger_: just making sure, should we land that revert ASAP ? | 14:03 |
| Shrews | pabelanger: restarted nl02. i'd really like to understand how it gets into this state, but i'm having trouble tracking down how it happens | 14:03 |
| *** baoli has joined #openstack-infra | 14:04 | |
| dmsimard | AJaeger_: nevermind the revert already landed | 14:04 |
| * dmsimard properly wakes up | 14:04 | |
| *** makowals has joined #openstack-infra | 14:04 | |
| *** mat128 has joined #openstack-infra | 14:05 | |
| *** mat128 has quit IRC | 14:05 | |
| dmsimard | AJaeger_: so build-openstack-releasenotes is not self-tested then... sigh | 14:05 |
| *** Goneri has joined #openstack-infra | 14:08 | |
| *** kgiusti has joined #openstack-infra | 14:09 | |
| pabelanger | Shrews: okay, looks to have cleared it out. Yah, would be good to understand how that happened | 14:13 |
| pabelanger | Shrews: how did you know nodepool-launcher was holding the lock? | 14:13 |
| pabelanger | Shrews: trying to see if that is the same case with rax-iad / inap nodes in ready / locked for 12+hrs | 14:14 |
| Shrews | pabelanger: output of the 'dump' command in zk-shell. i saw that the session holding one of the ready node locks also had a launcher ID ephemeral node associated with it | 14:15 |
| Shrews | pabelanger: i did not restart nl01. you may do so if you feel it would help | 14:15 |
| sean-k-mooney | AJaeger_: ok cool in that case ill propose a pathch to add functional jobs to os-vif's experimental pipline but ill try and convert to it over to zuul v3 in repo job before queens is released | 14:17 |
| pabelanger | Shrews: okay, I can restart nl01 also | 14:19 |
| pabelanger | Shrews: okay, that appears to have released them | 14:21 |
| pabelanger | Shrews: Oh, I might have spoke too soon | 14:22 |
| pabelanger | | 0001287466 | inap-mtl01 | ubuntu-xenial | 3d608769-9f89-4ff6-910b-a3ef1fc5f10c | 198.72.124.218 | | ready | 00:14:17:37 | locked | | 14:22 |
| pabelanger | so, how can I tell using zk-shell, what is holding that lock? | 14:22 |
| pabelanger | I don't think it is 'get nodepool/nodes/0001287466' | 14:23 |
| openstackgerrit | Dan Prince proposed openstack-infra/reviewday master: Fix reviewday URLs https://review.openstack.org/526686 | 14:23 |
| *** esberglu has joined #openstack-infra | 14:24 | |
| *** dansmith is now known as superdan | 14:25 | |
| *** mat128 has joined #openstack-infra | 14:35 | |
| Shrews | pabelanger: ok, i think clarkb's fix here https://review.openstack.org/526234 would have caught this particular problem. there's another fix that's needed, but i'll add that one | 14:36 |
| *** links has quit IRC | 14:37 | |
| *** ramishra has joined #openstack-infra | 14:38 | |
| pabelanger | looking | 14:39 |
| AJaeger_ | dmsimard: most of the jobs are not self-tested ;( we're not there yet. Fortunately a depends-on helps - if you know what to test ;( | 14:39 |
| Shrews | oh, clarkb's fix is enough actually. good | 14:40 |
| *** martinkopec has quit IRC | 14:42 | |
| pabelanger | okay great | 14:42 |
| AJaeger_ | Shrews: do you want to fix clarkb's change so that we can merge it and move forward? | 14:43 |
| AJaeger_ | or wait for clarkb to wake up? | 14:43 |
| Shrews | pabelanger: the command is 'dump'. you'll see the session holding /nodepool/nodes/0001287466/lock/ecdfcddcda21407890a07a9ef651309e__lock__0000007098 also lists the ephemeral node /nodepool/launchers/nl01.openstack.org-13040-PoolWorker.inap-mtl01-main, which is created by nl01 (per the name) | 14:45 |
| Shrews | pabelanger: you'll probably find it useful to first do a 'script out.txt' before using zk-shell. that way you can more easily grep the output | 14:46 |
| *** yamahata has quit IRC | 14:47 | |
| Shrews | AJaeger_: let's wait. there's an issue with his parent change, too | 14:47 |
| mordred | yay - my internet is broken so I get to tether to my phone! | 14:48 |
| Shrews | mordred: OMGSOLUCKY | 14:48 |
| *** pbourke has quit IRC | 14:50 | |
| * Shrews makes mooor coffeeee | 14:51 | |
| Guest9054 | FYI http://zuul.openstack.org/ is no longer redirecting to http://zuulv3.openstack.org/ | 14:52 |
| *** Guest9054 is now known as melwitt | 14:52 | |
| *** dhill_ has quit IRC | 14:53 | |
| *** ihrachys has joined #openstack-infra | 14:54 | |
| *** pbourke has joined #openstack-infra | 14:55 | |
| *** rcernin has quit IRC | 14:55 | |
| *** Hal has quit IRC | 14:58 | |
| *** Hal has joined #openstack-infra | 14:58 | |
| *** dhill_ has joined #openstack-infra | 14:59 | |
| *** ldnunes has quit IRC | 15:00 | |
| fungi | melwitt: huh, i wonder how that got undone. maybe zuul.o.o somehow managed to get puppet reapplied | 15:01 |
| fungi | it's still in the emergency disable list | 15:02 |
| *** andreas_s has joined #openstack-infra | 15:02 | |
| fungi | oh! the old zuul.o.o server is completely unresponsive. nice! | 15:02 |
| *** trown is now known as trown|brb | 15:02 | |
| melwitt | heh | 15:02 |
| *** slaweq has quit IRC | 15:02 | |
| mordred | dmsimard: http://logs.openstack.org/68/526668/1/check/build-openstack-releasenotes/14e8844/ara/ shows "Initialize virtual environment" to be skipped ... | 15:03 |
| fungi | wonder if we should just repoint the old dns name at the zuulv3.o.o addresses | 15:03 |
| mordred | dmsimard: but when I run that logic locally, it's totally working | 15:03 |
| *** slaweq has joined #openstack-infra | 15:03 | |
| dmsimard | mordred: yeah, same here or I wouldn't have submitted it :D | 15:03 |
| fungi | infra-root: ^ should i attempt to resuscitate the old zuul.openstack.org instance so we get our redirect back, or just move its dns entries to the v3 server? | 15:04 |
| dmsimard | mordred: I'm actually working to debug it through https://review.openstack.org/#/c/526666/1 | 15:04 |
| *** salv-orlando has joined #openstack-infra | 15:04 | |
| mordred | fungi: I'd vote for just moving the dns entries | 15:04 |
| fungi | i'm leaning that direction as well | 15:04 |
| pabelanger | yah, DNS sounds much easier | 15:04 |
| dmsimard | mordred: haven't submitted anything yet, but I want to submit a tmp patch just to add some debug tasks and stat tasks to see if the files are there or not | 15:04 |
| mordred | dmsimard: cool | 15:04 |
| *** Hal has quit IRC | 15:05 | |
| *** trown|brb is now known as trown | 15:06 | |
| *** hongbin has joined #openstack-infra | 15:07 | |
| fungi | we have 5 new trouble tickets for our ci tenant in rackspace, mostly having to do with dfw migrations | 15:07 |
| fungi | the first ticket says they had to force migrate stackalytics.o.o which also seems unresponsive to me | 15:09 |
| *** hemna_ has joined #openstack-infra | 15:09 | |
| AJaeger_ | fungi, move | 15:09 |
| mriedem | zuul says that 330285 is in both the check and gate queues? | 15:10 |
| *** signed8bit has joined #openstack-infra | 15:10 | |
| mriedem | is that normal or side effect of other problems? | 15:10 |
| AJaeger_ | mordred: wokring on releasenotes for openstack-doc-tools or python-troveclient as well? | 15:11 |
| AJaeger_ | mordred: releasenotes job is the one broken | 15:11 |
| *** ramishra has quit IRC | 15:11 | |
| *** ldnunes has joined #openstack-infra | 15:11 | |
| *** rraja has quit IRC | 15:11 | |
| *** eumel8 has joined #openstack-infra | 15:11 | |
| AJaeger_ | mriedem: if you +A - and have a +1 but issue(d) a recheck - yes, then this is normal | 15:12 |
| fungi | mriedem: it can happen if someone rechecks a change which was in the gate after a zuul restart but before we reenqueue our dumped lists of changes | 15:12 |
| fungi | there are a few circumstances that can result in that, right | 15:12 |
| fungi | looks like the reason stackalytics.o.o isn't up is that it hit filesystem issues on restarting after the reboot they performed in order to migrate it | 15:15 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/526695 | 15:16 |
| *** hemna_ has quit IRC | 15:18 | |
| *** Apoorva has joined #openstack-infra | 15:22 | |
| *** Wei_Liu has joined #openstack-infra | 15:23 | |
| *** Hal has joined #openstack-infra | 15:24 | |
| *** armax has joined #openstack-infra | 15:24 | |
| *** rraja has joined #openstack-infra | 15:27 | |
| *** hemna_ has joined #openstack-infra | 15:27 | |
| *** claudiub has quit IRC | 15:28 | |
| *** Apoorva has quit IRC | 15:28 | |
| *** rbrndt has joined #openstack-infra | 15:29 | |
| fungi | after asking fsck to proceed in fixing filesystem errors the instance eventually booted to a login prompt on the console but never set up networking. successive soft reboots didn't seem to solve it. now after hard rebooting it seems to no longer be possible to connect to the console for that instance in the provider's dashboard | 15:29 |
| fungi | we may need to delete and rebuild it from scratch, which i guess can happen as part of the xenial upgrades next week | 15:30 |
| fungi | since that server never went into production, i'm not too concerned about leaving it down for now | 15:30 |
| pabelanger | wfm | 15:31 |
| pabelanger | agree to rebuild as xenial, and maybe finally roll it into productoin | 15:32 |
| openstackgerrit | Dan Prince proposed openstack-infra/reviewday master: Add support for custom namespaces https://review.openstack.org/526707 | 15:32 |
| fungi | on a second hard reboot i got the console back again. will give it one more chance before i move on to looking into other servers | 15:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Expose serial keyword to base-test playbook https://review.openstack.org/526708 | 15:32 |
| fungi | but yeah, still acting like it's not bringing up the network successfully | 15:32 |
| *** makowals has quit IRC | 15:34 | |
| *** Goneri has quit IRC | 15:34 | |
| *** salv-orlando has quit IRC | 15:34 | |
| *** salv-orlando has joined #openstack-infra | 15:35 | |
| fungi | #status log rebooted zuul.openstack.org after it became unresponsive in what looked like a host migration activity | 15:36 |
| openstackstatus | fungi: finished logging | 15:36 |
| fungi | infra-root: melwitt: ^ solved with just a nova reboot for now | 15:36 |
| *** Goneri has joined #openstack-infra | 15:36 | |
| fungi | #status log the current stackalytics.openstack.org instance is not recovering via reboot after a failed host migration, and will likely need to be deleted and rebuilt when convenient | 15:37 |
| openstackstatus | fungi: finished logging | 15:38 |
| jeblair | i killed the zuul-server process which had started | 15:38 |
| fungi | oh, thanks jeblair. i thought we had disabled the initscript on it but i guess not | 15:38 |
| jeblair | probably wouldn't have done anything without any mergers or launchers anyway | 15:38 |
| fungi | sounds likely | 15:39 |
| frickler | jeblair: after the zuul restart it seems now openstackclient-plugin-jobs is suddenly running on python-designateclient, though that definition has been in place for quite some time. not sure whether that was to be expected with the new patches, but might be interesting | 15:39 |
| *** martinkopec has joined #openstack-infra | 15:39 | |
| *** salv-orlando has quit IRC | 15:40 | |
| melwitt | cool fungi | 15:40 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci master: Update reviewday project list https://review.openstack.org/526712 | 15:40 |
| melwitt | thanks | 15:40 |
| jeblair | frickler: so it's doing the right thing now? i thought the version we were previously running had all the fixes related to that; i wasn't aware of that problem. | 15:42 |
| fungi | #status log elasticsearch02, elasticsearch04 and review-dev are scheduled to be rebooted as part of a provider host migration at 2017-12-11 at 04:00 UTC | 15:43 |
| openstackstatus | fungi: finished logging | 15:43 |
| fungi | #status log zuul.openstack.org is scheduled to be rebooted as part of a provider host migration at 2017-12-12 at 04:00 UTC | 15:43 |
| openstackstatus | fungi: finished logging | 15:43 |
| frickler | jeblair: I wasn't aware of that, either, I just noticed the new check after rechecking a patch. seems it's also failing, but that's a different story https://review.openstack.org/526410 | 15:44 |
| frickler | jeblair: and looking who had created that check, I saw that it is in the openstackclient repo since october | 15:44 |
| fungi | mordred: During a recent audit of our Rackspace Internal Cloud Accounts we discovered this account (DDI#610275) is assigned to Monty Taylor who is no longer with Rackspace. Since there is no current owner for this account we are contacting their last documented manager. Could you look into this account to determine whether these servers are currently being used. If this is the case, please update the | 15:45 |
| fungi | contact information for this account. | 15:45 |
| fungi | not sure what we need to do there... it looks like the contact information is still relatively reasonable | 15:46 |
| *** shardy has quit IRC | 15:47 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:47 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul feature/zuulv3: WIP: Git driver https://review.openstack.org/525614 | 15:47 |
| fungi | that's for the openstackci tenant. granted the contact info for our openstackjenkins tenant is still pvo | 15:48 |
| fungi | maybe he wants to be the primary account contact for both of those? | 15:48 |
| mordred | fungi: yah - probably shifting them to pvo is a good idea | 15:49 |
| dmsimard | mordred: is it worth cutting a 1.25.1 with https://review.openstack.org/#/c/526127/ merged ? | 15:50 |
| dmsimard | argh, maybe not on a friday | 15:50 |
| dmsimard | but I'm sure it would yield a very beneficial impact on our nodepool with cloud providers with floating IPs .. | 15:50 |
| dmsimard | btw I didn't get to circle back on the issue *without* caching enabled but I added whatever I found in the bug | 15:51 |
| *** felipemonteiro__ has joined #openstack-infra | 15:51 | |
| *** eumel8 has quit IRC | 15:52 | |
| pabelanger | dmsimard: Are you seeing any api-timeout messages in nodepool logs now? | 15:53 |
| *** slaweq has quit IRC | 15:53 | |
| frickler | ah, seems it is failing also due to the restart ... with_items: "{{ zuul.projects | selectattr('required') | map(attribute='name') | list }}" | 15:54 |
| dmsimard | pabelanger: didn't really check but we haven't had any problems since applying cache/api-timeout and the shade patch | 15:54 |
| pabelanger | dmsimard: okay, that is good news | 15:54 |
| pabelanger | wasn't sure if you'd see anything in debug logs | 15:55 |
| *** kiennt26 has quit IRC | 15:55 | |
| dmsimard | pabelanger: if anything, it probably helps the load on rdo cloud tremendously | 15:55 |
| pabelanger | yup | 15:55 |
| *** felipemonteiro_ has quit IRC | 15:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove tripleo-ci patches from infra IRC channel https://review.openstack.org/526716 | 15:56 |
| mordred | dmsimard: yes - definitely worth cutting a release ... | 15:57 |
| *** rfolco is now known as rfolco|brb | 15:57 | |
| AJaeger_ | EmilienM, mwhahaha , heads up on 526716 ^ | 15:57 |
| mwhahaha | AJaeger_: works for me | 15:58 |
| AJaeger_ | mordred: thanks, I wondered as well for some time... | 15:58 |
| openstackgerrit | Dan Prince proposed openstack-infra/reviewday master: Fix pep8 test https://review.openstack.org/526717 | 15:58 |
| mordred | dmsimard: there is one more patch that's going through the gate now - as soon as it lands I think we'll be ready for a release - it'll be a 1.26 as it'll also contain thelast its of python-ironicclient-ectomy | 15:58 |
| EmilienM | AJaeger_: you don't like our spams? :P | 15:59 |
| dmsimard | mordred: keep that release for monday? :D | 15:59 |
| mordred | AJaeger_, mwhahaha: I also pushed up https://review.openstack.org/526715 to governance | 15:59 |
| EmilienM | or it's monty | 15:59 |
| EmilienM | he doesn't like us anymore :( | 15:59 |
| * EmilienM can troll, it's friday | 15:59 | |
| mwhahaha | mordred: sounds good | 15:59 |
| EmilienM | after years, tripleo-ci finally moved to tripleo project ! :) | 16:00 |
| mordred | EmilienM: anymore? | 16:01 |
| * mordred can troll on fridays too ... | 16:01 | |
| EmilienM | ahaha | 16:01 |
| AJaeger_ | enough votes for 526716 -> +2A | 16:01 |
| AJaeger_ | that was qucik ;) | 16:02 |
| mwhahaha | see we're responsive! | 16:03 |
| mwhahaha | dat communication | 16:03 |
| AJaeger_ | ;) | 16:04 |
| clarkb | looks like I need to get my nodepool patches passings tests | 16:05 |
| clarkb | I'll work on getting that done and then will attempt to review specs and mailman things but still somewhat of a zombie today | 16:05 |
| *** vhosakot has joined #openstack-infra | 16:07 | |
| dmsimard | AJaeger_: may I take over https://review.openstack.org/#/c/526666/ to experiment ? | 16:07 |
| mordred | dmsimard: I'd say go for it - AJaeger_ has plenty of stackalytics points already | 16:10 |
| dmsimard | lol | 16:11 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Clarify terminology around node request locks https://review.openstack.org/526233 | 16:12 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Handle race between handler and request cleanup https://review.openstack.org/526234 | 16:12 |
| mordred | clarkb: https://review.openstack.org/#/c/526715/ proably needs a +1 from you as well | 16:12 |
| *** jaosorior has quit IRC | 16:13 | |
| clarkb | mordred: done | 16:13 |
| *** xarses has quit IRC | 16:14 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul feature/zuulv3: Git driver https://review.openstack.org/525614 | 16:14 |
| *** weshay|ruck is now known as weshay|ruck|MOD | 16:15 | |
| AJaeger_ | dmsimard: it was meant for others to take over;) So, please do! Next time just do it ;) | 16:15 |
| *** rraja has quit IRC | 16:16 | |
| AJaeger_ | mordred: Can I share those points somehow? Or redeem them? ;) | 16:16 |
| mordred | AJaeger_: if you gain enough of them, you can redeem them for a lovely box of burnout. :) | 16:17 |
| * AJaeger_ hates pandora's boxes | 16:18 | |
| andreykurilin | hi folks! Why some of job statuses are not displayed in the top table of the gerrit and there is only ability to see them in the comments? for example, "RETRY_LIMIT" | 16:20 |
| dmsimard | andreykurilin: have an example ? | 16:20 |
| *** Hal has quit IRC | 16:20 | |
| andreykurilin | dmsimard: sure https://review.openstack.org/#/c/526569/ legacy-tempest-dsvm-neutron-src job | 16:20 |
| *** Hal has joined #openstack-infra | 16:20 | |
| AJaeger_ | andreykurilin, dmsimard our javascript filter in system-config needs updating for some of these new ones | 16:20 |
| dmsimard | AJaeger_: oh, yeah, that. I remember that. | 16:21 |
| dmsimard | http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/gerrit/hideci.js I think ? | 16:21 |
| fungi | yep, that's the one | 16:22 |
| *** MasterOfBugs has joined #openstack-infra | 16:22 | |
| *** pramodrj07 has joined #openstack-infra | 16:22 | |
| dmsimard | I remember looking at it to fix it but I couldn't figure out how to test it locally before submitting | 16:22 |
| clarkb | dmsimard: you can use your browser's dev tools to replcae the js content and reload (I say that like I actually know how to do it, btu really I fumble around in there any time I have to debug web things) | 16:23 |
| dmsimard | clarkb: yeah that's what I did, but when reloading it would reload the real version, not the modified one | 16:23 |
| dmsimard | clarkb: so I tried adding stuff like breakpoints, modify code, and then run it, but that didn't work either | 16:24 |
| dmsimard | I tried in chrome fwiw, haven't tried with firefox | 16:24 |
| fungi | this one time, sdague explained to me how to do it. but i didn't take good notes and now i've forgotten | 16:24 |
| mgagne | currently investigating an issue with inap-mtl01 region | 16:24 |
| pabelanger | mgagne: thanks | 16:24 |
| fungi | thanks for the heads up, mgagne! | 16:24 |
| dmsimard | I'm sure I could figure it out, but I need to be very motivated for javascript stuff to keep me interested | 16:24 |
| *** vhosakot has quit IRC | 16:25 | |
| *** caphrim007 has quit IRC | 16:25 | |
| fungi | dmsimard: just keep reminding yourself that the next generation of programmers will be writing control systems for nuclear power plants in javascript, and that it's better we find the bugs before they do | 16:25 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Clarify terminology around node request locks https://review.openstack.org/526233 | 16:25 |
| *** Hal has quit IRC | 16:25 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Handle race between handler and request cleanup https://review.openstack.org/526234 | 16:25 |
| dmsimard | fungi: I really don't know why javascript was the language people seem to have settled on to manage everything from frontend to backend to shell | 16:26 |
| dmsimard | whenever I see a shell script that involves "npm", I cry tears of sadness | 16:27 |
| *** mat128 has quit IRC | 16:27 | |
| fungi | yeah, me^2 | 16:27 |
| clarkb | I get the desire to only have one language to keep track of | 16:27 |
| *** baoli has quit IRC | 16:27 | |
| *** felipemonteiro__ has quit IRC | 16:28 | |
| *** Hal has joined #openstack-infra | 16:28 | |
| *** felipemonteiro__ has joined #openstack-infra | 16:28 | |
| dmsimard | javascript just moves so fast.. every year there's a new favorite framework.. jquery, angular, react, vue, whatever | 16:28 |
| *** hashar has quit IRC | 16:29 | |
| *** rfolco|brb is now known as rfolco | 16:32 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Add system-required for openstack/self-healing-sig https://review.openstack.org/525271 | 16:32 |
| fungi | dmsimard: i replied to your questions about 526148 | 16:37 |
| *** kjackal has joined #openstack-infra | 16:38 | |
| *** baoli has joined #openstack-infra | 16:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Expose serial keyword to base-test playbook https://review.openstack.org/526708 | 16:39 |
| dmsimard | fungi: "only config projects can define base jobs" ? | 16:40 |
| dmsimard | fungi: I'm not sure I follow, what's preventing me from defining what I'd call foo-base and then parenting a bunch of jobs to it ? | 16:40 |
| fungi | dmsimard: the actual job named "base" | 16:40 |
| dmsimard | fungi: there's code in zuul that recognizes that name ? | 16:41 |
| fungi | there's configuration in our ci system which recognizes that name as the name of our base job | 16:41 |
| fungi | and zuul has a sanity check to prevent any old repo from redefining whatever name is set as the base job | 16:42 |
| dmsimard | interesting, I didn't realize that. | 16:42 |
| fungi | dmsimard: check out the exceptions in the comments zuul left on https://review.openstack.org/526140 | 16:43 |
| dmsimard | oh, wait, that makes sense -- it's what makes "base" the implicit base job if you don't specify a parent ? | 16:43 |
| fungi | yeah | 16:43 |
| fungi | "Base jobs must be defined in config projects" is the actual error zuul reports on that change | 16:43 |
| fungi | discussing in #zuul, our options were to either configure zul to skip loading any configuration from that repo, or set it as a config repo shadowing project-config | 16:44 |
| fungi | er, configure zuul | 16:44 |
| fungi | sorry zul! | 16:44 |
| *** kjackal has quit IRC | 16:46 | |
| fungi | dmsimard: it may actually not so much be the name of the job as the "parent: null" in it | 16:47 |
| fungi | in fact, i suspect it's more of the latter | 16:48 |
| dmsimard | fungi: parent:null just says that the job doesn't have a parent job | 16:48 |
| fungi | though beyond that, we need teh shadowing because the job name is the same | 16:48 |
| dmsimard | fungi: you can do that anywhere to get rid of the implicit base parent | 16:48 |
| fungi | i didn't realize you could get rid of the implicit base parent anywhere besides config projects | 16:48 |
| fungi | though your base parent had to be a job from a config project | 16:49 |
| *** niedbalski has quit IRC | 16:49 | |
| *** niedbalski has joined #openstack-infra | 16:50 | |
| fungi | at any rate, the upshot is that we want a usable base job provided by the stdlib when we release, which means we need it in a separate repo from the rest of the stdlib either so we can ignore it or set it as a config repo, and the latter gets it a little bit of extra validation | 16:50 |
| fungi | but by all means ask questions, it seems to me like something likely to expose corner case problems we haven't turned up yet | 16:51 |
| *** d0ugal has quit IRC | 16:52 | |
| fungi | close review definitely needed there | 16:52 |
| *** d0ugal has joined #openstack-infra | 16:54 | |
| *** d0ugal has quit IRC | 16:54 | |
| *** d0ugal has joined #openstack-infra | 16:54 | |
| *** david-lyle has quit IRC | 16:57 | |
| *** liusheng has quit IRC | 16:57 | |
| *** liusheng has joined #openstack-infra | 16:57 | |
| fungi | pabelanger: i replied to your question on 526510 | 16:58 |
| pabelanger | thanks! looking | 16:59 |
| *** baoli_ has joined #openstack-infra | 16:59 | |
| pabelanger | fungi: +3 | 16:59 |
| zul | fungi: sorry im not so easily configurable | 16:59 |
| pabelanger | thanks | 16:59 |
| *** andreas_s has quit IRC | 17:00 | |
| fungi | zul: some consider that a feature! fewer configuration options leads to less user confusion, after all | 17:00 |
| *** andreas_s has joined #openstack-infra | 17:00 | |
| *** baoli has quit IRC | 17:01 | |
| *** slaweq has joined #openstack-infra | 17:03 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Handle race between handler and request cleanup https://review.openstack.org/526234 | 17:03 |
| mgagne | issue should be fixed in inap-mtl01 region. I'm sure there are a couple of failed instances, hopefully nodepool will clean them up shortly | 17:05 |
| *** andreas_s has quit IRC | 17:05 | |
| *** xarses has joined #openstack-infra | 17:07 | |
| fungi | thanks again mgagne! | 17:07 |
| *** felipemonteiro_ has joined #openstack-infra | 17:07 | |
| *** rosmaita has quit IRC | 17:09 | |
| *** slaweq_ has joined #openstack-infra | 17:09 | |
| *** slaweq has quit IRC | 17:10 | |
| *** slaweq_ has quit IRC | 17:10 | |
| *** rosmaita has joined #openstack-infra | 17:11 | |
| *** felipemonteiro__ has quit IRC | 17:11 | |
| *** slaweq has joined #openstack-infra | 17:11 | |
| dmsimard | fungi: yeah, it's just a bit confusing :) | 17:12 |
| *** derekh has quit IRC | 17:12 | |
| *** lucasagomes is now known as lucas-afk | 17:12 | |
| *** jpich has quit IRC | 17:12 | |
| dmsimard | FWIW we should probably retire openstack-infra/openstack-zuul-roles | 17:13 |
| dmsimard | unless we want to make use of it in the future | 17:13 |
| *** gyee has joined #openstack-infra | 17:14 | |
| fungi | yeah, i think we're likely to retire that | 17:17 |
| *** Hal has quit IRC | 17:17 | |
| *** panda is now known as panda|off | 17:18 | |
| *** baoli_ has quit IRC | 17:20 | |
| *** dtantsur is now known as dtantsur|afk | 17:24 | |
| *** baoli has joined #openstack-infra | 17:27 | |
| *** tesseract has quit IRC | 17:29 | |
| *** dhill_ has quit IRC | 17:29 | |
| *** jkilpatr has joined #openstack-infra | 17:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove tripleo-ci patches from infra IRC channel https://review.openstack.org/526716 | 17:35 |
| *** dhill_ has joined #openstack-infra | 17:35 | |
| *** david-lyle has joined #openstack-infra | 17:36 | |
| *** eumel8 has joined #openstack-infra | 17:38 | |
| *** jkilpatr has quit IRC | 17:39 | |
| *** caphrim007 has joined #openstack-infra | 17:40 | |
| fungi | clarkb: ^ it's probably time to revisit whether tripleo-ci should become a tripleo deliverable instead of an infra one, with the final decommissioning of the tripleo cloud from nodepool | 17:40 |
| *** fried_rice is now known as fried_rolls | 17:41 | |
| fungi | mordred: dmsimard: AJaeger_: ^ since you also reviewed that change and thought it wasn't an infra deliverable | 17:41 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul feature/zuulv3: Git driver https://review.openstack.org/525614 | 17:41 |
| AJaeger_ | fungi: https://review.openstack.org/526715 - mordred agreed with us here ;) | 17:44 |
| fungi | ahh, excellent. i missed that got proposed already | 17:44 |
| *** signed8bit is now known as signed8bit_Zzz | 17:45 | |
| *** dhill_ has quit IRC | 17:45 | |
| *** signed8bit_Zzz has quit IRC | 17:45 | |
| *** smatzek has joined #openstack-infra | 17:47 | |
| *** dhill_ has joined #openstack-infra | 17:47 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove docs-draft vhost from static.o.o https://review.openstack.org/526510 | 17:47 |
| *** pblaho has quit IRC | 17:48 | |
| pabelanger | mgagne: looks like we are still seeing some failure to alloate network errors in inap. | 17:49 |
| pabelanger | anything I can share? | 17:49 |
| mgagne | pabelanger: at least one UUID? will take a look right now | 17:49 |
| pabelanger | mgagne: 70272c45-4903-4f1b-9709-cf9106cde045 | 17:49 |
| clarkb | fungi: yup I and mwhahaha have +1'd the governance change for that | 17:50 |
| fungi | i've rollcall +1'd it now as well | 17:50 |
| *** salv-orlando has joined #openstack-infra | 17:50 | |
| fungi | on the expectation that the tripleo cloud will be gone from our nodepool configs rsn | 17:51 |
| *** claudiub has joined #openstack-infra | 17:51 | |
| *** jkilpatr has joined #openstack-infra | 17:53 | |
| *** dhill_ has quit IRC | 17:56 | |
| *** dhill_ has joined #openstack-infra | 17:56 | |
| mgagne | pabelanger: I suspect leaked Neutron ports. I will see if I can find them and clean them up. | 17:56 |
| pabelanger | mgagne: ack! Thanks for looking | 17:57 |
| *** salv-orlando has quit IRC | 18:01 | |
| *** e0ne has quit IRC | 18:01 | |
| *** xarses has quit IRC | 18:02 | |
| pabelanger | dmsimard: mordred: https://review.openstack.org/526708/ is a follow change to base-test now that the shared ansible_host connection in zuul is live. Would appreciate feedback. I know there is a few way to deal with the issue, but in the past I've used serial keyword to do so | 18:02 |
| *** yamamoto has quit IRC | 18:02 | |
| *** openstackgerrit has quit IRC | 18:03 | |
| mgagne | pabelanger: deleting 104 orphan/zombie ports | 18:05 |
| *** jkilpatr has quit IRC | 18:06 | |
| pabelanger | mgagne: ack | 18:06 |
| pabelanger | mgagne: do you think that is something we need to look at on our side? eg: shade | 18:06 |
| *** [HeOS] has quit IRC | 18:06 | |
| fungi | wow. that's unfortunate. anything we should have been more diligent about cleaning up in nodepool? | 18:06 |
| fungi | yeah, what pabelanger asked | 18:06 |
| mgagne | pabelanger: I think it was caused by the issue we had earlier, probably cleanup step got skipped by Nova | 18:07 |
| pabelanger | okay, good to know | 18:07 |
| fungi | we used to do a lot of similar manual cleanup of leaked ports in certain providers due to nova/neutron interaction races where boot failures could leave requested neutron ports hanging around indefinitely | 18:07 |
| mgagne | pabelanger: has rabbitmq clustering been of any help for you? =) | 18:07 |
| *** thorre has quit IRC | 18:11 | |
| *** thorre has joined #openstack-infra | 18:12 | |
| pabelanger | mgagne: specific to what? I wasn't aware of any rabbitmq issues in the context of inap | 18:14 |
| mgagne | just a general complain about rabbitmq clustering =) | 18:14 |
| pabelanger | ah | 18:15 |
| mgagne | rant* | 18:15 |
| pabelanger | yah, we just have a single rabbitmq in infracloud today | 18:15 |
| mgagne | =) | 18:15 |
| *** r-daneel has joined #openstack-infra | 18:15 | |
| pabelanger | but, nothing in infracloud is HA :) | 18:15 |
| fungi | infra-root: i've removed the old docs-draft logical volume from static.o.o and am trying to decide... should we add the ~1.5tb that freed to the logs volume, the tarballs volume, divide it up between them somehow, or leave it unallocated for now? | 18:16 |
| *** sdague has quit IRC | 18:16 | |
| *** yamamoto has joined #openstack-infra | 18:16 | |
| fungi | block utilization on /srv/static/tarballs is 204G used out of 345G so 60% | 18:17 |
| fungi | block utilization on /srv/static/logs is 10T used out of 12T or 85%, with 541M out of 768M inodes used or 71% (inode capacity for ext4 will grow proportionally with block count) | 18:20 |
| clarkb | fungi: maybe give tarballs a couple hundred gigs then the rest to logs? | 18:20 |
| *** david-lyle has quit IRC | 18:21 | |
| fungi | like bump tarballs up to 0.5t and logs to 13t? | 18:21 |
| *** yamamoto has quit IRC | 18:21 | |
| clarkb | ya | 18:21 |
| *** slaweq has quit IRC | 18:22 | |
| *** jkilpatr has joined #openstack-infra | 18:23 | |
| *** rbrndt has quit IRC | 18:23 | |
| *** slaweq has joined #openstack-infra | 18:25 | |
| *** electrofelix has quit IRC | 18:28 | |
| *** slaweq has quit IRC | 18:29 | |
| pabelanger | +1 | 18:29 |
| fungi | i'll give others a little more time to notice and chime in before committing, since this is relatively irreversible and we don't get many opportunities to repurpose what little space remains on that system | 18:30 |
| *** tosky has quit IRC | 18:30 | |
| *** yamamoto has joined #openstack-infra | 18:32 | |
| jeblair | fungi: ++ i've been a little worried about increased use of tarballs with images, etc, so bumping it some is good i think. | 18:34 |
| *** yamamoto has quit IRC | 18:36 | |
| *** sdague has joined #openstack-infra | 18:40 | |
| *** signed8bit has joined #openstack-infra | 18:41 | |
| *** openstackgerrit has joined #openstack-infra | 18:42 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: WIP: Revert "Revert "Add sphinx_python variable to sphinx role and job"" https://review.openstack.org/526666 | 18:42 |
| *** ldesimone has quit IRC | 18:43 | |
| *** ldesimone has joined #openstack-infra | 18:43 | |
| *** yamamoto has joined #openstack-infra | 18:44 | |
| *** yamamoto has quit IRC | 18:44 | |
| *** yamamoto has joined #openstack-infra | 18:44 | |
| *** yamamoto has quit IRC | 18:45 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Omit legacy npm job with nodejs4-publish-to-npm https://review.openstack.org/526752 | 18:45 |
| fungi | smcginnis: mordred: ^ part of what i suspect is the solution for the monasca-grafana-datasource release problem | 18:45 |
| fungi | though we also need to figure out what's wrong with the release-openstack-javascript job. i suspect some of its tasks should be set to use localhost instead of the job node | 18:46 |
| *** ldesimone has quit IRC | 18:48 | |
| fungi | either that, or when the playbook was written the author confused node paths and executor paths | 18:50 |
| *** camunoz has joined #openstack-infra | 18:51 | |
| fungi | we're using zuul.executor.work_root at http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/upload-npm/tasks/main.yaml#n23 but it's being run on the job node | 18:52 |
| *** niedbalski has quit IRC | 18:54 | |
| *** niedbalski has joined #openstack-infra | 18:54 | |
| *** tosky has joined #openstack-infra | 18:54 | |
| pabelanger | which playbook is the role run from? | 18:55 |
| fungi | given the playbook calls this first i expect we should run upload-npm's upload task on localhost: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/fetch-javascript-tarball/tasks/main.yaml | 18:56 |
| fungi | pabelanger: good question. hunting it down again now | 18:57 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/javascript/post.yaml | 18:57 |
| pabelanger | i wouldn't expect that to run on localhost | 18:57 |
| fungi | yep, there | 18:57 |
| fungi | okay, so why would we do a synchronize task in fetch-javascript-tarball and use executor paths for the upload source? | 18:58 |
| fungi | seems very split-brained | 18:58 |
| pabelanger | i think the issue is, line 6 on th post.yaml, should be run with host: localhost | 18:58 |
| pabelanger | as the upload-npm role, is expected to run on the executor | 18:59 |
| fungi | yeah, that's what i was suggesting | 18:59 |
| fungi | i misunderstood your use of "expect" | 18:59 |
| fungi | i would expect it to run on localhost, but the author seems to have missed telling it to | 18:59 |
| pabelanger | however, it is possible when we move this to localhost, we might run into brap issues running npm | 19:00 |
| pabelanger | and nvm | 19:00 |
| pabelanger | (not sure what nvm is) | 19:00 |
| fungi | something nodeish | 19:00 |
| pabelanger | yah, must manage creds | 19:00 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Run upload-npm role on executor https://review.openstack.org/526755 | 19:03 |
| fungi | pabelanger: like that? ^ | 19:03 |
| pabelanger | fungi: yup! +2 | 19:05 |
| fungi | i can reenqueue the failing release tag once that and its parent merge and see if we're any closer to working | 19:06 |
| pabelanger | wfm | 19:06 |
| *** dhill_ has quit IRC | 19:07 | |
| *** dhill_ has joined #openstack-infra | 19:08 | |
| *** Goneri has quit IRC | 19:10 | |
| *** pramodrj07 has quit IRC | 19:13 | |
| *** MasterOfBugs has quit IRC | 19:13 | |
| *** rbrndt has joined #openstack-infra | 19:16 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: WIP: Revert "Revert "Add sphinx_python variable to sphinx role and job"" https://review.openstack.org/526666 | 19:19 |
| pabelanger | mgagne: I think we might have about 15 nodes stuck deleting in inap. is that something you could look at when you have time? | 19:19 |
| dmsimard | jeblair: did you want to look at https://review.openstack.org/#/q/topic:base-rename before I started merging them ? | 19:20 |
| mgagne | pabelanger: looking into it right now | 19:20 |
| pabelanger | mgagne: http://paste.openstack.org/show/628477/ has the UUIDs | 19:20 |
| *** harlowja has joined #openstack-infra | 19:21 | |
| *** jascott1 has joined #openstack-infra | 19:22 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-openstackci master: Add zookeeper to allinone manifest https://review.openstack.org/526758 | 19:23 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-openstackci master: Add partial support for zuulv3 scheduler https://review.openstack.org/526759 | 19:23 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/system-config master: Move zuulv3.o.o definitions into ::openstackci https://review.openstack.org/526760 | 19:24 |
| *** slaweq has joined #openstack-infra | 19:24 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/system-config master: Move zuulv3.o.o definitions into ::openstackci https://review.openstack.org/526760 | 19:24 |
| dmsimard | er, why am I getting a 404 not found on stream.html ? | 19:25 |
| dmsimard | http://zuulv3.openstack.org/stream.html?uuid=799719e1a1494c17a9c2bfc74f0328a5&logfile=console.log | 19:25 |
| dmsimard | infra-root ^ | 19:25 |
| pabelanger | looking | 19:26 |
| *** rossella_s has joined #openstack-infra | 19:26 | |
| tobiash | pabelanger, dmsimard: some zuul-web changes have landed | 19:26 |
| dmsimard | tobiash: ah, maybe we broke something | 19:26 |
| tobiash | the rewrite rules probably need to be adapted | 19:26 |
| pabelanger | okay, that might explain it | 19:27 |
| *** MasterOfBugs has joined #openstack-infra | 19:27 | |
| *** pramodrj07 has joined #openstack-infra | 19:27 | |
| tobiash | the stream is now tenant scoped | 19:27 |
| dmsimard | yeah looks like http://git.openstack.org/cgit/openstack-infra/zuul/commit/?h=feature/zuulv3&id=a4996f12ae0e19b9340232cbc25e0d4bbd4d7989 is the culprit | 19:28 |
| pabelanger | puppet-zuul needs to be update then | 19:28 |
| dmsimard | so /openstack/stream.html ? | 19:28 |
| pabelanger | anybody mind working on that? | 19:28 |
| dmsimard | /openstack/stream.html doesn't work either :/ | 19:28 |
| tobiash | the rewrite rules probably need only a slight change | 19:30 |
| dmsimard | pabelanger: what needs fixing ? the vhost ? | 19:30 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/templates/zuul.vhost.erb is where our rewrite rules live | 19:30 |
| pabelanger | so, that needs to be updated, but not sure to what without looking at the patch | 19:31 |
| dmsimard | ah it was /static/stream.html before | 19:31 |
| *** e0ne has joined #openstack-infra | 19:31 | |
| dmsimard | and /console-stream no longer exists | 19:32 |
| tobiash | the new is /{tenant}/stream.html | 19:32 |
| dmsimard | I can take a stab at it but it's kind of in the dark | 19:32 |
| tobiash | and /{tenant}/console-stream | 19:32 |
| pabelanger | we'll also have to restart zuul-web, it hasn't been restarted in a few days | 19:32 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Run ansible-lint-jobs on tripleo-upgrades https://review.openstack.org/526763 | 19:36 |
| tobiash | and there is another one coming which changes the location of the pubic keys: https://review.openstack.org/#/c/504807/ | 19:36 |
| tobiash | that will probably also need a change to the rewrite rules | 19:37 |
| dmsimard | tobiash: can we prevent regressions like those ? | 19:38 |
| dmsimard | tobiash: by running puppet-zuul or something | 19:38 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Clarify terminology around node request locks https://review.openstack.org/526233 | 19:38 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Handle race between handler and request cleanup https://review.openstack.org/526234 | 19:38 |
| tobiash | dmsimard: hm, good question | 19:38 |
| dmsimard | tobiash: also, the URLs in zuul-web aren't tenant scoped right now | 19:38 |
| dmsimard | tobiash: it's just /static | 19:38 |
| tobiash | this is basically a change in the api | 19:38 |
| dmsimard | tobiash: er, /stream.html | 19:38 |
| EmilienM | dmsimard, pabelanger: if we can get https://review.openstack.org/#/c/526763/ soon, it would really help the upgrade team - thanks | 19:39 |
| *** slaweq has quit IRC | 19:39 | |
| dmsimard | EmilienM: hunting down broken zuul consoles for now, I'll look after | 19:39 |
| EmilienM | dmsimard: good luck | 19:40 |
| pabelanger | dmsimard: yah, we need to properly make puppet-zuul support tenants, right now we just support it via: http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n1397 | 19:40 |
| tobiash | dmsimard: most of the urls in zuul web are tenant scoped and the latest and coming changes essentially make all of that tenant scoped | 19:40 |
| tobiash | afaik openstack does a rewrite to strip out their only tenant | 19:41 |
| dmsimard | tobiash: This is an url from zuulv3.openstack.org right now: http://zuulv3.openstack.org/stream.html?uuid=c28999b124ce44cdae98c98d2d690469&logfile=console.log | 19:41 |
| jeblair | yeah, this is an api change we just need to roll with | 19:41 |
| dmsimard | tobiash: it should be /openstack/stream.html according to the code ? | 19:41 |
| dmsimard | hmm, now I'm not sure http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/model.py?h=feature/zuulv3&id=a4996f12ae0e19b9340232cbc25e0d4bbd4d7989#n1866 | 19:42 |
| jeblair | yeah, i think that's one level too far | 19:43 |
| jeblair | we're not finding the stream.html page | 19:43 |
| jeblair | (which then needs to look for the console-stream websocket) | 19:43 |
| tobiash | dmsimard: the stram.html is relative to the status page (which is normally tenant scoped) | 19:43 |
| dmsimard | jeblair: yeah we need to fix the rewrite rules to account for the change but I'm trying to figure out why the link from the current web page seems incorrect | 19:43 |
| dmsimard | tobiash: oh so zuulv3.openstack.org should actually be zuulv3.openstack.org/openstack ? | 19:44 |
| tobiash | dmsimard: exactly | 19:44 |
| tobiash | that's stripped out by the rewrite rules | 19:44 |
| dmsimard | hmm | 19:44 |
| dmsimard | well I'm not sure it's stripped out by the rules so much as we're hosting it at the root | 19:45 |
| jeblair | it's the static change that has broken us | 19:45 |
| jeblair | http://zuulv3.openstack.org/static/stream.html?uuid=f485d8b09ac44067be22e1f81ecc8132&logfile=console.log works | 19:45 |
| jeblair | (inasmuch as it finds stream.html) | 19:46 |
| jeblair | (it's also broken because of console-stream, but that's the next problem) | 19:46 |
| *** yamamoto has joined #openstack-infra | 19:46 | |
| tobiash | jeblair: with tristanC's last change I think the rewrite rules could be simplified as all paths are tenant scoped then | 19:47 |
| mgagne | pabelanger: still working on it, we hit a bug in Nova that prevents instance from being deleted | 19:48 |
| tobiash | for now I think a rewrite rule like 'RewriteRule ^/stream.html <%= @zuul_web_url %>/stream.html [P]' would do it | 19:48 |
| tobiash | hrm | 19:50 |
| tobiash | what's the value of the zuul_web_url variable? | 19:50 |
| jeblair | oh yeah, i bet if we restarted zuul-web, even that static link wouldn't work anymore. i think that's still partially the old system | 19:50 |
| jeblair | tobiash: i don't think we have that set | 19:51 |
| jeblair | only related var set is [webapp] status_url=https://zuulv3.openstack.org/ | 19:51 |
| tobiash | ah, then my rule would be wrong | 19:51 |
| jeblair | oh that's a puppet url? i don't know | 19:52 |
| tobiash | then I would try 'RewriteRule ^/stream.html <%= @zuul_web_url %>/openstack/stream.html [P]' | 19:52 |
| jeblair | sorry, i thought you were asking about zuul.conf | 19:52 |
| tobiash | and similar the console stream | 19:52 |
| dmsimard | $zuul_web_url = 'http://127.0.0.1:9000' | 19:52 |
| jeblair | dmsimard: ++ | 19:52 |
| *** yamamoto has quit IRC | 19:52 | |
| tobiash | ok, so we need to add /openstack/ into the rewrite rules | 19:52 |
| jeblair | is *everything* moved to zuul-web and tenant scoped nw? | 19:52 |
| jeblair | now? | 19:53 |
| dmsimard | The rules are all here: http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/templates/zuul.vhost.erb#n31 | 19:53 |
| tobiash | jeblair: one patch is still pending | 19:53 |
| dmsimard | tobiash: hardcoding /openstack/ seems to be a bit against the point | 19:53 |
| *** fried_rolls is now known as fried_rice | 19:53 | |
| tobiash | dmsimard: ah, thought, this is the openstack specific puppet module | 19:54 |
| tobiash | jeblair: according to https://review.openstack.org/#/c/504807/4/zuul/web/__init__.py once it lands pretty much everything is tenant scoped | 19:54 |
| tobiash | except tenants.yaml | 19:55 |
| *** rossella_s has quit IRC | 19:55 | |
| tobiash | s/yaml/html | 19:55 |
| jeblair | the thing we're aiming for is to get everything hosted in the tenant scope, and then have zuul.openstack.org drop the url prefix with rewrite rules. that probably means the structure of puppet-zuul needs to change. | 19:55 |
| dmsimard | have to step away for a few, brb | 19:56 |
| tobiash | jeblair: do you really keep the drop of openstack? | 19:56 |
| jeblair | tobiash: yes, as long as openstack has one tenant, i don't want urls like 'zuul.openstack.org/openstack' | 19:56 |
| tobiash | you also could have a redirect on / (in case multi tenantcy might be a thing for openstack in the future) | 19:56 |
| jeblair | tobiash: i think multitenancy is much more likely to come with new vhosts than new sub-urls. | 19:57 |
| tobiash | ah, ok | 19:57 |
| jeblair | (like we add a vhost for zuul.foobar.org if we add the foobar tenant) | 19:58 |
| jeblair | i think that puppet vhost template can no longer support both v2 and v3 | 19:58 |
| jeblair | tobiash: i think you're rewrite rule above would work though, and keep us limping forward a bit longer | 19:59 |
| jeblair | the status page move will be problematic. | 20:00 |
| jeblair | i'm going to restart zuul-web, manually apply tobiash's redirect, and verify that works | 20:01 |
| jeblair | just so our iteration cycle isn't so long | 20:01 |
| jeblair | http://zuulv3.openstack.org/stream.html?uuid=2710d1238603494f93dc3b01f4e618ff&logfile=console.log works now | 20:04 |
| tobiash | \o/ | 20:04 |
| *** gouthamr has joined #openstack-infra | 20:05 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add finger gateway https://review.openstack.org/525276 | 20:05 |
| dmsimard | jeblair: so we'd need to add a tenant parameter to puppet-zuul, default that to openstack and plumb that in the rewrite rule ? | 20:05 |
| jeblair | dmsimard: yes -- unless we do what we're doing with zuul_status_url and stick openstack/ on the end of that | 20:06 |
| jeblair | we can do that if all the uses of zuul_web_url in puppet-zuul are now tenant scoped | 20:06 |
| jeblair | they are: console-stream (yes), static (?), jobs (?) | 20:07 |
| *** kjackal has joined #openstack-infra | 20:07 | |
| jeblair | i'm trying to figure out how to look into static or jobs | 20:07 |
| dmsimard | AJaeger_, mordred: found the issue for the sphinx venv.. it's kind of ugly. It looks like "with_first_found" seeks for files in localhost, not the target node | 20:08 |
| jeblair | are we serving status.json from both services now? zuul-web and the scheduler? | 20:08 |
| pabelanger | mgagne: thanks for hte update | 20:09 |
| jeblair | dmsimard, tobiash: i think we might be able to do what dmsimard suggested now, if we carve out an exception for keys | 20:11 |
| openstackgerrit | Dan Prince proposed openstack-infra/reviewday master: Fix pep8 test https://review.openstack.org/526717 | 20:11 |
| jeblair | so maybe we can go ahead and put in the 'drop the tenant' rewrite rule, along with a specific rewrite rule for keys until that patch lands, then drop the exception | 20:11 |
| pabelanger | jeblair: dmsimard: we should also land https://review.openstack.org/526463/ to properly restart scheduler / mergers now | 20:12 |
| jeblair | RewriteRule ^/(.*) http://127.0.0.1:9000/openstack/$1 [P] | 20:12 |
| jeblair | nope, that doesn't work... why not? | 20:13 |
| jeblair | er, is it because we have no index page? | 20:13 |
| dmsimard | what doesn't work if you do that ? | 20:13 |
| fungi | jeblair: does it end up being recursive? | 20:13 |
| fungi | oh, nevermind, you're effectively redirecting to a different server | 20:14 |
| fungi | so don't need to worry about the target getting passed back through the redirect again | 20:14 |
| jeblair | http://zuulv3.openstack.org/test/ | 20:14 |
| jeblair | RewriteRule ^/test/(.*) http://127.0.0.1:9000/openstack/$1 [P] | 20:14 |
| jeblair | i did that so we can poke at it | 20:14 |
| tobiash | jeblair: static may be not needed anymore | 20:14 |
| *** sbezverk has joined #openstack-infra | 20:14 | |
| dmsimard | I get a console at http://zuulv3.openstack.org/test/stream.html?uuid=418bc2786b8b46f6b65ac964df8c04d8&logfile=console.log but no stream | 20:15 |
| jeblair | dmsimard: well, it may have a link for the stream that doesn't include test | 20:15 |
| dmsimard | jeblair: we still need the ws:// URL somewhere, no ? | 20:15 |
| jeblair | so it's not perfect -- it will just let us check initial urls, not dependencies | 20:16 |
| tobiash | the ws:// must be specified separately | 20:16 |
| tobiash | apache cannot handle ws transparently :( | 20:16 |
| jeblair | i know the the console stream stull will work | 20:16 |
| jeblair | stuff | 20:16 |
| *** david-lyle has joined #openstack-infra | 20:16 | |
| jeblair | what i don't know is how to get the status page | 20:16 |
| jeblair | ('GET', '/{tenant}/status.html', self._handleStaticRequest), | 20:16 |
| jeblair | that made me think that http://zuulv3.openstack.org/test/status.html would be the status page | 20:17 |
| jeblair | it is not | 20:17 |
| *** ChanServ has quit IRC | 20:17 | |
| dmsimard | well status.json works | 20:19 |
| dmsimard | http://zuulv3.openstack.org/test/status.json | 20:19 |
| jeblair | huh, sure does | 20:19 |
| dmsimard | actually http://zuulv3.openstack.org/test/status.html just started (kind of) working I guess you're hacking on things | 20:20 |
| *** kjackal has quit IRC | 20:20 | |
| fungi | apache error.log seems to contain a bunch of "Not a git repository" errorsfrom the time of the 20:14:11 graceful restart | 20:20 |
| *** slaweq has joined #openstack-infra | 20:20 | |
| jeblair | dmsimard: that doesn't look like a status page to me though | 20:20 |
| jeblair | i just see 4 links | 20:20 |
| dmsimard | jeblair: yeah it's like the zuul dashboard thing tristanC has been working on, with jobs and builds | 20:20 |
| jeblair | okay, i think it's supposed to be that page but it's not getting dependencies? | 20:21 |
| dmsimard | jeblair: oh, eh, look: https://softwarefactory-project.io/zuul3/local/status.html | 20:21 |
| tobiash | jeblair: yes | 20:21 |
| jeblair | yep, that looks right according to the source | 20:21 |
| tobiash | the dependencies are hard coded to be absolute | 20:21 |
| tobiash | I have to use ProxyHTMLURLMap as I don't run zuul on / | 20:23 |
| jeblair | http://zuulv3.openstack.org/test/static/js/jquery-visibility.min.js | 20:23 |
| jeblair | okay, that's one of the deps, but that url 404s | 20:23 |
| *** ldnunes has quit IRC | 20:23 | |
| dmsimard | This one seems relative and works <script src="../static/javascripts/zuul.angular.js"></script> | 20:24 |
| *** huanxie has quit IRC | 20:24 | |
| *** ChanServ has joined #openstack-infra | 20:24 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 20:24 | |
| tobiash | these deps are not tenant scoped | 20:24 |
| *** Goneri has joined #openstack-infra | 20:26 | |
| jeblair | okay, so we can't do it in one rewrite rule now, we still need a bunch | 20:26 |
| *** huanxie has joined #openstack-infra | 20:26 | |
| jeblair | i need to grab lunch | 20:28 |
| jeblair | i think we should drop zuulv3.o.o in the emergency file so puppet doesn't run and the manually applied fixes will stay | 20:28 |
| dmsimard | +1 | 20:28 |
| *** pcrews has quit IRC | 20:28 | |
| dmsimard | This is what the rewrite rules look like for softwarefactory-project.io: https://github.com/softwarefactory-project/sf-config/blob/2e9db4f198a64e6cc4cadf10899aac70abdf3773/ansible/roles/sf-gateway/templates/gateway.common.j2#L155-L176 | 20:28 |
| *** sbezverk has quit IRC | 20:29 | |
| jeblair | then i think probably add a bunch of rewrite rules in puppet zuul, and have a zuulv3 switch that selects one set or the other | 20:29 |
| dmsimard | jeblair: you mean to keep supporting v2 simultaneously ? | 20:29 |
| jeblair | dmsimard: puppet-zuul has to :( | 20:30 |
| dmsimard | I'll start a patch, will probably need a bit of help though | 20:30 |
| jeblair | dmsimard: cool thx | 20:30 |
| jeblair | RewriteRule ^/console-stream ws://127.0.0.1:9000/openstack/console-stream [P] | 20:30 |
| jeblair | RewriteRule ^/stream.html http://127.0.0.1:9000/openstack/stream.html [P] | 20:30 |
| jeblair | those are the 2 changes i've made manually | 20:30 |
| jeblair | #status log added zuulv3.openstack.org to emergency file due to manual fixes to apache rewrite rules | 20:31 |
| openstackstatus | jeblair: finished logging | 20:31 |
| *** sbezverk has joined #openstack-infra | 20:31 | |
| *** david-lyle has quit IRC | 20:32 | |
| jeblair | and it looks like puppet just overwrote my changes, so i re-made them. i think that means we should be okay since we're past this point in the cycle and i have the emergency stop in place | 20:33 |
| jeblair | (so it shouldn't run on the next cycle) | 20:33 |
| jeblair | lunch now. biab. | 20:33 |
| jeblair | infra-root: ^ fyi | 20:33 |
| *** Odd_Bloke has quit IRC | 20:37 | |
| *** Odd_Bloke has joined #openstack-infra | 20:37 | |
| *** Apoorva has joined #openstack-infra | 20:38 | |
| *** Apoorva has quit IRC | 20:38 | |
| *** Apoorva has joined #openstack-infra | 20:39 | |
| *** pcrews has joined #openstack-infra | 20:41 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Make all zuul-web urls relative https://review.openstack.org/526770 | 20:42 |
| tobiash | this could make the url rewriting simpler ^^ | 20:43 |
| tobiash | just tried it in my environment and it looks much easier for nonroot/rewriting deployments | 20:45 |
| *** jkilpatr has quit IRC | 20:47 | |
| *** yamamoto has joined #openstack-infra | 20:48 | |
| mordred | dmsimard: wow. well, that explains why my tests on localhost didn't work | 20:51 |
| *** yamamoto has quit IRC | 20:52 | |
| mordred | jeblair, dmsimard: we probably also need to fix and land this: https://review.openstack.org/#/c/521630/ | 20:53 |
| *** ttx has quit IRC | 20:57 | |
| *** ttx has joined #openstack-infra | 20:57 | |
| dmsimard | mordred: yeah I'll think of something | 20:57 |
| mordred | tobiash: that looks fine - also, now that the bulk of the initial dashboard stuff has landed I'm going to work on fixing up the javascript tooling patch | 20:58 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Clarify terminology around node request locks https://review.openstack.org/526233 | 21:00 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Handle race between handler and request cleanup https://review.openstack.org/526234 | 21:00 |
| mgagne | pabelanger: ok so I found the issue. If your instance has task_state=deleting and somehow, it fails to delete because whatever reasons, you can't ask for a delete again through API. You need to set task_state to NULL in the database. Now I don't know how but the instance will now get deleted by Nova very shortly after. If you are in a hurry, you can delete again through API and it will get deleted. | 21:00 |
| *** xarses has joined #openstack-infra | 21:01 | |
| *** thiagolib has quit IRC | 21:01 | |
| *** sbezverk has quit IRC | 21:02 | |
| mordred | tobiash: yah - +2 from me on that | 21:04 |
| tobiash | \o/ | 21:04 |
| pabelanger | mgagne: okay, good to know. cc mordred | 21:05 |
| pabelanger | (I'm sure mordred likely knows this) :) | 21:05 |
| *** gouthamr has quit IRC | 21:07 | |
| mordred | mgagne, pabelanger: awesome. and yes, we've seen that happen before - I betcha we could at the very least add support in shade for detecting task_state=deleting and at the very least log something if someone tries to delete something in that state (since it'll pretty much never work) | 21:08 |
| mordred | I'm not sure an Exception is the right choice, since it's not resolvable by the user | 21:08 |
| smcginnis | mordred: Hey! Are you aware of any outstanding work yet on those NPM release jobs? | 21:09 |
| smcginnis | mordred: re: conversation from a few days back if you can still remember that. :) | 21:10 |
| mordred | smcginnis: nope - only on me workingon them less than I was supposed to have | 21:10 |
| *** rlandy has quit IRC | 21:10 | |
| smcginnis | mordred: Cool, I'll take a look. Hopefully soon. | 21:10 |
| *** fried_rice is now known as efried_cya_jan | 21:12 | |
| fungi | mordred: smcginnis: i pushed some patches up a few hours ago--i'll get you urls | 21:13 |
| fungi | mordred: smcginnis: https://review.openstack.org/526752 and its child https://review.openstack.org/526755 | 21:14 |
| fungi | feedback would be helpful since i'm to some extent attempting to divine the intent of the original author | 21:15 |
| dmsimard | tobiash, jeblair: I'm not able to find a non-ugly way to plumb the notion of tenant down to the vhost rewrite rules | 21:15 |
| dmsimard | Assuming a single tenant is easy, but probably not what we want | 21:15 |
| mordred | fungi: second +A'd, maybe dmsimard or pabelanger will +3 https://review.openstack.org/#/c/526752 real quick | 21:16 |
| fungi | cool. once those land i'll reenqueue the failing tag yet again | 21:17 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Serve keys from canonical project name https://review.openstack.org/504807 | 21:18 |
| mordred | dmsimard: I think maybe assuming single tenant isn't terrible - I think the idea is "this is vhost {{ url }} for zuul tenant {{ tenant_name }}" ... and if someone doesn't want vhost-mapped-tenants but instead wants their web thing rooted up a level, maybe there should be a mapping passthrough version? | 21:19 |
| mordred | dmsimard: just thinking outloud, I could also be very wrong | 21:19 |
| clarkb | fungi: aha! looks like elasticsearch04 may have had a rax induced lunch break | 21:19 |
| clarkb | fungi: I think that is why the logstash processes are having sads | 21:19 |
| clarkb | I will hard reboot it | 21:19 |
| *** ganso has quit IRC | 21:19 | |
| fungi | clarkb: oh, poop, i should have tried logging into that one since rackspace opened that ticket saying something about a failed migration and a scheduled reboot for this weekend | 21:20 |
| fungi | they also mentioned elasticsearch02 but it seems to be reachable | 21:20 |
| clarkb | ya 02 is up | 21:20 |
| fungi | mordred: dmsimard: taking it a little further, if you have multiple tenants then it's possible you don't want redirects to gobble the tenant field from the url anyway? | 21:21 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-zuul master: Zuulv3: Change rewrite rules to get tenant scoped consoles https://review.openstack.org/526777 | 21:22 |
| clarkb | 04 is back in the cluster again, I am manually running the curator script now so that we don't try to recover the indexes on 04 | 21:22 |
| dmsimard | fungi: jeblair was of the opinion earlier that zuulv3.openstack.org was better than zuulv3.openstack.org/openstack | 21:22 |
| fungi | dmsimard: when you have only one tenant, right | 21:22 |
| fungi | mordred: dmsimard: though based on jeblair's earlier assertion about how we might end up doing multi-tenant zuul deployments, maybe you actually want to have "hard-coded" (really parameterized) tenants but multiple apache vhosts one for each tenant | 21:22 |
| jeblair | dmsimard, mordred: yeah, i think we can assume single tenant there for now. and now that we understand things more, we may just be able to reuse the zuul_web_url var | 21:23 |
| dmsimard | fungi: yeah, different vhosts | 21:23 |
| dmsimard | jeblair: https://review.openstack.org/526777 is up, the new template is a copy paste from the old one with the new rules in | 21:23 |
| *** david-lyle has joined #openstack-infra | 21:24 | |
| dmsimard | fungi: yeah, but if we want to take that approach (of one vhost per tenant), that means we need to re-think puppet-zuul apache configuration to some extent | 21:24 |
| dmsimard | but I left an editorial comment about that in my patch ^ | 21:24 |
| fungi | dmsimard: well, or make the puppet-zuul module a lowest-common-denominator opinionated deployment with a flag allowing you to just turn off the apache proxy bit and declare your own vhosts separately if you need to differentiate | 21:25 |
| *** dave-mccowan has quit IRC | 21:25 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-zuul master: Zuulv3: Change rewrite rules to get tenant scoped consoles https://review.openstack.org/526777 | 21:25 |
| *** e0ne has quit IRC | 21:26 | |
| fungi | it's often fair to punt on complicated designs if they're less common and just stick the opinionated bit in an isolated class the consumer can opt out of or hidden behind an enable flag they can choose to disable | 21:26 |
| clarkb | I wonder if this breaks because we list all of the elasticsearch nodes in the logstash output conf | 21:27 |
| dmsimard | so I was wondering, pretend kata starts using zuul -- they'll be in their own tenant, right ? | 21:27 |
| clarkb | (idea there being it can talk to any of them but maybe if it can't talk to one of them it braeks) | 21:27 |
| *** smatzek has quit IRC | 21:27 | |
| jeblair | dmsimard: i want to poke at maybe an alternative implementation... gimme a sec | 21:28 |
| fungi | dmsimard: maybe? we haven't really approached that discussion yet | 21:28 |
| dmsimard | jeblair: in case you missed it, tobiash sent a patch to make URLs relative https://review.openstack.org/#/c/526770/ | 21:28 |
| fungi | clarkb: yeah, it's possible we're just increasing the odds of breakage 6x | 21:28 |
| *** david-lyle has quit IRC | 21:29 | |
| dmsimard | fungi: it'd bring the question of multi-tenancy faster if so, but I was just mostly curious | 21:29 |
| mordred | I think we'd be more likely to set up a zuul.kataproject.io than have them look at a zuul.openstack.org/kata | 21:29 |
| mordred | but - yah - we don't *actually* have an answer for that yet | 21:29 |
| openstackgerrit | Merged openstack-infra/project-config master: Omit legacy npm job with nodejs4-publish-to-npm https://review.openstack.org/526752 | 21:29 |
| *** camunoz has quit IRC | 21:30 | |
| dmsimard | mordred, tobiash: I added a question in https://review.openstack.org/#/c/526770/ | 21:30 |
| openstackgerrit | Merged openstack-infra/project-config master: Run upload-npm role on executor https://review.openstack.org/526755 | 21:31 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add finger gateway https://review.openstack.org/525276 | 21:31 |
| clarkb | mordred: as I mentioned in the spec I think we run single services as much as possible and reduce the brand overload. So in that case you'd CNAME zuul.kataproject.io to zuul.openstack.org then use /kata | 21:31 |
| mordred | dmsimard: I think you're right - but I'll let tobiash weigh in since he's running it locally and testing paths and stuff | 21:31 |
| fungi | smcginnis: yeah, wrt your comment on 526752 i agree and thought about trying to solve it for the others, but since they weren't yet running the same jobs as this one i figured it was safer to make sure we have at least one working example first | 21:31 |
| smcginnis | fungi: ++ | 21:32 |
| clarkb | (we just can't scale to running duplicate infrastructures for every new project) | 21:32 |
| dmsimard | clarkb: it'd be single service | 21:32 |
| dmsimard | clarkb: but two tenants, and two vhosts | 21:32 |
| clarkb | I see you just mean collapse the path down too | 21:32 |
| jeblair | yep. putting the service name before or after the / means equal amount of work for us. :) | 21:33 |
| fungi | right, not multiple zuul deployments | 21:33 |
| dmsimard | well that's what we're already doing today | 21:33 |
| jeblair | putting it before the / is better for branding | 21:33 |
| jeblair | s/service name/project name/ | 21:33 |
| *** e0ne has joined #openstack-infra | 21:33 | |
| *** leakypipes has quit IRC | 21:33 | |
| tobiash | dmsimard, mordred: builds.html is served tenant scoped by zuul-web: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/web/__init__.py?h=feature/zuulv3#n389 | 21:33 |
| clarkb | I think my ideal would be zuul.fooci.org/openstack and zuul.fooci.org/kata and so on. That way its still less vhost configuration | 21:34 |
| tobiash | 21:34 | |
| clarkb | tl;dr people seem to be ok hosting their projects using "neutral" branding like github + travis + cname to project website hosting | 21:34 |
| tobiash | (but stored in static) | 21:34 |
| dmsimard | tobiash: why are static things scoped ? | 21:34 |
| * clarkb goes back to fixing logstash | 21:34 | |
| *** kgiusti has quit IRC | 21:34 | |
| jeblair | clarkb: i think we can consider that when we have a neutral brand | 21:35 |
| *** e0ne has quit IRC | 21:35 | |
| tobiash | dmsimard: because then they can just use relative paths for status.json, console stream etc | 21:35 |
| *** kgiusti has joined #openstack-infra | 21:35 | |
| fungi | clarkb: up-side to putting more resources in their own individual domains (even if only symbolically) is that if they move between platforms they can avoid having to update references all over the place and instead can just redirect things so old links keep working | 21:35 |
| mordred | dmsimard: so that the url is zuul.foo.com/openstack/builds.html is the builds for the openstack tenant as opposed to zuul.foo.com/builds.html?tenant=openstack | 21:35 |
| pabelanger | darn, neutral.org is taken :) | 21:36 |
| dmsimard | mordred: I get that, I meant why is static/bootstrap.css scoped under tenant/static/bootstrap.css | 21:36 |
| mordred | dmsimard: oh- that I think mostly just because there is just one folder with all the static bits | 21:36 |
| fungi | pabelanger: there have been some ideas floated for domain names, but problem is as soon as you start trying to have a public discussion about picking a domain you risk someone sniping and squatting it :/ | 21:36 |
| dmsimard | instead of just being "public" under /static/boostrap.css | 21:37 |
| jeblair | dmsimard: well, one reason to do that is so that you can drop the tenant from the url as we want to do. | 21:37 |
| tobiash | dmsimard: zuul-web currently lists all tenant scoped files by name | 21:37 |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Build/publish sphinx-feature-classification docs https://review.openstack.org/526782 | 21:37 |
| pabelanger | fungi: yah | 21:37 |
| tobiash | so only the 'main' pages are tenant scoped such they can easily request tenant scoped resources | 21:37 |
| jeblair | https://etherpad.openstack.org/p/c9XUD9WULG | 21:39 |
| clarkb | jeblair: ya I agree I don't think we should focus on that now as we don't have such a brand yet | 21:39 |
| jeblair | tobiash, dmsimard, mordred: ^ etherpad has my proposed set of rewrites | 21:39 |
| *** Apoorva has quit IRC | 21:39 | |
| jeblair | i think we can do that just by continuing to use the zuul_web_url variable, since that's a v3 only thing | 21:39 |
| jeblair | that moves everything over except keys, which we'll be able to accomodate once that patch lands | 21:40 |
| jeblair | that's basically just a translation of the route table from zuul-web | 21:40 |
| jeblair | the other thing i'd do is stop adding the zuulv2 rewrites if zuul_web_url is set | 21:40 |
| jeblair | does that sound like maybe it might work? if so, i can try it out in prod real quick to sanity check, then make the puppet change | 21:41 |
| mordred | jeblair: didn't we already land 'serve keys from zuul-web' ? | 21:41 |
| jeblair | mordred: yes, but not at a good url | 21:41 |
| jeblair | it's not tenant scoped, and my current plan hinges on including the tenant name in the zuul_web_url puppet var | 21:42 |
| mordred | jeblair: ah - gotcha | 21:43 |
| tobiash | jeblair: I think the keys rule is not yet correct | 21:43 |
| jeblair | tobiash: that's the current proxy rule | 21:43 |
| jeblair | note the 8001 rather than 9000 | 21:43 |
| tobiash | ah | 21:43 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Update jobs for tripleo-upgrades https://review.openstack.org/526763 | 21:43 |
| jeblair | so it's still using the old server | 21:43 |
| mordred | also - i was wrong, we haven't landed https://review.openstack.org/#/c/504807 yet anyway (the 8001 was the reason I was asking that) | 21:43 |
| *** rcernin has joined #openstack-infra | 21:44 | |
| fungi | mordred: pabelanger: okay, so what did i get wrong? http://logs.openstack.org/b1/b18a06caa5955d6b72535cdb88e60b09f3e692d0/release/release-openstack-javascript/c4a57b8/job-output.txt.gz#_2017-12-08_21_40_56_996559 | 21:44 |
| *** trown is now known as trown|outtypewww | 21:44 | |
| jeblair | mordred: ('GET', '/{source}/{project}.pub', self._handleKeyRequest), | 21:44 |
| jeblair | mordred: that's in zuul-web now, the upcoming change would move that to tenant/project/... | 21:45 |
| tobiash | jeblair, mordred: I added the rule for the upcoming change | 21:45 |
| clarkb | fungi: https://www.elastic.co/guide/en/logstash/2.4/plugins-outputs-elasticsearch.html#_retry_policy says network errors are retried infinitely and I'm guessing not to a different host | 21:46 |
| jeblair | tobiash: that should be 9001, ya? | 21:46 |
| clarkb | fungi: so I think that is likely the cause | 21:46 |
| tobiash | oh, yes | 21:46 |
| jeblair | er 9000 | 21:46 |
| fungi | mordred: pabelanger: seems it thinks project_ver isn't set, so i guess this is a latent bug in the original job, not something i broke | 21:46 |
| mordred | yah ++ makes sense | 21:46 |
| clarkb | fungi: however it doesn't seem to recover without restarting the service making me think we may want a timeout value set | 21:46 |
| mordred | fungi: looking | 21:46 |
| jeblair | okay, i'm going to reload apache with those in prod to validate this idea | 21:46 |
| fungi | clarkb: sounds likely | 21:47 |
| clarkb | I'll get a patch up for that in a sec | 21:47 |
| jeblair | nope, 404 -- what are we still missing? | 21:47 |
| mordred | fungi: version-from-git role is the thing that sets that | 21:47 |
| jeblair | oh / handling itself i think | 21:47 |
| smcginnis | Missing a step in the role? | 21:48 |
| *** Apoorva has joined #openstack-infra | 21:48 | |
| mordred | fungi: it should be run before upload-npm ... I'm guessing facts don't persist across plays then ... hrm | 21:49 |
| *** yamamoto has joined #openstack-infra | 21:49 | |
| fungi | yeah, i see it putting "project_ver": "1.2.0" in facts earlier | 21:49 |
| jeblair | the static deps still aren't working, why? | 21:50 |
| jeblair | mordred, tobiash, dmsimard: ^? | 21:50 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Timeout logstash es pushes after 300s https://review.openstack.org/526785 | 21:50 |
| clarkb | fungi: ^ | 21:50 |
| jeblair | oh, they aren't tenant scoped | 21:51 |
| tobiash | jeblair: yes | 21:51 |
| dmsimard | jeblair: do we need to land tobiash's relative URL fix ? | 21:51 |
| jeblair | okay, etherpad updated with current state | 21:51 |
| jeblair | i didn't think so? | 21:52 |
| mordred | jeblair: yes. that. | 21:52 |
| jeblair | mordred: well, that fixed half of them | 21:52 |
| tobiash | jeblair: added static rule to the etherpad | 21:52 |
| tobiash | that could help | 21:52 |
| jeblair | tobiash: shouldn't line 11 take care of that? | 21:52 |
| tobiash | oh right, for some reason I read openstack there... | 21:53 |
| jeblair | oh | 21:53 |
| jeblair | er | 21:53 |
| jeblair | i think we're missing files | 21:53 |
| *** yamamoto has quit IRC | 21:54 | |
| jeblair | because zuul-web is service static content from the installation directory | 21:54 |
| jeblair | and apparently we're supposed to download stuff and copy into the installation directory | 21:54 |
| jeblair | that doesn't sound right at all | 21:54 |
| tobiash | jeblair: the relative resources work, the absolute not | 21:54 |
| tobiash | so the relative path patch might help in this case | 21:55 |
| jeblair | tobiash: well, i think the resources that are part of the source code are working, and the external things are not | 21:55 |
| tobiash | do you host them from apache? | 21:55 |
| *** edmondsw has joined #openstack-infra | 21:55 | |
| *** edmondsw has quit IRC | 21:56 | |
| jeblair | one sec. i'm going to revert my manual changes | 21:56 |
| mordred | jeblair: this story will be much better with the webpack (no internal/external split) ... but for now I think we want to add a /static directory to the vhost that serves the directory where we put the external static things? | 21:56 |
| mordred | smcginnis, fungi: ok. I have a fix for that var issue coming | 21:57 |
| tobiash | in my deployment I install them into the static dir served by zuul-web so all static stuff (internal and external) comes from the same location | 21:57 |
| *** smatzek has joined #openstack-infra | 21:57 | |
| jeblair | mordred, tobiash: we previously/currently install all the external dependencies via puppet and serve them from /var/lib/zuul/www which is the documentroot, so what gets served if no rewrite rules match | 21:58 |
| tobiash | jeblair: so then we would need a rewrite rule for every static internal resource right? | 21:58 |
| jeblair | mordred: so yeah, it looks like the intended thing is to serve this all from /var/www/html/static | 21:59 |
| jeblair | tobiash: if we do that, we shoudn't need it | 21:59 |
| jeblair | i think "/" is static files served by zuul-web. "/static/" is external dependencies | 21:59 |
| tobiash | yes | 22:00 |
| tobiash | but you have to take care of updating them when updating zuul-web | 22:00 |
| jeblair | i think the only new dependency is angular | 22:01 |
| jeblair | so we should be able to have puppet-zuul install all the existing deps into /static pretty easily | 22:01 |
| *** smatzek has quit IRC | 22:01 | |
| mordred | yes, I agree | 22:02 |
| jeblair | i'll try to write that change first | 22:02 |
| *** rcernin has quit IRC | 22:03 | |
| mordred | jeblair: however, I think we should do whichever is easier, since ultimately having webpack make a deployment dir that has internal and external things in it and not have puppet fetch the external deps at all ... | 22:03 |
| *** rcernin has joined #openstack-infra | 22:03 | |
| jeblair | mordred: yeah, i think the puppet thing is going to be the easiest way to get zuulv3 out of the emergency file | 22:04 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Handle race between handler and request cleanup https://review.openstack.org/526234 | 22:04 |
| mordred | jeblair: ok. cool | 22:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Grab package_ver from remote host https://review.openstack.org/526790 | 22:05 |
| mordred | fungi, smcginnis: ^^ | 22:05 |
| mordred | jeblair: the angular patch has a puppet-lint error I was going to fix right now - but I don't want to step on what you're doing | 22:06 |
| mordred | jeblair: https://review.openstack.org/#/c/521630/ ifyou want to take it over while you're there - or else I can fix it independently | 22:06 |
| fungi | pabelanger: any chance you can look at 526790 real quick (in the same series where we're trying to fix js releasing)? | 22:07 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Update jobs for tripleo-upgrades https://review.openstack.org/526763 | 22:07 |
| inc0 | hey guys, not sure if you noticed but this RETRY_LIMIT seems to be striking again | 22:07 |
| mordred | tristanC, dmsimard: I think you said the dashboard code is expecting v1.5.6 yeah? | 22:07 |
| clarkb | inc0: have an example? | 22:08 |
| jeblair | mordred: i'll take it over, thanks! | 22:08 |
| dmsimard | mordred: on my phone right now. I think we discussed it a while back in #zuul.. dont remember by heart | 22:08 |
| inc0 | https://review.openstack.org/#/c/526421/ this is rechcecking but already have 2 of them | 22:08 |
| mordred | jeblair: cool. I'm investigating the version again real quick - I think it wants to be 1.5.6 instead of 1.5.8 - but I'm verifying | 22:09 |
| *** felipemonteiro_ has quit IRC | 22:11 | |
| jeblair | mordred: https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/web/static/README?id=feature/zuulv3#n1 says 156 | 22:12 |
| jeblair | (that's the script i'm working from, translating that to puppet basically) | 22:12 |
| *** gouthamr has joined #openstack-infra | 22:12 | |
| mordred | jeblair: well, 1.3.7 seems to be what tristanC has installed on https://softwarefactory-project.io/zuul3 and that seems to be working | 22:13 |
| openstackgerrit | Merged openstack-infra/reviewday master: Fix pep8 test https://review.openstack.org/526717 | 22:14 |
| fungi | inc0: that change seems to be making changes to job definitions so my gut says we wouldn't have noticed because this is probably a behavior that change is attempting to introduce | 22:14 |
| fungi | inc0: are you seeing it happen on changes not altering jobs? | 22:14 |
| *** ldesimone has joined #openstack-infra | 22:14 | |
| jeblair | mordred: hrm. which you want to go with? :) | 22:14 |
| tobiash | I'm running on 1.5.8 | 22:15 |
| mordred | tobiash: and it's working for you? | 22:16 |
| *** ldesimone has quit IRC | 22:16 | |
| tobiash | let's say I didn't notice anything not working besides builds | 22:16 |
| tobiash | but that's because I have no db yet ;) | 22:16 |
| mordred | I vote 1.5.8 if it's working for tobiash - and I'll put up a patch to fix the readme | 22:16 |
| melwitt | could anyone help point out what I've got wrong on my attempt to run a tempest job in-tree? this used to work 7 weeks ago and now it fails. I'm trying to use it to test a patch series https://review.openstack.org/#/c/513160 | 22:17 |
| *** ianychoi_ has joined #openstack-infra | 22:17 | |
| melwitt | the error is "ERROR Unable to find playbook /var/lib/zuul/builds/849c1ed8bb554f698adfd8bf2be40ed4/work/src/git.openstack.org/openstack/nova/playbooks/legacy/tempest-dsvm-neutron-nova-next-full/run" | 22:17 |
| inc0 | fungi: it's cherry pick from master, and master is fine | 22:17 |
| inc0 | this also changes the gates https://review.openstack.org/#/c/526469/ and got hit with retry_limit before recheck which just rightfully failed | 22:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Install angular.js and other static files for zuul web https://review.openstack.org/521630 | 22:18 |
| jeblair | mordred: i squashed ^ | 22:18 |
| jeblair | mostly because i moved your thing over to web.pp | 22:18 |
| mordred | melwitt: you need the .yaml suffixes now ... comment left | 22:19 |
| mordred | jeblair: cool | 22:19 |
| mordred | jeblair: lgtm | 22:19 |
| *** ianychoi has quit IRC | 22:19 | |
| melwitt | mordred: awesome, much appreciated | 22:20 |
| *** gmann_afk is now known as gmann | 22:21 | |
| clarkb | inc0: fungi http://paste.openstack.org/show/628486/ thats the cause | 22:21 |
| clarkb | at least for the build centos job | 22:21 |
| clarkb | log files are currently massive so grepping took a while | 22:22 |
| fungi | clarkb: oh, that's the issue we had going on early today | 22:22 |
| fungi | when not all the zuul services got restarted onto the same version | 22:22 |
| clarkb | hrm thats from 1859UTC | 22:23 |
| fungi | but this result looks like it came up just a few hours ago | 22:23 |
| fungi | yeah | 22:23 |
| clarkb | which is I think roughly when those set of jobs ran | 22:23 |
| clarkb | ze10 is where I pulled that log | 22:23 |
| clarkb | zuul executor there has been running since the 2nd | 22:24 |
| clarkb | so I'm guessing it didn't get restarted? | 22:24 |
| *** askb_ has joined #openstack-infra | 22:24 | |
| fungi | yeah, i wonder if it didn't properly shutdown when someone issued a restart and never terminated the old daemon | 22:24 |
| fungi | oh! or maybe someone forgot we added a couple of executors recently-ish | 22:25 |
| * mordred puts mooney on that | 22:25 | |
| clarkb | 09 was restarted | 22:26 |
| clarkb | anyways, should I go ahead and stop start the service on ze10? | 22:26 |
| fungi | yes, please | 22:26 |
| clarkb | ok doing thatn ow | 22:26 |
| mordred | clarkb: ++ | 22:26 |
| fungi | all of that went on before i came online since i had errands to run this morning first thing | 22:27 |
| fungi | but i gathered from scrollback that the expectation was they were all restarted | 22:27 |
| clarkb | stop issued, now we wait for it to actually stop I think | 22:27 |
| clarkb | 5 S zuul 19918 1 26 80 0 - 1226849 - Dec02 ? 1-15:57:48 /usr/bin/python3 /usr/local/bin/zuul-executor fwiw | 22:27 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Endpoint get summit schedule empty spots https://review.openstack.org/526795 | 22:28 |
| *** askb_ has quit IRC | 22:29 | |
| mordred | clarkb, fungi, pabelanger, dmsimard: have a sec to review jeblair's https://review.openstack.org/#/c/521630/ real quick? | 22:30 |
| jeblair | mordred, dmsimard, tobiash: we're in a weird situation where zuul-web serves *some* static files, but not others. and we need rewrite rules to hit the ones that zuul-web serves, and the external ones to fall through. | 22:31 |
| *** askb_ has joined #openstack-infra | 22:31 | |
| jeblair | so we could add rewrite rules for all the static stuff that's currently in zuul-web, or we could do what the readme says and copy those files from the zuul install location into /var/www/zuul/static | 22:31 |
| jeblair | maybe, as much as i don't want to do it, we should go with the second thing for now just so we're not having to keep things quite as much in sync between zuul and puppet-zuul? | 22:32 |
| clarkb | mordred: jeblair will that apache vhost follow symlinks? and I guess we are just going to install things such that both versions would work? | 22:33 |
| tobiash | jeblair: you could symlink then | 22:33 |
| *** slaweq has quit IRC | 22:33 | |
| *** baoli has quit IRC | 22:33 | |
| clarkb | ze10 is starting zuul executor now | 22:33 |
| mordred | jeblair: I think the second thing for now makes sense to me - and also intended to be a temporary awkwardness | 22:34 |
| *** baoli has joined #openstack-infra | 22:34 | |
| jeblair | well, the static dir itself has subdirs to hold the external things, so i don't think we can symlink it in, i think we have to recursively copy | 22:34 |
| *** baoli has quit IRC | 22:35 | |
| *** slaweq has joined #openstack-infra | 22:36 | |
| jeblair | mordred, tobiash, clarkb: updated etherpad to maybe make clearer | 22:36 |
| openstackgerrit | Merged openstack-infra/project-config master: Grab package_ver from remote host https://review.openstack.org/526790 | 22:37 |
| clarkb | jeblair: is one used by v2 the other by v3? | 22:37 |
| jeblair | clarkb: nope, all v3 | 22:37 |
| clarkb | why do we need two angulars? | 22:37 |
| jeblair | those aren't two angulars | 22:37 |
| mordred | jeblair: yah - I think just copy | 22:37 |
| fungi | clarkb: oh, good point, do we need options +symlinksifownermatch ? | 22:37 |
| jeblair | one is angular. the other is our application. | 22:37 |
| clarkb | jeblair: ah | 22:37 |
| clarkb | fungi: ya I think we may if its not already allowed | 22:38 |
| jeblair | we might be able to smylink images javascripts and styles directories | 22:38 |
| clarkb | or make copies and don't symlink | 22:38 |
| jeblair | since all the files at the root are already covered by rewrite rules | 22:38 |
| fungi | ahh, or options followsymlinks might be safe enough in this case | 22:38 |
| tobiash | jeblair: you probably need to remove the last rewrite rule | 22:39 |
| jeblair | okay, i think i want to try symlinking those 3 dirs. i think that may be cleaner than copying. but if it goes south, copy is best. | 22:39 |
| jeblair | tobiash: yep | 22:39 |
| clarkb | inc0: I think you can recheck at this point and it should be fine | 22:40 |
| inc0 | thanks clarkb | 22:41 |
| *** jascott1 has quit IRC | 22:43 | |
| *** wolverineav has quit IRC | 22:45 | |
| *** wolverineav has joined #openstack-infra | 22:46 | |
| fungi | #status log old docs-draft volume deleted from static.openstack.org, and the recovered extents divvied up between the tarballs and logs volumes (now 0.5tib and 13.4tib respectively) | 22:47 |
| openstackstatus | fungi: finished logging | 22:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Update Zuulv3 url rewrites https://review.openstack.org/526796 | 22:48 |
| jeblair | tobiash, clarkb, dmsimard, mordred, fungi: ^ | 22:48 |
| jeblair | i think there's one more change needed to system-config. i'm on that now | 22:48 |
| *** hashar has joined #openstack-infra | 22:49 | |
| *** bobh has quit IRC | 22:49 | |
| *** yamamoto has joined #openstack-infra | 22:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Update zuul_web_url https://review.openstack.org/526797 | 22:51 |
| jeblair | clarkb, fungi: the existing content in /var/lib/zuul/www is almost entirely symlinks | 22:52 |
| fungi | oh, and worknig? | 22:53 |
| fungi | wfm then | 22:53 |
| jeblair | yep | 22:53 |
| jeblair | i think it's set in apache2.conf | 22:54 |
| *** yamamoto has quit IRC | 22:54 | |
| fungi | indeed | 22:55 |
| fungi | <Directory /> Options FollowSymLinks ... | 22:55 |
| fungi | <Directory /var/www/> Options Indexes FollowSymLinks ... | 22:55 |
| fungi | so should be covered | 22:56 |
| mordred | jeblair, fungi: lgtm - gonna +A it | 22:56 |
| fungi | thanks! | 22:56 |
| *** wolverineav has quit IRC | 23:00 | |
| *** huanxie has quit IRC | 23:00 | |
| *** wolverineav has joined #openstack-infra | 23:00 | |
| *** askb_ has quit IRC | 23:02 | |
| *** askb_ has joined #openstack-infra | 23:03 | |
| *** slaweq has quit IRC | 23:04 | |
| *** wolverineav has quit IRC | 23:05 | |
| *** bobh has joined #openstack-infra | 23:05 | |
| fungi | mordred: we got slightly further now, but this error is beyond my ken: http://logs.openstack.org/b1/b18a06caa5955d6b72535cdb88e60b09f3e692d0/release/release-openstack-javascript/6faa118/job-output.txt.gz#_2017-12-08_23_03_14_058095 | 23:07 |
| fungi | er, farther | 23:07 |
| mordred | fungi: well! that's exciting | 23:08 |
| clarkb | I'm going back to zombie mode now | 23:08 |
| jeblair | fungi: can you +3 https://review.openstack.org/526797 ? | 23:09 |
| *** bobh has quit IRC | 23:09 | |
| fungi | yup. lgtm | 23:10 |
| *** esberglu has quit IRC | 23:10 | |
| fungi | mordred: maybe https://github.com/npm/npm/issues/16723 ? | 23:11 |
| fungi | maybe we need newer npm | 23:11 |
| *** baoli has joined #openstack-infra | 23:11 | |
| *** signed8b_ has joined #openstack-infra | 23:12 | |
| fungi | or older npm | 23:12 |
| fungi | or to take up dirt farming | 23:13 |
| bkero | a respectable profession | 23:14 |
| bkero | something that you can be proud of | 23:14 |
| mordred | fungi: yup | 23:14 |
| mordred | fungi: that would be it - and I have now read the entire thing ... we're using 5.5.1 in that job. 5.5.1-canary.8 seems to contain the fix - I am not sure if 5.5.1-canary.8 is 5.5.1 + something else or not | 23:15 |
| *** signed8bit has quit IRC | 23:15 | |
| mordred | fungi: I believe in the short term we can update to install npm4 instead of npm5 | 23:15 |
| fungi | i'll give it a shot. patch otw | 23:15 |
| *** baoli has quit IRC | 23:16 | |
| *** jascott1 has joined #openstack-infra | 23:16 | |
| *** signed8b_ has quit IRC | 23:16 | |
| fungi | hrm, i have to figure out what version of node provides npm4 | 23:17 |
| *** bobh has joined #openstack-infra | 23:17 | |
| fungi | or do i just add a separate step to nvm install npm4 after installing node? | 23:17 |
| fungi | oh, i guess it would be npm install npm<5 after the nvm use node line? | 23:18 |
| mordred | fungi: oh - sorry - patch coming ... | 23:19 |
| fungi | ahh, i'll review then ;) | 23:19 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Install angular.js and other static files for zuul web https://review.openstack.org/521630 | 23:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Pin to npm4 until npm 5.6.0 comes out https://review.openstack.org/526799 | 23:22 |
| *** bobh has quit IRC | 23:22 | |
| mordred | fungi: I believe that should do it | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Update Zuulv3 url rewrites https://review.openstack.org/526796 | 23:22 |
| fungi | eww, you have to invoke it as node_modules/.bin/npm? | 23:23 |
| mordred | fungi: that basically uses npm5 from the nvm node installation to install npm4 into the local dir, then uses that copy of npm to upload the tarball | 23:23 |
| fungi | got it | 23:23 |
| fungi | rather than asking npm5 to downgrade itself | 23:23 |
| mordred | fungi: yah - ./node_modules/.bin is the equiv of ~/.local/bin from pip install --user | 23:23 |
| mordred | fungi: so basically the default behavior of "npm install" is to install things into node_modules in the current directory, with the .bin dir inside that being where it puts any executables | 23:24 |
| fungi | i see | 23:24 |
| * mordred has learned things | 23:25 | |
| *** Wei_Liu has quit IRC | 23:27 | |
| *** rbrndt_ has joined #openstack-infra | 23:31 | |
| *** rbrndt has quit IRC | 23:31 | |
| *** bobh has joined #openstack-infra | 23:32 | |
| fungi | jeblair: i approved 526797 already (about 20 minutes before you). apologies for not mentioning in-channel | 23:33 |
| jeblair | fungi: i don't think zuul picked it up, so i reapproved | 23:34 |
| jeblair | we should check and see if puppet-zuul + system-config share a queue | 23:35 |
| fungi | ahh, yep, saw that in the timeline once i unhid the ci comments | 23:35 |
| fungi | oh! right | 23:35 |
| fungi | if they don't, it won't automatically enqueue if the dependency isn't meged | 23:35 |
| fungi | merged | 23:35 |
| *** bobh has quit IRC | 23:37 | |
| *** yamahata has joined #openstack-infra | 23:37 | |
| *** felipemonteiro has joined #openstack-infra | 23:39 | |
| *** xarses has quit IRC | 23:44 | |
| *** wolverineav has joined #openstack-infra | 23:44 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update zuul_web_url https://review.openstack.org/526797 | 23:46 |
| *** hashar has quit IRC | 23:47 | |
| *** markvoelker has quit IRC | 23:49 | |
| *** markvoelker has joined #openstack-infra | 23:50 | |
| jeblair | okay, i'll try to get that in place on zuulv3.o.o now | 23:50 |
| *** yamamoto has joined #openstack-infra | 23:51 | |
| *** markvoelker has quit IRC | 23:54 | |
| *** rbrndt_ has quit IRC | 23:54 | |
| jeblair | fungi, clarkb, mordred, dmsimard, tobiash, tristanC: i think we have a zuul dashboard! :) | 23:55 |
| *** yamamoto has quit IRC | 23:55 | |
| * fungi rejoices with beer | 23:56 | |
| jeblair | http://zuulv3.openstack.org/builds.html?job_name=announce-release | 23:56 |
| mwhahaha | :o | 23:57 |
| mwhahaha | yay | 23:57 |
| fungi | http://zuulv3.openstack.org/builds.html?job_name=release-openstack-javascript | 23:57 |
| jeblair | that looks useful and relevant! | 23:58 |
| fungi | recently so, even | 23:58 |
| fungi | 526799 should (hopefully) finally fix it | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!