| *** diablo_rojo has quit IRC | 00:00 | |
| *** ltomasbo has joined #openstack-infra | 00:00 | |
| *** jamesmcarthur has quit IRC | 00:02 | |
| *** jamesmcarthur has joined #openstack-infra | 00:03 | |
| *** rcernin_ has joined #openstack-infra | 00:04 | |
| *** rcernin has quit IRC | 00:04 | |
| *** dingyichen has joined #openstack-infra | 00:04 | |
| openstackgerrit | Merged openstack-infra/puppet-gerrit master: Update Gemfile for Zuulv3 https://review.openstack.org/582018 | 00:04 |
|---|---|---|
| *** jamesmca_ has joined #openstack-infra | 00:07 | |
| clarkb | fedora 28 is updated across the board and should be using new bindep | 00:08 |
| *** jamesmcarthur has quit IRC | 00:08 | |
| ianw | clarkb: oh, did it break? | 00:08 |
| * ianw still going through emails | 00:09 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config master: Run propose-update-constraints under Python3.6 https://review.openstack.org/583065 | 00:10 |
| clarkb | ianw: no, but I wanted to get an image with new bindep pushed out sooner than later | 00:10 |
| clarkb | ianw: we added support for alpine and in the process switched from using lsb_release to python distro library | 00:10 |
| clarkb | should be backward compatible and rely on the unittests a bit to ensure that using /etc/os-release fixtures | 00:13 |
| clarkb | but want to catch it early if there are problems | 00:13 |
| *** linkmark has quit IRC | 00:13 | |
| ianw | oh right, yeah saw that fly by. i guess distro just wasn't a think when bindep started? | 00:15 |
| *** Goneri has joined #openstack-infra | 00:15 | |
| clarkb | ianw: yes, I think it is somewhat newer | 00:15 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: sql: add default_url column https://review.openstack.org/583066 | 00:16 |
| fungi | i can confirm, the distro package was either nonexistent or nowhere near usable when we started bindep | 00:18 |
| ianw | another example of openstack solving problems people didn't even know they had ;) | 00:21 |
| clarkb | packethost has updaed to use 100 test instances | 00:21 |
| ianw | yay! i saw that ipv6 was next on the list | 00:22 |
| clarkb | of my todo list today the only one remaining is to followup with mnaser about vexxhost max servers | 00:23 |
| clarkb | and topic:puppet-4 mass approvals for gemfile updates are mostly done | 00:23 |
| clarkb | need to review the changes that add tests and similar though | 00:23 |
| clarkb | ianw: my only real concern with bindep change is we missed a case where os-release and lsb_release differ in naming and we didn't properly map between them. Definitely worked to map between them though | 00:27 |
| ianw | astroid requires Python '>=3.4.*' but the running Python is 2.7.12 | 00:27 |
| ianw | ^ do we know about this? | 00:28 |
| clarkb | ianw: where do you see that? | 00:28 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Summit entity event serialization issue https://review.openstack.org/583067 | 00:29 |
| clarkb | I've got to do a grocery run | 00:29 |
| ianw | clarkb: latest results on https://review.openstack.org/#/c/542350/ ... but could be job specific, i'll look a little harder, i just wondered if it was known issue | 00:29 |
| ianw | it looks like "astroid 2.0" released yesterday | 00:31 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Summit entity event serialization issue https://review.openstack.org/583067 | 00:32 |
| *** aeng has quit IRC | 00:32 | |
| ianw | oh, chatter in openstack-requirements anyway | 00:33 |
| fungi | yeah, there was some discussion of that earlier in my day before i woke up | 00:35 |
| *** Goneri has quit IRC | 00:37 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Update pylint to 1.7.6 https://review.openstack.org/583068 | 00:37 |
| *** bobh has quit IRC | 00:38 | |
| *** bobh has joined #openstack-infra | 00:38 | |
| *** bobh has quit IRC | 00:43 | |
| *** eernst has quit IRC | 00:44 | |
| *** hongbin has joined #openstack-infra | 00:44 | |
| *** hongbin_ has joined #openstack-infra | 00:56 | |
| *** jamesmca_ has quit IRC | 00:57 | |
| *** hongbin has quit IRC | 00:58 | |
| *** jamesmcarthur has joined #openstack-infra | 01:03 | |
| *** jamesmcarthur has quit IRC | 01:12 | |
| *** sthussey has quit IRC | 01:17 | |
| *** mrsoul_ has joined #openstack-infra | 01:23 | |
| *** manisht has quit IRC | 01:24 | |
| openstackgerrit | Fatema Khalid Sherif proposed openstack-infra/storyboard-webclient master: Expose ordering criteria in search UI https://review.openstack.org/577870 | 01:24 |
| *** mschuppert has quit IRC | 01:27 | |
| *** jamesmcarthur has joined #openstack-infra | 01:32 | |
| *** mriedem has quit IRC | 01:36 | |
| *** jamesmcarthur has quit IRC | 01:36 | |
| *** hongbin_ has quit IRC | 01:45 | |
| *** hongbin has joined #openstack-infra | 01:45 | |
| *** diablo_rojo has joined #openstack-infra | 01:46 | |
| *** yamahata has quit IRC | 01:54 | |
| *** roman_g has quit IRC | 01:55 | |
| *** iyamahat_ has quit IRC | 01:56 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Update pylint to 1.7.6, uncap networkx https://review.openstack.org/583068 | 01:57 |
| *** agopi is now known as agopi|afk | 02:01 | |
| openstackgerrit | Tony Breeds proposed openstack/diskimage-builder master: Support networkx 2.1 https://review.openstack.org/506524 | 02:04 |
| *** dave-mccowan has joined #openstack-infra | 02:08 | |
| *** jiapei has joined #openstack-infra | 02:10 | |
| *** armax has quit IRC | 02:16 | |
| *** zhangfei has joined #openstack-infra | 02:19 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Update pylint to 1.7.6, uncap networkx https://review.openstack.org/583068 | 02:21 |
| ianw | tonyb: ^ will try that ... | 02:21 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Update pylint to 1.7.6, uncap networkx https://review.openstack.org/583068 | 02:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Update pylint to 1.7.6, uncap networkx https://review.openstack.org/583068 | 02:24 |
| openstackgerrit | Tony Breeds proposed openstack/diskimage-builder master: Support networkx 2.1 https://review.openstack.org/506524 | 02:24 |
| *** dave-mccowan has quit IRC | 02:25 | |
| *** dave-mcc_ has joined #openstack-infra | 02:26 | |
| openstackgerrit | Fatema Khalid Sherif proposed openstack-infra/storyboard-webclient master: Exposing task creator name in search results https://review.openstack.org/583085 | 02:27 |
| *** rlandy has quit IRC | 02:28 | |
| openstackgerrit | Tony Breeds proposed openstack/diskimage-builder master: Support networkx 2.1 https://review.openstack.org/506524 | 02:28 |
| *** jamesmcarthur has joined #openstack-infra | 02:29 | |
| *** _ari_ has quit IRC | 02:30 | |
| *** weshay has quit IRC | 02:30 | |
| *** kzaitsev_pi has quit IRC | 02:30 | |
| *** _ari_ has joined #openstack-infra | 02:31 | |
| *** rasca has quit IRC | 02:31 | |
| *** rnoriega has quit IRC | 02:31 | |
| *** pabelanger has quit IRC | 02:31 | |
| *** _ari_ has quit IRC | 02:31 | |
| *** _ari_ has joined #openstack-infra | 02:31 | |
| *** pabelanger has joined #openstack-infra | 02:32 | |
| *** rasca has joined #openstack-infra | 02:32 | |
| *** rnoriega has joined #openstack-infra | 02:32 | |
| *** weshay has joined #openstack-infra | 02:32 | |
| *** jamesmcarthur has quit IRC | 02:33 | |
| *** aeng has joined #openstack-infra | 02:40 | |
| *** diablo_rojo has quit IRC | 02:42 | |
| *** psachin` has joined #openstack-infra | 02:44 | |
| *** diablo_rojo has joined #openstack-infra | 02:48 | |
| openstackgerrit | Tony Breeds proposed openstack/diskimage-builder master: Update pylint to 1.7.6, uncap networkx https://review.openstack.org/583068 | 02:48 |
| *** vaidy has joined #openstack-infra | 02:49 | |
| *** isviridov_away has joined #openstack-infra | 02:50 | |
| openstackgerrit | Tony Breeds proposed openstack/diskimage-builder master: Update pylint to 1.7.6, uncap networkx https://review.openstack.org/583068 | 02:51 |
| *** lbragstad has quit IRC | 03:05 | |
| *** jamesmcarthur has joined #openstack-infra | 03:11 | |
| openstackgerrit | Tony Breeds proposed openstack/diskimage-builder master: Support networkx 2.1 https://review.openstack.org/506524 | 03:13 |
| *** jamesmcarthur has quit IRC | 03:16 | |
| *** ramishra has joined #openstack-infra | 03:21 | |
| *** jamesmcarthur has joined #openstack-infra | 03:23 | |
| *** janki has joined #openstack-infra | 03:25 | |
| *** jamesmcarthur has quit IRC | 03:29 | |
| *** dave-mcc_ has quit IRC | 03:30 | |
| *** janki has quit IRC | 03:31 | |
| *** janki has joined #openstack-infra | 03:35 | |
| *** diablo_rojo has quit IRC | 03:40 | |
| *** lpetrut has joined #openstack-infra | 03:49 | |
| *** sshnaidm|bbl has quit IRC | 03:49 | |
| *** udesale has joined #openstack-infra | 03:50 | |
| *** sshnaidm|bbl has joined #openstack-infra | 03:50 | |
| *** sshnaidm|bbl has quit IRC | 03:56 | |
| *** sshnaidm|bbl has joined #openstack-infra | 03:57 | |
| *** ykarel|away has joined #openstack-infra | 04:00 | |
| *** hongbin has quit IRC | 04:08 | |
| openstackgerrit | Tony Breeds proposed openstack/diskimage-builder master: Update pylint to 1.7.6, uncap networkx https://review.openstack.org/583068 | 04:18 |
| openstackgerrit | Tony Breeds proposed openstack/diskimage-builder master: Support networkx 2.1 https://review.openstack.org/506524 | 04:18 |
| *** Bhujay has joined #openstack-infra | 04:22 | |
| *** rkukura has joined #openstack-infra | 04:26 | |
| *** ykarel|away is now known as ykarel | 04:29 | |
| *** aeng has quit IRC | 04:30 | |
| *** eernst has joined #openstack-infra | 04:30 | |
| *** jchhatbar has joined #openstack-infra | 04:31 | |
| *** janki has quit IRC | 04:31 | |
| *** jchhatbar has quit IRC | 04:32 | |
| *** cshastri has joined #openstack-infra | 04:32 | |
| *** jchhatbar has joined #openstack-infra | 04:33 | |
| *** lpetrut has quit IRC | 04:39 | |
| *** dhajare has joined #openstack-infra | 04:51 | |
| *** dhajare has quit IRC | 04:55 | |
| *** eernst has quit IRC | 05:01 | |
| *** dhajare has joined #openstack-infra | 05:08 | |
| *** aeng has joined #openstack-infra | 05:10 | |
| *** ykarel has quit IRC | 05:15 | |
| *** e0ne has joined #openstack-infra | 05:18 | |
| *** e0ne has quit IRC | 05:20 | |
| *** kjackal has joined #openstack-infra | 05:23 | |
| *** rcernin_ has quit IRC | 05:25 | |
| *** rcernin has joined #openstack-infra | 05:26 | |
| *** links has joined #openstack-infra | 05:33 | |
| dirk | infra-root: using codesearch.o.o gives (occasionally): ERROR:open /home/hound/data/idx-51adabe5f18b92d1/raw/LICENSE: too many open files | 05:34 |
| *** udesale_ has joined #openstack-infra | 05:35 | |
| *** udesale has quit IRC | 05:35 | |
| *** udesale_ has quit IRC | 05:40 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: make ZuulService load info api to manage paths https://review.openstack.org/579418 | 05:44 |
| *** Bhujay has quit IRC | 05:44 | |
| *** quiquell|off is now known as quiquell | 05:45 | |
| *** ykarel has joined #openstack-infra | 05:46 | |
| ianw | dirk: is it on a very large search result? | 05:46 |
| ianw | or to say another way, a fairly generic search term | 05:46 |
| dirk | ianw: no, actually the result set is zero | 05:47 |
| dirk | I was looking for 'exercises.sh' instead of 'exercise.sh' (which finds hits) | 05:48 |
| dirk | If you look for the former, about half the time the error message from above pops up | 05:49 |
| ianw | # lsof | grep houndd | wc -l | 05:50 |
| ianw | 52340 | 05:50 |
| ianw | hrm | 05:50 |
| *** stakeda has joined #openstack-infra | 05:51 | |
| *** tianhui_ has joined #openstack-infra | 05:56 | |
| ianw | # ls /home/hound/data/idx-* -ld | wc -l | 05:58 |
| ianw | 1446 | 05:59 |
| *** tianhui has quit IRC | 05:59 | |
| *** alex_xu has quit IRC | 05:59 | |
| ianw | it seems to hold these files open. it's file limit is 2048 so it seems under, but maybe there's leaks. something to look at next time it happens | 05:59 |
| *** alex_xu has joined #openstack-infra | 06:02 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/583111 | 06:03 |
| *** kjackal has quit IRC | 06:09 | |
| *** udesale_ has joined #openstack-infra | 06:11 | |
| *** janki has joined #openstack-infra | 06:13 | |
| *** jchhatbar has quit IRC | 06:14 | |
| *** janki has quit IRC | 06:15 | |
| *** mrsoul_ is now known as mschuppert | 06:15 | |
| *** janki has joined #openstack-infra | 06:15 | |
| *** kjackal has joined #openstack-infra | 06:23 | |
| *** andreww has joined #openstack-infra | 06:28 | |
| *** Bhujay has joined #openstack-infra | 06:28 | |
| *** xarses_ has quit IRC | 06:29 | |
| *** Douhet has quit IRC | 06:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/583111 | 06:30 |
| *** Douhet has joined #openstack-infra | 06:30 | |
| *** aojea has joined #openstack-infra | 06:31 | |
| *** janki has quit IRC | 06:35 | |
| dirk | ianw: i guess the daemon isn't restarted on index update? | 06:36 |
| *** janki has joined #openstack-infra | 06:36 | |
| *** dbecker has joined #openstack-infra | 06:36 | |
| *** alex_xu has quit IRC | 06:36 | |
| *** alex_xu has joined #openstack-infra | 06:36 | |
| openstackgerrit | Michel Peterson proposed openstack-infra/project-config master: Add Fluorine functional to grafana and remove fullstack https://review.openstack.org/582749 | 06:42 |
| *** andreas_s has joined #openstack-infra | 06:42 | |
| ianw | dirk: it should restart http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/resync-hound-config.sh | 06:44 |
| *** tesseract has joined #openstack-infra | 06:48 | |
| *** lpetrut has joined #openstack-infra | 06:50 | |
| *** yamamoto has joined #openstack-infra | 07:02 | |
| *** pgadiya has joined #openstack-infra | 07:03 | |
| *** pgadiya has quit IRC | 07:03 | |
| *** peereb has joined #openstack-infra | 07:04 | |
| *** shardy has joined #openstack-infra | 07:06 | |
| *** jcoufal has joined #openstack-infra | 07:09 | |
| *** ccamacho has joined #openstack-infra | 07:09 | |
| *** rcernin has quit IRC | 07:11 | |
| *** jchhatbar has joined #openstack-infra | 07:15 | |
| *** janki has quit IRC | 07:16 | |
| *** kjackal_ has joined #openstack-infra | 07:17 | |
| *** kjackal has quit IRC | 07:17 | |
| *** kzaitsev_pi has joined #openstack-infra | 07:17 | |
| *** Bhujay has quit IRC | 07:21 | |
| *** zoli is now known as zoli|wfh | 07:24 | |
| *** zoli|wfh is now known as zoli | 07:25 | |
| *** jchhatbar has quit IRC | 07:34 | |
| *** hashar has joined #openstack-infra | 07:38 | |
| *** tosky has joined #openstack-infra | 07:38 | |
| *** pguimaraes has joined #openstack-infra | 07:39 | |
| *** andreas_s has quit IRC | 07:41 | |
| *** florianf has joined #openstack-infra | 07:46 | |
| *** florianf has quit IRC | 07:47 | |
| *** florianf has joined #openstack-infra | 07:47 | |
| *** rfolco__ has joined #openstack-infra | 07:47 | |
| *** alexchadin has joined #openstack-infra | 07:49 | |
| *** rfolco_ has quit IRC | 07:50 | |
| *** jpich has joined #openstack-infra | 07:50 | |
| *** dpawlik has joined #openstack-infra | 07:51 | |
| *** janki has joined #openstack-infra | 07:51 | |
| *** yamahata has joined #openstack-infra | 07:53 | |
| *** derekh has joined #openstack-infra | 07:55 | |
| *** sshnaidm|bbl is now known as sshnaidm|rover | 07:56 | |
| *** gongysh has joined #openstack-infra | 07:57 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant yaml validation option to zuul client https://review.openstack.org/574265 | 08:00 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: make ZuulService load info api to manage paths https://review.openstack.org/579418 | 08:02 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add selenium tests for the status page https://review.openstack.org/579039 | 08:02 |
| *** vabada has quit IRC | 08:05 | |
| *** e0ne has joined #openstack-infra | 08:05 | |
| *** iyamahat has joined #openstack-infra | 08:05 | |
| *** vabada has joined #openstack-infra | 08:06 | |
| *** andreas_s has joined #openstack-infra | 08:06 | |
| *** niedbalski has quit IRC | 08:07 | |
| *** shoogz has quit IRC | 08:07 | |
| *** russellb has quit IRC | 08:07 | |
| *** vabada has quit IRC | 08:07 | |
| *** vabada has joined #openstack-infra | 08:08 | |
| *** russellb has joined #openstack-infra | 08:09 | |
| *** iyamahat has quit IRC | 08:09 | |
| *** vabada has quit IRC | 08:10 | |
| *** niedbalski has joined #openstack-infra | 08:10 | |
| *** vabada has joined #openstack-infra | 08:10 | |
| *** shoogz has joined #openstack-infra | 08:10 | |
| *** electrofelix has joined #openstack-infra | 08:10 | |
| *** vabada has quit IRC | 08:12 | |
| *** vabada has joined #openstack-infra | 08:13 | |
| *** andreas_s has quit IRC | 08:13 | |
| *** andreas_s has joined #openstack-infra | 08:13 | |
| *** roman_g has joined #openstack-infra | 08:16 | |
| *** janki has quit IRC | 08:18 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: sql: add default_url column https://review.openstack.org/583066 | 08:20 |
| *** shardy has quit IRC | 08:24 | |
| *** Bhujay has joined #openstack-infra | 08:28 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/devstack-gate master: Remove support for devstack exercises https://review.openstack.org/583146 | 08:30 |
| openstackgerrit | Frank Kloeker proposed openstack-infra/project-config master: Add patch submitting for project doc translation https://review.openstack.org/581000 | 08:31 |
| *** pbourke has quit IRC | 08:35 | |
| *** Bhujay has quit IRC | 08:35 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/system-config master: Remove mention of excercises.sh https://review.openstack.org/583147 | 08:35 |
| *** shardy has joined #openstack-infra | 08:37 | |
| openstackgerrit | Merged openstack-infra/puppet-translation_checksite master: Update Gemfile for Zuulv3 https://review.openstack.org/582071 | 08:38 |
| *** yamahata has quit IRC | 08:40 | |
| *** ralonsoh has joined #openstack-infra | 08:44 | |
| *** alex_xu has quit IRC | 08:45 | |
| *** alex_xu has joined #openstack-infra | 08:46 | |
| *** pbourke has joined #openstack-infra | 08:52 | |
| *** zhangfei has quit IRC | 08:53 | |
| *** dingyichen has quit IRC | 08:55 | |
| *** priteau has joined #openstack-infra | 08:55 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/devstack-gate master: Read ntp.conf as root to avoid permission issues https://review.openstack.org/486388 | 08:57 |
| *** zhangfei has joined #openstack-infra | 09:05 | |
| *** zhangfei has quit IRC | 09:10 | |
| *** derekh has quit IRC | 09:17 | |
| *** derekh has joined #openstack-infra | 09:17 | |
| *** gfidente has joined #openstack-infra | 09:18 | |
| *** gfidente has joined #openstack-infra | 09:18 | |
| *** annp has quit IRC | 09:18 | |
| *** ykarel has quit IRC | 09:23 | |
| *** annp has joined #openstack-infra | 09:26 | |
| *** zhangfei has joined #openstack-infra | 09:31 | |
| *** yamamoto has quit IRC | 09:37 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: trivial: Remove 'tools/releasenotes_tox.sh' https://review.openstack.org/580414 | 09:39 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: trivial: Fix file permissions https://review.openstack.org/580415 | 09:39 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Switch to stestr https://review.openstack.org/580413 | 09:39 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: builddoc: Remove support for Sphinx < 1.6 https://review.openstack.org/580416 | 09:39 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: packaging: Remove support for pyN requirement files https://review.openstack.org/580417 | 09:39 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: doc: Fix formatting of packagers guide https://review.openstack.org/580418 | 09:39 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: doc: Add documentation for missing '[pbr]' options https://review.openstack.org/580419 | 09:39 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Add Sphinx extension https://review.openstack.org/580420 | 09:39 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Deprecate 'build_sphinx' integration https://review.openstack.org/580421 | 09:39 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Deprecate 'test' integration https://review.openstack.org/580422 | 09:39 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: trivial: Increase integration test timeout https://review.openstack.org/583178 | 09:39 |
| *** dtantsur|afk is now known as dtantsur | 09:39 | |
| *** Goneri has joined #openstack-infra | 09:40 | |
| *** ykarel has joined #openstack-infra | 09:44 | |
| *** ykarel is now known as ykarel|away | 09:48 | |
| *** Goneri has quit IRC | 09:54 | |
| *** ykarel|away has quit IRC | 09:57 | |
| *** shardy_ has joined #openstack-infra | 09:59 | |
| *** shardy has quit IRC | 10:02 | |
| *** dpawlik has quit IRC | 10:07 | |
| *** dpawlik has joined #openstack-infra | 10:09 | |
| *** andreas_s has quit IRC | 10:16 | |
| *** markmcd has quit IRC | 10:22 | |
| *** pguimaraes has quit IRC | 10:22 | |
| *** markmcd has joined #openstack-infra | 10:33 | |
| *** dpawlik has quit IRC | 10:34 | |
| *** gongysh has quit IRC | 10:36 | |
| *** dpawlik has joined #openstack-infra | 10:38 | |
| *** zoli is now known as zoli|lunch | 10:41 | |
| *** boden has joined #openstack-infra | 10:44 | |
| *** pguimaraes has joined #openstack-infra | 10:47 | |
| *** kjackal_ has quit IRC | 10:58 | |
| *** rosmaita has joined #openstack-infra | 10:58 | |
| *** sambetts|afk is now known as sambetts | 10:59 | |
| *** rcarrill1 is now known as rcarrillocruz | 11:02 | |
| *** andreas_s has joined #openstack-infra | 11:03 | |
| *** kjackal has joined #openstack-infra | 11:09 | |
| *** andreas_s has quit IRC | 11:09 | |
| *** andreas_s has joined #openstack-infra | 11:10 | |
| *** udesale_ has quit IRC | 11:12 | |
| slaweq | hi | 11:14 |
| slaweq | I am trying to figure out why sometimes UT in neutron are failing because of timeouts, it's reported in https://bugs.launchpad.net/neutron/+bug/1779077 | 11:14 |
| openstack | Launchpad bug 1779077 in neutron "Unit test jobs (py35) fails with timeout often" [High,Confirmed] - Assigned to Slawek Kaplonski (slaweq) | 11:14 |
| slaweq | what I found is that in such failed runs some tests (maybe all, I'm not sure) are just going slower | 11:15 |
| *** d0ugal has quit IRC | 11:15 | |
| slaweq | and I also found that similar timeouts happens not only for neutron, see https://bit.ly/2JuNljk and check results from last week for example | 11:16 |
| slaweq | so if same issue happens for more than one project, maybe it's somehow related to infra, e.g. slow nodes or something like that? do You know anything about it maybe? | 11:17 |
| *** dave-mccowan has joined #openstack-infra | 11:18 | |
| *** d0ugal has joined #openstack-infra | 11:18 | |
| *** d0ugal has quit IRC | 11:18 | |
| *** d0ugal has joined #openstack-infra | 11:18 | |
| *** derekh has quit IRC | 11:22 | |
| *** lennyb has quit IRC | 11:23 | |
| *** andreas_s has quit IRC | 11:23 | |
| *** jamesdenton has joined #openstack-infra | 11:26 | |
| *** ldnunes has joined #openstack-infra | 11:28 | |
| *** andreas_s has joined #openstack-infra | 11:33 | |
| *** jluk has joined #openstack-infra | 11:38 | |
| *** janki has joined #openstack-infra | 11:39 | |
| *** andreas_s has quit IRC | 11:42 | |
| *** rh-jelabarre has joined #openstack-infra | 11:49 | |
| *** andreas_s has joined #openstack-infra | 11:51 | |
| *** zoli|lunch is now known as zoli | 11:52 | |
| *** zoli is now known as zoli|wfh | 11:53 | |
| *** rpioso|afk is now known as rpioso | 11:56 | |
| *** eharney has quit IRC | 11:58 | |
| *** andreas_s has quit IRC | 12:00 | |
| *** andreas_s has joined #openstack-infra | 12:02 | |
| *** agopi|afk has quit IRC | 12:02 | |
| *** roman_g has quit IRC | 12:05 | |
| *** alexchadin has quit IRC | 12:05 | |
| *** roman_g has joined #openstack-infra | 12:05 | |
| *** edmondsw has joined #openstack-infra | 12:06 | |
| *** zhangfei has quit IRC | 12:12 | |
| *** alexchadin has joined #openstack-infra | 12:13 | |
| *** wolverineav has joined #openstack-infra | 12:14 | |
| *** rfolco__ is now known as rfolco | 12:18 | |
| *** ralonsoh has quit IRC | 12:19 | |
| *** alexchadin has quit IRC | 12:25 | |
| *** rlandy has joined #openstack-infra | 12:28 | |
| *** zhangfei has joined #openstack-infra | 12:28 | |
| *** gongysh has joined #openstack-infra | 12:30 | |
| *** lbragstad has joined #openstack-infra | 12:31 | |
| *** andreas_s has quit IRC | 12:31 | |
| *** andreas_s has joined #openstack-infra | 12:32 | |
| *** andreas_s has quit IRC | 12:33 | |
| *** andreas_s has joined #openstack-infra | 12:33 | |
| *** camunoz has joined #openstack-infra | 12:34 | |
| *** agopi|afk has joined #openstack-infra | 12:34 | |
| *** agopi|afk is now known as agopi | 12:34 | |
| *** graphene has quit IRC | 12:36 | |
| *** graphene has joined #openstack-infra | 12:38 | |
| *** andreas_s has quit IRC | 12:38 | |
| *** camunoz has quit IRC | 12:40 | |
| *** lbragstad has quit IRC | 12:41 | |
| *** eharney has joined #openstack-infra | 12:42 | |
| *** udesale has joined #openstack-infra | 12:43 | |
| *** andreas_s has joined #openstack-infra | 12:43 | |
| *** andreas_s has quit IRC | 12:47 | |
| *** andreas_s has joined #openstack-infra | 12:49 | |
| *** andreas_s has quit IRC | 12:50 | |
| *** zhangfei has quit IRC | 12:50 | |
| *** camunoz has joined #openstack-infra | 12:55 | |
| *** peereb has quit IRC | 12:56 | |
| fungi | slaweq: the performance we get out of different service providers varies under different sorts of loads. does it seem to mostly impact one provider? | 12:56 |
| *** tosky has quit IRC | 12:56 | |
| *** tosky has joined #openstack-infra | 12:56 | |
| slaweq | fungi: no, problem is that it's not on nodes from one provider | 12:56 |
| fungi | some have slower storage than others, some slower processors (but in some cases more of them), some slower networking or have to communicate over greater distances to reach centralized resources, et cetera | 12:57 |
| *** andreas_s has joined #openstack-infra | 12:57 | |
| slaweq | fungi: ok, so there is nothing "special" now what could cause such issue and what You would be aware | 12:58 |
| slaweq | fungi: thx for confirmation | 12:58 |
| slaweq | I will continue checking it on neutron's side then | 12:58 |
| fungi | it _can_ also happen across different nova compute hosts in the same provider for a number of reasons (noisy neighbor problem, heterogenous hardware...) | 12:59 |
| fungi | i want to say a couple of weeks ago the tripleo team had run some basic stats on percentage of job timeouts seen by provider, which indicated some of our providers accounted for disproportionately larger proportions of our over-all total number of job timeouts | 13:00 |
| *** andreas_s has quit IRC | 13:01 | |
| *** andreas_s has joined #openstack-infra | 13:01 | |
| *** andreas_s has quit IRC | 13:01 | |
| fungi | it's also more likely to happen at certain peak loads since some of our providers are managed private clouds or have us quarantined to dedicated hardware with special scheduling, so we start seeing signs of oversubscription when running at quota | 13:02 |
| fungi | basically becoming our own noisy neighbors | 13:03 |
| *** tpsilva has joined #openstack-infra | 13:05 | |
| mordred | fungi: are we having known issues with the pypi mirror? | 13:11 |
| *** e0ne has quit IRC | 13:11 | |
| mordred | fungi: I'm still getting bindep 2.7 from our mirrors in a job instead of 2.8 | 13:11 |
| fungi | not known to me | 13:11 |
| mordred | awesome | 13:11 |
| *** andreas_s has joined #openstack-infra | 13:11 | |
| fungi | interesting. i'll take a quick look too | 13:11 |
| * mordred likes being interesting | 13:11 | |
| fungi | OSError: [Errno 122] Disk quota exceeded | 13:12 |
| mordred | \o/ | 13:12 |
| fungi | i'd call that a problem | 13:12 |
| fungi | time to refresh my memory on afs quota management | 13:12 |
| *** andreas_s has quit IRC | 13:14 | |
| *** andreas_s has joined #openstack-infra | 13:14 | |
| *** eharney has quit IRC | 13:16 | |
| *** zhangfei has joined #openstack-infra | 13:19 | |
| fungi | $ fs quota /afs/.openstack.org/mirror/pypi | 13:19 |
| fungi | 100% of quota used. | 13:19 |
| fungi | yup | 13:19 |
| fungi | 1999990055 of 2000000000 used | 13:20 |
| fungi | according to `fs listquota /afs/.openstack.org/mirror/pypi` | 13:20 |
| fungi | and 88% used on the partition | 13:20 |
| fungi | so probably can't/shouldn't bump it by too much | 13:20 |
| fungi | without adding another cinder volume to the vg and extending that lv | 13:21 |
| fungi | looking at the logs, i think we filled it up at 2018-07-16 06:25:28 | 13:23 |
| fungi | so this has been stuck for ~31 hours | 13:24 |
| *** Raybih has joined #openstack-infra | 13:25 | |
| *** graphene has quit IRC | 13:28 | |
| roman_g | Hello team. https://review.openstack.org/#/c/579434/ - "Use readthedocs webhook to trigger build" authored by Ian and has +2 from Joshua Hesketh - I would like to kindly ask you for a review and another +2 followed by merge. Pending since July, 3rd. Thank you! | 13:28 |
| *** graphene has joined #openstack-infra | 13:29 | |
| roman_g | I'm from Airship team, working on documentation publishing, and we are intresed in utilizing this patch set. | 13:29 |
| *** alexchadin has joined #openstack-infra | 13:30 | |
| *** eharney has joined #openstack-infra | 13:31 | |
| *** sthussey has joined #openstack-infra | 13:33 | |
| *** armax has joined #openstack-infra | 13:36 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Replace shade and os-client-config with openstacksdk. https://review.openstack.org/572829 | 13:38 |
| mordred | fungi: clarkb has been suggesting we cache fewer things in AFS and use the apache proxy cache for more | 13:41 |
| mordred | fungi: maybe it's worth considering shifting pypi from afs to apache | 13:41 |
| mordred | fungi: I tihnk we should continue to keep the main ubuntu archives in afs because of vos release | 13:41 |
| *** psachin` has quit IRC | 13:42 | |
| mordred | but maybe vos release isn't as important for pypi? | 13:42 |
| *** annp_ has joined #openstack-infra | 13:42 | |
| mordred | fungi: speaking of - https://review.openstack.org/#/c/580882/ and https://review.openstack.org/#/c/582584/1 would both be nice | 13:43 |
| *** owalsh has quit IRC | 13:49 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack-infra/bindep master: Add OracleLinux support https://review.openstack.org/536355 | 13:50 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: tox: default envlist to supported python versions https://review.openstack.org/583243 | 13:50 |
| *** mriedem has joined #openstack-infra | 13:51 | |
| *** gongysh has quit IRC | 13:52 | |
| *** bobh has joined #openstack-infra | 13:53 | |
| *** andreas_s has quit IRC | 13:55 | |
| *** dhajare has quit IRC | 13:56 | |
| *** radez has joined #openstack-infra | 13:57 | |
| mark-burnett | Hey, can we get another review on adding an airship specs repo please? https://review.openstack.org/#/c/581369/ | 13:57 |
| *** bobh has quit IRC | 13:58 | |
| mordred | mark-burnett: done! | 13:58 |
| mark-burnett | thanks :) | 13:58 |
| *** hongbin has joined #openstack-infra | 14:00 | |
| *** gongysh has joined #openstack-infra | 14:01 | |
| roman_g | *thumbs up* | 14:02 |
| *** owalsh has joined #openstack-infra | 14:02 | |
| *** andreas_s has joined #openstack-infra | 14:05 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 14:06 | |
| *** jamesmcarthur has joined #openstack-infra | 14:09 | |
| *** andreas_s has quit IRC | 14:10 | |
| *** alexchad_ has joined #openstack-infra | 14:13 | |
| *** AhmadM_ has joined #openstack-infra | 14:13 | |
| *** andreas_s has joined #openstack-infra | 14:14 | |
| *** alexchadin has quit IRC | 14:14 | |
| *** alexchad_ has quit IRC | 14:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create airship-specs repo https://review.openstack.org/581369 | 14:18 |
| *** mihalis68 has joined #openstack-infra | 14:19 | |
| *** esarault has joined #openstack-infra | 14:20 | |
| *** Raybih has quit IRC | 14:21 | |
| clarkb | mordred: fungi if we rebuild pypi mirror with our package blacklist I expect the size to almost halve | 14:22 |
| clarkb | but that will take some time to rebuild | 14:22 |
| mordred | clarkb: I think I'd vote for turning it in to proxy cache | 14:23 |
| mordred | clarkb: the bandersnatch version has always been a bit fragile - and we don't have the index/content out-of-sync issue we have with apt | 14:23 |
| clarkb | ya and hit rate tends to be pretty good once caches warm (it like 98% for the not docker vhost) | 14:24 |
| *** graphene has quit IRC | 14:26 | |
| clarkb | I could go either way. Now that we proper mirror there isnt really any functionality we are adding in | 14:27 |
| clarkb | and the growth of the AI/ML and cuda linked libs is pretty insane on pypi | 14:27 |
| *** graphene has joined #openstack-infra | 14:28 | |
| *** alexchadin has joined #openstack-infra | 14:28 | |
| *** andreas_s has quit IRC | 14:28 | |
| *** bobh has joined #openstack-infra | 14:28 | |
| *** graphene has quit IRC | 14:28 | |
| mordred | yeah | 14:28 |
| *** andreas_s has joined #openstack-infra | 14:29 | |
| mordred | (obviously we should keep the wheel mirror as is) | 14:29 |
| fungi | also the fedora mirror is at 100% quota too | 14:29 |
| mordred | clarkb: would we just proxy /pypi to http://pypi.org do you think? | 14:29 |
| *** graphene has joined #openstack-infra | 14:30 | |
| fungi | i've temporarily switched gears anyway. turns out https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/5ER2YET54CSX4FV2VP24JA57REDDW5OI/ is something we need to address | 14:30 |
| clarkb | mordred: we can do that on port 8080 and test it then switch i on port 80 if we want to avoid explicitly migrating existing users to port 8080 | 14:30 |
| fungi | i've asked warehouse to send a verification e-mail to our primary (and only) address there... review@o.o, but that mailbox has ~20k messages worth of backscatter in it so taking a while to load | 14:31 |
| fungi | once i can get it verified i'll look at adding and changing the primary address for it | 14:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add proxy cache for PyPI https://review.openstack.org/583263 | 14:32 |
| mordred | clarkb: ^^ | 14:32 |
| clarkb | mordred: see line 164 | 14:33 |
| clarkb | we alreadydo something but needs an update for this | 14:33 |
| *** jcoufal_ has joined #openstack-infra | 14:34 | |
| *** quiquell is now known as quiquell|off | 14:35 | |
| *** jcoufal has quit IRC | 14:36 | |
| mordred | clarkb:ah - gotcha | 14:36 |
| *** graphene has quit IRC | 14:37 | |
| *** d0ugal has quit IRC | 14:37 | |
| *** esarault_ has joined #openstack-infra | 14:37 | |
| *** andreas_s has quit IRC | 14:37 | |
| *** andreas_s has joined #openstack-infra | 14:38 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add proxy cache for PyPI https://review.openstack.org/583263 | 14:38 |
| *** graphene has joined #openstack-infra | 14:39 | |
| *** esarault has quit IRC | 14:40 | |
| *** andreas_s has quit IRC | 14:41 | |
| *** d0ugal has joined #openstack-infra | 14:44 | |
| *** andreas_s has joined #openstack-infra | 14:46 | |
| *** cshastri has quit IRC | 14:46 | |
| *** dpawlik has quit IRC | 14:46 | |
| clarkb | mordred: +2 should be straightforward to test once updated as well | 14:51 |
| *** anteaya has joined #openstack-infra | 14:53 | |
| clarkb | fungi: if we need to we can always create a new account | 14:53 |
| *** sshnaidm|afk is now known as sshnaidm|rover | 14:54 | |
| clarkb | grafana shows that packethost seems to be reasonably healthy from nodepool's perspective | 14:56 |
| clarkb | there are some consistent boot failures we may want to look into | 14:56 |
| mordred | clarkb: stab. it doesn't work | 14:56 |
| mordred | clarkb: because, of course, the urls the index returns are like https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl | 14:56 |
| *** lbragstad has joined #openstack-infra | 14:57 | |
| clarkb | mordred: ya so we'll need to do like docker and proxy multiple urls for it to work | 14:57 |
| clarkb | it is frustrating that CDN ( at least in the docker case) isn't reliable for us but makes it more complicated to make it reliable | 14:57 |
| mordred | clarkb: but how do we get pip to use the proxy url for the pythonhosted urls? | 14:58 |
| *** kaiokmo has joined #openstack-infra | 14:58 | |
| clarkb | mordred: you point pip at your /pypi then proxy files.pythonhosted.org to /pythonhosted or similar | 14:58 |
| clarkb | mordred: as long as they use proper redirects the reverse proxying will rewrite those headers | 14:58 |
| mordred | clarkb: oh - you're thinking the links in the pypi index are "pypi.org/something" and then they have redirects to files.pythonhosted ? | 14:59 |
| clarkb | mordred: yes | 14:59 |
| clarkb | at least that is how docker does it | 14:59 |
| *** andreas_s has quit IRC | 14:59 | |
| mordred | clarkb: cause I just put the pypi proxy rule in a local apache | 14:59 |
| mordred | and it doesn't work | 14:59 |
| *** andreas_s has joined #openstack-infra | 14:59 | |
| mordred | it fetches straight from files.pythonhosted.org still | 14:59 |
| * clarkb has a meeting but will look at browser debug panel after | 15:00 | |
| *** annp_ has quit IRC | 15:01 | |
| clarkb | mordred: it looks like if you proxy files.pythonhosted.org too it would work | 15:02 |
| clarkb | indexes are at pypi.org and files at files.pythonhosted.org | 15:02 |
| mordred | ok. something may be wrong with my local config | 15:02 |
| *** alexchadin has quit IRC | 15:02 | |
| *** d0ugal has quit IRC | 15:03 | |
| *** mriedem has quit IRC | 15:04 | |
| fungi | #status log changed validated e-mail address for openstackci account on pypi per https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/5ER2YET54CSX4FV2VP24JA57REDDW5OI/ | 15:05 |
| openstackstatus | fungi: finished logging | 15:05 |
| *** mriedem has joined #openstack-infra | 15:05 | |
| mordred | clarkb: ok - my local testing was borked - testing better now | 15:06 |
| *** andreas_s has quit IRC | 15:07 | |
| *** ccamacho has quit IRC | 15:07 | |
| *** andreas_s has joined #openstack-infra | 15:08 | |
| clarkb | I wonder if bandersnatch could grow a mirror real releases only mode | 15:08 |
| clarkb | (another thought on how we might approach this without needing to curate it) | 15:08 |
| *** florianf has quit IRC | 15:09 | |
| *** zhangfei has quit IRC | 15:09 | |
| mordred | clarkb: the links from the simple page are directly to files.pythonhosted.org | 15:11 |
| fungi | #status log switched primary address for openstackci pypi account from review@o.o to infra-root@o.o so that it doesn't get mixed in with gerrit backscatter (we can switch to a dedicated alias later if needed) | 15:11 |
| openstackstatus | fungi: finished logging | 15:11 |
| clarkb | mordred: bah | 15:11 |
| mordred | clarkb: https://pypi.org/simple/keystoneauth/ | 15:11 |
| *** florianf has joined #openstack-infra | 15:11 | |
| mordred | clarkb: so to pass-through proxy that we'd need to rewrite the html being returned | 15:11 |
| clarkb | mordred: ya that is unfrotunate | 15:11 |
| mordred | which I *thought* proxypassreverse did - but I guess not | 15:12 |
| clarkb | it won't change the body only headers | 15:12 |
| mordred | that really only does headers doesn't it? | 15:12 |
| mordred | yeah | 15:12 |
| mordred | well that's just unfortunate | 15:12 |
| clarkb | we could run a layer 7 proxy but that gets complicated quickly | 15:13 |
| *** d0ugal has joined #openstack-infra | 15:14 | |
| mordred | yeah | 15:15 |
| mordred | 958386 | 15:15 |
| mordred | gah | 15:15 |
| *** dgonzalez has left #openstack-infra | 15:15 | |
| fungi | now mordred has been replaced by a numbers station | 15:15 |
| fungi | clarkb: so do you think we should (temporarily) bump the afs quotas for the pypi and fedora mirrors to get them back on track in the interim? | 15:16 |
| *** andreas_s has quit IRC | 15:16 | |
| clarkb | fungi: we can't bump pypi. That is the limit for an afs volume. We could delete large packages manually and just have broken links to them until we rebuild the mirror | 15:17 |
| fungi | oh, thanks. i didn't realize we were at max quota there | 15:17 |
| clarkb | for fedora I think we can bump but also maybe we can delete older release packages if not already done? | 15:17 |
| fungi | also, i'm legally changing my name to "max quota" | 15:18 |
| clarkb | fungi: ya its 2 terabytes | 15:18 |
| clarkb | we might be able to get a few more bytes if it is binary not decimal (not sure) | 15:18 |
| corvus | mordred, clarkb, fungi: https://httpd.apache.org/docs/2.4/mod/mod_substitute.html can rewrite bodies | 15:18 |
| *** rkukura has quit IRC | 15:19 | |
| mordred | corvus: ooh. I will try that | 15:19 |
| *** greghaynes has quit IRC | 15:19 | |
| *** dpawlik has joined #openstack-infra | 15:21 | |
| *** lpetrut has quit IRC | 15:23 | |
| *** andreas_s has joined #openstack-infra | 15:24 | |
| *** eernst has joined #openstack-infra | 15:25 | |
| *** dpawlik has quit IRC | 15:25 | |
| *** d0ugal has quit IRC | 15:26 | |
| mordred | corvus: wanna hear something fun? | 15:26 |
| mordred | corvus: https://pypi.org/simple/keystoneauth does not return back a content type header at all | 15:27 |
| corvus | no. nope. definitely not. | 15:27 |
| corvus | drat too late | 15:27 |
| *** esarault has joined #openstack-infra | 15:27 | |
| mordred | so AddOutputFilterByType SUBSTITUTE text/html ... sigh | 15:27 |
| *** myoung is now known as myoung|biaf | 15:27 | |
| corvus | wow :( | 15:27 |
| *** greghaynes has joined #openstack-infra | 15:29 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed issue on SS entity serialization https://review.openstack.org/583289 | 15:31 |
| mordred | corvus: I tried SetOutputFilter and that also did not work | 15:31 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed issue on SS entity serialization https://review.openstack.org/583289 | 15:31 |
| *** esarault_ has quit IRC | 15:31 | |
| mordred | corvus: but it's also entirely possible I'm doing something horribly wrong | 15:32 |
| corvus | mordred: i'm no expert, but if you paste your config snippet, maybe i'll see something? | 15:33 |
| mordred | yup - one sec, I'm removing the cache entries for simplifying | 15:33 |
| *** camunoz has quit IRC | 15:34 | |
| *** andreas_s has quit IRC | 15:34 | |
| *** andreas_s has joined #openstack-infra | 15:34 | |
| mordred | corvus: http://paste.openstack.org/show/726121/ | 15:35 |
| corvus | mordred: is that inside of a Location, Directory, or bare VirtualHost? | 15:36 |
| mordred | corvus: bare virtualhost | 15:36 |
| mordred | corvus: also, I did a2enmod substitute | 15:37 |
| *** r-daneel has joined #openstack-infra | 15:37 | |
| mordred | corvus: I'm testing by looking at http://localhost/pypi/keystoneauth/ fwiw | 15:37 |
| corvus | mordred: substitute says it's only valid in a 'directory' context: https://httpd.apache.org/docs/2.4/mod/directive-dict.html#Context | 15:37 |
| mordred | oh! wait | 15:37 |
| corvus | (which also includes location) | 15:37 |
| mordred | browser cache browser cache | 15:37 |
| mordred | it works fine when I try it from curl | 15:38 |
| corvus | though usually apache lets you know if you use something where it's not supported | 15:38 |
| corvus | so maybe it's fine, undocumented behavior :) | 15:38 |
| *** jistr is now known as jistr|afk | 15:38 | |
| mordred | BOOYAH - that totally worked | 15:39 |
| mordred | corvus: so - I guess we should probably put the /pypi proxy in its own virtualhost so that we don't try to substitute filter all of the content | 15:39 |
| clarkb | mordred: thats similar to why docker is its own port/vhost | 15:39 |
| mordred | (since we can't limit it by content type) | 15:39 |
| mordred | clarkb: yup | 15:39 |
| corvus | mordred: yeah, or maybe you can do <Location /pypi> ? | 15:39 |
| clarkb | mordred: we override caching rules for docker | 15:39 |
| mordred | clarkb: I can try that - lemme give it a stab | 15:40 |
| clarkb | and in latest docker it assumes it owns / | 15:40 |
| mordred | corvus, clarkb: putting it in Location seems to have worked | 15:41 |
| mordred | patch coming | 15:41 |
| *** yamahata has joined #openstack-infra | 15:41 | |
| *** felipemonteiro has joined #openstack-infra | 15:41 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config master: Add Grafana dashboards for stable Neutron releases https://review.openstack.org/578191 | 15:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add proxy cache for PyPI https://review.openstack.org/583263 | 15:44 |
| mordred | corvus, clarkb: that works for me locally | 15:44 |
| *** andreas_s has quit IRC | 15:44 | |
| mordred | and I have tested it both by looking at the html returned via curl, but also by running a pip install -v -v -v and watching where it fetches from | 15:45 |
| clarkb | mordred: you probably need to enable the mod in the .pp for the mirror hosts as well | 15:45 |
| corvus | mordred: if it's a patch to proxy nodepool nodes through your laptop, we're all set then! :) | 15:45 |
| mordred | clarkb: it's already enabled | 15:45 |
| mordred | clarkb: I'm not sure why it's already enabled - but it is :) | 15:45 |
| mordred | corvus: what's funnier is that back in the drizzle days some of the build nodes did actually sit at my house under my desk | 15:46 |
| corvus | clarkb: we simul+2d | 15:47 |
| clarkb | mordred: I +2'd willing to give that a try. Did you double check our xenial nodes have substitute enabled by default? | 15:47 |
| clarkb | mordred: my only concern is that we'll try to restart and it will fail then no mirrors for anything | 15:47 |
| *** andreas_s_ has joined #openstack-infra | 15:47 | |
| *** dbecker has quit IRC | 15:48 | |
| mordred | clarkb: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/mirror.pp#n303 | 15:48 |
| clarkb | huh | 15:48 |
| *** greghaynes has quit IRC | 15:49 | |
| mordred | corvus: while I've got your attention on this topic - feel like a 2nd +2 on 580882 and 582584 ? | 15:49 |
| mordred | clarkb: right? | 15:49 |
| *** jiapei has quit IRC | 15:49 | |
| corvus | mordred: is clarkb overthinking it in his comment on 882? | 15:49 |
| mordred | corvus: I think so | 15:50 |
| mordred | corvus: but I didn't want to say so where he could hear me | 15:50 |
| corvus | clarkb: you're over-thinking it :) | 15:50 |
| corvus | mordred: totes | 15:50 |
| * clarkb looks | 15:50 | |
| corvus | is the question: might we proxy the registry in the future? | 15:51 |
| *** andreas_s_ has quit IRC | 15:52 | |
| *** greghaynes has joined #openstack-infra | 15:52 | |
| *** andreas_s_ has joined #openstack-infra | 15:53 | |
| clarkb | corvus: no, we already project the registry but for reasons it is hosted on its own port and vhost without a path (beacuse docker client refuses to actually do http and will only talk to a rooted hostname) | 15:53 |
| clarkb | corvus: my concern is that if we have a /docker that some people might assume that will work as a registry proxy (but it will not) | 15:53 |
| corvus | oh, i see | 15:53 |
| corvus | clarkb: yeah, not a bad suggestion, but we all seem to fall on the +2 rather than -1 side of that fence so i'll +3 it | 15:54 |
| *** d0ugal has joined #openstack-infra | 15:56 | |
| *** tesseract has quit IRC | 15:57 | |
| *** andreas_s_ has quit IRC | 15:57 | |
| *** andreas_s has joined #openstack-infra | 15:59 | |
| *** dklyle has quit IRC | 16:00 | |
| mark-burnett | It looks like the airship-specs repo was successfully created - can we get reviews on adding zuul jobs? https://review.openstack.org/#/c/581470/ Thanks :) | 16:01 |
| roman_g | +1 | 16:01 |
| roman_g | https://review.openstack.org/#/c/579434/ - "Use readthedocs webhook to trigger build" authored by Ian and has +2 from Joshua Hesketh - I would like to kindly ask you for a review and another +2 followed by merge. Pending since July, 3rd. Thank you! | 16:03 |
| clarkb | infra-root friendly reminder that if you haven't yet, reviewing mordreds containers spec would be really useful. I think we have opportunity to dig into that today in our meeting and beyond) | 16:03 |
| corvus | oh, thanks. it's been updated since i last looked :) | 16:04 |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Run kata static checks https://review.openstack.org/583296 | 16:06 |
| *** greghaynes has quit IRC | 16:07 | |
| mordred | roman_g: I left a coment- but that is not going to work like people may intend | 16:07 |
| *** jpich has quit IRC | 16:08 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config master: Change neutron CI dashboard to a week view https://review.openstack.org/583299 | 16:10 |
| *** d0ugal has quit IRC | 16:18 | |
| *** harlowja has joined #openstack-infra | 16:18 | |
| *** myoung|biaf is now known as myoung | 16:19 | |
| *** jcoufal_ has quit IRC | 16:20 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Docs: add a cross-ref in secrets https://review.openstack.org/583301 | 16:21 |
| *** camunoz has joined #openstack-infra | 16:23 | |
| *** links has quit IRC | 16:24 | |
| corvus | i added a logs-in-swift topic to the meeting if we feel like chatting about that a bit | 16:25 |
| *** andreas_s has quit IRC | 16:26 | |
| *** andreas_s has joined #openstack-infra | 16:27 | |
| *** lpetrut has joined #openstack-infra | 16:28 | |
| *** dklyle has joined #openstack-infra | 16:28 | |
| *** udesale has quit IRC | 16:29 | |
| *** rkukura has joined #openstack-infra | 16:29 | |
| mordred | corvus, clarkb: for https://review.openstack.org/#/c/583263/3 - should we wait for other humans to review? | 16:29 |
| mordred | corvus: I always like talking about logs-in-swift | 16:30 |
| *** d0ugal has joined #openstack-infra | 16:30 | |
| clarkb | mordred: I don't think so I've gone ahead and approved | 16:30 |
| mordred | woot | 16:30 |
| corvus | mordred, clarkb: aiui, that's a semi-urgent fix, so ++ | 16:30 |
| mordred | clarkb: then I guess to test that it works we can just do a DNM patch with pypi_url set in a variable | 16:31 |
| clarkb | yup | 16:31 |
| clarkb | also can test with pip from laptop | 16:31 |
| *** e0ne has joined #openstack-infra | 16:31 | |
| mordred | and if we're happy with that - we can then move it from the 8080 to the / vhost yeah? | 16:31 |
| clarkb | yes, then everything will transparently be proxied. That said we've kept the proxies in a different vhost for organizational simplicity, we might consider moving to :8080 as an option (though may not be a good one) | 16:32 |
| *** caphrim007 has joined #openstack-infra | 16:33 | |
| mordred | clarkb: we could do that - we'd need to set pypi_url in site-variables for that to work for configure mirrors - and then a second patch to ozj to update the mirror-info | 16:33 |
| mordred | pabelanger: you stood up an internal mirror based on the AFS content in /afs/openstack.org right? are you using the pypi mirror for that? if so, and if our experiment above work, will you be in a position to switch to a proxy-cache rule? | 16:34 |
| mordred | pabelanger: (I mean, whether you are in a position to make that switch or not, you should be aware that our AFS pypi mirror is broken and unlikley to unbreak) | 16:34 |
| *** gongysh has quit IRC | 16:35 | |
| *** andreas_s has quit IRC | 16:36 | |
| *** jamesmcarthur has quit IRC | 16:39 | |
| *** andreas_s has joined #openstack-infra | 16:41 | |
| *** dhajare has joined #openstack-infra | 16:41 | |
| * smcginnis notices the last comment and refrains from asking | 16:42 | |
| *** andreas_s has quit IRC | 16:43 | |
| clarkb | smcginnis: tldr is tensorflow and other libs that make binary packages that link against cuda result in ~300MB python packges. Then you take the matrix of windows * linux * OSX * cuda version X * cuda version Y * cuda version Z and publish nightly packages and you end up with gigabytes of new packages every day | 16:43 |
| fungi | smcginnis: in this case "broken" means "perpetually stale until we replace it" | 16:44 |
| pabelanger | mordred: yes, we are using AFS for pypi today in rdoproject.org, I'm sure we can make changes to apache if needed. | 16:44 |
| clarkb | smcginnis: eventually we run out of disk (currently at the afs limit for a volume) and are investigating alternatives | 16:44 |
| pabelanger | mordred: not me, but can pass along the info for other SF members to do | 16:44 |
| *** hashar is now known as hasharAway | 16:45 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add caching proxy config for download.docker.com https://review.openstack.org/580882 | 16:45 |
| fungi | pabelanger: not sure if you saw earlier, but the fedora mirror is similarly at quota (though we _can_ bump its quota some if it makes sense to do so) | 16:45 |
| openstackgerrit | Merged openstack-infra/system-config master: Add caching proxy config for Alpine Linux https://review.openstack.org/582584 | 16:45 |
| mordred | pabelanger: cool. we've got https://review.openstack.org/#/c/583263/ up with rules that theoretically will work (pending further testing once it lands) | 16:47 |
| pabelanger | fungi: I wonder if we ever delete old mirror content, we should only have fedora-28 online now | 16:47 |
| clarkb | pabelanger: ya that was my question earlier, I'm guessing there is older stuff we can delete, but haven't checked | 16:47 |
| *** agopi is now known as agopi|lunch | 16:47 | |
| pabelanger | mordred: not today, but for 3pci we maybe should extra this data from system-config to make it easy for downstream AFS users. | 16:48 |
| pabelanger | clarkb: +1 | 16:48 |
| mordred | pabelanger: ++ | 16:48 |
| pabelanger | s/extra/extract | 16:48 |
| clarkb | mordred: pabelanger: extract what data? | 16:48 |
| mordred | pabelanger: so, given that that mirror is currently not updating with any new packages, I'd recommend letting people know that they may want to go ahead and start investigating https://review.openstack.org/#/c/583263/ | 16:48 |
| mordred | clarkb: mirror config | 16:48 |
| pabelanger | clarkb: vhost config | 16:48 |
| mordred | clarkb: like "as a 3pci, I'd like to run a mirroring infrastructure like infra does for openstack" | 16:49 |
| pabelanger | mordred: ack | 16:49 |
| clarkb | hrm I'm not sure I'm super comfortable on prescribing how to make networking in $randomcorpenv reliable | 16:49 |
| mordred | I agree - and I don't think it should be required to run on | 16:49 |
| mordred | one | 16:50 |
| mordred | but the mirror vhost config is pretty decently standalone already | 16:50 |
| clarkb | thats true, mostly just concerned we'd be adding unnecesary layers for many users if they already have the coporate squid proxy | 16:50 |
| mordred | so someone else picking it up and deploying a mirror host with it without buying in to other things should be *fairly* easy | 16:50 |
| mordred | clarkb: agree | 16:50 |
| smcginnis | fungi, clarkb: So we have a requirements job that failed to find the newly released package. Do we need to wait for a manual intervention before rechecking that? | 16:50 |
| mordred | smcginnis: this is the issue we're working through now | 16:51 |
| mordred | no new python packages are hitting the mirrors right now | 16:51 |
| openstackgerrit | Merged openstack-infra/system-config master: Add proxy cache for PyPI https://review.openstack.org/583263 | 16:51 |
| smcginnis | mordred: Ack, got it. | 16:51 |
| mordred | smcginnis: once that is applied ^^ we're going to want to do some tests of it with DNM jobs setting a pypi_url variable | 16:52 |
| mordred | smcginnis: requirements job might be a good one | 16:52 |
| *** squid is now known as calebb | 16:52 | |
| clarkb | mordred: ++ that should exercise it quite a bit and check that new packages work :) | 16:52 |
| pabelanger | clarkb: yah, but so far it just works, if you join our AFS cell. And now that zuul jobs are be written to expect our mirror URI formats, I think we need to at least discuss a better way for zuul users to maybe support the same way as openstack | 16:52 |
| clarkb | pabelanger: ya it gets a bit weird when you start talking about proxies especially for corporate environments | 16:53 |
| clarkb | pabelanger: beacuse they tend to already have them and it should be a noop | 16:53 |
| mordred | yah. this is why it should not be required | 16:53 |
| clarkb | (though cases like docker likely present trouble) | 16:53 |
| mordred | and possibly not even suggested | 16:53 |
| *** tobasco is now known as tobasco_away | 16:54 | |
| mordred | but - there is an interface lurking around in here - and for a user who has the desire and the right environment to do one, I think it can make a big differece | 16:54 |
| clarkb | ya. The next thing we'll run into is why not nginx/squid/devpi. But I think we can offer what we do and let others figure out alternatives if they need them | 16:55 |
| pabelanger | Indeed, all I know is in rdoproject, we'd have a much large failure rate without the AFS mirror :) | 16:56 |
| fungi | up side to afs is that this is a fairly graceful breakage | 16:56 |
| clarkb | fwiw I set up apache as the proxy beacuse we already had apache on the mirror nodes making it easy. If apache becomes no longer easy we can/should reconsider it | 16:57 |
| *** greghaynes has joined #openstack-infra | 16:57 | |
| fungi | jobs are merely running with a stale snapshot of pypi | 16:57 |
| openstackgerrit | Merged openstack-infra/zuul master: zk: retry initial zookeeper connection attempts https://review.openstack.org/576048 | 16:57 |
| mordred | clarkb: well, right now the substitute module has become essential | 16:57 |
| fungi | if pypi goes down entirely, this is the same state the afs mirror would be ni | 16:57 |
| openstackgerrit | Merged openstack-infra/zuul master: dashboard: make ZuulService load info api to manage paths https://review.openstack.org/579418 | 16:57 |
| openstackgerrit | Merged openstack-infra/zuul master: scheduler: fix enqueue event to use canonical project name https://review.openstack.org/580040 | 16:57 |
| clarkb | fungi: ++ maybe we should consider getting by with proxy in short term, rebuild pypi mirror on afs after we switch and see how much smaller it is | 16:57 |
| fungi | on the other hand, if pypi goes down entirely and we're doing apache proxying... | 16:58 |
| *** agopi|lunch is now known as agopi | 16:58 | |
| mordred | clarkb: but also yes - we can offer our apache config - and if someone is like "why not nginx/squid/devpi" - they are more than welcome to solve the issue again ;) | 16:58 |
| mordred | clarkb: ++ to rebuilding pypi afs mirror and testing | 16:59 |
| pabelanger | ++ to only apache configs for offering | 16:59 |
| ssbarnea1 | does anyone knows a trick for converting a gerrit query into a bookmark? | 16:59 |
| clarkb | ssbarnea1: if you put stuff in the search box it should all go into url query parameters which you can bookmark | 17:00 |
| ssbarnea1 | clarkb: nope, it does work only for most simplistic queries | 17:00 |
| clarkb | ssbarnea1: what sort of query is failing for you? | 17:01 |
| fungi | ssbarnea1: alternatively, i think the gerrit-dash-creator can do that for you (turn a query string into a url) | 17:01 |
| ssbarnea1 | because browser adds %abcd which makes gerrit choke | 17:01 |
| ssbarnea1 | any query containing characters like: , " > = ' --- is likely to break. | 17:01 |
| fungi | https://pypi.org/project/gerrit-dash-creator/ | 17:02 |
| ssbarnea1 | fungi: thanks, testing now. | 17:02 |
| ssbarnea1 | ... not to mention the query window size :D | 17:02 |
| clarkb | https://review.openstack.org/#/q/Code-Review%253E%253D%252B2 seems to have worked for me | 17:02 |
| clarkb | ya bookmarked and opened in firefox and chrome | 17:03 |
| ssbarnea1 | newer versions of gerrit improved the UI a little bit but we don't have them. | 17:03 |
| fungi | well, latest versions of gerrit completely replaced the webui with one written in polyglot | 17:04 |
| clarkb | ssbarnea1: the characters have to be url encoded | 17:04 |
| openstackgerrit | Merged openstack-infra/zuul master: Stop publishing docs to docs.openstack.org https://review.openstack.org/582323 | 17:04 |
| clarkb | ssbarnea1: you get that automatically if you use the search box in gerrit to produce the url | 17:04 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix line numbers in job repr https://review.openstack.org/581909 | 17:04 |
| *** jmorgan1 has quit IRC | 17:07 | |
| *** aojea has quit IRC | 17:09 | |
| *** lbragstad has quit IRC | 17:11 | |
| *** dtantsur is now known as dtantsur|afk | 17:12 | |
| ssbarnea1 | fungi: thanks for the link, the tool proves really useful for creating the magic urls | 17:13 |
| *** bobh has quit IRC | 17:13 | |
| openstackgerrit | Merged openstack-infra/zuul master: Log cpu times of ansible executions https://review.openstack.org/582713 | 17:14 |
| *** jmorgan1 has joined #openstack-infra | 17:14 | |
| *** e0ne has quit IRC | 17:15 | |
| *** esarault_ has joined #openstack-infra | 17:16 | |
| *** ramishra has quit IRC | 17:17 | |
| *** esarault has quit IRC | 17:18 | |
| *** jluk has quit IRC | 17:19 | |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Stop publishing to docs.openstack.org https://review.openstack.org/582326 | 17:20 |
| *** e0ne has joined #openstack-infra | 17:21 | |
| pabelanger | clarkb: mordred: https://review.openstack.org/#/q/topic:afs-mirror-centos seems to be a thing for afs mirroring | 17:22 |
| ssbarnea1 | does anyone also happen to have a CSS hack that makes gerrit reviews table readable on not so wide screens? On 13" i need to zoom out to max to make it fix all columns. | 17:23 |
| *** felipemonteiro_ has joined #openstack-infra | 17:26 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Stop publishing docs to docs.openstack.org https://review.openstack.org/582325 | 17:28 |
| *** trown is now known as trown|lunch | 17:29 | |
| *** felipemonteiro has quit IRC | 17:30 | |
| *** jistr|afk is now known as jistr | 17:30 | |
| *** dpawlik has joined #openstack-infra | 17:31 | |
| ssbarnea1 | i added the infra meeting in my calendar but i see no activity on #opensack-meeting,... maybe I got the timezone conversion wrong. | 17:34 |
| fungi | ssbarnea1: the infra team meeting? | 17:34 |
| ssbarnea1 | which is bit funny as I am as close as possible to UTC | 17:34 |
| fungi | 19:00 utc is in ~1.5 hours from now | 17:34 |
| ssbarnea1 | yep, i wanted to start attending them. | 17:35 |
| *** dpawlik has quit IRC | 17:35 | |
| ssbarnea1 | yep, I did the math wrong. | 17:35 |
| mordred | clarkb: http://mirror.ca-ymq-1.vexxhost.openstack.org:8080/pypi/simple/ in index-url of ~/.config/pip.conf worked | 17:35 |
| mordred | for me locally | 17:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: DNM Set pypi mirror to use 8080 instead of 80 https://review.openstack.org/583317 | 17:39 |
| ssbarnea1 | fungi: done. see you there. mainly i want to help with few infra project that got neglected over time (gerritlib/review/bashate/...). there are lots of valid CRs that failed to get merged due to various reasons, most of the time was lack of 2x+2 | 17:39 |
| fungi | yep, having our reviewers doing double-duty as systems administrators leaves a lot of us with less time to review proposed improvements/fixes | 17:40 |
| mordred | clarkb: https://review.openstack.org/583318 DNM Test that new mirror fixes lifes problems <-- test patch to see that the zuul-jobs patch works | 17:40 |
| mordred | smcginnis: ^^ you might want a similar one for requirements? just depends-on that zuul-jobs patch should do the trick | 17:41 |
| *** sambetts is now known as sambetts|afk | 17:41 | |
| mordred | fungi, corvus: ^^ fyi | 17:42 |
| *** zoli|wfh is now known as zoli|gone | 17:44 | |
| *** myoung is now known as myoung|lunch | 17:50 | |
| *** r-daneel has quit IRC | 17:50 | |
| *** harlowja has quit IRC | 17:54 | |
| *** shardy_ has quit IRC | 17:54 | |
| clarkb | mordred: re 583318 that should affect all the jobs right? pep8 and py35 both passed | 18:01 |
| mordred | clarkb: yes. it should | 18:02 |
| *** bobh has joined #openstack-infra | 18:10 | |
| *** pcichy has quit IRC | 18:14 | |
| *** janki has quit IRC | 18:14 | |
| *** ianychoi has quit IRC | 18:14 | |
| corvus | mordred: when we merge for reals, should we just change pypi_mirror in the base job (or site vars) rather than changing the role default? | 18:15 |
| *** yamahata has quit IRC | 18:15 | |
| *** e0ne has quit IRC | 18:15 | |
| electrofelix | any chance of a review on https://review.openstack.org/582395 later today (it's to make check-requirements non-voting on git-upstream) heading off now so might not see a response | 18:16 |
| *** electrofelix has quit IRC | 18:16 | |
| clarkb | electrofelix is gone now, but cases like this are good candidates for moving config in tree | 18:18 |
| mordred | corvus: yes | 18:19 |
| mordred | corvus: I think the for-real switch should be a site-var | 18:20 |
| corvus | cool sounds good | 18:20 |
| *** jistr is now known as jistr|off | 18:25 | |
| *** rh-jelabarre has quit IRC | 18:25 | |
| clarkb | does updating that require restarting zuul? | 18:25 |
| clarkb | or will subsequent ansible processes pick it up? | 18:25 |
| *** r-daneel has joined #openstack-infra | 18:27 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use proxy/cache for pypi instead of AFS mirror https://review.openstack.org/583328 | 18:28 |
| mordred | corvus, clarkb: ^^ there's a patchfor when we're ready for it | 18:28 |
| zul | hey im seeing some weirdness on one of the mirrors: https://pastebin.com/1GyrXBFT | 18:29 |
| dirk | Are there known issues with static.openstack.org? | 18:29 |
| *** mihalis68 has quit IRC | 18:29 | |
| dirk | We get from one host connection-reset-by-peer every few minutes. Is the service bein constantly restarted? (Http/web) | 18:30 |
| pabelanger | zul: mordred is making changes to pypi, will defer to him if known issue | 18:30 |
| *** trown|lunch is now known as trown | 18:31 | |
| zul | ack | 18:31 |
| *** rh-jelabarre has joined #openstack-infra | 18:31 | |
| *** diablo_rojo has joined #openstack-infra | 18:31 | |
| *** r-daneel has quit IRC | 18:32 | |
| clarkb | zul: pabelanger the changes mordred made should only be if you talk to port 8080 which that isn't doing | 18:32 |
| clarkb | dirk: performing what action? | 18:32 |
| *** harlowja has joined #openstack-infra | 18:32 | |
| clarkb | there are different vhosts on that server so may help to know what exactly is resetting | 18:33 |
| dirk | clarkb: wget of various $foo-master.tar:gz on tarballs.o.o | 18:34 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/system-config master: Add logging config for openstacksdk https://review.openstack.org/583332 | 18:34 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/system-config master: Remove shade logging config for nodepool. https://review.openstack.org/583333 | 18:34 |
| clarkb | zul: pabelanger f5260a6e679d2ff42ec91ec5252f4eeffdcf21053db9113bd0a8e4d953769c00 is the sha256sum I get for that file. One thought is perhaps we ran out of disk on the afs volume and only wrote half the file | 18:34 |
| dirk | It used to be super stable but we have issues for the last week or so | 18:35 |
| pabelanger | clarkb: thinking that too | 18:35 |
| clarkb | 0.5.5 was released over a year ago though | 18:35 |
| Shrews | mordred: 583332 & 583333 put up for when we're ready to try nodepool+openstacksdk | 18:35 |
| clarkb | oh interesting pypi upstream says f5260a6e679d2ff42ec91ec5252f4eeffdcf21053db9113bd0a8e4d953769c00 is the sha256sum not the sha in that url | 18:36 |
| clarkb | zul: where is the requirements file for that installation? | 18:36 |
| *** sshnaidm|rover has quit IRC | 18:37 | |
| mordred | Shrews: I think we shold go ahead and land 583332 | 18:38 |
| mordred | clarkb, corvus: ^^ 583332 when you get a sec | 18:38 |
| *** sshnaidm has joined #openstack-infra | 18:39 | |
| clarkb | dirk: the youngest apache process is more than an hour old so I don't think the processes are crashing | 18:43 |
| *** sshnaidm is now known as sshnaidm|rover | 18:44 | |
| fungi | dirk: are you seeing this with release tarballs or maybe branch snapshot tarballs? the latter are overwritten and it's possible you could be requesting some mid-write | 18:44 |
| fungi | ann, you did say -master | 18:44 |
| clarkb | we are also well under connection limits from our cloud provider | 18:45 |
| fungi | dirk: if you can get some recent timestamps we could maybe try to correlate them to overwrite times for the respective files | 18:45 |
| fungi | though i'm not sure i would expect a connection reset in those cases | 18:46 |
| dirk | clarkb: ok, thanks | 18:46 |
| dirk | fungi: no, it also happens with foo-1.2.3.tar.gz tarballs | 18:46 |
| fungi | not entirely sure what apache's behavior is if it's streaming a file which gets truncated/replaced | 18:46 |
| dirk | it is entirely possible that it is an issue on our funky firewall or the like | 18:47 |
| fungi | ahh, sounding less likely to be that | 18:47 |
| *** lbragstad has joined #openstack-infra | 18:47 | |
| fungi | less likely to be the overwrite, i mean | 18:47 |
| dirk | yeaph, definitely not that problem | 18:47 |
| dirk | it used to run 100% stable all the time for many years | 18:47 |
| fungi | dirk: can you try to recreate the behavior from another network not behind that firewall? | 18:47 |
| dirk | so $something changed, just not understanding what changed | 18:47 |
| *** myoung|lunch is now known as myoung | 18:47 | |
| dirk | fungi: I am trying, but not successful so far | 18:48 |
| clarkb | also maybe check ipv4 vs ipv6 | 18:48 |
| zul | clarkb: http://logs.openstack.org/69/583069/1/gate/openstack-tox-pep8/afbcd1a/ | 18:48 |
| *** yamahata has joined #openstack-infra | 18:48 | |
| dirk | its ipv4 | 18:48 |
| fungi | dirk: i haven't heard similar complaints, though doesn't mean it isn't happening to other people | 18:48 |
| zul | clarkb: its for browbeat | 18:48 |
| dirk | fungi: yeah, its more likely an issue on our side, just wanted to check before I spend weeks on hunting a ghost | 18:48 |
| *** e0ne has joined #openstack-infra | 18:49 | |
| mordred | clarkb: something is very weird with mirror config | 18:51 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix devstack plugin for using openstacksdk https://review.openstack.org/583338 | 18:51 |
| mordred | clarkb: for https://review.openstack.org/#/c/583317/ - if you look in http://logs.openstack.org/17/583317/1/check/tox-py35-on-zuul/7c6d26e/tox/py35-1.log | 18:51 |
| mordred | it shows pulling from http://mirror.ord.rax.openstack.org/pypi/packages | 18:52 |
| clarkb | mordred: so it didn't test what we thought it was testing | 18:53 |
| mordred | exactly. it did not - but why the heck not? | 18:53 |
| clarkb | zul: the cmd2 dep which pulls in contextlib2 doesn't seem to specify a sha1. What is odd to me is the expected value and the received value do not match the actual value in pypi (or the value I get curling from that mirror myself) | 18:53 |
| clarkb | zul: is this a persistent issue? I'd almost be willing to attribute this to bit flips | 18:54 |
| *** bobh has quit IRC | 18:54 | |
| zul | clarkb: i just saw it so i dont think its persistent | 18:54 |
| zul | clarkb: yet at least ;) | 18:54 |
| fungi | dirk: most common cause i've seen for that behavior (was a network engineer and firewall admin for years) is an insufficiently-sized overload nat/pat pool | 18:54 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix devstack plugin for using openstacksdk https://review.openstack.org/583338 | 18:55 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix devstack plugin for using openstacksdk https://review.openstack.org/583338 | 18:56 |
| fungi | dirk: basically the nat builds up a lot of near-concurrent session states it's tracking and mapping to different tcp ports, but either hasn't been allocated sufficient memory to track them or enough global ip addresses to handle the number of distinct ports it needs | 18:56 |
| fungi | and then starts rejecting any additional connection attempst with tcp/rst responses | 18:56 |
| *** eventingmonkey has quit IRC | 18:57 | |
| *** pfallenop has quit IRC | 18:58 | |
| *** pfallenop has joined #openstack-infra | 18:58 | |
| *** pfallenop has joined #openstack-infra | 18:58 | |
| *** eventingmonkey has joined #openstack-infra | 18:58 | |
| dirk | fungi: its not a new connection. the issue that happens is that we're almost done with download (near 80-90%) and then get a conn reset | 18:59 |
| *** Qiming has quit IRC | 18:59 | |
| dirk | fungi: nf_conntrack_max is 65535, nf_conntrack_count is 32 | 18:59 |
| dirk | so it doesn't seem likely to exceed that | 18:59 |
| fungi | ahh, in that case there can be a lot of potential causes | 18:59 |
| *** pcichy has joined #openstack-infra | 19:00 | |
| fungi | is there a content inspection firewall? | 19:00 |
| clarkb | meeting time | 19:00 |
| fungi | those can frequently end up rejecting random binary streams because they get spooked by something they think matches an overzealous malware signature | 19:00 |
| fungi | which tends to happen mid-stream | 19:01 |
| dirk | fungi: its not repeatable on the same file. over 90% of the time we can download the identical file successfully | 19:01 |
| *** Qiming has joined #openstack-infra | 19:01 | |
| fungi | next guess in that regard is a non-graceful packet shaper trying to cope with overall traffic levels exceeding its configured limit | 19:02 |
| *** panda is now known as panda|off | 19:03 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix plugin and examples for using openstacksdk https://review.openstack.org/583338 | 19:03 |
| *** kjackal has quit IRC | 19:04 | |
| *** graphene has quit IRC | 19:07 | |
| *** graphene has joined #openstack-infra | 19:08 | |
| *** wolverineav has quit IRC | 19:09 | |
| *** wolverineav has joined #openstack-infra | 19:10 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config master: Change neutron CI dashboard to a week view https://review.openstack.org/583299 | 19:12 |
| *** efried has quit IRC | 19:14 | |
| *** efried has joined #openstack-infra | 19:14 | |
| *** wolverineav has quit IRC | 19:14 | |
| *** tosky has quit IRC | 19:20 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add logging config for openstacksdk https://review.openstack.org/583332 | 19:27 |
| *** dhajare has quit IRC | 19:40 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add role to ensure per-node output dirs exist https://review.openstack.org/511823 | 19:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add role to fetch zuul logs from nodes https://review.openstack.org/583346 | 19:41 |
| *** e0ne has quit IRC | 19:41 | |
| *** diablo_rojo has quit IRC | 19:47 | |
| *** dpawlik has joined #openstack-infra | 19:47 | |
| pabelanger | frickler: do you mind removing your -2 on https://review.openstack.org/511823/, I'd like to start testing and able to monitor | 19:51 |
| *** dpawlik has quit IRC | 19:51 | |
| *** gfidente has quit IRC | 19:52 | |
| *** eernst has quit IRC | 20:03 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add role to fetch zuul logs from nodes https://review.openstack.org/583346 | 20:04 |
| fungi | now that the meeting's through, i need to disappear to run some pre-travel errands and get something to eat, but will be back on later this evening | 20:04 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add tenant-scoped autohold, enqueue (change, ref) https://review.openstack.org/576907 | 20:04 |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Run kata static checks https://review.openstack.org/583296 | 20:05 |
| *** esarault_ has quit IRC | 20:05 | |
| clarkb | cmurphy: I'm popping out for a bit now to work on this paper at the coffeeshop but when I get back I will try to review puppet-4 changes. Iguess I'll start with the changes around using future parser | 20:08 |
| clarkb | mordred: also let me know if you figure out the test ing of pypi | 20:08 |
| *** fresta has quit IRC | 20:13 | |
| mordred | clarkb: I'm ... very confused | 20:13 |
| clarkb | mordred my guess is its being run from trusted job ontext so nog using your new parameter | 20:14 |
| clarkb | we may have go use base-test? | 20:14 |
| mordred | clarkb: THANK YOU | 20:15 |
| mordred | clarkb: I was completely and totally confused as to why it was running with those values | 20:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add role to fetch zuul docs from nodes https://review.openstack.org/583350 | 20:18 |
| mordred | clarkb: I think we can test it just with setting the variable in the DNM test patches rather than relying on the default update in zuul-jobs | 20:18 |
| mordred | lemme respin them | 20:18 |
| mordred | clarkb, corvus: remote: https://review.openstack.org/583318 DNM Test that new mirror fixes lifes problems | 20:21 |
| mordred | maybe that one will work better | 20:22 |
| cmurphy | clarkb: there's only one change right now to turn on the future parser, i figure we see how that goes and make sure i got the ansible right and then i can spin up some more | 20:22 |
| mordred | if it doesn't, we can do a bast-test dance | 20:22 |
| cmurphy | there are some template scope issues that should be fixed before we go crazy | 20:22 |
| jungleboyj | Anyone out here willing to provide some basic assistance to a poor Cinder guy trying to help my team in China get an Ironic CI set up? | 20:23 |
| *** hasharAway is now known as hashar | 20:24 | |
| mnaser | hi clarkb -- sorry for the mia-ness, we havent gotten around to the bottom of the ipv6 stuff | 20:25 |
| * mordred shakes fist at mnaser ... hands him a pie | 20:26 | |
| mnaser | there's a lot going on here and if i cant guarantee that it's stable, i wouldn't want to enable it and cause jobs to fail | 20:26 |
| mnaser | mordred: pie would be nice | 20:26 |
| mordred | mnaser: YES IT WOULD BE | 20:26 |
| mnaser | also. we're opening a new region in san jose/sillicon valley | 20:26 |
| mnaser | so i fully expect openstack infra to stress test it when its up. | 20:26 |
| mnaser | 40g internal networking, 10g public internet access to each vm, faster cpus, etc | 20:27 |
| mnaser | so hopefully the new region can help while we get ipv6 stuff figured out in montreal | 20:27 |
| * corvus lives near there and can attest that it is in fact sillycon valley | 20:27 | |
| mnaser | corvus: oh really, around where? i've been here since the 4th | 20:27 |
| *** lbragstad has quit IRC | 20:28 | |
| corvus | mnaser: oakland | 20:28 |
| mnaser | oh cool, yeah i'm about just ready to get back home | 20:29 |
| mnaser | living out of a hotel for the past 2 weeks takes a toll, heh | 20:29 |
| corvus | i'm a mere 1 hour drive from san jose. or 3 in traffic. :) | 20:29 |
| mnaser | nature is gorgeous here though, i took time go on a few hikes and it was really great | 20:30 |
| jungleboyj | mordred: Do you have pointers to start setting up a v3 CI. | 20:30 |
| jungleboyj | Are these directions still correct? https://docs.openstack.org/infra/openstackci/third_party_ci.html | 20:30 |
| *** priteau has quit IRC | 20:31 | |
| corvus | mnaser: how's your swift cluster? would you object to us, um, putting some objects in it? how about a lot of objects? | 20:32 |
| *** eernst has joined #openstack-infra | 20:33 | |
| *** lpetrut has quit IRC | 20:36 | |
| *** wolverineav has joined #openstack-infra | 20:38 | |
| *** rosmaita has quit IRC | 20:41 | |
| *** eharney has quit IRC | 20:47 | |
| *** fresta has joined #openstack-infra | 20:48 | |
| mordred | jungleboyj: they are not even close to being correct, no - we have not yet written new instructions for v3-based third-party-ci | 20:48 |
| jungleboyj | mordred: Oy. Ok. Are they right for v2? | 20:48 |
| jungleboyj | mordred: and if so can we still get Zuul v2 somewhere? | 20:49 |
| *** florianf has quit IRC | 20:50 | |
| mordred | jungleboyj: I'd avoid a new v2 install if it can at all be avoided. I know that humans are definitely using v3 for 3rd-party-ci though ... pabelanger is using software-factory - maybe he or tristanC have a getting started pointer | 20:50 |
| pabelanger | I'm just a user of software-factory, will defer to tristanC on how to set it up. But 3pci is working with zuulv3 | 20:51 |
| jungleboyj | mordred: Ok, we are supposed to have CI up and running by the end of the month for TheJulia and feel like we are kind of flying blind. | 20:52 |
| *** fresta_ has joined #openstack-infra | 20:53 | |
| mordred | jungleboyj: nod. we'll see if we can't get you some guidance ... sorry I don't have a doc to point you at yet | 20:54 |
| *** fresta has quit IRC | 20:55 | |
| *** camunoz has quit IRC | 20:55 | |
| jungleboyj | mordred: Thanks. In the mean time if we want to follow the old directions to see if we can make more progress is there a way to still get Zuul V2? | 20:57 |
| jungleboyj | Or we need to see if TheJulia will grant us a grace period. :-) | 20:57 |
| pabelanger | zuulv2 should work still | 20:58 |
| *** rosmaita has joined #openstack-infra | 20:58 | |
| mordred | yah - v2 should totally still work | 20:58 |
| pabelanger | rdoproject was still running as of 2 weeks ago | 20:58 |
| pabelanger | but now they are zuulv3 | 20:58 |
| *** d0ugal has quit IRC | 20:58 | |
| jungleboyj | pabelanger: The directions here just say 'Start Zuul' and when I go and follow the directions to get Zuul I get v3. | 20:59 |
| jungleboyj | :-) | 20:59 |
| jungleboyj | https://docs.openstack.org/infra/openstackci/third_party_ci.html | 21:00 |
| *** d0ugal has joined #openstack-infra | 21:00 | |
| *** d0ugal has quit IRC | 21:00 | |
| *** d0ugal has joined #openstack-infra | 21:00 | |
| pabelanger | jungleboyj: yah, I am guessing we haven't capped it in puppet-openstackci? | 21:01 |
| *** jtomasek has quit IRC | 21:01 | |
| TheJulia | jungleboyj: I look at it this way, you guys are working on it. That in itself goes a long way. :) | 21:01 |
| jungleboyj | Oh, probably true. | 21:01 |
| openstackgerrit | Merged openstack-infra/zuul master: Build container images using pbrx https://review.openstack.org/580160 | 21:01 |
| jungleboyj | TheJulia: Ok, wanted to make sure you were as nice as I am ... sounds like nicer. | 21:02 |
| jungleboyj | ;-) | 21:02 |
| TheJulia | :) | 21:02 |
| rajinir | jungleboyj: https://github.com/openstack-infra/system-config/blob/master/doc/source/third_party.rst - better instructions ( old though, have to updated for V3) You have to clone the project-config from github to get zuul and other codes | 21:04 |
| pabelanger | jungleboyj: zuul_revision looks to be capped at 2.6.0 if undef: http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/single_node_ci.pp#n213 | 21:04 |
| pabelanger | jungleboyj: likely need to see what version puppet installed | 21:04 |
| jungleboyj | TheJulia: Since I am able to get some support from rajinir who has this working with V2 I am going to see if I can get my team to make more progress that way and then if someone who has good V3 pointers jumps in we can move that direction. | 21:05 |
| TheJulia | jungleboyj: ack, thanks :) | 21:05 |
| jungleboyj | I am trying to lead a new team on the other side of the world and would rather they have more concrete instuctions. | 21:05 |
| *** jtomasek has joined #openstack-infra | 21:06 | |
| jungleboyj | pabelanger zuul==3.1.1.dev130 | 21:06 |
| pabelanger | jungleboyj: k, seems like something uncapped it | 21:07 |
| jungleboyj | Ok. so we need to follow the install instructions for Zuul but make sure we get a version less than 3.1.1 | 21:07 |
| jungleboyj | Less than 3. | 21:07 |
| pabelanger | yah, 2.6.0 I believe was last release | 21:08 |
| jungleboyj | pabelanger: Great. Thank you. | 21:08 |
| openstackgerrit | Merged openstack-infra/zuul master: Specify a prefix for building the images https://review.openstack.org/582396 | 21:08 |
| openstackgerrit | Merged openstack-infra/zuul master: Install less than alpine-sdk https://review.openstack.org/583062 | 21:08 |
| mordred | clarkb, corvus: I could use someone with a smarter brain than I have | 21:09 |
| mordred | clarkb, corvus: curl http://mirror.us-west-1.packethost.openstack.org:8080/pypi/simple/keystoneauth/ | 21:09 |
| mordred | returns what I expect | 21:09 |
| mordred | view-source:http://mirror.us-west-1.packethost.openstack.org:8080/pypi/simple/keystoneauth/ | 21:09 |
| mordred | does not | 21:09 |
| mordred | and http://logs.openstack.org/18/583318/3/check/pbrx-build-zuul-containers/19d2c9e/job-output.txt.gz#_2018-07-17_20_28_14_930401 | 21:10 |
| corvus | mordred: browser cache? :) | 21:10 |
| mordred | shows that it's using the new url for the index, but not for the content | 21:10 |
| mordred | so I can only assume it's getting the cached version not thenew fresh version | 21:10 |
| mordred | http://logs.openstack.org/18/583318/3/check/pbrx-build-zuul-containers/19d2c9e/job-output.txt.gz#_2018-07-17_20_28_18_818448 (is the bit where it's doing the wrong thing) | 21:10 |
| clarkb | oh we may have cached it in apache from.before the substitute? | 21:12 |
| corvus | clarkb: why would curl work then? | 21:14 |
| *** ldnunes has quit IRC | 21:15 | |
| mordred | (I could see us having it cachedin apache from before the substitute though) | 21:15 |
| mordred | corvus, clarkb: maybe pip sends appropriate "please obey cache" headers and curl doesn't and that affects something *I am grasping at straws* | 21:16 |
| clarkb | corvus: good point | 21:16 |
| *** kaisers1 has joined #openstack-infra | 21:17 | |
| *** kaisers has quit IRC | 21:18 | |
| corvus | when i enable accept-encoding gzip,deflate i get the the un-substituted version in curl | 21:18 |
| corvus | curl http://mirror.us-west-1.packethost.openstack.org:8080/pypi/simple/keystoneauth/ -o - -H "Accept-Encoding: gzip, deflate"|zcat | 21:19 |
| mordred | oh good | 21:19 |
| corvus | that's a header that firefox sends | 21:19 |
| mordred | quick digression - how's this for an awesome log entry: | 21:20 |
| mordred | 202.78.240.7 - - [17/Jul/2018:21:19:33 +0000] "GET /pypi/simple/keystoneauth/ HTTP/1.1" 200 666 conditional cache hit: entity refreshed "-" "Mozilla/5.0 (compatible; The Lounge IRC Client; +https://github.com/thelounge/thelounge)" | 21:20 |
| mordred | I believe each time we put the url in here it's causing someone's irc client to do a GET | 21:20 |
| corvus | yeah... it's part of why i avoid pasting urls of broken things in irc now :( | 21:20 |
| mordred | *sigh* | 21:21 |
| corvus | mordred: https://serverfault.com/questions/843905/apache-mod-substitute-works-in-curl-but-not-on-browser | 21:21 |
| jungleboyj | pabelanger: Would we need to make sure we get an older version of nodepool as well or should we be fine with what we get from Pip by default? | 21:21 |
| mordred | corvus: so maybe try replacing SUBSTITUTE with INFLATE;SUBSTITUTE;DEFLATE ? | 21:22 |
| clarkb | that is curious that substitute doesn't imply an inflate first | 21:23 |
| corvus | mordred: seems like a good starting point; can you test this locally? | 21:24 |
| mordred | yup. on it | 21:24 |
| mordred | I did see this behavior before locally, but chalked it up to browser cache | 21:24 |
| corvus | mordred: that seems safer than unsetting the header; i'm not clear on what side-effects would have in this situation. | 21:24 |
| pabelanger | jungleboyj: yah, nodepool should also be capped, you'd want 0.5.0 | 21:24 |
| mordred | corvus: worked locally | 21:25 |
| clarkb | mordred: corvus is that going to break if we get non gzipped content? | 21:26 |
| clarkb | or can we make it conditional? | 21:26 |
| clarkb | mordred: taking the examples above I guess pip works now but does curl break | 21:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Surround substitute with inflate/deflate https://review.openstack.org/583401 | 21:27 |
| mordred | clarkb: good quesiton - checking | 21:27 |
| *** felipemonteiro_ has quit IRC | 21:27 | |
| mordred | clarkb: curl also works | 21:28 |
| mordred | clarkb: I think that INFLATE/DEFLATE just runs those filters - so in this case, the filter doesn't do anything without the header | 21:28 |
| *** wolverineav has quit IRC | 21:28 | |
| clarkb | mordred: ya I guess inflate is a noop if already inflated, and deflate won't happen if client can't accept it? | 21:29 |
| clarkb | apache is probably smarter than me :) | 21:29 |
| jungleboyj | pabelanger: Ok. Good. Glad I asked. | 21:29 |
| mordred | clarkb, corvus: so https://review.openstack.org/#/c/583401/ should fix - and we can recheck that pbrx patch once it hits | 21:30 |
| *** rh-jelabarre has quit IRC | 21:31 | |
| clarkb | has my +2 | 21:32 |
| clarkb | nothing is using it (yet) so should be relatively safe to push | 21:32 |
| clarkb | do we need to enmod deflate or whatever it is called? | 21:32 |
| mordred | don't think so? | 21:33 |
| corvus | clarkb, pabelanger, ianw: should we maybe move forward on https://review.openstack.org/563849 ? | 21:34 |
| corvus | i'm really uncomfortable watching folks recommend new deployments of EOL versions of software | 21:34 |
| *** slaweq has quit IRC | 21:34 | |
| ianw | corvus: i've slowed on that. i can prioritise updating from comments and we can go for another round of reviews | 21:36 |
| *** edmondsw has quit IRC | 21:36 | |
| *** hashar has quit IRC | 21:36 | |
| clarkb | ianw: the goal is to select one of those options and list the others as alternatives right? | 21:38 |
| clarkb | the comment about how to configure zuul to be a third party ci is probably more important than I had initially realized. Mostly learning that there is confusion around where config goes in that situation (based on that recent infra list thread) | 21:39 |
| clarkb | Maybe if we start with that we can be a little flexible on the deployment tooling but still keep people using up to date software? | 21:39 |
| corvus | clarkb: well, among other reasons, the zuul-from-scratch document stops before it gets into that. | 21:39 |
| mordred | WELL ... | 21:41 |
| mordred | we're potentially about to have direction for config-management for infra | 21:42 |
| corvus | ianw, mordred, pabelanger: i just wrote a comment on 563849 that may be of interest | 21:42 |
| mordred | so maybe coming back to 563849 through that lens isn't a terrible idea? | 21:42 |
| * mordred goes to read | 21:42 | |
| mordred | corvus: I think pabelanger is planning on adding support to windmill for deploying from those container images | 21:43 |
| clarkb | ya, I'm mostly pointing out that deploying zuul is only half the problem (based on that list thread) | 21:43 |
| corvus | clarkb: which list thread? | 21:43 |
| clarkb | corvus: http://lists.openstack.org/pipermail/openstack-infra/2018-July/006006.html | 21:43 |
| clarkb | in that thread they appear to have gotten zuul running fine | 21:43 |
| corvus | ah right | 21:44 |
| corvus | yes, the "you need a local git repo" thing :) | 21:44 |
| *** bobh has joined #openstack-infra | 21:44 | |
| corvus | does anyone have thoughts on my response to that? http://lists.openstack.org/pipermail/openstack-infra/2018-July/006008.html | 21:45 |
| corvus | are we willing to trade customized recheck syntax and report messages for ease-of-deployment? | 21:45 |
| *** edmondsw_ has joined #openstack-infra | 21:45 | |
| clarkb | I'd be ok with that, I expect that many third party ci operators are not | 21:45 |
| clarkb | but if they are I think we should host that for them | 21:46 |
| corvus | we could go so far as defining pipelines for third-party ci operators, and save that trouble. | 21:46 |
| corvus | maybe even base jobs (and just use site variables to indicate where logs should be kept) | 21:46 |
| corvus | whatever tool we recommend folks use for deployment could create a local git repo and pre-populate it, but then let folks further customize it | 21:47 |
| fungi | jungleboyj: sorry, jiapei reached out to me just after i went to bed yesterday. something to do with your zuul-merger daemon getting an exit code -13 out of `https://www.irccloud.com/pastebin/B9QCBp4u/executor.log` and i haven't had much luck tracking down what that exit code means in the git source code (nor do they appear to publish a document with detailed exit code mappings that i could find) | 21:49 |
| fungi | am in a scramble getting ready for a two-week vacation starting tomorrow so having trouble following up with everyone | 21:49 |
| *** edmondsw_ has quit IRC | 21:49 | |
| jungleboyj | Hey fungi ! Ok, I am glad that jiapei did reach out. | 21:49 |
| corvus | jungleboyj: oh, if you're asking about running zuul v2 because of that issue, that may not be a solution. almost everything about that is exactly the same in v2 and v3. | 21:50 |
| jungleboyj | Understood. I have kind-of taken over and I have found out that there is more wrong than just Zuul. | 21:50 |
| clarkb | if anyone has some spare cycles https://review.openstack.org/#/q/topic:puppet-4+status:open+label:code-review%253D%252B2 should give you the list of puppet-4 changes i've already reviewed and +2'd I think ianw may have also +2'd some | 21:50 |
| fungi | odds are git is encountering a credential or connectivity problem | 21:50 |
| clarkb | if you don't feel comfortable approving changes I'm happy to go back through and approve as I can babysit | 21:50 |
| jungleboyj | corvus: Well, that is where my concerns started but then as I dug further into documentation they hadn't done anything but try to install Zuul. | 21:51 |
| jungleboyj | fungi: corvus I think I have gotten enough info today to get them down a better path. | 21:52 |
| corvus | jungleboyj: a lot of folks have tried to help with that, but we're all stumped. we've tried producing the command that zuul is running for manual replication of the problem, but that seems to work. so it's one of those things that feels like some subtle differences in environments like unix perms, or selinux contexts, etc. | 21:52 |
| jungleboyj | corvus: :-) | 21:52 |
| jungleboyj | I spent the last day and a half digging through permissions and also disabling selinux and stuff. | 21:52 |
| jungleboyj | I even rebuilt another node and ended up in the same place. | 21:53 |
| jungleboyj | So there is something we were doing wrong there. | 21:53 |
| jungleboyj | Going to try following directions using Ubuntu and see if we have better luck. | 21:54 |
| fungi | jungleboyj: did you have a working ci system on that account at some point? | 21:54 |
| jungleboyj | fungi: No. This is a new account but when I try to checkout by hand it works fine. I am not getting a host-key error. | 21:54 |
| clarkb | cmurphy: can you see comment on https://review.openstack.org/#/c/582524/1 when you have a chance | 21:56 |
| fungi | jungleboyj: if it helps, gerrit's not logging any auth failures for that account, nor do i see any sign of excessive concurrent connections such that it would be getting blocked | 21:57 |
| * fungi hesitates to point fingers at the great firewall | 21:58 | |
| clarkb | could also be corporate firewall? | 21:58 |
| fungi | s/greater/lesser/ | 21:58 |
| jungleboyj | Well, why would it work by hand and then not as part of the automation. | 21:58 |
| jungleboyj | Also didn't matter if it was on a server in China or on my laptop at home. :-) | 21:59 |
| fungi | a very good question to which i have no answer | 21:59 |
| clarkb | corporate firewall may explain that if you have to use a proxy and zuul daemon process is unaware | 21:59 |
| jungleboyj | clarkb: Ok. I am not aware of any proxies. | 22:00 |
| jungleboyj | I am going to have them start using the documentation that is out there and just try to get something simple working on Zuul V2 and see if we get further. | 22:01 |
| fungi | jungleboyj: you're sure the account running the zuul-merger daemon has access to the ssh key you're using for the lenovo_lxca_ci account and that it's the same key you're manually testing with which is set in your zuul configuration | 22:01 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix broken templates for projects in dependent pipelines https://review.openstack.org/581888 | 22:01 |
| fungi | like, testing with ssh -i | 22:01 |
| fungi | -i /path/to/id_rsa | 22:01 |
| jungleboyj | fungi: Yes and we don't have zuul-merger. This was coming from zuul-executor | 22:01 |
| jungleboyj | The keys are owned by zuul:zuul | 22:02 |
| fungi | traceback was from zuul.merger.merger but the executors do run a merger thread | 22:02 |
| fungi | so yeah | 22:02 |
| fungi | the merger thread of your zuul-executor daemon in that case | 22:03 |
| jungleboyj | fungi: Ah, ok. | 22:03 |
| corvus | GIT_SSH_COMMAND="ssh -i /var/lib/zuul/.ssh/id_rsa" git clone ssh://lenovo_lxca_ci@review.openstack.org:29418/openstack/ironic /var/lib/zuul/executor-git/review.openstack.org/openstack/ironic | 22:04 |
| corvus | that's the command i came up with for jiapei to try to replicate it | 22:04 |
| corvus | that's what zuul-executor should be running | 22:04 |
| jungleboyj | corvus: As what user? | 22:04 |
| jungleboyj | Zuul? | 22:04 |
| fungi | yeah, so that's working but when it's run under python git exits -13? very bizarre | 22:04 |
| corvus | yes, assuming zuul-executor is running as the zuul user | 22:04 |
| corvus | (that's worth checking :) | 22:05 |
| *** bobh has quit IRC | 22:05 | |
| fungi | as long as /var/lib/zuul/.ssh/id_rsa and all the directories leading up to it are readable by the zuul user and that file and the parent dir are only readable/owned by the zuul user that should work fine | 22:05 |
| jungleboyj | Just checked. It is. | 22:05 |
| corvus | jungleboyj: jiapei earlier posted this config snippet: http://paste.openstack.org/show/724846/ | 22:06 |
| corvus | which had the wrong path for the ssh key | 22:06 |
| fungi | also, is there a /var/lib/zuul/.ssh/id_rsa.pub and does it have the same last modified time as /var/lib/zuul/.ssh/id_rsa? ssh has some strange and inconsistent behaviors if those weren't generated together even though the .pub is technically never used by sshclient | 22:07 |
| corvus | fungi: oh good point i've seen that | 22:07 |
| corvus | jungleboyj: it should be: [executor] private_key_file=/var/lib/zuul/.ssh/id_rsa | 22:07 |
| corvus | (or, i think, omitting it entirely would use that as the default) | 22:07 |
| *** boden has quit IRC | 22:07 | |
| *** apetrich has quit IRC | 22:08 | |
| corvus | well, ~/.ssh/id_rsa is the default -- https://zuul-ci.org/docs/zuul/admin/components.html#attr-executor.private_key_file | 22:08 |
| jungleboyj | https://www.irccloud.com/pastebin/8gI3U6uE/ | 22:08 |
| jungleboyj | Yeah, I made sure the keys were right. | 22:08 |
| corvus | jungleboyj: can you try with this at the beginning of the command? GIT_SSH_COMMAND="ssh -i /var/lib/zuul/.ssh/id_rsa" | 22:09 |
| jungleboyj | corvus: I did set that before I ran git. | 22:09 |
| fungi | just helps rule out that $HOME is somewhere other than /var/lib/zuul resulting in you using a different set of ssh keys than you think | 22:10 |
| corvus | jungleboyj: and exported it? | 22:10 |
| jungleboyj | https://www.irccloud.com/pastebin/RwtlFMlr/ | 22:11 |
| *** slaweq has joined #openstack-infra | 22:11 | |
| corvus | jungleboyj: and /etc/zuul/zuul.conf has [executor] private_key_file set to /var/lib/zuul/.ssh/id_rsa or is unset? | 22:11 |
| *** apetrich has joined #openstack-infra | 22:12 | |
| jungleboyj | corvus: Actually we have nodepool_rsa in there but same result either way: | 22:13 |
| jungleboyj | https://www.irccloud.com/pastebin/VZj33aDS/ | 22:13 |
| corvus | jungleboyj: you added both of those keys to gerrit? | 22:13 |
| jungleboyj | Yes. | 22:13 |
| jungleboyj | :-) Have tried just a few things lately. | 22:13 |
| fungi | i'll double check it's the same account | 22:14 |
| clarkb | fungi: you may know the answe rto my question on https://review.openstack.org/#/c/581308/1 | 22:15 |
| fungi | grr... irccloud's pasteserv hijacks my mouse buffer | 22:15 |
| clarkb | (but feel free to help debug third party ci) | 22:15 |
| jungleboyj | fungi: Sorry. | 22:15 |
| *** slaweq has quit IRC | 22:16 | |
| *** dave-mccowan has quit IRC | 22:16 | |
| corvus | jungleboyj: if it's a possibility for you (ie, you have a node you can rebuild, or you just irrationally trust me), i'd be happy to log in and poke at things. if not, i completly understand. i've rarely been in a position to allow that. :) | 22:16 |
| fungi | just confirming here... username:lenovo_lxca_ci as maps to gerrit account_id 28429 which has no rows in the account_ssh_keys table | 22:17 |
| fungi | jungleboyj: you're adding ssh keys, but to what gerrit account? | 22:18 |
| jungleboyj | :-) I am not sure that I can get you into a node but I could set up screen sharing here at some point. | 22:18 |
| openstackgerrit | Merged openstack-infra/system-config master: Surround substitute with inflate/deflate https://review.openstack.org/583401 | 22:18 |
| jungleboyj | lenovo.lxca.ci@gmail.com | 22:18 |
| * fungi suspects there is more than one gerrit account | 22:18 | |
| jungleboyj | fungi: ^^^ | 22:18 |
| fungi | checking | 22:18 |
| fungi | that's the preferred_email for account_id 28429 | 22:19 |
| corvus | jungleboyj: is ssh-agent running in your shell? env|grep SSH_AUTH_SOCK | 22:19 |
| fungi | jungleboyj: when you get a moment, how were you adding ssh keys to that account? through the webui? and could you confirm the account id number it displays under settings for that? | 22:20 |
| jungleboyj | fungi: Looking. | 22:20 |
| fungi | wondering if it's something other than 28429 | 22:20 |
| jungleboyj | corvus: Nothing comes back for SSH_AUTH_SOCK Shouldn't be an agent running. | 22:21 |
| fungi | because gerrit's database has no ssh keys on record for 28429 | 22:21 |
| roman_g | mordred: thank you for comments (read the docs patch set) | 22:22 |
| jungleboyj | fungi: 28429 is what I show. Yes, I used the WebUI to add keys. | 22:23 |
| mordred | roman_g: you're welcome! I want to find a good solution for that - I need it for one of my repos too | 22:23 |
| jungleboyj | fungi: Have 10 keys that show up in the web UI. | 22:23 |
| corvus | i wonder if a "ssh -v" would be useful. like: GIT_SSH_COMMAND="ssh -v -i /var/lib/zuul/.ssh/id_rsa" git clone ssh://lenovo_lxca_ci@review.openstack.org:29418/openstack/ironic /tmp/ironic | 22:23 |
| fungi | bizarre i don't see any rows in the ssh_keys table where account_id=28429 | 22:23 |
| roman_g | mordred: we have situation which is worse: we have ~8-9 repos | 22:24 |
| roman_g | with RTD | 22:24 |
| fungi | er, the account_ssh_keys table | 22:24 |
| mordred | roman_g: this https://review.openstack.org/#/c/581023/ is what I started playing with - but that's an annoying amount to copy/paste | 22:24 |
| jungleboyj | I am on the 'SSH Public Keys' tab for review.o.o | 22:24 |
| mordred | roman_g: yah. it's the end of the day here so I'm not super useful anymore today- but I do want to help try to find a good solution | 22:25 |
| roman_g | mordred: currently we use something like you do. | 22:25 |
| corvus | roman_g, mordred: how about we do a central job? | 22:25 |
| roman_g | corvus: https://review.openstack.org/#/c/579434/ it's here | 22:25 |
| corvus | it seems like something we can support without too much trouble in openstack-infra | 22:26 |
| roman_g | we would love to have something like this | 22:26 |
| corvus | roman_g: no that's not what i meant. i left a comment on that suggesting that we proceed with monty's first option. | 22:26 |
| roman_g | aah | 22:26 |
| *** sambetts|afk has quit IRC | 22:26 | |
| roman_g | that would be even better | 22:26 |
| corvus | roman_g: so the role in that change should be fine. we'd just move the job definition to project-config and add the secret for the openstack-infra rtd account there too. | 22:27 |
| corvus | fungi: fwiw, i agree with your sql results. | 22:28 |
| *** sambetts_ has joined #openstack-infra | 22:28 | |
| fungi | by comparison, my gerrit account id has a couple of keys in that table | 22:28 |
| jungleboyj | Then wouldn't it fail if I was doing it by hand too? | 22:28 |
| clarkb | fungi: if you add a new key does it end up there too? (possibly they go somewhere else now?) | 22:28 |
| fungi | also possible they're cached in memory and not written out to the db frequently | 22:29 |
| corvus | clarkb: maybe: max(account_id) from that table is 26913 | 22:29 |
| * fungi is also mowing the lawn, so somewhat limited in troubleshooting capabilities | 22:29 | |
| jungleboyj | fungi: Hey, I have to go do that too. :-) | 22:30 |
| corvus | jungleboyj: yes it should; i suspect we don't understand something about gerrit. | 22:30 |
| jungleboyj | Ok. *Phew* | 22:30 |
| fungi | yeah, the account_ssh_keys table may be cruft from an older gerrit version | 22:30 |
| fungi | just hanging around since the last upgrade | 22:30 |
| roman_g | corvus: and then what would be with https://docs.openstack.org/infra/openstack-zuul-jobs/project-templates.html#project_template-docs-on-readthedocs ? | 22:30 |
| *** myoung is now known as myoung|off | 22:31 | |
| corvus | clarkb, fungi, jungleboyj: ssh keys are now in All-Users.git | 22:32 |
| fungi | neat! | 22:32 |
| fungi | part of the notedb transition apparently | 22:32 |
| *** rcernin has joined #openstack-infra | 22:32 | |
| corvus | i see some keys there for that account | 22:33 |
| fungi | okay, scratching that from my list of troubleshooting options | 22:33 |
| corvus | roman_g: we should replace the existing trigger-readthedocs jobs | 22:34 |
| corvus | since they don't work | 22:34 |
| corvus | but keep them in that template | 22:34 |
| roman_g | corvus: got it. Thank you. | 22:35 |
| *** jtomasek has quit IRC | 22:36 | |
| roman_g | When to expect that to happen? (should I implement workaround, or it will be completed pretty fast?) | 22:36 |
| corvus | ianw: ^ thoughts on 579434? | 22:38 |
| ianw | corvus: yeah, i think possibly the add-zuul-account-to-your-publisher-account thing is probably the way to go | 22:40 |
| corvus | jungleboyj, infra-root: neat new debugging method for gerrit doesn't require admin access: http://paste.openstack.org/show/726153/ | 22:40 |
| fungi | clarkb: on 581308 i have no idea when ed25519 support landed in openssh. mailing list discussions suggest some time in 2013 but i'd be hard-pressed to pin it to a particular release without further digging and ultimately no way to know what red hat has backported other than to test really | 22:41 |
| jungleboyj | corvus: Yikes. | 22:42 |
| corvus | jungleboyj: yikes? (that doesn't scare me -- they're only public keys) | 22:42 |
| jungleboyj | corvus: Oh, true enough. | 22:42 |
| roman_g | ianw: corvus: create account, and I would make it to be added to our readthedocs sites. Notify me of the username/e-mail when you will have it, please. Thank you! | 22:43 |
| fungi | does leave me curious as to what else is in that tree | 22:43 |
| * mordred really likes that interface - super cool ability for humans to access stuff | 22:43 | |
| fungi | but i don't have time to dissect the repo | 22:43 |
| corvus | fungi: currently that's the only file | 22:43 |
| fungi | hah, okay then | 22:43 |
| corvus | oh there's more | 22:44 |
| corvus | i checked out my branch and have preferences.config and watch.config | 22:44 |
| mordred | corvus: zomg | 22:44 |
| ianw | roman_g: yes, i'll have a go at doing it like this, keep an eye on that review | 22:44 |
| corvus | here's my content (other than ssh keys) http://paste.openstack.org/show/726156/ | 22:45 |
| mordred | corvus: are you authenticated? | 22:45 |
| roman_g | ianw: thank you! I'm subscribed on that PS. | 22:45 |
| * roman_g went home | 22:45 | |
| corvus | mordred: i just did an https clone, so i hope not, unless i left some netrc like thing laying around | 22:45 |
| mordred | nod | 22:46 |
| mordred | corvus: also - what's the path foryou and me? | 22:46 |
| mordred | I just tried: users/2/2 | 22:46 |
| corvus | mordred: users/02/2 | 22:46 |
| mordred | of course. silly me | 22:46 |
| corvus | mordred: you have a lot of dashboards :) | 22:48 |
| mordred | I do! | 22:48 |
| *** lbragstad has joined #openstack-infra | 22:50 | |
| fungi | who love dashboards? mordred--that's who! | 22:50 |
| clarkb | I think mirrors just updated | 22:50 |
| clarkb | double checking then will recheck pbrx change | 22:50 |
| clarkb | yup looks like it | 22:50 |
| mordred | clarkb: I concur | 22:51 |
| pabelanger | mordred: yah, now that the pbrx stuff is working, I should be able to test POC locally | 22:52 |
| mordred | pabelanger: sweet. | 22:52 |
| *** graphene has quit IRC | 22:52 | |
| clarkb | assuming this works, plan to switch over tomorrow morning maybe? | 22:54 |
| *** graphene has joined #openstack-infra | 22:54 | |
| clarkb | this being pypi proxy | 22:54 |
| *** eernst has quit IRC | 22:54 | |
| mordred | clarkb: ++ | 22:54 |
| clarkb | unless ianw wants to watch it | 22:54 |
| pabelanger | corvus: clarkb: I do like the idea of 3pci pipelines and central base jobs, was also part of my thought for http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-July/000508.html for keeping log secrets in trusted base, and rest of base untrusted in a shared 3pci-jobs repo | 22:54 |
| pabelanger | I'm hoping in next week to try that idea too | 22:54 |
| *** tpsilva has quit IRC | 22:55 | |
| ianw | clarkb: sorry watch what? | 22:55 |
| clarkb | ianw: we ran out of disk on the pypi volume. Current plan is to switch to a proxy to pypi.org. Then we can possibly rebuild the mirror on afs with our blacklists (or maybe decide pypi proxy is good enough) | 22:56 |
| clarkb | mor ethan happy to watch the switch happen in my morning | 22:57 |
| ianw | clarkb: it might be best to do that, by the time i context switch it all in if it goes pear shaped it will probably be your morning anyway :) | 22:59 |
| *** caphrim007_ has joined #openstack-infra | 23:00 | |
| *** caphrim007_ has quit IRC | 23:01 | |
| fungi | to be fair, it was already pear-shaped when we got here | 23:01 |
| clarkb | mordred: looks like 583318 is still talking to :80 not :8080 | 23:01 |
| clarkb | so we may need base-test dance afterall | 23:02 |
| clarkb | I just ate a burrito that was spicier than I anticipated | 23:02 |
| fungi | "something wrong with the pypi mirror? some new package isn't showing up" ... errors in bandersnatch log about filesystem full ... afs shows volume quota at 100% since sometime early utc monday ... alternatives get discussed | 23:03 |
| *** caphrim007 has quit IRC | 23:03 | |
| fungi | clarkb: there's a place i miss next to where i used to live in raleigh: "flying burrito" | 23:03 |
| fungi | their "ultimate sauce" for burritos was, i kid you not, pureed habanero | 23:04 |
| fungi | firey yet delectable | 23:04 |
| clarkb | that sounds worse than my current situation as I bet I could not type if I ate that | 23:04 |
| fungi | definitely hitting them on the way through raleigh on vacation | 23:05 |
| jungleboyj | fungi: corvus Thanks for the continued help. I need to drop off for now. | 23:05 |
| mordred | clarkb: I disagree - I see it talking to :8080 | 23:05 |
| jungleboyj | Will ping you if we figure anything else out or need mor ehelp. | 23:05 |
| mordred | clarkb: http://logs.openstack.org/18/583318/3/check/pbrx-build-zuul-containers/dd1d60d/job-output.txt.gz#_2018-07-17_22_55_15_433395 | 23:05 |
| fungi | as we pass through asheville, "nine mile" (jamaican/caribbean cuisine) has an undocumented spiciness level for some of their curries called "face melt" which... fairly literal. that's day #2 on the road trip | 23:06 |
| mordred | clarkb: oh - duh. nm. that was the job that set it in .zuul.yaml | 23:07 |
| fungi | i truly feel my face going numb and sliding off when i eat a raggy road done face melt | 23:07 |
| fungi | if you order an extra hot curry and finish it, the wait staff _might_ clue you in to "face melt" | 23:08 |
| corvus | jungleboyj: ok. let me know if i can help further (including screen sharing, etc). | 23:09 |
| *** tosky has joined #openstack-infra | 23:09 | |
| clarkb | mordred: is there a better thing to recheck? (I'm somewhat lost between all the things I've paged in today) | 23:10 |
| jungleboyj | corvus: Will do. Thank you so much! | 23:10 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** rpioso is now known as rpioso|afk | 23:13 | |
| *** slaweq has quit IRC | 23:16 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Run kata static checks https://review.openstack.org/583296 | 23:19 |
| openstackgerrit | Tony Breeds proposed openstack/diskimage-builder master: Support different versions of networkx https://review.openstack.org/506162 | 23:21 |
| *** rlandy has quit IRC | 23:24 | |
| openstackgerrit | Tony Breeds proposed openstack/diskimage-builder master: Update pylint to 1.7.6, uncap networkx https://review.openstack.org/583413 | 23:27 |
| mnaser | I figure most people here might appreciate this but we work with an IT recycling company that I visited here | 23:31 |
| ianw | corvus / mordred : so, for RTD the shared account isn't as nice as pypi. each project has a unique webhook URL to build it. so ISTM we can either keep all the URLS encrypted in project-config ... or require each project that wants to publish to RTD to call the publisher role manually with secrets in their playbook | 23:31 |
| mnaser | And I found this gem... | 23:31 |
| mnaser | https://usercontent.irccloud-cdn.com/file/rhxFBnja/IMG_4248.JPG | 23:31 |
| mnaser | https://usercontent.irccloud-cdn.com/file/jzkCJMII/IMG_4250.JPG | 23:31 |
| mnaser | Funny how I ran I got them | 23:32 |
| mnaser | Into them** | 23:32 |
| fungi | bwahahaha | 23:33 |
| fungi | the nebula cloud is alive and well somewhere inside vexxhost? | 23:33 |
| fungi | bits of their network gear anyway | 23:34 |
| fungi | or this is inside their facility? | 23:34 |
| fungi | yeah, i guess none of that is plugged in | 23:34 |
| fungi | so... on the scrap pile | 23:34 |
| corvus | ianw: the webhook url isn't secret though, right? so we can keep the account creds and the job definition in project-config, and then add the webhook url as a job variable which can be overridden in the project-pipeline variant? | 23:36 |
| *** agopi is now known as agopi|off | 23:36 | |
| ianw | corvus: well i'm not sure if it supposed to be secret or not? | 23:37 |
| corvus | oh. fascinating. | 23:37 |
| ianw | in the old api, anyone could hit it and basically request a new job. in this new one, you have to log in and get this integration token | 23:37 |
| ianw | the results of hitting it don't seem to be worse ... but i'm guessing they wanted to move to a secret model? | 23:37 |
| ianw | s/job/docs build/ | 23:38 |
| fungi | mnaser: aha, there is/was one (or more) chinese company going by "nebula" manufacturing network hardware, so sayeth some web searching on those part numbers | 23:40 |
| *** dpawlik has joined #openstack-infra | 23:48 | |
| *** pguimaraes has quit IRC | 23:48 | |
| mnaser | fungi: not using any of it but I thought it was a bit fun to see them | 23:52 |
| *** dpawlik has quit IRC | 23:52 | |
| *** bdodd_ has joined #openstack-infra | 23:57 | |
| *** bdodd_ has quit IRC | 23:57 | |
| *** bdodd has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
