| *** aspiers[m] has quit IRC | 00:05 | |
| *** ilpianista_ has quit IRC | 00:05 | |
| *** masayukig[m] has quit IRC | 00:05 | |
| *** bandini has quit IRC | 00:09 | |
| *** bandini has joined #openstack-infra | 00:10 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** masayukig[m] has joined #openstack-infra | 00:12 | |
| *** bgmccollum has quit IRC | 00:13 | |
| *** slaweq has quit IRC | 00:15 | |
| *** d0ugal has quit IRC | 00:15 | |
| *** AJaeger has quit IRC | 00:16 | |
| *** sree has joined #openstack-infra | 00:18 | |
| dmsimard | ianw: you're back? :D | 00:18 |
|---|---|---|
| *** comstud has quit IRC | 00:19 | |
| *** bgmccollum has joined #openstack-infra | 00:20 | |
| *** claudiub has quit IRC | 00:20 | |
| *** masber has joined #openstack-infra | 00:21 | |
| ianw | dmsimard: yes, currently context switching back in from swap :) | 00:21 |
| dmsimard | oh, you missed out on a bunch of fun | 00:21 |
| *** comstud has joined #openstack-infra | 00:22 | |
| ianw | next time we find structural flaws in every modern processor, it would be nice if didn't happen over the new year break | 00:22 |
| clarkb | ++ | 00:22 |
| dmsimard | no worries, that vulnerability is about 20 years old | 00:22 |
| clarkb | we are stil lwaiting on ubuntu patches fwiw | 00:23 |
| clarkb | also my debian jessie server doesn't have a new kernel yet either | 00:23 |
| dmsimard | yeah I think Ubuntu was waiting for ianw to be back to make sure he's there to help us :D | 00:23 |
| ianw | now every system call basically implies a context switch we might as well all just move to a microkernel anyway | 00:26 |
| ianw | brb, just checking how puppet works with minix :) | 00:27 |
| dmsimard | containers ? I was promised that containers would fix all my problems | 00:27 |
| *** d0ugal has joined #openstack-infra | 00:27 | |
| *** AJaeger has joined #openstack-infra | 00:27 | |
| ianw | containers seem like they will be an almost infinite source of side-channel attacks imo | 00:29 |
| *** masber has quit IRC | 00:29 | |
| *** sree has quit IRC | 00:30 | |
| *** pcichy has joined #openstack-infra | 00:34 | |
| *** ilpianista_ has joined #openstack-infra | 00:35 | |
| *** aspiers[m] has joined #openstack-infra | 00:35 | |
| *** xarses_ has joined #openstack-infra | 00:39 | |
| *** Apoorva has quit IRC | 00:40 | |
| *** tosky has quit IRC | 00:41 | |
| *** sree has joined #openstack-infra | 00:42 | |
| *** xarses_ has quit IRC | 00:49 | |
| *** sree has quit IRC | 00:51 | |
| *** sree has joined #openstack-infra | 00:56 | |
| *** dave-mccowan has joined #openstack-infra | 01:07 | |
| *** masber has joined #openstack-infra | 01:10 | |
| *** sshnaidm has quit IRC | 01:11 | |
| *** sshnaidm has joined #openstack-infra | 01:12 | |
| *** sree has quit IRC | 01:13 | |
| *** liusheng has joined #openstack-infra | 01:19 | |
| *** daidv has joined #openstack-infra | 01:20 | |
| *** daidv_ has joined #openstack-infra | 01:20 | |
| *** edmondsw has joined #openstack-infra | 01:21 | |
| *** liujiong has joined #openstack-infra | 01:22 | |
| *** daidv_ has quit IRC | 01:24 | |
| *** edmondsw has quit IRC | 01:26 | |
| *** sree has joined #openstack-infra | 01:33 | |
| *** kiennt26 has joined #openstack-infra | 01:36 | |
| *** sree has quit IRC | 01:37 | |
| *** leakypipes has quit IRC | 01:39 | |
| *** sshnaidm has quit IRC | 01:39 | |
| *** sshnaidm has joined #openstack-infra | 01:40 | |
| *** cuongnv has joined #openstack-infra | 01:45 | |
| *** annp has joined #openstack-infra | 01:54 | |
| *** hongbin has joined #openstack-infra | 02:03 | |
| *** openStackUser has joined #openstack-infra | 02:24 | |
| *** liusheng has quit IRC | 02:25 | |
| *** liusheng has joined #openstack-infra | 02:26 | |
| openStackUser | msg nickserv register Passcode1 mshankar4142@gmail.com | 02:30 |
| *** zhurong has joined #openstack-infra | 02:46 | |
| *** openStackUser has quit IRC | 03:01 | |
| *** liujiong has quit IRC | 03:03 | |
| *** sree has joined #openstack-infra | 03:07 | |
| *** hongbin has quit IRC | 03:09 | |
| *** dave-mccowan has quit IRC | 03:09 | |
| *** hongbin has joined #openstack-infra | 03:09 | |
| *** edmondsw has joined #openstack-infra | 03:09 | |
| *** hongbin has quit IRC | 03:10 | |
| *** hongbin has joined #openstack-infra | 03:10 | |
| *** hongbin has quit IRC | 03:10 | |
| *** hongbin has joined #openstack-infra | 03:11 | |
| *** hongbin has quit IRC | 03:12 | |
| *** hongbin has joined #openstack-infra | 03:13 | |
| *** edmondsw has quit IRC | 03:14 | |
| *** hongbin has quit IRC | 03:14 | |
| *** hongbin has joined #openstack-infra | 03:15 | |
| *** kong has quit IRC | 03:17 | |
| *** kong has joined #openstack-infra | 03:17 | |
| *** liujiong has joined #openstack-infra | 03:36 | |
| *** ramishra has joined #openstack-infra | 03:44 | |
| *** ddurst has quit IRC | 03:46 | |
| *** yamamoto has joined #openstack-infra | 03:58 | |
| *** yamamoto has quit IRC | 04:03 | |
| *** ddurst has joined #openstack-infra | 04:11 | |
| *** dhajare has joined #openstack-infra | 04:14 | |
| *** hongbin has quit IRC | 04:19 | |
| *** sflanigan has joined #openstack-infra | 04:27 | |
| *** sflanigan has joined #openstack-infra | 04:27 | |
| *** aeng has quit IRC | 04:32 | |
| *** bhavik1 has joined #openstack-infra | 04:33 | |
| *** gcb has joined #openstack-infra | 04:41 | |
| *** dhajare has quit IRC | 04:46 | |
| *** aeng has joined #openstack-infra | 04:47 | |
| *** bhavik1 has quit IRC | 04:50 | |
| *** zhurong has quit IRC | 04:52 | |
| *** udesale has joined #openstack-infra | 04:54 | |
| *** edmondsw has joined #openstack-infra | 04:58 | |
| *** edmondsw has quit IRC | 05:02 | |
| *** yamamoto has joined #openstack-infra | 05:04 | |
| *** openStackUser has joined #openstack-infra | 05:05 | |
| *** sflanigan has quit IRC | 05:07 | |
| *** liujiong has quit IRC | 05:08 | |
| *** yamamoto has quit IRC | 05:12 | |
| *** pcichy has quit IRC | 05:24 | |
| *** armaan has joined #openstack-infra | 05:26 | |
| *** janki has joined #openstack-infra | 05:28 | |
| *** mikal has quit IRC | 05:33 | |
| *** psachin has joined #openstack-infra | 05:33 | |
| openstackgerrit | Goutham Pratapa proposed openstack-infra/project-config master: Add Templates to Kingbird-Dashboard. https://review.openstack.org/530452 | 05:38 |
| *** dsariel has joined #openstack-infra | 05:39 | |
| *** dsariel has quit IRC | 05:44 | |
| *** zhurong has joined #openstack-infra | 05:52 | |
| *** pgadiya has joined #openstack-infra | 05:53 | |
| *** liujiong has joined #openstack-infra | 05:55 | |
| AJaeger | openStackUser: hope you changed your password... | 06:01 |
| chandankumar | AJaeger: Good morning | 06:04 |
| chandankumar | AJaeger: is the publish-to-pypi jobs fixed? | 06:04 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Use tempest-plugin-jobs project template for all tempest plugins https://review.openstack.org/531339 | 06:10 |
| AJaeger | chandankumar: we haven't discussed that yet, sorry | 06:14 |
| *** liusheng has quit IRC | 06:14 | |
| chandankumar | AJaeger: some projects need tempest plugins on pypi, will i send a seperate patch for them? | 06:15 |
| chandankumar | congress requires telemetry and mistral tempest plugin | 06:15 |
| AJaeger | chandankumar: put it on our agenda at https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting - for tomorrow's IRC meeting | 06:17 |
| *** janki has quit IRC | 06:17 | |
| chandankumar | AJaeger: sure :-) | 06:17 |
| AJaeger | chandankumar: yes, splitting those up might be best | 06:17 |
| AJaeger | chandankumar: I meant to say : I *just* put it there... | 06:17 |
| *** janki has joined #openstack-infra | 06:18 | |
| *** aeng has quit IRC | 06:18 | |
| *** openStackUser has quit IRC | 06:21 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't treat finger client disconnect as exception https://review.openstack.org/531408 | 06:24 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] Support networkx 2.0 https://review.openstack.org/506524 | 06:30 |
| *** khappone has joined #openstack-infra | 06:35 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] Support networkx 2.0 https://review.openstack.org/506524 | 06:36 |
| *** florianf has joined #openstack-infra | 06:39 | |
| *** dsariel has joined #openstack-infra | 06:41 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Share a fake pull request database across connections https://review.openstack.org/531506 | 06:41 |
| *** claudiub has joined #openstack-infra | 06:43 | |
| *** udesale__ has joined #openstack-infra | 06:45 | |
| *** edmondsw has joined #openstack-infra | 06:46 | |
| *** jbadiapa has joined #openstack-infra | 06:47 | |
| *** udesale has quit IRC | 06:48 | |
| *** edmondsw has quit IRC | 06:50 | |
| *** coolsvap has joined #openstack-infra | 06:51 | |
| *** udesale has joined #openstack-infra | 06:52 | |
| *** udesale__ has quit IRC | 06:54 | |
| *** dhajare has joined #openstack-infra | 06:56 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove the default argument to secure in nodepoolcmd https://review.openstack.org/480310 | 06:58 |
| *** kjackal has joined #openstack-infra | 07:00 | |
| *** pcaruana has joined #openstack-infra | 07:04 | |
| *** mat128 has quit IRC | 07:09 | |
| *** Kevin_Zheng has quit IRC | 07:13 | |
| *** Shrews has quit IRC | 07:13 | |
| *** knikolla has quit IRC | 07:13 | |
| *** mgkwill has quit IRC | 07:13 | |
| *** jungleboyj has quit IRC | 07:13 | |
| *** knikolla has joined #openstack-infra | 07:13 | |
| *** mgkwill has joined #openstack-infra | 07:13 | |
| *** jungleboyj has joined #openstack-infra | 07:13 | |
| *** Shrews has joined #openstack-infra | 07:14 | |
| *** gaurangt has quit IRC | 07:15 | |
| *** Krenair has quit IRC | 07:15 | |
| *** Kevin_Zheng has joined #openstack-infra | 07:16 | |
| *** gaurangt has joined #openstack-infra | 07:18 | |
| *** Krenair has joined #openstack-infra | 07:22 | |
| *** liusheng has joined #openstack-infra | 07:22 | |
| *** slaweq has joined #openstack-infra | 07:30 | |
| *** makowals has joined #openstack-infra | 07:36 | |
| *** armaan_ has joined #openstack-infra | 07:36 | |
| *** armaan has quit IRC | 07:37 | |
| *** dsariel has quit IRC | 07:49 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Use tempest plugin jobs and pypi jobs for Mistral tempest plugin https://review.openstack.org/531676 | 07:54 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: tempest plugin and pypi jobs for Mistral tempest plugin https://review.openstack.org/531676 | 07:57 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: tempest plugin and pypi jobs for Telemetry tempest plugin https://review.openstack.org/531678 | 08:00 |
| *** eumel8 has joined #openstack-infra | 08:03 | |
| *** armaan_ has quit IRC | 08:03 | |
| *** eumel8 has quit IRC | 08:08 | |
| *** tesseract has joined #openstack-infra | 08:09 | |
| *** ralonsoh has joined #openstack-infra | 08:09 | |
| *** namnh has joined #openstack-infra | 08:09 | |
| *** jtomasek has joined #openstack-infra | 08:12 | |
| *** shardy has joined #openstack-infra | 08:18 | |
| *** pabelanger has quit IRC | 08:19 | |
| *** weshay has quit IRC | 08:19 | |
| *** aviau has quit IRC | 08:19 | |
| *** lucas-afk has quit IRC | 08:19 | |
| *** rook has quit IRC | 08:19 | |
| *** aviau has joined #openstack-infra | 08:19 | |
| *** honza has quit IRC | 08:20 | |
| *** honza has joined #openstack-infra | 08:20 | |
| *** pabelanger has joined #openstack-infra | 08:20 | |
| *** weshay has joined #openstack-infra | 08:20 | |
| *** lucasagomes has joined #openstack-infra | 08:20 | |
| *** honza is now known as Guest14500 | 08:20 | |
| *** rook has joined #openstack-infra | 08:21 | |
| *** armaan has joined #openstack-infra | 08:21 | |
| *** rook is now known as Guest92761 | 08:21 | |
| *** eumel8 has joined #openstack-infra | 08:30 | |
| *** edmondsw has joined #openstack-infra | 08:34 | |
| *** hashar has joined #openstack-infra | 08:36 | |
| *** jpena has joined #openstack-infra | 08:38 | |
| *** edmondsw has quit IRC | 08:39 | |
| *** gibi_away is now known as gibi | 08:44 | |
| *** nmagnezi has joined #openstack-infra | 08:45 | |
| *** dalvarez has joined #openstack-infra | 08:47 | |
| *** rossella_s has joined #openstack-infra | 08:53 | |
| openstackgerrit | Merged openstack/ptgbot master: Remove ethercalc support https://review.openstack.org/525167 | 09:00 |
| *** jascott1 has quit IRC | 09:02 | |
| openstackgerrit | Merged openstack/ptgbot master: Rooms are now called Tracks https://review.openstack.org/525169 | 09:04 |
| openstackgerrit | Merged openstack/ptgbot master: Specify and load event schedule https://review.openstack.org/525168 | 09:04 |
| openstackgerrit | Merged openstack/ptgbot master: Add scheduled / additional rooms https://review.openstack.org/529345 | 09:04 |
| openstackgerrit | Merged openstack/ptgbot master: Display scheduled and additional rooms in web page https://review.openstack.org/529346 | 09:04 |
| openstackgerrit | Merged openstack/ptgbot master: Add ability to book a room https://review.openstack.org/529587 | 09:04 |
| openstackgerrit | Merged openstack/ptgbot master: Select current day tab by default https://review.openstack.org/529589 | 09:04 |
| openstackgerrit | Merged openstack/ptgbot master: Update location automatically https://review.openstack.org/529838 | 09:04 |
| openstackgerrit | Merged openstack/ptgbot master: Add command to clean up at start of day https://review.openstack.org/529842 | 09:04 |
| openstackgerrit | Merged openstack/ptgbot master: Reject some commands if a track is not scheduled https://review.openstack.org/529847 | 09:04 |
| openstackgerrit | Merged openstack/ptgbot master: Also find room for tracks in additional rooms https://review.openstack.org/529848 | 09:04 |
| openstackgerrit | Merged openstack/ptgbot master: Rewrite documentation in more user-friendly way https://review.openstack.org/529853 | 09:04 |
| *** armaan_ has joined #openstack-infra | 09:06 | |
| *** dsariel has joined #openstack-infra | 09:07 | |
| *** sshnaidm is now known as sshnaidm|afk | 09:07 | |
| *** rcernin has quit IRC | 09:07 | |
| *** sshnaidm|afk has quit IRC | 09:08 | |
| *** jpich has joined #openstack-infra | 09:08 | |
| *** sshnaidm has joined #openstack-infra | 09:08 | |
| *** armaan has quit IRC | 09:09 | |
| *** armaan_ has quit IRC | 09:10 | |
| *** armaan has joined #openstack-infra | 09:10 | |
| *** sbezverk has quit IRC | 09:11 | |
| *** Guest14500 is now known as honza | 09:14 | |
| *** dbecker has joined #openstack-infra | 09:15 | |
| *** dbecker has quit IRC | 09:15 | |
| *** owalsh_ is now known as owalsh | 09:15 | |
| *** dbecker has joined #openstack-infra | 09:16 | |
| *** finucannot is now known as stephenfin | 09:16 | |
| *** Hal has joined #openstack-infra | 09:18 | |
| *** Hal has quit IRC | 09:18 | |
| *** zoli has quit IRC | 09:21 | |
| *** zoli has joined #openstack-infra | 09:22 | |
| *** danpawlik has joined #openstack-infra | 09:23 | |
| *** shardy has quit IRC | 09:27 | |
| *** armaan has quit IRC | 09:28 | |
| *** armaan has joined #openstack-infra | 09:29 | |
| *** nmagnezi has quit IRC | 09:30 | |
| *** shardy has joined #openstack-infra | 09:32 | |
| *** efoley has joined #openstack-infra | 09:32 | |
| *** e0ne has joined #openstack-infra | 09:33 | |
| AJaeger | infra-root, kingbird-dashboard new repo creation was merged yesterday but the repo is not fully created, see http://git.openstack.org/cgit/openstack/kingbird-dashboard/ . | 09:36 |
| *** pcichy has joined #openstack-infra | 09:38 | |
| AJaeger | infra-root, I cannot reach http://zuulv3.openstack.org/ and cacti shows that zuulv3 uses 6+ GB swap | 09:40 |
| *** liujiong has quit IRC | 09:43 | |
| *** liujiong has joined #openstack-infra | 09:43 | |
| *** udesale has quit IRC | 09:45 | |
| *** udesale has joined #openstack-infra | 09:46 | |
| *** derekh has joined #openstack-infra | 09:46 | |
| *** e0ne has quit IRC | 09:47 | |
| *** erlon has joined #openstack-infra | 09:54 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool feature/zuulv3: Add a separate module for node management commands https://review.openstack.org/531703 | 09:58 |
| *** rossella_s has quit IRC | 09:59 | |
| *** armaan has quit IRC | 09:59 | |
| *** rossella_s has joined #openstack-infra | 09:59 | |
| *** sree has quit IRC | 10:02 | |
| *** gcb has quit IRC | 10:02 | |
| *** zhurong has quit IRC | 10:02 | |
| *** jascott1 has joined #openstack-infra | 10:03 | |
| *** tosky has joined #openstack-infra | 10:04 | |
| *** armaan has joined #openstack-infra | 10:05 | |
| *** kiennt26 has quit IRC | 10:07 | |
| *** liujiong has quit IRC | 10:07 | |
| *** tinwood_ is now known as tinwood | 10:10 | |
| *** pbourke has quit IRC | 10:12 | |
| *** pbourke has joined #openstack-infra | 10:12 | |
| frickler | AJaeger: ack, seem to be a bit better currently, but I'll prepared for a restart anyway. experience seems to be that this doesn't last for long | 10:14 |
| *** namnh has quit IRC | 10:18 | |
| *** jklare has quit IRC | 10:19 | |
| *** pcichy has quit IRC | 10:19 | |
| *** pcichy has joined #openstack-infra | 10:20 | |
| *** jklare has joined #openstack-infra | 10:20 | |
| *** edmondsw has joined #openstack-infra | 10:22 | |
| frickler | infra-root: I restarted zuul-(scheduler|web), but restoring the queues fails with traceback http://paste.openstack.org/show/640784/ | 10:23 |
| frickler | #status notice zuul has been restarted, all queues have been reset. please recheck your patches when appropriate | 10:24 |
| openstackstatus | frickler: sending notice | 10:24 |
| -openstackstatus- NOTICE: zuul has been restarted, all queues have been reset. please recheck your patches when appropriate | 10:25 | |
| frickler | https://storyboard.openstack.org/#!/story/2001448 | 10:26 |
| openstackstatus | frickler: finished sending notice | 10:27 |
| *** edmondsw has quit IRC | 10:27 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Set remote url on every getRepo in merger https://review.openstack.org/529293 | 10:29 |
| *** cuongnv has quit IRC | 10:30 | |
| frickler | ianw: happy new year to you, too. if you are still looking to fill up your empty swap space, AJaeger has collected a nice list of reviews https://etherpad.openstack.org/p/Nvt3ovbn5x :) | 10:31 |
| *** annp has quit IRC | 10:32 | |
| *** udesale has quit IRC | 10:34 | |
| openstackgerrit | liusheng proposed openstack-infra/zuul feature/zuulv3: Use hotlink instead log url in github job report https://review.openstack.org/531545 | 10:34 |
| *** udesale has joined #openstack-infra | 10:35 | |
| frickler | oh, that traceback happens when one doesn't have permission to read the zuul.conf file, pretty bad error handling I say. :( queues restored now | 10:35 |
| *** gema has joined #openstack-infra | 10:39 | |
| gema | need to add a new cloud to infra, can someone point me in the right direction as to how big the nodepool project needs to be and who to talk to ? | 10:40 |
| *** ldnunes has joined #openstack-infra | 10:44 | |
| *** rossella_s has quit IRC | 10:46 | |
| *** rossella_s has joined #openstack-infra | 10:46 | |
| *** sambetts|afk is now known as sambetts | 10:50 | |
| *** simondodsley has quit IRC | 10:50 | |
| *** simondodsley has joined #openstack-infra | 10:51 | |
| *** jamespage has quit IRC | 10:51 | |
| *** Krinkle has quit IRC | 10:51 | |
| *** jamespage has joined #openstack-infra | 10:52 | |
| *** Krinkle has joined #openstack-infra | 10:52 | |
| *** ldnunes has quit IRC | 10:52 | |
| *** ldnunes has joined #openstack-infra | 10:54 | |
| *** udesale has quit IRC | 10:54 | |
| *** armaan has quit IRC | 10:55 | |
| *** electrofelix has joined #openstack-infra | 10:56 | |
| *** mat128 has joined #openstack-infra | 10:56 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool feature/zuulv3: [WIP] webapp: add optional admin endpoint https://review.openstack.org/531718 | 11:00 |
| *** wolverineav has joined #openstack-infra | 11:00 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool feature/zuulv3: Refactor status functions, add web endpoints, allow params https://review.openstack.org/531390 | 11:02 |
| *** armaan has joined #openstack-infra | 11:02 | |
| *** numans has quit IRC | 11:06 | |
| *** numans has joined #openstack-infra | 11:08 | |
| *** rfolco has joined #openstack-infra | 11:21 | |
| openstackgerrit | Shachar Snapiri proposed openstack-infra/system-config master: Add openstack-dragonflow channel to statusbot https://review.openstack.org/531720 | 11:23 |
| snapiri | Hi Infra, could anyone please help review ^^^ | 11:23 |
| *** Guest92761 is now known as rook | 11:27 | |
| *** rcarrill1 has joined #openstack-infra | 11:32 | |
| *** rcarrillocruz has quit IRC | 11:35 | |
| *** zhurong has joined #openstack-infra | 11:35 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: doc: Rework features doc https://review.openstack.org/512248 | 11:39 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: doc: Minor rework of usage doc https://review.openstack.org/512249 | 11:39 |
| *** smatzek has joined #openstack-infra | 11:40 | |
| *** smatzek has quit IRC | 11:40 | |
| *** smatzek has joined #openstack-infra | 11:40 | |
| *** jkilpatr has quit IRC | 11:50 | |
| *** ldnunes has quit IRC | 11:53 | |
| *** ldnunes has joined #openstack-infra | 11:55 | |
| *** jistr is now known as jistr|mtg | 11:58 | |
| *** lucasagomes is now known as lucas-hungry | 12:00 | |
| *** sdague has joined #openstack-infra | 12:00 | |
| *** zhongjun has quit IRC | 12:06 | |
| *** dave-mccowan has joined #openstack-infra | 12:09 | |
| *** edmondsw has joined #openstack-infra | 12:10 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul-jobs master: Add roles for static web content build and publication https://review.openstack.org/531406 | 12:11 |
| *** gcb has joined #openstack-infra | 12:13 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul-jobs master: Add roles for static web content build and publication https://review.openstack.org/531406 | 12:14 |
| *** edmondsw has quit IRC | 12:15 | |
| *** udesale has joined #openstack-infra | 12:16 | |
| *** rhallisey has joined #openstack-infra | 12:17 | |
| *** jkilpatr has joined #openstack-infra | 12:22 | |
| *** janki has quit IRC | 12:24 | |
| *** rcarrill1 is now known as rcarrillocruz | 12:24 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul-jobs master: Add roles for static web content build and publication https://review.openstack.org/531406 | 12:26 |
| *** jpena is now known as jpena|lunch | 12:33 | |
| *** sree has joined #openstack-infra | 12:36 | |
| *** sree has quit IRC | 12:41 | |
| openstackgerrit | Frank Kloeker proposed openstack-infra/system-config master: Upgrade translate-dev.o.o to Zanata 4.3.3 https://review.openstack.org/531736 | 12:45 |
| eumel8 | clarkb, ianw: ^^ the patch for Zanata 4.3.3 | 12:46 |
| *** ldnunes has quit IRC | 12:47 | |
| *** ldnunes has joined #openstack-infra | 12:47 | |
| *** dave-mccowan has quit IRC | 12:47 | |
| *** armaan has quit IRC | 12:48 | |
| *** armaan has joined #openstack-infra | 12:49 | |
| *** dave-mccowan has joined #openstack-infra | 12:49 | |
| *** armaan has quit IRC | 12:49 | |
| *** armaan has joined #openstack-infra | 12:50 | |
| *** rosmaita has joined #openstack-infra | 12:54 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Also prefix the indexes if needed https://review.openstack.org/531742 | 12:55 |
| *** rcernin has joined #openstack-infra | 12:59 | |
| *** TLAXKIT has joined #openstack-infra | 13:01 | |
| *** zhurong has quit IRC | 13:01 | |
| *** dhajare has quit IRC | 13:04 | |
| *** lucas-hungry is now known as lucasagomes | 13:05 | |
| *** kjackal has quit IRC | 13:13 | |
| *** sree has joined #openstack-infra | 13:13 | |
| *** sshnaidm has quit IRC | 13:14 | |
| *** kjackal has joined #openstack-infra | 13:14 | |
| *** sshnaidm has joined #openstack-infra | 13:15 | |
| *** kjackal has quit IRC | 13:15 | |
| *** rcernin has quit IRC | 13:16 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Pin version of webob to 1.7.4 https://review.openstack.org/531507 | 13:24 |
| *** kjackal has joined #openstack-infra | 13:24 | |
| *** pcaruana has quit IRC | 13:25 | |
| *** pcaruana has joined #openstack-infra | 13:25 | |
| *** yamamoto has joined #openstack-infra | 13:28 | |
| *** jpena|lunch is now known as jpena | 13:31 | |
| *** jkilpatr has quit IRC | 13:31 | |
| *** jistr|mtg is now known as jistr | 13:35 | |
| *** coolsvap has quit IRC | 13:36 | |
| *** rlandy has joined #openstack-infra | 13:38 | |
| *** sree has quit IRC | 13:41 | |
| *** sree has joined #openstack-infra | 13:41 | |
| *** jkilpatr has joined #openstack-infra | 13:45 | |
| *** kgiusti has joined #openstack-infra | 13:46 | |
| *** sree has quit IRC | 13:46 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul-jobs master: Add roles for static web content build and publication https://review.openstack.org/531406 | 13:48 |
| *** edmondsw has joined #openstack-infra | 13:51 | |
| *** dprince has joined #openstack-infra | 13:51 | |
| *** kiennt26 has joined #openstack-infra | 13:53 | |
| *** katkapilatova1 has joined #openstack-infra | 13:54 | |
| openstackgerrit | Merged openstack-dev/pbr master: Support PEP 345 Project-URL metadata https://review.openstack.org/521689 | 13:54 |
| *** jaypipes has joined #openstack-infra | 13:55 | |
| *** yamamoto has quit IRC | 13:56 | |
| *** yamamoto has joined #openstack-infra | 13:57 | |
| *** yamamoto has quit IRC | 14:01 | |
| *** esberglu has joined #openstack-infra | 14:03 | |
| dmsimard | frickler: I created a few stories recently around zuul operations (and logging) | 14:08 |
| dmsimard | like https://storyboard.openstack.org/#!/story/2001445 and https://storyboard.openstack.org/#!/story/2001446 | 14:09 |
| dmsimard | so please document your experience as well so we can improve that | 14:09 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: switch whereto doc job to python3 https://review.openstack.org/531766 | 14:11 |
| dmsimard | I just updated https://storyboard.openstack.org/#!/story/2001445 with two other use cases | 14:13 |
| *** mriedem has joined #openstack-infra | 14:14 | |
| *** esberglu has quit IRC | 14:17 | |
| *** esberglu has joined #openstack-infra | 14:17 | |
| *** smatzek has quit IRC | 14:21 | |
| fungi | gema: we'd probably want to know some more about your goals for donating resources to our pool, but the basic details are at https://docs.openstack.org/infra/system-config/contribute-cloud.html | 14:21 |
| *** Goneri has joined #openstack-infra | 14:21 | |
| fungi | gema: from your linaro cloak, i'm guessing this is revisiting the earlier arm64 cloud discussions? | 14:22 |
| mordred | fungi: extrapolating from the irc nick cloak, I'm guessing maybe ARM? | 14:22 |
| mordred | fungi: jinx | 14:22 |
| *** esberglu has quit IRC | 14:22 | |
| fungi | if so, adding a new (second) architecture is going to be a lot more involved than what we have listed on that page | 14:22 |
| mordred | fungi: yah. having a second architecture will, I'm *sure*, flush out a few assumptions | 14:23 |
| *** eharney has joined #openstack-infra | 14:24 | |
| *** dhill_ has joined #openstack-infra | 14:25 | |
| fungi | gema: and now i see your e-mail to me. will follow up once i'm more awake and caught up | 14:27 |
| *** makowals has quit IRC | 14:27 | |
| *** makowals has joined #openstack-infra | 14:29 | |
| *** esberglu has joined #openstack-infra | 14:32 | |
| gema | fungi: ack, thanks :) | 14:32 |
| *** esberglu_ has joined #openstack-infra | 14:33 | |
| *** esberglu_ is now known as esberglu__ | 14:34 | |
| *** e0ne has joined #openstack-infra | 14:37 | |
| *** esberglu has quit IRC | 14:37 | |
| *** armaan_ has joined #openstack-infra | 14:38 | |
| *** armaan has quit IRC | 14:38 | |
| *** armaan_ has quit IRC | 14:38 | |
| *** armaan has joined #openstack-infra | 14:39 | |
| *** makowals has quit IRC | 14:39 | |
| *** makowals has joined #openstack-infra | 14:39 | |
| *** yamamoto has joined #openstack-infra | 14:42 | |
| *** psachin has quit IRC | 14:45 | |
| *** yamamoto has quit IRC | 14:47 | |
| openstackgerrit | Merged openstack-dev/cookiecutter master: Switch to use stestr in README https://review.openstack.org/530001 | 14:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Add some information about OpenStack IRC usage https://review.openstack.org/531000 | 14:49 |
| mordred | dmsimard: ^^ fixed from your comments | 14:49 |
| dmsimard | +2 | 14:50 |
| openstackgerrit | Frank Kloeker proposed openstack-infra/puppet-zanata master: [WiP] Change Zanata install medium to war files https://review.openstack.org/531809 | 14:54 |
| *** esberglu__ is now known as esberglu | 14:54 | |
| snapiri | Hi, any chance anyone could review: openstack-infra/system-config master: Add openstack-dragonflow channel to statusbot https://review.openstack.org/531720 ? | 14:56 |
| *** yamahata has joined #openstack-infra | 14:56 | |
| dmsimard | snapiri: are you looking for gerrit notifications ? or status updates (such as zuul restarts) ? | 14:56 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul feature/zuulv3: Make Zuul able to start with a broken config https://review.openstack.org/529060 | 14:56 |
| *** afred312 has quit IRC | 14:57 | |
| *** afred312 has joined #openstack-infra | 14:57 | |
| snapiri | dmisimard: the latter please | 14:58 |
| *** armaan has quit IRC | 14:59 | |
| snapiri | we have the first already | 14:59 |
| dmsimard | okay | 15:00 |
| dmsimard | snapiri: +2, was just asking to make sure you were doing the right thing :) | 15:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Fix yaml snippet for sphinx for python3 https://review.openstack.org/531811 | 15:00 |
| *** hongbin has joined #openstack-infra | 15:01 | |
| *** Guest8485 is now known as dansmith | 15:02 | |
| dmsimard | It's a shame that codesearch (hound) is largely unmaintained, there's a feature that would really be useful but there's no one to do it (and merge it) :/ | 15:02 |
| snapiri | dmsimard: Thanks! | 15:02 |
| *** dansmith is now known as Guest62501 | 15:02 | |
| dmsimard | I tried pinging one of the authors on twitter a few weeks/months back but nothing came of it (although he did reply) | 15:03 |
| fungi | hard to find stackers who want to hack on go projects, but maybe we could put out a call for help on hound's behalf? | 15:03 |
| *** sdague has quit IRC | 15:04 | |
| fungi | taron was looking at getting the online config reloading implemented, but their internship ran out before that part got tackled | 15:04 |
| dmsimard | fungi: the upstream is largely unresponsive :( | 15:04 |
| dmsimard | the authors don't work at etsy anymore | 15:04 |
| *** Guest62501 is now known as dansmith | 15:05 | |
| *** pgadiya has quit IRC | 15:07 | |
| *** sbezverk has joined #openstack-infra | 15:09 | |
| *** rlandy_ has joined #openstack-infra | 15:09 | |
| *** rkukura has quit IRC | 15:10 | |
| *** rkukura has joined #openstack-infra | 15:11 | |
| *** gouthamr has joined #openstack-infra | 15:11 | |
| *** smatzek has joined #openstack-infra | 15:15 | |
| *** katkapilatova1 has quit IRC | 15:15 | |
| *** smatzek has quit IRC | 15:15 | |
| *** smatzek has joined #openstack-infra | 15:15 | |
| *** smatzek has quit IRC | 15:15 | |
| *** felipemonteiro has joined #openstack-infra | 15:15 | |
| dmsimard | We'd like to use it for RDO, but the problem is actually a bit silly -- hound only indexes the master branch and some of our repos don't have a master branch so the branch would need to be configurable somehow. There's been a few issues/PRs around that topic but none that quite does that (and work) | 15:16 |
| *** smatzek has joined #openstack-infra | 15:16 | |
| dmsimard | I also trick hound into indexing locally cloned git repos instead of querying github | 15:16 |
| *** Apoorva has joined #openstack-infra | 15:17 | |
| smcginnis | Not really an infra questions, but I figure someone here probably knows. I haven't seen a BoD ballot in the email I would expect it in, the email it possibly could have been sent to, nor the email it shouldn't go to. Have those all been sent out? | 15:17 |
| Adri2000 | hello | 15:19 |
| dmsimard | smcginnis: I got my ballot email sometime this weekend | 15:20 |
| Adri2000 | who should I get in touch with regarding a sudden increase of spam received on the -owner@ alias of an openstack.org mailing-list? | 15:20 |
| dmsimard | Adri2000: there's been a few patches to address that recently, what email is it ? | 15:20 |
| Adri2000 | (like almost 100 - similar - spam per hour) | 15:20 |
| Adri2000 | openstack-fr-owner@ | 15:20 |
| dmsimard | Adri2000: for example http://git.openstack.org/cgit/openstack-infra/system-config/commit/?id=19037f2cbd2c554692160817a5aa9a05eeadfe07 | 15:21 |
| dmsimard | Adri2000: if you send a patch to system-config along those lines, we can merge it quickly | 15:21 |
| *** makowals has quit IRC | 15:22 | |
| Adri2000 | dmsimard: perfect, that should solve the problem indeed, will prepare a patch shortly | 15:22 |
| *** makowals has joined #openstack-infra | 15:22 | |
| *** pcichy has quit IRC | 15:24 | |
| AJaeger | infra-root, kingbird-dashboard new repo creation was merged yesterday but the repo is not fully created, see http://git.openstack.org/cgit/openstack/kingbird-dashboard/ - could somebody check and fix, please? | 15:26 |
| dmsimard | AJaeger: It's missing the gitreview I guess ? | 15:26 |
| AJaeger | dmsimard: and github mirroring. not sure what else | 15:27 |
| dmsimard | hmm it's not even replicated to github or gerrit | 15:27 |
| *** yamamoto has joined #openstack-infra | 15:27 | |
| dmsimard | I can't look *right now*, but iirc that patch landed after fixing the zuul issues from yesterday so it might be related | 15:28 |
| *** katkapilatova1 has joined #openstack-infra | 15:28 | |
| *** e0ne has quit IRC | 15:28 | |
| fungi | it's not even created in gerrit yet: https://review.openstack.org/#/admin/projects/openstack/kingbird-dashboard | 15:28 |
| AJaeger | mordred: did you see my patch to your change https://review.openstack.org/#/c/531430/ - is that fine? | 15:29 |
| chandankumar | dmsimard: AJaeger please have a look at these two reviews https://review.openstack.org/#/c/531678/ and https://review.openstack.org/#/c/531676/ | 15:29 |
| *** eharney has quit IRC | 15:29 | |
| openstackgerrit | Adrien Cunin proposed openstack-infra/system-config master: Treat mail sent to openstack-fr-owner as spam https://review.openstack.org/531820 | 15:30 |
| Adri2000 | dmsimard: ^ | 15:30 |
| *** SamYaple_ has quit IRC | 15:30 | |
| *** SamYaple has joined #openstack-infra | 15:30 | |
| dmsimard | Adri2000: thanks +2 | 15:30 |
| AJaeger | chandankumar: I'm not reviewing right now new pypi additions until I know what to require | 15:30 |
| AJaeger | chandankumar: let's wait until we discussed that in our meeting tomorrow, please | 15:30 |
| *** ramishra has quit IRC | 15:31 | |
| chandankumar | AJaeger: sure, keeping on hold | 15:31 |
| *** yamamoto has quit IRC | 15:32 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool feature/zuulv3: [WIP] webapp: add optional admin endpoint https://review.openstack.org/531718 | 15:33 |
| mordred | AJaeger: +2 thanks | 15:34 |
| *** caphrim007_ has quit IRC | 15:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Publish neutron and horizon to pypi https://review.openstack.org/531822 | 15:34 |
| *** caphrim007 has joined #openstack-infra | 15:35 | |
| mordred | ttx, smcginnis, dhellmann, mnaser: ^^ straw-man to get movement on that topic at least for neutron and horizon | 15:35 |
| AJaeger | config-core, I have written Zuul v3 native jobs for bindep and would love a review on https://review.openstack.org/531470 | 15:35 |
| smcginnis | mordred: Great, I'll take a look shortly. | 15:35 |
| snapiri | dmsimard: sorry for the rudeness, anyone else who can review openstack-infra/system-config master: Add openstack-dragonflow channel to statusbot https://review.openstack.org/531720 ? | 15:35 |
| dmsimard | snapiri: someone will get to it eventually, it's still early where a lot of the core reviewers live | 15:36 |
| snapiri | thanks :) | 15:36 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Update tox env to match zuul PTI https://review.openstack.org/530999 | 15:38 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Add some information about OpenStack IRC usage https://review.openstack.org/531000 | 15:38 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Fix yaml snippet for sphinx for python3 https://review.openstack.org/531811 | 15:38 |
| *** caphrim007 has quit IRC | 15:39 | |
| *** makowals has quit IRC | 15:40 | |
| dmsimard | snapiri: alternatively, all notices are also posted here: https://twitter.com/openstackinfra and here: https://wiki.openstack.org/wiki/Infrastructure_Status | 15:41 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add zuul_work_dir defaults to base-test https://review.openstack.org/531431 | 15:42 |
| *** bobh has joined #openstack-infra | 15:42 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Set defaults for zuul_work_dir and bindep_dir https://review.openstack.org/531432 | 15:42 |
| *** armaan has joined #openstack-infra | 15:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Publish bifrost to PyPI https://review.openstack.org/531824 | 15:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Publish everything but keystone to PyPI https://review.openstack.org/531825 | 15:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: DNM Release keystone to PyPI https://review.openstack.org/531826 | 15:43 |
| AJaeger | mordred: pypi has changed the way projects are registered and https://docs.openstack.org/infra/manual/creators.html#give-openstack-permission-to-publish-releases is not accurate anymore. | 15:44 |
| *** nicolasbock has joined #openstack-infra | 15:45 | |
| snapiri | dmsimard: thanks! this is eye-opening... | 15:45 |
| AJaeger | I put this on the agenda for tomorrow - but currently do not approve new pypi publish jobs since I do not know how to reserve the names | 15:45 |
| fungi | did something change with our our puppeting of servers on friday? or late thursday? last puppet run on review.openstack.org was 23:33 on january 4 | 15:45 |
| fungi | and it's not in the emergency file | 15:46 |
| dhellmann | AJaeger : I think someone needs to run "setup.py register" and then adjust the resulting project's permissions | 15:46 |
| dhellmann | AJaeger : oh, maybe you have to actually upload a release by hand once | 15:47 |
| fungi | dhellmann: i _think_ twine is supposed to do that automagically if you're uploading a not-yet-existent project | 15:47 |
| fungi | and that setup.py register is deprecated now | 15:47 |
| dhellmann | fungi : yeah. It didn't used to, but I think you're right based on the instructions on pypi.python.org | 15:47 |
| dhellmann | I think the change is you can't register a name without actually uploading a package | 15:48 |
| *** gcb has quit IRC | 15:48 | |
| fungi | problem with requiring someone to manually register the project now that doing so requires at least a stub of a package is that if we're reviewing for it we can't add new projects with pypi publication jobs, and have to wait for the pypi jobs to get added later | 15:48 |
| fungi | of course, the down-side to just relying on the first twine run to create the project from our release jobs (aside from the risk a squatter will grab it in the meantime) is that only our openstackci account will be an index owner | 15:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Publish everything but keystone to PyPI https://review.openstack.org/531825 | 15:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: DNM Release keystone to PyPI https://review.openstack.org/531826 | 15:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove release-openstack-python-without-pypi https://review.openstack.org/531829 | 15:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Remove release-openstack-server template https://review.openstack.org/531830 | 15:50 |
| mordred | fungi, dhellmann, AJaeger: yah- so - I just did a twine upload of bifrost to get it registered, but I grabbed the 4.0.0 release from tarballs.o.o ... | 15:51 |
| dhellmann | fungi : we could build a little tool for creating a mostly empty sdist for a given name and publish 0.0.0 releases | 15:52 |
| fungi | mordred: thanks for confirming it works as described | 15:52 |
| mordred | in the past, to work around register not being a thing anymore, I've just done a 'python setup.py sdist upload' on a random commit in a project, then have gone and deleted the release | 15:52 |
| mordred | dhellmann: ++ | 15:52 |
| mordred | dhellmann: I was just thinking that same thing | 15:52 |
| dhellmann | oh, that sounds like a good process, too | 15:52 |
| dhellmann | although it has the problem fungi pointed out | 15:52 |
| *** ram5391 has joined #openstack-infra | 15:52 | |
| fungi | dhellmann: we could, but i think that likely flies in the face of the reasons pypi has started requiring you to register by uploading a package, and wouldn't want to look like we're producing a reusable tool to skirt that | 15:52 |
| *** yamamoto has joined #openstack-infra | 15:53 | |
| *** yamamoto has quit IRC | 15:53 | |
| fungi | basically, i wouldn't want one of our release tools to become the thing that package name squatters use to mass register pypi projects | 15:53 |
| mordred | fungi: ++ | 15:54 |
| dhellmann | sure | 15:54 |
| mordred | fungi: we have things that are only owned by openstackci already | 15:54 |
| dhellmann | perhaps it's not a tool a person runs. | 15:54 |
| fungi | but yeah, perhaps we should encourage projects to "release early (and often)" by tagging a release as soon as the packaging boilerplate is merged | 15:54 |
| mordred | maybe we just let the release process be the thing | 15:54 |
| mordred | fungi: ++ | 15:54 |
| *** kiennt26 has quit IRC | 15:54 | |
| mordred | fungi: we publish alphas and betas now, right? | 15:54 |
| *** felipemonteiro has quit IRC | 15:54 | |
| fungi | yeah, tag 0.0.0.0a1 | 15:55 |
| mordred | a person could just tag an 0.0.0a1 | 15:55 |
| mordred | jinx | 15:55 |
| dhellmann | do we support uploading those? | 15:55 |
| *** felipemonteiro has joined #openstack-infra | 15:55 | |
| fungi | we do | 15:55 |
| mordred | we do - pip ignores them by default | 15:55 |
| dhellmann | ok, I thought we set the job to only run for full releases | 15:55 |
| fungi | we used to only upload wheels for prerelease tag strings, to work around (now very old) pip releases which didn't know how to distinguish pep 440 prereleases from releases | 15:56 |
| mordred | you have to explicitly opt-in to a pre-release version number in a pip requirement for it to find it | 15:56 |
| *** katkapilatova1 has quit IRC | 15:56 | |
| mordred | dhellmann, fungi: double-checked - publish-to-pypi does upload pre-releases | 15:56 |
| dhellmann | ok, cool | 15:56 |
| fungi | as of a while ago (maybe the past year?) we switched to just do both wheels and sdists for prereleases too | 15:56 |
| dhellmann | so yeah, an alpha release is a way to go | 15:56 |
| mordred | that said - the release team are the ones who have tagging acls these days | 15:56 |
| fungi | mordred: also, to be clear, it would be 0.0.0.0a1 not 0.0.0a1 (pbr would attempt to normalize the latter to the former and then our upload job wouldn't find the corresponding artifacts) | 15:57 |
| mordred | we maybe we need to have a job that runs when a new project is added to $something that submits the 0.0.0a1 tag? | 15:57 |
| dhellmann | the releases repo validation stuff may have to change to support the very first release of a library being an alpha, because we don't allow pre-release versioning for all deliverable types right now | 15:57 |
| *** SamYaple has quit IRC | 15:57 | |
| *** slaweq has quit IRC | 15:57 | |
| dhellmann | smcginnis : ^^ | 15:57 |
| *** SamYaple has joined #openstack-infra | 15:57 | |
| dhellmann | mordred : proposing the tag needs to be manual because we have to wait for the repo to contain enough stuff to actually be packagable | 15:58 |
| mordred | dhellmann: ah. good point | 15:58 |
| *** slaweq has joined #openstack-infra | 15:58 | |
| *** SamYaple has quit IRC | 15:58 | |
| *** SamYaple has joined #openstack-infra | 15:58 | |
| mordred | dhellmann: does the release team have permissions to propose tags by hand? | 15:58 |
| fungi | mordred: the catch-22 is that projects might add the pypi jobs before they've nailed down successful sdist/wheel generation | 15:58 |
| *** e0ne has joined #openstack-infra | 15:58 | |
| dhellmann | so, create the repo in project-config, populate it the normal way, propose a release as early as possible to reserve the name | 15:58 |
| dhellmann | use an alpha version if that release is not actually consumable | 15:59 |
| mordred | dhellmann: ++ | 15:59 |
| fungi | mordred: they do, last i checked, yes | 15:59 |
| *** xarses_ has joined #openstack-infra | 15:59 | |
| fungi | mordred: the way the release tooling works is it uses a gerrit account in the release managers group, which is how it gets sufficient permissions to push tags | 15:59 |
| fungi | so actual human release managers can in theory run the same commands | 16:00 |
| mordred | fungi, dhellmann: making an infra-manual patch with the above in it | 16:01 |
| AJaeger | mordred: could you document this on the creators page, please? | 16:02 |
| *** slaweq has quit IRC | 16:02 | |
| *** ijw has joined #openstack-infra | 16:03 | |
| fungi | that's where i assumed he would add it | 16:03 |
| AJaeger | fungi, mordred : sorry, didn't read everything when I wrote this - sorry in a meeting | 16:05 |
| AJaeger | chandankumar: see above discussion, did you reserve the entreis already on pypi for all the tempest plugins | 16:06 |
| dmsimard | zuul-merger doesn't appear to be running on the ze nodes again... but grafana is not reporting any anomalies in merger counts | 16:06 |
| dmsimard | wait, is merger supposed to be running on the executors ? I thought it was | 16:07 |
| dmsimard | 10x ze + 8 zm, that's the 18 mergers we're supposed to have | 16:07 |
| fungi | dmsimard: last i remember, yes | 16:09 |
| dmsimard | there's nothing in /var/lib/zuul/git so.. I guess I'm confused | 16:09 |
| dmsimard | I was actually troubleshooting why there was no merger-logging.conf on the zuul-executors but the puppet zuul merger class doesn't seem applied to the zuul-executors so far as I can tell | 16:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove centos-release-openstack-ocata https://review.openstack.org/531837 | 16:10 |
| clarkb | dmsimard: that is correct | 16:10 |
| pabelanger | ianw: clarkb: okay, I think we can just remove centos RDO ocata repos now^. I don't believe we need anything from there for our git servers. | 16:11 |
| dmsimard | clarkb: what is correct ? :) | 16:11 |
| clarkb | dmsimard: that we don't apply merger config to the executors | 16:11 |
| dmsimard | clarkb: but they *are* mergers, right ? | 16:11 |
| dmsimard | I recall discussing with corvus that ze's relied on themselves for merging (or something to that effect) | 16:12 |
| clarkb | dmsimard: only because executors run an implied merger aiui | 16:12 |
| fungi | the executors were running local mergers at one point | 16:12 |
| pabelanger | zuul-executor will launch their own process for mergers | 16:12 |
| fungi | ahh, that makes sense | 16:12 |
| pabelanger | they don't need zuul-merger class | 16:12 |
| clarkb | dmsimard: its built in requirement of the executor so it handles that itself | 16:12 |
| dmsimard | oh, so the zuul-merger bits would be logged in the executor logfile ? | 16:12 |
| clarkb | dmsimard: yes | 16:12 |
| pabelanger | yup | 16:12 |
| dmsimard | ok, that was confusing (hard monday this morning) | 16:13 |
| dmsimard | I didn't know executors ran an implied merger | 16:13 |
| dmsimard | without the actual zuul-merger process running | 16:13 |
| fungi | okay, so it looks like the centos upgrades late last week likely broke puppet, which is why it never gets around to running against review.o.o | 16:16 |
| fungi | i'm digging into logs on the git servers now to confirm | 16:16 |
| clarkb | fungi: oh you know what I may not have accepted new ssh host key for git.o.o | 16:16 |
| clarkb | but it shouldn't be part of the stuff that blocks review.o.o only git0*.openstack.org should be | 16:16 |
| fungi | 2018-01-08 16:02:33,806 p=1490 u=root | fatal: [git05.openstack.org]: FAILED! => {"changed": false, "failed": true, "msg": "/usr/bin/timeout -s 9 30m /usr/bin/puppet apply /opt/system-config/production/manifests/site.pp --logdest syslog --environment 'production' --no-noop --detailed-exitcodes failed with return code: 6", "rc": 6, "stderr": "", "stdout": "", "stdout_lines": []} | 16:16 |
| fungi | is what we see on the puppetmaster end | 16:17 |
| AJaeger | infra-root, can I ask again for help in finishing creation of kingbird-dashboard? fungi noticed that puppet did not run on review.o.o since January 4th - so that might be the first thing to look at, please | 16:17 |
| clarkb | ah ya hopefully syslog has more info | 16:17 |
| dmsimard | git05 was the server that wasn't updating properly iirc | 16:17 |
| clarkb | dmsimard: ya it was | 16:17 |
| fungi | AJaeger: that's what i've been looking into for the past half hour | 16:17 |
| AJaeger | infra-root, I see fungi is commenting already above | 16:17 |
| pabelanger | fungi: python-six looks to be the issue | 16:17 |
| AJaeger | fungi, thanks! Wasn't clear to me. | 16:17 |
| fungi | pabelanger: yeah, trying to upgrade jeepyb | 16:18 |
| fungi | Jan 7 08:32:23 git01 puppet-user[10116]: #011python-six is needed by (installed) libstoragemgmt-python-1.4.0-5.el7_4.noarch | 16:19 |
| sshnaidm | pabelanger, can we send our custom metrics from multinode jobs in tripleo to http://graphite.openstack.org/ ? | 16:19 |
| fungi | pabelanger: guessing something changed with that package when we updated to latest centos point release? | 16:19 |
| AJaeger | smcginnis: please see my note on https://review.openstack.org/#/c/531440 - and review your backports to properly checkout the correct branch | 16:20 |
| pabelanger | sshnaidm: not currently, the ports are firewalled. I know we have talked about a graphite publisher before, but we likely need to discuss it again | 16:21 |
| smcginnis | AJaeger: Yep, I've been updating outstanding patches this morning and just getting to the ones that had already merged. | 16:21 |
| pabelanger | fungi: yah, I'm trying to see what is going on now myself | 16:21 |
| AJaeger | smcginnis: cool, thanks | 16:21 |
| *** armax has joined #openstack-infra | 16:21 | |
| smcginnis | AJaeger: Thanks for the heads up! | 16:21 |
| fungi | pabelanger: i'll let you continue on that front. you're likely going to be faster than me at troubleshooting rpm installation issues | 16:21 |
| dmsimard | sshnaidm: graphite.o.o access is restricted by a firewall right now | 16:22 |
| corvus | fungi, clarkb, dmsimard: zuul executors have internal mergers. it doesn't run as a second process, it just implements both functions. | 16:22 |
| dmsimard | corvus: yeah, TIL, thanks | 16:22 |
| sshnaidm | pabelanger, yeah, you wanted us to use graphite.o.o for multinode jobs.. | 16:22 |
| pabelanger | fungi: http://git.openstack.org/cgit/openstack-infra/puppet-jeepyb/tree/manifests/init.pp#n60 | 16:22 |
| pabelanger | fungi: seems we are forcing an removal, but other library needs it now | 16:22 |
| dmsimard | sshnaidm: if there would be a way for jobs to send data to graphite.o.o, it would probably need to be a task running on the local executor at the end of a job or something like that -- not unlike how we send data to logstash and openstack-health | 16:23 |
| dmsimard | sshnaidm: would you like to come discuss that at tomorrow's openstack-infra meeting ? | 16:23 |
| pabelanger | sshnaidm: right, I wanted to see how we can remove the private infrastructure tripleo was using. If you want to add the topic to our weekly meeting tomorrow, we can start to go over the requirements | 16:23 |
| sshnaidm | dmsimard, I think clouds IP ranges are well known, why not to allow them in firewall? | 16:23 |
| sshnaidm | pabelanger, dmsimard ok, let's raise it in meeting | 16:24 |
| dmsimard | agenda is here: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting | 16:24 |
| *** e0ne has quit IRC | 16:24 | |
| pabelanger | fungi: seems https://review.openstack.org/505785/ from mordred might have broken things after point release | 16:25 |
| dmsimard | need a +3 to merge https://review.openstack.org/#/c/531820/ to reduce spam to one of our ML owners | 16:25 |
| clarkb | sshnaidm: because they aren't well known in every case and in many cases are reused by other tenants | 16:25 |
| sshnaidm | clarkb, hmm, I see.. | 16:26 |
| pabelanger | libstoragemgmt, anybody konw what that is for? | 16:27 |
| *** krtaylor_ has joined #openstack-infra | 16:27 | |
| *** krtaylor has quit IRC | 16:27 | |
| *** felipemonteiro has quit IRC | 16:27 | |
| corvus | sshnaidm: see also http://specs.openstack.org/openstack/qa-specs/specs/devstack/counter-inspection.html for some partially completed prior work on the subject | 16:27 |
| sshnaidm | clarkb, are zuul executors IP ranges not known too? | 16:27 |
| dmsimard | infra-root: yesterday all zuul-executors and zuul-mergers stopped due to a msgpack issue (see status log), while I reinstalled msgpack==0.5.0, there's still an outlying msgpack-python package we probably need to clean up | 16:27 |
| dmsimard | sshnaidm: zuul executors IP are whitelisted on a node basis | 16:27 |
| corvus | dmsimard: why did they stop? | 16:28 |
| clarkb | corvus: my hunch was that msgpack 0.5.0 release was initially bad (broken wheels maybe) but have not been able toconfirm that | 16:28 |
| fungi | sshnaidm: i really don't think we have a good way to determine the possible ip allocations used by all our providers, and even if we did that would be a pain to keep on top of what with the current state of ipv4 fragmentation and resale markets | 16:29 |
| *** ijw has quit IRC | 16:29 | |
| clarkb | I can't find anywhere where pypi says I deleted this thing | 16:29 |
| dmsimard | corvus: I'm not 100% positive on why they stopped, my understanding is that there was an update of several python packages (amongst which was msgpack) and msgpack did not install properly .. leading zuul-executor and zuul-merger to fail starting | 16:29 |
| corvus | clarkb: right, but we don't have any puppet mechanism to actually stop the process | 16:29 |
| fungi | sshnaidm: but also, that would allow anyone in any of those clouds to pollute our stats if they wanted, rather than being limited just to jobs we run | 16:29 |
| dmsimard | corvus: so it's probably not so much a stop than a failed restart (or reload) or something else | 16:29 |
| corvus | i understand why they didn't start -- i just don't understand why they stopped | 16:29 |
| sshnaidm | fungi, I see.. I thought it's something constant | 16:30 |
| *** jamesmcarthur has joined #openstack-infra | 16:30 | |
| dmsimard | corvus: amongst the updated packages, there was zuul itself, github3 and a few others.. does that cause a reload/restart of some sort to load the new code ? | 16:30 |
| *** udesale has quit IRC | 16:30 | |
| corvus | sshnaidm, fungi: the spec i linked you has suggestions to address that issue | 16:30 |
| fungi | sshnaidm: i agree that having the executors emit stats is the way to go for this. we know their addresses and can whitelist them easily | 16:30 |
| dmsimard | corvus: otherwise, the rename of the msgpack module (from msgpack-python to msgpack) probably confused python (pip was confused itself) | 16:31 |
| corvus | dmsimard: nope. no component of zuul ever automatically starts or restarts. | 16:31 |
| corvus | or stops | 16:32 |
| dmsimard | pip thought both msgpack and msgpack-python were installed, it showed up in pip3 freeze but msgpack was not actually installed (only had distinfo folders in /usr/local/lib/python3.5) | 16:32 |
| *** slaweq has joined #openstack-infra | 16:32 | |
| fungi | dmsimard: what was the approximate date/time when you noticed they weren't running? | 16:32 |
| dmsimard | fungi: it's super obvious looking at grafana | 16:32 |
| dmsimard | http://grafana.openstack.org/dashboard/db/zuul-status see yesterday | 16:33 |
| *** jistr is now known as jistr|off|mtg | 16:33 | |
| dmsimard | dead 19:00 UTC according to graphs | 16:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-jeepyb master: Revert "Remove python-six on centos" https://review.openstack.org/531851 | 16:34 |
| fungi | dmsimard: and this roughly corresponds with when the unattended-upgrades runs broke? | 16:34 |
| dmsimard | also, there should probably be a "job count" governor.. since ze01 started first, it picked up 200 jobs by itself and loaded up to 150+ | 16:35 |
| corvus | Jan 7 18:26:10 ze01 kernel: [8895847.097575] traps: zuul-executor[10850] general protection ip:513ef4 sp:7f2525ff9068 error:0 in python3.5[400000+3a9000] | 16:35 |
| corvus | dmsimard: not yet. ram first. then rate limit. | 16:35 |
| *** d0ugal has quit IRC | 16:35 | |
| pabelanger | fungi: clarkb: mordred: ^ should get puppet working again on git servers, is a revert of python-six removal. I think moving forward, we should discuss our DIB control plane effort (tomorrow) to finally solve cloud-init issues | 16:35 |
| dmsimard | fungi: the unattended-upgrades log I sent was probably unrelated but I stumbled on it when I was troubleshooting the issue | 16:35 |
| dmsimard | it was isolated to ze10 only afaik, which had crashed last week so it probably died in the middle of a package upgrade | 16:36 |
| corvus | Jan 7 18:56:03 ze02 kernel: [12449956.548285] traps: zuul-executor[27272] general protection ip:513ef4 sp:7f05aeffJJaJaJan 7 18:56:03 ze02 kernel: [124499Jan 7 18:5Jan 7Jan JaJanJJaJJan 7 18:56:03 ze02 kernel: [12449956.548285] traps: zuul-executor[27272] general protection ip:513ef4 sp:7f05aeff9918 error:0 in python3.5[400000+3a9000] | 16:36 |
| fungi | yeah, the gpf corvus pasted above was some 36 hours after the last unattended-upgrades run did anything with packages on ze01 according to dpkg.log, so likely unrelated | 16:36 |
| *** slaweq has quit IRC | 16:37 | |
| corvus | wow that was som weird pasting | 16:37 |
| * fungi blames tabs | 16:37 | |
| clarkb | pabelanger: fwiw there shouldn't be any cloud-init issues currently as we explicitly remove it as part of launch node execution | 16:37 |
| clarkb | pabelanger: its unrelated to the hostname stuff too. We explicitly set the hostname to the hostname we don't want | 16:37 |
| dmsimard | corvus: there's probably no mistaking that timestamp, it correlates to when everything died | 16:38 |
| *** kmalloc has joined #openstack-infra | 16:38 | |
| pabelanger | clarkb: are you sure? ze09 and ze10 have cloud-init installed | 16:39 |
| dmsimard | something that I noticed when troubleshooting is that while this was not our problem -- do we really want to point to the develop branch of github3 ? There's a lot of commits there on a regular basis | 16:39 |
| *** gyee has joined #openstack-infra | 16:39 | |
| dmsimard | I'll copy/pasta that in #zuul | 16:39 |
| clarkb | pabelanger: ya its the step before we set the hostname in https://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/roles/set_hostname/tasks/main.yml#n2 | 16:39 |
| corvus | dmsimard, fungi: http://paste.openstack.org/show/640975/ | 16:39 |
| clarkb | pabelanger: we do remove cloud-init then set the hostname to the value we don't want | 16:40 |
| fungi | comparing the gpf from ze01 with puppet events, i don't see any tight correlations | 16:40 |
| pabelanger | okay, something must be pulling it back in | 16:40 |
| pabelanger | 2018-01-05 23:54:10,759 - cc_update_hostname.py[DEBUG]: Updating hostname to ze09 (ze09) | 16:40 |
| *** kjackal has quit IRC | 16:41 | |
| pabelanger | that is from /var/log/cloud-init.log | 16:41 |
| fungi | pabelanger: it boots with cloud-init initially because the images we're using (provided by rax) have it installed. then we uninstall it | 16:41 |
| clarkb | right so cloud init comes up does whatever it does, then we should uninstall it and set the hostname to what we actually want | 16:42 |
| clarkb | but we instead set it to ze09 too | 16:42 |
| dmsimard | brb more shovelling (I love snow but this is a lot of snow now...) and lunch.. | 16:42 |
| clarkb | because of the split at https://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/roles/set_hostname/tasks/main.yml#n12 | 16:42 |
| *** jkilpatr has quit IRC | 16:44 | |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/531430/ , please? - we need this to fix publishing of openstack-manuals and friends | 16:44 |
| *** jpena is now known as jpena|brb | 16:44 | |
| pabelanger | fungi: clarkb: but dpkg -l cloud-init returns a result, doesn't that mean it is still installed? This is ze09 | 16:46 |
| pabelanger | or did we just remove it and haven't launched a new server? | 16:46 |
| clarkb | pabelanger: right its entirely possible that that ansible isn't doing what we want. My point is more even if cloud-init was completely gone we then set the hostname to what we don't want. So no amount of changing cloud-init will fix this | 16:47 |
| *** david-lyle has quit IRC | 16:47 | |
| clarkb | its also possible this isn't running for some reason? though the same part of launch node that executes puppet is what executes the hostname update | 16:48 |
| *** david-lyle has joined #openstack-infra | 16:48 | |
| *** pcaruana has quit IRC | 16:48 | |
| *** yamahata has quit IRC | 16:48 | |
| *** fultonj has quit IRC | 16:48 | |
| pabelanger | okay, so 2 things. Figure out why cloud-init is still installed, 2) figure how what hostname should be set ti | 16:49 |
| fungi | pabelanger: when you `dpkg -l cloud-init` is shows "rc" as the status | 16:50 |
| fungi | that means "removed, configuration still present" | 16:50 |
| fungi | which is what you get if you remove without --purge | 16:50 |
| pabelanger | fungi: ah, okay. that makes sense | 16:51 |
| *** ram5391 has quit IRC | 16:52 | |
| pabelanger | fungi: so, we can update ansible to purge that | 16:52 |
| *** david-lyle has quit IRC | 16:52 | |
| clarkb | we may not want it to purge | 16:52 |
| clarkb | so that if we do reinstall cloud-init it doesn't act like its on a fresh cloud init | 16:52 |
| fungi | if you want, it's not hurting anything leaving the conffile behind though afaik | 16:52 |
| pabelanger | clarkb: good point | 16:52 |
| *** e0ne has joined #openstack-infra | 16:53 | |
| *** yamamoto has joined #openstack-infra | 16:53 | |
| *** fultonj has joined #openstack-infra | 16:56 | |
| *** ram5391 has joined #openstack-infra | 16:56 | |
| *** caphrim007 has joined #openstack-infra | 16:57 | |
| *** d0ugal has joined #openstack-infra | 16:59 | |
| *** jkilpatr has joined #openstack-infra | 17:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Update the PyPI registration docs https://review.openstack.org/531859 | 17:00 |
| *** slaweq has joined #openstack-infra | 17:00 | |
| mordred | AJaeger, dhellmann, fungi: ^^ there's a stab | 17:00 |
| *** iyamahat has joined #openstack-infra | 17:01 | |
| *** yamamoto has quit IRC | 17:02 | |
| clarkb | pabelanger: fungi mordred corvus probably the first thing we need to do is decide if we want hostnames to be ze09 or ze09.openstack.org. Then modify the launch script's set hostname role to use foo.openstack.org if that is what we want | 17:04 |
| clarkb | I know SpamapS was running into similar things with glean recently but I think he wants something slight different which is use the name that nova provides by combining hostname and domainname into an fqdn. Which for us would be foo.novalocal because we don't have the integrated designate stuff | 17:05 |
| mordred | clarkb: I mean, I for one would like 'hostname' to return 'ze09' and 'hostname --fqdn' to return 'ze09.openstack.org' | 17:05 |
| clarkb | mordred: thats what we currently have and I think you implemented it that way :) | 17:05 |
| mordred | clarkb: the hostname provided by nova is completely broken | 17:05 |
| mordred | although I would LOVE IT if it wasn't | 17:05 |
| clarkb | mordred: yes I know, but you need to convince spamaps it isn't | 17:06 |
| mordred | well - it's completely broken for public cloud | 17:06 |
| mordred | I could see it being totally correct for a private cloud - since the problem with it is that .novalocal is a cloud-admin setting | 17:06 |
| mnaser | ^ it relies on dhcp_domain which is a used by nova-network and usually takes <server-name>.<CONF.dhcp_domain> | 17:06 |
| mnaser | but if you're using a private network (with a recent release of openstack), you can set the dhcp domain for the network | 17:07 |
| clarkb | mnaser: oh thats good to know (though won't fix all cases) | 17:07 |
| *** eumel8 has quit IRC | 17:07 | |
| clarkb | mordred: I thik the confusion specific to us was that we always had ze09.openstack.org as the hostname | 17:07 |
| clarkb | mordred: then we change it to be what you describe (and cloud-init also possibly changed things on its endtoo) | 17:08 |
| *** bhavik1 has joined #openstack-infra | 17:10 | |
| *** bhavik1 has quit IRC | 17:11 | |
| *** slaweq has quit IRC | 17:11 | |
| *** d0ugal has quit IRC | 17:11 | |
| fungi | yeah, my understanding of wanting glean to support the nova-supplied domain is that in cases where it's configurable by the cloud user for their network, it's convenient to not have to hard-code domains into configuration management | 17:13 |
| fungi | though that still gets back to... if your instance has multiple network interfaces on different networks, which one sets the primary domain name? the one with the default route? | 17:14 |
| mordred | fungi: totally - and I am rather sympathetic to that desire | 17:14 |
| AJaeger | mordred: a few nits on your change. But a more fundamental question: | 17:15 |
| mordred | fungi: this is amongst the reasons that I think that not having designate integrated fully is crazypants and that we should consider designate an essential part of our IaaS story | 17:15 |
| *** pcichy has joined #openstack-infra | 17:15 | |
| AJaeger | What is the workflow: 1) add publish-to-pypi to project-config; 2) "Prepare and Publish an Initial Release for Registration" - or the other way? | 17:15 |
| *** dsariel has quit IRC | 17:16 | |
| mordred | AJaeger: yah - I think that actually we should do $something to ensure that we reserve the name on PyPI before creating the project in the system otherwise | 17:16 |
| AJaeger | mordred: right I check before publish-to-pypi is set that the project exists but your description uses git push etc. so assume publish-to-pypi | 17:16 |
| *** jascott1 has quit IRC | 17:17 | |
| mordred | AJaeger: yah. also - if one adds the project to the system, then goes to register on pypi and only then realizes the name is taken, now we've got a renaming problem | 17:17 |
| *** jascott1 has joined #openstack-infra | 17:17 | |
| AJaeger | mordred: then your description needs to add the two stesp "$something" and "add publish-to-pypi" | 17:17 |
| *** ijw has joined #openstack-infra | 17:17 | |
| mordred | yah. I'm starting to think I need to rewrite it ... here's an idea ... | 17:17 |
| corvus | AJaeger, mordred, dhellmann: sorry i missed the conversation earlier (well, only a little sorry -- i was asleep) -- but i would feel really uncomfortable going through the whole project creation process without having registered the name in pypi. indeed, the most recent projects i've created, i did just upload an empty release in order to register the name. | 17:17 |
| AJaeger | mordred: we always have - if people do this later. But yes, we should register directly | 17:18 |
| AJaeger | corvus: I think both mordred and myself are in agreement here. That's what I require today and I suggest to keep that. | 17:18 |
| mordred | AJaeger, fungi, dhellmann, corvus: what if we add 2 fields to gerrit/projects.yaml - python: <bool, defaults to true> - python-package: <str, defaults to name> ... | 17:18 |
| corvus | AJaeger, mordred, dhellmann: if you think creating the repos then uploading is the right thing to recommend, i won't object -- but i feel a little weird about it because i don't think i'd ever be comfortable following that process myself. | 17:18 |
| *** jamesmcarthur has quit IRC | 17:19 | |
| mordred | and then we add a check job for projects.yaml that verifies for python projects if the project name is either available or if it is already registered if the openstackci user has correct permissions | 17:19 |
| mordred | and a post job that creates the project with a null release | 17:19 |
| mordred | it's a bit more automation work on our part, but it protects us from adding pytohn projects we can't publish | 17:20 |
| corvus | mordred: that's not bad.... maybe also have jeepyb check that happened before creating gerrit projects, etc? | 17:20 |
| AJaeger | mordred: would work for me - and remove a manual step for setup | 17:20 |
| corvus | (so if the post job failed, we don't proceed until someone manually fixes) | 17:20 |
| *** yamahata has joined #openstack-infra | 17:20 | |
| mordred | corvus: ++ | 17:21 |
| *** jamesmcarthur has joined #openstack-infra | 17:21 | |
| *** trown is now known as trown|lunch | 17:22 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies https://review.openstack.org/530806 | 17:23 |
| *** jpena|brb is now known as jpena | 17:24 | |
| openstackgerrit | Adam Spiers proposed openstack-infra/infra-manual master: promote IRC channel creation info into its own section https://review.openstack.org/531133 | 17:27 |
| *** ram5391 has quit IRC | 17:27 | |
| *** kjackal has joined #openstack-infra | 17:28 | |
| *** Apoorva has quit IRC | 17:29 | |
| clarkb | corvus your feedback on mordred's ideal hostname situation would be nice as I think you aren't entirely.happy woth the current situation (which I think achieves mordred's ideal) | 17:31 |
| dmsimard | Last Ubuntu update was yesterday about candidate kernels being spun up.. https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown | 17:32 |
| dmsimard | They were really caught off guard and they were at the last minute already.. | 17:32 |
| *** jpich has quit IRC | 17:33 | |
| *** david-lyle has joined #openstack-infra | 17:34 | |
| *** jascott1 has quit IRC | 17:34 | |
| *** eharney has joined #openstack-infra | 17:37 | |
| *** gema has quit IRC | 17:38 | |
| corvus | clarkb: the thing i'm unhappy about is that every time we spin up a machine, we have a 50% chance it will get one kind of hostname or the other, and we spend a long time going through the whole process to try to figure out why "this server" did it "this way" instead of "that way" like "that other server". if we can get a consistent behavior, i'll be happy. can you or mordred reply to that email thread with what you think the final ... | 17:39 |
| corvus | ... state should be, and what we have to do to get there? | 17:39 |
| *** yamamoto has joined #openstack-infra | 17:43 | |
| *** yamamoto has quit IRC | 17:43 | |
| *** felipemonteiro has joined #openstack-infra | 17:45 | |
| *** david-lyle has quit IRC | 17:47 | |
| *** jascott1 has joined #openstack-infra | 17:48 | |
| clarkb | I think it is consistent now | 17:48 |
| SpamapS | clarkb: You all convinced me that I was wrong about wanting an fqdn in hostname. :) | 17:48 |
| clarkb | which is what set hostname does during launch node | 17:49 |
| clarkb | I think without that it would depend on cloud init/nova agent | 17:50 |
| corvus | clarkb: didn't you say set_hostname does a split? | 17:50 |
| *** ram5391 has joined #openstack-infra | 17:51 | |
| corvus | clarkb: oh, disregard that, sorry | 17:51 |
| clarkb | corvus: yes so remove the .openstack.org from hodtname | 17:51 |
| corvus | clarkb: so why are all the other executors ze.o.o ? | 17:51 |
| corvus | did we change something since we launched them? it wasn't that long ago | 17:52 |
| *** lucasagomes is now known as lucas-afk | 17:52 | |
| *** ram5391 has quit IRC | 17:53 | |
| clarkb | yes set hostname is new | 17:55 |
| dhellmann | mordred , corvus , AJaeger : automatically creating things on pypi when we create the repo seems ok, but there's a chance the name used in setup.cfg won't match the name used in projects.yaml so we need to either enforce that or allow for it in other processes later | 17:55 |
| clarkb | as is the cloud init uninstall | 17:55 |
| *** e0ne has quit IRC | 17:55 | |
| corvus | dhellmann: i think mordred suggested we have a field in projects.yaml to 'override' the normal project name, so folks can set it to what they know they will put in setup.cfg later | 17:56 |
| *** Apoorva has joined #openstack-infra | 17:57 | |
| dhellmann | corvus : sure. I'm anticipating a source of mistakes with manually replicating that value. | 17:57 |
| dhellmann | I don't know how likely that is to actually come up | 17:57 |
| corvus | dhellmann: ah i see -- if someone forgets to set it there, we could end up with a spurious registration... i guess we could just delete it in that case | 17:57 |
| dhellmann | yeah | 17:58 |
| *** Goneri has quit IRC | 17:58 | |
| corvus | clarkb: set_hostname is almost a year old... the executors are only a few months | 17:58 |
| dhellmann | to fungi's earlier point if the purpose of this api change on pypi was to avoid having people park names, maybe we're better off just asking teams to prepare real releases quickly | 17:58 |
| dhellmann | even if the "real" release is just an alpha with some packaging metadata | 17:59 |
| *** greghayn1 is now known as greghaynes | 17:59 | |
| *** derekh has quit IRC | 18:01 | |
| corvus | dhellmann: that sounds reasonable, but i don't think i could convince myself to do that. i would never feel comfortable doing all the work to create a new project without knowing i had secured the name. the cost to secure the name is trivial; the cost of a rename is very high. | 18:01 |
| dhellmann | I'm probably in the same camp. In fact, I'm pretty sure I've done exactly that -- publish from a local set of files before things were even on github | 18:02 |
| *** Goneri has joined #openstack-infra | 18:02 | |
| corvus | dhellmann: yeah, same here, but s/github/gerrit/ :) | 18:03 |
| dhellmann | well, yeah, not everything I do is openstack or self-hosted | 18:03 |
| corvus | pretty sure i did that for the most recent projects i created | 18:03 |
| mordred | corvus, dhellmann: npm recently added namespaces to their uploads - which are signified with an @ sign -so like angular-ui-router (ui-router project for angular but not by angular) became @uirouter/angularjs ... I wonder if we should propose to donald something similar for warehouse | 18:04 |
| *** slaweq has joined #openstack-infra | 18:05 | |
| mordred | it would be nice to be able to publish to @openstack/nova and @openstack/keystone (or even openstack/nova and openstack/keystone - no clue why the @ was added) | 18:05 |
| *** ralonsoh has quit IRC | 18:05 | |
| dhellmann | "Namespaces are one honking great idea — let's do more of those!" | 18:05 |
| *** efoley has quit IRC | 18:06 | |
| dhellmann | mordred : so the thing with the / could be distinguised from a filename? | 18:06 |
| mordred | dhellmann: ah - yah - probably so | 18:06 |
| fungi | corvus: to be fair, the cost to secure the name on pypi is trivial for someone who's at least minimally versed in the python packaging ecosystem... we have people wanting to create new python projects who seemed to have trouble even following our earlier instructions for registering packages (back when that still worked) | 18:06 |
| fungi | now, whether or not that barrier to entry is a good thing is debatable | 18:07 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config master: Update the list of ironic jobs in nova's experimental queue https://review.openstack.org/531868 | 18:07 |
| * fungi feels like letting some things remain hard for people who won't take the time to gain familiarity with the ecosystem is entirely justified, but often gets accused of not taking "developer friendliness" into account | 18:08 | |
| openstackgerrit | Gage Hugo proposed openstack-infra/project-config master: Remove legacy jobs from Keystone https://review.openstack.org/524232 | 18:12 |
| corvus | fungi: yes. i'm not opposed to merging mordred's change as-is if we think that's the best way (i said so above). i would personally have trouble +2ing it because i would be recommending something other than what i would do myself. so if it were all up to me, i would probably work out how to make an empty release and then document that for developer friendliness, since that's what i would do. | 18:14 |
| *** Goneri has quit IRC | 18:16 | |
| corvus | the most importanting thing for developer friendliness is that we have *something (that works)* documented :) | 18:16 |
| fungi | yep, i completely agree | 18:16 |
| *** slaweq has quit IRC | 18:17 | |
| fungi | we could also recommend both creating a basically empty package (my old instructions there are already partly sufficient), and then mention that if the submitter is comfortable taking the risk that they should make sure to tag a very early (even alpha) release as soon as they merge packaging boilerplate to properly exercise the automation anyway | 18:17 |
| fungi | i wonder if simply stuffing that dummy PKG-INFO file into the right path in a tarball would do the trick | 18:18 |
| *** jpena is now known as jpena|off | 18:18 | |
| *** hashar is now known as hasharDinner | 18:19 | |
| *** jkilpatr has quit IRC | 18:19 | |
| fungi | the primary hurdle we ran into with the last iteration of instructions (before the upload form on pypi.p.o was taken away for good) was people who only had access to windows systems trying to figure out how to use notepad to create a file without an extension | 18:19 |
| *** dtantsur is now known as dtantsur|afk | 18:20 | |
| corvus | fungi: mordred's proposed automation is probably the best response then :) | 18:20 |
| fungi | but again, if they can't get far enough to make a stub python package, they're going to have a devil of a time running any tests on their software locally | 18:21 |
| *** slaweq has joined #openstack-infra | 18:21 | |
| fungi | i'm also perfectly fine with the idea that our instructions strongly urge the reader to have (and be proficient with) a unix/linux environment or else make friends with someone who can do the necessary steps for them | 18:23 |
| fungi | it's not our job to teach people how to computer | 18:23 |
| dhellmann | why do they need to be able to create a file without an extension? | 18:24 |
| fungi | the old instructions said to make a PKG-INFO file | 18:25 |
| dhellmann | oh, ok | 18:25 |
| fungi | for uploading in the (no longer available) registration form | 18:25 |
| dhellmann | ah | 18:25 |
| fungi | after they took away the parts of the form where you could enter the fields directly, but before they got rid of the place where you could still upload a PKG-INFO file | 18:26 |
| fungi | they were progressively paring away the places where name squatters could spam empty project creation with minimal effort | 18:27 |
| *** shardy is now known as shardy_afk | 18:27 | |
| corvus | s/squatters/developers/;s/spam/perform/ | 18:28 |
| fungi | right. basically squatters need automation to be able to do what they want to do on a mass scale... so the pypi response has been to make project creation increasingly hostile to automation (and hence anyone who wants to legitimately automate stuff) | 18:29 |
| corvus | (i'm baffled why this workflow is okay in any situation) | 18:29 |
| corvus | fungi: i don't think it's hostile to automation, i think it's hostile to python developers. | 18:29 |
| *** trown|lunch is now known as trown | 18:29 | |
| fungi | i agree that's the end result. it takes very little additional effort to adjust automation to create and upload empty sdists | 18:30 |
| fungi | yet significantly more effort for people creating new projects on pypi on a smaller scale to wrap their heads around | 18:30 |
| dhellmann | did they really have a problem with bulk squatters? | 18:30 |
| dhellmann | I figured it was just an issue with a few common names or something | 18:31 |
| fungi | yup. for a while there was a rash of typo squatters uploading (thankfully only prankish that anyone found) packages for slight misspellings of popular projects | 18:31 |
| dhellmann | ah | 18:31 |
| *** slaweq has quit IRC | 18:31 | |
| fungi | and also bulk registering discionary words | 18:32 |
| fungi | er, dictionary | 18:32 |
| corvus | i'dve thought a captcha or something would be a preferred approach | 18:32 |
| fungi | ostensibly auctioning off control of them | 18:32 |
| dhellmann | I guess I missed all of that | 18:32 |
| fungi | if memory serves some of them had readmes with instructions on how to purchase control (obviously those got taken down by the pypi admins once reported) | 18:32 |
| corvus | fungi: yeah, that last thing is why the scheme is obviously dumb | 18:33 |
| corvus | "how about instead of paying you, i just ask the nice human to tell you to take a hike?" | 18:33 |
| *** jkilpatr has joined #openstack-infra | 18:33 | |
| *** slaweq has joined #openstack-infra | 18:34 | |
| fungi | but yeah, they have (or at least had anyway) only a couple of part-time sysadmins taking care of pypi so some of the reactions seemed (to me at any rate) to have been of the knee-jerk variety | 18:34 |
| *** beekneemech is now known as bnemec | 18:34 | |
| fungi | they just recently got a very generous infrastructure funding donation so have been able to hire on a lot of additional humans to finish retiring teh cheeseshop and get the remaining missing bits of warehouse implemented | 18:35 |
| *** shardy_afk has quit IRC | 18:35 | |
| fungi | so hopefully the situation will improve in the nearish future | 18:35 |
| *** tesseract has quit IRC | 18:35 | |
| *** armaan has quit IRC | 18:36 | |
| *** shardy_afk has joined #openstack-infra | 18:36 | |
| *** armaan has joined #openstack-infra | 18:36 | |
| fungi | also i gather the removal of the registration form was only partly driven by abuse, and was also seen as a continuation of the plan from some several years ago to transition away from being a general package index separate from package hosting | 18:37 |
| fungi | so well before the registration forms were touched, they stopped having pypi index externally-hosted artifacts | 18:37 |
| fungi | and i'm pretty sure as a result of requiring that projects in the index have their artifacts hosted in pypi itself, the warehouse spec didn't include any form for registering a new project separate from uploading the initial artifact | 18:38 |
| fungi | thus the registration form was seen as a legacy feature of the (deprecated) cheeseshop platform | 18:39 |
| dhellmann | that makes a lot of sense. I still think it's an oversight, but I can see how they would come to that conclusion. | 18:39 |
| dhellmann | maybe 'reserve a name' is a separate operation that creates a record that times out after some period of time | 18:40 |
| corvus | dhellmann: ++ | 18:40 |
| dhellmann | s/is/should be/ | 18:40 |
| fungi | yeah, feels a bit like tunnel-vision to me. but also could be a pragmatic choice they made to keep the hitlist of warehouse requirements for production availability limited to an achievable scope | 18:40 |
| dhellmann | namespaces seems like a better ultimate solution | 18:41 |
| corvus | though, in some ways it shifts the problem | 18:41 |
| dhellmann | although then you have to deal with things like moving a package from @dhellmann/virtualenvwrapper to @virtualenvwrapper/virtualenvwrapper or whatever as new owners take over | 18:42 |
| dhellmann | so you have different issues, too | 18:42 |
| dhellmann | what do the container image registries do about that? do they care? | 18:42 |
| dhellmann | maybe it's not quite the same problem for them | 18:42 |
| fungi | basically, from what i've been able to gather (and encountered myself recently trying to track down the Project_URL support pieces for my setuptools/pbr implementations) the cheeseshop code is an unmaintainable mess of >decade-old spaghetti, and that's contributed to the reluctance of the current team to really implement anything new in it (even something relatively simple like a captcha, or fixing | 18:43 |
| fungi | the now-not-so-recent google openid changes) | 18:43 |
| *** yamamoto has joined #openstack-infra | 18:44 | |
| fungi | and that's driven the move to get warehouse in place with a minimal required featureset so they can officially shut down the cheeseshop before they even begin to talk about more significant changes to the archive | 18:45 |
| dhellmann | yeah, that's definitely my impression, too | 18:46 |
| dhellmann | iirc, the legacy code was built before there were popular web frameworks for python | 18:46 |
| fungi | s/for python// ;) | 18:47 |
| dhellmann | the approach they're taking for replacement makes sense | 18:47 |
| fungi | (depends on your definition of "popular" too i guess) | 18:47 |
| dhellmann | I believe pypi predates even zope | 18:47 |
| fungi | ASPs. very dangerous. you go first | 18:48 |
| clarkb | corvus: ok back from optometrist | 18:49 |
| clarkb | corvus: ya looks like launch node started doing the set hostname thing in march. now to see if set hostname has always had the same behavior | 18:50 |
| clarkb | corvus: we started removing cloud init in october | 18:51 |
| fungi | one of the consistency concerns on the hostname front is that debuntu take a different approach than rhel/centos | 18:51 |
| corvus | clarkb: oh, so maybe up till then, we were still fighting it, but now we've stopped that and it's stabilized? but servers launched before then had the full hostname due to cloud init? | 18:51 |
| fungi | (i think the red hat guidance on setting hostname to an fqdn is patently wrong, but admit my judgement may be clouded by years as a debian user) | 18:52 |
| corvus | fungi: our setting it explicitly will "correct" that though (either approach should work on either system, afaik) | 18:52 |
| dmsimard | The zuul executors losing connections to the hosts are starting to be quite problematic... http://status.openstack.org/elastic-recheck/#1721093 with http://status.openstack.org/elastic-recheck/#1718197 combined account for ~1200 gate failures over the past 10 days. When I last wanted to check, ze09 and ze10 were crashed so I got sidetracked by that but I'll try to understand what's going on now. | 18:52 |
| clarkb | corvus: ya thats my current theory | 18:52 |
| *** yamamoto has quit IRC | 18:52 | |
| dmsimard | A low hanging fruit I was wondering.. would it be worth considering adding a ssh config for keeping the connection alive client-side ? Like .ssh/config "ServerAliveInterval 60" | 18:53 |
| corvus | clarkb: so we should 'correct' all hosts other than ze09? | 18:53 |
| dmsimard | ze09 should have been corrected now (manually) | 18:53 |
| dmsimard | or, well, at least be consistent with the other zuul executors | 18:54 |
| corvus | dmsimard: then it's been 'un-corrected' :) | 18:54 |
| clarkb | corvus: or correct all the xenial hosts and fix things when we go to trusty? | 18:54 |
| clarkb | er | 18:54 |
| clarkb | when we upgrade off of trusty | 18:54 |
| clarkb | fungi: our git.o.o server does the same as ubuntu servers in our case. hostname -> git, hostname -f -> git.openstack.org | 18:54 |
| corvus | dmsimard: i would not consider any changes to ssh in zuul to be low-hanging-fruit due to the thoroughness that must be involved in making any changes, but do please look into it. | 18:54 |
| fungi | dmsimard: it's worth a try as long as paramiko supports it, but the particular incident i looked at last week acted like the node's filesystem had turned read-only and so the ssh connection wasn't actually broken, it just never responded to new commands over the socket | 18:55 |
| dmsimard | corvus: this would be puppet-zuul IMO | 18:55 |
| corvus | dmsimard: this==? | 18:55 |
| dmsimard | corvus: setting up the .ssh/config keepalive | 18:55 |
| fungi | dmsimard: are we using openssh and not paramiko with ansible? | 18:55 |
| corvus | dmsimard: no, zuul manages its own ssh client config | 18:55 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add support for a secure ZooKeeper configuration https://review.openstack.org/531882 | 18:56 |
| * fungi will readily admit the ansible ssh backend is a bit of a black box to him | 18:56 | |
| dmsimard | ansible doesn't use paramiko unless you explicitely tell it to (if ssh is available) | 18:56 |
| dmsimard | paramiko is the fallback | 18:56 |
| corvus | fungi: what dmsimard and we do not tell it to use paramiko so we use openssh | 18:56 |
| fungi | oic, so if no ssh client is present, paramiko gets used | 18:56 |
| dmsimard | corvus: hmm, I know that executor server sets up some ssh configuration for ansible (such as pipelining), I'll look there. | 18:57 |
| corvus | dmsimard: just make sure you consider the whole environment, including especially controlpersist. | 18:57 |
| fungi | dmsimard: also, if the goal is dead peer detection, see the ServerAliveCountMax and ClientAliveCountMax options (for ssh_config and sshd_config respectively) | 18:58 |
| fungi | though, again, if the particular node i looked at last week is representative, i don't think _any_ of thos will help | 18:59 |
| dmsimard | fungi: yeah I know about ServerAliveInterval for client side, sshd_config would need to go into the nodepool VMs I guess ? | 18:59 |
| fungi | s/thos/this/ | 18:59 |
| dmsimard | fungi: there's been ~1200 failures in recent history, I doubt they all match that particular pattern | 18:59 |
| dmsimard | but it's something I want to find out | 18:59 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Rename _useBuilder method to useBuilder https://review.openstack.org/531883 | 18:59 |
| fungi | dmsimard: right, if we want the nodes to send keepalives and to detect the executor going away, then presumably have dib adjust the /etc/ssh/sshd_config | 19:00 |
| * dmsimard nods | 19:01 | |
| corvus | what would that accomplish? | 19:01 |
| *** slaweq has quit IRC | 19:01 | |
| fungi | though just doing the client side on the executor may be sufficient given that i doubt there's any graceful anything we need the nodes to concern themselves with | 19:01 |
| *** yamamoto has joined #openstack-infra | 19:01 | |
| dmsimard | yeah just ServerAliveInterval by itself is magic... I'll even recover ssh connections after waking up my laptop from hibernation :) | 19:01 |
| *** slaweq has joined #openstack-infra | 19:02 | |
| fungi | the keepalive is a ping/pong so doing it from either client or server should be sufficient to keep firewalls and nats from destroying what it sees as idle connections. from a dead peer detection standpoint we probably only care that the executor can detect the client failing to respond and not the other way around | 19:03 |
| fungi | since an unresponsive node means the job fails and nodepool deletes it | 19:03 |
| dmsimard | from looking elastic-recheck, it seems to happen in waves | 19:03 |
| dmsimard | There was two large peaks recently, one today and one Jan 4th | 19:04 |
| clarkb | fungi: I've also just now sorted out the ssh hostkey for ansible and git.openstack.org | 19:05 |
| clarkb | mordred: fungi can you review https://review.openstack.org/#/c/531851/1 to fix puppet on git0*.o.o? | 19:06 |
| *** slaweq has quit IRC | 19:06 | |
| fungi | thanks! i missed that one getting uploaded | 19:06 |
| dmsimard | 126x "Ansible timeout exceeded" on ze09 alone | 19:08 |
| dmsimard | nevermind, half that because it was picking up both debug and normal logs | 19:08 |
| dmsimard | a few OOM kills but not that many | 19:09 |
| *** xarses_ has quit IRC | 19:09 | |
| *** slaweq has joined #openstack-infra | 19:09 | |
| *** xarses_ has joined #openstack-infra | 19:09 | |
| dmsimard | starting a pad.. I'll write my notes there: https://etherpad.openstack.org/p/executor-timeouts | 19:10 |
| *** harlowja has joined #openstack-infra | 19:19 | |
| *** slaweq has quit IRC | 19:19 | |
| *** pramodrj07 has joined #openstack-infra | 19:22 | |
| *** e0ne has joined #openstack-infra | 19:22 | |
| *** pramodrj07 has quit IRC | 19:22 | |
| *** wolverineav has quit IRC | 19:23 | |
| *** wolverineav has joined #openstack-infra | 19:23 | |
| *** wolverineav has quit IRC | 19:26 | |
| clarkb | dmsimard: you might try correlating it to high load, cacti should help with historical load data | 19:26 |
| *** wolverineav has joined #openstack-infra | 19:26 | |
| dmsimard | yeah, I am to some extent with grafana | 19:26 |
| dmsimard | for today's wave, there has been no OOM killers though | 19:26 |
| dmsimard | only 2 on ze09, nothing on the others | 19:26 |
| clarkb | load is orthogonal to oomkiller though | 19:26 |
| clarkb | but if load gets high enough it may not be able to keep all those connections open reliable | 19:27 |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/531430/ , please? - we need this to fix publishing of openstack-manuals and friends. It removes some explicit parameter, the defaults work just fine | 19:28 |
| *** slaweq has joined #openstack-infra | 19:28 | |
| * AJaeger waves good bye | 19:29 | |
| dmsimard | clarkb: we're on the same wavelength for that, load is generally high (above 10) but I'm trying to see if there's anything in particular | 19:29 |
| fungi | load is a relative measurement. what's "high" for some systems may be entirely reasonable for others (for example an mta with a system load >250 might still be entirely healthy) | 19:30 |
| dmsimard | the notion of keeping those 80+ concurrent ansible playbooks running (over ssh) reliably must be a bit challenging, I mentioned the client-side (executor) ssh keepalive we could add but perhaps there's something else | 19:30 |
| dmsimard | fungi: right, it also depends on a multitude of factors.. cpu, ram but also i/o (and i/o wait) | 19:31 |
| *** jamesmcarthur has quit IRC | 19:31 | |
| dmsimard | for example, keeping ze09's top open, I see a consistent ~10% i/o wait | 19:31 |
| *** dhajare has joined #openstack-infra | 19:32 | |
| dmsimard | kwapd taking a large amount of CPU cycles | 19:32 |
| dmsimard | kswapd* | 19:32 |
| dmsimard | so we're likely swapping out to a (slow) disk because we're running out of ram and it exacerbates everything else ? | 19:33 |
| *** jkilpatr has quit IRC | 19:33 | |
| fungi | dmsimard: yeah, the executor is using a fair amount of ram | 19:34 |
| fungi | dipping 2.2gb into swap there is likely unhealthy | 19:34 |
| fungi | 5660080 virt for zuul-executor pid 9718 | 19:35 |
| dmsimard | the ram usage is why I was inquiring about where ansible kept it's output buffer the other day | 19:36 |
| dmsimard | I know we end up streaming it, but we likely don't discard the "standard" output (which is, amongst other things, passed to ara) | 19:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies https://review.openstack.org/530806 | 19:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move CRD tests to test_gerrit_legacy_crd https://review.openstack.org/531886 | 19:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add skipped CRD tests https://review.openstack.org/531887 | 19:37 |
| dmsimard | some of the "command" (legacy and other) tasks end up running for an hour just by themselves, that's one hour worth of buffering output | 19:38 |
| dmsimard | for example "run devstack" (cue 30+ minutes of waiting for this one task) | 19:38 |
| * dmsimard asks #ansible-devel | 19:39 | |
| *** Goneri has joined #openstack-infra | 19:39 | |
| *** e0ne has quit IRC | 19:40 | |
| openstackgerrit | Merged openstack-infra/puppet-jeepyb master: Revert "Remove python-six on centos" https://review.openstack.org/531851 | 19:41 |
| *** felipemonteiro_ has joined #openstack-infra | 19:41 | |
| dmsimard | okay, so the output is buffered in RAM on the target node -- so not on the executor node. | 19:41 |
| corvus | dmsimard: we don't discard it. the output is buffered on the remote host while the task is running. then it's sent back to ansible running on the executor at the end. | 19:42 |
| *** wolverineav has quit IRC | 19:42 | |
| corvus | dmsimard: ansible will need to process it. i don't know how efficient it is with processing large json blobs | 19:43 |
| dmsimard | corvus: how do you feel about the ram usage on the executors ? is that something we want to look into or do we want to throw more machines at it ? | 19:43 |
| SpamapS | Curious.. have we seen impact from meltdown fixes at the infra node providers? | 19:43 |
| corvus | dmsimard: i absolutely want to have the ram governor in place | 19:43 |
| corvus | dmsimard: i don't want to spin up more executors until it is | 19:43 |
| fungi | SpamapS: no directly observed impact so far | 19:43 |
| SpamapS | I'd guess by now some of them are rolling out PTI enabled hypers. | 19:43 |
| fungi | SpamapS: at least as far as i'm aware | 19:43 |
| *** david-lyle has joined #openstack-infra | 19:43 | |
| *** wolverineav has joined #openstack-infra | 19:44 | |
| dmsimard | corvus: yeah, but beyond that, I mean, do you think the ram usage is legitimate right now ? | 19:44 |
| corvus | dmsimard: hard to say without the ram governor. the distribution is not sufficiently even | 19:44 |
| fungi | SpamapS: the vast majority of our control plane in in rax, which runs xen, and the guest risk on xen is mitigated by using hvm, so we may see no impact there | 19:44 |
| SpamapS | pabelanger: weren't you resurrecting my RAM governor patch? | 19:44 |
| dmsimard | there's a zuul meeting today, let's chat ram governor then :D | 19:44 |
| SpamapS | fungi: guest risk, but you still have to update your hyper kernel. | 19:45 |
| *** felipemonteiro has quit IRC | 19:45 | |
| fungi | SpamapS: yeah, in theory they live migrate us around to do that (in practice, it breaks for some of our instances, but that's been a constant thing well before now) | 19:45 |
| fungi | SpamapS: the main impact i'm expecting to observe is performance | 19:46 |
| fungi | but that's going to be tough to measure | 19:46 |
| SpamapS | I'm also pretty sure zuul executors will run for crap with PTI. | 19:46 |
| SpamapS | Since the worst case is "lots of network I/O with CPU in between" | 19:46 |
| dmsimard | yeah, it sucks. | 19:47 |
| SpamapS | Have we already built test images with Ubuntu's new kernels? | 19:47 |
| dmsimard | they're not out yet unless mistaken | 19:47 |
| SpamapS | IIRC they just published today | 19:47 |
| dmsimard | last I looked earlier today .. | 19:47 |
| dmsimard | released kernels are still TBD https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown | 19:48 |
| SpamapS | https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/pti/ | 19:48 |
| *** TLAXKIT has quit IRC | 19:48 | |
| SpamapS | PPA only right now | 19:48 |
| fungi | what a marvellous place for them | 19:48 |
| dmsimard | it's a staging ground before release afaik | 19:49 |
| SpamapS | I believe they're taking it slow because of the perf problems they raise. | 19:49 |
| SpamapS | I mean, there's nothing to protect from exploits if the DC burns down. ;) | 19:49 |
| fungi | makes sense, assuming they consider potential performance regression to be a bigger problem than your users taking over your servers | 19:50 |
| dmsimard | yeah... meltdown seems scaringly easy to exploit | 19:50 |
| clarkb | SpamapS: rax is largely unaffected because hvm guests cant attack the hypervisor | 19:50 |
| dmsimard | especially if it has really existed for the past 20 years .. | 19:50 |
| clarkb | SpamapS: so havent really seen anything on our control plane | 19:50 |
| dmsimard | clarkb: do we know if rax is up to date ? | 19:50 |
| clarkb | dmsimard: they dont have to update if they dont have pv nodes | 19:51 |
| dmsimard | Well, regardless of virtualization attack vectors, they likely need to update their hypervisors and that would have an impact on performance (does ephemeral live on the compute nodes?) | 19:52 |
| dmsimard | s/hypervisors/compute nodes/ | 19:52 |
| clarkb | thats what I'm saying they likely dont have to because hvm | 19:53 |
| clarkb | xen is only affected if running 64bit pv guests | 19:53 |
| *** yamamoto has quit IRC | 19:53 | |
| dmsimard | meltdown isn't just about virtualization, it's about processes being able to read RAM from other processes (cross boundary) -- so I would think they want PTI enabled on their compute nodes even if the VMs can't attack them, it's still a door opened for anything else.. right ? | 19:55 |
| fungi | unless they simply assume that any account on the hypervisor host is equivalent to root | 19:55 |
| clarkb | xen wrote a blog about it | 19:56 |
| clarkb | they seemed to bein a pretty good position | 19:56 |
| dmsimard | yeah, but the node has other software (l3 agent, ovs, dnsmaq or whatever) | 19:56 |
| clarkb | aiui beause they already do multiple virtual address spaces | 19:56 |
| fungi | i think dmsimard's point is that local processes on the hypervisor host are still a risk even if the guests aren't | 19:56 |
| dmsimard | fungi: +1 | 19:56 |
| clarkb | sure but those are all privileged anyeays | 19:57 |
| clarkb | every one of those is | 19:57 |
| dmsimard | ¯\_(ツ)_/¯ | 19:57 |
| fungi | which is what i meant by "assume that any account on the hypervisor host is equivalent to root" | 19:57 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add support for a secure ZooKeeper configuration https://review.openstack.org/531882 | 19:57 |
| fungi | odds are in a hands-off deployment that very well may be a safe enough assumption | 19:57 |
| clarkb | also those dont run on the hypervisor with xen right? | 19:58 |
| clarkb | so guests wont see a cpu impact | 19:58 |
| clarkb | js I guess what I am getting at as they already run in this safer mode | 19:58 |
| fungi | depends on how you have xen deployed, but yes commonly the control processes actually run in a separate guest as well | 19:59 |
| fungi | and so there's really no process other than the kernel in the "host" | 19:59 |
| openstackgerrit | Monty Taylor proposed openstack-infra/jeepyb master: Add checking for PyPI project ownership https://review.openstack.org/531898 | 20:00 |
| *** sdague has joined #openstack-infra | 20:00 | |
| fungi | granted, it's been a long while since i ran any xen hosts myself so some of that may have changed | 20:00 |
| mordred | fungi, dhellmann, AJaeger, corvus: ^^ there's project creation validation added to manage-projects as well as a utility script we can use in a gate job to do the same thing in check/gate | 20:00 |
| clarkb | but yes guests should update but thats on the customer (us) | 20:01 |
| clarkb | and we may see an impact once ubuntu pushes out kernels | 20:01 |
| *** Apoorva has quit IRC | 20:01 | |
| *** Apoorva has joined #openstack-infra | 20:02 | |
| *** Goneri has quit IRC | 20:02 | |
| dhellmann | mordred : added to my review queue | 20:02 |
| *** jgriffith_ is now known as jgriffith | 20:03 | |
| EmilienM | pabelanger: when you have time, please look https://review.openstack.org/#/q/status:open+project:openstack-infra/system-config+branch:master+topic:puppetlabs/mirrors | 20:04 |
| fungi | who was working on zuul v3 support in puppet-openstackci? | 20:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/jeepyb master: Add checking for PyPI project ownership https://review.openstack.org/531898 | 20:04 |
| clarkb | fungi: mmedvede | 20:04 |
| fungi | thanks clarkb | 20:04 |
| *** yamamoto has joined #openstack-infra | 20:04 | |
| mordred | dhellmann: it SHOULD be possible to try out the check-pypi command with just a local copy of projects.yaml | 20:05 |
| dhellmann | k | 20:05 |
| openstackgerrit | Amy Marrich (spotz) proposed openstack-infra/irc-meetings master: Add entry for the Diversity WG https://review.openstack.org/531900 | 20:05 |
| *** dsariel has joined #openstack-infra | 20:06 | |
| *** snapiri1 has joined #openstack-infra | 20:07 | |
| *** snapiri1 has quit IRC | 20:08 | |
| *** CrayZee has joined #openstack-infra | 20:08 | |
| *** yamamoto has quit IRC | 20:09 | |
| *** jkilpatr has joined #openstack-infra | 20:11 | |
| CrayZee | Hi, Is there something wrong with zuul or is it just a problem on my side? | 20:13 |
| mwhahaha | i'm getting a service unavailable from zuulv3.o.o so probably not just you | 20:14 |
| *** ldnunes has quit IRC | 20:15 | |
| dmsimard | web interface is working for me right now | 20:16 |
| dmsimard | but there was a definite drop in RAM usage, maybe the scheduler restarted ? | 20:16 |
| dmsimard | corvus: ^ | 20:17 |
| mwhahaha | must be nice, i'm still getting 503 | 20:17 |
| dmsimard | oh crap | 20:17 |
| dmsimard | [Mon Jan 8 19:55:28 2018] traps: zuul-web[26027] general protection ip:513ef4 sp:7f2f9a521798 error:0 in python3.5[400000+3a9000] | 20:17 |
| CrayZee | same here | 20:17 |
| dmsimard | [Mon Jan 8 19:55:40 2018] traps: zuul-scheduler[25965] general protection ip:7f1ffbfbe746 sp:7f1fadff8718 error:0 in libc-2.23.so[7f1ffbf33000+1c0000] | 20:17 |
| dmsimard | clarkb, fungi, corvus: msgpack failure propagated to zuulv3.o.o just now | 20:18 |
| dmsimard | modules were updated not long ago http://paste.openstack.org/raw/641017/ | 20:18 |
| dmsimard | module update timestamps correlate with the gpf | 20:19 |
| dmsimard | infra-root: unless someone wants to investigate why this has happened, I'll hotfix like zm and ze yesterday | 20:20 |
| fungi | i wonder if upgrading/rebuilding linked c extensions underneath a running python is unsafe | 20:20 |
| dmsimard | fungi: the msgpack rename has to have something to do with it | 20:20 |
| dmsimard | hotfixing | 20:20 |
| dmsimard | zuul-web and zuul-scheduler have been started | 20:22 |
| *** Goneri has joined #openstack-infra | 20:22 | |
| CrayZee | dmsimard: +1 - working on my side | 20:22 |
| dmsimard | I'm seeing a lot of those errors in the zuulv3.o.o logs: http://paste.openstack.org/raw/641019/ | 20:24 |
| dmsimard | zuulv3.o.o seems to still have all the ongoing jobs listed... really ? | 20:24 |
| dmsimard | hmm, now I realize that it was pending scheduler reload -- should have taken the opportunity to dump the queues | 20:26 |
| *** edmondsw has quit IRC | 20:26 | |
| *** beagles has joined #openstack-infra | 20:27 | |
| dmsimard | fungi: is this okay ? #status notice The jobs and queues in Zuul between 19:55UTC and 20:20UTC have been lost after recovering from a crash, you might need to re-check your patches if they were being tested during that period. | 20:32 |
| fungi | dmsimard: sure, good enough | 20:38 |
| fungi | i sometimes mention reapplying approvals, but that's long enough prose already ;) | 20:38 |
| fungi | dmsimard: you likely also want to #status log a summary of the actions you took on the server | 20:38 |
| fungi | for the benefit of our sysadmins in other timezones | 20:38 |
| dmsimard | #status log (dmsimard) the msgpack issue experienced yesterday on zm and ze nodes propagated to zuulv3.o.o and crashed zuul-web and zuul-scheduler with the same python general protection fault. They were started after re-installing msgpack but the contents of the queues were lost. | 20:38 |
| openstackstatus | dmsimard: finished logging | 20:38 |
| dmsimard | was already on it :) | 20:38 |
| fungi | perfect! | 20:38 |
| *** jamesmcarthur has joined #openstack-infra | 20:38 | |
| dmsimard | #status notice The jobs and queues in Zuul between 19:55UTC and 20:20UTC have been lost after recovering from a crash, you might need to re-check your patches if they were being tested during that period. | 20:38 |
| openstackstatus | dmsimard: sending notice | 20:38 |
| dmsimard | brb | 20:38 |
| *** openstack has quit IRC | 20:38 | |
| *** openstack has joined #openstack-infra | 20:40 | |
| *** ChanServ sets mode: +o openstack | 20:40 | |
| pabelanger | SpamapS: Yup, hopefully can rebase that today and push it up | 20:43 |
| SpamapS | pabelanger: rebasing it is not easy. I suggest manually re-doing the patch. | 20:44 |
| SpamapS | pabelanger: the bits have changed too much since I did it. | 20:44 |
| SpamapS | git can't figure it out. | 20:44 |
| pabelanger | SpamapS: ack | 20:44 |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul master: zuul-cloner: recognizes bare repos in cache dir https://review.openstack.org/249207 | 20:45 |
| tosky | the last message from statusbot was sent on twitter as usual, but it was split; I guess that the bot plugin that handles twitter should be updated to support the 280 character tweets | 20:47 |
| pabelanger | fungi: looks like git servers are working again! | 20:50 |
| fungi | yup. thanks for the revert | 20:51 |
| pabelanger | https://review.openstack.org/531837/ and https://review.openstack.org/531401 could also use a review to help get automated updates applied to centos | 20:52 |
| *** jamesmcarthur has quit IRC | 20:54 | |
| *** jamesmcarthur has joined #openstack-infra | 20:54 | |
| Shrews | tosky: depends on python-twitter lib. looks like we're waiting for 3.3.1 to be released: https://github.com/bear/python-twitter/pull/525 | 20:55 |
| fungi | yeah, ttx was going to patch statusbot to support long tweets until i pointed out that we need a new python-twitter supporting the protocol change first | 20:57 |
| tosky | Shrews: ah,good to know | 20:57 |
| fungi | easy enough to fix once the lib has support | 20:57 |
| *** jtomasek has quit IRC | 20:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add charm-interface-designate project https://review.openstack.org/517336 | 21:01 |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul master: Add __repr__ to a few classes https://review.openstack.org/361064 | 21:01 |
| *** openstackgerrit has quit IRC | 21:03 | |
| *** CrayZee has left #openstack-infra | 21:03 | |
| *** dprince has quit IRC | 21:06 | |
| *** dhajare has quit IRC | 21:07 | |
| *** smatzek has quit IRC | 21:11 | |
| *** smatzek has joined #openstack-infra | 21:11 | |
| *** jamesmcarthur has quit IRC | 21:12 | |
| *** rhallisey has quit IRC | 21:14 | |
| *** smatzek has quit IRC | 21:16 | |
| *** rcernin has joined #openstack-infra | 21:17 | |
| *** openstackgerrit has joined #openstack-infra | 21:18 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies https://review.openstack.org/530806 | 21:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove updateChange history from github driver https://review.openstack.org/531904 | 21:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/jeepyb master: Add checking for PyPI project ownership https://review.openstack.org/531898 | 21:19 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 21:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/jeepyb master: Add checking for PyPI project ownership https://review.openstack.org/531898 | 21:20 |
| clarkb | fungi: so you think the compile process for linking against python may trip up running python processes? | 21:22 |
| *** gouthamr has quit IRC | 21:22 | |
| dmsimard | What makes it so pip tries to get a package that isn't yet on the mirror ? There's a lot of "No matching distribution found" for various packages (it's our most popular elastic-recheck item) | 21:22 |
| dmsimard | For example, here we see a failure getting os-brick==1.15.5 @ 2018-01-08 16:59:21 http://logs.openstack.org/36/531836/1/check/legacy-grenade-dsvm-neutron-multinode/f84f74d/logs/grenade.sh.txt#_2018-01-08_16_59_21_956 | 21:23 |
| clarkb | dmsimard: some of it is "normal" iirc osa or something tries to install a package that was deleted from pypi on a stable branch | 21:23 |
| dmsimard | And Bandersnatch recovers it ~3 minutes before that: 2018-01-08 16:56:40,050 INFO: Downloading: https://pypi.python.org/packages/37/29/820cdc09fec4701a615cf4293f1dbfbfa5b5e12ba4bcd1949b4a660b4d31/os-brick-1.15.5.tar.gz | 21:23 |
| clarkb | dmsimard: so one case is when pypi deletes a package so you can no longer satisfy the requirement(s) | 21:23 |
| dmsimard | clarkb: the couple cases I've isolated were all "legit" like the one above | 21:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add memory awareness to system load governor https://review.openstack.org/508960 | 21:25 |
| clarkb | dmsimard: that is interesting, looks like that release came out today so likely a race between updating constraints and updating our mirror? | 21:25 |
| dmsimard | I can't find a reference to os-brick==1.15.5 in openstack/requirements | 21:26 |
| clarkb | dmsimard: I think releases push constraints updates and if someone manages to merge that constraint update before our mirror update pulls it in that could happen | 21:26 |
| dmsimard | but it might just be me looking at the wrong place | 21:26 |
| clarkb | we might also push the constriant update before we actually publish to pypi | 21:26 |
| clarkb | which could be another source of racing | 21:26 |
| * clarkb looks at zuul config | 21:26 | |
| dmsimard | clarkb: that's exactly the kind of thing I was expecting to see and fix | 21:26 |
| *** Anticimex has joined #openstack-infra | 21:27 | |
| dmsimard | well, I'm not sure how fixable it is -- the commit will always be merged before the post job (that publishes to pypi) runs I would guess | 21:27 |
| *** r-daneel has joined #openstack-infra | 21:27 | |
| clarkb | propose-update-constraints depends on release-openstack-python | 21:27 |
| clarkb | dmsimard: the commit/tag to make a release is what triggers the job to make a new ocmmit to update constraints | 21:28 |
| *** david-lyle has quit IRC | 21:28 | |
| clarkb | publish-to-pypi is the template in ozj | 21:28 |
| * dmsimard looks | 21:28 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/jeepyb master: Add checking for PyPI project ownership https://review.openstack.org/531898 | 21:29 |
| dmsimard | well, I found the propose-constraints for os-brick (thanks zuul dashboard) http://logs.openstack.org/b8/b8bb0ab8b33d67786c3673b3de29be4ed68d4ee5/release/propose-update-constraints/7f4e9b4/ | 21:29 |
| *** aeng has joined #openstack-infra | 21:30 | |
| dmsimard | from https://review.openstack.org/531836 | 21:30 |
| fungi | clarkb: we discussed maybe having the constraints update job wait until it could find the package on the mirror, but nobody's worked on solving the race yet. requirements reviewers just recheck it | 21:30 |
| dmsimard | wait that's not merged yet o_O | 21:30 |
| *** kgiusti has left #openstack-infra | 21:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add PyPI settings for jeepyb https://review.openstack.org/531909 | 21:31 |
| dmsimard | hah, bummer. The errors for os-brick seem to be all coming from that propose-constraints patch. | 21:31 |
| clarkb | dmsimard: ok so those are legit fails but its failing in a way that is good :) eg preventing us from merging something that would break | 21:31 |
| clarkb | I think that probably indicates our pypi mirroring may be broken | 21:32 |
| dmsimard | fungi: seems like an opportunity to get an ansible retry/wait going | 21:32 |
| clarkb | dmsimard: mirror-update.openstack.org is the host that updates pypi and other mirrors. It does so out of cron | 21:32 |
| dmsimard | yeah, already looked at the bandersnatch logs to see if there was anything unusual | 21:32 |
| clarkb | pabelanger: I don't see a cron entry for pypi on ^ | 21:32 |
| clarkb | does it rn as not root /me checks | 21:33 |
| dmsimard | it does | 21:33 |
| *** rcernin has quit IRC | 21:33 | |
| pabelanger | clarkb: checking | 21:33 |
| clarkb | ah ok | 21:33 |
| dmsimard | */5 * * * * flock -n /var/run/bandersnatch/mirror.lock bandersnatch-mirror-update /var/log/bandersnatch/mirror.log | 21:33 |
| clarkb | dmsimard: the end of the log file seems to indicate it htinks it is happy | 21:33 |
| clarkb | dmsimard: it could be a bug on the pypi side | 21:34 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/mirror_update.pp#n71 | 21:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Run jeepyb-check-pypi on project-config patches https://review.openstack.org/531911 | 21:34 |
| dmsimard | clarkb: could it just be stale cache from the mirror/afs ? | 21:34 |
| dmsimard | the package was downloaded by bandersnatch ~3 minutes before pip got a "404" | 21:34 |
| pabelanger | dmsimard: clarkb: bandersnatch looks to be running every 5mins or so, according to log | 21:35 |
| mordred | fungi, dhellmann, corvus: ^^ added a zuul job to the jeepyb patch and a patch to configure it to run on project-config changes - I think that should be about good for today | 21:35 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Correctly use project name in getGitUrl https://review.openstack.org/528309 | 21:35 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Initialize github auth in getGitUrl if needed https://review.openstack.org/528310 | 21:35 |
| clarkb | dmsimard: in that case it could be a bug in the only release afs when we need to check | 21:35 |
| clarkb | dmsimard: since a 404 likely made it fail, then next time we run we only check against the current run not previous runs | 21:36 |
| *** hemna_ has joined #openstack-infra | 21:37 | |
| *** gouthamr has joined #openstack-infra | 21:37 | |
| clarkb | ya bandersnatch-mirror-update is set -e so if a bandersnatch run fails it won't check if it needs to sync | 21:39 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul feature/zuulv3: Link to zuul-base-jobs docs from User's Guide https://review.openstack.org/531912 | 21:39 |
| clarkb | You'd expect the next run would sync something though since the previous failed | 21:39 |
| *** yamamoto has joined #openstack-infra | 21:39 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use configured github server in app mode https://review.openstack.org/528311 | 21:39 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix github app authentication in _getNeededByFromPR https://review.openstack.org/528312 | 21:39 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add the project to GithubUser https://review.openstack.org/528313 | 21:39 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Reprime the installation map in getGitUrl if needed https://review.openstack.org/529506 | 21:39 |
| smcginnis | clarkb: Is it possible it didn't update the index and now it needs a PURGE on that one? | 21:41 |
| dmsimard | clarkb: bandersnatch didn't get a 404, it's pip who did | 21:41 |
| clarkb | smcginnis: it seems to hvae grabbed the package so I don't think so | 21:41 |
| dmsimard | pip fails to get os-brick==1.15.5, but it was downloaded by bandersnatch ~3 minutes prior so it's possible the mirror's AFS or Apache cache was stale | 21:42 |
| clarkb | dmsimard: I'm comparing timestamps. 16:56 is when bandersnatch grabbed it | 21:42 |
| dmsimard | yeah, and it failed at 16:59 | 21:42 |
| smcginnis | I ran bandersnatch internally for a little over a year. Probably had about 3-4 times I had to run the PURGE thing. | 21:42 |
| dmsimard | (the pip in the job) | 21:42 |
| clarkb | dmsimard: 2018-01-08T16:59:52,888404332+0000 is when we synced it | 21:42 |
| clarkb | so thats the race there | 21:43 |
| clarkb | dmsimard: since the mirrors won't see it until the vos release | 21:43 |
| clarkb | it is interesting that it took almost an hour for bandersnatch to pick it up from when it was published to pypi | 21:45 |
| clarkb | but we likely can't do much to speed that up as pypi drives those events | 21:45 |
| dmsimard | clarkb: pip doesn't query pypi from a nodepool node, does it ? | 21:45 |
| dmsimard | I mean, we're configuring it to query our mirrors ? | 21:45 |
| clarkb | dmsimard: you mean we don't use upstream pypi from our jobs? no they should mostly use our mirrors (but there are some exceptions) | 21:45 |
| SamYaple | looking for review love https://review.openstack.org/#/c/517105/ bindep -- 2 months and counting | 21:46 |
| dmsimard | When I set up the configure-mirrors role in z-j, I don't remember setting up something for pypi.. wonder how that ends up being set up /me looks | 21:46 |
| clarkb | dmsimard: I'm not sure I understand | 21:46 |
| clarkb | you juts mean how do we configure out mirrors? | 21:47 |
| *** yamamoto has quit IRC | 21:47 | |
| dmsimard | yeah | 21:47 |
| clarkb | ah, it was part of the nodepool ready script stuff before zuulv3 but now its part of a very base job I think | 21:47 |
| pabelanger | it should be setup with configure-mirror role, I remember writing it for pip.conf | 21:48 |
| dmsimard | right, I'm the one who translated the mirror setup from readyscript to a role and I don't remember setting something up for pip | 21:48 |
| dmsimard | looking at configure-mirror I don't see pip stuff either .. | 21:48 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/configure-mirrors/tasks/mirror.yaml#n1 | 21:48 |
| dmsimard | well, there it is | 21:49 |
| dmsimard | pabelanger: makes sense I don't remember writing it if it was you who wrote it :p | 21:49 |
| pabelanger | so, something is hitting pypi directly? | 21:50 |
| *** slaweq has quit IRC | 21:50 | |
| pabelanger | at one point we did that for pep8 jobs to test wheels building, not sure we kept that for zuulv3 migration | 21:50 |
| dmsimard | pabelanger: I don't think but I wanted to make sure -- what seems to be happening is that an index is referencing a package that hasn't yet been pushed | 21:51 |
| dmsimard | so pip tries to download it but it's not there yet | 21:51 |
| clarkb | ya its definitely hitting our mirror, that is why the package doesn't exist yet | 21:51 |
| clarkb | its just racing bandersnatch + vos release | 21:51 |
| *** smatzek has joined #openstack-infra | 21:52 | |
| pabelanger | I thought index was only updated after vos release, to avoid this issue | 21:52 |
| *** eharney has quit IRC | 21:53 | |
| dmsimard | pabelanger: well, it's either that or the requirements jobs force that version (this particular case ocurred in a constraints update) | 21:53 |
| clarkb | pabelanger: the issue is we made a release of os-brick and then immediately push a change to requirements to bump constraints up to use it | 21:53 |
| clarkb | pabelanger: that change races our mirroring | 21:53 |
| *** esberglu has quit IRC | 21:53 | |
| pabelanger | clarkb: okay, that makes sense | 21:54 |
| clarkb | pabelanger: vos releasing ensures that we always publish a consistent pypi it does not guaruntee that it will be up to date | 21:54 |
| pabelanger | yah | 21:54 |
| fungi | pabelanger: the wheel building deps test was that we skipped our custom wheel mirror and went straight to our bandersnatch-built pypi mirror | 21:54 |
| fungi | not to global pypi | 21:55 |
| *** Goneri has quit IRC | 21:55 | |
| fungi | we also, further back, used to have requirements test jobs use real pypi instead of our pypi mirror, mainly because of races (though that came from a time when we only maintained a partial mirror and so you couldn't otherwise test proposed additions which we weren't yet configured to add to the mirror) | 21:56 |
| *** smatzek has quit IRC | 21:56 | |
| pabelanger | speaking of wheels, I guess we could have requirements job trigger building them again? Now that they are native zuulv3 jobs. Or actually, I think we want to wait for the enqueue job once into post pipeline patch for that | 21:57 |
| *** trown is now known as trown|outtypewww | 21:58 | |
| dmsimard | I'd like to try something in the propose-constraints job but it's running out of project-config so wouldn't be able to use depends-on :( | 21:58 |
| fungi | i'd also like to see our wheel mirror stop building wheels which are already available from pypi/our pypi mirror | 21:59 |
| pabelanger | fungi: Yah, for the most part, it is only 30mins per wheel type now. But agree, would make the jobs much faster | 21:59 |
| fungi | because we technically aren't testing how projects work with official wheels of deps since we host rebuilt ones and use those instead | 21:59 |
| pabelanger | there was also a bug in zuul, that kept old branches around, so we end up building wheels for ocata | 22:00 |
| clarkb | fungi: that has saved us once or twice after pypi deletes a wheel but ya not ideal | 22:00 |
| corvus | it's zuul meeting time in #openstack-meeting-alt | 22:00 |
| pabelanger | even liberty | 22:00 |
| fungi | less about speed of building the mirror and more about that we're not testing what we think we are | 22:00 |
| clarkb | fungi: right, and while we may keep working no one else does making our software non deployable | 22:00 |
| *** rkukura has quit IRC | 22:04 | |
| *** esberglu has joined #openstack-infra | 22:04 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add support for a secure ZooKeeper configuration https://review.openstack.org/531882 | 22:05 |
| *** rcernin has joined #openstack-infra | 22:06 | |
| *** jamesmcarthur has joined #openstack-infra | 22:08 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Clarify provider manager vs provider config https://review.openstack.org/531618 | 22:11 |
| *** jamesmcarthur has quit IRC | 22:13 | |
| *** florianf has quit IRC | 22:18 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] Support networkx 2.0 https://review.openstack.org/506524 | 22:21 |
| *** afred312 has quit IRC | 22:22 | |
| *** erlon has quit IRC | 22:24 | |
| *** e0ne has joined #openstack-infra | 22:28 | |
| *** rkukura has joined #openstack-infra | 22:31 | |
| *** dave-mccowan has quit IRC | 22:37 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add ansible/ansible to zuul https://review.openstack.org/504510 | 22:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Run shade functional tests on ansible OpenStack PRs https://review.openstack.org/531916 | 22:38 |
| *** armax has quit IRC | 22:40 | |
| *** wolverineav has quit IRC | 22:41 | |
| *** hongbin has quit IRC | 22:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Run shade functional tests on ansible OpenStack PRs https://review.openstack.org/531916 | 22:47 |
| *** rlandy is now known as rlandy|bbl | 22:47 | |
| mordred | corvus, pabelanger: ^^ ... https://review.openstack.org/#/q/topic:turn-on-ansible has the whole thing | 22:48 |
| *** e0ne has quit IRC | 22:49 | |
| *** david-lyle has joined #openstack-infra | 22:55 | |
| *** slaweq has joined #openstack-infra | 22:58 | |
| *** hashar has quit IRC | 22:58 | |
| *** slaweq has quit IRC | 23:00 | |
| *** afred312 has joined #openstack-infra | 23:00 | |
| *** gcb has joined #openstack-infra | 23:01 | |
| ianw | https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe <-- this had passed me by but is good | 23:04 |
| mordred | clarkb, fungi, pabelanger: https://review.openstack.org/#/c/504510 is safe to land right now- it'll only make ansible/ansible available in required-projects | 23:04 |
| ianw | i'll get to a dib release so we can try out fedora27 soon, could simplify a number of things longer term | 23:04 |
| mordred | ianw: ZOMG. that's awesome! | 23:05 |
| clarkb | ianw: looks like they are going the debian route of using /usr/local for pip | 23:05 |
| mordred | yah | 23:06 |
| mordred | now if only debian would revert the dist-packages thing... | 23:06 |
| clarkb | fungi: pabelanger AJaeger looks like git0* are running puppet successfully now? | 23:06 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool feature/zuulv3: [WIP] webapp: add optional admin endpoint https://review.openstack.org/531718 | 23:06 |
| * mordred is thrilled wit that fedora change and will need to email beers to someone | 23:06 | |
| clarkb | mordred: your ansible/ansible change looks wrong to me. include: - job is global to the source and not per repo? | 23:07 |
| clarkb | but also you have projects: under untrusted-projects: | 23:08 |
| *** bobh has quit IRC | 23:09 | |
| clarkb | kingbird-dashboard exists in gerrit now https://review.openstack.org/#/admin/projects/openstack/kingbird-dashboard,access | 23:10 |
| *** jaypipes has quit IRC | 23:12 | |
| *** david-lyle has quit IRC | 23:12 | |
| mordred | clarkb: that's defining a list of projects for-which to only load jobs - it's an etry that's basically {'projects': ['ansible/ansible'], 'include': ['jobs']} | 23:12 |
| pabelanger | clarkb: yah, we merged a fix for git servers | 23:12 |
| mordred | dmsimard: oh - https://review.openstack.org/#/c/504510 to you too | 23:13 |
| clarkb | mordred: it doesn't read well :/ | 23:13 |
| *** david-lyle has joined #openstack-infra | 23:13 | |
| pabelanger | I need to step away for a bit, will catch up on backscroll | 23:14 |
| mordred | corvus: ^^ I'm doing that right, yeah? | 23:14 |
| clarkb | mordred: I think the nested projects is what confuses me | 23:15 |
| clarkb | we list projects at the top level but now there is a third type of project being listed under them | 23:15 |
| clarkb | also does that mean we can't have a project called include? | 23:16 |
| mordred | clarkb: it does not - include is a key in the dict that's in the untrusted-projects list | 23:17 |
| clarkb | mordred: so we type check it? | 23:17 |
| clarkb | I guess in this case we'd hvae to | 23:17 |
| mordred | so - basically, untrusted-projects is a list that contains the string gtest-org/ansible and an anonymous dict that has the keys "include" and "project" | 23:18 |
| mordred | so - basically, untrusted-projects is a list that contains the string gtest-org/ansible and an anonymous dict that has the keys "include" and "projects" | 23:18 |
| mordred | clarkb: yah - there's a common pattern in the zuul config reading that is "the value here can be either a string or a dict" | 23:18 |
| clarkb | ya I'm reading the voluptuous schema for it now | 23:19 |
| *** gcb has quit IRC | 23:20 | |
| mordred | that has led to me look at what the yaml would look like if one of the entries of that list was a list ... it's weird :) | 23:20 |
| corvus | clarkb, mordred: i believe that's correct for an anonymous project group. | 23:21 |
| mordred | http://paste.openstack.org/show/641039/ | 23:21 |
| corvus | which is, unfortunately, undocumented | 23:21 |
| corvus | that's my bad, i will correct it | 23:22 |
| *** sambetts is now known as sambetts|afk | 23:22 | |
| clarkb | corvus: mordred the thing I descirbe appears to be supported as well | 23:22 |
| clarkb | my human readable preference would be for what I describe over the anonymous nested list of projects | 23:23 |
| corvus | clarkb: what's the thing you describe? | 23:23 |
| clarkb | corvus: set the includes per project entry in untrusted-projects rather than have a nested list of projects | 23:23 |
| clarkb | project_dict in the schema | 23:24 |
| corvus | clarkb: yes, that works too. | 23:24 |
| mordred | clarkb: but what about when we have 50 github projects and we only want to load jobs from each of them? | 23:24 |
| clarkb | (which is how I thought it had to be done but schema says both are valid) | 23:24 |
| clarkb | mordred: we don't have that problem yet and I think it will likely still be more readable in the non anonymous dict case | 23:25 |
| clarkb | at least for humans | 23:25 |
| mordred | maybe it would make more sense when there are more than one and for now doing it as an include on the one project itself would be better | 23:25 |
| corvus | mordred: that sounds reasonable. | 23:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add ansible/ansible to zuul https://review.openstack.org/504510 | 23:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Run shade functional tests on ansible OpenStack PRs https://review.openstack.org/531916 | 23:26 |
| mordred | clarkb, corvus: ^^ | 23:26 |
| clarkb | corvus: but ya I Think greatest source of confusion is likely the docs only doing the project_dict case and not the anonymous dict case | 23:27 |
| corvus | (also, ftr, "include: job" works as well) | 23:27 |
| clarkb | (so fixing that ++) | 23:27 |
| *** dciabrin has joined #openstack-infra | 23:28 | |
| mordred | corvus: should I collapse it to include: job? since this is our first example in the config file for later cargo-culting? | 23:28 |
| corvus | mordred: maybe so? that feels like a common case where brevity would be nice. | 23:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add ansible/ansible to zuul https://review.openstack.org/504510 | 23:30 |
| mordred | k. there we go | 23:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Run shade functional tests on ansible OpenStack PRs https://review.openstack.org/531916 | 23:31 |
| mordred | corvus: wanna do the honors? :) | 23:32 |
| corvus | done :) | 23:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies https://review.openstack.org/530806 | 23:36 |
| openstackgerrit | Merged openstack-infra/project-config master: Add ansible/ansible to zuul https://review.openstack.org/504510 | 23:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add job for testing against devel of ansible https://review.openstack.org/531934 | 23:43 |
| mordred | corvus, clarkb: ^^ there's a similar patch for zuul itself | 23:43 |
| *** felipemonteiro_ has quit IRC | 23:46 | |
| *** olaph has quit IRC | 23:49 | |
| *** dingyichen has joined #openstack-infra | 23:52 | |
| *** olaph has joined #openstack-infra | 23:53 | |
| *** tosky has quit IRC | 23:56 | |
| *** stakeda has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!