| *** AhmadM has quit IRC | 00:01 | |
| pabelanger | clarkb: yah, would agree | 00:01 |
|---|---|---|
| pabelanger | I guess we try restarting zuul services tomorrow? | 00:03 |
| *** david-lyle has joined #openstack-infra | 00:03 | |
| clarkb | ya in theory demand will be way down and smcginnis won't be waiting on release jobs | 00:04 |
| pabelanger | wfm | 00:04 |
| smcginnis | ++ | 00:05 |
| *** harlowja has joined #openstack-infra | 00:05 | |
| *** dklyle has quit IRC | 00:06 | |
| *** linkmark has quit IRC | 00:07 | |
| *** bobh has quit IRC | 00:08 | |
| *** caphrim007_ has quit IRC | 00:12 | |
| *** dklyle has joined #openstack-infra | 00:12 | |
| *** dklyle_ has joined #openstack-infra | 00:14 | |
| *** david-lyle has quit IRC | 00:14 | |
| *** jamesde__ has quit IRC | 00:15 | |
| *** rcarrill1 has joined #openstack-infra | 00:15 | |
| *** dklyle has quit IRC | 00:17 | |
| *** jamesden_ has joined #openstack-infra | 00:17 | |
| *** rcarrillocruz has quit IRC | 00:17 | |
| *** caphrim007 has joined #openstack-infra | 00:19 | |
| *** gyee has quit IRC | 00:23 | |
| *** caphrim007 has quit IRC | 00:24 | |
| openstackgerrit | Austin Sun proposed openstack-infra/project-config master: Add gerritbot for StarlingX (openstack/stx-*) projects to #starlingx IRC channel https://review.openstack.org/585919 | 00:32 |
| *** felipemonteiro has quit IRC | 00:55 | |
| *** slaweq has joined #openstack-infra | 01:05 | |
| *** slaweq has quit IRC | 01:10 | |
| *** harlowja has quit IRC | 01:17 | |
| *** xarses_ has quit IRC | 01:17 | |
| *** bobh has joined #openstack-infra | 01:46 | |
| *** yamahata has quit IRC | 01:48 | |
| *** dklyle has joined #openstack-infra | 01:49 | |
| *** dklyle_ has quit IRC | 01:50 | |
| *** bobh has quit IRC | 01:52 | |
| *** larainema has joined #openstack-infra | 01:58 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for live migration unbound vif bug 1783917 https://review.openstack.org/586389 | 01:59 |
| openstack | bug 1783917 in OpenStack Compute (nova) "live migration raises NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" [Undecided,New] https://launchpad.net/bugs/1783917 | 01:59 |
| *** david-lyle has joined #openstack-infra | 02:01 | |
| *** dingyichen has joined #openstack-infra | 02:02 | |
| *** dklyle has quit IRC | 02:02 | |
| *** david-lyle has quit IRC | 02:03 | |
| *** dklyle_ has joined #openstack-infra | 02:04 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for live migration unbound vif bug 1783917 https://review.openstack.org/586389 | 02:15 |
| openstack | bug 1783917 in OpenStack Compute (nova) "live migration fails with NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" [High,Confirmed] https://launchpad.net/bugs/1783917 | 02:15 |
| *** gongysh has joined #openstack-infra | 02:22 | |
| *** psachin`` has joined #openstack-infra | 02:23 | |
| *** psachin`` has quit IRC | 02:33 | |
| *** chinna100 has quit IRC | 02:35 | |
| *** myoung has quit IRC | 02:39 | |
| *** caphrim007 has joined #openstack-infra | 02:45 | |
| *** caphrim007 has quit IRC | 02:50 | |
| *** rosmaita has quit IRC | 02:55 | |
| *** armax has quit IRC | 03:00 | |
| *** armax has joined #openstack-infra | 03:04 | |
| *** yamamoto has joined #openstack-infra | 03:09 | |
| *** armax has quit IRC | 03:12 | |
| *** armax has joined #openstack-infra | 03:16 | |
| *** dave-mccowan has quit IRC | 03:24 | |
| *** armax has quit IRC | 03:29 | |
| *** eernst has quit IRC | 03:29 | |
| *** annp has quit IRC | 03:31 | |
| *** xarses_ has joined #openstack-infra | 03:32 | |
| *** annp has joined #openstack-infra | 03:32 | |
| *** links has joined #openstack-infra | 03:52 | |
| *** gongysh has quit IRC | 03:52 | |
| *** yamahata has joined #openstack-infra | 03:54 | |
| *** diablo_rojo has quit IRC | 03:58 | |
| *** mriedem has quit IRC | 04:04 | |
| *** mschuppert has joined #openstack-infra | 04:06 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** slaweq has quit IRC | 04:16 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add a dequeue command to zuul client https://review.openstack.org/95035 | 04:21 |
| *** pcaruana has joined #openstack-infra | 04:28 | |
| *** pcaruana has quit IRC | 04:30 | |
| *** diablo_rojo has joined #openstack-infra | 04:46 | |
| *** gongysh has joined #openstack-infra | 04:50 | |
| *** kjackal has joined #openstack-infra | 04:58 | |
| *** yamamoto has quit IRC | 04:59 | |
| *** yamamoto has joined #openstack-infra | 05:01 | |
| *** diablo_rojo has quit IRC | 05:01 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add a dequeue command to zuul client https://review.openstack.org/95035 | 05:03 |
| *** n-saito has joined #openstack-infra | 05:03 | |
| *** slaweq has joined #openstack-infra | 05:13 | |
| *** cshastri has joined #openstack-infra | 05:14 | |
| *** pgadiya has joined #openstack-infra | 05:16 | |
| *** pgadiya has quit IRC | 05:16 | |
| *** Bhujay has joined #openstack-infra | 05:17 | |
| *** slaweq has quit IRC | 05:17 | |
| *** Bhujay has quit IRC | 05:21 | |
| *** kjackal has quit IRC | 05:39 | |
| *** quiquell has joined #openstack-infra | 05:41 | |
| *** diablo_rojo has joined #openstack-infra | 05:46 | |
| *** annp has quit IRC | 05:51 | |
| *** annp has joined #openstack-infra | 05:52 | |
| *** AJaeger has quit IRC | 05:52 | |
| *** zigo_ has joined #openstack-infra | 05:53 | |
| *** zigo has quit IRC | 05:53 | |
| *** AJaeger has joined #openstack-infra | 05:55 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add trigger-readthedocs-webhook job https://review.openstack.org/583449 | 06:05 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Use webhook update for docs-on-readthedocs https://review.openstack.org/583834 | 06:06 |
| *** hashar has joined #openstack-infra | 06:15 | |
| *** alexchadin has joined #openstack-infra | 06:15 | |
| *** mgoddard has joined #openstack-infra | 06:41 | |
| *** pguimaraes has joined #openstack-infra | 06:47 | |
| *** mgoddard has quit IRC | 06:50 | |
| *** tesseract has joined #openstack-infra | 06:52 | |
| *** kjackal has joined #openstack-infra | 06:54 | |
| *** dhajare has joined #openstack-infra | 06:56 | |
| *** rcernin has quit IRC | 07:00 | |
| *** ccamacho has joined #openstack-infra | 07:20 | |
| *** dtantsur|afk is now known as dtantsur | 07:21 | |
| *** zoli|gone is now known as zoli | 07:30 | |
| *** shardy has joined #openstack-infra | 07:37 | |
| *** jtomasek has joined #openstack-infra | 07:48 | |
| *** jtomasek has quit IRC | 07:48 | |
| *** jtomasek has joined #openstack-infra | 07:49 | |
| *** alexchadin has quit IRC | 07:52 | |
| *** rpittau has quit IRC | 07:57 | |
| *** rpittau has joined #openstack-infra | 07:57 | |
| *** pblaho has joined #openstack-infra | 07:58 | |
| *** dtantsur is now known as dtantsur|bbl | 08:00 | |
| *** alexchadin has joined #openstack-infra | 08:05 | |
| *** jpich has joined #openstack-infra | 08:06 | |
| *** jamesmcarthur has joined #openstack-infra | 08:06 | |
| *** jamesmcarthur has quit IRC | 08:11 | |
| *** mgoddard has joined #openstack-infra | 08:12 | |
| *** roman_g has quit IRC | 08:13 | |
| tinwood | morning | 08:15 |
| tinwood | I'm having a problem with a nova-lxd backport to stable/queens; the issue is that although the branch it is proposed to is stable/queens in the nova-lxd project, it is actually testing against the master in the nova project; I'm sure I just have to twiddle some bits somewhere in the various project configs, but I'm not sure where to start. The issues are the tests openstack-tox-{py27|py35}. I tried an override in .zuul. | 08:17 |
| tinwood | conf, but that didn't work? Any pointers, please? | 08:17 |
| AJaeger | tinwood: have a change up to check log files? | 08:19 |
| *** bauzas is now known as PapaOurs | 08:19 | |
| tinwood | AJaeger, sure: https://review.openstack.org/#/c/584900/ | 08:19 |
| tinwood | AJaeger, I added openstack-tox-py27-stable-queens as a test; that's coming out; I was trying to see if an override would work at all. | 08:20 |
| *** e0ne has joined #openstack-infra | 08:22 | |
| kashyap | Hey folks, I'm hitting a "POST_FAILURE" state for the 'nova-live-migration' CI job (for this patch: https://review.openstack.org/#/c/567258/); is this a Zuul problem? | 08:22 |
| AJaeger | tinwood: why do you think it checks out master? | 08:22 |
| tinwood | AJaeger, I'll find the relevant line I think is doing it: | 08:22 |
| AJaeger | kashyap: check the log files, please | 08:22 |
| kashyap | When I look in the log (http://logs.openstack.org/58/567258/10/check/nova-live-migration/1998129/job-output.txt.gz) | 08:22 |
| kashyap | 2018-07-26 18:18:54.522168 | primary | ERROR | 08:22 |
| kashyap | 2018-07-26 18:18:54.541519 | primary | { | 08:22 |
| kashyap | 2018-07-26 18:18:54.541725 | primary | "msg": "SSH Error: data could not be sent to remote host \"147.75.38.160\". Make sure this host can be reached over ssh", | 08:22 |
| *** diablo_rojo has quit IRC | 08:22 | |
| kashyap | 2018-07-26 18:18:54.541843 | primary | "unreachable": true | 08:22 |
| kashyap | 2018-07-26 18:18:54.541953 | primary | } | 08:23 |
| kashyap | AJaeger: Yeah, the above is what the job-output seem to say | 08:23 |
| AJaeger | kashyap: please no multi-line pastes - you can link to that line... | 08:23 |
| tinwood | AJaeger, in the py27 tox run: | 08:23 |
| tinwood | 2018-07-26 16:17:15.870344 | ubuntu-xenial | from microversion_parse import middleware as mp_middleware | 08:23 |
| tinwood | 2018-07-26 16:17:15.870445 | ubuntu-xenial | ImportError: cannot import name middleware | 08:23 |
| AJaeger | kashyap: could have been a cloud problems, I would recheck | 08:23 |
| *** derekh has joined #openstack-infra | 08:23 | |
| kashyap | AJaeger: Ah, I too am averse to multi-line paste. But anything around 5 lines or less OK (if done only once :-)) | 08:24 |
| tinwood | AJaeger, and I think that middleware appeared in master; it doesn't appear to be in stable/queens. (or I'm not checking correctly). | 08:24 |
| kashyap | AJaeger: Thanks! I'll do the 'recheck' | 08:24 |
| AJaeger | tinwood: http://logs.openstack.org/00/584900/5/check/openstack-tox-py27/a9cc494/job-output.txt.gz#_2018-07-26_16_12_23_456162 has stable/queens in it | 08:24 |
| tinwood | AJaeger, yes, for nova-lxd. But "nova" is still master; so it fails (I think!) | 08:25 |
| AJaeger | tinwood: ah, nova! let me check... | 08:25 |
| AJaeger | tinwood: http://logs.openstack.org/00/584900/5/check/openstack-tox-py27/a9cc494/job-output.txt.gz#_2018-07-26_16_12_23_030838 does not check out nova at all | 08:25 |
| AJaeger | tinwood: you check it out in tox.ini yourself, that's your problem. You should use required-projects and follow the work we done for horizon and neutron to be able to install them from source | 08:26 |
| tinwood | AJaeger, I'm confused then. How can nova-lxd be tested at all without nova? (it doesn't mock everything out and the import error is in nova) | 08:27 |
| AJaeger | tinwood: check how the networking projects and the -dashboard ones are setup. You need the same setup. | 08:27 |
| tinwood | AJaeger, okay, re: neutron and horizon. I'll do some research on that; thanks for the pointers! :) | 08:27 |
| AJaeger | tinwood: your tox.ini has egit+https://github.com/openstack/nova.git#egg=nova - that is master | 08:27 |
| * tinwood sometimes only sees the wood, and didn't notice the trees. | 08:28 | |
| AJaeger | if you add to the jobs a "required-projects: openstack/nova", then we would check out the correct branch for you ;) | 08:28 |
| tinwood | AJaeger, thanks -- I will try that. | 08:28 |
| AJaeger | tinwood: sorry, no more time to guide you through the rest - if you need more help, best ask rest of team later. | 08:28 |
| tinwood | AJaeger, no problem; I'll look at the work in the other projects you mentioned and see where I get to. Thanks very much. | 08:29 |
| *** mgoddard has quit IRC | 08:34 | |
| *** jaosorior has quit IRC | 08:38 | |
| *** pbourke has quit IRC | 08:41 | |
| *** roman_g has joined #openstack-infra | 08:41 | |
| *** vivsoni has quit IRC | 08:41 | |
| *** pbourke has joined #openstack-infra | 08:42 | |
| *** lifeless has quit IRC | 08:54 | |
| *** jaosorior has joined #openstack-infra | 08:58 | |
| *** vivsoni has joined #openstack-infra | 09:05 | |
| *** d0ugal has joined #openstack-infra | 09:08 | |
| *** lifeless has joined #openstack-infra | 09:11 | |
| *** caphrim007 has joined #openstack-infra | 09:14 | |
| *** caphrim007 has quit IRC | 09:18 | |
| *** quiquell has quit IRC | 09:27 | |
| *** quiquell has joined #openstack-infra | 09:27 | |
| *** zoli is now known as zoli|lunch | 09:32 | |
| *** rcarrill1 is now known as rcarrillocruz | 09:41 | |
| *** andymccr- has joined #openstack-infra | 09:47 | |
| *** jaosorior has quit IRC | 09:49 | |
| *** andymccr_ has quit IRC | 09:50 | |
| *** johnthetubaguy has quit IRC | 09:52 | |
| *** stakeda has quit IRC | 10:03 | |
| *** andymccr has quit IRC | 10:04 | |
| *** andymccr- is now known as andymccr | 10:05 | |
| *** sshnaidm|afk has quit IRC | 10:09 | |
| *** alexchadin has quit IRC | 10:26 | |
| *** yamamoto has quit IRC | 10:30 | |
| *** alexchadin has joined #openstack-infra | 10:39 | |
| *** kjackal has quit IRC | 10:43 | |
| *** gongysh has quit IRC | 10:47 | |
| *** dtantsur|bbl is now known as dtantsur | 10:49 | |
| *** zoli|lunch is now known as zoli | 11:02 | |
| *** cshastri has quit IRC | 11:02 | |
| *** dhill_ has quit IRC | 11:03 | |
| *** dave-mccowan has joined #openstack-infra | 11:06 | |
| *** larainema has quit IRC | 11:15 | |
| *** cshastri has joined #openstack-infra | 11:17 | |
| *** yamamoto has joined #openstack-infra | 11:18 | |
| *** quiquell has quit IRC | 11:21 | |
| *** vivsoni has quit IRC | 11:23 | |
| *** quiquell has joined #openstack-infra | 11:24 | |
| *** jamesde__ has joined #openstack-infra | 11:31 | |
| *** jamesden_ has quit IRC | 11:32 | |
| *** jamesde__ has quit IRC | 11:34 | |
| *** zul has quit IRC | 11:34 | |
| *** kjackal has joined #openstack-infra | 11:34 | |
| *** alexchadin has quit IRC | 11:34 | |
| *** alexchadin has joined #openstack-infra | 11:35 | |
| *** alexchadin has quit IRC | 11:35 | |
| *** alexchadin has joined #openstack-infra | 11:36 | |
| *** alexchadin has quit IRC | 11:36 | |
| *** alexchadin has joined #openstack-infra | 11:36 | |
| *** alexchadin has quit IRC | 11:37 | |
| *** quiquell has quit IRC | 11:40 | |
| *** d0ugal has quit IRC | 11:41 | |
| *** quiquell has joined #openstack-infra | 11:42 | |
| *** quiquell has quit IRC | 11:43 | |
| *** rh-jelabarre has joined #openstack-infra | 11:47 | |
| *** n-saito has quit IRC | 11:48 | |
| *** yamamoto has quit IRC | 11:54 | |
| *** tpsilva has joined #openstack-infra | 11:54 | |
| *** rfolco|off is now known as rfolco|ruck | 11:55 | |
| *** yamamoto has joined #openstack-infra | 11:59 | |
| *** pawelzny has joined #openstack-infra | 12:00 | |
| *** linkmark has joined #openstack-infra | 12:03 | |
| *** pawelzny has left #openstack-infra | 12:05 | |
| *** boden has joined #openstack-infra | 12:07 | |
| *** briancurtin has quit IRC | 12:07 | |
| *** sshnaidm has joined #openstack-infra | 12:08 | |
| *** sshnaidm is now known as sshnaidm|off | 12:08 | |
| *** caphrim007 has joined #openstack-infra | 12:14 | |
| *** dhill_ has joined #openstack-infra | 12:15 | |
| *** alexchadin has joined #openstack-infra | 12:16 | |
| *** edmondsw has joined #openstack-infra | 12:17 | |
| *** johnthetubaguy has joined #openstack-infra | 12:17 | |
| *** caphrim007 has quit IRC | 12:18 | |
| *** alexchadin has quit IRC | 12:20 | |
| *** zul has joined #openstack-infra | 12:22 | |
| *** armaan has joined #openstack-infra | 12:22 | |
| openstackgerrit | Tobias Urdin proposed openstack-infra/system-config master: Mirror puppet5 for Ubuntu Bionic https://review.openstack.org/586526 | 12:22 |
| *** jcoufal has joined #openstack-infra | 12:25 | |
| *** wolverineav has joined #openstack-infra | 12:26 | |
| *** annp has quit IRC | 12:27 | |
| *** zul has quit IRC | 12:28 | |
| *** zul has joined #openstack-infra | 12:30 | |
| *** alexchadin has joined #openstack-infra | 12:33 | |
| *** pblaho has quit IRC | 12:33 | |
| *** diablo_rojo has joined #openstack-infra | 12:33 | |
| *** trown|outtypewww is now known as trown | 12:34 | |
| *** mriedem has joined #openstack-infra | 12:35 | |
| *** zul has quit IRC | 12:36 | |
| *** agopi has quit IRC | 12:37 | |
| *** zul has joined #openstack-infra | 12:37 | |
| *** armaan has quit IRC | 12:41 | |
| *** yamamoto has quit IRC | 12:42 | |
| *** zul has quit IRC | 12:44 | |
| *** armaan has joined #openstack-infra | 12:45 | |
| *** zul has joined #openstack-infra | 12:45 | |
| *** rlandy has quit IRC | 12:48 | |
| *** rlandy has joined #openstack-infra | 12:48 | |
| *** armaan has quit IRC | 12:49 | |
| *** armaan has joined #openstack-infra | 12:50 | |
| *** zul has quit IRC | 12:51 | |
| *** zul has joined #openstack-infra | 12:53 | |
| *** armaan has quit IRC | 12:54 | |
| *** zul has quit IRC | 12:58 | |
| *** yamamoto has joined #openstack-infra | 12:59 | |
| *** zul has joined #openstack-infra | 13:00 | |
| *** rcarrill1 has joined #openstack-infra | 13:01 | |
| *** hamerins has joined #openstack-infra | 13:01 | |
| *** rcarrill2 has joined #openstack-infra | 13:03 | |
| *** rpioso|afk is now known as rpioso | 13:03 | |
| *** rcarrillocruz has quit IRC | 13:03 | |
| *** sthussey has joined #openstack-infra | 13:04 | |
| *** rcarrill2 is now known as rcarrillocruz | 13:05 | |
| *** rcarrill1 has quit IRC | 13:06 | |
| *** rcarrill1 has joined #openstack-infra | 13:08 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: scheduler: return project_canonical in status page https://review.openstack.org/582451 | 13:08 |
| *** rcarrillocruz has quit IRC | 13:10 | |
| *** zul has quit IRC | 13:14 | |
| *** zul has joined #openstack-infra | 13:15 | |
| AJaeger | config-core, could you go over the review queues, please? We have collected a few changes... Some older changes with already one +2 are: https://review.openstack.org/#/c/584366/ https://review.openstack.org/#/c/581096/ https://review.openstack.org/#/c/581064/ https://review.openstack.org/585493 | 13:16 |
| *** jcoufal has quit IRC | 13:16 | |
| *** mandre is now known as mandre_away | 13:18 | |
| *** rcarrillocruz has joined #openstack-infra | 13:19 | |
| pabelanger | +3 | 13:19 |
| *** jcoufal has joined #openstack-infra | 13:20 | |
| *** zul has quit IRC | 13:21 | |
| *** rcarrill1 has quit IRC | 13:21 | |
| *** rcarrill1 has joined #openstack-infra | 13:22 | |
| *** agopi has joined #openstack-infra | 13:22 | |
| *** zul has joined #openstack-infra | 13:23 | |
| *** rcarrillocruz has quit IRC | 13:25 | |
| *** pblaho has joined #openstack-infra | 13:25 | |
| *** pblaho has quit IRC | 13:25 | |
| *** pblaho has joined #openstack-infra | 13:25 | |
| AJaeger | thanks, pabelanger | 13:25 |
| *** mdrabe has joined #openstack-infra | 13:26 | |
| *** zul has quit IRC | 13:28 | |
| *** caphrim007 has joined #openstack-infra | 13:29 | |
| *** caphrim007_ has joined #openstack-infra | 13:29 | |
| *** rosmaita has joined #openstack-infra | 13:30 | |
| *** zul has joined #openstack-infra | 13:30 | |
| *** auristor has quit IRC | 13:30 | |
| *** rcarrill1 is now known as rcarrillocruz | 13:31 | |
| *** jistr is now known as jistr|mtg | 13:32 | |
| *** kgiusti has joined #openstack-infra | 13:32 | |
| *** caphrim007 has quit IRC | 13:33 | |
| *** myoung has joined #openstack-infra | 13:35 | |
| *** caphrim007_ is now known as caphrim007 | 13:35 | |
| *** rcarrill1 has joined #openstack-infra | 13:35 | |
| *** quiquell has joined #openstack-infra | 13:35 | |
| AJaeger | config-core, a few more changes for review https://review.openstack.org/582317 https://review.openstack.org/#/c/585966/ https://review.openstack.org/#/c/586280/ https://review.openstack.org/#/c/585919/ | 13:36 |
| *** alexchadin has quit IRC | 13:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Remove Zuul v2 content https://review.openstack.org/586549 | 13:36 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove sahara-extra legacy jobs, moving in-tree https://review.openstack.org/581096 | 13:37 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-tempest-dsvm-nova-os-vif https://review.openstack.org/585493 | 13:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add a yarn role https://review.openstack.org/586550 | 13:37 |
| *** auristor has joined #openstack-infra | 13:37 | |
| *** rcarrillocruz has quit IRC | 13:37 | |
| AJaeger | mordred: shall I +2A https://review.openstack.org/#/c/511827/ and https://review.openstack.org/#/c/511823/ ? Or do you wait for the rest first? | 13:38 |
| mordred | AJaeger: I think they're both safe to land - let's go ahead | 13:39 |
| *** yamamoto has quit IRC | 13:39 | |
| AJaeger | done - +3 | 13:39 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Add instructions for deploying zuul with openSUSE https://review.openstack.org/581255 | 13:41 |
| *** hamerins has quit IRC | 13:41 | |
| *** rcarrillocruz has joined #openstack-infra | 13:41 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Change npm reference to yarn https://review.openstack.org/586552 | 13:41 |
| *** alexchadin has joined #openstack-infra | 13:42 | |
| *** zul has quit IRC | 13:43 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Drop references to system-required template https://review.openstack.org/579180 | 13:43 |
| *** rcarrill1 has quit IRC | 13:43 | |
| *** hamerins has joined #openstack-infra | 13:45 | |
| *** yamamoto has joined #openstack-infra | 13:45 | |
| *** rcarrill1 has joined #openstack-infra | 13:48 | |
| *** alexchadin has quit IRC | 13:49 | |
| *** rcarrillocruz has quit IRC | 13:50 | |
| *** rcarrillocruz has joined #openstack-infra | 13:51 | |
| *** rcarrill1 has quit IRC | 13:54 | |
| *** ianychoi has joined #openstack-infra | 13:56 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add pbrx-release team for pbrx https://review.openstack.org/584366 | 13:58 |
| openstackgerrit | Merged openstack-infra/project-config master: import networking-cisco jobs into net-cisco tree https://review.openstack.org/581064 | 13:58 |
| *** zul has joined #openstack-infra | 13:58 | |
| *** ianychoi_ has quit IRC | 14:00 | |
| openstackgerrit | Merged openstack-infra/system-config master: Replace mirror01.us-west-1.packethost.openstack.org https://review.openstack.org/586337 | 14:06 |
| *** links has quit IRC | 14:10 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy scenario multinode job for cinder https://review.openstack.org/583853 | 14:11 |
| *** jamesmcarthur has joined #openstack-infra | 14:12 | |
| *** pblaho has quit IRC | 14:14 | |
| *** eharney has joined #openstack-infra | 14:15 | |
| *** jlvacation is now known as jlvillal | 14:18 | |
| *** shardy has quit IRC | 14:18 | |
| *** r-daneel has joined #openstack-infra | 14:18 | |
| openstackgerrit | Merged openstack-infra/project-config master: Set zuul_output_dir in site-variables https://review.openstack.org/511827 | 14:18 |
| *** felipemonteiro has joined #openstack-infra | 14:19 | |
| *** dhajare has quit IRC | 14:20 | |
| mordred | pabelanger: lookie there ^^ ! | 14:21 |
| mordred | pabelanger: progress! | 14:21 |
| pabelanger | yay | 14:22 |
| *** diablo_rojo has quit IRC | 14:22 | |
| pabelanger | mordred: I think once we restart zuul today (assuming today) I can finished up some testing that depends on rsync patch. Then will reply again to ML with some notes | 14:22 |
| * mordred does a little dance | 14:24 | |
| *** alexchadin has joined #openstack-infra | 14:26 | |
| *** mdrabe has quit IRC | 14:26 | |
| *** links has joined #openstack-infra | 14:29 | |
| *** alexchadin has quit IRC | 14:30 | |
| *** ramishra has quit IRC | 14:31 | |
| pabelanger | clarkb: email sent to packethost to debug building nodes again, only 3 nodes in use ATM | 14:37 |
| pabelanger | http://grafana.openstack.org/d/U462abNik/nodepool-packethost | 14:37 |
| *** jistr|mtg is now known as jistr | 14:39 | |
| *** shardy has joined #openstack-infra | 14:40 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:42 | |
| *** felipemonteiro has quit IRC | 14:46 | |
| *** diablo_rojo has joined #openstack-infra | 14:46 | |
| *** dhajare has joined #openstack-infra | 14:46 | |
| *** ramishra has joined #openstack-infra | 14:49 | |
| *** efried is now known as fried_rice | 14:49 | |
| Adri2000 | hello, would appreciate a quick review for this planet.o.o addition https://review.openstack.org/#/c/586254/ :) | 14:51 |
| *** links has quit IRC | 15:02 | |
| mordred | pabelanger, clarkb: http://logs.openstack.org/62/586262/4/check/openstack-tox-lower-constraints/535b826/job-output.txt.gz#_2018-07-27_13_30_48_575676 | 15:03 |
| mordred | pabelanger, clarkb: also - mirror issues there | 15:03 |
| *** alexchadin has joined #openstack-infra | 15:04 | |
| *** mdrabe has joined #openstack-infra | 15:05 | |
| pabelanger | ugh | 15:05 |
| pabelanger | let me see if server is off again | 15:06 |
| pabelanger | yah, cannot SSH again | 15:07 |
| pabelanger | this is new mirror too | 15:07 |
| pabelanger | I'll reboot via API again | 15:07 |
| *** r-daneel_ has joined #openstack-infra | 15:08 | |
| *** r-daneel has quit IRC | 15:09 | |
| *** r-daneel_ is now known as r-daneel | 15:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route https://review.openstack.org/550978 | 15:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/projects and /{tenant}/project/{project} routes https://review.openstack.org/550979 | 15:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 15:12 |
| mordred | pabelanger: joy | 15:12 |
| *** ramishra has quit IRC | 15:13 | |
| pabelanger | mordred: clarkb: is back, but maybe we should disable packethost until we resolve mirror issues | 15:13 |
| mordred | yeah | 15:13 |
| mordred | seems reasonable | 15:13 |
| *** ginopc has joined #openstack-infra | 15:13 | |
| mnaser | fwiw we have an expected insane spike today of infra, but after that i should be able to restore our stuff and be able to actively troubleshoot stuf together | 15:13 |
| *** ginopc has quit IRC | 15:14 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Disable packethost in nodepool https://review.openstack.org/586601 | 15:14 |
| pabelanger | mnaser: clarkb: ^ | 15:15 |
| pabelanger | will follow up again with john on email | 15:15 |
| *** cshastri has quit IRC | 15:15 | |
| *** e0ne has quit IRC | 15:16 | |
| pabelanger | mnaser: yay | 15:16 |
| smcginnis | Are the mirror issues mentioned above the reason for a bunch of job failures with a finger:// link as the result with RETRY_LIMIT status? | 15:21 |
| mordred | smcginnis: probably? | 15:22 |
| *** eernst has joined #openstack-infra | 15:23 | |
| mordred | smcginnis: got an example? | 15:23 |
| pabelanger | Hmm | 15:23 |
| pabelanger | looking | 15:23 |
| smcginnis | Take your pick? http://zuul.openstack.org/ | 15:24 |
| smcginnis | :) | 15:24 |
| pabelanger | nope | 15:24 |
| pabelanger | we have broke jobs | 15:24 |
| *** alexchadin has quit IRC | 15:24 | |
| pabelanger | 2018-07-27 15:06:23,037 DEBUG zuul.AnsibleJob: [build: 635c8ce2eaf34e21a490fea197fed485] Ansible output: b"The error appears to have been in '/etc/zuul/site-variables.yaml': line 11, column 1, but may" | 15:24 |
| quiquell | Hello | 15:25 |
| quiquell | I see some retry_limit at zuul.o.o at the gates, are they important ? | 15:25 |
| quiquell | Here for example 586499,1 | 15:25 |
| mordred | uhoh | 15:26 |
| pabelanger | 1 sec | 15:26 |
| pabelanger | have patch | 15:26 |
| mordred | pabelanger: we're gonna need to force-merge it I'm betting | 15:26 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix site-variables typo https://review.openstack.org/586602 | 15:26 |
| pabelanger | yes | 15:26 |
| pabelanger | mordred: ^ | 15:26 |
| pabelanger | then kick puppet | 15:26 |
| mordred | pabelanger: on it | 15:26 |
| *** eernst has quit IRC | 15:27 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix site-variables typo https://review.openstack.org/586602 | 15:27 |
| pabelanger | now we need to kick.sh all zuul-executors | 15:27 |
| pabelanger | or wait | 15:27 |
| mordred | pabelanger: you on that or want me to? | 15:27 |
| smcginnis | Surprised the linter jobs let that through in the first palce. | 15:27 |
| smcginnis | *place | 15:27 |
| mordred | I believe we might not be linting that file - which is a clear mistake | 15:28 |
| pabelanger | mordred: i can | 15:28 |
| * mordred works on getting linters applied to that file to prevent in future | 15:28 | |
| *** eernst has joined #openstack-infra | 15:28 | |
| pabelanger | kicking now | 15:29 |
| prometheanfire | gate sad? https://review.openstack.org/586553 | 15:29 |
| pabelanger | yes, fixing now | 15:29 |
| *** annp has joined #openstack-infra | 15:30 | |
| prometheanfire | thanks | 15:30 |
| prometheanfire | rechecks needed? | 15:30 |
| prometheanfire | guess bot will announce if so | 15:30 |
| pabelanger | yah, we'll announce | 15:31 |
| pabelanger | once fixed | 15:31 |
| smcginnis | Might be an opportune time to do that restart now I suppose. | 15:31 |
| mordred | how about: #status alert A zuul config error slipped through and caused a pile of job failures. A fix has been applied, but jobs showing finger urls and RETRY_LIMIT will need to be rechecked. | 15:32 |
| smcginnis | mordred: That looks good to me. | 15:33 |
| mordred | smcginnis: not a terrible idea ... but I don't know if we want to conflate/confound things? | 15:33 |
| pabelanger | kick.sh still running, a little slow due to executors load | 15:33 |
| mordred | infra-root: since we just bombed out a ton of jobs for people, should we take this opportunity to restart zuul? | 15:33 |
| quiquell | mordred: That's why we see somw retry_limit at the gates ? | 15:34 |
| mordred | quiquell: yup | 15:34 |
| quiquell | Damn ok | 15:34 |
| mordred | how about: #status alert A zuul config error slipped through and caused a pile of job failures with retry_limit - a fix is being applied and should be back up in a few minutes | 15:34 |
| *** kashyap has left #openstack-infra | 15:34 | |
| *** andymccr has quit IRC | 15:34 | |
| mordred | now - and we can follow up with a fix announcement when kick is done? | 15:34 |
| pabelanger | lets wait, load is really high on executor and puppet is running slow | 15:35 |
| smcginnis | That might be good to head off all the questions that will be coming this way. | 15:35 |
| mordred | #status alert A zuul config error slipped through and caused a pile of job failures with retry_limit - a fix is being applied and should be back up in a few minutes | 15:35 |
| openstackstatus | mordred: sending alert | 15:35 |
| *** andymccr has joined #openstack-infra | 15:35 | |
| *** quiquell is now known as quique|luckyluck | 15:36 | |
| pabelanger | we might have to stop zuul-executors for puppet to run, ansible-playbooks launching a lot of builds | 15:36 |
| *** quique|luckyluck is now known as quiquell | 15:37 | |
| *** xinliang3 has joined #openstack-infra | 15:37 | |
| *** xinliang has quit IRC | 15:37 | |
| -openstackstatus- NOTICE: A zuul config error slipped through and caused a pile of job failures with retry_limit - a fix is being applied and should be back up in a few minutes | 15:38 | |
| *** ChanServ changes topic to "A zuul config error slipped through and caused a pile of job failures with retry_limit - a fix is being applied and should be back up in a few minutes" | 15:38 | |
| * clarkb catches up | 15:38 | |
| mordred | clarkb: tl;dr - we don't lint the site-variables file and landed a variable addition with a syntax error which is now causing all zuul jobs to fail | 15:39 |
| *** quiquell has quit IRC | 15:39 | |
| *** hongbin_ has joined #openstack-infra | 15:39 | |
| corvus | why did we change site vars? | 15:39 |
| mordred | to add the zuul_output_dir variable | 15:39 |
| pabelanger | okay, puppet has started fixing the files | 15:39 |
| mordred | woot | 15:40 |
| clarkb | pabelanger: re packethost we lost the new mirror node too? | 15:40 |
| mordred | for the logging change series - we missed a closing double quote | 15:40 |
| pabelanger | clarkb: only new mirror node, mirror01 still running | 15:40 |
| clarkb | huh | 15:40 |
| clarkb | well that change is approved for whenever zuul is running jobs again | 15:40 |
| pabelanger | okay, kick.sh is done | 15:41 |
| mordred | woot | 15:41 |
| roman_g | Hello. Please, review/+2/merge: minor thing - adding statusbot to the #airshipit channel - https://review.openstack.org/#/c/581704/ | 15:41 |
| openstackstatus | mordred: finished sending alert | 15:42 |
| *** diablo_rojo has quit IRC | 15:42 | |
| pabelanger | I can see jobs running again | 15:42 |
| mordred | shall we send #status ok ? | 15:43 |
| *** yamamoto has quit IRC | 15:44 | |
| clarkb | mordred: are we also going to restart zuul? | 15:44 |
| clarkb | or shall we wait on that? | 15:44 |
| pabelanger | okay, just seen successful job | 15:45 |
| pabelanger | can send ok, or restart zuul. | 15:45 |
| pabelanger | happy to do either | 15:45 |
| corvus | i think we've spent our allotment of good will for the day. | 15:46 |
| *** mdrabe has quit IRC | 15:46 | |
| clarkb | ya if it is already running jobs then we may want to wait | 15:46 |
| *** mdrabe has joined #openstack-infra | 15:46 | |
| pabelanger | wfm | 15:46 |
| clarkb | the problem was a yaml syntax error in a file we didn't lint? | 15:46 |
| corvus | yep | 15:47 |
| corvus | clarkb: change in question: https://review.openstack.org/511827 | 15:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add linters check to make sure site-variables is yaml https://review.openstack.org/586604 | 15:47 |
| *** yamamoto has joined #openstack-infra | 15:48 | |
| mordred | fwiw, I tried yamllint first - but it got unhappy about a line being too long and I didn't really feel like trying to deal with that | 15:48 |
| *** yamamoto has quit IRC | 15:49 | |
| *** yamamoto has joined #openstack-infra | 15:49 | |
| corvus | ++ | 15:49 |
| *** yamamoto has quit IRC | 15:49 | |
| *** yamamoto has joined #openstack-infra | 15:50 | |
| *** yamamoto has quit IRC | 15:50 | |
| corvus | i actually think that's best. i don't think i want to force everyone to figure out how to break "{{ some really long jinja formatted string }} | with_filters_and_stuff" across 2 lines in yaml and still maintain internal validity. | 15:50 |
| mordred | corvus: me either | 15:50 |
| *** yamamoto has joined #openstack-infra | 15:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add upload-logs-swift role https://review.openstack.org/584541 | 15:51 |
| pabelanger | mordred: send all clear? | 15:51 |
| mordred | how about #status ok Zuul config error has been rectified (and a test added to prevent in the future) Please recheck any RETRY_LIMIT failures from the last half hour or so | 15:52 |
| pabelanger | ++ | 15:52 |
| corvus | wfm | 15:52 |
| clarkb | mordred: should we add pyyaml to the deps list in tox.ini? | 15:52 |
| clarkb | mordred: status message lgtm | 15:52 |
| *** gyee has joined #openstack-infra | 15:52 | |
| mordred | clarkb: probably yes | 15:52 |
| mordred | oh - no - it already has ansible and ansible-lint | 15:53 |
| mordred | so it's got pyyaml | 15:53 |
| *** yamamoto has quit IRC | 15:54 | |
| clarkb | approved (I even double checked it failed agains the old broken file | 15:55 |
| *** rpittau has quit IRC | 15:58 | |
| *** r-daneel_ has joined #openstack-infra | 15:58 | |
| mordred | yay! | 15:58 |
| openstackgerrit | Merged openstack-infra/project-config master: Minor typo fix: duplicate HDD label https://review.openstack.org/586280 | 15:58 |
| openstackgerrit | Merged openstack-infra/project-config master: Add gerritbot for StarlingX (openstack/stx-*) projects to #starlingx IRC channel https://review.openstack.org/585919 | 15:59 |
| *** r-daneel has quit IRC | 16:00 | |
| *** r-daneel_ is now known as r-daneel | 16:00 | |
| *** shardy has quit IRC | 16:01 | |
| *** openstackgerrit has quit IRC | 16:04 | |
| *** jpich has quit IRC | 16:06 | |
| corvus | i have to run some errands, biab. | 16:08 |
| *** zoli is now known as zoli|gone | 16:09 | |
| *** zoli|gone is now known as zoli | 16:09 | |
| *** openstackgerrit has joined #openstack-infra | 16:10 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Fixed UX on ODIC RP initiated logout https://review.openstack.org/586608 | 16:10 |
| *** lbragstad_ is now known as lbragstad | 16:11 | |
| *** links has joined #openstack-infra | 16:14 | |
| *** mriedem is now known as mriedem_away | 16:15 | |
| *** trown is now known as trown|lunch | 16:15 | |
| *** jamesmcarthur has quit IRC | 16:16 | |
| *** links has quit IRC | 16:17 | |
| *** links has joined #openstack-infra | 16:17 | |
| *** 07IADF5PV has joined #openstack-infra | 16:18 | |
| *** armax has joined #openstack-infra | 16:21 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add role to ensure per-node output dirs exist https://review.openstack.org/511823 | 16:21 |
| *** yolanda has quit IRC | 16:22 | |
| *** links has quit IRC | 16:23 | |
| *** gema has quit IRC | 16:26 | |
| *** harlowja has joined #openstack-infra | 16:27 | |
| *** gema has joined #openstack-infra | 16:27 | |
| *** gema has joined #openstack-infra | 16:27 | |
| *** yolanda has joined #openstack-infra | 16:29 | |
| *** derekh has quit IRC | 16:30 | |
| *** tesseract has quit IRC | 16:32 | |
| *** fried_rice is now known as fried_rolls | 16:33 | |
| clarkb | pabelanger: should we manually disable packethost while we wait for zuul to catch up? | 16:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Disable packethost in nodepool https://review.openstack.org/586601 | 16:37 |
| clarkb | pabelanger: ^ nevermind that should get in soon enough | 16:39 |
| *** annp has quit IRC | 16:39 | |
| *** pguimaraes has quit IRC | 16:39 | |
| pabelanger | kk | 16:43 |
| *** john_studarus has joined #openstack-infra | 16:49 | |
| *** felipemonteiro__ has joined #openstack-infra | 16:52 | |
| *** felipemonteiro_ has quit IRC | 16:52 | |
| *** diablo_rojo has joined #openstack-infra | 17:00 | |
| clarkb | corvus: pabelanger mordred thinking ahead here a bit zuul demand tends to fall off around 2-3pm pacific. Thoughts on planning to restart then? It may even happen quicker on a friday | 17:02 |
| *** panda|rover is now known as panda|rover|off | 17:03 | |
| *** trown|lunch is now known as trown | 17:05 | |
| pabelanger | should be around still | 17:06 |
| *** felipemonteiro_ has joined #openstack-infra | 17:06 | |
| pabelanger | 2pmPST a little better | 17:06 |
| *** yamahata has quit IRC | 17:07 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add linters check to make sure site-variables is yaml https://review.openstack.org/586604 | 17:09 |
| *** felipemonteiro__ has quit IRC | 17:10 | |
| *** dtantsur is now known as dtantsur|afk | 17:10 | |
| openstackgerrit | Merged openstack-infra/openstackid master: Fixed UX on ODIC RP initiated logout https://review.openstack.org/586608 | 17:11 |
| *** felipemonteiro__ has joined #openstack-infra | 17:18 | |
| *** felipemonteiro_ has quit IRC | 17:18 | |
| *** hashar has quit IRC | 17:19 | |
| clarkb | johnsom: https://review.openstack.org/#/c/585184/2 any reason for me to not approve that now? | 17:19 |
| johnsom | clarkb No, that is fine. Thanks for the heads up | 17:20 |
| johnsom | clarkb Looks like the job don't have a link to the rendered docs anymore. Is that fixed somewhere? | 17:22 |
| *** mriedem_away is now known as mriedem | 17:23 | |
| clarkb | johnsom: which job? | 17:23 |
| johnsom | clarkb The job Carlos linked: https://review.openstack.org/#/c/585185/ | 17:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Drop requirements-check from renderspec https://review.openstack.org/584753 | 17:24 |
| clarkb | http://logs.openstack.org/85/585185/2/check/openstack-tox-docs/7cb8e2d/ for anyone else following along | 17:24 |
| clarkb | johnsom: I'm not sure why that happened. Maybe because its running the normal tox role which doesn't know about the docs artifacts | 17:25 |
| johnsom | Previously the job link was to the rendered docs. It was pretty handy. Trying to remember where that was configured | 17:25 |
| clarkb | mordred: AJaeger ^ have been involved with that I think | 17:25 |
| johnsom | This was the old config: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n273 | 17:29 |
| clarkb | johnsom: I don't think the change above changed that (since you cannot depends on a trusted repo and have it change pre merge) | 17:30 |
| clarkb | johnsom: some other change must've changed that | 17:30 |
| *** harlowja has quit IRC | 17:31 | |
| clarkb | ah ya the change I just approved will switch you to that job | 17:32 |
| clarkb | johnsom: it should work as expected once that change merges | 17:32 |
| johnsom | Hmm, ok | 17:32 |
| *** felipemonteiro_ has joined #openstack-infra | 17:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update oslo.limit testing jobs https://review.openstack.org/579690 | 17:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Run propose-updates for requirements-constraints to Python3.6 https://review.openstack.org/585569 | 17:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch to publish-openstack-docs-pti in Octavia https://review.openstack.org/585184 | 17:34 |
| *** felipemonteiro__ has quit IRC | 17:37 | |
| *** yamahata has joined #openstack-infra | 17:44 | |
| *** dhajare has quit IRC | 17:48 | |
| openstackgerrit | Merged openstack-infra/project-config master: Change neutron CI dashboard to a week view https://review.openstack.org/583299 | 18:02 |
| *** 07IADF5PV has quit IRC | 18:06 | |
| *** myoung is now known as myoung|lunch | 18:09 | |
| *** jtomasek has quit IRC | 18:10 | |
| *** mriedem1 has joined #openstack-infra | 18:14 | |
| *** mriedem has quit IRC | 18:14 | |
| *** john_studarus has quit IRC | 18:15 | |
| *** harlowja has joined #openstack-infra | 18:15 | |
| *** diablo_rojo has quit IRC | 18:16 | |
| *** gema has quit IRC | 18:17 | |
| *** jamesmcarthur has joined #openstack-infra | 18:19 | |
| *** mriedem1 is now known as mriedem | 18:22 | |
| *** jamesmcarthur has quit IRC | 18:23 | |
| *** dhajare has joined #openstack-infra | 18:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: Set up translation for cloudkitty-dashboard https://review.openstack.org/582317 | 18:27 |
| *** Qiming has quit IRC | 18:27 | |
| *** jamesmcarthur has joined #openstack-infra | 18:27 | |
| *** pguimaraes has joined #openstack-infra | 18:29 | |
| johnsom | clarkb Nope, this just isn't configured for those new jobs: https://review.openstack.org/#/c/586625/ | 18:29 |
| johnsom | It doesn't look like the new job is even collecting the rendered docs.... | 18:29 |
| *** Qiming has joined #openstack-infra | 18:29 | |
| clarkb | johnsom: I don't think that is the new job | 18:30 |
| clarkb | that is the old one | 18:30 |
| johnsom | Nope, the old one rendered this stuff | 18:30 |
| clarkb | Its the job that was running before the change I approved | 18:31 |
| johnsom | No, this is a DNM job I put in after the project-config change merged to test it | 18:31 |
| johnsom | Looking at the job definitions it is clearly missing | 18:31 |
| *** rkukura has quit IRC | 18:32 | |
| clarkb | ah | 18:32 |
| johnsom | This is an example of the old job http://logs.openstack.org/64/585864/2/check/build-openstack-sphinx-docs/df0c553/html/ | 18:32 |
| johnsom | I am super booked today, but maybe later I can take a minute and propose fixes | 18:32 |
| *** sshnaidm|off has quit IRC | 18:32 | |
| clarkb | I think the reason this is happening is openstack-tox-docs is "run tox docs env" | 18:33 |
| clarkb | and implies nothing about copyting additional artifacts | 18:33 |
| clarkb | (at least as it is currently written) | 18:33 |
| johnsom | No, that is perfectly fine, it's just that the post task is not collecting the output and not setting the success link | 18:33 |
| clarkb | right | 18:33 |
| johnsom | lol, maybe we are saying the same thing... | 18:34 |
| clarkb | looking at openstack-zuul-jobs more closely what is the difference between publish-openstack-sphinx-docs and publish-openstack-docs-pti supposed to be? functionality its this job that doesn't publish the docs vs the one that does | 18:34 |
| corvus | clarkb: i'll be around (more reliably than this morning) then | 18:35 |
| clarkb | corvus: ok, I think we can probably do a restart around then assuming demand falls off as it has in prior days | 18:36 |
| openstackgerrit | Merged openstack-infra/zuul master: Add debug message to job freezing https://review.openstack.org/582750 | 18:36 |
| openstackgerrit | Merged openstack-infra/project-config master: Aslo apply the py35 job for trove-dashboard queens https://review.openstack.org/585966 | 18:36 |
| *** njohnston has joined #openstack-infra | 18:41 | |
| *** jamesmcarthur has quit IRC | 18:43 | |
| njohnston | Hi! I saw a job failure with an odd reason why, wanted to let you know. So the test run for this neutron-functional job actually passed, but the failure happened in the post-testing portion. trying to install os-log-merger it got 'No route to host' installing from pypi mirror on http://mirror.us-west-1.packethost.openstack.org | 18:45 |
| njohnston | http://logs.openstack.org/89/585489/10/check/neutron-functional/0f04b1e/job-output.txt.gz#_2018-07-27_16_36_40_708330 | 18:45 |
| *** diablo_rojo has joined #openstack-infra | 18:45 | |
| clarkb | njohnston: thanks for the heads up. That mirror had crashed earlier today, we've removed that cloud region from nodepool temporarily while we figure out why that happened | 18:46 |
| njohnston | Thanks clarkb and all the infra team for being on it as always! | 18:46 |
| njohnston | FYI the error in question happened about 10 minutes ago | 18:46 |
| clarkb | njohnston: the logged timestamp was from just over 2 hours ago (16:36UTC vs 18:47UTC now) | 18:47 |
| *** r-daneel_ has joined #openstack-infra | 18:47 | |
| *** r-daneel has quit IRC | 18:47 | |
| *** r-daneel_ is now known as r-daneel | 18:47 | |
| njohnston | oh gosh, sorry about that, I misread | 18:48 |
| *** gema has joined #openstack-infra | 18:49 | |
| *** gema has joined #openstack-infra | 18:49 | |
| *** gema has quit IRC | 18:54 | |
| *** v1a4 has joined #openstack-infra | 18:55 | |
| *** fried_rolls is now known as fried_rice | 18:59 | |
| *** sthussey has quit IRC | 19:03 | |
| *** hamerins has quit IRC | 19:03 | |
| *** r-daneel has quit IRC | 19:04 | |
| *** gema has joined #openstack-infra | 19:05 | |
| *** gema has joined #openstack-infra | 19:05 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Copy inventory file first for validate-host https://review.openstack.org/586639 | 19:11 |
| clarkb | pabelanger: re ^ the plan is to remove that task from validate hosts entirely | 19:14 |
| clarkb | pabelanger: the new log-inventory role is used instead | 19:14 |
| pabelanger | clarkb: oh | 19:15 |
| pabelanger | as long as we run it before validate-host, I'm fine with it | 19:15 |
| clarkb | pabelanger: https://review.openstack.org/#/c/563789/2 | 19:15 |
| clarkb | pabelanger: you might suggest flpping the order of the roles in that change | 19:16 |
| pabelanger | yah | 19:16 |
| clarkb | pabelanger: feel free to just push a new ps to that change | 19:17 |
| clarkb | (to get the order right) | 19:17 |
| *** myoung|lunch is now known as myoung | 19:18 | |
| *** jtomasek has joined #openstack-infra | 19:18 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Use log-inventory in base jobs https://review.openstack.org/563789 | 19:19 |
| pabelanger | done | 19:19 |
| *** jtomasek has quit IRC | 19:19 | |
| clarkb | ianw: I think we can work to get ^ in next week now that feature freeze is behind us | 19:19 |
| *** rtjure has quit IRC | 19:25 | |
| pabelanger | clarkb: node requests have haven't dropped to 22 seem to have caught up now | 19:28 |
| *** rkukura has joined #openstack-infra | 19:28 | |
| clarkb | ya maybe after lunch we go for it | 19:30 |
| clarkb | I need to eat now and expect corvus is doing similar | 19:30 |
| *** eharney has quit IRC | 19:31 | |
| *** rtjure has joined #openstack-infra | 19:35 | |
| zxiiro | we released JJB 2.2.0 this morning but I don't see it in pypi yet. Seems like the release job didn't kick off? | 19:36 |
| zxiiro | anyway to force it to start? | 19:36 |
| openstackgerrit | Merged openstack-infra/zuul master: fix zuul from scratch user and group creation https://review.openstack.org/581254 | 19:37 |
| openstackgerrit | Merged openstack-infra/zuul master: Add instructions for deploying zuul with openSUSE https://review.openstack.org/581255 | 19:37 |
| clarkb | zxiiro: it may have gotten caught it in the zuul misconfiguration fallout that caused a bunch of tests to fail early | 19:38 |
| clarkb | zxiiro: we can reenqueue it | 19:38 |
| fungi | http://zuul.openstack.org/builds.html confirms that's what happened | 19:38 |
| clarkb | I'm grabbing lunch now and can look at enqueuing it after | 19:38 |
| clarkb | or maybe someone else will beat me to it | 19:38 |
| fungi | i'll take a quick look | 19:39 |
| zxiiro | makes sense | 19:39 |
| zxiiro | is there a keyword to requeue these things? or do I need to poke you guys when these things happen? | 19:39 |
| *** lbragstad_ has joined #openstack-infra | 19:40 | |
| *** lbragstad has quit IRC | 19:41 | |
| fungi | nope | 19:41 |
| fungi | i ran the following from shell on zuul.o.o: sudo zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=release --project=openstack-infra/jenkins-job-builder --ref=refs/tags/2.2.0 --newrev=014aa39477bad6eadba3c351ce24935e749cd235 | 19:42 |
| fungi | the jobs for it should be queuing up already | 19:42 |
| zxiiro | fungi: ok, thanks! | 19:43 |
| fungi | nope to there being a user-facing api for it (yet), yep to needing to raise it with one of the zuul admins (or tag a higher numbered release) | 19:43 |
| *** jamesmcarthur has joined #openstack-infra | 19:47 | |
| *** dhajare has quit IRC | 19:49 | |
| fungi | zxiiro: it's running now http://zuul.openstack.org/stream.html?uuid=e3241a29ac6743a2af7a767b37ab9edb&logfile=console.log | 19:49 |
| zxiiro | great! | 19:50 |
| *** eernst has quit IRC | 19:51 | |
| *** eernst has joined #openstack-infra | 19:53 | |
| *** eernst has quit IRC | 19:53 | |
| *** eernst has joined #openstack-infra | 19:54 | |
| *** rlandy is now known as rlandy|brb | 19:55 | |
| *** lbragstad_ is now known as lbragstad | 19:56 | |
| fungi | zxiiro: and seems to have worked fine this time: https://pypi.org/project/jenkins-job-builder/ | 19:57 |
| *** hamerins has joined #openstack-infra | 19:57 | |
| zxiiro | woohoo | 19:57 |
| corvus | clarkb, fungi, mordred, pabelanger: i have replenished myself with a burrito at my local taqueria: https://imgur.com/a/0yeGj8I | 19:59 |
| pabelanger | ++ | 20:00 |
| *** ccamacho1 has joined #openstack-infra | 20:00 | |
| fungi | i'm technically still vacationing but am back in the cabin for a bit should help be required | 20:00 |
| corvus | i swear i didn't put that there :) | 20:00 |
| *** ccamacho has quit IRC | 20:01 | |
| clarkb | I had a giant bowl of fresh picked blueberries I mixed into cereal :) | 20:04 |
| *** jcoufal has quit IRC | 20:05 | |
| *** e0ne has joined #openstack-infra | 20:06 | |
| *** kgiusti has left #openstack-infra | 20:07 | |
| *** rosmaita has quit IRC | 20:11 | |
| clarkb | zuul==3.1.1.dev200 # git sha 22ad98a is the commit we have installed on all of the zuul nodes | 20:13 |
| clarkb | that seems to be tip of master | 20:13 |
| *** slaweq has joined #openstack-infra | 20:15 | |
| pabelanger | looks like it | 20:17 |
| *** rlandy|brb is now known as rlandy | 20:17 | |
| *** jmorgan1 has quit IRC | 20:18 | |
| fungi | 22ad98a matches what i have too | 20:22 |
| *** e0ne has quit IRC | 20:22 | |
| fungi | for master branch tip on origin remote | 20:22 |
| *** dtruong_ has quit IRC | 20:23 | |
| *** dtruong_ has joined #openstack-infra | 20:26 | |
| *** jmorgan1 has joined #openstack-infra | 20:28 | |
| *** dmsimard has quit IRC | 20:29 | |
| *** jamesmcarthur has quit IRC | 20:30 | |
| *** jamesmcarthur has joined #openstack-infra | 20:31 | |
| *** trown is now known as trown|outtypewww | 20:31 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add upload-logs-swift role https://review.openstack.org/584541 | 20:38 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Point Suse users are zookeeper releases page https://review.openstack.org/586682 | 20:39 |
| *** slaweq has quit IRC | 20:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add upload-logs-swift role https://review.openstack.org/584541 | 20:40 |
| *** felipemonteiro_ has quit IRC | 20:40 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:40 | |
| *** v1a4 has quit IRC | 20:40 | |
| clarkb | pabelanger: corvus should we work on restarting now? | 20:44 |
| pabelanger | ready here | 20:46 |
| corvus | it's still a bit busy, but i think it'd recovery quickly. i reckon we could go for it. | 20:49 |
| clarkb | ya its busy but not backlog busy I think we go for it too | 20:49 |
| fungi | sounds good | 20:50 |
| corvus | clarkb: what's your preferred sequence? | 20:50 |
| corvus | i think we worked this out last time, and you had come up with the optimal | 20:50 |
| *** jamesmcarthur has quit IRC | 20:50 | |
| clarkb | probably should've written it down :) | 20:50 |
| corvus | it's in eavesdrop somewhere :) | 20:50 |
| clarkb | I think it was stop scheduler, then stop mergers (to avoid merge failures), then stop executors. Start scheduler as soon as others are told to stop then start mergers and executors | 20:51 |
| clarkb | with a save and restart job queues bookending that | 20:51 |
| corvus | yeah, with a note that we need to wait for the mergers to stop (fast) before starting them, then wait for the executors to stop (slow) before starting them | 20:52 |
| pabelanger | +1 | 20:53 |
| corvus | (i mean, if you follow the rule don't start it until it's stopped, you're fine, but considering the executors take 15m to stop, it's worth a note :) | 20:53 |
| *** david-lyle has joined #openstack-infra | 20:54 | |
| clarkb | I'm happy to drive it or let someone else do it if they want a go at it | 20:54 |
| corvus | clarkb: i feel like i've done the last few planned full restarts, so why don't you take a drive. should be able to do it all from puppetmaster with ad-hoc ansible commands | 20:55 |
| clarkb | ya Itend to do the executors and mergers from there since there are many of them then do the zuul01 stuf fon host | 20:55 |
| clarkb | give me a minute to prep some commands but then I'll start | 20:56 |
| *** dklyle_ has quit IRC | 20:57 | |
| corvus | clarkb: hang on just a second | 20:57 |
| clarkb | ok | 20:57 |
| *** diablo_rojo has quit IRC | 20:58 | |
| corvus | clarkb: i've been working on spiffing up pabelanger's playbooks. i think i have a stop playbook which matches our process. maybe we can try that out and confirm it works? | 20:58 |
| *** eernst_ has joined #openstack-infra | 20:58 | |
| corvus | http://paste.openstack.org/show/726768/ | 20:58 |
| clarkb | sure | 20:58 |
| * clarkb looks | 20:58 | |
| corvus | pabelanger: ^ | 20:58 |
| *** eernst has quit IRC | 20:59 | |
| corvus | we'll want to run that with "-f 20" or something for all the executors | 21:00 |
| pabelanger | corvus: looks right at first glance | 21:00 |
| clarkb | corvus: we may be able to optomize it a bit by running some of the stops in parallel, but otherwise I think that should work | 21:00 |
| clarkb | don't forget to save queues first as well | 21:00 |
| corvus | and obviously, it's not supposed to return until the executors are all stopped, but we'll be wanting to start things before then. but hopefully we can at least confirm that the stop commands and wait_for's all work | 21:00 |
| clarkb | yup and we can work things on a side channel while that runs | 21:01 |
| corvus | ++ | 21:01 |
| clarkb | corvus: do you want to run it? | 21:02 |
| corvus | clarkb: probably best if you do; you can self-coordinate the starts, etc. | 21:02 |
| clarkb | ok | 21:03 |
| *** edmondsw has quit IRC | 21:04 | |
| clarkb | `sudo ansible-playbook -f 20 /home/clarkb/playbooks/stop_zuul.yaml` is what I'll be running that look right to you all? | 21:04 |
| corvus | clarkb: yep | 21:05 |
| *** edmondsw has joined #openstack-infra | 21:05 | |
| clarkb | ok I'll save queues first then run that here in a minute | 21:05 |
| *** yamahata has quit IRC | 21:05 | |
| *** eernst has joined #openstack-infra | 21:05 | |
| *** eernst_ has quit IRC | 21:05 | |
| *** r-daneel has joined #openstack-infra | 21:05 | |
| clarkb | queues saved, now running playbook | 21:07 |
| clarkb | ok I think there is a bug | 21:08 |
| clarkb | but it should be fine | 21:08 |
| fungi | i guess someone will push that up as an addition to the system-config repo once it's tested? or do we have somewhere to start putting zuul-specific deployment and lifecycle management bits? | 21:08 |
| clarkb | or maybe not | 21:08 |
| clarkb | the wait didn't wait as long as I expected but the scheduler is indeed stopped | 21:08 |
| corvus | fungi: yeah, i'm prepping that now | 21:08 |
| fungi | awesome | 21:08 |
| clarkb | I'm starting scheduler and tertiary services now | 21:08 |
| corvus | clarkb: scheduler takes maybe 10 seconds to stop? the executors should be a really obvious test | 21:09 |
| *** edmondsw has quit IRC | 21:10 | |
| clarkb | corvus: it returned from executors already too | 21:10 |
| corvus | okay so that's still broke. | 21:10 |
| *** pbourke has quit IRC | 21:10 | |
| clarkb | checking if they stopped now (scheduler and mergers should be starting now | 21:10 |
| clarkb | I double checked ps before starting those though | 21:11 |
| corvus | there are indeed no pidfiles in /var/run/zuul on the executors | 21:11 |
| clarkb | ya executors have not stopped yet | 21:11 |
| corvus | i wouldn't expect them to be removed until it's stopped | 21:12 |
| clarkb | summary to this point is queues saved and all services have been asked to stop. Scheduler, web, fingergw and mergers have been started after stopping. Waiting on executors to actually stop now so they can be restarted | 21:12 |
| clarkb | corvus: possible bug in the init script | 21:12 |
| clarkb | corvus: also looks like paramiko still tries the ed25519 key :/ | 21:13 |
| *** njohnston has left #openstack-infra | 21:13 | |
| clarkb | it would be nice if people didn't configure broken keys on gerrit | 21:13 |
| clarkb | (but maybe that is a gerrit bug in that if you don't have one it makes one?) | 21:14 |
| clarkb | enqueuing changes now | 21:14 |
| corvus | clarkb: hrm, i'll look into the paramiko issue. | 21:15 |
| corvus | clarkb: (oh! i bet our init scripts remove the pid file on stop. i think we can stop doing that now) | 21:15 |
| clarkb | corvus: ya Ithink that may be the init script bug | 21:16 |
| *** e0ne has joined #openstack-infra | 21:16 | |
| *** r-daneel has quit IRC | 21:16 | |
| clarkb | zuul mergers are stopping I'll start them one by one so that we can start processing jobs | 21:16 |
| corvus | clarkb: you mean executors? | 21:17 |
| clarkb | yes sorry | 21:17 |
| fungi | i believe at start gerrit generates any "missing" host keys it thinks it supports (i've certainly never manually generated host keys when bootstrapping it) | 21:17 |
| pabelanger | corvus: I think you are right, pid file deleted by init script | 21:18 |
| pabelanger | can look into that | 21:18 |
| *** yamahata has joined #openstack-infra | 21:18 | |
| corvus | ok hrm | 21:21 |
| clarkb | All services are running new code now. Check queue is being reenqueued (gate is done) | 21:22 |
| corvus | using the known_hosts that puppet wrote does not cause paramiko to use rsa. but it's a valid known_hosts -- if i use it with openssh, it's fine. no complaints about unknown keys. | 21:22 |
| corvus | but if i clear it out and use openssh and accept the host key (so openssh writes it), *that* makes paramiko use the rsa key | 21:23 |
| clarkb | weird | 21:23 |
| corvus | the main difference being that openssh is writing the hashed index | 21:23 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Stop deleting PIDFILE on stop https://review.openstack.org/586699 | 21:23 |
| clarkb | maybe paramiko treats hashed index as higher priority than unhashed? | 21:23 |
| pabelanger | corvus: corvus: ^pidfile for zuul | 21:23 |
| clarkb | check is enqueued, from my position of performing the mechanical changes we are done | 21:25 |
| clarkb | I haven't seen any errors other than the ssh host key failure to opendaylight | 21:25 |
| pabelanger | clarkb: corvus: not today, we should also look to delete git repo on mergers / executors to ensure they are not created with 0755 permissions. As they still are 777. | 21:26 |
| corvus | oh. the port. | 21:27 |
| *** pbourke has joined #openstack-infra | 21:27 | |
| clarkb | corvus: oh! | 21:27 |
| corvus | [git.opendaylight.org]:29418,[2600:1f14:421:f500:7b21:2a58:ab0a:2d17]:29418 ssh-rsa | 21:28 |
| corvus | AAAAB3NzaC1yc2EAAAABIwAAAQEAyRXyHEw/P1iZr/fFFzbodT5orVV/ftnNRW59Zh9rnSY5Rmbc9aygsZHdtiWBERVVv8atrJSdZool75AglPDDYtPICUGWLR91YBSDcZwReh5S9es1dlQ6fyWTnv9QggSZ98KTQEuE3t/b5SfH0T6tXWmrNydv4J2/mejKRRLU2+oumbeVN1yB+8Uau/3w9/K5F5LgsDDzLkW35djLhPV8r0OfmxV/cAnLl7AaZlaqcJMA+2rGKqM3m3Yu+pQw4pxOfCSpejlAwL6c8tA9naOvBkuJk+hYpg5tDEq2QFGRX5y1F9xQpwpdzZROc5hdGYntM79VMMXTj+95dwVv/8yTsw== | 21:28 |
| corvus | that's what openssh generates when i tell it to disable hashing. that makes it and paramiko happy | 21:28 |
| corvus | i think since zuul should keep retrying, it's probably connected to ODL during one of the times when i manually had a working config in place. so we probably don't need to revert or restart. we can just fix going forward. | 21:29 |
| clarkb | corvus: ++ | 21:29 |
| clarkb | zuul web seems happy, log streaming works. failed jobs appear to be valid failures | 21:29 |
| corvus | 2018-07-27 21:28:40,548 DEBUG zuul.Scheduler: Processing trigger event <GerritTriggerEvent comment-added git.opendaylight.org/mdsal master 74289,7 Verified:0, Code-Review:0> | 21:30 |
| pabelanger | yay | 21:30 |
| fungi | ahh, yep, without the port number it assumes it's for the default ssh port and probably just ignores it | 21:30 |
| clarkb | we should double check that we report success/failure properly since we made that change to didalljobssucceed | 21:30 |
| clarkb | fungi: whats odd is openssh seems to consider it vlaid for 29418 too based on corvus' testing | 21:31 |
| fungi | odd that openssh treats it more like a port wildcard | 21:31 |
| fungi | yep, that | 21:31 |
| corvus | maybe it's a generous fallback behavior | 21:32 |
| corvus | or, maybe it's an exploitable error :) | 21:32 |
| *** rfolco|ruck is now known as rfolco|off | 21:33 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add playbooks to start/stop/restart zuul https://review.openstack.org/586706 | 21:33 |
| corvus | i'll work on updating the kesy | 21:33 |
| corvus | keys | 21:33 |
| corvus | there was an empty line between the two keys, presumably because {gerrit_ssh_host_key} has a newline at the end of it, and i put a \n after that. would you like me to keep doing that (to protect against us removing the newline from gerrit_ssh_host_key in the future), or remove the extra \n i added to keep the file tidy? | 21:38 |
| clarkb | it is a machine managed and read file I think its probably fine to have extra whitespace if that makes the config mgmt easier to read | 21:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Fix syntax of gerrit host ssh keys https://review.openstack.org/586710 | 21:39 |
| fungi | i like the newline in the template as insurance against a missing newline in the variable itself | 21:39 |
| fungi | or, rather, i think expecting newlines at the end of variable values is a mistake | 21:40 |
| corvus | (also, this is something that would be great to make better when we next revisit how we handle these data; string concatenation is not going to scale here) | 21:40 |
| fungi | better to assume they'll be missing and end up with a blank line than to end up accidentally concatenating two of them | 21:40 |
| fungi | yeah, that too. string concatenation is a poor substitute for an actual template | 21:41 |
| *** boden has quit IRC | 21:42 | |
| openstackgerrit | Merged openstack-infra/zuul master: Point Suse users are zookeeper releases page https://review.openstack.org/586682 | 21:43 |
| pabelanger | okay, I have to leave now, but can check back later this evening! | 21:43 |
| clarkb | pabelanger: thanks | 21:43 |
| clarkb | pabelanger: also its friday you should probably do something other than work this evening :) | 21:43 |
| *** kjackal has quit IRC | 21:44 | |
| *** eernst has quit IRC | 21:44 | |
| *** pguimaraes has quit IRC | 21:44 | |
| pabelanger | ++ | 21:45 |
| *** rtjure has quit IRC | 21:46 | |
| *** eernst has joined #openstack-infra | 21:48 | |
| *** eernst_ has joined #openstack-infra | 21:49 | |
| *** eernst has quit IRC | 21:49 | |
| *** neiloy has quit IRC | 21:53 | |
| *** eernst_ has quit IRC | 21:53 | |
| *** eernst has joined #openstack-infra | 21:54 | |
| *** eernst has quit IRC | 21:59 | |
| *** e0ne has quit IRC | 22:00 | |
| *** sshnaidm|off has joined #openstack-infra | 22:04 | |
| *** hamerins has quit IRC | 22:05 | |
| *** felipemonteiro_ has quit IRC | 22:06 | |
| *** shaner has quit IRC | 22:06 | |
| *** shaner has joined #openstack-infra | 22:07 | |
| *** rh-jelabarre has quit IRC | 22:07 | |
| *** rtjure has joined #openstack-infra | 22:13 | |
| corvus | so... anyone have thoughts on how to get openstacksdk installed on the executors? | 22:13 |
| corvus | i guess we could just add it to puppet with the pip3 provider? | 22:14 |
| clarkb | corvus tobiash seemed to think we could add it as a zuul dep | 22:14 |
| clarkb | (I'm not sure that is best route though) | 22:15 |
| *** rh-jelabarre has joined #openstack-infra | 22:15 | |
| corvus | hrm, i'm not sure i'm keen on that. zuul-jobs could end up having a lot of optional dependencies. i guess we could argue that *this one* is special.... | 22:15 |
| clarkb | maybe list it as an extras require? | 22:15 |
| corvus | if we get around to supporting multiple ansible versions, and we have zuul build the venvs, then we could add things like this to a zuul config file so it could install them. that sounds ideal, but that's rather a bit in the future. | 22:16 |
| clarkb | I'd be ok with an explicit out of band install to start that will be forward compatible if we add it to the dep list | 22:16 |
| corvus | we should also probably think about image builds.... | 22:17 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add upload-logs-swift role https://review.openstack.org/584541 | 22:17 |
| corvus | ie, if we want to run our executors from pbrx containers | 22:17 |
| *** neiloy has joined #openstack-infra | 22:17 | |
| clarkb | Does pbrx handle extras requires yet? that may be a good option there | 22:18 |
| corvus | that's probably an easy one too. there's a way to add extra packages to the zuul-executor image by adding them to setup.cfg. | 22:18 |
| clarkb | then you can pip install zuul[executor] and get that stuff only on that image | 22:18 |
| corvus | ara is added like that. | 22:18 |
| corvus | clarkb: yeah, it uses extras | 22:18 |
| corvus | http://git.zuul-ci.org/cgit/zuul/tree/setup.cfg#n55 | 22:18 |
| corvus | okay, i think i like the idea of us adding it to puppet and extras, but keeping it out of the main requirements file for now. | 22:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add openstacksdk to executor extras https://review.openstack.org/586717 | 22:20 |
| *** shaner has quit IRC | 22:23 | |
| *** shaner has joined #openstack-infra | 22:24 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Install openstacksdk on zuul executors https://review.openstack.org/586718 | 22:24 |
| *** sambetts_ has quit IRC | 22:24 | |
| *** sambetts_ has joined #openstack-infra | 22:26 | |
| *** tpsilva has quit IRC | 22:26 | |
| clarkb | both changes lgtm. I think generally being helpful with dependency lists even at the cost of disk is probably a good thing | 22:26 |
| corvus | clarkb: yeah, except that the actual requirements.txt is very frequently automatically installed in a way that is difficult or impossible for people to override. some users of zuul have requested that we keep the fixed requirement to a minimum. so it's a tricky balancing act. | 22:31 |
| corvus | happily, openstacksdk doesn't impose a lot of additional requirements of its own. | 22:32 |
| fungi | extras are also problematic come uninstall or upgrade time | 22:32 |
| clarkb | corvus: ya I think extras is a reasonable compromise, you have to opt into using them but its still an automatable thing if you chose to consume them | 22:32 |
| clarkb | fungi: how so? | 22:32 |
| clarkb | fungi: you can put them on the same pip intsall/uninstall command and it should sort it all out | 22:33 |
| fungi | last i heard (unless it's been fixed) pip doesn't know that packages installed via extras are dependencies when you do upgrades | 22:33 |
| clarkb | fungi: ya they are like distinct lists, you have to list them together if using them | 22:33 |
| fungi | though, yes, including the extras on the upgrade command i guess works | 22:33 |
| fungi | as for container images, if the size isn't significantly different i expect having one zuul image which can be used in a variety of configurations for different combinations of subservices might be more user-friendly | 22:34 |
| clarkb | fungi: I think the tricky thing doing that is that (docker at least) bakes in the command to run along with the image itself | 22:35 |
| fungi | but i'm not familiar enough with container ecosystems to know for sure whether that makes a difference to people using such installation methods | 22:35 |
| clarkb | so you end up needing different images for different commands even if they are basically hte same | 22:35 |
| fungi | mmm, so if you wanted a container which ran multiple daemons how would it handle that? | 22:36 |
| clarkb | fungi: I think the container crowd would say not to do that, but aiui if you want that you make your command to run an init system | 22:36 |
| fungi | i have a hard time believing docker containers only ever have one running process per container | 22:36 |
| clarkb | fungi: thats the design pattern | 22:36 |
| fungi | lots of software expects supporting processes | 22:37 |
| clarkb | fungi: ya 'sidecar' containers in k8s terminology are how you handle that | 22:37 |
| * fungi checks to see how many processes his web browser is currently comprised of | 22:37 | |
| clarkb | Forking like a web browser or zuul-executor is fine | 22:37 |
| clarkb | but it won't start distinct processes for you | 22:38 |
| clarkb | just the one (and thta one has to fork) | 22:38 |
| corvus | fungi: the pbrx system is building a container image for each setuptools entrypoint. so we have a distinct executor image (but some common things are shared in a base image which gets layered in) | 22:38 |
| *** agopi has quit IRC | 22:38 | |
| clarkb | corvus: ya and I think the design there is largely dictated by how docker expects to run containers (one top level process per container image) | 22:39 |
| corvus | and yeah, there's even a "zuul" image for running the zuul CLI | 22:39 |
| fungi | amusing | 22:39 |
| corvus | (so, in the container world, when we need to enqueue, the command will be a wee bit longer, but substantially similar -- we'll still log into zuul01 and "run the enqueue command") | 22:40 |
| fungi | so we wouldn't run zuul-web and apache in the same container, there's be a zuul-web container and an apache container and they'd communicate via virtual interfaces exposed into each of those? | 22:41 |
| corvus | fungi: i think you can turn off the network namespacing to make that less weird, and i think that's what mordred has planned | 22:42 |
| clarkb | fungi: yes (except I think to start we are using the host's networking so they will see regularl ol' eth0 and have to use distinct ports) | 22:42 |
| clarkb | corvus: yup in part because it makes things like the zuul enqueue commands simpler | 22:42 |
| clarkb | (since they rely on gearman which relies on tcp) | 22:42 |
| *** hongbin_ has quit IRC | 22:42 | |
| corvus | at the end of the day, it's probably going to feel more like a (very baroque) chroot for us. :) | 22:43 |
| clarkb | fungi: if we wanted different network stacks we can do that too and it sets up a bridge like system like neutron | 22:44 |
| corvus | (albiet, one which is not convenient to cd into) | 22:44 |
| clarkb | fungi: but then you hvae to sort out how to plug one system into another and that is made much easier if you jump to kubernetes | 22:44 |
| clarkb | (though I use docker swarm on my single node server at home and it works well enough for that) | 22:44 |
| clarkb | corvus: you can exec a shell into a container pretty easily as long as the shell exists on the image | 22:44 |
| corvus | yep | 22:45 |
| clarkb | docker exec | 22:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Support line comments in Gerrit https://review.openstack.org/577035 | 22:48 |
| fungi | doesn't sound too bad, just cumbersome to operate (in exchange for being less cumbersome to deploy i suppose) | 22:49 |
| fungi | and more portable (albeit in much the same way that java applications are "portable") | 22:50 |
| corvus | yeah -- concievably with some helper scripts installed for each of the images, it could be nearly transparent "zuul enqueue" could just work | 22:50 |
| clarkb | fungi: ya control over what is installed and when it changes is a big part of the draw. Youcan do neat stuff with namespacing beyond that but it seem to be the major win for most | 22:52 |
| *** shaner has quit IRC | 22:57 | |
| *** shaner has joined #openstack-infra | 22:59 | |
| *** mschuppert has quit IRC | 23:03 | |
| clarkb | this conversation inspired me to look at changes to the zuul container spec again. Other than one small nit I think it is ready to go | 23:08 |
| clarkb | corvus: ^ | 23:08 |
| clarkb | and now I must find somewhere to plug laptop into the wall | 23:08 |
| *** rpioso is now known as rpioso|afk | 23:09 | |
| *** harlowja has quit IRC | 23:09 | |
| *** neiloy has quit IRC | 23:15 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Support line comments in Gerrit https://review.openstack.org/577035 | 23:20 |
| *** myoung has quit IRC | 23:22 | |
| *** shaner has quit IRC | 23:23 | |
| pabelanger | clarkb: If okay with you, I might experiment with upgrading ze01 hwe kernel this weekend, that we've restarted zuul services. Based on reading kernel release notes, i think might be what is giving us such an improvement on ze11.o.o for ansible-playbooks | 23:27 |
| *** shaner has joined #openstack-infra | 23:29 | |
| mordred | pabelanger: that would be a great piece of info to learn if it turns out to be the case | 23:29 |
| pabelanger | indeed, for reasons unknown, ze11.o.o is way ahead with number of concurrent ansible-playbook processes. Hopefully ze01.o.o gets the same boost with latest kernel | 23:30 |
| *** rh-jelabarre has quit IRC | 23:33 | |
| clarkb | ya I think it is worth trying | 23:33 |
| openstackgerrit | Merged openstack-infra/zuul master: scheduler: return project_canonical in status page https://review.openstack.org/582451 | 23:34 |
| pabelanger | clarkb: actually, looks already installed just need to reboot server to pick it up | 23:35 |
| pabelanger | I'll do that in a bit | 23:36 |
| *** rlandy has quit IRC | 23:37 | |
| *** gongysh has joined #openstack-infra | 23:43 | |
| mordred | pabelanger: what's the change in the kernel changelog? | 23:45 |
| mordred | or release notes or whatever? | 23:46 |
| *** diablo_rojo has joined #openstack-infra | 23:47 | |
| clarkb | There were pti improvements iirc | 23:49 |
| pabelanger | let me see where I found it | 23:52 |
| pabelanger | https://kernelnewbies.org/Linux_4.15#Memory_management | 23:53 |
| mordred | pabelanger: thanks | 23:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Support line comments in Gerrit https://review.openstack.org/577035 | 23:53 |
| pabelanger | some interesting patches for speedup | 23:53 |
| pabelanger | have zero idea, if that is the reason | 23:53 |
| pabelanger | :) | 23:53 |
| *** wolverineav has quit IRC | 23:53 | |
| *** wolverineav has joined #openstack-infra | 23:54 | |
| mordred | pabelanger: it's a worthy hypothesis | 23:54 |
| mordred | if it turns out to be true- you should write a little blog post about it | 23:55 |
| mordred | seems like the sort of thing people would like - and we've got the ability to show the impact on a real world workload at scale | 23:55 |
| mordred | "we upgrade the kernel and you'll never believe what happened next ..." | 23:55 |
| *** eernst has joined #openstack-infra | 23:56 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Support line comments in Gerrit https://review.openstack.org/577035 | 23:56 |
| openstackgerrit | Michael Johnson proposed openstack-infra/openstack-zuul-jobs master: Fix publish-openstack-docs-pti check/gate html https://review.openstack.org/586729 | 23:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add role to fetch zuul logs from nodes https://review.openstack.org/583346 | 23:58 |
| pabelanger | mordred: ++ | 23:58 |
| *** wolverineav has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!