| fungi | update on the ssh problems, i think we're looking at two different ones | 00:01 |
|---|---|---|
| fungi | searching logstash for message:"REMOTE HOST IDENTIFICATION HAS CHANGED" turns up failures almost exclusively in inap-mtl01 | 00:02 |
| mgagne | fungi: could be that orphan ports issue again... | 00:02 |
| mgagne | looks like it's not the case? maybe zombie instances | 00:04 |
| fungi | that's my guess but i haven't tried correlating ip addresses in the error messages to find out whether it's the same ones cropping up over and over | 00:04 |
| mgagne | will take a look | 00:06 |
| fungi | i used to know how to get a query link from kibana, but can't figure it out in the new version | 00:06 |
| *** dosaboy has quit IRC | 00:08 | |
| *** dosaboy has joined #openstack-infra | 00:09 | |
| fungi | clarkb: i have 5 held nodes already... any idea how to trace them back to specific builds? is there a more effective way that poking around in zuul scheduler logs? | 00:10 |
| *** yamamoto has joined #openstack-infra | 00:10 | |
| clarkb | fungi: I grep the number suffix in executor logs | 00:11 |
| clarkb | innefficifnt but works | 00:11 |
| fungi | how do you know which executor? | 00:11 |
| fungi | or you grep across all 12 of them? | 00:11 |
| *** sthussey has quit IRC | 00:12 | |
| *** jcoufal has joined #openstack-infra | 00:14 | |
| *** yamamoto has quit IRC | 00:16 | |
| clarkb | I grep all | 00:17 |
| *** Weifan has quit IRC | 00:17 | |
| fungi | huh, none of these are showing up in any of the executor logs | 00:18 |
| fungi | not even the executor debug logs | 00:18 |
| *** yamamoto has joined #openstack-infra | 00:18 | |
| *** yamamoto has quit IRC | 00:18 | |
| *** yamamoto has joined #openstack-infra | 00:19 | |
| fungi | for n in 0{1,2,3,4,5,6,7,8,9} 1{0,1,2};do ssh ze$n.openstack.org grep 0005541845 /var/log/zuul/executor.log;done | 00:19 |
| corvus | fungi: try executor-debug.log | 00:20 |
| *** yamamoto has quit IRC | 00:20 | |
| *** yamamoto has joined #openstack-infra | 00:20 | |
| fungi | i tried both, but only gave one as an example | 00:20 |
| fungi | or i thought i tried both, but trying debug one again now just to be sure | 00:21 |
| fungi | takes longer to complete for obvious reasons | 00:21 |
| fungi | oh, nope, i guess i didn't do the full loop, there it is | 00:21 |
| fungi | but yeah, only in executor-debug.log not in executor.log | 00:23 |
| fungi | will convert these to build ids | 00:23 |
| *** yamamoto has quit IRC | 00:27 | |
| *** nicolasbock has quit IRC | 00:27 | |
| fungi | huh, there's no build result in zuul for one of these | 00:28 |
| fungi | http://zuul.opendev.org/t/openstack/build/b3494a8a5847483891e9c375759d20f9 | 00:28 |
| *** dayou_ has quit IRC | 00:29 | |
| fungi | for 3 of the 5 in fact | 00:30 |
| fungi | of the build ids listed in the executor debug logs for these 5 held nodes, only 2 of them have build records | 00:30 |
| fungi | i wonder if the other 3 were retried | 00:31 |
| corvus | fungi: or their buildsets were reset | 00:33 |
| fungi | ahh, makes sense | 00:33 |
| fungi | of the two which have build results, both are merely failing some failing unit tests | 00:33 |
| fungi | not our ssh disconnect | 00:33 |
| fungi | i'll delete these 5 nodes and look for more | 00:33 |
| corvus | fungi: you can grep for the build ids in the scheduler log | 00:34 |
| corvus | fungi: check the other 3 first | 00:34 |
| fungi | oh, good idea | 00:34 |
| corvus | and we can find their logs out of band | 00:34 |
| *** Weifan has joined #openstack-infra | 00:34 | |
| corvus | fungi: oh, actually, the last log line on the executor is useful: eg 2019-04-23 23:52:58,617 DEBUG zuul.AnsibleJob: [build: b3494a8a5847483891e9c375759d20f9] Sending result: {"result": "FAILURE", "data": {"zuul": {"log_url": "http://logs.openstack.org/07/655107/1/check/openstack-tox-py36/b3494a8/"}}, "warnings": []} | 00:35 |
| corvus | fungi: that has the result and the log url | 00:36 |
| fungi | oh, convenient! | 00:36 |
| fungi | thanks, way simpler | 00:36 |
| corvus | and, ftr, that one is not a hit :) | 00:36 |
| johnsom | Looks like the hostname change is breaking jobs: http://logs.openstack.org/75/655275/1/check/openstack-tox-pep8/6c3e302/job-output.txt.gz#_2019-04-24_00_34_06_971348 | 00:37 |
| mgagne | fungi: I had to write a script to automate the check. Now double checking the list of orphan instances before deleting them. | 00:38 |
| fungi | thanks mgagne! that'll be a huge help | 00:39 |
| corvus | johnsom: that's the zombie instance mgagne is working on | 00:39 |
| corvus | er, "zombie instance problem" | 00:39 |
| fungi | beware of rogue virtual machines | 00:40 |
| johnsom | Ah, cool. A docs change hit three post-failures so I took a look. Thanks folks! | 00:40 |
| *** armax has quit IRC | 00:42 | |
| *** Weifan has quit IRC | 00:42 | |
| *** Weifan has joined #openstack-infra | 00:43 | |
| mgagne | fungi: what's frustrating is that I think Nova is able to fix those kind of issues by itself. Just not with CellsV1 and/or the version we are running. | 00:45 |
| fungi | speaking of which, all five of the held nodes were failing random nova unit tests... none were executor connection timeouts | 00:47 |
| fungi | recycling those 5 now and looking for more | 00:47 |
| *** diablo_rojo has quit IRC | 00:48 | |
| mgagne | fungi: deleted all zombie instances. I reran the script and none were found. | 00:49 |
| mgagne | I have to go now. If there are any issues, guess they will have to wait until tomorrow. just let me know, will read backlog tomorrow. | 00:50 |
| fungi | thanks again mgagne! | 00:50 |
| fungi | have a great evening | 00:51 |
| *** jamesmcarthur has joined #openstack-infra | 00:52 | |
| fungi | corvus: clarkb: one thing which was suggested was that it would be helpful it we had a way to record a traceroute to the instance when the ssh connection dies... i'm not sure what options we might have for that, ansible failure callback? | 00:53 |
| *** Weifan has quit IRC | 00:54 | |
| *** jamesmcarthur has quit IRC | 00:54 | |
| clarkb | fungi: hrm maybe if we get an ansible return code 4 then run the traceroute | 00:55 |
| *** Weifan has joined #openstack-infra | 00:58 | |
| *** Weifan has quit IRC | 00:58 | |
| *** jamesmcarthur has joined #openstack-infra | 00:58 | |
| *** slaweq has quit IRC | 00:58 | |
| *** Weifan has joined #openstack-infra | 00:58 | |
| *** Weifan has quit IRC | 01:00 | |
| *** Weifan has joined #openstack-infra | 01:00 | |
| *** yamamoto has joined #openstack-infra | 01:00 | |
| clarkb | fungi: I've figured out my stuggling finding logs for these exit code 4 situations. We just retry them and never record build logs in the logs server? | 01:02 |
| *** mriedem has quit IRC | 01:03 | |
| *** Weifan has quit IRC | 01:03 | |
| *** hwoarang has quit IRC | 01:06 | |
| *** hwoarang has joined #openstack-infra | 01:08 | |
| fungi | that seems to be the case | 01:13 |
| *** markvoelker has joined #openstack-infra | 01:13 | |
| *** jamesmcarthur has quit IRC | 01:14 | |
| *** rfolco has joined #openstack-infra | 01:15 | |
| *** whoami-rajat has joined #openstack-infra | 01:16 | |
| *** jamesmcarthur_ has joined #openstack-infra | 01:18 | |
| *** arunkant has left #openstack-infra | 01:31 | |
| *** jamesmcarthur_ has quit IRC | 01:34 | |
| openstackgerrit | liuwei proposed openstack/os-performance-tools master: update git.openstack.org to opendev https://review.opendev.org/655280 | 01:39 |
| openstackgerrit | liuwei proposed openstack/os-testr master: update git.openstack.org to opendev https://review.opendev.org/655284 | 01:48 |
| *** michael-beaver has quit IRC | 01:49 | |
| *** apetrich has quit IRC | 01:56 | |
| corvus | we do place logs in the log server | 01:57 |
| corvus | we just don't record logs of incomplete builds in the database | 01:57 |
| clarkb | ah | 02:01 |
| *** jamesmcarthur has joined #openstack-infra | 02:05 | |
| *** ricolin has joined #openstack-infra | 02:06 | |
| *** ykarel|away has joined #openstack-infra | 02:07 | |
| fungi | yeah, that was my initial misunderstanding as well | 02:08 |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Change some OpenStack to OpenDev or ttygroup https://review.opendev.org/655286 | 02:09 |
| *** roman_g has quit IRC | 02:12 | |
| *** jamesmcarthur has quit IRC | 02:12 | |
| *** Weifan has joined #openstack-infra | 02:16 | |
| *** gmann_afk is now known as gmann | 02:16 | |
| *** Weifan has quit IRC | 02:16 | |
| *** jamesmcarthur has joined #openstack-infra | 02:18 | |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Change the default location for gertty config file https://review.opendev.org/581868 | 02:21 |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Change some OpenStack to OpenDev or ttygroup https://review.opendev.org/655286 | 02:21 |
| corvus | fascinating; somehow gertty grew a test-release-openstack job | 02:24 |
| corvus | looks like it was automatically given release jobs in the v2->v3 transition, which we never noticed because it doesn't use our release machinery, and then it was automatically given the publish-to-pypi template because it had release jobs. | 02:25 |
| *** bhavikdbavishi has joined #openstack-infra | 02:26 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Remove publish-to-pypi from ttygroup repos https://review.opendev.org/655287 | 02:27 |
| corvus | anyone around to +3 that? ^ | 02:27 |
| fungi | yup | 02:28 |
| fungi | it's amazing how jobs magically accumulate on projects | 02:29 |
| *** bhavikdbavishi has quit IRC | 02:35 | |
| openstackgerrit | Merged ttygroup/gertty master: Change the default location for gertty config file https://review.opendev.org/581868 | 02:37 |
| openstackgerrit | Merged ttygroup/gertty master: Change some OpenStack to OpenDev or ttygroup https://review.opendev.org/655286 | 02:37 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: add OpenAPI documentation https://review.opendev.org/535541 | 02:39 |
| *** igordc has joined #openstack-infra | 02:43 | |
| openstackgerrit | Merged openstack/project-config master: Remove publish-to-pypi from ttygroup repos https://review.opendev.org/655287 | 02:50 |
| openstackgerrit | Merged openstack/os-performance-tools master: Dropping the py35 testing https://review.opendev.org/654289 | 02:52 |
| *** Weifan has joined #openstack-infra | 02:52 | |
| *** Weifan has quit IRC | 02:52 | |
| openstackgerrit | Merged openstack/os-performance-tools master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/655055 | 02:53 |
| corvus | i've released gertty 1.6.0 and sent announcements to openstack-infra, openstack-discuss, and repo-discuss | 03:01 |
| fungi | awesome! | 03:02 |
| *** psachin has joined #openstack-infra | 03:02 | |
| *** xinranwang has joined #openstack-infra | 03:06 | |
| *** yamamoto has quit IRC | 03:09 | |
| *** yamamoto has joined #openstack-infra | 03:10 | |
| *** jcoufal has quit IRC | 03:12 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:19 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:22 | |
| *** bhavikdbavishi has quit IRC | 03:23 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:23 | |
| openstackgerrit | Merged opendev/system-config master: Blackhole spam for airship-discuss-owner address https://review.opendev.org/655227 | 03:32 |
| *** whoami-rajat has quit IRC | 03:35 | |
| *** ykarel|away has quit IRC | 03:40 | |
| *** smarcet has joined #openstack-infra | 03:44 | |
| *** ramishra has joined #openstack-infra | 03:51 | |
| *** udesale has joined #openstack-infra | 03:58 | |
| *** whoami-rajat has joined #openstack-infra | 04:06 | |
| *** smarcet has quit IRC | 04:16 | |
| *** smarcet has joined #openstack-infra | 04:17 | |
| *** jamesmcarthur has quit IRC | 04:17 | |
| *** jamesmcarthur has joined #openstack-infra | 04:18 | |
| *** jamesmcarthur has quit IRC | 04:23 | |
| *** ykarel|away has joined #openstack-infra | 04:24 | |
| *** jamesmcarthur has joined #openstack-infra | 04:29 | |
| *** ykarel_ has joined #openstack-infra | 04:30 | |
| *** ykarel|away has quit IRC | 04:32 | |
| *** smarcet has quit IRC | 04:36 | |
| *** jamesmcarthur has quit IRC | 04:45 | |
| *** jamesmcarthur has joined #openstack-infra | 04:45 | |
| *** jamesmcarthur has quit IRC | 04:50 | |
| *** jamesmcarthur has joined #openstack-infra | 04:50 | |
| *** yamamoto has quit IRC | 04:51 | |
| *** yamamoto has joined #openstack-infra | 04:53 | |
| *** rkukura has quit IRC | 04:53 | |
| *** rkukura has joined #openstack-infra | 04:53 | |
| *** jamesmcarthur has quit IRC | 04:54 | |
| *** jamesmcarthur has joined #openstack-infra | 05:06 | |
| *** e0ne has joined #openstack-infra | 05:10 | |
| *** markvoelker has quit IRC | 05:10 | |
| *** jamesmcarthur has quit IRC | 05:11 | |
| *** hwoarang has quit IRC | 05:13 | |
| *** hwoarang has joined #openstack-infra | 05:14 | |
| *** quiquell|off is now known as quiquell|rover | 05:17 | |
| *** jamesmcarthur has joined #openstack-infra | 05:17 | |
| *** e0ne has quit IRC | 05:17 | |
| *** jamesmcarthur has quit IRC | 05:22 | |
| *** rkukura has quit IRC | 05:22 | |
| *** rkukura has joined #openstack-infra | 05:23 | |
| *** jamesmcarthur has joined #openstack-infra | 05:25 | |
| *** yamamoto has quit IRC | 05:27 | |
| *** yamamoto has joined #openstack-infra | 05:29 | |
| *** jamesmcarthur has quit IRC | 05:32 | |
| *** igordc has quit IRC | 05:49 | |
| *** ykarel_ is now known as ykarel | 05:53 | |
| *** kopecmartin|off is now known as kopecmartin | 06:08 | |
| *** dayou has joined #openstack-infra | 06:09 | |
| quiquell|rover | Good morning | 06:10 |
| *** electrofelix has joined #openstack-infra | 06:11 | |
| *** jamesmcarthur has joined #openstack-infra | 06:11 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 06:15 | |
| *** jamesmcarthur has quit IRC | 06:15 | |
| *** bhavikdbavishi has quit IRC | 06:16 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 06:16 | |
| *** yamamoto has quit IRC | 06:20 | |
| *** d34dh0r53 has quit IRC | 06:22 | |
| *** cloudnull has quit IRC | 06:22 | |
| *** yamamoto has joined #openstack-infra | 06:23 | |
| *** eglute has quit IRC | 06:23 | |
| *** pcaruana has joined #openstack-infra | 06:24 | |
| *** bgmccollum has quit IRC | 06:24 | |
| *** pgaxatte has joined #openstack-infra | 06:26 | |
| *** kjackal has joined #openstack-infra | 06:30 | |
| *** zhangfei has joined #openstack-infra | 06:38 | |
| *** ralonsoh has joined #openstack-infra | 06:52 | |
| *** quiquell|rover is now known as quique|rover|brb | 06:54 | |
| *** rpittau|afk is now known as rpittau | 06:56 | |
| *** yamamoto has quit IRC | 06:57 | |
| *** slaweq has joined #openstack-infra | 06:57 | |
| *** yamamoto has joined #openstack-infra | 06:59 | |
| *** yamamoto has quit IRC | 07:00 | |
| *** yamamoto has joined #openstack-infra | 07:01 | |
| *** egonzalez has quit IRC | 07:03 | |
| *** egonzalez has joined #openstack-infra | 07:04 | |
| *** rcernin has quit IRC | 07:05 | |
| *** fmount has left #openstack-infra | 07:07 | |
| *** ginopc has joined #openstack-infra | 07:09 | |
| *** yamamoto has quit IRC | 07:09 | |
| *** markvoelker has joined #openstack-infra | 07:12 | |
| *** apetrich has joined #openstack-infra | 07:13 | |
| *** quique|rover|brb is now known as quiquell|rover | 07:14 | |
| *** xinranwang has quit IRC | 07:22 | |
| *** amoralej has joined #openstack-infra | 07:25 | |
| *** tosky has joined #openstack-infra | 07:26 | |
| openstackgerrit | Slawek Kaplonski proposed ttygroup/gertty master: Update information about config file location https://review.opendev.org/655338 | 07:30 |
| *** dtantsur|afk is now known as dtantsur | 07:40 | |
| *** jpena|off is now known as jpena | 07:48 | |
| *** ykarel is now known as ykarel|lunch | 07:53 | |
| *** jpich has joined #openstack-infra | 08:00 | |
| *** helenafm has joined #openstack-infra | 08:06 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: web: add build modal with a parameter form https://review.opendev.org/644485 | 08:07 |
| aspiers | is it just me or is gerrit search autocompletion for author: completely broken? | 08:07 |
| aspiers | I type author:aspiers and when I accept the completion to author:"Adam Spiers <aspiers@suse.com>" it finds nothing | 08:08 |
| aspiers | in contrast if I very carefully avoid the completion (which is surprisingly hard because for some incredibly strange reason Escape *accepts* rather than cancels it), author:aspiers@suse.com works fine | 08:09 |
| aspiers | I really cannot wait for us to get off this ancient Gerrit! | 08:09 |
| paladox | You could see if author:self works | 08:11 |
| *** derekh has joined #openstack-infra | 08:13 | |
| *** yamamoto has joined #openstack-infra | 08:26 | |
| *** lucasagomes has joined #openstack-infra | 08:26 | |
| asettle | fungi, okay - re-read the backlog and looked through the commit in question (https://github.com/openstack/openstackdocstheme/commit/d31e4ded8941a69b36de413f1bcf56c91bece779\) | 08:27 |
| asettle | I have... no idea :/ I'm trying to think why we even had that login button in the first place. | 08:28 |
| asettle | AJaeger, do you have any recollection as to why we included a "LOG IN" button on the top right of the openstackdocstheme? The only thing I can think of is either using it for launchpad ID? | 08:28 |
| asettle | But that doesn't make a tonne of sense. | 08:28 |
| *** ykarel|lunch is now known as ykarel | 08:30 | |
| *** yamamoto has quit IRC | 08:33 | |
| *** jpich has quit IRC | 08:56 | |
| *** jpich has joined #openstack-infra | 08:56 | |
| *** jpich has quit IRC | 08:57 | |
| *** roman_g has joined #openstack-infra | 09:02 | |
| *** jpich has joined #openstack-infra | 09:04 | |
| *** sshnaidm|afk is now known as sshnaidm| | 09:16 | |
| *** sshnaidm| is now known as sshnaidm | 09:16 | |
| *** dtantsur is now known as dtantsur|brb | 09:19 | |
| *** e0ne has joined #openstack-infra | 09:20 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: add support for checkbox and list parameters https://review.opendev.org/648661 | 09:24 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: add support for checkbox and list parameters https://review.opendev.org/648661 | 09:25 |
| *** gfidente has joined #openstack-infra | 09:27 | |
| tristanC | it seems like https://opendev.org doesn't replicate the gerrit ref like git.openstack.org used to do, and this is breaking the "gitweb" link on review.opendev.org | 09:33 |
| *** d34dh0r53 has joined #openstack-infra | 09:45 | |
| *** Weifan has joined #openstack-infra | 09:46 | |
| *** eglute has joined #openstack-infra | 09:47 | |
| *** ramishra has quit IRC | 09:51 | |
| *** tkajinam has quit IRC | 09:52 | |
| *** seyeongkim has quit IRC | 09:53 | |
| *** yamamoto has joined #openstack-infra | 09:53 | |
| *** dirk has quit IRC | 09:54 | |
| *** gmann has quit IRC | 09:54 | |
| *** dirk has joined #openstack-infra | 09:54 | |
| *** rpittau has quit IRC | 09:55 | |
| *** vdrok has quit IRC | 09:55 | |
| *** mrhillsman has quit IRC | 09:55 | |
| *** rm_work has quit IRC | 09:55 | |
| *** seyeongkim has joined #openstack-infra | 09:55 | |
| *** gmann has joined #openstack-infra | 09:56 | |
| *** vdrok has joined #openstack-infra | 09:57 | |
| *** rpittau has joined #openstack-infra | 09:57 | |
| *** mrhillsman has joined #openstack-infra | 09:58 | |
| *** bhavikdbavishi has quit IRC | 10:00 | |
| *** ykarel_ has joined #openstack-infra | 10:01 | |
| *** Weifan has quit IRC | 10:02 | |
| *** ykarel has quit IRC | 10:04 | |
| *** rm_work has joined #openstack-infra | 10:06 | |
| *** yamamoto has quit IRC | 10:06 | |
| *** ramishra has joined #openstack-infra | 10:08 | |
| *** Weifan has joined #openstack-infra | 10:09 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 10:13 |
| *** kjackal has quit IRC | 10:13 | |
| *** ykarel_ is now known as ykarel | 10:14 | |
| *** Weifan has quit IRC | 10:16 | |
| *** rtjure has joined #openstack-infra | 10:18 | |
| *** ykarel_ has joined #openstack-infra | 10:19 | |
| *** yamamoto has joined #openstack-infra | 10:22 | |
| *** ykarel has quit IRC | 10:22 | |
| *** ykarel_ is now known as ykarel | 10:23 | |
| *** yamamoto has quit IRC | 10:24 | |
| *** ykarel is now known as ykarelaway | 10:33 | |
| *** dtantsur|brb is now known as dtantsur | 10:35 | |
| *** candles has joined #openstack-infra | 10:36 | |
| *** nicolasbock has joined #openstack-infra | 10:36 | |
| *** yamamoto has joined #openstack-infra | 10:39 | |
| *** yamamoto has quit IRC | 10:41 | |
| *** gmann has quit IRC | 10:44 | |
| *** Weifan has joined #openstack-infra | 10:46 | |
| *** kjackal has joined #openstack-infra | 10:48 | |
| *** Weifan has quit IRC | 10:48 | |
| *** yamamoto has joined #openstack-infra | 11:02 | |
| *** ykarelaway is now known as ykarel | 11:05 | |
| *** happyhemant has joined #openstack-infra | 11:13 | |
| *** ginopc has quit IRC | 11:13 | |
| *** udesale has quit IRC | 11:16 | |
| *** jpena is now known as jpena|lunch | 11:18 | |
| *** yamamoto has quit IRC | 11:21 | |
| *** aaronsheffield has joined #openstack-infra | 11:25 | |
| *** rtjure has quit IRC | 11:28 | |
| *** zhangfei has quit IRC | 11:30 | |
| *** jbadiapa has quit IRC | 11:33 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:34 | |
| *** yamamoto has joined #openstack-infra | 11:36 | |
| *** cloudnull has joined #openstack-infra | 11:37 | |
| *** ccamacho has joined #openstack-infra | 11:44 | |
| *** Emine has joined #openstack-infra | 11:44 | |
| *** helenafm has quit IRC | 11:45 | |
| *** jbadiapa has joined #openstack-infra | 11:46 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 11:50 | |
| *** bhavikdbavishi has quit IRC | 11:50 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 11:50 | |
| *** yamamoto has quit IRC | 11:55 | |
| *** yamamoto has joined #openstack-infra | 11:56 | |
| *** quiquell|rover is now known as quique|rover|eat | 12:00 | |
| *** rlandy has joined #openstack-infra | 12:01 | |
| *** bhavikdbavishi has quit IRC | 12:01 | |
| *** rlandy is now known as rlandy|ruck | 12:02 | |
| *** jcoufal has joined #openstack-infra | 12:03 | |
| *** yamamoto has quit IRC | 12:05 | |
| *** amoralej is now known as amoralej|lunch | 12:06 | |
| *** markvoelker has quit IRC | 12:08 | |
| *** jamesmcarthur has joined #openstack-infra | 12:10 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:14 | |
| *** jamesmcarthur has quit IRC | 12:15 | |
| *** jamesmcarthur has joined #openstack-infra | 12:16 | |
| AJaeger | mordred: we need https://review.opendev.org/654238 merged before your nodepool change https://review.opendev.org/654230 can merge - but 238 fails testsuide | 12:18 |
| AJaeger | asettle: I don't remember - and I'm fine with removing it. | 12:19 |
| asettle | Coolio. thanks AJaeger | 12:19 |
| AJaeger | tristanC: indeed, that's why mordred pushed https://review.opendev.org/655238 - needs another +2 | 12:20 |
| *** yamamoto has joined #openstack-infra | 12:22 | |
| *** yamamoto has quit IRC | 12:24 | |
| *** yamamoto has joined #openstack-infra | 12:24 | |
| *** candles has quit IRC | 12:24 | |
| *** yamamoto has quit IRC | 12:25 | |
| *** udesale has joined #openstack-infra | 12:29 | |
| *** jamesmcarthur has quit IRC | 12:32 | |
| *** quique|rover|eat is now known as quiquell|rover | 12:41 | |
| tristanC | AJaeger: alright, thanks | 12:41 |
| *** kgiusti has joined #openstack-infra | 12:42 | |
| *** sthussey has joined #openstack-infra | 12:43 | |
| *** jamesmcarthur has joined #openstack-infra | 12:44 | |
| *** eharney has quit IRC | 12:50 | |
| *** yamamoto has joined #openstack-infra | 12:52 | |
| *** yamamoto has quit IRC | 12:52 | |
| *** yamamoto has joined #openstack-infra | 12:52 | |
| *** markmcd has quit IRC | 12:52 | |
| *** irclogbot_2 has quit IRC | 12:55 | |
| *** irclogbot_1 has joined #openstack-infra | 12:55 | |
| *** jistr is now known as jistr|afk | 12:56 | |
| *** altlogbot_3 has quit IRC | 12:57 | |
| *** yamamoto has quit IRC | 12:57 | |
| *** yamamoto has joined #openstack-infra | 12:57 | |
| *** markmcd has joined #openstack-infra | 12:57 | |
| *** altlogbot_2 has joined #openstack-infra | 12:58 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Update references for opendev https://review.opendev.org/654238 | 12:59 |
| *** gfidente has quit IRC | 13:01 | |
| *** markmcd has quit IRC | 13:01 | |
| *** gfidente has joined #openstack-infra | 13:01 | |
| *** kranthikirang has joined #openstack-infra | 13:05 | |
| *** markmcd has joined #openstack-infra | 13:07 | |
| openstackgerrit | Sean McGinnis proposed zuul/zuul-jobs master: ensure-twine: Don't install --user if running in venv https://review.opendev.org/655241 | 13:09 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 13:11 | |
| *** amoralej|lunch is now known as amoralej | 13:17 | |
| *** rfolco is now known as rfolco_sick | 13:18 | |
| *** markmcd has quit IRC | 13:20 | |
| openstackgerrit | Sean McGinnis proposed zuul/zuul-jobs master: ensure-twine: Don't install --user if running in venv https://review.opendev.org/655241 | 13:22 |
| *** ykarel is now known as ykarel|away | 13:23 | |
| *** Goneri has joined #openstack-infra | 13:23 | |
| *** bhavikdbavishi has quit IRC | 13:23 | |
| *** markmcd has joined #openstack-infra | 13:24 | |
| *** markmcd has left #openstack-infra | 13:27 | |
| *** lseki has joined #openstack-infra | 13:32 | |
| *** smarcet has joined #openstack-infra | 13:33 | |
| *** rkukura has quit IRC | 13:36 | |
| *** jpena|lunch is now known as jpena | 13:40 | |
| openstackgerrit | Brian Haley proposed openstack/project-config master: Update networking-ovn dashboard https://review.opendev.org/655433 | 13:42 |
| openstackgerrit | Ildiko Vancsa proposed openstack/project-config master: Create election repository for StarlingX https://review.opendev.org/655435 | 13:42 |
| rpittau | hello fellow openstackers! we're experiencing a weird issue in the ironic jobs in stable branches where, even if a project is in the required-projects list, it's not correctly cloned | 13:44 |
| rpittau | for instance http://logs.openstack.org/98/655398/1/check/ironic-dsvm-standalone/52ca325/job-output.txt.gz#_2019-04-24_10_51_12_181429 | 13:44 |
| openstackgerrit | Sean McGinnis proposed zuul/zuul-jobs master: Add environment debugging to ensure-twine role https://review.opendev.org/655437 | 13:44 |
| rpittau | it seems working correctly on master | 13:44 |
| rpittau | and weird thing, it works if it's a legacy project and the project is added to the PROJECTS variable explicitly | 13:45 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea templates for 1.8.0 https://review.opendev.org/655439 | 13:45 |
| rpittau | any thought on why this might happen? | 13:46 |
| *** ykarel|away has quit IRC | 13:46 | |
| mordred | corvus: ^^ that patch should allow making a jeblair/gitea branch based on 1.8.0 if we want | 13:47 |
| mordred | corvus: although if we decide to do that, we should probably just make a second branch in your repo, then have that patch update the git ref too | 13:47 |
| mordred | rpittau: looking | 13:48 |
| rpittau | thanks mordred :) | 13:48 |
| sshnaidm | hi, if I run ansible separately inside the job (like ansible-playbook my_playbook.yaml ...) and want to have there job variables, what is best practice to do it? | 13:48 |
| *** eharney has joined #openstack-infra | 13:48 | |
| mordred | rpittau: oh - there's a devstack patch we should backport to stable devstacks I think | 13:49 |
| mordred | rpittau: one sec, lemme make the backport patches | 13:50 |
| rpittau | mordred: thanks for checking! Was going a bit crazy here :D | 13:50 |
| *** gmann has joined #openstack-infra | 13:52 | |
| *** rfolco_sick is now known as rfolco | 13:52 | |
| mordred | rpittau: https://review.opendev.org/#/q/I8a55522a5fee46f415f0c0ce580ded3476133460 | 13:53 |
| *** mriedem has joined #openstack-infra | 13:53 | |
| mordred | rpittau: that should take care of the issues on the stable branches | 13:53 |
| rpittau | mordred: thanks again! I'll wait for that to merge before rechecking on our side | 13:54 |
| mordred | frickler, clarkb: ^^ if you have a sec, those are all backports of the namespace addition patches | 13:54 |
| corvus | mordred: cool -- i've rebased my redirect patches on 1.8.0 as i'm debugging them, so maybe once i get that worked out we can do the upgrade | 13:54 |
| mordred | corvus: awesome | 13:54 |
| corvus | mordred: i've learned that the test failure is due to a behavior change for private repos, which explains why i didn't catch it in my quick testing on friday, and it's probably not a big deal for our production site | 13:54 |
| mordred | corvus: if you have a sec, same ping re the link just above - quick/easy devstack backport patches for the namespace stuff | 13:55 |
| corvus | still haven't quite worked out the fix | 13:55 |
| mordred | corvus: ah - cool | 13:55 |
| mordred | corvus: incidentally, the template updates include support for showing if a repo is archived or not - happy about that new feature being in 1.8.0 | 13:55 |
| corvus | mordred: yeah, i mentioned that yesterday :) that will be nice | 13:56 |
| mordred | corvus: yeah | 13:56 |
| *** kmarc has quit IRC | 13:58 | |
| AJaeger | config-core, there're a couple of smaller changes up to review, could you put the following on your queue, please? https://review.opendev.org/652111 https://review.opendev.org/652947 https://review.opendev.org/653364 https://review.opendev.org/652993 https://review.opendev.org/655178 https://review.opendev.org/655433 | 13:59 |
| AJaeger | infra-root, should we enable ovh again? See https://review.opendev.org/653879 | 13:59 |
| openstackgerrit | Sean McGinnis proposed zuul/zuul-jobs master: Add environment debugging to ensure-twine role https://review.opendev.org/655437 | 14:00 |
| *** jamesmcarthur has quit IRC | 14:00 | |
| AJaeger | config-core, one more: https://review.opendev.org/#/c/654241/ | 14:00 |
| *** jamesmcarthur has joined #openstack-infra | 14:00 | |
| *** udesale has quit IRC | 14:06 | |
| *** udesale has joined #openstack-infra | 14:07 | |
| *** udesale has quit IRC | 14:08 | |
| *** udesale has joined #openstack-infra | 14:08 | |
| openstackgerrit | Brian Rosmaita proposed openstack/devstack-gate master: Remove CINDER_PERIODIC_INTERVAL setting https://review.opendev.org/655451 | 14:09 |
| fungi | has anyone seen any more signs of ssh dropouts so far today? | 14:16 |
| * AJaeger has seen a quite stable system and no complaints - that's why I suggest to renambe ovh-bhs again with 653879 | 14:18 | |
| *** ykarel has joined #openstack-infra | 14:18 | |
| openstackgerrit | Merged opendev/system-config master: Use internal gitweb instead of gitea for now https://review.opendev.org/655238 | 14:20 |
| fungi | sounds like a good idea to me, but maybe we give it another couple hours just to make sure we don't see any resurgence of yesterday's issues | 14:21 |
| clarkb | re: ovh my VM can lookup dns records against 1.1.1.1 currently | 14:28 |
| *** gfidente has quit IRC | 14:28 | |
| clarkb | which was failing when we turned it off | 14:28 |
| *** gfidente has joined #openstack-infra | 14:29 | |
| *** yamamoto has quit IRC | 14:35 | |
| *** electrofelix has quit IRC | 14:37 | |
| *** sreejithp has joined #openstack-infra | 14:42 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add environment debugging to ensure-twine role https://review.opendev.org/655437 | 14:42 |
| *** igordc has joined #openstack-infra | 14:42 | |
| smcginnis | fungi: ^ :) | 14:43 |
| openstackgerrit | Bernard Cafarelli proposed openstack/project-config master: Update Grafana dashboards for stable Neutron releases https://review.opendev.org/653354 | 14:43 |
| openstackgerrit | Ildiko Vancsa proposed openstack/project-config master: Create election repository for StarlingX https://review.opendev.org/655435 | 14:43 |
| fungi | thanks smcginnis, reenqueuing again | 14:44 |
| *** iurygregory_ has joined #openstack-infra | 14:46 | |
| *** jamesmcarthur has quit IRC | 14:48 | |
| mriedem | do url depends-on work yet? | 14:48 |
| mriedem | e.g. Depends-On: https://review.opendev.org/653514/ | 14:48 |
| clarkb | mriedem: they should haev always worked with the new domain name | 14:49 |
| mriedem | oh ok | 14:49 |
| clarkb | (my email was about the old domain not working) | 14:49 |
| mriedem | just not review.openstack.org | 14:49 |
| mriedem | yeah ok | 14:49 |
| *** iurygregory has quit IRC | 14:49 | |
| *** quiquell|rover is now known as quiquell|off | 14:49 | |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: Fix LP access credentials https://review.opendev.org/655465 | 14:53 |
| smcginnis | fungi: Hmm, apparently I really messed that debug print up: http://logs.openstack.org/59/59ce65e9e66fe3ea203b77812ee14e69ebdb192a/release/release-openstack-python/abaae0d/job-output.txt.gz#_2019-04-24_14_51_49_340224 | 14:54 |
| *** michael-beaver has joined #openstack-infra | 14:55 | |
| smcginnis | Ah, -debug: message: "{{ foo }}" | 14:55 |
| openstackgerrit | Sean McGinnis proposed zuul/zuul-jobs master: Correct debug statement for ensure-twine environment https://review.opendev.org/655466 | 14:57 |
| smcginnis | fungi: ^ | 14:57 |
| *** iurygregory_ is now known as iurygregory | 14:58 | |
| fungi | mriedem: right, the point was that as of the cut-over during the maintenance, zuul then knew our gerrit as review.opendev.org and had no knowledge of a review.openstack.org so ignored the old depends-on footers with those earlier urls | 14:59 |
| *** dave-mccowan has joined #openstack-infra | 15:01 | |
| openstackgerrit | Merged opendev/puppet-nodepool master: Install yappi and objgraph packages https://review.opendev.org/654462 | 15:05 |
| *** pcaruana has quit IRC | 15:06 | |
| *** dave-mccowan has quit IRC | 15:07 | |
| *** jamesmcarthur has joined #openstack-infra | 15:07 | |
| *** amotoki_ is now known as amotoki | 15:08 | |
| *** rh-jelabarre has joined #openstack-infra | 15:09 | |
| *** ccamacho has quit IRC | 15:15 | |
| *** diablo_rojo has joined #openstack-infra | 15:15 | |
| prometheanfire | for the new gertty, you should probably update and add py3 to setup.cfg if you claim support :P | 15:16 |
| prometheanfire | https://opendev.org/ttygroup/gertty/src/branch/master/setup.cfg | 15:16 |
| *** rlandy|ruck|mtg is now known as rlandy | 15:16 | |
| prometheanfire | I'll state support for py35,36,37 though | 15:17 |
| *** yamamoto has joined #openstack-infra | 15:17 | |
| *** ricolin has quit IRC | 15:17 | |
| *** rfolco is now known as rfolco|ruck | 15:19 | |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Update python version trove classifiers https://review.opendev.org/655471 | 15:19 |
| corvus | prometheanfire, fungi: how's that look? ^ | 15:19 |
| prometheanfire | corvus: I was using master as of a week ago with py2, it seems to work | 15:21 |
| prometheanfire | using it now in fact | 15:21 |
| corvus | prometheanfire: ok, want me to stick py2 back in there? | 15:21 |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Update python version trove classifiers https://review.opendev.org/655471 | 15:22 |
| openstackgerrit | Merged openstack/project-config master: Fix LP access credentials https://review.opendev.org/655465 | 15:22 |
| fungi | i think it's fine to list python versions you expect to work, or versions you want to continue supporting with that release | 15:22 |
| *** sshnaidm is now known as sshnaidm|off | 15:23 | |
| AJaeger | mordred: are https://review.opendev.org/#/c/654034/2 and https://review.opendev.org/654040 obsolete with other changes that merged - or do they need rebase and resolve of conflicts? | 15:23 |
| AJaeger | corvus: want to abandon https://review.opendev.org/654629 - another gitea double stack change on system-config? | 15:23 |
| *** yamamoto has quit IRC | 15:24 | |
| clarkb | re the starlingx request to talk about things with us. I'm going to propose midday friday if that works? I expect thursday we'll all be wanting to dive right into infra specific stuff so friday is better for cross projecty stuff | 15:25 |
| clarkb | let me know if I've missed somethign that makes that undesireable scheduling | 15:25 |
| prometheanfire | fungi: that's true, I packaged it with py27/35/36/37 | 15:25 |
| prometheanfire | next will be py3 though | 15:25 |
| clarkb | corvus: ^ I know you leave on friday. Does just after the lunch break work for you or should we plan for before? | 15:26 |
| *** ricolin has joined #openstack-infra | 15:28 | |
| corvus | clarkb: sgtm | 15:28 |
| corvus | clarkb: i'm probably good until 4ish? maybe later? haven't quite worked it out yet | 15:28 |
| clarkb | corvus: ok that should be enough if we have an hour ish meeting at 1 ish | 15:28 |
| *** armax has joined #openstack-infra | 15:29 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Correct debug statement for ensure-twine environment https://review.opendev.org/655466 | 15:31 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add release-zuul-python to post pipeline https://review.opendev.org/655474 | 15:32 |
| *** kjackal has quit IRC | 15:33 | |
| corvus | infra-root: i have updated https://github.com/go-gitea/gitea/pull/6688 with a new approach which works for public and private repos; it passes tests locally | 15:33 |
| mordred | corvus: woot | 15:34 |
| corvus | mordred: should i push an "opendev-180" branch and update your patch and start the upgrade? | 15:34 |
| mordred | corvus: yeah - I think so | 15:35 |
| clarkb | I'm up for that. Especially since I don't plan to be around much tomorrow or day after (so if I'm gonna be around to help today is the day) | 15:35 |
| corvus | mordred: oh, we need to get the registry fixed | 15:35 |
| smcginnis | fungi: That debug msg fix merged. Mind reenqueuing that again when you have a chance? | 15:35 |
| clarkb | I'm in a meeting now. Then will get the renaming playbook updates pushed. Then will start on a summit schedule for myself and can interrupt that for gitea things | 15:35 |
| mordred | corvus: oh - yeah. the secret update didn't seem to, you know, fix it | 15:35 |
| AJaeger | smcginnis: I have just rechecked a job that uses the role... | 15:36 |
| AJaeger | smcginnis: http://logs.openstack.org/77/654477/1/check/test-release-openstack/24f58a8/ passed! | 15:36 |
| AJaeger | that one failed 10 mins ago... | 15:37 |
| corvus | mordred: i'll double check your secret change | 15:37 |
| mordred | corvus: it's entirely possible I don't know how to computer | 15:37 |
| smcginnis | AJaeger: Hmm, that's actually too bad. We want to capture a failure. | 15:37 |
| smcginnis | Odd that it didn't actually print anything: http://logs.openstack.org/77/654477/1/check/test-release-openstack/24f58a8/job-output.txt.gz#_2019-04-24_15_35_17_935265 | 15:38 |
| smcginnis | Does the verbosity need to be raised? | 15:38 |
| mordred | smcginnis: isn't life weird when we *want* things to break? | 15:38 |
| smcginnis | ;) | 15:38 |
| AJaeger | smcginnis: http://logs.openstack.org/77/654477/1/check/test-release-openstack/24f58a8/ara-report/ | 15:38 |
| AJaeger | smcginnis: better: http://logs.openstack.org/77/654477/1/check/test-release-openstack/24f58a8/ara-report/result/8537481b-3908-4016-8165-0b4650aeb758/ | 15:38 |
| smcginnis | Oh, great! | 15:39 |
| smcginnis | That looks like what we should expect every time. So now it will be interesting to compare that to one of the failing cases to see what is different. And hopefully it points us to what is causing it to be different in the first place. | 15:39 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Omnibus rename repo playbook fixes https://review.opendev.org/655476 | 15:40 |
| clarkb | infra-root ^ that includes fixes from friday's playbooking | 15:40 |
| clarkb | feel free to amend it with coauthored by lines if you fixed one of thsoe on friday (I don't recall all who edited that stuff) | 15:40 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Correct intermediate registry key https://review.opendev.org/655477 | 15:43 |
| corvus | infra-root: ^ | 15:43 |
| *** amansi26 has joined #openstack-infra | 15:45 | |
| clarkb | looking | 15:45 |
| *** pcaruana has joined #openstack-infra | 15:46 | |
| corvus | clarkb: i have a suggestion on https://review.opendev.org/655476 | 15:48 |
| clarkb | corvus: makes sense. I'll do that update | 15:48 |
| *** iurygregory has quit IRC | 15:49 | |
| fungi | smcginnis: sorry, catching up after a meeting, reenqueued now | 15:49 |
| smcginnis | fungi: Thanks. If this fails, we will have a good comparison with a passing case. | 15:49 |
| smcginnis | Is there a way to see which executor it would have run on? Maybe we can compare if jobs pass on one but not others? | 15:50 |
| clarkb | smcginnis: it is in the inventory file iirc | 15:51 |
| *** pgaxatte has quit IRC | 15:51 | |
| smcginnis | OK, good. Maybe another variable we can investigate if we need to. | 15:51 |
| pabelanger | corvus: +3 | 15:51 |
| pabelanger | thanks! | 15:51 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Omnibus rename repo playbook fixes https://review.opendev.org/655476 | 15:52 |
| clarkb | corvus: ^ that better? | 15:52 |
| corvus | clarkb: thanks! | 15:52 |
| corvus | clarkb: derp, tiny fix needed | 15:53 |
| *** diablo_rojo has quit IRC | 15:53 | |
| Shrews | clarkb: Is https://docs.openstack.org/infra/system-config/puppet.html up-to-date? | 15:55 |
| corvus | Shrews: it's about half right | 15:55 |
| corvus | it's.... "cron driving ansible driving puppet" for many hosts | 15:56 |
| corvus | and "cron driving ansible" for some | 15:56 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Omnibus rename repo playbook fixes https://review.opendev.org/655476 | 15:56 |
| clarkb | my bad copy pasta sorry | 15:56 |
| Shrews | and there doesn't appear to be a puppetmaster host | 15:56 |
| *** diablo_rojo has joined #openstack-infra | 15:56 | |
| corvus | Shrews: right, we call it "bridge" now | 15:56 |
| smcginnis | fungi: Definitely running within ansibles venv. | 15:56 |
| smcginnis | Failure: http://logs.openstack.org/59/59ce65e9e66fe3ea203b77812ee14e69ebdb192a/release/release-openstack-python/781edd8/ara-report/result/a6f1d958-2d17-4b8a-a6bf-97cb1bf6e31d/ | 15:56 |
| corvus | we do need to update that page | 15:56 |
| *** bgmccollum has joined #openstack-infra | 15:57 | |
| smcginnis | Pass: http://logs.openstack.org/77/654477/1/check/test-release-openstack/24f58a8/ara-report/result/8537481b-3908-4016-8165-0b4650aeb758/ | 15:57 |
| mordred | yeah - I almost deleted it when doing the docs updates | 15:57 |
| mordred | but corvus pointed out that half of it is still real | 15:57 |
| AJaeger | smcginnis: one running on localhost, the other on the remote host. | 15:58 |
| Shrews | corvus: yeah, i figured it was on bridge now, but wasn't quite sure how things operated there (beyond the ansibling) | 15:58 |
| AJaeger | smcginnis: let me rephrase: One running on executor, the other on remote host? | 15:58 |
| corvus | Shrews: "run_all.sh" is called by cron -- that's the root of the whole thing | 15:58 |
| smcginnis | AJaeger: Where do I see that? | 15:58 |
| AJaeger | smcginnis: second line "host: localhost" and "host: ubuntu-bionic". AFAIU "localhost" means executor | 15:59 |
| AJaeger | fungi, corvus, what do you make of these twine findings? ^ | 16:00 |
| corvus | let's move to #zuul | 16:00 |
| openstackgerrit | Merged opendev/base-jobs master: Correct intermediate registry key https://review.opendev.org/655477 | 16:00 |
| *** tosky has quit IRC | 16:00 | |
| smcginnis | AJaeger: Oh, I completely looked right past that information. | 16:00 |
| smcginnis | Running locally could explain why it would fail. | 16:00 |
| *** lucasagomes has quit IRC | 16:01 | |
| *** rpittau is now known as rpittau|afk | 16:03 | |
| *** _erlon_ has joined #openstack-infra | 16:03 | |
| *** efried is now known as efried_rollin | 16:04 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Trim the openstack root list https://review.opendev.org/655487 | 16:04 |
| clarkb | infra-root ^ fyi | 16:04 |
| *** kambiz has quit IRC | 16:07 | |
| *** kambiz has joined #openstack-infra | 16:08 | |
| *** altlogbot_2 has quit IRC | 16:09 | |
| Shrews | infra-root: I'd like to restart the nodepool launchers to pick up changes for dumping thread/obj data with sigusr2. Objections? | 16:11 |
| mordred | Shrews: none from me | 16:11 |
| *** ykarel is now known as ykarel|away | 16:11 | |
| *** altlogbot_3 has joined #openstack-infra | 16:12 | |
| clarkb | none from me | 16:12 |
| pabelanger | Shrews: +1 | 16:13 |
| fungi | Shrews: sounds great | 16:13 |
| Shrews | excellent | 16:13 |
| fungi | nice to finally have that added | 16:13 |
| fungi | it's been super useful in zuul | 16:13 |
| clarkb | ok Monday evening Lowry Beer Garden trip is official. Just sent email out with details | 16:14 |
| *** ianychoi has quit IRC | 16:14 | |
| clarkb | We'll figure out something day of if the weather turns bad (current forecasting keeps going back and forth) | 16:14 |
| *** ianychoi has joined #openstack-infra | 16:15 | |
| Shrews | #status log Restarted nodepool launchers on commit f8ac79661a8d2e2ee980e511c9bdf19a41d156f8 | 16:16 |
| openstackstatus | Shrews: finished logging | 16:16 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Revert "Prepend path with bin dir of ansible virtualenv" https://review.opendev.org/655491 | 16:19 |
| *** ricolin has quit IRC | 16:20 | |
| fungi | infra-root: once ^ merges (hopefully later today?) we're probably going to want to do another executor restart | 16:21 |
| mordred | fungi: ++ | 16:21 |
| fungi | after that, there's a slew of failed release tags i need to reenqueue | 16:22 |
| *** ccamacho has joined #openstack-infra | 16:25 | |
| clarkb | I'm told everyone should have gotten email about this already (but I didn't maybe beacuse i'm osf staff? so posting here too) https://www.eventbrite.com/e/the-denver-party-during-open-infrastructure-summit-tickets-58863817262 is the summit party signup site. password is "denver" and for the phone number you can just put in 555-555-5555 | 16:28 |
| clarkb | (I don't know how important the rsvp actually is but figure if you possibly want to check it out signing up now is a good idea) | 16:29 |
| *** dtantsur is now known as dtantsur|afk | 16:29 | |
| fungi | there are a bunch of folks sharing that telephone | 16:29 |
| fungi | must be the payphone in the summit hotel lobby | 16:29 |
| fungi | also, probably worth pointing out, this is sort of like what happened with the "pub crawl" in berlin... some sponsors decided to throw it and told the summit event coordinators about it, but it's not really coordinated by the foundation | 16:31 |
| smcginnis | I had not seen anything about RSVPing for anything there. | 16:31 |
| clarkb | smcginnis: you have to type in the password first | 16:31 |
| fungi | hence the creepy requests for personal details on the sign-up form | 16:31 |
| corvus | or we can go back to the beer garden | 16:32 |
| fungi | yeah, i hear the beer garden doesn't ask for phone numbers | 16:32 |
| corvus | aka "the denver office" | 16:32 |
| clarkb | ya this isn't an indication I think we should all go to this party | 16:32 |
| fungi | just what beer you're buying | 16:32 |
| clarkb | more of a "its happening and I don't want people to feel they missed out due to potential communication issues" | 16:32 |
| *** jpich has quit IRC | 16:33 | |
| *** derekh has quit IRC | 16:35 | |
| anteaya | fungi: places ask for phone numbers to serve you in denver? | 16:35 |
| anteaya | oh sorry eventbrite does | 16:36 |
| fungi | anteaya: the marketing companies sponsoring the party clarkb linked to ask for phone numbers | 16:36 |
| anteaya | glad that many fives work in place of a real number | 16:36 |
| fungi | presumably because they are trading mediocre beer and finger food for the ability to call and try to sell you things as soon as you get home | 16:37 |
| *** rkukura has joined #openstack-infra | 16:37 | |
| anteaya | good luck with that | 16:37 |
| *** ijw has joined #openstack-infra | 16:37 | |
| *** ccamacho has quit IRC | 16:38 | |
| fungi | oh, before i forget, possible gitea bug? ttx noticed that if you browse to https://zuul.opendev.org/t/openstack/builds?job_name=release-openstack-python&project=openstack%2Fpython-freezerclient and click on the "change" there (it's a tag object) gitea returns a cryptic error, but other tags there work fine | 16:41 |
| fungi | and i need to disappear for a dental checkup, but should return shortly. bbiab | 16:41 |
| *** ijw has quit IRC | 16:42 | |
| *** altlogbot_3 has quit IRC | 16:43 | |
| *** manjeets_ is now known as manjeets | 16:43 | |
| *** altlogbot_3 has joined #openstack-infra | 16:44 | |
| *** gyee has joined #openstack-infra | 16:44 | |
| *** igordc has quit IRC | 16:45 | |
| *** amansi26 has quit IRC | 16:45 | |
| *** mattw4 has joined #openstack-infra | 16:47 | |
| mugsie | hey - would anyone in here know why a planet.openstack.org would not be showing new articles? | 16:49 |
| mugsie | I have a blog in the feed from the 20th, but it looks like the last time it got content was the 19th ... | 16:49 |
| clarkb | mugsie: it is beacuse the repo it uses to run planet has a git:// remote | 16:52 |
| clarkb | I'll fix it | 16:52 |
| *** altlogbot_3 has quit IRC | 16:53 | |
| *** psachin has quit IRC | 16:53 | |
| *** altlogbot_1 has joined #openstack-infra | 16:54 | |
| clarkb | #status log Moved /var/lib/planet/openstack aside on planet.o.o so that puppet will reclone it using the correct remote. This should fix planet publishing updates. | 16:54 |
| openstackstatus | clarkb: finished logging | 16:54 |
| clarkb | mugsie: ^ fyi next puppet run should get things sorted now | 16:54 |
| corvus | fungi, ttx: maybe so -- at first guess, the freezerclient issue may be related to the deleted file in that tagged change | 16:54 |
| openstackgerrit | Merged ttygroup/gertty master: Update python version trove classifiers https://review.opendev.org/655471 | 16:55 |
| corvus | mordred: did you restart the registry with the swift config? | 16:58 |
| corvus | i think the uptime is right for that, and i see no files in /var/registry/data or /opt/registry/data | 16:59 |
| mugsie | clarkb: thanks! | 16:59 |
| corvus | so i think the intermediate registry is using swift now | 16:59 |
| openstackgerrit | Jason Lee proposed opendev/storyboard master: WIP: Migrates Users to Blueprints, Writer Script WIP https://review.opendev.org/654812 | 17:01 |
| *** e0ne has quit IRC | 17:02 | |
| *** raissa has joined #openstack-infra | 17:05 | |
| *** ijw has joined #openstack-infra | 17:06 | |
| *** jpena is now known as jpena|off | 17:06 | |
| clarkb | corvus: that is really exciting. With swift + the GC'ing we should have sustainable intermediate registry now | 17:07 |
| corvus | clarkb: yeah, though we don't have gcing yet, i still need to refresh https://review.opendev.org/653148 | 17:09 |
| corvus | clarkb: but that can wait a week or 2 :) | 17:09 |
| clarkb | indeed | 17:09 |
| clarkb | I'm going to add it to the PTG ideas list though in case that is something people want to address there | 17:10 |
| *** mattmceuen has left #openstack-infra | 17:10 | |
| *** jamesmcarthur_ has joined #openstack-infra | 17:13 | |
| *** jamesmcarthur has quit IRC | 17:15 | |
| *** kopecmartin is now known as kopecmartin|off | 17:16 | |
| *** igordc has joined #openstack-infra | 17:16 | |
| clarkb | For anyone wanting to follow along I think I've started a pretty decent list at https://etherpad.openstack.org/p/2019-denver-ptg-infra-planning | 17:17 |
| AJaeger | Should we enable ovh again? See https://review.opendev.org/653879 | 17:18 |
| *** amoralej is now known as amoralej|off | 17:18 | |
| clarkb | AJaeger: yes I'll approve now | 17:19 |
| AJaeger | thanks | 17:20 |
| openstackgerrit | Ben Nemec proposed openstack/reviewstats master: Load subproject data from governance https://review.opendev.org/653024 | 17:22 |
| openstackgerrit | Ben Nemec proposed openstack/reviewstats master: Handle all exceptions loading pickled data https://review.opendev.org/653106 | 17:22 |
| openstackgerrit | Ben Nemec proposed openstack/reviewstats master: Switch default server to opendev.org https://review.opendev.org/655504 | 17:22 |
| openstackgerrit | Ben Nemec proposed openstack/reviewstats master: Don't fail when writing cache either https://review.opendev.org/655505 | 17:22 |
| openstackgerrit | Ben Nemec proposed openstack/reviewstats master: Switch to stestr https://review.opendev.org/655506 | 17:22 |
| openstackgerrit | Ben Nemec proposed openstack/reviewstats master: DNM: Debug zuul failure https://review.opendev.org/655507 | 17:22 |
| *** markvoelker has joined #openstack-infra | 17:22 | |
| *** yamamoto has joined #openstack-infra | 17:23 | |
| clarkb | ok etherpad now has a rough schedule | 17:26 |
| clarkb | I think the start PTG with a braindump idea will help get everyone up to speed as there have been many moving pieces and not everyone can keep it all paged in | 17:26 |
| clarkb | sets a foundation for further work at the PTG | 17:26 |
| *** markvoelker has quit IRC | 17:26 | |
| *** kjackal has joined #openstack-infra | 17:27 | |
| clarkb | infra-root can you please review https://review.opendev.org/#/c/655487/ when you have a moment. This trims the root list down per the email you were all cc'd on | 17:27 |
| clarkb | https://review.opendev.org/#/c/655476/ is the omnibus fix to rename projects which would be good to get in too | 17:27 |
| *** mattw4 has quit IRC | 17:28 | |
| openstackgerrit | Merged openstack/project-config master: Revert "nodepool: pause ovh" https://review.opendev.org/653879 | 17:32 |
| *** kjackal has quit IRC | 17:33 | |
| *** ramishra has quit IRC | 17:34 | |
| *** kjackal has joined #openstack-infra | 17:36 | |
| clarkb | dmsimard: is the openstack github admin team still in your ara github org? I get emails about activity there. If you are happy with the results of the move care to remove us? | 17:36 |
| *** markvoelker has joined #openstack-infra | 17:37 | |
| dmsimard | I think I took care of removing it but I'll double check | 17:37 |
| dmsimard | Yeah, it's removed | 17:38 |
| clarkb | thanks! | 17:38 |
| clarkb | time for some early lunch/late breakfast. back in a bit | 17:42 |
| *** eharney_ has joined #openstack-infra | 17:42 | |
| *** eharney has quit IRC | 17:44 | |
| *** eharney_ is now known as eharney | 17:45 | |
| *** electrofelix has joined #openstack-infra | 17:46 | |
| *** yamamoto has quit IRC | 17:47 | |
| *** udesale has quit IRC | 17:49 | |
| *** Weifan has joined #openstack-infra | 17:55 | |
| *** rkukura_ has joined #openstack-infra | 17:57 | |
| *** rkukura has quit IRC | 17:59 | |
| *** rkukura_ is now known as rkukura | 17:59 | |
| *** ralonsoh has quit IRC | 18:00 | |
| *** Goneri has quit IRC | 18:05 | |
| *** jamesmcarthur_ has quit IRC | 18:08 | |
| *** ijw has quit IRC | 18:11 | |
| *** ijw has joined #openstack-infra | 18:12 | |
| *** jamesmcarthur has joined #openstack-infra | 18:19 | |
| openstackgerrit | sebastian marcet proposed osf/openstackid-resources master: Added endpoints for speakers profile edit request https://review.opendev.org/655518 | 18:21 |
| *** ykarel|away has quit IRC | 18:22 | |
| *** kjackal has quit IRC | 18:26 | |
| mnaser | AJaeger: if you have sometime today https://review.opendev.org/655520 ! | 18:27 |
| *** happyhemant has quit IRC | 18:27 | |
| *** smarcet has quit IRC | 18:27 | |
| *** kjackal has joined #openstack-infra | 18:29 | |
| AJaeger | mnaser: will do. If you have time, please see above my ping for config-core ;) | 18:29 |
| mnaser | AJaeger: ill trade ya that | 18:29 |
| AJaeger | mnaser: LGTM | 18:30 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use local fork of gitea and upgrade to 1.8.0 https://review.opendev.org/655522 | 18:31 |
| corvus | clarkb, fungi, mordred: ^ that's my "use a local fork" patch squashed with mordred's "upgrade to 1.8.0" with an update to use a new branch of my gitea fork, which is 1.8.0 + the new version of the git clone redirect patch. | 18:31 |
| *** smarcet has joined #openstack-infra | 18:32 | |
| *** kjackal has quit IRC | 18:37 | |
| mnaser | AJaeger: I just kinda realized "Wayward Changes (Changes with no code review in the last 2days)" and "Needs Feedback (Changes older than 5 days that have not been reviewed by anyone)" is pretty much the same thing? | 18:37 |
| mnaser | except with different ages | 18:37 |
| mnaser | is this.. common.. or should I just converge them to one.. just kinda curious what folks do mostly :) | 18:38 |
| clarkb | I usually try to pick a day and get throw the entire backlog when not otherwise occupied | 18:39 |
| *** Weifan has quit IRC | 18:41 | |
| *** Weifan has joined #openstack-infra | 18:41 | |
| *** Weifan has quit IRC | 18:41 | |
| *** Weifan has joined #openstack-infra | 18:42 | |
| AJaeger | mnaser: there are subtile difference - but do whatever you like. | 18:42 |
| openstackgerrit | Jeremy Freudberg proposed openstack/project-config master: Minor/useful tweaks to Sahara repo descriptions https://review.opendev.org/655527 | 18:42 |
| *** Weifan has quit IRC | 18:42 | |
| *** Weifan has joined #openstack-infra | 18:42 | |
| *** Weifan has quit IRC | 18:43 | |
| *** Weifan has joined #openstack-infra | 18:43 | |
| mnaser | AJaeger: https://review.opendev.org/655528 thanks :) | 18:43 |
| fungi | i have simply resigned myself to the fact that my backlog grows unbounded, and bounce from problem to problem when people bring things to my attention | 18:44 |
| *** Weifan has quit IRC | 18:44 | |
| fungi | which is probably terrible, but gotta sleep sometime ;) | 18:44 |
| *** Weifan has joined #openstack-infra | 18:44 | |
| openstackgerrit | Merged openstack/project-config master: Announce stable branches for Kuryr https://review.opendev.org/652947 | 18:44 |
| AJaeger | config-core, could you review https://review.opendev.org/654241 https://review.opendev.org/653354 https://review.opendev.org/655527 , please? Then we can taggle the real stuff ;) | 18:44 |
| openstackgerrit | Merged openstack/project-config master: Update networking-ovn dashboard https://review.opendev.org/655433 | 18:45 |
| openstackgerrit | Merged openstack/project-config master: Switch py35 periodic jobs to py36 in Neutron's dashboard https://review.opendev.org/655178 | 18:45 |
| *** Weifan has quit IRC | 18:45 | |
| *** Weifan has joined #openstack-infra | 18:45 | |
| *** Weifan has quit IRC | 18:45 | |
| *** Weifan has joined #openstack-infra | 18:46 | |
| *** Weifan has quit IRC | 18:46 | |
| *** Weifan has joined #openstack-infra | 18:46 | |
| AJaeger | mnaser: sure. I thought there was a subtile difference - but looking at the conditions, it is only in wording ;( | 18:47 |
| mnaser | AJaeger: yep! | 18:47 |
| *** Weifan has quit IRC | 18:47 | |
| *** Weifan has joined #openstack-infra | 18:47 | |
| *** e0ne has joined #openstack-infra | 18:48 | |
| *** Weifan has quit IRC | 18:48 | |
| openstackgerrit | Merged openstack/project-config master: Add openstacksdk-functional-devstack-networking job to Neutron dashboard https://review.opendev.org/652993 | 18:49 |
| openstackgerrit | sebastian marcet proposed osf/openstackid-resources master: Added endpoints for speakers profile edit request https://review.opendev.org/655518 | 18:49 |
| openstackgerrit | sebastian marcet proposed osf/openstackid-resources master: Update Presentation Slide Serializer https://review.opendev.org/655532 | 18:53 |
| clarkb | corvus: AJaeger I'm back and pulling up those cahnges now | 18:56 |
| openstackgerrit | Merged openstack/project-config master: Adding the OUI training VM repo under docs https://review.opendev.org/653364 | 18:56 |
| clarkb | corvus: one thing is your repo will show the version as 1.7.3.stuff because there is no 1.8.0 tag I think | 18:57 |
| *** openstackgerrit has quit IRC | 18:57 | |
| clarkb | corvus: you might want to sync tags to your repo so taht we get a sane version reported? | 18:57 |
| clarkb | corvus: also is https://github.com/jeblair/gitea/commit/fa3078792e8e16a098d8de6e39eea079053bec4d the old fix for that or the new one that incorporates private repo support? | 18:58 |
| *** smarcet has quit IRC | 18:58 | |
| clarkb | we wait that is just general redirecting for git clones | 18:58 |
| clarkb | are we missing the fix for .git? | 18:58 |
| corvus | clarkb: it's unecessary (i left a comment on that) | 18:59 |
| corvus | clarkb: the old approach needed extra .git handling, but the new approach does not | 18:59 |
| *** Goneri has joined #openstack-infra | 18:59 | |
| corvus | clarkb: at least "git clone httsp://hostname/org/old.git" redirects correctly | 18:59 |
| clarkb | I see so the commit above fixes both | 18:59 |
| corvus | clarkb: that's what was broken, right? | 19:00 |
| clarkb | yes | 19:00 |
| corvus | ok. i tested that with the new patch | 19:00 |
| clarkb | also https://hostname/org/new.git was failing | 19:00 |
| clarkb | to git clone | 19:00 |
| corvus | ah, yeah. i'm pretty sure the first part of my old fix broke that. | 19:00 |
| *** rkukura has quit IRC | 19:00 | |
| corvus | clarkb: git clone https://try.gitea.io/stest12/Chrome_music_lab.git | 19:01 |
| corvus | that works | 19:01 |
| clarkb | ah | 19:01 |
| *** openstackgerrit has joined #openstack-infra | 19:01 | |
| openstackgerrit | Merged osf/openstackid-resources master: Update Presentation Slide Serializer https://review.opendev.org/655532 | 19:01 |
| corvus | clarkb: somehow my locally built gitea ended up reporting its version as opendev-1.8.0 | 19:02 |
| *** electrofelix has quit IRC | 19:03 | |
| clarkb | corvus: does `git tag` show you a 1.8.0 tag? | 19:03 |
| clarkb | I think your fork on github just needs the tag pushed to it so it has it too | 19:03 |
| corvus | that shouldn't cause it to say "opendev-1.8.0" | 19:03 |
| mordred | to be fair - opendev-1.8.0 is a great version to show | 19:03 |
| clarkb | (I'm basing this off of our 1.7.3. version on opendev.org now) | 19:04 |
| corvus | yeah, opendev-1.8.0 is the branch name | 19:05 |
| *** smarcet has joined #openstack-infra | 19:05 | |
| corvus | it's possible i built the 1.7.3 one with a slightly different dockerfile; this one is built with the one in https://review.opendev.org/655522 | 19:05 |
| clarkb | got it | 19:05 |
| corvus | maybe either setting GITEA_VERSION does it | 19:06 |
| clarkb | I've approved the change | 19:06 |
| corvus | s/either// | 19:06 |
| openstackgerrit | sebastian marcet proposed osf/openstackid-resources master: Update Presentation Slide Serializer https://review.opendev.org/655536 | 19:06 |
| corvus | neat! | 19:06 |
| corvus | clarkb, mordred, fungi: since it was handy, i tried pushing a full mirror of some repos to my local gertty | 19:06 |
| corvus | nova errored out, but looking at the logs, i suspect it was a timeout error because it was doing a lot of stuff to each ref that got pushed | 19:07 |
| fungi | local gitea? | 19:07 |
| corvus | fungi: yes....and here's the source of the typo | 19:07 |
| corvus | i also pushed gertty (as a much smaller repo) and it succeeded | 19:07 |
| fungi | cool, was trying to figure out how gertty fit in there | 19:08 |
| corvus | i am able to fetch change refs | 19:08 |
| clarkb | corvus: does that mean you think gitea 1.8.0 fixes the refs/changes problem? | 19:08 |
| corvus | no | 19:08 |
| corvus | i was mostly trying to remember what the problem was | 19:08 |
| clarkb | ah | 19:08 |
| fungi | is refs/notes also affected similarly? | 19:08 |
| corvus | what should i look for to test refs/notes? | 19:09 |
| clarkb | I think we don't replicate refs/notes because when we updated the replication rules we made them very specific to avoid refs/changes | 19:09 |
| clarkb | for that I think if you can push refs/notes to gitea maybe we just update our replication rules to allow that too? | 19:10 |
| fungi | i suspect if you pull up a commit view of a gerrit change, if gitea acts like gh then it may show you the git notes in the webui | 19:10 |
| AJaeger | thanks, clarkb | 19:11 |
| fungi | alternatively i add [remote "origin"] fetch = +refs/notes/*:refs/notes/* to my ~/.gitconfig and see them automatically in git show and git log | 19:11 |
| corvus | in github, where would i see that for this commit? https://github.com/openstack/gertty/commit/99124106aaf79e415ae2e16e993c8c5c5ed2ff4b | 19:11 |
| *** smarcet has quit IRC | 19:12 | |
| fungi | oh, you know what, it was cgit which displayed them... i don't actually know for sure gh did now that i think about it | 19:12 |
| openstackgerrit | Merged opendev/puppet-zuul master: Install objgraph package https://review.opendev.org/654599 | 19:12 |
| clarkb | https://github.blog/2010-08-25-git-notes-display/ fwiw github displayed them at one point in time | 19:13 |
| clarkb | but that post is almost 9 years old | 19:13 |
| openstackgerrit | Merged osf/openstackid-resources master: Update Presentation Slide Serializer https://review.opendev.org/655536 | 19:13 |
| corvus | what should they look like? | 19:13 |
| corvus | er | 19:13 |
| corvus | what should the ref path look like? | 19:13 |
| fungi | i'll clone directly from gerrit and find out | 19:14 |
| corvus | this is confusing. in my copy of the gertty.git directory from review.o.o, i only see this ref: http://paste.openstack.org/show/749717/ | 19:15 |
| corvus | a bare refs/notes/review -- with no change info | 19:15 |
| corvus | that's the only ref with 'notes' in it | 19:15 |
| corvus | that's for the most recently merged change | 19:15 |
| fungi | From https://review.openstack.org/ttygroup/gertty | 19:15 |
| fungi | * [new ref] refs/notes/review -> refs/notes/review | 19:15 |
| fungi | corvus: here's what git log looks like for me: http://paste.openstack.org/show/749718 | 19:16 |
| fungi | that's cloning directly from gerrit and fetching refs/notes/* | 19:17 |
| corvus | what am i missing? where are the other notes stored? | 19:17 |
| fungi | .git/refs/notes/review | 19:17 |
| corvus | are they magic that gerrit inserts from the database, but not actually in the git repo? | 19:17 |
| paladox | I doin't think github displays refs/notes/** (at least we replicate refs/* for some repos | 19:17 |
| corvus | on the filesystem in gerrit, there is a single ref called "refs/notes/review" which contains info only about the most recently submitted change | 19:18 |
| fungi | ahh, i see what you're asking... i don't know what the underlying implementation in git is for notes, but i do know that it seems to fetch them when they're available on the remote if i configure git to do so | 19:19 |
| fungi | and if they're fetched, then git will display them to me for all commits which have them (particularly those which correspond directly to gerrit changes) | 19:20 |
| mnasiadka | seems ovh bhs1 region has some dns resolution problems? http://logs.openstack.org/69/655369/1/gate/kolla-build-ubuntu-source/509c989/job-output.txt.gz#_2019-04-24_18_43_31_074787 | 19:20 |
| fungi | mnasiadka: that's unfortunate if so, it's the reason we had disabled it a while back and we reenabled today thinking it was fixed | 19:21 |
| mnasiadka | fungi: seems it's not, it's a fresh gate job | 19:21 |
| fungi | AJaeger: ^ i guess that answers your earlier question | 19:22 |
| fungi | "git.openstack.org: Temporary failure in name resolution" | 19:23 |
| *** _erlon_ has quit IRC | 19:23 | |
| fungi | i'll push up a revert of the revert | 19:23 |
| *** eharney has quit IRC | 19:23 | |
| AJaeger | argh ;( | 19:24 |
| AJaeger | clarkb: ^ | 19:24 |
| corvus | fungi: git clone https://review.opendev.org/ttygroup/gertty; cd gertty; git fetch origin +refs/notes/*:refs/notes/*; git log | 19:24 |
| corvus | fungi: that doesn't do anything different for me | 19:24 |
| openstackgerrit | Merged openstack/project-config master: Update links for opendev https://review.opendev.org/654241 | 19:24 |
| corvus | fungi: i still don't see any notes | 19:24 |
| clarkb | AJaeger: fungi ok fwiw I tested it on my personal VM but maybe it wasn't as good a test platform anymore | 19:24 |
| fungi | corvus: oh! also add [core] notesRef = refs/notes/review | 19:25 |
| AJaeger | corvus: you need to add to ~/.gitconfig a [notes] section with displayRef = refs/notes/review in it AFAIR | 19:25 |
| corvus | is there a way i can do this without modifying .gitconfig? | 19:25 |
| corvus | oh, i guess this could be repo local? | 19:25 |
| fungi | yes, could add it to .git/config i expect | 19:25 |
| corvus | okay, now git log shows notes, thanks | 19:27 |
| corvus | and so does the copy i cloned from gitea | 19:27 |
| openstackgerrit | Merged openstack/project-config master: Update Grafana dashboards for stable Neutron releases https://review.opendev.org/653354 | 19:27 |
| openstackgerrit | Merged openstack/project-config master: Minor/useful tweaks to Sahara repo descriptions https://review.opendev.org/655527 | 19:27 |
| corvus | fungi: so yes, however this is working (/me waves arms wildly) it seems to transit through gitea. i'm not seeing it display in gitea, but also, as paladox said, not github either. | 19:27 |
| clarkb | mugsie: your post is on planet now | 19:28 |
| fungi | right, i was somewhat spoilt by cgit displaying git notes by default | 19:28 |
| clarkb | I think that issue can be considered fixed. Thank you for bringing it to our attention | 19:28 |
| corvus | fungi: happily, we have the source to gitea :) | 19:28 |
| clarkb | I was thinking about ^ the other day in relation to our upgrade. I don't even want to think about how painful it would be to deal with the redirects issue if say using GHE and well it doesn't work | 19:29 |
| clarkb | something somethign free software is awesome | 19:29 |
| corvus | i concur | 19:30 |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Revert "Revert "nodepool: pause ovh"" https://review.opendev.org/655540 | 19:30 |
| fungi | mnasiadka: AJaeger: clarkb: ^ | 19:30 |
| clarkb | fungi: approved | 19:30 |
| mnasiadka | fungi: thanks | 19:31 |
| *** e0ne has quit IRC | 19:32 | |
| AJaeger | fungi, corvus, if I run "git config --add remote.origin.fetch refs/notes/review:refs/notes/review", then a "git pull" tells me "fatal: Couldn't find remote ref refs/notes/review". origin is opendev.org; with review.openstack.org it works | 19:32 |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: Output contents of /etc/hosts after modifying it https://review.opendev.org/655541 | 19:33 |
| fungi | AJaeger: right, we're not currently replicating refs/notes to gitea | 19:33 |
| fungi | hence the discussion | 19:33 |
| *** rlandy is now known as rlandy|biab | 19:33 | |
| fungi | we were replicating it to the old git server farm | 19:34 |
| fungi | mostly just trying to determine whether it's safe to replicate to gitea | 19:34 |
| * paladox remembers when we had that problem (with certain refs not being in diffusion). | 19:34 | |
| AJaeger | fungi: ah, ok. Then I misunderstood something in the discussion. | 19:35 |
| *** e0ne has joined #openstack-infra | 19:35 | |
| fungi | corvus was testing with a full git mirror of a subset of our repos to a local gitea he has running | 19:36 |
| AJaeger | fungi: now it makes sense to me... | 19:36 |
| fungi | anyway, i need to go grab some food, but will return forthwith | 19:36 |
| clarkb | as a heads up I have approved https://review.opendev.org/#/c/655487/ | 19:37 |
| *** kjackal has joined #openstack-infra | 19:37 | |
| *** openstackgerrit has quit IRC | 19:42 | |
| *** yamamoto has joined #openstack-infra | 19:45 | |
| *** Weifan has joined #openstack-infra | 19:47 | |
| *** smarcet has joined #openstack-infra | 19:51 | |
| *** dave-mccowan has joined #openstack-infra | 19:56 | |
| clarkb | random brainhole idea that just escaped: have status.opendev.org print whatever our last #status commnad was | 19:57 |
| clarkb | or something along those lines | 19:57 |
| * mnaser can suggest a deployment of cachethq | 19:59 | |
| mnaser | https://cachethq.io | 19:59 |
| clarkb | mnaser: I'm always wary of a status sytem that is more complicated than the system it reports on :P | 20:00 |
| clarkb | (certainly you get better data out of that and it looks niceer) | 20:00 |
| mnaser | heh, true, it's also something folks are used to see (because of the statuspage.io similarity) | 20:00 |
| *** e0ne has quit IRC | 20:08 | |
| *** yamamoto has quit IRC | 20:10 | |
| *** openstackgerrit has joined #openstack-infra | 20:11 | |
| openstackgerrit | Merged opendev/system-config master: Trim the openstack root list https://review.opendev.org/655487 | 20:11 |
| openstackgerrit | Merged openstack/project-config master: Revert "Revert "nodepool: pause ovh"" https://review.opendev.org/655540 | 20:11 |
| *** efried_rollin is now known as efried | 20:14 | |
| *** jcoufal has quit IRC | 20:16 | |
| clarkb | remote: https://review.opendev.org/655556 Update the project update for Denver 2019 | 20:19 |
| corvus | clarkb: there's a latent feature in statusbot for that; let me dig it up | 20:21 |
| *** mattw4 has joined #openstack-infra | 20:22 | |
| *** igordc has quit IRC | 20:22 | |
| *** igordc has joined #openstack-infra | 20:23 | |
| *** igordc has quit IRC | 20:23 | |
| corvus | clarkb: the alertfile interface writes a json file with the latest alert: https://opendev.org/opendev/statusbot/src/branch/master/statusbot/bot.py#L291 | 20:25 |
| corvus | corvus@eavesdrop01:~$ cat /var/lib/statusbot/www/alert.json | 20:25 |
| corvus | {"alert": null} | 20:25 |
| clarkb | cool so we just need a little bit of js and a webserver to render that | 20:25 |
| corvus | exactly | 20:25 |
| *** gtmanfred has quit IRC | 20:25 | |
| corvus | apache isn't configured to serve that dir, but that should be easy | 20:26 |
| corvus | just make sure to add a cors header | 20:26 |
| corvus | then fetch that from anywhere | 20:26 |
| corvus | gerrit, zuul, whatever | 20:26 |
| *** gtmanfred has joined #openstack-infra | 20:26 | |
| corvus | back to the refs replication thing -- it seems that the main thing is that gitea is doing too much processing on those refs and timing out | 20:27 |
| corvus | it's possible that we could bypass gitea and do an initial push of the repo with it offline, but i worry that would complicate the process of bringing up servers | 20:28 |
| corvus | so we should probably pursue the idea of making gitea ignore them | 20:28 |
| *** raissa has quit IRC | 20:29 | |
| *** raissa has joined #openstack-infra | 20:30 | |
| *** raissa has quit IRC | 20:30 | |
| *** raissa has joined #openstack-infra | 20:31 | |
| *** raissa has quit IRC | 20:31 | |
| openstackgerrit | sebastian marcet proposed osf/openstackid-resources master: Update allowed slides attachment extensions https://review.opendev.org/655564 | 20:31 |
| *** raissa has joined #openstack-infra | 20:32 | |
| *** raissa has quit IRC | 20:32 | |
| corvus | 2019/04/24 18:51:30 [...io/gitea/cmd/hook.go:109 runHookPreReceive()] [F] retrieve protected branches information failed: Get http://web:3000/api/internal/branch/4/refs/changes/16/89916/2: dial tcp 172.20.0.2:3000: connect: cannot assign requested address | 20:33 |
| corvus | wow, maybe that wasn't a timeout | 20:33 |
| corvus | maybe that was a DOS on the internal rest api | 20:33 |
| openstackgerrit | Merged osf/openstackid-resources master: Update allowed slides attachment extensions https://review.opendev.org/655564 | 20:33 |
| *** raissa has joined #openstack-infra | 20:34 | |
| *** raissa has quit IRC | 20:34 | |
| *** Goneri has quit IRC | 20:34 | |
| clarkb | heads up that zuul memory use has skyrocketed again | 20:35 |
| clarkb | on the scheduler specifically | 20:35 |
| mnaser | hey uh, did we end up fixing the gitweb urls? | 20:35 |
| clarkb | we still have ~7GB headroom | 20:35 |
| *** raissa has joined #openstack-infra | 20:35 | |
| clarkb | mnaser: not yet | 20:35 |
| mnaser | ack okay | 20:35 |
| mnaser | thought we did, alirghtly | 20:35 |
| clarkb | the change merged but we have to restart gerrit to pick it up | 20:36 |
| *** raissa has quit IRC | 20:39 | |
| corvus | that gitea error happened after processing 28233 refs | 20:39 |
| *** pcaruana has quit IRC | 20:39 | |
| corvus | my local port range has a span of 28231 | 20:40 |
| *** rlandy|biab is now known as rlandy | 20:40 | |
| corvus | i think it ran out of ports | 20:40 |
| *** raissa has joined #openstack-infra | 20:40 | |
| *** raissa has quit IRC | 20:40 | |
| openstackgerrit | Ben Nemec proposed openstack/pbr master: Stop using pbr sphinx integration https://review.opendev.org/655565 | 20:40 |
| openstackgerrit | sebastian marcet proposed osf/openstackid-resources master: Update allowed slides attachment extensions https://review.opendev.org/655566 | 20:41 |
| clarkb | corvus: it creates a new listening socket for each ref? | 20:42 |
| openstackgerrit | Merged osf/openstackid-resources master: Update allowed slides attachment extensions https://review.opendev.org/655566 | 20:42 |
| *** raissa has joined #openstack-infra | 20:42 | |
| *** raissa has quit IRC | 20:43 | |
| *** raissa has joined #openstack-infra | 20:43 | |
| *** raissa has quit IRC | 20:43 | |
| corvus | clarkb: the pre-recieve hook (run by ssh) contacts the internal rest api to check whether the ref is a protected branch; each of those is an http request. | 20:44 |
| corvus | clarkb: i think the solution here is to not do that for things that aren't branches | 20:44 |
| clarkb | gotcha | 20:44 |
| clarkb | (it is the source side of the connection that needs a port) | 20:44 |
| corvus | ya | 20:45 |
| *** smarcet has quit IRC | 20:47 | |
| *** Goneri has joined #openstack-infra | 20:47 | |
| *** eharney has joined #openstack-infra | 20:52 | |
| *** smarcet has joined #openstack-infra | 20:59 | |
| *** whoami-rajat has quit IRC | 21:25 | |
| *** kranthikirang has quit IRC | 21:29 | |
| *** kjackal has quit IRC | 21:37 | |
| corvus | i should have asked this at the meeting yesterday -- what's still outstanding before we are okay creating projects? | 21:41 |
| clarkb | I think the last remaining item is double checking that we create the org (if necessary) first | 21:42 |
| clarkb | I want to say we do this for gitea and that gerrit will auto create the org on disk if it needs to | 21:42 |
| clarkb | with cgit "off" we are probably set | 21:42 |
| fungi | yeah, i'm not aware of any other gotchas, with manage-projects confirmed working | 21:44 |
| clarkb | then it is just down to sorting out a general set of rules I think | 21:44 |
| *** MarkMaglana has joined #openstack-infra | 21:44 | |
| clarkb | but if we stick to "stackforge" type rules for now we are probably fine | 21:45 |
| clarkb | basiaclly is it open source | 21:45 |
| *** rkukura has joined #openstack-infra | 21:45 | |
| fungi | right, we probably need a semi-formal tos documented somewhere to point to in the case of random questions | 21:45 |
| fungi | this is worth talking through at the ptg but i don't think it should hold up new project creation in the interim | 21:46 |
| fungi | tos should mention things like "must be open source" | 21:46 |
| *** slaweq has quit IRC | 21:47 | |
| *** slaweq has joined #openstack-infra | 21:48 | |
| *** bgmccollum has quit IRC | 21:49 | |
| corvus | which can be helpfully realized more concretely as "licensed under osi approved license" | 21:49 |
| *** kgiusti has left #openstack-infra | 21:50 | |
| fungi | yes, i concur. was using short-hand | 21:50 |
| fungi | did not mean to imply any open source vs free software dichotomy | 21:51 |
| corvus | i've made a local modification to gitea to skip a bunch of processing of non-branch/tag refs; i've successfully pushed gertty with the expected functionality. i'm pushing nova now. (have been for a few minutes) | 21:51 |
| *** rh-jelabarre has quit IRC | 21:52 | |
| fungi | i'm mostly sucked into generating "contributor lists" for decorating conference badges, but need to go build a lawn mower and then push it around the yard for a few minutes before the sun is swallowed by the earth again | 21:53 |
| *** igordc has joined #openstack-infra | 21:53 | |
| clarkb | I have a feeling we are all getting sucked into the pre summit wormhole | 21:53 |
| fungi | (my old mower went for a swim in the last flood, replacement arrived last night) | 21:53 |
| clarkb | fungi: I thoguht you used a human powered mower? | 21:54 |
| corvus | clarkb: do you think you can do a scheduler restart with reverts before you vanish? | 21:54 |
| clarkb | corvus: yes I likely can do that | 21:54 |
| fungi | clarkb: it is, but "still some assembly required" | 21:54 |
| clarkb | corvus: thinking of process out loud here, make local branch in /opt/zuul with the two reverts there. pip3 install /opt/zuul. Capture queues and stop/start? | 21:55 |
| fungi | i can help with scheduler and hopefully executor restarts... hoping 655491 merges after my last recheck | 21:55 |
| clarkb | oh if we need to restart executors too then lets wait for that and try to minimize the impact we have | 21:55 |
| *** nicolasbock has quit IRC | 21:55 | |
| *** dciabrin has quit IRC | 21:57 | |
| fungi | surprising amount of backlog this close to summit | 21:59 |
| fungi | 655491 has only just started to get node allocations in check since the recheck so could be a while | 21:59 |
| fungi | should i enqueue it into the gate? | 21:59 |
| *** rlandy is now known as rlandy|bbl | 22:00 | |
| clarkb | I'm good with enqueing to the gate | 22:00 |
| *** bgmccollum has joined #openstack-infra | 22:00 | |
| *** imacdonn has quit IRC | 22:01 | |
| fungi | gonna go ahead with that as it still doesn't have most of its node allocation fulfilled in check anyway so could be quite a long time coming | 22:02 |
| *** imacdonn has joined #openstack-infra | 22:02 | |
| *** dciabrin has joined #openstack-infra | 22:02 | |
| corvus | clarkb: i usually make the local branch in my homedir | 22:04 |
| clarkb | corvus: rgr | 22:04 |
| * clarkb clones repo in homedir | 22:04 | |
| *** sreejithp has quit IRC | 22:06 | |
| clarkb | ok zuul01:~clarkb/zuul is checked out on a branch that is current zuul master with 9f7c642a and 3704095c reverted | 22:09 |
| *** yamamoto has joined #openstack-infra | 22:09 | |
| clarkb | I can pip3 install that once we're ready with the executor stuff | 22:09 |
| clarkb | at this rate maybe plan to restart tomorrow morning? I can be around early for that before I disappear | 22:10 |
| *** ijw has quit IRC | 22:11 | |
| fungi | i should be around all day tomorrow to if we want to do the restart then... will scheduler memory hold out that long? | 22:13 |
| clarkb | fungi: I think it will be close | 22:14 |
| *** slaweq has quit IRC | 22:24 | |
| corvus | i think we just learned in #zuul that we're too close | 22:31 |
| clarkb | I agree. | 22:31 |
| corvus | i think we should restart the scheduler now with clarkb's staged changes, then enqueue the zuul fixes and plan for executor restart later | 22:31 |
| clarkb | given we now believe the many issues to be related to the memory leak (and swapping) I agree | 22:32 |
| corvus | we're starting to tie the registry failures and random node connectivity failures all to the memory leak | 22:32 |
| corvus | so if we fix, or paper over, that, we get a system that should be stable enough to land changes like the executor one, then we restart and can hopefully release software again | 22:32 |
| corvus | clarkb: you want to proceed with that now? | 22:33 |
| clarkb | yup. My plan is to pip3 install /home/clarkb/zuul, grab queues, stop the scheduler, restore queues | 22:34 |
| clarkb | roughyl | 22:34 |
| corvus | ++ | 22:34 |
| clarkb | corvus: `sudo pip3 install -U /home/clarkb/zuul` that look right to you? | 22:35 |
| *** ijw has joined #openstack-infra | 22:35 | |
| clarkb | maybe drop the sudo and just run as root so that caches are used | 22:36 |
| clarkb | I might alway wait for the puppet run at ~22:45 to pass so that it doesn't undo my install | 22:36 |
| clarkb | *might also wait. I think might is now definite. It just gives me more time to work without worrying about starting on the wrong version | 22:38 |
| clarkb | so in about 10 minutes I will proceed with this plan | 22:38 |
| *** slaweq has joined #openstack-infra | 22:40 | |
| fungi | caught up after mower assembly, and concur with analysis | 22:40 |
| fungi | in summary, swapping bad | 22:40 |
| corvus | clarkb: oh, that's one of the reasons i install from my homedir -- you don't have to worry about puppet | 22:43 |
| corvus | (since you aren't updating /opt/zuul, puppet won't think it needs to do anything) | 22:43 |
| corvus | clarkb: but yes, the "pip3 install -U /home/clarkb/zuul" looks right and should work (modulo sudo or running as root) | 22:44 |
| clarkb | oh right | 22:44 |
| clarkb | ok starting now | 22:44 |
| *** gfidente has quit IRC | 22:44 | |
| *** yamamoto has quit IRC | 22:45 | |
| clarkb | zuul==3.8.1.dev3 # git sha 02b07a3 is installed | 22:45 |
| clarkb | capturing queues nowish | 22:45 |
| *** Goneri has quit IRC | 22:45 | |
| clarkb | I have captured queues and asked the scheduler to stop | 22:46 |
| clarkb | just waiting for it to actually stop | 22:46 |
| clarkb | run handler and mysql connections are stopping according to hte log but the process is still alive | 22:47 |
| * clarkb tries to be patient :) | 22:47 | |
| clarkb | it stopped so I started it again | 22:48 |
| fungi | such restart | 22:49 |
| clarkb | corvus: any idea if zuul-web still needs a restart when the scheduler is restarted? | 22:50 |
| corvus | clarkb: unclear | 22:50 |
| clarkb | ok I'll leave it be and only restart it if things are sad after the scheduler is back up | 22:50 |
| clarkb | cat jobs are happening now | 22:51 |
| fungi | past scheduler restarts i've discovered it necessary | 22:51 |
| fungi | but no clue whether it still is | 22:51 |
| corvus | let's find out | 22:51 |
| fungi | yep. it may also be that i was impatient in recent history and took indefinite "fetching info..." as a sign it wasn't going to eventually connect on its own | 22:52 |
| *** slaweq has quit IRC | 22:53 | |
| *** tkajinam has joined #openstack-infra | 22:53 | |
| corvus | clarkb: ftr which changes did you revert? | 22:54 |
| *** rcernin has joined #openstack-infra | 22:54 | |
| corvus | and it looks like the answer is yes we do need to restart web | 22:54 |
| clarkb | 9f7c642a and 3704095c | 22:54 |
| clarkb | ok I'll restart web then reload queues | 22:54 |
| fungi | mine finally popped up a "Service Unavailable (503) http://zuul.openstack.org/api/info" | 22:55 |
| fungi | and now it's loaded | 22:55 |
| corvus | yeah, that's after the service is stopped | 22:55 |
| fungi | okay, so that was the zuul-web restart then | 22:55 |
| fungi | noted | 22:55 |
| clarkb | gate is on its way back in | 22:55 |
| corvus | i will direct-enqueue the 2 changes we're interested in | 22:55 |
| corvus | done | 22:56 |
| *** yamamoto has joined #openstack-infra | 22:57 | |
| clarkb | and now check is on its way | 22:57 |
| *** ijw has quit IRC | 22:57 | |
| *** yamamoto has quit IRC | 23:01 | |
| clarkb | check is done | 23:03 |
| clarkb | I think that concludes this portion of our broadcast | 23:03 |
| corvus | same bat channel same bat time tomorrow? :| | 23:03 |
| clarkb | please tune back in in about an hour? to see if the zuul executor change has merged | 23:03 |
| corvus | ooh look | 23:04 |
| corvus | if you look at the status page right now you can see the provides/requires in action | 23:04 |
| corvus | the zuul-upload-image job for 655491 is waiting on the zuul-upload-image job in 654238 to complete so it can use it | 23:04 |
| clarkb | neat | 23:05 |
| corvus | s/use it/use its artifacts/ | 23:05 |
| *** yamamoto has joined #openstack-infra | 23:06 | |
| *** yamamoto has quit IRC | 23:06 | |
| corvus | https://screenshots.firefox.com/q526xtkCjNE3RQCa/zuul.openstack.org | 23:07 |
| *** yamamoto has joined #openstack-infra | 23:07 | |
| *** ijw has joined #openstack-infra | 23:09 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** yamamoto has quit IRC | 23:11 | |
| *** ijw has quit IRC | 23:17 | |
| clarkb | looking at the paused state in the ui we may want to change its color to blue since blue indicates in progress? | 23:21 |
| clarkb | but instead of a progress bar it can just be a blue b ar | 23:22 |
| clarkb | that might actually be a webui change I can figure out too | 23:24 |
| *** slaweq has quit IRC | 23:25 | |
| corvus | yeah | 23:27 |
| corvus | okay we got that weird of again | 23:27 |
| corvus | i guess we should have put the autoholds back :( | 23:28 |
| clarkb | not to late for the next one? | 23:28 |
| corvus | done | 23:29 |
| corvus | well, the next one is going to fail due to the name error | 23:29 |
| corvus | let's rebase it | 23:29 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Revert "Prepend path with bin dir of ansible virtualenv" https://review.opendev.org/655491 | 23:30 |
| corvus | k that's now based on the opendev fix, so we shouldn't need to worry about invalid results now | 23:30 |
| corvus | i will re-enqueue both in gate | 23:31 |
| *** Weifan has quit IRC | 23:33 | |
| *** Weifan has joined #openstack-infra | 23:33 | |
| *** Weifan has quit IRC | 23:34 | |
| *** Weifan has joined #openstack-infra | 23:34 | |
| *** Weifan has quit IRC | 23:34 | |
| *** Weifan has joined #openstack-infra | 23:35 | |
| *** dayou has quit IRC | 23:35 | |
| *** Weifan has quit IRC | 23:35 | |
| *** Weifan has joined #openstack-infra | 23:35 | |
| *** yamamoto has joined #openstack-infra | 23:36 | |
| *** Weifan has quit IRC | 23:36 | |
| *** Weifan has joined #openstack-infra | 23:36 | |
| *** dayou has joined #openstack-infra | 23:36 | |
| *** Weifan has quit IRC | 23:37 | |
| *** Weifan has joined #openstack-infra | 23:37 | |
| *** yamamoto has quit IRC | 23:37 | |
| *** Weifan has quit IRC | 23:38 | |
| corvus | real118m20.905s | 23:40 |
| corvus | that's how long it took to push nova into gitea locally, but it worked | 23:40 |
| *** lseki has quit IRC | 23:40 | |
| clarkb | \o/ | 23:42 |
| clarkb | presumably it is quicker if the data already exists on both ends and git can figure out no data needs to move | 23:42 |
| fungi | who knows | 23:45 |
| fungi | i wouldn't presume that ;) | 23:45 |
| corvus | a no-op push finishes in 5 seconds | 23:48 |
| corvus | how does this look? https://github.com/jeblair/gitea/commit/0da1af345c2e17144716f56e0946b0b1ac11d2f4 | 23:49 |
| corvus | suitable for a PR? | 23:49 |
| *** Goneri has joined #openstack-infra | 23:51 | |
| corvus | i made a minor clarification to the commit msg; url is now: https://github.com/jeblair/gitea/commit/03b85f4bd2b14e4c00701ab566d9631598282423 | 23:51 |
| corvus | wow reviewing go code in github is harsh | 23:52 |
| clarkb | I was just thinking about how I don't like the diffs | 23:52 |
| clarkb | (side by side would be much easier to read in this case) | 23:52 |
| clarkb | oh hey I can hit the split button for that | 23:52 |
| corvus | yep. you will need a *very* wide monitor. | 23:53 |
| corvus | i'm using both of mine. | 23:53 |
| clarkb | corvus: it lgtm | 23:53 |
| clarkb | commit message aligns with the change being made | 23:53 |
| *** yamamoto has joined #openstack-infra | 23:54 | |
| *** yamamoto has quit IRC | 23:54 | |
| *** yamamoto has joined #openstack-infra | 23:54 | |
| fungi | makes you wonder if there's a reason the company who pioneered go also wrote gerrit | 23:57 |
| fungi | (though not necessarily in that sequence) | 23:57 |
| fungi | i'm curious how the k community gets by this way | 23:58 |
| *** yamamoto has quit IRC | 23:59 | |
| fungi | for as much as i barely grok golang, lgtm | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!