| bknudson | I thought the backup time was going to be changed so gerrit didn't hang at this time. | 00:00 |
|---|---|---|
| anteaya | I am trying to vote +1 but the screen is frozen | 00:01 |
| bknudson | ok, it's back | 00:01 |
| anteaya | yeah, I hit it too | 00:01 |
| *** Ryan_Lane has quit IRC | 00:08 | |
| *** HenryG has quit IRC | 00:11 | |
| mordred | woot. now I can bother jhesketh for reviews on my things | 00:11 |
| mordred | jhesketh: review all of my things! | 00:11 |
| jhesketh | mordred: this is a two way street my friend ;-) | 00:11 |
| mordred | jhesketh: blast. you actually know how this works :( | 00:12 |
| jhesketh | I have plenty of unmerged patches :-) | 00:12 |
| *** nati_ueno has joined #openstack-infra | 00:12 | |
| mordred | jhesketh: really? I actually see none from you in my revew queue... | 00:12 |
| anteaya | jhesketh: look at that +2, very nice | 00:12 |
| anteaya | since you are both here: https://review.openstack.org/#/c/92475/ | 00:12 |
| anteaya | that is the initial commit for the infra-manual repo | 00:13 |
| anteaya | which has licensing stuff in it | 00:13 |
| jhesketh | mordred: https://review.openstack.org/#/q/owner:%22Joshua+Hesketh%22+status:open,n,z | 00:13 |
| * mordred wonders if he is missing project watches | 00:13 | |
| anteaya | you don't have to review it now, but I will keep bugging you | 00:13 |
| jhesketh | heh, anteaya also knows how this works... | 00:14 |
| anteaya | when the crowd gathers... | 00:14 |
| mordred | ooh! | 00:14 |
| mordred | the diff view shows images | 00:14 |
| mordred | has it always done that? | 00:14 |
| mordred | https://review.openstack.org/#/c/92475/1/doc/source/code_review.png | 00:14 |
| anteaya | I don't think so, no | 00:14 |
| anteaya | no we used to have to download the patch and open with a browser | 00:15 |
| mattoliverau | Wow, awesome! Now I can submit all my code in pictures! | 00:15 |
| anteaya | ha ha ha | 00:15 |
| anteaya | I like pictures | 00:15 |
| * mattoliverau wonders if it also supports animated gifs... this could be fun ;P | 00:16 | |
| *** LLKCKfan has joined #openstack-infra | 00:16 | |
| LLKCKfan | How can someone who is almost 30 and never had a job get one? I have been applying just to be told I am not what they are looking for or they are not hiring. | 00:16 |
| *** mordred sets mode: +b LLKCKfan!*@* | 00:17 | |
| *** LLKCKfan was kicked by mordred (LLKCKfan) | 00:17 | |
| *** nati_ueno has quit IRC | 00:17 | |
| jhesketh | mordred: I saw the patches to allow imgs, it was cool | 00:18 |
| *** packet has quit IRC | 00:18 | |
| anteaya | we need to ban this person from all #openstack channels | 00:18 |
| anteaya | (s)he was in -meeting earlier | 00:19 |
| mordred | hrm. I smell an accessbot feature :) | 00:19 |
| anteaya | jhesketh: do you have links? | 00:19 |
| jhesketh | anteaya: links to which now? | 00:19 |
| mordred | anteaya: there is a small flaw in that patch - I had to -1 it - if you wanted to correct it and resubmit, I betcha it would not be offensive | 00:20 |
| anteaya | jhesketh: the patches to allow images | 00:20 |
| anteaya | kk | 00:20 |
| jhesketh | oh they merged a while back | 00:20 |
| jhesketh | it was just a config change to gerrit | 00:20 |
| anteaya | I'll nuke the blank space I noticed as well | 00:20 |
| *** nati_uen_ has joined #openstack-infra | 00:22 | |
| ianw | fungi: yeah, my fault from https://review.openstack.org/#/c/93862/ (Handle Workflow in comment matching) i think | 00:24 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/infra-manual: Initial commit https://review.openstack.org/92475 | 00:26 |
| *** matsuhashi has joined #openstack-infra | 00:26 | |
| mordred | jhesketh: did you see the "Hide CI comments in gerrit" mailing list message? | 00:27 |
| mordred | there is a javascript thing: https://gist.github.com/rgerganov/35382752557cb975354a | 00:27 |
| *** nati_ue__ has joined #openstack-infra | 00:27 | |
| jhesketh | mordred: yes I did | 00:27 |
| mordred | makes me wonder if perhaps it might be the sort of thing we might want to just include in our local javascript header for gerrit in general | 00:27 |
| anteaya | there is developer support for it | 00:28 |
| anteaya | based on feedback from the summit | 00:28 |
| jhesketh | mordred: that was what I was thinking | 00:28 |
| anteaya | folks were also requesting a seperate column for ci reports | 00:28 |
| jhesketh | it's a good first pass and probably the easiest solution, but it'd be neat to pull the comments to the side in the future | 00:29 |
| jhesketh | although that probably requires quite a bit of UI redesign | 00:29 |
| mordred | yah | 00:29 |
| mordred | I'm going to respond to the list | 00:29 |
| *** nati_uen_ has quit IRC | 00:30 | |
| jhesketh | mordred: if my JS patches ever land I can get CI status' overlaying the pages | 00:31 |
| * jhesketh is less than subtle before coffee sorry | 00:31 | |
| mordred | jhesketh: oh. well | 00:31 |
| mordred | jhesketh: it seems your patch is WIP'd | 00:32 |
| *** nati_ue__ has quit IRC | 00:32 | |
| jhesketh | eh? which one... | 00:32 |
| mordred | your zuul status overlay | 00:33 |
| *** matsuhashi has quit IRC | 00:33 | |
| jhesketh | oh right, yes, that's WIP until all the zuul javascript patches are merged | 00:34 |
| *** matsuhashi has joined #openstack-infra | 00:34 | |
| mordred | oh. gotcha | 00:34 |
| jhesketh | mordred: Here's the tail https://review.openstack.org/#/c/91316/ | 00:35 |
| *** nati_uen_ has joined #openstack-infra | 00:35 | |
| mordred | jhesketh: I will work through those ... you may find https://review.openstack.org/#/c/90565/ and its ancestors interesting reading... | 00:35 |
| anteaya | jenkins is happy with the first infra-manual commit: https://review.openstack.org/#/c/92475/2 | 00:36 |
| anteaya | anyone know what the zanata demo server is? | 00:36 |
| jhesketh | thanks mordred | 00:36 |
| *** zhiyan_ is now known as zhiyan | 00:38 | |
| *** nati_uen_ has quit IRC | 00:39 | |
| *** yongli has quit IRC | 00:39 | |
| mordred | anteaya: yes. pleia2 is working on it - it's step one in trying out zanata as a replacement for transifex | 00:41 |
| anteaya | ah cool | 00:46 |
| anteaya | I've invited Carlos A. Munoz to join us on irc and chat about zanata | 00:48 |
| anteaya | he emailed the infra ml | 00:49 |
| *** gokrokve_ has quit IRC | 00:51 | |
| *** gokrokve has joined #openstack-infra | 00:52 | |
| *** igor_ has joined #openstack-infra | 00:52 | |
| *** nati_uen_ has joined #openstack-infra | 00:52 | |
| *** nati_ue__ has joined #openstack-infra | 00:54 | |
| *** gokrokve has quit IRC | 00:56 | |
| mordred | jhesketh: stack reviewed | 00:57 |
| mordred | anteaya: I have rebased my email response on top of yours | 00:57 |
| *** igor_ has quit IRC | 00:57 | |
| *** nati_uen_ has quit IRC | 00:57 | |
| *** nati_ueno has joined #openstack-infra | 00:57 | |
| jhesketh | mordred: awesome, thanks :-) | 00:59 |
| *** nati_ue__ has quit IRC | 00:59 | |
| anteaya | mordred: thanks for doing so | 00:59 |
| *** nati_ueno has quit IRC | 00:59 | |
| *** nati_ueno has joined #openstack-infra | 01:00 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Include rule numbers in HACKING.rst https://review.openstack.org/93470 | 01:00 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Add Installation section to the readme https://review.openstack.org/93471 | 01:01 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Drop 'not in' and 'is not' tests from HACKING.rst https://review.openstack.org/93472 | 01:01 |
| openstackgerrit | A change was merged to openstack-dev/hacking: update Commit Message guidelines https://review.openstack.org/93473 | 01:01 |
| *** nati_ueno has quit IRC | 01:04 | |
| *** nati_ueno has joined #openstack-infra | 01:09 | |
| anteaya | mordred: thanks | 01:10 |
| *** yaguang has joined #openstack-infra | 01:11 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add in sparklines to status page pipelines https://review.openstack.org/84922 | 01:14 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Fix up fetching jquery.visibility https://review.openstack.org/91316 | 01:15 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Move status dom into js app for easy reuse https://review.openstack.org/95049 | 01:15 |
| lifeless | mordred: since you are around | 01:26 |
| lifeless | mordred: I want to have a CI job that runs devtest.sh --build-only | 01:26 |
| lifeless | mordred: this requires root (since it builds images) - I presume that makes it a dsvm job ? | 01:27 |
| lifeless | mordred: also have you read https://etherpad.openstack.org/p/infra-no-floating-ip-slaves ? | 01:28 |
| *** david-lyle has quit IRC | 01:30 | |
| *** matsuhashi has quit IRC | 01:33 | |
| *** nati_ueno has quit IRC | 01:33 | |
| mordred | lifeless: "dsvm" stands for DevStack VM - you just want a bare node | 01:33 |
| mordred | lifeless: and no - looking | 01:34 |
| *** matsuhashi has joined #openstack-infra | 01:34 | |
| mordred | lifeless: first response - we don't wat to add any more features that take advantage of any features of jenkins, since we're making jenkins go away. I don't think that's a problem - just pointing it out since the text talks about teaching jenkins about jumphosts | 01:35 |
| mordred | lifeless: I _think_ we already have some related work done though that you might be able to build off of | 01:36 |
| mordred | lifeless: specifically, the work done to enable multi-node devstack | 01:36 |
| mordred | lifeless: I don't see any general issues with the strategy though | 01:37 |
| *** yamahata has joined #openstack-infra | 01:37 | |
| mordred | lifeless: "By setting a ProxyCommand in ~/.ssh/config for all the hosts that connect to slaves, we can transparently trigger jump host use for openssh use. " seems potentially weird | 01:38 |
| mordred | lifeless: how do you see that working? when we nova boot a thing, we're going to get back the "public" ip and we're going to want to connect to that - how would we manage that? have each cloud have it's own 10.x subnet for it's 'public' addresses? | 01:40 |
| mordred | and then have an entry for that cloud's 10.123.* with the jump host config? | 01:40 |
| *** oomichi has joined #openstack-infra | 01:40 | |
| *** alugovoi has joined #openstack-infra | 01:41 | |
| fungi | ianw: thanks for the refresher, actually--your incomplete change points out that my completion is also incomplete! ;) (fixing now) | 01:42 |
| mordred | lifeless: also, if you want to cry, you may want to look at: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/provider_manager.py#n44 | 01:43 |
| *** gokrokve has joined #openstack-infra | 01:44 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Broaden recheck/reverify pattern for vote matching https://review.openstack.org/95611 | 01:44 |
| *** nosnos has joined #openstack-infra | 01:44 | |
| fungi | ianw: ^ | 01:44 |
| ianw | fungi: not sure if it's worth inbuilt tests because it doesn't change much, but even just a comment saying "make sure the regex matches ...." and some examples might help | 01:45 |
| mordred | lifeless: we have clouds that do not give us an indication as to what network range we shoudl use to talk to them | 01:45 |
| mordred | actually ... | 01:46 |
| mordred | fungi, corvus: ^^ in writing that reply to lifeless I just realized, that if we teach nodepool about neutron and about creating/deleting routers/networks | 01:46 |
| mordred | fungi, corvus: then WE get to give it names for things, and that means we can stop guessing about which ip we should be conencting to, because we'd be creating and managing all of that | 01:47 |
| mordred | we'd need to keep a special case for rackspace until they expose a neutron API - but actually going through and making the primary control path very neutron aware might-could get us to a pretty sane place | 01:47 |
| clarkb | mordred: yup. I think the tricky bit will be scaling down | 01:48 |
| mordred | clarkb: yup | 01:48 |
| *** gokrokve has quit IRC | 01:48 | |
| clarkb | since we will fragment and cant really defrag | 01:48 |
| lifeless | mordred: not entirely sure | 01:49 |
| mordred | clarkb: yah. although maybe scaling down networks isnt' as important? | 01:49 |
| fungi | ianw: honestly, i should have just looked at all the comment_filter lines | 01:49 |
| clarkb | but we can just avoid it entirely | 01:49 |
| mordred | clarkb: since we're not using them throwaway | 01:49 |
| clarkb | mordred exactly | 01:49 |
| lifeless | mordred: but the jump host for jenkins would be on the same private network as the nodes | 01:49 |
| clarkb | but people that pay for resources may want scale down | 01:49 |
| mordred | lifeless: right. that makes sense | 01:49 |
| lifeless | mordred: and whether that is a unique private network or a cloud-wide one is irrelevant | 01:49 |
| lifeless | mordred: so I think we'd want to make it simple and just not use floating ips at all in that case | 01:50 |
| mordred | lifeless: I'm just asking about how nodepool itself knows to connect to a cloud's jump host when it gets a nova instance back if the jump hosts are configured in .ssh/config | 01:50 |
| lifeless | mordred: I think we'd want a 'use jump host' option in nodepool which could trigger simpler codepaths throughout | 01:50 |
| lifeless | is there an infra-specs repo now btw? | 01:51 |
| mordred | lifeless: we've got non-floating-ip code paths too - I'm not worried about that - although I agree with you - it's purely the "how do we map a host to the appropriate jump host" problem I'm concerned with | 01:51 |
| mordred | lifeless: and a mild concern that having that in .ssh/config and not nodepool.yaml might get ... confusing | 01:51 |
| mordred | lifeless: still working on specs repo | 01:52 |
| mordred | lifeless: I mean, specs repo exists- but the initial commit is not landed | 01:52 |
| *** saschpe has quit IRC | 01:53 | |
| mordred | lifeless: however, https://review.openstack.org/#/c/94440/ already has some children | 01:53 |
| *** igor_ has joined #openstack-infra | 01:53 | |
| lifeless | mordred: reworked the work items in the etherpad. | 01:53 |
| mordred | clarkb: now that I'm thinking about using both glance and neutron in nodepool, I REALLY want python-openstacksdk to exist | 01:53 |
| *** alugovoi has quit IRC | 01:54 | |
| *** saschpe has joined #openstack-infra | 01:55 | |
| mordred | clarkb: and I'm going to get cranky if we wind up writing one ourselves | 01:55 |
| lifeless | mordred: should I move this to the specs repo now ? | 01:56 |
| mordred | lifeless: sure! | 01:56 |
| mordred | that way we can capture some of this | 01:56 |
| lifeless | hah | 01:57 |
| lifeless | the one that isn't in the openstack namespace | 01:57 |
| *** igor_ has quit IRC | 01:57 | |
| mordred | :) | 01:58 |
| *** nati_ueno has joined #openstack-infra | 02:00 | |
| *** wenlock has joined #openstack-infra | 02:01 | |
| *** persia has quit IRC | 02:01 | |
| *** persia_ has joined #openstack-infra | 02:03 | |
| *** persia_ is now known as Guest77850 | 02:04 | |
| *** persia has joined #openstack-infra | 02:04 | |
| *** persia is now known as Guest9975 | 02:04 | |
| *** Guest9975 has joined #openstack-infra | 02:04 | |
| *** Guest9975 is now known as persia | 02:04 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/infra-specs: Make use of IP per slave optional. https://review.openstack.org/95625 | 02:08 |
| *** nati_ueno has quit IRC | 02:10 | |
| lifeless | mordred: ^tada | 02:11 |
| anteaya | lifeless: whitespace on line 23 | 02:12 |
| lifeless | anteaya: thanks, but I'm sure there are many more issues than that | 02:14 |
| anteaya | fair, that was the one I spotted | 02:16 |
| *** nati_ueno has joined #openstack-infra | 02:19 | |
| lifeless | jhesketh: you should hang in #tripleo :) | 02:19 |
| jhesketh | lol, then I'd just have more things to do | 02:19 |
| *** mayu_ has joined #openstack-infra | 02:25 | |
| *** nati_ueno has quit IRC | 02:25 | |
| lifeless | jhesketh: ... and? :) | 02:26 |
| *** alugovoi has joined #openstack-infra | 02:26 | |
| mayu_ | who can tell that how does jenkins tell slave node to git pull the specific pathch set ? | 02:27 |
| *** zhiyan is now known as zhiyan_ | 02:29 | |
| *** david-lyle has joined #openstack-infra | 02:31 | |
| *** nati_ueno has joined #openstack-infra | 02:33 | |
| *** david-lyle has quit IRC | 02:35 | |
| *** lcheng_ has joined #openstack-infra | 02:35 | |
| *** HenryG has joined #openstack-infra | 02:37 | |
| jhesketh | lifeless: I'm not that gullible | 02:38 |
| *** gokrokve has joined #openstack-infra | 02:41 | |
| *** hdd_ has joined #openstack-infra | 02:42 | |
| *** gokrokve has quit IRC | 02:46 | |
| *** zhiyan_ is now known as zhiyan | 02:46 | |
| mordred | mayu_: zuul prepares a set of repo states and passes those refs as env vars - then we have a script called "gerrit-git-prep" which reads those env vars and does the appropriate git actions | 02:50 |
| mayu_ | thanks, mordred | 02:51 |
| *** zhiyan is now known as zhiyan_ | 02:52 | |
| *** zhiyan_ is now known as zhiyan | 02:52 | |
| mayu_ | @mordred: is there some references to the process ? | 02:53 |
| *** igor_ has joined #openstack-infra | 02:54 | |
| *** igor_ has quit IRC | 02:58 | |
| *** signed8bit has joined #openstack-infra | 03:04 | |
| *** dims has quit IRC | 03:16 | |
| *** nosnos has quit IRC | 03:25 | |
| *** unicell has joined #openstack-infra | 03:29 | |
| *** unicell has quit IRC | 03:29 | |
| *** unicell has joined #openstack-infra | 03:29 | |
| *** plars has quit IRC | 03:31 | |
| *** david-lyle has joined #openstack-infra | 03:32 | |
| *** plars has joined #openstack-infra | 03:32 | |
| *** david-lyle has quit IRC | 03:36 | |
| *** david-lyle has joined #openstack-infra | 03:38 | |
| *** signed8bit has quit IRC | 03:40 | |
| *** gokrokve has joined #openstack-infra | 03:42 | |
| *** david-lyle has quit IRC | 03:43 | |
| *** gokrokve has quit IRC | 03:46 | |
| *** unicell has quit IRC | 03:47 | |
| *** igor_ has joined #openstack-infra | 03:55 | |
| *** bhuvan has joined #openstack-infra | 03:55 | |
| *** igor_ has quit IRC | 03:59 | |
| openstackgerrit | Steve Baker proposed a change to openstack-infra/devstack-gate: Enable dib service by default https://review.openstack.org/95636 | 04:01 |
| *** zhiyan is now known as zhiyan_ | 04:14 | |
| *** zhiyan_ is now known as zhiyan | 04:21 | |
| *** david-lyle has joined #openstack-infra | 04:24 | |
| *** nati_ueno has quit IRC | 04:31 | |
| *** nati_ueno has joined #openstack-infra | 04:32 | |
| *** gokrokve has joined #openstack-infra | 04:43 | |
| *** Longgeek has joined #openstack-infra | 04:45 | |
| jhesketh | mordred: ping | 04:46 |
| *** nosnos has joined #openstack-infra | 04:46 | |
| *** gokrokve has quit IRC | 04:48 | |
| *** gokrokve has joined #openstack-infra | 04:48 | |
| *** e0ne has joined #openstack-infra | 04:48 | |
| *** e0ne has quit IRC | 04:49 | |
| *** e0ne has joined #openstack-infra | 04:49 | |
| *** praneshp has joined #openstack-infra | 04:50 | |
| *** e0ne has quit IRC | 04:53 | |
| *** igor_ has joined #openstack-infra | 04:55 | |
| *** ildikov has quit IRC | 04:56 | |
| *** igor_ has quit IRC | 05:00 | |
| *** lcheng_ has quit IRC | 05:01 | |
| *** wenlock has quit IRC | 05:04 | |
| *** hdd_ has quit IRC | 05:08 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Run checklang gate only on master https://review.openstack.org/95642 | 05:11 |
| *** vkdrao has joined #openstack-infra | 05:12 | |
| *** doude has quit IRC | 05:16 | |
| *** gokrokve has quit IRC | 05:17 | |
| *** gokrokve has joined #openstack-infra | 05:18 | |
| *** gokrokve has quit IRC | 05:22 | |
| *** Longgeek has quit IRC | 05:25 | |
| *** starmer has joined #openstack-infra | 05:28 | |
| stevebaker | sdague: building an image during devstack gate adds ~11 minutes http://logs.openstack.org/17/95617/3/check/check-tempest-dsvm-neutron-heat-slow/7ef53b4/logs/devstacklog.txt.gz#_2014-05-27_04_15_46_073 | 05:31 |
| *** wenlock has joined #openstack-infra | 05:35 | |
| *** zhiyan is now known as zhiyan_ | 05:40 | |
| *** ildikov has joined #openstack-infra | 05:40 | |
| *** zhiyan_ is now known as zhiyan | 05:41 | |
| *** lcheng_ has joined #openstack-infra | 05:43 | |
| *** Ryan_Lane has joined #openstack-infra | 05:44 | |
| *** wenlock has quit IRC | 05:46 | |
| *** _nadya_ has joined #openstack-infra | 05:46 | |
| *** gokrokve has joined #openstack-infra | 05:48 | |
| *** yfried__ has joined #openstack-infra | 05:49 | |
| *** lcheng_ has quit IRC | 05:52 | |
| *** gokrokve has quit IRC | 05:54 | |
| *** igor_ has joined #openstack-infra | 05:56 | |
| *** Ryan_Lane has quit IRC | 05:57 | |
| *** igor_ has quit IRC | 06:00 | |
| *** yfried__ has quit IRC | 06:01 | |
| *** yfried__ has joined #openstack-infra | 06:01 | |
| *** yfried__ has quit IRC | 06:02 | |
| *** yfried__ has joined #openstack-infra | 06:03 | |
| *** W00dy_ has joined #openstack-infra | 06:03 | |
| *** yfried__ has quit IRC | 06:04 | |
| *** yfried__ has joined #openstack-infra | 06:04 | |
| *** yfried__ has quit IRC | 06:04 | |
| *** yfried has joined #openstack-infra | 06:05 | |
| *** _nadya_ has quit IRC | 06:08 | |
| *** yfried has quit IRC | 06:11 | |
| *** _nadya_ has joined #openstack-infra | 06:11 | |
| *** W00dy_ has quit IRC | 06:18 | |
| *** dstanek_zzz is now known as dstanek | 06:24 | |
| *** gpocentek has quit IRC | 06:31 | |
| *** LIYUEZHOU has joined #openstack-infra | 06:32 | |
| *** gpocentek has joined #openstack-infra | 06:34 | |
| *** W00dy_ has joined #openstack-infra | 06:42 | |
| *** gokrokve has joined #openstack-infra | 06:49 | |
| *** rgerganov has joined #openstack-infra | 06:52 | |
| *** camunoz has joined #openstack-infra | 06:54 | |
| *** gokrokve has quit IRC | 06:54 | |
| *** david-lyle has quit IRC | 06:54 | |
| *** alugovoi has quit IRC | 06:55 | |
| *** igor_ has joined #openstack-infra | 06:57 | |
| *** bhuvan has quit IRC | 06:58 | |
| *** jhesketh has quit IRC | 06:58 | |
| *** jhesketh has joined #openstack-infra | 06:59 | |
| *** dstanek is now known as dstanek_zzz | 07:00 | |
| *** igor_ has quit IRC | 07:01 | |
| *** boris-42 has quit IRC | 07:03 | |
| *** afazekas has joined #openstack-infra | 07:08 | |
| *** boris-42 has joined #openstack-infra | 07:10 | |
| mattoliverau | Night all, have a great night/day everyone! | 07:16 |
| *** jcoufal has joined #openstack-infra | 07:17 | |
| *** salv-orlando has joined #openstack-infra | 07:17 | |
| *** nati_ueno has quit IRC | 07:18 | |
| *** nati_ueno has joined #openstack-infra | 07:19 | |
| *** hashar has joined #openstack-infra | 07:19 | |
| *** starmer has quit IRC | 07:21 | |
| *** Longgeek has joined #openstack-infra | 07:22 | |
| *** skolekonov has joined #openstack-infra | 07:22 | |
| *** markmcclain has joined #openstack-infra | 07:23 | |
| *** nati_ueno has quit IRC | 07:23 | |
| *** wenlock has joined #openstack-infra | 07:25 | |
| *** flaper87|afk is now known as flaper87 | 07:26 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Extract translations for log messages https://review.openstack.org/95377 | 07:27 |
| *** praneshp has quit IRC | 07:28 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Extract translations for log messages https://review.openstack.org/95377 | 07:29 |
| *** jlibosva has joined #openstack-infra | 07:31 | |
| *** markmcclain has quit IRC | 07:35 | |
| *** wenlock has quit IRC | 07:35 | |
| *** talluri has joined #openstack-infra | 07:37 | |
| *** ihrachyshka has joined #openstack-infra | 07:38 | |
| *** jhesketh has quit IRC | 07:42 | |
| *** matrohon has joined #openstack-infra | 07:43 | |
| *** Clabbe has joined #openstack-infra | 07:48 | |
| *** amotoki has joined #openstack-infra | 07:49 | |
| *** gokrokve has joined #openstack-infra | 07:49 | |
| *** nati_ueno has joined #openstack-infra | 07:50 | |
| *** Longgeek has quit IRC | 07:54 | |
| *** gokrokve has quit IRC | 07:54 | |
| *** nati_ueno has quit IRC | 07:54 | |
| *** jhesketh has joined #openstack-infra | 07:56 | |
| *** jpich has joined #openstack-infra | 08:02 | |
| *** jgallard has joined #openstack-infra | 08:04 | |
| *** talluri has quit IRC | 08:06 | |
| *** rdopiera has joined #openstack-infra | 08:07 | |
| rdopiera | hello, I'm wondering what is the process for adding a package to the global requirements -- is it enough to send the patch for review, or should I also attend some metting or write a bug or e-mail? | 08:08 |
| fifieldt | rdopiera, normally in addition to patches, I see emails on the -dev mailing list for those kind of things | 08:09 |
| fifieldt | but I'm no expert :) | 08:09 |
| *** talluri has joined #openstack-infra | 08:09 | |
| StevenK | I added os-cloud-config to the global requirements with only a review and no discussion | 08:10 |
| StevenK | Depends what you're proposing to add, I guess. | 08:10 |
| *** Hal_ has joined #openstack-infra | 08:10 | |
| *** talluri_ has joined #openstack-infra | 08:12 | |
| *** pblaho has joined #openstack-infra | 08:13 | |
| *** pblaho has joined #openstack-infra | 08:14 | |
| *** talluri has quit IRC | 08:16 | |
| *** dizquierdo has joined #openstack-infra | 08:16 | |
| *** derekh_ has joined #openstack-infra | 08:17 | |
| *** talluri_ has quit IRC | 08:18 | |
| *** Hal_ has quit IRC | 08:19 | |
| *** Hal has joined #openstack-infra | 08:20 | |
| *** Hal is now known as Guest51544 | 08:20 | |
| rdopiera | fifieldt, StevenK: thank you | 08:23 |
| *** andreykurilin_ has joined #openstack-infra | 08:24 | |
| *** talluri has joined #openstack-infra | 08:28 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Extract translations for log messages https://review.openstack.org/95377 | 08:35 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Do not run unneeded update_catalog https://review.openstack.org/95687 | 08:39 |
| *** jamielennox is now known as jamielennox|away | 08:40 | |
| *** igor_ has joined #openstack-infra | 08:41 | |
| *** mrda is now known as mrda_away | 08:45 | |
| *** habib has joined #openstack-infra | 08:45 | |
| *** nosnos has quit IRC | 08:48 | |
| *** habib has quit IRC | 08:48 | |
| *** yjiang has joined #openstack-infra | 08:48 | |
| *** nosnos has joined #openstack-infra | 08:48 | |
| *** gokrokve has joined #openstack-infra | 08:49 | |
| *** nati_ueno has joined #openstack-infra | 08:50 | |
| *** doude has joined #openstack-infra | 08:53 | |
| *** nosnos has quit IRC | 08:53 | |
| *** gokrokve has quit IRC | 08:53 | |
| *** nati_ueno has quit IRC | 08:55 | |
| *** habib has joined #openstack-infra | 08:56 | |
| *** andreaf has joined #openstack-infra | 08:59 | |
| *** Longgeek has joined #openstack-infra | 09:00 | |
| *** Longgeek has quit IRC | 09:00 | |
| *** Longgeek has joined #openstack-infra | 09:00 | |
| *** habib has quit IRC | 09:07 | |
| *** jp_at_hp has joined #openstack-infra | 09:08 | |
| chmouel | jogo, mordred: you guys in paris? | 09:09 |
| *** jooools has joined #openstack-infra | 09:10 | |
| *** msabramo has quit IRC | 09:16 | |
| *** ihrachyshka has quit IRC | 09:18 | |
| *** amotoki has quit IRC | 09:22 | |
| *** andreykurilin_ has quit IRC | 09:30 | |
| *** andreykurilin_ has joined #openstack-infra | 09:30 | |
| *** nosnos has joined #openstack-infra | 09:35 | |
| *** salv-orlando_ has joined #openstack-infra | 09:35 | |
| *** salv-orlando has quit IRC | 09:37 | |
| *** salv-orlando_ is now known as salv-orlando | 09:37 | |
| *** zhiyan is now known as zhiyan_ | 09:39 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Textareas now autoresize their height. https://review.openstack.org/92939 | 09:42 |
| *** gokrokve has joined #openstack-infra | 09:49 | |
| *** gokrokve_ has joined #openstack-infra | 09:51 | |
| *** nati_ueno has joined #openstack-infra | 09:51 | |
| *** mayu_ has quit IRC | 09:52 | |
| *** gokrokve has quit IRC | 09:53 | |
| *** nati_ueno has quit IRC | 09:55 | |
| *** gokrokve_ has quit IRC | 09:55 | |
| *** ihrachyshka has joined #openstack-infra | 09:56 | |
| *** matsuhashi has quit IRC | 09:58 | |
| *** che-arne has joined #openstack-infra | 09:59 | |
| *** matsuhashi has joined #openstack-infra | 10:00 | |
| *** andreykurilin_ is now known as andreykurilin | 10:06 | |
| *** matsuhashi has quit IRC | 10:07 | |
| *** nosnos has quit IRC | 10:07 | |
| *** matsuhashi has joined #openstack-infra | 10:07 | |
| *** jgallard has quit IRC | 10:08 | |
| *** jgallard has joined #openstack-infra | 10:08 | |
| *** matsuhashi has quit IRC | 10:12 | |
| *** jgallard has quit IRC | 10:13 | |
| *** markmc has joined #openstack-infra | 10:16 | |
| *** hdd_ has joined #openstack-infra | 10:23 | |
| *** ominakov has joined #openstack-infra | 10:24 | |
| *** pcm__ has joined #openstack-infra | 10:27 | |
| *** salv-orlando has quit IRC | 10:29 | |
| *** vkdrao has quit IRC | 10:29 | |
| *** talluri_ has joined #openstack-infra | 10:29 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add sahara-specs repo https://review.openstack.org/95715 | 10:30 |
| *** talluri has quit IRC | 10:33 | |
| *** yfried has joined #openstack-infra | 10:37 | |
| *** talluri_ has quit IRC | 10:38 | |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack-infra/config: Add XStatic-* projects with packaged static files for Horizon https://review.openstack.org/95716 | 10:38 |
| *** yfried has quit IRC | 10:42 | |
| *** _nadya_ has quit IRC | 10:44 | |
| *** Alexei_987 has left #openstack-infra | 10:46 | |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack-infra/config: Add XStatic-* projects with packaged static files for Horizon https://review.openstack.org/95716 | 10:47 |
| *** gokrokve has joined #openstack-infra | 10:49 | |
| *** nati_ueno has joined #openstack-infra | 10:52 | |
| *** gokrokve has quit IRC | 10:53 | |
| *** nati_ueno has quit IRC | 10:56 | |
| *** Ajaeger has joined #openstack-infra | 10:56 | |
| *** yjiang has quit IRC | 10:58 | |
| *** e0ne has joined #openstack-infra | 10:59 | |
| *** e0ne has quit IRC | 11:04 | |
| *** zhiyan_ is now known as zhiyan | 11:17 | |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack-infra/config: Add XStatic-* projects with packaged static files for Horizon https://review.openstack.org/95716 | 11:18 |
| openstackgerrit | Christian Berendt proposed a change to openstack-infra/gerritbot: replace dict.iteritems() with six.iteritems(dict) https://review.openstack.org/95727 | 11:21 |
| anteaya | chmouel: mordred is in palo alto still, he is sick | 11:23 |
| anteaya | chmouel: I'm not sure where jogo is | 11:24 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Fix Unknown events in timeline https://review.openstack.org/95729 | 11:26 |
| *** dims has joined #openstack-infra | 11:28 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Small fix to a method name https://review.openstack.org/95730 | 11:28 |
| sdague | stevebaker: that's probably still ok for now, especially if it leads to more repeatable results | 11:32 |
| sdague | http://logs.openstack.org/17/95617/3/check/check-tempest-dsvm-neutron-heat-slow/7ef53b4/logs/devstacklog.txt.gz#_2014-05-27_04_24_18_771 - grub install seems to take 2 minutes | 11:32 |
| sdague | which is interesting | 11:32 |
| *** yamahata has quit IRC | 11:36 | |
| *** hdd_ has quit IRC | 11:42 | |
| *** mburned_out is now known as mburned | 11:42 | |
| openstackgerrit | João Cravo proposed a change to openstack-infra/jenkins-job-builder: Add support for reverse build trigger https://review.openstack.org/95734 | 11:45 |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack-infra/config: Add XStatic-* projects with packaged static files for Horizon https://review.openstack.org/95716 | 11:45 |
| *** rlandy has joined #openstack-infra | 11:49 | |
| *** salv-orlando has joined #openstack-infra | 11:49 | |
| *** e0ne has joined #openstack-infra | 11:49 | |
| *** gokrokve has joined #openstack-infra | 11:49 | |
| *** gokrokve has quit IRC | 11:53 | |
| *** jgallard has joined #openstack-infra | 11:54 | |
| *** e0ne has quit IRC | 11:54 | |
| *** nati_uen_ has joined #openstack-infra | 11:55 | |
| *** _nadya_ has joined #openstack-infra | 11:55 | |
| *** yfried has joined #openstack-infra | 11:57 | |
| openstackgerrit | Antoine Musso proposed a change to stackforge/python-jenkins: Speed up job existence tests by fetching less info https://review.openstack.org/89589 | 11:58 |
| *** weshay has joined #openstack-infra | 11:58 | |
| *** nati_uen_ has quit IRC | 11:59 | |
| *** _nadya__ has joined #openstack-infra | 12:00 | |
| *** _nadya_ has quit IRC | 12:00 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Permit pre-release versions with git metadata https://review.openstack.org/80857 | 12:01 |
| *** Ajaeger has quit IRC | 12:02 | |
| *** ArxCruz has joined #openstack-infra | 12:02 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Make setUp fail if sdist fails. https://review.openstack.org/94106 | 12:03 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Modify horizon log copy for Fedora https://review.openstack.org/93251 | 12:03 |
| *** lcostantino has joined #openstack-infra | 12:06 | |
| *** afazekas has quit IRC | 12:07 | |
| *** salv-orlando has quit IRC | 12:07 | |
| *** mwagner_lap has quit IRC | 12:08 | |
| *** mbacchi has joined #openstack-infra | 12:09 | |
| *** IvanBerezovskiy has joined #openstack-infra | 12:09 | |
| *** flaper87 is now known as flaper87|afk | 12:11 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Remove unnecessary files https://review.openstack.org/95741 | 12:13 |
| *** dstanek_zzz is now known as dstanek | 12:14 | |
| *** yaguang has quit IRC | 12:14 | |
| *** e0ne has joined #openstack-infra | 12:16 | |
| *** dprince has joined #openstack-infra | 12:18 | |
| *** dims has quit IRC | 12:20 | |
| openstackgerrit | Radoslav Gerganov proposed a change to openstack-infra/config: Add button that shows/hides CI comments in Gerrit https://review.openstack.org/95743 | 12:20 |
| *** ok_delta has joined #openstack-infra | 12:21 | |
| *** afazekas has joined #openstack-infra | 12:21 | |
| rgerganov | hi folks | 12:24 |
| rgerganov | I am trying to add a piece of javascirpt in GerritSiteHeader.html that will add "Toggle CI" button | 12:25 |
| rgerganov | in order to show/hide CI comments | 12:25 |
| rgerganov | you can see the patch above | 12:25 |
| *** dprince has quit IRC | 12:25 | |
| *** dprince has joined #openstack-infra | 12:26 | |
| rgerganov | my question is how can I test such a change? I can install it as userscript in my browser but now I am trying to push this on the server side | 12:27 |
| *** salv-orlando has joined #openstack-infra | 12:27 | |
| *** yfried_ has joined #openstack-infra | 12:29 | |
| *** yfried_ has quit IRC | 12:30 | |
| hashar | rgerganov: Openstack has a Gerrit dev box so they can probably fetch your change there and try it out | 12:30 |
| *** yfried_ has joined #openstack-infra | 12:30 | |
| *** afazekas is now known as _afazekas_mtg | 12:31 | |
| rgerganov | hashar, thanks. who would be to proper contact for this? | 12:31 |
| hashar | I have no clue :-] | 12:31 |
| hashar | rgerganov: and during the summit there was apparently a discussion to normalize the name of third party CI bots. | 12:31 |
| rgerganov | hashar, yes, that would be nice | 12:32 |
| hashar | I am sure I have seen a mail about normalization | 12:32 |
| hashar | but can't find it :D | 12:32 |
| *** aysyd has joined #openstack-infra | 12:33 | |
| *** yfried has quit IRC | 12:33 | |
| rgerganov | I guess I can test my change by hacking an http proxy and inserting the script behind the scenes | 12:34 |
| rgerganov | but I am looking for something easier :) | 12:34 |
| hashar | they will wake/show up in a few hours | 12:34 |
| hashar | or you can ping the openstack-infra mailing list http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra | 12:35 |
| rgerganov | hashar, ok, thanks for the info | 12:35 |
| *** dstanek is now known as dstanek_zzz | 12:41 | |
| *** yamahata has joined #openstack-infra | 12:42 | |
| *** yfried_ has quit IRC | 12:42 | |
| *** eharney has joined #openstack-infra | 12:43 | |
| *** dims has joined #openstack-infra | 12:45 | |
| *** yamahata has quit IRC | 12:45 | |
| *** che-arne has quit IRC | 12:45 | |
| *** andreykurilin has quit IRC | 12:46 | |
| *** yamahata has joined #openstack-infra | 12:46 | |
| *** habib has joined #openstack-infra | 12:48 | |
| *** habib has quit IRC | 12:49 | |
| *** gokrokve has joined #openstack-infra | 12:49 | |
| *** habib has joined #openstack-infra | 12:49 | |
| *** dstanek_zzz is now known as dstanek | 12:50 | |
| *** salv-orlando has quit IRC | 12:51 | |
| *** jgrimm has quit IRC | 12:53 | |
| *** gokrokve has quit IRC | 12:54 | |
| *** nati_ueno has joined #openstack-infra | 12:55 | |
| *** radez_g0n3 is now known as radez | 12:56 | |
| *** heyongli has joined #openstack-infra | 12:57 | |
| *** nati_ueno has quit IRC | 13:00 | |
| *** pblaho has quit IRC | 13:00 | |
| dhellmann | mordred: is the intent for the wheel publishing change to stop publishing tarballs? I don't see anything in https://review.openstack.org/#/c/56760 that's doing that, it seems to just be adding wheel publishing | 13:00 |
| Alex_Gaynor | dhellmann: this is for our mirror, not pypi right? | 13:01 |
| *** pblaho has joined #openstack-infra | 13:01 | |
| Alex_Gaynor | (pip will already prefers wheels, so it seems like adding them is more backwards compatible) | 13:01 |
| dhellmann | Alex_Gaynor: the scripts use twine so I thought it was pypi | 13:01 |
| dhellmann | Alex_Gaynor: https://review.openstack.org/#/c/56760/7/modules/openstack_project/files/jenkins_job_builder/config/pypi-jobs.yaml | 13:01 |
| dhellmann | also "pypi-jobs" | 13:02 |
| *** yfried_ has joined #openstack-infra | 13:02 | |
| *** yfried_ has quit IRC | 13:03 | |
| *** yfried__ has joined #openstack-infra | 13:03 | |
| fungi | rgerganov: dhellmann some of the reasoning behind building and uploading wheels was that we could publish wheels to pypi for prereleases (but not tarballs) since versions of pip which grok wheels also don't install prerelease version patterns by default | 13:03 |
| *** _afazekas_mtg has quit IRC | 13:04 | |
| fungi | rgerganov: er, sorry, runaway tab completion there | 13:04 |
| dhellmann | fungi: that's what I thought, but that's not what mordred's change (which I'm trying to rebase) seems to be doing | 13:04 |
| dhellmann | fungi: it calls both upload scripts from the pypi-upload job | 13:05 |
| *** julim has joined #openstack-infra | 13:05 | |
| fungi | dhellmann: well, it calls both scripts from the tarball builder... still digging to see whether anything besides the wheel publisher does anything with the results | 13:06 |
| *** jcoufal has quit IRC | 13:07 | |
| *** jcoufal_ has joined #openstack-infra | 13:07 | |
| fungi | dhellmann: ahh, yeah it does seem to build tarballs too. i suspect we need a tarball-pypi uploader and a wheel-pypi uploader separate | 13:07 |
| dhellmann | fungi: makes sense, and I had actually already made that split to prevent tarball upload issues from breaking wheel upload issues | 13:08 |
| *** pblaho has quit IRC | 13:08 | |
| fungi | dhellmann: yeah, i just meant they probably need to be separate jobs so we can add both to the release pipeline, but only wheels to the pre-release pipeline | 13:09 |
| dhellmann | ah, right | 13:09 |
| *** pblaho has joined #openstack-infra | 13:10 | |
| *** signed8bit has joined #openstack-infra | 13:11 | |
| dhellmann | fungi: the {name}-tarball job template builds both tarballs and wheels, what do you think about renaming that {name}-dists or something similar? | 13:11 |
| *** yfried__ is now known as yfried | 13:12 | |
| fungi | dhellmann: seems fine to me. i'm trying to think about what implications this has for my tarball validation and signing plan, but i haven't had enough coffee yet so i've got nothing | 13:12 |
| dhellmann | the tarball builder does both, too | 13:12 |
| fungi | building both, and even publishing both on tarballs.o.o, seems okay | 13:13 |
| dhellmann | it looks like he piggybacked on the existing tarball stuff pretty heavily | 13:13 |
| *** changbl has quit IRC | 13:13 | |
| dhellmann | fungi: ok, I'll leave the builder alone | 13:13 |
| fungi | it's mainly which we upload to pypi that i'm concerned about being able to split up | 13:13 |
| Alex_Gaynor | dhellmann: for pypi we definitely still want to upload sdists | 13:14 |
| *** oomichi has quit IRC | 13:14 | |
| dhellmann | Alex_Gaynor: sure | 13:15 |
| dhellmann | fungi: I'm having some trouble figuring out how to express the pipeline in the layout file | 13:15 |
| dhellmann | do I want both the tarball-upload and wheel upload to run before the post-mirror-* jobs? | 13:16 |
| *** _afazekas_mtg has joined #openstack-infra | 13:16 | |
| dhellmann | and if so, how would I express that with the tree structure? maybe that's why he kept both uploads in one job? | 13:16 |
| *** pblaho has quit IRC | 13:17 | |
| fungi | dhellmann: hrm, yeah right now zuul doesn't allow a child job with multiple parents. we've discussed the possibility of being able to set dependencies on job groups, but that's still on the drawing board | 13:17 |
| fungi | er, not job groups in the jjb sense, but some as of yet not implemented grouping structure in zuul | 13:17 |
| dhellmann | fungi: how about this: http://paste.openstack.org/show/81690/ | 13:17 |
| dhellmann | (the changes are in the pre-release and release sections) | 13:17 |
| dhellmann | I'll need to define separate "both-upload" and "wheel-upload" jobs | 13:18 |
| fungi | dhellmann: yep, that would work | 13:18 |
| dhellmann | fungi: great, thanks | 13:18 |
| *** _afazekas_mtg is now known as afazekas | 13:19 | |
| anteaya | hashar rgerganov that is the 4th item on today's infra meeting agenda: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting | 13:20 |
| anteaya | hashar rgerganov it would be great to have your input for that item | 13:20 |
| hashar | ah I knew I see it somewhere :-] | 13:20 |
| fungi | Alex_Gaynor: you wouldn't happen to have any ideas on how to go about validating/reversing a wheel build to confirm whether it was built from tampered sources, given the wheel file and the tagged git repo, would you? i know how to roughly go about doing that for a tarball, but... | 13:20 |
| anteaya | hashar: :D | 13:21 |
| *** _nadya__ has quit IRC | 13:21 | |
| hashar | anteaya: rgerganov wrote a short JS that would let one hide the reviews proposed by CI system. So I am sure normalizing the names will help a lot ! | 13:21 |
| *** andreykurilin has joined #openstack-infra | 13:21 | |
| Alex_Gaynor | fungi: build a wheel from the sources and diff with the wheel? I'm not sure if that process is detemrinistic, but it's a good starting point /cc dstufft | 13:21 |
| anteaya | yes, I am following the email thread, there are many folks who would use that feature gladly | 13:22 |
| Alex_Gaynor | fungi: alternatively, install from source, unpack the wheel on disk, and diff those | 13:22 |
| dhellmann | fungi: do the templates in layout.yaml support nesting? could I have a "publish-wheels" template and a "publish-tarballs" template with the dependency structure, instead of repeating myself? | 13:22 |
| *** jaypipes has joined #openstack-infra | 13:22 | |
| fungi | Alex_Gaynor: yeah, the challenge i'm trying to tackle is to build the dist on an untrusted system (since it has to run untrusted code to build) but then validate it on a trusted system without running untrusted code prior to uploading | 13:23 |
| dhellmann | Alex_Gaynor: would you have to install from source or just compile what was checked out? | 13:23 |
| fungi | dhellmann: i'm not actually sure if zuul project templates support nesting/recursion | 13:23 |
| dhellmann | fungi: not recursion, just the tree structure | 13:24 |
| *** mriedem has joined #openstack-infra | 13:24 | |
| dhellmann | fungi: can I have "cliff-tarball: cliff-pypi-wheel-upload: ..." as a tree? | 13:24 |
| fungi | dhellmann: i'm looking to see if there are existing examples, but presumably the layout validator would complain if you tried and it didn't support doing that | 13:24 |
| dhellmann | I'm not even sure how to validate what I have locally | 13:24 |
| anteaya | rgerganov hashar yeah if we could figuring out a nameing format for third party ci that was easy to catch with a regex that would be awesome | 13:24 |
| *** malini_afk is now known as malini | 13:24 | |
| anteaya | s/figuring/figure | 13:25 |
| Alex_Gaynor | fungi: I think my second solution is safe, if you trust the source version; just unpacking the wheel is safe | 13:25 |
| fungi | Alex_Gaynor: ahh, yeah that's what i was planning to do with the tarballs, but i'm not familiar enough with wheel internal formatting to know how feasible that would be (seems like it's more cooked than a plain tarball, but i'll dig into it) | 13:26 |
| Alex_Gaynor | fungi: a wheel is literally a zipped up version of whatever ends up in site-packages when you setup.py install (basically); /cc dstufft | 13:27 |
| krtaylor | anteaya, yes, and use the same tag for email so we can filter | 13:27 |
| fungi | Alex_Gaynor: ahh, in that case i can probably just filter out entrypoint wrappers and a few other things. thanks | 13:28 |
| fungi | i had assumed it was all byte-compiled .pyc files inside | 13:28 |
| anteaya | krtaylor: agreed | 13:29 |
| krtaylor | anteaya, nm, I am reading the agenda, that is the subject of the discussion, I need more coffee | 13:29 |
| anteaya | krtaylor: :D | 13:30 |
| anteaya | glad you are here | 13:30 |
| anteaya | yeah, I have to forage for some breakfast | 13:30 |
| *** W00dy_ has quit IRC | 13:31 | |
| Alex_Gaynor | fungi: ah, no, I don't even think PYCs are included, although truth be told I'm not sure | 13:31 |
| *** Longgeek has quit IRC | 13:31 | |
| *** Longgeek has joined #openstack-infra | 13:31 | |
| *** _nadya_ has joined #openstack-infra | 13:32 | |
| fungi | dhellmann: http://git.openstack.org/cgit/openstack-infra/config/tree/tools/run-layout.sh | 13:33 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Name fields checked with regex https://review.openstack.org/95763 | 13:33 |
| *** habib has quit IRC | 13:33 | |
| dhellmann | fungi: thanks | 13:33 |
| fungi | dhellmann: we run it in the gate-config-layout job with "./tools/run-layout.sh openstack-infra config" | 13:33 |
| *** habib has joined #openstack-infra | 13:33 | |
| dhellmann | fungi: I'm not sure how much benefit I'll get from using a template anyway, since some of the repos support python 3 and some don't | 13:34 |
| *** dripton_ is now known as dripton | 13:35 | |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Enabled pylint job for manila project https://review.openstack.org/95765 | 13:36 |
| fungi | dhellmann: and to the other question, i don't see any current examples of a project-template definition invoking another template, so probably not supported (yet anyway) | 13:36 |
| dhellmann | fungi: this is what I was going to try as a template: http://paste.openstack.org/show/81698/ | 13:36 |
| *** _nadya_ has quit IRC | 13:37 | |
| *** sballe_ has joined #openstack-infra | 13:37 | |
| fungi | dhellmann: looks sane enough | 13:37 |
| *** habib has quit IRC | 13:38 | |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add hyperlinks https://review.openstack.org/95766 | 13:38 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Correct a problem with tables at very small widths https://review.openstack.org/95767 | 13:38 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add patchset selection in diff https://review.openstack.org/95768 | 13:38 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Handle (ignore) no-diff renames https://review.openstack.org/95769 | 13:38 |
| *** habib has joined #openstack-infra | 13:39 | |
| *** sballe has quit IRC | 13:39 | |
| *** dkliban_gone is now known as dkliban | 13:40 | |
| mkoderer | dhellmann: ping | 13:41 |
| *** sballe_ has quit IRC | 13:42 | |
| dhellmann | mkoderer: pong | 13:42 |
| *** flaper87|afk is now known as flaper87 | 13:44 | |
| *** msabramo has joined #openstack-infra | 13:44 | |
| mkoderer | dhellmann: about your comment on https://review.openstack.org/#/c/95411/ | 13:45 |
| mkoderer | dhellmann: your are pointing to a merged patch that wasn't merged as I initially uploaded the patch | 13:46 |
| mkoderer | how can this be related? is the wrong sphinx version installed? | 13:47 |
| *** prad_ has joined #openstack-infra | 13:47 | |
| *** jgrimm has joined #openstack-infra | 13:47 | |
| *** wenlock has joined #openstack-infra | 13:47 | |
| *** liyuezho has joined #openstack-infra | 13:48 | |
| *** liyuezho has quit IRC | 13:48 | |
| dhellmann | mkoderer: time passes, it wasn't merged when I left the comment | 13:48 |
| mkoderer | dhellmann: ah ok, so I rerun the test but it's still failing | 13:48 |
| dhellmann | mkoderer: I think if you rebase your patch and resubmit it, it should work | 13:49 |
| dhellmann | mkoderer: hang on, let me check if my other stuff landed | 13:49 |
| mkoderer | dhellmann: ok | 13:49 |
| *** gokrokve has joined #openstack-infra | 13:49 | |
| *** jcoufal has joined #openstack-infra | 13:49 | |
| dhellmann | mkoderer: yeah, https://review.openstack.org/#/c/95343/ landed so yours should work if you rebase | 13:49 |
| mkoderer | dhellmann: k cool thx | 13:50 |
| *** jcoufal_ has quit IRC | 13:52 | |
| *** doug-fish has joined #openstack-infra | 13:53 | |
| *** gokrokve has quit IRC | 13:53 | |
| *** sballe has joined #openstack-infra | 13:54 | |
| *** igor_ has quit IRC | 13:54 | |
| *** homeless has joined #openstack-infra | 13:55 | |
| *** nati_ueno has joined #openstack-infra | 13:56 | |
| *** pblaho has joined #openstack-infra | 13:57 | |
| *** zhiyan is now known as zhiyan_ | 13:57 | |
| *** sballe has quit IRC | 13:59 | |
| *** annegentle has joined #openstack-infra | 13:59 | |
| *** nati_ueno has quit IRC | 14:00 | |
| *** beekneemech is now known as bnemec | 14:01 | |
| *** Longgeek has quit IRC | 14:03 | |
| *** wenlock has quit IRC | 14:05 | |
| *** lcheng_ has joined #openstack-infra | 14:06 | |
| *** hashar has quit IRC | 14:06 | |
| *** Longgeek has joined #openstack-infra | 14:07 | |
| *** gokrokve has joined #openstack-infra | 14:07 | |
| *** heyongli has quit IRC | 14:07 | |
| *** gokrokve_ has joined #openstack-infra | 14:08 | |
| *** duran has joined #openstack-infra | 14:08 | |
| *** yfried has quit IRC | 14:09 | |
| *** gokrokve has quit IRC | 14:11 | |
| *** pblaho has quit IRC | 14:12 | |
| *** flaper87 is now known as flaper87|afk | 14:12 | |
| Alex_Gaynor | Is some part of zuul having troubles? Just sent a thing to gerrit and it hasn't produced a job | 14:12 |
| fungi | Alex_Gaynor: what thing? | 14:12 |
| Alex_Gaynor | fungi: https://review.openstack.org/#/c/95777/ | 14:12 |
| *** rdopiera has quit IRC | 14:13 | |
| fungi | Alex_Gaynor: i do see "Queue lengths: 37 events, 317 results." on http://status.openstack.org/zuul/ so it may be dealing with a config reload or something... checking | 14:13 |
| Alex_Gaynor | fungi: doh, I forgot those lengths were there; thanks! | 14:13 |
| fungi | Alex_Gaynor: the results queue seems to be falling, fwiw, so it will probably right itself here shortly | 14:14 |
| Alex_Gaynor | fungi: yeah, sorry about the noise | 14:14 |
| *** Kiall_ is now known as Kiall | 14:14 | |
| *** habib has quit IRC | 14:15 | |
| fungi | well, those swift changes in the gate seem to be cycling through devstack-precise nodes at a steady clip, and we did just hit our nodepool image rebuild time | 14:15 |
| fungi | playing "spot the broken" now | 14:15 |
| corvus | fungi: i'm online if you need a hand | 14:16 |
| Alex_Gaynor | Did a jenkins worker die? | 14:16 |
| *** corvus is now known as jeblair | 14:16 | |
| *** habib has joined #openstack-infra | 14:17 | |
| fungi | doesn't seem likely to be a new image causing issues... none have been building long enough to become ready and spawn new nodes. must be something more external | 14:18 |
| *** msabramo has quit IRC | 14:18 | |
| *** yaguang has joined #openstack-infra | 14:20 | |
| *** lcheng_ has quit IRC | 14:22 | |
| *** lcheng_ has joined #openstack-infra | 14:24 | |
| *** timrc is now known as timrc-afk | 14:24 | |
| *** habib has quit IRC | 14:25 | |
| *** habib has joined #openstack-infra | 14:25 | |
| *** james_li has joined #openstack-infra | 14:26 | |
| jeblair | fungi: zuul just lost its gearman connection | 14:26 |
| fungi | aha | 14:26 |
| fungi | and aborted all running jobs | 14:26 |
| jeblair | which, on the one hand is bad, but on the other, hopefully it means we have enough log entries this time to debug it | 14:26 |
| *** ihrachyshka has quit IRC | 14:26 | |
| fungi | that's why i'm not finding any new jobs exhibiting an obvious problem | 14:26 |
| jeblair | fungi: that _just_ happened though; i'm not sure if that's related to anything prior | 14:27 |
| jeblair | i'm going to eat breakfast then dive into debugging that | 14:28 |
| *** sileht has quit IRC | 14:28 | |
| fungi | sounds good. i'll see if i can dig up anything | 14:28 |
| *** pcrews has joined #openstack-infra | 14:29 | |
| *** habib has quit IRC | 14:29 | |
| *** sileht has joined #openstack-infra | 14:29 | |
| *** david-lyle has joined #openstack-infra | 14:30 | |
| *** sballe_ has joined #openstack-infra | 14:31 | |
| *** gokrokve has joined #openstack-infra | 14:33 | |
| *** atiwari has joined #openstack-infra | 14:35 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for Neutron SSH EOFError bug 1323658 https://review.openstack.org/95782 | 14:35 |
| uvirtbot | Launchpad bug 1323658 in neutron "SSH EOFError - Public network connectivity check failed" [Undecided,New] https://launchpad.net/bugs/1323658 | 14:35 |
| *** gokrokve_ has quit IRC | 14:36 | |
| *** otherwiseguy has joined #openstack-infra | 14:36 | |
| phschwartz | morning infra | 14:37 |
| anteaya | morning phschwartz | 14:37 |
| *** sileht has quit IRC | 14:37 | |
| fungi | jeblair: looks like it's happened four times today, the earliest was 14:06:26 | 14:39 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Create and upload wheels https://review.openstack.org/56760 | 14:39 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Make it possible to run zuul layout test locally https://review.openstack.org/95783 | 14:39 |
| *** vhoward has left #openstack-infra | 14:39 | |
| *** sileht has joined #openstack-infra | 14:39 | |
| *** vhoward has joined #openstack-infra | 14:39 | |
| dhellmann | fungi: ^^ changes related to what we were discussing earlier | 14:39 |
| jeblair | fungi: ah that might explain the event backlog then | 14:39 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add sahara-specs repo https://review.openstack.org/95715 | 14:40 |
| fungi | seeing what else of note preceded it in the log | 14:40 |
| *** Longgeek_ has joined #openstack-infra | 14:41 | |
| SergeyLukjanov | anteaya, thanks for the top | 14:41 |
| SergeyLukjanov | tip* | 14:41 |
| *** msabramo has joined #openstack-infra | 14:42 | |
| anteaya | SergeyLukjanov: np | 14:44 |
| *** rgerganov has quit IRC | 14:44 | |
| * anteaya reviews again | 14:44 | |
| openstackgerrit | A change was merged to stackforge/gertty: Use alembic https://review.openstack.org/92744 | 14:44 |
| openstackgerrit | A change was merged to stackforge/gertty: Add some helper methods to deal with sqlite migrations https://review.openstack.org/92745 | 14:44 |
| openstackgerrit | A change was merged to stackforge/gertty: Increase the status field width https://review.openstack.org/92746 | 14:44 |
| openstackgerrit | A change was merged to stackforge/gertty: Sync parent changes https://review.openstack.org/92753 | 14:44 |
| *** Longgeek has quit IRC | 14:44 | |
| openstackgerrit | A change was merged to stackforge/gertty: Fix handling no newline at EOF in both files https://review.openstack.org/93997 | 14:44 |
| openstackgerrit | A change was merged to stackforge/gertty: Removed closed changes from unreviewed list. https://review.openstack.org/92747 | 14:44 |
| anteaya | +1 | 14:44 |
| anteaya | SergeyLukjanov: thanks | 14:44 |
| SergeyLukjanov | anteaya, thx ;) | 14:45 |
| *** sballe_ has quit IRC | 14:45 | |
| *** habib has joined #openstack-infra | 14:45 | |
| *** alugovoi has joined #openstack-infra | 14:46 | |
| *** habib has quit IRC | 14:46 | |
| *** habib has joined #openstack-infra | 14:47 | |
| *** skolekonov has quit IRC | 14:48 | |
| *** sileht has quit IRC | 14:49 | |
| *** wenlock has joined #openstack-infra | 14:49 | |
| mordred | dhellmann: woot! | 14:49 |
| *** timrc-afk is now known as timrc | 14:50 | |
| *** sileht has joined #openstack-infra | 14:51 | |
| cody-somerville | clarkb: mordred: Can you review https://review.openstack.org/#/c/93870/ s'il-vous-plait? :) | 14:51 |
| signed8bit | yup | 14:52 |
| dhellmann | mordred: those layout changes are a little scary, so please look them over closely -- I did manage to run some tests locally that make me think I have them right, but still. | 14:52 |
| * signed8bit didn't mean to type that... wrong focus | 14:52 | |
| *** thedodd has joined #openstack-infra | 14:53 | |
| jpich | Hello! Is there a way to block or report LP users? Someone filed a couple of bugs containing only spam. Marking them as Invalid is fine for now but it's going to get irritating | 14:55 |
| mordred | cody-somerville: lgtm | 14:55 |
| fungi | jpich: either try to get someone's attention in #launchpad or file a bug against "launchpad itself" noting the problem behavior | 14:55 |
| jeblair | fungi, jpich: opening a "question" on the "launchpad itself" may be better than filing a bug? | 14:56 |
| ttx | jeblair: yes, they are actually reactive on "questions" | 14:56 |
| ttx | whereas bugs... not so much | 14:57 |
| fungi | oops, yes i forgot it was lp answers not bugs they used for support requests | 14:57 |
| *** nati_ueno has joined #openstack-infra | 14:57 | |
| *** otherwiseguy has quit IRC | 14:57 | |
| jpich | fungi jeblair ttx: Fair enough, will do. Thanks! | 14:57 |
| *** otherwiseguy has joined #openstack-infra | 14:58 | |
| openstackgerrit | Dan Prince proposed a change to openstack-infra/config: Add yum.openstack.org lightweight Fedora 20 mirror https://review.openstack.org/90875 | 14:58 |
| openstackgerrit | Dan Prince proposed a change to openstack-infra/config: Import puppet-yum project https://review.openstack.org/90874 | 14:59 |
| openstackgerrit | Dan Prince proposed a change to openstack-infra/config: Install the openstackci-yum module. https://review.openstack.org/95787 | 14:59 |
| *** Longgeek_ has quit IRC | 15:00 | |
| *** alugovoi has quit IRC | 15:00 | |
| *** Longgeek has joined #openstack-infra | 15:01 | |
| *** lascii is now known as alaski | 15:01 | |
| *** nati_ueno has quit IRC | 15:01 | |
| *** KurtMartin is now known as kmartin | 15:01 | |
| *** malini is now known as malini_afk | 15:02 | |
| *** malini_afk is now known as malini | 15:03 | |
| *** yaguang has quit IRC | 15:03 | |
| *** morganfainberg_Z is now known as morganfainberg | 15:04 | |
| *** moted has joined #openstack-infra | 15:05 | |
| *** zhiyan_ is now known as zhiyan | 15:05 | |
| *** blamar has quit IRC | 15:06 | |
| *** malini has left #openstack-infra | 15:07 | |
| *** blamar has joined #openstack-infra | 15:10 | |
| *** terryw has joined #openstack-infra | 15:10 | |
| Alex_Gaynor | fungi: I think CI jobs actually aren't starting this time :-) | 15:10 |
| *** otherwiseguy has quit IRC | 15:11 | |
| *** andreykurilin has quit IRC | 15:11 | |
| *** andreykurilin has joined #openstack-infra | 15:12 | |
| *** BadCub has joined #openstack-infra | 15:12 | |
| *** talluri has joined #openstack-infra | 15:13 | |
| mordred | dhellmann: the changes look good so far! | 15:14 |
| *** zhiyan is now known as zhiyan_ | 15:14 | |
| annegentle | ttx: around? | 15:15 |
| mordred | dhellmann: why not have publish-wheels for openstack/sahara? | 15:15 |
| dhellmann | mordred: I was trying to replace the jobs that were there, without making decisions about adding new ones. Did they have a pre-release publish job before? | 15:16 |
| dhellmann | mordred: ah, they have a tarball but they weren't doing any mirror syncing | 15:17 |
| dhellmann | mordred: do we need that for things that aren't libraries? | 15:17 |
| *** che-arne has joined #openstack-infra | 15:17 | |
| *** rfolco has joined #openstack-infra | 15:18 | |
| *** talluri has quit IRC | 15:18 | |
| *** afazekas has quit IRC | 15:18 | |
| fungi | Alex_Gaynor: it looks like we've had zuul internal gearman timeouts between 14:06 and 14:44, so we may also have considerable node starvation from all the job restarts | 15:20 |
| Alex_Gaynor | fungi: but it'll heal itself? | 15:20 |
| fungi | Alex_Gaynor: not sure yet--still digging but it seems to have stopped flailing | 15:20 |
| Alex_Gaynor | fungi: ok, good to know -- I wont' bother folks in the future if patience is all it takes | 15:21 |
| annegentle | Hi all, I'm trying to push a new tag for openstack-doc-tools. It's 0.15, which doesn't exist on github (https://github.com/openstack/openstack-doc-tools/tags) and I don't see how to view tags on gerrit web view... any ideas for me? | 15:21 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/config: Index logs/tempest.txt for logstash queries https://review.openstack.org/95796 | 15:21 |
| mriedem | sdague: mtreinish: clarkb: ^ | 15:21 |
| annegentle | The error I'm seeing is "error: src refspec 0.15 matches more than one." | 15:21 |
| ttx | annegentle: multiplexing, but yes | 15:21 |
| annegentle | ttx: ok I was looking for guidance on the defcore task, I've worked on many of those line items but wanted to know what we want to do in the tc meeting | 15:22 |
| annegentle | ttx: is it research for the red rows? | 15:22 |
| ttx | We want to verify scores on the "TC direction" column | 15:22 |
| ttx | annegentle: especially the 0.5 | 15:23 |
| *** gyee has joined #openstack-infra | 15:23 | |
| ttx | annegentle: the red lines are first for PTLs to fill, we'll step up if it's blocked | 15:23 |
| annegentle | ttx: ok got it, and does a 1 mean "matches TC direction" | 15:23 |
| fungi | Alex_Gaynor: no, please do bother us ;) | 15:23 |
| annegentle | ttx: ok good | 15:23 |
| annegentle | fungi: oh good then I'll bother you with my refspec 0.15 multi match! :) | 15:23 |
| ttx | annegentle: yes, 1 means we care | 15:23 |
| ttx | 0 means it's probably deprecated tomorrow | 15:24 |
| ttx | 0.5 is "will die some day" | 15:24 |
| fungi | annegentle: i have no idea what that means, but sure, why not | 15:24 |
| annegentle | ttx: and 0.5 means they didn't know? | 15:24 |
| ttx | at least that's how I'd score it | 15:24 |
| clarkb | mriedem: would it be mad of me to -1 that on grounds of that log file being huge and full of noise? | 15:24 |
| annegentle | fungi: oh right it's an error I'm seeing when trying to push a tag to gerrit for openstack-doc-tools (scroll up) | 15:24 |
| annegentle | I'm multistasking too much :) | 15:24 |
| clarkb | INFO started http connection over and over and over | 15:24 |
| annegentle | ttx: do we get to use "die in a fire" | 15:24 |
| annegentle | ttx: actually I don't feel that strongly about any of those to mark them diaf | 15:25 |
| ttx | heh, yes | 15:25 |
| fungi | annegentle: aha. i missed that earlier | 15:25 |
| ttx | annegentle: I found the scores mostly correct imho | 15:25 |
| mriedem | clarkb: is there a way to configure it such that it only indexes on certain log levels? | 15:25 |
| ttx | annegentle: but we'll discuss them later | 15:25 |
| annegentle | ttx: yeah I think so too (and I helped with some of them so I hope I remain consistent ha) | 15:25 |
| mriedem | clarkb: like WARNING and higher? | 15:25 |
| annegentle | ttx: sounds good thanks | 15:25 |
| *** mkerrin1 has quit IRC | 15:26 | |
| clarkb | mriedem: it does INFO and higher | 15:26 |
| mriedem | clarkb: i know, but wondering if i can tell the tooling to ignore everything below WARNING | 15:26 |
| mriedem | until we can clean up the logging | 15:26 |
| annegentle | fungi: I definitely have a 0.15 locally | 15:26 |
| clarkb | I don't think you can configure that today | 15:26 |
| annegentle | fungi: just can't figure out why git thinks there's already one remotely | 15:26 |
| clarkb | but its just python so that can be changed | 15:26 |
| fungi | annegentle: what command are you running that you're getting that error? 'git tag ...' or 'git push ...' or something else? | 15:28 |
| annegentle | fungi: git push gerrit 0.15 | 15:28 |
| mriedem | clarkb: let me take a look at the tempest log at INFO level and see if there should also be a change in tempest, i.e. INFO started http connection to debug | 15:28 |
| annegentle | fungi: first, git tag -s 0.15 | 15:28 |
| mriedem | clarkb: then i can get all the deps lined up | 15:28 |
| *** mrodden has quit IRC | 15:29 | |
| clarkb | ok | 15:29 |
| fungi | annegentle: so that error seems to imply that you have more than one local ref named "0.15" | 15:29 |
| fungi | annegentle: probably a branch? | 15:29 |
| mriedem | clarkb: feel free to -1 though | 15:29 |
| mriedem | until that happens | 15:29 |
| fungi | annegentle: does 'git branch -a | grep 0\.15' give you something? | 15:30 |
| *** mkerrin has joined #openstack-infra | 15:30 | |
| annegentle | fungi: yep, I have two branches with 0.15 in the name | 15:30 |
| *** Ajaeger has joined #openstack-infra | 15:30 | |
| annegentle | fungi: one from the first time I tried and got the error, then deleted the tag locally, then tried again with the second branch | 15:30 |
| fungi | annegentle: try renaming those with 'git branch -m old_branch new_branch' | 15:31 |
| annegentle | fungi: ok those are renamed now | 15:32 |
| annegentle | fungi: trying again | 15:33 |
| fungi | annegentle: basically git is complaining that it doesn't know whether you're wanting to push a tag named 0.15 or a branch named 0.15 | 15:33 |
| annegentle | fungi: ha that did it. Dangit. | 15:33 |
| fungi | since you had both | 15:33 |
| annegentle | fungi: thanks much! Something about memorial day made me not think about that. | 15:33 |
| annegentle | fungi: :) | 15:33 |
| Ajaeger | fungi, great that you could solve the mystery! | 15:33 |
| fungi | annegentle: lots of things about memorial day just made me not think, so i undertand | 15:34 |
| fungi | understand | 15:34 |
| fungi | annegentle: Ajaeger: yep, i see your new tag now at http://git.openstack.org/cgit/openstack/openstack-doc-tools | 15:35 |
| Ajaeger | Great! | 15:35 |
| *** unicell has joined #openstack-infra | 15:35 | |
| *** NithyaG is now known as NithyaG_afk | 15:35 | |
| Ajaeger | fungi, in that case I have a few more favors to ask ;) Could you review https://review.openstack.org/95414, please? | 15:36 |
| Ajaeger | We're getting of the special asciidoc handling - and openstack-doc-tools 0.15 contains a corresponding change. | 15:36 |
| *** pdmars has joined #openstack-infra | 15:36 | |
| *** alugovoi has joined #openstack-infra | 15:37 | |
| Ajaeger | Anybody else around that can review https://review.openstack.org/95414 as well, please? | 15:37 |
| *** msabramo has quit IRC | 15:39 | |
| *** jgrimm has quit IRC | 15:39 | |
| anteaya | Ajaeger: +1 | 15:39 |
| *** zhiyan_ is now known as zhiyan | 15:39 | |
| Ajaeger | thanks, anteaya | 15:40 |
| *** mrodden has joined #openstack-infra | 15:41 | |
| *** msabramo has joined #openstack-infra | 15:43 | |
| *** dangers_away is now known as dangers | 15:43 | |
| *** ominakov has quit IRC | 15:43 | |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Perform http calls outside of the db session https://review.openstack.org/94097 | 15:44 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Don't show closed changes in the open list https://review.openstack.org/94085 | 15:44 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Change review toggle keybinding to 'v'. https://review.openstack.org/94086 | 15:44 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add 'killthread' https://review.openstack.org/94087 | 15:44 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add jump to change https://review.openstack.org/94088 | 15:44 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Fix closing stacked dialogs https://review.openstack.org/94089 | 15:44 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Ensure single-threaded db access https://review.openstack.org/94090 | 15:44 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add local cherry-pick button https://review.openstack.org/94091 | 15:44 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add a project list header https://review.openstack.org/94092 | 15:44 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Change config file to YAML https://review.openstack.org/94093 | 15:44 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add custom palettes and commentlinks https://review.openstack.org/94094 | 15:44 |
| mriedem | clarkb: bug 1323726 - 17880 hits of tempest.common.rest_client in one log file, so yeah, a bit excessive | 15:46 |
| uvirtbot | Launchpad bug 1323726 in tempest "tempest.common.rest_client is too chatty at INFO level" [Undecided,New] https://launchpad.net/bugs/1323726 | 15:46 |
| mriedem | i'll make a change there | 15:46 |
| *** markwash has joined #openstack-infra | 15:49 | |
| *** zhiyan is now known as zhiyan_ | 15:51 | |
| mordred | dhellmann: I thnk the new idea in my head is that we should go ahead and start doing the wheels for pre-releases and tarballs for releases for everything now ... but we can also take that as a second pass | 15:51 |
| dhellmann | mordred: yeah, that makes sense | 15:51 |
| mordred | dhellmann: like, if we have access to the pypi account for something, we should just across teh board do the pre-release/release pattern | 15:51 |
| * dhellmann nods | 15:52 | |
| *** jgrimm has joined #openstack-infra | 15:52 | |
| *** zehicle_at_dell has quit IRC | 15:52 | |
| *** msabramo has quit IRC | 15:53 | |
| mordred | dhellmann: in fact, I wonder if maybe we shouldn't just collapse your two layout templates into a single publish-to-pypi one- so that if we publish to pypi for someone, we do it one way | 15:53 |
| mordred | (or two - publish-to-pypi and publish-2only) | 15:54 |
| Alex_Gaynor | mordred, dhellmann: I know we said that uploading wheels of pre-releases would be fine, but when Django tried to do it, we hit some snag, trying to remember what it was /cc dstufft | 15:54 |
| dhellmann | mordred: that would be a clean way to do it; should I do it in this patch? | 15:54 |
| mordred | Alex_Gaynor: oh, that would be a good thing to know | 15:54 |
| fungi | digging into zuul's gearman debug logs, the first ERROR entry is about an unknown job at 14:06, right when we got the first gearman timeouts in the zuul daemon log | 15:55 |
| Alex_Gaynor | mordred: I'm hoping donald will remember, in his role as "Keeper of packaging lore" | 15:55 |
| mordred | dhellmann: yeah - let's do it in this patch - I just said the same thing in a review too | 15:55 |
| *** msabramo has joined #openstack-infra | 15:55 | |
| dhellmann | mordred: ok | 15:55 |
| *** hashar has joined #openstack-infra | 15:56 | |
| mordred | dhellmann: but overall, the patch looks awesome | 15:56 |
| *** zz_gondoi is now known as gondoi | 15:59 | |
| *** alexpilotti has joined #openstack-infra | 15:59 | |
| *** IvanBerezovskiy has left #openstack-infra | 15:59 | |
| *** Ryan_Lane has joined #openstack-infra | 16:00 | |
| *** jlibosva has quit IRC | 16:02 | |
| *** gondoi has quit IRC | 16:04 | |
| openstackgerrit | Monty Taylor proposed a change to stackforge/gertty: Add alembic to requirements https://review.openstack.org/95814 | 16:05 |
| openstackgerrit | A change was merged to stackforge/gertty: Add alembic to requirements https://review.openstack.org/95814 | 16:05 |
| ildikov | hi everyone | 16:05 |
| mordred | haha | 16:05 |
| ildikov | I have a newbie question realted to gate jobs | 16:05 |
| ildikov | s/realted/related/ | 16:06 |
| mordred | that was the quickest patch land I think I've ever seen | 16:06 |
| openstackgerrit | A change was merged to stackforge/gertty: Don't show closed changes in the open list https://review.openstack.org/94085 | 16:06 |
| openstackgerrit | A change was merged to stackforge/gertty: Change review toggle keybinding to 'v'. https://review.openstack.org/94086 | 16:06 |
| jeblair | mordred: i'm waiting for a grep to finish :) | 16:06 |
| openstackgerrit | A change was merged to stackforge/gertty: Add 'killthread' https://review.openstack.org/94087 | 16:06 |
| jeblair | i need to land the rest of those changes; gertty doesn't really work very well until the end of that series | 16:07 |
| ildikov | I have a bug, about having false success on the docs gate job of Ceilometer: https://bugs.launchpad.net/ceilometer/+bug/1317648 | 16:07 |
| uvirtbot | Launchpad bug 1317648 in ceilometer "The documentation gate job wrongly reports success" [Medium,New] | 16:07 |
| anteaya | mordred: 27 seconds, impressive | 16:08 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Perform http calls outside of the db session https://review.openstack.org/94097 | 16:08 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add jump to change https://review.openstack.org/94088 | 16:08 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Fix closing stacked dialogs https://review.openstack.org/94089 | 16:08 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Ensure single-threaded db access https://review.openstack.org/94090 | 16:08 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add local cherry-pick button https://review.openstack.org/94091 | 16:08 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add a project list header https://review.openstack.org/94092 | 16:08 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Change config file to YAML https://review.openstack.org/94093 | 16:08 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add custom palettes and commentlinks https://review.openstack.org/94094 | 16:08 |
| openstackgerrit | A change was merged to stackforge/gertty: Add jump to change https://review.openstack.org/94088 | 16:08 |
| *** dkliban is now known as dkliban_brb | 16:08 | |
| openstackgerrit | A change was merged to stackforge/gertty: Fix closing stacked dialogs https://review.openstack.org/94089 | 16:09 |
| openstackgerrit | A change was merged to stackforge/gertty: Ensure single-threaded db access https://review.openstack.org/94090 | 16:09 |
| *** asselin has joined #openstack-infra | 16:09 | |
| *** doude has quit IRC | 16:09 | |
| mordred | jeblair: I look forward to our new gertty patchseries overlords | 16:09 |
| openstackgerrit | A change was merged to stackforge/gertty: Add local cherry-pick button https://review.openstack.org/94091 | 16:09 |
| anteaya | ildikov: I see the bug, what is the question? | 16:09 |
| ildikov | I found scripts and gate job templates in the infra config repo, but I'm not 100% sure, I found the place, where some change could be applied to solve this issue and I'm also not familiar how this could be tested locally | 16:09 |
| anteaya | ildikov: link to what you found? | 16:10 |
| openstackgerrit | A change was merged to stackforge/gertty: Add a project list header https://review.openstack.org/94092 | 16:10 |
| openstackgerrit | A change was merged to stackforge/gertty: Change config file to YAML https://review.openstack.org/94093 | 16:10 |
| *** doude has joined #openstack-infra | 16:10 | |
| hashar | gertty: \O/ | 16:10 |
| ildikov | anteaya: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/slave_scripts/run-docs.sh | 16:10 |
| ildikov | anteaya: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/python-jobs.yaml#L148 | 16:10 |
| openstackgerrit | A change was merged to stackforge/gertty: Add custom palettes and commentlinks https://review.openstack.org/94094 | 16:10 |
| openstackgerrit | A change was merged to stackforge/gertty: Perform http calls outside of the db session https://review.openstack.org/94097 | 16:10 |
| ildikov | anteaya: the run-docs script seems to run the sphinx build command and the other one is the template for the docs gate job | 16:11 |
| jeblair | mordred: okay, gertty HEAD is now a good place; though the series at 95769 is also good if you want to try out the hyperlink stuff i've been working on | 16:11 |
| *** marcoemorais has joined #openstack-infra | 16:11 | |
| *** Ryan_Lane has quit IRC | 16:11 | |
| anteaya | ildikov: so far so good | 16:12 |
| ildikov | anteaya: I cannot really see the mapping between these too and also it's not 100% clear to me, that where should I change prolly the script to check the logs of the sphinx build | 16:12 |
| anteaya | ildikov: this is the template for many jobs to run the docs job | 16:12 |
| anteaya | the question is why is it breaking for ceilometer doc jobs | 16:13 |
| anteaya | do you have a patch url that has a broken docs job? | 16:13 |
| ildikov | anteaya: it does not fail in case of errors or at least for one specific error for sure | 16:13 |
| ildikov | anteaya: one sec, I will try to find the last occurence | 16:13 |
| *** nati_ueno has joined #openstack-infra | 16:14 | |
| *** talluri has joined #openstack-infra | 16:14 | |
| anteaya | ildikov: for reference: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n632 | 16:15 |
| ildikov | anteaya: https://review.openstack.org/#/c/92365/6/doc/source/measurements.rst | 16:15 |
| *** zz_gondoi has joined #openstack-infra | 16:15 | |
| *** zz_gondoi is now known as gondoi | 16:16 | |
| anteaya | ildikov: note that python-jobs are defined here: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n238 | 16:16 |
| *** markmc has quit IRC | 16:16 | |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Handle (ignore) no-diff renames https://review.openstack.org/95769 | 16:16 |
| ildikov | anteaya: the patch is corrected now, but for instance in patch set 6 the table for Cinder was not correctly formed | 16:16 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add patchset selection in diff https://review.openstack.org/95768 | 16:16 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Correct a problem with tables at very small widths https://review.openstack.org/95767 | 16:16 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Add hyperlinks https://review.openstack.org/95766 | 16:16 |
| morganfainberg | i need to poke at gertty more. | 16:16 |
| ildikov | anteaya: and in this case when I ran sphinx build locally, it throws an Error message with malformed table text | 16:16 |
| ildikov | anteaya: but the docs gate job reports success and the result is that the affected table is missing from the generated doc | 16:17 |
| anteaya | ildikov: here is the docs build for patchset 6: http://docs-draft.openstack.org/65/92365/6/check/gate-ceilometer-docs/cc989fb/doc/build/html/ | 16:17 |
| anteaya | can you show me the broken parts? | 16:17 |
| *** amotoki has joined #openstack-infra | 16:17 | |
| ildikov | anteaya: http://docs-draft.openstack.org/65/92365/6/check/gate-ceilometer-docs/cc989fb/doc/build/html/measurements.html#volume-cinder | 16:18 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Create and upload wheels https://review.openstack.org/56760 | 16:18 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Make it possible to run zuul layout test locally https://review.openstack.org/95783 | 16:18 |
| clarkb | http://logs.openstack.org/65/92365/10/gate/gate-ceilometer-docs/0e286b8/console.html.gz is the build log. ceilometer docs isnt failing on warnings. not sure if this would help | 16:18 |
| dhellmann | mordred: ^^ | 16:18 |
| clarkb | but there are no errors in the build | 16:18 |
| ildikov | anteaya: this is the result of the comment I added in the previous link of that patch | 16:18 |
| *** ihrachyshka has joined #openstack-infra | 16:19 | |
| anteaya | ildikov: I see tables | 16:19 |
| *** talluri has quit IRC | 16:19 | |
| *** jooools has quit IRC | 16:19 | |
| anteaya | ildikov: what is missing? | 16:19 |
| ildikov | anteaya: the Cinder table is missing | 16:19 |
| anteaya | yeah I confirm the cinder table it missing | 16:20 |
| ildikov | anteaya: in the correct docs it looks like this: http://docs-draft.openstack.org/65/92365/10/gate/gate-ceilometer-docs/0e286b8/doc/build/html/measurements.html#volume-cinder | 16:20 |
| mordred | jeblair: raise MultipleInvalid([e]) | 16:20 |
| ildikov | anteaya: ah, ok | 16:20 |
| mordred | voluptuous.MultipleInvalid: expected a dictionary | 16:20 |
| clarkb | so you want python setup.py build_sphinx to catch that | 16:20 |
| *** habib has quit IRC | 16:21 | |
| mordred | jeblair: I moved my config to yaml as per instructions - and started gertty and got that - known issue? or should I debug? | 16:21 |
| jeblair | mordred: unknown issue | 16:21 |
| *** thedodd has quit IRC | 16:21 | |
| *** thedodd has joined #openstack-infra | 16:21 | |
| mordred | jeblair: I see it. my bad | 16:21 |
| anteaya | ildikov: as clarkb pointed out the build log for the docs job throws no warnings | 16:22 |
| anteaya | ildikov: according to the log the build succeeds | 16:22 |
| ildikov | clarkb: and if I make it to catch this issue than it will mark the doc job as failed, right? | 16:22 |
| clarkb | ildikov: anteaya if you can get tox -evenv -- python setup.py build_sphinx to catch that then it will be gated on | 16:22 |
| mordred | jeblair: I missed the top level "servers" | 16:22 |
| *** e0ne has quit IRC | 16:22 | |
| jeblair | oops | 16:22 |
| clarkb | yes | 16:22 |
| ildikov | clarkb: anteaya: I use sphinx build to check my doc related patches, so it throws an error, I never tried to run it in any other way | 16:23 |
| clarkb | anteaya: it has 10 warnings no errors | 16:23 |
| *** e0ne has joined #openstack-infra | 16:23 | |
| clarkb | ildikov it may be related to the version of sphinx | 16:23 |
| ildikov | clarkb: hmm, ok, I will check that also | 16:24 |
| anteaya | clarkb: looking at the testenv for docs: http://git.openstack.org/cgit/openstack/ceilometer/tree/tox.ini | 16:25 |
| clarkb | also it should gate on warnings if using pbr properly | 16:25 |
| *** ihrachyshka has quit IRC | 16:25 | |
| anteaya | clarkb: would a change need to happen in the tox.ini for ceilometer? | 16:25 |
| clarkb | mordred ^ | 16:25 |
| *** ihrachyshka has joined #openstack-infra | 16:25 | |
| clarkb | anteaya no we dont use a docs venv | 16:25 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add zuul template for rtfd jobs https://review.openstack.org/95825 | 16:25 |
| *** jgallard has quit IRC | 16:26 | |
| clarkb | we use tox -evenv -- python setup.py build_sphinx | 16:26 |
| anteaya | clarkb: is the ceilometer docs venv in the tox.ini file mis-leading? | 16:26 |
| *** derekh_ has quit IRC | 16:26 | |
| hashar | iirc the docs/doc testenv in repository is just a convenience for devs | 16:27 |
| *** e0ne has quit IRC | 16:27 | |
| hashar | so they can easily generate doc by tox -edocs | 16:27 |
| clarkb | potentially since the venv virtualenv does some extra things | 16:27 |
| *** Longgeek has quit IRC | 16:27 | |
| clarkb | it doesnt do that typically but ceilometer appears to have a snowflake | 16:28 |
| zaro | morning | 16:28 |
| mordred | oh! | 16:28 |
| anteaya | morning zaro | 16:29 |
| *** ArxCruz has quit IRC | 16:29 | |
| mordred | clarkb: well, wow | 16:29 |
| ildikov | hmm, does the venv have anything to do with the false success? | 16:29 |
| mordred | that means that ceilo docs jobs in our stuff are going to be trying to start mongo | 16:29 |
| ildikov | sorry for the silly question, I'm not the expert of this part | 16:30 |
| clarkb | and who knows what else | 16:30 |
| fungi | jeblair: i think this could be where things started to go wrong, but i don't see any smoking gun in the gearman-server.log (this was a fraction of a second before the first ERROR in that log): http://paste.openstack.org/show/81722/ (trying now to correlate with the 5 other disconnects we saw after that one during the time of troubles) | 16:30 |
| ildikov | hmm, anyhow that sounds bad :S | 16:30 |
| mordred | clarkb: I'm not 100% sure what the right choice is here - but I think we might need to dive in | 16:30 |
| mordred | I'm not sure I believe it should be tox's job to start mongodb - but I will admit I have spent all of 30 seconds thinking about this | 16:31 |
| anteaya | mordred: where do you see tox starts mongodb? | 16:31 |
| anteaya | I am not seeing that | 16:31 |
| jeblair | fungi: thx | 16:31 |
| mordred | anteaya: in setup-test-env.sh | 16:32 |
| clarkb | anteaya: the setup env thing | 16:32 |
| fungi | jeblair: these were the 6 local gearman server disconnects we logged... http://paste.openstack.org/show/81726/ | 16:32 |
| anteaya | ah thanks | 16:32 |
| *** jcoufal has quit IRC | 16:32 | |
| ildikov | anteaya: testenv:venv would be my vote | 16:32 |
| clarkb | fungi: did sys or kern log log anything ? | 16:33 |
| fungi | clarkb: crickets | 16:33 |
| clarkb | I wonder if the host had a sad | 16:33 |
| clarkb | :( | 16:33 |
| clarkb | ildikov does tox -e docs do the right thing? | 16:34 |
| clarkb | if not then the setup test env thing is probably not to blame | 16:34 |
| *** gokrokve has quit IRC | 16:34 | |
| ildikov | clarkb: TBH, I've never tried it locally, I use the plain sphinx build command | 16:34 |
| *** gokrokve has joined #openstack-infra | 16:35 | |
| *** mwagner_lap has joined #openstack-infra | 16:35 | |
| ildikov | clarkb: should the doc section and venv in tox.ini look like for instance the one in nova's tox.ini: https://github.com/openstack/nova/blob/master/tox.ini#L45 | 16:35 |
| *** wenlock_ has joined #openstack-infra | 16:35 | |
| clarkb | just the venv one matters for this | 16:36 |
| clarkb | the docs section is unused by the gate | 16:36 |
| *** vhoward has left #openstack-infra | 16:36 | |
| *** terryw is now known as otherwiseguy | 16:36 | |
| ildikov | clarkb: yeap, you're right, I messed it up a bit | 16:36 |
| jeblair | fungi: ~corvus/logs has just the 14:xx hour of log entries | 16:38 |
| *** BadCub has left #openstack-infra | 16:38 | |
| ildikov | clarkb: do the deps section needed there for instance? I mean in the 'venv' as it is added already in testenv at the beginning | 16:38 |
| clarkb | I think it may need to be there if you override somethibg | 16:39 |
| fungi | jeblair: thanks! much faster. i should have done something similar | 16:39 |
| clarkb | but otherwise it should be fine without it | 16:39 |
| *** gokrokve has quit IRC | 16:39 | |
| *** wenlock_ has quit IRC | 16:40 | |
| jeblair | fungi: it's interesting that the packets that zuul times out on are eventually received by the gearman server | 16:41 |
| fungi | yeah | 16:41 |
| jeblair | http://paste.openstack.org/show/81727/ | 16:41 |
| fungi | clearly communication isn't kaput, just taking too long | 16:42 |
| *** zhiyan_ is now known as zhiyan | 16:42 | |
| fungi | focusing on the conversation leading up to each disconnect, i don't see any commonalities whatsoever... http://paste.openstack.org/show/81728/ | 16:43 |
| *** msabramo has quit IRC | 16:43 | |
| fungi | all in various states | 16:43 |
| *** lakshmiS has joined #openstack-infra | 16:44 | |
| clarkb | arg being hauled to breakfast with family before returning to northern lands | 16:44 |
| *** msabramo has joined #openstack-infra | 16:44 | |
| clarkb | back before meeting | 16:44 |
| fungi | have fun, clarkb | 16:44 |
| ildikov | clarkb: hmm, I'm not 100% sure that what is overwritten where, so I will leave it as is for the first round | 16:44 |
| *** ihrachyshka has quit IRC | 16:46 | |
| ildikov | clarkb: anteaya: thanks for the help and the pointers, I will try to play a bit with tox then and see what happens, I guess there should be the solution somewhere | 16:46 |
| * anteaya nods | 16:46 | |
| fungi | this is leading me to conclude there was something environmental affecting zuul's local performance around that time, slowing it down just enough that some local connections exceeded the 30-second timeout by ~10% | 16:46 |
| *** pdmars has quit IRC | 16:47 | |
| anteaya | I hope you find a solution | 16:47 |
| *** ihrachyshka has joined #openstack-infra | 16:48 | |
| fungi | https://status.rackspace.com/ looks sort of bad, but nothing obvious there which would impact dfw | 16:48 |
| *** olaph has joined #openstack-infra | 16:48 | |
| *** w_ has joined #openstack-infra | 16:48 | |
| ildikov | anteaya: I will come back with some new questions if not, but hopefully it will not be needed, I'm a bit confused now with that venv section, but anyway I'm ready to play with it a bit :) | 16:48 |
| ildikov | anteaya: so thanks again :) | 16:49 |
| *** w_ has quit IRC | 16:49 | |
| *** yfried has joined #openstack-infra | 16:49 | |
| fungi | oh! https://status.rackspace.com/index/viewincidents?start=1401163200 "10:59 AM EDT Our Engineers have identified an issue with one of the storage devices in the DFW1 data center which is causing some sites to experience slow response or timeouts." | 16:49 |
| fungi | maybe? about the right timeframe | 16:49 |
| *** yfried has quit IRC | 16:50 | |
| *** yfried has joined #openstack-infra | 16:50 | |
| anteaya | ildikov: questions are always welcome | 16:51 |
| *** yfried has quit IRC | 16:51 | |
| *** zhiyan is now known as zhiyan_ | 16:51 | |
| *** yfried has joined #openstack-infra | 16:51 | |
| *** david-lyle has quit IRC | 16:51 | |
| *** david-lyle has joined #openstack-infra | 16:52 | |
| ildikov | anteaya: cool, thanks | 16:52 |
| *** david-lyle has quit IRC | 16:52 | |
| jeblair | fungi: i'm seeing geard logging mostly idle during those timeframes too; so either it's affected by the 'paused host', or it's spending 35 seconds doing something it's not logging | 16:52 |
| zaro | clarkb: would you be able to comment? https://review.openstack.org/#/c/92773/ | 16:53 |
| jeblair | fungi: (i wonder what a live migration actually looks like from the host pov) | 16:53 |
| fungi | jeblair: omg, it's full of clouds | 16:53 |
| *** nati_ueno has quit IRC | 16:53 | |
| *** dkliban_brb is now known as dkliban | 16:54 | |
| *** SumitNaiksatam has quit IRC | 16:54 | |
| jeblair | fungi: http://paste.openstack.org/show/81729/ that's a call/response pair to a jenkins master separated by 35 seconds | 16:54 |
| clarkb | zaro: yes use the macro in the defaults | 16:54 |
| *** harlowja_away is now known as harlowja_ | 16:55 | |
| *** yamahata has quit IRC | 16:55 | |
| mordred | dhellmann: your patches make me happy | 16:55 |
| jeblair | fungi: do you feel that /var/log/jenkins/jenkins.log on jenkins07 should be greater than zero bytes? | 16:55 |
| dhellmann | mordred: :-) | 16:56 |
| mordred | dhellmann: they also raise the question of what we shoudl be doing with rtfd again - especially since there are two repos using rtfd that may be able to become "official" | 16:56 |
| mordred | which means it might be time for us to actually take a stance | 16:56 |
| mordred | s/able/about/ | 16:56 |
| *** nati_ueno has joined #openstack-infra | 16:56 | |
| dhellmann | mordred: yeah, some of those oslo libs have docs published there as legacy urls; I don't think it hurts to keep them. | 16:57 |
| *** ildikov has quit IRC | 16:57 | |
| Ajaeger | infra team, could you review https://review.openstack.org/#/c/95414/ and https://review.openstack.org/95345, please? The first one is important for manual changes (and thus if you only review one, take that one;), and the second the base for some further work I do. | 16:57 |
| *** hogepodge has joined #openstack-infra | 16:57 | |
| mordred | dhellmann: I kinda think we should do rtfd across the board - AND publish to docs.o.o - as long as we're consistently able to keep both up to date without much effort - biggest effort is getting the darned rtfd id for each project, which makes project creation/registration much harder | 16:58 |
| *** praneshp has joined #openstack-infra | 16:58 | |
| *** dizquierdo has quit IRC | 16:58 | |
| mordred | Ajaeger: looking | 16:58 |
| jeblair | mordred: it uses names now | 16:58 |
| Alex_Gaynor | mordred: you don't need IDs, anymore, the project name is enough | 16:58 |
| fungi | jeblair: ooh, good catch | 16:59 |
| *** alugovoi has quit IRC | 16:59 | |
| dhellmann | jeblair: does it still require a project to be registered manually? I guess that's no different than pypi. | 16:59 |
| Ajaeger | thanks, mordred | 16:59 |
| mordred | oh! neat | 16:59 |
| Alex_Gaynor | dhellmann: yeah, but, like, you can write some software | 16:59 |
| mordred | dhellmann: I've been meaning to add pypi project registration to manage-projects too | 16:59 |
| Alex_Gaynor | dhellmann: hell, it'd probably even be easy to add an API end point to rtd, I'm sure they'd take it | 16:59 |
| dhellmann | Alex_Gaynor: true | 16:59 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Add dib-utils project https://review.openstack.org/90281 | 16:59 |
| jeblair | mordred: i'm in favor of not using rtfd because it's an unecessary extra thing to deal with | 17:00 |
| *** _nadya_ has joined #openstack-infra | 17:00 | |
| mordred | jeblair: I'm also in favor of doing that | 17:00 |
| *** hashar is now known as hasharOut | 17:00 | |
| fungi | jeblair: /var/log/jenkins/jenkins.log on jenkins07 is owned by a nonexistent user/group | 17:01 |
| *** bhuvan has joined #openstack-infra | 17:01 | |
| fungi | jeblair: which would explain why the daemon can't write to it | 17:01 |
| mordred | jeblair: I think the thing I'm least in favor of is both dealing with it and also not just having it be done everywhere | 17:01 |
| jeblair | fungi: i'm going to see if any of the other failures had a similar interaction with a different jenkins | 17:02 |
| fungi | jeblair: yeah, i think the zero-byte log on jenkins07 is merely an unfortunate coincidence. i'll fix the ownership on it to something sane and consistent with the other masters | 17:03 |
| fungi | looks like it's probably been broken since january 26 (was that when we built it?) | 17:04 |
| fungi | looks like it | 17:04 |
| jeblair | fungi: istr some package/puppet conflicts around users; maybe we didn't do some needed cleanup on that host | 17:04 |
| fungi | last modified time on /etc/hosts is from the same week anyway | 17:04 |
| fungi | seems quite likely | 17:05 |
| fungi | it's owned by jenkins:jenkins now, but will probably need a restart before it writes to it again | 17:05 |
| *** ok_delta has quit IRC | 17:05 | |
| mordred | jeblair: I think my argument in favor of rtfd is that, like pypi, it's a thing that "python projects use" - and I think that where we can consistently engage with python ecosystem might help mitigate the perception that we're over off in the corner. but I don't feel strongly enough about it to die on a hill or anything | 17:06 |
| fungi | doing a sweep of that host for anything else under the previous user/group and will clean that up too | 17:07 |
| fungi | looks like it was only logs | 17:07 |
| anteaya | mordred: don't die on a hill for that | 17:07 |
| Alex_Gaynor | mordred: Do you feel strongly enough to die in a place that isn't a hill? | 17:07 |
| fungi | so presumably jenkins got started under the wrong uid/gid and the logs it created were left behind with incorrect ownership | 17:08 |
| *** dims has quit IRC | 17:08 | |
| *** sarob has joined #openstack-infra | 17:09 | |
| *** _nadya_ has quit IRC | 17:09 | |
| mordred | Alex_Gaynor: I feel strongly that I should only die on hills | 17:09 |
| *** isviridov is now known as isviridov|away | 17:09 | |
| *** _nadya_ has joined #openstack-infra | 17:09 | |
| *** yfried has quit IRC | 17:10 | |
| *** sarob_ has joined #openstack-infra | 17:10 | |
| fungi | (mordred of the hill people) | 17:11 |
| zaro | clarkb: using macro in defaults will not work the same way because the default timeout sets timeout value to 30, while the macro sets it to {timeout}. if we use the macro in the default (as you suggest) then no deault timeout value would be set. | 17:11 |
| *** yfried has joined #openstack-infra | 17:11 | |
| anteaya | mordred: stay away from hills | 17:12 |
| clarkb | zaro: you set the timeout with the macro to 30 | 17:12 |
| *** esker has joined #openstack-infra | 17:12 | |
| *** dims has joined #openstack-infra | 17:13 | |
| *** sarob has quit IRC | 17:13 | |
| *** UtahDave has joined #openstack-infra | 17:13 | |
| *** talluri has joined #openstack-infra | 17:15 | |
| zaro | clarkb: so macro timeout is set to static '30' how do jobs overrite this value? I thought | 17:15 |
| zaro | {timeout} was the thing that lets the override happen? | 17:15 |
| * JayF increments "number of times jay has been unsub'd from openstack-dev@" to 4 | 17:16 | |
| *** gokrokve has joined #openstack-infra | 17:16 | |
| fungi | JayF: you need one of those workplace safety posters for your wall... "0 days since last unsolicited unsubscribe from an openstack mailing list" | 17:16 |
| jeblair | fungi: the packet that caused the timeout in the second instance was processed 6 minutes after it was sent | 17:17 |
| fungi | jeblair: okay, that's certainly a little more than 30 seconds | 17:17 |
| fungi | i could not for the life of me make out any system performance problems exhibiting on cacti graphs for zuul during the problem period | 17:18 |
| mordred | jeblair: autoabandon (or the current lackthereof) came up on another channel, and it seems some teams really miss it - which made me wonder about re-thinking it into a thing similar to the channel logging... | 17:18 |
| mordred | jeblair: that is, something with a yaml config somewhere where a ptl/core-team could opt-in if it's the sort of thing that's important for them | 17:19 |
| *** esker has quit IRC | 17:19 | |
| Ajaeger | thanks, fungi! | 17:19 |
| mordred | (or if their solution to would be "write our own bot that does the same thing") | 17:19 |
| morganfainberg | mordred, jeblair, ++ I know i've heard at least the question about autoabandon in 2 projects now. | 17:19 |
| *** gokrokve_ has joined #openstack-infra | 17:19 | |
| fungi | mordred: btw, the current lack of the old behavior would be fixed by https://review.openstack.org/92884 | 17:19 |
| clarkb | zaro use the variable in the defaults | 17:19 |
| jeblair | mordred: i think it's a really bad idea not to be able to predict which of the patches you submit to openstack will be automatically abandoned by the system | 17:19 |
| *** talluri has quit IRC | 17:20 | |
| *** ArxCruz has joined #openstack-infra | 17:20 | |
| clarkb | zaro: the same way you do in a job | 17:20 |
| Ajaeger | morganfainberg: https://review.openstack.org/92884 AFAIK | 17:20 |
| jeblair | morganfainberg: how about instead we figure out what about the things that are supposed to make it unecessary aren't working and try to fix that | 17:20 |
| morganfainberg | jeblair, perhaps an autowip? | 17:20 |
| morganfainberg | jeblair, and yes we should be doing that as well | 17:20 |
| *** andreaf has quit IRC | 17:20 | |
| fungi | mordred: looks like jeblair and clarkb already +2'd my fix for auto-abandon. any objections to restoring it to working order? | 17:21 |
| morganfainberg | Ajaeger, thanks :) | 17:21 |
| jeblair | fungi: that would cause it to start working again? | 17:21 |
| *** gokrokve has quit IRC | 17:21 | |
| anteaya | fungi: I have no objections | 17:21 |
| jeblair | i just -2d it | 17:22 |
| fungi | jeblair: yes, it's been failing because its credentials were missing | 17:22 |
| *** ArxCruz has quit IRC | 17:22 | |
| fungi | fair enough | 17:22 |
| *** _nadya_ has quit IRC | 17:22 | |
| mordred | fungi: I think we should have a more comprehensive answer to what should be happening- and I actually appreciate it not happening on infra things now | 17:22 |
| jeblair | it was our intent to stop it with the gerrit upgrade; something we confirmed at the summit | 17:22 |
| morganfainberg | wait, core's can abandon any change right directly? | 17:23 |
| fungi | morganfainberg: yes, or wip | 17:23 |
| morganfainberg | fungi, then no need to have a bot do it. if the code really is defunct the core team can cleanup. | 17:23 |
| jeblair | so if we re-enable it, it should not be in the mode of correcting a puppet error, but as a more deliberate thing | 17:23 |
| jeblair | morganfainberg: ++ | 17:23 |
| mordred | jeblair: ++ | 17:23 |
| fungi | morganfainberg: core wip was around before the upgrade, but core abandon/unabandon is new | 17:23 |
| mordred | morganfainberg: perhaps, instead of thinking about it as a solution to stale reviews | 17:23 |
| *** amotoki has quit IRC | 17:23 | |
| jeblair | fungi: i'm not even sure core wip was universal | 17:23 |
| jeblair | before | 17:24 |
| openstackgerrit | A change was merged to openstack-infra/config: Create common translation functions https://review.openstack.org/95345 | 17:24 |
| mordred | morganfainberg: we should have people get into the habit of WIP-ing patches as they review them if it's something that really does need a new version | 17:24 |
| mordred | like, be more aggressive with using that feature up front | 17:24 |
| morganfainberg | mordred, ++++++++++ | 17:24 |
| fungi | jeblair: well, it wasn't necessarily universal, but it was possible (though i think i/someone had a large patch to turn it on pretty much everywhere earlier this year) | 17:24 |
| anteaya | like if the infra-manual initial commit lands, I can create a patch for that suggesting that for core reviewers | 17:25 |
| zaro | clarkb: that doesn't make sense to me. why would you set a variable in defaults? shoudn't default values should be static? | 17:25 |
| mordred | so that rather than the problem being "how do we deal with stale patches" it turns in to "how do we communicate more effectively to people that we expect them to come back with more work" | 17:25 |
| *** changbl has joined #openstack-infra | 17:25 | |
| fungi | jeblair: anyway, i'll abandon 92884 in favor of a patch to properly turn off auto-abandon in that case | 17:25 |
| mordred | that way, I can -1 something if I dont' like it but want more feedback from other people | 17:25 |
| jeblair | so maybe we should write a message to the list describing how core-WIP, core-abandon and the dashboards can be used effectively | 17:25 |
| *** ArxCruz has joined #openstack-infra | 17:26 | |
| mordred | but -1 + -1 WIP something if I don't like it, don't want to blocking--2 it - but know it needs to be fixed without question | 17:26 |
| clarkb | zaro it is static you set it there | 17:26 |
| jeblair | mordred: that's a huge timesaver for other cores | 17:26 |
| clarkb | zaro the point is there are other options we want to be the same everywhere and the macro does that for us | 17:26 |
| mordred | jeblair: maybe I'll write a message to the list about that | 17:26 |
| *** praneshp has quit IRC | 17:26 | |
| clarkb | the timeout variable is a variable but the other options arent | 17:27 |
| morganfainberg | mordred, i like that messaging. though, the big scary red X for WIP is historically a hard block, we should make sure to be clear when people see that more | 17:27 |
| mordred | morganfainberg: the big scary red X for WIP is my least favorite thing about eh current impl | 17:27 |
| morganfainberg | mordred, wonder if we could CSS that to something more distinct | 17:28 |
| anteaya | do we have any control over the colour of the wip X? can it be yellow? | 17:28 |
| morganfainberg | anteaya, ++ my thought exactly | 17:28 |
| *** hasharOut has quit IRC | 17:28 | |
| zaro | clarkb: it seems like what i already have is what you are describing. let me do etherpad of your suggestion. | 17:28 |
| sdague | morganfainberg: it's unfortunately not an easy css fix iirc | 17:30 |
| morganfainberg | sdague, yeah i'm finding that out w/ inspection of the elements | 17:31 |
| *** zehicle_at_dell has joined #openstack-infra | 17:31 | |
| *** praneshp has joined #openstack-infra | 17:31 | |
| morganfainberg | sdague, it's a raw data load it looks like. ugh. | 17:31 |
| *** nati_ueno has quit IRC | 17:31 | |
| sdague | yeh - http://paste.openstack.org/show/81737/ | 17:31 |
| morganfainberg | yep | 17:32 |
| morganfainberg | ick. | 17:32 |
| sdague | we'd have to see if zaro wanted to hack us a class in there instead in the gerrit src | 17:32 |
| mordred | what if we made a -2 workflow status ... but didn't give anyone access to set it | 17:32 |
| mordred | just so that gerrit wouldn't render a -1 as an X | 17:33 |
| sdague | it wouldn't block the change then without prolog hacking | 17:33 |
| jeblair | mordred: you'd want to make sure you can make -1 block | 17:33 |
| morganfainberg | mordred, yeah i wouldn't want it to be a soft looking -1 | 17:33 |
| morganfainberg | i like the X, just... the color | 17:33 |
| *** alugovoi has joined #openstack-infra | 17:33 | |
| mordred | jeblair: yeah. it would need testing for sure | 17:33 |
| sdague | because the default prolog rules are = 'at least one of the largest value, and non of the smallest value) | 17:33 |
| mordred | or we can just grow used to it and wait for vinz to solve the world | 17:34 |
| morganfainberg | might be able to "fix" it w/ a dirty jquery hack | 17:34 |
| jeblair | there _is_ a workinprogress plugin, but it's missing some minor functionality on the 2.8 change screen; apparently works with 2.9. | 17:34 |
| jeblair | so before we start hacking gerrit to make the currenty hacky thing better, we might want to focus on that instead. :) | 17:34 |
| sdague | so realistically it would just be nice if all the check / x markers were actually themable instead of inline data. | 17:34 |
| morganfainberg | sdague, ++ | 17:35 |
| mordred | jeblair: ++ | 17:35 |
| sdague | because inline base64 png files is... ug | 17:35 |
| mordred | sdague: that would probably be a nice upstream patch | 17:35 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Disable Gerrit auto-abandon https://review.openstack.org/95836 | 17:35 |
| fungi | jeblair: ^ | 17:36 |
| anteaya | I wouldn't suggest a prolog hack | 17:36 |
| anteaya | when might we upgrade to 2.9? | 17:36 |
| zaro | clarkb: https://etherpad.openstack.org/p/change-84637 | 17:36 |
| morganfainberg | jeblair, would we want to change "WIP" mechanism again in any reasonable timeframe? I'd be concerned about changing workflow too much. | 17:37 |
| morganfainberg | not sure how the WIP plugin works though (tbh) | 17:37 |
| mordred | jeblair: oh wow. that actually gets rid of the final use of launchpadsync? | 17:37 |
| fungi | mordred: yep | 17:37 |
| fungi | one more reason to be in favor ;) | 17:38 |
| mordred | clarkb: the 1.1 regions in hp seem to be unhappy again | 17:38 |
| mordred | clarkb: if I'm reading the graph right | 17:38 |
| jeblair | morganfainberg: the wip plugin is based on our old wip patch we were running on 2.4, so it should be similar to what we were using before | 17:39 |
| *** alexpilotti has quit IRC | 17:39 | |
| fungi | mordred: though the lp sync credentials are still used for update bug/bp | 17:39 |
| fungi | i believe | 17:39 |
| fungi | but the cron bits are no more | 17:40 |
| *** Ryan_Lane has joined #openstack-infra | 17:40 | |
| jeblair | fungi: the jenkins logs aren't quite detailed enough for me to know when it responded to the NO_JOB packet in the second instance | 17:40 |
| zaro | sdague, morganfainberg : i think the scary red X is an image. which would mean it would be change to image not just color. | 17:40 |
| morganfainberg | jeblair, hm. i think i like this method better (the workflow), but that is purely personal bias. i wouldn't argue not going back to the old system if it made life easier | 17:40 |
| morganfainberg | zaro, if it was a themeable element, it would solve the issue. and it's a raw b64 png dataload, not just CSS or similar. | 17:41 |
| *** alexpilotti has joined #openstack-infra | 17:41 | |
| jeblair | morganfainberg: [switching to ux feedback collection mode] why do you like it better? | 17:41 |
| morganfainberg | jeblair, strictly because it uses the same mechanism for marking WIP as reviewing. the WIP button and the odd "status" never sat well with me. | 17:42 |
| *** nati_ueno has joined #openstack-infra | 17:42 | |
| morganfainberg | jeblair, also, part of the approved column at a glance centralizes the information on WIP/Non WIP, CRV | 17:42 |
| morganfainberg | jeblair, like i said, pure personal preference, but not strong enough to jump on a "don't go back" train. | 17:43 |
| sdague | zaro: right, but like morganfainberg said, if we had a handle to theme it, it would be nice | 17:43 |
| *** zhiyan_ is now known as zhiyan | 17:43 | |
| sdague | to make it something which doesn't look like the a -2 | 17:43 |
| sdague | I've had multiple people ask me how to remove the X from their code, because they know it's bad | 17:43 |
| fungi | orange "under construction" sign | 17:43 |
| sdague | when the only X was the WIP they set themselves | 17:43 |
| sdague | fungi: yeh, that would be much better | 17:44 |
| morganfainberg | fungi, can we have that be animated too? | 17:44 |
| morganfainberg | :P | 17:44 |
| * morganfainberg goes back to web 1.0 days and blink tags. | 17:44 | |
| fungi | and swap the approved green checkmark with a thumbs-up | 17:44 |
| sdague | fungi: yeh, it's kind of sad they didn't do it in unicode :) | 17:45 |
| fungi | beer mug and snowman | 17:45 |
| morganfainberg | sdague, -2 can become a table-flip ascii guy then! | 17:45 |
| *** pblaho has joined #openstack-infra | 17:45 | |
| anteaya | no blinking | 17:46 |
| openstackgerrit | Arun Kant proposed a change to openstack/requirements: Adding ldappool module dependency as needed by keystone bug #1320997. https://review.openstack.org/95842 | 17:46 |
| uvirtbot | Launchpad bug 1320997 in keystone "Common Ldap handler connection pooling" [Medium,In progress] https://launchpad.net/bugs/1320997 | 17:46 |
| sdague | the thing that's also weird is the meta data is completely lost on vote columns that are +2 or -2 | 17:46 |
| anteaya | I like snowmen | 17:46 |
| morganfainberg | <blink>Whats wrong with blinking</blink> :P | 17:46 |
| sdague | they don't have the negative or positive class on them | 17:46 |
| morganfainberg | sdague, yeah it's a little odd | 17:47 |
| morganfainberg | sdague, that was my hope so we could just CSS it up. | 17:47 |
| *** Lingo has joined #openstack-infra | 17:47 | |
| *** vhoward has joined #openstack-infra | 17:49 | |
| jeblair | morganfainberg: cool, thanks for the feedback on status vs approval category | 17:50 |
| morganfainberg | jeblair, happy to help. | 17:50 |
| morganfainberg | fungi, nice topic on disabling autoabandon bot. just noticed | 17:52 |
| *** zhiyan is now known as zhiyan_ | 17:52 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/infra-specs: Added specification for storyboard subscription (Story:96) https://review.openstack.org/95307 | 17:52 |
| *** doude has quit IRC | 17:54 | |
| *** pblaho has quit IRC | 17:54 | |
| *** e0ne has joined #openstack-infra | 17:55 | |
| mrodden | mordred: https://review.openstack.org/#/c/93986/ fixes the bash8 ignore thing | 17:56 |
| mrodden | hasn't been release yet; i should probably do that since others were asking about fixes for it | 17:57 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added search icon to typeahead fields https://review.openstack.org/94273 | 17:57 |
| *** Guest51544 has quit IRC | 17:57 | |
| mordred | jeblair, morganfainberg, sdague: https://etherpad.openstack.org/p/M5qQPxxPyy | 17:59 |
| mordred | there's a draft email to send out about using WIP more aggressively | 17:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove special handling of high-availability manual https://review.openstack.org/95414 | 18:00 |
| morganfainberg | mordred quick glance looks good, need to hop over to keystone irc meething though | 18:00 |
| openstackgerrit | Arun Kant proposed a change to openstack/requirements: Adding ldappool module dependency as needed by keystone bug #1320997. https://review.openstack.org/95842 | 18:00 |
| uvirtbot | Launchpad bug 1320997 in keystone "Common Ldap handler connection pooling" [Medium,In progress] https://launchpad.net/bugs/1320997 | 18:00 |
| anteaya | mordred: you can have turquoise I have switched to purple | 18:01 |
| jeblair | mordred: made some teensy changes; do you want to mention core-abandon in this? | 18:02 |
| *** esker has joined #openstack-infra | 18:03 | |
| *** esker has quit IRC | 18:03 | |
| fungi | mordred: on a related note, you could mention that core reviewers can now abandon/restore patches for their projects | 18:03 |
| *** alexpilotti has quit IRC | 18:03 | |
| fungi | er, what jeblair said :/ | 18:03 |
| *** esker has joined #openstack-infra | 18:03 | |
| mrodden | mriedem: were you still looking for a bash8 release for that python26 fix? | 18:04 |
| mriedem | mrodden: at some point, but not anytime soon | 18:04 |
| mrodden | k | 18:04 |
| mriedem | now that i have other things to do after the long break | 18:04 |
| mrodden | long is a relative term | 18:04 |
| *** james_li has quit IRC | 18:04 | |
| mriedem | after the > 2 day weekend | 18:05 |
| *** david-lyle has joined #openstack-infra | 18:05 | |
| mrodden | 'll probably do a 1.1 | 18:05 |
| mrodden | err | 18:05 |
| mrodden | 0.1.1 | 18:05 |
| mrodden | ignore is nice to have :) | 18:05 |
| *** jamielennox|away is now known as jamielennox | 18:05 | |
| mrodden | its not even implemented in the devstack version which i found really surprising | 18:06 |
| *** sarob_ has quit IRC | 18:07 | |
| Ajaeger | clarkb: I'm working on the translation scripts to add extraction of log level messages and thus stumble about some odd things. I'd like to know whether we really need to run the update - could you review https://review.openstack.org/95687, please? | 18:08 |
| *** _nadya_ has joined #openstack-infra | 18:09 | |
| *** hashar has joined #openstack-infra | 18:09 | |
| sdague | mordred: did you see my analysis of -1ed merged code last week? | 18:10 |
| *** krtaylor_ has joined #openstack-infra | 18:10 | |
| sdague | honestly, I think this is going to just confuse things. And people are already -1ing meaning -1 and WIP today anyway | 18:11 |
| clarkb | Ajaeger lgtm will +2 when I sit with laptop | 18:11 |
| Ajaeger | clarkb: Thanks! Then I can code easier on the log level extraction ;) | 18:11 |
| *** Ryan_Lane has quit IRC | 18:12 | |
| *** krtaylor_ has quit IRC | 18:12 | |
| *** nati_ueno has quit IRC | 18:13 | |
| *** nati_ueno has joined #openstack-infra | 18:14 | |
| *** rfolco has quit IRC | 18:14 | |
| jeblair | fungi: in the third instance, the geard gap was bracketed by two 'receive packet' log entries. that makes it seem unlikely that geard was blocked on sending network traffic | 18:15 |
| *** jp_at_hp has quit IRC | 18:16 | |
| jeblair | fungi: during all three gaps, geard continued to get new workers connecting to it periodically | 18:16 |
| jeblair | fungi: it's starting to look like all network traffic on existing connections was stopped, then resumed | 18:17 |
| fungi | jeblair: this would be consistent with something like a live migration while rackspace tried to deal with block storage backend issues in that region | 18:17 |
| *** rfolco has joined #openstack-infra | 18:18 | |
| *** hashar is now known as hasharOut | 18:18 | |
| fungi | phschwartz: do you happen to know if there was anything going on related to the block storage issues in dfw earlier today which might have caused one of our instances there to get paused for brief periods? | 18:18 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Extract translations for log messages https://review.openstack.org/95377 | 18:19 |
| *** jpich has quit IRC | 18:20 | |
| Ajaeger | dhellmann: now it should look much nicer ^ | 18:20 |
| jeblair | fungi: same pattern for fourth occurrence | 18:21 |
| *** krtaylor has quit IRC | 18:22 | |
| dhellmann | Ajaeger: that is much nicer; one comment inline where I think you're missing a $ | 18:22 |
| Ajaeger | dhellmann: good catch, thanks! | 18:24 |
| Ajaeger | will update later | 18:25 |
| *** reed has joined #openstack-infra | 18:25 | |
| anteaya | hello reed | 18:26 |
| anteaya | are you in austin? | 18:27 |
| clarkb | jeblair: the new gertty HEAD does it have anything new that I don't already have? | 18:27 |
| clarkb | eg any bug fixes made against existing patches | 18:27 |
| jeblair | clarkb: nope, new stuff is in review | 18:27 |
| mtreinish | ArxCruz, rfolco: is there a reason you guys are reporting running the tempest unit tests on powerkvm ci? I'm not sure there is much value in doing that. | 18:28 |
| phschwartz | fungi: I have not heard of anything. Let me look at notifications and see what needs to get looked at. | 18:28 |
| mtreinish | ArxCruz, rfolco: oh and the log links for those jobs are dead... | 18:28 |
| clarkb | jesusaurus: your jenkins changes are happy now? | 18:28 |
| reed | anteaya, on my way, I'll have an answer for you tomorrow | 18:28 |
| mordred | sdague: so - I think we might be talking about different concerns or problem spaces? | 18:29 |
| phschwartz | fungi: oh, and we don't currently do live migrations of active customer instances. | 18:29 |
| sdague | mordred: that's possible | 18:29 |
| mordred | sdague: the thing the WIP and/or auto-abandon is trying to solve is slimming down or focusing the set of things I should be looking at | 18:29 |
| fungi | phschwartz: between 1300 and 1400 utc (which was shortly before they posted an impact notice for dfw) we saw multiple periods of inexplicable time skips for zuul.openstack.org which resulted in timeouts for some of its internal communication, so just curious whether there was any possible correlation | 18:29 |
| mordred | and isn't really about whether things get merged or not | 18:29 |
| jesusaurus | clarkb: yep :) | 18:29 |
| sdague | mordred: sure, but you are doing that by offloading more work to other reviewers | 18:30 |
| fungi | phschwartz: er, actually between 1400 and 1500 | 18:30 |
| phschwartz | fungi: Let me look in that time frame | 18:30 |
| clarkb | zaro: updated your etherpad | 18:30 |
| *** che-arne has quit IRC | 18:30 | |
| sdague | when i think most of them are already signaling you that same info with a -1 today | 18:30 |
| mordred | sdague: what more work? just WIP something when you -1 it then | 18:30 |
| openstackgerrit | A change was merged to openstack-infra/config: Do not run unneeded update_catalog https://review.openstack.org/95687 | 18:30 |
| jeblair | fungi, phschwartz: http://paste.openstack.org/show/81770/ are the exact times where we either saw gaps in expected network traffic, or geard was doing something very unexpected | 18:30 |
| mordred | it's on the same screen | 18:30 |
| mordred | sdague: the problem is - non-core folks also -1 things | 18:31 |
| anteaya | reed: great, thanks, safe journey | 18:31 |
| sdague | mordred: sure | 18:31 |
| mordred | sdague: adn those -1's are different | 18:31 |
| mordred | take: https://review.openstack.org/#/c/90234/ | 18:31 |
| mordred | sdague: zaro's original -1 vote is a valid thing for him to express and I'm glad he did. but he's not core, which means that this patch still needs to be reviewed by the cores | 18:32 |
| sdague | mordred: sure, but that was the point on my merge bit. | 18:32 |
| mordred | sdague: so for -1 there to imply WIP is actually wrong | 18:32 |
| anteaya | mtreinish: any dates selected for qa mid-cycle meetup? | 18:32 |
| phschwartz | do any of you have the uuid for the zuul instance handy? | 18:33 |
| jeblair | phschwartz: i'll get it | 18:33 |
| fungi | phschwartz: getting it for you | 18:33 |
| sdague | mordred: ok, so at least in the nova team that would trigger zaro coming back and saying "oh, gotcha, I was wrong +1" | 18:33 |
| clarkb | mordred: yes 1.1 looks really unhappy | 18:33 |
| mordred | sdague: right. I think the nova team might be different than many of the other teams | 18:33 |
| clarkb | mordred: and we are using 100 nodes per router:network now | 18:34 |
| fungi | phschwartz: fbc5f4f3-8f46-41be-88f1-3374b0ed1975 | 18:34 |
| clarkb | mordred: so I think that debunks the theory | 18:34 |
| mordred | clarkb: grumble | 18:34 |
| sdague | it could be, we do the same ish thing on tempest | 18:34 |
| *** shivharis has joined #openstack-infra | 18:34 | |
| jeblair | fungi: i agree. :) | 18:34 |
| phschwartz | fungi: ty | 18:34 |
| clarkb | mordred: so uh ya | 18:34 |
| mordred | sdague: so I think the thing is - it's a tool that's available to people if they want to use it | 18:34 |
| sdague | sure | 18:34 |
| mordred | there are teams actively asking for the auto-abandon bot | 18:34 |
| fungi | phschwartz: thank YOU for weighing in ;) | 18:34 |
| clarkb | mordred: any chance we can get the hpcloud people that care in here so that we can talk to them as a group? | 18:34 |
| mordred | actually: | 18:34 |
| sdague | mordred: yes, and I'm one of them :) | 18:34 |
| mordred | 17:26:59 lifeless | I hate autoabandon when it happens to my branches | 18:34 |
| mordred | 17:27:12 lifeless | I love it when it happens to those of drive-by contributors | 18:34 |
| clarkb | mordred: its a bit frustrating to get all of this third hand | 18:35 |
| mordred | there's a good example ^^ | 18:35 |
| phschwartz | fungi: That is why I am here :) | 18:35 |
| mordred | clarkb: let me see if I can do that | 18:35 |
| sdague | to the point that I might just cron it for projects I feel entitled to | 18:35 |
| mordred | sdague: right. which is why I was suggesting that if people are about to start doing that, we should re-think our current bot - but we'd like to explore not needing it if possible | 18:35 |
| *** rlandy_ has joined #openstack-infra | 18:36 | |
| mordred | sdague: if autoabandon is a thing that's useful, then perhaps the "-1 == WIP" your'e talking about actually _isn't_ working | 18:36 |
| mordred | other than that a -1 is an effective block to merging | 18:36 |
| mordred | but it may not actualy be serving a job as an effective workflow flag | 18:36 |
| sdague | sure, the issue is -1 not meaning - please respin this, is the exception | 18:36 |
| sdague | so the fact that it's more work to do the default case, is annoying, and means I'd probably never do that workflow | 18:37 |
| mordred | hrm | 18:37 |
| sdague | if I'm not sure, I leave a 0 review | 18:37 |
| sdague | or I ask someone | 18:37 |
| clarkb | mordred: it apepars to be load related | 18:37 |
| clarkb | mordred: like if we change the nodepool rate limit it might be happier but with less throughput | 18:37 |
| *** chuckC has quit IRC | 18:38 | |
| clarkb | mordred: and that should be enforced on their end imo | 18:38 |
| mordred | clarkb: as in, related to number of requested instances? | 18:38 |
| clarkb | mordred: ya | 18:38 |
| phschwartz | fungi: hmm, nothing looking out of whack with the hv, will have it watched, but if it happens again, lets open a ticket and get it moved to a different hv so we can see if it is truly an hv issue. | 18:38 |
| jeblair | clarkb: are you sure it's the rate and not the total number of instances? | 18:38 |
| *** rlandy has quit IRC | 18:38 | |
| clarkb | jeblair: no, it may also just be the totals | 18:38 |
| fungi | phschwartz: definitely, and thanks. we see similar issues occasionally, but only recently got additional logging in place to have a better understanding of what could be triggering it | 18:39 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Extract translations for log messages https://review.openstack.org/95377 | 18:39 |
| jeblair | sdague: could an alteration to the dashboard queries you use that ignores changes with a negative review that are older than a certain age help? | 18:39 |
| jeblair | sdague: essentially, i think that a good dashboard system should make auto-abandon unecessary | 18:39 |
| phschwartz | fungi: as it is seen more, I would love to get as much log info as possible so we can troubleshoot it. | 18:40 |
| sdague | jeblair: maybe, we've still got the issue of iterating on dashboards | 18:40 |
| jeblair | sdague: istr you wrote a patch to support that | 18:40 |
| sdague | yep | 18:40 |
| Ajaeger | dhellmann, clarkb: The above patch (95377) is ready for review and merge now. It's not urgent (unless dhellmann is really eager ;) but if there are questions, I'm here for discussion. | 18:40 |
| sdague | jeblair: 7 days no reviews :) | 18:41 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Added search icon to typeahead fields https://review.openstack.org/94273 | 18:41 |
| mordred | clarkb: I have copied you on an email where I have also requested someone from HP show up in channel to talk to us | 18:43 |
| *** zhiyan_ is now known as zhiyan | 18:44 | |
| clarkb | mordred: thank | 18:44 |
| *** markwash has quit IRC | 18:44 | |
| clarkb | mriedem: sdague: so behavior I have noticed is that on "mondays" logstash indexing falls behind | 18:45 |
| clarkb | though that may be related to the network things that fungi is debugging for zuul | 18:45 |
| clarkb | will need a larger sample size to be confident in blaming monday rechecks | 18:45 |
| *** chuckC has joined #openstack-infra | 18:46 | |
| jeblair | mordred, sdague: i _think_ adding something like " -(age:14d label:code-review-1)" to the dashboard queries would get you the same filtering effect as auto-abandon | 18:47 |
| sdague | jeblair: not quite | 18:48 |
| sdague | because if people continue to comment on it, the age gets reset | 18:48 |
| sdague | and even a 0 comment will trigger clean check | 18:48 |
| jeblair | sdague: sure, but if they comment on it, it's clearly not abandoned | 18:48 |
| sdague | it might not be them | 18:48 |
| sdague | it might be a helpful other person coming through saying "you need to handle x" | 18:49 |
| sdague | which I've seen | 18:49 |
| jeblair | that still sounds like it's not abandoned | 18:49 |
| mtreinish | anteaya: sorry was eating late lunch, planning on the week of the 14th. But working through all the details, nothing set in stone yet. | 18:49 |
| mtreinish | that will probably come later this week :) | 18:49 |
| sdague | jeblair: maybe, i feel like it still is | 18:49 |
| fungi | clarkb: i believe the six gearman disconnects followed by all jobs getting restarted chewed through a lot of additional nodes and probably spiked logs as well (since they kept running, zuul merely lost track of them) | 18:50 |
| sdague | the problem I've got with people posting log lived, not being worked, patches in gerrit is it's a mutex | 18:50 |
| jeblair | sdague: i think the thing is that we really need to start thinking of abandoned as the Really Big Deal that it is. | 18:50 |
| clarkb | fungi: ah that would do it | 18:50 |
| sdague | "should I work on this... oh, mordred has an active set of patches on it, so no" | 18:50 |
| clarkb | fungi: so I will wait on a larger sample size before I blame anything specific | 18:50 |
| sdague | when they really aren't active | 18:50 |
| mordred | sdague: right, but I might have an active train of dev on a thing that sits with a -1 on it for 8 weeks | 18:51 |
| sdague | it's like assigning a bug in progress to yourself | 18:51 |
| jeblair | sdague: yeah, but you have all the info you need to work out if it's really active or not, and how to follow up with the author. | 18:51 |
| fungi | clarkb: yeah, we're still trying to catch up from that i think (noting time-in-queue for a lot of changes is still a bit higher than it should be) | 18:51 |
| jeblair | sdague: if those patches are abandoned then that is all lost | 18:51 |
| openstackgerrit | A change was merged to openstack-infra/config: Run checklang gate only on master https://review.openstack.org/95642 | 18:51 |
| anteaya | mtreinish: sorry to disturb your lunch, thanks, any idea how many days you will meet? | 18:51 |
| sdague | jeblair: sure, I'm not saying the other way is unicorns and rainbows | 18:51 |
| jeblair | sdague: it definitely has spiky horns | 18:52 |
| *** jesusaurus has quit IRC | 18:53 | |
| *** hasharOut is now known as hashar | 18:53 | |
| mtreinish | anteaya: well it'll be a shared infra/qa midcycle so it'll be 4 or 5 days | 18:53 |
| mordred | clarkb: what are our current node spinup timeouts? | 18:53 |
| mordred | clarkb: still 5 minutes/ | 18:54 |
| openstackgerrit | A change was merged to openstack-infra/config: Reindex Gerrit after project move/rename https://review.openstack.org/95603 | 18:54 |
| mordred | ? | 18:54 |
| clarkb | mordred: its not that | 18:54 |
| clarkb | they are comming back with an error status | 18:54 |
| mordred | oh! they are? awesome | 18:54 |
| clarkb | mordred: /var/log/nodepool$ grep 'LaunchStatusException launching node id' debug.log | grep hpcloud-b | wc -l | 18:54 |
| anteaya | mtreinish: ah cool it is the the shared infra meetup, great | 18:54 |
| clarkb | 2413 when I ran it | 18:54 |
| *** zhiyan is now known as zhiyan_ | 18:55 | |
| anteaya | reed so if we can steer our event for the week of June 30th or the week of July 7th, that would be the best for me | 18:55 |
| anteaya | I'll be disappointed to miss Canada Day, but oh well | 18:55 |
| mordred | clarkb: those are 2014-05-27 00:11:59,383 ERROR nodepool.NodeLauncher: LaunchStatusException launching node id: 4332962 in provider: tripleo-test-cloud-rh1 error: | 18:55 |
| reed | anteaya, ACK | 18:55 |
| anteaya | thanks | 18:55 |
| mordred | oh - wait | 18:55 |
| mordred | clarkb: sorry, I suck | 18:56 |
| mordred | clarkb: those aren't hard errors - that's the exception that gets thrown after a timeout | 18:57 |
| clarkb | mordred: no it specifically says that the node went into ERROR | 18:57 |
| mordred | clarkb: nod. thank you | 18:57 |
| clarkb | I can read code to be double sure but status: ERROR implies to me that it went into ERROR | 18:58 |
| *** nati_ueno has quit IRC | 18:58 | |
| jeblair | fungi: options: a) increase the geard debug level. we will need a cinder volume (we're already at 40G/day, i expect that to 4x) b) increase the zuul timeout to 300s and carry on. | 18:59 |
| *** nati_ueno has joined #openstack-infra | 18:59 | |
| *** arnaud__ has joined #openstack-infra | 18:59 | |
| clarkb | mordred: ya that looks like hard error, link with code incoming | 18:59 |
| jeblair | fungi: a.1) add packet logs to the mix to help with debugging | 18:59 |
| *** arnaud__ has quit IRC | 18:59 | |
| zaro | clarkb: i don't think your suggestion would work because there's a ton of jobs that still use the default 'timeout'. I think your suggestion would require a big refactor to change a bunch of jobs to use 'build-timeout'. is this correct? | 18:59 |
| jeblair | meeting time in #openstack-meeting | 19:00 |
| clarkb | mordred: https://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/nodepool.py#n388 it happens after wait for server but it is an explicti check of the status | 19:00 |
| clarkb | mordred: so pretty sure waitforserver returns because node went into hard error | 19:00 |
| clarkb | zaro: yes, but that is a different change | 19:00 |
| clarkb | zaro: this first change is just setting up thedefault appropriately | 19:01 |
| *** derekh_ has joined #openstack-infra | 19:02 | |
| *** michchap_ has quit IRC | 19:04 | |
| *** Lingo is now known as BadCub | 19:05 | |
| *** melwitt has joined #openstack-infra | 19:06 | |
| *** pdmars has joined #openstack-infra | 19:06 | |
| *** ihrachyshka has quit IRC | 19:06 | |
| *** lcheng_ has quit IRC | 19:08 | |
| *** lcheng_ has joined #openstack-infra | 19:08 | |
| dhellmann | should I write up the cross-test blueprint in infra-specs, or is that repo for experimentation still? | 19:08 |
| openstackgerrit | A change was merged to openstack-infra/config: Test python-ironicclient changes symmetrically https://review.openstack.org/95563 | 19:09 |
| *** james_li has joined #openstack-infra | 19:12 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add PostgreSQL integration testing for Gnocchi https://review.openstack.org/95463 | 19:19 |
| openstackgerrit | A change was merged to openstack-infra/config: Gate Gnocchi on Python 3 https://review.openstack.org/95543 | 19:20 |
| mordred | wow. what's gnocchi? | 19:20 |
| Alex_Gaynor | mordred: it's a type of pasta | 19:20 |
| morganfainberg | Alex_Gaynor, ++ beat me to it! | 19:21 |
| mordred | Alex_Gaynor wins. and now I'm hungry | 19:21 |
| rcarrillocruz | a pasta that can be hard to swallow :P | 19:21 |
| mordred | rcarrillocruz: ++ | 19:22 |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Enabled pylint job for manila project https://review.openstack.org/95765 | 19:22 |
| ArxCruz | mtreinish: hey, I'm looking for the unity tests results | 19:24 |
| ArxCruz | mtreinish: we started to listen the tempest, and was a good idea start unity tests as well :) | 19:24 |
| fungi | dhellmann: you can propose it as a child of the initial commit change which is still up for review (there are several which have done so already) | 19:25 |
| SpamapS | hm | 19:29 |
| SpamapS | is heat-specs misconfigured? | 19:29 |
| SpamapS | http://paste.ubuntu.com/7531507/ ... | 19:29 |
| SpamapS | git review -s tries to grab 'orchestration-specs' | 19:29 |
| clarkb | SpamapS: yes it was renamed | 19:30 |
| clarkb | see .gitreview | 19:30 |
| mordred | SpamapS: I submitted a patch | 19:30 |
| *** _nadya_ has quit IRC | 19:31 | |
| SpamapS | which was abandoned? | 19:31 |
| SpamapS | let me just wave that through | 19:31 |
| SpamapS | mordred: no reason given for the abandon. ? | 19:32 |
| *** markmc has joined #openstack-infra | 19:33 | |
| *** nati_ueno has quit IRC | 19:33 | |
| SpamapS | ah and I see because there's no tox.ini | 19:33 |
| SpamapS | and then there's no specs | 19:33 |
| dhellmann | fungi: ok, I'll do that, I just wasn't sure if you were actually using the repo for regular changes or not | 19:33 |
| SpamapS | shouldn't we like, test that a repo has a working gate, before adding the not-noop gate to it? :-P | 19:34 |
| clarkb | SpamapS: no | 19:34 |
| clarkb | you just make it work on your first commit | 19:34 |
| SpamapS | aight, let me just do that then | 19:35 |
| mtreinish | ArxCruz: for example: on https://review.openstack.org/#/c/95843/ , http://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/95843/1/gate-ibm-tempest-python27/1e8e096 is broken | 19:35 |
| ArxCruz | mtreinish: yes, I'm checking what's wrong, it's the swift script I will let you know when it's fixed :) | 19:35 |
| mtreinish | ArxCruz: the unit tests shouldn't be any different on PowerKVM vs anything else, it's just python code that doesn't call anything outside the tree | 19:35 |
| mtreinish | so you huys are just wasting resources running them again | 19:35 |
| mtreinish | *guys | 19:36 |
| ArxCruz | mtreinish: hmmm, not really, we found some tests broken in nova for example, due the lack of ide support on Power | 19:36 |
| fungi | SpamapS: for example https://review.openstack.org/94440 | 19:36 |
| SpamapS | sphinx.errors.SphinxWarning: /home/clint/src/heat-specs/doc/source/index.rst:9: WARNING: toctree glob pattern u'specs/*' didn't match any documents | 19:36 |
| SpamapS | but specs/* has files | 19:36 |
| SpamapS | so.. ?? | 19:36 |
| *** yfried has quit IRC | 19:37 | |
| mordred | SpamapS: https://review.openstack.org/#/c/95297/ | 19:38 |
| mtreinish | ArxCruz: the nova unit tests may different. I'm saying that you won't ever hit a unit test issue like that. The tempest unit tests don't make any external calls. | 19:38 |
| SpamapS | mordred: yes, that is the error I'm trying to fix on that review | 19:38 |
| mtreinish | if you don't believe me look, there are <500 of them | 19:38 |
| mordred | SpamapS: gotcha. so, it turns out that the cookiecutter repo is bs | 19:38 |
| SpamapS | :) | 19:38 |
| mordred | and doesn't have a good set of content in it | 19:38 |
| mordred | look at the infra-specs patch | 19:38 |
| SpamapS | clearly | 19:38 |
| ArxCruz | mriedem: oh, okay. I will talk with kurt, and remove the test :) | 19:39 |
| mordred | I had to 'fix' what cookiecutter did there | 19:39 |
| mriedem | ArxCruz: looking for mtreinish? | 19:39 |
| ArxCruz | oh | 19:39 |
| ArxCruz | damn, again! | 19:39 |
| mordred | at least, I thought I did that for infra - maybe I did it for heat and didnt' psuh? | 19:39 |
| ArxCruz | mtreinish: ^ | 19:39 |
| mordred | I fixed this somewhere | 19:39 |
| jeblair | mordred: what did you have to fix for infra? | 19:39 |
| jeblair | mordred: (i think you're making things up since _I_ wrote the init patch for infra; you must have done it somewhere else) | 19:40 |
| dhellmann | fungi: the change I want to make is in openstack-infra/config, which doesn't appear as an option in storyboard. Should I just say that I want to change the specs repo? | 19:40 |
| fungi | dhellmann: probably | 19:41 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added field restrictions and error messages to project forms https://review.openstack.org/95873 | 19:41 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added field restrictions and error messages to project forms https://review.openstack.org/95873 | 19:42 |
| *** rlandy_ has quit IRC | 19:43 | |
| *** SumitNaiksatam has joined #openstack-infra | 19:44 | |
| *** dprince has quit IRC | 19:45 | |
| mordred | jeblair: I probably did it elsewhere then | 19:45 |
| *** zhiyan_ is now known as zhiyan | 19:46 | |
| *** andreaf has joined #openstack-infra | 19:46 | |
| mordred | SpamapS: the tip of heat-specs works for me | 19:46 |
| mordred | SpamapS: oh! I think I'm missing a git add | 19:47 |
| mtreinish | ArxCruz: were you talking about: https://review.openstack.org/#/c/93621/ on nova? | 19:47 |
| SpamapS | mordred: tip of heat-specs fails because there's no toxini | 19:47 |
| *** lcheng_ has quit IRC | 19:47 | |
| mordred | SpamapS: I meant my change | 19:48 |
| mordred | SpamapS: I just pushed an updated version | 19:48 |
| mordred | sorry - I forgot to git add a file | 19:48 |
| SpamapS | fantastic | 19:48 |
| *** nati_ueno has joined #openstack-infra | 19:50 | |
| *** lcheng_ has joined #openstack-infra | 19:50 | |
| mtreinish | ArxCruz: because the bug there is not a bug because in your env, it's an any non default arch bug. It's still open IMO, you should be mocking the get_arch stuff like on L2030 or L907 of the same file. | 19:51 |
| *** james_li has quit IRC | 19:51 | |
| mordred | dhellmann: btw- see above re: specs-cookiecutter | 19:51 |
| mordred | dhellmann: it does not produce specs repos that can actually build via build_sphinx - I copied what nova did for heat, but I'm not sure it's the _right_ thing | 19:52 |
| mtreinish | ArxCruz: and I'm not sure I'd view one hard coded assumption in a unit test as a justification to run the unit tests on every commit in your 3rd party ci. | 19:52 |
| *** dims has quit IRC | 19:54 | |
| mtreinish | mordred: did I screw up something in specs-cookiecutter? Patches are welcome :) | 19:54 |
| *** zhiyan is now known as zhiyan_ | 19:55 | |
| dhellmann | mordred: yeah, I had to make a bunch of changes for oslo-specs, too. I should submit those back to the cookiecutter repo. | 19:55 |
| mordred | dhellmann: same here | 19:56 |
| *** signed8bit has quit IRC | 19:56 | |
| mtreinish | dhellmann: yeah I made the mistake of doing qa-specs before I did the cookiecutter. So I didn't have to go through the pain firsthand... | 19:57 |
| *** gyee has quit IRC | 19:57 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Update openstack-doc-tools to at least 0.15 https://review.openstack.org/95879 | 19:57 |
| *** saschpe has quit IRC | 19:58 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Update openstack-doc-tools to at least 0.15 https://review.openstack.org/95879 | 19:59 |
| *** saschpe has joined #openstack-infra | 19:59 | |
| *** signed8bit has joined #openstack-infra | 20:00 | |
| *** nati_ueno has quit IRC | 20:00 | |
| *** markmcclain has joined #openstack-infra | 20:00 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/infra-specs: Add spec for adding cross-project unit test jobs https://review.openstack.org/95885 | 20:01 |
| *** melwitt has quit IRC | 20:02 | |
| SergeyLukjanov | jeblair, fungi, clarkb, mordred, anteaya, sorry folks, I'm not very active in IRC last several days and probably will not be active till the end of week - /me totally destructed by jet lag and backlog | 20:02 |
| clarkb | mordred: so, I think you can configure what the valid prefixes are | 20:02 |
| *** melwitt has joined #openstack-infra | 20:02 | |
| anteaya | SergeyLukjanov: yes, I understand jet lag | 20:02 |
| clarkb | mordred: and that is what I meant by making it config. because encoding it in source is ugh | 20:02 |
| mordred | clarkb: ECONTEXT | 20:02 |
| anteaya | SergeyLukjanov: get some sleep | 20:02 |
| clarkb | mordred: test-matrix | 20:02 |
| mordred | clarkb: ah | 20:02 |
| anteaya | SergeyLukjanov: glad to have you back when you are rested | 20:02 |
| lifeless | hi | 20:02 |
| mordred | clarkb: so - I think the main thing where we're missing each other | 20:02 |
| clarkb | mordred: the .py file shouldn't know that hp/* exists or */foo | 20:02 |
| lifeless | the tripleo HP region | 20:02 |
| mordred | clarkb: is that I think there is an expansion of the algorithm which does not need to know about hp/* | 20:03 |
| clarkb | mordred: instead it should be given a list of prefixes (maybe regexes with a substr map) | 20:03 |
| lifeless | I'm trying to investigate the errors there | 20:03 |
| clarkb | mordred: I grok that | 20:03 |
| mordred | clarkb: and does not need for hp/* (or redhat/*) to be put in the config | 20:03 |
| clarkb | mordred: but that algorithm reduces our checks upstream | 20:03 |
| SergeyLukjanov | anteaya, the issue is that I'm sleeping ok just permanently tired due to the feeling of the incorrect timezone ;) that's really funny to feel 8h diff tz | 20:03 |
| clarkb | mordred: we lose a feature by doing that | 20:03 |
| anteaya | SergeyLukjanov: yup | 20:03 |
| anteaya | and dizzyness and headaches | 20:03 |
| clarkb | mordred: a couple really | 20:03 |
| clarkb | mordred: first we map feature/* to master | 20:04 |
| anteaya | sleep my friend | 20:04 |
| clarkb | mordred: and second we check that you don't have hp/* branches upstream | 20:04 |
| mordred | clarkb: ok - maybe configurable prefix regexes will do what I'm talking about | 20:04 |
| *** shivharis has quit IRC | 20:05 | |
| mordred | I just want to be able to carry the logic encoded by test-matrix in wholesale except with the existence of a namespace- so $prefix/stable/grizzly should be able to pick up the feature matrix as it is there | 20:05 |
| mordred | from stable/grizzly | 20:05 |
| mordred | clarkb: we may be getting close to an understanding... | 20:05 |
| lifeless | can I get someone to tell me what nodepool thinks the current status is ? | 20:05 |
| derekh_ | on a related note has nodepool stoppd talking to the other tripleo region in the last few minutes ? | 20:06 |
| *** krtaylor has joined #openstack-infra | 20:06 | |
| *** marcoemorais has quit IRC | 20:07 | |
| *** marcoemorais has joined #openstack-infra | 20:07 | |
| clarkb | mordred: right so I think you do a branchmap:\n ^.*/(stable/.*)$: \1 | 20:08 |
| mikal | jeblair: can you please add me to nova-coresec? | 20:08 |
| *** BadCub01_ has joined #openstack-infra | 20:08 | |
| clarkb | mordred: maybe thats gross to write and we can make the config of it less gross | 20:08 |
| clarkb | mordred: but I think that describes what you want | 20:08 |
| mordred | clarkb: where do you do that? | 20:08 |
| clarkb | mordred: in the test-matrix config | 20:08 |
| sdague | on the list of patches I've got outstanding, I'd like to get opinions on - https://review.openstack.org/#/c/91799/ - which is removing all the pypy jobs | 20:08 |
| *** BadCub has quit IRC | 20:08 | |
| clarkb | sdague: pypy should be fine now | 20:08 |
| sdague | it's still got a 20% fail rate in the gate | 20:09 |
| clarkb | except for az2 and we can remove the pypy nodes from there | 20:09 |
| clarkb | ya its hpcloud hating us | 20:09 |
| mordred | clarkb: I'd love to have a way to have some overlay config, so that forking/patching the test-matrix config was not needed | 20:09 |
| clarkb | mordred: but tahts the whole point | 20:09 |
| clarkb | its config it is needed | 20:09 |
| clarkb | otherwise we haev the problem jeblair described | 20:10 |
| clarkb | which is we have no way to test and unbreak downstreams | 20:10 |
| sdague | clarkb: so does someone else have the alternate patch up to make it voting again | 20:10 |
| clarkb | lifeless: I haev no idea what is going on | 20:10 |
| sdague | ? | 20:10 |
| clarkb | sdague: yes Alex_Gaynor has a patchto make it voting | 20:10 |
| jeblair | mikal: done | 20:10 |
| mikal | jeblair: thanks | 20:10 |
| mordred | clarkb: I'm sorry - I'm very dense - I do not know what you mean by that. I'm basically just talking about some way to run-parts a config, or have a second config that can be there and optional or something, so that a downstream can consume the upstream config unedited | 20:11 |
| clarkb | mordred: we could do what jjb does | 20:11 |
| sdague | clarkb: cool | 20:11 |
| clarkb | and combine all the yaml docs | 20:11 |
| *** mbacchi has quit IRC | 20:12 | |
| *** nati_ueno has joined #openstack-infra | 20:12 | |
| anteaya | jeblair: we talked about making me gerrit admin at the summit | 20:12 |
| sdague | mordred: can you sketch out what you need after the tc meeting? I only have half the context here | 20:12 |
| mordred | sdague: yah. let's come back to it after meeting | 20:12 |
| anteaya | did we want to go ahead and do that or do we need something else to happen first | 20:12 |
| lifeless | clarkb: I see three vms in state error | 20:12 |
| anteaya | like do you want to post to the ml | 20:12 |
| *** hashar has quit IRC | 20:12 | |
| jeblair | anteaya: yeah, i've been looking into it and am not quite ready with a proposal yet | 20:12 |
| anteaya | okay great | 20:13 |
| clarkb | lifeless: looks like we haev a bunch of nodes in delete and building states | 20:13 |
| anteaya | do you need to see something more from me? | 20:13 |
| *** chuckC has quit IRC | 20:13 | |
| clarkb | lifeless: and they have been that way for 7 hours | 20:13 |
| lifeless | clarkb: and a f20 template in running state | 20:13 |
| anteaya | jeblair: do you need to see something more from me? | 20:13 |
| lifeless | clarkb: want to bet that we managed to glitch the firewall | 20:13 |
| clarkb | lifeless: I would never bet against borked networking in your cloud >_> | 20:14 |
| lifeless | clarkb: and thus need to reset tcp again? | 20:14 |
| mordred | lifeless: it's a firewall. it is a glitch by design :) | 20:14 |
| lifeless | clarkb: said firewall is at the rackspace end AFAICT | 20:14 |
| lifeless | anyhow | 20:14 |
| lifeless | pretty please? | 20:14 |
| clarkb | how did we bork the firewall in rackspace? | 20:14 |
| jeblair | anteaya: not at the moment | 20:14 |
| anteaya | kk | 20:14 |
| clarkb | especially since rax and hp and rh endpoints not in that DC are fine | 20:15 |
| lifeless | clarkb: hypothesis - too long a period with no response on a socket and it forgets the socket exists, then when you should be getting RST from your packets, you don't | 20:15 |
| *** radez is now known as radez_g0n3 | 20:15 | |
| *** dhellman_ has joined #openstack-infra | 20:15 | |
| lifeless | clarkb: oh yes, I *know* the fundamental issue is our end, but the firewall that prevents TCP's state machine doing its thing is separate to the cause of the issue. | 20:16 |
| lifeless | so this is an interesting change in the -22 build of linux | 20:16 |
| lifeless | * vlan: Set correct source MAC address with TX VLAN offload enabled | 20:16 |
| lifeless | tx-vlan-offload: on [fixed] | 20:17 |
| lifeless | and we have a vlan | 20:17 |
| lifeless | so thats a candidate for the issue | 20:17 |
| derekh_ | lifeless: is it possible you've done something in R1 that would cause nodepool to stop talking to R2 ? | 20:18 |
| *** primeministerp has joined #openstack-infra | 20:18 | |
| lifeless | derekh_: No. | 20:18 |
| *** otherwiseguy has quit IRC | 20:19 | |
| lifeless | derekh_: a) I haven't done anything to R1 yet, b) nodepool is entirely segregated on that stuff. | 20:19 |
| clarkb | so what connection needs killing? 138.35.77.16:13774 (ESTABLISHED) that one? | 20:19 |
| derekh_ | lifeless: ok | 20:19 |
| clarkb | mordred: also your plea to get people on IRC didn't work at all | 20:19 |
| clarkb | unless folks are lurking here? | 20:20 |
| *** primeministerp has quit IRC | 20:20 | |
| *** primeministerp has joined #openstack-infra | 20:20 | |
| mordred | clarkb: it did not - although they are investigating errors on their end | 20:20 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added simple logged-in dashboard https://review.openstack.org/92662 | 20:21 |
| jeblair | lifeless: why do you think this is a firewall issue? i believe our understanding of the problem was lack of keepalive support in novaclient combined with the tripleo controller dropping its connection with no termination packet. | 20:21 |
| ArxCruz | mtreinish: okay, I will talk with kurt :) | 20:21 |
| *** boris-42 has quit IRC | 20:21 | |
| *** boris-42 has joined #openstack-infra | 20:21 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Update openstack-doc-tools to at least 0.15 https://review.openstack.org/95879 | 20:22 |
| *** reed has quit IRC | 20:23 | |
| zaro | clarkb: i think we are in agreement as to what the default.yaml and macro.yaml should look like. i was taking a different approach on steps to get there. I think your approach is to prepare macro.yaml and default.yaml (as 1st change) for the big refactoring of jobs (2nd change). | 20:25 |
| lifeless | jeblair: I'm convinced that the reason it doesn't self correct is a firewall somewhere. | 20:25 |
| *** afazekas has joined #openstack-infra | 20:25 | |
| lifeless | jeblair: tcp keepalive on would allow it to self correct without the RST from the target system. | 20:25 |
| clarkb | zaro: ok | 20:25 |
| lifeless | jeblair: removing the firewall(s) that prevent the RST being triggered/received would also allow it to self correct. | 20:25 |
| jeblair | lifeless: i would not expect an RST without an outgoing packet; in these cases, nodepool is not transmitting, it's waiting for a responses | 20:26 |
| *** dims has joined #openstack-infra | 20:26 | |
| lifeless | jeblair: I am fairly sure the tcp state machine on the nodepool end is in fact trying to get an ACK for the last request sent | 20:27 |
| zaro | clarkb: i think your approach works as well. mine was to do both in 1 change (but has not been done yet). | 20:27 |
| lifeless | jeblair: so there are outgoing packets, but we can check this easily enough | 20:27 |
| clarkb | zaro: the reason I want to do it that way is so that they don't drift apart | 20:28 |
| clarkb | zaro: by using the macro from the start any changes to macro end up everywhere | 20:28 |
| clarkb | is port 13774 the nova endpoint? | 20:29 |
| *** pdmars has quit IRC | 20:29 | |
| lifeless | clarkb: yes | 20:29 |
| jeblair | lifeless: at the summit i was told the underlying network problem was mellanox related. do you still think that's the case? when do you expect it to be fixed? | 20:30 |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/devstack-gate: Allow to configure git base URL https://review.openstack.org/95901 | 20:30 |
| lifeless | jeblair: its clearly more than just that. | 20:30 |
| clarkb | I can kill 138.35.77.16:13774 (ESTABLISHED) whenever people are ready | 20:30 |
| lifeless | clarkb: please do | 20:30 |
| lifeless | jeblair: I've asked internally for firmware patchse for the hardware | 20:30 |
| zaro | clarkb: yeah, i see the point. it's a little confusing with the changes that i've already pushed (84637 & 92773) for review. So i think i'll just abandone those and redo with the approach you suggested. probably will work out better. | 20:31 |
| lifeless | jeblair: and hopefully we'll be able to redeploy with an HA control plane very very soon | 20:31 |
| lifeless | greghaynes is on that feature along with jprovazn and one of the dmitry's | 20:31 |
| clarkb | fungi: jeblair: do you want to do any other debugging before I `killcx 138.35.77.16:13774` | 20:32 |
| zaro | clarkb: did we restart gerrit over the weekend? | 20:32 |
| clarkb | zaro: yesterday fungi did it to reindex | 20:32 |
| lifeless | clarkb: oh you haven't? lets look for traffic | 20:33 |
| lifeless | clarkb: tcpdump -i $whatever host 138.35.77.16 | 20:33 |
| lifeless | clarkb: or similar | 20:33 |
| zaro | clarkb, fungi : darn, gerrit still wrapping files into zip on download https://review.openstack.org/#/c/93108/ | 20:34 |
| lifeless | clarkb: whats your external IP for the nodepool machine ? I will dumpon the server side | 20:34 |
| clarkb | lifeless: addr:192.237.211.91 | 20:35 |
| lifeless | ohhhh ho ho ho this is interesting | 20:35 |
| lifeless | I just spotted what looks like an egregious misconfiguration in the vlan setup | 20:35 |
| clarkb | I see nothing going in or out with host 138.35.77.16 | 20:35 |
| lifeless | bear with me while I tickle that | 20:36 |
| clarkb | ok | 20:36 |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/devstack-gate: Allow to configure git base URL https://review.openstack.org/95901 | 20:39 |
| *** otherwiseguy has joined #openstack-infra | 20:41 | |
| lifeless | the ip address was not added to the vlan, but instead to the external bridge | 20:41 |
| lifeless | its a wonder its working at all | 20:41 |
| *** BadCub has joined #openstack-infra | 20:41 | |
| clarkb | vlans don't care about addresses really | 20:42 |
| clarkb | its a frame tag | 20:42 |
| *** BadCub has quit IRC | 20:42 | |
| lifeless | clarkb: it is, but ovs is generally a lot better about strictness than linxubridge | 20:45 |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/devstack-gate: Make the master branch configurable https://review.openstack.org/95904 | 20:45 |
| lifeless | clarkb: you are not meant to process traffic for not-your-vlan on a raw interface | 20:45 |
| fungi | unless it has some equivalent of cisco's default<->native translation | 20:46 |
| fungi | but generally correct | 20:46 |
| *** zhiyan_ is now known as zhiyan | 20:47 | |
| fungi | clarkb: so tcpdump shows no outgoing packets matching that destination address+port? | 20:47 |
| fungi | if that's the case (not even retransmits), then yes you don't need a firewall silently discarding things to see this behavior | 20:48 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Bump min hacking version to 0.8.1 https://review.openstack.org/82339 | 20:48 |
| lifeless | its a bit of a thread-the-needle condition, but yes. | 20:49 |
| lifeless | keepalive ftw | 20:49 |
| fungi | now someone just needs to fix the requests library to support it correctly | 20:49 |
| clarkb | fungi: correct there is nothing | 20:49 |
| lifeless | it does support it | 20:49 |
| clarkb | tcpdump -i eth0 host whateverip | 20:49 |
| lifeless | there's just some horrid mess somewhere in the stack | 20:50 |
| *** lcheng_ has quit IRC | 20:50 | |
| *** hashar has joined #openstack-infra | 20:50 | |
| *** marcoemorais has quit IRC | 20:51 | |
| fungi | i thought there was something with the underlying calls from requests to urllib not setting it up, but i forget the details now | 20:51 |
| *** marcoemorais has joined #openstack-infra | 20:51 | |
| clarkb | fungi: yes see jeblair's comments | 20:51 |
| lifeless | that may be the mess | 20:51 |
| *** andreykurilin has quit IRC | 20:51 | |
| fungi | oh, right, and this is bubbling up through novaclient | 20:51 |
| *** gokrokve_ has quit IRC | 20:52 | |
| clarkb | fungi: lifeless: in any case let me know if/when you want the existing connection to be shot | 20:55 |
| clarkb | jeblair: can you review 95302 as an alternative to 95277 | 20:56 |
| *** zhiyan is now known as zhiyan_ | 20:56 | |
| lifeless | clarkb: will do | 20:58 |
| lifeless | clarkb: I'm stomping on this bug first | 20:58 |
| *** markwash has joined #openstack-infra | 20:58 | |
| jeblair | clarkb: can you respond to jhesketh on https://review.openstack.org/#/c/95302/1 ? | 21:00 |
| clarkb | looking | 21:00 |
| *** hashar has quit IRC | 21:01 | |
| clarkb | jeblair: hrm hard to as there is no explanation to why | 21:01 |
| *** julim has quit IRC | 21:01 | |
| *** alexpilotti has joined #openstack-infra | 21:01 | |
| clarkb | but I will respond with what I discovred and see what jhesketh says | 21:01 |
| jeblair | clarkb: i thought your commit message was pretty clear | 21:02 |
| *** gyee has joined #openstack-infra | 21:02 | |
| anteaya | jeblair: jhesketh has a -1 on your infra-manual initial commit: https://review.openstack.org/#/c/92475/ | 21:02 |
| anteaya | I'd really like to see it merged this week, if possible | 21:02 |
| derekh_ | nodepool still doesn't seem to be talking to tripleo-test-cloud-rh1 (for about an hour now), anybody got any idea why? | 21:03 |
| *** bhuvan_ has joined #openstack-infra | 21:03 | |
| *** hashar has joined #openstack-infra | 21:03 | |
| clarkb | jeblair: done | 21:03 |
| lifeless | clarkb: ok, I've sorted by head out now. Looks good - please kick. | 21:04 |
| *** ArxCruz has quit IRC | 21:04 | |
| jeblair | clarkb: zuul config change +2 | 21:04 |
| Alex_Gaynor | Are new jenkins builds not starting again? | 21:05 |
| *** bhuvan has quit IRC | 21:05 | |
| derekh_ | Alex_Gaynor: +1 , could explain why I'm not seeing any new instances being spawned on tripleo-test-cloud-rh1 | 21:06 |
| fungi | jeblair: back on the zuul gearman disconnects, i worry that a 300s timeout might still occasionally get tripped, especially for downstream users who may run it on more resource-constrained systems than we have. i'm happy to add a 0.25t cinder volume at /var/log/zuul (we've got available quota) and take the next quiet opportunity for a quick zuul restart | 21:06 |
| clarkb | Alex_Gaynor: yes that apepars to be the case | 21:06 |
| clarkb | which would explain not takling to rh1 | 21:07 |
| *** chuckC has joined #openstack-infra | 21:07 | |
| *** eharney has quit IRC | 21:07 | |
| *** gyee has quit IRC | 21:07 | |
| clarkb | lifeless: actually looks like we have building nodes to your cloud now | 21:07 |
| * clarkb looks at zuul | 21:08 | |
| lifeless | ok | 21:08 |
| lifeless | so we really need to fix this | 21:08 |
| clarkb | lifeless: we being you? | 21:08 |
| fungi | clarkb: i'm checking now to see if we ended up with new disconnects | 21:08 |
| clarkb | fungi: did zuul disconnect again? | 21:08 |
| clarkb | thanks | 21:08 |
| lifeless | we being the royal us | 21:08 |
| *** nati_ueno has quit IRC | 21:08 | |
| clarkb | lifeless: I didn't kill anything btw | 21:08 |
| lifeless | SpamapS: still around? Did we file a bug over novaclient not doing keepalive properly? | 21:08 |
| clarkb | so ya I don't know | 21:08 |
| lifeless | clarkb: ?! | 21:08 |
| lifeless | clarkb: no, I see the same nodes before | 21:09 |
| lifeless | clarkb: or do you mean the rh1 region ? | 21:09 |
| clarkb | I see new ones building | 21:09 |
| lifeless | clarkb: you do? using what tool | 21:09 |
| fungi | clarkb: are you sure the builds which started 8 hours ago didn't just time out (they were due for it) and get replaced with new builds which are also stuck? | 21:09 |
| derekh_ | clarkb: yup, thats the rh region, lifeless is concerned with the hp region | 21:10 |
| clarkb | lifeless: nodepool list | 21:10 |
| clarkb | fungi: oh that could be | 21:10 |
| fungi | and yes, we've seen two more local gearman disconnects on zuul. getting timestamps now | 21:10 |
| clarkb | derekh_: its not the rh region | 21:10 |
| clarkb | but what fungi describes may be te case | 21:10 |
| clarkb | lifeless: I will kill connection now | 21:10 |
| lifeless | clarkb: sec | 21:10 |
| lifeless | clarkb: check for tcp traffic again | 21:10 |
| jeblair | fungi: i think we should go for a full 1tb | 21:10 |
| lifeless | clarkb: if nodepool wasn't actively trying anything, that would explain no traffic :) | 21:11 |
| clarkb | jeblair: 1tb for what? | 21:11 |
| *** nati_ueno has joined #openstack-infra | 21:11 | |
| *** e0ne has quit IRC | 21:11 | |
| fungi | jeblair: we can. we've got just over 2t of quota open at the moment | 21:11 |
| fungi | i'll get it spinning up | 21:11 |
| *** markmc has quit IRC | 21:11 | |
| jeblair | fungi: oh we have quota! neat. | 21:11 |
| jeblair | clarkb: for geard debug level logs | 21:11 |
| *** e0ne has joined #openstack-infra | 21:11 | |
| derekh_ | clarkb: ok, fair enough, at the same time nodepool has started creating new instances on the rh region (after a 1hr window of silence) | 21:12 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Add a build-timeout macro https://review.openstack.org/95912 | 21:12 |
| fungi | jeblair: according to cinderclient, we're using 22628 of our 25600 maxTotalVolumeGigabytes | 21:12 |
| fungi | so nearly 3tb quota open in fact | 21:12 |
| jeblair | clarkb: also, istr you said you thought you could get logrotate working with python logging; that would probably really help here | 21:12 |
| *** aysyd has quit IRC | 21:12 | |
| jeblair | fungi: hopefully this is temporary | 21:12 |
| clarkb | jeblair: yes we do it with logstash workers | 21:12 |
| * clarkb find sa link | 21:12 | |
| *** dims has quit IRC | 21:13 | |
| *** lcheng_ has joined #openstack-infra | 21:13 | |
| clarkb | jeblair: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/log_processor/manifests/worker.pp#n54 copytruncate is the important bit | 21:14 |
| fungi | btw, the new gearman disconnects were at 19:31:58 and 19:44:44 in the logs | 21:14 |
| *** e0ne has quit IRC | 21:14 | |
| fungi | so ~1.5 hours ago | 21:14 |
| lifeless | clarkb: nothing ? if so reset please | 21:14 |
| *** dkliban is now known as dkliban_afk | 21:15 | |
| clarkb | lifeless: nothing, reseting connection now | 21:15 |
| jeblair | clarkb: would you be up for making such a change? | 21:15 |
| clarkb | jeblair: sure | 21:15 |
| clarkb | lifeless: done | 21:15 |
| lifeless | boom traffic | 21:16 |
| jeblair | clarkb: maybe we can land that and the swift change, add fungi's volume, and then restart zuul | 21:16 |
| *** markwash_ has joined #openstack-infra | 21:16 | |
| clarkb | jeblair: ++ | 21:16 |
| clarkb | jeblair: do I need to worry about merger logs too? | 21:16 |
| *** markwash has quit IRC | 21:17 | |
| * fungi finds it disappointing that nova volume-attach understands server display names but not volume display names | 21:17 | |
| jeblair | clarkb: probably best to be consistent | 21:17 |
| jeblair | clarkb: so yes? | 21:17 |
| mordred | fungi: there are so many things I find disappointing | 21:17 |
| *** gyee has joined #openstack-infra | 21:18 | |
| clarkb | jeblair: actually looks like merger logs are rotated with python already | 21:19 |
| *** dhellman_ has quit IRC | 21:19 | |
| *** lcostantino has quit IRC | 21:19 | |
| jeblair | clarkb: er, isn't the goal to rotate with logrotate (so we can compress)? | 21:20 |
| jeblair | (and stop rotating with python) | 21:21 |
| fungi | jeblair: okay, we have a 1tb filesystem on zuul:/dev/main/logs | 21:21 |
| *** duran has quit IRC | 21:21 | |
| *** markwash_ has quit IRC | 21:21 | |
| clarkb | jeblair: oh I think I missed the wanting compression goal | 21:21 |
| *** markmcclain has quit IRC | 21:21 | |
| clarkb | but now that I know that I will do all the things | 21:21 |
| *** markwash has joined #openstack-infra | 21:21 | |
| fungi | jeblair: i'll add it to /etc/fstab as /var/log/zuul | 21:21 |
| *** e0ne has joined #openstack-infra | 21:21 | |
| lifeless | whee some terrifying things happen in client libs | 21:21 |
| lifeless | https://bugs.launchpad.net/python-novaclient/+bug/1297796 | 21:21 |
| jeblair | clarkb: cool | 21:21 |
| uvirtbot | Launchpad bug 1297796 in python-novaclient "nova python client is not process safe " [High,Fix committed] | 21:21 |
| fungi | oh, nm, we have /opt/log/zuul right now | 21:22 |
| jeblair | fungi: yeah; i think we want to swap it out | 21:22 |
| jeblair | fungi: so maybe we need to stop;umount;mount;start -- and all that should wait for clarkb's change to merge | 21:22 |
| fungi | okay, so yeah, replace the bindmount with a device mount after we stop the service | 21:22 |
| *** pcm__ has quit IRC | 21:22 | |
| fungi | i'll just hold off editing fstab for the moment | 21:22 |
| *** markwash has quit IRC | 21:23 | |
| clarkb | jeblair: fungi: how many days do you want? 30? | 21:24 |
| jeblair | clarkb: let's go to 7 for now; these will be huge | 21:24 |
| fungi | for the gearman debug logs a week is probably more than sufficien | 21:24 |
| fungi | t | 21:25 |
| fungi | or what jeblair proposes | 21:25 |
| jeblair | yeah, actually... | 21:25 |
| jeblair | i don't think we need to make zuul larger | 21:25 |
| clarkb | ok | 21:25 |
| jeblair | so i think we can stick with 30 days for zuul itself, and 7 days for the gearman server which we will drop to debug | 21:25 |
| fungi | so have the gearman logs go to a new sub-tree? | 21:25 |
| fungi | or do you still want them in the same dir? | 21:26 |
| clarkb | fungi: they are already in a different location | 21:26 |
| *** prad_ has quit IRC | 21:26 | |
| fungi | clarkb: right now they're in different files but in the same dir | 21:26 |
| *** arnaud__ has joined #openstack-infra | 21:26 | |
| fungi | just wondering if we want the gearman debug logs isolated to a new volume, or move all zuul logging to the new volume | 21:27 |
| jeblair | fungi: let's move everything to the new vol | 21:27 |
| derekh_ | fungi: another gearman disconnect ? looks like zuul is running nothing status.openstack.org/zuul/ | 21:27 |
| fungi | easily done | 21:27 |
| fungi | derekh_: we're about to restart it i think | 21:27 |
| derekh_ | fungi: ahh ok, never mind me :-) | 21:28 |
| fungi | but yes it does seem to have completely given up running things this time | 21:28 |
| jeblair | zuul seems to be stuck in a loop due to an error | 21:28 |
| *** mrda_away is now known as mrda | 21:28 | |
| fungi | i know how it feels | 21:28 |
| jeblair | i've stopped zuul | 21:29 |
| fungi | i'll get to work setting up the new mount point | 21:29 |
| *** e0ne has quit IRC | 21:29 | |
| jeblair | #status alert Zuul is offline due to an operational issue; ETA 2200 UTC. | 21:30 |
| openstackstatus | jeblair: sending alert | 21:30 |
| fungi | clarkb: your cwd is in /var/log/zuul... can you switch to /opt/log/zuul and look at the files from there so i can umount the bindmount? | 21:31 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Rotate zuul logs with logrotate https://review.openstack.org/95915 | 21:31 |
| jeblair | fungi: i was in that dir; changed | 21:31 |
| clarkb | fungi: yes sorry | 21:31 |
| clarkb | done | 21:31 |
| fungi | jeblair: clarkb: thanks! | 21:31 |
| fungi | someone has a process as root less'ing the debug log still | 21:32 |
| clarkb | er was that me? | 21:32 |
| jeblair | fungi: that was me | 21:32 |
| fungi | all better--thanks@ | 21:32 |
| clarkb | no not me | 21:32 |
| jeblair | fungi: let me know when it's safe for me to find out why i had to stop zuul. :) | 21:32 |
| -openstackstatus- NOTICE: Zuul is offline due to an operational issue; ETA 2200 UTC. | 21:32 | |
| *** ChanServ changes topic to "Zuul is offline due to an operational issue; ETA 2200 UTC." | 21:32 | |
| openstackgerrit | A change was merged to openstack-infra/config: Pass tenant_name to zuul config. https://review.openstack.org/95302 | 21:33 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add query for bug 1308715 https://review.openstack.org/95918 | 21:33 |
| uvirtbot | Launchpad bug 1308715 in nova/icehouse "Deadlock on quota_usages" [High,In progress] https://launchpad.net/bugs/1308715 | 21:33 |
| *** jesusaurus has joined #openstack-infra | 21:33 | |
| fungi | jeblair: want me to copy the logs from /opt/log/zuul into /var/log/zuul or just leave them there? | 21:34 |
| *** eharney has joined #openstack-infra | 21:34 | |
| jeblair | fungi: i think we can leave em | 21:34 |
| openstackstatus | jeblair: finished sending alert | 21:34 |
| fungi | weird, filesystem is reported as the wrong size. investigating | 21:35 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Add --submit-immediately/-S command to submit immediately after push https://review.openstack.org/93952 | 21:35 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Switch zuul geard to debug https://review.openstack.org/95920 | 21:36 |
| *** nati_ueno has quit IRC | 21:37 | |
| jeblair | mordred: do you want to review https://review.openstack.org/#/c/95915/1 and the one after? | 21:37 |
| *** Ajaeger has quit IRC | 21:38 | |
| fungi | okay, resized up to 1tb. i got mixed up between --size and --extents on the lvcreate :/ | 21:39 |
| fungi | resize is still going but should finish in moments | 21:39 |
| *** davidlenwell_ has joined #openstack-infra | 21:40 | |
| fungi | df now reports 1008G avail on /var/log/zuul | 21:42 |
| *** nati_ueno has joined #openstack-infra | 21:43 | |
| *** marcoemorais has quit IRC | 21:43 | |
| *** zzelle has joined #openstack-infra | 21:43 | |
| *** SumitNaiksatam has quit IRC | 21:43 | |
| fungi | do zuul's daemons need a hup signal to flush and close old file descriptors and open fresh ones, and if so does logrotate take care of that by default or does it need to be added as a service notify? | 21:44 |
| *** davidlenwell_ has quit IRC | 21:44 | |
| *** marcoemorais has joined #openstack-infra | 21:44 | |
| jeblair | clarkb: ^ | 21:44 |
| *** melwitt has quit IRC | 21:44 | |
| clarkb | no clue | 21:44 |
| clarkb | oh wait | 21:44 |
| clarkb | copytruncate is the trick | 21:44 |
| fungi | aside from not knowing the answer to that, the change lgtm | 21:44 |
| *** melwitt has joined #openstack-infra | 21:44 | |
| clarkb | it copies the files then truncates the existing one | 21:44 |
| clarkb | so you never have to change file descriptors | 21:45 |
| *** marcoemorais has quit IRC | 21:45 | |
| jeblair | but i guess you briefly have 2x the current data on disk | 21:45 |
| fungi | and the pythonlogging implementation knows to reset its pointer to 0 when that happens? | 21:45 |
| openstackgerrit | Christian Berendt proposed a change to openstack-infra/git-review: replaced unicode() with six.text_type() https://review.openstack.org/95925 | 21:45 |
| *** marcoemorais has joined #openstack-infra | 21:45 | |
| *** SumitNaiksatam has joined #openstack-infra | 21:45 | |
| clarkb | fungi: yes | 21:45 |
| *** davidlenwell has quit IRC | 21:45 | |
| *** dims has joined #openstack-infra | 21:45 | |
| clarkb | fungi: it works with logstash workers | 21:45 |
| fungi | i've definitely seen some services deal poorly with truncating open file descriptors in append | 21:45 |
| fungi | okay, awesome | 21:45 |
| *** mriedem has quit IRC | 21:46 | |
| fungi | the 2x one log file isn't too worrysome given how much breathing room we have for it | 21:46 |
| openstackgerrit | A change was merged to openstack-infra/config: Rotate zuul logs with logrotate https://review.openstack.org/95915 | 21:46 |
| clarkb | ya and it may do the compress as it copies | 21:46 |
| clarkb | so not quite 2x | 21:46 |
| jeblair | fungi: i would not be surprised if we end up with a 200G log file. | 21:47 |
| fungi | wowza | 21:47 |
| fungi | processing that will be fun | 21:47 |
| fungi | heck, logrotate very well may struggle to compress that | 21:47 |
| clarkb | we should sahara boot a cluster | 21:47 |
| *** davidlenwell has joined #openstack-infra | 21:47 | |
| clarkb | fungi: ya, I would be more worried about the size of things than python logging getting confused | 21:48 |
| *** dangers is now known as dangers_away | 21:48 | |
| *** zhiyan_ is now known as zhiyan | 21:48 | |
| fungi | agreed | 21:48 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch zuul geard to debug https://review.openstack.org/95920 | 21:48 |
| SergeyLukjanov | clarkb, /me reading scrollback | 21:48 |
| jeblair | oh no you woke up SergeyLukjanov! :) | 21:48 |
| clarkb | whoops | 21:49 |
| jeblair | running puppet on zuul hosts | 21:49 |
| jeblair | fungi: ready for me to start zuul? | 21:50 |
| fungi | jeblair: any time you're ready | 21:51 |
| SergeyLukjanov | storing logs in hdfs, mmm, than we could process Tbs of them | 21:51 |
| clarkb | SergeyLukjanov: yeah and logstash + ES can use hdfs as a long term store | 21:51 |
| jeblair | ImportError: No module named FileHandler | 21:52 |
| clarkb | what | 21:52 |
| clarkb | I even erad the docs | 21:52 |
| *** mwagner_lap has quit IRC | 21:52 | |
| clarkb | jeblair: arg its in the root module | 21:52 |
| clarkb | jeblair: can you remove the .handlers and see if that fixes it? | 21:52 |
| clarkb | should be logging.FileHandler I fail | 21:53 |
| jeblair | clarkb: then we get: TypeError: __init__() takes at most 5 arguments (33 given) | 21:54 |
| jeblair | clarkb: i think your tuple was missing a ',' | 21:54 |
| fungi | ahh, yep | 21:54 |
| fungi | args=('/var/log/zuul/zuul.log',) | 21:54 |
| fungi | et cetera | 21:55 |
| fungi | it was trying to enumerate a string | 21:55 |
| jeblair | clarkb: it works with those 2 corrections | 21:55 |
| jeblair | clarkb: want to push up a change to fix that and we'll go aheand and merge it and start with that? | 21:56 |
| clarkb | ok new patch coming | 21:56 |
| fungi | once we get this running, i need to disappear and start rolling out pizza dough | 21:56 |
| *** harlowja_ is now known as harlowja_away | 21:57 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Fix python FileHandler loggers https://review.openstack.org/95931 | 21:57 |
| fungi | in good news, this has given the logstash job queue time to catch back up! | 21:57 |
| SergeyLukjanov | I'd like to ask you folks for the infra root world tour this week | 21:57 |
| clarkb | jeblair: fungi ^ there it is | 21:57 |
| *** zhiyan is now known as zhiyan_ | 21:57 | |
| jeblair | i haven't tracked down what was causing the 'already submitted errors', though the timing suggests it could be an edge case with gearman disconnects | 21:57 |
| *** harlowja_away is now known as harlowja_ | 21:57 | |
| fungi | lgtm | 21:57 |
| jeblair | er, 'already reported' | 21:57 |
| *** lakshmiS has quit IRC | 21:58 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix python FileHandler loggers https://review.openstack.org/95931 | 21:58 |
| jeblair | re-puppeting | 21:58 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Simply jobs by using the build-timeout macro https://review.openstack.org/95933 | 21:59 |
| zzelle | if you want to use logrotate, you should use WatchedFileHandler not FileHandler | 21:59 |
| jeblair | zzelle: oh neat | 22:00 |
| clarkb | zzelle: looks like that would be a way around copytruncate | 22:00 |
| mikal | Is there some way for me as PTL to override a core's -2 on a patch? | 22:00 |
| fungi | 136M /var/log/zuul/gearman-server.log already | 22:00 |
| jeblair | mikal: it's never come up before | 22:00 |
| clarkb | I can prep another change that will use watchedfilehandler | 22:00 |
| fungi | mikal: not really other than getting a gerrit admin to delete the vote (with some clear explanation) | 22:01 |
| zaro | clarkb: i hope this is what you were looking for.. https://review.openstack.org/95933 | 22:01 |
| mikal | Ok, I've emailed the core and asked him to tweak his vote | 22:01 |
| jeblair | clarkb: might be worth it; fungi what do you think? | 22:01 |
| mikal | The problem being he is on vacation | 22:01 |
| mikal | So if he doesn't reply in a couple of days I might ask for some help | 22:01 |
| jeblair | mikal: okay. i think we'll want a good paper trail for something like that. | 22:02 |
| clarkb | jeblair: also did swift config make it in? | 22:02 |
| clarkb | jeblair: and it worked this time? | 22:02 |
| jeblair | clarkb: yes | 22:02 |
| fungi | jeblair: clarkb: if the current configuration is tested and working for the logstash workers, then i'm fine with considering that an improvement for another day | 22:02 |
| mikal | jeblair: I don't think its contentious, the -2 was "please land the spec first", which is now done. | 22:02 |
| *** wenlock_ has joined #openstack-infra | 22:02 | |
| mikal | jeblair: its just the core involved isn't around to remove the -2 | 22:02 |
| mikal | jeblair: but like I said, I emailed him and maybe he'll notice | 22:02 |
| clarkb | jeblair: woot | 22:02 |
| jeblair | mikal: ok cool, so probably leave a note on the review asking us to do it w/explanation, and then ping us | 22:03 |
| jeblair | mikal: when/if you decide to proceed | 22:03 |
| *** hashar has quit IRC | 22:04 | |
| mikal | jeblair: cool, thanks | 22:04 |
| fungi | jeblair: clarkb: probably worth evaluating anywhere else we need to be using logrotate and double-checking that we do it consistently | 22:04 |
| jeblair | started mergers and reloading gate queue | 22:04 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Use WatchedFileHandler to avoid copytruncate https://review.openstack.org/95935 | 22:04 |
| sdague | mordred: you about to explain what you are running into on the feature matrix? | 22:05 |
| Alex_Gaynor | Will any CRs that had pending zuul stuff be auto-restarted? | 22:05 |
| *** nati_ueno has quit IRC | 22:05 | |
| jeblair | Alex_Gaynor: things that were in the check or gate queues at shutdown will be restored; but not changes during the downtime | 22:05 |
| clarkb | we should probably test that though, the current setup is tested in logstash workers | 22:05 |
| Alex_Gaynor | jeblair: k, thanks. | 22:05 |
| *** otherwiseguy has quit IRC | 22:06 | |
| *** yamahata has joined #openstack-infra | 22:07 | |
| fungi | clarkb: looks like the only other place we're obviously using that pattern is the log_processor module | 22:07 |
| jeblair | #status ok Zuul is started and processing changes that were in the queue when it was stopped. Changes uploaded or approved since then will need to be re-approved or rechecked. | 22:07 |
| openstackstatus | jeblair: sending ok | 22:07 |
| fungi | clarkb: so probably worth fixing that one too | 22:07 |
| clarkb | fungi: yup, and we can test there first with less effort | 22:08 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Small fix to a method name https://review.openstack.org/95730 | 22:08 |
| *** ildikov has joined #openstack-infra | 22:08 | |
| *** thedodd has quit IRC | 22:08 | |
| fungi | stuff looks like it's testing/merging, so i'm gonna vanish for a bit | 22:09 |
| mordred | sdague: one sec | 22:09 |
| jeblair | fungi: enjoy rolling in your dough | 22:09 |
| -openstackstatus- NOTICE: Zuul is started and processing changes that were in the queue when it was stopped. Changes uploaded or approved since then will need to be re-approved or rechecked. | 22:09 | |
| fungi | indeed i shall | 22:09 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer Infrastructure | docs http://ci.openstack.org | bugs https://launchpad.net/openstack-ci/ | https://git.openstack.org/cgit/openstack-infra/config/tree/" | 22:09 | |
| *** wenlock_ has quit IRC | 22:10 | |
| openstackstatus | jeblair: finished sending ok | 22:11 |
| clarkb | ok I need to drive back to seattle before it gets too late | 22:12 |
| clarkb | jeblair: mordred anything you want me to do first? | 22:12 |
| jeblair | clarkb: drive safely! | 22:12 |
| *** gondoi is now known as zz_gondoi | 22:13 | |
| mordred | clarkb: nope. except for the drive safely. We just hit 6 infra core- I don't want to fall back down to 5... | 22:13 |
| mattoliverau | Morning! | 22:13 |
| *** gabriel-bezerra has joined #openstack-infra | 22:13 | |
| clarkb | ya I don't want that either | 22:13 |
| mordred | ola mattoliverau ! | 22:14 |
| gabriel-bezerra | Hi guys. Is there a way in DevStack's scripts to get the Apache or Ubuntu version that it is running on? | 22:14 |
| gabriel-bezerra | I'd like to check whether devstack is running on Ubuntu with Apache 2.2 or Apache 2.4, so I can configure the scripts with the right names. | 22:14 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Fix Unknown events in timeline https://review.openstack.org/95729 | 22:14 |
| anteaya | morning mattoliverau | 22:15 |
| mattoliverau | hola mordred and anteaya, have started reading scroll back, but as a cheat, anything interesting happen while I slept? :) | 22:17 |
| jeblair | fungi, clarkb: i think i found why zuul was stuck in that loop... | 22:19 |
| *** SumitNaiksatam has quit IRC | 22:19 | |
| jeblair | fungi, clarkb: the submit job packet for a jenkins "describe" job to update the description for a build was the one that timed out; i think it may have been the final description update and may have affected removing the reported change from the queue | 22:20 |
| anteaya | mattoliverau: well zuul just got restarted | 22:21 |
| anteaya | mattoliverau: that was interesting | 22:21 |
| anteaya | mattoliverau: that is the biggest most recent fire | 22:21 |
| jeblair | mattoliverau: we're seeing an unusually high number of incidences of zuul disconnecting from geard because geard is unresponsive. we've enabled an obscene amount of debugging to try to learn why. | 22:22 |
| mattoliverau | OK, thanks anteaya, so we may get people compaining about how long will it take for there changes to merge then :) | 22:22 |
| *** SumitNaiksatam has joined #openstack-infra | 22:23 | |
| anteaya | or people reporting that their patch isnt' being tested | 22:24 |
| anteaya | like jeblair said, due to zuul disconnecting from geard | 22:25 |
| jeblair | wow, it happened again. that didn't take long. | 22:25 |
| anteaya | 754 zuul results | 22:26 |
| anteaya | never seen it that high | 22:26 |
| mattoliverau | jeblair: so we might have finally seen a limit to the current geard infrastructure then. Could we throw in another geard server and start moving different type of servers on it, like the mergers etc. to lower the load? | 22:26 |
| *** jgrimm has quit IRC | 22:27 | |
| jeblair | mattoliverau: perhaps, but i'm not at all sure it's that simple. geard has long periods of gaps in its logs where it is receiving new connections but apparently not receiving gearman packets otherwise | 22:27 |
| jeblair | mattoliverau: we don't know what's happening during those periods, thus the logging increase | 22:27 |
| *** melwitt has quit IRC | 22:30 | |
| mattoliverau | jeblair: hmm, annoying, fair enough, recieving new connections but not getting packets.. is the network link saturated. If the debug logs aren't giving much away, and also have large gaps, maybe the packets aren't reaching the application layer. | 22:31 |
| anteaya | anything new showing up in the logs yet? | 22:31 |
| anteaya | or too early? | 22:31 |
| *** arnaud__ has quit IRC | 22:31 | |
| *** melwitt has joined #openstack-infra | 22:31 | |
| *** melwitt has quit IRC | 22:31 | |
| mattoliverau | jeblair: sorry, just thinking out loud | 22:31 |
| *** melwitt has joined #openstack-infra | 22:31 | |
| *** nati_ueno has joined #openstack-infra | 22:32 | |
| *** melwitt has quit IRC | 22:32 | |
| jhesketh | Morning | 22:32 |
| *** melwitt has joined #openstack-infra | 22:33 | |
| jeblair | we definitely have more data; trying to sort through it now | 22:33 |
| *** rfolco has quit IRC | 22:34 | |
| mattoliverau | jeblair: traffic on zuul, does seem a little abnormal on eth1, but this might be normal, or due to all the restarts: http://is.gd/hXo4Sv | 22:36 |
| anteaya | morning jhesketh | 22:36 |
| anteaya | jhesketh: zuul is unhappy due to geard issues | 22:37 |
| jhesketh | hmm, anything I can do to help? | 22:37 |
| anteaya | probably, try the backscroll from the last 90 minutes | 22:37 |
| anteaya | that should give you the tl;dr | 22:37 |
| *** marcoemorais has quit IRC | 22:38 | |
| *** marcoemorais has joined #openstack-infra | 22:38 | |
| *** Sukhdev has joined #openstack-infra | 22:38 | |
| *** marcoemorais has quit IRC | 22:38 | |
| *** marcoemorais has joined #openstack-infra | 22:39 | |
| anteaya | queue length 1180 results | 22:41 |
| anteaya | wow | 22:41 |
| *** nati_uen_ has joined #openstack-infra | 22:42 | |
| *** nati_ueno has quit IRC | 22:42 | |
| jeblair | i've started a tcpdump too | 22:42 |
| hemna | anyone familiar with the gerritlib/gerritbot code ? | 22:42 |
| anteaya | hemna: somewhat, what is on your mind? | 22:43 |
| hemna | anteaya, I'm trying to plug a local gerritbot into a locally installed gerrit. I'm getting json errors | 22:43 |
| hemna | anteaya, pastebin.com/1DtG3SRY | 22:43 |
| jeblair | geard does not log admin requests, which makes it a little tricky to determine if nodepool's interactions with it are having an effect. | 22:43 |
| anteaya | :/ | 22:44 |
| hemna | my local gerrit install is version 2.8.4 | 22:44 |
| hemna | not sure if there is something that has to be configured on the gerrit side to allow this ? | 22:44 |
| hemna | I'm using a non-admin user in my gerritbot config | 22:45 |
| anteaya | okay on local gerrit can you ssh and stream events? | 22:45 |
| anteaya | the error is about being unable to ssh | 22:45 |
| anteaya | can you do so manually? | 22:45 |
| hemna | I can ssh in manually | 22:45 |
| anteaya | or consuming the ssh stream | 22:45 |
| hemna | gerrit drops me out | 22:45 |
| hemna | saying interactive shells are disabled.... | 22:46 |
| anteaya | what do you mean, gerrit drops me out | 22:46 |
| anteaya | so you can't stream events via your shell? | 22:46 |
| anteaya | sounds like a gerrit permissions error | 22:46 |
| anteaya | you have to give your gerrit account permissions to read the stream | 22:46 |
| hemna | that's a gerrit config option ? | 22:47 |
| anteaya | in gerrit 2.8.4 stream events is limited | 22:47 |
| anteaya | not sure where it is set, zaro would know | 22:47 |
| *** arnaud has joined #openstack-infra | 22:47 | |
| anteaya | we made everyone able to stream events for our gerrit | 22:47 |
| anteaya | but that is a change from 2.4 to 2.8 | 22:47 |
| hemna | ok | 22:47 |
| *** changbl has quit IRC | 22:47 | |
| anteaya | everyone used to be able to read the stream in 2.4 | 22:47 |
| anteaya | now your account has to have permission to read the stream | 22:48 |
| *** zhiyan_ is now known as zhiyan | 22:48 | |
| jogo | chmouel: no paris for me | 22:49 |
| anteaya | no | 22:49 |
| anteaya | where are you jogo? | 22:49 |
| *** derekh_ has quit IRC | 22:51 | |
| *** jhesketh has quit IRC | 22:51 | |
| mordred | jeblair: is there anything I can do to help? | 22:52 |
| *** jhesketh_ has joined #openstack-infra | 22:52 | |
| *** dstanek is now known as dstanek_zzz | 22:52 | |
| *** jhesketh_ is now known as jhesketh | 22:52 | |
| zaro | hemna : i believe 'registered users' are allowed to view gerrit stream events. which means you must at least have a gerrit account. | 22:54 |
| jeblair | mordred: not atm... | 22:54 |
| jeblair | mordred, jhesketh, mattoliverau: https://etherpad.openstack.org/p/XrzCW0EARb | 22:54 |
| jeblair | those are interesting log entries i'm collecting | 22:54 |
| *** atiwari has quit IRC | 22:54 | |
| hemna | zaro, and I do. | 22:55 |
| jeblair | in the last two events, i've seen a correlation with nodepool commands | 22:55 |
| anteaya | zaro: that was in 2.4 | 22:55 |
| anteaya | zaro: in 2.8 they have to have express permissions | 22:56 |
| zaro | hemna, anteaya : look at global capabilities.. https://review.openstack.org/#/admin/projects/All-Projects,access | 22:56 |
| anteaya | fungi said that we gave them to all registered uses for our gerrit to save time with all the third party ci | 22:56 |
| *** msabramo has quit IRC | 22:56 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added sort parameters to API https://review.openstack.org/95959 | 22:56 |
| anteaya | yeah that is us | 22:56 |
| anteaya | hemna: do you have such a page for your gerrit? | 22:57 |
| anteaya | what do you see under global capabilities > stream events | 22:57 |
| *** zhiyan is now known as zhiyan_ | 22:58 | |
| jhesketh | jeblair: were there no geard logs corresponding to the submit job request? | 22:58 |
| jeblair | jhesketh: added; it is logged after zuul times out | 22:59 |
| *** signed8bit has quit IRC | 22:59 | |
| hemna | trying a different user to my local gerrit to see if that works (different privs) | 22:59 |
| *** andreaf has quit IRC | 22:59 | |
| jeblair | jhesketh: this time it was pretty close, previous incidences have it considerably later | 22:59 |
| jhesketh | what's the load on the machine like? | 22:59 |
| anteaya | hemna: kk | 22:59 |
| jeblair | jhesketh: cacti should tell you | 23:00 |
| anteaya | hemna: can you navigate to the All-Projects,access page in your gui for your gerrit? is that an option? | 23:00 |
| hemna | I can't with my normal user. | 23:01 |
| anteaya | who can? | 23:01 |
| jhesketh | jeblair: so it looks like the memory is full and the load is reasonably high | 23:04 |
| jhesketh | are you able to tell which process is chewing all the memory? | 23:04 |
| jeblair | jhesketh: there's 29G of ram free | 23:05 |
| jhesketh | lol, it's because I can't read graphs | 23:05 |
| jhesketh | ignore me | 23:05 |
| jeblair | :) | 23:05 |
| mattoliverau | jeblair: Zuul's connection errors all seem to relate to lo packets, tcpdump output looks like it is only showing eth* packets (i.e. nodepool), can you tcpdump -i lo to show loopback packets.. even though you would think lo packets wouldn't have any issues. | 23:07 |
| *** mwagner_lap has joined #openstack-infra | 23:08 | |
| *** msabramo has joined #openstack-infra | 23:09 | |
| mattoliverau | Loopback traffic did spike.. but that might just be zuul being reconnected (multiple times). | 23:10 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added sort parameters to API https://review.openstack.org/95959 | 23:10 |
| jeblair | mattoliverau: started; | 23:10 |
| jeblair | mattoliverau: if it happens again we'll have those logs | 23:10 |
| mattoliverau | nice | 23:11 |
| zzelle | hemna, look at https://review.openstack.org/#/admin/projects/All-Projects,access | 23:13 |
| *** yamahata has quit IRC | 23:16 | |
| openstackgerrit | A change was merged to openstack-infra/config: Publish api-site for DE and JA https://review.openstack.org/95451 | 23:16 |
| openstackgerrit | A change was merged to openstack-infra/config: Name the integrated queue https://review.openstack.org/95046 | 23:16 |
| *** rfolco has joined #openstack-infra | 23:17 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Log admin requests https://review.openstack.org/95964 | 23:18 |
| jeblair | i'm going to restart zuul with that change | 23:20 |
| jeblair | and it just happened again; this time the timed-out packet went over eth1 | 23:20 |
| mordred | jeblair: I support that decision | 23:20 |
| jeblair | er eth0 | 23:21 |
| jeblair | nevermind that's wrong | 23:21 |
| jeblair | anyway | 23:21 |
| jeblair | i'll checkout the lo tcpdumps after the restart | 23:21 |
| *** SumitNaiksatam has quit IRC | 23:23 | |
| *** otherwiseguy has joined #openstack-infra | 23:23 | |
| *** SumitNaiksatam has joined #openstack-infra | 23:24 | |
| *** david-lyle has quit IRC | 23:25 | |
| tchaypo | I hate to nag, but https://review.openstack.org/#/c/86746/ has been sitting waiting for some love for a long time. It has several +1s, a +2, and jenkins is happy... | 23:29 |
| jeblair | i've been updating the etherpad | 23:29 |
| jeblair | i think that's confirmation the status command took 34 seconds to run | 23:29 |
| *** michchap has joined #openstack-infra | 23:32 | |
| *** dstanek_zzz is now known as dstanek | 23:33 | |
| mattoliverau | jeblair: yup 35 seconds to run the admin request.. is it just me, or is that a rather long time? How long is the resulting status list? | 23:34 |
| anteaya | tchaypo: hey there | 23:34 |
| *** eharney has quit IRC | 23:35 | |
| *** wenlock has quit IRC | 23:36 | |
| jeblair | mattoliverau: 244901 bytes and it typically takes a few tenths of a second | 23:37 |
| jeblair | (it's not super efficient, but it's not _that_ inefficient) | 23:37 |
| mattoliverau | lol | 23:38 |
| mordred | jeblair, clarkb: btw- the hp folks looking in to the 1.1 issues have done some digging and seem to think the errors are actually related to the nova schedule | 23:38 |
| mordred | scheduler | 23:38 |
| jeblair | mordred: i love helping the project! | 23:38 |
| mordred | jeblair: I provided them with the uuid for one of the errors in the log and they ran $things and found inconclusive evidence that points in that direction | 23:38 |
| mordred | jeblair: I'll let you know more things when I learn them | 23:39 |
| mordred | jeblair: it's entirely possible that it has something do to with requesting that many 30G nodes all at once and is a real scheduler issue | 23:39 |
| mordred | as in - like - the schuduler is unable to schedule $blah | 23:39 |
| jeblair | mordred: let's increase the large ops value! | 23:40 |
| mordred | :) | 23:40 |
| mordred | jeblair: or, I mean, say it with me ... we could just fix this with AFS | 23:40 |
| * mordred has no idea how ... | 23:40 | |
| jeblair | mordred: heh | 23:40 |
| jeblair | mordred: the zuul/geard problem seems to be happening every few minutes now | 23:40 |
| jeblair | so i think we're dead until we fix it | 23:41 |
| mordred | jeblair: awesome | 23:41 |
| jeblair | it has occurred to me that all the extra logging could be an issue | 23:41 |
| mordred | oh. well yes. it could be | 23:41 |
| *** gokrokve has joined #openstack-infra | 23:41 | |
| jeblair | i've been running the status command manually and just saw it take 41 seconds | 23:41 |
| mordred | jeblair: how much logging does it write to perform that command? | 23:42 |
| jeblair | mordred: about 4 lines | 23:42 |
| mordred | oh. well, that's not terrible. | 23:42 |
| mordred | does the logger serialize globally? | 23:42 |
| mrodden | i think it does | 23:42 |
| mrodden | we have had issues with it and eventlet i know... | 23:43 |
| *** dstanek is now known as dstanek_zzz | 23:43 | |
| mrodden | comstud said it tries to acquire a lock on the file stream or something? | 23:43 |
| anteaya | hemna: did we lose you? | 23:43 |
| mrodden | python logging | 23:43 |
| *** fbo has quit IRC | 23:44 | |
| *** unicell has quit IRC | 23:45 | |
| anteaya | morganfainberg: yay | 23:47 |
| morganfainberg | anteaya, hi :) | 23:47 |
| jeblair | there's only about 15 log entries between the start/end of handling the status command | 23:47 |
| mattoliverau | jeblair: good call on the extra logging, extra logging = more time = more timeouts.. Still something is timing out causeing zuul to disconnect. So the output of the status command doesn't seem anything out of the ordinary then, it isn't extra large or anything? | 23:48 |
| jeblair | so even if the logging is having an impact, it doesn't seem to be doing so during the time period when it's dealing with the admin command | 23:48 |
| mordred | jeblair: hrm. that seems not huge - I wouldn't expect it to take 41 seconds of blocking to write 15 lines | 23:48 |
| mgagne | anyone ever encountered a redirect loop with gerrit after login? (not on review.o.o) | 23:48 |
| anteaya | morganfainberg: you need to do a gitdm patch: https://review.openstack.org/#/q/project:openstack-infra/gitdm,n,z | 23:48 |
| morganfainberg | anteaya, correct it looked like that was in config? or is that in gitdm directly? | 23:49 |
| anteaya | in gitdm directly | 23:49 |
| mgagne | nvm, found it | 23:49 |
| morganfainberg | anteaya, hmm. | 23:49 |
| anteaya | it has its own repo | 23:49 |
| anteaya | openstack-infra/gitdm | 23:49 |
| anteaya | you have a few samples to choose from | 23:49 |
| mgagne | I had the great idea to set a secure cookie on an insecure url | 23:49 |
| anteaya | nice | 23:50 |
| morganfainberg | anteaya, right. but i saw other commits about gitdm in config :P so i was confused where to put it. i'll get that posted today once i get my keystone spec (well 1st spec) written up | 23:50 |
| anteaya | kk | 23:50 |
| anteaya | BadCub01_: there you are | 23:51 |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add tox "coverage" target https://review.openstack.org/87382 | 23:51 |
| jeblair | mordred: i'd like to propose the following "go home" solution: | 23:52 |
| jeblair | mordred: set gear log levels to WARNING and the zuul gear timeout to 300 | 23:53 |
| *** marcoemorais has quit IRC | 23:53 | |
| jeblair | mordred: and then work on a way to track this down out of production | 23:53 |
| *** arnaud has quit IRC | 23:55 | |
| mordred | jeblair: ++ | 23:56 |
| mordred | jeblair: I support that solution | 23:56 |
| mattoliverau | jeblair: +1. I think you deserve some sleep! Thanks for staying at it so long. I'm sorry I can't really get in there and help so it isn't all on you in the middle of the night. | 23:56 |
| jeblair | i think tomorrow i'll try to reproduce locally; i have managed to scale well past our environment on my workstation; i'll try to do that and see if i can get a really slow status command | 23:57 |
| *** lcheng_ has quit IRC | 23:58 | |
| mattoliverau | jeblair: let me know if there is anything I can do. If there are zuul issues during my day, I'll alert the devs who come in channel to complain :) | 23:58 |
| openstackgerrit | Mark Sturdevant proposed a change to openstack/requirements: Remove hp3parclient from global-requirements https://review.openstack.org/95971 | 23:59 |
| jhesketh | +1 | 23:59 |
| jhesketh | also happy to help if I can | 23:59 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Reduce gearman logging level https://review.openstack.org/95972 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
