| *** hogepodge has quit IRC | 00:00 | |
| clarkb | the only way I can see us working around it is to require people to quote when using tox, eg tox -efoo 'my args here' | 00:02 |
|---|---|---|
| anteaya | boo | 00:02 |
| sdague | fungi / anteaya: I love you guys :) | 00:04 |
| *** amcrn has quit IRC | 00:04 | |
| anteaya | sdague: ah thanks | 00:05 |
| anteaya | I love you too | 00:05 |
| anteaya | how is Korea? | 00:05 |
| clarkb | https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror commented upstream | 00:06 |
| clarkb | fungi: where were we requiring oslo.messaging as a tarball? I checked nova and glance but they seem to use the normal requirement | 00:07 |
| clarkb | fungi: maybe we are already patched against that in most places? | 00:07 |
| fungi | clarkb: i hope so | 00:07 |
| fungi | dev-reqs may still have the tarball link? | 00:07 |
| clarkb | yeah dev-reqs will need to be rm'd | 00:09 |
| sdague | it's good, about to head back over, which means I'm about to be disconnected for the next many hours | 00:09 |
| clarkb | sdague: you were there for just a cuople days? wow | 00:09 |
| sdague | sorry, back over to the samsung campus | 00:09 |
| sdague | but yes only a few days as well | 00:10 |
| sdague | flying back friday | 00:10 |
| anteaya | how is the samsung campus? | 00:10 |
| anteaya | anything note worthy, that you are able to share? | 00:11 |
| sdague | kind of amazingly huge | 00:12 |
| clarkb | anteaya: fungi: how much time do you think we need to give to unpin things? A week from friday? | 00:12 |
| clarkb | I am wary of interferring with proposal freezes, but seriously we have to get better at this | 00:13 |
| *** david-lyle has joined #openstack-infra | 00:13 | |
| anteaya | sounds like a reasonable timeline to me | 00:13 |
| anteaya | clarkb: any timeline you propose is sufficent for those paying attention | 00:13 |
| fungi | clarkb: i think that's plenty. people will take as long as we let them, so best to keep that constrained | 00:13 |
| anteaya | and insufficient for those not | 00:13 |
| anteaya | you are just deciding what day everything breaks for the folks not paying attention | 00:13 |
| fungi | yep | 00:14 |
| anteaya | and the weekend or a friday sounds good to me | 00:14 |
| clarkb | https://etherpad.openstack.org/p/tox-virtualenv-pip-upgrades | 00:15 |
| clarkb | how does that sound? | 00:15 |
| clarkb | jeblair: ^ before I send anything want to make sure you are on board | 00:15 |
| lifeless | clarkb: jeblair: so on node deletes; I'm waiting on an updated log to decide what to do next | 00:15 |
| *** ryanpetrello has quit IRC | 00:16 | |
| clarkb | lifeless: ok | 00:16 |
| anteaya | sdague: *amazingly huge* kind of like the Atlantic? | 00:17 |
| pleia2 | well, we went from 266 to 245 bugs, and triaged 45 new ones, good day :) | 00:17 |
| *** sarob has joined #openstack-infra | 00:18 | |
| jeblair | clarkb: reading | 00:18 |
| *** lcheng_ has joined #openstack-infra | 00:19 | |
| *** thedodd has quit IRC | 00:19 | |
| *** sarob_ has quit IRC | 00:19 | |
| *** paul-- has joined #openstack-infra | 00:21 | |
| *** CaptTofu has joined #openstack-infra | 00:24 | |
| *** matsuhashi has joined #openstack-infra | 00:25 | |
| anteaya | pleia2: yay, thanks for taking charge of bug day | 00:25 |
| *** CaptTofu has quit IRC | 00:28 | |
| jeblair | pleia2: thank you! | 00:29 |
| *** slong has quit IRC | 00:30 | |
| pleia2 | sure thing | 00:32 |
| *** pcrews has quit IRC | 00:32 | |
| StevenK | pleia2: That's a massive stack of disks. Are you planning to store the internet at home? :-) | 00:33 |
| *** atiwari has quit IRC | 00:33 | |
| pleia2 | StevenK: haha, it's only 2! | 00:33 |
| pleia2 | woke up to disk errors on monday, doing raid1 on my desktop | 00:34 |
| *** zehicle_at_dell has quit IRC | 00:35 | |
| *** zehicle_at_dell has joined #openstack-infra | 00:36 | |
| *** sarob has quit IRC | 00:39 | |
| *** jcoufal has quit IRC | 00:42 | |
| *** jcoufal has joined #openstack-infra | 00:42 | |
| *** kraman1 has quit IRC | 00:42 | |
| *** kraman has joined #openstack-infra | 00:43 | |
| *** mgagne has quit IRC | 00:44 | |
| clarkb | jeblair: additional edits have been made | 00:46 |
| *** kraman has quit IRC | 00:48 | |
| fungi | clarkb: out of curiosity, what's the difference between --test-args and --testr-args? (is the latter something specific to testr?) | 00:48 |
| *** smurugesan has quit IRC | 00:48 | |
| clarkb | --testr-args is the actual thing did I tyop somewhere? | 00:48 |
| clarkb | pbr makes it a thing | 00:49 |
| jeblair | clarkb: i don't think we should ask devs to use this | 00:49 |
| jeblair | clarkb: i won't. | 00:49 |
| jeblair | clarkb: maybe we should ask mordred about the feasibility of making tox more user friendly for this | 00:50 |
| jeblair | (assuming that hasn't already been attempted) | 00:50 |
| clarkb | jeblair: we can do that too | 00:50 |
| *** gokrokve_ has joined #openstack-infra | 00:50 | |
| clarkb | jeblair: so far https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror is as far as I have gotten upstream | 00:50 |
| clarkb | jeblair: upstream makes it sound like they won't be fixing it but perhaps the quote thing can be handled differently | 00:51 |
| fungi | clarkb: my brain may have just thought it saw --test-args, because i don't see it any more | 00:51 |
| clarkb | fungi: I fixed one case of it | 00:51 |
| fungi | ah | 00:51 |
| *** sarob has joined #openstack-infra | 00:52 | |
| *** yamahata has joined #openstack-infra | 00:53 | |
| *** gokrokve has quit IRC | 00:53 | |
| *** reed has quit IRC | 00:54 | |
| clarkb | jeblair: and I have responded to my own comment upstream | 00:56 |
| *** sarob has quit IRC | 00:56 | |
| jeblair | clarkb: the troubling thing about that bug report is that it does not identify what has changed | 00:57 |
| *** nosnos has joined #openstack-infra | 00:57 | |
| *** gokrokve_ has quit IRC | 00:57 | |
| jeblair | clarkb: that makes it unclear whether offbyone understands the problem | 00:57 |
| clarkb | jeblair: good point | 00:58 |
| fungi | the commit messages around the posargs changes talked about supposed improvements... finding those again | 00:58 |
| clarkb | though offbyone apparently write a bunch of changes | 00:58 |
| anteaya | offbyone replied to you clarkb | 00:59 |
| fungi | clarkb: it sounds like he's saying tox is not meant to be used, and that's by design | 01:00 |
| anteaya | that's what I got from it too | 01:00 |
| anteaya | echoes of the *mark* is not to be used from earlier this afternoon | 01:01 |
| clarkb | ya I am having a hard time writing a response :) | 01:01 |
| *** nosnos has quit IRC | 01:01 | |
| *** nosnos has joined #openstack-infra | 01:01 | |
| fungi | perhaps this means we need to have tox call a wrapper which picks posargs up from a secondary file/pipe and hands those to testr | 01:02 |
| jeblair | clarkb: one way might be to focus on the regression | 01:02 |
| *** lcheng_ has quit IRC | 01:02 | |
| fungi | from the changelog, the goal seems to have been to "rework and simplify 'commands' parsing and in particular posargs substitutions to avoid various win32/posix related quoting issues." | 01:04 |
| jeblair | basically anything that doesn't follow the template "run python27 test.name.goes.here" is silly because it's just asking developers to type extra meaningless characters and we shouldn't do it. | 01:04 |
| * anteaya thinks about the obligatory 22 placesetting of china one *must* possess but must never ever use, just figure out how to dole them out after death | 01:05 | |
| clarkb | jeblair: I agree | 01:05 |
| *** shivh has quit IRC | 01:06 | |
| *** VijayT has quit IRC | 01:07 | |
| lifeless | is netifaces still a problem child mirror wise? | 01:08 |
| lifeless | run-mirror completed for me but didn't include netifaces... | 01:08 |
| jeblair | clarkb: good clear response (one that starts with "Given") | 01:09 |
| lifeless | oh | 01:09 |
| lifeless | run-mirror it didn't pull requirements | 01:09 |
| lifeless | for some reason | 01:09 |
| * lifeless will poke at it | 01:10 | |
| fungi | lifeless: run-mirror tells pip to allow netifaces to be insecure and external (along with a couple other packages) | 01:12 |
| fungi | clarkb: is this where things started to go wrong? https://bitbucket.org/hpk42/tox/commits/88a503e7e5aefe509be8f6c3108a84baa20c3c26#Ltests/test_config.pyT1091 | 01:13 |
| clarkb | fungi: oh man is it unquoting {posargs} first then applying the transform? | 01:14 |
| fungi | er, might have meant T710 | 01:14 |
| fungi | _config.pyT710 | 01:15 |
| fungi | but yeah, looks like it | 01:15 |
| fungi | strip quotes, then combine | 01:15 |
| clarkb | fungi: ya, so I think that is a legit bug | 01:15 |
| *** ryanpetrello has joined #openstack-infra | 01:15 | |
| clarkb | it should expand then strip quotes | 01:15 |
| *** mestery has quit IRC | 01:15 | |
| clarkb | or why not leave quotes alone? | 01:15 |
| fungi | because EWINDOWS | 01:15 |
| *** mrodden has quit IRC | 01:16 | |
| fungi | the commit message (and such a descriptive one) says "fix windows parsing/escaping" | 01:17 |
| *** cody-somerville has quit IRC | 01:17 | |
| fungi | er, "fix parsing/escaping bugs on windows32" | 01:17 |
| anteaya | but why break parsing for linux by doing so? | 01:17 |
| anteaya | is there a better way to accomplish both goals? | 01:18 |
| clarkb | anteaya: either it was a mistake or offbyone did actually intend on breaking it (see his comment on how the behavior is supposed to be different) | 01:18 |
| *** sandywalsh has quit IRC | 01:19 | |
| * anteaya looks | 01:19 | |
| anteaya | sorry, how do I see offbyone's comment? | 01:20 |
| *** melwitt has quit IRC | 01:20 | |
| clarkb | anteaya: https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror | 01:20 |
| fungi | anteaya: where he says "posargs is not supposed to be a quoted insertion into the commands in the tox command list, it's supposed to be an in-place insertion into the command. The distinction is perhaps a subtle one, but important. The placeholder is there to indicate the point in the list of command arguments into which we should insert the list of additional command line positional arguments before | 01:21 |
| fungi | constructing the complete command." | 01:21 |
| anteaya | oh that one, sorry, I was looking for it in the commit | 01:21 |
| clarkb | they are being too smart. | 01:22 |
| clarkb | which is my cue to head home | 01:22 |
| *** thuc_ has quit IRC | 01:23 | |
| *** smurugesan has joined #openstack-infra | 01:23 | |
| *** thuc has joined #openstack-infra | 01:24 | |
| anteaya | this is the issue that patch was addressing: https://bitbucket.org/hpk42/tox/issue/128/install_command-doesnt-interpolate | 01:24 |
| *** smurugesan has quit IRC | 01:25 | |
| anteaya | just for comparison | 01:27 |
| anteaya | the bug it was addressing had 0 votes, and 2 watchers | 01:27 |
| anteaya | the issue it has created has 0 votes, and 6 watchers | 01:27 |
| clarkb | jeblair: can I be a bit selfish and ask you to prioritize https://review.openstack.org/#/c/69442/ it will fix several problems with our slave builds | 01:27 |
| jeblair | clarkb: marked for review tomorrow when i have brains | 01:28 |
| *** thuc has quit IRC | 01:28 | |
| *** markmcclain has quit IRC | 01:30 | |
| clarkb | awesome thanks | 01:30 |
| fungi | clarkb: yeah, looks like that commit got further reworked in https://bitbucket.org/hpk42/tox/commits/f15199a9ec78e052b9f9513a34e0e5768e8affdd | 01:30 |
| *** changbl has joined #openstack-infra | 01:31 | |
| fungi | _processcommand() grew a lot more logic | 01:31 |
| *** CaptTofu has joined #openstack-infra | 01:32 | |
| anteaya | it looks to me like their tests are targeted for os=win32 | 01:33 |
| anteaya | but it doesn't look like the code is only targeted for win32 | 01:33 |
| *** VijayT has joined #openstack-infra | 01:35 | |
| fungi | anteaya: i think they're testing whether \ gets left alone on windows (since it's a directory separator not an escape) | 01:35 |
| anteaya | ah | 01:36 |
| anteaya | thank you | 01:36 |
| anteaya | so I must be missing the part where they test this new code then | 01:37 |
| * anteaya moseys over to tests/test_config.py | 01:37 | |
| jeblair | anteaya: lol :) | 01:37 |
| *** packet has joined #openstack-infra | 01:41 | |
| *** ryanpetrello has quit IRC | 01:42 | |
| anteaya | well I'm probably wrong, since I usually am, but I am not seeing the test for _processcommand within the IniReader function: https://bitbucket.org/hpk42/tox/src/b0360a54ab368ef428c7f83601ba6b64f6fec64f/tests/test_config.py | 01:44 |
| anteaya | I see the IniReader function used a lot to set up the tests | 01:44 |
| anteaya | but I can't see the test for it | 01:44 |
| * anteaya is prepared to be told how wrong she is | 01:44 | |
| anteaya | sorry IniReader class | 01:46 |
| *** malini_afk is now known as malini | 01:46 | |
| *** pcrews has joined #openstack-infra | 01:47 | |
| *** sandywalsh has joined #openstack-infra | 01:48 | |
| *** amcrn has joined #openstack-infra | 01:48 | |
| anteaya | wait I found something: https://bitbucket.org/hpk42/tox/src/b0360a54ab368ef428c7f83601ba6b64f6fec64f/tests/test_config.py?at=default#cl-1215 | 01:49 |
| anteaya | woohoo https://review.openstack.org/#/c/67537/ is about to be merged | 01:55 |
| anteaya | it made it | 01:56 |
| anteaya | yay dansmith | 01:56 |
| anteaya | and https://review.openstack.org/#/c/66501/ went in too | 01:56 |
| anteaya | markmcclain has a list of high priority patches he is working on first | 01:57 |
| *** sarob has joined #openstack-infra | 01:57 | |
| anteaya | 6 in post, 2 of them neutron | 01:58 |
| anteaya | yay | 01:58 |
| *** sarob has quit IRC | 02:03 | |
| *** ktbenton has joined #openstack-infra | 02:06 | |
| *** sarob has joined #openstack-infra | 02:08 | |
| *** sarob has quit IRC | 02:12 | |
| *** freyes has joined #openstack-infra | 02:12 | |
| *** nati_ueno has quit IRC | 02:14 | |
| *** amotoki has joined #openstack-infra | 02:14 | |
| *** AaronGr is now known as AaronGr_Zzz | 02:21 | |
| *** jasondotstar has joined #openstack-infra | 02:22 | |
| *** thuc has joined #openstack-infra | 02:24 | |
| *** dims_ has quit IRC | 02:31 | |
| HenryG | fungi: ping | 02:32 |
| *** thuc has quit IRC | 02:32 | |
| lyxus | how do yo test a patch in general . I would like to use the cherry-pick option but my files are getting overwritten. My current option is to add my fetch at the setup_neutron function | 02:33 |
| *** dims_ has joined #openstack-infra | 02:34 | |
| *** coolsvap_away has quit IRC | 02:34 | |
| *** dstanek has joined #openstack-infra | 02:36 | |
| anteaya | lyxus: can you expand? | 02:36 |
| anteaya | lyxus: which patch would you like to test and what command are you trying to run to test it? | 02:37 |
| *** rakhmerov has quit IRC | 02:39 | |
| lyxus | anteaya, For the 3rd party testing, I am doing patch verification. At the moment, when I want to try a neutron patch. I am editing the lib/neutron and adding the cherry=pick after function install_neutron() | 02:41 |
| *** gokrokve has joined #openstack-infra | 02:41 | |
| clarkb | lyxus: have you seen devstack-gate? it does everything you need | 02:41 |
| lyxus | Then I run a stack.sh. I was thinking if there was no other (smarter) way | 02:41 |
| StevenK | lyxus: Ah, that was you posting on openstack-dev recently? | 02:41 |
| lyxus | clarkb, Nope | 02:42 |
| *** jcoufal has quit IRC | 02:42 | |
| clarkb | lyxus: https://git.openstack.org/cgit/openstack-infra/devstack-gate | 02:42 |
| lyxus | StevenK, Yes the post on the -dev was more toward the speed | 02:42 |
| StevenK | lyxus: I'd certainly suggest you look into devstack-gate, since that means you can also automate it | 02:43 |
| *** pcrews has quit IRC | 02:43 | |
| *** sarob has joined #openstack-infra | 02:43 | |
| *** jcoufal has joined #openstack-infra | 02:43 | |
| lyxus | clarkb, StevenK , Ok reading it now | 02:43 |
| *** sarob has quit IRC | 02:45 | |
| *** sarob has joined #openstack-infra | 02:45 | |
| *** sarob_ has joined #openstack-infra | 02:47 | |
| fungi | HenryG: sort of around--what's up? | 02:49 |
| *** AlexF has joined #openstack-infra | 02:49 | |
| *** oubiwann_ has joined #openstack-infra | 02:50 | |
| *** sarob has quit IRC | 02:50 | |
| HenryG | fungi: remember I asked some time back if there is a way to identify CI bots in the email headers from gerrit? Just wondering if anything has happened on that front. | 02:50 |
| *** AlexF has quit IRC | 02:52 | |
| *** CaptTofu has quit IRC | 02:53 | |
| fungi | HenryG: not yet. we've got the beginnings of a set of requirements in place now for third-party test accounts, and i think a consistent name prefix or suffix would be a good candidate to add to that before we notify the existing accounts of required changes... http://git.openstack.org/cgit/openstack-infra/config/tree/doc/source/third_party.rst#n17 | 02:53 |
| fungi | the display names for those accounts are basically under the control of infrastructure and don't change the way they authenticate, so should be fairly simple to introduce en masse should we find consensus around that idea | 02:54 |
| HenryG | fungi: thanks! happy to know things are trundling forward. :) | 02:54 |
| *** CaptTofu has joined #openstack-infra | 02:55 | |
| anteaya | fungi: are you still getting private emails from people about 3rd party testing? | 02:56 |
| anteaya | I haven't had time to talk to Mark about it, but I thought about it and if people are emailing you in private, how is he to know about it and respond? | 02:57 |
| anteaya | or anyone else for that matter | 02:57 |
| *** jamielennox is now known as jamielennox|away | 02:57 | |
| *** sarob_ has quit IRC | 02:57 | |
| fungi | anteaya: it happens from time to time, so i generally cc the -infra ml when responding to them | 02:57 |
| fungi | hoping they might get the hint | 02:57 |
| anteaya | okay thanks | 02:58 |
| anteaya | hopefully | 02:58 |
| *** sarob has joined #openstack-infra | 02:58 | |
| anteaya | I haven't seen anything of note lately, if there is something I should see, do ping | 02:58 |
| *** slong has joined #openstack-infra | 02:58 | |
| fungi | it was mainly just a couple of previously disabled accounts whose owners were private-messaging me in irc or e-mailing me directly about getting reinstated, but it's been a week or two since that's happened | 02:59 |
| *** jcoufal has quit IRC | 02:59 | |
| anteaya | okay | 02:59 |
| anteaya | 50% in the neutron check jobs so far | 03:00 |
| clarkb | since the tox thing seems to be a case of ugh, I am probably going to test my zmq plugin change, merge that if it works, do a release of the zmq plugin and upgrade that everywhere next | 03:00 |
| anteaya | now I have just been rechecking everything so perhaps some of them were still to be worked on | 03:00 |
| anteaya | so better, but still a ways to go | 03:00 |
| clarkb | also, wheels. I should figure out if we can enable wheels without pip 1.5 | 03:00 |
| anteaya | looking forward to bug fixes next | 03:01 |
| fungi | clarkb: we need to add --use-wheel in a lot of places for that to work, i think | 03:01 |
| anteaya | clarkb: glad you have a direction that takes you away from the ugh | 03:01 |
| *** sarob has quit IRC | 03:02 | |
| *** freyes has quit IRC | 03:04 | |
| *** morganfainberg is now known as morganfainberg_Z | 03:04 | |
| *** pcrews has joined #openstack-infra | 03:05 | |
| *** ianw has quit IRC | 03:08 | |
| *** ianw has joined #openstack-infra | 03:08 | |
| *** gokrokve has quit IRC | 03:10 | |
| *** malini is now known as malini_afk | 03:12 | |
| *** gokrokve has joined #openstack-infra | 03:13 | |
| *** rakhmerov has joined #openstack-infra | 03:15 | |
| *** amcrn has quit IRC | 03:25 | |
| *** jamielennox|away is now known as jamielennox | 03:27 | |
| *** mriedem has quit IRC | 03:27 | |
| *** mriedem has joined #openstack-infra | 03:28 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/config: Change mysql package for Fedora >= 19 for bare-precise https://review.openstack.org/70152 | 03:30 |
| *** paul-- has quit IRC | 03:30 | |
| *** packet has quit IRC | 03:31 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/config: Change mysql package for Fedora >= 19 for bare-precise https://review.openstack.org/70152 | 03:31 |
| *** sarob has joined #openstack-infra | 03:33 | |
| *** lcheng_ has joined #openstack-infra | 03:34 | |
| *** hcc has joined #openstack-infra | 03:42 | |
| *** mriedem has quit IRC | 03:55 | |
| *** pcrews has quit IRC | 03:56 | |
| openstackgerrit | Evgeny Fadeev proposed a change to openstack-infra/askbot-theme: fixed css for the search drop menu https://review.openstack.org/71186 | 04:00 |
| *** jhesketh__ has quit IRC | 04:01 | |
| *** hcc is now known as hdd_ | 04:02 | |
| *** harlowja is now known as harlowja_away | 04:03 | |
| *** jhesketh__ has joined #openstack-infra | 04:04 | |
| openstackgerrit | A change was merged to openstack-infra/askbot-theme: fixed css for the search drop menu https://review.openstack.org/71186 | 04:05 |
| *** sarob has quit IRC | 04:07 | |
| *** sarob has joined #openstack-infra | 04:08 | |
| *** CaptTofu has quit IRC | 04:09 | |
| *** CaptTofu has joined #openstack-infra | 04:09 | |
| *** paul-- has joined #openstack-infra | 04:10 | |
| *** paul-- has quit IRC | 04:11 | |
| *** paul-- has joined #openstack-infra | 04:11 | |
| *** sarob has quit IRC | 04:12 | |
| *** CaptTofu has quit IRC | 04:14 | |
| *** lcheng_ has quit IRC | 04:16 | |
| *** paul--_ has joined #openstack-infra | 04:17 | |
| *** paul-- has quit IRC | 04:18 | |
| *** jd__ has quit IRC | 04:21 | |
| *** jd__ has joined #openstack-infra | 04:22 | |
| *** jkt has quit IRC | 04:22 | |
| *** jkt has joined #openstack-infra | 04:22 | |
| *** jkt has joined #openstack-infra | 04:22 | |
| *** exz has quit IRC | 04:30 | |
| *** exz has joined #openstack-infra | 04:31 | |
| *** bhuvan has joined #openstack-infra | 04:33 | |
| *** rakhmerov has quit IRC | 04:34 | |
| *** lcheng_ has joined #openstack-infra | 04:35 | |
| *** gokrokve has quit IRC | 04:36 | |
| *** ArxCruz has quit IRC | 04:36 | |
| *** rcleere has joined #openstack-infra | 04:38 | |
| *** gokrokve has joined #openstack-infra | 04:40 | |
| *** nicedice has quit IRC | 04:42 | |
| *** nicedice has joined #openstack-infra | 04:43 | |
| *** paul--_ has quit IRC | 04:44 | |
| *** paul-- has joined #openstack-infra | 04:45 | |
| openstackgerrit | Bhuvan Arumugam proposed a change to openstack-infra/jeepyb: Split config between yaml and ini https://review.openstack.org/71188 | 04:46 |
| *** slong has quit IRC | 04:53 | |
| *** paul-- has quit IRC | 04:54 | |
| *** oubiwann_ has quit IRC | 04:58 | |
| *** marun has quit IRC | 05:02 | |
| *** marun has joined #openstack-infra | 05:03 | |
| *** rcleere has quit IRC | 05:05 | |
| *** lcheng_ has quit IRC | 05:08 | |
| *** lcheng_ has joined #openstack-infra | 05:08 | |
| *** sarob has joined #openstack-infra | 05:09 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Send swift upload instructions to workers https://review.openstack.org/68297 | 05:09 |
| *** shivh has joined #openstack-infra | 05:11 | |
| *** shivh has quit IRC | 05:14 | |
| *** sarob has quit IRC | 05:15 | |
| bhuvan | team, a quick question. what's the right way to test fixes to jeepyb? | 05:15 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Add a slave-script to Jenkins for pushing to swift https://review.openstack.org/71189 | 05:18 |
| *** jcooley_ has quit IRC | 05:19 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Add a slave-script to Jenkins for pushing to swift https://review.openstack.org/71189 | 05:20 |
| *** nicedice has quit IRC | 05:23 | |
| *** gokrokve has quit IRC | 05:24 | |
| *** rakhmerov has joined #openstack-infra | 05:30 | |
| *** gokrokve has joined #openstack-infra | 05:31 | |
| *** igor__ has joined #openstack-infra | 05:37 | |
| *** mestery has joined #openstack-infra | 05:38 | |
| *** mestery has quit IRC | 05:38 | |
| *** bhuvan has quit IRC | 05:38 | |
| *** mestery has joined #openstack-infra | 05:39 | |
| *** markmcclain has joined #openstack-infra | 05:41 | |
| *** mestery_ has joined #openstack-infra | 05:44 | |
| *** markmcclain has quit IRC | 05:45 | |
| *** mestery has quit IRC | 05:46 | |
| *** gokrokve has quit IRC | 05:46 | |
| *** gokrokve has joined #openstack-infra | 05:47 | |
| *** jcooley_ has joined #openstack-infra | 05:48 | |
| *** nati_ueno has joined #openstack-infra | 05:51 | |
| *** gokrokve has quit IRC | 05:51 | |
| *** paul-- has joined #openstack-infra | 05:53 | |
| *** lcheng_ has quit IRC | 05:56 | |
| *** lcheng_ has joined #openstack-infra | 05:56 | |
| *** mestery_ has quit IRC | 06:00 | |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Image logs accessible via HTTP https://review.openstack.org/71190 | 06:01 |
| *** weshay has quit IRC | 06:02 | |
| *** hdd_ has quit IRC | 06:03 | |
| *** e0ne has joined #openstack-infra | 06:04 | |
| *** emagana has joined #openstack-infra | 06:12 | |
| *** sarob has joined #openstack-infra | 06:12 | |
| *** sarob has quit IRC | 06:17 | |
| *** igor__ has quit IRC | 06:21 | |
| *** miqui has quit IRC | 06:22 | |
| *** jasondotstar has quit IRC | 06:22 | |
| *** e0ne has quit IRC | 06:26 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Update openstack-doc-tools to 0.5 https://review.openstack.org/69685 | 06:29 |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Image logs accessible via HTTP https://review.openstack.org/71190 | 06:38 |
| *** odyssey4me has joined #openstack-infra | 06:39 | |
| *** vogxn has joined #openstack-infra | 06:41 | |
| *** mrda is now known as mrda_away | 06:44 | |
| *** VijayT has quit IRC | 06:44 | |
| *** dstanek has quit IRC | 06:45 | |
| *** markmcclain has joined #openstack-infra | 06:50 | |
| *** afazekas has joined #openstack-infra | 06:54 | |
| *** rakhmerov has quit IRC | 06:55 | |
| *** lcheng_ has quit IRC | 06:56 | |
| *** yolanda has quit IRC | 07:00 | |
| *** boris-42_ has quit IRC | 07:05 | |
| *** nati_ueno has quit IRC | 07:05 | |
| *** nati_ueno has joined #openstack-infra | 07:06 | |
| *** uvirtbot has joined #openstack-infra | 07:08 | |
| *** coolsvap has joined #openstack-infra | 07:09 | |
| *** jcooley_ has quit IRC | 07:12 | |
| *** vogxn has quit IRC | 07:12 | |
| *** jcooley_ has joined #openstack-infra | 07:12 | |
| *** nati_ueno has quit IRC | 07:14 | |
| *** nati_ueno has joined #openstack-infra | 07:14 | |
| *** nati_ueno has quit IRC | 07:15 | |
| *** sarob has joined #openstack-infra | 07:15 | |
| *** nati_ueno has joined #openstack-infra | 07:15 | |
| *** nati_ueno has quit IRC | 07:16 | |
| openstackgerrit | A change was merged to openstack-infra/config: Run the tests for the python-openstacksdk project https://review.openstack.org/69667 | 07:17 |
| *** nati_ueno has joined #openstack-infra | 07:17 | |
| *** jcooley_ has quit IRC | 07:17 | |
| *** yolanda has joined #openstack-infra | 07:18 | |
| *** DinaBelova_ is now known as DinaBelova | 07:19 | |
| *** sarob has quit IRC | 07:19 | |
| *** Ryan_Lane has quit IRC | 07:21 | |
| *** nati_ueno has quit IRC | 07:25 | |
| *** nati_ueno has joined #openstack-infra | 07:26 | |
| *** nati_ueno has quit IRC | 07:28 | |
| *** nati_ueno has joined #openstack-infra | 07:29 | |
| *** pblaho has joined #openstack-infra | 07:30 | |
| *** yamahata has quit IRC | 07:36 | |
| *** nati_ueno has quit IRC | 07:37 | |
| *** nati_ueno has joined #openstack-infra | 07:37 | |
| *** nati_ueno has quit IRC | 07:40 | |
| *** wenlock has quit IRC | 07:41 | |
| *** Ryan_Lane has joined #openstack-infra | 07:57 | |
| *** vogxn has joined #openstack-infra | 08:04 | |
| *** flaper87|afk is now known as flaper87 | 08:05 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Publish draft docs on draft-docs https://review.openstack.org/70594 | 08:06 |
| *** yamahata has joined #openstack-infra | 08:07 | |
| *** CaptTofu has joined #openstack-infra | 08:11 | |
| *** yamahata has quit IRC | 08:11 | |
| *** CaptTofu has quit IRC | 08:16 | |
| *** hashar has joined #openstack-infra | 08:19 | |
| *** markmcclain has quit IRC | 08:22 | |
| *** che-arne has joined #openstack-infra | 08:23 | |
| *** che-arne has quit IRC | 08:29 | |
| *** che-arne has joined #openstack-infra | 08:31 | |
| *** Ryan_Lane has quit IRC | 08:35 | |
| *** Ryan_Lane has joined #openstack-infra | 08:36 | |
| *** krtaylor has quit IRC | 08:47 | |
| *** krtaylor has joined #openstack-infra | 08:49 | |
| *** Ryan_Lane has quit IRC | 08:52 | |
| *** jooools has joined #openstack-infra | 08:54 | |
| *** matsuhashi has quit IRC | 08:54 | |
| *** krtaylor has quit IRC | 08:55 | |
| *** matsuhashi has joined #openstack-infra | 08:55 | |
| *** jcoufal has joined #openstack-infra | 08:55 | |
| *** e0ne has joined #openstack-infra | 08:56 | |
| *** krtaylor has joined #openstack-infra | 08:56 | |
| *** reed has joined #openstack-infra | 08:59 | |
| *** jpich has joined #openstack-infra | 09:01 | |
| *** bhuvan has joined #openstack-infra | 09:02 | |
| *** mancdaz has joined #openstack-infra | 09:05 | |
| *** derekh has joined #openstack-infra | 09:05 | |
| mordred | /buffer ##gozer-team | 09:09 |
| *** bradm1 has joined #openstack-infra | 09:10 | |
| *** bradm has quit IRC | 09:10 | |
| *** bradm1 is now known as bradm | 09:10 | |
| mordred | gah | 09:11 |
| *** fungi has quit IRC | 09:12 | |
| *** fbo_away is now known as fbo | 09:13 | |
| hashar | time to change that channel name ! /D | 09:13 |
| *** xqueralt has joined #openstack-infra | 09:14 | |
| *** fungi has joined #openstack-infra | 09:16 | |
| *** sarob has joined #openstack-infra | 09:17 | |
| *** andreaf has joined #openstack-infra | 09:17 | |
| *** yassine has joined #openstack-infra | 09:18 | |
| *** dizquierdo has joined #openstack-infra | 09:22 | |
| *** sarob has quit IRC | 09:23 | |
| *** boris-42_ has joined #openstack-infra | 09:23 | |
| *** odyssey4me has quit IRC | 09:25 | |
| *** jooools has quit IRC | 09:29 | |
| *** odyssey4me has joined #openstack-infra | 09:34 | |
| *** jooools has joined #openstack-infra | 09:35 | |
| *** e0ne_ has joined #openstack-infra | 09:38 | |
| openstackgerrit | Sylvain Bauza proposed a change to openstack-infra/reviewstats: Fix Launchpad ID for Nikolay Starodubtsev https://review.openstack.org/71213 | 09:40 |
| *** dims_ has quit IRC | 09:41 | |
| *** e0ne has quit IRC | 09:41 | |
| *** xqueralt has quit IRC | 09:42 | |
| *** julim has joined #openstack-infra | 09:42 | |
| *** dims_ has joined #openstack-infra | 09:44 | |
| openstackgerrit | Dina Belova proposed a change to openstack-infra/reviewstats: Fix list of core members due to Climate core team clean-up https://review.openstack.org/71215 | 09:46 |
| *** talluri has joined #openstack-infra | 09:49 | |
| *** jcoufal has quit IRC | 09:54 | |
| *** bhuvan has quit IRC | 09:55 | |
| *** bhuvan has joined #openstack-infra | 09:56 | |
| *** jasondotstar has joined #openstack-infra | 10:00 | |
| *** max_lobur_afk is now known as max_lobur | 10:04 | |
| *** cody-somerville has joined #openstack-infra | 10:05 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Sort tracked projects list https://review.openstack.org/71220 | 10:06 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Add rally to tracked projects list https://review.openstack.org/63741 | 10:06 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Sort tracked projects list https://review.openstack.org/63770 | 10:08 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Add rally to tracked projects list https://review.openstack.org/63741 | 10:08 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Add climate-nova to tracked projects list https://review.openstack.org/71221 | 10:09 |
| *** johnthetubaguy has joined #openstack-infra | 10:09 | |
| *** CaptTofu has joined #openstack-infra | 10:12 | |
| *** CaptTofu has quit IRC | 10:16 | |
| *** zehicle_at_dell has quit IRC | 10:16 | |
| *** jooools has quit IRC | 10:16 | |
| *** vogxn has quit IRC | 10:19 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Add 2m timeout to tests https://review.openstack.org/71223 | 10:19 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to test_remote https://review.openstack.org/70936 | 10:20 |
| *** sarob has joined #openstack-infra | 10:20 | |
| *** jcoufal has joined #openstack-infra | 10:23 | |
| *** sarob has quit IRC | 10:27 | |
| *** boris-42_ has quit IRC | 10:28 | |
| *** markmc has joined #openstack-infra | 10:31 | |
| *** bookwar has quit IRC | 10:32 | |
| *** e0ne_ has quit IRC | 10:35 | |
| *** e0ne has joined #openstack-infra | 10:35 | |
| *** jooools has joined #openstack-infra | 10:36 | |
| openstackgerrit | Noorul Islam K M proposed a change to openstack-infra/reviewstats: Update solum core-team https://review.openstack.org/71227 | 10:37 |
| *** bhuvan has quit IRC | 10:38 | |
| *** boris-42_ has joined #openstack-infra | 10:40 | |
| *** bookwar has joined #openstack-infra | 10:41 | |
| *** nosnos has quit IRC | 10:53 | |
| *** pblaho has quit IRC | 10:58 | |
| *** johnthetubaguy has quit IRC | 10:59 | |
| *** jhesketh__ has quit IRC | 10:59 | |
| *** jasondotstar has quit IRC | 11:00 | |
| *** che-arne has quit IRC | 11:01 | |
| *** johnthetubaguy has joined #openstack-infra | 11:03 | |
| *** jhesketh has quit IRC | 11:06 | |
| *** jasondotstar has joined #openstack-infra | 11:13 | |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Toplevel collection typecheck. https://review.openstack.org/70997 | 11:17 |
| *** sarob has joined #openstack-infra | 11:24 | |
| *** emagana_ has joined #openstack-infra | 11:29 | |
| *** sarob has quit IRC | 11:29 | |
| *** emagana has quit IRC | 11:32 | |
| *** talluri has quit IRC | 11:35 | |
| *** bhuvan has joined #openstack-infra | 11:37 | |
| *** dizquierdo has quit IRC | 11:38 | |
| *** rfolco has joined #openstack-infra | 11:48 | |
| *** ewindisch has quit IRC | 11:54 | |
| *** emagana_ has quit IRC | 11:55 | |
| *** igor_ has joined #openstack-infra | 11:57 | |
| *** dims_ has quit IRC | 11:57 | |
| *** paul-- has quit IRC | 11:57 | |
| *** dims_ has joined #openstack-infra | 11:57 | |
| *** dims_ has quit IRC | 11:57 | |
| *** dims has joined #openstack-infra | 11:58 | |
| *** jhesketh_ has joined #openstack-infra | 11:59 | |
| *** ruhe has joined #openstack-infra | 11:59 | |
| *** lcestari has joined #openstack-infra | 12:03 | |
| *** hcc has joined #openstack-infra | 12:03 | |
| *** ruhe is now known as ruhe_ | 12:06 | |
| *** ruhe_ is now known as ruhe | 12:06 | |
| *** CaptTofu has joined #openstack-infra | 12:07 | |
| *** jkt has quit IRC | 12:10 | |
| *** yamahata has joined #openstack-infra | 12:19 | |
| *** ArxCruz has joined #openstack-infra | 12:20 | |
| *** paul-- has joined #openstack-infra | 12:23 | |
| *** hashar has quit IRC | 12:24 | |
| *** jooools has quit IRC | 12:25 | |
| *** emagana has joined #openstack-infra | 12:26 | |
| *** coolsvap has quit IRC | 12:33 | |
| *** emagana has quit IRC | 12:35 | |
| *** CaptTofu has quit IRC | 12:36 | |
| *** CaptTofu has joined #openstack-infra | 12:36 | |
| *** mgagne has joined #openstack-infra | 12:36 | |
| *** hcc has quit IRC | 12:37 | |
| *** cody-somerville has quit IRC | 12:39 | |
| *** CaptTofu has quit IRC | 12:41 | |
| *** jasondotstar has quit IRC | 12:41 | |
| *** paul-- has quit IRC | 12:42 | |
| *** ruhe is now known as ruhe_ | 12:44 | |
| *** mkerrin has joined #openstack-infra | 12:44 | |
| *** ruhe_ has quit IRC | 12:48 | |
| *** pblaho has joined #openstack-infra | 12:53 | |
| *** amotoki has quit IRC | 12:55 | |
| *** apevec has joined #openstack-infra | 12:56 | |
| apevec | hi, I noticed tempest grizzly is failing e.g. http://logs.openstack.org/periodic-stable/periodic-tempest-dsvm-full-grizzly/f25a89a/console.html | 12:57 |
| apevec | tests pass but then it fails with: | 12:57 |
| apevec | /opt/stack/new/devstack-gate/devstack-vm-gate.sh: line 358: tools/check_logs.py: No such file or directory | 12:57 |
| *** alexpilotti has joined #openstack-infra | 13:03 | |
| *** oubiwann_ has joined #openstack-infra | 13:06 | |
| *** oubiwan__ has joined #openstack-infra | 13:07 | |
| *** hashar has joined #openstack-infra | 13:08 | |
| *** andre__ has quit IRC | 13:09 | |
| *** oubiwann_ has quit IRC | 13:10 | |
| *** dprince has joined #openstack-infra | 13:11 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/nodepool: Adding the possibility to pass a cleanup timeout to nodepool https://review.openstack.org/71261 | 13:14 |
| *** CaptTofu has joined #openstack-infra | 13:17 | |
| *** cody-somerville has joined #openstack-infra | 13:18 | |
| *** cody-somerville has quit IRC | 13:18 | |
| *** cody-somerville has joined #openstack-infra | 13:18 | |
| *** smarcet has joined #openstack-infra | 13:19 | |
| *** che-arne has joined #openstack-infra | 13:22 | |
| *** ruhe has joined #openstack-infra | 13:22 | |
| *** paul-- has joined #openstack-infra | 13:22 | |
| *** andre__ has joined #openstack-infra | 13:24 | |
| *** oubiwan__ has quit IRC | 13:24 | |
| *** homeless_ has joined #openstack-infra | 13:26 | |
| *** homeless has quit IRC | 13:26 | |
| *** igor_ has quit IRC | 13:27 | |
| *** sarob has joined #openstack-infra | 13:27 | |
| *** igor_ has joined #openstack-infra | 13:28 | |
| *** dkranz has joined #openstack-infra | 13:29 | |
| *** eharney has joined #openstack-infra | 13:30 | |
| *** igor_ has quit IRC | 13:33 | |
| *** sarob has quit IRC | 13:34 | |
| *** sandywalsh has quit IRC | 13:34 | |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/config: XenServer installer script https://review.openstack.org/60581 | 13:36 |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Do not produce extra whitespaces on Python 2.7.3. https://review.openstack.org/71266 | 13:39 |
| *** thomasem has joined #openstack-infra | 13:43 | |
| *** CaptTofu has quit IRC | 13:46 | |
| *** ruhe is now known as ruhe_ | 13:46 | |
| *** sandywalsh has joined #openstack-infra | 13:48 | |
| *** dims has quit IRC | 13:48 | |
| *** yassine has quit IRC | 13:49 | |
| *** thomasem has quit IRC | 13:49 | |
| *** yassine_ has joined #openstack-infra | 13:49 | |
| *** dims has joined #openstack-infra | 13:50 | |
| *** thomasem has joined #openstack-infra | 13:50 | |
| *** zigo_ is now known as zigo | 13:50 | |
| *** CaptTofu has joined #openstack-infra | 13:50 | |
| *** ruhe_ is now known as ruhe | 13:54 | |
| *** ruhe has quit IRC | 13:54 | |
| *** igor_ has joined #openstack-infra | 13:58 | |
| *** mriedem has joined #openstack-infra | 13:58 | |
| *** mfer has joined #openstack-infra | 13:58 | |
| *** miqui has joined #openstack-infra | 14:00 | |
| *** sandywalsh has quit IRC | 14:00 | |
| *** igor__ has joined #openstack-infra | 14:00 | |
| *** sandywalsh has joined #openstack-infra | 14:01 | |
| *** igor_ has quit IRC | 14:03 | |
| *** mriedem has quit IRC | 14:06 | |
| *** mriedem has joined #openstack-infra | 14:07 | |
| *** mrodden has joined #openstack-infra | 14:08 | |
| *** dstufft|laptop has joined #openstack-infra | 14:09 | |
| ArxCruz | if someone ask... http://devopsreactions.tumblr.com/post/75576729444/explaining-how-git-gerrit-and-jenkins-work-together | 14:12 |
| fungi | ArxCruz: people constantly ask about that. might be worth reposting to the -infra ml too | 14:14 |
| *** bknudson has quit IRC | 14:14 | |
| *** medberry is now known as med_ | 14:14 | |
| ArxCruz | hehehe | 14:14 |
| *** jasondotstar has joined #openstack-infra | 14:14 | |
| *** prad_ has joined #openstack-infra | 14:15 | |
| *** afazekas has quit IRC | 14:16 | |
| fungi | apevec: weird. looks like dkranz never backported that script to tempest's stable/grizzly branch. i see a couple of possible fixes (either backport it, or have devstack-gate conditionally run it only when present) | 14:16 |
| fungi | sdague: dkranz: preferences? ^ | 14:16 |
| fungi | i'm leaning toward the latter, since grizzly is going to be chock full o' errors we won't be fixing anyway | 14:17 |
| fungi | starting on a d-g patch for that now | 14:17 |
| bhuvan | fungi: https://bugs.launchpad.net/openstack-ci/+bug/1275824 | 14:19 |
| uvirtbot | bhuvan: Error: Could not parse data returned by Launchpad: The read operation timed out | 14:19 |
| bhuvan | any clue on howto test jeepyb fixes? | 14:19 |
| bhuvan | i'm unable to setup gerrit using puppet, as i use oraclelinux vm | 14:20 |
| dkranz | fungi: backport which script? | 14:20 |
| fungi | dkranz: tools/check_logs.py | 14:20 |
| *** ryanpetrello has joined #openstack-infra | 14:20 | |
| dkranz | fungi: devstack gate should not be running that script at all on grizzly | 14:21 |
| fungi | grizzly tempest tests are failing because devstack-gate currently seems to assume it exists | 14:21 |
| dkranz | fungi: I did make that change | 14:21 |
| fungi | oh, right, we already have conditional logic around that... i remember now. so why is it trying... | 14:21 |
| *** dkliban_afk is now known as dkliban | 14:21 | |
| *** sarob has joined #openstack-infra | 14:22 | |
| bhuvan | fungi: i've a patch to fix ProjectsYamlRegistry() -> ProjectsRegistry() and derive defaults from ini or yaml, if ini exist | 14:22 |
| fungi | bhuvan: for the pull request closer, i think you'd just need a github project to test with and a minimal projects.ini and projects.yaml with references to the right project name/account credentials | 14:22 |
| bhuvan | great. i'll create sample ini/yml file and test this script | 14:23 |
| bhuvan | manage_projects.py? | 14:23 |
| fungi | dkranz: worth noting, this is in the periodic jobs only. maybe we're not checking the right envvars for those | 14:23 |
| fungi | bhuvan: oh, i was thinking you meant just for testing the broken state of the github pull request closer in jeepyb... for a more thorough test of anything which calls into jeepyb's projects.py or utils.py you'd also need a working gerrit with your sample project set up in it and a corresponding launchpad project | 14:25 |
| *** talluri has joined #openstack-infra | 14:26 | |
| fungi | bhuvan: and a cgit server too, actually | 14:26 |
| fungi | bhuvan: if it's a fairly minimal patch, we's probably just review it first and then we could hand-patch it onto review-dev.openstack.org and exercise it there | 14:27 |
| bhuvan | cboylan already proposed a patch to fix temporarily. https://review.openstack.org/#/c/71110/ | 14:27 |
| bhuvan | i'm working on fixing the issue long term | 14:27 |
| fungi | bhuvan: yep, appreciated. i was hoping we could get a second core +2 on 71110 but i think i'll just go ahead and approve it now so i can make sure it wasn't hiding other bugs (i think the pull-request closer is still broken in other ways, prior to the projects.yaml format change) | 14:28 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Determine yaml format when loading yaml configs. https://review.openstack.org/71110 | 14:30 |
| dkranz | fungi: devstack-vm-gate.sh is checking GRENADE_OLD_BRANCH suggested by jeblair | 14:31 |
| dkranz | fungi: Perhaps that does not work in periodic builds? | 14:31 |
| fungi | dkranz: that's what i'm trying to determine... | 14:31 |
| *** boris-42_ has quit IRC | 14:31 | |
| dkranz | fungi: ok | 14:32 |
| *** zehicle_at_dell has joined #openstack-infra | 14:32 | |
| *** weshay has joined #openstack-infra | 14:34 | |
| *** dstufft|laptop has quit IRC | 14:34 | |
| fungi | dkranz: yeah it looks like that may not be set. we do have BRANCH_OVERRIDE=stable/grizzly and OVERRIDE_ZUUL_BRANCH=stable/grizzly according to the log... checking now to see whether there are other conditions which would cause us not to additionally negative-match on one of those | 14:34 |
| *** ruhe has joined #openstack-infra | 14:35 | |
| fungi | dkranz: i think the existing code maybe still doesn't do what we want... or it recently changed to not do what we want... because GRENADE_OLD_BRANCH is only ever set in devstack-vm-gate-wrap.sh and devstack-vm-gate.sh when one of DEVSTACK_GATE_GRENADE or DEVSTACK_GATE_GRENADE_FORWARD is equal to 1 | 14:39 |
| fungi | which i think means that we should see the same errors cropping up on non-grenade tempest jobs for stable/grizzly changes too? | 14:40 |
| fungi | i'm going hunting now to see whether i can prove that theory | 14:41 |
| ArxCruz | fungi: hey, about that scp plugin ? did you guys release the source code ? | 14:43 |
| *** dstanek has joined #openstack-infra | 14:43 | |
| fungi | ArxCruz: the source code itself it public, as for whether a new release has been built/uploaded from it i'm not sure... i'll check in a bit | 14:44 |
| ArxCruz | ok | 14:45 |
| ArxCruz | fungi: where's the source code? | 14:45 |
| *** sarob has quit IRC | 14:46 | |
| *** ruhe is now known as ruhe_ | 14:46 | |
| fungi | ArxCruz: https://github.com/jenkinsci/scp-plugin | 14:46 |
| *** Ajaeger has joined #openstack-infra | 14:47 | |
| krtaylor | yeah, that's for 1.8 | 14:47 |
| fungi | ArxCruz: last commit was two weeks ago, last tag was three years ago | 14:47 |
| *** malini_afk is now known as malini | 14:49 | |
| krtaylor | fungi, there must be a -infra version that ArxCruz is referring to that is tagged 1.9 | 14:49 |
| *** ruhe_ is now known as ruhe | 14:49 | |
| fungi | ArxCruz: i believe all the patches for the unofficial "1.9" pre-release snapshot we're using is reflected in those commits on master | 14:50 |
| fungi | krtaylor: ^ | 14:50 |
| ArxCruz | fungi: cool | 14:50 |
| krtaylor | fungi, excellent, thanks | 14:50 |
| fungi | ArxCruz: krtaylor: and that pull request of zaro's | 14:51 |
| fungi | i believe what i heard was that we wanted to try to reincorporate these fixes and the console log collection feature itself back into upstream | 14:51 |
| *** alexpilotti has quit IRC | 14:52 | |
| *** Ajaeger has quit IRC | 14:52 | |
| ArxCruz | fungi: are you aware about any swift plugin ? | 14:52 |
| krtaylor | fungi, mtreinish had mentioned to me that there was interest in a swift plugin | 14:52 |
| *** mestery has joined #openstack-infra | 14:52 | |
| krtaylor | ArxCruz, hehheh | 14:53 |
| ArxCruz | sorry boss :P | 14:53 |
| krtaylor | hehheh, great minds... | 14:53 |
| *** mestery has quit IRC | 14:53 | |
| fungi | ArxCruz: krtaylor: sounds neat. not up to date on what efforts are ongoing there though | 14:53 |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add script trigger. https://review.openstack.org/70971 | 14:53 |
| openstackgerrit | Florian Klink proposed a change to openstack-infra/git-review: Use LC_ALL="C" for run_command_status() https://review.openstack.org/71276 | 14:53 |
| *** ruhe is now known as ruhe_ | 14:53 | |
| *** ruhe_ is now known as ruhe | 14:53 | |
| *** mestery has joined #openstack-infra | 14:53 | |
| krtaylor | ArxCruz, let's see if we can find anyone that might be working on one, else we'll look into it | 14:54 |
| fungi | krtaylor: ArxCruz: check with jhesketh_/mikal/mattoliverau when they're around... i think maybe they had some interest in it as well (might already be working on something similar?) | 14:55 |
| krtaylor | fungi, great, will do | 14:56 |
| krtaylor | fungi, we will be using a swift service for our ci artifacts for powerkvm | 14:56 |
| fungi | dkranz: apevec: we're seeing the same error on stable/grizzly changes being tested... http://logs.openstack.org/52/63752/1/check/check-tempest-dsvm-full/c50af5e/console.html#_2014-02-02_05_24_38_224 | 14:56 |
| *** freyes has joined #openstack-infra | 14:56 | |
| apevec | fungi, ok, so we need d-g fix asap | 14:57 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Update solum core-team https://review.openstack.org/71227 | 14:57 |
| *** alexpilotti has joined #openstack-infra | 14:58 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 15:01 | |
| *** zehicle_at_dell has quit IRC | 15:01 | |
| fungi | apevec: looking more closely at your periodic job log, that error is misleading but not the reason the job failed. tempest.tests.compute.volumes.test_volumes_get.* errored, tempest.tests.volume.test_volumes_get.* failed as did tempest.tests.volume.admin.test_volume_types.VolumeTypesTest.test_create_get_delete_volume_with_volume_type_and_extra_specs... | 15:01 |
| *** ruhe is now known as ruhe_ | 15:01 | |
| fungi | i think you're being thrown by the cli tests passing since they run after the main body of tempest tests (which reported as overall failed) | 15:02 |
| jeblair | fungi, ArxCruz: zaro's pull request is merged into master of scp plugin | 15:02 |
| fungi | jeblair: i just saw that. yesterday i guess | 15:02 |
| jeblair | fungi, ArxCruz: jclouds plugin supports swift to some extent, but not how we want to use it | 15:02 |
| jeblair | fungi, ArxCruz: our swift work is focused on replacing jenkins, so we don't plan on adding the bits we need to any jenkins plugin | 15:03 |
| jeblair | fungi, ArxCruz: rather, we will start with a simple python script to do the uploads, and then incorporate that into the job-runner that we will use instead of jenkins in the future | 15:03 |
| jeblair | fungi, ArxCruz: and yes, jhesketh is working on that | 15:04 |
| fungi | apevec: "XML: nosetests-full.xml [...] FAILED (SKIP=72, errors=11, failures=7) | 15:04 |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add script trigger. https://review.openstack.org/70971 | 15:04 |
| *** rossella_s has joined #openstack-infra | 15:04 | |
| *** ruhe_ is now known as ruhe | 15:05 | |
| *** Ajaeger has joined #openstack-infra | 15:06 | |
| *** ArxCruz has quit IRC | 15:07 | |
| *** ruhe is now known as ruhe_ | 15:07 | |
| *** jergerber has joined #openstack-infra | 15:09 | |
| *** ruhe_ is now known as ruhe | 15:09 | |
| malini | fungi: ping | 15:10 |
| *** ArxCruz has joined #openstack-infra | 15:10 | |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add script trigger. https://review.openstack.org/70971 | 15:10 |
| *** mestery has quit IRC | 15:10 | |
| *** rossella_s has quit IRC | 15:11 | |
| *** gokrokve has joined #openstack-infra | 15:12 | |
| *** jergerber has quit IRC | 15:12 | |
| *** freyes has quit IRC | 15:14 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Don't check logs on stable/grizzly https://review.openstack.org/71285 | 15:16 |
| fungi | dkranz: apevec: i think this ^ will quiet the misleading error message | 15:16 |
| fungi | malini: what's up? | 15:16 |
| malini | fungi: wanted to get your thoughts on https://review.openstack.org/#/c/65140/6/modules/openstack_project/files/zuul/layout.yaml | 15:17 |
| *** matsuhashi has quit IRC | 15:17 | |
| malini | what are the general thoughts on adding a non-voting gate job? | 15:17 |
| dkranz | fungi: Thanks for that. | 15:17 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Don't check logs on stable/grizzly https://review.openstack.org/71285 | 15:18 |
| * fungi made an embarrassing typo in the first commit message | 15:18 | |
| *** dprince has quit IRC | 15:18 | |
| *** matsuhashi has joined #openstack-infra | 15:18 | |
| *** talluri has quit IRC | 15:18 | |
| *** talluri has joined #openstack-infra | 15:19 | |
| *** dprince has joined #openstack-infra | 15:19 | |
| *** mrodden has quit IRC | 15:19 | |
| *** amotoki has joined #openstack-infra | 15:20 | |
| fungi | malini: i agree with SergeyLukjanov's comment. generally nobody sees/pays attention to non-voting gate job results, so they're mostly just a waste of resources. when you're ready to remove the non-voting filter for check-tempest-dsvm-marconi, you can add gate-tempest-dsvm-marconi at the same time. as long as they're both built from the same job template they should be identical anyway (aside from the | 15:21 |
| fungi | pipeline name) | 15:21 |
| malini | fungi:I see quite a few of non voting gate jobs in the layout.yaml. The reason I made mine non-voting, was to just flip a switch & make it voting once our tempest patches are in place. I dont want to block other marconi patches while I am working on tempest | 15:21 |
| malini | looks like we typed at the same time :) | 15:21 |
| *** esker has quit IRC | 15:21 | |
| *** rcleere has joined #openstack-infra | 15:22 | |
| *** esker has joined #openstack-infra | 15:22 | |
| *** esker has quit IRC | 15:23 | |
| fungi | skimming changes currently running in the gate pipeline, i only see gate-grenade-dsvm-neutron marked as non-voting, and it's sort of a special case since we're trying to collect gating statistics on that job | 15:23 |
| *** esker has joined #openstack-infra | 15:23 | |
| *** talluri has quit IRC | 15:23 | |
| fungi | you'll still get (non-voting) results for check-tempest-dsvm-marconi in the check pipeline on all your changes | 15:24 |
| malini | fungi: is an experimental gate job different from a gate gate job? | 15:24 |
| ttx | jeblair: wanted to discuss https://review.openstack.org/#/c/65103/ with you | 15:24 |
| *** rossella_s has joined #openstack-infra | 15:24 | |
| fungi | malini: jobs in the experimental pipeline are only run when you leave a comment of "check experimental" on an open change | 15:24 |
| ttx | jeblair: the proposed/* support | 15:24 |
| malini | fungi: aah ..ok.. | 15:25 |
| ttx | jeblair: there is some marginal value to it -- but maybe we should rather aim for juno-1 for such a change | 15:25 |
| ttx | jeblair: I'm also fine with dropping it completely if you think value is 0 | 15:26 |
| malini | fungi: so can I just remove gate-tempest-dsvm-marconi from the gate section, but leave 'name: ^(gate|check)-tempest-dsvm-marconi' as is? | 15:26 |
| fungi | malini: the comment only referred to the job you added in the gate pipeline (which runs after the change is approved but prior to it being merged to the branch). jobs in the check pipeline are what your devs are going to be looking at during review anyway | 15:26 |
| fungi | malini: that should be fine | 15:26 |
| malini | great! thanks fungi! | 15:27 |
| *** esker has quit IRC | 15:27 | |
| *** ruhe is now known as ruhe_ | 15:27 | |
| *** jnoller has joined #openstack-infra | 15:28 | |
| *** kraman has joined #openstack-infra | 15:29 | |
| fungi | malini: you're welcome | 15:29 |
| *** coolsvap has joined #openstack-infra | 15:29 | |
| *** ruhe_ has quit IRC | 15:32 | |
| *** boris-42_ has joined #openstack-infra | 15:32 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add proof-of-concept savanna-guestagent project https://review.openstack.org/71293 | 15:32 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack-infra/config: Add Job for marconi-tempest integration https://review.openstack.org/65140 | 15:32 |
| fungi | SergeyLukjanov: proof of concept ^ worries me a little as a project addition, unless you expect it to transition into a working project and not just get abandoned once the concept is proven | 15:33 |
| SergeyLukjanov | fungi, it's a poc from the implementation side | 15:33 |
| *** matsuhashi has quit IRC | 15:34 | |
| SergeyLukjanov | we'd like to harden it on stackforge and there are two options in future | 15:34 |
| SergeyLukjanov | convert it to inified agent | 15:34 |
| SergeyLukjanov | or it'll be savanna guest agent | 15:34 |
| SergeyLukjanov | no options for abandoning it :) | 15:34 |
| *** mrodden has joined #openstack-infra | 15:34 | |
| SergeyLukjanov | fungi, is it ok for you? | 15:35 |
| fungi | okay, just bear in mind that every new project we add is additional workload for everyone, and we can't really just delete projects from stackforge, so yeah i'm -2 on any project which is expected to possibly be abandoned. but since you expect it to continue to exist, i'm cool with that | 15:35 |
| * fungi was just worried by the commit subject, that's all ;) | 15:36 | |
| SergeyLukjanov | ok, I'll make it more clear | 15:36 |
| SergeyLukjanov | the guest agents approach was discussed at summit and we'd like to use them | 15:36 |
| fungi | agreed, adjusting that subject might avoid similar knee-jerk reactions from other reviewers | 15:37 |
| SergeyLukjanov | and now we have some code that we'd like to publish on stackforge and start using it as an option | 15:37 |
| fungi | sounds awesome then | 15:37 |
| SergeyLukjanov | in the mostly best case we'll ask to rename it to openstack-super-mega-awesome-guestagent | 15:38 |
| fungi | i think it merits an even longer name if it's that super-mega-awesome | 15:38 |
| SergeyLukjanov | :) | 15:40 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add project for savanna-ci configs https://review.openstack.org/66521 | 15:42 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add savanna-guestagent project https://review.openstack.org/71293 | 15:42 |
| *** salv-orlando has quit IRC | 15:42 | |
| malini | fungi,SergeyLukjanov: thank you for the reviews !! I love all +1s & +2s I can get :) | 15:43 |
| malini | fungi: do I need another core reviewer to +2, before it can be approved? | 15:43 |
| SergeyLukjanov | malini, you're welcome, I'm really happy that I can help with reviewing some infra stuff | 15:43 |
| *** salv-orlando has joined #openstack-infra | 15:44 | |
| *** wenlock has joined #openstack-infra | 15:44 | |
| *** markwash has quit IRC | 15:45 | |
| mriedem | i really really wish we didn't test against trunk level code of the clients in the gate | 15:45 |
| mriedem | https://review.openstack.org/#/c/70935/ | 15:45 |
| mriedem | to avoid this every week | 15:45 |
| mriedem | fungi: clarkb: ^ | 15:45 |
| mriedem | sdague: ^ | 15:46 |
| *** homeless_ has quit IRC | 15:47 | |
| fungi | malini: since nobody's really objected to the job configuration itself and several infra core reviewers have commented on the patch for the past month, i guess it's safe to go ahead and approve | 15:47 |
| *** homeless has joined #openstack-infra | 15:48 | |
| malini | yayy..thanks!! I hope jenkins likes it too | 15:48 |
| fungi | mriedem: the alternative to testing trunk of clients in the gate is finding out when they release that they don't actually work with the servers | 15:49 |
| mriedem | gah, i assumed it would be so easy :) | 15:49 |
| *** max_lobur is now known as max_lobur_afk | 15:49 | |
| *** bhuvan has quit IRC | 15:50 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add Job for marconi-tempest integration https://review.openstack.org/65140 | 15:51 |
| fungi | mriedem: but yes, it has been proposed that we should also run (periodic?) tests of our stated minimum requirements... pip's dependency solver (or lack thereof) makes that a bit challenging to actually implement though | 15:51 |
| *** dhellmann is now known as dhellmann_ | 15:52 | |
| *** bhuvan has joined #openstack-infra | 15:52 | |
| mriedem | fungi: i propose that mrodden makes it work in juno... | 15:52 |
| mriedem | we have to justify getting him to summit | 15:52 |
| fungi | since dependencies have their own dependencies, many of which are not explicitly stated, and pip thinks you should always want the latest package which matches your version spec | 15:52 |
| mriedem | ah right, and version caps are naughty | 15:53 |
| *** DinaBelova is now known as DinaBelova_ | 15:53 | |
| fungi | so testing that we work with the minimum specified second- and higher-order dependencies we don't even list in our requirements is not something i think is likely to really work without reinventing some things | 15:53 |
| mrodden | mriedem: hi, what am i being volunteered for? | 15:54 |
| *** jgrimm_ has joined #openstack-infra | 15:54 | |
| mriedem | mrodden: testing with the released client versions in the gate rather than the trunk code | 15:55 |
| mriedem | mrodden: but it's fraught with issues | 15:55 |
| fungi | and if someone wants to work on figuring out how to pre-map the version interactions for a pip dependency tree, they might be better off helping dstufft work that into pip (since i know he wants it in his utopian future anyway) | 15:55 |
| *** igor__ has quit IRC | 15:57 | |
| *** sarob has joined #openstack-infra | 15:57 | |
| *** pblaho has quit IRC | 15:57 | |
| mrodden | lol pip is the future... | 15:58 |
| *** hcc has joined #openstack-infra | 15:59 | |
| mrodden | oh yeah pip doesn't build a tree, just grabs the version req. it sees first | 15:59 |
| mrodden | always forget about that... | 15:59 |
| *** reed has quit IRC | 16:00 | |
| *** sarob has quit IRC | 16:01 | |
| *** emagana has joined #openstack-infra | 16:01 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 16:01 | |
| mrodden | i don't think it would be terrible to do tests with the latest released versions | 16:02 |
| mrodden | rather than latest git master | 16:02 |
| mrodden | we still want that to detect breakage | 16:02 |
| *** reed has joined #openstack-infra | 16:03 | |
| *** oubiwann_ has joined #openstack-infra | 16:03 | |
| *** igor_ has joined #openstack-infra | 16:03 | |
| mrodden | testing minimal would definitely be trickier | 16:04 |
| *** oubiwann_ has quit IRC | 16:04 | |
| *** thuc has joined #openstack-infra | 16:04 | |
| *** thuc_ has joined #openstack-infra | 16:05 | |
| *** oubiwann_ has joined #openstack-infra | 16:05 | |
| *** AlexF has joined #openstack-infra | 16:06 | |
| fungi | i *definitely* agree on the need for additional tests against latest release versions of our stuff. we should prevent ourselves from releasing something which only works with unreleased features/fixes in its dependencies, which is not something we currently test | 16:07 |
| fungi | at least not when we develop the dependencies as well | 16:07 |
| *** emagana has quit IRC | 16:08 | |
| *** igor_ has quit IRC | 16:08 | |
| *** thuc has quit IRC | 16:09 | |
| fungi | we also don't currently test that changes proposed for clients/libraries continue to work with supported stable releases of the integrated servers | 16:12 |
| fungi | which is a bit of a problem too | 16:13 |
| *** hcc is now known as hdd_ | 16:14 | |
| *** dhellmann_ is now known as dhellmann | 16:15 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 16:18 | |
| *** Ajaeger has quit IRC | 16:18 | |
| *** mestery has joined #openstack-infra | 16:18 | |
| *** pblaho has joined #openstack-infra | 16:21 | |
| *** jasondotstar has quit IRC | 16:21 | |
| *** pblaho has quit IRC | 16:21 | |
| *** jcooley_ has joined #openstack-infra | 16:21 | |
| *** pblaho has joined #openstack-infra | 16:21 | |
| *** DinaBelova_ is now known as DinaBelova | 16:26 | |
| *** esker has joined #openstack-infra | 16:27 | |
| *** jcooley_ has quit IRC | 16:28 | |
| *** AaronGr_Zzz is now known as AaronGr | 16:29 | |
| *** weshay is now known as weshay_bbiab | 16:30 | |
| *** jasondotstar has joined #openstack-infra | 16:31 | |
| jeblair | ttx: okay, i understand. i think i'm probably okay with the change, but let me think about it a bit more. | 16:31 |
| *** jcooley_ has joined #openstack-infra | 16:33 | |
| ttx | jeblair: been discussed for two summits already, but there is no hurry | 16:34 |
| openstackgerrit | Ilya Sviridov proposed a change to openstack-infra/config: Added MagentoDB (https://wiki.openstack.org/wiki/MagnetoDB) project to Stackforge https://review.openstack.org/71308 | 16:36 |
| *** gokrokve has quit IRC | 16:36 | |
| *** bhuvan has quit IRC | 16:37 | |
| *** gokrokve has joined #openstack-infra | 16:37 | |
| *** jpich has quit IRC | 16:38 | |
| jeblair | ttx: oh, is there an etherpad? | 16:40 |
| *** mestery has quit IRC | 16:40 | |
| *** gokrokve has quit IRC | 16:41 | |
| *** beagles has quit IRC | 16:41 | |
| jeblair | ttx: (i mostly wanted to think about it in case i think of something in infra that it would break) | 16:41 |
| ttx | jeblair: it's usually mentioned in the release session, but by that point most people are too exhausted to discuss it | 16:42 |
| ttx | see bottom of https://etherpad.openstack.org/p/IcehouseReleaseSchedule | 16:43 |
| fungi | ttx: in atlanta, we should put process track discussions on the first day of the design summit | 16:44 |
| *** hcc has joined #openstack-infra | 16:44 | |
| *** hdd_ has quit IRC | 16:44 | |
| *** hcc is now known as hdd_ | 16:44 | |
| ttx | fungi: and have dissent ? What are you thinking | 16:44 |
| jeblair | fungi: look at the last line of that etherpad? :) | 16:44 |
| fungi | heh | 16:44 |
| jeblair | s/?/./ | 16:44 |
| fungi | jeblair: i think i wrote the last line of that etherpad some months ago | 16:45 |
| fungi | while sitting in a somewhat uncomfortable conference room chair on the far side of the world | 16:45 |
| *** dtroyer has quit IRC | 16:46 | |
| jeblair | ttx: i think we were all planning our ski vacations at that point :) | 16:46 |
| ttx | jeblair: couldn't find it mentioned in earlier summits. i may have dreamt that | 16:46 |
| clarkb | this morning I had vuvuzelas as alarms... yay parades I guess | 16:46 |
| *** dtroyer has joined #openstack-infra | 16:46 | |
| *** jcoufal has quit IRC | 16:47 | |
| ttx | jeblair: maybe it's urgent to wait. Storyboard workflow might also call for one solution or the other | 16:47 |
| fungi | clarkb: that's one hell of a way to wake up | 16:47 |
| jeblair | ttx: how? | 16:47 |
| *** exz has quit IRC | 16:48 | |
| jeblair | ttx: i wasn't expecting storyboard to have m-p branches... oh, do you mean the workflow of the openstack project facilitated by storyboard? | 16:48 |
| *** CaptTofu has quit IRC | 16:48 | |
| ttx | jeblair: remember the discussion about associating tasks to branches vs. milestones | 16:48 |
| jeblair | ttx: (as opposed to the workflow of storyboard _itself_) | 16:48 |
| ttx | jeblair: yes | 16:48 |
| jeblair | ttx: that makes more sense. ;) | 16:49 |
| ttx | it does | 16:49 |
| jeblair | ttx: yeah. i think it's a good idea. i'll +2 it tomorrow if i don't think of any problems. | 16:50 |
| *** exz has joined #openstack-infra | 16:50 | |
| ttx | jeblair: I don't really feel like experimenting around icehouse-3 though | 16:50 |
| ttx | jeblair: that said, passing that review doesn't force us to experiment. | 16:50 |
| ttx | it just supports additional cases | 16:51 |
| ttx | so maybe it's sane | 16:51 |
| *** salv-orlando has quit IRC | 16:53 | |
| *** pblaho has quit IRC | 16:53 | |
| *** salv-orlando has joined #openstack-infra | 16:53 | |
| *** pblaho has joined #openstack-infra | 16:54 | |
| clarkb | logstash stopped indexing console logs overnight for some reason, I will look into that shortly | 16:55 |
| *** gyee has joined #openstack-infra | 16:56 | |
| *** dtroyer has left #openstack-infra | 16:57 | |
| jeblair | clarkb: what's the status of gerrit re-vote updatedOn? | 16:57 |
| clarkb | jeblair: I haven't had a chance to look at it beyond "this is the line that needs to change" | 16:57 |
| jeblair | clarkb: think we should go ahead and merge the check+0 change then? | 16:58 |
| clarkb | I think so | 16:58 |
| jeblair | clarkb: i can probably write the plan (c) change to zuul next week (search for comments by text) | 16:59 |
| clarkb | I am also worried that if I change gerrit properly upstream won't like it (the proper fix is to remove the complex dirty bit checking and just update the DB each time) | 16:59 |
| *** dtroyer has joined #openstack-infra | 16:59 | |
| *** nati_ueno has joined #openstack-infra | 16:59 | |
| jeblair | clarkb: does it suppress anything else other than same-vote db updates? | 16:59 |
| clarkb | no, the method is very specific to applying votes/comments | 17:00 |
| *** bhuvan has joined #openstack-infra | 17:01 | |
| jeblair | clarkb: well, let's just try to make the case for our use-case -- knowing when the last time an automated system voted on a change is fairly universally useful. | 17:01 |
| *** emagana has joined #openstack-infra | 17:01 | |
| *** markmcclain has joined #openstack-infra | 17:01 | |
| openstackgerrit | A change was merged to openstack-infra/config: Use ZUUL_URL in ggp https://review.openstack.org/68997 | 17:02 |
| openstackgerrit | A change was merged to openstack-infra/config: Ensure a recent(48h) check result https://review.openstack.org/69508 | 17:02 |
| *** markmcclain has quit IRC | 17:02 | |
| clarkb | jeblair: I agree, I suppose I can just rip out all the unneeded code, push it, and see what happens | 17:03 |
| jeblair | clarkb, fungi: pontentially breaking change, followed by a potentially runaway change ^ | 17:03 |
| fungi | great combination! | 17:03 |
| fungi | a good time for me to grab lunch in that case ;) | 17:03 |
| clarkb | I worry because someone clearly put a lot of time and effort into doing it wrong :) | 17:03 |
| *** markmcclain has joined #openstack-infra | 17:03 | |
| *** e0ne has quit IRC | 17:03 | |
| fungi | clarkb: now you sound like you're talking about tox posargs again | 17:04 |
| jeblair | clarkb: yeah, if we followed that logic we'd get nothing done | 17:04 |
| jeblair | ;) | 17:04 |
| clarkb | fungi: don't get me started on tox posrags, have you seen the latest on that thread? | 17:04 |
| fungi | i will momentarily | 17:04 |
| *** igor_ has joined #openstack-infra | 17:04 | |
| openstackgerrit | A change was merged to openstack-infra/config: Separate handling of ChangeIds and Git SHAs in Gerrit https://review.openstack.org/64738 | 17:05 |
| clarkb | well that explains it, logstash is 100k deep into a gearman queue | 17:05 |
| *** gokrokve has joined #openstack-infra | 17:05 | |
| clarkb | and my logstash-worker* nodes don't seem reachable | 17:05 |
| clarkb | wth | 17:06 |
| jeblair | gearman says they are registered | 17:06 |
| jeblair | clarkb: remember they are "logstash-worker01.openstack.org" now | 17:06 |
| clarkb | jeblair: ah | 17:07 |
| clarkb | jeblair: thanks | 17:07 |
| jeblair | clarkb: the gearman debug log shows activity... | 17:07 |
| fungi | sounds like we'll either need to start recommending use of openstack-infra/xot instead, or wait until we can get what we want into tox 1.8 | 17:07 |
| *** gokrokve_ has joined #openstack-infra | 17:07 | |
| clarkb | fungi: fwiw I was || close to forking tox and throwing it up on github last night | 17:07 |
| jeblair | clarkb: s/github/stackforge/ | 17:08 |
| clarkb | jeblair: sure :) | 17:08 |
| jeblair | with like regression tests | 17:08 |
| *** Ryan_Lane has joined #openstack-infra | 17:08 | |
| *** dangers_away is now known as dangers | 17:08 | |
| fungi | openstack infrastructure is probably the biggest regression test known to man | 17:09 |
| *** igor_ has quit IRC | 17:09 | |
| *** morganfainberg_Z is now known as morganfainberg | 17:09 | |
| jeblair | fungi: thanks for including the word 'test' :) | 17:09 |
| *** hogepodge has joined #openstack-infra | 17:09 | |
| fungi | heh | 17:09 |
| *** gokrokve has quit IRC | 17:10 | |
| jeblair | clarkb: graphite says worker02 is fubar | 17:10 |
| jeblair | s/graphite/cacti/ | 17:10 |
| *** gokrokve has joined #openstack-infra | 17:10 | |
| jeblair | clarkb: worker02-04 | 17:10 |
| jeblair | worker01 is ok | 17:10 |
| *** bogdando has quit IRC | 17:11 | |
| jeblair | clarkb: oh apparently we had 01-05 running, so 02-05 are out, 01 is ok | 17:11 |
| jeblair | clarkb: (did someone add 05? i thought it was supposed to be idle) | 17:12 |
| *** amotoki is now known as amotoki_ | 17:12 | |
| jeblair | clarkb: it looks like crm may be busy-looping on 02 | 17:12 |
| clarkb | yeah looks like they all went sideways | 17:13 |
| clarkb | I am restarting the services there which seems to get things moving | 17:13 |
| *** gokrokve_ has quit IRC | 17:13 | |
| jeblair | clarkb: give me a host to continue to debug | 17:13 |
| fungi | weeks ago i started additional workers on 05 and 06 | 17:14 |
| clarkb | jeblair: 02 | 17:14 |
| jeblair | clarkb: thx | 17:14 |
| jeblair | fungi: ok | 17:14 |
| clarkb | jeblair: or do you want one that hasn't had services restarted? | 17:14 |
| fungi | back when our gate volume was burdening it (based on cacti graphs) | 17:14 |
| clarkb | I have 01-04 happy again, if you want derped hosts 05 or 06 will work | 17:14 |
| jeblair | clarkb: i want one that hasn't restarted :) | 17:14 |
| jeblair | clarkb: i'll take 05 | 17:14 |
| clarkb | jeblair: ok | 17:14 |
| jeblair | clarkb, fungi: 06 fell over yesterday | 17:14 |
| jeblair | 05 is not useful | 17:15 |
| fungi | should be plenty safe to stop the workers on 05 and 06 under current volume | 17:15 |
| jeblair | it's not doing what 02 was doing | 17:15 |
| fungi | once troubleshooting has completed | 17:15 |
| clarkb | jeblair: 06 hasn't been touched | 17:15 |
| clarkb | if you want to look at it too | 17:15 |
| *** julim has quit IRC | 17:15 | |
| jeblair | clarkb: 05 and 06 show similar symptoms; all workers are running and the debug logs indicate they are between having pushed their data and sent a WORK_COMPLETE packet | 17:19 |
| mordred | clarkb, jeblair: I got pinged by hp about us still being on 1.0 - 1.1 is still too slow, right? I just want to make sure I'm telling them the right thing | 17:19 |
| clarkb | jeblair: I wonder if the logrotate hit them in a weird way? | 17:19 |
| *** markmc has quit IRC | 17:19 | |
| clarkb | mordred: yes 1.1 is unusable | 17:19 |
| jeblair | clarkb: oh hrm, crm is running though | 17:19 |
| jeblair | mordred: what is 1.1? | 17:19 |
| *** Ryan_Lane has quit IRC | 17:20 | |
| jesusaurus | jeblair: the new region | 17:21 |
| clarkb | jeblair: hpcloud region b basically | 17:21 |
| jeblair | clarkb, jesusaurus: ah, last time people talked about that with me they used works like "region" and not numbers. :) thanks | 17:21 |
| jeblair | clarkb, mordred: so sdague's graph http://dague.net/testing/check-tempest-dsvm-full.png | 17:21 |
| *** hemna_ is now known as hemna | 17:22 | |
| *** jcooley_ has quit IRC | 17:22 | |
| jeblair | clarkb, mordred: makes me think that we should extend our testing of region-b because the numbers don't look terrible. | 17:22 |
| jesusaurus | clarkb: did you try both the new east and new west? | 17:22 |
| jesusaurus | ive heard rumor that 1.1 west is better, but havent tested | 17:22 |
| clarkb | jesusaurus: I tried new west and was immediately told to stop | 17:22 |
| jesusaurus | ha, okay | 17:22 |
| clarkb | jesusaurus: those are 30GB test nodes | 17:23 |
| *** jcooley_ has joined #openstack-infra | 17:23 | |
| clarkb | er jeblair ^ | 17:23 |
| jeblair | clarkb, mordred: they are close to az1's standard deviation, and within az2 and az3's std. | 17:23 |
| jeblair | clarkb: oh | 17:23 |
| jeblair | clarkb: if we use 8g it fails? | 17:23 |
| clarkb | jeblair: which I suppose we can use, but that is ridiculous imo | 17:23 |
| clarkb | jeblair: yes | 17:23 |
| *** odyssey4me2 has joined #openstack-infra | 17:23 | |
| jeblair | clarkb: i don't think we can use them without limiting the ram usage, otherwise, within a few hours you won't be able to test openstack on less than 8g of ram. :) | 17:24 |
| clarkb | ha | 17:24 |
| jeblair | i mean less than 30g. | 17:24 |
| *** odyssey4me has quit IRC | 17:24 | |
| *** odyssey4me2 is now known as odyssey4me | 17:24 | |
| clarkb | jeblair: I do note that the end of logging in the last rotated log coincides with when the files were log rotated | 17:24 |
| clarkb | jeblair: which makes me suspicious | 17:24 |
| jeblair | clarkb: ok, so bad logs... | 17:25 |
| jeblair | clarkb: are those being logrotated? | 17:25 |
| jeblair | clarkb: because i think that's incompatible with the python logging module | 17:25 |
| clarkb | jeblair: yes, I should fix that, it has been on a todo list to use python logging directly for that | 17:25 |
| clarkb | jeblair: its not incompatible, or shouldn't be at least | 17:25 |
| *** pcrews has joined #openstack-infra | 17:25 | |
| clarkb | it does a file copy then truncates the old file | 17:25 |
| clarkb | so fds don't change, but perhaps in some circumstances this makes python unhappy | 17:25 |
| jeblair | clarkb: i've never seen it work | 17:26 |
| *** amotoki_ has quit IRC | 17:26 | |
| jeblair | clarkb: anyway, reconsidering the evidence, it does look like crm is stuck on those hosts | 17:26 |
| fungi | i've definitely seen weird issues arise when pythonlogging and logrotate come together | 17:26 |
| jeblair | clarkb: the start time for the crm processes is right around the same time though | 17:27 |
| *** david-lyle has quit IRC | 17:28 | |
| *** morganfainberg is now known as morganfainberg_Z | 17:28 | |
| clarkb | I will make replacing logrotate with python logging a much higher priority | 17:28 |
| jeblair | clarkb: oh, looking more closely, the log lines are consistent with crm being stuck (the last log enty is pre-crm) | 17:28 |
| jeblair | clarkb: i think the log rotating may be a red herring after all and the logs may be correct | 17:29 |
| clarkb | cool | 17:29 |
| jeblair | clarkb: it's not super-accurate, but the crm processes are at least processing the same filenames as the most recent jobs the workers fetched | 17:30 |
| mordred | jeblair, clarkb: thanks. that's helpful! | 17:30 |
| jeblair | clarkb: (that is, 3 console logs and one syslog) | 17:31 |
| *** bknudson has joined #openstack-infra | 17:31 | |
| *** hogepodge has quit IRC | 17:31 | |
| pabelanger | I know I've asked in the past, but any chance up uploading horizon 2013.2 to pypi manually? I'm hitting the "Some externally hosted files were ignored" error in pip. | 17:32 |
| *** CaptTofu has joined #openstack-infra | 17:33 | |
| *** jcooley_ has quit IRC | 17:34 | |
| jeblair | clarkb: the crm processes still have the worker's fds open; could the logrotation have borked it? | 17:34 |
| *** jcooley_ has joined #openstack-infra | 17:34 | |
| clarkb | jeblair: maybe? I wouldn't expect it to because I am using the bad form of logrotation which shouldn't change any file descriptors | 17:34 |
| openstackgerrit | David Johansen proposed a change to openstack-infra/jenkins-job-builder: Added support for least load plugin. https://review.openstack.org/71316 | 17:34 |
| openstackgerrit | Dina Belova proposed a change to openstack-infra/config: Fix Climate jobs https://review.openstack.org/71317 | 17:35 |
| clarkb | but who knows what magicalness python expects under it | 17:35 |
| *** talluri has joined #openstack-infra | 17:35 | |
| jeblair | mordred: see pabelanger's question | 17:35 |
| jeblair | pabelanger: i don't think we currently upload any of the server projects, but are considering it. | 17:35 |
| clarkb | jeblair: I think the timestamps are definitely suspicious | 17:36 |
| Alex_Gaynor | Is there a plan to get the tripleo test runner back online? It makes looking at teh zuul status hard :-) | 17:36 |
| jeblair | pabelanger: currently the only distribution method we use for them is launchpad | 17:36 |
| pabelanger | jeblair, Ya, the conversation in the past was once horizon splits into a library we'd do it. However there is an already an older version of horizon uploaded | 17:36 |
| openstackgerrit | Dean Troyer proposed a change to openstack-infra/config: Gate openstackclient on DevStack https://review.openstack.org/71318 | 17:36 |
| *** krotscheck has joined #openstack-infra | 17:36 | |
| pabelanger | jeblair, I can work around it with --allow-external, but figured I'd ask again | 17:36 |
| mordred | yea - we were waiting until we had wheel uploading to start uploading the server projects | 17:37 |
| jeblair | pabelanger: it looks like it _isn't_ uploaded, but has a description that pip will follow | 17:37 |
| jeblair | Alex_Gaynor: lifeless &co are on it | 17:37 |
| Alex_Gaynor | cool | 17:37 |
| clarkb | jeblair: I will make fixing the logging to use python logging for rotation a bigger priority (work on it today probably) | 17:37 |
| clarkb | jeblair: as it should be fixed either way | 17:37 |
| pabelanger | jeblair, Oh, you are right. I _thought_ it was at one point. | 17:37 |
| jeblair | clarkb: i was wondering if the log rotation broke crm | 17:38 |
| jeblair | clarkb: since crm was holding those fds open | 17:38 |
| clarkb | jeblair: crm was holding which fds? the log files? | 17:38 |
| jeblair | clarkb: yes | 17:38 |
| jeblair | clarkb: and all of the worker's fds | 17:38 |
| *** bknudson has quit IRC | 17:38 | |
| jeblair | clarkb: at any rate, we should also probably have subprocess close extra fds | 17:38 |
| clarkb | because it is a fork, gotcha | 17:38 |
| *** atiwari has joined #openstack-infra | 17:39 | |
| devananda | zuul now says when it starts check jobs, too. nice touch! | 17:39 |
| clarkb | jeblair: the fds don't change, but crm is a special beast perhaps it noticed files got truncated and got cranky | 17:39 |
| *** Ajaeger has joined #openstack-infra | 17:39 | |
| jeblair | clarkb: yeah. it's all pretty weird. | 17:40 |
| clarkb | using python logging would possibly make it crankier as fds would change | 17:40 |
| jeblair | clarkb: crm is waiting on stdin. | 17:40 |
| clarkb | so ya, we should clean those up before the exec probably | 17:40 |
| jeblair | okay, i'll restart 05 and 06 now | 17:40 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: gerritbot: create an #openstack-merges channel https://review.openstack.org/71319 | 17:40 |
| clarkb | jeblair: thanks | 17:41 |
| fungi | devananda: we'd rather it didn't, but that's a workaround for gerrit not updating timestamps on plain votes | 17:41 |
| jeblair | heh, i guess that change is live now. :) | 17:41 |
| *** jcooley_ has quit IRC | 17:41 | |
| fungi | devananda: so this allows zuul to be able to tell how fresh its last check was | 17:41 |
| *** jcooley_ has joined #openstack-infra | 17:42 | |
| fungi | hmmm... i got an unstable result from check-tempest-dsvm-ironic-nv | 17:42 |
| *** yassine_ has quit IRC | 17:42 | |
| fungi | com.jcraft.jsch.JSchException: Session.connect: java.net.SocketException: Connection reset | 17:43 |
| *** yassine has joined #openstack-infra | 17:43 | |
| *** yassine has quit IRC | 17:43 | |
| jeblair | devananda: i assume you were noting https://review.openstack.org/#/c/68018/ ? | 17:44 |
| jeblair | clarkb, fungi, devananda: ^ that change is the new thing in action (it has been idle for 5 days until this morning) | 17:45 |
| fungi | it works! | 17:45 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Close unneeded fds before execing CRM 114. https://review.openstack.org/71320 | 17:46 |
| *** jasondotstar has quit IRC | 17:46 | |
| clarkb | jeblair: ^ completely untested but should do what we want. I need to step out now, but can test this more later | 17:46 |
| devananda | jeblair: wait, that's odd ... zuul started a check just because I posted a comment? | 17:46 |
| *** VijayT has joined #openstack-infra | 17:46 | |
| *** denis_makogon has quit IRC | 17:46 | |
| fungi | devananda: that's the intent | 17:46 |
| *** denis_makogon_ has joined #openstack-infra | 17:46 | |
| fungi | devananda: it had been at least 48 hours since that change's confession^Wcheck run | 17:47 |
| devananda | fungi: oh. that is a bit wierd | 17:47 |
| fungi | devananda: zuul reacts to the gerrit event stream... it saw your comment in the stream, noticed that the change you were commenting on hadn't been checked recently, and fired up a fresh check | 17:48 |
| devananda | fungi: i was referrign to https://review.openstack.org/#/c/70468/ -- zuul posted that it started a check in response to a new patch set, which makes sense | 17:48 |
| devananda | fungi: that seems like it's going to waste a lot of resources checking things when folks are just discussing a long-lived patch | 17:48 |
| jeblair | devananda: the goal is to make sure that the check results don't get stale; it will only check a patch once every 48 hours (later we will reduce it to 24h) | 17:49 |
| fungi | devananda: it's less of a waste of resources than people approving changes which had bitrotted | 17:49 |
| devananda | fungi: maybe not that many resources. i don't know how many patches get discussed longer than 48 hrs between revisions outside of ironic | 17:49 |
| fungi | devananda: and if discussion on a change goes silent for a week (or a month) then zuul won't be running rechecks on it anyway | 17:50 |
| devananda | gotcha | 17:50 |
| jeblair | devananda: triggering it on comment events means that changes no one cares about won't have this done to them, but actively developed and reviewed changes will have current check results | 17:50 |
| devananda | ok, seems reasonable :) | 17:50 |
| fungi | this way the devs/revs will notice fairly quickly while the patch is still under discussion if it suddenly ceases to become fit for approval (merge conflicts, bitrot, et cetera) | 17:51 |
| devananda | ++ | 17:51 |
| *** gokrokve has quit IRC | 17:51 | |
| fungi | so ultimately, this helps save reviewer and developer time, possibly at the expense of some additional test resources | 17:51 |
| *** gokrokve has joined #openstack-infra | 17:52 | |
| sdague | jeblair: so are you working with 1 time line, or 2 right now? | 17:52 |
| *** markwash has joined #openstack-infra | 17:52 | |
| *** hashar has quit IRC | 17:53 | |
| sdague | because when I was originally thinking of this, I was thinking about it as a soft limit and hard limit | 17:53 |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/config: XenServer installer script https://review.openstack.org/60581 | 17:53 |
| sdague | where soft limit would be something like 5 or 7 days, where we're do a refresh of results if there was any activity on the patch | 17:53 |
| sdague | and a hard limit of 24 hrs which would for a rerun of check before it goes into the gate on +A | 17:54 |
| fungi | sdague: the accompanying change does the latter | 17:54 |
| sdague | ok | 17:54 |
| sdague | so the idle sweep / soft limit, is still tbd? | 17:54 |
| sdague | or is there thinking not to do that? | 17:55 |
| openstackgerrit | David Johansen proposed a change to openstack-infra/jenkins-job-builder: Added support for least load plugin. https://review.openstack.org/71316 | 17:55 |
| sdague | (also, excuse my slowness, weird timezones, and jetlag, and up in the middle of the night :) ) | 17:55 |
| fungi | sdague: https://review.openstack.org/68817 | 17:56 |
| *** dhellmann is now known as dhellmann_ | 17:56 | |
| *** hogepodge has joined #openstack-infra | 17:56 | |
| *** gokrokve has quit IRC | 17:56 | |
| sdague | ok, gotcha great | 17:56 |
| Ajaeger | infra team, I'm coming back to yesterday's short discussion on using tox for building the localized manuals. | 17:56 |
| *** pblaho has quit IRC | 17:57 | |
| jeblair | sdague: i think the current implementation requires they be matched (eg, check queue is trigger by change older than X, gate queue requires change newer than X); i don't think the current approach would work with two values | 17:57 |
| sdague | jeblair: ok | 17:57 |
| sdague | so it might be nice to also have the soft limit later | 17:57 |
| Ajaeger | We've hit now one problem: generatedocbook uses xml2po - and that one is installed on the system but not in the virtualenv. How do I get it into the venv? | 17:57 |
| sdague | but maybe it's an optimization we don't need | 17:57 |
| Ajaeger | pip install xml2po does not work ;( | 17:57 |
| jeblair | sdague: i'll see if my subconcious comes up with anything | 17:58 |
| *** DeeJay1 has joined #openstack-infra | 17:58 | |
| fungi | sdague: yeah, i say we watch how things play out with just rechecking everything daily if it's got activity, and requiring the same for approval to work | 17:58 |
| sdague | it does also bring up the other thing we talked about a while ago which would be really handy, which is to do a new merge conflict check on all the patches in a project after one merges | 17:58 |
| openstackgerrit | Bhuvan Arumugam proposed a change to openstack-infra/jeepyb: Load project settings from ini file, if present. https://review.openstack.org/71323 | 17:58 |
| sdague | because we send way too many merge conflict patches from check -> gate | 17:58 |
| fungi | sdague: i also thing that merge conflict check is probably less necessary when active patches are getting rechecked daily | 17:59 |
| jeblair | sdague: i think that's the thing new gerrit can do; however, merge conflicts are cheap for us to detect | 17:59 |
| fungi | s/thing/think | 17:59 |
| jeblair | sdague: (and about to get cheaper with the army of zuul mergers i'm raising) | 17:59 |
| sdague | jeblair: right | 17:59 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Fix list of core members due to Climate core team clean-up https://review.openstack.org/71215 | 17:59 |
| jeblair | Ajaeger: the virtualenv should only affect python packages, not binaries | 18:00 |
| fungi | jeblair: you and your press gang will shanghai them into service? | 18:00 |
| Ajaeger | jeblair: xml2po is both a binary and a python package. | 18:00 |
| sdague | fungi: so the reason I think it's good, is it's very frustrating as a reviewer to spend time +2 code, then Jenkins goes an -1s it on a new check. It would ensure that all the code that reviewers looked at would be able to be merged | 18:00 |
| *** derekh has quit IRC | 18:00 | |
| sdague | which would also I think help reviewer morale | 18:00 |
| Ajaeger | jeblair: generatedocbook does a "import xml2po" - but what should I add as requirement? | 18:01 |
| sdague | still a race window based on changes in the gate, but it's remarkable how often 10% of gate is merge conflicts | 18:01 |
| sdague | or even higher | 18:01 |
| jeblair | sdague: agreed; but i say we go with the incremental improvement that the 48h check will get us and wait to see if gerrit 2.8 fills the rest of the void before we do extra work to deal with that on our own | 18:01 |
| sdague | yep, definitely | 18:02 |
| Ajaeger | it comes with gnome-doc-utils on my system ;( | 18:02 |
| sdague | just wanted to refresh the idea and make sure it wasn't entirely invalid | 18:02 |
| jeblair | Ajaeger: looking | 18:03 |
| Ajaeger | Should I just add "sitepackages=True" to the tox.ini? | 18:03 |
| jeblair | Ajaeger: that is the easiest solution but can bring other problems | 18:03 |
| jeblair | Ajaeger: is generatedocbook our tool or someone elses? | 18:04 |
| Ajaeger | It's our tool - part of openstack-doc-tools package | 18:05 |
| *** gyee has quit IRC | 18:05 | |
| *** jasondotstar has joined #openstack-infra | 18:05 | |
| jeblair | Ajaeger: another option would be to avoid importing it and only using it as a library | 18:05 |
| bhuvan | fungi: as discussed, rewritten jeepyb.utils to use ini file https://review.openstack.org/#/c/71323/ and cleaned the config parser code in rest of scripts | 18:05 |
| *** igor__ has joined #openstack-infra | 18:05 | |
| jeblair | Ajaeger: a third option is that we start building it and uploading it to pypi | 18:05 |
| jeblair | Ajaeger: so there are 3 bad options! :) | 18:05 |
| Ajaeger | sitepackages does not work - it tries to install packages into /usr/lib on my system. | 18:06 |
| Ajaeger | how can I only use it as library? Do you have a pointer? | 18:06 |
| jeblair | Ajaeger: that's not right; it doesn't change where it's writing | 18:06 |
| *** andreaf has quit IRC | 18:06 | |
| *** cody-somerville has quit IRC | 18:07 | |
| jeblair | Ajaeger: http://paste.openstack.org/show/62627/ | 18:08 |
| *** morganfainberg_Z is now known as morganfainberg | 18:08 | |
| fungi | bhuvan: great--thanks! i still need to check and see if the pull-request closer was fixed by clarkb's change or if it merely got us back to the earlier problem i was seeing on it | 18:08 |
| jeblair | Ajaeger: setting sitepackages=true should just do that | 18:08 |
| clarkb | fungi: bhuvan: my change is something to build on. Ideally everything uses the jeepyb.projects import | 18:08 |
| clarkb | my change just fixes status quo in existing yaml reads | 18:09 |
| fungi | clarkb: yep, totally | 18:09 |
| *** freyes has joined #openstack-infra | 18:09 | |
| fungi | stop the bleeding hopefully | 18:09 |
| clarkb | ya | 18:09 |
| clarkb | ok back to afk | 18:09 |
| *** david-lyle has joined #openstack-infra | 18:09 | |
| * fungi needs to step away for a bit, but should bbrsn | 18:09 | |
| bhuvan | fungi: ok. fwiw i tested changes to utils locally. i didn't test the scripts, as i couldn't setup gerrit in my vm using puppet | 18:10 |
| *** igor__ has quit IRC | 18:10 | |
| jeblair | clarkb, fungi, lifeless: what's the state on nodepool deletes? i think we are at node exhaustion due to that. | 18:10 |
| bhuvan | clarkb: in 71323, i've rewritten to use jeepyb.utils.ProjectRegistry() for everything ... jeepyb.projects already use jeepyb.utils | 18:11 |
| *** yassine has joined #openstack-infra | 18:11 | |
| *** nati_ueno has quit IRC | 18:11 | |
| Ajaeger | jeblair: that's my problem: http://paste.openstack.org/show/62629/ | 18:12 |
| *** rossella_s has quit IRC | 18:12 | |
| *** hogepodge has quit IRC | 18:14 | |
| jeblair | clarkb, mordred: can you look at that ^ | 18:15 |
| *** gokrokve has joined #openstack-infra | 18:15 | |
| jeblair | Ajaeger: i don't immediately understand why it's trying to do that; it's definitely not supposed to happen | 18:15 |
| *** gokrokve has quit IRC | 18:17 | |
| *** gokrokve has joined #openstack-infra | 18:17 | |
| *** emagana has quit IRC | 18:18 | |
| *** pmoosh has joined #openstack-infra | 18:18 | |
| fungi | clarkb: jeblair: lifeless: fwiw, snapshot of the amount of time current nodes have been waiting in a delete state, summarized by 0.1 hour increments... http://paste.openstack.org/show/62631/ | 18:18 |
| *** gokrokve_ has joined #openstack-infra | 18:19 | |
| *** jcooley_ has quit IRC | 18:19 | |
| *** talluri has quit IRC | 18:21 | |
| Ajaeger | jeblair: ok, let me talk with our local python experts to figure out what's wrong... | 18:21 |
| Ajaeger | jeblair: thanks | 18:21 |
| *** jcooley_ has joined #openstack-infra | 18:21 | |
| *** burt1 has joined #openstack-infra | 18:21 | |
| jeblair | Ajaeger: sorry; it should work. i believe nova uses it. | 18:21 |
| *** gokrokve has quit IRC | 18:22 | |
| Ajaeger | jeblair: it should indeed, will debug now- thanks | 18:22 |
| *** freyes has quit IRC | 18:23 | |
| *** jcooley_ has quit IRC | 18:23 | |
| *** jcooley_ has joined #openstack-infra | 18:23 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Clean up the tmp directories - NOT READY FOR REVIEW https://review.openstack.org/71331 | 18:25 |
| *** freyes has joined #openstack-infra | 18:26 | |
| clarkb | fungi: jeblair: lifeless has a change up to add more logging to nodepool so we can debug the deletes | 18:26 |
| * fungi finds | 18:26 | |
| clarkb | https://review.openstack.org/#/c/71137/ | 18:26 |
| fungi | thanks | 18:26 |
| openstackgerrit | Dina Belova proposed a change to openstack-infra/config: Fix Climate jobs https://review.openstack.org/71317 | 18:27 |
| fungi | clarkb: lgtm, though we'll need to hand-patch that and restart nodepool for it to help | 18:28 |
| clarkb | ya | 18:29 |
| jeblair | fungi: hand-patch? | 18:29 |
| clarkb | well git update by hand | 18:29 |
| clarkb | (puppet isn't running) | 18:29 |
| fungi | or i can temporarily whip up a change to officially rip the tripleo-ci provider back out of nodepool.yaml and start puppet back up. it's been several days now | 18:29 |
| jeblair | fungi: let's do that too | 18:30 |
| fungi | on it | 18:30 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Also log provider name when debugging deletes. https://review.openstack.org/71137 | 18:30 |
| *** bookwar has quit IRC | 18:31 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Log state names not numbers https://review.openstack.org/71333 | 18:32 |
| *** nicedice has joined #openstack-infra | 18:32 | |
| fungi | my status questions in #tripleo as to the progress on the rebuild have been falling on deaf ears, so i assume lifeless is the only one who knows what's going on with it | 18:32 |
| *** hogepodge has joined #openstack-infra | 18:32 | |
| jeblair | clarkb: can you aprv that one too? ^ :) | 18:33 |
| *** zul has quit IRC | 18:33 | |
| fungi | oh, i completely missed that was the numeric keys not the names | 18:34 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Clean up the tmp directories - NOT READY FOR REVIEW https://review.openstack.org/71331 | 18:34 |
| *** weshay_bbiab is now known as weshay | 18:36 | |
| *** zul has joined #openstack-infra | 18:37 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Temporarily remove tripleo-ci nodepool provider https://review.openstack.org/71334 | 18:37 |
| *** yassine has quit IRC | 18:39 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Log state names not numbers https://review.openstack.org/71333 | 18:40 |
| *** bknudson has joined #openstack-infra | 18:41 | |
| *** nati_ueno has joined #openstack-infra | 18:41 | |
| *** AlexF has quit IRC | 18:41 | |
| dims | fungi, clarkb - what does the "66 hr 8 min" on the first job of the check queue mean? we have not been able to run it for that long? | 18:44 |
| fungi | dims: tripleo changes need special nodes from their cloud, and it's been down for a few days | 18:45 |
| fungi | dims: so zuul is waiting for workers in their cloud to pick up those jobs | 18:46 |
| *** AlexF has joined #openstack-infra | 18:46 | |
| dims | fungi, ah. thanks. got scared looking at the number :) | 18:46 |
| clarkb | jeblair: I am watching parade stuff now sorry | 18:47 |
| clarkb | also there are a ton of people | 18:47 |
| SergeyLukjanov | clarkb, fungi, could you, please, take a look / approve https://review.openstack.org/#/c/68066/ ? :) the direction was +1'd by sdague | 18:47 |
| wenlock | jeblair: thanks for the review on https://review.openstack.org/#/c/51425/26 , much appreciate it! guess we need more before it gets approved, right? | 18:48 |
| jeblair | wenlock: yeah, we'll need at least one other infra-core reviewer, and someone to babysit the change and look for errors. unfortunately our puppet dashboard broke and no one has been able to fix it yet so that means someone has to log into servers and look for errors manually. | 18:49 |
| *** jnoller_ has joined #openstack-infra | 18:50 | |
| jeblair | it looks like the check queue is now 25 minutes behind due to node starvation | 18:51 |
| wenlock | jeblair, yep thats understandable. We use the code on our side so it should go fairly smooth i think | 18:51 |
| *** jnoller has quit IRC | 18:51 | |
| jeblair | wenlock: excellent! | 18:51 |
| *** apevec has quit IRC | 18:51 | |
| *** ryanpetrello has quit IRC | 18:52 | |
| *** ryanpetrello has joined #openstack-infra | 18:52 | |
| clarkb | jeblair: finally managed to pull up your nodepool change lgtm | 18:54 |
| *** markmcclain has quit IRC | 18:55 | |
| fungi | clarkb: jeblair: any objections on https://review.openstack.org/71334 ? | 18:55 |
| jog0 | "Starting check jobs. http://status.openstack.org/zuul/ | 18:55 |
| jog0 | ? | 18:55 |
| fungi | i'll go ahead and start doing mass deletes of nodes which are hanging around in delete state, then get ready to start puppet and restart nodepool | 18:56 |
| jeblair | jog0: see scrollback with devananda | 18:56 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Clean up the tmp directories - NOT READY FOR REVIEW https://review.openstack.org/71331 | 18:56 |
| jeblair | fungi: lgtm | 18:57 |
| jog0 | jeblair: ahh | 18:57 |
| clarkb | fungi lgtm | 18:57 |
| jog0 | jeblair: should it only do that if there hasn't been any activity in a while? | 18:58 |
| clarkb | barely any bits flowing so many people | 18:58 |
| jog0 | or will it always do that | 18:58 |
| jeblair | jog0: (hopefully we can remove the comment spam in a few weeks, but it's necessary while we work on better implementations) | 18:58 |
| jeblair | jog0: it triggers on comments on changes where check hasn't run within 48 hours | 18:58 |
| jog0 | jeblair: ahh, that makes more sense now, thanks | 18:58 |
| jog0 | https://review.openstack.org/#/c/69799/ | 18:58 |
| jog0 | was where I first saw it, where it ran on a new patchset | 18:59 |
| jeblair | jog0: oh, sorry, the comment is always posted when check jobs start (which still happen on new patchsets) | 18:59 |
| fungi | jog0: though the "starting check jobs" comment is new spam we need to be able to support that because currently gerrit does not update timestamps unless an actual comment is left--just resetting a vote leaves the timestamp untouched | 18:59 |
| jeblair | jog0: but additionally check jobs are now triggered when someone posts a comment to an old change | 18:59 |
| jog0 | fungi: ohh gerrit | 19:00 |
| jog0 | jeblair: neato | 19:00 |
| fungi | apparently voting is not "activity" but comments are. wth | 19:00 |
| jeblair | jog0: yeah, we will either fix gerrit or work around it. possibly both. | 19:00 |
| *** jnoller_ is now known as jnoller | 19:01 | |
| openstackgerrit | A change was merged to openstack-infra/config: Enable tempest/savanna gate tests https://review.openstack.org/68066 | 19:01 |
| *** hemna has quit IRC | 19:01 | |
| *** che-arne has quit IRC | 19:02 | |
| fungi | while i'm waiting on these deletes, i suppose i should look for other nodepool changes we might want in this restart | 19:02 |
| *** harlowja_away is now known as harlowja | 19:05 | |
| *** AlexF has quit IRC | 19:05 | |
| portante | fungi, clarkb, sdague: is there a graph, or at least data on how many patches in the gate have landed all at once? | 19:05 |
| *** igor__ has joined #openstack-infra | 19:06 | |
| clarkb | portante see openstack/openstack repo | 19:06 |
| clarkb | you get exact commits too | 19:06 |
| jeblair | portante: also you could probably get numbers from graphite | 19:06 |
| *** b3nt_pin has joined #openstack-infra | 19:06 | |
| *** b3nt_pin is now known as beagles | 19:06 | |
| *** weshay has quit IRC | 19:07 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Use venv to build documentation https://review.openstack.org/65987 | 19:07 |
| *** burt1 has quit IRC | 19:07 | |
| *** jcooley_ has quit IRC | 19:07 | |
| *** jcooley_ has joined #openstack-infra | 19:08 | |
| openstackgerrit | A change was merged to openstack-infra/config: Temporarily remove tripleo-ci nodepool provider https://review.openstack.org/71334 | 19:09 |
| *** igor__ has quit IRC | 19:10 | |
| morganfainberg | out of curiosity, would it make sense if reviewday lowered the score on ancient WIP reviews [mostly the WIP part]? It seems odd some are very high priority. | 19:11 |
| *** balar has joined #openstack-infra | 19:12 | |
| fungi | pleia2: dprince: ^ | 19:12 |
| pleia2 | morganfainberg: seems reasonable to me, I haven't looked at the logic for it in a while, can you submit a bug? | 19:12 |
| *** emagana has joined #openstack-infra | 19:13 | |
| morganfainberg | pleia2, absolutely. just noticed for example: keystone has https://review.openstack.org/#/c/47480/1 at the top of the priority. i don't even know if it's relevant anymore | 19:13 |
| dprince | morganfainberg: Yes. So WIP reviews should always be at the bottom of the queue I think. They just sort of slipped in there | 19:13 |
| morganfainberg | dprince, cool. will get a bug filed | 19:14 |
| morganfainberg | against openstack-core-infra? | 19:14 |
| morganfainberg | openstack-ci in LP | 19:14 |
| morganfainberg | that is | 19:14 |
| dprince | morganfainberg: lets just do it! | 19:15 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools for openstack-operations-guide-{lang} https://review.openstack.org/71056 | 19:15 |
| * dprince doesn't review the reviewday bugs anyways | 19:15 | |
| *** coolsvap is now known as coolsvap_away | 19:15 | |
| morganfainberg | dprince, hehe well i'll see if i can quickly provide a fix. I am hoping to add reviewday json support to dolphm's nextreview once the json is available publically | 19:15 |
| openstackgerrit | A change was merged to openstack-infra/config: Don't run non-voting gate-grenade-dsvm-neutron https://review.openstack.org/67485 | 19:16 |
| fungi | i think a side effect of the latest nodepoold deletion changes makes it such that 'nodepool delete' out of band won't get rid of nodes until nodepoold gets around to taking care of them itself anyway | 19:20 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Fix up `zuul --version` https://review.openstack.org/66629 | 19:20 |
| jeblair | fungi: why is that? | 19:20 |
| *** pmoosh has quit IRC | 19:20 | |
| jeblair | fungi: is it because 'nodepool delete' is now a no-op? since deletion is only in the cron? | 19:20 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Log gerrit comment https://review.openstack.org/70287 | 19:20 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove unused main in elasticRecheck.py https://review.openstack.org/70471 | 19:20 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Convert failed_job into an object https://review.openstack.org/70537 | 19:20 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Map failed jobs to bugs in gerrit comment https://review.openstack.org/71148 | 19:20 |
| fungi | jeblair: that's my suspicion, because i did a 'nodepool delete' across all the nodes listed as being in a delete state, and when it was done there were even more than when i started | 19:21 |
| openstackgerrit | A change was merged to openstack-infra/config: Run tempest-dsvm-postgres-full with nova-api-metadata binary https://review.openstack.org/68305 | 19:22 |
| fungi | i'm doing one more pass just to make sure | 19:23 |
| *** weshay has joined #openstack-infra | 19:24 | |
| jog0 | clarkb: it looks like elasticSearch isn't getting any console data | 19:25 |
| openstackgerrit | A change was merged to openstack/requirements: Depending on python-swiftclient>=1.6 https://review.openstack.org/70935 | 19:25 |
| jog0 | filename:"console.html" | 19:25 |
| jog0 | shows no hits in last 15 minutes | 19:26 |
| clarkb | yes see sb | 19:26 |
| clarkb | it is about 5 hours behind. will add workers after parade | 19:26 |
| jeblair | clarkb: i think they stopped again | 19:27 |
| clarkb | really? :( | 19:27 |
| jeblair | clarkb: and not long after starting | 19:27 |
| clarkb | want to merge my patch and see if it helps? | 19:27 |
| jog0 | clarkb: ahh thanks | 19:28 |
| jeblair | clarkb: without logrotate to blame for triggering it, i don't expect it to help | 19:28 |
| clarkb | ok | 19:28 |
| jeblair | clarkb: i'll give it a shot but i'm not hopeful | 19:29 |
| clarkb | how is disk on the nodes? crm needs disk right? | 19:29 |
| jeblair | clarkb: 10% | 19:30 |
| *** igor___ has joined #openstack-infra | 19:32 | |
| *** johnthetubaguy has quit IRC | 19:34 | |
| SergeyLukjanov | looks like propose-requirements-updates is still failing | 19:34 |
| *** Ajaeger has quit IRC | 19:35 | |
| fungi | did clarkb's proposed fix for that merge yet? | 19:35 |
| clarkb | I dont think so | 19:36 |
| *** igor___ has quit IRC | 19:37 | |
| fungi | right, https://review.openstack.org/69969 | 19:38 |
| fungi | i was hoping mordred would have time to chime in on that | 19:38 |
| bknudson | there is something wacky going on -- https://review.openstack.org/#/c/66247/ | 19:38 |
| bknudson | says it's gating | 19:38 |
| bknudson | but I don't think anyone +A ? | 19:38 |
| fungi | bknudson: it says "Starting check jobs." | 19:38 |
| bknudson | ahh, not gating. | 19:38 |
| bknudson | thanks | 19:38 |
| fungi | bknudson: see discussion with devananda, sdague and jog0 in scrollback | 19:39 |
| dkehn | clarkb: in running tox -py27 locally I'm see errors with the commands = ... --testr-args='{posargs'}, when I remmove wroks, what am I missing | 19:39 |
| fungi | bknudson: but in short, the additional comment spam is needed, at least temporarily, to support new freshness checks from zuul | 19:39 |
| clarkb | dkehn downgrade tox | 19:39 |
| dkehn | clarkb: yah thx | 19:39 |
| clarkb | they did a release and its broken | 19:39 |
| bknudson | fungi: ok, I was just freaking out! | 19:40 |
| clarkb | 1.6.1 is the version you want | 19:40 |
| fungi | dkehn: or join us in forking tox over https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror ;) | 19:40 |
| clarkb | if we fork tox can we call it detox? | 19:40 |
| * fungi checks to see if that's taken | 19:40 | |
| openstackgerrit | A change was merged to openstack-infra/config: Close unneeded fds before execing CRM 114. https://review.openstack.org/71320 | 19:40 |
| *** odyssey4me has quit IRC | 19:40 | |
| *** markmcclain has joined #openstack-infra | 19:40 | |
| fungi | yep, it's taken... https://pypi.python.org/pypi/detox | 19:41 |
| fungi | "detox is the distributed version of tox..." | 19:41 |
| clarkb | :( I wonder if detox is sane as an existing fork | 19:42 |
| fungi | clarkb: they share one bitbucket account, so probably not | 19:43 |
| fungi | and "Author: holger krekel" | 19:44 |
| *** bhuvan has quit IRC | 19:44 | |
| *** shivh has joined #openstack-infra | 19:45 | |
| clarkb | what about botox >_> | 19:45 |
| fungi | toxin | 19:46 |
| dims | fungi, looks like Daniel has merged the bug we found in libvirt (https://bugzilla.redhat.com/show_bug.cgi?id=929412) - is there a vm image i can download (as close to what we deploy) so i can compile the libvirt from sources by hand to try stuff out | 19:46 |
| uvirtbot | dims: Error: Could not parse XML returned by bugzilla.redhat.com: HTTP Error 404: Not Found | 19:46 |
| fungi | clarkb: pypi and apt-cache searches turn up no hits for toxin | 19:46 |
| openstackgerrit | Dolph Mathews proposed a change to openstack-infra/elastic-recheck: add query for bug 1276778 https://review.openstack.org/71357 | 19:46 |
| uvirtbot | Launchpad bug 1276778 in tempest "test_lock_unlock_server: failed to reach ACTIVE status and task state "None" within the required time" [Undecided,New] https://launchpad.net/bugs/1276778 | 19:46 |
| openstackgerrit | Morgan Fainberg proposed a change to openstack-infra/reviewday: Ensure WORKINPROGRESS reviews are at the bottom of the list. https://review.openstack.org/71358 | 19:47 |
| fungi | dims: the closest i can offer is the instructions in devstack-gate's readme.rst | 19:47 |
| dims | fungi, k thx | 19:47 |
| fungi | dims: the providers where we're currently running jobs don't provide a means to download (or upload) server images | 19:48 |
| dims | fungi, thanks. understood | 19:48 |
| jeblair | clarkb: crm is a red herring; it's stuck sending data to logstash | 19:50 |
| jeblair | clarkb: (i'm doing foregroud debugging) | 19:52 |
| clarkb | jeblsir: rgr. possible elasticsearch backup then | 19:52 |
| clarkb | if itvfell into gc hell thst could explain it or network fail splitting the cluster | 19:53 |
| SergeyLukjanov | jeblair, it looks like zuul is still failing unit tests sometime, is it ok if I'll create an issue for it? | 19:54 |
| *** e0ne has joined #openstack-infra | 19:55 | |
| *** davidhadas has joined #openstack-infra | 19:56 | |
| markmcclain | the gate is seeing name resolution failures for git.openstack.org | 19:57 |
| markmcclain | https://jenkins03.openstack.org/job/gate-devstack-dsvm-cells/798/console | 19:57 |
| *** hogepodge has quit IRC | 19:57 | |
| fungi | i wonder if conficker traffic is bringing the resolvers at our providers to their knees | 19:59 |
| jeblair | SergeyLukjanov: yep! | 20:00 |
| fungi | markmcclain: bug 1270382 | 20:00 |
| SergeyLukjanov | jeblair, ok, I'm trying to understand the errors in logs now | 20:00 |
| uvirtbot | Launchpad bug 1270382 in openstack-ci "Name resolution failure on gate job" [Medium,Incomplete] https://launchpad.net/bugs/1270382 | 20:00 |
| lifeless | Alex_Gaynor: yes, am on it; trunk being broken didn't help :( | 20:02 |
| fungi | markmcclain: logstash says 10 incidents of that in the past 24 hours | 20:02 |
| *** igor___ has joined #openstack-infra | 20:03 | |
| *** igor____ has joined #openstack-infra | 20:05 | |
| fungi | clarkb: jeblair: nodepool.o.o has config updates and additional debugging patches in place. i'm restarting the daemon now, unless there are objections | 20:05 |
| markmcclain | fungi: thanks for the link | 20:05 |
| fungi | markmcclain: i wish i had a better answer... moar internets? | 20:06 |
| fungi | the one you linked happened in rackspace, so i'll see if their status page says anything about dns issues | 20:07 |
| jeblair | fungi: no objections | 20:07 |
| *** igor___ has quit IRC | 20:07 | |
| fungi | on the assumption that clarkb will also not object, restarting it now ;) | 20:08 |
| markmcclain | fungi: looking back 7 days…. there are 34 incidents on rax.. must be some kind of transient error | 20:09 |
| markmcclain | haven't seen it on HP | 20:09 |
| *** mrodden has quit IRC | 20:09 | |
| fungi | markmcclain: probably not related, but "02/05/14 01:01 PM EST Our Cloud engineering team is investigating an issue impacting a portion of Next Gen Cloud Server customers attempting to build new server instances from custom images. During this time, customers may be experiencing extended delays." | 20:09 |
| *** hogepodge has joined #openstack-infra | 20:09 | |
| fungi | markmcclain: what's extra sad is that rackspace is where we host that domain. so you'd think anywhere had trouble not resolving those records it would be other providers | 20:10 |
| markmcclain | fungi: that is a good point | 20:11 |
| *** julim has joined #openstack-infra | 20:11 | |
| fungi | markmcclain: one thing we *could* consider would be adding a local caching resolver daemon on each slave and having the first step of the job be to prime that cache with multiple resolution retries | 20:13 |
| jeblair | clarkb: i've tracked the pusher problem to "self.logq.put(out_event)". that makes me suspicious that we're filling the queue. i'll debug the other side of the queue after lunch | 20:13 |
| jeblair | clarkb: but we should start considering that logstash is the problem | 20:13 |
| *** yolanda has quit IRC | 20:13 | |
| *** markmcclain has quit IRC | 20:15 | |
| *** thuc_ has quit IRC | 20:16 | |
| *** thuc has joined #openstack-infra | 20:17 | |
| *** mrodden has joined #openstack-infra | 20:17 | |
| *** thuc has quit IRC | 20:21 | |
| *** gokrokve_ has quit IRC | 20:21 | |
| *** DeeJay1 has quit IRC | 20:21 | |
| *** gokrokve has joined #openstack-infra | 20:21 | |
| fungi | since the nodepool restart has been waiting for about 15 minutes now, i'm going to kill it for the sake of expediency and clean up the mess instead | 20:24 |
| *** gokrokve has quit IRC | 20:26 | |
| fungi | #status note nodepoold restarted with additional debugging patches | 20:27 |
| clarkb | jeblair: I just double checked elasticsearch and it appears to be happy | 20:27 |
| clarkb | feeling is starting to return to my toes and fingers | 20:28 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-dev/pbr: Do not force log verbosity level to info https://review.openstack.org/71374 | 20:29 |
| clarkb | jeblair: did you change someting? a minute ago or so the bigdesk index requests graphs spiked | 20:31 |
| *** hogepodge has quit IRC | 20:32 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add zuul_version to status.json samples https://review.openstack.org/66043 | 20:34 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Display zuul_version in zuul status page https://review.openstack.org/66052 | 20:34 |
| *** freyes has quit IRC | 20:37 | |
| *** thuc has joined #openstack-infra | 20:38 | |
| clarkb | jeblair: restarting the logstash indexer seems to help. unfortunately they don't log very verbosely | 20:42 |
| clarkb | jeblair: I did that on 06, I can go ahead and do it for the others, but will wait to see if you want ot capture anything else | 20:43 |
| *** hashar has joined #openstack-infra | 20:43 | |
| zaro | clarkb: hey, i double checked that regex and it works. talk about this https://review.openstack.org/#/c/70865 | 20:48 |
| clarkb | zaro: ? | 20:49 |
| zaro | clarkb: loaded up yaml and verified pattern match with re.match(..) on a clean ubuntu precise vm. | 20:49 |
| *** rlandy has joined #openstack-infra | 20:50 | |
| zaro | clarkb: not sure why i thought it didn't match before, i think i must have been doing something wrong. | 20:50 |
| clarkb | jeblair: I am stracing the working on logstash-worker05 | 20:51 |
| zaro | clarkb: i'm not sure why zuul-dev is not doing the recheck, do you have any ideas what i should try next? | 20:51 |
| clarkb | zaro: probably need to try a recheck and capture the logs for that then digi nto zuul | 20:52 |
| zaro | clarkb: unfortunately i don't have access to zuul-dev. | 20:52 |
| clarkb | jeblair: a lot of epoll waiting and futex waits. I think it may have gotten deadlocked inside logstash | 20:54 |
| *** kraman has quit IRC | 20:56 | |
| clarkb | jeblair: you can find the trace files at logstash-worker05:~clarkb/traces | 20:56 |
| *** kraman has joined #openstack-infra | 20:56 | |
| *** sarob has joined #openstack-infra | 20:57 | |
| *** kraman has quit IRC | 20:57 | |
| *** kraman has joined #openstack-infra | 20:57 | |
| clarkb | jeblair: is there any other data we should collect before restarting the indexers? | 20:57 |
| *** sarob has joined #openstack-infra | 20:57 | |
| clarkb | jeblair: going to grab lunch while I wait to hear back from you. | 20:59 |
| clarkb | fwiw I have not seen this behavior before, typically the backups happened in elasticsearch when gc got out of hand | 20:59 |
| *** amcrn has joined #openstack-infra | 21:00 | |
| clarkb | zaro: ya, I can do that for you after lunch | 21:01 |
| clarkb | zaro: did you see the parade? | 21:02 |
| clarkb | zaro: 700k people according to SPD | 21:02 |
| *** oubiwann_ has quit IRC | 21:02 | |
| *** nati_ueno has quit IRC | 21:02 | |
| *** oanufriev has joined #openstack-infra | 21:03 | |
| *** malini is now known as malini_afk | 21:04 | |
| *** nati_ueno has joined #openstack-infra | 21:04 | |
| *** melwitt has joined #openstack-infra | 21:05 | |
| *** dprince has quit IRC | 21:05 | |
| zaro | clarkb: no but that's insane! did you catch it? | 21:06 |
| *** CaptTofu_ has joined #openstack-infra | 21:06 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Display last_reconfigured in zuul status page https://review.openstack.org/66053 | 21:06 |
| clarkb | yup, it was cold | 21:06 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: give zaro access to zuul-dev server https://review.openstack.org/71380 | 21:07 |
| zaro | clarkb: ^ | 21:08 |
| *** CaptTofu has quit IRC | 21:09 | |
| *** DinaBelova is now known as DinaBelova_ | 21:09 | |
| *** rfolco has quit IRC | 21:10 | |
| clarkb | wow they are asking people to stop using their phones now because ETOOMANYPHONES | 21:11 |
| *** hogepodge has joined #openstack-infra | 21:11 | |
| zaro | haha! wonder if it's the data or voice? i'm guessing too much facebook or instagram. | 21:13 |
| *** rlandy has quit IRC | 21:14 | |
| *** mrda_away is now known as mrda | 21:17 | |
| *** dripton has quit IRC | 21:17 | |
| mattoliverau | morning | 21:21 |
| *** oubiwann_ has joined #openstack-infra | 21:22 | |
| *** dripton has joined #openstack-infra | 21:22 | |
| *** smarcet has left #openstack-infra | 21:23 | |
| notmyname | is it possible to search by group in gerrit? | 21:24 |
| *** blamar has joined #openstack-infra | 21:25 | |
| *** dcramer_ has joined #openstack-infra | 21:26 | |
| *** blamar has quit IRC | 21:26 | |
| *** blamar has joined #openstack-infra | 21:26 | |
| fungi | notmyname: search by what kind of group? user groups? | 21:28 |
| notmyname | fungi: ya, eg I'd like to search for something like reviewer:swift-core | 21:28 |
| notmyname | (which doesn't work) | 21:28 |
| zaro | notmyname: not available on review.o.o, but is available in review-dev.o.o | 21:29 |
| fungi | oh, like search for changes where the owner is a member of a group? | 21:29 |
| zaro | it's 'ownerin' criteria, available in newer gerrit. | 21:29 |
| notmyname | fungi: ya, essentially I want to find stuff that hasn't been reviewed by a core memeber | 21:29 |
| fungi | zaro: oh, they added that in 2.8? wonderful! | 21:29 |
| notmyname | ok | 21:29 |
| zaro | here's the doc on it https://gerrit-review.googlesource.com/Documentation/user-search.html | 21:30 |
| zaro | jeblair: ping | 21:30 |
| jeblair | clarkb: okay, yeah i'm not in the mood to debug logstash, so let's just restart 05 | 21:33 |
| jeblair | zaro: pong | 21:33 |
| shivh | fungi: requested new service account as suggested. seems email did not work yesterday, no moderator response, so created a bug report. anything else i should do? | 21:34 |
| *** nati_ueno has quit IRC | 21:35 | |
| *** jhesketh__ has joined #openstack-infra | 21:35 | |
| jhesketh_ | krtaylor: ping | 21:36 |
| jhesketh__ | Morning all | 21:36 |
| zaro | jeblair: want to ask a few questions about proposed 'workflow' label, https://etherpad.openstack.org/p/gerrit-workflow-label | 21:36 |
| *** oubiwann_ has quit IRC | 21:37 | |
| *** oubiwann_ has joined #openstack-infra | 21:37 | |
| jeblair | zaro: answers good? | 21:39 |
| zaro | jeblair: yes, thanks. | 21:40 |
| *** paul-- has quit IRC | 21:42 | |
| *** sparkycollier has joined #openstack-infra | 21:42 | |
| *** markmcclain has joined #openstack-infra | 21:43 | |
| *** alexpilotti has quit IRC | 21:43 | |
| *** markmcclain1 has joined #openstack-infra | 21:43 | |
| *** alexpilotti has joined #openstack-infra | 21:45 | |
| *** alexpilotti has quit IRC | 21:47 | |
| *** markmcclain has quit IRC | 21:47 | |
| *** hashar has quit IRC | 21:48 | |
| krtaylor | jhesketh_, pong | 21:48 |
| *** dizquierdo has joined #openstack-infra | 21:49 | |
| jhesketh__ | krtaylor: Hey, I think I was mentioned to you in the backscroll about something to do with swift? | 21:49 |
| jhesketh__ | anything I can help with? | 21:49 |
| jeblair | clarkb, fungi: logstash restarted on all hosts (except 01), and the workers too. workers 01-16 are in the pool | 21:50 |
| jeblair | status: push-log1946536566 | 21:50 |
| jeblair | ugh | 21:50 |
| jeblair | queue: 194653 | 21:50 |
| krtaylor | jhesketh_, ArxCruz, and I are looking at a jenkins swift plugin, we were wondering if there were others looking at it also | 21:50 |
| *** oubiwan__ has joined #openstack-infra | 21:50 | |
| jeblair | krtaylor: jclouds plugin supports swift | 21:51 |
| reed | fungi, re:https://review.openstack.org/69636 jimmy is right: we're not moving silverstripe to trove at this time | 21:51 |
| krtaylor | jhesketh_, it doesnt look like it would be too bad with joss.swift | 21:51 |
| *** oubiwann_ has quit IRC | 21:51 | |
| fungi | shivh: sometimes we get behind on moderating non-subscriber posts to the openstack-infra mailing list, sorry... i'll look at the bug soon hopefully | 21:51 |
| krtaylor | jeblair, looking at that now, thanks | 21:52 |
| shivh | fungi: np, wasnt sure i had missed something. no hurry. | 21:52 |
| jhesketh__ | krtaylor: oh cool, interesting | 21:53 |
| fungi | reed: instead we're going to have the openid server connect to a mysqld presented from whatever is embedded within cloudsites? | 21:53 |
| jhesketh__ | krtaylor: so I was looking at doing it in a way where we could use formpost (or tempurl) as opposed to storing credentials on Jenkins | 21:53 |
| jhesketh__ | krtaylor: I put this together yesterday as a test: https://review.openstack.org/#/c/71189/ | 21:54 |
| *** mestery has joined #openstack-infra | 21:54 | |
| jhesketh__ | but it's clearly not a Jenkins plugin and we're going to have to figure out how to get the console logs along for the ride too | 21:54 |
| jhesketh__ | but as you can see it uses information given to it by zuul | 21:54 |
| *** oubiwan__ has quit IRC | 21:54 | |
| *** oubiwann_ has joined #openstack-infra | 21:55 | |
| krtaylor | jhesketh_, we did similar, we just have a script uploading into swift | 21:56 |
| clarkb | there is already a jenkins swift plugin | 21:56 |
| clarkb | the jclouds plugin supports blob stores | 21:56 |
| jeblair | krtaylor: the main thing we're solving here is that we want untrusted workers to be able to push to swift; jenkins, by its architecture is trusted, so the problem space is a bit different. but we want to support non-jenkins job runners. | 21:56 |
| clarkb | and it *should* work since I fixed big chunks of it | 21:56 |
| *** jasondotstar has quit IRC | 21:56 | |
| krtaylor | jeblair, ok that helps | 21:56 |
| jeblair | krtaylor: so what jhesketh__'s change to zuul does is provide the formpost glue so a job runner can only upload into restricted paths in swift | 21:57 |
| jeblair | clarkb: did you work on jclouds or something else? | 21:57 |
| *** jasondotstar has joined #openstack-infra | 21:57 | |
| clarkb | jeblair: jclouds, the swift stuff didn't work at all | 21:57 |
| clarkb | but it works now | 21:57 |
| krtaylor | clarkb, good to know, I wish ArxCruz was on, he'll want to hear this :) | 21:58 |
| krtaylor | jeblair, clarkb, so is the plan to move to swift-based log artifacts? | 21:59 |
| jeblair | queue: 181764 | 22:00 |
| clarkb | jeblair: yeah it is slowly but surely catching up | 22:00 |
| jeblair | clarkb: eta 126 mins | 22:00 |
| zaro | jeblair: workflow label is now on review-dev.o.o | 22:01 |
| jeblair | krtaylor: yes, and possibly other artifacts too | 22:01 |
| jeblair | tarballs, etc | 22:01 |
| annegentle | fungi: sooo.... I really need to do some more editing before cutting that branch... can I ping you tomorrow to ask for that O'REilly branch to be set up? | 22:01 |
| *** gokrokve has joined #openstack-infra | 22:01 | |
| *** stack555 has joined #openstack-infra | 22:02 | |
| clarkb | annegentle: feel free to ping any of us | 22:02 |
| clarkb | annegentle: we just need to create a branch in gerrit ya? | 22:02 |
| annegentle | fungi: after I beg our editor for an extension ha ha | 22:02 |
| fungi | annegentle: sure. as clarkb says me or any of the gerrit admins | 22:02 |
| *** nati_ueno has joined #openstack-infra | 22:02 | |
| stack555 | hello clarkb | 22:03 |
| annegentle | fungi: clarkb: I think another branch in gerrit right? I don't think oreilly needed to work outside of gerrit | 22:03 |
| clarkb | stack555: hi, can you repeat your question for the folks here | 22:03 |
| stack555 | sure | 22:03 |
| annegentle | and I want to bring stuff in | 22:03 |
| stack555 | hello all, I am individual contributer ..while creating the jerrit account I accidentally signed USG CLA and system CLA. Now I cant delete them from my account and they are stuck in pending state.. | 22:03 |
| fungi | stack555: i can go ahead and clear those | 22:03 |
| fungi | just a sec | 22:03 |
| stack555 | do you need my account id or something | 22:04 |
| fungi | annegentle: yes, that sounds like all we'd need | 22:04 |
| annegentle | and now a general question about gerrit, how do I get a copy of a specific patch that is prior to the current patch? on https://review.openstack.org/#/c/68936/ I specifically want patch 13. | 22:04 |
| clarkb | fungi: so that I am aware of the proper fix, basically find rows for those CLAs in the CLA tables and delete them? | 22:04 |
| fungi | stack555: no, i just clear them all. you'll probably be one of many, but the only one who noticed or bothered to ask | 22:04 |
| stack555 | account id is 10162 | 22:04 |
| fungi | clarkb: yes | 22:04 |
| clarkb | fungi: that reminds me, you should take a look at account 10226 when you are in there | 22:05 |
| clarkb | fungi: I think we should probably disable it | 22:05 |
| clarkb | if you look at the name you will see why | 22:05 |
| fungi | clarkb: i saw your comment in scrollback the other day and mentioned it over in #launchpad, but i think all the launchpad admins have either died or fallen into an eternal slumber | 22:06 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Set appropriate rate-limit for RAX clouds. https://review.openstack.org/68509 | 22:06 |
| clarkb | fungi: ya, I think in the accounts table is a column that allows us to disable accounts on our end | 22:06 |
| clarkb | fungi: but it was late at night and I didn't feel like updating the DB randomly | 22:06 |
| *** mrodden has quit IRC | 22:07 | |
| fungi | clarkb: however, that was the only comment left by that account anyway, and if they want to try again they'll just come in from a different lp account anyway | 22:07 |
| clarkb | this is true | 22:07 |
| *** david-lyle has quit IRC | 22:08 | |
| fungi | clarkb: delete from account_agreements where status = 'n'; | 22:08 |
| fungi | (for future reference) | 22:09 |
| stack555 | Hello do you have all the info that you need ?? | 22:09 |
| fungi | stack555: i've cleared your pending agreements | 22:09 |
| clarkb | fungi: thanks | 22:09 |
| stack555 | yes | 22:09 |
| stack555 | thanks a lot | 22:09 |
| fungi | you were one of 14 accounts in a similar situation | 22:09 |
| clarkb | jeblair: for https://review.openstack.org/#/c/69442/ if I whip up a small change to that to support dev slaves better will you be able to rereview? or should I approve that as is and work on the dev slave improvements later? | 22:10 |
| *** cody-somerville has joined #openstack-infra | 22:10 | |
| fungi | well, one of 8 accounts. i was counting duplicates because it looks like most people who think to agree to one usually agree to both just for good measure | 22:10 |
| clarkb | jeblair: I think I want to add a flage to single_use_slave that selects the appropriate key | 22:10 |
| clarkb | and use a single class | 22:10 |
| *** mfer has quit IRC | 22:11 | |
| jeblair | openstack gerrit's main page now shows only 5 minutes of activity per page (because it's so active). | 22:11 |
| *** mrodden has joined #openstack-infra | 22:12 | |
| jeblair | gerrit's gerrit shows 3 days of activity. | 22:12 |
| *** dims has quit IRC | 22:12 | |
| jeblair | aosp shows about 4 hours. | 22:12 |
| clarkb | wow | 22:12 |
| fungi | jinkies | 22:12 |
| jeblair | as does cyanogenmod | 22:12 |
| jeblair | we're also about to catch up to the review numbers for aosp. | 22:13 |
| jeblair | they are at 81000 | 22:13 |
| jeblair | looks like we passed gerrit itself a while ago | 22:13 |
| jeblair | clarkb: it's broken now, let's merge your change as is then extend and fix | 22:14 |
| clarkb | jeblair: wfm | 22:14 |
| openstackgerrit | A change was merged to openstack-infra/config: Set appropriate rate-limit for RAX clouds. https://review.openstack.org/68509 | 22:15 |
| jog0 | clarkb: whats your ETA for logstash catching up | 22:16 |
| jeblair | clarkb, fungi, lifeless: ^ lets see what impact that has | 22:16 |
| clarkb | jog0: jeblair thought about 2 hours a little while back | 22:16 |
| mordred | jeblair, jhesketh__ ooh neat! | 22:17 |
| stack555 | Hello All, I amgetting the following error on doing git commit fatal: ICLA contributor agreement requires current contact information. Please review your contact information: https://review.openstack.org/#/settings/contact | 22:17 |
| *** mrodden has quit IRC | 22:17 | |
| jog0 | clarkb: cool thanks | 22:17 |
| stack555 | i think i might have to resign ICLA?? | 22:17 |
| fungi | stack555: did you follow that url and enter contact information? | 22:17 |
| clarkb | jeblair: I am going to approve my change then, I can babysit and kick off an image build or two | 22:17 |
| clarkb | fungi: jeblair ^ does that interfere with the nodepool work? | 22:17 |
| stack555 | on doing that it tells me that my email is not a part of openstack.org/regiter | 22:18 |
| *** mrda is now known as mrda_away | 22:18 | |
| stack555 | is that required??? | 22:18 |
| fungi | jeblair: hopefully it will help--at least half the waiting deletes are in rackspace, but really it looks roughly proportional to our quotas on each provider | 22:18 |
| clarkb | stack555: yes, foundation membership with emails that match is required | 22:18 |
| *** Shrews has quit IRC | 22:18 | |
| fungi | stack555: https://wiki.openstack.org/wiki/CLA-FAQ | 22:19 |
| stack555 | sure thank you | 22:20 |
| clarkb | and approved, will babysit the slave manifest change | 22:21 |
| jeblair | clarkb: should be fine; will only bother us if we need to restart np | 22:21 |
| jog0 | clarkb: now that the config patch landed I am reteting https://review.openstack.org/#/c/68304/ | 22:22 |
| *** markmcclain1 has quit IRC | 22:23 | |
| *** jergerber has joined #openstack-infra | 22:25 | |
| fungi | jeblair: actually, looking at the breakdown more closely, it's really just rackspace and hpcloud-az2 which have the lion's share of delete state nodes. hpcloud-az1 and -az3 are fairly clean in that regard | 22:26 |
| *** mestery has quit IRC | 22:26 | |
| fungi | so maybe it will help quite a bit if we're overrunning the rate limits for those api calls | 22:26 |
| jeblair | fungi: az2 does have way more servers than az1 or az3 | 22:27 |
| fungi | yeah, but this is more than an order of magnitude difference | 22:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Redo slave manifests for clarity and correctness. https://review.openstack.org/69442 | 22:27 |
| fungi | jeblair: http://paste.openstack.org/show/62680/ | 22:28 |
| fungi | jeblair: at this point, all nodes in az2 are stuck in a delete state | 22:28 |
| *** e0ne has quit IRC | 22:28 | |
| fungi | it's been climbing steadily, but just hit the quota moments agi | 22:29 |
| fungi | ago | 22:29 |
| *** dims has joined #openstack-infra | 22:29 | |
| *** thomasem has quit IRC | 22:30 | |
| fungi | oh, nevermind. we must have more quota there? it's up to 104 in az2 now | 22:30 |
| *** denis_makogon_ is now known as denis_makogon | 22:30 | |
| *** gokrokve has quit IRC | 22:30 | |
| fungi | right, we have 192 quota there these days | 22:30 |
| jeblair | fungi: yeah, 4x the 48 in the other azs | 22:31 |
| fungi | so it's just past the 50% mark in a delete state for that provider | 22:31 |
| fungi | anyway, still considerably worse than the quota proportions | 22:31 |
| clarkb | zaro: 2014-02-05 22:29:46,226 WARNING zuul.Scheduler: Project gtest-org/gtest not found is why recheck no bug is not doing anything | 22:32 |
| fungi | sounds like it needs adding to layout-dev.yaml? | 22:33 |
| jeblair | clarkb: nodepool.o.o is uptodate via puppet | 22:33 |
| clarkb | fungi: ya, I think layout-dev must be missing something. I don't know what that is yet it doesn't seem readily apparently | 22:33 |
| clarkb | there is a projects: whith - name: gtest-org/gtest in it | 22:34 |
| clarkb | aha! I see the bug | 22:34 |
| *** markmcclain has joined #openstack-infra | 22:34 | |
| fungi | so, some of the delete problem seems to be hpcloud-az2. we're holding fairly steady on delete counts for other providers but continually climbing in stale delete numbers for az2 | 22:34 |
| clarkb | zaro: there is an extra 'projects:' in layout-dev | 22:35 |
| *** dangers is now known as dangers_away | 22:35 | |
| *** markmcclain1 has joined #openstack-infra | 22:36 | |
| *** markmcclain1 has quit IRC | 22:36 | |
| *** markmcclain1 has joined #openstack-infra | 22:36 | |
| *** markmcclain1 has quit IRC | 22:36 | |
| *** markmcclain1 has joined #openstack-infra | 22:37 | |
| *** markmcclain1 has quit IRC | 22:37 | |
| fungi | the 0.1-hour quantization of delete node times in az2 looks like http://paste.openstack.org/show/62685/ | 22:37 |
| *** markmcclain has quit IRC | 22:39 | |
| *** dims has quit IRC | 22:39 | |
| jeblair | after the configuration reload, some of the providers are doing nothing | 22:40 |
| *** kraman has quit IRC | 22:44 | |
| *** oubiwan__ has joined #openstack-infra | 22:44 | |
| *** oubiwann_ has quit IRC | 22:44 | |
| *** CaptTofu_ has quit IRC | 22:45 | |
| jeblair | yeah, none of the rax task managers are running | 22:45 |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Image logs accessible via HTTP https://review.openstack.org/71190 | 22:47 |
| *** mriedem has quit IRC | 22:47 | |
| *** miqui has quit IRC | 22:47 | |
| jeblair | clarkb, fungi: how do you feel about reverting? | 22:47 |
| *** wchrisj_ has joined #openstack-infra | 22:48 | |
| *** jasondotstar has quit IRC | 22:48 | |
| fungi | the hpcloud-az2 delete state count has dropped at least | 22:49 |
| fungi | though the ones in rax have climbed | 22:49 |
| jeblair | the thread dump says the rax taskmanagers are waiting for something to do | 22:49 |
| *** apevec has joined #openstack-infra | 22:50 | |
| apevec | fungi, fyi tempest grizzly should be fixed now with https://review.openstack.org/#/q/Iea57598698b4faf906e82a2974497221d07f4d16,n,z | 22:50 |
| jeblair | unfortunately the new per-provider cleanup threads are not named so i can't tell which is which | 22:50 |
| apevec | it was reqs mismatch | 22:50 |
| jog0 | https://review.openstack.org/#/c/71191/ | 22:50 |
| jog0 | cisco is just spamming us | 22:50 |
| apevec | ending with pkg_resources.DistributionNotFound: SQLAlchemy>=0.7.3,<=0.7.9 in c-vol | 22:51 |
| clarkb | jog0: I am ok with a revert | 22:51 |
| clarkb | jeblair: ^ | 22:51 |
| mordred | jeblair: I'm ok with a revert | 22:51 |
| jog0 | nulljob/TempestForN1KV/6/ : SUCCESS | 22:51 |
| mordred | I'm also ok with reverting jog0 | 22:51 |
| mordred | except I don't know what that means | 22:51 |
| fungi | jeblair: clarkb: i assume we're talking about reverting the entire patch series | 22:52 |
| jeblair | fungi: yep | 22:52 |
| clarkb | jeblair: if logstash locks up like that again, I will probably look into upgrading logstash. Their change log says things like 'bug fixes galore' and digging into their jira I find a couple cases of hanging that have been fixed | 22:52 |
| *** oubiwann_ has joined #openstack-infra | 22:52 | |
| *** oubiwan__ has quit IRC | 22:52 | |
| *** dizquierdo has quit IRC | 22:53 | |
| *** oubiwann_ has quit IRC | 22:53 | |
| mordred | clarkb: well, we know that bug fixes never bring new bugs | 22:53 |
| mordred | clarkb: so that's good | 22:53 |
| clarkb | mordred: right:) most of my time upgrading last round was spent working around new bugs | 22:53 |
| *** gokrokve has joined #openstack-infra | 22:53 | |
| fungi | jeblair: clarkb: i'll go ahead and put together a series of reverts real quick | 22:53 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Revert delete-rework branch https://review.openstack.org/71406 | 22:53 |
| jeblair | fungi: done | 22:53 |
| fungi | oh, or that | 22:54 |
| jeblair | fungi: but do double check i got everything | 22:54 |
| * clarkb reviews | 22:54 | |
| *** Sukhdev has joined #openstack-infra | 22:54 | |
| fungi | it looks like a far simpler change when squashed like that, but yeah i'm comparing to the commit log now | 22:55 |
| clarkb | fungi: it is... I am not a fan of the lifeless style of review | 22:56 |
| *** mgagne has quit IRC | 22:57 | |
| jog0 | clarkb: he once gave me a -1 without any comment ;) | 22:57 |
| *** ryanpetrello has quit IRC | 22:57 | |
| clarkb | jog0: apparently this is common in tripleo land but you guys review the entire chain of things rather than the diff between states A and B | 22:57 |
| jog0 | clarkb: to be clear revert means making Cisco Neutron CI shut up? | 22:58 |
| clarkb | jog0: cisco neutron ci? | 22:58 |
| jog0 | clarkb: yeah | 22:58 |
| harlowja | hi guys, is there anyway to get more info when a build has the following 'Build timed out (after 40 minutes)' | 22:58 |
| jog0 | clarkb: see https://review.openstack.org/#/c/71191/ | 22:58 |
| mordred | harlowja: well - it usually means the the build took longer than 40 minutes | 22:59 |
| * mordred hides | 22:59 | |
| harlowja | :) | 22:59 |
| Sukhdev | will somebody here be able to answer a question regarding neutron-debug probe-create? | 22:59 |
| jeblair | harlowja: which build? | 22:59 |
| clarkb | jog0: pretty sure nodepool reverting will have nothing to do with that | 22:59 |
| harlowja | mordred i would suspect so, http://logs.openstack.org/12/71312/4/check/gate-taskflow-tox-py27-sa8-mysql/ccef4d9/console.html | 22:59 |
| fungi | harlowja: did the subunit output get captured as a separate file? | 22:59 |
| clarkb | Sukhdev: no probably not | 22:59 |
| harlowja | fungi ah, this goes back to the subunit thing i gotta fix, let me do that | 23:00 |
| fungi | harlowja: right, looks like we only got the console log there | 23:00 |
| harlowja | ya, let me fix up my patch to fix that problem | 23:00 |
| jeblair | jog0, fungi, clarkb: shall we remove cisco neutron ci from the voting group? | 23:00 |
| jog0 | clarkb: ok that is why I was so confused | 23:00 |
| Sukhdev | clarkb: thanks - do you know what is IRC for devstack related questions? | 23:00 |
| jog0 | jeblair: ++, its a noop right now | 23:00 |
| fungi | jeblair: i'll do that and follow up to their account request e-mail thread | 23:01 |
| jog0 | I can live with comments but noop +1s | 23:01 |
| jeblair | fungi: ok cool thx | 23:01 |
| Sukhdev | Will somebody here be able to answer a question on Jenkins setting for "recheck no bug" setting? | 23:01 |
| jeblair | clarkb, mordred: are you reviewing 71406? | 23:02 |
| clarkb | Sukhdev: #openstack-dev for devstack questions | 23:02 |
| Sukhdev | clarkb:thanks | 23:02 |
| mordred | jeblair: yeah. but go ahead. I'm not expecting to find a blocker | 23:02 |
| clarkb | Sukhdev: yes we can talk about jenkins settings for rechecks | 23:02 |
| jeblair | mordred, clarkb: no prob, i can wait, i just didn't want to wait if you weren't doing anything | 23:03 |
| clarkb | Sukhdev: short story is (there is a thread on the infra mail list) gerrit trigger plugin can't do it but zuul can | 23:03 |
| clarkb | jeblair: I just scanned the list, I think you got everything. Looking at diffs now | 23:03 |
| mordred | +2 | 23:04 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: remove extra projects section https://review.openstack.org/71410 | 23:04 |
| zaro | clarkb: ^ | 23:04 |
| Sukhdev | clarkb: - I am setting jenkin's Gerrit plugin. I am able to get the notifications for recheck, but, am able to get it for patch create - | 23:04 |
| ArxCruz | krtaylor: clarkb wow what I didn't hear? | 23:05 |
| clarkb | jeblair: and approved | 23:05 |
| Sukhdev | clarkb: would you happen to know how/where do the create Jenkins config? | 23:05 |
| jeblair | ArxCruz: search for swift in http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-02-05.log | 23:05 |
| clarkb | Sukhdev: no, we haven't used the jenkins gerrit trigger plugin in almost 2 years | 23:05 |
| clarkb | Sukhdev: please see the thread at http://lists.openstack.org/pipermail/openstack-infra/2014-February/000759.html | 23:06 |
| mordred | clarkb: wow. has it been that long? | 23:06 |
| *** weshay has quit IRC | 23:06 | |
| clarkb | mordred: yes will be 2 years in July? | 23:06 |
| clarkb | maybe closer to 18 months at this point | 23:06 |
| mordred | still. | 23:07 |
| mordred | we shoudl throw zuul a birthday you know | 23:07 |
| openstackgerrit | A change was merged to openstack/requirements: Sort tracked projects list https://review.openstack.org/63770 | 23:07 |
| mordred | Date: Tue May 29 14:49:32 2012 -0700 | 23:07 |
| mordred | Initial commit. | 23:07 |
| *** wchrisj_ has quit IRC | 23:07 | |
| *** esp has joined #openstack-infra | 23:08 | |
| openstackgerrit | Jenkins proposed a change to openstack-dev/hacking: Updated from global requirements https://review.openstack.org/71412 | 23:08 |
| mordred | I consider initial commits from jeblair to be essentially production code. all the rest is just polish | 23:08 |
| Sukhdev | clarkb: thanks for the link - where can get more information about how does zuul create the configuration? | 23:08 |
| clarkb | Sukhdev: http://ci.openstack.org/zuul | 23:09 |
| ArxCruz | jeblair: I didn't understand what you mean with the plans is to replace jenkins with swift... I'm not a expert, but I though swift was the object storage, how it can replace jenkins? | 23:09 |
| ArxCruz | oh, the job-runner okay, got it | 23:09 |
| openstackgerrit | A change was merged to openstack/requirements: Add rally to tracked projects list https://review.openstack.org/63741 | 23:10 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Add a testr specific builder https://review.openstack.org/70511 | 23:10 |
| harlowja | fungi clarkb (if i didn't do it right let me know ) ;) ^ | 23:10 |
| harlowja | damn ;) ^ doesnt look right, lol | 23:10 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Add a testr specific builder https://review.openstack.org/70511 | 23:11 |
| clarkb | I am doing zmq testing on jenkins-dev again, if this pans out I will approve my zmq change | 23:12 |
| clarkb | but I can put this down if nodepool or whatever needs more attention | 23:12 |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Image logs accessible via HTTP https://review.openstack.org/71190 | 23:14 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Revert delete-rework branch https://review.openstack.org/71406 | 23:15 |
| *** krotscheck has quit IRC | 23:15 | |
| ArxCruz | jhesketh__: hey, so https://review.openstack.org/#/c/71189/ very cool! | 23:15 |
| *** esp has left #openstack-infra | 23:15 | |
| ArxCruz | jhesketh_: In this case, jenkins will continue to publish using scp plugin right ? | 23:15 |
| *** dcramer_ has quit IRC | 23:16 | |
| * fungi is so pleased... | 23:16 | |
| jhesketh__ | ArxCruz: well that entirely depends on how you configure Jenkins | 23:16 |
| jhesketh__ | but yes | 23:17 |
| ArxCruz | cool, and when fungi jeblair clarkb will approve this? :D | 23:17 |
| fungi | The following message to <cisco-openstack-neutron-ci@cisco.com> was undeliverable. The reason for the problem: 5.1.0 - Unknown address error 550-'5.1.1 <cisco-openstack-neutron-ci@cisco.com>... User unknown' | 23:17 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Log gerrit comment https://review.openstack.org/70287 | 23:18 |
| fungi | guess it's time to add another paragraph to my message and resend it to the original account requesters | 23:18 |
| *** slong has joined #openstack-infra | 23:18 | |
| jhesketh__ | ArxCruz: the proposed change actually won't get run. The next step will be to test it out with JJB | 23:18 |
| jhesketh__ | possibly using http://ci.openstack.org/jenkins-job-builder/publishers.html#publishers.post-tasks | 23:19 |
| jhesketh__ | but the zuul swift instructions needs merging first | 23:19 |
| clarkb | fungi: :/ | 23:20 |
| ArxCruz | jhesketh__: that's what I'm talking | 23:20 |
| ArxCruz | jhesketh__: today I've started a swift plugin on jenkins, based on the scp plugin | 23:20 |
| ArxCruz | But will take a while to be functional | 23:20 |
| ArxCruz | your work seems to be already pretty functional | 23:20 |
| jhesketh__ | ArxCruz: I thought clarkb said that already existed? | 23:20 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove unused main in elasticRecheck.py https://review.openstack.org/70471 | 23:20 |
| ArxCruz | jhesketh__: didn't knew about jclouds before begin lol | 23:21 |
| clarkb | ArxCruz: jhesketh__ it does exist | 23:21 |
| *** david-lyle has joined #openstack-infra | 23:21 | |
| krtaylor | ArxCruz, jhesketh_ Y, the jclouds is news to us | 23:22 |
| *** masayukig has joined #openstack-infra | 23:22 | |
| krtaylor | ArxCruz, do you have time to look at that tonight? | 23:23 |
| *** rcleere has quit IRC | 23:23 | |
| jeblair | fungi: nodepool is ready to restart | 23:24 |
| jeblair | fungi: you've been doing that a lot recently, want to again? | 23:24 |
| fungi | jeblair: sure, i'll take care of it | 23:25 |
| ArxCruz | krtaylor: a few, it's WoW time ;D | 23:25 |
| fungi | yep, looks like it has the change | 23:25 |
| ArxCruz | krtaylor: but let's continue with our script, then move on to jclouds | 23:25 |
| openstackgerrit | David Johansen proposed a change to openstack-infra/jenkins-job-builder: Added support for least load plugin. https://review.openstack.org/71316 | 23:26 |
| *** jgrimm_ has quit IRC | 23:26 | |
| *** wchrisj__ has joined #openstack-infra | 23:26 | |
| *** CaptTofu has joined #openstack-infra | 23:27 | |
| *** mriedem has joined #openstack-infra | 23:28 | |
| *** esker has quit IRC | 23:29 | |
| *** esker has joined #openstack-infra | 23:29 | |
| *** flaper87 is now known as flaper87|afk | 23:30 | |
| *** denis_makogon has quit IRC | 23:32 | |
| *** jnoller has quit IRC | 23:32 | |
| *** paul-- has joined #openstack-infra | 23:33 | |
| *** kraman1 has joined #openstack-infra | 23:33 | |
| *** yamahata has quit IRC | 23:33 | |
| *** esker has quit IRC | 23:34 | |
| clarkb | jeblair: fungi https://review.openstack.org/#/c/70283/2 is another fun change we might consider merging before spinning up image builds | 23:37 |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Nodepool image logs accessible via HTTP https://review.openstack.org/71190 | 23:37 |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Add oslo fixture module and update everything else https://review.openstack.org/69830 | 23:39 |
| *** e0ne has joined #openstack-infra | 23:40 | |
| jeblair | clarkb: should we add the ext4 features during the image build? (can you do that with an online fs?) | 23:40 |
| clarkb | jeblair: you can do it on an online / but it is trickier than my change there | 23:40 |
| *** HenryG has quit IRC | 23:41 | |
| jeblair | clarkb: seems like it's worth doing though; the more consistent the envs the better i think | 23:42 |
| jeblair | clarkb: but we can start with that | 23:42 |
| mordred | jeblair: do you have a sec to explain credentials-id to me? | 23:42 |
| *** julim has quit IRC | 23:42 | |
| jeblair | mordred: create a credential using the web ui, then find the id it used in the xml file | 23:43 |
| jeblair | mordred: (the file is one that we copy around with the tarballs so new jenkinses get the same credential id) | 23:43 |
| jeblair | mordred: it's for an ssh key | 23:43 |
| jeblair | for the slaves | 23:44 |
| mordred | yah. grok that. did we pre-precreate the ssh key and then add it? | 23:44 |
| *** e0ne has quit IRC | 23:44 | |
| jeblair | mordred: the ssh key is in hiera and installed on the system, the credential just points to it | 23:44 |
| mordred | ah - good - I see that option | 23:44 |
| *** mrda_away is now known as mrda | 23:45 | |
| fungi | mordred: in production for us it's pointing to ~/jenkins/.ssh/id_rsa | 23:45 |
| jeblair | mordred: so, yes, the actual key is created in the normal manner (ssh-keygen) | 23:45 |
| fungi | er, ~jenkins/.ssh/id_rsa | 23:45 |
| mordred | god I hate jenkins | 23:45 |
| jeblair | mordred: almost gone | 23:45 |
| fungi | which i think is the default behavior for the webui if you just say "this credential is an ssh key on the filesystem" | 23:45 |
| jeblair | fungi, clarkb: looks like rax api requests are happening > 1/s now | 23:46 |
| fungi | jeblair: i'm also mass-deleting things which were hanging around from before the restart | 23:46 |
| clarkb | http://www.debian-administration.org/articles/643 indicates conversion should happen in single user mode which isn't a thing on ubuntu anymore aiui | 23:46 |
| clarkb | but should be safe to try since we will just discard a failed image build | 23:46 |
| fungi | old nodes in building and ready state primarily | 23:46 |
| clarkb | I will probably need to run through that on rax by hand just to get a feel for what happens | 23:47 |
| jeblair | fungi: ++; i'm just watching the log though which at this point is probably just spinning up new machines | 23:47 |
| fungi | excellent | 23:47 |
| clarkb | but even then most files won't use extents | 23:48 |
| clarkb | I think for now mounting as ext4 is simple and a big win | 23:48 |
| jeblair | clarkb: there's the dir trees thing... | 23:48 |
| jeblair | clarkb: what about ext4 do people think will be better? | 23:49 |
| mordred | jeblair: it fixes neutron | 23:49 |
| clarkb | jeblair: it has been clarified since then that the guest fs for nova VMs is more important in the nbd network namespace situation | 23:49 |
| mordred | jeblair: don't ask way | 23:49 |
| mordred | why | 23:49 |
| clarkb | jeblair: but others think the switch to ext4 for the test slaves themselves may help on the performance front for our slaves | 23:50 |
| *** eharney has quit IRC | 23:50 | |
| *** prad_ has quit IRC | 23:50 | |
| jeblair | mordred: well, why is important... we only get some ext4 features when mounting ext3 as ext4 | 23:50 |
| * mordred does not believe it will significantly impact performance, but is willing to be wrong | 23:50 | |
| jeblair | i don't think it will either | 23:50 |
| mordred | jeblair: sorry - too dry humor - I meant to imply hand waving | 23:51 |
| jeblair | mordred: are you in the uk? | 23:51 |
| mordred | jeblair: yes | 23:51 |
| jeblair | mordred: that explains the humour then | 23:51 |
| mordred | yup | 23:51 |
| clarkb | jeblair: basically folks were grasping at whatever they could. they tried newer kernels, saw ext3 was used and suggested ext4 | 23:51 |
| mordred | the weather certainly doesn't | 23:51 |
| clarkb | jeblair: I think it is insane that ext4 is not the default | 23:51 |
| clarkb | but do also agree that I don't expect it to radically change anything | 23:52 |
| jeblair | clarkb: indeed | 23:52 |
| *** oubiwann_ has joined #openstack-infra | 23:52 | |
| clarkb | and had relatively harsh comments to that affect when it was suggested we change | 23:52 |
| *** sparkycollier has quit IRC | 23:53 | |
| jeblair | fungi: we can probably parelellize those deletes | 23:55 |
| jeblair | fungi: i've run up to 10 at once | 23:55 |
| clarkb | fungi: ping me when you are comfortable with me kicking off an image build | 23:55 |
| fungi | jeblair: i did (well, 5 loops running) | 23:55 |
| *** zehicle_at_dell has joined #openstack-infra | 23:55 | |
| fungi | got through the building/ready now i'm going to try to knock down the old images in delete state | 23:55 |
| jeblair | fungi: ah ok. cool. | 23:56 |
| jeblair | the gate queue is now waiting on nodes :( | 23:56 |
| *** VijayT has quit IRC | 23:59 | |
| *** apevec has quit IRC | 23:59 | |
| StevenK | Oh, its an image build that is impacting the gate? | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
