| jeblair | deleting nodepool nodes that were marked used during the zuul downtime | 00:00 |
|---|---|---|
| openstackgerrit | Samuel Merritt proposed a change to openstack-infra/config: Add soft timeout to Swift functional tests https://review.openstack.org/68802 | 00:00 |
| clarkb | StevenK: the zuul status json blobs include window sizes now. You can query them that way until I figure out how to make them pretty in html + js | 00:01 |
| openstackgerrit | A change was merged to openstack-infra/config: Enable jenkins0[5-7] in nodepool https://review.openstack.org/68759 | 00:01 |
| jeblair | clarkb: i recommend a grey dot | 00:01 |
| jeblair | for inactive | 00:01 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: fix template directory for uncategorized page https://review.openstack.org/68803 | 00:01 |
| sdague | clarkb: ... face-palm | 00:01 |
| sdague | re: ^^^^^ | 00:01 |
| clarkb | sdague: ha | 00:02 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Keep logs in $BASE instead of $WORKSPACE https://review.openstack.org/68782 | 00:02 |
| anteaya | pleia2: do you know where the gerrit patch is that mentions the chanserv command for adding openstackinfra to a channel? | 00:06 |
| anteaya | I can't even remember what repo that is in | 00:06 |
| *** derekh has joined #openstack-infra | 00:06 | |
| clarkb | anteaya: its going to be the openstack-infra/config repo | 00:06 |
| clarkb | anteaya: you will want to look at reviews for commits that touch the bot config | 00:07 |
| pleia2 | anteaya: see the bottom of http://ci.openstack.org/irc.html | 00:07 |
| anteaya | thanks | 00:08 |
| anteaya | yeah all paths lead to config | 00:08 |
| *** gokrokve has quit IRC | 00:08 | |
| *** gokrokve has joined #openstack-infra | 00:09 | |
| hashar | jeblair: thank you for the Zuul reviews :-D | 00:09 |
| *** smurugesan has quit IRC | 00:11 | |
| *** rnirmal has quit IRC | 00:12 | |
| anteaya | jeblair: can you run /msg chanserv access #openstack-neutron add markmcclain +AFRfiorstv | 00:13 |
| anteaya | please | 00:13 |
| *** gokrokve has quit IRC | 00:13 | |
| anteaya | I tried but can not | 00:13 |
| fungi | lifeless: could it be that nodepool is smart enough to not try building a tripleo-fedora image if there are no nodes of that type specified for any provider? i see nothing in the log suggesting that it even tried to create the image | 00:14 |
| *** slong has joined #openstack-infra | 00:15 | |
| *** salv-orlando has quit IRC | 00:16 | |
| *** salv-orlando has joined #openstack-infra | 00:16 | |
| *** dkliban_afk has quit IRC | 00:16 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/devstack-gate: Adding the tripleo repositories to PROJECTS https://review.openstack.org/68645 | 00:17 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Clone the TripleO repositories https://review.openstack.org/68807 | 00:18 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Switch path to toci_gate_test.sh https://review.openstack.org/68662 | 00:18 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Remove TRIPLEO_ROOT and pull-tools https://review.openstack.org/68661 | 00:18 |
| sdague | fungi: hey, while you've been near jbryce this week, any chance you've chatted about publishing the tc docs to openstack.org? | 00:18 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Reflect zuul window in status page. https://review.openstack.org/68808 | 00:19 |
| *** flaper87 is now known as flaper87|afk | 00:19 | |
| clarkb | jeblair: ^ that wasn't so bad | 00:19 |
| *** marun has joined #openstack-infra | 00:19 | |
| fungi | sdague: i have not, but i can make a point of asking him and toddmorey | 00:19 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Build tripleo-fedora nodepool nodes https://review.openstack.org/68809 | 00:21 |
| fungi | lifeless: ^ | 00:21 |
| sdague | fungi: that would be awesome | 00:21 |
| anteaya | clarkb: I found one, but it doesn't have the command I was looking for, it was on the page pleia2 linked | 00:22 |
| anteaya | http://git.openstack.org/cgit/openstack-infra/config/commit/?id=01a900a772379af2eb9d16fcb068502d6b5017ed | 00:22 |
| anteaya | this is not the command you're looking for | 00:22 |
| anteaya | thanks pleia2 | 00:22 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: stop being rediculous with our time formats https://review.openstack.org/68765 | 00:24 |
| lifeless | fungi: ahahahaha | 00:26 |
| lifeless | thanks | 00:26 |
| *** slong has quit IRC | 00:26 | |
| *** matsuhashi has joined #openstack-infra | 00:27 | |
| fungi | np | 00:27 |
| jeblair | anteaya: markmcclain: added Aot, which I think is what's necessary. | 00:28 |
| jeblair | anteaya: we're not generally in the business of setting these things by hand, so please limit such requests until someone writes the bot. | 00:29 |
| markmcclain | jeblair: thank you | 00:29 |
| jeblair | clarkb: istr that we need an extra puppet line to explcitly copy the new .png? | 00:31 |
| clarkb | jeblair: yes we do, I will add that (I was running it directly out of that dir...) | 00:31 |
| jeblair | clarkb: 1 sec i have another suggestion | 00:32 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Reflect zuul window in status page. https://review.openstack.org/68808 | 00:33 |
| clarkb | jeblair: whoops :) | 00:33 |
| *** jergerber has quit IRC | 00:33 | |
| jeblair | clarkb: done! | 00:33 |
| *** hogepodge has quit IRC | 00:34 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Stop logging skipped bugs https://review.openstack.org/68810 | 00:34 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Make IRC messages provide more context https://review.openstack.org/68811 | 00:34 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Clarify required parameters in query_builder https://review.openstack.org/67756 | 00:34 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Use short build_uuids in elasticSearch queries https://review.openstack.org/67596 | 00:34 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Reflect zuul window in status page. https://review.openstack.org/68808 | 00:34 |
| clarkb | jeblair: there we go | 00:34 |
| openstackgerrit | A change was merged to openstack-infra/config: fix template directory for uncategorized page https://review.openstack.org/68803 | 00:35 |
| *** rcleere has quit IRC | 00:35 | |
| jeblair | clarkb: neat! | 00:36 |
| *** DennyZhang has quit IRC | 00:36 | |
| StevenK | clarkb: I'm not sure about the alt text, but I can't come up with anything better. :-( | 00:37 |
| *** senk1 has joined #openstack-infra | 00:37 | |
| clarkb | StevenK: ya, without writing a paragraph I don't know how to make it better | 00:37 |
| jeblair | StevenK: does it leave you with unanswered questions? | 00:37 |
| *** senk has quit IRC | 00:38 | |
| jeblair | StevenK: (I think it answers the question "why hasn't my change landed?" which is the main thing the status page is designed to supply to devs) | 00:38 |
| *** markmcclain has quit IRC | 00:39 | |
| *** marun has quit IRC | 00:39 | |
| StevenK | Mind you, I'm not objecting, just stating that I'm not sure about the text. | 00:39 |
| *** mrodden has quit IRC | 00:39 | |
| jeblair | StevenK: i wasn't sure whether 'closer' or 'nearer' would be best. so i'm not sure it's perfect either. ;) | 00:40 |
| *** markwash has quit IRC | 00:46 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Keep logs in $BASE instead of $WORKSPACE https://review.openstack.org/68782 | 00:48 |
| *** markwash has joined #openstack-infra | 00:54 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Ensure a recent (48h) check result https://review.openstack.org/68816 | 00:54 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Require a recent (48h) positive check result https://review.openstack.org/68817 | 00:54 |
| jeblair | sdague: ^ | 00:55 |
| *** hashar has quit IRC | 00:58 | |
| jeblair | clarkb, fungi: the new jenkins servers can't connect to any of their slaves | 01:01 |
| jeblair | the credentials seem to be missing | 01:02 |
| jog0 | whoa 1 job away from a 15+ patch merge | 01:02 |
| jog0 | lets hope it doesn't fail | 01:02 |
| clarkb | jog0: you just jinxed it | 01:03 |
| jeblair | clarkb: i will fix the 3 jenkins masters | 01:04 |
| *** markmcclain has joined #openstack-infra | 01:05 | |
| jeblair | it did fail | 01:06 |
| jeblair | 2014-01-24 01:05:53.866 | 2014-01-24 00:52:07.299 30766 ERROR heat.common.keystoneclient [-] Failed to add user heat-tempest-560097030-CfnUser-tbvazbsvgup3 to role heat_stack_user, check role exists! | 01:06 |
| jeblair | logs have errors | 01:06 |
| clarkb | jeblair: actually we ignore that right now | 01:06 |
| jeblair | clarkb: oh nice, and we print errors we ignore, yeah? | 01:06 |
| clarkb | jeblair: because they all have errors, if logs copy successfully that test should pass | 01:06 |
| clarkb | it has been sitting there for a long time though | 01:07 |
| jeblair | why does it say "FAILED" ? | 01:07 |
| clarkb | jeblair: yup we print the errors we ignore | 01:07 |
| clarkb | jeblair: because ugh | 01:07 |
| clarkb | lets see how long it spins there | 01:07 |
| jeblair | jenkins bootstrap tarball updated | 01:08 |
| clarkb | fungi: jeblair: where are we on the root fs thing for slaves? | 01:08 |
| clarkb | I think I can give that cycles now | 01:08 |
| jeblair | clarkb: fungi has a change up for review; tests aren't back yet | 01:09 |
| jeblair | yay success on that change! | 01:09 |
| sdague | the tarballs take 3 minutes to process at the end | 01:10 |
| jog0 | clarkb: looks like no jinks today | 01:10 |
| lifeless | jeblair: thats a regression in heat, its a real error | 01:10 |
| openstackgerrit | A change was merged to openstack/requirements: Update python-neutronclient version to 2.3.3 https://review.openstack.org/67491 | 01:11 |
| sdague | lifeless: well because heat doesn't have any tests for it, it's not a tempest error | 01:12 |
| *** oubiwann_ has joined #openstack-infra | 01:12 | |
| lifeless | sure | 01:12 |
| lifeless | just digging the thing up, sec | 01:12 |
| openstackgerrit | A change was merged to openstack-infra/config: Reflect zuul window in status page. https://review.openstack.org/68808 | 01:13 |
| jeblair | sdague, jog0, clarkb: in case you're wondering, the top of queue reset was due to a recursive git merge error: https://review.openstack.org/#/c/65018/ | 01:13 |
| sdague | jeblair: ah | 01:14 |
| clarkb | I figured | 01:14 |
| sdague | that's news to me | 01:14 |
| clarkb | the important changes got in though and the window stuff seems to have updated nicely | 01:14 |
| lifeless | https://bugs.launchpad.net/heat/+bug/1271190 | 01:14 |
| clarkb | my local grey blobs are so shiny | 01:14 |
| *** dpyzhov has joined #openstack-infra | 01:15 | |
| russellb | i feel like we're turning the tide on things :) | 01:15 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: [wip] Add NPM mirror https://review.openstack.org/68818 | 01:15 |
| lifeless | sdague: so while tis not a tempest error, it has broken heat comprehensively | 01:15 |
| lifeless | and https://review.openstack.org/#/c/68135/ - we're stuck on that in tripleo too | 01:16 |
| sdague | so that would be good to get tested in the integrated gate then. If it's enough to trigger an error log message in the current set of things, it should be enough to detect and block on | 01:16 |
| *** dstanek is now known as dstanekafk | 01:16 | |
| lifeless | sdague: we would have caught it in tripleo FWIW :(. | 01:16 |
| sdague | lifeless: sure, but tripleo isn't running on every change | 01:16 |
| lifeless | sdague: I know, we're working on that | 01:17 |
| jeblair | lifeless: we would have caught it with the log error catching script if it weren't ignoring it. | 01:17 |
| *** krotscheck has quit IRC | 01:17 | |
| jeblair | sdague: ^ | 01:17 |
| lifeless | true | 01:17 |
| sdague | jeblair: well, we turned it back off because it was causing a lot of resets over the weekend | 01:17 |
| *** dstanekafk has quit IRC | 01:17 | |
| jeblair | sdague: and bam, an error got through. :) | 01:17 |
| sdague | the 50hr gate queue meant it was kind of impossible to keep up with the whitelist | 01:17 |
| sdague | jeblair: no, that got through a week ago | 01:18 |
| sdague | or some time over the weekend, before we had the log checker fixed | 01:18 |
| jeblair | sdague: why is keeping up with the whitelist hard? you mean adding new entries? | 01:18 |
| sdague | jeblair: because d-g got completely broken with the check over christmas | 01:18 |
| sdague | so we didn't have it running for all of january, until last week | 01:19 |
| sdague | and the new errors weren't picked up | 01:19 |
| clarkb | I am going to an OpenStack meetup here in seattle shortly will be mostly afk | 01:19 |
| sdague | and, in reality, the errors that trigger there are intermitent, so they actually become a new class of reset issues. | 01:19 |
| jeblair | sdague: sure, until they are all known. i agree now is not the time to find them all. | 01:20 |
| sdague | and when we dropped to merging 1 change / 3 hrs over the weekend, we said "give up, turn it off" | 01:20 |
| sdague | jeblair: yep | 01:20 |
| sdague | though it actually was the n-cond one that made us do that, as that was showing up about 50% of the time | 01:21 |
| sdague | the heat one was more infrequent than that | 01:21 |
| clarkb | bam refresh your zuul statuses | 01:23 |
| clarkb | StevenK: ^ | 01:23 |
| sdague | jeblair: next time we get a promote slot - 68673 | 01:24 |
| sdague | that will actually speed up the tempest runs | 01:24 |
| sdague | as it drops all the nova v3 xml tests | 01:24 |
| jeblair | i'm deleting all the nodes from jenkins05 since creating them without creds doesn't seem to be automatically fixable | 01:25 |
| sdague | so does nodepool have insight into zuul? | 01:25 |
| jeblair | sdague: yes, it bursts based on zuul's current queue | 01:26 |
| *** salv-orlando has quit IRC | 01:26 | |
| jeblair | sdague: it is currently hampered by the fact that it added like 300 nodes to jenkins masters that didn't know how to talk to them | 01:26 |
| jeblair | sdague: that's what i'm fixing | 01:26 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Make gerrit/ES timeout configurable https://review.openstack.org/68799 | 01:26 |
| jeblair | sdague: all those ready nodes are not actually available | 01:26 |
| sdague | so could we dynamically up the ready size to try to keep in reserve the number of nodes that the gate window would need? | 01:26 |
| sdague | ah, ok | 01:27 |
| *** salv-orlando has joined #openstack-infra | 01:27 | |
| jeblair | 05 is getting real nodes now | 01:28 |
| jeblair | #status ok | 01:28 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 01:28 | |
| jog0 | sdague: did the unclassified page go up? | 01:28 |
| clarkb | jog0: yes | 01:29 |
| clarkb | http://status.openstack.org/elastic-recheck/data/uncategorized.html | 01:29 |
| clarkb | jog0: and the new zuul status is in place | 01:29 |
| *** mgagne has quit IRC | 01:29 | |
| jog0 | ohh purty | 01:29 |
| jeblair | deleting nodes from 06 | 01:29 |
| *** mgagne1 has joined #openstack-infra | 01:29 | |
| *** mgagne1 has quit IRC | 01:29 | |
| *** mgagne1 has joined #openstack-infra | 01:29 | |
| jog0 | clarkb: grey, nice | 01:29 |
| jog0 | http://logs.openstack.org/19/67119/3/gate/gate-tempest-dsvm-postgres-full/d512b6f/console.html#_2014-01-23_14_15_29_161 http://nova.clouds.archive.ubuntu.com/ubuntu/pool/main/p/postgresql-common/postgresql-client-common_129ubuntu1_all.deb... | 01:30 |
| jog0 | sigh | 01:30 |
| russellb | clarkb: ooh, love the fancy grey update | 01:31 |
| *** blamar_ has joined #openstack-infra | 01:31 | |
| *** mrodden has joined #openstack-infra | 01:32 | |
| clarkb | russellb: thanks, it was jeblair's suggestion | 01:32 |
| clarkb | and I am really going to afk now. | 01:32 |
| jeblair | this subway is going to cover the whole city soon | 01:32 |
| jeblair | 06 is getting real nodes now | 01:32 |
| jeblair | i have enabled the gearman plugin on 05 | 01:32 |
| jeblair | it is now running an enormous number of jobs | 01:33 |
| sdague | jog0: categorization is down to 72% | 01:33 |
| *** beagles has quit IRC | 01:33 | |
| *** morganfainberg has quit IRC | 01:33 | |
| *** morganfainberg has joined #openstack-infra | 01:33 | |
| *** b3nt_pin has joined #openstack-infra | 01:33 | |
| sdague | but at least we can now farm it out to people, though we do need to get the grenade logs in there | 01:33 |
| *** blamar has quit IRC | 01:34 | |
| *** blamar_ is now known as blamar | 01:34 | |
| jeblair | 06 is now running many many jobs | 01:34 |
| jeblair | deleting nodes from 07 | 01:34 |
| jog0 | sdague: we should get https://review.openstack.org/#/c/68478/ in to get better numbers | 01:34 |
| *** senk1 has quit IRC | 01:35 | |
| jog0 | sdague: I will work on getting the numbers up tomorrow | 01:35 |
| sdague | jog0: lgtm | 01:35 |
| sdague | I just pushed +A | 01:35 |
| jog0 | sdague: found one above ^ apt-get timeout | 01:35 |
| sdague | nice | 01:35 |
| *** nosnos has joined #openstack-infra | 01:35 | |
| jog0 | will file a bug on it tomorrow | 01:36 |
| sdague | jog0: can you send an email to the openstack-dev list pointing people at this page if they want to help categorize jobs that have failed in the gate | 01:36 |
| jog0 | sdague: sure, I will do that tonight | 01:36 |
| sdague | I'm at the limits of english being useful right now :) | 01:36 |
| jog0 | sdague: heh I know the feeling | 01:36 |
| sdague | yeh, I am very much looking forward to the weekend :) | 01:37 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add per job classification rate to uncategorized.html https://review.openstack.org/68478 | 01:37 |
| *** derekh has quit IRC | 01:38 | |
| jog0 | so the check window thing should really help with all the unit test failures | 01:40 |
| openstackgerrit | Aaron Greengrass proposed a change to openstack-infra/config: Extend user creation with more granularity https://review.openstack.org/68776 | 01:40 |
| *** ArxCruz has quit IRC | 01:41 | |
| *** jhesketh has quit IRC | 01:41 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Increate rax IAD/DFW server limits https://review.openstack.org/68826 | 01:41 |
| *** jhesketh has joined #openstack-infra | 01:41 | |
| sdague | jog0: yep | 01:42 |
| sdague | also, not running unit tests in the gate | 01:42 |
| sdague | then nova unit test issue only impacts nova | 01:43 |
| sdague | which is appropriate | 01:43 |
| jog0 | although we did fix em | 01:43 |
| *** morganfainberg is now known as morganfainberg|z | 01:43 | |
| sdague | yep, definitely which is good | 01:43 |
| sdague | so I'll pick on glance instead :) | 01:44 |
| sdague | which I don't think has yet | 01:44 |
| jeblair | sdague: fwiw, that's the aspect i'm least comfortable with. i'm willing to try it, but i also would be happy if we find we are able to keep or restore that aspect. | 01:44 |
| jog0 | :) | 01:44 |
| sdague | jeblair: yeh, lets give it a shot and see how it all works | 01:44 |
| jog0 | I agree with jeblair, its a good experiment to run at the very least | 01:44 |
| jog0 | bbiab relo | 01:45 |
| jeblair | yep. just sharing my feelings. i hear that's good to do. | 01:45 |
| sdague | heh | 01:45 |
| jeblair | okay, all the new jenkinseses are running jobs now | 01:45 |
| jeblair | so nodepool is no longer being lied to | 01:45 |
| mriedem | we can rechekc no bug stable/havana patches now right? | 01:46 |
| jeblair | sdague: though note that the estimated times are going to be really off (you can see it in the progress bars) because the new jenkins masters don't have run time data yet | 01:46 |
| sdague | nice | 01:46 |
| mriedem | sdague: ^ ? | 01:46 |
| sdague | mriedem: yes stable/havana should be able to pass | 01:46 |
| mriedem | recheck no bug? | 01:46 |
| sdague | yeh | 01:46 |
| sdague | mriedem: then we can start roping you into helping tear through the uncategorized list - http://status.openstack.org/elastic-recheck/data/uncategorized.html :) | 01:47 |
| mriedem | oh barf | 01:48 |
| mriedem | i don't even know what that is yet | 01:48 |
| sdague | mriedem: that's every failed gate job | 01:48 |
| mriedem | i'm watching 'peppa pig' with a 2 year old and have a plastic tub on my head for some reason | 01:48 |
| sdague | which we don't match in ER yet | 01:48 |
| mriedem | oh nice | 01:48 |
| mriedem | so need fingerprints? | 01:48 |
| sdague | yes | 01:48 |
| sdague | and bugs | 01:48 |
| mriedem | let's have watson do it | 01:49 |
| mriedem | :( | 01:49 |
| sdague | watson cloud solves everything! | 01:49 |
| mriedem | i wish | 01:50 |
| mriedem | we'll see | 01:50 |
| sdague | see, it's funny, because I don't work there any more :) | 01:50 |
| mriedem | i know... :( | 01:50 |
| mriedem | funny | 01:50 |
| mriedem | jokes on me | 01:50 |
| clarkb | woot I can update the older jenkinses now | 01:52 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: do not be rediculous about precision https://review.openstack.org/68827 | 01:54 |
| sdague | jog0: ^^^ | 01:54 |
| sdague | because - gate-grenade-dsvm-neutron : 118 Uncategorized Fails. 4.06504065041% Classification Rate (123 Total Fails) | 01:55 |
| sdague | is kind of rediculous | 01:55 |
| *** jcooley_ has joined #openstack-infra | 01:57 | |
| mriedem | ha | 01:57 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Stashing https://review.openstack.org/68828 | 02:00 |
| *** jcooley_ has quit IRC | 02:00 | |
| sdague | ok, night all | 02:01 |
| *** jcooley_ has joined #openstack-infra | 02:02 | |
| sdague | based on what got fixed today, and - http://status.openstack.org/elastic-recheck/ - I'm definitely feeling like we're seeing some light at the end of the tunnel | 02:05 |
| clarkb | nice | 02:06 |
| sdague | I think if - Bug 1268732 - Slave went offline during log upload gets sorted, we should have some nice flow | 02:06 |
| *** obondarev has quit IRC | 02:09 | |
| *** obondarev has joined #openstack-infra | 02:10 | |
| *** jasondotstar has quit IRC | 02:11 | |
| bknudson | does the jenkins06 indicate this ran on a new jenkins? https://jenkins06.openstack.org/job/gate-grenade-dsvm/5/ | 02:13 |
| bknudson | (and failed there) | 02:13 |
| *** jp_at_hp has quit IRC | 02:16 | |
| clarkb | yes new jenkins | 02:17 |
| *** jcooley_ has quit IRC | 02:17 | |
| *** jcooley_ has joined #openstack-infra | 02:18 | |
| *** jcooley_ has quit IRC | 02:19 | |
| openstackgerrit | daisy-ycguo proposed a change to openstack-infra/config: Job to push Horizon translation to Transifex https://review.openstack.org/68042 | 02:20 |
| *** markwash has quit IRC | 02:26 | |
| *** SumitNaiksatam has quit IRC | 02:27 | |
| *** eharney has quit IRC | 02:30 | |
| *** praneshp has quit IRC | 02:30 | |
| *** esker has joined #openstack-infra | 02:30 | |
| *** esker has quit IRC | 02:32 | |
| *** rakhmerov has quit IRC | 02:34 | |
| *** jcooley_ has joined #openstack-infra | 02:35 | |
| *** dstanek has joined #openstack-infra | 02:35 | |
| *** eharney has joined #openstack-infra | 02:43 | |
| *** markmcclain has quit IRC | 02:44 | |
| *** dpyzhov has quit IRC | 02:48 | |
| *** jcooley_ has quit IRC | 02:49 | |
| *** emagana has quit IRC | 02:51 | |
| *** yamahata has quit IRC | 02:52 | |
| *** dguitarbite has joined #openstack-infra | 02:56 | |
| *** SumitNaiksatam has joined #openstack-infra | 02:56 | |
| *** dstufft has quit IRC | 02:58 | |
| *** dstufft has joined #openstack-infra | 02:59 | |
| *** thomasem has joined #openstack-infra | 02:59 | |
| *** praneshp has joined #openstack-infra | 02:59 | |
| *** dstufft_ has joined #openstack-infra | 03:02 | |
| *** markmcclain has joined #openstack-infra | 03:02 | |
| *** gokrokve has joined #openstack-infra | 03:03 | |
| *** thomasem has quit IRC | 03:03 | |
| *** markmcclain has quit IRC | 03:04 | |
| *** dstufft has quit IRC | 03:05 | |
| *** markmcclain has joined #openstack-infra | 03:07 | |
| *** dstufft_ is now known as dstufft | 03:10 | |
| *** praneshp has quit IRC | 03:11 | |
| *** matsuhashi has quit IRC | 03:14 | |
| *** melwitt has quit IRC | 03:15 | |
| *** dguitarbite has quit IRC | 03:16 | |
| *** mriedem has quit IRC | 03:27 | |
| *** UtahDave has joined #openstack-infra | 03:27 | |
| mordred | yay new jenkins! | 03:31 |
| mordred | clarkb: how bad did I screw up setting them up? | 03:31 |
| *** ryanpetrello has quit IRC | 03:32 | |
| *** senk has joined #openstack-infra | 03:34 | |
| *** oubiwann has quit IRC | 03:37 | |
| *** dguitarbite has joined #openstack-infra | 03:37 | |
| *** senk1 has joined #openstack-infra | 03:37 | |
| *** praneshp has joined #openstack-infra | 03:38 | |
| *** senk has quit IRC | 03:38 | |
| *** salv-orlando has quit IRC | 03:40 | |
| *** ryanpetrello has joined #openstack-infra | 03:41 | |
| *** senk1 has quit IRC | 03:41 | |
| *** ryanpetrello has quit IRC | 03:42 | |
| *** yaguang has joined #openstack-infra | 03:45 | |
| *** UtahDave has quit IRC | 03:46 | |
| *** ryanpetrello has joined #openstack-infra | 03:49 | |
| *** thuc_ has joined #openstack-infra | 03:49 | |
| *** praneshp has quit IRC | 03:49 | |
| Alex_Gaynor | Hmm, so there's a patch in the gate pipeline whose pep8 job isn't running for some reason (head of the queue) | 03:50 |
| *** ryanpetrello has quit IRC | 03:53 | |
| dims | Alex_Gaynor, looks like its running now | 03:57 |
| Alex_Gaynor | dims: indeed, I wonder what happened | 03:58 |
| dims | no clue! | 04:00 |
| *** eharney has quit IRC | 04:03 | |
| *** pcrews has quit IRC | 04:06 | |
| *** rakhmerov has joined #openstack-infra | 04:07 | |
| *** gokrokve has quit IRC | 04:09 | |
| *** gokrokve has joined #openstack-infra | 04:09 | |
| *** jcooley_ has joined #openstack-infra | 04:13 | |
| *** jcooley_ has quit IRC | 04:13 | |
| *** gokrokve has quit IRC | 04:13 | |
| *** jcooley_ has joined #openstack-infra | 04:13 | |
| Alex_Gaynor | Hmm, so now like none of the jobs seem to be going, some issue with nodepool? | 04:15 |
| *** gokrokve has joined #openstack-infra | 04:16 | |
| *** harlowja is now known as harlowja_away | 04:19 | |
| openstackgerrit | A change was merged to openstack-infra/gitdm: Add Gregory Haynes to HP https://review.openstack.org/68277 | 04:20 |
| *** gyee has quit IRC | 04:20 | |
| *** weshay has quit IRC | 04:22 | |
| *** rcleere has joined #openstack-infra | 04:25 | |
| clarkb | mordred: not too bad, jeblair's tarballs were incomplete | 04:29 |
| clarkb | mordred: basically needed the secret.key and ssh key data | 04:29 |
| clarkb | once it had that all was well | 04:29 |
| clarkb | Alex_Gaynor: dims: not an 'issue' just delay in spinning up a node capable of running the test | 04:29 |
| Alex_Gaynor | clarkb: okey doke! | 04:30 |
| clarkb | Alex_Gaynor: dims: possible there were other jobs ahead of that one in the gearman queue that wanted the same label | 04:30 |
| clarkb | probably from check jobs | 04:30 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support to list running jobs to zuul client https://review.openstack.org/68828 | 04:33 |
| clarkb | Alex_Gaynor: I do think something funny is happening with the available nodes though, there is a lot of green on the graph at status.o.o/zuul | 04:33 |
| clarkb | its a bit late for me to debug it now though | 04:34 |
| *** changbl has joined #openstack-infra | 04:35 | |
| clarkb | I think nodepool isn't properly connected to the new jenkins so isn't deleting used nodes from them | 04:36 |
| clarkb | if anyone is awake enough to poke at it ^ would be a start | 04:36 |
| *** matsuhashi has joined #openstack-infra | 04:39 | |
| *** markmcclain has quit IRC | 04:40 | |
| *** dkliban_afk has joined #openstack-infra | 04:47 | |
| mordred | clarkb: I think there are several things in those tarballs that can be easily added to the puppet | 04:51 |
| clarkb | mordred: probably | 04:52 |
| clarkb | mordred: if you are awake and want to figure out why nodepool isn't deleting nodes from new jenkinses that would be awesome | 04:52 |
| clarkb | mordred: I am guessing either nodepool didn't manage to connect to the zmq event publishers or jenkins did not open sockets on 8888 for some reason | 04:52 |
| clarkb | (was the zmq-event-publisher plugin installed properly? | 04:52 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support to list running jobs to zuul client https://review.openstack.org/68828 | 04:54 |
| mordred | clarkb: that's an excellent question | 04:55 |
| fungi | sorry for abandoning. my battery died just as i needed to split for dinner | 04:55 |
| mordred | clarkb: 8888 is not open on jenkins05 | 04:56 |
| openstackgerrit | Ivan Melnikov proposed a change to openstack-dev/hacking: Trigger warnings for raw and unicode docstrings https://review.openstack.org/68774 | 04:57 |
| mordred | clarkb: zmq-event-publisher isnt' ther AT ALL | 04:57 |
| clarkb | mordred: yeah, its one that requries install by hand iirc | 04:58 |
| mordred | clarkb: do we install the normal one? | 04:58 |
| mordred | ah. | 04:58 |
| mordred | where do I get it? copy from 4? | 04:58 |
| clarkb | mordred: yes from 04 | 04:58 |
| clarkb | mordred: though we may not need to do that since zaro did add publishing to jenkins maven repo | 04:58 |
| clarkb | and we haven't touched that plugin in a long time | 04:58 |
| fungi | i like the new zuul status visualization | 04:58 |
| clarkb | mordred: before you stop jenkins you want to delete all of the slaves | 04:58 |
| clarkb | otehrwise they may come back up as online | 04:58 |
| *** coolsvap has joined #openstack-infra | 04:59 | |
| clarkb | mordred: so I think process is shutdown mode, delete all slaves, stop jenkins install plugin start jenkins | 04:59 |
| * mordred cries | 04:59 | |
| mordred | clarkb: when you say delete slaves, do you mean anything other than delete from config.xml? | 04:59 |
| mordred | do I also need to delete from nodepool? | 04:59 |
| clarkb | mordred: yes you need to delete in both places | 05:00 |
| fungi | wow, yeah the "ready" node count is crazy high | 05:00 |
| * mordred cries more | 05:00 | |
| *** starmer has joined #openstack-infra | 05:00 | |
| fungi | mordred: the main issue is just that when jenkins restarts, it will automatically try to online any previously offline slaves it knows about | 05:01 |
| fungi | and then they will get new jobs, even if they were single-use slaves | 05:01 |
| fungi | so deleting slaves which are marked offline in the jenkins master is the best workaround we have for that | 05:02 |
| *** yamahata has joined #openstack-infra | 05:02 | |
| fungi | anyway, i'm going back to prodding devstack logging into shape | 05:04 |
| fungi | looks like my most recent test still didn't work as intended | 05:05 |
| *** kraman1 has quit IRC | 05:06 | |
| clarkb | fungi: if logging hasn't been sorted by tomorrow I intend on pitching in on that | 05:06 |
| mordred | fungi, clarkb: I have copied the plugin - I'm going to step away for a minute. if it hasn't been done when I'm back I'll get it done | 05:06 |
| *** UtahDave has joined #openstack-infra | 05:08 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: do not be rediculous about precision https://review.openstack.org/68827 | 05:11 |
| *** mrodden1 has joined #openstack-infra | 05:12 | |
| *** mrodden has quit IRC | 05:12 | |
| fungi | clarkb: tomorrow will be appreciated. i'll be on airplanes and in airports again for much of the day, so probably not remarkably useful | 05:14 |
| zaro | clarkb, mordred : latest release of zmq event publisher should be available from jenkins plugin update page. you might need to click the check updates button on jenkis. | 05:14 |
| *** vkozhukalov has joined #openstack-infra | 05:18 | |
| *** ryanpetrello has joined #openstack-infra | 05:19 | |
| *** ryanpetrello has quit IRC | 05:20 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 05:22 | |
| *** yamahata has quit IRC | 05:23 | |
| *** yamahata has joined #openstack-infra | 05:23 | |
| openstackgerrit | Ivan Melnikov proposed a change to openstack-dev/hacking: Fix python 3 incompatibility in _get_import_type https://review.openstack.org/68849 | 05:24 |
| *** talluri has joined #openstack-infra | 05:24 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: change user from zaro to zaro0508 to match LP user account https://review.openstack.org/68850 | 05:27 |
| clarkb | zaro: I am not sure I like that change. It doesn't remove the old one and there is no reason to have them match | 05:30 |
| *** dkliban_afk has quit IRC | 05:31 | |
| zaro | clarkb: i wanted that change because i was trying to use my account to integrate a gerritbot with review-dev. i'm having a difficult time getting it to work. | 05:34 |
| zaro | clarkb: i'll abandon if i can get you to help me with the integration. | 05:34 |
| *** nicedice has quit IRC | 05:35 | |
| zaro | clarkb: had a hard time using 'zaro' because that's not my account on review-dev and since review-dev is tied to LP, i didn't want to create new LP account. | 05:35 |
| *** dosaboy has quit IRC | 05:36 | |
| *** ogelbukh has quit IRC | 05:36 | |
| *** max_lobur_afk has quit IRC | 05:36 | |
| *** dosaboy has joined #openstack-infra | 05:37 | |
| *** max_lobur_afk has joined #openstack-infra | 05:37 | |
| *** ogelbukh has joined #openstack-infra | 05:37 | |
| lifeless | https://review.openstack.org/#/c/68645/ should be good for +2s now | 05:38 |
| *** gokrokve has quit IRC | 05:47 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Added rbenv wrapper https://review.openstack.org/65352 | 05:47 |
| *** gokrokve has joined #openstack-infra | 05:48 | |
| *** yamahata has quit IRC | 05:50 | |
| *** yamahata has joined #openstack-infra | 05:50 | |
| *** amotoki_zzz is now known as amotoki | 05:51 | |
| *** thuc_ has quit IRC | 05:51 | |
| fungi | clarkb: do i need to delete all those ready nodes on jenkins05-07 so we'll start running jobs again? | 05:52 |
| *** thuc has joined #openstack-infra | 05:52 | |
| fungi | i didn't entirely grasp what the current state is on the new masters | 05:53 |
| *** gokrokve has quit IRC | 05:53 | |
| clarkb | fungi: all of the nodes should be offline | 05:55 |
| clarkb | they ran jobs and were offlined by gearman plugin but nodepool diddn't delete them due to missing plugin | 05:55 |
| fungi | well, they're still marked ready in nodepool | 05:55 |
| clarkb | zaro: you should be able to specify a username and keyfile | 05:55 |
| fungi | i'll clean them up | 05:55 |
| clarkb | fungi: right, we need to delete only those that were on newer jenkins | 05:55 |
| fungi | yep | 05:56 |
| clarkb | zaro: the name of the user it is running the process under doesn't matter | 05:56 |
| *** thuc has quit IRC | 05:56 | |
| fungi | clarkb: do the masters still need new plugins (should i put them in prepare for shutdown mode so they don't get more assigned to them yet)? | 05:56 |
| zaro | clarkb: yeah, i tried. i have to use the gerrit user that has ssh access to view the gerrit stream. that would be 'zaro0508' | 05:57 |
| clarkb | zaro: yeah but you can specify that | 05:57 |
| clarkb | it doesn't have to be the user the process is running as | 05:57 |
| zaro | clarkb: yeah, i did specify but it doesn't seem to want to connect. | 05:59 |
| zaro | clarkb: i can't get the logging to work either so i can't tell what the error might be. | 05:59 |
| zaro | clarkb: i tried connecting to gerrit stream outside of gerritbot, was able to connect but had to enter my passphrase to work. | 06:00 |
| zaro | clarkb: i'm thinking that might be the problem. | 06:00 |
| clarkb | zaro: thats probably the problem | 06:00 |
| zaro | clarkb: it seems to only ask for passphase only when i run 'ssh -i', but not when i run just 'ssh' | 06:01 |
| clarkb | zaro: are they two different keys one with a passphrase and one without? | 06:02 |
| zaro | clarkb: nope same key. | 06:03 |
| clarkb | it should always ask you for the passphrase then unless you have a key agent running | 06:03 |
| zaro | maybe key-agent is what i need? | 06:03 |
| *** mrda is now known as mrda_away | 06:03 | |
| *** praneshp has joined #openstack-infra | 06:04 | |
| clarkb | or a second passphrase less key | 06:04 |
| clarkb | not sure if gerritbot is smart enough to take advantage of an agent (or paramiko) | 06:05 |
| *** wenlock has quit IRC | 06:06 | |
| *** otherwiseguy has quit IRC | 06:06 | |
| zaro | ok. i was confused. it's always asking for my passphrase when using ssh | 06:07 |
| zaro | clarkb: i'll try to setup a key agent to see if that works | 06:07 |
| fungi | yes, for production gerritbot we just use an unencrypted private key, so no decryption prompt needed | 06:07 |
| zaro | i can create a new passphrase less key for myself, but wouldn't that require a lot of changes to everthing i have setup? | 06:09 |
| clarkb | zaro: no, put it in a different file | 06:11 |
| clarkb | then put the public key in gerrit | 06:11 |
| zaro | i assume i'll also need to put the pub key for all other systems as well, like hpcloud, review-dev, etc.. | 06:12 |
| *** jcooley_ has quit IRC | 06:13 | |
| zaro | clarkb: ?? | 06:13 |
| clarkb | zaro: no, just the things you are talking to | 06:13 |
| clarkb | in this case gerritbot only talks to gerrit | 06:13 |
| *** praneshp_ has joined #openstack-infra | 06:14 | |
| *** gokrokve has joined #openstack-infra | 06:14 | |
| zaro | clarkb: ohh, you mean only use new key for gerrit and others stay the same? | 06:14 |
| zaro | i think that's doable. | 06:15 |
| clarkb | yes | 06:15 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Create a git cache for tripleo-gate. https://review.openstack.org/68807 | 06:16 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Switch path to toci_gate_test.sh https://review.openstack.org/68662 | 06:16 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Remove TRIPLEO_ROOT and pull-tools https://review.openstack.org/68661 | 06:16 |
| zaro | cool. that sounds like the easiest thing todo. i'll give that a try. | 06:16 |
| *** praneshp has quit IRC | 06:17 | |
| *** praneshp_ is now known as praneshp | 06:17 | |
| lifeless | clarkb: any change you could cast an eye over https://review.openstack.org/#/q/status:open+branch:master+topic:ci-work,n,z | 06:21 |
| clarkb | lifeless: sure | 06:23 |
| lifeless | thanks! | 06:24 |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 06:29 | |
| openstackgerrit | Ivan Melnikov proposed a change to openstack-dev/hacking: More portable way to detect modules for H302 https://review.openstack.org/68858 | 06:32 |
| openstackgerrit | Ivan Melnikov proposed a change to openstack-dev/hacking: More portable way to detect modules for H302 https://review.openstack.org/68858 | 06:33 |
| *** otherwiseguy has joined #openstack-infra | 06:34 | |
| clarkb | lifeless: reviewed | 06:36 |
| lifeless | oh | 06:37 |
| lifeless | d-g will fail on missing repos ? | 06:37 |
| clarkb | it won't fail, but it will actually do a real clone | 06:37 |
| clarkb | normally we copy the cached repo into the workspace then update the repos instead of doing a full clone | 06:38 |
| clarkb | that way the delta is small, less than 24 hours | 06:38 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Create a git cache for tripleo-gate. https://review.openstack.org/68807 | 06:39 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Switch path to toci_gate_test.sh https://review.openstack.org/68662 | 06:39 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Remove TRIPLEO_ROOT and pull-tools https://review.openstack.org/68661 | 06:39 |
| *** yamahata has quit IRC | 06:40 | |
| lifeless | clarkb: sorted | 06:40 |
| clarkb | lifeless: lgtm | 06:41 |
| lifeless | clarkb: so, next up,mordred? | 06:42 |
| lifeless | mordred: https://review.openstack.org/#/q/status:open+branch:master+topic:ci-work,n,z | 06:42 |
| lifeless | since jim is sick and fungi should be asleep | 06:42 |
| *** otherwiseguy has quit IRC | 06:43 | |
| jog0 | sdague: so I ran out of time to send the email about uncategorized.html tonight (I got distracted by life) so if you want to send it out in the AM otherwise I will (but in my AM which is your PM) | 06:43 |
| mordred | lifeless: I'm several cachaças into the evening. I think my +A flag should not be used | 06:47 |
| mordred | lifeless: although I support and affirm you and your endeavors | 06:47 |
| lifeless | hah | 06:51 |
| lifeless | fungi - still up :) | 06:51 |
| *** vkozhukalov has quit IRC | 06:55 | |
| *** yamahata has joined #openstack-infra | 06:56 | |
| *** odyssey4me has joined #openstack-infra | 07:01 | |
| *** jcooley_ has joined #openstack-infra | 07:07 | |
| *** yolanda_ has joined #openstack-infra | 07:08 | |
| *** jcooley_ has quit IRC | 07:12 | |
| *** jcooley_ has joined #openstack-infra | 07:14 | |
| openstackgerrit | A change was merged to openstack-infra/config: Create a git cache for tripleo-gate. https://review.openstack.org/68807 | 07:15 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 07:16 | |
| *** DinaBelova_ is now known as DinaBelova | 07:18 | |
| *** jcooley_ has quit IRC | 07:18 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove TRIPLEO_ROOT and pull-tools https://review.openstack.org/68661 | 07:19 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch path to toci_gate_test.sh https://review.openstack.org/68662 | 07:19 |
| lifeless | \o/ | 07:19 |
| * lifeless does the happy happy testy testy dance | 07:19 | |
| *** dpyzhov has joined #openstack-infra | 07:20 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 07:22 | |
| lifeless | clarkb: if possible, getting that config onto the nodepool environment and triggering an image update would be awesome, unless the next once is going to happen automatically in the next few hours | 07:25 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Make gate-tripleo-deploy check for tripleo stuff. https://review.openstack.org/68866 | 07:26 |
| *** zhiyan has joined #openstack-infra | 07:26 | |
| *** eharney has joined #openstack-infra | 07:27 | |
| *** zhiyan has quit IRC | 07:31 | |
| *** dpyzhov has quit IRC | 07:31 | |
| *** oubiwann_ has quit IRC | 07:31 | |
| *** afazekas has joined #openstack-infra | 07:32 | |
| *** UtahDave has quit IRC | 07:35 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_a | 07:39 | |
| *** SergeyLukjanov_a is now known as SergeyLukjanov_ | 07:40 | |
| *** matsuhashi has quit IRC | 07:42 | |
| *** nosnos has quit IRC | 07:43 | |
| *** dstanek has quit IRC | 07:44 | |
| *** senk has joined #openstack-infra | 07:45 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 07:49 | |
| *** senk has quit IRC | 07:50 | |
| *** bauzas has joined #openstack-infra | 08:01 | |
| *** jcooley_ has joined #openstack-infra | 08:02 | |
| *** mancdaz_away is now known as mancdaz | 08:02 | |
| *** jcoufal has joined #openstack-infra | 08:05 | |
| *** gokrokve has quit IRC | 08:08 | |
| *** gokrokve has joined #openstack-infra | 08:08 | |
| *** gokrokve has quit IRC | 08:12 | |
| *** luqas has joined #openstack-infra | 08:14 | |
| *** flaper87|afk is now known as flaper87 | 08:15 | |
| *** fbo is now known as fbo_away | 08:18 | |
| *** mancdaz is now known as mancdaz_away | 08:22 | |
| *** starmer has quit IRC | 08:23 | |
| *** starmer has joined #openstack-infra | 08:24 | |
| *** vkozhukalov has joined #openstack-infra | 08:27 | |
| *** pblaho has joined #openstack-infra | 08:30 | |
| *** jcooley_ has quit IRC | 08:36 | |
| *** dpyzhov has joined #openstack-infra | 08:39 | |
| *** cyeoh has quit IRC | 08:41 | |
| *** cyeoh has joined #openstack-infra | 08:42 | |
| *** yassine has joined #openstack-infra | 08:54 | |
| *** pblaho has quit IRC | 08:56 | |
| *** fbo_away is now known as fbo | 08:59 | |
| *** mancdaz_away is now known as mancdaz | 08:59 | |
| *** jpich has joined #openstack-infra | 09:02 | |
| *** yassine has quit IRC | 09:03 | |
| *** yassine has joined #openstack-infra | 09:03 | |
| *** yassine has quit IRC | 09:04 | |
| *** yassine has joined #openstack-infra | 09:04 | |
| *** yassine has quit IRC | 09:05 | |
| *** yassine has joined #openstack-infra | 09:06 | |
| *** yassine has quit IRC | 09:06 | |
| *** eharney has quit IRC | 09:07 | |
| *** yassine has joined #openstack-infra | 09:07 | |
| *** yassine has quit IRC | 09:07 | |
| *** eharney has joined #openstack-infra | 09:08 | |
| *** yassine has joined #openstack-infra | 09:08 | |
| *** yassine has quit IRC | 09:09 | |
| *** yassine has joined #openstack-infra | 09:10 | |
| *** yassine has quit IRC | 09:12 | |
| *** yassine has joined #openstack-infra | 09:12 | |
| *** eharney has quit IRC | 09:14 | |
| *** yassine has quit IRC | 09:17 | |
| *** yassine has joined #openstack-infra | 09:17 | |
| *** jcooley_ has joined #openstack-infra | 09:26 | |
| *** derekh has joined #openstack-infra | 09:27 | |
| *** jcooley_ has quit IRC | 09:28 | |
| *** jcooley_ has joined #openstack-infra | 09:29 | |
| *** shardy_afk is now known as shardy | 09:31 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_a | 09:31 | |
| *** SergeyLukjanov_a is now known as SergeyLukjanov_ | 09:32 | |
| *** yassine has quit IRC | 09:33 | |
| *** yassine has joined #openstack-infra | 09:33 | |
| *** jcooley_ has quit IRC | 09:35 | |
| *** pblaho has joined #openstack-infra | 09:36 | |
| *** yassine has quit IRC | 09:37 | |
| *** yassine has joined #openstack-infra | 09:38 | |
| *** yassine has quit IRC | 09:39 | |
| *** yassine has joined #openstack-infra | 09:40 | |
| *** yassine has quit IRC | 09:42 | |
| *** DinaBelova is now known as DinaBelova_ | 09:42 | |
| *** yassine has joined #openstack-infra | 09:42 | |
| *** luqas has quit IRC | 09:47 | |
| *** salv-orlando has joined #openstack-infra | 09:49 | |
| openstackgerrit | Pavel Sedlák proposed a change to openstack-infra/jenkins-job-builder: Add support for Test Stability with Junit https://review.openstack.org/68152 | 09:50 |
| *** bauzas has left #openstack-infra | 09:52 | |
| *** johnthetubaguy has joined #openstack-infra | 09:55 | |
| *** mkerrin has joined #openstack-infra | 09:55 | |
| *** chandankumar_ has quit IRC | 09:56 | |
| *** BobBallAway is now known as BobBall | 09:59 | |
| *** starmer has quit IRC | 10:00 | |
| *** jasondotstar has joined #openstack-infra | 10:01 | |
| *** DinaBelova_ is now known as DinaBelova | 10:10 | |
| *** max_lobur_afk is now known as max_lobur | 10:11 | |
| *** che-arne has joined #openstack-infra | 10:11 | |
| *** che-arne|2 has joined #openstack-infra | 10:11 | |
| *** che-arne|2 has quit IRC | 10:11 | |
| *** wenlock has joined #openstack-infra | 10:14 | |
| *** wenlock has quit IRC | 10:18 | |
| *** jcooley_ has joined #openstack-infra | 10:20 | |
| *** jcooley_ has quit IRC | 10:25 | |
| *** luqas has joined #openstack-infra | 10:27 | |
| *** cody-somerville has quit IRC | 10:28 | |
| *** che-arne has quit IRC | 10:30 | |
| *** che-arne has joined #openstack-infra | 10:32 | |
| *** EmilienM has quit IRC | 10:32 | |
| *** EmilienM has joined #openstack-infra | 10:32 | |
| *** praneshp has quit IRC | 10:41 | |
| *** cody-somerville has joined #openstack-infra | 10:42 | |
| *** vogxn has joined #openstack-infra | 10:51 | |
| *** DinaBelova is now known as DinaBelova_ | 10:54 | |
| *** fbo has quit IRC | 10:55 | |
| *** dachary has quit IRC | 10:55 | |
| *** fbo has joined #openstack-infra | 10:56 | |
| *** dachary has joined #openstack-infra | 10:58 | |
| *** yaguang has quit IRC | 11:13 | |
| *** jcooley_ has joined #openstack-infra | 11:15 | |
| *** jcooley_ has quit IRC | 11:19 | |
| *** ArxCruz has joined #openstack-infra | 11:26 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: Upgrade six to 1.5.2 https://review.openstack.org/68424 | 11:26 |
| *** ok_delta has joined #openstack-infra | 11:26 | |
| *** jp_at_hp has joined #openstack-infra | 11:27 | |
| *** yamahata has quit IRC | 11:33 | |
| *** vogxn has quit IRC | 11:36 | |
| *** vogxn has joined #openstack-infra | 11:38 | |
| *** jasondotstar has quit IRC | 11:40 | |
| *** rfolco has joined #openstack-infra | 11:41 | |
| *** sirushti has quit IRC | 11:43 | |
| *** ProfFalken has joined #openstack-infra | 11:44 | |
| *** sirushti has joined #openstack-infra | 11:45 | |
| *** ok_delta has quit IRC | 11:45 | |
| *** _ruhe is now known as ruhe | 11:49 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 11:49 | |
| *** coolsvap_away has joined #openstack-infra | 11:54 | |
| *** coolsvap has quit IRC | 11:55 | |
| *** coolsvap_away has quit IRC | 11:58 | |
| *** dpyzhov has quit IRC | 12:02 | |
| ArxCruz | ALL: in nodepool.yaml the credentials-id is an rsa key or the id for some key I've added on jenkins ? | 12:06 |
| sdague | fungi: when you rise ... I think we want to raise the floor again on the gate, especially with the new resources | 12:06 |
| sdague | I'm regularly seeing us merge chunks of 10 changes | 12:06 |
| *** jcooley_ has joined #openstack-infra | 12:09 | |
| *** vogxn has quit IRC | 12:11 | |
| *** jcooley_ has quit IRC | 12:18 | |
| *** pblaho has quit IRC | 12:21 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Support recursiving processing of paths https://review.openstack.org/68895 | 12:21 |
| *** luqas has quit IRC | 12:23 | |
| *** matrohon has quit IRC | 12:30 | |
| *** chandankumar_ has joined #openstack-infra | 12:31 | |
| *** chandankumar_ has quit IRC | 12:34 | |
| *** markmc has joined #openstack-infra | 12:36 | |
| *** vogxn has joined #openstack-infra | 12:37 | |
| *** dpyzhov has joined #openstack-infra | 12:42 | |
| *** gmurphy has left #openstack-infra | 12:44 | |
| *** pblaho1 has joined #openstack-infra | 12:45 | |
| *** smarcet has joined #openstack-infra | 12:50 | |
| *** pblaho1 has quit IRC | 12:54 | |
| sdague | more swift failing unit tests in the gate - 68485,2 | 12:56 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 12:56 | |
| *** talluri has quit IRC | 12:59 | |
| *** talluri has joined #openstack-infra | 12:59 | |
| *** DinaBelova_ is now known as DinaBelova | 13:00 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 13:01 | |
| *** matrohon has joined #openstack-infra | 13:01 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 13:02 | |
| *** dpyzhov has quit IRC | 13:02 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 13:03 | |
| *** jcooley_ has joined #openstack-infra | 13:04 | |
| *** talluri has quit IRC | 13:04 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 13:06 | |
| *** dpyzhov has joined #openstack-infra | 13:06 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 13:06 | |
| *** ruhe is now known as _ruhe | 13:07 | |
| *** jcooley_ has quit IRC | 13:08 | |
| *** prad has joined #openstack-infra | 13:10 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 13:11 | |
| *** pblaho has joined #openstack-infra | 13:15 | |
| *** mugsie has quit IRC | 13:17 | |
| *** _ruhe is now known as ruhe | 13:18 | |
| *** mugsie has joined #openstack-infra | 13:19 | |
| *** vogxn has quit IRC | 13:20 | |
| *** johnthetubaguy1 has joined #openstack-infra | 13:21 | |
| *** pblaho has quit IRC | 13:21 | |
| *** johnthetubaguy has quit IRC | 13:21 | |
| *** pblaho has joined #openstack-infra | 13:21 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 13:23 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 13:24 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 13:26 | |
| *** jasondotstar has joined #openstack-infra | 13:32 | |
| *** Ajaeger has joined #openstack-infra | 13:33 | |
| *** thomasem has joined #openstack-infra | 13:33 | |
| *** julim has joined #openstack-infra | 13:34 | |
| *** pblaho has quit IRC | 13:34 | |
| *** pblaho has joined #openstack-infra | 13:35 | |
| bknudson | sdague: I always look forward to the "state of the gate" address. | 13:37 |
| sdague | bknudson: heh... | 13:38 |
| sdague | I do think we are turning a corner | 13:38 |
| *** mfink has quit IRC | 13:39 | |
| anteaya | I too enjoy the daily mail | 13:39 |
| anteaya | better than my national newspaper | 13:40 |
| bknudson | I agree.76 and falling better than 120. | 13:40 |
| *** dpyzhov has quit IRC | 13:40 | |
| *** thomasem_ has joined #openstack-infra | 13:41 | |
| *** pblaho has quit IRC | 13:41 | |
| *** pblaho has joined #openstack-infra | 13:41 | |
| *** ArxCruz has quit IRC | 13:42 | |
| *** openstackgerrit has quit IRC | 13:42 | |
| *** spiffxp has quit IRC | 13:42 | |
| anteaya | we lost openstackgerrit | 13:42 |
| *** pblaho` has joined #openstack-infra | 13:42 | |
| anteaya | another few hours before anyone is awake | 13:42 |
| *** ArxCruz has joined #openstack-infra | 13:42 | |
| *** dpyzhov has joined #openstack-infra | 13:42 | |
| *** dhellmann has quit IRC | 13:42 | |
| anteaya | unless mordred is about? | 13:42 |
| *** alaski has quit IRC | 13:43 | |
| *** westmaas has quit IRC | 13:43 | |
| anteaya | can you bring back openstackgerrit? | 13:43 |
| *** smarcet has quit IRC | 13:43 | |
| *** medberry has quit IRC | 13:43 | |
| *** aude has quit IRC | 13:43 | |
| *** pblaho` has quit IRC | 13:43 | |
| *** pblaho has quit IRC | 13:43 | |
| *** jeremyb has quit IRC | 13:43 | |
| *** davidlenwell has quit IRC | 13:43 | |
| *** julim has quit IRC | 13:43 | |
| *** dolphm has quit IRC | 13:43 | |
| *** davidlenwell has joined #openstack-infra | 13:43 | |
| *** zaro has quit IRC | 13:43 | |
| *** belliott has quit IRC | 13:43 | |
| *** hdd has quit IRC | 13:43 | |
| *** Hefeweizen has quit IRC | 13:43 | |
| *** blamar has quit IRC | 13:43 | |
| *** soren has quit IRC | 13:43 | |
| *** thomasem has quit IRC | 13:44 | |
| *** MarkAtwood_ has joined #openstack-infra | 13:44 | |
| *** hughsaunders has quit IRC | 13:44 | |
| *** andreaf has joined #openstack-infra | 13:44 | |
| *** MarkAtwood has quit IRC | 13:44 | |
| *** rfolco has quit IRC | 13:44 | |
| *** yamahata_ has quit IRC | 13:44 | |
| *** alaski has joined #openstack-infra | 13:44 | |
| *** bookwar has quit IRC | 13:45 | |
| *** westmaas has joined #openstack-infra | 13:45 | |
| *** aude has joined #openstack-infra | 13:45 | |
| *** jeremyb has joined #openstack-infra | 13:45 | |
| *** soren has joined #openstack-infra | 13:45 | |
| *** soren has quit IRC | 13:45 | |
| *** soren has joined #openstack-infra | 13:45 | |
| *** mrmartin has joined #openstack-infra | 13:45 | |
| *** harlowja_away has quit IRC | 13:45 | |
| *** spiffxp has joined #openstack-infra | 13:46 | |
| *** julim has joined #openstack-infra | 13:46 | |
| *** asadoughi has quit IRC | 13:46 | |
| *** Hefeweizen has joined #openstack-infra | 13:46 | |
| *** blamar has joined #openstack-infra | 13:47 | |
| *** pblaho has joined #openstack-infra | 13:48 | |
| *** hughsaunders has joined #openstack-infra | 13:48 | |
| *** belliott has joined #openstack-infra | 13:49 | |
| *** dolphm has joined #openstack-infra | 13:49 | |
| *** dhellmann has joined #openstack-infra | 13:49 | |
| *** yamahata_ has joined #openstack-infra | 13:49 | |
| *** miqui has joined #openstack-infra | 13:50 | |
| *** DinaBelova is now known as DinaBelova_ | 13:50 | |
| *** DinaBelova_ is now known as DinaBelova | 13:52 | |
| *** thuc has joined #openstack-infra | 13:54 | |
| *** rlandy has joined #openstack-infra | 13:55 | |
| *** mfink has joined #openstack-infra | 13:55 | |
| *** rfolco has joined #openstack-infra | 13:56 | |
| *** gsamfira has joined #openstack-infra | 13:56 | |
| *** hdd has joined #openstack-infra | 13:56 | |
| *** xchu has joined #openstack-infra | 13:57 | |
| *** zaro has joined #openstack-infra | 13:57 | |
| *** smarcet has joined #openstack-infra | 13:58 | |
| *** jcooley_ has joined #openstack-infra | 13:58 | |
| *** bookwar has joined #openstack-infra | 13:58 | |
| *** luqas has joined #openstack-infra | 13:59 | |
| *** mfer has joined #openstack-infra | 14:00 | |
| *** jcooley_ has quit IRC | 14:02 | |
| *** dims has quit IRC | 14:02 | |
| *** dpyzhov has quit IRC | 14:04 | |
| *** rlandy has quit IRC | 14:04 | |
| *** weshay has joined #openstack-infra | 14:05 | |
| *** dims has joined #openstack-infra | 14:05 | |
| *** changbl has quit IRC | 14:06 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 14:07 | |
| *** eharney has joined #openstack-infra | 14:07 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 14:09 | |
| anteaya | sdague: any thoughts on why so many in the gate window have queued tests? | 14:17 |
| sdague | anteaya: did we just reset? | 14:17 |
| *** johnthetubaguy1 is now known as johnthetubaguy | 14:17 | |
| anteaya | the head of teh gate seemed to have been running for almost an hour | 14:17 |
| anteaya | so I would go with no, the last reset appears to have been an hour ago | 14:17 |
| mrmartin | re | 14:17 |
| anteaya | unless I am missing some information | 14:17 |
| sdague | anteaya: the job at top looks like it just started | 14:18 |
| sdague | so I'm going reset | 14:18 |
| *** b3nt_pin is now known as beagles | 14:18 | |
| *** ryanpetrello has joined #openstack-infra | 14:19 | |
| *** jdurgin has quit IRC | 14:19 | |
| anteaya | ah then I am missunderstanding the meaning of 53minutes in the status bar for that patch | 14:19 |
| sdague | 53 minutes is estimated time to completion | 14:19 |
| anteaya | ah | 14:19 |
| anteaya | I was going the wrong way | 14:19 |
| *** mriedem has joined #openstack-infra | 14:19 | |
| anteaya | then yes, it had just reset | 14:19 |
| anteaya | my mistake | 14:19 |
| mrmartin | sdague: could you check me this https://review.openstack.org/#/c/68912/ review if you have a few minutes. The gate-config-layout fails, but I cannot determine the exact reason from the backtrace. | 14:20 |
| *** milki has quit IRC | 14:20 | |
| *** kraman has quit IRC | 14:21 | |
| mriedem | looking at http://status.openstack.org/elastic-recheck/data/uncategorized.html | 14:23 |
| mriedem | does that somehow exclude failures from patches that have actual problems in their code that makes the unit tests fail? | 14:23 |
| mriedem | or does it cross multiple patches for common failures? | 14:24 |
| *** oubiwann has joined #openstack-infra | 14:24 | |
| *** fifieldt has joined #openstack-infra | 14:28 | |
| *** dprince has joined #openstack-infra | 14:28 | |
| *** thuc has quit IRC | 14:31 | |
| *** thuc has joined #openstack-infra | 14:32 | |
| *** jdurgin has joined #openstack-infra | 14:33 | |
| *** thuc has quit IRC | 14:36 | |
| *** milki has joined #openstack-infra | 14:39 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 14:41 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 14:41 | |
| *** burt1 has joined #openstack-infra | 14:43 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 14:51 | |
| *** jcooley_ has joined #openstack-infra | 14:52 | |
| *** mrmartin has quit IRC | 14:53 | |
| *** jcooley_ has quit IRC | 14:57 | |
| *** odyssey4me has quit IRC | 14:59 | |
| *** dstanek has joined #openstack-infra | 15:01 | |
| *** fallenpegasus has joined #openstack-infra | 15:03 | |
| *** alexpilotti has joined #openstack-infra | 15:03 | |
| mriedem | sdague: question about uncategorized e-r hits above? ^ | 15:04 |
| mriedem | wanted to clear that up before i start digging into those | 15:04 |
| *** xchu has quit IRC | 15:04 | |
| *** jgrimm has joined #openstack-infra | 15:08 | |
| *** mgagne1 has quit IRC | 15:09 | |
| *** mgagne has joined #openstack-infra | 15:10 | |
| *** CaptTofu has joined #openstack-infra | 15:11 | |
| *** tristanC has joined #openstack-infra | 15:14 | |
| *** bookwar has quit IRC | 15:15 | |
| *** fifieldt has quit IRC | 15:17 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 15:19 | |
| *** kraman has joined #openstack-infra | 15:21 | |
| *** jgrimm has quit IRC | 15:22 | |
| *** dcramer_ has joined #openstack-infra | 15:22 | |
| bknudson | things are merging at breakneck speed. | 15:26 |
| mriedem | yeah. on another note, need an e-r fingerprint for bug 1267326. | 15:33 |
| *** jgrimm has joined #openstack-infra | 15:34 | |
| *** mrmartin has joined #openstack-infra | 15:36 | |
| *** pblaho has quit IRC | 15:38 | |
| dims | mriedem, gate length down to 69 as well | 15:38 |
| *** rossella_s has quit IRC | 15:40 | |
| *** vkozhukalov has quit IRC | 15:43 | |
| HenryG | sdague: ping | 15:43 |
| *** jeremyb has quit IRC | 15:44 | |
| *** jeremyb has joined #openstack-infra | 15:44 | |
| mriedem | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRkFJTDogdGVtcGVzdC5hcGkuY29tcHV0ZS5zZXJ2ZXJzLnRlc3Rfc2VydmVyX2FjdGlvbnMuU2VydmVyQWN0aW9uc1Rlc3RKU09OLnRlc3RfY3JlYXRlX2JhY2t1cFwiIEFORCBmaWxlbmFtZTpcImNvbnNvbGUuaHRtbFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzkwNTc4MTcwMzAxfQ== | 15:44 |
| *** AlexF_ has joined #openstack-infra | 15:44 | |
| mriedem | that's not a great query but i don't know of a better way w/o multiline query support | 15:44 |
| *** afazekas has quit IRC | 15:45 | |
| *** asadoughi has joined #openstack-infra | 15:46 | |
| *** jcooley_ has joined #openstack-infra | 15:46 | |
| HenryG | Question about neutron unit tests: if I run them locally they fail due to the new pip not liking the psutil version. Does the gate not use the new pip? | 15:47 |
| Alex_Gaynor | HenryG: the gate has it's own mirror which it uses, it doesn't try to install from whatever random site psutil is hosted on | 15:48 |
| ttx | sdague: couldn't find any nova stable/* changes gating, although https://review.openstack.org/#/c/61924/ and https://review.openstack.org/#/c/60550/ look approved. Let me know if you need my help to kick anything out | 15:49 |
| HenryG | Alex_Gaynor: thanks | 15:50 |
| *** jcooley_ has quit IRC | 15:51 | |
| *** oubiwann has quit IRC | 15:51 | |
| *** dkliban has joined #openstack-infra | 15:52 | |
| *** matrohon has quit IRC | 15:53 | |
| mordred | HenryG: also, we have the gate pinned at a previous version of pip | 15:53 |
| mordred | HenryG: due to a bug | 15:54 |
| mordred | HenryG: we're ready to remove the pin - but with the state the gate has been in, we didnt' want to stress it more in case something else went wrong | 15:54 |
| mordred | shoudl all be sorted out soon | 15:54 |
| *** jergerber has joined #openstack-infra | 15:54 | |
| *** jergerber has quit IRC | 15:55 | |
| *** jergerber has joined #openstack-infra | 15:55 | |
| Alex_Gaynor | mordred: gate seems to be moving faster in the last 12 hours, was one of the big issues fixed, or just the increase in workers? | 15:56 |
| mordred | Alex_Gaynor: actually, the biggest thing we landed was a throttling feature to zuul | 15:56 |
| mordred | and - then - also more workers | 15:56 |
| HenryG | mordred: OK, because with the new pip neutron needs https://review.openstack.org/65209 | 15:56 |
| mordred | HenryG: yeah - Alex_Gaynor ^^ were we adding those directly? or doing an env var thing? do you remember? | 15:57 |
| *** oubiwann has joined #openstack-infra | 15:57 | |
| Alex_Gaynor | mordred: I don't remember what the decision was, clearly we should just replace those deps in order to avoid needing ot make a decision | 15:58 |
| mordred | HenryG: also, seriously, if someone could find the psutil maintainer that woudl be great - they really need to upload a proper release | 15:58 |
| mordred | Alex_Gaynor: I agree | 15:58 |
| Alex_Gaynor | I thought I filed a bug against psutil | 15:58 |
| mordred | Alex_Gaynor: there's only four of them IIRC | 15:58 |
| *** wenlock has joined #openstack-infra | 15:58 | |
| HenryG | mordred: psutil is fine. The neutron requirements.txt is not. | 15:58 |
| mordred | HenryG: how is that? | 15:59 |
| HenryG | mordred: neutron is pinned to psutil>=0.6.1,<1.0 for no good reason | 15:59 |
| mordred | HenryG: is there a newer psutil that's actually on pypi? | 15:59 |
| HenryG | mordred: psutil>=1.1.0 is in PyPI | 15:59 |
| *** beagles is now known as beagles_brb | 16:00 | |
| mordred | YES! | 16:00 |
| mordred | sweet | 16:00 |
| mordred | then please let's update to that and not add allow-external | 16:00 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 16:00 | |
| *** UtahDave has joined #openstack-infra | 16:00 | |
| dhellmann | I've been holding off on approving requirements changes because of the milestone. Is it safe to look at those again? | 16:00 |
| dhellmann | oh, and, good morning everyone | 16:01 |
| HenryG | I left my comment on that review. For some reason the interest is low. I will ping markmclain about it. | 16:01 |
| dhellmann | :-) | 16:01 |
| *** gokrokve has joined #openstack-infra | 16:01 | |
| mordred | dhellmann: https://review.openstack.org/68946 | 16:01 |
| mordred | dhellmann: consider it safe | 16:01 |
| mordred | dhellmann: to review - not that that review is specifically safe | 16:01 |
| *** changbl has joined #openstack-infra | 16:02 | |
| mordred | however, that review gets us past one of our 4 remaining not-on-pypi issues | 16:02 |
| dhellmann | mordred: understood | 16:02 |
| *** emagana has joined #openstack-infra | 16:02 | |
| dhellmann | mordred: it looks fine to me; I'll wait for the check job to +1 before I +2 | 16:02 |
| mordred | dhellmann: yah. :) | 16:02 |
| * mordred is excited potentially | 16:03 | |
| * mordred hopes new version of psutil isn't crazy different | 16:03 | |
| *** jcooley_ has joined #openstack-infra | 16:03 | |
| mordred | HenryG: have you tried neutron with newer version? | 16:03 |
| mordred | ooh... | 16:03 |
| mordred | dhellmann: just had an idea... | 16:03 |
| Alex_Gaynor | mordred: Will passing requirements tests inform us if it's safe to upgrade for all downstream projects? | 16:03 |
| mordred | Alex_Gaynor: yes | 16:03 |
| Alex_Gaynor | mordred: Great, will +2 once it passes | 16:04 |
| HenryG | mordred: yes, but neutron only uses it on one line of code | 16:04 |
| *** jcooley_ has quit IRC | 16:05 | |
| *** emagana has quit IRC | 16:05 | |
| *** jcooley_ has joined #openstack-infra | 16:05 | |
| jeblair | clarkb, mordred: i updated jenkins.o.o/~root/bootstrap/README | 16:05 |
| jeblair | so i think we now actually have full install instructions | 16:06 |
| *** coolsvap has joined #openstack-infra | 16:06 | |
| *** dolphm is now known as dolphm_food | 16:07 | |
| *** yamahata has joined #openstack-infra | 16:08 | |
| * dhellmann worries about mordred's ideas sometimes | 16:08 | |
| *** dangers_away is now known as dangers | 16:09 | |
| *** AlexF_ has quit IRC | 16:09 | |
| *** jcooley_ has quit IRC | 16:10 | |
| *** jcooley_ has joined #openstack-infra | 16:12 | |
| *** openstackgerrit has joined #openstack-infra | 16:12 | |
| jeblair | #status log restarted gerritbot because it seemed to be on the wrong side of a netsplit | 16:13 |
| *** alexpilotti_ has joined #openstack-infra | 16:14 | |
| Ajaeger | Thanks, jeblair | 16:14 |
| *** pballand has joined #openstack-infra | 16:15 | |
| *** Ajaeger has quit IRC | 16:15 | |
| *** mrmartin has quit IRC | 16:15 | |
| *** alexpilotti has quit IRC | 16:16 | |
| *** alexpilotti_ is now known as alexpilotti | 16:16 | |
| *** markmcclain has joined #openstack-infra | 16:17 | |
| *** markmcclain1 has joined #openstack-infra | 16:19 | |
| *** markmcclain1 has quit IRC | 16:19 | |
| *** markmcclain has quit IRC | 16:19 | |
| *** markmcclain has joined #openstack-infra | 16:20 | |
| *** ociuhandu has joined #openstack-infra | 16:21 | |
| *** jog0 is now known as flashgordon | 16:22 | |
| sdague | 93 patches merged in last 24hrs, zuul is starting to do it's thing again | 16:23 |
| *** jcoufal has quit IRC | 16:23 | |
| jeblair | yaaay | 16:24 |
| sdague | so we have patches in the active window which got approved after the zuul reset yesterday | 16:25 |
| sdague | which is real progress | 16:25 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 16:25 | |
| *** gokrokve has quit IRC | 16:26 | |
| *** smarcet has quit IRC | 16:31 | |
| *** thuc has joined #openstack-infra | 16:33 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 16:33 | |
| mordred | jeblair: awesome! | 16:35 |
| *** senk1 has joined #openstack-infra | 16:36 | |
| devananda | jeblair: g'morning! I'm wondering if you guys are planning on landing https://review.openstack.org/#/c/65845/ now that i2 is passed? | 16:36 |
| *** smarcet has joined #openstack-infra | 16:36 | |
| fungi | mordred: last night i put jenkins05-07 in shutdown mode and cleared out all the nodepool nodes which got assigned to them. did they still need a newer plugin installed before reenabling? | 16:36 |
| devananda | or s/planning/open to/ | 16:36 |
| *** apevec has joined #openstack-infra | 16:36 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 16:37 | |
| fungi | mordred: (nodepool had basically leaked enough ready nodes onto them which weren't picking up jobs/clearing out that we ended up with no actual nodepool nodes running any jobs) | 16:37 |
| *** AlexF_ has joined #openstack-infra | 16:38 | |
| apevec | sdague, ping - russellb said you're looking for me, I assume about nova stable http://lists.openstack.org/pipermail/openstack-stable-maint/2014-January/002050.html ? | 16:39 |
| *** DinaBelova is now known as DinaBelova_ | 16:41 | |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Expand DNS failures to catch all "errno" string output https://review.openstack.org/68961 | 16:42 |
| portante | sdague: ^^^ | 16:42 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Expand DNS failures to catch all "errno" string output https://review.openstack.org/68961 | 16:43 |
| *** gokrokve has joined #openstack-infra | 16:43 | |
| *** nicedice has joined #openstack-infra | 16:43 | |
| *** beagles_brb is now known as beagles | 16:44 | |
| *** dguitarbite has quit IRC | 16:44 | |
| flashgordon | mtreinish: https://review.openstack.org/67596 | 16:45 |
| *** SumitNaiksatam has quit IRC | 16:45 | |
| flashgordon | should help improve e-r gerritbot output | 16:45 |
| *** russellb is now known as rustlebee | 16:46 | |
| *** malini has joined #openstack-infra | 16:47 | |
| *** gokrokve has quit IRC | 16:49 | |
| *** dizquierdo has joined #openstack-infra | 16:50 | |
| *** dprince has quit IRC | 16:50 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Expand DNS failures to catch all "errno" string output https://review.openstack.org/68961 | 16:52 |
| *** yamahata has quit IRC | 16:53 | |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Recheck PIP "Cannot fetch index base URL" https://review.openstack.org/68965 | 16:56 |
| *** jcooley_ has quit IRC | 16:57 | |
| *** wendar has joined #openstack-infra | 16:58 | |
| mtreinish | flashgordon: +A | 16:59 |
| flashgordon | mtreinish: https://review.openstack.org/#/c/68745/ one more ? | 16:59 |
| flashgordon | that is sdague's patch | 16:59 |
| flashgordon | mtreinish: thanks | 16:59 |
| *** david-lyle is now known as david-lyle_ | 17:00 | |
| mtreinish | flashgordon: done | 17:01 |
| mtreinish | that one was simple enough | 17:01 |
| *** starmer has joined #openstack-infra | 17:02 | |
| flashgordon | mtreinish: yup, as more people submit patches I want to make sure we have that test | 17:02 |
| *** SumitNaiksatam has joined #openstack-infra | 17:02 | |
| *** marun has joined #openstack-infra | 17:03 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Use short build_uuids in elasticSearch queries https://review.openstack.org/67596 | 17:03 |
| mtreinish | flashgordon: well that was why I started: https://review.openstack.org/#/c/68778/ I just need to get them working | 17:03 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add tests for loading the queries https://review.openstack.org/68745 | 17:03 |
| flashgordon | mtreinish: yeah I am looking forward to your patch | 17:03 |
| *** ruhe is now known as _ruhe | 17:04 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1270182 https://review.openstack.org/68968 | 17:05 |
| *** david-lyle has joined #openstack-infra | 17:05 | |
| *** bnemec is now known as beekneemech | 17:06 | |
| *** AlexF_ has quit IRC | 17:11 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Add logging to #openstack-gate https://review.openstack.org/68969 | 17:13 |
| *** dolphm_food is now known as dolphm | 17:13 | |
| *** smarcet has quit IRC | 17:14 | |
| *** johnthetubaguy has quit IRC | 17:14 | |
| *** thomasem_ has left #openstack-infra | 17:15 | |
| *** rakhmerov has quit IRC | 17:16 | |
| *** DinaBelova_ is now known as DinaBelova | 17:16 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 17:17 | |
| *** krotscheck has joined #openstack-infra | 17:17 | |
| *** senk1 has quit IRC | 17:18 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add multi-project irc support to the bot https://review.openstack.org/67540 | 17:21 |
| *** rnirmal has joined #openstack-infra | 17:26 | |
| *** mat-lowery has joined #openstack-infra | 17:26 | |
| *** senk1 has joined #openstack-infra | 17:26 | |
| sdague | apevec: yeh, just trying to get things to depend on russellb's backport fixes so that we not have the stable changes break the gate | 17:27 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add --noirc to tox run job https://review.openstack.org/68972 | 17:27 |
| apevec | sdague, or get rustlebee's patches merge asap - but I just noticed we might want to squash them all together | 17:27 |
| *** smarcet has joined #openstack-infra | 17:28 | |
| apevec | e.g. check for https://review.openstack.org/68921 failed on the issue fixed by the next patch in that queue | 17:28 |
| apevec | https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/havana+topic:unit-test-backports,n,z | 17:28 |
| rustlebee | or ninja-merge them | 17:28 |
| rustlebee | is probably easiest | 17:29 |
| sdague | yeh, or ninja-merge | 17:29 |
| apevec | who's the ninja in the house? | 17:29 |
| sdague | I'd be ok with that as well | 17:29 |
| mat-lowery | Forgive me if this is not the correct place to ask this. For Gerrit, it doesn't appear to be formatting lists (lines that start with - or *) in comments. It is rendering the <li> items but the CSS specifies no bullet. | 17:29 |
| rustlebee | would rather do that than squash | 17:29 |
| sdague | fungi / jeblair / or clarkb | 17:29 |
| apevec | sdague, note I already approved https://review.openstack.org/68920 before I realized this | 17:29 |
| apevec | so that one is in the gate now | 17:29 |
| * apevec crosses fingers | 17:29 | |
| sdague | apevec: ok, well if we ninja merge they'll get ahead of it | 17:30 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Add a sample config file https://review.openstack.org/68620 | 17:30 |
| fungi | mat-lowery: is that consistent with the behavior of upstream gerrit as well, or unique to openstack's gerrit instance? | 17:30 |
| *** markmcclain has quit IRC | 17:31 | |
| sdague | and we're about to reset, volumes issues | 17:31 |
| sdague | it will be another 5 minutes for the job to fail | 17:31 |
| fungi | sdague: yeah, was there a change which fixes that which should be promoted ahead of the failing one? | 17:31 |
| mat-lowery | fungi, I tried to verify that but I couldn't get a couple of gerrits that I tried to render it at all in draft mode (the only mode I tried). Is there some place I can try a vanilla gerrit? | 17:32 |
| sdague | fungi: no, but when we reset let's promote - https://review.openstack.org/#/c/68673/ | 17:32 |
| *** rakhmerov has joined #openstack-infra | 17:32 | |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/config: Enable tempest/ironic gate tests https://review.openstack.org/65845 | 17:32 |
| sdague | that will make everything run a little faster | 17:32 |
| fungi | sdague: okay | 17:32 |
| devananda | fungi: rebased ^ | 17:32 |
| sdague | I'd like to wait for the fail to get a full log capture though | 17:32 |
| sdague | because we addressed some volumes issues by just turning off some volumes tests | 17:33 |
| sdague | so any fails in that area we want to have data for | 17:33 |
| fungi | mat-lowery: well, there's https://gerrit-review.googlesource.com/ | 17:33 |
| *** nrede has joined #openstack-infra | 17:34 | |
| *** nrede has quit IRC | 17:34 | |
| fungi | mat-lowery: that's official upstream gerrit's gerrit, which is basically the version for which we're in the process of upgrading openstack's gerrit | 17:34 |
| *** gokrokve has joined #openstack-infra | 17:34 | |
| *** nrede has joined #openstack-infra | 17:34 | |
| *** nrede has quit IRC | 17:34 | |
| *** jp_at_hp has quit IRC | 17:34 | |
| *** yamahata has joined #openstack-infra | 17:35 | |
| mordred | fungi: I installed the zmq plugin from 04 onto 5-7 last night - so they shoudl be good for a restart | 17:35 |
| fungi | mordred: okay, i'll go ahead and do that now--thanks1 | 17:36 |
| *** nrede has joined #openstack-infra | 17:36 | |
| *** nrede has quit IRC | 17:36 | |
| fungi | i just didn't want to turn them back on if they were going to grind us back to a halt again | 17:36 |
| mordred | ++ | 17:36 |
| *** nrede has joined #openstack-infra | 17:37 | |
| mordred | thanks for the shutdown/nodeclear - I thought I was goign to be able to get to it later -and then I was in judgement-impared state and figured I'd do more harm than help | 17:37 |
| *** nrede has quit IRC | 17:37 | |
| fungi | mordred: it's fine. thanks for working on getting these set up | 17:37 |
| mat-lowery | fungi: It shows bullets. I'll just wait for that version then. Thanks! | 17:38 |
| *** danilonrede has joined #openstack-infra | 17:38 | |
| *** danilonrede has quit IRC | 17:38 | |
| *** fbo is now known as fbo_away | 17:38 | |
| *** emagana has joined #openstack-infra | 17:38 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Add logging to #openstack-gate https://review.openstack.org/68969 | 17:38 |
| fungi | mat-lowery: cool. if you also want to test our already upgraded sandbox instance at review-dev.openstack.org, it's possible we have some css applied which is breaking it | 17:39 |
| *** DinaBelova is now known as DinaBelova_ | 17:41 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_a | 17:42 | |
| openstackgerrit | Dolph Mathews proposed a change to openstack-infra/reviewstats: update keystone-core membership https://review.openstack.org/68976 | 17:42 |
| mat-lowery | fungi: No bullets on review-dev.openstack.org. Note: The <li>'s are there (as they are on live gerrit). It's the CSS I'm pretty sure. | 17:42 |
| fungi | mat-lowery: good to know | 17:42 |
| *** DinaBelova_ is now known as DinaBelova | 17:42 | |
| *** SergeyLukjanov_a is now known as SergeyLukjanov_ | 17:43 | |
| fungi | zaro: ^ possibly one more item to look into for the upgrade? | 17:43 |
| mat-lowery | fungi: Should I log it somewhere? If so where? | 17:43 |
| *** jpich has quit IRC | 17:44 | |
| apevec | sdague, fungi - could we increase timeout for periodic-neutron-python26-havana - it passes tests just under 40min and then fails while copying results | 17:45 |
| fungi | mat-lowery: you could open a bug at https://launchpad.net/openstack-ci pointing out that something in openstack's css for review.openstack.org (and review-dev) is hiding bullets for <li> elements | 17:45 |
| mat-lowery | fungi: Got it. Will do. Thanks! | 17:45 |
| mordred | uhm | 17:45 |
| *** luqas has quit IRC | 17:46 | |
| fungi | apevec: can you give me a link to the failure? i just want to double-check it, but timeouts are easy enough for us to bump | 17:46 |
| mordred | david-lyle_: not for nothing - but I just ran "/bin/bash run_tests.sh -N --pep8" in a horizon repo and it's showing errors | 17:46 |
| sdague | apevec: yes, we should | 17:46 |
| apevec | fungi, http://logs.openstack.org/periodic-stable/periodic-neutron-python26-havana/426a2d3/console.html | 17:46 |
| *** yassine has quit IRC | 17:46 | |
| sdague | python 26 is lsow, I'd bump it to 60 minutes | 17:46 |
| mordred | are we somehow not gating horizon on flake8? | 17:46 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1223469 https://review.openstack.org/68943 | 17:47 |
| emagana | Hello infra team! | 17:47 |
| emagana | I was wondering if somebody could help with the gerrit account for PLUMgrid | 17:48 |
| emagana | I sent the email last Tuesday and I have not received any response yet | 17:48 |
| sdague | fungi: ok, so we're uploading logs now. So about another minute on the reset | 17:48 |
| mordred | david-lyle_: oh - it may be a version thing... | 17:49 |
| clarkb | morning | 17:49 |
| clarkb | is there a tldr on the state of test slave root partitions filling? | 17:49 |
| clarkb | fungi ^ | 17:49 |
| fungi | emagana: we're a bit backed up on requests for third-party testing systems. i'll have time to look through them on my flights today i hope | 17:49 |
| fungi | clarkb: yeah, i'm basically iterating on https://review.openstack.org/68782 and fixing the places where i see it break. the commit message also links to the bug | 17:50 |
| *** jhesketh has quit IRC | 17:50 | |
| emagana | !fungi: I do appreciate it! He had a deadline (yesterday) for the testing but I hope we should be fine.. :-) | 17:50 |
| openstack | emagana: Error: "fungi:" is not a valid command. | 17:50 |
| emagana | fungi: I do appreciate it! He had a deadline (yesterday) for the testing but I hope we should be fine.. :-) | 17:50 |
| *** harlowja has joined #openstack-infra | 17:51 | |
| emagana | fungi: Do you you know if there is an available script to cherry pick the patch and re-start the devstack process and run tempest? | 17:51 |
| fungi | clarkb: looks like i needs moar sudo still... devstack-vm-gate-wrap.sh: line 276: /home/jenkins/workspace/check-tempest-dsvm-full/logs/devstack-gate-cleanup-host.txt: Permission denied | 17:51 |
| *** marun has quit IRC | 17:52 | |
| emagana | I know, I am asking for a lot but having so many third party systems pushing notification to gerri already, I was wondering if we can find one available | 17:52 |
| *** derekh has quit IRC | 17:52 | |
| fungi | emagana: we use this for what you describe... https://git.openstack.org/cgit/openstack-infra/devstack-gate | 17:52 |
| emagana | we are planning to build the replicate the entire OpenStack CI but we want to start with something simple | 17:53 |
| fungi | emagana: see the readme for devstack-gate for details--it probably answers some of your questions... https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst | 17:53 |
| emagana | fungi: Let me take a look! My Jenkins is ready for sending notifications and running tests | 17:54 |
| *** pballand has quit IRC | 17:54 | |
| emagana | fungi: awesome! | 17:54 |
| emagana | it will help a lot! | 17:54 |
| fungi | mordred: jenkins05-07 are back online, have nodepool nodes and are running tests. i'll keep an eye on the node list for them to make sure they're behaving properly | 17:54 |
| *** pballand has joined #openstack-infra | 17:54 | |
| *** jhesketh_ has quit IRC | 17:55 | |
| sdague | fungi: reset time | 17:55 |
| mordred | fungi: w00t! | 17:55 |
| *** mat-lowery has left #openstack-infra | 17:55 | |
| fungi | sdague: thanks. i've popped 68673,4 to the top | 17:55 |
| sdague | fungi: thanks! | 17:55 |
| sdague | heh, new jenkins means no good time estimates :) | 17:56 |
| *** fallenpegasus has quit IRC | 17:56 | |
| mordred | heh | 17:57 |
| *** DinaBelova is now known as DinaBelova_ | 17:57 | |
| *** vkozhukalov has joined #openstack-infra | 17:58 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Storyboard API Interface and basic project management https://review.openstack.org/67582 | 17:59 |
| *** Ajaeger has joined #openstack-infra | 18:00 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Keep logs in $BASE instead of $WORKSPACE https://review.openstack.org/68782 | 18:00 |
| harlowja | hi guys, now that it appears to be calming down (slightly) is it possible to get https://review.openstack.org/#/c/65135/ pulled in? | 18:01 |
| *** hashar has joined #openstack-infra | 18:01 | |
| *** yolanda_ has quit IRC | 18:02 | |
| *** emagana has quit IRC | 18:02 | |
| *** yamahata has quit IRC | 18:03 | |
| *** yamahata has joined #openstack-infra | 18:03 | |
| jeblair | fungi, zaro: mat-lowery is gone, but the bullets are intentionally disabled in css to make the jenkins report formatting nicer. | 18:04 |
| *** AlexF_ has joined #openstack-infra | 18:04 | |
| *** dprince has joined #openstack-infra | 18:04 | |
| fungi | oh | 18:05 |
| jeblair | (in our local css) | 18:05 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: update keystone-core membership https://review.openstack.org/68976 | 18:05 |
| jeblair | fungi, sdague: i'm approving the change that will automatically trigger check runs on changes with old (>48h) check results. i'm also approving the rax server limit increase | 18:06 |
| fungi | i suppose that makes some sense. did you play around at all with the way gerrit handles indented lines in comments (treats them as unbreakable monospaced)? | 18:06 |
| sdague | jeblair: cool | 18:06 |
| jeblair | fungi: no | 18:06 |
| sdague | jeblair: that's only on comments right? | 18:06 |
| jeblair | sdague: yes | 18:06 |
| sdague | cool | 18:06 |
| sdague | works for me | 18:06 |
| *** hogepodge has joined #openstack-infra | 18:08 | |
| *** markmc has quit IRC | 18:09 | |
| *** mfink has quit IRC | 18:10 | |
| Alex_Gaynor | mordred: do you know if there's any in-progress work to speed up the tempest runs? | 18:10 |
| openstackgerrit | A change was merged to openstack-infra/config: Ensure a recent (48h) check result https://review.openstack.org/68816 | 18:10 |
| *** mfink has joined #openstack-infra | 18:11 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Experiment: set concurrency to 6 https://review.openstack.org/68982 | 18:11 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add basic unit tests for the bot https://review.openstack.org/68778 | 18:11 |
| clarkb | fungi: moar! | 18:12 |
| jeblair | Alex_Gaynor: rustlebee and sdague are the ones to ask about that. I believe rustlebee is working on fixing some nova problems so that we might consider increasing the concurrency again | 18:12 |
| openstackgerrit | A change was merged to openstack-infra/config: Increase rax IAD/DFW server limits https://review.openstack.org/68826 | 18:12 |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 18:12 | |
| *** hogepodge has quit IRC | 18:12 | |
| *** markmcclain has joined #openstack-infra | 18:12 | |
| jeblair | Alex_Gaynor: that reminded me that we also wanted to experiment with higher concurrency on rax performance nodes, so https://review.openstack.org/68982 is to see what that does | 18:13 |
| jeblair | rustlebee: ^ | 18:13 |
| Alex_Gaynor | jeblair: do you know how close to a linear speedup increased concurrency gives us? Will 2->6 bring the jobs to 20 minutes? | 18:13 |
| jeblair | Alex_Gaynor: going from 4 -> 2 increased the runtime by ~1.5 istr | 18:14 |
| *** harlowja has quit IRC | 18:14 | |
| sdague | Alex_Gaynor: for 2x concurency you probably make the tempest run go at 60% the time | 18:14 |
| rustlebee | 4 on the 4 vCPU nodes had the CPU pegged the whole time and we were seeing failures due to random things taking too long. | 18:15 |
| sdague | there is fixed cost in all those runs though | 18:15 |
| jeblair | Alex_Gaynor: (at least part of the nonlinearity is a fixed startup cost of devstack) | 18:15 |
| sdague | 7 minutes setup, 4 minutes teardown | 18:15 |
| Alex_Gaynor | Interesting, thanks. | 18:15 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1270182 https://review.openstack.org/68968 | 18:15 |
| rustlebee | i'd like to try increasing again once a certain nova patch series lands | 18:15 |
| *** starmer has quit IRC | 18:16 | |
| *** hashar has quit IRC | 18:16 | |
| sdague | Alex_Gaynor: honestly, right now, the best thing to do to help us raise concurency is help figure out where we are spending all the time | 18:16 |
| SergeyLukjanov | looks like Hyper-V CI is commenting to all changes :( https://review.openstack.org/#/c/68814/ | 18:16 |
| *** emagana has joined #openstack-infra | 18:16 | |
| *** harlowja has joined #openstack-infra | 18:16 | |
| rustlebee | alexpilotti: ping | 18:16 |
| rustlebee | alexpilotti: ^^^^^^ | 18:16 |
| sdague | with a bogus url, nice :) | 18:16 |
| *** AlexF_ has quit IRC | 18:17 | |
| fungi | at least it's not leaving votes | 18:17 |
| SergeyLukjanov | sdague, yup | 18:17 |
| rustlebee | SergeyLukjanov: at least it didn't say it failed ... | 18:17 |
| rustlebee | fungi: heh right | 18:17 |
| alexpilotti | lol | 18:17 |
| *** morganfainberg|z is now known as morganfainberg | 18:17 | |
| flashgordon | at least we passed the noop | 18:17 |
| SergeyLukjanov | yep, at least w/o voting | 18:17 |
| alexpilotti | be patient, fixing it :-) | 18:17 |
| rustlebee | failing the noop test would be a bad day | 18:17 |
| openstackgerrit | Peter Liljenberg proposed a change to openstack-infra/jenkins-job-builder: Fix filtering when templates are used https://review.openstack.org/68984 | 18:17 |
| sdague | alexpilotti: well yuo need to make it stop commenting as a top priority, even if it means pulling the plug on the machine. gerrit spam is bad. | 18:18 |
| *** esker has joined #openstack-infra | 18:18 | |
| rustlebee | it's bad mmmmkay | 18:18 |
| flashgordon | I was thinking the same thing | 18:19 |
| flashgordon | mmmkay | 18:19 |
| *** mkerrin has quit IRC | 18:19 | |
| sdague | heh | 18:19 |
| alexpilotti | so basically this is what we try to get into the message :-) http://64.119.130.115/68690/1/ | 18:19 |
| rustlebee | good thing noop only took 0 seconds | 18:19 |
| rustlebee | jeblair: a change we could probably merge now is programatically determiine how many vCPUs devstack-gate is running on, and set it to that / 2, instead of == 2 | 18:21 |
| rustlebee | change was assuming all test nodes were 4 vCPU nodes | 18:21 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Remove mock API interfaces from storyboard. https://review.openstack.org/68464 | 18:21 |
| jeblair | rustlebee: yes, or maybe vcpu-2... | 18:22 |
| jeblair | rustlebee: my test change ran entirely on hpcloud nodes. :( | 18:22 |
| rustlebee | sure | 18:22 |
| rustlebee | and boo | 18:22 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Experiment: set concurrency to 6 https://review.openstack.org/68982 | 18:23 |
| jeblair | spin again! | 18:23 |
| *** pcrews has joined #openstack-infra | 18:23 | |
| rustlebee | if only life were like that. | 18:23 |
| jeblair | (big money, big money....) | 18:23 |
| rustlebee | no whammy | 18:23 |
| *** dhellmann is now known as dhellmann_ | 18:23 | |
| primeministerp | rustlebee: ping | 18:24 |
| rustlebee | pong. | 18:25 |
| primeministerp | rustlebee: hey sorry we're finializing formatting | 18:25 |
| rustlebee | all good | 18:25 |
| primeministerp | rustlebee: it's been a ride | 18:25 |
| rustlebee | happy to see it getting closer!! | 18:25 |
| primeministerp | rustlebee: we're lighting live today | 18:25 |
| primeministerp | rustlebee: as soon as we've cleaned up the remaining gotchas | 18:25 |
| rustlebee | primeministerp: can't wait :) | 18:25 |
| *** DinaBelova_ is now known as DinaBelova | 18:26 | |
| primeministerp | rustlebee: sorry for being so quite, pretty much spent most of dec/jan in the dc | 18:27 |
| primeministerp | rustlebee: as you could imagine | 18:27 |
| *** che-arne has quit IRC | 18:27 | |
| *** dizquierdo has quit IRC | 18:27 | |
| *** BullSherd has joined #openstack-infra | 18:28 | |
| *** BullSherd has left #openstack-infra | 18:28 | |
| rustlebee | primeministerp: able to share any info on your environment? | 18:28 |
| clarkb | fungi: in your logs change, is there a reason we move things up into $WORKSPACE? I think $WORKSPACE/logs is cleaner | 18:28 |
| rustlebee | just for my curiosity | 18:28 |
| primeministerp | rustlebee: right now it's a mix of anything we could get our hands on | 18:29 |
| ttx | sdague: hmm, just re-approved https://review.openstack.org/#/c/60550/ which is nova stable/grizzly. Should I cancel that ? | 18:29 |
| primeministerp | rustlebee: i'm working on changing that | 18:29 |
| ttx | rustlebee: same question ^ | 18:29 |
| primeministerp | rustlebee: most of the infra code is here: http://github.com/openstack-hyper-v | 18:29 |
| fungi | clarkb: i was just wanting to make sure i was writing to a location where jenkins had permission, then using sudo to move those files to the right place | 18:29 |
| sdague | ttx: do not read email... :) | 18:30 |
| primeministerp | rustlebee: i'm quite pleased w/ this piece http://github.com/openstack-hyper-v/puppet-quartermaster | 18:30 |
| fungi | clarkb: part of the challenge is that we're generating files as several users but want them to end up in a common directory, and i didn't want to have to monkey with communal group write on the logs tree | 18:30 |
| ttx | sdague: read it, but accidentally approved it while I was goinf through stuck security fixes | 18:30 |
| sdague | heh | 18:30 |
| ttx | sdague: which is why I'm asking | 18:30 |
| ttx | 'oops' | 18:31 |
| clarkb | fungi: the sudo mv's have me confused when you can just have it tee to the $BASE/logs dir? | 18:31 |
| rustlebee | sdague: ttx looks ok ... it's not the unit test failures | 18:31 |
| clarkb | fungi: oh its a permissions thing | 18:31 |
| clarkb | fungi: I understand now | 18:31 |
| fungi | clarkb: yeah, it was previously trying to write there on the prior patchset and died on insufficient perms | 18:31 |
| sdague | rustlebee: well, the unit test stuff is a race | 18:31 |
| sdague | so they might show up | 18:32 |
| rustlebee | sdague: right, but i'm saying that patch didn't fail on those | 18:32 |
| rustlebee | sdague: and i only backported those to havana anyway | 18:32 |
| fungi | clarkb: we can clean it up however you like once i get it working, but right now is mostly hatchet time | 18:32 |
| sdague | the fconcurrent didnt' got to grizzly? | 18:32 |
| rustlebee | no | 18:32 |
| rustlebee | i mean, it can | 18:32 |
| sdague | I saw grizzly fail on it this morning | 18:32 |
| rustlebee | ok | 18:32 |
| sdague | just on that one | 18:32 |
| rustlebee | i'll backport that one then | 18:33 |
| sdague | so if you could bring that one back, that would be good | 18:33 |
| ttx | sdague: so that would be a YES cancel it (sorry I'm a bit thick this morning) | 18:33 |
| * rustlebee would let it ride and hope | 18:33 | |
| sdague | ttx: we're going to promote or merge these other fixes anyway | 18:33 |
| sdague | so it's probably ok | 18:33 |
| clarkb | fungi: sounds good | 18:33 |
| ttx | sdague: ok | 18:33 |
| fungi | clarkb: i'm also cheating, holding the nodes these checks run on and looking at fs perms and things as i go | 18:33 |
| clarkb | fungi: any chance you can give me the IP of one of those? I want to look at the root filesystem in general | 18:34 |
| jeblair | rustlebee: finally there's a pg-full-grizzly job running on a rax node: https://jenkins05.openstack.org/job/check-tempest-dsvm-postgres-full-grizzly/1/ | 18:34 |
| rustlebee | yay | 18:34 |
| fungi | clarkb: 15.185.223.67 | 18:34 |
| clarkb | fungi: thanks | 18:35 |
| fungi | clarkb: the current challenge i worked around in that last patchset is that by the end of the job /opt/stack/logs ends up owned by stack:stack | 18:35 |
| fungi | even though it starts out owned by jenkins when the setup functions are being run | 18:36 |
| *** crank has quit IRC | 18:36 | |
| fungi | clarkb: i also realized i can kill two birds with one stone since this is as a side effect going to solve what i originally set out to do in https://review.openstack.org/64937 | 18:37 |
| fungi | so once it's working correctly, we should also be able to merge https://review.openstack.org/64938 | 18:38 |
| clarkb | fungi: nice | 18:39 |
| flashgordon | sdague: can you look at https://review.openstack.org/#/c/67485/ that gets us very far for overall classification rate | 18:39 |
| *** max_lobur is now known as max_lobur_afk | 18:39 | |
| flashgordon | sdague: second biggest is sort out grenade logs | 18:40 |
| clarkb | lol du -h on the root of the filesystem (I did pipe it through sort -h) | 18:41 |
| clarkb | fungi: so the mysql slow log is nearly half a gig | 18:41 |
| clarkb | then there is a bunch of stuff related to apt and dpkg caches using up a bunch of disk | 18:41 |
| fungi | clarkb: yeah we can probably do more to make room there | 18:41 |
| jeblair | clarkb: we stopped collecting the slow log a while ago; perhaps it's time to disable it | 18:42 |
| clarkb | and /home/jenkins seems bloated, I assume due to file copying? | 18:42 |
| *** smarcet has quit IRC | 18:42 | |
| clarkb | jeblair: yeah or tell it to only log things that are really slow | 18:42 |
| clarkb | rather than logging everything | 18:42 |
| fungi | i think we'd need to restart mysql after setup_workspace runs to be able to move /var/log/mysql before starting devstack, but should be doable yeah? | 18:42 |
| fungi | oh, or disabe it if we don't want it, sure | 18:42 |
| jeblair | fungi: i don't think it's worth doing for something we are ignoring | 18:42 |
| clarkb | fungi: I don't think we should move where it logs, just make it smarter about what it logs | 18:42 |
| fungi | sure | 18:43 |
| clarkb | the workspace-cache is the largest offender in jenkins homedir | 18:43 |
| clarkb | can we use /opt/git for that or not because of the weird remounting that happens? | 18:44 |
| *** dkliban has quit IRC | 18:44 | |
| fungi | though we're working toward emptying workspace-cache anyway right? | 18:44 |
| *** dkliban has joined #openstack-infra | 18:44 | |
| fungi | we could at least delete workspace-cache once the job reaches a point where it's not used any further | 18:44 |
| jeblair | fungi, clarkb: yes; direction should be to use opt/git and remove workspace cache | 18:45 |
| fungi | similarly, an option to devstack to run apt-get clean after package installs might help free some space | 18:45 |
| jeblair | that transition is just stalled in progress | 18:45 |
| clarkb | fungi: that makes sense for single use slaves | 18:47 |
| *** markwash has joined #openstack-infra | 18:48 | |
| fungi | clarkb: right, i would expect that to be one of those rare "off by default" options which we'd enable for our single-use slave situation | 18:50 |
| *** nati_ueno has joined #openstack-infra | 18:50 | |
| fungi | and we'd probably want it to do the rpm/yum equivalent of that too | 18:50 |
| fungi | though that also assumes that other jobs using a base devstack install but then adding other packages in a separate hook function would end up hitting the internet, so maybe not | 18:52 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1272468 https://review.openstack.org/68987 | 18:52 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-dev/hacking: Add a tool that can fix the import order https://review.openstack.org/68988 | 18:52 |
| fungi | however, i doubt we're successfully precaching those in the nodepool images anyway, so maybe irrelevant | 18:52 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-dev/hacking: Add a tool that can fix the import order https://review.openstack.org/68988 | 18:53 |
| clarkb | fungi: good point | 18:53 |
| clarkb | ideally we wouldn't need to worry about package cahces so much, they are hand | 18:53 |
| clarkb | y | 18:53 |
| jeblair | let's do lower hanging fruit first | 18:54 |
| fungi | yep. i'll throw up a change to turn off slow log | 18:54 |
| *** ryanpetrello has quit IRC | 18:54 | |
| jeblair | i'll do another step in using /opt/git | 18:54 |
| *** matel has quit IRC | 18:54 | |
| *** ryanpetrello has joined #openstack-infra | 18:55 | |
| clarkb | jeblair: note the tripleo prepare node scripts use workspace-cache too, will need to move them as well to /opt/git | 18:55 |
| clarkb | how does /opt/git work if we remount a thing over /opt ? | 18:55 |
| clarkb | mordred: is the horizon thing a tox vs not tox problem? | 18:55 |
| *** mgagne1 has joined #openstack-infra | 18:56 | |
| *** mgagne has quit IRC | 18:56 | |
| jeblair | clarkb: we rsync before remount | 18:56 |
| jeblair | clarkb: (mkfs, mount on /mnt, rsync opt to mnt, remount on /opt) | 18:56 |
| mordred | clarkb: yes | 18:56 |
| jeblair | clarkb: we should probably mv instead of rsync though to free sapec | 18:57 |
| jeblair | space | 18:57 |
| mordred | clarkb: so I think it was not a "there is a missing gate" which was my confusion earlier | 18:57 |
| *** hogepodge has joined #openstack-infra | 18:57 | |
| clarkb | jeblair: ++ | 18:57 |
| clarkb | mordred: right, its a special snowflake problem | 18:57 |
| mordred | clarkb: no - I mean, the gate and tox both do the right thing | 18:57 |
| clarkb | oh | 18:58 |
| mordred | I was running it outside of tox locally to test - so it was my bad | 18:58 |
| clarkb | then why is it not "clean" | 18:58 |
| *** jasondotstar has quit IRC | 18:59 | |
| mordred | clarkb: newer versions of pep8/pyflakes | 18:59 |
| mordred | I believe | 18:59 |
| mordred | I stopped looking in to it | 18:59 |
| mgagne1 | Who should I contact if a 3rd party CI tested a change for our puppet stuff (which I feel they shouldn't be testing) and left an unreachable link for the result? https://review.openstack.org/#/c/68977/1 | 18:59 |
| mordred | because it wasn't a bug in the gate as best I coudl tell | 18:59 |
| *** mgagne1 is now known as mgagne | 18:59 | |
| fungi | mgagne: alexpilotti and primeministerp are shutting that off while they fix it | 19:00 |
| mgagne | fungi: ok thanks | 19:00 |
| clarkb | fungi: also does that account not have an email address? I think we should require email addresses | 19:00 |
| fungi | clarkb: i agree we should. that was an early entrant from before we started discussing making some of the optional info required | 19:01 |
| clarkb | gotcha | 19:01 |
| clarkb | alexpilotti: primeministerp: can you provide a contact email address for hyperv ci? | 19:01 |
| fungi | clarkb: i think we should go ahead and merge the doc update on it if there is no further dissent, then notify people that we're requiring additional details/functionality, then remove voting or access for accounts which are not complying | 19:02 |
| clarkb | fungi: 68782 claims success, the change looked fine to me | 19:02 |
| *** gokrokve_ has joined #openstack-infra | 19:02 | |
| clarkb | fungi: ++ | 19:02 |
| mgagne | clarkb: can you take a look at https://review.openstack.org/#/c/65406/ ? It removes a job from the puppet modules pipeline | 19:02 |
| clarkb | fungi: but first 68782, I suppose we dig into workspace cleanup logs and make sure it is happy then approve? | 19:02 |
| fungi | clarkb: yeah, i want to go over all the result links and make sure they have the right contents (since some jobs run different functions which handle different artifacts) | 19:03 |
| mgagne | clarkb: I feel this could be of interest as it frees jenkins resources | 19:04 |
| clarkb | fungi: http://logs.openstack.org/82/68782/4/check/check-dg-tempest-dsvm-full/a141492/logs/devstack-gate-cleanup-host.txt any idea why there are a bunch of files iwth numbers in them getting deleted? | 19:04 |
| clarkb | mgagne: ok | 19:04 |
| alexpilotti | clarkb: for the moment ociuhandu@cloudbasesolutions.com | 19:04 |
| *** beagles is now known as seagulls | 19:04 | |
| *** morganfainberg is now known as needscoffee | 19:04 | |
| alexpilotti | clarkb: we're also in the process of setting up the details for the support phase, so we'll have a dedicated email | 19:05 |
| *** gokrokve has quit IRC | 19:05 | |
| clarkb | perfect | 19:05 |
| fungi | clarkb: you mean the "sudo rm /opt/stack/logs/screen-c-api.2014-01-24-180947.txt /opt/stack/logs/screen-c-bak.2014-01-24-180947.txt [...]"? | 19:06 |
| *** emagana has quit IRC | 19:06 | |
| alexpilotti | about the noop *ahem* featured comments, it got sent to 13 patches | 19:06 |
| clarkb | fungi: yup | 19:06 |
| clarkb | fungi: we aren't log rotating those files are we? | 19:06 |
| *** praneshp has joined #openstack-infra | 19:06 | |
| fungi | clarkb: https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n408 | 19:07 |
| fungi | comment for that command says "remove duplicate logs" | 19:07 |
| *** jerryz has joined #openstack-infra | 19:07 | |
| *** dstanek has quit IRC | 19:07 | |
| *** dstanek has joined #openstack-infra | 19:07 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Use /opt/git instead of workspace cache https://review.openstack.org/68990 | 19:07 |
| *** gokrokve has joined #openstack-infra | 19:08 | |
| jeblair | clarkb, fungi: devstack creates symlinks with timestamps | 19:08 |
| jeblair | or the other way around | 19:08 |
| jeblair | either way, the jenkins publisher would get 2 copies | 19:08 |
| clarkb | jeblair: thanks | 19:08 |
| *** praneshp has quit IRC | 19:11 | |
| *** jcooley_ has joined #openstack-infra | 19:11 | |
| *** gokrokve_ has quit IRC | 19:12 | |
| fungi | clarkb: i think we're missing logs from grenade still... http://logs.openstack.org/82/68782/4/check/check-grenade-dsvm/971381c/logs/ | 19:12 |
| fungi | digging now to see if i can tell why | 19:12 |
| clarkb | fungi: oh yep, there should be a new/ and old/ dir | 19:12 |
| fungi | i may need to recheck and hold the host grenade jobs end up on | 19:14 |
| *** smarcet has joined #openstack-infra | 19:14 | |
| *** dcramer_ has quit IRC | 19:15 | |
| fungi | i don't see an error right off to explain it, but i'll go back through the script to see if i missed something obvious first | 19:15 |
| *** david-lyle has quit IRC | 19:15 | |
| *** markmcclain1 has joined #openstack-infra | 19:16 | |
| *** markmcclain1 has quit IRC | 19:16 | |
| *** marun has joined #openstack-infra | 19:16 | |
| *** markmcclain1 has joined #openstack-infra | 19:16 | |
| *** yolanda_ has joined #openstack-infra | 19:18 | |
| *** markmcclain has quit IRC | 19:18 | |
| *** emagana has joined #openstack-infra | 19:19 | |
| *** dims has quit IRC | 19:21 | |
| *** dhellmann_ is now known as dhellmann | 19:24 | |
| *** praneshp has joined #openstack-infra | 19:26 | |
| *** Vijay has joined #openstack-infra | 19:26 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Keep logs in $BASE instead of $WORKSPACE https://review.openstack.org/68782 | 19:26 |
| fungi | well, i don't immediately see the grenade log collection problem, but spotted a couple others i missed solving | 19:26 |
| * clarkb says semi on the record that testing mysql, maridb, postgres, and mongodb is insane | 19:26 | |
| fungi | i'll hold the host for te grenade run on this patchset | 19:26 |
| clarkb | fungi: ok, I will take a look at grenade too then | 19:26 |
| *** mfink has quit IRC | 19:27 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Genericize javascript release artifact creation https://review.openstack.org/67731 | 19:27 |
| *** mfink has joined #openstack-infra | 19:27 | |
| fungi | clarkb: jeblair: looking at the mysql slow log, devstack doesn't have a knob to disable that. do you think i should add one or just remove those lines (or comment them out as documentation for how to hack it back in easily)? | 19:27 |
| clarkb | fungi: we never used the slow logs for anything aiui, it was too far forward thinking | 19:28 |
| clarkb | fungi: so I think we just remove it | 19:28 |
| *** marun has quit IRC | 19:29 | |
| jeblair | fungi: i think some devs want the slow log, so i think a knob so that it can be used by devs but disabled by devstack-gate would be good | 19:30 |
| *** jcooley_ has quit IRC | 19:30 | |
| *** marun has joined #openstack-infra | 19:30 | |
| clarkb | fungi: cp: target `/opt/stack/logs/old/' is not a directory <- happens for new as well. I believe that is why we have not new and old logs | 19:30 |
| clarkb | fungi: we need a mkdir before the cp | 19:30 |
| *** nati_ueno has quit IRC | 19:32 | |
| fungi | oho, good catch | 19:33 |
| *** VijayT has joined #openstack-infra | 19:34 | |
| *** nati_ueno has joined #openstack-infra | 19:35 | |
| *** ociuhandu_ has joined #openstack-infra | 19:36 | |
| *** jasondotstar has joined #openstack-infra | 19:36 | |
| clarkb | fungi: did you catch that we have been asked to not rename that stackfprge project this weekend? | 19:37 |
| fungi | okay | 19:37 |
| fungi | wfm | 19:37 |
| clarkb | apparently it will break a bunch of stuff and the person requesting it won't be around to fix it | 19:37 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Keep logs in $BASE instead of $WORKSPACE https://review.openstack.org/68782 | 19:38 |
| fungi | let's see if ^ that is any better | 19:38 |
| *** crank has joined #openstack-infra | 19:38 | |
| *** gokrokve has quit IRC | 19:39 | |
| clarkb | fungi: you need to mkdir -p the grenade logs dir too | 19:39 |
| fungi | also trying not to get carsick in the back of the airport shuttle, so typing slowly (twisty mountain roads) | 19:39 |
| fungi | ahh, yep--fixing. thx | 19:39 |
| *** markmcclain1 has quit IRC | 19:39 | |
| *** ociuhandu_ has quit IRC | 19:39 | |
| *** pballand has quit IRC | 19:42 | |
| *** ociuhandu_ has joined #openstack-infra | 19:42 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use ZUUL_URL in ggp https://review.openstack.org/68997 | 19:47 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Stop supplying the zuul url to ggp https://review.openstack.org/68998 | 19:47 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Cleanup unused param in ggp https://review.openstack.org/68999 | 19:47 |
| *** ociuhandu_ has quit IRC | 19:47 | |
| *** ociuhandu_ has joined #openstack-infra | 19:47 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1272468 https://review.openstack.org/68987 | 19:49 |
| *** ociuhandu_ has quit IRC | 19:50 | |
| jeblair | rustlebee: https://review.openstack.org/#/c/66950/ :( | 19:53 |
| *** harlowja is now known as harlowja_away | 19:53 | |
| *** denis_makogon_ has joined #openstack-infra | 19:54 | |
| fungi | wow. nothing like 10 miles through a granite canyon when it comes to wireless dead zones | 19:54 |
| *** ociuhandu_ has joined #openstack-infra | 19:54 | |
| rustlebee | jeblair: wat | 19:54 |
| rustlebee | jeblair: how did i even pull that off | 19:55 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Keep logs in $BASE instead of $WORKSPACE https://review.openstack.org/68782 | 19:55 |
| *** marun has quit IRC | 19:57 | |
| jerryz | fungi: are we able to and allowed to configure gate pipeline outside of CI-infra's zuul and use our own? | 19:57 |
| *** ociuhandu_ has quit IRC | 19:57 | |
| mordred | jerryz: nope | 19:59 |
| mordred | there needs to be one owner of gating | 19:59 |
| *** jcooley_ has joined #openstack-infra | 19:59 | |
| mordred | otherwise we get robots fighting, and that leads to the destruction of all life as we know it | 19:59 |
| *** ociuhandu_ has joined #openstack-infra | 20:00 | |
| dstufft | mordred: https://lh3.ggpht.com/_dfClHjAuXrE/TDP72YVReCI/AAAAAAAAB3k/UNrwOwSNY5A/s1600/Classic+BattleBots.jpg ? | 20:00 |
| mordred | dstufft: yup. exactly like that | 20:01 |
| *** needscoffee is now known as needscoffee|z | 20:01 | |
| jerryz | mordred: there is a heavy test and we want it to be in the gate but not check since it takes time | 20:02 |
| jerryz | mordred: i can add it in the third party test job but it seems only check pipeline is used. | 20:03 |
| mordred | jerryz: we've been finding that running thigns in gate without running and passing them in check winds up killing folks, oddly enough | 20:04 |
| jerryz | mordred: to be specific, our project is in stackforge. and other than python jobs, we want to add some integration test in third party jobs. | 20:04 |
| mordred | hrm | 20:04 |
| *** smurugesan has joined #openstack-infra | 20:05 | |
| fungi | grrr... my flights today through chicago got cancelled and the airline's only other flight which could get me home today is already sold out | 20:06 |
| clarkb | fungi time for a party | 20:06 |
| *** rfolco has quit IRC | 20:07 | |
| jerryz | mordred: that's why we want a gate pipeline out of openstack's zuul. | 20:07 |
| fungi | or time to complain to someone at southwest's ticket counter until they agree to transfer my fare to another airline with open seats | 20:07 |
| *** emagana has quit IRC | 20:08 | |
| fungi | don't have high hopes for that though | 20:08 |
| *** vkozhukalov has quit IRC | 20:08 | |
| *** ociuhandu_ has quit IRC | 20:08 | |
| clarkb | :( | 20:08 |
| anteaya | fungi: where are you stuck? | 20:09 |
| anteaya | in utah still? | 20:09 |
| clarkb | re other gates taanstaafl. the zuul gate is part of stackforge | 20:09 |
| fungi | anteaya: yup | 20:09 |
| anteaya | does southwest fly to texas? | 20:09 |
| fungi | it does | 20:10 |
| anteaya | can you get home that way? | 20:10 |
| anteaya | or is that part of the complaining at teh ticket counter? | 20:10 |
| fungi | the trick is getting a flight to an airport which has a later flight departing for where i need to get to. the only other one southwest has is already sold out | 20:10 |
| anteaya | ah | 20:10 |
| fungi | anyway, i'll figure something out | 20:11 |
| anteaya | go you | 20:11 |
| *** dkliban has quit IRC | 20:11 | |
| jerryz | mordred: how could we add the integration test for a stackforge project? | 20:11 |
| *** emagana has joined #openstack-infra | 20:12 | |
| *** prad has quit IRC | 20:12 | |
| *** dkliban has joined #openstack-infra | 20:12 | |
| *** rwsu has quit IRC | 20:14 | |
| *** dstanek has quit IRC | 20:14 | |
| *** dstanek has joined #openstack-infra | 20:15 | |
| alexpilotti | ok, we're on the right track: https://review.openstack.org/#/c/62896/ | 20:17 |
| anteaya | fungi: if you can get to Baltimore before 7:10pm | 20:17 |
| alexpilotti | we still have an issue with zuul reporting build succeeded, although the job failed. We're hacking on that | 20:18 |
| anteaya | fungi: you are trying to get to charlotte? | 20:18 |
| *** Ajaeger has quit IRC | 20:19 | |
| *** dstufft has quit IRC | 20:19 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Add NPM mirror https://review.openstack.org/68818 | 20:19 |
| anteaya | the only way I can see getting to charlotte not through chicago is by going through baltimore, every thing else stops in chicago | 20:20 |
| *** rwsu has joined #openstack-infra | 20:20 | |
| *** dstufft has joined #openstack-infra | 20:20 | |
| *** yolanda_ has quit IRC | 20:20 | |
| *** melwitt has joined #openstack-infra | 20:22 | |
| *** ok_delta has joined #openstack-infra | 20:23 | |
| *** praneshp has quit IRC | 20:24 | |
| *** dstufft_ has joined #openstack-infra | 20:25 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Add NPM mirror https://review.openstack.org/68818 | 20:27 |
| *** nati_ueno has quit IRC | 20:28 | |
| *** dstufft has quit IRC | 20:28 | |
| *** dstufft_ is now known as dstufft | 20:30 | |
| sdague | regrouping after lunch, rustlebee, what are the review numbers for all the stable/ fixes? | 20:32 |
| *** krotscheck has quit IRC | 20:32 | |
| *** lcestari has quit IRC | 20:33 | |
| rustlebee | https://review.openstack.org/#/c/68986/ https://review.openstack.org/#/c/68922/ https://review.openstack.org/#/c/68921/ https://review.openstack.org/#/c/68920 | 20:34 |
| *** Vijay has quit IRC | 20:34 | |
| *** smurugesan has quit IRC | 20:37 | |
| *** jasondotstar has quit IRC | 20:37 | |
| *** colettecello has joined #openstack-infra | 20:39 | |
| *** colettecello has quit IRC | 20:39 | |
| *** krotscheck has joined #openstack-infra | 20:40 | |
| *** jasondotstar has joined #openstack-infra | 20:40 | |
| *** praneshp has joined #openstack-infra | 20:41 | |
| *** malini is now known as malini_afk | 20:41 | |
| *** DinaBelova is now known as DinaBelova_ | 20:42 | |
| *** CaptTofu has quit IRC | 20:43 | |
| *** mgagne has quit IRC | 20:43 | |
| openstackgerrit | Jesse Noller proposed a change to openstack-infra/config: Add openstacksdk, see https://wiki.openstack.org/wiki/UnifiedSDK https://review.openstack.org/69008 | 20:44 |
| *** needscoffee|z is now known as morganfainberg | 20:45 | |
| *** jcooley_ has quit IRC | 20:46 | |
| *** jcooley_ has joined #openstack-infra | 20:48 | |
| *** SumitNaiksatam has quit IRC | 20:50 | |
| *** yolanda_ has joined #openstack-infra | 20:50 | |
| *** SumitNaiksatam has joined #openstack-infra | 20:51 | |
| sdague | fungi: you in a possition to do ninja merges? | 20:51 |
| sdague | or clarkb ? | 20:52 |
| openstackgerrit | Jesse Noller proposed a change to openstack-infra/config: Add openstacksdk, see https://wiki.openstack.org/wiki/UnifiedSDK https://review.openstack.org/69008 | 20:53 |
| *** _ruhe is now known as ruhe | 20:53 | |
| *** jcooley_ has quit IRC | 20:54 | |
| *** senk1 has quit IRC | 20:54 | |
| *** CaptTofu has joined #openstack-infra | 20:54 | |
| *** emagana_ has joined #openstack-infra | 20:57 | |
| *** jcooley_ has joined #openstack-infra | 20:57 | |
| *** emagana has quit IRC | 20:57 | |
| clarkb | sdague: in a bit | 20:58 |
| sdague | clarkb: cool, ping me when you can | 20:58 |
| *** dcramer_ has joined #openstack-infra | 20:58 | |
| openstackgerrit | Jesse Noller proposed a change to openstack-infra/config: Add openstacksdk, see https://wiki.openstack.org/wiki/UnifiedSDK https://review.openstack.org/69008 | 20:59 |
| *** smarcet has left #openstack-infra | 20:59 | |
| *** yolanda_ has quit IRC | 21:00 | |
| *** jcooley_ has quit IRC | 21:00 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 21:00 | |
| *** mfink has quit IRC | 21:00 | |
| *** SumitNaiksatam has quit IRC | 21:00 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 21:00 | |
| *** pcrews has quit IRC | 21:02 | |
| ociuhandu | anyone could give us a hint why jenkins says "Finished: FAILURE" in console and on status "Reported result: SUCCESS" on the same job? | 21:02 |
| *** pcrews has joined #openstack-infra | 21:03 | |
| *** seagulls has quit IRC | 21:03 | |
| *** marun has joined #openstack-infra | 21:05 | |
| *** yamahata has quit IRC | 21:06 | |
| clarkb | ociuhandu: can you link to the log? | 21:07 |
| clarkb | sdague: ok | 21:07 |
| clarkb | sdague: are we ninja ninjaing or promoting? | 21:07 |
| *** jnoller has joined #openstack-infra | 21:07 | |
| *** harlowja_away is now known as harlowja | 21:07 | |
| * clarkb hopes for promoting because it feels dirty to actually ninja something | 21:07 | |
| ociuhandu | clarkb: https://review.openstack.org/#/c/66951 is an issue | 21:08 |
| clarkb | ociuhandu: can you link to the specific test log? | 21:08 |
| *** senk1 has joined #openstack-infra | 21:08 | |
| ociuhandu | clarkb: the jenkins result log? | 21:09 |
| clarkb | ociuhandu: yes the one that said finished failure | 21:09 |
| clarkb | but reported a result of success | 21:09 |
| ociuhandu | clarkb: http://64.119.130.115/66951/10/ | 21:09 |
| openstackgerrit | Jesse Noller proposed a change to openstack-infra/config: Add openstacksdk, see https://wiki.openstack.org/wiki/UnifiedSDK https://review.openstack.org/69008 | 21:10 |
| clarkb | ociuhandu: that is a third part test, looks like hyper V. YOu will want to ping alexpilotti or primeministerp | 21:10 |
| sdague | clarkb: I think ninja merging | 21:10 |
| sdague | for the following reason | 21:10 |
| *** krotscheck has quit IRC | 21:10 | |
| sdague | they are backport unit test fixes | 21:10 |
| alexpilotti | clarkb: ociuhandu is sitting next to me :-) | 21:11 |
| clarkb | alexpilotti: ok, I am confused then | 21:11 |
| sdague | and when they ran in check patch #2 failed needing the fix for patch #3 | 21:11 |
| sdague | but the races can go in any order | 21:11 |
| ociuhandu | clarkb: and this is the zuul and jenkins output: http://paste.openstack.org/show/61856/ | 21:11 |
| sdague | clarkb: https://review.openstack.org/#/c/68986/ https://review.openstack.org/#/c/68922/ https://review.openstack.org/#/c/68921/ https://review.openstack.org/#/c/68920 | 21:11 |
| *** praneshp has quit IRC | 21:11 | |
| clarkb | sdague: so we solve that in a different but also dirty way, squash them | 21:12 |
| alexpilotti | clarkb: ociuhandu and gsamfira are both working on teh Hyper-V CI from the Cloudbase side | 21:12 |
| *** jcooley_ has joined #openstack-infra | 21:12 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-dev/hacking: Add a tool that can fix the import order https://review.openstack.org/68988 | 21:12 |
| *** jcooley_ has quit IRC | 21:12 | |
| sdague | clarkb: sure, we could do that as well | 21:12 |
| clarkb | alexpilotti: ociuhandu that job appears to be non voting | 21:13 |
| clarkb | if a job is non voting it won't flip the success/fail bit and it defaults to success | 21:13 |
| *** zaro has quit IRC | 21:13 | |
| sdague | clarkb: ok, how about I squash the havana ones then, and we promote that | 21:14 |
| clarkb | sdague: ++ | 21:14 |
| alexpilotti | clarkb: tx, so, since the Hyper-V CI is non voting, it implies that the stuatus is always success? | 21:14 |
| clarkb | alexpilotti: yes | 21:14 |
| alexpilotti | clarkb: tx and forgive us for the zuul n00bness ;-) | 21:14 |
| ociuhandu | clarkb: thanks for the explanation | 21:15 |
| clarkb | alexpilotti: ociuhandu: no problem. I was initially confused by the question :) | 21:15 |
| *** praneshp has joined #openstack-infra | 21:15 | |
| *** praneshp has quit IRC | 21:16 | |
| alexpilotti | clarkb: looks like we are done then. The Hyper-V CI is ready :-) | 21:16 |
| *** praneshp has joined #openstack-infra | 21:17 | |
| alexpilotti | clarkb: we still have a bug with non master branches, as we don't properly checkout the right branch in case of Grizzly, Havana | 21:17 |
| ociuhandu | clarkb: please ping me or mail me at ociuhandu@cloudbasesolutions.com for any issue you identify with the Hyper-V CI | 21:17 |
| alexpilotti | clarkb: we're gonna fix that one over the week end | 21:18 |
| gsamfira | the branch selection bit is being worked on now | 21:18 |
| clarkb | alexpilotti: ociuhandu: cool, have you seen what devstack-gate and gerrit-git-prep do to setup the git trees? | 21:18 |
| clarkb | would be a good place to start probably | 21:18 |
| gsamfira | clarkb: yes, we use gerrit-git-prep for nova, but we did not use it for the rest of the projects. A fist for that should be commited soon | 21:20 |
| gsamfira | first=fix | 21:20 |
| openstackgerrit | Jesse Noller proposed a change to openstack-infra/config: Add openstacksdk, see https://wiki.openstack.org/wiki/UnifiedSDK https://review.openstack.org/69008 | 21:20 |
| jnoller | sdague: I fixed the name | 21:21 |
| *** esker has quit IRC | 21:21 | |
| sdague | jnoller: cool, thanks | 21:21 |
| *** dprince has quit IRC | 21:22 | |
| *** VijayT has quit IRC | 21:22 | |
| *** ociuhandu_ has joined #openstack-infra | 21:22 | |
| *** VijayT has joined #openstack-infra | 21:22 | |
| lifeless | clarkb: fungi: any chance - https://review.openstack.org/#/c/68866/ ? | 21:23 |
| *** ociuhandu_ has quit IRC | 21:26 | |
| clarkb | lifeless: done | 21:26 |
| clarkb | jeblair: fungi's change to log procssing in devstack-gate looks good now | 21:26 |
| clarkb | jeblair: any chance I can get you to review that? | 21:27 |
| clarkb | mordred: ^ | 21:27 |
| *** virmitio has joined #openstack-infra | 21:27 | |
| *** mrodden1 is now known as mrodden | 21:27 | |
| lifeless | clarkb: cool. | 21:28 |
| *** thuc has quit IRC | 21:29 | |
| mordred | clarkb: link? | 21:30 |
| *** thuc has joined #openstack-infra | 21:30 | |
| clarkb | mordred: https://review.openstack.org/#/c/68782/ | 21:30 |
| mordred | lifeless: +A | 21:30 |
| jeblair | clarkb: ack | 21:30 |
| ttx | clarkb: I seem to be quite unsuccessful at pushing things back to the gate queue by APRVing them. Isn't that what we're supposed to do now ? (instead of reverifying) ? | 21:31 |
| clarkb | jeblair: https://review.openstack.org/#/c/68990/1 is unhappy. I can poke at that in a bit | 21:31 |
| ttx | example: https://review.openstack.org/#/c/67186/ | 21:31 |
| ttx | or https://review.openstack.org/#/c/61714/ | 21:31 |
| *** etoews1 has joined #openstack-infra | 21:31 | |
| clarkb | ttx: yes and interesting | 21:31 |
| ttx | I don't see any of them i nthe gate queue, although the veent queue seems to be empty | 21:32 |
| clarkb | ttx: ya, let me look at zuul logs | 21:32 |
| annegentle | clarkb: jeblair: anything we can do about five builds behind on the operations-guide builds? | 21:32 |
| ttx | probably me not dong it right | 21:32 |
| *** changbl has quit IRC | 21:33 | |
| annegentle | er, six builds? https://review.openstack.org/#/q/status:open+project:openstack/operations-guide,n,z | 21:33 |
| jeblair | perhaps we are enqueing nothing in the gate | 21:33 |
| annegentle | clarkb: jeblair: May not have run for 7 hours? https://jenkins01.openstack.org/view/Openstack-manuals/ | 21:33 |
| *** ruhe is now known as _ruhe | 21:33 | |
| jeblair | annegentle: jenkins is not a reliable place to look for that | 21:34 |
| *** mfink has joined #openstack-infra | 21:34 | |
| clarkb | jeblair: ttx http://paste.openstack.org/show/61859/ | 21:34 |
| *** thuc has quit IRC | 21:34 | |
| jeblair | clarkb: i see it | 21:34 |
| clarkb | jeblair: that is related to your layout changes? | 21:34 |
| annegentle | jeblair: ah ok, how can I further troubleshoot? | 21:35 |
| jeblair | so, neat, we can have approvals without usernames | 21:35 |
| jeblair | annegentle: no need | 21:35 |
| lifeless | clarkb: so I did a 'check experimental' but no results and not in zuul status | 21:36 |
| *** thuc has joined #openstack-infra | 21:37 | |
| clarkb | lifeless: can you link to the change? | 21:37 |
| clarkb | I see no occurrences of check experimental in the zuul log but it may not be that verbose | 21:37 |
| lifeless | clarkb: https://review.openstack.org/#/c/68841/ | 21:37 |
| *** zaro0508 has joined #openstack-infra | 21:38 | |
| clarkb | lifeless: you have hit the same problem as ttx apparently | 21:39 |
| clarkb | jeblair: are you working on that or should I look into fixing it? | 21:39 |
| jeblair | #status alert Zuul is ignoring some enqueue events; fix in progress | 21:39 |
| openstackstatus | NOTICE: Zuul is ignoring some enqueue events; fix in progress | 21:39 |
| *** ChanServ changes topic to "Zuul is ignoring some enqueue events; fix in progress" | 21:39 | |
| *** ociuhandu_ has joined #openstack-infra | 21:39 | |
| *** zaro0508 is now known as zaro | 21:39 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 21:39 | |
| jeblair | clarkb: can you find a change with an approval with no username? | 21:40 |
| clarkb | jeblair: ttx's change | 21:40 |
| clarkb | 61714 | 21:40 |
| jeblair | clarkb: all approvals seem to have a username | 21:41 |
| *** jasondotstar has quit IRC | 21:41 | |
| clarkb | jeblair: http://paste.openstack.org/show/61859/ that one didn't | 21:42 |
| clarkb | unless I am misparsing what you are asking | 21:42 |
| annegentle | jeblair: is that notice to blame for our behind builds? | 21:42 |
| jeblair | annegentle: yes, and the current conversation | 21:42 |
| annegentle | jeblair: got it, thanks for 'splainin' :) | 21:42 |
| jeblair | ssh review gerrit query 61714 --all-approvals --current-patch-set | 21:42 |
| jeblair | clarkb: i don't see the problem | 21:42 |
| *** mfink has quit IRC | 21:43 | |
| jeblair | clarkb: oh, i think i see it | 21:43 |
| clarkb | jeblair: approval['by']['username'] perhaps? | 21:43 |
| jeblair | yep | 21:43 |
| jeblair | miscopied the test data | 21:44 |
| jeblair | patching now | 21:44 |
| clarkb | ok | 21:44 |
| *** ociuhandu_ has quit IRC | 21:47 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Gerrit approval records have a 'by' field https://review.openstack.org/69025 | 21:49 |
| jeblair | clarkb: ^ | 21:50 |
| *** apevec has quit IRC | 21:50 | |
| jeblair | oh good grief | 21:50 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Gerrit approval records have a 'by' field https://review.openstack.org/69025 | 21:50 |
| clarkb | jeblair: +2 | 21:51 |
| clarkb | sdague: rustlebee: so we will probably do a zuul restart to apply that fix | 21:52 |
| clarkb | sdague: rustlebee: can you supply the change numbers for stable havana to jeblair and we can promote in the restart | 21:52 |
| sdague | clarkb: yeh | 21:52 |
| rustlebee | thanks guys | 21:52 |
| sdague | https://review.openstack.org/#/c/68922/ https://review.openstack.org/#/c/68921/ https://review.openstack.org/#/c/68920 | 21:53 |
| jeblair | clarkb: i believe i need to ninja merge that, yeah? | 21:53 |
| sdague | 68920, 68921, 68922 | 21:53 |
| sdague | jeblair: any way you want to handle them | 21:53 |
| jeblair | sdague: yeah, i'll promote them, i meant ninja merge the zuul change because nothing is going in the gate queue right now | 21:54 |
| sdague | the only concern with running them was in check the middle one failed on the bug fixed in the last one. Even though the last one was the less common bug | 21:54 |
| sdague | oh, ok | 21:54 |
| jeblair | zuul has passed everything except the 26 job, so merging it now | 21:54 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Gerrit approval records have a 'by' field https://review.openstack.org/69025 | 21:55 |
| jeblair | #status alert Zuul is restarting to pick up a bug fix | 21:55 |
| openstackstatus | NOTICE: Zuul is restarting to pick up a bug fix | 21:55 |
| *** ChanServ changes topic to "Zuul is restarting to pick up a bug fix" | 21:55 | |
| sdague | jeblair: so we have 9 changes that might merge in the next 12 minutes | 21:56 |
| jeblair | sdague: i'll work slowly | 21:56 |
| sdague | ok | 21:56 |
| sdague | then there is a 30 minute jump after that | 21:56 |
| sdague | so if we could see how many of the top of the queue we could land, that would be cool | 21:56 |
| sdague | gate < 70 now | 21:57 |
| jeblair | sdague: well, that's partly because of this bug | 21:57 |
| flashgordon | github reports we have 99 patches in o/o today! | 21:57 |
| *** sarob has joined #openstack-infra | 21:58 | |
| sdague | jeblair: oh, what did the bug do? | 21:58 |
| flashgordon | with 2 hours to go | 21:58 |
| jeblair | sdague: prevent anything from going into check, gate, or experimental except for new patchsets going into check | 21:58 |
| sdague | heh | 21:58 |
| jeblair | yeah, that's all. :) | 21:58 |
| sdague | blah, who cares about new approves | 21:59 |
| *** changbl has joined #openstack-infra | 21:59 | |
| jeblair | sdague: for about the last 3-4 hours | 21:59 |
| sdague | well top of gate passed | 21:59 |
| flashgordon | our record is 139 patches in one day | 21:59 |
| sdague | 2 more minutes to maybe merge another 6 | 22:00 |
| flashgordon | sdague: woot, its good to see things moving again | 22:00 |
| krtaylor | 139, wow, has anyone figured an average? | 22:00 |
| *** praneshp has quit IRC | 22:00 | |
| flashgordon | krtaylor: https://github.com/openstack/openstack/graphs/commit-activity | 22:00 |
| flashgordon | is what I was using | 22:01 |
| lifeless | should I attempt to requeue those patches ? | 22:01 |
| jeblair | lifeless: after the restart | 22:02 |
| sdague | flashgordon: I get 104 patches in the last day | 22:02 |
| sdague | git log --author=jenkins --since=1day --oneline | wc -l | 22:02 |
| sdague | on openstack/openstack | 22:02 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1272503 https://review.openstack.org/69027 | 22:02 |
| flashgordon | sdague: different day | 22:02 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1272511 https://review.openstack.org/69028 | 22:02 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1272509 https://review.openstack.org/69029 | 22:02 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1224208 https://review.openstack.org/69030 | 22:02 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1271962 https://review.openstack.org/69031 | 22:02 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1272417 https://review.openstack.org/68965 | 22:02 |
| flashgordon | github uses UTC I think | 22:02 |
| bknudson | 104 is impressive given where it was. | 22:02 |
| mordred | yah | 22:03 |
| sdague | yeh, 104 is an immediate last 24hrs | 22:03 |
| flashgordon | btw sdague looks like neutron is still a wreaking ball | 22:03 |
| sdague | yeh | 22:03 |
| sdague | well it's helped that anteaya has been keeping them out of the queue | 22:03 |
| flashgordon | one snuck in | 22:04 |
| bknudson | change to requirements.txt | 22:04 |
| lifeless | jeblair: oh, thought it had | 22:04 |
| flashgordon | https://review.openstack.org/68245 | 22:04 |
| sdague | yeh, it almost passed | 22:04 |
| jeblair | lifeless: i'm holding for the top of the queue to merge (to avoid setting us back 1 hour) | 22:04 |
| *** ociuhandu has quit IRC | 22:04 | |
| flashgordon | portante: in the future don't make your e-r fingerprints dependant | 22:05 |
| sdague | jeblair: it's still at least another 5 minutes based on where the jenkins logs are | 22:06 |
| portante | flashgordon: okay | 22:06 |
| *** ArxCruz has quit IRC | 22:06 | |
| flashgordon | portante: and thanks | 22:06 |
| portante | welcome, that should cover all the uncategorized swift gate failures | 22:07 |
| sdague | portante: awesome, thanks | 22:07 |
| flashgordon | portante: \o/ | 22:07 |
| jeblair | sdague: did the xml tests get removed? | 22:07 |
| *** pcrews has quit IRC | 22:07 | |
| sdague | jeblair: yes | 22:07 |
| sdague | that merged about 3 hrs ago | 22:07 |
| jeblair | sdague: https://jenkins03.openstack.org/job/gate-tempest-dsvm-postgres-full/2904/console | 22:07 |
| lifeless | file uploading on -full | 22:07 |
| jeblair | i just saw XML show up a lot in the test names | 22:07 |
| *** praneshp has joined #openstack-infra | 22:08 | |
| flashgordon | jeblair: v2 API in nova has xml | 22:08 |
| *** shardy is now known as shardy_afk | 22:08 | |
| jeblair | oh, it's v3 xml only. that makes sense. | 22:08 |
| sdague | yeh, we are only killing for new api | 22:08 |
| jeblair | okay, so what about tempest.api.identity.admin.v3.test_roles.RolesV3TestXML. ? | 22:08 |
| jeblair | that says v3 and xml. :) | 22:08 |
| jeblair | oh, keystone | 22:09 |
| sdague | that's keystone | 22:09 |
| jeblair | ok. i'm learning things. | 22:09 |
| sdague | :) | 22:09 |
| flashgordon | sdague: want to make status.openstack.org/elastic-recheck/data/uncategorized.html discoverable from http://status.openstack.org/elastic-recheck/ | 22:09 |
| sdague | I could have missed one | 22:09 |
| sdague | flashgordon: yes... but I'm out of steam | 22:09 |
| sdague | and I'm about to completely run away from computers for the weekend | 22:09 |
| rustlebee | sdague: i hear ya | 22:09 |
| *** ArxCruz has joined #openstack-infra | 22:09 | |
| rustlebee | same here. | 22:09 |
| flashgordon | sdague: good luck trying to stay away | 22:10 |
| sdague | heh | 22:10 |
| sdague | I need some recharge time | 22:10 |
| sdague | way too much firefighting last weekend | 22:11 |
| flashgordon | for the big day on mondaY! | 22:11 |
| sdague | yeh | 22:11 |
| *** mfer has quit IRC | 22:13 | |
| mordred | jeblair: eek! you're learning things about openstack. this is very dangerous. ;) | 22:13 |
| *** dangers is now known as dangers_away | 22:13 | |
| sdague | so our fixed cost on setup on those pg runs that are taking a little time right now has grown to 14 minutes | 22:14 |
| sdague | before we start tempest | 22:14 |
| jeblair | mordred: i've already forgotten. we use a SOAP api, right? ;) | 22:14 |
| sdague | jeblair: corba | 22:14 |
| sdague | it's great | 22:14 |
| jeblair | sdague: oh it is! but i know we don't -- i'd recognize a corba api. :) | 22:15 |
| sdague | heh | 22:15 |
| sdague | fun fact of the day, there are 11 cobol projects on github | 22:15 |
| *** denis_makogon_ has quit IRC | 22:16 | |
| jeblair | do any of them have a license? | 22:16 |
| *** changbl has quit IRC | 22:16 | |
| clarkb | sdague: stackforge clearly needs 12 then | 22:16 |
| sdague | it's github... nothing has a license | 22:16 |
| *** branen has quit IRC | 22:17 | |
| mtreinish | sdague: only 11? | 22:17 |
| mordred | sdague: you may be amused to know that jeblair and I wrote a distributed lighting control system several years ago that was in python and C++ and communicated using corba | 22:17 |
| rustlebee | nice | 22:18 |
| sdague | https://github.com/azac/cobol-on-wheelchair | 22:18 |
| mordred | it worked quite well - I even ran a show with is where my laptop was communicating with the hardware control node over this new wireless protocol called 802.11b | 22:18 |
| *** jcooley_ has joined #openstack-infra | 22:19 | |
| sdague | mordred: nice | 22:19 |
| jeblair | it was _fast_; it was the only major rpc system in use at the time that could keep up the dmx refresh rate over the network | 22:19 |
| sdague | oh, when I am less fried, i need to ask you about lighting things for my screened in porch, as I really want rgb controlable led lighting out there | 22:20 |
| openstackgerrit | Jesse Noller proposed a change to openstack-infra/config: Add openstacksdk, see https://wiki.openstack.org/wiki/UnifiedSDK https://review.openstack.org/69008 | 22:20 |
| jeblair | sdague: what's the extra pg setup time? | 22:21 |
| jeblair | i mean what's it doing? | 22:21 |
| sdague | I don't know that it's extra | 22:21 |
| sdague | so there is 6 minutes in the first missing block | 22:21 |
| sdague | which I don't think we are timestamping, in the setupworkspace bits | 22:22 |
| jeblair | sdague: yeah, you can assign most of that to git fetches | 22:22 |
| sdague | we're in the cli tests in the last job | 22:23 |
| sdague | so that should finish in the next 60 seconds, then 3 minutes to upload logs | 22:23 |
| sdague | then you are good to go | 22:23 |
| *** marun has quit IRC | 22:24 | |
| clarkb | jeblair: things I hope to get around to next week include, pickling window sizes and reloading them on zuul restarts, and decoupling the state of the window size from the layout configs | 22:24 |
| clarkb | also I figure I should add an rpc command that lets you configure it on the fly | 22:24 |
| jeblair | clarkb: 'decoupling the state of the window size from the layout configs' ? | 22:24 |
| clarkb | jeblair: each time the layout is reloaded the window size is set to 20 for us | 22:24 |
| jeblair | clarkb: oh, i thought that's what you meant by pickling them | 22:24 |
| clarkb | jeblair: I think it is related | 22:25 |
| clarkb | but I need to handle the non zuul restart case too | 22:25 |
| sdague | clarkb: it might be nice to have more general state dump to something like yaml | 22:25 |
| sdague | jeblair: does zuul keep it's job list on restart? or do you have to save that off manually? | 22:25 |
| clarkb | sdague: we have json :) | 22:26 |
| sdague | or that | 22:26 |
| clarkb | sdague: if you let it coalesce it will nicely save lists of things | 22:26 |
| clarkb | but that is slow | 22:26 |
| jeblair | sdague: save it manually; having it dump the queues in the same place would be nice. | 22:26 |
| clarkb | jeblair: I am trying to debug your devstack-gate change now | 22:27 |
| jeblair | clarkb: thx | 22:27 |
| sdague | finally on the last test | 22:27 |
| sdague | this node is exceptionally slow | 22:27 |
| sdague | devstack-precise-rax-iad-1201417 | 22:28 |
| jeblair | sdague: yeah, all the nodes we were waiting on were iad | 22:28 |
| sdague | tests passed, 3 minutes to log upload | 22:28 |
| *** jcooley_ has quit IRC | 22:29 | |
| clarkb | jeblair: fatal: Not a git repository (or any parent up to mount parent ) the find /opt/ that copies repos isn't copying repos apparently | 22:30 |
| clarkb | jeblair: do we need a sync before the umount? | 22:31 |
| portante | flashgordon: okay, I'll rebase these rechecks independently and add the related bug reference | 22:31 |
| *** changbl has joined #openstack-infra | 22:31 | |
| flashgordon | portante: yeah | 22:31 |
| flashgordon | making them independent means less rebasing in the future | 22:31 |
| *** gyee has joined #openstack-infra | 22:31 | |
| flashgordon | portante: also I think we want at least two hits before adding to e-r | 22:31 |
| openstackgerrit | A change was merged to openstack/requirements: Update python-keystoneclient to 0.4.2 https://review.openstack.org/66494 | 22:32 |
| openstackgerrit | A change was merged to openstack/requirements: Bump paramiko version to 1.9.0 https://review.openstack.org/68088 | 22:32 |
| openstackgerrit | A change was merged to openstack/requirements: Bump version of sphinxcontrib-pecanwsme https://review.openstack.org/66077 | 22:32 |
| flashgordon | re: https://review.openstack.org/#/c/69028/1/queries/1272511.yaml | 22:32 |
| flashgordon | sdague: ^ thoughts? | 22:32 |
| portante | okay, so the goal is to only categorize failing jobs if there are two or more hits | 22:32 |
| flashgordon | portante: yeah less then that and we are saying its not a pattern | 22:33 |
| flashgordon | thats the current thinking at least | 22:33 |
| sdague | portante: unless you think it might come back | 22:33 |
| flashgordon | right | 22:33 |
| sdague | there are truly aberant events that may not | 22:33 |
| sdague | jeblair: so the bunch merged | 22:33 |
| flashgordon | sdague: I'm looking forward to adding the jobs on trunk so we get even better data | 22:34 |
| sdague | the nova change at top is probably another 6 - 8 minutes (not 0 like it says) | 22:34 |
| jeblair | sdague: eta for https://jenkins07.openstack.org/job/gate-tempest-dsvm-postgres-full/21/console ? | 22:34 |
| portante | I suspect that the one hit one you are referring to (test_node_write_timeout) is actually related to one of the other ones (test_node_read_timeout_retry) | 22:34 |
| jnoller | can you change a gerrit topic without dumping the entire thing so people know you renamed a thing? >_< | 22:34 |
| sdague | jeblair: yeh, 6 - 8 minutes | 22:34 |
| jeblair | sdague: too bad :( | 22:34 |
| portante | jnoller: I think as long as the Change-Id does not change | 22:34 |
| flashgordon | portante: I am looking at test_ratelimit_max_rate_multiple_acc right now | 22:34 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Added storyboard API to webclient venv https://review.openstack.org/68523 | 22:35 |
| sdague | jeblair: so I'd say restart whenever you are good | 22:35 |
| portante | flashgordon: that one has failed in the past | 22:35 |
| sdague | we got the big mass of work through | 22:35 |
| flashgordon | portante: ahh, so can you update the commit message | 22:35 |
| portante | yes, will do | 22:35 |
| jnoller | or someone other than jenkins could +1 me | 22:35 |
| sdague | jeblair: the problem is we're not actually reseting much | 22:35 |
| sdague | you know... good problem to have | 22:35 |
| flashgordon | so the commit message here should say why we should track it -- if its not obvious | 22:35 |
| flashgordon | which it usually is | 22:35 |
| *** ociuhandu_ has joined #openstack-infra | 22:36 | |
| portante | flashgordon: agreed, will do | 22:36 |
| openstackgerrit | Jesse Noller proposed a change to openstack-infra/config: Add python-openstacksdk, see https://wiki.openstack.org/wiki/UnifiedSDK https://review.openstack.org/69008 | 22:36 |
| flashgordon | portante: sweet | 22:36 |
| fungi | loos like nothing important for me in scrollback. i'm still semi-afk, but at least i have flights now. i get to spend the night in phoenix--can't wait | 22:36 |
| fungi | looks | 22:36 |
| sdague | fungi: bummer | 22:36 |
| flashgordon | sounds like you will have a nother rev of all these patches so gonna wait until then to continue reviewing 'em | 22:36 |
| sdague | why's that? | 22:36 |
| portante | flashgordon: yes, in a few minutes | 22:37 |
| flashgordon | portante: cool | 22:37 |
| *** dcramer_ has quit IRC | 22:37 | |
| flashgordon | thanks again for doing this | 22:37 |
| portante | flashgordon: you are welcome | 22:37 |
| fungi | sdague: flights through midway got cancelled | 22:37 |
| sdague | ah bummer | 22:37 |
| * fungi should have learned by now not to book northern layovers in the winter | 22:37 | |
| fungi | got me coming and going | 22:38 |
| *** mrodden1 has joined #openstack-infra | 22:38 | |
| sdague | yeh | 22:38 |
| clarkb | fungi: jeblair : I have changed the lable on 15.185.166.197 and am using it to poke around and see why jeblair's d-g change was so unhappy | 22:38 |
| sdague | this is why I'm going direct to korea week after next | 22:38 |
| *** mrodden has quit IRC | 22:38 | |
| sdague | layovers in winter are dangerous | 22:38 |
| fungi | clarkb: yeah, i saw the fails on it but hadn't looked closer yet | 22:38 |
| clarkb | fungi: jeblair: the image has /opt with the things I expect. I am now thinking the find to move stuff is not working correctly | 22:39 |
| clarkb | possibly because it crosses filesystem boundaries? | 22:39 |
| jeblair | sdague: what were those changes again? | 22:40 |
| fungi | find -x will cause it to not | 22:40 |
| sdague | jeblair: ... which changes? | 22:40 |
| *** julim has quit IRC | 22:40 | |
| jeblair | sdague: to promote | 22:40 |
| sdague | 68920, 68921, 68922 | 22:40 |
| *** praneshp has quit IRC | 22:41 | |
| fungi | clarkb: er, sorry, -xdev (or -mount). i was thinking of du | 22:41 |
| clarkb | fungi: no the find is fine, I think copying somewhere isn't | 22:41 |
| fungi | mmm | 22:41 |
| sdague | git log --author=jenkins --since=1day --oneline | wc -l | 22:42 |
| sdague | 110 | 22:42 |
| anteaya | flashgordon: 68245 came from jenkins, I can't turn off the jenkins | 22:42 |
| flashgordon | anteaya: ahh | 22:42 |
| fungi | sdague: there can actually be some not authored by jenkins if it didn't have to merge (promoted changes in particular are common for that) | 22:42 |
| jeblair | sdague: 68921 does not have an aprv | 22:42 |
| sdague | jeblair: let me fix | 22:43 |
| sdague | now it does | 22:43 |
| clarkb | fungi: we mount vdb2 on /mnt then use find to list all dirs one level under /opt and run a -exec on those to mv from /opt to /mnt which is a mv across fs boundaries | 22:44 |
| anteaya | and neutron in check is fine right now, since we have resources to spare, I don't think I have seen any patchsets doing recheck over and over | 22:44 |
| clarkb | fungi: we then umount /mnt/ and remount it over /opt | 22:44 |
| anteaya | and I haven't seen any neutron in the gate | 22:44 |
| clarkb | fungi: then, there is an rsync -a from /opt/git/openstack-infra/devstack-gate to /opt/stack/new/devstack-gate | 22:45 |
| clarkb | fungi: the git repo we expect in /opt/stack/new/devstack-gate is not actually a git repo according to the logs | 22:45 |
| *** mriedem has quit IRC | 22:46 | |
| fungi | huh, that's weird | 22:47 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1224208 https://review.openstack.org/69030 | 22:47 |
| *** sarob has quit IRC | 22:48 | |
| *** praneshp has joined #openstack-infra | 22:50 | |
| *** gsamfira has quit IRC | 22:50 | |
| clarkb | fungi: oh I think I see the problem | 22:51 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1271664 https://review.openstack.org/69056 | 22:51 |
| clarkb | give me a minute to make me confident in that statement | 22:51 |
| jeblair | fungi: i might have forkbombed nodepool and caused the oom killer to kill it. | 22:52 |
| *** ociuhandu_ has quit IRC | 22:53 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Use /opt/git instead of workspace cache https://review.openstack.org/68990 | 22:53 |
| clarkb | jeblair: fungi: ^ slashes matter :) | 22:53 |
| jeblair | fungi: it's restarted now. just letting you know. | 22:53 |
| sdague | heh | 22:54 |
| sdague | clarkb: ah, rsync slashs.... the bane of my world :) | 22:54 |
| fungi | jeblair: oh, heh. any tips for not forkbombing nodepool? | 22:54 |
| fungi | and yes, rsync is very pedantic about meanings of trailing slash vs no trailing slash on source and destination | 22:55 |
| fungi | always trips me up too | 22:55 |
| sdague | I think the problem is tridge is just smarter than any normal human | 22:55 |
| jeblair | fungi: i just botched a mass-delete | 22:55 |
| sdague | so the idea that anyone would mess that up is beyond him | 22:56 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1271962 https://review.openstack.org/69031 | 22:56 |
| *** crank has quit IRC | 22:56 | |
| fungi | jeblair: ok. too many in parallel? | 22:56 |
| *** crank has joined #openstack-infra | 22:57 | |
| clarkb | fungi: jeblair: I am going to change the node label on that host now | 22:57 |
| fungi | k | 22:57 |
| *** CaptTofu has quit IRC | 22:58 | |
| fungi | i need to go find some food anyway. i've got a long 24 hours of airports and planes ahead of me | 22:58 |
| fungi | back in a bit | 22:58 |
| jeblair | #status ok | 22:58 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 22:58 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1271664 https://review.openstack.org/69056 | 22:59 |
| *** sarob has joined #openstack-infra | 22:59 | |
| anteaya | fungi: ugh | 22:59 |
| *** sarob has quit IRC | 22:59 | |
| flashgordon | jgriffith: https://review.openstack.org/#/c/69056/ | 23:00 |
| flashgordon | looks like you have a new gate bug | 23:00 |
| *** sarob has joined #openstack-infra | 23:01 | |
| *** burt1 has quit IRC | 23:02 | |
| *** david-lyle has joined #openstack-infra | 23:02 | |
| sdague | jeblair: so is zuul going to pick up the +A bits that people set previously? | 23:02 |
| sdague | or do we need to jiggle them | 23:03 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1272503 https://review.openstack.org/69027 | 23:04 |
| *** ociuhandu_ has joined #openstack-infra | 23:04 | |
| *** AlexF_ has joined #openstack-infra | 23:04 | |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1272417 https://review.openstack.org/68965 | 23:04 |
| jeblair | sdague: it will not | 23:04 |
| jeblair | sdague: i'm working on a query to try to get all those | 23:05 |
| sdague | ok | 23:05 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1272509 https://review.openstack.org/69029 | 23:05 |
| *** etoews1 has left #openstack-infra | 23:06 | |
| jnoller | bleargh, how do I get jenkins to love me again with https://review.openstack.org/#/c/69008/6 | 23:07 |
| *** jergerber has quit IRC | 23:07 | |
| jeblair | jnoller: sorry, it got caught up in a zuul bug i just fixed; i left a recheck comment | 23:07 |
| jnoller | thank you! | 23:08 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1272511 https://review.openstack.org/69028 | 23:08 |
| *** che-arne has joined #openstack-infra | 23:09 | |
| portante | flashgordon: all six of them have been rebased, and appropriate headers added | 23:10 |
| flashgordon | portante: cool, will review shortly | 23:11 |
| portante | flashgordon: thx | 23:12 |
| *** ociuhandu_ has quit IRC | 23:12 | |
| *** virmitio has quit IRC | 23:13 | |
| *** ryanpetrello has quit IRC | 23:13 | |
| *** AlexF_ has quit IRC | 23:14 | |
| *** sarob has quit IRC | 23:15 | |
| *** AlexF_ has joined #openstack-infra | 23:16 | |
| *** andreaf has quit IRC | 23:17 | |
| *** alexpilotti has quit IRC | 23:18 | |
| jeblair | sdague: i just reapproved 52 changes | 23:18 |
| portante | flashgordon: one of those above e-r patches failed because pip could not install pyelasticsearch | 23:19 |
| jeblair | hopefully that will get the bulk of them | 23:19 |
| jeblair | annegentle: that looks to have picked up your ops guide changes | 23:20 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1223469 https://review.openstack.org/68943 | 23:21 |
| *** AlexF_ has quit IRC | 23:21 | |
| flashgordon | portante: heh, not sure if you saw my comment on https://review.openstack.org/#/c/68965/3 | 23:22 |
| flashgordon | portante: I thinnk it needs another revision, which will also cause a recheck | 23:23 |
| portante | ooops | 23:23 |
| portante | sorry | 23:23 |
| flashgordon | that was a funny one | 23:23 |
| portante | I forgot to do that | 23:23 |
| flashgordon | portante: np, thats the query I am the most eager to land | 23:24 |
| portante | sec | 23:24 |
| *** eharney has quit IRC | 23:25 | |
| jeblair | lifeless: there are 325134 instances of the nodepool sleep message in the log for today (30 minutes remain in the utc day). 282094 of them are for about 0.49 seconds. | 23:26 |
| jeblair | lifeless: 20584 are for 0.99 seconds | 23:26 |
| lifeless | jeblair: thanks; your patch to revert it hasn't landed yet I presume ? | 23:26 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1272417 https://review.openstack.org/68965 | 23:26 |
| portante | flashgordon: how's that? | 23:27 |
| jeblair | lifeless: not yet. i'd still like to. anything else you'd like to know? | 23:27 |
| lifeless | jeblair: no | 23:27 |
| lifeless | jeblair: I'll add something we can toggle on or off I think for future instrumentation | 23:27 |
| jeblair | lifeless: ok. this and things like it might be a good statsd metric too | 23:27 |
| *** praneshp has quit IRC | 23:28 | |
| lifeless | jeblair: so when we make a change that should make things better, we can turn on the stats, gather data, then turn it off - without restarting | 23:28 |
| lifeless | jeblair: yeah, though fungi had some concern about network traffic for that | 23:28 |
| flashgordon | portante: LGTM, waiting for check queue then +A | 23:28 |
| flashgordon | portante: so http://status.openstack.org/elastic-recheck/data/uncategorized.html gets regenerated on every commit to e-r so we should see the results pretty soon | 23:28 |
| jeblair | lifeless: what exactly are we trying to make better? | 23:28 |
| *** changbl has quit IRC | 23:28 | |
| portante | oh good | 23:29 |
| annegentle | jeblair: looks like it | 23:30 |
| *** MarkAtwood_ has quit IRC | 23:32 | |
| lifeless | I want to make reduce the excess API calls we make to make deletes happen faster | 23:32 |
| lifeless | in my copious spare time | 23:32 |
| portante | flashgordon: mind if I take on a neutron one? | 23:32 |
| *** praneshp has joined #openstack-infra | 23:33 | |
| flashgordon | portante: go for it | 23:33 |
| flashgordon | portante: with a little luck we can get the overall categorization rate up to at least 80 for monday's big gate fix | 23:34 |
| flashgordon | but I think we hit all the low hanging fruit already | 23:35 |
| lifeless | jeblair: which that data confirmed for me - we're making lots of API calls to do what we do, and we're hitting the rate limit a lot | 23:37 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1272417 https://review.openstack.org/68965 | 23:39 |
| *** ativelkov has joined #openstack-infra | 23:40 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1272503 https://review.openstack.org/69027 | 23:40 |
| jeblair | lifeless: yes, i don't think there's a way to instantaneously spin up 500 servers and not hit the rate limit. so nodepool is designed to run at or near the limits most of the time | 23:42 |
| jeblair | lifeless: reducing extraneous calls will be great though as it should improve overall throughput | 23:42 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1272511 https://review.openstack.org/69028 | 23:42 |
| lifeless | jeblair: so I'm focusing on deletes right now, but spawn we could just request 500 instances | 23:42 |
| lifeless | jeblair: via num_instances, but I saw code about diagnosing instances by hostname which I would want to discuss w/you before removing | 23:43 |
| flashgordon | lifeless: heh nova doesn't handle request 500 instaces very well | 23:43 |
| lifeless | flashgordon: someone should fix that | 23:43 |
| flashgordon | lifeless: we made it *better* | 23:43 |
| lifeless | flashgordon: even requesting batches of 20 would massively reduce our API calls | 23:43 |
| jeblair | lifeless: surely the bulk of the api calls are waiting for the instances, yeah? | 23:44 |
| flashgordon | lifeless: true dat | 23:44 |
| jeblair | lifeless: can you batch query instances? | 23:44 |
| flashgordon | lifeless: if we don't support that well now sounds like a great v3 feature | 23:45 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1224208 https://review.openstack.org/69030 | 23:45 |
| flashgordon | lifeless: batch query for just instance status | 23:45 |
| jeblair | flashgordon: that would be very nice | 23:46 |
| flashgordon | rustlebee: ^ | 23:46 |
| flashgordon | cyeoh: ^ | 23:46 |
| lifeless | we do batch query already - adding a new field to it shold be fairly easy | 23:46 |
| jeblair | lifeless: i don't _think_ we need the hostname for anything; but don't hold me to that until we look closer at specifics | 23:46 |
| jeblair | lifeless: what's the batch query that we do do? | 23:47 |
| lifeless | jeblair: /servers - I don't mean in nodepool, I mean in nova | 23:47 |
| jeblair | lifeless: i know; i'm switching 'we's here; what's the field you would need to add? | 23:47 |
| lifeless | jeblair: status | 23:48 |
| lifeless | jeblair: name + id isn't enough to tell if the server should be running or has failed | 23:48 |
| jeblair | lifeless: roger. sounds promising. | 23:48 |
| flashgordon | http://docs.openstack.org/api/openstack-compute/2/content/List_Servers-d1e2078.html | 23:48 |
| lifeless | flashgordon: oh, its in the JSON | 23:49 |
| lifeless | I was checking http://api.openstack.org/api-ref-compute.html | 23:49 |
| lifeless | so, done | 23:49 |
| lifeless | easy | 23:49 |
| jeblair | neat | 23:50 |
| lifeless | ok, must run. ciao. | 23:50 |
| flashgordon | lifeless: ah thats what I wanted thanks | 23:50 |
| flashgordon | jeblair: list servers which changes-since | 23:50 |
| flashgordon | *should* work | 23:51 |
| flashgordon | http://api.openstack.org/api-ref-compute.html#compute_servers | 23:51 |
| jeblair | flashgordon: oh that's a good idea too | 23:51 |
| lifeless | flashgordon: oh - https://review.openstack.org/#/c/69060/ | 23:51 |
| lifeless | flashgordon: ^ bit of a WTF moment for tripleo. Would love a +2 once jenkins acks it | 23:51 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Updated fingerprint for bug 1261182 https://review.openstack.org/69067 | 23:51 |
| lifeless | (its possible it will fail unit tests) | 23:52 |
| flashgordon | lifeless: will look at that one on monday, still working on gate stuff today | 23:52 |
| lifeless | au revoir | 23:52 |
| portante | flashgordon: the above casts a wider net for that bug and should catch more of what is considered uncategorized | 23:52 |
| flashgordon | no tests tisk tisk | 23:52 |
| flashgordon | portante: put that in the commit msg | 23:52 |
| *** rnirmal has quit IRC | 23:52 | |
| ativelkov | Folks, I am getting strange messages from Jenkins to one of the changesets: it posts "gate-noop SUCCESS in 1s" every several seconds | 23:54 |
| jeblair | ativelkov: i just noticed that; looking into it | 23:54 |
| portante | flashgordon: will do | 23:54 |
| ativelkov | jeblair: thanks | 23:55 |
| openstackgerrit | A change was merged to openstack-infra/config: Make gate-tripleo-deploy check for tripleo stuff. https://review.openstack.org/68866 | 23:55 |
| flashgordon | jeblair: https://review.openstack.org/#/c/52427 | 23:55 |
| flashgordon | heh | 23:55 |
| openstackgerrit | Peter Portante proposed a change to openstack-infra/elastic-recheck: Updated fingerprint for bug 1261182 https://review.openstack.org/69067 | 23:55 |
| *** xchu has joined #openstack-infra | 23:57 | |
| *** ociuhandu_ has joined #openstack-infra | 23:58 | |
| *** praneshp has quit IRC | 23:58 | |
| *** senk1 has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
