| *** dkehn__ is now known as dkehn_ | 00:00 | |
| *** mrodden has quit IRC | 00:01 | |
| *** dkranz has quit IRC | 00:03 | |
| *** banix has joined #openstack-infra | 00:03 | |
| *** gaelL has joined #openstack-infra | 00:04 | |
| *** chadlung has quit IRC | 00:06 | |
| *** dkehn has quit IRC | 00:06 | |
| *** dkehn has joined #openstack-infra | 00:07 | |
| *** dkehn_ has quit IRC | 00:09 | |
| *** dkehn is now known as dkehn_ | 00:09 | |
| *** harlowja is now known as harlowja_away | 00:09 | |
| *** dkehn_ has quit IRC | 00:09 | |
| *** lyle has quit IRC | 00:10 | |
| *** dkehn has joined #openstack-infra | 00:10 | |
| *** dkehn_ has joined #openstack-infra | 00:10 | |
| *** virmitio has quit IRC | 00:12 | |
| *** lyle has joined #openstack-infra | 00:12 | |
| openstackgerrit | Tim Landscheidt proposed a change to openstack-infra/git-review: Retrieve remote pushurl independently of user's locale https://review.openstack.org/64307 | 00:14 |
|---|---|---|
| *** banix has quit IRC | 00:22 | |
| *** dkehn__ has joined #openstack-infra | 00:24 | |
| *** mrodden has joined #openstack-infra | 00:25 | |
| *** krtaylor has joined #openstack-infra | 00:25 | |
| *** dkehn__ has quit IRC | 00:26 | |
| *** dkehn_ has quit IRC | 00:26 | |
| *** ryanpetrello has quit IRC | 00:26 | |
| *** dkehn has quit IRC | 00:26 | |
| *** dkehn__ has joined #openstack-infra | 00:27 | |
| *** dkehn has joined #openstack-infra | 00:28 | |
| clarkb | down to 16 remaining shards. bumping the concurrent repplication number seems to have helped a bit | 00:28 |
| clarkb | starting to dig into logstash.o.o swap slightly | 00:30 |
| wenlock | fungi, that problem i had earlier was resolved by adjusting our config for https://github.com/openstack-infra/zuul/commit/01c2e63057a77f1c58bdcc6b1321de346f759c23 | 00:30 |
| clarkb | and down to 14 | 00:31 |
| fungi | wenlock: oh, are you pushing your zuul refs into your gerrit repositories? | 00:32 |
| wenlock | yeah, we've been using the git plugin section | 00:32 |
| wenlock | with ZUUL_REF as the refspec | 00:32 |
| clarkb | you really don't want to though :( it made everyting so slow | 00:32 |
| fungi | i had forgotten that was even an option, we've been serving them from zuul for so long | 00:33 |
| fungi | speaking of which, i want time to focus on the zuul merger/git server scale-out changes | 00:33 |
| jeblair | wenlock: that option won't be in the next version of zuul either. | 00:33 |
| wenlock | ahh so something we might need to revisit | 00:33 |
| wenlock | jeblair, good to know! so that means scm section is depricated? | 00:34 |
| jeblair | wenlock: no, just that zuul will need to serve the refs itself; you should be able to use the git plugin to fetch it from zuul instead of gerrit | 00:34 |
| openstackgerrit | A change was merged to openstack/requirements: Oslo program has adopted cliff, pycadf, stevedore, and taskflow https://review.openstack.org/72330 | 00:36 |
| wenlock | jeblair, because zuul is always pulled as latest with vcsrepo, it means we have to make sure to keep up with changes like that quickly , right? | 00:36 |
| openstackgerrit | A change was merged to openstack/requirements: Use OS's pypi mirror for Marconi https://review.openstack.org/72646 | 00:36 |
| jeblair | wenlock: if you run it like that, yes; we occasionally merge potentially breaking changes though we try not to | 00:38 |
| wenlock | jeblair, makes sense... thinking maybe it should be parameter to offer an option to use a specific tag | 00:38 |
| jeblair | wenlock: safest thing if you want to do that is to review upcoming changes in gerrit. | 00:38 |
| jeblair | wenlock: not sure i understand that suggestion | 00:38 |
| *** banix has joined #openstack-infra | 00:39 | |
| wenlock | jeblair, making a parameter to the zuul class in config/modules/zuul/manifests/init.pp that implements an argument for pulling some tag instead of latest as an option, while defaulting to latest. | 00:39 |
| clarkb | 12 shards now | 00:40 |
| *** gokrokve_ has quit IRC | 00:41 | |
| jeblair | wenlock: ah yes, that's a good idea. i'd accept that patch. :) | 00:41 |
| *** gokrokve has joined #openstack-infra | 00:41 | |
| jeblair | wenlock: also, fyi i try to make sure that major or breaking changes are noted in the NEWS file, so you can check that on new releases. | 00:42 |
| wenlock | cool | 00:42 |
| wenlock | jeblair, awesome | 00:42 |
| clarkb | 27MB into swap on logstash.o.o | 00:43 |
| jeblair | wenlock: if you run it in the same configuration we do, it's much safer to run CD off of master | 00:43 |
| wenlock | jeblair, agreed... we were following the wiki setup instructions...i think we might default to something closer to a devstack gate in our side | 00:44 |
| jeblair | wenlock: we very rarely break that configuration; and i actually don't expect us to in the forseable future | 00:44 |
| wenlock | jeblair, im not even sure we understood what value refspecs were giving us | 00:44 |
| *** dcramer_ has joined #openstack-infra | 00:45 | |
| *** gokrokve has quit IRC | 00:45 | |
| wenlock | kinda learning it as we go | 00:45 |
| jeblair | wenlock: well, zuul wants to tell a worker to "test something" where something is specified by the ZUUL_REF. how zuul comes up with that is a long story and best ignored for now. | 00:46 |
| jeblair | wenlock: so the issue at hand is where that ZUUL_REF (which is just a git ref) is located; on the zuul server itself or on gerrit | 00:47 |
| jeblair | having gerrit serve it is convenient and easy to set up, but we found in the long run, a bad idea | 00:47 |
| jeblair | because your gerrit server gets full of silly useless one-time refs | 00:47 |
| jeblair | so instead, it's better to just have zuul serve that to the workers directly | 00:47 |
| wenlock | jeblair... ahh that makes sense.... so there is no clean up for them? | 00:47 |
| jeblair | nope. we have a cron on our zuul server that packs them at least, but the only way to get rid of them is actually quite intrusive... | 00:48 |
| *** senthil has joined #openstack-infra | 00:48 | |
| jeblair | wenlock: so we occasionally just delete all the repos on our zuul server to clean them up. | 00:48 |
| jeblair | (it reclones them automatically) | 00:49 |
| clarkb | now 42MB of swap, I guess in a real emergency I can stop apache and kibana for a short while | 00:49 |
| *** yamahata has joined #openstack-infra | 00:49 | |
| jeblair | hopefully in the future, we can have zuul take care of this for us. | 00:49 |
| jeblair | clarkb: kibana looked to be using a lot of ram | 00:49 |
| clarkb | jeblair: yeah the ruby vm like python and java isn't cheap | 00:50 |
| wenlock | jeblair.... i was wrong, second glance, there is already a revision option on that git clone for zuul... my bad | 00:53 |
| *** talluri has quit IRC | 00:56 | |
| *** david-lyle has joined #openstack-infra | 00:58 | |
| *** rcleere has quit IRC | 00:59 | |
| *** ryanpetrello has joined #openstack-infra | 01:00 | |
| *** mgagne has quit IRC | 01:02 | |
| *** senthil has quit IRC | 01:04 | |
| *** jhesketh_ has quit IRC | 01:04 | |
| *** bhuvan has joined #openstack-infra | 01:05 | |
| *** cody-somerville has quit IRC | 01:06 | |
| *** ryanpetrello has quit IRC | 01:06 | |
| *** senthil has joined #openstack-infra | 01:11 | |
| *** bhuvan_ has joined #openstack-infra | 01:11 | |
| *** bhuvan has quit IRC | 01:11 | |
| *** david-lyle has quit IRC | 01:11 | |
| *** VijayT has joined #openstack-infra | 01:13 | |
| clarkb | jeblair: I am leaning towards stopping kibana on logstash.o.o so that I can walk home without watching that host | 01:13 |
| jeblair | clarkb: sounds good | 01:13 |
| clarkb | jeblair: thoughts? then from home I will spin up the workers as we are getting closish to full recovery | 01:13 |
| clarkb | 9 shards in process now | 01:14 |
| clarkb | kibana stopped, heading home now back on in about half an hour | 01:16 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Use client_id as part of the logger name https://review.openstack.org/73118 | 01:16 |
| jeblair | that ^ might seem out of left field, but i think i need it to help track down one of the nondeterministic zuul test bugs | 01:16 |
| *** dkehn_ has joined #openstack-infra | 01:17 | |
| *** prad__ has quit IRC | 01:18 | |
| *** alexpilotti has quit IRC | 01:18 | |
| *** cody-somerville has joined #openstack-infra | 01:18 | |
| fungi | the bare-centos6 image build in rax-ord is running for close to 90 minutes now, but it's still doing things, not hung | 01:19 |
| openstackgerrit | A change was merged to openstack/requirements: Add pbr to projects.txt https://review.openstack.org/71834 | 01:19 |
| jeblair | fungi: yeah, i just had the same thought and checked. | 01:19 |
| *** dkehn_ has quit IRC | 01:19 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: add dstat to our list of statistics collectors https://review.openstack.org/72786 | 01:19 |
| *** dkehn has quit IRC | 01:19 | |
| *** dkehn_ has joined #openstack-infra | 01:20 | |
| fungi | hoping once it finishes i can shoot one of the rax-iad nodes in the head and watch one successfully build in ord | 01:20 |
| *** dkehn__ has quit IRC | 01:20 | |
| fungi | i would say then we're probably okay to merge the bare-centos6 job config switch, but it's getting late and if my power goes out i won't be around to look after it | 01:20 |
| openstackgerrit | Jenkins proposed a change to openstack-dev/pbr: Updated from global requirements https://review.openstack.org/73119 | 01:21 |
| jeblair | fungi: yeah, i have about an hour left before i'm supposed to be elsewhere, so we may want to wait. | 01:21 |
| jeblair | oh look at that. ^ :) | 01:21 |
| *** dkehn has joined #openstack-infra | 01:21 | |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/reviewstats: Update the Ironic core team list https://review.openstack.org/73068 | 01:22 |
| jeblair | mordred: can you confirm that we really do want automatic updates for pbr? i could see how we might not due to it's location at the bottom of the dependency stack | 01:23 |
| jeblair | mordred: https://review.openstack.org/#/c/71834/ and https://review.openstack.org/73119 | 01:23 |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Nodepool image logs accessible via HTTP https://review.openstack.org/71190 | 01:26 |
| *** senthil has quit IRC | 01:28 | |
| *** yaguang has joined #openstack-infra | 01:31 | |
| *** oubiwann_ has joined #openstack-infra | 01:34 | |
| *** atiwari has quit IRC | 01:34 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/reviewstats: Add Sergey Lukjanov to infra-core https://review.openstack.org/73122 | 01:35 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/reviewstats: Add new infra projects https://review.openstack.org/73123 | 01:35 |
| *** mgagne has joined #openstack-infra | 01:38 | |
| *** david-lyle has joined #openstack-infra | 01:40 | |
| *** harlowja_away is now known as harlowja | 01:43 | |
| *** w_ has joined #openstack-infra | 01:48 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerrit: replace submodule url reference from relative to absolute https://review.openstack.org/73125 | 01:49 |
| *** alexpilotti has joined #openstack-infra | 01:50 | |
| *** olaph has quit IRC | 01:50 | |
| *** zul has joined #openstack-infra | 01:51 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerrit: add a .gitreview file https://review.openstack.org/73126 | 01:52 |
| *** pcrews has quit IRC | 01:53 | |
| zaro | fungi: i figured out the submodule situation. need to make a few changes to openstack/2.8 branch. wondering if you have time to help me push thru? | 01:55 |
| fungi | zaro: i can try | 01:55 |
| zaro | fungi: https://review.openstack.org/#/c/73126 and https://review.openstack.org/#/c/73125 | 01:56 |
| zaro | fungi: those failed, but tests are invalid. | 01:56 |
| *** nosnos has joined #openstack-infra | 02:00 | |
| *** morganfainberg has quit IRC | 02:02 | |
| zaro | fungi: what i mean is that changes to openstack/2.8 branch triggers tests meant to for master branch, master is on gerrit 2.4 | 02:04 |
| fungi | zaro: what needs to be done to get check-gerrit-unittests passing? | 02:04 |
| *** zehicle_at_dell has joined #openstack-infra | 02:04 | |
| *** bradm has quit IRC | 02:05 | |
| fungi | we should probably add a gerrit-specific noop job which is voting, and then make check-gerrit-unittests nonvoting on everything except the '^openstack/2\.4\..*$' branch pattern | 02:06 |
| clarkb | down to 4 shards now | 02:06 |
| fungi | awesome | 02:06 |
| zaro | fungi: this change sets up correctly testing on the right branches https://review.openstack.org/#/c/60348 | 02:07 |
| *** bradm has joined #openstack-infra | 02:07 | |
| fungi | zaro: do we have a unit test job for gerrit 2.8? | 02:07 |
| fungi | or at least the information we need to write one? | 02:07 |
| zaro | fungi: yes, it's in 60348 | 02:07 |
| *** salv-orlando_ has joined #openstack-infra | 02:07 | |
| zaro | fungi: but 60348 depends on 73126 | 02:08 |
| fungi | ahh, yes, i had reviewed a previous patchset for the job addition | 02:08 |
| zaro | fungi: sorry i mean 60348 depends on 73125 | 02:08 |
| *** sarob has quit IRC | 02:08 | |
| zaro | fungi: we need to setup the git submodules correctly before merging 60348 | 02:09 |
| *** nicedice_ has joined #openstack-infra | 02:09 | |
| fungi | got it, so we need 73125 force-merged to be able to run the correct jobs | 02:09 |
| *** sarob has joined #openstack-infra | 02:09 | |
| zaro | fungi: yup | 02:09 |
| fungi | though... i think we should be able to merge 60348 first | 02:10 |
| *** mkerrin1 has joined #openstack-infra | 02:10 | |
| *** homeless_ has joined #openstack-infra | 02:10 | |
| *** lttrl has quit IRC | 02:10 | |
| fungi | it will leave us unable to merge anything new for gerrit 2.8 until we merge 73125 | 02:10 |
| zaro | fungi: 60348 run 'buck build release' that type of build requires the plugins | 02:10 |
| fungi | but we ought to be able to recheck 73125 once 60348 is applied on jenkins/zuul | 02:10 |
| *** amotoki__ has joined #openstack-infra | 02:10 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Remove tox locale overrides https://review.openstack.org/72207 | 02:11 |
| zaro | fungi: ok if this is confusing. i can roll 73125 into 60348. | 02:11 |
| zaro | fungi: how's that? | 02:11 |
| fungi | you can't roll them together--they're for separate projects | 02:11 |
| zaro | fungi: ohh yeah, your right. | 02:12 |
| fungi | the config change implementing the job can go in, so long as we don't run that job nothing breaks, right? | 02:12 |
| fungi | and then the first change which would run that job is 73125, which adds the submodule corrections | 02:12 |
| clarkb | 3 shards now | 02:12 |
| fungi | we ought to be able to recheck 73125 and see the job run and hopefully pass | 02:12 |
| *** mtaylor has joined #openstack-infra | 02:13 | |
| zaro | fungi: hmm, but that change sets up the job and it will run. | 02:13 |
| *** harlowja_ has joined #openstack-infra | 02:13 | |
| *** locke105 has quit IRC | 02:13 | |
| *** SergeyLukjanov has quit IRC | 02:13 | |
| *** orionNebula has joined #openstack-infra | 02:13 | |
| *** sc68cal has quit IRC | 02:13 | |
| zaro | fungi: are you suggesting maybe adding noop job to that change and running that initially instead? | 02:13 |
| *** sarob has quit IRC | 02:14 | |
| fungi | zaro: those jobs being added in 60348 only run in check, gate and post pipelines for changes to gerrit | 02:14 |
| *** mikal_ has joined #openstack-infra | 02:14 | |
| *** DinaBelova_ has quit IRC | 02:14 | |
| *** mikal has quit IRC | 02:14 | |
| *** ogelbukh1 has joined #openstack-infra | 02:14 | |
| *** tristanC_ has joined #openstack-infra | 02:14 | |
| *** nico_ has joined #openstack-infra | 02:14 | |
| zaro | fungi: yes, 60348 will fail unless 73125 is merged. | 02:15 |
| fungi | it's a "if a tree falls in the forest and there's noone round to hear it" question | 02:15 |
| *** bnemec_ has joined #openstack-infra | 02:15 | |
| *** SergeyLukjanov has joined #openstack-infra | 02:15 | |
| fungi | does it matter if that jobs broken, if the first change it runs on is the one which fixes it? | 02:15 |
| *** mkerrin has quit IRC | 02:15 | |
| *** bhuvan_ has quit IRC | 02:15 | |
| *** amotoki_ has quit IRC | 02:15 | |
| *** NikitaKonovalov has quit IRC | 02:15 | |
| *** yaguang has quit IRC | 02:15 | |
| *** jhesketh__ has quit IRC | 02:15 | |
| *** mordred has quit IRC | 02:15 | |
| *** SpamapS has quit IRC | 02:15 | |
| *** salv-orlando has quit IRC | 02:15 | |
| *** changbl has quit IRC | 02:15 | |
| *** homeless has quit IRC | 02:15 | |
| *** bnemec has quit IRC | 02:15 | |
| *** Clabbe has quit IRC | 02:15 | |
| *** ruhe has quit IRC | 02:15 | |
| *** nkinder has quit IRC | 02:15 | |
| *** kevinbenton has quit IRC | 02:15 | |
| *** gaelL has quit IRC | 02:15 | |
| *** vladan has quit IRC | 02:15 | |
| *** rockstar has quit IRC | 02:15 | |
| *** salv-orlando_ is now known as salv-orlando | 02:15 | |
| *** nosnos_ has joined #openstack-infra | 02:15 | |
| *** plomakin_ has joined #openstack-infra | 02:15 | |
| zaro | fungi: ohh, i get what you mean. sorry. i'm brain dead. | 02:15 |
| *** changbl has joined #openstack-infra | 02:15 | |
| *** davidlenwell_ has joined #openstack-infra | 02:15 | |
| *** max_lobur has joined #openstack-infra | 02:16 | |
| *** dkehn__ has joined #openstack-infra | 02:16 | |
| *** sc68cal has joined #openstack-infra | 02:16 | |
| zaro | fungi: ok. so 60348 can go in before 73125. | 02:16 |
| *** jhesketh has joined #openstack-infra | 02:16 | |
| *** bodepd_ has joined #openstack-infra | 02:16 | |
| *** mkerrin has joined #openstack-infra | 02:16 | |
| fungi | so we approve the job change, which will merge to the config repo just fine. then we recheck the gerrit change which corrects the submodules, which ends up self-testing on the new job | 02:16 |
| *** kevinbenton has joined #openstack-infra | 02:17 | |
| fungi | yep | 02:17 |
| *** gaelL has joined #openstack-infra | 02:17 | |
| fungi | and if the recheck fails, we analyze the reason, and either adjust it or land another config patch correcting the job further | 02:17 |
| openstackgerrit | Cyril Roelandt proposed a change to openstack-infra/config: python-savannaclient: enable the py33 gates https://review.openstack.org/73128 | 02:17 |
| zaro | yeah, ok. i'm heading out now, but will 60348 take it off WIP status. | 02:17 |
| fungi | sounds great | 02:18 |
| *** Clabbe has joined #openstack-infra | 02:18 | |
| *** b3nt_pin has joined #openstack-infra | 02:18 | |
| *** mrodden has quit IRC | 02:18 | |
| *** CaptTofu has quit IRC | 02:18 | |
| *** max_lobur_afk has quit IRC | 02:18 | |
| *** ProfFalken has quit IRC | 02:18 | |
| *** plomakin has quit IRC | 02:18 | |
| *** branen_ has joined #openstack-infra | 02:18 | |
| *** david-lyle has quit IRC | 02:18 | |
| *** bodepd has quit IRC | 02:18 | |
| *** dkehn_ has quit IRC | 02:18 | |
| *** ProfFalken has joined #openstack-infra | 02:19 | |
| *** mtreinish has quit IRC | 02:19 | |
| *** esmute has quit IRC | 02:19 | |
| *** bradm has quit IRC | 02:19 | |
| *** zehicle_at_dell has quit IRC | 02:19 | |
| *** alexpilotti has quit IRC | 02:19 | |
| *** VijayT has quit IRC | 02:19 | |
| *** yamahata has quit IRC | 02:19 | |
| *** lyle has quit IRC | 02:19 | |
| *** beagles has quit IRC | 02:19 | |
| *** yamamoto has quit IRC | 02:19 | |
| *** gyee has quit IRC | 02:19 | |
| *** ogelbukh has quit IRC | 02:19 | |
| *** nicedice has quit IRC | 02:19 | |
| *** freyes has quit IRC | 02:19 | |
| *** harlowja has quit IRC | 02:19 | |
| *** exz has quit IRC | 02:19 | |
| *** tristanC has quit IRC | 02:19 | |
| *** lcestari has quit IRC | 02:19 | |
| *** gema has quit IRC | 02:19 | |
| *** dkliban has quit IRC | 02:19 | |
| *** yamahata_ has quit IRC | 02:19 | |
| *** mattymo has quit IRC | 02:19 | |
| *** cp16net has quit IRC | 02:19 | |
| *** kruskakli has quit IRC | 02:19 | |
| fungi | yay! we have bare-centos6 nodepool images in both rax-iad and rax-ord now | 02:19 |
| *** zehicle_at_dell has joined #openstack-infra | 02:19 | |
| openstackgerrit | Yongli He proposed a change to openstack-infra/storyboard: Use six.moves.urllib.parse instead of urlparse https://review.openstack.org/72887 | 02:19 |
| *** ogelbukh1 has quit IRC | 02:20 | |
| *** asalkeld has joined #openstack-infra | 02:20 | |
| *** gema has joined #openstack-infra | 02:20 | |
| fungi | deleted one of the 14 ready rax-iad nodes, waiting for one to hopefully start building in rax-ord | 02:21 |
| *** esmute has joined #openstack-infra | 02:22 | |
| *** mkerrin1 has quit IRC | 02:22 | |
| *** nosnos has quit IRC | 02:22 | |
| *** zul has quit IRC | 02:22 | |
| *** dcramer_ has quit IRC | 02:22 | |
| *** banix has quit IRC | 02:22 | |
| *** branen has quit IRC | 02:22 | |
| *** kmartin has quit IRC | 02:22 | |
| *** davidlenwell has quit IRC | 02:22 | |
| *** bogdando has quit IRC | 02:22 | |
| *** medieval1_ has quit IRC | 02:22 | |
| *** uvirtbot has quit IRC | 02:22 | |
| clarkb | woot | 02:22 |
| asalkeld | hi folks where is the best place to look for setting up a full genkins/zuul gating system? | 02:22 |
| clarkb | are they pvhvm? | 02:23 |
| asalkeld | jenkin/gerrit | 02:23 |
| *** cody-somerville has quit IRC | 02:23 | |
| asalkeld | (so install/configuration instructions) | 02:23 |
| openstackgerrit | Cyril Roelandt proposed a change to openstack-infra/config: python-tuskarclient: enable the py33 gate https://review.openstack.org/73130 | 02:23 |
| clarkb | asalkeld: there is a running your own doc at http://ci.openstack.org which is probably a good place to start | 02:23 |
| *** nkinder has joined #openstack-infra | 02:23 | |
| *** NikitaKonovalov_ has joined #openstack-infra | 02:24 | |
| *** kevinbenton_ has joined #openstack-infra | 02:24 | |
| *** SpamapS has joined #openstack-infra | 02:24 | |
| *** SpamapS has quit IRC | 02:24 | |
| *** SpamapS has joined #openstack-infra | 02:24 | |
| *** lttrl has joined #openstack-infra | 02:25 | |
| *** gaelL_ has joined #openstack-infra | 02:25 | |
| *** esmute has quit IRC | 02:25 | |
| *** ProfFalken has quit IRC | 02:25 | |
| fungi | asalkeld: http://ci.openstack.org/running-your-own.html | 02:25 |
| *** CaptTofu has joined #openstack-infra | 02:25 | |
| *** tru_tru has quit IRC | 02:25 | |
| *** lcestari_ has joined #openstack-infra | 02:25 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 02:26 | |
| *** yamahata__ has joined #openstack-infra | 02:26 | |
| *** nico_ is now known as exz | 02:26 | |
| *** nicedice_ has quit IRC | 02:26 | |
| asalkeld | thanks clarkb fungi | 02:26 |
| *** DinaBelova_ has joined #openstack-infra | 02:26 | |
| *** DinaBelova_ is now known as DinaBelova | 02:27 | |
| *** VijayT has joined #openstack-infra | 02:27 | |
| *** eharney has quit IRC | 02:27 | |
| *** ruhe has joined #openstack-infra | 02:27 | |
| *** cody-somerville has joined #openstack-infra | 02:27 | |
| *** davidlenwell has joined #openstack-infra | 02:28 | |
| *** max_lobur has quit IRC | 02:28 | |
| *** esmute has joined #openstack-infra | 02:28 | |
| *** david-lyle has joined #openstack-infra | 02:28 | |
| *** nicedice has joined #openstack-infra | 02:28 | |
| *** yamahata has joined #openstack-infra | 02:28 | |
| fungi | clarkb: yeah, the bare-centos slaves use base-image: 'CentOS 6.5 (PVHVM)' | 02:28 |
| fungi | and of course name-filter: 'Performance' | 02:28 |
| *** yaguang has joined #openstack-infra | 02:29 | |
| *** freyes has joined #openstack-infra | 02:29 | |
| *** SpamapS has quit IRC | 02:29 | |
| *** eharney has joined #openstack-infra | 02:30 | |
| *** lttrl has quit IRC | 02:30 | |
| *** julim has quit IRC | 02:30 | |
| *** SpamapS has joined #openstack-infra | 02:30 | |
| *** VijayT has quit IRC | 02:30 | |
| *** coolsvap has quit IRC | 02:30 | |
| *** kmartin has joined #openstack-infra | 02:30 | |
| *** zul has joined #openstack-infra | 02:30 | |
| *** dkliban has joined #openstack-infra | 02:30 | |
| *** dripton_ has joined #openstack-infra | 02:30 | |
| *** max_lobur has joined #openstack-infra | 02:30 | |
| *** bhuvan has joined #openstack-infra | 02:30 | |
| *** dripton has quit IRC | 02:30 | |
| *** gaelL_ has quit IRC | 02:31 | |
| *** lttrl has joined #openstack-infra | 02:31 | |
| *** ruhe has quit IRC | 02:32 | |
| *** julim_ has joined #openstack-infra | 02:32 | |
| *** julim_ has quit IRC | 02:32 | |
| *** NikitaKonovalov has quit IRC | 02:32 | |
| *** tristanC_ has quit IRC | 02:32 | |
| *** nosnos has joined #openstack-infra | 02:32 | |
| *** zehicle_at_dell has quit IRC | 02:33 | |
| *** b3nt_pin has quit IRC | 02:33 | |
| *** Clabbe has quit IRC | 02:33 | |
| *** kevinbenton has quit IRC | 02:33 | |
| *** bodepd_ has quit IRC | 02:33 | |
| *** jhesketh has quit IRC | 02:33 | |
| *** gaelL has quit IRC | 02:33 | |
| *** dkehn__ has quit IRC | 02:33 | |
| *** davidlenwell_ has quit IRC | 02:33 | |
| *** nosnos_ has quit IRC | 02:33 | |
| *** oubiwann_ has quit IRC | 02:33 | |
| *** kevinbenton_ is now known as kevinbenton | 02:33 | |
| *** andre__ has quit IRC | 02:34 | |
| *** locke105 has joined #openstack-infra | 02:34 | |
| *** dkehn has quit IRC | 02:35 | |
| *** lcestari_ has quit IRC | 02:35 | |
| *** lttrl has quit IRC | 02:35 | |
| *** DinaBelova has quit IRC | 02:35 | |
| *** bhuvan has quit IRC | 02:36 | |
| *** SpamapS has quit IRC | 02:36 | |
| *** cp16net has joined #openstack-infra | 02:36 | |
| *** SpamapS has joined #openstack-infra | 02:36 | |
| *** SpamapS has quit IRC | 02:36 | |
| *** SpamapS has joined #openstack-infra | 02:36 | |
| *** Clabbe has joined #openstack-infra | 02:37 | |
| *** vladan has joined #openstack-infra | 02:37 | |
| *** mgagne has quit IRC | 02:37 | |
| *** dripton__ has joined #openstack-infra | 02:37 | |
| *** NikitaKonovalov has joined #openstack-infra | 02:38 | |
| *** bhuvan has joined #openstack-infra | 02:38 | |
| *** jhesketh has joined #openstack-infra | 02:38 | |
| *** oubiwann_ has joined #openstack-infra | 02:38 | |
| *** tristanC has joined #openstack-infra | 02:38 | |
| *** gaelL_ has joined #openstack-infra | 02:38 | |
| *** jhesketh has quit IRC | 02:38 | |
| *** asalkeld has quit IRC | 02:38 | |
| *** dripton_ has quit IRC | 02:38 | |
| *** exz has quit IRC | 02:38 | |
| *** andre__ has joined #openstack-infra | 02:38 | |
| *** ruhe has joined #openstack-infra | 02:38 | |
| *** jhesketh has joined #openstack-infra | 02:38 | |
| *** mtreinish has joined #openstack-infra | 02:38 | |
| *** julim has joined #openstack-infra | 02:38 | |
| *** weshay has quit IRC | 02:38 | |
| *** weshay has joined #openstack-infra | 02:39 | |
| *** VijayT has joined #openstack-infra | 02:40 | |
| *** DinaBelova_ has joined #openstack-infra | 02:40 | |
| *** andre__ has quit IRC | 02:40 | |
| *** DinaBelova_ is now known as DinaBelova | 02:41 | |
| *** thedodd has joined #openstack-infra | 02:41 | |
| *** amotoki_ has joined #openstack-infra | 02:41 | |
| *** dkehn_ has joined #openstack-infra | 02:42 | |
| *** esker has joined #openstack-infra | 02:42 | |
| *** tru_tru has joined #openstack-infra | 02:42 | |
| *** gaelL has joined #openstack-infra | 02:43 | |
| *** gaelL_ has quit IRC | 02:43 | |
| *** lttrl has joined #openstack-infra | 02:43 | |
| *** mkerrin1 has joined #openstack-infra | 02:43 | |
| *** bnemec has joined #openstack-infra | 02:43 | |
| *** lcestari_ has joined #openstack-infra | 02:43 | |
| *** bodepd has joined #openstack-infra | 02:44 | |
| *** mgagne has joined #openstack-infra | 02:44 | |
| *** ilyashakhat_ has joined #openstack-infra | 02:45 | |
| *** SergeyLukjanov2 has joined #openstack-infra | 02:45 | |
| *** DinaBelova has quit IRC | 02:45 | |
| *** saper_ has joined #openstack-infra | 02:45 | |
| *** plomakin has joined #openstack-infra | 02:45 | |
| *** russellb_ has joined #openstack-infra | 02:45 | |
| *** b3nt_pin has joined #openstack-infra | 02:47 | |
| *** zul has quit IRC | 02:47 | |
| *** DinaBelova_ has joined #openstack-infra | 02:47 | |
| *** DinaBelova_ is now known as DinaBelova | 02:47 | |
| *** dhellmann_ has joined #openstack-infra | 02:48 | |
| *** motoki has joined #openstack-infra | 02:48 | |
| *** Ryan_Lane2 has joined #openstack-infra | 02:48 | |
| *** mikal has joined #openstack-infra | 02:49 | |
| *** bhuvan has quit IRC | 02:49 | |
| *** lyle has joined #openstack-infra | 02:49 | |
| *** bhuvan has joined #openstack-infra | 02:49 | |
| *** dstanek has quit IRC | 02:49 | |
| *** vladan has quit IRC | 02:49 | |
| *** nosnos has quit IRC | 02:49 | |
| *** david-lyle has quit IRC | 02:49 | |
| *** kevinbenton has quit IRC | 02:49 | |
| *** mkerrin has quit IRC | 02:49 | |
| *** sc68cal has quit IRC | 02:49 | |
| *** plomakin_ has quit IRC | 02:49 | |
| *** SergeyLukjanov has quit IRC | 02:49 | |
| *** bnemec_ has quit IRC | 02:49 | |
| *** mikal_ has quit IRC | 02:49 | |
| *** amotoki__ has quit IRC | 02:50 | |
| *** krtaylor has quit IRC | 02:50 | |
| *** dims has quit IRC | 02:50 | |
| *** rwsu has quit IRC | 02:50 | |
| *** ilyashakhat has quit IRC | 02:50 | |
| *** lifeless has quit IRC | 02:50 | |
| *** afazekas|sick has quit IRC | 02:50 | |
| *** amotoki has quit IRC | 02:50 | |
| *** gothicmindfood has quit IRC | 02:50 | |
| *** ianw has quit IRC | 02:50 | |
| *** saper has quit IRC | 02:50 | |
| *** belliott has quit IRC | 02:50 | |
| *** Ryan_Lane has quit IRC | 02:50 | |
| *** jpeeler has quit IRC | 02:50 | |
| *** dhellmann has quit IRC | 02:50 | |
| *** anteaya has quit IRC | 02:50 | |
| *** mburned has quit IRC | 02:50 | |
| *** russellb has quit IRC | 02:50 | |
| *** SergeyLukjanov2 is now known as SergeyLukjanov | 02:50 | |
| *** asalkeld has joined #openstack-infra | 02:50 | |
| *** dhellmann_ is now known as dhellmann | 02:50 | |
| *** russellb_ is now known as russellb | 02:50 | |
| *** dstanek has joined #openstack-infra | 02:50 | |
| *** vladan has joined #openstack-infra | 02:50 | |
| *** sc68cal has joined #openstack-infra | 02:51 | |
| *** kevinbenton has joined #openstack-infra | 02:51 | |
| *** NikitaKonovalov_ has joined #openstack-infra | 02:52 | |
| *** zul has joined #openstack-infra | 02:52 | |
| *** gothicmindfood has joined #openstack-infra | 02:54 | |
| *** mtreinish_ has joined #openstack-infra | 02:54 | |
| *** nosnos has joined #openstack-infra | 02:56 | |
| *** anteaya has joined #openstack-infra | 02:56 | |
| *** NikitaKonovalov has quit IRC | 02:57 | |
| *** mtreinish has quit IRC | 02:57 | |
| *** jhesketh has quit IRC | 02:57 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 02:57 | |
| *** mtreinish_ is now known as mtreinish | 02:57 | |
| *** jpeeler1 has joined #openstack-infra | 02:57 | |
| *** dims has joined #openstack-infra | 02:57 | |
| *** andre__ has joined #openstack-infra | 02:57 | |
| *** ianw has joined #openstack-infra | 02:57 | |
| *** lifeless has joined #openstack-infra | 02:58 | |
| *** gothicmi1dfood has joined #openstack-infra | 02:58 | |
| *** jhesketh has joined #openstack-infra | 02:59 | |
| *** thomasem has joined #openstack-infra | 02:59 | |
| *** thomasem has quit IRC | 02:59 | |
| *** krtaylor has joined #openstack-infra | 03:00 | |
| *** sarob has joined #openstack-infra | 03:01 | |
| *** rwsu has joined #openstack-infra | 03:01 | |
| *** whayutin_ has joined #openstack-infra | 03:02 | |
| *** dklyle has joined #openstack-infra | 03:03 | |
| *** pleia2 has quit IRC | 03:03 | |
| *** sarob has quit IRC | 03:03 | |
| *** pleia2 has joined #openstack-infra | 03:03 | |
| *** HenryG has quit IRC | 03:04 | |
| *** sarob has joined #openstack-infra | 03:04 | |
| *** gothicmindfood has quit IRC | 03:04 | |
| *** lttrl has quit IRC | 03:04 | |
| *** sc68cal has quit IRC | 03:04 | |
| *** belliott has joined #openstack-infra | 03:04 | |
| *** weshay has quit IRC | 03:04 | |
| *** yaguang has quit IRC | 03:04 | |
| *** lyle has quit IRC | 03:04 | |
| *** afazekas|sick has joined #openstack-infra | 03:04 | |
| *** bhuvan_ has joined #openstack-infra | 03:06 | |
| *** bhuvan has quit IRC | 03:06 | |
| *** belliott has quit IRC | 03:06 | |
| clarkb | woot cluster is green starting workers now | 03:07 |
| *** branen_ has quit IRC | 03:08 | |
| *** yaguang has joined #openstack-infra | 03:08 | |
| *** sarob has quit IRC | 03:08 | |
| *** branen_ has joined #openstack-infra | 03:09 | |
| anteaya | go clarkb | 03:09 |
| *** yamahata__ has quit IRC | 03:09 | |
| *** sc68cal has joined #openstack-infra | 03:10 | |
| *** yamahata__ has joined #openstack-infra | 03:10 | |
| *** belliott has joined #openstack-infra | 03:11 | |
| *** esker has quit IRC | 03:17 | |
| *** lttrl has joined #openstack-infra | 03:18 | |
| *** mburned has joined #openstack-infra | 03:20 | |
| *** dkehn_ has quit IRC | 03:21 | |
| *** pcrews has joined #openstack-infra | 03:21 | |
| clarkb | 64 gearman workers are running through 16 lgostash indexers | 03:21 |
| clarkb | and the backlog is falling | 03:21 |
| clarkb | leaving kibana off for now, will start that later thsi evening when memory use on logstash.o.o shouldn't be too bad | 03:22 |
| *** dkehn has joined #openstack-infra | 03:22 | |
| *** lttrl has quit IRC | 03:24 | |
| *** esker has joined #openstack-infra | 03:24 | |
| *** lttrl has joined #openstack-infra | 03:25 | |
| *** CaptTofu has quit IRC | 03:26 | |
| *** mgagne has quit IRC | 03:27 | |
| *** mgagne has joined #openstack-infra | 03:28 | |
| *** exz has joined #openstack-infra | 03:29 | |
| *** ProfFalken has joined #openstack-infra | 03:29 | |
| *** gyee has joined #openstack-infra | 03:29 | |
| *** bradm has joined #openstack-infra | 03:29 | |
| *** bogdando has joined #openstack-infra | 03:29 | |
| *** flaper87|afk has quit IRC | 03:29 | |
| *** flaper87|afk has joined #openstack-infra | 03:29 | |
| *** homeless_ has quit IRC | 03:30 | |
| *** homeless_ has joined #openstack-infra | 03:30 | |
| *** harlowja_ has quit IRC | 03:30 | |
| *** harlowja_ has joined #openstack-infra | 03:30 | |
| *** kmartin has quit IRC | 03:30 | |
| *** kmartin has joined #openstack-infra | 03:30 | |
| *** mkerrin1 has quit IRC | 03:30 | |
| *** mkerrin1 has joined #openstack-infra | 03:30 | |
| *** ianw has quit IRC | 03:30 | |
| *** ianw has joined #openstack-infra | 03:30 | |
| *** dklyle has quit IRC | 03:30 | |
| *** dklyle has joined #openstack-infra | 03:30 | |
| *** branen_ has quit IRC | 03:30 | |
| *** branen_ has joined #openstack-infra | 03:30 | |
| *** yamahata__ has quit IRC | 03:30 | |
| *** yamahata__ has joined #openstack-infra | 03:30 | |
| *** mburned has quit IRC | 03:30 | |
| *** mburned has joined #openstack-infra | 03:30 | |
| *** zul has quit IRC | 03:31 | |
| *** lttrl has quit IRC | 03:33 | |
| clarkb | going to step away for a bit now while it chews through the backlog | 03:34 |
| clarkb | its down from 390k to about 370k already | 03:34 |
| *** lttrl has joined #openstack-infra | 03:34 | |
| *** zul has joined #openstack-infra | 03:36 | |
| *** mgagne has quit IRC | 03:37 | |
| *** jishaom has joined #openstack-infra | 03:38 | |
| *** pcrews has quit IRC | 03:42 | |
| *** bhuvan_ has quit IRC | 03:42 | |
| *** bhuvan has joined #openstack-infra | 03:44 | |
| *** rcleere has joined #openstack-infra | 03:44 | |
| *** jhesketh_ has joined #openstack-infra | 03:47 | |
| *** mgagne has joined #openstack-infra | 03:48 | |
| *** senthil has joined #openstack-infra | 03:51 | |
| *** zul has quit IRC | 03:52 | |
| *** gokrokve has joined #openstack-infra | 03:55 | |
| *** chandan_kumar has joined #openstack-infra | 03:58 | |
| *** chandan_kumar has quit IRC | 04:09 | |
| *** chandan_kumar has joined #openstack-infra | 04:10 | |
| *** sarob has joined #openstack-infra | 04:14 | |
| *** asalkeld has left #openstack-infra | 04:16 | |
| *** bhuvan has quit IRC | 04:18 | |
| *** sarob has quit IRC | 04:19 | |
| clarkb | down to 337k, this is going to take some time | 04:20 |
| *** ogelbukh1 has joined #openstack-infra | 04:20 | |
| *** hdd has joined #openstack-infra | 04:21 | |
| *** mikal has quit IRC | 04:26 | |
| *** zehicle_at_dell has joined #openstack-infra | 04:26 | |
| *** mikal has joined #openstack-infra | 04:27 | |
| *** senthil has quit IRC | 04:32 | |
| *** amcrn has quit IRC | 04:42 | |
| *** dcramer_ has joined #openstack-infra | 04:49 | |
| clarkb | now 317k, so about 1k every couple minutes | 05:00 |
| clarkb | ~10 hours? | 05:00 |
| clarkb | my maths are fuzzy | 05:01 |
| jesusaurus | how much swap were you eating up? | 05:01 |
| jesusaurus | and did you bump your es flush size? | 05:01 |
| StevenK | clarkb: 316/30 == 10.53, so roughly | 05:01 |
| jesusaurus | clarkb: ? | 05:01 |
| clarkb | jesusaurus: yes bumped es flush queue, es is doing over 10k events per second right now | 05:02 |
| clarkb | jesusaurus: right now I am down to 30MB of swap | 05:02 |
| jesusaurus | nice | 05:02 |
| jesusaurus | 10k/s is pretty good | 05:03 |
| *** cadenzajon has quit IRC | 05:05 | |
| *** krtaylor has quit IRC | 05:06 | |
| clarkb | ya I am happy with it, once it is caught up it should have no problem keeping up | 05:06 |
| clarkb | memory use for logstash looks a lot better than before | 05:08 |
| *** zul has joined #openstack-infra | 05:10 | |
| *** krtaylor has joined #openstack-infra | 05:11 | |
| *** krtaylor has quit IRC | 05:11 | |
| *** thedodd has quit IRC | 05:12 | |
| *** DinaBelova is now known as DinaBelova_ | 05:15 | |
| *** mriedem has quit IRC | 05:18 | |
| *** hdd has quit IRC | 05:18 | |
| *** krtaylor has joined #openstack-infra | 05:19 | |
| *** Guest46806 has quit IRC | 05:20 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add Sergey Lukjanov to infra-core https://review.openstack.org/73122 | 05:20 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add new infra projects https://review.openstack.org/73123 | 05:21 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Update the Ironic core team list https://review.openstack.org/73068 | 05:21 |
| *** lillie has joined #openstack-infra | 05:21 | |
| *** lillie is now known as Guest75317 | 05:21 | |
| *** masayukig has joined #openstack-infra | 05:22 | |
| *** nicedice has quit IRC | 05:24 | |
| *** CaptTofu has joined #openstack-infra | 05:27 | |
| *** CaptTofu has quit IRC | 05:31 | |
| *** dstanek has quit IRC | 05:32 | |
| *** UtahDave has joined #openstack-infra | 05:32 | |
| *** masayukig has quit IRC | 05:32 | |
| *** masayukig has joined #openstack-infra | 05:32 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerrit: add a .gitreview file https://review.openstack.org/73126 | 05:34 |
| *** jamielennox is now known as jamielennox|away | 05:34 | |
| *** oubiwann_ has quit IRC | 05:35 | |
| *** gyee has quit IRC | 05:40 | |
| *** zehicle_at_dell has quit IRC | 05:40 | |
| *** amcrn has joined #openstack-infra | 05:42 | |
| *** chandankumar_ has joined #openstack-infra | 05:49 | |
| *** chandankumar_ has joined #openstack-infra | 05:50 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update comment filter for gerrit 2.8 https://review.openstack.org/70864 | 05:50 |
| *** fifieldt has joined #openstack-infra | 05:51 | |
| *** chandankumar_ has quit IRC | 05:52 | |
| *** dkehn has quit IRC | 05:52 | |
| *** tomhe has quit IRC | 05:52 | |
| *** chandan_kumar has quit IRC | 05:53 | |
| *** chandan_kumar has joined #openstack-infra | 05:54 | |
| *** dkehn has joined #openstack-infra | 05:54 | |
| *** gokrokve has quit IRC | 05:54 | |
| *** gokrokve has joined #openstack-infra | 05:55 | |
| *** UtahDave has quit IRC | 05:57 | |
| openstackgerrit | Yongli He proposed a change to openstack-infra/storyboard: Remove extraneous vim configuration comments https://review.openstack.org/73156 | 05:57 |
| *** gokrokve has quit IRC | 05:58 | |
| *** Ryan_Lane2 is now known as Ryan_Lane | 06:01 | |
| *** Ryan_Lane has joined #openstack-infra | 06:01 | |
| openstackgerrit | Nathan Kinder proposed a change to openstack-infra/config: Add openstack-security-notes stackforge project https://review.openstack.org/73157 | 06:02 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Fix Launchpad ID for Nikolay Starodubtsev https://review.openstack.org/73158 | 06:05 |
| *** dkehn_ has joined #openstack-infra | 06:05 | |
| *** zul has quit IRC | 06:08 | |
| *** amotoki_ has quit IRC | 06:09 | |
| SergeyLukjanov | jeblair, fungi, clarkb, good evening | 06:13 |
| SergeyLukjanov | hm, or night already | 06:13 |
| fungi | SergeyLukjanov: good morning! | 06:13 |
| clarkb | fungi: you are still awake? | 06:14 |
| clarkb | we are down to 291k | 06:14 |
| clarkb | about 100k less than where we started | 06:14 |
| SergeyLukjanov | is there anything with that I can help while you sleeping? :) | 06:14 |
| fungi | it's only a little after 1am local time where i am | 06:14 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Fix Launchpad ID for Nikolay Starodubtsev https://review.openstack.org/73158 | 06:14 |
| fungi | currently trying to get caught up on e-mail | 06:15 |
| *** afazekas|sick has quit IRC | 06:16 | |
| *** yolanda has joined #openstack-infra | 06:21 | |
| *** e0ne has joined #openstack-infra | 06:22 | |
| *** e0ne has quit IRC | 06:23 | |
| *** masayuki_ has joined #openstack-infra | 06:23 | |
| *** masayuki_ is now known as masayukig_ | 06:23 | |
| *** masayukig has quit IRC | 06:24 | |
| *** masayukig_ is now known as masayukig | 06:24 | |
| *** w_ is now known as olaph | 06:26 | |
| *** sarob has joined #openstack-infra | 06:31 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update All-Project configuration for gerrit 2.8 https://review.openstack.org/70818 | 06:33 |
| *** marun has quit IRC | 06:40 | |
| *** harlowja_ is now known as harlowja_away | 06:44 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: disable the jeepyb trivial rebase hook on review-dev.o.o https://review.openstack.org/69800 | 06:45 |
| clarkb | fungi: re the opendaylight third party tests | 06:46 |
| clarkb | fungi: I thought they were open and could just be tested upstream | 06:47 |
| fungi | was there a separate thread on that? i've been behind on e-mail | 06:48 |
| clarkb | fungi: no, | 06:48 |
| clarkb | it just occurs to me that we went from the extreme of having no testing to the other where none of the interesting testing happens upstream... | 06:48 |
| clarkb | docker is in that boat too | 06:48 |
| fungi | wonder why they're requesting a third party account | 06:48 |
| clarkb | fungi: I think because it is "required" | 06:48 |
| fungi | ah | 06:49 |
| clarkb | there is a disconnect between what we actually want which is testing, and what people think they need to do that | 06:49 |
| clarkb | granted trying to spin up docker or opendaylight tests upstream just before i3 probably isn't going to happen :) | 06:49 |
| clarkb | starting to get worried that the third party tests will just make it more difficult to upstream the tests | 06:50 |
| clarkb | because each of them will do things differently | 06:50 |
| clarkb | but in the case of open source testable things we want consistency | 06:50 |
| *** pblaho has joined #openstack-infra | 06:50 | |
| clarkb | but having tests is a good thing so I should just stop worrying and poke markmcclain | 06:51 |
| fungi | probably the same for any other plugin which could be tested upstream but waited until they had a couple weeks to go to look into it | 06:51 |
| zaro | clarkb, fungi : does change id link work for you? https://review.openstack.org/#/c/69800 | 06:52 |
| zaro | one in the last comment | 06:52 |
| fungi | zaro i'm not where i can pull up gerrit easily but i'll try to have a look tomorrow | 06:53 |
| *** fifieldt_ has joined #openstack-infra | 06:53 | |
| *** fifieldt has quit IRC | 06:53 | |
| clarkb | zaro: no, look at the url it is using, the $1 isn't being expanded | 06:54 |
| zaro | clarkb: yeah, is that new problem? | 06:54 |
| clarkb | I think so | 06:54 |
| clarkb | did we restart gerrit to pick up more changes? | 06:54 |
| *** masayukig has quit IRC | 06:55 | |
| zaro | i didn't if you are asking me. | 06:56 |
| clarkb | zaro: I know :) it may have been picekd up when gerrit restarted for project renames last week | 06:57 |
| clarkb | but I wasn't aware of us merging any additional changes to the comment links | 06:57 |
| zaro | anyways it might be good to restart or something. i've noticed that it's been a little flaky. sometimes it fails to submit a review. | 06:57 |
| clarkb | zaro: we need to fix the config | 06:57 |
| clarkb | restarting won't help otherwis | 06:57 |
| zaro | i really noticed flakeness today, but previously it's been ok. | 06:58 |
| zaro | or maybe i'm just making more commits today than usual. getting all the gerrit upgrade patches in shape. | 06:59 |
| *** motoki has quit IRC | 07:00 | |
| *** garyk has joined #openstack-infra | 07:02 | |
| *** denis_makogon has joined #openstack-infra | 07:04 | |
| *** wenlock has quit IRC | 07:12 | |
| *** dkehn__ has joined #openstack-infra | 07:19 | |
| *** sarob has quit IRC | 07:21 | |
| *** dkehn__ has quit IRC | 07:21 | |
| *** sarob has joined #openstack-infra | 07:21 | |
| *** dkehn__ has joined #openstack-infra | 07:21 | |
| *** dkehn_ has quit IRC | 07:22 | |
| *** dkehn has quit IRC | 07:22 | |
| *** dkehn has joined #openstack-infra | 07:24 | |
| *** sarob has quit IRC | 07:26 | |
| *** CaptTofu has joined #openstack-infra | 07:27 | |
| *** CaptTofu has quit IRC | 07:32 | |
| *** flaper87|afk is now known as flaper87 | 07:33 | |
| fungi | so...much...e-mail | 07:38 |
| *** UtahDave has joined #openstack-infra | 07:38 | |
| fungi | the rest will just have to wait for tomorrow | 07:39 |
| fungi | <- zzz | 07:39 |
| openstackgerrit | Lin Tan proposed a change to openstack-infra/jeepyb: Rename Openstack to OpenStack https://review.openstack.org/73180 | 07:42 |
| openstackgerrit | A change was merged to openstack-infra/git-review: Add 2m timeout to tests https://review.openstack.org/71223 | 07:43 |
| *** e0ne has joined #openstack-infra | 07:47 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Make Gerrit port and dir selection deterministic https://review.openstack.org/70685 | 07:51 |
| *** VijayT has quit IRC | 07:56 | |
| *** rlandy has joined #openstack-infra | 07:58 | |
| *** mkoderer has quit IRC | 07:59 | |
| *** denis_makogon_ has joined #openstack-infra | 07:59 | |
| *** afazekas has joined #openstack-infra | 08:00 | |
| *** mkoderer has joined #openstack-infra | 08:00 | |
| *** denis_makogon has quit IRC | 08:02 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Publish link to docs-drafts for successful builds https://review.openstack.org/73185 | 08:06 |
| *** cyeoh has quit IRC | 08:12 | |
| *** cyeoh has joined #openstack-infra | 08:12 | |
| *** fifieldt_ has quit IRC | 08:13 | |
| *** locke105 has quit IRC | 08:13 | |
| *** bodepd has quit IRC | 08:13 | |
| *** SpamapS has quit IRC | 08:14 | |
| *** greghaynes has quit IRC | 08:14 | |
| *** bodepd has joined #openstack-infra | 08:14 | |
| *** fifieldt_ has joined #openstack-infra | 08:14 | |
| *** locke105 has joined #openstack-infra | 08:15 | |
| *** SpamapS has joined #openstack-infra | 08:15 | |
| *** SpamapS has quit IRC | 08:15 | |
| *** SpamapS has joined #openstack-infra | 08:15 | |
| *** openstackgerrit has quit IRC | 08:16 | |
| *** persia has quit IRC | 08:16 | |
| *** branen_ has quit IRC | 08:16 | |
| *** branen_ has joined #openstack-infra | 08:17 | |
| *** leifmadsen has quit IRC | 08:17 | |
| *** lifeless has quit IRC | 08:17 | |
| *** persia has joined #openstack-infra | 08:18 | |
| *** persia is now known as Guest43781 | 08:18 | |
| *** lifeless has joined #openstack-infra | 08:19 | |
| *** Guest43781 has quit IRC | 08:20 | |
| *** Guest43781 has joined #openstack-infra | 08:20 | |
| *** Guest43781 is now known as persia | 08:20 | |
| *** orionNebula has quit IRC | 08:20 | |
| *** orionNebula has joined #openstack-infra | 08:21 | |
| *** sarob has joined #openstack-infra | 08:22 | |
| *** mattymo has joined #openstack-infra | 08:23 | |
| *** mrmartin has joined #openstack-infra | 08:23 | |
| *** che-arne has joined #openstack-infra | 08:24 | |
| AJaeger | Is anybody around to tell me why Zuul/Jenkins are still using for openstack-operations-guide-ja the old job? This was changed 12 hours ago | 08:24 |
| *** cp16net has quit IRC | 08:25 | |
| *** cp16net has joined #openstack-infra | 08:25 | |
| *** leifmadsen has joined #openstack-infra | 08:27 | |
| *** sarob has quit IRC | 08:28 | |
| *** mattymo has quit IRC | 08:28 | |
| *** DeeJay1 has joined #openstack-infra | 08:32 | |
| *** jcoufal has joined #openstack-infra | 08:35 | |
| *** shardy_afk is now known as shardy | 08:35 | |
| *** jpich has joined #openstack-infra | 08:36 | |
| *** bogdando has quit IRC | 08:38 | |
| *** bogdando has joined #openstack-infra | 08:39 | |
| *** dizquierdo has joined #openstack-infra | 08:44 | |
| *** saju_m has joined #openstack-infra | 08:50 | |
| *** fifieldt_ has quit IRC | 08:57 | |
| *** hashar has joined #openstack-infra | 09:01 | |
| *** llu has joined #openstack-infra | 09:03 | |
| *** llu has left #openstack-infra | 09:03 | |
| *** jgallard has joined #openstack-infra | 09:03 | |
| *** derekh has joined #openstack-infra | 09:09 | |
| *** yassine has joined #openstack-infra | 09:11 | |
| *** mkerrin1 has quit IRC | 09:13 | |
| *** dkliban has quit IRC | 09:21 | |
| *** sarob has joined #openstack-infra | 09:25 | |
| *** rossella_s has joined #openstack-infra | 09:25 | |
| *** markmc has joined #openstack-infra | 09:26 | |
| *** denis_makogon_ has quit IRC | 09:28 | |
| *** CaptTofu has joined #openstack-infra | 09:28 | |
| *** sneelakantan has joined #openstack-infra | 09:30 | |
| *** DinaBelova_ is now known as DinaBelova | 09:32 | |
| *** CaptTofu has quit IRC | 09:33 | |
| *** IvanBerezovskiy has joined #openstack-infra | 09:36 | |
| *** jooools has joined #openstack-infra | 09:47 | |
| *** jishaom has quit IRC | 09:56 | |
| SergeyLukjanov | AJaeger, hey, looking on it | 09:56 |
| AJaeger | SergeyLukjanov, thanks! | 09:56 |
| AJaeger | here's the logfile https://jenkins01.openstack.org/job/openstack-operations-guide-ja/81/ | 09:56 |
| *** sarob has quit IRC | 09:57 | |
| AJaeger | And this is the patch you approved last night: https://review.openstack.org/#/c/71056/ | 09:57 |
| AJaeger | I see updates for the other patch I pinged active - just not this one. | 09:58 |
| * AJaeger is completely puzzled | 09:58 | |
| SergeyLukjanov | AJaeger, hm zuul Last reconfigured: Thu Feb 13 2014 02:39:08 GMT+0400 (VOLT) | 09:59 |
| AJaeger | so, that looks fine | 10:00 |
| AJaeger | it wsa approved 4 hours or so before | 10:00 |
| SergeyLukjanov | AJaeger, oh, it's not about zuul, you only changed corresponding job | 10:00 |
| AJaeger | SergeyLukjanov, you're right. So, this is jenkins problem? | 10:00 |
| AJaeger | Sorry for misleading you | 10:00 |
| SergeyLukjanov | AJaeger, np, it looks like job hasn't been updated | 10:02 |
| SergeyLukjanov | AJaeger, thinking about how to check it | 10:02 |
| *** bookwar has quit IRC | 10:03 | |
| *** bookwar has joined #openstack-infra | 10:03 | |
| *** mattymo has joined #openstack-infra | 10:03 | |
| mattymo | I noticed that in the last 8 hours openstackgerrit died | 10:03 |
| mattymo | anyone monitoring it? maybe our dearest SergeyLukjanov ? | 10:03 |
| mrmartin | SergeyLukjanov: hi, I have the same problem here: http://logs.openstack.org/94/72294/2/check/gate-groups-unittests/a20b42b/console.html#_2014-02-13_08_46_04_530 | 10:03 |
| mrmartin | I made same modification yesterday on related gating job, but the previous one was executed. | 10:04 |
| SergeyLukjanov | mattymo, hey, I have no root access to this server to reboot it | 10:04 |
| mattymo | SergeyLukjanov, how about some puppet triggers :) | 10:05 |
| SergeyLukjanov | mrmartin, could you please share the link to your config change? | 10:06 |
| mrmartin | of course: https://review.openstack.org/#/c/68912/ | 10:06 |
| AJaeger | SergeyLukjanov, my issue is not urgent that it needs fixing *now*, if you want to wait for others to show up, no problem. Just wanted to bring it up here... | 10:08 |
| SergeyLukjanov | AJaeger, ok | 10:09 |
| *** sneelakantan has left #openstack-infra | 10:13 | |
| *** greghaynes has joined #openstack-infra | 10:14 | |
| SergeyLukjanov | AJaeger, the only idea is that jenkins job update wasn't applied on the moment of job execution, but it's really strange, I have no access to check the job correctness | 10:15 |
| SergeyLukjanov | AJaeger, but probably, you could try to submit another change to check how it works now | 10:16 |
| AJaeger | SergeyLukjanov, my job was submitted two hours ago which should have been enough time. The first job after the merge was after 40 mins and also failed | 10:16 |
| SergeyLukjanov | AJaeger, oh | 10:17 |
| *** sarob has joined #openstack-infra | 10:22 | |
| SergeyLukjanov | AJaeger, I confirm that "./tools/generatedocbook -l ja -b openstack-ops" is still in job "openstack-operations-guide*" on all jenkins servers :( | 10:23 |
| AJaeger | SergeyLukjanov, so no need to test - this is wrong. | 10:24 |
| SergeyLukjanov | AJaeger, yup | 10:24 |
| AJaeger | SergeyLukjanov, thanks for checking! | 10:24 |
| SergeyLukjanov | AJaeger, I have no access to retrigger jjb but I'll think about how to workaround it :) | 10:24 |
| SergeyLukjanov | mrmartin, looking on your issue now | 10:24 |
| mrmartin | thanks | 10:25 |
| *** CaptTofu has joined #openstack-infra | 10:25 | |
| AJaeger | SergeyLukjanov, I'm still fine to wait for other that teach you how to do it properly later ;) | 10:26 |
| SergeyLukjanov | AJaeger, I think that it requires just puppet-agent --test but I have no root access :) | 10:26 |
| AJaeger | SergeyLukjanov, neither have I ;) | 10:26 |
| SergeyLukjanov | AJaeger, heh, looks like I need to ask access to some servers to be able to repair some stuff like this | 10:27 |
| *** jp_at_hp has joined #openstack-infra | 10:27 | |
| AJaeger | SergeyLukjanov, you're still new in the job - so I expect some more of these missing permissions ;( | 10:28 |
| *** che-arne has quit IRC | 10:29 | |
| SergeyLukjanov | AJaeger, yeah, but I'm not the admin guy, so, I'd like to keep amount of servers that I have access to as small as possible :) | 10:30 |
| AJaeger | Ah, ok. | 10:30 |
| SergeyLukjanov | AJaeger, heh, don't want to break the world while everyone else in team sleeping ;) | 10:31 |
| AJaeger | If you rather review, here's one think I noticed that could help my last two changes: https://review.openstack.org/73185 | 10:31 |
| SergeyLukjanov | AJaeger, anyway, I hope we'll find the solution to grant me enough permissions to help :) | 10:31 |
| AJaeger | SergeyLukjanov, I thought the expectation is that you fix all the problems of the world while the rest of the team is sleeping ;) | 10:32 |
| mrmartin | I think we need to wait until the US infra team wake up. | 10:33 |
| SergeyLukjanov | mrmartin, I confirm that you have correct jobs on all jenkins servers now for groups, both unittests and release | 10:33 |
| *** e0ne has quit IRC | 10:33 | |
| mrmartin | SergeyLukjanov, had you made any change, or just checked the files? | 10:33 |
| *** andreaf has joined #openstack-infra | 10:33 | |
| *** mrfatstrat has joined #openstack-infra | 10:34 | |
| SergeyLukjanov | mrmartin, I've check it directly in jenkins | 10:34 |
| mrmartin | so it is a mysterious bug | 10:34 |
| SergeyLukjanov | mrmartin, oh, it looks like you're calling apt-get update under the bootstrap.sh | 10:36 |
| mrmartin | I now, but it was the previous script, and yesterday's patch eliminated the bootstrap.sh call :) | 10:37 |
| SergeyLukjanov | mrmartin, and it fails due to permission denied | 10:37 |
| SergeyLukjanov | mrmartin, oops, I was looking on https://review.openstack.org/#/c/68912/5/modules/openstack_project/files/jenkins_job_builder/config/groups.yaml :( | 10:37 |
| mrmartin | my question is why the bootstrap.sh was called, when we removed it | 10:37 |
| SergeyLukjanov | mrmartin, it means that jenkins jobs configs wasn't updated for some time | 10:38 |
| SergeyLukjanov | mrmartin, but not for your case, https://review.openstack.org/#/c/72921/2 isn't yet merged :) | 10:39 |
| mrmartin | ok, Sorry | 10:39 |
| mrmartin | https://review.openstack.org/#/c/72921/ | 10:39 |
| SergeyLukjanov | mrmartin, so, you should wait until it'll be merged and re-check after that | 10:39 |
| SergeyLukjanov | I'm adding it to my baklog | 10:40 |
| SergeyLukjanov | backlog* | 10:40 |
| mrmartin | I see now, I thought it was merged yesterday, I just see that jeblair just review it :) | 10:40 |
| SergeyLukjanov | AJaeger, anyway, your changes to jobs wasn't applied and it's bas | 10:40 |
| mrmartin | oh thanks | 10:40 |
| SergeyLukjanov | bad* | 10:40 |
| AJaeger | SergeyLukjanov, yeah, agreed. | 10:41 |
| *** mkerrin has joined #openstack-infra | 10:42 | |
| *** sarob has quit IRC | 10:55 | |
| *** UtahDave has quit IRC | 10:57 | |
| *** hashar is now known as hasharCodingDojo | 11:01 | |
| *** ArxCruz has joined #openstack-infra | 11:02 | |
| *** yaguang has quit IRC | 11:03 | |
| *** w_ has joined #openstack-infra | 11:08 | |
| *** jgallard has quit IRC | 11:10 | |
| *** olaph has quit IRC | 11:10 | |
| *** CaptTofu has quit IRC | 11:13 | |
| *** hasharCodingDojo has quit IRC | 11:21 | |
| *** sarob has joined #openstack-infra | 11:22 | |
| *** jooools has quit IRC | 11:24 | |
| *** jooools has joined #openstack-infra | 11:24 | |
| *** ArxCruz has quit IRC | 11:29 | |
| *** notmyname has quit IRC | 11:29 | |
| *** ArxCruz has joined #openstack-infra | 11:30 | |
| *** CaptTofu has joined #openstack-infra | 11:31 | |
| *** pleia2 has quit IRC | 11:31 | |
| *** dragondm has quit IRC | 11:31 | |
| *** dhellmann has quit IRC | 11:31 | |
| *** pleia2 has joined #openstack-infra | 11:32 | |
| *** dhellmann has joined #openstack-infra | 11:32 | |
| *** notmyname has joined #openstack-infra | 11:32 | |
| *** sileht has quit IRC | 11:32 | |
| *** dkehn_ has joined #openstack-infra | 11:32 | |
| *** dragondm has joined #openstack-infra | 11:32 | |
| *** antonym has quit IRC | 11:32 | |
| *** apevec has joined #openstack-infra | 11:33 | |
| *** krtaylor has quit IRC | 11:33 | |
| *** sileht has joined #openstack-infra | 11:33 | |
| apevec | Daviey, ping | 11:33 |
| *** tteggel has joined #openstack-infra | 11:33 | |
| *** tteggel_ has joined #openstack-infra | 11:33 | |
| *** sc68cal has quit IRC | 11:34 | |
| apevec | Daviey, last time adam_g reported Ubuntu CI results running against stable branch, I'd like to see those now | 11:34 |
| *** tteggel_ has quit IRC | 11:34 | |
| apevec | last patch exception was merged to neutron havana, and I'm ready to tag | 11:34 |
| *** dkehn_ has quit IRC | 11:35 | |
| *** tteggel_ has joined #openstack-infra | 11:35 | |
| *** dkehn__ has quit IRC | 11:35 | |
| *** dkehn has quit IRC | 11:35 | |
| *** dkehn_ has joined #openstack-infra | 11:35 | |
| *** sc68cal has joined #openstack-infra | 11:37 | |
| *** dkehn has joined #openstack-infra | 11:37 | |
| *** krtaylor has joined #openstack-infra | 11:38 | |
| *** tteggel_ has quit IRC | 11:40 | |
| *** tteggel_ has joined #openstack-infra | 11:41 | |
| *** tteggel_ has joined #openstack-infra | 11:42 | |
| *** tteggel_ has quit IRC | 11:42 | |
| *** tteggel has quit IRC | 11:42 | |
| *** tteggel has joined #openstack-infra | 11:42 | |
| *** mrmartin has quit IRC | 11:43 | |
| *** e0ne has joined #openstack-infra | 11:44 | |
| *** e0ne has quit IRC | 11:48 | |
| *** e0ne has joined #openstack-infra | 11:51 | |
| *** aude has quit IRC | 11:55 | |
| *** sarob has quit IRC | 11:55 | |
| *** mrfatstrat has quit IRC | 11:57 | |
| Daviey | apevec: Ah, good call. I presume Adam hasn't offered the results already? | 11:57 |
| apevec | no, he just mentioned on irc last time, but w/o public url | 11:58 |
| apevec | is there any? | 11:58 |
| Daviey | Good question... | 11:58 |
| Daviey | (looking) | 11:58 |
| *** vkozhukalov has joined #openstack-infra | 11:59 | |
| *** dstanek has joined #openstack-infra | 12:00 | |
| *** aude has joined #openstack-infra | 12:04 | |
| *** ociuhandu has quit IRC | 12:10 | |
| *** jgallard has joined #openstack-infra | 12:16 | |
| *** jcooley has quit IRC | 12:17 | |
| *** EntropyWorks has quit IRC | 12:18 | |
| *** mrmartin has joined #openstack-infra | 12:20 | |
| *** jcooley has joined #openstack-infra | 12:20 | |
| *** yamahata has quit IRC | 12:22 | |
| *** EntropyWorks has joined #openstack-infra | 12:22 | |
| *** sarob has joined #openstack-infra | 12:22 | |
| *** rfolco has joined #openstack-infra | 12:23 | |
| *** talluri has joined #openstack-infra | 12:26 | |
| *** hashar has joined #openstack-infra | 12:28 | |
| *** CaptTofu has quit IRC | 12:39 | |
| *** jooools has quit IRC | 12:40 | |
| *** jooools has joined #openstack-infra | 12:41 | |
| *** jhesketh has quit IRC | 12:42 | |
| isviridov | Hi infra team. Hi SergeyLukjanov. I would like to ask about https://review.openstack.org/#/c/71308/, should i do anything to make it merged? | 12:42 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 12:43 | |
| isviridov | SergeyLukjanov, around? | 12:48 |
| dolphm | can anyone push this one over the edge to gate keystoneclient on py33? https://review.openstack.org/#/c/72831/ | 12:48 |
| dolphm | i'd like to cut a release gated against py33 | 12:49 |
| *** xBsd has joined #openstack-infra | 12:52 | |
| *** sarob has quit IRC | 12:55 | |
| *** smarcet has joined #openstack-infra | 12:59 | |
| *** dstanek has quit IRC | 13:01 | |
| SergeyLukjanov | dolphm, sure, I'll do it | 13:02 |
| *** b3nt_pin is now known as beagles | 13:02 | |
| dolphm | SergeyLukjanov: thanks! | 13:02 |
| SergeyLukjanov | isviridov, hey, I'll take a look | 13:02 |
| *** xBsd has quit IRC | 13:02 | |
| SergeyLukjanov | dolphm, np, done | 13:03 |
| SergeyLukjanov | isviridov, we currently have some problems with manage-projects script that is used for new projects creation, so, you should wait for the fix | 13:03 |
| *** jcoufal has quit IRC | 13:04 | |
| SergeyLukjanov | isviridov, no estimates, here is an issue - https://bugs.launchpad.net/openstack-ci/+bug/1242569 | 13:05 |
| *** pdmars has joined #openstack-infra | 13:05 | |
| *** jcoufal has joined #openstack-infra | 13:05 | |
| isviridov | SergeyLukjanov, thx for update | 13:05 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 13:05 | |
| *** jhesketh_ has quit IRC | 13:07 | |
| *** nosnos has quit IRC | 13:10 | |
| *** dprince has joined #openstack-infra | 13:10 | |
| *** heyongli has joined #openstack-infra | 13:12 | |
| *** zul has joined #openstack-infra | 13:14 | |
| SergeyLukjanov | fungi, clarkb, jeblair, short update - looks like gerrit bot is down, job configs from https://review.openstack.org/#/c/71056/ doesn't applied to jenkins jobs :( | 13:20 |
| *** senthil has joined #openstack-infra | 13:21 | |
| *** sarob has joined #openstack-infra | 13:22 | |
| *** esker has quit IRC | 13:23 | |
| *** esker has joined #openstack-infra | 13:23 | |
| *** lcostantino has joined #openstack-infra | 13:25 | |
| *** dcramer_ has quit IRC | 13:27 | |
| *** dstanek has joined #openstack-infra | 13:27 | |
| *** mrmartin has quit IRC | 13:28 | |
| *** esker has quit IRC | 13:28 | |
| ewindisch | I'm inclined to remove this section from 3rd-party testing: http://ci.openstack.org/third_party.html#the-jenkins-gerrit-trigger-plugin-way | 13:28 |
| ewindisch | the jenkins-gerrit-trigger doesn't support rechecks. | 13:29 |
| ewindisch | unless the 'support rechecks' requirement isn't (a requirement). | 13:30 |
| *** mrmartin has joined #openstack-infra | 13:30 | |
| *** antonym has joined #openstack-infra | 13:31 | |
| *** jamespage has joined #openstack-infra | 13:32 | |
| jamespage | apevec, hi - you where after adam_g to run testing against stable branch packages right? | 13:33 |
| *** daniil has joined #openstack-infra | 13:38 | |
| *** w_ is now known as olaph | 13:43 | |
| *** prad has joined #openstack-infra | 13:44 | |
| *** oubiwann_ has joined #openstack-infra | 13:46 | |
| *** yamahata has joined #openstack-infra | 13:49 | |
| *** prad_ has joined #openstack-infra | 13:50 | |
| *** prad has quit IRC | 13:51 | |
| *** prad_ is now known as prad | 13:51 | |
| *** mrfatstrat has joined #openstack-infra | 13:52 | |
| *** che-arne has joined #openstack-infra | 13:52 | |
| *** markwash has joined #openstack-infra | 13:55 | |
| *** oanufriev has joined #openstack-infra | 13:55 | |
| *** sarob has quit IRC | 13:55 | |
| *** gokrokve has joined #openstack-infra | 13:57 | |
| *** heyongli has quit IRC | 14:03 | |
| *** che-arne has quit IRC | 14:06 | |
| *** dkranz has joined #openstack-infra | 14:11 | |
| *** mrmartin has quit IRC | 14:13 | |
| *** mrmartin has joined #openstack-infra | 14:13 | |
| *** ociuhandu has joined #openstack-infra | 14:14 | |
| *** oubiwann_ has quit IRC | 14:16 | |
| *** nkinder has quit IRC | 14:17 | |
| *** sarob has joined #openstack-infra | 14:22 | |
| *** saju_m has quit IRC | 14:28 | |
| *** ok_delta has joined #openstack-infra | 14:30 | |
| *** esker has joined #openstack-infra | 14:34 | |
| *** esker has quit IRC | 14:34 | |
| *** esker has joined #openstack-infra | 14:35 | |
| *** max_lobur is now known as max_lobur_afk | 14:36 | |
| *** homeless_ has quit IRC | 14:38 | |
| *** homeless has joined #openstack-infra | 14:38 | |
| *** dims has quit IRC | 14:38 | |
| *** CaptTofu has joined #openstack-infra | 14:40 | |
| *** fifieldt has joined #openstack-infra | 14:40 | |
| *** sarob has quit IRC | 14:40 | |
| *** jhesketh has joined #openstack-infra | 14:41 | |
| sdague | gerrit bot dead? | 14:41 |
| anteaya | according to SergeyLukjanov, yes | 14:42 |
| *** fifieldt has quit IRC | 14:43 | |
| *** dcramer_ has joined #openstack-infra | 14:43 | |
| SergeyLukjanov | sdague, anteaya, yup | 14:43 |
| sdague | ok, cool | 14:44 |
| sdague | on the up side, git review was extra fast for me this morning (faster than I can remember it being ever) | 14:44 |
| sdague | so something got better on the gerrit side | 14:44 |
| SergeyLukjanov | sdague, we could upload/merge some dirty stuff ;) | 14:44 |
| *** CaptTofu has quit IRC | 14:44 | |
| *** olaph has quit IRC | 14:45 | |
| anteaya | SergeyLukjanov: bad code would be removed | 14:45 |
| anteaya | poor taste would be shamed | 14:45 |
| *** ryanpetrello has joined #openstack-infra | 14:45 | |
| SergeyLukjanov | anteaya, ;( | 14:45 |
| anteaya | and good morning | 14:47 |
| *** oubiwann_ has joined #openstack-infra | 14:47 | |
| *** oubiwann_ has quit IRC | 14:48 | |
| *** oubiwann_ has joined #openstack-infra | 14:48 | |
| SergeyLukjanov | yeah, morning ;) | 14:48 |
| *** oubiwann_ has quit IRC | 14:49 | |
| *** mrfatstrat has quit IRC | 14:50 | |
| *** oubiwann_ has joined #openstack-infra | 14:50 | |
| *** dims has joined #openstack-infra | 14:51 | |
| *** wenlock has joined #openstack-infra | 14:52 | |
| ttx | fungi: are we still using tools/atc/email-stats.sh to generate ATC lists ? Or are you consuming governance/programs.yaml instead ? | 14:53 |
| SergeyLukjanov | ttx, does governance/programs contain all the repos from email-stats? | 14:57 |
| SergeyLukjanov | ttx, and non-code atcs? | 14:57 |
| ttx | email-stats is not the reference of anything. | 14:57 |
| SergeyLukjanov | oh, rollback, it does | 14:57 |
| SergeyLukjanov | ttx, ok | 14:57 |
| ttx | it duplicates information, so it should die | 14:58 |
| ttx | and just read info from where it lives | 14:58 |
| SergeyLukjanov | ttx, agreed | 14:58 |
| *** olaph has joined #openstack-infra | 14:58 | |
| *** AlexF has joined #openstack-infra | 15:00 | |
| apevec | jamespage, yes, I'd like to get ack from adam_g before pushing tags | 15:01 |
| apevec | jamespage, do you have CI running against latest stable/havana? | 15:01 |
| jamespage | apevec, just kicked off a run now | 15:01 |
| apevec | thanks! | 15:02 |
| jamespage | (needed to fixup a few package build failures first - now done) | 15:02 |
| mrmartin | SergeyLukjanov, this patch 72294,3 was posted into check more than 1 hour ago and the gate-groups-unittests job is queued. Could we check somewhere what happens? | 15:02 |
| *** nkinder has joined #openstack-infra | 15:03 | |
| anteaya | apevec: you understand the purpose for my comment on 72754? | 15:03 |
| *** mfer has joined #openstack-infra | 15:03 | |
| apevec | anteaya, "This patch has failed in the gate 4 times in a row." - not quite :) | 15:04 |
| anteaya | would you like to know? | 15:04 |
| apevec | yes please | 15:04 |
| anteaya | I was letting you know I was watching the patch | 15:04 |
| apevec | btw it merged, looks like stable gate likes EU morning | 15:04 |
| anteaya | and that it had failed in the gate 4 times in a row | 15:04 |
| SergeyLukjanov | mrmartin, it's misconfigured now, looks like by the latest patch | 15:04 |
| dims | logstash seems to be down | 15:04 |
| apevec | anteaya, but poor patch is innocent :) | 15:04 |
| anteaya | and that if it would be sent back in and block other patches, I would remove it | 15:05 |
| anteaya | apevec: I think we have a failure of understanding between us | 15:05 |
| fungi | SergeyLukjanov: i'll get gerritbot restarted--this is not an uncommon occurrence (it has a tendency not to recover from netslipts due to a bug in the underlying framework) | 15:05 |
| anteaya | all code is innocent | 15:05 |
| *** oanufriev has quit IRC | 15:05 | |
| apevec | anteaya, ihrachys is now looking what Tempest patches should be backported | 15:05 |
| *** markwash has quit IRC | 15:05 | |
| anteaya | all is made up of ones and zeros which alone and by themselves are innocent | 15:05 |
| anteaya | it is only in combination that problems occur | 15:05 |
| fungi | ttx: yes, using that but i modified it to parse the governance repo. i'll see if i forgot to submit a review with that improvement | 15:05 |
| SergeyLukjanov | fungi, morning, great | 15:06 |
| ttx | fungi: cool. Just want people to stop proposig patch to that list as if it was the canonical list | 15:06 |
| anteaya | apevec: great, well that should improve stablility for testing for stable/havana | 15:06 |
| mrmartin | SergeyLukjanov, what is misconfigured? | 15:06 |
| anteaya | and decrease failures in the gate, decreasing my need to remove them | 15:06 |
| SergeyLukjanov | mrmartin, looks like we've missed that https://github.com/openstack-infra/config/commit/15761d181a8110afa1cfdcb15922de88bcb047d8 going to use {node} but it's not set for project | 15:07 |
| anteaya | apevec: do you have a better sense of my motivation for my actions? | 15:07 |
| mrmartin | SergeyLukjanov, ok, what we need to change? | 15:07 |
| apevec | anteaya, also separate gate queue for stable would help, since master and stable and not equal re. stability | 15:07 |
| *** mgagne has quit IRC | 15:07 | |
| anteaya | that is an interesting idea | 15:07 |
| apevec | anteaya, I understand your actions completely | 15:07 |
| SergeyLukjanov | mrmartin, looking on it atm | 15:07 |
| anteaya | a separate gate queue for stable | 15:08 |
| anteaya | apevec: great, thanks I am glad we are there | 15:08 |
| *** wenlock has quit IRC | 15:08 | |
| anteaya | apevec: if you would like to propose a separate gate for stable, I would encourage you to add an item to the infra meeting agenda so that it can be discussed | 15:08 |
| mrmartin | SergeyLukjanov, I cloned those parts from storyboard-webclient, seems to me something was missed | 15:08 |
| mtreinish | anteaya: we can't do a separate gate queue for stable because stable has implications for grenade tests | 15:08 |
| mtreinish | it needs to be integrated with everything because of that | 15:09 |
| anteaya | mtreinish: thank you | 15:09 |
| anteaya | ah | 15:09 |
| mtreinish | otherwise we'll wedge | 15:09 |
| *** nicedice has joined #openstack-infra | 15:09 | |
| anteaya | no weding | 15:09 |
| SergeyLukjanov | mrmartin, I see in job config label: {node} | 15:09 |
| anteaya | wedging even | 15:09 |
| SergeyLukjanov | mrmartin, I mean I see it in jenkins, so, {node} should be replaced with actual value | 15:09 |
| apevec | anteaya, iirc fungi & co entertained the idea already, I'll add as an item (I'm not sure what it would take) | 15:09 |
| anteaya | neither do I | 15:10 |
| anteaya | hopefully mtreinish will also attend and bring his perspective to the discussion | 15:10 |
| mtreinish | anteaya: I'm normally around during the infra meeting | 15:11 |
| anteaya | mtreinish: great | 15:11 |
| anteaya | should make for a good discussion | 15:11 |
| SergeyLukjanov | mrmartin, IMO it should be taken from the node definition here - https://review.openstack.org/#/c/72921/2/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml | 15:11 |
| fungi | apevec: the problem is that stable can't be gated separately, because we have upgrade testing | 15:11 |
| mrmartin | yeap, so, do I need to change anything? I thought the same. | 15:12 |
| apevec | fungi, why is that a problem? Master tests h->i, stable/havana g->h ? | 15:13 |
| SergeyLukjanov | mrmartin, I'm trying to understand why it's not working, no specific ideas atm | 15:13 |
| *** AlexF has quit IRC | 15:13 | |
| mrmartin | SergeyLukjanov, ok thanks | 15:13 |
| fungi | apevec: we test that patches to stable don't prevent upgrading to the next release, and that patches to the next release don't prevent you from being unable to upgrade from stable | 15:13 |
| SergeyLukjanov | fungi, could you please clarify that in this case https://review.openstack.org/#/c/72921/2/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml "node=precise" will be passed to gate-groups-unittests job template? | 15:14 |
| *** oanufriev has joined #openstack-infra | 15:14 | |
| fungi | apevec: but regardless, separatin the gate pipelines for stable and master is just a workaround. the proper solution is to get enough people who care about stable to backport fixes to it and keep it in shape. with just a handful of us it's fairly insurmountable | 15:14 |
| apevec | true that | 15:15 |
| fungi | but i'm afraid unless we just declare it bankrupt, not many other entities are likely to step up and help | 15:15 |
| apevec | yeah, I tried making noise on the lists, I'll keep trying :) | 15:15 |
| fungi | SergeyLukjanov: having a look | 15:16 |
| *** jnoller has joined #openstack-infra | 15:16 | |
| anteaya | fungi apevec right now how many people work on stable maintenance? | 15:16 |
| SergeyLukjanov | fungi, thx | 15:16 |
| apevec | anteaya, in theory it's stable-maint group https://review.openstack.org/#/admin/groups/120,members | 15:16 |
| *** jnoller_ has joined #openstack-infra | 15:16 | |
| fungi | anteaya: depends on how you count them | 15:16 |
| *** jooools has quit IRC | 15:16 | |
| *** mfer has quit IRC | 15:16 | |
| *** jooools1 has joined #openstack-infra | 15:16 | |
| apevec | but not all are active | 15:16 |
| *** jnoller_ has quit IRC | 15:16 | |
| apevec | and we have active backporters who aren't core | 15:17 |
| *** jnoller_ has joined #openstack-infra | 15:17 | |
| *** CaptTofu has joined #openstack-infra | 15:17 | |
| anteaya | fungi: how do you count them? | 15:17 |
| apevec | also stable-maint doesn't have +2 on Tempest | 15:17 |
| *** jnoller_ has quit IRC | 15:17 | |
| apevec | stable/* branches of Tempest that is | 15:17 |
| anteaya | apevec: so you need someone who has +2 on Tempest to be in the group? | 15:18 |
| anteaya | would that help? | 15:18 |
| *** jergerber has joined #openstack-infra | 15:18 | |
| apevec | I hope so | 15:18 |
| mtreinish | apevec, anteaya: the tempest core team has +2 on tempest stable | 15:18 |
| *** banix has joined #openstack-infra | 15:18 | |
| mtreinish | when we went to do our first backport no one had permission to do it :) | 15:18 |
| apevec | mtreinish, would you like to join stable-maint ? | 15:18 |
| apevec | we need some intersection | 15:18 |
| anteaya | would seem to fill a need for both | 15:18 |
| mtreinish | I knew I shouldn't have said anything... | 15:19 |
| mtreinish | apevec: sure I guess | 15:19 |
| *** jnoller_ has joined #openstack-infra | 15:19 | |
| *** jnoller has quit IRC | 15:19 | |
| anteaya | apevec: how does he join? | 15:19 |
| apevec | for all other projects each PTL is stable-maint member by default | 15:19 |
| anteaya | ah | 15:19 |
| apevec | anteaya, I'll fast track him right now :) | 15:19 |
| fungi | anteaya: i think he just did ;) | 15:19 |
| anteaya | hence the length of the list | 15:19 |
| anteaya | apevec fungi well it helps | 15:19 |
| anteaya | I seen neither jeblair or fungi on the stable-maint list | 15:20 |
| fungi | much like core security reviewers, each project ideally would have at least a couple people taking care to propose or at least shepherd in backports and watch the nightly bitrot job results and so on | 15:20 |
| anteaya | thought you were on it, fungi | 15:21 |
| fungi | anteaya: i don't want to be in the position of approving changes to stable branches | 15:21 |
| anteaya | ah I am thinking of core security | 15:21 |
| anteaya | fair enough | 15:21 |
| anteaya | apevec: who is active for stable-maint | 15:21 |
| fungi | i mostly notice stable breakage when trying to get security fixes tested | 15:21 |
| anteaya | ah | 15:21 |
| *** mkerrin has quit IRC | 15:21 | |
| apevec | ttx, ^^^ do you have your stats for stable-maint? | 15:21 |
| fungi | which is fairly often, and then i do my best to track down and fix whatever's gone awry | 15:21 |
| anteaya | fungi: ah so you offer fixes but are not core | 15:22 |
| apevec | ttx, it's perhaps time to review membership | 15:22 |
| apevec | or we do it at I release? | 15:22 |
| *** sarob has joined #openstack-infra | 15:22 | |
| SergeyLukjanov | fungi, additionally, https://review.openstack.org/#/c/71056/ still wasn't applied (>15h) due to the https://jenkins04.openstack.org/view/All/job/openstack-operations-guide-ja/configure | 15:22 |
| sdague | apevec: so we could add stable-maint to approve on tempest stable branches | 15:22 |
| sdague | the reality is that when we did our first backport, the stable-maint team at the time had no interest in being part of that group | 15:23 |
| ttx | apevec: well, the trick is the original script isn't stable/* compatible | 15:23 |
| ttx | I remember tweaking it though, let me see if I kept that around | 15:24 |
| *** prad has quit IRC | 15:25 | |
| *** ArxCruz has quit IRC | 15:25 | |
| *** prad has joined #openstack-infra | 15:25 | |
| *** AlexF has joined #openstack-infra | 15:25 | |
| dhellmann | sdague: do you know anything about the history of https://bugs.launchpad.net/devstack/+bug/1260723 ? I seem to be hitting it even after a fix has merged. | 15:25 |
| apevec | sdague, looks like that need to change and stable-maint should learn Tempest better | 15:25 |
| *** zhiyan has joined #openstack-infra | 15:25 | |
| apevec | sdague, I'd like we have intersection, so I've added mtreinish to stable-maint | 15:26 |
| apevec | mtreinish, welcome to stable-maint https://review.openstack.org/#/admin/groups/120,members ! | 15:26 |
| sdague | sure | 15:26 |
| mtreinish | apevec: thanks | 15:26 |
| anteaya | yay | 15:26 |
| *** zhiyan is now known as zhiyan_ | 15:26 | |
| anteaya | dhellmann: you are on the stable-maint list | 15:27 |
| dhellmann | anteaya: yes | 15:27 |
| fungi | anteaya: but yeah, i'm a release cycle management project core team member by way of being a vulnerability management team member, though we have some separation-of-concerns in place to keep release management and vulnerability management and stable branch management siloed, aside from our illustrious ptl of course | 15:27 |
| anteaya | dhellmann: is this an accurate reflection of your interest? | 15:27 |
| *** sarob has quit IRC | 15:27 | |
| *** ArxCruz has joined #openstack-infra | 15:27 | |
| fungi | s/project/program/ | 15:27 |
| dhellmann | anteaya: I am interested in helping, yes | 15:27 |
| anteaya | fungi: fair enough | 15:27 |
| anteaya | dhellmann: great, how do I support that? | 15:27 |
| sdague | dhellmann: yes, though that's not an actual failure | 15:27 |
| anteaya | since apevec is needing some stable-maint help | 15:28 |
| sdague | sahid was working on that yesterday | 15:28 |
| dhellmann | anteaya: I'm not sure I understand the question | 15:28 |
| dhellmann | sdague: http://logs.openstack.org/37/72437/1/check/check-devstack-dsvm-cells/6272c5f/console.html | 15:28 |
| apevec | anteaya, ttx will do periodic membership cleanups (was already done in the past) | 15:28 |
| apevec | based on (in)activity | 15:28 |
| dhellmann | sdague: am I misinterpreting that log? | 15:28 |
| apevec | as in reviews | 15:28 |
| anteaya | dhellmann: apevec is feeling that stable-maint is not getting enough attention, and slow backports are affecting release timing | 15:28 |
| sdague | dhellmann: I think so | 15:28 |
| sdague | the actually failure is a server 500 on volumes create | 15:29 |
| sdague | http://logs.openstack.org/37/72437/1/check/check-devstack-dsvm-cells/6272c5f/console.html#_2014-02-13_13_49_57_793 | 15:29 |
| ttx | apevec: will pastebin in a few | 15:29 |
| *** chandan_kumar has quit IRC | 15:29 | |
| *** chandankumar_ has joined #openstack-infra | 15:29 | |
| anteaya | dhellmann: I am helping to acertain who can help to improve the speed of stable-maint backports to help releases | 15:29 |
| dhellmann | sdague: ok, I thought that might have been caused by the credentials thing | 15:29 |
| dhellmann | sdague: I'll look for another bug to recheck against | 15:29 |
| anteaya | dhellmann: so I am asking people on the list if their presence is an accurate reflection of interest | 15:29 |
| dhellmann | anteaya: my time is limited, but I do try to help | 15:30 |
| *** mgagne has joined #openstack-infra | 15:30 | |
| sdague | dhellmann: http://logs.openstack.org/37/72437/1/check/check-devstack-dsvm-cells/6272c5f/logs/screen-c-api.txt.gz | 15:30 |
| anteaya | so apevec knows whom he can talk to regarding getting stable-maint work done | 15:30 |
| sdague | taskflow / stevedore thing | 15:30 |
| *** mkerrin has joined #openstack-infra | 15:30 | |
| anteaya | dhellmann: you are a helping kind of person, which is why I asked you | 15:30 |
| anteaya | apevec is there something dhellmann can help with? | 15:30 |
| sdague | dhellmann: the fact that it fails all the devstack jobs is usually a good indication that it's a pretty core issue :) | 15:31 |
| *** tomhe has joined #openstack-infra | 15:31 | |
| *** jcoufal has quit IRC | 15:31 | |
| sdague | basically I wonder if taskflow git actually works | 15:32 |
| dhellmann | sdague: hrm, I'll have to ask harlowja_away what that means | 15:32 |
| dhellmann | sdague: thanks | 15:32 |
| apevec | anteaya, dunno, up to him, but latest failures are nova/neutron area afaict | 15:32 |
| *** rcleere has quit IRC | 15:33 | |
| anteaya | apevec: okay, well going to be difficult for me to support getting some people to help with stable-maint if they don't have something to help with | 15:33 |
| *** mgagne1 has joined #openstack-infra | 15:34 | |
| dhellmann | apevec: were any of those neutron fixes for the isolated gate jobs backported? is that what needs review? | 15:34 |
| *** marun has joined #openstack-infra | 15:34 | |
| apevec | ihrachys has dug out some master Tempest changes which could help | 15:34 |
| fungi | anteaya: i think probably one of the biggest helps to stable branch management would be people subscribing to openstack-stable-maint@lists.openstack.org and looking at the bitrot job failure reports ("Stable check of <fooproject> failed" from jenkins) | 15:34 |
| *** mrodden has joined #openstack-infra | 15:34 | |
| dhellmann | apevec: I saw mtreinish's email to the list, but haven't looked at the reviews he linked | 15:34 |
| fungi | anteaya: and then trying to stay on top of fixing them as they crop up | 15:34 |
| mtreinish | dhellmann: there were a lot of them | 15:35 |
| *** mgagne has quit IRC | 15:35 | |
| mtreinish | improving the isolated job was one of the big pushes at montreal | 15:35 |
| dhellmann | mtreinish: right, I'm trying to understand if we're trying to backport them or if they are only going into master | 15:35 |
| dhellmann | mtreinish: because if we're backporting then it would be good to have a list to review and if we're not then I agree we should talk about turning that gate job off | 15:35 |
| * anteaya joins openstack-stable-maint@lists.openstack.org | 15:36 | |
| *** wenlock has joined #openstack-infra | 15:36 | |
| *** thomasem has joined #openstack-infra | 15:37 | |
| mtreinish | dhellmann: I really think it'll be better to just turn it off. We mostly added the job so neutron would feel the pain and try to prioritize fixing it. | 15:37 |
| *** AlexF has quit IRC | 15:37 | |
| mtreinish | the job has been broken for master on and off the whole cycle | 15:38 |
| dhellmann | mtreinish: yeah, if we know it doesn't work it's just pissing everyone off to keep it | 15:38 |
| *** rockstar has joined #openstack-infra | 15:38 | |
| *** rockstar has quit IRC | 15:38 | |
| *** rockstar has joined #openstack-infra | 15:38 | |
| fungi | anteaya: but also, these branches are supported for the benefit of distributions and deployers who use them, so the hope is that they step forward with sufficient prople to keep it working. if they don't, then i'm not entirely sure why the project as a whole puts forth the effort to coordinate it | 15:38 |
| fungi | s/prople/people/ | 15:38 |
| anteaya | hmmmm | 15:39 |
| fungi | and to that end, we've scaled back the supported duration and release count multiple times | 15:39 |
| * anteaya feels a summit session brewing | 15:39 | |
| apevec | fungi, yes that was the idea, to share the pain of keep old branch working | 15:39 |
| *** atiwari has joined #openstack-infra | 15:40 | |
| *** AlexF has joined #openstack-infra | 15:40 | |
| fungi | anteaya: there is a long-standing session at every summit which might as well be entitled "we're reducing stable support even more this cycle" | 15:40 |
| dhellmann | fungi: heh | 15:40 |
| anteaya | ah I have missed prior sessions, obviously | 15:40 |
| *** dripton__ is now known as dripton_shovelin | 15:41 | |
| * anteaya just wishes everything would just work in the gate | 15:41 | |
| fungi | anteaya: they're some of my favorite sessions, just remember to bring your anti-depressants | 15:41 |
| anteaya | no no no | 15:41 |
| mtreinish | anteaya: I've done that before it kind of defeats the purpose | 15:41 |
| anteaya | brewer's yeast and soy lecithin | 15:41 |
| *** hashar has quit IRC | 15:42 | |
| * fungi drinks plenty of brewer's yeast | 15:42 | |
| *** vkozhukalov has quit IRC | 15:42 | |
| anteaya | you don't have bad effects like from anti-depressents | 15:42 |
| anteaya | fungi: I keep trying to convince myself it tastes like chocolate milk | 15:42 |
| *** pcrews has joined #openstack-infra | 15:42 | |
| anteaya | so far I have been unsuccessful | 15:42 |
| *** pblaho has quit IRC | 15:43 | |
| anteaya | mtreinish: rather, yes | 15:43 |
| anteaya | but I can dream | 15:43 |
| *** hashar has joined #openstack-infra | 15:43 | |
| sdague | I think a piece of this whole problem is that if you surveyed core reviewers across the projects, I suspect < 50% have ever run tempest | 15:43 |
| anteaya | sdague: I would agree with you | 15:43 |
| anteaya | a running tempest focused effort? | 15:44 |
| anteaya | an educational push? | 15:44 |
| sdague | or figuring out what needs to be done to make that easier | 15:44 |
| mtreinish | sdague: to be fair I rarely run it | 15:44 |
| anteaya | sdague: interesting | 15:44 |
| *** hdd has joined #openstack-infra | 15:44 | |
| mtreinish | sdague: that was going to be summit session I was going to submit, that combined with tempest doc gaps | 15:45 |
| fungi | sdague: i would say, from what i've seen from my vantage point on the vmt, way fewer than 50% actually bother to run their patches through tempest locally on embargoed fixes too | 15:45 |
| sdague | mtreinish: so lets start there, what sort of setup would make it simpler for you to run it locally? Would a good vagrant guest be helpful | 15:45 |
| mtreinish | sdague: no it's just a matter of me being lazy or killing python deps on a node | 15:45 |
| mtreinish | I've got a cloud snapshot that I can spin up and rerun devstack and get there fairly quickly | 15:46 |
| mtreinish | it's just a matter of doing that vs pushing and waiting for the check queue to come back | 15:46 |
| sdague | mtreinish: right, so if it was easier to pull a good known revision, sort of conceptually a tox for tempest, would that help? | 15:46 |
| fungi | i can still try to get my devstack-gate manual control scripts into a publishable condition if it will help | 15:46 |
| fungi | but the barrier to entry there is having a cloud provider | 15:47 |
| fungi | access to one, at any rate | 15:47 |
| *** openstackgerrit has joined #openstack-infra | 15:47 | |
| mtreinish | sdague: like having one command that spins up a node and has everything setup? | 15:47 |
| anteaya | that is becoming less of a barrier all the time | 15:47 |
| sdague | fungi: so, I actually bet we could adapt them to also work on local vagrant very easily | 15:47 |
| jeblair | or a workstation that can run a vm. | 15:47 |
| *** AlexF has quit IRC | 15:47 | |
| mtreinish | that's basically what I have now | 15:47 |
| fungi | jeblair: sdague: true enough | 15:47 |
| jeblair | openstackgerrit: welcome back | 15:47 |
| fungi | i just meant in the current shape i have them in, they assume novaclient | 15:47 |
| jeblair | i think gerritbot was netsplit | 15:48 |
| fungi | yeah, i was looking at its logs | 15:48 |
| sdague | fungi: sure, if you want to stackforge all that, I can play with adapting it to other envs as well | 15:48 |
| sdague | because I think a coherent launch script to do this for local testing would bring more folks into the fold | 15:48 |
| *** AlexF has joined #openstack-infra | 15:48 | |
| sdague | and I think that would be a huge benefit to the project as a whole | 15:49 |
| openstackgerrit | Diane Fleming proposed a change to openstack-infra/config: OpenStack Image Service API v2 Reference link broken https://review.openstack.org/73331 | 15:49 |
| *** jasond` has joined #openstack-infra | 15:49 | |
| jeblair | fungi, SergeyLukjanov: would you like to review https://review.openstack.org/#/c/68817/ before i self-approve? | 15:49 |
| fungi | sdague: sure thing. it's been on my to do list, just not high enough to get any traction. also it's broken again with recent devstack-gate changes so i need to track it down. we don't really have an api for d-g so anything like this which tries to call bits of it has a tendency to break | 15:50 |
| jasond` | can anybody tell me why this isn't merging? https://review.openstack.org/#/c/71082/ | 15:50 |
| SergeyLukjanov | jeblair, morning, looking on it | 15:50 |
| sdague | basically I have a lot of lesser hacks, that are hard to motivate on making generic, but if I had someone to work with, like yourself, I think I could stay more motivated | 15:50 |
| jasond` | i don't see it in the zuul status page | 15:50 |
| fungi | jeblair: ahh, yes, i had looked at that one before, but not since it was un-wip'd and rebased | 15:50 |
| fungi | lgtm | 15:50 |
| *** mrmartin has quit IRC | 15:51 | |
| *** atiwari has quit IRC | 15:51 | |
| anteaya | jasond`: the gate jobs failed then you have been running recheck which sends it to the check queue | 15:51 |
| anteaya | to get it back into the gate queue you have to run reverify bug <bug #> or reapprove | 15:51 |
| jeblair | fungi, SergeyLukjanov: thanks, it's the kind of change i want to merge early in my day | 15:52 |
| jasond` | anteaya: i see. at this point, what should i run to get it back to the gate queue? | 15:52 |
| jasond` | reverify no bug? | 15:52 |
| anteaya | no that command has been removed | 15:52 |
| fungi | jasond`: right, it's not currently failing, so just get shardy to remove and readd his approval, or get another core reviewer on that project to add an approve vote | 15:52 |
| anteaya | you are core? you can reapprove | 15:52 |
| anteaya | or | 15:52 |
| anteaya | `reverify bug <bug #>` | 15:52 |
| jasond` | anteaya: i am. i'll try that | 15:52 |
| jasond` | thanks for the help | 15:52 |
| anteaya | k | 15:52 |
| anteaya | np | 15:52 |
| anteaya | good luck | 15:53 |
| fungi | jeblair: makes sense. i'd like to go ahead and do the bare-centos6 one now too if possible, since there are several of us around to quickly revert it if needed | 15:53 |
| *** gothicmi1dfood is now known as gothicmindfood | 15:53 | |
| openstackgerrit | Alexander Jones proposed a change to openstack-infra/git-review: Fix parsing of SCP-style URLs, as these are valid in Git itself https://review.openstack.org/72751 | 15:53 |
| anteaya | jasond`: this is your patch, so this is the only case where your reapproval is okay, just to send it back to the gate | 15:53 |
| anteaya | normally self approval is verboten | 15:54 |
| jeblair | fungi: yep | 15:54 |
| jasond` | anteaya: i made a note to that effect. thanks | 15:54 |
| anteaya | np | 15:54 |
| *** DeeJay1 has quit IRC | 15:54 | |
| *** IvanBerezovskiy has left #openstack-infra | 15:54 | |
| fungi | SergeyLukjanov: did you want to take a look at https://review.openstack.org/73107 real quick? | 15:54 |
| *** dcramer_ has quit IRC | 15:55 | |
| SergeyLukjanov | fungi, looking | 15:55 |
| SergeyLukjanov | fungi, I'm ok with it | 15:56 |
| jeblair | fungi: we have iad and ord ready now? | 15:56 |
| SergeyLukjanov | fungi, just find open tab with non-submitted +2 :) | 15:56 |
| fungi | jeblair: yeah, we had nodes in both before i went to sleep last nighth | 15:56 |
| jeblair | cool. both those are approved now | 15:57 |
| fungi | night | 15:57 |
| jeblair | i'm also merging this potentially breaking change: https://review.openstack.org/#/c/68998/ | 15:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Require a recent (48h) positive check result https://review.openstack.org/68817 | 15:58 |
| fungi | jeblair: for completing the bare-precise transition, i'm thinking we may want to refactor the way we're applying the single-use parameter function, yeah? since only a handful of jobs will now go to long-running trusted nodes, single-use becomes the common case. or should we just do a negative pattern match on the proposal/upload jobs and be done with it? | 15:59 |
| mtaylor | jeblair: I'm behind - are you not passing in the url anymore because you're pushing the merges to git.o.o ? | 15:59 |
| fungi | 68998 lgtm too, and i'll keep an eye out for all jobs failing to get any commits | 16:00 |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/config: XenServer installer script https://review.openstack.org/60581 | 16:00 |
| fungi | mtaylor: zuul passing the git url in a parameter | 16:00 |
| jeblair | mtaylor: no, it's part of a series; if you walk up to the first change, you'll see the rationale: https://review.openstack.org/#/c/68997/1 | 16:00 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch to bare-centos6 single-use nodes https://review.openstack.org/73107 | 16:00 |
| jeblair | mtaylor: it's to support the zuul merger split | 16:00 |
| mtaylor | jeblair: ok. cool. I'll go read | 16:01 |
| *** rcleere has joined #openstack-infra | 16:01 | |
| jeblair | fungi: i think we should reverse the pattern at some point | 16:01 |
| mtaylor | jeblair: got it. thanks | 16:02 |
| *** andreaf has quit IRC | 16:02 | |
| fungi | jeblair: would you rather two-step that? first do the negative regex, then add a new parameter function (or two) and swap it around in a subsequent change? | 16:03 |
| *** mgagne1 is now known as mgagne | 16:03 | |
| openstackgerrit | A change was merged to openstack-infra/config: Stop supplying the zuul url to ggp https://review.openstack.org/68998 | 16:04 |
| jeblair | fungi: i think it can be in one change. | 16:04 |
| ttx | apevec: http://paste.openstack.org/show/65076/ | 16:04 |
| ttx | apevec: hand-made a bit | 16:05 |
| ttx | (my script did not add the stable-maint core ** so I adedd them manually) | 16:05 |
| ttx | removing anyone** under 9 sounds fair, as would adding anyone above 30 | 16:06 |
| *** mtaylor is now known as mordred | 16:07 | |
| *** mordred has quit IRC | 16:07 | |
| *** mordred has joined #openstack-infra | 16:07 | |
| ttx | although we could review the quality of their reviews and their knowledge of the stable policy first | 16:07 |
| *** saju_m has joined #openstack-infra | 16:08 | |
| apevec | yeah, also this should be an offer, not forced :) | 16:08 |
| mtreinish | ttx: where are those numbers pulled from because I -2d like 7 tempest backports last week. | 16:09 |
| *** virmitio has joined #openstack-infra | 16:09 | |
| ttx | mtreinish: I quickly hacked a script because reviewstats doesn't support stable/* -- might be inaccurate. Only stable/havana considered | 16:11 |
| *** AlexF has quit IRC | 16:11 | |
| *** mrfatstrat has joined #openstack-infra | 16:11 | |
| *** AlexF has joined #openstack-infra | 16:11 | |
| anteaya | there are 20 people listed with 0 reviews | 16:12 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: OpenstackID Documentation https://review.openstack.org/69620 | 16:12 |
| mtreinish | ttx: ok | 16:12 |
| ttx | anteaya: yeah. Weird huh. I should probably take the time to properly hack that script. | 16:12 |
| mtreinish | ttx: just for reference: https://review.openstack.org/#/q/status:abandoned+project:openstack/tempest+branch:stable/havana+topic:bug/1252620,n,z | 16:13 |
| anteaya | or we can just admire the anomolies | 16:13 |
| anteaya | seems faster | 16:13 |
| mtreinish | it might be because he abandonded them | 16:13 |
| ttx | mtreinish: yes.. only status:merged is taken into account | 16:13 |
| ttx | because of some strange query limitation iirc | 16:13 |
| anteaya | ah so -2'ing be definition is not counted as a review | 16:14 |
| ttx | let me see if running without status:merged looks better | 16:14 |
| *** dizquierdo has quit IRC | 16:14 | |
| *** jpeeler1 has quit IRC | 16:14 | |
| ttx | running... | 16:15 |
| *** chandankumar__ has joined #openstack-infra | 16:16 | |
| *** dripton_shovelin is now known as dripton | 16:17 | |
| ttx | apevec, mtreinish: http://paste.openstack.org/show/65084/ | 16:17 |
| *** garyk has quit IRC | 16:18 | |
| *** VijayT has joined #openstack-infra | 16:18 | |
| *** chandankumar_ has quit IRC | 16:19 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack/requirements: Add requirements for taskflow https://review.openstack.org/73342 | 16:19 |
| openstackgerrit | Andreas Rehn proposed a change to openstack-infra/jenkins-job-builder: Added support for Delivery Pipeline Plugin https://review.openstack.org/71658 | 16:20 |
| anteaya | apevec: who are tamazur and jsbryant? | 16:21 |
| mtreinish | ttx: well now I think you might be overcounting a bit. :) I only counted 10 -2's from me. | 16:21 |
| apevec | they're now stable-maint members, just people doing reviews on stable patches | 16:22 |
| apevec | s/now/not/ | 16:22 |
| ttx | mtreinish: right, I think that's why I ended up adding status:merged to the query | 16:22 |
| ttx | because otherwise ther e was some doublecounts | 16:22 |
| anteaya | because if they have reliable knowledge of stable policy and good quality reivews, I would support their consideration for membership | 16:22 |
| anteaya | same with zhaoqin | 16:23 |
| apevec | anteaya, yes, as ttx mentioned: we need to review their reviews then ask them | 16:23 |
| *** mrfatstrat has quit IRC | 16:23 | |
| anteaya | apevec: are you in a position to review their reviews? | 16:23 |
| anteaya | or do you need some additional people for that? | 16:23 |
| anteaya | plus whichever list you go by, it appears some folks are not interested in maintaining membership | 16:24 |
| apevec | yeah, ttx/me/adam_g should go over the list in the next few days | 16:24 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Modernize ATC list format https://review.openstack.org/73347 | 16:25 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Retrieve ATC project list from governance repo https://review.openstack.org/73348 | 16:25 |
| fungi | ttx: ^ | 16:25 |
| fungi | jeblair: ^ | 16:25 |
| ttx | apevec: i should probably fix the stats generation properly then | 16:25 |
| jeblair | fungi: neato! | 16:25 |
| fungi | that's been rotting in a topic branch on my machine for too long | 16:26 |
| jeblair | btw, it's halfway through storyboard meeting time. i should get better at remembering to announce that here | 16:26 |
| fungi | gah | 16:26 |
| anteaya | apevec: that sounds great | 16:26 |
| *** AlexF has quit IRC | 16:27 | |
| *** mfer has joined #openstack-infra | 16:28 | |
| *** AlexF has joined #openstack-infra | 16:29 | |
| *** david-lyle has joined #openstack-infra | 16:31 | |
| *** pafuent has joined #openstack-infra | 16:31 | |
| fungi | jenkins02 and 04 have some bare-centos6 nodes running jobs now | 16:33 |
| *** AlexF has quit IRC | 16:34 | |
| *** pafuent has left #openstack-infra | 16:34 | |
| *** pafuent has joined #openstack-infra | 16:35 | |
| fungi | this one succeeded... https://jenkins02.openstack.org/job/gate-cinder-python26/6160/console | 16:35 |
| jeblair | yaay! | 16:35 |
| fungi | as did https://jenkins02.openstack.org/job/gate-ceilometer-python26/4105/console | 16:35 |
| fungi | so two different jobs | 16:35 |
| fungi | this one looks sort of bound up though... https://jenkins02.openstack.org/job/gate-ceilometer-python26/4103/console | 16:36 |
| *** gyee has joined #openstack-infra | 16:37 | |
| *** afazekas has quit IRC | 16:37 | |
| jeblair | a bunch of heat and horizon changes in the gate just turned into a christmas tree, but the failures don't look like they could be related to any of our changes | 16:38 |
| *** chandankumar__ has quit IRC | 16:38 | |
| *** AlexF has joined #openstack-infra | 16:38 | |
| sdague | jeblair: all failures with mock. Did we take a new version? | 16:40 |
| sdague | or fail to have one in the mirror at the right level | 16:40 |
| *** chandan_kumar has joined #openstack-infra | 16:41 | |
| fungi | the bare-centos6 fail i saw looks like zuul retried it on a different node a minute later... not entirely certain why | 16:41 |
| fungi | check pipeline, so not a gate reset | 16:42 |
| fungi | compare https://jenkins02.openstack.org/job/gate-ceilometer-python26/4103/ and https://jenkins02.openstack.org/job/gate-ceilometer-python26/4105/ | 16:42 |
| *** AlexF has quit IRC | 16:43 | |
| fungi | same change and patchset number | 16:44 |
| *** AlexF has joined #openstack-infra | 16:44 | |
| *** andreaf has joined #openstack-infra | 16:46 | |
| jeblair | sdague: the failing job and last successful job both had mock 1.0.1 | 16:46 |
| sdague | jeblair: odd | 16:46 |
| *** dcramer_ has joined #openstack-infra | 16:47 | |
| sdague | jeblair: so is your sufficiency check through now? | 16:47 |
| sdague | because, honestly, once that's through, I want to purge all the non devstack jobs on the gate side, as then unit test fails like this will only affect the project in question, and not everyone else | 16:48 |
| anteaya | sdague: https://review.openstack.org/#/c/68817/ | 16:48 |
| jeblair | sdague: i believe it is based on the zuul reload time | 16:48 |
| sdague | jeblair: ok, so then step 1 job trimming, remove all the unit test, pep8, and docs jobs from the gate? | 16:49 |
| *** mrodden has quit IRC | 16:49 | |
| sdague | I want a head nod before I work up the patch :) | 16:49 |
| sdague | so I'm not wasting my time | 16:49 |
| sdague | I'm also testing that it all works as expected with an older devstack change | 16:51 |
| jamespage | apevec, apologies - this is taking a bit longer than expected | 16:52 |
| jog0 | I'm getting a 503 from from elasticSearch | 16:52 |
| jamespage | apevec, the havana/iscsi topology tested OK | 16:52 |
| apevec | jamespage, np, just let me know when you have results | 16:52 |
| sdague | 503 is typically either over capacity (cold cache) or bad params | 16:52 |
| jamespage | apevec, unfortunately the lab ran out of IP's during the havana/ceph test | 16:52 |
| jamespage | apevec, re-running that now | 16:52 |
| sdague | the proxy isn't great at distingusihing them | 16:52 |
| apevec | jamespage, cool, is there maybe public url with results? | 16:53 |
| jog0 | sdague: I am concerned about dropping pep8 and unit tests from gate | 16:53 |
| jamespage | apevec, apparently not (I was surprised) | 16:54 |
| jog0 | http://status.openstack.org/elastic-recheck/data/uncategorized.html shows we have a good number of valid bugs that are cought on those jobs in gate http://status.openstack.org/elastic-recheck/data/uncategorized.html | 16:54 |
| jamespage | apevec, not going to fix that now - will do for next | 16:54 |
| *** geekinutah has joined #openstack-infra | 16:54 | |
| apevec | ok | 16:54 |
| apevec | jamespage, re. ceph - there's one patch which is Ceph related but came too late: https://review.openstack.org/72575 | 16:55 |
| apevec | so I'll wait for you ceph results before tagging nova | 16:55 |
| *** DinaBelova is now known as DinaBelova_ | 16:55 | |
| jog0 | sdague: I am afraid that if we drop pep8 and unit from gate we will constantly be fixing them in check queue | 16:55 |
| jamespage | apevec, the ceph scenarios does not do boot from rbd via nova | 16:56 |
| jamespage | but lets check it out anyway | 16:56 |
| sdague | jog0: then they are fixable on the check queue in the project in question | 16:56 |
| sdague | because right now, it's the same issue, except it kills changes from unrelated projects | 16:57 |
| jog0 | sdague: then each project will be blockeda nd playing fix up | 16:57 |
| sdague | jog0: instead of each project blocking all the other projects? | 16:57 |
| jog0 | sdague: what about this: | 16:57 |
| sdague | jog0: they have to have clean check score to make it into the gate | 16:57 |
| fungi | (as of a few minutes ago) | 16:58 |
| jog0 | if the gate has TOP -> nova -> cinder. | 16:58 |
| *** e0ne has quit IRC | 16:58 | |
| jog0 | and nova unit tests fail, we don't invalidate cinder | 16:58 |
| jog0 | but just all nova patches | 16:58 |
| sdague | jog0: ok, so go and rewrite zuul to do that in the next week | 16:59 |
| jog0 | so if nova-1 -> cinder-1 -> nova-2. nova-2 gets invalidated | 16:59 |
| sdague | right now the dependency setup is one dimensional | 16:59 |
| jog0 | sdague: I am afraid your proposal will make each project play catch up all the time | 16:59 |
| *** garyk has joined #openstack-infra | 17:00 | |
| sdague | jog0: well the alternative, given the failure rate of unit tests, is a 60 hr merge queue again on icehouse-3 | 17:00 |
| *** DinaBelova_ is now known as DinaBelova | 17:00 | |
| sdague | this was all outlined in that change proposal | 17:00 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 17:00 | |
| jog0 | if we get there we can take drastic action but I don't think we don't need to yet | 17:00 |
| sdague | jog0: we really do | 17:00 |
| jog0 | sdague: right now a failed nova unit test doesn't 'block' a cinder patch it just slows it down | 17:01 |
| *** mrodden has joined #openstack-infra | 17:01 | |
| *** gokrokve has quit IRC | 17:01 | |
| sdague | jog0: so then people will fix the unit tests on the check side | 17:01 |
| jog0 | sdague: with the gate queue flushing out every night, why do you think we need this now? | 17:01 |
| sdague | jog0: because we're at really low inbound | 17:01 |
| *** gokrokve has joined #openstack-infra | 17:02 | |
| *** AlexF has quit IRC | 17:02 | |
| jog0 | sdague: that violates the major idea behind the gate, keep trunk working so we don't block eachother | 17:02 |
| sdague | and doing changes like this when everything is backed up 60 hrs and thrashing, is the wrong time to do it | 17:02 |
| sdague | jog0: you can't get in without working check | 17:02 |
| *** afazekas has joined #openstack-infra | 17:02 | |
| mordred | what if we remove bad tests from the unittest set? | 17:02 |
| sdague | and you are cogating on all the tempest | 17:02 |
| anteaya | jog0: https://review.openstack.org/#/c/68817/ | 17:02 |
| mordred | if there are flaky unit tests - that means they are bad tests | 17:02 |
| sdague | mordred: or bad code | 17:03 |
| mordred | no | 17:03 |
| anteaya | do you mean remove or fix? | 17:03 |
| anteaya | I'm all for fixing, I'm against removing | 17:03 |
| mordred | if they are hitting race conditions, then they are functional tests | 17:03 |
| jog0 | sdague: even with the run check >2 days before gate we are seeing the gate catch valid pep8 and py27 issues | 17:03 |
| mordred | and they shoudl not be in the unittests | 17:03 |
| sdague | mordred: ok, fine semantics | 17:03 |
| sdague | jog0: the gate should never be catching valid pep8 / py27 errors | 17:03 |
| sdague | ever | 17:03 |
| jog0 | sdague: but it is | 17:03 |
| jog0 | I agree with that statement | 17:03 |
| mordred | sdague: so, I'm just saying, if we have tests in the unittests that are not 1005 consistent, remove them | 17:03 |
| jog0 | but its invalid today | 17:04 |
| sdague | because people are being lazy and not fixing it in check | 17:04 |
| jog0 | mordred: this is things that fail 100% of the time | 17:04 |
| anteaya | mordred: remove, as in delete, or move as in put them where they belong? | 17:04 |
| jog0 | mordred: see status.openstack.org/elastic-recheck/data/uncategorized.html | 17:04 |
| sdague | mordred: so glance, swift, nova, heat, neutron all have races in their unit tests | 17:04 |
| jog0 | skim the pep8 issues for trivial examples | 17:04 |
| *** mfer has quit IRC | 17:04 | |
| sdague | have for 6 months | 17:04 |
| mordred | sdague: great. delete the unittests | 17:05 |
| mordred | I mean, delete the racey tests. full stop. if someone wants to move the thing it's testing to tempest, that's awesome | 17:05 |
| fungi | jog0: as of today, it's impossible for changes to get approved into the gate when they're not getting a +1 from check jobs first. and shortly we're looking at upping that from 48 hours to 24 hours to narrow the eye of the needle a bit more | 17:05 |
| mordred | but they are, by all measures, bad tests adn in the wrong place | 17:05 |
| *** gokrokve has quit IRC | 17:06 | |
| sdague | mordred: that's fine, that's not addressable by icehouse | 17:06 |
| *** andre__ has quit IRC | 17:06 | |
| *** AlexF has joined #openstack-infra | 17:06 | |
| jog0 | fungi: but we are still seeing things being catched in gate today | 17:06 |
| jog0 | see pepe8 | 17:06 |
| anteaya | fungi: from jenkins check jobs, or any check jobs? | 17:07 |
| mordred | gotta step away for a sec | 17:07 |
| sdague | basically it's a ton of grunt work that no one is willing to do. I think having projects wedge themselves on check, in a way that's completely fixable by the projects independently, is the right call | 17:07 |
| sdague | because that will put the incentive in the right place | 17:07 |
| anteaya | sdague: I agree | 17:07 |
| fungi | jog0: and rerunning those in the check queue within the same 24-hour period would have succeeded? | 17:07 |
| anteaya | if they won't fix their tests, keep them in check | 17:07 |
| jeblair | sqlalchemy migration numbers are a concise example of the kind of thing that the gate is designed to prevent landing | 17:08 |
| sdague | because the answer to failed unit test in gate is just reverify, I've never seen anyone actually work to fix a race there | 17:08 |
| anteaya | fungi: would a +1 verified from a third party testing system enough to get a patch into the gate, even if jenkins hasn't come back with results yet? | 17:08 |
| jeblair | it's a perfectly legitimate problem, and i know we ran into it all the time | 17:08 |
| sdague | jeblair: and devstack / tempest wouldn't catch that? | 17:08 |
| jog0 | how about this: | 17:08 |
| jeblair | sdague: i'm citing it as an example of the process | 17:08 |
| fungi | anteaya: no, the filter is on the specific gerrit account | 17:08 |
| sdague | jeblair: sure | 17:08 |
| anteaya | fungi: yay! | 17:08 |
| jog0 | wait a 2 weeks since https://review.openstack.org/#/c/68817/ was landed | 17:09 |
| jog0 | and see what http://status.openstack.org/elastic-recheck/data/uncategorized.html looks like | 17:09 |
| *** HenryG has joined #openstack-infra | 17:09 | |
| anteaya | jog0: that puts us into feature freeze ramp up | 17:09 |
| anteaya | just for timing | 17:09 |
| jog0 | then we will have real data on this | 17:09 |
| anteaya | I agree we need to access the effects of 68817 | 17:09 |
| jog0 | anteaya: if we make a bad choice now things will get worse IMHO | 17:10 |
| anteaya | I agree | 17:10 |
| sdague | jog0: why not make the change now, see how bad it is in 2 weeks instead? | 17:10 |
| anteaya | just letting you know what will be happening in two weeks from now | 17:10 |
| *** markmc has quit IRC | 17:10 | |
| jeblair | sdague: to be honest, this is the part of the process i'm most hesitant about. i do think that lifeless made some good points. but i also said that i was willing to try it and i'll stick to that. | 17:10 |
| sdague | you make this assumption that the alternative is going to be terrible, but we don't actually know | 17:10 |
| *** andre__ has joined #openstack-infra | 17:10 | |
| jog0 | sdague: because if you are wrong you move the burdon off of a computer onto people | 17:10 |
| *** markmcclain has joined #openstack-infra | 17:11 | |
| sdague | jog0: no... we move it from impacting unrelated projects, to only impacting single projects | 17:11 |
| jeblair | sdague: (if you change your mind, i will be somewhat relieved) | 17:11 |
| sdague | so here's where I stand | 17:11 |
| jog0 | sdague: if we let in a blocking patch (something that breaks pep8 or unit tests 100% of time) then we have a blocked repo | 17:11 |
| sdague | during the month of january, I spent about 3 weeks straight, including weekends, trying to unwedge the gate | 17:11 |
| jog0 | vs a slighly slower gate | 17:11 |
| jeblair | sdague: my personal preference would be, if it's possible, to see what impact the freshness requirement alone has first, and preferably at a 24h interval, before we remove the unit test jobs | 17:12 |
| sdague | that meant manually pulling changes out of the gate (hundreds) that were failing unit tests | 17:12 |
| jog0 | jeblair: ++ | 17:12 |
| sdague | and manually getting 40ish changes promoted | 17:12 |
| sdague | and I think the biggest issue is that stuff would fail the gate that had no reason being there, like pep8 changes failing on a git name resolve issue | 17:13 |
| sdague | or unit tests failing 8% of the time in swift | 17:13 |
| jog0 | sdague: so the freshness logic will catch all that | 17:13 |
| sdague | it won't catch the races | 17:13 |
| jog0 | no, but we can do a better job of classififying and dealing with those | 17:13 |
| jog0 | (disable them, yell at people etc.) | 17:13 |
| ttx | apevec: how is the release going ? Need me around ? | 17:13 |
| sdague | and I brought them up numerous times with people, and the general reaction was *schrug* don't know | 17:14 |
| anteaya | jog0: can you define a better job? | 17:14 |
| sdague | by the core team members | 17:14 |
| anteaya | I think sdague is saying he doesn't want to do what he did in January again in March | 17:14 |
| sdague | and people typing reverify 123456789 | 17:14 |
| jog0 | anteaya: yes, a better job at tracking gate failruse | 17:14 |
| anteaya | jog0: how? | 17:14 |
| jog0 | anteaya: aka stairing at http://status.openstack.org/elastic-recheck/data/uncategorized.html | 17:14 |
| sdague | anteaya: I think what I'm saying is I *will not* do it again, period | 17:14 |
| anteaya | sdague: okay, and I agree with you | 17:15 |
| anteaya | the same flavour of frustration drove me into Neutron | 17:15 |
| sdague | so my intent is to provide systematic changes to push the pain point back to the right place | 17:15 |
| sdague | because the issue right now, is the pain point is in the wrong place | 17:15 |
| anteaya | and now I will snipe a Neutron patch from the gate without mercy | 17:15 |
| sdague | so it's not getting fixed | 17:15 |
| anteaya | so I feel better coming into this ff | 17:15 |
| apevec | ttx, started tagging, will wait for CI result from jamespage before pushing | 17:15 |
| apevec | ttx, I should be all set | 17:15 |
| jog0 | sdague: btw docs failed once in gate with an unknown error | 17:16 |
| anteaya | sdague: I agree, if the pain point is in the correct place, action either happens or it doesn't, but the rest of the folks carry on | 17:16 |
| sdague | and to try this experiment before the queue is already jammed | 17:16 |
| sdague | because we can always revert | 17:16 |
| anteaya | sdague: how long would you like to try it? | 17:16 |
| jog0 | sdague: so this is a two part experiment | 17:16 |
| anteaya | what length of time to experiment? | 17:16 |
| jog0 | 1) the freshness aspect. 2) removing things from gate | 17:17 |
| notmyname | sdague: I'm in a meeting now, but I want to follow up with you a little later on this | 17:17 |
| jog0 | more info on 1) gives us more confidence on the impact on 2 | 17:17 |
| anteaya | I'm in agreement with you | 17:17 |
| jog0 | sdague: if you are wrong on 2, you cause a lot more pain for a lot more people that we can avoid. | 17:17 |
| anteaya | however given our timing to ff, we need to come to agreement soon | 17:17 |
| anteaya | as in this week-ish | 17:18 |
| sdague | jog0: no, it causes different pain | 17:18 |
| anteaya | so we are not testing things during traffic | 17:18 |
| sdague | it's not actually more pain | 17:18 |
| jog0 | anteaya: we should have decent Preliminary results ian week | 17:18 |
| sdague | and it's pain back at the point of fixing it | 17:18 |
| anteaya | jog0: how much time does that give us to evaluate the second part? | 17:18 |
| jog0 | sdague: the pain is we wedge pep8 on repo x and everything stops | 17:18 |
| sdague | jog0: yep | 17:19 |
| anteaya | and enough time to revert before we are overrun? | 17:19 |
| sdague | and you know what, nova core will fix that pep8 issue really fast | 17:19 |
| jog0 | anteaya: well two weeks of data gets us clean ES data, but a week is decent | 17:19 |
| jog0 | sdague: if we are all asleep? if its a unit test issue? | 17:19 |
| jog0 | its a big distraction that we would rather avoid | 17:19 |
| pafuent | Hi. I'm trying to get working a devstack-dsvm gate for Climate, and after some tries I get it working, except that I can't see any screen logs uploaded in the gate results. What is the meaning of this? The screens failed for some reason or I need to do something special in order to get those logs uploaded? | 17:19 |
| anteaya | jog0: but that gives us 0 time to test sdague's proposal before the traffic hits | 17:19 |
| *** mfer has joined #openstack-infra | 17:19 | |
| jog0 | anteaya: so I see two possible impacts of removing pep8/unit tests fro mgate | 17:20 |
| sdague | jog0: so instead it's preferable to crater heat landing changes? | 17:20 |
| sdague | and make heat core devs have to babysit gate queue for days | 17:20 |
| jog0 | 1) we wedge more repos more often. 2) make gate faster by moving the burdon of racey unit tests | 17:21 |
| sdague | because basically what I'm hearing is - we should put more burden on all the other 9 integrated projects instead of fixing our unit tests | 17:21 |
| *** atiwari has joined #openstack-infra | 17:21 | |
| jog0 | without implimenting removing thigns from gate we will get a clear idea of 1) and 2 I am fairly confident about | 17:21 |
| anteaya | jog0: yes, those are the impacts | 17:21 |
| jog0 | sdague: the burdon is only a slower gate | 17:21 |
| anteaya | jog0: you have yet to give a time frame for evaluating 1 that leaves enough time to evaluate 2 | 17:22 |
| sdague | jog0: that's a real burden | 17:22 |
| zaro | morning | 17:22 |
| sdague | when you need to do things like land dependent changes across projects | 17:22 |
| jog0 | sdague: which we can live with for a bi | 17:22 |
| sdague | jog0: we really can't | 17:22 |
| jog0 | and the gate is in the best shape it has been in a long long time | 17:22 |
| jog0 | we landed 435 patches two weeks ago | 17:22 |
| *** AlexF has quit IRC | 17:22 | |
| sdague | jog0: because we're at a lull | 17:22 |
| sdague | that's super low | 17:22 |
| jog0 | sdague: that is a record for us | 17:23 |
| anteaya | jog0: yes take a look at traffic | 17:23 |
| anteaya | volume changes everything | 17:23 |
| jog0 | and the gate flushes out over the weekend ... and every night | 17:23 |
| anteaya | right now yes | 17:23 |
| zaro | fungi: i'm can't seem to push new PS in gerrit due to https://review.openstack.org/#/c/73126 | 17:23 |
| sdague | jog0: we're at a lull | 17:23 |
| jog0 | anteaya: yes, if that changes we can re-evalute the timeline | 17:23 |
| zaro | fungi: can you help? | 17:23 |
| anteaya | that is not fair | 17:23 |
| jog0 | sdague: I don't think landing 141 patches this Tuesday is a lul | 17:23 |
| anteaya | the only change will be more traffic, which is coming, which elimnates the ability test anything | 17:24 |
| sdague | well, you have my point of view. | 17:24 |
| *** chandan_kumar has quit IRC | 17:24 | |
| jog0 | sdague: at the very least I think we should wait a week | 17:24 |
| sdague | I'm basically at the point where I don't think we'll make it through icehouse 3 | 17:24 |
| anteaya | I still think it is possible to set a time frame to test 1 which allows enough time to test 2 within the next week | 17:24 |
| jog0 | I think we have a better solution which is much more invasive but safer (zuul changes) | 17:24 |
| jeblair | jog0: what solution is that? | 17:25 |
| sdague | and honestly, if that crap hits the fan, I have no interest in helping to unwind this during i3 | 17:25 |
| sdague | because, I completely burnt out on diagnosing and hotfixing during i2 | 17:25 |
| anteaya | sdague: I hear you | 17:25 |
| jog0 | jeblair: given TOP_OF_GATE -> nova-1 -> cinder-1 -> nova-2 | 17:25 |
| jeblair | sdague: i will +2 the change to remove unit tests, etc from the gate but i think we should set the interval to 24h first (i think that was the plan) | 17:26 |
| sdague | jeblair: yep, I'm fine with that | 17:26 |
| jog0 | jeblair: if pep8 or unit tests fail in nova 1, don't invalidate cinder-1, only nova-2 | 17:26 |
| sdague | I think 24hr freshness is good | 17:26 |
| jeblair | jog0: that won't work; the nova change still affects the system that is being tested with the cinder change | 17:26 |
| sdague | jeblair: can you seperate the auto freshness from the required freshness? | 17:26 |
| *** AlexF has joined #openstack-infra | 17:26 | |
| jog0 | jeblair: oh, how so? | 17:26 |
| sdague | because I think we can actually live with required freshness at 3 or 4 days | 17:26 |
| sdague | sorry, auto freshness at 3 or 4 days | 17:27 |
| sdague | and required at 24h | 17:27 |
| jeblair | sdague: i don't think so; we're counting on side effects of their being the same | 17:27 |
| *** yassine has quit IRC | 17:27 | |
| jeblair | sdague: particularly, the fact that if you approve something and it doesn't meet required freshness, it auto-triggers auto-freshness | 17:27 |
| sdague | jeblair: ok, that just means more check queue traffic, but we can probably deal with that for now | 17:27 |
| sdague | ok | 17:27 |
| sdague | jeblair: where do we change the 24hr bump? | 17:27 |
| jeblair | sdague: i mean, if we wanted to give that up, we could, but basically we'll spend all day telling people "that wasn't enqueued because the check result is too old, type 'recheck no bug'" | 17:28 |
| sdague | jeblair: no, I think this is good | 17:28 |
| fungi | zaro: why is 73126 preventing you from pushing patches into gerrit? | 17:28 |
| jeblair | sdague: just s/48/24/ in zayout.yaml | 17:28 |
| sdague | I think there could be an optimization in the future | 17:28 |
| jog0 | jeblair: what is the issue with my idea above ^ | 17:29 |
| sdague | jeblair: oh... could we trigger off of different votes? | 17:29 |
| zaro | fungi: i'm trying to push new PS for https://review.openstack.org/#/c/65133 but when i gitreview it says no .gitreview file. | 17:29 |
| sdague | so +2 / +A triggered at a different interval than -1,0,+1 | 17:29 |
| jeblair | jog0: you might land a change to cinder that only works if the change in nova is present | 17:30 |
| sdague | anyway, future optimization | 17:30 |
| zaro | fungi: when i add .gitreview it won't work either because it's not untracke. | 17:30 |
| jog0 | jeblair: that isn't true for unit tests | 17:30 |
| jog0 | err pep8 | 17:30 |
| jog0 | not sure how that is possible for unit tests | 17:30 |
| zaro | fungi: *it's untracked | 17:30 |
| jog0 | unless you are ceilometer and pull in nova repo during unit tests | 17:31 |
| jeblair | jog0: this would be a failed devstack test | 17:31 |
| sdague | jog0: but you aren't kicking out tests, you are kicking out changes | 17:31 |
| sdague | so you need to reset the whole queue | 17:31 |
| jeblair | jog0: except it's not failing because it's still being tested with the nova change because your proposal was that we keep the cinder tests running | 17:31 |
| fungi | zaro: why would an untracked .gitreview file in your repo be a problem? | 17:31 |
| jog0 | jeblair: ohh I see now | 17:31 |
| sdague | jog0: believe me, a ton of brain power has been spent on the possible interactions | 17:32 |
| clarkb | morning | 17:32 |
| jog0 | sdague: the 'just trust me' answer is never a good one | 17:32 |
| jog0 | clarkb: I can't hit http://logstash.openstack.org/ | 17:33 |
| fungi | zaro: we should land the change to update the gerrit unit test job, then the change which fixes the submodules, then with working tests for that branch we can land the .gitreview file addition | 17:33 |
| jog0 | clarkb: and good morning | 17:33 |
| fungi | zaro: if we do it right, those should be able to merge properly in that sequence | 17:33 |
| clarkb | gearman logstash queue is better than yesterday (under 90k) but we did lose 3 indexers (last night we lost all 15 on old code so an improvement) going to scale back to two workers per indexer on new indexers to see if that helps more | 17:33 |
| clarkb | jog0 yup I turned off kibana | 17:33 |
| jog0 | clarkb: oh? | 17:34 |
| clarkb | will correct when everything is caught up | 17:34 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: narrow the freshness window to 24h https://review.openstack.org/73366 | 17:34 |
| jog0 | clarkb: ack | 17:34 |
| jog0 | jeblair sdague: what about having a pre-gate pipeline that runs pep8 and unit tests only | 17:34 |
| zaro | fungi: let me check again. maybe typed the wrong git cmd | 17:34 |
| jog0 | right before gate | 17:35 |
| sdague | jog0: ok, don't trust me. I'll stop looking at any gate things, because apparently the aggregate knowledge gained there isn't useful in convincing you of things :) | 17:35 |
| jog0 | sdague: all I am saying its good to share knoweldge and reasoning | 17:35 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerrit: Add a Change Owner group https://review.openstack.org/65133 | 17:35 |
| zaro | fungi: sorry about that. i forgot to specify the branch when doing the git review | 17:36 |
| fungi | zaro: also, note my comment on that change. you can solve that by adding defaultbranch to the .gitreview file | 17:37 |
| jog0 | clarkb: any ETA when things will be back (no rush, just wondering so I can plan accordingly) | 17:37 |
| clarkb | I need to run a few errands this morning and will finish es/logstash cleanup when I return | 17:37 |
| jog0 | clarkb: thanks | 17:37 |
| clarkb | jog0 ~3 hours? depends on how much new stuff rolls in this morning | 17:37 |
| jog0 | clarkb: awesome | 17:38 |
| *** hashar has quit IRC | 17:38 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerrit: Fix Schema_86 to use correct UUID for Change Owner group https://review.openstack.org/65134 | 17:39 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 17:39 | |
| jog0 | sdague: so I unit tests can have race conditions. but docs and pep8? | 17:40 |
| jog0 | I don't know of any pep8 races | 17:41 |
| jog0 | and docs appear to never fail due to docs issues | 17:41 |
| *** bhuvan has joined #openstack-infra | 17:41 | |
| jog0 | (but that is neither here nor there) | 17:42 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerrit: Add a Change Owner group https://review.openstack.org/65133 | 17:42 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerrit: Fix Schema_86 to use correct UUID for Change Owner group https://review.openstack.org/65134 | 17:42 |
| *** hashar has joined #openstack-infra | 17:43 | |
| *** vkozhukalov has joined #openstack-infra | 17:43 | |
| fungi | jog0: i would count doc tooling issues as ways those jobs can break | 17:44 |
| *** jpich has quit IRC | 17:44 | |
| zaro | fungi: thanks. i have added defaultbranch. | 17:44 |
| jog0 | fungi: sure, so what about pep8? | 17:45 |
| jog0 | if there is a pep8 issue that I don't know about? | 17:45 |
| jog0 | (gate related that is) | 17:45 |
| ewindisch | jog0: btw, in regard to rechecks, the 3rd-party requirements page is actually kind of broken. | 17:45 |
| fungi | there are plenty of ways to break pep8... are you asking about nondeterministic results from pep8 or just general failures in pep8 jobs? | 17:46 |
| jog0 | ewindisch: oh? | 17:46 |
| jog0 | fungi: non-deterministic | 17:46 |
| jog0 | ewindisch: why aren't you skiing? | 17:46 |
| ewindisch | jog0: at the top it says you must support rechecks, but then suggests using the jenkins gerrit-trigger (and a configuration for it), but it doesn't support rechecks. | 17:46 |
| ewindisch | jog0: because I have to work. I'm going to hit the slopes after lunch | 17:46 |
| jog0 | ewindisch: ahh yes that is an issue | 17:46 |
| jog0 | ewindisch: ahh enjoy | 17:47 |
| ewindisch | jog0: I thought you said you were staying today, too? | 17:47 |
| jog0 | jeblair: ^ w/ regard to third_party tseting docs | 17:47 |
| fungi | ewindisch: i agree, we've had mailing list threads about that too. alternatives are to either expect third-parties to clean up/manually inspect all failures reported, or run something more capable than the gerrit trigger plugin | 17:47 |
| jog0 | ewindisch: something came up so have go back today | 17:47 |
| fungi | i'm not really comfortable even continuing to recommend the gerrit trigger plugin, because even though i've never myself used or even configured it, people constantly ask me to troubleshoot their installations of it | 17:48 |
| lifeless | jeblair: sdague: oh hi | 17:48 |
| ewindisch | fungi: I'm using it without problems, other than it doesn't do rechecks. | 17:48 |
| ewindisch | fungi: my personal preference would be some leniency in getting it implemented in a timely fashion IRT the hypervisor deprecation plan | 17:49 |
| lifeless | I'm not against trying it; what I'd ask is that we be willing to revert if it is worse - but how do we assess worse? | 17:49 |
| *** senthil has quit IRC | 17:49 | |
| ewindisch | fungi: but I did start on a solution for long-term: https://github.com/ewindisch/gherkin-trigger | 17:49 |
| fungi | ewindisch: love the name too! | 17:49 |
| jog0 | lifeless: w/ respeect to pulling unit tests out of gate? | 17:49 |
| jeblair | ewindisch, fungi: yeah, we should probably just recommend zuul. | 17:50 |
| lifeless | jog0: that broad discussion yes | 17:50 |
| lifeless | jog0: it was AIUI more than just unit tests that were going to be pulled | 17:50 |
| ewindisch | jeblair: that's another option, although I didn't really want to go down that road. It's probably an irrational fear of the unknown | 17:50 |
| jeblair | ewindisch: why don't you patch the gerrit-trigger-plugin? | 17:50 |
| ewindisch | jeblair: java. | 17:51 |
| jog0 | lifeless: yes, sdague wants all !devstack jobs to be pulled | 17:51 |
| fungi | zuul is lightweight enough it could run on a jenkins master and talk over loopback | 17:51 |
| fungi | depending on your scaling needs | 17:51 |
| jeblair | ewindisch: do you realize you just re-wrote commit 1 of zuul? | 17:51 |
| jog0 | lifeless: so we can assess worse by seeing if we wedge unit and pep8 tests for individual repos more | 17:51 |
| ewindisch | but yeah, there are countless options. My main problem is that I have an impossible deadline | 17:51 |
| jog0 | lifeless: not sure best way to collect that data though | 17:52 |
| fungi | gerrit trigger plugin being unable to parse gerrit comments -> zuul | 17:52 |
| * jog0 gets ready to board his flight | 17:53 | |
| anteaya | ewindisch: gherkin is a tool used with cucumber testing framework, if you care | 17:53 |
| ewindisch | anteaya: bleh. It's also an alternative pickle library. | 17:53 |
| jeblair | ewindisch: zuul can actually be pretty lightweight for just a check queue; we had a lot of problems with that approach that we have solved along the way. | 17:54 |
| anteaya | ewindisch: as long as you know | 17:54 |
| ewindisch | anteaya: I didn't, actually :( | 17:54 |
| anteaya | k | 17:54 |
| ewindisch | jeblair: it isn't very clear how to run zuul without gearman | 17:54 |
| *** hashar has quit IRC | 17:54 | |
| jeblair | ewindisch: if you are into writing python-based test tools, i'd encourage you to contribute to the ones we're working on | 17:55 |
| anteaya | ewindisch: the biggest problem is how to convey this information to people standing up these systems will zero knowledge of openstack development | 17:55 |
| fungi | zuul has a gearman server built in. attach any gearman workers you like, including the jenkins-gearman plugin | 17:55 |
| jeblair | ewindisch: you can't; you'll need the gearman plugin for jenkins. | 17:55 |
| ewindisch | jeblair: my problem is that I have negative time. I'm taking every shortcut I can. | 17:55 |
| lifeless | jog0: so, I want to add tripleo-gate jobs into the gate :) | 17:55 |
| ewindisch | jeblair: my interest is in gating the docker code so Russell doesn't rip it out of Nova. | 17:55 |
| HenryG | When submitting a "git review", how is the 'topic' determined? | 17:55 |
| anteaya | they also seem to have little knowledge of opensource, how to use mailing lists or how to work on anything other than windows apps | 17:55 |
| anteaya | just some general observations | 17:56 |
| fungi | HenryG: it's guessed from your local branch name, or specified with the -t command-line option | 17:56 |
| lifeless | jog0: that is actually my biggest concern, but there's a path to doing that - we need the check-on-+2-only queue. | 17:56 |
| ewindisch | jeblair: I have deadline-insanity for that, so no, things won't get done the best way. Throwaway work will get done. | 17:56 |
| anteaya | HenryG: it is the topic branch you are on in your local git repo | 17:56 |
| fungi | HenryG: and if your local branch name is the same as the remote branch and you don't use -t, then it empties the topic field on the assumption you didn't want one (because a topic named the same as the remote branch is pretty useless usually) | 17:57 |
| anteaya | HenryG: before editing, run git checkout -b <branch name> | 17:57 |
| anteaya | for a new branch | 17:57 |
| ewindisch | jeblair: because spending an hour writing my own python script is faster than learning anyone else's tool, even if it's a whole lot better | 17:57 |
| anteaya | or git checkout <branch name> for an existing branch | 17:57 |
| ewindisch | (and yes, that sucks) | 17:57 |
| jeblair | ewindisch: okay, well when things calm down, a lot of people are setting up zuul servers for 3rd party testing, so the setup process is going to get a lot better and there will be a lot of support for it. | 17:57 |
| HenryG | anteaya: that's what I did, but my local branch and the topic in gerrit do not seem in any way related. | 17:58 |
| jeblair | ewindisch: and we'd love your contributions to that. :) | 17:58 |
| ewindisch | jeblair: cool. and I wish I was using zuul/gearman. I do plan to move to it, once there are cycles. | 17:58 |
| fungi | ewindisch: some of the people installing zuul for third-party test integration are also writing howto documentation | 17:58 |
| anteaya | HenryG: did you create the branch before editing and commiting? | 17:58 |
| sdague | lifeless: I think we want to consider if a smaller set of configurations for integration points work in the gate. Phase 1, just get rid of the non devstack jobs, as those are clearly not cross project. | 17:58 |
| HenryG | anteaya: yes | 17:58 |
| anteaya | HenryG: what url for the patch | 17:59 |
| fungi | HenryG: oh, also, we have a hook which will change your topic to bug/NNNNNNN if you include a bug header line in your commit message | 17:59 |
| anteaya | HenryG: and paste the output of git status | 17:59 |
| HenryG | anteaya: https://review.openstack.org/73355 | 17:59 |
| *** senthil has joined #openstack-infra | 17:59 | |
| ewindisch | er.. anyway, the whole point was that the 3rd-party testing page is wrong to suggest using gerrit-trigger | 17:59 |
| ewindisch | ;-) | 17:59 |
| lifeless | sdague: so the conceptual problem I have is that pre-merge gating is valuable for projects on their own, in it's own right. | 17:59 |
| anteaya | fungi: looks like the hook hits bp for as well | 18:00 |
| ewindisch | (unless 3rd-party rechecks aren't really necessary) | 18:00 |
| lifeless | sdague: and this feels like we're taking that away from projects and saying 'you choose - pre-merge gating, or integrated pre-merge gating' | 18:00 |
| sdague | lifeless: so if the functionality is caught in a tempest jobs, it will be fine. | 18:00 |
| lifeless | sdague: I think we actually need to do more of a pipeline arrangement in zuul | 18:00 |
| anteaya | HenryG: seems the hook has picked up the blueprint for in the commit message | 18:01 |
| sdague | because most of the real issues I've seen crop up were because projects, like heat, weren't actually testing in tempest | 18:01 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: OpenStack Image Service API v2 Reference link broken https://review.openstack.org/73331 | 18:01 |
| lifeless | sdague: where we do the isolated project gating - avoiding races within the project, and then it goes into an integrated queue for cross project testing, and then finally lands at the end | 18:01 |
| anteaya | ewindisch: the problem is that rechecks are necessary | 18:01 |
| *** hashar has joined #openstack-infra | 18:01 | |
| *** Ajaeger1 has joined #openstack-infra | 18:01 | |
| lifeless | sdague: this doesn't capture cross-project breakage of internal tests - but we make no commitment to catching that today anyway | 18:01 |
| HenryG | anteaya: ah, ok, I have the text '... see the blueprint for more info...' and it picked that up. | 18:02 |
| jeblair | lifeless: that's essentially what sdaque is proposing | 18:02 |
| sdague | and projects should not be running cross project internal tests | 18:02 |
| anteaya | ewindisch: and the knowledge base of the folks implementing is far away from the knowledge required to implement the ideal system | 18:02 |
| fungi | anteaya: HenryG: i see that, it apparently had trouble identifying the blueprint name or something | 18:02 |
| anteaya | ewindisch: you have correctly identified a gap | 18:02 |
| sdague | lifeless: so I realize there is an exposure, especially if the merge queue gets long | 18:02 |
| lifeless | sdague: projects that are built on top of the very core layers use the core layer clients in their test code | 18:02 |
| jeblair | lifeless: the requirement for a recent positive check result is very similar to that. | 18:02 |
| anteaya | but getting rid of the requirement is not a winning approach | 18:02 |
| HenryG | fungi, anteaya: I assume I can use -t to change the topic in a follow-up patch? | 18:02 |
| anteaya | ewindisch: the ideal approach would be, you must be this tall to ride | 18:03 |
| *** loquacities has quit IRC | 18:03 | |
| anteaya | but we can't do that either | 18:03 |
| fungi | HenryG: i believe so. no guarantees it won't change it back but give that a try on your next patchset | 18:03 |
| lifeless | sdague: e.g. everyone depends on pbr, and we see issues with unit tests there that tempest liekly won't catch | 18:03 |
| sdague | however this is a good approximation of that, and maybe zuul could be extended to cover it | 18:03 |
| lifeless | sdague: and heat depends on neutronclient and novaclient | 18:03 |
| *** AlexF has quit IRC | 18:03 | |
| lifeless | sdague: and its unit tests - rightly or wrongly - involve codepaths within those libraries | 18:03 |
| sdague | lifeless: pbr doesn't currently gate on everyone's unit tests | 18:04 |
| jamespage | apevec, cinder/iscsi and cinder/ceph scenarios test OK - just waiting for the neutron scenario to complete | 18:04 |
| *** morganfainberg_Z has joined #openstack-infra | 18:04 | |
| sdague | so that's not an arguement | 18:04 |
| *** loquacities has joined #openstack-infra | 18:04 | |
| *** morganfainberg_Z is now known as morganfainberg | 18:04 | |
| jeblair | sdague, lifeless: i'm not sure how extending zuul to cover it would be different than either sdague's approach on the one hand, or the current system on the other. the fundamental difference is whether a unit test occupies a position in the gate queue. one approach says no, the other yes. | 18:04 |
| *** jgallard has quit IRC | 18:04 | |
| *** morganfainberg has quit IRC | 18:05 | |
| *** morganfainberg has joined #openstack-infra | 18:05 | |
| *** morganfainberg is now known as Guest86576 | 18:05 | |
| *** Guest86576 is now known as adskfgjsdkl | 18:05 | |
| HenryG | fungi, anteaya: thanks for the insights. No big deal, and happy just to understand things a little better. | 18:05 |
| sdague | jeblair: I think the item would be another tier of queues, so the nova patches that all enter final gate in an order where unit tests are known to work in that order | 18:05 |
| *** hashar is now known as hasharMeeting | 18:05 | |
| sdague | but a reset would kill that anyway | 18:05 |
| *** adskfgjsdkl is now known as needscoffee | 18:05 | |
| lifeless | sdague: I know pbr doesn't - but AIUI there is interest in having it do so | 18:05 |
| anteaya | HenryG: thanks for asking, I didn't know the reason for it either, glad to learn something new myself | 18:05 |
| sdague | lifeless: well we have real issues today | 18:05 |
| fungi | HenryG: we might ought to consider putting more standardization around blueprint header lines in commit messages and then restricting the actions of that hook to that pattern | 18:05 |
| sdague | and lets try to address those | 18:05 |
| Ajaeger1 | SergeyLukjanov, did you solve the mystery of the non-updated Jenkins jobs? | 18:05 |
| lifeless | sure | 18:05 |
| sdague | and solve the future when we get there | 18:06 |
| *** needscoffee is now known as morganfainberg | 18:06 | |
| *** morganfainberg is now known as morganfainberg_z | 18:06 | |
| sdague | because the alternative is kind of give up, and let the gate collapse in i3, imo | 18:06 |
| *** morganfainberg_z is now known as morganfainberg | 18:06 | |
| * anteaya just wants to ensure sdague knows his work is valued | 18:06 | |
| anteaya | because we need you | 18:07 |
| sdague | thanks anteaya :) | 18:07 |
| anteaya | :D | 18:07 |
| *** mriedem has joined #openstack-infra | 18:08 | |
| *** e0ne has joined #openstack-infra | 18:09 | |
| sdague | jeblair / clarkb: https://review.openstack.org/#/c/73366/ the freshness change passed if anyone wants to approve it | 18:09 |
| *** pafuent has left #openstack-infra | 18:10 | |
| * anteaya +1's the freshness change and heads out for a walk | 18:10 | |
| jeblair | sdague: let me think a moment about whether we could do the split you were talking about | 18:10 |
| sdague | ok, no worries | 18:11 |
| sdague | I need to go shovel some snow anyway | 18:11 |
| *** e0ne has quit IRC | 18:12 | |
| *** e0ne has joined #openstack-infra | 18:12 | |
| *** esker has quit IRC | 18:12 | |
| *** esker has joined #openstack-infra | 18:13 | |
| *** thomasbiege has joined #openstack-infra | 18:13 | |
| *** jooools1 has quit IRC | 18:13 | |
| *** derekh has quit IRC | 18:13 | |
| SergeyLukjanov | Ajaeger1, nope :( we need to ask someone from the root team to check logs | 18:14 |
| *** jooools has joined #openstack-infra | 18:15 | |
| *** harlowja_away is now known as harlowja_ | 18:15 | |
| *** pafuent1 has joined #openstack-infra | 18:15 | |
| *** talluri has quit IRC | 18:17 | |
| *** esker has quit IRC | 18:18 | |
| *** talluri has joined #openstack-infra | 18:18 | |
| salv-orlando | Hi! We would like to enable filtering logs by severity when they're displayed through HTTP in our CI (vmware minesweeper). Do you have any pointers? | 18:18 |
| salv-orlando | in other words we would like them to appear as the do on logs.openstack.org | 18:19 |
| fungi | salv-orlando: sdague wrote most of that. the project is openstack-infra/os-loganalyze i believe | 18:19 |
| sdague | yep, that's the project | 18:19 |
| sdague | salv-orlando: it's a pretty simple wsgi filter | 18:20 |
| Ajaeger1 | Who's around from the root team and can tell me why Jenkins is still using a more than 20hours old job? The patch was merged yesterday... | 18:20 |
| fungi | salv-orlando: and if you look in openstack-infra/config in the vhost template for logs.openstack.org you can see how we apply it | 18:20 |
| fungi | Ajaeger1: i am. just a sec let me get back off my lunch computer to a proper system | 18:20 |
| *** nkinder has left #openstack-infra | 18:20 | |
| salv-orlando | fungi: thanks | 18:21 |
| Ajaeger1 | fungi: sure! Just tell me once you've got time... | 18:21 |
| geekinutah | clarkb: hey I had jog0 in my office the other day explaining how we use Rackspace and HP cloud to do gating | 18:22 |
| *** asomya has joined #openstack-infra | 18:22 | |
| *** talluri has quit IRC | 18:22 | |
| geekinutah | clarkb: I'd like to donate some capacity, but not sure if I can support all the API calls that you need (we broke the scheduler for example, so I have to jimmy rig that) | 18:23 |
| geekinutah | clarkb: is there some code you could point me at so I can figure out what else I need to fix? | 18:23 |
| *** hasharMeeting has quit IRC | 18:23 | |
| fungi | geekinutah: we use python-novaclient | 18:24 |
| jaypipes | SergeyLukjanov, fungi, jeblair: apologies for bugging you again on this :( I'm trying to figure out the use of the 'ssh_key' parameter to the jenkins::slave puppet manifest. I had previously thought that this was intended for a private SSH key for that Jenkins slave to register with the master Jenkins, but I am questioning that assumption now, since I see ssh_key = '' in the openstack_proejct::bare_slave class inclu | 18:24 |
| jaypipes | sion. | 18:24 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: narrow the freshness window to 24h https://review.openstack.org/73366 | 18:24 |
| jeblair | sdague: ^ i _think_ that might do it... | 18:24 |
| geekinutah | fungi: okay, but what drives the client? is that nodepool? | 18:24 |
| fungi | geekinutah: so having a functional nova endpoint is the big need. also a usable glance and working cinder are a big plus | 18:24 |
| fungi | geekinutah: yes, nodepool uses novaclient to create and delete nodes | 18:25 |
| jeblair | SergeyLukjanov, fungi: can you double check that you think https://review.openstack.org/73366 will do what i think it will. :) | 18:25 |
| SergeyLukjanov | jeblair, looking on it | 18:25 |
| geekinutah | fungi: glance shouldn't be a problem and we don't run cinder yet | 18:26 |
| *** shardy is now known as shardy_afk | 18:26 | |
| geekinutah | fungi: besides nodepool are there other things that use the nova endpoint? | 18:27 |
| jeblair | fungi, geekinutah: i don't think we need cinder for testing at the moment; only for our long-running servers | 18:27 |
| fungi | geekinutah: we mostly currently just use cinder for infrastructure servers | 18:27 |
| fungi | er, what jeblair said | 18:27 |
| geekinutah | oh, what do the long-running servers do? | 18:28 |
| fungi | jeblair: i believe 73366 will do what it says, but i'm having trouble holding the full state in my head to think it through | 18:28 |
| jeblair | geekinutah: not gating jobs, but public services like gerrit, etc. | 18:28 |
| fungi | geekinutah: we run our gerrit code review, jenkins masters, zuul server(s), nodepool server, wiki, logs, and so on, and so on... on long-running systems | 18:29 |
| jeblair | geekinutah: i think nodepool is the only thing that would interact with the endpoint. | 18:30 |
| SergeyLukjanov | fungi, jeblair, agreed, it looks like the commit message describes it but quite difficult to collect the whole context in head :) | 18:30 |
| openstackgerrit | Bhuvan Arumugam proposed a change to openstack-infra/jeepyb: Load project settings from ini file, if present. https://review.openstack.org/71323 | 18:31 |
| jeblair | sdague: if you're around can you review my update to https://review.openstack.org/73366 | 18:31 |
| geekinutah | jeblair, fungi: cool, I'll check it out, I'm assuming you use nodepool for long running servers also? | 18:31 |
| *** pafuent1 is now known as pafuent | 18:31 | |
| fungi | geekinutah: no, only for single-use systems | 18:31 |
| geekinutah | Like, I couldn't just give you guys some baremetal with IPMI or something? | 18:31 |
| fungi | geekinutah: for bare metal, you may want to talk to lifeless about federating with the tripleo cloud as another region | 18:32 |
| geekinutah | okay, thanks much :-) | 18:32 |
| lifeless | hi | 18:32 |
| lifeless | yes | 18:32 |
| fungi | geekinutah: they're organizing multi-region bare-metal cloud for project use | 18:32 |
| geekinutah | yeah, lifeless and I have chatted previously, I'm still moving that through company politics, budget and such... | 18:33 |
| geekinutah | :-( | 18:33 |
| lifeless | we'd love more contributed machines, we're running checks now, waiting for the second region (RH's) to come on line before we start voting | 18:33 |
| fungi | Ajaeger1: just for context, you're saying that https://review.openstack.org/70594 doesn't seem to have taken effect, or am i looking at the wrong change? | 18:33 |
| lifeless | 'checks' == zuul check jobs upstream | 18:33 |
| Ajaeger1 | fungi, patch: https://review.openstack.org/#/c/71056/ but Jenkins uses old config: https://jenkins04.openstack.org/view/All/job/openstack-operations-guide-ja/configure | 18:34 |
| jaypipes | fungi, jeblair, SergeyLukjanov: do you all have some time today to chat with me about some Jenkins-related questions? maybe 20 minutes or so? | 18:34 |
| sdague | jeblair: lgtm | 18:34 |
| Ajaeger1 | fungi: so, yes, you looked at the wrong patch | 18:34 |
| fungi | Ajaeger1: aha. makes slightly more sense now | 18:35 |
| clarkb | jog0: down to 67k events now if that gives you a feel for how quickly we are keeping up | 18:35 |
| Ajaeger1 | 70594 works just fine, I'm really happy - will be perfect once this is approved: https://review.openstack.org/#/c/73185/ | 18:35 |
| *** andrew_plunk has joined #openstack-infra | 18:35 | |
| bhuvan | SergeyLukjanov: i've rebased https://review.openstack.org/71323, due to conflict. can you review/approve it again? | 18:36 |
| openstackgerrit | Pablo Andres Fuente proposed a change to openstack-infra/config: Added climate-a/climate-m to ENABLED_SERVICES https://review.openstack.org/73375 | 18:36 |
| SergeyLukjanov | bhuvan, adding to the backlog | 18:36 |
| SergeyLukjanov | jaypipes, I'm on a meeting now, sorry | 18:37 |
| jaypipes | SergeyLukjanov: no, I was asking if you had some time today... | 18:37 |
| SergeyLukjanov | fungi, Ajaeger1, I've checked all jenkins node and all of them contain obsolete job config | 18:37 |
| *** thomasbiege has quit IRC | 18:37 | |
| fungi | SergeyLukjanov: yeah, i'm going to dig deeper and see whether jjb is breaking or something | 18:38 |
| fungi | SergeyLukjanov: Ajaeger1: i agree that config change did not seem to get applied to the jobs in question | 18:39 |
| SergeyLukjanov | jaypipes, should have some time today | 18:40 |
| jaypipes | thx. I've been struggling to get this working for a couple days now and I've exhausted my levels of knowledge on Jenkins-related things :( | 18:40 |
| clarkb | SergeyLukjanov must be on a polyphasic sleep cycle | 18:41 |
| *** mriedem has quit IRC | 18:41 | |
| jaypipes | clarkb: do you know why the jenkins master manifest doesn't install a jenkins user, but the jenkins slaves do? | 18:43 |
| bhuvan | jeblair or any other volunteers to review https://review.openstack.org/#/c/72445/, log analyzer for openstack irc logs, using super serious stats ... | 18:43 |
| clarkb | jaypipes: because the jenkins package should install the user for us | 18:44 |
| jeblair | jaypipes: i think there was a problem with that recently the last time we spun up a master. i'm not sure at this point whether we expect puppet to or the jenkins .deb. i think we may have decided that we need puppet to do it for us before installing the deb. | 18:44 |
| fungi | jaypipes: i believe the jenkins user on the master gets created by the jenkins software package installation scripts (and it may be doing it in a way which doesn't jive with our puppetry too, based on findings during the last couple we built). we puppe a jenkins user onto the slaves so we can configure it to accept ssh connections from the masters | 18:44 |
| clarkb | but that assumption may not be true | 18:44 |
| clarkb | re 3rd party requirements and showing people how to do it with gerrit trigger plugin. I think those things are orthogonal | 18:44 |
| *** cadenzajon has joined #openstack-infra | 18:44 | |
| jeblair | fungi: we're like the same person today. :) | 18:45 |
| clarkb | just because gerrit trigger plugin doesn't meet the requirements for rechecking on openstack/* projects doesn't mean it can't be useful for say stackforge projects | 18:45 |
| fungi | jeblair: clearly we should deduplicate our efforts--we'd be much more efficiene that way | 18:45 |
| jaypipes | ok. when trying to figure out what was going on, I just noticed that there was no /home/jenkins on the master, but there was on the slave, so I wondered if that was problematic.. | 18:45 |
| fungi | efficient | 18:45 |
| * jeblair goes to do something else | 18:45 | |
| clarkb | jaypipes: jenkins master homedir is /var/lib/jenkins | 18:45 |
| jaypipes | clarkb: ah.. that makes sense, of course. | 18:46 |
| jaypipes | well.. as much sense as jenkins installations go... | 18:46 |
| *** ywu has quit IRC | 18:46 | |
| fungi | about this ][ much sense | 18:46 |
| jaypipes | hehe | 18:46 |
| * fungi does maxwell smart gesture | 18:47 | |
| jaypipes | clarkb: I gather from reading the nodepool code that the upstream jenkins slaves are set up as JNPL slaves and not SSH-driven slaves. | 18:47 |
| jaypipes | clarkb: how exactly does that work? sorry, I'm not understanding how the masters communicate with the slaves if there are no keys exchanged or anything.. | 18:47 |
| jaypipes | clarkb: does nodepool register each jenkins slave with one of the masters when it launches a node? | 18:48 |
| jaypipes | (me trying to figure out how to set up this simple external testing platform without needing nodepool...) | 18:48 |
| clarkb | jaypipes: they are ssh slaves | 18:49 |
| clarkb | so nothing fancy, nodepool via puppet installs public key into known_hosts for jenkins user, once node is ready nodepool adds it to jenkins as an ssh slave, and the connection is made from maste rto slave | 18:49 |
| jaypipes | clarkb: is that the nodepool_ssh_key variable? | 18:50 |
| jaypipes | used in the install_jenkins_slave.sh script? | 18:50 |
| clarkb | jaypipes: yes it can be, that variable allows you to override our default key easily | 18:50 |
| clarkb | our default key is baked into the puppet | 18:50 |
| jaypipes | oh... | 18:51 |
| fungi | btw, power flickering has started in earnest here... if i disappear longer than now through the weekend, send in the national guard i guess | 18:51 |
| jaypipes | perhaps that is the magic I am missing... | 18:51 |
| *** rossella_s has quit IRC | 18:51 | |
| clarkb | mordred: sdague: a lot of flaky unittest fails aren't due to races in bad code, but instead due to test ordering issues which won't be fixed by disabling one test, it will just affect the next vulnerable test and so on | 18:51 |
| jeblair | fungi: good luck! | 18:51 |
| clarkb | fungi: gl, I saw pictures of burning cars on the freeway outside of raleigh | 18:52 |
| clarkb | so I pretty much assume the zombies have taken over | 18:52 |
| *** marun has quit IRC | 18:52 | |
| apevec | how long should it take for tagged tarball to appear at tarballs.o.o ? | 18:52 |
| fungi | clarkb: i have my shotgun and trusty zombie bat within reach. should be fine through the worst of it | 18:53 |
| apevec | I've pushed https://github.com/openstack/nova/releases/tag/2013.2.2 | 18:53 |
| *** mrfatstrat has joined #openstack-infra | 18:53 | |
| *** jooools has quit IRC | 18:53 | |
| apevec | still not there http://tarballs.openstack.org/nova/?C=M;O=D | 18:53 |
| jaypipes | clarkb: so I created an SSH key on my jenkins slave, supplied that key as the ssh_key parameter to the jenkins::slave manifest, and puppet applied it. I saw the message in puppet output about authorized_keys being added to... I then manually added a node in the master jenkins web UI with that private key and the IP of the slave host... but get this on the jenkins master: http://paste.openstack.org/show/64787/ | 18:53 |
| nibalizer | fungi: locate ham radio | 18:53 |
| jaypipes | clarkb: there are no iptables rules on any of the hosts | 18:53 |
| fungi | nibalizer: already got one | 18:53 |
| *** jooools has joined #openstack-infra | 18:54 | |
| jaypipes | clarkb: and I'm wondering if I am adding the ssh keys in the right place... it's quite confusing to me, sorry :( | 18:54 |
| fungi | apevec: the tarball job may have failed. looking in a moment | 18:54 |
| apevec | fungi, thanks | 18:54 |
| jaypipes | clarkb: perhaps I just need to create *one* key pair and automatically inject that somewhere on all hosts in the system? | 18:54 |
| clarkb | jaypipes: np, the jenkins master will fall back on using the ssh key in /var/lib/jenkins/.ssh | 18:54 |
| clarkb | jaypipes: is that hte key you supplied to the slave? | 18:55 |
| jaypipes | lemme check. | 18:55 |
| clarkb | jaypipes: yes one key, with public and private in /var/lib/jenkins/.ssh on master, then give that public key to all slave known_hosts | 18:55 |
| *** VijayT has quit IRC | 18:55 | |
| clarkb | you can do fancier stuff than that, but that is sort of simple and useable | 18:55 |
| *** asomya has left #openstack-infra | 18:55 | |
| apevec | fungi, looks like it's just nova, other projects are ok | 18:56 |
| openstackgerrit | Bhuvan Arumugam proposed a change to openstack-infra/jenkins-job-builder: allow missing reports for HTML publisher https://review.openstack.org/73378 | 18:59 |
| jaypipes | clarkb: so there is only a single key in /var/lib/jenkins/.ssh/ and it is id_rsa[.pub] that is the value of the $jenkins_ssh_private_key and $jenkins_ssh_public_key parameters to the jenkins::master puppet manifest. The SSH key I added manually in the web interface is not there. Must be stored somewhere else. | 18:59 |
| fungi | apevec: just so you can follow along, git show-ref 2013.2.2 will give you the sha of the tag object (rather than the commit it references). log url for that object's associated jobs is http://logs.openstack.org/c6/c671c2dc66fb983c19193c7b6af7b757d3a1c32d/ | 18:59 |
| jaypipes | clarkb: I'm going to rework things to use a single key. I'll check back with you later. thanks for talking with me. | 19:00 |
| fungi | apevec: tarball job log does indeed indicate failure http://logs.openstack.org/c6/c671c2dc66fb983c19193c7b6af7b757d3a1c32d/release/nova-tarball/3e5ac77/console.html | 19:00 |
| lifeless | whats up with https://review.openstack.org/#/c/71954/ not merging? it has no deps, though it does have something abandonded depending on it | 19:00 |
| apevec | fungi, hmm what did I do? | 19:01 |
| fungi | apevec: for some reason, https://review.openstack.org/p/openstack/nova did not have the tag (that /p/ goes to a local git replica on the same server as gerrit, but it still might be out of sync somehow) | 19:01 |
| clarkb | jaypipes: np | 19:02 |
| *** mriedem has joined #openstack-infra | 19:02 | |
| fungi | apevec: it looks properly signed... checking gerrit now to see if it was just behind for some reason | 19:02 |
| clarkb | lifeless: were its tests lost when the tripleo cloud was down? | 19:03 |
| apevec | fungi, tag is on github so yeah looks like something wasn't in sync in Jenkins? | 19:03 |
| fungi | clarkb: jeblair: mordred: SergeyLukjanov: ^ btw, worth thinking about, maybe release jobs should look directly to gerrit's git server for tags rather than a replica | 19:03 |
| fungi | so as to reduce racyness | 19:03 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: install gerrit core plugins https://review.openstack.org/65368 | 19:03 |
| SergeyLukjanov | fungi, sounds reasonable IMO | 19:03 |
| lifeless | clarkb: they are check-non-vote only | 19:03 |
| clarkb | fungi: ++ | 19:03 |
| clarkb | lifeless: right, but if they never ran zuul would never +2 | 19:04 |
| clarkb | which is why it hasn't merged | 19:04 |
| lifeless | clarkb: Jenkins +1 | 19:04 |
| clarkb | but you need +2 to merge | 19:04 |
| lifeless | clarkb: they aren't set for gate at all, so no, its not that | 19:04 |
| lifeless | clarkb: I know | 19:04 |
| lifeless | clarkb: gate-noop SUCCESS in 0s | 19:04 |
| lifeless | was the only comment until the recheck | 19:05 |
| lifeless | can I reverify no bug perhaps? | 19:05 |
| clarkb | lifeless: not no bug, but you can apply a +1 approve | 19:05 |
| lifeless | ok | 19:05 |
| lifeless | done | 19:05 |
| lifeless | merged | 19:05 |
| lifeless | weird | 19:05 |
| fungi | apevec: review.openstack.org:/var/lib/git/openstack/nova.git/refs/tags/2013.2.2 definitely exists now, so i have to assume some sort of replication delay there. we may see more of these if zuul/jenkins start to get more efficient faster than gerrit does :/ | 19:06 |
| zaro | fungi, jeblair, clarkb : ok, now officially bugging infra-core to review needed changes for gerrit upgrade. first one that needs to go in is https://review.openstack.org/#/c/60348 | 19:06 |
| fungi | apevec: i'll retrigger the tarball job now | 19:06 |
| jeblair | lifeless: the tripelo failure exposed a bug in zuul and so we had to do a hard-stop and lost its queue. if that approval happened around that time, it's possible that was the cause. | 19:06 |
| apevec | fungi, thanks, just wanted to ask how could I retrigger | 19:06 |
| lifeless | jeblair: ah, ack - thanks | 19:06 |
| *** mrfatstrat has quit IRC | 19:06 | |
| jeblair | for the record, that bug is fixed in https://review.openstack.org/#/c/72042/ | 19:07 |
| fungi | apevec: we don't have an easy way. at the moment it's a python utility which is run locally on the zuul server to retrigger post/pre-release/release pipeline jobs manually | 19:07 |
| *** lcestari_ is now known as lcestari | 19:07 | |
| *** mriedem has quit IRC | 19:08 | |
| jeblair | fungi: yes, either gerrit directly, or after the new merger is in place, perhaps via zuul. i think it should dtrt there. | 19:09 |
| *** oanufriev has quit IRC | 19:09 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 19:09 | |
| openstackgerrit | Bhuvan Arumugam proposed a change to openstack-infra/config: log analyzer for openstack IRC logs https://review.openstack.org/72445 | 19:09 |
| zaro | SergeyLukjanov: forgot to put you on the bugging list. look a few lines above. | 19:09 |
| SergeyLukjanov | zaro, see it | 19:10 |
| *** jgrimm has joined #openstack-infra | 19:10 | |
| jeblair | zaro: that changed a lot; why did we go from building buck in the job to installing the vcsrepo on the slave? | 19:10 |
| *** mrmartin has joined #openstack-infra | 19:11 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 19:11 | |
| notmyname | sdague: you mentioned earlier something about swift unit tests failing nearly 10% of the time. can you give me some more info there? I don't see that in what I'm seeing, so I want to make sure that we fix a problem if it's there | 19:11 |
| fungi | jeblair: does trigger-job.py need an update to start passing ZUUL_URL? | 19:12 |
| jeblair | fungi: probably so | 19:12 |
| fungi | jeblair: cool, i'll fix it then | 19:12 |
| fungi | just making sure i hadn't missed something obvious | 19:12 |
| jeblair | zaro: oh, it always did that | 19:13 |
| fungi | apevec: slight delay while i fix our tools to support our tools ;) | 19:13 |
| jeblair | zaro, fungi: i'm sorry i didn't catch this earlier, but do we really want to build and install buck on all the slaves? | 19:14 |
| apevec | fungi, np, I'm uploading other projects | 19:14 |
| *** jomara has joined #openstack-infra | 19:14 | |
| fungi | jeblair: i believe the reasoning is that buck doesn't provide binary releases we can consume, so we have to build buck before building gerrit | 19:15 |
| fungi | zaro: was that right? | 19:15 |
| zaro | fungi, jeblair : yes, fungi is correct. | 19:15 |
| jeblair | fungi, zaro: but can we just do that in the gerrit build job? | 19:15 |
| fungi | jeblair: i assume so. consider it part of the gerrit build process | 19:15 |
| zaro | jeblair: i guess that's possible. clarkb suggested the current implementation. | 19:16 |
| jeblair | fungi, zaro: we don't build and install any other job-specific software in the slave manifest, i feel weird doing it for buck. | 19:16 |
| jeblair | clarkb: thoughts ^? | 19:17 |
| clarkb | I agree it is weird, but I think it is a reasonable option | 19:17 |
| zaro | jeblair: ohh that would me we would need to build buck for any of the 3rd party plugins jobs as well. | 19:17 |
| clarkb | since it doesn't require root and keeps us up to date | 19:17 |
| mrmartin | fungi, jeblair: re, could you check me what was the exact reason behind the failure of this test: https://review.openstack.org/#/c/72294/ this patch also related, because it executes the gate-groups-unittests: https://review.openstack.org/#/c/72921/ | 19:18 |
| fungi | zaro: jeblair: agreed. i overlooked it in reviewing, but that ant call should just live in the (check|gate)-gerrit-buck job, maybe as a builder macro both can include | 19:18 |
| jeblair | clarkb: presumably building buck doesn't require root | 19:18 |
| zaro | so like this change for building javamelody https://review.openstack.org/#/c/70014 | 19:18 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 19:18 | |
| zaro | no building buck does not require root. | 19:18 |
| Ajaeger1 | mrmartin: lost job - some kind of hickup of gerrit ;8 | 19:19 |
| jeblair | clarkb: we tend to take a pretty hard line on what gets included on openstack's unit test slaves; we've said no to installs of other random software before | 19:19 |
| jeblair | mrmartin: it's possible that job wasn't automatically created... | 19:20 |
| clarkb | jeblair: oh I misread, are we installing this at a slave level and not a job level? | 19:20 |
| clarkb | I was thinking we could install it at the job level | 19:20 |
| clarkb | local to the workspace | 19:20 |
| fungi | clarkb: yes, i misread it the same way. glad i'm not the only one | 19:20 |
| jeblair | fungi: have you gotten to looking into the wrong job for Ajaeger1? | 19:20 |
| fungi | jeblair: i'm looking at the syslog on jenkins01 | 19:21 |
| jeblair | fungi: i wonder if Ajaeger1 and mrmartin are seeing manifestations of the same problem | 19:21 |
| mrmartin | jeblair: ok, so it means, that something wrong with the gating scripts, or I can fix it with a simple commit --amend | 19:21 |
| fungi | we had one puppet timeout on the jenkins-jobs exec, but others in the timeframe since that change merged which succeeded. however i don't see puppet noticing that particular yaml file having a change, which is odd | 19:21 |
| fungi | still digging | 19:21 |
| jeblair | mrmartin: when it's fixed you can leave a comment with "recheck no bug" and it will re-run it | 19:22 |
| *** sarob has joined #openstack-infra | 19:22 | |
| pleia2 | Mithrandir: oh! nice meeting you last night :) | 19:23 |
| *** VijayT has joined #openstack-infra | 19:23 | |
| *** alexpilotti has joined #openstack-infra | 19:23 | |
| fungi | jeblair: the file is definitely updated in /etc though | 19:23 |
| jeblair | gate-groups-unittests exists on jenkins01 and 02 | 19:24 |
| jeblair | oops | 19:24 |
| jeblair | mrmartin: the jjb config for that job is broken, it wants to run on "{node}" | 19:25 |
| jeblair | fungi: so there doesn't seem to be a problem with jjb execution related to groups | 19:25 |
| mrmartin | jeblair, yeap, I copy this part from storyboard-webclient, where this {node} value come from? | 19:25 |
| fungi | mrmartin: they specify theirs in the projects.yaml file in that same directory | 19:26 |
| clarkb | jog0: down to 54k now, so slowly chewing through the backlog | 19:27 |
| jeblair | mrmartin: substitunions are only valid for job-templates, but gate-groups-unittests is a job (not a template) | 19:27 |
| mrmartin | fungi: as I do, https://review.openstack.org/#/c/72921/2/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml | 19:27 |
| fungi | jeblair: i found in syslog where puppet saw manuals.yaml receive the update, and then it triggered the jenkins-jobs run which seemed to complete error-free | 19:27 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Fix groups unit tests job https://review.openstack.org/73381 | 19:28 |
| jeblair | mrmartin: ^ | 19:28 |
| mrmartin | jeblair: there's the point. so I need to replace {node} with precise? | 19:28 |
| jeblair | yep | 19:28 |
| SergeyLukjanov | jeblair, heh, I was really sure that it works for jobs too :( | 19:28 |
| *** talluri has joined #openstack-infra | 19:28 | |
| *** sarob has quit IRC | 19:29 | |
| mrmartin | SergeyLukjanov: no prob, part of the continous learning process | 19:29 |
| fungi | jeblair: SergeyLukjanov: mrmartin: oh! i always miss that too. should be job-template | 19:29 |
| fungi | job doesn't expect variable substitution, job-template does | 19:30 |
| mrmartin | fungi: a typical hard to find typo | 19:30 |
| *** tomhe has quit IRC | 19:30 | |
| SergeyLukjanov | mrmartin, fungi, jeblair, probably we'd like it to use bare-precise? | 19:30 |
| devananda | logstash down? | 19:30 |
| clarkb | devananda: yes | 19:30 |
| clarkb | but not really | 19:30 |
| devananda | ? | 19:30 |
| fungi | SergeyLukjanov: well, we have a ton of them we still need to transition to the new node label | 19:30 |
| jeblair | SergeyLukjanov: probably, but it would need a change to zuul's layout.yaml to set the single use flag, so we can take care of it later | 19:30 |
| mrmartin | SergeyLukjanov: we need the php5-cli package | 19:30 |
| devananda | trying to track down why i'm getting py26 failures in ironic's gate recently | 19:31 |
| clarkb | I killed the UI to free up memory so that the gearman server wouldn't OOM. But all of the data is currently be processed and you can query ES directly if you want | 19:31 |
| fungi | devananda: how recently? | 19:31 |
| SergeyLukjanov | jeblair, got it | 19:31 |
| devananda | fungi: this morning. | 19:31 |
| fungi | devananda: just in the past few hours | 19:31 |
| devananda | fungi: we also landed like 10 changes :) | 19:31 |
| devananda | yes | 19:31 |
| SergeyLukjanov | jeblair, so +2 on it | 19:31 |
| fungi | devananda: oh, we also switched the node type for py26 jobs | 19:31 |
| devananda | fungi: i was going to assume it was something we did ... but that seems more likely | 19:31 |
| fungi | devananda: let me have a link to a failure | 19:31 |
| jamespage | apevec, neutron tests passed as well - all good from my perpsective | 19:31 |
| clarkb | unfortunately the gearman server doesn't GC as well as I had hoped so I haven't been able to start the UI now that we are down to 53k gearman jobs | 19:31 |
| mrmartin | jeblair: I thing it won't work with bare-precise because I added the php5-cli package and it was not included in bare | 19:31 |
| devananda | fungi: http://logs.openstack.org/15/67415/15/gate/gate-ironic-python26/399c460/ | 19:31 |
| jeblair | clarkb: is log-gearman-client leaking? | 19:32 |
| clarkb | I think I will need to restart the gearman serer when it hits 0 | 19:32 |
| clarkb | jeblair: yes | 19:32 |
| apevec | jamespage, excellent, thanks for jumping in! | 19:32 |
| jeblair | clarkb: is log-geraman-client also the server? | 19:32 |
| fungi | clarkb: jeblair: also i noticed that nodepool is swapping very heavily again | 19:32 |
| clarkb | jeblair: yup | 19:32 |
| clarkb | they are bundled together :/ | 19:32 |
| apevec | jamespage, release is on its way | 19:32 |
| jamespage | apevec, \o/ | 19:32 |
| *** talluri has quit IRC | 19:32 | |
| jeblair | fungi: not anymore. 19:33:11 up 1:06, 1 user, load average: 0.21, 0.18, 0.15 | 19:33 |
| fungi | gah | 19:33 |
| jeblair | fungi: that probably means we have some zombie nodes. | 19:33 |
| * fungi shakes his fist at the sky in futility and gets back to debugging things | 19:34 | |
| *** dcramer_ has quit IRC | 19:34 | |
| jeblair | fungi: i will remove the nodes that have been building for > 1 hour | 19:35 |
| fungi | jeblair: thanks! | 19:35 |
| *** jooools has quit IRC | 19:37 | |
| fungi | jeblair: so, looking at the openstack-operations-guide-ja job on jenkins01, the /var/lib/jenkins/jobs/openstack-operations-guide-ja/config.xml file reflects the updated job configuration, but the jenkins webui for the same job shows the old configuration | 19:37 |
| *** jooools has joined #openstack-infra | 19:37 | |
| jeblair | fungi: wow! | 19:37 |
| jeblair | fungi: oh! | 19:37 |
| fungi | so for some reason the running jenkins server doesn't know the xml updated? | 19:37 |
| jeblair | fungi: this is a maven->freestyle change isn't it | 19:37 |
| fungi | YES | 19:38 |
| jeblair | fungi: i just remembered that is broken in jenkins and can't be handled by jjb | 19:38 |
| fungi | what dark magic do i not know about there? | 19:38 |
| jeblair | fungi: so we need to delete that job and let jjb recreate it | 19:38 |
| fungi | jeblair: can do | 19:38 |
| jeblair | Ajaeger1: and we'll need to do that for other maven->freestyle changes | 19:38 |
| fungi | Ajaeger1: ^ mystery solved. blame jenkins | 19:38 |
| Ajaeger1 | fungi, jeblair thanks for solving this! | 19:38 |
| devananda | fungi: any leads on the py26 issue? should I just retry for now? | 19:38 |
| fungi | Ajaeger1: i'll get this one fixed asap | 19:39 |
| jeblair | (but it's definetly worth doing; we'll just have to try to remember to do that when we approve thes changes) | 19:39 |
| Ajaeger1 | jeblair, fungi: If I do another maven->freestyl change, should I mark this somehow special so that you know to delete the job first? | 19:39 |
| fungi | devananda: you're 3rd in line. i need to fix one of our tools next so i can help apevec work around a race condition we encountered for the nova stable release | 19:39 |
| *** e0ne has quit IRC | 19:39 | |
| *** e0ne has joined #openstack-infra | 19:40 | |
| fungi | Ajaeger1: or just bug one of us with root access after it merges, because it's possible we'll forget between approving and running | 19:40 |
| clarkb | jeblair: fungi https://review.openstack.org/#/c/73037/ any idea what was causing jobs to be lost this morning? also that change was approved again but not queued | 19:40 |
| Ajaeger1 | fungi: will try to remember. Thanks! | 19:40 |
| clarkb | jeblair: wondering if that is related to the stuff you and sdague were talking about earlier | 19:40 |
| jeblair | clarkb: let's investigate those in the opposite order | 19:40 |
| clarkb | sure | 19:40 |
| devananda | fungi: ack, np :) | 19:41 |
| jeblair | clarkb: i think the enqueue problem is a hole in the 24->72 hour freshness window change | 19:42 |
| *** e0ne_ has joined #openstack-infra | 19:42 | |
| *** mrmartin has quit IRC | 19:42 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 19:42 | |
| fungi | jeblair: we just need to wait another 48 hours and everything will be working as intended, right? ;) | 19:43 |
| *** gyee has quit IRC | 19:43 | |
| jeblair | oh, actually, i don't think it's related to the 24-72 hour change, i think this was an expected behavior | 19:43 |
| jeblair | that -2'd changes would need to have a recheck applied manually, you couldn't just aprv them | 19:44 |
| *** e0ne has quit IRC | 19:45 | |
| jeblair | clarkb: so i think the current state with the new gate configuration is that if jenkins leaves a -2 vote on a patch, you need to do 'recheck bug xxx' to get a clean check result. | 19:46 |
| *** mfer has quit IRC | 19:46 | |
| notmyname | jeblair: when did that change? and in this case, what bug number do I use? | 19:47 |
| jeblair | clarkb: i think that means that reverify is broken | 19:47 |
| boris-42_ | jeblair clarkb could you pls take a look at this patch (adding requirements that is actually already presented) https://review.openstack.org/#/c/73025/ (already have 2x+2) | 19:47 |
| fungi | jeblair: i think recheck followed by reverify might work? | 19:47 |
| jeblair | fungi: yes | 19:47 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 19:47 | |
| *** fbo is now known as fbo_away | 19:47 | |
| fungi | assuming the recheck passes anyway | 19:48 |
| jeblair | notmyname: let's hold 'what bug to use' until i figure out why it was LOST | 19:48 |
| jeblair | sdague: ping | 19:48 |
| notmyname | jeblair: k | 19:48 |
| *** mfer has joined #openstack-infra | 19:48 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 19:49 | |
| notmyname | jeblair: thanks for looking into it. FWIW, I'm (initially) not a fan of removing core reviewer's ability to approve something and have is re-enqueued. but this is the first I've heard of it, so it's still new to me | 19:49 |
| jeblair | sdague, clarkb, fungi: when sdague is back from shoveling snow, let's see about being more clear about the expected process when jenkins leaves a -2; cause i think we broke reverify. | 19:49 |
| clarkb | ok | 19:50 |
| jeblair | notmyname: i think we have a rough edge here that i bet we can work out, but i don't want to go into details without sdague | 19:51 |
| jeblair | so moving on to LOST... | 19:51 |
| jeblair | 2014-02-13 18:15:42,489 INFO zuul.Gearman: Gearman job <gear.Job 0x7f0dde0c9890 handle: H:127.0.0.1:1626291 name: build:gate-tempest-dsvm-full unique: d36b728b64c141e6a7fc2ba3d2f1e2cd> lost due to disconnect | 19:51 |
| Ajaeger1 | infra team, could you quickly approve this one as well, please? This would be soo nice for the docs-drafts: https://review.openstack.org/#/c/73185/ | 19:51 |
| fungi | Ajaeger1: okay, openstack-operations-guide-ja has been deleted and recreated on all the jenkins masters. that's the only job affected by that template so far, right? | 19:52 |
| jeblair | it looks like we lost 3063 jobs around 1815 due to disconnects | 19:53 |
| *** lcheng_ has joined #openstack-infra | 19:53 | |
| Ajaeger1 | fungi: yes, that's the only job you had to touch. | 19:53 |
| Ajaeger1 | I wanted to start the maven migration very slowly... | 19:53 |
| Ajaeger1 | ... with some baby steps ;) | 19:54 |
| *** wenlock_ has joined #openstack-infra | 19:54 | |
| fungi | jeblair: that's *suspiciously* close to when nodepool.o.o got rebooted too | 19:54 |
| fungi | reboot system boot Thu Feb 13 18:26 | 19:55 |
| jeblair | fungi: interesting; zuul has been up that whole time | 19:55 |
| fungi | i don't see any way it could be related, unless they rebooted a bunch of our vms rather than just nodepool | 19:56 |
| fungi | i don't suppose we know if those disconnects were only to rackspace nodes | 19:56 |
| jeblair | fungi: well, because of the way the gearman plugin for jenkins works, there are really only connections to zuul's gearman from the jenkins masters | 19:57 |
| fungi | oh! those are jenkins masters disconnecting from zuul? | 19:58 |
| jeblair | fungi: (there's still a connection for each worker, but they'll all be from the master hosts) | 19:58 |
| jeblair | fungi: or zuul disconnecting from geard | 19:58 |
| fungi | nothing obvious related on status.rackspace.com | 19:58 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 19:59 | |
| jeblair | fungi: and actually my gear log patch would really help in this case | 19:59 |
| clarkb | does cacti show a spike in tcp errors? | 19:59 |
| fungi | rackspace ticket saying "The server 23.253.76.129 has been rate-limited due to excessive queries against our caching nameservers." | 20:00 |
| clarkb | no apparent errors caught by cacti | 20:00 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 20:00 | |
| fungi | no reverse dns on 23.253.76.129 so probably a nodepool node | 20:00 |
| *** pafuent has quit IRC | 20:01 | |
| fungi | and the usual friendly "1st Notice: Server nodepool.openstack.org, Out of Memory/Swapping and Rebooted" | 20:01 |
| *** e0ne has joined #openstack-infra | 20:03 | |
| fungi | zuul says "Last reconfigured: Thu Feb 13 2014 16:25:04 GMT+0000 (UTC)" so this doesn't coincide with a config reload either | 20:03 |
| jeblair | 2014-02-13 18:15:41,797 ERROR gear.Client: Connection <gear.Connection 0x25b7190 host: 127.0.0.1 port: 4730> timed out waiting for a response to a submit job request: <gear.Job 0x7f0e8029dd10 handle: None name: build:gate-horizon-pep8 unique: 1ca1dfc3b17e408f9bad8aad74af7be4> | 20:03 |
| jeblair | so i think zuul abandoned its own connection to its own gearman server | 20:04 |
| *** ArxCruz has quit IRC | 20:04 | |
| *** rfolco has quit IRC | 20:04 | |
| *** mrodden1 has joined #openstack-infra | 20:05 | |
| jeblair | that's a 30 second timeout | 20:05 |
| *** andreaf2 has joined #openstack-infra | 20:05 | |
| *** mrodden has quit IRC | 20:05 | |
| *** DinaBelova is now known as DinaBelova_ | 20:05 | |
| *** andreaf has quit IRC | 20:06 | |
| *** e0ne_ has quit IRC | 20:06 | |
| apevec | fungi, drop me an email when you retrigger tarball job - I'll check later after dinner | 20:06 |
| apevec | all other tarballs are uploaded to LP | 20:06 |
| *** boris-42_ has quit IRC | 20:07 | |
| *** DinaBelova_ is now known as DinaBelova | 20:08 | |
| fungi | apevec: it's running now--i'll let you know once it's there | 20:08 |
| apevec | ok, thanks | 20:08 |
| apevec | fungi, actually I'll see it when it shows up on tarballs.o.o :) | 20:09 |
| sdague | jeblair: back | 20:09 |
| sdague | what's up? | 20:09 |
| *** mrodden1 has quit IRC | 20:10 | |
| fungi | apevec: well, i should say, it's in the gearman queue, but i think it may still be waiting to get picked up by a worker | 20:10 |
| *** dcramer_ has joined #openstack-infra | 20:13 | |
| *** CaptTofu has quit IRC | 20:14 | |
| fungi | apevec: it worked. http://logs.openstack.org/c6/c671c2dc66fb983c19193c7b6af7b757d3a1c32d/release/nova-tarball/3e5ac77,2/console.html http://tarballs.openstack.org/nova/nova-2013.2.2.tar.gz | 20:14 |
| apevec | fungi, thank you! | 20:14 |
| * apevec uploads | 20:14 | |
| jeblair | notmyname: https://bugs.launchpad.net/openstack-ci/+bug/1279963 | 20:15 |
| jeblair | notmyname: use that bug for rechecks please | 20:15 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/zuul: Update trigger script for new zuul url parameter https://review.openstack.org/73392 | 20:15 |
| jeblair | fungi: nodepool does talk to that gear server; i wonder if gear got stuck trying to reply to nodepool and blocked processing of other clients (eg zuul) | 20:16 |
| notmyname | jeblair: thanks | 20:16 |
| fungi | jeblair: quite possible. that could be around the time it ran out of swap | 20:16 |
| * fungi checks cacti | 20:16 | |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=979&rra_id=all | 20:17 |
| fungi | not really | 20:17 |
| fungi | though there is a nice gap in all nodepool's cacti graphs a little while prior | 20:18 |
| jeblair | fungi: it was certainly within the last 15 minutes of the server's life | 20:18 |
| jeblair | fungi: the iowait and swap activity metrics were as high as they were going to get | 20:19 |
| *** jooools has quit IRC | 20:19 | |
| fungi | definitely | 20:19 |
| *** pafuent has joined #openstack-infra | 20:19 | |
| *** saju_m has quit IRC | 20:19 | |
| fungi | i mean, the server was probably thrashing hard for a while there, so it's anybody's guess what could go on under thiose conditions | 20:20 |
| jeblair | fungi: so i think i should fix the nodepool leak; it may be worth a look in gear to see if there's something i need to do to prevent clients from blocking each other. | 20:20 |
| * clarkb is busy trying to spin up a second laptop sorry I haven't been more helpful | 20:20 | |
| sdague | jeblair: so what was the -2 issue that showed up? | 20:20 |
| clarkb | am still watching logstash though under 40k gearman jobs | 20:20 |
| fungi | devananda: okay, looking at your failure now to see if maybe there's something different about our bare-centos6 nodes than the previous static centos6 slaves | 20:21 |
| *** jooools has joined #openstack-infra | 20:21 | |
| sdague | can we get another requirements core to +2 this - https://review.openstack.org/#/c/72776/ ? | 20:21 |
| jeblair | sdague: https://review.openstack.org/#/c/73037/ | 20:21 |
| jeblair | sdague: we have some rough edges around jenkins leaving a -2 | 20:22 |
| sdague | ok | 20:22 |
| sdague | so can we trigger it all over again on a +A toggle? | 20:22 |
| jeblair | sdague: currently you have to "recheck bug xxx" then "reverify bug xxx". | 20:23 |
| jeblair | sdague: possibly. but now i need to go to lunch | 20:23 |
| sdague | or recheck bug, and +A toggle | 20:23 |
| sdague | right? | 20:23 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Replace install-deploy-guide-$LANG jobs https://review.openstack.org/73397 | 20:23 |
| sdague | ok, enjoy lunch | 20:23 |
| jeblair | sdague: thx, let's pick this up when i get back | 20:24 |
| sdague | yep | 20:24 |
| *** jooools has quit IRC | 20:26 | |
| *** cadenzajon has quit IRC | 20:26 | |
| *** melwitt has joined #openstack-infra | 20:27 | |
| *** Sukhdev has joined #openstack-infra | 20:27 | |
| *** jooools has joined #openstack-infra | 20:28 | |
| devananda | fungi: i can get you a few more links to py26 failures if it helps | 20:28 |
| dstanek | bknudson: in https://review.openstack.org/#/c/73026 why is it --log_config instead of --log-config? | 20:29 |
| fungi | devananda: having the earliest one you know of would also be great | 20:30 |
| fungi | devananda: because if it ran on a slave named like centos6-somenumber instead of bare-centos6.stuff then we know it's not related to the node type change | 20:31 |
| bknudson | dstanek: what is --log_config? | 20:31 |
| *** jooools has quit IRC | 20:31 | |
| dstanek | bknudson: that's what i was wondering :-) https://review.openstack.org/#/c/73026/1/doc/source/man/keystone-all.rst | 20:31 |
| bknudson | dstanek: it's probably a bug in oslo.config? | 20:32 |
| *** jooools has joined #openstack-infra | 20:32 | |
| dstanek | bknudson: both of those see to work, but you change the docs to use the _ version; i didn't know if that was intentional or not | 20:33 |
| bknudson | dstanek: not sure why you want to discuss this on -infra? | 20:33 |
| devananda | k, looking | 20:33 |
| devananda | fwiw, having logstash would help ;) | 20:33 |
| *** boris-42_ has joined #openstack-infra | 20:34 | |
| dstanek | bknudson: oops wrong channel | 20:34 |
| openstackgerrit | Terry Howe proposed a change to openstack-infra/gitdm: Terry Howe added to other files https://review.openstack.org/71651 | 20:34 |
| fungi | devananda: also, if test__get_node_vif_ids_no_ports is likely to use particular system libraries, that would be good to know too. we at least discovered that rackspace didn't necessarily install all the same packages by default on the pvhvm centos images as on the traditional non-pvhvm ones, so it could be some unpuppeted system dependency we're suddenly missing | 20:34 |
| *** senthil has quit IRC | 20:35 | |
| dtroyer | jeblair, fungi, clarkb: When you get a minute for a not-fix-broken-gate review this one is hopefully simple: https://review.openstack.org/#/c/71318/ to add devstack top openstackclient's gate…we now have 2 devstack reviews that use OSC, I don't want the approve them without this change | 20:37 |
| devananda | fungi: "DriverNotFound: Failed to load driver fake_pxe" seems to be the error, which is coming from stevedore | 20:38 |
| *** jooools has quit IRC | 20:38 | |
| devananda | we wrap the stevedore error and raise that | 20:38 |
| *** jooools has joined #openstack-infra | 20:40 | |
| *** jooools has quit IRC | 20:40 | |
| apevec | If anyone has a second, please proof-read https://wiki.openstack.org/wiki/ReleaseNotes/2013.2.2 | 20:40 |
| *** jooools has joined #openstack-infra | 20:42 | |
| *** oanufriev has joined #openstack-infra | 20:46 | |
| *** CaptTofu has joined #openstack-infra | 20:48 | |
| annegentle | apevec: nothing comes up in spellcheck nor read-through... I am not the best copy editor but wanted to take a look :) | 20:48 |
| apevec | annegentle, thanks, much appreciated! | 20:49 |
| annegentle | apevec: you might say "low risk" rather than "risk free" as nothing is free of risk :) | 20:49 |
| annegentle | apevec: otherwise, lgtm | 20:49 |
| apevec | good point, I'll update our template | 20:49 |
| *** bhuvan has quit IRC | 20:50 | |
| *** Sukhdev has quit IRC | 20:50 | |
| openstackgerrit | Diane Fleming proposed a change to openstack-infra/config: OpenStack Image Service API v2 Reference link broken https://review.openstack.org/73331 | 20:50 |
| *** gokrokve has joined #openstack-infra | 20:51 | |
| *** yolanda has quit IRC | 20:51 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 20:51 | |
| *** bhuvan has joined #openstack-infra | 20:52 | |
| *** oanufriev has quit IRC | 20:53 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 20:54 | |
| *** jooools has quit IRC | 20:54 | |
| fungi | devananda: if it helps, here's the difference in system packages between the old static centos6 slaves and the new bare-centos6 nodepool nodes... http://paste.openstack.org/show/65171/ | 20:54 |
| apevec | annegentle, although, we do say "relatively risk free" which is rather strange wording, now that I read it | 20:55 |
| *** jooools has joined #openstack-infra | 20:55 | |
| fungi | devananda: i'd pay particular attention to any of those which are python libraries, if ironic is using system site packages in tox/virtualenv like nova does | 20:55 |
| fungi | apevec: you can blame the english language. it's full of horrible self-contradictory phrases like that one | 20:56 |
| apevec | annegentle, "low risk" is better | 20:56 |
| apevec | fungi, that's what makes it fun for me as a non-native speaker | 20:56 |
| fungi | apevec: i'll grant you it's fun enough as a native speaker | 20:57 |
| Ajaeger1 | infra team, could you review and approve Diane's patch https://review.openstack.org/#/c/73331/, please? This is needed since she reworked the image-api repo a bit | 21:00 |
| *** andre__ has quit IRC | 21:00 | |
| Mithrandir | maybe "no known regressions"? | 21:01 |
| *** andreaf2 has quit IRC | 21:01 | |
| lifeless | got another approved-but-not-merging https://review.openstack.org/#/c/73078/ | 21:03 |
| lifeless | suggestions? ANd this is new, so definitely not the preior disucssion | 21:03 |
| *** denis_makogon_ has joined #openstack-infra | 21:05 | |
| fungi | lifeless: it's got a recent successful check run. i wonder if another side effect of the recent change is that approvals which are applied before the check completes get ignored by zuul... | 21:05 |
| fungi | jeblair: ^ (when you're back) | 21:05 |
| pafuent | Hi. Can someone take a look at https://review.openstack.org/#/c/73375/? This will help Climate for incubation. BTW the change is only one line | 21:05 |
| *** smarcet has left #openstack-infra | 21:05 | |
| devananda | hm, after a gate failure, does re-ticking +A no longer trigger jenkins to retry? | 21:06 |
| fungi | devananda: you need to recheck first at the moment | 21:06 |
| fungi | recheck bug somenumber | 21:06 |
| devananda | fungi: some patches are coming back fine now. are you running a mix of py26 hosts? | 21:06 |
| lifeless | fungi: so, I'll hit +A and see what happens? | 21:06 |
| lifeless | /? | 21:06 |
| fungi | lifeless: right, see if that kicks it | 21:06 |
| *** rlandy has quit IRC | 21:06 | |
| fungi | devananda: no, at this point they should all be bare-precise nodepool nodes, but i'll make sure jenkins isn't still doing something wierd like handing some jobs to the old nodes | 21:07 |
| devananda | fungi: is there a bug for the work ya'll did this morning? | 21:07 |
| *** ociuhandu has quit IRC | 21:07 | |
| devananda | otherwise, there's just a few left, and i'll recheck no bug | 21:07 |
| lifeless | fungi: yes, that tickled it. I like your theory. | 21:07 |
| fungi | devananda: looks like jenkins isn't assigning any new jobs to the old static centos6 slaves at this point. your py26 errors aren't happening any longer? | 21:08 |
| fungi | oof, power blackouts are worsening | 21:08 |
| *** apevec has quit IRC | 21:08 | |
| devananda | fungi: a few tests that finished in the last 15 min were fine. i'm retrying the ones that failed | 21:08 |
| devananda | may have jsut been transitory | 21:09 |
| *** thedodd has joined #openstack-infra | 21:09 | |
| fungi | devananda: might have been something in one of your requirements then, which got released broken and then a quick brown bag release to fix. you could compare pip freeze results between runs | 21:09 |
| *** denis_makogon_ is now known as denis_makogon | 21:10 | |
| *** DinaBelova is now known as DinaBelova_ | 21:10 | |
| jeblair | dtroyer: wow, uh, is openstackclient an official project? | 21:11 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 21:11 | |
| mgagne | Is check gate run on -1? They look to have been triggered by my comment: https://review.openstack.org/#/c/69511/ | 21:13 |
| mgagne | Same here with -2: https://review.openstack.org/#/c/71108/ | 21:14 |
| jeblair | fungi: i think your theory is correct | 21:14 |
| fungi | mgagne: it's triggered on any comment if it's been a while since the last check run | 21:14 |
| mgagne | fungi: oh, funny. thanks! | 21:14 |
| *** Ajaeger1 has quit IRC | 21:14 | |
| fungi | mgagne: currently 48 hours, soon to probably be 24 | 21:14 |
| mgagne | fungi: a while means? 48h ? | 21:15 |
| *** lcostantino has quit IRC | 21:15 | |
| * fungi nods | 21:15 | |
| jeblair | fungi: i think that merged | 21:15 |
| mgagne | ty | 21:15 |
| jeblair | fungi: except it should be 72 if not approved | 21:15 |
| jeblair | 24 if approved | 21:15 |
| fungi | oh, right | 21:15 |
| fungi | i keep forgetting that stretched the unapproved rechecks back to 72 | 21:16 |
| jeblair | fungi: but back to your theory... actually it should have triggered the gate pipeline; that's supposed to trigger on jenkins +1 | 21:16 |
| * SergeyLukjanov going to write the note about it somewhere | 21:16 | |
| fungi | good point... so something about that didn't match up... looking at the zuul debug log to see if i can tell what | 21:16 |
| jeblair | sdague: i'm back, you still here? | 21:17 |
| jeblair | dtroyer: i really like the idea of openstackclient; i'm just thinking that if we're going to start using it everywhere, including in the gate, it ought to be an official project | 21:18 |
| *** pafuent has left #openstack-infra | 21:18 | |
| jeblair | dtroyer: hrm, we put it in openstack/; it's just not listed in openstack/governance | 21:19 |
| jeblair | dtroyer: who's running it? | 21:19 |
| openstackgerrit | Doug Hellmann proposed a change to openstack/requirements: Add cmd2 for cliff https://review.openstack.org/73412 | 21:19 |
| jeblair | fungi: oh, the 24h change hasn't merged due to the same thing you're looking into. | 21:21 |
| fungi | relevant line for the start of zuul seeing its own comment is i think at... | 21:22 |
| fungi | 2014-02-13 20:15:57,499 DEBUG zuul.Gerrit: Change <Change 0x7f0ddc187a50 73078,4> status: NEW | 21:22 |
| dtroyer | jeblair: I run it… it isn't official, and was created before any of the guidelines were in place for repos, etc (almost 2 years ago) | 21:22 |
| *** derekh has joined #openstack-infra | 21:23 | |
| openstackgerrit | A change was merged to openstack-infra/config: narrow the freshness window to 24h https://review.openstack.org/73366 | 21:23 |
| jeblair | dtroyer: it's all coming back to me now. :) we should find a program for it... | 21:23 |
| *** marun has joined #openstack-infra | 21:24 | |
| dtroyer | jeblair: we've talked about sticking it under the Oslo program, dhellmann was kind enough to give me some slots in HK | 21:24 |
| dtroyer | jeblair: but I think longer-term it makes sense with the SDK stuff that should be coming up soon | 21:24 |
| fungi | i mean, it's called "openstack/python-openstackclient" so how much more official can it get? says openstack twice right there ;) | 21:24 |
| jeblair | dtroyer: yes, i agree | 21:24 |
| jeblair | fungi: they cancel out | 21:24 |
| jeblair | dtroyer: openstackclient + sdk may make a good fit | 21:25 |
| *** CaptTofu has quit IRC | 21:25 | |
| dtroyer | jeblair: I'm not sure about the timing of that though | 21:25 |
| jeblair | dtroyer: devstack-gate change lgtm; aprvd | 21:26 |
| fungi | yeah, it does match up with the other general service-agnostic openstack sdks for various languages | 21:26 |
| dtroyer | jeblair: thanks | 21:26 |
| *** cadenzajon has joined #openstack-infra | 21:26 | |
| dtroyer | fungi: which is why I'm sticking my nose into that work too :) | 21:27 |
| sdague | jeblair: yes | 21:27 |
| *** jooools has quit IRC | 21:27 | |
| *** Clabbe has quit IRC | 21:28 | |
| *** jooools has joined #openstack-infra | 21:28 | |
| *** mrmartin has joined #openstack-infra | 21:28 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Restore expected reverify/reapprove behavior https://review.openstack.org/73418 | 21:28 |
| *** mrodden has joined #openstack-infra | 21:29 | |
| jeblair | sdague: okay, while i was waiting, i went ahead and threw that together ^ but we can talk about whether that's desirable | 21:29 |
| openstackgerrit | A change was merged to openstack-infra/config: Gate openstackclient on DevStack https://review.openstack.org/71318 | 21:29 |
| *** amcrn has quit IRC | 21:29 | |
| *** mriedem has joined #openstack-infra | 21:29 | |
| jeblair | sdague: so i think the current state is that we broke reverify and reapprove because they will never satisfy the freshness critera (which is all about having a +1; there's no current way to get to a +1 with those actions) | 21:30 |
| *** alexpilotti has quit IRC | 21:30 | |
| jeblair | sdague: so that change makes it so that reverify now triggers the _check_ pipeline iff jenkins has left a -2 | 21:30 |
| devananda | fungi: what's the difference between http://logs.openstack.org/82/70382/4/check/gate-ironic-python26/ca57a41 and http://logs.openstack.org/21/66621/6/check/gate-ironic-python26/b37dde3 | 21:30 |
| jeblair | sdague: it also makes it so that an approved vote triggers the _check_ pipeline iff jenkins has left a -2 | 21:31 |
| devananda | fungi: first one passed, second one failed, within a minute of each other. | 21:31 |
| openstackgerrit | A change was merged to openstack-infra/config: Added climate-a/climate-m to ENABLED_SERVICES https://review.openstack.org/73375 | 21:31 |
| jeblair | sdague: reverify has no effect on the gate now. but the +1 result from check runs should trigger gate | 21:31 |
| *** cody-somerville has quit IRC | 21:31 | |
| fungi | jeblair: so, looking at the debug log, zuul saw its vrif:1 event on the stream, and then again at 20:19 it saw when its parent change (71725,3) got enqueued to the gate pipeline, but didn't enqueue it then either | 21:31 |
| fungi | i'm going to stare at the event config patch again for a bit and see if the answer just comes to me | 21:33 |
| *** cody-somerville has joined #openstack-infra | 21:33 | |
| *** cody-somerville has quit IRC | 21:33 | |
| *** cody-somerville has joined #openstack-infra | 21:33 | |
| jeblair | fungi: ugh, what if there's a race condition and gerrit wasn't reporting the verified vote when zuul queried it | 21:33 |
| jeblair | fungi: (like gerrit sends out events, and then, a few ms later it updates its cache/db so that queries report the same data) | 21:34 |
| fungi | zuul gets the vrif:1 comment-added event on the stream. does it then go recheck gerrit to find out whether that's legit? | 21:34 |
| *** mrmartin has quit IRC | 21:35 | |
| jeblair | fungi: yeah, it's not too bright that way. basically each time there is an event from gerrit, zuul requeries the associated change so it has all of the latest data | 21:35 |
| jeblair | fungi: but it doesn't try to match what gerrit is doing though. | 21:36 |
| fungi | then that sounds like a potential design flaw in our assumptions | 21:36 |
| jeblair | fungi: we'd need to verify that's the case. i'm just brainstorming. | 21:36 |
| fungi | i agree it's a distinct possibility | 21:36 |
| jeblair | fungi: i think we can find out easily by enabling zuul's gerrit debug logging | 21:38 |
| jeblair | (which will require a restart i think) | 21:38 |
| *** derekh is now known as derekh_afk | 21:39 | |
| *** gokrokve has quit IRC | 21:39 | |
| *** gokrokve has joined #openstack-infra | 21:40 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Enable zuul gerrit debug logging https://review.openstack.org/73423 | 21:40 |
| jeblair | fungi: ^ | 21:40 |
| *** oanufriev has joined #openstack-infra | 21:40 | |
| *** vkozhukalov has quit IRC | 21:41 | |
| fungi | devananda: same type of node on the same provider built from the same image, consistent pip freeze output (eyeballing it, so it's possible i missed something). my guess is either a nondeterministic problem in the code, or one of those patches is at fault? | 21:42 |
| devananda | fungi: neither patch changes code near the error. yay nondeterminism | 21:43 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix groups unit tests job https://review.openstack.org/73381 | 21:43 |
| devananda | fungi: thanks for checking on the node types to confirm it is in fact ironic | 21:44 |
| *** gokrokve has quit IRC | 21:44 | |
| fungi | devananda: well, i'm still not 100% sure, but sure enough that i'm comfortable it's not likely impacting any other projects, and so have other things i need to look at now | 21:44 |
| *** hemna has joined #openstack-infra | 21:45 | |
| devananda | fungi: ack :) | 21:45 |
| openstackgerrit | A change was merged to openstack/requirements: Add requirements for taskflow https://review.openstack.org/73342 | 21:47 |
| *** ok_delta has quit IRC | 21:47 | |
| jhesketh | Hey | 21:49 |
| jeblair | jhesketh: hello | 21:49 |
| *** jhesketh__ has joined #openstack-infra | 21:50 | |
| *** jooools has quit IRC | 21:52 | |
| *** jooools has joined #openstack-infra | 21:53 | |
| jhesketh | hey jeblair. If you have some time to take a look at my swift changes to make sure I'm on the right track before I write some tests that'd be awesome https://review.openstack.org/#/c/68297/ | 21:53 |
| *** CaptTofu has joined #openstack-infra | 21:55 | |
| hemna | any chance I can get a +3 on my requirements change ? https://review.openstack.org/#/c/72776/ | 21:56 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-dev/pbr: Updated from global requirements https://review.openstack.org/73119 | 21:58 |
| *** MIDENN_ has joined #openstack-infra | 21:59 | |
| fungi | hemna: i think vishy is the only person who wields the transcendent +3 vote | 22:00 |
| *** gokrokve has joined #openstack-infra | 22:01 | |
| fungi | hemna: anyway, the people who regularly +2 requirements changes are listed at https://review.openstack.org/#/admin/groups/131,members | 22:01 |
| *** dcramer_ has quit IRC | 22:03 | |
| clarkb | down to 8k events | 22:03 |
| clarkb | when it hits 0 I will restart kibana and puppet on that node after restarting the leaky process | 22:04 |
| hemna | fungi, ok thanks | 22:04 |
| fungi | jeblair: a race is feeling more likely... look at the sequence of check result, approval and gating on your 73423 change | 22:05 |
| fungi | jeblair: oh, or did you use zuul enqueue from the command line on that one? | 22:05 |
| *** arborism has joined #openstack-infra | 22:05 | |
| anteaya | hemna: markmcclain is looking at it | 22:06 |
| fungi | jeblair: particularly since that change should (in theory) have emitted no events for ~6 minutes before it was spontaneously enqueued | 22:06 |
| hemna | thank you | 22:06 |
| openstackgerrit | A change was merged to openstack-infra/config: Enable zuul gerrit debug logging https://review.openstack.org/73423 | 22:07 |
| markmcclain | hemna: approved | 22:08 |
| anteaya | sdague: who is in charge of culling the requirements-core list? I am wondering having Dave Walker, Joe Heck and Gabriel Hurley listed is still accurate. | 22:08 |
| anteaya | also should anyone else be added? | 22:08 |
| *** arborism is now known as amcrn | 22:09 | |
| anteaya | hemna: you are +A'd | 22:09 |
| lifeless | redeploying the testenvironments for the tripleo ci cloud | 22:09 |
| *** skraynev is now known as skraynev_afk | 22:09 | |
| lifeless | this will stall jobs, but they should recover shortly | 22:09 |
| *** dcramer_ has joined #openstack-infra | 22:11 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Don't update pbr's setup.py https://review.openstack.org/73431 | 22:11 |
| SergeyLukjanov | dhellmann, crazy 2am patch :) ^^ | 22:12 |
| *** mfer has quit IRC | 22:13 | |
| *** pdmars has quit IRC | 22:13 | |
| dhellmann | SergeyLukjanov: do you think it's better to look at the contents of setup.py or the path to detect pbr? | 22:13 |
| hemna | anteaya, thank you | 22:13 |
| jeblair | fungi: oh, i think i reapproved it but i guess gerrit didn't record that | 22:14 |
| SergeyLukjanov | dhellmann, 50-50, it could be used manually by users, so, it's hard to rely on path | 22:14 |
| fungi | weiiiird | 22:14 |
| dhellmann | SergeyLukjanov: good point | 22:14 |
| jeblair | fungi: i think it's the thing clarkb is fixing? did you see 2 aprv events for it? | 22:15 |
| dhellmann | SergeyLukjanov: how about looking at the name in setup.cfg? I'm worried if we ever do change the way that setup.py works we'll break this and not realize it | 22:15 |
| clarkb | jeblair: fungi sounds like it | 22:15 |
| SergeyLukjanov | dhellmann, that's a good point too :) | 22:15 |
| dhellmann | SergeyLukjanov: crazy 5 PM thinking ;-) | 22:15 |
| SergeyLukjanov | dhellmann, good option to look into the setup.cfg, will upload a new patch | 22:16 |
| SergeyLukjanov | dhellmann, like this idea much more than relying on content of setup.py | 22:16 |
| *** thomasem has quit IRC | 22:17 | |
| *** senthil has joined #openstack-infra | 22:17 | |
| *** thomasem has joined #openstack-infra | 22:17 | |
| *** thomasem has quit IRC | 22:17 | |
| mordred | ++ | 22:17 |
| mordred | nice solution | 22:17 |
| mordred | dhellmann: so - I had this crazy idea ... | 22:18 |
| * dhellmann reaches for the bourbon | 22:18 | |
| mordred | dhellmann: which is that, as part of sdist in a pbr project, we generate a setup.py that is self-contained and doesn't use pbr | 22:18 |
| mordred | dhellmann: so that a tarball on pypi behaves as a tarball on pypi does with no crazy | 22:18 |
| * dhellmann thinks | 22:18 | |
| mordred | dhellmann: and then write a pbr command-line tool that you can install on your system if you're going to be developing from git repos on projects that use pbr | 22:19 |
| dhellmann | could pbr just generate all of the metadata and put it in the package with a generic setup.py? | 22:19 |
| mordred | and not have a setup.py in the git repo at all | 22:19 |
| mordred | dhellmann: it already does that | 22:19 |
| dhellmann | except that the setup.py we have now also wants pbr | 22:19 |
| mordred | (except we might be talking about different thigns) | 22:19 |
| dhellmann | is all of the metadata already in the package? or is some being generated at install time? | 22:20 |
| dhellmann | mordred: either way, I like the idea of dropping the install time use of pbr | 22:20 |
| *** dstanek has quit IRC | 22:20 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Don't update pbr's setup.py https://review.openstack.org/73431 | 22:20 |
| mordred | dhellmann: all the metadata is in the sdist tarball - it's self-contaiend - EXCEPT, it setup_requires pbr | 22:21 |
| dhellmann | mordred: we would need to do something for the tox case, right? | 22:21 |
| SergeyLukjanov | dhellmann, mordred, ^^ | 22:21 |
| fungi | jeblair: it saw the approval event reflected in the gerrit comment histry at 21:45 and then one more at 21:59 coinciding with when it started enqueuing | 22:21 |
| fungi | based on my reading of the debug log | 22:21 |
| fungi | clarkb: is that what you're tracking? | 22:21 |
| SergeyLukjanov | mordred, dhellmann, do we really need to increasy complexity of pbr's usage? | 22:21 |
| SergeyLukjanov | (crazy question) | 22:22 |
| dhellmann | SergeyLukjanov: I'd rather parse that file but that looks close to right | 22:22 |
| clarkb | fungi: I am fixing https://gerrit-review.googlesource.com/#/c/54490/1 upstream | 22:22 |
| dhellmann | SergeyLukjanov: the idea is to find a way to reduce it, by making it something that distributors of packages use, but installers do not need to know about | 22:22 |
| clarkb | fungi: tl;dr is that sometimes when you don't change the info going into a comment it is never recorded in the DB but events are generated | 22:22 |
| clarkb | like a phantom comment | 22:22 |
| mordred | dhellmann: nope. tox is easy - tox already has requirements list - and it processes them before it processes the local repo | 22:22 |
| dhellmann | mordred: so the requirements would include pbr, and then what? | 22:23 |
| mordred | so we could tell it to requirements = pbr - and then we could have the install command be "pbr install ." or some shit | 22:23 |
| *** e0ne has quit IRC | 22:23 | |
| fungi | clarkb: aha, that sounds right then | 22:23 |
| dhellmann | mordred: is that install command used just for the current package, or for all things? | 22:23 |
| jeblair | fungi: so yeah, it still missed enqueuing based on the verify+1, and only enqueued after i gave it a second aprv which is not in the ui because i left no text both times | 22:23 |
| mordred | SergeyLukjanov: I'm actually trying to make it less complex | 22:23 |
| mordred | dhellmann: jsut for current package | 22:23 |
| mordred | SergeyLukjanov: but I might be failing :) | 22:23 |
| dhellmann | mordred: ok, that was never clear to me | 22:23 |
| *** e0ne has joined #openstack-infra | 22:23 | |
| SergeyLukjanov | dhellmann, I understand you :) | 22:24 |
| * fungi needs to heat up some dinner... bbiab | 22:24 | |
| jeblair | fungi: this is going to explode zuul's logs | 22:24 |
| jeblair | fungi: so i'm going to delete some old ones to make room | 22:24 |
| dhellmann | mordred: I like that direction | 22:24 |
| jeblair | fungi: root fs is at 80& | 22:24 |
| dhellmann | mordred: I wonder if we could get a GSoC project to work on some of this? | 22:24 |
| *** gokrokve has quit IRC | 22:25 | |
| mordred | clarkb: are the upstream people ok with you fixing that? | 22:25 |
| mordred | dhellmann: zomg. then I'd actually have to clearly write down thoughts | 22:25 |
| *** gokrokve has joined #openstack-infra | 22:25 | |
| dhellmann | mordred: life is all about trade-offs | 22:25 |
| jeblair | dhellmann: ooh, you know what, we never applied to gsoc for infra because we were working on other peoples tools... we have enough of our own now i bet we could do something. | 22:25 |
| mordred | dhellmann: in a form other than "dhellman: I had a crazy idea about pbr" | 22:25 |
| mordred | jeblair: ++ | 22:25 |
| anteaya | mordred: or ensure dhellmann and GSoC had lots of bourbon | 22:25 |
| dhellmann | jeblair: quick! deadline is tomorrow | 22:25 |
| clarkb | mordred: they seemed to be, dborowitz indicated it should be fixed and I think my second patchset meets his criteria, still need to figure out buck and test it though | 22:26 |
| *** dcramer_ has quit IRC | 22:26 | |
| jeblair | dhellmann: maybe next year then. i don't want to do a rush job (i think it merits more time and consideration than i can give it by tomorrow) | 22:26 |
| mordred | jeblair: I just forwarded you an email about mentoring org applications | 22:26 |
| mordred | kk | 22:26 |
| dhellmann | jeblair: definitely | 22:26 |
| mordred | anteaya: :) | 22:26 |
| dhellmann | anteaya: :-) | 22:27 |
| *** gokrokve_ has joined #openstack-infra | 22:27 | |
| sdague | jeblair: ok back | 22:28 |
| sdague | sorry about that | 22:28 |
| *** e0ne has quit IRC | 22:28 | |
| jeblair | sdague: np | 22:28 |
| *** marun has quit IRC | 22:29 | |
| clarkb | down to 2.3k gearman jobs | 22:29 |
| *** gokrokve has quit IRC | 22:29 | |
| jeblair | clarkb: i'm going to hard-stop zuul; remove git repos, then start it again | 22:30 |
| jeblair | clarkb: this is mostly to pick up the logging change | 22:30 |
| dhellmann | can I ask where things stand with that bug that means importing new repos is a manual task? I know there was some debugging work going on | 22:30 |
| jeblair | clarkb: but i figure we'd clean the repos up while it's down | 22:31 |
| anteaya | dhellmann: so far we have logging | 22:31 |
| jaypipes | clarkb: I hate to bother you again, but I'm really struggling to get stuff to work in my testing platform. Do you have some time to do a hangout with me? | 22:31 |
| jeblair | clarkb: since we're about to get busy | 22:31 |
| anteaya | and the script sometimes works automatically | 22:31 |
| dhellmann | anteaya: ok, thanks | 22:31 |
| anteaya | I have it on my list of things to do to stand up a mock gerrit and test the manage-projects script | 22:31 |
| jeblair | clarkb: does that work for you? | 22:31 |
| anteaya | but so far have been distracted by expense reports, which are now done | 22:32 |
| * anteaya hopes mordred has checked his inbox | 22:32 | |
| jeblair | dhellmann: do you have openstack projects to import? we'll just do those; don't let the lack of automation slow you down | 22:32 |
| anteaya | dhellmann: so my obstacles are decreasing | 22:32 |
| dhellmann | oslo has 2 libraries we're working on setting up, so if we could schedule some time to take care of those, maybe early next week, it would help | 22:32 |
| dhellmann | jeblair: sure, I'm trying to be careful with your time, too | 22:32 |
| jeblair | dhellmann: are they stackforge renames? | 22:32 |
| dhellmann | jeblair: there are changes up for oslo.vmware and oslo.test | 22:32 |
| dhellmann | jeblair: no, brand new ones | 22:32 |
| dhellmann | well, importing from github | 22:33 |
| openstackgerrit | Boris Pavlovic proposed a change to openstack/requirements: Adding Mako>=0.4.0 to global requirements https://review.openstack.org/73025 | 22:33 |
| dhellmann | jeblair: https://review.openstack.org/#/c/70761/ and https://review.openstack.org/#/c/73036/ | 22:33 |
| jeblair | dhellmann: ok. scheduling can be pretty loose then, no downtime required, you'll just want to know when the switch actually happens | 22:33 |
| dhellmann | jeblair: yes, we're not committing to those upstream repos, just providing them as the source for importing stuff | 22:34 |
| jeblair | clarkb: i'm taking your silence as assent. :) | 22:34 |
| *** jnoller_ has quit IRC | 22:34 | |
| jeblair | #static alert restarting zuul for a configuration change | 22:34 |
| jeblair | #status alert restarting zuul for a configuration change | 22:34 |
| openstackstatus | NOTICE: restarting zuul for a configuration change | 22:34 |
| *** ChanServ changes topic to "restarting zuul for a configuration change" | 22:35 | |
| *** tru_tru has quit IRC | 22:35 | |
| *** dstanek has joined #openstack-infra | 22:36 | |
| fungi | jeblair: sorry, popping back in while stuff heats up. sounds good to me | 22:36 |
| jeblair | fungi: cool, in progress | 22:36 |
| jeblair | it's cloning now | 22:36 |
| fungi | we blew away old zuul logs the last time we rebuilt the server too (well, i think we haven't deleted the old server yet, but anyway...) | 22:37 |
| jaypipes | WARNING zuul.Scheduler: Project openstack-dev/sandbox not found <-- how does one track down why that would be... | 22:38 |
| jeblair | jaypipes: it means the project isn't defined in the layout | 22:39 |
| openstackgerrit | Boris Pavlovic proposed a change to openstack/requirements: Adding Mako>=0.4.0 to global requirements https://review.openstack.org/73025 | 22:39 |
| jaypipes | jeblair: *sigh*... debugged that yesterday, forgot to change my source repo to put projects: instead of - projects: in the layout.yaml. Ty sir. | 22:40 |
| * vishy votes for hemna | 22:42 | |
| *** dstanek has quit IRC | 22:42 | |
| zaro | pleia2: how do you determine which python version is on which ubuntu release? | 22:42 |
| pleia2 | zaro: http://packages.ubuntu.com/search?keywords=python | 22:43 |
| pleia2 | that will show you defaults | 22:43 |
| anteaya | vishy: should hemna be on requirements-core? | 22:43 |
| jeblair | #status ok | 22:43 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 22:43 | |
| zaro | pleia2: thanks. | 22:43 |
| pleia2 | zaro: can search for python3 to see what python3 is has: http://packages.ubuntu.com/search?keywords=python3 | 22:43 |
| anteaya | vishy: was that your meaning? | 22:43 |
| pleia2 | (you can install python3 instead of just python) | 22:44 |
| vishy | anteaya: no he asked for a +3 and fungi said I weilded the transcendant +3 vote | 22:44 |
| anteaya | ah, sorry I mis-understood | 22:44 |
| vishy | (fairly far back in scrollback) :) | 22:44 |
| zaro | pleia2: i'm using precise vm, can i just upgrade to 2.7.5? | 22:44 |
| pleia2 | zaro: there might be a ppa, but the easy answer is "not really" | 22:44 |
| jaypipes | ok, now back to the job not registered with Gearman problem... | 22:44 |
| anteaya | since you wield the +3, can you also update the requirements-core list? | 22:45 |
| anteaya | yes I remember fungi saying that, was getting my wires crossed | 22:45 |
| hemna | :) | 22:45 |
| anteaya | hemna: I'm trying buddy | 22:45 |
| hemna | heh thanks for that :) | 22:45 |
| anteaya | vishy: since at least 3 members or req-core appear stale to my eyes | 22:46 |
| anteaya | s/or/of | 22:46 |
| hemna | fwiw, I'll have a follow up on that patch to update the lower number in that requirements. | 22:46 |
| hemna | the current one was just updating the upper number to make the mirror happy | 22:47 |
| harlowja_ | hi guys, do u have a sec to help the taskflow figure out a requirement that it wants to use but which isn't on the pypi openstack mirror | 22:47 |
| harlowja_ | https://bugs.launchpad.net/openstack-ci/+bug/1280008 | 22:47 |
| harlowja_ | is there a way to selectively use the global pypi, and only for certain cases use the openstack pypi/requirements | 22:47 |
| openstackgerrit | Ivan Melnikov proposed a change to openstack/requirements: Add pyMySQL to global requirements https://review.openstack.org/73437 | 22:47 |
| harlowja_ | instead of it being all or nothing | 22:47 |
| harlowja_ | the all or nothing approach, makes it hard for taskflow to be compatiable with dependencies that do exist in the public realm, but which are not direct openstack requirements | 22:48 |
| harlowja_ | *since certain pieces of openstack are not that compatible yet | 22:48 |
| jeblair | harlowja_: so the purpose of the openstack/requirements project is to ensure that all parts of openstack are using a compatible set of requirements, and that change to that go through review | 22:49 |
| dhellmann | jeblair: in this case, we want to test with those requirements, and with newer versions of the lib -- sqlalchemy in particular | 22:49 |
| dhellmann | so we want *both* sets of tests | 22:49 |
| dhellmann | if that's not readily possible, we can live with it, but if there's a way to make it work that would be nice | 22:50 |
| jeblair | dhellmann: i do not believe that is possible with the current design | 22:51 |
| *** geekinutah has left #openstack-infra | 22:51 | |
| dhellmann | jeblair: that's what I thought | 22:51 |
| *** iv_m has joined #openstack-infra | 22:51 | |
| harlowja_ | jeblair i can understand setting a minimum set of requirements, but also bounding the upper limit seems odd :-/ | 22:52 |
| *** med_ has quit IRC | 22:52 | |
| *** med_ has joined #openstack-infra | 22:52 | |
| harlowja_ | since in taskflows case at least it is compatible with 0.7,0.8 and 0.9 | 22:52 |
| *** tru_tru has joined #openstack-infra | 22:52 | |
| dhellmann | harlowja_: we've had far far too many cases of libraries outside our control releasing breaking versions | 22:52 |
| jeblair | dhellmann: a change to the openstack/requirements repo will run jobs with new versions, so you can test whether a new version breaks by proposing such a change | 22:52 |
| clarkb | jog0: kibana has been restored | 22:52 |
| dhellmann | jeblair: we want to test taskflow against many versions of sqlalchemy in different jobs | 22:52 |
| anteaya | clarkb: \o/ | 22:53 |
| dhellmann | jeblair: we were doing this when it was on stackforge, but there it was not restricted to the openstack requirements list | 22:53 |
| harlowja_ | jeblair another option is to possibly have a way to mark jobs to use global, or openstack pypi | 22:53 |
| jeblair | dhellmann: i understand; just making sure you knew what capabilities were there. | 22:53 |
| dhellmann | jeblair: sure, it's just not quite the same thing | 22:53 |
| ryanpetrello | so, infra review/zuul question | 22:54 |
| *** e0ne has joined #openstack-infra | 22:54 | |
| jaypipes | anybody know how I can debug why my jenkins jobs aren't being registered with gearman? there's no information in jenkins logs and nothing in zuul logs other than "job not regsistered" messages. or perhaps I should be asking *how* jobs get registered with gearman? on jenkins start? on zuul start? | 22:54 |
| *** beagles has quit IRC | 22:54 | |
| ryanpetrello | if I open a review, and then issue a ps2 ~3 minutes later | 22:54 |
| ryanpetrello | is zuul smart enough to cancel the first patchset? | 22:54 |
| jeblair | harlowja_: this system is focused on making sure that there is a set of requirements that all of openstack can work with. it's really designed to make openstack work, not to make projects work outside of openstack. | 22:54 |
| StevenK | ryanpetrello: Yes. | 22:55 |
| ryanpetrello | \o/ | 22:55 |
| jeblair | jaypipes: did you enable the gearman plugin? | 22:55 |
| dhellmann | jeblair: yeah, we can work with that, we just wanted to see if there was a way around | 22:55 |
| harlowja_ | jeblair with more oslo libraries does that start to make sense anymore :( | 22:55 |
| jaypipes | jeblair: yep, it's enabled and running fine, according to jenkins logs. | 22:55 |
| *** e0ne has quit IRC | 22:55 | |
| *** e0ne has joined #openstack-infra | 22:56 | |
| dhellmann | harlowja_: we can address that in the next cycle. I think moving to libraries instead of the incubator is going to make managing those requirements easier across the board | 22:56 |
| harlowja_ | dhellmann sure | 22:56 |
| *** thedodd has quit IRC | 22:56 | |
| jeblair | harlowja_: most of this comes from the fact that pip doesn't have a true dependency resolver, so there isn't a good way to say "this project can have different minimum/maximum versions than /requirements" in a reasonable way | 22:56 |
| jeblair | harlowja_: dstufft will fix all that though. :) | 22:57 |
| harlowja_ | jeblair sure, understandable, but could there be a way to allow certain zuul jobs to use/not use the global-requirements? | 22:57 |
| harlowja_ | like for py26, py27, py3, use the restricted requirements, for anything else, dont | 22:57 |
| harlowja_ | a flag possibly in yaml | 22:58 |
| harlowja_ | ? | 22:58 |
| jeblair | harlowja_: not easily. | 22:58 |
| harlowja_ | hmm | 22:58 |
| harlowja_ | durn | 22:58 |
| *** esker has joined #openstack-infra | 22:59 | |
| jeblair | harlowja_: i think the changes would be to the select-mirror script; but note that you would also need to have a different requirements.txt for the different tox envs... | 22:59 |
| jeblair | harlowja_: and then, how do you publish an artifact that works with the other versions? | 22:59 |
| jeblair | harlowja_: i think that's the real problem. testing it is tractable, but what do you do with the results? | 22:59 |
| harlowja_ | keep as much compatiblity as we can | 23:00 |
| jeblair | the thing published to pypi needs to use the openstack requirements | 23:00 |
| harlowja_ | hmmm | 23:00 |
| *** ryanpetrello has quit IRC | 23:00 | |
| harlowja_ | interesting, but if it works with a superset of the openstack requirements, then it should still work with openstack right? | 23:01 |
| *** e0ne has quit IRC | 23:01 | |
| jeblair | harlowja_: you're back at "pip doesn't have a dependency resolver". | 23:01 |
| harlowja_ | durn :-P | 23:01 |
| jaypipes | jeblair: does the registration happen when zuul starts? | 23:02 |
| jeblair | honestly, if that gets fixed, we can do a lot more interesting things. dstufft really is working on that | 23:02 |
| harlowja_ | jeblair k, thx, guess we might have to adjust our jobs then for now | 23:02 |
| jeblair | jaypipes: and on certain other events | 23:02 |
| fungi | 'pip install a b c' does not necessarily get you the same versions of packages installed as 'pip install a c b' and you also have to keep in mind that it's *very* possible for pip to install versions of packages which don't actually satisfy later declarations of required versions | 23:02 |
| fungi | so it's all extremely fiddly | 23:03 |
| jaypipes | jeblair: is there a way to manually trigger the event of registering a job? or some other way to try to debug the process of registration? | 23:03 |
| *** amcrn_ has joined #openstack-infra | 23:03 | |
| jeblair | jaypipes: start jenkins, change a job config, change a node config | 23:03 |
| jeblair | jaypipes: did you enable the gearman plugin? | 23:03 |
| dstufft | fungi: to be fair, we can atleast promise that pip abc will give you the same thing as pip a b c :V | 23:03 |
| dstufft | er | 23:03 |
| dstufft | pip a b c and pip a b c | 23:03 |
| dstufft | joke ruined by bad typing :( | 23:04 |
| jaypipes | jeblair: yes, the gearman plugin is definitely enabled. | 23:04 |
| jeblair | jaypipes: does your jenkins show up in the status command to the gearman server? | 23:04 |
| fungi | dstufft: repeatability is certainly a virtue | 23:04 |
| fungi | dstufft: as long as those two install commands are soon enough together than stuff on pypi doesn't change ;) | 23:04 |
| *** amcrn has quit IRC | 23:05 | |
| jaypipes | jeblair: all I get from the status command is this: | 23:05 |
| jaypipes | zuul:promote001 | 23:05 |
| jaypipes | zuul:enqueue001 | 23:05 |
| jaypipes | . | 23:05 |
| dstufft | actually techincally even if you have the exact same packages we can't guarentee that :( | 23:05 |
| jeblair | jaypipes: jenkins isn't connected to gearman. | 23:05 |
| jeblair | jaypipes: oh, sorry, i meant 'workers'. | 23:05 |
| dstufft | becuase insall_requires=random.sample(["a", "b", c"], 2) | 23:05 |
| fungi | dstufft: ha! | 23:06 |
| *** ryanpetrello has joined #openstack-infra | 23:06 | |
| *** maurosr has quit IRC | 23:06 | |
| *** gyee has joined #openstack-infra | 23:07 | |
| fungi | me conceives a chaos package that declares completely random pip requirements from pypi on every run | 23:07 |
| *** hdd_ has quit IRC | 23:07 | |
| dstufft | fungi: I tried to register requirements.txt becuase I was going to do basically that :3 | 23:08 |
| fungi | nice | 23:08 |
| dstufft | or have it depend on everything on PyPI | 23:08 |
| dstufft | but jannis beat me too it :D | 23:08 |
| fungi | great for when someone forgets a -r | 23:09 |
| jaypipes | jeblair: so in the jenkins UI, under manage plugins -> enabled, gearman plugin is showing up. I don't see any way to configure the plugin (and it was installed with the openstack-infra/config jenkins::plugin manifest in the same way as upstream jenkins masters. Sorry for asking so many questions. I'm just not entirely sure where to look for the connection between jenkins and gearman. | 23:09 |
| *** sarob has joined #openstack-infra | 23:09 | |
| fungi | jaypipes: go to configure jenkins and it will have a section there | 23:09 |
| fungi | jaypipes: when the plugin is enabled, it is configured in the main jenkins configuration screen. that's typical for jenkins. silly, but typical | 23:10 |
| *** mrodden has quit IRC | 23:10 | |
| jeblair | jaypipes: it defaults to disabled. it needs to be configured to talk to the zuul server and it needs to be enabled in the screen fungi mentioned. there's a connection test there too. | 23:10 |
| *** mrodden1 has joined #openstack-infra | 23:10 | |
| jaypipes | fungi: OK, that was really weird. Yeah, the plugin was enabled, but in the Configure System -> Gearman Plugin Config , the Enable Gearman box was unchecked :( | 23:10 |
| fungi | the plugin and gearman enabling are separate, so you can disable communication without unconfiguring | 23:11 |
| jaypipes | I see.. | 23:11 |
| fungi | at least i assume that was the reason | 23:12 |
| jaypipes | fungi: so do you guys manually check "Enable Gearman" then? | 23:12 |
| jaypipes | fungi: or is there some puppet thing I'm missing? | 23:12 |
| *** ryanpetrello has quit IRC | 23:12 | |
| fungi | jaypipes: we manually...lots of things for jenkins | 23:12 |
| *** e0ne has joined #openstack-infra | 23:12 | |
| fungi | jaypipes: because doing otherwise mostly means guessing at xml | 23:12 |
| jaypipes | fungi: ok, thx | 23:12 |
| fungi | jenkins-job-builder is a primary exception. it has raised guessing at jenkins xml to a high art | 23:13 |
| jaypipes | son of a... I restart Jenkins and it unchecks the damn Enable Gearman box again :( | 23:13 |
| fungi | that sounds wrong | 23:13 |
| fungi | we definitely start jenkins fairly often without it disabling | 23:14 |
| lifeless | fungi: 70376 | 23:14 |
| *** e0ne has quit IRC | 23:14 | |
| lifeless | fungi: yes,we certainly do | 23:14 |
| fungi | though i think it may require a successful connection test to actually enable? | 23:14 |
| jeblair | jaypipes: did you click save? | 23:14 |
| *** e0ne has joined #openstack-infra | 23:14 | |
| fungi | or that, right | 23:14 |
| jeblair | jaypipes, fungi: we can't configure that part of jenkins with puppet. that is one of the many reasons we're getting rid of jenkins. | 23:15 |
| jeblair | jaypipes: and the reason it doesn't default to enabled is because there is no sensible default for the host it should connect to. | 23:15 |
| *** e0ne has quit IRC | 23:15 | |
| jeblair | jaypipes: so defaulting it to enabled would mean defaulting to on-and-broken. | 23:15 |
| fungi | lifeless: restoring context...failed | 23:15 |
| *** e0ne has joined #openstack-infra | 23:16 | |
| jeblair | fungi: so the first vrfy+1 i examined in the log does have the correct data in the query which is returned 0.185s after zuul got the event. | 23:17 |
| lifeless | fungi: review 70376 | 23:17 |
| jaypipes | jeblair: lol. had no idea there was a save button below :( when I clicked Enable Gearman, it did something and said "Test successful!" so I figured it was saved :/ Anyway, saved now, and after restarting zuul, BOOM I have my jobs in gearman registered. w00t. on to the next obstacle. thx again you guys for your patience with me. | 23:17 |
| fungi | jeblair: but it still did not get enqueued? | 23:17 |
| lifeless | harlowja_: ping | 23:17 |
| lifeless | harlowja_: space & confirmation etc | 23:18 |
| jeblair | fungi: it wasn't supposed to be; i wasn't looking for a gate one specifically | 23:18 |
| harlowja_ | lifeless sup | 23:18 |
| harlowja_ | lifeless u see seans email? | 23:18 |
| fungi | jeblair: oh, okay | 23:18 |
| jog0 | clarkb: awesome, btw what happend? | 23:19 |
| jeblair | jog0: we upgraded | 23:19 |
| jaypipes | https://review.openstack.org/#/c/72790/ <-- \o/ it all worked. | 23:19 |
| jog0 | jeblair: ohh nice I thought so | 23:19 |
| jaypipes | now... to quickly write this down before I forget :) | 23:19 |
| anteaya | jaypipes: I am looking forward to reading your notes | 23:21 |
| *** e0ne has quit IRC | 23:21 | |
| jeblair | sdague: had a chance to look at https://review.openstack.org/#/c/73418/ ? | 23:21 |
| *** dims has quit IRC | 23:22 | |
| sdague | jeblair: just did | 23:22 |
| sdague | that looks fine to me. Honestly, I'm not 100% sure I parse the matrix correctly in my head. This is one of those places where my zuul grokking remains low | 23:22 |
| sdague | conceptually, I think it's the right behavior | 23:23 |
| clarkb | jeblair: you use xfce right? are you using the ssh-agent that it starts automagically for you? I was doing that with xfce on precise and it worked iwth ssh-add -c no problem. xfce + ssh agent on saucy xfce will accept the -c option but when I try to load the key the agent socket is removed | 23:23 |
| clarkb | I wonder if I need to set SSH_ASKPASS | 23:24 |
| lifeless | harlowja_: ah, yes I have | 23:24 |
| jeblair | sdague: it's new to me too. :) | 23:24 |
| lifeless | cody-somerville: you were copied on it | 23:24 |
| lifeless | cody-somerville: about 30m ago | 23:24 |
| harlowja_ | k | 23:25 |
| jeblair | clarkb: yeah i'm using the default agent: /usr/bin/ssh-agent /usr/bin/dbus-launch --exit-with-session startxfce4 | 23:26 |
| jeblair | clarkb: but this was never a gnome machine, so it doesn't have the gnome agent installed; if it did, i think it might use it instead | 23:26 |
| jeblair | clarkb: also, it's precise. | 23:27 |
| *** oubiwann_ has quit IRC | 23:27 | |
| clarkb | jeblair: yeah that is what I have on my precise box | 23:27 |
| clarkb | and it works. on saucy it is just ssh-agent without the extra stuff | 23:27 |
| clarkb | I guess I can try it with those options if I can find the magical place ssh-agent is invoked by xfce | 23:27 |
| jeblair | fungi: i'm testing with this change: https://review.openstack.org/#/c/69800/ | 23:27 |
| fungi | k | 23:28 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to set_hooks_commit_msg https://review.openstack.org/70746 | 23:28 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Retrieve remote pushurl independently of user's locale https://review.openstack.org/64307 | 23:28 |
| fungi | clarkb: i just use keychain and invoke it in my ~/.xinitrc | 23:29 |
| fungi | but maybe i'm weird | 23:29 |
| jeblair | fungi: oh, i think nodepool's connection to zuul's gearman broke. | 23:29 |
| fungi | oh, though i use startx via an alias and have the local command-line agent fir for that, so i get prompted for that in the vtty | 23:30 |
| jeblair | restarted nodepool | 23:30 |
| fungi | jeblair: you mean when zuul restarted? | 23:30 |
| jeblair | fungi: probably; i think there's still a gear bug that could cause that | 23:30 |
| mordred | clarkb: let me know if you find how to do that - I would like to change the options too | 23:30 |
| fungi | clarkb: anyway, caching your key prior to starting your x session isn't hard as long as you don't use a login manager | 23:31 |
| jeblair | 2014-02-13 23:31:34,258 DEBUG nodepool.NodePool: Deficit: devstack-precise: 97 (start: 123 min: 123 ready: 26) | 23:31 |
| jeblair | that's more like it | 23:31 |
| *** sarob has quit IRC | 23:31 | |
| *** sarob has joined #openstack-infra | 23:32 | |
| clarkb | fungi: yeah there are alternatives, but the reason I bother withxfce is so that it does this stuff for me :) | 23:35 |
| *** dims_ has joined #openstack-infra | 23:35 | |
| fungi | xfce uses xdm? | 23:36 |
| clarkb | mordred: figured it out, need to install ssh-askpass | 23:36 |
| fungi | oh, yeah you definitely need that regardless | 23:37 |
| StevenK | Huh | 23:37 |
| StevenK | I have a working agent setup without ssh-askpass | 23:37 |
| clarkb | StevenK: we use ssh-add -c | 23:38 |
| clarkb | it works fine without the -c | 23:38 |
| StevenK | Ah | 23:38 |
| *** gokrokve_ has quit IRC | 23:39 | |
| *** gokrokve has joined #openstack-infra | 23:39 | |
| clarkb | fungi: I think on my old box the gnome ask pass was installed because unity, but this has no unity therefore needed to isntall | 23:39 |
| mordred | clarkb: wait- so where are you doing your ssh-add -c ? | 23:39 |
| clarkb | mordred: I just do that by hand when I start a new session | 23:40 |
| mordred | oh. weird | 23:40 |
| mordred | mine happens automatically. I guess I'll investigate | 23:40 |
| jeblair | clarkb: i have a startup task that runs ssh-add -c | 23:41 |
| fungi | clarkb: oh, you don't use keychain for that? | 23:41 |
| jeblair | mordred: if it's happening automatically, it's probably not doing "-c". | 23:42 |
| mordred | jeblair: right. that's what I'm investigating right now | 23:42 |
| clarkb | fungi: no, it is just starting ssh-agent which is all I really want | 23:42 |
| clarkb | then I can manually add keys as appropriate | 23:42 |
| fungi | got it | 23:42 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Remove taskflow sqlalchemy 0.9 testing https://review.openstack.org/73445 | 23:43 |
| zaro | clarkb: questions about available packages on jenkins slaves | 23:43 |
| clarkb | zaro: sure, there is a list in the slave.pp manifset | 23:43 |
| zaro | clarkb: i don't see git defined in pp files so is it available? | 23:43 |
| clarkb | git is available, it gets in there as part of the image install iirc | 23:43 |
| clarkb | and probably as part of a lower level puppet manifest | 23:44 |
| boris-42_ | somebody could pls take a look at this patch https://review.openstack.org/#/c/73025/ ?) | 23:44 |
| zaro | clarkb: how would i know that? | 23:44 |
| *** gokrokve has quit IRC | 23:44 | |
| fungi | clarkb: well, early on, our install shell scripts install git so that they can get a copy of the puppet manifest | 23:45 |
| fungi | so we install git before we apply puppetry | 23:45 |
| clarkb | right | 23:45 |
| zaro | clarkb: so i spend most of my day troubleshooting gerrit build and it turned out that the problem was that zip package was not on the vm i was using. | 23:45 |
| clarkb | zaro: see the install puppet script | 23:45 |
| clarkb | zaro: what are we using zip files for? | 23:45 |
| zaro | clarkb: not me it's the buck buildding gerrit | 23:45 |
| *** dhellmann is now known as dhellmann_ | 23:45 | |
| clarkb | ah | 23:46 |
| fungi | oh, and they don't assume you might build buck on a system with no windows-centric zip tool? funny | 23:46 |
| zaro | fungi: they are all mac lovers | 23:46 |
| fungi | i guess it helps that java jarfiles are actually zip with a different extension | 23:46 |
| *** rcleere has quit IRC | 23:46 | |
| fungi | so java devs most likely assume you install zip everywhere, since it's baked into their toolchain anyway | 23:47 |
| lifeless | fungi: s/assume/know that/ | 23:47 |
| fungi | heh | 23:47 |
| zaro | zip installs unzip, but not the other way around. | 23:47 |
| zaro | should both be explicitly stated in pp file? or just zip since it install unzip. | 23:48 |
| lifeless | be explicit IMO | 23:48 |
| lifeless | zip installing unzip is as weird as unzip not installing zip | 23:48 |
| lifeless | so I wouldn't bet on it never changing | 23:48 |
| fungi | yeah, there's no guarantee that in $future it might change | 23:48 |
| fungi | what lifeless said | 23:49 |
| zaro | so why not be explicit with git too? | 23:49 |
| mordred | are we not? | 23:49 |
| zaro | i don't see it in slave.pp | 23:49 |
| zaro | or param.pp | 23:49 |
| fungi | zaro: if we don't have it in our puppet manifest somewhere there or at a higher level then i would consider that a latent bug | 23:50 |
| sdague | honestly, jar is a better zip than zip | 23:50 |
| mordred | oh - I know why | 23:50 |
| sdague | as it takes the same flags as tar, and works on archives over 4G | 23:50 |
| mordred | it's because our "bootstrap this node" script installs it I think | 23:50 |
| *** bhuvan has quit IRC | 23:50 | |
| fungi | mordred: yep, we already discussed that. we should still be explicit and have puppet insist on it anyway | 23:51 |
| mordred | nope. wrong | 23:51 |
| mordred | it's in base.pp | 23:51 |
| mordred | s/wrong/me was wrong/ | 23:51 |
| fungi | yeah, so sounds like we already do, in which case no bug | 23:51 |
| *** denis_makogon has quit IRC | 23:52 | |
| *** bhuvan has joined #openstack-infra | 23:52 | |
| *** gokrokve has joined #openstack-infra | 23:54 | |
| zaro | ok, i see it now. would it be safe to assume that the slave will also have gcc? | 23:54 |
| fungi | zaro: the build-essential package depends on the gnu libc toolchain among other things | 23:55 |
| zaro | fungi: ohh yeah, i found that. thanks. | 23:56 |
| fungi | explicitly listing all packages brought in by build-essential might count as overly pedantic | 23:56 |
| *** derekh_afk is now known as derekh | 23:56 | |
| jeblair | fungi: same behavior observed with https://review.openstack.org/#/c/69800/ | 23:56 |
| jeblair | fungi: it looks like it has the correct data. i'll try to reproduce locally | 23:56 |
| *** sarob_ has joined #openstack-infra | 23:56 | |
| jog0 | fungi: I want to follow up in bug 1270382 | 23:57 |
| fungi | jeblair: i see that. looking at the debug log | 23:57 |
| *** andrew_plunk has quit IRC | 23:57 | |
| jog0 | or jeblair ^, its marked as incomplete, is there any other data you want etc | 23:57 |
| jog0 | as we e-r thinks thats thh top gate bug | 23:57 |
| dstufft | sdague: zip64 $$ | 23:58 |
| fungi | jog0: so it's still happening? | 23:58 |
| *** jgrimm has quit IRC | 23:58 | |
| jeblair | fungi: i wonder if that's related to that ticket you mentioned | 23:58 |
| sdague | dstufft: but then I have to remember zip flags :) | 23:58 |
| sdague | jar xvf foo.zip :) | 23:58 |
| *** bhuvan has quit IRC | 23:59 | |
| jeblair | jog0: i think based on a ticket rackspace openened on our account, it looks like their resolvers may be rate limiting us | 23:59 |
| fungi | jeblair: it's possible, but they only ever opened one ticket about one slave, and it was long after that bug got reported | 23:59 |
| mtreinish | sdague: jar or tar? :) | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!