Wednesday, 2013-12-18

*** ^d has quit IRC00:01
jog0sdague clarkb: we are getting build timeout after 90 minutes00:01
jog0in {check,gate}-temepst-dsvm-{,postgres}-full00:02
anteayayeah, I noticed those had jumped up00:04
anteayawhat might cause the sudden increase in those bugs?00:04
jog0hmm its because suspend and resume and reboot tests are failing in nova00:04
jog0I think00:04
jog0so if things timeout we don't collect logs :(00:05
*** markmcclain has joined #openstack-infra00:05
jog0jeblair: ^00:05
jog0err testr doesn't save logs or something00:05
jog0lifeless: ^00:06
jeblairjog0: if jenkins times out logs are not copied to logs.o.o00:06
jeblairjog0: two solutions: try to make the jobs timeout internally (eg testr) before hitting the jenkins timeout00:06
lifelessjog0: logs are written to a temp file in real time00:06
jog0jeblair: some appear t o be http://logs.openstack.org/90/54290/4/check/check-tempest-dsvm-full/d03c4f1/console.html#_2013-12-17_19_51_29_70300:06
jog0jeblair: I like the testr timeout option00:07
lifelessjog0: so should be able to just grab them00:07
jog0lifeless: is that doable, a global 90 minute timeout ?00:07
jog0for tempest00:07
jog0or we can wire up logic to look for those temp files00:07
lifelesssure, new feature00:07
lifelessshortest path is to grab the temp file00:07
jog0where do they live?00:07
lifeless.testrepository/00:08
jeblairjog0: those logs were put in position before the test run started...00:08
jeblairjog0: you want the testr logs or the openstack logs?00:08
*** jhesketh__ has quit IRC00:08
jog0jeblair: testr for starters00:08
jog0ideally both openstack and testr00:08
*** sarob has joined #openstack-infra00:08
jog0but one or the other is pretty good00:08
jeblairjog0: the openstack logs are moved into position by devstack-gate for jenkins to find after tempest finishes00:09
jog0I have some failures in tempest here http://logs.openstack.org/90/54290/4/check/check-tempest-dsvm-full/d03c4f1/console.html#_2013-12-17_18_48_58_24700:09
jog0and want to get some more info on them00:09
jeblairjog0: perhaps we could replace the tempest invocation with "timeout 85m tox -e full" in devstack-gate00:10
jog0lifeless: ^ thoughts00:10
jeblairjog0: and use that to kill tempest before the jenkins timeout while giving devstack-gate enough time to copy the logs into position for jenkins00:10
jog0I think we loose the testr logs taht way, but we get openstack logs, so win00:11
jeblairjog0: if they are written in real time we should get whatever is run00:12
jog0jeblair: we have to add logic to copy the temp file  Ithink00:12
*** sarob has quit IRC00:13
jog0oh this should n00:13
jog0patch coming soon00:13
*** rnirmal has joined #openstack-infra00:13
lifelessthere is some buffering in the logs atm00:13
lifelessonce I finish the migration to v2 it will be gone00:13
lifelessor so low as to be invisible00:13
lifelessspecifically you won't see hung tests today, only slow ones00:13
openstackgerritJames E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timing out tempest  https://review.openstack.org/6278100:14
jog0lifeless: that is fine,00:14
jeblairjog0: ^ quick poc of the idea00:14
*** cody-somerville has joined #openstack-infra00:14
*** cody-somerville has quit IRC00:14
*** cody-somerville has joined #openstack-infra00:14
jeblairjog0: if it works and we do it for real, we'll probably want to pass in the timeout values from the JJB config so that we can accomodate the different runtimes for different jobs00:15
cody-somervilleclarkb: Hey. Where are you guys?00:15
pleia2cody-somerville: clarkb went to the airport, the rest of us are in the gravity room on the 7th floor00:16
cody-somervilleAh, ok.00:16
*** rongze has joined #openstack-infra00:16
cody-somervillepleia2: Who all is up there?00:16
jeblairpleia2: sounds heavy.00:17
*** mrodden has quit IRC00:17
StevenKjeblair: Boo. Hiss.00:17
pleia2cody-somerville: AaronGr and I with the hp folks00:17
*** herndon has quit IRC00:17
pleia2er, other hp folks, non-monty hp folks :)00:17
pleia2jeblair: hehe00:18
jog0lifeless jeblair: http://paste.openstack.org/show/5522600:18
fungi"heavy." why are things so heavy in the future?00:18
jog0something like that would let us colelct the testr temp file I think00:19
*** loq_mac has quit IRC00:20
ewindischin SF - anyone up for beer, coffee, or dinner?00:20
jeblairjog0: the testr part looks good (the timeout we'll want to make less hacky like above)00:20
openstackgerritA change was merged to openstack/requirements: Add python-ironicclient to global requirements  https://review.openstack.org/6063800:21
*** rongze has quit IRC00:21
jog0jeblair: testing the idea out locally using unit tests00:21
pleia2ewindisch: aww, I'm in mt view tonight, how long are you in town?00:22
jeblairjog0: oh, and the other thing is we need to budget for setup time; so the tempest timeout should be (job_timeout - devstack_setup_time - log_prep_time - margin_of_error)00:22
jeblairjog0: oh, i had another idea...00:22
jog0jeblair: cool, I am making sure the testr logic works for now00:23
jeblairjog0: devstack and tempest are all in the gate script, whereas log copying is in the wrap script... so we may be able to timeout the gate script00:23
jog0looks like it does00:23
jeblairjog0: which makes the timeout calculation less fiddly00:24
jog0jeblair: oh I like00:24
ewindischpleia2: until Thursday, but I have something Thursday night00:24
lifeless+100:24
jog0jeblair: confirmed timeout will make testr leave its temp file around00:24
jog0going to push a patch that does only the testr copy logic00:24
openstackgerritJames E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2  https://review.openstack.org/6278400:25
jeblairjog0: testing the other timeout idea00:25
jeblairjog0: ++copy logic patch00:25
jeblairjog0: 5m tempest timeout seems to have worked: https://jenkins01.openstack.org/job/check-dg-tempest-dsvm-full/30/console00:29
cody-somervilleIf I want to write a new config module and sort of trail and error a bit on a cloud instance, is there a particular version of puppet I should be using?00:29
openstackgerritJoe Gordon proposed a change to openstack-infra/devstack-gate: Preserve testr temp files  https://review.openstack.org/6278600:29
jeblaircody-somerville: we're using 2.7.21 from puppetlabs on prod servers00:29
jog0jeblair: cool I did a 1m timeout locally and saw the same thing00:30
jog0ohh nice00:30
jeblaircody-somerville: http://ci.openstack.org/sysadmin.html#making-a-change-in-puppet00:30
jeblaircody-somerville: if you follow the instructions there, you'll get exactly the puppet version we use in prod00:30
*** dizquierdo has quit IRC00:31
jog0lifeless: https://review.openstack.org/#/c/62786/00:32
cody-somervillejeblair: Awesome. Thanks.00:33
openstackgerritJoe Gordon proposed a change to openstack-infra/devstack-gate: Preserve testr temp files  https://review.openstack.org/6278600:33
openstackgerritJames E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2  https://review.openstack.org/6278400:36
*** wenlock has joined #openstack-infra00:38
jeblairjog0: 5m timeout of the gate script correctly killed during devstack, though parts of devstack kept running in the bg...00:38
jeblairhttps://jenkins02.openstack.org/job/check-dg-tempest-dsvm-full/26/console00:39
jog0jeblair: is that a problem?00:39
jog0its clearly not good00:39
jeblairjog0: i don't think so?  it's not very elegant but i don't think it'll happen in practice and i think we'll still get the bulk of the data00:40
jeblairjog0: tox and testr seemed better behaved so i don't think this case is worth worrying about too much00:40
jog0agreed00:40
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Refactor logstash_worker into log_processor module  https://review.openstack.org/6278700:41
fungithe gate is empty, so as soon as there are no queued jobs in the post pipeline (one hasn't been started yet) i'll put jenkins.o.o in prepare for shutdown00:42
fungiand there it goes00:42
fungionce the rest of those finish, i'll switch the dns entry, fire off one last catch-up rsync and update the root path for the tarballs.o.o publisher site, wait ~5 minutes and cancel the jenkins shutdown00:43
fungiactually as soon as the last running tarball job there finishes, i'll do the rsync00:44
anteayapleia2: so is the onboarding continuing tomorrow?00:44
*** wenlock has quit IRC00:44
anteayapleia2: I still need to hear from mordred otherwise I will do what I have to do and just accept the fingerwagging when it happens00:44
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Increase to 16 logstash worker nodes  https://review.openstack.org/6278800:44
*** jergerber has quit IRC00:45
jeblairfungi: ^ the first of those logstash changes should let us double-up the workers on the nodes, and then the second should let us scale up00:45
fungiexciting--having a look00:45
pleia2anteaya: no, folks are already heading home00:46
jeblairfungi: we should probably do both of those and then give crm114 another shot; keeping an eye on it.  then hopefully we can scale down.00:46
anteayapleia2: k, thanks - I hope everyone had a good time00:46
fungifinal tarballs rsync done00:46
anteayaand that mordred reads my email00:46
anteayaone day left to wait00:46
* anteaya makes pointy fingers at mordred00:47
fungii think it's probably safe to let the requirements post mirror jobs keep running while i cut over the tarballs site00:47
jeblairfungi: yes00:47
*** yamahata has joined #openstack-infra00:47
pleia2anteaya: we split up into teams today, I haven't actually seen mordred since last night, I'll let him know you have a query pending if I do00:48
*** zhiyan has joined #openstack-infra00:48
clarkbdouble glad I left a little early screening line is huge00:49
fungijeblair: do you think the jvm name resolver jenkins uses will notice the address change automatically, or should i wait until it's fully quiesced and completely restart the jenkins service for good measure?00:50
StevenKclarkb: The TSA are just being extra careful.00:50
StevenKWait, I can't claim that with a straight face.00:50
jeblairfungi: jenkins seems to handle it correctly, so i don't think it's necessary00:50
fungiStevenK: extra careful to jusitfy their pay checks00:50
fungijeblair: okay, good enough. a few more minutes and i'll cancel shutdown then00:51
fungiand i'll do some spot checks after the next tarball job i see complete, just to make sure it's landing on the correct server00:51
jog0jeblair: this is the related bug https://bugs.launchpad.net/tempest/+bug/125868200:52
uvirtbotLaunchpad bug 1258682 in tempest "timeout causing gate-tempest-dsvm-full to fail" [Undecided,Invalid]00:52
jog0the generic things timed out and I don't have logs00:52
fungijeblair: i ended up adding the additional filesystem graph through the cacti admin webui. hopefully that doesn't cause issues00:52
fungi(couldn't find a sane way to inject the info into the db)00:53
jeblairfungi: oh, yeah, that's probably the best way to deal with a one-off00:53
anteayapleia2: sure, clarkb was my messenger yesterday, so a smile and reminder should do it00:53
anteayait's just I only have one day left00:54
anteayaotherwise I could wait00:54
*** ^d has joined #openstack-infra00:57
*** nicedice has quit IRC00:57
*** nicedice has joined #openstack-infra00:57
*** johnthetubaguy has joined #openstack-infra01:00
*** esker has joined #openstack-infra01:01
*** ^d has quit IRC01:02
*** wenlock has joined #openstack-infra01:02
fungijust dawned on me that it's jenkins01 and jenkins02 which do the tarball builds, so i quickly ninja'd the tarballs.o.o site root path on them just now. no new tarball jobs have started yet anyway01:03
*** blamar has quit IRC01:03
*** johnthetubaguy has quit IRC01:04
fungiin fact, 90% of our developers seem to have decided to go on holiday this week, judging from the gate01:06
jerryzjog0: does grenade failure log get classified if tempests are successful?01:06
jog0jerryz: we don't report on grenade failures at the moment01:07
jerryzjog0: but it seems to me that if tempest and grenade fail at the same time, grenade failures will still get reported, as grenade logs are in elastic search01:08
jog0jerryz: we may have changed it we wantto report on grenade01:09
*** praneshp has joined #openstack-infra01:11
jerryzjog0: i mean i found that tempest failure triggered the classification, but in the end, as a matter of fact, a bug is found in grenade logs .01:12
*** wenlock has quit IRC01:12
jog0jerryz: that sounds possible01:13
*** praneshp_ has joined #openstack-infra01:13
*** praneshp has quit IRC01:15
*** praneshp_ is now known as praneshp01:15
jerryzjog0: that happens when i fight the grenade failures. i found Bug 1253896 failed my grenade test a lot01:15
uvirtbotLaunchpad bug 1253896 in neutron "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,In progress] https://launchpad.net/bugs/125389601:15
fungithat one seems vague enough to be a symptom of more than one actual bug01:16
jog0yeah we don't know much about what is causing that bug01:17
jerryzfungi, jog0: in my env, 3 rechecks may give me luck to pass.01:19
*** senk1 has quit IRC01:20
*** freyes_ has joined #openstack-infra01:21
*** arosen has quit IRC01:22
*** jhesketh__ has joined #openstack-infra01:23
jerryzjog0: why not report grenade failures at the moment?01:23
jog0jerryz: its a bug01:23
jerryzjog0: return "FAILURE" in line and "tempest" in line  . can i add 'or "grenade" in line' here01:25
*** blamar has joined #openstack-infra01:26
*** nosnos has joined #openstack-infra01:26
jerryzjog0: http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/elastic_recheck/elasticRecheck.py#n8201:29
*** yaguang has joined #openstack-infra01:29
jog0jerryz: that isn't enough01:30
jog0jerryz: can you find an example where 1253896 is failing greande01:30
jog0grenade01:30
jog0grenade has different files01:30
jeblairfungi: gah, sorry about that (tarball jobs)01:31
fungijeblair: no damage done. it's quiet in our gated community tonight01:32
fungii think we're coming up on an hour straight with nothing at all in the gate01:32
jeblairthat is a very flat sparkline!01:33
jerryzjog0: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcIkJhZFJlcXVlc3Q6IEludmFsaWQgdm9sdW1lOiBWb2x1bWUgc3RhdHVzIG11c3QgYmUgYXZhaWxhYmxlIG9yIGVycm9yXCIgQU5EIGZpbGVuYW1lOiBcImNvbnNvbGUuaHRtbFwiIEFORCBidWlsZF9uYW1lOiBcImNoZWNrLWdyZW5hZGUtZHN2bVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzg3MzMwNDUxMTg0fQ==01:35
*** senk1 has joined #openstack-infra01:36
*** crank has quit IRC01:38
*** crank has joined #openstack-infra01:38
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add support for grenade console.html  https://review.openstack.org/6279101:40
jog0jerryz: ^ that should support your fingerprint01:40
jog0sdague: ^01:40
jog0jerryz: I need to go AFK for a bit01:40
jog0o/01:40
jerryzthanks01:40
openstackgerritJames E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2  https://review.openstack.org/6278401:41
*** CaptTofu has quit IRC01:42
*** CaptTofu has joined #openstack-infra01:42
*** ryanpetrello has joined #openstack-infra01:42
*** sdake has quit IRC01:42
*** sdake has joined #openstack-infra01:42
*** sdake has quit IRC01:43
*** sdake has joined #openstack-infra01:43
*** cody-somerville has quit IRC01:48
*** crank has quit IRC01:50
*** senk1 has quit IRC01:50
*** crank has joined #openstack-infra01:51
fungichanges in the gate finally. soon i'll be able to confirm tarball uploads are working02:00
fungijhesketh_ really should have added tarball jobs for turbo-hipster. boo02:00
fungihave to wait for the nova change to (hopefully) merge instead02:01
jhesketh__fungi: yes, I need to neaten up the release stuff for that a little first02:01
fungithat makes your project a horribly unsuitable guinea pig for my completely unrelated needs02:01
* jhesketh__ is very unsure of what fungi is doing02:02
fungihoping to confirm i didn't break tarball uploads02:02
jhesketh__ah right02:02
fungiwithout going to the trouble of manually retriggering a job, which would be several seconds of my time wasted of course02:03
*** guohliu has joined #openstack-infra02:03
*** jhesketh_ has quit IRC02:05
*** melwitt1 has quit IRC02:07
*** jhesketh_ has joined #openstack-infra02:10
*** crank has quit IRC02:13
*** crank has joined #openstack-infra02:14
*** rongze has joined #openstack-infra02:14
*** krotscheck1 has joined #openstack-infra02:18
*** krotscheck1 has left #openstack-infra02:21
*** ryanpetrello has quit IRC02:22
*** zhiyan has quit IRC02:24
*** ryanpetrello has joined #openstack-infra02:29
*** krotscheck has joined #openstack-infra02:30
*** crank has quit IRC02:31
*** crank has joined #openstack-infra02:32
openstackgerritYuuichi Fujioka proposed a change to openstack-dev/hacking: checking for metaclass to be Python 3.x compatible  https://review.openstack.org/5689002:34
*** praneshp has quit IRC02:34
*** praneshp has joined #openstack-infra02:34
*** krotscheck has quit IRC02:40
fungigah, upload failure... http://paste.openstack.org/show/55247/02:40
fungido the jenkins masters need to have their known_hosts file entries for the old tarballs.o.o host key removed/replaced?02:41
fungiclarkb: jeblair: mordred: ^02:41
*** krotscheck_ has joined #openstack-infra02:43
fungiif so, it's not the ~jenkins/.ssh/known_hosts file on the masters at any rate02:43
fungiand i confirmed that sudo -u jenkins ssh tarballs.openstack.org works and gets a shell on static02:44
anteayahow does jenkins do auth?02:44
fungiit has references to its ssh key on the filesystem (the one i just authenticated with)02:45
anteayahmm02:46
anteayathe build step changed the result to unstable02:47
fungiyep02:47
fungithat's typical when it can't upload02:48
fungiohhhhhhhhhhhh02:48
fungithe publisher's remote username was "hudson" on old-wiki02:48
fungifixing that to "jenkins" should solve it02:49
anteayaah02:51
*** freyes_ has quit IRC02:53
fungiupdated on all 3 masters for completeness, and retriggered the job https://jenkins02.openstack.org/job/nova-branch-tarball/590/console02:54
*** vkozhukalov has joined #openstack-infra02:54
anteayafungi I get finished unstable again02:57
fungiyup, and i see why02:57
anteayado share, after you fix02:58
fungithis time it's a permissions issue, because the files were owned by hudson when they were copied over. fixing02:58
*** praneshp has quit IRC02:58
fungilatest attempt running now... https://jenkins01.openstack.org/job/nova-branch-tarball/929/console02:59
fungithat worked03:00
fungishould be all set now03:00
fungiconfirmed that the timestamp for that branch tarball was updated at the new remote end03:01
* fungi switches to low-power mode for the rest of the evening03:01
*** rongze has quit IRC03:02
*** wenlock has joined #openstack-infra03:02
*** mriedem has quit IRC03:04
*** nati_uen_ has quit IRC03:04
anteayayay03:05
*** rongze has joined #openstack-infra03:10
*** alexpilotti has quit IRC03:11
*** krotscheck_ has quit IRC03:13
*** rnirmal has quit IRC03:15
*** krotscheck has joined #openstack-infra03:15
*** pcrews has quit IRC03:18
*** gyee has quit IRC03:20
*** krtaylor has quit IRC03:25
*** ryanpetrello has quit IRC03:27
jhesketh__If anybody has time to look at this simple fix I'd really appreciate it: https://review.openstack.org/#/c/62523/03:27
*** UtahDave has quit IRC03:31
*** AaronGr_Zzz is now known as AaronGr03:32
fungijhesketh__: do you have a test gerrit you've tried that on? if not, i'll put it on my list of things to hopefully try out manually tomorrow03:33
*** slong has joined #openstack-infra03:34
jhesketh__fungi: no, I was hoping a close review would be enough but I guess we really don't want to break gerrit03:34
*** krotscheck has quit IRC03:34
fungiwell, i won't speak for others, but at least *i* personally break plenty enough already with out needing to tempt fate ;)03:36
jhesketh__heh, fair enough03:37
jhesketh__well I don't have a local jenkins/infra to test on, do you guys have a dev deployment?03:37
*** ArxCruz has quit IRC03:37
fungiof gerrit, yes and no. it's currently not usable for this because we're testing a gerrit version upgrade03:38
funginormally i'd say just tweak the change to only apply to review-dev first, but not for this week at any rate03:40
*** AaronGr is now known as AaronGr_Zzz03:40
fungiso i'll test rolling back our failed upgrade first and then give it a shot03:41
jhesketh__sure okay03:43
jhesketh__sounds good03:43
*** nicedice has quit IRC03:46
*** nicedice has joined #openstack-infra03:46
*** CaptTofu has quit IRC03:48
*** CaptTofu has joined #openstack-infra03:48
*** nicedice has quit IRC03:49
*** nicedice has joined #openstack-infra03:50
*** jhesketh__ has quit IRC03:50
*** lyz has quit IRC03:50
*** jhesketh__ has joined #openstack-infra03:50
*** rcarrillocruz has joined #openstack-infra03:51
*** harlowja is now known as harlowja_away03:51
*** rcarrillocruz1 has joined #openstack-infra03:52
*** ljjjustin has joined #openstack-infra03:54
*** rcarrillocruz has quit IRC03:55
*** rcarrillocruz1 has quit IRC03:56
*** rcleere has joined #openstack-infra04:00
*** CaptTofu has quit IRC04:10
*** CaptTofu has joined #openstack-infra04:10
*** ryanpetrello has joined #openstack-infra04:11
*** niska has quit IRC04:14
*** chandankumar has joined #openstack-infra04:15
*** CaptTofu has quit IRC04:17
*** julim has quit IRC04:18
*** niska has joined #openstack-infra04:21
*** slong has quit IRC04:24
*** slong has joined #openstack-infra04:24
*** talluri has joined #openstack-infra04:26
*** CaptTofu has joined #openstack-infra04:33
SpamapS                                       /win 1304:35
*** guohliu has quit IRC04:36
*** bingbu has joined #openstack-infra04:37
openstackgerritBill Maxwell proposed a change to openstack-infra/jenkins-job-builder: Implements: Archive publisher allow-empty setting.  https://review.openstack.org/6280604:39
*** Ryan_Lane has quit IRC04:47
*** Ryan_Lane has joined #openstack-infra04:52
*** Ryan_Lane has quit IRC04:56
*** ryanpetrello has quit IRC04:58
*** rongze has quit IRC04:59
*** masayukig has joined #openstack-infra04:59
*** talluri has quit IRC05:15
*** Ryan_Lane has joined #openstack-infra05:18
*** wenlock has quit IRC05:21
*** Ryan_Lane has quit IRC05:23
*** SergeyLukjanov has joined #openstack-infra05:24
*** guohliu has joined #openstack-infra05:26
*** CaptTofu has quit IRC05:29
*** CaptTofu has joined #openstack-infra05:29
*** rongze has joined #openstack-infra05:30
*** praneshp has joined #openstack-infra05:30
*** sdake_ has quit IRC05:31
*** rongze has quit IRC05:35
*** talluri has joined #openstack-infra05:38
*** rcleere has quit IRC05:44
*** crank has quit IRC05:45
*** crank has joined #openstack-infra05:45
*** davidhadas has joined #openstack-infra05:46
*** slong has quit IRC05:50
*** nati_ueno has joined #openstack-infra05:51
*** slong has joined #openstack-infra05:54
*** praneshp has quit IRC05:55
*** SergeyLukjanov is now known as _SergeyLukjanov05:56
*** _SergeyLukjanov has quit IRC05:56
*** crank has quit IRC06:00
*** crank has joined #openstack-infra06:01
*** dkliban has quit IRC06:02
*** fifieldt has joined #openstack-infra06:07
*** rongze has joined #openstack-infra06:08
*** zhiyan has joined #openstack-infra06:08
*** SergeyLukjanov has joined #openstack-infra06:12
*** vkozhukalov has quit IRC06:15
*** vkozhukalov has joined #openstack-infra06:15
*** senk has joined #openstack-infra06:17
*** MarkAtwood has quit IRC06:18
*** Ryan_Lane has joined #openstack-infra06:19
*** Ryan_Lane has quit IRC06:24
*** senk has quit IRC06:26
zaro~.06:31
*** crank has quit IRC06:31
*** crank has joined #openstack-infra06:32
*** dkliban has joined #openstack-infra06:36
*** sarob has joined #openstack-infra06:39
*** dkehn has quit IRC06:40
*** rwsu_ has quit IRC06:41
*** dkehn has joined #openstack-infra06:42
*** dstufft_ is now known as dstufft06:42
*** praneshp has joined #openstack-infra06:49
*** vkozhukalov has quit IRC06:52
*** dims has quit IRC06:55
*** davidhadas has quit IRC07:03
*** yolanda has joined #openstack-infra07:08
*** ljjjustin has quit IRC07:09
*** jcooley_ has joined #openstack-infra07:10
*** ljjjustin has joined #openstack-infra07:11
*** pblaho has joined #openstack-infra07:12
lifelessbtw https://bugs.launchpad.net/nova/+bug/1262075 might be the slowdown culprit? 196 seconds to rescue a server..07:14
uvirtbotLaunchpad bug 1262075 in nova "tempest.api.compute.servers.test_server_rescue.ServerRescueTestXML.test_rescue_unrescue_instance[gate,smoke] failed on postgresql only" [Undecided,New]07:14
*** SergeyLukjanov has quit IRC07:14
*** Ryan_Lane has joined #openstack-infra07:19
*** dkliban has quit IRC07:19
*** zhiyan has quit IRC07:20
*** Ryan_Lane has quit IRC07:23
*** yolanda has quit IRC07:24
*** ljjjustin has quit IRC07:34
*** sarob has quit IRC07:40
*** sarob has joined #openstack-infra07:41
*** zhiyan has joined #openstack-infra07:41
*** dims_ has joined #openstack-infra07:43
*** flaper87|afk is now known as flaper8707:43
*** jcooley_ has quit IRC07:45
*** sarob has quit IRC07:46
*** yolanda has joined #openstack-infra07:47
*** dkliban has joined #openstack-infra07:56
*** oubiwann has quit IRC07:58
*** fifieldt has quit IRC08:02
*** dkliban has quit IRC08:10
*** sarob has joined #openstack-infra08:11
*** jcoufal has joined #openstack-infra08:13
*** davidhadas has joined #openstack-infra08:14
*** flaper87 is now known as flaper87|afk08:14
*** flaper87|afk is now known as flaper8708:16
*** sarob has quit IRC08:17
*** jcoufal has quit IRC08:18
*** Ryan_Lane has joined #openstack-infra08:19
*** bingbu has quit IRC08:20
*** Ryan_Lane has quit IRC08:20
*** Ryan_Lane has joined #openstack-infra08:21
*** SushilKM has joined #openstack-infra08:22
*** shardy_afk is now known as shardy08:24
*** Ryan_Lane has quit IRC08:25
*** vkozhukalov has joined #openstack-infra08:31
*** masayukig has quit IRC08:32
*** SergeyLukjanov has joined #openstack-infra08:32
*** boris-42 has joined #openstack-infra08:37
*** mancdaz_away is now known as mancdaz08:38
*** jcooley_ has joined #openstack-infra08:42
*** thomasbiege has joined #openstack-infra08:43
*** thomasbiege has quit IRC08:44
*** jcooley_ has quit IRC08:48
*** jcoufal has joined #openstack-infra08:52
*** hashar has joined #openstack-infra08:53
*** yassine has joined #openstack-infra08:54
*** jcoufal_ has joined #openstack-infra08:56
*** jcoufal has quit IRC08:58
*** spredzy has quit IRC09:02
*** mugsie has quit IRC09:04
*** LinuxJedi_ has quit IRC09:09
*** guohliu has quit IRC09:10
*** katyafervent has quit IRC09:10
*** ianw has quit IRC09:10
*** Hefeweizen has quit IRC09:10
*** talluri has quit IRC09:10
*** markmcclain has quit IRC09:10
*** salv-orlando has quit IRC09:10
*** simonmcc has quit IRC09:10
*** openstackgerrit has quit IRC09:11
*** med_ has quit IRC09:11
*** samalba has quit IRC09:11
*** jeblair has quit IRC09:11
*** jcooley has quit IRC09:11
*** dragondm has quit IRC09:11
*** vkozhukalov has quit IRC09:11
*** yolanda has quit IRC09:11
*** sdake has quit IRC09:11
*** esker has quit IRC09:11
*** bknudson has quit IRC09:11
*** briancline has quit IRC09:11
*** freyes has quit IRC09:11
*** mgagne has quit IRC09:11
*** vishy has quit IRC09:11
*** zigo has quit IRC09:11
*** Adri2000 has quit IRC09:11
*** cyeoh has quit IRC09:11
*** persia has quit IRC09:11
*** morganfainberg has quit IRC09:11
*** pblaho has quit IRC09:11
*** nati_ueno has quit IRC09:11
*** blamar has quit IRC09:11
*** branen has quit IRC09:11
*** vipul has quit IRC09:11
*** HenryG has quit IRC09:11
*** akscram has quit IRC09:11
*** ilyashakhat has quit IRC09:11
*** agordeev has quit IRC09:11
*** dkranz has quit IRC09:11
*** dims_ has quit IRC09:11
*** praneshp has quit IRC09:11
*** dkehn has quit IRC09:11
*** jhesketh__ has quit IRC09:11
*** yaguang has quit IRC09:11
*** rockstar has quit IRC09:11
*** portante has quit IRC09:11
*** jpeeler has quit IRC09:11
*** thingee has quit IRC09:11
*** basic` has quit IRC09:11
*** pleia2 has quit IRC09:11
*** SpamapS has quit IRC09:11
*** odyi has quit IRC09:11
*** hub_cap has quit IRC09:11
*** koolhead17 has quit IRC09:11
*** hemanth_ has quit IRC09:11
*** kiall has quit IRC09:11
*** BobBall has quit IRC09:11
*** rahmu has quit IRC09:11
*** metabro_ has quit IRC09:11
*** chmouel has quit IRC09:11
*** Daviey has quit IRC09:11
*** wayneeseguin has quit IRC09:11
*** SlickNik has quit IRC09:11
*** Mithrandir has quit IRC09:11
*** antonym has quit IRC09:11
*** sdague has quit IRC09:11
*** EntropyWorks has quit IRC09:11
*** jgriffith has quit IRC09:11
*** mika has quit IRC09:11
*** rainya has quit IRC09:11
*** esmute has quit IRC09:11
*** adam_g has quit IRC09:11
*** stevebaker has quit IRC09:11
*** ttx has quit IRC09:11
*** kpepple has quit IRC09:11
*** jcoufal_ has quit IRC09:11
*** niska has quit IRC09:11
*** nosnos has quit IRC09:11
*** danger_f` has quit IRC09:11
*** jog0 has quit IRC09:11
*** harlowja_away has quit IRC09:11
*** lifeless has quit IRC09:11
*** dstufft has quit IRC09:11
*** kashyap has quit IRC09:11
*** tian has quit IRC09:11
*** jswarren has quit IRC09:11
*** miqui has quit IRC09:11
*** jerryz has quit IRC09:11
*** ywu has quit IRC09:11
*** hashar has quit IRC09:11
*** boris-42 has quit IRC09:11
*** crank has quit IRC09:11
*** sandywalsh_ has quit IRC09:11
*** shardy has quit IRC09:11
*** sirushti has quit IRC09:11
*** primeministerp has quit IRC09:11
*** devananda has quit IRC09:11
*** gema has quit IRC09:11
*** avishayb has quit IRC09:11
*** davidhadas has quit IRC09:11
*** chandankumar has quit IRC09:11
*** mkerrin has quit IRC09:11
*** StevenK has quit IRC09:11
*** pabelanger has quit IRC09:11
*** jd__ has quit IRC09:11
*** ekarlso has quit IRC09:11
*** Grishkin has quit IRC09:11
*** milki has quit IRC09:11
*** SergeyLukjanov has quit IRC09:11
*** zhiyan has quit IRC09:11
*** rongze has quit IRC09:11
*** yamahata has quit IRC09:11
*** gaelL has quit IRC09:11
*** juice has quit IRC09:11
*** mattymo has quit IRC09:11
*** jheiss has quit IRC09:11
*** vladan has quit IRC09:11
*** uvirtbot has quit IRC09:11
*** jesusaurus has quit IRC09:11
*** bogdando has quit IRC09:11
*** rpodolyaka has quit IRC09:11
*** yassine has quit IRC09:11
*** jhesketh_ has quit IRC09:11
*** Shrews has quit IRC09:11
*** jaypipes has quit IRC09:11
*** mordred has quit IRC09:11
*** dansmith has quit IRC09:11
*** spiffxp_ has quit IRC09:11
*** Loquacities has quit IRC09:11
*** pvo has quit IRC09:11
*** Guest33069 has quit IRC09:11
*** AaronGr_Zzz has quit IRC09:11
*** sileht has quit IRC09:11
*** mancdaz has quit IRC09:11
*** jlk has quit IRC09:11
*** EmilienM has quit IRC09:11
*** russellb has quit IRC09:11
*** yongli has quit IRC09:11
*** dhellmann has quit IRC09:11
*** torgomatic has quit IRC09:11
*** davidlenwell has quit IRC09:11
*** iccha-dinner has quit IRC09:11
*** hughsaunders has quit IRC09:11
*** maurosr has quit IRC09:11
*** saper has quit IRC09:11
*** Protux has quit IRC09:11
*** olaph has quit IRC09:11
*** zaro has quit IRC09:11
*** mindjiver has quit IRC09:11
*** david-lyle has quit IRC09:11
*** alaski has quit IRC09:11
*** _cyril_ has quit IRC09:11
*** guitarzan has quit IRC09:11
*** GheRivero has quit IRC09:11
*** hdd has quit IRC09:11
*** clarkb has quit IRC09:11
*** Ng has quit IRC09:11
*** dachary has quit IRC09:11
*** mikal has quit IRC09:11
*** nikhil__ has quit IRC09:11
*** westmau5 has quit IRC09:11
*** jeremyb has quit IRC09:11
*** greghaynes has quit IRC09:11
*** Alex_Gaynor has quit IRC09:11
*** fbo_away has quit IRC09:11
*** notmyname has quit IRC09:11
*** gfa has quit IRC09:11
*** ewindisch has quit IRC09:11
*** ogelbukh has quit IRC09:11
*** adarazs has quit IRC09:11
*** annegentle_ has quit IRC09:11
*** enikanorov has quit IRC09:11
*** mkoderer_ has quit IRC09:11
*** tsufiev_ has quit IRC09:11
*** belliott has quit IRC09:11
*** nijaba has quit IRC09:11
*** SushilKM has quit IRC09:11
*** slong has quit IRC09:11
*** CaptTofu has quit IRC09:11
*** nicedice has quit IRC09:11
*** zul has quit IRC09:11
*** bnemec has quit IRC09:11
*** ChanServ has quit IRC09:11
*** Hunner has quit IRC09:11
*** andreaf has quit IRC09:11
*** adalbas has quit IRC09:11
*** luisg has quit IRC09:11
*** mtreinish has quit IRC09:11
*** yamahata__ has quit IRC09:11
*** marun has quit IRC09:11
*** michchap has quit IRC09:11
*** fungi has quit IRC09:11
*** sbadia has quit IRC09:11
*** anteaya has quit IRC09:11
*** LinuxJedi has joined #openstack-infra09:12
*** thomasbiege has joined #openstack-infra09:12
*** afazekas has joined #openstack-infra09:12
*** jpich has joined #openstack-infra09:12
*** mugsie_ has joined #openstack-infra09:12
*** spredzy_ has joined #openstack-infra09:12
*** jcoufal has joined #openstack-infra09:12
*** yassine has joined #openstack-infra09:12
*** hashar has joined #openstack-infra09:12
*** boris-42 has joined #openstack-infra09:12
*** SergeyLukjanov has joined #openstack-infra09:12
*** vkozhukalov has joined #openstack-infra09:12
*** SushilKM has joined #openstack-infra09:12
*** davidhadas has joined #openstack-infra09:12
*** yolanda has joined #openstack-infra09:12
*** dims_ has joined #openstack-infra09:12
*** zhiyan has joined #openstack-infra09:12
*** pblaho has joined #openstack-infra09:12
*** praneshp has joined #openstack-infra09:12
*** dkehn has joined #openstack-infra09:12
*** crank has joined #openstack-infra09:12
*** rongze has joined #openstack-infra09:12
*** slong has joined #openstack-infra09:12
*** nati_ueno has joined #openstack-infra09:12
*** talluri has joined #openstack-infra09:12
*** CaptTofu has joined #openstack-infra09:12
*** guohliu has joined #openstack-infra09:12
*** niska has joined #openstack-infra09:12
*** chandankumar has joined #openstack-infra09:12
*** jhesketh__ has joined #openstack-infra09:12
*** nicedice has joined #openstack-infra09:12
*** jhesketh_ has joined #openstack-infra09:12
*** sdake has joined #openstack-infra09:12
*** yaguang has joined #openstack-infra09:12
*** nosnos has joined #openstack-infra09:12
*** blamar has joined #openstack-infra09:12
*** esker has joined #openstack-infra09:12
*** yamahata has joined #openstack-infra09:12
*** markmcclain has joined #openstack-infra09:12
*** branen has joined #openstack-infra09:12
*** bknudson has joined #openstack-infra09:12
*** briancline has joined #openstack-infra09:12
*** sandywalsh_ has joined #openstack-infra09:12
*** Hunner has joined #openstack-infra09:12
*** miqui has joined #openstack-infra09:12
*** salv-orlando has joined #openstack-infra09:12
*** vipul has joined #openstack-infra09:12
*** freyes has joined #openstack-infra09:12
*** simonmcc has joined #openstack-infra09:12
*** andreaf has joined #openstack-infra09:12
*** jswarren has joined #openstack-infra09:12
*** katyafervent has joined #openstack-infra09:12
*** adalbas has joined #openstack-infra09:12
*** HenryG has joined #openstack-infra09:12
*** luisg has joined #openstack-infra09:12
*** rpodolyaka has joined #openstack-infra09:12
*** _cyril_ has joined #openstack-infra09:12
*** mtreinish has joined #openstack-infra09:12
*** yamahata__ has joined #openstack-infra09:12
*** akscram has joined #openstack-infra09:12
*** ilyashakhat has joined #openstack-infra09:12
*** marun has joined #openstack-infra09:12
*** zul has joined #openstack-infra09:12
*** agordeev has joined #openstack-infra09:12
*** avishayb has joined #openstack-infra09:12
*** michchap has joined #openstack-infra09:12
*** greghaynes has joined #openstack-infra09:12
*** dkranz has joined #openstack-infra09:12
*** mgagne has joined #openstack-infra09:12
*** ianw has joined #openstack-infra09:12
*** openstackgerrit has joined #openstack-infra09:12
*** bnemec has joined #openstack-infra09:12
*** bogdando has joined #openstack-infra09:12
*** Hefeweizen has joined #openstack-infra09:12
*** jerryz has joined #openstack-infra09:12
*** fungi has joined #openstack-infra09:12
*** shardy has joined #openstack-infra09:12
*** Shrews has joined #openstack-infra09:12
*** rockstar has joined #openstack-infra09:12
*** jaypipes has joined #openstack-infra09:12
*** portante has joined #openstack-infra09:12
*** mordred has joined #openstack-infra09:12
*** dansmith has joined #openstack-infra09:12
*** dragondm has joined #openstack-infra09:12
*** BobBall has joined #openstack-infra09:12
*** jcooley has joined #openstack-infra09:12
*** kiall has joined #openstack-infra09:12
*** jeblair has joined #openstack-infra09:12
*** samalba has joined #openstack-infra09:12
*** koolhead17 has joined #openstack-infra09:12
*** med_ has joined #openstack-infra09:12
*** kpepple has joined #openstack-infra09:12
*** ttx has joined #openstack-infra09:12
*** stevebaker has joined #openstack-infra09:12
*** adam_g has joined #openstack-infra09:12
*** hub_cap has joined #openstack-infra09:12
*** esmute has joined #openstack-infra09:12
*** odyi has joined #openstack-infra09:12
*** mika has joined #openstack-infra09:12
*** jgriffith has joined #openstack-infra09:12
*** EntropyWorks has joined #openstack-infra09:12
*** SpamapS has joined #openstack-infra09:12
*** pleia2 has joined #openstack-infra09:12
*** basic` has joined #openstack-infra09:12
*** thingee has joined #openstack-infra09:12
*** hemanth_ has joined #openstack-infra09:12
*** jpeeler has joined #openstack-infra09:12
*** sdague has joined #openstack-infra09:12
*** rainya has joined #openstack-infra09:12
*** antonym has joined #openstack-infra09:12
*** Mithrandir has joined #openstack-infra09:12
*** vishy has joined #openstack-infra09:12
*** zigo has joined #openstack-infra09:12
*** Adri2000 has joined #openstack-infra09:12
*** cyeoh has joined #openstack-infra09:12
*** persia has joined #openstack-infra09:12
*** morganfainberg has joined #openstack-infra09:12
*** SlickNik has joined #openstack-infra09:12
*** wayneeseguin has joined #openstack-infra09:12
*** chmouel has joined #openstack-infra09:12
*** metabro_ has joined #openstack-infra09:12
*** rahmu has joined #openstack-infra09:12
*** Daviey has joined #openstack-infra09:12
*** spiffxp_ has joined #openstack-infra09:12
*** mkerrin has joined #openstack-infra09:12
*** yongli has joined #openstack-infra09:12
*** dhellmann has joined #openstack-infra09:12
*** torgomatic has joined #openstack-infra09:12
*** danger_f` has joined #openstack-infra09:12
*** Alex_Gaynor has joined #openstack-infra09:12
*** annegentle_ has joined #openstack-infra09:12
*** sirushti has joined #openstack-infra09:12
*** davidlenwell has joined #openstack-infra09:12
*** primeministerp has joined #openstack-infra09:12
*** fbo_away has joined #openstack-infra09:12
*** notmyname has joined #openstack-infra09:12
*** devananda has joined #openstack-infra09:12
*** gfa has joined #openstack-infra09:12
*** jog0 has joined #openstack-infra09:12
*** ewindisch has joined #openstack-infra09:12
*** ogelbukh has joined #openstack-infra09:12
*** adarazs has joined #openstack-infra09:12
*** harlowja_away has joined #openstack-infra09:12
*** enikanorov has joined #openstack-infra09:12
*** lifeless has joined #openstack-infra09:12
*** mkoderer_ has joined #openstack-infra09:12
*** belliott has joined #openstack-infra09:12
*** tsufiev_ has joined #openstack-infra09:12
*** dstufft has joined #openstack-infra09:12
*** kashyap has joined #openstack-infra09:12
*** gaelL has joined #openstack-infra09:12
*** tian has joined #openstack-infra09:12
*** juice has joined #openstack-infra09:12
*** guitarzan has joined #openstack-infra09:12
*** GheRivero has joined #openstack-infra09:12
*** iccha-dinner has joined #openstack-infra09:12
*** StevenK has joined #openstack-infra09:12
*** hughsaunders has joined #openstack-infra09:12
*** maurosr has joined #openstack-infra09:12
*** pabelanger has joined #openstack-infra09:12
*** jd__ has joined #openstack-infra09:12
*** saper has joined #openstack-infra09:12
*** gema has joined #openstack-infra09:12
*** ekarlso has joined #openstack-infra09:12
*** ywu has joined #openstack-infra09:12
*** Loquacities has joined #openstack-infra09:12
*** Protux has joined #openstack-infra09:12
*** olaph has joined #openstack-infra09:12
*** Grishkin has joined #openstack-infra09:12
*** pvo has joined #openstack-infra09:12
*** Guest33069 has joined #openstack-infra09:12
*** hdd has joined #openstack-infra09:12
*** AaronGr_Zzz has joined #openstack-infra09:12
*** mattymo has joined #openstack-infra09:12
*** clarkb has joined #openstack-infra09:12
*** zaro has joined #openstack-infra09:12
*** sileht has joined #openstack-infra09:12
*** mancdaz has joined #openstack-infra09:12
*** milki has joined #openstack-infra09:12
*** mindjiver has joined #openstack-infra09:12
*** sbadia has joined #openstack-infra09:12
*** jheiss has joined #openstack-infra09:12
*** Ng has joined #openstack-infra09:12
*** david-lyle has joined #openstack-infra09:12
*** jlk has joined #openstack-infra09:12
*** vladan has joined #openstack-infra09:12
*** dachary has joined #openstack-infra09:12
*** uvirtbot has joined #openstack-infra09:12
*** EmilienM has joined #openstack-infra09:12
*** russellb has joined #openstack-infra09:12
*** jesusaurus has joined #openstack-infra09:12
*** mikal has joined #openstack-infra09:12
*** alaski has joined #openstack-infra09:12
*** nikhil__ has joined #openstack-infra09:12
*** anteaya has joined #openstack-infra09:12
*** nijaba has joined #openstack-infra09:12
*** westmau5 has joined #openstack-infra09:12
*** jeremyb has joined #openstack-infra09:12
*** ChanServ has joined #openstack-infra09:12
*** hobana.freenode.net sets mode: +o ChanServ09:12
*** flaper87 has quit IRC09:13
*** flaper87 has joined #openstack-infra09:13
*** sarob has joined #openstack-infra09:13
*** crank has quit IRC09:15
*** guohliu has quit IRC09:16
*** katyafervent has quit IRC09:16
*** ianw has quit IRC09:16
*** Hefeweizen has quit IRC09:16
*** talluri has quit IRC09:16
*** markmcclain has quit IRC09:16
*** salv-orlando has quit IRC09:16
*** simonmcc has quit IRC09:16
*** openstackgerrit has quit IRC09:16
*** med_ has quit IRC09:16
*** samalba has quit IRC09:16
*** jeblair has quit IRC09:16
*** jcooley has quit IRC09:16
*** dragondm has quit IRC09:16
*** sarob has quit IRC09:16
*** vkozhukalov has quit IRC09:16
*** yolanda has quit IRC09:16
*** sdake has quit IRC09:16
*** esker has quit IRC09:16
*** bknudson has quit IRC09:16
*** briancline has quit IRC09:16
*** freyes has quit IRC09:16
*** mgagne has quit IRC09:16
*** vishy has quit IRC09:16
*** zigo has quit IRC09:16
*** Adri2000 has quit IRC09:16
*** cyeoh has quit IRC09:16
*** persia has quit IRC09:16
*** morganfainberg has quit IRC09:16
*** spredzy_ has quit IRC09:16
*** pblaho has quit IRC09:16
*** nati_ueno has quit IRC09:16
*** blamar has quit IRC09:16
*** branen has quit IRC09:16
*** vipul has quit IRC09:16
*** HenryG has quit IRC09:16
*** akscram has quit IRC09:16
*** ilyashakhat has quit IRC09:16
*** agordeev has quit IRC09:16
*** dkranz has quit IRC09:16
*** dims_ has quit IRC09:16
*** praneshp has quit IRC09:16
*** dkehn has quit IRC09:16
*** jhesketh__ has quit IRC09:16
*** yaguang has quit IRC09:16
*** rockstar has quit IRC09:16
*** portante has quit IRC09:16
*** jpeeler has quit IRC09:16
*** thingee has quit IRC09:16
*** basic` has quit IRC09:16
*** pleia2 has quit IRC09:16
*** SpamapS has quit IRC09:16
*** odyi has quit IRC09:16
*** hub_cap has quit IRC09:16
*** koolhead17 has quit IRC09:16
*** hemanth_ has quit IRC09:16
*** kiall has quit IRC09:16
*** BobBall has quit IRC09:16
*** rahmu has quit IRC09:16
*** metabro_ has quit IRC09:16
*** chmouel has quit IRC09:16
*** Daviey has quit IRC09:16
*** wayneeseguin has quit IRC09:16
*** SlickNik has quit IRC09:16
*** Mithrandir has quit IRC09:16
*** antonym has quit IRC09:16
*** sdague has quit IRC09:16
*** EntropyWorks has quit IRC09:16
*** jgriffith has quit IRC09:16
*** mika has quit IRC09:16
*** rainya has quit IRC09:16
*** esmute has quit IRC09:16
*** adam_g has quit IRC09:16
*** stevebaker has quit IRC09:16
*** ttx has quit IRC09:16
*** kpepple has quit IRC09:16
*** LinuxJedi has quit IRC09:16
*** afazekas has quit IRC09:16
*** jcoufal has quit IRC09:16
*** niska has quit IRC09:16
*** nosnos has quit IRC09:16
*** danger_f` has quit IRC09:16
*** jog0 has quit IRC09:16
*** harlowja_away has quit IRC09:16
*** lifeless has quit IRC09:16
*** dstufft has quit IRC09:16
*** kashyap has quit IRC09:16
*** tian has quit IRC09:16
*** jswarren has quit IRC09:16
*** miqui has quit IRC09:16
*** jerryz has quit IRC09:16
*** ywu has quit IRC09:16
*** hashar has quit IRC09:16
*** boris-42 has quit IRC09:16
*** sandywalsh_ has quit IRC09:16
*** shardy has quit IRC09:16
*** sirushti has quit IRC09:16
*** primeministerp has quit IRC09:16
*** devananda has quit IRC09:16
*** gema has quit IRC09:16
*** avishayb has quit IRC09:16
*** davidhadas has quit IRC09:16
*** chandankumar has quit IRC09:16
*** mkerrin has quit IRC09:16
*** StevenK has quit IRC09:16
*** pabelanger has quit IRC09:16
*** jd__ has quit IRC09:16
*** ekarlso has quit IRC09:16
*** Grishkin has quit IRC09:16
*** milki has quit IRC09:16
*** jpich has quit IRC09:16
*** mugsie_ has quit IRC09:16
*** SergeyLukjanov has quit IRC09:16
*** zhiyan has quit IRC09:16
*** rongze has quit IRC09:16
*** yamahata has quit IRC09:16
*** gaelL has quit IRC09:16
*** juice has quit IRC09:16
*** mattymo has quit IRC09:16
*** jheiss has quit IRC09:16
*** vladan has quit IRC09:16
*** uvirtbot has quit IRC09:16
*** jesusaurus has quit IRC09:16
*** bogdando has quit IRC09:16
*** rpodolyaka has quit IRC09:16
*** thomasbiege has quit IRC09:16
*** yassine has quit IRC09:16
*** jhesketh_ has quit IRC09:16
*** Shrews has quit IRC09:16
*** jaypipes has quit IRC09:16
*** mordred has quit IRC09:16
*** dansmith has quit IRC09:16
*** spiffxp_ has quit IRC09:16
*** Loquacities has quit IRC09:16
*** pvo has quit IRC09:16
*** Guest33069 has quit IRC09:16
*** AaronGr_Zzz has quit IRC09:16
*** sileht has quit IRC09:16
*** mancdaz has quit IRC09:16
*** jlk has quit IRC09:16
*** EmilienM has quit IRC09:16
*** russellb has quit IRC09:16
*** yongli has quit IRC09:16
*** dhellmann has quit IRC09:16
*** torgomatic has quit IRC09:16
*** davidlenwell has quit IRC09:16
*** iccha-dinner has quit IRC09:16
*** hughsaunders has quit IRC09:16
*** maurosr has quit IRC09:16
*** saper has quit IRC09:16
*** Protux has quit IRC09:16
*** olaph has quit IRC09:16
*** zaro has quit IRC09:16
*** mindjiver has quit IRC09:16
*** david-lyle has quit IRC09:16
*** alaski has quit IRC09:16
*** _cyril_ has quit IRC09:16
*** guitarzan has quit IRC09:16
*** GheRivero has quit IRC09:16
*** hdd has quit IRC09:16
*** clarkb has quit IRC09:16
*** Ng has quit IRC09:16
*** dachary has quit IRC09:16
*** mikal has quit IRC09:16
*** nikhil__ has quit IRC09:16
*** westmau5 has quit IRC09:16
*** jeremyb has quit IRC09:16
*** greghaynes has quit IRC09:16
*** Alex_Gaynor has quit IRC09:16
*** fbo_away has quit IRC09:16
*** notmyname has quit IRC09:16
*** gfa has quit IRC09:16
*** ewindisch has quit IRC09:16
*** ogelbukh has quit IRC09:16
*** adarazs has quit IRC09:16
*** annegentle_ has quit IRC09:16
*** enikanorov has quit IRC09:16
*** mkoderer_ has quit IRC09:16
*** tsufiev_ has quit IRC09:16
*** belliott has quit IRC09:16
*** nijaba has quit IRC09:16
*** SushilKM has quit IRC09:16
*** slong has quit IRC09:16
*** CaptTofu has quit IRC09:16
*** nicedice has quit IRC09:16
*** zul has quit IRC09:16
*** bnemec has quit IRC09:16
*** ChanServ has quit IRC09:17
*** Hunner has quit IRC09:17
*** andreaf has quit IRC09:17
*** adalbas has quit IRC09:17
*** luisg has quit IRC09:17
*** mtreinish has quit IRC09:17
*** yamahata__ has quit IRC09:17
*** marun has quit IRC09:17
*** michchap has quit IRC09:17
*** fungi has quit IRC09:17
*** sbadia has quit IRC09:17
*** anteaya has quit IRC09:17
*** Ryan_Lane has joined #openstack-infra09:22
*** derekh has joined #openstack-infra09:22
*** LinuxJedi has joined #openstack-infra09:22
*** thomasbiege has joined #openstack-infra09:22
*** afazekas has joined #openstack-infra09:22
*** jpich has joined #openstack-infra09:22
*** mugsie_ has joined #openstack-infra09:22
*** spredzy_ has joined #openstack-infra09:22
*** jcoufal has joined #openstack-infra09:22
*** yassine has joined #openstack-infra09:22
*** hashar has joined #openstack-infra09:22
*** boris-42 has joined #openstack-infra09:22
*** SergeyLukjanov has joined #openstack-infra09:22
*** vkozhukalov has joined #openstack-infra09:22
*** SushilKM has joined #openstack-infra09:22
*** davidhadas has joined #openstack-infra09:22
*** yolanda has joined #openstack-infra09:22
*** dims_ has joined #openstack-infra09:22
*** zhiyan has joined #openstack-infra09:22
*** pblaho has joined #openstack-infra09:22
*** praneshp has joined #openstack-infra09:22
*** dkehn has joined #openstack-infra09:22
*** rongze has joined #openstack-infra09:22
*** slong has joined #openstack-infra09:22
*** nati_ueno has joined #openstack-infra09:22
*** talluri has joined #openstack-infra09:22
*** CaptTofu has joined #openstack-infra09:22
*** guohliu has joined #openstack-infra09:22
*** niska has joined #openstack-infra09:22
*** chandankumar has joined #openstack-infra09:22
*** nicedice has joined #openstack-infra09:22
*** sdake has joined #openstack-infra09:22
*** yaguang has joined #openstack-infra09:22
*** nosnos has joined #openstack-infra09:22
*** blamar has joined #openstack-infra09:22
*** esker has joined #openstack-infra09:22
*** yamahata has joined #openstack-infra09:22
*** markmcclain has joined #openstack-infra09:22
*** branen has joined #openstack-infra09:22
*** bknudson has joined #openstack-infra09:22
*** briancline has joined #openstack-infra09:22
*** sandywalsh_ has joined #openstack-infra09:22
*** Hunner has joined #openstack-infra09:22
*** miqui has joined #openstack-infra09:22
*** salv-orlando has joined #openstack-infra09:22
*** vipul has joined #openstack-infra09:22
*** freyes has joined #openstack-infra09:22
*** simonmcc has joined #openstack-infra09:22
*** andreaf has joined #openstack-infra09:22
*** jswarren has joined #openstack-infra09:22
*** katyafervent has joined #openstack-infra09:22
*** adalbas has joined #openstack-infra09:22
*** HenryG has joined #openstack-infra09:22
*** luisg has joined #openstack-infra09:22
*** rpodolyaka has joined #openstack-infra09:22
*** _cyril_ has joined #openstack-infra09:22
*** mtreinish has joined #openstack-infra09:22
*** yamahata__ has joined #openstack-infra09:22
*** akscram has joined #openstack-infra09:22
*** ilyashakhat has joined #openstack-infra09:22
*** marun has joined #openstack-infra09:22
*** zul has joined #openstack-infra09:22
*** agordeev has joined #openstack-infra09:22
*** avishayb has joined #openstack-infra09:22
*** michchap has joined #openstack-infra09:22
*** greghaynes has joined #openstack-infra09:22
*** dkranz has joined #openstack-infra09:22
*** mgagne has joined #openstack-infra09:22
*** ianw has joined #openstack-infra09:22
*** openstackgerrit has joined #openstack-infra09:22
*** bnemec has joined #openstack-infra09:22
*** bogdando has joined #openstack-infra09:22
*** Hefeweizen has joined #openstack-infra09:22
*** jerryz has joined #openstack-infra09:22
*** fungi has joined #openstack-infra09:22
*** shardy has joined #openstack-infra09:22
*** Shrews has joined #openstack-infra09:22
*** rockstar has joined #openstack-infra09:22
*** jaypipes has joined #openstack-infra09:22
*** portante has joined #openstack-infra09:22
*** mordred has joined #openstack-infra09:22
*** dansmith has joined #openstack-infra09:22
*** dragondm has joined #openstack-infra09:22
*** BobBall has joined #openstack-infra09:22
*** jcooley has joined #openstack-infra09:22
*** kiall has joined #openstack-infra09:22
*** jeblair has joined #openstack-infra09:22
*** samalba has joined #openstack-infra09:22
*** koolhead17 has joined #openstack-infra09:22
*** med_ has joined #openstack-infra09:22
*** kpepple has joined #openstack-infra09:22
*** ttx has joined #openstack-infra09:22
*** stevebaker has joined #openstack-infra09:22
*** adam_g has joined #openstack-infra09:22
*** hub_cap has joined #openstack-infra09:22
*** esmute has joined #openstack-infra09:22
*** odyi has joined #openstack-infra09:22
*** mika has joined #openstack-infra09:22
*** jgriffith has joined #openstack-infra09:22
*** EntropyWorks has joined #openstack-infra09:22
*** SpamapS has joined #openstack-infra09:22
*** pleia2 has joined #openstack-infra09:22
*** basic` has joined #openstack-infra09:22
*** thingee has joined #openstack-infra09:22
*** hemanth_ has joined #openstack-infra09:22
*** jpeeler has joined #openstack-infra09:22
*** sdague has joined #openstack-infra09:22
*** rainya has joined #openstack-infra09:22
*** antonym has joined #openstack-infra09:22
*** Mithrandir has joined #openstack-infra09:22
*** vishy has joined #openstack-infra09:22
*** zigo has joined #openstack-infra09:22
*** Adri2000 has joined #openstack-infra09:22
*** cyeoh has joined #openstack-infra09:22
*** persia has joined #openstack-infra09:22
*** morganfainberg has joined #openstack-infra09:22
*** SlickNik has joined #openstack-infra09:22
*** wayneeseguin has joined #openstack-infra09:22
*** chmouel has joined #openstack-infra09:22
*** metabro_ has joined #openstack-infra09:22
*** rahmu has joined #openstack-infra09:22
*** Daviey has joined #openstack-infra09:22
*** spiffxp_ has joined #openstack-infra09:22
*** mkerrin has joined #openstack-infra09:22
*** yongli has joined #openstack-infra09:22
*** dhellmann has joined #openstack-infra09:22
*** torgomatic has joined #openstack-infra09:22
*** danger_f` has joined #openstack-infra09:22
*** Alex_Gaynor has joined #openstack-infra09:22
*** annegentle_ has joined #openstack-infra09:22
*** sirushti has joined #openstack-infra09:22
*** davidlenwell has joined #openstack-infra09:22
*** primeministerp has joined #openstack-infra09:22
*** fbo_away has joined #openstack-infra09:22
*** notmyname has joined #openstack-infra09:22
*** devananda has joined #openstack-infra09:22
*** gfa has joined #openstack-infra09:22
*** jog0 has joined #openstack-infra09:22
*** ewindisch has joined #openstack-infra09:22
*** ogelbukh has joined #openstack-infra09:22
*** adarazs has joined #openstack-infra09:22
*** harlowja_away has joined #openstack-infra09:22
*** enikanorov has joined #openstack-infra09:22
*** lifeless has joined #openstack-infra09:22
*** mkoderer_ has joined #openstack-infra09:22
*** belliott has joined #openstack-infra09:22
*** tsufiev_ has joined #openstack-infra09:22
*** dstufft has joined #openstack-infra09:22
*** kashyap has joined #openstack-infra09:22
*** gaelL has joined #openstack-infra09:22
*** tian has joined #openstack-infra09:22
*** juice has joined #openstack-infra09:22
*** guitarzan has joined #openstack-infra09:22
*** GheRivero has joined #openstack-infra09:22
*** iccha-dinner has joined #openstack-infra09:22
*** StevenK has joined #openstack-infra09:22
*** hughsaunders has joined #openstack-infra09:22
*** maurosr has joined #openstack-infra09:22
*** pabelanger has joined #openstack-infra09:22
*** jd__ has joined #openstack-infra09:22
*** saper has joined #openstack-infra09:22
*** gema has joined #openstack-infra09:22
*** ekarlso has joined #openstack-infra09:22
*** ywu has joined #openstack-infra09:22
*** Loquacities has joined #openstack-infra09:22
*** Protux has joined #openstack-infra09:22
*** olaph has joined #openstack-infra09:22
*** Grishkin has joined #openstack-infra09:22
*** pvo has joined #openstack-infra09:22
*** Guest33069 has joined #openstack-infra09:22
*** hdd has joined #openstack-infra09:22
*** AaronGr_Zzz has joined #openstack-infra09:22
*** mattymo has joined #openstack-infra09:22
*** clarkb has joined #openstack-infra09:22
*** zaro has joined #openstack-infra09:22
*** sileht has joined #openstack-infra09:22
*** mancdaz has joined #openstack-infra09:22
*** milki has joined #openstack-infra09:22
*** mindjiver has joined #openstack-infra09:22
*** sbadia has joined #openstack-infra09:22
*** jheiss has joined #openstack-infra09:22
*** Ng has joined #openstack-infra09:22
*** david-lyle has joined #openstack-infra09:22
*** jlk has joined #openstack-infra09:22
*** vladan has joined #openstack-infra09:22
*** dachary has joined #openstack-infra09:22
*** uvirtbot has joined #openstack-infra09:22
*** EmilienM has joined #openstack-infra09:22
*** russellb has joined #openstack-infra09:22
*** jesusaurus has joined #openstack-infra09:22
*** mikal has joined #openstack-infra09:22
*** alaski has joined #openstack-infra09:22
*** nikhil__ has joined #openstack-infra09:22
*** anteaya has joined #openstack-infra09:22
*** nijaba has joined #openstack-infra09:22
*** westmau5 has joined #openstack-infra09:22
*** jeremyb has joined #openstack-infra09:22
*** ChanServ has joined #openstack-infra09:22
*** hobana.freenode.net sets mode: +o ChanServ09:22
*** crank has joined #openstack-infra09:22
*** Ryan_Lane has quit IRC09:24
*** crank has quit IRC09:27
*** crank has joined #openstack-infra09:28
*** jcooley_ has joined #openstack-infra09:30
*** tsufiev_ has quit IRC09:31
*** crank has quit IRC09:33
*** crank has joined #openstack-infra09:34
*** jcooley_ has quit IRC09:35
*** fbo_away is now known as fbo09:38
*** crank has quit IRC09:39
*** ruhe has joined #openstack-infra09:46
*** crank has joined #openstack-infra09:47
*** crank has quit IRC09:52
*** crank_ has joined #openstack-infra09:52
*** fifieldt has joined #openstack-infra09:54
*** crank_ has quit IRC09:57
*** crank has joined #openstack-infra09:58
*** fifieldt has quit IRC10:08
*** jooools has joined #openstack-infra10:08
*** crank has quit IRC10:12
*** sarob has joined #openstack-infra10:12
*** crank has joined #openstack-infra10:13
*** rongze has quit IRC10:14
*** dhellmann has quit IRC10:14
*** mancdaz is now known as mancdaz_away10:15
*** elasticio has joined #openstack-infra10:15
*** dhellmann has joined #openstack-infra10:15
*** sarob has quit IRC10:17
*** crank has quit IRC10:17
*** crank has joined #openstack-infra10:18
*** jhesketh_ has joined #openstack-infra10:19
*** Ryan_Lane has joined #openstack-infra10:19
*** jhesketh_ has quit IRC10:20
*** mancdaz_away is now known as mancdaz10:21
*** ruhe has quit IRC10:23
*** salv-orlando has quit IRC10:23
*** jcooley_ has joined #openstack-infra10:23
*** Ryan_Lane has quit IRC10:24
*** jcooley_ has quit IRC10:28
*** fifieldt has joined #openstack-infra10:30
*** nati_ueno has quit IRC10:34
*** yamahata has quit IRC10:42
*** yamahata__ has quit IRC10:42
*** yamahata__ has joined #openstack-infra10:42
*** yamahata__ has quit IRC10:43
*** yamahata__ has joined #openstack-infra10:43
*** elasticio has quit IRC10:48
*** crank has quit IRC10:49
*** crank has joined #openstack-infra10:50
*** yolanda has quit IRC10:51
*** yolanda has joined #openstack-infra10:51
*** fifieldt has quit IRC11:04
*** ArxCruz has joined #openstack-infra11:04
*** sarob has joined #openstack-infra11:12
*** nati_ueno has joined #openstack-infra11:13
*** SergeyLukjanov is now known as _SergeyLukjanov11:13
*** _SergeyLukjanov has quit IRC11:14
*** sarob has quit IRC11:17
*** jcooley_ has joined #openstack-infra11:18
*** crank has quit IRC11:18
*** Ryan_Lane has joined #openstack-infra11:19
*** crank has joined #openstack-infra11:19
*** NikitaKonovalov has joined #openstack-infra11:20
*** lcestari has joined #openstack-infra11:22
*** jcooley_ has quit IRC11:23
*** Ryan_Lane has quit IRC11:24
*** rongze has joined #openstack-infra11:24
*** mugsie_ has quit IRC11:24
*** mugsie_ has joined #openstack-infra11:24
*** mugsie_ is now known as mugsie11:24
*** che-arne has joined #openstack-infra11:25
*** elasticio has joined #openstack-infra11:35
*** guohliu has quit IRC11:36
*** chenglin has joined #openstack-infra11:44
*** marun has quit IRC11:48
*** praneshp has quit IRC11:51
*** chenglin has quit IRC11:52
*** chenglin has joined #openstack-infra11:52
*** chenglin has quit IRC11:53
*** nosnos has quit IRC11:53
*** plomakin has joined #openstack-infra11:53
*** chenglin has joined #openstack-infra11:54
*** chenglin has quit IRC11:54
*** chenglin has joined #openstack-infra11:54
*** chenglin has quit IRC11:54
*** vogxn has joined #openstack-infra11:55
*** lcheng has joined #openstack-infra11:55
*** elasticio has quit IRC11:55
*** crank has quit IRC12:00
*** talluri has quit IRC12:01
*** afazekas_ has joined #openstack-infra12:01
*** crank has joined #openstack-infra12:01
*** talluri has joined #openstack-infra12:01
*** dolphm has joined #openstack-infra12:02
*** CaptTofu has quit IRC12:04
*** CaptTofu has joined #openstack-infra12:04
BobBallsdague: Regarding nose rename of create_test_server - I was requested to restore the change by matthew treinish due to people trying to increase the quotas to fix a different test.12:05
*** talluri has quit IRC12:06
BobBallsdague: I'm more than happy for the change to be abandoned (as it had been before) - but do you have an issue with just renaming the method?  or is the issue the reason that I want to rename it?12:06
*** markmc has joined #openstack-infra12:08
*** zhiyan has quit IRC12:08
*** jcooley_ has joined #openstack-infra12:11
*** rfolco has joined #openstack-infra12:11
*** sarob has joined #openstack-infra12:12
*** zhiyan has joined #openstack-infra12:13
*** miqui has quit IRC12:13
*** yaguang has quit IRC12:16
*** salv-orlando has joined #openstack-infra12:16
*** jcooley_ has quit IRC12:17
*** Ryan_Lane has joined #openstack-infra12:19
*** tma996 has joined #openstack-infra12:19
*** krtaylor has joined #openstack-infra12:21
*** avishayb has quit IRC12:23
*** Ryan_Lane has quit IRC12:24
*** smarcet has joined #openstack-infra12:29
*** avishayb has joined #openstack-infra12:31
*** crank has quit IRC12:34
*** Abhishek has joined #openstack-infra12:34
*** crank has joined #openstack-infra12:34
*** sandywalsh_ has quit IRC12:36
sdagueso we keep putting bandaids around nose12:36
sdagueand we need to just fix whatever is keeping people from using testr12:36
sdagueand explode if they try to run under nose12:36
BobBallI understand that - but last time I talked with _someone_ (can't remember who) the suggestion was that we can't deprecate nose for a while?12:37
BobBallI figured that this change - even though it's only a bandaid - was a good thing to have to save the current frustrations12:38
*** vogxn has quit IRC12:39
*** dolphm has quit IRC12:39
*** dolphm has joined #openstack-infra12:41
*** thomasem has joined #openstack-infra12:43
*** zhiyan has left #openstack-infra12:43
*** sarob has quit IRC12:45
*** crank has quit IRC12:48
sdagueso the problem is it's not trivial, but lifeless had a path to get us there12:49
*** sandywalsh_ has joined #openstack-infra12:49
*** crank has joined #openstack-infra12:49
*** yolanda has quit IRC12:50
sdagueso I'm just putting my foot down and someone should work on it, no new nose fixes.12:50
*** yolanda has joined #openstack-infra12:50
sdaguebecause nose working in the tree actually means we get tests in that aren't parallel safe12:50
sdagueand I've had to revert a couple of those12:50
BobBallOK - perhaps an alternative change would be to require the "--use-deprecated-nose" flag or abort ;)12:51
BobBallI'll abandon my change12:52
BobBallI'm just not sure that most of the active people will care much because "everyone uses tox"12:52
*** dprince has joined #openstack-infra12:59
sdagueman grizzly changes are just full of fails13:00
freyeshello everybody13:06
*** jcooley_ has joined #openstack-infra13:07
freyesI abandoned this patch https://review.openstack.org/#/c/61547/ in favor of this one https://review.openstack.org/#/c/60066/, but we haven't had any news since a couple of weeks, what's the recommended time to wait for updates?13:10
sdaguefreyes: yuo had -1 feedback from Kai on Dec 5 which was not addressed13:12
*** xchu has joined #openstack-infra13:12
*** sarob has joined #openstack-infra13:12
sdaguebasically, if a patch has -1 on it, it's going to be largely ignored by other folks, as everyone has a lot on their review queue. So address those comments and upload a new change13:13
*** jcooley_ has quit IRC13:13
*** dkranz has quit IRC13:16
*** sarob has quit IRC13:16
freyessdague, I abandoned my patch, because mathieu ask me to do it, 60066 arrived before and we didn't want to duplicate efforts13:18
*** psedlak_clone is now known as psedlak13:18
*** rongze has quit IRC13:19
*** Ryan_Lane has joined #openstack-infra13:19
*** Protux has quit IRC13:19
*** eharney has joined #openstack-infra13:19
*** rongze has joined #openstack-infra13:21
*** freyes has quit IRC13:23
*** freyes has joined #openstack-infra13:23
*** mriedem has joined #openstack-infra13:23
*** Ryan_Lane has quit IRC13:24
*** esker has quit IRC13:26
*** esker has joined #openstack-infra13:27
*** herndon has joined #openstack-infra13:28
*** dolphm has quit IRC13:28
*** dolphm has joined #openstack-infra13:30
*** esker has quit IRC13:31
*** spredzy_ is now known as spredzy13:42
*** beagles is now known as beagles_brb13:48
*** NikitaKonovalov has quit IRC13:50
*** NikitaKonovalov has joined #openstack-infra13:50
*** SergeyLukjanov has joined #openstack-infra13:51
*** jpich has quit IRC13:52
*** jpich has joined #openstack-infra13:53
*** che-arne has quit IRC13:53
sdaguefreyes: so I would try to ping stewart, and if nothing else you can take over that patch (git review -d ) and make the fixes needed13:53
*** che-arne has joined #openstack-infra13:53
freyessdague, thanks :)13:53
*** amotoki has joined #openstack-infra13:55
*** mattymo has quit IRC13:55
*** mattymo has joined #openstack-infra13:56
*** gothicmindfood has joined #openstack-infra13:58
*** julim has joined #openstack-infra13:58
*** jcooley_ has joined #openstack-infra14:00
*** gothicmindfood has quit IRC14:04
*** jcooley_ has quit IRC14:06
*** yaguang has joined #openstack-infra14:07
*** zul has quit IRC14:09
*** weshay has joined #openstack-infra14:09
*** dkranz has joined #openstack-infra14:09
*** CaptTofu has quit IRC14:10
*** CaptTofu has joined #openstack-infra14:10
*** nati_ueno has quit IRC14:10
*** CaptTofu has quit IRC14:11
*** zul has joined #openstack-infra14:11
*** sarob has joined #openstack-infra14:12
*** plomakin has quit IRC14:12
openstackgerritSalvatore Orlando proposed a change to openstack-infra/elastic-recheck: Add query for bug 1254890  https://review.openstack.org/6289114:13
uvirtbotLaunchpad bug 1254890 in tempest ""Timed out waiting for thing" causes tempest-dsvm-neutron-* failures" [Low,In progress] https://launchpad.net/bugs/125489014:13
*** herndon has quit IRC14:15
*** mfer has joined #openstack-infra14:15
*** dstanek has joined #openstack-infra14:17
*** sarob has quit IRC14:17
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1254890  https://review.openstack.org/6289114:23
uvirtbotLaunchpad bug 1254890 in tempest ""Timed out waiting for thing" causes tempest-dsvm-neutron-* failures" [Low,In progress] https://launchpad.net/bugs/125489014:23
*** oubiwann has joined #openstack-infra14:25
openstackgerritA change was merged to openstack-infra/elastic-recheck: Machine remins in HARD_REBOOT status  https://review.openstack.org/6181514:25
*** dizquierdo has joined #openstack-infra14:28
*** yassine has quit IRC14:33
*** yassine has joined #openstack-infra14:33
*** dkliban has joined #openstack-infra14:39
flaper87guys, any chance I can get your help on figuring out what's happening here? https://review.openstack.org/#/c/32544/14:40
flaper87For some reason, that test fails in the gate but neither me nor other guys I've asked can replicate it14:41
*** dims_ has quit IRC14:44
fungiflaper87: unit.cache.test_cache.TestCacheModule.test_get_cache_options fails on assertEqual(driver._options, {'default_ttl': '4'}) with MismatchError: {} != {'default_ttl': '4'}14:47
fungiflaper87: you're unable to reproduce that under python 2.7 on ubuntu 12.04?14:47
fungii can try and see what i get14:48
*** dims_ has joined #openstack-infra14:48
flaper87I tested it on f19 and Alejandro tested it on Ubuntu (IIRC)14:49
*** jlibosva has joined #openstack-infra14:49
*** SushilKM has quit IRC14:49
*** johnthetubaguy has joined #openstack-infra14:49
flaper87fungi: if you can give it a try, I'd really appreciate it14:49
anteayajlibosva: here you are14:51
anteayafungi: when you have a minute, jlibosva and I were wondering if you can ssh into a vm running his patch and do some exploration14:52
anteayait is the neutron grenade patch and he is unable to replicate failures locally14:52
fungianteaya: perhaps. what is it a patch of/for?14:52
fungiyou anticipate my questions before i can ask them ;)14:52
anteayajlibosva: can we have the patch url please?14:53
anteayafungi: I try14:53
* jlibosva searching14:53
jlibosvahttps://review.openstack.org/#/c/58695/14:53
fungiflaper87: i've got the offending testset running in a vm here... i'll let you know what i get14:53
fungijlibosva: okay, so it looks like grenade is setting up an initial stable/havana devstack with neutron in preparation for your change, but devstack exercises cannot complete14:57
fungiis it possible devstack with neutron is broken in havana presently?14:57
jlibosvaidk, are there still running any tests with havana?14:58
fungisdague: ^ do you know about any possible havana devstack+neutron issues at the moment?14:58
*** yolanda has quit IRC15:00
fungispecifically, devstack exercises failing on [ERROR] /opt/stack/old/devstack/lib/neutron:824 [Fail] Couldn't ping server15:00
*** yolanda has joined #openstack-infra15:01
fungijlibosva: anteaya: i'm leaning toward it being bug 126118215:01
uvirtbotLaunchpad bug 1261182 in neutron "[ERROR] /opt/stack/old/devstack/lib/neutron:824 [Fail] Couldn't ping server" [Undecided,New] https://launchpad.net/bugs/126118215:01
*** dcramer_ has joined #openstack-infra15:02
jlibosvadamn, why am I spending so much time debugging when it's already reported? :D15:02
*** esker has joined #openstack-infra15:02
jlibosvaah, it's not debugged yet, so maybe it was worth it :)15:02
anteayafungi: you seem to have found the issue15:03
anteayajlibosva: you have exercises working locally, correct?15:03
jlibosvafungi: I tried that locally on my environment and I'm suspecting that ovs is not used correctly, I'd like to see whether it behaves the same on gate job15:03
jlibosvaanteaya: no, my floating_ip exercise fails too15:03
*** beagles_brb is now known as beagles15:04
*** flaper87 is now known as flaper87|afk15:04
anteayajlibosva: ah15:04
* fungi looks at the floating_ip results15:04
fungiyeah, same behavior though15:04
jlibosvaanteaya: but I'm not sure I understand that correctly. I see that interfaces in qdhcp and qrouter uses tag 1 , but instance uses tag 215:05
jlibosvathat could explain why dhcp packets cannot reach dhcp15:05
fungijlibosva: keep in mind that step #1 in grenade is to set up the previous stable release in devstack and make sure it's operable (so your patch isn't even applied at the point where that's failing)15:05
*** NikitaKonovalov has quit IRC15:06
fungithat's a safety valve to keep us from testing an upgrade from a broken stable state15:06
jlibosvafungi: I'm aware of that :) But I was asked to have exercises running before upgrade15:06
*** SergeyLukjanov has quit IRC15:06
fungiwhat i'm saying is, i don't see any way this bug is related to your patch. if devstack exercises at the start of grenade for a change to master are failing, that indicates a need to fix something in the stable/havana branch15:08
*** miqui has joined #openstack-infra15:08
*** rongze has quit IRC15:10
*** miqui has quit IRC15:10
*** flaper87|afk is now known as flaper8715:11
*** miqui has joined #openstack-infra15:12
*** sarob has joined #openstack-infra15:12
*** rcleere has joined #openstack-infra15:14
fungii'd suggest rechecking against 1261182 and see if grenade will get any farther on the next run so we can see if your patch is doing what you intend15:15
fungibut as far as broken devstack exercises, those are broken in stable/havana on that test run15:15
*** ruhe has joined #openstack-infra15:16
fungi(the *current* state of the stable/havana branch tip, unrelated to your change to master)15:16
*** sarob has quit IRC15:16
*** pblaho has quit IRC15:17
anteayafungi: thanks for the help15:18
fungiflaper87: on a debian/testing vm i had sitting around, your change fails under a python 2.7 tox environment for me at unit.reports.test_guru_meditation_report.TestGuruMeditationReport.test_bas15:18
fungiic_report15:18
anteayaI'll follow up with -qa and see if we can keep jlibosva moving forward15:18
fungiflaper87: i'll build an ubuntu vm and see what i get there15:18
*** Ryan_Lane has joined #openstack-infra15:19
flaper87fungi: mmh, weird. Pls, let me know! THanks for the support!15:20
*** talluri has joined #openstack-infra15:24
*** wenlock has joined #openstack-infra15:24
*** rnirmal has joined #openstack-infra15:24
*** herndon has joined #openstack-infra15:26
*** ruhe has quit IRC15:30
*** wenlock has quit IRC15:32
*** sdake_ has joined #openstack-infra15:33
*** talluri_ has joined #openstack-infra15:34
*** mrodden has joined #openstack-infra15:34
*** talluri has quit IRC15:35
*** xchu has quit IRC15:36
*** UtahDave has joined #openstack-infra15:36
*** SergeyLukjanov has joined #openstack-infra15:37
*** CaptTofu has joined #openstack-infra15:37
*** Ryan_Lane has quit IRC15:38
*** davidhadas has quit IRC15:39
*** senk has joined #openstack-infra15:40
*** vkozhukalov has quit IRC15:40
*** rongze has joined #openstack-infra15:40
*** agordeev has quit IRC15:44
*** senk has quit IRC15:45
*** jcooley_ has joined #openstack-infra15:48
*** dkehn has quit IRC15:48
*** david-lyle_ has joined #openstack-infra15:49
*** dkehn has joined #openstack-infra15:50
*** senk has joined #openstack-infra15:52
*** david-lyle_ has quit IRC15:53
fungiflaper87: i'm able to perfectly reproduce the error in your unit test log15:53
*** david-lyle_ has joined #openstack-infra15:53
flaper87fungi: mmh15:53
flaper87fungi: ubuntu? What python version?15:53
fungiflaper87: i built an ubuntu 12.04 vm in rackspace following the instructions at https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst#n100 and then installed the needed libmysqlclient-dev package15:54
*** jcooley_ has quit IRC15:54
*** johnthetubaguy1 has joined #openstack-infra15:55
fungiflaper87: then checked out oslo-incubator master branch from git, cherry-picked your change onto it, and ran tox -epy2715:55
flaper87fungi: is there a chance I can access that box you created?15:55
flaper87It'd make it easier to figure out what's happening and why15:55
flaper87I can't replicated it in my f19 box15:55
flaper87:/15:55
*** johnthetubaguy has quit IRC15:55
*** rongze has quit IRC15:55
fungiflaper87: sure, is your ssh public key on your lp account profile?15:55
flaper87fungi: yes sir15:56
fungiflaper87: you should be able to ssh into jenkins@162.209.101.7 or jenkins@2001:4802:7800:1:3bc3:d7f6:ff20:2f46 with that key15:57
fungithe oslo-incubator checkout i was testing with is in the jenkins homedir15:57
flaper87fungi: I'm in, thanks a lot!15:57
fungilet me know when you're done so i can discard that vm15:57
flaper87fungi: I'll let you know when I'm done with it15:57
flaper87hehe15:57
fungiperfect15:58
*** mrodden has quit IRC16:01
russellbwere is the code for status.openstack.org/zuul/ ?16:03
russellbwhere16:03
russellbi'm interested in trying to add a compact display option ...16:03
openstackgerritJakub Libosvar proposed a change to openstack-infra/config: Adding non-voting neutron exercises gate job  https://review.openstack.org/6292416:04
fungirussellb: i would love that--i've wanted one as well and just haven't had the time (something to collapse/expand one or all changes to/from just the heading bars?) http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul16:05
fungithe status.* files are the ones you want, i think16:05
sdagueyeh status.html is the main page16:05
russellbfungi: yeah that's what i was thinking16:05
sdagueall the real work is in https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/status.js16:06
russellbi have 0 web skill ... sounds like a good way to learn something, heh16:06
russellbthanks guys16:06
fungithank YOU!16:06
russellbthank me if i do it16:06
fungiheh16:06
russellbbut i plan to try16:06
sdaguerussellb: basically I think yuo want to write an alternate format_change16:07
*** ryanpetrello has joined #openstack-infra16:07
sdagueand do a conditional around that16:07
*** Abhishek has quit IRC16:07
sdagueif you can slice it at that level, it won't be too bad, if yuo need to go above that level, things will get more complicated16:07
*** jlibosva has quit IRC16:08
UtahDavefungi: Hey, I've got an example of using the Salt reactor at  https://etherpad.openstack.org/p/Salt-Event-System16:09
fungiUtahDave: great! taking a look now16:10
UtahDaveLet me know when you have a minute to go over it16:10
jeblairrussellb: oh hey16:10
russellbohai16:10
jeblairrussellb: i actually designed the current layout to work with the change box collapsed16:10
jeblairrussellb: i just didn't get around to that...16:10
russellbnice16:10
*** blamar has quit IRC16:10
*** SushilKM__ has joined #openstack-infra16:10
russellbyeah, i just want a way to have it collapsed by default, and have a little thingy i can click to expand for changes i care about16:10
jeblairrussellb: so if you make it so that the part below the box header isn't drawn16:10
russellbor something like that16:10
jeblairrussellb: it should still work -- the subway map and everything16:10
russellbwoot16:11
jeblairrussellb: sounds perfect!16:11
fungiUtahDave: i'll spend a few minutes digesting it and then ask the myriad of questions i'm still likely to harbor16:11
russellbthink it could be collapsed by default?  or would people be annoyed16:11
UtahDaveSounds good.  I'm going to add some notes as well.16:11
russellbso it doesn't have to be a toggle somewhere16:11
russellbmeh, i'll figure something out16:11
*** sarob has joined #openstack-infra16:12
jeblairrussellb: i think ability to toggle individual and toggle all would be nice, and collapsed by default is good16:12
russellback16:12
jeblairrussellb: (i think infra/qa people are likely the folks who want to see whole system status, so we should support that but it doesn't need to be default)16:13
hasharjeblair: hey, I am looking at the Zuul templates changes you made recently :)16:13
jeblairhashar: hey!16:13
*** yamahata has joined #openstack-infra16:13
*** dcramer_ has quit IRC16:13
fungijeblair: russellb: the other usability thing i've wanted is a highlighter similar to the project filter but extended to also match changes/gitshas and make matching items visually distinct (rather than hiding the others) so you can find/follow them when it's busy16:15
jeblairfungi: yeah, that'd be nice16:15
*** jcooley_ has joined #openstack-infra16:15
fungimaybe bolding the heading text or changing the fill tint16:15
*** tma996 has quit IRC16:16
russellbyet another thing i'd like is a 2nd line in the header bar with the change summary line :)16:17
russellbas that's how i generally remember patches instead of review numbers16:17
*** sarob has quit IRC16:17
russellboh the possibilities!16:17
jeblairrussellb: how human of you.  upgrade your cyborg implant!16:18
fungiinferior thinkmeat16:18
*** ruhe has joined #openstack-infra16:19
*** Abhishek has joined #openstack-infra16:20
jeblairthey're made out of meat.16:21
fungifrightening16:21
*** thomasbiege1 has joined #openstack-infra16:23
*** markmcclain has quit IRC16:25
*** ruhe has quit IRC16:27
*** thomasbiege has quit IRC16:27
*** dcramer_ has joined #openstack-infra16:27
*** SushilKM__ has quit IRC16:27
*** yassine has quit IRC16:28
*** yassine has joined #openstack-infra16:29
hasharjeblair: left you a long meaningless comment on the Zuul template change https://review.openstack.org/#/c/62498/16:30
hashargetting fresh air be back in a few :-]16:31
jeblairhashar: cool, thanks.  i didn't think of that situation.16:31
ArxCruzjeblair: fungi do you know why I'm getting this error? ERROR zuul.Gearman: Job <gear.Job 0x7fec98209810 handle: None name: build:gate-heat-python26 unique: 4525449a714049d0b2c1054397621852> is not registered with Gearman16:32
jeblairArxCruz: are you using jenkins with the gearman plugin?16:33
ArxCruzjeblair: yes16:33
*** yaguang has quit IRC16:33
jeblairArxCruz: that means it hasn't registered that job (or 'project' in jenkins-speak); either because the job doesn't exist in jenkins or some other reason (possibly a bug)16:34
sdaguejeblair: one other thing that occurred to me this morning, in dealing with a gate reset. We are currently putting grizzly changes into the same gate as master changes, however there is actually no link between those (no overlapping grenade job). How hard would it be to have zuul understand that?16:34
jeblairArxCruz: do any jobs work or is it just that one?16:34
ArxCruzright now the gate-heat-python27 start works16:34
jeblairArxCruz: oh, another possibility is that there is no jenkins node currently capable of running it -- if that's the case, gearman-plugin won't register it16:34
*** mrodden has joined #openstack-infra16:34
fungisdague: we're putting grizzly changes in the same gate as havana changes and havana changes in the same gate as master changes16:34
ArxCruzjeblair: oh, you mean, the number of executors is full ?16:35
fungisdague: also, grizzly branches of server projects interact with master branches of client libraries16:35
*** SushilKM__ has joined #openstack-infra16:35
sdaguefungi: if any of those are in the gate16:35
*** dkranz has quit IRC16:35
jeblairArxCruz: no, i mean that there are no executors; for instance, we run 2.6 jobs on centos nodes, so there have to be centos nodes available for the job to be registered initially16:35
fungisdague: true16:35
sdaguethe instance I had this morning was a bunch of tempest master changes, and a heat grizzly one was in the queue16:36
sdaguewhich blew up on a race16:36
ArxCruzjeblair: fair enough, I have only one node added now :)16:36
sdagueI didn't think I had havana tempest in there.... but maybe I did16:36
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Turn on devstack exercises for neutron (non-voting)  https://review.openstack.org/6293016:36
jeblairArxCruz: one node should be enough for it to register; you might try taking the node offline and putting it back online again.  sometimes we see gearman-plugin not register.  :(16:36
*** pcrews has joined #openstack-infra16:37
jeblairArxCruz: you can telnet to gearman on port 4730 and type 'status' to see the list of registered jobs16:37
fungisdague: i think shared queue definition would have to become dynamic if it were to depend on which changes were in the queue at some point in time, and then how do you merge two separate queues when a change which needs to be in both of them enters the gate16:37
sdagueI guess if we just had master tempest and grizzly heat, would they form 2 queues?16:37
hasharjeblair: merging jobs trees is a bit tricky but I am sure python has a solution build-in :-]16:37
jeblairhashar: i'm still thinking about it, but i'm not sure that one way or the other is correct...16:37
sdaguefungi: right, so in that case it would have to wait to enter until the queues ahead resolved16:38
sdagueyeh, it gets complicated16:38
sdagueok, just thinking aloud16:38
*** dstanek has quit IRC16:38
*** yassine has quit IRC16:38
fungisdague: i think it's probably solvable, but at the cost of a lot of complexity (bringing risk of new hard-to-diagnose bugs) and time probably better spent elsewhere16:39
sdagueyep, definitely16:39
sdaguethanks for explaining the complexity of it16:39
sdaguehadn't thought it all through16:39
ArxCruzjeblair: cool, thanks16:39
*** yassine has joined #openstack-infra16:39
jeblairhashar: i'm thinking maybe the default should be what's in there now (don't collapse), and maybe add an option to the template "collapse-parents: True" to toggle the behavior you described16:40
jeblairhashar: and i'm thinking maybe not writing that right now, but documenting the current behavior and leaving a TODO note.  :)16:40
hasharjeblair: lazy guy :-D :-D16:40
*** sdake has quit IRC16:41
hasharjeblair: that would work.  The code is fine anyway, I did test it out with my layout and it works.  Not sure what happens when a job is referenced twice though, I guess it would be triggered two times.16:41
*** krotscheck_nexus has joined #openstack-infra16:41
*** sdake has joined #openstack-infra16:41
hasharjeblair:  I might give a shot at merging the job trees16:41
jeblairhashar: yeah, i recently made it so that you could list a job twice and it would run twice (we wanted that as a tool to deal with nondeterministic bugs), which is why i think both behaviors are desirable16:42
hasharjeblair: nasty but valid use case :-]16:43
hasharso yeah merging should be done if and only if it got requested16:43
jeblairhashar: that's what i'm thinking; not merging seems to match the current behavior so should probably be the default16:43
*** romcheg1 has joined #openstack-infra16:44
jeblairhashar: cool, i'll go make doc/TODO updates16:44
romcheg1Hi folks16:44
*** romcheg1 is now known as romcheg16:44
hasharjeblair: will not be connected tonight, so will only review that tomorrow.  But if doc/TODO is around I guess it is good for me16:45
hasharjeblair: the following change (provide 'name' to templates) is fine to me though haven't tested it16:45
hasharI +1ed it already16:45
*** krotscheck_nexus has quit IRC16:45
*** krotscheck_nexus has joined #openstack-infra16:45
romchegAs I see some Neutron jobs define DEVSTACK_GATE_SMOKE_SERIAL=1 flag because there was a bug that failed to run Neutron tests in parallel.16:46
jeblairhashar: cool, i think it would only be a problem if you used 'name' as a substitution variable (and used it to mean something other than what zuul is going to set it to)16:46
romchegI found that the bug is in Fix released state16:46
romchegCan I add my new neutron job without setting that flag?16:47
jeblairromcheg: nice.  i'm not fully familiar with this; you might want to check with sdague to see if he knows of any other reasons it should still be set...16:48
romchegjeblair: thanks.16:48
romchegsdague: Are you around?16:48
jeblairromcheg: also, you can propose an experimental change to devstack-gate that just ignores that flag; that will run all of the jobs and you can see how well it works16:49
*** senk has quit IRC16:49
jeblairromcheg: that's a quick way to check what will happen without having to define a new job16:49
romchegjeblair: That's what I did. Just want to get more details16:49
*** krotscheck_nexus has quit IRC16:50
*** dkranz has joined #openstack-infra16:50
hasharjeblair: I use "prefix" for the Zuul job-template :-]16:53
*** yassine has quit IRC16:53
*** avishayb has quit IRC16:55
*** rongze has joined #openstack-infra16:55
*** smarcet has quit IRC16:57
openstackgerritRoman Prykhodchenko proposed a change to openstack-infra/config: Adds devstack-gate tests for Ironic  https://review.openstack.org/5391716:57
openstackgerritRoman Prykhodchenko proposed a change to openstack-infra/config: Add devstack-gate jobs for Ironic  https://review.openstack.org/5391717:00
*** jgriffith has quit IRC17:02
*** dstanek has joined #openstack-infra17:04
*** krotscheck has joined #openstack-infra17:04
*** davidhadas has joined #openstack-infra17:06
*** jcoufal has quit IRC17:06
*** sarob has joined #openstack-infra17:07
*** rwsu has joined #openstack-infra17:08
*** mancdaz is now known as mancdaz_away17:08
*** dstanek has quit IRC17:09
fungiUtahDave: i've added some comments to the end of that etherpad i'd like to get your feedback on when you have time17:12
UtahDavefungi: sure!17:13
*** agordeev has joined #openstack-infra17:17
*** blamar has joined #openstack-infra17:18
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Turn on devstack exercises for neutron (non-voting)  https://review.openstack.org/6293017:19
*** SushilKM__ has quit IRC17:20
*** SushilKM has joined #openstack-infra17:20
devanandaclarkb: ping17:22
*** fbo is now known as fbo_away17:27
clarkbdevananda: pong17:30
*** dkranz has quit IRC17:30
zarogood morning17:31
*** senk1 has joined #openstack-infra17:31
devanandaclarkb: following up on a chat romcheg and I were just having -- the plan for ironic/tempest stuff. I thought there was concensus, but now i'm not sure what order we should be adding the tests where...17:32
*** ruhe has joined #openstack-infra17:33
fungizaro: since we said we'd approve 61542 if we didn't get mordred's feedback by last night, i went ahead and did17:33
clarkbdevananda: I think you start with devstack-gate job where ironic gets installed and configured. This requires the creation of "baremetal" VMs and the enrollment of them into ironic17:33
clarkbdevananda: then you can have tempest tests do whatever with those resources17:33
*** vkozhukalov has joined #openstack-infra17:34
fungizaro: i want to try manually rolling back review-dev to our 2.4.4 fork briefly just to see if it works (and to test a link regex change on it real quick), if that's cool with you17:35
openstackgerritA change was merged to openstack-infra/config: give zaro access to review-dev server  https://review.openstack.org/6154217:35
romchegclarkb: I think the concern it about the pipeline17:35
zarofungi: no problem, take your time.17:35
clarkbromcheg: devananda: what is the concern?17:35
devanandaclarkb: two different types of testing -- i'm referring to API testing by tempest, romcheg's patches that have been up for a few weeks17:36
clarkbdevananda: I don't think they are any different they both require ironic to be running right?17:36
devanandaclarkb: not the "devstack-gate spins up a cloud using ironic" tests17:36
romchegIn this patch I add these baremetal jobs to the experimental branch https://review.openstack.org/#/c/53917/17:36
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Allow layered templates  https://review.openstack.org/6249817:36
devanandaclarkb: they are very different :-/17:36
clarkbdevananda: but not from an infrastructure perspective17:36
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Automatically provide 'name' to project templates  https://review.openstack.org/6250017:36
devanandaclarkb: yes, from an infra perspective too17:37
clarkbdevananda: you need a cloud with ironic running to do either tests17:37
devanandaclarkb: but in the currently-proposed tests, we don't need any VMs17:37
devanandaclarkb: it's just API exercises against a fake driver17:37
*** DennyZhang has joined #openstack-infra17:38
clarkbright but that fake driver still needs ironic to be running and to be configured...17:38
devanandaexercise our client, services, db, rpc, etc -- but not the drivers17:38
devanandaright17:38
devanandaand devstack can start ironic17:38
*** jcooley_ has quit IRC17:38
*** senk has joined #openstack-infra17:38
clarkbenrollment of "baremetal" VMs seems to be the one difference17:38
devanandathe other stuff - -actually bringing up nodes with cirros on them, using ironic, will require changes to devstack-gate, a separate d-g pool, etc.17:38
devanandaclarkb: precisely17:39
romchegdevananda, clarkb: All the changes to devstack and devstack-gate are merged already17:39
clarkbI don't think that matters much from infra perspective, that would just be a step in the test that either runs or doesn't17:39
devanandaromcheg: no they're not :)17:39
romchegdevananda: yes, they are :)17:39
*** sarob has quit IRC17:39
romchegdevananda: the one that's left is to infra config17:39
devanandaromcheg: the devstack-gate changes I'm referrign to (for ironic to actually start 100's of nodes with the pxe_ssh driver) aren't written yet. unless they are and I dont know about it17:40
clarkbso anyways going back to where this discussion started, what is the concern?17:40
devanandathere are two TOTALLY DIFFERENT sets of tests taht I want: 1) fake driver 2) real driver.17:40
romchegdevananda: Ah, I was talking about the API tests17:40
clarkbdevananda: right17:40
*** senk1 has quit IRC17:41
devanandaall the things for testing ironic with FAKE driver are proposed, but it's been weeks and they haven't merged17:41
devanandaand there's been lots of back-and-forth about what test pipeline they should be in17:41
clarkbwhere is the back and forth?17:41
devanandahttps://review.openstack.org/#/c/48109/8 and https://review.openstack.org/#/c/53917/1417:43
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Turn on devstack exercises for neutron (non-voting)  https://review.openstack.org/6293017:43
*** MarkAtwood has joined #openstack-infra17:43
clarkbdevananda: I don't think there was any back and forth in the infra change17:43
devanandaand on this thread  http://lists.openstack.org/pipermail/openstack-dev/2013-December/021948.html17:43
clarkbwe just wanted the jobs to go in with the new templating17:43
*** jooools has quit IRC17:44
jeblairSergeyLukjanov: it looks like the alphabetized job isn't quite working: http://logs.openstack.org/17/53917/17/check/check-projects-yaml-alphabetized/71b08e3/console.html17:44
*** dkranz has joined #openstack-infra17:44
clarkbdevananda: romcheg: can you guys tl;dr for me because I think I have read this stuff completely differently17:44
*** mkerrin has quit IRC17:44
*** chandankumar has quit IRC17:45
clarkbdevananda: romcheg: basically tempest wants to merge code that is actually tested upstream (which we are doing), and jobs can start in the experimental queue17:45
romchegdevananda clarkb: the patch to tempest is actually waiting for the one to infra to test changes17:45
*** dolphm has quit IRC17:45
clarkbwhen the jobs are in the experimental queue you can `check experimental` on the tempest change and once it works you can switch to the check/gate queues17:45
romchegclarkb: I think devananda wants to be sure that the jobs will to to gate and check pipelines after they are checked in the experimental one17:46
flaper87fungi: fixed, thanks! :) https://review.openstack.org/#/c/32544/17:53
*** flaper87 is now known as flaper87|afk17:56
*** agordeev has quit IRC17:56
*** amotoki has quit IRC17:56
*** jpich has quit IRC17:56
*** ArxCruz has quit IRC17:56
*** oubiwann has quit IRC17:56
*** thomasem has quit IRC17:56
*** lcheng has quit IRC17:56
*** lcestari has quit IRC17:56
*** yamahata__ has quit IRC17:56
*** romcheg has quit IRC17:56
*** derekh has quit IRC17:56
*** katyafervent has quit IRC17:56
*** ianw has quit IRC17:56
*** Hefeweizen has quit IRC17:56
*** talluri_ has quit IRC17:56
*** simonmcc has quit IRC17:56
*** openstackgerrit has quit IRC17:56
*** med_ has quit IRC17:56
*** samalba has quit IRC17:56
*** jeblair has quit IRC17:56
*** jcooley has quit IRC17:56
*** dragondm has quit IRC17:56
*** DennyZhang has quit IRC17:56
*** Abhishek has quit IRC17:56
*** CaptTofu has quit IRC17:56
*** sdake_ has quit IRC17:56
*** mattymo has quit IRC17:56
*** bknudson has quit IRC17:56
*** briancline has quit IRC17:56
*** mgagne has quit IRC17:56
*** vishy has quit IRC17:56
*** zigo has quit IRC17:56
*** Adri2000 has quit IRC17:56
*** cyeoh has quit IRC17:56
*** persia has quit IRC17:56
*** morganfainberg has quit IRC17:56
*** vkozhukalov has quit IRC17:56
*** pcrews has quit IRC17:56
*** johnthetubaguy1 has quit IRC17:56
*** spredzy has quit IRC17:56
*** branen has quit IRC17:56
*** vipul has quit IRC17:56
*** HenryG has quit IRC17:56
*** akscram has quit IRC17:56
*** ilyashakhat has quit IRC17:56
*** krtaylor has quit IRC17:56
*** afazekas_ has quit IRC17:56
*** dhellmann has quit IRC17:56
*** rockstar has quit IRC17:56
*** portante has quit IRC17:56
*** jpeeler has quit IRC17:56
*** thingee has quit IRC17:56
*** basic` has quit IRC17:56
*** pleia2 has quit IRC17:56
*** SpamapS has quit IRC17:56
*** odyi has quit IRC17:56
*** hub_cap has quit IRC17:56
*** koolhead17 has quit IRC17:56
*** hemanth_ has quit IRC17:56
*** kiall has quit IRC17:56
*** BobBall has quit IRC17:56
*** dkranz has quit IRC17:56
*** yamahata has quit IRC17:56
*** ryanpetrello has quit IRC17:56
*** david-lyle_ has quit IRC17:56
*** SergeyLukjanov has quit IRC17:56
*** herndon has quit IRC17:56
*** rnirmal has quit IRC17:56
*** dizquierdo has quit IRC17:56
*** rahmu has quit IRC17:56
*** metabro_ has quit IRC17:56
*** chmouel has quit IRC17:56
*** Daviey has quit IRC17:56
*** wayneeseguin has quit IRC17:56
*** SlickNik has quit IRC17:56
*** Mithrandir has quit IRC17:56
*** antonym has quit IRC17:56
*** sdague has quit IRC17:56
*** EntropyWorks has quit IRC17:56
*** mika has quit IRC17:56
*** rainya has quit IRC17:56
*** esmute has quit IRC17:56
*** adam_g has quit IRC17:56
*** stevebaker has quit IRC17:56
*** ttx has quit IRC17:56
*** kpepple has quit IRC17:56
*** markmc has quit IRC17:56
*** LinuxJedi has quit IRC17:56
*** afazekas has quit IRC17:56
*** niska has quit IRC17:57
*** danger_f` has quit IRC17:57
*** jog0 has quit IRC17:57
*** harlowja_away has quit IRC17:57
*** lifeless has quit IRC17:57
*** dstufft has quit IRC17:57
*** kashyap has quit IRC17:57
*** tian has quit IRC17:57
*** miqui has quit IRC17:57
*** jswarren has quit IRC17:57
*** jerryz has quit IRC17:57
*** ywu has quit IRC17:57
*** mfer has quit IRC17:57
*** mriedem has quit IRC17:57
*** rfolco has quit IRC17:57
*** salv-orlando has quit IRC17:57
*** hashar has quit IRC17:57
*** boris-42 has quit IRC17:57
*** shardy has quit IRC17:57
*** sirushti has quit IRC17:57
*** primeministerp has quit IRC17:57
*** devananda has quit IRC17:57
*** gema has quit IRC17:57
*** krotscheck has quit IRC17:57
*** rongze has quit IRC17:57
*** dkehn has quit IRC17:57
*** UtahDave has quit IRC17:57
*** dims_ has quit IRC17:57
*** julim has quit IRC17:57
*** che-arne has quit IRC17:57
*** StevenK has quit IRC17:57
*** pabelanger has quit IRC17:57
*** jd__ has quit IRC17:57
*** ekarlso has quit IRC17:57
*** Grishkin has quit IRC17:57
*** milki has quit IRC17:57
*** MarkAtwood has quit IRC17:57
*** rwsu has quit IRC17:57
*** sdake has quit IRC17:57
*** dcramer_ has quit IRC17:57
*** rcleere has quit IRC17:57
*** esker has quit IRC17:57
*** zul has quit IRC17:57
*** weshay has quit IRC17:57
*** freyes has quit IRC17:57
*** mugsie has quit IRC17:57
*** gaelL has quit IRC17:57
*** juice has quit IRC17:57
*** jheiss has quit IRC17:57
*** vladan has quit IRC17:57
*** uvirtbot has quit IRC17:57
*** jesusaurus has quit IRC17:57
*** bogdando has quit IRC17:57
*** rpodolyaka has quit IRC17:57
*** senk has quit IRC17:57
*** ruhe has quit IRC17:57
*** mrodden has quit IRC17:57
*** dprince has quit IRC17:57
*** crank has quit IRC17:57
*** Shrews has quit IRC17:57
*** jaypipes has quit IRC17:57
*** mordred has quit IRC17:57
*** dansmith has quit IRC17:57
*** spiffxp_ has quit IRC17:57
*** Loquacities has quit IRC17:57
*** pvo has quit IRC17:57
*** Guest33069 has quit IRC17:57
*** AaronGr_Zzz has quit IRC17:57
*** sileht has quit IRC17:57
*** mancdaz_away has quit IRC17:57
*** jlk has quit IRC17:57
*** EmilienM has quit IRC17:57
*** russellb has quit IRC17:57
*** yongli has quit IRC17:57
*** torgomatic has quit IRC17:57
*** davidlenwell has quit IRC17:57
*** iccha-dinner has quit IRC17:57
*** hughsaunders has quit IRC17:57
*** maurosr has quit IRC17:57
*** saper has quit IRC17:57
*** olaph has quit IRC17:57
*** zaro has quit IRC17:57
*** mindjiver has quit IRC17:57
*** david-lyle has quit IRC17:57
*** alaski has quit IRC17:57
*** _cyril_ has quit IRC17:57
*** guitarzan has quit IRC17:57
*** GheRivero has quit IRC17:57
*** hdd has quit IRC17:57
*** clarkb has quit IRC17:57
*** Ng has quit IRC17:57
*** dachary has quit IRC17:57
*** mikal has quit IRC17:57
*** nikhil__ has quit IRC17:57
*** westmau5 has quit IRC17:57
*** jeremyb has quit IRC17:57
*** greghaynes has quit IRC17:57
*** Alex_Gaynor has quit IRC17:57
*** fbo_away has quit IRC17:57
*** notmyname has quit IRC17:57
*** gfa has quit IRC17:57
*** ewindisch has quit IRC17:57
*** ogelbukh has quit IRC17:57
*** adarazs has quit IRC17:57
*** annegentle_ has quit IRC17:57
*** enikanorov has quit IRC17:57
*** mkoderer_ has quit IRC17:57
*** belliott has quit IRC17:57
*** nijaba has quit IRC17:57
*** SushilKM has quit IRC17:57
*** eharney has quit IRC17:57
*** sandywalsh_ has quit IRC17:57
*** slong has quit IRC17:57
*** nicedice has quit IRC17:57
*** bnemec has quit IRC17:57
*** ChanServ has quit IRC17:57
*** blamar has quit IRC17:57
*** davidhadas has quit IRC17:57
*** thomasbiege1 has quit IRC17:57
*** yolanda has quit IRC17:57
*** dkliban has quit IRC17:57
*** Hunner has quit IRC17:57
*** andreaf has quit IRC17:57
*** adalbas has quit IRC17:57
*** luisg has quit IRC17:57
*** mtreinish has quit IRC17:57
*** michchap has quit IRC17:57
*** fungi has quit IRC17:57
*** sbadia has quit IRC17:57
*** anteaya has quit IRC17:57
*** openstack has joined #openstack-infra18:02
*** openstack has joined #openstack-infra18:06
ruhekrotscheck: jfyi there is a #storyboard channel18:06
hub_capfungi: fair enough18:06
*** herndon has quit IRC18:07
krotscheckruhe: Huhn- I thought that'd be called #openstack-storybaord18:07
SergeyLukjanovfungi, anteaya, thanks for catching it, I'll fix it when I'll be home ;)18:07
fungiSergeyLukjanov: sounds like anteaya may beat you to it anyway18:08
*** dcramer_ has joined #openstack-infra18:08
fungikrotscheck: i think the goal is to make storyboard generally reusable beyond the openstack community, thus it's not an openstack-specific project18:08
* anteaya is checking to see if she is working on the correct file18:08
anteayaSergeyLukjanov: config/infra.yaml?18:09
fungianteaya: the script being run in that log18:09
*** krotscheck has left #openstack-infra18:09
*** krotscheck has joined #openstack-infra18:09
krotscheckfungi: So there's a naming convention there, got it.18:09
*** harlowja_away is now known as harlowja18:09
*** dstanek has quit IRC18:10
anteayafungi: ah18:10
* anteaya looks again18:10
clarkbdevananda: romcheg: I am guessing that clears up the confusion? I hope to catch up on reviews today between running errands. will rereview the infra change18:10
*** herndon has joined #openstack-infra18:11
fungianteaya: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/check_projects_yaml_alphabetized.sh#n1218:11
romchegclarkb: For testing api is clear.18:11
anteayafungi: thanks, just found it18:11
fungianteaya: make it $OLDPWD/modules/openstack_project/templates/review.projects.yaml.erb on that line18:12
anteayawill do18:12
romchegclarkb: Maybe you can help me with that DEVSTACK_GATE_SMOKE_SERIAL flag?18:12
*** gyee has joined #openstack-infra18:12
*** psedlak has joined #openstack-infra18:13
*** mestery has joined #openstack-infra18:13
*** saschpe has joined #openstack-infra18:13
*** beagles has joined #openstack-infra18:13
*** dripton_ has joined #openstack-infra18:13
*** locke105 has joined #openstack-infra18:13
*** changbl has joined #openstack-infra18:13
*** flaper87|afk has joined #openstack-infra18:13
*** NobodyCam has joined #openstack-infra18:13
*** openstackstatus has joined #openstack-infra18:13
*** soren has joined #openstack-infra18:13
*** dtroyer_zz has joined #openstack-infra18:13
romchegclarkb: https://review.openstack.org/#/c/53917/16/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml18:13
romchegclarkb: the bug seems to be fixed but several neutron jobs still set the flag.18:13
*** SushilKM has quit IRC18:13
clarkbromcheg: I don't think neutron has fixed it. If that have they have not update their jobs to actually gate on it...18:15
*** Alex_Gaynor has quit IRC18:15
openstackgerritA change was merged to openstack-infra/config: Add status.openstack.org vm  https://review.openstack.org/6155218:15
clarkbromcheg: the jobs with neutron-isolated-parallel in their name do run in parallel18:15
clarkbyou can check if those are passing18:15
*** BobBall is now known as BobBallAway18:15
*** Abhishek has quit IRC18:15
lifelesssdague: o/18:16
clarkbromcheg: they are in the experimental queue for tempest and neutron iirc, so you check check experimental on your tempest api test change to see if they pass18:17
*** Abhishek has joined #openstack-infra18:17
*** Alex_Gaynor_ has joined #openstack-infra18:17
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1262154  https://review.openstack.org/6294818:18
uvirtbotLaunchpad bug 1262154 in nova "HTTP 500 on update_server_metadata (DB deadlock)" [Undecided,Confirmed] https://launchpad.net/bugs/126215418:18
anteayafungi: I didn't get my prior patch up before changing to a new branch for the script change, now I have them all folded into one commit18:18
anteayafungi: looking at git reset, should I be looking elsewhere?18:18
fungianteaya: might be easiest to just undo your trivial edit to that one file and git commit -a --amend again, then switch to a new branch and reapply that edit18:19
anteayaI'll do that18:19
*** jcooley_ has joined #openstack-infra18:20
*** jcooley_ has quit IRC18:21
fungiif it were tougher to undo, i'd suggest you look at resetting to a sha from your reflog18:21
anteayafungi: yeah, let's wait on that dance18:22
jeblairfungi, clarkb: i'm stopping puppet on the logstash workers and client and applying changes18:22
*** nati_ueno has joined #openstack-infra18:22
fungijeblair: sounds good18:22
*** jgriffith has joined #openstack-infra18:23
openstackgerritA change was merged to openstack-infra/config: Refactor logstash_worker into log_processor module  https://review.openstack.org/6278718:27
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Fix the yaml-alphabetized job  https://review.openstack.org/6295118:27
openstackgerritA change was merged to openstack-infra/config: Increase to 16 logstash worker nodes  https://review.openstack.org/6278818:29
jlkmordred: hey there, got another favor to ask wrt rackspace-auth-openstack. Could you ask github to make the rackerlabs/rackspace-auth-openstack fork the new root of "rackspace-auth-openstack" ?18:30
jlkmordred: since you own the current root github says you gotta ask for it.18:30
*** johnthetubaguy1 has quit IRC18:30
reedfungi, jeblair: https://review.openstack.org/#/c/61250/ time to merge it?18:32
fungiclearly freenode is under orbital bombardment again... i keep getting timeouts to my current irc server18:33
*** dkranz has quit IRC18:35
*** dstufft has quit IRC18:35
*** dstufft has joined #openstack-infra18:36
*** Alex_Gaynor_ is now known as Alex_Gaynor18:37
*** Alex_Gaynor has quit IRC18:37
*** Alex_Gaynor has joined #openstack-infra18:37
*** jpich has quit IRC18:38
*** dstufft has joined #openstack-infra18:38
*** ruhe has quit IRC18:40
* sdague catching up on scrollback18:42
sdagueromcheg: as far as I know neutron still won't work parallel18:43
*** blamar has quit IRC18:43
sdagueif someone think that's fixed, we should revisit18:43
anteayasdague: as far as I know, not working parallel yet18:44
anteayasalv-orlando is working on it18:44
*** markmcclain has joined #openstack-infra18:45
romchegsdague: Thank you18:45
anteayahttps://blueprints.launchpad.net/neutron/+spec/neutron-tempest-parallel18:45
salv-orlandocurrent trunk code has perhaps at beast a 15% success rate on parallel job. We have already patches under review which *hopefully* should bring that to 80% at least.18:46
jeblairfungi, clarkb, mordred: you okay with canceling the next two infra meetings?  they are on christmas eve day and new years eve day respectively.18:47
clarkbjeblair: wfm18:47
clarkbI will probably be around if we decide there is areason to have those meetings18:47
fungijeblair: i'll chair those, in an overstuffed arm chair, with my special eggnog which is made by pouring burbon in a glass18:48
*** markmc has quit IRC18:49
fungibut yeah, cancelling is fine with me18:49
jeblairfungi: that's not how you normally chair them?18:49
*** dkranz has joined #openstack-infra18:49
*** salv-orlando_ has joined #openstack-infra18:49
fungigood point18:49
*** salv-orlando has quit IRC18:50
clarkbfungi: you need a monocle as well18:50
*** salv-orlando_ is now known as salv-orlando18:50
fungiand a calabash18:50
anteayasalv-orlando: thanks for the update18:50
*** SushilKM has joined #openstack-infra18:50
* anteaya is looking forward to the next two infra meetings18:51
jeblairlaunching worker01-08 nodes (4g performance)18:51
fungireed: i think https://review.openstack.org/61250 is ready to merge, just want to get clarkb or jeblair to skim the final patchset in case they have any other comments18:51
* fungi would rather avoid self-approving a new infra project addition, mostly on principle18:51
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Added .gitignore for IntelliJ specific configuration files  https://review.openstack.org/6295618:52
*** kgriffs has joined #openstack-infra18:52
jeblairfungi, reed: lgtm18:52
clarkbfungi: if you approve with a glass of bourbon while wearing a monocle in a large overstuffed chair I don't think we can complain18:52
clarkbreviewing as well18:53
fungiheh18:53
krotscheckWoo, first CR18:53
fungizaro: the review-dev rollback worked as far as i can tell. i imported the musqldump from the day before the upgrade while gerrit was stopped, then reverted 7011082 in a current puppet development environment and applied18:53
fungikrotscheck: welcome aboard officially now!18:54
krotscheckfungi: Thanks!18:54
krotscheckExcept that jenkings barfed.18:54
sdaguelifeless: what's up?18:54
lifelesssdague: you invoked my name18:54
pleia2anteaya: did you catch that the next two meetings are next year? (not doing xmas eve and nye)18:54
lifelesssdague: in the context of testr/nose18:54
fungikrotscheck: you will get used to that. jenkins hates us and our attempts to write clean code18:54
krotscheckfungi: ...yeah, in this case it really seems to hate gitignore.18:55
openstackgerritA change was merged to openstack-infra/config: Import the openstackid authentication project  https://review.openstack.org/6125018:55
fungikrotscheck: it looks to me like storyboard.stories.tests.SimpleTest.test_even_more_addtion may have bitrotted or something18:56
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Support launching rax performance nodes  https://review.openstack.org/6295818:56
jeblairfungi, clarkb: ^ just tested in production18:56
fungikrotscheck: or that job has been perpetually busted, or doesn't clean up properly before running or something18:56
fungijeblair: that's where i test all my code18:57
krotscheckfungi: Thanks for pointing me in the right direction, I'm still new to python :)18:57
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Support launching rax performance nodes  https://review.openstack.org/6295818:57
krotscheckfungi: Awwww yeah. Nothing like a helloworld test case breaking your jenkins.18:58
*** afazekas_ has quit IRC18:59
fungikrotscheck: given the project is still in its infancy, i wouldn't be surprised if the voting unit test job was added even though the unit tests didn't run18:59
anteayapleia2: my tongue in cheek response to saying that fungi chairing with eyepiece and beverage should be a good time18:59
anteayapleia2: and yes, got that they are cancelled19:00
*** dstanek has joined #openstack-infra19:00
sdaguelifeless: right, I want to remove nose support from tempest with a vengence19:00
lifelesswhat do you need from testr to do that?19:00
anteayapleia2: thanks for making sure I am up to date, now that I am rarely here19:00
sdaguebut I remember there being some weird condition on 2.6 + testr that is still not resolved19:00
anteaya:D19:00
clarkblifeless: sdague: the weird case is skip test exceptions19:00
fungianteaya: i can take the meeting as an opportunity for story time. maybe a dramatic rendition of 20-year-old bofh newsletters or something19:00
pleia2anteaya: oh ok :)19:00
clarkbnose adds them itself in 2.6 and making that work under 2.6 is weird apparently. also setup class but unittest2 can apparently handle that19:01
lifelessright, i spent some time with afazekas on that at the summit19:01
lifelessbugs everywhere19:01
krotscheckfungi: Makes sense. Also def. the right decision to make.19:01
sdagueok, any idea how close it got with afazekas ?19:01
*** jswarren has quit IRC19:01
anteayafungi krotscheck they were19:02
anteayafungi: I'm looking forward to storytime19:02
fungianteaya: krotscheck: so if there's a pending change in review to fix the unit tests, you can check that out and rebase your .gitignore change on top of it19:02
*** sarob has joined #openstack-infra19:02
fungiand then they should be tested together and pass19:03
anteayaturns out a django project must have at least 4 tests otherwise jenkins gets upset19:03
clarkbanteaya: we fixed that19:03
anteayaah, see so long ago19:03
clarkbanteaya: now you need 1. running 0 tests is a fail19:03
anteayaha ha ha19:04
fungiin the before-time, in the long-long-ago. like, days ago19:04
anteayawhere was that changed, in the jenkins job script?19:04
anteayafungi: yeah, that time frame19:04
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1262154  https://review.openstack.org/6294819:05
uvirtbotLaunchpad bug 1262154 in nova "HTTP 500 on update_server_metadata (DB deadlock)" [Medium,In progress] https://launchpad.net/bugs/126215419:05
fungianteaya: modules/jenkins/files/slave_scripts/run-unittests.sh 8c28621b (Clark Boylan  2013-12-06 12:57:07 -0800)19:06
fungiso very, very long ago19:06
krotscheckfungi: Looks like lifeless & nikita have an initial test set, I'll rebase on top of that.19:07
*** dstufft_ has joined #openstack-infra19:07
fungikrotscheck: if it's shown as passing the defined jenkins jobs, then that ought to be what you want19:07
*** Abhishek has quit IRC19:07
anteayaages19:08
*** dkranz has quit IRC19:08
fungigranted, things move so fast here that a week seems like a year sometimes19:08
*** jergerber has joined #openstack-infra19:08
sdaguenice... try to submit feedback on https://www.openstack.org/foundation/technical-committee/ with the form at the bottom of the page19:09
reedfungi, jeblair, woot! thanks!19:09
*** davidhadas has quit IRC19:09
sdagueit thinks it a cross site scripting attack19:09
*** dstufft has quit IRC19:10
openstackgerritClark Boylan proposed a change to openstack-infra/config: Publish elastic-recheck documentation.  https://review.openstack.org/6130019:10
clarkbsdague: jeblair ^ that should publish it to the desired location19:10
reedsdague, who thinks it is a xss?19:10
fungisdague: are you sure you're not trying to pwn us all? (but in seriousness we get on the order of a report a week for some vulnerability in that site, so tipit have been putting in a lot of paranoia checks to deal with those)19:11
sdaguereed the site I think19:12
*** dstufft_ has quit IRC19:12
sdagueoh, no, chrome19:12
sdague"Security token doesn't match, possible CSRF attack."19:12
fungiahh, neat19:12
sdaguehonestly, no idea really, but again with the broken.19:13
*** dstufft has joined #openstack-infra19:13
*** shivh has joined #openstack-infra19:13
fungisdague: you probably just want to file something at https://bugs.launchpad.net/openstack-org19:14
clarkbfungi: https://review.openstack.org/#/admin/projects/openstack-infra/openstackid,access looks like that still isn't working quite right. github was created and code exists there. the openstackid-core group was created but not the openstackid-ptl group which I am guessing caused the acl push to fail19:15
clarkbI have no idea why that is happening. If one group creation works the other should as well in theory. I mean the code itself isn't broken right?19:16
jeblairerr: Could not prefetch package provider 'pip':19:16
jeblairclarkb: ^ have you seen that puppet error ?19:16
fungiclarkb: yep, last time i got lucky and encountered it manually, the error was "ERROR:manage_projects:Unable to get UUID for group gantt-milestone."19:16
shivhclarkb: I am experimenting with tempest tests posting verify (third party testing) and have some problem with posting "verified=1", however i dont have a problem posting "verifyed=0". Anyone out there know why. THanks.19:17
clarkbfungi: right so I think it must be a race around creating the group and it being in the tables?19:17
fungiclarkb: then re-ran it and it worked fine19:17
fungiclarkb: that's my current suspicion19:17
clarkbjeblair: the pip provider is something that will be copied to the host before puppet runs the catalog19:17
jeblairshivh: you're probably not using an account registered with gerrit; see http://ci.openstack.org/third_party.html19:17
clarkbjeblair: I think it is saying that that initial copy failed for some reason. I don't recall running into that particular error but ^ is what I believe is happening19:17
jeblairshivh: rather, it's not registered as a third party test system19:17
jeblairclarkb: retry doesn't help19:18
clarkbfungi: maybe we put the busy wait around the db querying19:18
jeblairclarkb: ++ (busy wait)19:18
*** dstanek has quit IRC19:19
jeblairshivh: see http://ci.openstack.org/third_party.html#posting-result-to-gerrit and http://ci.openstack.org/third_party.html#requesting-a-service-account19:19
fungiclarkb: sounds like an excellent idea19:19
*** dkranz has joined #openstack-infra19:20
clarkbjeblair: do you have accesss to the host? is pip installed? I wonder if maybe the new pip stuff mordred did depends on having pip ahead of time?19:20
jeblairclarkb: http://paste.openstack.org/show/55365/19:20
jeblairhrm.19:20
clarkbjeblair: ok I bet it is related to broken pip install19:21
clarkbjeblair: that looks like classic case of need to uninstall pip and pkg-resources, reinstall pip then immediately update setuptools19:22
*** dstanek has joined #openstack-infra19:23
*** mrodden has quit IRC19:26
anteayaclarkb jeblair pleia2 krotscheck AaronGr_Zzz can I get some reviews? I'd like to get this in so I can get another patch in: https://review.openstack.org/#/c/62951/19:26
anteayaand zaro19:27
openstackgerritRussell Bryant proposed a change to openstack-infra/config: Make gate-solum-devstack-dsvm voting  https://review.openstack.org/6296419:27
jeblairanteaya: please be patient; a few hours is not a long time to wait for reviews; expect a few days.  and to continue working, make your new patchset depend on that one.19:27
anteayajeblair: it isn't, I can get the other in without this test passing, would just be nice if it was19:28
anteayasorry to bother you19:28
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Added .gitignore for IntelliJ specific configuration files  https://review.openstack.org/6295619:29
anteayafungi: so if the turning on of devstack exercises gets in, I can have jlibovar keep working on it tomorrow19:29
jeblairclarkb: do you know how the system got into this state?19:30
jeblairclarkb: during the initial run, it encountered this error: http://paste.openstack.org/show/55367/19:32
*** CaptTofu has quit IRC19:32
jeblairclarkb: it seems to be able to run pip at that time, and i don't see any pip-related actions after it19:32
*** CaptTofu has joined #openstack-infra19:33
clarkbjeblair: could it be a bad interaction between old pip and gear+pbr?19:34
*** mrodden has joined #openstack-infra19:34
clarkbjeblair: also note that that is /usr/lib/python*/pip not /usr/local/lib... I think puppet is upgrading pip from 1.0 to 1.4.1 with pip. That could also possibly be at fault19:35
*** dkranz has quit IRC19:36
jeblairclarkb: where does pip get upgraded?19:36
jeblairclarkb: indeed, there seems to be no /usr/bin/pip anymore, despite the debian package being installed.19:37
openstackgerritClark Boylan proposed a change to openstack-infra/jeepyb: Wait for DB to reflect group UUID updates  https://review.openstack.org/6296719:37
clarkbjeblair: in the pip module iirc19:37
jeblairclarkb: i'm not seeing it; it looks like it should just install python-pip from the os19:38
clarkbhmm nevermind that doesn't seem to be it19:38
clarkbya19:38
jeblairclarkb: which is what i see in the logs too19:38
fungii think the intent was for the distro package of pip to get uninstalled, then get-pip retrieves and installs it from github/pypi19:38
clarkbjeblair: pbr requires new pip perhaps it is doing it for us?19:38
*** herndon has left #openstack-infra19:38
jeblairfungi: i don't see any code to do that :/19:39
openstackgerritRussell Bryant proposed a change to openstack-infra/config: Make gate-solum-devstack-dsvm voting  https://review.openstack.org/6296419:39
fungioh, but mordred's pip provider refactor rotted on the vine i think, which is where that was happening19:39
jeblairclarkb: possibly so.19:39
fungiit was broken for python 3.3 pip i believe when i tested it, and i'm not sure it got a new patchset after that19:39
*** krotscheck has quit IRC19:40
jeblairso i believe we do not have the ability to launch a new server that runs a python package that uses pbr.19:41
jeblairwhich is quite a few of our servers.19:41
mordredanything I can look at to be helpful?19:42
openstackgerritA change was merged to openstack-infra/devstack-gate: Don't fix disks before rexecing  https://review.openstack.org/6096119:43
jeblairmordred: yes.  i have no idea how to fix that.  fungi says you were working on it.  perhaps you could indicate what is needed?19:43
fungimordred: i think https://review.openstack.org/51425 needs restoring and continuing19:43
mordredwell - it's a really hard problem which ultimately always makes us want to move forward with pabelanger's packaging work of our stuff19:43
jeblairmordred: a brand new spinup of a server gets this error: http://paste.openstack.org/show/55367/ and then pip is broken: http://paste.openstack.org/show/55365/19:44
jeblairmordred: i do seem to recall that being the case.  i didn't think we were quite so broken at the moment though.19:44
mordredjeblair: I didn't either19:44
*** dcramer_ has quit IRC19:45
mordredjeblair: it's fixable by hand - the above patch is the one that should fix it systemically19:45
mordredjeblair: I have an idea for a short-term fix19:46
jeblairmordred: it looks like that patch is not fit to land atm, so how do i fix it by hand?19:46
jeblairmordred: i'll need to do that on the 16 machines i'm spinning up19:46
mordredjeblair: why don't we add the get-pip invocation to launch-node.py19:46
mordredso that we pre-get pip installed properly on the machine before we try to run puppet19:46
mordredwhich will get you unstuck19:46
mordredit's not ideal, obviously19:47
jeblairmordred: i think it should be added to install_puppet.sh so it works for people who don't use launch-node19:48
mordredjeblair: tools/install_pip.sh in devstack describes how pip wants to get installed19:48
mordredjeblair: ++19:48
mordredjeblair: lemme make a patch real quick19:48
jd__mordred: can I stare at you and frown because you never answered https://review.openstack.org/#/c/58770/ ?19:48
mordredjd__: yeah, but lemme get this done for jeblair first19:49
jeblairmordred: i think puppet wants python-pip package installed, but tools/install_pip wants it removed; so maybe your patch should also remove that?19:49
mordredI haven't been very useful recently19:49
mordredjeblair: yes. I believe you are correct19:49
jd__mordred: sure! ;-)19:50
* jd__ stares at jeblair 19:50
jeblairmordred: https://etherpad.openstack.org/p/RHbv5NLdi019:51
*** dkranz has joined #openstack-infra19:51
shivhclarkb: I should be "registered with gerrit". I have been contributing code etc. looks like something subtle i don't understand.19:52
*** jcooley_ has joined #openstack-infra19:53
anteayajd__: don't do that, you will get a slap on the wrist19:53
anteayagive him his work space19:53
jd__I can stare from a very far place to avoid that :-)19:54
anteayaha ha ha19:55
anteayajim has long arms19:55
*** davidhadas has joined #openstack-infra19:56
fungishivh: the point was that to leave a verify vote you need to request a specific gerrit service account as described in the links you were provided earlier. it gets added to a special group which has verify voting ability19:56
fungishivh: normal gerrit user accounts are not allowed to add verify votes19:56
fungi(in openstack's configuration at any rate)19:57
shivhthanks. got it.19:57
*** dizquierdo has quit IRC19:57
shivhfungi, clarkb, jeblair: Thanks.19:57
*** jcooley_ has quit IRC19:57
*** dcramer_ has joined #openstack-infra19:58
fungiflaper87|afk: looks like you logged off that test host. are you finished investigating the failure there? if so i'll discard it20:01
jeblairmordred: etherpad now works; the python-pip package seems to be mostly irrelevant20:02
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Support launching rax performance nodes  https://review.openstack.org/6295820:03
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Install pip using get-pip in launch node  https://review.openstack.org/6297320:04
mordredjeblair: ^^20:04
mordredjeblair: the larger patch I thnkn still wants to be fixed, btw20:04
mordredand we shoudl talk about python2 and python3 colocated on the same host with the patch20:05
jeblaircool.  i thought that wasn't going to work?20:05
mordredit won't20:05
*** shivh has quit IRC20:05
fungii'm rather amused that this mechanism for installing pip requires access to bitbucket, github and pypi20:06
jeblairfungi: what could possibly go wrong?20:06
mordredbut I think there was concern with how python3-pip installed a pip3 thing20:06
*** david-lyle_ has quit IRC20:06
*** dkranz has quit IRC20:07
fungimordred: particularly, that the module search path for colocated python2.7 and python3.3 interpreters on debian-derived distros is different, and calling the pip3 vs pip entrypoint is how pip knows where to install packages so that they'll be found by the right interpreter20:08
fungii wanted to test out the rewritten provider to see whether it would "just work" anyway, but currently puppet doesn't like it (error messages linked from my last comment in the old review)20:09
*** davidhadas has quit IRC20:10
*** sandywalsh_ has quit IRC20:12
fungibut the crux of the issue is that if you python2.7 -c 'import sys;print(sys.path)' and then do the same for 3.3 and compare them, you'll see that the only entry they share is the current working directory20:12
fungiall others are qualified by the interpreter version20:12
fungi(and none seem to be symlinked to one another that i could find)20:13
*** derekh has joined #openstack-infra20:13
* clarkb is going to afk for lunch and errands now. back later20:14
mordredfungi: oh! I think I know how to fix it20:19
fungiawesome!20:19
*** dkranz has joined #openstack-infra20:21
openstackgerritA change was merged to openstack-infra/config: Support launching rax performance nodes  https://review.openstack.org/6295820:21
*** prad has joined #openstack-infra20:22
*** praneshp has joined #openstack-infra20:22
mordredfungi: if we make pip.rb call "python -m pip.main" instead of "pip" - and the other one "python3 -m pip.main"20:22
mordredthen we're set20:22
*** elasticio has joined #openstack-infra20:23
*** sandywalsh_ has joined #openstack-infra20:24
jeblairclarkb, fungi, mordred: logstash worker01 A and B are running now20:24
mordredjeblair: woot20:25
jeblairone oversight: they both report the same worker name to gearman; but that's not that important and we can fix later.20:26
fungimordred: sounds reasonable20:27
*** vkozhukalov has quit IRC20:28
*** yolanda has quit IRC20:33
*** _david_ has joined #openstack-infra20:35
openstackgerritJames E. Blair proposed a change to openstack-infra/devstack-gate: Remove strange duplication of setup_host and output  https://review.openstack.org/6298320:36
openstackgerritDarragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Use yaml local tags to support including files  https://review.openstack.org/4878320:36
openstackgerritA change was merged to openstack-infra/config: Install pip using get-pip in launch node  https://review.openstack.org/6297320:36
openstackgerritRoman Prykhodchenko proposed a change to openstack-infra/config: Add devstack-gate jobs for Ironic  https://review.openstack.org/5391720:37
*** sdake_ is now known as sdake220:38
*** sarob has quit IRC20:40
*** sarob has joined #openstack-infra20:40
jeblairclarkb, mordred, fungi: all of the old log workers are stopped, and the first 8 of the new hosts (16 workers) are running20:41
*** AaronGr_Zzz is now known as AaronGr20:41
*** hashar has joined #openstack-infra20:42
AaronGrback now, hi folks.20:42
fungijeblair: i guess we didn't add them to cacti yet20:42
fungiAaronGr: welcome back!20:42
fungijeblair: i'll get the cacti addition whipped up20:43
mordredhey it's AaronGr20:43
jeblairfungi: cool, thanks20:44
AaronGrmordred: just made it home, going to get email and vpn working, then get back to gerrit and ldap.20:44
*** derekh has quit IRC20:45
fungiAaronGr: protip, you don't need that hp vpn or e-mail. i haven't looked at mine in months20:45
fungioh, wait20:45
AaronGrfungi: having a few conversations at the moment where that's the endpoint.. and all the boxes i need to hit today/tomorrow are behind a vpn20:46
fungiAaronGr: don't listen to me--i haven't worked for hp since last spring ;)20:46
*** dcramer_ has quit IRC20:47
AaronGrfungi: no worries20:47
*** smarcet has left #openstack-infra20:50
*** dprince has quit IRC20:50
*** senk has quit IRC20:53
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Further setup of openstack-doc-tools  https://review.openstack.org/6298420:54
*** mrodden has quit IRC20:54
*** mrodden has joined #openstack-infra20:56
*** marun has joined #openstack-infra20:56
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Add performance flavor logstash workers to cacti  https://review.openstack.org/6298520:56
fungithis way ^ we can trend how loaded they get and see how many we really need20:57
*** sarob has quit IRC20:58
*** hogepodge has joined #openstack-infra21:00
*** krotscheck has joined #openstack-infra21:02
*** sarob has joined #openstack-infra21:02
*** krotscheck has quit IRC21:08
openstackgerritA change was merged to openstack-infra/config: Add performance flavor logstash workers to cacti  https://review.openstack.org/6298521:11
*** mancdaz_away is now known as mancdaz21:12
*** SergeyLukjanov has quit IRC21:12
*** miqui has quit IRC21:14
*** fbo_away is now known as fbo21:21
gyeesdague, I am trying to chase down a grenade upgrade failure for https://review.openstack.org/#/c/56333/21:23
gyeethe log seem to suggest that keystone-paste.ini is not upgraded because I don't see the new filter being called21:24
fungigyee: grenade tests new openstack with old configuration, so it won't be21:24
gyeeah21:24
fungigyee: we want to make sure that deployers can upgrade from havana to icehouse and still have a working openstack *before* they need to go making configuration adjustments21:25
fungigyee: that makes things like rolling upgrades more reasonable to accomplish in production21:25
gyeefungi, so its running IceHouse with Havana config?21:25
sdagueyes21:25
fungigyee: yes21:25
gyeek, let me make the adjustment then21:26
gyeefungi, sdague, thanks21:26
fungineeds to degrade gracefully using reasonable built-in defaults, basically21:26
sdagueif we really can't get around that we can flag it as UpgradeImpact and provide a work around, but we are really trying to basically make the work arounds noop21:27
*** dcramer_ has joined #openstack-infra21:27
gyeebtw, I did tried to run grenade on my local VM, not sure if I was following the steps correctly21:27
gyeeI basically checkout devstack and grenade, ran setup-grenade localhost21:28
gyeethan modify grenaderc to use my review branch21:28
gyeethen grenade.sh21:28
gyeebut it doesn't seem to do any IceHouse upgrade at all21:28
jeblairfungi: the new logstash workers seem to be, uh, stashing logs.  so i'm going to delete the old ones21:30
jeblairthen add the next 821:30
morganfainbergclarkb, fwiw, i think i'm going to wokr on httpd keystone and ceiliometer during the holidays here21:30
fungijeblair: sounds great21:30
morganfainbergclarkb, figured you'd want to know.21:30
morganfainbergclarkb, in devstack that is21:30
fungigyee: i've only ever run grenade from the openstack-infra/devstack-gate scripts, so i can't provide much of a suggestion beyond do what d-g does to run grenade'21:31
gyeefungi, good to know! let me see if I can duplicate that in my local VM21:33
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Remove old logstash worker nodes  https://review.openstack.org/6299221:34
jeblairclarkb, mordred, fungi: the launch of worker09 completed without error.  thanks.21:36
fungiyay!21:36
*** hashar has quit IRC21:36
jeblairand fast too21:36
fungilooks like we're starting to trend snmp oids from the first half of the new workers21:40
fungithe slivers i see look mostly nominal21:40
openstackgerritJames E. Blair proposed a change to openstack-infra/devstack-gate: Remove strange duplication of setup_host and output  https://review.openstack.org/6298321:41
*** jhesketh_ has joined #openstack-infra21:42
*** tjones has joined #openstack-infra21:45
*** tjones has left #openstack-infra21:45
*** eharney has quit IRC21:46
jeblairfungi: i just set the default cacti tree to alpha sorting21:47
*** eharney has joined #openstack-infra21:48
fungiyet another thing i kept meaning to look into but never found the time21:48
fungiit was tolerable at 10 servers. i've had to resort to keyword searching for months21:48
*** senk has joined #openstack-infra21:49
reedfungi, now that the git repo from tipit is in our repo, too, what's next step?21:50
*** senk1 has joined #openstack-infra21:51
jeblairreed: first, don't commit any more changes to the tipit repo and start using gerrit21:51
reedjeblair, makes sense...21:52
reedso they need to pull from git.openstack.org, right?21:52
*** senk1 has quit IRC21:53
jeblairreed, fungi: i imagine two things, possibly in parallel; if the code in the repo is installable, add that to the puppet module for the server21:53
jeblairreed, fungi: and spin up the server21:53
jog0jeblair: https://review.openstack.org/#/c/62784/21:53
*** senk has quit IRC21:53
*** senk has joined #openstack-infra21:54
fungijeblair: well, the server's already there and deploying that project, but from github at the moment21:54
fungireed: yes, we need to update the puppet configuration vcsrepo object for that project to switch to using a git.o.o origin21:54
reedfungi, jeblair, ok, I'm shooting an email to all of you now21:54
jeblairreed: oh, then what i've said is wrong.  :)21:54
*** dcramer_ has quit IRC21:56
fungireed: and from what smarcet was saying earlier, it sounds like he needs to let us know where his application would normally consume database connection details/credentials from (probably a config file?) so that we can template that and inject the appropriate bits into it from hiera21:56
jeblairjog0: yeah, i want to kill the whole process group on timeout; not sure how to do that yet21:56
reedfungi, clear21:56
openstackgerritJoe Gordon proposed a change to openstack-infra/devstack-gate: Preserve testr temp files  https://review.openstack.org/6278621:57
fungireed: basically the database for openstackid-dev.o.o already exists in a rackspace trove instance, but his app needs to be given the info for how to reach it21:57
fungireed: another to do item is for us to add some additional puppetry to set up https on that machine (i can put a self-signed cert and key for that into hiera as well)21:58
reedfungi, great, so you need to know which file to edit? or you need to tell smarcet how to configure his app21:58
jog0jeblair: cool, just wanted to make sure it didn't get forgotton about21:59
fungireed: mainly just want to know where the configuration would go and in what format (an example config file and path for that would suffice)21:59
*** elasticio has quit IRC22:00
reedfungi, for development not to happen on github anymore, any documentation for them to read?22:00
fungireed: well, it'll be the same thing which happened for the openstack-infra/groups project. that's one tipit's working on as well right (just different devs there)?22:01
*** MarkAtwood has quit IRC22:02
fungithe development process will be somewhat familiar to smarcet already at this point since he used the same workflow to propose patches to openstack-infra/config, but now he also gets the ability to +2 and approve changes for openstack-infra/openstackid22:02
clarkbmorganfainberg: cool (though working over the holidays is not required :) )22:03
reedfungi, no, groups is done by marton kiss, he's not affiliated with tipit22:03
clarkbjeblair: \o/22:03
reedfungi, does he need to change his remote git repo?22:03
jog0sdague: reviewing your e-r patches now22:04
clarkbjeblair: on 09 can we enable crm and see how it handles the extra load?22:04
jeblairclarkb: sure22:04
*** fbo is now known as fbo_away22:05
*** vickleford has joined #openstack-infra22:05
fungireed: yeah, he'll want to change his git remotes from github.com to git.openstack.org22:06
fungireed: and more importantly, to openstack-infra/openstackid22:07
fungias the remote repository name22:07
*** jcooley_ has joined #openstack-infra22:07
*** agordeev has quit IRC22:07
fungireed: something along the lines of sed -i 's,github.com/blah/idproject,git.openstack.org/openstack-infra/openstackid' .git/config22:08
fungier, missed a trailing , there but you get the idea22:08
*** dcramer_ has joined #openstack-infra22:09
*** rnirmal has quit IRC22:09
fungireed: i didn't realize marton was working independently on that. very cool22:10
openstackgerritA change was merged to openstack-infra/config: Remove old logstash worker nodes  https://review.openstack.org/6299222:10
openstackgerritA change was merged to openstack-infra/elastic-recheck: allow facets to work at different resolutions  https://review.openstack.org/6265222:11
*** MarkAtwood has joined #openstack-infra22:13
*** lcheng has quit IRC22:16
*** mfer has quit IRC22:19
*** dkranz has quit IRC22:19
*** elasticio has joined #openstack-infra22:20
*** dizquierdo has joined #openstack-infra22:21
*** kgriffs is now known as kgriffs_afk22:23
*** mancdaz is now known as mancdaz_away22:23
zaroclarkb: question about copy console log.22:25
clarkbzaro: go for it22:25
zaroclarkb: what is the significance of the wait here?  https://github.com/jenkinsci/scp-plugin/blob/master/src/main/java/be/certipost/hudson/plugin/SCPRepositoryPublisher.java#L37822:25
zaroclarkb: i mean why 500? because that's the problem.22:25
clarkbzaro: it was just a non zero value that was picked22:26
clarkbwhy is 500 a problem?22:26
zaroclarkb: seems like that's too long now.22:26
clarkbzaro: too long? if it was too long wouldn't the entire log file be copied?22:26
clarkbzaro: basically what is happening there is we are yielding to other threads so that the job can finish and have a complete console log22:27
clarkbotherwise we don't copy the entire log file (which is what is happening here so the waiting probably isn't working properly) because the job and IO are not complete yet22:28
zaroclarkb: i thought getLogText only gets pieces that's why you need to check with isComplete?22:29
jd__mordred: did you just disappear on me after that Python 3 question?22:30
clarkbzaro: it only gets pieces because the log isn't complete usually22:30
*** thomasem has quit IRC22:30
zaroclarkb: actually i haven't figured out the root cause, but let me try some more things.22:31
zaroclarkb: just letting you know that dialing that down will get the entire log.22:32
clarkbzaro: what are you dialing it down to?22:32
clarkbzaro: I wonder if that thread is being killed for some reason when it waits for half a second?22:32
zaroclarkb: worked when dialed down to 50.22:33
clarkbzaro: how large is your console log? I wonder if that still works when the console log is >6MB22:33
zaroclarkb: that's what i'm guessing as well.22:33
jeblairclarkb, fungi: worker16 is running crm114 (and has puppet stopped)22:34
zaroclarkb: not that large.22:34
fungijeblair: glued to the graph ;)22:34
*** dstanek has quit IRC22:34
zaroclarkb: only about 50K22:35
fungiclarkb: zaro: that was the suggestion jeblair floated... the job ends and the console log collector thread is terminated before it's done retrieving all the data22:35
jeblairwe've just jumped from 8 to 32 workers, and only 2 of them are running crm114, so i expect a pretty mininmal load (in fact it's mostly idle right now; it can barely pick up a job due to all the competition)22:35
clarkbquick run more jobs >_>22:36
clarkbfungi: zaro: jeblair: that seems likely. Mean of jenkins to do that, but likely22:36
jeblairclarkb, fungi: but we can start changing the config on the other servers and ramp it up.22:36
fungii suppose i could stick a broken python package on the pypi mirror and watch all the approved patches start piling up in the gate ;)22:36
*** iccha-dinner is now known as iccha22:36
* fungi would not *actually* do that of course22:37
jeblairfungi: i think we should send a mail to the -dev mailing list asking people to approve patches without testing and keep bashing recheck until changes land.  ;)22:37
fungibwahaha22:37
clarkbI wonder if that is because the thread object lives in perform()'s scopr22:37
clarkbzaro: ^22:37
*** yamahata has quit IRC22:39
openstackgerritJames E. Blair proposed a change to openstack-infra/devstack-gate: Remove strange duplication of setup_host and output  https://review.openstack.org/6298322:41
sdaguejog0: can you look again there on responses - https://review.openstack.org/#/c/62694/22:42
sdagueI'll fix the docs, but I want to get a feel on the functionality changes to work on22:42
jeblairjog0: I'm stumped; i've produced the whole stack of shell scripts and tox (including pretty_tox.sh) in a test environment, and timeout does the right thing with process groups22:42
jeblairjog0: my guess is that something about running devstack itself changes that22:43
jeblairfungi: do you have any ideas?  http://logs.openstack.org/84/62784/3/check/check-dg-tempest-dsvm-full/abe3d68/console.html22:44
jeblairfungi: note at the bottom after /usr/bin/timeout times out, tempest keeps running22:44
jeblaircompare to http://logs.openstack.org/81/62781/1/check/check-dg-tempest-dsvm-full/4227047/console.html where putting the timeout on tox kills it much more cleanly22:45
clarkbwow you even gave it -922:45
*** senk has quit IRC22:46
jeblairjog0: at any rate, we could just go ahand and live with a few extra tests running after the timeout.  i mean, there aren't likely to be many -- after all the job is timing out.22:46
jog0sdague: responded22:46
zaroclarkb: so you think the thread for perform is terminated taking down the thread to copy console?22:46
sdaguejog0: only on the top one :)22:47
clarkbzaro: maybe, but a quick read on java threads seems to indicate losing the reference to a thread object doesn't make it go away necessarily. Trying to sort out if that makes the thread GC able22:47
fungijeblair: yeah, i see that... um... maybe the file descriptor is attached but the child is disassociated from the controlling process and in a different process group?22:48
*** sarob has quit IRC22:48
clarkbzaro: ya doesn't look like java Threads are GCable22:49
jeblairfungi: yeah, i made a test env with a wrap script calling a gate script calling sudo calling tox calling pretty_tox.sh calling testr, and everything worked..22:49
jog0jeblair: I think hte current solution although imperfect is  a great starting point and worth going forward with22:49
jog0sdague: just did bottom22:49
jeblairfungi: which is why i was looking at devstack suspiciously22:50
openstackgerritMonty Taylor proposed a change to openstack-infra/storyboard: Add SQLalchemy database model  https://review.openstack.org/6223922:50
fungijeblair: if we can identify the process group to target, kill -9 -- -<pgroup>22:50
clarkbif at first you don't succeed, clone the jenkins source code :)22:50
*** dstanek has joined #openstack-infra22:51
*** senk has joined #openstack-infra22:52
sdaguejog0: ok, so I fundamentally disagree about the job list. I think it's important to keep the unit test and docs fails in the total fail count, because they are often good indicators of infra fails.22:53
sdagueAnd while we don't bot them, the queries can match on them22:53
*** rongze has joined #openstack-infra22:53
sdagueso we can bundle them after the fact, like we did with Bug 126065422:54
uvirtbotLaunchpad bug 1260654 in openstack-ci "Could not initialize class jenkins.model.Jenkins$MasterComputer" [Critical,Fix released] https://launchpad.net/bugs/126065422:54
*** gothicmindfood has joined #openstack-infra22:55
jog0sdague: fair enough, but what about in a seperate category or something22:55
jog0sdague: true22:56
sdagueif it is causing a gate reset, we really shouldn't ignore it22:56
openstackgerritJames E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2  https://review.openstack.org/6278422:57
jeblairfungi: ^ that's an attempt to see what's going on.  and yeah, maybe we can have the top of the gate script write out its pgrp and then have the wrap kill it after the timeout22:57
*** dims_ has quit IRC22:57
jog0I agree don't ignore, but I think some sort of differentiation might be worth it22:58
jog0but we can revisit that point later22:58
jog0not worth holding up for that22:58
*** rongze has quit IRC22:59
*** jhesketh_ has quit IRC22:59
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: add per job percentages for responsibility of fails  https://review.openstack.org/6273823:01
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: add support for classification rate  https://review.openstack.org/6269423:01
*** _david_ has quit IRC23:01
sdagueok that has the doc fixes and the collection iterator, and now I have to figure out how I broke the histogram for the graphs23:01
clarkbzaro: AbstractBuild has a cleanUp() method that may be causing the problem23:01
clarkbzaro: though that appears to release the workspace lease, clear the env variables, and trigger downstream builds. Nothing that should be killing that thread23:04
*** vickleford has quit IRC23:06
*** SushilKM has quit IRC23:09
clarkbzaro: a quick scan of git log doesn't show anything promising. We may need to attach a debugger23:10
*** dims_ has joined #openstack-infra23:13
*** gothicmindfood has quit IRC23:13
clarkbjeblair: are we ready to try your swap fix for rax performance nodes yet?23:14
*** senk has quit IRC23:14
jeblairclarkb: if i can ever find it again; it should get resurrected and rechecked23:15
jeblairi really hate auto-abandon23:15
*** MarkAtwood has quit IRC23:17
*** esker has quit IRC23:17
*** sarob has joined #openstack-infra23:18
mordredjeblair: yeah. I'm liking it less myself as well23:18
*** mriedem has quit IRC23:19
*** senk1 has joined #openstack-infra23:19
*** gyee has quit IRC23:20
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: correct issue of dangling µs in buckets  https://review.openstack.org/6301523:22
sdaguejog0: so that one should be looked at quickly23:23
sdagueit will fix the graphs when it lands23:23
*** sarob has quit IRC23:23
*** dstanek has quit IRC23:25
sdaguefungi: thanks23:27
fungisdague: non-ascii in commit subject... uncleeeeean! (seriously though, you're going to wear out your altgr key with all that mooing)23:28
openstackgerritClark Boylan proposed a change to openstack-infra/devstack-gate: Use the ephemeral disk on rackspace performance nodes  https://review.openstack.org/6034723:28
clarkbjeblair: ^ rebased for you23:28
*** mrodden has quit IRC23:28
sdaguefungi: we're in a utf8 world man :)23:30
*** weshay has quit IRC23:30
sdagueI also thought it would be a nice test of our pipeline23:30
clarkbfungi: sdague: my weecaht in screen can render it properly now so I don't care :)23:30
sdagueyep23:30
sdagueit's been fine in everything I've looked at it in23:31
fungisdague: clarkb: looks right to me in gerrit and gerritbot/irc23:31
fungiwe win at international characterset encoding!23:31
sdagueyou'll appreciate the utf8 char I used for utf8 testing image names23:31
clarkbsdague: screen requires special bit flipping or options to do unicode23:31
*** MarkAtwood has joined #openstack-infra23:31
fungisdague: snowman? get with the times... there's a beer mug glyph with an official codepoint23:32
fungijeblair: timeout: failed to run command `/opt/stack/devstack-vm-gate.sh': No such file or directory23:32
sdaguehttps://review.openstack.org/#/c/61747/9/tempest/api/compute/images/test_images_oneserver.py23:32
sdagueMasayuki's comment is funny too23:32
*** UtahDave has quit IRC23:32
sdague"I'm not sure this unicode character is "valid" for the "Enterprise" though :)."23:33
fungihahaha23:33
fungii think i have actually seen that one before as well23:33
*** dcramer_ has quit IRC23:39
marunI know I've asked before, but how do I go about adding a new gate job (functional) for neutron?  It will need to run with sudo privileges (iirc running as the tempest user should work).  It will also need things like openvswitch installed (hopefully the devstack vm prep will do that?).23:43
marunsdague: ^23:43
openstackgerritA change was merged to openstack-infra/elastic-recheck: correct issue of dangling µs in buckets  https://review.openstack.org/6301523:44
clarkbmarun: give me a minute I will link you to some commit shas that have done it23:44
marunclarkb: danke23:45
clarkbhrm all of them are old enough that maybe better to just describe the current process23:46
*** rcleere has quit IRC23:46
*** romcheg has quit IRC23:46
anteayamarun: this adds an existing job, devstack exercises: https://review.openstack.org/#/c/62930/23:46
clarkbeg not sufficient to give you one diff, need many23:46
clarkboh anteaya has one perfect23:47
clarkbmarun: see that change23:47
anteayait doesn't write the job though23:47
anteayajust uses an existing one23:47
clarkbright but the glue is there23:47
clarkbmarun: if you look at the other jobs in those files some of them set export hook functions which run at different times in the devstack-gate process23:48
clarkbcurrently you can do post devstack pre test, hook to run tests, and hook after tests iirc23:48
*** jergerber has quit IRC23:49
marunclarkb: hmm. this test won't require services to target, just that some of the neutron os dependencies like ovs have been installed23:50
clarkbmarun: devstack vm prep is basically run devstack23:51
clarkbmarun: your other option is to have a thing do the prep itself23:51
marunclarkb: I was under the impression that installing prereqs and then snapshotting the resulting vm was a part of the jenkins pipeline.23:52
marunclarkb: or do I just have an active imagination?23:52
fungiheading out to dinner, back on later23:53
clarkbmarun: close. we cache the prereqs so that devsatck can run as quickly as possible. but installation is done by devstack23:54
marunclarkb: ah, gotcha23:54
marunclarkb: so the installation has to happen as part of the job.  I guess I'll have to see if I can install things without doing a full devstack run.23:55
clarkbcorrect, installation would need to be part of the job23:55
clarkbmarun: fwiw the pbr integration test which all it does is install openstack stuff with pbr in a variety of ways uses devstack to make sure all of the dependencies are in place23:56
*** dcramer_ has joined #openstack-infra23:56
marunclarkb: ok, cool.23:56
sdaguemarun: so if you only care about neutron in this job, you should be able to set the service list pretty small in the devstack config yuo'd use for it23:58
sdagueso you'd use devstack to install, but only keystone + neutron23:59
*** prad has quit IRC23:59
sdaguethe devstack install time is about 2 minutes for all of it, so you could probably get to < 60s overhead to do that23:59
marunsdague: I don't actually want anything running, though23:59
openstackgerritRussell Bryant proposed a change to openstack-infra/config: zuul status: allow changes to be collapsed  https://review.openstack.org/6301823:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!