| *** yamamoto has quit IRC | 00:00 | |
| tsg | clarkb: looks like there have been a ton of changes to openstack CI (review, zuul) in the last week or so .. is there a summary somewhere? | 00:00 |
|---|---|---|
| *** yamamoto has joined #openstack-infra | 00:00 | |
| *** evgenyf has joined #openstack-infra | 00:01 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Subscription https://review.openstack.org/104004 | 00:02 |
| clarkb | tsg: the git logs? | 00:02 |
| clarkb | tsg: we do also capture things in each meeting that are large | 00:02 |
| tsg | clarkb: that's a good suggestion - let me look at the meeting logs | 00:02 |
| *** yamamoto_ has joined #openstack-infra | 00:02 | |
| *** rhsu has joined #openstack-infra | 00:03 | |
| jeblair | jhesketh: right, i think we've been proceeding under the assumption that a single pipeline would only ever be interested in one gerrit per source, trigger, or reporter | 00:03 |
| jeblair | jhesketh: it sounded like you wanted to change that when you said "I don't know if we'll ever need to have other gerrit sections and thus it may get confusing" | 00:03 |
| jhesketh | which is perhaps fair enough | 00:04 |
| *** yamamoto_ has quit IRC | 00:04 | |
| *** yamamot__ has joined #openstack-infra | 00:04 | |
| jhesketh | oh, that was more around not being a fan of the naming of sections | 00:04 |
| *** mkerrin has quit IRC | 00:04 | |
| jeblair | jhesketh: in zuul.ini or layout.yaml? | 00:04 |
| *** stevemar has joined #openstack-infra | 00:04 | |
| jhesketh | I can't think of a good reason why, but we may want another section to configure gerrit options across all gerrits for example | 00:04 |
| jhesketh | in zuul.ini | 00:04 |
| tsg | clarkb: also a ton of Swift gate jobs were consistently failing in gate-swift-python27 for reasons outside what was changed in the patch .. the interesting thing those jobs started to pass this afternoon after about 15+ "recheck no bug"s so far :) | 00:04 |
| *** yamamoto has quit IRC | 00:05 | |
| jhesketh | jeblair: So if we don't think about specific sources/triggers/etc and we think abstractly about them, I think we want one source per pipeline and any combination of the others | 00:05 |
| tsg | clarkb: are the gate-* jobs run in parallel in the same instance? | 00:05 |
| *** svarnau__ has quit IRC | 00:05 | |
| *** yamamoto has joined #openstack-infra | 00:06 | |
| *** shayneburgess has quit IRC | 00:06 | |
| jeblair | jhesketh: but only one of each type per pipeline, yeah? so only one gerrit, one timer, etc.. | 00:06 |
| clarkb | tsg: they are not. each test runs on its own dedicated single use slave | 00:06 |
| tsg | clarkb: FYI, the test that fails every now and then is "test.unit.common.middleware.test_xprofile.TestProfileMiddleware" | 00:07 |
| clarkb | tsg have example failures? | 00:07 |
| tsg | which we cannot reproduce locally in SAIO | 00:07 |
| clarkb | links to logs | 00:07 |
| tsg | clarkb: checking | 00:07 |
| tsg | http://logs.openstack.org/44/103644/10/check/gate-swift-python27/4c3ed6a/console.html | 00:07 |
| jhesketh | jeblair: Possibly... but unless you're thinking specifically about gerrit it shouldn't actually matter | 00:07 |
| clarkb | tsg the way to reproduce is `tox -epy27` | 00:07 |
| *** yamamoto_ has joined #openstack-infra | 00:07 | |
| clarkb | saio is different | 00:07 |
| tsg | clarkb: can't repro it with tox either :( | 00:08 |
| tsg | the failure is really intermittent | 00:08 |
| tsg | a number of times the check jobs passed but then the same py27 job failed at gate | 00:08 |
| fungi | tsg: the tests within the jobs may (and usually are) run in parallel on one instance, though from one *job* to the next they're on separate instances | 00:09 |
| *** yamamot__ has quit IRC | 00:09 | |
| openstackgerrit | Aishwarya Thangappa proposed a change to openstack-infra/storyboard: Added a subscription_events table & subscription hander https://review.openstack.org/113016 | 00:09 |
| clarkb | swift is still nose and serial tests so not parallel test related | 00:09 |
| jeblair | jhesketh: i think all three of those syntaxes accomodate that | 00:09 |
| fungi | but yeah, not for swift | 00:09 |
| *** yamamot__ has joined #openstack-infra | 00:09 | |
| tsg | clarkb: okay | 00:09 |
| tsg | clarkb: any ideas looking at the log? | 00:10 |
| jeblair | jhesketh: i suppose for option 3 you could have a '[gerrit]' section too | 00:10 |
| *** yamamoto has quit IRC | 00:10 | |
| clarkb | tsg: last time swift ran into something like this it was a timing issue | 00:11 |
| jhesketh | jeblair: so I don't think it's an issue, I just prefer option 2's syntax... If we think of them like connectors (for lack of a better name) then as a user I know I need to set up everything I want to connect to... when I want to connect to gerrit, I specify that as the driver | 00:11 |
| jhesketh | it's probably just a preference thing, both are just as compatible | 00:11 |
| *** yamamoto has joined #openstack-infra | 00:11 | |
| *** yamamot__ has quit IRC | 00:11 | |
| jeblair | jhesketh: so what's the dislike for option 1? | 00:11 |
| tsg | clarkb: what could cause the timing issues, system load? | 00:11 |
| *** shashankhegde has joined #openstack-infra | 00:11 | |
| clarkb | tsg: all kinds of things | 00:11 |
| clarkb | tsg: sleep() can return early | 00:11 |
| clarkb | for example | 00:12 |
| jhesketh | jeblair: show me how you'd implement a timer trigger for option 1 | 00:12 |
| jhesketh | is that a special singleton? | 00:12 |
| tsg | clarkb: :) thus trying to understand the CI setup a little better | 00:12 |
| *** yamamoto_ has quit IRC | 00:12 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Exposed subscription_events in API https://review.openstack.org/114399 | 00:12 |
| fungi | but also, we run in cloud virtual machines with no guarantee of resource continuity, and we do so on multiple providers and boot from potentially slightly different base images in each of them and... | 00:12 |
| *** yamamoto_ has joined #openstack-infra | 00:13 | |
| fungi | there is limitless variability | 00:13 |
| jeblair | jhesketh: same as in option 2 | 00:14 |
| clarkb | looks like that test uses a fake wsgiapp so you would hope it wasn't time dependent | 00:14 |
| clarkb | and the old style assert equal is hard to read :( | 00:14 |
| *** wenlock has quit IRC | 00:14 | |
| jeblair | jhesketh: it's special either way. in option 1, you don't need to add a connector entry to zuul.ini. in option 2 you don't need to add a connector entry to zuul.ini, and you also don't need to add a connector entry to layout.yaml. | 00:14 |
| *** yamamot__ has joined #openstack-infra | 00:15 | |
| *** yamamoto_ has quit IRC | 00:15 | |
| jhesketh | jeblair: it's special in option 1, but not in option 2... much like we document different parameters to current triggers/reporters etc we'd just document that the gerrit trigger takes the connector attribute, and the timer does not | 00:15 |
| *** mmaglana has quit IRC | 00:15 | |
| jeblair | jhesketh: i'm not buying that. the timer does not need a connector, gerrit does. that's true regardless (and it's even true now) | 00:16 |
| clarkb | tsg: it may help to clean up that assert | 00:16 |
| *** yamamoto_ has joined #openstack-infra | 00:16 | |
| *** yamamot__ has quit IRC | 00:16 | |
| clarkb | tsg: it is asserting two things are euqal then giving a different value back as the message | 00:17 |
| clarkb | tsg: which is confusing | 00:17 |
| clarkb | anddoens't help debug the error | 00:17 |
| *** yamamoto has quit IRC | 00:17 | |
| jeblair | jhesketh: the difference between option 1 and option 2 is that in option 2, you tell zuul that you're talking about gerrit twice, which means that you have an additional chance to get it wrong (and are telling it something it should be able to figure out on its own) | 00:17 |
| clarkb | tsg: but it looks like the fake app should be returning 404 but instead it returns a 200? | 00:17 |
| *** Ish__ has left #openstack-infra | 00:17 | |
| clarkb | tsg: maybe the "mock" is breaking? | 00:17 |
| *** tomoe_ has quit IRC | 00:18 | |
| jhesketh | jeblair: I'm confused, I think that's what I was saying (re timer not needing a connector) | 00:18 |
| *** yamamoto has joined #openstack-infra | 00:18 | |
| tsg | clarkb: I will take a closer look .. the problem is I have been able to reproduce it locally 0% of the time | 00:19 |
| jeblair | jhesketh: you asserted that timer is not special in option 2 but is special in option 1. it's exactly the same in either case. | 00:19 |
| *** atiwari has quit IRC | 00:19 | |
| clarkb | tsg: we really aren't doing much special. These are 8 core 8gb rax and hpcloud trusty VMs and they do `tox -epy27` | 00:19 |
| *** yamamoto has quit IRC | 00:20 | |
| *** yamamoto has joined #openstack-infra | 00:20 | |
| jeblair | jhesketh: anyway, let's get back to why you don't like option 1 | 00:20 |
| *** daneleblanc has joined #openstack-infra | 00:20 | |
| jhesketh | jeblair: in option 1 triggers becomes a named connector except for things like timer and zuul... however in option 2 the change is that the gerrit trigger (which is always named gerrit) takes a connection parameter while the timer trigger has no such parameter | 00:20 |
| *** linkid has left #openstack-infra | 00:20 | |
| jhesketh | so it's consistent in that each trigger has its own set of parameters | 00:20 |
| jeblair | jhesketh: i agree; though it does so at the cost of redundancy and a potential error point for users writing a layout | 00:21 |
| jhesketh | jeblair: I don't like option 1 because it mixes named connectors with (seemingly) hard-coded ones (ie 'openstack-gerrit' vs 'timer') | 00:21 |
| *** yamamoto_ has quit IRC | 00:21 | |
| clarkb | oh I see it is asking for profile 135 and getting back profile 13586 | 00:22 |
| clarkb | tsg: ^ | 00:22 |
| *** yamamoto_ has joined #openstack-infra | 00:22 | |
| jeblair | jhesketh: okay, i can live with option 2 | 00:22 |
| jeblair | (i'm playing with calling them 'connection' instead of 'connector') | 00:23 |
| clarkb | tsg: is 13586 the pid of the test process? | 00:23 |
| tsg | clarkb: let me look closer .. thank you! | 00:23 |
| clarkb | tsg: and the profiler matches prefixes? | 00:23 |
| jeblair | jhesketh: what else do you want to decide? | 00:23 |
| tsg | clarkb: need to understand the middleware test still :) | 00:23 |
| clarkb | tsg: so it could have to do with post boot state on machines to which pid they have reached | 00:23 |
| jeblair | jhesketh: object and class implementation, yeah? | 00:23 |
| *** dane_leblanc has quit IRC | 00:23 | |
| tsg | but good points | 00:23 |
| *** yamamoto_ has quit IRC | 00:23 | |
| clarkb | tsg: which on our slaves may be pretty consistent by the time a test runs | 00:23 |
| *** yamamoto_ has joined #openstack-infra | 00:24 | |
| jhesketh | jeblair: so I do agree with what you're saying with option 2 regarding the redundancy | 00:24 |
| jhesketh | I'm not sure which is best at this point | 00:24 |
| *** yamamoto has quit IRC | 00:24 | |
| jeblair | jhesketh: option 2 is more easily backwards compatible :) | 00:24 |
| jhesketh | jeblair: but yeah, I think that'd be good to get an idea of | 00:24 |
| jhesketh | clarkb: do you have an opinion? | 00:25 |
| jeblair | jhesketh: so i think you were suggesting 4 classes earlier | 00:25 |
| *** yamamoto has joined #openstack-infra | 00:25 | |
| *** melwitt has quit IRC | 00:26 | |
| *** shashankhegde has quit IRC | 00:26 | |
| jhesketh | jeblair: yeah, so a class for each action (so to speak), where the gerrit ones would expect to be provided with a gerrit connection (which has a loaded gerrit driver) | 00:26 |
| clarkb | I liked 3 | 00:26 |
| clarkb | my brain understood it best | 00:26 |
| jhesketh | and I'm guessing the gerrit driver would look a little like the lib/gerrit.py | 00:26 |
| jhesketh | clarkb: so 3 is very close to 2 | 00:27 |
| clarkb | yup the big thing is it names the gerrit thibg | 00:27 |
| clarkb | and skips connector as a magical thibg | 00:27 |
| *** yamamot__ has joined #openstack-infra | 00:27 | |
| jeblair | clarkb: that's actually why i was thinking of changing 'connector' to 'connection', to try to make the abstraction a little more obvious | 00:27 |
| clarkb | because to me connector == driver for the most part | 00:27 |
| clarkb | jeblair ya that helps too | 00:28 |
| *** gokrokve has quit IRC | 00:28 | |
| *** yamamoto_ has quit IRC | 00:28 | |
| *** gokrokve has joined #openstack-infra | 00:29 | |
| *** yamamot__ has quit IRC | 00:29 | |
| *** yamamoto_ has joined #openstack-infra | 00:29 | |
| jeblair | jhesketh, clarkb: what do you think of the class layout notes i jotted down? | 00:29 |
| *** bknudson has joined #openstack-infra | 00:29 | |
| jeblair | jhesketh, clarkb: i'm basically imagining the connector(or connection) defined by the ini is a global singleton | 00:30 |
| *** yamamoto has quit IRC | 00:30 | |
| jeblair | jhesketh, clarkb: and each source/trigger/reporter that references it is passed a reference to it | 00:30 |
| *** shashankhegde has joined #openstack-infra | 00:30 | |
| *** yamamoto has joined #openstack-infra | 00:31 | |
| jeblair | jhesketh, clarkb: that way it can hold things that can be shared (such as an ssh connection), or just be used to pass connection info around if something needs to make its own connection | 00:31 |
| jeblair | (as in the case of the stream-events connection) | 00:31 |
| *** yamamot__ has joined #openstack-infra | 00:33 | |
| jhesketh | jeblair: that looks like what I was imagining | 00:33 |
| *** gokrokve has quit IRC | 00:33 | |
| *** yamamot__ has quit IRC | 00:34 | |
| *** pblaho_ has quit IRC | 00:34 | |
| *** yamamot__ has joined #openstack-infra | 00:34 | |
| clarkb | jeblair++ | 00:35 |
| *** yamamoto_ has quit IRC | 00:35 | |
| jeblair | jhesketh: an smtpreporter wouldn't get a gerrit connection object, right? it'd get an smtpconnection | 00:35 |
| jhesketh | yeah, I realised that was wrong | 00:35 |
| *** yamamoto has quit IRC | 00:35 | |
| jhesketh | everything would need to get a copy of the source though | 00:35 |
| *** yamamot__ has quit IRC | 00:36 | |
| *** tomoe_ has joined #openstack-infra | 00:36 | |
| *** yamamoto has joined #openstack-infra | 00:36 | |
| jeblair | jhesketh: hrm, maybe not? | 00:37 |
| jhesketh | jeblair: so it's expected that changes only ever come from one source right? | 00:38 |
| *** yamamoto_ has joined #openstack-infra | 00:38 | |
| jhesketh | maybe this only applies to triggers, but antyhing that is triggering a change to be queued should know about the source | 00:38 |
| jhesketh | it'll also allow reporters to do fancy extra queries of the change before sending a report if it needed to | 00:38 |
| *** yamamoto_ has quit IRC | 00:39 | |
| *** yamamoto_ has joined #openstack-infra | 00:40 | |
| *** yamamoto has quit IRC | 00:40 | |
| jeblair | jhesketh: i think having triggers not know about the source helps clear up the abstraction; i think that's what makes the timer and zuul triggers much easier to reason about | 00:41 |
| *** yamamoto has joined #openstack-infra | 00:42 | |
| jeblair | jhesketh: but i think you're right | 00:42 |
| jeblair | jhesketh: they do still need to know the source | 00:42 |
| *** evgenyf has quit IRC | 00:43 | |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Raise an error if preversion versions are too low https://review.openstack.org/108269 | 00:43 |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Look for and process sem-ver pseudo headers in git https://review.openstack.org/108270 | 00:43 |
| *** yamamoto has quit IRC | 00:43 | |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Refactor oneline changelog walking. https://review.openstack.org/108271 | 00:43 |
| *** yamamot__ has joined #openstack-infra | 00:43 | |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Only consider tags that look like versions. https://review.openstack.org/114403 | 00:43 |
| jhesketh | isn't the reason the source is abstacted so the triggers/etc can know about it? (ie I'm not sure how that muddles the abstraction) | 00:43 |
| *** imcsk8_ has quit IRC | 00:43 | |
| *** imcsk8 has joined #openstack-infra | 00:44 | |
| jeblair | jhesketh: yep, i think we're agreed | 00:44 |
| *** mkerrin has joined #openstack-infra | 00:44 | |
| *** yamamoto_ has quit IRC | 00:44 | |
| jhesketh | jeblair: Another way we could possibly do it is not have the gerrit trigger take a connection param... it could work out the connection details by using the source object | 00:45 |
| *** yamamoto has joined #openstack-infra | 00:45 | |
| *** yamamoto has quit IRC | 00:47 | |
| *** yamamoto has joined #openstack-infra | 00:47 | |
| lifeless | fungi: ^ bugfix for the bug you pointed me at | 00:47 |
| jeblair | jhesketh: i think that's an option; it would be a little weird that you'd still have to provide a connection to the smtp reporter | 00:47 |
| jhesketh | yeah, I hadn't thought about the need for other connection types at the time | 00:48 |
| lifeless | fungi: sadly its at the top of the stack, and depends on the haveged slaves working (or we drop python2.6) | 00:48 |
| jeblair | jhesketh: but also wouldn't suggest that it's sensible to have your source connected to one gerrit and a trigger connected to a different one :) | 00:48 |
| jhesketh | jeblair: I mean we could do the smtp reporter the same way it is currently done | 00:48 |
| *** yamamot__ has quit IRC | 00:48 | |
| *** yamamoto_ has joined #openstack-infra | 00:49 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: [WIP] Subscription events now appear in UI https://review.openstack.org/114404 | 00:49 |
| jeblair | jhesketh: i think we have enough general agreement to take a stab at this, yeah? | 00:50 |
| *** yamamot__ has joined #openstack-infra | 00:51 | |
| fungi | lifeless: both wonderful and tragic, agreed. i'm still managing to narrow down the cause/find a workaround (but am also trying to participate in an end user working working group kick-off call at the same time and just realized i was typing this with my clicky keyboard without muting my phone!) | 00:51 |
| jeblair | jhesketh: would you mind reviewing my two changes with an eye to anything that would block this, and if they don't, let's work on this for the multi-connection change? | 00:52 |
| *** yamamoto has quit IRC | 00:52 | |
| jeblair | jhesketh: i'm happy to do some of the hacking on this too, i'm not meaning to throw this all into your lap :) | 00:52 |
| *** shashankhegde has quit IRC | 00:52 | |
| *** yamamoto has joined #openstack-infra | 00:52 | |
| jeblair | jhesketh: just that i don't want to block those if it's not needed | 00:53 |
| jhesketh | yep, will do | 00:54 |
| jeblair | jhesketh: cool, thanks. i'm going to see what i can scrape up for dinner now :) | 00:54 |
| *** yamamo___ has joined #openstack-infra | 00:54 | |
| jhesketh | no worries, thanks for hanging around :-) | 00:54 |
| *** yamamoto_ has quit IRC | 00:54 | |
| *** yamamot__ has quit IRC | 00:55 | |
| *** flwang_ has joined #openstack-infra | 00:55 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added a subscription_events table & subscription hander https://review.openstack.org/113016 | 00:55 |
| lifeless | fungi: anyhow, I'm going to leave this now :) | 00:55 |
| fungi | lifeless: yep, thanks! | 00:56 |
| *** yamamoto_ has joined #openstack-infra | 00:56 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added a subscription_events table & subscription hander https://review.openstack.org/113016 | 00:57 |
| *** yamamoto has quit IRC | 00:57 | |
| *** yamamoto_ has quit IRC | 00:57 | |
| *** yamamoto has joined #openstack-infra | 00:58 | |
| *** yamamo___ has quit IRC | 00:58 | |
| *** yamamoto has quit IRC | 00:59 | |
| *** flwang_ has quit IRC | 00:59 | |
| *** yamamoto has joined #openstack-infra | 01:00 | |
| openstackgerrit | JC Delay proposed a change to openstack-infra/git-review: get_remote_url(): honor any "url.<base>.insteadOf" config setting https://review.openstack.org/109851 | 01:00 |
| *** yamamoto has quit IRC | 01:01 | |
| *** yamamoto_ has joined #openstack-infra | 01:01 | |
| openstackgerrit | JC Delay proposed a change to openstack-infra/git-review: get_remote_url(): honor any "url.<base>.insteadOf" config setting https://review.openstack.org/109851 | 01:02 |
| *** yamamoto_ has quit IRC | 01:03 | |
| *** yamamoto has joined #openstack-infra | 01:03 | |
| *** yamamoto has quit IRC | 01:05 | |
| *** rhsu has quit IRC | 01:05 | |
| *** yamamoto has joined #openstack-infra | 01:05 | |
| *** bhuvan_ has quit IRC | 01:06 | |
| *** yamamoto has quit IRC | 01:06 | |
| *** yamamoto has joined #openstack-infra | 01:07 | |
| mordred | clarkb I miss anything fun? | 01:07 |
| *** unicell has joined #openstack-infra | 01:09 | |
| *** annegentle has joined #openstack-infra | 01:11 | |
| *** salv-orlando has quit IRC | 01:13 | |
| clarkb | uhm I found beer | 01:14 |
| *** gokrokve has joined #openstack-infra | 01:14 | |
| *** gokrokve has quit IRC | 01:14 | |
| clarkb | jeblair and jhesketh had good conversation | 01:14 |
| *** jyuso has joined #openstack-infra | 01:14 | |
| clarkb | fungi is debugging centos | 01:14 |
| jhesketh | so really clarkb won | 01:14 |
| jhesketh | I found green tea and biscuits if that counts | 01:14 |
| jhesketh | it's also AM here | 01:15 |
| *** gokrokve has joined #openstack-infra | 01:15 | |
| *** nosnos has joined #openstack-infra | 01:15 | |
| fungi | i should continue debugging centos with beer. my fridge (*is* right over *there* after all | 01:15 |
| *** ramashri has quit IRC | 01:15 | |
| clarkb | nodepool is happy | 01:16 |
| fungi | when nodepool is happy, everyone is happy | 01:16 |
| clarkb | yolanda and or anyone else need to rethink where we start client sessions | 01:16 |
| * mordred found cachaça | 01:18 | |
| mordred | clarkb: oh yeah? what's up with client sessions? | 01:18 |
| fungi | mordred: is that like finding religion? | 01:18 |
| mordred | fungi: YES | 01:18 |
| mordred | fungi: I was out last night, which made me a sad mordred | 01:18 |
| fungi | rejoice | 01:19 |
| mordred | fungi: indeed he is cachaça | 01:19 |
| fungi | for the universe hath brought forth sugarcane and rendered it unto you as booze | 01:19 |
| clarkb | mordred https://review.openstack.org/#/c/114388/ | 01:20 |
| *** Administrator_ has quit IRC | 01:20 | |
| clarkb | mordred basically that fails today | 01:20 |
| mordred | fungi: sandy really liked that last bit | 01:20 |
| *** annegentle has quit IRC | 01:20 | |
| clarkb | if a cloud is down nodepool dies its related to the dib change | 01:20 |
| mordred | clarkb: spectacular | 01:20 |
| mordred | clarkb: WOOT | 01:20 |
| mordred | yolanda will fix with magic | 01:20 |
| clarkb | so we installed head~1 and restarted | 01:20 |
| clarkb | yup thats the hope :) | 01:20 |
| mordred | clarkb: oh - so, we installed without dib? | 01:21 |
| mordred | to fix the thing? | 01:21 |
| *** dims has quit IRC | 01:22 | |
| *** nosnos has quit IRC | 01:22 | |
| clarkb | ya | 01:22 |
| *** dims has joined #openstack-infra | 01:22 | |
| clarkb | but didnt revert | 01:22 |
| mordred | kk | 01:23 |
| clarkb | because its onlybbroken if your clouds are down | 01:23 |
| *** nosnos has joined #openstack-infra | 01:23 | |
| mordred | it's probably something weird or something | 01:23 |
| clarkb | not really. client timesout forces nodepool to try again | 01:23 |
| fungi | nice. according to the all-knowing wikipedia, cachaça is (among other names) known as "água-benta" (holy water) | 01:23 |
| clarkb | nodepool never gets beyond initial config | 01:23 |
| clarkb | need to push client sessions further down to make timeouts/errors cheap | 01:24 |
| mordred | fungi: NICE | 01:24 |
| mordred | ahh. yes. this makes sense | 01:24 |
| *** dims_ has joined #openstack-infra | 01:25 | |
| *** dims has quit IRC | 01:25 | |
| *** crc32 has quit IRC | 01:33 | |
| fungi | crappity. when i distill the presumed trigger case down to this trivial manifest, the bug does not emerge... http://paste.openstack.org/show/95290/ | 01:33 |
| *** yamamoto has quit IRC | 01:33 | |
| fungi | python-requests is yum removed, requests is pip installed and then libffi-devel is yum installed | 01:33 |
| fungi | so clearly the cause of this is more subtle than simply "it's running under puppet" | 01:34 |
| *** zz_dimtruck is now known as dimtruck | 01:35 | |
| *** jergerber has quit IRC | 01:36 | |
| *** Sukhdev has quit IRC | 01:36 | |
| fungi | i'm basically going to have to hack on it in the context of the entire manifest, which is slow to iterate given its runtime | 01:36 |
| fungi | unless anyone else has bright ideas | 01:36 |
| *** Sukhdev has joined #openstack-infra | 01:36 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:36 | |
| mordred | fungi: if I say "rewrite it in ansible", will you kill me? | 01:37 |
| fungi | mordred: nah. my telekinesis doesn't reach more than a few hundred miles | 01:38 |
| *** gokrokve has quit IRC | 01:39 | |
| mordred | fungi: sweet | 01:39 |
| *** gokrokve has joined #openstack-infra | 01:39 | |
| *** Sukhdev_ has joined #openstack-infra | 01:39 | |
| *** shayneburgess has joined #openstack-infra | 01:40 | |
| *** yaguang has joined #openstack-infra | 01:42 | |
| *** jaypipes has quit IRC | 01:42 | |
| *** Sukhdev has quit IRC | 01:42 | |
| *** Ryan_Lane has quit IRC | 01:43 | |
| *** xchu has joined #openstack-infra | 01:43 | |
| *** jaypipes has joined #openstack-infra | 01:43 | |
| *** paulrad has joined #openstack-infra | 01:43 | |
| *** gokrokve has quit IRC | 01:43 | |
| *** marun has quit IRC | 01:44 | |
| *** Sukhdev_ has quit IRC | 01:44 | |
| *** marun has joined #openstack-infra | 01:44 | |
| *** Ish__ has joined #openstack-infra | 01:48 | |
| *** paulrad has quit IRC | 01:51 | |
| *** ramashri has joined #openstack-infra | 01:53 | |
| *** krtaylor has joined #openstack-infra | 01:54 | |
| *** flaviof is now known as flaviof_zzz | 01:55 | |
| *** jaypipes has quit IRC | 01:55 | |
| *** SumitNaiksatam has quit IRC | 01:57 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:57 | |
| *** ramashri has quit IRC | 01:58 | |
| *** ramashri has joined #openstack-infra | 02:00 | |
| openstackgerrit | Maru Newby proposed a change to openstack-infra/config: Fix neutron-func job to unstack in gate_hook https://review.openstack.org/114416 | 02:04 |
| *** dims_ has quit IRC | 02:04 | |
| *** annegentle has joined #openstack-infra | 02:05 | |
| *** dims has joined #openstack-infra | 02:05 | |
| *** homeless has quit IRC | 02:06 | |
| *** jaypipes has joined #openstack-infra | 02:07 | |
| *** annegentle has quit IRC | 02:10 | |
| *** yamamoto has joined #openstack-infra | 02:10 | |
| *** shayneburgess has quit IRC | 02:12 | |
| *** tomoe__ has joined #openstack-infra | 02:13 | |
| *** tomoe_ has quit IRC | 02:13 | |
| *** otter768 has quit IRC | 02:15 | |
| *** tomoe_ has joined #openstack-infra | 02:15 | |
| *** tomoe__ has quit IRC | 02:18 | |
| *** ramashri has quit IRC | 02:19 | |
| *** otter768 has joined #openstack-infra | 02:22 | |
| *** tomoe_ has quit IRC | 02:24 | |
| *** tomoe_ has joined #openstack-infra | 02:24 | |
| *** zehicle_at_dell has quit IRC | 02:25 | |
| *** zehicle_at_dell has joined #openstack-infra | 02:25 | |
| *** ramashri has joined #openstack-infra | 02:26 | |
| mgagne | Are CI bots allowed to comment with "Starting check jobs." on new change or should it only be the actual results? | 02:30 |
| *** Krast has joined #openstack-infra | 02:33 | |
| openstackgerrit | Aishwarya Thangappa proposed a change to openstack-infra/storyboard: Added a subscription_events table & subscription hander https://review.openstack.org/113016 | 02:34 |
| *** SumitNaiksatam has quit IRC | 02:35 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/jenkins-job-builder: Add test for templated- and nested- builders https://review.openstack.org/114420 | 02:35 |
| *** SumitNaiksatam has joined #openstack-infra | 02:36 | |
| *** Ish__ has quit IRC | 02:38 | |
| trinaths | mgagne, as I remember its only result no starting jobs | 02:38 |
| trinaths | SumitNaiksatam,: Hi sir | 02:39 |
| mgagne | trinaths: where should I complain? =) | 02:39 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/jenkins-job-builder: Add test for templated- and nested- builders https://review.openstack.org/114420 | 02:43 |
| *** dustins has joined #openstack-infra | 02:43 | |
| jesusaurus | clarkb: im not sure how to go about making ^that^ test pass, but thats what i found earlier | 02:45 |
| trinaths | mgagne, mail it in infra/neutron mailing list | 02:47 |
| mgagne | trinaths: thanks. mostly cinder bots for me =) | 02:47 |
| *** dustins has quit IRC | 02:48 | |
| *** dustins has joined #openstack-infra | 02:48 | |
| *** dims has quit IRC | 02:53 | |
| *** flwang_ has joined #openstack-infra | 02:56 | |
| *** flwang_ has quit IRC | 03:00 | |
| *** annegentle_ has quit IRC | 03:02 | |
| *** arosen1 has quit IRC | 03:05 | |
| *** arosen1 has joined #openstack-infra | 03:05 | |
| *** zengyunling__ has quit IRC | 03:06 | |
| *** Krast has quit IRC | 03:10 | |
| *** Krast has joined #openstack-infra | 03:10 | |
| *** praneshp has quit IRC | 03:12 | |
| *** adalbas has quit IRC | 03:13 | |
| *** shashankhegde has joined #openstack-infra | 03:15 | |
| trinaths | SumitNaiksatam: Can you look into the change view.openstack.org/#/c/109659/ | 03:16 |
| trinaths | SumitNaiksatam, review.openstack.org/#/c/109659/ - typo | 03:16 |
| trinaths | mestery: Can you now access the CI log server at http://git.freescale.com/openstack_logs/564/112146/13/Freescale-FWaaS-Plugin | 03:17 |
| *** mmaglana has joined #openstack-infra | 03:18 | |
| *** flaviof_zzz is now known as flaviof | 03:18 | |
| *** ominakov has joined #openstack-infra | 03:20 | |
| *** daneleblanc has quit IRC | 03:23 | |
| *** Ryan_Lane has joined #openstack-infra | 03:26 | |
| *** nosnos has quit IRC | 03:28 | |
| *** dane_leblanc has joined #openstack-infra | 03:29 | |
| *** yfried__ has quit IRC | 03:31 | |
| *** gokrokve has joined #openstack-infra | 03:31 | |
| *** xchu has quit IRC | 03:31 | |
| *** gokrokve has quit IRC | 03:33 | |
| *** AlexF has joined #openstack-infra | 03:34 | |
| *** dimtruck is now known as zz_dimtruck | 03:34 | |
| *** ramashri has quit IRC | 03:40 | |
| *** nosnos has joined #openstack-infra | 03:41 | |
| *** harlowja is now known as harlowja_away | 03:42 | |
| *** daya_k has joined #openstack-infra | 03:43 | |
| *** Sukhdev has joined #openstack-infra | 03:46 | |
| *** trinaths has quit IRC | 03:49 | |
| *** ramashri has joined #openstack-infra | 03:49 | |
| *** changbl has joined #openstack-infra | 03:50 | |
| *** chandankumar has joined #openstack-infra | 03:54 | |
| *** shashankhegde has quit IRC | 03:55 | |
| *** amotoki has joined #openstack-infra | 03:56 | |
| *** lttrl has quit IRC | 03:59 | |
| anteaya | mgagne: you can complain to me | 04:02 |
| anteaya | mgagne: one comment per patchset for third party ci accounts | 04:02 |
| *** dustins has quit IRC | 04:02 | |
| *** pcrews has quit IRC | 04:04 | |
| *** praneshp_ has joined #openstack-infra | 04:05 | |
| *** Ryan_Lane has quit IRC | 04:06 | |
| *** chandankumar has quit IRC | 04:06 | |
| *** shashankhegde has joined #openstack-infra | 04:08 | |
| *** medieval1 has quit IRC | 04:09 | |
| *** tsg has quit IRC | 04:09 | |
| *** medieval1 has joined #openstack-infra | 04:09 | |
| *** garyh has quit IRC | 04:09 | |
| *** amotoki has quit IRC | 04:10 | |
| *** zehicle_at_dell has quit IRC | 04:13 | |
| *** zehicle_at_dell has joined #openstack-infra | 04:13 | |
| *** medieval1 has quit IRC | 04:14 | |
| *** shashankhegde has joined #openstack-infra | 04:15 | |
| *** harlowja_at_home has joined #openstack-infra | 04:16 | |
| *** Ish__ has joined #openstack-infra | 04:18 | |
| *** amcrn has quit IRC | 04:23 | |
| *** harlowja_at_home has quit IRC | 04:24 | |
| *** amotoki has joined #openstack-infra | 04:33 | |
| *** ramashri has quit IRC | 04:37 | |
| *** ildikov has quit IRC | 04:51 | |
| *** rfolco has quit IRC | 04:54 | |
| r1chardj0n3s | if any infra folk are still awake, I'd love some review love for https://review.openstack.org/#/c/114137/ so I can kick the pip-missing-reqs party off in nova land :) | 04:57 |
| *** flwang_ has joined #openstack-infra | 04:57 | |
| *** harlowja_at_home has joined #openstack-infra | 04:59 | |
| *** yfried__ has joined #openstack-infra | 04:59 | |
| *** garyh has joined #openstack-infra | 05:01 | |
| *** flwang_ has quit IRC | 05:01 | |
| anteaya | r1chardj0n3s: hey there | 05:11 |
| r1chardj0n3s | hi anteaya :) | 05:11 |
| anteaya | how art thou? | 05:18 |
| *** otter768 has quit IRC | 05:18 | |
| r1chardj0n3s | anteaya: great, and busy, how're you? | 05:19 |
| anteaya | good thanks | 05:19 |
| anteaya | just finished sprint number 3 | 05:19 |
| r1chardj0n3s | sprinty sprinty :) | 05:20 |
| anteaya | yeah exactly | 05:20 |
| anteaya | listening to a bit of jazz | 05:20 |
| anteaya | not bad | 05:20 |
| r1chardj0n3s | :) | 05:20 |
| r1chardj0n3s | I gotta go now tho and pick my daughter up from school :) | 05:20 |
| anteaya | can't be late for that | 05:20 |
| anteaya | later | 05:20 |
| * anteaya reviews jhesketh's patch | 05:21 | |
| *** shashankhegde has quit IRC | 05:21 | |
| anteaya | +1 | 05:22 |
| *** reed has quit IRC | 05:24 | |
| *** ominakov has quit IRC | 05:24 | |
| *** Ish__ has quit IRC | 05:29 | |
| *** chenglch__ has joined #openstack-infra | 05:30 | |
| *** Krast has quit IRC | 05:31 | |
| *** zz_dimtruck is now known as dimtruck | 05:34 | |
| *** jamielennox is now known as jamielennox|away | 05:35 | |
| *** harlowja_at_home has quit IRC | 05:35 | |
| *** AlexF has quit IRC | 05:36 | |
| *** rfolco has joined #openstack-infra | 05:38 | |
| *** Longgeek has joined #openstack-infra | 05:40 | |
| *** arosen1 has quit IRC | 05:41 | |
| *** arosen2 has joined #openstack-infra | 05:41 | |
| *** ildikov has joined #openstack-infra | 05:41 | |
| *** vermarah_ has joined #openstack-infra | 05:44 | |
| *** dimtruck is now known as zz_dimtruck | 05:44 | |
| *** Sukhdev has quit IRC | 05:46 | |
| *** vermarah_ has quit IRC | 05:48 | |
| *** dims has joined #openstack-infra | 05:54 | |
| *** dims has quit IRC | 05:58 | |
| *** _nadya_ has joined #openstack-infra | 06:01 | |
| *** Krast has joined #openstack-infra | 06:09 | |
| yolanda | sorry clarkb, mordred, i think i missed context. What's the problem with nodepool client? | 06:14 |
| *** _nadya_ has quit IRC | 06:16 | |
| *** marun has quit IRC | 06:20 | |
| *** marun has joined #openstack-infra | 06:20 | |
| *** maxbit has joined #openstack-infra | 06:20 | |
| *** stevemar has quit IRC | 06:22 | |
| *** otherwiseguy has quit IRC | 06:24 | |
| *** otherwiseguy has joined #openstack-infra | 06:24 | |
| *** daya_k has quit IRC | 06:28 | |
| *** maxbit has quit IRC | 06:31 | |
| *** otherwiseguy has quit IRC | 06:31 | |
| *** otherwiseguy has joined #openstack-infra | 06:32 | |
| *** otherwiseguy has quit IRC | 06:37 | |
| *** otherwiseguy has joined #openstack-infra | 06:38 | |
| *** pcrews has joined #openstack-infra | 06:40 | |
| *** jcoufal has joined #openstack-infra | 06:44 | |
| *** pblaho_ has joined #openstack-infra | 06:45 | |
| *** zz_naotok is now known as naotok | 06:46 | |
| *** mrda is now known as mrda-weekend | 06:46 | |
| *** pblaho_ has quit IRC | 06:46 | |
| *** pblaho_ has joined #openstack-infra | 06:47 | |
| *** dtantsur|afk is now known as dtantsur | 06:47 | |
| *** pblaho_ is now known as pblaho | 06:48 | |
| *** _nadya_ has joined #openstack-infra | 06:50 | |
| *** otherwiseguy has quit IRC | 06:51 | |
| *** salv-orlando has joined #openstack-infra | 06:56 | |
| *** salv-orlando has quit IRC | 06:57 | |
| *** flwang_ has joined #openstack-infra | 06:58 | |
| *** _nadya_ has quit IRC | 07:01 | |
| *** e0ne has joined #openstack-infra | 07:01 | |
| *** salv-orlando has joined #openstack-infra | 07:02 | |
| *** flwang_ has quit IRC | 07:02 | |
| *** salv-orlando has quit IRC | 07:03 | |
| *** marun has quit IRC | 07:03 | |
| *** e0ne has quit IRC | 07:09 | |
| *** salv-orlando has joined #openstack-infra | 07:13 | |
| *** salv-orlando has quit IRC | 07:13 | |
| *** e0ne has joined #openstack-infra | 07:13 | |
| *** salv-orlando has joined #openstack-infra | 07:14 | |
| *** e0ne has quit IRC | 07:14 | |
| *** salv-orlando has quit IRC | 07:16 | |
| *** e0ne has joined #openstack-infra | 07:19 | |
| *** e0ne has quit IRC | 07:21 | |
| *** e0ne has joined #openstack-infra | 07:24 | |
| *** e0ne has quit IRC | 07:26 | |
| *** arosen2 has quit IRC | 07:26 | |
| *** e0ne has joined #openstack-infra | 07:30 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 07:30 | |
| *** e0ne has quit IRC | 07:32 | |
| *** zz_dimtruck is now known as dimtruck | 07:34 | |
| *** Longgeek has quit IRC | 07:37 | |
| *** maxbit has joined #openstack-infra | 07:37 | |
| *** Longgeek has joined #openstack-infra | 07:37 | |
| *** maxbit has quit IRC | 07:38 | |
| *** _nadya_ has joined #openstack-infra | 07:40 | |
| *** Longgeek has quit IRC | 07:42 | |
| *** dimtruck is now known as zz_dimtruck | 07:44 | |
| *** skolekonov has joined #openstack-infra | 07:46 | |
| *** Longgeek has joined #openstack-infra | 07:51 | |
| *** mmaglana has quit IRC | 07:54 | |
| *** ihrachyshka has joined #openstack-infra | 07:58 | |
| *** e0ne has joined #openstack-infra | 08:00 | |
| *** jpich has joined #openstack-infra | 08:00 | |
| *** e0ne has quit IRC | 08:04 | |
| *** jedimike has joined #openstack-infra | 08:08 | |
| mattoliverau | Friday night!! I'm calling it a night, have agreat weekend all. | 08:10 |
| *** ihrachyshka has quit IRC | 08:11 | |
| *** _nadya_ has quit IRC | 08:11 | |
| *** jistr has joined #openstack-infra | 08:12 | |
| *** derekh has joined #openstack-infra | 08:13 | |
| *** penguinRaider has quit IRC | 08:13 | |
| *** _nadya_ has joined #openstack-infra | 08:15 | |
| *** ihrachyshka has joined #openstack-infra | 08:17 | |
| *** jlibosva has joined #openstack-infra | 08:22 | |
| *** RicoLin has joined #openstack-infra | 08:25 | |
| *** penguinRaider has joined #openstack-infra | 08:28 | |
| *** linkid has joined #openstack-infra | 08:28 | |
| *** praneshp_ has quit IRC | 08:29 | |
| *** amuller has joined #openstack-infra | 08:31 | |
| *** RicoLin has quit IRC | 08:35 | |
| *** RicoLin has joined #openstack-infra | 08:35 | |
| *** daya_k has joined #openstack-infra | 08:38 | |
| *** RicoLin has quit IRC | 08:40 | |
| *** RicoLin has joined #openstack-infra | 08:40 | |
| *** bmwiedemann has left #openstack-infra | 08:43 | |
| *** Longgeek has quit IRC | 08:48 | |
| *** RicoLin has quit IRC | 08:49 | |
| *** Longgeek has joined #openstack-infra | 08:50 | |
| *** RicoLin has joined #openstack-infra | 08:51 | |
| *** Longgeek has quit IRC | 08:52 | |
| *** dmellado has quit IRC | 08:52 | |
| *** RicoLin has quit IRC | 08:52 | |
| *** RicoLin has joined #openstack-infra | 08:53 | |
| *** Longgeek has joined #openstack-infra | 08:53 | |
| *** flwang_ has joined #openstack-infra | 08:59 | |
| *** RicoLin has quit IRC | 09:01 | |
| *** RicoLin has joined #openstack-infra | 09:01 | |
| *** flwang_ has quit IRC | 09:03 | |
| *** amuller has quit IRC | 09:05 | |
| *** pelix has joined #openstack-infra | 09:07 | |
| *** alexpilotti has joined #openstack-infra | 09:10 | |
| *** e0ne has joined #openstack-infra | 09:14 | |
| *** chenglch__ has quit IRC | 09:15 | |
| *** daya_k has quit IRC | 09:16 | |
| *** jyuso has quit IRC | 09:18 | |
| *** markmc has joined #openstack-infra | 09:18 | |
| *** rlandy has joined #openstack-infra | 09:21 | |
| *** linkid has quit IRC | 09:22 | |
| *** linkid has joined #openstack-infra | 09:23 | |
| *** dtantsur is now known as dtantsur|lunch | 09:24 | |
| *** RicoLin has quit IRC | 09:33 | |
| *** annegentle has joined #openstack-infra | 09:35 | |
| *** Longgeek has quit IRC | 09:35 | |
| *** pblaho is now known as pblaho|afk | 09:36 | |
| *** Alexei_987 has quit IRC | 09:36 | |
| *** Longgeek has joined #openstack-infra | 09:47 | |
| *** Krast has quit IRC | 09:56 | |
| *** unicell has quit IRC | 10:01 | |
| *** e0ne has quit IRC | 10:01 | |
| *** RicoLin has joined #openstack-infra | 10:02 | |
| *** e0ne has joined #openstack-infra | 10:04 | |
| *** RicoLin has quit IRC | 10:06 | |
| *** RicoLin has joined #openstack-infra | 10:06 | |
| *** yaguang has quit IRC | 10:07 | |
| *** yaguang has joined #openstack-infra | 10:07 | |
| *** yaguang has quit IRC | 10:10 | |
| *** yamamoto has quit IRC | 10:15 | |
| *** salv-orlando has joined #openstack-infra | 10:17 | |
| *** pcm_ has joined #openstack-infra | 10:20 | |
| *** pcm_ has quit IRC | 10:21 | |
| *** pcm_ has joined #openstack-infra | 10:21 | |
| *** dims has joined #openstack-infra | 10:25 | |
| *** Longgeek has quit IRC | 10:30 | |
| *** dims has quit IRC | 10:32 | |
| *** naotok is now known as zz_naotok | 10:32 | |
| *** asselin has quit IRC | 10:34 | |
| *** annegentle has quit IRC | 10:34 | |
| *** zehicle_at_dell has quit IRC | 10:37 | |
| *** kmartin has quit IRC | 10:41 | |
| *** lttrl has joined #openstack-infra | 10:42 | |
| *** RicoLin has quit IRC | 10:44 | |
| *** ihrachyshka_ has joined #openstack-infra | 10:45 | |
| *** ihrachyshka has quit IRC | 10:46 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Refactor WSME models https://review.openstack.org/114225 | 10:49 |
| *** dmellado has joined #openstack-infra | 10:51 | |
| *** yamamoto has joined #openstack-infra | 10:51 | |
| tsufiev | anteaya, Hello, Anita! I've addressed all your comments in the commit for creating a new stackforge project repo ( https://review.openstack.org/#/c/110325/ ). Could you please take a look when you'll have some idle moments? | 11:00 |
| *** flwang_ has joined #openstack-infra | 11:00 | |
| *** Longgeek has joined #openstack-infra | 11:00 | |
| *** flwang_ has quit IRC | 11:04 | |
| *** pblaho|afk is now known as pblaho | 11:05 | |
| *** ociuhandu_ has joined #openstack-infra | 11:05 | |
| *** ociuhandu has quit IRC | 11:05 | |
| *** ociuhandu_ is now known as ociuhandu | 11:05 | |
| *** dims has joined #openstack-infra | 11:06 | |
| *** linkid has quit IRC | 11:10 | |
| *** linkid has joined #openstack-infra | 11:11 | |
| *** RicoLin has joined #openstack-infra | 11:11 | |
| *** dims has quit IRC | 11:12 | |
| *** dims has joined #openstack-infra | 11:12 | |
| *** flaviof is now known as flaviof_zzz | 11:16 | |
| *** flaviof_zzz is now known as flaviof | 11:18 | |
| *** pblaho is now known as pblaho|afk | 11:20 | |
| *** sandywalsh has quit IRC | 11:22 | |
| *** tomoe__ has joined #openstack-infra | 11:22 | |
| *** tomoe_ has quit IRC | 11:25 | |
| *** RicoLin has quit IRC | 11:25 | |
| *** tomoe_ has joined #openstack-infra | 11:26 | |
| *** tomoe__ has quit IRC | 11:26 | |
| *** tomoe__ has joined #openstack-infra | 11:31 | |
| *** Longgeek_ has joined #openstack-infra | 11:34 | |
| *** tomoe_ has quit IRC | 11:35 | |
| *** Longgeek has quit IRC | 11:37 | |
| DinaBelova | jhesketh, hello | 11:37 |
| DinaBelova | jhesketh, are you around? | 11:37 |
| DinaBelova | :) | 11:37 |
| *** AlexF has joined #openstack-infra | 11:40 | |
| *** tomoe__ has quit IRC | 11:41 | |
| *** daya_k has joined #openstack-infra | 11:43 | |
| openstackgerrit | Mike Heald proposed a change to openstack-infra/zuul: Enforce ref only for gerrit events that supply a ref https://review.openstack.org/114510 | 11:45 |
| *** daneleblanc has joined #openstack-infra | 11:46 | |
| *** dane_leblanc has quit IRC | 11:49 | |
| *** unicell has joined #openstack-infra | 11:53 | |
| *** RicoLin has joined #openstack-infra | 11:53 | |
| *** plars has quit IRC | 11:54 | |
| *** plars has joined #openstack-infra | 11:56 | |
| *** e0ne has quit IRC | 11:56 | |
| *** ildikov has quit IRC | 11:56 | |
| openstackgerrit | Dina Belova proposed a change to openstack-infra/config: Add Ceilometer MongoDB jobs to the Tempest experimental https://review.openstack.org/114514 | 11:57 |
| *** ildikov has joined #openstack-infra | 11:57 | |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Define -T/--no-topic to disable review submit with topic https://review.openstack.org/114038 | 11:58 |
| *** ihrachyshka_ is now known as ihrachyshka | 11:58 | |
| *** rlandy has quit IRC | 12:02 | |
| *** daya_k has quit IRC | 12:03 | |
| *** dtantsur|lunch is now known as dtantsur | 12:03 | |
| *** RicoLin has quit IRC | 12:05 | |
| *** dprince has joined #openstack-infra | 12:06 | |
| *** AlexF has quit IRC | 12:09 | |
| *** sandywalsh has joined #openstack-infra | 12:13 | |
| *** yfried__ has quit IRC | 12:17 | |
| *** amotoki has quit IRC | 12:17 | |
| *** afazekas has joined #openstack-infra | 12:19 | |
| *** maxbit has joined #openstack-infra | 12:20 | |
| *** amalagon_ has joined #openstack-infra | 12:22 | |
| *** pblaho|afk is now known as pblaho | 12:24 | |
| *** rfolco has quit IRC | 12:26 | |
| *** rfolco has joined #openstack-infra | 12:26 | |
| *** adalbas has joined #openstack-infra | 12:26 | |
| ihrachyshka | afazekas: hey! I have a question for you. I'm interested in creating a separate job for mysqlconnector. I'm looking at how it's implemented for postgresql to get the idea. So what I see is that there is a variable DEVSTACK_GATE_POSTGRES passed to devstack-vm-gate.sh, though I don't see where the list of enabled services is modified. Do you know how passing the variable results in postgresql deployment in gate? | 12:28 |
| *** Ajaeger1 has joined #openstack-infra | 12:29 | |
| *** sandywalsh has quit IRC | 12:30 | |
| afazekas | ihrachyshka: https://github.com/openstack-infra/devstack-gate/blob/master/features.yaml#L10 | 12:30 |
| *** dkranz has joined #openstack-infra | 12:31 | |
| afazekas | https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L36 | 12:31 |
| afazekas | https://github.com/openstack-infra/devstack-gate/blob/master/features.yaml#L123 | 12:32 |
| afazekas | https://github.com/openstack-infra/devstack-gate/blob/master/test-matrix.py#L68 | 12:33 |
| afazekas | ihrachyshka: the last link reads the the env vars | 12:34 |
| *** sandywalsh has joined #openstack-infra | 12:34 | |
| ihrachyshka | afazekas: thanks a lot! I somehow forgot about magic possibly being involved :) | 12:34 |
| openstackgerrit | Simon McCartney proposed a change to openstack-infra/jenkins-job-builder: Add support for the SSH Agent Plugin as a build wrapper https://review.openstack.org/113222 | 12:35 |
| *** nosnos has quit IRC | 12:37 | |
| *** doug-fish has quit IRC | 12:38 | |
| *** doug-fish has joined #openstack-infra | 12:38 | |
| *** pblaho is now known as pblaho|afk | 12:39 | |
| afazekas | ihrachyshka: FYI it is expected to solve the missing (libffi-devel) package issue https://review.openstack.org/#/c/107946/ | 12:44 |
| afazekas | ihrachyshka: I have no idea what causes the missing rpm or yum issue, or how to reproduce it | 12:45 |
| ihrachyshka | afazekas: ok, fine. I'll track stable periodic jobs to see whether they still fail | 12:46 |
| ihrachyshka | afazekas: if you're interested, devstack run with mysql connector: https://review.openstack.org/#/c/114207/ | 12:49 |
| *** daneleblanc_ has joined #openstack-infra | 12:49 | |
| ihrachyshka | just three patches to merge and release (for migrate library), and we're ready to go :) | 12:50 |
| *** garyh has quit IRC | 12:50 | |
| *** aysyd has joined #openstack-infra | 12:51 | |
| *** ominakov has joined #openstack-infra | 12:51 | |
| *** garyh has joined #openstack-infra | 12:52 | |
| *** daneleblanc has quit IRC | 12:52 | |
| *** amuller has joined #openstack-infra | 12:53 | |
| *** oanufriev has quit IRC | 12:53 | |
| afazekas | psedlak: ^ | 12:54 |
| afazekas | ihrachyshka: thx | 12:54 |
| *** garyh has quit IRC | 12:57 | |
| *** radez_g0n3 is now known as radez | 12:59 | |
| ihrachyshka | afazekas: am I correct that we don't pass arbitrary env vars from infra job to devstack-gate script? in general, I would be ok with just passing SQLALCHEMY_DATABASE_DRIVER="mysqlconnector" to devstack script. should I even cope with all those features and stuff? or will it be enough to just pass it from infra job description? [no idea how to test it, so hard to check myself] | 12:59 |
| *** flwang_ has joined #openstack-infra | 13:01 | |
| *** jaypipes is now known as leakypipes | 13:03 | |
| *** russellb is now known as rustlebee | 13:03 | |
| *** leifmadsen is now known as blitzrage | 13:03 | |
| afazekas | https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst#n169 | 13:03 |
| *** dustins has joined #openstack-infra | 13:04 | |
| afazekas | `cat ~/.ssh/id_rsa.pub| awk '{print $2}'` | 13:05 |
| *** flwang_ has quit IRC | 13:05 | |
| afazekas | The above document describes how to reproduce gate jobs | 13:05 |
| *** mfer has joined #openstack-infra | 13:06 | |
| afazekas | The ssh key definition seams incorrect , the above version worked for me | 13:06 |
| fungi | wow... https://www.stackmonkey.com (take the stupid cloud survey... it's marvellous) | 13:08 |
| *** skath has quit IRC | 13:08 | |
| *** skath_ has joined #openstack-infra | 13:09 | |
| *** sandywalsh has quit IRC | 13:09 | |
| *** johnthetubaguy has quit IRC | 13:09 | |
| *** mancdaz has quit IRC | 13:09 | |
| *** pcm__ has joined #openstack-infra | 13:11 | |
| *** dims_ has joined #openstack-infra | 13:13 | |
| *** YorikSar_ has joined #openstack-infra | 13:13 | |
| afazekas | ihrachyshka: does not seams consistent when the variable needs to use DEVSTACK or DEVSTACK_GATE prefix, but usually you should not expect an env defined in the job config will be used by devstack. You need to add to the localrc explicitly in devstack-gate | 13:13 |
| ihrachyshka | afazekas: something like: http://paste.fedoraproject.org/125837/10845214 ? | 13:14 |
| *** pcrews_ has joined #openstack-infra | 13:15 | |
| afazekas | ihrachyshka: LGTM | 13:15 |
| ihrachyshka | ok then, hopefully it will be enough :) I guess I need to wait with this one before devstack changes are merged... | 13:16 |
| *** maxbit has quit IRC | 13:20 | |
| *** Ajaeger1 has quit IRC | 13:20 | |
| *** dims has quit IRC | 13:20 | |
| *** pcm_ has quit IRC | 13:20 | |
| *** jpich has quit IRC | 13:20 | |
| *** pcrews has quit IRC | 13:20 | |
| *** YorikSar has quit IRC | 13:20 | |
| *** froek has quit IRC | 13:20 | |
| *** bookwar has quit IRC | 13:20 | |
| *** fifieldt has quit IRC | 13:20 | |
| *** Svedrin has quit IRC | 13:20 | |
| *** bknudson has quit IRC | 13:22 | |
| *** e0ne has joined #openstack-infra | 13:23 | |
| *** mriedem has joined #openstack-infra | 13:23 | |
| *** zz_johnthetubagu has joined #openstack-infra | 13:25 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 13:25 | |
| *** jpich has joined #openstack-infra | 13:26 | |
| *** mancdaz has joined #openstack-infra | 13:29 | |
| *** froek has joined #openstack-infra | 13:29 | |
| *** homeless has joined #openstack-infra | 13:29 | |
| *** Svedrin has joined #openstack-infra | 13:29 | |
| *** hdd has joined #openstack-infra | 13:31 | |
| *** dansmith is now known as superdan | 13:31 | |
| *** dkliban_afk is now known as dkliban | 13:34 | |
| *** annegentle has joined #openstack-infra | 13:35 | |
| *** bookwar has joined #openstack-infra | 13:35 | |
| *** RicoLin has joined #openstack-infra | 13:36 | |
| *** skolekonov has quit IRC | 13:37 | |
| *** bknudson has joined #openstack-infra | 13:40 | |
| *** RicoLin has quit IRC | 13:40 | |
| *** RicoLin has joined #openstack-infra | 13:41 | |
| *** Ish__ has joined #openstack-infra | 13:42 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Rest controllers fix https://review.openstack.org/114534 | 13:42 |
| openstackgerrit | Ihar Hrachyshka proposed a change to openstack/requirements: Bump oslotest version https://review.openstack.org/114535 | 13:44 |
| *** pdmars has joined #openstack-infra | 13:45 | |
| *** hdd has quit IRC | 13:49 | |
| *** AlexF has joined #openstack-infra | 13:51 | |
| *** hdd has joined #openstack-infra | 13:51 | |
| *** annegentle has quit IRC | 13:51 | |
| *** pblaho|afk is now known as pblaho | 13:55 | |
| *** tsg has joined #openstack-infra | 14:01 | |
| *** jgrimm has joined #openstack-infra | 14:01 | |
| *** markmcclain has joined #openstack-infra | 14:03 | |
| *** jistr is now known as jistr|afk | 14:04 | |
| *** yfried__ has joined #openstack-infra | 14:07 | |
| *** stevemar has joined #openstack-infra | 14:08 | |
| *** nelsnelson has quit IRC | 14:09 | |
| *** AlexF has quit IRC | 14:09 | |
| *** nelsnelson has joined #openstack-infra | 14:09 | |
| *** wenlock has joined #openstack-infra | 14:12 | |
| *** markmcclain1 has joined #openstack-infra | 14:13 | |
| *** zz_dimtruck is now known as dimtruck | 14:14 | |
| *** hdd has quit IRC | 14:14 | |
| *** jistr|afk is now known as jistr | 14:15 | |
| *** markmcclain has quit IRC | 14:15 | |
| erw | fungi: nice. I like seeing that someone has finally done this. | 14:15 |
| erw | I thought someone would do it over Tor with onion addresses though. | 14:16 |
| erw | clarkb: Our nova gate for docker seems stable. We’re ready to move it to silent, if infra is. | 14:16 |
| erw | acknowledging that Juno, FF, etc is going to be much more important to you :) | 14:17 |
| fungi | erw: some of the design description suggests that different providers in the pool may offer different network architectures and tunneled addresses, so that's still a possibility | 14:17 |
| erw | fungi: the interesting thing about this being paid for with bitcoin is that the market will assure the price/hr is higher than the BTC/hr generation of the compute instances. | 14:19 |
| erw | (which I suppose is probably guaranteed anyway, considering how hard it is to CPU mine these days) | 14:19 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Revert "Added subscriber and publisher modules" https://review.openstack.org/114346 | 14:20 |
| fungi | erw: yeah, i was amused that one of their survey questions is whether you want to buy instances in their cloud to use for mining bitcoins... :; | 14:21 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Rest controllers fix https://review.openstack.org/114534 | 14:21 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Added subscriber and publisher modules https://review.openstack.org/114353 | 14:21 |
| *** wenlock has quit IRC | 14:22 | |
| openstackgerrit | Eric Windisch proposed a change to openstack-infra/config: Promote Nova gate for docker to silent https://review.openstack.org/114547 | 14:23 |
| *** AlexF has joined #openstack-infra | 14:28 | |
| *** pblaho has quit IRC | 14:28 | |
| *** AlexF has quit IRC | 14:28 | |
| *** vhoward has joined #openstack-infra | 14:30 | |
| *** ZZelle has joined #openstack-infra | 14:30 | |
| *** gargola has joined #openstack-infra | 14:31 | |
| *** dimtruck is now known as zz_dimtruck | 14:32 | |
| *** paulrad has joined #openstack-infra | 14:34 | |
| openstackgerrit | afazekas proposed a change to openstack-infra/devstack-gate: Fix ssh key specification in the README.rst https://review.openstack.org/114553 | 14:36 |
| *** alaski is now known as lascii | 14:36 | |
| jedimike | https://review.openstack.org/#/c/113287/ has 2 +2, could someone +A? thanks | 14:36 |
| *** denis_makogon has quit IRC | 14:36 | |
| *** wenlock has joined #openstack-infra | 14:37 | |
| phschwartz | morning infra | 14:38 |
| anteaya | morning phschwartz | 14:39 |
| *** dims_ is now known as dimsum | 14:40 | |
| *** dimsum is now known as dimsum_ | 14:40 | |
| * krotscheck groans himself out of bed. | 14:40 | |
| clarkb | yolanda if a cloud is configures in nodepool but the cloud is down nodepool will constantly reconfigure itself when the clients through exceptions during the reconfigure step | 14:41 |
| *** pelix has quit IRC | 14:41 | |
| clarkb | yolanda https://review.openstack.org/#/c/114388/ makes it fail like that in the tests | 14:41 |
| clarkb | yolanda we need to push the client session starts closer to the actual client requests so that if/when they fail the rest of nodepool continues to operate | 14:43 |
| *** ildikov has quit IRC | 14:43 | |
| *** mmaglana has joined #openstack-infra | 14:45 | |
| *** Ish__ has quit IRC | 14:45 | |
| *** prad has joined #openstack-infra | 14:46 | |
| *** tonytan4ever has joined #openstack-infra | 14:48 | |
| *** YorikSar_ is now known as YorikSar | 14:52 | |
| *** amuller has quit IRC | 14:53 | |
| *** zhiyan has quit IRC | 14:53 | |
| *** dangers_away is now known as dangers | 14:54 | |
| *** zhiyan has joined #openstack-infra | 14:54 | |
| *** atiwari has joined #openstack-infra | 14:54 | |
| *** datsun180b has joined #openstack-infra | 14:55 | |
| *** zz_dimtruck is now known as dimtruck | 14:58 | |
| YorikSar | Good morning, infra-cores! Please take a look at https://review.openstack.org/112666 (oslo.concurrency repo creation). I have +1 from Oslo team and +2 from SergeyLukjanov. | 15:02 |
| *** flwang_ has joined #openstack-infra | 15:02 | |
| *** ildikov has joined #openstack-infra | 15:05 | |
| *** atiwari has quit IRC | 15:06 | |
| *** flwang_ has quit IRC | 15:06 | |
| *** tsg has quit IRC | 15:07 | |
| *** mestery is now known as mestery_afk | 15:10 | |
| *** david-lyle has joined #openstack-infra | 15:12 | |
| *** homeless has quit IRC | 15:13 | |
| *** paulrad has quit IRC | 15:14 | |
| openstackgerrit | Dolph Mathews proposed a change to openstack-infra/elastic-recheck: ade query for nova quota reservation deadlock https://review.openstack.org/114566 | 15:14 |
| *** mmaglana has quit IRC | 15:14 | |
| *** tsg has joined #openstack-infra | 15:14 | |
| *** gokrokve has joined #openstack-infra | 15:15 | |
| *** mflobo has quit IRC | 15:16 | |
| openstackgerrit | Dolph Mathews proposed a change to openstack-infra/elastic-recheck: add query for bug 1316926 https://review.openstack.org/102932 | 15:16 |
| uvirtbot | Launchpad bug 1316926 in tempest "failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: BUILD. Current task state: spawning." [Undecided,New] https://launchpad.net/bugs/1316926 | 15:16 |
| *** denis_makogon has joined #openstack-infra | 15:17 | |
| zaro | morning | 15:19 |
| anteaya | morning zaro | 15:20 |
| * anteaya prepares to move locations | 15:20 | |
| *** homeless has joined #openstack-infra | 15:21 | |
| *** bhuvan has joined #openstack-infra | 15:22 | |
| *** _nadya_ has quit IRC | 15:23 | |
| openstackgerrit | Dolph Mathews proposed a change to openstack-infra/elastic-recheck: Add query for bin/pip suddenly missing https://review.openstack.org/111361 | 15:23 |
| *** reed has joined #openstack-infra | 15:25 | |
| *** reed has quit IRC | 15:25 | |
| rustlebee | fungi: point well stated re: google on the more general issue | 15:25 |
| *** _nadya_ has joined #openstack-infra | 15:25 | |
| *** reed has joined #openstack-infra | 15:25 | |
| fungi | rustlebee: thanks! i try not to preach too often | 15:26 |
| fungi | sometimes my soapbox gets a little dusty and i need to make sure it's still solid | 15:26 |
| rustlebee | fungi: it was good though | 15:26 |
| rustlebee | :) | 15:26 |
| *** atiwari has joined #openstack-infra | 15:27 | |
| * clarkb looks for church of yuggoth signs | 15:28 | |
| *** ihrachyshka has quit IRC | 15:29 | |
| clarkb | found it. well said indeed | 15:30 |
| * rustlebee considers himself corrected | 15:30 | |
| *** yfried__ has quit IRC | 15:31 | |
| *** medieval1 has joined #openstack-infra | 15:32 | |
| *** dtantsur is now known as dtantsur|afk | 15:32 | |
| *** paulrad has joined #openstack-infra | 15:35 | |
| *** gokrokve has quit IRC | 15:36 | |
| *** shashankhegde has joined #openstack-infra | 15:36 | |
| *** bhuvan has quit IRC | 15:36 | |
| *** gokrokve has joined #openstack-infra | 15:36 | |
| *** sandywalsh has joined #openstack-infra | 15:38 | |
| *** andreaf has joined #openstack-infra | 15:39 | |
| *** mriedem is now known as mattykins | 15:41 | |
| *** gokrokve has quit IRC | 15:41 | |
| *** paulrad has quit IRC | 15:41 | |
| *** mattykins is now known as matty_kins | 15:41 | |
| *** jpich has quit IRC | 15:42 | |
| *** homeless_ has joined #openstack-infra | 15:42 | |
| *** jpich has joined #openstack-infra | 15:43 | |
| *** medieval1 has quit IRC | 15:44 | |
| *** praneshp has joined #openstack-infra | 15:44 | |
| *** ramashri has joined #openstack-infra | 15:45 | |
| *** homeless has quit IRC | 15:45 | |
| *** garyh has joined #openstack-infra | 15:46 | |
| *** praneshp_ has joined #openstack-infra | 15:47 | |
| *** svarnau has joined #openstack-infra | 15:47 | |
| *** markmcclain1 has quit IRC | 15:48 | |
| *** svarnau has quit IRC | 15:48 | |
| *** praneshp has quit IRC | 15:49 | |
| *** praneshp_ is now known as praneshp | 15:49 | |
| *** pblaho has joined #openstack-infra | 15:51 | |
| *** emagana has joined #openstack-infra | 15:51 | |
| *** tonytan4ever has quit IRC | 15:52 | |
| *** tonytan4ever has joined #openstack-infra | 15:55 | |
| *** shayneburgess has joined #openstack-infra | 15:55 | |
| *** SumitNaiksatam has quit IRC | 15:55 | |
| *** paulrad has joined #openstack-infra | 15:55 | |
| *** dustins has quit IRC | 15:57 | |
| *** bhuvan has joined #openstack-infra | 15:58 | |
| *** otherwiseguy has joined #openstack-infra | 15:58 | |
| *** dustins has joined #openstack-infra | 15:59 | |
| *** svarnau has joined #openstack-infra | 16:00 | |
| *** asselin has joined #openstack-infra | 16:00 | |
| *** bnemec is now known as beekneemech | 16:01 | |
| *** armax has joined #openstack-infra | 16:02 | |
| mordred | krotscheck: that's never the right choice | 16:02 |
| *** e0ne has quit IRC | 16:02 | |
| krotscheck | mordred: It’s really not. | 16:06 |
| nibalizer | what choice is this? | 16:07 |
| nibalizer | oh getting out of bed | 16:07 |
| krotscheck | mordred: btw- Ben has 24 bottles of my wine. Burn consumption is usually about 15. That means y’all have to drink at least 9 bottles next week. | 16:07 |
| mordred | krotscheck: done | 16:07 |
| mordred | krotscheck: actually _I_ have 24 bottles of your wine | 16:07 |
| krotscheck | mordred: Oh! | 16:07 |
| clarkb | you all needs kegs | 16:08 |
| mordred | krotscheck: we switched cars when you weren't looking | 16:08 |
| krotscheck | mordred: sneaky! | 16:08 |
| mordred | clarkb: you should bring us kegs | 16:08 |
| krotscheck | I would LOVE to bring a keg. | 16:08 |
| clarkb | Mine are currently empty :( | 16:08 |
| * krotscheck ponders playa-kegerators. | 16:08 | |
| *** jlibosva has quit IRC | 16:09 | |
| *** bhuvan has quit IRC | 16:13 | |
| *** leakypipes is now known as leakypipes-codin | 16:14 | |
| krotscheck | clarkb: We’ve yet to make booze together. We should fix this. | 16:15 |
| *** leakypipes-codin is now known as leakypipes | 16:15 | |
| krotscheck | clarkb: Since you know barley, and I know wine, I suggest we make a barley wine! | 16:16 |
| clarkb | ooh we could. needs such patience though for big beers | 16:17 |
| *** mmaglana has joined #openstack-infra | 16:18 | |
| krotscheck | clarkb: I’ll bring the patience :). That’s what I’m good at. | 16:18 |
| *** medieval1 has joined #openstack-infra | 16:18 | |
| krotscheck | clarkb: And then, to finish it off, we’ll put it on nitro. | 16:18 |
| * nibalizer would like to be there for the final part of this ( the drinking ) | 16:18 | |
| krotscheck | nibalizer! I have a review chain for you :) https://review.openstack.org/#/c/114383/ | 16:21 |
| krotscheck | nibalizer: And you’re in portland, right? barley wining would be super close. | 16:22 |
| nibalizer | krotscheck: https://github.com/nanliu/puppet-staging | 16:22 |
| *** markmcclain has joined #openstack-infra | 16:22 | |
| nibalizer | so that likely would be able to accomplish what you were trying to do with puppi | 16:22 |
| nibalizer | i'm just really a big hater of execs for curl | 16:23 |
| krotscheck | nibalizer: I actually tried that. | 16:23 |
| nibalizer | sometimes its the only way | 16:23 |
| nibalizer | krotscheck: no dice? | 16:23 |
| krotscheck | nibalizer: Not idempotent | 16:23 |
| nibalizer | nu uh | 16:23 |
| * nibalizer doesn't believe you | 16:23 | |
| krotscheck | nibalizer: Executes the download every time. | 16:23 |
| *** mmaglana has quit IRC | 16:24 | |
| krotscheck | nibalizer: https://github.com/nanliu/puppet-staging/blob/master/manifests/file.pp | 16:24 |
| *** mmaglana has joined #openstack-infra | 16:24 | |
| *** gokrokve has joined #openstack-infra | 16:25 | |
| *** baoli has quit IRC | 16:25 | |
| krotscheck | nibalizer: The problem is that there’s no good way of doing a file age check via puppet short of the ‘onlyif’ command. Once you add a ‘creates’, it never runs the command again. | 16:26 |
| *** comstud is now known as bearhands | 16:26 | |
| *** derekh has quit IRC | 16:26 | |
| *** markmcclain has quit IRC | 16:27 | |
| *** Ryan_Lane has joined #openstack-infra | 16:27 | |
| *** gokrokve has quit IRC | 16:27 | |
| *** chuckC has quit IRC | 16:27 | |
| nibalizer | krotscheck: does this idempotent? https://forge.puppetlabs.com/maestrodev/wget | 16:27 |
| nibalizer | its just not that hard to curl -Z | 16:28 |
| nibalizer | or -z | 16:28 |
| *** jogo is now known as flashgordon | 16:28 | |
| nibalizer | or get the headers down and look at last-modified and compare to the file on disk | 16:28 |
| yolanda | hi clarkb, sorry, i was out, today is a bank holiday | 16:28 |
| openstackgerrit | Mark Vanderwiel proposed a change to openstack-infra/config: Fix gate job timing issues with bundler https://review.openstack.org/114593 | 16:28 |
| mordred | yolanda: stupid banks | 16:28 |
| yolanda | let me take a look at it | 16:29 |
| *** mmaglana has quit IRC | 16:29 | |
| yolanda | mordred, clarkb, but you already merged the change, right? it should be going into a new one? | 16:29 |
| *** emagana has quit IRC | 16:30 | |
| *** SumitNaiksatam has joined #openstack-infra | 16:30 | |
| mordred | yolanda: yes | 16:30 |
| *** emagana has joined #openstack-infra | 16:30 | |
| yolanda | ok, let me take a look | 16:30 |
| krotscheck | nibalizer: It doesn’t look like it. It does have an unless => command, but the test is a file existence check. | 16:30 |
| mordred | nibalizer: we don't want puppet to report "changed" just because it had to run something to check to see if it should run something | 16:31 |
| krotscheck | nibalizer: That’s what I do in mine. | 16:31 |
| krotscheck | nibalizer: Any opinion on this change? https://review.openstack.org/#/c/114089/2 | 16:32 |
| *** yamamoto has quit IRC | 16:32 | |
| *** emagana has quit IRC | 16:34 | |
| nibalizer | krotscheck: looks good | 16:35 |
| nibalizer | managing the apache2.4 changeover is a bit painful | 16:36 |
| nibalizer | two ways to do it | 16:36 |
| nibalizer | one is to maintain separate vhost files | 16:36 |
| nibalizer | the other is to do big if statements | 16:36 |
| krotscheck | nibalizer: Well, I just want the default install on trusty to not fail. | 16:36 |
| *** yamamoto has joined #openstack-infra | 16:36 | |
| YorikSar | fungi: https://review.openstack.org/112666 | 16:37 |
| YorikSar | fungi: If you're about repo creation. | 16:37 |
| fungi | YorikSar: thanks! | 16:37 |
| *** jpich has quit IRC | 16:37 | |
| *** emagana has joined #openstack-infra | 16:38 | |
| *** tonytan4ever has quit IRC | 16:38 | |
| nibalizer | krotscheck: ya | 16:39 |
| nibalizer | i think this is fine | 16:39 |
| krotscheck | clarkb: I don’t recall what the resolution of the discussion two days ago was re: infra using the most recent of their own puppet modules. Are we still pinning install_modules to specific sha’s, or should we just hit master? | 16:39 |
| nibalizer | it occurs to me that apache2.2 vs 2.4 will be a problem for a very long time | 16:40 |
| *** Hefeweizen has quit IRC | 16:40 | |
| krotscheck | nibalizer: yyyep. | 16:40 |
| *** medieval1 has quit IRC | 16:40 | |
| nibalizer | krotscheck: right now we're using a hash | 16:40 |
| *** dimtruck is now known as zz_dimtruck | 16:40 | |
| krotscheck | nibalizer: There are still apache 1 hosts out there. | 16:40 |
| nibalizer | krotscheck: jeblair asked us to go to 'master' | 16:40 |
| nibalizer | im not sure that the code does that | 16:40 |
| krotscheck | nibalizer: Oh, I know. Lemme go do that. | 16:40 |
| nibalizer | i haven't had time to dig into install_moudles.sh | 16:40 |
| nibalizer | that stuff at the end of that down there is .... pretty gross | 16:41 |
| linkid | hi | 16:42 |
| *** zz_dimtruck is now known as dimtruck | 16:42 | |
| linkid | fungi: oh, if you could do the same for mines, it would be great and I will be able to push my code this week-end (while Jenkins is cool down ^^) | 16:43 |
| reed | Ryan_Lane, hello there, are you still the best person to ask to fiddle our mediawiki? I'd like to install the freemind extension | 16:43 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Use puppet-storyboard@master https://review.openstack.org/114597 | 16:43 |
| krotscheck | Let’s see if that works. | 16:43 |
| Ryan_Lane | reed: yeah. let me see if that extension is easy | 16:43 |
| *** markmcclain has joined #openstack-infra | 16:44 | |
| reed | http://www.mediawiki.org/wiki/Extension:FreeMind | 16:44 |
| reed | thanks Ryan_Lane | 16:44 |
| fungi | linkid: which one? if it's for stackforge rather than openstack, i can't really prioritize that (i'm in the middle of trying to fix things in openstack at the moment) | 16:44 |
| Ryan_Lane | heh. ewww. flash or java app? | 16:44 |
| *** Guest35108 has joined #openstack-infra | 16:44 | |
| *** UtahDave has joined #openstack-infra | 16:45 | |
| *** UtahDave has left #openstack-infra | 16:45 | |
| linkid | fungi: ah, it is for stackforge. And if you have some other things to do, it can wait, then. | 16:45 |
| openstackgerrit | A change was merged to openstack-infra/config: Create repo for oslo.concurrency https://review.openstack.org/112666 | 16:46 |
| *** yamamoto has quit IRC | 16:46 | |
| *** markmcclain has quit IRC | 16:47 | |
| *** markmcclain has joined #openstack-infra | 16:47 | |
| krotscheck | Any infra-core out there available to look at https://review.openstack.org/#/c/114383/ and https://review.openstack.org/#/c/114089/2 ? The storyboard webclient isn’t updating at the moment :/ | 16:48 |
| *** pballand has joined #openstack-infra | 16:48 | |
| reed | Ryan_Lane, choose your poison :) both? | 16:48 |
| *** markmcclain has quit IRC | 16:48 | |
| *** markmcclain has joined #openstack-infra | 16:49 | |
| Ryan_Lane | it looks like this extension likely has XSS issues | 16:49 |
| reed | wonderful... we're back to square 1 | 16:49 |
| reed | one has to wonder why google docs becomes popular | 16:50 |
| * reed rants and goes to grab more coffee and rant some more | 16:50 | |
| Ryan_Lane | well, I haven't met anyone that uses mind maps seriously :) | 16:50 |
| fungi | reed: google docs does mindmaps? | 16:50 |
| reed | thanks Ryan_Lane | 16:50 |
| Ryan_Lane | which is likely why there's no real support for it in mediawiki | 16:50 |
| reed | Ryan_Lane, maybe you have a better suggestion: here is the use case | 16:51 |
| Ryan_Lane | let me read the code more closely | 16:51 |
| Ryan_Lane | it may not actually have issues | 16:51 |
| reed | we have a taxonomy prepared for the wiki | 16:51 |
| reed | and currently this taxonomy is a ahem spreadsheet | 16:51 |
| Ryan_Lane | oh. why not use graphviz? | 16:51 |
| reed | with colum as the 'levels', to separate branches and leaves | 16:51 |
| *** markmcclain has quit IRC | 16:51 | |
| reed | Ryan_Lane, would that work better? I have nothing against it | 16:52 |
| fungi | but yeah, to echo Ryan_Lane's comment, at a previous job we used a very large mediawiki instance for internal documentation collaboration. one of the more hipsterish development groups hounded me for ages to install that exact extension, but once i did they never went past one or two experimental mindmaps which were abandoned shortly thereafter. a year or so later the extension wasn't keeping up with | 16:52 |
| Ryan_Lane | http://www.mediawiki.org/wiki/Extension:GraphViz | 16:52 |
| fungi | mediawiki releases so i removed it in an upgrade and never looked back | 16:52 |
| Ryan_Lane | the nice thing about graphviz is that it's useful for software development, too | 16:52 |
| reed | Ryan_Lane, so are mindmaps but hey, whatever :) | 16:53 |
| Ryan_Lane | I've never met a developer who uses mindmaps :) | 16:53 |
| reed | Ryan_Lane, indeed... sw devs can be pretty primitives in the tools they choose :) | 16:53 |
| reed | muahahhahha | 16:54 |
| Ryan_Lane | so, the taxonomy needs to be an image? | 16:54 |
| *** linkid has left #openstack-infra | 16:54 | |
| reed | Ryan_Lane, not an image, better it to be something that can be navigated | 16:54 |
| reed | it's a pretty big tree | 16:54 |
| * Ryan_Lane uses omnigraffle, and it makes amazing graphs :) | 16:54 | |
| reed | i'm afraid an image would not be good | 16:54 |
| Ryan_Lane | what's the taxonomy going to do? | 16:54 |
| Ryan_Lane | is it a category tree? | 16:55 |
| reed | Ryan_Lane, it's a tool to communicate how to use the wiki, how to add categories to pages | 16:55 |
| reed | Ryan_Lane, yes, it's a category tree | 16:55 |
| *** dimsum_ has quit IRC | 16:55 | |
| Ryan_Lane | made with actual categories? :) | 16:55 |
| Ryan_Lane | https://www.mediawiki.org/wiki/Extension:CategoryTree | 16:56 |
| reed | Ryan_Lane, maybe | 16:57 |
| reed | the problem is that we're still working on it... it's a 'live' document | 16:57 |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/nodepool: Fix nodepool crashing when cloud is down https://review.openstack.org/114599 | 16:57 |
| Ryan_Lane | reed: CategoryTree references a category and all of its subcategories | 16:57 |
| Ryan_Lane | it'll update automatically | 16:57 |
| yolanda | pushed to test into onte instance in the cloud, not ready yet | 16:57 |
| *** dims has joined #openstack-infra | 16:58 | |
| *** isviridov is now known as isviridov|away | 16:58 | |
| reed | Ryan_Lane, so that means that we could develop the tree using a mindmap or anything, implement the Categories on the wiki and see the result on the CatTree? | 16:58 |
| *** bookwar has quit IRC | 16:58 | |
| Ryan_Lane | reed: yeah | 16:59 |
| clarkb | krotscheck lets hit master | 16:59 |
| jeblair | krotscheck: when i search in storyboard, they typeahead search isn't keeping up with my typing, such that if i type "storyboard<enter>" i might end up with "story" in the field. | 16:59 |
| reed | Ryan_Lane, sounds like this is what we need then | 16:59 |
| Ryan_Lane | cool. that's already installed | 16:59 |
| reed | Ryan_Lane, where? | 16:59 |
| krotscheck | jeblair: There’s a patch that will hopefully mitigate that: https://review.openstack.org/#/c/109372/ | 16:59 |
| reed | https://wiki.openstack.org/wiki/Special:CategoryTree ? | 16:59 |
| krotscheck | clarkb: Hitting master - https://review.openstack.org/#/c/114597/ | 17:00 |
| Ryan_Lane | reed: check the documentation, it's a wiki tag: https://www.mediawiki.org/wiki/Extension:CategoryTree | 17:00 |
| jeblair | krotscheck: cool, thanks | 17:00 |
| Ryan_Lane | <categorytree mode=pages>Manual</categorytree> | 17:00 |
| reed | that page is as cryptic as a gpg encrypted helloworld | 17:00 |
| *** svarnau_ has joined #openstack-infra | 17:00 | |
| Ryan_Lane | you just put that into a page, and it will show the tree for whatever category you add inside of the tags | 17:00 |
| krotscheck | jeblair: The next iteration on top of that patch will be to stop managing the typeahead responses as a batch | 17:01 |
| reed | how about the whole shebang? | 17:01 |
| Ryan_Lane | you just need to look at the usage section ;) https://www.mediawiki.org/wiki/Extension:CategoryTree#Usage | 17:01 |
| Ryan_Lane | the tag I just added should work, if you're using a Manual category | 17:01 |
| *** openstackgerrit has quit IRC | 17:02 | |
| Ryan_Lane | basically, you just want to add <categorytree mode=pages>My Category</categorytree> | 17:02 |
| YorikSar | fungi: Thanks a lot! | 17:02 |
| *** svarnau has quit IRC | 17:02 | |
| Ryan_Lane | to whatever page you want this to show up on | 17:02 |
| *** pblaho has quit IRC | 17:02 | |
| YorikSar | fungi: When will the repo get imported? | 17:02 |
| *** openstackgerrit has joined #openstack-infra | 17:02 | |
| Ryan_Lane | there's a bunch of options you can pass it, but that mode should likely do what you want | 17:02 |
| *** svarnau__ has joined #openstack-infra | 17:02 | |
| *** flwang_ has joined #openstack-infra | 17:03 | |
| *** harlowja_away is now known as harlowja | 17:03 | |
| reed | I don't get it | 17:04 |
| reed | Ryan_Lane, I don't get how to show *every* category starting from the root of the wiki | 17:04 |
| jeblair | yolanda: i don't think 114599 will fix the problem; it will just raise an exception in the same place | 17:05 |
| Ryan_Lane | reed you need to actually set up a category hierarchy | 17:05 |
| yolanda | jeblair, i was wondering if that didn't happen before, maybe because it was only a call to Nova, not involving keystone and glance | 17:05 |
| yolanda | i think the ideal should be to return None for the client, and ignore that provider in some way until it works | 17:06 |
| reed | Ryan_Lane, ok, this exhausted me and my time today, I'll have Shari look into it... | 17:06 |
| *** svarnau_ has quit IRC | 17:06 | |
| jeblair | yolanda: yeah, i think the original code did not actually make an api call in that function. but since you actually have to make an api call to keystone, now it fails | 17:06 |
| Ryan_Lane | ok. basically you want to set up a top level category, and include other categories from it | 17:06 |
| * Ryan_Lane waves | 17:06 | |
| * reed thanks Ryan_Lane :) | 17:07 | |
| yolanda | jeblair, so you think that adding code for ignoring the provider should be fine? instead of raising exception, catch and return None ? | 17:07 |
| Ryan_Lane | yw | 17:07 |
| *** flwang_ has quit IRC | 17:07 | |
| *** bhuvan has joined #openstack-infra | 17:08 | |
| jeblair | yolanda: if we ignore the provider, then we will make it to the main loop, but every time through the loop it will try the operation again and it takes 1 minute to time out | 17:08 |
| jeblair | yolanda: (because it creates missing providers each time through the loop, which is good) | 17:08 |
| yolanda | jeblair, then what do you suggest? flagging that in some way? | 17:08 |
| yolanda | maybe we could maintain a list of providers to ignore, let's say for 10 minutes | 17:09 |
| *** dustins has quit IRC | 17:09 | |
| *** tonytan4ever has joined #openstack-infra | 17:09 | |
| jeblair | that's an option | 17:10 |
| jeblair | let's call that (a) | 17:10 |
| jeblair | (b) could be to not use keystone and manually configure the glance endpoint | 17:11 |
| fungi | YorikSar: seems to be done now. who goes in the new groups? | 17:11 |
| *** MarkAtwood has joined #openstack-infra | 17:11 | |
| *** maxbit has joined #openstack-infra | 17:11 | |
| yolanda | jeblair, hardcoding the endpoint? that should be flexible enough? | 17:11 |
| clarkb | manually setting glance endpoint will fix rax glance too | 17:11 |
| clarkb | I like that option for this reason | 17:12 |
| *** nelsnelson has quit IRC | 17:12 | |
| yolanda | clarkb, but we need endpoint and auth token | 17:12 |
| jeblair | (c) could be to make it so that we don't do the keystone call and create the glance client until a call actually needs it -- that means that the cost of this is pushed down into the launch threads which can handle it, but we may end up with some race conditions to either deal with and ignore | 17:12 |
| jeblair | wow, using openstack makes no sense | 17:12 |
| jeblair | deal with _or_ ignore | 17:13 |
| *** markmc has quit IRC | 17:13 | |
| jeblair | yolanda: how long is an auth token good for? | 17:13 |
| *** chuckC has joined #openstack-infra | 17:14 | |
| yolanda | jeblair, don't know exact expiration time | 17:14 |
| jeblair | morganfainberg: ^? | 17:14 |
| *** dustins has joined #openstack-infra | 17:15 | |
| yolanda | jeblair, i think it's configurable and depends on provider | 17:15 |
| morganfainberg | jeblair, yolanda, in icehouse the default is 1h, but it can be set to anything. many operators set it to 1d, or 3h | 17:15 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add pipeline source https://review.openstack.org/112410 | 17:15 |
| jeblair | how do we get a new one or refresh it? | 17:15 |
| yolanda | so actually it would make sense to configure glance only when pushing images | 17:15 |
| *** tonytan4ever has quit IRC | 17:15 | |
| morganfainberg | jeblair, yolanda, you can get a token issued and look for 'expires' or 'expires_at' depending on the token version and see the difference from the issued_at attribute | 17:15 |
| jedimike | jeblair, as you +2'd it, could you +A https://review.openstack.org/#/c/113287/ please? cheers :) | 17:16 |
| jeblair | yolanda: i'm starting to think that may be the case. maybe for simplicity in this case we should not keep the glanceclient around, but rather just do the keystone+glance calls when we need to upload | 17:16 |
| yolanda | jeblair, yes, i'm with you on it | 17:16 |
| *** dustins has quit IRC | 17:16 | |
| jeblair | jedimike: +3 | 17:16 |
| jedimike | thanks :) | 17:16 |
| yolanda | honestly i didn't thought about auth token expiration | 17:17 |
| jeblair | heh, we would have found out eventually :) | 17:17 |
| *** jedimike has quit IRC | 17:17 | |
| morganfainberg | most apps don't re-use tokens well :( | 17:17 |
| morganfainberg | and just get new tokesn all the time | 17:17 |
| *** dustins has joined #openstack-infra | 17:17 | |
| jeblair | putting on my user hat: "man it's weird that using glance is so different than using nova" | 17:18 |
| yolanda | jeblair, so i'm fine on restructuring that a bit, leave nova client as it was before, and configure glance each time we need to upload an image | 17:18 |
| morganfainberg | the new session object clients are using should *help* make that less of an issue, not sure if glanceclient uses that yet, similar not sure about novaclient | 17:18 |
| *** homeless_ has quit IRC | 17:18 | |
| jeblair | yolanda: yeah, i think that works for now since we only perform one glance call, and we're only likely to perform it once per day per provider | 17:19 |
| yolanda | ok, let me rewrite that a bit | 17:19 |
| morganfainberg | jeblair, yolanda anyway, if you need me to look into any of this stuff, let me know i'm happy to help - but i need breakfast :P | 17:19 |
| morganfainberg | so.. be back in a bit | 17:19 |
| jeblair | morganfainberg: cool, thanks | 17:19 |
| *** hogepodge has quit IRC | 17:19 | |
| *** homeless has joined #openstack-infra | 17:20 | |
| *** maxbit has quit IRC | 17:20 | |
| jeblair | did gerritbot dissappear? | 17:20 |
| *** maxbit has joined #openstack-infra | 17:20 | |
| jeblair | oh no i just missed it. nm. | 17:21 |
| *** dustins has quit IRC | 17:22 | |
| *** bhuvan has quit IRC | 17:22 | |
| *** hogepodge has joined #openstack-infra | 17:23 | |
| *** tsg has quit IRC | 17:24 | |
| *** maxbit has quit IRC | 17:25 | |
| *** maxbit has joined #openstack-infra | 17:25 | |
| *** julim has quit IRC | 17:29 | |
| *** julim has joined #openstack-infra | 17:30 | |
| rustlebee | jeblair: so i just discovered https://jitsi.org/Projects/JitsiVideobridge ... open source thing that's very close to google hangouts. just tried it on a friend's setup | 17:32 |
| *** matty_kins is now known as mriedem | 17:32 | |
| rustlebee | major technical difference, with hangouts, all video streams go through google (so you only have 1 in/out stream), with this all streams come to you, so it won't scale well to large groups | 17:32 |
| rustlebee | but for small group things, might be worth setting up | 17:32 |
| *** emagana has quit IRC | 17:33 | |
| *** dustins has joined #openstack-infra | 17:34 | |
| *** dustins has quit IRC | 17:34 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add a Zuul trigger https://review.openstack.org/112411 | 17:35 |
| *** emagana has joined #openstack-infra | 17:36 | |
| jeblair | rustlebee: neat; you know i've been itching to use webrtc :) | 17:36 |
| rustlebee | yep :) | 17:36 |
| jeblair | clarkb: can you speedily re-review https://review.openstack.org/112411 ? i just changed the test to fix a nondeterminism the gate caught | 17:37 |
| jeblair | fungi: or you if you're around? ^ | 17:38 |
| fungi | jeblair: sure | 17:38 |
| jeblair | rustlebee: think that's a better investment that trying out webrtc video with asterisk? | 17:38 |
| jeblair | s/that try/than try/ | 17:39 |
| *** garyh has quit IRC | 17:39 | |
| openstackgerrit | afazekas proposed a change to openstack-infra/elastic-recheck: Add query for 1357476 https://review.openstack.org/114610 | 17:40 |
| fungi | jeblair: 112411 update lgtm | 17:40 |
| jeblair | cool, back in the queue it goes | 17:41 |
| *** emagana has quit IRC | 17:41 | |
| *** maxbit has quit IRC | 17:41 | |
| *** emagana has joined #openstack-infra | 17:41 | |
| *** maxbit has joined #openstack-infra | 17:41 | |
| rustlebee | jeblair: yes | 17:42 |
| rustlebee | i've never seen someone set up something this nice/usable with asterisk with webrtc | 17:43 |
| jeblair | rustlebee: so probably we set up ejabberd and plug jvb into it? | 17:44 |
| rustlebee | yeah looks like it | 17:45 |
| rustlebee | seems worth experimenting with anyway | 17:46 |
| *** krtaylor is now known as krtaylor_away | 17:46 | |
| rustlebee | tempted to do a quick manual setup first to play, and then puppetize if it's worth keeping | 17:47 |
| *** dustins has joined #openstack-infra | 17:47 | |
| *** yamamoto has joined #openstack-infra | 17:47 | |
| clarkb | jeblair: +2 | 17:47 |
| * mordred still hates video conferencing | 17:48 | |
| jeblair | mordred: yeah, one of the thinks i like about potentially using asterisk is the idea of still being able to call in on my phone :) | 17:48 |
| * jeblair dials "78" to connect to the openstack confbridge | 17:49 | |
| jeblair | but i could probably convince my local asterisk to do something with jitsi+xmpp | 17:50 |
| rustlebee | jeblair: what we have today does that in theory ... if you use a video client like jitsi and call into the same bridge as phone callers, you'll get video from other talkers that called in with video | 17:50 |
| rustlebee | or at least in theory, i never tested it on this server :) | 17:50 |
| rustlebee | but i intended that with the config | 17:51 |
| rustlebee | but you have to use jitsi. | 17:51 |
| rustlebee | (or whatever) | 17:51 |
| rustlebee | and not webrtc ... i'm not sure how to make webrtc work with this | 17:51 |
| *** praneshp has quit IRC | 17:51 | |
| rustlebee | admittedly the vast majority of the webrtc work in asterisk happened after i stopped doing asterisk full time | 17:51 |
| rustlebee | so my knowledge is a bit dated | 17:51 |
| *** baoli has joined #openstack-infra | 17:51 | |
| *** jistr is now known as jistr|afk | 17:52 | |
| *** yamamoto has quit IRC | 17:52 | |
| *** dims is now known as dimsum_ | 17:52 | |
| *** melwitt has joined #openstack-infra | 17:54 | |
| *** yamamoto has joined #openstack-infra | 17:54 | |
| *** hogepodge has quit IRC | 17:55 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/storyboard: Add loader subcommand to sync from launchpad https://review.openstack.org/113624 | 17:55 |
| *** paulrad has quit IRC | 17:55 | |
| mordred | NikitaKonovalov: ^^ ttx followed up on your comments, could you re-review when you get a chance? | 17:56 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add source entries for all Zuul pipelines https://review.openstack.org/114615 | 17:57 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add merge-check pipeline https://review.openstack.org/114616 | 17:57 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Enqueue unfresh changes in check when parent in gate https://review.openstack.org/114617 | 17:57 |
| jeblair | clarkb, fungi, mordred: ^ that's where the new zuul fun starts :) i'm going to work on restart zuul to prepare for those if you want to go ahead and review (but not approve) them | 17:58 |
| *** yamamoto has quit IRC | 17:59 | |
| *** tonytan4ever has joined #openstack-infra | 17:59 | |
| *** dustins has quit IRC | 18:02 | |
| *** hogepodge has joined #openstack-infra | 18:02 | |
| *** amcrn has joined #openstack-infra | 18:03 | |
| *** Guest35108 has quit IRC | 18:04 | |
| *** bhuvan has joined #openstack-infra | 18:04 | |
| jeblair | clarkb, fungi, mordred: the 72h freshness check was triggered the following number of times on these (UTC) days this week: M:441, T:293, W:160, H:111, F:81 (so far) | 18:07 |
| *** Ajaeger1 has joined #openstack-infra | 18:09 | |
| *** ramashri has quit IRC | 18:09 | |
| jeblair | 383/244/156/193/186 last week | 18:09 |
| fungi | welp, some progress with the centos6 image update issue in rax. if i rpm -e python-requests before applying the manifest, it no longer breaks | 18:10 |
| jeblair | the 24h check is not hit as much: 15/55/35/66/56 | 18:11 |
| *** MarkAtwood has quit IRC | 18:12 | |
| *** _nadya_ has quit IRC | 18:12 | |
| *** Longgeek_ has quit IRC | 18:12 | |
| Ajaeger1 | Good afternoon, how often is http://pypi.openstack.org updated? I'm not seeing the new version of openstack-doc-tools that was tagged four hours ago ;( | 18:13 |
| Ajaeger1 | It's up here: https://pypi.python.org/pypi/openstack-doc-tools/ | 18:13 |
| Ajaeger1 | as version 0.18 but not on http://pypi.openstack.org/simple/openstack-doc-tools/ | 18:14 |
| fungi | Ajaeger1: looks like bandersnatch may be stuck *again* | 18:14 |
| fungi | checking into it now | 18:14 |
| Ajaeger1 | Thanks fungi for checking | 18:14 |
| *** marun has joined #openstack-infra | 18:15 | |
| fungi | yeah, running since 13:05 utc | 18:15 |
| jeblair | fungi: does this happen often? | 18:15 |
| *** dimtruck is now known as zz_dimtruck | 18:15 | |
| *** annegentle has joined #openstack-infra | 18:16 | |
| fungi | jeblair: i think we've been running into it a couple times a week | 18:16 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add a Zuul trigger https://review.openstack.org/112411 | 18:16 |
| fungi | clarkb: mordred: ^ ? | 18:16 |
| jeblair | fungi: is there a bug? | 18:16 |
| jeblair | filed | 18:16 |
| jesusaurus | zaro: you seem to be an active jjb contributor, do you know how i can make https://review.openstack.org/#/c/114420/ pass? i dont really understand why deep_format is failing in that case | 18:16 |
| fungi | most of the time it's been issues on the cdn or pypi itself which bandersnatch is incapable of figuring out | 18:17 |
| *** paulrad has joined #openstack-infra | 18:17 | |
| YorikSar | fungi: Yay! :) | 18:17 |
| fungi | looking now to see whether this is a stale negative cache in the cdn or something else | 18:17 |
| *** marun has quit IRC | 18:17 | |
| YorikSar | fungi: They should definitely inherit oslo groups. | 18:17 |
| clarkb | I ran into it a couple times | 18:18 |
| fungi | last time it was broken, it was because there was a cached redirect for case insensitivity in the cdn but the destination was redirected | 18:18 |
| *** marun has joined #openstack-infra | 18:18 | |
| fungi | and ended up being a circular redirect | 18:18 |
| clarkb | it shouldnt be stale cache. mordred fixed that | 18:18 |
| YorikSar | fungi: Let me find out if someone else should be added.. | 18:18 |
| fungi | oh, right | 18:18 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Require openstack-doc-tools >= 0.18 https://review.openstack.org/114621 | 18:18 |
| fungi | strace shows it in a select loop | 18:19 |
| fungi | no obvious child processes | 18:19 |
| fungi | :q | 18:19 |
| fungi | heh, wrong window ;) | 18:19 |
| *** paulrad has quit IRC | 18:21 | |
| fungi | the mirror log from the cron job is updating with http://paste.openstack.org/show/95680/ but i think that's just new invocations dying because there's already one running? | 18:21 |
| fungi | TooManyRedirects: Exceeded 30 redirects. | 18:23 |
| fungi | maybe oligotyping? | 18:24 |
| *** bcrochet has quit IRC | 18:24 | |
| fungi | it looks like it was renamed from Oligotyping to oligotyping | 18:24 |
| *** hdd has joined #openstack-infra | 18:26 | |
| *** andreaf_ has joined #openstack-infra | 18:27 | |
| *** emagana has quit IRC | 18:28 | |
| fungi | i'll kill the stuck process and see if it's better now--those cached redirects can take up to 24 hours to expire, so if not i'll see if i can dig up what mordred (i think it was) did last week when this same issue arose | 18:28 |
| *** emagana has joined #openstack-infra | 18:28 | |
| *** terryw has joined #openstack-infra | 18:28 | |
| *** otherwiseguy has quit IRC | 18:28 | |
| mordred | fungi: if you run the cache invalidatino thing ... | 18:29 |
| fungi | mordred: was that working for cached redirect loops after a rename? | 18:29 |
| *** homeless has quit IRC | 18:29 | |
| mordred | yah | 18:29 |
| mordred | fungi: curl -X PURGE https://pypi.python.org/pypi/python-novaclient/json | 18:29 |
| mordred | for instance | 18:29 |
| YorikSar | fungi: Ok, now I have it. oslo-concurrency-ptl: oslo-ptl group and yorik-sar; oslo-concurrency-core: oslo-core group and yorik-sar. | 18:29 |
| mordred | fungi: can you see in the logs what it is that's failing? | 18:30 |
| *** andreaf has quit IRC | 18:30 | |
| fungi | mordred: yeah, "TooManyRedirects: Exceeded 30 redirects." and seems to have come when trying to delete Oligotyping and sync oligotyping | 18:30 |
| *** dustins has joined #openstack-infra | 18:30 | |
| *** homeless has joined #openstack-infra | 18:30 | |
| mordred | yeah - maybe try the purge on both? | 18:31 |
| fungi | mordred: i believe dstufft said this was a known issue with fastly caching case-sensitivity redirects and then a package getting renamed after it from one case to another | 18:31 |
| * anteaya has to go to lunch and and then spend the weekend in the mountains | 18:32 | |
| anteaya | tty Monday | 18:32 |
| Ajaeger1 | anteaya: you have to spend the weekend in the mountains? | 18:32 |
| Ajaeger1 | I would consider that something really great ;) | 18:32 |
| Ajaeger1 | anteaya: Enjoy! | 18:32 |
| fungi | so someone requested oligotyping from fastly, which pypi was redirecting to Oligotyping, and that redirect was cached. then Oligotyping was renamed to oligotyping on pypi. now requesting oligotyping gets redirected to Oligotyping which is redirected back to oligotyping ad infinitum | 18:33 |
| anteaya | Ajaeger1: thanks, oh I am looking forward to it | 18:33 |
| anteaya | Ajaeger1: rereading what I wrote it did come across as a chore didn't it | 18:33 |
| clarkb | jeblair your config changes for new zuul failed the layout test | 18:33 |
| anteaya | I didn't pick good words | 18:33 |
| *** emagana has quit IRC | 18:33 | |
| Ajaeger1 | anteaya: I wanted to interpret it this way ;) | 18:34 |
| *** emagana has joined #openstack-infra | 18:34 | |
| jeblair | clarkb: the zuul change may not have merged yet | 18:35 |
| fungi | mordred: yep, still breaking on that redirect loop, so i'll try to invalidate both versions of that package in fastly | 18:35 |
| clarkb | jeblair yup thats it | 18:35 |
| *** yamamoto has joined #openstack-infra | 18:36 | |
| jeblair | it's merged now so i'm rechecking | 18:36 |
| *** ramashri has joined #openstack-infra | 18:36 | |
| *** bcrochet has joined #openstack-infra | 18:36 | |
| *** nelsnelson has joined #openstack-infra | 18:36 | |
| fungi | i've tried purging oligotyping from the cdn first expecting that's sufficient. we'll see when the bandersnatch cron job re-fires in a couple minutes | 18:37 |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/nodepool: Refactor nodepool to configure glance only when uploading imaes https://review.openstack.org/114599 | 18:38 |
| *** nelsnelson has quit IRC | 18:39 | |
| *** praneshp has joined #openstack-infra | 18:39 | |
| fungi | jeblair: rustlebee: ha! coincidence? https://lists.debian.org/debian-devel-announce/2014/08/msg00007.html | 18:39 |
| *** nelsnelson has joined #openstack-infra | 18:40 | |
| rustlebee | heh, cool | 18:41 |
| yolanda | jeblair, submitted change again | 18:41 |
| yolanda | it reverts all the changes we did for client, and adds new method for configuring glance on uploadimage | 18:42 |
| *** otter768 has joined #openstack-infra | 18:44 | |
| *** cody-somerville has quit IRC | 18:46 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add Zuul merge-check to all projects https://review.openstack.org/114625 | 18:48 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove freshness check https://review.openstack.org/114626 | 18:48 |
| openstackgerrit | patrick-crews proposed a change to openstack-infra/config: Document crm114 for logstash https://review.openstack.org/114627 | 18:48 |
| *** Alexei_987 has joined #openstack-infra | 18:49 | |
| *** jistr|afk has quit IRC | 18:50 | |
| openstackgerrit | Aishwarya Thangappa proposed a change to openstack-infra/storyboard: Added a subscription_events table & subscription hander https://review.openstack.org/113016 | 18:51 |
| openstackgerrit | Aidan McGinley proposed a change to openstack-infra/jenkins-job-builder: Adds support for Change Assembly Version plugin https://review.openstack.org/114630 | 18:52 |
| *** praneshp has quit IRC | 18:54 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/nodepool: Refactor nodepool to configure glance only when uploading imaes https://review.openstack.org/114599 | 18:54 |
| *** hashar has joined #openstack-infra | 18:55 | |
| jeblair | i'm waiting about 5 more minutes, then i'm going to restart zuul | 18:56 |
| *** aysyd has quit IRC | 18:56 | |
| fungi | mordred: purging all four of {pypi,simple}/{O,o}ligotyping from fastly seems not to have solved the redirect loop | 18:57 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for test_postgresql_opportunistically bug 1355929 https://review.openstack.org/113581 | 18:58 |
| uvirtbot | Launchpad bug 1355929 in nova "test_postgresql_opportunistically fails in stable/havana due to: "ERROR: source database "template1" is being accessed by other users"" [Undecided,New] https://launchpad.net/bugs/1355929 | 18:58 |
| fungi | dstufft: did you happen to have other suggestions for workarounds there? | 18:58 |
| jeblair | yolanda: i left a comment on your change | 18:58 |
| clarkb | fungi that is unfortunate | 18:58 |
| fungi | i'll go ahead an leave an issue for bandersnatch upstream in case there isn't one already tracking this failure mode | 18:59 |
| jeblair | r1chardj0n3s_afk: ^ fyi | 18:59 |
| yolanda | jeblair, ok, just let it throw | 19:00 |
| *** prad has quit IRC | 19:00 | |
| jeblair | yolanda: yep; it will be propogated to the correct thread, and caught and logged there | 19:01 |
| yolanda | ok, i'll correct that | 19:01 |
| *** e0ne has joined #openstack-infra | 19:01 | |
| *** yolanda has left #openstack-infra | 19:01 | |
| *** yolanda has joined #openstack-infra | 19:02 | |
| *** e0ne has quit IRC | 19:03 | |
| *** flwang_ has joined #openstack-infra | 19:04 | |
| *** linkid has joined #openstack-infra | 19:04 | |
| dstufft | fungi: blech | 19:04 |
| dstufft | fungi: moment | 19:04 |
| dstufft | fungi: should be fixed now | 19:04 |
| fungi | dstufft: thanks--is there something we can do from our end the next time this happens? | 19:05 |
| *** zz_dimtruck is now known as dimtruck | 19:05 | |
| fungi | dstufft: and is it something that mirroring tools should be smart enough to work around? | 19:05 |
| jeblair | zuul/nodepool restarted | 19:05 |
| dstufft | fungi: if purging /simple/{O,o}ligotying didn't work, then no | 19:06 |
| dstufft | there's nothing you can do | 19:06 |
| dstufft | and nothing the mirroring tools can/should do | 19:06 |
| dstufft | I need to finish up my better purging stuff | 19:06 |
| zaro | jesusaurus: will take a look at the test failure. | 19:06 |
| fungi | dstufft: okay sp this really is only solvable between cheeseshop/fastly implementation and we need to suffer through until it's eventually rearchitected | 19:06 |
| dstufft | fungi: basically yea | 19:06 |
| *** paulrad has joined #openstack-infra | 19:06 | |
| fungi | dstufft: noted, and thanks | 19:06 |
| dstufft | so most likely remeditation involves yelling at me to go purge all | 19:07 |
| * fungi aborts his bandersnatch issue filing | 19:07 | |
| dstufft | or waiting | 19:07 |
| fungi | 24hrs timeout on those, right? | 19:07 |
| *** e0ne has joined #openstack-infra | 19:07 | |
| dstufft | yea | 19:07 |
| funzo | fungi: clarkb: do one of you have time to look at https://review.openstack.org/#/c/113333/ | 19:07 |
| *** flwang_ has quit IRC | 19:08 | |
| fungi | dstufft: thanks again! | 19:08 |
| jeblair | oops, there are some jobs that got hit with not_registered, probably due to lack of devstack-precise nodes at the time | 19:08 |
| dstufft | fungi: you can techincally do some tricksies to get the Fastly cache to see it as a seperate thing, but ti requires modifying bandernsatch and it's super hacky | 19:08 |
| dstufft | reworking the whole mirroring protocol si on my todo list at some point too | 19:08 |
| dstufft | a slightly specialized wget --mirror is not that great | 19:09 |
| fungi | dstufft: cool--i'll keep an eye out | 19:09 |
| *** annegentle has quit IRC | 19:09 | |
| *** annegentle has joined #openstack-infra | 19:10 | |
| *** shashankhegde has quit IRC | 19:10 | |
| fungi | funzo: your commit message is such a cliff-hanger | 19:11 |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/nodepool: Refactor nodepool to configure glance only when uploading imaes https://review.openstack.org/114599 | 19:11 |
| *** paulrad has quit IRC | 19:11 | |
| flashgordon | fungi: so the nova/neutron issues in the public clouds you are consuming | 19:11 |
| funzo | fungi: haha, I didn't realize that | 19:11 |
| funzo | but now I do | 19:11 |
| *** cody-somerville has joined #openstack-infra | 19:11 | |
| *** denis_makogon_ has joined #openstack-infra | 19:12 | |
| flashgordon | fungi: are those still the issue where an instance errors out and you cannot delete it, or is something else now? | 19:12 |
| funzo | fungi: allow me to fixy fix | 19:12 |
| dstufft | fungi: fixing packaging http://i.minus.com/ibaDjk7AeIcvxv.gif | 19:12 |
| *** denis_makogon has quit IRC | 19:12 | |
| *** denis_makogon_ is now known as denis_makogon | 19:12 | |
| openstackgerrit | calfonso proposed a change to openstack-infra/config: Run Tempest tests on Fedora 20 with the Docker driver https://review.openstack.org/113333 | 19:12 |
| fungi | flashgordon: one of them, yes, though not being able to delete them is more of a symptom/subsequent bug (though it's the part that's directly impacting us in that particular provider) | 19:12 |
| *** dmakogon_ has joined #openstack-infra | 19:13 | |
| fungi | flashgordon: for the provider themselves, they're interested in figuring out *why* neutron fails the port removal request during instance deletion | 19:13 |
| flashgordon | fungi: right this is a secondary bug of sorts | 19:13 |
| flashgordon | fungi: but the providers aren't on a new enough version of openstack that has the fix I landed? | 19:14 |
| fungi | your patch to make the instance deletable anyway is in their pending queue for their next upgrade apparently, at which point it stops affecting us | 19:14 |
| flashgordon | fungi: ahh cool | 19:14 |
| flashgordon | just wanted to make sure there wasn't another bug like that that needs fixing | 19:15 |
| flashgordon | (bugs blocking you from deleting | 19:15 |
| jeblair | zuul appears to be doing it's thing, so i'm going to grab some lunch | 19:15 |
| jeblair | after i return, i'll look at merging the zuul config changes | 19:15 |
| fungi | flashgordon: but the other issue which was particularly causing problems over the last week or so is that sometimes floating ip requests fail. there's work in progress to make nodepool work around it, i believe, and prevent that from eating too far into our floating ip quota, but that still implies another underlying nova/neutron integration issue in their environment i think | 19:15 |
| flashgordon | fungi: yeah I think arosen has a patch up on that | 19:16 |
| flashgordon | trying to find it | 19:16 |
| *** baoli has quit IRC | 19:16 | |
| clarkb | jeblair sounds good | 19:16 |
| fungi | flashgordon: right, i think it was under discussion in the neutron channel earlier in the week | 19:16 |
| *** baoli has joined #openstack-infra | 19:16 | |
| clarkb | it wont fix things... | 19:17 |
| clarkb | it makes the nodepool code less redundant with novaclient | 19:17 |
| clarkb | but nova client does disassociate then delete | 19:17 |
| fungi | dstufft: btw your global purge seems to have gotten our bandersnatch back on track--thank you again | 19:17 |
| flashgordon | fungi: ahh https://review.openstack.org/#/c/113754/ | 19:17 |
| clarkb | if the first thing succeeds and the second fails we will leak the ip | 19:18 |
| fungi | Ajaeger1: http://pypi.openstack.org/simple/openstack-doc-tools/ is updated now--sorry about the delay! | 19:18 |
| Ajaeger1 | fungi, thanks a lot! | 19:18 |
| dstufft | fungi: np, sorry it was required at all :/ | 19:18 |
| flashgordon | clarkb: ahh, any way to make novaclient smarter | 19:18 |
| fungi | Ajaeger1: the thanks on this one go to dstufft | 19:18 |
| Ajaeger1 | dstufft: thanks a lot! | 19:19 |
| flashgordon | clarkb: https://review.openstack.org/#/c/113754/ is a nova patch not novaclient | 19:19 |
| * fungi gets back to banging his head against centos | 19:19 | |
| clarkb | flashgordon imo no. I think this is an api bug. delete should imply disassociate in an atomic manner | 19:19 |
| clarkb | I was talking about a nodepool patch | 19:19 |
| *** ramashri has quit IRC | 19:19 | |
| clarkb | I hadnt seen that patch. looking now | 19:20 |
| flashgordon | clarkb: ahh thanks | 19:20 |
| clarkb | looks like arosen is doing an api update there so that would fix it | 19:21 |
| flashgordon | clarkb: hopefully | 19:21 |
| *** paulrad has joined #openstack-infra | 19:23 | |
| openstackgerrit | Joel Coffman proposed a change to openstack/requirements: Add pykmip https://review.openstack.org/114037 | 19:24 |
| yolanda | wow, big check queue now | 19:25 |
| *** hdd has joined #openstack-infra | 19:26 | |
| *** paulrad has quit IRC | 19:27 | |
| fungi | earlier in the week it was ~350. not so big by comparison | 19:28 |
| *** maxbit has quit IRC | 19:29 | |
| fungi | pretty high for a friday though | 19:29 |
| *** shashankhegde has joined #openstack-infra | 19:30 | |
| *** hdd has quit IRC | 19:30 | |
| yolanda | clarkb, jeblair, disconnecting for a while, but if you have some comments about my change i can read the comments on it | 19:30 |
| *** cody-somerville has quit IRC | 19:31 | |
| clarkb | yolanda ok thank you | 19:32 |
| *** jcoufal has quit IRC | 19:34 | |
| *** Hefeweizen has joined #openstack-infra | 19:34 | |
| annegentle | hey clarkb, I'm trying to chase down a doc build problem with "db4-upgrade.xsl not found" and wondered how to know if the switch to Trusty affects a particular job? | 19:34 |
| annegentle | clarkb: gate-identity-api-tox-doc-publish-checkbuild is the job | 19:34 |
| annegentle | clarkb: basically the markdown gets converted to docbook then gets built, and it's the second build that's failing with db4-upgrade.xsl not found | 19:35 |
| daneleblanc_ | Greetings all, we have an 3rd party CI account which is no longer used which should be deleted. What do I need to do? | 19:36 |
| fungi | daneleblanc_: i can deactivate the account--which one is it? also why is it being retired (out of curiosity)? | 19:36 |
| *** yamamoto has quit IRC | 19:36 | |
| daneleblanc_ | Account is: 9270 cisco-openstack-ci Cisco Unknown Function CI cisco-openstack-ci@external.cisco.com | 19:37 |
| daneleblanc_ | This account was used for some testing that our COI (CIsco OpenStack Installer) group was doing that is no longer needed. | 19:39 |
| *** hashar is now known as hashSpeleology | 19:39 | |
| daneleblanc_ | We have another account "Cisco CI" which is active for our Neutron plugin/driver testing. | 19:39 |
| daneleblanc_ | I've confirmed with the former owners of the old account that they're not using it. | 19:40 |
| fungi | daneleblanc_: okay, cool. if you have no need of account 9270 any longer, i'll go ahead and disable it. thanks for the heads up! | 19:40 |
| daneleblanc_ | fungi: Thanks very much! | 19:40 |
| fungi | daneleblanc_: if there were any wiki pages created for that account, they should be retired/removed as well | 19:41 |
| *** yamamoto has joined #openstack-infra | 19:41 | |
| *** Ish__ has joined #openstack-infra | 19:41 | |
| daneleblanc_ | fungi: I'll check re. wiki pages. Thanks. | 19:41 |
| *** Ish__ has quit IRC | 19:41 | |
| fungi | hogepodge: if you're around, i have an interesting puppet debugging puzzle... we're hitting some sort of rpmdb concurrency problem i think, as exhibited on line 2409 of http://pastebin.com/wRX8B8ED and i can reproduce it on a fresh CentOS 6.5 (PVHVM) instance in rackspace by running these commands http://paste.openstack.org/show/95656/ | 19:44 |
| hogepodge | fungi ok | 19:44 |
| fungi | hogepodge: if i rpm -e cloud-init python-requests before applying that manifest, the problem goes away | 19:44 |
| annegentle | fungi: I pinged clarkb on the Trusty environment question since he sent the note to the mailing list, but do you know if db4-upgrade.xsl not found could be caused by 14.04 enviro? | 19:45 |
| fungi | hogepodge: i think the issue is arising because we ensure an rpm is removed, ensure the equivalent pip package is present, and then at the next yum invocation it goes sideways presumably because it still briefly thinks the original package we removed is still installed, but the files have now changed | 19:46 |
| hogepodge | https://bugzilla.redhat.com/show_bug.cgi?id=973375 | 19:46 |
| uvirtbot | hogepodge: Error: Could not parse XML returned by bugzilla.redhat.com: HTTP Error 404: Not Found | 19:46 |
| fungi | hogepodge: ha! i saw that bug and didn't connect the dots until just now | 19:47 |
| fungi | hogepodge: is there any good way to insert a cache clearing checkpoint (stuff it into the before chain as an exec maybe)? | 19:48 |
| hogepodge | The workaround looks yuck | 19:48 |
| *** yamamoto has quit IRC | 19:48 | |
| hogepodge | Yeah, that’s how I would do it. Add a dependency. You could use the spaceship operator to make sure the cache is cleared before all packages. | 19:49 |
| fungi | annegentle: possibly. have a link to a failure log or something i can get more context on, or does it happen on every run of gate-identity-api-tox-doc-publish-checkbuild right now? | 19:49 |
| annegentle | fungi: yep, just a sec | 19:50 |
| fungi | hogepodge: thanks! i have a suspicion just clearing it after we remove the rpm and before we replace it with the pip package ought to do it in that case | 19:50 |
| annegentle | fungi: clarkb: http://logs.openstack.org/98/113998/4/check/gate-identity-api-tox-doc-publish-checkbuild/71a1715/console.html | 19:50 |
| fungi | annegentle: just a sec and i'll see what i can spot | 19:51 |
| *** ruhe has quit IRC | 19:51 | |
| annegentle | fungi: thanks! | 19:51 |
| *** NikitaKonovalov has quit IRC | 19:51 | |
| *** dmitryme has quit IRC | 19:52 | |
| *** SergeyLukjanov has quit IRC | 19:52 | |
| *** DinaBelova has quit IRC | 19:52 | |
| openstackgerrit | afazekas proposed a change to openstack-infra/elastic-recheck: Add query for 1357476 https://review.openstack.org/114610 | 19:52 |
| *** dmitryme has joined #openstack-infra | 19:55 | |
| *** e0ne has quit IRC | 19:55 | |
| *** chmouel has quit IRC | 19:57 | |
| *** DinaBelova has joined #openstack-infra | 19:58 | |
| *** chmouel has joined #openstack-infra | 19:58 | |
| *** e0ne has joined #openstack-infra | 19:59 | |
| fungi | annegentle: so just to clarify, that job ran on an ubuntu precise worker. are you asking whether switching it to trusty might fix it? | 19:59 |
| *** SergeyLukjanov has joined #openstack-infra | 20:00 | |
| *** tonytan4ever has quit IRC | 20:00 | |
| annegentle | fungi: hm, ok, it doesn't run for me locally, I get the same error | 20:01 |
| *** tonytan4ever has joined #openstack-infra | 20:01 | |
| annegentle | fungi: and I thought I remembered it as an xml enviro error | 20:01 |
| zaro | jesusaurus: reviewed your jjb failure. | 20:01 |
| fungi | annegentle: the java stacktrace implies that the xml file is truncated (maybe malformed in some way?) | 20:02 |
| *** hdd has joined #openstack-infra | 20:02 | |
| *** NikitaKonovalov has joined #openstack-infra | 20:02 | |
| annegentle | fungi: ok, one thing Ajaeger1 found was that using < instead of angle brackets avoids the error. Hm | 20:02 |
| *** ruhe has joined #openstack-infra | 20:03 | |
| fungi | annegentle: specifically noticing "...identity-api-v3.xml: Premature end of file." | 20:03 |
| Ajaeger1 | annegentle: it needs to be marked as "verbatim" - intented by four spaces | 20:03 |
| *** unicell has quit IRC | 20:03 | |
| *** dangers is now known as dangers_away | 20:03 | |
| annegentle | ah four spaces does it? | 20:03 |
| Ajaeger1 | pandoc fails on the file and creates an empty one - also locally for me. | 20:03 |
| annegentle | ok! | 20:03 |
| Ajaeger1 | annegentle: at least for me- pushed a new revision, let's see whether Jenkins likes it as well ;) | 20:04 |
| annegentle | fingers crossed! | 20:04 |
| krotscheck | zaro: https://review.openstack.org/#/c/114383/ | 20:05 |
| *** tonytan4ever has quit IRC | 20:05 | |
| jesusaurus | zaro: that test i added is something that i want to make pass but i dont understand why the current deep_format isnt formatting that builder | 20:06 |
| Ajaeger1 | annegentle: where do you see "db4-upgrade.xsl not found"? | 20:06 |
| *** hdd has quit IRC | 20:06 | |
| *** ramashri has joined #openstack-infra | 20:06 | |
| *** DinaBelova has quit IRC | 20:07 | |
| annegentle | Ajaeger1: locally when I run tox -echeckbuild | 20:07 |
| Ajaeger1 | annegentle: might it be that you don't have everything installed? Not sure in which package db4-upgrade.xsl is... | 20:08 |
| Ajaeger1 | annegentle: on openSUSE it's part of docbook_5-5.0-71.1.2.noarch | 20:09 |
| *** DinaBelova has joined #openstack-infra | 20:09 | |
| *** bknudson has quit IRC | 20:09 | |
| *** notmyname has quit IRC | 20:10 | |
| *** chmouel has quit IRC | 20:10 | |
| *** mriedem1 has joined #openstack-infra | 20:11 | |
| *** notmyname has joined #openstack-infra | 20:11 | |
| *** mriedem1 has quit IRC | 20:11 | |
| *** mriedem1 has joined #openstack-infra | 20:12 | |
| *** mriedem has quit IRC | 20:13 | |
| *** e0ne has quit IRC | 20:15 | |
| *** paulrad has joined #openstack-infra | 20:15 | |
| *** chmouel has joined #openstack-infra | 20:15 | |
| salv-orlando | fungi, clarkb: after the latest fixes the neutron full job failure rate is around 6% - can we switch it to voting before some patch break it again? | 20:17 |
| salv-orlando | patch —> https://review.openstack.org/#/c/88289/ | 20:17 |
| *** cody-somerville has joined #openstack-infra | 20:17 | |
| *** garyh has joined #openstack-infra | 20:19 | |
| *** e0ne has joined #openstack-infra | 20:19 | |
| *** paulrad has quit IRC | 20:19 | |
| *** annegentle has quit IRC | 20:20 | |
| YorikSar | fungi: (or anyone in infra-core) Please set up oslo-concurrency-core and -ptl groups. | 20:20 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bin/pip suddenly missing https://review.openstack.org/111361 | 20:20 |
| clarkb | whats the difference between the two tempest _FULL vars? | 20:20 |
| clarkb | one is testr full and the other is just full | 20:20 |
| *** tonytan4ever has joined #openstack-infra | 20:21 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for 1357476 https://review.openstack.org/114610 | 20:21 |
| fungi | clarkb: looks like DEVSTACK_GATE_TEMPEST_TESTR_FULL is used in a couple existing jobs too | 20:22 |
| *** emagana has quit IRC | 20:23 | |
| *** emagana has joined #openstack-infra | 20:23 | |
| openstackgerrit | Steve Varnau proposed a change to openstack-infra/zuul: Update gerrit change attributes even if merged https://review.openstack.org/114102 | 20:24 |
| fungi | clarkb: seems to run tox -etestr-full instead of tox -efull | 20:24 |
| clarkb | mtreinish can you weigh in here? | 20:24 |
| *** paulrad has joined #openstack-infra | 20:25 | |
| clarkb | otherwise I am happy to change the job | 20:25 |
| fungi | unless i'm missing something, those two tox envs are identical? | 20:25 |
| fungi | both seem to just run: bash tools/pretty_tox.sh '(?!.*\[.*\bslow\b.*\])(^tempest\.(api|scenario|thirdparty|cli)) {posargs}' | 20:26 |
| fungi | i wonder if this is left over from the tempest testr transition? | 20:26 |
| openstackgerrit | A change was merged to openstack-infra/config: Add source entries for all Zuul pipelines https://review.openstack.org/114615 | 20:27 |
| *** garyh has quit IRC | 20:27 | |
| *** annegentle has joined #openstack-infra | 20:27 | |
| *** emagana has quit IRC | 20:28 | |
| clarkb | oh could be | 20:28 |
| fungi | i suspect all current use of DEVSTACK_GATE_TEMPEST_TESTR_FULL can be switched to DEVSTACK_GATE_TEMPEST_TESTR_FULLDEVSTACK_GATE_TEMPEST_FULL, then the logic ripped out of devstack-gate and tempest's tox.ini | 20:28 |
| lifeless | fungi: how did the new centos slaves in rax go ?? | 20:28 |
| lifeless | fungi: OOI what testr transition? | 20:28 |
| jesusaurus | zaro: oh, so my failure is expected behaviour? and builders with parameters need those parameters passed in every time they are used? | 20:28 |
| fungi | lifeless: i think i have a solution, now i'm trying to figure out how to puppet it in a platform-dependent manner. we're apparently hitting https://bugzilla.redhat.com/show_bug.cgi?id=973375 | 20:28 |
| uvirtbot | fungi: Error: Could not parse XML returned by bugzilla.redhat.com: HTTP Error 404: Not Found | 20:28 |
| openstackgerrit | A change was merged to openstack-infra/config: Add merge-check pipeline https://review.openstack.org/114616 | 20:29 |
| lifeless | no snacks for you, uvirtbot | 20:29 |
| *** paulrad has quit IRC | 20:29 | |
| *** emagana has joined #openstack-infra | 20:29 | |
| lifeless | fungi: fun race | 20:30 |
| fungi | lifeless: as for testr transition, talking about the (a while back) transition tempest made to run via testr rather than nose | 20:30 |
| clarkb | NOTABUG | 20:30 |
| clarkb | of course | 20:30 |
| fungi | clarkb: any bug i find hard to fix i just mark notabug. much easier that way ;) | 20:30 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for neutron subnet delete race bug 1357055 https://review.openstack.org/114359 | 20:30 |
| uvirtbot | Launchpad bug 1357055 in nova "Race to delete shared subnet in Tempest neutron full jobs" [Undecided,New] https://launchpad.net/bugs/1357055 | 20:30 |
| clarkb | salv-orlando: can I ask for one tiny little change to that change? | 20:32 |
| *** dustins has quit IRC | 20:32 | |
| fungi | salv-orlando: i'm good to +2 it again once clarkb is satisfied | 20:32 |
| clarkb | salv-orlando: fungi see inline comment | 20:32 |
| fungi | YorikSar: yep, i'll knock it out now, sorry about the lag | 20:32 |
| clarkb | but once thats in lets do it | 20:32 |
| salv-orlando | fungi, clarkb : I’ll do anything you ask! (as long as it’s fairly legal) | 20:33 |
| YorikSar | fungi: np, I see you're busy. Just reminding so that you won't forget about it :) | 20:34 |
| fungi | jeblair: mordred: clarkb: to work around the rpmdb cache/corruption issue on centos6 in rax, i can via some convoluted platform-dependent puppetry to run 'yum clean rpmdb' around the troublesome package operations, or i can avoid it by preemptively removing the packages in a nodepool prep script or the install_puppet.sh... preferences? | 20:34 |
| *** Ish__ has joined #openstack-infra | 20:35 | |
| clarkb | fungi: this is removing python-requests so that pip will install it? | 20:35 |
| lifeless | that bug appears very easy to fix | 20:35 |
| clarkb | lifeless: but it isn't a bug | 20:35 |
| fungi | clarkb: yeah | 20:35 |
| clarkb | fungi: I think we should do it all in puppet since this isn't a bootstrap package | 20:36 |
| lifeless | change <= to < and sleep to the border of the current second before closing the cache | 20:36 |
| fungi | clarkb: well, it's arguably a bootstrap problem | 20:36 |
| clarkb | fungi: not really. It isn't a package necessary to run puppet | 20:37 |
| fungi | clarkb: we only see it in rax because they preinstall cloud-init which in turn pulls in python-requests | 20:37 |
| clarkb | fungi: we currently only install packages in the scripts to do that | 20:37 |
| fungi | true | 20:37 |
| clarkb | then puppet does the rest | 20:37 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bin/pip suddenly missing https://review.openstack.org/111361 | 20:37 |
| fungi | clarkb: so... can i spaceship around an ordering checkpoint but have it be a no-op except on rpm-based platforms? | 20:38 |
| *** signed8bit has joined #openstack-infra | 20:38 | |
| jeblair | fungi: that all seems weird; is actually getting rpm to work correctly a dead end? | 20:38 |
| fungi | so that we can be sure if on centos6 we clear the rpmdb cache before puppet does any other packaeg operations which might try to use that cache | 20:38 |
| jeblair | have you asked ianw or other folks about it? | 20:39 |
| *** paulrad has joined #openstack-infra | 20:39 | |
| fungi | jeblair: red hat seems to feel that *is* working correctly | 20:39 |
| fungi | jeblair: good point... ianw: dprince: ^ ? | 20:39 |
| lifeless | well, they've decided that its a better tradeoff this way | 20:39 |
| jeblair | fungi: then perhaps we should do nothing? :) | 20:39 |
| clarkb | fungi: ya I think you can do an exec before the package resource for libffi and clear the cache | 20:39 |
| lifeless | thats not quite the same thing :) | 20:39 |
| clarkb | fungi: or force the pip install to happen last | 20:39 |
| fungi | clarkb: well, the package resource for libffi there is arbitrary. it's really "whatever you try to install next via yum will fail" | 20:40 |
| harlowja | qq, if someone gets a free second, i tagged doc8 and pushed a signed tag up to gerrit earlier today, doc8 afaik has the publish-to-pypi jobs, but i'm not seeing it showup on pypi, the doc8 pypi page has openstackci as maintainer and all, so seems like thats fine, anyone know what could be up? | 20:40 |
| fungi | clarkb: so i need to make sure after we ensure=>absent on python-requests we clear the rpmdb cache before any other yum activities puppet might try (which we can't predict since ordering is going to be potentially variable) | 20:41 |
| jeblair | harlowja: will check | 20:41 |
| salv-orlando | clarkb: re your comment there is already a patch for fixing that -> https://review.openstack.org/#/c/114283 | 20:42 |
| harlowja | jeblair thx much | 20:42 |
| clarkb | salv-orlando: that wfm thanks | 20:42 |
| lifeless | oh another way to fix it would be to write a 'invalid' file when the db changes | 20:42 |
| salv-orlando | cool | 20:42 |
| *** david-ly_ has joined #openstack-infra | 20:43 | |
| dprince | fungi: calling 'yum clean all' or rpmdb doesn't seem to be a horrible solution to me | 20:43 |
| lifeless | and when the cache is written remove it at the time the cache is opened | 20:43 |
| clarkb | salv-orlando: +2 on both changes | 20:43 |
| clarkb | haven't approved the first because eyes are good on it but I want to approve it :) | 20:43 |
| *** paulrad has quit IRC | 20:43 | |
| salv-orlando | clarkb: yay!!!! | 20:43 |
| fungi | dprince: yeah, i wonder if the puppet package provider shouldn't just clean it after every remove action to avoid that issue | 20:44 |
| fungi | dprince: but i'd rather we didn't continue making a habit of reimplementing package providers | 20:44 |
| jeblair | harlowja: http://logs.openstack.org/47/47281acf5028fb4cf376eb1076b52f2ab5cc0e46/release/doc8-tarball/01c6a9d/console.html | 20:44 |
| harlowja | jeblair ah, poopy | 20:45 |
| jeblair | harlowja: an unfortunate, hopefully transient, failure. i'll retrigger the job | 20:45 |
| harlowja | jeblair thx :) | 20:45 |
| *** david-lyle has quit IRC | 20:45 | |
| clarkb | jeblair: fungi mtreinish https://review.openstack.org/#/c/88289/ shall we do it? | 20:46 |
| fungi | clarkb: too late | 20:47 |
| clarkb | fungi: woot | 20:47 |
| fungi | unless you feel obligated to unapprove/-2 ;) | 20:47 |
| *** e0ne has quit IRC | 20:47 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for a rescue related failure https://review.openstack.org/112549 | 20:47 |
| *** pdmars has quit IRC | 20:47 | |
| openstackgerrit | A change was merged to openstack-infra/config: Linkify storyboard stories in gerrit https://review.openstack.org/105895 | 20:48 |
| *** pdmars has joined #openstack-infra | 20:48 | |
| clarkb | fungi: puppet has runlevels or whatever they call them | 20:49 |
| fungi | YorikSar: i added the requested initial members to those groups | 20:49 |
| clarkb | fungi: we could put that set of operations in an earler run stage | 20:49 |
| clarkb | I think stage is the word I was looking for | 20:49 |
| openstackgerrit | A change was merged to openstack-infra/config: Git rev-list --max-count needs an = for assignment https://review.openstack.org/106174 | 20:51 |
| clarkb | fungi: https://docs.puppetlabs.com/puppet/latest/reference/lang_run_stages.html | 20:51 |
| *** e0ne has joined #openstack-infra | 20:51 | |
| fungi | they do *seem* an awful lot like system v runlevels though, so i got what you meant | 20:51 |
| *** rwsu has quit IRC | 20:51 | |
| zaro | jesusaurus: yes, my suggest make sense? | 20:52 |
| *** dprince has quit IRC | 20:52 | |
| *** pdmars has quit IRC | 20:52 | |
| pleia2 | oof, work desktop went sideways this morning, hopefully back to business now | 20:52 |
| clarkb | pleia2: oh no | 20:53 |
| clarkb | did you take a hammer to it? | 20:53 |
| pleia2 | ext4 became corrupted on my SSD :\ reinstalled the OS (fortunately all my LVs are on the spinning drive) | 20:53 |
| pleia2 | SSD may still be a problem, since ext4 is pretty resiliant, we'll see | 20:53 |
| fungi | clarkb: so i could just wrap a tiny class around removing python-requests and cleaning the rpmdb cache and then assign that to a stage which i create to execute before main? | 20:53 |
| clarkb | fungi: yup exactly | 20:54 |
| clarkb | then you know it will run before all other possible yum operations | 20:54 |
| jesusaurus | zaro: i guess so, i just expected the builder macro to be able to get its parameters from the job calling it without the job having to pass them in to the builder | 20:54 |
| clarkb | nibalizer: ^ do you have a better suggestion? | 20:54 |
| fungi | clarkb: that seems reasonable, for definitions of reasonable which don't leak into my weekend | 20:54 |
| jeblair | fungi, clarkb: are we seriously considering this as a solution? | 20:54 |
| jeblair | fungi, clarkb: i mean, is there something particular about this set of packages that makes us think that this is the only way this can happen? | 20:55 |
| clarkb | jeblair: I am not sure I understand your question. THis is a yum/rpm interaction breaking us specific to how puppet works | 20:55 |
| clarkb | if we want to remove one package and install another with puppet I think we need to | 20:56 |
| jeblair | clarkb: can you point me to the bug report? | 20:56 |
| clarkb | jeblair: https://bugzilla.redhat.com/show_bug.cgi?id=973375 | 20:56 |
| uvirtbot | clarkb: Error: Could not parse XML returned by bugzilla.redhat.com: HTTP Error 404: Not Found | 20:56 |
| fungi | jeblair: the specific failure mode seems to be "1. remove a system package, 2. install something which replaces its files, 3. perform some other system package operation -> *boom* yum complains that the rpmdb is corrupt because it's looking at a stale cache" | 20:56 |
| *** bknudson has joined #openstack-infra | 20:56 | |
| *** nelsnelson has quit IRC | 20:56 | |
| clarkb | another alternatiev is to have nodepool scripts remove the package for us with yum to keep the cace consistent if I understand this properly | 20:57 |
| *** bknudson1 has joined #openstack-infra | 20:57 | |
| nibalizer | oh got please lets not use stages | 20:57 |
| *** SumitNaiksatam has quit IRC | 20:57 | |
| clarkb | nibalizer: is there another way to perform an operation before all other operations? | 20:57 |
| *** crc32 has joined #openstack-infra | 20:58 | |
| nibalizer | i'd like to understand the problem better | 20:58 |
| fungi | options i'm aware of include forcing the rpmdb cache to be cleared between these operations, or removing the offending packages before we apply the puppet manifest (again this is a "rackspace makes their own images and puts things on them which are not in the upstream cloud images" issue we're running into) | 20:58 |
| nibalizer | clarkb: there is a way to do that yea | 20:58 |
| *** nelsnelson has joined #openstack-infra | 20:59 | |
| clarkb | nibalizer: pupept installs pacakges with yum and removes them with rpm. This makes the rpmdb cache inconcistent across those tools | 20:59 |
| nibalizer | so we need to exec a clean and remove a package? | 20:59 |
| clarkb | nibalizer: so if you install a package from pip that writes to where those files were yum complains | 20:59 |
| clarkb | nibalizer: and we have to do that exec to clean the cache before any additional yum operations happen | 20:59 |
| jeblair | apparently the best solution, according to the bugzilla, is to run 'yum clean rpmdb' after running 'rpm -e'; i'm assuming that's a change to the puppet rpm/yum package provider | 21:00 |
| clarkb | nibalizer: oh I see if we combine the two things into a single exec we can enforce the order we want | 21:00 |
| *** bknudson has quit IRC | 21:01 | |
| clarkb | heck if we just exec yum to remove the packae that should work too | 21:01 |
| *** Ajaeger1 has quit IRC | 21:01 | |
| clarkb | nevermind cleaning the cache | 21:01 |
| fungi | probably--while i suspect that would work i haven't tested the theory yet | 21:01 |
| fungi | i'll give that a shot next and make sure it's also a viable option | 21:01 |
| nibalizer | clarkb: can you force puppet to remove with yum by specifying provider => yum in the package stanza? | 21:02 |
| clarkb | nibalizer: no idea | 21:02 |
| * clarkb reads some ruby | 21:02 | |
| *** garyh has joined #openstack-infra | 21:03 | |
| nibalizer | clarkb: steps to reproduce this locally? | 21:04 |
| openstackgerrit | A change was merged to openstack-infra/config: Make neutron full job voting https://review.openstack.org/88289 | 21:04 |
| clarkb | nibalizer: remove python-requests with rpm, pip install requests, yum install libffi-dev | 21:04 |
| *** david-ly_ is now known as david-lyle | 21:05 | |
| *** flwang_ has joined #openstack-infra | 21:05 | |
| clarkb | nibalizer: you may need to yum install python-requests first depending on your base image | 21:05 |
| nibalizer | okay ill try to go do this | 21:05 |
| clarkb | nibalizer: the yum package provider does not appear to support uninstall | 21:05 |
| clarkb | nibalizer: the rpm provider does | 21:05 |
| fungi | SO... i have one other possibly elegant solution which would allow us to sidestep the entire issue i think | 21:06 |
| clarkb | there is a purge | 21:06 |
| *** terryw has quit IRC | 21:06 | |
| nibalizer | clarkb: :?/ | 21:06 |
| nibalizer | :(< | 21:07 |
| clarkb | so I think we can purge the package | 21:07 |
| openstackgerrit | A change was merged to openstack-infra/config: Use proper name for neutron full job in gate queue https://review.openstack.org/114283 | 21:07 |
| clarkb | fungi: ^ that might be "easy" | 21:07 |
| *** radez is now known as radez_g0n3 | 21:07 | |
| clarkb | but will also remove cloud init | 21:07 |
| fungi | according to git history, the reason we're installing requests on our slaves is twofold... because we use it for uploading to swift and for twine which uploads to pypi | 21:07 |
| clarkb | yup | 21:08 |
| clarkb | so we need a relatively new requests version | 21:08 |
| clarkb | the twine uploads can happen in a virtalenv though. So really its the swift uploads | 21:08 |
| fungi | of the two, only twine needs a newer version than is present in precise, so it's why we started trying to replace the package with a pip version | 21:08 |
| fungi | yeah, so you see where this is heading | 21:08 |
| clarkb | yup | 21:08 |
| *** flwang_ has quit IRC | 21:09 | |
| clarkb | we can try ensure => purged instaed of abset | 21:09 |
| clarkb | but ya that may break a bunch of stuff like cloud init | 21:09 |
| fungi | options are to either special-case the long-running worker where twine gets run and install a newer requests only there. it's ubuntu and so won't exhibit this rpmdb cache weirdness | 21:09 |
| fungi | or to run twine in a virtualenv, as you pointed out | 21:10 |
| clarkb | jeblair: for https://review.openstack.org/#/c/111094/ do you want me to make those two changes and make the possibly use ironic comment a bit more verbose? | 21:10 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/puppet-storyboard: StoryBoard Webclient now auto-updates again. https://review.openstack.org/114383 | 21:12 |
| *** _nadya_ has joined #openstack-infra | 21:12 | |
| *** annegent_ has joined #openstack-infra | 21:13 | |
| jeblair | clarkb: oh i just learned about a relevant change, 1 sec | 21:13 |
| jeblair | clarkb: check out https://review.openstack.org/#/c/112134/ and see if it changes things | 21:13 |
| clarkb | looking | 21:14 |
| *** garyh has quit IRC | 21:14 | |
| *** e0ne has quit IRC | 21:14 | |
| *** annegentle has quit IRC | 21:14 | |
| *** mfer has quit IRC | 21:14 | |
| *** leakypipes has quit IRC | 21:15 | |
| *** atiwari has quit IRC | 21:16 | |
| clarkb | jeblair: ya so that change would make it so that all ironic jobs use the ironic virt driver | 21:16 |
| clarkb | jeblair: and my comment can be stronge rand say use ironic | 21:16 |
| *** _nadya_ has quit IRC | 21:16 | |
| fungi | clarkb: i tried the exec command => '/usr/bin/yum uninstall -y python-requests' suggestion in place of package ensure => absent and it seems to work around the problem (as long as we make that a red hat family affair only via conditional) | 21:17 |
| clarkb | fungi: cool | 21:18 |
| harlowja | jeblair u retriggered that doc8 job right (still seems like not updated on pypi) | 21:18 |
| mordred | fungi: I would like to make a new yum provider for puppet | 21:19 |
| mordred | fungi: the current one is clearly very poor | 21:19 |
| clarkb | mordred: given how that has gone with pip I am -1 | 21:19 |
| clarkb | mordred: it is very hard to merge changes like that | 21:19 |
| fungi | mordred: without knowing their reasons for the current implementation i'm also somewhat worried we'll end up rediscovering why they've made the current set of compromises | 21:20 |
| mordred | clarkb: ah - I was thinking overwriting the current one - not making a parallel one | 21:20 |
| clarkb | mordred: its the same problem | 21:20 |
| mordred | it is? | 21:20 |
| clarkb | mordred: there isn't a good way to ease it in. Its a one shot and go | 21:20 |
| *** SumitNaiksatam has joined #openstack-infra | 21:21 | |
| *** lascii is now known as alaski | 21:21 | |
| *** homeless has quit IRC | 21:21 | |
| fungi | i can imagine contributing improvements (or a rewrite) to puppet stdlib, but would hate for us to just carry a local yum package provider indefinitely (even a weapper around the stdlib one for that matter) | 21:21 |
| mordred | well, my other suggestion is to just start ripping out puppet since it can't properly install packages using either yum or pip | 21:21 |
| fungi | s/weapper/wrapper/ | 21:21 |
| mordred | but that may just me being snarky | 21:21 |
| notmyname | FYI swift team (portante and acoles and peluse) just clicked approve on v3 auth support for swiftclient https://review.openstack.org/#/c/91788/ | 21:21 |
| nibalizer | i really do think i can probably find a non-gross way to do tis | 21:22 |
| clarkb | mordred: my reaction was to stop using rpm based distros >_> (yum as a few problematic bugs that are O_O) | 21:22 |
| nibalizer | i do need a bit of time though | 21:22 |
| clarkb | nibalizer: the exec is not gross | 21:22 |
| jeblair | harlowja: yeah, the tarball job finished: http://logs.openstack.org/47/47281acf5028fb4cf376eb1076b52f2ab5cc0e46/release/doc8-tarball/01c6a9d/console.html | 21:22 |
| *** ominakov has quit IRC | 21:22 | |
| jeblair | harlowja: now i need to trigger the upload job | 21:22 |
| *** crc32 has quit IRC | 21:22 | |
| harlowja | jeblair ah, thx | 21:22 |
| clarkb | mordred: it doesn't look like latest puppet is any better though | 21:23 |
| mordred | clarkb: I imagine not | 21:23 |
| *** pcm__ has quit IRC | 21:25 | |
| fungi | clarkb: i would consider stopping the bleeding by replacing the package remove in that class with an exec to yum (probably on an onlyif to rpm -qa|grep python-requests or something so that it doesn't fire on every puppet run) followed by another patch to stop installing python-requests except where we need it and only overriding the version we need where twine is being run as well | 21:25 |
| clarkb | jeblair: so I should go ahead and remove the empty set variable stuff and make those comments a bit more clear and specific | 21:25 |
| *** homeless has joined #openstack-infra | 21:25 | |
| jeblair | clarkb: yeah, want to base your change on the other one to make it simple? | 21:26 |
| *** dalgaaf_ has joined #openstack-infra | 21:26 | |
| clarkb | jeblair: the other one? mine is a d-g change not a config change | 21:26 |
| *** homeless has quit IRC | 21:27 | |
| *** crc32 has joined #openstack-infra | 21:27 | |
| *** homeless has joined #openstack-infra | 21:28 | |
| lifeless | fungi: what about applying the fix from the bug | 21:29 |
| jeblair | clarkb: oh, right; will merging the other one make yours simpler now? | 21:29 |
| fungi | lifeless: that's actually harder to orchestrate in puppet | 21:29 |
| jeblair | harlowja: http://logs.openstack.org/47/47281acf5028fb4cf376eb1076b52f2ab5cc0e46/release/doc8-pypi-both-upload/01c6a9d/console.html | 21:29 |
| harlowja | jeblair much appreciated | 21:29 |
| clarkb | jeblair: not much simpler. It would just change the comment about the driver | 21:29 |
| fungi | lifeless: since in our case we need to make sure it happens between one package operation and any other package operatioj | 21:30 |
| clarkb | jeblair: I think I would still use the tmp var and set it back again | 21:30 |
| *** hogepodge has quit IRC | 21:30 | |
| jeblair | clarkb: where does it get set back? | 21:30 |
| fungi | lifeless: and puppet likes to arbitrarily order things so we can say "do this thing before this other thing" but "do this thing before any calls of this other kind" is a lot more complicated | 21:31 |
| *** emagana has quit IRC | 21:31 | |
| *** MarkAtwood has joined #openstack-infra | 21:31 | |
| lifeless | fungi: I thought we were building with dib now, so we could bring it in there? | 21:31 |
| clarkb | jeblair: in that same block. we preserve values set in job, set "old" values, generate old localrc, reset values, generate new localrc | 21:31 |
| clarkb | jeblair: that way you describe the end state that you want | 21:31 |
| clarkb | and grenade will upgrade you to that point | 21:31 |
| *** emagana has joined #openstack-infra | 21:32 | |
| fungi | lifeless: longer-term, i think we can avoid the problem entirely anyway, so not sure dib makes a difference in that regard. the only place we actually need to do this is one static worker which runs a particular tool | 21:32 |
| *** hogepodge has joined #openstack-infra | 21:32 | |
| jeblair | clarkb: gotcha. yeah, i think dropping 'possibly' and just saying you're setting it back to whatever the job requested the end state be will be more clear | 21:32 |
| fungi | lifeless: and it's not an rpm-based system so wouldn't exhibit the same problem anyway | 21:33 |
| clarkb | jeblair: yup and I can remove the setting of empty var since it is redundant and confusing | 21:33 |
| clarkb | I was trying to preserve as much of the old logic as possible but I Think just making it clearer is better | 21:33 |
| lifeless | fungi: now I'm confused ;) | 21:33 |
| lifeless | fungi: not to worry, leaving it with you | 21:33 |
| *** baoli has quit IRC | 21:35 | |
| *** amcrn has quit IRC | 21:35 | |
| fungi | lifeless: a summary... we need python-requests installed to support workers uploading to swift. we also have a static worker which inherits from the same class where we need to have a newer version of requests to support uploading things to pypi with a tool called twine. that latter case is what's causing the pattern which triggers this side effect on centos workers | 21:35 |
| *** todd_dsm has joined #openstack-infra | 21:36 | |
| fungi | but to solve it properly is a larger refactor to make our static workers stop inheriting from a common class used by our single-use slaves | 21:36 |
| nibalizer | clarkb: so i can't really repro: https://etherpad.openstack.org/p/puppet_problems | 21:36 |
| nibalizer | can you advise why that isn't causing the failure? | 21:36 |
| *** andreaf_ has quit IRC | 21:37 | |
| clarkb | nibalizer: you probably started on a maching without python-requests installed | 21:37 |
| fungi | nibalizer: i was never able to recreate the bug if i distilled down the manifest, so suspect it has something to do with the volume of package operations being performed when the openstack_project::thick_slave class is applied | 21:37 |
| clarkb | nibalizer: install it manually via yum first | 21:37 |
| clarkb | oh huh | 21:37 |
| openstackgerrit | Aaron Rosen proposed a change to openstack/requirements: Add congress and python-congressclient to tracked projects list https://review.openstack.org/114092 | 21:38 |
| *** annegent_ has quit IRC | 21:38 | |
| fungi | nibalizer: i can recreate it with http://paste.openstack.org/show/95656/ on a fresh centos6 node in rackspace (where they preinstall python-requests as a prerequisite of cloud-init) | 21:38 |
| fungi | that's part of what has made it slow to iterate on possible fixes | 21:39 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Allow for grenade upgrades within a release https://review.openstack.org/111094 | 21:39 |
| clarkb | jeblair: ^ ow is that | 21:39 |
| fungi | clarkb has started typing with a cockney accent. must be friday | 21:39 |
| nibalizer | fungi: okay ill try a whole thick node | 21:40 |
| *** todd_dsm has quit IRC | 21:40 | |
| *** mriedem1 has quit IRC | 21:41 | |
| *** sdake has quit IRC | 21:41 | |
| fungi | nibalizer: obviously if you can narrow down the trigger case at all to something faster to run and reset which still produces the error, that would help... my limited puppet-fu just wasn't taking me very far in that regard | 21:41 |
| *** annegentle has joined #openstack-infra | 21:41 | |
| clarkb | fungi: my keyboard has grown an accent | 21:41 |
| jeblair | my known hosts file breaks paramiko | 21:43 |
| fungi | http://paste.openstack.org/show/95743/ seems to work around the issue, so i'm just getting it into proper conditionals now | 21:43 |
| clarkb | jeblair: wow achievement unlocked | 21:43 |
| mordred | jeblair: that's magical | 21:43 |
| jeblair | of course, i have no idea how to clean it up. it has important stuff and 20000 random nodepool hosts | 21:43 |
| clarkb | fungi: you going to add an onlyif too? | 21:44 |
| jeblair | and the entries are all hashed | 21:44 |
| fungi | yup | 21:44 |
| mordred | jeblair: sounds like you might have to add debug statements to paramkiko to find out where it's breaking? | 21:44 |
| jeblair | mordred: 'mv known_hosts foo' | 21:44 |
| jeblair | (i'm already debugging something; maybe some other time) | 21:45 |
| *** arnaud__ has joined #openstack-infra | 21:45 | |
| mordred | jeblair: ++ | 21:45 |
| clarkb | fungi: seems like a realtively clean way to fix it for now | 21:45 |
| fungi | clarkb: yeah, and as i said once we have things running again, a refactor to further separate our static worker configurations from our single-use worker configurations so we don't have bleed over on these special cases | 21:46 |
| fungi | or we hack up the pypi upload job to make a venv and pip install twine into it | 21:47 |
| fungi | like we do for wheel generation jobs | 21:47 |
| *** e0ne has joined #openstack-infra | 21:47 | |
| nibalizer | fungi: well first i want to see it fail | 21:50 |
| nibalizer | but yea | 21:50 |
| nibalizer | expensive in terms of time to iterate | 21:50 |
| fungi | nibalizer: yep, i'm once again waiting on a fresh victim instance to finish bootstrapping | 21:50 |
| nibalizer | fungi: have we actually met? we're you in portland? | 21:51 |
| nibalizer | for oscon, i mean | 21:52 |
| *** otherwiseguy has joined #openstack-infra | 21:52 | |
| fungi | nibalizer: i think i helped you sneak into a summit session in portland last year | 21:52 |
| fungi | but yeah i was at oscon | 21:52 |
| *** Ryan_Lane has quit IRC | 21:53 | |
| nibalizer | okay sweet | 21:53 |
| nibalizer | hrm im trying to remember | 21:53 |
| nibalizer | oh well, there will be future opportunities! | 21:53 |
| *** Ish__ has left #openstack-infra | 21:53 | |
| fungi | nibalizer: i'm the short dorky guy with long hair and a hawaiian shirt | 21:53 |
| jeblair | fungi: ssh review gerrit query 35520 | 21:54 |
| jeblair | fungi: returns no results :/ | 21:54 |
| jeblair | exists though: https://review.openstack.org/#/c/35520/ | 21:54 |
| fungi | jeblair: this sounds familiar... trying to remember why though | 21:55 |
| *** shayneburgess has quit IRC | 21:55 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Start running bashate on config (nonvoting) https://review.openstack.org/112446 | 21:55 |
| fungi | jeblair: yeah, search gerrit's webui for its change id and you get the same (Ie17b9a2c96720892e4140b36c2ebf81fb2f3e058) | 21:56 |
| jeblair | fungi: ssh review gerrit query change:Ie17b9a2c96720892e4140b36c2ebf81fb2f3e058 also returns 0 | 21:56 |
| fungi | jeblair: i want to say we spotted a change with these symptoms causing an issue for zuul... | 21:57 |
| fungi | because gerrit would report it as a child/parent but then when zuul tried to query that it got zilch | 21:57 |
| fungi | but now for the life of me i can't recall if we identified the cause | 21:59 |
| fungi | it's like something is causing it not to get included in the lucene search index | 22:00 |
| *** e0ne has quit IRC | 22:01 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add query for lbaas create member fail AttributeError https://review.openstack.org/114325 | 22:02 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove fingerprint for bug 1333410 https://review.openstack.org/114674 | 22:03 |
| uvirtbot | Launchpad bug 1333410 in nova "sqla 0.9.5 breaks the world" [Critical,Fix released] https://launchpad.net/bugs/1333410 | 22:03 |
| *** datsun180b has quit IRC | 22:03 | |
| nibalizer | fungi: wow this takes forrreveerrrr | 22:03 |
| clarkb | fungi drafts were an issue once iirc | 22:04 |
| clarkb | but this is different ya? | 22:04 |
| fungi | clarkb: yeah, seems not to be a draft | 22:04 |
| fungi | nor was it likely to have ever been given that it was a change of yours | 22:04 |
| fungi | from a year ago | 22:05 |
| *** nelsnelson has quit IRC | 22:05 | |
| clarkb | and its old enough to have been indrxed | 22:05 |
| fungi | yeah | 22:05 |
| *** bhuvan_ has joined #openstack-infra | 22:05 | |
| fungi | perhaps the gerrit log has enlightening search errors or something | 22:06 |
| jeblair | fungi: the zuul behavior you described (10 minutes ago) is what i'm seeing | 22:07 |
| jeblair | it's one of 2 things tripping up the merge-check stuff | 22:07 |
| fungi | jeblair: right, my vague recollection is that's what brought a similar change to my attention, just can't recall the resolution (if there was one) | 22:08 |
| *** bhuvan has quit IRC | 22:08 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove queries for fixed bugs https://review.openstack.org/114676 | 22:08 |
| *** denis_makogon has quit IRC | 22:13 | |
| *** dimsum_ has quit IRC | 22:16 | |
| *** dimsum_ has joined #openstack-infra | 22:16 | |
| *** homeless has quit IRC | 22:17 | |
| *** Ryan_Lane has joined #openstack-infra | 22:17 | |
| *** annegentle has quit IRC | 22:18 | |
| *** markmcclain has joined #openstack-infra | 22:19 | |
| *** tonytan4ever has quit IRC | 22:19 | |
| *** dimsum_ has quit IRC | 22:20 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/infra-specs: Put the puppet modules in their own projects/repos https://review.openstack.org/99990 | 22:21 |
| *** e0ne has joined #openstack-infra | 22:23 | |
| *** dimsum_ has joined #openstack-infra | 22:26 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Safely remove python-requests on workers https://review.openstack.org/114679 | 22:26 |
| *** dimsum_ has quit IRC | 22:26 | |
| *** dimsum_ has joined #openstack-infra | 22:26 | |
| fungi | nibalizer: clarkb: jeblair: mordred: tested that ^ and it seems to work | 22:26 |
| *** dimsum_ has quit IRC | 22:28 | |
| *** dimsum_ has joined #openstack-infra | 22:28 | |
| mordred | fungi: jeez | 22:28 |
| *** dimsum_ has quit IRC | 22:30 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for nova unit test bug 1357578 https://review.openstack.org/114680 | 22:30 |
| uvirtbot | Launchpad bug 1357578 in nova "Unit test: nova.tests.integrated.test_multiprocess_api.MultiprocessWSGITest.test_terminate_sigterm timing out in gate" [Undecided,New] https://launchpad.net/bugs/1357578 | 22:30 |
| *** dimsum_ has joined #openstack-infra | 22:30 | |
| *** linkid has left #openstack-infra | 22:33 | |
| *** shashankhegde has quit IRC | 22:34 | |
| *** e0ne has quit IRC | 22:35 | |
| *** chuckC has quit IRC | 22:37 | |
| *** MarkAtwood has quit IRC | 22:37 | |
| *** annegentle has joined #openstack-infra | 22:38 | |
| *** melwitt has quit IRC | 22:41 | |
| *** baoli has joined #openstack-infra | 22:42 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Fix queries for Zuul trigger https://review.openstack.org/114682 | 22:43 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Stop removing changes from the cache https://review.openstack.org/114683 | 22:43 |
| jeblair | fungi, clarkb, mordred: ^ that should fix the issues i'm seeing with the zuul trigger as well as make it faster (at the cost of a slow memory leak) | 22:44 |
| fungi | unlikely that'll be the only memory leak in zuul anyway, and that seems like a very, very slow one | 22:45 |
| *** dimsum_ has quit IRC | 22:46 | |
| *** dimsum_ has joined #openstack-infra | 22:47 | |
| *** hashSpeleology has quit IRC | 22:47 | |
| *** stevemar has quit IRC | 22:49 | |
| *** amcrn has joined #openstack-infra | 22:51 | |
| *** dimsum_ has quit IRC | 22:51 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 22:54 | |
| *** Sukhdev has joined #openstack-infra | 22:57 | |
| *** MarkAtwood has joined #openstack-infra | 22:58 | |
| *** reed has quit IRC | 23:01 | |
| *** annegentle has quit IRC | 23:02 | |
| *** nelsnelson has joined #openstack-infra | 23:04 | |
| *** nelsnelson has quit IRC | 23:04 | |
| *** nelsnelson has joined #openstack-infra | 23:04 | |
| *** dimsum_ has joined #openstack-infra | 23:04 | |
| *** flwang_ has joined #openstack-infra | 23:05 | |
| *** gargola has quit IRC | 23:06 | |
| *** rwsu has joined #openstack-infra | 23:06 | |
| *** emagana has quit IRC | 23:06 | |
| asselin | dtroyer, why is this line needed. what would happen if the if is removed? https://github.com/openstack-dev/devstack/blob/master/lib/cinder#L473 | 23:07 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Make smtp test more robust https://review.openstack.org/114689 | 23:07 |
| *** emagana has joined #openstack-infra | 23:07 | |
| *** morganfainberg is now known as morganfainberg_Z | 23:07 | |
| *** chuckC has joined #openstack-infra | 23:08 | |
| dtroyer | asselin: you'd get an error if the configure_cinder_backend_${BE_TYPE} function is undefined, which for some of those tests is totally valid | 23:08 |
| *** ZZelle has quit IRC | 23:08 | |
| dtroyer | oh, wait | 23:09 |
| asselin | ok | 23:09 |
| asselin | (ok on the oh, wait) | 23:09 |
| dtroyer | so in that specific case, that's how we decide to create a volume type..if it isn't configured we skip the type | 23:09 |
| jeblair | fungi, clarkb, mordred: i'm going to manually install my zuul changes into production while things are quiet | 23:09 |
| *** dimsum_ has quit IRC | 23:10 | |
| fungi | jeblair: cool--i'm reviewing them now | 23:10 |
| fungi | but may retire to more weekendish pursuits soon | 23:10 |
| *** dimsum_ has joined #openstack-infra | 23:10 | |
| *** flwang_ has quit IRC | 23:10 | |
| asselin | dtroyer, ok, i'm thinking of deleting the "if". disucssing it in -cinder | 23:10 |
| mordred | jeblair: ++ | 23:10 |
| jeblair | fungi: yeah, i figure this is a good time to work out the kinks since it's doing things like "query all changes and generate events for them" :) | 23:11 |
| dtroyer | asselin: you want a type created in all cases? | 23:11 |
| fungi | jeblair: absolutely! | 23:11 |
| jgriffith | dtroyer: actually.... | 23:11 |
| jgriffith | dtroyer: it's kinda the opposite | 23:11 |
| jgriffith | dtroyer: that if statement actually prevents us from creating a type | 23:11 |
| jgriffith | if we don't have a cinder_backend/xxx file | 23:12 |
| jgriffith | dtroyer: as long as ENABLED_CINDER_BACKENDS=x:x is set it works fine | 23:12 |
| *** emagana has quit IRC | 23:12 | |
| jgriffith | and honestly seems better than requiring a backend file for every driver | 23:12 |
| jgriffith | dtroyer: ie I can do: https://gist.github.com/a38cc7a955b69e17a1cb | 23:12 |
| jeblair | pleia2: i can hardly believe you reviewed that change! and you found stuff! i'm having trouble finding your comments! :) | 23:13 |
| jgriffith | and things just *work* and I don't need to submit a backend file | 23:13 |
| *** david-lyle has quit IRC | 23:13 | |
| jgriffith | 8924abe7-d3e4-437a-a416-2d3ae5c0b08a | solidfire | {u'volume_backend_name': u'solidfire'} | 23:13 |
| jgriffith | of course... I see your point | 23:13 |
| jgriffith | about the lvm driver... but that should still work as well | 23:13 |
| jgriffith | ie default | 23:13 |
| dtroyer | jgriffith: ok, so without the backend you don't get a type then either... | 23:14 |
| dtroyer | I suppose I don't quite get it | 23:14 |
| jgriffith | dtroyer: sure do | 23:14 |
| asselin | dtroyer, we want to be able to create the type without defining configure_cinder_backend_${BE_TYPE} | 23:14 |
| jgriffith | dtroyer: if I leave the if statement of course I don't get it without the backend file | 23:15 |
| dtroyer | ah, that's the bit I didn't connect... | 23:15 |
| *** dimsum_ has quit IRC | 23:15 | |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/publications: Add link to instructions for using template https://review.openstack.org/98906 | 23:15 |
| pleia2 | jeblair: haha | 23:15 |
| dtroyer | so just do types for everything in CINDER_ENABLED_BACKENDS? | 23:15 |
| asselin | yes | 23:15 |
| dtroyer | jgriffith, asselin: that's cool. I think I was being careful about creating types | 23:16 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Enqueue unfresh changes in check when parent in gate https://review.openstack.org/114617 | 23:16 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add Zuul merge-check to all projects https://review.openstack.org/114625 | 23:16 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove freshness check https://review.openstack.org/114626 | 23:16 |
| asselin | dtroyer, ok I'll submit a patch. This will simplify our 3rd party cinder testing. | 23:16 |
| dtroyer | good deal | 23:17 |
| asselin | dtroyer, fyi http://lists.openstack.org/pipermail/openstack-dev/2014-August/043086.html | 23:17 |
| *** otter768 has quit IRC | 23:21 | |
| asselin | dtroyer, https://review.openstack.org/114693 | 23:23 |
| clarkb | fungi: your exec change didn't pass the linter | 23:26 |
| clarkb | looking at it now | 23:26 |
| clarkb | fungi: you need to wrap vars in ${} when used in "" | 23:27 |
| fungi | clarkb: yep, fixing | 23:28 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Safely remove python-requests on workers https://review.openstack.org/114679 | 23:29 |
| fungi | clarkb: thx | 23:29 |
| clarkb | jeblair: I am reviewing 114682 and child | 23:29 |
| jeblair | clarkb: i am disabling 4 third-party ci accounts | 23:29 |
| openstackgerrit | patrick-crews proposed a change to openstack-infra/config: Document crm114 for logstash https://review.openstack.org/114627 | 23:29 |
| clarkb | jeblair: ok | 23:29 |
| jeblair | fungi: are we doing some kind of double-accounting thing with voting third-party ci? | 23:30 |
| fungi | jeblair: for 114683 is there any malleable status information zuul collects on a change which could vary later and now not get noticed? | 23:30 |
| jeblair | fungi: that is, if i remove someone from that, do i need to add them to third-party ci? | 23:31 |
| fungi | jeblair: yeah, there is no double-accounting currently | 23:31 |
| *** annegentle has joined #openstack-infra | 23:31 | |
| jeblair | sorry, i meant to say 'double-entry' | 23:31 |
| fungi | jeblair: so i'd been just removing from and and adding to the other to enable/disable voting for a project | 23:31 |
| jeblair | fungi: i think i'd much prefer 'third-party ci' to be an explicit list of everyone, and 'voting third-party ci' to be a list of only those who can vote | 23:32 |
| jeblair | fungi: that way it's so much easier to immediately disable them | 23:32 |
| fungi | we have the voting group set as included by the non-voting group so recursion works, but we can change that if we want | 23:32 |
| jeblair | fungi: yeah, it's just i don't want this to be hard | 23:32 |
| jeblair | because it happens all the time | 23:33 |
| *** dimtruck is now known as zz_dimtruck | 23:33 | |
| fungi | jeblair: agreed, ultimately i still think we simplify it by making them per-program and letting the programs control these things | 23:33 |
| *** annegentle has quit IRC | 23:33 | |
| fungi | but that requires slighthly more complicated acls for everyone | 23:33 |
| jeblair | fungi: yeah, though even in that case, i'm going to remove folks for policy violiations | 23:33 |
| fungi | jeblair: sure, we have the ability to do so | 23:33 |
| clarkb | jeblair: I am going to approve 114682 | 23:33 |
| clarkb | I assume that is ok | 23:34 |
| jeblair | clarkb: probably :) i haven't gotten around to doing my manual install/restart yet | 23:34 |
| clarkb | jeblair: ok | 23:34 |
| clarkb | jeblair: would you prefer to manual install first without puppet udpating it? | 23:34 |
| jeblair | clarkb: nah, go for it | 23:34 |
| clarkb | kk | 23:35 |
| jeblair | this third-party thing may take up the rest of my day | 23:35 |
| jeblair | grr | 23:35 |
| clarkb | jeblair: want me to take that on so you can zuul? | 23:35 |
| *** markmcclain has quit IRC | 23:35 | |
| *** rwsu has quit IRC | 23:35 | |
| jeblair | clarkb: nah, i'll get it | 23:35 |
| fungi | jeblair: which third-party thing? can i help? | 23:35 |
| * fungi feels like he's overlooked an ml thread or something | 23:36 | |
| jeblair | i just need to write up 4 emails about the systems i'm disabling | 23:36 |
| jeblair | fungi: see message from hashar to infra | 23:36 |
| fungi | oh, the ones adding "starting check jobs" messahes? | 23:36 |
| jeblair | yep. they all also have other problems | 23:36 |
| fungi | i assume these are cookie-cutter zuul installs with no local tuning | 23:37 |
| fungi | probably using our layout.yaml | 23:37 |
| jeblair | who's telling them to put the... | 23:37 |
| jeblair | i can't even begin to say how wrong that is | 23:37 |
| jeblair | also, i'm pretty sure i'm now in favor of keeping the email addr in gerrit | 23:37 |
| fungi | it would make it marginally easier to track these people down, yes ;) | 23:38 |
| clarkb | ++ | 23:38 |
| mordred | ++ | 23:38 |
| fungi | (or at least find and block their ip addresses) | 23:38 |
| jeblair | i'm assuming these folks would not have a wiki entry (they've done everything else wrong, why would they do that?) and it's too much cross-referencing for something like this | 23:38 |
| fungi | yeah | 23:39 |
| clarkb | fungi: exec change lgtm | 23:39 |
| clarkb | mordred: ^ can you rereview | 23:39 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Fix queries for Zuul trigger https://review.openstack.org/114682 | 23:39 |
| fungi | the down-side to making them supply e-mail addresses at account creation time is that they invariably pick a personal address and then come back later asking us to change it for them | 23:39 |
| mordred | clarkb: yah. which one/ | 23:40 |
| clarkb | mordred: https://review.openstack.org/#/c/114679/ | 23:40 |
| mordred | oh - the fungi ... | 23:40 |
| fungi | also, when they do supply a business e-mail address, more often than not it seems to bounce because their employers consider e-mail addresses as being something you only use for internal or whitelisted communication | 23:40 |
| fungi | from teh internet at large, you only allow messages to sales@, marketing@ and info@ (and probably filter those into a black hole or use them to collect for your mass mail spam) | 23:41 |
| clarkb | :/ | 23:41 |
| mordred | fungi: people suck | 23:42 |
| * jeblair hits enter 86 times | 23:43 | |
| clarkb | I am reviewing yolanda's nodepool fix. Probably won't get that in today but hopefully on monday I can give it a go again | 23:43 |
| fungi | mordred: yeah, humans are useless. i'm ready to rebel and start my own species | 23:43 |
| pleia2 | hah :) | 23:44 |
| mordred | clarkb: I am excited about such things | 23:44 |
| jeblair | fungi: third-party ci no longer includes voting third-party ci | 23:44 |
| fungi | jeblair: noted and i'll adjust any further edits to those groups accordingly | 23:44 |
| openstackgerrit | A change was merged to openstack-infra/config: Safely remove python-requests on workers https://review.openstack.org/114679 | 23:45 |
| mordred | clarkb: can I ask a stupid question? | 23:45 |
| clarkb | mordred: sure | 23:46 |
| mordred | clarkb: why does this code explicitly get a token from keystone? | 23:46 |
| fungi | jeblair: so all of the offenders hashar mentioned were in the voting third-party ci group? of course, any account can leave a "starting whatever..." comment on a change, so we ultimately probably just --inactive the offenders? | 23:46 |
| jeblair | fungi: ah, as it turns out none of them are voting | 23:46 |
| jeblair | fungi: so yeah, i'll be --inactiving them | 23:46 |
| fungi | jeblair: okay, that is unsurprising | 23:46 |
| mordred | clarkb: I mean, why is it that it cant' do what novaclient does and handle an auth url? | 23:46 |
| jeblair | fungi: i'm wondering if we should tweak the acls so that they can't even comment on anything except sandbox unless they are voting | 23:47 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Stop removing changes from the cache https://review.openstack.org/114683 | 23:47 |
| fungi | jeblair: that's probably doable, though i'd have to ponder how we subtract them from registered users in the all-projects acl | 23:47 |
| *** daya_k has joined #openstack-infra | 23:47 | |
| clarkb | mordred: its a good question. I am sure morganfainberg_Z can expand more but the tl;dr is that each of our clients does keystone different. Some speak keystone api directly (novaclient iirc), others import keystoneclient for you and do that properly (swiftclient) and finally others expect you to give it a token (glanceclient) | 23:48 |
| fungi | jeblair: it's probably trivial to implement and friday night is merely crippling my brains | 23:48 |
| clarkb | mordred: basically if you want to talk about a "stabilization" release to fix bugs we should also be looking at fixing things like ^ | 23:48 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Test nodepool handles failed cloud providers. https://review.openstack.org/114388 | 23:49 |
| notmyname | clarkb: interesting summary. we (swift devs) have considered speaking the keystone api instead of using keystoneclient in order to limit dependencies and make adoption easier. but keystone's api is getting more complicated, so that's not as simple as it once was | 23:49 |
| mordred | clarkb: that makes me want to kill someone | 23:49 |
| clarkb | notmyname: ya, I don't think any given way is right or wrong but we should probably be consistent | 23:50 |
| mordred | notmyname: I think also morganfainberg_Z is working to try to minimize keystoneclient depends | 23:50 |
| * notmyname likes that swift is called out as the right way ;-) | 23:50 | |
| mordred | notmyname: since the 'right' way would be to just use keystoneclient- but as you say, doing that currently brings in unnecessary crap | 23:50 |
| fungi | notmyname: the choice might be a little easier if keystoneclient didn't also depend on lxml | 23:50 |
| *** otherwiseguy has quit IRC | 23:51 | |
| * mordred votes for the lxml depend to be removed from keystoneclient | 23:51 | |
| fungi | if arbitrarily voting on things makes them magically happen, then count me in | 23:51 |
| mordred | fungi: ZOMG | 23:51 |
| mordred | fungi: ZOMG ZOMG ZOMG | 23:51 |
| mordred | lxml is only there to support a contrib extension | 23:51 |
| mordred | fungi: keystoneclient/contrib/auth/v3/saml2.py:from lxml import etree | 23:52 |
| fungi | mordred: also not particularly surprising | 23:52 |
| clarkb | fungi: ok your change merged, I am going to give rebuilding a rax centos6 image a go | 23:52 |
| clarkb | fungi: unless you want to but i figure its late for your so I can do it | 23:52 |
| fungi | we could just propose a one-line patch to keystoneclient, i guess | 23:52 |
| clarkb | dfw is building | 23:52 |
| jeblair | jeez: | 23:52 |
| jeblair | ssh review gerrit set-account --inactive '"ProphetStor CI"' | 23:52 |
| jeblair | fatal: user "ProphetStor CI" not found | 23:52 |
| jeblair | i had to do that one by number. no idea why. | 23:53 |
| fungi | clarkb: i appreciate your takeover, but do give me a heads up if it ends up failing and i'll try to keep an eye on irc/jump in and help | 23:53 |
| jeblair | of course, i had to do ls-members to get the number. | 23:53 |
| clarkb | fungi: will do | 23:53 |
| fungi | jeblair: at least there's an api call for that now. i still forget from time to time and log into the db to query for the account number | 23:54 |
| fungi | select * from accounts where full_name like "%foo%"; | 23:54 |
| clarkb | jeblair: possibly because there are two or three prophetstor CIs | 23:55 |
| jeblair | clarkb, fungi, mordred: http://ci.openstack.org/third_party.html | 23:55 |
| clarkb | jeblair: they created a couple lp accounts before reading the docs or some such | 23:55 |
| jeblair | do we not actually say "you may only leave one comment per patch?" | 23:56 |
| fungi | jeblair: possibly not. some of that may only be lore and oral tradition still | 23:56 |
| mordred | https://review.openstack.org/114701 | 23:56 |
| jeblair | (luckily if so, they all violated the 'public link' clause) | 23:56 |
| mordred | there. taht fixes keystoneclient ^^ | 23:56 |
| jeblair | see i told you this third party thing would take the rest of my day | 23:57 |
| clarkb | jeblair: can you see my comments on 114599 when you have a moment | 23:58 |
| clarkb | mordred: OMGBBQ | 23:58 |
| clarkb | mordred: we can fall back to stdlib? | 23:59 |
| clarkb | mordred: can we do this to every single thing in oepnstack that imports lxml? | 23:59 |
| jeblair | clarkb: it's a good point, but it might still be good as a regression test in case we try to do something like that again | 23:59 |
| mordred | clarkb: I don't know if there are features that are needed in lxml ... but I thnk it's just about performance | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
