| jeblair | zuul is moving again | 00:00 |
|---|---|---|
| fungi | bare-precise nodes take a long time to build... the one old nodepool started builfing half an hour ago in hpcloud is still going | 00:01 |
| fungi | er, new nodepool | 00:01 |
| *** groknix has quit IRC | 00:01 | |
| *** bo_sh has joined #openstack-infra | 00:01 | |
| fungi | devstack-precise i mean | 00:01 |
| fungi | still hasn't successfully added one | 00:02 |
| *** groknix has joined #openstack-infra | 00:02 | |
| clarkb | gah I may have derped my strace /me tries agin no wonder I was confused | 00:02 |
| *** amitgandhinz has quit IRC | 00:02 | |
| sweston | asselin: the merge didn't work, or the history was not saved/ | 00:03 |
| sweston | ? | 00:03 |
| *** dims has quit IRC | 00:03 | |
| jeblair | fungi: maybe the important question is why there seems to be no queue of merge jobs | 00:03 |
| asselin | sweston, it merged all of master, undoing the subtree | 00:03 |
| *** dims has joined #openstack-infra | 00:03 | |
| asselin | sweston, not sure why I though it worked before....?? | 00:04 |
| sweston | asselin: eeeew | 00:04 |
| asselin | sweston, --rejoin seems to be what we need. testing it out now | 00:05 |
| fungi | jeblair: yeah, i think as soon as they end up in the gearman queue the merge workers snap them up, as gearman perpetually reports none are waiting | 00:05 |
| mordred | fungi: but a merge worker should only snap up one at a time | 00:06 |
| fungi | mordred: right. for whatever reason merge work requests seem to be trickling into gearman very slowly | 00:07 |
| *** dims has quit IRC | 00:08 | |
| *** dangers is now known as dangers_away | 00:08 | |
| jeblair | okay, no one objected. i'm making the call to revert everything | 00:09 |
| sweston | asselin: so the workflow is to start with your local repo, then do a git remote add, then a fetch and merge. you are doing it differently, right, by starting with subtree split? | 00:09 |
| jeblair | clarkb, fungi: can you get old nodepool in shape? | 00:09 |
| clarkb | ya I will update dns now | 00:09 |
| *** groknix has quit IRC | 00:09 | |
| clarkb | someone should stop new nodepool service | 00:10 |
| clarkb | fungi: ^ you want to do that? | 00:10 |
| jeblair | i will clean up zuul | 00:10 |
| fungi | jeblair: yeah, doing | 00:10 |
| sweston | asselin: if --rejoin doesn't work, then i'd try with the steps here http://git-scm.com/book/en/v1/Git-Tools-Subtree-Merging | 00:10 |
| clarkb | fungi: we should also run puppet on old nodepool prior to starting to pull in changes | 00:10 |
| fungi | agreed | 00:10 |
| *** camunoz has joined #openstack-infra | 00:11 | |
| fungi | nodepoold stopped and puppet agent disabled on new nodepool | 00:11 |
| asselin | sweston, here's what I'm doing: http://paste.openstack.org/show/137782/ | 00:12 |
| fungi | puppet agent enabled and updated on old nodepool server | 00:12 |
| clarkb | A and AAAA records updated | 00:12 |
| fungi | thanks, was about to ask | 00:12 |
| clarkb | so ~5 minutes from now I will kick iptables on zuul and the jenkinses | 00:13 |
| fungi | as soon as puppetmaster sees the updated dns entries i'll refresh the known_hosts entry for it | 00:13 |
| sweston | asselin: ok | 00:13 |
| fungi | okay, known_hosts updated on puppetmaster | 00:14 |
| *** sarob has quit IRC | 00:14 | |
| fungi | gear 0.5.2 is the one we want, right? | 00:15 |
| jeblair | fungi: yep | 00:15 |
| clarkb | yup | 00:15 |
| fungi | that's the one currently on old nodepool, so no changes needed | 00:15 |
| clarkb | ya old nodepool should be ok | 00:15 |
| jeblair | i'm running a geard on zuul atm so nodepool can gracefully start without zuul running | 00:15 |
| fungi | we need to reupdate iptables on zuul? | 00:16 |
| clarkb | fungi: yup once dns goes propogates | 00:16 |
| clarkb | s/goes// | 00:16 |
| jeblair | fungi: it currently has both actually | 00:16 |
| fungi | or is it still open from when jeblair manually added it back earlier? | 00:17 |
| jeblair | since i manually added it for testing | 00:17 |
| fungi | yeah, that | 00:17 |
| *** tnovacik has quit IRC | 00:17 | |
| clarkb | oh right then not on zuul but on jenkinses | 00:17 |
| *** david-lyle is now known as david-lyle_afk | 00:17 | |
| fungi | ahh, right. need to update iptables on the 7 jenkins masters | 00:17 |
| *** rushiagr_away is now known as rushiagr | 00:17 | |
| fungi | just start the iptables-persistent initscript on them? | 00:17 |
| clarkb | yup I can do that in a minute | 00:17 |
| clarkb | want to make sure dns is updated first | 00:18 |
| fungi | ahh, yep | 00:18 |
| jeblair | zuul is ready when we are | 00:18 |
| clarkb | ok I get new records locally going to get the jenkinses | 00:18 |
| clarkb | fungi: jenkins01 is done can you test telnet from old nodepool to that node over port 8888 | 00:19 |
| fungi | works | 00:19 |
| *** MarkAtwood has quit IRC | 00:19 | |
| clarkb | all 8 are done now | 00:20 |
| fungi | i just checked them all too | 00:20 |
| clarkb | I think that means we can start nodepool service on old server | 00:20 |
| fungi | i think you were opening each just ahead of me | 00:20 |
| jeblair | wait on that | 00:20 |
| fungi | k | 00:20 |
| jeblair | i think we need to delete things in the alien list first, right? | 00:20 |
| fungi | some of those aliens may be running jobs right now? | 00:20 |
| jeblair | also | 00:20 |
| jeblair | requests.exceptions.ConnectionError: HTTPSConnectionPool(host='ci-overcloud.tripleo.org', port=13000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'socket.error'>: [Errno 110] Connection timed out) | 00:20 |
| jeblair | fungi: but zuul is stopped | 00:20 |
| clarkb | jeblair: I didn't use the alien list I just used nodepool cli on server to list then delete all nodes with xargs -n 5 | 00:21 |
| fungi | oh | 00:21 |
| fungi | i will do that on the trusty nodepool now | 00:21 |
| *** dims has joined #openstack-infra | 00:21 | |
| jeblair | fungi: ok thanks | 00:21 |
| fungi | clarkb: what was trusty nodepool's ipa? | 00:21 |
| clarkb | it worked well something like nodepool list | cut -d'|' -f 2 | xargs -n 1 -P 5 nodepool delete --now | 00:21 |
| *** mbacchi has quit IRC | 00:22 | |
| clarkb | fungi: 104.130.155.213 | 00:22 |
| mmedvede | asselin: can you explain in two words the main idea behind the split job? I do not understand the part when it does curl of existing split repo, overlaying it on top of system-config | 00:23 |
| fungi | they're deleting now | 00:23 |
| fungi | this can happen in parallel with starting precise nodepool presumably? | 00:23 |
| clarkb | fungi: I think so | 00:24 |
| clarkb | jeblair: ^ | 00:24 |
| asselin | the curl creates the remote repository. it either succeds, or errors saying it already exists. | 00:24 |
| jeblair | sounds reasonable | 00:24 |
| clarkb | I did it in parallel when we went the other direction | 00:24 |
| *** banix has quit IRC | 00:24 | |
| fungi | i'll do another pass for alien nodes from precise nodepool's perspective once this is completed | 00:24 |
| fungi | any other last-minute objections before i start precise nodepoold? | 00:24 |
| clarkb | none here | 00:24 |
| asselin | mmedvede, the idea in 2 words is "automate sub-tree" | 00:24 |
| mmedvede | asselin: ah, neat | 00:25 |
| mmedvede | asselin: and what is the benefit of --rejoin? | 00:25 |
| fungi | okay, started | 00:25 |
| fungi | 57 nodes building | 00:26 |
| asselin | mmedvede, still unconfirmed, the idea is to be able to do a re-subtree and only process the new commits, insead of starting over. | 00:26 |
| asselin | mmedvede, so obviously much faster | 00:26 |
| *** thingee has joined #openstack-infra | 00:27 | |
| fungi | 63 building now | 00:27 |
| *** rushiagr is now known as rushiagr_away | 00:28 | |
| *** AlexF has joined #openstack-infra | 00:28 | |
| *** prad has quit IRC | 00:28 | |
| jeblair | some nodes are registering | 00:29 |
| fungi | yeah, we have ready nodes | 00:30 |
| fungi | though no bare-precise or devstack-precise yet | 00:30 |
| jeblair | fungi: let me know when we have bare and devstack versions of precise and trusty | 00:31 |
| *** dims has quit IRC | 00:31 | |
| fungi | will do. it just errored out a bare-precise and started building a new one | 00:31 |
| *** dims has joined #openstack-infra | 00:31 | |
| *** melwitt has quit IRC | 00:33 | |
| *** AlexF has quit IRC | 00:34 | |
| *** dims has quit IRC | 00:36 | |
| fungi | there's a bare-precise ready now, devstack-precise still building | 00:37 |
| fungi | clarkb: looks like the centos7 diskimage isn't happy | 00:37 |
| fungi | clarkb: devstack-centos7-dib seems to be in an image build loop | 00:37 |
| clarkb | ya known issue with precise | 00:38 |
| clarkb | its the reason we upgraded to trusty | 00:38 |
| fungi | oh, right. hurrr | 00:38 |
| fungi | i think only half my pistons are firing today | 00:38 |
| *** ZZelle_ has quit IRC | 00:38 | |
| mmedvede | asselin: thanks for explaining. I am going to do a couple of quick tests to check if --rejoin does help. I have a bad feeling it does not | 00:39 |
| asselin | mmedvede, please | 00:39 |
| *** AlexF has joined #openstack-infra | 00:40 | |
| *** mpavlase has quit IRC | 00:40 | |
| *** bo_sh has quit IRC | 00:41 | |
| fungi | about 5 minutes into a new devstack-precise build. the last one fell victim to a nova boot error in rax-dfw | 00:43 |
| clarkb | current one is in rax iad | 00:43 |
| openstackgerrit | Surojit Pathak proposed openstack-dev/hacking: Fixing broken while loop in imports.py https://review.openstack.org/136517 | 00:44 |
| *** david-lyle_afk has quit IRC | 00:45 | |
| *** che-arne has quit IRC | 00:45 | |
| *** che-arne has joined #openstack-infra | 00:45 | |
| *** wenlock has quit IRC | 00:46 | |
| *** emagana has quit IRC | 00:46 | |
| *** MarkAtwood has joined #openstack-infra | 00:46 | |
| jeblair | why in the world is this taking so long? | 00:46 |
| *** emagana has joined #openstack-infra | 00:46 | |
| clarkb | I just remember to update iptables on graphite.o.o so thats done | 00:48 |
| clarkb | now our graphs should better reflect reality | 00:48 |
| *** emagana_ has joined #openstack-infra | 00:48 | |
| *** emagana has quit IRC | 00:48 | |
| fungi | rackspace boot from snapshot seems to take 10-20 minutes under good conditions, and a significant percentage of the time ends in a nova boot error. doesn't help that nodepool has a min-ready of 1 for those precise labels now too | 00:48 |
| jeblair | i'm going to go ahead and start zuul | 00:49 |
| jeblair | the additional demand might help us get a working node faster | 00:49 |
| fungi | here's hoping | 00:49 |
| clarkb | sounds good | 00:49 |
| jeblair | and too bad about the not_registered for icehouse jobs | 00:49 |
| fungi | i'll keep an eye out for signs in the precise nodepool logs of any similar gearman client timeouts | 00:49 |
| clarkb | ugh looks like the precise node that was building failed | 00:50 |
| fungi | yeah, another LaunchStatusException but this time in rad-iad | 00:51 |
| fungi | er, rax-iad | 00:51 |
| *** ryanpetrello has quit IRC | 00:52 | |
| jeblair | File "/usr/local/lib/python2.7/dist-packages/novaclient/exceptions.py", line 217, in from_response | 00:53 |
| jeblair | return cls(**kwargs) | 00:53 |
| jeblair | TypeError: __init__() got an unexpected keyword argument 'retry_after' | 00:53 |
| jeblair | wtf? | 00:53 |
| jeblair | that's some nice forward compatability there, novaclient | 00:53 |
| clarkb | jeblair: is that from nodepool? | 00:53 |
| jeblair | that's happening a lot | 00:53 |
| jeblair | yep | 00:53 |
| jroll | fungi: yeah, rackspace's boot times on custom images are really really sad | 00:53 |
| clarkb | if we update novaclient we also need to update six | 00:54 |
| jeblair | it kind of looks like rackspace is running a version of nova that is returning extra information in error responses that novaclient does not know how to deal with | 00:54 |
| clarkb | ya | 00:55 |
| jeblair | this is the sort of thing that makes me hate using openstack | 00:55 |
| clarkb | telling us to try again later | 00:55 |
| *** dannywilson has quit IRC | 00:55 | |
| jeblair | mordred: ^ fyi | 00:55 |
| *** dannywilson has joined #openstack-infra | 00:56 | |
| jeblair | ugh and you know what? | 00:56 |
| jeblair | because it's an error that we don't know about | 00:56 |
| jeblair | nodepool deletes it | 00:56 |
| fungi | looks like jogo reported that as bug 1365251 | 00:56 |
| uvirtbot | Launchpad bug 1365251 in python-novaclient "TypeError: __init__() got an unexpected keyword argument 'retry_after'" [High,Fix released] https://launchpad.net/bugs/1365251 | 00:56 |
| *** FallenPegasus has joined #openstack-infra | 00:56 | |
| clarkb | looks like it was fixed in novaclient though | 00:57 |
| fungi | 2.20.0 | 00:57 |
| clarkb | which means we need latest novaclient... | 00:57 |
| *** MarkAtwood has quit IRC | 00:57 | |
| *** FallenPegasus is now known as MarkAtwood | 00:57 | |
| fungi | yeah | 00:57 |
| *** AlexF has quit IRC | 00:58 | |
| jeblair | that's not a fix :/ | 00:58 |
| clarkb | jeblair: right | 00:58 |
| jeblair | and yep | 00:58 |
| jeblair | the fix is specific to this thing | 00:58 |
| jeblair | so the next new "feature" that's added will have the same problem | 00:58 |
| *** unicell has joined #openstack-infra | 00:59 | |
| jeblair | well, rackspace is currently useless to us because of this :( | 00:59 |
| clarkb | should I upgrade novaclient? | 00:59 |
| clarkb | however new novaclient/six may have contributed to new nodepool issues... | 00:59 |
| jeblair | clarkb: exactly | 00:59 |
| clarkb | jeblair: though is nodepool only deleting nodes that errored anyways? | 01:00 |
| clarkb | I guess figuring out what the exception is may be helpful | 01:00 |
| fungi | oh, is six where it was theorized async i/o issues were arising? | 01:00 |
| jeblair | clarkb: that exception causes nodepool to delete nodes | 01:00 |
| *** bswartz has joined #openstack-infra | 01:00 | |
| *** dannywilson has quit IRC | 01:00 | |
| clarkb | jeblair: right but its in exception handling | 01:01 |
| clarkb | jeblair: so maybe it was already a failure? | 01:01 |
| clarkb | fungi: no, but its a thing that was different between the nodes | 01:01 |
| jeblair | clarkb: i assume "retry_after" comes with a rate limit error | 01:01 |
| jeblair | which normally does not cause us to delete nodes | 01:01 |
| fungi | yeah, digging in nova now to find out | 01:02 |
| clarkb | jeblair: looking at that code it seems assocaited to overlimit errors | 01:02 |
| jeblair | yep | 01:02 |
| *** ryanpetrello has joined #openstack-infra | 01:02 | |
| clarkb | except that class appears to handle kwargs so likely its another exception that does not handle kwargs /me looks | 01:02 |
| *** unicell has quit IRC | 01:03 | |
| clarkb | rate limit and overlimite should handle it properly | 01:03 |
| clarkb | so likely a different exception | 01:03 |
| fungi | RateLimitFault._retry_after(retry_time) | 01:03 |
| fungi | so, er, yep | 01:04 |
| *** salv-orlando has quit IRC | 01:04 | |
| jeblair | oh, i was mistaken, the bugfix for that in novaclient is supposed to make novaclient more resilient generally | 01:04 |
| jeblair | so that's good at least | 01:04 |
| fungi | and that _retry_after() method is responsible for adding headers = {'Retry-After': '%d' % retry_after} | 01:04 |
| clarkb | jeblair: ya I think nova server started putting that data in more responses | 01:04 |
| clarkb | so the bug is really in nova server imo | 01:05 |
| jeblair | it looks like there's a bug in nova-api where retry_after is being added to responses where it should not be | 01:05 |
| jeblair | ya | 01:05 |
| *** MaxV has quit IRC | 01:06 | |
| jeblair | jogo, mikal: fyi https://bugs.launchpad.net/python-novaclient/+bug/1365251 is hitting us for real, and causing quite a lot of issues booting nodes in rackspace | 01:06 |
| uvirtbot | Launchpad bug 1365251 in python-novaclient "TypeError: __init__() got an unexpected keyword argument 'retry_after'" [High,Fix released] | 01:06 |
| jogo | jeblair: lookking | 01:06 |
| fungi | JayF: ^ ;) | 01:06 |
| jogo | jeblair: can you pastebin the stacktrace you are getting | 01:07 |
| *** r-daneel has quit IRC | 01:07 | |
| jeblair | jogo, mikal, JayF: and we are not in a position to upgrade novaclient right now to get that fix :( | 01:07 |
| fungi | nor are we in a position to patch rackspace's nova installs ;) | 01:08 |
| jeblair | jogo: http://paste.openstack.org/show/137795/ | 01:08 |
| *** marun has joined #openstack-infra | 01:08 | |
| JayF | fungi: I'm not in that position either (thankfully) | 01:08 |
| mordred | jeblair: wow | 01:08 |
| clarkb | jeblair: fungi: what if we install nodepool to a virtualenv | 01:09 |
| jogo | jeblair: can you get the underlying error? for example in http://paste.ubuntu.com/8229361/plain/ | 01:09 |
| jogo | its a quota limit | 01:09 |
| clarkb | on precise | 01:09 |
| *** teran has quit IRC | 01:09 | |
| clarkb | then we can always fall back on the global install if it exhibits old behavior with gear | 01:09 |
| jeblair | jogo: that page wants me to log in but i don't want to :( | 01:09 |
| clarkb | (I am thinking out loud here. feel free to tell me its a bad idea) | 01:09 |
| jogo | jeblair: http://paste.ubuntu.com/8229361/plain/ is that better? | 01:09 |
| jogo | jeblair: RESP BODY: {"forbidden": {"message": "Quota exceeded for instances: Requested 1, but already used 10 of 10 instances", "code": 403}} | 01:10 |
| jeblair | clarkb: we've lost a day of work because the ci system was broken, i'm just trying to get a working system going again, and to do that, i'd really like to start with a known working state | 01:10 |
| jeblair | jogo: no | 01:10 |
| *** signed8bit has quit IRC | 01:10 | |
| jeblair | still asks for login | 01:10 |
| clarkb | unfortunately rax appears to have moved beyond known working independently | 01:10 |
| jeblair | clarkb: well, hopefully it will work well enough with the extra failures | 01:11 |
| clarkb | ok | 01:11 |
| clarkb | thats certainly possible | 01:11 |
| jeblair | clarkb: (it's designed for clouds to fail, that doesn't bother me as much) | 01:11 |
| clarkb | ya | 01:11 |
| mordred | I have very bad internet right now | 01:11 |
| jeblair | clarkb: then we can decide how to proceed | 01:11 |
| mordred | extra useless mordred | 01:12 |
| jogo | jeblair: can you tcpdump or something? | 01:12 |
| *** ChuckC has quit IRC | 01:12 | |
| jogo | jeblair: if you cannot upgrade novaclient, and we don't know what the specific error you are getting is. not sure what to try next | 01:12 |
| jeblair | jogo: i really hope the connection is encrypted | 01:12 |
| fungi | jogo: https | 01:12 |
| jogo | jeblair: hehe woops yeah | 01:13 |
| *** adalbas has quit IRC | 01:13 | |
| fungi | we'd need to inject a proxy | 01:13 |
| jroll | 01:10:09 jogo | jeblair: RESP BODY: {"forbidden": {"message": "Quota exceeded for instances: Requested 1, but already used 10 of 10 instances", "code": 403}} <- what flavor are you booting? | 01:13 |
| jroll | also, why is openstack CI limited to 10 instances? | 01:13 |
| jogo | jroll: not sure if that was from rax | 01:13 |
| jroll | oh, ok | 01:13 |
| jeblair | yeah, we don't know the actual exception | 01:13 |
| fungi | jroll: that wasn't an example from our system | 01:13 |
| jogo | jroll: jroll that was from the bug from a while ago | 01:13 |
| jroll | right, ok | 01:13 |
| jroll | thanks | 01:13 |
| jogo | jeblair: can you turn novaclient debug logging on? | 01:14 |
| fungi | all we know is we're getting an api response that novaclient 2.14.1 can't interpret | 01:14 |
| jroll | fungi: jeblair: if you can get the actual error message, I can poke things | 01:14 |
| jroll | which isn't the real fix | 01:14 |
| *** wuhg has joined #openstack-infra | 01:14 | |
| jroll | but it might help get things working at least | 01:14 |
| clarkb | we can probably turn on debug logging via the logging config file | 01:14 |
| *** david-lyle_afk has joined #openstack-infra | 01:14 | |
| clarkb | would require a nodepool restart thoug | 01:15 |
| jogo | clarkb: is this happening every time or just sometimes? | 01:15 |
| fungi | might be a better idea to look at nova limits reported for that tenant/region first. i'll see if anything looks maxxed out | 01:15 |
| jogo | fungi: yeah, that is what I am thinking | 01:15 |
| jogo | you may have been leaking some resource | 01:16 |
| jroll | there's known bugs in quotas where we can end up with them out of sync | 01:16 |
| fungi | seeing it in at least dfw and ord | 01:16 |
| jroll | fungi: what's the tenant? | 01:17 |
| jeblair | i have to go now. i have not seen any timeouts like we saw earlier; hopefully we're at least not hitting the errors from earlier | 01:17 |
| fungi | jroll: openstackjenkins | 01:17 |
| fungi | thanks again jeblair! | 01:17 |
| jroll | fungi: have a tenant ID? | 01:17 |
| jroll | it's a number | 01:17 |
| fungi | jroll: yeah, just a sec | 01:18 |
| JayF | jroll: you can get there from that | 01:18 |
| JayF | jroll: I'll show you | 01:18 |
| jroll | fungi: I've got it, thanks | 01:18 |
| JayF | brb, keystone cops operating a cloud | 01:18 |
| fungi | jroll: okay, thanks | 01:18 |
| *** ChuckC has joined #openstack-infra | 01:18 | |
| *** MarkAtwood has quit IRC | 01:19 | |
| clarkb | fungi: are you happy with how things generally look? I should be outside raking before it gets too late | 01:19 |
| fungi | clarkb: yep. i too am about to knock off once i shepherd the last of the alien nodes cleanup through | 01:20 |
| fungi | nova rate-limits and absolute-limits look sane for dfw | 01:21 |
| clarkb | ok I am going to step out and do that. will check back in after | 01:22 |
| jroll | ^ I'm seeing this too | 01:22 |
| jroll | I'm going to see if I can get it reset anyway, just to be sure | 01:22 |
| *** camunoz is now known as camunoz_mtg | 01:22 | |
| *** ChuckC has quit IRC | 01:22 | |
| *** ChuckC has joined #openstack-infra | 01:23 | |
| *** otter768 has joined #openstack-infra | 01:23 | |
| fungi | wondering if maybe we were hitting the api to rapidly and running afoul of an actual api call rate limit | 01:23 |
| fungi | er, too rapidly | 01:24 |
| fungi | because rate-limits and absolute-limits look reasonable on ord too | 01:24 |
| jroll | looks like the rate limits are pretty high | 01:24 |
| *** emagana_ has quit IRC | 01:26 | |
| *** emagana has joined #openstack-infra | 01:27 | |
| *** ivar-laz_ has joined #openstack-infra | 01:27 | |
| *** otter768 has quit IRC | 01:28 | |
| *** sarob has joined #openstack-infra | 01:29 | |
| *** sarob has quit IRC | 01:29 | |
| *** che-arne has quit IRC | 01:29 | |
| *** ivar-lazzaro has quit IRC | 01:31 | |
| *** emagana has quit IRC | 01:31 | |
| *** weshay has joined #openstack-infra | 01:34 | |
| *** ryanpetrello has quit IRC | 01:35 | |
| *** yamamoto has quit IRC | 01:36 | |
| *** sarob has joined #openstack-infra | 01:37 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Set up ssh_known_host based on hostname https://review.openstack.org/136596 | 01:38 |
| *** ivar-laz_ has quit IRC | 01:39 | |
| *** ivar-lazzaro has joined #openstack-infra | 01:39 | |
| *** dims has joined #openstack-infra | 01:42 | |
| *** unicell has joined #openstack-infra | 01:45 | |
| fungi | trusty nodepool seems to have failed to delete 21 nodes in total | 01:46 |
| fungi | giving it a second pass before i switch to checking for alien nodes from precise nodepool | 01:46 |
| clarkb | fungi: try a second pass? there were a handful old nodepool didn't get either but a second pass made it happy | 01:47 |
| *** emagana has joined #openstack-infra | 01:47 | |
| *** achanda has quit IRC | 01:50 | |
| dims | hi all, anyone point me to the mox3 git and bug tracker? | 01:52 |
| *** mrmartin has joined #openstack-infra | 01:53 | |
| *** weshay has quit IRC | 01:54 | |
| fungi | mordred: ^ ? | 01:54 |
| fungi | doesn't seem to be in our gerrit that i can find | 01:55 |
| pleia2 | dims: looks like https://code.google.com/p/pymox/ | 01:55 |
| fungi | also hasn't released in more than a year | 01:55 |
| *** gyee_ has quit IRC | 01:55 | |
| fungi | pleia2: i think that's the pymox page, not the mox3 page | 01:55 |
| pleia2 | hm, or not | 01:55 |
| pleia2 | yeah | 01:55 |
| dims | pleia2: fungi: i was scratching my head as pypi says it came from us - https://pypi.python.org/pypi/mox3/0.7.0 | 01:56 |
| pleia2 | I was following the link on the pypi page | 01:56 |
| *** yaguang has joined #openstack-infra | 01:56 | |
| lifeless | Package Index Owner: openstackci | 01:56 |
| lifeless | thats fairly conclusive | 01:56 |
| fungi | lifeless: conclusive that someone who has the password to the openstackci account uploaded a tarball | 01:56 |
| lifeless | yes | 01:57 |
| dims | weird | 01:57 |
| fungi | dims: it's a monty side project afaik. probably https://github.com/emonty/pymox is as official as you're going to find | 01:57 |
| dims | thanks fungi | 01:57 |
| dims | lifeless: and pleia2 as well | 01:58 |
| *** sarob has quit IRC | 01:59 | |
| dims | fyi, http://stackoverflow.com/questions/8905472/python-unit-test-how-to-use-mox-to-mock-the-gzip-with-statement does not work well with mox3 (just switch "import mox" to "from mox3 import mox" | 01:59 |
| *** dims has quit IRC | 02:00 | |
| *** dims has joined #openstack-infra | 02:00 | |
| *** nosnos has joined #openstack-infra | 02:01 | |
| *** jungleboyj has joined #openstack-infra | 02:02 | |
| *** MarkAtwood has joined #openstack-infra | 02:03 | |
| *** Vek has quit IRC | 02:04 | |
| *** sarob has joined #openstack-infra | 02:05 | |
| *** dims has quit IRC | 02:05 | |
| *** ecosta has quit IRC | 02:05 | |
| jogo | I assume its known zuul's queue is backed up | 02:06 |
| *** MaxV has joined #openstack-infra | 02:07 | |
| *** Daisy has joined #openstack-infra | 02:07 | |
| fungi | oh yes | 02:08 |
| fungi | it's about caught back up though | 02:08 |
| pleia2 | it was a long day in CI land | 02:08 |
| *** sputnik13 has quit IRC | 02:09 | |
| fungi | we ended up switching back to the nodepool server we were running before friday | 02:09 |
| *** yamamoto has joined #openstack-infra | 02:09 | |
| fungi | zuul only has about 1000 waiting jobs now and is churning through about 500 in parallel | 02:09 |
| *** teran has joined #openstack-infra | 02:09 | |
| *** sarob has quit IRC | 02:10 | |
| jeblair | fungi: 2 timeouterrors from 1:20 and 1:22 | 02:11 |
| jeblair | fungi: but i don't see the hysteresis in the graph from earlier | 02:11 |
| *** MaxV has quit IRC | 02:12 | |
| fungi | the frequency of timeouts seems to be considerably less | 02:12 |
| fungi | perhaps it's something about network performance on the trusty nodepool, or somethign to do with the network path between it and zuul which the precise nodepool isn't seeing? | 02:13 |
| *** shashankhegde has quit IRC | 02:13 | |
| jeblair | fungi: that's my best wildly unsupported guess atm | 02:13 |
| *** MarkAtwood has quit IRC | 02:13 | |
| *** teran has quit IRC | 02:15 | |
| fungi | or it's related to time of day and we'll see it pick back up again tomorrow | 02:15 |
| clarkb | new node is oerformance flavor | 02:16 |
| clarkb | and pvhvm image | 02:16 |
| jeblair | then we'll have learned something | 02:16 |
| fungi | indeed | 02:16 |
| clarkb | iz zuul performance yet? | 02:16 |
| jeblair | i think today we did a great job of identifying all the variables we can change; tomorrow we can regroup and decide in what order we want to change them. :) | 02:17 |
| fungi | and hope that in the process we don't get stuck with some new variables | 02:17 |
| clarkb | ++ | 02:17 |
| *** sputnik13 has joined #openstack-infra | 02:18 | |
| fungi | second pass of the 21 straggler nodes on trusty nodepool was able to delete 6, leaving 15 still in the list | 02:18 |
| fungi | quite a lot of them timeout waiting 10 minutes for a response from nova | 02:19 |
| *** banix has joined #openstack-infra | 02:19 | |
| jeblair | 2014-11-25 02:18:51,660 DEBUG zuul.Gearman: Function set_description:jenkins04.openstack.org is not registered | 02:20 |
| jeblair | that is going to contribute a bit of load ^ | 02:20 |
| jeblair | for some reason that function didn't get registered (there should be one for each jenkins master) | 02:20 |
| fungi | looks like the node deletion timeouts are mostly in dfw and some in ord (same places we were getting odd ratelimit responses from novaclient) | 02:21 |
| jeblair | zuul is going to ask gearman for the status list every 5 seconds | 02:21 |
| fungi | jeblair: would restarting the gearman plugin on jenkins04 (or restarting jenkins04) be likely to help? | 02:21 |
| jeblair | fungi: probably... | 02:21 |
| fungi | i'll give it a go | 02:22 |
| jeblair | the stop function for 04 is also not registered | 02:22 |
| clarkb | hold on | 02:22 |
| fungi | sounds like 04 may just be misbehaving then | 02:22 |
| *** emagana has quit IRC | 02:22 | |
| * fungi holds on | 02:22 | |
| jeblair | but it's running jobs.... | 02:22 |
| clarkb | just disable enable a job | 02:22 |
| clarkb | that forces registration | 02:22 |
| fungi | oh, good point. that might help | 02:22 |
| fungi | trying | 02:22 |
| jeblair | clarkb: are you sure that will update the master's registration? (it's worth a shot as a first thing anyway) | 02:22 |
| *** emagana has joined #openstack-infra | 02:23 | |
| clarkb | ya I think it will | 02:23 |
| fungi | done. checking gearman status i'm not seeing it though | 02:24 |
| *** banix has quit IRC | 02:24 | |
| jeblair | yeah, not showing up in 'workers' or 'status' | 02:25 |
| fungi | i can give disabling and enabling the plugin a shot unless clarkb wants to try/check anything first | 02:25 |
| clarkb | no if its not there its not there | 02:25 |
| *** Ryan_Lane has quit IRC | 02:26 | |
| jeblair | fungi: that will probably kill all the running jobs on 04 | 02:26 |
| jeblair | maybe we should leave it be until tomorrow | 02:26 |
| fungi | oh, in that case i'll just put it into prepare for shutdown and gracefully restart it | 02:26 |
| jeblair | fungi: that should work | 02:26 |
| jeblair | fungi: maybe even don't restart it, but rather: | 02:27 |
| *** emagana has quit IRC | 02:27 | |
| fungi | it's preparing for shutdown now. once it quiesces i can try toggling the plugin first | 02:27 |
| jeblair | fungi: prepare for shutdown; wait until idle; disable gearman plugin; enable gearman plugin; check 'status'/'workers'; if ok, exit prepare for shutdown... | 02:27 |
| jeblair | fungi: yep that :) | 02:27 |
| jeblair | fungi: and then if that doesn't work, do a restart | 02:28 |
| fungi | 'zactly | 02:28 |
| *** dkranz has joined #openstack-infra | 02:28 | |
| jeblair | anyway, i'm sure that can wait until later; i think it's probably slowing things down a little, but it's not killing us | 02:29 |
| *** sputnik13 has quit IRC | 02:31 | |
| *** chandankumar has joined #openstack-infra | 02:32 | |
| *** jerryz has joined #openstack-infra | 02:34 | |
| *** camunoz_mtg has quit IRC | 02:36 | |
| *** amcrn has quit IRC | 02:37 | |
| *** patrickeast has quit IRC | 02:40 | |
| *** dkranz has quit IRC | 02:40 | |
| *** banix has joined #openstack-infra | 02:42 | |
| *** vipul has quit IRC | 02:43 | |
| *** esmute has quit IRC | 02:44 | |
| *** mrodden has quit IRC | 02:46 | |
| *** mrmartin has quit IRC | 02:51 | |
| *** jyuso has quit IRC | 02:53 | |
| *** KanagarajM has joined #openstack-infra | 02:54 | |
| mmedvede | asselin: if you're around - I have tested, the subtree --rejoin would not work. There is a workaround to make something similar to --rejoin work, but it shows that rejoin would make the split slower. Ping me, I can show you the test script | 02:55 |
| *** banix has quit IRC | 02:55 | |
| *** mrodden has joined #openstack-infra | 02:56 | |
| *** yamahata has joined #openstack-infra | 02:57 | |
| *** bhunter71 has joined #openstack-infra | 02:58 | |
| *** chandankumar has quit IRC | 03:02 | |
| *** mmaglana has joined #openstack-infra | 03:02 | |
| *** chandankumar has joined #openstack-infra | 03:03 | |
| *** ddieterly has quit IRC | 03:06 | |
| *** ddieterly has joined #openstack-infra | 03:07 | |
| *** eharney has quit IRC | 03:13 | |
| *** hdd has joined #openstack-infra | 03:16 | |
| *** talluri has joined #openstack-infra | 03:18 | |
| *** otter768 has joined #openstack-infra | 03:24 | |
| *** mudassirlatif has quit IRC | 03:24 | |
| openstackgerrit | Merged openstack-infra/nodepool: Clean up 'images' documentation https://review.openstack.org/115168 | 03:25 |
| fungi | okay, remaining cruft from the nodepool cut-back is cleaned up excepting 9 nodes in dfw and 4 in ord which refuse to nova delete | 03:27 |
| *** jerryz has quit IRC | 03:27 | |
| fungi | the last job running on jenkins04 is wrapping up now, so i'll have a go at wanging on it until i get it registered properly in zuul again | 03:27 |
| *** JayJ has joined #openstack-infra | 03:28 | |
| *** chandankumar has quit IRC | 03:28 | |
| *** otter768 has quit IRC | 03:29 | |
| *** nosnos has quit IRC | 03:29 | |
| fungi | indeed, bouncing the gearman plugin did in fact get set_description:jenkins04.openstack.org registered in zuul | 03:30 |
| fungi | cancelling shutdown for jenkins04 now | 03:30 |
| *** ivar-lazzaro has quit IRC | 03:30 | |
| *** Sincler has quit IRC | 03:31 | |
| fungi | and it's running jobs again | 03:31 |
| *** talluri has quit IRC | 03:31 | |
| fungi | no new gearmanclient timeouts for precise nodepool since the two a couple hours ago around 01:20 utc | 03:32 |
| *** JayJ has quit IRC | 03:33 | |
| fungi | and zuul seems to be nearly caught up | 03:33 |
| fungi | think i'm going to take this opportunity to pack it in. night all | 03:33 |
| *** ryanpetrello has joined #openstack-infra | 03:34 | |
| *** KanagarajM has quit IRC | 03:36 | |
| *** teran has joined #openstack-infra | 03:41 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Create puppet-forge credentials on the pypi slave https://review.openstack.org/134835 | 03:43 |
| *** hdd has quit IRC | 03:44 | |
| *** teran has quit IRC | 03:45 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-httpd: Update Rakefile and module info https://review.openstack.org/136959 | 03:58 |
| *** Vek has joined #openstack-infra | 04:00 | |
| *** nosnos has joined #openstack-infra | 04:03 | |
| *** smoser has quit IRC | 04:08 | |
| *** smoser has joined #openstack-infra | 04:09 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Add puppet-httpd to modules.env https://review.openstack.org/136962 | 04:13 |
| *** shashankhegde has joined #openstack-infra | 04:13 | |
| *** julim has quit IRC | 04:18 | |
| *** hdd has joined #openstack-infra | 04:21 | |
| *** smoser has quit IRC | 04:21 | |
| *** MarkAtwood has joined #openstack-infra | 04:23 | |
| *** sputnik13 has joined #openstack-infra | 04:27 | |
| *** jerryz has joined #openstack-infra | 04:28 | |
| *** vipul has joined #openstack-infra | 04:28 | |
| *** bhunter71 has quit IRC | 04:32 | |
| *** MarkAtwood has quit IRC | 04:34 | |
| *** stevemar has quit IRC | 04:36 | |
| *** sputnik13 has quit IRC | 04:38 | |
| *** oomichi has quit IRC | 04:38 | |
| *** koolhead17 has joined #openstack-infra | 04:39 | |
| *** sputnik13 has joined #openstack-infra | 04:40 | |
| *** armax has joined #openstack-infra | 04:40 | |
| *** ddieterly has quit IRC | 04:45 | |
| *** ryanpetrello has quit IRC | 04:45 | |
| *** hdd has quit IRC | 04:54 | |
| *** subscope_ has joined #openstack-infra | 04:55 | |
| *** camunoz has joined #openstack-infra | 04:55 | |
| *** esmute has joined #openstack-infra | 05:04 | |
| *** MaxV has joined #openstack-infra | 05:06 | |
| *** mikedillion has quit IRC | 05:09 | |
| *** sarob has joined #openstack-infra | 05:10 | |
| *** AlexF has joined #openstack-infra | 05:11 | |
| *** jamielennox has quit IRC | 05:11 | |
| *** MaxV has quit IRC | 05:11 | |
| *** jamielennox has joined #openstack-infra | 05:11 | |
| *** teran has joined #openstack-infra | 05:11 | |
| *** viglesias has quit IRC | 05:13 | |
| *** sarob has quit IRC | 05:15 | |
| *** ddieterly has joined #openstack-infra | 05:16 | |
| *** teran has quit IRC | 05:16 | |
| *** ddieterly has quit IRC | 05:17 | |
| *** ddieterl_ has joined #openstack-infra | 05:17 | |
| sweston | mmedvede: still around? I have some working tests on the git commands for the puppet modules | 05:18 |
| sweston | asselin: ^ | 05:18 |
| *** AlexF has quit IRC | 05:19 | |
| *** viglesias has joined #openstack-infra | 05:19 | |
| *** ddieterl_ has quit IRC | 05:22 | |
| *** ryanpetrello has joined #openstack-infra | 05:23 | |
| *** otter768 has joined #openstack-infra | 05:25 | |
| *** chandankumar has joined #openstack-infra | 05:25 | |
| *** subscope_ has quit IRC | 05:25 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Flush out the db api documenation https://review.openstack.org/129405 | 05:27 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Reorganize the python api documentation https://review.openstack.org/129406 | 05:27 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add section on the basic data model https://review.openstack.org/129407 | 05:29 |
| *** otter768 has quit IRC | 05:30 | |
| *** BharatK has joined #openstack-infra | 05:30 | |
| *** nosnos has quit IRC | 05:30 | |
| *** nosnos_ has joined #openstack-infra | 05:30 | |
| harlowja | qq, anyone know why the following would fail? https://review.openstack.org/#/c/136938/ | 05:31 |
| harlowja | is it due to those 'NOT_REGISTERED' ? | 05:31 |
| *** oomichi has joined #openstack-infra | 05:32 | |
| *** Ryan_Lane has joined #openstack-infra | 05:33 | |
| jogo | what user does nodepool set the ssh keys up for in aiopcpu? | 05:34 |
| jogo | jenkins? | 05:37 |
| *** AlexF has joined #openstack-infra | 05:38 | |
| adam_g | jogo, i think 'stack' is the user | 05:38 |
| jogo | http://logs.openstack.org/96/136596/8/experimental/check-tempest-dsvm-neutron-aiopcpu/9770468/logs/screen-n-cpu.txt.gz#_2014-11-25_04_03_12_291 | 05:39 |
| jeblair | jogo: jenkins: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/multinode_setup.sh | 05:39 |
| jogo | jeblair: thanks, that makes sense. So the libvirt URI should be libvirt URI qemu+ssh://jenkins@... ? | 05:40 |
| jogo | or should we set up the keypairs for stack as well? | 05:41 |
| jogo | (in devstack-gate) | 05:41 |
| adam_g | jogo, i think devstack gate already does for stack | 05:41 |
| jogo | adam_g: I may be reading the stacktrace above wrong, but it doesn't appear to be | 05:42 |
| adam_g | jogo, http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n390 | 05:42 |
| jogo | (working) | 05:42 |
| adam_g | jogo, yeah--so i ran into some weirdness last week where it would work in one direction and not the other | 05:42 |
| adam_g | even tho manual testing the connection worked fine. the debug output is not very informative about what user/keypair its trying to use | 05:43 |
| jogo | adam_g: hmm what is stack's home directory | 05:43 |
| adam_g | jogo, /opt/stack/new/ IRRC | 05:43 |
| adam_g | *IIRC | 05:43 |
| *** tnovacik has joined #openstack-infra | 05:45 | |
| jogo | adam_g: yup sudo useradd -U -s /bin/bash -d /opt/stack/new -m stack | 05:46 |
| *** Ryan_Lane has quit IRC | 05:47 | |
| jogo | adam_g: hmm I get a different error in the other direction - http://logs.openstack.org/96/136596/8/experimental/check-tempest-dsvm-neutron-aiopcpu/9770468/logs/10.176.199.103/screen-n-cpu.txt.gz?level=TRACE#_2014-11-25_03_47_52_744 | 05:48 |
| adam_g | jogo, maybe archive /var/log/auth.log to check out what the other ssh server is claiming? | 05:48 |
| jogo | adam_g: well found one bug so far: https://review.openstack.org/#/c/136158/6/devstack-vm-gate.sh,cm | 05:51 |
| adam_g | jogo, eh? | 05:52 |
| jogo | missing a quote | 05:52 |
| adam_g | im not familiar enough with the nodepool host naming scheme. does that look correct? one with 'devstack-trusty-2-node-rax-iad-3210505' and the other '$that+-415'? | 05:52 |
| jogo | so I was writting to /etc/hosts/ on the same node twice | 05:53 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: pulling in unmerged devstack patch https://review.openstack.org/136504 | 05:53 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Set up ssh_known_host based on hostname https://review.openstack.org/136596 | 05:53 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Setup /etc/hosts for aiopcpu https://review.openstack.org/136158 | 05:53 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT COMMIT disable config drive https://review.openstack.org/135831 | 05:53 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/134639 | 05:53 |
| jogo | adam_g: https://review.openstack.org/#/c/136158/6..7/devstack-vm-gate.sh,cm | 05:54 |
| *** ildikov has quit IRC | 05:55 | |
| adam_g | jogo, ah | 05:56 |
| adam_g | i gotta run. good luck | 05:56 |
| *** shashankhegde has quit IRC | 05:58 | |
| jogo | adam_g: o/ thanks | 06:02 |
| *** shashankhegde has joined #openstack-infra | 06:02 | |
| SpamapS | jeblair: Been out for a few days in the mountains. Seems like the thing Clark mentioned was resolved. I don't really know which way is right either. :-P | 06:03 |
| SpamapS | jeblair: (re the gear review from earlier today) | 06:03 |
| *** emagana has joined #openstack-infra | 06:05 | |
| *** renlt has joined #openstack-infra | 06:06 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/136504 | 06:10 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Set up ssh_known_host based on hostname https://review.openstack.org/136596 | 06:10 |
| *** shashankhegde has quit IRC | 06:10 | |
| *** yaguang has quit IRC | 06:13 | |
| *** shashankhegde has joined #openstack-infra | 06:14 | |
| *** ryanpetrello has quit IRC | 06:14 | |
| *** mrmartin has joined #openstack-infra | 06:16 | |
| *** Hal has joined #openstack-infra | 06:18 | |
| *** Hal has quit IRC | 06:18 | |
| *** rushiagr_away is now known as rushiagr | 06:19 | |
| *** yaguang has joined #openstack-infra | 06:21 | |
| *** jyuso has joined #openstack-infra | 06:23 | |
| *** k4n0 has joined #openstack-infra | 06:24 | |
| *** vigneshvar has joined #openstack-infra | 06:25 | |
| *** Ryan_Lane has joined #openstack-infra | 06:25 | |
| *** Daisy has quit IRC | 06:26 | |
| *** Ryan_Lane has quit IRC | 06:27 | |
| *** ildikov has joined #openstack-infra | 06:28 | |
| *** emagana has quit IRC | 06:30 | |
| *** emagana has joined #openstack-infra | 06:31 | |
| *** vigneshvar has quit IRC | 06:33 | |
| *** nosnos_ has quit IRC | 06:34 | |
| *** nosnos has joined #openstack-infra | 06:34 | |
| *** emagana has quit IRC | 06:36 | |
| *** camunoz has quit IRC | 06:36 | |
| *** camunoz has joined #openstack-infra | 06:37 | |
| *** shashankhegde has quit IRC | 06:40 | |
| *** Longgeek has joined #openstack-infra | 06:42 | |
| *** Longgeek has quit IRC | 06:42 | |
| *** Longgeek has joined #openstack-infra | 06:42 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:45 | |
| *** shashankhegde has joined #openstack-infra | 06:45 | |
| *** ddieterly has joined #openstack-infra | 06:46 | |
| *** shashankhegde has quit IRC | 06:47 | |
| *** sputnik13 has quit IRC | 06:48 | |
| *** sputnik13 has joined #openstack-infra | 06:49 | |
| *** ddieterly has quit IRC | 06:51 | |
| *** HACKING-FACEBOOK has quit IRC | 06:53 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:53 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:54 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:54 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:54 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:55 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:55 | |
| *** koolhead17 has quit IRC | 06:55 | |
| *** koolhead17 has joined #openstack-infra | 06:56 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:56 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:57 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:57 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:58 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:58 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:58 | |
| *** HACKING-FACEBOOK has quit IRC | 06:58 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:59 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:59 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 06:59 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:00 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:00 | |
| *** teran has joined #openstack-infra | 07:00 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:01 | |
| *** koolhead17 has quit IRC | 07:01 | |
| *** harlowja is now known as harlowja_away | 07:01 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:01 | |
| *** Longgeek has quit IRC | 07:01 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:01 | |
| *** Longgeek has joined #openstack-infra | 07:02 | |
| *** teran has quit IRC | 07:06 | |
| *** Longgeek has quit IRC | 07:06 | |
| *** koolhead17 has joined #openstack-infra | 07:07 | |
| *** ashaeron has joined #openstack-infra | 07:08 | |
| *** achanda has joined #openstack-infra | 07:08 | |
| *** cody-somerville has quit IRC | 07:09 | |
| *** camunoz has quit IRC | 07:10 | |
| *** AlexF has quit IRC | 07:13 | |
| *** Longgeek has joined #openstack-infra | 07:14 | |
| *** AlexF has joined #openstack-infra | 07:14 | |
| *** AlexF has quit IRC | 07:16 | |
| *** koolhead17 has quit IRC | 07:18 | |
| *** AlexF has joined #openstack-infra | 07:19 | |
| *** HACKING-FACEBOOK has quit IRC | 07:20 | |
| *** cody-somerville has joined #openstack-infra | 07:23 | |
| *** yfried has joined #openstack-infra | 07:24 | |
| *** talluri has joined #openstack-infra | 07:24 | |
| *** isaacb has joined #openstack-infra | 07:25 | |
| *** otter768 has joined #openstack-infra | 07:26 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:27 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:27 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:27 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:28 | |
| *** HACKING-FACEBOOK has quit IRC | 07:28 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:28 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:28 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:29 | |
| *** talluri has quit IRC | 07:29 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:29 | |
| *** ddieterly has joined #openstack-infra | 07:29 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:29 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:30 | |
| *** belmoreira has joined #openstack-infra | 07:30 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:30 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:30 | |
| *** otter768 has quit IRC | 07:30 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:31 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:31 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:31 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:32 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:32 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:32 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:33 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:33 | |
| *** HACKING-FACEBOOK has quit IRC | 07:33 | |
| tchaypo | fatal: Couldn't find remote ref refs/changes/94/136894/2 | 07:33 |
| tchaypo | boo, hiss. doesn't that usually mean someone has uploaded a change as a draft? | 07:33 |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:33 | |
| *** ddieterly has quit IRC | 07:34 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:34 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 07:34 | |
| *** HeOS has quit IRC | 07:35 | |
| *** achanda has quit IRC | 07:35 | |
| *** ashaeron has quit IRC | 07:35 | |
| *** teran has joined #openstack-infra | 07:37 | |
| *** ashaeron has joined #openstack-infra | 07:37 | |
| *** e0ne has joined #openstack-infra | 07:40 | |
| *** achanda has joined #openstack-infra | 07:43 | |
| *** achanda has quit IRC | 07:46 | |
| *** AlexF has quit IRC | 07:47 | |
| *** achanda has joined #openstack-infra | 07:48 | |
| *** armax has quit IRC | 07:49 | |
| *** jamielennox is now known as jamielennox|away | 07:49 | |
| *** obondarev_ has joined #openstack-infra | 07:51 | |
| *** e0ne has quit IRC | 07:53 | |
| *** achanda has quit IRC | 07:53 | |
| *** e0ne has joined #openstack-infra | 07:57 | |
| *** achanda has joined #openstack-infra | 07:58 | |
| *** e0ne has quit IRC | 07:59 | |
| *** jcoufal has joined #openstack-infra | 08:00 | |
| *** ZZelle has quit IRC | 08:02 | |
| *** ZZelle has joined #openstack-infra | 08:03 | |
| *** e0ne has joined #openstack-infra | 08:03 | |
| *** ildikov has quit IRC | 08:03 | |
| *** ala_ has joined #openstack-infra | 08:06 | |
| *** koolhead17 has joined #openstack-infra | 08:07 | |
| *** salv-orlando has joined #openstack-infra | 08:08 | |
| *** e0ne has quit IRC | 08:08 | |
| *** koolhead17 has quit IRC | 08:10 | |
| *** achanda has quit IRC | 08:10 | |
| *** arxcruz has joined #openstack-infra | 08:10 | |
| *** koolhead17 has joined #openstack-infra | 08:11 | |
| *** e0ne has joined #openstack-infra | 08:12 | |
| *** amuller has joined #openstack-infra | 08:13 | |
| *** ci-testing has quit IRC | 08:14 | |
| *** btran has joined #openstack-infra | 08:14 | |
| *** ddieterly has joined #openstack-infra | 08:16 | |
| *** jistr has joined #openstack-infra | 08:18 | |
| *** yamamoto has quit IRC | 08:18 | |
| *** e0ne has quit IRC | 08:18 | |
| *** doude has joined #openstack-infra | 08:19 | |
| *** ildikov has joined #openstack-infra | 08:19 | |
| *** bo_sh has joined #openstack-infra | 08:20 | |
| *** ddieterly has quit IRC | 08:21 | |
| *** bo_sh has left #openstack-infra | 08:24 | |
| *** HACKING-FACEBOOK has quit IRC | 08:28 | |
| *** koolhead17 has quit IRC | 08:29 | |
| *** sputnik13 has quit IRC | 08:33 | |
| *** afazekas has joined #openstack-infra | 08:34 | |
| *** doude has quit IRC | 08:35 | |
| *** talluri has joined #openstack-infra | 08:35 | |
| *** talluri has quit IRC | 08:40 | |
| *** talluri has joined #openstack-infra | 08:42 | |
| *** tnovacik has quit IRC | 08:42 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 08:42 | |
| *** ddieterly has joined #openstack-infra | 08:45 | |
| *** mkerrin has quit IRC | 08:48 | |
| *** ddieterly has quit IRC | 08:50 | |
| *** HeOS has joined #openstack-infra | 08:51 | |
| *** talluri has quit IRC | 08:55 | |
| *** talluri has joined #openstack-infra | 08:58 | |
| *** jpich has joined #openstack-infra | 09:01 | |
| *** MaxV has joined #openstack-infra | 09:01 | |
| openstackgerrit | Mehdi Abaakouk proposed openstack-infra/project-config: Run ceilometer rally scenarios https://review.openstack.org/132650 | 09:05 |
| *** obondarev_ is now known as obondarev | 09:06 | |
| openstackgerrit | Mehdi Abaakouk proposed openstack-infra/project-config: Run ceilometer rally scenarios https://review.openstack.org/132650 | 09:07 |
| *** katyafervent is now known as katyafervent_awa | 09:08 | |
| *** katyafervent_awa is now known as katyafervent | 09:09 | |
| *** yamamoto has joined #openstack-infra | 09:11 | |
| *** yamahata has quit IRC | 09:11 | |
| *** jedimike has joined #openstack-infra | 09:12 | |
| *** HACKING-FACEBOOK has quit IRC | 09:14 | |
| *** adalbas has joined #openstack-infra | 09:14 | |
| *** derekh has joined #openstack-infra | 09:14 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 09:16 | |
| *** ihrachyshka has joined #openstack-infra | 09:18 | |
| *** yamamoto has quit IRC | 09:18 | |
| *** adalbas has quit IRC | 09:21 | |
| *** andreykurilin_ has joined #openstack-infra | 09:22 | |
| *** yfried is now known as yfried|afk | 09:24 | |
| *** matel has joined #openstack-infra | 09:24 | |
| *** renlt has quit IRC | 09:25 | |
| *** yfried|afk is now known as yfried | 09:25 | |
| *** e0ne has joined #openstack-infra | 09:25 | |
| *** otter768 has joined #openstack-infra | 09:27 | |
| *** luqas has joined #openstack-infra | 09:27 | |
| *** ashaeron has quit IRC | 09:31 | |
| *** otter768 has quit IRC | 09:31 | |
| *** oomichi has quit IRC | 09:33 | |
| *** jistr has quit IRC | 09:33 | |
| *** HACKING-FACEBOOK has quit IRC | 09:35 | |
| *** achuprin_ has quit IRC | 09:41 | |
| *** vdo has joined #openstack-infra | 09:41 | |
| *** ashaeron has joined #openstack-infra | 09:42 | |
| *** achuprin_ has joined #openstack-infra | 09:44 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 09:44 | |
| *** dizquierdo has joined #openstack-infra | 09:47 | |
| *** belmoreira has quit IRC | 09:48 | |
| *** BobBall_AWOL is now known as BobBall | 09:48 | |
| *** ashaeron has quit IRC | 09:59 | |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/project-config: Setting up a StoryBoard python client https://review.openstack.org/136004 | 09:59 |
| *** andreykurilin_ has quit IRC | 10:00 | |
| *** andreykurilin_ has joined #openstack-infra | 10:00 | |
| *** salv-orlando has quit IRC | 10:00 | |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/project-config: Setting up a StoryBoard python client https://review.openstack.org/136004 | 10:01 |
| *** salv-orlando has joined #openstack-infra | 10:02 | |
| *** ashaeron has joined #openstack-infra | 10:03 | |
| *** jistr has joined #openstack-infra | 10:03 | |
| *** nadya has joined #openstack-infra | 10:05 | |
| *** mpavlase has joined #openstack-infra | 10:05 | |
| *** andreykurilin_ has quit IRC | 10:05 | |
| *** nadya is now known as Guest4876 | 10:05 | |
| *** andreykurilin_ has joined #openstack-infra | 10:06 | |
| *** apevec has joined #openstack-infra | 10:06 | |
| apevec | what does it mean when jenkins job is NOT_REGISTERED e.g. https://review.openstack.org/136899 ? | 10:08 |
| *** nosnos has quit IRC | 10:08 | |
| *** nosnos has joined #openstack-infra | 10:09 | |
| *** bigo has joined #openstack-infra | 10:11 | |
| *** mkerrin has joined #openstack-infra | 10:11 | |
| bigo | hi, all , when i giva a reivew on the code , why is there no mail send out ? | 10:12 |
| *** marcusvrn has joined #openstack-infra | 10:12 | |
| bigo | i gave a code review and comments , | 10:14 |
| bigo | the review and comments should be send to all reviewer via email , | 10:15 |
| Mithrandir | why do you believe there's no mail sent? | 10:15 |
| Mithrandir | afaik it's just not sent to yourself. | 10:15 |
| *** jp_at_hp has joined #openstack-infra | 10:16 | |
| bigo | i've watched the projct | 10:16 |
| bigo | all changes and commented should be send mail to me | 10:16 |
| *** koolhead17 has joined #openstack-infra | 10:17 | |
| *** marcusvrn has quit IRC | 10:17 | |
| bigo | but my comments on code review is not send out mail to the reviewer | 10:17 |
| *** marcusvrn has joined #openstack-infra | 10:18 | |
| bigo | i did recieved mail on code reviewed and comments by others | 10:18 |
| *** yamamoto has joined #openstack-infra | 10:19 | |
| tsufiev | SergeyLukjanov, hello! do you know what's happening with zuul? 2 approved patches here https://review.openstack.org/#/q/owner:%22Vlad+Okhrimenko%22+status:open,n,z but no sign of them in http://status.openstack.org/zuul/ | 10:21 |
| *** yfried is now known as yfried|afk | 10:22 | |
| *** yamamoto has quit IRC | 10:24 | |
| *** apevec has quit IRC | 10:25 | |
| tchaypo | bigo: https://review.openstack.org/#/settings/projects has settings where you can control what notifications you get for projects you watch | 10:32 |
| tchaypo | bigo: https://review.openstack.org/#/settings/preferences has an option to CC you on comments you write | 10:33 |
| *** rlandy has joined #openstack-infra | 10:33 | |
| tchaypo | bigo: if you think Gerrit should support some other behaviour, there's a "report bug" on the bottom right | 10:33 |
| *** mpaolino has joined #openstack-infra | 10:37 | |
| *** ldnunes has joined #openstack-infra | 10:37 | |
| *** markus_z has joined #openstack-infra | 10:39 | |
| *** erlon has joined #openstack-infra | 10:41 | |
| *** vigneshvar has joined #openstack-infra | 10:41 | |
| *** yfried|afk is now known as yfried | 10:42 | |
| bigo | tchaypo:thanks , | 10:42 |
| bigo | other reviewers suppose to recieve email when i give a code review comments,right? | 10:43 |
| bigo | tchaypo:other reviewers suppose to recieve email notification when i give a code review comments,right? | 10:44 |
| chandankumar | fungi, ping | 10:44 |
| bigo | whethere i'm a new comer or a core reviewer | 10:45 |
| openstackgerrit | Tim Kelsey proposed openstack/requirements: Bumping PyKMIP version from 0.1.0 to 0.2.0 https://review.openstack.org/137016 | 10:46 |
| *** isaacb has quit IRC | 10:50 | |
| *** Guest4876 has quit IRC | 10:54 | |
| *** isaacb has joined #openstack-infra | 10:56 | |
| *** teran has quit IRC | 10:57 | |
| *** salv-orlando has quit IRC | 10:57 | |
| *** vdo has quit IRC | 10:58 | |
| *** salv-orlando has joined #openstack-infra | 11:02 | |
| *** luqas has quit IRC | 11:03 | |
| gilliard | Hi. This patch is approved a few hours ago but not merged yet and I don't see it anywhere in zuul: https://review.openstack.org/#/c/136266/ | 11:05 |
| gilliard | I've probably missed something easy but shouldn't it be in the check or gate queue after a +a? | 11:05 |
| *** unicell has quit IRC | 11:06 | |
| *** isaacb has quit IRC | 11:06 | |
| *** koolhead17 has quit IRC | 11:07 | |
| *** ildikov has quit IRC | 11:09 | |
| *** salv-orlando has quit IRC | 11:11 | |
| *** salv-orlando has joined #openstack-infra | 11:11 | |
| *** andreykurilin_ has quit IRC | 11:12 | |
| mpavlase | Hi, I'd like to ask, what does mean NOT_REGISTERED result in Jenkins gate (tempest project) from several jobs? I did recheck (is in progress now) and the same subset of jobs got NOT_REGISTERED again... | 11:15 |
| *** rfolco has joined #openstack-infra | 11:16 | |
| *** arxcruz has quit IRC | 11:17 | |
| *** talluri has quit IRC | 11:17 | |
| *** luqas has joined #openstack-infra | 11:17 | |
| *** k4n0 has quit IRC | 11:20 | |
| *** dims has joined #openstack-infra | 11:20 | |
| *** yfried has quit IRC | 11:20 | |
| *** rfolco has quit IRC | 11:20 | |
| *** yamamoto has joined #openstack-infra | 11:21 | |
| *** yfried has joined #openstack-infra | 11:22 | |
| *** ildikov has joined #openstack-infra | 11:22 | |
| *** rfolco has joined #openstack-infra | 11:25 | |
| *** koolhead17 has joined #openstack-infra | 11:26 | |
| *** vigneshvar has quit IRC | 11:27 | |
| *** otter768 has joined #openstack-infra | 11:27 | |
| *** isaacb has joined #openstack-infra | 11:28 | |
| *** vigneshvar has joined #openstack-infra | 11:30 | |
| *** otter768 has quit IRC | 11:32 | |
| *** katyafervent is now known as katyafervent_awa | 11:33 | |
| *** dizquierdo is now known as dizquierdo_afk | 11:34 | |
| *** yfried is now known as yfried|afk | 11:35 | |
| *** aysyd has joined #openstack-infra | 11:37 | |
| *** che-arne has joined #openstack-infra | 11:38 | |
| *** marcusvrn1 has joined #openstack-infra | 11:38 | |
| *** marcusvrn has quit IRC | 11:39 | |
| *** yfried|afk is now known as yfried | 11:40 | |
| *** yamamoto has quit IRC | 11:44 | |
| yolanda | hi, i have a question about recheck/reverify comments. Are they supposed to do the same action? I got the feedback from users, that recheck is retrying the check queue, and reverify is retrying the gate queue, but looking at the code i don't see where is that achieved | 11:47 |
| *** che-arne has quit IRC | 11:47 | |
| *** yaguang has quit IRC | 11:47 | |
| *** isaacb has quit IRC | 11:48 | |
| *** vigneshvar has quit IRC | 11:49 | |
| sdague | yeh, something is wrong with all the icehouse nodes | 11:51 |
| sdague | yolanda: they do exactly the same thing | 11:53 |
| yolanda | sdague, that's what i thought | 11:53 |
| kashyap | chandankumar, A gentl note - blank pings are not useful when people come back to check scroll. You might want to ping with context - http://blogs.gnome.org/markmc/2014/02/20/naked-pings/ | 11:53 |
| sdague | jhesketh: you back from vacation yet? | 11:53 |
| markus_z | mpavlase: I was told that this is an issue with the gates not your review. I ran into the same issue. I hope that some information about that will be provided in this channel in the near future. | 11:54 |
| * sdague tries to figure out which infra-core can look into the no icehouse jobs registered issue | 11:54 | |
| sdague | which probably means.... precise nodes don't exit? | 11:54 |
| markus_z | sdague: thanks | 11:55 |
| *** ildikov has quit IRC | 11:55 | |
| mpavlase | markus_z: thanks, at least good to know something (I'm not only one who hit this issue) that nothing | 11:56 |
| mpavlase | please ping me, when you notice something new about that | 11:56 |
| markus_z | mpavlase: sure, I'll do | 11:57 |
| mpavlase | markus_z: thanks! | 11:58 |
| *** ildikov has joined #openstack-infra | 11:59 | |
| *** adalbas has joined #openstack-infra | 12:01 | |
| *** BharatK has left #openstack-infra | 12:02 | |
| *** vigneshvar has joined #openstack-infra | 12:02 | |
| *** yfried is now known as yfried|afk | 12:02 | |
| *** markus_z has quit IRC | 12:03 | |
| *** teran has joined #openstack-infra | 12:05 | |
| *** teran_ has joined #openstack-infra | 12:06 | |
| *** isaacb has joined #openstack-infra | 12:06 | |
| *** dizquierdo_afk has quit IRC | 12:06 | |
| *** katyafervent_awa is now known as katyafervent | 12:06 | |
| *** yfried|afk is now known as yfried | 12:08 | |
| *** hdd has joined #openstack-infra | 12:08 | |
| yolanda | sdague, user says in the wiki there is that info "To re-run the check jobs (before a change has been approved), leave a comment with the form "recheck bug ####"." / | 12:08 |
| yolanda | "To re-run the gate jobs (after a change has been approved), leave a comment with the form "reverify bug ####"." . What's the point for that? just track the information on where the process fails? | 12:08 |
| *** teran has quit IRC | 12:10 | |
| sdague | yolanda: there is probably that info in the wiki, it is horribly out of date | 12:11 |
| yolanda | sdague so yes, the reality is that both commands do the same, as spotted on layout.yaml, and the users can choose the command they want to use, right? | 12:12 |
| *** pblaho has joined #openstack-infra | 12:13 | |
| sdague | yolanda: yeh, it should be collapsed to 1 command. It needs a volunteer to go clean up all the places reverify is referenced. | 12:13 |
| yolanda | sdague, but even on the pipeline reverify is used, this command is deprecated? | 12:15 |
| *** talluri has joined #openstack-infra | 12:16 | |
| *** markus_z has joined #openstack-infra | 12:16 | |
| sdague | yolanda: yes, it's deprecated | 12:21 |
| yolanda | sdague, ok, thx for the feedback | 12:21 |
| *** salv-orlando has quit IRC | 12:25 | |
| *** dizquierdo has joined #openstack-infra | 12:25 | |
| *** arxcruz has joined #openstack-infra | 12:29 | |
| *** vigneshvar has quit IRC | 12:30 | |
| *** luqas has quit IRC | 12:31 | |
| *** jistr is now known as jistr|english | 12:32 | |
| *** dizquierdo has quit IRC | 12:39 | |
| *** che-arne has joined #openstack-infra | 12:41 | |
| *** koolhead17 has quit IRC | 12:41 | |
| *** AJaeger has joined #openstack-infra | 12:41 | |
| *** AJaeger has joined #openstack-infra | 12:41 | |
| *** weshay has joined #openstack-infra | 12:42 | |
| *** Longgeek_ has joined #openstack-infra | 12:44 | |
| *** yamamoto has joined #openstack-infra | 12:44 | |
| *** koolhead17 has joined #openstack-infra | 12:44 | |
| *** Longgeek has quit IRC | 12:44 | |
| *** bradjones has quit IRC | 12:45 | |
| *** bradjones has joined #openstack-infra | 12:45 | |
| *** yamamoto has quit IRC | 12:49 | |
| *** hdd has quit IRC | 12:49 | |
| gilliard | This change has been +A twice and both times I don't see anything happening (no gate fail, no merge fail, just nothing) https://review.openstack.org/#/c/136266/ - What can I do? | 12:54 |
| *** teran_ has quit IRC | 12:54 | |
| *** jcoufal has quit IRC | 12:56 | |
| *** jcoufal has joined #openstack-infra | 12:57 | |
| *** teran has joined #openstack-infra | 12:58 | |
| *** teran_ has joined #openstack-infra | 12:58 | |
| *** che-arne has quit IRC | 12:59 | |
| *** mbacchi has joined #openstack-infra | 13:01 | |
| *** pc_m has joined #openstack-infra | 13:02 | |
| *** teran has quit IRC | 13:02 | |
| *** pc_m has quit IRC | 13:03 | |
| *** che-arne has joined #openstack-infra | 13:03 | |
| *** pc_m has joined #openstack-infra | 13:03 | |
| *** dprince has joined #openstack-infra | 13:06 | |
| AJaeger | gilliard, just try a simple "recheck" | 13:07 |
| gilliard | Thanks AJaeger - it's popped into the check queue now. Will it go to the gate automatically after that? | 13:08 |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| openstackgerrit | Jaroslav Henner proposed openstack-infra/jenkins-job-builder: Allow multiple comment-added events in gerrit trig. https://review.openstack.org/124568 | 13:08 |
| AJaeger | gilliard, it should - let's wait and see... | 13:09 |
| gilliard | :) Thanks. | 13:09 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: make nova functional tests voting https://review.openstack.org/137050 | 13:09 |
| *** nadya_ has joined #openstack-infra | 13:10 | |
| *** eharney has joined #openstack-infra | 13:11 | |
| *** kgiusti has joined #openstack-infra | 13:11 | |
| *** mpaolino has quit IRC | 13:12 | |
| *** yamamoto has quit IRC | 13:13 | |
| *** mdbooth has quit IRC | 13:15 | |
| *** marcusvrn1 has quit IRC | 13:16 | |
| *** JayJ has joined #openstack-infra | 13:17 | |
| *** chandankumar has quit IRC | 13:21 | |
| sdague | gilliard: yeh, I expect that some things got lost during gerrit / zuul restarts | 13:21 |
| jklare | hi, somebody there who could help me with implementing nonvoting tests for the openstack-chef project? | 13:22 |
| *** JayJ has quit IRC | 13:22 | |
| jklare | i basically need a hint where to get started or where to look for more info | 13:22 |
| *** JayJ has joined #openstack-infra | 13:22 | |
| *** Longgeek has joined #openstack-infra | 13:24 | |
| *** markus_z has quit IRC | 13:25 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard-webclient: Story project is defaulted to the current project if available https://review.openstack.org/137057 | 13:25 |
| *** bswartz has quit IRC | 13:26 | |
| AJaeger | jklare, what is your challenge: Defining a job - or making it non-voting? | 13:26 |
| *** ddieterly has joined #openstack-infra | 13:26 | |
| *** JayJ has quit IRC | 13:27 | |
| jklare | i basically want to run all existing jobs for openstack-chef also on a ubuntu14.04 instance but make the result non-voting | 13:27 |
| *** Longgeek_ has quit IRC | 13:27 | |
| jklare | currently they are locked to precise | 13:27 |
| jklare | because the cookbooks dont work on trusty | 13:27 |
| *** JayJ has joined #openstack-infra | 13:27 | |
| jklare | but to get started with implementing support for trusty we decided it might be a good idea to start tests as nonvoting | 13:28 |
| AJaeger | jklare, create jobs with a different name and change zuul/layout.yaml to have "voting: false" for the jobs defined. | 13:28 |
| AJaeger | jklare, yes, good idea. | 13:28 |
| *** otter768 has joined #openstack-infra | 13:28 | |
| AJaeger | jklare, so you want to have a gate-openstack-chef-repo job for trusty? | 13:29 |
| *** koolhead17 has quit IRC | 13:29 | |
| jklare | yep | 13:29 |
| *** yaguang has joined #openstack-infra | 13:29 | |
| sdague | gilliard: yeh, it looks like a bunch of stuff was lost. I just mass rechecked the things that jenkins lost | 13:30 |
| *** koolhead17 has joined #openstack-infra | 13:30 | |
| sdague | jklare: typically if you know things don't work, you should put them in the experimental queue | 13:30 |
| sdague | so they aren't burning test nodes on unrelated patches | 13:30 |
| AJaeger | job is in jenkins/jobs/chef-jobs.yaml. You could copy that job with a new name and then add it to zuul/layout.yaml | 13:30 |
| AJaeger | sdague, yeah, indeed. First in experimental, then non-voting, then voting. | 13:31 |
| jklare | this experimental queue sound good | 13:31 |
| jklare | ok so let me check these files quickly | 13:31 |
| sdague | jklare: jobs in experimental queue are only run on a patch if you add 'check experimental' as a comment | 13:31 |
| *** JayJ has quit IRC | 13:32 | |
| *** JayJ has joined #openstack-infra | 13:32 | |
| *** otter768 has quit IRC | 13:33 | |
| *** koolhead17 has quit IRC | 13:34 | |
| *** dizquierdo has joined #openstack-infra | 13:34 | |
| jklare | so i would create something like chef-jobs-experimental.yaml in jenkins/jobs and then run it from the zuul/layout.yaml as non-voting right | 13:35 |
| *** jeson has joined #openstack-infra | 13:35 | |
| jklare | ? | 13:35 |
| AJaeger | jklare, http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n274 | 13:36 |
| AJaeger | You add the job to the experimental queue - not the check or gate one | 13:36 |
| AJaeger | No need to make it non-voting | 13:36 |
| *** marcusvrn has joined #openstack-infra | 13:36 | |
| AJaeger | jklare, send a patch and we'll help polishing it up ;) | 13:38 |
| jklare | AJaeger: i am on it ;) thx for the help so far | 13:38 |
| *** markus_z has joined #openstack-infra | 13:38 | |
| yfried | anyone knows what NOT_REGISTERED means in jenkins results? https://review.openstack.org/#/c/136981/ | 13:40 |
| gilliard | sdague: thanks. That was overnight for me, so I didn't know what had gone on. | 13:41 |
| AJaeger | yfried, on icehouse jobs? Broken setup of jenkins ;( | 13:41 |
| AJaeger | yfried, it means that a job should be available but somehow there's nobody registered to take it :( | 13:41 |
| *** mpaolino has joined #openstack-infra | 13:41 | |
| yfried | AJaeger: it happend twice. should I recheck? | 13:41 |
| tchaypo | bigo: yep, that's my understanding - anyone listed in the "reviewer" section + anyone who has starred the issue + anyone watching the project with appropriate notifications set up should all get an email for each review (except their own, unless they also turned on that setting) | 13:43 |
| AJaeger | yfried, those are icehouse related - wait until some of the infra experts have it fixed. | 13:43 |
| *** IvanBerezovskiy has quit IRC | 13:45 | |
| yolanda | sdague, what needs to be cleaned in order to get rid of the reverify comment? zuul layout.yaml in project-config, doc in wiki? what else? in the past, it was working in a different way? seems that users here have the opinion that recheck is for retrying the check queue, and reverify is for retrying the gate queue | 13:46 |
| AJaeger | yolanda, that was the case in the past. Nowadays reverify and recheck do the same - both enqueue in the check queue | 13:48 |
| AJaeger | and if the patch is approved, it goes afterwards in the gate | 13:48 |
| *** nosnos has quit IRC | 13:48 | |
| *** enikanorov__ has quit IRC | 13:48 | |
| *** nosnos has joined #openstack-infra | 13:49 | |
| jeson | Hello | 13:51 |
| jeson | When I setup the 3rd CI master, I got the following errors: | 13:51 |
| jeson | Error: Error: /Stage[main]/Os_ext_testing::Master/File[/etc/zuul/openstack_functions.py]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///modules/openstack_project/zuul/openstack_functions.py | 13:51 |
| jeson | Why this file openstack_functions.py not in system-config? | 13:52 |
| jklare | AJaeger: how can is set the node option for these experimental tests to trusty? https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L508 its set to precise by default for all these jobs if i understand this correctly | 13:52 |
| *** salv-orlando has joined #openstack-infra | 13:53 | |
| AJaeger | jklare, put for this specific job after the general regex a new rule that uses set_node_options_default_trusty | 13:53 |
| *** nosnos has quit IRC | 13:54 | |
| asselin | mmedvede, sweston I'm around now | 13:54 |
| AJaeger | jklare, and specify "bare-trusty" in the node line of the job | 13:54 |
| *** otherwiseguy has joined #openstack-infra | 13:55 | |
| yfried | AJaeger: is there a bug open for this? | 13:56 |
| *** unicell has joined #openstack-infra | 13:56 | |
| AJaeger | yfried, I don't know. | 13:56 |
| mmedvede | asselin: yesterday I used this script to test if --rejoin would work to speed up the split: http://paste.openstack.org/show/138180/ | 13:57 |
| *** markus_z has quit IRC | 13:57 | |
| sdague | yolanda: I think that sounds about right | 13:57 |
| *** enikanorov_ has joined #openstack-infra | 13:57 | |
| *** dims has quit IRC | 13:57 | |
| *** dizquierdo has quit IRC | 13:58 | |
| mmedvede | asselin: it did show that it would not work with vanilla --rejoin, and if I try to mimic the rejoin mysel, it does take longer with it, than without | 13:58 |
| *** dims has joined #openstack-infra | 13:58 | |
| *** yamahata has joined #openstack-infra | 13:58 | |
| *** bswartz has joined #openstack-infra | 13:58 | |
| openstackgerrit | Balazs Gibizer proposed openstack-infra/project-config: Increase gate-{name}-pylint job timeout to 40 mins https://review.openstack.org/137072 | 13:58 |
| jeson | Does anyone have seen this error when set up 3rd CI: | 13:59 |
| yolanda | AJaeger, what was the reason for that change of behaviour? | 14:00 |
| asselin | mmedvede, I did notice it wasn't as 'fast' as I expected. I was thinking it is because there are too many sub-trees. | 14:00 |
| *** amuller has quit IRC | 14:00 | |
| asselin | mmedvede, I was thinking of creating a clean clone of system-config for each subtree, and try that. | 14:00 |
| AJaeger | yolanda, the infra team changed the way jobs are queued. Sorry, don't have the full background on it. | 14:00 |
| AJaeger | yolanda, this was more than 6 months ago AFAIR | 14:00 |
| *** bhunter71 has joined #openstack-infra | 14:00 | |
| jklare | AJaeger: so basically i want to run all the tests defined in check (https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L422-425) also on the experimental queue but with {node} defined as bare-trusty | 14:01 |
| asselin | mmedvede, but in my test, it took 2 hours for the first, and 30 minutes for the subsequent | 14:01 |
| mmedvede | asselin: I think the culprit with git subtree split is that it would always go through ALL commit history for every split. The only way to speed it up is to make something custom. | 14:02 |
| mmedvede | asselin: It could be because on subsequent runs you had it cached, so it did not have to do as many disk reads | 14:02 |
| asselin | mmedvede, here it says it won't go through all the history: https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt#L234 | 14:02 |
| jklare | AJaeger: can i somehow only set this node = bare-trusty for the experimental queue | 14:03 |
| jklare | ? | 14:03 |
| AJaeger | jklare, sorry, can't help with that question.... | 14:03 |
| AJaeger | jklare, in another meeting now... | 14:03 |
| jklare | AJaeger: ok ty anyway | 14:03 |
| *** jcoufal has quit IRC | 14:03 | |
| asselin | mmedvede, in your tests, did you observe that it produces the incorrect result? | 14:04 |
| *** jcoufal has joined #openstack-infra | 14:04 | |
| mmedvede | asselin: in my test, the --rejoin prevented the further splits altogether, it failed on first update | 14:04 |
| asselin | mmedvede, :( | 14:05 |
| *** talluri has quit IRC | 14:05 | |
| mmedvede | asselin: It could work if you have separate repository per split, I did not test that | 14:05 |
| *** dustins has joined #openstack-infra | 14:06 | |
| *** dkliban_afk is now known as dkliban | 14:07 | |
| asselin | mmedvede, ok, I'll play around with it some more | 14:07 |
| sweston | asselin: mmedvede: Good morning! So I have a script for you all to try out: https://github.com/Triniplex/puppet-module-split | 14:09 |
| *** otherwiseguy has quit IRC | 14:09 | |
| *** changbl has quit IRC | 14:09 | |
| jklare | hi, anybody out there who knows if one can set the node to run on only for a specific queue? | 14:09 |
| sweston | asselin: mmedvede: let me know when you are ready, and i can walk you through it | 14:10 |
| *** koolhead17 has joined #openstack-infra | 14:10 | |
| asselin | sweston, reading through it now | 14:10 |
| *** markus_z has joined #openstack-infra | 14:10 | |
| fungi | jklare: the parameters for a given job are immutable. instead you create two different jobs from a common template with the node parameterized, and then add those differently named jobs to different pipelines in zuul | 14:10 |
| sweston | asselin: ok, I'll give you a few minutes :-) | 14:11 |
| sdague | fungi: morning | 14:12 |
| sdague | fungi: so all icehouse dsvm jobs are now NOT_REGISTERED | 14:12 |
| sweston | asselin: the test is only with one repo, however it will be really trivial to add a loop to iterate through all the modules | 14:12 |
| sdague | which means tempest / python libraries / most of oslo is blocked | 14:12 |
| *** jistr|english is now known as jistr | 14:13 | |
| fungi | sdague: yes, oddly we have bare-precise and devstack-precise nodes ready in nodepool. i'm looking in gearman now to see if i can tell what's still not right | 14:13 |
| *** ryanpetrello has joined #openstack-infra | 14:13 | |
| fungi | the chances of me getting to, well, anything i carried over on my to do list are seeming slim at best | 14:14 |
| *** yamamoto has joined #openstack-infra | 14:14 | |
| *** dkranz has joined #openstack-infra | 14:14 | |
| jedimike | clarkb, I found the exact problem with the new gitpython and zuul :) there are a few ways we could fix it, emailing openstack-infra later so we can decide which one is the right way | 14:15 |
| asselin | sweston, this won't pull in all of master back into the subtree? https://github.com/Triniplex/puppet-module-split/blob/master/module_split.sh#L70 | 14:16 |
| *** otherwiseguy has joined #openstack-infra | 14:16 | |
| *** Sincler has joined #openstack-infra | 14:16 | |
| sweston | asselin: nope, right before that we switch to the module branch created earlier, which only includes the subtree we're interested in | 14:17 |
| asselin | sweston, I see | 14:17 |
| fungi | nevermind, there are no devstack-precise nodes (i was mistaking devstack-precise-dib nodes for them) | 14:18 |
| *** dustins has quit IRC | 14:18 | |
| *** yamamoto has quit IRC | 14:18 | |
| fungi | nodepool's trying to build one right now. looking to see if we have any detail on why none have booted to completion | 14:18 |
| sweston | asselin: once the initial split runs on each of the modules, just pass the -s flag in, and the merge is extremely fast | 14:19 |
| jeblair | fungi: wow, so none built since yesterday? | 14:19 |
| fungi | jeblair: seems that way | 14:19 |
| *** dustins has joined #openstack-infra | 14:20 | |
| *** ddieterly has quit IRC | 14:20 | |
| *** dustins has quit IRC | 14:20 | |
| jeblair | fungi: istr that lack of a certain node type was one of the issues we were dealing with early yesterday, but we chalked it up to not receiving demand and the allocator's behavior there. was that node type devstack-precise by any chance? | 14:21 |
| fungi | LaunchStatusException with status: ERROR for the first couple i've looked at | 14:21 |
| asselin | sweston,"-s ours" so that's why it didn't work for me yesterday. I see | 14:21 |
| fungi | jeblair: it was. we saw similar issues with bare-precise but after half a dozen tries it seemed to finally successfully launch one | 14:21 |
| jeblair | fungi: hrm. we may have a lot of things that are just a little broken. :/ | 14:22 |
| fungi | jeblair: nodepool seems to only want to try launching devstack-precise nodes in rax-iad | 14:22 |
| jeblair | fungi: devstack-precise-check is configured for iad only, but devstack-precise is configured for all of them | 14:24 |
| *** rushiagr is now known as rushiagr_away | 14:24 | |
| *** Hal has joined #openstack-infra | 14:24 | |
| *** Hal is now known as Guest27016 | 14:24 | |
| fungi | weird. i haven't seen any devstack-precise-check nodes at all | 14:24 |
| fungi | min-ready is 0 for those | 14:25 |
| jeblair | fungi: we don't use them, but they are still in the config | 14:25 |
| sweston | asselin: yes, and the other step that is crucial is switching the config repo to the target subtree, doing a merge from upstream, and then pulling those changes into the split repo | 14:25 |
| jeblair | yeah | 14:25 |
| jeblair | ianw: ping | 14:25 |
| fungi | it's picking the last entry in the providers list for the devstack-precise label, fwiw | 14:26 |
| *** mriedem has joined #openstack-infra | 14:26 | |
| fungi | we have built other precise-based nodes in rax-iad | 14:28 |
| sweston | asselin: this will make sure we don't have to pull down the history every time, and preserves the history on the split repo, allowing a clean merge | 14:28 |
| jklare | fungi: so i already have these job-templates (https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/chef-jobs.yaml) which are grouped and assigned to the queues in this project-template (https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L421-428). The node parameter is set here (https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L508-509) on basis of the singl | 14:28 |
| jklare | e job templates. How can i run all these chef-jobs (https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L3997-L4115) with node=precise for the check and gate queues and node=trusty for the experimental queue (without duplicating a lot of code)? | 14:28 |
| asselin | sweston, so what's next, run it on all the modules? | 14:30 |
| *** Ark has joined #openstack-infra | 14:30 | |
| *** Ark has quit IRC | 14:31 | |
| sweston | yup. I wanted to run it by you first to make sure it made sense to you. let me update the code to run on all the modules. | 14:31 |
| fungi | jklare: i can have a look some time when i'm not firefighting, otherwise maybe someone else can help walk you through it | 14:31 |
| sweston | asselin: ^ | 14:31 |
| jklare | fungi: ty, i will also keep looking | 14:32 |
| *** Ark has joined #openstack-infra | 14:32 | |
| jeblair | fungi: i think it's because it never requests more than one devstack-precise at a time, so it gets filled from the first provider in the list (which is a hash, so is in "random" order, but consistent throughout the run) | 14:32 |
| *** Ark is now known as Guest32131 | 14:32 | |
| fungi | jeblair: aha, that's what i was just digging in the source to try to confirm | 14:32 |
| asselin | sweston, yeah, take a quick look at what mmedvede did: http://paste.openstack.org/show/138180/ | 14:32 |
| jeblair | fungi: have we tried forcing an image rebuild? | 14:33 |
| fungi | jeblair: i suppose there's something unluckily wrong with the most recent devstack-precise image in rax-iad that nova fails to successfully boot from it, and we keep getting persisted to it | 14:33 |
| fungi | jeblair: an image update? | 14:33 |
| fungi | i was about to try that | 14:33 |
| jeblair | fungi: yeah, i think we went to the same place. :) sounds good. | 14:33 |
| *** smoser has joined #openstack-infra | 14:34 | |
| jeblair | fungi: also, deleting devstack-precise in rax-iad may get us immediate relief (and simultaneously try the experiment of booting a similar image in another region) | 14:34 |
| fungi | i was thinking the same | 14:34 |
| *** banix has joined #openstack-infra | 14:35 | |
| *** jesusaurus has quit IRC | 14:35 | |
| jeson | Any one have seen this error when setting up 3rd CI: | 14:35 |
| jeson | Error: Error: /Stage[main]/Os_ext_testing::Master/File[/etc/zuul/openstack_functions.py]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///modules/openstack_project/zuul/openstack_functions.py | 14:35 |
| *** baoli has quit IRC | 14:35 | |
| *** cpowell has joined #openstack-infra | 14:35 | |
| *** jungleboyj has quit IRC | 14:35 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support for Gerrit Trigger Comment Contains Expression https://review.openstack.org/136605 | 14:36 |
| jeson | Seems can't find the file "openstack_functions.py" | 14:36 |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/project-config: Setting up a StoryBoard python client https://review.openstack.org/136004 | 14:37 |
| fungi | jeblair: one thing i find worrisome, nodepool.yaml is set to start image updates at 14:14 which was only 23 minutes ago, but i see no images building right now at all | 14:37 |
| *** mjturek has joined #openstack-infra | 14:38 | |
| fungi | okay, most recent devstack-precise image in rax-iad (from 95 hours ago) is deleted now and the next boot there should try with the remaining one (167 hours old) | 14:39 |
| *** mjturek has quit IRC | 14:39 | |
| *** mjturek has joined #openstack-infra | 14:39 | |
| sweston | asselin: looking over it now. seems like the script is doing a subtree split every time it runs? | 14:40 |
| *** ryanpetrello has quit IRC | 14:40 | |
| fungi | and i've got a new image building for it now | 14:41 |
| asselin | sweston, yes, but with --rejoin, which is supposed to make it 'smart'. He found that the performance is not good, but perhaps using a syste-config repo per module can help. | 14:41 |
| fungi | and i shuffled the providers list for that node label in the nodepool.yaml temporarily to see if that gets it building in another provider | 14:41 |
| *** signed8bit has joined #openstack-infra | 14:41 | |
| *** MarkAtwood has joined #openstack-infra | 14:42 | |
| *** otherwiseguy has quit IRC | 14:43 | |
| jeblair | fungi: i think it ends up in dict hash order, so not sure if it really got shuffled. look at the output from the allocator to see the order it's using. | 14:44 |
| fungi | jeblair: yeah, i thought that might happen. maybe i should temporarily remove rax-iad from that list instead | 14:45 |
| jeblair | fungi: well, it will be interesting to learn if rax iad can boot the older image | 14:45 |
| *** ryanpetrello has joined #openstack-infra | 14:46 | |
| fungi | jeblair: as for periodic image updates, it looks like it's processing dib image updates but not snapshot image updates | 14:46 |
| jeblair | fungi: yes, i was just reading that code | 14:46 |
| jeblair | fungi: i believe that the periodic update will not run snapshot image updates until all of the dib updates have completed | 14:46 |
| *** ddieterly has joined #openstack-infra | 14:46 | |
| fungi | ohhhhh | 14:46 |
| jeblair | <sigh> | 14:46 |
| mmedvede | sweston, asselin: I did try doing separate repository per split with --rejoin, the subtree fails after more than one update. My implementation of rejoin was wrong, so this is why my tests took longer with it. There is still hope to use custom rejoin to speed things up, but I am not going to have much time until later. Here is the script to reproduce the failure: https://gist.github.com/mmedvede/cf3a62fa395c18714b75 | 14:46 |
| fungi | that seems like a bug | 14:46 |
| jeblair | yolanda: ^ | 14:47 |
| jeblair | fungi: so i think for now we need to revert centos7 out of the config? | 14:47 |
| fungi | jeblair: apparently something at least caused it to start its next devstack-precise node build in rax-dfw instead | 14:47 |
| jeblair | ok | 14:48 |
| mmedvede | sweston, asselin: subtree split expects rejoin commits to be of certain form. Was looking into subtree implementation | 14:48 |
| jeblair | fungi: oh maybe we don't need to revert centos7 | 14:48 |
| *** yfried is now known as yfried|afk | 14:48 | |
| fungi | jeblair: do tell | 14:48 |
| *** ddieterly has quit IRC | 14:48 | |
| asselin | mmedvede, yes, I see those commits in my job's repo. | 14:48 |
| Sincler | hi, excuse me i have a question about the communication between github and gerrit, you can make am mirror from github to gerrit using jeepby, but if i made commits to gerrit those are not being pushed to github, how are you doing that? | 14:48 |
| jeblair | fungi: maybe it will just fail that single build and then proceed. so maybe we just need to wait until all the dib builds that can complete are complete, and then we should see snapshot builds? | 14:49 |
| *** markus_z has quit IRC | 14:49 | |
| asselin | mmedvede, they happen automatically with --rejoin | 14:49 |
| fungi | jeblair: entirely possible | 14:49 |
| fungi | Sincler: you have to configure gerrit's replication plug-in | 14:49 |
| *** ddieterly has joined #openstack-infra | 14:50 | |
| mmedvede | asselin: yes, it would have been nice if it worked. In tests I did it fails after some time. So you might not see a failure right away. Could be a bug with subtree | 14:50 |
| *** jeson has quit IRC | 14:50 | |
| asselin | mmedvede, ok, btw, sweston has another approach: https://github.com/Triniplex/puppet-module-split/blob/master/module_split.sh | 14:51 |
| *** r-daneel has joined #openstack-infra | 14:51 | |
| fungi | jeblair: the devstack-precise it tried to launch in rax-dfw also ended with a LaunchStatusException of status: ERROR | 14:51 |
| *** andymaier has joined #openstack-infra | 14:51 | |
| *** rushiagr_away is now known as rushiagr | 14:51 | |
| *** markus_z has joined #openstack-infra | 14:51 | |
| *** yfried|afk is now known as yfried | 14:51 | |
| fungi | jeblair: i'm tempted to stop building devstack-precise nodes in rackspace | 14:51 |
| jeblair | fungi: yeah; i do want to know if the replacement image boots, but until then, removing them seems like a good idea. | 14:52 |
| yolanda | jeblair, yes, the updateImages call is waiting for dib images to be created prior to uploading all the snapshots | 14:52 |
| jeblair | removing rax from the config for that image that is | 14:52 |
| fungi | jeblair: okay, change incoming | 14:53 |
| jeblair | yolanda: but it's also waiting for them before starting the snapshot image builds | 14:53 |
| *** jcoufal_ has joined #openstack-infra | 14:53 | |
| sweston | asselin: mmedvede: also, before each rejoin, we should switch to the branch the split was done from. I am not seeing where the script is doing that. | 14:53 |
| *** jcoufal has quit IRC | 14:53 | |
| *** jcoufal_ has quit IRC | 14:54 | |
| *** jcoufal has joined #openstack-infra | 14:54 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support for Gerrit Trigger Comment Contains Expression https://review.openstack.org/136605 | 14:54 |
| yolanda | jeblair, yes, as it's not applying any difference in the type of snapshots. It builds the dib images, to be sure that all sources are available. Then it uploads the dib snapshots or the normal snapshots in same pack | 14:54 |
| yolanda | i could redo the logic a bit, first upload the normal snapshots, then to the dib builds + upload dib snapshots, by that way we are not blocking normal snapshots | 14:55 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Don't build devstack-precise in rax for now https://review.openstack.org/137089 | 14:56 |
| mmedvede | sweston: good point. Could be this is why my test was failing | 14:56 |
| *** markus_z has quit IRC | 14:57 | |
| fungi | yolanda: i guess the confusion there is that snapshot image updates aren't uploads. so what's being delayed is the start of building the images, not just an upload | 14:57 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for cinder volume delete hang bug 1396186 https://review.openstack.org/137090 | 14:57 |
| uvirtbot | Launchpad bug 1396186 in cinder "(TestVolumeBootPattern:_run_cleanups) Failed to delete volume within the required time" [High,Confirmed] https://launchpad.net/bugs/1396186 | 14:57 |
| *** jesusaurus has joined #openstack-infra | 14:58 | |
| *** banix has quit IRC | 14:58 | |
| fungi | jeblair: okay, after a couple retries it successfully built a devstack-precise in rax-dfw after all | 14:59 |
| *** baoli has joined #openstack-infra | 14:59 | |
| jeblair | yolanda: i think your proposal sounds good -- it will start the snapshot builds at the scheduled time as well as start doing the dib builds (serially) and then upload the dib images when all dib builds are complete... | 14:59 |
| *** andymaier has quit IRC | 14:59 | |
| fungi | jeblair: so we could still do 137089 if we want, or abandon that now | 14:59 |
| jeblair | fungi: does that sound good ^ ? | 14:59 |
| *** andymaier has joined #openstack-infra | 14:59 | |
| fungi | jeblair: yolanda: yeah that does seem preferable to the present behavior | 15:00 |
| yolanda | jeblair, yes, that waitforbuildimages is blocking all the other processes to be executed | 15:00 |
| jeblair | fungi: lean toward ignoring/abandoning that change if it's not necessary | 15:00 |
| fungi | jeblair: same. abandoned! | 15:00 |
| sweston | mmedvede: I think so. I've also separated the split process from the update process, although there should probably be some logic added to only do a split when there is no local repo to pull the history into the split branch | 15:00 |
| openstackgerrit | Merged openstack-infra/release-tools: Add autokick.py https://review.openstack.org/136820 | 15:00 |
| yolanda | when i built that i was thinking mostly on exclusive builds, either by dib or snapshots, so i didn't consider the 2 types of builds case, and the blocking it was causing | 15:00 |
| yolanda | i'll send a change for that | 15:01 |
| jeblair | yolanda: yeah, and i think we all hoped we would have switched by now :) | 15:01 |
| *** markus_z has joined #openstack-infra | 15:01 | |
| fungi | jeblair: i worry that boot from snapshot in rackspace may just be getting increasingly less reliable | 15:01 |
| *** jungleboyj has joined #openstack-infra | 15:01 | |
| jeblair | fungi: yeah, i'm hoping with a little more poking, we'll have a reasonable problem statement for the rax folks | 15:02 |
| *** belmoreira has joined #openstack-infra | 15:02 | |
| fungi | jeblair: yolanda: as we discovered yesterday, we have a second roadblock to using dib with rackspace, which is that we'll probably have to install nova agent to get ip addresses configured there | 15:02 |
| jeblair | fungi: in other news, the async gear io change passes zuul's tests :( | 15:03 |
| jeblair | fungi: what? | 15:03 |
| *** zz_jgrimm is now known as jgrimm | 15:03 | |
| fungi | jeblair: apparently rackspace doesn't currently support setting instance addresses via dhcp or config drive | 15:04 |
| fungi | jeblair: if you look on any of our servers or nodepool nodes in rax, you'll see we have nova-agent running | 15:05 |
| jeblair | i'm pretty sure we talked about the question of how to configure networking early on | 15:05 |
| yolanda | fungi, why do you need ip addresses for dib? | 15:05 |
| jeblair | i thought someone looked into that | 15:05 |
| *** AlexF has joined #openstack-infra | 15:05 | |
| fungi | we were told yesterday that without it we won't get ip addresses | 15:05 |
| fungi | yolanda: we need something to set the ip address of the instance booted from the image dib creates | 15:05 |
| yolanda | fungi, ah, but that's on next step, when you are spinning up a node based on dib, right? | 15:06 |
| jeblair | fungi: i thought mordred looked into that :/ | 15:06 |
| dteselkin | Hi | 15:06 |
| fungi | jeblair: mordred was about to go on a rant yesterday when it got brought up. apparently it was news to him too | 15:06 |
| jeblair | i'm not entirely sure why we embarked on the whole dib thing without knowing something basic like that | 15:07 |
| fungi | yolanda: well, yes, but the image needs to contain whatever mechanisms are needed for the provider to be able to set the instance's ip address | 15:07 |
| dteselkin | Sometimes I see that zuul stops processing of events from 'events stream', however two 'zuul-server' processes are alive. | 15:07 |
| yolanda | ok, understand now | 15:07 |
| *** cnesa has quit IRC | 15:07 | |
| jeblair | fungi: how do i find this conversation? | 15:07 |
| fungi | jeblair: yesterday it was mentioned that rackspace hopes to experimentally have network information presented in the configdrive data real soon now | 15:07 |
| fungi | jeblair: irc log. lemme get you a timestamp | 15:08 |
| *** yamahata has quit IRC | 15:08 | |
| dteselkin | In such situations I see only 'Looking for lost builds' messages in debug.log | 15:08 |
| dteselkin | And I have to restart zuul service to get it working. | 15:09 |
| dteselkin | Is it a known issue? | 15:09 |
| *** belmoreira has quit IRC | 15:09 | |
| *** dizquierdo has joined #openstack-infra | 15:09 | |
| jeblair | fungi: oh found it | 15:10 |
| *** belmoreira has joined #openstack-infra | 15:10 | |
| fungi | jeblair: 2014-11-24 16:27:30 | 15:10 |
| *** afazekas has quit IRC | 15:11 | |
| dteselkin | For example, here is a part of logs http://paste.openstack.org/show/138207/ | 15:11 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support for Gerrit Trigger Comment Contains Expression https://review.openstack.org/136605 | 15:11 |
| jeblair | fungi: so mordred is going to get us options :) | 15:12 |
| jeblair | fungi: do you know what nova agent _does_ with networking? | 15:12 |
| *** mwagner_lap has quit IRC | 15:13 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for cinder volume delete hang bug 1396186 https://review.openstack.org/137090 | 15:13 |
| uvirtbot | Launchpad bug 1396186 in cinder "(TestVolumeBootPattern:_run_cleanups) Failed to delete volume within the required time" [High,Confirmed] https://launchpad.net/bugs/1396186 | 15:13 |
| *** mattfarina has joined #openstack-infra | 15:13 | |
| fungi | jeblair: i'm guessing it modifies files on the filesystem or something... i'm checking now to see if rax publishes the source code | 15:14 |
| jeblair | https://github.com/rackerlabs/openstack-guest-agents-unix ? | 15:15 |
| *** nikhil_k|vacay is now known as nikhil_k | 15:15 | |
| fungi | it's also python, so i'm looking at the copy on a recent server | 15:15 |
| *** HACKING-FACEBOOK has joined #openstack-infra | 15:15 | |
| *** zz_dimtruck is now known as dimtruck | 15:16 | |
| jeblair | fungi: it's very confusing that one of the things it does is "file injection" | 15:16 |
| *** yamahata has joined #openstack-infra | 15:16 | |
| jeblair | fungi: i thought when people referred to that, they meant modifying the image filesystem before booting | 15:16 |
| *** ryanpetrello has quit IRC | 15:16 | |
| *** jdandrea has quit IRC | 15:16 | |
| fungi | jeblair: yep, that looks like the same source code | 15:17 |
| *** otherwiseguy has joined #openstack-infra | 15:17 | |
| jeblair | fungi: but in reality we have a thing that can write to our config files at any time. that may actually explain some of the weird things we saw when working on unbound | 15:17 |
| fungi | jeblair: and agreed, it never dawned on me that they didn't mean file injection in the traditional sense of injecting files into the filesystem pre-boot | 15:18 |
| *** emagana has joined #openstack-infra | 15:18 | |
| jeblair | fungi: what nova agent does should be called "rudely overwriting your sysadmin's carefully configuration-managed files without asking" | 15:18 |
| fungi | jeblair: sounds about right | 15:19 |
| jroll | jeblair: fungi: AIUI, nova-agent should only modify things on (first?) boot and hotplug events (like adding a block storage volume) | 15:19 |
| gilliard | What's the lifetime of CI logs in logstash? Should I be able to search logs from July? | 15:20 |
| fungi | jroll: any chance the provider sometimes "generates" hotplug events to update things like network configuration? | 15:20 |
| *** signed8b_ has joined #openstack-infra | 15:20 | |
| fungi | gilliard: we only have enough room for about 10 days | 15:20 |
| *** ryanpetrello has joined #openstack-infra | 15:20 | |
| jroll | fungi: great question, but I kind of doubt it? | 15:20 |
| jroll | why would it update network configs t random | 15:20 |
| gilliard | fungi: yeah I thought 1/3 of a year was a bit optimistic :) | 15:20 |
| jroll | at random? | 15:21 |
| fungi | jroll: no idea... backend route changes? gateway updates? dns resolver address changes? | 15:21 |
| clarkb | jeblair mordred did look into it and rax jas docs on how to not use nova agent. what they dont tell is oy doesnt work | 15:21 |
| jroll | fungi: but someone else likely knows more, I like to pretend nova-agent doesn't exist | 15:22 |
| clarkb | you have to ask johnthetubaguy for that info | 15:22 |
| jroll | fungi: those should be very rare :| | 15:22 |
| *** banix has joined #openstack-infra | 15:22 | |
| fungi | jroll: "very rare" is not the same as nonexistent, and we've seen things like our custom dns resolver configuration get quietly undone | 15:22 |
| jroll | sigh | 15:23 |
| fungi | jroll: not sure if it was nova agent doing it, but it's a potential culprit | 15:23 |
| *** BharatK has joined #openstack-infra | 15:23 | |
| jroll | right | 15:23 |
| *** signed8bit has quit IRC | 15:23 | |
| *** HACKING-FACEBOOK has quit IRC | 15:23 | |
| jeblair | (we have a custom dns resolver config because rackspace dns has a blacklist that lags behind reality, so we boot new machines that can't resolve names quite a lot if we use it) | 15:24 |
| *** tonytan4ever has joined #openstack-infra | 15:24 | |
| fungi | i was looking at setting up my own recursive resolvers in rackspace so i can work around rackspace's disturbing lack of dnssec support, but now i'm wondering if it will periodically get undone and my virtual machines might just get set back to using the old resolvers | 15:24 |
| clarkb | fungi my new boot check at git.o.o resolves seems to have worked around it | 15:24 |
| *** HACKING-FACEBOOK has joined #openstack-infra | 15:24 | |
| clarkb | at least according to e-r last I looked | 15:24 |
| jroll | jeblair: what kind of blacklist? like, the dns servers blocking IPs? | 15:25 |
| fungi | clarkb: that's good news at least | 15:25 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Revert "Add query for cinder volume delete hang bug 1396186" https://review.openstack.org/137098 | 15:25 |
| uvirtbot | Launchpad bug 1396186 in cinder "(TestVolumeBootPattern:_run_cleanups) Failed to delete volume within the required time" [High,Confirmed] https://launchpad.net/bugs/1396186 | 15:25 |
| jeblair | jroll: yep; aiui, it's an auto-blacklist of abusive tenant ips | 15:25 |
| *** bigo has quit IRC | 15:25 | |
| * jroll cries | 15:26 | |
| fungi | jroll: rackspace has a mechanism in place to blacklist abusive internal consumers of their resolvers. it looks like abusers may just frequently recycle instances to get new ip addresses and work around it. unfortunately the blacklist doesn't un-blacklist those addresses before handing them out to new instances for other tenants | 15:26 |
| *** Guest27016 has quit IRC | 15:26 | |
| jroll | fungi: right, I've accidentally gotten onto that blacklist, it isn't fun | 15:26 |
| fungi | so we get a significant percentage of nodes booted in rackspace which are unable to resolve dns through rackspace's recursive resolvers | 15:26 |
| jeblair | fungi, clarkb: i hesitate to say this, but if mordred comes back with no other options, we could consider running a modified nova agent. | 15:27 |
| * johnthetubaguy wonders if he can help? | 15:27 | |
| fungi | not because our instances were blacklisted, but rather because the previous asignees of those addresses were blacklisted. rackspace support basically explained that this was the case and that there was nothing they could do about it | 15:27 |
| *** jgallard has joined #openstack-infra | 15:28 | |
| clarkb | jeblair yes I think that is where we got yesterday. or we wait for config drive | 15:28 |
| fungi | i could understand if this were some sort of external blacklist over which rackspace had no control, but it's systems within their network denying the connection. seems poorly-thought out to me | 15:28 |
| clarkb | we have about 5 other issues with dib + rax so time may be abundant | 15:29 |
| *** achanda has joined #openstack-infra | 15:29 | |
| *** otter768 has joined #openstack-infra | 15:29 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add cli option to subunit2sql to specify run_id https://review.openstack.org/131298 | 15:29 |
| fungi | jeblair: clarkb: it does indeed appear that we could fork rackspace's nova agent. it's apache 2 licensed | 15:29 |
| *** nikhil_k is now known as nikhil_k|vacay | 15:29 | |
| jeblair | https://github.com/rackerlabs/openstack-guest-agents-unix/blob/master/gitconfig | 15:30 |
| *** david-lyle has joined #openstack-infra | 15:30 | |
| jeblair | fungi: we'll want to, um, change the gitconfig before we make any commits. | 15:30 |
| johnthetubaguy | fungi: what do you need the agent to do? | 15:30 |
| jroll | wat | 15:30 |
| fungi | jeblair: ha! | 15:30 |
| fungi | i find it amusing that nova-agent.py is copyright openstack llc too | 15:30 |
| *** erikwilson has joined #openstack-infra | 15:30 | |
| sdague | jeblair: that's hilarious :) | 15:30 |
| johnthetubaguy | fungi: we can try merge changes you need, if you need them | 15:31 |
| * jroll PRs git rm gitconfig | 15:31 | |
| clarkb | if only there was some dynamic host configuration protocol we could use... | 15:31 |
| *** pc_m has quit IRC | 15:31 | |
| jroll | johnthetubaguy: if you have merge rights there, https://github.com/rackerlabs/openstack-guest-agents-unix/pull/57 | 15:32 |
| jeblair | clarkb: yeah, i'm wondering if something like "have nova agent drive a dhcp server" is a sane idea | 15:32 |
| fungi | johnthetubaguy: basically neuter it to handle the initial network setup but otherwise not be able to do anything destructive (and also we should probably uninstall it as soon as initial boot is over) | 15:32 |
| *** achanda has quit IRC | 15:32 | |
| jeblair | fungi: yep | 15:32 |
| *** che-arne has quit IRC | 15:32 | |
| *** alexpilotti has joined #openstack-infra | 15:32 | |
| *** ayoung has quit IRC | 15:33 | |
| johnthetubaguy | fungi: we might have image properties for most of that stuff | 15:33 |
| *** ayoung has joined #openstack-infra | 15:33 | |
| johnthetubaguy | fungi: what specific thing do you not want? | 15:33 |
| johnthetubaguy | I guess you only want network changes, and nothing more? | 15:33 |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (Jenkins) https://review.openstack.org/137100 | 15:33 |
| *** che-arne has joined #openstack-infra | 15:33 | |
| *** che-arn7 has joined #openstack-infra | 15:33 | |
| *** che-arne has quit IRC | 15:33 | |
| fungi | johnthetubaguy: well, keep in mind this is from the perspective of us building custom images to upload into glance. we basically would want it to set the interface addresses and routes, but likely nothing else | 15:34 |
| *** koolhead17 has quit IRC | 15:34 | |
| *** otter768 has quit IRC | 15:34 | |
| *** che-arn7 is now known as che-arne | 15:34 | |
| johnthetubaguy | fungi: got you there | 15:34 |
| *** banix has quit IRC | 15:35 | |
| johnthetubaguy | fungi: rather than over-write the whole file, just modify it in place? | 15:35 |
| johnthetubaguy | fungi: I think there was a "i am finished" call back thing | 15:35 |
| fungi | johnthetubaguy: potentially, yes. mostly just brainstorming at the momenty | 15:35 |
| *** wenlock has joined #openstack-infra | 15:36 | |
| *** che-arne has quit IRC | 15:36 | |
| johnthetubaguy | fungi: I would be cool merging that, its been a TODO for a while | 15:36 |
| *** sputnik13 has joined #openstack-infra | 15:36 | |
| fungi | but since boot from snapshot seems to be getting considerably worse based on recent observations, we have more urgency around working out the remaining show-stoppers for switching to glance uploads | 15:37 |
| *** BharatK has left #openstack-infra | 15:37 | |
| jeblair | johnthetubaguy: (for full context what we really want is to use the same image on hp and rax, so getting network config via a standard like dhcp is ideal; otherwise we want to minimize the differences) | 15:37 |
| johnthetubaguy | fungi: yeah, I have been hotfix snapshots for the last week, agreed, there was an incident open for that | 15:37 |
| *** mattfarina has quit IRC | 15:38 | |
| johnthetubaguy | jeblair: yeah, that totally makes sense, making config drive is my hope there | 15:38 |
| *** che-arn0 has joined #openstack-infra | 15:38 | |
| *** che-arne has joined #openstack-infra | 15:38 | |
| *** che-arn0 has quit IRC | 15:38 | |
| fungi | johnthetubaguy: glad to know it's a somewhat known situation then. i didn't see anything mentioned for it on status.rackspace.com | 15:38 |
| johnthetubaguy | fungi: latest openstack code pull broke us badly, to be fair it was a patch a rackspace person wrote | 15:38 |
| fungi | bwahahaha | 15:39 |
| johnthetubaguy | fungi: it should have been up there earlier in the week | 15:39 |
| fungi | johnthetubaguy: ahh, i likely missed it then | 15:39 |
| mrmartin | morning | 15:39 |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (Jenkins dev) https://review.openstack.org/137102 | 15:39 |
| fungi | johnthetubaguy: though it seems to be ongoing, at least what we're witnessing | 15:39 |
| johnthetubaguy | fungi: basically we had images goning active, even though they failed | 15:39 |
| jeblair | johnthetubaguy: the beauty is that now it's hard to get the fix merged because we're having trouble booting nodes from snapshots :) | 15:39 |
| *** pradk has joined #openstack-infra | 15:39 | |
| johnthetubaguy | fungi: right now, the failure rates are too high, but it fails safe, we got the error rates down somewhat, but we can't fix this right now | 15:40 |
| *** esker has joined #openstack-infra | 15:40 | |
| fungi | johnthetubaguy: oh, i definitely found one of those. nodepool got a response from rax that the snapshot was successfully created, but then boot from it was perpetually failing with "requested image is not active" or similar | 15:40 |
| sweston | asselin: mmedvede ok, running live against github now, on all modules | 15:40 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Revert "Add query for cinder volume delete hang bug 1396186" https://review.openstack.org/137098 | 15:40 |
| uvirtbot | Launchpad bug 1396186 in cinder "(TestVolumeBootPattern:_run_cleanups) Failed to delete volume within the required time (dup-of: 1373513)" [High,Confirmed] https://launchpad.net/bugs/1396186 | 15:41 |
| uvirtbot | Launchpad bug 1373513 in cinder "Lvm hang during tempest tests" [Critical,Confirmed] https://launchpad.net/bugs/1373513 | 15:41 |
| jeblair | johnthetubaguy: fails safe? | 15:41 |
| fungi | johnthetubaguy: but yeah it was from like friday i think | 15:41 |
| johnthetubaguy | fungi: yeah, thats been fixed now | 15:41 |
| sweston | asselin: mmedvede anybody know if github has a limit on number of public repositories? :-D | 15:41 |
| *** HACKING-FACEBOOK has quit IRC | 15:41 | |
| fungi | sweston: we have a few hundred in stackforge at this point, so if they do it's at least higher than that | 15:41 |
| johnthetubaguy | jeblair: well, when its a partial upload, it now fails, we had an issue where image when active, but nova eventually deleted it, once nova detected the upload failed | 15:41 |
| mmedvede | sweston: I think you should be fine | 15:42 |
| *** ildikov has quit IRC | 15:42 | |
| jeblair | ok | 15:42 |
| sweston | fungi: hehe ok .. (crosses fingers) | 15:43 |
| fungi | sdague: jeblair: i see icehouse dsvm jobs running now, so at least the not_registered should be under control at this point | 15:43 |
| mrmartin | fungi, can I help somehow, and issue a self-signed certs for groups-dev.openstack.org ? | 15:43 |
| *** david-lyle has quit IRC | 15:43 | |
| *** che-arne has quit IRC | 15:43 | |
| sdague | fungi: \o/ | 15:43 |
| *** che-arne has joined #openstack-infra | 15:43 | |
| *** che-arn3 has joined #openstack-infra | 15:44 | |
| fungi | mrmartin: sorry, i didn't mean to hold you up. we've had a series of different failures and bugs sucking up most of our time for the past 5+ days | 15:44 |
| mrmartin | fungi, I see and can help in cert issue, the question here, how to exchange them in a safe way | 15:44 |
| markus_z | fungi: What should I do with my review if I ran into the not_registrated issue? | 15:45 |
| fungi | mrmartin: the cert generation isn't the time consuming part | 15:45 |
| fungi | markus_z: recheck | 15:45 |
| bknudson | what does "NOT_REGISTERED" result mean? https://review.openstack.org/#/c/102403/ | 15:45 |
| mrmartin | ok. | 15:45 |
| fungi | markus_z: should be fixed now | 15:45 |
| asselin | sweston, I didn't have any issues yesterday | 15:45 |
| clarkb | fungi jeblair so we are in stableish spot now with nodepool? | 15:45 |
| markus_z | fungi: thanks, I'll do that | 15:46 |
| fungi | clarkb: seems so. i checked about an hour ago and saw no new gearmanclient timeouts since the two we observed around 01:20 | 15:46 |
| markus_z | mpavlase: You ran into this issue, too, right? | 15:46 |
| *** che-arne has quit IRC | 15:46 | |
| hogepodge | What does "NOT_REGISTERED" mean in the Jenkins logs? | 15:47 |
| hogepodge | Oh, just saw someone else asked. | 15:47 |
| hogepodge | I'll recheck no bug. :-) | 15:47 |
| *** achanda has joined #openstack-infra | 15:48 | |
| clarkb | it means we had no knowm location to run that job | 15:49 |
| clarkb | I believe fungi has reported that this is corrected now | 15:49 |
| bknudson | great, thanks | 15:50 |
| fungi | yep | 15:50 |
| mestery | Hi infra folks! If I could get some time for this super simple review https://review.openstack.org/#/c/134309/. It enables an updated version of ODL in devstack, which fixes IPV6 there and lets the Tempest API tests run again. We need this to land to re-enable our CI. | 15:50 |
| mestery | Thanks for yoru time! | 15:50 |
| mestery | tykeal: ^^^ | 15:50 |
| tykeal | mestery: thanks :) | 15:51 |
| *** sdake has quit IRC | 15:51 | |
| fungi | jeblair: also the manual image update of devstack-precise in rax-iad succeeded | 15:53 |
| mpavlase | markus_z: NOT_REGISTERED issue? Yes, I do | 15:53 |
| *** skolekonov has quit IRC | 15:53 | |
| clarkb | mestery that is a devstack change most ofus here are not devstack cores | 15:53 |
| clarkb | we typically point at openstack-qa for that | 15:53 |
| mestery | clarkb: Ah, will move to #opestack-qa, thanks for the correction! :) | 15:53 |
| fungi | jeblair: and as predicted, the scheduled image updates did eventually proceed | 15:53 |
| fungi | they're all running now | 15:54 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Update snapshots prior to dib image builts https://review.openstack.org/137110 | 15:54 |
| markus_z | mpavlase: fungi has reported that this is corrected now. Leave a "recheck" comment (without the quotation marks) in your review. | 15:54 |
| clarkb | why not use two crons? | 15:54 |
| mpavlase | hogepodge: I also tried the same, but I've got NOT_REG again before that jobs even started | 15:54 |
| clarkb | yolanda ^ | 15:54 |
| mpavlase | markus_z: great! Thanks for notice :-) | 15:54 |
| *** sputnik13 has quit IRC | 15:55 | |
| markus_z | mpavlase: you're welcome! | 15:55 |
| *** sdake has joined #openstack-infra | 15:55 | |
| *** sdake has quit IRC | 15:55 | |
| *** sdake has joined #openstack-infra | 15:55 | |
| yolanda | clarkb, actually they are running on independent threads, the blocker was waiting untill all dib items in the queue were completed | 15:55 |
| clarkb | but now ww wait on snapshots? | 15:55 |
| yolanda | actually i've always seen that as a problem that all snapshot updates are being executed at the same time | 15:55 |
| clarkb | not sure how that is more correct | 15:56 |
| openstackgerrit | Aysy Anne A. Duarte proposed openstack-infra/elastic-recheck: Add query for insufficient free space bug 1386249 https://review.openstack.org/132118 | 15:56 |
| uvirtbot | Launchpad bug 1386249 in tempest "Tests from package tempest.api.volume.admin sometimes fails due "insufficient free space"" [Undecided,New] https://launchpad.net/bugs/1386249 | 15:56 |
| clarkb | or does the snapshot stuff not block? | 15:57 |
| *** fandi has joined #openstack-infra | 15:57 | |
| jeblair | clarkb: it does not block | 15:57 |
| *** ildikov has joined #openstack-infra | 15:58 | |
| yolanda | it's non-blocking | 15:58 |
| *** armax has joined #openstack-infra | 15:58 | |
| mrmartin | fungi, groups-dev.openstack.org seems to be down: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=138 | 15:58 |
| clarkb | ok I can rebase my change to allow mixed image types on that change then | 15:58 |
| *** isaacb has quit IRC | 15:59 | |
| *** emagana has quit IRC | 16:00 | |
| fungi | mrmartin: i was able to ssh into it just a moment ago | 16:00 |
| fungi | and apache is running | 16:00 |
| mrmartin | fungi | 16:00 |
| *** emagana has joined #openstack-infra | 16:00 | |
| clarkb | jeblair what do you think next step in debugging is? do we nees to connect to zuul geard from new nodepool and do more debhgging of gear? | 16:01 |
| mrmartin | yeap, too late here, I rewrite the hostname in local /etc/hosts and forget | 16:01 |
| mrmartin | :) | 16:01 |
| *** achanda has quit IRC | 16:02 | |
| *** matel has quit IRC | 16:02 | |
| *** ociuhandu has joined #openstack-infra | 16:02 | |
| jeblair | clarkb: not sure yet; i'm going to grab some breakfast; maybe regroup in 30 mins? | 16:02 |
| clarkb | wfm | 16:02 |
| clarkb | I need food too | 16:03 |
| *** yaguang has quit IRC | 16:04 | |
| *** emagana_ has joined #openstack-infra | 16:04 | |
| *** emagana has quit IRC | 16:05 | |
| *** rediskin has joined #openstack-infra | 16:06 | |
| *** amuller has joined #openstack-infra | 16:07 | |
| *** david-lyle_afk is now known as david-lyle | 16:07 | |
| *** reed has joined #openstack-infra | 16:08 | |
| *** belmoreira has quit IRC | 16:08 | |
| rediskin | Hi all. I sent an email to third-party-announce a week ago, but still no response. What I'm doing wrong? | 16:11 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: remove heat-slow job and definition https://review.openstack.org/137123 | 16:11 |
| *** chandankumar has joined #openstack-infra | 16:12 | |
| rediskin | fungi jeblair SergeyLukjanov someone please help T_T | 16:12 |
| *** isaacb has joined #openstack-infra | 16:12 | |
| *** xyang0 has joined #openstack-infra | 16:13 | |
| *** yfried has quit IRC | 16:14 | |
| *** tailwong has joined #openstack-infra | 16:16 | |
| *** yamamoto has joined #openstack-infra | 16:16 | |
| *** radez is now known as radez_g0n3 | 16:17 | |
| *** ashaeron has quit IRC | 16:19 | |
| *** mattfarina has joined #openstack-infra | 16:20 | |
| *** dannywilson has joined #openstack-infra | 16:20 | |
| *** yamamoto has quit IRC | 16:21 | |
| sdague | fungi / clarkb / jeblair - https://review.openstack.org/#/c/136846/ - removes the pylint job from nova, has 3 nova core +1s on it. | 16:21 |
| *** ala_ has quit IRC | 16:22 | |
| fungi | sdague: though so did https://review.openstack.org/136013 which merged last night :/ | 16:24 |
| fungi | rediskin: we've been extremely backed up on those, sorry. i haven't looked but we're presently working on options to make it self-service | 16:24 |
| sdague | fungi: mine had an ML thread | 16:24 |
| sdague | no one is looking at that job | 16:24 |
| *** changbl has joined #openstack-infra | 16:25 | |
| *** cnesa has joined #openstack-infra | 16:25 | |
| *** AJaeger has quit IRC | 16:25 | |
| sdague | we had the PTL +1 on the ML and a number of active cores vote on the ML and in gerrit - http://lists.openstack.org/pipermail/openstack-dev/2014-November/051436.html | 16:25 |
| sdague | I mean, we could just keep wasting test nodes, but I'd kind of like them to do useful things :) | 16:26 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Update snapshots prior to dib image builts https://review.openstack.org/137110 | 16:27 |
| *** cnesa has quit IRC | 16:27 | |
| sdague | fungi: thanks, though apparently I need to rebase | 16:28 |
| *** JayJ has quit IRC | 16:29 | |
| *** otherwiseguy has quit IRC | 16:29 | |
| *** cnesa has joined #openstack-infra | 16:29 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: remove nova pylint https://review.openstack.org/136846 | 16:29 |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Doc job specification branch: handles refs https://review.openstack.org/137130 | 16:29 |
| sdague | fungi: rebased - https://review.openstack.org/#/c/136846/2 | 16:30 |
| *** emagana_ has quit IRC | 16:31 | |
| *** emagana has joined #openstack-infra | 16:31 | |
| *** obondarev has quit IRC | 16:32 | |
| *** emagana has quit IRC | 16:33 | |
| *** emagana_ has joined #openstack-infra | 16:33 | |
| *** mmaglana has quit IRC | 16:36 | |
| fungi | mrmartin: i've got the dev and production certs/keys in hiera now. i've marked a few inline comments on 136846 but am happy to make the indicated edits myself if you like | 16:36 |
| mrmartin | checking that | 16:36 |
| *** alexpilotti has quit IRC | 16:37 | |
| *** AJaeger has joined #openstack-infra | 16:38 | |
| *** AJaeger has joined #openstack-infra | 16:38 | |
| *** alexpilotti has joined #openstack-infra | 16:38 | |
| *** Mike24 has joined #openstack-infra | 16:39 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: Added experimental-{name}-chef-rake job-template https://review.openstack.org/137134 | 16:39 |
| mmedvede | fungi, clarkb: would of you volunteer to shepherd elasticsearch module split merges, or would you volunteer some one else? | 16:40 |
| *** banix has joined #openstack-infra | 16:40 | |
| *** sputnik13 has joined #openstack-infra | 16:40 | |
| mrmartin | fungi: do you think that it is better to provide a snake-oil cert instead of leaving those values undef? | 16:40 |
| *** sergsh has quit IRC | 16:40 | |
| *** shashankhegde has joined #openstack-infra | 16:42 | |
| fungi | mrmartin: if we do, then someone copying the groups_dev.pp to set up their own dev server doesn't need to pass in site_ssl_cert_file_contents and site_ssl_key_file_contents (though we need to inject them from hiera because we'll need the same cert added to the trust chain on openstackid-dev) | 16:43 |
| *** dkehn has quit IRC | 16:44 | |
| *** fifieldt has quit IRC | 16:45 | |
| mrmartin | ok, but then I need to rewrite the entire vhost part | 16:45 |
| mrmartin | doing that anyway | 16:45 |
| *** dkehn has joined #openstack-infra | 16:46 | |
| *** mpaolino has quit IRC | 16:47 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/infra-specs: Use Change-Id instead of URL for dependent patches https://review.openstack.org/137135 | 16:49 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Add ironic-discoverd to #openstack-ironic IRC channel https://review.openstack.org/137136 | 16:49 |
| *** ildikov has quit IRC | 16:50 | |
| *** sergsh has joined #openstack-infra | 16:51 | |
| *** sergsh is now known as sergsh_away | 16:51 | |
| *** chandankumar has quit IRC | 16:51 | |
| *** thedodd has joined #openstack-infra | 16:52 | |
| fungi | mrmartin: yeah, that makes it more consistent with how we're doing ssl in other vhost templates anyway | 16:52 |
| mrmartin | almost ready | 16:52 |
| fungi | mmedvede: i agreed to shepherd two more puppet modules last week for nibalizer and then let him down, so i shouldn't agree to do any more until i get those pushed through | 16:53 |
| thingee | I was noticing the python-cinderclient is still getting some jobs as not registered https://review.openstack.org/#/c/91946/ | 16:53 |
| fungi | thingee: we didn't have it solved until a couple hours ago | 16:54 |
| thingee | fungi: ok thanks | 16:54 |
| fungi | thingee: that result is from long, long before | 16:54 |
| openstackgerrit | Merged openstack-infra/project-config: remove nova pylint https://review.openstack.org/136846 | 16:54 |
| *** e0ne has quit IRC | 16:54 | |
| dteselkin | fungi, sorry to bother you, but I faced the problem with zuul again. Do you ever met the situation when it stops fetching events? | 16:54 |
| mmedvede | fungi: np. Would you suggest who else I should ping? | 16:55 |
| nibalizer | fungi: no worries :) | 16:55 |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Enable SSL in groups-dev.openstack.org https://review.openstack.org/135708 | 16:55 |
| mrmartin | fungi: I moved the cert override up to site.pp | 16:56 |
| *** radez_g0n3 is now known as radez | 16:56 | |
| clarkb | mmedvede: right now puppet splits are low on my priority list. Trying to get nodepool sorted | 16:57 |
| *** sarob has joined #openstack-infra | 16:57 | |
| mmedvede | clarkb: Understand. I am not trying to push it through now. Just need somebody to put into commit message as a shepherd | 16:57 |
| mmedvede | I'll ask around | 16:58 |
| *** Longgeek has quit IRC | 16:58 | |
| *** talluri has joined #openstack-infra | 16:58 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Enable SSL in groups-dev.openstack.org https://review.openstack.org/135708 | 16:58 |
| clarkb | jeblair: fungi: ok I am back from breakfast. going to review yolanda's change now then rebase on it | 16:58 |
| fungi | dteselkin: probably running tcpdump on your zuul server looking for packets coming back from the gerrit event stream would help you to know if you have a network issue. it's possible the socket is disconnected due to a network problem (such as an aggressive timeout in a gateway) and your zuul is unaware it's a dead socket | 16:58 |
| pleia2 | good morning | 16:58 |
| clarkb | pleia2: morning | 16:59 |
| *** davideagnello has quit IRC | 17:00 | |
| dteselkin | fungi, ok, I'll check it. Thanks! | 17:00 |
| *** amotoki has quit IRC | 17:00 | |
| *** davideagnello has joined #openstack-infra | 17:01 | |
| *** markus_z has quit IRC | 17:01 | |
| *** mjg1 has joined #openstack-infra | 17:01 | |
| sweston | asselin: mmedvede: ok, it seems to be working, with all the modules now. the initial subtree splits will take a while, but once it runs through once, it will be much faster | 17:01 |
| *** yfried has joined #openstack-infra | 17:01 | |
| *** salv-orlando has quit IRC | 17:01 | |
| asselin | sweston, yeah, my initial one took 2 hours. if yours is working, I'll take a look a the next steps. | 17:02 |
| *** SumitNaiksatam has quit IRC | 17:03 | |
| mtreinish | clarkb, fungi: if you get a sec can you run this query on the subunit2sql db: "SELECT tests.test_id,tests.success,tests.failure,tests.run_count FROM tests ORDER BY tests.failure DESC LIMIT 25;" | 17:03 |
| mtreinish | I'm curious what a weeks worth of failures looks like | 17:03 |
| sweston | asselin: ok, will keep you updated when I have some results | 17:03 |
| *** SumitNaiksatam has joined #openstack-infra | 17:04 | |
| *** kmartin has quit IRC | 17:05 | |
| *** isaacb has quit IRC | 17:06 | |
| *** julim has joined #openstack-infra | 17:07 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Allow labels to have snapshot and dib images https://review.openstack.org/130878 | 17:08 |
| clarkb | ok there is the rebase | 17:08 |
| *** dims has quit IRC | 17:09 | |
| *** mmaglana has joined #openstack-infra | 17:09 | |
| *** bswartz has quit IRC | 17:09 | |
| fungi | mrmartin: closer, but i still have a couple comments on that last patchset | 17:12 |
| fungi | mtreinish: just a sec | 17:12 |
| jeblair | clarkb, fungi: i'm back too | 17:13 |
| asselin | is there an infra meeting today? | 17:14 |
| fungi | asselin: at 19:00 utc, yes | 17:14 |
| *** yamahata has quit IRC | 17:14 | |
| *** mmaglana has quit IRC | 17:14 | |
| mordred | clarkb: I'll be on an working in about 45 ... do we have captured the deets on the novaclient nonsense? | 17:14 |
| asselin | fungi, is the agenda up-to-date? looks like last week's | 17:14 |
| clarkb | mordred: we have a stacktrace | 17:14 |
| jeblair | let's clean the agenda up | 17:15 |
| mordred | ossum | 17:15 |
| fungi | asselin: i probably forgot to clean up any of my addressed entries. i'll do that real quick | 17:15 |
| asselin | fungi, thanks | 17:15 |
| mmedvede | jhesketh, mordred, jeblair: any of you can shepherd elasticsearch puppet module split merges? | 17:15 |
| clarkb | mordred: jeblair had put up a paste of it yesterday | 17:15 |
| fungi | mtreinish: http://paste.openstack.org/show/138278 | 17:16 |
| jeblair | mmedvede: could probably do it after 2100 utc | 17:16 |
| mtreinish | fungi: awesome, thanks | 17:17 |
| mmedvede | jeblair: ok, I would add you on commit message than. Thx | 17:17 |
| mmedvede | *then | 17:17 |
| jeblair | we're adding that to commit messages? | 17:17 |
| clarkb | I was just going to ask... | 17:18 |
| *** AlexF has quit IRC | 17:18 | |
| jeblair | seems like that should be a comment, not a commit message thing | 17:18 |
| clarkb | ++ | 17:18 |
| *** StevenK has quit IRC | 17:18 | |
| jeblair | mmedvede: ^ | 17:19 |
| *** mmaglana has joined #openstack-infra | 17:19 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:19 | |
| mmedvede | jeblair, clarkb : we want it to be clear who is supposed to approve the merge, as discussed previously | 17:19 |
| jeblair | mmedvede: i'm totally okay with being clear. i'm not sure it belongs in the commit message. that creates unecessary reviewer churn | 17:20 |
| *** StevenK has joined #openstack-infra | 17:20 | |
| jeblair | and it's not related to the commit -- it's temporal | 17:20 |
| *** shashankhegde has quit IRC | 17:21 | |
| fungi | okay, meeting agenda is straightened up relative to what we covered last week | 17:21 |
| *** AlexF has joined #openstack-infra | 17:21 | |
| jeblair | fungi: heh, and i just got to see mediawiki's conflict detection in action :) | 17:22 |
| *** nadya_ has quit IRC | 17:22 | |
| fungi | jeblair: ha, oops! sorry... | 17:22 |
| jeblair | fungi: np, you (understandably) did a slightly better job than i did :) | 17:22 |
| fungi | too many cooks in the kitchen | 17:22 |
| mrmartin | fungi: checking that | 17:24 |
| mmedvede | jeblair: there is this spec for splits were we try to get the work flow right: http://specs.openstack.org/openstack-infra/infra-specs/specs/puppet-modules.html . It is desired that related merges be synchronized. Leaving that information in a comment would mean accident merge could happen. Making patch WIP would mean none would review it... | 17:24 |
| jeblair | mmedvede: yeah, i don't see anything in the spec about changing the commit messages | 17:24 |
| *** derekh has quit IRC | 17:25 | |
| mmedvede | jeblair: sorry, It has not been merged yet https://review.openstack.org/#/c/135452/ . But it has been discussed in past meetings | 17:25 |
| *** mmaglana has quit IRC | 17:26 | |
| jeblair | 19:30:56 <jeblair> #agreed when dependent puppet-module splits are completely ready to merge, a core reviewer will commit to approving them in the appropriate order or coordinate with another reviewer to take over | 17:26 |
| jeblair | 19:31:00 <jeblair> #agreed system-config module removals should include Depends-On: in commit message referencing corresponding project-config module adds | 17:27 |
| *** mudassirlatif has joined #openstack-infra | 17:27 | |
| jeblair | yeah, nothing about putting core reviewer names in commit messages. it's just an understanding that whoever approves a project-config module add must ensure that the removal is approved | 17:28 |
| jeblair | everyone who is core on project config shuld know that, and if they do it anyway, then we need to have a conversation with them | 17:28 |
| *** alexpilotti has quit IRC | 17:28 | |
| fungi | fwiw, sdague expressed a concern that having the ability to accidentally merge a patch which is not ready or which is waiting for someone specific to approve and monitor results in infra having a separate core review workflow from other projects when we should not, to reduce burden on cross-program cores | 17:30 |
| *** otter768 has joined #openstack-infra | 17:30 | |
| jeblair | fungi: i can not parse that sentence | 17:30 |
| fungi | i had trouble distilling it | 17:30 |
| fungi | let me try to rephrase | 17:31 |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Enable SSL in groups-dev.openstack.org https://review.openstack.org/135708 | 17:32 |
| fungi | sdague said that he doesn't like that we have patches we don't want merged which are not either wip or -2, and that expecting core reviewers to pay attention to other reviewer's comments before approving a change means that we're deviating from how other programs do core review, or some such (he can likely explain his viewpoint better than i can) | 17:32 |
| jeblair | oh, "monitor <end of phrase/> results" not "<phrase>monitor results>" | 17:33 |
| jeblair | sdague: ping | 17:33 |
| fungi | putting directed comments in the commit message was suggested as a compromise | 17:33 |
| sdague | jeblair: pong | 17:33 |
| *** dustins has joined #openstack-infra | 17:33 | |
| clarkb | I still don't grok why WIP is not acceptable | 17:33 |
| mrmartin | fungi: the changes are there ^^^^ | 17:33 |
| clarkb | if you are corralling people to do the reviews anways WIP shouldn't be a problem | 17:34 |
| fungi | thanks mrmartin | 17:34 |
| *** dustins has quit IRC | 17:34 | |
| jeblair | sdague: so here's the thing -- we're in the middle of this big project that's a lot of work and is an exceptional situation. during this time, we need project-config cores to know that there's some tricky sequencing to specific types of changes -- those that are puppet module splits from system-config. | 17:34 |
| sdague | jeblair: ok, so my question is why WIP wasn't part of that workflow. | 17:35 |
| *** otter768 has quit IRC | 17:35 | |
| jeblair | sdague: so, yeah, it's exceptional, but that's exactly it -- it's exceptional, and it will be over eventually :) | 17:35 |
| clarkb | sdague: because people are afraid that marking things WIP will kill reviews | 17:35 |
| nibalizer | ya you dont get reviews if you WIP | 17:35 |
| clarkb | nibalizer: but this is different | 17:35 |
| nibalizer | unless you badger people in chat, and that feels improper too | 17:36 |
| clarkb | nibalizer: because y'all are here asking us specifically for reviews | 17:36 |
| clarkb | nibalizer: no you are alread doing that | 17:36 |
| sdague | and they have to go in a specific times with people monitoring them | 17:36 |
| fungi | well, there's a timing-related issue as well. essentially the patch gets into a ready state where it is not expected to change, and then wip is being used not to indicate that the patch is still in progress, but rather that there are external dependencies/workflow which will need addressing in parallel with it | 17:36 |
| sdague | anyway, I can also just blanket ignore all the puppet stuff | 17:36 |
| nibalizer | clarkb: i've tried not to be a bother ~~ | 17:36 |
| jeblair | sdague: that's not exactly it | 17:36 |
| nibalizer | this feels a bit like the growing pains of a split project-config/core and system-config/core group | 17:37 |
| nibalizer | more than anything else | 17:37 |
| jeblair | sdague: it's this -- please don't approve the change unless you know that someone is able to approve the system-config change too | 17:37 |
| fungi | and also can confirm or can have someone confirm that the module being split out hasn't received additional changes since the last time the split copy was refreshed | 17:38 |
| jeblair | sdague: i would love for you to continue to review things. i acknowledge that this is exceptional, but i also think that it's not unreasonable to keep some context about what's going on in the project in mind when reviewing. | 17:38 |
| *** SumitNaiksatam has quit IRC | 17:38 | |
| *** dustins has joined #openstack-infra | 17:38 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:39 | |
| *** mmaglana has joined #openstack-infra | 17:39 | |
| *** dkehn has quit IRC | 17:39 | |
| *** guest764 has joined #openstack-infra | 17:39 | |
| *** e0ne has joined #openstack-infra | 17:40 | |
| *** AlexF has quit IRC | 17:40 | |
| *** AlexF_ has joined #openstack-infra | 17:40 | |
| *** jcoufal has quit IRC | 17:40 | |
| sdague | jeblair: so... I think it's safer if I just remember to ignore puppet changes. Because I look at a lot of code, and I'll lose context that something else had to happen, and nothing in gerrit was holding back the patches. | 17:41 |
| *** dkehn has joined #openstack-infra | 17:41 | |
| *** shashankhegde has joined #openstack-infra | 17:41 | |
| *** andreykurilin_ has joined #openstack-infra | 17:42 | |
| *** dustins has quit IRC | 17:42 | |
| *** melwitt has joined #openstack-infra | 17:42 | |
| *** dustins has joined #openstack-infra | 17:42 | |
| *** wuhg has quit IRC | 17:43 | |
| *** luqas has joined #openstack-infra | 17:43 | |
| *** luqas has quit IRC | 17:45 | |
| *** shashankhegde has quit IRC | 17:46 | |
| clarkb | adam_g: see comment on https://review.openstack.org/#/c/131604/ | 17:47 |
| clarkb | adam_g: does that remove the need for the -1? also not sure how to debug the metadata thing | 17:47 |
| clarkb | adam_g: do you have thoughts on that? | 17:47 |
| mmedvede | Opinions are split :) I'll WIP my split patches then | 17:47 |
| *** harlowja_away is now known as harlowja | 17:49 | |
| *** jerryz has quit IRC | 17:49 | |
| *** jerryz has joined #openstack-infra | 17:49 | |
| *** dimsum__ has joined #openstack-infra | 17:51 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/system-config: Split out elasticsearch module https://review.openstack.org/137155 | 17:51 |
| *** jistr has quit IRC | 17:52 | |
| harlowja | qq, are the 'NOT_REGISTERED' issues that i'm seeing across oslo libraries gonna be resolved sometime today? :-/ | 17:52 |
| *** fandi has quit IRC | 17:52 | |
| *** nadya_ has joined #openstack-infra | 17:53 | |
| fungi | harlowja: are those still happening somewhere? they should have been solved at least several hours ago now | 17:53 |
| harlowja | hmmm | 17:53 |
| adam_g | clarkb, oh, yeah. i must have changed defaults or something locally. re metadata, that was my bad. i forgot that you need to run a nova-metadata-ec2 service on each compute in multi host | 17:53 |
| harlowja | let me see what happens with a recheck i'm doing fungi | 17:53 |
| *** jerryz has quit IRC | 17:53 | |
| clarkb | adam_g: cool for metadata does devstack do that for us? or do I need to update that? | 17:53 |
| adam_g | clarkb, i have a patch, one sec | 17:53 |
| *** mwagner_lap has joined #openstack-infra | 17:53 | |
| jklare | can somebody help me with this? http://logs.openstack.org/34/137134/1/check/gate-project-config-layout/30a9254/console.html | 17:53 |
| jklare | or point me into the right direction? | 17:54 |
| adam_g | clarkb, https://review.openstack.org/#/c/136179/ | 17:54 |
| adam_g | jogo ^ | 17:54 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/project-config: Add puppet-elasticsearch as split out module https://review.openstack.org/137156 | 17:54 |
| clarkb | adam_g: cool thanks | 17:54 |
| jeblair | sdague, clarkb, fungi: so i don't think we should put names in the commit message, but i think we could say we should add a warning about needing coordination to the project-config commit message (in addition to the needed-by) | 17:54 |
| clarkb | adam_g: any idea why icehouse failed there? | 17:54 |
| clarkb | jklare: if you scroll to near the very bottom of that log you will find a list of jobs that are not defined | 17:55 |
| clarkb | jklare: so 137134 is trying to run those jobs in zuul but we don't have corresponding jobs in jenkins. you need to add the jobs to jenkins via jjb as well | 17:55 |
| adam_g | clarkb, wrapt | 17:56 |
| fungi | jeblair: works for me. it's at least a more explicit reminder as to the expected workflow | 17:56 |
| clarkb | adam_g: rgr | 17:56 |
| *** gyee_ has joined #openstack-infra | 17:56 | |
| jklare | clarkb: ty, i was acutally looking at the bottom but saw this reolve error first | 17:56 |
| jklare | ^^resolve | 17:56 |
| jeblair | fungi: added suggesting wording in https://review.openstack.org/#/c/135452/ | 17:57 |
| *** sarob has quit IRC | 17:57 | |
| *** Ryan_Lane has joined #openstack-infra | 17:57 | |
| sdague | adam_g: I'm pretty sure that's not needed at all | 17:58 |
| *** mrmartin has quit IRC | 17:58 | |
| sdague | and more importantly, I don't think it does what you think it does | 17:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Add puppet-elasticsearch as split out module https://review.openstack.org/137156 | 17:58 |
| clarkb | I will let sdague review that change then :0 | 17:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Add puppet-elasticsearch as split out module https://review.openstack.org/137156 | 17:58 |
| clarkb | er :) | 17:58 |
| adam_g | sdague, nova-network multi host /w no config drive | 17:58 |
| sdague | adam_g: yep | 17:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Split out elasticsearch module https://review.openstack.org/137155 | 17:58 |
| sdague | the metadata server is always there in the api | 17:59 |
| jklare | clarkb: now i am even more confused. why is the node set in the jenkins/jobs/projects.yaml as well as in the zuul/layout.yaml? | 17:59 |
| *** talluri has quit IRC | 17:59 | |
| clarkb | jklare: I don't follow | 17:59 |
| clarkb | the node? | 17:59 |
| sdague | that change will just make it run in a separate process | 17:59 |
| sdague | which is not needed | 17:59 |
| *** rushiagr is now known as rushiagr_away | 17:59 | |
| jeblair | mmedvede, sdague, clarkb: how's this look? https://review.openstack.org/#/c/137156/ | 17:59 |
| clarkb | jklare: keep in mind jenkins/jobs only configures jenkins. and zuul/layout.yaml onl configures zuul | 17:59 |
| sdague | jeblair: wfm | 17:59 |
| adam_g | sdague, right | 18:00 |
| clarkb | jeblair: looks good | 18:00 |
| adam_g | sdague, for 2 node devstack thing, running n-api-ec2 on the second node with n-cpu and n-net | 18:00 |
| mmedvede | jeblair: lgtm. I did WIP it though | 18:01 |
| clarkb | adam_g: oh you should base that on the change that runs n-net on the subnode too if its only nova multi host that needs it | 18:01 |
| sdague | https://github.com/openstack-dev/devstack/blob/master/lib/nova#L444-L447 | 18:01 |
| adam_g | sdague, wasn't able to get ec2 metadata working for instances on node 2 from the central API server on the first node | 18:01 |
| clarkb | adam_g: since its based on code that doesn't nova multi host yet | 18:01 |
| jeblair | asselin: https://review.openstack.org/#/c/135452/2 some comments on your proposed spec change; if you agree, we should get it going soon | 18:01 |
| openstackgerrit | Merged openstack-infra/project-config: Precreate temp holding dir in static publish job https://review.openstack.org/136921 | 18:01 |
| *** nadya_ has quit IRC | 18:01 | |
| openstackgerrit | Terry Howe proposed openstack-infra/project-config: Add automated pypi release job for openstacksdk https://review.openstack.org/137159 | 18:01 |
| sdague | adam_g: that's because if you want to use the metadata server you're supposed to set up ip forward rules at the switch level | 18:02 |
| asselin | jeblair, will look into it now | 18:02 |
| clarkb | sdague: that sounds terrible | 18:02 |
| jklare | clarkb: so the job needs to run on a specific node right? and zuul should take care of that? why is node: 'bare-precise || bare-trusty' defined in the projects.yaml while the parameter-function in zuul/layout.yaml is set_node_options_default_precise? | 18:02 |
| sdague | clarkb: which is why I say it's bonghits :) | 18:02 |
| sdague | and that people should use config drive | 18:02 |
| clarkb | jklare: so that jenkins restricts those jobs to those nodes | 18:02 |
| adam_g | sdague, in this case the node's ENABLED_SERVICES=n-cpu,n-net,n-api-meta | 18:02 |
| jeblair | mmedvede: so i can approve that this afternoon, but we do need to collect some +2s on those before then | 18:03 |
| clarkb | jklare: otherwise it could run on any node (it basically helps prevent fires if things go bad) | 18:03 |
| *** ivar-laz_ has joined #openstack-infra | 18:03 | |
| yfried | clarkb: AJaeger: any news about the NOT_REGISTERED icehouse jobs on gate? | 18:03 |
| jeblair | mmedvede: i wouldn't worry about WIPing the system-config changes | 18:03 |
| *** fandi has joined #openstack-infra | 18:03 | |
| jklare | clarkb: so this job will run on precise only or on precise and trusty? | 18:03 |
| *** arxcruz has quit IRC | 18:03 | |
| sdague | so... we're already doing live block migration in this attempted test, I'm not sure why we don't just fix config drive to be able to be block migrated | 18:04 |
| clarkb | yfried: its been fixed | 18:04 |
| mmedvede | jeblair: you mean WIP only project-config? | 18:04 |
| jeblair | clarkb, fungi: okay, i'm going to try to start the "how should we proceed with nodepool" conversation again :) | 18:04 |
| yfried | clarkb: so I should recheck my jobs? | 18:04 |
| yfried | clarkb: tnx | 18:04 |
| openstackgerrit | Merged openstack-infra/project-config: Add missing docimpact for fuel repos https://review.openstack.org/134986 | 18:04 |
| openstackgerrit | Merged openstack-infra/project-config: Add missing docimpact for fuel repos https://review.openstack.org/134986 | 18:04 |
| openstackgerrit | Merged openstack-infra/project-config: Add Cue rtfd hook https://review.openstack.org/136624 | 18:04 |
| *** BobBall is now known as BobBall_AWOL | 18:04 | |
| *** e0ne has quit IRC | 18:05 | |
| *** ivar-lazzaro has quit IRC | 18:06 | |
| fungi | jeblair: eating, but here and can discuss | 18:06 |
| jeblair | clarkb, fungi: so i think we need to decide whether to debug with the current gear, or get async io working first | 18:06 |
| *** mriedem has quit IRC | 18:06 | |
| *** sarob has joined #openstack-infra | 18:06 | |
| *** e0ne has joined #openstack-infra | 18:06 | |
| *** AlexF_ has quit IRC | 18:07 | |
| jeblair | i sort of lean towards getting async io working first, because if there are any network/throughput/etc related problems, it _should_ make the system more resilient to them | 18:07 |
| fungi | jeblair: and also, the work to support that is mostly written | 18:07 |
| clarkb | that sounds reasonable to me | 18:07 |
| jeblair | otoh, if we go with debug the current system, it may be simpler. though we may just end up with "we need robust async io" 2 days later :) | 18:07 |
| sdague | adam_g: ok, I guess we can go down that path | 18:08 |
| *** sarob_ has joined #openstack-infra | 18:08 | |
| *** mrmartin has joined #openstack-infra | 18:08 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: Added experimental-{name}-chef-rake job-template https://review.openstack.org/137134 | 18:08 |
| sdague | I'll have to say all of - http://docs.openstack.org/admin-guide-cloud/content/section_metadata-service.html is just confusing and terrible. It seems like the metadata service should just be in n-net | 18:08 |
| *** kmartin has joined #openstack-infra | 18:08 | |
| *** MaxV has quit IRC | 18:08 | |
| jeblair | so if we want to work on async io, i have two thoughts -- one: read the zuul log and see if we can figure out where it went wrong; two: run a local zuul and see if the behavior manifests again | 18:08 |
| sdague | because you basically have to pair it with it all along anyway | 18:08 |
| jeblair | (because all of the local testing of gear i have done, including running the zuul tests, doesn't show a problem) | 18:09 |
| fungi | jeblair: agreed. i'm not one for dwelling on the past if we already know where we're going | 18:09 |
| *** btran has quit IRC | 18:09 | |
| clarkb | jeblair: I like the idea of local zuul simply because I don't expect the zuul logs to expose much about the async event loop | 18:09 |
| adam_g | sdague, yeah.. running the (n-net, n-api-meta, n-cpu) per compute host is how i've been doing it since multi host came along, at least for something that works out of the box | 18:09 |
| sdague | adam_g: though on https://review.openstack.org/#/c/136179/1 you *also* need to run it on the server I think, otherwise things are going to be wokny | 18:10 |
| sdague | adam_g: yeh, that's fair | 18:10 |
| adam_g | sdague, on which server? the central one? | 18:10 |
| adam_g | node1? | 18:10 |
| sdague | yeh | 18:10 |
| fungi | btw, just checked the logs again, still no gearmanclient timeouts on precise nodepool | 18:10 |
| jeblair | okay, i will start on that this afternoon; i don't know what else we could do in the mean time though :/ | 18:10 |
| sdague | if nothing else, for consistency | 18:10 |
| adam_g | sdague, its enabled by default and served by the monolithic nova-api | 18:10 |
| *** HeOS has quit IRC | 18:10 | |
| jeblair | maybe give new nodepool a second dns entry and add it to the firewall config so it's easier to switch? :) | 18:10 |
| clarkb | jeblair: so thats partially my question. did you want ot hold off on all nodepool changes until the gear stuff is sorted? | 18:10 |
| sdague | adam_g: yeh, but is it relying on the special address in the all in one case? | 18:11 |
| clarkb | because there are now ~3 nodepool changes that would be good to get in | 18:11 |
| *** jamielennox|away is now known as jamielennox | 18:11 | |
| *** cpowell has quit IRC | 18:11 | |
| adam_g | sdague, 169.254.169.254? yeah | 18:11 |
| *** russellb has quit IRC | 18:11 | |
| jeblair | clarkb: i think that it's okay to evolve the nodepool code, but not to switch the server | 18:11 |
| clarkb | jeblair: ok | 18:11 |
| sdague | which it won't be on the compute, right? | 18:11 |
| adam_g | sdague, in all cases, n-net sets the iptables rule to forward instance traffic to the appropriate server | 18:11 |
| clarkb | I can put together a new nodepool dns record and add that to iptables rules | 18:12 |
| adam_g | sdague, to 127.0.0.1:8775 by default, or $my_ip or whatever | 18:12 |
| *** ivar-laz_ has quit IRC | 18:12 | |
| clarkb | any preference for that name? nodepool2.openstack.org? | 18:12 |
| clarkb | or even nodepool-dev.openstack.org? | 18:12 |
| fungi | jeblair: clarkb: right, we're running the same nodepool checkout on both servers, so it seems unlikely that pending nodepool improvements will touch those code paths (if they do, then maybe we put just those on hold) | 18:12 |
| *** russellb has joined #openstack-infra | 18:12 | |
| *** ivar-lazzaro has joined #openstack-infra | 18:12 | |
| *** bcourt has joined #openstack-infra | 18:13 | |
| *** bradjones has quit IRC | 18:13 | |
| *** wenlock has quit IRC | 18:13 | |
| *** sarob has quit IRC | 18:13 | |
| sdague | adam_g: ok | 18:13 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/infra-specs: Update module issues_url & include storyboard https://review.openstack.org/135452 | 18:13 |
| *** shashankhegde has joined #openstack-infra | 18:13 | |
| *** che-arn3 has quit IRC | 18:13 | |
| *** e0ne has quit IRC | 18:14 | |
| *** guest764 has quit IRC | 18:14 | |
| *** guest764 has joined #openstack-infra | 18:14 | |
| clarkb | nibalizer: mmedvede do we want to merge any changes like https://review.openstack.org/#/c/137155/ before nibalizer's dedup change of the integration testing project list gets in? | 18:14 |
| clarkb | nibalizer: have a link to that chang handy? | 18:14 |
| nibalizer | sure sec | 18:14 |
| *** emagana_ has quit IRC | 18:15 | |
| fungi | anybody know if puppet agent is intentionally disabled on git.openstack.org right now? | 18:15 |
| *** emagana has joined #openstack-infra | 18:15 | |
| nibalizer | https://review.openstack.org/#/c/134723/ | 18:15 |
| sdague | adam_g: ok, +2. Also I did the recheck, the icehouse requirements issue should be addressed | 18:15 |
| *** e0ne has joined #openstack-infra | 18:16 | |
| *** pblaho has quit IRC | 18:16 | |
| *** pblaho has joined #openstack-infra | 18:16 | |
| *** cpowell has joined #openstack-infra | 18:16 | |
| bcourt | hi, I'm poking at the setup scripts used by nodepool when configuring an image. nodepool.py seems to explicitly copy only the immediate contents of the scripts directory over to the image. Any subdirectories are ignored. This makes things a little more complicated if I want to use a custom puppet module to apply changes in absense of a puppet master. would folks consider a patch to nodepool t | 18:17 |
| bcourt | o include subdirectories of the script-dir? | 18:17 |
| adam_g | sdague, cool. related, juno grenade is still failing. hoping these will check clean https://review.openstack.org/#/q/I7f06a030cd9f31f3741c1ea4c97a2be3b76b978c,n,z and fix it today | 18:17 |
| clarkb | fungi: it may be related to me rebuilding git.o.o | 18:17 |
| clarkb | fungi: possibly I derped the ssh host key there too | 18:17 |
| fungi | clarkb: oh, maybe. the last reported date is about the same as the uptime | 18:17 |
| clarkb | ya Ibet that is the problem | 18:18 |
| * clarkb fails | 18:18 | |
| fungi | also it seems the dns resolvers on that server are not pointing to unbound | 18:18 |
| fungi | fun times | 18:18 |
| clarkb | fungi: jeblair: I am going to create a nodepool-dev set of DNS records if there is no opposition for that name | 18:18 |
| sdague | adam_g: ... don't you need that in juno? | 18:18 |
| *** jpich has quit IRC | 18:18 | |
| *** dizquierdo has quit IRC | 18:19 | |
| mordred | clarkb, jeblair: ok - from reading the scrollback in a more connected world, it doesn't seem like there are specific things that need to be attacked regarding the novaclient issues from yesterday | 18:19 |
| sdague | I'm +2 on all of that... also, it amazes me that pip fails to do the right thing there | 18:19 |
| *** guest7641 has joined #openstack-infra | 18:19 | |
| mordred | other than needing a newer novaclient and needing to vet whether or not newer novaclient was part of our problems on trusty nodepool | 18:19 |
| clarkb | mordred: no, I think mostly we yell at nova. say don't do that again | 18:19 |
| adam_g | sdague, nope. the juno eventlet had alreayd been bumped to >=0.15.1 | 18:19 |
| mordred | right. but there isn't work to be done by us in that arena right now | 18:20 |
| clarkb | I don't think so | 18:20 |
| fungi | mordred: other than possibly turning on debug logging for novaclient so we can see what the conditions are surrounding that unparseable (in older release) error response | 18:20 |
| mmedvede | clarkb: I forgot about the nibalizer 's refactor patch. +1 on holding off on new module splits until it lands | 18:20 |
| sdague | maybe instead of xstatic packages we should package all our python as js npm packages as that actually knows how to install a thing that depends on another thing | 18:20 |
| clarkb | oh I +2'd that change and it failed tests | 18:20 |
| openstackgerrit | Spencer Krum proposed openstack-infra/infra-specs: WIP of Docker spec https://review.openstack.org/137163 | 18:20 |
| * clarkb rereviews nibalizer's change | 18:20 | |
| *** guest764 has quit IRC | 18:21 | |
| adam_g | sdague, whats more annoying is that we dont hit this conflict on icehouse tempest runs because those run on precise with a distro packaged eventlet of 0.9.x, causing pip to do the correct thing | 18:21 |
| fungi | clarkb: okay, you predicted correctly on the git.o.o puppet issue. i've corrected the known_hosts entry now | 18:21 |
| sdague | adam_g: except when it completely fails to realize that it has to upgrade a dependency | 18:21 |
| clarkb | fungi: ty | 18:21 |
| *** pblaho has quit IRC | 18:22 | |
| sdague | I'm actually not even clear how that's the case | 18:22 |
| *** e0ne has quit IRC | 18:22 | |
| *** teran_ has quit IRC | 18:23 | |
| *** davideagnello has quit IRC | 18:23 | |
| clarkb | nibalizer: lgtm | 18:24 |
| *** emagana has quit IRC | 18:24 | |
| nibalizer | clarkb: woo | 18:24 |
| *** davideagnello has joined #openstack-infra | 18:24 | |
| *** emagana has joined #openstack-infra | 18:24 | |
| openstackgerrit | Merged openstack-infra/project-config: Cleanup gerrit ACLs https://review.openstack.org/136222 | 18:25 |
| jamielennox | is there any point to continually reverifying https://review.openstack.org/#/c/135903/ ? the last few fails have all been due to tests NOT_REGISTERED - is that going to go away if we keep trying? | 18:25 |
| fungi | sdague: it's the situation dstufft was describing last week where pip (in absence of -U) believes the first version directives it encounters for a package name in a given run, so your most specific version constraints need to be mentioned earliest when taking the (depth-first i think?) transitive dependency tree into account | 18:25 |
| clarkb | jamielennox: that should be fixed now /me looks at timestamps | 18:26 |
| clarkb | jamielennox: ya the most recent one is from forever ago | 18:26 |
| fungi | sdague: i'm also wondering if this is a more recent behavior change, because we definitely saw later version constraints causing pip to replace earlier ones it had already installed | 18:27 |
| jamielennox | clarkb: most recent fail? | 18:27 |
| fungi | in old pip | 18:27 |
| clarkb | jamielennox: on that change you linked is from ~5hours ago | 18:27 |
| clarkb | jamielennox: we have since fixed the issue | 18:27 |
| clarkb | if you have evidence of failures from the last hour pointing us at those would be great | 18:27 |
| jamielennox | clarkb: ok, was just thinking forever ago might have been longer than my yesterday | 18:28 |
| jamielennox | clarkb: thanks | 18:28 |
| *** emagana has quit IRC | 18:29 | |
| *** MarkAtwood has quit IRC | 18:29 | |
| clarkb | nodepool-dev records added. | 18:29 |
| clarkb | puppet change to allow that node in firewalls is incoming | 18:29 |
| sdague | fungi: yeh, I don't know. But it seems odd that it doesn't process each stanza atomically | 18:30 |
| sdague | I require this, is it done, no do it | 18:30 |
| sdague | that's I think what used to happen | 18:31 |
| *** emagana has joined #openstack-infra | 18:31 | |
| *** MarkAtwood has joined #openstack-infra | 18:31 | |
| sdague | which ... granted, can cause other crazies | 18:31 |
| *** patrickeast has joined #openstack-infra | 18:31 | |
| sdague | and was the reason for global requirements | 18:31 |
| fungi | sdague: yeah, i recall that being how it worked as well, but maybe that was with version caps combined with install -U | 18:31 |
| sdague | but... it at least does the things it's asked to do | 18:31 |
| sdague | the current behavior is definitively broken | 18:32 |
| fungi | dstufft was suggesting that it likely won't improve until someone implements a dep solver in pip | 18:32 |
| mordred | sdague: amusingly enough, I've had the same npm thought before | 18:32 |
| *** salv-orlando has joined #openstack-infra | 18:33 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Allow traffic from nodepool-dev https://review.openstack.org/137167 | 18:33 |
| *** alexpilotti has joined #openstack-infra | 18:33 | |
| clarkb | jeblair: fungi ^ | 18:33 |
| mordred | sdague: we could also go back to using distro packages of everything in the gate and maintaining our own overlay distro repo. I turns out that _those_ have excellent dependency management - we'd just have to get into the business of maintaining a defacto derivative distro | 18:34 |
| *** sputnik13 has quit IRC | 18:35 | |
| *** rushiagr_away is now known as rushiagr | 18:36 | |
| sdague | mordred: you are vaguely tempting me to build a package management system into devstack | 18:36 |
| *** sputnik13 has joined #openstack-infra | 18:36 | |
| sdague | don't do that | 18:36 |
| *** sarob_ has quit IRC | 18:36 | |
| mordred | sdague: well, if we're going to build a package management system, I'd like to do so generally - I have thoughts | 18:39 |
| *** yfried has quit IRC | 18:40 | |
| *** yfried has joined #openstack-infra | 18:40 | |
| *** jgallard has quit IRC | 18:41 | |
| *** dimtruck is now known as zz_dimtruck | 18:41 | |
| clarkb | meh just run gentoo/arch/sabayon | 18:41 |
| openstackgerrit | Dan Prince proposed openstack-infra/reviewday: Add os-net-config. https://review.openstack.org/137168 | 18:41 |
| clarkb | (seriously though some distros have solved these issues not sure why we would solve them in the context of rhel/ubuntu_ | 18:41 |
| mordred | clarkb: what do you mean? | 18:43 |
| *** Guest32131 has quit IRC | 18:43 | |
| clarkb | mordred: well a couple things. one is gentoos slots | 18:43 |
| mordred | clarkb: like, not being obtuse - how would gentoo/arch/sabayon help? I'm pretty much all ears | 18:43 |
| clarkb | mordred: one of the places this comes up is we want to optionally have latest libvirt | 18:43 |
| clarkb | well gentoos has solved that | 18:43 |
| dprince | pleia2: thanks for looking at the reviewday fix | 18:44 |
| mordred | oh - I'm _just_ talking about solving the fact that pip doesn't do dependencies | 18:44 |
| clarkb | the other thing is simple packaging system that makes it easy to slap stuff over the OS | 18:44 |
| clarkb | arch's ports are really damn good at that | 18:44 |
| mordred | which means we'd have to express the depends in some alternate scheme - but if we get extra features from gentoo like slots, neat - I'd be happy to maintain a gentoo whatever if that's the right thing | 18:44 |
| *** Ark has joined #openstack-infra | 18:44 | |
| clarkb | rather than taking say ubuntu and making a derivative distro | 18:44 |
| *** emagana has quit IRC | 18:44 | |
| mordred | nod | 18:45 |
| pleia2 | dprince: np | 18:45 |
| *** Ark is now known as Guest19472 | 18:45 | |
| *** sputnik13 has quit IRC | 18:45 | |
| mtreinish | clarkb: +1 for arch :) | 18:45 |
| mtreinish | except for you know the python==python3 craziness... | 18:46 |
| *** sputnik13 has joined #openstack-infra | 18:47 | |
| *** achanda has joined #openstack-infra | 18:47 | |
| *** nfedotov has joined #openstack-infra | 18:47 | |
| *** emagana has joined #openstack-infra | 18:48 | |
| *** ZZelle_ has joined #openstack-infra | 18:49 | |
| *** stevemar has joined #openstack-infra | 18:50 | |
| fungi | sdague: on the ironic/nova co-gating thread, i believe devananda mentioned in a summit session that the triggering incident was that someone approved changes to nova which broke the contract and simultaneously adjusted the contract tests to make them pass | 18:50 |
| mordred | yes | 18:51 |
| *** baoli has quit IRC | 18:51 | |
| *** achanda_ has joined #openstack-infra | 18:52 | |
| fungi | sdague: which still in my mind is not a reason to go back to enforcing it with an external testset, just a human (reviewer) problem which needs to get solved | 18:52 |
| *** achanda has quit IRC | 18:52 | |
| sdague | fungi: agreed, and I want to get the instance | 18:53 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 18:53 | |
| *** koolhead17 has joined #openstack-infra | 18:54 | |
| *** BharatK has joined #openstack-infra | 18:54 | |
| *** gokrokve has joined #openstack-infra | 18:54 | |
| *** timrc-afk is now known as timrc | 18:54 | |
| *** jp_at_hp has quit IRC | 18:54 | |
| *** AJaeger_ has joined #openstack-infra | 18:55 | |
| *** mmaglana has quit IRC | 18:57 | |
| *** ivar-lazzaro has quit IRC | 18:58 | |
| *** MaxV has joined #openstack-infra | 18:58 | |
| *** harlowja is now known as harlowja_away | 18:58 | |
| vishy | so does anyone know why the stable postgres tests are failing? | 18:59 |
| *** ivar-lazzaro has joined #openstack-infra | 18:59 | |
| *** mwagner_lap has quit IRC | 18:59 | |
| *** ivar-lazzaro has quit IRC | 19:00 | |
| *** gokrokve has quit IRC | 19:00 | |
| *** gokrokve has joined #openstack-infra | 19:00 | |
| *** ivar-lazzaro has joined #openstack-infra | 19:00 | |
| fungi | meeting time | 19:00 |
| mordred | jeblair, clarkb: so - here's what I'm thinking re: dib images and rackspace | 19:00 |
| mordred | oh | 19:00 |
| mordred | I guess I can say it there | 19:01 |
| mordred | :) | 19:01 |
| *** Mike24 has left #openstack-infra | 19:01 | |
| *** teran has joined #openstack-infra | 19:03 | |
| *** bswartz has joined #openstack-infra | 19:03 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/136504 | 19:03 |
| mordred | essentially - but in an init script that tries to mount a config drive partition if it's there, and if it is able to do that and there is network information in the config drive instance, replace the DHCP config with static config based on what's pulled from config drive. if there isn't a config drive, then fail with no error | 19:03 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Add ssh key to root user https://review.openstack.org/137176 | 19:03 |
| devananda | sdague: so I started digging into that (caveat distractions) this morning | 19:04 |
| devananda | sdague: the original API contract tests were deleted from Nova, and in their place is a more automated thing now which I haven't looked at much | 19:04 |
| *** achanda_ has quit IRC | 19:04 | |
| *** teran has quit IRC | 19:04 | |
| devananda | sdague: see https://review.openstack.org/#/c/111425/ | 19:04 |
| *** achanda has joined #openstack-infra | 19:04 | |
| *** emagana has quit IRC | 19:04 | |
| *** teran has joined #openstack-infra | 19:04 | |
| sdague | right, those tests were deleted on the driver land, which had all kinds of ironic folks on it | 19:04 |
| *** ivar-laz_ has joined #openstack-infra | 19:04 | |
| sdague | I just went and found that | 19:04 |
| *** ccowley has joined #openstack-infra | 19:05 | |
| sdague | that is definitely not the commit that broke ironic | 19:05 |
| devananda | nope | 19:05 |
| *** achanda has quit IRC | 19:05 | |
| sdague | so I want to know the commit that broke ironic | 19:05 |
| *** BharatK has quit IRC | 19:05 | |
| sdague | so that we can work backwards from that and figure out what the actual failure condition was | 19:06 |
| *** achanda has joined #openstack-infra | 19:07 | |
| *** harlowja_away is now known as harlowja | 19:07 | |
| *** ivar-lazzaro has quit IRC | 19:08 | |
| dolphm | what does elastic-recheck-success do? the help message isn't too helpful | 19:08 |
| sdague | it's kind of the text version of the webpage | 19:09 |
| *** jistr has joined #openstack-infra | 19:09 | |
| dolphm | sdague: so i should just run it and find out? | 19:10 |
| sdague | yeh, or read the code | 19:10 |
| dolphm | sdague: but lazy | 19:10 |
| sdague | yeh, then just run it | 19:11 |
| dolphm | sdague: fwiw, i'm finally looking at moving spandex into elastic-recheck/ | 19:11 |
| sdague | dolphm: nice :) | 19:11 |
| *** mmaglana has joined #openstack-infra | 19:11 | |
| devananda | sdague: there are three commits which show up as having changed that file (aside from the deletion), none of which broke Ironic's tests during that time | 19:12 |
| *** ecosta has joined #openstack-infra | 19:13 | |
| devananda | sdague: I think I was thinking of this patch, though - https://review.openstack.org/#/c/68942/ | 19:14 |
| dstufft | fungi: sdague fwiw the "first found" thing is regardless of -U, the fact -U fixed it was more ana ccident becasue the first found was a >= so upgrading it upgraded to the latest | 19:15 |
| *** ihrachyshka has quit IRC | 19:15 | |
| *** btran has joined #openstack-infra | 19:15 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: Added experimental-{name}-chef-rake job-template https://review.openstack.org/137134 | 19:16 |
| fungi | dstufft: oh, fair point. if the first round was a cap at <1.2.3 and then a later entry requested the same package at <1.0 then you'd end up with a later version than you actually wanted | 19:16 |
| *** koolhead17 has quit IRC | 19:17 | |
| *** koolhead17 has joined #openstack-infra | 19:17 | |
| *** gokrokve has quit IRC | 19:17 | |
| dstufft | fungi: yea | 19:18 |
| *** baoli has joined #openstack-infra | 19:18 | |
| *** gokrokve has joined #openstack-infra | 19:18 | |
| *** MaxV has quit IRC | 19:19 | |
| *** achanda_ has joined #openstack-infra | 19:19 | |
| *** achanda has quit IRC | 19:20 | |
| harlowja | anyway i can get https://review.openstack.org/#/c/136657/ merged so that elastic recheck finds this one (it still seems to be happening alot) | 19:20 |
| *** arosen has joined #openstack-infra | 19:20 | |
| *** emagana has joined #openstack-infra | 19:21 | |
| *** ociuhandu has quit IRC | 19:21 | |
| *** baoli has quit IRC | 19:22 | |
| *** koolhead17 has quit IRC | 19:22 | |
| *** gokrokve has quit IRC | 19:22 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/elastic-recheck: Add query for bug 1395368 https://review.openstack.org/136657 | 19:23 |
| uvirtbot | Launchpad bug 1395368 in tempest "ExternalNetworksTestJSON.test_delete_external_networks_with_floating_ip (icehouse) failures" [Undecided,Incomplete] https://launchpad.net/bugs/1395368 | 19:23 |
| *** amcrn has joined #openstack-infra | 19:23 | |
| *** mmaglana has quit IRC | 19:24 | |
| *** rfolco has quit IRC | 19:25 | |
| *** mmaglana_ has joined #openstack-infra | 19:26 | |
| *** ayoung has quit IRC | 19:31 | |
| *** ivar-laz_ has quit IRC | 19:31 | |
| *** otter768 has joined #openstack-infra | 19:31 | |
| *** ivar-lazzaro has joined #openstack-infra | 19:32 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add MongoDB plugin support https://review.openstack.org/131301 | 19:33 |
| *** dangers_away is now known as dangers | 19:35 | |
| *** mwagner_lap has joined #openstack-infra | 19:35 | |
| *** mpavlase has quit IRC | 19:36 | |
| sdague | devananda: so that was litterally a million years ago, and should have just been a conversation with danpb and vishy | 19:36 |
| *** otter768 has quit IRC | 19:36 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/136504 | 19:36 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Add ssh key to root user https://review.openstack.org/137176 | 19:36 |
| vishy | sdague any info on the postgres stable fails? | 19:37 |
| vishy | is anyone looking at it? | 19:37 |
| sdague | vishy: you have a link to a failed job? | 19:37 |
| vishy | is it a known issue? | 19:37 |
| vishy | sure | 19:37 |
| vishy | the stable builds have been failing for 3 days | 19:37 |
| *** adalbas has quit IRC | 19:37 | |
| sdague | I don't know anyone looking at it, but I won't count as being totally in the know | 19:37 |
| devananda | sdague: Juno cycle isn't a million years ago. That's Iceage, I mean Icehouse, territory :) | 19:37 |
| adam_g | vishy, yeah | 19:37 |
| sdague | devananda: in this conversation it's a really long time ago | 19:38 |
| adam_g | viglesias, if its the failure im thinking of, https://review.openstack.org/#/c/136538/ is meant to address it | 19:38 |
| devananda | sdague: three months is a "really long time ago"?? | 19:38 |
| adam_g | vishy, ^^ | 19:38 |
| sdague | devananda: it was before the driver land. I thought we were talking about something after | 19:38 |
| vishy | sdague: http://logs.openstack.org/periodic-stable/periodic-nova-python26-juno/6f8aa31 | 19:39 |
| adam_g | vishy, yeah, thats the one. see 136538 | 19:39 |
| sdague | so it really is a long time ago, because we are talking about changes to code that is now obsolete | 19:39 |
| vishy | adam_g: ah thx | 19:39 |
| adam_g | vishy, we're waiting on a couple of other things to fix icehouse before the grenade job will let that merge, tho | 19:39 |
| vishy | i see | 19:40 |
| vishy | ok i so sounds like it is being looked at | 19:40 |
| devananda | sdague: are we talking about whether non-voting tests provide input which is valued? or something else? | 19:40 |
| vishy | I won’t try and dig in myself then | 19:40 |
| vishy | adam_g: if you link me the other reviews i can review them though | 19:40 |
| *** ChuckC has quit IRC | 19:40 | |
| adam_g | vishy, review on that backport welcome, it wasn't a clean cherry-pick and required some refactoring | 19:40 |
| adam_g | vishy, they're both gating now, https://review.openstack.org/#/q/I7f06a030cd9f31f3741c1ea4c97a2be3b76b978c,n,z | 19:40 |
| *** ddieterly has quit IRC | 19:40 | |
| vishy | ok cool | 19:41 |
| sdague | devananda: I'm trying to figure out why we need that job in the check queue when this should be something which is fully covered by defined interfaces | 19:41 |
| devananda | sdague: let's actually clarify what it is we're talking about. Having Ironic provide non-voting test results? Ironic running in the check but not gate queue? And whether we're discussing well-defined interfaces, poorly-defined interfaces, or support from other projects like tempest and devstack? | 19:42 |
| *** Guest19472 has quit IRC | 19:45 | |
| *** ecosta has quit IRC | 19:45 | |
| devananda | sdague: for reference, I've dug up a change in each of devstack, devstack-gate, and tempest -- all which broke Ironic's gate during Juno, but before it landed in Nova. | 19:45 |
| sdague | devananda: sure, before the driver landed, which changes the surfaces dramatically | 19:45 |
| devananda | sdague: so I'm going to stick to saying that non-voting check tests *were* ignored. Yes, that's a social problem. | 19:45 |
| *** tnovacik has joined #openstack-infra | 19:47 | |
| devananda | sdague: so I'm going to stick to saying that non-voting check tests *were* ignored. Yes, that's a social problem, and yes, we all have been using a technical solution | 19:47 |
| sdague | I'm not arguing to make them non-voting. I'm talking about figuring out what actually needs to be tested to keep things working, instead of running the world | 19:47 |
| devananda | ah | 19:47 |
| sdague | because I think that in a bunch of cases we don't need that job | 19:47 |
| devananda | ok - so that's why I'm asking for clarification :) | 19:47 |
| sdague | we can solve it with the right tests in other jobs | 19:47 |
| sdague | right, so I thought that my email made that clear | 19:48 |
| sdague | if not, I messed up on intent | 19:48 |
| devananda | your email and the -2 vote on my patch seem like you object to ironic being in the check or gate queue for [nova, devstack, tempest, ...] | 19:48 |
| devananda | that was my read, anyway | 19:48 |
| *** mmaglana_ has quit IRC | 19:48 | |
| *** mmaglana has joined #openstack-infra | 19:49 | |
| *** achanda_ has quit IRC | 19:49 | |
| devananda | I'm delighted to have a narrow-but-meaningful test surface, and really don't want to test the world on every commit | 19:49 |
| sdague | the -2 vote on your patch was to not put it into the gate queue | 19:49 |
| sdague | which I think we don't need with clean check | 19:49 |
| sdague | the email came far before the -2 | 19:51 |
| *** andymaier has quit IRC | 19:52 | |
| devananda | sdague: consider me perpetually behind on email ... | 19:52 |
| *** Ark has joined #openstack-infra | 19:52 | |
| *** Ark is now known as Guest46583 | 19:53 | |
| devananda | sdague: so, question. is your objection to putting ironic in the gate queue based on existing precedent, or on the big-tent-discussions // longer term goals // ongoing refactoring? | 19:53 |
| *** mrmartin has quit IRC | 19:54 | |
| sdague | existing precedent | 19:54 |
| sdague | there are so many jobs on devstack / tempest that if we replicated them on gate as well we'd basically never be able to merge code | 19:54 |
| *** Ryan_Lane has quit IRC | 19:55 | |
| sdague | a project's spurious failure ends up being massively magnified there | 19:55 |
| sdague | which is often the place that we need to change code to help with races | 19:55 |
| sdague | we haven't had check match gate in devstack or tempest since we instituted clean check | 19:56 |
| sdague | and it's been fine | 19:56 |
| devananda | I know I'm behind on many of the things in project-config | 19:57 |
| devananda | should Ironic have a "template: integrated-gate" entry? | 19:57 |
| *** ildikov has joined #openstack-infra | 19:57 | |
| mtreinish | sdague: fyi, we did manage to thread the needle last week when a stale check pass had a really high stable failure rate during the reqs fires | 19:57 |
| mtreinish | it was fun to try and unwedge that... | 19:58 |
| *** gokrokve has joined #openstack-infra | 19:59 | |
| sdague | mtreinish: well, stale check is a thing that we might need to address again then | 19:59 |
| sdague | devananda: so in a world where we are blindly following policy vs. being smart, yes. However, I think we should be more smart :) | 19:59 |
| devananda | sdague: heh :) | 20:00 |
| *** cnesa has quit IRC | 20:00 | |
| *** HeOS has joined #openstack-infra | 20:01 | |
| *** mmaglana has quit IRC | 20:01 | |
| *** ChuckC has joined #openstack-infra | 20:01 | |
| *** mmaglana has joined #openstack-infra | 20:02 | |
| devananda | sdague: not to go into this bikeshed, but just to point out that that's very inconsistent with respect to other recently-integrated projects | 20:02 |
| pleia2 | also, AJaeger and I got the Ansible playbooks for Zanata from Red Hat, I'll be working on translating them into a Puppet module, and more eyeballs on the related spec would be great too: https://review.openstack.org/#/c/133222 | 20:02 |
| * pleia2 should add things to the agenda \o/ | 20:02 | |
| sdague | devananda: so we've got to start somewhere. Apparently everyone wants to do this be smarter thing, unless we start with their project. | 20:02 |
| *** CI has joined #openstack-infra | 20:03 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: adjust swift testing definitions to things that actually test swift https://review.openstack.org/137184 | 20:03 |
| devananda | sdague: nope. I'm fine starting with Ironic. but also, I'm not an infra-config expert and continually playing catch up to how to model tests there | 20:03 |
| sdague | yep, you will notice I did start 3 different threads today about smarter testing | 20:04 |
| mordred | pleia2: we could also just run them as ansible playbooks ... | 20:04 |
| sdague | not just the ironic one | 20:04 |
| devananda | sdague: if you're saying that "clean check" + ironic being in the check queue for devstack, tempest, etc, means that someone breaking Ironic's tempest tests will notice it when they propose a patch to tempest | 20:04 |
| pleia2 | mordred: consensus on this was "no" | 20:04 |
| *** jedimike has quit IRC | 20:05 | |
| sdague | devananda: yes I am saying that | 20:05 |
| mordred | pleia2: darn | 20:05 |
| devananda | sdague: and thus there's no need to have that same job in tempest's gate -- and that that's teh direction you want some?many?all? projects to go | 20:05 |
| devananda | sdague: then great :) | 20:05 |
| sdague | devananda: yes | 20:05 |
| devananda | \o/ | 20:05 |
| pleia2 | mordred: we don't want to hurt the brains of people duplicating our infra by introducing another config mgmt system for service configuration (it's ok for the weird things ansible does for us now) | 20:05 |
| sdague | I'm also saying, for devstack, I'd like to figure out a way to do service testing without running all of tempest | 20:05 |
| mtreinish | sdague: bring back the exercise? :) | 20:06 |
| sdague | because having every job decide to test their devstack setup by "do all the tests run" is actually very far away from where the devstack code ends | 20:06 |
| devananda | sdague: also to note, I originally proposed this change two months ago. which is like, for ever ago :) | 20:06 |
| devananda | sdague: I've been asking clarkb how to do that for a year now! (without using really long regex's) | 20:06 |
| *** amuller has quit IRC | 20:07 | |
| *** achanda has joined #openstack-infra | 20:07 | |
| mtreinish | sdague: so we just need to cleanup the tempest tags and make them useful | 20:07 |
| sdague | devananda: so... actually, I think you and I are talking different things | 20:07 |
| clarkb | devananda: asking me how to do what?g | 20:07 |
| devananda | sdague: I'll be sooo happy when there is a job which tests only the things that Ironic cares about, and when I can see that job voting in the check queue for other things that want it there | 20:07 |
| sdague | devananda: sure, but that job is not going to be running on every devstack commit | 20:07 |
| clarkb | oh that. I have nothing to do with that | 20:07 |
| *** andreykurilin_ has quit IRC | 20:07 | |
| *** andreykurilin_ has joined #openstack-infra | 20:08 | |
| *** mpavlase has joined #openstack-infra | 20:08 | |
| sdague | because we need to define a smaller set of things which is did devstack do it's job | 20:08 |
| *** bcourt has quit IRC | 20:08 | |
| dteselkin | fungi, I've checked network on my zuul server. tcpdump shows traffic from port 29418, and in the logs there are messages from https://github.com/openstack-infra/zuul/blob/master/zuul/lib/gerrit.py#L40 | 20:08 |
| *** aysyd has quit IRC | 20:08 | |
| dteselkin | Here they are http://paste.openstack.org/show/138358/ | 20:08 |
| dteselkin | So it doesn't look like a network issue | 20:09 |
| sdague | mtreinish: which... is sort of the exercise. Honestly I'd be happy if it was a healthcheck tag in tempest or something | 20:09 |
| fungi | dteselkin: it looks like it's saying it received no data from the gerrit event stream | 20:09 |
| *** mmaglana has quit IRC | 20:09 | |
| sdague | the exercises were completely terrible to debug though, and also kind of ran into the issue that they depended on the client output | 20:09 |
| fungi | dteselkin: or rather received empty events | 20:09 |
| sdague | which again, was actually way outside devstack control | 20:10 |
| *** shashankhegde has quit IRC | 20:10 | |
| devananda | sdague: if/once devstack and tempest are able to consume libraries from within the trees of the projects we want to test, then sure. but we're not there yet AFAIK | 20:10 |
| sdague | devananda: a job which tests only the things that Ironic cares about == ironic functional job | 20:11 |
| dteselkin | Oh, I forget to sayd that I modified line where logs are generated | 20:11 |
| dteselkin | * to say | 20:11 |
| sdague | devananda: well, on the devstack side, soon. chmouel is putting up a proposal for that in qa-specs | 20:11 |
| dteselkin | I replaced .debug with .info and remove line 41 | 20:11 |
| fungi | dteselkin: i was about to mention that i don't see that string in our zuul ;) | 20:11 |
| *** ayoung has joined #openstack-infra | 20:11 | |
| *** achanda has quit IRC | 20:11 | |
| *** achanda has joined #openstack-infra | 20:12 | |
| fungi | dteselkin: er, in our zuul production debug logs i mean | 20:12 |
| dteselkin | :) | 20:12 |
| fungi | i do see it in the source tree of course | 20:12 |
| *** mwagner_lap has quit IRC | 20:13 | |
| *** mwagner_lap has joined #openstack-infra | 20:13 | |
| dteselkin | I've added line 41 so now I'll be able to see the messages next time it hangs | 20:13 |
| *** jgrimm is now known as zz_jgrimm | 20:13 | |
| devananda | sdague: great. I think we are (and have been) saying very similar things about that. we might just be talking about things in different times | 20:13 |
| sdague | devananda: yes, I think I agree | 20:14 |
| *** ddieterly has joined #openstack-infra | 20:14 | |
| *** ccowley_ has joined #openstack-infra | 20:14 | |
| nibalizer | fungi: so im about to step out for some lunch | 20:14 |
| *** ccowley has quit IRC | 20:14 | |
| fungi | dteselkin: yes, it would be useful to compare the json blobs from before and after the change in behavior | 20:14 |
| nibalizer | so if you need me around you have me for about 15 mintues or wil have to wait a couple hours | 20:15 |
| fungi | nibalizer: cool, i'm pulling up those changes now | 20:15 |
| fungi | nibalizer: the refactor one first | 20:15 |
| dteselkin | fungi, and it's useful to know what data it receives when the problem happen | 20:16 |
| *** ccowley_ is now known as ccowley | 20:16 | |
| *** ldnunes has quit IRC | 20:16 | |
| devananda | sdague: patch abandoned | 20:16 |
| fungi | nibalizer: you want to rebase your system-config split changes for pip and hithub on top of 134723? | 20:17 |
| *** zz_jgrimm is now known as jgrimm | 20:18 | |
| nibalizer | sure | 20:19 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Support multiple image formats in a diskimage https://review.openstack.org/126747 | 20:19 |
| mordred | clarkb: wow. ok. so - the information we need to get an IP up and going is there | 20:19 |
| *** jgrimm is now known as zz_jgrimm | 20:19 | |
| mordred | clarkb: in the rax config drive info | 20:19 |
| *** emagana has quit IRC | 20:19 | |
| clarkb | mordred: does it match what was nova agent configured? | 20:19 |
| fungi | nibalizer: i'm about to approve it, just going over the puppet apply job logs carefully to make sure it did those things | 20:19 |
| mordred | HOWEVER, it is amazing how much information that nova knows about this instance is _not_ in config drive | 20:19 |
| *** achanda has quit IRC | 20:19 | |
| mordred | clarkb: yes. | 20:19 |
| mordred | clarkb: or, at least, it matches enough that we can do what we need to do | 20:19 |
| clarkb | ok all three of those nodepool hcnages I linked in the meeting should be ready for review | 20:19 |
| *** emagana has joined #openstack-infra | 20:20 | |
| fungi | mordred: which file path in the configdrive? i didn't see ip addresses/gateways/et cetera in there | 20:20 |
| mordred | clarkb: which is, it'll make sure that the node has a public ip | 20:20 |
| *** emagana has quit IRC | 20:20 | |
| jroll | mordred: is there something in particular you want in configdrive, that isn't? | 20:20 |
| clarkb | mordred: what about private ip? we need those too | 20:20 |
| mordred | jroll: literally everything that's in the server_info entry for the nova host in the external api | 20:20 |
| *** xyang0 has quit IRC | 20:21 | |
| *** achanda has joined #openstack-infra | 20:21 | |
| jroll | mordred: is that a thing nova typically does? | 20:21 |
| mordred | jroll: also, intended gateway for the network would be nice ... but we can fudge it | 20:21 |
| *** emagana has joined #openstack-infra | 20:21 | |
| mordred | jroll: no idea - I mean, just having dhcp work would be stellar | 20:21 |
| jroll | mordred: I want to see, where is configdrive located | 20:21 |
| jroll | right... | 20:21 |
| clarkb | lol fudging your gateway | 20:21 |
| mordred | you have to boot with --config-drive set | 20:21 |
| jroll | blah | 20:21 |
| fungi | jroll: on my systems you need to then mount /dev/xvdd | 20:21 |
| mordred | then | 20:21 |
| mordred | mount /dev/disk/by-label/config-2 /mnt/config | 20:22 |
| *** rushiagr is now known as rushiagr_away | 20:22 | |
| mordred | cat /mnt/config/./ec2/latest/meta-data.json | 20:22 |
| clarkb | I am going to go to lunch in a few minutes, but reviews on those nodepool changes would be excellent | 20:23 |
| fungi | EC2!!! | 20:23 |
| mordred | and you'll see the IP in local-ipv4 | 20:23 |
| mordred | fungi: don't get me started | 20:23 |
| fungi | i just assumed i shouldn't need to look in ec2 :/ | 20:23 |
| mordred | it's not in the openstack namespace | 20:23 |
| mordred | it's also wrong | 20:23 |
| fungi | because, you know, there next to it is a directory called "openstack" | 20:23 |
| *** ccowley has quit IRC | 20:23 | |
| mordred | because it's the public IP | 20:23 |
| mordred | BUT | 20:23 |
| mordred | I don't care | 20:23 |
| mordred | because the data is there | 20:23 |
| clarkb | mordred: we need private as well | 20:23 |
| mordred | clarkb: why? | 20:23 |
| clarkb | mordred: multi node testing | 20:23 |
| *** ccowley has joined #openstack-infra | 20:23 | |
| *** andreykurilin_ has quit IRC | 20:24 | |
| clarkb | but maybe nodepool can give the nodes that info | 20:24 |
| mordred | clarkb: sorry - I have not fleshed out step 2 here | 20:24 |
| fungi | hey, lookit there... ip addresses | 20:24 |
| mordred | yeah | 20:24 |
| mordred | I just want this to get a basic workable ip address | 20:24 |
| clarkb | do you get masks? | 20:24 |
| *** andreykurilin_ has joined #openstack-infra | 20:24 | |
| clarkb | or is that left out along with the router? | 20:24 |
| mordred | nope. but they're all the same | 20:24 |
| *** achanda has quit IRC | 20:24 | |
| clarkb | that seems dangerous to rely on | 20:24 |
| fungi | um | 20:24 |
| fungi | yeah, no netmask, no gateway | 20:24 |
| mordred | I don't think so - I find it highly unlikely that they're going to start getting clever there | 20:25 |
| fungi | not sure this is safely sufficient | 20:25 |
| clarkb | so given the derpyness of that data | 20:25 |
| clarkb | is it better than just using nova agent? | 20:25 |
| mordred | and we run that on our hp nodes too? | 20:25 |
| *** achanda has joined #openstack-infra | 20:25 | |
| clarkb | mordred: sure, they will noop on hp ya? | 20:25 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Pull pip module out. https://review.openstack.org/126205 | 20:25 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Removing github module. https://review.openstack.org/126450 | 20:25 |
| fungi | also no ipv6 in there | 20:25 |
| nibalizer | fungi: okay all set | 20:26 |
| JayF | mordred: on Rackspace cloud, look at vendor_data and some of that network info is there (I /think/ we enabled that patch on virt cloud as well as OnMetal, imbw) ... it's a downstreamed version of the patch here: https://review.openstack.org/#/c/85673/ | 20:26 |
| fungi | nibalizer: thanks, going over those now too | 20:26 |
| fungi | nibalizer: have a good lunch and i'll catch up with you when you're done | 20:26 |
| clarkb | or is nova agent really stupid | 20:26 |
| nibalizer | ya github depends on pip too so we dont have merge conflicts when we try to land it right after pip | 20:26 |
| mordred | JayF: nope | 20:26 |
| mordred | JayF: only a list of IP whitelists | 20:26 |
| jroll | JayF: that's only onmetal | 20:26 |
| clarkb | anyways must food. curious to see what you figure out | 20:26 |
| *** markmcclain has joined #openstack-infra | 20:26 | |
| JayF | jroll: mordred: :( | 20:27 |
| *** esker has quit IRC | 20:27 | |
| markmcclain | I've got a question about a skipped job in zuul: https://review.openstack.org/127257 | 20:27 |
| jroll | JayF: though maybe we could talk virt into enabling that | 20:27 |
| mordred | jroll: if we can't have dhcp, putting the actual network info (all of it) into the config drive would be hella nice | 20:27 |
| markmcclain | trying to determine why it is marked as skipped | 20:27 |
| nibalizer | fungi: okay thanks! | 20:27 |
| fungi | mordred: i think that's what johnthetubaguy said was in the works | 20:27 |
| openstackgerrit | Merged openstack-infra/system-config: Source modules.env from apply-test.sh https://review.openstack.org/134723 | 20:27 |
| jroll | mordred: we do that in onmetal, we can likely do it in virt as well but that wouldn't be up to me personally | 20:27 |
| *** tnovacik has quit IRC | 20:27 | |
| jroll | it's just a config option afaik | 20:28 |
| *** ChuckC_ has joined #openstack-infra | 20:28 | |
| *** sarob has joined #openstack-infra | 20:28 | |
| JayF | jroll: we'd also have to make sure no bad info leaked from nova | 20:29 |
| *** dprince has quit IRC | 20:29 | |
| *** xyang0 has joined #openstack-infra | 20:29 | |
| *** MaxV has joined #openstack-infra | 20:29 | |
| jroll | JayF: what sort of bad info? idgi | 20:29 |
| *** Guest46583 has quit IRC | 20:29 | |
| mordred | JayF: you're already putting the root password into the stuff in config drive - I cannot possibly imagine what _worse_ thigns could leak | 20:29 |
| jroll | JayF: obviously we would need to test the output before deploying, I don't think it will be an issue though | 20:29 |
| JayF | jroll: like when we enabled it for onmetal, we had to play with the policy.json to get the right fields exposed and some undesired fields unexposed | 20:29 |
| openstackgerrit | Terry Howe proposed openstack-infra/project-config: Add automated pypi release job for openstacksdk https://review.openstack.org/137159 | 20:29 |
| jroll | JayF: ugh | 20:29 |
| JayF | mordred: ++ there's a reason OnMetal only supports key based auth | 20:29 |
| * morganfainberg stepped into a scary conversation. | 20:30 | |
| mordred | morganfainberg: back away slowly | 20:30 |
| JayF | this isn't that bad of a conversation :) | 20:30 |
| fungi | morganfainberg: it's nova not keystone, so you can just pretend you never saw that ;) | 20:30 |
| * morganfainberg reads "root password" and "config drive". | 20:30 | |
| JayF | just mordred giving us support for https://review.openstack.org/#/c/85673/ :P | 20:30 |
| morganfainberg | I think I should pretend I didn't see it. | 20:30 |
| fungi | morganfainberg: yeah, pretty insane | 20:30 |
| *** nadya_ has joined #openstack-infra | 20:31 | |
| jroll | to be fair, how else do you get the root password generated by nova into the instance (and still support password auth) | 20:31 |
| mordred | jroll: other than being snarky in IRC, is there anything I can do to get such a thing prioritized by the fine folks in virt? | 20:31 |
| mordred | jroll: I'd say "don't support password auth" | 20:31 |
| ZZelle_ | fungi, clarkb, hi | 20:32 |
| jroll | mordred: I would agree :P | 20:32 |
| yfried | anyone familiar with the process of creating a new job (experimental) in openstack-infra/project-config? | 20:32 |
| *** ChuckC has quit IRC | 20:32 | |
| morganfainberg | So related to infra stuff. Anyone have contacts at Microsoft that like us? I'm trying to get some support for 3rd party CI on adfs and keystone. | 20:32 |
| jroll | mordred: so, I can push for it or you can, I suppose, idk what the easiest route is | 20:32 |
| ZZelle_ | fungi, clarkb, when you have some times, could you have a look to https://review.openstack.org/116035 (was approved) | 20:32 |
| ZZelle_ | ? | 20:32 |
| jroll | mordred: the only extra thing I can do is spam internal IRC and bug trackers :P | 20:32 |
| * jroll makes a ticket | 20:33 | |
| * jroll or something | 20:33 | |
| JayF | mordred: we already do support key based; there's just no option to not set a root p/w | 20:33 |
| dolphm | elastic-recheck is only storing 10 days of logs now, correct? | 20:33 |
| mordred | jroll: well, with troy gone and pvo not hanging out in channel anymore, I'm not sure who the right person to poke is | 20:33 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Fixed bug in working directories. https://review.openstack.org/137197 | 20:33 |
| jroll | mordred: ok, yeah, poking people now | 20:34 |
| fungi | ZZelle_: that rebase looks fine, so i reapproved it | 20:34 |
| ZZelle_ | fungi, thanks | 20:34 |
| dolphm | or elastic search / kibana, anyway | 20:34 |
| jroll | mordred: idk how soon it could get out but will do what I can | 20:34 |
| fungi | dolphm: yep | 20:35 |
| *** timrc is now known as timrc-afk | 20:36 | |
| fungi | dolphm: it's all we can reasonably store given the volume of log data we collect and the extent to which it explodes when indexed into elasticsearch (no compression and annotated with tags/metadata) | 20:36 |
| fungi | dolphm: the crazy thing is, we strip debug-level log lines before we load them too, so that's just the volume of non-debug logs from jobs | 20:36 |
| *** timrc-afk is now known as timrc | 20:37 | |
| dolphm | fungi: thanks! | 20:37 |
| fungi | dolphm: we've gotten the log volume down a bit and could probably extend that back out some, but not more than maybe a week best case | 20:38 |
| fungi | we've got 8tb of storage across the cluster and seem to be utilizing it at ~50% presently | 20:38 |
| fungi | but also change volume has been lower as usual since the summit, so this is slump-size | 20:38 |
| sdague | fungi: it picked up again last week | 20:39 |
| fungi | sdague: yep, i expect to see it around 75% utilization next week | 20:39 |
| sdague | we merged 525 commits last week | 20:39 |
| fungi | not too shabby | 20:40 |
| sdague | https://github.com/openstack/openstack/graphs/commit-activity | 20:40 |
| fungi | highest weekly total across those repos for a couple months | 20:40 |
| clarkb | 6tb right? | 20:41 |
| *** jistr has quit IRC | 20:41 | |
| clarkb | also we need ~1tb free to handle failovers | 20:41 |
| sdague | clarkb: https://review.openstack.org/#/c/136179/ adam_g's metadata server change is passing tests now | 20:41 |
| sdague | fungi: yeh, people were busy last week | 20:42 |
| fungi | clarkb: oh, yes 6tb not 8 | 20:42 |
| fungi | for some reason i forgot it was a 5-way cluster | 20:42 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Fixed bug in working directories. https://review.openstack.org/137197 | 20:42 |
| fungi | gah | 20:42 |
| fungi | 6-way | 20:42 |
| krotscheck | Ugh rookie bugs | 20:43 |
| openstackgerrit | Merged openstack-infra/git-review: Prefer git-config over git-review config files https://review.openstack.org/116035 | 20:44 |
| fungi | krotscheck: at least you killed three bugs with one patch | 20:44 |
| krotscheck | fungi: True | 20:45 |
| krotscheck | Still, I should have written tests in the first place. | 20:45 |
| *** e0ne has joined #openstack-infra | 20:45 | |
| *** nfedotov has left #openstack-infra | 20:45 | |
| fungi | well, they're easy fixes to review ;) | 20:46 |
| clarkb | sdague adam_g that should be rebased on my change to enavle nova multihost | 20:46 |
| clarkb | otherwise it will conflict and does no good without that change | 20:46 |
| morganfainberg | krotscheck: I need to talk to you re: storyboard after thanksgiving. | 20:46 |
| morganfainberg | krotscheck: want to know if there are small things I can do to help out. | 20:47 |
| krotscheck | morganfainberg: LOTS | 20:47 |
| morganfainberg | But can't think too much about it till then. | 20:47 |
| *** sputnik13 has quit IRC | 20:47 | |
| *** thedodd has quit IRC | 20:47 | |
| krotscheck | morganfainberg: Ping me when you’re ready :) | 20:47 |
| morganfainberg | krotscheck: ++ | 20:48 |
| fungi | krotscheck: oh, also, i dunno if clarkb has discussed it with you, but i think for the interface drafts we may want to switch to a stroyboard-dev server to cut out an easy source of social engineering | 20:48 |
| fungi | er, storyboard-dev | 20:48 |
| krotscheck | Social engineering? | 20:49 |
| krotscheck | Oh, right | 20:49 |
| *** apevec has joined #openstack-infra | 20:49 | |
| fungi | krotscheck: that's also something we can work on spinning up next week if you have time | 20:50 |
| krotscheck | Let me add a story and put it on my list. | 20:50 |
| jhesketh | Morning | 20:50 |
| krotscheck | Or it could be something morganfainberg can help with :) | 20:50 |
| * jhesketh is back | 20:50 | |
| fungi | jhesketh: welcome back! we assigned all things to you. mattoliverau said it would be fine | 20:51 |
| *** MaxV has quit IRC | 20:51 | |
| jhesketh | fungi: the flaw in your trap there is that mattoliverau is still on vacation ;-) | 20:51 |
| *** nadya_ has quit IRC | 20:51 | |
| jhesketh | however, that does mean we can assign everything to mattoliverau and I can say it's fine | 20:52 |
| fungi | jhesketh: done | 20:52 |
| krotscheck | fungi: Does this capture it all? https://storyboard.openstack.org/#!/story/2000028 | 20:52 |
| *** Ark has joined #openstack-infra | 20:53 | |
| *** zz_jgrimm is now known as jgrimm | 20:53 | |
| *** yfried is now known as yfried|afk | 20:53 | |
| *** Ark is now known as Guest62143 | 20:53 | |
| *** jgrimm is now known as zz_jgrimm | 20:53 | |
| alexpilotti | Hi guys, I have a stackforge patch taht for some reasons does not trigger Jenkins: https://review.openstack.org/#/c/136740/ | 20:54 |
| *** banix has quit IRC | 20:54 | |
| *** shashankhegde has joined #openstack-infra | 20:55 | |
| alexpilotti | Does anybody have an idea on the possible reason? | 20:55 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/infra-specs: Update module issues_url & include storyboard https://review.openstack.org/135452 | 20:56 |
| fungi | krotscheck: yeah, i'm unclear on what the project-config change is for (review-dev projects.yaml maybe?) but the rest makes sense | 20:56 |
| AJaeger_ | alexpilotti: let's recheck it. | 20:56 |
| alexpilotti | Nevermind, looks a recheck did the trick | 20:56 |
| krotscheck | fungi: We can always re-examine that later. | 20:56 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/system-config: Split out elasticsearch module https://review.openstack.org/137155 | 20:56 |
| alexpilotti | AJaeger_: yep, tx | 20:56 |
| AJaeger_ | alexpilotti: we lost some events with jenkins - and the +2 will not start without a recheck. | 20:56 |
| krotscheck | fungi: Also, having a dev instance allows us to selectively disable emails on that box. | 20:56 |
| krotscheck | fungi: Because that would be BAD | 20:57 |
| alexpilotti | AJaeger_: tx, I was actually expecting the +2a to retrigger it, that explains everything | 20:57 |
| fungi | alexpilotti: AJaeger_: looks like the second patchset for that was uploaded while we had zuul offline, thus it wasn't around to se the gerrit event | 20:57 |
| fungi | krotscheck: yep, agreed | 20:57 |
| sdague | alexpilotti: yeh a bunch of events were lost, I did a bulk recheck of the openstack/ that looked lost | 20:58 |
| sdague | but didn't do stackforge | 20:58 |
| *** ildikov has quit IRC | 20:59 | |
| sdague | oh... a thing I discovered | 20:59 |
| sdague | there are 6 open patches on attic - https://review.openstack.org/#/q/project:%255Eopenstack-attic.*+status:open,n,z | 20:59 |
| sdague | sorry 7 | 20:59 |
| alexpilotti | fungi sdague: np tx, didn’t know the +2a was not retriggering it, everything makes sense | 20:59 |
| *** thedodd has joined #openstack-infra | 21:00 | |
| *** julim has quit IRC | 21:01 | |
| mordred | clarkb: have you done a nova flavor-list in rackspace recently? there are some really interesting looking flavors that might be useful for elasticsearch | 21:01 |
| clarkb | I have not | 21:01 |
| mordred | clarkb: for instance: | memory1-240 | 240 GB Memory v1 | 245760 | 0 | 0 | | 32 | 10000.0 | N/A | | 21:01 |
| *** nikhil_k|vacay is now known as nikhil_k | 21:01 | |
| *** yfried|afk is now known as yfried | 21:02 | |
| mordred | clarkb: http://paste.openstack.org/show/138383/ | 21:02 |
| jroll | mordred: /me plugs | onmetal-io1 | OnMetal IO v1 | 131072 | 32 | 3200 | | 40 | 10000.0 | N/A | | 21:02 |
| JayF | Those new memory flavors are diskless (CBS only) | 21:03 |
| mordred | JayF: yah. some of our things might enjoy being diskless | 21:03 |
| JayF | jroll: I think they need cbs | 21:03 |
| JayF | jroll: which onmetal doesn't have | 21:03 |
| jroll | elasticsearch seems like it should have decent IO | 21:03 |
| jroll | ugh | 21:03 |
| jroll | ignore me, then | 21:03 |
| mordred | JayF: we may or may not need cinder for some things | 21:03 |
| JayF | the perfect fit for es is onmetal-memory1 + CBS, but we don't support that yet so :/ | 21:03 |
| jroll | also, onmetal-io1 is missing from mordred's paste O.o | 21:03 |
| *** alexpilotti has quit IRC | 21:04 | |
| *** rlandy has quit IRC | 21:04 | |
| mordred | jroll: I'd love to think about ways in which we can leverage onmetal for some stuff | 21:04 |
| mordred | jroll: but I havent' quite gotten there yet :) | 21:04 |
| *** mmaglana has joined #openstack-infra | 21:04 | |
| jroll | mordred: anything that would benefit from super fast IO is a candidate | 21:04 |
| *** Guest62143 has quit IRC | 21:05 | |
| asselin | jeblair, removed "READ" | 21:05 |
| *** ddieterly has quit IRC | 21:05 | |
| *** guest764 has joined #openstack-infra | 21:05 | |
| *** guest7641 has quit IRC | 21:06 | |
| *** imcsk8 has quit IRC | 21:07 | |
| *** imcsk8 has joined #openstack-infra | 21:07 | |
| *** yfried has quit IRC | 21:07 | |
| *** unicell has quit IRC | 21:07 | |
| clarkb | es on metal would be great | 21:09 |
| fungi | sdague: hah, i guess setting acls to read-only for those repos was insufficient to get them to reject new changes | 21:09 |
| fungi | sdague: oh, wait. those are from before they were shelved. so i think it probably is fine i should just abandon them | 21:10 |
| *** dizquierdo has joined #openstack-infra | 21:10 | |
| *** MaxV has joined #openstack-infra | 21:11 | |
| *** xyang0 has quit IRC | 21:11 | |
| SergeyLukjanov | fungi, jeblair, clarkb, mordred, could you please help a bit - in sahara we're going to rename endpoint name to data-processing (from underscore) for consistency | 21:11 |
| SergeyLukjanov | but it means need to update devstack and tempest in the same time | 21:11 |
| SergeyLukjanov | only ninja merge could help? | 21:12 |
| fungi | SergeyLukjanov: did you discuss that with the qa core reviewers? | 21:12 |
| SergeyLukjanov | nope | 21:12 |
| fungi | SergeyLukjanov: they're the ones who probably need to weigh in on any possible alternatives for simultaneous changes in devstack and tempest | 21:12 |
| mordred | SergeyLukjanov: I think you'll need to figure out how to support both for a bit, no? I mean, otherwise you're going to have upgrade/compat problems in the real world too | 21:12 |
| SergeyLukjanov | mordred, client is compat, but tempest have hardcoded endpoints | 21:13 |
| mordred | ah | 21:13 |
| mtreinish | SergeyLukjanov: shouldn't be | 21:13 |
| mtreinish | it should be a config flag the default might use the underscore, but it pulls the endpoint from keystone catalog | 21:14 |
| jroll | SergeyLukjanov: what about people using the rest api directly, rather than the client? | 21:14 |
| openstackgerrit | Merged openstack-infra/system-config: Pull pip module out. https://review.openstack.org/126205 | 21:14 |
| SergeyLukjanov | jroll, it'll not break anything, just endpoint rename, different openstack installation / distributions could have different endpoint type names | 21:14 |
| fungi | huh, even after strapping on my gerrit superpowers cape, i can't get an abandon button on read-only projects | 21:15 |
| SergeyLukjanov | mtreinish, yeah, the default endpoint type name | 21:15 |
| jroll | SergeyLukjanov: oh, the name in the service catalog, not the actual URL? | 21:15 |
| SergeyLukjanov | jroll, yeah | 21:15 |
| jroll | got it | 21:15 |
| mordred | ah - so the name that you use to look up the url in the service catalog | 21:15 |
| * mordred understands | 21:15 | |
| SergeyLukjanov | I mean to rename data_processing to data-processing | 21:15 |
| SergeyLukjanov | it's just a type | 21:15 |
| *** MaxV has quit IRC | 21:15 | |
| SergeyLukjanov | like a compute and object-storage | 21:15 |
| mtreinish | SergeyLukjanov: ok, then just push a devstack change out which changes the name there and overrides the tempest default | 21:15 |
| jroll | hat could still break people that have written code around that, but probably more minimal impact | 21:15 |
| mtreinish | then we can change the tempest default | 21:15 |
| mtreinish | and then remove the config override | 21:16 |
| *** dkliban is now known as dkliban_biab | 21:16 | |
| SergeyLukjanov | mtreinish, oh, just override the default from the devstack code/ | 21:16 |
| *** AJaeger_ has quit IRC | 21:16 | |
| SergeyLukjanov | mtreinish, yup, got it, thanks for the advise | 21:16 |
| ccowley | \quit | 21:16 |
| *** ccowley has quit IRC | 21:16 | |
| mtreinish | SergeyLukjanov: yeah, devstack writes the tempest.conf so if we're changing the catalog type just have devstack makes sure tempest knows that | 21:17 |
| mtreinish | and we can fix the defaults everywhere after | 21:17 |
| mtreinish | SergeyLukjanov: sure np | 21:17 |
| SergeyLukjanov | mtreinish, any tips on the correct way to re-write default? | 21:17 |
| mtreinish | SergeyLukjanov: I would just add an iniset in lib/tempest, based on the var which you use to add the catalog entry for sahara | 21:18 |
| mtreinish | but, I haven't looked at the service catalog bits in devstack in a while | 21:19 |
| SergeyLukjanov | mtreinish, ok, got it, thanks | 21:20 |
| mordred | jeblair: so - clarkb and fungi don't like my idea of pulling just the IP address from the config drive info and inferring that the netmask is pretty much always the same on rackspace | 21:21 |
| mordred | jeblair: which leads us back to just running the nova-agent on the host | 21:22 |
| jeblair | mordred: netmask is not provided? | 21:22 |
| mordred | jeblair: nope | 21:22 |
| fungi | nor gateways/routes | 21:22 |
| fungi | nor any ip address besides the eth0 interface for that matter, and just the ipv4 address for that even | 21:22 |
| mordred | jeblair: we've also asked rackspace to include more info in their configdrive - but that may not hit tomorrow or anything | 21:23 |
| jeblair | ok cool | 21:23 |
| mordred | so maybe we start with nova-agent, then move to configdrive once all the datas are there | 21:23 |
| jeblair | mordred: i feel like nova-agent may be a reasonable interim step | 21:23 |
| jeblair | yep | 21:23 |
| *** jamielennox is now known as jamielennox|away | 21:23 | |
| mordred | cool. I will work on adding that to some elements | 21:23 |
| mordred | also ... while we're on the subject | 21:23 |
| fungi | yeah, the current state of config drive is "we can probably identify a reachable ipv4 address to set on eth0 and then guess everything else and hope get it right" | 21:24 |
| mordred | the elements I was working on the past couple of days was for infra _Servers_ ... but it seems there is some stuff that would want to be shared between infra servers and nodepool nodes | 21:24 |
| jeblair | as long as we think we can eventually move to configdrive, doesn't seem worth engineering around it. i look forward to helping the rax folks test this out when it's ready :) | 21:24 |
| mordred | jeblair: so it seems to me that those should live in system-config - and we shoudl add system-config/elements to the nodepool elements path | 21:24 |
| jeblair | mordred: network configuration among them, i think :) | 21:25 |
| mordred | jeblair: yah | 21:25 |
| mordred | :) | 21:25 |
| jeblair | mordred: that seems reasonable | 21:25 |
| *** dkranz has quit IRC | 21:25 | |
| jeblair | though | 21:25 |
| mordred | cool. also, I think the puppet element can be split | 21:25 |
| jeblair | that may make it harder for downstreams | 21:25 |
| jeblair | but that may make it easier again :) | 21:25 |
| mordred | into "get puppet installed and bootstrapped on this image" and "run puppet stuffs" | 21:25 |
| *** davideagnello has quit IRC | 21:25 | |
| jeblair | mordred: basically, we want downstreams to be able to use our image elements, right? i mean, cause they will anyway so we may as well not make it hard on us | 21:26 |
| mordred | yes | 21:26 |
| mordred | btw - I've pushed a lot of yesterdays work to the dib repo itself | 21:26 |
| fungi | though some downstreams may turn out to be more interested in just reusing our images instead | 21:27 |
| jeblair | fungi: true | 21:27 |
| mordred | yay for images with my ssh keys on them in the wild! | 21:27 |
| fungi | heh | 21:27 |
| fungi | that's a very good point... we set up our accounts in our nodepool dib images now, and that is probably a negative for reusability of those images | 21:28 |
| *** achanda has quit IRC | 21:28 | |
| *** cpowell has quit IRC | 21:28 | |
| *** bo_sh has joined #openstack-infra | 21:28 | |
| *** jamielennox|away is now known as jamielennox | 21:28 | |
| fungi | though on the other hand, we need to at least configure the images with credentials sufficient to allow our nodepool to ssh in and do post-boot setup | 21:28 |
| jeblair | mordred: i'm sort of uncertain that having project-config rely on system-config elements is a good idea | 21:29 |
| fungi | so either way we're potentially distributing images downstream that we'd have ssh access into | 21:29 |
| fungi | (once we decide to start making them available for download anyway) | 21:29 |
| *** emagana has quit IRC | 21:30 | |
| jeblair | mordred: i think that should take a back seat to dib nodepool for now | 21:30 |
| *** emagana has joined #openstack-infra | 21:31 | |
| mordred | jeblair: yah. well, most all of the issues I need to solve are the same | 21:31 |
| jeblair | and i kind of want to avoid making that more complicated for us and downstreams yet; at least i'd like to actually start using it first | 21:31 |
| jeblair | mordred: k, so maybe we let them be parallel for a bit and see what it looks like? | 21:31 |
| mordred | jeblair: and where the elements will live doesn't hurt me either way in the hacking on it | 21:31 |
| ryanpetrello | is there anything you all need from me in regards to pecan and the py26 support removal? | 21:32 |
| *** emagana has quit IRC | 21:32 | |
| mordred | jeblair: actually, I betcha a nova-agent element is a thing that should live in dib - because it's general useful | 21:32 |
| ryanpetrello | or will py26 jobs just disappear one day? | 21:32 |
| jeblair | yeah | 21:32 |
| *** otter768 has joined #openstack-infra | 21:32 | |
| *** emagana has joined #openstack-infra | 21:32 | |
| *** zz_jgrimm is now known as jgrimm | 21:32 | |
| clarkb | it will just disappear if you dont scream | 21:32 |
| mordred | jeblair: but yeah - I'll focus on getting dib nodepool stuff going and can reorg later if/when it seems necessary | 21:33 |
| *** CI has quit IRC | 21:33 | |
| ryanpetrello | clarkb: that's fine, I'm perfectly equipped to run my own tooling to ensure py26 support in the future | 21:33 |
| ryanpetrello | I do the same for py32 | 21:33 |
| *** MaxV has joined #openstack-infra | 21:33 | |
| jeblair | i'm going to go into a hole and work on gear/zuul. bbl. | 21:33 |
| ryanpetrello | just wanted to make sure you all wouldn't be waiting on me | 21:33 |
| mordred | kk | 21:33 |
| *** thedodd has quit IRC | 21:33 | |
| fungi | jeblair: don't forget to take a torch | 21:34 |
| * jeblair was eaten by a grue | 21:34 | |
| *** dizquierdo has quit IRC | 21:35 | |
| *** kgiusti has left #openstack-infra | 21:37 | |
| fungi | sdague: okay, i got those attic changes abandoned | 21:37 |
| fungi | thanks for the heads up | 21:37 |
| sdague | fungi: no worries | 21:37 |
| *** otter768 has quit IRC | 21:37 | |
| sdague | jeblair: I always felt like that was the appropriate statement everytime you eventually found your way to the urinals in that wc near the infra room | 21:38 |
| fungi | sdague: it was a twisty maze of white tile passages, to be certain | 21:39 |
| mordred | weirdest bathroom ever | 21:39 |
| *** davideagnello has joined #openstack-infra | 21:40 | |
| *** ociuhandu has joined #openstack-infra | 21:43 | |
| fungi | i'm testing an hpcloud-b1 bare-trusty image update now that the changes splitting out the puppet pip module have merged, to ensure it hasn't unexpectedly hosed us | 21:45 |
| *** dkliban_biab is now known as dkliban | 21:45 | |
| sdague | can I get a +A on this - https://review.openstack.org/#/c/137050 so that I can get the nova functional tests seeded without a test gap | 21:46 |
| sdague | nibalizer: and your -1 on that, we still need branch selection | 21:46 |
| *** e0ne has quit IRC | 21:47 | |
| fungi | sdague: it claims to depend on a change of yours in nova which is still wip | 21:47 |
| *** bradjones has joined #openstack-infra | 21:47 | |
| fungi | sdague: was that supposed to say "needed by" instead | 21:47 |
| fungi | ? | 21:47 |
| sdague | oh, sorry, I stuck a change under the one listed | 21:48 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Plugins may now register cron workers. https://review.openstack.org/129609 | 21:48 |
| sdague | so that it would work to go in before it | 21:48 |
| fungi | i see, the "dummy patch" change | 21:48 |
| sdague | yeh | 21:48 |
| *** mudassirlatif has quit IRC | 21:49 | |
| fungi | jogo: you were taken aback when there was insufficient nova core reviewer visibility into an infra change adding voting jobs for nova. care to bless 137050 before it gets approved? | 21:50 |
| *** changbl has quit IRC | 21:51 | |
| * jogo /ooks | 21:52 | |
| sdague | jogo: tl;dr it's the functional test job | 21:52 |
| * krotscheck is pondering his cron change. | 21:52 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: make nova functional tests voting https://review.openstack.org/137050 | 21:52 |
| *** ddieterly has joined #openstack-infra | 21:52 | |
| *** ddieterly has quit IRC | 21:53 | |
| fungi | corrected the commit message accordingly ^ | 21:53 |
| sdague | fungi: ok, coolio | 21:53 |
| fungi | thanks jogo | 21:53 |
| *** Ark has joined #openstack-infra | 21:54 | |
| jogo | fungi: thank you | 21:54 |
| jogo | fungi: although that was proposed by a nova-core which is enough IMHO | 21:54 |
| *** Ark is now known as Guest85398 | 21:54 | |
| fungi | jogo: wfm. didn't know how many nova cores make it sufficient | 21:54 |
| sdague | jogo: so, as soon as that lands, I have the giant move | 21:54 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/136504 | 21:54 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Add ssh key to root user https://review.openstack.org/137176 | 21:54 |
| *** emagana has quit IRC | 21:54 | |
| sdague | jogo: https://review.openstack.org/#/c/136772 - which is the nova side | 21:55 |
| *** emagana has joined #openstack-infra | 21:55 | |
| *** nikhil_k is now known as nikhil_k|vacay | 21:56 | |
| jogo | sdague: nice! | 21:57 |
| jogo | good start | 21:57 |
| *** davideagnello has quit IRC | 21:57 | |
| *** emagana_ has joined #openstack-infra | 21:57 | |
| *** emagana has quit IRC | 21:57 | |
| sdague | yeh, once that is split over I'll start refactoring it into hopefully more sanity, and shorter run time with testresources | 21:57 |
| jogo | cool | 21:58 |
| jogo | I'm still surprised live migration works atall | 21:59 |
| *** davideagnello has joined #openstack-infra | 21:59 | |
| openstackgerrit | Merged openstack-infra/system-config: Removing github module. https://review.openstack.org/126450 | 21:59 |
| openstackgerrit | Merged openstack-infra/system-config: Add in postbuildscript plugin for Jenkins https://review.openstack.org/133179 | 22:02 |
| fungi | jhesketh: ^ | 22:02 |
| *** mudassirlatif has joined #openstack-infra | 22:03 | |
| jhesketh | fungi: oh shiny! | 22:04 |
| * jhesketh has heaps of backlog to work through | 22:04 | |
| *** mudassirlatif has quit IRC | 22:04 | |
| *** mattfarina has quit IRC | 22:04 | |
| fungi | clarkb: minor question related to 137167... why do we have nodepool subscribing to zmq on jenkins.o.o at all anyway? there shouldn't be any nodepool nodes there... | 22:06 |
| *** emagana_ has quit IRC | 22:07 | |
| *** mudassirlatif has joined #openstack-infra | 22:07 | |
| *** Bart___ has joined #openstack-infra | 22:07 | |
| *** Bart has quit IRC | 22:07 | |
| *** jungleboyj has quit IRC | 22:07 | |
| clarkb | I think thats old from befoee many jenkinses | 22:07 |
| fungi | ahh, yeah, perhaps. that part of the timeline is a little fuzzy for me now. anyway, i suppose we can clean that up | 22:09 |
| fungi | but in another patch | 22:09 |
| *** ryanpetrello has quit IRC | 22:10 | |
| openstackgerrit | Adam Gandelman proposed openstack-infra/devstack-gate: Enable nova-api-metadata for multi host nova-network https://review.openstack.org/136179 | 22:14 |
| adam_g | clarkb, sdague, sorry had to run out. there ya go ^ | 22:14 |
| sdague | adam_g: oh... hey, you lost n-net | 22:15 |
| adam_g | oh, hey | 22:15 |
| adam_g | thanks | 22:15 |
| sdague | I must have missed it the first time | 22:15 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/devstack-gate: Enable nova-api-metadata for multi host nova-network https://review.openstack.org/136179 | 22:16 |
| sdague | +2 | 22:16 |
| *** dimsum__ has quit IRC | 22:16 | |
| *** dimsum__ has joined #openstack-infra | 22:17 | |
| *** melwitt has quit IRC | 22:17 | |
| *** melwitt has joined #openstack-infra | 22:18 | |
| *** JayJ has joined #openstack-infra | 22:18 | |
| openstackgerrit | Merged openstack-infra/nodepool: Update snapshots prior to dib image builts https://review.openstack.org/137110 | 22:19 |
| *** emagana has joined #openstack-infra | 22:20 | |
| *** alexpilotti has joined #openstack-infra | 22:20 | |
| *** emagana has quit IRC | 22:20 | |
| *** yamamoto has joined #openstack-infra | 22:21 | |
| *** dimsum__ has quit IRC | 22:21 | |
| *** Hefeweizen has joined #openstack-infra | 22:22 | |
| *** emagana has joined #openstack-infra | 22:22 | |
| sdague | fungi: so... what's the puppet apply jobs on project-config doing? | 22:23 |
| jeblair | clarkb, fungi: i have a local zuul watching prod with one worker and 2 mergers and i haven't seen an issue yet :/ | 22:23 |
| sdague | because I didn't think there was any puppet really left in there | 22:23 |
| openstackgerrit | Merged openstack-infra/project-config: make nova functional tests voting https://review.openstack.org/137050 | 22:23 |
| fungi | sdague: our puppet manifests use some of the files from that repository | 22:24 |
| sdague | fungi: ah, ok | 22:24 |
| fungi | sdague: so in theory a change, say, removing one of them might show up as a failure | 22:24 |
| sdague | fungi: well it seems to be mostly erroring | 22:24 |
| sdague | even though it's passing | 22:24 |
| sdague | https://jenkins06.openstack.org/job/gate-infra-puppet-apply-centos6/398/console | 22:24 |
| jeblair | that's a new feature of puppet 3! | 22:24 |
| fungi | sdague: yeah, same on system-config. it's a bit of a mess right now | 22:25 |
| sdague | ok | 22:25 |
| sdague | well it was weird enough I thought I'd bring it up | 22:25 |
| fungi | apparently some errors aren't errors | 22:25 |
| fungi | or something | 22:25 |
| *** yamamoto has quit IRC | 22:25 | |
| fungi | so sayeth puppet 3 | 22:26 |
| sdague | so say us all | 22:26 |
| fungi | also it's crazy full of ansi escapes | 22:26 |
| sdague | ok, time for me to call it a day and do family things. Enjoy folks | 22:26 |
| jeblair | sdague: 'night | 22:26 |
| fungi | have a good evening sdague | 22:26 |
| fungi | i will be vanishing soonish once i get an opportunity to sync back up with nibalizer | 22:27 |
| *** MaxV has quit IRC | 22:27 | |
| fungi | so far no catastrophic failures with those modules split out, and the image update i tested didn't complain either | 22:27 |
| *** achanda has joined #openstack-infra | 22:28 | |
| fungi | also i won't be around much from wednesday afternoon through the weekend. cooking, driving, visiting, the usual holiday drill | 22:29 |
| *** mattfarina has joined #openstack-infra | 22:35 | |
| *** gokrokve has quit IRC | 22:36 | |
| nibalizer | fungi: hi | 22:37 |
| *** rediskin has left #openstack-infra | 22:38 | |
| *** achanda has quit IRC | 22:39 | |
| fungi | nibalizer: so far so good | 22:39 |
| fungi | anything you want to check? | 22:39 |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Add job that tests rally and tempest integration https://review.openstack.org/137232 | 22:39 |
| nibalizer | fungi: uhhhh | 22:40 |
| nibalizer | i mean if pip was busted... we would know :P | 22:40 |
| *** gokrokve has joined #openstack-infra | 22:40 | |
| fungi | nibalizer: yep, that's why i tried an image build | 22:41 |
| *** e0ne has joined #openstack-infra | 22:42 | |
| *** e0ne has quit IRC | 22:42 | |
| clarkb | jeblair maybe it is triggered by accessing the admin commands? | 22:44 |
| *** signed8b_ has quit IRC | 22:45 | |
| clarkb | jeblair which zuul + merger + worker wont do | 22:45 |
| jeblair | clarkb: i keep typing 'status' in a telnet session | 22:45 |
| *** mattfarina has quit IRC | 22:45 | |
| jeblair | clarkb: that does leave 'cancel job' as untested though | 22:45 |
| jeblair | clarkb: i'm digging through the zuul logs now looking for a clue | 22:46 |
| mordred | fungi, jeblair: I thought we were doing the good stuff with puppet to get proper errors on errors | 22:46 |
| *** Guest85398 has quit IRC | 22:46 | |
| mordred | do we need to pass "puppet apply --no-seriously-please-die-on-errors-kthxbai" or something/ | 22:46 |
| *** tonytan4ever has quit IRC | 22:46 | |
| *** mbacchi has quit IRC | 22:46 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Set modulepath to the first element of puppet's modulepath. https://review.openstack.org/137233 | 22:46 |
| nibalizer | sdague: oh okay my bad | 22:46 |
| fungi | apparently there are issues with ask.openstack.org and evgeny is working on them | 22:46 |
| fungi | not related to anything we did though | 22:47 |
| nibalizer | i guess i don't know what the branch selection does | 22:47 |
| *** MaxV has joined #openstack-infra | 22:47 | |
| openstackgerrit | Dolph Mathews proposed openstack-infra/elastic-recheck: add CLI command for executing queries https://review.openstack.org/137234 | 22:47 |
| fungi | mordred: no i think it's that we're running them in noop so it complains loudly about all the things it's not doing because they depend on resources it won't update in --noop mode | 22:48 |
| mordred | oh. right | 22:48 |
| mordred | that | 22:48 |
| jeblair | oh, i thought it was more than that | 22:48 |
| fungi | mordred: like complaining that it didn't do stuff related to the apache module because it didn't install the package for apache or whatever | 22:48 |
| *** dims_ has joined #openstack-infra | 22:49 | |
| fungi | there may be more, that's just the main class of errors i end up wading through trying to look at the puppet apply logs usually | 22:49 |
| clarkb | execs trip noop mode a lot | 22:49 |
| mordred | right. because it didn't actually make the change to the system, since you told it not to, so things that depend on that thing having been created complain because their resource isn't there | 22:49 |
| clarkb | since execs are black boxes to puppet | 22:49 |
| jeblair | looking at http://logs.openstack.org/50/137050/2/gate/gate-infra-puppet-apply-centos6/4ca633a/console.html | 22:50 |
| jeblair | there's a lot of deprecation errors | 22:50 |
| *** MaxV_ has joined #openstack-infra | 22:50 | |
| jeblair | then there's: | 22:50 |
| jeblair | 2014-11-25 22:16:46.510 | [1;31mError: /Stage[main]/Cgit::Selinux/Selboolean[httpd_enable_cgi]: Could not evaluate: Execution of '/usr/sbin/getsebool httpd_enable_cgi' returned 1: /usr/sbin/getsebool: SELinux is disabled[0m | 22:50 |
| jeblair | i dunno if that's a thing... | 22:51 |
| jeblair | 2014-11-25 22:16:48.247 | mError: /Stage[main]/Haproxy/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]: Could not evaluate: Could not retrieve information from environment production source(s) file:/var/lib/puppet/concat/_etc_haproxy_haproxy.cfg/fragments.concat.outm | 22:51 |
| *** MaxV has quit IRC | 22:51 | |
| jeblair | don't know why that's an error | 22:51 |
| nibalizer | concat fragments fail really hard on noop | 22:51 |
| *** MarkAtwood has quit IRC | 22:52 | |
| * jeblair looks askance at puppet | 22:52 | |
| nibalizer | since the concat module puts files in a temp directory before concating them together | 22:52 |
| jeblair | that seems like the sort of "noop" that you could "op" :) | 22:52 |
| *** dustins has quit IRC | 22:52 | |
| *** MarkAtwood has joined #openstack-infra | 22:52 | |
| nibalizer | ya this module https://github.com/richardc/puppet-datacat is the future, replacing concat, with such features as working in noop contexts | 22:53 |
| *** mattfarina has joined #openstack-infra | 22:53 | |
| fungi | i guess "needs to create files somewhere" is one of the places it draws the line on what is a non-operation | 22:53 |
| jeblair | i mean, maybe my lisp is showing, but i don't think string concatenation should have side effects :) | 22:53 |
| jeblair | nibalizer: working is an excellent idea for a new feature! :) | 22:54 |
| fungi | release notes: "now works!" | 22:54 |
| * mordred looks at jeblair's lisp | 22:54 | |
| nibalizer | yea... concat is an example of a hack that was so useful but gross but now is in use pretty much everywhre | 22:54 |
| mordred | s/concat/puppet/ | 22:55 |
| * mordred ducks | 22:55 | |
| pleia2 | hah | 22:55 |
| *** ayoung is now known as ayoung-dad-mode | 22:56 | |
| *** emagana has quit IRC | 22:57 | |
| *** pradk has quit IRC | 22:57 | |
| *** stevemar has quit IRC | 23:00 | |
| *** teran has quit IRC | 23:00 | |
| *** enikanorov_ has quit IRC | 23:01 | |
| *** enikanorov_ has joined #openstack-infra | 23:01 | |
| *** andreykurilin_ has quit IRC | 23:01 | |
| *** stevemar has joined #openstack-infra | 23:01 | |
| *** teran has joined #openstack-infra | 23:01 | |
| openstackgerrit | Dolph Mathews proposed openstack-infra/elastic-recheck: add CLI command for executing elastic-recheck queries https://review.openstack.org/137234 | 23:01 |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Add job that tests rally and tempest integration https://review.openstack.org/137232 | 23:01 |
| *** banix has joined #openstack-infra | 23:02 | |
| dims_ | mordred: ping | 23:02 |
| *** shashankhegde_ has joined #openstack-infra | 23:03 | |
| *** achanda has joined #openstack-infra | 23:03 | |
| *** jklare has quit IRC | 23:03 | |
| *** emagana has joined #openstack-infra | 23:04 | |
| dims_ | mordred: i see a mox3 on your github, Anyone we know familiar with internals? got a nasty bug - https://bugs.launchpad.net/openstack-ci/+bug/1396144 | 23:04 |
| uvirtbot | Launchpad bug 1396144 in openstack-ci "mox3 fails for context manager based mock testing" [Undecided,New] | 23:04 |
| openstackgerrit | Dolph Mathews proposed openstack-infra/elastic-recheck: add CLI command for executing elastic-recheck queries https://review.openstack.org/137234 | 23:05 |
| *** jklare has joined #openstack-infra | 23:05 | |
| *** shashankhegde has quit IRC | 23:06 | |
| *** shashankhegde_ is now known as shashankhegde | 23:06 | |
| *** sarob has quit IRC | 23:07 | |
| *** sarob has joined #openstack-infra | 23:07 | |
| *** sarob has quit IRC | 23:07 | |
| *** mattfarina has quit IRC | 23:08 | |
| clarkb | jeblair: anything else I can do to help? | 23:14 |
| *** markmcclain1 has joined #openstack-infra | 23:15 | |
| *** markmcclain has quit IRC | 23:15 | |
| jeblair | clarkb: not atm; i'm piecing together what happened from the logs. i'm hoping i'll find something interesting -- i have found where zuul took 8 minutes to complete an enqueue event; trying to figure out what it was doing during that time | 23:15 |
| clarkb | cool going to start putting together the third party self serve changes then | 23:15 |
| *** gokrokve has quit IRC | 23:16 | |
| mordred | jeblair: maybe it took a break to watch TV | 23:16 |
| mordred | dims_: I have very little knowledge there - I think zul was doing some stuff with it? | 23:17 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Create puppet-forge credentials on the pypi slave https://review.openstack.org/134835 | 23:17 |
| dims_ | mordred: ack, will ask | 23:17 |
| dims_ | mordred: thanks | 23:17 |
| *** achanda has quit IRC | 23:17 | |
| clarkb | dims_: also aren't new test cases supposed to use mock? | 23:17 |
| *** dkliban is now known as dkliban_afk | 23:17 | |
| jeblair | mordred: the real housewives of gozeria? | 23:17 |
| dims_ | clarkb: true. takes time...i hit it when i was trying to use oslotest (uses mox3) with nova | 23:18 |
| clarkb | why would oslotest use the thing we want ot deprecate | 23:18 |
| * clarkb decides it isn't important and goes back to third party things | 23:19 | |
| dims_ | clarkb: hehe, phased move if you really want to see (which i don't think you do :) - https://review.openstack.org/#/c/135944/ | 23:19 |
| clarkb | right but oslotest is new... you don't phase move the new thing you just use new things | 23:20 |
| clarkb | why do 2 migrations when you can do one? | 23:20 |
| *** sputnik13 has joined #openstack-infra | 23:20 | |
| *** jungleboyj has joined #openstack-infra | 23:21 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Use storyboard for all infra puppet modules https://review.openstack.org/136594 | 23:21 |
| mordred | nibalizer: zomg. are we actually getting automated uploads to puppet-forge? | 23:22 |
| dims_ | clarkb: i hear ya. we can do this on oslo channel when dhellmann is around :) | 23:23 |
| *** achanda has joined #openstack-infra | 23:23 | |
| nibalizer | mordred: yup | 23:25 |
| nibalizer | a rooter has to make the new module first time i think | 23:25 |
| mordred | that's fine - we have to do a similar thing for PyPI | 23:25 |
| *** HACKING-FACEBOOK has joined #openstack-infra | 23:26 | |
| *** teran_ has joined #openstack-infra | 23:27 | |
| *** camunoz has joined #openstack-infra | 23:29 | |
| *** erikwilson has quit IRC | 23:29 | |
| *** markmcclain1 has quit IRC | 23:29 | |
| *** markmcclain has joined #openstack-infra | 23:29 | |
| *** gokrokve has joined #openstack-infra | 23:30 | |
| *** markmcclain has quit IRC | 23:30 | |
| *** teran has quit IRC | 23:30 | |
| *** dangers is now known as dangers_away | 23:31 | |
| *** apevec has left #openstack-infra | 23:31 | |
| *** otter768 has joined #openstack-infra | 23:33 | |
| *** camunoz has quit IRC | 23:34 | |
| *** mjturek has quit IRC | 23:36 | |
| *** arosen has left #openstack-infra | 23:37 | |
| *** otter768 has quit IRC | 23:38 | |
| jeblair | clarkb: okay, i have a lead -- during those 8 minutes, it wanted to launch jobs for an icehouse change and none of the devstack-precise jobs were registered. each time it wanted to launch one of those, it took ~90 seconds to determine they were not registered (which it did one at a time [there is a cache, but the ttl on the cache is set at the beginning of the query and is 5 seconds, so a 90 second query will always bust it]) | 23:38 |
| *** denis_makogon has quit IRC | 23:38 | |
| jeblair | that happened at least 5 times, so that accounts for almost all of the time it was hung | 23:38 |
| *** dmakogon_ has joined #openstack-infra | 23:38 | |
| jeblair | (also, one of them timed out) | 23:39 |
| jeblair | so it's almost certainly related to the status admin command... possibly in using it on the same connection that's also being used for job submissions; i don't think that i have tested that | 23:40 |
| asselin | anyone seen this issue with jjb? pkg_resources.VersionConflict: (python-jenkins 0.2 (/usr/lib/python2.7/dist-packages), Requirement.parse('python-jenkins>=0.3.4')) | 23:41 |
| clarkb | jeblair: oh interesting | 23:42 |
| *** shashankhegde has quit IRC | 23:42 | |
| jeblair | also, btw, gearman has 12484 jobs registered | 23:43 |
| sweston | asselin: I recall having some problems with it on install ... it ended up being a python path problem/conflict with the os packages and pip | 23:44 |
| asselin | sweston, how did you resolve? | 23:44 |
| asselin | reinstall? | 23:44 |
| *** hashar has joined #openstack-infra | 23:45 | |
| sweston | asselin: uninstalled the os packages, re-installed the pip requirements, updated jjb from source, and compiled it, then installed | 23:45 |
| asselin | sweston, which os packages? | 23:46 |
| *** oomichi has joined #openstack-infra | 23:46 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/136504 | 23:47 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Set up ssh_known_host based on hostname https://review.openstack.org/136596 | 23:47 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Setup /etc/hosts for aiopcpu https://review.openstack.org/136158 | 23:47 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Add ssh key to root user https://review.openstack.org/137176 | 23:47 |
| sweston | asselin: it was a long time ago, I am having trouble remembering | 23:47 |
| sweston | asselin: let me check my puppet manifest git logs | 23:47 |
| asselin | sweston, pip freeze shows python-jenkins==0.2, but fails during the uninstall | 23:48 |
| *** camunoz has joined #openstack-infra | 23:48 | |
| *** emagana has quit IRC | 23:48 | |
| *** gyee_ has quit IRC | 23:49 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Document new third party self service accounts https://review.openstack.org/137240 | 23:49 |
| jeblair | jogo: i'm not paying close attention to what you're doing, but fyi in case this helps -- i believe we have sshd configured to disallow logins as the root user by default | 23:49 |
| clarkb | jeblair: krtaylor fungi anteaya ^ | 23:50 |
| clarkb | change to create ci-sandbox is next | 23:50 |
| jogo | jeblair: that is probably a good and thing and not a problem here | 23:50 |
| clarkb | jeblair: jogo ya we do but jogo should be setting up root to other accounts so it should be fine | 23:50 |
| clarkb | ya | 23:50 |
| jogo | jeblair: libvirt(which runs as root) needs to be able to login as stack | 23:50 |
| *** melwitt has quit IRC | 23:50 | |
| jeblair | ok cool :) | 23:51 |
| *** HACKING-FACEBOOK has quit IRC | 23:52 | |
| jogo | jeblair: now that I look at that commit message title, it isn't very clear | 23:52 |
| sweston | asselin: is the python-pip class installed? | 23:53 |
| asselin | sweston, no | 23:53 |
| sweston | asselin: python-yaml? | 23:54 |
| fungi | asselin: sweston: we *just* merged a change to split the puppet pip module out into its own project separate from the system-config repo | 23:54 |
| *** davideagnello has quit IRC | 23:54 | |
| fungi | asselin: sweston: not sure if you're bumping into that at all | 23:54 |
| asselin | sweston, no | 23:55 |
| asselin | fungi, ok, maybe...? | 23:55 |
| sweston | fungi: good to know, thank you | 23:55 |
| *** davideagnello has joined #openstack-infra | 23:56 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Add a ci-sandbox project. https://review.openstack.org/137242 | 23:56 |
| clarkb | boom and that shouldn'tmerge until we fix project creation | 23:56 |
| clarkb | fungi: have a link to your jeepyb change handy? I will review that next | 23:56 |
| *** Ark has joined #openstack-infra | 23:56 | |
| *** Ark is now known as Guest8043 | 23:57 | |
| fungi | clarkb: oh, right that's something i wanted to try out. i believe my jeepyb change is still hand-applied on review.o.o and rudimentary tests were not failing like we had been seeing | 23:57 |
| clarkb | fungi: woot | 23:57 |
| fungi | clarkb: https://review.openstack.org/136079 | 23:57 |
| fungi | clarkb: apparently mysqldb.Connection.ping() is magicsauce | 23:58 |
| fungi | especially if you pass True as its one poorly-documented positional parameter | 23:58 |
| fungi | since that tells it to automatically reconnect if it detects that it's not connected | 23:59 |
| sweston | asselin: do a git checkout of jjb from github, and then do a pip install --upgrade -r test-requirements.txt | 23:59 |
| clarkb | fungi: looks like magic | 23:59 |
| fungi | clarkb: unfortunately i found no equivalent for psycopg2 so left that unimplemented | 23:59 |
| *** HACKING-FACEBOOK has joined #openstack-infra | 23:59 | |
| asselin | sweston, ok, seems it was a local thing. I did this and it worked: sudo apt-get remove python-jenkins | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!