| *** smatzek has joined #openstack-powervm | 00:42 | |
| *** smatzek_ has joined #openstack-powervm | 00:44 | |
| *** tjakobs has joined #openstack-powervm | 00:45 | |
| *** seroyer has joined #openstack-powervm | 00:45 | |
| *** smatzek has quit IRC | 00:48 | |
| *** smatzek_ has quit IRC | 01:54 | |
| *** thorst has quit IRC | 01:55 | |
| *** thorst has joined #openstack-powervm | 01:56 | |
| *** thorst has quit IRC | 02:04 | |
| *** seroyer has quit IRC | 02:36 | |
| *** thorst has joined #openstack-powervm | 03:02 | |
| *** thorst has quit IRC | 03:10 | |
| *** thorst has joined #openstack-powervm | 04:09 | |
| *** thorst has quit IRC | 04:15 | |
| *** thorst has joined #openstack-powervm | 05:12 | |
| *** thorst has quit IRC | 05:20 | |
| *** thorst has joined #openstack-powervm | 06:19 | |
| *** thorst has quit IRC | 06:25 | |
| *** k0da has joined #openstack-powervm | 07:21 | |
| *** thorst has joined #openstack-powervm | 07:23 | |
| *** k0da has quit IRC | 07:29 | |
| *** thorst has quit IRC | 07:30 | |
| *** openstackgerrit has quit IRC | 07:48 | |
| *** openstackgerrit has joined #openstack-powervm | 07:48 | |
| *** thorst has joined #openstack-powervm | 08:27 | |
| *** apearson has joined #openstack-powervm | 08:29 | |
| *** k0da has joined #openstack-powervm | 08:34 | |
| *** thorst has quit IRC | 08:35 | |
| *** apearson has quit IRC | 08:45 | |
| *** thorst has joined #openstack-powervm | 09:32 | |
| *** thorst has quit IRC | 09:39 | |
| *** apearson has joined #openstack-powervm | 09:42 | |
| *** apearson has quit IRC | 10:27 | |
| *** apearson has joined #openstack-powervm | 10:28 | |
| *** apearson has quit IRC | 10:28 | |
| *** apearson has joined #openstack-powervm | 10:28 | |
| *** apearson has quit IRC | 10:28 | |
| *** apearson has joined #openstack-powervm | 10:29 | |
| *** apearson has quit IRC | 10:29 | |
| *** apearson has joined #openstack-powervm | 10:30 | |
| *** apearson has quit IRC | 10:30 | |
| *** apearson has joined #openstack-powervm | 10:31 | |
| *** apearson has quit IRC | 10:31 | |
| *** apearson has joined #openstack-powervm | 10:31 | |
| *** apearson has quit IRC | 10:32 | |
| *** apearson has joined #openstack-powervm | 10:32 | |
| *** apearson has quit IRC | 10:33 | |
| *** apearson has joined #openstack-powervm | 10:33 | |
| *** apearson has quit IRC | 10:33 | |
| *** apearson has joined #openstack-powervm | 10:34 | |
| *** apearson has joined #openstack-powervm | 10:34 | |
| *** apearson has quit IRC | 10:35 | |
| *** thorst has joined #openstack-powervm | 10:37 | |
| *** apearson has joined #openstack-powervm | 10:39 | |
| *** thorst has quit IRC | 10:45 | |
| *** apearson has quit IRC | 10:45 | |
| *** apearson has joined #openstack-powervm | 10:55 | |
| *** apearson has joined #openstack-powervm | 11:02 | |
| *** seroyer has joined #openstack-powervm | 11:23 | |
| *** smatzek_ has joined #openstack-powervm | 11:29 | |
| *** thorst has joined #openstack-powervm | 11:49 | |
| *** wangqwsh has joined #openstack-powervm | 11:51 | |
| *** edmondsw has joined #openstack-powervm | 12:21 | |
| *** apearson has quit IRC | 12:24 | |
| *** apearson has joined #openstack-powervm | 12:25 | |
| *** apearson has quit IRC | 12:25 | |
| *** apearson has joined #openstack-powervm | 12:25 | |
| *** apearson has quit IRC | 12:26 | |
| *** apearson has joined #openstack-powervm | 12:52 | |
| *** wangqwsh has quit IRC | 13:06 | |
| *** wangqwsh_ has joined #openstack-powervm | 13:06 | |
| *** wangqwsh_ is now known as wangqwsh | 13:06 | |
| *** mdrabe has joined #openstack-powervm | 13:20 | |
| *** seroyer has quit IRC | 13:24 | |
| adreznec | Jeez thorst, an hour long scrum. Must have a lot of status to report | 13:31 |
|---|---|---|
| thorst | adreznec: I just love taking your time | 13:31 |
| thorst | efried esberglu wangqwsh: around for virtual scrum? | 13:31 |
| efried | o/ | 13:32 |
| * adreznec runs and hides | 13:32 | |
| wangqwsh | :) | 13:32 |
| thorst | o yes, lets make this semi-official | 13:32 |
| thorst | \o | 13:32 |
| adreznec | I wonder if the meeting controls work outside the meeting channels... | 13:32 |
| thorst | heh | 13:33 |
| efried | Do we need to re-summarize what broke the CI? | 13:33 |
| thorst | nah, I know | 13:33 |
| thorst | I borked it | 13:33 |
| adreznec | #startmeeting CI Scrum | 13:33 |
| openstack | Meeting started Wed Nov 2 13:33:44 2016 UTC and is due to finish in 60 minutes. The chair is adreznec. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:33 |
| efried | With good intentions, of course. | 13:33 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:33 |
| openstack | The meeting name has been set to 'ci_scrum' | 13:33 |
| thorst | whoa. | 13:33 |
| adreznec | #topic Overview of current status | 13:33 |
| efried | noyce. | 13:34 |
| adreznec | Hmm | 13:34 |
| adreznec | Anyway | 13:34 |
| adreznec | I'll turn the floor over to thorst | 13:34 |
| thorst | OK - so the CI runs on Tempest VMs. They run remote to the NovaLink. So the fact that I made them with a file path is what was broken | 13:34 |
| *** tblakes has joined #openstack-powervm | 13:35 | |
| thorst | unfortunately we didn't catch cause CI was down, and now we can't get the CI running until we get a fix in | 13:35 |
| *** smatzek_ has quit IRC | 13:35 | |
| thorst | here are my thoughts... | 13:35 |
| thorst | we could add a config option to go back to old way. I think for the image size its fine. We needed this new path for when people like kriskend and seroyer deploys twenty 20-gig images | 13:36 |
| thorst | but our CI is doing very small images, and most are linked clones (though not all) | 13:36 |
| efried | So a secret config option or a public one? | 13:36 |
| efried | (I'm down with the idea, btw. | 13:36 |
| efried | ) | 13:36 |
| adreznec | We're open source, hard for it to be a secret config option :P | 13:36 |
| thorst | efried: I think it should be public | 13:36 |
| esberglu | thorst: We are moving to large images once we add in the OSA stuff | 13:37 |
| thorst | and I think that we need it public for our new in tree driver too. | 13:37 |
| thorst | esberglu: larger for the under cloud, but not the VMs that tempest deploys | 13:37 |
| efried | So we're talking about reinstating the IterableToFileAdapter and all that | 13:37 |
| thorst | the under cloud would actually use the new path...because it is running on the novalink | 13:37 |
| esberglu | No larger for the vms. OSA nodes need 50G free space | 13:37 |
| thorst | efried: well, I replaced it with a ChunkyFileIter in an old path... | 13:37 |
| efried | in an old patch on the same change set? | 13:38 |
| adreznec | esberglu: yeah, but not for the guest VMs deployed as part of the CI run | 13:38 |
| esberglu | NM i'm dumb | 13:38 |
| thorst | esberglu: run...but that's all VMs that the under cloud provisions, not what the Tempest running in the VM provisions | 13:38 |
| esberglu | Yeah I was confused by what you meant | 13:38 |
| thorst | run -> right | 13:38 |
| esberglu | I get it now | 13:38 |
| efried | Nope. In an abandoned change set? | 13:38 |
| * adreznec can't help but laugh every time he sees ChunkFileIter | 13:39 | |
| adreznec | *Chunky | 13:39 |
| thorst | efried: yeah, its actually a previous version of what merged | 13:39 |
| thorst | but what I'm not sure about is what to call this opt | 13:39 |
| thorst | "compat_upload_mode" or something | 13:39 |
| adreznec | Anyway. So basically this configopt would provide a way to revert back to that not-quite-as-old behavior | 13:39 |
| thorst | right. | 13:40 |
| adreznec | I guess it depends on whether we want this configopt to be specific for toggling to the old upload function | 13:40 |
| adreznec | Or if we want it to be a development-only configopt for allowing the driver to work remotely in the future | 13:40 |
| thorst | hmmm | 13:40 |
| thorst | that's kinda neat. | 13:40 |
| adreznec | Where we could have other things this configopt "fixes" | 13:40 |
| thorst | but whatever it is, it will need to persist into the future in-tree driver too | 13:40 |
| adreznec | Something like "remote_driver_dev" | 13:41 |
| thorst | adreznec: slippery slope...next thing you know we'll have config opts for remote ips | 13:41 |
| thorst | and look like a remote driver. | 13:41 |
| efried | Yeah, that's what I meant by a "private" option - basically either undocumented, or documented as "don't use this unless you're us" | 13:41 |
| adreznec | thorst: yeah | 13:41 |
| adreznec | Idk | 13:42 |
| thorst | efried adreznec: another idea... | 13:42 |
| thorst | is there a way we could ... hide this somehow? | 13:42 |
| thorst | put a pypowervm patch in that says 'no, you don't get to do that' | 13:42 |
| efried | I was looking at it, and I couldn't see a way to do it easily. | 13:42 |
| thorst | we'd have to read in from the file path...and then pipe that into the REST layer | 13:42 |
| efried | Or we put the IterableToFileAdapter (or the artist formerly known as) into pypowervm. | 13:43 |
| thorst | efried: well, really a FileToIterableAdapter | 13:43 |
| efried | And then pypowervm detects remote and overrides the specified option, ignoring the function. | 13:43 |
| thorst | well, you need the function...cause that's the only interlock into glance | 13:43 |
| thorst | I kinda prefer that cause we need to patch the pypowervm in OSA already... | 13:44 |
| thorst | and in our devstack... | 13:44 |
| thorst | its trickier but I think it prevents slipperiness | 13:44 |
| efried | oh, you want this as part of local2remote patch, not a permanent fixture in pypowervm? | 13:44 |
| efried | I guess that makes sense. | 13:44 |
| thorst | efried: right. | 13:44 |
| adreznec | Yeah | 13:44 |
| adreznec | Basically this would be another library tweak for CI | 13:45 |
| thorst | those were only two ways I could think of it... Config opt or hide it in pypowervm local2remote. I prefer the second cause it will just work with everything and limit it to our CI | 13:45 |
| efried | btw, does local2remote become moot if we decide to support remote pypowervm officially? | 13:45 |
| thorst | efried: nah | 13:45 |
| *** seroyer has joined #openstack-powervm | 13:45 | |
| thorst | because its really a question of whether or not we support nova-powervm remotely | 13:46 |
| thorst | which we don't, except for CI (to allow scale) | 13:46 |
| adreznec | Right | 13:46 |
| efried | Okay, so presumably... | 13:46 |
| efried | #action thorst to propose the local2remote patch to make this work | 13:46 |
| efried | ? | 13:46 |
| adreznec | And I can't see a compelling reason to want to outside development... kind of defeats the purpose of the driver | 13:47 |
| thorst | can we swap the owner to efried? | 13:47 |
| thorst | cause I want a different action later in meeting :-D | 13:47 |
| thorst | (I want to spend time updating the nova-powervm proposal rst) | 13:47 |
| efried | You'd be quicker at it, but if you show me this interim thing you mentioned (which I have not been able to find), I'll take it on. | 13:47 |
| thorst | OK - we'll work it together. | 13:48 |
| thorst | apearson would be the quickest, but he's decided to be in Europe and be afk | 13:48 |
| efried | #agreed thorst & efried to work the local2remote patch | 13:48 |
| adreznec | #action efried and thorst work together to bring harmony to the CI system | 13:48 |
| adreznec | Ok, so that would get applied as part of our existing patch path then | 13:48 |
| efried | yuh | 13:48 |
| thorst | so that brings to second point | 13:48 |
| adreznec | No new code required from esberglu so far | 13:48 |
| efried | yuh | 13:49 |
| thorst | how are we going to do our CI for proper nova | 13:49 |
| thorst | I think this patch solves one aspect of it | 13:49 |
| adreznec | #topic In-tree driver CI discussion | 13:49 |
| *** tblakes has quit IRC | 13:49 | |
| thorst | but the other was that I was planning on localdisk for in-tree. I think we need SSP at a minimum for CI 'harmony' | 13:49 |
| adreznec | Yeah, a bit of a wrinkle | 13:49 |
| apearson | @thorst - so I don't have to read through a ton (yeah, I'm lazy), is there a short summary I can look at to help? | 13:50 |
| thorst | but...I think its not that awful? We could lead with SSP... | 13:50 |
| thorst | apearson: don't worry about it - I was just poking on how you're supposedly afk | 13:50 |
| apearson | oh fine - I know when I'm not wanted... | 13:50 |
| adreznec | #link https://review.openstack.org/#/c/381772/ <-- Driver blueprint | 13:51 |
| adreznec | thorst: would we really lead with SSP only? | 13:51 |
| adreznec | I think we'd also want localdisk in the mix | 13:51 |
| thorst | adreznec: I think we throw both in | 13:52 |
| thorst | see what sticks | 13:52 |
| adreznec | That would allow us to run the most basic case of the driver | 13:52 |
| adreznec | Ok | 13:52 |
| thorst | but we probably develop SSP first. | 13:52 |
| efried | There's a matter of staging, in any case. We would probably want to ... yeah. | 13:52 |
| thorst | so that we get CI running ASAP | 13:52 |
| adreznec | #agreed on including both localdisk and SSP in the first pass of the in-tree driver | 13:53 |
| thorst | alright...amazing. We have a plan on those. | 13:53 |
| adreznec | Yep | 13:53 |
| thorst | #action thorst to update powervm blueprint | 13:53 |
| thorst | does that actually do anything? | 13:53 |
| adreznec | Are there any other things we need to decide in the blueprint? | 13:54 |
| adreznec | It should in the meeting minutes | 13:54 |
| thorst | not sure...probably, but I haven't looked. | 13:54 |
| efried | I think it makes stuff appear in a different font in the meeting minutes. | 13:54 |
| adreznec | (if we get meeting minutes) | 13:54 |
| thorst | well, haven't looked in depth. E-mail hell. | 13:54 |
| adreznec | Looking at comments now | 13:54 |
| adreznec | First one up was about an overview of old powervm vs powervc vs powerkvm vs new powervm driver | 13:54 |
| thorst | yeah, I can put that stuff in. None of this is really heart burn. | 13:55 |
| adreznec | I think a couple lines on that is fine | 13:55 |
| efried | I responded to a few of the comments with links to the WIP change sets. | 13:55 |
| thorst | and some we already discussed in unconference...so I think we're really OK here. | 13:56 |
| adreznec | Yeah | 13:56 |
| adreznec | Ok | 13:56 |
| thorst | next topic? | 13:56 |
| efried | There's probably only three or four comments that need some nontrivial text added. | 13:56 |
| adreznec | #topic Next steps on stabilizing CI | 13:56 |
| esberglu | I have a couple things for that | 13:56 |
| adreznec | So esberglu once we land the updated local2remote patch, what's next? | 13:56 |
| *** tblakes has joined #openstack-powervm | 13:56 | |
| esberglu | adreznec: I just saw your comment about disabling stable/mitaka runs. stable/mitaka is not compat. with 16.04, which we have now moved to | 13:57 |
| adreznec | Ah, right | 13:57 |
| adreznec | Ok, I think I'm ok with dropping that from CI runs... | 13:57 |
| esberglu | But also I think there is another issue. The run where we discover the above remote thing only took 1 hour. Some are still taking forever / timing out | 13:58 |
| adreznec | We'd be stuck with it through the next cycle without CI going, but... I'm not sure it's a big deal | 13:58 |
| esberglu | I think there is a devstack config option to force runs even though devstack hasn't been tested on 16.04 | 13:58 |
| *** smatzek has joined #openstack-powervm | 13:59 | |
| adreznec | Yeah | 13:59 |
| esberglu | If we want to try that on staging at some point and see what happens | 13:59 |
| adreznec | You can always force the run | 13:59 |
| thorst | adreznec esberglu: we need 16.04 because OSA, right? | 13:59 |
| esberglu | Yeah | 13:59 |
| adreznec | Not sure it's worth the headache down the road | 13:59 |
| adreznec | Well and for Ocata | 13:59 |
| adreznec | Ocata isn't going to support trusty for most projects by the end of the cycle | 14:00 |
| adreznec | So we'd be here in a month or two anyway | 14:00 |
| thorst | OK - yeah, I'm OK with that. | 14:01 |
| thorst | unfortunate, but OK. | 14:01 |
| thorst | can't do something like that when in tree... | 14:01 |
| efried | So the timeouts appear to be related to our multiplexed image upload algorithm. | 14:01 |
| adreznec | Yeah | 14:01 |
| thorst | efried: when you say multiplexed... | 14:01 |
| efried | We need a deeper debug (I'm probably on the hook for that); but I think a broader design discussion may be in order. | 14:01 |
| thorst | do you mean my code or your marker lu thing? | 14:01 |
| adreznec | We'll need to figure out handling multiple image "flavors" for different branches down the road | 14:01 |
| efried | probably the wrong term. I mean the marker LU thing. | 14:02 |
| adreznec | Fortunately we have ~2 years to figure that out, probably | 14:02 |
| thorst | efried: and how much of that is due to marker lu or the fact that the file never actually uploads (my thing) | 14:02 |
| efried | thorst, you mean the thing that _just_ happened? Not related. The marker-based upload stuff behaves properly in that scenario. | 14:03 |
| efried | Which is why this is kinda bizarre. | 14:03 |
| esberglu | adreznec: That multiple flavor thing will be a piece of cake once zuul v3 comes out | 14:03 |
| efried | It should be behaving the same on any other kind of failure. | 14:03 |
| adreznec | Yep | 14:03 |
| adreznec | That's why I don't think it's worth chasing now | 14:03 |
| adreznec | When we get more complex config (static nodes, etc) with zuulv3 | 14:03 |
| thorst | so revisit when we have things a bit more stable (patch landed) | 14:03 |
| thorst | ready to move onto the issues that wangqwsh is hitting? | 14:04 |
| adreznec | Ok, we'll need to have a deeper dive into this once we land the local2remote stuff | 14:04 |
| efried | Wait | 14:04 |
| efried | aren't we still discussing the upload hangs? | 14:05 |
| * thorst waiting... | 14:05 | |
| adreznec | Yes | 14:05 |
| efried | 1) I wonder if we need to move the marker LU *creation* inside the try/finally; 2) I wonder if we somehow need to handle the scenario where deleting the marker LU fails; but most profoundly, 3) should we consider a timeout of some kind, where I can delete a marker LU I didn't create if a certain amount of time has elapsed? (scary) | 14:05 |
| efried | It's possible 3a) we can detect whether the real image LU hasn't been created for "a while" and act then. | 14:06 |
| thorst | a timeout scares the hell out of me | 14:06 |
| thorst | a timeout where we see no progress being made doesn't scare me | 14:06 |
| efried | Yeah, there's no way we can really set expectation for the speed of the actual upload. | 14:06 |
| thorst | well, if we see any bytes moving...then ok | 14:07 |
| thorst | but do we even get that visibility? | 14:07 |
| efried | Yeah, I don't know if there's a way to detect how much of the upload has happened. | 14:07 |
| *** tjakobs has joined #openstack-powervm | 14:08 | |
| adreznec | Do we really have visibility into the rate of data happening in the upload? | 14:08 |
| efried | The schema doesn't provide anything but the capacity as far as the LU itself is concerned. | 14:08 |
| efried | And remember, the whole point is that the upload is happening from a different nvl that we can't talk to (except through the SSP). | 14:09 |
| efried | So... we could theoretically use the marker LU as a message bus. This gets pretty complicated. | 14:09 |
| efried | Have the owner of the marker LU write heartbeats of some kind, and the other guys read the heartbeats. | 14:09 |
| efried | Now we have clock sync problems and everything; but we can get around that. | 14:10 |
| thorst | can you see a last touched thing? | 14:10 |
| thorst | get a time that the marker was last touched and have the one uploading actually touch the marker | 14:10 |
| efried | Not via REST. Would at the very least have to map & mount it. | 14:10 |
| thorst | though, we get into the same lock contention we'd be in otherwise | 14:10 |
| thorst | whoa, no mounts | 14:10 |
| adreznec | Ew ew ew | 14:11 |
| efried | Maybe not mount. | 14:11 |
| efried | What metadata does linux provide on a mapped device? | 14:11 |
| efried | So yeah, not map, but read. | 14:11 |
| efried | Basically have raw, dd-able data written by the marker owner, read by the waiters. | 14:12 |
| thorst | should we table that for further discussion? I want to make sure we get to wangqwsh's item because it is late for him | 14:12 |
| thorst | we can swing back to it? | 14:12 |
| efried | sure. | 14:12 |
| adreznec | Ok | 14:12 |
| efried | If I propose a patch for 1 & 2... | 14:12 |
| adreznec | esberglu: any other CI stabilizing topics? | 14:12 |
| efried | Can it be tested without merging it? | 14:12 |
| adreznec | efried: that would probably be a good place to start discussion | 14:12 |
| adreznec | and I think we could? | 14:12 |
| efried | k | 14:12 |
| esberglu | I think thats it | 14:13 |
| adreznec | Ok | 14:13 |
| adreznec | #topic OpenStack-Ansible CI bring-up | 14:14 |
| adreznec | wangqwsh: thorst the floor is yours | 14:14 |
| adreznec | Oh, right | 14:14 |
| adreznec | #action efried to start proposing discussion patches on marker LU enhancements | 14:14 |
| adreznec | As you were | 14:14 |
| efried | (#1 is kinda dead in the water, alas) | 14:14 |
| thorst | alright. wangqwsh I think you were seeing odd Configparser import issues due to the use of the local2remote patch in your OSA CI | 14:15 |
| thorst | as of last night when we discussed, we didn't really have a plan... | 14:15 |
| adreznec | I think we have two options | 14:16 |
| thorst | wangqwsh: did you make any progress on it or is that still the latest? | 14:16 |
| *** efried_otm has joined #openstack-powervm | 14:16 | |
| wangqwsh | no progress... | 14:16 |
| adreznec | Either install configparser into the nova-master venv for the compute node | 14:16 |
| adreznec | Or make it so the local2remote patch doesn't require configparser | 14:16 |
| thorst | let me look at that patch... | 14:17 |
| thorst | ewww...the patch has a tab in it! | 14:17 |
| wangqwsh | repo container builds the wheels. | 14:17 |
| adreznec | wangqwsh: that doesn't really matter, we could patch it in post-build | 14:18 |
| thorst | adreznec efried: I feel like ConfigParser could easily be removed...for something more trivial. Though its probably a few hours of work. | 14:18 |
| thorst | are we using the 'setup.ini'? | 14:19 |
| adreznec | Hmm ok | 14:19 |
| adreznec | Let me look at the patch | 14:19 |
| thorst | sorry...pypowervm.ini | 14:19 |
| efried_otm | I could rewrite the confit parsing if I had to. not trivial. | 14:20 |
| thorst | yeah, but are we even using it... | 14:20 |
| thorst | it looks like it could fall back to nothing | 14:20 |
| efried_otm | other than in the patch? I don't think so. | 14:20 |
| efried_otm | ugh, and do the discovery every time, which is slow. | 14:21 |
| thorst | efried_otm: yeah, but | 14:21 |
| thorst | discovery when you start the adapter. | 14:21 |
| efried_otm | but yeah, that's the east path. | 14:21 |
| thorst | which is once. | 14:21 |
| thorst | maybe twice. | 14:21 |
| efried_otm | easy* | 14:22 |
| *** wangqwsh_ has joined #openstack-powervm | 14:23 | |
| *** wangqwsh has quit IRC | 14:23 | |
| *** wangqwsh_ is now known as wangqwsh | 14:23 | |
| adreznec | Hmm | 14:23 |
| thorst | flip side... | 14:23 |
| thorst | how hard is it to add that dependency to the container? | 14:23 |
| thorst | adreznec / esberglu? | 14:24 |
| adreznec | Lets see | 14:24 |
| adreznec | So the actual action of adding it to the container is really easy, right? | 14:24 |
| adreznec | The path is consistent, so we'd source /path/to/nova/venv/bin/activate | 14:24 |
| adreznec | and pip install configparser == v.whatever there | 14:25 |
| adreznec | timing might be more complicated | 14:25 |
| adreznec | wangqwsh: at what point in the run were you seeing the failure? | 14:26 |
| *** efried_otm has quit IRC | 14:26 | |
| adreznec | Would it be enough to let the OSA AIO finish running, then before we kick tempest patch configparser into the venv, restart nova-compute, validate it comes up, then do the tempest run? | 14:26 |
| wangqwsh | start the nova-compute service, it printed | 14:26 |
| thorst | but will OSA be OK if nova-compute just dies | 14:27 |
| adreznec | I think so | 14:27 |
| adreznec | Easy to test locally | 14:27 |
| wangqwsh | yes | 14:27 |
| adreznec | I'll just break my driver settings, kick off an AIO and find out :) | 14:27 |
| adreznec | I think it will though | 14:27 |
| adreznec | I don't think it checks service state for long enough to notice the failure | 14:28 |
| adreznec | Ok, so 2 minutes left here | 14:28 |
| adreznec | wangqwsh: do you want to try patching configparser into the venv and seeing if nova-compute works? | 14:29 |
| adreznec | Should just need to run "source /openstack/venvs/nova-master/bin/activate" and "pip install configparser" | 14:29 |
| wangqwsh | how to install the pkg? via pip? | 14:29 |
| adreznec | The restart nova-compute | 14:29 |
| adreznec | I'll test the driver breakage situation on my AIO | 14:30 |
| adreznec | To see if that timing would work | 14:30 |
| wangqwsh | the pip config was changed to repo containter. | 14:30 |
| adreznec | Ah, and configparser isn't there? | 14:30 |
| wangqwsh | pip install would not find the pkg. | 14:30 |
| wangqwsh | yes | 14:30 |
| adreznec | Hmm ok | 14:30 |
| adreznec | That's inconvenient | 14:31 |
| thorst | I'm wondering if maybe we try to remove the dependency in pypowervm... Maybe wangqwsh could drive that and efried could review? | 14:31 |
| thorst | just seems...simpler... | 14:31 |
| adreznec | I wonder if we could just patch in configparser as an additional dependency for nova-powervm in CI runs | 14:31 |
| adreznec | and then it would just end up in the venv | 14:32 |
| wangqwsh | repo container builds the wheels using openstack requirement files. | 14:32 |
| thorst | wangqwsh: want to try those two approaches? 1) Change the nova-powervm requirements to include Configparser (I find that eww) and 2) work on the local2remote patch with efried to see how to remove that dependency | 14:33 |
| wangqwsh | ok | 14:34 |
| adreznec | wangqwsh: right, we could basically add configparser to the list of requirements needed for nova-powervm, but only for CI runs | 14:34 |
| adreznec | Sure | 14:34 |
| adreznec | I'll take #1 | 14:34 |
| adreznec | #action adreznec to test patching nova-powervm requirements to include configparser in OSA CI runs | 14:34 |
| adreznec | #action wangqwsh and efried to evaluate removing configparser dependency from local2remote patch | 14:34 |
| adreznec | #topic Future meetings | 14:35 |
| adreznec | So I think this has been pretty productive | 14:35 |
| thorst | I liked this. We should do it again | 14:35 |
| adreznec | What do you guys think about doing this weekly | 14:35 |
| adreznec | I can get something scheduled | 14:35 |
| thorst | +2 | 14:35 |
| adreznec | efried: esberglu wangqwsh ^^ ?? | 14:35 |
| thorst | we should get a wiki out there too, like the nova meetings. | 14:35 |
| adreznec | Right | 14:36 |
| adreznec | that was my plan | 14:36 |
| adreznec | Formalize this as a driver meeting | 14:36 |
| esberglu | Yeah I think this was better than phone calls | 14:36 |
| esberglu | Plus there is now a chat history | 14:36 |
| adreznec | Cool | 14:36 |
| wangqwsh | if the #1 not work, i can try #2 | 14:36 |
| wangqwsh | sure | 14:36 |
| wangqwsh | 1 question: | 14:36 |
| adreznec | Does this time slot work for people? | 14:37 |
| thorst | works for me | 14:37 |
| thorst | unless one of us has to SDB present... | 14:37 |
| adreznec | Right | 14:37 |
| wangqwsh | hscipaddess issue | 14:37 |
| adreznec | Which would be an issue in 2 weeks | 14:37 |
| adreznec | Ok, I'll look at calendars | 14:37 |
| wangqwsh | thorst: do you mean the hscipaddress works for you? | 14:39 |
| thorst | wangqwsh: I think that will go away with the ConfigParser dependency | 14:39 |
| thorst | I only saw that error once, so I think it was an anomoly | 14:39 |
| adreznec | Yeah | 14:40 |
| adreznec | I think that was a timing issue | 14:40 |
| wangqwsh | ok, | 14:40 |
| wangqwsh | i will try it again | 14:41 |
| adreznec | #action adreznec to schedule weekly driver team meeting | 14:41 |
| adreznec | All right, I think we're done here? | 14:41 |
| adreznec | And we're over time | 14:41 |
| adreznec | Thanks everyone! | 14:41 |
| thorst | damn...almost made it | 14:41 |
| thorst | thx! | 14:41 |
| adreznec | #endmeeting | 14:41 |
| openstack | Meeting ended Wed Nov 2 14:41:43 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:41 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/ci_scrum/2016/ci_scrum.2016-11-02-13.33.html | 14:41 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/ci_scrum/2016/ci_scrum.2016-11-02-13.33.txt | 14:41 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/ci_scrum/2016/ci_scrum.2016-11-02-13.33.log.html | 14:41 |
| adreznec | Neat | 14:42 |
| adreznec | Man | 14:42 |
| adreznec | esberglu: has no action items | 14:42 |
| adreznec | How did that happen | 14:42 |
| esberglu | Yeah I was just gonna ask if I could help with anything | 14:42 |
| esberglu | Otherwise if efried still needs help getting his devstack going | 14:42 |
| adreznec | Your job is to Make CI Great Again | 14:42 |
| thorst | yeah...how did we talk CI for 50 min and esberglu gets none :-) | 14:43 |
| esberglu | I'm sneaky | 14:43 |
| thorst | seriously | 14:43 |
| adreznec | Because the rest of us are all CI slackers | 14:43 |
| adreznec | And he already has work items | 14:43 |
| adreznec | :P | 14:43 |
| thorst | asking efried what he needs help with is best route | 14:43 |
| thorst | he's got a lot on his plate. | 14:43 |
| thorst | and we just added more | 14:43 |
| adreznec | Yep | 14:43 |
| esberglu | Yeah just let me know | 14:44 |
| esberglu | efried ^^ | 14:44 |
| thorst | I'm told efried's IRC is screwed up at the moment | 14:45 |
| thorst | so slack him if he's unresponsive | 14:45 |
| esberglu | Those meeting minutes are slick | 14:47 |
| thorst | they really are | 14:48 |
| adreznec | Yeah | 15:06 |
| adreznec | Definitely doing more of these | 15:06 |
| efried | thorst, I'm looking at the get_or_upload_image_lu code, and I don't think my above suggestions #1 & #2 make sense. | 15:13 |
| *** wangqwsh has quit IRC | 15:14 | |
| thorst | looking up that code | 15:14 |
| thorst | efried: what I just don't understand is what is causing the thing that claims the marker to not actually upload | 15:16 |
| thorst | is it just...failing/ | 15:16 |
| thorst | cause if so...we delete the marker_lu | 15:16 |
| efried | Right - that's what 'finally' is for. I haven't been able to figure out why that's not happening in the CI scenarios we've seen. | 15:17 |
| efried | Perhaps the thing to do is add some more debug logging and reproduce. | 15:17 |
| thorst | efried: my guess is that there is no winner | 15:17 |
| thorst | and they're all just waiting. | 15:17 |
| efried | No, that shouldn't be the case. | 15:17 |
| efried | And actually, the logs disprove that. | 15:17 |
| efried | Cause we do see the second guy come in, bounce off the first one, delete his marker, and wait. | 15:18 |
| efried | ...and wait, and wait, and wait... | 15:18 |
| thorst | but we don't see anything from the first... | 15:18 |
| *** mdrabe has quit IRC | 15:18 | |
| efried | We see the first one create his marker LU and start the upload. Then... nothing. | 15:19 |
| efried | esberglu, do you happen to still have any logs around from runs manifesting this timeout behavior? | 15:20 |
| thorst | adreznec kriskend: that kinda sounds like what we saw with our localdisk uploads | 15:20 |
| esberglu | http://184.172.12.213/12/392312/1/check/nova-powervm-pvm-dsvm-tempest-full/d63c182/ | 15:20 |
| thorst | and one of the reasons to move to this new model... | 15:20 |
| efried | Yeah, if the upload itself hangs, we really have no recourse. | 15:20 |
| efried | as far as this algorithm is concerned. | 15:20 |
| efried | esberglu, thanks. thorst ^^ take a look with me. | 15:21 |
| adreznec | Yeah... | 15:22 |
| adreznec | that sounds really familiar | 15:22 |
| efried | One issue we have is that we turn off debug logging for pypowervm because it's so bloody verbose (we see all the REST requests & responses). | 15:22 |
| adreznec | Yeah... | 15:23 |
| efried | Which is as it should be - those logs are WAY too much to deal with in nearly 100% of cases. | 15:23 |
| efried | In fact... I can't think of the last time we had to use 'em. | 15:23 |
| efried | I wonder... | 15:23 |
| adreznec | Mhm | 15:23 |
| adreznec | It's enough data as to make the logs kind of useless is most cases | 15:23 |
| efried | If only there was *another* log level. | 15:23 |
| efried | ooo | 15:24 |
| efried | TRACE | 15:24 |
| efried | Is that less than DEBUG? | 15:24 |
| efried | yessss. | 15:24 |
| efried | Ima propose that change RIGHT NOW. | 15:24 |
| adreznec | Amazing | 15:24 |
| efried | esberglu, your action will be to change the local.confs to make pypowervm DEBUG (it's INFO right now). | 15:25 |
| adreznec | and then TRACE becomes the log level we all wish didn't exist, most of the time | 15:25 |
| efried | 1.0.0.4? | 15:25 |
| thorst | neat | 15:25 |
| thorst | efried: yeah. | 15:25 |
| efried | rockin. | 15:25 |
| esberglu | You mean change to TRACE? | 15:26 |
| esberglu | Wait which is the more verbose one, trace or debug | 15:26 |
| *** mdrabe has joined #openstack-powervm | 15:26 | |
| efried | trace is more verbose. | 15:26 |
| esberglu | Okay so debug | 15:26 |
| esberglu | is what we want | 15:26 |
| efried | Background: pypowervm reports all the REST payloads - request & response - under DEBUG currently. This makes debug logging in pypowervm pretty much useless - which takes away our ability to use it for useful stuff - like this upload problem. | 15:27 |
| efried | So we always set pypowervm to INFO, which is manageable. | 15:27 |
| efried | I'm going to move all the REST request/response stuff to TRACE so we can set the default level to DEBUG and have it be actually useful. | 15:27 |
| efried | So | 15:27 |
| efried | thorst, it sure would be nice if I didn't have to make all these changes in the to-be-removed caching code. | 15:28 |
| efried | Can we get that guy approved? | 15:28 |
| efried | 4405 | 15:28 |
| thorst | o yeah | 15:29 |
| thorst | forgot about that | 15:29 |
| thorst | sec. | 15:29 |
| esberglu | adreznec: I like your idea of just commenting out the stable/mitaka stuff for now, and then trying to bring it back with zuul v3. I'm going to move forward with that | 15:36 |
| adreznec | esberglu: cool, I'll +2 an updated patch with that in it | 15:38 |
| efried | thorst, re-review please - added a note to README. | 15:39 |
| thorst | thx | 15:40 |
| thorst | +2 | 15:40 |
| thorst | good removal of debt... | 15:40 |
| efried | Are our jenkins builds affected by the network outage? | 15:40 |
| thorst | yes | 15:40 |
| efried | bleh. | 15:40 |
| efried | thorst, 4421 for debug=>trace. | 15:41 |
| efried | Considered leaving in the ones that don't have full body text, but figured that would still be a lot. Whatcha think? | 15:42 |
| adreznec | Right | 15:42 |
| adreznec | yeah... going to be a tough day to merge patches internally | 15:42 |
| *** tblakes has quit IRC | 15:45 | |
| adreznec | esberglu: so looking at that patch again... we're only pulling 12.0.0, 13.0.0, etc | 15:45 |
| adreznec | What do we do when tempest tags point releases? | 15:46 |
| adreznec | e.g. 12.1.0, 12.2.0? | 15:46 |
| esberglu | Probably need a discussion on that | 15:48 |
| esberglu | Would need to be tested on staging | 15:48 |
| esberglu | And changes made accordingly | 15:48 |
| esberglu | Why doesn't tempest follow the openstack release model? | 15:48 |
| adreznec | Good question | 16:03 |
| adreznec | Probably because tempest is really a library | 16:04 |
| adreznec | and not an actual core project | 16:04 |
| adreznec | libraries tend to have independent release schedules | 16:04 |
| efried | thorst, esberglu: here's a thought. Maybe I should generate the marker LU name to include some representation of the host name. That would help with debugging. | 16:09 |
| efried | We run into problems with name length, unfortunately. | 16:11 |
| esberglu | adreznec: I set it up so I will get emails of future point releases on tempest. So when they come out I will know and can test at that point | 16:14 |
| adreznec | Cool | 16:14 |
| efried | esberglu, in that log you noted above -- did you at any point go in and delete marker LUs? | 16:22 |
| esberglu | No | 16:22 |
| efried | Well, that's.... | 16:22 |
| esberglu | Didn't touch anything | 16:22 |
| efried | esberglu (thorst) So several things jump out of that log. | 16:24 |
| efried | Remember the "ssp_primer" instance? This is the one we create from os_ci_tempest.sh to prime the SSP with the image LU to make subsequent deploys faster. | 16:24 |
| esberglu | Yeah | 16:25 |
| efried | First of all, os_ci_tempest.sh doesn't create that guy if the image LU already exists in the SSP. In this log, I see we're trying to create it - so os_ci_tempest.sh must've thought we needed to. As if this was the first run against this SSP. Which clearly can't be true because... | 16:27 |
| efried | ...the ssp_primer VM creation is finding (supposedly) in-progress uploads - that is, existing marker LUs. | 16:27 |
| efried | But not just one. Like six. That's really weird. Unless half a dozen different nodes were all coming up at the exact same time. esberglu, is that possible? | 16:28 |
| efried | Probably not, because... | 16:29 |
| thorst | 6 could come up....but with jitter I think | 16:30 |
| esberglu | Maybe? If it is right when I finish redeploying the CI env. Basically zuul starts the queue while the initial image builds. Then spawns 20+ nodes and start all of the tempest runs for all of the changes that came in while that stuff was going on at the same time | 16:30 |
| esberglu | If that made any sense | 16:30 |
| esberglu | What time did this happen at? | 16:31 |
| efried | ...several *other* uploads try to happen over the next seven minutes, but they *also* bounce off of (supposedly) in-progress uploads. And those also show various large numbers of marker LUs. Like more than ten. | 16:31 |
| efried | 2016-11-01 16:47:17.489 is when the ssp_primer first bounces off the in-progress uploads. | 16:31 |
| efried | six marker LUs. | 16:31 |
| thorst | undercloud could have 6 hit at once... | 16:32 |
| esberglu | That would have been right around the time that the first runs were going on yesterday I think | 16:32 |
| thorst | but I thought we only had 4 hosts in a given SSP or something | 16:32 |
| efried | So let's assume for a second that we really did have six (or more) of the 20+ nodes trying their uploads at the same time. What should be happening is that they should all see all the marker LUs, compare them, and whoever's "first" (lowest sort order of marker LU name) should "win" - but the rest ought to delete their markers and spin. | 16:33 |
| thorst | yep | 16:33 |
| efried | So at 2016-11-01 16:48:41.456 and 2016-11-01 16:48:44.270 we see two other threads bounce off the markers - now ten of 'em. | 16:34 |
| thorst | are we talking about an undercloud rebuild or a tempest run | 16:35 |
| efried | Even if we managed to create six at once, no new marker LUs should have come into the picture after that, cause all the other guys should have just seen the existing ones and entered into a wait loop. | 16:35 |
| efried | Then at 2016-11-01 16:54:16.053 things get _really_ weird. | 16:35 |
| efried | The ssp_primer thread CREATES A MARKER LU AND TRIES TO UPLOAD. | 16:36 |
| efried | Which must mean it came out of a sleep, looked around, and found a) no marker LUs, and b) NO completed image LU. | 16:36 |
| efried | At that point, he tries his upload and fails (because APINotLocal). | 16:36 |
| efried | Which brings us to the first potentially actionable item - it's possible os_ci_tempest.sh should actually wait for the ssp_primer to come up successfully before it allows testing to start. | 16:37 |
| thorst | efried: but even with the SSP primer... | 16:38 |
| thorst | things are going to be uploaded | 16:38 |
| thorst | snapshot and then deploy from snapshot | 16:38 |
| efried | Yeah, but those are different images. | 16:38 |
| thorst | right. | 16:38 |
| efried | The snapshots need to be created from existing instances, which can't be created until the primer is there. | 16:39 |
| efried | The only thing this would buy us is the ability for os_ci_tempest.sh to bail out early and not run tempest if the ssp_primer fails to create. | 16:39 |
| efried | It wouldn't actually help the tests themselves. | 16:39 |
| efried | Call it early error detection. | 16:39 |
| efried | So anyway, I'm scouring the get_or_upload_image_lu algorithm for anything that would allow a marker LU to be created if we already see marker LUs in the SSP. | 16:40 |
| thorst | yeah...but I think we hit this thing even in undercloud build out | 16:40 |
| thorst | are marker_lu's guaranteed to be unique? | 16:41 |
| efried | As unique as the first 8 chars of a randomly-generated UUID. | 16:41 |
| efried | Which is more guaranteed-to-be-unique than the frequency with which we're seeing this issue. | 16:41 |
| thorst | so HIGHLY unlikely | 16:41 |
| thorst | +1 | 16:41 |
| efried | what's the significant difference in the undercloud? | 16:42 |
| thorst | it deploys across 20 or so hosts | 16:42 |
| thorst | but the SSP is shared across 4 hosts | 16:42 |
| thorst | so there are 5 or so SSPs | 16:42 |
| thorst | I think we've seen this issue there. | 16:42 |
| efried | still remote pypowervm? | 16:42 |
| esberglu | *deploys across 14 hosts | 16:42 |
| thorst | local pypowervm | 16:42 |
| efried | okay. I don't think that's related to the upload marker stuff - it just results in the primer failing ultimately. | 16:43 |
| efried | (remote, that is) | 16:43 |
| efried | So as I said, I'm scouring the get_or_upload_image_lu algorithm, and would welcome some more eyes. | 16:44 |
| efried | The obvious, but hopefully-impossible, thing that could cause this is our SSP GETs returning the wrong data. | 16:45 |
| efried | Like we do a GET and it comes back empty, when it's really not. | 16:45 |
| efried | I wonder if that can happen when a VIOS goes bad (busy, RMC dead, etc.) | 16:46 |
| efried | apearson? | 16:46 |
| efried | Or the other way - maybe VIOS and/or REST is caching improperly and reporting the LU list in a state it's not in anymore. | 16:48 |
| efried | Beyond that, trying to inspect the algorithm for holes where we could create a marker even if we find some already there. | 16:48 |
| efried | Sigh, this is pretty clear: | 16:49 |
| efried | if _upload_in_progress(lus, luname, first): | 16:49 |
| efried | first = False | 16:49 |
| efried | _sleep_for_upload() | 16:49 |
| efried | continue | 16:49 |
| thorst | efried: I've heard rumors of cache issues in the SSP code | 16:49 |
| efried | on VIOS or REST? | 16:49 |
| thorst | REST. | 16:49 |
| thorst | I think csky and shyama hit that. | 16:50 |
| efried | This is, of course, going to be tough as hell to nail down. | 16:50 |
| esberglu | efried: It would be weird if it was the algorithm right? Nothing has changed in that recently. If it was an algorithm thing we likely would have been hitting it at some point previously | 16:50 |
| thorst | has the algorithm ever been 100%? | 16:51 |
| efried | esberglu, have we not been hitting this more or less ever since we really cranked up the number of nodes? | 16:51 |
| thorst | well, maybe the algorithm is... | 16:51 |
| thorst | but the cache was always the issue... | 16:51 |
| esberglu | We have been seeing issues with the marker lu stuff, but the underlying cause has always been something else. | 16:52 |
| esberglu | Then the underlying cause is fixed, we run fine for a while, then it manifests again | 16:52 |
| efried | thorst, adreznec, esberglu: do me this, if you please. Take 15 minutes and scrutinize this again for holes: https://github.com/powervm/pypowervm/blob/develop/pypowervm/tasks/cluster_ssp.py#L155 | 16:53 |
| efried | With its helper methods (all in same file) it's only about 170 lines including docs, comments, and whitespace | 16:54 |
| thorst | efried: I have been :-) | 16:54 |
| efried | If I got it to pass sonar's complexity check, it can't be too complicated, can it? ;-) | 16:54 |
| thorst | while I'm reviewing...maybe reach out to Hsien to see if he knows of any issues with the cache there? | 16:55 |
| efried | rgr | 16:55 |
| thorst | specifically what Hsien saw was something with the tier...which we're using here. | 16:56 |
| efried | One thought: in the 'finally' clause, when we delete the marker, maybe we spin doing GETs until it really disappears? (And warn like hell if we have to spin even once - dammit, when delete() returns, the thing should be GONE.) | 16:58 |
| *** k0da has quit IRC | 16:58 | |
| efried | But it wouldn't be super surprising if VIOS claims completion before the thing is really purged from the SSP. They gave us that guff about LU mappings recently, if you recall. | 16:59 |
| thorst | one thing I can possibly maybe see here... | 17:00 |
| thorst | the crt_lu fails (but actually manages to succeed) | 17:00 |
| thorst | and ends up creating a marker lu. | 17:00 |
| thorst | but we think it failed... | 17:00 |
| thorst | *maybe* | 17:00 |
| *** apearson has quit IRC | 17:00 | |
| efried | Yeah, I was looking at that. That was #1 waay above. The only way we could detect that would be, after that crt_lu "fails", do a GET and see if that LU really got created. | 17:01 |
| thorst | seems unlikely | 17:02 |
| efried | Once again, that's a bunch of extra code on the theme of essentially distrusting the atomicity/transactionalism of the API. | 17:02 |
| thorst | efried: some extra logging in here feels like it could go a long way. | 17:06 |
| thorst | maybe something in the _find_lus to log the names of the LU's returned. | 17:06 |
| efried | The 'Waiting for in-progress' log records the names of the LUs we're waiting on. That's how I found out how many of those there were. | 17:08 |
| efried | Being able to use the debug log will produce those on every iteration instead of just the first one, which will be very helpful. | 17:08 |
| *** esberglu has left #openstack-powervm | 17:12 | |
| thorst | efried: true... | 17:14 |
| *** tblakes has joined #openstack-powervm | 17:20 | |
| *** tblakes has quit IRC | 17:41 | |
| *** esberglu has joined #openstack-powervm | 18:11 | |
| *** openstackgerrit has quit IRC | 18:18 | |
| *** openstackgerrit has joined #openstack-powervm | 18:18 | |
| esberglu | thorst: Any progress on the LU stuff while I was at lunch? | 18:23 |
| *** k0da has joined #openstack-powervm | 18:45 | |
| *** tblakes has joined #openstack-powervm | 19:29 | |
| *** tblakes has quit IRC | 19:35 | |
| *** seroyer_ has joined #openstack-powervm | 19:36 | |
| *** seroyer has quit IRC | 19:37 | |
| *** seroyer_ is now known as seroyer | 19:37 | |
| thorst | esberglu: not from my side...been looking at SDN stuff unfortunately | 19:50 |
| *** openstack has joined #openstack-powervm | 19:59 | |
| *** dwayne_ has quit IRC | 20:11 | |
| *** openstackgerrit has quit IRC | 20:18 | |
| *** openstackgerrit has joined #openstack-powervm | 20:18 | |
| thorst | man...Ubuntu is so nice | 20:25 |
| thorst | can't say enough nice things about Ubuntu | 20:25 |
| *** dwayne_ has joined #openstack-powervm | 20:27 | |
| *** smatzek has quit IRC | 20:29 | |
| *** tblakes has joined #openstack-powervm | 20:41 | |
| *** edmondsw has quit IRC | 20:50 | |
| *** esberglu has quit IRC | 21:03 | |
| *** esberglu has joined #openstack-powervm | 21:04 | |
| *** esberglu has quit IRC | 21:08 | |
| *** tblakes has quit IRC | 21:18 | |
| efried | thorst, and RHEL is HEL(l) | 21:21 |
| thorst | I can not confirm nor deny | 21:21 |
| thorst | but man, that ubuntu is classy | 21:21 |
| efried | adreznec, queue up topic for our next scrummyscrum (or preferably sooner): we need pypi fixed for pypowervm so we can get https://review.openstack.org/391286 going as a prereq for any serious driver integration. | 21:22 |
| efried | It needs to *happen*. | 21:22 |
| adreznec | Yeah | 21:22 |
| adreznec | I was just talking to esberglu about driver meetings/scrums/etc | 21:22 |
| efried | We can get away with the live migration object as soon as the blueprint is approved, but pypowervm is gonna block everything else. | 21:22 |
| adreznec | I'll set up a discussion with Dom | 21:23 |
| adreznec | So we can talk getting a job created to do it | 21:23 |
| adreznec | Hopefully for tomorrow if I can find calendar space | 21:23 |
| adreznec | thorst: you still here? | 21:23 |
| thorst | adreznec: kinda | 21:24 |
| thorst | about to leave | 21:24 |
| adreznec | K | 21:24 |
| adreznec | I want to cancel our existing OSA phone scrums | 21:24 |
| adreznec | and replace them with IRC discussions | 21:24 |
| adreznec | It seemed more productive for me | 21:24 |
| thorst | my only concern is with wangqing | 21:25 |
| adreznec | And wangqwsh | 21:25 |
| adreznec | Why? | 21:25 |
| thorst | given timing | 21:25 |
| adreznec | Well right | 21:25 |
| adreznec | We can discuss more when you have more time | 21:25 |
| thorst | but I'm in support otherwise, lets just make sure it works with him | 21:25 |
| thorst | it was super duper awesome sauce | 21:25 |
| adreznec | But it seemed easier for us to communicate that way | 21:25 |
| adreznec | Easier in text | 21:26 |
| adreznec | And it has logs and action items | 21:26 |
| adreznec | I'll poke at it a bit | 21:26 |
| thorst | adreznec: +1 | 21:27 |
| thorst | alright...I'm out | 21:27 |
| adreznec | see ya | 21:28 |
| *** thorst has quit IRC | 21:31 | |
| *** esberglu has joined #openstack-powervm | 21:34 | |
| *** tjakobs has quit IRC | 21:37 | |
| *** seroyer has quit IRC | 21:38 | |
| *** esberglu has quit IRC | 21:38 | |
| *** smatzek has joined #openstack-powervm | 21:57 | |
| *** tblakes has joined #openstack-powervm | 21:58 | |
| *** mdrabe has quit IRC | 22:09 | |
| *** smatzek has quit IRC | 22:30 | |
| *** apearson has joined #openstack-powervm | 22:32 | |
| *** tjakobs has joined #openstack-powervm | 22:38 | |
| *** tblakes has quit IRC | 22:58 | |
| *** apearson has quit IRC | 23:00 | |
| *** apearson has joined #openstack-powervm | 23:00 | |
| *** thorst has joined #openstack-powervm | 23:13 | |
| *** thorst has quit IRC | 23:18 | |
| *** seroyer has joined #openstack-powervm | 23:34 | |
| *** seroyer has quit IRC | 23:35 | |
| *** esberglu has joined #openstack-powervm | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!