| *** jwcroppe has joined #openstack-powervm | 00:40 | |
| *** jwcroppe has quit IRC | 00:44 | |
| *** jwcroppe has joined #openstack-powervm | 00:45 | |
| *** thorst has joined #openstack-powervm | 00:55 | |
| *** thorst has quit IRC | 00:58 | |
| *** tjakobs has joined #openstack-powervm | 01:40 | |
| *** smatzek has joined #openstack-powervm | 01:49 | |
| *** thorst has joined #openstack-powervm | 01:54 | |
| openstackgerrit | Drew Thorstensen (thorst) proposed openstack/nova-powervm: Check for instance state fix https://review.openstack.org/416315 | 02:20 |
|---|---|---|
| *** thorst has quit IRC | 02:21 | |
| *** thorst has joined #openstack-powervm | 02:21 | |
| *** smatzek has quit IRC | 02:22 | |
| *** thorst has quit IRC | 02:25 | |
| *** thorst has joined #openstack-powervm | 02:47 | |
| *** thorst has quit IRC | 02:49 | |
| *** thorst has joined #openstack-powervm | 03:35 | |
| *** thorst has quit IRC | 03:36 | |
| *** tlian2 has quit IRC | 04:26 | |
| *** thorst has joined #openstack-powervm | 04:37 | |
| *** thorst has quit IRC | 04:41 | |
| *** kotra03 has joined #openstack-powervm | 06:05 | |
| *** tjakobs has quit IRC | 06:20 | |
| *** thorst has joined #openstack-powervm | 06:38 | |
| *** thorst has quit IRC | 06:42 | |
| *** k0da has joined #openstack-powervm | 07:59 | |
| *** thorst has joined #openstack-powervm | 08:39 | |
| *** thorst has quit IRC | 08:43 | |
| *** thorst has joined #openstack-powervm | 10:39 | |
| *** thorst has quit IRC | 10:44 | |
| *** smatzek has joined #openstack-powervm | 11:40 | |
| *** kotra03 has quit IRC | 11:41 | |
| *** kotra03 has joined #openstack-powervm | 12:29 | |
| *** openstackgerrit has quit IRC | 12:33 | |
| *** thorst has joined #openstack-powervm | 12:41 | |
| *** openstackgerrit has joined #openstack-powervm | 12:48 | |
| openstackgerrit | Drew Thorstensen (thorst) proposed openstack/nova-powervm: Revert "Add delay queue for events" https://review.openstack.org/416545 | 12:48 |
| thorst | efried: FYI - just to see I've proposed a revert of the event change set...if that is the cause we'll know for sure | 12:48 |
| thorst | and if it is, we revert it, and repropose it back up...that way we've isolated the bad change set. | 12:48 |
| *** edmondsw has joined #openstack-powervm | 13:16 | |
| efried | thorst - we'll know after several rechecks, I suppose. | 13:21 |
| thorst | right. | 13:24 |
| thorst | but we bombed throughout the night too | 13:24 |
| thorst | I'm debating just shutting it off till esberglu gets back...but qing wu is his backup | 13:25 |
| thorst | I also wonder if the novalinks just need a restart | 13:25 |
| efried | thorst, is it always those same two or three tests? | 13:26 |
| thorst | in my experience | 13:26 |
| thorst | I've checked about 10 change sets. | 13:26 |
| efried | Then I think rebooting the novalinks won't help. Those failures are totally a timing bug. | 13:26 |
| thorst | we'll see. | 13:27 |
| thorst | nothing is sticking out to me as a timing bug. | 13:27 |
| efried | I'm not confident that your delay queue is the culprit, mind you. | 13:27 |
| thorst | but I don't disagree...that's what we're seeing at the moment. But I do remember a lot more successes before this. | 13:27 |
| efried | But it's clearly a timing issue in the resize flow. | 13:27 |
| thorst | me either... | 13:27 |
| thorst | the other option is to turn off those tests, then when esberglu gets back re-enable them for just nova-powervm | 13:28 |
| thorst | or in staging | 13:28 |
| thorst | but I really don't like that we puked all night on the CI | 13:28 |
| *** svenkat has joined #openstack-powervm | 13:28 | |
| thorst | I'm going to let mriedem know that our CI is acting up. | 13:28 |
| efried | thorst, we're not close to being ready to merge yet anyway. | 13:29 |
| thorst | and that we'll give an update on the PowerVM change sets tomorrow. Would you be able to do that in the IRC meeting? | 13:29 |
| efried | I can do that; will you be out or something? | 13:29 |
| thorst | well, our CI is puking on EVERY nova change set for out OOT driver. | 13:29 |
| efried | Yes, but even if it wasn't, we shouldn't expect to merge until we've got in-tree passing. | 13:29 |
| efried | Which won't happen until we have the second config/skip-list set up. | 13:30 |
| thorst | nope, but you're driving it so I'm hoping you can give them status. Plus with baby on the way, I won't be going to Boston summit, so its good that you give updates now for when you go to Boston | 13:30 |
| efried | okay. | 13:30 |
| efried | What time is that meeting? I should put it on my calendar in case I don't get flagged. | 13:30 |
| thorst | efried: I'm not sure they would agree that we need to wait for our in tree (for at least the first couple of change sets) for the separate CI | 13:30 |
| thorst | https://wiki.openstack.org/wiki/Meetings/Nova | 13:31 |
| thorst | 2100 UTC -> 3 PM CT I think | 13:31 |
| thorst | we should also add ourselves to agenda. | 13:32 |
| thorst | I can do that. | 13:32 |
| efried | Yeah, I don't see the thing where there's a list of handles they mention to alert folks. | 13:33 |
| efried | Assuming GMT and UTC are the same this time of year, it is indeed 3PM CT. | 13:34 |
| *** mdrabe has joined #openstack-powervm | 13:36 | |
| efried | thorst, I'm going to set up a recurring meeting on my calendar for these suckers. You want me to invite you? | 13:36 |
| thorst | efried: I put us at the bottom here: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | 13:36 |
| thorst | sure, but just note they change it around once and a while :-) | 13:36 |
| efried | great | 13:36 |
| efried | BTW, even if the cores don't care, I think we should have our CI passing before we merge our in-tree change sets. | 13:37 |
| *** jpasqualetto has joined #openstack-powervm | 13:37 | |
| efried | adreznec, you want in on this invite too? | 13:37 |
| thorst | I think it needs to be worked in parallel. Devstack sure...and also I'm OK if it takes 5 stacked change sets before it works. | 13:38 |
| thorst | if we try to make it work on change set 1...then change set 1 could be colossal...which is bad for review bandwidth. | 13:38 |
| thorst | but hopefully I'm wrong on that. | 13:38 |
| thorst | be cognizant of the Ocata release schedule too...I think we've missed the boat. | 13:38 |
| thorst | :-( | 13:38 |
| efried | I declined yours, gonna use mine. | 13:41 |
| thorst | fair | 13:42 |
| *** efried has quit IRC | 13:44 | |
| jpasqualetto | Hello everyone | 13:45 |
| jpasqualetto | I'm working in a test environment with the powervm drivers. I'm trying to find powervm compatible images (or how to create them). Is there any images available for download at this moment? | 13:45 |
| *** tlian has joined #openstack-powervm | 13:53 | |
| thorst | jpasqualetto: I don't think there is a public repo at the moment, but I *think* we put an Ubuntu image up on box at one point | 13:53 |
| thorst | adreznec: Do you recall? | 13:53 |
| jpasqualetto | thorst, I saw this in the irc logs in september 29. But I don't have access to the box.. is there any other way to get access to this image? | 13:57 |
| thorst | jpasqualetto: do you have a box ID? We could maybe grant access. It is a large image, so e-mail won't work :-) | 13:58 |
| thorst | if you have a box ID, just PM me your address and I can work with adreznec to get you added? | 13:58 |
| jpasqualetto | thorst, unfortunately I have no box ID | 14:00 |
| *** jwcroppe has quit IRC | 14:00 | |
| jpasqualetto | thorst, i'll try to create one | 14:00 |
| thorst | cool | 14:00 |
| *** apearson has joined #openstack-powervm | 14:06 | |
| *** tblakes has joined #openstack-powervm | 14:10 | |
| *** kriskend has joined #openstack-powervm | 14:15 | |
| adreznec | Sorry, just catching up on backscroll. thorst we did, let me find it | 14:18 |
| *** jwcroppe has joined #openstack-powervm | 14:21 | |
| *** smatzek has quit IRC | 14:21 | |
| *** efried has joined #openstack-powervm | 14:40 | |
| *** k0da has quit IRC | 14:48 | |
| *** smatzek has joined #openstack-powervm | 14:56 | |
| openstackgerrit | Drew Thorstensen (thorst) proposed openstack/nova-powervm: Check for instance state fix https://review.openstack.org/416315 | 15:03 |
| *** kotra03 has quit IRC | 15:07 | |
| *** kotra03 has joined #openstack-powervm | 15:07 | |
| *** kotra03_ has joined #openstack-powervm | 15:11 | |
| *** kotra03 has quit IRC | 15:13 | |
| *** kotra03_ is now known as kotra03 | 15:13 | |
| *** tjakobs has joined #openstack-powervm | 15:35 | |
| *** kriskend_ has joined #openstack-powervm | 15:35 | |
| *** kriskend has quit IRC | 15:39 | |
| thorst | efried: https://review.openstack.org/#/c/416315/ | 16:36 |
| thorst | success...but slower | 16:36 |
| thorst | doing another recheck | 16:36 |
| thorst | if it works...I'll post proper... | 16:36 |
| thorst | efried adreznec: I was thinking if we don't fix by EOD, we should have qing wu turn off the publishing of the PowerVM logs to nova | 16:38 |
| thorst | until we get it sorted...it's a lot of red for other people's reviews. | 16:38 |
| thorst | but that way we'd still run the job and get the results for us. | 16:39 |
| adreznec | Mhm | 16:40 |
| adreznec | Seems fair | 16:40 |
| efried | thorst, sowait, PS5 just turns off the delay queue? | 16:42 |
| thorst | efried: it turns off all queining | 16:42 |
| thorst | sorry...all eventing | 16:42 |
| efried | thorst, how much slower was this? | 16:44 |
| thorst | 20 min | 16:44 |
| thorst | unclear if it is because my change or not | 16:44 |
| thorst | doing another run | 16:44 |
| efried | k | 16:44 |
| *** edmondsw_ has joined #openstack-powervm | 16:45 | |
| *** edmondsw_ has quit IRC | 16:45 | |
| *** kotra03 has quit IRC | 16:57 | |
| openstackgerrit | Adam Reznechek proposed openstack/networking-powervm: Switch to manual service enablement for devstack plugins https://review.openstack.org/416667 | 17:13 |
| *** kotra03 has joined #openstack-powervm | 17:14 | |
| *** smatzek has quit IRC | 17:19 | |
| *** kotra03 has quit IRC | 17:36 | |
| *** k0da has joined #openstack-powervm | 17:45 | |
| *** smatzek has joined #openstack-powervm | 17:46 | |
| thorst | efried adreznec: even after turning off all of eventing...we have this issue. | 18:04 |
| *** efried has quit IRC | 18:13 | |
| *** tblakes has quit IRC | 18:25 | |
| *** tblakes has joined #openstack-powervm | 18:29 | |
| *** apearson has quit IRC | 18:50 | |
| thorst | adreznec: how hard is it to change a conf file option in the CI? | 18:53 |
| adreznec | thorst: Which conf | 18:56 |
| thorst | local.conf | 18:56 |
| thorst | I'm not thinking that it is the sync_power_state coming in | 18:56 |
| adreznec | For the undercloud? or the dsvms | 18:56 |
| thorst | saying that a VM is down when it is just in the middle of a resize | 18:56 |
| thorst | dsvms | 18:56 |
| adreznec | uhh we'd have to change the conf, then rebuild the base image since I think it's in there | 18:57 |
| adreznec | then rebuild all the slaves | 18:57 |
| thorst | so quite hard. | 18:58 |
| *** apearson has joined #openstack-powervm | 19:04 | |
| *** jpasqualetto has quit IRC | 19:18 | |
| *** dwayne__ has quit IRC | 19:36 | |
| *** dwayne__ has joined #openstack-powervm | 19:36 | |
| thorst | latest theory... | 19:46 |
| thorst | the novalink is giving back an old status. | 19:46 |
| adreznec | thorst: Like... old partition state? e.g. bad cache? | 19:56 |
| thorst | I dunno... | 19:56 |
| adreznec | Hmm k | 19:56 |
| adreznec | that'd be ugly | 19:56 |
| *** apearson has quit IRC | 19:57 | |
| thorst | well its weird. We can see that in the finish_migration they tell us to power it on. | 19:58 |
| thorst | which we do | 19:58 |
| thorst | then in the confirm_migration (which runs right after), they check the instance.power_state | 19:58 |
| thorst | and detect that it's shutdown | 19:58 |
| thorst | so then they tell us to shut down | 19:58 |
| thorst | which we do... | 19:58 |
| *** efried has joined #openstack-powervm | 20:00 | |
| *** efried has quit IRC | 20:02 | |
| *** efried has joined #openstack-powervm | 20:02 | |
| *** efried has left #openstack-powervm | 20:03 | |
| *** efried has joined #openstack-powervm | 20:08 | |
| *** efried has quit IRC | 20:09 | |
| *** apearson has joined #openstack-powervm | 20:10 | |
| *** efried has joined #openstack-powervm | 20:16 | |
| *** efried has quit IRC | 20:17 | |
| *** efried has joined #openstack-powervm | 20:37 | |
| *** tblakes has quit IRC | 21:17 | |
| openstackgerrit | Drew Thorstensen (thorst) proposed openstack/nova-powervm: Check for instance state fix https://review.openstack.org/416315 | 21:54 |
| *** thorst has quit IRC | 21:58 | |
| *** thorst has joined #openstack-powervm | 22:00 | |
| *** thorst has quit IRC | 22:05 | |
| *** kriskend_ has quit IRC | 22:12 | |
| *** smatzek has quit IRC | 22:13 | |
| *** kriskend has joined #openstack-powervm | 22:13 | |
| *** svenkat has quit IRC | 22:14 | |
| *** thorst has joined #openstack-powervm | 22:18 | |
| *** thorst has quit IRC | 22:21 | |
| *** thorst has joined #openstack-powervm | 22:22 | |
| *** tblakes has joined #openstack-powervm | 22:29 | |
| *** thorst has quit IRC | 22:39 | |
| *** edmondsw has quit IRC | 22:41 | |
| *** edmondsw has joined #openstack-powervm | 22:42 | |
| *** edmondsw has quit IRC | 22:47 | |
| *** edmondsw has joined #openstack-powervm | 22:48 | |
| *** openstack has joined #openstack-powervm | 22:55 | |
| *** tjakobs has quit IRC | 23:02 | |
| *** tblakes has quit IRC | 23:08 | |
| *** kriskend_ has joined #openstack-powervm | 23:16 | |
| *** mdrabe has quit IRC | 23:16 | |
| *** kriskend has quit IRC | 23:20 | |
| *** edmondsw has joined #openstack-powervm | 23:32 | |
| *** apearson has quit IRC | 23:33 | |
| *** thorst_ has joined #openstack-powervm | 23:35 | |
| *** edmondsw has quit IRC | 23:37 | |
| *** k0da has quit IRC | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!