| *** thorst_afk has joined #openstack-powervm | 00:19 | |
| *** thorst_afk has quit IRC | 00:24 | |
| *** apearson has joined #openstack-powervm | 00:29 | |
| *** adreznec has quit IRC | 00:59 | |
| *** adreznec has joined #openstack-powervm | 01:01 | |
| *** thorst_afk has joined #openstack-powervm | 01:02 | |
| *** apearson has quit IRC | 01:03 | |
| *** thorst_afk has quit IRC | 01:04 | |
| *** lan2 has joined #openstack-powervm | 01:51 | |
| *** thorst_afk has joined #openstack-powervm | 01:58 | |
| *** thorst_afk has quit IRC | 01:58 | |
| *** tjakobs has joined #openstack-powervm | 02:25 | |
| *** jay1_ has joined #openstack-powervm | 02:42 | |
| *** thorst_afk has joined #openstack-powervm | 02:57 | |
| *** thorst_afk has quit IRC | 02:57 | |
| *** lan1 has joined #openstack-powervm | 03:28 | |
| *** thorst_afk has joined #openstack-powervm | 03:28 | |
| *** lan2 has quit IRC | 03:31 | |
| *** jay1_ has quit IRC | 03:34 | |
| *** thorst_afk has quit IRC | 03:46 | |
| *** apearson has joined #openstack-powervm | 05:05 | |
| *** apearson has quit IRC | 05:31 | |
| *** arunman has joined #openstack-powervm | 05:34 | |
| *** jay1_ has joined #openstack-powervm | 05:39 | |
| *** thorst_afk has joined #openstack-powervm | 05:43 | |
| *** thorst_afk has quit IRC | 05:48 | |
| *** chas has joined #openstack-powervm | 06:19 | |
| *** jay1_ has quit IRC | 06:34 | |
| *** chas has quit IRC | 06:37 | |
| *** chas has joined #openstack-powervm | 06:37 | |
| *** chas has quit IRC | 06:41 | |
| *** chas has joined #openstack-powervm | 06:41 | |
| *** thorst_afk has joined #openstack-powervm | 06:44 | |
| *** thorst_afk has quit IRC | 06:49 | |
| *** chas has quit IRC | 06:55 | |
| *** chas has joined #openstack-powervm | 06:55 | |
| *** chas has quit IRC | 06:57 | |
| *** chas has joined #openstack-powervm | 06:57 | |
| *** chas has quit IRC | 07:06 | |
| *** chas has joined #openstack-powervm | 07:06 | |
| *** chas has quit IRC | 07:11 | |
| *** k0da has joined #openstack-powervm | 07:38 | |
| *** thorst_afk has joined #openstack-powervm | 07:45 | |
| *** chas has joined #openstack-powervm | 07:47 | |
| *** thorst_afk has quit IRC | 07:50 | |
| *** tjakobs has quit IRC | 08:21 | |
| *** thorst_afk has joined #openstack-powervm | 08:46 | |
| *** chhavi has joined #openstack-powervm | 08:55 | |
| *** thorst_afk has quit IRC | 09:05 | |
| *** k0da has quit IRC | 09:46 | |
| *** lan2 has joined #openstack-powervm | 09:49 | |
| *** lan1 has quit IRC | 09:53 | |
| *** k0da has joined #openstack-powervm | 10:00 | |
| *** thorst_afk has joined #openstack-powervm | 10:02 | |
| *** thorst_afk has quit IRC | 10:07 | |
| *** chas has quit IRC | 10:08 | |
| *** chas has joined #openstack-powervm | 10:09 | |
| *** chas has quit IRC | 10:13 | |
| *** chas has joined #openstack-powervm | 10:43 | |
| *** chas has quit IRC | 10:47 | |
| *** chas has joined #openstack-powervm | 10:47 | |
| *** smatzek has joined #openstack-powervm | 11:01 | |
| *** thorst_afk has joined #openstack-powervm | 11:18 | |
| *** chhavi has quit IRC | 11:20 | |
| *** arunman has quit IRC | 11:52 | |
| *** edmondsw has joined #openstack-powervm | 12:15 | |
| *** edmondsw has quit IRC | 12:16 | |
| *** jwcroppe has quit IRC | 12:33 | |
| *** jwcroppe has joined #openstack-powervm | 12:33 | |
| *** mdrabe has joined #openstack-powervm | 12:35 | |
| *** jwcroppe has quit IRC | 12:37 | |
| *** chhavi has joined #openstack-powervm | 12:39 | |
| *** jwcroppe has joined #openstack-powervm | 12:58 | |
| *** lan2 has quit IRC | 13:00 | |
| *** esberglu has joined #openstack-powervm | 13:01 | |
| *** esberglu has quit IRC | 13:03 | |
| *** esberglu has joined #openstack-powervm | 13:05 | |
| *** esberglu_ has joined #openstack-powervm | 13:06 | |
| *** esberglu has quit IRC | 13:09 | |
| efried | esberglu_ CI sane at this point? | 13:10 |
|---|---|---|
| esberglu_ | efried: I'm looking into it. OOT runs are working fine but lots of failing tests | 13:16 |
| esberglu_ | The cell setup stuff isn't working in tree | 13:17 |
| efried | boo, okay. Let me know if you need help | 13:17 |
| esberglu_ | I'm gonna look into the in tree first since its failing everything. You can look into OOT failures if you want | 13:19 |
| esberglu_ | I still haven't been able to figure out the source of that rebuild failure we were looking at last week | 13:20 |
| esberglu_ | efried: The cell thing is weird. I can see it running the same nova-manage command that works OOT. And worked IT on staging | 13:22 |
| efried | esberglu_ What's the error? | 13:23 |
| esberglu_ | http://184.172.12.213/30/461630/4/check/nova-in-tree-pvm/3aaf8e5/powervm_os_ci.html | 13:23 |
| esberglu_ | Host is not mapped to any cell | 13:23 |
| esberglu_ | I'm wondering if we run the nova manage too quickly after we restart the services | 13:23 |
| esberglu_ | But I would think it would fail in that case, not return true | 13:24 |
| efried | esberglu_ Where's the code that runs the nova-manage command? | 13:27 |
| esberglu_ | prep_devstack | 13:27 |
| esberglu_ | You can see it run in the console log | 13:27 |
| *** apearson has joined #openstack-powervm | 13:27 | |
| esberglu_ | Just search for "nova-manage cell_v2 discover_hosts" | 13:27 |
| esberglu_ | I'm running through the script manually on a test instance right now | 13:28 |
| *** kylek3h has quit IRC | 13:30 | |
| *** smatzek has quit IRC | 13:34 | |
| *** kylek3h has joined #openstack-powervm | 13:56 | |
| *** smatzek has joined #openstack-powervm | 13:58 | |
| *** smatzek_ has joined #openstack-powervm | 14:02 | |
| *** tjakobs has joined #openstack-powervm | 14:03 | |
| *** smatzek has quit IRC | 14:06 | |
| efried | esberglu_ Do you have "Consume HttpNotFound" on your list? | 14:18 |
| esberglu_ | efried: Nope | 14:20 |
| efried | esberglu_ Okay, I'll knock that one out real quick. | 14:20 |
| esberglu_ | Sweet | 14:21 |
| *** arunman has joined #openstack-powervm | 14:25 | |
| *** edmondsw has joined #openstack-powervm | 14:29 | |
| esberglu_ | efried: All signs seem to be pointing to a timing issue | 14:35 |
| esberglu_ | I can see in the logs that the discover_hosts command worked on the manual run | 14:35 |
| esberglu_ | My understanding from the discussion in nova is that for now | 14:37 |
| esberglu_ | there isn't a great way to determine if it worked | 14:37 |
| esberglu_ | efried: What do you think our best option is short term? | 14:38 |
| efried | esberglu_ There is no best option really-short-term. The bad option is "add a sleep". | 14:39 |
| efried | esberglu_ Whyncha see if you can add that --strict flag | 14:39 |
| efried | If that looks like it's going to take longer than a day or so, then we can add the sleep. | 14:40 |
| *** smatzek_ is now known as smatzek | 14:42 | |
| esberglu_ | efried: As long as everyone is cool with failing in-tree runs for the time being | 14:44 |
| efried | esberglu_ For "a day or so", yeah. | 14:45 |
| efried | The change sets we have in flight aren't time critical, I don't think. | 14:45 |
| efried | We'll let people scream if they feel otherwise. | 14:45 |
| *** tjakobs has quit IRC | 14:48 | |
| *** dwayne has joined #openstack-powervm | 14:57 | |
| *** apearson has quit IRC | 15:20 | |
| *** dwayne has quit IRC | 15:22 | |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: Consume HttpNotFound https://review.openstack.org/466768 | 15:24 |
| efried | esberglu_ http://184.172.12.213/25/466425/1/check/nova-powervm-out-of-tree-pvm/761dd61/powervm_os_ci.html <== can I just recheck this sucker, or do we actually need to fix something? | 15:25 |
| esberglu_ | Recheck unless you feel like diving into the logs to see what happened | 15:26 |
| *** apearson has joined #openstack-powervm | 15:26 | |
| esberglu_ | efried: OOT success rates are also down since the redeploy last friday | 15:27 |
| efried | esberglu_ Are OOT and/or IT runs using systemd at this point? | 15:28 |
| esberglu_ | But there have been a decent number failing the jenkins checks too | 15:28 |
| esberglu_ | Yeah | 15:28 |
| *** dwayne has joined #openstack-powervm | 15:29 | |
| efried | Awesome. Log colors are working beautifully. | 15:29 |
| esberglu_ | efried: Yeah zipping the files works consistently now too | 15:30 |
| efried | I noticed that on in-tree, but not this one. | 15:30 |
| efried | esberglu_ ^ | 15:30 |
| esberglu_ | The coloring or the zip? | 15:30 |
| *** tjakobs has joined #openstack-powervm | 15:31 | |
| esberglu_ | efried: ^ | 15:31 |
| efried | esberglu_ The zipping. Oh - I guess the logs themselves are zipped fine; but I noticed in the last IT run I looked at that the console & summary HTMLs were zipped; but not so in OOT. | 15:32 |
| efried | esberglu_ Not that I think that's a big deal, really. But may point to an inconsistency in the CI code. | 15:32 |
| esberglu_ | Yeah. I see whats happening. We only gzip *.log | 15:33 |
| esberglu_ | Not a big deal. But an easy fix. Added to my low hanging fruit list | 15:33 |
| efried | esberglu_ But in-tree we zip all? | 15:33 |
| esberglu_ | efried: Where are you seeing that? | 15:34 |
| efried | eeehhhh, lemme find it... | 15:34 |
| efried | esberglu_ E.g. http://184.172.12.213/34/412634/27/check/nova-in-tree-pvm/0c64b5b/ | 15:34 |
| efried | The HTMLs are zipped (but not the subunit) | 15:34 |
| esberglu_ | When is that run from? | 15:34 |
| esberglu_ | Oh nvm its right there | 15:35 |
| *** apearson has quit IRC | 15:35 | |
| esberglu_ | efried: Yeah that would have been friday before the redeploy | 15:35 |
| efried | esberglu_ I'm not looking at the source, but if you wanted to zip everything, rather than having to figure out which suffixes to use, you can say something like: find . -type f | xargs gzip | 15:35 |
| esberglu_ | Wait maybe not | 15:35 |
| efried | esberglu_ ...if you wanna. | 15:36 |
| esberglu_ | efried: Yeah this is a remnant of the way screen had a bunch of symlinks and stuff | 15:36 |
| esberglu_ | I can clean it up the log publishing stuff now that we are on systemd | 15:36 |
| efried | esberglu_ Or if you just want to zip stuff over a certain size: find . -type f -size +4096c | xargs gzip | 15:37 |
| efried | (that'll zip anything over 4KB) | 15:38 |
| efried | But anyway, yeah, way low on the priority list. | 15:38 |
| *** apearson has joined #openstack-powervm | 15:41 | |
| *** k0da has quit IRC | 15:49 | |
| *** apearson has quit IRC | 16:07 | |
| *** chas has quit IRC | 16:09 | |
| *** apearson has joined #openstack-powervm | 16:14 | |
| *** apearson has quit IRC | 16:24 | |
| *** apearson has joined #openstack-powervm | 16:25 | |
| *** apearson has quit IRC | 16:25 | |
| *** arunman has quit IRC | 16:27 | |
| *** kjw3 has joined #openstack-powervm | 16:37 | |
| *** apearson has joined #openstack-powervm | 16:56 | |
| openstackgerrit | Eric Larese proposed openstack/nova-powervm master: Clean up orphan tap devices on migrate https://review.openstack.org/466400 | 17:46 |
| *** k0da has joined #openstack-powervm | 17:46 | |
| *** kjw3 has quit IRC | 17:49 | |
| *** k0da has quit IRC | 17:55 | |
| *** kjw3 has joined #openstack-powervm | 18:02 | |
| *** chas has joined #openstack-powervm | 18:09 | |
| *** chas has quit IRC | 18:14 | |
| *** chas has joined #openstack-powervm | 18:31 | |
| *** chas has quit IRC | 18:35 | |
| *** esberglu_ has quit IRC | 18:42 | |
| *** esberglu has joined #openstack-powervm | 18:44 | |
| *** smatzek has quit IRC | 19:02 | |
| *** smatzek has joined #openstack-powervm | 19:02 | |
| *** k0da has joined #openstack-powervm | 19:04 | |
| *** chas has joined #openstack-powervm | 19:12 | |
| *** chas has quit IRC | 19:17 | |
| *** chas has joined #openstack-powervm | 19:33 | |
| *** chas has quit IRC | 19:38 | |
| *** chhavi has quit IRC | 19:48 | |
| *** chas has joined #openstack-powervm | 19:54 | |
| *** smatzek has quit IRC | 19:55 | |
| *** smatzek has joined #openstack-powervm | 19:58 | |
| *** chas has quit IRC | 19:59 | |
| *** chas has joined #openstack-powervm | 20:15 | |
| *** chas has quit IRC | 20:19 | |
| *** smatzek has quit IRC | 20:34 | |
| *** chas has joined #openstack-powervm | 20:36 | |
| *** chas has quit IRC | 20:41 | |
| *** chas has joined #openstack-powervm | 20:57 | |
| *** thorst_afk has quit IRC | 20:58 | |
| *** thorst_afk has joined #openstack-powervm | 20:59 | |
| *** chas has quit IRC | 21:02 | |
| *** thorst_afk has quit IRC | 21:03 | |
| openstackgerrit | Taylor Jakobson proposed openstack/nova-powervm master: Add support for file-backed ephemeral disk https://review.openstack.org/462248 | 21:12 |
| *** chas has joined #openstack-powervm | 21:18 | |
| *** chas has quit IRC | 21:22 | |
| *** esberglu has quit IRC | 21:24 | |
| *** thorst_afk has joined #openstack-powervm | 21:36 | |
| *** chas has joined #openstack-powervm | 21:39 | |
| *** esberglu has joined #openstack-powervm | 21:39 | |
| *** thorst_afk has quit IRC | 21:40 | |
| *** jwcroppe has quit IRC | 21:42 | |
| *** jwcroppe has joined #openstack-powervm | 21:43 | |
| *** chas has quit IRC | 21:43 | |
| *** jwcroppe has quit IRC | 21:47 | |
| efried | esberglu How's that --strict thing going? | 21:50 |
| efried | Need any help? | 21:50 |
| *** chas has joined #openstack-powervm | 21:59 | |
| *** tjakobs has quit IRC | 22:01 | |
| *** chas has quit IRC | 22:04 | |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: Consume HttpNotFound https://review.openstack.org/466768 | 22:09 |
| *** kylek3h has quit IRC | 22:12 | |
| esberglu | efried: I think it's working minus the UT. But not sure that it will fix our problems CI-wise | 22:19 |
| efried | why not? | 22:19 |
| esberglu | I haven't had a chance to run it through expect manually. So hard to say, because manually the steps are separated enough that I don't think it hitting us | 22:20 |
| *** chas has joined #openstack-powervm | 22:20 | |
| esberglu | So I wan't to confirm it works in the script | 22:21 |
| esberglu | By "not sure" I just mean I haven't tested it yet | 22:21 |
| efried | esberglu Okay, you'll have to change the script to say something like, `while ! nova-manage cell_v2 discover_hosts --strict; do sleep 1; done` | 22:22 |
| esberglu | Yep | 22:22 |
| *** chas has quit IRC | 22:25 | |
| *** tjakobs has joined #openstack-powervm | 22:26 | |
| *** jwcroppe has joined #openstack-powervm | 22:28 | |
| *** k0da has quit IRC | 22:34 | |
| *** chas has joined #openstack-powervm | 22:41 | |
| *** chas has quit IRC | 22:46 | |
| *** thorst_afk has joined #openstack-powervm | 23:00 | |
| *** chas has joined #openstack-powervm | 23:02 | |
| *** thorst_afk has quit IRC | 23:04 | |
| *** thorst_afk has joined #openstack-powervm | 23:05 | |
| *** chas has quit IRC | 23:06 | |
| *** thorst_afk has quit IRC | 23:09 | |
| *** jwcroppe has quit IRC | 23:18 | |
| *** chas has joined #openstack-powervm | 23:23 | |
| *** jpasqualetto has joined #openstack-powervm | 23:26 | |
| *** chas has quit IRC | 23:27 | |
| *** tjakobs has quit IRC | 23:32 | |
| *** jwcroppe has joined #openstack-powervm | 23:43 | |
| *** chas has joined #openstack-powervm | 23:44 | |
| *** chas has quit IRC | 23:48 | |
| *** jpasqualetto has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!