| *** wangqwsh has quit IRC | 00:09 | |
| *** wangqwsh has joined #openstack-powervm | 00:09 | |
| *** esberglu has joined #openstack-powervm | 00:34 | |
| *** thorst has joined #openstack-powervm | 00:36 | |
| *** thorst has quit IRC | 00:42 | |
| *** thorst has joined #openstack-powervm | 00:43 | |
| openstackgerrit | Drew Thorstensen proposed openstack/networking-powervm: Support VIOS CNAs https://review.openstack.org/344379 | 00:48 |
|---|---|---|
| openstackgerrit | Drew Thorstensen proposed openstack/networking-powervm: Support VIOS CNAs https://review.openstack.org/344379 | 00:49 |
| wangqwsh | thorst: ibmveth6 is the new NIC that you help attach to neo4, right? | 00:55 |
| *** esberglu has quit IRC | 00:59 | |
| thorst | wangqwsh: nope | 00:59 |
| thorst | its the en 0 | 00:59 |
| wangqwsh | enp96s0f0? | 01:00 |
| thorst | wangqwsh: yep. That sounds right. You have to use Ashana's controller though | 01:00 |
| thorst | its got the right VLANs to the powervm novalink | 01:01 |
| *** arnoldje has joined #openstack-powervm | 01:15 | |
| *** svenkat has quit IRC | 01:28 | |
| *** openstackgerrit has quit IRC | 01:33 | |
| *** openstackgerrit has joined #openstack-powervm | 01:34 | |
| *** tsjakobs has joined #openstack-powervm | 01:42 | |
| *** jwcroppe has quit IRC | 01:42 | |
| *** tsjakobs has quit IRC | 01:47 | |
| *** thorst has quit IRC | 01:48 | |
| *** thorst has joined #openstack-powervm | 01:48 | |
| *** thorst has quit IRC | 01:57 | |
| *** thorst has joined #openstack-powervm | 02:06 | |
| *** thorst has quit IRC | 02:06 | |
| *** svenkat has joined #openstack-powervm | 02:12 | |
| *** svenkat has quit IRC | 02:33 | |
| *** svenkat has joined #openstack-powervm | 02:35 | |
| *** svenkat has quit IRC | 02:49 | |
| *** svenkat has joined #openstack-powervm | 02:59 | |
| *** wangqwsh has quit IRC | 03:01 | |
| *** thorst has joined #openstack-powervm | 03:07 | |
| *** svenkat has quit IRC | 03:15 | |
| *** thorst has quit IRC | 03:16 | |
| *** seroyer has joined #openstack-powervm | 03:37 | |
| *** thorst has joined #openstack-powervm | 04:15 | |
| *** thorst has quit IRC | 04:21 | |
| *** seroyer has quit IRC | 04:24 | |
| *** kotra03 has joined #openstack-powervm | 04:49 | |
| *** seroyer has joined #openstack-powervm | 04:51 | |
| *** seroyer has quit IRC | 04:58 | |
| *** arnoldje has quit IRC | 05:09 | |
| *** thorst has joined #openstack-powervm | 05:20 | |
| *** thorst has quit IRC | 05:26 | |
| *** kotra03 has quit IRC | 05:58 | |
| *** thorst has joined #openstack-powervm | 06:01 | |
| *** thorst has quit IRC | 06:08 | |
| *** kotra03 has joined #openstack-powervm | 06:41 | |
| *** thorst has joined #openstack-powervm | 07:08 | |
| *** thorst has quit IRC | 07:14 | |
| *** adreznec has quit IRC | 07:35 | |
| *** adreznec has joined #openstack-powervm | 07:39 | |
| *** thorst has joined #openstack-powervm | 08:11 | |
| *** k0da has joined #openstack-powervm | 08:12 | |
| *** thorst has quit IRC | 08:18 | |
| *** thorst has joined #openstack-powervm | 09:16 | |
| *** thorst has quit IRC | 09:23 | |
| *** k0da has quit IRC | 09:52 | |
| *** k0da has joined #openstack-powervm | 10:00 | |
| *** thorst has joined #openstack-powervm | 10:23 | |
| *** thorst has quit IRC | 10:29 | |
| *** svenkat has joined #openstack-powervm | 11:05 | |
| *** thorst has joined #openstack-powervm | 11:26 | |
| *** thorst has quit IRC | 11:31 | |
| *** thorst has joined #openstack-powervm | 11:49 | |
| *** thorst has quit IRC | 11:56 | |
| *** thorst has joined #openstack-powervm | 12:02 | |
| *** Ashana has joined #openstack-powervm | 12:15 | |
| *** kotra03_ has joined #openstack-powervm | 12:37 | |
| *** kotra03 has quit IRC | 12:40 | |
| *** kotra03_ is now known as kotra03 | 12:40 | |
| *** seroyer has joined #openstack-powervm | 13:00 | |
| *** tblakeslee has joined #openstack-powervm | 13:05 | |
| *** mdrabe has joined #openstack-powervm | 13:11 | |
| *** seroyer has quit IRC | 13:14 | |
| *** seroyer has joined #openstack-powervm | 13:16 | |
| *** edmondsw has joined #openstack-powervm | 13:17 | |
| *** seroyer has quit IRC | 13:23 | |
| *** lmtaylor has joined #openstack-powervm | 13:28 | |
| *** apearson has joined #openstack-powervm | 13:38 | |
| *** efried has joined #openstack-powervm | 13:40 | |
| Ashana | adreznec: I just ran your repo-discovery.yml and I got an error saying | 13:49 |
| Ashana | The conditional check 'ansible_architecture not in \"{{ groups['repo_all'] | map('extract', hostvars, 'repo_master') | list }}\"' failed. | 13:49 |
| Ashana | adreznec: for that conditional should there be brackets around ansible architecture | 13:50 |
| adreznec | Ashana: Are you just running repo-discovery on it's own? Or as part of the whole repo-build flow? | 13:51 |
| -openstackstatus- NOTICE: Gerrit is being restarted now to relieve performance degradation | 13:53 | |
| Ashana | on its own that was on the list of Qing Wu playbooks that he wanted me to run | 13:54 |
| thorst | adreznec: Qing Wu ran them all as independent playbooks | 13:54 |
| adreznec | Oh, I hadn't run it that way. I bet it's an issue with the fact for repo_master not being set then | 13:55 |
| adreznec | Because that gets set to false when the repo_server setup happens | 13:55 |
| adreznec | Running them all independently is a weird use case | 13:55 |
| adreznec | For now just try running all of setup-infrastructure and see if that fixes it | 13:56 |
| adreznec | If so I'll try and find a way to make the playbooks independent again | 13:56 |
| Ashana | Ok I will try that thank you! | 13:59 |
| *** lmtaylor has quit IRC | 14:03 | |
| *** jwcroppe has joined #openstack-powervm | 14:03 | |
| *** seroyer has joined #openstack-powervm | 14:05 | |
| *** arnoldje has joined #openstack-powervm | 14:14 | |
| efried | thorst, mdrabe and I were discussing the bug about mappings not being cleaned up on destroy_evacuated. Wanted to get your input on an approach we considered. | 14:15 |
| *** dwayne has joined #openstack-powervm | 14:16 | |
| efried | My take was: we're being real gentle and careful about how we clean up mappings on destroy - generally feeding everything through Disconnect etc. Is there some reason we shouldn't just feed everything through the LPAR-specific scrub on destroy? | 14:17 |
| *** tsjakobs has joined #openstack-powervm | 14:19 | |
| *** lmtaylor1 has joined #openstack-powervm | 14:20 | |
| *** mdrabe has quit IRC | 14:57 | |
| thorst | efried: I don't really think there would be an issue with that...except isn't the LPAR there? | 14:57 |
| thorst | isn't the LPAR scrub generally used when we know the LPAR, but before mappings are attached to it? | 14:57 |
| efried | Correct on both counts. | 14:58 |
| thorst | and doesn't the destroy really try to do what the scrub does? | 14:58 |
| efried | Not exactly, no. Disconnect is gentler (and slower). | 14:58 |
| efried | It involves discover_hdisk, for one thing, which I don't think we need. | 14:58 |
| efried | ever. | 14:58 |
| efried | But I could be wrong - mdrabe will have to investigate to verify. | 14:59 |
| thorst | well, wouldn't we need that to clear out the hdisk | 14:59 |
| efried | The scrubber removes the storage (if unused) as well, in a post-execute. I think that's better for this situation because, if it fails, the mappings are gone anyway. | 14:59 |
| *** kotra03 has quit IRC | 14:59 | |
| openstackgerrit | Taylor Jakobson proposed openstack/nova-powervm: [WIP] Add iscsi support https://review.openstack.org/346920 | 14:59 |
| efried | which would resolve this problem. | 14:59 |
| *** esberglu has joined #openstack-powervm | 14:59 | |
| thorst | hmm | 15:01 |
| thorst | implicitly on the covers I don't see anything wrong | 15:01 |
| thorst | how hard is it to write up and let me stew on in gerrit? | 15:01 |
| efried | Guess that's a mdrabe question, and he seems to be afk at the moment. | 15:02 |
| *** mdrabe has joined #openstack-powervm | 15:05 | |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm: Rebase on pypowervm.tasks.hdisk refactor https://review.openstack.org/347889 | 15:08 |
| thorst | efried: I'm not digging that the pypowervm change requires us to update nova-powervm | 15:09 |
| thorst | that smells bad. | 15:09 |
| efried | thorst, did you look at the change I just posted? | 15:09 |
| thorst | whoops, just the name | 15:09 |
| thorst | :-D | 15:09 |
| thorst | sloppy | 15:09 |
| efried | Basically, we were doing something semi-wrong in our UT anyway. | 15:10 |
| thorst | by going to lua_recovery instead of discover_hdisk | 15:10 |
| thorst | ? | 15:10 |
| efried | Yeah. I think what happened is that we lazily neglected to change those mocks after we wrapped lua_recovery in discover_hdisk (to put in that retry) | 15:10 |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm: Rebase on pypowervm.tasks.hdisk refactor https://review.openstack.org/347889 | 15:14 |
| *** dwayne has quit IRC | 15:14 | |
| thorst | svenkat: https://review.openstack.org/#/c/344379/ please? | 15:15 |
| tsjakobs | thorst: do you know if there is a good way to get the udid of a device from the device_name in nova-powervm? | 15:28 |
| tsjakobs | the iscsi discovery only returns the dev_name | 15:28 |
| efried | tsjakobs, what's the schema type of the device? | 15:33 |
| efried | Does it show up as a PhysicalVolume? | 15:33 |
| tsjakobs | ya, i think so | 15:33 |
| thorst | tsjakobs: is it not passed in as part of the connection_info? | 15:33 |
| thorst | (since this is a cinder volume) | 15:33 |
| tsjakobs | the connection_info has host_ip and IQN. Then we do the iscsi discovery/login then a query to get the dev_name | 15:34 |
| tsjakobs | best i can tell | 15:34 |
| efried | It sounds like you will need to GET the VirtualIOServer with the ViosStorage extended attribute group, then loop through the PhysicalVolumes therein looking for the one with a matching dev_name. | 15:35 |
| efried | What do you need to do with the UDID? | 15:35 |
| tsjakobs | I'm using it in the connect/disconnect of iscsi. It looks like I could probably get away with just the device_name, it just seemed like the refactor i did worked better | 15:37 |
| tsjakobs | if the udid was easily available | 15:38 |
| efried | tsjakobs, I think it would be a new thing if we were mapping a PV directly to a VM. thorst, keep me honest here. | 15:49 |
| efried | damn, that CAN'T be right. Looking more... | 15:49 |
| *** dwayne has joined #openstack-powervm | 15:51 | |
| efried | Whew. | 15:52 |
| efried | Okay, tsjakobs, for the existing disk type, we're also just using the device name to create the mappings. | 15:53 |
| efried | See nova_powervm/virt/powervm/volume/vscsi.py:563 | 15:54 |
| *** jwcroppe has quit IRC | 15:55 | |
| tsjakobs | what about nova_powervm/virt/powervm/volume/vscsi.py:310 | 15:57 |
| tsjakobs | would i have to make a dev_name_to_scsi_mapping? | 15:57 |
| tpeoples | fyi, https://github.com/openstack/nova/commit/3b62698235364057ec0c6811cc89ac85511876d2 broke migration for the nova-powervm driver | 15:57 |
| tpeoples | efried: fyi | 15:57 |
| tsjakobs | efried: ^^^ | 15:57 |
| thorst | tpeoples: is that just a rename of a method? | 15:57 |
| thorst | nothing more severe? | 15:57 |
| tpeoples | yeah, so easy fix | 15:57 |
| thorst | thank goodness. | 15:57 |
| thorst | thx dude. | 15:57 |
| thorst | mind making a bug for it? | 15:57 |
| tpeoples | but it'll blow up | 15:57 |
| tpeoples | alrighty | 15:58 |
| thorst | thx...easy fix. | 15:58 |
| thorst | efried tsjakobs: stuck in meeting...haven't read the previous stuff. | 15:58 |
| adreznec | thorst: esberglu: That reminds me... once we get the CI stablized, we should probably look at getting the multi-node live migration jobs working in the CI | 15:59 |
| adreznec | Doesn't have to be for every run on Nova, but even just a periodic job we set up somewhere or just on nova-powervm runs would give us more heads up in this area | 16:00 |
| thorst | adreznec: yeppers | 16:00 |
| adreznec | Something to keep on the back burner for after our CI is stable | 16:01 |
| adreznec | So next week, right esberglu? | 16:01 |
| thorst | your optimism knows no bounds | 16:01 |
| *** thorst is now known as thorst_afk | 16:01 | |
| esberglu | adreznec: The only thing that is really still holding us back is the slow instance build times | 16:02 |
| efried | tpeoples, fix is ready. Got that bug report yet? | 16:07 |
| tpeoples | efried: https://bugs.launchpad.net/nova-powervm/+bug/1606981 | 16:07 |
| openstack | Launchpad bug 1606981 in nova-powervm "Driver does not implement cleanup_live_migration_destination_check method" [Undecided,New] | 16:07 |
| efried | thx | 16:07 |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm: Refactor: cleanup_live_migration_destination_check https://review.openstack.org/347928 | 16:08 |
| efried | tpeoples thorst ^^ | 16:09 |
| *** k0da has quit IRC | 16:10 | |
| *** jwcroppe has joined #openstack-powervm | 16:13 | |
| adreznec | esberglu: How slow is slow? | 16:13 |
| esberglu | I’ve seen as bad as 500 seconds | 16:14 |
| thorst_afk | esberglu: with the new code that tells us each step...where is it spending the majority of its time now? | 16:25 |
| *** thorst_afk is now known as thorst | 16:27 | |
| esberglu | thorst_afk: Task plug_vifs completed in 155 seconds | 16:31 |
| thorst | ugh...that's awful | 16:36 |
| thorst | OK - time to fix that. | 16:36 |
| adreznec | esberglu: What was the rest of the breakdown? | 16:38 |
| adreznec | Just trying to wrap my head around a 500 second spawn... | 16:39 |
| *** Ashana has quit IRC | 16:39 | |
| *** Ashana has joined #openstack-powervm | 16:40 | |
| *** Ashana has quit IRC | 16:44 | |
| *** Ashana has joined #openstack-powervm | 16:47 | |
| esberglu | adreznec: This one was only 384 seconds! crt_disk_from_img took 43, cfg_drive took 23. Those are the big ones | 16:51 |
| *** Ashana has quit IRC | 16:51 | |
| *** Ashana has joined #openstack-powervm | 16:53 | |
| *** Ashana has quit IRC | 16:58 | |
| thorst | esberglu: link? | 17:09 |
| esberglu | http://184.172.12.213/36/338736/3/silent/nova-pvm-dsvm-tempest-full/7d3dbea | 17:10 |
| esberglu | d872d03a-eed4-404c-80d4-9b35af942447 | 17:11 |
| esberglu | Filter by that if you want the build I was looking at | 17:11 |
| thorst | I was just looking at all the instances | 17:12 |
| thorst | so in general, we have two time sinks | 17:12 |
| thorst | plugging vifs for some reason | 17:12 |
| thorst | I'll sign up to take that | 17:12 |
| thorst | then crt_disk_from_img and connect_disk are wild | 17:13 |
| thorst | efried: Maybe that's one you take and work with clbush on? | 17:13 |
| thorst | esberglu: somethings wrong with the networking-powervm agent...I'm not getting any debug :-( | 17:15 |
| thorst | I thought we had enabled that? | 17:16 |
| esberglu | thorst: Crap. I just had it on a local branch and forgot to add it back in last time I redeployed. | 17:17 |
| thorst | can we get that added and then I can actually debug that? | 17:17 |
| esberglu | Yep | 17:17 |
| thorst | rockin | 17:18 |
| *** Ashana has joined #openstack-powervm | 17:21 | |
| esberglu | thorst: Is that something we want to permanently enable? | 17:22 |
| *** Ashana_ has joined #openstack-powervm | 17:23 | |
| *** Ashana has quit IRC | 17:26 | |
| *** Ashana_ has quit IRC | 17:28 | |
| *** Ashana has joined #openstack-powervm | 17:29 | |
| *** Ashana has quit IRC | 17:34 | |
| *** Ashana has joined #openstack-powervm | 17:34 | |
| Ashana | adreznec: I ran the setup-infrastructure and there was an error with the repo_package_sync.yml I got this error failed: [infra01_repo_container-e0986c2a] (item=/var/www/repo/pools) => {"changed": true, "cmd": ["rsync", "--delay-updates", "-F", "--compress", "--archive", "--rsh", "ssh -l nginx -o stricthostkeychecking=no", "--out-format=<<CHANGED>>%i %n%L", "/var/www/repo/pools", "nginx@compute01_repo_container-1b7b4616:/ | 17:42 |
| Ashana | var/www/repo"] | 17:42 |
| thorst | esberglu: Yeah, perm | 17:52 |
| thorst | Ashana: does the compute node have an ssh key pair to the source? | 18:01 |
| thorst | and adreznec does it? | 18:01 |
| adreznec | Ashana: Ok.... what was the actual error? | 18:02 |
| adreznec | That just says it failed | 18:02 |
| adreznec | You'll probably have to run again with -vvv | 18:02 |
| adreznec | and yes thorst, the repo containers all have SSH keys shared between them | 18:02 |
| thorst | adreznec: do the playbooks set that up | 18:03 |
| adreznec | Yes | 18:03 |
| thorst | or ahead of time is that done? | 18:03 |
| adreznec | Done in the playbooks | 18:03 |
| thorst | goooood | 18:03 |
| adreznec | I'm just using the existing ones put in place during repo server setup | 18:03 |
| openstackgerrit | Merged openstack/networking-powervm: Support VIOS CNAs https://review.openstack.org/344379 | 18:29 |
| efried | thorst, are you saying crt_disk_from_img and connect_disk are performing poorly? | 19:36 |
| *** Ashana has quit IRC | 19:41 | |
| openstackgerrit | Drew Thorstensen proposed openstack/networking-powervm: Simplify host_uuid and gets https://review.openstack.org/344399 | 19:45 |
| thorst | efried: yeppers | 19:45 |
| thorst | sorry for delay | 19:45 |
| efried | thorst, What's the context here? CI run? | 19:47 |
| thorst | yes | 19:47 |
| thorst | so we're seeing two major slow downs. | 19:47 |
| thorst | 1) Networking -> I'm on it. esberglu is increasing log levels and then I get to figure it out | 19:47 |
| thorst | 2) crt_disk_from_image and connect_disk. Hoping you, esberglu and clbush can dive a little bit into? We need to figure out if only solution is to decrease SSP size... | 19:48 |
| *** Ashana has joined #openstack-powervm | 19:48 | |
| efried | thorst, we're seeing a slowdown from previous runs, or we're just now finding a bottleneck that has always existed? | 19:49 |
| thorst | efried: so we opened it up to all of nova | 19:49 |
| thorst | so much more throughput | 19:49 |
| thorst | so its a bottleneck that has always existed that we just never percieved | 19:50 |
| thorst | we were able to open up to all of nova after we solved other bottlenecks | 19:50 |
| efried | So the bottleneck must be at the SSP, because that's the only shared component? | 19:50 |
| thorst | likely....I think we just need to get clbush and changh on it...and get them whatever data they need. | 19:51 |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm: WIP/Proposal: Simplify storage deletion on destroy https://review.openstack.org/348014 | 19:53 |
| *** Ashana has quit IRC | 19:53 | |
| *** Ashana has joined #openstack-powervm | 19:54 | |
| *** Ashana has quit IRC | 19:55 | |
| *** Ashana has joined #openstack-powervm | 19:56 | |
| efried | thorst, mdrabe - take a look at the approach suggested by ^^ | 19:56 |
| efried | And try to break it. | 19:56 |
| Ashana | @thorst just got back and I ran it with -vvv but it still | 19:56 |
| Ashana | @adreznec | 19:56 |
| Ashana | adreznec: it says [WARNING]: Consider using synchronize module rather than running rsync then it was on the infra01_repo_container-e0986c2a | 19:57 |
| adreznec | Yeah | 19:58 |
| adreznec | That's not the error though, that's just a warning | 19:58 |
| adreznec | Do you want to shoot me the full log in a pastebin | 19:58 |
| adreznec | thorst: https://etherpad.openstack.org/p/osa-midcycle-newton | 19:59 |
| Ashana | Yes I will | 20:02 |
| openstackgerrit | Merged openstack/nova-powervm: Refactor: cleanup_live_migration_destination_check https://review.openstack.org/347928 | 20:10 |
| *** miltonm has quit IRC | 20:31 | |
| *** thorst has quit IRC | 20:55 | |
| *** svenkat has quit IRC | 20:58 | |
| *** k0da has joined #openstack-powervm | 20:58 | |
| *** Ashana has quit IRC | 21:05 | |
| *** seroyer has quit IRC | 21:07 | |
| *** lmtaylor1 has left #openstack-powervm | 21:27 | |
| *** apearson has quit IRC | 21:46 | |
| *** esberglu has quit IRC | 21:55 | |
| *** tblakeslee has quit IRC | 21:59 | |
| *** mdrabe has quit IRC | 22:03 | |
| *** efried has quit IRC | 22:04 | |
| *** svenkat has joined #openstack-powervm | 22:10 | |
| *** tsjakobs has quit IRC | 22:13 | |
| *** edmondsw has quit IRC | 22:25 | |
| *** arnoldje has quit IRC | 22:25 | |
| *** svenkat has quit IRC | 22:27 | |
| *** k0da has quit IRC | 22:28 | |
| *** svenkat has joined #openstack-powervm | 22:32 | |
| *** jwcroppe has quit IRC | 22:40 | |
| *** jwcroppe has joined #openstack-powervm | 22:41 | |
| *** jwcroppe has quit IRC | 22:45 | |
| *** thorst_ has joined #openstack-powervm | 22:47 | |
| *** seroyer has joined #openstack-powervm | 22:54 | |
| *** thorst_ has quit IRC | 22:55 | |
| *** thorst_ has joined #openstack-powervm | 22:56 | |
| *** svenkat has quit IRC | 23:03 | |
| *** thorst_ has quit IRC | 23:04 | |
| *** seroyer has quit IRC | 23:18 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!