| *** k_stev has joined #openstack-ansible | 00:02 | |
| *** jaypipes has quit IRC | 00:04 | |
| cloudnull | Jwitko you may have to remove / change the force config drive option everywhere and restart the various nova services. | 00:08 |
|---|---|---|
| cloudnull | I'm out and about on mobile so I can't specifically look but I think there's an override for it. | 00:08 |
| cloudnull | is_nova/defaults/main.yml | 00:09 |
| cloudnull | *os_nova | 00:09 |
| jwitko | aye, looks like its here "roles/os_nova/defaults/main.yml" | 00:09 |
| cloudnull | Otherwise you can reconfigure it with the config template overrides. | 00:10 |
| jwitko | cloudnull, I'm seeing different things online though. Some saying set it to "False" some saying to remove the line entirely | 00:10 |
| cloudnull | I'd set it to false. | 00:10 |
| jwitko | cloudnull, any idea how to use OSAD to only apply that template and restart appropriate services across the board? | 00:11 |
| jwitko | looks like tag "nova-config" | 00:12 |
| cloudnull | Set the override. Then run : openstack-ansible os-nova-install.yml --tags nova-config | 00:12 |
| jwitko | thanks | 00:12 |
| cloudnull | You got it. ;-) sorry slow to type on the mobile keyboard. | 00:13 |
| jwitko | no problem! i really appreciate your help | 00:13 |
| cloudnull | Anytime. I'm off bbl let us know how it goes | 00:14 |
| jwitko | cloudnull, one question before you go. for every instance that is booted already I'll have to do a hard-reset on it to come back up without the drive attached correct? | 00:15 |
| jwitko | or can I simply just en-mass issue the --eject command | 00:15 |
| *** k_stev has quit IRC | 00:24 | |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Updated the repo-build process https://review.openstack.org/230716 | 00:44 |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Implement shippable venvs https://review.openstack.org/236183 | 00:50 |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Always drop lxc_net bridge configuration https://review.openstack.org/237266 | 00:52 |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Stop configure_diskspace from using small partitions https://review.openstack.org/216880 | 01:00 |
| *** kerwin_bai has joined #openstack-ansible | 01:00 | |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Load glance metadata definitions https://review.openstack.org/235425 | 01:02 |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Removed deprecated vif driver https://review.openstack.org/237362 | 01:07 |
| jwitko | Hey All, Can anyone please help me work out a live migration issue? The full debug error is here http://paste.openstack.org/show/476794/ -- but the basics are that its saying the compute node is not on shared storage, but it is! | 01:15 |
| jwitko | I have disabled all config drive across all nova.conf's and restarted all appropriate services | 01:15 |
| *** BjoernT is now known as Bjoern_zZzZzZzZ | 01:23 | |
| *** Bjoern_zZzZzZzZ is now known as BjoernT | 01:28 | |
| *** BjoernT is now known as Bjoern_zZzZzZzZ | 01:29 | |
| *** Bjoern_zZzZzZzZ is now known as BjoernT | 01:30 | |
| neillc | cloudnull: thanks for rebase. I promise to do something about that patchset as soon as I finish my madatory work video watching :/ | 01:30 |
| *** BjoernT has quit IRC | 01:34 | |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Updated the repo-build process https://review.openstack.org/230716 | 01:38 |
| *** markvoelker has joined #openstack-ansible | 01:40 | |
| *** greg_a has joined #openstack-ansible | 01:47 | |
| cloudnull | Jwitko sorry wasn't watching irc. You could do mass virsh ejects. Ansible to shell commands could help make that go. | 02:03 |
| jwitko | cloudnull, so Sam-I-Am was discussing with me in #openstack | 02:03 |
| cloudnull | Also is it still happening with it all disabled ? | 02:03 |
| jwitko | He believes it is because my /var/lib/nova/instances is not a shared directory | 02:03 |
| jwitko | Yes its still happening with config drive disabled accross the board | 02:04 |
| cloudnull | Not a shared dir? | 02:04 |
| jwitko | Yes, its not shared across compute hosts. AKA its not an NFS mount | 02:04 |
| cloudnull | I don't think that's needed. | 02:04 |
| jwitko | all of my VMs live on the NetApp, but the libvirt metadata lives in /var/lib/nova/instances | 02:05 |
| cloudnull | One sec | 02:05 |
| jwitko | ok | 02:05 |
| Sam-I-Am | cloudnull: i'm pretty sure thats a requirement for live migration | 02:05 |
| cloudnull | jwitko: are you running the live-migrate from the CLI or horizon ? | 02:12 |
| cloudnull | i remember reading this a while back https://blog.codecentric.de/en/2015/03/true-kvm-live-migration-openstack-icehouse-ceph-based-vm-storage/ which was using netapp for livemigrate | 02:13 |
| cloudnull | and they patched nova to skip the shared storage check | 02:13 |
| cloudnull | however i seem to remember passing --block_migrate would skip that check | 02:14 |
| *** devlaps has quit IRC | 02:15 | |
| jwitko | cloudnull, I have been attempting to run it from horizon | 02:15 |
| jwitko | cloudnull, I attempted to use the "block migrate" option on horizon as I saw that in a stack exchange article | 02:15 |
| jwitko | but it still complains about config drive! | 02:15 |
| cloudnull | even with it all disabled in config ? | 02:17 |
| jwitko | yes, one second I'll paste the stack debug | 02:18 |
| jwitko | http://paste.openstack.org/show/476798/ | 02:19 |
| jwitko | 2015-10-19 22:17:08.011 7724 TRACE oslo_messaging.rpc.dispatcher NoLiveMigrationForConfigDriveInLibVirt: Live migration of instances with config drives is not supported in libvirt unless libvirt instance path and drive data is shared across compute nodes. | 02:19 |
| jwitko | but config drive is disabled EVERYWHERE | 02:19 |
| jwitko | and all appropriate nova services were restarted | 02:19 |
| cloudnull | so with config drive disabled i still http://cdn.pasteraw.com/n1z5mp8hziqbdxpj6v6hz63hxt13npa on my instances | 02:27 |
| jwitko | aye | 02:28 |
| jwitko | i still see that as well | 02:28 |
| jwitko | and am still able to --eject | 02:28 |
| cloudnull | building another instance that section is gone | 02:28 |
| jwitko | ah interesting | 02:28 |
| jwitko | i wonder if i "rebuild instance" | 02:28 |
| cloudnull | so it'd be interesting to see if you can remove that section , restart the instance , and then live migrate | 02:29 |
| *** devlaps has joined #openstack-ansible | 02:29 | |
| jwitko | I tried to remove it before | 02:29 |
| jwitko | it added it back lol | 02:29 |
| jwitko | maybe I did it wrong though | 02:29 |
| jwitko | checking now | 02:29 |
| cloudnull | maybe remove it restart and migrate with the block option ? | 02:30 |
| jwitko | instance is stopped | 02:30 |
| jwitko | i just removed | 02:30 |
| jwitko | starting instance now | 02:30 |
| jwitko | it adds it back :\ | 02:30 |
| jwitko | i wonder what would happen if i made the file immutable | 02:31 |
| jwitko | restarting the instance errors as "permission denied" if i make the file immutable | 02:33 |
| jwitko | why the hell is openstack forcing the cd-rom in there | 02:33 |
| cloudnull | if you boot a test instance does it still do the same ? | 02:35 |
| bgmccollum | jwitko: probably cause it was initially booted with the config drive...wild guess...try updating the instance in the database to remove the config drive...then see if it regenerates the libvirt.xml | 02:35 |
| jwitko | bgmccollum, what db/table contains that info, do you know? | 02:36 |
| bgmccollum | not offhand | 02:36 |
| jwitko | cloudnull, no it does not | 02:36 |
| jwitko | cloudnull, new instances look to be without it. let me try to live migrate the new instance | 02:37 |
| cloudnull | bgmccollum: jwitko: nova.instances | 02:37 |
| cloudnull | config_drive: | 02:37 |
| jwitko | lol cloudnull, you're going to love this | 02:38 |
| cloudnull | update instances set config_drive="True"; | 02:38 |
| bgmccollum | False ? | 02:38 |
| cloudnull | that would do all | 02:38 |
| cloudnull | yes False | 02:39 |
| cloudnull | update instances set config_drive="False"; # will disable it | 02:39 |
| bgmccollum | and it sounds like nova-compute regenerates the libvirt.xml on state changes or something? | 02:40 |
| *** devlaps has quit IRC | 02:40 | |
| cloudnull | actually it looks like -- update instances set config_drive=""; | 02:40 |
| cloudnull | building a new instance without config drive leaves the value blank | 02:40 |
| bgmccollum | jwitko: dont leave us hanging... | 02:41 |
| jwitko | so | 02:41 |
| jwitko | if I live migrate the new instance without the cdrom | 02:41 |
| jwitko | horizon reports success and that its starting the process | 02:41 |
| jwitko | but then never finishes | 02:41 |
| jwitko | I don't see anything reporting errors though, pasting a stack trace now | 02:41 |
| jwitko | http://paste.openstack.org/show/476800/ | 02:42 |
| jwitko | if I do it with "block migrate" option selected | 02:42 |
| jwitko | I get this error, which makes no damn sense | 02:42 |
| jwitko | InvalidLocalStorage: oss-comp07 is not on local storage: Block migration can not be used with shared storage. | 02:42 |
| jwitko | to caller | 02:42 |
| jwitko | raise exception.InvalidLocalStorage(reason=reason, path=source)\n\nInvalidLocalStorage: oss-comp07 is not on local storage: Block migration can not be used with shared storage. | 02:42 |
| jwitko | so on one hand its error because I'm NOT using shared storage | 02:43 |
| *** sdake has quit IRC | 02:43 | |
| jwitko | and now it errors because i AM using shared storage! | 02:43 |
| bgmccollum | jwitko: block migrate literally SCPs the vm between hosts...so its saying you can't do that with shared storage (its shouldn't have to move any bits) | 02:44 |
| jwitko | bgmccollum, wouldn't it want to move the libvirt meta-data? | 02:44 |
| jwitko | that is stored locally on the compute node | 02:44 |
| bgmccollum | it should regenerate that from the DB | 02:45 |
| jwitko | oh wow, awesome | 02:45 |
| jwitko | so then lets ignore block migrate | 02:45 |
| jwitko | and ponder why the live migrate option by itself doesn't work :) | 02:45 |
| Sam-I-Am | jwitko: all of this sounds like it needs better docs :) | 02:45 |
| Sam-I-Am | the ins-and-outs of migration | 02:46 |
| jwitko | Sam-I-Am, so with the above from bgmccollum, the libvirt meta-data being regen'd from DB | 02:46 |
| jwitko | I guess the shared drive is not necessary ? | 02:46 |
| Sam-I-Am | in that case, i guess not | 02:46 |
| Sam-I-Am | there's two ways to skin the cat | 02:46 |
| bgmccollum | jwitko: your initial issue is the VM was on shared storage, but the config drive was on local storage...which is why it wouldn't live migrate | 02:47 |
| jwitko | bgmccollum, makes sense | 02:47 |
| bgmccollum | if the VM and config drive were on shared storage, it should have worked | 02:48 |
| jwitko | however bgmccollum, now I have a VM with no config drive | 02:48 |
| jwitko | and lvie migrate is still not working | 02:48 |
| bgmccollum | you said you're using boot from volume | 02:48 |
| jwitko | bgmccollum, yes I boot from a volume snapshot into a new volume for all VMs | 02:48 |
| bgmccollum | ok...and after updating the DB to cloud out config drive for that instance...did you shutdown / start the instance (which you indicated was regenerating your libvirt.xml)...and verify there is not config drive configured... | 02:49 |
| bgmccollum | freudian...cloud out...clear out | 02:50 |
| jwitko | bgmccollum, I created a new instance. "Test1", and this instance booted without a cdrom at all | 02:51 |
| jwitko | I then selected to live migrate this instance | 02:51 |
| jwitko | and again, I see the following | 02:53 |
| jwitko | Returning exception Live migration of instances with config drives is not supported in libvirt unless libvirt instance path and drive data is shared across compute nodes. to caller | 02:53 |
| jwitko | if i go into the libvirt.xml of that instance | 02:53 |
| jwitko | there is only one <disk block, and its iSCSI for the source | 02:53 |
| jwitko | <disk type="block" device="disk"> | 02:53 |
| jwitko | <driver name="qemu" type="raw" cache="none"/> | 02:53 |
| jwitko | <source dev="/dev/disk/by-path/ip-10.1.2.17:3260-iscsi-iqn.1992-08.com.netapp:sn.b81c6744a63611e38425123478563412:vs.3-lun-0"/> | 02:53 |
| jwitko | <target bus="virtio" dev="vda"/> | 02:53 |
| jwitko | <serial>fb845640-f430-4a4b-a517-66108691832d</serial> | 02:53 |
| jwitko | </disk> | 02:53 |
| bgmccollum | can you paste the contents of your instance directory? | 02:54 |
| jwitko | Yes | 02:55 |
| jwitko | you want /var/liv/nova/instances or /var/liv/nova/instances/<instance>/ ? | 02:55 |
| bgmccollum | the 2nd | 02:56 |
| jwitko | liveops@oss-comp07:~$ ls -al /var/lib/nova/instances/95358532-5633-4cb2-8b6a-d6e64516bba8/ | 02:56 |
| jwitko | total 16 | 02:56 |
| jwitko | drwxr-xr-x 2 nova nova 4096 Oct 19 22:35 . | 02:56 |
| jwitko | drwxr-xr-x 4 nova nova 4096 Oct 19 22:35 .. | 02:56 |
| jwitko | -rw-rw---- 1 libvirt-qemu kvm 1 Oct 19 22:36 console.log | 02:56 |
| jwitko | -rw-r--r-- 1 nova nova 2704 Oct 19 22:35 libvirt.xml | 02:56 |
| bgmccollum | so...thats a terrible error...cause there is indeed no config drive. | 02:58 |
| jwitko | Would you like me to paste the full error log > | 02:58 |
| jwitko | ? | 02:58 |
| Sam-I-Am | i wonder if any of this was addressed in liberty | 02:59 |
| bgmccollum | sounds like /var/lib/nova/instance does in fact need to be shared storage...probably just to preserve the console log, since the libvirt.xml could easily be regenrated | 03:00 |
| jwitko | bgmccollum, anything necessary in that console.log ? | 03:01 |
| jwitko | if I was to move it to a tmp directory, would that hurt anything? | 03:02 |
| jwitko | and would it then allow us to see if thats the issue ? | 03:02 |
| bgmccollum | either that or its not smart enough to see its was booted from a volume...just looks at the instance dir and see its not shared | 03:02 |
| bgmccollum | the console log is just the serial console | 03:02 |
| jwitko | bgmccollum, so here is something interesting... I just recreated one of the VMs from its heat stack | 03:04 |
| jwitko | and it was created to the same hypervisor as the test machine, and it comes up with a cdrom | 03:05 |
| cloudnull | ohh. maybe config_drive is enabled via heat when booting the vm ? | 03:05 |
| jwitko | yup. just looked | 03:05 |
| jwitko | config_drive: true | 03:05 |
| jwitko | in all my heat templates | 03:05 |
| jwitko | a good find, but the VM i manually created wouldn't have this issue | 03:06 |
| jwitko | cloudnull, bgmccollum, Sam-I-Am, so How can I test this? Shut down this test VM, create a nfs mount to my netapp, copy over all of /var/lib/nova/instances, mount the nfs over /var/lib/nova/instances, and then also mount that on the 'receiving' compute node and try to migrate? | 03:07 |
| bgmccollum | jwitko: yes | 03:07 |
| cloudnull | that might work | 03:07 |
| jwitko | ok will report back shortly | 03:09 |
| bgmccollum | cloudnull: the cinder udev whitespace issue was fixed in the new upgrade script, and looks like it will be addressed in 10.2.0 in the cinder role...should the same fix be applied to the old upgrade script, or just wait till 10.2.0 is cut...upgrade to that...then 11.0? | 03:10 |
| bgmccollum | im guessing its going to be "dont touch anything" for a while... | 03:11 |
| cloudnull | 10.2 might never happen | 03:11 |
| bgmccollum | sad panda | 03:11 |
| bgmccollum | so side port to old upgrade script...or out of band patch | 03:12 |
| cloudnull | but in the new script it fixes that issue when cinder-volume is in containers | 03:12 |
| bgmccollum | yeah...but RPC | 03:12 |
| cloudnull | you can fix it manually by calling the play | 03:12 |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/kilo/doc/source/upgrade-guide/process.rst | 03:12 |
| cloudnull | exists regardless of rpc | 03:12 |
| cloudnull | openstack-ansible ${UPGRADE_PLAYBOOKS}/cinder-adjustments.yml | 03:13 |
| cloudnull | you can run the broken play and when that doesn't work you can follow the intree process docs | 03:13 |
| cloudnull | *broken script | 03:13 |
| bgmccollum | yup...i can make it work...its a matter of what *should* we do...fine line between rpc docs, osa docs, and creative off script interpretation | 03:14 |
| bgmccollum | ;) | 03:15 |
| cloudnull | rpc needs to ditch the old script because its prone to failure | 03:15 |
| cloudnull | new script has same functionality but has the ability to recover | 03:16 |
| bgmccollum | im in agreement...but i have to follow the RPC way for the upgrade class | 03:16 |
| cloudnull | thats fair . idk how to fix that ... | 03:16 |
| bgmccollum | i was happy to see the new upgrade script... | 03:16 |
| cloudnull | welcome to the lobbying game | 03:17 |
| bgmccollum | i guess its a moot point, because of the container -> metal migration that supposed to happen | 03:18 |
| bgmccollum | i was just being lazy in my upgrade testing | 03:18 |
| cloudnull | migrating cinder-volume to the host from a container makes the most sense for long term supportability when using cinder+lvm | 03:19 |
| cloudnull | however if using netapp, vnx cinder-volume in a container makes better sense | 03:19 |
| cloudnull | so the upgrade should be able to support both cases | 03:20 |
| bgmccollum | ill throw up a review and see how much hell i get | 03:20 |
| cloudnull | the good thing is that the new script is available and we simply need to convince rpc that the old one is bad | 03:21 |
| cloudnull | not to mention as is all of the documentation. | 03:22 |
| cloudnull | rpc is using this https://github.com/rcbops/rpc-openstack/commit/304a63d8b69f33410ffc05d1c7f2286433a790eb | 03:23 |
| * cloudnull shakes head | 03:23 | |
| cloudnull | bgmccollum: http://upgrade-test.cloudnull.io/ | 03:23 |
| jwitko | oh boy | 03:23 |
| jwitko | the VM won't even power on ;\ | 03:23 |
| cloudnull | these are the results for the upgrade tests , testing the last 5 juno tags to the latest kilo tag | 03:23 |
| cloudnull | all of the failues are the old script, the success jobs are upgrades from the latest juno to the kilo branch (which uses the new script) | 03:24 |
| jwitko | 2015-10-19 23:23:37.290 9927 TRACE oslo_messaging.rpc.dispatcher NovaException: iSCSI device not found at /dev/disk/by-path/ip-10.1.2.17:3260-iscsi-iqn.1992-08.com.netapp:sn.b81c6744a63611e38425123478563412:vs.3-lun-2 | 03:24 |
| cloudnull | jwitko: is that with nfs for /var/lib/nova/instances | 03:25 |
| jwitko | aye | 03:25 |
| cloudnull | + netapp iscsi vm ? | 03:25 |
| jwitko | aye | 03:25 |
| cloudnull | :( | 03:25 |
| jwitko | i think i see the issue | 03:25 |
| jwitko | permissions got messed up in the nfs dir | 03:25 |
| jwitko | and it is not allowing me to change them | 03:25 |
| jwitko | also seeing | 03:28 |
| jwitko | 2015-10-19 23:28:28.004 9927 WARNING nova.virt.libvirt.volume [req-78a610fa-169d-48a9-98a5-342448b4f51d 99903c87e9ab4bb3b29bc51b0943330c 7f43cc16e3714f1c84e525455ddb7495 - - -] ISCSI volume not yet found at: vda. Will rescan & retry. Try number: 3 | 03:28 |
| jwitko | fixed the permissions errors | 03:29 |
| jwitko | but still seeing the above messages about iscsi | 03:29 |
| bgmccollum | cloudnull: fixed -- ln -s run-upgrade.sh run-upgrade-old.sh | 03:29 |
| cloudnull | hahahah | 03:30 |
| cloudnull | bgmccollum: that wont work you need ln -fs run-upgrade.sh run-upgrade-old.sh | 03:30 |
| cloudnull | :) | 03:30 |
| bgmccollum | state: absent | 03:30 |
| cloudnull | ha | 03:31 |
| jwitko | fuck... now its not working even without the nfs mount | 03:31 |
| *** agireud has quit IRC | 03:33 | |
| *** agireud has joined #openstack-ansible | 03:33 | |
| jwitko | so somehow through all of this iscsi got hosed on this compute node | 03:37 |
| jwitko | 2015-10-19 23:37:28.776 7040 WARNING nova.virt.libvirt.volume [req-a5b03677-a26c-42c3-b878-4cbd91863b51 99903c87e9ab4bb3b29bc51b0943330c 7f43cc16e3714f1c84e525455ddb7495 - - -] ISCSI volume not yet found at: vda. Will rescan & retry. Try number: 3 | 03:37 |
| jwitko | looks like just that VMs data got corrupted | 03:40 |
| jwitko | I almost want to give it another try after fixing the NFS permissions | 03:40 |
| *** tlian2 has quit IRC | 03:41 | |
| cloudnull | yea , anything worth doing is worth doing a dozen times or so :) | 03:41 |
| jwitko | cloudnull, should I be shutting off libvirtd service before doing this? | 03:41 |
| jwitko | root@oss-comp07:~# ps -ef | grep libvirt | 03:41 |
| jwitko | root 7240 1 1 23:36 ? 00:00:04 /usr/sbin/libvirtd -d | 03:41 |
| cloudnull | i'd not assume so. | 03:42 |
| cloudnull | but maybe ? idk for sure | 03:42 |
| *** subscope has joined #openstack-ansible | 03:42 | |
| cloudnull | bgmccollum, jwagner_away if you guys can give the "new" upgrade script a thorough testing im sure we can get the new advances into the rpc tags . but it may be a bit | 03:45 |
| cloudnull | --cc d34dh0r53 | 03:45 |
| jwitko | ok, the instance is running now on nfs | 03:46 |
| jwitko | so it was the perms issue | 03:46 |
| jwitko | going to redo nfs on other compute node now | 03:46 |
| cloudnull | jwitko: i have faith :) | 03:48 |
| cloudnull | its going to work amazing this time | 03:48 |
| jwitko | cloudnull, here is an interesting error when live migrating | 03:48 |
| jwitko | 1c84e525455ddb7495 - - -] [instance: 6039bc1a-f7a9-4b52-b618-4ea9b1ab7460] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://oss-comp09/system: unable to connect to server at 'oss-comp09:16509': Connection refused | 03:48 |
| * cloudnull fingers crossed | 03:48 | |
| jwitko | there is nothing running at 16509 on any of my compute nodes though ? | 03:49 |
| cloudnull | jwitko: can you check /etc/libvirt/libvirtd.conf | 03:51 |
| jwitko | looks like I need listen_tcp = 1 | 03:51 |
| larsks | jwitko: that is the port that libvirt will listen on if you have both listen_tcp=1 in /etc/libvirt/libvirtd.conf and and --listen on the libvirtd command line. That' | 03:51 |
| cloudnull | and auth_tcp = "none" | 03:51 |
| larsks | jwitko: and also --listen (-l) on libvirtd. | 03:51 |
| jwitko | larsks, where does libvirtd start from that i can append this? | 03:51 |
| jwitko | found it | 03:52 |
| jwitko | /etc/init/libvirt-bin.conf | 03:52 |
| * larsks returns to lurker mode and goes to bed... | 03:52 | |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/kilo/playbooks/roles/os_nova/files/libvirtd.conf | 03:52 |
| cloudnull | o/ larsks | 03:53 |
| jwitko | wow this is a lot of modifications to make this work | 03:53 |
| jwitko | cloudnull, hm if its in the OSAD repo then why wasn't it set? | 03:53 |
| cloudnull | you need to set https://github.com/openstack/openstack-ansible/blob/kilo/playbooks/roles/os_nova/defaults/main.yml#L300-L302 | 03:54 |
| *** skamithi13 has quit IRC | 03:55 | |
| jwitko | cloudnull, so those should be "0", "1", and "none" respectively ? | 03:55 |
| cloudnull | yes | 03:56 |
| cloudnull | which will enable listen mode for libvirt | 03:56 |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/kilo/playbooks/roles/os_nova/tasks/nova_compute_kvm.yml#L43-L53 | 03:56 |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/kilo/playbooks/roles/os_nova/tasks/nova_compute_kvm.yml#L49 | 03:56 |
| jwitko | libvirtd failing to start | 03:58 |
| jwitko | 2015-10-20 03:57:37.447+0000: 9509: error : virNetTLSContextCheckCertFile:117 : Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory | 03:58 |
| cloudnull | with 0 1 none respectivly ? | 03:59 |
| jwitko | did it manually, running through playbook now | 03:59 |
| jwitko | this should do it right? | 03:59 |
| jwitko | openstack-ansible os-nova-install.yml --limit oss-comp09 --ask-vault-pass --tags 'nova-libvirt' | 03:59 |
| cloudnull | that'll do it | 04:00 |
| jwitko | alright, its listening | 04:00 |
| cloudnull | woot | 04:00 |
| cloudnull | as long as tls is not enabled it should to the pem look up. | 04:00 |
| jwitko | not failing.... | 04:01 |
| jwitko | yet... | 04:01 |
| jwitko | yet... | 04:01 |
| jwitko | holy | 04:01 |
| jwitko | fucking | 04:01 |
| jwitko | shit | 04:01 |
| jwitko | :D | 04:01 |
| cloudnull | maybe that was the error all along :) | 04:01 |
| jwitko | restarting libvirtd will definitely restart all VMs right ? | 04:02 |
| cloudnull | no it should not | 04:03 |
| cloudnull | the /usr/bin/qemu-system-x86_64 should remain | 04:03 |
| jwitko | so I can run that across all nodes immediately? | 04:03 |
| cloudnull | yes that should be ok | 04:03 |
| cloudnull | Sam-I-Am: bgmccollum: correct me if im wront | 04:03 |
| cloudnull | *wrong | 04:03 |
| cloudnull | but yes that should be fine | 04:04 |
| Sam-I-Am | i think its ok | 04:04 |
| cloudnull | jwitko: if you change the user variable and run openstack-ansible os-nova-install.yml --ask-vault-pass --tags 'nova-libvirt' | 04:05 |
| cloudnull | itll restart libvirtd as part of the task | 04:05 |
| jwitko | aye, thank you | 04:05 |
| cloudnull | we should document enabling live-migrate with all this glorius data --cc Sam-I-Am | 04:07 |
| cloudnull | wink wink nudge nudge | 04:07 |
| cloudnull | that would be most useful | 04:07 |
| Sam-I-Am | haha | 04:07 |
| cloudnull | especially with config_drive being configurable in kilo and beyond | 04:07 |
| Sam-I-Am | does o-a not technically support live migration now? | 04:08 |
| Sam-I-Am | i know its sort of a corner case for 'cloud' | 04:08 |
| cloudnull | it seems so . however theres some config bits that we need to set to make it go . | 04:08 |
| cloudnull | which jwitko is now an SME on :D | 04:08 |
| cloudnull | jwitko: you going to the summit ? | 04:09 |
| jwitko | in tokyo? negative | 04:09 |
| * Sam-I-Am neither | 04:10 | |
| cloudnull | :( | 04:10 |
| jwitko | i wish | 04:10 |
| jwitko | but my company doesnt do much in that way for me | 04:10 |
| cloudnull | next one is in Austin | 04:10 |
| jwitko | now that I could almost certainly swing | 04:10 |
| jwitko | as they just moved the HQ to austin | 04:10 |
| *** sdake has joined #openstack-ansible | 04:10 | |
| jwitko | but i thought the next one was in atlanta | 04:10 |
| Sam-I-Am | its austin | 04:11 |
| cloudnull | sounds like a work trip if ive ever heard | 04:11 |
| Sam-I-Am | which should be... interesting | 04:11 |
| jwitko | i work remote from NYC | 04:11 |
| cloudnull | atlanta was two summits ago | 04:11 |
| Sam-I-Am | three | 04:11 |
| cloudnull | ^ that one | 04:11 |
| cloudnull | :) | 04:11 |
| jwitko | ok | 04:11 |
| jwitko | so if i could just take a moment so summarize the work | 04:11 |
| Sam-I-Am | either way, you can contribute bits to o-a :) | 04:11 |
| jwitko | 1) override nova_libvirtd_listen_tls, nova_libvirtd_listen_tcp, nova_libvirtd_auth_tcp in openstack_user_config.yml to 0, 1, none respectively | 04:12 |
| jwitko | 2) execute openstack-ansible os-nova-install.yml --ask-vault-pass --tags 'nova-libvirt' | 04:12 |
| jwitko | 3) Shut down all VMs on an instance | 04:12 |
| jwitko | 4) Copy all data from /var/lib/nova/instances/ to a tmp directory | 04:13 |
| jwitko | 4a) stop nova-compute service | 04:13 |
| jwitko | 5) mount NFS directory over /var/lib/nova/instances | 04:13 |
| jwitko | 6) start nova compute services | 04:13 |
| jwitko | 7) start VMs back up | 04:13 |
| jwitko | I think that covers it correct? | 04:13 |
| cloudnull | 1a) disable force_config_drive | 04:14 |
| jwitko | ah, good catch | 04:14 |
| cloudnull | 2) we can do ## openstack-ansible os-nova-install.yml --ask-vault-pass --tags 'nova-libvirt,nova-config' | 04:14 |
| Sam-I-Am | this would be to enable live migrate on an existing system, vs. configuring to begin with | 04:14 |
| jwitko | Sam-I-Am, yes | 04:15 |
| cloudnull | yes | 04:15 |
| jwitko | but just trying to summarize what I did | 04:15 |
| Sam-I-Am | sure | 04:15 |
| jwitko | so 1) should also include "nova_force_config_drive: False" | 04:15 |
| cloudnull | sweet thanks for sticking with that jwitko | 04:15 |
| Sam-I-Am | i wonder how the upstream live migrate docs are | 04:15 |
| jwitko | cloudnull, I'm like a dog with lock jaw man | 04:15 |
| jwitko | i made Sam-I-Am spend an entire weekend with me once lol | 04:15 |
| cloudnull | i have that problem too :D | 04:15 |
| jwitko | but then I sent a gift card :) | 04:15 |
| cloudnull | nice! | 04:16 |
| cloudnull | Sam-I-Am: kick down man =P | 04:16 |
| jwitko | haha to be fair it was not enough to share with others | 04:16 |
| jwitko | capital grill expensive as fuk | 04:16 |
| cloudnull | Sam-I-Am: is a helpful dude | 04:17 |
| cloudnull | im off to bed , jwitko im going to try and replicate your success in the morning | 04:17 |
| cloudnull | i have a lab im going to go beat up | 04:17 |
| jwitko | thank you so much for your time | 04:17 |
| jwitko | have a great night | 04:18 |
| cloudnull | thank you for figuring it out | 04:18 |
| cloudnull | im off take care, ttyl | 04:18 |
| Sam-I-Am | enjoy | 04:18 |
| Sam-I-Am | jwitko: yeah, we had a long stretch of stuff there | 04:18 |
| *** kerwin_bai has quit IRC | 04:20 | |
| *** daneyon has joined #openstack-ansible | 04:21 | |
| *** daneyon_ has joined #openstack-ansible | 04:24 | |
| *** daneyon has quit IRC | 04:27 | |
| jwitko | hey Sam-I-Am, what config file should those variables go in to over-ride? | 04:40 |
| jwitko | user_variables.yml? openstack_user_config.yml? | 04:41 |
| jwitko | looks like user_variables? | 04:43 |
| Sam-I-Am | https://github.com/openstack/openstack-ansible/blob/kilo/doc/source/install-guide/configure-openstack.rst | 04:45 |
| jwitko | oh wow thats complicated | 04:46 |
| jwitko | the examples in user_variables.yml just have them as straight one line entries | 04:46 |
| Sam-I-Am | well, its and no | 04:51 |
| Sam-I-Am | er, yes and no | 04:52 |
| Sam-I-Am | stupid fingers | 04:52 |
| Sam-I-Am | it was getting too complex to manage all of the potential overrides, so along came a more generic way to set them... and arguably more complex | 04:52 |
| *** javeriak has joined #openstack-ansible | 05:01 | |
| *** greg_a has quit IRC | 05:07 | |
| *** javeriak_ has joined #openstack-ansible | 05:16 | |
| *** javeriak has quit IRC | 05:18 | |
| *** mcarden has joined #openstack-ansible | 05:44 | |
| *** javeriak_ has quit IRC | 05:52 | |
| *** sdake has quit IRC | 06:05 | |
| *** greg_a has joined #openstack-ansible | 06:05 | |
| *** kerwin_bai has joined #openstack-ansible | 06:07 | |
| *** gardenshed has joined #openstack-ansible | 06:08 | |
| *** manas has joined #openstack-ansible | 06:10 | |
| *** gardenshed has quit IRC | 06:22 | |
| *** markvoelker_ has joined #openstack-ansible | 06:26 | |
| *** markvoelker has quit IRC | 06:29 | |
| *** jmckind is now known as jmckind_ | 06:33 | |
| *** javeriak has joined #openstack-ansible | 06:41 | |
| *** greg_a has quit IRC | 06:50 | |
| openstackgerrit | Merged openstack/openstack-ansible: Always drop lxc_net bridge configuration https://review.openstack.org/237262 | 06:54 |
| openstackgerrit | Merged openstack/openstack-ansible: Fixes TCP and UDP strings in rsyslog client log shipping. https://review.openstack.org/237029 | 06:57 |
| *** karimb has joined #openstack-ansible | 07:00 | |
| openstackgerrit | Merged openstack/openstack-ansible: Additional RabbitMQ SSL fixes https://review.openstack.org/236731 | 07:01 |
| openstackgerrit | Jesse Pretorius proposed openstack/openstack-ansible: Fixes TCP and UDP strings in rsyslog client log shipping. https://review.openstack.org/237430 | 07:03 |
| openstackgerrit | Jesse Pretorius proposed openstack/openstack-ansible: Additional RabbitMQ SSL fixes https://review.openstack.org/237431 | 07:04 |
| *** gardenshed has joined #openstack-ansible | 07:10 | |
| *** javeriak has quit IRC | 07:11 | |
| *** javeriak has joined #openstack-ansible | 07:16 | |
| *** neillc has quit IRC | 07:28 | |
| *** daneyon_ has quit IRC | 07:31 | |
| *** daneyon has joined #openstack-ansible | 07:34 | |
| *** neillc has joined #openstack-ansible | 07:34 | |
| *** sdake has joined #openstack-ansible | 07:35 | |
| *** gardenshed has quit IRC | 07:38 | |
| *** gardenshed has joined #openstack-ansible | 07:42 | |
| *** gardenshed has quit IRC | 07:42 | |
| *** openstackgerrit has quit IRC | 07:46 | |
| *** openstackgerrit has joined #openstack-ansible | 07:46 | |
| *** karimb has quit IRC | 07:56 | |
| *** karimb has joined #openstack-ansible | 07:57 | |
| *** jhesketh has quit IRC | 08:07 | |
| *** sdake has quit IRC | 08:08 | |
| *** jhesketh has joined #openstack-ansible | 08:09 | |
| odyssey4me | jwitko user_variables if it's an ansible variable, the configure openstack page is for openstack variables | 08:19 |
| openstackgerrit | Merged openstack/openstack-ansible: Standardise ownership of *_venv_bin directories https://review.openstack.org/236985 | 08:24 |
| odyssey4me | mattt hughsaunders if you guys can take the time to work through https://review.openstack.org/230716 today, it'd be hugely appreciated | 08:25 |
| hughsaunders | odyssey4me: will have a look | 08:26 |
| openstackgerrit | Jesse Pretorius proposed openstack/openstack-ansible: Updated the neutron l3HA tool to use v3 https://review.openstack.org/237460 | 08:27 |
| *** mgoddard has joined #openstack-ansible | 08:28 | |
| *** gardenshed has joined #openstack-ansible | 08:35 | |
| *** gardenshed has quit IRC | 08:35 | |
| *** sdake has joined #openstack-ansible | 08:36 | |
| *** shausy has joined #openstack-ansible | 08:38 | |
| *** kerwin_bai has quit IRC | 08:40 | |
| *** javeriak has quit IRC | 08:52 | |
| *** sdake has quit IRC | 08:56 | |
| *** manas has quit IRC | 09:08 | |
| *** neilus has quit IRC | 09:08 | |
| *** neilus has joined #openstack-ansible | 09:08 | |
| *** shausy has quit IRC | 09:09 | |
| *** shausy has joined #openstack-ansible | 09:10 | |
| *** gardenshed has joined #openstack-ansible | 09:16 | |
| *** metral is now known as metral_zzz | 09:22 | |
| *** tiagogomes__ has joined #openstack-ansible | 09:24 | |
| *** tiagogomes has quit IRC | 09:24 | |
| *** javeriak has joined #openstack-ansible | 09:28 | |
| *** metral_zzz is now known as metral | 09:33 | |
| *** manas has joined #openstack-ansible | 09:36 | |
| *** javeriak has quit IRC | 09:40 | |
| *** javeriak has joined #openstack-ansible | 09:40 | |
| *** fawadkhaliq has joined #openstack-ansible | 09:46 | |
| *** javeriak_ has joined #openstack-ansible | 09:48 | |
| *** sdake has joined #openstack-ansible | 09:48 | |
| *** javeriak has quit IRC | 09:48 | |
| *** sdake has quit IRC | 10:03 | |
| *** subscope has quit IRC | 10:10 | |
| *** subscope has joined #openstack-ansible | 10:12 | |
| *** subscope has quit IRC | 10:13 | |
| *** markvoelker_ has quit IRC | 10:13 | |
| *** gardenshed has left #openstack-ansible | 10:15 | |
| *** gardenshed has joined #openstack-ansible | 10:15 | |
| *** javeriak_ has quit IRC | 10:17 | |
| openstackgerrit | Merged openstack/openstack-ansible: Removed deprecated vif driver https://review.openstack.org/237362 | 10:38 |
| *** javeriak has joined #openstack-ansible | 10:54 | |
| *** manas has quit IRC | 10:56 | |
| openstackgerrit | Merged openstack/openstack-ansible: Correct OS_IDENTITY_API_VERSION https://review.openstack.org/236978 | 11:04 |
| *** javeriak has quit IRC | 11:05 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible: [WIP] Install Guide Cleanup https://review.openstack.org/237513 | 11:09 |
| *** markvoelker has joined #openstack-ansible | 11:14 | |
| *** manas has joined #openstack-ansible | 11:15 | |
| *** javeriak has joined #openstack-ansible | 11:18 | |
| *** javeriak has quit IRC | 11:22 | |
| *** gcivitella has joined #openstack-ansible | 11:28 | |
| *** markvoelker has quit IRC | 11:32 | |
| openstackgerrit | Jesse Pretorius proposed openstack/openstack-ansible: Only bind mount images dir /w local Glance storage https://review.openstack.org/237145 | 11:35 |
| *** manas has quit IRC | 11:38 | |
| *** javeriak has joined #openstack-ansible | 11:39 | |
| *** markvoelker has joined #openstack-ansible | 11:39 | |
| *** markvoelker has quit IRC | 11:39 | |
| *** fawadkhaliq has quit IRC | 11:58 | |
| openstackgerrit | Merged openstack/openstack-ansible: Ensure that the vnc console is disabled when spice is enabled https://review.openstack.org/237161 | 12:08 |
| openstackgerrit | Merged openstack/openstack-ansible: Fix to correctly set the nova_management_address https://review.openstack.org/236711 | 12:08 |
| openstackgerrit | Merged openstack/openstack-ansible: Added rotating logging to the neutron ha tool https://review.openstack.org/237095 | 12:08 |
| *** jmckind_ has quit IRC | 12:08 | |
| openstackgerrit | Merged openstack/openstack-ansible: Added rotating logging to the neutron ha tool https://review.openstack.org/237092 | 12:08 |
| mhayden | morning | 12:11 |
| *** elo has quit IRC | 12:15 | |
| *** jaypipes has joined #openstack-ansible | 12:24 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-security: V-3857{4,6,7}: Password hashing algorithms https://review.openstack.org/233071 | 12:26 |
| mhayden | thanks for the backport on the RabbitMQ SSL fixes, odyssey4me -- i didn't notice it merged | 12:27 |
| *** markvoelker_ has joined #openstack-ansible | 12:35 | |
| odyssey4me | mhayden :) | 12:42 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-security: V-38660: SNMPv3 https://review.openstack.org/233226 | 12:43 |
| mhayden | i need to see if nova has a buried option for kombu SSL cert verification | 12:44 |
| mhayden | on first pass, i couldn't find it | 12:44 |
| *** neilus has quit IRC | 12:49 | |
| *** neilus has joined #openstack-ansible | 12:50 | |
| *** javeriak has quit IRC | 12:53 | |
| *** markvoelker_ has quit IRC | 12:54 | |
| *** alejandrito has joined #openstack-ansible | 12:54 | |
| *** fawadkhaliq has joined #openstack-ansible | 12:59 | |
| *** gardensh_ has joined #openstack-ansible | 13:00 | |
| *** javeriak has joined #openstack-ansible | 13:02 | |
| *** gardenshed has quit IRC | 13:03 | |
| *** fawadkhaliq has quit IRC | 13:04 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible: Set neutron_lib_dir depending on neutron_venv_enabled https://review.openstack.org/237573 | 13:12 |
| *** tlian has joined #openstack-ansible | 13:20 | |
| *** gardensh_ has quit IRC | 13:21 | |
| *** gardenshed has joined #openstack-ansible | 13:23 | |
| *** Bjoern_ has joined #openstack-ansible | 13:23 | |
| *** Bjoern_ is now known as Bjoern_zZzZzZzZ | 13:23 | |
| *** gardenshed has quit IRC | 13:24 | |
| *** gardenshed has joined #openstack-ansible | 13:25 | |
| *** javeriak has quit IRC | 13:25 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible: [WIP] Install Guide Cleanup https://review.openstack.org/237513 | 13:28 |
| *** gardenshed has quit IRC | 13:34 | |
| *** Bjoern_zZzZzZzZ is now known as Bjoern_ | 13:35 | |
| *** javeriak has joined #openstack-ansible | 13:37 | |
| *** Mudpuppy has joined #openstack-ansible | 13:39 | |
| odyssey4me | mhayden would that be for setting nova to have a ca, cert, etc - or to enable/disabled cert verification? | 13:39 |
| mhayden | odyssey4me: give the option for disabling verification (if self signed certs are in use) | 13:40 |
| odyssey4me | mhayden typically that option is 'insecure' | 13:40 |
| odyssey4me | is that when doing keystone auth? | 13:40 |
| mhayden | odyssey4me: well, i'm looking at how nova and rabbitmq could potentially talk over SSL/TLS reliably | 13:41 |
| odyssey4me | mhayden http://docs.openstack.org/developer/openstack-ansible/install-guide/configure-sslcertificates.html#self-signed-certificates | 13:41 |
| mhayden | haha, i wrote that :P | 13:41 |
| odyssey4me | ah, so then you'll need to dig up the rpc configuration... nothing to do with nova | 13:41 |
| odyssey4me | it'll be in oslo.messaging | 13:41 |
| *** Mudpuppy has quit IRC | 13:43 | |
| *** Mudpuppy has joined #openstack-ansible | 13:43 | |
| *** gardenshed has joined #openstack-ansible | 13:46 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible: Search for existing aodh binary https://review.openstack.org/237589 | 13:50 |
| cloudnull | morning | 13:51 |
| d34dh0r53 | mornings | 13:54 |
| *** fawadkhaliq has joined #openstack-ansible | 13:56 | |
| tiagogomes__ | anyone configured Keystone for using LDAP with SQL as fallback | 13:57 |
| *** phalmos has joined #openstack-ansible | 13:58 | |
| *** javeriak has quit IRC | 13:59 | |
| *** phalmos has quit IRC | 14:00 | |
| *** fawadkhaliq has quit IRC | 14:07 | |
| *** KLevenstein has joined #openstack-ansible | 14:07 | |
| openstackgerrit | Merged openstack/openstack-ansible: Only bind mount images dir /w local Glance storage https://review.openstack.org/237145 | 14:08 |
| *** spotz_zzz is now known as spotz | 14:09 | |
| *** sdake has joined #openstack-ansible | 14:10 | |
| *** shausy has quit IRC | 14:18 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:20 | |
| *** KLevenstein has quit IRC | 14:23 | |
| *** wmlynch has joined #openstack-ansible | 14:23 | |
| d34dh0r53 | reviews please https://review.openstack.org/#/c/237460/1 | 14:24 |
| *** KLevenstein has joined #openstack-ansible | 14:26 | |
| odyssey4me | d34dh0r53 https://review.openstack.org/237430 and https://review.openstack.org/237431 too | 14:26 |
| *** KLevenstein has quit IRC | 14:27 | |
| *** jmckind has joined #openstack-ansible | 14:28 | |
| *** mgoddard_ has joined #openstack-ansible | 14:31 | |
| *** jmckind is now known as jmckind_ | 14:32 | |
| *** jmckind_ is now known as jmckind | 14:33 | |
| *** mgoddard has quit IRC | 14:34 | |
| *** mgoddard_ has quit IRC | 14:34 | |
| *** mgoddard has joined #openstack-ansible | 14:34 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible: [WIP] Install Guide Cleanup https://review.openstack.org/237513 | 14:34 |
| *** mgoddard has quit IRC | 14:42 | |
| *** mgoddard has joined #openstack-ansible | 14:43 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible: Install Guide Cleanup https://review.openstack.org/237513 | 14:48 |
| *** phalmos has joined #openstack-ansible | 14:50 | |
| tiagogomes__ | can I use both LDAP and SQL for the same domain? | 14:50 |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Updated the repo-build process https://review.openstack.org/230716 | 14:51 |
| *** greg_a has joined #openstack-ansible | 14:52 | |
| cloudnull | odyssey4me hughsaunders updated repo build process for the regex update and to create an absolute requirement file | 14:55 |
| *** k_stev has joined #openstack-ansible | 14:55 | |
| openstackgerrit | Jesse Pretorius proposed openstack/openstack-ansible: Update Neutron Configuration for Liberty https://review.openstack.org/234926 | 14:56 |
| *** gardenshed has quit IRC | 15:01 | |
| *** jwagner_away is now known as jwagner | 15:02 | |
| spotz | morning | 15:04 |
| *** phalmos has quit IRC | 15:06 | |
| *** jlvillal has quit IRC | 15:11 | |
| *** jlvillal has joined #openstack-ansible | 15:17 | |
| tiagogomes__ | mmm, ok you hardcode the ldap configuration to be on keystone.Default.conf. It would be nice to allow to specify the domain name | 15:32 |
| tiagogomes__ | Otherwise I can't auth with SQL | 15:33 |
| tiagogomes__ | for the service and admin users | 15:34 |
| openstackgerrit | Merged openstack/openstack-ansible: Always drop lxc_net bridge configuration https://review.openstack.org/237266 | 15:44 |
| *** mgoddard_ has joined #openstack-ansible | 15:46 | |
| *** mgoddard has quit IRC | 15:48 | |
| *** alop has joined #openstack-ansible | 15:49 | |
| *** fawadkhaliq has joined #openstack-ansible | 15:53 | |
| *** gardenshed has joined #openstack-ansible | 15:55 | |
| *** gardenshed has quit IRC | 15:57 | |
| *** gardenshed has joined #openstack-ansible | 15:58 | |
| *** fawadkhaliq has quit IRC | 15:58 | |
| *** jwagner is now known as jwagner_away | 16:01 | |
| stevelle | Bug triage? | 16:03 |
| palendae | I think most of our UK group are getting back to desks | 16:03 |
| *** mgoddard_ has quit IRC | 16:04 | |
| *** mgoddard has joined #openstack-ansible | 16:04 | |
| *** phalmos has joined #openstack-ansible | 16:04 | |
| *** jwagner_away is now known as jwagner | 16:04 | |
| *** elo has joined #openstack-ansible | 16:08 | |
| odyssey4me | tiagogomes__ a patch would be welcome - that is an area that could do with some work :) | 16:08 |
| *** elo has quit IRC | 16:08 | |
| *** Bjoern_ is now known as BjoernT | 16:09 | |
| *** elo has joined #openstack-ansible | 16:09 | |
| odyssey4me | bug triage cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, mancdaz, dolphm, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, mhayden, scarlisle | 16:09 |
| prometheanfire | neat | 16:10 |
| odyssey4me | righto - first up: https://bugs.launchpad.net/openstack-ansible/+bug/1473525 | 16:11 |
| openstack | Launchpad bug 1473525 in openstack-ansible trunk " Duplicate sources.list entry for mariadb during kilo upgrade" [Low,New] | 16:11 |
| jwagner | i am seeing this as well | 16:11 |
| jwagner | i just remove the old http and it is fine | 16:12 |
| BjoernT | yeah so I will review my old patch | 16:14 |
| *** galstrom_zzz is now known as galstrom | 16:15 | |
| cloudnull | yea thats a thing on upgrade . | 16:15 |
| BjoernT | correct | 16:15 |
| BjoernT | so prio 2 kilo ? | 16:17 |
| odyssey4me | ok, invalid on trunk - who who's taking this on for juno->kilo? | 16:17 |
| cloudnull | its a hard one. | 16:17 |
| cloudnull | because depending on when the deployment was done that repo may be different | 16:17 |
| cloudnull | EG icehouse, juno. | 16:17 |
| BjoernT | i can look at it again | 16:18 |
| odyssey4me | can the apt task not add files with a specific name? | 16:18 |
| cloudnull | its really something that , if it causes issues it needs to be fixed, but idk if we should really try to program around that type of failure. | 16:18 |
| cloudnull | odyssey4me: it can | 16:18 |
| cloudnull | but those repos may be in various files named differently | 16:18 |
| odyssey4me | cloudnull then perhaps we should change how we add the sources file and name them according to the purpose? | 16:18 |
| BjoernT | yes | 16:19 |
| *** jmckind is now known as jmckind_ | 16:19 | |
| BjoernT | no dynamic file names | 16:19 |
| BjoernT | that's bad when changing mirror urls | 16:19 |
| *** k_stev has quit IRC | 16:19 | |
| BjoernT | etc | 16:19 |
| odyssey4me | then this issue can be resolved by deployers - it can be noted as a known issue | 16:19 |
| cloudnull | rpc-repo.rax, mariadb.com, mirror.rax, other maira mirror, etc. | 16:19 |
| *** dstanek has quit IRC | 16:19 | |
| BjoernT | that should be fixed in OSA not deployer | 16:19 |
| odyssey4me | a simple ansible command can remove the old file, whatever it is, then the plays can simply re-add | 16:19 |
| cloudnull | i think we can standardize on a name, but fixing it may break a deployer using a repo for other specific purposes if we simply remove the entries | 16:20 |
| BjoernT | sure but it doesnt make sure that if wont happen again | 16:20 |
| BjoernT | it's caused by how we configure the sources.list so it needs to be fixed there | 16:20 |
| odyssey4me | cloudnull yep, that's why it needs to be a manual fix | 16:20 |
| cloudnull | we'd have to search for, interpret, and attempt to resolve | 16:20 |
| BjoernT | cleanup has to happen in OSA, if no one volunteers I work on it | 16:20 |
| odyssey4me | and we can't implement this fix except for a major version | 16:20 |
| cloudnull | BjoernT: if you can figure out a good way to make it work then +1 | 16:21 |
| cloudnull | but it seems like a pit of edge cases | 16:21 |
| *** k_stev has joined #openstack-ansible | 16:21 | |
| odyssey4me | sure, but bear in mind that there is no standard to how a deployer has configured the repo | 16:21 |
| cloudnull | ^ that | 16:21 |
| odyssey4me | if it's internal, it'll be different | 16:21 |
| *** k_stev has quit IRC | 16:21 | |
| *** k_stev has joined #openstack-ansible | 16:21 | |
| odyssey4me | so the best is to simply list it as a known issue and provide an example resolution for it | 16:21 |
| BjoernT | not really, this is only true for the ubuntu base repos | 16:21 |
| *** gcivitella has quit IRC | 16:22 | |
| odyssey4me | BjoernT not it is not - it could be anything and will hit anyone when they change the repo | 16:22 |
| *** dstanek has joined #openstack-ansible | 16:22 | |
| BjoernT | I disagree | 16:22 |
| BjoernT | we put in the mariadb over OSA not anyone else | 16:22 |
| odyssey4me | BjoernT in https://bugs.launchpad.net/openstack-ansible/+bug/1473525 it is mariadb, not the ubuntu base repo | 16:22 |
| openstack | Launchpad bug 1473525 in openstack-ansible kilo " Duplicate sources.list entry for mariadb during kilo upgrade" [Low,New] | 16:22 |
| cloudnull | BjoernT: if you can put toether a PoC i'd be happy to work on it with you. but i do have some reservations on a fix . | 16:23 |
| odyssey4me | ok BjoernT if you can figure out an acceptable solution then great | 16:23 |
| BjoernT | sure I can spend some cycles | 16:23 |
| odyssey4me | assigned to BjoernT | 16:23 |
| odyssey4me | next: https://bugs.launchpad.net/openstack-ansible/+bug/1506173 | 16:24 |
| openstack | Launchpad bug 1506173 in openstack-ansible "Failed upgrades should output list of full commands to run" [Undecided,New] | 16:24 |
| *** greg_a has quit IRC | 16:24 | |
| cloudnull | thats been completed | 16:24 |
| odyssey4me | oh? | 16:24 |
| palendae | cloudnull: This is a newer one | 16:25 |
| palendae | Filed the 14th | 16:25 |
| cloudnull | oh , nevermind | 16:25 |
| * cloudnull was going off the title | 16:25 | |
| palendae | Basically saying the output is incomplete | 16:25 |
| palendae | Yeah, I thought it was the old one too | 16:25 |
| odyssey4me | clearly an enhancement request, rather than a bug | 16:25 |
| odyssey4me | happy to accept the enhancement request? | 16:26 |
| BjoernT | yes it's a enhancement to add the openstack-ansible command line so we can cut and paste the output | 16:26 |
| cloudnull | nevermind its covered | 16:26 |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/kilo/doc/source/upgrade-guide/process.rst#getting-started | 16:26 |
| cloudnull | "While it's not required, running all of the following playbooks with the openstack-ansible command is recommended. Additionally during the upgrade it's recommended to pass the flag, -e 'pip_install_options=--force-reinstall'. This flag will ensure all pip packages are reinstalled and running the expected versions upon the completion of the upgrade." | 16:26 |
| odyssey4me | ok, but this appears to be referring to the run-upgrade script's output | 16:27 |
| odyssey4me | fyi - a little trick: http://docs.openstack.org/developer/openstack-ansible/kilo/ | 16:28 |
| *** gardenshed has quit IRC | 16:28 | |
| BjoernT | I think just asks to complete the output to prepend openstack-ansible so we can cut and paste the output | 16:29 |
| palendae | Yeah, sounds like it's about making sure the script will output more than just the playbook name if a failure occurs | 16:29 |
| odyssey4me | agreed | 16:29 |
| odyssey4me | so is this an enhancement that someone wants to take on? is it one that is accepted as a valid enhancement? | 16:31 |
| palendae | Soudns valid to me | 16:31 |
| palendae | I can take a stab | 16:32 |
| *** KLevenstein has joined #openstack-ansible | 16:32 | |
| odyssey4me | palendae great, thanks | 16:32 |
| odyssey4me | next: https://bugs.launchpad.net/openstack-ansible/+bug/1506285 | 16:33 |
| openstack | Launchpad bug 1506285 in openstack-ansible "11.2.1 : openstack client with V3 auth causes usability issues" [Undecided,New] - Assigned to Ian Cordasco (icordasc) | 16:33 |
| *** jmckind_ is now known as jmckind | 16:33 | |
| odyssey4me | any update on this sigmavirus24 d34dh0r53 ? | 16:34 |
| sigmavirus24 | odyssey4me: waiting for an environment to debug this in | 16:34 |
| *** k_stev has quit IRC | 16:35 | |
| odyssey4me | ok, shall we leave the status as-is or change it? | 16:35 |
| tiagogomes__ | mm, for multidomain I need Keystone API v3 for the private endpoint. How do I do that? | 16:35 |
| sigmavirus24 | as-is if you don't mind | 16:35 |
| sigmavirus24 | I'll update it | 16:35 |
| odyssey4me | sigmavirus24 ok cool | 16:36 |
| odyssey4me | next: https://bugs.launchpad.net/openstack-ansible/+bug/1506291 | 16:36 |
| openstack | Launchpad bug 1506291 in openstack-ansible "11.2.1: swift playbooks fails with missing swift_pubkey" [Undecided,New] | 16:36 |
| *** KLevenstein has quit IRC | 16:37 | |
| odyssey4me | that will come from cat {{ swift_system_home_folder }}/.ssh/id_rsa.pub | 16:37 |
| odyssey4me | so it sounds like the host was missing a public key | 16:37 |
| openstackgerrit | Merged openstack/openstack-ansible: Set neutron_lib_dir depending on neutron_venv_enabled https://review.openstack.org/237573 | 16:37 |
| *** k_stev has joined #openstack-ansible | 16:37 | |
| odyssey4me | yep, https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_swift_sync/tasks/swift_key_populate.yml#L18 | 16:39 |
| cloudnull | yea that seems like the swift user was missing the key | 16:39 |
| cloudnull | which should be generated here https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_swift/tasks/swift_pre_install.yml#L46 | 16:40 |
| cloudnull | but its likely that the user existed and the key did not | 16:40 |
| odyssey4me | it may have been due to a limited/scoped execution? | 16:41 |
| BjoernT | no | 16:42 |
| BjoernT | I did run without --limit just to make sure | 16:42 |
| odyssey4me | oh - it could be due to it being an upgrade so the user existed, but the ssh key did not | 16:42 |
| odyssey4me | so this can be used: https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_swift/tasks/swift_pre_install.yml#L32 | 16:43 |
| odyssey4me | hmm, actually - this seems like a gap | 16:43 |
| BjoernT | yeah lets debug it offline | 16:43 |
| odyssey4me | if the recreate is actioned, it'll remove the keys - but will they ever be re-added | 16:43 |
| BjoernT | I run the playboogks right now | 16:43 |
| odyssey4me | one would hope that https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_swift/tasks/swift_pre_install.yml#L37 would validate all the things, but maybe it doesn't | 16:44 |
| odyssey4me | ok, let's check the next bug so long: https://bugs.launchpad.net/openstack-ansible/+bug/1506597 | 16:45 |
| openstack | Launchpad bug 1506597 in openstack-ansible "LDAP config conversion from Juno to Kilo" [Undecided,New] | 16:45 |
| cloudnull | i think both the new and old upgade scripts do the conversion . | 16:46 |
| odyssey4me | yep | 16:47 |
| BjoernT | yes but not enough as I had outlined | 16:47 |
| BjoernT | server needs to be converted to url | 16:47 |
| BjoernT | for example | 16:47 |
| odyssey4me | any volunteers to figure this one out? | 16:47 |
| cloudnull | so user_bind has become user ? | 16:48 |
| BjoernT | correct | 16:48 |
| BjoernT | the changes are in the ticket | 16:49 |
| cloudnull | hum ... so we'd need to map out all of the vars to other vars. | 16:49 |
| cloudnull | and do so here https://github.com/openstack/openstack-ansible/blob/kilo/scripts/upgrade-utilities/scripts/juno-kilo-ldap-conversion.py | 16:50 |
| BjoernT | yes correct | 16:50 |
| odyssey4me | importance? | 16:51 |
| cloudnull | medium ? | 16:51 |
| BjoernT | yes | 16:51 |
| odyssey4me | we need someone to actually verify this to confirm it | 16:51 |
| dolphm | odyssey4me: cloudnull: just shared this on the mailing list, but this is a personalized review queue (inbox zero behavior) of all reviews starred by keystone-core in keystone repos http://bit.ly/1GnOuqw (would be easy to adapt to openstack-ansible) | 16:52 |
| BjoernT | I can verify the output | 16:52 |
| odyssey4me | BjoernT you registered the bug - someone else needs to verify | 16:52 |
| BjoernT | ok | 16:52 |
| cloudnull | its verified we'll need to do the remapping | 16:52 |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Use full command when reporting upgrade failure https://review.openstack.org/237689 | 16:53 |
| odyssey4me | dolphm we have one too: https://goo.gl/03qZPi :) | 16:53 |
| odyssey4me | ok, confirmed by cloudnull - any volunteers to pick up a fix? | 16:54 |
| *** fawadkhaliq has joined #openstack-ansible | 16:54 | |
| dolphm | odyssey4me: awesome! | 16:54 |
| odyssey4me | ok, no volunteers so we'll leave it unassigned for now | 16:55 |
| odyssey4me | next: https://bugs.launchpad.net/openstack-ansible/+bug/1507311 | 16:55 |
| openstack | Launchpad bug 1507311 in openstack-ansible " nf_conntrack_ftp needed on computes to use active ftp with instances " [Undecided,New] | 16:55 |
| *** jmckind_ has joined #openstack-ansible | 16:56 | |
| *** sdake has quit IRC | 16:56 | |
| *** phalmos has quit IRC | 16:56 | |
| *** karimb has quit IRC | 16:58 | |
| odyssey4me | I thought it was well known that active ftp doesn't work through a nat? | 16:58 |
| *** fawadkhaliq has quit IRC | 16:58 | |
| dolphm | odyssey4me: looking at the source of your dashboard, mine has a different goal. it's not *all* reviews, it's crowd sourcing the starred reviews by *anyone* in core, and shows nothing but starred reviews | 16:58 |
| odyssey4me | or is this between instances on the same net | 16:58 |
| odyssey4me | dolphm oh, very nice :) | 16:59 |
| *** gardenshed has joined #openstack-ansible | 16:59 | |
| *** jmckind has quit IRC | 16:59 | |
| dolphm | odyssey4me: also, your dashboard will continue showing reviews that you've downvoted? what's the logic to that? that you need to follow up? | 16:59 |
| *** gardenshed has quit IRC | 17:00 | |
| cloudnull | odyssey4me: i say wont fix | 17:00 |
| *** gardenshed has joined #openstack-ansible | 17:00 | |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: [WIP] Test enabling prevent_arp_spoofing https://review.openstack.org/236995 | 17:01 |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: [WIP] Use full command when reporting upgrade failure https://review.openstack.org/237689 | 17:01 |
| odyssey4me | yeah, ok - there's an existing way to make this work for a deployment, so won't fix | 17:02 |
| cloudnull | +1 | 17:03 |
| odyssey4me | alright, thanks all | 17:03 |
| odyssey4me | dolphm I'll check yours and mine out - although part of the reason for including the down-voted in some sections is to try and get a broader set of reviews | 17:04 |
| odyssey4me | but yeah, downvoted reviews tend not to get reviewed - so I may as well have those removed | 17:05 |
| odyssey4me | dolphm why're you including the attic repo? | 17:05 |
| dolphm | odyssey4me: because it's one of our repos | 17:06 |
| cloudnull | https://review.openstack.org/#/c/230716 <- hughsaunders odyssey4me if you could give it another review it'd be appreciated. | 17:06 |
| odyssey4me | dolphm I thought that the attic repo was where retired stuff went to die? | 17:06 |
| dolphm | odyssey4me: yes, but if there's a starred review against it - why wouldn't it be included? no discrimination! | 17:07 |
| odyssey4me | dolphm also, to cater for future needs you may wish to try using regex for the foreach :) | 17:07 |
| dolphm | odyssey4me: to match other projects? | 17:07 |
| odyssey4me | eg: project:^openstack/keystone.* | 17:07 |
| dolphm | odyssey4me: that'd be smart; i was thinking that this config file could be genereted by gerrit pretty easily per *-core group, so the explicit list of projects would be a natural consequence | 17:08 |
| dolphm | odyssey4me: query for all *-core groups, query projects those groups have permissions on, build a dashboard for the group, publish permalink | 17:08 |
| odyssey4me | dolphm are you aware of gerrit-dash-creator? | 17:09 |
| odyssey4me | eg: https://github.com/openstack/gerrit-dash-creator/blob/master/dashboards/openstack-ansible.dash | 17:09 |
| odyssey4me | they publish a list of dashboards to: http://ghostcloud.net/openstack_gerrit_dashboards/ | 17:09 |
| dolphm | odyssey4me: yes | 17:10 |
| odyssey4me | heh, it appears that the publishing hasn't been done in some time | 17:10 |
| dolphm | odyssey4me: https://github.com/dolph/dotfiles/blob/master/gerrit-dashboard-keystone | 17:10 |
| odyssey4me | dolphm oh nice, I'll take a peek and see if there're some bits we can include for the core review dash - and then I like your idea of crowdsourcing reviews :) | 17:12 |
| *** woodard has joined #openstack-ansible | 17:12 | |
| *** sdake has joined #openstack-ansible | 17:14 | |
| *** KLevenstein has joined #openstack-ansible | 17:16 | |
| *** elo has quit IRC | 17:18 | |
| *** sdake_ has joined #openstack-ansible | 17:21 | |
| *** sdake has quit IRC | 17:23 | |
| *** elo has joined #openstack-ansible | 17:23 | |
| mhayden | i hear that the keystone thing is kinda popular | 17:25 |
| *** harlowja has quit IRC | 17:30 | |
| *** woodard has quit IRC | 17:31 | |
| *** harlowja has joined #openstack-ansible | 17:36 | |
| *** greg_a has joined #openstack-ansible | 17:37 | |
| *** elo has quit IRC | 17:40 | |
| *** woodard has joined #openstack-ansible | 17:41 | |
| *** elo has joined #openstack-ansible | 17:43 | |
| *** woodard has quit IRC | 17:46 | |
| *** sdake_ is now known as sdake | 17:47 | |
| *** woodard has joined #openstack-ansible | 17:54 | |
| *** phalmos has joined #openstack-ansible | 17:58 | |
| *** greg_a has quit IRC | 17:58 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-security: V-38660: SNMPv3 https://review.openstack.org/233226 | 18:01 |
| *** KLevenstein has quit IRC | 18:10 | |
| *** woodard has quit IRC | 18:14 | |
| *** woodard has joined #openstack-ansible | 18:15 | |
| *** greg_a has joined #openstack-ansible | 18:18 | |
| *** woodard has quit IRC | 18:25 | |
| *** b3rnard0 is now known as b3rnard0_away | 18:33 | |
| *** woodard has joined #openstack-ansible | 18:41 | |
| openstackgerrit | Merged openstack/openstack-ansible: Search for existing aodh binary https://review.openstack.org/237589 | 18:42 |
| *** fawadkhaliq has joined #openstack-ansible | 18:44 | |
| *** sdake has quit IRC | 18:47 | |
| *** cbits has joined #openstack-ansible | 18:47 | |
| *** galstrom is now known as galstrom_zzz | 18:48 | |
| openstackgerrit | Charles Bitter proposed openstack/openstack-ansible: Removing extra 'provides' from doc https://review.openstack.org/237737 | 18:48 |
| *** javeriak has joined #openstack-ansible | 18:48 | |
| *** harlowja has quit IRC | 18:53 | |
| *** woodard_ has joined #openstack-ansible | 19:02 | |
| *** woodard has quit IRC | 19:02 | |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Test enabling prevent_arp_spoofing https://review.openstack.org/236995 | 19:03 |
| *** harlowja has joined #openstack-ansible | 19:11 | |
| jwitko | cloudnull, thanks again for the help last night | 19:11 |
| *** b3rnard0_away is now known as b3rnard0 | 19:11 | |
| cloudnull | anytime | 19:11 |
| jwitko | I'm seeing some pretty nasty issues popping up with live migration though due to nova users having different gid/uid on different servers | 19:11 |
| jwitko | on the nfs mount | 19:12 |
| jwitko | ownership changing and such | 19:12 |
| cloudnull | yuk | 19:12 |
| cloudnull | is that because of the nfs users or the compute host users? | 19:13 |
| jwitko | compute host users | 19:14 |
| jwitko | the nova user specifically | 19:14 |
| jwitko | I'm not specifying a specific uid upon creation | 19:14 |
| jwitko | so it just allocates next available | 19:14 |
| jwitko | so some are 997, some are 998 | 19:15 |
| *** spotz is now known as spotz_zzz | 19:15 | |
| jwitko | the newest model is 1001! | 19:15 |
| jwitko | so I am wondering how much it will screw shit up | 19:15 |
| jwitko | if i usermod -u 1001 nova | 19:15 |
| jwitko | on the older servers | 19:15 |
| cloudnull | so if we cordinated the openstack user ids that would solve that issue | 19:15 |
| *** greg_a has quit IRC | 19:16 | |
| jwitko | yes | 19:16 |
| jwitko | obviously thats a trickier subject than it appears at surface level | 19:17 |
| jwitko | you'd have to let users know "these uids must be reserved for openstack" | 19:17 |
| jwitko | and they'd probably have to be some far-reaching uids so as to avoid common use | 19:17 |
| cloudnull | yes and on an upgrade you'd have to change the perms on all of the files/folders owned | 19:17 |
| jwitko | well the following seems to work well for me: service nova-compute stop && sleep 1 && usermod -u 1001 nova ; service nova-compute start | 19:18 |
| *** greg_a has joined #openstack-ansible | 19:18 | |
| *** gparaskevas has joined #openstack-ansible | 19:20 | |
| cloudnull | jwitko: i dont think that will change perms on things like /etc/nova/ | 19:21 |
| cloudnull | and the group mod will need to be updated | 19:22 |
| *** spotz_zzz is now known as spotz | 19:22 | |
| *** woodard_ has quit IRC | 19:22 | |
| *** woodard has joined #openstack-ansible | 19:22 | |
| *** woodard has quit IRC | 19:24 | |
| *** sdake has joined #openstack-ansible | 19:25 | |
| *** woodard has joined #openstack-ansible | 19:25 | |
| *** qba73 has joined #openstack-ansible | 19:26 | |
| jwitko | cloudnull, yea definitely missed those. Is there anything else owned by nova I should look out for? | 19:28 |
| *** qba73 has quit IRC | 19:28 | |
| *** daneyon has quit IRC | 19:28 | |
| *** javeriak_ has joined #openstack-ansible | 19:32 | |
| *** javeriak has quit IRC | 19:34 | |
| *** woodard has quit IRC | 19:35 | |
| openstackgerrit | Jesse Pretorius proposed openstack/openstack-ansible: Update Neutron Configuration for Liberty https://review.openstack.org/234926 | 19:35 |
| jwitko | odyssey4me, cloudnull, do you happen to know the tags to popualte the /etc/hosts file on machines? | 19:37 |
| jwitko | for some reason when I added these hosts it looks like it didn't make it around to the hosts files with the new compute nodes | 19:37 |
| jwitko | causing live migration to fail | 19:37 |
| openstackgerrit | Jesse Pretorius proposed openstack/openstack-ansible: Update Neutron Configuration for Liberty https://review.openstack.org/234926 | 19:37 |
| jwitko | openstack-host-hostfile? | 19:38 |
| jwitko | probably because I ran it with a limit... but i thought it saw to go to all hosts | 19:39 |
| *** sdake has quit IRC | 19:40 | |
| *** galstrom_zzz is now known as galstrom | 19:42 | |
| bgmccollum | docs should be updated to remove the example of using --limit...so many things break in that scenario | 19:42 |
| palendae | cloudnull, sigmavirus24 in this block, is there a reason that '$2' is used in the eval line instead of '$run_item'? https://github.com/openstack/openstack-ansible/blob/kilo/scripts/run-upgrade.sh#L55-L79 | 19:43 |
| jwitko | bgmccollum, yup i just reran it using that tag and it definitely skipped over that part | 19:44 |
| *** karimb has joined #openstack-ansible | 19:45 | |
| *** karimb has quit IRC | 19:45 | |
| bgmccollum | jwitko: tag or limit? their two different things... | 19:45 |
| bgmccollum | * they're | 19:46 |
| *** karimb has joined #openstack-ansible | 19:47 | |
| jwitko | bgmccollum, i had to run it with that tag and no limit because when I added my new compute nodes I ran setup-everything with a limit | 19:47 |
| jwitko | and so it didn't add the compute node hosts to the /etc/hosts file | 19:48 |
| jwitko | of all other servers besides the new compute nodes | 19:48 |
| sigmavirus24 | Good question palendae | 19:48 |
| *** galstrom is now known as galstrom_zzz | 19:48 | |
| bgmccollum | jwitko: so its fixed? | 19:48 |
| sigmavirus24 | That said, that `-e` should be before the $2 | 19:48 |
| jwitko | bgmccollum yes live migration currently works | 19:49 |
| sigmavirus24 | palendae: cloudnull ^ | 19:49 |
| *** galstrom_zzz is now known as galstrom | 19:49 | |
| sigmavirus24 | Otherwise for tasks like L144 you're passing -e to `true` | 19:49 |
| jwitko | bgmccollum, the final permissions change was as follows | 19:50 |
| jwitko | service nova-compute stop ; sleep 1 ; usermod -u 1001 nova; groupmod -g 1001 nova; chown nova:nova /var/lib/nova /var/lock/nova /var/log/nova/* /var/log/nova /etc/nova -R ; service nova-compute start | 19:50 |
| *** sdake has joined #openstack-ansible | 19:51 | |
| palendae | sigmavirus24: so openstack-ansible -e pip_stuff $run_item | 19:52 |
| palendae | What I'd like to do is make the string that's evaled into a template so I can interpolate the variable into it | 19:52 |
| palendae | variables* | 19:52 |
| palendae | Thus, when output happens on a failure, the entire command is output, not just the pieces in RUN_TASKS | 19:52 |
| sigmavirus24 | palendae: so you can do `cmd="openstack-ansible -e pip_stuff $2"; eval $cmd` | 19:54 |
| sigmavirus24 | or `eval "$cmd"` | 19:55 |
| palendae | sigmavirus24: Which works in that 1 case; I'm more interested in applying it to the failure loop (https://github.com/openstack/openstack-ansible/blob/kilo/scripts/run-upgrade.sh#L92-L94) | 19:55 |
| sigmavirus24 | That let's you show exactly the command run, what BjoernT and others wanted was just the steps to run for the rest of the upgrade though and I don't know if they'd rather have the full command or not | 19:55 |
| palendae | https://review.openstack.org/#/c/237689/2, too | 19:55 |
| palendae | They would | 19:55 |
| palendae | Working on an enhancement request from bgmccollum | 19:56 |
| sigmavirus24 | okay | 19:56 |
| palendae | I can fix the 1 case, where we use $2 like you said | 19:56 |
| palendae | But the loop is more general, and bash doesn't have something like % or .format from python | 19:56 |
| *** fawadkhaliq has quit IRC | 19:58 | |
| *** spotz is now known as spotz_zzz | 19:58 | |
| sigmavirus24 | palendae: bash is more akin to ruby in this case | 19:59 |
| sigmavirus24 | It's why so many ruby scripts are as unreadable as bash =P | 19:59 |
| palendae | heh | 19:59 |
| *** KLevenstein has joined #openstack-ansible | 20:00 | |
| *** spotz_zzz is now known as spotz | 20:02 | |
| *** daneyon has joined #openstack-ansible | 20:05 | |
| openstackgerrit | Merged openstack/openstack-ansible: Additional RabbitMQ SSL fixes https://review.openstack.org/237431 | 20:05 |
| *** daneyon has quit IRC | 20:11 | |
| *** jmckind_ has quit IRC | 20:20 | |
| *** wmlynch has quit IRC | 20:21 | |
| openstackgerrit | Merged openstack/openstack-ansible: Updated the neutron l3HA tool to use v3 https://review.openstack.org/237460 | 20:27 |
| openstackgerrit | Merged openstack/openstack-ansible: Fixes TCP and UDP strings in rsyslog client log shipping. https://review.openstack.org/237430 | 20:27 |
| *** woodard has joined #openstack-ansible | 20:44 | |
| *** KLevenstein has quit IRC | 20:53 | |
| *** woodard has quit IRC | 21:04 | |
| *** woodard has joined #openstack-ansible | 21:05 | |
| mhayden | cloudnull: do you know of any stress test available for kombu and/or oslo.messaging? | 21:14 |
| mhayden | trying to see what impact nova<->SSL<->rabbitmq might have | 21:14 |
| *** woodard has quit IRC | 21:14 | |
| *** woodard has joined #openstack-ansible | 21:15 | |
| *** javeriak_ has quit IRC | 21:15 | |
| cloudnull | no i do not | 21:15 |
| sigmavirus24 | mhayden: what about splitting out multiple threads to send tons of messages? | 21:15 |
| cloudnull | :( | 21:15 |
| *** jimchou has joined #openstack-ansible | 21:16 | |
| *** woodard has quit IRC | 21:16 | |
| mhayden | might be worth a try | 21:18 |
| mhayden | i doubt it would be terribly impactful especially with AES offloading in modern CPUs | 21:18 |
| sigmavirus24 | mhayden: I generally agree with that | 21:18 |
| *** woodard has joined #openstack-ansible | 21:20 | |
| *** gparaskevas has quit IRC | 21:22 | |
| *** woodard has quit IRC | 21:27 | |
| *** greg_a has quit IRC | 21:29 | |
| *** woodard has joined #openstack-ansible | 21:31 | |
| jwitko | hey cloudnull, do you know if I disable a hypervisor under the "Admin --> Hypervisors --> Compute Host" tab clicking the "Disable service" button | 21:33 |
| jwitko | will this leave current VMs working without issue and will just disable provisioning new VMs to those hosts? | 21:33 |
| *** CheKoLyN has joined #openstack-ansible | 21:34 | |
| *** phalmos has quit IRC | 21:37 | |
| cloudnull | jwitko: that should leave the running instances online but make it so you're no longer able to schedule vms to that target . | 21:39 |
| openstackgerrit | Bjoern Teipel proposed openstack/openstack-ansible: Adding new RabbitMQ alarms (fd,proc,sockets) https://review.openstack.org/237783 | 21:40 |
| *** KLevenstein has joined #openstack-ansible | 21:46 | |
| *** alejandrito has quit IRC | 21:47 | |
| openstackgerrit | Bjoern Teipel proposed openstack/openstack-ansible: Adding new RabbitMQ alarms (fd,proc,sockets) https://review.openstack.org/237783 | 21:51 |
| *** woodard has quit IRC | 21:51 | |
| *** woodard has joined #openstack-ansible | 21:52 | |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: [WIP] Use full command when reporting upgrade failure https://review.openstack.org/237689 | 21:54 |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: [WIP] Use full command when reporting upgrade failure https://review.openstack.org/237689 | 21:57 |
| *** woodard has quit IRC | 21:58 | |
| *** jongchoi_ has joined #openstack-ansible | 21:58 | |
| *** karimb has quit IRC | 22:06 | |
| *** aslaen has joined #openstack-ansible | 22:17 | |
| *** aslaen has quit IRC | 22:17 | |
| *** sdake has quit IRC | 22:18 | |
| *** sdake has joined #openstack-ansible | 22:18 | |
| *** galstrom is now known as galstrom_zzz | 22:24 | |
| *** jongchoi_ has quit IRC | 22:26 | |
| *** spotz is now known as spotz_zzz | 22:27 | |
| *** k_stev has quit IRC | 22:28 | |
| *** jongchoi has joined #openstack-ansible | 22:36 | |
| *** jongchoi has quit IRC | 22:36 | |
| *** galstrom_zzz is now known as galstrom | 22:39 | |
| *** jimchou_ has joined #openstack-ansible | 22:43 | |
| *** jimchou has quit IRC | 22:44 | |
| *** jimchou_ has quit IRC | 22:48 | |
| *** sdake_ has joined #openstack-ansible | 22:48 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:49 | |
| *** sdake has quit IRC | 22:49 | |
| *** KLevenstein has quit IRC | 23:00 | |
| *** jhesketh has quit IRC | 23:00 | |
| *** jhesketh has joined #openstack-ansible | 23:03 | |
| cloudnull | im off take care all | 23:04 |
| *** daneyon has joined #openstack-ansible | 23:12 | |
| *** galstrom is now known as galstrom_zzz | 23:17 | |
| *** jimchou has joined #openstack-ansible | 23:34 | |
| *** daneyon_ has joined #openstack-ansible | 23:34 | |
| *** jaypipes has quit IRC | 23:36 | |
| *** daneyon has quit IRC | 23:37 | |
| *** jimchou has quit IRC | 23:39 | |
| *** BjoernT has quit IRC | 23:50 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!