| *** hkominos has quit IRC | 00:03 | |
| *** gregwork has quit IRC | 01:05 | |
| *** yamamoto has joined #openstack-lbaas | 01:54 | |
| *** yamamoto has quit IRC | 02:02 | |
| *** yamamoto has joined #openstack-lbaas | 02:10 | |
| *** yamamoto has quit IRC | 02:16 | |
| *** yamamoto has joined #openstack-lbaas | 02:33 | |
| openstackgerrit | zhangxuanyuan proposed openstack/octavia master: Remove option status_update_threads https://review.opendev.org/699146 | 03:03 |
|---|---|---|
| *** psachin has joined #openstack-lbaas | 03:34 | |
| *** yamamoto has quit IRC | 03:39 | |
| *** yamamoto has joined #openstack-lbaas | 03:42 | |
| openstackgerrit | zhangxuanyuan proposed openstack/octavia master: Remove configuration option 'amp_ssh_access_allowed' https://review.opendev.org/699151 | 03:50 |
| *** openstackgerrit has quit IRC | 04:08 | |
| *** dmellado has quit IRC | 04:08 | |
| *** Dinesh_Bhor has quit IRC | 04:08 | |
| *** strigazi has quit IRC | 04:08 | |
| *** gthiemonge has quit IRC | 04:08 | |
| *** sapd1_ has quit IRC | 04:08 | |
| *** larsks has quit IRC | 04:08 | |
| *** jmccrory has quit IRC | 04:08 | |
| *** openstackstatus has quit IRC | 04:11 | |
| *** openstackstatus has joined #openstack-lbaas | 04:13 | |
| *** ChanServ sets mode: +v openstackstatus | 04:13 | |
| *** goldyfruit has joined #openstack-lbaas | 04:27 | |
| *** goldyfruit has quit IRC | 04:58 | |
| *** yamamoto has quit IRC | 05:53 | |
| *** yamamoto has joined #openstack-lbaas | 05:54 | |
| *** yamamoto has quit IRC | 05:59 | |
| *** ramishra has joined #openstack-lbaas | 06:04 | |
| *** yamamoto has joined #openstack-lbaas | 06:42 | |
| *** rcernin has quit IRC | 06:58 | |
| *** maciejjozefczyk_ has joined #openstack-lbaas | 07:09 | |
| *** maciejjozefczyk_ is now known as mjozefcz | 07:11 | |
| *** mjozefcz is now known as maciejjozefczyk | 07:11 | |
| *** pcaruana has joined #openstack-lbaas | 07:16 | |
| *** yamamoto has quit IRC | 07:46 | |
| *** yamamoto has joined #openstack-lbaas | 07:46 | |
| *** gcheresh has joined #openstack-lbaas | 07:48 | |
| *** tesseract has joined #openstack-lbaas | 07:48 | |
| *** yamamoto has quit IRC | 07:58 | |
| *** yamamoto has joined #openstack-lbaas | 07:58 | |
| *** tkajinam has quit IRC | 08:06 | |
| *** tesseract has quit IRC | 08:11 | |
| *** rpittau|afk is now known as rpittau | 08:34 | |
| *** ivve has joined #openstack-lbaas | 08:35 | |
| *** yamamoto has quit IRC | 08:42 | |
| *** yamamoto has joined #openstack-lbaas | 08:45 | |
| *** rcernin has joined #openstack-lbaas | 08:54 | |
| *** tesseract has joined #openstack-lbaas | 09:03 | |
| *** tesseract has quit IRC | 09:04 | |
| *** tesseract has joined #openstack-lbaas | 09:04 | |
| *** mugsie has quit IRC | 09:19 | |
| *** mugsie has joined #openstack-lbaas | 09:21 | |
| *** rcernin has quit IRC | 09:36 | |
| *** rcernin has joined #openstack-lbaas | 09:36 | |
| *** hkominos has joined #openstack-lbaas | 09:55 | |
| *** strigazi has joined #openstack-lbaas | 10:05 | |
| *** jmccrory has joined #openstack-lbaas | 10:05 | |
| *** larsks has joined #openstack-lbaas | 10:06 | |
| *** sapd1 has joined #openstack-lbaas | 10:06 | |
| *** gthiemonge has joined #openstack-lbaas | 10:07 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 10:07 | |
| *** dmellado has joined #openstack-lbaas | 10:07 | |
| *** irclogbot_3 has quit IRC | 10:08 | |
| *** irclogbot_2 has joined #openstack-lbaas | 10:09 | |
| *** rcernin has quit IRC | 10:16 | |
| *** salmankhan has joined #openstack-lbaas | 10:22 | |
| *** salmankhan has quit IRC | 10:23 | |
| *** salmankhan has joined #openstack-lbaas | 10:23 | |
| *** yamamoto has quit IRC | 10:53 | |
| *** yamamoto has joined #openstack-lbaas | 10:53 | |
| *** yamamoto has quit IRC | 11:03 | |
| hkominos | morning all. Is it possible to get ssh access to the loadbalancer to check the logs ? (maybe obtain octavia-ssh-key somehow ) | 11:12 |
| cgoncalves | hkominos, it is. connect to the amphora on the lb-mgmt-net. get the IP from "openstack loadbalancer amphora list" | 11:17 |
| cgoncalves | user name depends on the distribution you're using for the amphorae. given in your case it's TripleO deployment, the amphorae are CentOS thus username 'centos' (or cloud-user? I always mix up) | 11:18 |
| cgoncalves | hkominos, this doc should apply to TripleO upstream too: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/networking_guide/sec-octavia#accessing_amphora_logs | 11:19 |
| *** rpittau is now known as rpittau|bbl | 11:22 | |
| hkominos | cgoncalves: suprisingly openstack loadbalancer amphora list is empty | 11:31 |
| hkominos | I can find the internal ips from the console | 11:31 |
| hkominos | of course | 11:31 |
| hkominos | the boot console which shows a user ssh key inserted and an ip | 11:32 |
| cgoncalves | hkominos, that's odd. is the list empty or your user does not have admin permissions? | 11:32 |
| cgoncalves | it might also be that something wrong happened that led octavia to delete the amphora but in the process it failed to delete it in nova | 11:33 |
| cgoncalves | aka zombie amphora | 11:33 |
| hkominos | so it seems that the instance has been deleted in nova as well | 11:33 |
| hkominos | So octavia does lifecycle mgmt as well ? | 11:33 |
| cgoncalves | hkominos, lifecycle mgmt of its resources, yes | 11:34 |
| cgoncalves | for example, if the octavia health manager service for some reason detected that the amphora is not healthy it triggers an amphora failover | 11:35 |
| hkominos | I assume this is what happened. I am just tryting to login to VM to see logs to see why I am seeeing this networking (?) error. https://pastebin.com/3bLWt910 | 11:36 |
| *** yamamoto has joined #openstack-lbaas | 11:41 | |
| *** yamamoto has quit IRC | 11:41 | |
| *** yamamoto has joined #openstack-lbaas | 11:42 | |
| cgoncalves | hkominos, I'm guessing the certificates were not well configured. did you let TripleO handle them or you specified your owns? | 11:43 |
| hkominos | I created a set of certificates according to the instructions.But If the handshake fails I would see it in worker.log . | 11:45 |
| hkominos | Now I see something like 2019-12-16 11:37:46.574 24 INFO octavia.certificates.generator.local [-] Signing a certificate request using OpenSSL locally.2019-12-16 11:37:46.574 24 INFO octavia.certificates.generator.local [-] Using CA Certificate from config.2019-12-16 11:37:46.574 24 INFO octavia.certificates.generator.local [-] Using CA Private Key | 11:46 |
| hkominos | from config.2019-12-16 11:37:46.574 24 INFO octavia.certificates.generator.local [-] Using CA Private Key Passphrase from config. | 11:46 |
| hkominos | so I assume it all went well | 11:46 |
| hkominos | So i want to connect into the VM | 11:47 |
| hkominos | But I am missing the private key I guess | 11:47 |
| hkominos | the ssh private key i mean to connect from the controllers | 11:49 |
| cgoncalves | hkominos, not necessarily. those messages are logged when the worker creates the server-side (amphora) certificate. later, the worker tries to TLS connect to the amphora. if the two cannot establish a handshake, the connection fails and thus the amphora create reverts (that is why your amphora list output is empty) | 11:52 |
| cgoncalves | hkominos, TripleO upstream does not document how to provide own certificates. OSP documentation does but is not much clear at the moment. try this (credits to johnsom): http://paste.openstack.org/show/e8YMI5z5NL9kclTdXdur/ | 11:55 |
| hkominos | cgoncalves: so we follow the instructions from RDO (kinda) Some of the config does not propagate at the moment correctly But I have access to the controllers so I login and manually inject the client and the server certificates in the containers. And it seemed to have worked until now | 11:58 |
| hkominos | Perhaps something is not injected properly to the VM though | 11:58 |
| hkominos | I will try again | 11:58 |
| *** gthiemonge has quit IRC | 12:13 | |
| *** hkominos has quit IRC | 12:14 | |
| *** gthiemonge has joined #openstack-lbaas | 12:14 | |
| *** yamamoto has quit IRC | 12:20 | |
| *** yamamoto has joined #openstack-lbaas | 12:21 | |
| *** ivve has quit IRC | 12:22 | |
| *** pcaruana has quit IRC | 12:27 | |
| *** ivve has joined #openstack-lbaas | 12:30 | |
| *** pcaruana has joined #openstack-lbaas | 12:33 | |
| *** yamamoto has quit IRC | 12:43 | |
| *** goldyfruit has joined #openstack-lbaas | 13:09 | |
| *** rpittau|bbl is now known as rpittau | 13:09 | |
| *** servagem has joined #openstack-lbaas | 13:13 | |
| *** yamamoto has joined #openstack-lbaas | 13:34 | |
| *** yamamoto has quit IRC | 13:34 | |
| *** yamamoto has joined #openstack-lbaas | 13:34 | |
| *** goldyfruit has quit IRC | 13:37 | |
| *** haleyb has joined #openstack-lbaas | 13:56 | |
| *** psachin has quit IRC | 14:01 | |
| *** hkominos has joined #openstack-lbaas | 14:27 | |
| hkominos | cgoncalve: Are you also part of octavia integration in OOO ? | 14:27 |
| cgoncalves | hkominos, yes | 14:28 |
| hkominos | do you mind if I run debug log by you ? I am trying to rerun the octavia installation with the new keys but some ssh key on the undercloud is giving me trouble | 14:28 |
| cgoncalves | hkominos, sure | 14:29 |
| hkominos | https://pastebin.com/wYBApxr4 | 14:32 |
| hkominos | This is the failing step ansible-playbook -i inventory.yaml --extra-vars group_vars /usr/share/tripleo-common/playbooks/octavia-files.yaml --private-key /var/lib/mistral/overcloud/ssh_private_key | 14:32 |
| hkominos | I see that the error is found here : /usr/share/openstack-tripleo-common/playbooks/roles/octavia-undercloud/tasks/main.yml L35 | 14:34 |
| hkominos | which is an ssh key check | 14:34 |
| hkominos | maybe created here ? https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/octavia/octavia-deployment-config.j2.yaml#L248 | 14:37 |
| cgoncalves | hkominos, tripleo rocky? | 14:42 |
| hkominos | yes | 14:42 |
| hkominos | -rwxr-xr-x. 1 42430 42430 1675 Dec 16 12:58 ssh_private_key | 14:43 |
| hkominos | I just wanted to add the the permissins for the key seem ok. I see the ssh key from the mistral container. | 14:44 |
| hkominos | but I think the test seem to think that it is unreadble for some reason | 14:45 |
| hkominos | ls | 14:47 |
| cgoncalves | hmm, sorry, trying to remember the little details of rocky. it handles things a bit differently from queens and >=stein | 14:50 |
| cgoncalves | what is amp_ssh_key_path set to? | 14:51 |
| hkominos | excellent question ! | 14:51 |
| hkominos | I dont know. I think that is is empty but it should not | 14:51 |
| hkominos | I mean I dont know. | 14:51 |
| hkominos | I dont set it anywhere | 14:51 |
| cgoncalves | so you didn't set OctaviaAmphoraSshKeyFile? | 14:52 |
| cgoncalves | when not set, default is to use public key from stack@undercloud:~/.ssh/id_rsa.pub | 14:52 |
| hkominos | OctaviaAmphoraSshKeyFile is new to me. I only set as per ouyr previous discussion the yaml files here | 14:53 |
| hkominos | http://paste.openstack.org/show/e8YMI5z5NL9kclTdXdur/ | 14:53 |
| cgoncalves | folks, if this too much TripleO for the channel let us know. we can move the conversation to #tripleo | 14:53 |
| hkominos | is amp_ssh_key_path visible in the plan ? | 14:54 |
| hkominos | let me check | 14:54 |
| hkominos | Ok. It is basically what you said. amp_ssh_key_path: { get_param: OctaviaAmphoraSshKeyFile } | 14:55 |
| hkominos | I dont set it up so it should be using the keys from the UC which should be fine | 14:55 |
| cgoncalves | hkominos, ok. do you have the keypair named "default" in the UC? | 14:56 |
| hkominos | keypair ??? | 14:58 |
| cgoncalves | yeah, nova keypair | 14:59 |
| hkominos | I assume we are talking about the ansible user ? | 14:59 |
| hkominos | a | 14:59 |
| cgoncalves | anyway, if you haven't set it it shouldn't be entering that code path | 14:59 |
| hkominos | xm. I did not remember setting anything | 14:59 |
| cgoncalves | if must have amp_ssh_key_path set (even if empty string) to something, else I don't know how it's entering the ansible block https://github.com/openstack/tripleo-common/blob/stable/rocky/playbooks/roles/octavia-undercloud/tasks/main.yml#L28 | 15:00 |
| cgoncalves | if only I could remember where the ansible vars are stored in the undercloud... | 15:00 |
| hkominos | I am speculating that is entering because a keypair is already present there because this deployement of octavia is overwriting a previous one (also broken) but with different keys | 15:01 |
| hkominos | lets find the ansible vars then | 15:02 |
| hkominos | let me search | 15:02 |
| *** KeithMnemonic has joined #openstack-lbaas | 15:02 | |
| hkominos | aha! | 15:03 |
| hkominos | it looks empty | 15:03 |
| hkominos | "amp_ssh_key_name": "octavia-ssh-key", "amp_ssh_key_path": "", | 15:03 |
| hkominos | or at least that is visible in ansible.log | 15:04 |
| cgoncalves | being it set to "" (empty) I'd have expected it to not enter that ansible block... :/ | 15:05 |
| hkominos | I dont know. I am too confused | 15:05 |
| *** psachin has joined #openstack-lbaas | 15:07 | |
| hkominos | Does it make any sense to restart the mistral containers ? | 15:07 |
| hkominos | doubt it, but what do I know. | 15:07 |
| *** mloza has quit IRC | 15:08 | |
| cgoncalves | nop | 15:09 |
| cgoncalves | argh, now that I'm thinking more about this I think I once got into the same/similar issue. I'm looking around | 15:10 |
| *** TrevorV has joined #openstack-lbaas | 15:17 | |
| *** ivve has quit IRC | 15:21 | |
| *** goldyfruit has joined #openstack-lbaas | 15:23 | |
| *** goldyfruit has quit IRC | 15:45 | |
| hkominos | There is also a permission denied issue in the previous log. Dunno where it comes from | 15:45 |
| hkominos | [WARNING]: Could not create retry file '/usr/share/tripleo-common/playbooks/octavia-files.retry'. [Errno 13] Permission denied: u'/usr/share/tripleo-common/playbooks/octavia-files.retry' | 15:45 |
| *** maciejjozefczyk is now known as mjozefcz|afk | 16:04 | |
| *** gcheresh has quit IRC | 16:05 | |
| cgoncalves | hkominos, sudo ansible-playbook -i "/var/lib/mistral/overcloud/octavia-ansible/inventory.yaml" --extra-vars @"/var/lib/mistral/overcloud/octavia-ansible/group_vars/octavia_vars.yaml" /usr/share/tripleo-common/playbooks/octavia-files.yaml --private-key "/var/lib/mistral/overcloud/ssh_private_key" | 16:07 |
| cgoncalves | should be applicable to rocky | 16:08 |
| cgoncalves | actually, that is for Rocky yeah | 16:08 |
| hkominos | sudo ? | 16:11 |
| hkominos | I run it from within mistral_executor | 16:11 |
| hkominos | xm. now it run somehow and bypassed the previous error | 16:13 |
| hkominos | and hit another error on a controller | 16:13 |
| hkominos | sayting that ssh key changed | 16:13 |
| hkominos | https://pastebin.com/mtNRTMJR | 16:13 |
| hkominos | xmmm | 16:14 |
| cgoncalves | that I'm not sure, sorry | 16:15 |
| cgoncalves | hkominos, can't you just run overcloud_deploy.sh again? | 16:15 |
| hkominos | of course. This still happens. that is what I am trying to disect | 16:15 |
| hkominos | a wait | 16:16 |
| hkominos | yeah I rerun the deployment. If i remove octavia it works like a charm | 16:16 |
| hkominos | only when I enable octavia I hit this | 16:16 |
| cgoncalves | yeah, it's something specific to octavia in tripleo so makes sense | 16:17 |
| cgoncalves | can you share the content of /var/lib/mistral/overcloud/octavia-ansible/group_vars/octavia_vars.yaml please? redact any sensitive information | 16:17 |
| hkominos | sure. I was just looking at that file. | 16:18 |
| cgoncalves | that last error (keys changed), I really don't know. it's just a SSH from the undercloud to controller-2 | 16:18 |
| *** goldyfruit has joined #openstack-lbaas | 16:20 | |
| hkominos | https://pastebin.com/HnEQTKRZ | 16:21 |
| hkominos | btw I thought that ansible is using the ssh keys of the stack user | 16:21 |
| *** gregwork has joined #openstack-lbaas | 16:21 | |
| hkominos | but the private key found in /var/lib/mistral/overcloud is not hte same as my stack user. | 16:22 |
| hkominos | IS that default behaviour | 16:22 |
| hkominos | ? | 16:22 |
| cgoncalves | tripleo is in some places expected to run from the "stack" user. probably the case for Octavia | 16:27 |
| cgoncalves | it may escalate to "root" at some parts, I don't know | 16:27 |
| cgoncalves | hkominos, can you get around the ssh key changed issue and re-run the same ansible-playbook command? | 16:28 |
| *** yamamoto has quit IRC | 16:28 | |
| *** yamamoto has joined #openstack-lbaas | 16:30 | |
| hkominos | I can try | 16:32 |
| *** yamamoto has quit IRC | 16:35 | |
| hkominos | ok something went horribly wrong. All the nodes are unreachable from ansible | 16:41 |
| cgoncalves | could you expand? what have you changed? | 16:49 |
| *** sapd1_x has joined #openstack-lbaas | 17:00 | |
| *** sapd1_x has quit IRC | 17:09 | |
| hkominos | nothing really. I just tried to rerun openstsack_ansible_command.sh and none of the nodes is reachable. I SPECULATE that as part of the command that we run before (https://pastebin.com/mtNRTMJR) we might have accidentally created a new key for the ansible-user. But that key is not injected to the undercloud nodes. therefore we fail. | 17:14 |
| hkominos | I rekicked a deployment to see what is going on | 17:14 |
| *** goldyfruit has quit IRC | 17:17 | |
| cgoncalves | the octavia ansible playbook does not touch system-wide ssh keys, so very unlikely unless I'm missing something | 17:23 |
| *** goldyfruit has joined #openstack-lbaas | 17:43 | |
| *** ramishra has quit IRC | 17:48 | |
| *** openstackgerrit has joined #openstack-lbaas | 17:52 | |
| openstackgerrit | Mikhail Ushanov proposed openstack/octavia stable/rocky: Limit spares pool to the spare_amphora_pool_size https://review.opendev.org/699257 | 17:52 |
| *** goldyfruit has quit IRC | 17:58 | |
| *** psachin has quit IRC | 18:19 | |
| *** salmankhan has quit IRC | 18:33 | |
| *** gcheresh has joined #openstack-lbaas | 18:37 | |
| *** rpittau is now known as rpittau|afk | 18:55 | |
| *** tesseract has quit IRC | 19:06 | |
| *** gcheresh_ has joined #openstack-lbaas | 19:52 | |
| *** gcheresh has quit IRC | 19:56 | |
| *** gcheresh_ has quit IRC | 19:58 | |
| *** gcheresh_ has joined #openstack-lbaas | 19:58 | |
| *** mjozefcz|afk has quit IRC | 19:59 | |
| *** gcheresh_ has quit IRC | 20:17 | |
| *** gregwork has quit IRC | 20:34 | |
| openstackgerrit | Brian Haley proposed openstack/octavia master: Fix tests to correctly call reset_mock() https://review.opendev.org/699287 | 20:45 |
| *** salmankhan has joined #openstack-lbaas | 20:50 | |
| *** salmankhan has quit IRC | 20:54 | |
| *** salmankhan has joined #openstack-lbaas | 20:54 | |
| *** yamamoto has joined #openstack-lbaas | 21:18 | |
| *** rcernin has joined #openstack-lbaas | 21:21 | |
| *** yamamoto has quit IRC | 21:23 | |
| *** TrevorV has quit IRC | 21:35 | |
| *** goldyfruit has joined #openstack-lbaas | 21:44 | |
| *** pcaruana has quit IRC | 21:55 | |
| openstackgerrit | Brian Haley proposed openstack/octavia master: Support hacking 2.0.0 https://review.opendev.org/699302 | 21:59 |
| *** mloza has joined #openstack-lbaas | 22:20 | |
| *** rcernin has quit IRC | 22:39 | |
| *** rcernin has joined #openstack-lbaas | 22:39 | |
| *** tkajinam has joined #openstack-lbaas | 23:07 | |
| *** salmankhan has quit IRC | 23:13 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!