| michchap | hello, I'm trying to do separated control/compute upgrades for rocky->stein, but am hitting an error when upgrading a single compute node since --limit removes the control nodes and thus the nova upgrade check that is meant to run on groups['nova-api'][0] doesn't run. Am I missing something or is using --limit to upgrade compute nodes one by one not supported? | 00:31 |
|---|---|---|
| michchap | I also noticed that the delegation of gather-facts in --limit scenarios causes a single unreachable node not within the limit to cause another (random) node to be excluded from everything since it will fail, but proceed. I'll file a bug for that one. | 00:33 |
| EugenMayer | Good morning. Deployed openstack with TLS yesterday (from scratch, not an TLS upgrade). AFAICS everything is TLS except the tty console. It does not show up when clicked, since the controller tryies https:// - when i click on the dedicated tty console window, it does not load (no socket), when i change to http it connects (but credentials are | 07:44 |
| EugenMayer | wrong). Seems like some services are not correctly configured to use tlss? | 07:44 |
| EugenMayer | not sure which api should run on :6080 | 07:46 |
| mnasiadka | dmsimard: ugh, nice. | 08:00 |
| mnasiadka | kolla-ansible stable/ussuri patches: https://review.opendev.org/q/branch:stable/ussuri+status:open+project:openstack/kolla-ansible - what do we need to merge yoctozepto ? | 08:05 |
| *** amoralej|off is now known as amoralej | 08:26 | |
| EugenMayer | anybody uses TLS and can connect to novnc. I seem to have a very similar issue as this one https://bugzilla.redhat.com/show_bug.cgi?id=1722089. I'am using officially signed certificates for all services | 09:40 |
| jingvar | let me check | 09:42 |
| EugenMayer | Seems like something from https://docs.openstack.org/nova/xena/admin/remote-console-access.html might be missing in the kolla setup | 09:42 |
| EugenMayer | checking the nova-novnc logs i see https://gist.github.com/EugenMayer/82528fcfca6e22b818f865852606f28c | 09:44 |
| jingvar | victoria letsencrypt tls for all endpoints, can see graphical console via web browser via admin url | 09:51 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Build overcloud host image directly with DIB https://review.opendev.org/c/openstack/kayobe/+/772609 | 09:59 |
| yoctozepto | mnasiadka: I have no idea what you need to merge me :-) what's the question? :D | 10:03 |
| EugenMayer | jingvar could you paste our nova-novncproxy/nova.conf config like this https://gist.github.com/EugenMayer/fba4eb20a49ccd717ba70f38188a8e1e .. please be sure to remove the passwords. Thanks | 10:03 |
| mnasiadka | yoctozepto: just thinking we should merge/abandon/whatever and mark as EM, right? | 10:03 |
| priteau | mnasiadka: I've updated the Kayobe CI status to GREEN | 10:07 |
| yoctozepto | mnasiadka: ah, in that context | 10:10 |
| yoctozepto | ok, let me see then | 10:10 |
| yoctozepto | mnasiadka; we have open changes on train as well | 10:32 |
| yoctozepto | so we can leave them be | 10:32 |
| yoctozepto | I can abandon the CI ones obviously | 10:32 |
| mnasiadka | well, rather if we want to merge anything on Ussuri before EM (last release) | 10:32 |
| mnasiadka | right | 10:32 |
| yoctozepto | I w+1'ed one of the trivials | 10:32 |
| yoctozepto | others not really ready | 10:32 |
| yoctozepto | so meh | 10:33 |
| opendevreview | Merged openstack/kolla-ansible master: mariadb: use add_host to include inactive hosts in shard grouping https://review.opendev.org/c/openstack/kolla-ansible/+/814276 | 11:02 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/xena: mariadb: use add_host to include inactive hosts in shard grouping https://review.opendev.org/c/openstack/kolla-ansible/+/816652 | 11:04 |
| EugenMayer | jingvar: try openstack console url show <instanceNameOrId> - does it show an https:// url? | 11:19 |
| EugenMayer | with the help in nova it is pretty clear right now, that kolla does and did not support TLS on vnc, since the configuration of it seems to be missing and copying the certificates is missing too | 11:21 |
| EugenMayer | either https://github.com/openstack/kolla-ansible/blob/stable/wallaby/ansible/roles/nova-cell/templates/nova.conf.j2#L2 should have cert/key as value if TLS is active (to a custom path) or coping them to self.pem is missing. Anyway, obviously not used at all, so less of an interest here | 11:27 |
| opendevreview | Merged openstack/kolla-ansible stable/ussuri: Fix missing Ansible version in the error message https://review.opendev.org/c/openstack/kolla-ansible/+/815809 | 11:27 |
| opendevreview | Seena Fallah proposed openstack/kolla-ansible stable/wallaby: mariadb: use add_host to include inactive hosts in shard grouping https://review.opendev.org/c/openstack/kolla-ansible/+/816653 | 11:30 |
| jingvar | EugenMayer: nova-novncproxy/nova.conf seems the same | 11:33 |
| jingvar | EugenMayer: https://my.cloud:6080/vnc_auto.html?path=%3Ftoken%XXXXXXXXXXXXXXXXXXX&title=nfvm(4450304a-588c-4b06-9a33-57fac246f20f) | 11:34 |
| hrw | mnasiadka: https://review.opendev.org/c/openstack/kolla/+/816374 maybe? | 11:35 |
| jingvar | EugenMayer: nova-novnc is behind haproxy | 11:37 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Fix missing Ansible version in the error message https://review.opendev.org/c/openstack/kolla-ansible/+/815806 | 11:48 |
| opendevreview | Merged openstack/kolla-ansible stable/xena: Fix missing Ansible version in the error message https://review.opendev.org/c/openstack/kolla-ansible/+/815805 | 11:50 |
| opendevreview | Merged openstack/kolla-ansible stable/victoria: Fix missing Ansible version in the error message https://review.opendev.org/c/openstack/kolla-ansible/+/815807 | 11:50 |
| *** amoralej is now known as amoralej|lunch | 11:57 | |
| mnasiadka | hrw: sure, but it's a bit weird that we have two sources for the same package... | 12:05 |
| yoctozepto | mnasiadka: it's pretty normal in general, we just need to prefer the variant we want :-) | 12:31 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] CI: Enable TLS in all jobs https://review.opendev.org/c/openstack/kolla-ansible/+/782387 | 12:35 |
| hrw | mnasiadka: a bit, yes | 12:39 |
| opendevreview | Radosław Piliszek proposed openstack/kolla master: Use MariaDB 10.6 https://review.opendev.org/c/openstack/kolla/+/811015 | 12:53 |
| *** amoralej|lunch is now known as amoralej | 13:17 | |
| kevko | hello gentlemen | 13:21 |
| opendevreview | Merged openstack/kolla master: Debian: we want Openvswitch and ovn from backports https://review.opendev.org/c/openstack/kolla/+/816374 | 13:31 |
| hrw | mnasiadka: do we want it in Xena too? | 14:17 |
| mnasiadka | hrw: I guess so, if we don't - we're bound to that one version that we're installing now? | 14:17 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla stable/xena: Debian: we want Openvswitch and ovn from backports https://review.opendev.org/c/openstack/kolla/+/816657 | 14:18 |
| hrw | let's check how it builds | 14:18 |
| EugenMayer | jingvar which HAproxy? there is non running on the container | 15:31 |
| yoctozepto | heads-up: adjutant is going retired, no contributors to care about it | 15:40 |
| yoctozepto | mnasiadka: ^ | 15:43 |
| jingvar | EugenMayer: ctl01 : ls /etc/kolla/haproxy/services.d | 16:02 |
| mnasiadka | yoctozepto: once it's retired we'll abandon the change and that's it | 16:10 |
| EugenMayer | jingvar ok interesting, backend points to the non VIP address, while frontend is bound on the VIP address - this explains it, thank yoy | 16:11 |
| jingvar | when I logged in via admin horizon url - should be the novnc url accordingly via admin? | 16:12 |
| EugenMayer | i think the point is what kolla_internal_fqdn and kolla_external_fqdn is configured to, i guess | 16:14 |
| jingvar | In my mind all useful - users services should work - in case HA, it means via VIP | 16:14 |
| jingvar | EugenMayer: ofcourse | 16:15 |
| EugenMayer | what is kolla_external_vip_address used for in general? i guess the services use kolla_internal_vip_address to talk right, so when the nova agents and those talk, they use they just internal? is the kolla_external_vip_address just for the access to horizon? | 16:16 |
| EugenMayer | looking at the HAproxy all services see to be bound to external | 16:17 |
| EugenMayer | oh wait, that's internal. Odd, what i do not get is how the 'initial IP on the management network' is considered inte setup | 16:18 |
| EugenMayer | so my cluster network, before provisioning openstack, has 3 nodes, controller (10.0.0.2), compute1(.3), compute2(.4). I configure the internal_vip to be .253 | 16:19 |
| EugenMayer | looking at the haproxy, all frontends are bound to .253 and all backends are bound to 10.0.0.2 (on the controller) | 16:19 |
| jingvar | there are 2 places to set fqdn | 16:21 |
| jingvar | internal vip in your case should be controller ip | 16:22 |
| EugenMayer | which one, .2? | 16:23 |
| EugenMayer | that cannot be done, when deploying, internal VIP must be an unused ip | 16:23 |
| jingvar | yep | 16:23 |
| EugenMayer | so i cannot use .2 for internal_vip, i have to use something else | 16:23 |
| jingvar | hmm, disable HA | 16:23 |
| jingvar | I don't rememmber how works AIO | 16:24 |
| jingvar | which step fails with internal=controller | 16:25 |
| EugenMayer | what setting disables ha? | 16:25 |
| jingvar | kolla_enable_haproxy: yes kolla_enable_keepalived: yes | 16:28 |
| jingvar | ^^^for enable ha | 16:28 |
| EugenMayer | i guess with multinode it is enabled by default | 16:30 |
| EugenMayer | before i do that i would really like to understand the semantic meaning of internal and external VIP | 16:30 |
| jingvar | extrernal network should be filtered | 16:33 |
| EugenMayer | https://docs.openstack.org/kolla/newton/advanced-configuration.html seems like both are the 'targeted' internal network, but one can have separated networks for internal and external | 16:33 |
| jingvar | if you want direct accsess for services you use intenal | 16:34 |
| EugenMayer | well then i'am confused why all backend definitions in haproxy use the 'initial IP on the management plan interface, the frontend use the internal_vip definition | 16:35 |
| EugenMayer | so 10.0.0.3 for the backend, 10.0.0.253 for the frontend - why is the backend not using .253? | 16:35 |
| EugenMayer | so frontend uses | 16:36 |
| EugenMayer | bind 10.0.0.253:6080 ssl crt /etc/haproxy/haproxy-internal.pem | 16:36 |
| EugenMayer | and backend uses server controller 10.0.0.3:6080 check inter 2000 rise 2 fall 5 | 16:36 |
| jingvar | you should think about kolla-ansible as a tool to configure openstack . the main is openstack doc | 16:36 |
| EugenMayer | AFAIU this is for HA then. since we only have one upstream, it is the 'local one' identified by .3 - the LB runs on .253 and forwards to the upstreams | 16:37 |
| EugenMayer | jingvar that really wont work. since the OpenStack docs are not talking about those containers. they have no intneral/external definition. Kolla is a macro level, which is nice, since it saves you a lot of time. But this hides details and to decrypt those takes some effort (fair enough) | 16:38 |
| EugenMayer | for example, in a non kolla env the nova-novnc service exposes it's cerficate for tLS directly (so i got told in nova today), which in kolla we have an haproxy for SSL offloaind and HA | 16:39 |
| jingvar | i've spent 5months for it :) | 16:39 |
| EugenMayer | so readind novas novnc docs will not work for kolla, since HAproxy makes a huge difference. That is what makes it difficult for the starter i think - or it is just me | 16:40 |
| jingvar | not only for you :) | 16:41 |
| EugenMayer | so what i did now is, my initial fqdn (controller.cluster.domain.tld) points to .3, internal and external VIP are .253 and i define the internal fqdn to controller.cluster.domain.tld, but the external to openstack.cluster.domain.tld (points to .253) | 16:43 |
| EugenMayer | using openstack.cluster.domain.tld to access the UI then works, since openstack.cluster.domain.tld is used for novnc | 16:44 |
| jingvar | in that case I go reverse path | 16:45 |
| jingvar | I understand what should be in a config and look how it generetes in kolla ansible scripts | 16:46 |
| EugenMayer | i think my biggest mistake was using controller for my hostname for the controller | 16:46 |
| EugenMayer | it should have been controller1 | 16:46 |
| EugenMayer | then my internal fqdn would be controller.cluster pointing to .253, controller1 to .3 and external is not needed at all | 16:47 |
| jingvar | I use ctl01 etc because if you need grep controller .... | 16:47 |
| EugenMayer | let's see if i can fix that with a reconfiguration or redo the entire cluster like i did yesterday (to change the passwords) | 16:50 |
| jingvar | in case find message about all my controllers I can use "ctl0" | 16:50 |
| EugenMayer | i see | 16:50 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Systemd container control PoC https://review.opendev.org/c/openstack/kolla-ansible/+/816724 | 16:51 |
| EugenMayer | jingvar thank you for your insight and helk! | 16:55 |
| EugenMayer | s/helk/help | 16:55 |
| jingvar | YAW | 16:57 |
| EugenMayer | nice, redeploying and fixing the hostname just worked | 17:21 |
| *** amoralej is now known as amoralej|off | 18:05 | |
| rainwadj | Does anything happen here, outside the scheduled meeting time? | 20:26 |
| opendevreview | Adrian Andreias proposed openstack/kolla-ansible master: Add kolla-ansible --version option https://review.opendev.org/c/openstack/kolla-ansible/+/816748 | 20:28 |
| opendevreview | Adrian Andreias proposed openstack/kolla-ansible master: docs: Fix python-openstackclient package name and init-runonce path https://review.opendev.org/c/openstack/kolla-ansible/+/816074 | 20:33 |
| opendevreview | Adrian Andreias proposed openstack/kolla-ansible master: docs: Install openstack-client with upper constraints https://review.opendev.org/c/openstack/kolla-ansible/+/816076 | 21:01 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!