| *** hindret has quit IRC | 00:03 | |
| *** elduderino80 has quit IRC | 00:03 | |
| *** hindret has joined #openstack-ansible | 00:04 | |
| *** macz_ has quit IRC | 00:18 | |
| *** mmethot has quit IRC | 00:19 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_manila master: Define condition for the first play host one time https://review.opendev.org/760715 | 00:58 |
|---|---|---|
| *** fanfi has quit IRC | 00:58 | |
| *** gyee has quit IRC | 01:02 | |
| *** macz_ has joined #openstack-ansible | 01:11 | |
| *** macz_ has quit IRC | 01:16 | |
| *** cshen has quit IRC | 01:18 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Add ability to define network MTU https://review.opendev.org/760187 | 01:23 |
| *** NewJorg has quit IRC | 01:25 | |
| *** NewJorg has joined #openstack-ansible | 01:27 | |
| *** macz_ has joined #openstack-ansible | 01:33 | |
| *** macz_ has quit IRC | 01:37 | |
| *** cloudxtiny has quit IRC | 01:53 | |
| *** ThiagoCMC has quit IRC | 02:02 | |
| *** cshen has joined #openstack-ansible | 02:14 | |
| *** cshen has quit IRC | 02:18 | |
| *** ThiagoCMC has joined #openstack-ansible | 02:27 | |
| *** yolanda has quit IRC | 02:57 | |
| *** cshen has joined #openstack-ansible | 04:14 | |
| *** cshen has quit IRC | 04:19 | |
| *** macz_ has joined #openstack-ansible | 04:34 | |
| *** macz_ has quit IRC | 04:39 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:33 | |
| *** yasemind1 has joined #openstack-ansible | 05:54 | |
| *** cshen has joined #openstack-ansible | 06:14 | |
| *** cshen has quit IRC | 06:19 | |
| *** recyclehero has quit IRC | 06:26 | |
| *** recyclehero has joined #openstack-ansible | 06:28 | |
| *** macz_ has joined #openstack-ansible | 06:28 | |
| *** macz_ has quit IRC | 06:33 | |
| *** recyclehero has quit IRC | 06:38 | |
| *** recyclehero has joined #openstack-ansible | 06:39 | |
| *** miloa has joined #openstack-ansible | 06:58 | |
| *** cshen has joined #openstack-ansible | 07:00 | |
| *** cshen has quit IRC | 07:04 | |
| *** cshen has joined #openstack-ansible | 07:22 | |
| *** jbadiapa has quit IRC | 07:33 | |
| *** pcaruana has quit IRC | 07:43 | |
| *** pcaruana has joined #openstack-ansible | 07:55 | |
| *** spatel has joined #openstack-ansible | 07:58 | |
| *** spatel has quit IRC | 08:03 | |
| *** sshnaidm|afk is now known as sshnaidm|rover | 08:08 | |
| noonedeadpunk | mornings | 08:08 |
| *** andrewbonney has joined #openstack-ansible | 08:09 | |
| *** jbadiapa has joined #openstack-ansible | 08:09 | |
| *** macz_ has joined #openstack-ansible | 08:16 | |
| *** shyamb has joined #openstack-ansible | 08:18 | |
| *** shyam89 has joined #openstack-ansible | 08:18 | |
| *** shyamb has quit IRC | 08:19 | |
| *** shyam89 has quit IRC | 08:19 | |
| *** shyam89 has joined #openstack-ansible | 08:20 | |
| *** shyamb has joined #openstack-ansible | 08:20 | |
| *** shyamb has quit IRC | 08:20 | |
| *** shyam89 has quit IRC | 08:20 | |
| *** macz_ has quit IRC | 08:20 | |
| *** shyamb has joined #openstack-ansible | 08:21 | |
| *** pto has joined #openstack-ansible | 08:34 | |
| *** shyam89 has joined #openstack-ansible | 08:37 | |
| *** shyamb has quit IRC | 08:39 | |
| *** tosky has joined #openstack-ansible | 08:42 | |
| *** yasemind1 has quit IRC | 08:49 | |
| *** rpittau|afk is now known as rpittau | 09:05 | |
| pto | Why does the haproxy_keepalived_internal_vip_cidr includes a netmask? I cant find anything in the keepalive docs why is defined? | 09:13 |
| noonedeadpunk | iirc it's needed to add VIP to the interface. I think without netmask it will be /32 but I'd rather specify it explicitly | 09:18 |
| *** macz_ has joined #openstack-ansible | 09:24 | |
| pto | iirc? | 09:25 |
| noonedeadpunk | if i recall correctly | 09:25 |
| pto | oh. thx | 09:25 |
| pto | But what is the purpose of specifying a mask on the VIP address? | 09:28 |
| noonedeadpunk | Well it's needed to be added as alias to the interface | 09:28 |
| *** macz_ has quit IRC | 09:29 | |
| noonedeadpunk | and maybe you want vip from another network that is already used on the interface (ie ddos protected one) | 09:29 |
| noonedeadpunk | and to make this network routable you will need to provide netmask | 09:30 |
| noonedeadpunk | but agree that generally it's not probably required and you probavbly can try omiting netmask to get it /32 | 09:33 |
| *** pto has quit IRC | 09:46 | |
| *** pto has joined #openstack-ansible | 09:46 | |
| *** shyamb has joined #openstack-ansible | 09:48 | |
| *** shyam89 has quit IRC | 09:49 | |
| pto | noonedeadpunk: thanks for the clarifications. I suggest we remove it from the examples, or at least document it | 09:52 |
| noonedeadpunk | not sure it makes much sense, since in docs we're following general instructions for keepalived configuration. If you look through all keepalived howtos you will find that all of them have netmask provided | 09:53 |
| noonedeadpunk | well, redhat is not https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/load_balancer_administration/ch-initial-setup-vsa | 09:55 |
| noonedeadpunk | anyway I don't think it matters much, but you can suggest change if you feel it's worth it | 09:55 |
| pto | noonedeadpunk: Agree. Its just rather confusing the first time you deploy openstack - i recall. | 09:56 |
| pto | noonedeadpunk: https://keepalived.readthedocs.io/en/latest/case_study_failover.html and https://keepalived.readthedocs.io/en/latest/configuration_synopsis.html#vrrp-instance-definitions-synopsis specifies @ip | 09:57 |
| *** fanfi has joined #openstack-ansible | 09:57 | |
| pto | Just a curious question, why do you run python3 -m http.server some seconds before certbot? | 09:58 |
| noonedeadpunk | I think jrosser may explain more, but I think the reason here that in order to start haproxy you need ssl configured, and to configure ssl you need webserver to pass verification. | 09:59 |
| *** macz_ has joined #openstack-ansible | 10:00 | |
| noonedeadpunk | so we spawn temorary web server to issue certificates which will be used so spawn haproxy which will serve for their futher renewals | 10:00 |
| noonedeadpunk | ah, well, you're right. just my google showed my old articels at first place, like https://docs.oracle.com/cd/E37670_01/E41138/html/section_uxg_lzh_nr.html | 10:01 |
| *** sauloasilva1 has quit IRC | 10:01 | |
| noonedeadpunk | but again, I don't think it matters, as it's just what will be passed to `ip address add` command, as without netmask it will be just /32 by default | 10:02 |
| noonedeadpunk | and feel free to push a patch to explain this in docs | 10:02 |
| *** fresta has joined #openstack-ansible | 10:03 | |
| pto | Make sense... Thanks for clarifying | 10:05 |
| *** macz_ has quit IRC | 10:05 | |
| *** yann-kaelig has joined #openstack-ansible | 10:06 | |
| openstackgerrit | Merged openstack/ansible-role-uwsgi master: Allow to globaly override uwsgi params https://review.opendev.org/761198 | 10:06 |
| *** fresta has quit IRC | 10:09 | |
| *** fresta has joined #openstack-ansible | 10:11 | |
| *** pto has quit IRC | 10:15 | |
| *** fresta_ has joined #openstack-ansible | 10:23 | |
| *** pto has joined #openstack-ansible | 10:23 | |
| *** yann-kaelig has quit IRC | 10:32 | |
| *** spatel has joined #openstack-ansible | 10:59 | |
| *** pto has quit IRC | 11:04 | |
| *** spatel has quit IRC | 11:04 | |
| *** fresta has quit IRC | 11:08 | |
| *** pto has joined #openstack-ansible | 11:14 | |
| pto | Keepalived_vrrp[154790]: (Line 37) VRRP parsed invalid IP strato-new.claaudia.aau.dk. skipping IP... | 11:18 |
| pto | haproxy_keepalived_external_vip_cidr: "{{external_lb_vip_address}}" | 11:18 |
| pto | external_lb_vip_address: strato-new.claaudia.aau.dk | 11:19 |
| pto | Is the external_lb_vip_address supposed to be an fqdn or an IP? The example suggest an fqdn | 11:27 |
| noonedeadpunk | external_lb_vip_address should be fqdn if needed, but haproxy_keepalived_external_vip_cidr needs to be ip | 11:32 |
| noonedeadpunk | well it's not cshould but can and preffered to be fqdn if you're going to use SSL | 11:33 |
| pto | noonedeadpunk: https://github.com/openstack/openstack-ansible/blob/b98646c10121b03a93d1b4e644b90f10e286474c/etc/openstack_deploy/user_variables.yml#L176 | 11:36 |
| pto | noonedeadpunk: The other examples are fine. | 11:36 |
| *** shyamb has quit IRC | 11:44 | |
| *** NewJorg has quit IRC | 11:48 | |
| noonedeadpunk | well, it can be set that way only if external_lb_vip_address is ip... | 11:55 |
| noonedeadpunk | maybe worth mentioning that | 11:55 |
| pto | I think there is a problem with the letsencryp | 11:56 |
| pto | I think there is a problem with the letsencrypt play: Problem binding to port 80: Could not bind to IPv4 or IPv6. | 11:57 |
| pto | shell: timeout 5 python3 -m http.server 8888 --bind 172.21.212.11 || true && certbot certonly --standalone --agree-tos --non-interactive --text --rsa-key-size 4096 --email perat@its.aau.dk --domains strato-new.claaudia.aau.dk | 11:57 |
| pto | This will make the certbot try to bind on port 80, which is allready bound by haproxy | 11:57 |
| pto | Or did i miss some logic? | 11:58 |
| *** NewJorg has joined #openstack-ansible | 12:00 | |
| noonedeadpunk | I think it should also have --http-01-port and --http-01-address defined | 12:04 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/tasks/haproxy_ssl_letsencrypt.yml#L64-L83 | 12:04 |
| noonedeadpunk | and haproxy_ssl_letsencrypt_certbot_backend_port is 8888 | 12:04 |
| noonedeadpunk | in U it was handled with haproxy_ssl_letsencrypt_setup_extra_params | 12:06 |
| *** rh-jelabarre has quit IRC | 12:06 | |
| noonedeadpunk | would be awesome to get https://review.opendev.org/#/c/757937/ merged with all cherry-picks | 12:15 |
| pto | noonedeadpunk: Certbot is executed on each infra host in sequence, and the haproxy_ssl_letsencrypt_certbot_bind_address should then be a hostvar of the host running the certbot. I dont see why certbot should be called three times and why you should define a hostvar for each host? | 12:18 |
| pto | noonedeadpunk: Why not just generate the certs on infra[0] and then copy them to the others? | 12:19 |
| noonedeadpunk | it's usually just set `haproxy_ssl_letsencrypt_certbot_bind_address: "{{ ansible_host }}"` | 12:19 |
| noonedeadpunk | not much reason of doing that, except it was implemented that way one day. from other side it ensures that certbot is working and will be able to care about certificate renewal later on | 12:21 |
| noonedeadpunk | anyway we have huge ssl refactoring topic for the upcoming release | 12:22 |
| noonedeadpunk | btw I'm not sure how it's all applicable for non-master... | 12:22 |
| pto | noonedeadpunk: I guess that will fix the problem. Cant try it now, as I have used my le quota for today | 12:23 |
| noonedeadpunk | yeah:( it's really a problem:( | 12:23 |
| pto | noonedeadpunk: But thanks allot for helping out anyway. I dont think the current stable is working without the haproxy_ssl_letsencrypt_certbot_bind_address: "{{ ansible_host }}" as it will try to bind on the same port as haproxy uses | 12:26 |
| noonedeadpunk | pto: yeah in U it was `haproxy_ssl_letsencrypt_setup_extra_params: "--http-01-address {{ ansible_host }} --http-01-port 8888"` | 12:26 |
| noonedeadpunk | https://docs.openstack.org/openstack-ansible/ussuri/user/security/index.html#letsencrypt-certificates | 12:26 |
| noonedeadpunk | it get's simplified in master (V) | 12:27 |
| *** spatel has joined #openstack-ansible | 12:27 | |
| pto | noonedeadpunk: Awsome! I think it very hard to find the correct conf in the docs. There are som many different versions scattered in many different places | 12:28 |
| noonedeadpunk | well yes. but everyone is welcome to contribute here! we really don't have resources to take care about everything | 12:29 |
| *** spatel has quit IRC | 12:32 | |
| pto | noonedeadpunk: I am aware of that. Btw. you are doing a great job :-D | 12:32 |
| pto | noonedeadpunk: Huge improvements since my last install of queens 2+ years ago | 12:33 |
| *** yann-kaelig has joined #openstack-ansible | 12:40 | |
| *** pto has quit IRC | 12:45 | |
| *** mgariepy has quit IRC | 12:47 | |
| *** pto has joined #openstack-ansible | 12:53 | |
| jrosser | pto: certbot is run multiple times because each haproxy node is responsible for its own certificate | 12:56 |
| jrosser | we had user request for making the certs unique per node and also you have to consider what happens at renewal time | 12:56 |
| noonedeadpunk | jrosser: I think we still could do init on 1 host only and then for renewals they can be on their own? | 12:57 |
| noonedeadpunk | but yeah, I think we have such user story of having unique certs | 12:57 |
| jrosser | perhaps, but I’m not sure you can re-renew more than once from the same config, never tried that. there’s all sorts of keys and stuff get generated | 12:58 |
| jrosser | pto: the python web server is run for a few seconds on the backend port to make sure that the active haproxy sees that the certbot on the renewing (or initialising) haproxy node is “up” | 12:59 |
| jrosser | one haproxy will have the vip and needs to direct the challenge to whichever backend certbot is renewing. this is important because the ssl cert fqdn resolves to the vip, not the specific ip of each haproxy | 13:01 |
| *** rh-jelabarre has joined #openstack-ansible | 13:01 | |
| pto | jrosser: thanks for the explanation. It didnt work for me with 3 infra nodes. certbot binded on the same ports as haproxy. I have added haproxy_ssl_letsencrypt_setup_extra_params... which should resolve the binding problem. I cant test today because i have reached the le quota | 13:01 |
| jrosser | you should test this all out with —staging includes in the extra paeans | 13:02 |
| jrosser | then you get huge quota but invalid certs | 13:02 |
| jrosser | when you are happy with how it works remove —staging, delete all the LE files from the haproxy nodes and re-run to get valid ones | 13:03 |
| jrosser | in Ussuri really only basic support got added to haproxy role and lots of overrides were needed | 13:04 |
| jrosser | this is now much much cleaner in master branch | 13:04 |
| pto | Its very nice with the le feature :-) | 13:07 |
| *** rfolco has joined #openstack-ansible | 13:07 | |
| pto | jrosser: Why didnt you merge https://review.opendev.org/#/c/758845/ to master? | 13:08 |
| jrosser | I left a comment on it about needing to check if the libraries are compatible on Debian as well as Ubuntu | 13:09 |
| *** macz_ has joined #openstack-ansible | 13:09 | |
| jrosser | and I’ve not had time to do that.... | 13:10 |
| pto | jrosser: No worries. I can confirm it work perfectly. | 13:11 |
| jrosser | pto by the way the quotas for LE production and staging endpoints are independent so if you need to work on it more with the staging endpoint that will still work today | 13:14 |
| *** macz_ has quit IRC | 13:14 | |
| pto | jrosser: I have stopped haproxty & keepalived and run certbot on each node. So for now, it works as expected. I have left a note for myself to test renewal later | 13:14 |
| pto | Do you want me to do further testing on federated identity? | 13:17 |
| jrosser | if you have time to just try installing both oidc and saml packages on a debian10 vm and verify they both want libcurl4, that would mean we can merge / adjust the patch | 13:19 |
| pto | I dont have a setup available where I cant test that atm. I have a physical stage and a physical prod env i need to finish soon with SAML2 federated id. | 13:21 |
| pto | The stage setup works for now, with the workaround by commenting out keystone_federation_sp_idp_setup.yml on the first run of keystone-install.yml and then comment it in and run it again. | 13:21 |
| *** mgariepy has joined #openstack-ansible | 13:25 | |
| *** luksky has joined #openstack-ansible | 13:41 | |
| pto | Would it make sense to include https://github.com/openstack/openstack-ansible/blob/b98646c10121b03a93d1b4e644b90f10e286474c/playbooks/os-keystone-install.yml#L135 in the os_keystone role when federated identity is configured? | 13:42 |
| Adri2000 | should I use something other than "recheck" when the gate is failing? https://review.opendev.org/#/c/758413/ looks like when doing "recheck" it goes through the check queue again instead of "rechecking" the gate... | 13:53 |
| mgariepy | Adri2000, recheck is the way to do it. did you check what was the failure? | 14:05 |
| mgariepy | https://zuul.opendev.org/t/openstack/build/c9e2923c93bf41449d534797e561880b/log/job-output.txt#9574 failed to update apt cache. | 14:06 |
| mgariepy | :/ | 14:06 |
| Adri2000 | mgariepy: ok, so basically in order to merge the patch needs to pass the tests two times in a row? (check queue + gate queue) | 14:06 |
| Adri2000 | yes it's some failures unrelated to the patch itself :( | 14:06 |
| *** nurdie has quit IRC | 14:07 | |
| Adri2000 | it's failing 80% of the time or so :( so I've been trying to get it merged since October 19th (when it was approved)... | 14:07 |
| *** nurdie has joined #openstack-ansible | 14:07 | |
| mgariepy | do you know whihc repo fail to updatE? | 14:11 |
| *** nurdie has quit IRC | 14:12 | |
| mgariepy | fun. not always the same check that fails. | 14:14 |
| mgariepy | Adri2000, i'll try to monitor it and recheck if it fails. | 14:15 |
| *** spatel has joined #openstack-ansible | 14:25 | |
| mgariepy | Adri2000, https://docs.openstack.org/contributors/common/zuul-status.html#why-do-changes-go-first-in-the-check-queue | 14:25 |
| *** mmethot has joined #openstack-ansible | 14:25 | |
| *** pcaruana has quit IRC | 14:27 | |
| *** mmethot has quit IRC | 14:33 | |
| *** mmethot has joined #openstack-ansible | 14:33 | |
| spatel | jamesdenton: morning! | 14:35 |
| *** cloudxtiny has joined #openstack-ansible | 14:38 | |
| cloudxtiny | Hello all. question regarding CentOS 8. how do I install bridge-utils? I noticed a few of the openstack-ansible task seem to use it when configuring container networks | 14:40 |
| *** pcaruana has joined #openstack-ansible | 14:48 | |
| spatel | I don't think OSA by default install bridge-utils with CentOS-8 | 14:48 |
| spatel | noonedeadpunk: ^^ | 14:48 |
| spatel | we should add that utility because its very hand. | 14:49 |
| noonedeadpunk | spatel: isn't bridge-utils are just missing as class from centos 8? | 14:49 |
| noonedeadpunk | cloudxtiny: can you point to these tasks? As I think we should be using systemd-networkd everywhere | 14:50 |
| noonedeadpunk | bridge-utils is part of the https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.7_release_notes/deprecated_functionality | 14:51 |
| spatel | noonedeadpunk: this is the playbook right to rollout that right? https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/vars/redhat-host.yml | 14:51 |
| noonedeadpunk | rollout what? I think there's no official source from which bridge-utils can be installed for centos 8? | 14:54 |
| spatel | bridge-utils RPM is part of rdo-deps and we do have rdo-repo part of OSA so worth installing that tool | 14:58 |
| spatel | noonedeadpunk: ^ | 14:58 |
| noonedeadpunk | yeah, it's really in rdo-deps | 15:01 |
| noonedeadpunk | however, I'd install it rather only in case if it's needed for roles. otherwise I'd say everyone can use openstack_host_extra_metal_distro_packages variable to define packages theyd love to see installed on the hosts | 15:03 |
| noonedeadpunk | as we're aiming to install bare minimum of stuff | 15:04 |
| *** macz_ has joined #openstack-ansible | 15:06 | |
| spatel | openstack_host_extra_metal_distro_packages so i can define it in user_variable.yml right ? | 15:10 |
| noonedeadpunk | yep | 15:10 |
| *** macz_ has quit IRC | 15:11 | |
| spatel | what if i want to some utility inside my container (like tcpdump/netstat/iostat etc?) | 15:11 |
| noonedeadpunk | also we have openstack_host_extra_distro_packages but it's for all hosts (including lxc) | 15:11 |
| *** yann-kaelig has quit IRC | 15:12 | |
| *** mmethot has quit IRC | 15:19 | |
| *** mmethot has joined #openstack-ansible | 15:24 | |
| mgariepy | fun how the neutron agent doesn't work well in container .. | 15:28 |
| mgariepy | noonedeadpunk, would you mind if we add back that test? | 15:31 |
| noonedeadpunk | mgariepy: if I knew what test we're talking about:) | 15:32 |
| mgariepy | moving the neutron agent in a lxc container.. | 15:32 |
| mgariepy | maybe we could add it to the check queue, and not gate. | 15:33 |
| noonedeadpunk | and in our functional tests aren't agents in lxc? | 15:34 |
| mgariepy | the check queue does build aio .. | 15:35 |
| noonedeadpunk | (not sure) | 15:35 |
| noonedeadpunk | it's.... ovn related?:) just not sure what it is | 15:36 |
| mgariepy | https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-playbooks/neutron.yml#L54-L64 | 15:36 |
| *** macz_ has joined #openstack-ansible | 15:36 | |
| mgariepy | not really | 15:36 |
| noonedeadpunk | ah I see | 15:36 |
| mgariepy | this doesn't work on focal. | 15:36 |
| mgariepy | and the ovs deployment in container doesn't either. | 15:37 |
| noonedeadpunk | I can recall some ideas to abandon possibility to deploy neutron agents in lxc at all | 15:38 |
| mgariepy | i like to have the network natting service in a container and not on the hosts directly. | 15:38 |
| mgariepy | i know it's not quite a standard way of doing it but. it was like that a long time ago and i still prefer to do it that way. | 15:39 |
| spatel | Don't you think reason to run neutron-agent on metal was neutron l3 performance? | 15:40 |
| spatel | i may be wrong because i am not running l3-agent in my cloud | 15:40 |
| mgariepy | i don't have dedicated network nodes on all my setup. | 15:43 |
| noonedeadpunk | me neither actually | 15:43 |
| mgariepy | if i had that i would but it's not the case currently. | 15:43 |
| *** nurdie has joined #openstack-ansible | 15:43 | |
| noonedeadpunk | but l3 nat is done inside namespace isn't it? | 15:44 |
| mgariepy | yes. | 15:44 |
| noonedeadpunk | so kind of safe for host itself. but if you feel it's good usecase, we can make some test | 15:44 |
| noonedeadpunk | eventually we can probably all lxc CI do with neutron agent inside lxc as well | 15:45 |
| noonedeadpunk | and metal jobs should prevent things from being broken in default scenario | 15:46 |
| noonedeadpunk | or we can just add another integrated test for os_neutron role | 15:46 |
| noonedeadpunk | the only thing I'm not sure about is how to handle env.d file - can't recall if we just copy it or they can be templates | 15:47 |
| jrosser | there was a migration long ago to move the l3 agents out of containers | 15:49 |
| jrosser | iirc big difficulty when any maintanence restarted infra node lxc you got the l3 agent stopped/started | 15:51 |
| Adri2000 | mgariepy: thanks for that link, it's a good explanation | 15:51 |
| *** klamath_atx has joined #openstack-ansible | 15:54 | |
| openstackgerrit | Rafael Folco proposed openstack/openstack-ansible-os_tempest stable/train: Switch tripleo jobs to content provider https://review.opendev.org/761021 | 15:55 |
| *** nurdie has quit IRC | 16:02 | |
| cloudxtiny | another point. I want to install clouditty as part of my set. I have added the playbook to the opesntack-ansible conf.d playbook path, but I noticed when I run setup-host and setup-infra, it doesn't get added. Am i missing something? | 16:02 |
| *** nurdie has joined #openstack-ansible | 16:03 | |
| mgariepy | jrosser, yep, but that's why there is a lxc_container_restart variable ;) haha | 16:06 |
| *** nurdie has quit IRC | 16:07 | |
| *** pto has quit IRC | 16:08 | |
| *** pto has joined #openstack-ansible | 16:08 | |
| *** pto has quit IRC | 16:15 | |
| *** pto has joined #openstack-ansible | 16:15 | |
| ThiagoCMC | Hey guys, I'm trying to install OSA Ussuri with qdrouterd but the TASK "qdrouterd : Install Qpid Dispatch Router (qdrouterd) packages" is failing, error: '"No package matching 'python-qpid-proton' is available"', the correct package name is actually "python3-qpid-proton"! | 16:25 |
| ThiagoCMC | Any tips? ^_^ | 16:25 |
| ThiagoCMC | I'll try to just change the package name at /etc/ansible/roles/qdrouterd/vars/ubuntu.yml for now... | 16:26 |
| noonedeadpunk | cloudxtiny: nah, we just still didn't put cloudkitty playbook into our integrated repo as not much ppl using it | 16:29 |
| noonedeadpunk | you can use https://opendev.org/openstack/openstack-ansible-os_cloudkitty/src/branch/master/tests/test-install-cloudkitty.yml as sample and create it somewhere | 16:30 |
| noonedeadpunk | we will fix this in the next release | 16:30 |
| cloudxtiny | noonedeadpunk thank mate :-) | 16:30 |
| noonedeadpunk | ThiagoCMC: seems we suck :( | 16:31 |
| dmsimard | noonedeadpunk: I am told the openvswitch collection issue should be fixed "soon" | 16:32 |
| cloudxtiny | noonedeadpunk this might be a dumb question as I am still new to ansible :-). how do I run this cloudkitty playbook? | 16:32 |
| noonedeadpunk | cloudxtiny: well, you create file let's say /root/os-cloudkitty.yml and run it as `openstack-ansible /root/os-cloudkitty.yml` | 16:32 |
| noonedeadpunk | dmsimard: great news, thanks) | 16:33 |
| noonedeadpunk | really waiting for it:) | 16:33 |
| cloudxtiny | noonedeadpunk thanks. | 16:33 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Bump ansible version to 2.10.3 https://review.opendev.org/761443 | 16:34 |
| ThiagoCMC | noonedeadpunk, :-P | 16:35 |
| ThiagoCMC | It worked! | 16:35 |
| noonedeadpunk | ThiagoCMC: what release is that? | 16:35 |
| ThiagoCMC | stable/ussuri | 16:35 |
| noonedeadpunk | https://review.opendev.org/#/c/761444/ | 16:36 |
| noonedeadpunk | thanks for reporting it | 16:36 |
| ThiagoCMC | My pleasure! =) | 16:37 |
| noonedeadpunk | another vote on https://review.opendev.org/#/c/760188 would be awesome | 16:37 |
| ThiagoCMC | I'm interested on this idea: https://docs.openstack.org/openstack-ansible/ussuri/user/messaging/messaging.html | 16:38 |
| noonedeadpunk | oh, well, I think you will find more things then | 16:39 |
| ThiagoCMC | lol | 16:39 |
| *** djhankb has quit IRC | 16:40 | |
| *** djhankb has joined #openstack-ansible | 16:40 | |
| noonedeadpunk | I'm not sure we were paying enough attention to use appropriate variables everywhere, but would be gret to get it really tested by someone (not sure who might be using this scenario) | 16:41 |
| *** cloudxtiny has quit IRC | 16:41 | |
| *** pto has quit IRC | 16:43 | |
| *** pto has joined #openstack-ansible | 16:43 | |
| ThiagoCMC | I'll give it a try! My intention is to reduce the load out of RabbitMQ, I faced many problems with it in the past, where it consumes a lot of CPU and it's very hard to clean it up. | 16:45 |
| noonedeadpunk | actually if you don't have ceilometer or designate, qrouterd will just replace rabbit | 16:46 |
| noonedeadpunk | but yeah - let us know about how it goes) | 16:47 |
| ThiagoCMC | Hmm... Interesting! I don't have those two. | 16:47 |
| ThiagoCMC | I'll! | 16:47 |
| noonedeadpunk | notification is used by these 2 services mostly (if I'm not missing smth) | 16:48 |
| noonedeadpunk | so if you don't have them you won't be using notifications queue | 16:48 |
| ThiagoCMC | Got it | 16:49 |
| ThiagoCMC | I can double check this at, for example, nova.conf and others, right? | 16:50 |
| noonedeadpunk | yep | 16:50 |
| noonedeadpunk | we should have conditions in template like this https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/templates/nova.conf.j2#L62 everywhere | 16:51 |
| noonedeadpunk | otherwise rabbit really goes wild | 16:51 |
| noonedeadpunk | as it will has dozens of messages without any reader | 16:52 |
| noonedeadpunk | *will have | 16:52 |
| spatel | ThiagoCMC: i had same issue with my rabbitMQ and reducing all timer in neutron and nova help me a lot to reduce MQ cpu load | 16:52 |
| ThiagoCMC | Cool! Thanks spatel! | 16:53 |
| noonedeadpunk | so messeging driver should be noop in case you don't have ceilometer or designate | 16:53 |
| ThiagoCMC | Ok | 16:54 |
| ThiagoCMC | I'll test everything and pastebin the nova.conf/neutron.conf/etc files so you can also double check it | 16:55 |
| spatel | ThiagoCMC: This is what my timers looks reduce load of MQ - http://paste.openstack.org/show/799700/ | 16:58 |
| *** pto has quit IRC | 16:59 | |
| *** pto has joined #openstack-ansible | 16:59 | |
| ThiagoCMC | Awesome | 17:01 |
| *** mgariepy has quit IRC | 17:02 | |
| jrosser | i think noone uses qdrouterd with OSA at all tbh | 17:07 |
| jrosser | it's in the same category as nspawn, it's 'interesting' but never went beyond POC | 17:08 |
| jrosser | and notably theres no CI jobs for it at all really | 17:08 |
| noonedeadpunk | well, nspawn was used actually by several ppl | 17:09 |
| noonedeadpunk | so, who knows... | 17:09 |
| noonedeadpunk | maybe we should add some CI for it one day... | 17:09 |
| noonedeadpunk | as sometimes I feel like I'm done with rabbit | 17:10 |
| spatel | its used by amqp right? | 17:10 |
| jrosser | it would be interesting to prove qdrouterd was really a drop in replacement for rabbit | 17:11 |
| noonedeadpunk | but at the moment have no idea about qdrouterd.... | 17:11 |
| *** pto has quit IRC | 17:12 | |
| jrosser | i asked about this ant the london ops meetup and there was much concern about too much reliance on the specific behaviour of rabbitmq | 17:12 |
| *** pto has joined #openstack-ansible | 17:12 | |
| noonedeadpunk | I think everything using oslo.messaging? | 17:13 |
| noonedeadpunk | Wich is kind of abstraction layer | 17:13 |
| noonedeadpunk | but who knows how do they test qdrouterd support :) | 17:14 |
| *** tosky has quit IRC | 17:15 | |
| *** nurdie has joined #openstack-ansible | 17:22 | |
| ThiagoCMC | I'll test it! :-P | 17:26 |
| *** rpittau is now known as rpittau|afk | 17:35 | |
| *** pto has quit IRC | 17:36 | |
| *** pto has joined #openstack-ansible | 17:36 | |
| *** pto has quit IRC | 17:45 | |
| *** pto has joined #openstack-ansible | 17:45 | |
| *** klamath_atx has quit IRC | 18:09 | |
| *** klamath_atx has joined #openstack-ansible | 18:09 | |
| *** pto has quit IRC | 18:10 | |
| *** mgariepy has joined #openstack-ansible | 18:17 | |
| *** mugsie has quit IRC | 18:18 | |
| *** miloa has quit IRC | 18:18 | |
| *** mugsie has joined #openstack-ansible | 18:21 | |
| *** andrewbonney has quit IRC | 18:24 | |
| djhankb | Hey folks - I am doing some maintenance on my cluster and one of of my controllers I have had several of the containers lose their gateways and DNS | 18:34 |
| djhankb | Where would I check to see why the gateway would be missing: from within the container, "ip -4 route" shows only connected networks. | 18:35 |
| *** pto has joined #openstack-ansible | 18:42 | |
| *** pto has quit IRC | 18:44 | |
| *** pto has joined #openstack-ansible | 18:45 | |
| djhankb | actually it looks like my lxcbr0 is not working correctly on that controller | 18:48 |
| djhankb | figured it out, had to restart the lxc-dnsmasq service | 18:57 |
| noonedeadpunk | yeah, sorry for not answering earlier, but good that you've figured it out:) | 19:02 |
| *** luksky has quit IRC | 19:12 | |
| mgariepy | jamesdenton, is there any reason why ovs doesn't use the openvswitch firewall instead of iptables_hybrid ? | 19:18 |
| mgariepy | jamesdenton, https://github.com/openstack/openstack-ansible-os_neutron/blob/master/templates/plugins/ml2/openvswitch_agent.ini.j2#L33-L36 | 19:22 |
| *** luksky has joined #openstack-ansible | 19:24 | |
| dmsimard | noonedeadpunk: https://github.com/ansible-collections/openvswitch.openvswitch/pull/60/files | 19:27 |
| *** bl0m1 has joined #openstack-ansible | 19:34 | |
| *** NewJorg has quit IRC | 19:55 | |
| spatel | mgariepy: I have configured dpdk with ovs and cpu usage showing 100% for ovs-vswitchd. does it normal ? | 19:55 |
| *** NewJorg has joined #openstack-ansible | 19:56 | |
| *** pto has quit IRC | 19:57 | |
| *** pto has joined #openstack-ansible | 19:57 | |
| *** luksky has quit IRC | 20:01 | |
| mgariepy | spatel, i have not idea. | 20:03 |
| spatel | no worry | 20:03 |
| mgariepy | spatel, it's my first ovs install since .... fuel.. | 20:03 |
| spatel | i am on same boat, trying to learn everything about OVS before pushing out to production | 20:04 |
| *** pto has quit IRC | 20:04 | |
| *** pto has joined #openstack-ansible | 20:04 | |
| mgariepy | meh way easier to fix stuff under pressure no ? | 20:05 |
| spatel | :) | 20:07 |
| spatel | I am so happy with LinuxBridge that now hard to leave it. | 20:07 |
| spatel | is there any openvswitch IRC channel to talk ? | 20:09 |
| ThiagoCMC | spatel, yes, it's normal for an DPDK App to consume an entire CPU Core. | 20:09 |
| spatel | ThiagoCMC: thanks for confirmation. | 20:10 |
| ThiagoCMC | In my lab, with a Supermicro server, 2 x 10G, at least, 2 CPU Cores. With multiqueue, 4. | 20:10 |
| ThiagoCMC | NP! | 20:10 |
| spatel | ThiagoCMC: now i have big problem with bonding with DPDK | 20:10 |
| spatel | I have only 2x10G nic card so how do i bound them in OVS ? | 20:10 |
| spatel | because DPDK doesn't support bonding outside ovs right? | 20:11 |
| ThiagoCMC | You can bond with OVS with or without DPKD | 20:11 |
| ThiagoCMC | But with DPDK is really tricky | 20:11 |
| ThiagoCMC | I did this about 6 months ago... Don't remember the syntax :-( | 20:11 |
| spatel | How do i bond because i will loose connection right during building bonding | 20:12 |
| ThiagoCMC | Oh, good question... I always had an USB Ethernet as "OOBM Network", outside of OVS, so I don't kick myself out of it | 20:13 |
| *** luksky has joined #openstack-ansible | 20:13 | |
| spatel | ThiagoCMC: that is what i am thinking, how to do bonding while SSHing on same interface. (its not like during OS install you have it ready) | 20:14 |
| spatel | Even OSA will install openvswitch during playbook run | 20:15 |
| spatel | if i have 3rd nic then i can solve this issue but not with 2 nic | 20:15 |
| ThiagoCMC | spatel, your best bet would be to have an extra interface, or access the server's IPMI/iLo. Here is where I got docs about bonding with ovs dpdk: https://software.intel.com/content/www/us/en/develop/articles/link-aggregation-configuration-and-usage-in-open-vswitch-with-dpdk.html | 20:15 |
| spatel | I have 300 compute nodes and i can't do that via ILO :) i need something very automated process. | 20:16 |
| ThiagoCMC | true lol | 20:16 |
| ThiagoCMC | I wasn't aware that OSA is installing openvswitch now! That's... weird | 20:16 |
| spatel | OSA does install openvswitch during os-neutron-install.yml playbook | 20:17 |
| spatel | currently i am running my production workload on SRIOV (reason i was looking for dpdk because it support bonding but look like i hit the wall) | 20:17 |
| spatel | SR-IOV doesn't support bonding. | 20:18 |
| ThiagoCMC | I see... Well, keep in mind that ovs-dpdk is a much more complex environment! | 20:30 |
| ThiagoCMC | You must play with it without OpenStack... Just Ubuntu 20.40 host, ovs-dpdk and QEMU, nothing else. Then, you'll better understand how it works. | 20:31 |
| spatel | ThiagoCMC: totally with you | 20:33 |
| ThiagoCMC | =) | 20:34 |
| *** tosky has joined #openstack-ansible | 20:35 | |
| djhankb | spatel: are you using OVS for your controllers and container networking? | 20:38 |
| spatel | djhankb: currently in my lab i am just playing with OVS | 20:38 |
| spatel | I used LinuxBridge+OVS combo | 20:39 |
| spatel | br-mgmt in lxb and br-vlan/br-vxlan in OVS | 20:39 |
| djhankb | spatel: gotcha. I have been running OVS for my container networking, i.e. br-mgmt, br-vxlan, br-storage. And I have hit quite a few pitfalls... | 20:40 |
| spatel | OVS isn't fun to run until unless you have army of developer and code digger | 20:40 |
| djhankb | I'm not sure if this has been addressed yet, I'm still on Train/20.0.1 but when building containers there are assumptions made that you are using linuxbridge, and things like 'veth-cleanup.sh' will not delete the ports from OBX | 20:41 |
| djhankb | OVS | 20:41 |
| spatel | I am asking this question to everyone what is the biggest advantage of using OVS over lxb (for get about dpdk etc) | 20:41 |
| djhankb | For me - I initially set it up as I wanted to try out using Hyper-V for some of my compute nodes, and the Hyper-V VXlan did not work with Linuxbridge VXlan, and OVS was the happy medium as it worked on both. | 20:42 |
| spatel | make sense. | 20:46 |
| djhankb | That being said, everything about it is a horrible pain. Trying to get it all set up using netplan? Forget it, which means you'll have to set up /etc/network/interfaces and deal with that. Then there are custom scripts needed for container networking as when you stop a container, it doesn't automatically remove the port from OVS and then the | 20:48 |
| djhankb | container won't start until you manually remove it. | 20:48 |
| *** cshen has quit IRC | 20:48 | |
| djhankb | I just found another one, where if a container uses br-storage or if one of your bridges uses a VLAN tag, it won't automatically tag the port with the VLAN without editing the .ini file for the eth in the /usr/lib/lxc/foo_container directory | 20:49 |
| ThiagoCMC | Are you guys using Ussuri OSA's Ceph with Erasure Code? | 21:02 |
| ThiagoCMC | I'm curious about something from the OSA Ussuri release notes as well, can someone clarify it for me? Here: https://docs.openstack.org/releasenotes/openstack-ansible/ussuri.html - there is: "Any ceph infrastructure components (OSDs, MONs etc) deployed using the OSA/ceph-ansible tooling will be upgraded to the Ceph Nautilus release." ... | 21:04 |
| ThiagoCMC | However, Ceph on Ubuntu 20.04 is Octopus (15), not Nautilus! | 21:05 |
| *** fanfi has quit IRC | 21:06 | |
| *** cshen has joined #openstack-ansible | 21:07 | |
| jrosser | ThiagoCMC: the ceph packages are taken from download.ceph.com not from the distro iirc | 21:27 |
| ThiagoCMC | My file "/etc/openstack_deploy/user_variables-ceph.yml" contains: "ceph_origin: distro", so, it actually installed from Ubuntu itself, which is the way I expect... Is this bad from OSA point of view? | 21:28 |
| jrosser | possibly, a particular OSA release will bring in a specific major version of ceph-ansible, which in turn will imply a specific ceph major version | 21:30 |
| ThiagoCMC | Oh, okay | 21:30 |
| ThiagoCMC | Kinda bad to see OSA supporting Ubuntu 20.04 but downgrading Ceph... | 21:31 |
| kleini | ThiagoCMC: I am always deploying Ceph separately from OSA using ceph-ansible. This is very easy and allows then to run different versions. | 21:32 |
| jrosser | are you sure that is actually a release it’s from ussuri? | 21:32 |
| ThiagoCMC | kleini, cool, I'm thinking about doing that too! | 21:32 |
| ThiagoCMC | jrosser, I'm deploying from OSA's stable/ussuri git branch | 21:32 |
| jrosser | rather than a previous release note which is mistakenly on the ussuri release notes page? | 21:32 |
| ThiagoCMC | Good question... | 21:33 |
| ThiagoCMC | lol | 21:33 |
| kleini | djhankb: I use LXB on controller nodes for bridges and LXC containers and OVS on compute and network nodes for bridges and ML2 | 21:33 |
| jrosser | because something truly wierd has happened with the rendering of the release notes | 21:33 |
| ThiagoCMC | O_o | 21:33 |
| jrosser | and no one knows what has caused it | 21:33 |
| ThiagoCMC | Well, I just deployed Ceph 15 from OSA/Ussuri branch | 21:34 |
| ThiagoCMC | "ceph status" is healthy | 21:34 |
| jrosser | well here is the version which would be installed from downloads.ceph.com for ussuri https://github.com/openstack/openstack-ansible/blob/stable/ussuri/inventory/group_vars/all/ceph.yml#L30 | 21:35 |
| jrosser | so i think that the releasenote is unfortunately from Train | 21:35 |
| ThiagoCMC | Damn | 21:35 |
| ThiagoCMC | Ok! I'm at Ussuri with Octopus! | 21:35 |
| ThiagoCMC | The Ussuri's release notes is wrong! | 21:36 |
| jrosser | yes, it seems to have included releasenotes from previous releases | 21:36 |
| ThiagoCMC | Oh, I see... | 21:36 |
| jrosser | the infra/docs folk are not able to figure out why this happens | 21:36 |
| jrosser | this is all automated somehow in the docs generation | 21:37 |
| ThiagoCMC | Ok, no problem, good to know =) | 21:37 |
| jrosser | anyway - if you override the ceph version then thats fine, but you'll have to take care that the OSA / Ansible / ceph-ansible / ceph combination you are trying to deploy is valid | 21:37 |
| jrosser | those things are all dependant on each other, unfortunately | 21:38 |
| jrosser | this is one of the biggest reasons to decide to deploy ceph seperately without OSA, because you can decouple the versions and points you decide to upgrade completely | 21:39 |
| djhankb | kleini: I think that's probably the ideal way to use OVS. As I mentioned, my use case was sort of non-traditional, and I had to learn everything about it the hard way of course. | 21:41 |
| ThiagoCMC | jrosser, right, I'll try to deploy ceph-ansible first, and not call "ceph-install.yaml" from OSA | 21:43 |
| ThiagoCMC | jrosser, kleini are you guys using Erasure Code with Ceph pools for OpenStack? | 21:49 |
| jrosser | i don't have any EC pools currently | 21:49 |
| ThiagoCMC | ok | 21:50 |
| jrosser | everything is replica 3 right now | 21:50 |
| ThiagoCMC | 3 is kinda overkill for me | 21:50 |
| ThiagoCMC | My cloud is in my basement lol | 21:50 |
| jrosser | worth careful reading becasue i think for rbd you need the metadata on a replica pool anyway | 21:54 |
| jrosser | there is no omap on en ec pool | 21:54 |
| ThiagoCMC | Hmm... | 21:56 |
| ThiagoCMC | Researching about: ceph erasure code openstack | 21:57 |
| ThiagoCMC | =) | 21:57 |
| djhankb | on a deployment host, I could've swore there was a file containing the output of playbook execution. Does anyone recall where that would be? | 22:05 |
| *** spatel has quit IRC | 22:06 | |
| ThiagoCMC | jrosser, looks like that this: https://docs.ceph.com/en/latest/rados/operations/erasure-code/#erasure-coding-with-overwrites - is required to make sure OpenStack can use Erasure Coded pools! - Blog: https://themeanti.me/technology/2018/08/23/ceph_erasure_openstack.html | 22:27 |
| *** cshen has quit IRC | 22:27 | |
| *** cshen has joined #openstack-ansible | 22:29 | |
| *** rfolco has quit IRC | 22:30 | |
| *** luksky has quit IRC | 22:48 | |
| *** spatel has joined #openstack-ansible | 23:12 | |
| *** spatel has quit IRC | 23:17 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!