| jrosser | _jralbert: the neatest way to do that is to locally host an rpm repo and add it to the basic configuration you give your hosts when you pxeboot/provision them | 05:19 |
|---|---|---|
| jrosser | the extra repo config should be copied into lxc containers already by OSA, and your locally hosted packages will then be available to install needing no other modification or adjustment | 05:21 |
| noonedeadpunk | morning | 06:39 |
| *** rpittau|afk is now known as rpittau | 07:21 | |
| noonedeadpunk | really nasty thing I can't really understand :( https://zuul.opendev.org/t/openstack/build/321c754f399742b2b19e476743a2066d/log/job-output.txt#15628 | 07:34 |
| jrosser | that will be error in neutron API service contacting keystone | 07:36 |
| noonedeadpunk | oh... | 07:36 |
| noonedeadpunk | just wouldn't expect 503 | 07:36 |
| jrosser | here https://zuul.opendev.org/t/openstack/build/321c754f399742b2b19e476743a2066d/log/logs/host/neutron-server.service.journal-18-07-45.log.txt#1817 | 07:36 |
| noonedeadpunk | but yeah, I think I got idea now, that we send several requyest in a row | 07:37 |
| noonedeadpunk | which with the last one ends as 503 | 07:37 |
| jrosser | so this suggests that the systemd defaultenvironment thing may not be there? | 07:37 |
| jrosser | i had the same locally here btw | 07:38 |
| noonedeadpunk | http://paste.openstack.org/show/806905/ | 07:38 |
| noonedeadpunk | I dobt about spaces between ` = ` | 07:38 |
| jrosser | do you restart the neutron api service? | 07:40 |
| jrosser | i think i will rebuild my AIO from scratch with the current patches | 07:40 |
| jrosser | its in a wierd state because i tried all sorts of things / hacks | 07:41 |
| opendevreview | Merged openstack/openstack-ansible master: [doc] Fix compatability -> compatibility https://review.opendev.org/c/openstack/openstack-ansible/+/797673 | 07:48 |
| noonedeadpunk | well, neutron-api was deployed with this setting | 08:04 |
| noonedeadpunk | so it's fresh aio with all these patches | 08:05 |
| noonedeadpunk | jrosser: can put your key there to save some time | 08:05 |
| jrosser | oh sure yes | 08:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Set REQUESTS_CA_BUNDLE env var https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/797129 | 08:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_openrc master: Add OS_CACERT env variable https://review.opendev.org/c/openstack/openstack-ansible-openstack_openrc/+/797818 | 08:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Don't set keystone URI as unsecure https://review.opendev.org/c/openstack/openstack-ansible/+/796809 | 08:43 |
| *** sshnaidm|afk is now known as sshnaidm | 08:45 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Set buster jobs to non-voting https://review.opendev.org/c/openstack/openstack-ansible/+/797819 | 08:49 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-haproxy_server master: Add variable to disable stick-table https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/797642 | 10:54 |
| noonedeadpunk | jrosser: ok, so smth does pass now https://review.opendev.org/c/openstack/openstack-ansible/+/796809 but failures are different and weird... | 11:04 |
| jrosser | so buster we understand? | 11:04 |
| jrosser | bullseye failure is interesting, very early in keystone | 11:11 |
| * jrosser boots bullseye AIO | 11:16 | |
| noonedeadpunk | also focal has same failure | 11:18 |
| noonedeadpunk | sorry, not same | 11:18 |
| jrosser | then the proxy job is this https://zuul.opendev.org/t/openstack/build/1c83c08d0ff848d2bfe3ed4930de5c71/log/logs/host/squid/access.log.txt#1970-1995 | 11:18 |
| noonedeadpunk | uh, we didn't set squid proxy to consume https I guess? | 11:19 |
| jrosser | it will connect to whatever it has been asked to | 11:20 |
| jrosser | perhaps the proxy job fails because of this https://github.com/openstack/openstack-ansible/blob/306f57c31a00aeda589ef9189ba2fd01f1e27db1/tests/roles/bootstrap-host/files/squid.conf#L1 | 11:29 |
| noonedeadpunk | yeah, feels like this might be an issue | 11:50 |
| -opendevstatus- NOTICE: Our Zuul gating CI/CD services will be offline starting around 14:00 UTC (in roughly two hours from now) in order to apply some critical security updates, and is not expected to remain offline for more than 30 minutes. | 12:01 | |
| jrosser | noonedeadpunk: btw why are the haproxy server certs ending up in /etc/ssl/certs now? | 12:01 |
| noonedeadpunk | they should be in /etc/ssl/private/ | 12:08 |
| noonedeadpunk | `haproxy_ssl_key_path` is used for output file https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/796940/8/handlers/main.yml#18 | 12:08 |
| jrosser | a bunch of it ends up in /etc/ssl/certs now | 12:09 |
| noonedeadpunk | ah! | 12:09 |
| jrosser | http://paste.openstack.org/show/806915/ | 12:10 |
| noonedeadpunk | yeah, eventually we had that before don't we? | 12:10 |
| jrosser | well /etc/ssl/certs is kind of for roots | 12:10 |
| jrosser | not server things | 12:10 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/defaults/main.yml#L87 | 12:10 |
| noonedeadpunk | I was just following it.... | 12:10 |
| noonedeadpunk | But let's change that then? | 12:10 |
| jrosser | i guess i understand this though https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/defaults/main.yml#L90 | 12:11 |
| noonedeadpunk | yeah, that makes sense | 12:11 |
| jrosser | but imho in this new world we need to keep the certificates somehow elsewhere | 12:11 |
| jrosser | the root is installed independantly now | 12:12 |
| jrosser | also good news / bad news on bullseye | 12:12 |
| jrosser | i got the same failure as the CI job | 12:12 |
| noonedeadpunk | is it good news?:) | 12:12 |
| jrosser | then ran playbooks/os-keystone-install.yml again and it worked | 12:12 |
| noonedeadpunk | yeah, these were good for sure... | 12:12 |
| jrosser | so i wonder if there is different behaviour there with ssh connection persistence on bullseye, that we're still somehow using the same session | 12:13 |
| jrosser | same session from before /etc/environment got modified | 12:13 |
| noonedeadpunk | we should totally disconnect after setup-hosts | 12:14 |
| jrosser | there is a meta for that i think? | 12:14 |
| noonedeadpunk | yeah, there was some. but it makes sense only if we ran setup-everything | 12:14 |
| jrosser | ah reset_connection | 12:14 |
| noonedeadpunk | and we end play and run new one... | 12:14 |
| jrosser | yeah, i did setup-everything here | 12:14 |
| noonedeadpunk | ah, yes, for setup-everything we need meta somewhere after setup-hosts | 12:15 |
| noonedeadpunk | for gate-cehck-commit though it would be more weird | 12:15 |
| noonedeadpunk | hm, I can recall writing smth for resetting session there though... | 12:16 |
| jrosser | it is a bit odd how this is only failing on bullseye | 12:16 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/gate-check-commit.sh#L196-L199 | 12:17 |
| jrosser | though that sets the vars in the context of the gate-check-commit script though? | 12:21 |
| jrosser | ControlPersist=60s so i could see the persistent connection being maintained across different playbooks even inside the gate-check-commit scripte | 12:26 |
| jrosser | buy you quite likely avoid that during local development | 12:26 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Allow proxy zuul job to connect to any port https://review.opendev.org/c/openstack/openstack-ansible/+/797890 | 12:29 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Reset deploy host SSH connection after running openstack_hosts role https://review.opendev.org/c/openstack/openstack-ansible/+/797892 | 12:32 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Don't set keystone URI as unsecure https://review.opendev.org/c/openstack/openstack-ansible/+/796809 | 12:33 |
| jrosser | i think that https://review.opendev.org/c/openstack/openstack-ansible/+/797892/ might be working for bullseye in CI | 13:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Generate self-signed SSL per listen IP https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/796940 | 13:19 |
| noonedeadpunk | I wonder why it was like that... some sshd config? | 13:33 |
| noonedeadpunk | hope we get some results until zuul go down :( | 13:35 |
| -opendevstatus- NOTICE: Our Zuul gating CI/CD services are being taken offline now in order to apply some critical security updates, and are not expected to remain offline for more than 30 minutes. | 13:56 | |
| admin1 | what will be the best way to troubeshoot when the instance fails to get dhcp ip from dhcp-agent | 14:09 |
| noonedeadpunk | set ip manually and check if there's connectivity at all? | 14:11 |
| noonedeadpunk | between l3 namespace and instance | 14:11 |
| jrosser | tcpdump in the router namespace and look for arp and dhcp requests | 14:11 |
| noonedeadpunk | jrosser: do you think we should create smth like /etc/ssl/haproxy and /etc/ssl/rabbitmq? Or just make ssl dir under /etc/haproxy and /etc/rabbitmq ? | 14:30 |
| noonedeadpunk | second probably is more logical | 14:31 |
| noonedeadpunk | recheck time | 14:52 |
| _jralbert | jrosser: hosting the package in my own repo will work fine for me, but it won't help other folks who want to use the websso elements of OpenStack on CO7 - it'd be nicest if OSA either pulled in the packages with URLs in distro_packages (what I'm currently doing) or staged them to a local repo for install itself | 15:07 |
| jrosser | well the distro_packages variables are fed to the ansible package: module across all the roles to be OS independant | 15:09 |
| jrosser | however some of the roles like rabbitmq in the past have installed by giving a URL to a specific .deb / .rpm | 15:12 |
| _jralbert | Sorry, I should be precise: I'm adding package URLs to keystone_sp_distro_packages in vars/redhat.yml, so they'll only be included on CentOS Keystone containers when IDP SP features are enabled | 15:16 |
| _jralbert | Would you consider a bug/patchset to include those changes? | 15:16 |
| jrosser | and is that sufficient for whatever lies under here to just install those from the URL? https://opendev.org/openstack/openstack-ansible-os_keystone/src/branch/master/tasks/keystone_install.yml#L93-L105 | 15:18 |
| _jralbert | Yep, it works just fine - the "package" ansible module flows through to yum/dnf on RH/CO systems, and they happily take URLs as arguments | 15:19 |
| jrosser | ah right the yum module will take a URL https://docs.ansible.com/ansible/latest/collections/ansible/builtin/yum_module.html | 15:20 |
| jrosser | well if it's completely broken for OIDC as it stands then a patch is fine | 15:20 |
| _jralbert | Awesome, I'll put something together for it, hopefully before I'm gone on vacation | 15:20 |
| _jralbert | Would you like a single patchset for several issues around OIDC on CO (eg several elements of the Shib install could never have worked on CO because they expect usernames and paths from the APT installation of Shib that are different in the RPM install), or should I split each issue into its own bug/patchset? | 15:22 |
| jrosser | for something like the centos-7 support we'd need a patch against the OSA stable branch that still supports centos, so Ussuri I guess | 15:23 |
| jrosser | and then if there are more fundamental troubles with OIDC on Centos they may well also be wrong on the master branch, so those should be fixed on master then we backport bugfixes back to the stable branches | 15:24 |
| _jralbert | yeah, I'm doing this work in our Train environment, but it should apply to Train and Ussuri; there'll be a whole separate question about how CO8 differs in post-Ussuri installations | 15:24 |
| jrosser | fwiw the OIDC support got a lot of work in the V release | 15:28 |
| jrosser | would be worth reviewing what's been done recently in the os_keystone role | 15:28 |
| _jralbert | I'll take a look! | 15:29 |
| noonedeadpunk | jrosser: is it worth placing user provided CA to `/etc/ssl/private/` | 15:30 |
| jrosser | only the keys go in private | 15:30 |
| jrosser | and the keys should never leave the deploy host | 15:30 |
| noonedeadpunk | keys in terms of ssl rsa? | 15:31 |
| jrosser | no, the private key associated with the CA cert | 15:31 |
| noonedeadpunk | aha | 15:31 |
| jrosser | for a server (like haproxy, rabbit, blah), they need the key, the intermediate CA and the server cert | 15:33 |
| noonedeadpunk | yep | 15:33 |
| noonedeadpunk | gotcha | 15:33 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Generate self-signed SSL per listen IP https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/796940 | 15:35 |
| *** admin1 is now known as a0 | 15:59 | |
| admin1 | is there a way to setup specific host only for dhcp service ? .. tag 22.1.4 if that helps | 16:02 |
| admin1 | dhcp agents | 16:02 |
| *** rpittau is now known as rpittau|afk | 16:09 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Generate self-signed SSL per listen IP https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/796940 | 16:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Generate self-signed SSL per listen IP https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/796940 | 16:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Generate self-signed SSL per listen IP https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/796940 | 16:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_barbican master: Allow to symlink barbican_user_libraries https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/797991 | 17:06 |
| noonedeadpunk | admin1: I think right now neutron doesn't have appropriate groups, so env.d change would be required for that | 17:06 |
| admin1 | baiscally i want to have the routers in 1 node and dhcp agents in another node | 17:14 |
| noonedeadpunk | yeah, got you, but currently without env.d file change that's not possible. I guess spatel had some sample how to change env.d | 17:15 |
| noonedeadpunk | well, actually, high time to patch that... | 17:16 |
| noonedeadpunk | oh, well, now looking at env.d I'm not sure about it... | 17:20 |
| noonedeadpunk | admin1: I think you can actually create /etc/openstack_deploy/env.d/neutron_dhcp.yml with smth like that http://paste.openstack.org/show/806929/ | 17:30 |
| noonedeadpunk | worth to backup your openstack_inventory.json first though :) | 17:30 |
| noonedeadpunk | then you will be able to define `network-dhcp_hosts` in openstack_user_config and provide it IP of hosts where you want to run dhcp agent | 17:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Split neutron server and neutron agent hosts https://review.opendev.org/c/openstack/openstack-ansible/+/798001 | 17:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Split neutron server and neutron agent hosts https://review.opendev.org/c/openstack/openstack-ansible/+/798001 | 17:59 |
| noonedeadpunk | fwiw, kolla released W today | 18:04 |
| admin1 | noonedeadpunk, thank you .. i will try | 18:30 |
| opendevreview | Merged openstack/openstack-ansible-os_tempest stable/victoria: Install py3-dev when not building wheels https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/797031 | 20:43 |
| opendevreview | Merged openstack/openstack-ansible master: Set buster jobs to non-voting https://review.opendev.org/c/openstack/openstack-ansible/+/797819 | 23:49 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!