| noonedeadpunk | morning | 05:35 |
|---|---|---|
| * noonedeadpunk looking at haproxy | 05:35 | |
| jrosser | looks like buster is working again | 05:45 |
| noonedeadpunk | lol | 05:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Revert "Set buster jobs to non-voting" https://review.opendev.org/c/openstack/openstack-ansible/+/797952 | 05:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Revert "Set buster jobs to non-voting" https://review.opendev.org/c/openstack/openstack-ansible/+/797952 | 05:48 |
| noonedeadpunk | wait, wtf zuul did in https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/796940 - why patchset 9 got even reported now... | 05:51 |
| noonedeadpunk | also feels like https://review.opendev.org/c/openstack/openstack-ansible/+/797892 doesn't really work | 05:54 |
| noonedeadpunk | oh well, maybe it works as now debian fails the way futher | 05:57 |
| jrosser | yeah cinder db migration again | 06:22 |
| jrosser | yes that is confusing with ps9 results but ps13 being the latest to complete | 06:23 |
| jrosser | wonder if it's all to do with the restart/upgrade of zuul yesterday, something may be broken | 06:26 |
| noonedeadpunk | I have same thought actually. | 06:35 |
| noonedeadpunk | nice thing that haproxy eventually should pass | 06:35 |
| noonedeadpunk | also I tested it with user provided certs and it was fine I think | 06:35 |
| jrosser | awesome | 06:36 |
| * noonedeadpunk running buster aio atm | 06:40 | |
| noonedeadpunk | eventually debian failed not on cinder but on nova from what i saw | 06:41 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/9b332681e54644a091faaeb1e497aa93/log/job-output.txt#20664 | 06:42 |
| noonedeadpunk | well, godd thing that ci failure reproduced | 07:03 |
| jrosser | oh thats probably becasue it's using command and not shell | 07:05 |
| noonedeadpunk | hm, no, not really.... | 07:05 |
| noonedeadpunk | ah, my fault. I thought that I can just source /etc/environmnat :( | 07:06 |
| noonedeadpunk | `export REQUESTS_CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt"` and `/openstack/venvs/nova-23.0.0.0b2.dev20/bin/nova-status upgrade check` works | 07:07 |
| noonedeadpunk | interesting part is why this only raised for debian | 07:07 |
| noonedeadpunk | I mean https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/tasks/nova_db_post_setup.yml#L22-L26 is not failing and just passing | 07:08 |
| noonedeadpunk | bacuase it doesn't connect to api I guess... | 07:09 |
| noonedeadpunk | and might be smth related to become.... | 07:10 |
| jrosser | perhaps sudo config env_keep might help with that | 07:13 |
| noonedeadpunk | yep, it's become | 07:13 |
| *** rpittau|afk is now known as rpittau | 07:15 | |
| noonedeadpunk | `Defaults env_reset` | 07:16 |
| jrosser | yeah, thats going to clobber our var i think | 07:17 |
| noonedeadpunk | I believe same is set for ubuntu as well... | 07:19 |
| noonedeadpunk | well, I don't actually want to patch all roles now for patching sudoers... | 07:26 |
| noonedeadpunk | I bet sudo should get env from /etc/environment after reset normally.... | 07:27 |
| noonedeadpunk | that's result on bionic http://paste.openstack.org/show/806943/ | 07:35 |
| noonedeadpunk | oh, well, it's same on debian... | 07:37 |
| noonedeadpunk | but it's not used by command? | 07:37 |
| jrosser | try command vs. shel | 07:38 |
| jrosser | *shell | 07:38 |
| noonedeadpunk | doesn't work either | 07:39 |
| noonedeadpunk | if you want to look into and around - same ip as yestarday | 07:40 |
| noonedeadpunk | s/around/have time/ | 07:40 |
| noonedeadpunk | there's /root/openstack-ansible/test.yml I'm playing with | 07:41 |
| opendevreview | OpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/798052 | 07:41 |
| noonedeadpunk | env_keep doesn't help either btw | 07:44 |
| noonedeadpunk | maybe become_method is wrong | 07:44 |
| jrosser | did you change anything? | 07:49 |
| jrosser | it's working now somehow.... | 07:49 |
| noonedeadpunk | um... | 07:51 |
| jrosser | i tried it once, failed, added some debug and tried again, now it's ok..... | 07:51 |
| noonedeadpunk | I set `Defaults env_keep="REQUESTS_CA_BUNDLE"` instead of doing it specificly for nova user | 07:51 |
| jrosser | oh, well it has to be for the root user? | 07:52 |
| jrosser | as we're doing root->nova | 07:52 |
| noonedeadpunk | at least `Defaults:nova env_keep="REQUESTS_CA_BUNDLE"` doesn't work | 07:52 |
| noonedeadpunk | oh, yes, indeed, it should be for root, stupid me | 07:52 |
| jrosser | anyway \o/ | 07:52 |
| jrosser | but super odd it doesnt do this on ubuntu | 07:53 |
| noonedeadpunk | well, still not super clear why ansible does see env var when become | 07:53 |
| noonedeadpunk | but it's not taken into account | 07:53 |
| noonedeadpunk | yeah and ubuntu actually the same | 07:54 |
| jrosser | i added a task just like the nova-status one which ran /usr/bin/env | 07:54 |
| jrosser | and you can see the vars there when run with -vv, "stdout_lines": ["SUDO_GID=0", "MAIL=/var/mail/nova", "USER=nova", "HOME=/var/lib/nova", "SUDO_UID=0", "LOGNAME=nova", "TERM=unknown", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt", "LANG=en_US.UTF-8", "SUDO_COMMAND=/bin/sh -c echo BECOME-SUCCESS-bmpwqtzqouwizivbpofjbhrvlrxtltbp ; | 07:55 |
| jrosser | /usr/bin/python3", "SHELL=/bin/bash", "SUDO_USER=root", "PWD=/var/lib/nova/.ansible/tmp/ansible-moduletmp-1624607180.1686323-b4zjj8pw"] | 07:55 |
| noonedeadpunk | oh | 07:56 |
| noonedeadpunk | I think it might be because of pam.d session for sudo... | 07:56 |
| noonedeadpunk | ubuntu has `session required pam_env.so readenv=1 user_readenv=0` in /etc/pam.d/sudo | 07:57 |
| noonedeadpunk | also I think become_method su will also fix issue | 07:58 |
| noonedeadpunk | yeah... | 07:59 |
| noonedeadpunk | eventually we can specificly provide envfile with `session required pam_env.so readenv=1 envfile=/etc/environment user_readenv=0` | 08:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Set REQUESTS_CA_BUNDLE env var https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/797129 | 08:12 |
| noonedeadpunk | it's super weird it's not done for sudo but done for su... | 08:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Set REQUESTS_CA_BUNDLE env var https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/797129 | 08:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Set REQUESTS_CA_BUNDLE env var https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/797129 | 08:15 |
| jrosser | noonedeadpunk: you have ansible_facts['distribution'] | lower == 'Debian' which looks wrong? | 08:16 |
| noonedeadpunk | yeah... | 08:16 |
| noonedeadpunk | already fixed | 08:16 |
| jrosser | ah ok | 08:16 |
| noonedeadpunk | also nice things to merge might be https://review.opendev.org/c/openstack/openstack-ansible/+/798001 as it was one of goals for release | 08:19 |
| opendevreview | Jonathan Heathcote proposed openstack/openstack-ansible stable/victoria: Bump SHAs for stable/victoria https://review.opendev.org/c/openstack/openstack-ansible/+/795000 | 08:54 |
| noonedeadpunk | feels like 796809 going to pass now :) | 10:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Enable senlin tempest plugin when senlin is deployed https://review.opendev.org/c/openstack/openstack-ansible/+/754105 | 10:05 |
| jrosser | cool - need to push to merge all these small fixes that it needs | 10:09 |
| noonedeadpunk | will monitor them and ping ppl once they pass ci | 10:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Add centos-8 support for ovs-dpdk https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/762729 | 10:16 |
| mjwales | Hi, we have completed an upgrade from Train to Ussuri but neutron is playing up. As far as we can tell the l3 agent is not creating any routers. Are there any known issues? | 10:22 |
| noonedeadpunk | and what does l3 agent logs say? | 10:23 |
| mjwales | Not a lot it stops logging after running for 10 minutes or so | 10:24 |
| noonedeadpunk | are agents even running? | 10:25 |
| noonedeadpunk | also are you checking journald logs, right? | 10:25 |
| noonedeadpunk | actually I may think about some rootwrap issue, but logs would help a lot | 10:25 |
| mjwales | The agents are showing as active via a systemctl status | 10:26 |
| noonedeadpunk | as well as openstack network agent list? | 10:27 |
| mjwales | We rebooted a controller node at 08:20 this morning. The l3 agent then outputs logs to journald until 08:31 then nothing since that | 10:27 |
| mjwales | openstack network agent list shows them as active | 10:27 |
| noonedeadpunk | and happy smile as well?:) | 10:28 |
| mjwales | and a happy smile too | 10:29 |
| noonedeadpunk | well, at least that means that it's not rabbitmq communication issue. | 10:29 |
| noonedeadpunk | but not so sure.... | 10:29 |
| noonedeadpunk | as node reboot might trigger rabbitmq issue | 10:30 |
| noonedeadpunk | and if there're no logs then l3 agent doesn't recieve any messages... | 10:31 |
| mjwales | We enabled debug logging for l3 on one of the nodes. That does continually log but only items such as "Lock "_check_child_processes" acquired by " | 10:32 |
| mjwales | We initially thought this might be because we have a large number of networks and routers (700+) | 10:32 |
| noonedeadpunk | and issues are only with l3s? so instance are created and managed normaly without issues? | 10:34 |
| mjwales | Logs from the controller that was rebooted at http://paste.openstack.org/show/806950/ | 10:35 |
| opendevreview | Merged openstack/openstack-ansible master: Revert "Set buster jobs to non-voting" https://review.opendev.org/c/openstack/openstack-ansible/+/797952 | 10:36 |
| mjwales | Instances launch as show in a running state | 10:37 |
| mjwales | Trying to look at the console in horizon just show "Connecting" | 10:37 |
| noonedeadpunk | `oslo.messaging._drivers.impl_rabbit` are actually concerning | 10:37 |
| noonedeadpunk | and you can telnet to rabbit ports from controller node? | 10:38 |
| mjwales | I put those down to rabbit starting on the node | 10:38 |
| noonedeadpunk | might be actually. I think there should be message tha connection was re-initiated though... | 10:40 |
| noonedeadpunk | and all l3 agents are not creating new namspaces and do nothing? | 10:40 |
| mjwales | Trying to telnet using 'telnet infra3-rabbit-mq-container-15d1d426 5672' gives a response of 'Connection closed by foreign host.' | 10:42 |
| noonedeadpunk | and what does rabbitmqctl cluster_status say? if run it from rabbit container? | 10:42 |
| mjwales | That says everything is fine. 3 disk nodes, 3 running nodes, all on the same version and no alarms or partitions | 10:43 |
| mjwales | Running a host 'host infra1-rabbit-mq-container-45cefef2' give two entires however a 'infra1-rabbit-mq-container-45cefef2 has address 172.29.239.195' and 'Host infra1-rabbit-mq-container-45cefef2 not found: 3(NXDOMAIN)' | 10:45 |
| mjwales | There are connections to each of the rabbit nodes. Checked via rabbitmqctl status | 10:52 |
| mjwales | noonedeakpunk: the l3 agents do create a handful of namespaces after a restart but that then stops | 11:06 |
| noonedeadpunk | well... I'm not sure it will work, but worth trying? with some messages related issues that helps | 11:08 |
| noonedeadpunk | running openstack-ansible rabbitmq-install.yml -e rabbitmq_upgrade=true | 11:08 |
| mjwales | Will give it a shot | 11:10 |
| *** rpittau is now known as rpittau|bbl | 11:17 | |
| opendevreview | Merged openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/798052 | 12:09 |
| noonedeadpunk | spotz: jrosser ebbex would be great to get votes on https://review.opendev.org/q/topic:%22osa%252Fpki%22+(status:open)+label:Verified | 12:20 |
| jrosser | noonedeadpunk: we set this to False https://review.opendev.org/c/openstack/openstack-ansible/+/796809/8/inventory/group_vars/all/keystone.yml | 12:25 |
| jrosser | but the default in group vars is still http https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/all.yml#L87 | 12:26 |
| noonedeadpunk | agree- stupid | 12:26 |
| noonedeadpunk | ah, well, not really | 12:26 |
| noonedeadpunk | because it more about what we do by default vs what we want to test? | 12:26 |
| noonedeadpunk | because insecure makes sense only when https+cert untrusted | 12:27 |
| jrosser | oh right of course | 12:28 |
| jrosser | so now it is an actual error when the cert is not trusted | 12:28 |
| jrosser | rather than something to hack around | 12:28 |
| noonedeadpunk | yep | 12:28 |
| noonedeadpunk | also votes on https://review.opendev.org/c/openstack/openstack-ansible/+/798001 https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/797991 and https://review.opendev.org/c/openstack/ansible-config_template/+/796151 would be a plus | 12:34 |
| jrosser | urgh linters on stable/victoria need attention https://zuul.opendev.org/t/openstack/build/8682a29bfa74484bb5a450696e743260 | 12:41 |
| noonedeadpunk | yeah... | 12:41 |
| noonedeadpunk | I saw that, will take a look now. I guess it's related to integrated repo only | 12:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Enable senlin tempest plugin when senlin is deployed https://review.opendev.org/c/openstack/openstack-ansible/+/754105 | 12:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Don't set keystone URI as unsecure https://review.opendev.org/c/openstack/openstack-ansible/+/796809 | 12:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Enable senlin tempest plugin when senlin is deployed https://review.opendev.org/c/openstack/openstack-ansible/+/754105 | 12:46 |
| noonedeadpunk | damn | 12:46 |
| noonedeadpunk | I meant to rebase only top patch :( | 12:46 |
| opendevreview | Merged openstack/openstack-ansible-openstack_openrc master: Add OS_CACERT env variable https://review.opendev.org/c/openstack/openstack-ansible-openstack_openrc/+/797818 | 12:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump SHAs for master https://review.opendev.org/c/openstack/openstack-ansible/+/794998 | 12:55 |
| *** rpittau|bbl is now known as rpittau | 12:56 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Workaround linting issue https://review.opendev.org/c/openstack/openstack-ansible/+/798083 | 13:10 |
| spotz | noonedeadpunk: Looking | 13:21 |
| opendevreview | Amy Marrich proposed openstack/openstack-ansible-openstack_hosts master: Set REQUESTS_CA_BUNDLE env var https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/797129 | 13:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Set REQUESTS_CA_BUNDLE env var https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/797129 | 13:26 |
| spotz | Ack hope we didn't collide! | 13:27 |
| noonedeadpunk | just edited your small typo) https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/797129/10..11/releasenotes/notes/requests_ca_bundle-1a678a22b3375976.yaml | 13:27 |
| spotz | hehe, II really need a new keyboard. Sometimes I'm better at catching the extra letters then other times | 13:28 |
| spotz | And sometimes I just hit enter in chhats cause my friends read through it:) | 13:28 |
| noonedeadpunk | yeah, and I totally need to attend more english classes :( or buy smth like grammarly at least | 13:29 |
| opendevreview | Amy Marrich proposed openstack/openstack-ansible-haproxy_server master: Generate self-signed SSL per listen IP https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/796940 | 13:30 |
| spotz | Ok no typos there:) | 13:30 |
| spotz | I add some the's I change afeew words out. It's not that the words are wrong but for a smoother flow. And it's why you keep me around:) | 13:31 |
| spotz | +2'd cause the code did look good on both of those. The other one is already +w | 13:32 |
| noonedeadpunk | we eventually do this for https://review.opendev.org/c/openstack/openstack-ansible/+/796809 | 13:33 |
| noonedeadpunk | (which stopps us from finally releasing W) | 13:33 |
| noonedeadpunk | btw would be also awesome to merge nspawn deprecation :) https://review.opendev.org/c/openstack/openstack-ansible-nspawn_container_create/+/797726 https://review.opendev.org/c/openstack/openstack-ansible-nspawn_hosts/+/797724 | 13:35 |
| opendevreview | Merged openstack/ansible-role-uwsgi master: Use the system trust store for python requests rather than certifi https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/797600 | 13:51 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Add variable to disable stick-table https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/797642 | 14:19 |
| *** rpittau is now known as rpittau|afk | 14:47 | |
| noonedeadpunk | hopefully stuff will land today and wil be able to push branching PR for roles tomorrow | 15:01 |
| jrosser | we need another vote on this https://review.opendev.org/c/openstack/openstack-ansible/+/797890/ | 15:31 |
| jrosser | and a little care with small changes/nits just now on anything to avoid interfering with the currently running jobs | 15:33 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible/+/796809 is 90% passed in CI... | 15:36 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/train: Use version from repo_packages for SPICE HTML5 https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/797312 | 16:01 |
| noonedeadpunk | spotz: can you please check these 2 patches as well if still around ?:) | 16:10 |
| noonedeadpunk | and https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/797991 :) | 16:11 |
| noonedeadpunk | jrosser: btw https://review.opendev.org/c/openstack/openstack-ansible/+/798083 seems to ublock V | 16:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Bump SHAs for stable/victoria https://review.opendev.org/c/openstack/openstack-ansible/+/795000 | 16:15 |
| jrosser | commit message is saying tempest is commented out, which the patch doesnt do | 16:41 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/victoria: Workaround linting issue https://review.opendev.org/c/openstack/openstack-ansible/+/798083 | 16:42 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/victoria: Bump SHAs for stable/victoria https://review.opendev.org/c/openstack/openstack-ansible/+/795000 | 16:42 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Set REQUESTS_CA_BUNDLE env var https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/797129 | 16:44 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Generate self-signed SSL per listen IP https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/796940 | 17:21 |
| spotz | noonedeadpunk: I think I got them all | 19:27 |
| jrosser | spotz: theres a bunch on the dashboard needing approval too http://bit.ly/osa-review-board-v3 | 19:29 |
| spotz | jrosser: bit.ly is rrerouting me to my dashboard:) | 19:32 |
| spotz | Ha cliicking vs copying worked:) | 19:32 |
| opendevreview | Merged openstack/openstack-ansible-nspawn_hosts master: Deprecate OpenStack-Ansible nspawn repositories https://review.opendev.org/c/openstack/openstack-ansible-nspawn_hosts/+/797724 | 19:41 |
| opendevreview | Merged openstack/openstack-ansible-nspawn_container_create master: Deprecate OpenStack-Ansible nspawn repositories https://review.opendev.org/c/openstack/openstack-ansible-nspawn_container_create/+/797726 | 19:43 |
| opendevreview | Merged openstack/openstack-ansible master: Allow proxy zuul job to connect to any port https://review.opendev.org/c/openstack/openstack-ansible/+/797890 | 20:07 |
| opendevreview | Merged openstack/openstack-ansible master: Reset deploy host SSH connection after running openstack_hosts role https://review.opendev.org/c/openstack/openstack-ansible/+/797892 | 21:02 |
| opendevreview | Merged openstack/openstack-ansible-os_barbican master: Allow to symlink barbican_user_libraries https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/797991 | 21:09 |
| opendevreview | Merged openstack/ansible-config_template master: Add PKI role to functional testing https://review.opendev.org/c/openstack/ansible-config_template/+/796151 | 21:26 |
| opendevreview | Merged openstack/ansible-config_template master: Use ansible_facts[] instead of fact variables https://review.opendev.org/c/openstack/ansible-config_template/+/780753 | 21:26 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Add centos-8 support for ovs-dpdk https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/762729 | 22:11 |
| opendevreview | Merged openstack/openstack-ansible master: Split neutron server and neutron agent hosts https://review.opendev.org/c/openstack/openstack-ansible/+/798001 | 23:34 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!