| *** rpittau|afk is now known as rpittau | 07:54 | |
| DanG | Hey folks, is it a known problem that I am unable to use a cidr range on openstack instances that conflicts with the controller lxc cidr range (e.g. 10.0.3.0/24 by default)? for me this results in the instance being unable to access metadata or the internet but can be reached by SSH | 09:38 |
|---|---|---|
| DK4 | anyway to find out why "/openstack/venvs/keystone-23.1.0.dev9/bin/keystone-manage db_sync check" returns 1? it does not seem to throw any error | 10:02 |
| noonedeadpunk | DanG: oh, that's interesting. eventually metadata is reached through 169.254.169.254 and then routed down the way. And this should not ever result internet access of the instances themselves | 10:48 |
| noonedeadpunk | but how it's reached via ssh? from network namespace? | 10:49 |
| noonedeadpunk | DK4: iirc it should log output/result into journald | 10:51 |
| noonedeadpunk | But I guess it would be `Your database is out of sync` | 10:51 |
| noonedeadpunk | based on the exit code | 10:52 |
| DanG | I've used a bastion instance to reach them, my initial thought was a conflict within the controller as to where to route the reply | 11:11 |
| noonedeadpunk | While it sounds really plausable I'm not sure how that is possible... As neutron part is placed all in net namespaces and shouldn't care about host networks | 12:12 |
| noonedeadpunk | Maybe things go wrong when neutron forwards request from ns to nova metadata... But I think it's using nova api endpoints rather then tenant vxlan network... | 12:14 |
| spatel | noonedeadpunk are you around? | 13:01 |
| noonedeadpunk | yep | 13:15 |
| spatel | https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803990 | 13:18 |
| spatel | could you give me example what we can do to make it more better | 13:19 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible master: Add guide for distribution upgrades to docs https://review.opendev.org/c/openstack/openstack-ansible/+/804875 | 13:19 |
| spatel | lets fix it now and not later | 13:19 |
| andrewbonney | noonedeadpunk / kleini: first draft of dist upgrade guide just uploaded https://review.opendev.org/c/openstack/openstack-ansible/+/804875 | 13:20 |
| noonedeadpunk | ٩(^ᴗ^)۶ | 13:21 |
| noonedeadpunk | spatel: I will comment in patch | 13:21 |
| spatel | perfect! | 13:22 |
| spatel | stupid question related apt-get, i am messing with my lab and running rabbitMQ playbook end stuck here - https://paste.opendev.org/show/808144/ | 13:22 |
| spatel | any idea what apt-get trying to downgrade here? | 13:22 |
| noonedeadpunk | I think it's related to erlang-base-hipe package deprecation | 13:23 |
| spatel | hmm | 13:23 |
| noonedeadpunk | Maybe https://opendev.org/openstack/openstack-ansible-rabbitmq_server/commit/bf53f24dd40b9938ba3d10ba35cc87489d285980 fixes that? | 13:23 |
| DanG | noonedeadpunk thanks for your thoughts! do you think this is likely to be a misconfiguration in our installation or a bug present which we should consequently advise folks to avoid that IP range in project networks? | 13:30 |
| noonedeadpunk | let me try to reproduce in my prod env :) | 13:32 |
| spatel | noonedeadpunk This patch additionally switches deployments to the 'erlang-base' | 13:32 |
| spatel | rather than 'erlang-base-hipe' package which appears to have been | 13:32 |
| spatel | discontinued. | 13:32 |
| noonedeadpunk | spatel: you think it's bad idea? | 13:33 |
| spatel | how do i check which repo apt-get is using? | 13:33 |
| spatel | i am upgrading my openstack from 22.0.0.rc2 to 22.2.0 | 13:34 |
| spatel | curious why i hit that error if we have that patch in place? | 13:34 |
| noonedeadpunk | we just merged it | 13:34 |
| noonedeadpunk | it's not in 22.2.0 | 13:34 |
| dmsimard | noonedeadpunk: welcome back o/ | 13:34 |
| noonedeadpunk | thanks! | 13:34 |
| noonedeadpunk | I started missing stuff during vacation :) | 13:35 |
| spatel | can we backport this patch otherwise it will break my production upgrade also :) | 13:35 |
| noonedeadpunk | andrewbonney: ^ ? I guess we can? | 13:37 |
| andrewbonney | Yeah, the only reason I hadn't got round to it was wondering if we should add a release note with backports? If you don't run it with 'rabbitmq_upgrade=True' it might break things | 13:37 |
| noonedeadpunk | wtf... https://docs.openstack.org/openstack-ansible/latest/ | 13:38 |
| noonedeadpunk | I bet it's that locale stuff.... | 13:39 |
| noonedeadpunk | So, I mean https://docs.openstack.org/openstack-ansible/latest/en_GB/admin/upgrades/minor-upgrades.html is covering that? | 13:39 |
| noonedeadpunk | we just should add there galera_upgrade as well... | 13:40 |
| noonedeadpunk | But anyway - my point was that the way to get this code is minor upgrade or major upgrade. During both of them you're supposed to provide that flag? | 13:40 |
| andrewbonney | That's fine if you're happy with that. It's only really an issue if someone is already running latest rabbitmq as otherwise the role won't let you continue without providing that flag | 13:42 |
| spatel | andrewbonney should we add version condition to detect if its latest (skip) | 13:43 |
| andrewbonney | I was trying not to over-complicate the role with extra checks if possible | 13:44 |
| noonedeadpunk | IMO that's fine. Anytime you do upgrade - you run with that flag. And this instruction hasn't changed for years now. | 13:44 |
| spatel | i am running with openstack-ansible -e rabbitmq_upgrade=true rabbitmq-install.yml | 13:48 |
| spatel | still getting that error | 13:48 |
| spatel | let me patch it by hand and see if it pass that step | 13:48 |
| kleini | andrewbonney: great! | 13:53 |
| andrewbonney | spatel: which operating system are you running, and have you customised the rabbitmq_install_method at all? I note that the rabbitmq role didn't change between 22.0.0.rc2 and 22.2.0 | 13:57 |
| spatel | ubuntu 20.04 | 13:57 |
| spatel | i have applied that patch and still getting same error | 13:57 |
| spatel | no, i don't have anything in users_variables.yaml | 13:58 |
| andrewbonney | Ok, 22.2.0 used distro install for rabbitmq on 20.04. That might be part of the issue. 22.3.0 attempted to conform this: https://github.com/openstack/openstack-ansible-rabbitmq_server/commit/bd355fc461b5adc82d2478c93e908b05a5edb847 | 13:58 |
| spatel | rabbitmq-server 3.9.3-1 | 13:58 |
| andrewbonney | Would that patch plus the erlang one help? | 13:58 |
| andrewbonney | Ah 3.9.3 seems very new | 13:59 |
| spatel | i don't know from where that came from | 13:59 |
| andrewbonney | Yeah, the distro version for 20.04 is 3.8.2 | 14:00 |
| spatel | currently my rabbitmq cluster is down | 14:00 |
| spatel | so do you think i should remove 3.9.3 ? and re-run playbook ? | 14:00 |
| andrewbonney | What version is installed on your other two hosts? | 14:00 |
| spatel | ah! on other nodes i have 3.8.14-1 | 14:03 |
| spatel | very messed up stage | 14:03 |
| spatel | i think solution would be remove rabbitmq-server on infra-1 and give it a try | 14:04 |
| andrewbonney | Yeah, I'd remove the new version. If you were using the 22.3.0 release that ought to get the apt pins correct for 3.8.14-1 to match your other hosts. Hopefully with that plus the erlang patch you can get it going again | 14:05 |
| spatel | andrewbonney let me get this straight, do minor upgrade to 22.3.0 + apply your patch right? | 14:18 |
| andrewbonney | Yeah, either that or apply both patches to the rabbitmq role. There are only those two as part of the minor upgrade for that role | 14:19 |
| spatel | andrewbonney perfect! | 14:37 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-rabbitmq_server stable/wallaby: Fix erlang upgrades for rabbitmq external_repo installs https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/804861 | 14:41 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-rabbitmq_server stable/victoria: Fix erlang upgrades for rabbitmq external_repo installs https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/804884 | 14:46 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Aug 17 15:00:17 2021 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| noonedeadpunk | o/ | 15:02 |
| noonedeadpunk | #topic open discussion | 15:13 |
| noonedeadpunk | I don't have an agenda for today meeting | 15:13 |
| noonedeadpunk | The main problem now is our docs that are broken :( | 15:13 |
| spatel | how that happened overnight? | 15:14 |
| spatel | seems like web server issue which is not looking into correct directors | 15:17 |
| noonedeadpunk | it's issue that sphinx doesn't finish rendering and does not create index.html | 15:24 |
| noonedeadpunk | and that is because of docutils version | 15:24 |
| spatel | i remembered we hit some kind of sphinx bug few week ago | 15:28 |
| spatel | not sure if that is causing this issue | 15:28 |
| mgariepy | https://review.opendev.org/c/openstack/openstackdocstheme/+/802758 | 15:38 |
| mgariepy | that's the issue with sphinx | 15:38 |
| DanG | hey noonedeadpunk thanks for looking into the issue I mentioned, were you able to reproduce it? | 15:40 |
| noonedeadpunk | mgariepy: that is stupid solution kind of... | 15:41 |
| noonedeadpunk | we're dropping so much work for ID... | 15:42 |
| mgariepy | i do prefer mine. | 15:42 |
| mgariepy | of course haha | 15:42 |
| mgariepy | you can add a comment ! | 15:42 |
| mgariepy | not just for ID. for all lang that are not in the docutils list. | 15:43 |
| spatel | Any idea about this error? https://paste.opendev.org/show/808148/ | 15:47 |
| spatel | look like i don't have nova-22.3.0 director root@os-infra-2-nova-api-container-93628d14:~# /openstack/venvs/nova-22.3.0/bin/nova-manage | 15:48 |
| spatel | let me debug and see | 15:49 |
| spatel | https://paste.opendev.org/show/808150/ | 15:51 |
| noonedeadpunk | mgariepy: yeah, I think we need to push patches to docutils indeed.... But for OSA we can remove -W for now as quick fix | 15:52 |
| noonedeadpunk | the problem with -W that all linting issues would be skipped :( | 15:55 |
| mgariepy | it would be nice to add all the lang to docutils but i think that removing -W for all the ""unssupported lang"" | 15:57 |
| mgariepy | would be better for now. | 15:57 |
| mgariepy | or a way to filter out some warnings for generating errors on the sphinx side. | 15:59 |
| noonedeadpunk | #endmeeting | 16:04 |
| opendevmeet | Meeting ended Tue Aug 17 16:04:00 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:04 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-08-17-15.00.html | 16:04 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-08-17-15.00.txt | 16:04 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-08-17-15.00.log.html | 16:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Fix doc upper constraints before docutils bump https://review.opendev.org/c/openstack/openstack-ansible/+/804891 | 16:18 |
| noonedeadpunk | DanG: for me it's working nicely https://paste.opendev.org/show/808151/ | 16:21 |
| noonedeadpunk | I also assigned floating IP to the isntance and was able to ssh from public net | 16:21 |
| noonedeadpunk | btw, if metadata wasn't working, you won't have SSH key on the machine after it is spawned | 16:22 |
| noonedeadpunk | but dunno, maybe you placed ssh keys in the image or did some recovery process | 16:23 |
| DanG | ahh thanks noonedeadpunk! yeah I used one instance which did have metadata access to make a snapshot and applied it to all so yeah ssh keys were baked in, we were making instances that were 10.0.[1,2,3,4].* so access was only appearing to fail sometimes, quite baffling until I worked out it was only on .3 | 16:27 |
| *** rpittau is now known as rpittau|afk | 16:31 | |
| spatel | andrewbonney i am seeing very odd issue with 22.3.0 tag specially with nova | 16:51 |
| spatel | its trying to deploy wallaby version of nova_api schema and that is why its failing.. | 16:51 |
| spatel | noonedeadpunk this SHA doesn't exist look like - https://github.com/openstack/openstack-ansible/commit/378a4d834d760ad4f3b242e843bf517da0f95fca#diff-12f9c58e857caaeb47fc942e54562395816c1b3fccff80c1ffd9ceace759683bR199 | 17:04 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Update galera to 10.5.12 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/804284 | 17:10 |
| opendevreview | Merged openstack/openstack-ansible master: Add shallow_since to parallel git clone https://review.opendev.org/c/openstack/openstack-ansible/+/802145 | 17:25 |
| mgariepy | noonedeadpunk, maybe until it's fixed we can push this one https://review.opendev.org/c/openstack/openstack-ansible/+/803635 ? | 17:41 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!