| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/zed: Remove nova-scheduler from ironic compute container_skel https://review.opendev.org/c/openstack/openstack-ansible/+/879482 | 05:42 |
|---|---|---|
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/zed: Add documentation on refreshing hosts file https://review.opendev.org/c/openstack/openstack-ansible/+/879483 | 05:43 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/yoga: Add documentation on refreshing hosts file https://review.opendev.org/c/openstack/openstack-ansible/+/879484 | 05:43 |
| kleini | Good morning, if you setup bonded network interfaces, which bond type do you use? OpenStack docs have active-backup in their examples. I used LACP and I am stumbling now across a larger amount of packet loss and try to find the cause for that. | 07:26 |
| jrosser | kleini: it kind of depends on how you have your switches set up really | 08:01 |
| kleini | two redundant switches | 08:02 |
| kleini | so if one fails, the other one should be able to take over all traffic | 08:02 |
| jrosser | a single switch with bonds to it, or a pair of switches with MLAG/VPC you can do active/active, if it's just a pair of switches with no fancy multichassis features then you need to do active/backup | 08:02 |
| jrosser | i have nexus9k and use cisco VPC to make portchannels across both switches, then can use active/active mode 4 bond between the two, for example | 08:04 |
| kleini | a pair of Lenovo switches doing MLAG | 08:04 |
| jrosser | having said that, it has been very vulnerable to bugs in both the switch firmware and NIC firmware | 08:05 |
| jrosser | and if i was starting again from scratch would probably look at trying to do L3 routing from each host instead | 08:05 |
| kleini | we're doing here the same, too. but I see a lot of packet loss and bad network throughput. multiple documents, howtos and guides now claim, that 802.3ad is not fully compliant implemented everywhere. Especially OVS should have problems with it. | 08:05 |
| kleini | Are there any guides/introduction into L3 routing? | 08:07 |
| jrosser | we also have a set of steps which must be followed *exactly* to upgrade switch firmware otherwise things go pretty bad if you don't do it exactly right | 08:07 |
| jrosser | i don't think i've seen a pure L3 implementation | 08:09 |
| jrosser | that would be super interesting to work on | 08:09 |
| noonedeadpunk | mornings | 08:34 |
| jrosser | good morning | 08:35 |
| jrosser | need to try to get some of the haproxy stuff moving this morning | 08:35 |
| jrosser | if damiandabrowski is around this is the bottom of the dependancy tree i think https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/875779 | 08:37 |
| noonedeadpunk | I was just looking at this patch | 08:38 |
| jrosser | i see on a bunch of others there are unaddressed comments | 08:39 |
| jrosser | i also saw some repo gpg key errors on stable branch backports of the ovs fixes | 08:40 |
| noonedeadpunk | I assume that was mariadb? | 08:40 |
| noonedeadpunk | As they've updated for centos, but should affect only new releases... | 08:40 |
| jrosser | it was for the ovs pacakge actually | 08:41 |
| noonedeadpunk | huh | 08:41 |
| jrosser | also TIL you can `bind blah.example.com:8000-9000` in a haproxy frontend config to handle an entire port range in one statement | 08:46 |
| noonedeadpunk | oh, that's interesting, didn't know that | 09:16 |
| noonedeadpunk | wtf. So you wanna say that gpg is wrong on Zed but not in Yoga while jobs finished at exact same time? | 10:11 |
| noonedeadpunk | And well. Zed was passing at some point as well... | 10:11 |
| noonedeadpunk | and it doesn't affect rocky on top | 10:12 |
| hamidlotfi | Hello, | 10:17 |
| hamidlotfi | I'm back again with a duplicate error message but this time in adding a Compute node to my environment. | 10:17 |
| hamidlotfi | Run the following command, I want to add the `compute03` | 10:17 |
| hamidlotfi | `openstack-ansible playbooks/setup-openstack.yml --limit localhost,compute03` | 10:17 |
| hamidlotfi | show this error message: | 10:17 |
| hamidlotfi | ` | 10:17 |
| hamidlotfi | TASK [Gather software version list] ******************************************** | 10:17 |
| hamidlotfi | fatal: [compute03 -> localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'ansible_local'\n\nThe error appears to be in '/opt/openstack-ansible/playbooks/os-nova-install.yml': line 52, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n | 10:17 |
| hamidlotfi | # venv tag for all hosts in the 'cinder_all' host group.\n - name: Gather software version list\n ^ here\n"} | 10:17 |
| hamidlotfi | ` | 10:17 |
| hamidlotfi | @jrosser | 10:17 |
| noonedeadpunk | yeah, I think your nodes are deployed already. | 10:29 |
| noonedeadpunk | as this failure happens only after al lrequired steps are done. | 10:30 |
| hamidlotfi | let me check | 10:30 |
| noonedeadpunk | So unless you have zun or masakari - you should be good enough now | 10:31 |
| hamidlotfi | Yes, right, the new compute node is added successfully. | 10:32 |
| noonedeadpunk | but it's actualy exact same bug you had with controllers | 10:33 |
| hamidlotfi | Yes, but in the controller add the `cinder_all` in the `limit` section and skiped | 10:35 |
| hamidlotfi | Here also I added the same order but it also shows me the error. | 10:37 |
| jrosser | well, it's a different error really | 10:37 |
| jrosser | this time is it missing information about the nova_all group, not cinder_all | 10:38 |
| jrosser | root cause is different i mean | 10:38 |
| noonedeadpunk | well... it's different place but same issue | 10:38 |
| noonedeadpunk | but yeah | 10:39 |
| hamidlotfi | ohh, but the end of the error message says ` venv tag for all hosts in the 'cinder_all' host group` | 10:39 |
| noonedeadpunk | oh wait, indeed | 10:40 |
| noonedeadpunk | brr, how cinder_all ended up being asked in os-nova-install.yml | 10:40 |
| hamidlotfi | I don't know | 10:42 |
| jrosser | isnt that a bug that i already patched? | 10:44 |
| jrosser | https://github.com/openstack/openstack-ansible/commit/7ddc02946dfebf6fd39540b131ac5137f711fd00 | 10:45 |
| hamidlotfi | I work with ZED version. | 10:45 |
| jrosser | it's backported to Zed as well https://review.opendev.org/c/openstack/openstack-ansible/+/876678 | 10:46 |
| jrosser | hamidlotfi: it's probably that you're using a point release of Zed that does not contain my fix to that comment | 10:47 |
| hamidlotfi | Yes, in my file don't have your patch. | 10:49 |
| hamidlotfi | https://www.irccloud.com/pastebin/F0bblMAs/ | 10:49 |
| hamidlotfi | @jrosser Thank you for your hint. | 10:51 |
| jrosser | i don't think that anything has actually failed, even though its bad that the playbook ends in error | 10:51 |
| jrosser | unfortunately there is not a trivial fix to this and it's going to need some consideration of what the correct approach is | 10:52 |
| hamidlotfi | Yes, the mission is completed successfully 🫡 | 10:54 |
| noonedeadpunk | hamidlotfi: it's not functional patch just in case | 11:22 |
| hamidlotfi | I understood. | 11:24 |
| noonedeadpunk | It looks like we indeed have some gpg issue for centos.... | 11:35 |
| noonedeadpunk | wrt ovn/ovs which is provided by SIG | 11:36 |
| noonedeadpunk | that is soooooooooo annoying | 11:36 |
| noonedeadpunk | and why only centos and not rocky.... | 11:36 |
| noonedeadpunk | it basically means, that `centos-release-nfv-openvswitch` has issue in packaging on centos, while rocky pack it correctly | 11:40 |
| noonedeadpunk | well, obviously I can't reproduce a thing locally | 11:55 |
| noonedeadpunk | BUT, what I see in /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-NFV in VM and https://www.centos.org/keys/RPM-GPG-KEY-CentOS-SIG-NFV are 2 completely different GPGs | 11:56 |
| noonedeadpunk | AND fingerprint is kinda same https://paste.openstack.org/show/b9TiKYNc44AmldSXZU5X/ | 12:04 |
| noonedeadpunk | with one provided here https://www.centos.org/keys/ | 12:05 |
| noonedeadpunk | well, fingerprint is exactly the same in both, so likely just gpg format is different | 12:18 |
| NeilHanlon | I am meeting with the NFV sig this morning.. will talk to them about this noonedeadpunk | 12:52 |
| noonedeadpunk | I've already bothered amoralej just in case... This could potentialy be infra mirror issue, but weird... | 12:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Drop `echo` from add-compute.sh script https://review.opendev.org/c/openstack/openstack-ansible/+/879476 | 13:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Drop `echo` from add-compute.sh script https://review.opendev.org/c/openstack/openstack-ansible/+/879475 | 13:57 |
| noonedeadpunk | NeilHanlon: I _think_ that centos-release-nfv-openvswitch simply ships gpg key in SHA1 that's not supported anymore by gpg https://paste.openstack.org/show/bXtPkgJ96bEzleXjcAW6/ | 14:33 |
| noonedeadpunk | Probably it's time to switch jobs to rocky, as it's some nightmare with CentOS going on recently and I'm quite fed up with it | 14:40 |
| noonedeadpunk | https://bugzilla.redhat.com/show_bug.cgi?id=2184640 | 14:52 |
| NeilHanlon | ugh | 14:59 |
| noonedeadpunk | so. all current container/cloud images of centos are completely borked | 14:59 |
| NeilHanlon | sick | 15:00 |
| NeilHanlon | what can I do to help? I am happy to go through and submit reviews on projects switching centos to NV and replacing them with Rocky | 15:01 |
| NeilHanlon | i mean not "happy" in that it brings me joy, since I hate to really have to do it, but. | 15:01 |
| noonedeadpunk | NeilHanlon: most helpful would be to ensure rocky is not backporting that as well atm :) | 15:02 |
| noonedeadpunk | change is brought by gnupg2-2.3.3-3 | 15:04 |
| NeilHanlon | gnupg2-2.3.3-2.el9_0.x86_64.rpm 20-Sep-2022 20:42 2599359 | 15:06 |
| NeilHanlon | rhel 9.2 beta shipping -2, as well | 15:07 |
| NeilHanlon | https://skiprocky.linuxdn.org/tmp/rockygitcompare/rhel9beta_vs_rocky9_full.html | 15:07 |
| noonedeadpunk | I wonder if they've released 2.3.3-3 after this page being generated... | 15:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Replace CentOS 9 Jobs with Rockylinux https://review.opendev.org/c/openstack/openstack-ansible/+/879671 | 16:16 |
| noonedeadpunk | I wonder how far we are from marking CentOS 9 S as experimental | 16:16 |
| noonedeadpunk | But I kinda hate where this ends up | 16:17 |
| noonedeadpunk | And clueless what else we can do | 16:17 |
| noonedeadpunk | except wait for couple of weeks to all repos updating their gpg keys... | 16:18 |
| noonedeadpunk | Well NFV has already pushed their and it should be in mirrors really soon | 16:18 |
| noonedeadpunk | So we will be technically unblocked, until zuul images will be updated so it will hit Base repo | 16:19 |
| damiandabrowski | do you think it's reasonable to (temporarily) set c9s to non-voting for now? | 16:20 |
| noonedeadpunk | Yeah, I've pushed 879671 actually for that as well. | 16:23 |
| noonedeadpunk | But I'd expect things being fixed in the morning to be frank even without it. | 16:23 |
| noonedeadpunk | Unless they will break again | 16:23 |
| damiandabrowski | ahhh sorry, I missed it. Great then | 16:25 |
| opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible master: Add support for TLS backends https://review.opendev.org/c/openstack/openstack-ansible/+/879085 | 17:24 |
| opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible master: [DNM] Enable openstack_backend_service_https https://review.opendev.org/c/openstack/openstack-ansible/+/879501 | 17:24 |
| opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible-os_horizon master: Change default value for horizon_enable_ssl https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/879515 | 17:25 |
| opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible-os_horizon master: Fix horizon_enable_ssl logic https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/879514 | 17:25 |
| opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible-os_horizon master: Rename horizon_enable_ssl to horizon_backend_https https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/879516 | 17:25 |
| opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible-os_horizon master: Add PKI support to horizon backends https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/879517 | 17:25 |
| opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible-os_horizon master: Add PKI support to horizon backends https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/879517 | 17:29 |
| opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible-os_horizon master: Add PKI support to horizon backends https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/879517 | 18:04 |
| spatel | jamesdenton noonedeadpunk i have booked my flight for Vancouver :) | 19:01 |
| noonedeadpunk | sweet ) | 19:01 |
| spatel | I will see you there with some action plan!! | 19:01 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Remove old GPG key in advance https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/879680 | 19:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Replace CentOS 9 Jobs with Rockylinux https://review.opendev.org/c/openstack/openstack-ansible/+/879671 | 19:19 |
| noonedeadpunk | oh, seems like NFV repo is already fixed | 19:21 |
| noonedeadpunk | as centos metal is passing now | 19:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Remove nova-scheduler from ironic compute container_skel https://review.opendev.org/c/openstack/openstack-ansible/+/879482 | 19:25 |
| noonedeadpunk | so hopefully I will be able to fix stable branches tomorrow... | 19:32 |
| * NeilHanlon still is thinking about sneaking away to Vancouver... | 20:02 | |
| NeilHanlon | here is my dilemma... i need to be in Charlotte, NC on June 9-10... and then Infra summit is 13-15... and we're trying to buy a house this summer 😅 | 20:03 |
| opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible master: [DNM] Enable openstack_backend_service_https https://review.opendev.org/c/openstack/openstack-ansible/+/879501 | 22:47 |
| opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible-repo_server master: Add TLS support to repo_server backends https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/876429 | 22:59 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Provide custom handler name to PKI role https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/875779 | 23:19 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Fix haproxy_service_configs format conversion https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/878771 | 23:19 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!