| opendevreview | Takashi Kajinami proposed openstack/tripleo-validations master: Remove six https://review.opendev.org/c/openstack/tripleo-validations/+/841176 | 00:29 |
|---|---|---|
| opendevreview | Merged openstack/validations-libs master: Expand validations_logs table with reason of failure https://review.opendev.org/c/openstack/validations-libs/+/804392 | 00:46 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-validations master: Remove six https://review.opendev.org/c/openstack/tripleo-validations/+/841176 | 00:48 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: Designate: Replace hiera by lookup https://review.opendev.org/c/openstack/puppet-tripleo/+/841183 | 01:07 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add tripleo_container_standalone role https://review.opendev.org/c/openstack/tripleo-ansible/+/838905 | 01:08 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: tripleo_keystone: Use tripleo_container_standalone https://review.opendev.org/c/openstack/tripleo-ansible/+/838906 | 01:08 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add tripleo_iscsid role https://review.opendev.org/c/openstack/tripleo-ansible/+/838907 | 01:08 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add tripleo_mysql_client role https://review.opendev.org/c/openstack/tripleo-ansible/+/840321 | 01:08 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add initial standalone playbooks and inventory for a compute node https://review.opendev.org/c/openstack/tripleo-ansible/+/840509 | 01:08 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add tripleo_bootstrap role to standlone install phase https://review.opendev.org/c/openstack/tripleo-ansible/+/840675 | 01:08 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add tripleo_ssh_known_hosts role to standalone configure phase https://review.opendev.org/c/openstack/tripleo-ansible/+/840676 | 01:08 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add standalone bootstrap phase and playbook https://review.opendev.org/c/openstack/tripleo-ansible/+/840677 | 01:08 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add pre-network and network phase and playbooks https://review.opendev.org/c/openstack/tripleo-ansible/+/840903 | 01:08 |
| opendevreview | Merged openstack/validations-common master: Setting default inventory to 0 length string https://review.opendev.org/c/openstack/validations-common/+/839862 | 01:13 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: Octavia: Replace hiera by lookup https://review.opendev.org/c/openstack/puppet-tripleo/+/841184 | 01:13 |
| opendevreview | Merged openstack/validations-common master: Removing references to Babel https://review.opendev.org/c/openstack/validations-common/+/840178 | 01:13 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: Metrics/QDR: Replace hiera by lookup https://review.opendev.org/c/openstack/puppet-tripleo/+/841185 | 01:19 |
| *** rlandy|bbl is now known as rlandy|out | 01:22 | |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: Octavia: Replace hiera by lookup https://review.opendev.org/c/openstack/puppet-tripleo/+/841184 | 01:30 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: Designate: Replace hiera by lookup https://review.opendev.org/c/openstack/puppet-tripleo/+/841183 | 01:30 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: Designate: Replace hiera by lookup https://review.opendev.org/c/openstack/puppet-tripleo/+/841183 | 01:31 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: tripleo_keystone: Use tripleo_container_standalone https://review.opendev.org/c/openstack/tripleo-ansible/+/838906 | 01:32 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add tripleo_iscsid role https://review.opendev.org/c/openstack/tripleo-ansible/+/838907 | 01:32 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add tripleo_mysql_client role https://review.opendev.org/c/openstack/tripleo-ansible/+/840321 | 01:32 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add initial standalone playbooks and inventory for a compute node https://review.opendev.org/c/openstack/tripleo-ansible/+/840509 | 01:32 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add tripleo_bootstrap role to standlone install phase https://review.opendev.org/c/openstack/tripleo-ansible/+/840675 | 01:32 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add tripleo_ssh_known_hosts role to standalone configure phase https://review.opendev.org/c/openstack/tripleo-ansible/+/840676 | 01:32 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add standalone bootstrap phase and playbook https://review.opendev.org/c/openstack/tripleo-ansible/+/840677 | 01:32 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add pre-network and network phase and playbooks https://review.opendev.org/c/openstack/tripleo-ansible/+/840903 | 01:32 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: Ironic: Replace hiera by lookup https://review.opendev.org/c/openstack/puppet-tripleo/+/841186 | 01:35 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: Ironic-inspector: Replace hiera by lookup https://review.opendev.org/c/openstack/puppet-tripleo/+/841187 | 01:35 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: RabbitMQ: Replace hiera by lookup https://review.opendev.org/c/openstack/puppet-tripleo/+/841188 | 01:44 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: haproxy: Enable httplog for horizon endpoint https://review.opendev.org/c/openstack/puppet-tripleo/+/841192 | 02:22 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: Horizon: Support Strict-Transport-Security header https://review.opendev.org/c/openstack/puppet-tripleo/+/841194 | 02:38 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Horizon: Support Strict-Transport-Security header https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841196 | 02:46 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: dnm: test hsts header configuration https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841197 | 02:46 |
| opendevreview | Steve Baker proposed openstack/openstack-virtual-baremetal master: Remove centos-7 support https://review.opendev.org/c/openstack/openstack-virtual-baremetal/+/841199 | 03:32 |
| opendevreview | Steve Baker proposed openstack/openstack-virtual-baremetal master: WIP Also start a redfish based BMC for OVB nodes https://review.opendev.org/c/openstack/openstack-virtual-baremetal/+/841200 | 03:32 |
| *** soniya29 is now known as soniya29|ruck | 04:38 | |
| *** ysandeep|out is now known as ysandeep|rover | 04:42 | |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Horizon: Support Strict-Transport-Security header https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841196 | 04:49 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: dnm: test hsts header configuration https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841197 | 04:49 |
| opendevreview | Steve Baker proposed openstack/openstack-virtual-baremetal master: Also start a redfish based BMC for OVB nodes https://review.opendev.org/c/openstack/openstack-virtual-baremetal/+/841200 | 04:53 |
| *** pojadhav|afk is now known as pojadhav | 05:23 | |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-common master: Remove reference to Undercloud services https://review.opendev.org/c/openstack/tripleo-common/+/841203 | 05:31 |
| opendevreview | Takashi Kajinami proposed openstack/python-tripleoclient master: Undercloud: Deprecate options for removed services https://review.opendev.org/c/openstack/python-tripleoclient/+/841205 | 05:57 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Remove leftover of Undercloud-specific services https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841206 | 06:06 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Remove unused environment files for undercloud https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841207 | 06:08 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: haproxy: Enable httplog for horizon endpoint https://review.opendev.org/c/openstack/puppet-tripleo/+/841192 | 06:10 |
| opendevreview | Takashi Kajinami proposed openstack/python-tripleoclient master: Remove unused object storage client implementation https://review.opendev.org/c/openstack/python-tripleoclient/+/841208 | 06:29 |
| opendevreview | Takashi Kajinami proposed openstack/python-tripleoclient master: Remove unused object storage client implementation https://review.opendev.org/c/openstack/python-tripleoclient/+/841208 | 06:32 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Horizon: Support Strict-Transport-Security header https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841196 | 06:37 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: dnm: test hsts header configuration https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841197 | 06:37 |
| opendevreview | Francesco Pantano proposed openstack/tripleo-heat-templates master: Do not deploy ceph-nfs during the overcloud deploy https://review.opendev.org/c/openstack/tripleo-heat-templates/+/839474 | 06:42 |
| opendevreview | Takashi Kajinami proposed openstack/python-tripleoclient master: Fix typo in heat.conf for standalone heat https://review.opendev.org/c/openstack/python-tripleoclient/+/841214 | 07:23 |
| opendevreview | Takashi Kajinami proposed openstack/python-tripleoclient master: Remove ineffective parameter from standalone/ephemeral heat.conf https://review.opendev.org/c/openstack/python-tripleoclient/+/841215 | 07:23 |
| *** jpena|off is now known as jpena | 07:30 | |
| opendevreview | Jiri Podivin proposed openstack/validations-libs master: Validation help improvement https://review.opendev.org/c/openstack/validations-libs/+/807828 | 07:45 |
| opendevreview | Marios Andreou proposed openstack/tripleo-ci master: Remove centos-7 job definitions and zuul layouts https://review.opendev.org/c/openstack/tripleo-ci/+/839518 | 07:56 |
| opendevreview | Marios Andreou proposed openstack/tripleo-ci master: Revert "Adds tripleo-repos/-operator-ansible collection push jobs" https://review.opendev.org/c/openstack/tripleo-ci/+/841223 | 07:56 |
| *** soniya29|ruck is now known as soniya29|ruck|lunch | 08:09 | |
| *** ykarel is now known as ykarel|away | 08:25 | |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Add back FRR to Undercloud https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841235 | 08:31 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Add back FRR to Undercloud https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841235 | 08:33 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Add back FRR to Undercloud https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841235 | 08:34 |
| opendevreview | Takashi Kajinami proposed openstack/python-tripleoclient master: Undercloud: Deprecate options for removed services https://review.opendev.org/c/openstack/python-tripleoclient/+/841205 | 08:37 |
| opendevreview | Khomesh Thakre proposed openstack/tripleo-upgrade master: Install pkgs required to apply workarounds https://review.opendev.org/c/openstack/tripleo-upgrade/+/839683 | 08:37 |
| opendevreview | Khomesh Thakre proposed openstack/tripleo-upgrade master: Install pkgs required to apply workarounds https://review.opendev.org/c/openstack/tripleo-upgrade/+/839683 | 08:46 |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 08:49 | |
| opendevreview | Marios Andreou proposed openstack/tripleo-heat-templates master: DNM TESTING tripleo-ci/+/839518 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841242 | 09:13 |
| opendevreview | Khomesh Thakre proposed openstack/tripleo-upgrade master: Install pkgs required to apply workarounds https://review.opendev.org/c/openstack/tripleo-upgrade/+/839683 | 09:17 |
| opendevreview | Marios Andreou proposed openstack/tripleo-common master: DNM testing tripleo-ci/+/839518 https://review.opendev.org/c/openstack/tripleo-common/+/841243 | 09:18 |
| opendevreview | Marios Andreou proposed openstack/tripleo-heat-templates stable/wallaby: DNM TESTING tripleo-ci/+/839518 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840945 | 09:20 |
| *** soniya29|ruck|lunch is now known as soniya29|ruck | 09:21 | |
| opendevreview | Marios Andreou proposed openstack/tripleo-heat-templates stable/train: DNM TESTING tripleo-ci/+/839518 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841244 | 09:25 |
| opendevreview | Marios Andreou proposed openstack/tripleo-common stable/wallaby: DNM testing tripleo-ci/+/839518 https://review.opendev.org/c/openstack/tripleo-common/+/840946 | 09:30 |
| *** bhagyashris is now known as bhagyashris|out | 09:33 | |
| opendevreview | Marios Andreou proposed openstack/tripleo-common stable/train: DNM testing tripleo-ci/+/839518 https://review.opendev.org/c/openstack/tripleo-common/+/841245 | 09:33 |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 09:47 | |
| slaweq | Tengu: I have the same issue with iptables rules on OSP-17 on rhel 9 | 09:59 |
| slaweq | so it's not only on master | 09:59 |
| Tengu | slaweq: iptables, or iptables-nft ? | 10:01 |
| slaweq | iptables-nft | 10:01 |
| Tengu | ok.. | 10:02 |
| Tengu | I'm checking the nftables content, with the default things we're supposed to have. also, I'll check to switch to nftables in tripleo. | 10:02 |
| Tengu | apparently, there are things in neutron that are already hitting it anyway. | 10:02 |
| Tengu | time to switch away from iptables imho. | 10:02 |
| opendevreview | A R proposed openstack/diskimage-builder master: Preserve local mirrors when using Ubuntu element https://review.opendev.org/c/openstack/diskimage-builder/+/841247 | 10:09 |
| *** rlandy|out is now known as rlandy | 10:33 | |
| slaweq | Tengu: I think that I found something and I'm one step closer | 11:08 |
| Tengu | slaweq: oh?? please, share :) | 11:09 |
| slaweq | Tengu: it seems that we our icmp packets are for some reason first hitting rule from raw table to "notrack" them (it's only for geneve tunnels but for some reason it hits it) and because of that they aren't matched later in INPUT chain | 11:09 |
| Tengu | o_O | 11:10 |
| slaweq | if You would remove those 2 rules from raw table or add rule to match all UNTRACKED traffic in INPUT table, it should works - at least it works in my lab like that | 11:10 |
| bogdando | rlandy: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/831259 is likely good to go | 11:10 |
| Tengu | and then they hit the chain policy but not the actual drop rule ? | 11:10 |
| slaweq | I still don't understand exactly why it's like that but, as I said, it's one step closer at least | 11:10 |
| slaweq | for some reason yes, but I don't know exactly why | 11:11 |
| slaweq | ahh, righ | 11:11 |
| slaweq | it's because DROP rule at the end is only for conntrack NEW packets | 11:11 |
| slaweq | and those are UNTRACKED | 11:11 |
| Tengu | yep | 11:11 |
| opendevreview | chandan kumar proposed openstack/tripleo-ansible master: Rename jobs to centos-stream and use stream9 image as base https://review.opendev.org/c/openstack/tripleo-ansible/+/839688 | 11:11 |
| Tengu | was about to say that :) | 11:11 |
| slaweq | that makes sense | 11:11 |
| Tengu | hmm... can't we track that UNTRACKED? | 11:12 |
| Tengu | sounds terrible. | 11:12 |
| slaweq | I only don't understand why those packets are encapsulated | 11:12 |
| slaweq | that "notrack" rule was added there on purpose | 11:12 |
| slaweq | and we should have it for the geneve tunnel traffic | 11:12 |
| slaweq | anyway, I'm going for lunch now but I will later update LP and will continue work on that | 11:13 |
| rlandy | bogdando: thank - checking testproject | 11:14 |
| bogdando | rlandy: ok, I've also linked a testproject job there it had been tested | 11:15 |
| Tengu | slaweq: soooo - we may allow UNTRACKED icmp only? | 11:15 |
| rlandy | bogdando; yep - thank you - merging | 11:15 |
| rlandy | ysandeep|rover: soniya29|ruck: ^^ fyi | 11:15 |
| rlandy | fs039 on train | 11:15 |
| Tengu | ouch... nope. it's.. well. it's for UDP, not icmp anyway ? | 11:15 |
| bogdando | one little fixed tempest test, one huge step for tripleo :D | 11:16 |
| soniya29|ruck | rlandy, ack | 11:18 |
| opendevreview | Merged openstack/openstack-tempest-skiplist master: Revert "Skip failing test on train c8" https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/831259 | 11:25 |
| *** dviroel|afk is now known as dviroel | 11:28 | |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add tripleo_iscsid role https://review.opendev.org/c/openstack/tripleo-ansible/+/838907 | 11:39 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add tripleo_mysql_client role https://review.opendev.org/c/openstack/tripleo-ansible/+/840321 | 11:39 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add initial standalone playbooks and inventory for a compute node https://review.opendev.org/c/openstack/tripleo-ansible/+/840509 | 11:39 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add tripleo_bootstrap role to standlone install phase https://review.opendev.org/c/openstack/tripleo-ansible/+/840675 | 11:39 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add tripleo_ssh_known_hosts role to standalone configure phase https://review.opendev.org/c/openstack/tripleo-ansible/+/840676 | 11:39 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add standalone bootstrap phase and playbook https://review.opendev.org/c/openstack/tripleo-ansible/+/840677 | 11:39 |
| opendevreview | James Slagle proposed openstack/tripleo-ansible master: Add pre-network and network phase and playbooks https://review.opendev.org/c/openstack/tripleo-ansible/+/840903 | 11:39 |
| opendevreview | Francesco Pantano proposed openstack/tripleo-ansible stable/wallaby: Avoid failing if no pools/keys are specified https://review.opendev.org/c/openstack/tripleo-ansible/+/841252 | 11:44 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Horizon: Support Strict-Transport-Security header https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841196 | 11:46 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: dnm: test hsts header configuration https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841197 | 11:46 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: Horizon: Support Strict-Transport-Security header https://review.opendev.org/c/openstack/puppet-tripleo/+/841194 | 11:50 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Horizon: Support Strict-Transport-Security header https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841196 | 11:52 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: dnm: test hsts header configuration https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841197 | 11:52 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Add back FRR to Undercloud https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841235 | 11:53 |
| opendevreview | Takashi Kajinami proposed openstack/python-tripleoclient master: Undercloud: Deprecate options for removed services https://review.opendev.org/c/openstack/python-tripleoclient/+/841205 | 11:54 |
| *** soniya29|ruck is now known as soniya29|ruck|break | 11:58 | |
| opendevreview | Slawek Kaplonski proposed openstack/tripleo-heat-templates master: Iptables rule for geneve traffic should match UNTRACKED state https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841259 | 12:26 |
| opendevreview | Slawek Kaplonski proposed openstack/tripleo-ansible master: Get all of the INPUT rules in a dedicated chain https://review.opendev.org/c/openstack/tripleo-ansible/+/839163 | 12:26 |
| slaweq | Tengu: I proposed patch which should fix that issue ^^ | 12:27 |
| *** soniya29|ruck|break is now known as soniya29|ruck | 12:27 | |
| slaweq | and I already updated Your patch and rechecked testproject's patch | 12:27 |
| slaweq | so lets see if that will work with default policy DROP | 12:27 |
| slaweq | TBH I think that it always worked only because geneve traffic was matching default policy which was ACCEPT | 12:28 |
| Tengu | slaweq: more than probably, yep. Wow. that was a good hunt at least! | 12:30 |
| Tengu | slaweq: finger crossed :). In parallel, I'm working on moving to nftables. Shouldn't be THAT hard, especially with the dedicated chain. | 12:30 |
| Tengu | though I see we're hitting some other tables than "filter", such as that "raw" one, and iirc I saw some "nat" as well. | 12:31 |
| Tengu | might be good to get some dedicated chains in there as well. | 12:31 |
| slaweq | I'm not nftables expert but IIRC it shouldn't be mixed e.g. nftables and iptables-nft and neutron is only working with iptables-nft currently | 12:32 |
| slaweq | but maybe I'm wrong there | 12:32 |
| Tengu | hmm, I see the neutron things in nft list ruleset at least. | 12:32 |
| Tengu | buuut... humpf. yeah. I think I've seen something about that in the nftables doc. | 12:32 |
| Tengu | though I suspect it wouldn't be that hard to make neutron nftables-aware. | 12:32 |
| slaweq | I'm not so sure, I know ralonsoh was working on that some time ago and he had some troubles with it | 12:35 |
| slaweq | but I don't remember now exactly what was the problem there | 12:35 |
| slaweq | probably at some point we will have to move to it, or we will not use iptables in neutron when we will move completly to OVN but that's not current state for sre | 12:36 |
| slaweq | *sure | 12:36 |
| Tengu | ^^ | 12:37 |
| Tengu | so my goal here is to expose this new possibility, and make it switchable (i.e. toggle a var to switch from iptables_nft to nftables). Same interface in the end, just the engine under the hood. | 12:37 |
| Tengu | that will allow some nice testing capabilities. | 12:37 |
| ysandeep|rover | hjensas: following yesterday conversation, looks like the ANSIBLE_SSH_RETRIES is already set to 3 in tripleoclient. https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/utils.py#L481 | 12:53 |
| slaweq | Tengu++ | 12:54 |
| ysandeep|rover | hjensas, I think there is no option to enhance ansible verbosity in node provision command, I think I need to set "ANSIBLE_VERBOSITY" and "ANSIBLE_DEBUG" in node_provision script before calling node provision - if I want to check debug logs. | 12:56 |
| ysandeep|rover | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/0222a8a/logs/undercloud/home/zuul/tripleo_overcloud_node_provision.sh.txt.gz | 12:57 |
| ysandeep|rover | hjensas, We are still hitting node provisioning issue in master with ssh timeout but not seeing same in wallaby c9. | 12:58 |
| hjensas | ysandeep|rover: I think adding '--verbose' to the overcloud node provision command will give you debug logs? | 12:58 |
| ysandeep|rover | thanks, let me try that | 12:59 |
| ysandeep|rover | we are still awaiting today's load input from vexx, but not seeing same issue in wallaby c9 which also run on same env - is a bit weird. | 13:00 |
| ysandeep|rover | hjensas: --verbose worked thanks! | 13:02 |
| hjensas | ysandeep|rover: yes, if it was a performance issue in infra we should see it with wallaby as well. | 13:07 |
| opendevreview | A R proposed openstack/diskimage-builder master: Preserve local mirrors when using Ubuntu element https://review.opendev.org/c/openstack/diskimage-builder/+/841247 | 13:07 |
| ysandeep|rover | hjensas, for ex. fs020 node provisioning failed in master vs passed in wallaby | 13:15 |
| ysandeep|rover | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/69a4390/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz | 13:15 |
| ysandeep|rover | https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/de43aed/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz | 13:15 |
| ysandeep|rover | hjensas, to debug this issue further I am thinking about rerunning wallaby and master in single testproject so that we know load on infra is equal for both release jobs with 2 changes:- 1) Increased ANSIBLE_SSH_RETRIES B) With --verbose in node provisioning command.. | 13:17 |
| ysandeep|rover | and see in logs if for master - ssh_retries connect after few more retries | 13:19 |
| ysandeep|rover | hjensas, please let me know if you have any other pointers to debug in any alternative way. | 13:19 |
| hjensas | ysandeep|rover: Looking at recent patches, https://review.opendev.org/c/openstack/diskimage-builder/+/840825 is ssh related. But dib is branchless so should see in Wallaby as well. | 13:24 |
| ysandeep|rover | hjensas, we currently don't call cloud-init element in image so I think that will not affect us. | 13:26 |
| ysandeep|rover | https://review.opendev.org/c/openstack/tripleo-common/+/841067/1/image-yaml/overcloud-hardened-images-uefi-python3.yaml is not merged yet. | 13:26 |
| hjensas | ysandeep|rover: ah, yes you are right. Just saw the comment on that review. | 13:26 |
| opendevreview | Douglas Viroel proposed openstack/tripleo-quickstart master: Update centos-compose dependecy config https://review.opendev.org/c/openstack/tripleo-quickstart/+/841268 | 13:30 |
| *** dasm|off is now known as dasm | 13:31 | |
| opendevreview | Merged openstack/tripleo-heat-templates stable/wallaby: Set rx permissions for all in /var/lib/config-data https://review.opendev.org/c/openstack/tripleo-heat-templates/+/838755 | 13:35 |
| opendevreview | Merged openstack/validations-common master: Normalizing VF metadata https://review.opendev.org/c/openstack/validations-common/+/840460 | 13:35 |
| hjensas | ysandeep|rover: We may want to try to capture journal on the node we fail SSH connection to? | 13:57 |
| ysandeep|rover | hjensas, https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/69a4390/logs/overcloud-controller-0/var/log/extra/journal.txt.gz | 14:01 |
| ysandeep|rover | "controller-0: Failed to connect to the host via ssh: ssh: connect to host" failed on controller-0 | 14:02 |
| slaweq | Tengu: test job failed but I don't think it failed due to my patch | 14:03 |
| slaweq | can You check it maybe? | 14:03 |
| slaweq | as You know those tripleo bits much better than me :) | 14:03 |
| Tengu | slaweq: gimme a minute | 14:10 |
| * Tengu sticks his head out of nftables dark places | 14:10 | |
| hjensas | ysandeep|rover: oh, thanks. Cool, I thought we had failed to capture that. *looking* | 14:10 |
| slaweq | thx Tengu | 14:11 |
| Tengu | slaweq: ah. you just hit the RDO issue with node provisionning. | 14:11 |
| slaweq | good, that it's not my patch's fault this time :) | 14:11 |
| Tengu | it's known. and under review right now. maybe rlandy and/or ysandeep|rover have info about that one? | 14:11 |
| Tengu | ysandeep|rover, rlandy any news for RDO node provision issue? :) | 14:12 |
| ysandeep|rover | Tengu, not yet, that's what I am discussing with hjensas currently. | 14:13 |
| Tengu | ysandeep|rover: ah, good - sorry, didn't really pay attention to the backlog.. | 14:13 |
| Tengu | slaweq: I may be able to run a test here with both patches in. though tomorrow morning will be a bit terrible for me, have different appointments. | 14:14 |
| slaweq | Tengu: no problem, I'm pretty confident that my patch should help (it did for me locally for sure) | 14:15 |
| slaweq | and I think I know why there wasn't that problem in Your lab | 14:15 |
| Tengu | conntrack is a mess. | 14:15 |
| slaweq | the reason is that You had only one controller so no L3HA and because of that this geneve traffic wasn't needed | 14:15 |
| Tengu | oh. | 14:16 |
| Tengu | yay small infra right? :) | 14:16 |
| slaweq | yeah, try lab with 2 controllers and You will reproduce the same issue :) | 14:16 |
| Tengu | I can run the deploy on the big builder and get 3 controllers :). 2 aren't actually a great idea. | 14:16 |
| hjensas | ysandeep|rover: Wonder if the legacy network service is causing a blipp - https://paste.opendev.org/show/bzM0IYC23hcCNKycWWlj/ | 14:30 |
| hjensas | ysandeep|rover: May 10 06:12:38 overcloud-controller-0 NetworkManager[1130]: <info> [1652177558.7952] device (enp3s0): Activation: successful, device activated. | 14:30 |
| hjensas | ysandeep|rover: and then May 10 06:13:38 Starting LSB: Bring up/down networking... | 14:31 |
| hjensas | ysandeep|rover: ansible failed 2022-05-10 06:13:37.289975 | 14:31 |
| rlandy | Tengu: sorry - rotating meetings - what ysandeep|rover said | 14:32 |
| Tengu | rlandy: no problem - thanks for jumping in :) | 14:33 |
| ysandeep|rover | hjensas, let me check c9 wallaby journal logs for comparision | 14:34 |
| hjensas | ysandeep|rover: also cloud-init is creating the head-admin user after the ansible login failure? | 14:35 |
| hjensas | May 10 06:13:39 overcloud-controller-0 useradd[1493]: new user: name=heat-admin, UID=1000, GID=1001, home=/home/heat-admin, shell=/bin/bash, from=none | 14:35 |
| ysandeep|rover | hjensas: sry was in a mtg, reading back | 14:38 |
| hjensas | ysandeep|rover: :) No worries. | 14:38 |
| ysandeep|rover | hjensas, if we can trust the ntp is in sync right away after boot between undercloud and overcloud nodes, then looks like cloud-init was still running and by the time ansible failed. | 14:42 |
| hjensas | ysandeep|rover: yes, looks like chrony already started | 14:43 |
| hjensas | May 10 06:12:43 overcloud-controller-0 chronyd[953]: Selected source 206.108.0.133 (2.centos.pool.ntp.org) | 14:43 |
| hjensas | May 10 06:12:43 overcloud-controller-0 chronyd[953]: System clock TAI offset set to 37 seconds | 14:43 |
| * hjensas is not sure if chrony would immideatly sync time at that point. | 14:45 | |
| ysandeep|rover | https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/playbooks/cli-overcloud-node-growvols.yaml#L48-L52 | 14:45 |
| ysandeep|rover | I need to check wait_for_connection: --> what does it test if ssh or something else. | 14:45 |
| Tengu | slaweq: also, it seems the ansible "iptables" module wants to use contrack for every single rule, and is adding the --state NEW when nothing is set, instead of just ignoring the state. | 14:49 |
| Tengu | slaweq: this also leads to the issue we just hit imho. | 14:49 |
| Tengu | I'm not sure I'll do the same thing within my nftables role.... | 14:49 |
| Tengu | state is nice, but.. | 14:49 |
| Tengu | though, if there are package with the NOTRACK, it may lead to some issues... ? | 14:50 |
| hjensas | ysandeep|rover: also, ANSIBLE_SSH_RETRIES - does it wait between retries? Is it tunable? | 14:50 |
| ysandeep|rover | hjensas, I was doing some testing in my lab, I killed overcloud node and reran the playbook: http://pastebin.test.redhat.com/1050814 | 14:52 |
| ysandeep|rover | looks like the wait increase in each loop | 14:52 |
| ysandeep|rover | pausing for 0 seconds .... pausing for 1 seconds ... pausing for 3 seconds | 14:53 |
| *** dviroel is now known as dviroel|lunch|afk | 14:53 | |
| hjensas | ysandeep|rover: yes, looks like incremental retries. indeed. | 14:53 |
| ysandeep|rover | hjensas, I am scratching my head on how this worked for same controller: " OK | Wait for provisioned nodes to boot | overcloud-controller-0" | 14:56 |
| ysandeep|rover | may be wait_for_connection don't test for ssh connection or cloud-init restarted networking after this task in between. | 14:56 |
| hjensas | ysandeep|rover: network is up, and wait_for_connection does not try to log-in? It may just ping, or try to connect to SSH server without logging in? | 14:57 |
| ysandeep|rover | ack, and login fail because cloud-init is creating the heat-admin after ansible already failed. | 14:59 |
| ysandeep|rover | yeah https://docs.ansible.com/ansible/latest/collections/ansible/builtin/wait_for_connection_module.html : This module makes use of internal ansible transport (and configuration) and the ping/win_ping module to guarantee correct end-to-end functioning. | 15:00 |
| slaweq | Tengu: yes, that also can be the problem | 15:00 |
| Tengu | slaweq: I'll do a 1:1 version, and we'll be able to iterate. | 15:01 |
| Tengu | now I just need to extract the "chain" and "table" passed in the rules so that I can create them as custom beforehand, and edit the actual original chain in order to push the traffic through the custom chain. | 15:01 |
| Tengu | i.e. "map" INPUT -> TRIPLEO_INPUT, and redirect INPUT into TRIPLEO_INPUT (and same for any other mentioned chain in any mentioned table) | 15:02 |
| opendevreview | Alan Bishop proposed openstack/tripleo-heat-templates stable/wallaby: Manila: Deprecate ineffctive *DriverHandlesShareServers https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840582 | 15:04 |
| opendevreview | Alan Bishop proposed openstack/tripleo-heat-templates stable/wallaby: Deprecate ineffective ManilaCephFSCephFSEnableSnapshots https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840581 | 15:04 |
| hjensas | ysandeep|rover: do we get "UNREACHABLE" if the wrong user is used by ansible? | 15:05 |
| hjensas | wrong as in "non existing" user? | 15:05 |
| ysandeep|rover | hjensas, let me do a quick test on my env | 15:07 |
| ysandeep|rover | hjensas, hah.. i don't see cloud-init logs in wallaby: https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/de43aed/logs/overcloud-controller-0/var/log/extra/journal.txt.gz | 15:08 |
| ysandeep|rover | may be rotated.. | 15:08 |
| ysandeep|rover | ignore clou-init ran https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/de43aed/logs/overcloud-controller-0/var/log/cloud-init-output.log.txt.gz | 15:10 |
| *** afaranha__ is now known as afaranha | 15:11 | |
| ysandeep|rover | hjensas: unreachable with permission denied: http://pastebin.test.redhat.com/1050824 | 15:20 |
| *** soniya29|ruck is now known as soniya29|out | 15:20 | |
| hjensas | ysandeep|rover: ok, so looks different from port 22: Connection refused. | 15:26 |
| ysandeep|rover | hjensas: yeah.. if wait_for_connection don't test for ssh connection.. may be in pre_tasks we should also retry for ssh connection to become available. | 15:33 |
| ysandeep|rover | https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/playbooks/cli-overcloud-node-growvols.yaml#L47 | 15:33 |
| ysandeep|rover | or we can just increase ANSIBLE_SSH_RETRIES | 15:34 |
| hjensas | ysandeep|rover: I wonder if network-scripts blips the network? I wonder if we should ensure 'systemctl disable network.service' in the image? | 15:34 |
| ysandeep|rover | I am not in sync about status of network.service in recent version, but if we don't need it anymore we can disable in image itself. | 15:38 |
| ysandeep|rover | We can test that in third party check itself - if we plan to disable via diskimage-builder patch | 15:39 |
| ysandeep|rover | hjensas: looks like we use env['ANSIBLE_SSH_RETRIES'] = 3 from tripleoclient: https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/utils.py#L481 , let me try if bumping it works in parallel. | 15:41 |
| ysandeep|rover | ^^ as a dnm patch | 15:42 |
| hjensas | ysandeep|rover: we need it later, when os-net-config is used to configure networking. But at this stage NetworkManager is doing a good job. the tripleo_network_config ansible role will enable network service when it is required in case it is disabled in the image. | 15:43 |
| opendevreview | Alan Bishop proposed openstack/python-tripleoclient master: Undercloud: Deprecate options for removed services https://review.opendev.org/c/openstack/python-tripleoclient/+/841205 | 15:44 |
| hjensas | ysandeep|rover: afict package openstack-network-scripts provide network-scripts, it enables the service in post script. | 15:44 |
| hjensas | ysandeep|rover: but, I assume the same package is used on Wallaby C9. | 15:44 |
| hjensas | ysandeep|rover: so would make sense if we saw the same error there. | 15:45 |
| ysandeep|rover | openstack-network-scripts.x86_64 10.11.1-1.el9s @delorean-wallaby-testing | 15:46 |
| hjensas | ysandeep|rover: have a Wallaby C9 logs handy? | 15:47 |
| ysandeep|rover | hjensas, https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/de43aed/ | 15:47 |
| hjensas | ysandeep|rover: :( looks like logs rotated, I don't see the initial NetworkManager/network.service stuff in journals there. | 15:50 |
| ysandeep|rover | hjensas: Its getting late here.. I will be out soon.. If you get anything please left a note on https://bugs.launchpad.net/tripleo/+bug/1970400 or ping rlandy, I will pick up in my morning. | 15:50 |
| rlandy | ysandeep|rover: hjensas: thanks for the investigation here | 15:50 |
| ysandeep|rover | hjensas, let me see a different job for wallaby and see if logs from boot time are available | 15:51 |
| hjensas | ysandeep|rover: ack, I can dig for one myself as well. | 15:51 |
| hjensas | ysandeep|rover: +1 on trying to increase ANSIBLE_SSH_RETRIES. | 15:51 |
| ysandeep|rover | hjensas, https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/utils.py#L481 i am bumping here for now, We can figure our later if that can be done at playbook level. | 15:53 |
| opendevreview | Sandeep Yadav proposed openstack/python-tripleoclient master: [DNM ]bump ANSIBLE_SSH_RETRIES for a test https://review.opendev.org/c/openstack/python-tripleoclient/+/841304 | 16:01 |
| ysandeep|rover | ahh, all the jobs i have checked so far for wallaby have journal logs rotated | 16:01 |
| ysandeep|rover | hjensas, rlandy fyi.. testing with ANSIBLE_SSH_RETRIES bump here: https://review.rdoproject.org/r/c/testproject/+/31954 | 16:04 |
| *** marios is now known as marios|out | 16:17 | |
| ysandeep|rover | hjensas, o/ I am leaving for the day, will catch up with you tomorrow, Thank you for the help and have a great rest of your day. | 16:23 |
| *** ysandeep|rover is now known as ysandeep|out | 16:23 | |
| opendevreview | Merged openstack/tripleo-validations master: Setting default inventory for multinode job to 0 length string https://review.opendev.org/c/openstack/tripleo-validations/+/840186 | 16:43 |
| opendevreview | Merged openstack/validations-libs master: Validation help improvement https://review.opendev.org/c/openstack/validations-libs/+/807828 | 16:43 |
| *** jpena is now known as jpena|off | 17:17 | |
| *** artom_ is now known as artom | 17:24 | |
| *** dasm is now known as dasm|bbl | 17:27 | |
| *** tweining is now known as tweining|off | 18:00 | |
| opendevreview | Merged openstack/tripleo-heat-templates master: Fix manila policy override config https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840440 | 18:01 |
| opendevreview | Merged openstack/tripleo-heat-templates master: Remove legacy network-isolation env files https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841112 | 18:04 |
| *** dviroel|lunch|afk is now known as dviroel\ | 18:53 | |
| *** dviroel\ is now known as dviroel | 18:53 | |
| opendevreview | Merged openstack/tripleo-docs master: fix yaml typo and add syntax highlighting https://review.opendev.org/c/openstack/tripleo-docs/+/824602 | 18:56 |
| opendevreview | Merged openstack/tripleo-heat-templates master: Add IPA client service to Cell ctrlr role https://review.opendev.org/c/openstack/tripleo-heat-templates/+/839497 | 19:04 |
| opendevreview | Merged openstack/tripleo-image-elements stable/wallaby: Migrate from testr to stestr, disable auto discovery https://review.opendev.org/c/openstack/tripleo-image-elements/+/835684 | 19:04 |
| opendevreview | Merged openstack/tripleo-puppet-elements stable/wallaby: Migrate from testr to stestr, disable auto discovery https://review.opendev.org/c/openstack/tripleo-puppet-elements/+/835792 | 19:04 |
| *** dviroel is now known as dviroel|out | 21:22 | |
| opendevreview | Merged openstack/openstack-virtual-baremetal master: Add bmc host support for centos-9-stream https://review.opendev.org/c/openstack/openstack-virtual-baremetal/+/834612 | 21:59 |
| *** rlandy is now known as rlandy|bbl | 22:17 | |
| opendevreview | Sofer Athlan-Guyot proposed openstack/tripleo-heat-templates stable/train: Ensure container's image get updated if their name stay the same. https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841347 | 22:35 |
| opendevreview | Sofer Athlan-Guyot proposed openstack/tripleo-heat-templates stable/train: Ensure container's image get updated if their name stay the same. https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841347 | 22:47 |
| opendevreview | Merged openstack/tripleo-ansible stable/train: Wrap stopping podman -t with systemd timeouts (squash) https://review.opendev.org/c/openstack/tripleo-ansible/+/840663 | 23:03 |
| opendevreview | Merged openstack/tripleo-heat-templates master: Don't cleanup catalog entries in check mode https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840463 | 23:03 |
| opendevreview | Merged openstack/tripleo-heat-templates master: Remove parameter to run stunnel by systemd https://review.opendev.org/c/openstack/tripleo-heat-templates/+/839292 | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!