Thursday, 2021-02-18

openstackgerritMichael Johnson proposed openstack/octavia master: Add support for scoped tokens and default roles  https://review.opendev.org/c/openstack/octavia/+/77595700:24
*** luksky has quit IRC00:33
*** eandersson has joined #openstack-lbaas01:35
*** eandersson has quit IRC01:42
*** eandersson has joined #openstack-lbaas01:42
*** zzzeek has quit IRC02:47
*** zzzeek has joined #openstack-lbaas02:50
*** mchlumsky0 has joined #openstack-lbaas02:55
*** mchlumsky has quit IRC02:59
*** priteau has quit IRC02:59
*** nicolasbock has quit IRC02:59
*** mchlumsky0 is now known as mchlumsky02:59
*** bcafarel has quit IRC03:00
*** nicolasbock has joined #openstack-lbaas03:00
*** bcafarel has joined #openstack-lbaas03:25
*** ramishra has quit IRC03:27
*** rcernin has quit IRC03:36
*** psachin has joined #openstack-lbaas03:42
*** rcernin has joined #openstack-lbaas04:16
*** ramishra has joined #openstack-lbaas04:21
rm_workjohnsom: you aware of any time recently where the cent8 amp image build was broken?05:06
rm_workand/or cgoncalves / gthiemonge05:07
rm_worki remember someone was working on something cent8 breakage related05:07
gthiemongerm_work: https://review.opendev.org/c/openstack/octavia/+/77541905:20
gthiemongerm_work: it works but it's ugly :D05:21
rm_workwhat was the symptom05:21
rm_workof the broken image05:21
gthiemongecgoncalves is working on a fix in devstack05:21
rm_workdid the image just NOT BUILD?05:21
rm_workor did it build but fail to boot?05:21
gthiemongeyep, it doesn't build, pip updates setuptools and then breaks some selinux tools that are used by DIB05:22
gthiemongesee the comment in https://review.opendev.org/c/openstack/octavia/+/775419/4/devstack/plugin.sh05:22
*** psachin has quit IRC05:50
rm_workah k06:03
rm_worknot my issue06:03
rm_workbut, found my issue :D06:03
*** gcheresh has joined #openstack-lbaas06:12
*** vishalmanchanda has joined #openstack-lbaas06:34
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: WIP Fix two-node job configuration  https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/77388806:45
*** eandersson has quit IRC06:53
*** eandersson has joined #openstack-lbaas06:54
*** eandersson has quit IRC06:56
*** eandersson has joined #openstack-lbaas06:56
cgoncalvesthe centos 8 amphora image build is an issue only on devstack environments07:16
cgoncalvesthe nightly amphora image build job looks fine (non-devstack env): https://zuul.openstack.org/builds?job_name=publish-openstack-octavia-amphora-image-centos807:20
*** rcernin has quit IRC07:58
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: WIP Fix two-node job configuration  https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/77388808:00
*** rcernin has joined #openstack-lbaas08:02
*** rcernin has quit IRC08:07
*** rpittau|afk is now known as rpittau08:14
*** luksky has joined #openstack-lbaas08:17
*** yamamoto has quit IRC08:49
*** rcernin has joined #openstack-lbaas08:49
*** rcernin has quit IRC08:53
*** yamamoto has joined #openstack-lbaas08:57
lukskyhi. I tried to upgrade octavia from queens to victoria version. Well... this time all goes bad :/09:14
lukskyI ended with deleteing all database, and installing fresh db schema from victoria (7.1.0) version09:15
*** yamamoto has quit IRC09:16
lukskybut the amphoras are recreated09:16
lukskyover and over again09:17
lukskyhttp://paste.openstack.org/show/802768/09:19
lukskyI can access amphoras (by ssh) for the time they exists. Tried 4 build of amphoras.09:20
gthiemongeluksky: do you have any ERROR lines in the health-manager logs?09:25
lukskythe last few lines for all logs09:27
lukskyhttp://paste.openstack.org/show/802769/09:29
lukskyafter cleaning DB - there is no ERROR in health-manager, was fighthing with this until 1:30 CET09:30
lukskyI think this is more amphora/access to amphora related - checked the SG - but the looks OK09:32
gthiemongedo you mean that you don't have any amphorae/LBs in the DB?09:32
lukskythey are recreated every few minutes09:33
lukskyit looks like this:09:35
lukskyhttp://paste.openstack.org/show/802771/09:35
gthiemongespare amphorae, right?09:38
*** ccamposr has joined #openstack-lbaas09:38
lukskyyes, there is no LB now, creating any ends in ERROR staty anyway09:39
lukskybut most anoing is this:09:39
lukskyhttp://paste.openstack.org/show/802772/09:39
lukskythere is connection with amphora, she reutrns with 200, health manager says OK and.... recreate (why, o why ?? :) )09:40
gthiemongeI think you can find the reason in health-manager.log09:43
gthiemongeit should expain why the HM triggers a failover and creates the amphorae09:43
*** yamamoto has joined #openstack-lbaas09:44
lukskywell, my eyes may be tired. Maybe don't see some obvius thing here:09:47
lukskyhttp://paste.openstack.org/show/802773/09:47
lukskythis the tail -f for few last seconds from health-manager.log09:48
gthiemongeStale amphora's id is: cd8ff0c8-58eb-46a4-a568-1d7076568ade09:49
lukskyok, which was created few minutes ago09:49
lukskyand deleted...09:50
lukskyand over and over again09:50
gthiemongeso it means the HM detected that the amp is not sending heartbeat messages09:50
*** yamamoto has quit IRC09:50
gthiemongeand it considers that there's an error, and tries to failover it09:51
gthiemongeReceived packet from ('10.99.99.41', 11488)09:51
gthiemongebut it receives an heartbeat message 1min before the failover09:51
lukskyindeed09:52
lukskythat is the part, I don't understand09:52
gthiemongeluksky: what is the heartbeat_interval value in octavia.conf?09:52
gthiemongedefault value is 1009:52
lukskyit is set to heartbeat_interval = 3009:54
*** yamamoto has joined #openstack-lbaas09:59
lukskyin http://paste.openstack.org/show/802774/ - here is log from amphora10:01
gthiemongeluksky: can you check the output of 'sudo journalctl -lf' in a newly created amphora? I'd like to see if amphora-agent is throwing some errors there10:02
gthiemongeluksky: if you don't find anything there, I would use tcpdump on the management port of the amphora (or mangement interface of the controllers) to verify that it sends an heartbeat package every 30 seconds10:04
lukskyhm..10:06
lukskythere is some error10:06
lukskyhttp://paste.openstack.org/show/802775/10:07
lukskyunable to resolve host amphora-6b4c446e-def4-413b-8581-536472ab10d510:08
gthiemongenothing important here10:08
lukskyok...10:08
gthiemongejust wait a few minutes until the amphora is deleted10:08
lukskyI have one controller now - turned off all other, to have all connections to one controller10:09
gthiemongehmmm10:13
gthiemongeI think that if you turn off the other controllers, some heartbeat messages will be lost10:14
lukskyHm... they can by not consumed on amqp servers  ?10:15
gthiemongebasically the amphora sends a UDP packet to one controller at a time (in round-robin)10:15
gthiemongeso if you set the heartbeat_interval to 10, and heartbeat_timeout to 60, it should work with 3 controllers10:16
gthiemongeyou need to ensure that (heartbeat_interval * nb_controller < heartbeat_timeout)10:19
lukskyok10:19
lukskywill adjust config for one controller and check10:22
gthiemongethe new config will only affect newly created amp, existing amp will still have 30 sec heartbeat_interval, but they will eventually disappear with the failovers10:23
*** yamamoto has quit IRC10:25
*** yamamoto has joined #openstack-lbaas10:27
lukskyhm.. but I already have heartbeat_interval = 30 and heartbeat_timeout = 60, so assuming (heartbeat_interval * nb_controller < heartbeat_timeout) = 30 * 1 < 6010:28
gthiemongeluksky: do you have only one entry in the controller_ip_port_list setting in octavia.conf?10:32
lukskyyes, NOW i have one entry, I checked on (turned off) octavia's controllers tcpdump10:33
lukskyearlier it was 3 entries, I think this was the reason. Checking, and let You now if reducing number of entries in controller_ip_port_list from 3 to 1 solved the issue.10:34
*** yamamoto has quit IRC10:36
*** yamamoto has joined #openstack-lbaas10:37
lukskygthiemonge: thank You ! the amphoras aren't flapping now10:37
*** yamamoto has quit IRC10:37
gthiemongeluksky: cool!10:38
lukskychecking if LB are created correctly :D10:38
*** yamamoto has joined #openstack-lbaas10:46
*** yamamoto has quit IRC11:07
*** yamamoto has joined #openstack-lbaas11:30
*** priteau has joined #openstack-lbaas11:34
*** yamamoto has quit IRC11:49
*** yamamoto has joined #openstack-lbaas11:53
*** njohnston has quit IRC12:16
rm_workyou also uploaded a newly built amphora image before the upgrade, right?12:21
rm_workhighly recommend that, in the case that anything did change that wasn't backwards compatible (we guarantee amp <-> controller will be compatable +/- one release either way, but you're doing further)12:22
rm_workbut, worst case they will fail over, and if the new image is available, then everything will be fine and shouldn't even have any missed connections12:22
openstackgerritAdrian Vladu proposed openstack/octavia master: Add support for building arm64 images  https://review.opendev.org/c/openstack/octavia/+/77093712:47
openstackgerritAdrian Vladu proposed openstack/octavia master: Add support for building arm64 images  https://review.opendev.org/c/openstack/octavia/+/77093712:48
*** yamamoto has quit IRC12:50
*** mzylowski has joined #openstack-lbaas12:57
*** yamamoto has joined #openstack-lbaas13:05
*** yamamoto has quit IRC13:05
*** yamamoto has joined #openstack-lbaas13:05
*** yamamoto has quit IRC13:10
*** haleyb has joined #openstack-lbaas13:30
*** yamamoto has joined #openstack-lbaas13:37
lukskyall is working correctly, I will try to downgrade to queens version and simulate once more time upgrade.13:42
lukskyrm_wark: can You confirm that these steps are enough13:44
luksky1. stop health-manager and other octavia services13:44
luksky2. install new code13:44
luksky3. octavia-db-manage upgrade head13:44
luksky4. change tag on ampora image from old (queens) to new (victoria) version13:44
luksky5. start health-manager (and other octavia services13:44
lukskyrm_work: sorry for misspelling13:45
mchlumskyrm_work we had some stein amphoras running for almost a week in an Ussuri octavia environment without anyone complaining but I was anxious to get them all upgraded.13:48
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: WIP Fix two-node job configuration  https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/77388813:50
mzylowskiHello, I'm looking for a way to restrict some networks for Octavia. In other words, I want to make impossible to create loadbalancer for given network (or even better network type). Also I want to disallow creation pool members from those networks. I tried to achieve this by policy.json file, but no luck yet (looks like network fields in policies13:53
mzylowskiare not taken into account). Any thoughts?13:53
*** yamamoto has quit IRC13:54
*** sapd1 has joined #openstack-lbaas13:56
*** yamamoto has joined #openstack-lbaas14:08
openstackgerritGregory Thiemonge proposed openstack/octavia master: Fix health manager messages about failed failovers  https://review.opendev.org/c/openstack/octavia/+/77646714:23
rm_workthat should be good luksky14:26
rm_workmzylowski: yes this is possible14:26
lukskyok, let now how goes this time :P14:26
lukskylet know*14:26
rm_workmzylowski: check the config option "valid_vip_networks"14:29
rm_workthat will limit the creation of loadbalancers to a specific set of network_ids14:29
rm_workas far as not allowing members from specific networks... I think you'd need to do that using ACLs or something in your network fabric14:30
rm_workOctavia LBs will function with members with any routable address -- so you could add some random VM from AWS to your openstack octavia LB if you want, or your home server...14:31
rm_workah, if you are using flavors, you can also set valid_vip_networks for each flavor individually14:32
mzylowskiOkey I got it, but valid_vip_networks seems not be enough for me, because we want to allow our users to create custom private networks and those networks can be used with loadbalancers. I want to block some public network and allow all the rest, even those not created yet.14:37
rm_workIf the users are not allowed to create ports on the network, i believe that will prevent it? So long as you don't have "allow invisible resource use" on14:39
rm_workThough... Hmm we might only check read access14:39
rm_workjohnsom may know better14:39
rm_workHe may be awake soon14:40
mzylowskiOkey, great :)  Thanks for Your help.14:40
haleybrm_work: hey Adam, can you take a look at https://review.opendev.org/c/openstack/octavia/+/774426 - I think that should get out tempest gate green (it's the bottom of a depends-on chain)14:46
rm_workSorry I'm in bed but I can look when I wake up14:46
rm_workWhich is... Uhh14:46
rm_workIn an hour and ten minutes, FML14:47
rm_workBack in a bit :D14:47
haleybrm_work: thanks, i was just looking at my backlog and that was the blocker14:48
*** yamamoto has quit IRC14:57
openstackgerritGregory Thiemonge proposed openstack/octavia master: Fix health manager messages about failed failovers  https://review.opendev.org/c/openstack/octavia/+/77646715:06
*** ihti[m] has joined #openstack-lbaas15:13
johnsomluksky Yes, with a heartbeat_interval of 30 and heartbeat_timeout of 60 and two out of three controllers down, you are going to have the amphora marked as failed and a failover attempted.  With the default interval of 10, it would have been fine, but with 30 you aren't giving it a chance to hit a healthy HM instance.15:28
openstackgerritTakashi Kajinami proposed openstack/octavia-dashboard master: Disable Load Balancers panel when Octavias service is not deployed  https://review.opendev.org/c/openstack/octavia-dashboard/+/76687815:36
openstackgerritTakashi Kajinami proposed openstack/octavia-dashboard master: Disable Load Balancers panel when Octavias service is not deployed  https://review.opendev.org/c/openstack/octavia-dashboard/+/76687815:37
*** EmilienM has joined #openstack-lbaas15:57
EmilienMhello everyone, I'm currently seeing https://bugs.launchpad.net/octavia/+bug/1706836 in my environment, I run Ussuri, any idea what it could be?15:58
openstackLaunchpad bug 1706836 in octavia "Amphora agent 404 No suitable network interface found" [High,Triaged]15:58
EmilienMcgoncalves: ^ fyi15:58
cgoncalvesgthiemonge, ^^ fyi :P15:59
johnsomLaunchpad hasn't been used for Octavia for years15:59
johnsomhttps://bugs.launchpad.net/octavia15:59
EmilienMright16:00
EmilienMI just found the bug on search engine16:00
johnsomYeah, the storyboard migration and launchpad doesn't close out the bugs when it's disabled in launchpad and you get migrated16:01
*** vishalmanchanda has quit IRC16:02
EmilienMany idea what it could be?16:03
johnsomI know there was a recent issue with NetworkManager doing bad things. It might be related if you are using a CentOS/RHEL image.16:04
johnsomIt would be best if you could provide the amphora logs for the issue.16:04
EmilienMI use RHEL16:05
EmilienMok I'll try to get them16:05
*** mzylowski has quit IRC16:43
*** luksky has quit IRC17:05
EmilienMjohnsom: how do you get these logs?17:08
johnsomEmilienM If you are on OSP they are in with the other container logs. If it's upstream it is in log/controller/logs/octavia-amphora_log.txt17:43
gthiemongejohnsom: I'm working with EmilienM, it looks like the lookup of the mac address in the amphora doesn't work17:46
gthiemongehttps://opendev.org/openstack/octavia/src/branch/master/octavia/amphorae/backends/agent/api_server/plug.py#L218-L22817:46
gthiemonge^  it fails but the interface exists17:46
*** rpittau is now known as rpittau|afk17:47
johnsomOh! This was a regression in pyroute2 I bet. I reported that and got it fixed. One minute.17:47
EmilienM:-O17:47
johnsomhttps://github.com/svinota/pyroute2/issues/72417:48
gthiemongelol17:48
EmilienMhttps://github.com/openstack/octavia/commit/51b93c00223ef5529d98174ab422f99ac7c7570e17:48
EmilienMseems like you have a workaround17:48
johnsom0.5.13 broken it all, 0.5.14 fixed it again17:48
EmilienMlet me see what version of pyroute2 I have in the amphora17:49
johnsom0.5.13 should be on a deny list somewhere.17:49
EmilienMyeah I have python3-pyroute2-0.5.13-1.el8ost.noarch17:51
johnsomThat build is a train wreck. We need to fix that.17:51
cgoncalveshttps://github.com/openstack/requirements/blob/stable/train/global-requirements.txt#L25417:52
cgoncalvesthough not that OSP would care for g-r.txt ...17:53
*** luksky has joined #openstack-lbaas17:54
johnsomHmm, that is interesting. I could have sworn GR was updated to deny that build17:54
johnsomThere was a long discussion about it in the requirements channel.17:54
cgoncalvesEmilienM, how are you building the amphora image?18:06
EmilienMcgoncalves: I was using the OSP image that I got from rpm18:06
EmilienMbut now I'm using the one from https://images.rdoproject.org/octavia/master/18:06
EmilienMwe'll see how that works :D18:07
EmilienMin OSP there is a package named octavia-amphora-image-x86_64, I was using that18:07
EmilienMthe image was old from November of 2020, and it seems to have the wrong version of pyroute218:07
cgoncalvesEmilienM, master amphora image running on a train octavia deployment is strongly not recommended and more likely to fail spectacularly18:08
EmilienMI run OSP1718:08
EmilienMwhich is Ussuri I guess?18:09
cgoncalvesmaster Victoria I would hope18:09
EmilienMI guess I can download an ussuri image if it fails18:09
EmilienMit's hard to tell which version of Octavia I run with OSP17 :/18:09
EmilienMactually, I can tell on which commit was built the rpm18:10
cgoncalvessorry, I meant to say master Wallaby. OSP 17 will be based on Wallaby18:10
EmilienMopenstack-octavia-api-6.1.0-0.20200828161906.0b1d8dd.el8ost.noarch18:10
EmilienMthat's my version of Octavia18:10
EmilienMwhich is a commit in Aug 26th last year18:10
EmilienMso it's Victoria IIUC18:11
EmilienMhttps://releases.openstack.org/18:11
johnsom6.x.x is Ussuri18:12
cgoncalvesOK. I really thought OSP 17 was tracking master since it will be based on Wallaby18:12
johnsomhttps://docs.openstack.org/releasenotes/octavia/ussuri.html18:12
johnsom17 should be tracking Wallaby18:13
haleybthe tip of rhos-17.0-trunk-patches has a feb-2021 change18:13
haleybMerge "Fix pools going into ERROR when updating the pool"18:13
haleyband we're talking downstream on the upstream channel if it matters18:14
EmilienMpython3-pyroute2-0.5.13-1.1.el8.noarch on master18:14
EmilienMwhich seems to have worked, my deployment is running fine18:15
haleybEmilienM: i see octavia 7.1.0 numbered rpms downstream for 17 in the latest puddle, which was last November18:17
*** ccamposr has quit IRC18:22
*** sapd1 has quit IRC18:28
gthiemongehttps://zuul.openstack.org/builds?job_name=openstack-tox-cover&project=openstack%2Foctavia18:36
gthiemongeI think that merging the SCTP commit has broken the cover job18:37
johnsomI think my patch on that list was just an issue with the oslo policy bump causing follow on issues. I can't speak for the others18:37
gthiemongeI'm checking18:38
gthiemongejohnsom: the SCTP commit passed the cover job when the threshold was 90%18:40
gthiemongeand the gates don't run openstack-tox-cover18:40
johnsomI see 91%, but... that is still below the 92% the other patch flagged.18:41
johnsomOh, that is interesting.... So, SCTP passed check on Jan 29. 92% merged on Jan 30. Then SCTP merged Feb 17th. So, yeah, since cover isn't in the gate it didn't catch that it was now under the limit.18:45
johnsomSame on the cores that reviewed that for letting in un-covered code. Grin. But, maybe we need to add cover to the gate too.18:46
*** xarlos has joined #openstack-lbaas18:47
johnsomNeat, so gthiemonge Can you spin a patch to fix the coverage in SCTP?18:47
gthiemongesure, I have some ideas ;-)18:50
johnsomI haven't read through all of this yet, but ... This promises to be an adventure: https://docs.sqlalchemy.org/en/14/changelog/migration_20.html20:52
*** dougwig has quit IRC21:08
*** dougwig has joined #openstack-lbaas21:09
*** gcheresh has quit IRC21:18
openstackgerritGregory Thiemonge proposed openstack/octavia master: Add test coverage for SCTP health checker script  https://review.opendev.org/c/openstack/octavia/+/77653521:20
johnsomWow, thanks Greg21:20
haleybsomeone's up late21:21
gthiemonge;-)21:22
johnsomOMG, I just don't know what to say.21:38
johnsomhttps://www.irccloud.com/pastebin/180NtWOE/21:39
johnsomAn import, a long constants string, all for "W"21:39
johnsomEven if I created a "WALLABY" constant in our constants file it would be less bytes21:40
haleybshouldn't it be the other way around?  W = "WALLABY"  crikey21:53
johnsomhttps://github.com/openstack/oslo.log/blob/master/oslo_log/versionutils.py#L15721:54
johnsomI think the deprecation message would read better with Wallaby than just "w"21:55
johnsomBut.... whatever21:55
haleybyou should propose a patch for X = 'Xena' to add the sanity back21:55
johnsomI have other bike sheds21:56
haleyb:)  and you must have shaved your yak already21:56
johnsomAlready sent the yak down the river in the enchanted canoe.22:08
haleybmight need to get that yak back to fix the docs job failure, we just can't win22:11
johnsomReally? lol22:13
johnsomWell, my checked out version is rendering my docs, so I guess I will not rebase22:13
haleybi'm looking at it but it's weird, and will have to merge with greg's change to make the gate green22:16
*** yamamoto has joined #openstack-lbaas22:18
haleybthe file in question hasn't changed in a year, doc/source/install/install-ubuntu.rst so sphinx or something moved forward.  probably just a trivial formatting error that went from warning to error22:18
johnsomhaleyb Hmmm, might need this: https://review.opendev.org/c/x/wsme/+/76342222:22
johnsomI can rage merge that if you decide you need it22:23
johnsomOh, it looks like it is just the "console", so yeah, change it to bash will probably work22:24
haleybjohnsom: there's other occurences of console in that file too, :-/22:25
haleybdoc/source/install/install-ubuntu.rst:269:Could not lex literal_block as "console". Highlighting skipped.22:25
haleybmaybe just change them all to bash22:28
rm_workok what did i need to look at, slept through by accident22:42
haleybrm_work: oh, the one i had asked about was https://review.opendev.org/c/openstack/octavia/+/774426 but i doubt it will merge now, damn gate22:44
johnsomI will have another RBAC spin here soon-ish22:45
haleybjohnsom: changing that to bash seems to fix it, strange since vim isn't showing all those blocks correctly so i figured it was some line above there.22:46
haleybusually the '.. foo::' are in yellow22:46
* haleyb skeedaddles22:47
rm_workhmm k22:48
*** rcernin has joined #openstack-lbaas23:03
*** mchlumsky has quit IRC23:09
*** rcernin has quit IRC23:11
*** rcernin has joined #openstack-lbaas23:12
openstackgerritMichael Johnson proposed openstack/octavia master: Add support for scoped tokens and default roles  https://review.opendev.org/c/openstack/octavia/+/77595723:17
openstackgerritMichael Johnson proposed openstack/octavia master: Add support for scoped tokens and default roles  https://review.opendev.org/c/openstack/octavia/+/77595723:32
johnsomweee, lower constraints chains23:32
johnsomI really should spend the time to build a VM that is clean enough to run that locally.23:32
openstackgerritMichael Johnson proposed openstack/octavia master: Add support for scoped tokens and default roles  https://review.opendev.org/c/openstack/octavia/+/77595723:39
openstackgerritMichael Johnson proposed openstack/octavia master: Add support for scoped tokens and default roles  https://review.opendev.org/c/openstack/octavia/+/77595723:49

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!