Friday, 2024-01-26

kevkohttps://review.opendev.org/q/topic:%22kolla-neutron-dns-integration%2209:14
kevkoI think ^^ this is ready, can I get some reviews ? :) 09:14
kevkohttps://review.opendev.org/q/topic:%22kolla-horizon-fix%2209:16
kevkoAnd also this ^^ please 09:16
kevkobbezak ? 09:31
kevkofrickler ? 09:32
SvenKieskefrickler is afaik out of office for most of the day09:55
kevko:'(09:55
*** ravlew is now known as Guest44410:09
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: [DNM] build test  https://review.opendev.org/c/openstack/kolla-ansible/+/90682110:32
SvenKieskekevko: bbezak: do we have a (new?) CI/Gate problem? https://review.opendev.org/c/openstack/kolla/+/90647910:44
SvenKieskeah nvm, just a timeout10:44
SvenKieskekevko, could you elaborate a little around https://review.opendev.org/c/openstack/kolla-ansible/+/906229 ? I know it's just CI, but I seem to not know about the build failures.10:51
opendevreviewWill Szumski proposed openstack/kayobe master: Remove docker devicemapper support  https://review.opendev.org/c/openstack/kayobe/+/90638611:31
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: [DNM] build test  https://review.opendev.org/c/openstack/kolla-ansible/+/90682112:13
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Fix podman cross-dependency build  https://review.opendev.org/c/openstack/kolla-ansible/+/90622912:24
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Rework horizon role to support local_settings.d  https://review.opendev.org/c/openstack/kolla-ansible/+/90634712:24
opendevreviewGrzegorz Koper proposed openstack/kolla-ansible master: Configure missing nova services to expose vendordata over configdrive  https://review.opendev.org/c/openstack/kolla-ansible/+/90584312:26
kevkoSvenKieske: will chck 12:27
kevkoSvenKieske: well, it's important bugfix and we can merge probably ..I've fixed globals as bbezak asked in comment 12:28
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: [DNM] build test  https://review.opendev.org/c/openstack/kolla-ansible/+/90682112:35
kevkoguys, what about to rework kolla to install all openstack stuff from git 13:40
kevkofor example now i need to install glance_store for one customer ..and i really want to download from custom glance store git 13:41
kevkoi know how to rework kolla in downstream repo 13:41
kevkobut still ... for stable version which are not supported sometimes (quite often) upstream merged a fix ..but didn't release new version into pip 13:42
kevkoso idea is, install openstack python source from git based on stable branch 13:42
kevkoand other libs from pip as it is now 13:43
kevkoSvenKieske: ? 13:44
kevkothis of course can be an option for kolla-build command 13:44
SvenKieskemhm, I'm not sure I understand, you mean install everything in openstack from stable git branches, not just kolla-ansible?13:53
SvenKieskekevko: do you need a special glance patch? in my experience most our "upstream" openstack projects release frequent enough?13:55
kevkoSvenKieske: yeah I need .. it's normal that you need to customize your image ..13:56
kevkoSvenKieske: point is that services are downloaded from tarballs repo ...OR user can override source by his git and reversion (this works for services) ...but this don't work for libraries ...oslo...etc ..etc 13:57
kevkoSvenKieske: another example -> i need to fix this https://review.opendev.org/c/openstack/oslo.messaging/+/880189 in yoga 13:58
kevkoSvenKieske: but noone care ... and pip packe will not be released 13:58
SvenKieskewell, afaik our downstream doesn't really customize images, at least not with regards to custom backports, only for 1 or 2 edgecases maybe13:58
kevkoSvenKieske: we also try to not, but sometimes there is no other way13:59
kevkoSvenKieske: lastly we upgraded from wallaby to yoga and we realized that octavia changed octavia driver to point to v2 endpoint ...from v1 14:00
SvenKieskeI didn't even know kolla-build doesn't support this, that seems weird, so that should maybe be a two-step process, as usual: first) enable k-a to be able to use git branches for openstack projects second) maybe make them default14:00
kevkoSvenKieske: so is the way to failover 1000+ LBs ? 14:00
kevkoSvenKieske: no, we patched octavia 14:00
SvenKieskekevko: I'm pretty sure that octavia driver endpoint stuff was in some reno ;)14:00
kevkoSvenKieske: and what ? 14:01
SvenKieskelol wut? why not failover LBs?14:01
kevkoSvenKieske: 1000 ? 14:01
kevkoSvenKieske: at once ? 14:01
kevkoSvenKieske: k8s LBs ? 14:01
SvenKieskewhy do you need to do that at once?14:01
kevkoSvenKieske: because of k8s ...14:01
SvenKieskeif you have 1000 LBs and don't read release notes I don't know what to tell you, btw :)14:02
kevkoSvenKieske: 1500 clusters are still trying and trying ..and nothing 14:02
kevkoSvenKieske: we read ..we fixed images ...14:02
SvenKieskekevko: I also managed k8s LBs in octavia (not so many, true), and I needed to frequently failover them, and never did I failover all at once?14:02
kevkoSvenKieske: our customer said ..yes we can migrate to v2 ...but it consume some time ..and upgrade is planned now 14:03
kevkoSvenKieske: Okay, another example, we have a customer with VMWare connected to openstack , 1, openstack is removing VMWare 2. morevoer, they have some special usecase how to do it little bit different 14:04
SvenKieskeI think the reasoning "we need to failover 1000 lbs over at once because it's k8s" needs at least more explanation..14:04
kevkoi need to replace glance store, nova, neutron 14:04
kevkoforget on it..focus on second example 14:04
SvenKieskekevko: I'm all for the idea of the implementation but the ops stories behind this really sound..weird :)14:04
SvenKieskeregardless of motivation it seems like a nice feature, well at least if it doesn't require hundreds of lines of new code14:05
SvenKieskethis might make it e.g. also easier to test upstream code changes in dev clusters (deployed with k-a), no?14:06
kevkoSvenKieske: what is weird on the fact that some customer built some infra ...they have it for years ...for example on some old version ...and now after years they asked us to just ..help us 14:06
kevkoSvenKieske: yes, of course14:06
kevko*because we are not able to migrate ...14:07
SvenKieskekevko: ah don't trigger me :D I had the same type of customers as well. very weird. I'm the most surprised how - relatively - well their platforms run without any maintenance.14:07
kevkoSvenKieske: main question is ...if we install services from stable branch ...why not openstack libs ? 14:10
SvenKieskesure, good question at least imho :) maybe someone has an answer, or just write a patch and get shut down with a detailed answer why it's a bad idea ;)14:11
kevkoI discussed on PTG 2 years ago i think :D 14:12
SvenKieskeI guess providing this as an option should be fine, as it doesn't hurt and it should not increase the maintenance burden by a lot  I guess.14:12
SvenKieskeah I was not there, unfortunately, if there are logs it might be worth to look at those?14:13
kevkoi don't know 14:13
opendevreviewMerged openstack/kolla master: Revert "Fix gnocchi and skyline after requirements change"  https://review.opendev.org/c/openstack/kolla/+/90647914:13
kevkokevko: it was some time ago when binary images were still built 14:13
kevkoSvenKieske: I was the one who protested against ...14:13
SvenKieske:D14:14
kevkothat was quite good ...if I needed to fix something ..i've just built a package and place into repo 14:14
kevkorebuild image 14:14
SvenKieskekevko: thinking about it: are you really sure it doesn't work? just superficially looking at it, I think it should work? but I'm still a little bit ill, maybe my brain is making fun of me :D14:15
kevkoSvenKieske: where do you see apt install python-glance for example ? 14:19
SvenKieskewell, you could provide your own tarballs-base, technically? I can understand that this is not a great solution I guess.14:23
SvenKieskethinking about it, you can point it to a reverse proxy in front of opendev.org which only packages your patches on the fly into tarballs, all other requests go straight through :D14:24
SvenKieskeokay, a proper solution might be nicer14:25
SvenKieskemhm looking at the code I see at least no easy solution, it's at least a midsized amount of code needed I guess14:27
kevkoSvenKieske: you cannot 14:30
kevkoSvenKieske: you can't create a tarball for oslo.config and install it ..there is no support for it14:30
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: [DNM] test system scope for nova-compute-ironic  https://review.opendev.org/c/openstack/kolla-ansible/+/90685815:46
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: [DNM] test system scope for nova-compute-ironic  https://review.opendev.org/c/openstack/kolla-ansible/+/90685815:48
johnsomThere is no need to failover the LBs due to the amphorav2 change, they are compatible as far as I know. It was just a controller workflow change16:07
kevkojohnsom: it was not working :( 16:25
johnsomCan you open a launchpad bug for Octavia that has what you saw?16:27
johnsomWe strive to make the upgrades as smooth as possible.16:28
kevkojohnsom: there is a chance that it's actually bug in https://github.com/kubernetes/cloud-provider-openstack16:29
kevko(don't know the version)16:29
johnsomOk, the Octavia doesn’t maintain that, but it would still be good information for the team if that is causing upgrade problems for people. A launchpad bug would be useful16:31
kevkojohnsom: as it's customer ..i will try to ask them for a specific description ..16:31
SvenKieskejohnsom: well what is a common setting is, that amphoras/lbs completely fail when some automation is too trigger happy, we even created a script to clean it up in the database: https://github.com/osism/openstack-resource-manager/blob/main/src/loadbalancer.py16:33
johnsomWe have upgrade test jobs that check for issues like this, so it is best if we can learn more16:34
johnsomYou should never mess with the statuses in the database16:35
SvenKieskejohnsom: my problem might be something different, but it manifests as well with cloud-provider-openstack, which is an automation in kubernetes to use openstack, e.g. for loadbalancers16:35
SvenKieskejohnsom: I know, but octavia can get in a buggy state where a loadbalancer gets stuck..let me recollect the knowledge..16:36
johnsomI have not heard of this script or looked at what it does, but if it is in the Octavia database it is a bad idea and unnecessary16:36
SvenKieskejonsom: the only feasible thing to do then is go into the db, because you can't even delete the LB16:36
SvenKieskejohnsom: believe me, it is necessary. this was written by people with some good openstack knowledge. I should check if I ever reported a bug against octavia though..16:37
johnsomIt should not unless you do not gracefully shutdown your controllers, even then the jobboard feature addresses that.16:38
johnsomIt is a common misconception that things get “Stuck”, especially if you have non-production configs like devstack uses16:39
SvenKieskejohnsom: so, from the top of my head, the problem is like this: a) you have an LB in "Pending_update" right? b) amphoras get deleted/are unavailable. you can't failover because both amphoras are dead. you want to clean this up. but the database state of "pending update" makes this futile, because it marks all amphora items via API as read only until the update is complete. however the update can never 16:39
SvenKieskecomplete, because the instances are gone.16:39
SvenKieskejohnsom: yeah jobboard might help. unfortunately we are in the kolla channel, which means we hadn't jobbboard support for a long time, I think it's still not merged.16:40
kevkoyep16:41
johnsomWell, I have run thousands of lbs in production, was a former PTL, and was part of the team that created Octavia, so I also have “some” experience.16:41
SvenKieskeand my 2 cents are: all modern system must be designed the way that I can walk into the DC and pull power from a random server and everything still works :)16:41
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: [DNM] disable enforce scope ironic  https://review.opendev.org/c/openstack/kolla-ansible/+/90685816:41
SvenKieskemight be this is a kolla only problem then :)16:41
SvenKieskebut still a problem, because kolla is quite widely used16:42
SvenKieskeI wish some upstreams would contribute in downstream, boring, deploymentprojects like ours :D (which you seem to do, at least by taking part in the discussion, so thanks!)16:42
SvenKieskelet me dig up that jobboard patch16:43
johnsomNo, they will timeout and go to ERROR state at which point you can failover or delete via the API16:43
SvenKieskejohnsom: maybe my explanation from memory was wrong (very likely) but the bug I'm talking about, the LBs never timed out and never went to error state, thus you needed to force the errorstate via DB :(16:44
johnsomYeah, OpenStack has not addressed the “pull power on a controller with active work” well. This is why we implemented the jobboard feature. You will still have neutron and nova pain, but the state will be resumed in Octavia16:45
SvenKieskemhm, seems we actually did merge that: https://review.opendev.org/c/openstack/kolla-ansible/+/88858816:45
SvenKieskewell, merged november, so hopefully no problem for new deployments then..16:46
johnsomOur problem is there are something like 15 different deployment tooling projects that support Octavia, we can’t be hands on with all of them.16:47
SvenKieskesure! :) I can understand that :)16:47
SvenKieskeour reno has a good description of the problem: https://review.opendev.org/c/openstack/kolla-ansible/+/888588/12/releasenotes/notes/octavia_jobboard-823f44393f3e109e.yaml#516:47
* SvenKieske has still memories of fixing loadbalancers manually in the DB at 03:00 in the night..16:48
johnsomYeah, that is what we added jobboard to address16:48
SvenKieskejohnsom: so sorry for blaming octavia! had to blame kolla instead :D16:49
SvenKieskemy boss actually posted that patch only after I explained "no, we can't clean up this LBs any other way than going straight into the DB" :D16:49
johnsomYou should never need to touch the DB with Octavia. If you think you are in that situation, open a bug and come talk to us about it.16:50
SvenKieskeso I guess I raised awareness enough. I hadn't known jobboard myself until then.16:50
SvenKieskejohnsom: good to know, will do.16:50
kevkojohnsom: but i am sure that i can explain problem what we had 16:50
SvenKieskejohnsom: that being said; I haven't looked at newer release but I think the default log levels inside amphoras could be set to zero I guess? if this is not already the case.16:51
johnsomThe state model is rich and concurrency aware in Octavia, so messing with that in the DB can mean resources get lost and LBs get in bad states16:51
johnsomYeah, you have full control of the log levels from the config settings16:52
SvenKieskerest assured, I always try to find different solutions first. but sometimes I need also solutions for OPs stuck on past releases with no option to install a backport or a new feature :/16:52
SvenKieskejohnsom: yes I know, that's why I'm talking about _default_ ;)16:53
SvenKieskethe 2 GB default volume in an amphora quickly fills up with logs if you have some decent load on your loadbalancer. it's no big issue, but one I have seen quite often actually.16:54
SvenKieskeit often also leads to completely dead amphoras.16:54
johnsomRight, that is the devstack default, not a production setting. I also encourage using log offloading where you don’t store any logs in the amphora.16:55
johnsomI have to go to a meeting. I encourage you to ask questions in the lbaas channel and to open bugs.16:56
kevkojohnsom: thank you 16:56
kevkojohnsom: btw, you have a review from me I think 16:57
kevkojohnsom: https://review.opendev.org/c/openstack/octavia/+/90676916:57
kevkojohnsom: it's same code as for v2 ..description in commit message16:57
SvenKieskemhm, that conversation lead me to an idea for a small patch :D17:06
opendevreviewMerged openstack/kolla-ansible master: [CI] Fix podman cross-dependency build  https://review.opendev.org/c/openstack/kolla-ansible/+/90622917:07
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Correct glance.conf example in external Ceph guide  https://review.opendev.org/c/openstack/kolla-ansible/+/90688017:09
opendevreviewJake Hutchinson proposed openstack/kolla-ansible master: Ironic parameter rework and default NTP server  https://review.opendev.org/c/openstack/kolla-ansible/+/89303117:09
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/zed: Correct glance.conf example in external Ceph guide  https://review.opendev.org/c/openstack/kolla-ansible/+/90690117:10
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/yoga: Correct glance.conf example in external Ceph guide  https://review.opendev.org/c/openstack/kolla-ansible/+/90690217:10
opendevreviewSven Kieske proposed openstack/kolla-ansible master: remove redis precheck for octavia  https://review.opendev.org/c/openstack/kolla-ansible/+/90690417:14
opendevreviewSven Kieske proposed openstack/kolla-ansible master: remove redis precheck for octavia  https://review.opendev.org/c/openstack/kolla-ansible/+/90690417:16
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix neutron DNS integration  https://review.opendev.org/c/openstack/kolla-ansible/+/90585219:02
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Test neutron DNS integration and designate  https://review.opendev.org/c/openstack/kolla-ansible/+/90564419:02
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Test neutron DNS integration and designate  https://review.opendev.org/c/openstack/kolla-ansible/+/90564419:39

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!