| kevko | https://review.opendev.org/q/topic:%22kolla-neutron-dns-integration%22 | 09:14 |
|---|---|---|
| kevko | I think ^^ this is ready, can I get some reviews ? :) | 09:14 |
| kevko | https://review.opendev.org/q/topic:%22kolla-horizon-fix%22 | 09:16 |
| kevko | And also this ^^ please | 09:16 |
| kevko | bbezak ? | 09:31 |
| kevko | frickler ? | 09:32 |
| SvenKieske | frickler is afaik out of office for most of the day | 09:55 |
| kevko | :'( | 09:55 |
| *** ravlew is now known as Guest444 | 10:09 | |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: [DNM] build test https://review.opendev.org/c/openstack/kolla-ansible/+/906821 | 10:32 |
| SvenKieske | kevko: bbezak: do we have a (new?) CI/Gate problem? https://review.opendev.org/c/openstack/kolla/+/906479 | 10:44 |
| SvenKieske | ah nvm, just a timeout | 10:44 |
| SvenKieske | kevko, could you elaborate a little around https://review.opendev.org/c/openstack/kolla-ansible/+/906229 ? I know it's just CI, but I seem to not know about the build failures. | 10:51 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Remove docker devicemapper support https://review.opendev.org/c/openstack/kayobe/+/906386 | 11:31 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: [DNM] build test https://review.opendev.org/c/openstack/kolla-ansible/+/906821 | 12:13 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Fix podman cross-dependency build https://review.opendev.org/c/openstack/kolla-ansible/+/906229 | 12:24 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Rework horizon role to support local_settings.d https://review.opendev.org/c/openstack/kolla-ansible/+/906347 | 12:24 |
| opendevreview | Grzegorz Koper proposed openstack/kolla-ansible master: Configure missing nova services to expose vendordata over configdrive https://review.opendev.org/c/openstack/kolla-ansible/+/905843 | 12:26 |
| kevko | SvenKieske: will chck | 12:27 |
| kevko | SvenKieske: well, it's important bugfix and we can merge probably ..I've fixed globals as bbezak asked in comment | 12:28 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: [DNM] build test https://review.opendev.org/c/openstack/kolla-ansible/+/906821 | 12:35 |
| kevko | guys, what about to rework kolla to install all openstack stuff from git | 13:40 |
| kevko | for example now i need to install glance_store for one customer ..and i really want to download from custom glance store git | 13:41 |
| kevko | i know how to rework kolla in downstream repo | 13:41 |
| kevko | but still ... for stable version which are not supported sometimes (quite often) upstream merged a fix ..but didn't release new version into pip | 13:42 |
| kevko | so idea is, install openstack python source from git based on stable branch | 13:42 |
| kevko | and other libs from pip as it is now | 13:43 |
| kevko | SvenKieske: ? | 13:44 |
| kevko | this of course can be an option for kolla-build command | 13:44 |
| SvenKieske | mhm, I'm not sure I understand, you mean install everything in openstack from stable git branches, not just kolla-ansible? | 13:53 |
| SvenKieske | kevko: do you need a special glance patch? in my experience most our "upstream" openstack projects release frequent enough? | 13:55 |
| kevko | SvenKieske: yeah I need .. it's normal that you need to customize your image .. | 13:56 |
| kevko | SvenKieske: point is that services are downloaded from tarballs repo ...OR user can override source by his git and reversion (this works for services) ...but this don't work for libraries ...oslo...etc ..etc | 13:57 |
| kevko | SvenKieske: another example -> i need to fix this https://review.opendev.org/c/openstack/oslo.messaging/+/880189 in yoga | 13:58 |
| kevko | SvenKieske: but noone care ... and pip packe will not be released | 13:58 |
| SvenKieske | well, afaik our downstream doesn't really customize images, at least not with regards to custom backports, only for 1 or 2 edgecases maybe | 13:58 |
| kevko | SvenKieske: we also try to not, but sometimes there is no other way | 13:59 |
| kevko | SvenKieske: lastly we upgraded from wallaby to yoga and we realized that octavia changed octavia driver to point to v2 endpoint ...from v1 | 14:00 |
| SvenKieske | I didn't even know kolla-build doesn't support this, that seems weird, so that should maybe be a two-step process, as usual: first) enable k-a to be able to use git branches for openstack projects second) maybe make them default | 14:00 |
| kevko | SvenKieske: so is the way to failover 1000+ LBs ? | 14:00 |
| kevko | SvenKieske: no, we patched octavia | 14:00 |
| SvenKieske | kevko: I'm pretty sure that octavia driver endpoint stuff was in some reno ;) | 14:00 |
| kevko | SvenKieske: and what ? | 14:01 |
| SvenKieske | lol wut? why not failover LBs? | 14:01 |
| kevko | SvenKieske: 1000 ? | 14:01 |
| kevko | SvenKieske: at once ? | 14:01 |
| kevko | SvenKieske: k8s LBs ? | 14:01 |
| SvenKieske | why do you need to do that at once? | 14:01 |
| kevko | SvenKieske: because of k8s ... | 14:01 |
| SvenKieske | if you have 1000 LBs and don't read release notes I don't know what to tell you, btw :) | 14:02 |
| kevko | SvenKieske: 1500 clusters are still trying and trying ..and nothing | 14:02 |
| kevko | SvenKieske: we read ..we fixed images ... | 14:02 |
| SvenKieske | kevko: I also managed k8s LBs in octavia (not so many, true), and I needed to frequently failover them, and never did I failover all at once? | 14:02 |
| kevko | SvenKieske: our customer said ..yes we can migrate to v2 ...but it consume some time ..and upgrade is planned now | 14:03 |
| kevko | SvenKieske: Okay, another example, we have a customer with VMWare connected to openstack , 1, openstack is removing VMWare 2. morevoer, they have some special usecase how to do it little bit different | 14:04 |
| SvenKieske | I think the reasoning "we need to failover 1000 lbs over at once because it's k8s" needs at least more explanation.. | 14:04 |
| kevko | i need to replace glance store, nova, neutron | 14:04 |
| kevko | forget on it..focus on second example | 14:04 |
| SvenKieske | kevko: I'm all for the idea of the implementation but the ops stories behind this really sound..weird :) | 14:04 |
| SvenKieske | regardless of motivation it seems like a nice feature, well at least if it doesn't require hundreds of lines of new code | 14:05 |
| SvenKieske | this might make it e.g. also easier to test upstream code changes in dev clusters (deployed with k-a), no? | 14:06 |
| kevko | SvenKieske: what is weird on the fact that some customer built some infra ...they have it for years ...for example on some old version ...and now after years they asked us to just ..help us | 14:06 |
| kevko | SvenKieske: yes, of course | 14:06 |
| kevko | *because we are not able to migrate ... | 14:07 |
| SvenKieske | kevko: ah don't trigger me :D I had the same type of customers as well. very weird. I'm the most surprised how - relatively - well their platforms run without any maintenance. | 14:07 |
| kevko | SvenKieske: main question is ...if we install services from stable branch ...why not openstack libs ? | 14:10 |
| SvenKieske | sure, good question at least imho :) maybe someone has an answer, or just write a patch and get shut down with a detailed answer why it's a bad idea ;) | 14:11 |
| kevko | I discussed on PTG 2 years ago i think :D | 14:12 |
| SvenKieske | I guess providing this as an option should be fine, as it doesn't hurt and it should not increase the maintenance burden by a lot I guess. | 14:12 |
| SvenKieske | ah I was not there, unfortunately, if there are logs it might be worth to look at those? | 14:13 |
| kevko | i don't know | 14:13 |
| opendevreview | Merged openstack/kolla master: Revert "Fix gnocchi and skyline after requirements change" https://review.opendev.org/c/openstack/kolla/+/906479 | 14:13 |
| kevko | kevko: it was some time ago when binary images were still built | 14:13 |
| kevko | SvenKieske: I was the one who protested against ... | 14:13 |
| SvenKieske | :D | 14:14 |
| kevko | that was quite good ...if I needed to fix something ..i've just built a package and place into repo | 14:14 |
| kevko | rebuild image | 14:14 |
| SvenKieske | kevko: thinking about it: are you really sure it doesn't work? just superficially looking at it, I think it should work? but I'm still a little bit ill, maybe my brain is making fun of me :D | 14:15 |
| kevko | SvenKieske: where do you see apt install python-glance for example ? | 14:19 |
| SvenKieske | well, you could provide your own tarballs-base, technically? I can understand that this is not a great solution I guess. | 14:23 |
| SvenKieske | thinking about it, you can point it to a reverse proxy in front of opendev.org which only packages your patches on the fly into tarballs, all other requests go straight through :D | 14:24 |
| SvenKieske | okay, a proper solution might be nicer | 14:25 |
| SvenKieske | mhm looking at the code I see at least no easy solution, it's at least a midsized amount of code needed I guess | 14:27 |
| kevko | SvenKieske: you cannot | 14:30 |
| kevko | SvenKieske: you can't create a tarball for oslo.config and install it ..there is no support for it | 14:30 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: [DNM] test system scope for nova-compute-ironic https://review.opendev.org/c/openstack/kolla-ansible/+/906858 | 15:46 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: [DNM] test system scope for nova-compute-ironic https://review.opendev.org/c/openstack/kolla-ansible/+/906858 | 15:48 |
| johnsom | There is no need to failover the LBs due to the amphorav2 change, they are compatible as far as I know. It was just a controller workflow change | 16:07 |
| kevko | johnsom: it was not working :( | 16:25 |
| johnsom | Can you open a launchpad bug for Octavia that has what you saw? | 16:27 |
| johnsom | We strive to make the upgrades as smooth as possible. | 16:28 |
| kevko | johnsom: there is a chance that it's actually bug in https://github.com/kubernetes/cloud-provider-openstack | 16:29 |
| kevko | (don't know the version) | 16:29 |
| johnsom | Ok, the Octavia doesn’t maintain that, but it would still be good information for the team if that is causing upgrade problems for people. A launchpad bug would be useful | 16:31 |
| kevko | johnsom: as it's customer ..i will try to ask them for a specific description .. | 16:31 |
| SvenKieske | johnsom: well what is a common setting is, that amphoras/lbs completely fail when some automation is too trigger happy, we even created a script to clean it up in the database: https://github.com/osism/openstack-resource-manager/blob/main/src/loadbalancer.py | 16:33 |
| johnsom | We have upgrade test jobs that check for issues like this, so it is best if we can learn more | 16:34 |
| johnsom | You should never mess with the statuses in the database | 16:35 |
| SvenKieske | johnsom: my problem might be something different, but it manifests as well with cloud-provider-openstack, which is an automation in kubernetes to use openstack, e.g. for loadbalancers | 16:35 |
| SvenKieske | johnsom: I know, but octavia can get in a buggy state where a loadbalancer gets stuck..let me recollect the knowledge.. | 16:36 |
| johnsom | I have not heard of this script or looked at what it does, but if it is in the Octavia database it is a bad idea and unnecessary | 16:36 |
| SvenKieske | jonsom: the only feasible thing to do then is go into the db, because you can't even delete the LB | 16:36 |
| SvenKieske | johnsom: believe me, it is necessary. this was written by people with some good openstack knowledge. I should check if I ever reported a bug against octavia though.. | 16:37 |
| johnsom | It should not unless you do not gracefully shutdown your controllers, even then the jobboard feature addresses that. | 16:38 |
| johnsom | It is a common misconception that things get “Stuck”, especially if you have non-production configs like devstack uses | 16:39 |
| SvenKieske | johnsom: so, from the top of my head, the problem is like this: a) you have an LB in "Pending_update" right? b) amphoras get deleted/are unavailable. you can't failover because both amphoras are dead. you want to clean this up. but the database state of "pending update" makes this futile, because it marks all amphora items via API as read only until the update is complete. however the update can never | 16:39 |
| SvenKieske | complete, because the instances are gone. | 16:39 |
| SvenKieske | johnsom: yeah jobboard might help. unfortunately we are in the kolla channel, which means we hadn't jobbboard support for a long time, I think it's still not merged. | 16:40 |
| kevko | yep | 16:41 |
| johnsom | Well, I have run thousands of lbs in production, was a former PTL, and was part of the team that created Octavia, so I also have “some” experience. | 16:41 |
| SvenKieske | and my 2 cents are: all modern system must be designed the way that I can walk into the DC and pull power from a random server and everything still works :) | 16:41 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: [DNM] disable enforce scope ironic https://review.opendev.org/c/openstack/kolla-ansible/+/906858 | 16:41 |
| SvenKieske | might be this is a kolla only problem then :) | 16:41 |
| SvenKieske | but still a problem, because kolla is quite widely used | 16:42 |
| SvenKieske | I wish some upstreams would contribute in downstream, boring, deploymentprojects like ours :D (which you seem to do, at least by taking part in the discussion, so thanks!) | 16:42 |
| SvenKieske | let me dig up that jobboard patch | 16:43 |
| johnsom | No, they will timeout and go to ERROR state at which point you can failover or delete via the API | 16:43 |
| SvenKieske | johnsom: maybe my explanation from memory was wrong (very likely) but the bug I'm talking about, the LBs never timed out and never went to error state, thus you needed to force the errorstate via DB :( | 16:44 |
| johnsom | Yeah, OpenStack has not addressed the “pull power on a controller with active work” well. This is why we implemented the jobboard feature. You will still have neutron and nova pain, but the state will be resumed in Octavia | 16:45 |
| SvenKieske | mhm, seems we actually did merge that: https://review.opendev.org/c/openstack/kolla-ansible/+/888588 | 16:45 |
| SvenKieske | well, merged november, so hopefully no problem for new deployments then.. | 16:46 |
| johnsom | Our problem is there are something like 15 different deployment tooling projects that support Octavia, we can’t be hands on with all of them. | 16:47 |
| SvenKieske | sure! :) I can understand that :) | 16:47 |
| SvenKieske | our reno has a good description of the problem: https://review.opendev.org/c/openstack/kolla-ansible/+/888588/12/releasenotes/notes/octavia_jobboard-823f44393f3e109e.yaml#5 | 16:47 |
| * SvenKieske has still memories of fixing loadbalancers manually in the DB at 03:00 in the night.. | 16:48 | |
| johnsom | Yeah, that is what we added jobboard to address | 16:48 |
| SvenKieske | johnsom: so sorry for blaming octavia! had to blame kolla instead :D | 16:49 |
| SvenKieske | my boss actually posted that patch only after I explained "no, we can't clean up this LBs any other way than going straight into the DB" :D | 16:49 |
| johnsom | You should never need to touch the DB with Octavia. If you think you are in that situation, open a bug and come talk to us about it. | 16:50 |
| SvenKieske | so I guess I raised awareness enough. I hadn't known jobboard myself until then. | 16:50 |
| SvenKieske | johnsom: good to know, will do. | 16:50 |
| kevko | johnsom: but i am sure that i can explain problem what we had | 16:50 |
| SvenKieske | johnsom: that being said; I haven't looked at newer release but I think the default log levels inside amphoras could be set to zero I guess? if this is not already the case. | 16:51 |
| johnsom | The state model is rich and concurrency aware in Octavia, so messing with that in the DB can mean resources get lost and LBs get in bad states | 16:51 |
| johnsom | Yeah, you have full control of the log levels from the config settings | 16:52 |
| SvenKieske | rest assured, I always try to find different solutions first. but sometimes I need also solutions for OPs stuck on past releases with no option to install a backport or a new feature :/ | 16:52 |
| SvenKieske | johnsom: yes I know, that's why I'm talking about _default_ ;) | 16:53 |
| SvenKieske | the 2 GB default volume in an amphora quickly fills up with logs if you have some decent load on your loadbalancer. it's no big issue, but one I have seen quite often actually. | 16:54 |
| SvenKieske | it often also leads to completely dead amphoras. | 16:54 |
| johnsom | Right, that is the devstack default, not a production setting. I also encourage using log offloading where you don’t store any logs in the amphora. | 16:55 |
| johnsom | I have to go to a meeting. I encourage you to ask questions in the lbaas channel and to open bugs. | 16:56 |
| kevko | johnsom: thank you | 16:56 |
| kevko | johnsom: btw, you have a review from me I think | 16:57 |
| kevko | johnsom: https://review.opendev.org/c/openstack/octavia/+/906769 | 16:57 |
| kevko | johnsom: it's same code as for v2 ..description in commit message | 16:57 |
| SvenKieske | mhm, that conversation lead me to an idea for a small patch :D | 17:06 |
| opendevreview | Merged openstack/kolla-ansible master: [CI] Fix podman cross-dependency build https://review.opendev.org/c/openstack/kolla-ansible/+/906229 | 17:07 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Correct glance.conf example in external Ceph guide https://review.opendev.org/c/openstack/kolla-ansible/+/906880 | 17:09 |
| opendevreview | Jake Hutchinson proposed openstack/kolla-ansible master: Ironic parameter rework and default NTP server https://review.opendev.org/c/openstack/kolla-ansible/+/893031 | 17:09 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/zed: Correct glance.conf example in external Ceph guide https://review.opendev.org/c/openstack/kolla-ansible/+/906901 | 17:10 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/yoga: Correct glance.conf example in external Ceph guide https://review.opendev.org/c/openstack/kolla-ansible/+/906902 | 17:10 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible master: remove redis precheck for octavia https://review.opendev.org/c/openstack/kolla-ansible/+/906904 | 17:14 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible master: remove redis precheck for octavia https://review.opendev.org/c/openstack/kolla-ansible/+/906904 | 17:16 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix neutron DNS integration https://review.opendev.org/c/openstack/kolla-ansible/+/905852 | 19:02 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Test neutron DNS integration and designate https://review.opendev.org/c/openstack/kolla-ansible/+/905644 | 19:02 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Test neutron DNS integration and designate https://review.opendev.org/c/openstack/kolla-ansible/+/905644 | 19:39 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!