| opendevreview | Pierre Riteau proposed openstack/kayobe stable/zed: Handle transition of Yoga to unmaintained https://review.opendev.org/c/openstack/kayobe/+/908248 | 06:49 |
|---|---|---|
| opendevreview | Dr. Jens Harbott proposed openstack/kolla stable/zed: stable-only: Drop upgrade testing https://review.opendev.org/c/openstack/kolla/+/908249 | 06:54 |
| frickler | good point, we need to stop testing upgrades from yoga | 06:55 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Build letsencrypt images https://review.opendev.org/c/openstack/kayobe/+/907723 | 06:55 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible stable/zed: stable-only: Drop upgrade testing https://review.opendev.org/c/openstack/kolla-ansible/+/908250 | 07:00 |
| opendevreview | Pierre Riteau proposed openstack/kayobe-config-dev stable/2023.2: Use dummy1 as bridge port instead of eth1 https://review.opendev.org/c/openstack/kayobe-config-dev/+/907939 | 07:02 |
| opendevreview | Pierre Riteau proposed openstack/kayobe-config-dev stable/2023.1: Use dummy1 as bridge port instead of eth1 https://review.opendev.org/c/openstack/kayobe-config-dev/+/907940 | 07:03 |
| opendevreview | Pierre Riteau proposed openstack/kayobe-config-dev stable/zed: Use dummy1 as bridge port instead of eth1 https://review.opendev.org/c/openstack/kayobe-config-dev/+/908261 | 07:03 |
| frickler | oh, gerritbot doesn't show unmaintained patches for kolla. this is kind of good I think, wanted to ask about that in the meeting | 07:09 |
| frickler | but just for reference: remote: https://review.opendev.org/c/openstack/kolla/+/908251 Drop publishing and periodic jobs [NEW] | 07:09 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible stable/zed: stable-only: Drop upgrade testing https://review.opendev.org/c/openstack/kolla-ansible/+/908250 | 08:11 |
| mnasiadka | frickler: should we update dashboards with a separate section for unmaintained branches? | 08:26 |
| opendevreview | Merged openstack/kolla stable/zed: stable-only: Drop upgrade testing https://review.opendev.org/c/openstack/kolla/+/908249 | 08:29 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Rework horizon role to support local_settings.d https://review.opendev.org/c/openstack/kolla-ansible/+/906347 | 08:29 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Enable testing horizon https://review.opendev.org/c/openstack/kolla-ansible/+/907718 | 08:29 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: CI: Test custom routes & rules on EL9 https://review.opendev.org/c/openstack/kayobe/+/899941 | 08:45 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Support credentials for custom DNF repositories https://review.opendev.org/c/openstack/kayobe/+/908142 | 08:55 |
| frickler | mnasiadka: I would rather prefer to have all unmaintained things out of my view, but that's also something we can discuss in the meeting | 09:04 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Fix wording of confirm_deprovision docs https://review.opendev.org/c/openstack/kayobe/+/908259 | 09:16 |
| mnasiadka | frickler: any idea why https://review.opendev.org/c/openstack/kolla/+/907824 is failing on contraints? | 09:20 |
| frickler | mnasiadka: yes, because requirements hasn't got the new branch yet, so things are falling back to master constraints. should be fixed soon I hope | 09:20 |
| opendevreview | Dawud proposed openstack/kayobe master: Fix wipe-disks role to work on util-linux > 2.37 https://review.opendev.org/c/openstack/kayobe/+/907105 | 09:22 |
| mnasiadka | frickler: ack | 09:23 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: openvswitch: Stop setting fail_mode for external bridges https://review.opendev.org/c/openstack/kolla-ansible/+/908166 | 09:38 |
| opendevreview | Matt Crees proposed openstack/kayobe stable/2023.2: Fix: configure etc-hosts for overcloud group https://review.opendev.org/c/openstack/kayobe/+/908262 | 09:46 |
| opendevreview | Matt Crees proposed openstack/kayobe stable/2023.1: Fix: configure etc-hosts for overcloud group https://review.opendev.org/c/openstack/kayobe/+/908263 | 09:46 |
| opendevreview | Merged openstack/kolla-ansible master: Update keystone service user passwords https://review.opendev.org/c/openstack/kolla-ansible/+/903178 | 10:07 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Add service role to service users https://review.opendev.org/c/openstack/kolla-ansible/+/815577 | 10:23 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Ironic: enable elevated access for users with service role https://review.opendev.org/c/openstack/kolla-ansible/+/908007 | 10:29 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Template system scoped admin-openrc and clouds.yml files https://review.opendev.org/c/openstack/kolla-ansible/+/908168 | 10:29 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Revert "Disable new defaults and scope for Ironic (RBAC)" https://review.opendev.org/c/openstack/kolla-ansible/+/907274 | 10:29 |
| opendevreview | Merged openstack/kolla-ansible master: Enable HAProxy Prometheus metrics endpoint https://review.opendev.org/c/openstack/kolla-ansible/+/877118 | 10:33 |
| SvenKieske | bbezak: mnasiadka: and maybe anyone else: regarding https://review.opendev.org/c/openstack/kolla-ansible/+/815577 I'd like to hold down on merging please for a bit, because I quite don't understand why that patch is even necessary, the default keystone service user role should be "service" for a long time, so I don't know why this patch works the way it does | 10:52 |
| SvenKieske | commented on the patchset, my understanding might still be incomplete but I'm 75% sure this should be solved in a simpler fashion, but maybe I overlooked something | 11:00 |
| frickler | added some more -1 reasoning, thx for pointing that out SvenKieske | 11:10 |
| SvenKieske | frickler: thanks for the pointer about the etherpad, I was just about to comment that as well. It might also be worth to archive this etherpad somewhere longer lasting and/or update it. I did save a local copy just now. | 11:14 |
| SvenKieske | I'll take a short nap, fosdem made me sick (again) -.- see you later | 11:16 |
| frickler | unfiltered gaseous interchange with thousands of other humans makes ppl sick? what a surprise *scnr* | 11:17 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Fix: configure etc-hosts for overcloud group https://review.opendev.org/c/openstack/kayobe/+/907306 | 11:27 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Add service role to service users https://review.opendev.org/c/openstack/kolla-ansible/+/815577 | 11:28 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: openvswitch: Set fail_mode to standalone for external bridges https://review.opendev.org/c/openstack/kolla-ansible/+/908166 | 11:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: openvswitch: Set fail_mode to standalone for external bridges https://review.opendev.org/c/openstack/kolla-ansible/+/908166 | 11:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: openvswitch: Set fail_mode to standalone for external bridges https://review.opendev.org/c/openstack/kolla-ansible/+/908166 | 11:42 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: openvswitch: Set fail_mode to standalone for external bridges https://review.opendev.org/c/openstack/kolla-ansible/+/908166 | 11:44 |
| mnasiadka | frickler, SvenKieske, bbezak: https://review.opendev.org/c/openstack/kolla-ansible/+/908166 - I hope this has enough background in commit message right now :) | 11:44 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Rework horizon role to support local_settings.d https://review.opendev.org/c/openstack/kolla-ansible/+/906347 | 11:45 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Enable testing horizon https://review.opendev.org/c/openstack/kolla-ansible/+/907718 | 11:45 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix horizon deployment https://review.opendev.org/c/openstack/kolla-ansible/+/908293 | 11:45 |
| opendevreview | Alex Welsh proposed openstack/kayobe stable/yoga: Fix ansible requirements https://review.opendev.org/c/openstack/kayobe/+/908294 | 11:47 |
| frickler | mnasiadka: now that's what I call a commit message ;) thx. but now I wonder if that setting is just a workaround masking some hidden bug in the setup. why would that standalone mode be needed if OVN is running properly? | 11:51 |
| mnasiadka | so, in bash scripts we didn't supply fail_mode - so it was standalone | 11:51 |
| mnasiadka | Ansible module seems to have some weird issue on reconfigure - see here: https://b15374aad6eb87dbf340-28a36564c8fcedcf018c712a9987c604.ssl.cf5.rackcdn.com/908166/2/check/kolla-ansible-debian/dceef6d/primary/logs/ansible/reconfigure | 11:51 |
| frickler | mnasiadka: so then it was an older bug, not a new one? | 11:51 |
| mnasiadka | yeah, I'll raise a bug in openvswitch collection for tracking | 11:53 |
| frickler | "cmd": "/usr/bin/ovs-vsctl -t 5 set-fail-mode br-ex None" that looks like a bug in that module, too | 11:53 |
| frickler | but still I don't see why we need standalone instead of secure? | 11:53 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Refactor of docker worker https://review.opendev.org/c/openstack/kolla-ansible/+/908295 | 11:53 |
| mnasiadka | frickler: secure drops all flows if controller is not putting any flows in that bridge, seems in OVN case all flows are on br-int | 11:55 |
| mnasiadka | br-int though needs to have fail_mode secure | 11:55 |
| mnasiadka | as in https://patchwork.ozlabs.org/project/ovn/patch/20210507174947.1879798-1-flavio@flaviof.com/#2678886 | 11:56 |
| frickler | ah, one bridge to switch them all. I guess that's ok then, thx | 12:34 |
| SvenKieske | mhm | 12:58 |
| SvenKieske | there's a recent ovn bug with regards to snat btw, just fyi | 12:58 |
| opendevreview | Jake Hutchinson proposed openstack/kayobe master: Add NTP parameter configuration https://review.opendev.org/c/openstack/kayobe/+/895199 | 13:13 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: openvswitch: Set fail_mode to standalone for external bridges https://review.opendev.org/c/openstack/kolla-ansible/+/908166 | 13:34 |
| mnasiadka | SvenKieske: link? | 13:35 |
| mnasiadka | frickler: https://github.com/ansible-collections/openvswitch.openvswitch/issues/86 - seems there's a bug opened since 2021 :) | 13:38 |
| mnasiadka | frickler: and OSA does the same thing: https://github.com/openstack/openstack-ansible-os_neutron/commit/f94959745c59f8978d5ed2592b3f2007ff8b28aa | 13:39 |
| mnasiadka | mgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo jovial osmanlicilegi mattcrees dougszu - meeting in 9! | 13:51 |
| mnasiadka | #startmeeting kolla | 14:00 |
| opendevmeet | Meeting started Wed Feb 7 14:00:01 2024 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 14:00 |
| mnasiadka | #topic rollcall | 14:00 |
| mnasiadka | o/ | 14:00 |
| mmalchuk | \o | 14:00 |
| osmanlicilegi | o/ | 14:00 |
| dougszu | |o | 14:00 |
| jovial | \o | 14:00 |
| frickler | \o | 14:00 |
| mhiner | o/ | 14:00 |
| kevko | \o/ | 14:00 |
| bbezak | \o | 14:01 |
| mattcrees | o/ | 14:01 |
| jangutter | o/ | 14:01 |
| SvenKieske | o/ | 14:01 |
| mnasiadka | #topic agenda | 14:02 |
| mnasiadka | * CI status | 14:02 |
| mnasiadka | * Release tasks | 14:02 |
| mnasiadka | * Regular stable releases (first meeting in a month) | 14:02 |
| mnasiadka | * Current cycle planning | 14:02 |
| mnasiadka | * Additional agenda (from whiteboard) | 14:02 |
| mnasiadka | * Open discussion | 14:02 |
| mnasiadka | #topic CI status | 14:02 |
| mnasiadka | So, we broke OVN jobs - fixing that with https://review.opendev.org/c/openstack/kolla-ansible/+/908166 | 14:02 |
| mnasiadka | Rocky9 Ironic CI is suffering from ironic-api vs ironic-inspector race | 14:02 |
| mnasiadka | any other CI issues that I haven't noticed? | 14:02 |
| mnasiadka | And centos9 decided to break libvirt | 14:03 |
| bbezak | cephadm jobs? | 14:03 |
| bbezak | last time I checked those were failing | 14:03 |
| mnasiadka | yeah, those fail from time to time, but maybe there's something new in them | 14:03 |
| mnasiadka | anyway, CI needs some love | 14:03 |
| frickler | yoga is gone to unmaintained, so upgrade jobs in zed will be failing, too. I pushed patches already | 14:03 |
| mnasiadka | Ok, let's merge those if they pass | 14:04 |
| mnasiadka | #topic Release tasks | 14:04 |
| mnasiadka | R-8 week - nothing planned for it in release schedule | 14:04 |
| mnasiadka | #topic Regular stable releases | 14:05 |
| mnasiadka | Did we get in the stable releases in Jan? | 14:05 |
| bbezak | yes | 14:05 |
| mnasiadka | fantastic, so then it's time for Feb releases (excluding Yoga of course) | 14:05 |
| mnasiadka | Any volunteer? | 14:05 |
| bbezak | can do | 14:06 |
| bbezak | aka will do | 14:06 |
| frickler | updating the docs to drop yoga would also be nice | 14:06 |
| mnasiadka | good idea | 14:06 |
| mnasiadka | #topic Current cycle planning | 14:06 |
| mnasiadka | We bumped Ansible | 14:06 |
| mnasiadka | jovial: Would be nice to do the same in Kayobe | 14:07 |
| jovial | Sounds like a good idea | 14:07 |
| mnasiadka | I started working on the OVN BGP Agent | 14:07 |
| mnasiadka | The same for Ubuntu 24.04 LTS - so we don't do it last minute | 14:07 |
| SvenKieske | slightly OT but might be good to be aware of, this OVN SNAT bug, if you haven't seen it already: https://bugs.launchpad.net/bugs/2051935 | 14:08 |
| mnasiadka | SLURP patch would like some reviews I guess - https://review.opendev.org/c/openstack/kolla-ansible/+/905322 | 14:08 |
| bbezak | I think I'll add secure RBAC for ironic to the release tasks, as it looks we need to push it there this release | 14:08 |
| mnasiadka | SvenKieske: that's probably a bit unusual setup, as in two levels of routers | 14:08 |
| SvenKieske | +1 I'm happy to review the rbac and service role stuff, want to be done with it :D | 14:08 |
| SvenKieske | true, regarding the bug, but still a little bit disturbing. | 14:09 |
| mnasiadka | anybody working on anything from the list? | 14:09 |
| mnasiadka | list == whiteboard L231 | 14:09 |
| SvenKieske | I pestered some folks from OSBA regarding mirrors at fosdem | 14:10 |
| mnasiadka | Any luck? | 14:10 |
| SvenKieske | might be we actually get new mirrors for a more stable CI, but I believe it when I see it (I guess I will need to do more talking still) | 14:10 |
| mnasiadka | Ok then, good luck ;) | 14:11 |
| SvenKieske | it was promised to me under the influence of some alcohol, so let's see how the promise holds up once everyone is sober ;) | 14:11 |
| mnasiadka | Ok, let's move to topics from whiteboard | 14:12 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 14:12 |
| mnasiadka | (SvenKieske): https://bugs.launchpad.net/kolla-ansible/+bug/2049762 (service token verification in cinder wrong?) | 14:12 |
| mnasiadka | SvenKieske: I guess after bbezak's work - we could just stop setting service_token = admin? | 14:12 |
| SvenKieske | yeah | 14:12 |
| kevko | we should | 14:13 |
| SvenKieske | I proposed a singular patch for that, I was curious if our CI would break, it didn't, at least not obvious. I guess it's a matter of taste if we want two patches for that | 14:13 |
| mnasiadka | ok, we should get all service roles/tokens/ironic system scope patches as RP+1 and start reviewing them | 14:14 |
| SvenKieske | I'm fine either way, we probably need to discuss the service role patch distinctly. I think it can be much simpler than it currently is. | 14:14 |
| mnasiadka | bbezak: can you group them in a topic and do RP+1? | 14:14 |
| SvenKieske | big +1 from me on getting this stuff over the line :) | 14:14 |
| bbezak | service role is ready to review https://review.opendev.org/c/openstack/kolla-ansible/+/815577 | 14:14 |
| bbezak | the other one I have some more ideas | 14:14 |
| bbezak | but yeah, I'll group them in the topic | 14:15 |
| mnasiadka | goodie, thanks | 14:15 |
| mnasiadka | (bbezak): Service role discussion - https://review.opendev.org/c/openstack/kolla-ansible/+/815577/ | 14:15 |
| mnasiadka | anything to discuss here? | 14:15 |
| bbezak | indeed, there were some questions from frickler and SvenKieske if we need admin role still for service users | 14:16 |
| bbezak | and we probably don't for some services | 14:16 |
| bbezak | however not all projects implemented service role support | 14:16 |
| bbezak | https://etherpad.opendev.org/p/rbac-goal-tracking#L48 | 14:17 |
| frickler | but if this isn't ready upstream, do we need to adopt it at all already? | 14:17 |
| bbezak | ironic needs it, cinder apparently too for this service_token | 14:18 |
| SvenKieske | frickler: well some projects (cinder/nova) do regard our current handling of admin role as a security bug, if you read https://bugs.launchpad.net/kolla-ansible/+bug/2049762 | 14:18 |
| frickler | so then only change the accounts for those projects? also hurray for openstack doing wildly inconsistent stuff once again | 14:18 |
| jovial | Seems sane to adopt it for the projects that support it | 14:19 |
| mnasiadka | yeah, I think we should implement what works today, and track the rest of the projects | 14:19 |
| SvenKieske | I think we should configure stuff with the minimum needed roles possible, obviously. and if we need to maybe split up the existing service_ks_register role for that, fine. | 14:19 |
| mnasiadka | bbezak: seems you went in a nice rabbit hole | 14:19 |
| bbezak | :) | 14:20 |
| SvenKieske | I actually don't think we should let users override this, but if user are already using it, we can't of course deprecate this functionality this fast. | 14:20 |
| bbezak | I adopted old change that did the same thing and polished it with new services etc. | 14:20 |
| SvenKieske | it is a rabbit hole, for sure. at least I learned some stuff about ironic and rbac in keystone :) | 14:20 |
| bbezak | I'm fine with going with service role only for ironic/cinder for now | 14:20 |
| bbezak | let's see if it will work | 14:20 |
| bbezak | I'll focus on ironic | 14:21 |
| mnasiadka | There's Neutron mentioned in the rbac goals | 14:21 |
| bbezak | and SvenKieske could for cinder with his patch for service tokens | 14:21 |
| mnasiadka | Problem with service tokens in cinder is that we need to backport this all the way to zed (unmaintained/yoga ?) | 14:21 |
| mnasiadka | So maybe the question is what is the minimum set cinder needs | 14:21 |
| SvenKieske | okay for me, but then I guess I need to adapt it to explicitly use the "service" role. I'm not sure I understand our config merging code in this regard :D | 14:22 |
| SvenKieske | mnasiadka: ack | 14:22 |
| bbezak | I thought that adding service roles for all services is a solution that could would solve our issues in the future | 14:22 |
| mnasiadka | bbezak: as long as those services support it :) | 14:22 |
| bbezak | but we could do that selectively too | 14:22 |
| mnasiadka | and it seems some support it in 2023.1, some 2023.2, etc | 14:22 |
| bbezak | adding service role won't hurt :) | 14:23 |
| mnasiadka | So seems like fantastic mess | 14:23 |
| mnasiadka | yeah well, if it's not needed and supported, then maybe it doesn't make sense | 14:23 |
| bbezak | I agree | 14:23 |
| mnasiadka | so maybe we should have per service/service group patches | 14:23 |
| bbezak | 75% agree :) | 14:24 |
| mnasiadka | I know that's more work, but this way we can decide what to backport | 14:24 |
| bbezak | but yeah it is a mess. Ironic being in fact only service with system scope is somewhat breaking my mind | 14:24 |
| SvenKieske | bbezak: I think in the long run your approach is fine, I don't know if we need to patch each service for that, though. maybe have three widgets for this? $service_role_default=service $service_role_not_migrated_yet=admin $service_role_user_override_beware_here_be_dragons=foobar | 14:24 |
| bbezak | but that's different story | 14:24 |
| SvenKieske | bbezak: you are not alone in that, I still don't know if I understood all this really (I think I didn't) | 14:25 |
| SvenKieske | maybe someone needs to draw a nice flowchart how this works :D | 14:25 |
| mnasiadka | Well, I think it might make sense to implement service roles for those projects, that support that today | 14:26 |
| jovial | I guess the service user doesn't get any more perms with the service role over admin. So I can see bbezak's point of doing it in a big bang. | 14:26 |
| mnasiadka | we have a list on the etherpad | 14:26 |
| mnasiadka | I don't think just adding the role fixes anything, still we need some per-service configuration (e.g. cinder.conf) entries, right? | 14:27 |
| SvenKieske | I was under the impression bbezak did refresh that etherpad, is the information in there current, or stale? | 14:27 |
| bbezak | old etherpad with system scope is stale | 14:27 |
| SvenKieske | mnasiadka: at least for some services I think a customization is currently needed (not 100% sure) | 14:28 |
| bbezak | mnasiadka: adding service rolesis just initial thing yes | 14:28 |
| bbezak | ironic for instance needs also that - https://review.opendev.org/c/openstack/kolla-ansible/+/908007 | 14:28 |
| bbezak | if not then system scope member user | 14:28 |
| mnasiadka | yeah, I get that - and on Ironic side they enabled enforcing new defaults | 14:29 |
| mnasiadka | meaning enforcing system scope | 14:29 |
| mnasiadka | I'm not a fan of adding a role to a user just because 7 years later they might support service roles | 14:29 |
| bbezak | ok, I'll add it just for ironic. will check if it will cope with just service role, or it will need admin too | 14:30 |
| mnasiadka | I can have a look on Neutron, as in how to switch it to use service role in service-to-service communication | 14:31 |
| bbezak | furthermore the inital change for adding service roles to all projects was somewhat agreed within the comments of then PTL | 14:31 |
| bbezak | but I agree that things pivoted since then | 14:32 |
| bbezak | out of system scope most importantly | 14:32 |
| mnasiadka | We assumed every project will implement it in a reasonable time | 14:33 |
| mnasiadka | Now it seems it's not that simple | 14:33 |
| mnasiadka | scope implementation and service role are different phases | 14:33 |
| bbezak | and scope died, so :) | 14:34 |
| mnasiadka | well, system scope died | 14:34 |
| bbezak | yeah | 14:34 |
| bbezak | ok, I'm done with secure rbac for now thx :) | 14:34 |
| SvenKieske | xD | 14:34 |
| mnasiadka | #link https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#change-in-scope-implementation | 14:34 |
| mnasiadka | ok, let's move on | 14:35 |
| mnasiadka | (mnasiadka): Add unmaintained/* reviews to Gerrit dashboards (another section or stable backports) | 14:35 |
| mnasiadka | basically gerrit bot doesn't announce patches to that branch | 14:35 |
| mnasiadka | and we don't see them in the review dashboard as well | 14:36 |
| mnasiadka | so either we get a new section in the review dashboard called Unmaintained branches | 14:36 |
| mnasiadka | or we change the query for stable backports | 14:36 |
| mnasiadka | or we basically don't care | 14:36 |
| mnasiadka | which one do we choose? | 14:36 |
| SvenKieske | I'm confused (again) why do we want notifications on unmaintained branches? wasn't the point that the "unmaintained" team handles those? | 14:37 |
| * frickler don't core | 14:37 | |
| SvenKieske | so I would opt for don't care, but maybe someone has a compelling reason? I can't think of any though. | 14:37 |
| frickler | ehm ... care even ... but core is also not wrong ;) | 14:37 |
| mmalchuk | I'm confused with Kayobe - there is 2 commits in stable/yoga which are not in unmaintained/yoga | 14:37 |
| mnasiadka | Maybe another question | 14:37 |
| mnasiadka | mmalchuk: this one later | 14:37 |
| mnasiadka | Who wants to maintain unmaintained/yoga for Kolla/Kolla-Ansible apart SHPC? | 14:38 |
| mmalchuk | me | 14:38 |
| mmalchuk | I'm always care for backports | 14:38 |
| SvenKieske | I may contribute drive-by backports on a case by case basis, but I wouldn't count that officially :) | 14:38 |
| frickler | then likely you should create a kolla-unmaintained-core group and amend the gerrit acls | 14:39 |
| frickler | I can find an example patch after the meeting | 14:39 |
| SvenKieske | makes sense | 14:39 |
| mnasiadka | frickler: what if we would prefer that the kolla-core group is core for unmaintained branches? :) | 14:40 |
| frickler | mnasiadka: well I would prefer to not be core for unmaintained | 14:40 |
| SvenKieske | shouldn't kolla-core be cleaned up either way? I swear I've seen people in their where their last patch/contribution was somewhere in 2017 or so? | 14:41 |
| frickler | that's yet another topic | 14:41 |
| SvenKieske | there* | 14:41 |
| mnasiadka | frickler: but by default what are the ACLs? kolla-core has rights or some openstack-unmaintained-core? | 14:41 |
| frickler | the latter only | 14:41 |
| mnasiadka | ok | 14:41 |
| mnasiadka | I'll create kolla-unmaintained-core and kayobe-unmaintained-core | 14:42 |
| mnasiadka | and add current cores for starters, nobody is forced to review anything - just like in the usual EM branches | 14:42 |
| frickler | mnasiadka: cf. I169e52d5fb545c675549ce06fef1ca2f8eb1de86 | 14:43 |
| mnasiadka | frickler: thanks | 14:45 |
| mnasiadka | ok, let's go forward | 14:45 |
| mnasiadka | (dougszu): Discuss sending all service and infra logs to journald (don't shoot). | 14:45 |
| mnasiadka | dougszu: please elaborate ;-) | 14:45 |
| mmalchuk | about 2 commits above unmaintained/yoga in Kayobe? | 14:45 |
| mnasiadka | mmalchuk: they will be merged in unmaintained/yoga once requirements repo has unmaintained/yoga | 14:46 |
| dougszu | So basically, oslo.log can output additional logging info, that we don't currently get: https://docs.openstack.org/oslo.log/latest/admin/journal.html | 14:46 |
| mnasiadka | now nothing is mergable | 14:46 |
| dougszu | There are two ways to get the extra info - write out logs in JSON format, which means they are less readable on the box, or send everything to journald | 14:46 |
| SvenKieske | dougszu: I'm personally a big +1 on this, as it streamlines the logging infrastructure more. I don't know about the implementation though, but I guess I already commented on the patch. | 14:47 |
| kevko | only one +2 and +w for unmaintained branches ! :) | 14:47 |
| mmalchuk | mnasiadka stable/yoga would be dropped as after merge? | 14:47 |
| mnasiadka | mmalchuk: yup | 14:47 |
| mmalchuk | thanks | 14:47 |
| mnasiadka | kevko: once they start working we can go back to that | 14:47 |
| dougszu | thanks Sven - I could look at alternatives to writing direct to the journal in the patch | 14:47 |
| mnasiadka | I have a slightly complicated question - you know that RH-clones do not persist journal across reboots? | 14:48 |
| jovial | Is there a link to the patch? | 14:48 |
| dougszu | there is no patch yet - some thoughts on the etherpad: https://etherpad.opendev.org/p/KollaWhiteBoard#L63 | 14:48 |
| jovial | Persisting the journal can be enabled though, right? | 14:48 |
| frickler | mnasiadka: well thats configurable, isn't it? just create /var/log/journal | 14:48 |
| jovial | IIRC all you need to do is create the default direcotry | 14:49 |
| mnasiadka | frickler: true, but still that's something that needs to be included | 14:49 |
| mnasiadka | I assume we're speaking about not logging anymore to /var/log/kolla? | 14:49 |
| SvenKieske | well we should probably do a robust config, not just create the directory, that means taking care that it doesn't overflow etc. | 14:49 |
| dougszu | Correct - I am proposing to stop logging to /var/log/kolla, and hand over everything to journald | 14:50 |
| SvenKieske | we could also for the first part redirect /var/log/journal to /var/log/kolla, it's also configurable which directory to use, probably better for older users | 14:50 |
| mnasiadka | SvenKieske: that's not the same format, not really human readable | 14:50 |
| SvenKieske | the location is totally orthogonal to the mechanism being used.. | 14:50 |
| kevko | i like var/log/kolla :) | 14:50 |
| mmalchuk | me too | 14:50 |
| frickler | it is not only a matter of the directory, also text format vs. journal format | 14:50 |
| SvenKieske | sure it's human readable, just use "journalctl" ;) | 14:50 |
| mnasiadka | SvenKieske: cat vs journalctl, err... no | 14:51 |
| kevko | if i need to choose beetween journalctl and tail ..i am voting for tail | 14:51 |
| SvenKieske | but this is still a third orthogonal problem, can we please stop mixing problem spaces all the time? :) | 14:51 |
| dougszu | journalctl is pretty good if you take time to read the man page | 14:51 |
| jovial | journalctl has some nice filtering options such as log priority | 14:51 |
| SvenKieske | so we have three problems: a) using journald b) which location on the FS to log to c) which binary format to use (utmp is also a binary log file btw) | 14:52 |
| mnasiadka | We all understand that, but you know that proposal is like dropping Docker? | 14:52 |
| mmalchuk | filtering bad with multiline logs | 14:52 |
| dougszu | mmalchuk: there is no multi-line regex with this approach, it should fix some issues with that | 14:53 |
| SvenKieske | so which problem do we want to talk about? all at once? because you can of course configure journald to output plain text and the location is configurable, so this really has nothing to do | 14:53 |
| mnasiadka | I'm pretty sure there will be a lot of people disliking that approach | 14:53 |
| kevko | why to not provide user and option if he want to do a or b ? | 14:53 |
| dougszu | maintenance | 14:53 |
| dougszu | You could perhaps configure fluentd to write back out kolla logs from the journal | 14:54 |
| mnasiadka | So, we have 6 more minutes. | 14:54 |
| jovial | Don't we only care about giving the the services access to the journal socket. Where the logs are stored are up to how you configure the host OS. | 14:54 |
| mnasiadka | Unless dougszu can formulate the proposal in depth in a separate etherpad - we will need to discuss that at the PTG. | 14:54 |
| dougszu | PTG sounds good, thanks all | 14:55 |
| mnasiadka | I can't see an option we stop writing text files to /var/log/kolla/$service without a proper research, asking users on the ML and providing long deprecation phase | 14:55 |
| kevko | grep -ri error /var/log/kolla :D | 14:55 |
| SvenKieske | well the current state of affairs is at least very inconsistent, afaik, correct me if I'm wrong but: a) we have logs shipped with fluentd into opensearch b) we have(?) some local only logs in /var/log/{kolla}, c) we have stuff like docker logs which are not persistet anywhere afaik. d) I honestly don't know what podman does e) we have journald installed by default for stuff like kernel/systemd logs by | 14:55 |
| SvenKieske | default afaik anyway.. | 14:55 |
| mnasiadka | SvenKieske: PTG | 14:55 |
| mnasiadka | It's a very big change | 14:55 |
| mnasiadka | Unless we are going to support both modes | 14:55 |
| SvenKieske | it doesn't have to be. stop making things complicated. like, really! | 14:56 |
| dougszu | :D | 14:56 |
| jovial | Supporting both looks like it may be possible, right? | 14:56 |
| kevko | SvenKieske: yeah and only b is the place where you are sure where all logs are present :D | 14:56 |
| SvenKieske | s/syslog-ng/journald/ works, you know. if you do the proper config dance. | 14:56 |
| mmalchuk | mission impossible) | 14:56 |
| mnasiadka | Well, yes - but one of them won't have test coverage :) | 14:56 |
| jovial | Just use log_file and use_journal at the same time? | 14:56 |
| SvenKieske | kevko: not true, there aren't all logs in b) | 14:56 |
| kevko | SvenKieske: okay + docker logs | 14:57 |
| SvenKieske | but I agree it seems to be a PTG topic, or for some larger meeting at least | 14:57 |
| kevko | SvenKieske: fluent + openserarch works ...but until we will not drop parsing and regexp and will not use python fluent logger ...it's 75% working logging system | 14:57 |
| SvenKieske | the current logging state is a mess, to be honest. but still we need some careful planning to improve upon it and don't make it worse :) | 14:57 |
| halomiva | I want to ask, i started working on refactoring docker worker to not use low level client and instead use client similar with podman, i guess logical steps are to first merge this refactor and then try to put as many functions into container worker. What do you think about it? | 14:58 |
| SvenKieske | kevko: last time I looked there are no kernel logs in fluentd? :D it's a mess! ;) | 14:58 |
| mnasiadka | halomiva: I think we refactored Kolla to do the same, right? | 14:58 |
| SvenKieske | halomiva: sounds sane on the surface at least | 14:59 |
| mnasiadka | SvenKieske: nobody stops you to run syslog-ng and forward logs to fluentd | 14:59 |
| halomiva | mnasiadka: i think yes | 14:59 |
| mnasiadka | halomiva: so fine, I've seen the patch - is it ready for reviews? | 14:59 |
| SvenKieske | mnasiadka: right, and you can do the very same thing with journald, so that's not really a big thing, if you just talk about using journald (without all the binary log file blabla) | 14:59 |
| mnasiadka | ok, it's 16:00 | 15:00 |
| halomiva | waiting for tests to finish, i tested it locally on basic deployment and it worked so we will see after tests | 15:00 |
| mnasiadka | It's the first meeting in last 2 years when we did use the full hour | 15:00 |
| mnasiadka | Thanks for coming guys! | 15:00 |
| mnasiadka | #endmeeting | 15:00 |
| opendevmeet | Meeting ended Wed Feb 7 15:00:37 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-02-07-14.00.html | 15:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-02-07-14.00.txt | 15:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-02-07-14.00.log.html | 15:00 |
| mnasiadka | halomiva: good, I'll have a look :) | 15:00 |
| kevko | SvenKieske: kernel logs ? I meant openstack logging ... don't mix system logs | 15:00 |
| mmalchuk | mnasiadka gerrit dropped stable/yoga right now | 15:01 |
| SvenKieske | sorry, for extending the duration (I guess I'm partly to blame), but thanks for all the input | 15:01 |
| mnasiadka | mmalchuk: nice | 15:01 |
| mmalchuk | we will lost 2 commits | 15:01 |
| mnasiadka | mmalchuk: I'm sure jovial knows which ones need resubmitting | 15:01 |
| SvenKieske | kevko: today you want full traceability of all logs in a single system, of course you want system logs as well | 15:01 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Refactor of docker worker https://review.opendev.org/c/openstack/kolla-ansible/+/908295 | 15:01 |
| mmalchuk | mnasiadka jovial they both in detached state already) | 15:02 |
| jovial | I know priteau submitted one to test the status of the CI: https://review.opendev.org/c/openstack/kayobe/+/908281 | 15:03 |
| kevko | SvenKieske: agree, but kolla-ansible is focused to openstack deployment ... | 15:04 |
| mmalchuk | ok then, there are both from Pierre | 15:04 |
| kevko | SvenKieske: I just want to say that I agree that good logging of openstack stacks will be fine ....but this is huuuuge task | 15:04 |
| jovial | mmalchuk: Indeed, we may have to bundle a few up to get CI to pass | 15:06 |
| mmalchuk | i see | 15:07 |
| kevko | SvenKieske: some time ago i proposed elegant logging for openstack services via fluent logger ... gerrit XL change -> drop :D | 15:07 |
| jovial | We did ask the release team if there was anyway to move the commits across, but apparently resubmitting the changes is the only option | 15:08 |
| SvenKieske | okay, well then let's just take our time and get it right (or at least better). I'm sure we will find a good solution for everyone :) | 15:09 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Rework horizon role to support local_settings.d https://review.opendev.org/c/openstack/kolla-ansible/+/906347 | 15:13 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix horizon deployment https://review.opendev.org/c/openstack/kolla-ansible/+/908293 | 15:13 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Enable testing horizon https://review.opendev.org/c/openstack/kolla-ansible/+/907718 | 15:13 |
| mnasiadka | bbezak: cephadm jobs have some cinder upgrade problem - https://7892a49fff80be41bd93-7937b4b8835d06e87bcc77aa86f44280.ssl.cf5.rackcdn.com/908166/7/check/kolla-ansible-rocky9-upgrade-cephadm/99f369f/primary/logs/ansible/upgrade | 16:22 |
| mnasiadka | ubuntu fails on the same https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_095/908166/7/check/kolla-ansible-ubuntu-upgrade-cephadm/095ca6d/primary/logs/ansible/upgrade | 16:22 |
| spatel | I am running yoga release and now I want to add Gnocchi/ceilometer services. Can add zed release of Gnocchi & ceilometer? | 16:29 |
| spatel | Assuming it won't create any issue right? | 16:29 |
| spatel | We can mix release right? (ofc not for core services but atleast other services) | 16:30 |
| opendevreview | Merged openstack/kayobe master: reno: Update master for unmaintained/yoga https://review.opendev.org/c/openstack/kayobe/+/907819 | 20:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!