| opendevreview | OpenStack Proposal Bot proposed openstack/openstack-manuals master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-manuals/+/936215 | 03:14 |
|---|---|---|
| opendevreview | Merged openstack/openstack-manuals master: [www] Update static sitemap.xml https://review.opendev.org/c/openstack/openstack-manuals/+/931145 | 07:14 |
| opendevreview | Merged openstack/openstack-manuals master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-manuals/+/936215 | 07:18 |
| opendevreview | Merged openstack/openstack-manuals master: Add guideline for Permission denied (publickey) https://review.opendev.org/c/openstack/openstack-manuals/+/841577 | 07:18 |
| gouthamr | tc-members: a gentle reminder that we’re catching up here in ~40 minutes | 17:19 |
| clarkb | gouthamr: gmann JayF https://linuxfestnorthwest.org/ CFP closes January 10 | 17:25 |
| JayF | Bellingham is far enough for me for it to essentially be a nonlocal conference | 17:26 |
| JayF | I also will be taking a sabbatical this year and I'm unsure if it'll be started by then or not | 17:26 |
| gouthamr | nice thanks for sharing clarkb ! | 17:26 |
| clarkb | JayF: ya its even further for me. But its a fun small event | 17:28 |
| clarkb | similar to seagl | 17:28 |
| JayF | I didn't attend SeaGL this year and likely will never submit a talk there again | 17:28 |
| JayF | I did not have a good experience as a speaker; I was promised there was going to be quality recordings and remote participation (how I justified a small regional conference to $job) but the recording was completely unusable in a predictable way :/ | 17:29 |
| clarkb | I've only done seagl once and it was bring your own recording at the time. I'm partial to those small community run events because I think you get an audience that is really excited to be there (/me started working on openstack due to one of those events) | 17:48 |
| gmann | clarkb: thanks for sharing. I will check if i can make it. | 18:00 |
| gouthamr | #startmeeting tc | 18:01 |
| opendevmeet | Meeting started Tue Nov 26 18:01:04 2024 UTC and is due to finish in 60 minutes. The chair is gouthamr. Information about MeetBot at http://wiki.debian.org/MeetBot. | 18:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 18:01 |
| opendevmeet | The meeting name has been set to 'tc' | 18:01 |
| gouthamr | Welcome to the weekly meeting of the OpenStack Technical Committee. A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct. | 18:01 |
| gouthamr | Today's meeting agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee | 18:01 |
| gouthamr | #topic Roll Call | 18:01 |
| noonedeadpunk | o/ | 18:01 |
| frickler | \o | 18:01 |
| slaweq | o/ | 18:01 |
| gtema | o/ | 18:01 |
| gmann | o/ | 18:01 |
| gouthamr | noted absence: c a r d o e, b a u z a s | 18:01 |
| gouthamr | courtesy ping: spotz[m] | 18:03 |
| gouthamr | oh, spotz[m] may be away | 18:03 |
| gouthamr | lets get started | 18:04 |
| gouthamr | #topic Last week's AIs | 18:04 |
| gouthamr | there was one that we took note of: resolving the ownership of "watcher-drivers" on Launchpad.. | 18:05 |
| gouthamr | this has been addressed | 18:05 |
| gmann | yeah | 18:05 |
| gouthamr | thanks to LP admins, Billy Olsen.. and openstack-admins seems to have cleaned up everything for the team | 18:05 |
| gouthamr | i was tracking no other meeting AIs, did anyone else have any? | 18:06 |
| gmann | The next step is on watcher team side to add more members if needed and nothing specific for TC on this. | 18:06 |
| gouthamr | gmann: ++ | 18:06 |
| slaweq | gmann ++ | 18:06 |
| gouthamr | we'e a relatively short agenda today.. i.e., no new topics.. so would you folks want to round up on PTG notes? or would you prefer to do that next week over Zoom/IRC? | 18:07 |
| * frickler would not prefer zoom | 18:08 | |
| gouthamr | ^ yes; lets start today and see if we have a spillover | 18:08 |
| gouthamr | but, in the order of the topics there, I had: | 18:09 |
| gouthamr | #topic A check on gate health | 18:09 |
| gouthamr | we were talking about a number of UC bumps | 18:09 |
| gmann | I did not see any blocker or frequent failure this week. | 18:09 |
| gouthamr | it looks like we landed a lot more over this week | 18:10 |
| gouthamr | #link https://review.opendev.org/q/project:openstack/requirements+status:merged | 18:10 |
| gmann | I think those are settle down as projects are fixing the things. pillow bump one i remember is fixed multiple places | 18:10 |
| frickler | new keystoneauth still failing for horizon, something system_scope related? https://zuul.opendev.org/t/openstack/build/fabcf39cc611462488546bb9517b6266 | 18:10 |
| frickler | but a lot of other reqs bumps merged, yes | 18:10 |
| gmann | system scope should be disabled in horizon, my patch to enable it is still WIP | 18:11 |
| gmann | I will check it | 18:11 |
| frickler | https://etherpad.opendev.org/p/requirements-blockers is mostly up to date again | 18:11 |
| gouthamr | #link https://etherpad.opendev.org/p/requirements-blockers (OpenStack Global Requirements tracker) | 18:12 |
| * gouthamr wasn't aware of this tracker | 18:12 | |
| frickler | other than that, lots of jobs still have issues with the new dockerhub rate limits | 18:12 |
| gouthamr | thanks for sharing frickler | 18:12 |
| gouthamr | newer new rate limits? :| | 18:12 |
| frickler | new as of like 2 weeks ago? | 18:12 |
| clarkb | ya sometime mid november they seem to have changed | 18:13 |
| clarkb | they have a blog post about it but in my testing with their token system I wasn't able to get what their blog said in my token | 18:13 |
| clarkb | so it is very confusing. However jobs are definitely less happy about using docker hub and not using docker hub is probably a good recommendation at this point | 18:14 |
| frickler | sadly they seem to be the only source for plain debian/ubuntu images | 18:15 |
| frickler | (used as source for kolla image builds) | 18:15 |
| gouthamr | clarkb: this the post? https://www.docker.com/blog/checking-your-current-docker-pull-rate-limits-and-status/ | 18:16 |
| clarkb | there is work in zuul/zuul-jobs to add generic tooling to copy images from one registry to another and we could set up jobs to copy commonly used base images (like debian/ubuntu/mariadb/etc) | 18:16 |
| clarkb | gouthamr: thats the process for checking your rate limit according to the token values | 18:16 |
| frickler | clarkb: I'm not sure whether there could be legal issues with that? iiuc docker considers those their intellectual property | 18:17 |
| clarkb | https://www.docker.com/blog/november-2024-updated-plans-announcement/ is where they announce different lower rate limits that don't seem to be relfected in the tokens | 18:17 |
| * gouthamr doesn't know if we're opening another can of worms, but, | 18:18 | |
| gouthamr | is the recommendation to move to something like quay.io? | 18:18 |
| clarkb | frickler: maybe, we'd have to check for any images we do | 18:18 |
| clarkb | gouthamr: I think my recommendation at this point is avoid resources on dockerhub if you can. quay.io is one such alternative | 18:18 |
| clarkb | but github, google, etc have alternatives too that may or may not work | 18:18 |
| gouthamr | ack, and this is something that project teams have to individually opt-in to do? | 18:19 |
| fungi | well, they've individually opted into using dockerhub, right? | 18:19 |
| clarkb | its usually specific because projects indicate what image to pull which includes a registry name (or if not registry name is included the default is docker hub) | 18:19 |
| clarkb | fungi: yes it is explicit. Any opt out would be explicit | 18:19 |
| fungi | or merely un-opting-in | 18:20 |
| fungi | (i.e. choosing to use something else, rather than choosing to use dockerhub) | 18:20 |
| fungi | the challenge is that docker/moby built an entire ecosystem around assumptions that people could freely and conveniently download whatever they need from a central authority whenever they want, and now they've decided to upend that and invalidate those prior widespread assumptions throughout the container ecosystem | 18:21 |
| clarkb | https://github.com/docker-library/official-images/blob/master/LICENSE the source code for the official docker hub images is apache 2 licensed at least. Not sure what they will argue about the binary images themselves | 18:22 |
| fungi | so any solutions will necessarily involve some (or perhaps rather a lot of) pain to achieve | 18:22 |
| clarkb | "As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within." from https://hub.docker.com/_/debian | 18:24 |
| frickler | clarkb: yes, some time ago I looked into building images for our downstream ourselves, but it wasn't easy | 18:24 |
| clarkb | in any case we've attempted to mitigate the problem by disabling our caching proxy by default | 18:24 |
| frickler | "Use of Docker Official Images is subject to Docker's Terms of Service". on https://docs.docker.com/trusted-content/official-images/ | 18:25 |
| clarkb | since the caching proxy was a single IP it was far more likely to get rate limited vs spreading requests across many IPs in the CI system | 18:25 |
| clarkb | the rate limit errors still occur but at a lower rate I think since that change | 18:25 |
| clarkb | and every change to not use docker hub is at least one less request to docker hub and will further improve the situation | 18:26 |
| frickler | anyway I don't think we can solve this now and here, but the TC should know the status | 18:26 |
| fungi | (single ip per test node region anyway) | 18:26 |
| gouthamr | ++ if you folks have a recommendation to share, please do.. i can call this out as a concern so project teams that are explicitly opting into pulling from/uploading to dockerhub can notice and work on the recommended alternatives | 18:27 |
| gouthamr | #link https://docs.opendev.org/opendev/base-jobs/latest/docker-image.html | 18:28 |
| gouthamr | ^ this page for instance doesn't show a preference.. and i understand not writing one into the docs.. but, its possible some people would think we have a preference based on existing CI jobs/playbooks/roles that are easy to integrate with | 18:29 |
| clarkb | ya we're not prescribing where you host your images | 18:30 |
| clarkb | you can authenticate with docker hub and get better rate limits if you want | 18:30 |
| clarkb | you can use quay or github or whatever | 18:30 |
| fungi | though that document, for historical reasons, does mention docker hub rather a lot | 18:31 |
| clarkb | we just want peopel to be aware that the rate limits our not really under our control and you may or may not need to do something about it | 18:31 |
| clarkb | well it talks about docker images a lot not necessarily docker hub | 18:31 |
| clarkb | but if people want to make that less confusing via overloading of terms that fine | 18:31 |
| gouthamr | strive to eliminate something like that becoming a problem to test/ship your code.. | 18:32 |
| gouthamr | clarkb: with my employer-colored-hat on, i'd call them "container images" :D | 18:33 |
| clarkb | sure but for better or worse they've been called docker images for like a decade | 18:33 |
| fungi | (it does say quite a bit about docker hub in the publishing section) | 18:33 |
| gouthamr | true | 18:33 |
| clarkb | my priority right now isn't in sanitizing that document to make certain businesses happy about terms | 18:34 |
| clarkb | I've been focused on udnerstanding the udnerlying issues and adjusting job configs to alleviate the problem | 18:34 |
| gouthamr | ack, shouldn't be your concern.. | 18:34 |
| clarkb | then when things are better understood we can adjust the docuemntation if necessary | 18:34 |
| gouthamr | okay; thank you for raising the concern here, frickler and for seeding this discussion.. | 18:34 |
| gouthamr | any other gate concern to discuss here? | 18:35 |
| gouthamr | #topic PTG AIs and the TC Tracker | 18:36 |
| gouthamr | #link https://etherpad.opendev.org/p/tc-2025.1-tracker (Technical Committee activity tracker - 2025.1) | 18:36 |
| gouthamr | some things need an update on that tracker | 18:37 |
| gouthamr | #link https://etherpad.opendev.org/p/oct2024-ptg-os-tc-summary (OS TC Epoxy PTG Summary) | 18:37 |
| gmann | on Noble migration this is current status | 18:37 |
| gmann | #link https://etherpad.opendev.org/p/migrate-to-noble#L38 | 18:37 |
| gmann | Projects Green on Noble (already passing or changes mentioned below needs to be merged): 21 | 18:38 |
| gmann | Projects Failing on Noble: 18 | 18:38 |
| gmann | I send on ML but there are still many projects who have not ack/working on fixing the issue | 18:38 |
| gmann | I will ping them on IRC/add in their meeting agenda today if that can help | 18:39 |
| gouthamr | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/JOMDY26TCW7OX3NXRGOYQCIDXNNJ4E25/ ([all][tc][ptl][qa] Migrating upstream CI/CD jobs to Ubuntu Noble (24.04)) | 18:39 |
| gouthamr | thanks gmann.. we're at the original deadline in 3 days | 18:39 |
| gmann | yeah | 18:40 |
| gmann | that all from me on this | 18:40 |
| gouthamr | tkajinam brought up an eventlet related issue on the thread, and is discussing it further in #openstack-eventlet-removal | 18:41 |
| gouthamr | with these many projects having issues, i guess we can't meet that original deadline, gmann | 18:42 |
| gouthamr | not without overrides at least | 18:42 |
| gmann | yeah which is ok. otherwise these failure will stay until final release and that will be bigger problem | 18:43 |
| gouthamr | ack; would you be proposing these overrides? | 18:43 |
| gmann | switching it on Nov 29 gives a good pre-holidays heads up to projects | 18:43 |
| gmann | gouthamr: no, I will leave then failing and let project decide if they want to shift the migration for their job or fix them | 18:44 |
| gmann | overriding the nodeset has drawback of people might not even remove it so failing CI is good way | 18:44 |
| gouthamr | yes | 18:44 |
| frickler | I agree, if projects are not even able to pin their nodeset, we should reconsider their active status | 18:44 |
| gmann | and if project need more time then they can do but should be aware of what they are explicitly doing | 18:45 |
| gmann | frickler: ++ | 18:45 |
| gmann | gouthamr: I think we can move to next item | 18:48 |
| gouthamr | thank you | 18:48 |
| gouthamr | i was browsing through the PTG summary for topics that we don't have explicit AI owners for | 18:48 |
| gouthamr | i need to work on a couple of AIs | 18:49 |
| gouthamr | but there's one on the postgres discussion: | 18:49 |
| gouthamr | "Reviving postgres support will need volunteers. The TC last published a resolution in 2017 [3] explaining the state of support for postgresql. There must be a follow up to state that non-MySQL backends are not tested within the community." | 18:50 |
| gouthamr | i see that neutron dropped postgresql testing | 18:50 |
| gouthamr | other project teams may follow suite.. but this is a TC/doc issue; anyone wants to take a dig at this? | 18:51 |
| gmann | what is needed here 'a new resolution to remove the support' or documentation somewhere else? | 18:51 |
| gouthamr | #link https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.html (TC Resolution on the state of testing of database systems) | 18:52 |
| gouthamr | this was done as a resolution | 18:52 |
| spotz[m] | Sorry on PTO and got distracted | 18:53 |
| frickler | iiuc neutron was the last team to drop psql testing, at least from the core services | 18:53 |
| frickler | so I'm not sure what documentation you'd want to get updated? | 18:54 |
| gouthamr | np spotz[m].. hope the horse is well, | 18:54 |
| gouthamr | frickler: what's a "core" service | 18:54 |
| gmann | yeah, I think we are all good here and neutron did same as other services already did | 18:54 |
| spotz[m] | gouthamr: Doing much better thanks | 18:55 |
| frickler | gouthamr: essentially those installed by devstack itself without the need of a plugin? | 18:55 |
| gmann | gouthamr: that was the terminology used in past we do not need to clarify those in doc where ever ever it was used | 18:55 |
| frickler | but yes, the term is not exactly well defined | 18:55 |
| gmann | I think resolution still ok and service test other DB or not is up to them | 18:56 |
| gouthamr | okay; i can call this out to the ML again and round up any projects still running postgres jobs | 18:56 |
| gmann | if they do and want it is still ok right? | 18:56 |
| gmann | I mean we can say minimum things to test/support but max is something we should not limit | 18:57 |
| frickler | another can of worms will be opened when we see that we need to distinguish between mysql and mariadb. I think we mostly only test the latter these days | 18:57 |
| gouthamr | gmann: yes, i guess so.. iirc we came about this discussion because we had that stance.. and project teams thought they had to test it and were surprised when things started breaking because some other project teams weren't testing with postgresql | 18:58 |
| slaweq | frickler IIRC in neutron we are mostly testing with mysql and we have one periodic job with Maria DB | 18:58 |
| gouthamr | so we were reminding people of that old resolution, and someone suggested we need to reiterate it.. and claim we're NOT testing this (i don't know where we'd post this either) | 18:59 |
| gouthamr | oslo.db for instance supports it with no disclaimers | 18:59 |
| gouthamr | security guide suggests it as an equal alternative: https://docs.openstack.org/security-guide/databases/database-backend-considerations.html | 19:00 |
| gmann | if they support and test well then they do not any disclaimers right? | 19:00 |
| gouthamr | our install guide: https://docs.openstack.org/install-guide/environment-sql-database.html | 19:01 |
| gouthamr | "OpenStack services also support other SQL databases including PostgreSQL." | 19:01 |
| gouthamr | sorry; we're over time and i just noticed | 19:01 |
| gmann | I mean current resolution statements are still valid and there is no guarantee for non-MySQL support/test but if anyone test it is ok | 19:01 |
| gmann | Maria DB is good example frickler mentioned | 19:01 |
| gouthamr | but in essence, there's a bunch of documentation suggesting that postgres is well supported.. but, project teams aren't testing them anymore, so i think the AI was to help operators know that | 19:02 |
| gouthamr | i'll endmeeting and let you folks disperse to other places :) | 19:02 |
| gouthamr | thank you all for attending | 19:03 |
| slaweq | o/ | 19:03 |
| gouthamr | we'll pick up on this next week | 19:03 |
| gouthamr | hope to see you then | 19:03 |
| gouthamr | #endmeeting | 19:03 |
| opendevmeet | Meeting ended Tue Nov 26 19:03:31 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:03 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-26-18.01.html | 19:03 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-26-18.01.txt | 19:03 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-26-18.01.log.html | 19:03 |
| fungi | in case anyone is interested in taking on additional leadership responsibilities: a reminder that nominations for the 2025 open infrastructure foundation board of director individual member elections are currently open, closing at 23:59 utc on december 13: https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/thread/5R57DDIS7TARSUQGHL2Y5XBCEPNLGHVC/ | 19:03 |
| gouthamr | w00t... i should have made time for OpenDiscussion so we could have had that in the minutes | 19:04 |
| gouthamr | thanks for sharing that, fungi | 19:04 |
| fungi | yeah, i didn't want to interrupt other heated topics | 19:04 |
| JayF | Just adding: Ironic recently dropped postgresql testing, too | 20:21 |
| gouthamr | nice; thanks JayF | 20:21 |
| JayF | Also the security guide is so outdated as to be actively harmful IMHO | 20:21 |
| JayF | so it being documented there is less a comment on postgres support and more a comment on how dated that doc is | 20:22 |
| spotz[m] | JayF: What do you think about tagging it to say what versions? | 21:08 |
| JayF | Tagging what, exactly? | 21:09 |
| spotz[m] | Security Guide so it if't outdated since say Xena we make sure it's tagged visable so folks know it's outdted for later versions | 21:09 |
| JayF | IIRC fungi and I were talking about if it should just be retired at the security sig at the ptg | 21:10 |
| JayF | i'd have to check notes to be sure | 21:10 |
| JayF | eh, that sentiment didn't make it to the etherpad: https://etherpad.opendev.org/p/oct2024-ptg-os-security-sig | 21:11 |
| fungi | spotz[m]: there's a big yellow note at the very top of https://docs.openstack.org/security-guide/ which states "Important! This guide was last updated during the Train release..." | 22:01 |
| fungi | if my lettermath is correct, that's 10 releases or 5 years? | 22:02 |
| *** iurygregory__ is now known as iurygregory | 23:37 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!