Thursday, 2023-09-21

mnasiadkamorning06:11
opendevreviewrenliang proposed openstack/kolla master: Fixed an issue with zun-cin-daemon building images in aarch64.  https://review.opendev.org/c/openstack/kolla/+/88572906:44
opendevreviewMichal Nasiadka proposed openstack/kolla master: Fix an issue with zun-cni-daemon building images in aarch64  https://review.opendev.org/c/openstack/kolla/+/88572906:45
SvenKieskemorning07:50
SvenKieskecongratz for being voted as PTL again, mnasiadka ;)07:53
mnasiadkathanks ;)07:59
hrwmnasiadka: gratki!07:59
mnasiadkaI forgot which cycle in a row is that ;)07:59
mnasiadkahrw: do we really need https://review.opendev.org/c/openstack/kolla/+/886086 ? it's failing on SHA-1, and the currently used key should be the new one, based on their website08:00
hrwlet me abandon08:06
kevkomorning \o/08:09
kevkomnasiadka, hrw : can we remove https://github.com/openstack/kolla/blob/3d703b60e6fb1062313b57293696f1c34151c27f/docker/macros.j2#L64   ulimit from this command ? podman failing on it ...this is the last issue 08:11
hrwkevko: 'git blame' pointed at me?08:12
kevkohrw: i've tried to use git blame ..and went through 4 5 patches ... :D but i didn't find the commit which proposed 08:12
hrwkevko: I would say 'go for it and check is someone complain'08:13
mnasiadkakevko: and second question is why ulimit fails on podman?08:15
mnasiadkadefault ulimit is lower?08:15
hrwpodman does not run as root08:16
mnasiadkawe use become for all container related tasks, so I guess it does08:16
kevkoI don't think there is a sudo kolla-build 08:31
kevkoIn Kolla ci08:31
mnasiadkawell, we didn't - because we used docker engine to do all the work08:37
mnasiadkanow podman is running the build as a user08:37
mnasiadkapodman build supports --ulimit, maybe the default one is too small08:39
mnasiadkaI'm not saying we really need that ulimit in td-agent-gem install08:39
mnasiadkabut removing it might bring some issues on docker side of things08:39
mnasiadkabecause docker default is 104857608:41
mnasiadkaso maybe the easiest approach is to set to 1024 - just like Neutron does (https://review.opendev.org/c/openstack/tripleo-heat-templates/+/559268)08:41
mnasiadkaand 1024 is the podman default08:41
kevkoso change 65536 -> 1024 ? 08:42
opendevreviewMichal Arbet proposed openstack/kolla master: Add podman for image building  https://review.opendev.org/c/openstack/kolla/+/82579109:01
opendevreviewMichal Arbet proposed openstack/kolla master: [CI] Add podman build  https://review.opendev.org/c/openstack/kolla/+/89581709:01
opendevreviewMichal Arbet proposed openstack/kolla master: Lower limit for open files when installing fluent gem  https://review.opendev.org/c/openstack/kolla/+/89600709:01
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Add option for extra kernel modules in neutron role  https://review.opendev.org/c/openstack/kolla-ansible/+/89473509:32
opendevreviewWill Szumski proposed openstack/kolla-ansible master: Support exposing prometheus_server externally  https://review.opendev.org/c/openstack/kolla-ansible/+/82869509:48
opendevreviewWill Szumski proposed openstack/kolla-ansible master: Support exposing prometheus_server externally  https://review.opendev.org/c/openstack/kolla-ansible/+/82869509:50
opendevreviewBartosz Bezak proposed openstack/kayobe master: CI: Use nested nodepools for jobs that are spawning vms  https://review.opendev.org/c/openstack/kayobe/+/87264709:55
opendevreviewMichal Nasiadka proposed openstack/kayobe-config master: kayobe-env: Unstick KOLLA_SOURCE_PATH and KOLLA_VENV_PATH  https://review.opendev.org/c/openstack/kayobe-config/+/89588810:01
opendevreviewMichal Nasiadka proposed openstack/kolla master: Drop openvswitch-netcontrold  https://review.opendev.org/c/openstack/kolla/+/89165510:35
kevkomnasiadka, hrw: Podman passing now , can u review ? 10:39
mnasiadkakevko: can you tell me why podman build is so slow? :)10:51
mnasiadka23m 02s docker vs 1h 16m 00s podman10:51
kevkohaha :D i don't know :D 10:56
kevkomnasiadka: Podman provides faster startup thanks to its daemon-less structure. At any point in time, the host system stays protected. However, when it comes to overall building, Docker is a winner. The building part relies primarily on image building. Since Docker has native image building, it can build at least six times faster than Podman.10:57
kevkomnasiadka: https://www.solutelabs.com/blog/podman-vs-docker#:~:text=The%20building%20part%20relies%20primarily,six%20times%20faster%20than%20Podman.10:57
mnasiadkashould we add that to docs, so users don't ask that?10:57
kevkomnasiadka: hmm, where it should go ? 10:59
mnasiadkahttps://docs.openstack.org/kolla/latest/admin/index.html#building-container-images - somewhere here?10:59
kevkoSvenKieske: do you want to help me with docs :D ? I hate it and i know you like docs :) 11:00
mnasiadkaprobably here https://docs.openstack.org/kolla/latest/admin/image-building.html#building-kolla-images - e.g. that since Bobcat we support building with Podman (Buildah) and you use it like that, and expect it being slow? :)11:00
opendevreviewMerged openstack/kayobe stable/zed: Limit interface names to 15 characters  https://review.opendev.org/c/openstack/kayobe/+/88454011:06
opendevreviewMark Goddard proposed openstack/kolla-ansible master: prometheus: Fix etcd scrape when etcd uses TLS  https://review.opendev.org/c/openstack/kolla-ansible/+/89605211:28
SvenKieskekevko: where exactly do you need assistance? be warned though that I hate restructured text syntax :D11:32
kevkoSvenKieske ^^ just a note that we supports podman and it's slow 😂11:35
SvenKieskeinteresting that it's slow; I skimmed the blog you linked and googled a bit around, no real conclusion though (I don't know what "native" in this context should mean, written in C?).11:36
SvenKieskeso that would be kolla docs, I suppose?11:37
SvenKieskeBut what should it contain, besides what is already covered here? https://review.opendev.org/c/openstack/kolla/+/825791/34/releasenotes/notes/add-docker-engine-7402d3fe7349f1f3.yaml Let me take a look..11:38
SvenKieskeshould we maybe add a note to the central README.rst that we have podman support? I don't know if it is worth it to explicitly mention docker|podman everywhere?11:41
opendevreviewVerification of a change to openstack/kayobe stable/yoga failed: Limit interface names to 15 characters  https://review.opendev.org/c/openstack/kayobe/+/88454111:41
mnasiadkaSvenKieske: not saying everywhere, just in https://docs.openstack.org/kolla/latest/admin/image-building.html#building-kolla-images - should be easy to do11:43
mnasiadkathis is how you use podman, by default we use docker, end of story11:43
opendevreviewSven Kieske proposed openstack/kolla master: [docs] fix docker best practice link  https://review.opendev.org/c/openstack/kolla/+/89605411:45
SvenKieskemnasiadka: ok11:45
SvenKieskehow do I build podman images? xD11:46
SvenKieskelet me look up the option name..11:46
SvenKieskemhm11:51
SvenKieskedo we really want to make users install the podman-py manually? why not add it to requirements.txt?11:51
mnasiadkait should be added11:52
mnasiadkasince we have docker-py there11:52
SvenKieskeyep11:52
mnasiadkabut it might be too late for this cycle11:52
mnasiadkaanybody want to pursue that with the requirements team?11:53
mnasiadka(it needs to be in requirements repo in upper-constraints.txt)11:53
SvenKieskemhm11:53
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Add option for extra kernel modules in neutron role  https://review.opendev.org/c/openstack/kolla-ansible/+/89473511:54
SvenKieskecan't it be just in requirements.txt in kolla repo?11:54
SvenKieskeI'm still a noob when it comes to know when something is in upper constraints..11:54
SvenKieskeI really don't understand this, it's not like every project is running in the same venv and thus must share the same deps?11:55
fricklerthey must all be co-installable, yes11:56
fricklernot about venv mainly, but distro pkgs11:56
fricklerbut why should kolla require both docker-py and podman-py if only one of them is needed?11:57
SvenKieskeah, the good old dpkg way11:57
kevkoHmm, podman and docker were in requirements .. but I think frickler suggested to remove ? 11:57
frickleryes, because having both installed at the same time makes no sense to me11:57
SvenKieskeif we don't want to be one of them to be second class citizen they must be?11:57
kevkoi don't think it's needed to have it in requirements ..as it is not hard requirement ...it's just optional 11:58
fricklerwe should drop both from being required, ack11:58
kevko+111:58
SvenKieskeok so second class citizen and support requests go to frickler and kevko :)11:58
SvenKieskeboth?11:58
SvenKieskecan kolla-build work at all without them?11:58
fricklerno11:58
fricklerbut that's the price of having a choice: you actually have to make one11:59
SvenKieskeso, I don't know... the file is called "_requirements_.txt" you know?11:59
SvenKieskedefaults must work11:59
frickleryou could add extra sections for both11:59
SvenKieskeyeah, and make the installation docs even longer to..save 5mb disk spaceß11:59
mnasiadkamakes sense for me, remove docker from requirements.txt12:00
mnasiadkaare the errors in kolla-build enough for users to know how to install docker/podman-py? :)12:00
kevkoit is i think 12:00
SvenKieskewhy do you want to make your product less userfriendly? why make it harder to setup for beginners?12:00
mnasiadkait's not a product, it's open source software12:01
SvenKieskecan someone please provide some justification that is not "well we could user either lib a or lib b, thus none is _technically_ required"=12:01
SvenKieskeyeah it might as well be closed source or deleted from the universe without users.12:01
mnasiadkawell, none is technically required, you can template out dockerfiles and use whatever you want to build them12:01
kevkoyou can fork and do whatewer you want :D 12:02
mnasiadkayes12:02
mnasiadkafine by me12:02
SvenKieskethe man who does not read has no advantage over the man who can't read - Mark Twain12:02
mnasiadkakolla-ansible doesn't have docker-py in requirements, you want to tell us it's also not user friendly?12:03
SvenKieskeuhm, when does kolla-ansible need to interact with docker-py?12:04
opendevreviewMaksim Malchuk proposed openstack/kolla master: Drop openvswitch-netcontrold  https://review.opendev.org/c/openstack/kolla/+/89165512:04
kevkokolla-docker module ? :D 12:04
SvenKieskeI'm not aware I installed that manually, like, ever?12:04
mnasiadkakolla_docker12:04
mnasiadkabecause ansible-collection-kolla baremetal role did install that for you when you run "kolla-ansible bootstrap-servers"12:05
SvenKieskeokay? so you agree it's good to not need to install it manually? or should we remove that as well?12:05
SvenKieskeI don't yet see any point being made, quite the contrary.12:05
mnasiadkasituation is the same, we deliver software without required dependency, and you need to go the extra mile of installing it12:05
mnasiadkaif removing docker from requirements.txt in kolla is seen as not being user friendly, then I probably need to stop being the PTL12:06
SvenKieskeas I said, I'm pretty sure I never did type "pip install docker-py" into any terminal, so I beg to differ that I installed something manually :)12:06
mnasiadkaI did that too many times with kolla-ansible, but whatever ;)12:07
mnasiadkaso it's either we install both, or none12:07
kevkoSvenKieske: do you think that people using kolla and kolla-ansible don't know how to install docker or podman library if they know how to build images ? 12:07
SvenKieskeI don't know, there are dozens of users who just install releases from pypi instead from, so there you are..12:08
SvenKieske*git12:08
mnasiadkawell, even if they get docker-py installed12:08
mnasiadkait doesn't mean they have Docker Engine installed12:08
mnasiadkaso it won't work12:08
kevkoindeed 12:08
mnasiadkaI'm pretty sure we have conscious users that know how to install a python package and Docker Engine12:09
SvenKieskeI'm also not claiming it's "hard" or "users won't understand it" just "it's not necessary" and annoying12:09
mnasiadkawell, if we want to install both (add them to requirements.txt) - then we need to add podman-py to openstack/requirements and get it managed by the requirements team12:09
mnasiadkaotherwise requirements-check CI job is going to fail12:09
SvenKieskethe ideal software installation should imho be a one liner like "apt install $foo" or "pip install $bar" and not a book full of instructions12:10
mnasiadkaunless we don't care and remove the requirements-check job12:10
kevko+1 ...so podman feature which is ready to merge will stay unmerged for next half year :D 12:10
mnasiadkayeah, let's wait 2 more years12:10
kevkobecause of one line in separate requirements repo 12:10
kevkoSvenKieske: haha, have you ever install openstack manually  ? :D 12:11
mnasiadkaand then we'll get a bug report, that podman-py is a useless dependency, because the user doesn't want that package :)12:11
SvenKieskethen delete that from the requirements.txt, but then also add all the extra steps needed to the docs, that's why I brought this topic up to begin with, currently, it's a nice oneliner to install kolla: "python3 -m pip install kolla"12:12
kevkoSvenKieske: you can propose dockerfile and add installation steps inside dockerfile ...then it will not be the pip but docker ...solved 12:13
mnasiadkaisn't that what we wanted to do before this discussion?12:13
mnasiadkaremove docker from requirements.txt and amend the docs?12:13
SvenKieskekevko: why would I fix things you break? do it yourself, ty.12:13
SvenKieskebefore this discussion nobody seemed to be aware that we either need instructions to install podman-py or include it in requirements.txt12:14
opendevreviewBartosz Bezak proposed openstack/kayobe master: CI: Use nested nodepools for jobs that are spawning vms  https://review.opendev.org/c/openstack/kayobe/+/87264712:14
SvenKieskethere was only an error logged in kolla-build if the lib is not there.12:14
mnasiadkaSvenKieske: I was aware, frickler was aware because we discussed that at least two times :)12:14
SvenKiesketechnically that's also sufficient? who needs docs anyway? openstack users now to read the sourcecode anyway12:14
kevkoSvenKieske: I didn't break anything ...I didn't touch requirements repo ... i've just invested my time to get podman patch merged ...and it's optional 12:15
SvenKieskebut you didn't raise this topics in the code review12:15
kevkoi am ok to not touch anything ...for me help is enough :D 12:15
SvenKieskewe could also get rid of the error log, it's technically not required :)12:15
SvenKieskeI'm sure we only want the smartest users12:15
mnasiadkaI love those discussions that don't lead to anything12:16
mnasiadkacan we focus on THE problem?12:16
SvenKieskeyes, they lead to me not patching the docs :)12:16
mnasiadkaah, that's your plan12:16
SvenKieskeyou said there is no problem? so discussion is over. I'm just replying because you keep typing stuff into keyboards.12:16
mnasiadkayou could have responded to kevko that you will not help him - case solved12:16
mnasiadkainstead of wasting bandwidth12:17
kevkookay, i will probably not touch podman from this moment .. tired fs rom thi12:17
mnasiadkalol12:17
mnasiadkaSvenKieske: see what you've done?12:17
SvenKieskeokay, let me restate what happened, just to correct it for historical purposes: I tried to help with the docs. I noticed that we have no requirements for podman-py, I asked what's to be done if it should be added to requirements.txt12:18
mnasiadkaand then frickler proposed we don't need to add it, we can remove docker - so the user can choose whatever he should install based on docs or the wind direction12:18
SvenKieskethis let to neither podman-py being added to requirements, nor to put it in docs, but to docker-py to also be removed, making more work for the docs.12:18
mnasiadkaI can do that work, if you're complaining on the amount of work that needs to be done - no problem12:20
SvenKieskeand I frankly don't agree with this, it's user unfriendly. it's technically strictly worse to have to type "pip install kolla podman-py" than to omit the last part.12:20
mnasiadkawe just need to agree what do we want to do :)12:20
mnasiadkawell, we shouldn't even direct users to do pip install kolla12:20
SvenKieskeyou also need to list in the docs now the lower and upper tested versions of podman-py and docker-py12:20
SvenKieskenow you lost me, how are users supposed to install kolla if not by pip?12:21
mnasiadkaI'm saying that by pip, but not use the version in pypi - because we're crappy at doing point releases12:21
mnasiadka(but that's another topic)12:22
SvenKieskethat's another can of worms I don't want to talk about now, can we maybe stick to one topic, that already got expanded to two topicsß12:22
mnasiadkaSo, I understand that SvenKieske opts for adding podman-py to requirements.txt12:23
SvenKieskeyeah, that way we don't even need to change the docs, at least for the installation part12:23
fricklerbut these are all interrelated. like for my downstream, we wrap all this in a container build, no issue at all12:23
mnasiadkakevko, frickler, bbezak - what do you choose - adding podman-py to requirements.txt or dropping docker from requirements.txt and asking users in docs to install docker-py or podman-py?12:23
SvenKieskeI'd even volunteer to provide a patch for u-c, if that is needed12:23
SvenKieskefrickler: sure, but we should not make upstream more complicated to sell our wrapper, no? ;)12:24
SvenKieskethe optimal solution would be if pip would understand optional dependencies, but I guess such a thing doesn't exist?12:25
mnasiadkawell, there's option number three - we install podman or docker based on what user wants to do using subprocess ;-)12:25
SvenKieskeah, that seems to exists, called "extras" in pip12:25
fricklerwell that's the extra sections I mentioned. you would "pip install kolla[podman]" or kolla[docker]12:25
mnasiadkaah, you mean pip install kolla[podman]?12:25
SvenKieskethat's actually nice, no? is that something that's accepted in openstack?12:26
mnasiadkawell, what happens if user does pip install kolla without extras?12:26
mnasiadkano docker and no podman? :)12:26
SvenKieskeborked kolla? :D but then you can point at docs? or install docker if it's omitted?12:27
fricklerthey'll get an error when they try to run it, telling them they have to make a choice12:27
SvenKieskeI mean docker is historically the default12:27
SvenKieskeI like the extras stuff, if that works and has no weird problems with u-c and pbr and stuff12:28
SvenKieskewhich I really don't know, not a pip expert.12:28
fricklerbut it was you who demanded equal treatment for podman12:28
fricklerI'd need to check the u-c situation for extras12:29
SvenKieskeyeah I still think it would not hurt to just install it. but I can do compromises, actually :)12:29
SvenKieskeand it's technically better to install something only if it's needed I get that, e.g. for security reasons.12:29
* SvenKieske did run gentoo for far too long12:30
mnasiadkaI still think an error message saying "Package podman-py not present, please install it by running: python -m pip install podman-py" would be sufficient for majority of the population12:30
SvenKieskefor kolla-build that might be true, but on the other hand, many people just run this stuff in CI pipelines, I guess they need to dig that out of the logs than, but I'm agreeing..somewhat12:31
SvenKieskeI don't want to waste your time anymore on this, whatever.12:32
mnasiadkaCI pipelines fail every week because some external dependency went crazy12:32
mnasiadkaand we're not backporting this approach, so I don't see a problem12:32
SvenKieskeyeah, let's add one more to the list ;)12:32
SvenKieskesure12:32
mnasiadkawell, I can see it, but it's easily fixable from a downstream CI perspective ;)12:33
SvenKieskeas I got the file already open I'll sketch a doc patch with rudimentary instructions12:33
mnasiadkathe extras might be nice, but we would need to drop docker from requirements.txt12:34
mnasiadkaso the effect is most probably the same12:34
mnasiadkajust we give a convenient "pip3 install kolla[docker]" or "pip3 install kolla[podman]"12:34
mnasiadkaor we go the requirements route and add podman-py there :D12:35
opendevreviewMerged openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks  https://review.opendev.org/c/openstack/kayobe/+/89313712:36
opendevreviewMerged openstack/kayobe master: Add python 3.10 to setup.cfg metadata  https://review.opendev.org/c/openstack/kayobe/+/88654612:36
SvenKieskeI'll just go with your original suggestion - for the docs - to add the instructions there.12:36
SvenKieskeimho requirements.txt changes should go into the kolla build patch12:36
opendevreviewMerged openstack/kayobe master: Remove firewall-debian role  https://review.opendev.org/c/openstack/kayobe/+/88385812:39
opendevreviewMerged openstack/kayobe-config-dev master: Rename disable_selinux_do_reboot  https://review.opendev.org/c/openstack/kayobe-config-dev/+/88336812:39
opendevreviewSven Kieske proposed openstack/kolla master: [docs] add instructions for optional dependencies  https://review.opendev.org/c/openstack/kolla/+/89608112:39
opendevreviewSven Kieske proposed openstack/kolla master: [docs] add instructions for optional dependencies  https://review.opendev.org/c/openstack/kolla/+/89608112:41
opendevreviewSven Kieske proposed openstack/kolla master: [docs] add instructions for optional dependencies  https://review.opendev.org/c/openstack/kolla/+/89608112:42
opendevreviewMerged openstack/kayobe master: Fix ipa_build_dib_elements_default comments  https://review.opendev.org/c/openstack/kayobe/+/89476512:58
opendevreviewMerged openstack/kayobe-config master: kayobe-env: Unstick KOLLA_SOURCE_PATH and KOLLA_VENV_PATH  https://review.opendev.org/c/openstack/kayobe-config/+/89588812:58
opendevreviewPierre Riteau proposed openstack/kayobe-config-dev stable/2023.1: Rename disable_selinux_do_reboot  https://review.opendev.org/c/openstack/kayobe-config-dev/+/89610513:32
opendevreviewPierre Riteau proposed openstack/kayobe-config-dev stable/zed: Rename disable_selinux_do_reboot  https://review.opendev.org/c/openstack/kayobe-config-dev/+/89610613:32
opendevreviewWill Szumski proposed openstack/kolla-ansible master: Support exposing prometheus_server externally  https://review.opendev.org/c/openstack/kolla-ansible/+/82869514:57
opendevreviewMerged openstack/kayobe stable/yoga: Limit interface names to 15 characters  https://review.opendev.org/c/openstack/kayobe/+/88454115:19
-opendevstatus- NOTICE: The lists.openinfra.dev and lists.starlingx.io sites will be offline briefly for migration to a new server15:31
SvenKieskemhm, would it be possible to drop "kolla-ansible-ubuntu-binary-upgrade" job for zed release? is that even still supported?15:45
SvenKieskeI assume it's testing binary releases?15:45
SvenKieskejust noticed because it failed here: https://review.opendev.org/c/openstack/kolla-ansible/+/89538015:46
SvenKieskeah, is that needed for yoga upgrade? because binary is not in zed support matrix: https://docs.openstack.org/kolla/zed/support_matrix15:47
SvenKieskeI never did run binary containers, how is the upgrade story there, going from binary to source release?15:48
SvenKieskeah I see, this _is_ the binary->source upgrade test: https://review.opendev.org/c/openstack/kolla-ansible/+/83904715:52
opendevreviewMerged openstack/kayobe-config-dev stable/2023.1: Rename disable_selinux_do_reboot  https://review.opendev.org/c/openstack/kayobe-config-dev/+/89610516:27
opendevreviewPierre Riteau proposed openstack/kayobe master: Fix data file path detection with new pip  https://review.opendev.org/c/openstack/kayobe/+/89483116:29
opendevreviewMerged openstack/kayobe-config-dev stable/zed: Rename disable_selinux_do_reboot  https://review.opendev.org/c/openstack/kayobe-config-dev/+/89610616:57
opendevreviewMaksim Malchuk proposed openstack/kayobe-config stable/2023.1: kayobe-env: Unstick KOLLA_SOURCE_PATH and KOLLA_VENV_PATH  https://review.opendev.org/c/openstack/kayobe-config/+/89611417:52
opendevreviewMaksim Malchuk proposed openstack/kayobe-config stable/zed: kayobe-env: Unstick KOLLA_SOURCE_PATH and KOLLA_VENV_PATH  https://review.opendev.org/c/openstack/kayobe-config/+/89611517:52
opendevreviewMaksim Malchuk proposed openstack/kayobe-config stable/yoga: kayobe-env: Unstick KOLLA_SOURCE_PATH and KOLLA_VENV_PATH  https://review.opendev.org/c/openstack/kayobe-config/+/89611617:53

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!