Thursday, 2023-08-10

mnasiadkammalchuk: there's https://review.opendev.org/c/openstack/kolla-ansible/+/841239 for external vip, although in a different context05:43
SvenKieskemnasiadka: I'm preparing the UCA change for bobcat, looking at it: I guess someone else will push the EL9Stream Repos to bobcat?08:33
opendevreviewSven Kieske proposed openstack/kolla master: [release] Use UCA Bobcat  https://review.opendev.org/c/openstack/kolla/+/89101808:37
mnasiadkaSvenKieske: I'll handle the CentOS/Rocky part08:37
SvenKieskecool08:39
SvenKieskefrickler: mnasiadka: it would still be great if you could lend your +2 powers to these backports (all linked in the top right corner): https://review.opendev.org/c/openstack/kolla-ansible/+/89048308:50
SvenKieskeshould be fairly trivial (TM)08:51
mnasiadkaSvenKieske: took the liberty to merge them with only my superpower08:55
SvenKieskety!08:57
mnasiadkaSvenKieske: no bobcat yet in RDO, it's an empty repo at most, so let's wait09:03
opendevreviewMichal Nasiadka proposed openstack/kolla master: Revert "CentOS/Rocky: use CentOS Cloud SIG repo instead of Delorean"  https://review.opendev.org/c/openstack/kolla/+/89085109:04
mnasiadkabut let's maybe check out if master works09:05
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/2023.1: Enable nova libvirt driver skip_cpu_compare_on_dest workaround  https://review.opendev.org/c/openstack/kolla-ansible/+/89085209:50
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/zed: Enable nova libvirt driver skip_cpu_compare_on_dest workaround  https://review.opendev.org/c/openstack/kolla-ansible/+/89085309:50
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/yoga: Enable nova libvirt driver skip_cpu_compare_on_dest workaround  https://review.opendev.org/c/openstack/kolla-ansible/+/89085409:50
opendevreviewVerification of a change to openstack/kayobe master failed: Fix CentOS / Rocky route options  https://review.opendev.org/c/openstack/kayobe/+/88968009:52
opendevreviewlikui proposed openstack/kolla-ansible master: update ansible version  https://review.opendev.org/c/openstack/kolla-ansible/+/89102510:03
opendevreviewMerged openstack/kolla-ansible stable/2023.1: Fix OpenSearch Dashboards health check  https://review.opendev.org/c/openstack/kolla-ansible/+/89048310:06
opendevreviewMerged openstack/kolla-ansible stable/zed: Fix OpenSearch Dashboards health check  https://review.opendev.org/c/openstack/kolla-ansible/+/89048410:06
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Deny access to public /server-status in http Openstack services  https://review.opendev.org/c/openstack/kolla-ansible/+/88894310:07
opendevreviewMerged openstack/kolla-ansible stable/yoga: Fix OpenSearch Dashboards health check  https://review.opendev.org/c/openstack/kolla-ansible/+/89048510:07
kevkomnasiadka what about letsencrypt :) ? 10:20
mnasiadkaI'm planning to have a look at letsencrypt and podman tomorrow10:20
kevkoit would by nice if you will find some time today ..because tomorrow i have a vac :)10:21
mnasiadkaI'm still not convinced https://review.opendev.org/c/openstack/kolla-ansible/+/888943/14/releasenotes/notes/http-services-deny-server-status-39d0259664053e59.yaml should be fixes: instead of security: - either we want it to stay in the shadow of all fixes, or we want that to be visible for users in renos - ideas?10:21
kevkoit is fixes ...10:22
fricklerIMO since the bug is a security bug, it should be security:10:26
mnasiadkaso let's fix that, security: is for security fixes, fixes: is for bug fixes (not security)10:27
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Deny access to public /server-status in http Openstack services  https://review.opendev.org/c/openstack/kolla-ansible/+/88894310:28
mnasiadkadone10:28
mnasiadkait needs proper visibility10:28
janguttermnasiadka : I've got the  etcd 3.4 update email drafted, with [kolla-ansible][tooz] tagged, any other notables I should include?10:45
opendevreviewMerged openstack/kolla-ansible stable/2023.1: loadbalancer: Add option to not define track script  https://review.opendev.org/c/openstack/kolla-ansible/+/88706911:43
opendevreviewMerged openstack/kolla-ansible stable/zed: loadbalancer: Add option to not define track script  https://review.opendev.org/c/openstack/kolla-ansible/+/88717011:43
opendevreviewMerged openstack/kolla-ansible stable/2023.1: Trivial: Add deploy-containers for skyline  https://review.opendev.org/c/openstack/kolla-ansible/+/88980311:43
opendevreviewMerged openstack/kolla stable/2023.1: rabbitmq: Fix repo for ubuntu aarch64  https://review.opendev.org/c/openstack/kolla/+/88717711:53
SvenKieskeit would help if you guys would read all the comments, including mine: https://review.opendev.org/c/openstack/kolla-ansible/+/888943/comment/d01f1abf_39a12c21/11:58
SvenKieskeahyou already pushed it, thank you mnasiadka :+111:59
SvenKieskeit would also be great if nothing gets merged when there are unresolved discussions, that this is even possible is not good, isn't there a setting for such stuff in gerrit?12:00
kevkogit diff is best reno :)12:03
SvenKieskei violently disagree :)12:04
SvenKieskehttps://keepachangelog.com/en/1.0.0/12:05
SvenKieskequote: "Don’t let your friends dump git logs into changelogs."12:05
kevkoSvenKieske should you have the courage to let kolla-ansible upgrade to production without a test?12:06
SvenKieskeassuming that all people needing our changelogs are even familiar with git is wrong, also many commit messages don't really do a great service of informing the end user and end up being techno speak12:06
SvenKieskekevko: I don't know what you're referring to? what has upgrading to do with anything being said?12:07
kevkopeople reading release notes right before they are going to upgrade some component ...12:08
SvenKieskeIf you're thinking in the line of "in order to upgrade you need to study the git log anyway" -> you are right, but only because openstack relnotes often miss important stuff, which means they are suboptimal and should be improved12:08
SvenKieskeI read relnotes before I upgrade the test env?12:08
kevkobut i am also testing always .... because i don't remember when everything was included as reno in kolla ..almost never :D 12:09
SvenKieskeso you assume your testing catches everything that's mentioned in relnotes and git log? lulz12:09
kevkowhat i know that i can run dry run ...and i can see what is going to be changed ..what is going to be restarted .... etc ..12:10
SvenKieskeI do: read relnotes, read git (because most relnotes sadly suck), do testing deployment of new env, do upgrade test, let users test both deployment, after deploying to staging, wait some time, then deploy to prod12:10
SvenKieskedry run does not work, like at all, especially with openstack12:11
jangutterheh, the simplest way is just to define all bugs as features. those exceptions in the log are meant to be there.12:11
SvenKieskeand that procedure still doesn't catch all errors in advance12:11
jangutterrelying on only one method to catch and foresee problems is doomed to fail until we invent a perfect crystal ball.12:12
kevkoso, you are reading renos as first ... i am sure that you will realize that "fixes" from reno is "security fix" ...because you are also reading git log and checking the code ..12:13
SvenKieskeeven simple ansible playbooks do not execute meaningfully when using dry run, a former colleague tried to implement dry-run with k-a and found out that barely anything get's triggered this way, it's no meaningful test. I don't remember the details, but afaik we have to little asserts and tests included for dry-run to do meaningful testing with it, iirc.12:13
SvenKieskekevko: yes I do, I mean, I'm a contributor and do this stuff for some years. But I know there are inexperienced users who will glance over "fixes" and not realize that it's security related12:13
SvenKieskeI mean, just read openstack-discuss daily and you see large groups of beginner users who are basically clueless.12:14
SvenKieskewhich is okay.12:15
SvenKieskea security fix should be named a security fix, I'm no fan of the linux kernel policy to sweep such stuff under the rug12:15
kevkowhat I am trying to say is  that sometimes some important patch is waiting for weeks because of stupid (maybe so strong word - sorry :P ) reno or bad english or whatever ... but bug still present in upstream code :D 12:15
SvenKieskebecause the blackhats surely DO read relnotes and git logs and will find security issues and abuse them for bad stuff.12:15
SvenKieskekevko: yeah sure, time to release is always a bad tradeoff.. on the other hand: users that want the fix and are informed can already pull non merged stuff if it's only not merged because of relnotes12:16
kevkoMoreover, if it is security patch ...it should be merged as soon as possible ...then in another patch reno can be proposed 12:17
SvenKieskeif we had more people working on openstack I guess everything would be better ;)12:17
SvenKieskekevko: I think I also like that approach, but that approach works better if you have a higher, so called, "velocity" i.e. time to merge between patches is short.12:18
jangutterkevko: security is a nightmare - the clock is ticking and rushing things makes it much more likely for a broken fix to be merged.12:18
kevkojangutter: we are not talking about fix implementation ...but about release note ...12:19
SvenKieskeopenstack velocity is basically zero point 1 for most stuff, many people have their own feelings and objections to each patch. I still think the end result is a net positive :)12:19
jangutterkevko: I consider the release note part of the implementation...12:19
SvenKieskesee, 2 people, 3 opinions :D12:20
jangutterhehe12:20
kevkojangutter: okay - no problem, but release note is part of the release ...  stable/* or master is not release 12:21
SvenKieskeon the one hand i totally agree: changes without docs are useless (and relnotes are docs), on the other hand, well, we need the code changes at least. code changes without docs beat no code changes (most of the times)12:21
SvenKieskekevko: yeah maybe we need some sort of unstable/integration branch before going onto a release branch. "problem" is, we want to have master green in CI so CI works12:22
kevkoSvenKieske: consider this simple trivial patch 12:22
kevkohttps://review.opendev.org/c/openstack/kolla-ansible/+/88873812:22
kevkorelease note  ? closes-bug ....etc etc ? waste of  the time ....12:22
kevkoand still ....it's there for a month :D 12:22
SvenKieskeit's not a waste of time. without docs, there's a chance no user ever know's this new feature exists. a feature that is not used might as well not be implemented :)12:23
kevkoit's not new feature ...it's just a bug 12:23
SvenKieskeI literally searched a whole week to find out if virtio-blk supports discard, there's much documentation available which says it doesn't work, but it does (except for rhel). if that would've been documented that would have saved me days12:24
SvenKieskeso I'm coming from the ops/deployment side into the openstack dev side and I have to say: the thing in openstack which sucks the most is docs!12:25
jangutterI wonder if this case is particularly painful because it triggers a fundamental difference in opinion. (useful features should be enabled by default vs. useful features should be disabled by default)12:25
SvenKieskein theory you can do many things, just to find out the docs are wrong, outdated or there are easier not documented solutions.12:25
SvenKieskethat's the highest barrier to openstack adoption imho12:25
SvenKieskeyou need to waste 1-2 years to get familiar with openstack before really being able to run it, and that's mostly because of bad docs and the close second factor is the abysmal website. when there are docs you can't find them. if you google, you only find outdated pike docs :D12:26
jangutterYeah, it's very difficult to figure out how to operate openstack without having operated it before.12:27
SvenKieskeI'm even using chatgpt to let it explain openstack features/docs, which works better than openstack.org and google, of course I still need to double check for hallucination, but it's still faster.12:27
SvenKieskethat I have to do this after roughly 3 years is insane.12:27
* SvenKieske is done ranting for today, hopefully.12:28
SvenKieskefair point: at least the openstack docs are consistent, ansible documentation is still way worse, consistency wise (try to search for docs for a certain release, than clicking through the docs, only to realize that you're suddenly reading "latest" docs instead of 2.x)12:30
kevkoSvenKieske when i came from ops/dev side into the openstack dev side i also tried to read documentation .... then i just realized that best documentation is code :)12:31
SvenKieskeso I still think openstack does a good job, as someone has put it some years ago: 'openstack asks questions where other "solutions" didn't even grasp that there is a problem.'12:31
SvenKieskekevko: yes of course, me too. but I still think this is bad UX; it's nice to be able to read the code if I need it. But I should not be forced to read it. especially some parts of openstack are really..wild (coding practices wise).12:32
SvenKieskeI'm really cynical about it to the point that every time I read something in the docs I'm going "this might be all outdated and wrong, I should directly dive into the source code and check if this really works this way"12:33
SvenKieskethe dozens of abstraction layers inside openstack do also not help in grasping all the code/moving parts and how they fit together :D12:34
kevkolast 500 commits -> 12:34
kevkoGit commits without reno: 24812:34
kevkoGit commits with reno: 4212:34
SvenKieskewe are still living in the dark age of information technology. in 50 years people will hopefully look back like we do now and be baffled at our bad practices :D12:35
jangutterSvenKieske: but that's how I work with almost all open source. Unless I have a running system _doing_ the thing I want to do, there's still a reasonable chance that there's something missing.12:36
SvenKieskewe should just strive to have better practices imho. it's not always possible or enforceable, but we should at least try.12:36
SvenKieskeand never justify bad stuff with "well we always did it this way" or "but the project over there does the same bad thing"12:37
SvenKieskeI like page 10 of this famous kernel security talk: http://kernsec.org/files/lss2015/giant-bags-of-mostly-water.pdf12:38
SvenKieskethe whole thing is gold :)12:38
guesswhat[m]openstack will die in the near future anyway, there are projects like kubevirt, cloud-hypervisor where all the "toil" is abstracted with kubernetes and its ecosystem12:40
jangutterhave you ever read https://danluu.com/nothing-works/ ? at least the problem isn't isolated.12:40
SvenKieskeyeah, big danluu "fan" here12:40
SvenKieskeall k8s on baremetal stuff still uses ironic for baremetal under the hood last time I looked12:40
SvenKieskeso it's really just incorporating openstack, not supplementing with something else12:41
SvenKieske"We still design IT infrastructure like we designed cars in the 1960s."12:41
jangutterheh, I have to duck out of this conversation now or I'd say something stupid like "folks who believe k8s will take over openstack don't understand either"12:42
SvenKieskeand k8s development is the same shitshow like everything else, maybe with slightly better docs, but most commits also get no docs there.12:42
guesswhat[m]really? no..12:43
SvenKieskeand the dependency tree is even more kafkaesk than openstack, try to compile it yourself and cry. ;) 12:43
kevkoguesswhat[m]: this argument used my colleguage 6 years ago ...and we started to deliver k8s ...we had 2 clients for k8s and do you know what ? from that time ..no bigger k8s customer but openstack ? yeah ...several ..12:43
guesswhat[m]virtualization is still not matured tho12:44
guesswhat[m]but there are projects like kata containers 12:44
SvenKieskek8s has a nice api, and you can decently run stateless workloads, stateful requires quite some work. and if you're not on a public cloud you need a whole team of SREs just to run the thing. just like openstack12:45
guesswhat[m]the ecosystem is giant https://landscape.cncf.io/, there is such a traction ... , the only problem is the lack of engineers, but it will change in the future 12:46
SvenKieskeit's the same like when openstack came out, gardener hype cycle and all that.12:47
kevkokata containers are just vms with looks like containers :D no ? 12:47
guesswhat[m]there is vm isolation with capabilities of containers12:48
kevkoyeah, but there is again qemu :D 12:49
kevkofull virtualization ..12:49
guesswhat[m]networking, auditing, deployment, volumes, all that jazz out of the box12:49
guesswhat[m]wrong12:49
guesswhat[m]cloud-hypervisor12:49
kevkovirtualization != paravirtualization 12:50
guesswhat[m]openstack is sinking 18th centrury ship12:50
SvenKieskemaybe, maybe not. predictions are hard, especially if they involve the future.12:51
SvenKieskethere's tons of legacy industry who's barely virtualized and which can't and won't run in containers, and never in public clouds. half the cncf landscape only works in aws, azure or gcp.12:52
kevkowe might as well be the past ... AI will replace us :)12:52
guesswhat[m]clouds are here, old sysadmins are not anymore, no one wants to maintain own hw and software, every major cloud is compliant in terms of legal12:52
SvenKieskethe other half says in the docs "do not use just yet in production, to start a dev environment you can deploy this to GKE via..."12:52
kevkono one ? and what about companies which can't run in public cloud for example ? sensitive user data ? 12:53
kevkobanks for example ? 12:53
guesswhat[m]I dont even know if Openstack is ARM capable.., but ARMs are future12:53
guesswhat[m]We builded the bank in Azure, its not a problem anymore12:53
SvenKieskeopenstack runs on ARM and I'd argue RISC-V is the future and ARM is already a dead man walking but doesn't know it just yet.12:53
SvenKieskethere are banks in public clouds and banks who won't run in public clouds :)12:54
jangutter... and apparently for some public clouds ... cough ... finding banking credentials isn't too difficult according to recent leaks.12:56
guesswhat[m]the problem is elsewhere, there is a lack of experienced engineers in general, and its easy for everyone in the cloud build some infrastructure, tons of premaded blueprints, not possible for openstack .. and a lot of inexperienced engineers are creating a lot of infrastructure and ... cough ... everyone knows whats happening after13:00
guesswhat[m]azure,aws,gcp is literally .. moving knobs and sliders :D13:00
guesswhat[m]luckily they have always a paid service for auditing and remediation :D13:01
SvenKieskeguesswhat[m]: don't know your part of the world but I think you're missing huge industries where public cloud is just a no go, e.g. see european self sovereign stuff like this: https://digital-strategy.ec.europa.eu/en/policies/strategy-data13:02
SvenKieskethat's what I'm paid to do, actually, see: https://scs.community/index.html13:03
SvenKieskek8s public cloud might be larger, but there are still multi billion business and gov stuff that's never gonna go in a public cloud, especially military stuff, but also many other use cases13:04
guesswhat[m]it does not imply anything, every major cloud has government hosting solution, even with totally offloaded hw ...13:06
guesswhat[m]are You telling me that every non cloud company / government is auditing its own hosted solution ? 13:07
guesswhat[m]I am from EU and the government is not able to pay for experienced engineers, but enterprise companies are 13:08
jangutterbah, it all went bad when we migrated away from mainframes.13:15
SvenKieskecloud act in the US requires every US company to release customer data on a secret court order, without recurse or even knowledge of the customer, this includes spin offs of us companies. this is against current eu law. but even some countries in the EU try to ignore it because it's convenient.13:18
jangutterI remember in 2000-ish when Java was going to replace everything. There were even folks explaining that the whole OS is going to be written in Java.13:18
SvenKieskeyou may want to look up "schrems II" at the european court, the new "agreement" between EU and US will very likely be shot down by judges again.13:19
jangutterThere's always going to be something new that some advocates are pushing because their paycheck depends on it.13:20
guesswhat[m]jangutter +113:21
SvenKieskeit's also known as "cv driven development" ;)13:21
jangutterI'm stealing that.13:23
guesswhat[m]back to the reality,... openstack is still broken, currently listing jobs here, no one is hiring openstack engineer :D13:24
guesswhat[m]but there are plenty of aws,gcp,azure a kubernetes openings..13:24
guesswhat[m]why would i choose the openstack path then?13:24
guesswhat[m]I need to eat a provide food for my family 13:25
guesswhat[m]*and13:25
guesswhat[m]"It`s only logical"13:26
SvenKieskewell I can debug the complete openstack stack, in theory, down to baremetal, I have yet to see a production ready baremetal k8s deployment which is completely opensource, run by a foundation, not run by a single company13:27
SvenKieskeand openstack jobs exists, just not on openstack.org website, most of the time labeled as "cloud engineer", "devops" or "SRE"13:27
SvenKieskek8s is nice, sure, but most of it runs on proprietory infrastructure, which is sad13:28
SvenKieskewill be improved in time, I'm sure, the next hype topic seems to bei AI :)13:29
mnasiadkajangutter: it's ok, fire off :)13:34
jangutterhttps://lists.openstack.org/pipermail/openstack-discuss/2023-August/034695.html13:42
opendevreviewMerged openstack/kolla-ansible master: Deny access to public /server-status in http Openstack services  https://review.opendev.org/c/openstack/kolla-ansible/+/88894313:52
opendevreviewMichal Arbet proposed openstack/kolla-ansible stable/2023.1: Deny access to public /server-status in http Openstack services  https://review.opendev.org/c/openstack/kolla-ansible/+/89085713:59
opendevreviewMichal Arbet proposed openstack/kolla-ansible stable/zed: Deny access to public /server-status in http Openstack services  https://review.opendev.org/c/openstack/kolla-ansible/+/89085813:59
opendevreviewMichal Arbet proposed openstack/kolla-ansible stable/yoga: Deny access to public /server-status in http Openstack services  https://review.opendev.org/c/openstack/kolla-ansible/+/89085914:00
opendevreviewMichal Arbet proposed openstack/kolla-ansible stable/xena: Deny access to public /server-status in http Openstack services  https://review.opendev.org/c/openstack/kolla-ansible/+/89086014:00
opendevreviewMichal Arbet proposed openstack/kolla-ansible stable/yoga: Deny access to public /server-status in http Openstack services  https://review.opendev.org/c/openstack/kolla-ansible/+/89085914:04
opendevreviewMichal Arbet proposed openstack/kolla-ansible stable/xena: Deny access to public /server-status in http Openstack services  https://review.opendev.org/c/openstack/kolla-ansible/+/89086014:06
kevkoSvenkieske here ? 14:58
kevkoSvenkieske do you think it is good idea to do things in different way as it is already implemented in different Kolla images ? 14:58
SvenKieskepuh, depends?15:03
SvenKieskecan we discuss this tomorrow? also a concrete example would be nice15:04
SvenKieskein general I would say a different implementation can be good/justified if it's "better" in some way. we don't need to cargo cult old behaviour just for the sake of everything looking the same imho.15:04
SvenKieskeit should be significant better though, and that's of course subjective what "significant" means :D15:05
kevkoI have no problem to remove ifs and just set owners etc etc ... 15:16
kevkoAnd I will provide huge patch to also change on other places 15:17
opendevreviewJan Gutter proposed openstack/kolla-ansible master: CI: Add back ARA logging  https://review.opendev.org/c/openstack/kolla-ansible/+/89109718:35
opendevreviewJan Gutter proposed openstack/kolla-ansible master: CI: Add back ARA logging  https://review.opendev.org/c/openstack/kolla-ansible/+/89109718:49
opendevreviewMaksim Malchuk proposed openstack/kolla master: Add server-status handler to Rocky/Centos Apache conf  https://review.opendev.org/c/openstack/kolla/+/89109819:10
mmalchukfolks, Im ready for a new discussion^^^19:10
mmalchukmnasiadka kevko 19:10
opendevreviewJan Gutter proposed openstack/kolla-ansible master: CI: Add back ARA logging  https://review.opendev.org/c/openstack/kolla-ansible/+/89109719:51

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!