Tuesday, 2022-10-18

opendevreviewBoxiang Zhu proposed openstack/kolla-ansible master: [CI] Add skyline scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/86168702:43
opendevreviewBoxiang Zhu proposed openstack/kolla-ansible master: [CI] Add skyline scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/86168705:19
shyambHi05:26
shyambDoes Kolla yoga supports binary and source installation modes? or just source?05:26
hrwshyamb: yoga supports binary but it is deprecated there already05:49
shyambhrw: Okay.thanks. Is this deprecation started from Xena release?05:51
shyambAny rough plan of when we are planning to stop complete binary support in future?05:52
shyambYoga release notes does not have this information: https://docs.openstack.org/releasenotes/kolla-ansible/yoga.html05:54
hrwshyamb: kolla does: https://docs.openstack.org/releasenotes/kolla/yoga.html#prelude05:57
hrwshyamb: Zed has only one type of images05:58
hrwYoga will be supported for some time05:58
shyambhrw: Okay, thank you. 06:00
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm - use download.ceph.com packages for Ubuntu and Rocky  https://review.opendev.org/c/openstack/kolla-ansible/+/86170106:07
mnasiadkamorning06:10
shyambhrw: In source images, can we install rpm, debian packages or only pip packages are supported?07:21
shyambWe are building a container for our service - triliovault07:21
shyambwe have only rpm and debian packages07:21
hrwshyamb: binary/source describes where from we take openstack components. and only that07:31
hrwshyamb: mariadb container image is same in both cases07:31
shyambhrw: Okay07:31
hrwas at one point in time I moved few vars from base image into macro to make our images more 'the same'07:32
hrwshyamb: there was a time when we planned to add 3rd type - 'infra'07:33
hrwshyamb: I wrote stuff for it but we never enabled that by default07:33
hrwshyamb: so you would get debian-infra-mariadb + debian-source-nova-compute + debian-infra-nova-libvirt 07:33
shyambhrw: What would be our recommendation for third party applications running on kolla openstack? Should they also move their images to 'source' format?07:34
hrwshyamb: there is no binary/source anymore07:34
shyambhrw: Okay07:34
shyambI am from TrilioVault, we have only debian and rpm packages07:35
hrwshyamb: if your app is not openstack component then install as you like. if it is then show where source is07:35
hrw"TrilioVault is the only OpenStack-native data backup and recovery solution that gives cloud and backup administrators the ability to restore entire workloads in one click." one?07:35
shyambyes07:36
shyambhrw: So, I wanted to check if we can keep rpm, debian based container images instead of moving to pip format and virtual env07:36
hrwyou can07:36
shyambhrw: Okay07:37
hrwshyamb: I see that this is comercial product, right?07:38
hrw'free trial' gives 30 days07:38
hrw'free basic edition' is 16 nodes or 160 vcpus07:38
hrwhm. those are for k8s edition07:39
hrw'triliovault for openstack' only has 'request a demo'07:40
shyambhrw: Hi, I will check about licensing and will let you know07:40
shyambThere should be some free trial license.07:40
shyambhrw: Yes, it's commercial product.07:41
hrwI will leave the decision to mnasiadka as PTL then.07:41
mnasiadkaincluding a commercial licensed product in Kolla upstream open source repositories? I don't think that is possible07:43
hrwcause now we do not have images with 'binary only' software. even elasticsearch, with their weird licensing etc, has sources available07:43
mnasiadkaBut I remember at some point in time we wanted to enable users to build ,,bring your own container images'' - maybe that's a solution to that problem07:43
hrwwe have --docker-dir option for adding external ones07:44
shyambhrw: mnasiadka: We have not committed our code in upstream repos. We keep all our repos separate07:46
mnasiadkaso again - source images mean we install OpenStack projects code from source, nothing else - all the other services are installed from rpm/deb07:51
mnasiadkaas Marcin mentioned07:52
shyambmnasiadka: Okay, thanks.08:47
amorinhello kolla team, is there any reason not starting neutron as a wsgi process behind apache2, like nova?08:51
amorinhttps://github.com/openstack/kolla/blob/master/docker/neutron/neutron-server/Dockerfile.j208:51
mnasiadkahttps://review.opendev.org/c/openstack/kolla/+/74960109:11
mnasiadkasee last comment09:11
mnasiadkaI think nobody tried again since 202009:11
amorinack09:12
amorinthanks09:12
amorinwsgi was introduced in 2019 AFAIKS on neutron side09:12
amorinwe are facing some issue on our deployment using neutron in wsgi mod  as well (not using kolla), so I was wondering if the community solved the issue09:13
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add max_replication_lag to proxysql template  https://review.opendev.org/c/openstack/kolla-ansible/+/86172409:20
kevkoamorin: what is the issue ...because i wanted to reimplement neutron in kolla-ansible 09:22
amorinso,09:23
kevkobecause I think neutron should have neutron-rpc-server and neutron-api containers 09:23
amorinwhen using wsgi, you have a choice to disable lazy loading of wsgi09:23
kevkoand i wanted to implement neutron-api as other wsgi services in kolla-ansible 09:23
amorinwhen we disable lazy loading, some neutron rpc messages sent by api are lost09:23
amorinI am not 100% sure that upstream is affected, we are running a custom stein deployment09:23
kevkohmm09:31
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add heartbeat config option to rabbitmq.conf  https://review.opendev.org/c/openstack/kolla-ansible/+/86172709:37
opendevreviewBoxiang Zhu proposed openstack/kolla-ansible master: Add skyline service  https://review.opendev.org/c/openstack/kolla-ansible/+/82846409:48
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add max_message_size to rabbitmq config  https://review.opendev.org/c/openstack/kolla-ansible/+/86172809:57
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add heartbeat config option to rabbitmq.conf  https://review.opendev.org/c/openstack/kolla-ansible/+/86172709:59
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add heartbeat config option to rabbitmq.conf  https://review.opendev.org/c/openstack/kolla-ansible/+/86172710:01
opendevreviewBoxiang Zhu proposed openstack/kolla-ansible master: [CI] Add skyline scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/86168710:01
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add heartbeat config option to rabbitmq.conf  https://review.opendev.org/c/openstack/kolla-ansible/+/86172710:03
opendevreviewBoxiang Zhu proposed openstack/kolla-ansible master: [CI] Add skyline scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/86168710:06
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add max_replication_lag to proxysql template  https://review.opendev.org/c/openstack/kolla-ansible/+/86172410:08
kevkomnasiadka, yoctozepto >> https://review.opendev.org/c/openstack/kolla-ansible/+/861724 , https://review.opendev.org/c/openstack/kolla-ansible/+/861727, https://review.opendev.org/c/openstack/kolla-ansible/+/861728 ...can you then check after zuul will pass ? 10:15
kevkoit's only some config options 10:15
kevkorabbitmq config tuning - heartbeat configuration which we needed to modify, and max_message_size ...(we found that some messages from nova or neutron were very big and rabbitmq was dropping this)10:16
opendevreviewBoxiang Zhu proposed openstack/kolla-ansible master: [CI] Add skyline scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/86168710:17
kevkothese options are set to default rabbitmq values of course 10:17
kevkoproxysql - max_replication_lag ...because default value is 0 ..which is sometimes not what you want ..10:18
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: openvswitch: switch bridge setup to Ansible modules  https://review.opendev.org/c/openstack/kolla-ansible/+/84346011:07
hrwamorin: stein is a bit old11:19
frickler.oO(straight from the stone age)11:21
mnasiadkaStein? From the stone age? come on!11:21
fricklerthat's literally what it means11:22
frickleralso for reference https://bugs.launchpad.net/neutron/+bug/1912359 was mentioned in #-neutron regarding OVN+wsgi11:22
fricklerso no progress yet11:22
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Fix post-config of OVS for manila-share servers  https://review.opendev.org/c/openstack/kolla-ansible/+/82470711:30
mnasiadkaI started abandoning some really old patches that have been already covered or not applicable anymore.11:31
mnasiadkaI'll prepare a list of maybe-relevant for next weeks meeting, so we can decide what to do with them.11:31
mnasiadka(kolla-ansible only)11:31
fricklersounds great, thx mnasiadka 11:34
mnasiadkakevko: with your patches I have a bit of an issue - what happens if rabbitmq/proxysql changes the default? do we need to track it? it would be better to put it in the config only if the user defines it (well actually rabbitmq's conf can be easily overridden by a user without those variables).11:36
kevkomnasiadka: proxysql - in both cases this value is set and it's default is = 0 ...you are right that i put default 5, i will change to = 011:37
hrwkevko: -1 on one patch just after seeing commit message11:38
mnasiadkajust saying if proxysql in a years time changes the default to 1, we're still setting 0, just set it to an empty variable in defaults/main.yml and check if it's defined by user in the code, and set it only then.11:39
hrwmaybe I am too picky but commit message is a place to write "what changes" and "why it is changed" rather than "a few words which tell you to read whole patch and understand whole project"11:41
kevkomnasiadka: rabbitmq - yes you are right - but unfortunatelly rabbitmq can change between versions ..so that means i every version need to copy template from kolla11:41
kevkohrw: ok, i will change commit message11:42
hrwthx11:43
hrwkevko: when you also write release note then read it again and consider adding it into commit message11:45
hrw"rabbitmq: add ability to configure heartbeat" looks (imho) nicer than "Add heartbeat config option to rabbitmq.conf" as former one informs that it is configurable now. Second one just adds an option, but can it be altered/configured?11:47
mnasiadkakevko: right, rabbitmq.conf is nor ini, neither yaml, so we're not merging them - include that in the commit message - so everybody understands your motivation?11:50
kevkomnasiadka: ok11:50
mnasiadkaAnother stupid idea - RMQ 3.8 is unsupported now11:52
mnasiadkaShould we think about raising the version in stable branches somehow?11:52
mnasiadkaRabbitMQ 3.8 was released in October 2019, and reached end of life on July 31st, 2022, we strongly discourage its use11:53
mnasiadkahttps://www.rabbitmq.com/which-erlang.html11:53
hrwmnasiadka: rmq 3.10.8 + erlang 25.1.1?11:53
hrwmnasiadka: upgrade to rmq 3.10.8 + erlang 25.1.1?11:53
mnasiadkaI guess so - question if we don't break anything by doing so11:54
mnasiadkaseems 3.8 has a feature flag for rolling upgrades and allows interoperability with newer versions11:54
mnasiadkahttps://www.rabbitmq.com/upgrade.html#rolling-upgrades11:54
mnasiadkahrw/kevko/frickler: willing to +2 https://review.opendev.org/c/openstack/kolla/+/861355 ?11:55
hrwdone11:56
hrwmnasiadka: or 3.8 -> 3.9.23 as they do not mention 3.8 -> 3.10 upgrade yet11:57
hrwmnasiadka: still erlang 25.1.1 everywhere (needs my patches merge to master)11:58
fricklermnasiadka: hrw: reno question on the above11:59
hrwfrickler: we need one of those 'fix renos' patch before release to unify naming etc12:00
mnasiadkayes, that's right12:00
mnasiadka3.11 is out, but let's maybe not push it :)12:01
hrwmnasiadka: 'check do we need to bump rmq' is probably good add to release cycle notes12:02
mnasiadkayes, basically we should list components that need bumping - let me check our current docs12:02
opendevreviewMichal Nasiadka proposed openstack/kolla master: release-docs: Add infra components bump to R-5  https://review.opendev.org/c/openstack/kolla/+/86174512:08
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ovn: Break out role into ovn-db and ovn-controller roles  https://review.opendev.org/c/openstack/kolla-ansible/+/84776912:11
opendevreviewMichal Nasiadka proposed openstack/kolla master: release-docs: Add infra components bump to R-5  https://review.opendev.org/c/openstack/kolla/+/86174512:14
hrwmnasiadka: abandoned 3.10.8 patch - we move to newer erlang instead so no problem12:22
mnasiadkagood, I worried it's some maintenance burden attached ;)12:22
hrwmnasiadka: we need to merge https://review.opendev.org/c/openstack/kolla/+/861096 https://review.opendev.org/c/openstack/kolla/+/861327 so we have newest erlang everywhere. And then backport and backport12:23
hrwmnasiadka: and then backport after each release again probably ;D12:24
mnasiadkafirst one is merging12:24
mnasiadkafrickler: can you look into the second one?12:24
mnasiadkawell, I think we should bump up the version in stable branches to 3.9 instead of 3.10 and test that we don't break anything with it12:25
mnasiadka3.9 supports erlang 25 or only 24?12:25
mnasiadkaseems it supports 2512:26
hrwboth12:26
kevkoguys, I have a question regarding cinder upgrade - We are going to upgrade OS from victoria to wallaby, kolla-ansible is restarting all services, then it waits for configurable time (not ideal - but OK) and then online db migration is running ...problem is that we have two cinder volumes (clustered with redis coordination) ..but cinder-volume is reporting ONLINE after 25 minutes (because tousands and tousands volumes) ... we can't bump count of 12:30
kevkocinder-volume services ..because then ceph is killed by cinder-volumes ...12:30
amorinI know stein is old, but hey, I do with what I have :)12:30
kevkoso, we will be offline for 25 minutes 12:30
kevkoany suggestion how to deal with this ? 12:31
hrwamorin: you also waste your time a bit as upstream will tell 'rebase to master' anyway12:32
kevkoi think in kolla-ansible there should be a task which will check if cinder-volume is ONLINE ..then kolla-ansible will continue and restart other cinder-volumes to avoid time when we will be offline 12:32
fricklerhow do you handle this during normal maintenance? also not everyone identifies as "guys"12:32
hrw'folks' is often considered as more neutral12:34
kevkofrickler: sorry, no problem :) , from today i will use 'folks'12:34
amorinthis is not a waste IMO, most of the issues I currently have in stein are usually in master, but anyway, agree that it's old, I cant update yet unfortunately12:34
kevkofrickler: normal maintance ? normal maintance ussually not includes DB upgrades, API upgrades, RPC version upgrades ..etc 12:35
hrwamorin: do work on stein, compare with master, check which parts can be done by backporting from stein+12:36
mnasiadkakevko: ceph killer12:38
kevko?12:39
kevkodid you get my idea right ? :D 12:39
mnasiadkano, I didn't - you said you had a question, but didn't really ask the question :)12:45
mnasiadkaah, you did, a couple lines later12:45
mnasiadkaso, cinder-volume will be down for 25 minutes - is that a problem?12:45
opendevreviewDr. Jens Harbott proposed openstack/kolla master: Fix incorrect type conversion for cmd option '--docker-dir'  https://review.opendev.org/c/openstack/kolla/+/65660312:45
opendevreviewMarcin Juszkiewicz proposed openstack/kolla-ansible master: Update RHEL family information  https://review.opendev.org/c/openstack/kolla-ansible/+/86007512:46
hrwrebased to pass linters12:46
mnasiadkafrickler: do we actually need it? ;-)12:46
opendevreviewDr. Jens Harbott proposed openstack/kolla master: Fix incorrect type conversion for cmd option '--docker-dir'  https://review.opendev.org/c/openstack/kolla/+/65660312:47
kevkomnasiadka: well, it is problem ...because of 700+ kubernetes pernamently wants from cinder something 12:48
mnasiadkaah, that type of problem12:48
kevkoyeah :D 12:48
kevkoand it is big issue if it is not working ...12:48
mnasiadkaso, upgrade cinder-volume one by one?12:49
kevkoin kolla-ansible there should be a mechanism to wait for at least one cinder-volume in cluster for backend to be up 12:49
mnasiadkayes, that's what I meant12:49
opendevreviewMarcin Juszkiewicz proposed openstack/kolla-ansible master: Update RHEL family information  https://review.opendev.org/c/openstack/kolla-ansible/+/86007512:50
opendevreviewMarcin Juszkiewicz proposed openstack/kolla-ansible master: Default to Rocky Linux instead of CentOS Stream  https://review.opendev.org/c/openstack/kolla-ansible/+/86069212:50
kevkomnasiadka: yes, but upgrade will also upgrade DB ..so 12:50
fricklermnasiadka: I'm a big fan of considering warnings fatal, so yes, this should be fixed12:50
mnasiadkafrickler: I haven't seen this warning, but maybe I haven't been paying attention to tox output :)12:50
fricklermnasiadka: I verified the bug still applies to master, yes12:51
opendevreviewDr. Jens Harbott proposed openstack/kolla master: Fix incorrect type conversion for cmd option '--docker-dir'  https://review.opendev.org/c/openstack/kolla/+/65660312:59
opendevreviewwu.chunyang proposed openstack/kolla-ansible master: Switch trove-api to wsgi running under apache.  https://review.opendev.org/c/openstack/kolla-ansible/+/85475912:59
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: openvswitch: switch bridge setup to Ansible modules  https://review.opendev.org/c/openstack/kolla-ansible/+/84346013:00
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible master: Fix freezer deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/86175613:15
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ovn: Break out role into ovn-db and ovn-controller roles  https://review.opendev.org/c/openstack/kolla-ansible/+/84776913:18
opendevreviewwu.chunyang proposed openstack/kolla-ansible master: Switch trove-api to wsgi running under apache.  https://review.opendev.org/c/openstack/kolla-ansible/+/85475913:22
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: rabbitmq/aarch64: use COPR for Erlang  https://review.opendev.org/c/openstack/kolla/+/86132713:23
hrws/tabtab/spacesspaces/13:23
opendevreviewMerged openstack/kolla master: Bump node_exporter to 1.4.0  https://review.opendev.org/c/openstack/kolla/+/86135513:25
kevkomnasiadka: ok, i probably found a solution regarding cinder volumes 13:51
kevkomnasiadka: and kolla SHOULD provide configurable option to this variable 13:52
kevkomnasiadka: # Set to False if the pool is shared with other usages. On exclusive use driver13:52
kevko# won't query images' provisioned size as they will match the value calculated13:52
kevko# by the Cinder core code for allocated_capacity_gb. This reduces the load on13:52
kevko# the Ceph cluster as well as on the volume service. On non exclusive use13:52
kevko# driver will query the Ceph cluster for per image used disk, this is an13:52
kevko# intensive operation having an independent request for each image. (boolean13:52
kevko# value)13:52
kevko#rbd_exclusive_cinder_pool = true13:52
kevkomnasiadka: https://github.com/openstack/cinder/blob/ad184ca698d4b9cce912fcc1e936ef02c44b47d9/cinder/volume/drivers/rbd.py#L773-L77713:53
kevkowe have tens tousands ceph volumes ..and cinder volume is starting for 25 minutes only because of default cinder value true ...this should be kolla configurable 13:53
opendevreviewMerged openstack/kolla master: rabbitmq: use Erlang from PPA on Debian/Ubuntu  https://review.opendev.org/c/openstack/kolla/+/86109613:57
opendevreviewMerged openstack/kolla master: release-docs: Add infra components bump to R-5  https://review.opendev.org/c/openstack/kolla/+/86174513:57
mnasiadkakevko: nice, add it to external ceph guide with a variable - I'm all in :)14:00
kevkomnasiadka: hmm, but it looks like default is good (but it was changed from version to version - so anyway kolla should set this or document it ) ...because conditional is negation14:02
kevkoconditional in cinder14:03
kevkomnasiadka: ok, kolla should definitively set this option ... because default was changed during releases and it was also backported 14:11
kevkoI32c7746fa9149bce6cdec96ee9aa87b303de427114:11
opendevreviewMarcin Juszkiewicz proposed openstack/kolla stable/yoga: rabbitmq: use Erlang from PPA on Debian/Ubuntu  https://review.opendev.org/c/openstack/kolla/+/86176214:13
opendevreviewMarcin Juszkiewicz proposed openstack/kolla stable/yoga: rabbitmq: use Erlang from PPA on Debian/Ubuntu  https://review.opendev.org/c/openstack/kolla/+/86176214:19
mnasiadkakevko: should help in your case14:19
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: openvswitch: switch bridge setup to Ansible modules  https://review.opendev.org/c/openstack/kolla-ansible/+/84346014:23
mnasiadkahrw: https://review.opendev.org/c/openstack/kolla-ansible/+/860075 - planning to update per Radek's comments, or should I?14:27
atmarkwhy is there no enable_horizon_barbican?14:35
atmarknvm, i found the answer. looks like barbican-ui is abandoned14:36
wuchunyangHi, everyone, I find some project's keystone endpoint  was defined in all.yml, others was defined in default/main.yml. which one should be recommend?14:49
mnasiadkawuchunyang: depends if multiple roles need it, or just one14:57
wuchunyangmnasiadka: But it doesn't seem to be that way.  e.g. octavia_public_endpoint, this is only needed by octavia role.15:03
wuchunyang I was working on trove role, and I am not sure whether i should remove the endpoint to all.yml.15:04
wuchunyangIf this does not affect anything, i will keep it there.15:07
hrwmnasiadka: https://review.opendev.org/c/openstack/kolla-ansible/+/860692 does that15:42
* hrw out15:42
mnasiadkahrw: ah, thanks15:46
opendevreviewMerged openstack/kolla master: rabbitmq/aarch64: use COPR for Erlang  https://review.opendev.org/c/openstack/kolla/+/86132715:53
mnasiadkaMeeting in 5 minutes for Kolla Operator Hour is here: https://meetpad.opendev.org/kolla-antelope-ptg15:55
fricklermnasiadka: if we use the other etherpad then this is confusing15:59
mnasiadkawell, it was precreated - and I was asked to use the precreated one - what can I tell15:59
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: RBD: Don't query Ceph on stats for exclusive pools  https://review.opendev.org/c/openstack/kolla-ansible/+/86177416:00
kevkomnasiadka: https://review.opendev.org/c/openstack/kolla-ansible/+/86177416:01
mnasiadkalet me copy the content to kolla-antelope-ptg16:01
opendevreviewEbbex proposed openstack/kayobe master: Use dnf module to get rid of warning  https://review.opendev.org/c/openstack/kayobe/+/86177916:43
dardeleanwhat is redis used for in kolla?17:00
dardeleanas support for what?17:00
kevkocoordination url for cinder ha 17:09
kevkoor designate cooridenation 17:09
kevkoor osprofiler 17:10
kevkodardelean: ^ 17:10
dardeleaneye see, thanks kekvo17:10

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!