Wednesday, 2023-05-17

opendevreviewVishal Manchanda proposed openstack/kolla master: Revert "Pin setuptools=67.2.*"  https://review.opendev.org/c/openstack/kolla/+/87646010:32
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: fluentd: pin fluent-plugin-elasticsearch to <5.3.0  https://review.opendev.org/c/openstack/kolla/+/88335811:49
opendevreviewMichal Nasiadka proposed openstack/kolla master: Revert "Pin setuptools=67.2.*"  https://review.opendev.org/c/openstack/kolla/+/87646011:50
opendevreviewMichal Nasiadka proposed openstack/kolla master: Revert "toolbox: pin ansible.posix to <1.5.4"  https://review.opendev.org/c/openstack/kolla/+/88312711:55
opendevreviewDawud proposed openstack/kolla-ansible master: Use friendly prometheus instance labels  https://review.opendev.org/c/openstack/kolla-ansible/+/87635711:57
SvenKieskemnasiadka: sure master wasn't broken as well by ansible.posix?12:00
mnasiadkaSvenKieske: just want to check12:01
mnasiadkaif it is12:01
SvenKieskeah okay12:01
mnasiadkathe bug is not gaining any attention ;-)12:01
SvenKieskeah I wanted to talk about the bug https://github.com/ansible-collections/ansible.posix/issues/461 , thanks for the reminder12:02
SvenKieskeimho it's missing a clear output what went wrong and what is expected behaviour?12:03
SvenKieskethen again when I look at the repo history we're lucky if each forthnight a bug can be closed :D12:04
mnasiadkabasically kolla_toolbox is running ansible (in ad-hoc mode) and setting JSON output12:06
mnasiadkawhich is fine, but new posix.json callback plugin version requires play.path variable (which most probably only exists in normal mode, not ad-hoc mode)12:06
mnasiadkaa simple dumb if probably would suffice, but I don't know if I have the time to work on this :)12:07
opendevreviewMerged openstack/kolla stable/yoga: toolbox: pin ansible.posix to <1.5.4  https://review.opendev.org/c/openstack/kolla/+/88312212:15
opendevreviewDawud proposed openstack/kolla-ansible master: Use friendly prometheus instance labels  https://review.opendev.org/c/openstack/kolla-ansible/+/87635712:42
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: fluentd: pin fluent-plugin-elasticsearch to <5.3.0  https://review.opendev.org/c/openstack/kolla/+/88335812:44
opendevreviewMark Goddard proposed openstack/kayobe-config-dev master: Rename disable_selinux_do_reboot  https://review.opendev.org/c/openstack/kayobe-config-dev/+/88336812:50
mnasiadkamgoddard mnasiadka hrw  bbezak frickler kevko SvenKieske mmalchuk - meeting in 1012:50
mnasiadka#startmeeting kolla13:02
opendevmeetMeeting started Wed May 17 13:02:44 2023 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.13:02
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:02
opendevmeetThe meeting name has been set to 'kolla'13:02
mnasiadka#topic rollcall13:02
mnasiadkao/13:02
mmalchuko/13:03
SvenKieskeo/13:03
mhinero/13:03
mgoddard\o13:03
mattcreeso/13:03
mnasiadka#topic agenda13:05
mnasiadka* CI status13:05
mnasiadka* Release tasks13:05
mnasiadka* Regular stable releases (first meeting in a month)13:05
mnasiadka* Current cycle planning13:05
mnasiadka* Additional agenda (from whiteboard)13:05
mnasiadka* Open discussion13:05
mnasiadka#topic CI Status13:05
mnasiadkaSo, we've had multiple breakages recently13:06
mnasiadkajson callback plugin is the most recent13:06
mnasiadkayoga merged today, so tomorrow we can merge zed13:06
mnasiadkaand become ,,stable-ish'' again13:06
mnasiadka#topic Release tasks13:06
mmalchukcan we force rebuild images?13:06
mnasiadkaWe can, but there's no need, a couple of hours won't save anybody's life13:07
mmalchukcouple of days13:07
mnasiadkaRocky Linux 9.2 is out, so centos-release-openstack-antelope should be there - the patch to use it in master should be good to go13:08
mmalchukday for xena, day for yoga, day for zed atleast13:08
mmalchukone day pass13:08
mnasiadkaI updated the transition to DockerClient patch13:08
mnasiadkaMaybe we can get the podman/buildah support in Kolla as well this cycle13:08
mnasiadkabut we need to start branching fast - 1st June is deadline13:08
opendevreviewMark Goddard proposed openstack/kolla-ansible master: ironic: Enable automated cleaning by default  https://review.opendev.org/c/openstack/kolla-ansible/+/88336913:09
opendevreviewGaël THEROND proposed openstack/kolla-ansible master: Improve designate role support.  https://review.opendev.org/c/openstack/kolla-ansible/+/87827013:09
mnasiadkaI don't think we can get to working podman before the release, but let's see how that goes13:10
opendevreviewGaël THEROND proposed openstack/kolla-ansible master: Improve designate role support.  https://review.opendev.org/c/openstack/kolla-ansible/+/87827013:10
mnasiadkaLet's move to additional agenda and use the remaining time to do the RP+1 reviews in Kolla and Kolla-Ansible13:10
mnasiadka#topic Additional agenda (from whiteboard)13:11
mnasiadka(ihalomi): do we really need shared /run mount among containers?13:11
mnasiadkamhiner: ihalomi is not here - do you want to discuss that or should we wait for next week?13:11
mhinerlets wait13:12
mnasiadkaok then13:12
mnasiadka#topic Open discussion13:12
mnasiadkaAnybody anything?13:12
opendevreviewMichal Nasiadka proposed openstack/kolla master: Transition to DockerClient and docker 3.0.0+  https://review.opendev.org/c/openstack/kolla/+/82578613:12
SvenKieskeI'm on PTO for the rest of the week and tomorrow is a public holiday here, so don't expect frickler to be there either13:12
mmalchukplease review security fix: https://review.opendev.org/c/openstack/kolla-ansible/+/882100?usp=search13:13
mnasiadkaHopefully frickler will have time next week to push some of the patches forward13:13
mmalchukand CI fix in Kayobe: https://review.opendev.org/c/openstack/kayobe/+/881679?usp=search13:13
mmalchukmgoddard ^^^13:13
mmalchukthanks in advance13:14
mnasiadkaok, I assume no other topics13:18
mmalchukforce rebuild?13:18
mnasiadkaThanks for coming, please review all RP+1 patches at least in Kolla and Kolla-Ansible13:18
mnasiadkammalchuk: I said no, is that not enough?13:18
mmalchukok13:19
mmalchukthanks13:19
mnasiadkathanks13:19
mnasiadka#endmeeting13:19
opendevmeetMeeting ended Wed May 17 13:19:28 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)13:19
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-05-17-13.02.html13:19
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-05-17-13.02.txt13:19
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-05-17-13.02.log.html13:19
opendevreviewMichal Nasiadka proposed openstack/kolla master: Transition to DockerClient and docker 3.0.0+  https://review.opendev.org/c/openstack/kolla/+/82578613:22
opendevreviewPierre Riteau proposed openstack/kayobe master: CI: Remove disable_selinux_do_reboot flag  https://review.opendev.org/c/openstack/kayobe/+/88337213:42
opendevreviewMichal Nasiadka proposed openstack/kolla master: toolbox: use venv instead of virtualenv  https://review.opendev.org/c/openstack/kolla/+/87491613:58
opendevreviewMichal Nasiadka proposed openstack/kolla master: toolbox: use venv instead of virtualenv  https://review.opendev.org/c/openstack/kolla/+/87491613:59
opendevreviewMichal Nasiadka proposed openstack/kolla master: toolbox: use venv instead of virtualenv  https://review.opendev.org/c/openstack/kolla/+/87491614:00
opendevreviewMichal Nasiadka proposed openstack/kolla master: CentOS/Rocky: use CentOS Cloud SIG repo instead of Delorean  https://review.opendev.org/c/openstack/kolla/+/88255814:04
opendevreviewDawud proposed openstack/kolla-ansible master: Use friendly prometheus instance labels  https://review.opendev.org/c/openstack/kolla-ansible/+/87635714:07
opendevreviewMichal Nasiadka proposed openstack/kolla master: CentOS/Rocky: use CentOS Cloud SIG repo instead of Delorean  https://review.opendev.org/c/openstack/kolla/+/88255814:25
opendevreviewMichal Nasiadka proposed openstack/kolla master: ubuntu: use Antelope UCA  https://review.opendev.org/c/openstack/kolla/+/88338014:29
opendevreviewMichal Nasiadka proposed openstack/kolla master: debian: Use Antelope extrepo  https://review.opendev.org/c/openstack/kolla/+/88338114:32
opendevreviewVerification of a change to openstack/kolla master failed: mariadb: log bootstrap output to file  https://review.opendev.org/c/openstack/kolla/+/88224714:42
opendevreviewMichal Nasiadka proposed openstack/kolla master: Transition to DockerClient and docker 3.0.0+  https://review.opendev.org/c/openstack/kolla/+/82578615:09
opendevreviewMichal Nasiadka proposed openstack/kolla master: CentOS/Rocky: use CentOS Cloud SIG repo instead of Delorean  https://review.opendev.org/c/openstack/kolla/+/88255815:11
mnasiadkahrw: is it me or Antelope requires bookworm? https://salsa.debian.org/extrepo-team/extrepo-data/-/blob/master/repos/debian/osbpo.yaml15:19
mnasiadkazigo: can you comment on ^^?15:20
zigoCorrect.15:20
zigoZed is the release were I have Bullseye+Bookworm, though Antelope is Bookworm only.15:21
mnasiadkaholy crap15:21
mnasiadkawell then, we need to have extra fast route to bookworm it seems15:22
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Use Debian Bookworm  https://review.opendev.org/c/openstack/kolla/+/88338815:27
mnasiadkaSvenKieske: you guys use Debian, right?15:29
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Use Debian Bookworm  https://review.opendev.org/c/openstack/kolla/+/88338815:32
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Use Debian Bookworm  https://review.opendev.org/c/openstack/kolla/+/88338815:33
hrwmnasiadka, zigo: argh.15:34
hrwmnasiadka: there is no bookworm mirror on CI15:35
mnasiadkahrw: even better :)15:35
hrwmnasiadka: https://review.opendev.org/q/topic:for-debian-bookworm-upgrade15:36
hrwhttps://review.opendev.org/c/opendev/system-config/+/881952 is crucial15:36
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Use Debian Bookworm  https://review.opendev.org/c/openstack/kolla/+/88338815:37
mnasiadkahrw: well, mirrors can come later, I've seen the topic - let's try to glue it together somehow - I'm not keen on delaying final release for a month or more15:38
hrwmnasiadka: kick debian from release list?15:38
mnasiadkaDrop Debian? :)15:38
zigohrw: We should build one...15:38
mnasiadkaThat's an option15:38
hrwwe are 2 months behind Antelope already15:38
zigomnasiadka: Yeah, great, and drop ubuntu and CentOS... let's build on Alpine only.15:38
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Use Debian Bookworm  https://review.opendev.org/c/openstack/kolla/+/88338815:38
mnasiadkahrw: we might just release without Debian for now, but let's see how far are we from having images15:39
zigoGuys, you already droped pacakge deployment ...15:39
hrwmnasiadka: 5-7 images fail due to lack of 3rdparty repos15:40
mnasiadkabinary builds? meh15:40
mmalchuk+1 for Alpine)15:40
mnasiadkaargh, fantastic15:40
hrw17:40 marcin@puchatek:debian-book-master1$ ls 0*15:40
hrw000_FAILED_fluentd.log   000_FAILED_kolla-toolbox.log  000_FAILED_openvswitch-netcontrold.log  000_FAILED_rabbitmq.log15:40
hrw000_FAILED_influxdb.log  000_FAILED_mariadb-base.log   000_FAILED_proxysql.log                 000_FAILED_telegraf.log15:40
zigoIt's really not hard to add a repo to the infra...15:40
mnasiadkawell, mariadb, rabbitmq and fluentd are a bit important15:40
hrwno fluentd, influxdb, telegraf, mariadb, erlang, proxysql15:40
mnasiadkaif they are not there today, we're not going to make it15:41
hrwerlang probably can be taken from ubuntu/jammy 15:41
mnasiadkaso yes, let's release kolla rc1 without Debian and see how that goes15:41
zigoYou can take rabbit and mariadb from Bookworm, I can confirm that works.15:41
zigoI did validate Antelope like the week after the release with my own installer (puppet-openstack based).15:42
mnasiadkawell, we've taken mariadb from ubuntu repos for jammy in the past15:42
hrwif Debian bookworm gets released on 12th June then August is maybe time when 3rdparty repos catch up15:42
zigoAll of that is validated...15:42
zigohrw: The question is, why do you use 3rd party repos.15:42
zigoThat's LAME and often BROKEN.15:42
hrwzigo: wasn't my decision15:42
zigoIt's a generality: upstream people don't know how to package.15:43
zigoI've seen stupidity in so many upstream repos ...15:43
hrwand distro people don't know how^Wwhen to upgrade15:43
zigoWhen to upgrade what?15:43
hrwcomponents we use to versions supported upstream?15:43
mnasiadkaIn case of Debian they don't know when to update Ceph and OpenvSwitch :)15:43
zigomnasiadka: As the maintainer of both packages, I thank you a lot for saying that.15:44
zigoIsn't OVS 3.1 enough?!?15:44
mnasiadkaWell, in Bookworm, right?15:44
zigoOf course, yes ...15:44
zigoIf needed, I can always upload to official backports.15:44
zigoI did many times when needed.15:45
mnasiadkabullseye now has 2.17, incredible15:45
zigoAlso did unofficial backports in the debian.net namespace when needed.15:45
zigoThat's the concept of a stable release: it doesn't change.15:45
mnasiadkaBasically, I don't want to continue that thread - the problem is, that we lack Debian maintainers in Kolla15:45
zigoI never seen a bug report from anyone asking for a backport ...15:45
zigoIf you need one, feel free to ping me, and I'll be happy to help.15:45
hrwsorry folks, but I have to go15:46
zigoMe too. Have a great week-end.15:47
opendevreviewMichal Nasiadka proposed openstack/kolla master: CentOS/Rocky: use CentOS Cloud SIG repo instead of Delorean  https://review.opendev.org/c/openstack/kolla/+/88255816:44
opendevreviewMerged openstack/kolla master: mariadb: log bootstrap output to file  https://review.opendev.org/c/openstack/kolla/+/88224717:09
opendevreviewDawud proposed openstack/kolla-ansible master: Use friendly prometheus instance labels  https://review.opendev.org/c/openstack/kolla-ansible/+/87635717:16
opendevreviewGaël THEROND proposed openstack/kolla-ansible master: Improve designate role support.  https://review.opendev.org/c/openstack/kolla-ansible/+/87827017:25
spatelmnasiadka I have question related patching images during build. 17:51
spatelDo you know what is the best way to patch images using kolla-build?17:51
kevkooverrides17:51
kevkomnasiadka you probably also forgot that i am also DD in debian :D 17:53
mmalchukspatel I've answered in mailing list17:57
spatellet me check 17:58
mmalchukthe best way is build from git source17:58
mmalchukyou should clone repo, create new branch, apply your patch, and build from this branch17:59
spatelmmalchuk like this example https://paste.opendev.org/show/bGKXCcbXlc4xldTLnudG/18:01
spatelCan't we pass commit hash to reference? 18:01
spatelso it will pull/clone from remote repo instead of maintain local copy of repo? 18:02
mmalchukspatel sure18:03
mmalchukyou can use any git refference: branch, tag, hash, etc.18:03
spatelLet me run test and validate if that works. 18:04
spatelI have one more question, I did this just for testing. It build image but use tag zed instead of keystone-zed which i specified in keystone stanza - https://paste.opendev.org/show/bvo9LGbAAWgsDMw7gAPT/18:06
spatelIt should use tag:keystone-zed correct? 18:07
mmalchukuse tag = your-tag in kolla-build.conf or pass the --tag option18:08
mmalchukit not should it might be any tag you desire, but you also you should configure kolla-ansible to use images with your custom tag18:09
spatelThis is my kolla-build.conf file - https://paste.opendev.org/show/bwA3UORjL5OvVG592CIw/18:10
spatelYes i have to use custom tag in global.yml. I am just trying to run all kind of test for documentation work and learning. 18:11
mmalchuktag=keystone-zed - incorrect18:11
mmalchukreference = stable/zed18:11
mmalchukoption tag only in [DEFAULT] section18:12
spateloh! so I have to pass custom tag in command like with --tag my-keystone 18:13
spatelotherwise it will use [DEFAULT] one18:13
mmalchukhttps://docs.openstack.org/kolla/latest/admin/image-building.html#build-openstack-from-source18:13
mmalchukthe valid options for the repo is: type, location, reference, enabled18:14
spatel+1 18:15
mmalchukthere is no tag option18:15
mmalchukonly in [DEFAULT] section18:15
spatelPerfect!! 18:17
spatelI love the way kolla works and all pieces connected to each other, so simple to play :) 18:19
spatelDo you guys run deployment nodes in container or host itself? 18:20
spatelwhy kolla or team doesn't provide kolla-ansible inside container so we don't need to do venv etc.. 18:21
spatelMay be its personal choice of end users. 18:21
mmalchukyou can look at Kayobe project, there deployment host in the kvm vm18:25
mmalchukhttps://docs.openstack.org/kayobe/latest/18:25
spatelI heard about kayobe but not sure how does it work18:29
spateldoes it work like triple0, over/under cloud?18:30
spatelHow its different to legacy kolla-ansible deployment18:32
mmalchukit work as bifrost + kolla-ansible18:33
mmalchukbifrost for provision baremetal18:34
mmalchukkolla-ansible for deploy openstack18:34
spatelGot it, In short bifrost provision your newly installed server using ironic/ipxe etc and then kolla-ansible deploy containers to run specific roles. 18:37
spatelIn my DC, I don't have dedicated LAN for PXE boot which bifrost can control to inspect etc.. 18:38
spatelBut i will sure give it a try in lab and see how useful its. 18:39
mmalchukyou can use Tenks project to deploy kvm vm as a baremetal18:40
mmalchukhttps://docs.openstack.org/tenks/latest/18:40
mmalchuktake a look at this workshop as an example: https://github.com/stackhpc/a-universe-from-nothing18:42
spatelThanks, I will take a look 18:47
spatelWhat is the meaning of this block [keystone-base] why need to add -base what if i use [keystone-foo] ? 18:49
spatelin kolla-build.conf file 18:49
mmalchukkeystone-base is a base for other keystone-* images18:50
spatelHere is the list of images it built - https://paste.opendev.org/show/ba8iP6BtdwNYZuxgKnc7/18:52
spatelyou are saying it use keystone-base first and then fork other from base image correct?18:52
mmalchuknot first but base18:53
mmalchukfor example kolla/docker/keystone/keystone-ssh/Dockerfile.j218:53
mmalchukthe first line: FROM {{ namespace }}/{{ image_prefix }}keystone-base:{{ tag }}18:54
kevkorun kolla --template-only --no-pull --no-push18:54
kevkoto see what will be in dockerfiles on the end18:54
spatelkevko is kolla command? I don't have one18:56
kevkoyeah it is 18:56
spatelkolla: command not found18:56
kevkook, sorry 18:56
kevkohttps://github.com/openstack/kolla/blob/master/tools/build.py18:56
kevko./tools/build.py18:57
mmalchukspatel kolla-build command)18:58
spatel kolla-build: error: unrecognized arguments: --no-pull --no-push18:58
kevko--help18:58
kevko| egrep 'pull|push|template'18:58
mmalchuk--nopull18:58
mmalchuk--nopush18:58
mmalchukkolla-build --help18:59
kevkodockerfiles then will be generated in /tmp/ ...so you can check final dockerfiles18:59
spatelit dump all docker files in /tmp/kolla-2023-05-17_18-58-50_rdhxnrkp/docker/18:59
spatelThis is aweseome! 19:00
spatelawesome* 19:00
kevkoyep, now you can see everything ..19:00
kevkowhich image is FROM which ... etc19:00
spatelTotally get it, It use base image for sub images. That make sense 19:01
kevkobase and openstack-base19:01
kevkoopenstack-base is for services ...19:01
kevkooslo libs, services ..e.tc19:01
spateloh! 19:02
kevkobase is for rabbit, maria, memcached...etc19:02
spatelwhy this command not part of doc :) /opt/kolla/tools/build.py --template-only --nopull --nopush 19:02
spatelkevko Thanks for that command19:02
mmalchukbase also for openstack-base :)19:02
kevkobut all openstack images using oslo libs and other openstack developed libraries ...so on top of base openstack-base is build which includes all this libraries19:02
kevkommalchuk: also .. yeah ..kind of .. 19:03
spatelthat make sense, why need to re-build libs/oslo for each components, Just use reference image 19:03
kevkobase -> openstack-base -> nova-base -> nova-api19:03
spatel+119:04
mmalchuka - architecture)19:04
kevkoyeah, but i miss binary imaches :( 19:04
kevko*images19:04
kevkobuilds19:04
spatelmmalchuk I have use this snippet and it works - https://paste.opendev.org/show/boG1KZ0NdsiVFoljGyfM/19:06
mmalchukit should)19:07
spatelI use specific hash to build image19:07
mmalchukas I said - any git refference19:07
spatelHow do i validate it really use that hash. I believe in build output there must be reference of hash somewhere correct?19:07
mmalchuk--debug19:07
spatelGot it 19:07
mmalchuk /var/log/kolla-build.log19:08
spatelI can see this line in log - INFO:kolla.common.utils.glance-base:Clone dir /tmp/kolla-2023-05-17_18-26-47_tot4knbo/docker/glance/glance-base/glance-base-archive-41da684fe41e48eca934aca1f19fbb779be07fc4 exists. Removing it.19:08
spatelassuming it used 41da684fe41e48eca934aca1f19fbb779be07fc419:09
BobZAnnapolis_sorry, another newbie question. Using AWS, we've gotten the aios/quick starts to run to completion and get to the dashboard;most of the times we can actually create vms but we haven't been able to ssh into them - what piece / doc are we missing that'll get us past that hurdle ? tia19:40
BobZAnnapolis_Is it a user error in how we (1) created the vms ?, (2) created the OS networks, (3) a known limitation of running the aio in AWS, (4) other ? tia19:43
BobZAnnapolis_The run-once is doing some creation & testing, not sure if it tests ssh'ing in. But when we create a new vm, we can't ssh in to them. Sounding more & more like user error :-) We're trying to ssh in to the launched instances from the vm we ran the quickstart/aio playbooks from. 19:54

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!