Friday, 2021-04-16

« Thursday, 2021-04-15 Index Saturday, 2021-04-17 »
*** kevko has quit IRC00:08
*** k_mouza has joined #openstack-kolla00:45
*** k_mouza has quit IRC00:50
*** macz_ has joined #openstack-kolla01:21
*** macz_ has quit IRC01:25
*** macz_ has joined #openstack-kolla01:32
*** macz_ has quit IRC01:36
*** e0ne has joined #openstack-kolla01:39
*** e0ne has quit IRC01:46
*** zul has quit IRC03:15
*** macz_ has joined #openstack-kolla03:33
*** macz_ has quit IRC03:37
*** vishalmanchanda has joined #openstack-kolla04:29
*** dabukalam has quit IRC04:47
*** k_mouza has joined #openstack-kolla04:48
*** dabukalam has joined #openstack-kolla04:49
*** k_mouza has quit IRC04:52
*** cah_link has joined #openstack-kolla04:54
*** skramaja has joined #openstack-kolla05:10
*** shyamb has joined #openstack-kolla05:54
*** LinPeiWen has quit IRC06:04
*** shyam89 has joined #openstack-kolla06:35
*** shyamb has quit IRC06:38
*** kevko has joined #openstack-kolla06:39
*** luksky has joined #openstack-kolla06:58
*** bengates has joined #openstack-kolla07:13
*** jbadiapa has joined #openstack-kolla07:14
*** andrewbonney has joined #openstack-kolla07:15
mnasiadkamorning07:20
mnasiadkahrw, yoctozepto: bullseye change green as grass...07:21
*** gfidente|afk is now known as gfidente07:27
*** rpittau|afk is now known as rpittau07:32
hrwlgtm when it stays that way07:34
kevkomorning07:35
openstackgerritMichal Nasiadka proposed openstack/kolla master: rabbitmq: Move to packagecloud  https://review.opendev.org/c/openstack/kolla/+/78639007:37
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Bump up python version for Debian Bullseye  https://review.opendev.org/c/openstack/kolla-ansible/+/78659107:46
hrwo! rabbitmq stopped crashing07:46
mnasiadkaok, I'll prepare changes to bump up python version in correct way and remove the hack in zuul config ;_07:46
mnasiadka;)07:46
hrw+207:47
mnasiadkahrw: so do we want to recheck UCA change?07:47
hrwpuchatek.lan:5000/bullseye-0415-2/debian-source-mariadb-clustercheck:11.1.0 names are crazy07:47
mnasiadkahmm, I think we would need a reno in Bullseye change? :)07:49
hrwlet merge and add note in separate?07:51
hrwto not go another round of checks07:51
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: releasenotes: we move to Debian bullseye  https://review.opendev.org/c/openstack/kolla/+/78659407:55
hrwdone07:55
mnasiadkathanks08:00
mnasiadkayoctozepto: wake up, bullseye waiting for you ;)08:00
hrwhttps://review.opendev.org/c/openstack/kolla/+/786042 for morning coffee/drink (except mnasiadka as he already +2)08:05
*** shyam89 has quit IRC08:13
yoctozeptomnasiadka: it will have to wait08:25
*** LinPeiWen has joined #openstack-kolla08:26
*** ivan_lin has joined #openstack-kolla08:26
mnasiadkahrw: so with RDO we're waiting for python3-networking-vmware-nsx?08:28
*** e0ne has joined #openstack-kolla08:28
mnasiadkaok, seems it's there - I kicked recheck08:30
hrwmnasiadka: we wait for centos-release-openstack-wallaby08:38
hrwmnasiadka: rdo-release-wallaby package is temporary solution to check how we are without delorean repos08:38
mnasiadkahrw: last time it took 1-2 weeks for somebody to put a package on centos mirrors...08:38
mnasiadkahrw: that's why I commented if we really have to switch afterwards ;)08:38
*** e0ne has quit IRC08:38
hrwso we wait08:39
hrwcentos-release-openstack-wallaby is the only way08:40
hrwwe do rc1 without it, then rdo package kolla rc1 and release c-r-o-wallaby package. we move to it, do rc208:41
hrwin meantime bullseye, uca08:41
mnasiadkahrw: so as always, but we can do rc-1 with rdo-release-wallaby package ;)08:43
hrwmnasiadka: 09 12:06 < amoralej> rdo-release is conceived for rhel or non-centos rhel clones08:45
hrwFAILED - RETRYING: nova | Ensure RabbitMQ users exist (1 retries left).08:46
hrwmeh.08:46
mnasiadkahrw: so... maybe it's really the rabbitmq downgrade case? if we move to packagecloud we're getting 3.8.12 at least (or stick to bionic in the UCA change)08:48
yoctozeptobionic?08:49
mnasiadkayoctozepto: well, if you use focal as distro in rabbitmq repo - you get older rabbitmq than when using bionic (which we do in V) :D08:50
yoctozeptokill it with fire08:50
hrwyoctozepto: it is same package for bionic, focal, buster...08:52
hrwSAME08:52
*** e0ne has joined #openstack-kolla08:54
yoctozeptohrw: mnasiadka claims otherwise08:57
yoctozeptonot me08:57
mnasiadkaupgrade job: first 2021-04-15 10:57:36.713 Starting RabbitMQ 3.8.14 on Erlang 23.3.1 AND then 2021-04-15 11:20:01.253 Starting RabbitMQ 3.8.12 on Erlang 23.3.109:00
mnasiadkaso from where do we get this package? :)09:01
*** k_mouza has joined #openstack-kolla09:02
mnasiadkaActually I was wrong, nothing is really changed apart the UCA repo in this change...09:04
*** e0ne has quit IRC09:04
*** e0ne has joined #openstack-kolla09:06
mnasiadkaoh boy rabbitmq... packagecloud gpg key is only for repo, but not for packages09:07
*** e0ne has quit IRC09:10
*** dave-mccowan has quit IRC09:11
*** dave-mccowan has joined #openstack-kolla09:13
openstackgerritMichal Nasiadka proposed openstack/kolla master: rabbitmq: Move to packagecloud  https://review.opendev.org/c/openstack/kolla/+/78639009:14
mnasiadkahrw, yoctozepto: I already see that we have failing upgrade CI, because Nova master is X and requires at oldest Wallaby computes - time to branch?09:29
mnasiadkathey merged this 9 hours ago: https://github.com/openstack/nova/commit/74d6fc4c28125a5a43b20cd7ec1015f934266de209:30
*** macz_ has joined #openstack-kolla09:35
*** e0ne has joined #openstack-kolla09:37
*** macz_ has quit IRC09:39
*** LinPeiWen has quit IRC09:41
*** shyamb has joined #openstack-kolla09:42
*** LinPeiWen has joined #openstack-kolla09:48
*** k_mouza has quit IRC10:03
*** e0ne has quit IRC10:10
*** ivan_lin has quit IRC10:22
*** LinPeiWen has quit IRC10:23
*** ivan_lin has joined #openstack-kolla10:23
*** k_mouza has joined #openstack-kolla10:31
priteauyoctozepto: hrw: would you have the chance to review this? Fairly trivial https://review.opendev.org/c/openstack/kolla-ansible/+/78252510:37
*** LinPeiWen has joined #openstack-kolla10:38
*** k_mouza has quit IRC10:46
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Use stable/wallaby sources  https://review.opendev.org/c/openstack/kolla/+/78662010:47
*** k_mouza has joined #openstack-kolla11:01
*** dave-mccowan has quit IRC11:08
hrwmnasiadka: time to branch11:22
hrwyoctozepto: there are package files and package indexes. packages are the same. indexes differ11:23
yoctozeptohrw: I have no idea why you are telling me this; I understand the packaging for apt11:25
yoctozeptomnasiadka: so we are all RED now?11:26
mnasiadkayoctozepto: basically yes - on upgrade jobs11:26
yoctozeptoarar11:26
mnasiadkanova barfs and we're in hell11:26
yoctozeptook, one more argument for early switch11:27
hrwyoctozepto: bionic/buster index says 3.8.14, focal index says 3.8.1211:28
yoctozeptomnasiadka: https://review.opendev.org/c/openstack/kolla/+/78550011:28
priteauWhy don't we switch to the stable branch as soon as they are created?11:28
hrwyoctozepto: that's what mnasiadka meant11:28
mnasiadkayoctozepto: well then, do you need updates in it - or can we merge it?11:29
yoctozeptohrw: yes, I understand lol11:29
yoctozeptomnasiadka: the issue was vmware nsx11:29
yoctozeptoI wonder if it still holds11:29
yoctozeptoyour patch will show11:29
yoctozeptoso wait for the tests11:29
yoctozeptoand then we amend11:29
yoctozeptook, so11:30
yoctozeptotop priority is to switch to wallaby now11:30
yoctozeptoI can force image upload to quay should we need it11:30
yoctozeptobut we can also wait till tomorrow for publishing11:31
yoctozeptobut need to merge wallaby switch11:31
mnasiadkayup, then fix what we need (Bullseye, UCA), branch + rc, wait for RDO on official mirrors, rc211:31
yoctozepto++11:31
*** k_mouza has quit IRC11:48
*** k_mouza has joined #openstack-kolla11:55
*** cosmicsound has quit IRC12:11
*** macz_ has joined #openstack-kolla12:19
*** macz_ has quit IRC12:24
*** k_mouza has quit IRC12:26
mnasiadkayoctozepto: networking-ansible and telemetry-tempest-plugin not branched12:33
mnasiadkayoctozepto: https://222ead189344188148d2-a1dc26a377ecb78f56a1ade61b6bdb54.ssl.cf5.rackcdn.com/786620/1/check/kolla-build-centos8s-source/271d8ff/kolla/build/index.html12:34
yoctozeptomnasiadka: tempest does not branch12:36
yoctozeptowe are supposed to run master12:36
mnasiadkatelemetry-tempest-plugin also does not branch?12:36
mnasiadkaright, it doesn't12:36
yoctozeptoit should not12:36
mnasiadkaso needs update in config to skip it12:36
yoctozeptotempest should be master too12:36
mnasiadkaso we just need to mark networking-ansible as a bad example ;)12:37
yoctozeptobut vmware-nsx worked?12:37
yoctozeptoas wallaby?12:37
yoctozeptoah12:37
yoctozeptohard to tell12:37
yoctozeptosince it failed on ansible already12:38
mnasiadkaI think it failed on networking-ansible, so we don't know :)12:38
yoctozeptoyeah yeah12:38
hrwand we have the same again ;D12:38
*** k_mouza has joined #openstack-kolla12:38
hrwmaybe next cycle we should try something new and branch when others branch?12:38
yoctozeptohrw: we don't have to branch, but we have to switch to stable in config12:39
yoctozeptobut we can branch as well12:39
yoctozeptoI'm kinda SWOTing this out in kolla PTG etherpard12:40
yoctozeptoetherpad*12:40
hrwwe do stable-in-master and then revert revert revert...12:40
mnasiadkayoctozepto: so, all those you removed did not branch W still12:41
mnasiadkanow question is what do we do first - branch or merge this patch12:42
mnasiadkamaybe hrw is right it's better to branch and then start merging stuff there12:42
mnasiadkainstead of doing reverts12:42
hrwI am adding notes to PTG etherpad12:42
yoctozeptocan't branch/release broken either12:43
yoctozeptounless we remove the publishing from release jobs12:43
yoctozeptoI think we should12:43
yoctozeptonobody uses the versioned images12:43
yoctozeptoat least should not12:43
hrwxena-testing?12:43
yoctozeptofor branching/releasing we need mgoddard anyhow12:43
yoctozeptohrw: no, it's something like 22.0.0rc112:44
yoctozeptoI don't remember the version12:44
hrwopenstack.kolla/debian-source-binary-base:xena-we-told-you-to-not-use-them12:44
yoctozeptocause we honestly don't care12:44
yoctozeptono, it's these tags that appear nowhere12:44
yoctozeptoexcept for being published lol12:44
mnasiadkawell, they appear in pip?12:44
hrwand CI uses published ones12:44
hrwnumbers go to pip12:45
yoctozeptomnasiadka: I'm talking images12:45
yoctozeptodockerhub12:45
mnasiadkaso we should push xena-rc1 to quay/dockerhub12:45
yoctozeptoI think you misunderstood me12:45
yoctozeptobut anyway12:45
mnasiadkaso tell us again12:45
mnasiadka:)12:45
yoctozeptothe most obvious issue with releasing is we need to wait for mgoddard12:46
yoctozeptootoh, it's already Friday afternoon :-)12:46
mnasiadkaYeah, so let's speak Monday12:46
mnasiadkalet master be broken :D12:46
hrwah, shit. PTG is next week12:46
yoctozeptoyeah, duh12:47
hrwhave to remember that while arranging meetings12:47
hrwI am buying flat next week.12:47
yoctozeptomnasiadka, hrw12:47
yoctozeptoI was talking12:47
yoctozepto            tag: "{{ (zuul.tag if zuul.pipeline == 'release' else zuul.branch | basename) ~ tag_suffix }}"12:47
yoctozeptothis is what we do on release12:47
yoctozeptowe grab the release tag12:47
yoctozeptoand publish such images12:47
yoctozeptothis will fail12:47
mnasiadkaand publish once images with numeric version tag?12:47
yoctozeptoI guess we can disable it12:48
mnasiadkathis is some madness12:48
yoctozeptomnasiadka: yeah, it is12:48
hrw- tag: "{{ (zuul.tag if zuul.pipeline == 'release' else zuul.branch | basename) ~ tag_suffix }}"12:48
hrw+ tag: "do-not-use-may-kill-kittens"12:48
yoctozeptolol12:48
hrwand then revert12:48
yoctozeptowe can remove the jobs from release12:48
yoctozeptoI don't think they are worth it12:48
yoctozeptowe have periodics for these12:48
yoctozeptowith the usual branch tag12:49
hrwthere are ways12:49
yoctozeptook so12:49
yoctozeptoI prepare a patch12:49
hrwok, so which patches we need to get INTO wallaby?12:49
yoctozeptothat we do not publish release images12:49
yoctozeptothis gets merged12:49
yoctozeptoI propose to release kolla12:49
yoctozepto(do we want k-a too already?)12:49
hrwsomething outside of https://review.opendev.org/q/topic:%22prepare-for-kolla-wallaby-release%22+(status:open%20OR%20status:merged) ones?12:49
yoctozeptoand let mgoddard accept it12:49
yoctozeptowdyt?12:50
hrwbranch, do not release images? +712:50
yoctozeptothen we can resume xena testing on master properly (with upgrades from wallaby)12:50
yoctozeptohrw: superb12:50
yoctozeptoand what mnasiadka thinks12:50
yoctozeptohrw, mnasiadka: could you also double-check we never reference the versioned images?12:51
yoctozeptoI will handle the patching in a bit12:51
yoctozeptobbl12:51
*** e0ne has joined #openstack-kolla12:57
hrwok, so https://review.opendev.org/c/openstack/kolla/+/785500 from yoctozepto or https://review.opendev.org/c/openstack/kolla/+/786620 from mnasiadka?13:04
*** shyamb has quit IRC13:04
hrwor both?13:04
*** LinPeiWen has quit IRC13:05
hrwlooks like yoctozepto's13:06
yoctozeptoneither because we want to branch first, no?13:06
yoctozeptoI am confused by your question13:06
hrwso am I13:06
hrwsorry, too many things at once when I should just drop all of them13:07
yoctozeptowhen you are unsure what to drop...13:08
yoctozeptodrop the bass!13:08
* hrw scrolls 2 hours to reread13:08
hrwok. so let's branch13:11
*** samcat116 has joined #openstack-kolla13:26
*** kukacz has quit IRC13:47
*** skramaja has quit IRC13:56
ozzzoI successfully upgraded my test kolla cluster from Train->Ussuri, but VM build fails now, with rados.Rados.connect\nrados.PermissionDeniedError13:59
ozzzohttps://paste.ubuntu.com/p/bVNVDRhDvB/13:59
ozzzoDo I need to do something with ceph after upgrading?13:59
ozzzoI'm running external ceph13:59
*** e0ne has quit IRC14:03
*** sorin-mihai has joined #openstack-kolla14:05
yoctozeptoozzzo: no, but perhaps the ceph change has affected your config14:06
ozzzoI didn't upgrade ceph yet; I was planning to do that later14:07
ozzzoI'm on nautilus14:07
yoctozeptoozzzo: no, I mean change to ceph config that happened kolla-ansible-side14:08
yoctozeptoas we got rid of internal ceph14:08
*** brinzhang_ has quit IRC14:08
*** brinzhang_ has joined #openstack-kolla14:09
ozzzooic14:09
ozzzodo I need to change something to compensate?14:10
ozzzothe ceph cluster is healthy; it seems like I'm missing a password or key, or something14:10
yoctozeptoI think we wrote this in the release notes14:11
yoctozeptohttps://docs.openstack.org/releasenotes/kolla-ansible/ussuri.html#relnotes-10-0-0-stable-ussuri-upgrade-notes14:13
yoctozeptoyou might be interested in two, namely:14:13
yoctozepto"For cinder (cinder-volume and cinder-backup), glance-api and manila keyrings behavior has changed and Kolla Ansible deployment will not copy those keys using wildcards (ceph.*), instead will use newly introduced variables. Your environment may render unusable after an upgrade if your keys in /etc/kolla/config do not match default values for14:13
yoctozeptointroduced variables."14:13
yoctozeptooh, even three14:13
yoctozepto"The default behavior for generating the cinder.conf template has changed. An rbd-1 section will be generated when external Ceph functionality is used, i.e. cinder_backend_ceph is set to true. Previously it was only included when Kolla Ansible internal Ceph deployment mechanism was used."14:13
yoctozepto"The rbd section of nova.conf for nova-compute is now generated when nova_backend is set to "rbd". Previously it was only generated when both enable_ceph was "yes" and nova_backend was set to "rbd"."14:14
ozzzook reading14:15
*** rpittau is now known as rpittau|afk14:18
*** kevko has quit IRC14:18
ozzzoI have cinder_backend_ceph: "yes" but I don't have nova_backend14:19
ozzzoI have nova_backend_ceph: "yes"14:19
ozzzoDo I need to set nova_backend?14:19
*** macz_ has joined #openstack-kolla14:20
ozzzoit looks like my enable_ceph is defaulted to no: #enable_ceph: "no"14:21
ozzzoI was looking at https://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html last night and I think I have everything setup according to that. I didn't find anything there that needed to be changed14:23
*** macz_ has quit IRC14:24
*** kevko has joined #openstack-kolla14:24
ozzzoit looks like I'm missing the auth_*_required values from /etc/kolla/config/glance/ceph.conf - I'll try adding those14:26
yoctozeptothat should help, yes14:32
openstackgerritWill Szumski proposed openstack/kolla-ansible stable/stein: Switch octavia-api to wsgi running under apache.  https://review.opendev.org/c/openstack/kolla-ansible/+/78666514:40
*** bengates has quit IRC14:44
*** bengates has joined #openstack-kolla14:45
ozzzoyoctozepto: that didn't seem to help; I changed that and deployed successfully but I still get the nrados.PermissionDeniedError error when I build a VM14:47
*** bengates has quit IRC14:50
ozzzocould I be hitting this isssue? In the meantime, please ensure that, before upgrading to Ussuri, the backend_host option is set to its previous value (the default was rbd:volumes) via a config override.14:50
ozzzoMy backend_host is set to rbd:volumes14:50
openstackgerritWill Szumski proposed openstack/kolla-ansible stable/stein: Switch octavia-api to wsgi running under apache.  https://review.opendev.org/c/openstack/kolla-ansible/+/78666914:52
ozzzodo I need the auth_*_required values in other ceph.conf files or only the glance one?14:54
openstackgerritWill Szumski proposed openstack/kolla-ansible stable/stein: Switch octavia-api to wsgi running under apache.  https://review.opendev.org/c/openstack/kolla-ansible/+/78666915:17
*** sorin-mihai has quit IRC15:20
*** kukacz has joined #openstack-kolla15:23
*** bengates has joined #openstack-kolla15:31
ozzzoI added the auth_*_required values to all of the ceph.conf files but that didn't make a difference15:58
ozzzois it possible to add them after upgrading, or do I need to start over and add them before the upgrade?15:58
*** kevko has quit IRC16:02
*** kevko has joined #openstack-kolla16:04
*** bengates has quit IRC16:11
*** k_mouza has quit IRC16:12
*** bengates has joined #openstack-kolla16:12
*** k_mouza has joined #openstack-kolla16:12
*** e0ne has joined #openstack-kolla16:12
*** bengates has quit IRC16:17
*** k_mouza has quit IRC16:17
*** cah_link has quit IRC16:27
*** k_mouza has joined #openstack-kolla16:38
*** muhaha has joined #openstack-kolla16:53
muhahaguys, what hw do i need for single node installation ?16:53
*** k_mouza has quit IRC17:04
priteaumuhaha: for an all-in-one node? depends how many services you want to enable. you probably need 4g of ram at least17:16
muhahapriteau i just want to test nested kubernetes cluster with openstack api integration17:17
priteauYou will need to launch actual VMs then. Depends how many you need and how much memory is needed for each VM17:21
*** e0ne has quit IRC17:21
muhahaIs there any example how to run all-in-one installation ? I am not sure about storage...17:51
*** vishalmanchanda has quit IRC18:04
*** andrewbonney has quit IRC18:05
*** gfidente is now known as gfidente|afk18:21
*** k_mouza has joined #openstack-kolla18:25
*** k_mouza has quit IRC18:30
*** mushrushu has quit IRC18:32
*** mushrushu has joined #openstack-kolla18:32
muhahaIs 2 NIC hard requirement for installing openstack kolla on single node ?18:36
ozzzoyou can use a dummy interface for the 2nd18:44
ozzzomuhaha: this is the document I used for my single node: https://docs.openstack.org/kolla-ansible/latest/user/quickstart.html18:47
muhahahm, thanks, is there any ansible kolla container ? i guess that i dont need to install these dependencies on actual node, right ?19:00
ozzzoi installed it on a VM19:09
ozzzoI had to change nova_compute_virt_type from kvm to qemu; besides that it worked pretty much as described in the doc19:11
*** ozzzo has quit IRC19:17
muhahathanks, it should be possible to run these steps in container and use external ip for all-in-one host ... ( of course dockerd and python docked should be installed on target host )19:18
*** ozzzo has joined #openstack-kolla19:27
*** dciabrin_ has joined #openstack-kolla19:40
*** dciabrin has quit IRC19:42
openstackgerritRadosław Piliszek proposed openstack/kolla master: Stop publishing on release  https://review.opendev.org/c/openstack/kolla/+/78671619:44
yoctozeptomnasiadka hrw https://review.opendev.org/c/openstack/kolla/+/78671619:49
openstackgerritRadosław Piliszek proposed openstack/kolla master: Stop publishing on release  https://review.opendev.org/c/openstack/kolla/+/78671619:51
openstackgerritRadosław Piliszek proposed openstack/kolla master: [CI] Drop support for publishing versioned image tags  https://review.opendev.org/c/openstack/kolla/+/78671919:54
*** k_mouza has joined #openstack-kolla20:14
*** k_mouza has quit IRC20:18
*** samcat116 has quit IRC20:25
hrwyoctozepto: iirc you are in masakari team nowadays... go to http://osbpo.debian.net/deb-status/ and switch to bullseye-wallaby. then search for masakari.20:26
hrwyoctozepto: other package names than ubuntu uses so images need to be adapted20:26
hrwyoctozepto: then we can enable them back20:26
*** muhaha has quit IRC20:31
*** zzzeek has quit IRC20:38
*** zzzeek has joined #openstack-kolla20:44
*** admin0 has quit IRC21:16
*** luksky has quit IRC21:39
*** fuhrmannb has quit IRC23:41
*** ysirndjuro has quit IRC23:55
« Thursday, 2021-04-15 Index Saturday, 2021-04-17 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!