Thursday, 2021-04-15

*** bengates has joined #openstack-kolla01:38
*** bengates has quit IRC01:43
*** ivan_lin has quit IRC01:47
*** ivan_lin has joined #openstack-kolla01:48
*** cosmicsound has quit IRC01:55
*** cosmicsound has joined #openstack-kolla01:55
*** oyrogerg has quit IRC01:57
*** oyrogerg has joined #openstack-kolla01:57
*** brinzhang_ is now known as brinzhang02:06
openstackgerritJames Kirsch proposed openstack/kolla master: Add LetsEncrypt images for cert request/renewal  https://review.opendev.org/c/openstack/kolla/+/74133902:59
*** vishalmanchanda has joined #openstack-kolla04:23
*** skramaja has joined #openstack-kolla04:28
*** brinzhang_ has joined #openstack-kolla04:45
*** brinzhang has quit IRC04:48
*** ivan_lin has quit IRC05:16
*** ivan_lin has joined #openstack-kolla05:17
*** ivan_lin has quit IRC05:25
*** ivan_lin has joined #openstack-kolla05:26
*** cah_link has joined #openstack-kolla06:55
*** bengates has joined #openstack-kolla07:03
*** jaicaa has quit IRC07:10
*** jaicaa has joined #openstack-kolla07:13
*** bengates has quit IRC07:14
*** bengates has joined #openstack-kolla07:15
*** bengates has quit IRC07:16
*** bengates has joined #openstack-kolla07:17
*** bengates has quit IRC07:18
*** bengates has joined #openstack-kolla07:19
*** wuchunyang has joined #openstack-kolla07:20
*** andrewbonney has joined #openstack-kolla07:24
*** luksky has joined #openstack-kolla07:34
yoctozeptomorning07:38
*** rpittau|afk is now known as rpittau07:45
*** jovial[m] has joined #openstack-kolla07:58
mnasiadkamorning08:11
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: mariadb: Deprecate wsrep-notify.sh  https://review.opendev.org/c/openstack/kolla-ansible/+/78624508:12
*** fuhrmannb has joined #openstack-kolla08:16
mnasiadkahmm, seems the UCA Wallaby upgrade job failed because floating ip doesn't work, but Neutron claims everything is fine :)08:26
*** luksky has quit IRC08:31
*** luksky has joined #openstack-kolla08:31
mnasiadkaalthough neutron-l3-agent log claims it was applied and working08:35
mnasiadkaso I will need to reproduce it locally...08:35
*** luksky has quit IRC08:42
hrwelo08:48
*** gfidente|afk is now known as gfidente08:53
*** luksky has joined #openstack-kolla08:55
mnasiadkahrw: how's the Bullseye patch testing on aarch? I think yesterday you haven't pulled in k-a with the wsrep-notify patch?09:00
hrwmnasiadka: rabbitmq is crashing all the time09:02
hrwmnasiadka: so mariadb installs, several other ones too and then it reaches nova and fails due to no contact with rmq09:03
*** gfidente has quit IRC09:14
*** gfidente has joined #openstack-kolla09:15
mnasiadkahrw: why we are downgrading rabbitmq?09:21
mnasiadkahrw: 3.8.14 in Victoria and 3.8.12 in Wallaby?09:21
mnasiadkaStarting RabbitMQ 3.8.14 on Erlang 23.3.1 vs Starting RabbitMQ 3.8.12 on Erlang 23.3.109:22
*** gfidente has quit IRC09:24
hrwwhich distro/arch?09:25
*** vishalmanchanda has quit IRC09:27
mnasiadkaubuntu/x8609:27
hrwno idea09:28
hrwcentos and debian grab 3.8.1409:28
*** ivve has joined #openstack-kolla09:29
*** k_mouza has joined #openstack-kolla09:29
*** kevko_ has quit IRC09:30
*** kevko has joined #openstack-kolla09:30
kevkomorning \o/09:30
hrwmnasiadka: ok. it is upstream decision - they say '3.8.12' in focal Packages09:30
hrwhttps://dl.bintray.com/rabbitmq/debian/dists/focal/main/binary-amd64/:Packages09:31
mnasiadkahmm, rabbitmq website points to packagecloud, not to bintray09:33
mnasiadkaand still, we have focal in Victoria, so why there's 3.8.14? :)09:33
hrwlet me run a build09:37
hrwvictoria uses bionic repo09:39
hrwduring W we moved to use focal repo as someone mentioned that it exists now09:39
mnasiadkayeah, but it seems it gets older package09:40
hrwhttps://review.opendev.org/c/openstack/kolla/+/78511409:40
mnasiadkaAnd we could move to packagecloud, because google says bintray is getting shut down ;)09:40
hrwmnasiadka: how it went? ah, 'patches welcome'09:41
hrw;D09:41
mnasiadkayeah, I'll look into it today ;)09:41
hrwand we need to merge it before 'switch to new way of APT key/repo' patch so it will be easy to backport09:41
hrwmnasiadka: if possible please avoid using apt-key. fetching gpg key file is recommended09:44
*** vishalmanchanda has joined #openstack-kolla09:48
mnasiadkahrw: what is funny we use packagecloud already for non x86 rabbitmq repos09:54
hrwI see only bintray09:55
mnasiadkafor centos yes, for ubuntu we we have packagecloud in sources.list.ubuntu.aarch6409:59
hrwmnasiadka: commented out10:02
hrwas we do not use entries from there but from repos.yaml10:02
hrwI have a patch which drops sources.list.* files10:02
hrwbut it is on pile of other patches so wait for them to be merged first10:03
mnasiadkaah right10:04
*** wuchunyang has quit IRC10:11
*** gfidente has joined #openstack-kolla10:14
*** e0ne has joined #openstack-kolla10:24
hrwkolla-ansible-ubuntu-source-upgrade https://zuul.opendev.org/t/openstack/build/6403c9d1f7e8446bba9554a62572dd9b : SUCCESS in 58m 56s10:25
hrwUCA +2?10:25
mnasiadkaincredible10:26
hrwmnasiadka: give it +W before it fails10:26
mnasiadka+w and off we go10:26
hrw;D10:29
hrwthat whole fancy zuul build interface drives me mad10:32
*** e0ne has quit IRC10:37
*** gfidente has quit IRC10:56
*** k_mouza has quit IRC10:58
yoctozeptoit works for me10:58
hrwmaybe I should restart firefox more often than once per month10:59
*** gfidente has joined #openstack-kolla11:15
*** k_mouza has joined #openstack-kolla11:15
*** k_mouza has quit IRC11:15
*** k_mouza has joined #openstack-kolla11:16
openstackgerritMichal Nasiadka proposed openstack/kolla master: rabbitmq: Move to packagecloud  https://review.opendev.org/c/openstack/kolla/+/78639011:17
hrwhttps://review.opendev.org/c/openstack/kolla-ansible/+/783887 - opinions?11:18
mnasiadkawell, from one perspective it's nice, from another we're in feature freeze?11:20
hrwI meant opinion, now +w ;D11:20
*** shyamb has joined #openstack-kolla11:27
*** shyam89 has joined #openstack-kolla11:27
*** jovial[m] has quit IRC11:30
*** shyam89 has quit IRC11:43
*** shyamb has quit IRC11:43
*** k_mouza has quit IRC11:43
*** k_mouza has joined #openstack-kolla11:44
*** jovial[m] has joined #openstack-kolla11:54
*** jovial[m] has quit IRC11:59
*** jovial[m] has joined #openstack-kolla12:01
*** e0ne has joined #openstack-kolla12:08
*** shyamb has joined #openstack-kolla12:14
mnasiadkahrw: and gate failed12:19
hrwmeh12:20
openstackgerritMichal Nasiadka proposed openstack/kolla master: rabbitmq: Move to packagecloud  https://review.opendev.org/c/openstack/kolla/+/78639012:25
*** shyamb has quit IRC12:25
hrw+1 RP+1 +comment12:30
openstackgerritMichal Nasiadka proposed openstack/kolla master: rabbitmq: Move to packagecloud  https://review.opendev.org/c/openstack/kolla/+/78639012:32
mnasiadkaupdated ;)12:32
hrw+2 rp+1, waiting for zuul12:34
hrwand anther comment12:37
hrwso far bullseye erlang is newer but it may change12:38
hrwwill send comment once check12:38
hrwdone12:43
*** shyamb has joined #openstack-kolla12:50
*** ivve has quit IRC12:58
*** e0ne has quit IRC13:24
*** e0ne has joined #openstack-kolla13:26
*** e0ne has quit IRC13:26
*** samcat116 has joined #openstack-kolla13:27
*** e0ne has joined #openstack-kolla13:28
*** shyam89 has joined #openstack-kolla13:38
hrwmnasiadka: will push update to packagecloud patch. jsut need to build test13:38
*** shyam89 has quit IRC13:38
*** shyamb has quit IRC13:39
mnasiadkahrw: sure, feel free13:40
hrwjust informing ;D13:41
hrwmnasiadka: it should also fix ubuntu/aarch6413:41
yoctozeptofyi, gerrit->launchpad works again (big thanks to fungi)13:47
mnasiadkanice13:49
*** ivan_lin has quit IRC13:49
hrwmnasiadka: hm. that erlang repo provides just a bunch of erlang packages while debian/ubuntu have more13:54
mnasiadkahrw: well, does rabbitmq need that more?13:55
hrwmnasiadka: https://www.rabbitmq.com/install-debian.html mentions using ppa for erlang.13:59
mnasiadkahrw: but ppa is over http - so that's why I went for the other one - I doubt it has more packages?13:59
hrwmnasiadka: there are two different ways of building/packging erlang for debian/ubuntu14:00
hrwone is 'create few packages and hope for best', second is 'just reuse debian packaging'14:00
mnasiadkahrw: if erlang versions fit rabbitmq requirements - we might just use what ubuntu/debian provides14:00
mnasiadkaI just assumed we want to mimick centos - but maybe we don't14:01
*** e0ne has quit IRC14:02
*** e0ne has joined #openstack-kolla14:03
hrwbbl - meeting14:04
*** k_mouza has quit IRC14:05
*** k_mouza_ has joined #openstack-kolla14:06
yoctozeptomnasiadka: nice? it restores my hope in managing the bugs!14:47
mnasiadkayoctozepto: all hope is lost14:48
yoctozeptolol14:48
mnasiadkayoctozepto: mark all as invalid and wait if people will claim these are still needed :D14:48
yoctozeptoyeah, we can probably do this for old and hard to understand ones...14:48
mnasiadkawe also have some really old reviews, probably need some cleaning if we care14:49
yoctozeptoside-question: did stackhpc try out cyborg?14:49
yoctozeptomnasiadka: yeah, we clean up with Mark from time to time14:49
mnasiadkaI don't think so, we mainly use PCI pass through14:49
mnasiadkaif we're talking GPU14:50
mnasiadkabut I'm not the resident expert ;)14:50
*** ysirndjuro has joined #openstack-kolla15:07
openstackgerritMichal Nasiadka proposed openstack/kolla master: move to Debian 'bullseye'  https://review.opendev.org/c/openstack/kolla/+/77247915:09
mnasiadkahrw: now it should pass aarch6415:16
hrwbtw 'check arm64' is magic incantation to recheck on aarch64 only15:26
yoctozepto++15:28
openstackgerritMichal Nasiadka proposed openstack/kolla master: move to Debian 'bullseye'  https://review.opendev.org/c/openstack/kolla/+/77247915:33
*** bengates has quit IRC15:38
*** bengates has joined #openstack-kolla16:02
*** bengates has quit IRC16:06
ozzzoworking on my train->ussuri upgrade, I'm getting a rate limit error16:08
ozzzoInternal Server Error (\"toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\")\\n'"}16:08
ozzzowhat's the best way to work around that?16:08
ozzzoif I create an account, can I enter the account info somewhere in my config and use it for the upgrade?16:09
*** rpittau is now known as rpittau|afk16:09
mnasiadkaset up your own registry and build+push images into it?16:09
ozzzois there a document on that?16:12
mnasiadkaozzzo: https://docs.openstack.org/kolla-ansible/latest/user/multinode.html#option-1-local-registry16:15
mnasiadkaozzzo: of course you need to use kolla to build images and push them into that registry16:15
ozzzook ty16:16
mnasiadkaozzzo: https://docs.openstack.org/kolla/latest/admin/image-building.html (how to build)16:16
mnasiadkareplace latest with your release - I guess ussuri16:16
ozzzook16:17
hrwozzzo: use some tag on your images so you will know which one you deployed. handy when new nodes are added16:23
*** samcat116 has quit IRC16:26
hrwozzzo: Linaro Developer Cloud uses ussuri-ldc-BUILDNUMBER for example16:26
*** samcat116 has joined #openstack-kolla16:28
yoctozepto^ using different tags per build ensure you can easily rollback16:31
yoctozeptoensures*16:31
yoctozeptoshould you need it16:32
hrwyoctozepto: https://review.opendev.org/c/openstack/kolla/+/78639016:36
*** e0ne has quit IRC16:37
yoctozeptohrw: cloudsmith?16:37
hrwyoctozepto: mnasiadka's patch16:39
*** samcat116 has quit IRC16:42
*** k_mouza_ has quit IRC16:43
*** samcat116 has joined #openstack-kolla16:44
*** k_mouza has joined #openstack-kolla16:44
yoctozeptohrw: right16:45
yoctozeptomnasiadka: cloudsmith?16:45
yoctozeptoand erlang gone versionless16:45
*** k_mouza has quit IRC16:51
mnasiadkayoctozepto: cloudsmith - on rabbitmq webpage it says to use either ppa erlang repo or cloudsmith16:56
mnasiadkayoctozepto: ppa is http only, I tend to have http allergy16:56
yoctozeptowe need gpg over https16:56
yoctozeptoand then I don't really care16:57
yoctozeptonow it's a mix16:57
yoctozeptoanyhow16:57
yoctozeptoI understand we need to backport this16:57
yoctozeptoto all the branches...16:57
yoctozeptowhat about erlang pin? it gets removed16:57
mnasiadkayoctozepto: but hrw pointed we could use in-distro for ubu/deb16:57
*** gfidente is now known as gfidente|afk16:57
*** jonaspaulo has joined #openstack-kolla16:57
mnasiadkaWe can put it back16:57
yoctozeptooh my, switching back to distro-provided?16:58
* yoctozepto 's head spins16:58
ozzzook I'll read up on tags, ty16:59
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134017:01
*** cah_link has quit IRC17:22
*** zul has joined #openstack-kolla17:25
*** andrewbonney has quit IRC17:50
*** bsanjeewa has joined #openstack-kolla18:09
*** jonaspaulo has quit IRC18:09
bsanjeewasubscribe18:16
* yoctozepto knows nothing about a subscription service around18:18
*** vishalmanchanda has quit IRC18:27
*** skramaja has quit IRC18:28
*** bsanjeewa_ has joined #openstack-kolla18:30
*** bsanjeewa has quit IRC18:33
*** e0ne has joined #openstack-kolla19:03
*** e0ne has quit IRC19:04
*** e0ne has joined #openstack-kolla19:05
*** e0ne has quit IRC19:24
*** dave-mccowan has quit IRC19:43
*** dave-mccowan has joined #openstack-kolla19:45
*** bsanjeewa_ has quit IRC19:50
*** macz_ has joined #openstack-kolla19:55
*** k_mouza has joined #openstack-kolla20:45
*** k_mouza has quit IRC20:49
ozzzoI successfully upgraded my test kolla cluster from Train->Ussuri, but VM build fails now, with rados.Rados.connect\nrados.PermissionDeniedError21:01
ozzzohttps://paste.ubuntu.com/p/bVNVDRhDvB/21:02
*** jbadiapa has quit IRC21:02
*** samcat116 has quit IRC21:28
ozzzolooking at this document: https://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html21:35
ozzzoit looks like I should have a ceph_glance_keyring setting in globals.yml21:36
ozzzobut I didn't have that in Train and it was working. Maybe that is a new requirement in Ussuri?21:36
ozzzoI have /etc/kolla/config/glance/ceph.client.glance.keyring but no variable in globals.yml that points to it21:37
ozzzoer... no that's probably not it; it looks like that's the default21:38
*** zijlboot_ has joined #openstack-kolla22:42
*** zijlboot has quit IRC22:44
*** macz_ has quit IRC23:16
*** macz_ has joined #openstack-kolla23:20
*** macz_ has quit IRC23:25
*** luksky has quit IRC23:58

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!