Thursday, 2020-01-02

*** cloudnull has quit IRC00:16
*** cloudnull has joined #openstack-kolla00:18
*** tonythomas has quit IRC00:25
*** xinliang has joined #openstack-kolla00:40
aleccoderIs cloudkitty even still supported with kolla-ansible on train?01:52
*** xinliang has quit IRC01:54
*** xinliang has joined #openstack-kolla02:21
*** k_mouza has joined #openstack-kolla02:33
*** k_mouza has quit IRC02:38
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Fix deploy mariadb failed to reach primary node  https://review.opendev.org/69917202:45
*** k_mouza has joined #openstack-kolla03:05
*** k_mouza has quit IRC03:30
*** xinliang has quit IRC04:16
*** skramaja has joined #openstack-kolla04:49
*** zainub_wahid has joined #openstack-kolla05:15
*** zainub_wahid has joined #openstack-kolla05:21
*** goldyfruit has quit IRC05:24
*** goldyfruit has joined #openstack-kolla05:25
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-kolla05:33
*** shyamb has joined #openstack-kolla06:17
*** mrunge has quit IRC06:19
*** mrunge has joined #openstack-kolla06:24
zainub_wahidWhen is kolla-ansible 9.2.0 release expected? will there be 9.1.0 release first?06:52
hrwzainub_wahid: is there a rush for new release?07:02
osmanlicilegimorning07:25
*** shyamb has quit IRC07:26
hrwosmanlicilegi: hi Dincer07:26
*** jbadiapa has joined #openstack-kolla07:26
osmanlicilegihrw: how's going?07:27
osmanlicilegiI hope everybody had a good rest :]07:27
hrwosmanlicilegi: so far, so good07:27
hrwlooking now how our build works once we disable epel ;D07:30
*** cah_link has joined #openstack-kolla07:32
*** pcaruana has joined #openstack-kolla07:34
osmanlicilegihttps://review.opendev.org/#/c/699333/ is ready for reviews.07:35
patchbotpatch 699333 - kolla-ansible - Adds maximum supported version option for Ansible - 7 patch sets07:35
*** shyamb has joined #openstack-kolla07:36
*** jbadiapa has quit IRC07:38
hrwcommented07:40
osmanlicilegigot it07:42
osmanlicilegiah I see I've fixed it but didn't pushed07:45
hrw;D07:46
openstackgerritDincer Celik proposed openstack/kolla-ansible master: Adds maximum supported version option for Ansible  https://review.opendev.org/69933307:46
hrw+207:47
osmanlicilegithanks!07:47
yoctozeptomorning07:47
*** shyamb has quit IRC07:48
hrwyoctozepto: hi07:49
yoctozeptohi hrw07:49
hrwyoctozepto: you planned to disable epel, right?07:49
yoctozeptohrw: yeah, more like "wished to"07:49
hrwyoctozepto: https://paste.centos.org/view/1c42e39107:50
yoctozeptocurrently producing bandanas with "kill epel" emblem07:50
yoctozeptothis patch is evil07:50
hrwyoctozepto: it shows base for getting it done properly07:51
yoctozeptohrw: but it shows adding epel :D07:51
hrwyoctozepto: kill scsi-target-utils first?07:52
yoctozeptohrw: mandatory07:53
hrwyoctozepto: https://review.opendev.org/#/c/613815/ probably07:54
patchbotpatch 613815 - kolla - Remove tgtd and scsi-target-utils support in CentO... (ABANDONED) - 12 patch sets07:54
yoctozeptohrw: the issue is k-a07:54
yoctozeptoand general migration from tgtd to lio07:55
hrwin other words: not going to happen07:56
hrwmove to centos8 will just drop it07:56
yoctozeptohrw: most likely, unfortunately07:57
yoctozeptoI think epel8 still has it?07:57
yoctozeptoanyways, since we are forcing users to migrate to c8 on host07:57
yoctozeptothe current tgtd have to go anyway07:57
hrw()[root@3d033af0a58d yum.repos.d]# dnf info scsi-target-utils08:01
hrwLast metadata expiration check: 0:02:41 ago on Thu 02 Jan 2020 07:58:59 AM UTC.08:01
hrwError: No matching Packages to list08:01
hrwdoes not have08:01
yoctozeptohrw: so it's forced and we are done with epel by means of c808:08
*** numans has quit IRC08:25
*** numans has joined #openstack-kolla08:26
*** jistr_ is now known as jistr08:29
mnasiadkamorning08:38
*** rpittau|afk is now known as rpittau08:41
hrwhi mnasiadka08:44
*** shyamb has joined #openstack-kolla08:44
* hrw just dropped enable_repos 08:44
mnasiadkahi hrw08:44
mnasiadkatrying to learn typing on a keyboard after xmas && new year break ;)08:44
hrwspeaking of keyboard... during visit in Castorama I took another ms ergonomic 4000 keyboard from electrowaste basket08:45
hrwI do not care does it work. keys are like not used!08:45
hrwkolla.tests.test_methods.MethodsTest.test_disable_repos_centos  0.01908:47
hrw;D08:47
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Fix "deploy mariadb failed to reach primary node" - short edition  https://review.opendev.org/70087508:48
*** shyamb has quit IRC08:49
*** shyamb has joined #openstack-kolla08:50
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: CentOS IPv6  https://review.opendev.org/69181108:51
hrwyoctozepto: https://paste.centos.org/view/4b4fbe41 should look better08:52
yoctozeptohrw: nice08:53
*** shyam89 has joined #openstack-kolla08:54
*** shyamb has quit IRC08:54
mnasiadkawhy do we need epel at all? :)08:55
hrwmnasiadka: scsi-target-utils?08:59
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: centos: disable EPEL by default  https://review.opendev.org/70087708:59
mnasiadkahrw: that is tgtd I assume?09:00
hrwmnasiadka: base09:00
hrwcommit 184f01a0d645d58fd4c9b6c5004e94fe98a077f809:03
hrwthis added scsi-target-utils into base09:04
hrwhttps://review.opendev.org/#/c/291285/09:05
patchbotpatch 291285 - kolla - Adds iscsid and tgtd docker containers (MERGED) - 55 patch sets09:05
mnasiadkayeah, I guess09:05
mnasiadkabut still I remember me and yoctozepto wanted to get rid of tgtd in favor of lio09:05
yoctozeptoyou remember well, mnasiadka09:06
yoctozeptoc8 simply forces this on us09:06
yoctozeptoas it's nowhere there09:07
mnasiadkayeah, so why hrw is wasting time on this? :)09:07
yoctozeptoand users are forced anyway to turn off their target09:07
yoctozeptofor the c8 upgrade09:07
yoctozeptomnasiadka: I like the disable stuff anyway09:07
yoctozeptocould come in handy with something else09:07
hrwmnasiadka: software archeology09:07
hrwyoctozepto: collectd uses it to disable extras and opstools09:09
yoctozeptohrw: for example :D09:09
* hrw votes from drop of tripleo7 due to 202009:29
hrws/from/for09:29
hrw;)09:29
*** k_mouza has joined #openstack-kolla09:31
hrwhm. should login to aarch64 host, build/upload set of images and then write CI job to make multiarch images09:36
yoctozeptohrw: yeah, we be dropping09:36
hrwwhich will require adding --list-buildable-images to kolla-build first09:36
yoctozeptowe can have a job running after two other jobs I guess but not sure09:37
yoctozeptobest ask infra09:37
hrwI got help already09:38
hrwjust have to sit and write09:38
hrwhttps://review.opendev.org/695168 needs brave core09:39
patchbotpatch 695168 - kolla-ansible - Get rid of Python 2 support - 12 patch sets09:39
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Get rid of Python 2 support  https://review.opendev.org/69131609:42
hrwhttps://bugs.launchpad.net/tripleo/+bug/185667809:43
openstackLaunchpad bug 1856678 in tripleo "Use Python 3 in CentOS 7 overcloud image" [High,Triaged]09:43
hrwand no, I do not plan to dig into tripleo code09:46
hrwthey have multiple git repos09:46
hrwhttps://review.opendev.org/#/c/699555/ - vote!09:48
patchbotpatch 699555 - tripleo-ci - [DNM] Run container build with python3 since train... - 2 patch sets09:48
mnasiadkahrw: well, it's DNM :)09:48
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Get rid of Python 2 support  https://review.opendev.org/69131609:55
hrwmnasiadka: soon will not be09:55
yoctozeptoxD09:57
hrwhttps://review.opendev.org/#/c/699555 - go for it.09:59
patchbotpatch 699555 - tripleo-ci - Run container build with python3 since train on Ce... - 3 patch sets09:59
*** k_mouza has quit IRC10:02
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs  https://review.opendev.org/67637610:12
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs  https://review.opendev.org/67637610:15
hrwmnasiadka: 192 revisions. nice10:20
mnasiadkayeah, and I'm not finished it seems ;)10:20
hrwmnasiadka: 256 is not that far away10:21
mnasiadkamight be :)10:21
mnasiadkaok, 193 on the way10:21
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs  https://review.opendev.org/67637610:22
yoctozeptoyeah, new year, new resolutions10:24
yoctozeptoin 2020 we accept only powers of 210:24
yoctozeptoas revision numbers10:24
yoctozeptoso it's either 256 or you work till 512 :-)10:24
*** k_mouza has joined #openstack-kolla10:25
zainub_wahidis kolla-ansible 9.1.0 release in pipeline? and when will 9.2.0 be released?10:38
hrwzainub_wahid: let me repeat question from few hours ago: is there a rush for new release?10:39
hrwzainub_wahid: any patch merged in meantime qualifies to make release?10:39
hrwzainub_wahid: or is it because few months passed since last release?10:40
*** shyam89 has quit IRC10:41
zainub_wahidi am working on Magnum and Fedora-CoreOS is acceptable in 9.2.0.10:43
hrwzainub_wahid: version of kolla* does not have to be in sync with other components10:44
hrwzainub_wahid: if you need magnum 9.2.0 then you may need git version of kolla stable/train10:45
hrwwe updated to use 9.2.0 magnum10:45
*** shyamb has joined #openstack-kolla10:58
*** tonythomas has joined #openstack-kolla10:58
hrwuf. --list-images can be used.10:59
zainub_wahidokay. Thank you11:05
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Fix MariaDB galera deployment on CentOS 7  https://review.opendev.org/69917211:09
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: CentOS IPv6  https://review.opendev.org/69181111:11
hrwhttps://hub.docker.com/repository/registry-1.docker.io/haerwu/debian-source-base/tags - preparations in progress11:18
*** k_mouza has quit IRC11:20
*** sean-k-mooney has joined #openstack-kolla11:26
*** k_mouza has joined #openstack-kolla11:27
hrwI think that we push too late in build process...11:34
hrwI am slowly considering adding --quiet switch to kolla11:37
hrwso all build logs go to files but are not printed11:37
*** sean-k-mooney has quit IRC11:43
*** sean-k-mooney has joined #openstack-kolla11:44
hrwINFO:kolla.common.utils:Attempt number: 1 to run task: BuildTask(zaqar-wsgi)11:50
hrwINFO:kolla.common.utils:Attempt number: 1 to run task: PushIntoQueueTask(PushTask(qinling-engine)=><queue.Queue object at 0x7f056119cad0>)11:50
hrwhm.11:50
hrwwhat for we say queue.Queue here? :D11:52
hrwINFO:kolla.common.utils:Added next task PushIntoQueueTask(PushTask(watcher-applier)) to queue11:52
hrwINFO:kolla.common.utils:Attempt number: 1 to run task: PushIntoQueueTask(PushTask(watcher-applier))11:52
hrwlooks better11:52
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add '--quiet' switch  https://review.opendev.org/70088811:59
hrwhttps://paste.centos.org/view/69202e0612:00
hrwthis is how --quiet looks12:00
*** kplant has joined #openstack-kolla12:05
*** shyam89 has joined #openstack-kolla12:07
*** shyamb has quit IRC12:07
hrwdo someone else has a use for it?12:10
*** shyam89 has quit IRC12:12
*** vmixor has joined #openstack-kolla12:14
osmanlicilegi--quiet makes sense to me.12:18
osmanlicilegiquick and good job hrw12:18
hrwosmanlicilegi: I like such small patches12:19
osmanlicileginow I need to backport it to my stein branch :D12:19
hrwosmanlicilegi: move quiet just above debug in common/config.py and should be cherry-pickable12:20
*** mgoddard has joined #openstack-kolla12:25
*** shyamb has joined #openstack-kolla12:26
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs  https://review.opendev.org/67637612:28
mgoddardHello Koalas, Happy New Year!12:29
mgoddardI had a little IRC trouble this morning, I think we're good now12:29
mnasiadkaGood afternoon mgoddard, happy new year :)12:30
hrwmgoddard: HNY!12:33
mgoddardthanks hrw and mnasiadka12:33
osmanlicilegikoalas love you mgoddard, happy new year!12:33
mgoddard:)12:33
hrwmgoddard: good news: tripleo works on getting rid of using py2 to run kolla12:33
mgoddardhrw: that is good news12:34
osmanlicilegibad news, they still didn't do anything for the rmq update12:34
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs  https://review.opendev.org/67637612:34
hrwand speaking of py2... we may need to write/backport some py2/py3 changes in few weeks. Ceph 14 (nautilus) will land in Debian:buster and be py3 only12:35
mnasiadkaosmanlicilegi: go complain on #rdo - it's not really tripleo that should do something about it :)12:35
hrwI do not remember how far we are with py3 ceph stuff12:35
mnasiadkahrw: we are dropping kolla-ceph this cycle.12:36
hrwmnasiadka: I worry about stein/train this time12:36
mgoddardhrw: debian already supports py312:36
mgoddardhrw: debian/binary since stein, debian/source since train12:36
mnasiadkahrw: ugh12:36
hrwmgoddard: so stein can have problem then12:37
hrwwill see12:37
mgoddardhrw: also, we won't upgrade ceph on stable branches12:37
hrwmgoddard: will pin then ;D12:37
hrweven easier12:37
mgoddardhrw: yes12:37
mgoddardno surprise major upgrades please :)12:37
hrw--quiet rulz ;D12:37
hrwI just need number of tasks now12:38
hrwINFO:kolla.common.utils:Attempt number: 1 to run task: PushTask(swift-base)12:38
hrwsomething like "INFO:kolla.common.utils:Attempt number: 1 to run task (53/401): PushTask(swift-base) "12:39
hrwbut it is not doable without changes as we build queue during build not before12:39
*** spsurya has joined #openstack-kolla12:48
*** zainub_wahid has quit IRC12:49
openstackgerritMerged openstack/kolla stable/stein: Bump OpenStack versions for Stein  https://review.opendev.org/70018413:01
*** shyamb has quit IRC13:10
*** skramaja has quit IRC13:16
*** k_mouza has quit IRC13:19
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs  https://review.opendev.org/67637613:28
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs  https://review.opendev.org/67637613:42
openstackgerritLEDUC Florian proposed openstack/kolla-ansible master: Add ability to use the log service plugins for Neutron  https://review.opendev.org/70089513:49
*** dave-mccowan has joined #openstack-kolla13:51
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs  https://review.opendev.org/67637613:59
*** k_mouza has joined #openstack-kolla14:02
mgoddardmnasiadka: you may as well go for #200 now14:07
mgoddardany takers for https://review.opendev.org/#/c/698445/?14:10
patchbotpatch 698445 - kolla - CentOS 8: work around mod_ssl certificate issue - 5 patch sets14:10
* hrw did14:20
*** dave-mccowan has quit IRC14:21
*** dave-mccowan has joined #openstack-kolla14:26
openstackgerritLEDUC Florian proposed openstack/kolla-ansible master: Add ability to use the log service_plugins for Neutron  https://review.opendev.org/70089514:27
openstackgerritMerged openstack/kolla-ansible master: Docs: remove some bad recommendations  https://review.opendev.org/69181414:28
yoctozeptoHappy New Year mgoddard :-) you back for good?14:47
mgoddardyoctozepto: indeed I am. I will never leave again14:47
mgoddardyoctozepto: did anything happen while we were away? I saw you were active a bit14:51
*** jaicaa has quit IRC14:52
*** jaicaa has joined #openstack-kolla14:53
mnasiadkamgoddard: that's my plan15:10
openstackgerritMark Goddard proposed openstack/kayobe master: Fix seed VM provision  https://review.opendev.org/70090415:22
*** KeithMnemonic has joined #openstack-kolla15:25
hrw(venv3) 15:30 (2s) hrw@j13-qrep-04:kolla$ time ./tools/create-multiarch-images.sh15:31
hrwINFO:kolla.common.utils:Found the docker image folder at /home/hrw/devel/kolla/kolla/docker15:31
hrwINFO:kolla.common.utils:Found the docker image folder at /home/hrw/devel/kolla/kolla/docker15:31
hrwCreated manifest list docker.io/test/test-aodh-api:master15:31
hrwcloser ;D15:31
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs  https://review.opendev.org/67637615:36
hrwso the goal will be: 1st CI publish job does :master-aarch64 images, 2nd does :master-x86_64 and 3rd does multiarch ones.15:47
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs  https://review.opendev.org/67637615:52
hrwcan docker retag image in remote registry without pull/push?15:52
yoctozeptomgoddard: glad you are back; yeah, I was active "a bit", keeping the channel alive; 99cloud (incl. Jeffrey4l) contributed completely set of ipv6 patches to get c7 on its feet15:54
hrwhm. maybe have solution15:54
mgoddardyoctozepto: I saw those. Always nice to remove some exceptions15:54
mnasiadkahrw: well, you can pull & push the manifest only, using REST - but I don't know if this is what you want to do15:54
yoctozeptomgoddard: by introducing exceptions :D15:54
mgoddardwell, true15:55
hrwmnasiadka: I can create manifest for one arch too ;D15:55
yoctozeptoanyways, the assumptions these workarounds make are sane15:55
yoctozeptoso it's worth having c7 with ipv6 working on train15:55
hrwmnasiadka: yay! #200 :D15:56
mnasiadkahrw: yeah, it's all mgoddard's fault - asked me to do something properly :D15:57
hrwhttps://hub.docker.com/repository/docker/haerwu/debian-source-aodh-api yay ;)15:57
yoctozeptomnasiadka: remember about 256/51215:57
mnasiadkayoctozepto: I think I'll go big, and try to get to 102415:57
mnasiadka;)15:57
yoctozeptomnasiadka: you are welcome too15:58
yoctozepto(and then we discover gerrit breaks after 1000)15:58
hrw16:58 (4s) hrw@puchatek:~$ docker manifest inspect haerwu/debian-source-aodh-api:master15:58
hrw16:58 (0s) hrw@puchatek:~$ docker manifest inspect haerwu/debian-source-aodh-api:master|grep archi15:59
hrw            "architecture": "arm64",15:59
hrw            "architecture": "amd64",15:59
hrweasy way to tag remote image? "docker manifest create NEWTAG OLDTAG;docker manifest push NEWTAG"16:01
*** pcaruana has quit IRC16:04
hrwhttps://review.opendev.org/700888 https://review.opendev.org/700877 both passed zuul16:08
patchbotpatch 700888 - kolla - Add '--quiet' switch - 1 patch set16:08
patchbotpatch 700877 - kolla - centos: disable EPEL by default - 1 patch set16:08
hrwhttps://review.opendev.org/695168 needs vote16:08
patchbotpatch 695168 - kolla-ansible - Get rid of Python 2 support - 12 patch sets16:08
hrwand no idea what to do with https://review.opendev.org/699361 one - push, add extra argument and push or abandon16:09
patchbotpatch 699361 - kolla - build: remove not buildable templates for --templa... - 1 patch set16:09
hrwhttps://review.opendev.org/694797 will probably rot in gerrit ;d16:09
patchbotpatch 694797 - kolla-ansible - Some ansible-lint fixes - 12 patch sets16:09
hrwhttps://paste.debian.net/1124164/ is whole tools/create-multiarch-images.sh ;d16:11
openstackgerritLEDUC Florian proposed openstack/kolla-ansible master: Enable libvirt multipath ISCSI in nova multipathd is enabled  https://review.opendev.org/70091016:12
openstackgerritLEDUC Florian proposed openstack/kolla-ansible master: Enable libvirt multipath ISCSI in nova multipathd is enabled  https://review.opendev.org/70091016:24
hrwmy favourite type of commit message16:27
hrw'how' instead of 'why'16:28
yoctozeptomnasiadka: https://review.opendev.org/69917216:28
patchbotpatch 699172 - kolla-ansible - Fix MariaDB galera deployment on CentOS 7 - 9 patch sets16:28
yoctozeptohrw: well, it's not that bad, that 'in' should actually be 'if'16:29
yoctozeptothen it's quite nice16:29
hrw+116:30
yoctozeptohrw: only message was nice16:32
yoctozepto-116:32
hrw;D16:34
* hrw out16:34
hrwhave fun16:34
*** cah_link has quit IRC16:40
*** pcaruana has joined #openstack-kolla16:43
yoctozeptomgoddard, mnasiadka: I think I have found out our mariadb upgrade issues reason16:45
yoctozeptowe should wait after each "slave" to regain quorum16:46
yoctozeptobecause in the failed ones the "master" declares it has been split off (as it lost connection to 2 others)16:46
yoctozeptoand we lose the cluster16:47
*** rpittau is now known as rpittau|afk16:51
yoctozeptoany ideas how to serialize this cleanly?16:58
mgoddardyoctozepto: as in start slaves seriallu?17:05
yoctozeptomgoddard: stop and start, yeah17:06
yoctozeptoI commented in my change17:06
mgoddardyoctozepto: maybe we just need 3 phases17:06
yoctozepto3 phases?17:07
mgoddardrather than master/slave, phase 1 2 317:07
yoctozeptoyeah, and if someone runs 5?17:07
mgoddardfor each phase: restart all nodes in phase, wait17:07
yoctozeptoit can be phased as long as you can guarantee a quorum17:07
mgoddardgroups['mariadb'].index(inventory_hostname) % 3 == phase17:08
mgoddardalternatively, use etcd to roll through changes17:09
mgoddardI keep thinking we should look into that17:09
mgoddardwithout somehow building k8s17:09
yoctozeptorestart kolla-kubernetes? xD17:10
mgoddardlol17:10
yoctozeptowe really only need this for stateful stuff like mariadb17:10
mgoddardyes, but if there was a general mechanism we could avoid downtime17:10
yoctozeptoeven rmq simply stops servicing requests when quorum is lost, not declaring everything lost and damned17:11
mgoddardneutron agents keep coming up17:11
yoctozeptomhm17:11
yoctozeptoyou are thinking big image17:11
mgoddardalways yoctozepto, always17:11
yoctozeptoI am stuck at getting mariadb not to break now and then :D17:11
yoctozeptogood ptl17:12
mgoddardI think 3 phases could work for mariadb, if that is the issue17:12
yoctozeptoyeah, that seems to make sense17:13
yoctozeptoit gets rid of this messy (and wrong) master-slave relation17:13
mgoddardit makes sense - if 'master' notices both slaves have gone then it will lose quorum17:13
yoctozeptoyeah, in the original choice of master we were lucky enough for it to notice one after the other17:14
yoctozeptomost of the time17:14
mgoddardalternatively, could we use serial: 30%?17:14
yoctozepto(with rare occurence of this failure)17:14
yoctozeptomgoddard: can't have serial outside of play17:14
mgoddardno17:14
mgoddardbut can on a play17:14
yoctozeptostill need to establish which mariadb are already deployed17:15
yoctozeptoand which are not17:15
mgoddardfor cases where the cluster is half up?17:15
yoctozeptofor cases when one is carefree and does a pull-deploy on half-deployed cluster, yeah17:16
yoctozeptoor has a change in config17:16
mgoddardhmm17:16
yoctozeptowith this ipv6 I kept mariadb from changing config unless necessary17:16
yoctozeptobecause it's so destructive17:16
mgoddardsure there were no whitespace changes?17:16
yoctozeptothe patch helps with deploying new mariadb17:16
yoctozeptomgoddard: checked, our merger is stable and removes all17:17
mgoddardcool17:17
yoctozeptoback to topic17:17
yoctozeptothe patch helps with deploying new mariadb17:17
yoctozeptobut it seems with replacing containers in CI17:17
yoctozeptoit makes the procedure even less stable17:17
yoctozeptoprobably due to different load pattern17:18
mgoddardsure it's not because the 'master' has changed?17:19
mgoddard:)17:19
yoctozeptonot by galera design17:19
yoctozeptothey are all master17:19
yoctozeptoif you compare logs between success and failure17:19
yoctozeptoit is missing the step of reducing quorum to 2/217:20
yoctozepto(and 1/1 which is easy with galera)17:20
yoctozeptoit sees immediate drop of two friends17:20
yoctozeptoand thinks it's no longer in primary view (the quorum)17:20
yoctozeptoas a homework you can try unplugging power coords from majority of mariadb hosts and check the result17:21
mgoddardso in the case of a semi-deployed cluster, you need to start up the undeployed nodes first to avoid losing quorum?17:22
mgoddardI will try in production now17:22
yoctozeptoyeah, it's easy17:22
mgoddardthen restart deployed nodes17:22
yoctozeptoI think I've got all the stuff sorted out in my head, now need to implement it, yeah17:22
mgoddardwhich means the groups['mariadb'] thing earlier wouldn't work17:23
yoctozeptoexactly17:23
yoctozeptobut it can be refined17:23
mgoddardtry adding a sleep in kolla_docker to see if the issue is 100% reproducible17:25
mgoddardwould be a nice demo17:25
yoctozeptosleep where?17:26
mgoddardrestart_container17:26
mgoddardafter stop, before start17:26
yoctozeptonot sure what that would demo out now17:27
yoctozeptothe problem is sudden stop of both friends, what to sleep for :-)17:27
mgoddardIIUC, it's racy because the containers normally start up again quickly, so the 'master' does not register them as down17:29
mgoddardif we sleep it will see them as down17:29
yoctozeptoIIUC is False17:30
yoctozeptoFalse -> whatever is True17:30
yoctozepto:D17:30
yoctozeptonah, the problem is that it somehow does not see the first one going down17:30
yoctozeptoand it does not reduce the quorum size17:30
yoctozeptoand instead immediately sees both going down17:30
yoctozeptoand goes: "oh noez, I am alone"17:31
yoctozepto"I17:31
yoctozepto"I'm fine but not PRIMARY"17:31
yoctozepto"let's hope I'm rejoined with my friends soon"17:31
yoctozeptothe others come back17:31
yoctozeptoand go17:31
yoctozepto"where is the cluster?"17:31
yoctozeptoand every other responds "not here"17:32
yoctozeptocheck mate17:32
mgoddardok, so in the working case17:32
mgoddardfirst node goes down17:32
mgoddardcluster is now 2 nodes17:32
mgoddardwhy does it not lose quorum when the second node goes down?17:33
*** evrardjp has quit IRC17:33
*** evrardjp has joined #openstack-kolla17:33
yoctozeptomgoddard: this part is what makes me wonder, but it looks as if clean shutdown must send some info, or maybe split in 2 is just allowed by our config17:34
yoctozeptoanyways, it happily acknowledges having 1/1 quorum17:34
yoctozeptomgoddard: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ae8/699172/9/check/kolla-ansible-centos-source-upgrade-ceph/ae87dd1/secondary2/logs/kolla/mariadb/mariadb.txt.gz17:34
yoctozeptofind 2020-01-02 12:28:3817:35
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Sleep during container recreate/restart  https://review.opendev.org/70092017:40
mgoddardyoctozepto: ^ curious17:41
yoctozeptomgoddard: as always17:50
* yoctozepto curious what mgoddard breaks this way17:50
yoctozepto:D17:50
yoctozeptoknowing our inherent luck, it will break on something else17:50
*** mgoddard has quit IRC18:04
yoctozeptohmm, it looks like mariadb shutting properly sends a goodbye to the cluster18:05
yoctozeptohence probably why it can descale to 1/118:05
yoctozeptothen I wonder how shutting down two at once could break that18:06
yoctozeptovery mariadbish...18:06
*** jonmasters has joined #openstack-kolla18:18
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/train: Make fluentd-elasticsearch configuration more robust  https://review.opendev.org/70092719:12
*** pcaruana has quit IRC19:15
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Make fluentd-elasticsearch configuration more robust  https://review.opendev.org/67108019:15
*** mgoddard has joined #openstack-kolla19:27
*** vmixor has quit IRC19:36
openstackgerritMerged openstack/kolla-ansible master: Add AArch64 CI job  https://review.opendev.org/69946419:41
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Fix MariaDB galera IPv6 deployment on CentOS 7  https://review.opendev.org/69917220:20
openstackgerritDincer Celik proposed openstack/kolla-ansible master: Adds maximum supported version option for Ansible  https://review.opendev.org/69933320:27
*** kplant has quit IRC20:28
openstackgerritDincer Celik proposed openstack/kolla-ansible master: Adds maximum supported version check for Ansible  https://review.opendev.org/69933320:38
openstackgerritDincer Celik proposed openstack/kolla-ansible master: Adds Docker image pruning command  https://review.opendev.org/69869420:54
*** tonythomas has quit IRC21:46
*** goldyfruit has quit IRC22:14
jonmastershrw: what would you say the status of Kolla on CentOS7 is for aarch64?22:39
jonmastersI wanted to use CentOS 8 but that aint happening22:40
hrwjonmasters: use stable/train23:02
hrwjonmasters: and do not mention centos8 here too often. it is still in progress due to packages missing due to buildroot missing due to other centos issues23:03
hrwjonmasters: you can run all-in-one setup with c8 - we have patches for it23:03
hrwbut shared? forget - no ceph.23:03
* hrw off23:11
r3ap3rI have a quick question, I mean I hope it is quick. I have an All-in-One Deployment on CentOS 7.7 and I am trying to get LAN access to the VM/Containers but nothing I do appears to be working. After the deployment is successful and run through the post deploy procedures, there is a step where you run `init-runonce`. The default external net info doesn't work on my network so I went in and changed it to...23:25
r3ap3rwhat I though would be correct but when VM's/Containers get an IP from external net range, it doesn't make it out and I cannot ping it from the LAN, I was wondering if I had the config right in the `init-runonce` script or was I going about this the wrong way??23:26
r3ap3rhttps://www.irccloud.com/pastebin/mV6Wp4I7/23:26
jonmastersthanks hrw23:35
jonmastersI'll take a look at train23:35
cosmicsound      heat_container_agent_tag: ussuri-dev anyone knowshow this slipped in a train deployment, this is a magnum deployment23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!