| *** cloudnull has quit IRC | 00:16 | |
| *** cloudnull has joined #openstack-kolla | 00:18 | |
| *** tonythomas has quit IRC | 00:25 | |
| *** xinliang has joined #openstack-kolla | 00:40 | |
| aleccoder | Is cloudkitty even still supported with kolla-ansible on train? | 01:52 |
|---|---|---|
| *** xinliang has quit IRC | 01:54 | |
| *** xinliang has joined #openstack-kolla | 02:21 | |
| *** k_mouza has joined #openstack-kolla | 02:33 | |
| *** k_mouza has quit IRC | 02:38 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Fix deploy mariadb failed to reach primary node https://review.opendev.org/699172 | 02:45 |
| *** k_mouza has joined #openstack-kolla | 03:05 | |
| *** k_mouza has quit IRC | 03:30 | |
| *** xinliang has quit IRC | 04:16 | |
| *** skramaja has joined #openstack-kolla | 04:49 | |
| *** zainub_wahid has joined #openstack-kolla | 05:15 | |
| *** zainub_wahid has joined #openstack-kolla | 05:21 | |
| *** goldyfruit has quit IRC | 05:24 | |
| *** goldyfruit has joined #openstack-kolla | 05:25 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-kolla | 05:33 | |
| *** shyamb has joined #openstack-kolla | 06:17 | |
| *** mrunge has quit IRC | 06:19 | |
| *** mrunge has joined #openstack-kolla | 06:24 | |
| zainub_wahid | When is kolla-ansible 9.2.0 release expected? will there be 9.1.0 release first? | 06:52 |
| hrw | zainub_wahid: is there a rush for new release? | 07:02 |
| osmanlicilegi | morning | 07:25 |
| *** shyamb has quit IRC | 07:26 | |
| hrw | osmanlicilegi: hi Dincer | 07:26 |
| *** jbadiapa has joined #openstack-kolla | 07:26 | |
| osmanlicilegi | hrw: how's going? | 07:27 |
| osmanlicilegi | I hope everybody had a good rest :] | 07:27 |
| hrw | osmanlicilegi: so far, so good | 07:27 |
| hrw | looking now how our build works once we disable epel ;D | 07:30 |
| *** cah_link has joined #openstack-kolla | 07:32 | |
| *** pcaruana has joined #openstack-kolla | 07:34 | |
| osmanlicilegi | https://review.opendev.org/#/c/699333/ is ready for reviews. | 07:35 |
| patchbot | patch 699333 - kolla-ansible - Adds maximum supported version option for Ansible - 7 patch sets | 07:35 |
| *** shyamb has joined #openstack-kolla | 07:36 | |
| *** jbadiapa has quit IRC | 07:38 | |
| hrw | commented | 07:40 |
| osmanlicilegi | got it | 07:42 |
| osmanlicilegi | ah I see I've fixed it but didn't pushed | 07:45 |
| hrw | ;D | 07:46 |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: Adds maximum supported version option for Ansible https://review.opendev.org/699333 | 07:46 |
| hrw | +2 | 07:47 |
| osmanlicilegi | thanks! | 07:47 |
| yoctozepto | morning | 07:47 |
| *** shyamb has quit IRC | 07:48 | |
| hrw | yoctozepto: hi | 07:49 |
| yoctozepto | hi hrw | 07:49 |
| hrw | yoctozepto: you planned to disable epel, right? | 07:49 |
| yoctozepto | hrw: yeah, more like "wished to" | 07:49 |
| hrw | yoctozepto: https://paste.centos.org/view/1c42e391 | 07:50 |
| yoctozepto | currently producing bandanas with "kill epel" emblem | 07:50 |
| yoctozepto | this patch is evil | 07:50 |
| hrw | yoctozepto: it shows base for getting it done properly | 07:51 |
| yoctozepto | hrw: but it shows adding epel :D | 07:51 |
| hrw | yoctozepto: kill scsi-target-utils first? | 07:52 |
| yoctozepto | hrw: mandatory | 07:53 |
| hrw | yoctozepto: https://review.opendev.org/#/c/613815/ probably | 07:54 |
| patchbot | patch 613815 - kolla - Remove tgtd and scsi-target-utils support in CentO... (ABANDONED) - 12 patch sets | 07:54 |
| yoctozepto | hrw: the issue is k-a | 07:54 |
| yoctozepto | and general migration from tgtd to lio | 07:55 |
| hrw | in other words: not going to happen | 07:56 |
| hrw | move to centos8 will just drop it | 07:56 |
| yoctozepto | hrw: most likely, unfortunately | 07:57 |
| yoctozepto | I think epel8 still has it? | 07:57 |
| yoctozepto | anyways, since we are forcing users to migrate to c8 on host | 07:57 |
| yoctozepto | the current tgtd have to go anyway | 07:57 |
| hrw | ()[root@3d033af0a58d yum.repos.d]# dnf info scsi-target-utils | 08:01 |
| hrw | Last metadata expiration check: 0:02:41 ago on Thu 02 Jan 2020 07:58:59 AM UTC. | 08:01 |
| hrw | Error: No matching Packages to list | 08:01 |
| hrw | does not have | 08:01 |
| yoctozepto | hrw: so it's forced and we are done with epel by means of c8 | 08:08 |
| *** numans has quit IRC | 08:25 | |
| *** numans has joined #openstack-kolla | 08:26 | |
| *** jistr_ is now known as jistr | 08:29 | |
| mnasiadka | morning | 08:38 |
| *** rpittau|afk is now known as rpittau | 08:41 | |
| hrw | hi mnasiadka | 08:44 |
| *** shyamb has joined #openstack-kolla | 08:44 | |
| * hrw just dropped enable_repos | 08:44 | |
| mnasiadka | hi hrw | 08:44 |
| mnasiadka | trying to learn typing on a keyboard after xmas && new year break ;) | 08:44 |
| hrw | speaking of keyboard... during visit in Castorama I took another ms ergonomic 4000 keyboard from electrowaste basket | 08:45 |
| hrw | I do not care does it work. keys are like not used! | 08:45 |
| hrw | kolla.tests.test_methods.MethodsTest.test_disable_repos_centos 0.019 | 08:47 |
| hrw | ;D | 08:47 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix "deploy mariadb failed to reach primary node" - short edition https://review.opendev.org/700875 | 08:48 |
| *** shyamb has quit IRC | 08:49 | |
| *** shyamb has joined #openstack-kolla | 08:50 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: DNM: CentOS IPv6 https://review.opendev.org/691811 | 08:51 |
| hrw | yoctozepto: https://paste.centos.org/view/4b4fbe41 should look better | 08:52 |
| yoctozepto | hrw: nice | 08:53 |
| *** shyam89 has joined #openstack-kolla | 08:54 | |
| *** shyamb has quit IRC | 08:54 | |
| mnasiadka | why do we need epel at all? :) | 08:55 |
| hrw | mnasiadka: scsi-target-utils? | 08:59 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: centos: disable EPEL by default https://review.opendev.org/700877 | 08:59 |
| mnasiadka | hrw: that is tgtd I assume? | 09:00 |
| hrw | mnasiadka: base | 09:00 |
| hrw | commit 184f01a0d645d58fd4c9b6c5004e94fe98a077f8 | 09:03 |
| hrw | this added scsi-target-utils into base | 09:04 |
| hrw | https://review.opendev.org/#/c/291285/ | 09:05 |
| patchbot | patch 291285 - kolla - Adds iscsid and tgtd docker containers (MERGED) - 55 patch sets | 09:05 |
| mnasiadka | yeah, I guess | 09:05 |
| mnasiadka | but still I remember me and yoctozepto wanted to get rid of tgtd in favor of lio | 09:05 |
| yoctozepto | you remember well, mnasiadka | 09:06 |
| yoctozepto | c8 simply forces this on us | 09:06 |
| yoctozepto | as it's nowhere there | 09:07 |
| mnasiadka | yeah, so why hrw is wasting time on this? :) | 09:07 |
| yoctozepto | and users are forced anyway to turn off their target | 09:07 |
| yoctozepto | for the c8 upgrade | 09:07 |
| yoctozepto | mnasiadka: I like the disable stuff anyway | 09:07 |
| yoctozepto | could come in handy with something else | 09:07 |
| hrw | mnasiadka: software archeology | 09:07 |
| hrw | yoctozepto: collectd uses it to disable extras and opstools | 09:09 |
| yoctozepto | hrw: for example :D | 09:09 |
| * hrw votes from drop of tripleo7 due to 2020 | 09:29 | |
| hrw | s/from/for | 09:29 |
| hrw | ;) | 09:29 |
| *** k_mouza has joined #openstack-kolla | 09:31 | |
| hrw | hm. should login to aarch64 host, build/upload set of images and then write CI job to make multiarch images | 09:36 |
| yoctozepto | hrw: yeah, we be dropping | 09:36 |
| hrw | which will require adding --list-buildable-images to kolla-build first | 09:36 |
| yoctozepto | we can have a job running after two other jobs I guess but not sure | 09:37 |
| yoctozepto | best ask infra | 09:37 |
| hrw | I got help already | 09:38 |
| hrw | just have to sit and write | 09:38 |
| hrw | https://review.opendev.org/695168 needs brave core | 09:39 |
| patchbot | patch 695168 - kolla-ansible - Get rid of Python 2 support - 12 patch sets | 09:39 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Get rid of Python 2 support https://review.opendev.org/691316 | 09:42 |
| hrw | https://bugs.launchpad.net/tripleo/+bug/1856678 | 09:43 |
| openstack | Launchpad bug 1856678 in tripleo "Use Python 3 in CentOS 7 overcloud image" [High,Triaged] | 09:43 |
| hrw | and no, I do not plan to dig into tripleo code | 09:46 |
| hrw | they have multiple git repos | 09:46 |
| hrw | https://review.opendev.org/#/c/699555/ - vote! | 09:48 |
| patchbot | patch 699555 - tripleo-ci - [DNM] Run container build with python3 since train... - 2 patch sets | 09:48 |
| mnasiadka | hrw: well, it's DNM :) | 09:48 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Get rid of Python 2 support https://review.opendev.org/691316 | 09:55 |
| hrw | mnasiadka: soon will not be | 09:55 |
| yoctozepto | xD | 09:57 |
| hrw | https://review.opendev.org/#/c/699555 - go for it. | 09:59 |
| patchbot | patch 699555 - tripleo-ci - Run container build with python3 since train on Ce... - 3 patch sets | 09:59 |
| *** k_mouza has quit IRC | 10:02 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs https://review.opendev.org/676376 | 10:12 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs https://review.opendev.org/676376 | 10:15 |
| hrw | mnasiadka: 192 revisions. nice | 10:20 |
| mnasiadka | yeah, and I'm not finished it seems ;) | 10:20 |
| hrw | mnasiadka: 256 is not that far away | 10:21 |
| mnasiadka | might be :) | 10:21 |
| mnasiadka | ok, 193 on the way | 10:21 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs https://review.opendev.org/676376 | 10:22 |
| yoctozepto | yeah, new year, new resolutions | 10:24 |
| yoctozepto | in 2020 we accept only powers of 2 | 10:24 |
| yoctozepto | as revision numbers | 10:24 |
| yoctozepto | so it's either 256 or you work till 512 :-) | 10:24 |
| *** k_mouza has joined #openstack-kolla | 10:25 | |
| zainub_wahid | is kolla-ansible 9.1.0 release in pipeline? and when will 9.2.0 be released? | 10:38 |
| hrw | zainub_wahid: let me repeat question from few hours ago: is there a rush for new release? | 10:39 |
| hrw | zainub_wahid: any patch merged in meantime qualifies to make release? | 10:39 |
| hrw | zainub_wahid: or is it because few months passed since last release? | 10:40 |
| *** shyam89 has quit IRC | 10:41 | |
| zainub_wahid | i am working on Magnum and Fedora-CoreOS is acceptable in 9.2.0. | 10:43 |
| hrw | zainub_wahid: version of kolla* does not have to be in sync with other components | 10:44 |
| hrw | zainub_wahid: if you need magnum 9.2.0 then you may need git version of kolla stable/train | 10:45 |
| hrw | we updated to use 9.2.0 magnum | 10:45 |
| *** shyamb has joined #openstack-kolla | 10:58 | |
| *** tonythomas has joined #openstack-kolla | 10:58 | |
| hrw | uf. --list-images can be used. | 10:59 |
| zainub_wahid | okay. Thank you | 11:05 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix MariaDB galera deployment on CentOS 7 https://review.opendev.org/699172 | 11:09 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: DNM: CentOS IPv6 https://review.opendev.org/691811 | 11:11 |
| hrw | https://hub.docker.com/repository/registry-1.docker.io/haerwu/debian-source-base/tags - preparations in progress | 11:18 |
| *** k_mouza has quit IRC | 11:20 | |
| *** sean-k-mooney has joined #openstack-kolla | 11:26 | |
| *** k_mouza has joined #openstack-kolla | 11:27 | |
| hrw | I think that we push too late in build process... | 11:34 |
| hrw | I am slowly considering adding --quiet switch to kolla | 11:37 |
| hrw | so all build logs go to files but are not printed | 11:37 |
| *** sean-k-mooney has quit IRC | 11:43 | |
| *** sean-k-mooney has joined #openstack-kolla | 11:44 | |
| hrw | INFO:kolla.common.utils:Attempt number: 1 to run task: BuildTask(zaqar-wsgi) | 11:50 |
| hrw | INFO:kolla.common.utils:Attempt number: 1 to run task: PushIntoQueueTask(PushTask(qinling-engine)=><queue.Queue object at 0x7f056119cad0>) | 11:50 |
| hrw | hm. | 11:50 |
| hrw | what for we say queue.Queue here? :D | 11:52 |
| hrw | INFO:kolla.common.utils:Added next task PushIntoQueueTask(PushTask(watcher-applier)) to queue | 11:52 |
| hrw | INFO:kolla.common.utils:Attempt number: 1 to run task: PushIntoQueueTask(PushTask(watcher-applier)) | 11:52 |
| hrw | looks better | 11:52 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add '--quiet' switch https://review.opendev.org/700888 | 11:59 |
| hrw | https://paste.centos.org/view/69202e06 | 12:00 |
| hrw | this is how --quiet looks | 12:00 |
| *** kplant has joined #openstack-kolla | 12:05 | |
| *** shyam89 has joined #openstack-kolla | 12:07 | |
| *** shyamb has quit IRC | 12:07 | |
| hrw | do someone else has a use for it? | 12:10 |
| *** shyam89 has quit IRC | 12:12 | |
| *** vmixor has joined #openstack-kolla | 12:14 | |
| osmanlicilegi | --quiet makes sense to me. | 12:18 |
| osmanlicilegi | quick and good job hrw | 12:18 |
| hrw | osmanlicilegi: I like such small patches | 12:19 |
| osmanlicilegi | now I need to backport it to my stein branch :D | 12:19 |
| hrw | osmanlicilegi: move quiet just above debug in common/config.py and should be cherry-pickable | 12:20 |
| *** mgoddard has joined #openstack-kolla | 12:25 | |
| *** shyamb has joined #openstack-kolla | 12:26 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs https://review.opendev.org/676376 | 12:28 |
| mgoddard | Hello Koalas, Happy New Year! | 12:29 |
| mgoddard | I had a little IRC trouble this morning, I think we're good now | 12:29 |
| mnasiadka | Good afternoon mgoddard, happy new year :) | 12:30 |
| hrw | mgoddard: HNY! | 12:33 |
| mgoddard | thanks hrw and mnasiadka | 12:33 |
| osmanlicilegi | koalas love you mgoddard, happy new year! | 12:33 |
| mgoddard | :) | 12:33 |
| hrw | mgoddard: good news: tripleo works on getting rid of using py2 to run kolla | 12:33 |
| mgoddard | hrw: that is good news | 12:34 |
| osmanlicilegi | bad news, they still didn't do anything for the rmq update | 12:34 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs https://review.opendev.org/676376 | 12:34 |
| hrw | and speaking of py2... we may need to write/backport some py2/py3 changes in few weeks. Ceph 14 (nautilus) will land in Debian:buster and be py3 only | 12:35 |
| mnasiadka | osmanlicilegi: go complain on #rdo - it's not really tripleo that should do something about it :) | 12:35 |
| hrw | I do not remember how far we are with py3 ceph stuff | 12:35 |
| mnasiadka | hrw: we are dropping kolla-ceph this cycle. | 12:36 |
| hrw | mnasiadka: I worry about stein/train this time | 12:36 |
| mgoddard | hrw: debian already supports py3 | 12:36 |
| mgoddard | hrw: debian/binary since stein, debian/source since train | 12:36 |
| mnasiadka | hrw: ugh | 12:36 |
| hrw | mgoddard: so stein can have problem then | 12:37 |
| hrw | will see | 12:37 |
| mgoddard | hrw: also, we won't upgrade ceph on stable branches | 12:37 |
| hrw | mgoddard: will pin then ;D | 12:37 |
| hrw | even easier | 12:37 |
| mgoddard | hrw: yes | 12:37 |
| mgoddard | no surprise major upgrades please :) | 12:37 |
| hrw | --quiet rulz ;D | 12:37 |
| hrw | I just need number of tasks now | 12:38 |
| hrw | INFO:kolla.common.utils:Attempt number: 1 to run task: PushTask(swift-base) | 12:38 |
| hrw | something like "INFO:kolla.common.utils:Attempt number: 1 to run task (53/401): PushTask(swift-base) " | 12:39 |
| hrw | but it is not doable without changes as we build queue during build not before | 12:39 |
| *** spsurya has joined #openstack-kolla | 12:48 | |
| *** zainub_wahid has quit IRC | 12:49 | |
| openstackgerrit | Merged openstack/kolla stable/stein: Bump OpenStack versions for Stein https://review.opendev.org/700184 | 13:01 |
| *** shyamb has quit IRC | 13:10 | |
| *** skramaja has quit IRC | 13:16 | |
| *** k_mouza has quit IRC | 13:19 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs https://review.opendev.org/676376 | 13:28 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs https://review.opendev.org/676376 | 13:42 |
| openstackgerrit | LEDUC Florian proposed openstack/kolla-ansible master: Add ability to use the log service plugins for Neutron https://review.opendev.org/700895 | 13:49 |
| *** dave-mccowan has joined #openstack-kolla | 13:51 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs https://review.opendev.org/676376 | 13:59 |
| *** k_mouza has joined #openstack-kolla | 14:02 | |
| mgoddard | mnasiadka: you may as well go for #200 now | 14:07 |
| mgoddard | any takers for https://review.opendev.org/#/c/698445/? | 14:10 |
| patchbot | patch 698445 - kolla - CentOS 8: work around mod_ssl certificate issue - 5 patch sets | 14:10 |
| * hrw did | 14:20 | |
| *** dave-mccowan has quit IRC | 14:21 | |
| *** dave-mccowan has joined #openstack-kolla | 14:26 | |
| openstackgerrit | LEDUC Florian proposed openstack/kolla-ansible master: Add ability to use the log service_plugins for Neutron https://review.opendev.org/700895 | 14:27 |
| openstackgerrit | Merged openstack/kolla-ansible master: Docs: remove some bad recommendations https://review.opendev.org/691814 | 14:28 |
| yoctozepto | Happy New Year mgoddard :-) you back for good? | 14:47 |
| mgoddard | yoctozepto: indeed I am. I will never leave again | 14:47 |
| mgoddard | yoctozepto: did anything happen while we were away? I saw you were active a bit | 14:51 |
| *** jaicaa has quit IRC | 14:52 | |
| *** jaicaa has joined #openstack-kolla | 14:53 | |
| mnasiadka | mgoddard: that's my plan | 15:10 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Fix seed VM provision https://review.opendev.org/700904 | 15:22 |
| *** KeithMnemonic has joined #openstack-kolla | 15:25 | |
| hrw | (venv3) 15:30 (2s) hrw@j13-qrep-04:kolla$ time ./tools/create-multiarch-images.sh | 15:31 |
| hrw | INFO:kolla.common.utils:Found the docker image folder at /home/hrw/devel/kolla/kolla/docker | 15:31 |
| hrw | INFO:kolla.common.utils:Found the docker image folder at /home/hrw/devel/kolla/kolla/docker | 15:31 |
| hrw | Created manifest list docker.io/test/test-aodh-api:master | 15:31 |
| hrw | closer ;D | 15:31 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs https://review.opendev.org/676376 | 15:36 |
| hrw | so the goal will be: 1st CI publish job does :master-aarch64 images, 2nd does :master-x86_64 and 3rd does multiarch ones. | 15:47 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs https://review.opendev.org/676376 | 15:52 |
| hrw | can docker retag image in remote registry without pull/push? | 15:52 |
| yoctozepto | mgoddard: glad you are back; yeah, I was active "a bit", keeping the channel alive; 99cloud (incl. Jeffrey4l) contributed completely set of ipv6 patches to get c7 on its feet | 15:54 |
| hrw | hm. maybe have solution | 15:54 |
| mgoddard | yoctozepto: I saw those. Always nice to remove some exceptions | 15:54 |
| mnasiadka | hrw: well, you can pull & push the manifest only, using REST - but I don't know if this is what you want to do | 15:54 |
| yoctozepto | mgoddard: by introducing exceptions :D | 15:54 |
| mgoddard | well, true | 15:55 |
| hrw | mnasiadka: I can create manifest for one arch too ;D | 15:55 |
| yoctozepto | anyways, the assumptions these workarounds make are sane | 15:55 |
| yoctozepto | so it's worth having c7 with ipv6 working on train | 15:55 |
| hrw | mnasiadka: yay! #200 :D | 15:56 |
| mnasiadka | hrw: yeah, it's all mgoddard's fault - asked me to do something properly :D | 15:57 |
| hrw | https://hub.docker.com/repository/docker/haerwu/debian-source-aodh-api yay ;) | 15:57 |
| yoctozepto | mnasiadka: remember about 256/512 | 15:57 |
| mnasiadka | yoctozepto: I think I'll go big, and try to get to 1024 | 15:57 |
| mnasiadka | ;) | 15:57 |
| yoctozepto | mnasiadka: you are welcome too | 15:58 |
| yoctozepto | (and then we discover gerrit breaks after 1000) | 15:58 |
| hrw | 16:58 (4s) hrw@puchatek:~$ docker manifest inspect haerwu/debian-source-aodh-api:master | 15:58 |
| hrw | 16:58 (0s) hrw@puchatek:~$ docker manifest inspect haerwu/debian-source-aodh-api:master|grep archi | 15:59 |
| hrw | "architecture": "arm64", | 15:59 |
| hrw | "architecture": "amd64", | 15:59 |
| hrw | easy way to tag remote image? "docker manifest create NEWTAG OLDTAG;docker manifest push NEWTAG" | 16:01 |
| *** pcaruana has quit IRC | 16:04 | |
| hrw | https://review.opendev.org/700888 https://review.opendev.org/700877 both passed zuul | 16:08 |
| patchbot | patch 700888 - kolla - Add '--quiet' switch - 1 patch set | 16:08 |
| patchbot | patch 700877 - kolla - centos: disable EPEL by default - 1 patch set | 16:08 |
| hrw | https://review.opendev.org/695168 needs vote | 16:08 |
| patchbot | patch 695168 - kolla-ansible - Get rid of Python 2 support - 12 patch sets | 16:08 |
| hrw | and no idea what to do with https://review.opendev.org/699361 one - push, add extra argument and push or abandon | 16:09 |
| patchbot | patch 699361 - kolla - build: remove not buildable templates for --templa... - 1 patch set | 16:09 |
| hrw | https://review.opendev.org/694797 will probably rot in gerrit ;d | 16:09 |
| patchbot | patch 694797 - kolla-ansible - Some ansible-lint fixes - 12 patch sets | 16:09 |
| hrw | https://paste.debian.net/1124164/ is whole tools/create-multiarch-images.sh ;d | 16:11 |
| openstackgerrit | LEDUC Florian proposed openstack/kolla-ansible master: Enable libvirt multipath ISCSI in nova multipathd is enabled https://review.opendev.org/700910 | 16:12 |
| openstackgerrit | LEDUC Florian proposed openstack/kolla-ansible master: Enable libvirt multipath ISCSI in nova multipathd is enabled https://review.opendev.org/700910 | 16:24 |
| hrw | my favourite type of commit message | 16:27 |
| hrw | 'how' instead of 'why' | 16:28 |
| yoctozepto | mnasiadka: https://review.opendev.org/699172 | 16:28 |
| patchbot | patch 699172 - kolla-ansible - Fix MariaDB galera deployment on CentOS 7 - 9 patch sets | 16:28 |
| yoctozepto | hrw: well, it's not that bad, that 'in' should actually be 'if' | 16:29 |
| yoctozepto | then it's quite nice | 16:29 |
| hrw | +1 | 16:30 |
| yoctozepto | hrw: only message was nice | 16:32 |
| yoctozepto | -1 | 16:32 |
| hrw | ;D | 16:34 |
| * hrw out | 16:34 | |
| hrw | have fun | 16:34 |
| *** cah_link has quit IRC | 16:40 | |
| *** pcaruana has joined #openstack-kolla | 16:43 | |
| yoctozepto | mgoddard, mnasiadka: I think I have found out our mariadb upgrade issues reason | 16:45 |
| yoctozepto | we should wait after each "slave" to regain quorum | 16:46 |
| yoctozepto | because in the failed ones the "master" declares it has been split off (as it lost connection to 2 others) | 16:46 |
| yoctozepto | and we lose the cluster | 16:47 |
| *** rpittau is now known as rpittau|afk | 16:51 | |
| yoctozepto | any ideas how to serialize this cleanly? | 16:58 |
| mgoddard | yoctozepto: as in start slaves seriallu? | 17:05 |
| yoctozepto | mgoddard: stop and start, yeah | 17:06 |
| yoctozepto | I commented in my change | 17:06 |
| mgoddard | yoctozepto: maybe we just need 3 phases | 17:06 |
| yoctozepto | 3 phases? | 17:07 |
| mgoddard | rather than master/slave, phase 1 2 3 | 17:07 |
| yoctozepto | yeah, and if someone runs 5? | 17:07 |
| mgoddard | for each phase: restart all nodes in phase, wait | 17:07 |
| yoctozepto | it can be phased as long as you can guarantee a quorum | 17:07 |
| mgoddard | groups['mariadb'].index(inventory_hostname) % 3 == phase | 17:08 |
| mgoddard | alternatively, use etcd to roll through changes | 17:09 |
| mgoddard | I keep thinking we should look into that | 17:09 |
| mgoddard | without somehow building k8s | 17:09 |
| yoctozepto | restart kolla-kubernetes? xD | 17:10 |
| mgoddard | lol | 17:10 |
| yoctozepto | we really only need this for stateful stuff like mariadb | 17:10 |
| mgoddard | yes, but if there was a general mechanism we could avoid downtime | 17:10 |
| yoctozepto | even rmq simply stops servicing requests when quorum is lost, not declaring everything lost and damned | 17:11 |
| mgoddard | neutron agents keep coming up | 17:11 |
| yoctozepto | mhm | 17:11 |
| yoctozepto | you are thinking big image | 17:11 |
| mgoddard | always yoctozepto, always | 17:11 |
| yoctozepto | I am stuck at getting mariadb not to break now and then :D | 17:11 |
| yoctozepto | good ptl | 17:12 |
| mgoddard | I think 3 phases could work for mariadb, if that is the issue | 17:12 |
| yoctozepto | yeah, that seems to make sense | 17:13 |
| yoctozepto | it gets rid of this messy (and wrong) master-slave relation | 17:13 |
| mgoddard | it makes sense - if 'master' notices both slaves have gone then it will lose quorum | 17:13 |
| yoctozepto | yeah, in the original choice of master we were lucky enough for it to notice one after the other | 17:14 |
| yoctozepto | most of the time | 17:14 |
| mgoddard | alternatively, could we use serial: 30%? | 17:14 |
| yoctozepto | (with rare occurence of this failure) | 17:14 |
| yoctozepto | mgoddard: can't have serial outside of play | 17:14 |
| mgoddard | no | 17:14 |
| mgoddard | but can on a play | 17:14 |
| yoctozepto | still need to establish which mariadb are already deployed | 17:15 |
| yoctozepto | and which are not | 17:15 |
| mgoddard | for cases where the cluster is half up? | 17:15 |
| yoctozepto | for cases when one is carefree and does a pull-deploy on half-deployed cluster, yeah | 17:16 |
| yoctozepto | or has a change in config | 17:16 |
| mgoddard | hmm | 17:16 |
| yoctozepto | with this ipv6 I kept mariadb from changing config unless necessary | 17:16 |
| yoctozepto | because it's so destructive | 17:16 |
| mgoddard | sure there were no whitespace changes? | 17:16 |
| yoctozepto | the patch helps with deploying new mariadb | 17:16 |
| yoctozepto | mgoddard: checked, our merger is stable and removes all | 17:17 |
| mgoddard | cool | 17:17 |
| yoctozepto | back to topic | 17:17 |
| yoctozepto | the patch helps with deploying new mariadb | 17:17 |
| yoctozepto | but it seems with replacing containers in CI | 17:17 |
| yoctozepto | it makes the procedure even less stable | 17:17 |
| yoctozepto | probably due to different load pattern | 17:18 |
| mgoddard | sure it's not because the 'master' has changed? | 17:19 |
| mgoddard | :) | 17:19 |
| yoctozepto | not by galera design | 17:19 |
| yoctozepto | they are all master | 17:19 |
| yoctozepto | if you compare logs between success and failure | 17:19 |
| yoctozepto | it is missing the step of reducing quorum to 2/2 | 17:20 |
| yoctozepto | (and 1/1 which is easy with galera) | 17:20 |
| yoctozepto | it sees immediate drop of two friends | 17:20 |
| yoctozepto | and thinks it's no longer in primary view (the quorum) | 17:20 |
| yoctozepto | as a homework you can try unplugging power coords from majority of mariadb hosts and check the result | 17:21 |
| mgoddard | so in the case of a semi-deployed cluster, you need to start up the undeployed nodes first to avoid losing quorum? | 17:22 |
| mgoddard | I will try in production now | 17:22 |
| yoctozepto | yeah, it's easy | 17:22 |
| mgoddard | then restart deployed nodes | 17:22 |
| yoctozepto | I think I've got all the stuff sorted out in my head, now need to implement it, yeah | 17:22 |
| mgoddard | which means the groups['mariadb'] thing earlier wouldn't work | 17:23 |
| yoctozepto | exactly | 17:23 |
| yoctozepto | but it can be refined | 17:23 |
| mgoddard | try adding a sleep in kolla_docker to see if the issue is 100% reproducible | 17:25 |
| mgoddard | would be a nice demo | 17:25 |
| yoctozepto | sleep where? | 17:26 |
| mgoddard | restart_container | 17:26 |
| mgoddard | after stop, before start | 17:26 |
| yoctozepto | not sure what that would demo out now | 17:27 |
| yoctozepto | the problem is sudden stop of both friends, what to sleep for :-) | 17:27 |
| mgoddard | IIUC, it's racy because the containers normally start up again quickly, so the 'master' does not register them as down | 17:29 |
| mgoddard | if we sleep it will see them as down | 17:29 |
| yoctozepto | IIUC is False | 17:30 |
| yoctozepto | False -> whatever is True | 17:30 |
| yoctozepto | :D | 17:30 |
| yoctozepto | nah, the problem is that it somehow does not see the first one going down | 17:30 |
| yoctozepto | and it does not reduce the quorum size | 17:30 |
| yoctozepto | and instead immediately sees both going down | 17:30 |
| yoctozepto | and goes: "oh noez, I am alone" | 17:31 |
| yoctozepto | "I | 17:31 |
| yoctozepto | "I'm fine but not PRIMARY" | 17:31 |
| yoctozepto | "let's hope I'm rejoined with my friends soon" | 17:31 |
| yoctozepto | the others come back | 17:31 |
| yoctozepto | and go | 17:31 |
| yoctozepto | "where is the cluster?" | 17:31 |
| yoctozepto | and every other responds "not here" | 17:32 |
| yoctozepto | check mate | 17:32 |
| mgoddard | ok, so in the working case | 17:32 |
| mgoddard | first node goes down | 17:32 |
| mgoddard | cluster is now 2 nodes | 17:32 |
| mgoddard | why does it not lose quorum when the second node goes down? | 17:33 |
| *** evrardjp has quit IRC | 17:33 | |
| *** evrardjp has joined #openstack-kolla | 17:33 | |
| yoctozepto | mgoddard: this part is what makes me wonder, but it looks as if clean shutdown must send some info, or maybe split in 2 is just allowed by our config | 17:34 |
| yoctozepto | anyways, it happily acknowledges having 1/1 quorum | 17:34 |
| yoctozepto | mgoddard: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ae8/699172/9/check/kolla-ansible-centos-source-upgrade-ceph/ae87dd1/secondary2/logs/kolla/mariadb/mariadb.txt.gz | 17:34 |
| yoctozepto | find 2020-01-02 12:28:38 | 17:35 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: DNM: Sleep during container recreate/restart https://review.opendev.org/700920 | 17:40 |
| mgoddard | yoctozepto: ^ curious | 17:41 |
| yoctozepto | mgoddard: as always | 17:50 |
| * yoctozepto curious what mgoddard breaks this way | 17:50 | |
| yoctozepto | :D | 17:50 |
| yoctozepto | knowing our inherent luck, it will break on something else | 17:50 |
| *** mgoddard has quit IRC | 18:04 | |
| yoctozepto | hmm, it looks like mariadb shutting properly sends a goodbye to the cluster | 18:05 |
| yoctozepto | hence probably why it can descale to 1/1 | 18:05 |
| yoctozepto | then I wonder how shutting down two at once could break that | 18:06 |
| yoctozepto | very mariadbish... | 18:06 |
| *** jonmasters has joined #openstack-kolla | 18:18 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/train: Make fluentd-elasticsearch configuration more robust https://review.opendev.org/700927 | 19:12 |
| *** pcaruana has quit IRC | 19:15 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Make fluentd-elasticsearch configuration more robust https://review.opendev.org/671080 | 19:15 |
| *** mgoddard has joined #openstack-kolla | 19:27 | |
| *** vmixor has quit IRC | 19:36 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Add AArch64 CI job https://review.opendev.org/699464 | 19:41 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix MariaDB galera IPv6 deployment on CentOS 7 https://review.opendev.org/699172 | 20:20 |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: Adds maximum supported version option for Ansible https://review.opendev.org/699333 | 20:27 |
| *** kplant has quit IRC | 20:28 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: Adds maximum supported version check for Ansible https://review.opendev.org/699333 | 20:38 |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: Adds Docker image pruning command https://review.opendev.org/698694 | 20:54 |
| *** tonythomas has quit IRC | 21:46 | |
| *** goldyfruit has quit IRC | 22:14 | |
| jonmasters | hrw: what would you say the status of Kolla on CentOS7 is for aarch64? | 22:39 |
| jonmasters | I wanted to use CentOS 8 but that aint happening | 22:40 |
| hrw | jonmasters: use stable/train | 23:02 |
| hrw | jonmasters: and do not mention centos8 here too often. it is still in progress due to packages missing due to buildroot missing due to other centos issues | 23:03 |
| hrw | jonmasters: you can run all-in-one setup with c8 - we have patches for it | 23:03 |
| hrw | but shared? forget - no ceph. | 23:03 |
| * hrw off | 23:11 | |
| r3ap3r | I have a quick question, I mean I hope it is quick. I have an All-in-One Deployment on CentOS 7.7 and I am trying to get LAN access to the VM/Containers but nothing I do appears to be working. After the deployment is successful and run through the post deploy procedures, there is a step where you run `init-runonce`. The default external net info doesn't work on my network so I went in and changed it to... | 23:25 |
| r3ap3r | what I though would be correct but when VM's/Containers get an IP from external net range, it doesn't make it out and I cannot ping it from the LAN, I was wondering if I had the config right in the `init-runonce` script or was I going about this the wrong way?? | 23:26 |
| r3ap3r | https://www.irccloud.com/pastebin/mV6Wp4I7/ | 23:26 |
| jonmasters | thanks hrw | 23:35 |
| jonmasters | I'll take a look at train | 23:35 |
| cosmicsound | heat_container_agent_tag: ussuri-dev anyone knowshow this slipped in a train deployment, this is a magnum deployment | 23:51 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!