| *** zhanglong has joined #openstack-kolla | 00:04 | |
| *** k_mouza has quit IRC | 00:12 | |
| *** igordc has quit IRC | 00:12 | |
| *** wuchunyang has joined #openstack-kolla | 00:22 | |
| *** dosaboy has joined #openstack-kolla | 00:23 | |
| *** wuchunyang has quit IRC | 00:27 | |
| *** ricolin has joined #openstack-kolla | 00:43 | |
| *** xinliang has joined #openstack-kolla | 00:45 | |
| *** zhanglong has quit IRC | 00:49 | |
| *** zhanglong has joined #openstack-kolla | 00:56 | |
| *** ricolin has quit IRC | 01:09 | |
| *** ricolin has joined #openstack-kolla | 01:09 | |
| *** zhanglong has quit IRC | 01:34 | |
| *** zhanglong has joined #openstack-kolla | 01:37 | |
| *** gregoryo has joined #openstack-kolla | 02:17 | |
| *** schwicht has joined #openstack-kolla | 02:30 | |
| *** schwicht has quit IRC | 02:50 | |
| *** wuchunyang has joined #openstack-kolla | 02:52 | |
| *** wuchunyang has quit IRC | 02:56 | |
| *** skramaja has quit IRC | 03:44 | |
| *** zhanglong has quit IRC | 03:55 | |
| *** zhanglong has joined #openstack-kolla | 04:12 | |
| *** xinliang has quit IRC | 04:19 | |
| *** zhanglong has quit IRC | 04:22 | |
| *** zhanglong has joined #openstack-kolla | 04:24 | |
| *** zhanglong has quit IRC | 04:34 | |
| *** zhanglong has joined #openstack-kolla | 04:36 | |
| *** raukadah is now known as chkumar|rover | 05:08 | |
| *** zainub_wahid has joined #openstack-kolla | 05:12 | |
| *** dave-mccowan has quit IRC | 05:30 | |
| *** skramaja has joined #openstack-kolla | 06:00 | |
| mnasiadka | yoctozepto: replied :) | 06:01 |
|---|---|---|
| mnasiadka | so people are still using kolla-ceph even though it is deprecated? nice | 06:02 |
| *** pcaruana has joined #openstack-kolla | 06:02 | |
| *** ricolin has quit IRC | 06:10 | |
| *** sri_ has joined #openstack-kolla | 06:16 | |
| mnasiadka | yoctozepto: I'll be back around 11, so we can discuss :) | 06:16 |
| sri_ | Morning | 06:20 |
| *** gregoryo has quit IRC | 06:24 | |
| *** xinliang has joined #openstack-kolla | 06:30 | |
| *** jbadiapa has joined #openstack-kolla | 06:43 | |
| *** lemko has joined #openstack-kolla | 06:59 | |
| *** zhanglong has quit IRC | 07:06 | |
| *** zhanglong has joined #openstack-kolla | 07:08 | |
| *** logan- has quit IRC | 07:09 | |
| *** logan_ has joined #openstack-kolla | 07:10 | |
| *** logan_ is now known as logan- | 07:10 | |
| *** wuchunyang has joined #openstack-kolla | 07:46 | |
| *** wuchunyang has quit IRC | 07:51 | |
| *** rgogunskiy has quit IRC | 07:54 | |
| *** rgogunskiy has joined #openstack-kolla | 07:56 | |
| *** zhanglong has quit IRC | 07:56 | |
| *** zhanglong has joined #openstack-kolla | 07:58 | |
| *** wuchunyang has joined #openstack-kolla | 07:58 | |
| *** rgogunskiy has quit IRC | 07:59 | |
| *** rgogunskiy has joined #openstack-kolla | 07:59 | |
| *** rgogunskiy has quit IRC | 08:00 | |
| *** zhanglong has quit IRC | 08:02 | |
| *** wuchunyang has quit IRC | 08:02 | |
| *** zhanglong has joined #openstack-kolla | 08:04 | |
| *** ricolin has joined #openstack-kolla | 08:06 | |
| *** ricolin has quit IRC | 08:06 | |
| *** ricolin has joined #openstack-kolla | 08:07 | |
| *** wuchunyang has joined #openstack-kolla | 08:09 | |
| *** wuchunyang has quit IRC | 08:13 | |
| yoctozepto | morning | 08:15 |
| yoctozepto | mnasiadka: lovely indeed | 08:15 |
| *** zhanglong has quit IRC | 08:16 | |
| *** ricolin has quit IRC | 08:47 | |
| *** FlorianFa has quit IRC | 08:49 | |
| yoctozepto | mnasiadka: well, I learnt our approach is less problematic than using ceph-ansible | 08:52 |
| yoctozepto | so might be the "why" | 08:52 |
| hrw | morning | 08:54 |
| hrw | infra patch fails terribly on CI but I left some notes there. please take a look once you have a time | 08:55 |
| yoctozepto | hrw: agreed | 08:57 |
| hrw | yoctozepto: to? | 08:59 |
| yoctozepto | hrw: you rebase idea, replied in review | 08:59 |
| yoctozepto | your* | 08:59 |
| hrw | k | 09:00 |
| *** FlorianFa has joined #openstack-kolla | 09:01 | |
| yoctozepto | mgoddard: please look at https://review.opendev.org/676128 (specifically my comment re mariadb - I suppose you are the most well-versed in it) | 09:05 |
| patchbot | patch 676128 - kolla - Build images with healthcheck scripts for core ser... - 19 patch sets | 09:05 |
| yoctozepto | bad news | 09:08 |
| yoctozepto | stein->train upgrade permabroken | 09:08 |
| yoctozepto | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bee/698718/2/check/kolla-ansible-ubuntu-source-upgrade-ceph/bee3111/primary/logs/ansible/upgrade | 09:08 |
| yoctozepto | mariadb not accepting some migration | 09:08 |
| *** FlorianFa has quit IRC | 09:09 | |
| hrw | hm. I see that infra images can make some poeple angry as some overrides will be killed | 09:13 |
| hrw | base -> noninfra-base -> openstack-base -> nova-base -> nova-compute | 09:13 |
| hrw | base -> mariadb | 09:13 |
| *** dougsz has joined #openstack-kolla | 09:15 | |
| *** gfidente has joined #openstack-kolla | 09:15 | |
| *** k_mouza has joined #openstack-kolla | 09:15 | |
| yoctozepto | Row size too large. The maximum row size for the used table type, not counting BLOBs, is 8126. This includes storage overhead, check the manual | 09:19 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: CI: Set zuul's ansible_python_interpreter to python2 https://review.opendev.org/698718 | 09:20 |
| yoctozepto | https://mariadb.com/kb/en/library/troubleshooting-row-size-too-large-errors-with-innodb/#solving-the-problem | 09:21 |
| hrw | mgoddard: I unified base_centos_packages similar to your change in centos8 patch. will send for review in few minutes | 09:21 |
| *** FlorianFa has joined #openstack-kolla | 09:22 | |
| yoctozepto | In MariaDB 10.1 and before, and in MySQL 5.6 and before, the COMPACT row format was the default row format. | 09:22 |
| yoctozepto | anyone brave enough to write a script to convert table types for ubuntu upgrade? :D | 09:23 |
| yoctozepto | (otherwise we have b0rken CI, just so you know ;p) | 09:23 |
| mnasiadka | yoctozepto: well, maybe in V we could write something simple to use ceph orchestrator | 09:24 |
| mnasiadka | yoctozepto: I think you are brave enough :) | 09:25 |
| yoctozepto | mnasiadka: no, already deep in some s**t | 09:26 |
| yoctozepto | in fact we might want to convert everyone | 09:29 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: base: unify CentOS packages list https://review.opendev.org/698855 | 09:29 |
| yoctozepto | becasue if you are coming from like queens or rocky then you might also have these old format | 09:29 |
| hrw | mgoddard: ^^ | 09:30 |
| yoctozepto | in centos too | 09:30 |
| yoctozepto | hrw: nice | 09:31 |
| hrw | yoctozepto: mgoddard did that in centos8 patch and it is needed for infra images too | 09:31 |
| hrw | have to also find a way to not give install_type for infra images to make sure that dockerfile will be same for bin/src | 09:32 |
| hrw | INFO:kolla.common.utils.base:Step 6/34 : ENV KOLLA_BASE_DISTRO=centos KOLLA_INSTALL_TYPE=source KOLLA_INSTALL_METATYPE=mixed KOLLA_DISTRO_PYTHON_VERSION=2.7 KOLLA_BASE_ARCH=x86_64 | 09:36 |
| hrw | this block | 09:36 |
| hrw | found | 09:37 |
| yoctozepto | mgoddard: https://bugs.launchpad.net/nova/+bug/1856296 | 09:37 |
| openstack | Launchpad bug 1856296 in kolla-ansible "upgrade to Train might fail due to mariadb row format" [Critical,Triaged] | 09:37 |
| hrw | yoctozepto: how goes epel removal? | 09:37 |
| yoctozepto | hrw: 0 progress since then | 09:38 |
| yoctozepto | mostly due to all issues that pop up :-) | 09:38 |
| yoctozepto | with mnasiadka we decided that we "get rid" of the tgt issue | 09:38 |
| yoctozepto | because with c8 users are forced to reinstall | 09:38 |
| hrw | ok | 09:39 |
| yoctozepto | so we could just drop the tgt and not care | 09:39 |
| yoctozepto | I guess other uses were not that important | 09:39 |
| yoctozepto | but still to check | 09:39 |
| hrw | I had to remove epel to have unified base image | 09:39 |
| yoctozepto | and glad you did | 09:39 |
| yoctozepto | that's the goal | 09:39 |
| hrw | now base image should be install_type agnostic | 09:40 |
| hrw | INFO:kolla.common.utils.base:Successfully tagged infra/centos-infra-base:9.1.0 | 09:41 |
| hrw | INFO:kolla.common.utils.base:Built at 2019-12-13 10:41:07.566909 (took 0:01:05.919345) | 09:41 |
| hrw | and then next build with different install_type: | 09:41 |
| hrw | INFO:kolla.common.utils.base:Successfully tagged infra/centos-infra-base:9.1.0 | 09:41 |
| hrw | INFO:kolla.common.utils.base:Built at 2019-12-13 10:41:21.249479 (took 0:00:07.551533) | 09:41 |
| hrw | all steps from cache | 09:42 |
| hrw | so noninfra-base image will not be needed | 09:42 |
| hrw | let now build all centos binary/source images | 09:42 |
| hrw | and then go infra image by infra image and get rid of install_type checks | 09:43 |
| mgoddard | yoctozepto: any idea why we have started to see this DB row isse? | 09:44 |
| hrw | yoctozepto: ops. epel is still there. we just had two different places to install it | 09:45 |
| mgoddard | should nova migrations include a step to convert tables to dynamic row format? | 09:45 |
| yoctozepto | mgoddard: not sure which 'why' you are asking about :-) | 09:45 |
| mgoddard | yoctozepto: what changed to make it fail? | 09:46 |
| hrw | mgoddard: morning | 09:46 |
| mgoddard | morning hrw | 09:46 |
| yoctozepto | mgoddard: ah, nope | 09:46 |
| yoctozepto | I skimmed over recent changes but found nothing relevant | 09:46 |
| yoctozepto | indeed it worked before for us | 09:46 |
| mgoddard | same | 09:47 |
| mgoddard | could be a mariadb change? | 09:47 |
| yoctozepto | though the issue is legit, might be just mariadb started being serious about reporting | 09:47 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: base: unify CentOS packages list https://review.opendev.org/698855 | 09:47 |
| hrw | yoctozepto: indent fixed | 09:47 |
| yoctozepto | https://downloads.mariadb.org/mariadb/+releases/ | 09:48 |
| yoctozepto | yup | 09:48 |
| hrw | hm. I may see a new issue with infra images... We do 'binary' build and get infra+binary images. then we build 'source' on same machine and get infra+source. all is fine as newly tagged 'infra' images are just using cache layers from previous build. we can push infra+binary+source images to registry. | 09:54 |
| *** wuchunyang has joined #openstack-kolla | 09:54 | |
| hrw | but if we do 'binary' on one and 'source' on other machine (or with cleaning image between) then we have separate set of layers making 'infra' images | 09:55 |
| hrw | so kolla-publish jobs will have to do infra+binary+source per distro | 09:55 |
| hrw | instead of current binary-distro, source-distro | 09:55 |
| mnasiadka | back | 09:56 |
| *** wuchunyang has quit IRC | 09:59 | |
| yoctozepto | mnasiadka: mariadb heat | 09:59 |
| yoctozepto | (not openstack heat I mean) | 09:59 |
| mnasiadka | deprecate mariadb | 09:59 |
| hrw | mgoddard: thanks for review. will go on | 10:00 |
| mnasiadka | yoctozepto: isn't there a bloody tool for mariadb upgrades that converts everything? | 10:00 |
| mnasiadka | yoctozepto: like mysql_upgrade? | 10:00 |
| yoctozepto | mnasiadka: well, you might not want to convert :-) | 10:01 |
| yoctozepto | such long rows are really not so performant anyway ;p | 10:01 |
| mnasiadka | yoctozepto: I don't care about performance for now, just care that it works - shouldn't we run mysql_upgrade/mysqlcheck to find out if we need to do anything? | 10:02 |
| mnasiadka | oh geez, I think mysql_upgrade doesn't even check it | 10:04 |
| mnasiadka | so now we need to update to DYNAMIC? uhh | 10:04 |
| yoctozepto | mnasiadka: yeah | 10:04 |
| yoctozepto | best to do it on all tables - simplest and safest for the future | 10:04 |
| mnasiadka | well, just to be sure - it would be nice to run mysqlcheck --verbose to understand if mysql_upgrade would help us here | 10:05 |
| *** wuchunyang has joined #openstack-kolla | 10:06 | |
| yoctozepto | mnasiadka: it wouldn't - it's just the default that changed | 10:06 |
| mnasiadka | well then we need to alter all tables, and we're done | 10:06 |
| mnasiadka | should we do some backup before? :D | 10:06 |
| yoctozepto | mnasiadka: we leave that exercise to users | 10:07 |
| yoctozepto | I do wonder why it suddenly started appearing | 10:07 |
| yoctozepto | newer mariadb got released | 10:07 |
| yoctozepto | and that's it | 10:07 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: base: unify CentOS packages list https://review.opendev.org/698855 | 10:08 |
| yoctozepto | as if it started checking this more rigorously | 10:08 |
| hrw | mgoddard: commit message with explanation + note + overrides compatibility | 10:08 |
| mnasiadka | yoctozepto: well, then we just need to check for mariadb version after upgrade and generate an sql statement to update all tables to row_format=DYNAMIC | 10:09 |
| yoctozepto | mnasiadka: yeah, we do :-) the question was who was to do it | 10:09 |
| *** wuchunyang has quit IRC | 10:10 | |
| osmanlicilegi | morning | 10:11 |
| hrw | osmanlicilegi: hi | 10:11 |
| hrw | osmanlicilegi: https://review.opendev.org/698789 will make rebase to 3.8 a bit easier | 10:12 |
| patchbot | patch 698789 - kolla - Use APT pinning to set preferred version of RabbitMQ - 1 patch set | 10:12 |
| osmanlicilegi | hrw: I just saw it, thank you | 10:12 |
| hrw | osmanlicilegi: side effect of looking at too many files while working on infra images | 10:12 |
| osmanlicilegi | hrw: I agree with you. pinning something everywhere is totaly a mess | 10:13 |
| hrw | osmanlicilegi: and complicates dockerfiles | 10:14 |
| osmanlicilegi | mgoddard: with erlang 22, hipe support is dropped. | 10:15 |
| osmanlicilegi | https://www.rabbitmq.com/which-erlang.html | 10:16 |
| *** wuchunyang has joined #openstack-kolla | 10:17 | |
| *** wuchunyang has quit IRC | 10:22 | |
| yoctozepto | yeah, no hype :D | 10:23 |
| hrw | did infra+binary build. started infra+source and first 78 images were built in 0:00.00.* time - pure cache use | 10:33 |
| hrw | to compare: first source one was 3:34 | 10:34 |
| openstackgerrit | Merged openstack/kolla master: Build images with healthcheck scripts for core services https://review.opendev.org/676128 | 10:38 |
| mnasiadka | whoa, I haven't noticed +2s for that | 10:39 |
| hrw | ;D | 10:39 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.opendev.org/559706 | 10:40 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: base: unify CentOS packages list https://review.opendev.org/698855 | 10:42 |
| hrw | pep8 fixed | 10:42 |
| openstackgerrit | Merged openstack/kolla master: Add Networking-MLNX to Neutron-Base packages https://review.opendev.org/689110 | 10:47 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Test control plane deployment using tenks https://review.opendev.org/620074 | 10:48 |
| yoctozepto | mnasiadka: yea, I had a comment about different treatment of mariadb wsrep status | 10:56 |
| yoctozepto | mgoddard: ^ | 10:56 |
| mnasiadka | yoctozepto: yeah, found it - I based it on clustercheck script and mariadb docs | 10:56 |
| openstackgerrit | Merged openstack/kolla master: Use APT pinning to set preferred version of RabbitMQ https://review.opendev.org/698789 | 10:59 |
| *** ricolin has joined #openstack-kolla | 10:59 | |
| mgoddard | mnasiadka: we do mysql_upgrade during upgrade | 11:01 |
| mgoddard | no mention of this in release notes that I can see: https://mariadb.com/kb/en/library/mariadb-10321-release-notes/ | 11:01 |
| mnasiadka | mgoddard: good, then we need to alter the damn tables | 11:01 |
| mgoddard | yep | 11:01 |
| *** ricolin_ has joined #openstack-kolla | 11:02 | |
| *** ricolin has quit IRC | 11:04 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: docker: fix 'Empty continuation line found' warnings https://review.opendev.org/698875 | 11:17 |
| *** ricolin_ has quit IRC | 11:18 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.opendev.org/559706 | 11:26 |
| openstackgerrit | Doug Szumski proposed openstack/kayobe master: Support custom Kolla group_vars https://review.opendev.org/698882 | 11:28 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Test control plane deployment using tenks https://review.opendev.org/620074 | 11:29 |
| *** zainub_wahid has quit IRC | 11:51 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: WIP: Create 'infra' type of images https://review.opendev.org/698753 | 12:00 |
| hrw | CI will probably still fail but that's update of work | 12:01 |
| *** EmilienM has quit IRC | 12:12 | |
| *** EmilienM has joined #openstack-kolla | 12:12 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add Debian/source aarch64 experimental job https://review.opendev.org/557659 | 12:13 |
| hrw | CentOS job got dropped | 12:17 |
| *** EmilienM is now known as EvilienM | 12:17 | |
| *** kplant has joined #openstack-kolla | 12:19 | |
| *** wuchunyang has joined #openstack-kolla | 12:22 | |
| *** mgoddard has quit IRC | 12:30 | |
| *** kozhukalov has joined #openstack-kolla | 12:30 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Test control plane deployment using tenks https://review.opendev.org/620074 | 12:33 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.opendev.org/559706 | 12:37 |
| *** xinliang has quit IRC | 12:37 | |
| *** sm806 has quit IRC | 12:38 | |
| *** sm806 has joined #openstack-kolla | 12:39 | |
| *** dave-mccowan has joined #openstack-kolla | 12:40 | |
| yoctozepto | mgoddard (hopefully around), mnasiadka: I might be able to spare some cycles to fix upgrade this evening if you don't do before | 12:40 |
| yoctozepto | per the agreed actions ;-) | 12:40 |
| yoctozepto | btw, https://ethercalc.openstack.org/kolla-infra-service-matrix | 12:41 |
| yoctozepto | I think this might be helpful to have an overview over why something breaks in this or that release/flavor | 12:41 |
| yoctozepto | hrw: is that you? | 12:44 |
| osmanlicilegi | me | 12:45 |
| yoctozepto | osmanlicilegi: ack | 12:47 |
| yoctozepto | thanks | 12:47 |
| osmanlicilegi | yw | 12:47 |
| *** kozhukalov has quit IRC | 12:48 | |
| *** kozhukalov has joined #openstack-kolla | 12:48 | |
| openstackgerrit | Merged openstack/kayobe master: Adds support for Ansible 2.8, bump mimimum to 2.6 https://review.opendev.org/697702 | 12:58 |
| *** wuchunyang has quit IRC | 12:59 | |
| *** wuchunyang has joined #openstack-kolla | 12:59 | |
| *** goldyfruit has joined #openstack-kolla | 13:07 | |
| osmanlicilegi | yoctozepto: rmq and erlang are ok | 13:11 |
| hrw | nice sheet | 13:17 |
| hrw | let me add Debian | 13:18 |
| *** negronjl has quit IRC | 13:18 | |
| *** negronjl has joined #openstack-kolla | 13:20 | |
| *** mgoddard has joined #openstack-kolla | 13:21 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Initialize fernet keys before starting keystone container https://review.opendev.org/698710 | 13:33 |
| *** ktibi has joined #openstack-kolla | 13:39 | |
| *** negronjl has quit IRC | 13:49 | |
| *** negronjl has joined #openstack-kolla | 13:50 | |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe master: Bump stack.libvirt-vm role to 1.8.0 https://review.opendev.org/698906 | 13:55 |
| *** negronjl has quit IRC | 13:59 | |
| hrw | can we kill opendaylight images? | 14:00 |
| hrw | it failed again | 14:00 |
| *** negronjl has joined #openstack-kolla | 14:03 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.opendev.org/559706 | 14:03 |
| mnasiadka | hrw: no, but mark them as unbuildable and wait for tripleo to come :) | 14:06 |
| hrw | mnasiadka: ;D | 14:07 |
| hrw | hm. why infra images patch works locally and not on CI... | 14:07 |
| hrw | (venv3) 15:10 (0s) hrw@puchatek:kolla$ tox -e build-debian-source | 14:11 |
| hrw | also works | 14:11 |
| openstackgerrit | Doug Szumski proposed openstack/kayobe master: Support custom Kolla group_vars https://review.opendev.org/698882 | 14:12 |
| *** skramaja has quit IRC | 14:13 | |
| yoctozepto | hrw: where does it fail? | 14:23 |
| ktibi | Hi guys, anyone had use kolla on RHEL os ? | 14:23 |
| hrw | yoctozepto: https://review.opendev.org/#/c/698753/ | 14:24 |
| patchbot | patch 698753 - kolla - WIP: Create 'infra' type of images - 3 patch sets | 14:24 |
| yoctozepto | ktibi: tripleo kinda do | 14:24 |
| ktibi | yoctozepto, yeah but I don't like tripleo :p | 14:24 |
| yoctozepto | ktibi: you want k-a? | 14:25 |
| hrw | ktibi: use kolla as 'do builds with rhel as host' or 'do builds with rhel as container os' or 'deploys on rhel'? | 14:25 |
| yoctozepto | then follow for centos and let us know ;D | 14:25 |
| ktibi | yeah sorry , k-a | 14:25 |
| mgoddard | ktibi: it is 'supported' but untested | 14:26 |
| ktibi | hrw, yep the question is : anyone had deploy kolla with k-a on a rhel system ? | 14:26 |
| *** rgogunskiy has joined #openstack-kolla | 14:26 | |
| ktibi | mgoddard, ok thx ;) | 14:26 |
| *** rgogunskiy has quit IRC | 14:31 | |
| *** vmixor has joined #openstack-kolla | 14:32 | |
| *** dmellado has quit IRC | 14:36 | |
| *** dmellado has joined #openstack-kolla | 14:39 | |
| *** igordc has joined #openstack-kolla | 14:40 | |
| *** negronjl has quit IRC | 14:41 | |
| *** sri_ has quit IRC | 14:43 | |
| *** negronjl has joined #openstack-kolla | 14:44 | |
| *** graysonh has joined #openstack-kolla | 14:45 | |
| yoctozepto | INFO:kolla.common.utils.ceph-base:Step 1/7 : FROM kolla/debian-infra-base:9.1.0 | 14:48 |
| yoctozepto | INFO:kolla.common.utils.ceph-base:Step 2/7 : FROM kolla/debian-source-base:9.1.0 | 14:48 |
| yoctozepto | looking wrong | 14:48 |
| yoctozepto | hrw: ^ | 14:48 |
| yoctozepto | I guess you locally have this other image | 14:48 |
| mgoddard | should we call it debian-base? | 14:50 |
| mgoddard | debian-mariadb etc | 14:50 |
| openstackgerrit | Will Szumski proposed openstack/kolla-ansible master: Add extras directory to prometheus config https://review.opendev.org/688374 | 14:56 |
| *** kozhukalov has quit IRC | 14:56 | |
| vmixor | Hi | 14:57 |
| mgoddard | hi vmixor | 14:59 |
| openstackgerrit | Merged openstack/kayobe master: Improve detection of base path https://review.opendev.org/655864 | 15:01 |
| openstackgerrit | Will Szumski proposed openstack/kayobe-config master: Synchronise with kayobe for Train https://review.opendev.org/698927 | 15:01 |
| vmixor | I am trying to deploy Train and I am getting error: "The conditional check 'api_interface_address is not defined' failed. " What should I set to this variable? It had been initiated automaticaly up to Stein. | 15:02 |
| vmixor | hi mgoddard | 15:03 |
| yoctozepto | vmixor: does it mention kolla_address in there? | 15:03 |
| yoctozepto | if so, please ensure ansible and kolla_ansible are both installed in the same environment | 15:03 |
| yoctozepto | verify with pip list | 15:03 |
| yoctozepto | (or pip freeze) | 15:04 |
| graysonh | vmixor: In response to your reply Thursday, I am setting the bootstrap_bs label on the devices. Each host has 16 spinny disks and 8 SSDs and I'm trying to move the block.db to the latter while keeping block on the former. Here is what they look like right now https://pastebin.com/BT6tawcj, which is what my interpretation of the docs make me think they should be, but kolla never detects them | 15:04 |
| graysonh | And yeah, I know ceph kolla support is going away, but we've had no luck with ceph-ansible yet, and we already have a cluster in prod deployed this way that needs to be expanded :( | 15:04 |
| vmixor | yoctozepto: ah, you are right, Ansible is not present in that virtual environment | 15:06 |
| hrw | yoctozepto: infra-base got built just before infra-ceph-base etc | 15:07 |
| hrw | yoctozepto: have to find why it goes that way | 15:08 |
| hrw | yoctozepto: locally it works with clean docker | 15:08 |
| yoctozepto | hrw: I mean source-base | 15:08 |
| yoctozepto | look more carefully | 15:08 |
| hrw | yoctozepto: locally it works with clean docker | 15:08 |
| yoctozepto | mgoddard: looks like we should guard against the case that ansible is called from outside the virtualenv | 15:08 |
| yoctozepto | hrw: ok | 15:09 |
| hrw | yoctozepto: I use new namespace per build even | 15:09 |
| openstackgerrit | Doug Szumski proposed openstack/kayobe master: Support custom Kolla group_vars https://review.opendev.org/698882 | 15:13 |
| vmixor | yoctozepto: now it works, thanks | 15:17 |
| vmixor | graysonh: tbh, I have no experience with such setup in production, maybe someone else there? | 15:19 |
| hrw | 1444 images built with infra patch. | 15:20 |
| hrw | https://paste.centos.org/view/c3f6afad | 15:21 |
| yoctozepto | vmixor: yw | 15:22 |
| mgoddard | yoctozepto: hmm. a better check would be to run a playbook that uses ansible_playbook_python to import kolla_ansible | 15:24 |
| *** wuchunyang has quit IRC | 15:28 | |
| yoctozepto | mgoddard: mhm | 15:31 |
| yoctozepto | mgoddard: I guess it's the mariadb upgrade fix for this for rc3 and final release | 15:32 |
| mgoddard | hmm, yes I suppose we need to block the release on it | 15:34 |
| hrw | http://zuul.openstack.org/status/change/557659,58 runs on new arm64 pipeline ;D | 15:43 |
| yoctozepto | hrw: at least one good info today ;D | 15:43 |
| hrw | yoctozepto: I have few more good infos today. 4th season of The Expanse is on Amazon Prime. My infra images work (locally). Few more new patches done. Jira card for infra images done. Got info about new arm64 nodes work. | 15:45 |
| yoctozepto | :D | 15:45 |
| *** kozhukalov has joined #openstack-kolla | 15:46 | |
| hrw | week late birthday gifts arrived at their destination, xmas gifts arrived in paczkomat... | 15:47 |
| hrw | and few more | 15:47 |
| *** jbadiapa has quit IRC | 15:48 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add Debian/source aarch64 CI job https://review.opendev.org/557659 | 15:52 |
| hrw | no longer experimental | 15:53 |
| *** chkumar|rover is now known as raukadah | 15:53 | |
| *** ricolin_ has joined #openstack-kolla | 15:53 | |
| *** ktibi has quit IRC | 15:54 | |
| *** kozhukalov has quit IRC | 15:56 | |
| vmixor | May I ask for review of https://review.opendev.org/#/c/695054/ and https://review.opendev.org/#/c/674579/ | 15:58 |
| patchbot | patch 695054 - kolla-ansible - Support for custom api-paste files - 6 patch sets | 15:58 |
| patchbot | patch 674579 - kolla-ansible - Adds support of CADF notifying for other APIs next... - 10 patch sets | 15:58 |
| yoctozepto | asking is permitted :-) | 16:01 |
| vmixor | :) | 16:02 |
| *** schwicht has joined #openstack-kolla | 16:07 | |
| *** schwicht has quit IRC | 16:07 | |
| mgoddard | is anyone looking at https://bugs.launchpad.net/kolla-ansible/+bug/1856296? | 16:08 |
| openstack | Launchpad bug 1856296 in kolla-ansible "upgrade to Train might fail due to mariadb row format" [Critical,Triaged] | 16:08 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: CI: Set zuul's ansible_python_interpreter to python2 https://review.opendev.org/698945 | 16:08 |
| openstackgerrit | Will Szumski proposed openstack/kayobe-config master: Synchronise with kayobe for Train https://review.opendev.org/698927 | 16:09 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/rocky: CI: Set zuul's ansible_python_interpreter to python2 https://review.opendev.org/698946 | 16:09 |
| openstackgerrit | Will Szumski proposed openstack/kayobe-config master: Synchronise with kayobe for Train https://review.opendev.org/698927 | 16:11 |
| *** igordc has quit IRC | 16:15 | |
| yoctozepto | mgoddard: /me as promised | 16:15 |
| mgoddard | yoctozepto: great. I missed the promise | 16:15 |
| yoctozepto | mgoddard: already split off backport of your change | 16:15 |
| mgoddard | yoctozepto: will need to combine with ansible_python_interpreter | 16:15 |
| yoctozepto | yeah, yeah | 16:15 |
| mgoddard | doesn't the change need to go to train? | 16:16 |
| yoctozepto | mgoddard: to train | 16:16 |
| mgoddard | k | 16:16 |
| openstackgerrit | Will Szumski proposed openstack/kayobe-config master: Synchronise with kayobe for Train https://review.opendev.org/698927 | 16:32 |
| *** ricolin_ has quit IRC | 16:36 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/train: Fix upgrades from older (<=10.1) MariaDB releases https://review.opendev.org/698957 | 16:39 |
| yoctozepto | mgoddard, mnasiadka: a bit dirty ^ | 16:39 |
| yoctozepto | W-1 mostly due to missing comments, no_log and CI trimmed to just one job | 16:40 |
| yoctozepto | testing otherwise | 16:40 |
| yoctozepto | I checked my deploy on c7 and all tables are dynamic | 16:40 |
| yoctozepto | so we should be fine going for all in Ubuntu | 16:40 |
| *** kozhukalov has joined #openstack-kolla | 16:41 | |
| yoctozepto | and c7 if need be (for example for some old deployments) | 16:41 |
| yoctozepto | reviews welcome still ;-) | 16:41 |
| mgoddard | looks ok to me | 16:42 |
| mgoddard | I have left a very helpful comment | 16:42 |
| yoctozepto | mgoddard: game changer! | 16:43 |
| mgoddard | won't we have the same problem in stein for centos? | 16:44 |
| mgoddard | stein upgraded to 10.3 | 16:44 |
| yoctozepto | mgoddard: CI is checking | 16:44 |
| yoctozepto | we can always backport it | 16:44 |
| mgoddard | who needs CI when you have me | 16:44 |
| yoctozepto | but maybe relevant migrations are only in Train | 16:44 |
| mgoddard | true, could be | 16:44 |
| yoctozepto | ok, proposing to drop CI | 16:44 |
| mgoddard | :) | 16:44 |
| openstackgerrit | Merged openstack/kayobe-config master: Synchronise with kayobe for Train https://review.opendev.org/698927 | 16:44 |
| yoctozepto | all blame on Mark | 16:45 |
| yoctozepto | for now let Train amend deploys | 16:45 |
| yoctozepto | if ppl come complaining we can backport | 16:45 |
| yoctozepto | Stein already has mariadb issues :-) | 16:45 |
| yoctozepto | wonder how we avoided the circus with Ubuntu | 16:46 |
| yoctozepto | but somehow we did | 16:46 |
| yoctozepto | :D | 16:46 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/train: Fix upgrades from older (<=10.1) MariaDB releases https://review.opendev.org/698957 | 16:47 |
| yoctozepto | it did not start the job so I fixed the whitespace, eh ;D | 16:47 |
| yoctozepto | some reply from Matt here: https://bugs.launchpad.net/kolla-ansible/+bug/1856296 | 16:47 |
| openstack | Launchpad bug 1856296 in kolla-ansible "upgrade to Train might fail due to mariadb row format" [Critical,Triaged] | 16:47 |
| yoctozepto | mgoddard: ^ | 16:47 |
| mgoddard | I saw. Fair enough I suppose | 16:48 |
| yoctozepto | notified, closed | 16:49 |
| yoctozepto | all happy | 16:49 |
| *** schwicht has joined #openstack-kolla | 16:55 | |
| mnasiadka | back again - crazy day | 16:57 |
| yoctozepto | mnasiadka: you don't say | 16:58 |
| mnasiadka | it's good that it's Friday... | 16:58 |
| mnasiadka | ok, let's get back to this bloody fernet | 16:59 |
| *** graysonh has quit IRC | 17:01 | |
| *** hjensas has joined #openstack-kolla | 17:03 | |
| hrw | https://review.opendev.org/#/c/557659/ - please review and accept. it works | 17:12 |
| patchbot | patch 557659 - kolla - Add Debian/source aarch64 CI job - 59 patch sets | 17:12 |
| hrw | and will allow me to close oldest Jira card ;D | 17:13 |
| mnasiadka | that's interesting, keystone bootstrap task returns always changed=true | 17:20 |
| hrw | and it is now green on all CI jobs | 17:25 |
| hrw | yoctozepto: if you want another trivial change - go for it. | 17:27 |
| * hrw -> weekend ;D | 17:27 | |
| hrw | have a nice weekend folks | 17:27 |
| *** wuchunyang has joined #openstack-kolla | 17:29 | |
| *** openstackgerrit has quit IRC | 17:29 | |
| *** openstackgerrit has joined #openstack-kolla | 17:31 | |
| openstackgerrit | Doug Szumski proposed openstack/kayobe master: Support customising nova-conductor group for cells https://review.opendev.org/698970 | 17:31 |
| openstackgerrit | Doug Szumski proposed openstack/kayobe master: WIP: Cells documentation https://review.opendev.org/698971 | 17:31 |
| *** wuchunyang has quit IRC | 17:34 | |
| *** kozhukalov has quit IRC | 17:35 | |
| yoctozepto | mgoddard, mnasiadka: https://review.opendev.org/698957 it worked :D | 17:39 |
| patchbot | patch 698957 - kolla-ansible (stable/train) - Fix upgrades from older (<=10.1) MariaDB releases - 2 patch sets | 17:39 |
| *** graysonh has joined #openstack-kolla | 17:44 | |
| mnasiadka | mgoddard: cool, but I see a module mysql_command would be useful (if it existed) ;) | 17:49 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/train: Fix upgrades from older (<=10.1) MariaDB releases https://review.opendev.org/698957 | 17:52 |
| yoctozepto | mnasiadka: we rarely have to resort to doing this | 17:52 |
| mnasiadka | yeah, s/mgoddard/yoctozepto/ ;) | 17:53 |
| yoctozepto | the only pending issue is the check for ansible+kolla_ansible in the same venv | 17:54 |
| yoctozepto | complaints about the kolla_address missing are too numerous | 17:54 |
| *** dougsz has quit IRC | 17:57 | |
| *** k_mouza has quit IRC | 17:57 | |
| *** wuchunyang has joined #openstack-kolla | 18:06 | |
| *** wuchunyang has quit IRC | 18:11 | |
| yoctozepto | https://bugs.launchpad.net/kolla-ansible/+bug/1856346 | 18:16 |
| openstack | Launchpad bug 1856346 in kolla-ansible "check for kolla_ansible being installed does not take into account it's ansible that has to see it" [Medium,Triaged] | 18:16 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Initialize fernet keys before starting keystone container https://review.opendev.org/698710 | 18:25 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Initialize fernet keys before starting keystone container https://review.opendev.org/698710 | 18:26 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Initialize fernet keys before starting keystone container https://review.opendev.org/698710 | 18:30 |
| *** wuchunyang has joined #openstack-kolla | 18:30 | |
| *** wuchunyang has quit IRC | 18:35 | |
| *** gfidente has quit IRC | 18:40 | |
| *** wuchunyang has joined #openstack-kolla | 18:42 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Initialize fernet keys before starting keystone container https://review.opendev.org/698710 | 18:43 |
| *** wuchunyang has quit IRC | 18:47 | |
| *** lemko has quit IRC | 18:48 | |
| *** wuchunyang has joined #openstack-kolla | 18:51 | |
| *** vkmc has joined #openstack-kolla | 18:53 | |
| vkmc | hi folks | 18:53 |
| vkmc | maybe someone can help me out | 18:53 |
| vkmc | I'm building an image with kolla, it never finishes kolla_start | 18:54 |
| vkmc | looking at the error code, the container runs out of memory | 18:54 |
| vkmc | the host has memory | 18:54 |
| vkmc | but apparently there is something crazy going inside the container | 18:54 |
| vkmc | some suggestions on how to debug this? | 18:54 |
| vkmc | attaching a shell to the container is not possible because it respawns every 1min or less | 18:54 |
| *** wuchunyang has quit IRC | 18:59 | |
| *** igordc has joined #openstack-kolla | 19:05 | |
| klindgren_ | change the start command in /etc/kolla/<whatever/config.json to something like sleep 100000; <normal command> | 19:07 |
| klindgren_ | then you should be able to exec into the container and trying running the command yourself to see what happens | 19:08 |
| vkmc | klindgren_++ | 19:13 |
| vkmc | thanks, I will try that | 19:14 |
| *** rgogunskiy has joined #openstack-kolla | 19:31 | |
| *** rgogunskiy has quit IRC | 19:36 | |
| yoctozepto | mgoddard, mnasiadka: if you were so kind to approve patches to 3 stable branches to unbreak them, thanks :-) | 19:46 |
| *** kplant has quit IRC | 20:02 | |
| *** kozhukalov has joined #openstack-kolla | 20:10 | |
| *** vmixor has quit IRC | 20:12 | |
| *** gmann is now known as gmann_afk | 20:13 | |
| *** wuchunyang has joined #openstack-kolla | 20:14 | |
| *** wuchunyang has quit IRC | 20:19 | |
| *** wuchunyang has joined #openstack-kolla | 20:25 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Initialize fernet keys before starting keystone container https://review.opendev.org/698710 | 20:26 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Initialize fernet keys before starting keystone container https://review.opendev.org/698710 | 20:27 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Initialize fernet keys before starting keystone container https://review.opendev.org/698710 | 20:28 |
| *** wuchunyang has quit IRC | 20:30 | |
| *** kozhukalov has quit IRC | 20:31 | |
| *** graysonh has quit IRC | 20:35 | |
| *** kozhukalov has joined #openstack-kolla | 20:37 | |
| *** oncall-pokemon has joined #openstack-kolla | 20:56 | |
| *** EvilienM is now known as EmilienM | 20:57 | |
| *** wuchunyang has joined #openstack-kolla | 21:16 | |
| *** vmixor has joined #openstack-kolla | 21:17 | |
| *** wuchunyang has quit IRC | 21:20 | |
| *** wuchunyang has joined #openstack-kolla | 21:27 | |
| *** igordc has quit IRC | 21:30 | |
| *** wuchunyang has quit IRC | 21:32 | |
| *** wuchunyang has joined #openstack-kolla | 21:42 | |
| *** schwicht has quit IRC | 21:47 | |
| *** wuchunyang has quit IRC | 21:48 | |
| *** wuchunyang has joined #openstack-kolla | 21:49 | |
| *** vmixor has quit IRC | 21:53 | |
| *** wuchunyang has quit IRC | 21:54 | |
| *** kozhukalov has quit IRC | 21:54 | |
| *** logan- has quit IRC | 22:04 | |
| *** logan- has joined #openstack-kolla | 22:06 | |
| *** schwicht has joined #openstack-kolla | 22:10 | |
| *** KeithMnemonic1 has joined #openstack-kolla | 22:15 | |
| *** schwicht has quit IRC | 22:18 | |
| *** KeithMnemonic has quit IRC | 22:19 | |
| *** schwicht has joined #openstack-kolla | 22:20 | |
| *** wuchunyang has joined #openstack-kolla | 22:42 | |
| *** schwicht has quit IRC | 22:44 | |
| *** hjensas has quit IRC | 22:45 | |
| *** wuchunyang has quit IRC | 22:47 | |
| *** pcaruana has quit IRC | 22:48 | |
| *** wuchunyang has joined #openstack-kolla | 22:50 | |
| *** wuchunyang has quit IRC | 22:54 | |
| *** wuchunyang has joined #openstack-kolla | 22:56 | |
| openstackgerrit | Will Szumski proposed openstack/kayobe master: Switch default IPA images to centos https://review.opendev.org/699046 | 22:56 |
| *** wuchunyang has quit IRC | 23:06 | |
| *** schwicht has joined #openstack-kolla | 23:11 | |
| *** KeithMnemonic1 has quit IRC | 23:19 | |
| *** wuchunyang has joined #openstack-kolla | 23:33 | |
| *** wuchunyang has quit IRC | 23:42 | |
| *** wuchunyang has joined #openstack-kolla | 23:48 | |
| *** wuchunyang has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!