Friday, 2019-06-07

*** igordc has joined #openstack-kolla00:38
*** Sravan has joined #openstack-kolla01:32
*** Sravan has quit IRC01:37
*** Talion has joined #openstack-kolla01:44
*** schwicht has quit IRC01:55
*** gregoryo has joined #openstack-kolla02:00
sean-k-mooneyJeffrey4l: o/02:00
Jeffrey4lhey sean-k-mooney, sup02:01
sean-k-mooneystephen asked me to take over https://review.opendev.org/#/c/661366/ for him since he is not really that familar with kolla-ansible02:01
sean-k-mooneybut i was wondering if we wanted to go with your version or not02:02
sean-k-mooneyJeffrey4l: or ask if you want to take it over but if not ill set up a kolla env and start woring on the upgrade and stop changes02:04
Jeffrey4lsean-k-mooney,  either is ok. i can take this. i will restore mine and abandon stephen's02:06
sean-k-mooneyJeffrey4l: ok if you are happy to contiue working on it thats cool with me. if you dont have time then ill take it over from stephen instead02:07
Jeffrey4lokay, i can take over it. thanks.02:08
*** dpawlik has joined #openstack-kolla02:11
*** dpawlik has quit IRC02:15
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Remove nova-consoleauth  https://review.opendev.org/66142702:17
*** schwicht has joined #openstack-kolla02:19
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Remove nova-consoleauth  https://review.opendev.org/66142702:21
*** goldyfruit has quit IRC02:24
*** dpawlik has joined #openstack-kolla02:27
*** sean-k-mooney has quit IRC02:28
*** goldyfruit has joined #openstack-kolla02:29
*** dpawlik has quit IRC02:32
*** JamesBenson has joined #openstack-kolla02:50
*** goldyfruit has quit IRC02:53
*** threestrands has joined #openstack-kolla02:54
*** sean-k-mooney has joined #openstack-kolla03:00
*** schwicht has quit IRC03:01
*** whoami-rajat has joined #openstack-kolla03:10
*** sean-k-mooney has quit IRC03:14
*** sean-k-mooney has joined #openstack-kolla03:16
*** JamesBenson has quit IRC03:23
*** JamesBenson has joined #openstack-kolla03:24
*** igordc has quit IRC03:25
*** JamesBenson has quit IRC03:28
*** Sravan has joined #openstack-kolla03:29
*** Sravan has quit IRC03:37
*** Sravan has joined #openstack-kolla03:49
*** Sravan has quit IRC03:50
*** shyamb has joined #openstack-kolla03:51
*** Sravan has joined #openstack-kolla03:53
*** threestrands has quit IRC04:12
*** Sravan_ has joined #openstack-kolla04:15
*** Sravan has quit IRC04:18
*** sean-k-mooney has quit IRC04:19
*** sean-k-mooney has joined #openstack-kolla04:21
*** Sravan has joined #openstack-kolla04:30
*** radeks__ has joined #openstack-kolla04:31
*** Sravan__ has joined #openstack-kolla04:33
*** Sravan_ has quit IRC04:34
*** Sravan has quit IRC04:36
*** Sravan__ has quit IRC04:43
*** dpawlik has joined #openstack-kolla04:43
*** dpawlik has quit IRC04:48
*** shyamb has quit IRC04:57
*** shyam89 has joined #openstack-kolla04:57
*** shyam89 has quit IRC05:08
*** shyamb has joined #openstack-kolla05:18
*** dpawlik has joined #openstack-kolla05:30
*** shyamb has quit IRC05:37
*** luksky has joined #openstack-kolla05:44
*** shyamb has joined #openstack-kolla05:48
*** cah_link has joined #openstack-kolla06:03
*** jbadiapa has quit IRC06:13
*** gregoryo has quit IRC06:16
*** itlinux_ has joined #openstack-kolla06:34
*** itlinux has quit IRC06:34
*** pcaruana has joined #openstack-kolla06:35
*** itlinux_ has quit IRC06:54
*** itlinux has joined #openstack-kolla07:00
*** itlinux has quit IRC07:05
openstackgerritMichal Nasiadka proposed openstack/kolla master: Bump up pkg versions in kolla-toolbox  https://review.opendev.org/64241607:06
*** itlinux has joined #openstack-kolla07:12
yoctozeptohttps://review.opendev.org/#/c/662209/ - deploy guide blocker, may one +2?07:12
mnasiadkadone07:14
yoctozeptomnasiadka: thanks07:14
*** shyamb has quit IRC07:30
*** rwadolowski_ has joined #openstack-kolla07:45
*** Wasaac has joined #openstack-kolla07:52
*** Wasaac has quit IRC07:54
*** Wasaac has joined #openstack-kolla07:54
*** shyamb has joined #openstack-kolla07:57
*** JamesBenson has joined #openstack-kolla08:08
yoctozeptohttps://review.opendev.org/#/c/651601/ damn, no way it failed, were happy too soon08:10
yoctozeptocurl: (7) Failed connect to www.percona.com:443; Connection timed out08:10
yoctozeptoahh08:10
*** dougsz has joined #openstack-kolla08:11
*** JamesBenson has quit IRC08:12
*** k_mouza has joined #openstack-kolla08:12
mgoddardyoctozepto: did you get anywhere with https://review.opendev.org/#/c/658456/1 ?08:15
*** gfidente has joined #openstack-kolla08:16
yoctozeptomgoddard: sharing first name does not make us the same person :-)08:16
hrwmorning08:17
hrwyay percona ;(08:17
mgoddardyoctozepto: argh sorry! I even wrote down your name and cross checked :( still asleep08:17
yoctozeptomorning hrw08:18
*** k_mouza has quit IRC08:18
yoctozeptomgoddard: no problem08:18
mgoddardradeks__ is the owner of https://review.opendev.org/#/c/65845608:18
hrwmgoddard: s/owner/author08:19
mgoddardhrw: yeah08:19
openstackgerritMerged openstack/kolla-ansible master: Add deploy guide stuff to irrelevant file list of zuul  https://review.opendev.org/66220908:19
mgoddardhrw: although gerrit says also owner08:19
yoctozeptohrw: nah, gerrit says owner08:19
yoctozepto:D08:19
*** k_mouza has joined #openstack-kolla08:19
hrwgerrit...08:19
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Supporting monitoring time synchronisation with Monasca  https://review.opendev.org/66372208:20
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Add load monitoring plugin config for Monasca  https://review.opendev.org/66372308:20
hrwI consider myself to be author of patches I sent to gerrit. do not own them - they are now community owned08:20
yoctozeptohrw: I think this is like the 'owner' in 'service owner'08:21
yoctozeptonot owning too much08:21
hrwyoctozepto: probably another thing of 'you have to be more fluent in english culture to get it'08:21
mgoddardowners own reviews, authors created patches, committers last committed patches08:21
yoctozeptohrw: yeah, been there, felt that08:22
yoctozeptohttps://dictionary.cambridge.org/dictionary/english/process-owner08:25
yoctozeptocould not find better match08:26
mnasiadkamgoddard: I still feel percona issue is ovh related, couldn't find a job with percona issue that did not run in ovh08:26
mnasiadkahi guys08:26
yoctozeptobut the dictionary allows the definition of owner as the 'person responsible'08:26
yoctozeptomnasiadka: you mean that *did* run in ovh08:27
mnasiadkayoctozepto: I meant that I couldn't find a failed job due to percona issue, that ran on a *DIFFERENT* Zuul provider than OVH08:27
yoctozeptoI see08:28
yoctozeptoreally odd08:28
mnasiadkait's 10.30 am... I need coffee08:28
mnasiadkayoctozepto: it's not odd, let's say I'm used to it ;)08:28
openstackgerritMerged openstack/kolla master: Debian/Ubuntu: move to Python 3 for source images  https://review.opendev.org/64237508:29
openstackgerritMerged openstack/kolla master: Debian/Ubuntu: more python3 packages for binary  https://review.opendev.org/66337508:29
yoctozeptomnasiadka: nonetheless, it makes little sense without further insight08:29
yoctozeptoI wonder whether radeks__' way would help08:29
yoctozeptobut in general 5 times in 7 second intervals sound reasanable if the issues are really temporary08:30
yoctozeptolost long-running jobs are bad-bad-bad08:30
mnasiadkait's not long running, it fails just in the beginning :D08:32
mgoddardmnasiadka: have you seen jobs pass in OVH?08:34
yoctozeptomnasiadka: still 16 minutes are lost08:34
yoctozeptomgoddard: was about to ask that08:35
yoctozeptomaybe percona does not like ovh, I know I had a hosting provider once whose firewalls notoriously blocked ovh because of 'attacks'08:36
yoctozeptobut it was like years ago08:36
yoctozeptoand we did not care about ovh anyway because the target was the browser of a user, not ovh08:37
mgoddardso for percona packaging key we actually ship it in the kolla repo. That might be an easy fix08:38
yoctozeptomgoddard: +108:38
yoctozeptobut is it not the same server that packages are downloaded from anyway?08:39
mnasiadkamgoddard: sometimes yes08:41
yoctozepto> <yoctozepto> but is it not the same server that packages are downloaded from anyway?08:43
yoctozeptothe same, just checked08:43
yoctozeptoeven though the domain is different08:43
yoctozeptoit resolves to the same IP address08:43
yoctozeptothen it can fail just a bit later anyway :P08:44
mgoddardyoctozepto: true :(08:45
openstackgerritMerged openstack/kolla-ansible stable/rocky: CI Test: Support being source of a Ceph upgrade  https://review.opendev.org/65816208:46
yoctozeptomgoddard: out of curiosity - is there somewhere a documented reasoning behind percona vs maria?08:48
openstackgerritMerged openstack/kolla master: Ignore EM releases in version-check.py  https://review.opendev.org/66018608:49
mgoddardyoctozepto: as opposed to mariabackup?08:53
openstackgerritMerged openstack/kolla-ansible stable/stein: Test Ceph upgrade in CI  https://review.opendev.org/66356508:53
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix the deploy guide build failed  https://review.opendev.org/66359308:53
openstackgerritMerged openstack/kolla-ansible stable/stein: there is no easy_install in Debian 'buster'  https://review.opendev.org/66273308:53
openstackgerritMerged openstack/kolla-ansible master: Update contributor guide  https://review.opendev.org/66044208:53
mgoddardI'm not aware of anything08:53
openstackgerritMerged openstack/kolla-ansible master: Add blazar to fluentd aggregation  https://review.opendev.org/66333008:53
openstackgerritMerged openstack/kolla-ansible master: [heat] Multi-region support for bootstrap  https://review.opendev.org/66332708:53
openstackgerritMerged openstack/kolla-ansible master: Hide logs when looping over passwords  https://review.opendev.org/66332108:53
openstackgerritMerged openstack/kolla-ansible master: Fix docs - do not mention the old, deprecated CLI clients  https://review.opendev.org/66357408:53
*** heikkine has quit IRC08:53
openstackgerritMerged openstack/kolla-ansible master: Fix docs - simply run the script instead of sourcing it  https://review.opendev.org/66356608:53
mgoddardmerge fest!08:53
mgoddardyoctozepto: that is an iteresting idea - switch to mariabackup08:54
yoctozeptomgoddard: this and others, the whole maria stack08:54
yoctozeptofrom engine to HA to backup08:54
mgoddardyoctozepto: is it better?08:55
mgoddardwould it be painful to migrate a running cluster?08:56
yoctozeptothey always claim it is better but never really did enough percona to tell, probably irrelevant details for 99% of deployments08:56
yoctozeptono idea about migration, I only did mysql->maria in the past08:57
yoctozeptoobviously offline08:57
mgoddardswitching to mariabackup is tempting if it is compatible - would solve our percona headaches08:58
yoctozeptoMariabackup is based on XtraBackup08:58
yoctozeptofrom their docs08:59
yoctozeptomgoddard: yup, unless maria repos result in more instabilities :P08:59
yoctozeptooh, I see now, you are only using percona for the backup09:00
yoctozeptothought it was more09:00
yoctozeptothen the answer is clear09:00
yoctozeptoDROP PERCONA;09:01
yoctozeptoUSE MARIA;09:01
mgoddardyeah, seems like a good plan. Just needs someone to do it09:01
yoctozeptowell, replacing for new installs would be easy09:03
yoctozeptobut upgrades, eh :D09:03
hrw10:29 < openstackgerrit> Merged openstack/kolla master: Debian/Ubuntu: move to Python 3 for source images  https://review.opendev.org/64237509:04
*** k_mouza has quit IRC09:04
hrw10:29 < openstackgerrit> Merged openstack/kolla master: Debian/Ubuntu: more python3 packages for binary  https://review.opendev.org/66337509:04
hrwYES!09:05
mgoddardhrw: congrats!09:05
hrwhttps://review.opendev.org/662429 https://review.opendev.org/663300 and I may consider my work done for a moment (or two)09:05
yoctozeptohrw: congrats09:06
mgoddardyoctozepto: it's only backups, so we would just switch to a new backup tool. Restore has always been out of scope09:06
mgoddardyoctozepto: old backups could still be restored (manually)09:06
hrwlet me revive https://review.opendev.org/#/c/557659/ then09:07
mgoddardhrw: did you see my questions about rabbitmq?09:07
yoctozeptomgoddard: yeah, also by looking at the docs, the replacement for either new install or upgrade would be equally easy09:07
openstackgerritMerged openstack/kolla-ansible master: Support multi-region discovery of Nova cells  https://review.opendev.org/66331909:08
openstackgerritMerged openstack/kolla-ansible master: [cloudkitty] Remove unused option from configuration file template  https://review.opendev.org/66297209:08
openstackgerritMerged openstack/kolla-ansible master: Remove `/%(tenant_id)s` suffix from v2.1 endpoints  https://review.opendev.org/66252809:08
hrwmgoddard: not yet. backlog or gerrit?09:08
openstackgerritMerged openstack/kolla-ansible master: Add ansible_nodename (system hostname) to /etc/hosts  https://review.opendev.org/66046309:08
openstackgerritMerged openstack/kolla-ansible master: Add support for elasticsearch TLS and authentication in fluentd  https://review.opendev.org/66221509:08
openstackgerritMerged openstack/kolla-ansible master: Fix monasca grafana organisation check  https://review.opendev.org/66243309:08
openstackgerritMerged openstack/kolla-ansible master: Fix the document of external ceph for gnocchi  https://review.opendev.org/66222009:08
mgoddardhrw: gerrit09:08
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add kolla-build-debian-source-arm64 experimental job  https://review.opendev.org/55765909:08
yoctozeptomerge flood09:08
hrwmgoddard: replying09:10
mgoddardhrw: thanks09:10
*** Sravan has joined #openstack-kolla09:10
hrwreplied09:11
openstackgerritKrzysztof Klimonda proposed openstack/kolla-ansible master: Internal OpenStack endpoints encryption spec  https://review.opendev.org/66386509:14
*** Sravan has quit IRC09:15
*** k_mouza has joined #openstack-kolla09:19
hrwnow let's see how that debian/aarch64 job will go09:23
*** priteau has joined #openstack-kolla09:25
openstackgerritKrzysztof Klimonda proposed openstack/kolla-ansible master: Internal OpenStack endpoints encryption spec  https://review.opendev.org/66386509:29
*** zbr has joined #openstack-kolla09:30
hrwhttps://review.opendev.org/#/c/557659/ needs reviews09:30
mgoddardmnasiadka: or hrw could you add +2A back to https://review.opendev.org/#/c/651210/ - only one line changed09:33
mnasiadkamgoddard: done09:34
mgoddardthanks09:35
yoctozeptomgoddard: but you do mirror percona09:35
yoctozeptoI was evaluating possibility of move to maria09:36
yoctozeptoand discovered this :D09:36
yoctozepto-e "s|^baseurl=https://repo.percona.com|baseurl=http://mirror.gra1.ovh.openstack.org:8080/percona|g" \09:36
mgoddardyoctozepto: yeah - I saw that. It doesn't apply because the repo file has 'baseurl = ' not 'baseurl='09:37
mgoddardyoctozepto: I looked for the repo on one of the publicly visible mirrors (RAX?) and couldn't see one for percona09:37
yoctozeptohttp://mirror.gra1.ovh.openstack.org:8080/percona/yum/09:38
yoctozeptolooks legit09:38
yoctozeptoand has both keys09:38
yoctozeptoyeah09:38
mgoddardhmm09:39
yoctozeptomgoddard: may I prepare a patch?09:39
mgoddardyoctozepto: always!09:40
yoctozeptomgoddard: as for MariaDB - I could not find whether upstream is used, probably not?09:42
*** k_mouza has quit IRC09:42
*** k_mouza has joined #openstack-kolla09:43
hrwyoctozepto: we moved to use mariadb from distro repos09:43
hrwyoctozepto: Debian was on 10.1 while ubuntu/centos images used 10.0 one. Galera support got merged into 10.1 so deployment had to care about 'is it 10.0 or 10.1' so we moved everyone to 10.109:45
*** Wasaac has quit IRC09:45
yoctozeptohrw: how did you move everyone to 10.1 by not using upstream? did the distros agree so well?09:46
*** Wasaac has joined #openstack-kolla09:46
hrwyoctozepto: RDO had 10.1 we did not used earlier so it served centos. do not remember how it went with ubuntu. let me find a patch09:47
hrw6ff3a55aedf69173d6a1ef5008072273fb698edf fixed 10.0/10.1 then a729d132d6a8a891dc0ab92c9bda094a59cea231 moved centos to 10.1 from RDO09:49
hrwthen 502196d8fa9fae468089724937f66059e57ce3ac fixed ubuntu with their 10.009:49
hrwthen bionic arrived and we forgot that 10.0 existed09:50
yoctozeptohrw: I see, thanks for the commits09:51
hrwyoctozepto: we gave it to our Chinese developers as 'please do it while we celebrate xmas break' and some time later they celebrated Chinese New Year so we took care of other stuff etc09:52
openstackgerritCarlos Goncalves proposed openstack/kolla-ansible master: Remove Neutron LBaaS support  https://review.opendev.org/66387509:52
yoctozeptohrw: just how it's supposed to be done I think ;-)09:53
hrwyoctozepto: working with different cultures/timezones can be tricky09:53
hrwyoctozepto: but also allows to have 16h developer days09:54
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Supporting monitoring time synchronisation with Monasca  https://review.opendev.org/66372209:56
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Add load monitoring plugin config for Monasca  https://review.opendev.org/66372309:56
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Include cron_generator script in coverage report  https://review.opendev.org/66309709:56
openstackgerritCarlos Goncalves proposed openstack/kolla master: Remove Neutron LBaaS support  https://review.opendev.org/66302109:56
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add kolla-build-debian-source-arm64 experimental job  https://review.opendev.org/55765910:02
hrwcleaned up10:02
*** luksky has quit IRC10:11
yoctozeptomgoddard, hrw: while working on the percona patch - I have to decide on the path - is there a reason why GPG keys are added upfront instead of relying on repo definitions?10:13
mgoddardyoctozepto: I don't know10:13
hrwyoctozepto: for Debian/Ubuntu you need key before you run apt update. I do not remember do CentOS allows you to automatically accept new gpg keys10:13
mgoddardmaybe to prevent interactive prompt?10:14
yoctozeptoyum -y install accepts them10:14
mgoddardnot sure then10:16
mgoddardwe do always pull the gpgkey from the upstream source though rather than a mirror10:16
mgoddardso we might need to keep a copy of the percona key in the repo to avoid hitting percona.com10:17
mgoddard(we do already include docker/base/gpg-keys/PERCONA-PACKAGING-KEY, I'm still not sure why we have two keys)10:17
hrwyoctozepto: you may prepare patch which will do it your way. CI will tell does it work10:17
yoctozeptomgoddard, hrw: ok, thanks; I just wanted to handle Zuul/non-Zuul cleanly without rewriting the whole pipeline :-)10:20
yoctozeptomgoddard, hrw, mnasiadka: was Ubuntu ever affected? or only rhel?10:21
hrwyoctozepto: do not remember10:21
hrwyoctozepto: can you tell me which image uses something from percona?10:22
yoctozeptoxtrabackup10:22
yoctozepto(and it uses xtrabackup :-) )10:22
yoctozepto(finally a good naming scheme ;-) )10:22
*** tolisbar has quit IRC10:24
*** shyamb has quit IRC10:25
*** shyam89 has joined #openstack-kolla10:25
hrwyoctozepto: can we migrate to mariadb-backup instead?10:27
hrw" This backup tool is guaranteed to be compatible with MariaDB. Based on Xtrabackup, but improved to work with MariaDB. "10:27
*** Wasaac has quit IRC10:28
yoctozeptohrw: I know, ideally yes, I am setting it as my goal but it will take more work10:28
hrwheh. none in ubuntu ;(10:28
*** tolisbar has joined #openstack-kolla10:28
yoctozeptoexactly why I asked about upstream10:28
openstackgerritMerged openstack/kolla stable/rocky: Bump Monasca Notification version in Rocky  https://review.opendev.org/66304410:29
hrwDebian has it, RDO has it. Ubuntu as usual is behind10:29
yoctozeptohrw: more behind than CentOS Storage SIG or comparably? :D10:30
*** Wasaac has joined #openstack-kolla10:30
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Remove Neutron LBaaS support  https://review.opendev.org/66387510:30
*** Wasaac has quit IRC10:30
*** Wasaac has joined #openstack-kolla10:30
hrw;d10:31
hrwyoctozepto: so if we go mariadb-backup for centos/debian and xtrabackup for ubuntu then we can get rid of percona?10:35
yoctozeptohrw: of percona for centos/debian10:35
*** Wasaac has quit IRC10:36
yoctozeptothe repo of percona is used with ubuntu as well if I am not mistaken10:36
hrwyoctozepto: ubuntu takes xtrabackup from ubuntu not percona10:36
yoctozeptoconfirmed?10:36
hrwroot@a72820f30d51:/# apt-cache policy xtrabackup10:36
hrwxtrabackup:10:36
hrw  Installed: (none)10:36
hrw  Candidate: 2.4.9-0ubuntu210:36
hrw  Version table:10:36
hrw     2.4.9-0ubuntu2 50010:36
hrw        500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages10:36
yoctozeptook, then much cleaner this way10:37
yoctozeptoanyway, the patch for percona will land soon10:37
hrwyoctozepto: percona-xtrabackup is package name to use for ubuntu as xtrabackup is transitional one depending on former (both in ubuntu repo)10:37
yoctozeptohrw: ok, I just was not looking thoroughly, hence even asked about other OS failures10:38
hrwsure10:39
yoctozeptoyet see10:39
openstackgerritStephen Finucane proposed openstack/kolla master: Remove nova-consoleauth  https://review.opendev.org/66125110:39
yoctozeptobase Dockerfile10:39
yoctozeptoit installs10:39
yoctozeptoapt key for ubuntu10:40
yoctozeptofor percona10:40
yoctozepto(debian too)10:40
hrwyoctozepto: feel free to remove10:40
hrwI think that those keys would require review and dropping not needed ones10:40
yoctozeptoand installs the repo10:41
yoctozepto# Percona XtraBackup repo10:41
yoctozeptodeb http://repo.percona.com/apt bionic main10:41
yoctozeptonot touching it for now :-)10:41
hrwok10:43
*** Wasaac has joined #openstack-kolla10:44
*** zbr is now known as zbr|rover10:44
openstackgerritMerged openstack/kolla-ansible master: Use become for all docker tasks  https://review.opendev.org/65121010:47
openstackgerritMerged openstack/kolla-ansible master: Check for 'become' in tasks that use Docker in pep8  https://review.opendev.org/65122610:47
*** luksky has joined #openstack-kolla10:49
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Debian/Ubuntu: allow to use unsigned repositories  https://review.opendev.org/66389210:55
openstackgerritMerged openstack/kolla stable/stein: Switch to RDO Stein release on CentOS & Oracle  https://review.opendev.org/65160110:55
hrw663892 is interesting one. should just work and ubuntu jobs will use mirror so will be faster a bit10:56
openstackgerritRadosław Piliszek proposed openstack/kolla master: Fix issues with RHEL-family-based Percona in Zuul and clean up a little bit  https://review.opendev.org/66389310:56
*** janki has joined #openstack-kolla10:57
yoctozeptolet's see how I mastered the dark art of blind-programming10:57
hrwlooks like Kolla is slowly overtaken by Polish developers10:58
yoctozeptohrw: sssh!10:58
hrwyoctozepto: +1 for now. +2 once it pass zuul10:59
mgoddardyoctozepto: lets use xtrabackup or mariabackup everywhere - not different between OSs please11:00
hrwyoctozepto, kklimonda: will you attend OpenInfra Days next week?11:00
yoctozeptohrw: unfortunately not, have other duties to attend to :-(11:02
yoctozeptomgoddard: that was the original plan, but then - back to upstream11:03
yoctozeptohttps://bugs.launchpad.net/kolla/+bug/181390611:06
openstackLaunchpad bug 1813906 in kolla "xtrabackup image fails to build and blocks the gate" [Critical,Fix released]11:06
yoctozeptoguess this could be closed even before this patch11:07
yoctozeptosorry, it was11:07
yoctozeptothe comment suggested like it was not11:07
yoctozeptoall clear11:07
yoctozeptomgoddard, hrw: mariabackup is the way to go: Percona XtraBackup does not work with MariaDB 10.1 or greater if encryption or compression is used, or when innodb_page_size is set to some value other than 16K. It also does not work with MariaDB 10.2 or greater if innodb_safe_truncate=ON is set. It also does not work with MariaDB 10.3 or greater.11:09
yoctozeptoahm, I found an interesting release note11:11
yoctozeptoUpgrades MariaDB to version 10.3.10 in the ``mariadb`` image for11:11
yoctozepto    RHEL/CentOS/Oracle Linux.11:11
yoctozeptodoes it?11:11
yoctozepto(kolla notes)11:11
hrwyoctozepto: ubuntu has 10.1.40, centos has 10.3.10, debian has 10.3.1511:12
yoctozeptothen ahm11:13
yoctozeptoIt also does not work with MariaDB 10.3 or greater.11:13
yoctozeptohow does it sound to you11:13
yoctozepto:D11:13
hrwyoctozepto: 'gdzie jest krzyż^Wpatch' ;D11:13
yoctozeptohrw: haha11:14
yoctozeptofor now making Zuul happy, then making real progress11:14
hrwye11:14
yoctozeptoI need to flex my programming muscles11:14
kklimonda@hrw no, I had other plans already11:15
hrwok11:15
*** radeks has joined #openstack-kolla11:18
*** radeks__ has quit IRC11:19
hrwyoctozepto: your patch fails all over the place ;(11:19
hrwyoctozepto: jinja2.exceptions.TemplateSyntaxError: unexpected char u'#' at 325111:19
hrwhttp://logs.openstack.org/93/663893/1/check/kolla-build-centos-source/832db37/job-output.txt.gz#_2019-06-07_11_03_44_84444511:19
yoctozeptodamn, I amended to add this comment11:20
yoctozepto:D11:20
yoctozeptopunished for being pedant11:20
yoctozeptowonder why vscode does not see a problem with this...11:21
hrw;d11:21
yoctozeptoguess they made only a close-to-jinja parser :D11:23
openstackgerritRadosław Piliszek proposed openstack/kolla master: Fix issues with RHEL-family-based Percona in Zuul and clean up a little bit  https://review.opendev.org/66389311:27
yoctozeptowell, now verified with real jinja...11:27
hrwyoctozepto: doing local builds helps, you know?11:28
yoctozeptohrw: thought I would save some time ;-)11:28
hrwmgoddard: https://review.opendev.org/#/c/661324/ - may you write why -1 next time?11:30
*** Wasaac has quit IRC11:30
mgoddardhrw: I did in a previous comment11:30
yoctozeptohrw: what about two comments above?11:30
mgoddardhrw: all the jobs are failing, should be obvious :)11:30
hrw;d)11:30
hrwI should stop waking up at 5:3011:31
yoctozeptohrw: local build? no?11:31
yoctozeptotoday I woke up at 5:4511:31
yoctozeptonot much later :P11:31
hrw~curse pypi for being x86 centric11:34
yoctozeptohrw: that is my fear about other architectures, x86 is taken for granted :-(11:36
hrwyoctozepto: which ones you care about?11:37
yoctozeptohrw: where did you see those Zuul fails? I cannot see them via Zuul status page nor gerrit11:37
yoctozeptohrw: x86_64 but I am interested in the architecture you are currently pushing forward11:38
yoctozeptoand also did MIPS with my old router11:38
yoctozeptooh boy, then you can see how x86-centric the world is :-)11:38
hrwyoctozepto: I have zuul.openstack.org opened 24/711:38
yoctozeptohttp://zuul.openstack.org/builds?project=openstack%2Fkolla11:38
yoctozeptome blind, ask for help11:39
hrwyoctozepto: http://logs.openstack.org/93/663893/ and then below11:39
yoctozeptohrw: thanks... and the magic numbers 93/663893 are..?11:40
yoctozepto(1/2 are patchsets I presume)11:40
hrw663893 is your patch number. 93 are last two digits11:40
yoctozeptook, tried that, looked at the wrong tab11:41
yoctozeptoyou are right about early mornings in fact11:41
*** EmilienM is now known as EvilienM11:45
*** shyam89 has quit IRC11:47
*** shyamb has joined #openstack-kolla11:48
*** hrw has quit IRC11:50
*** hrw has joined #openstack-kolla11:51
*** Wasaac has joined #openstack-kolla12:04
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix quickstart for virtual environments  https://review.opendev.org/66273212:08
hrwmgoddard: in queue we have two CI jobs: debian/aarch64 #557659 and centos/aarch64 #662456. What do you think about adding debian/x86-64 one?12:12
*** Wasaac has quit IRC12:12
*** happyhemant has joined #openstack-kolla12:20
*** chrizl has joined #openstack-kolla12:24
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Debian/Ubuntu: allow to use unsigned repositories  https://review.opendev.org/66389212:25
hrwyoctozepto: thx for whitespace suggestion12:27
*** Wasaac has joined #openstack-kolla12:27
hrwmnasiadka: could you take a look at https://review.opendev.org/#/c/662429 https://review.opendev.org/#/c/663300/ https://review.opendev.org/663892 patches?12:29
*** Wasaac has quit IRC12:29
*** Wasaac_ has joined #openstack-kolla12:29
mnasiadkahrw: will do in a minute12:29
hrwmnasiadka: thanks12:29
openstackgerritMerged openstack/kolla-ansible stable/stein: Set kolla_source_version to stein  https://review.opendev.org/66355112:36
*** shyamb has quit IRC12:48
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Remove Neutron LBaaS support  https://review.opendev.org/66387512:52
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Add support for idempotent container stop and removal  https://review.opendev.org/66392812:52
kklimondais there a minimal ansible version that we can depend on?13:00
kklimondahmm, latest docs seem to indicate 2.4+13:01
kklimondaany chance of bumping it up to 2.7? ;)13:02
*** janki has quit IRC13:04
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Fix issue finding custom, host specific plugins  https://review.opendev.org/66393513:04
*** janki has joined #openstack-kolla13:04
hrwkklimonda: plan is to 2.713:04
hrwkklimonda: please synchronize with mnasiadka13:04
kklimondathanks, will do13:05
hrwkklimonda: we need 2.5+ due to python 3.7 in Debian so 2.7 was chosen as minimum. kolla-toolbox gets 2.8 in https://review.opendev.org/64241613:05
hrwkklimonda: https://etherpad.openstack.org/p/KollaWhiteBoard is worth reading13:06
*** sm806 has quit IRC13:12
*** sm806 has joined #openstack-kolla13:13
openstackgerritMerged openstack/kolla-ansible master: Fix the deploy guide build failed  https://review.opendev.org/66187913:16
*** priteau has quit IRC13:23
hrwkklimonda, mnasiadka: does it have sense? https://paste.fedoraproject.org/paste/UmD6H5~4ZCiKp2cygl2KIQ13:23
hrwwhen arch=aarch64 then I want other value for nodepool_mirror_host but rest has to be defined always13:23
*** dpawlik has quit IRC13:27
*** sm806 has quit IRC13:28
*** sm806 has joined #openstack-kolla13:29
mgoddardhrw: about debian/x86-64 - yes I guess it makes sense13:31
openstackgerritMark Goddard proposed openstack/kolla master: Remove crane image  https://review.opendev.org/66394213:32
*** priteau has joined #openstack-kolla13:37
kklimonda@hrw not really, I don't think you can use when for vars in the play - you could try moving that into tasks and using set_fact: https://paste.fedoraproject.org/paste/zP8MpE8F1thfE1OnicfpTg13:39
kklimonda@hrw or, if that's a zuul job perhaps you can create a variant based on the arch?13:40
*** Wasaac_ has quit IRC13:46
*** henriqueof has quit IRC13:46
*** dpawlik has joined #openstack-kolla13:46
*** Wasaac has joined #openstack-kolla13:47
*** cah_link has quit IRC13:50
mnasiadkahrw: yeah, try using job vars in zuul job definition13:51
*** JamesBenson has joined #openstack-kolla13:52
*** JamesBenson has quit IRC13:53
*** JamesBenson has joined #openstack-kolla13:53
mgoddardansible facts should be available from zuul and should have info on the arch13:54
hrwthanks13:57
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add CentOS/source aarch64 experimental job  https://review.opendev.org/66245614:07
hrwsynchronized code with Debian CI job14:08
*** janki has quit IRC14:08
hrwshit. Debian CI job timed out14:08
mnasiadkalol, can't be longer than OracleLinux :D14:15
hrwmnasiadka: aarch64 is slower14:16
hrwinfra has 8vcpu instances iirc14:16
*** igordc has joined #openstack-kolla14:23
*** dpawlik has quit IRC14:27
*** goldyfruit has joined #openstack-kolla14:28
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Use Ansible upstream modules  https://review.opendev.org/66396314:38
*** itlinux has quit IRC14:46
*** itlinux has joined #openstack-kolla14:47
*** schwicht has joined #openstack-kolla14:49
*** itlinux has quit IRC14:52
mgoddardmnasiadka: weird ceph upgrade failure: http://logs.openstack.org/75/663875/3/check/kolla-ansible-centos-source-upgrade-ceph/34224a4/primary/logs/ansible/upgrade14:56
mnasiadkamgoddard: weird indeed14:57
*** itlinux has joined #openstack-kolla14:59
*** radeks_ has joined #openstack-kolla15:00
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Include cron_generator script in coverage report  https://review.opendev.org/66309715:01
*** Wasaac has quit IRC15:02
*** Wasaac_ has joined #openstack-kolla15:02
*** radeks has quit IRC15:02
openstackgerritMerged openstack/kolla master: Use packagecloud/ubuntu rabbitmq repo for all archs  https://review.opendev.org/66330015:06
hrw3>15:07
*** Sravan has joined #openstack-kolla15:15
yoctozeptomgoddard: were monitors restarted?15:29
yoctozepto(ceph-mon monitors in case someone wonders)15:30
mgoddardyoctozepto: would have been restarted due to the upgrade, yes15:31
yoctozeptoit looks as if the quorum did not decide on newer caps15:32
yoctozeptocould be race condition?15:37
yoctozeptohrw> yoctozepto: thx for whitespace suggestion15:37
yoctozeptoyou are welcome, I added one more comment15:38
yoctozeptosorry for not noticing that earlier15:38
yoctozepto(let's blame the missing whitespace!)15:38
*** Wasaac_ is now known as Wasaac15:40
*** Sravan has quit IRC15:43
*** Sravan has joined #openstack-kolla15:44
*** goldyfruit has quit IRC15:45
hrw:D15:46
*** Sravan has quit IRC15:48
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: grafana/aarch64: use package from upstream repositories  https://review.opendev.org/66242915:54
*** Wasaac has quit IRC15:54
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Debian/Ubuntu: allow to use unsigned repositories  https://review.opendev.org/66389215:56
hrwyoctozepto: renamed to 99*15:56
yoctozeptohrw: thanks15:59
*** tolisbar has quit IRC16:00
hrwyoctozepto: and this was good comment. thanks16:00
yoctozeptohrw: working hard on making them all such16:02
hrwyoctozepto: practice makes perfect16:02
yoctozeptohrw: my thoughts exactly (or did I copy?)16:03
yoctozeptomgoddard: https://review.opendev.org/#/c/663893/ how many rechecks advised?16:03
yoctozeptoand a side-question: would it listen to my recheck comment?16:04
yoctozepto(not willing to test)16:04
hrwyoctozepto: anyone can do recheck16:08
*** luksky has quit IRC16:11
hrwmgoddard: I am off until next Friday. going to OpenInfra Days Poland16:11
hrwbeer, beer, some talks, beer ;D16:11
* hrw off16:12
*** priteau has quit IRC16:25
*** k_mouza_ has joined #openstack-kolla16:40
*** owalsh has quit IRC16:40
*** k_mouza has quit IRC16:42
*** k_mouza_ has quit IRC16:44
*** owalsh has joined #openstack-kolla16:52
*** dougsz has quit IRC16:54
*** Wasaac has joined #openstack-kolla17:08
*** KeithMnemonic has quit IRC17:09
*** factor has joined #openstack-kolla17:14
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add no_log to creating glance project  https://review.opendev.org/66400917:18
openstackgerritMichal Nasiadka proposed openstack/kolla master: Bump up pkg versions in kolla-toolbox  https://review.opendev.org/64241617:18
mnasiadkamgoddard: you can imagine I will be in the same place ;)17:21
mgoddardhave fun mnasiadka & hrw :)17:22
*** k_mouza has joined #openstack-kolla17:22
mgoddardyoctozepto: it's hard to say. Given that we do manage to get patches merged it clearly doesn't happen every time17:23
mgoddardyoctozepto: 4-5 passes in a row would be a good sign17:23
yoctozeptook, then I will try a recheck17:24
yoctozepto:-)17:24
blineis multi-domain setup supported?17:25
mnasiadkamgoddard: the ,,try without kolla_toolbox'' idea is rather my thinking how to get rid of some weird stuff we are doing, like shell: docker exec, become:True in those cases and so on - but true - there's a lot of libraries that need to be installed in order to install python libraries dependencies - mea culpa ;)17:25
mnasiadkabline: keystone multi domain? sure.17:25
blinethanks17:26
mnasiadkabline: one option is to use per-domain configs - it's supported, just check the config.yml in the keystone role - or you can define domain auth settings using API17:26
*** k_mouza has quit IRC17:27
blinehaving issues with magnum and magnum_trustee_domain_admin after adding in multidomain config17:27
blineFAILED! => {"changed": false, "extra_data": null, "msg": "Error in creating user magnum_trustee_domain_admin. (409) Client Error for url: http://10.1.0.100:35357/v3/users Conflict occurred attempting to store user - Duplicate entry found with name magnum_trustee_domain_admin at domain ID 9b43cfc70cea495eb84ce4593958afce."}17:27
mnasiadkabline: yeah, had that like 2-3 years ago, but that's something wrong with kolla_keystone_* modules and multidomain17:27
blinemnasiadka, is there a fix?17:28
mnasiadkabline: there is a workaround, deploy keystone single domain and then magnum, and then add additional domains ;)17:28
blineok, will give that a go; thanks mnasiadka17:29
mnasiadkabline: other option is to fix that, but I didn't have time for that those 2-3 years ago - there should be bug filed by me or someone from my former team ;)17:29
blinei'm not in production yet, so zapping and reinstalling is totally an option17:32
*** goldyfruit has joined #openstack-kolla17:48
openstackgerritMark Goddard proposed openstack/kolla master: Fix ownership of ironic inspector DHCP hostsdir  https://review.opendev.org/66401117:52
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Add CI job for ironic  https://review.opendev.org/56882917:53
*** Wasaac has quit IRC17:55
*** luksky has joined #openstack-kolla17:56
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Add CI job for ironic  https://review.opendev.org/56882917:58
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Add CI job for ironic  https://review.opendev.org/56882917:58
openstackgerritMark Goddard proposed openstack/kolla stable/stein: Debian/Ubuntu: more python3 packages for binary  https://review.opendev.org/66401217:59
openstackgerritMark Goddard proposed openstack/kolla stable/stein: Bump service versions for Stein  https://review.opendev.org/66016518:02
*** gfidente has quit IRC18:02
openstackgerritRadosław Piliszek proposed openstack/kolla master: Clean Zuul repo mirrors for RHEL family  https://review.opendev.org/66402618:34
yoctozeptomgoddard: https://review.opendev.org/#/c/664026/ success can be counted towards the percona change too18:34
yoctozepto;-)18:34
yoctozeptoalong with hrw's ubuntu patch it makes the Zuul build process cleaner18:35
*** Wasaac has joined #openstack-kolla18:46
yoctozeptohttps://blueprints.launchpad.net/kolla/+spec/percona-xtrabackup-to-mariabackup18:54
*** Sravan has joined #openstack-kolla18:54
yoctozeptoshare your thoughts with me18:54
*** Sravan has quit IRC19:00
*** JamesBenson has quit IRC19:02
*** Sravan has joined #openstack-kolla19:04
*** Sravan has quit IRC19:07
*** Sravan has joined #openstack-kolla19:09
*** Sravan has quit IRC19:16
*** Sravan has joined #openstack-kolla19:19
yoctozeptohrw: digging into xtrabackup, it actually fails miserably for debian (+1 for testing it) and ubuntu uses upstream repos but via http (not https) - hence probably why it failed less often or never19:23
blinemnasiadka, You deploy with non-multidomain and then reconfigure with the multidomain config?19:35
*** Sravan has quit IRC19:35
blinejust tried that; deploy went great but the reconfigure gave the same error with magnum19:35
mnasiadkabline: I remember it worked in the past, so maybe now it doesn’t :(19:38
bline:(19:38
blinethink i found the bug report: https://bugs.launchpad.net/kolla-ansible/+bug/171401119:38
openstackLaunchpad bug 1714011 in kolla-ansible "Reconfigure failing on creating magnum_trustee_domain_admin when domain_specific_drivers_enabled = True is set in keystone.conf" [Undecided,Incomplete] - Assigned to yuqian (roger-yu)19:38
*** Wasaac has quit IRC19:54
*** goldyfruit has quit IRC20:02
*** jonaspaulo has joined #openstack-kolla20:05
*** jonaspaulo has quit IRC20:09
*** igordc has quit IRC20:28
blineguess i could try to rebuild kolla-toolbox with the suggestions in that ticket20:35
*** radeks_ has quit IRC21:00
*** Talion has quit IRC21:03
hrwyoctozepto: https://review.opendev.org/#/c/631723/ - can you look and comment? we may abandon it if it is no longer needed due to your changes21:13
*** Sravan has joined #openstack-kolla21:13
hrwyoctozepto: https://review.opendev.org/#/c/638410/ as well21:14
*** Sravan has quit IRC21:17
*** Sravan has joined #openstack-kolla21:19
*** whoami-rajat has quit IRC21:34
openstackgerritAlex Schultz proposed openstack/kolla master: Do not gem install for binary type  https://review.opendev.org/63401921:35
*** Sravan has quit IRC21:50
*** pcaruana has quit IRC21:51
*** luksky has quit IRC21:55
mlozahello, I lost external vip address in controller nodes because I restarted networking, how can I regain it? Doing reconfigure didn't help22:06
*** Sravan has joined #openstack-kolla22:07
*** Sravan has quit IRC22:24
*** goldyfruit has joined #openstack-kolla22:28
*** schwicht has quit IRC22:28
*** Sravan has joined #openstack-kolla22:29
*** hamzy_ has joined #openstack-kolla22:30
openstackgerritMerged openstack/kolla-ansible master: fluentd: Configure td-agent only for x86_64  https://review.opendev.org/63872222:30
openstackgerritAlex Schultz proposed openstack/kolla master: Add script to check for forbidden instructions  https://review.opendev.org/63397122:32
*** hamzy has quit IRC22:32
*** Sravan has quit IRC22:36
openstackgerritAlex Schultz proposed openstack/kolla master: Do not gem install for binary type  https://review.opendev.org/63401922:36
openstackgerritAlex Schultz proposed openstack/kolla master: Do not pip install for binary type  https://review.opendev.org/63470622:36
*** EvilienM is now known as EmilienM22:42
*** Sravan has joined #openstack-kolla22:56
*** Sravan has quit IRC22:59
*** schwicht has joined #openstack-kolla23:06
*** Sravan has joined #openstack-kolla23:12
*** Sravan has quit IRC23:16
*** Sravan has joined #openstack-kolla23:25
*** schwicht has quit IRC23:25
*** goldyfruit has quit IRC23:29
*** Sravan has joined #openstack-kolla23:31
*** Sravan has quit IRC23:32
*** q[bline] has joined #openstack-kolla23:39
q[bline]anyone recommend which vnc console to use? tried novnc and spice (spice was unusable)23:40
*** bline has quit IRC23:51
*** jonaspaulo has joined #openstack-kolla23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!