Wednesday, 2019-06-26

« Tuesday, 2019-06-25 Index Thursday, 2019-06-27 »
*** hamzy has joined #openstack-kolla00:27
*** gregoryo has joined #openstack-kolla00:38
*** k_mouza has joined #openstack-kolla00:52
*** k_mouza has quit IRC00:57
*** mgoddard has quit IRC01:40
*** mgoddard has joined #openstack-kolla01:48
*** kplant has quit IRC01:52
*** igordc has quit IRC01:58
*** zhanglong has joined #openstack-kolla02:26
*** hongbin has joined #openstack-kolla02:44
*** goldyfruit has quit IRC02:52
*** whoami-rajat has joined #openstack-kolla03:04
*** hongbin has quit IRC03:18
*** Wasaac has joined #openstack-kolla03:31
*** Wasaac has quit IRC03:35
*** rgogunskiy has joined #openstack-kolla03:52
*** zhanglong has quit IRC03:52
*** hongbin has joined #openstack-kolla03:54
*** rgogunskiy has quit IRC03:56
*** skramaja has joined #openstack-kolla04:00
*** skramaja_ has joined #openstack-kolla04:03
*** skramaja has quit IRC04:04
*** hongbin has quit IRC04:12
*** Sravan has joined #openstack-kolla04:18
*** Sravan has quit IRC04:21
*** dave-mccowan has quit IRC04:23
*** Sravan has joined #openstack-kolla04:25
*** Sravan has quit IRC05:18
*** Sravan has joined #openstack-kolla05:19
*** ivve has joined #openstack-kolla05:36
*** mgoddard has quit IRC05:54
*** cah_link has joined #openstack-kolla06:00
*** mgoddard has joined #openstack-kolla06:00
*** dpawlik has joined #openstack-kolla06:05
*** Sravan has quit IRC06:07
yoctozeptomorning06:14
yoctozeptohrw: well, it did not06:14
*** openstackgerrit has joined #openstack-kolla06:20
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: kolla-ansible CI test (host: CentOS 7 -> Ubuntu Bionic)  https://review.opendev.org/66751806:20
openstackgerritRadosław Piliszek proposed openstack/kolla master: DNM: kolla CI test (host: CentOS 7 -> Ubuntu Bionic)  https://review.opendev.org/66751906:22
*** pcaruana has joined #openstack-kolla06:26
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: kolla-ansible CI test (host: CentOS 7 -> Ubuntu Bionic)  https://review.opendev.org/66751806:27
openstackgerritRadosław Piliszek proposed openstack/kolla master: DNM: kolla CI test (host: CentOS 7 -> Ubuntu Bionic)  https://review.opendev.org/66751906:30
*** luksky has joined #openstack-kolla06:45
*** cgrosjean has joined #openstack-kolla06:49
*** cgrosjean has quit IRC07:07
*** rgogunskiy has joined #openstack-kolla07:12
*** rgogunskiy has quit IRC07:12
*** rgogunskiy has joined #openstack-kolla07:12
*** rgogunskiy has quit IRC07:12
*** rgogunskiy has joined #openstack-kolla07:13
gregoryoStarting with configuring my Kolla OpenStack to look more like our prod infra, and I found this in glance-api.conf: use_forwarded_for07:24
gregoryoI can't find what it's for, especially not at https://docs.openstack.org/glance/rocky/configuration/glance_api.html - any ideas?07:24
yoctozeptogregoryo: http://codesearch.openstack.org/?q=use_forwarded_for seems irrelevant to glance07:29
yoctozeptorather nova and manila07:29
yoctozeptobut we have it for cinder and glance as well07:29
yoctozeptomight have been relevant at some point07:30
gregoryoOkay, I'll just ignore it.07:30
yoctozeptoit just tells the api to trust the proxy07:31
yoctozeptoas in using its reported address instead of real address07:31
yoctozeptoprobably relevant when you have api behind a load balancer07:32
*** cgrosjean has joined #openstack-kolla07:34
*** cgrosjean has quit IRC07:34
mnasiadkamorning07:40
hrwmorning07:40
hrw~curse zuul or whatever making my ci failing07:40
mnasiadkawell, I have a lot of fun with Nokia Jenkins today\07:41
yoctozeptohrw: we need to get this slowness out because it causes our current troubles07:42
yoctozeptoI am running tests to see if it is not the issue with the host machines we take for centos tests07:42
hrwmnasiadka: jenkins is fine as long as you keep all job definitions outside of it07:42
yoctozeptoas there should be no problem running centos images on ubuntu hosts :P07:42
mnasiadkahrw: Jenkins is fine, as long as you manage it, not trust some service team to deliver it, and they move it from your local DC to AWS, while your test env remains in your DC :D07:43
hrwyoctozepto: or ask infra for some hw/vm details of those runners07:43
hrwmnasiadka: ouch07:43
yoctozeptomnasiadka: fcuk07:44
hrwmnasiadka: at linaro we use some plugin so all job definitions are kept in git as yaml files. easy way to do reviews and no one can fsck jobs by playing with jankins ui07:44
yoctozeptohrw: yeah, you have my blessing07:44
mnasiadkahrw: yeah, here we keep pipeline definitions in repo, but nobody thoght about keeping job definitions in a repo, don't even ask :D07:45
yoctozeptomnasiadka: must be the reason why I am not seeing June security updates on my Nokia 5 :P07:46
mnasiadkayoctozepto: you use a Nokia phone? poor guy :)07:47
yoctozeptomnasiadka: lolz07:47
yoctozeptomnasiadka: actually it works better than all I had before07:47
mnasiadkayoctozepto: I have one as company cell, my kid is playing with that, had too many problems with it - and I don't like carrying two phones :)07:48
yoctozeptomnasiadka: what are you using then?07:49
mnasiadkayoctozepto: apple ;)07:49
yoctozeptomnasiadka: an apple a day keeps the doctor away07:51
mnasiadkasomething like that07:51
yoctozeptoI have one but somehow did not come to appreciate it, might be too used to android07:51
yoctozeptoI have this small form factor one07:52
yoctozeptodon't rememember the name07:52
hrwyoctozepto: SE?07:55
gregoryoI've just found https://github.com/openstack/kolla-ansible/blob/stable/rocky/doc/source/reference/external-ceph-guide.rst and very happy to do so. Great to think "hey how do I... oh there's the doco for *exactly* what I want to do"07:59
hrwmnasiadka: my official phone for work stuff is Nokia 101. because it can not install any apps ;D08:08
mnasiadkahrw: well, it has nice standby time if I remember correctly :)08:09
*** dougsz has joined #openstack-kolla08:10
hrwmnasiadka: not the same as 3310 but probably close08:10
hrwfor horizon we process 1600 locale files. 1100 of them are django ones08:12
hrwso let's remove django ones...08:15
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: horizon: process i18n files just once  https://review.opendev.org/66738608:15
* hrw out for some08:16
hrwhave to take family to the beach ;D08:16
mgoddardmorning all08:19
mgoddardsounds like CI wasn't magically fixed overnight?08:19
*** gregoryo has quit IRC08:21
mnasiadkaI think IT is not the place for magic overnight fixing...08:24
*** Luzi has joined #openstack-kolla08:31
*** rpittau|afk is now known as rpittau|mtg08:41
*** cgrosjean has joined #openstack-kolla08:46
*** Dubla has joined #openstack-kolla08:47
hrwmnasiadka: unless you are not in admin team08:48
mnasiadkahrw: I meant the magic is usually in somebody's hands, not in the air ;)08:49
hrwlove is in the air, not magic08:52
* hrw -> updating laptop system. tricky when python2 is on a list of 'do not even try to install' ;D08:53
openstackgerritZijian Guo proposed openstack/kolla-ansible master: Add extra volumes support for services that were not previously supported  https://review.opendev.org/66725609:03
hrw126GB used on / and 113GB of it is /var/lib/docker ;d09:04
*** k_mouza has joined #openstack-kolla09:09
openstackgerritMerged openstack/kolla-ansible master: Do not run Zuul kolla-ansible-base jobs when modifying globals.yml  https://review.opendev.org/66553609:27
*** gfidente has joined #openstack-kolla09:30
*** hamzaachi has joined #openstack-kolla09:35
yoctozepto<hrw> yoctozepto: SE?09:36
yoctozeptoyup09:36
yoctozepto* hrw -> updating laptop system. tricky when python2 is on a list of 'do not even try to install' ;D09:37
yoctozeptothat's how you take family to the beach?09:37
hrwyoctozepto: out of home, to the car, to the beach, they leave, I go back to work09:37
yoctozeptomgoddard: they fixed CI in terms of RETRY_LIMIT09:38
yoctozeptoand probably duplicate runs09:39
mgoddardyoctozepto: yeah, seems better now09:39
yoctozeptobut timeouts are due to us09:39
hrwyoctozepto: thanks for handling09:39
yoctozeptomgoddard, hrw: yeah, I am handling it still09:40
yoctozeptofrom infra:09:40
yoctozepto<ianw> cgoncalves has identified that centos-7 is a lot slower at times, we haven't figured out why though09:40
hrwIan ;D09:41
yoctozeptohrw: what about him?09:42
hrwyoctozepto: nothing. hard to catch sometimes due to Australia timezone09:42
yoctozeptohrw: see, I can catch anybody and anything09:44
yoctozeptotake flu for example09:44
hrw2019-06-26 09:46:03.784177 | primary | INFO:kolla.common.utils.designate-worker:http://mirror.iad.rax.opendev.org/centos/7/os/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to mirror.iad.rax.opendev.org:80; No route to host"09:46
hrwfsck.09:46
mnasiadkagood stuff09:47
yoctozeptonot again!09:47
yoctozepto<yoctozepto> https://review.opendev.org/66751809:49
yoctozepto<yoctozepto> https://review.opendev.org/66751909:49
yoctozeptoby the rule of single omnipotent observation, I declare using ubuntu as host did not help much09:49
yoctozepto:D09:49
hrwyoctozepto: was worth checking09:50
hrwlooks like horizon change is getting close to +109:54
hrwjust kolla-build-centos-source left...09:54
yoctozepto<hrw> just kolla-build-centos-source left...09:54
yoctozeptoyeah, that's the most likely to fail09:55
*** k_mouza has quit IRC09:56
hrwand will. looks like lost connection to infra mirror ;(09:58
yoctozeptodamn it09:59
yoctozeptohrw: seriously, now after you butcher django files I seriously consider my approach better :P10:02
hrwyoctozepto: it was a test how it will improve situation10:02
yoctozeptohrw: aaaah ;D10:03
hrwpatch needs recheck ;(10:04
yoctozeptohrw: the whole CI needs a big hug10:07
hrwyoctozepto: huge hug10:07
yoctozeptomonstrous hug10:07
hrwand I need to take care of aarch64 CI too ;(10:07
yoctozeptothough that does not seem nice anymore10:07
yoctozeptohrw: wonder if centos is slower there too ;D10:08
hrwyoctozepto: there I run all on debian-stretch VM10:08
yoctozeptothe good thing is we know octavia and osa teams notice the same slowness10:08
openstackgerritMerged openstack/kolla master: Fix deployment of ceph-nfs container failing due to wrong USER  https://review.opendev.org/66623910:08
yoctozeptohrw: but you had some centos ci build in there10:08
yoctozepto<openstackgerrit> Merged openstack/kolla master: Fix deployment of ceph-nfs container failing due to wrong USER  https://review.opendev.org/66623910:08
yoctozeptoa miracle!10:08
*** k_mouza has joined #openstack-kolla10:09
hrwyoctozepto: I have debian/aarch64 and centos/aarch64 CI job patches. both run on debian-stretch ;D10:09
yoctozeptohrw: but they build in different images10:09
yoctozeptoit affects x86_64 at least10:09
yoctozeptoas switching to ubuntu host did not help10:09
*** spiette has quit IRC10:11
hrwlet me restart my aarch64 rust10:12
hrwyoctozepto: SKIPPED_IMAGES == list of images which are not buildable so kolla will not even try to build them. right?10:14
yoctozeptohrw: that's how it was used according to git log10:15
yoctozeptoI slapped the arch support there as it seemed logical ;D10:16
hrwwill try to populate it with aarch64 stuff10:16
hrwas 'let me do something not involving ci' ;D10:16
yoctozeptohrw: you are welcome10:16
yoctozeptohttp://logs.openstack.org/19/667519/2/check/kolla-build-centos-source/dd7b3e9/logs/build/?C=M;O=D10:17
hrw(venv3) 10:17 (0s) hrw@j13-qrep-04:kolla$ ~/bin/docker-utils/kolla-build-images.sh 'centos debian ubuntu' 'source binary' ' ' skipped-images-test10:17
yoctozeptothis ubuntu+centos10:17
yoctozeptoalways horizon lame10:18
yoctozeptothe hit is so huge compared to ubuntu10:18
hrwyoctozepto: https://github.com/hrw/docker-utils/blob/master/kolla-build-images.sh - you may find it useful10:18
yoctozeptohrw: thanks10:19
hrweasy script to mass build for any distro/type/tag10:19
yoctozeptonice little scriptlet in there10:19
hrwyoctozepto: have fun. patches/issues welcome10:20
yoctozeptohrw: I will remind you you asked when I spam you with the issues I have :D10:21
hrwyoctozepto: I reserve myself an option to ignore them ;d10:22
yoctozeptoguys, following the logic it is either python (and its deps) or gettext (and its deps) that is so hellish slow in centos10:22
yoctozeptohrw: and so the dream developer mask has fallen ;D10:22
yoctozeptoyet osa and octavia do not depend that much on gettext10:23
yoctozeptoso it leaves python and the way it is used10:23
hrwyoctozepto: all hope in centos8 then10:24
* hrw -> beach to grab family back home10:24
yoctozeptohrw: that or we can actually try switching python version in 710:26
yoctozeptoit's a source build anyway10:26
*** luksky has quit IRC10:28
*** skramaja_ has quit IRC10:29
mnasiadkaso, what's the problem with horizon, but in short words? :D10:32
*** heikkine has quit IRC10:33
yoctozeptomnasiadka: builds the slowest on centos, comparing with ubuntu it's absurd10:34
yoctozeptoand times out our jobs ;_;10:35
mnasiadkayeah well, I noticed CentOS builds are slightly slower than Ubuntu, but I guess now it's dramatic :D10:35
mnasiadkalet's drop CentOS then :D10:36
yoctozeptomnasiadka: I would but my deployment is centos-centric :P10:36
mnasiadkayoctozepto: mine too, I'm only joking10:36
mnasiadkayoctozepto: question is what is wrong with CentOS...10:36
yoctozeptomnasiadka: good one10:36
yoctozeptomnasiadka: osa and octavia both noticed it10:37
yoctozeptowe have absurd slowness amplification with horizon10:37
yoctozeptodue to its compilemessages10:37
*** dpawlik has quit IRC10:37
yoctozepto(django localization)10:37
*** dpawlik has joined #openstack-kolla10:38
mgoddardany mariadb/galera gurus out there?10:38
mnasiadkayoctozepto: let me guess, horizon people don't see a problem with it? :)10:38
yoctozeptomgoddard: nope, only padawans :(10:38
yoctozeptomnasiadka: and you win the main prize10:39
mnasiadkamgoddard: you expect reading bits in wsrep and mangling packets on the fly, or some normal people? :)10:39
mgoddardI'm looking at https://launchpad.net/bugs/1834191, rocky to stein upgrade bug10:39
openstackLaunchpad bug 1834191 in kolla-ansible stein "Multinode galera upgrade from Rocky to Stein fails on CentOS" [Critical,In progress] - Assigned to Mark Goddard (mgoddard)10:39
yoctozeptomnasiadka: we fixed the extra work case10:39
yoctozeptobut it does not make centos faster10:39
*** SaiHarshaK has joined #openstack-kolla10:39
yoctozeptoand it's not horizon's fault in this regard10:39
mgoddardso here's the problem. We switched from xtrabackup to mariabackup for state transfer in stein, so when we restart the new containers they can fail if talking to a rocky container10:40
mgoddardwhat should we do? shutdown all containers and recover?10:41
yoctozepto"mgoddard: We switched from xtrabackup to mariabackup for state transfer in stein"10:41
yoctozeptowe did?10:41
mgoddardyes10:41
mnasiadkamgoddard: so you want to handle the upgrade case?10:42
yoctozeptoso why do we keep percona around? :O10:42
mgoddardfor backups10:42
*** dpawlik has quit IRC10:42
mgoddardkolla-ansible mariadb_backup uses xtrabackup10:42
mgoddardgalera uses mariabackup for state syc10:42
yoctozeptoah, you mean this10:43
yoctozeptothat's clear now10:43
yoctozeptomgoddard: can't see the flaw in your logic so far10:44
mgoddardjust means we lose the DB for a while10:44
*** dpawlik has joined #openstack-kolla10:45
SaiHarshaKHello, i got error at "Waiting for nova-compute service up", as mgoddard suggested i checked nova_compute logs, its not able to get connected to rabbitmq i think and i could'nt login to rabbitmq container(stuck in restart loop) to check logs.10:45
SaiHarshaKthis time i tried a fresh deploy with centos, source. same issue, but "docker logs rabbitmq" gave me this output -> http://paste.openstack.org/show/753431/10:47
mnasiadkamgoddard: well, during an upgrade it's not weird that we lose the DB for a while?10:47
mnasiadkamgoddard: as long as the data is there :)10:48
yoctozeptomnasiadka: the while is more like a whiiiiile10:48
mgoddardI guess10:48
mnasiadkayoctozepto: it still not 5 hours, right?10:48
yoctozeptomnasiadka: hard to tell, we fail, but probably not as slow as building horizon on centos-7 ;D10:49
mnasiadkayoctozepto: I'll tell you what - try doing FFU upgrade on tripleo - then you might think that horizon build is fast.10:50
*** dpawlik has quit IRC10:50
mnasiadkamgoddard: wait, we're changing SST or IST?10:52
yoctozeptomnasiadka: what's SST/IST?10:52
hrwre10:52
yoctozeptoah, state tranfer types10:53
yoctozeptonever seen them abbr.10:53
*** dpawlik has joined #openstack-kolla10:53
mnasiadkayoctozepto: so I guess you never had a lot of fun with Galera :D10:53
yoctozeptomnasiadka: nope :-(10:53
hrwmgoddard: I only know that galera is in mariadb 10.1+ and that's all10:53
yoctozeptohrw: by the looks of it it makes us happier people ;p10:56
hrwyoctozepto: my lack of galera knowledge you mean? :D10:56
SaiHarshaKi have the erl_crash.dump, its 3.3Mb should i link it here?10:56
yoctozeptohrw: and mine too, I only know the principles and the fact that kolla-ansible deployed it fine for me ;p10:57
hrwhttp://logs.openstack.org/86/667386/3/check/kolla-build-centos-source/212acff/logs/build/ that's fail...10:57
hrwrecheck in progress10:58
yoctozeptohrw: next time think about consequences of your actions :D10:58
mnasiadkahrw: rax is getting better and better :)10:58
hrwyoctozepto: now I do not get what you mean10:58
*** luksky has joined #openstack-kolla10:58
yoctozeptohrw: it was django's revenge for the butchery10:59
yoctozeptomnasiadka, hrw: joking aside, I regularly let infra know what I feel about iad.rax11:00
*** dpawlik has quit IRC11:00
mnasiadkayoctozepto: I have similar feelings towards ovh11:00
yoctozeptomnasiadka: they did not fail me yet, waiting for the prime time ;D11:01
*** dpawlik has joined #openstack-kolla11:01
hrwheh. spinning rust and 96GB ram ;(11:04
hrwwith rust+256GB ram I could move /var/lib/docker to tmpfs...11:04
hrwbut 1200 images will not fit in 64GB11:05
*** k_mouza has quit IRC11:06
*** k_mouza has joined #openstack-kolla11:09
*** k_mouza has quit IRC11:09
yoctozeptohrw: wonder if the speed gain would be that drastic, with that amount of RAM and proper caching it should fly anyway11:09
yoctozeptodrop the barriers on fs, set to aggressive caching and it should leave the stuff in RAM for as long as it is still useful11:11
hrwyoctozepto: yep11:12
hrwanyway no rush. will build finally11:12
yoctozeptohrw: tell it to CI11:12
hrw"load average: 17.33, 16.56, 13.27" tells that there is still reserve11:12
*** kplant has joined #openstack-kolla11:16
yoctozeptohrw: hard to tell it from average load, you could get these numbers running off RAM anyway11:17
hrwyep11:20
hrwanyway it is caching nice in memory11:20
*** happyhemant has joined #openstack-kolla11:28
hrwwould be nice to have "INFO:kolla.common.utils.nova-compute: started build at year-month-day hour:minute:second" and similar for end of build11:36
hrwor even "INFO:kolla.common.utils.nova-compute: Built in X:Y:Z"11:37
yoctozeptohrw: logs last modified time does it now11:38
hrwyoctozepto: it only says when it ended11:38
hrwnot how long it took11:38
yoctozepto- creation time11:38
yoctozeptosimple math :D11:38
yoctozeptoor you mean in CI11:39
yoctozeptothere it is not available :-(11:39
yoctozepto+1 for the idea though11:39
hrwINFO:kolla.common.utils.base:Building started at 2019-06-26 13:41:55.41753911:42
yoctozeptoyou're halfway there ;-)11:43
*** sean-k-mooney has joined #openstack-kolla11:43
hrw1/3 I would say11:44
yoctozeptooh, right, the math is going to be 1/311:44
yoctozeptoand to display it11:44
yoctozeptoyou might even be in 1/4 and not realizing it11:45
yoctozepto;D11:45
hrw;D11:46
hrwINFO:kolla.common.utils.base:Built at 2019-06-26 13:46:45.69643011:46
hrwINFO:kolla.common.utils.base:Built at 2019-06-26 13:47:55.714413 (took 0:00:04.144356)11:48
hrwdatetime.datetime.now() roxx11:48
hrwbut will move it to separate line11:48
*** ivve has quit IRC11:51
hrwINFO:kolla.common.utils.kibana:Built11:52
hrwINFO:kolla.common.utils.kibana:Ended at 2019-06-26 13:52:10.774162 (took 0:02:03.803041)11:52
yoctozeptowhy separate?11:55
yoctozeptolooks odd this way11:56
*** goldyfruit has joined #openstack-kolla11:59
hrwwanted to have it for failed images but self.logger.exception() prints and raise so will move back12:00
yoctozeptoI see12:02
*** luksky has quit IRC12:02
*** luksky has joined #openstack-kolla12:03
*** hrww has joined #openstack-kolla12:03
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: base: do not put grafana repo twice  https://review.opendev.org/66758712:05
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: build: add timestamps to start/end of build  https://review.opendev.org/66758812:05
*** goldyfruit has quit IRC12:05
*** hrw has quit IRC12:06
*** hrww is now known as hrw12:06
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Make nova external ceph key extraction tasks non-changing  https://review.opendev.org/66759612:23
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Make cron jobs generator command task in keystone non-changing  https://review.opendev.org/66759912:27
*** Luzi has quit IRC12:35
*** dpawlik has quit IRC12:40
*** gkadam has joined #openstack-kolla12:43
*** dave-mccowan has joined #openstack-kolla12:52
*** dpawlik has joined #openstack-kolla12:55
*** k_mouza has joined #openstack-kolla13:10
*** livelace has joined #openstack-kolla13:24
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: build: add timestamps to start/end of build  https://review.opendev.org/66758813:32
hrwyoctozepto: you comment included13:32
openstackgerritGiulio Fidente proposed openstack/kolla stable/stein: Move back to CentOS StorageSIG repos for Ceph and Ganesha  https://review.opendev.org/66761613:36
*** dougsz has quit IRC13:40
*** dougsz has joined #openstack-kolla13:40
hrwyoctozepto: https://pastebin.com/HXvenmY2 is proper way?13:40
openstackgerritMerged openstack/kolla-ansible stable/stein: Replace merge_configs with merge_yaml for merge mongodb.conf  https://review.opendev.org/66709213:44
*** mloza has quit IRC13:47
*** dougsz has quit IRC13:48
*** mloza has joined #openstack-kolla13:56
*** k_mouza has quit IRC13:58
*** jistr is now known as jistr|call14:05
hrwgfidente: +2 and let is go through zuul14:07
gfidentehrw hi thanks!14:08
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Wait for mariadb to stop after shutdown  https://review.opendev.org/66736314:17
*** jistr|call is now known as jistr14:22
*** spiette has joined #openstack-kolla14:23
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Add upgrade-bifrost command  https://review.opendev.org/66763014:24
*** heikkine has joined #openstack-kolla14:24
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Add upgrade-bifrost command  https://review.opendev.org/66763014:27
*** ktibi has joined #openstack-kolla14:35
hrwyoctozepto: you will have some SKIPPED_IMAGES stuff for review later this week ;d14:36
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Wait for mariadb to stop after shutdown  https://review.opendev.org/66736314:36
*** mloza has quit IRC14:40
*** k_mouza has joined #openstack-kolla14:43
*** luksky has quit IRC14:43
hrwheh those weird days when I run multidistro run and in meantime prepare bunch of patches for next run ;D14:47
*** k_mouza has quit IRC14:47
mgoddard== meeting in 10 minutes ==14:50
* hrw out14:50
hrwsorry14:50
mgoddardmnasiadka yoctozepto egonzalez ^14:51
*** dougsz has joined #openstack-kolla14:51
hrwmgoddard: probably subject #1: how to unbreak CI14:51
mgoddardhrw: the horizon issue?14:51
hrwmgoddard: and retrylimit14:52
mgoddardI thought that one had gone away?14:52
hrwmgoddard: due to retry limit and broken iad mirror it is in queue14:52
hrwhttps://review.opendev.org/#/c/667386/14:52
hrwsent for another recheck14:53
mgoddard"msg": "No package matching 'libffi-dev' is available"14:53
mgoddardmirror issues?14:53
hrwlooks like14:54
hrwas libffi-dev is present in distro14:54
hrwmgoddard: would be nice to have kind of 'super priority CI job' available for PTL devs. kind of "this is so important that all other our jobs can wait but his HAS to run"14:55
hrw;D14:56
hrws/his/this14:56
* hrw out14:57
hrwwill read notes tomorrow14:57
*** k_mouza has joined #openstack-kolla14:57
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed Jun 26 15:00:28 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
openstackThe meeting name has been set to 'kolla'15:00
mgoddard#topic rollcall15:00
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:00
mgoddard\o15:00
mgoddardping mnasiadka yoctozepto egonzalez15:02
mnasiadkao/15:02
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Don't rotate keystone fernet keys during deploy  https://review.opendev.org/66688215:03
mgoddardquiet one today15:05
mnasiadkaseems so15:06
mgoddard#topic agenda15:06
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:06
mgoddard* Roll-call15:06
mgoddard* Announcements15:06
mgoddard** Kolla Train priorities results are in https://etherpad.openstack.org/p/kolla-train-priorities15:06
mgoddard** Vote for kayobe becoming (or not) a kolla deliverable http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007250.html15:06
mgoddard** Discussion on ML about octavia in kolla http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007251.html15:06
mgoddard* Review action items from last meeting15:06
mgoddard* Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:06
mgoddard* Stein release status15:06
mgoddard* IPv6 community goal15:06
mgoddard#topic announcements15:06
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:06
mgoddard#info Kolla Train priorities results are in15:06
mgoddard#link https://etherpad.openstack.org/p/kolla-train-priorities15:06
mgoddard#info Vote for kayobe becoming (or not) a kolla deliverable15:07
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007250.html15:07
mgoddard#info Discussion on ML about octavia in kolla15:07
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007251.html15:07
mgoddard#topic review action items from last meeting15:07
*** openstack changes topic to "review action items from last meeting (Meeting topic: kolla)"15:07
mgoddardmgoddard to post to ML about octavia in kolla15:08
mgoddardmgoddard to ask openstack-discuss to vote on making kayobe a kolla deliverable15:08
mgoddardyoctozepto to create an etherpad with a taxonomy of CI errors and a count of how many times each has been hit15:08
mgoddarddone, done, done15:08
mgoddard#link https://etherpad.openstack.org/p/kolla-ci-errors15:08
mgoddard#topic Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:08
*** openstack changes topic to "Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard (Meeting topic: kolla)"15:08
mgoddardWe have some CI issues currently15:09
mgoddardYesterday it was RETRY_LIMIT and finger15:09
mgoddardThose seem to be not our fault15:09
mgoddardThere is a horizon issue which hrw and yoctozepto are looking at15:09
mgoddard#link https://review.opendev.org/66738615:10
mgoddardImage build times out CI job15:10
mgoddardAny other issues you're awar of mnasiadka ?15:10
mnasiadkanot really15:11
mgoddard#topic Stein release status15:11
*** openstack changes topic to "Stein release status (Meeting topic: kolla)"15:11
mgoddardStill waiting on the patch to switch back to centos ceph repos15:12
mgoddardSeems to be taking forever to get patches merged right now15:12
mgoddardbackported to stein here: https://review.opendev.org/66761615:12
mnasiadkaI'll try to watch that one - that's essential for stein release...15:13
mgoddardyeah15:13
mnasiadkado we have any more stein stoppers?15:13
mgoddardalso have https://bugs.launchpad.net/kolla-ansible/+bug/1834191 which I would say is essential15:13
openstackLaunchpad bug 1834191 in kolla-ansible stein "Multinode galera upgrade from Rocky to Stein fails on CentOS" [Critical,In progress] - Assigned to Mark Goddard (mgoddard)15:13
mgoddardhttps://review.opendev.org/66736315:14
*** hamzaachi_ has joined #openstack-kolla15:14
mgoddardI think that patch is in a good state now15:14
mgoddardSo I think we just need to keep pushing on those two patches15:15
*** hamzaachi has quit IRC15:16
mgoddardLet's leave the IPv6 discussion since yoctozepto isn't around15:16
mgoddard#topic open discussion15:16
*** openstack changes topic to "open discussion (Meeting topic: kolla)"15:16
mgoddardAnything to say for yourself mnasiadka?15:16
mnasiadkanaah, I'll try to make some progress next week on bumping Ansible version in kolla_toolbox15:17
mgoddardthat would be nice15:17
mgoddardlet's leave it there then. Thanks for joining15:18
mnasiadkathanks for hosting15:18
mgoddard#endmeeting15:18
*** openstack changes topic to "Topic for #openstack-kolla is: support: ask.openstack.org | New to Kolla: docs @ https://docs.openstack.org/kolla/latest/ | | Kolla IRC meetngs on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"15:18
openstackMeeting ended Wed Jun 26 15:18:46 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:18
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-26-15.00.html15:18
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-26-15.00.txt15:18
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-26-15.00.log.html15:18
*** cah_link has quit IRC15:21
*** whoami-rajat has quit IRC15:22
*** k_mouza has quit IRC15:25
openstackgerritMark Goddard proposed openstack/kolla master: Move back to CentOS StorageSIG repos for Ceph and Ganesha  https://review.opendev.org/66601515:27
openstackgerritMark Goddard proposed openstack/kolla master: Move back to CentOS StorageSIG repos for Ceph and Ganesha  https://review.opendev.org/66601515:27
*** dpawlik has quit IRC15:28
*** icarusfactor has joined #openstack-kolla15:28
*** factor has quit IRC15:29
*** rgogunskiy has quit IRC15:31
yoctozeptomy dear, leaving you for a few hours and what a ton of messages15:35
openstackgerritGiulio Fidente proposed openstack/kolla stable/stein: Move back to CentOS StorageSIG repos for Ceph and Ganesha  https://review.opendev.org/66761615:35
yoctozeptogee, I thought the meeting's on Thursday15:36
*** hamzaachi_ has quit IRC15:37
*** whoami-rajat has joined #openstack-kolla15:38
*** cgrosjean has quit IRC15:48
*** ktibi has quit IRC15:49
yoctozeptomgoddard, mnasiadka: sorry for being absent15:50
yoctozeptothe remaining issues with CI:15:50
yoctozepto1) timeout on centos-715:51
yoctozeptodue to horizon image15:51
yoctozeptowe know centos is slow for everyone15:51
yoctozeptowe can mitigate this a bit, seems like horizon posted some new patch worth checking out (ping hrw as well)15:51
yoctozepto2) mirror.iad.rax is having bad days again15:52
yoctozeptoinfra informed but what can they do about it ;D15:52
yoctozeptoit's a being in itself15:52
yoctozeptoit has its moods15:52
yoctozeptohrw: horizon patch -  https://review.opendev.org/66764015:53
*** k_mouza has joined #openstack-kolla15:57
*** igordc has joined #openstack-kolla15:59
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Add upgrade-bifrost command  https://review.opendev.org/66766516:03
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Configure Monasca Grafana user roles  https://review.opendev.org/66767116:13
mgoddardthanks for updates yoctozepto16:17
*** gfidente has quit IRC16:18
yoctozeptomgoddard: https://review.opendev.org/666882 I think you missed the meat16:19
yoctozeptofernet-push.sh.j2 in fact16:20
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Don't rotate keystone fernet keys during deploy  https://review.opendev.org/66688216:21
openstackgerritMerged openstack/kolla-ansible stable/stein: Make cron jobs generator command task in keystone non-changing  https://review.opendev.org/66759916:24
mgoddardyoctozepto: you are faster than CI :)16:24
mgoddardyoctozepto: and more reliable right now!16:24
*** k_mouza has quit IRC16:25
yoctozeptoyoctozepto: thanks! hahaha16:26
yoctozeptoin fact I wanted to reply about idempotency case, yet could not find the file to discuss16:26
yoctozepto;D16:26
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Don't rotate keystone fernet keys during deploy  https://review.opendev.org/66688216:31
*** k_mouza has joined #openstack-kolla16:34
yoctozeptomgoddard: that was fast16:38
mgoddardindeed16:38
mgoddardwe can check reconfigure logs for idempotency16:38
yoctozeptomgoddard: exactly how I find my fix targets16:43
mgoddardyoctozepto: if you're looking for things to do... http://bugs.launchpad.net/kolla and http://bugs.launchpad.net/kolla-ansible16:44
mgoddardor https://etherpad.openstack.org/p/kolla-train-priorities16:45
yoctozeptomgoddard: yeah, figured that out, I actually started working on one of the bugs and I should resume, the CI thing + regular work got higher prio16:47
*** k_mouza has quit IRC16:47
yoctozeptomgoddard: there is a ton of bugs to verify and fix, it will never get boring16:48
yoctozeptoremind me, we are looking back as far as ocata?16:48
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Wait for mariadb to stop after shutdown  https://review.opendev.org/66736316:54
mgoddardyoctozepto: ocata is EOL16:54
mgoddardwell it's extended maintenance, but we can EOL it soon16:54
mgoddardand pike is extended maintenance, so we don't need to actively maintain it, but can accept patches16:54
*** dougsz has quit IRC16:56
yoctozeptoso queens and up really, good17:03
*** iclon_ has joined #openstack-kolla17:17
*** zbr|ruck is now known as zbr17:18
*** iclon has quit IRC17:20
*** Sravan has joined #openstack-kolla17:22
*** hamzaachi has joined #openstack-kolla17:22
openstackgerritKrzysztof Klimonda proposed openstack/kolla-ansible master: Make fluentd-elasticsearch configuration more robust  https://review.opendev.org/66174717:26
*** luksky has joined #openstack-kolla17:29
*** gkadam has quit IRC17:30
*** gkadam has joined #openstack-kolla17:35
openstackgerritRadosław Piliszek proposed openstack/kolla master: Speed up builds for horizon slowed down by recent upstream change  https://review.opendev.org/66720417:38
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Wait for mariadb to stop after shutdown  https://review.opendev.org/66736317:40
openstackgerritRadosław Piliszek proposed openstack/kolla master: Speed up builds of horizon slowed down by recent upstream change  https://review.opendev.org/66720417:45
*** k_mouza has joined #openstack-kolla17:48
*** jistr_ has joined #openstack-kolla17:52
*** klindgren_ has joined #openstack-kolla17:52
*** k_mouza has quit IRC17:52
*** klindgren has quit IRC17:53
*** jistr has quit IRC17:53
*** altlogbot_2 has quit IRC17:55
*** altlogbot_2 has joined #openstack-kolla17:57
*** nathharp has joined #openstack-kolla17:57
*** altlogbot_2 has quit IRC17:57
*** nathharp has quit IRC17:57
*** nathharp has joined #openstack-kolla17:58
*** altlogbot_2 has joined #openstack-kolla17:59
*** altlogbot_2 has quit IRC18:01
*** altlogbot_3 has joined #openstack-kolla18:02
*** jonaspaulo has joined #openstack-kolla18:17
*** ironfoot has quit IRC18:21
*** ironfoot has joined #openstack-kolla18:22
*** ivve has joined #openstack-kolla18:45
*** nathharp has quit IRC18:46
mnasiadkayoctozepto, mgoddard, hrw: Just thinking - maybe it's better just to not run compilemessages for now, and wait when horizon bug will be fixed? I feel that we are overdoing it :)18:46
*** nathharp has joined #openstack-kolla18:46
yoctozeptomnasiadka: horizon is not going to "fix" this18:48
yoctozeptothis is not a bug for them18:48
yoctozeptoour usage needs adaptation anyway18:48
mnasiadkayoctozepto: it does, but do you really want to use django-admin compilemessages instead of the ,,horizon supported'' way?18:49
yoctozeptomnasiadka: nope, I would prefer it to be like they advertise it but it is not, now we are just doing the "Django supported" way ;-)18:51
*** nathharp has quit IRC18:51
mnasiadkayoctozepto: I just feel that we will need to redo it anyway ;)18:51
yoctozeptomnasiadka: I share your worries18:52
yoctozeptomnasiadka: best drop the compilemessages altogether and let users compile their languages ;D18:52
mnasiadkayoctozepto: well, it's master - as a workaround we can maybe drop the compilemessages and work on a better solution when some patches are merged to horizon ;)18:53
yoctozeptomnasiadka: yeah, we can also check whether there is anyone interested in translated horizon ;D18:54
mnasiadkayoctozepto: I personally hate the polish translations :D18:54
yoctozeptomnasiadka: unfortunately they suck18:54
yoctozeptothe forcibly translated terminology makes me uncomfortable18:55
yoctozeptomnasiadka: though it would be fun if some finally used "Poniechaj" for "Cancel", don't you think?18:56
mnasiadkauhh, I see we're in a kind of CI break loop... horizon needs to be fixed, then we need to go back to CentOS Storage SIG repos, and then we might survive...18:56
yoctozeptomnasiadka: yeah, I told you to +2 w+1 and now it's too late :D18:57
yoctozeptoit's like a game18:57
yoctozeptohrw: https://bugs.launchpad.net/kolla-ansible/+bug/1833835 - can you reproduce this? I tried hard today but cannot break it18:58
openstackLaunchpad bug 1833835 in kolla-ansible "[debian] MountFlags=shared fail for docker 18.09" [Undecided,Incomplete] - Assigned to Radosław Piliszek (yoctozepto)18:58
mnasiadkayoctozepto: well, I love stuff like this: https://opendev.org/openstack/horizon/src/branch/master/horizon/locale/pl_PL/LC_MESSAGES/django.po#L33219:00
yoctozeptomnasiadka: you mean "stosu"?19:01
mnasiadkayoctozepto: yeah :)19:01
yoctozeptomnasiadka: then we agree this translation should burn at "stos"19:02
yoctozeptoto be fair, these translations look good overally, I have no idea how to improve this sentence19:03
yoctozeptoit sounds bad, but seems we have nothing better19:03
mnasiadkayup19:07
*** Sravan has quit IRC19:18
*** dims has quit IRC19:20
*** hamzaachi has quit IRC19:20
*** whoami-rajat has quit IRC19:22
yoctozeptohttps://bugs.launchpad.net/kolla-ansible/+bug/1794422 - first step on the path to cleaning up the bug queue19:22
openstackLaunchpad bug 1794422 in kolla-ansible "1044, \"Access denied for user 'root'@'%' to database 'keystone'\")" [Undecided,Invalid]19:22
openstackgerritMerged openstack/kolla stable/ocata: Disable publishing images on Ocata  https://review.opendev.org/66574219:24
*** dims has joined #openstack-kolla19:28
*** Sravan has joined #openstack-kolla19:32
*** dims has quit IRC19:32
hrwyoctozepto, mgoddard, mnasiadka: what about we skip compilemessages in kolla completely and do that in extend_start on container start?19:33
*** dims has joined #openstack-kolla19:33
mnasiadkahrw: I had the same thought looking at the whole crapchain of commands in Dockerfile...19:35
hrw;)19:37
*** kplant has quit IRC19:37
yoctozeptohrw: except that we run it anyway? :D19:38
*** hamzaachi has joined #openstack-kolla19:38
yoctozeptomnasiadka, hrw: yeah, the Dockerfile is a mess, not sure whether all those lines are necessary anyway19:38
hrwcould be probably rewritten to do magic once with set of variables set for distro/type set19:39
hrwbut first we need to get it done in ANY way19:40
yoctozeptoalso, we don't really use the horizon user19:40
yoctozeptowhich we set up19:40
yoctozeptothat's why I managed to just skip the chown part19:41
yoctozeptoit is not even done for ubuntu binary19:41
yoctozeptomess all around if you ask me19:41
hrwbinary image should have permissions set by package manager19:41
yoctozeptohrw: tell it to centos path19:41
hrwpath or part?19:42
yoctozeptological path19:42
yoctozeptoexecution path19:42
yoctozeptosynonyms hee19:42
yoctozeptohere*19:42
yoctozeptoanyways, horizon probably needs more testing as some guy last week reported that it is broken in master anyway in some combinations19:43
yoctozeptodistro/type19:43
hrwyoctozepto: +2 to your patch19:44
yoctozeptohrw: thanks, I like it too19:44
hrwI do not care which horizon patch gets merged as long as one will19:44
yoctozeptoyours was better until you butchered django :P19:44
yoctozeptoI checked mine to the point that I can confirm we are not doing anything worse with what we have19:45
yoctozeptosince it results in the very same image sans that chown now but it matters not19:45
*** altlogbot_3 has quit IRC19:45
yoctozeptoit had root-owned files inside anyway19:46
yoctozeptoand apache was able to access it because it itself is run from root19:46
*** dims has quit IRC19:46
*** altlogbot_1 has joined #openstack-kolla19:47
yoctozeptomnasiadka: https://review.opendev.org/66720419:47
yoctozeptopress the right buttons and get us out of CI hell19:47
yoctozeptoor mgoddard or egonzalez ^19:49
mnasiadkaDone, let’s get out of CI jail and merge ceph stuff...19:49
yoctozeptomnasiadka: and all the other stuff19:49
yoctozeptoI'm taking a break now to get some good night sleep19:50
yoctozeptohave a good night19:50
*** dims has joined #openstack-kolla19:52
hrwI restart some builds for skipped images stuff19:53
hrwmy aarch64 box hang few hours ago ;(19:53
*** dims has quit IRC19:57
*** dims has joined #openstack-kolla19:57
*** Sravan has quit IRC19:58
*** dims has quit IRC20:08
*** altlogbot_1 has quit IRC20:15
*** altlogbot_2 has joined #openstack-kolla20:18
*** Sravan has joined #openstack-kolla20:29
*** Sravan has quit IRC20:34
*** hamzaachi has quit IRC20:36
*** kplant has joined #openstack-kolla20:38
*** SaiHarshaK has quit IRC20:42
*** hamzaachi has joined #openstack-kolla20:42
*** altlogbot_2 has quit IRC20:43
*** altlogbot_3 has joined #openstack-kolla20:44
*** altlogbot_3 has quit IRC21:00
*** altlogbot_0 has joined #openstack-kolla21:04
*** pcaruana has quit IRC21:05
*** ivve has quit IRC21:07
*** hamzaachi has quit IRC21:08
*** Sravan has joined #openstack-kolla21:14
*** mloza has joined #openstack-kolla21:33
*** hamzaachi has joined #openstack-kolla21:40
*** Sravan has quit IRC21:55
*** Sravan has joined #openstack-kolla21:55
openstackgerritMerged openstack/kolla stable/stein: Move back to CentOS StorageSIG repos for Ceph and Ganesha  https://review.opendev.org/66761621:56
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: monasca: add 'confluent-kafka' to base  https://review.opendev.org/66566221:57
*** Sravan has quit IRC21:58
*** Sravan has joined #openstack-kolla21:58
*** Sravan has quit IRC22:09
*** Sravan has joined #openstack-kolla22:10
*** Sravan has quit IRC22:13
*** Sravan has joined #openstack-kolla22:14
*** luksky has quit IRC22:16
*** iniazi has quit IRC22:21
*** goldyfruit has joined #openstack-kolla22:21
*** heikkine has quit IRC22:26
*** hamzaachi has quit IRC22:38
*** jonaspaulo has quit IRC22:40
*** schwicht has joined #openstack-kolla22:48
*** schwicht has quit IRC22:50
*** Sravan has quit IRC23:16
openstackgerritMerged openstack/kolla master: Speed up builds of horizon slowed down by recent upstream change  https://review.opendev.org/66720423:21
*** igordc has quit IRC23:25
*** goldyfruit has quit IRC23:30
*** goldyfruit has joined #openstack-kolla23:32
*** fabstao has joined #openstack-kolla23:47
*** icarusfactor has quit IRC23:51
*** Sravan has joined #openstack-kolla23:56
« Tuesday, 2019-06-25 Index Thursday, 2019-06-27 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!