Tuesday, 2019-07-09

*** igordc has quit IRC00:59
*** hongbin has joined #openstack-kolla01:03
*** happyhemant has quit IRC01:17
*** JangwonLee_ has joined #openstack-kolla01:27
*** JangwonLee has quit IRC01:30
*** altlogbot_0 has quit IRC01:30
*** altlogbot_2 has joined #openstack-kolla01:31
*** hongbin has quit IRC01:57
*** KeithMnemonic has quit IRC02:26
*** vmixor has quit IRC02:46
*** hongbin has joined #openstack-kolla02:57
*** icarusfactor has quit IRC03:03
*** whoami-rajat has joined #openstack-kolla03:12
openstackgerritMerged openstack/kolla-ansible stable/stein: CI: add periodic-stable-jobs Zuul project template  https://review.opendev.org/66963003:16
*** BjoernT has joined #openstack-kolla03:17
*** Sravan has joined #openstack-kolla03:17
*** shyamb has joined #openstack-kolla03:40
*** skramaja has joined #openstack-kolla03:53
*** hongbin has quit IRC04:04
*** Sravan has quit IRC04:15
*** factor has joined #openstack-kolla04:16
*** shyamb has quit IRC04:20
*** Sravan has joined #openstack-kolla04:27
*** Sravan has quit IRC04:28
*** Sravan has joined #openstack-kolla04:29
*** Sravan has quit IRC04:33
*** pcaruana has joined #openstack-kolla04:35
*** pcaruana has quit IRC04:38
*** Sravan has joined #openstack-kolla05:03
*** Sravan has quit IRC05:08
*** ivve has joined #openstack-kolla05:19
*** BjoernT has quit IRC05:26
*** luksky11 has joined #openstack-kolla05:30
*** shyamb has joined #openstack-kolla05:36
*** Luzi has joined #openstack-kolla05:37
*** shyam89 has joined #openstack-kolla05:41
*** shyamb has quit IRC05:41
*** cah_link has joined #openstack-kolla06:02
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Deprecate Ceph deployment  https://review.opendev.org/66979206:07
yoctozeptomorning06:15
yoctozeptomgoddard: if you check the failures for nova refix, you will find that mariadb fixes did not help :-(06:16
*** dpawlik has joined #openstack-kolla06:19
yoctozeptoand our ci errors queue grew too :-(06:22
egonzalezohwhyosa hi, looking into the ODL bug, can you check karaf.log, likely ODL havent started properly and so restconf service is not available06:41
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Upgrade ODL to fluorine release  https://review.opendev.org/60109406:49
*** shyam89 has quit IRC06:52
*** rpittau|afk is now known as rpittau06:58
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Upgrade ODL to fluorine release  https://review.opendev.org/62289607:00
*** shyamb has joined #openstack-kolla07:02
*** pcaruana has joined #openstack-kolla07:03
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Run OpenDaylight in NFV CI jobs  https://review.opendev.org/59861207:05
*** ianw is now known as ianw_pto07:16
openstackgerritJeffrey Zhang proposed openstack/kolla stable/rocky: Write hash after compressing horizon static assets  https://review.opendev.org/66980507:19
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: DNM: Troubleshoot ceph-nfs on ubuntu  https://review.opendev.org/66931507:33
openstackgerritMerged openstack/kolla-ansible master: Exit on failure in init-runonce  https://review.opendev.org/66814907:33
*** Sravan has joined #openstack-kolla07:34
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Exit on failure in init-runonce  https://review.opendev.org/66980807:36
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Test HAproxy in multinode  https://review.opendev.org/62508807:38
*** Sravan has quit IRC07:39
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Test HAproxy in multinode  https://review.opendev.org/62508807:39
*** shyamb has quit IRC07:40
*** shyamb has joined #openstack-kolla07:44
*** priteau has joined #openstack-kolla07:44
*** shyamb has quit IRC07:50
*** gkadam has joined #openstack-kolla07:55
*** gkadam has quit IRC07:55
*** fxpester has joined #openstack-kolla07:56
priteauGood morning. One of our gate jobs in Kayobe master, which uses kolla-ansible stein, is failing with the following error:07:57
priteauTASK [nova : Waiting for nova-compute services to register themselves] *********07:57
priteaufatal: [controller0]: FAILED! => {"msg": "The field 'vars' has an invalid value, which includes an undefined variable. The error was: 'nova_compute_services' is undefined\n\nThe error appears to have been in '/home/zuul/kolla-venv/share/kolla-ansible/ansible/roles/nova/tasks/discover_computes.yml': line 30, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe07:57
priteauoffending line appears to be:\n\n\n- name: Waiting for nova-compute services to register themselves\n  ^ here\n"}07:57
priteauIt looks like nova_compute_services is used before it is defined at https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/nova/tasks/discover_computes.yml#L3807:58
mgoddardpriteau: yes, we hit this yesterday due to a patch merged to stein which does not support ansible<2.807:58
mgoddardpriteau: there is a fix you can cherry pick for now07:58
egonzalezpriteau https://review.opendev.org/#/c/669730/207:59
*** Wasaac has joined #openstack-kolla08:00
mgoddardegonzalez beat me to it08:00
priteauThanks.08:01
priteauI see, it's allowed to declare the variable using the value from register as long as it is used in the until statement.08:02
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Test minimum supported and latest versions of Ansible  https://review.opendev.org/66841308:04
*** cgrosjean has joined #openstack-kolla08:07
*** k_mouza has joined #openstack-kolla08:16
*** k_mouza has quit IRC08:17
*** k_mouza has joined #openstack-kolla08:17
*** shyamb has joined #openstack-kolla08:23
yoctozeptopriteau: one just can't reference registered in vars in ansible<2.808:25
yoctozeptoseems to be a feature of 2.8 rather than bug of earlier08:25
*** cgrosjea_ has joined #openstack-kolla08:42
*** cgrosjean has quit IRC08:42
*** cgrosjean has joined #openstack-kolla08:43
mnasiadkamorning08:44
*** cgrosje__ has joined #openstack-kolla08:45
*** cgrosjea_ has quit IRC08:46
*** priteau has quit IRC08:47
*** cgrosjean has quit IRC08:47
*** priteau has joined #openstack-kolla08:48
mgoddardmorning mnasiadka08:55
mnasiadkahi mgoddard08:55
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: DNM: Troubleshoot ceph-nfs on ubuntu  https://review.opendev.org/66931509:01
*** k_mouza has quit IRC09:02
*** k_mouza has joined #openstack-kolla09:03
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: DNM: Troubleshoot ceph-nfs on ubuntu  https://review.opendev.org/66931509:05
*** cgrosje__ has quit IRC09:06
*** cgrosjean has joined #openstack-kolla09:11
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Do not require valid migration_interface for controllers  https://review.opendev.org/66963109:11
*** hamzaachi has joined #openstack-kolla09:14
yoctozeptomgoddard, mnasiadka: ^ would be pleased if you reviewed the above change ;-)09:22
*** cgrosjean has quit IRC09:26
*** cgrosjean has joined #openstack-kolla09:26
mgoddardyoctozepto: hadn't realised it was the same issue as discovery09:33
openstackgerritMerged openstack/kolla-ansible master: Fix nova deploy with Ansible<2.8  https://review.opendev.org/66973009:33
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Fix nova deploy with Ansible<2.8  https://review.opendev.org/66982809:34
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/rocky: Wait for all compute services before cell discovery  https://review.opendev.org/66969809:37
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/queens: Wait for all compute services before cell discovery  https://review.opendev.org/66970009:39
*** shyamb has quit IRC09:40
*** cgrosjean has quit IRC09:44
*** cgrosjean has joined #openstack-kolla09:45
*** pcaruana has quit IRC09:48
egonzalezdo anybody knows if second iface in infra VMs network range is reserved to each job or is used by other jobs? as for associate an IP address on that range or try to workaround something else09:48
ohwhyosaHey people! Morning!09:49
*** shyamb has joined #openstack-kolla09:52
*** priteau has quit IRC10:03
*** egonzalez has quit IRC10:07
*** egonzalez has joined #openstack-kolla10:09
stingrayzamgoddard: looks like https://review.opendev.org/#/c/668413/5 was failing while waiting for https://review.opendev.org/#/c/669730/2 to go in. now that the latter is in, the first one should recheck ok?10:12
mgoddardegonzalez: I'm not sure, would need to ask infra10:13
mgoddardhi ohwhyosa10:13
yoctozeptostingrayza: it hangs on https://review.opendev.org/66982810:14
yoctozepto(stein backport is required to pass it)10:14
stingrayzaah, gotcha :)10:15
yoctozepto<mgoddard> yoctozepto: hadn't realised it was the same issue as discovery10:32
yoctozepto^ did not get you, what did you mean?10:33
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: DNM: Troubleshoot ceph-nfs on ubuntu  https://review.opendev.org/66931510:34
*** shyamb has quit IRC10:35
*** luksky11 has quit IRC10:40
mnasiadkamgoddard: is that a huge sin, to enforce dbus installation on Ubuntu in bootstrap? (to make ceph-nfs work) :)10:43
mgoddardyoctozepto: looks like the migration_interface issue is a similar underlying issue to the nova discovery issue?10:43
mgoddardmnasiadka: probably not if you have ceph-nfs enabled10:44
yoctozeptomgoddard: not really, the only common points are that it's ansible's fault and touches variable expansion point10:48
yoctozeptomgoddard, mnasiadka: re: ceph - if we are dumping ceph and ceph-nfs does not seem to be too popular, then... ;-)10:49
mnasiadkayoctozepto: we can't deprecate and drop in the same moment10:49
mnasiadkayoctozepto: officially - it's supported :)10:49
mnasiadkayoctozepto: that it doesn't work and nobody uses that, that's something else :D10:49
mgoddardso we need to continue to support ceph for Train, and possibly U10:50
mgoddardwe could put ceph-nfs on the (long) list of things that are not really core and well maintained10:51
mgoddardif someone wants to fix things on that list that are broken, great10:51
mgoddardwe don't necessarily need to prioritise them too highly though IMO10:52
*** shyamb has joined #openstack-kolla10:52
mgoddardhopefully we can formalise some of these ideas this cycle with a support matrix10:52
*** serhatd has joined #openstack-kolla10:53
mgoddardif you want to get ceph-nfs going mnasiadka, go for it10:53
mgoddardbut don't feel obliged :)10:53
mgoddardthat's my 2c anyway10:53
mnasiadkawell, I'm nearly done - so it doesn't make sense to stop now :)10:53
mgoddardsure10:53
*** cgrosjean has quit IRC10:53
mnasiadkaand then still, in migration to ceph-ansible we should test ceph-nfs, which I will need to fix anyway ;)10:54
*** cgrosjean has joined #openstack-kolla11:00
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: DNM: Troubleshoot ceph-nfs on ubuntu  https://review.opendev.org/66931511:01
ohwhyosaShould the router gateway interface be on the same lan as the nodes?11:04
*** dosaboy has quit IRC11:05
ohwhyosaI kinda managed twice to deploy a multinode, but none of the times does the network, well... work11:05
jovial[m]When using kayobe, can I customize nova.conf on a per hypervisor level? How would I achieve that?11:06
ohwhyosamgoddard "we could put ceph-nfs on the (long) list of things that are not really core and well maintained" is that a figurative list of real, readable list?11:06
mgoddardohwhyosa: plan is for it to become a real list soon11:07
mgoddardjovial[m]: etc/kayobe/kolla/config/nova/{{ inventory_hostname }}/nova.conf11:08
mgoddardjovial[m]: or reference host/group vars in etc/kayobe/kolla/config/nova.conf11:09
*** dosaboy has joined #openstack-kolla11:09
jovial[m]thanks :)11:09
*** luksky11 has joined #openstack-kolla11:16
*** altlogbot_2 has quit IRC11:19
*** irclogbot_0 has quit IRC11:19
*** altlogbot_2 has joined #openstack-kolla11:20
*** zbr is now known as zbr|lunch11:22
*** altlogbot_2 has quit IRC11:25
*** hamdyk has joined #openstack-kolla11:40
hamdykHello, We are working on creating new container for neutron mlnx agent, how can we do that in kolla ?11:42
*** cgrosjean has quit IRC11:42
*** jistr_ has joined #openstack-kolla11:51
*** niceplace_ has joined #openstack-kolla11:52
*** jistr has quit IRC11:55
*** dannins has quit IRC11:55
*** markmcclain has quit IRC11:55
*** niceplace has quit IRC11:55
*** hogepodge has quit IRC11:55
*** mnasiadka has quit IRC11:55
*** TheJulia has quit IRC11:55
*** rwellum has quit IRC11:55
*** zbr|lunch is now known as zbr11:59
*** shyamb has quit IRC11:59
*** irclogbot_1 has joined #openstack-kolla12:00
*** cgrosjean has joined #openstack-kolla12:01
*** dannins has joined #openstack-kolla12:01
*** hogepodge has joined #openstack-kolla12:01
*** mnasiadka has joined #openstack-kolla12:01
*** TheJulia has joined #openstack-kolla12:01
*** rwellum has joined #openstack-kolla12:01
*** altlogbot_1 has joined #openstack-kolla12:02
*** irclogbot_1 has quit IRC12:05
*** altlogbot_1 has quit IRC12:05
*** altlogbot_3 has joined #openstack-kolla12:08
*** pcaruana has joined #openstack-kolla12:10
*** altlogbot_3 has quit IRC12:11
*** strigazi has quit IRC12:14
*** strigazi has joined #openstack-kolla12:15
*** cah_link has quit IRC12:18
*** shyamb has joined #openstack-kolla12:26
*** jistr_ is now known as jistr12:33
yoctozepto<ohwhyosa> Should the router gateway interface be on the same lan as the nodes?12:36
yoctozeptowhat do you mean?12:36
yoctozeptothere are oh so many routers nowadays ;D12:37
mgoddardmnasiadka: would you mind: https://review.opendev.org/#/c/669828, https://review.opendev.org/66979212:39
mgoddardhamdyk: hi. Do you want to create just an image or the kolla-ansible support too?12:40
yoctozeptomgoddard: http://logs.openstack.org/31/669631/3/check/kolla-ansible-ubuntu-source-upgrade-ceph/cbf3b5c/primary/ara-report/ <- sad :-(12:40
mgoddardhamdyk: some info on adding images here: https://docs.openstack.org/kolla/latest/contributor/CONTRIBUTING.html#adding-a-new-service12:40
mgoddardyoctozepto: I'm tired of mariadb12:42
mgoddardlet's switch to NoSQL12:43
yoctozeptomgoddard: lolz12:44
yoctozeptobut it looked so fixed!12:44
*** Wasaac has quit IRC12:45
hamdykmgoddard: I see the doc you sent for adding new services in kolla12:47
hamdykmgoddard: it's clear and detailed12:48
hamdyknot sure about kolla-ansible support12:48
hamdykbut in case we had to, is there such a document for kolla-ansible ?12:49
mgoddardhamdyk: yes: https://docs.openstack.org/kolla-ansible/latest/contributor/CONTRIBUTING.html#adding-a-new-service12:49
*** Wasaac has joined #openstack-kolla12:50
hamdykmgoddard: thanks man, you have been very helpful :D12:52
mgoddardhamdyk: np12:53
*** altlogbot_0 has joined #openstack-kolla12:54
*** altlogbot_0 has quit IRC12:57
*** skramaja has quit IRC13:09
*** priteau has joined #openstack-kolla13:14
*** goldyfruit has joined #openstack-kolla13:19
*** shyamb has quit IRC13:23
*** BjoernT has joined #openstack-kolla13:34
*** cgrosjean has quit IRC13:39
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Trivial fix: log stderr of init-runonce as well  https://review.opendev.org/66987313:40
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix nova deploy with Ansible<2.8  https://review.opendev.org/66982813:46
openstackgerritMerged openstack/kolla-ansible stable/stein: Deprecate Ceph deployment  https://review.opendev.org/66979213:46
*** BjoernT has quit IRC13:52
*** cah_link has joined #openstack-kolla13:56
*** BjoernT has joined #openstack-kolla13:57
*** Luzi has quit IRC13:57
*** cgrosjean has joined #openstack-kolla14:03
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: Testing Masakari role in gate  https://review.opendev.org/61605014:04
mgoddardyoctozepto mnasiadka what are we going to do about the stein release given the few mariadb failures that have popped up?14:10
*** heikkine has joined #openstack-kolla14:11
mnasiadkamgoddard: can we deprecate mariadb?14:14
mgoddard:)14:14
mnasiadka:)14:14
mgoddardwould love to14:15
kplantyeah, let's go ot microsoft sql!14:17
*** cah_link has quit IRC14:18
goldyfruitOracle ?14:18
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: Testing Masakari role in gate  https://review.opendev.org/61605014:18
*** whoami-rajat has quit IRC14:18
*** BjoernT_ has joined #openstack-kolla14:30
yoctozepto<kplant> yeah, let's go ot microsoft sql!14:31
yoctozepto<goldyfruit> Oracle ?14:31
yoctozeptohave you really worked with those or just heard the names? ;p14:31
yoctozeptothey fail all the same ;p14:31
yoctozepto<mgoddard> yoctozepto mnasiadka what are we going to do about the stein release given the few mariadb failures that have popped up?14:31
yoctozeptowhat do you propose?14:32
*** BjoernT has quit IRC14:32
yoctozeptois there anything better supported for the future (with a migration plan)?14:32
goldyfruityoctozepto, worked with both \o/14:32
mgoddardseriously, openstack == mysql/mariadb14:33
*** dpawlik has quit IRC14:34
mgoddardgalera is probably the issue here14:34
goldyfruitat the beginning postgres was one of the choice14:34
mgoddardreally we need to just make it more reliable. Question for now is, is it too unreliable to release?14:35
mgoddardand what happens if it fails? can you just run it again and expect it to work?14:37
yoctozeptogoldyfruit: postgres has nice clustering14:38
kplantyoctozepto: i've worked with mssql and it's cancer14:38
kplantis was being sarcastic14:38
kplanti*14:38
mgoddardpostgres seems to have a loyal following14:38
mgoddardbut overall its less popular than mariadb, and not well tested with openstack14:39
mgoddardat one point I was testing with a 'sleep 60'. I guess that wouldn't be so bad if it lets us release14:39
mgoddardhow many passes do we need to see to be sure it's working though?14:39
yoctozeptokplant: thanks for clarifying xD14:40
goldyfruitmgoddard, was it the issue with Kolla and MariaDB ?14:40
yoctozeptomgoddard: well, the thing I am worried the most14:40
yoctozeptoare scenarios like14:40
yoctozeptoit deploys14:40
mgoddardgoldyfruit: yeah14:40
yoctozeptosome API db calls go through just fine14:40
yoctozeptoand then it fails with wsrep14:40
yoctozeptowtf14:40
mgoddardI think that's due to the lack of haproxy14:40
mgoddardpossibly14:41
mgoddardif we're pointing at a node that isn't primary, maybe we'd see weirdness like that?14:41
yoctozeptomgoddard: but should not it then just work all the time? you could convince me that haproxy was to blame but not necessarily the other way around unless im missing something14:41
yoctozeptobecause you are pointing to the same node all the time?14:42
mgoddardwell we restart things so could end up with a different primary depending on timing14:42
mgoddardand non-primaries may or may not be up to date, depending on replication14:42
mgoddardhonestly I don't know14:43
goldyfruityoctozepto, mgoddard so what is the issue (sorry didn't follow everything)14:43
yoctozeptomgoddard: hmm, but it is much after the restarts and some API succeed some do not14:44
mgoddardyeah, could just be replication randomness14:44
yoctozeptomaybe there is something we are doing wrong that's so obvious that we miss it14:44
mgoddardi.e. it works if you catch it when up to date, but not if it needs to sync14:45
yoctozeptomgoddard: hmm, that's a valid reason14:45
mgoddardcould be, who knows14:45
yoctozeptogoldyfruit: mariadb issues14:45
yoctozeptomostly upgrade14:45
yoctozeptobut sometimes also in regular deploy14:46
yoctozeptowe have observed CI failures14:46
yoctozeptorelated to both mariadb failing to start properly14:46
yoctozeptoand other oddities in later stages14:46
yoctozeptorelated to wsrep issues14:46
yoctozeptogoldyfruit: https://etherpad.openstack.org/p/kolla-ci-errors14:46
goldyfruitRelated to MariaDB packages upgrade ?14:46
yoctozeptolook at mariadb/wsrep things14:46
goldyfruitoki14:47
yoctozeptogoldyfruit: we replace the containers14:47
yoctozeptobut some are for "fresh" deployments14:47
yoctozeptoso it's not bound to upgrades14:47
yoctozeptopossibly upgrades fail more often because we restart mariadb twice ;p14:47
yoctozepto(well, twice the normal times)14:47
*** hamdyk has quit IRC14:51
yoctozeptomgoddard: what if we are just hitting some bug in mariadb14:52
yoctozeptomost failures come from centos14:52
*** ivve has quit IRC14:53
goldyfruitand what about percona xtradb ?14:53
yoctozeptoI did not notice an ubuntu failure after patching14:53
mgoddardyoctozepto: the one you linked earlier was ubuntu: kef201.dh2.11p.lsw114:54
mgoddardor http://logs.openstack.org/31/669631/3/check/kolla-ansible-ubuntu-source-upgrade-ceph/cbf3b5c/primary/ara-report/14:54
yoctozeptohmm, then I missed them in etherpad :-(14:54
yoctozeptoah, I did not group them well14:55
yoctozeptolet me do some etherpad regrouping14:55
goldyfruitmgoddard, yoctozepto during the upgrade how do you stop the containers ?14:57
goldyfruitwaiting the 10s from Docker and then kill -9 from Docker ?14:57
goldyfruitMaybe not related at all but we got an issue with keepalived and docker within Kolla few years ago14:58
goldyfruitif keepalived is not stopped gracefully then keepalived doesn't clean the resources properly14:58
goldyfruitMaybe something like that happens with mariadb container14:59
yoctozeptogoldyfruit: keepalived/haproxy not used14:59
goldyfruitI saw some post about that with MariaDB into Docker14:59
goldyfruityoctozepto, it was just to give an example of how bad was the kill -9 of Docker during the stop step15:00
goldyfruitof Docker container*15:00
mgoddardgoldyfruit: it's a fair point - docker is very quick to kill -9, we should up that timeout15:01
yoctozepto!15:02
mgoddardin this case we use mysqladmin shutdown because docker stop might not work for some rocky containers15:02
yoctozeptoand since we do stop/start in deploy15:02
goldyfruityeah15:02
yoctozepto<mgoddard> in this case we use mysqladmin shutdown because docker stop might not work for some rocky containers15:02
yoctozeptowe do?15:02
yoctozeptono hits in master15:02
mgoddardyeah, only in stein, for upgrades15:03
yoctozeptomgoddard: but stein->master fails the same15:03
yoctozeptoor you mean rocky->stein does not now?15:03
yoctozeptoI have to rethink the CI issue presentation15:03
yoctozeptowould be better to use some excel-like thing probably15:04
mgoddardwe only added mysqladmin shutdown in stein upgrades. Master still uses docker restart15:04
yoctozeptoand then just sort/filter15:04
yoctozeptomgoddard: yeah, got you15:04
mgoddardethercalc?15:04
yoctozeptothe point is you might be right15:04
yoctozeptoethercalc? o.O15:04
yoctozeptohttps://ethercalc.openstack.org/kolla-ci-errors15:05
*** cgrosjean has quit IRC15:05
*** fxpester has quit IRC15:06
yoctozeptonot the best UI/UX15:07
yoctozeptonah, too crappy15:08
yoctozeptocan we use something else?15:08
yoctozeptoor no better ideas?15:08
*** altlogbot_0 has joined #openstack-kolla15:12
yoctozeptore: postgres -> https://docs.openstack.org/oslo.db/latest/install/index.html#using-with-postgresql15:13
yoctozeptoseems to be still supported15:13
*** cgrosjean has joined #openstack-kolla15:13
mgoddardyeah sort of, but not well tested. And was considered for dropping a while back15:15
*** altlogbot_0 has quit IRC15:17
yoctozeptomgoddard: then no touching15:21
mgoddardindeed15:23
mgoddardtoo late for stein anyway15:23
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Exit on failure in init-runonce  https://review.opendev.org/66980815:25
yoctozeptomgoddard: obviously, it was more about the future15:26
yoctozeptobut it seems it must be mariadb15:27
yoctozeptohonestly I've never had to upgrade a galera cluster15:27
yoctozeptoand also don't see what wrong we could be doing in deploy15:27
yoctozeptobut maybe it's worth checking the docker kill issue15:27
yoctozeptoit never failed me locally so might be that our vms are just a bit slower than normal15:28
yoctozeptoggm15:30
yoctozeptohmm*15:30
yoctozeptolook at15:30
yoctozeptohttp://logs.openstack.org/30/669730/2/check/kolla-ansible-centos-source-upgrade-ceph/6f71867/secondary2/logs/kolla/mariadb/mariadb.txt.gz15:30
yoctozepto2019-07-08 21:54:21 0 [ERROR] mysqld: Table './mysql/user' is marked as crashed and should be repaired15:30
yoctozepto2019-07-08 21:54:21 0 [Warning] Checking table:   './mysql/user'15:30
yoctozepto2019-07-08 21:54:21 0 [ERROR] mysql.user: 1 client is using or hasn't closed the table properly15:30
yoctozepto2019-07-08 21:54:21 0 [ERROR] mysqld: Table './mysql/db' is marked as crashed and should be repaired15:30
yoctozepto2019-07-08 21:54:21 0 [Warning] Checking table:   './mysql/db'15:30
yoctozepto2019-07-08 21:54:21 0 [ERROR] mysql.db: 1 client is using or hasn't closed the table properly15:30
yoctozepto2019-07-08 21:54:21 0 [Note] WSREP: Signalling provider to continue.15:30
yoctozepto2019-07-08 21:54:21 0 [Warning] WSREP: SST position can't be set in past. Requested: 12101, Current:  12680.15:30
yoctozepto2019-07-08 21:54:21 0 [Warning] WSREP: Can't continue.15:30
yoctozepto2019-07-08 21:54:21 0 [ERROR] Aborting15:31
yoctozeptogoldyfruit: you were riiiight15:32
yoctozeptogood node:15:32
goldyfruityeah \o/15:32
yoctozepto2019-07-08 21:54:02 0 [Note] /usr/libexec/mysqld (initiated by: unknown): Normal shutdown15:32
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Stop replication15:32
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Closing send monitor...15:32
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Closed send monitor.15:32
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: gcomm: terminating thread15:32
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: gcomm: joining thread15:32
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: gcomm: closing backend15:32
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: view(view_id(NON_PRIM,94e9a10d,6) memb {15:32
yoctozepto94e9a10d,015:33
yoctozepto} joined {15:33
*** hamzy has quit IRC15:33
yoctozepto} left {15:33
yoctozepto} partitioned {15:33
yoctozepto94f373f1,015:33
yoctozeptob005a5d0,015:33
yoctozepto})15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: view((empty))15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: gcomm: closed15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 115:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Flow-control interval: [16, 16]15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Trying to continue unpaused monitor15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Received NON-PRIMARY.15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 12680)15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Received self-leave message.15:33
yoctozepto2019-07-08 21:54:02 10 [Note] WSREP: New cluster view: global state: 87be6844-a1be-11e9-9356-b282b53359d4:12680, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 315:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Flow-control interval: [0, 0]15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Trying to continue unpaused monitor15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 12680)15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: RECV thread exiting 0: Success15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: recv_thread() joined.15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Closing replication queue.15:33
yoctozepto2019-07-08 21:54:02 0 [Note] WSREP: Closing slave action queue.15:33
yoctozepto2019-07-08 21:54:02 10 [Note] WSREP: New cluster view: global state: 87be6844-a1be-11e9-9356-b282b53359d4:12680, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 315:33
yoctozepto2019-07-08 21:54:02 10 [Note] WSREP: applier thread exiting (code:0)15:33
yoctozepto2019-07-08 21:54:02 2 [Note] WSREP: applier thread exiting (code:6)15:33
yoctozepto2019-07-08 21:54:02 9 [Note] WSREP: applier thread exiting (code:6)15:33
yoctozepto2019-07-08 21:54:02 12 [Note] WSREP: applier thread exiting (code:6)15:33
yoctozepto2019-07-08 21:54:04 1 [Note] WSREP: rollbacker thread exiting15:34
yoctozepto2019-07-08 21:54:04 0 [Note] Event Scheduler: Purging the queue. 0 events15:34
yoctozepto2019-07-08 21:54:04 0 [Note] InnoDB: FTS optimize thread exiting.15:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: dtor state: CLOSED15:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: mon: entered 12679 oooe fraction 0 oool fraction 015:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: mon: entered 12679 oooe fraction 0.0914899 oool fraction 015:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: mon: entered 12940 oooe fraction 0 oool fraction 7.72798e-0515:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: cert index usage at exit 015:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: cert trx map usage at exit 11315:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: deps set usage at exit 015:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: avg deps dist 22.855315:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: avg cert interval 0.3665915:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: cert index size 10115:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: Service thread queue flushed.15:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: wsdb trx map usage 0 conn query map usage 015:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 015:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: MemPool(SlaveTrxHandle): hit ratio: 0.987776, misses: 155, in use: 0, in pool: 15515:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 12680)15:34
yoctozepto2019-07-08 21:54:04 0 [Note] WSREP: Flushing memory map to disk...15:34
yoctozepto2019-07-08 21:54:04 0 [Note] InnoDB: Starting shutdown...15:34
yoctozepto2019-07-08 21:54:04 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool15:34
yoctozepto2019-07-08 21:54:04 0 [Note] InnoDB: Buffer pool(s) dump completed at 190708 21:54:0415:34
yoctozepto2019-07-08 21:54:06 0 [Note] InnoDB: Shutdown completed; log sequence number 25668564; transaction id 3817815:34
yoctozepto2019-07-08 21:54:06 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"15:34
yoctozepto2019-07-08 21:54:06 0 [Note] /usr/libexec/mysqld: Shutdown complete15:34
yoctozeptosorry for the spam!15:34
yoctozeptowill redo in paste15:34
yoctozeptogood node:15:34
yoctozeptohttp://paste.openstack.org/show/754226/15:35
yoctozepto(look at timestamps) I pasted the whole shutdown procedure15:35
yoctozeptobad node:15:35
yoctozeptov15:35
yoctozeptohttp://paste.openstack.org/show/754227/15:35
kplanthaha15:35
yoctozeptoit did not finish shutdown15:35
yoctozeptothough why it failed to recover at all15:35
yoctozeptois beyond me15:35
yoctozeptothe whole point of clustering is to make it more reliable ;p15:35
yoctozeptomgoddard, goldyfruit: ^15:35
mgoddardjeez15:35
kplantyour client is good at buffering15:35
yoctozeptoyeah, again sorry for the spam15:35
yoctozeptothe pastes should be clearer15:36
mgoddardyoctozepto: it depends on when that crash happened. Mariadb can recover from a crash in the same version, but rocky -> stein it cannot, which is why we need to be very careful about shutting down properly15:37
yoctozeptomgoddard: 'tis be stein->master15:38
yoctozeptospecifically the docker stop issue15:38
yoctozeptoseems it can never recover15:38
yoctozepto;p15:38
yoctozeptothat's it for HA15:39
goldyfruithttp://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2017-05-22.log.html15:41
goldyfruitlook for string "docker stop -t 30 mariadb will work better"15:41
*** altlogbot_2 has joined #openstack-kolla15:42
*** altlogbot_2 has quit IRC15:47
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: Testing Masakari role in gate  https://review.opendev.org/61605015:48
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: Set default timeout to 60 seconds for docker stop  https://review.opendev.org/66989715:50
mgoddardfrom those IRC logs: "just tested upgrade from ocata -=> master and mariadb keeps restarting"15:51
mgoddardsome things never change :)15:51
yoctozeptogoldyfruit: yeah, what mgoddard did is use shutdown15:51
yoctozeptomgoddard: we should really be using shutdown, not docker stop15:51
yoctozeptobecause if it needs that 61 seconds, then let it be15:52
mgoddardyoctozepto: docker stop should trigger a shutdown15:52
*** priteau has quit IRC15:52
yoctozeptomgoddard: but will wait only 60s, not until the real shutdown15:52
mgoddardyoctozepto: this may help us: https://review.opendev.org/66989715:53
mgoddardwell, currently it will wait 10 seconds :)15:53
mgoddardwas thinking, perhaps we need a mariadb ansible module15:53
yoctozeptomgoddard: yeah, the thing in general is fine by me - e.g. for all services15:53
mgoddardmight be nicer than docker exec + mysqladmin, and could wrap up multiple tasks into one15:53
goldyfruitThat could increase upgrade time but it should make it more consistent15:54
yoctozeptoyou mean in-house or some external?15:54
mgoddardin-house15:54
yoctozeptok15:54
mgoddardwould be good if we factored out our docker interactions into a python module that we import15:55
mgoddardthese are just pipe dreams15:55
yoctozeptoI still don't get this error15:55
yoctozepto2019-07-08 21:54:21 0 [Warning] WSREP: SST position can't be set in past. Requested: 12101, Current:  12680.15:55
yoctozeptoit was a slave15:55
yoctozeptoand restarted as slave15:56
yoctozeptoand it speaks about position in the past15:56
*** igordc has joined #openstack-kolla15:56
yoctozeptodafuq15:56
*** hamzy has joined #openstack-kolla16:02
yoctozeptohttps://stackoverflow.com/questions/54664565/unable-to-complete-sst-transfer-due-to-wsrep-sst-position-cant-be-set-in-past16:03
yoctozeptounresolved16:03
*** icarusfactor has joined #openstack-kolla16:04
*** factor has quit IRC16:06
*** factor__ has joined #openstack-kolla16:06
*** altlogbot_1 has joined #openstack-kolla16:08
*** ivve has joined #openstack-kolla16:08
*** icarusfactor has quit IRC16:08
*** BjoernT_ has quit IRC16:09
mgoddardnight all \o16:12
*** altlogbot_1 has quit IRC16:13
*** BjoernT_ has joined #openstack-kolla16:13
yoctozeptonight16:17
*** Sravan has joined #openstack-kolla16:17
yoctozeptoI expanded the CI failure listing for mariadb16:18
*** henriqueof has joined #openstack-kolla16:20
*** hamzaachi has quit IRC16:20
*** altlogbot_1 has joined #openstack-kolla16:20
*** whoami-rajat has joined #openstack-kolla16:20
*** altlogbot_1 has quit IRC16:23
*** hamzaachi has joined #openstack-kolla16:23
*** irclogbot_1 has joined #openstack-kolla16:24
*** irclogbot_1 has quit IRC16:27
*** factor__ has quit IRC16:29
*** factor__ has joined #openstack-kolla16:30
*** goldyfruit has quit IRC16:35
*** rpittau is now known as rpittau|afk16:37
*** luksky11 has quit IRC16:39
*** cgrosjean has quit IRC16:41
*** k_mouza has quit IRC16:41
openstackgerritMerged openstack/kolla-ansible master: Trivial fix: log stderr of init-runonce as well  https://review.opendev.org/66987316:43
*** factor__ has quit IRC16:44
*** mgoddard has quit IRC16:45
*** mgoddard has joined #openstack-kolla16:48
*** dpawlik has joined #openstack-kolla16:49
*** altlogbot_3 has joined #openstack-kolla17:00
*** altlogbot_3 has quit IRC17:05
*** irclogbot_3 has joined #openstack-kolla17:10
*** irclogbot_3 has quit IRC17:13
*** Sravan has quit IRC17:16
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: DNM: Troubleshoot ceph-nfs on ubuntu  https://review.opendev.org/66931517:19
*** k_mouza has joined #openstack-kolla17:19
*** dpawlik has quit IRC17:20
*** k_mouza has quit IRC17:24
*** igordc has quit IRC17:29
*** Sravan has joined #openstack-kolla17:32
*** cgrosjean has joined #openstack-kolla17:40
*** Sravan has quit IRC17:42
*** hamzy has quit IRC17:47
*** Sravan has joined #openstack-kolla17:51
*** hamzy has joined #openstack-kolla17:52
*** Sravan has quit IRC17:53
*** Sravan has joined #openstack-kolla17:54
openstackgerritMerged openstack/kolla-ansible stable/stein: Exit on failure in init-runonce  https://review.opendev.org/66980817:57
yoctozeptomgoddard: https://governance.openstack.org/tc/reference/upstream-investment-opportunities/2019/glance.html <- maybe advertise kolla in a similar way18:01
mnasiadkawe are not so critical as glance :)18:03
*** Wasaac has quit IRC18:03
mnasiadkanow mds has some problem in ceph, ugh18:05
*** luksky11 has joined #openstack-kolla18:09
*** henriqueof has quit IRC18:09
*** BjoernT_ has quit IRC18:09
*** Wasaac has joined #openstack-kolla18:23
*** whoami-rajat has quit IRC18:30
*** irclogbot_2 has joined #openstack-kolla18:36
*** irclogbot_2 has quit IRC18:39
*** Wasaac has quit IRC18:40
*** henriqueof has joined #openstack-kolla18:44
*** BjoernT has joined #openstack-kolla18:45
*** Sravan has quit IRC18:48
*** hamzy has quit IRC18:49
*** hamzy has joined #openstack-kolla18:49
*** Sravan has joined #openstack-kolla18:49
*** igordc has joined #openstack-kolla18:55
*** Sravan has quit IRC18:59
*** Wasaac has joined #openstack-kolla19:01
*** factor has joined #openstack-kolla19:06
*** Wasaac has quit IRC19:07
*** Sravan has joined #openstack-kolla19:10
*** jonaspaulo has joined #openstack-kolla19:11
*** Sravan has quit IRC19:12
*** Sravan has joined #openstack-kolla19:16
*** hamzaachi has quit IRC19:18
*** Sravan has quit IRC19:19
*** Sravan has joined #openstack-kolla19:21
*** Sravan has quit IRC19:22
*** nde has joined #openstack-kolla19:24
*** Sravan has joined #openstack-kolla19:24
ndeHi all.  My deployment seems to run fine except the ceph-osd containers never fire up.  Any ideas?19:25
*** goldyfruit has joined #openstack-kolla19:25
kplantcould be a number of reasons, did you mark your disks with the proper labels?19:25
ndeI did.  The disk was labeled proper and the proper ceph items tagged in the inventory file.  All other ceph containers come up fine.  ceph-osd just never gets created.  It's strange.19:28
kplantyou look in /var/log/kolla/ ?19:29
*** Sravan has quit IRC19:29
kplantor do a ceph -s in the monitor container19:29
ndeok19:32
kplanti would double check your labels as well, they should have been changed if they were found by kolla-ansible19:33
ndehttps://pastebin.com/MeSP8rHs19:34
ndeI trimmed out critical data19:34
kplantosd: 0 osds: 0 up, 0 in19:34
kplantright there19:34
kplantif it knew about your osds it would be reporting them down19:34
kplantyou also don't want an even number of monitors19:35
kplant2n+1 is usually best practice in a cluster19:35
*** nde has quit IRC19:37
*** nde has joined #openstack-kolla19:37
ndekplant: sorry was disconnected if I missed anything after sending the pastie19:37
kplant15:34 < kplant> osd: 0 osds: 0 up, 0 in19:37
kplant15:34 < kplant> right there19:37
kplant15:34 < kplant> if it knew about your osds it would be reporting them down19:37
kplant15:35 < kplant> you also don't want an even number of monitors19:38
kplant15:35 < kplant> 2n+1 is usually best practice in a cluster19:38
ndePerhaps thats the issue altogether.  We'll trim down the even number.19:38
kplantthat's not your problem here, it's only problem in the events of failures really19:38
kplantsplit-brain19:38
ndeI see.  Pretty stange.19:39
kplanti would look at your disks and make sure kolla did find them and make the partitions it needed to19:39
ndeOk.  Would there be anything stopping Kolla from hitting /dev/sda?19:39
kplantnope, what labels did you use?19:40
ndeparted /dev/sda -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -119:40
kplantare you using filestore or bluestore19:42
ndePerhaps that's the issue.19:43
yoctozeptonde, kplant: better yet: what release of kolla-ansible are you using? the one from PyPI currently still deploys rocky only19:43
ndeIn /etc/ansible/group_vars/all.yml the system is defined as bluestore19:43
nde8.0.0.0rc119:44
yoctozeptonde: k, that's for stein19:44
yoctozeptonde: why using /etc/ansible/group_vars/all.yml ?19:44
yoctozeptoif it's inventory in /etc/ansible then be aware that we might be overriding it by group_vars/all.yml in k-a playbooks dir19:45
ndeI see.19:45
yoctozeptothough we default to bluestore anyway19:45
ndeShould it be another location that is set?19:45
ndeOk.. .perhaps it's my disk label that is incorrect because I noticed it is not for bluestore.19:46
yoctozeptomore specific groups than 'all' will override playbook's 'all'19:46
yoctozeptoand for globals we use globals.yml in /etc/kolla19:46
yoctozeptonote that globals override anything19:47
yoctozeptobecause they are sourced as extra variables19:47
yoctozepto(that's how ansible works)19:47
ndeyoctozepto: That's very helpful.  Should I just wack all.yml in that case?19:47
yoctozeptonde: yeah, saving your customizations elsewhere19:48
*** Bico_Fino has quit IRC19:49
ndeyoctozepto:  Very good.  Thanks.  I'll do that and then properly tag my disks.  Perhaps that will help.19:49
yoctozeptonde: very likely since labels were wrong indeed19:50
ndeThank you guys!19:51
yoctozeptonde: you are welcome19:53
*** cgrosjean has quit IRC20:04
*** cgrosjean has joined #openstack-kolla20:05
*** Sravan has joined #openstack-kolla20:05
*** cgrosjean has quit IRC20:06
*** ivve has quit IRC20:06
*** cgrosjean has joined #openstack-kolla20:08
*** cgrosjean has quit IRC20:09
*** Sravan has quit IRC20:10
*** cgrosjean has joined #openstack-kolla20:12
*** cgrosjean has quit IRC20:16
*** nde has quit IRC20:21
*** hamzaachi has joined #openstack-kolla20:21
*** irclogbot_1 has joined #openstack-kolla20:24
*** irclogbot_1 has quit IRC20:27
*** hamzy has quit IRC20:37
*** strigazi has quit IRC20:42
*** BjoernT_ has joined #openstack-kolla20:43
*** strigazi has joined #openstack-kolla20:43
*** BjoernT has quit IRC20:44
*** Sravan has joined #openstack-kolla21:00
*** irclogbot_3 has joined #openstack-kolla21:14
*** pcaruana has quit IRC21:19
*** irclogbot_3 has quit IRC21:19
*** goldyfruit has quit IRC21:20
*** irclogbot_1 has joined #openstack-kolla21:38
*** irclogbot_1 has quit IRC21:43
*** BjoernT_ has quit IRC21:52
*** irclogbot_0 has joined #openstack-kolla21:54
*** altlogbot_3 has joined #openstack-kolla21:55
*** altlogbot_3 has quit IRC21:55
*** henriqueof has quit IRC21:58
*** irclogbot_0 has quit IRC21:59
*** igordc has quit IRC22:00
*** cgrosjean has joined #openstack-kolla22:09
*** luksky11 has quit IRC22:12
*** Sravan has quit IRC22:14
*** goldyfruit has joined #openstack-kolla22:28
*** Sravan has joined #openstack-kolla22:34
*** hamzaachi has quit IRC22:37
*** Sravan has quit IRC22:43
*** Sravan has joined #openstack-kolla22:43
*** altlogbot_2 has joined #openstack-kolla22:44
*** serhatd has quit IRC22:45
*** altlogbot_2 has quit IRC22:49
*** jonaspaulo has quit IRC22:50
*** Sravan has quit IRC22:54
*** goldyfruit has quit IRC23:14
*** Wasaac has joined #openstack-kolla23:46
*** Wasaac has quit IRC23:50
*** hamzy has joined #openstack-kolla23:59
*** kplant has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!