Tuesday, 2019-03-19

*** dave-mccowan has quit IRC00:11
*** luksky has quit IRC00:15
*** Sravan has quit IRC00:33
*** Sravan has joined #openstack-kolla00:38
*** Sravan has quit IRC00:47
*** Sravan has joined #openstack-kolla00:52
*** lemko has quit IRC01:04
*** Sravan has quit IRC01:07
*** hongbin has joined #openstack-kolla01:12
*** Sravan has joined #openstack-kolla01:16
*** Sravan has quit IRC01:19
*** wolverineav has joined #openstack-kolla01:24
*** shyamb has joined #openstack-kolla01:26
*** Sravan has joined #openstack-kolla01:29
*** wolverineav has quit IRC01:30
*** absubram has quit IRC01:33
*** Sravan has quit IRC01:35
*** Sravan has joined #openstack-kolla01:36
*** Sravan has joined #openstack-kolla01:39
*** dave-mccowan has joined #openstack-kolla01:51
*** Sravan has quit IRC01:56
*** Sravan has joined #openstack-kolla01:57
*** Sravan has quit IRC02:09
*** Sravan has joined #openstack-kolla02:16
*** unicell has joined #openstack-kolla02:17
*** unicell has quit IRC02:18
*** dave-mccowan has quit IRC02:23
*** wolverineav has joined #openstack-kolla02:36
*** shyam89 has joined #openstack-kolla02:50
*** shyamb has quit IRC02:53
*** Sravan has quit IRC03:04
*** itlinux has joined #openstack-kolla03:04
*** shyam89 has quit IRC03:08
*** zzzeek has quit IRC03:10
*** shyamb has joined #openstack-kolla03:11
*** zzzeek has joined #openstack-kolla03:16
*** Sravan has joined #openstack-kolla03:34
*** skramaja has joined #openstack-kolla03:38
*** Sravan has quit IRC03:39
*** janki has joined #openstack-kolla03:44
*** wolverineav has quit IRC03:45
*** shyamb has quit IRC03:52
*** shyamb has joined #openstack-kolla03:52
*** shyamb has quit IRC03:58
*** hongbin has quit IRC04:12
*** wolverineav has joined #openstack-kolla04:18
*** JamesBenson has joined #openstack-kolla04:31
*** JamesBenson has quit IRC04:33
*** Sravan has joined #openstack-kolla04:35
*** wolverineav has quit IRC04:49
*** Sravan has quit IRC05:15
*** sm806 has quit IRC05:20
*** sm806 has joined #openstack-kolla05:20
*** shyamb has joined #openstack-kolla05:39
*** shyamb has quit IRC05:41
*** shyamb has joined #openstack-kolla05:41
*** wolverineav has joined #openstack-kolla05:42
*** iniazi has quit IRC05:45
*** wolverineav has quit IRC05:51
*** iniazi has joined #openstack-kolla06:06
*** lemko has joined #openstack-kolla06:15
*** gfidente has joined #openstack-kolla06:15
*** ivve has joined #openstack-kolla06:19
*** andrein has joined #openstack-kolla06:20
*** shyamb has quit IRC06:32
*** shyamb has joined #openstack-kolla06:36
*** itlinux has quit IRC06:57
*** itlinux has joined #openstack-kolla06:58
*** Luzi has joined #openstack-kolla06:59
*** shyamb has quit IRC07:13
*** pcaruana has joined #openstack-kolla07:14
*** shyamb has joined #openstack-kolla07:16
*** itlinux has quit IRC07:21
*** wolverineav has joined #openstack-kolla07:22
*** tosky has joined #openstack-kolla07:25
*** pcaruana has quit IRC07:33
*** pcaruana has joined #openstack-kolla07:34
*** hamdyk has joined #openstack-kolla07:34
*** hamdyk has quit IRC07:37
*** hamdykhader has joined #openstack-kolla07:37
*** iniazi has quit IRC07:41
*** owalsh has quit IRC07:48
*** shyamb has quit IRC07:53
*** owalsh has joined #openstack-kolla07:54
*** hamdykhader has quit IRC07:55
*** wolverineav has quit IRC07:57
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-kolla08:02
*** hamzaachi has joined #openstack-kolla08:03
*** luksky has joined #openstack-kolla08:07
*** iniazi has joined #openstack-kolla08:07
*** luksky has quit IRC08:16
*** andrein has quit IRC08:25
mgoddardHi ivve, we recently made a change to use mariabackup instead of xtrabackup-v2, see commit 4b273eeeec086dda5a8047c807596b4095c2500308:25
mgoddardivve: could you check in the image to see if mariabackup is available?08:26
mgoddardMorning all08:27
mgoddardI've proposed some cycle highlights for kolla and kolla-ansible. Please review and see if I missed anything obvious. https://review.openstack.org/#/c/644506/08:28
mgoddardI will be offline for today and tomorrow. I expect to be online briefly to take the meeting tomorrow.08:29
ivvemgoddard: i noticed :)08:37
*** priteau has joined #openstack-kolla08:38
ivvemgoddard: i have a different question, ubuntu 18.04 repos don't have correct rabbitmq version as specified in https://github.com/openstack/kolla/blob/master/docker/rabbitmq/Dockerfile.j2#L3208:38
ivvethe one that exists is Version: 3.6.10-108:39
ivveperhaps a typo?08:39
mgoddardivve: no? it seems to be building ok in CI08:40
ivvecustom repos?08:40
ivvei performed an upgrade and i get undef errors from rabbit, removed containers and redeployed from master08:40
ivvethey are empty package & plugin wise08:41
*** shyamb has joined #openstack-kolla08:41
ivveso i started a fresh ubuntu 18.04 container and searched up the package. it is the one i pasted above ^08:41
ivve3.6.x not 3.7.x08:41
mgoddardivve: we have this in sources.list:08:42
mgoddarddeb https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu/ bionic main08:42
ivvelooking through base container atm08:43
mgoddardivve: do you have that repo in your image?08:43
ivvemgoddard: checking.. reason for me asking is i did an upgrade and it failed.. so i deleted containers + volumes (to force the bootstrap) and deployed new ones.. with a no-go on that08:46
mgoddardivve: which release from/to?08:46
ivverocky -> master08:46
ivve7.0.008:46
ivveso far only that is shaky and keystone (which required db_sync... no biggie)08:47
ivveso thats a heads up as it is not in the playbooks08:47
*** wolverineav has joined #openstack-kolla08:48
ivveshouldn't there always be a keystone-manage db_sync when upgrading keystone?08:48
*** Luzi has quit IRC08:59
*** dougsz has joined #openstack-kolla09:03
ivvemgoddard: you are right about the repo, i was missing that in my test.. so the question remains.. why rabbit containers break.. i use deploy with --tags rabbitmq make sure volumes and old containers are gone.09:05
ivvethey are redeployed and kolla-ansible exists properly but containers fail to start with "undef" and i can see that plugins are nowhere to be seen in the volume/mnesia dir09:06
mgoddardivve: didn't you say that your rabbit image did not have rabbit installed?09:07
ivverabbit yes but seems like plugins are missing.. i will doublecheck09:07
egonzalezivve keystone upgrades works a bit different since queens, it does an online upgrade of the services09:08
*** andrein has joined #openstack-kolla09:09
egonzalezregarding repos, maybe something is broken in external mirrors, we're currently using infra mirror proxies to install packages09:10
*** kukacz has quit IRC09:11
egonzalezivve kesytone upgrade commands https://github.com/openstack/kolla/blob/master/docker/keystone/keystone/extend_start.sh#L34 db_sync --expand db_sync --migrate and db_sync --contract09:12
*** k_mouza has joined #openstack-kolla09:12
*** jbadiapa has joined #openstack-kolla09:13
*** luksky has joined #openstack-kolla09:19
*** wolverineav has quit IRC09:19
openstackgerritMerged openstack/kolla master: Sahara: install also the split plugins  https://review.openstack.org/64337809:21
*** gfidente has quit IRC09:26
*** gfidente has joined #openstack-kolla09:26
hrwmorning09:26
*** wolverineav has joined #openstack-kolla09:27
*** kukacz has joined #openstack-kolla09:28
*** wolverineav has quit IRC09:29
*** kukacz has quit IRC09:29
*** kukacz has joined #openstack-kolla09:30
*** shyamb has quit IRC09:31
ivveegonzalez: hmm okay. i had to run manual keystone-manage db_sync, the playbook/container scripts didn't. it resulted in the task failing to insert into description in role table. the script inside the container i think is called kolla_keystone_manage?09:34
*** shyamb has joined #openstack-kolla09:34
*** wolverineav has joined #openstack-kolla09:35
egonzalezivve yes command is keystone-manage db_sync, expand migrate and contract basically does what only db_sync does in a single command but without API downtime09:39
*** kukacz has quit IRC09:39
*** kukacz has joined #openstack-kolla09:39
ivveit doesn't do it09:39
egonzaleznothing changed in keystone upgrades i'm aware of09:40
ivveegonzalez: so my question is, should i bugreport it?09:40
ivveshould it perform a db_sync when calling the upgrade.yml09:40
egonzalezit does but in 3 different steps, if something fails in between i think may be a keystone bug09:40
egonzalezcould you gather some logs?09:41
ivvebecause it fails running that kolla_keystone_manage script that tries to insert into role/description, because its missing09:41
ivveegonzalez: the log when it fails - https://hastebin.com/qupizerofe.makefile09:43
*** sshnaidm|afk is now known as sshnaidm|rover09:43
ivvesolution: a simple attach to the container and keystone-manage db_sync solves it09:43
*** k_mouza has quit IRC09:44
ivveoh here is the db before and after09:44
ivvehttps://hastebin.com/mogamazula.rb09:44
*** k_mouza has joined #openstack-kolla09:44
*** shyamb has quit IRC09:45
egonzalezduring upgrades we dont run register.yml playbook https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/keystone/tasks/upgrade.yml09:45
egonzalezivve are you running upgrade, reconfigure or deploy?09:46
ivveok, im just saying that if keystone changes and new version requries table upgrades you will run into this09:46
ivveupgrade09:46
egonzalezcould you show me all the ansible logs?09:47
ivvethe first one was the ansible log09:47
ivveor do you want more?09:48
egonzalezyes, where the upgrade tasks are done, db syncs, etc09:48
egonzalezalso if possible keystone manage logs too, to see the schema migrations09:49
ivveoh shit you are right09:49
ivvei used reconfigure09:49
ivveand not upgrade09:49
ivvedamn09:49
ivvesorry about that09:49
ivveshit behind the wheel, again :)09:49
egonzalezthats what i though, upgrade doesnt execute register.yml playbook which is what was failing due missing db_sync09:49
ivveaye09:50
egonzalezivve be aware that mariadb upgrade in centos/oraclelinux is now broken rocky-master upgrade, fix are under progress09:50
ivveegonzalez: thanks for the heads-up however im using ubuntu-source09:51
egonzalezroger, let me know if still failing (should not)09:51
ivvenah i fixed my own problem manually, lol09:52
ivve:)09:52
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: Add support for upgrade checkers  https://review.openstack.org/64452809:54
*** jbadiapa has quit IRC09:59
egonzalezmandre maybe you need to add a new override for mariadb image in tripleo when this gets merged https://review.openstack.org/#/c/644244/09:59
egonzalezmgoddard dump-init change was backported to rocky and queens https://review.openstack.org/#/c/612887/ , maybe we broke something in stable branchs10:04
*** k_mouza has quit IRC10:05
*** wolverineav has quit IRC10:09
*** k_mouza has joined #openstack-kolla10:10
*** jbadiapa has joined #openstack-kolla10:12
hrwis PKG-INFO in python packages required to be US-ASCII?10:14
egonzalezother placement-api issue, rdo packages copies a wsgi file with the same process name as us, so ansible deploy now fails with centos-binary10:17
hrwremote:   https://review.openstack.org/644531 README: make it us-ascii10:17
*** andrein has quit IRC10:19
*** andrein has joined #openstack-kolla10:20
*** rgogunskiy has quit IRC10:27
*** jbadiapa has quit IRC10:29
hrwremote:   https://review.openstack.org/644533 README: make it us-ascii10:29
hrwkarbor and kuryr-libnetwork10:29
egonzalezhrw when is failing in kolla, havent seen such issues10:30
egonzalezahh, with python 310:31
hrwegonzalez: https://review.openstack.org/#/c/642375/10:31
egonzalezmakes sense setuptools adds some check to maintain an standard in the packages10:32
mandreegonzalez: ack thanks for the heads up10:33
hrwegonzalez: no! those files should be UTF-8 just like Python source files.10:39
hrwmaybe in 2022 such PEP will be written10:39
hrwok. 642375 now waits for 4 code reviews. karbor, kuryr, monasca10:41
hrwbifrost crap left10:42
hrwimho we should not even build it10:43
*** jbadiapa has joined #openstack-kolla10:43
hrwmonasca accepted both10:44
*** janki has quit IRC10:45
mnasiadkahrw: first we would need to deprecate it - now we have a good alternative ;)10:59
hrwmnasiadka: I do not know the story. and probably do not want to ;d11:00
mnasiadkahrw: got approval to go to Krakow... so probably we'll meet there ;)11:01
hrwmnasiadka: lovely!11:01
hrwmnasiadka: there are some RH guys going from Wrocław by car iirc11:01
mnasiadkahrw: let me first find some normal hotel ;)11:05
hrwnot looked yet much11:06
*** priteau has quit IRC11:08
*** iniazi has quit IRC11:14
*** iniazi has joined #openstack-kolla11:16
hrwbifrost patch: https://review.openstack.org/#/c/64454111:28
*** wolverineav has joined #openstack-kolla11:37
*** janki has joined #openstack-kolla11:49
*** klindgren_ has quit IRC11:51
*** klindgren has joined #openstack-kolla12:03
*** skramaja_ has joined #openstack-kolla12:05
*** skramaja has quit IRC12:06
*** luksky has quit IRC12:07
*** jbadiapa has quit IRC12:11
*** skramaja has joined #openstack-kolla12:12
*** wolverineav has quit IRC12:12
*** skramaja_ has quit IRC12:12
*** luksky has joined #openstack-kolla12:20
*** iniazi has quit IRC12:21
*** iniazi has joined #openstack-kolla12:22
*** goldyfruit has quit IRC12:27
*** luksky has quit IRC12:29
*** mchlumsky has joined #openstack-kolla12:33
*** luksky has joined #openstack-kolla12:42
*** tolisbar has joined #openstack-kolla12:43
*** mchlumsky has quit IRC12:56
*** tolisbar has quit IRC12:57
*** baha has joined #openstack-kolla12:58
*** mchlumsky has joined #openstack-kolla12:58
*** rgogunskiy has joined #openstack-kolla12:59
*** wolverineav has joined #openstack-kolla13:01
*** skramaja has quit IRC13:07
*** Sravan has joined #openstack-kolla13:09
*** wolverineav has quit IRC13:12
*** Sravan has quit IRC13:13
*** jbadiapa has joined #openstack-kolla13:18
*** goldyfruit has joined #openstack-kolla13:18
openstackgerritChason Chan proposed openstack/kolla-ansible master: Add a soft link creating task to post_deploy.yml  https://review.openstack.org/64456913:26
*** iniazi has quit IRC13:26
*** gfidente has quit IRC13:27
*** gfidente has joined #openstack-kolla13:29
*** iniazi has joined #openstack-kolla13:37
*** Luzi has joined #openstack-kolla13:54
*** wolverineav has joined #openstack-kolla14:05
*** mjturek has joined #openstack-kolla14:07
*** tolisbar has joined #openstack-kolla14:10
*** wolverineav has quit IRC14:11
*** tolisbar has quit IRC14:22
hrwbifrost patches merged14:22
hrwwaiting for kuryr and karbur14:22
*** iniazi has quit IRC14:41
*** JamesBenson has joined #openstack-kolla14:42
*** JamesBenson has quit IRC14:46
*** itlinux has joined #openstack-kolla14:47
*** JamesBenson has joined #openstack-kolla14:50
*** rgogunskiy has quit IRC14:51
*** wolverineav has joined #openstack-kolla14:53
*** iniazi has joined #openstack-kolla14:53
*** luksky has quit IRC14:58
*** wolverineav has quit IRC15:00
*** diablo_rojo has joined #openstack-kolla15:12
*** baribalbear has joined #openstack-kolla15:12
*** Luzi has quit IRC15:12
baribalbearHello folks. Afaiu from here https://github.com/openstack/kolla-kubernetes kolla-kubernetes is obsoleted? Currently the recommended way to deploy Openstack into the K8s is to use openstack-helm?15:14
mnasiadkabaribalbear: right.15:16
baribalbearmnasiadka: and how mature is it? Is it ready for prod like deployments? Heard that TripleO also transitionling to the container based Openstack deployments. Is it better to use TrippleO or openstack-helm?15:23
JamesBensonbaribalbear: Helm is actually having a meeting in room-4 right now...15:24
JamesBensontalking about 1.0 release.15:24
mjturekmgoddard: thanks for the review on https://review.openstack.org/#/c/644389/ do you think we need to be concerned about the non-voting failures?15:41
*** yan0s has joined #openstack-kolla15:41
*** JamesBenson has quit IRC15:43
*** JamesBenson has joined #openstack-kolla15:44
*** luksky has joined #openstack-kolla15:51
*** hamzy has quit IRC15:55
*** wolverineav has joined #openstack-kolla15:59
*** itlinux has quit IRC16:17
*** jbadiapa has quit IRC16:17
*** hamzy has joined #openstack-kolla16:18
*** baha has quit IRC16:29
*** ivve has quit IRC16:31
*** jbadiapa has joined #openstack-kolla16:39
*** itlinux has joined #openstack-kolla16:45
*** henriqueof2 has quit IRC16:51
*** luksky has quit IRC16:54
*** Sravan has joined #openstack-kolla16:55
*** hamzaachi has quit IRC17:01
*** wolverineav has quit IRC17:04
*** hamzaachi has joined #openstack-kolla17:05
*** luksky has joined #openstack-kolla17:07
*** baha has joined #openstack-kolla17:08
*** Chaserjim has joined #openstack-kolla17:08
openstackgerritMerged openstack/kolla-ansible master: Run jobs on bionic instead of xenial  https://review.openstack.org/64409717:17
*** unicell has joined #openstack-kolla17:18
*** andrein has quit IRC17:22
*** priteau has joined #openstack-kolla17:25
*** janki has quit IRC17:29
*** wolverineav has joined #openstack-kolla17:31
*** luksky has quit IRC17:32
*** Sravan has quit IRC17:33
*** yan0s has quit IRC17:33
hrwok. added probably all core devs to karbur and kuryr patches. let's see how much time it will take17:43
*** hamzaachi has quit IRC17:44
*** Sravan has joined #openstack-kolla17:45
*** k_mouza has quit IRC17:47
*** Sravan has quit IRC17:50
*** Sravan has joined #openstack-kolla17:51
*** Sravan has quit IRC17:54
*** Sravan has joined #openstack-kolla17:55
*** dougsz has quit IRC17:57
*** diablo_rojo has left #openstack-kolla17:59
*** andrein has joined #openstack-kolla18:00
*** itlinux has quit IRC18:02
*** priteau has quit IRC18:07
*** ivve has joined #openstack-kolla18:07
*** k_mouza has joined #openstack-kolla18:07
*** wolverineav has quit IRC18:12
*** k_mouza has quit IRC18:12
*** andrein has quit IRC18:14
*** wolverineav has joined #openstack-kolla18:20
*** Sravan has quit IRC18:25
*** wolverineav has quit IRC18:25
*** Sravan has joined #openstack-kolla18:26
*** Sravan has quit IRC18:31
*** Sravan has joined #openstack-kolla18:32
openstackgerritJuan Badia Payno proposed openstack/kolla master: WIP: rsyslog: add plugins kafka, relp, mmkubernetes, gssapi  https://review.openstack.org/64462918:32
*** Sravan has quit IRC18:35
*** Sravan has joined #openstack-kolla18:38
*** hamzaachi has joined #openstack-kolla18:42
*** gfidente is now known as gfidente|afk18:43
*** jbadiapa has quit IRC18:50
*** baribalbear has quit IRC18:53
*** wolverineav has joined #openstack-kolla18:55
*** andrein has joined #openstack-kolla19:00
*** wolverineav has quit IRC19:04
mjturekegonzalez: thanks for the review on https://review.openstack.org/#/c/644389/ should we be concerned with the non-voting failures?19:19
hrwmjturek: yay! someone with Power!19:25
mjturek:)19:25
hrwmjturek: are you building kolla on power?19:25
hrwmjturek: I did last build nearly 2y ago19:26
mjturekhrw: we are spinning up a job to do it regularly and publish to rdo19:26
mjtureklemme grab a link19:26
hrwmjturek: lovely19:26
mjturekhttps://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/19:26
hrwmjturek: too bad that log is unreadable19:28
*** k_mouza has joined #openstack-kolla19:30
*** wolverineav has joined #openstack-kolla19:33
*** k_mouza has quit IRC19:35
mjturekhrw yeah it's rough :( the log uploads are better https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/412/logs/logs/19:51
*** itlinux has joined #openstack-kolla20:12
*** tosky has quit IRC20:30
*** gfidente|afk has quit IRC20:35
*** hamzaachi has quit IRC20:36
*** wolverineav has quit IRC20:43
*** mjturek has quit IRC20:45
*** hamzy has quit IRC20:46
*** wolverineav has joined #openstack-kolla20:53
*** luksky has joined #openstack-kolla20:54
*** baha has quit IRC21:04
*** priteau has joined #openstack-kolla21:07
*** goldyfruit has quit IRC21:08
*** goldyfruit has joined #openstack-kolla21:34
*** itlinux has quit IRC21:40
*** JamesBenson has quit IRC22:02
*** goldyfruit has quit IRC22:06
*** pcaruana has quit IRC22:08
hrwindeed22:15
*** itlinux has joined #openstack-kolla22:16
*** itlinux has quit IRC22:27
*** wolverineav has quit IRC22:32
*** wolverineav has joined #openstack-kolla22:33
*** wolverineav has quit IRC22:40
*** wolverineav has joined #openstack-kolla22:41
*** wolverineav has quit IRC22:45
*** priteau has quit IRC22:54
*** wolverineav has joined #openstack-kolla23:00
*** iniazi has quit IRC23:04
*** iniazi has joined #openstack-kolla23:04
*** mchlumsky has quit IRC23:08
*** calbers has quit IRC23:11
*** calbers has joined #openstack-kolla23:12
*** k_mouza has joined #openstack-kolla23:16
*** k_mouza has quit IRC23:21
*** luksky has quit IRC23:21
*** andrein has quit IRC23:25
*** henriqueof has joined #openstack-kolla23:36
*** unicell has quit IRC23:50
*** unicell has joined #openstack-kolla23:54
*** unicell has left #openstack-kolla23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!