Thursday, 2018-02-15

*** zhubingbing has joined #openstack-kolla00:03
*** mnasiadka has joined #openstack-kolla00:08
*** jbrooks has quit IRC00:09
*** jbrooks has joined #openstack-kolla00:11
*** mnasiadka has quit IRC00:13
*** rloo has left #openstack-kolla00:15
*** jtriley has quit IRC00:17
*** signed8bit is now known as signed8bit_Zzz00:18
*** mnasiadka has joined #openstack-kolla00:23
*** mnasiadka has quit IRC00:28
*** mnasiadka has joined #openstack-kolla00:38
*** k_mouza_ has quit IRC00:39
*** mnasiadka has quit IRC00:43
*** k_mouza has joined #openstack-kolla00:51
*** mnasiadka has joined #openstack-kolla00:53
*** dave-mccowan has joined #openstack-kolla00:54
*** k_mouza has quit IRC00:55
*** mnasiadka has quit IRC00:58
*** caoyuan has joined #openstack-kolla01:04
*** jbrooks has quit IRC01:04
*** zhubingb_ has joined #openstack-kolla01:04
*** zhubingbing has quit IRC01:05
*** jbrooks has joined #openstack-kolla01:05
*** mnasiadka has joined #openstack-kolla01:08
*** mnasiadka has quit IRC01:13
*** jbrooks has quit IRC01:16
*** mnasiadka has joined #openstack-kolla01:23
spsuryaJeffrey4l: Many Many Congratulations šŸŽ‰ šŸŽŠ šŸ˜Š01:27
spsuryaAll the best for your new role šŸ˜Š01:27
*** mnasiadka has quit IRC01:28
*** zhubingb_ has quit IRC01:30
*** numans has quit IRC01:34
*** mnasiadka has joined #openstack-kolla01:38
*** jbrooks has joined #openstack-kolla01:41
*** mnasiadka has quit IRC01:42
inc0Jeffrey4l: congrats:) well deserved!01:43
*** awidders_ has joined #openstack-kolla01:46
*** awiddersheim has quit IRC01:46
*** itlinux has joined #openstack-kolla01:47
*** dardelean has joined #openstack-kolla01:53
*** mnasiadka has joined #openstack-kolla01:53
*** spiette_ has quit IRC01:56
*** dardelean has quit IRC01:57
*** mnasiadka has quit IRC01:58
*** mnasiadka has joined #openstack-kolla02:08
*** mnasiadka has quit IRC02:13
*** mnasiadka has joined #openstack-kolla02:23
*** goldyfruit has joined #openstack-kolla02:24
*** mnasiadka has quit IRC02:28
*** zhubingbing has joined #openstack-kolla02:31
*** zhubingbing has quit IRC02:36
*** mnasiadka has joined #openstack-kolla02:38
*** jtriley has joined #openstack-kolla02:42
*** mnasiadka has quit IRC02:43
*** gkadam_ has quit IRC02:43
*** salv-orlando has joined #openstack-kolla02:44
*** awidders_ has quit IRC02:45
*** salv-orl_ has quit IRC02:47
*** aagate has quit IRC02:48
*** aagate has joined #openstack-kolla02:48
*** mnasiadka has joined #openstack-kolla02:53
*** awiddersheim has joined #openstack-kolla02:55
*** awiddersheim has quit IRC02:57
*** awiddersheim has joined #openstack-kolla02:57
*** mnasiadka has quit IRC02:58
*** chmarkus has quit IRC02:59
*** aagate has quit IRC02:59
*** chmarkus has joined #openstack-kolla03:01
*** mnasiadka has joined #openstack-kolla03:08
*** harlowja has quit IRC03:08
*** mnasiadka has quit IRC03:13
*** unicell has quit IRC03:18
sdakeJeffrey4l congrats dude :)03:18
*** goldyfruit has quit IRC03:20
*** mnasiadka has joined #openstack-kolla03:23
*** mnasiadka has quit IRC03:28
*** ArminderSingh has quit IRC03:29
*** ArminderSingh has joined #openstack-kolla03:33
*** hongbin has joined #openstack-kolla03:34
*** itlinux has quit IRC03:37
SamYapleJeffrey4l: much better than than pbourke dude!03:39
*** mnasiadka has joined #openstack-kolla03:39
*** awiddersheim has quit IRC03:42
*** mnasiadka has quit IRC03:44
*** skramaja has joined #openstack-kolla03:46
*** skramaja has quit IRC03:51
*** skramaja_ has joined #openstack-kolla03:51
*** ArminderSingh has quit IRC03:53
*** mnasiadka has joined #openstack-kolla03:54
sdakeSamYaple I noticed in the election results you applied loci for tc oversight03:54
sdakenice job :)03:55
sdakehow is your progress?03:55
sdakeoh - its chinese new year - bet Jeffrey4l is OOO03:58
*** mnasiadka has quit IRC03:59
*** itlinux has joined #openstack-kolla04:00
SamYaplesdake: pretty good. the images are running in production now. project doesnt have very high entropy normally, but htats to be expected04:02
sdakecool04:02
SamYaplekolla-ansible using loci containers is on the table still, we all just havent gotten around to doing it04:02
*** jtriley has quit IRC04:02
SamYaplehoping to get some cross gating going with that04:02
*** unicell has joined #openstack-kolla04:05
*** unicell has quit IRC04:05
sdakeyou doing this work at rax or somewhere else?04:05
sdakeor is it a free time kind of thing04:05
*** skramaja_ is now known as skramaja04:06
SamYaplei work for verizon digital media services now04:07
SamYaple9 months or so04:08
SamYaplewe are deploying them there04:08
*** mnasiadka has joined #openstack-kolla04:09
sdakenice04:10
sdakejay works there right?04:10
sdakehaivng a look at the code looks pretty straightforward04:10
SamYaplejay works for verizon wireless (VZW) i work for VDMS04:11
SamYaplevdms is basically a cdn. they used to be "edgecast"04:11
sdakeya - i imagine verizon is a huge sprawling co :)04:12
sdakemuch like other large public cos04:12
SamYapleyea i dont really keep track. i just do my thing04:12
sdakehope your doing well04:12
SamYapleso far so good. going to be in dublin?04:12
sdakeno, out of money error04:12
sdakei did book vancouver travel however04:13
SamYapletoo much hifi04:13
sdakeyo ugot it :)04:13
SamYapleyea i got a talk submitted for vancouver. ill go if its accepted04:13
sdakei could have paid for myself - but i am sick of travel04:13
SamYapleif not i dont really need to04:13
sdakei am platinum executive this year from all the trvcel last year04:13
sdake103k miles04:14
inc0that's a lot of tubing04:14
*** mnasiadka has quit IRC04:14
*** awiddersheim has joined #openstack-kolla04:14
SamYaplei did 40 last year04:14
SamYaplethis year is looking like 70k04:14
sdaketell me about it inc004:14
sdakeSamYaple a world travveler :)04:14
inc0I need to start counting miles04:15
sdakeI think I will hit about 60-75k this year04:15
sdakeinc0 if your not in a frequent flyer program, recommend getting in one ;)04:15
SamYapleyes. like yesterday04:15
inc0lol04:15
inc0well04:15
inc0I hope I won't be as frequent this year04:15
SamYapledude. the delta sky lounge has free pour liqour04:15
sdakefree miles04:16
sdakeI have over 300k miles04:16
SamYaplelike no ones around04:16
sdakejust waiting for flights/upgrades/whatever04:16
SamYapleit makes travelling way better04:16
sdakei have a membership to AA's lounge04:16
sdakeand a pass to various others04:16
*** devananda has quit IRC04:17
*** itlinux has quit IRC04:17
SamYaplei forgot to do globalentry in time04:17
sdakei am going to blow up my miles this year (hopefully) and my 4 systemwide upgrades for a vacation for th4e first time in 5+ years04:17
SamYapleim mad04:17
sdakei applied for globalentry04:17
sdakethey denied me04:17
SamYaplehaha04:17
sdake"You have had electornic equipment labeled cryptocurrency shipped from china to your house so your denied."04:17
sdakethe dude told me straight to my face04:17
SamYaplei actually got the "clear" membership thing with my delta stuff. its like tsaprecheck++04:18
sdakeglobal entry is a big deal to the us04:18
sdakei just use the phone app for entry in and out of us04:18
sdakeits much less painful04:18
SamYapleyup04:18
SamYapleim irratated04:19
inc0btw speaking of entry, hardest part of my green card is done \o/04:19
sdakewhat pisses me off bout globalentry is the dude fingerprinted me04:19
sdakeand then deined me04:19
inc0funny how they know exactly what got shipped from China to you04:19
sdakewell its on a waybill04:19
sdakeso Im not surprised04:19
sdakebut it isn't illegal04:19
sdakebitmain sent like 50k boxes in one week last year04:20
sdakecustoms doesn't care04:20
sdakethink they are mostly looking for bombs/money/drugs04:20
sdakemy crypto setup is borderline wacko :)04:21
sdake100 amp subpanel04:21
sdake7 crypto miners04:21
sdakeexhaust system04:21
sdakeair conditioning04:21
sdake100-300$/day04:22
*** mnasiadka has joined #openstack-kolla04:23
inc0lol04:24
inc0I can't buy new GPU because all that cryptocurrency minig04:24
inc0I need to wait till it's not as beneficial and buy second hand gpu for pennies04:24
*** dave-mccowan has quit IRC04:25
*** gkadam_ has joined #openstack-kolla04:26
*** mnasiadka has quit IRC04:28
sdakeya - gamers hate crypto nerds04:30
sdakeall my gear is asic- so don't blame me ;)04:30
sdakethe whole crypto thing for me was a cool and big project04:31
sdakebuilding the exhaust system from home depot and lowes parts was a real challenge04:31
sdakerunning 90 foot of wire and conduit was also a PITA04:31
*** zhubingbing has joined #openstack-kolla04:32
*** masber has joined #openstack-kolla04:33
*** skramaja has quit IRC04:35
*** gkadam_ has quit IRC04:36
*** zhubingbing has quit IRC04:37
sdakeinc0 yo ustll play diabolo3?04:37
inc0I turn it on every now and then04:38
inc0not much04:38
*** gkadam has joined #openstack-kolla04:38
*** mnasiadka has joined #openstack-kolla04:39
*** lpetrut has joined #openstack-kolla04:39
*** skramaja has joined #openstack-kolla04:42
*** hongbin has quit IRC04:43
*** ArminderSingh has joined #openstack-kolla04:43
*** mnasiadka has quit IRC04:44
gkadamJeffrey4l, congratulations! :)04:46
*** mnasiadka has joined #openstack-kolla04:47
*** mnasiadka has quit IRC04:52
*** mnasiadka has joined #openstack-kolla04:54
*** janki has joined #openstack-kolla04:57
*** mnasiadka has quit IRC04:58
*** janki has quit IRC05:03
*** janki has joined #openstack-kolla05:03
*** glycerine has quit IRC05:04
*** janki has quit IRC05:06
*** glycerine has joined #openstack-kolla05:06
*** janki has joined #openstack-kolla05:06
*** lpetrut has quit IRC05:08
*** janki has quit IRC05:08
*** mnasiadka has joined #openstack-kolla05:09
*** janki has joined #openstack-kolla05:09
*** threestrands_ has joined #openstack-kolla05:09
*** janki has quit IRC05:09
*** threestrands_ has quit IRC05:09
*** threestrands_ has joined #openstack-kolla05:09
*** threestrands has quit IRC05:09
*** janki has joined #openstack-kolla05:10
*** threestrands_ has quit IRC05:10
*** threestrands_ has joined #openstack-kolla05:11
*** mnasiadka has quit IRC05:14
*** harlowja has joined #openstack-kolla05:15
*** threestrands has joined #openstack-kolla05:15
*** threestrands has quit IRC05:15
*** threestrands has joined #openstack-kolla05:15
*** coolsvap has joined #openstack-kolla05:17
*** threestrands_ has quit IRC05:18
*** lpetrut has joined #openstack-kolla05:19
*** unicell has joined #openstack-kolla05:21
*** mnasiadka has joined #openstack-kolla05:23
*** mnasiadka has quit IRC05:28
*** threestrands has quit IRC05:28
*** lpetrut has quit IRC05:34
*** zhubingbing has joined #openstack-kolla05:34
*** jaosorior has quit IRC05:37
*** zhubingbing has quit IRC05:38
*** mnasiadka has joined #openstack-kolla05:38
*** janki has quit IRC05:40
*** janki has joined #openstack-kolla05:40
*** mnasiadka has quit IRC05:43
*** jaosorior has joined #openstack-kolla05:44
*** mdnadeem has joined #openstack-kolla05:46
*** unicell1 has joined #openstack-kolla05:47
*** unicell has quit IRC05:48
*** mnasiadka has joined #openstack-kolla05:53
*** mnasiadka has quit IRC05:58
*** lpetrut has joined #openstack-kolla06:06
*** mnasiadka has joined #openstack-kolla06:08
*** jchhatbar has joined #openstack-kolla06:09
*** janki has quit IRC06:09
*** awiddersheim has quit IRC06:12
*** mnasiadka has quit IRC06:13
*** harlowja has quit IRC06:14
*** lpetrut has quit IRC06:19
*** numans has joined #openstack-kolla06:34
*** mewald has joined #openstack-kolla06:36
*** mnasiadka has joined #openstack-kolla06:48
*** jbadiapa has joined #openstack-kolla06:48
*** jchhatbar has quit IRC06:56
*** janki has joined #openstack-kolla06:56
*** caoyuan_ has joined #openstack-kolla07:01
*** caoyuan has quit IRC07:02
*** olivierbourdon38 has quit IRC07:06
*** egonzalez has joined #openstack-kolla07:07
*** threestrands has joined #openstack-kolla07:09
*** threestrands has quit IRC07:09
*** threestrands has joined #openstack-kolla07:09
*** mewald has quit IRC07:09
*** janki has quit IRC07:09
*** janki has joined #openstack-kolla07:10
hrwall hail Jeffrey4l!07:19
*** lpetrut has joined #openstack-kolla07:29
dtkcongrats Jeffrey!07:39
*** pcaruana has joined #openstack-kolla07:40
*** sai___ has quit IRC07:51
*** gfidente has joined #openstack-kolla07:55
*** gfidente has quit IRC07:55
*** gfidente has joined #openstack-kolla07:55
*** egonzalez has quit IRC07:55
*** b_bezak has joined #openstack-kolla08:04
*** dardelean has joined #openstack-kolla08:14
*** egonzalez has joined #openstack-kolla08:15
*** caoyuan_ has quit IRC08:18
*** olivierbourdon38 has joined #openstack-kolla08:19
*** paken has joined #openstack-kolla08:23
*** ansiwen has joined #openstack-kolla08:23
*** dardelean has quit IRC08:25
*** dardelean has joined #openstack-kolla08:25
*** Angeh has joined #openstack-kolla08:29
*** dardelean has quit IRC08:30
*** gkadam_ has joined #openstack-kolla08:37
*** hamza21 has joined #openstack-kolla08:37
*** gkadam is now known as Guest820708:37
*** gkadam_ is now known as gkadam08:37
*** k_mouza has joined #openstack-kolla08:44
*** mgoddard_ has joined #openstack-kolla08:44
*** lvdombrkr has joined #openstack-kolla08:49
*** ktibi has joined #openstack-kolla08:54
*** sambetts|afk has quit IRC09:03
*** shardy has joined #openstack-kolla09:04
*** sambetts_ has joined #openstack-kolla09:07
*** coolsvap has quit IRC09:08
*** unicell1 has quit IRC09:12
*** unicell has joined #openstack-kolla09:12
*** unicell has quit IRC09:12
ktibiHi kolla, anyone have a doc for removing a compute node from a kolla cluster ?09:19
*** threestrands_ has joined #openstack-kolla09:21
*** threestrands has quit IRC09:21
*** jmccarthy has joined #openstack-kolla09:23
*** dardelean has joined #openstack-kolla09:29
dtkHello everybody09:30
*** caoyuan has joined #openstack-kolla09:31
dtkdoes anybody know something about the following error ?09:31
dtkTASK [keystone : Creating admin project, user, role, service, and endpoint] ******************************************************************************************************************************************************************09:31
dtkfatal: [efs10lab-infra-osmgmt01]: FAILED! => {"failed": true, "msg": "The conditional check '(keystone_bootstrap.stdout | from_json).changed' failed. The error was: Expecting ',' delimiter: line 1 column 421 (char 420)"}09:31
dtkwas uuid token provider removed?09:32
dtkI mean right now keystone defaults to fernet09:32
dtkcorrect ?09:32
dtkhttp://paste.openstack.org/show/hzKt9YX9aFD80yobDpan/09:34
dtkhere's the error shown by the keystone container after the "Creating admin project, user, role, service, and endpoint" inside the keystone register task runs09:35
pakenafaik fernet is default since Pike, and uuid tokens will be removed from Rocky09:35
dtkgotcha'09:35
dtkcan I submit a quick patch then ?09:35
dtkbecause inside the globals.yml file uuid is still the default09:35
pakenwell, I just started in this group, so maybe somebody else will give you a better feedback than I can; sorry.09:36
egonzalezcurrent code deployed to our queens branch is master, so we really using keystone's rocky now09:36
egonzalezwe have branch queens and use queens keystone code09:36
dtkmaster is what we tried installing09:37
dtk:D09:37
egonzalezyeah, but our master is queens, while keystone is rocky09:37
egonzalezwe and deployment tools release a couple weeks after openstack libs are relased09:38
egonzalezuse binary images for now, packagers probably havent promoted the code to master yet09:40
dtkahhh, gotcha'09:40
dtkso it's not a bug you are saying09:41
egonzaleznope, is just that we're using different releases09:41
egonzalezwe'll remove uuid for rocky, but cannot do such change during feature freeze09:41
*** lvdombrkr89 has joined #openstack-kolla09:42
*** caoyuan has quit IRC09:42
*** lvdombrkr has quit IRC09:44
*** haint has joined #openstack-kolla09:50
*** dardelean has quit IRC09:55
*** b_bezak has quit IRC09:57
jmccarthyegonzalez: morning ! I wanted to ask you about the question that came up yesterday, of looking into upgrades, from pike to queens ? I'm trying that out myself at the moment (well using master for queens of course)09:58
jmccarthyegonzalez: I'm trying it again soon today, and taking notes, seems ok so far, at least with a basic deploy using default settings more or less09:59
*** dardelean has joined #openstack-kolla09:59
egonzalezjmccarthy, cool, have instances running during upgrade?10:01
jmccarthyI did yep, just 1 cirros one but yep10:02
*** b_bezak has joined #openstack-kolla10:02
egonzalezjmccarthy, some services check if there are resources running to identify if is an upgrade or a fresh deploy (nova check flavors iirc)10:02
egonzalezcool10:02
egonzalezany issue found so far (except keystone tokens)10:02
jmccarthyegonzalez: Any special things I should look for ?10:03
jmccarthyegonzalez: I'm working on a review to update the upgrade docs, they don't give enough clues as to how to go about doing things10:03
egonzalezjmccarthy, +1000 for that10:04
jmccarthyegonzalez: No other issue found so far, was able to ssh into floating ip of instance again after upgrade was complete for example10:04
egonzalezcheck for errors in ovs, cinder usually have issues with mq queues10:05
egonzalezvalidate ceph, we upgraded to luminous10:05
egonzalezalso mariadb galera to 10.110:05
jmccarthyegonzalez: will try if I can add some of those, I didn't have them enabled, it was a basic deploy (no cinder or ceph, so far at least)10:05
jmccarthyegonzalez: yes ok will double check about galera10:06
egonzalezjmccarthy, i use to deploy tacker and test it. with it i validate cinder, nova, neutron, mistral, barbican, redis, etc in once ;)10:06
jmccarthyegonzalez: Ok I might ask you about that, I've not tried it out10:07
*** coolsvap has joined #openstack-kolla10:07
egonzalezor run rally and tempest10:08
*** caoyuan has joined #openstack-kolla10:09
jmccarthyegonzalez: Ok can check into that angle a bit more10:10
*** caoyuan has quit IRC10:11
*** caoyuan has joined #openstack-kolla10:11
*** cah_link has joined #openstack-kolla10:12
*** openstackgerrit has joined #openstack-kolla10:13
openstackgerritNick Jones proposed openstack/kolla-ansible master: [HAProxy] Allow extra options to be specified per listener  https://review.openstack.org/53483410:13
openstackgerritNick Jones proposed openstack/kolla-ansible master: [HAProxy] Allow extra options to be specified per listener  https://review.openstack.org/53483410:19
pakendtk: The bug https://bugs.launchpad.net/kolla-ansible/+bug/1748194 is not yet set as duplicated; do you have the link to the other bug? Thanks!10:19
openstackLaunchpad bug 1748194 in kolla-ansible ""Running MariaDB bootstrap container" fails" [Undecided,New]10:19
*** sambetts_ is now known as sambetts10:21
dtkstill looking for it, havenā€™t got a chance yet10:21
dtkwill do today10:21
*** cah_link has quit IRC10:21
*** dasTor_ has quit IRC10:31
*** skramaja has quit IRC10:31
*** dasTor has joined #openstack-kolla10:32
pakenok, thanks10:32
*** cah_link has joined #openstack-kolla10:39
*** cah_link has quit IRC10:52
*** b_bezak has quit IRC10:57
*** b_bezak has joined #openstack-kolla10:58
*** egonzalez has quit IRC10:58
mgoddard_inc0, pbourke: I added a few lines to the PTG etherpad about the HAProxy customisation discussion in yesterdays meeting10:59
mgoddard_yankcrime: ^^10:59
*** jistr is now known as jistr|mtg11:00
yankcrimethanks mgoddard_11:01
*** egonzalez has joined #openstack-kolla11:01
*** b_bezak has quit IRC11:02
*** JamesBenson has joined #openstack-kolla11:02
*** b_bezak has joined #openstack-kolla11:03
*** egonzalez_afk has joined #openstack-kolla11:04
*** egonzalez has quit IRC11:07
*** JamesBenson has quit IRC11:07
*** b_bezak has quit IRC11:08
*** pbourke-home has joined #openstack-kolla11:20
pbourke-homeJeffrey4l: congrats on the win :)11:20
*** b_bezak has joined #openstack-kolla11:21
*** b_bezak has quit IRC11:26
*** pbourke has quit IRC11:30
jmccarthyJeffrey41: Gratz on new PTL !11:31
*** pbourke has joined #openstack-kolla11:32
Jeffrey4lthanks for all guys support. i will do my best to server as my new job :D.11:35
Jeffrey4lbtw, i am in a china lunar new year 7 days holidays. So will be absent until Feb 22.11:37
*** Angeh has quit IRC11:39
*** b_bezak has joined #openstack-kolla11:40
mgoddard_yeah, congrats Jeffrey4l!11:41
Jeffrey4lmgoddard_, thanks ;D11:43
egonzalez_afkJeffrey4l, congrats man, enjoy your holidays :)11:48
sdakemorning peeps11:48
sdakehey Jeffrey4l grats on teh win11:49
sdakepbourke-home (and dounghq who isn't around) -  thanks for running as well! - hope to see you on next ballot :)11:49
pbourke-homecheers sdake11:49
*** Angeh has joined #openstack-kolla11:49
sdakeso proud of the organization we have buit :)11:50
gemaJeffrey4l: congrats!11:50
gemawhere are the results? I haven't seen an email :D11:50
sdakegema it was on the openstack mailing list11:51
gemasdake: thanks :D11:52
jmccarthysdake: +1 !11:58
jmccarthysdake: Obvi you and others are more active here than myself, just to say kolla is a great project :)12:08
*** dave-mccowan has joined #openstack-kolla12:10
*** janki is now known as janki|aafk12:12
*** janki|aafk is now known as janki|afk12:12
openstackgerritMerged openstack/kolla-ansible master: Workaround for Bifrost being unable to write to /etc/hosts  https://review.openstack.org/52954312:13
*** jistr|mtg is now known as jistr12:23
*** shardy has quit IRC12:35
*** coolsvap has quit IRC12:36
*** mgoddard_ has quit IRC12:42
*** gkadam has quit IRC12:48
mandreJeffrey4l: congrats on the PTL election!!12:51
*** hamza21 has quit IRC12:53
*** hamza21 has joined #openstack-kolla13:01
Jeffrey4lsdake, gema mandre egonzalez_afk thanks :D13:04
jmccarthyHmm I still need to do a pip install --upgrade docker==2.* when installing pike in order to get a successful deploy, is that expected ?13:07
*** rhallisey has joined #openstack-kolla13:15
caoyuanJeffrey4l: congratulations13:20
caoyuan:)13:20
ReepicheepI had some issues with ceph cache tier filling up because it was not flushing to the backend servers.  I added some values for max bytes, max objects and a few more osds and it then flushed data to the backend, my ceph pools are healthy now.13:20
*** mgoddard_ has joined #openstack-kolla13:21
Reepicheepthrough that process I also upgraded my kolla-ansible and kolla images but now my cinder-volume services all are down :(13:21
Reepicheepit's stable/ocata,  Any ideas.  I'm not able to seam to find anything in the logs regarding why cinder-volumes are not connecting.  cinder-backus on the same nodes are up13:23
Reepicheepand it is not NTP I have chrony running on all the nodes and double checked the status of the time sync13:24
caoyuanuse ceph -s command to ensure the ceph is ok13:26
caoyuanand check the cinder-volume's log13:26
caoyuanthe log path is /var/lib/docker/volumes/kolla_logs/cinder_volume.log on storage node13:27
*** lvdombrkr89 has quit IRC13:27
caoyuanmay need to stop the unnecessary cinder-volume container, just leave one run , and tail the log file , then restart the cinder-volume?13:28
caoyuanReepicheep are you still online?13:29
Reepicheepcaoyuan yeah ceph is HEALTHY with ceph -s cinder volume log doesn't seem to show much i will get you a paste bin13:31
ReepicheepI have restarded the cinder_volume containers a number of times.. I will do it again to show the logs of it starting up13:32
caoyuanhow many storage nodes?13:33
caoyuanrun one cinder-volume and stop the others?13:34
caoyuancheck the log again?13:34
caoyuanand show the info of the log13:34
Reepicheephttp://paste.openstack.org/show/673084/ that is my cinder-volume log on one of the storage nodes13:36
Reepicheephttp://paste.openstack.org/show/673085/ there are my storage nodes from cinder13:37
Reepicheepcaoyuan I will shutdown the cinder_volume container on all node and then start it on a single node and see what happens?13:39
caoyuanyeah, to get latest log13:40
*** signed8bit_Zzz is now known as signed8bit13:40
*** signed8bit has quit IRC13:41
caoyuanReepicheep: How is it going?13:48
Reepicheepcaoyuan same thing I stopped all the cinder_volume containers on all the nodes and started it on a single node.. same thing in the logs http://paste.openstack.org/show/673086/13:53
Reepicheepand the cinder-volume service is still down on that single node13:54
caoyuanemmm13:54
caoyuandocker top cinder_volume13:55
caoyuanis the cinder-volume service exist?13:55
*** awiddersheim has joined #openstack-kolla13:55
Reepicheepyes there are two cinder-volume processes in the container13:59
caoyuanweird14:02
caoyuantry to create a volume,  check the log?14:02
caoyuanhow about nova service-list14:02
caoyuanit is ok?14:02
Reepicheepit failes with no node to schedule and shows all nodes down on the log14:03
caoyuanemm14:04
Reepicheephttp://paste.openstack.org/show/673088/ nova service-list is fine14:04
caoyuanwhat did you do, before the problem occur?14:05
*** hachi has joined #openstack-kolla14:07
*** hamza21 has quit IRC14:08
Reepicheepmy problem started with an issue with my ceph cache tier.  through resolving that issue I upgade kolla containers and kolla-ansible both stable/ocata and once ceph was healthy again I notice the cinder-volume issue14:10
Reepicheepthrough the upgrade I did have to add ceph_rgw_keystone_password: because of a change to kolla-ansible14:12
Reepicheephttps://github.com/openstack/kolla-ansible/commit/fe6cbe9d668d6e293c9b0f4502e9fcdc315b7ead14:12
*** janki|afk has quit IRC14:16
caoyuanupdate O to P?14:16
Reepicheepcaoyuan I don't know if I ready to do that yet :|  I would like to wait until summer to do that upgrade, I'm in K12 education and the school districts will be out than which is normally our planned upgrade time14:19
*** dave-mccowan has quit IRC14:23
*** jaosorior has quit IRC14:23
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a spec for ceph-ansible  https://review.openstack.org/54498014:24
caoyuanI'm sure whether have any problem or not when you do this upgrade, maybe you should test it more,14:24
caoyuansorry ? I don't know how to reslove this issue14:25
caoyuanReepicheep: you can report a bug, and wait the ohter to reslove it.14:28
*** iclon has joined #openstack-kolla14:30
Reepicheepcaoyuan thanks for working with me, if I don't get it resolved here shortly I will file a bug14:30
caoyuanall right,14:30
Reepicheepbtw.. I'm noticing the service stays up for a bit sometimes almost a minute when I restart the cinder_volume containers14:30
Reepicheepstill trying to correlate any logs with it though14:31
caoyuan:|14:31
*** goldyfruit has joined #openstack-kolla14:32
Reepicheepthey stay up for right about a minute and nothing shows up in the logs at that time that I can see14:33
caoyuanso it is weird...14:33
caoyuanand no error in log14:34
*** jtriley has joined #openstack-kolla14:35
caoyuanrollback to O/stable14:36
caoyuan?14:36
*** zhubingbing has joined #openstack-kolla14:36
*** ktibi_ has joined #openstack-kolla14:38
caoyuanjust rollback the cinder service14:38
*** threestrands_ has quit IRC14:39
*** jdandrea_ has quit IRC14:40
jmccarthyDoes this mean my mariadb didn't come backup properly ? ... http://paste.openstack.org/show/673096/ everything was fine before I rebooted the nodes14:40
*** ktibi has quit IRC14:41
*** jdandrea has joined #openstack-kolla14:42
jmccarthyI get weird gateway timesout now, vip seems to be up fine, but it gives me failed login ?14:43
jmccarthy[Thu Feb 15 13:17:17.145150 2018] [:error] [pid 49] Login successful for user "admin", remote address <address>.14:43
jmccarthy[Thu Feb 15 13:44:57.185731 2018] [:error] [pid 14] Login failed for user "admin", remote address <address>.14:43
*** awiddersheim has quit IRC14:43
*** awiddersheim has joined #openstack-kolla14:43
*** ktibi_ has quit IRC14:46
jmccarthyDidn't there used to be a repair mariadb script or something like that someplace ?14:48
Reepicheepcaoyuan I going try rolling back the cinder_volume container14:48
*** ktibi has joined #openstack-kolla14:49
*** zhubingbing has quit IRC14:51
*** hachi has quit IRC14:55
sdakejmccarthy thanks - although I am not all that active with Kolla any longer - focused on bridging openstack and adjacent communities (mostly through tech called a service mesh)14:56
jmccarthysdake: Seems like it wasn't that long ago I discovered kolla (and you were running things) - time flies when your having fun as they say !15:00
jmccarthysdake: I still remember chat about vaccum tubes and amps and whatnot lol15:00
sdakeya - still into that :)15:00
sdakeI don't think I ever ran kolla15:00
sdakeit was more like kolla ran me ;)15:00
sdakethe community flogged me to deat h:)15:00
jmccarthylol15:01
sdakeof course, that is one of the key things of a good leader ;)15:01
jmccarthysdake: Hopefully the new gig is going as well !15:01
sdakewell still employed in cisco, continuing to work on openstack at the board level (e.g. macro planning)15:01
sdakejust have a slightly different technical focus15:02
jmccarthysdake: Nice :) !15:02
sdakeso not a new gig - more like an evolution of my work sine early days of opentack15:02
jmccarthysdake: I wish I didn't bring that up, my audio setup is non-existent =/15:04
jmccarthysdake: Keep putting it on the long finger .. one of these days !15:04
sdakeya its a bad hobby to get into15:05
sdakeexpensive15:05
jmccarthyWorth is tho ! I big shop near me, lots of vinyl that they never used to have (I mean lotta mainstream stuff) but still, new aisles for vinyl that were never there alongside the cds15:06
jmccarthy/is/it15:07
jmccarthy/I/a .. I need a new keyboard ;)15:07
*** caoyuan has quit IRC15:07
*** hamzy_ has quit IRC15:10
*** dave-mccowan has joined #openstack-kolla15:13
*** JamesBenson has joined #openstack-kolla15:15
*** paken has quit IRC15:16
*** mdnadeem has quit IRC15:17
*** mnasiadka has quit IRC15:18
sdakemore vinyl is manufactured then cds today15:23
sdakelol15:23
sdakei feel bad for younger artists as they don't necessarily have access to LP pressing15:24
sdakei'm a fan of this artist https://en.wikipedia.org/wiki/Sigrid_(singer) - yet no vinyl15:25
sdakeno 192/24 recordings either15:26
sdakeher cover of everybody knows is awesome15:27
*** jaosorior has joined #openstack-kolla15:27
*** iclon has quit IRC15:30
*** awiddersheim has quit IRC15:31
jmccarthyI must check that out !15:33
*** sai___ has joined #openstack-kolla15:34
Reepicheepwhere is it that I can have kolla ansible use my older version of my cinder_volume container instead of the version from "openstack_release" in globals.yml?15:34
*** robbbe has joined #openstack-kolla15:34
ReepicheepI want to override the openstack release value for just cinder to see if rolling back the container fixes my issue15:35
pbourke-homeReepicheep: check defaults.yml under the cinder role15:35
Reepicheeppbourke-home thanks15:35
pbourke-homeReepicheep: you can override the image used for any of the cinder containers15:35
*** hamzy_ has joined #openstack-kolla15:37
*** hamzy__ has joined #openstack-kolla15:39
*** hamzy__ is now known as hamzy15:39
*** hamzy has quit IRC15:39
*** hamzy has joined #openstack-kolla15:39
*** hamzy_ has quit IRC15:41
*** lpetrut has quit IRC15:42
*** dave-mccowan has quit IRC15:43
*** olivierbourdon38 has quit IRC15:44
*** iclon has joined #openstack-kolla15:44
*** gkadam has joined #openstack-kolla15:47
jmccarthyListening to Justice League soundtrack now lol15:48
*** hamzy has quit IRC15:49
*** hamzy has joined #openstack-kolla15:50
*** hamzy has quit IRC15:50
*** hamzy has joined #openstack-kolla15:51
*** zhubingbing has joined #openstack-kolla15:52
egonzalez_afkReepicheep, cinder_tag: "other_version" only for master and pike15:53
egonzalez_afkprevious versions cannot do it that simple, need to override each container tag https://github.com/openstack/kolla-ansible/blob/stable/ocata/ansible/roles/cinder/defaults/main.yml#L8415:53
egonzalez_afkbe aware of the ansible changes too, which may make downgrade config for other services to a previous release while the container in n+115:55
*** zhubingbing has quit IRC15:57
*** egonzalez_afk has quit IRC15:59
*** jmccarthy has left #openstack-kolla16:04
*** pcaruana has quit IRC16:09
*** jmccarthy has joined #openstack-kolla16:11
*** olivierbourdon38 has joined #openstack-kolla16:21
*** lpetrut has joined #openstack-kolla16:27
*** jaosorior has quit IRC16:29
*** kiennt26 has joined #openstack-kolla16:33
*** mewald has joined #openstack-kolla16:40
*** dardelean has quit IRC16:43
*** dardelean has joined #openstack-kolla16:44
*** dardelean has quit IRC16:48
*** olivierbourdon38 has quit IRC16:56
openstackgerritBrian Haley proposed openstack/kolla master: Create radvd container  https://review.openstack.org/54503617:02
*** mewald has quit IRC17:10
*** mnasiadka has joined #openstack-kolla17:13
*** b_bezak has quit IRC17:18
*** lpetrut has quit IRC17:18
*** sambetts is now known as sambetts|afk17:19
*** harlowja has joined #openstack-kolla17:19
*** pcaruana has joined #openstack-kolla17:20
*** mnasiadka has quit IRC17:21
*** marthydavid has joined #openstack-kolla17:27
*** marthydavid has left #openstack-kolla17:28
*** mnasiadka has joined #openstack-kolla17:28
*** marthydavid has joined #openstack-kolla17:28
*** mnasiadka has quit IRC17:32
*** lpetrut has joined #openstack-kolla17:34
*** zhubingbing has joined #openstack-kolla17:36
harlowjaqq, has anyone done like any kind of analysis of why kolla-ansible prints out passwords alot :(17:37
harlowjalike i don't think https://bugs.launchpad.net/kolla/+bug/1748528 is a special case17:37
openstackLaunchpad bug 1748528 in kolla "'Creating Nova databases user and setting permission' shows passwords" [Undecided,New]17:37
*** k_mouza has quit IRC17:38
harlowjait seems like there are a bunch of `no_log` and such missing17:40
*** zhubingbing has quit IRC17:41
*** mnasiadka has joined #openstack-kolla17:43
*** mnasiadka has quit IRC17:47
*** itlinux has joined #openstack-kolla17:47
*** rhallisey_ has joined #openstack-kolla17:48
*** rhallisey_ has quit IRC17:49
*** pbourke-home has quit IRC17:55
*** gkadam has quit IRC17:56
*** mnasiadka has joined #openstack-kolla17:57
*** mnasiadka has quit IRC18:02
*** bmace has quit IRC18:02
*** mgoddard_ has quit IRC18:04
*** jtriley has quit IRC18:05
*** hamza21 has joined #openstack-kolla18:08
*** Angeh has quit IRC18:09
*** mnasiadka has joined #openstack-kolla18:09
*** kiennt26 has quit IRC18:14
*** mnasiadka has quit IRC18:17
*** marthydavid has quit IRC18:18
*** mnasiadka has joined #openstack-kolla18:28
*** hamza21 has quit IRC18:29
*** bmace has joined #openstack-kolla18:29
*** mnasiadka has quit IRC18:32
*** hamza21 has joined #openstack-kolla18:33
*** lpetrut has quit IRC18:38
*** lpetrut has joined #openstack-kolla18:39
*** mgoddard_ has joined #openstack-kolla18:39
*** mnasiadka has joined #openstack-kolla18:43
*** mnasiadka has quit IRC18:47
*** jtriley has joined #openstack-kolla18:50
*** gfidente has quit IRC18:53
*** mnasiadka has joined #openstack-kolla18:58
*** paken has joined #openstack-kolla18:59
*** harlowja has quit IRC19:00
*** unicell has joined #openstack-kolla19:01
*** mnasiadka has quit IRC19:02
openstackgerritMark Hamzy proposed openstack/kolla master: ppc64le is built in delorean-deps.repo  https://review.openstack.org/54193319:05
*** jmccarthy has left #openstack-kolla19:12
*** mnasiadka has joined #openstack-kolla19:13
*** jmcevoy has joined #openstack-kolla19:17
*** mnasiadka has quit IRC19:17
openstackgerritMark Hamzy proposed openstack/kolla master: syslinux package is only built for x86_64  https://review.openstack.org/54509919:26
*** mnasiadka has joined #openstack-kolla19:28
*** mnasiadka has quit IRC19:32
*** harlowja has joined #openstack-kolla19:35
*** harlowja_ has joined #openstack-kolla19:37
*** zhubingbing has joined #openstack-kolla19:37
*** harlowja has quit IRC19:39
*** lpetrut has quit IRC19:40
*** zhubingbing has quit IRC19:42
*** mnasiadka has joined #openstack-kolla19:43
*** mgoddard_ has quit IRC19:45
*** mnasiadka has quit IRC19:48
*** munishmehan has joined #openstack-kolla19:53
*** pomac has quit IRC19:57
*** mnasiadka has joined #openstack-kolla19:58
*** mnasiadka has quit IRC20:02
*** pcaruana has quit IRC20:03
*** robbbe has quit IRC20:08
*** mnasiadka has joined #openstack-kolla20:13
*** mnasiadka has quit IRC20:17
wwriverratKolla-ansible folk :-) . I have a custom neutron service_plugin Iā€™m trying to shoe-horn in to kolla ansible. Any ideas how to append mine to generated list?20:20
*** mnasiadka has joined #openstack-kolla20:28
*** mnasiadka has quit IRC20:33
*** zhubingbing has joined #openstack-kolla20:38
*** mnasiadka has joined #openstack-kolla20:42
*** zhubingbing has quit IRC20:43
*** mgoddard_ has joined #openstack-kolla20:46
*** mnasiadka has quit IRC20:47
*** mnasiadka has joined #openstack-kolla20:57
*** mnasiadka has quit IRC21:02
openstackgerritMark Hamzy proposed openstack/kolla master: syslinux package is only built for x86_64  https://review.openstack.org/54509921:02
*** k_mouza has joined #openstack-kolla21:03
*** mgoddard_ has quit IRC21:05
*** hamza21 has quit IRC21:06
*** rhallisey has quit IRC21:10
*** k_mouza has quit IRC21:10
*** paken has quit IRC21:11
*** mnasiadka has joined #openstack-kolla21:12
*** k_mouza has joined #openstack-kolla21:13
*** mnasiadka has quit IRC21:17
*** k_mouza_ has joined #openstack-kolla21:21
*** k_mouza has quit IRC21:21
*** mnasiadka has joined #openstack-kolla21:27
*** pcaruana has joined #openstack-kolla21:31
*** k_mouza_ has quit IRC21:32
*** k_mouza has joined #openstack-kolla21:32
*** mnasiadka has quit IRC21:32
*** k_mouza_ has joined #openstack-kolla21:36
*** k_mouza has quit IRC21:36
*** mnasiadka has joined #openstack-kolla21:43
*** openstack has joined #openstack-kolla21:46
*** ChanServ sets mode: +o openstack21:46
*** mnasiadka has quit IRC21:48
*** pcaruana has quit IRC21:48
*** dardelean has quit IRC21:49
*** mnasiadka has joined #openstack-kolla21:58
*** dtk has quit IRC21:58
*** marian_tudosoiu has quit IRC21:58
*** dtk has joined #openstack-kolla21:58
*** marian_tudosoiu has joined #openstack-kolla21:58
*** mnasiadka has quit IRC22:02
*** jtriley has quit IRC22:04
*** dardelean has joined #openstack-kolla22:09
*** mnasiadka has joined #openstack-kolla22:13
*** mnasiadka has quit IRC22:17
*** mnasiadka has joined #openstack-kolla22:28
*** k_mouza_ has quit IRC22:31
*** mnasiadka has quit IRC22:33
*** k_mouza has joined #openstack-kolla22:38
*** zhubingbing has joined #openstack-kolla22:40
*** k_mouza has quit IRC22:43
*** munishmehan has quit IRC22:43
*** threestrands has joined #openstack-kolla22:43
*** threestrands has quit IRC22:43
*** threestrands has joined #openstack-kolla22:43
*** mnasiadka has joined #openstack-kolla22:44
*** zhubingbing has quit IRC22:44
*** k_mouza has joined #openstack-kolla22:45
*** ktibi has quit IRC22:47
*** mnasiadka has quit IRC22:48
*** k_mouza_ has joined #openstack-kolla22:50
*** k_mouza has quit IRC22:52
*** mnasiadka has joined #openstack-kolla22:58
*** JamesBenson has quit IRC23:01
*** mnasiadka has quit IRC23:03
*** itlinux has quit IRC23:09
*** mnasiadka has joined #openstack-kolla23:13
*** dardelean has quit IRC23:17
*** mnasiadka has quit IRC23:18
openstackgerritTony Breeds proposed openstack/kolla master: ppc64le is built in delorean-deps.repo  https://review.openstack.org/54193323:28
*** mnasiadka has joined #openstack-kolla23:28
*** mnasiadka has quit IRC23:36
*** zhubingbing has joined #openstack-kolla23:41
*** k_mouza has joined #openstack-kolla23:42
*** zhubingbing has quit IRC23:46
*** k_mouza_ has quit IRC23:46
*** JamesBenson has joined #openstack-kolla23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!