Wednesday, 2017-01-25

*** shardy has quit IRC00:01
kfox1111arg.. no, its just not working at all...00:03
*** schwicht has joined #openstack-kolla00:04
*** pramodrj07 has quit IRC00:07
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds  https://review.openstack.org/41855000:10
*** mattmceuen has quit IRC00:13
*** lrensing has quit IRC00:17
*** lamt has quit IRC00:17
*** schwicht has quit IRC00:20
sbezverkkfox1111: if you want to check the vswitchd fix: https://review.openstack.org/42490200:20
*** eaguilar has joined #openstack-kolla00:26
jascott1portdirect ping00:26
*** TxGirlGeek has quit IRC00:26
*** sdake has joined #openstack-kolla00:26
*** inc0 has joined #openstack-kolla00:31
inc0hey again00:31
inc0so, gates00:31
inc0red00:31
inc0I'm worried, any centos people aroudn?00:31
*** zhubingbing has joined #openstack-kolla00:35
duonghqhi inc000:35
duonghqmorning guys00:35
jascott1good morning00:36
duonghqhi jascott100:37
Pavosup inc0 I'm a centos user00:37
inc0hey duonghq00:38
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds  https://review.openstack.org/41855000:38
inc0Pavo, our gates are broken00:38
inc0centos00:38
inc0https://review.openstack.org/#/c/424784/ look at gate status00:38
Pavoah I have no clue on those, never looked at them but will take a look00:39
inc0yeah gates requires special kind of touch00:39
inc0but let me know if you run into any issues, they seem consistently broken00:39
inc0Jeffrey4l_, https://review.openstack.org/#/c/424784/ <- when you wake up please take a look00:39
jascott1new to this party but this is the error right? Unable to open /dev/sr0 read-write (Read-only file system).00:41
jascott1Error: /dev/sr0: unrecognised disk label00:41
jascott1weird chars in the file? before that error there is this00:42
jascott1cp: cannot stat ‘/var/lib/docker/volumes/kolla_logs/_data/*’: No such file or directory00:42
inc0chars were there for a while now00:43
inc0bit this /dev/sr0 looks suspicious00:43
*** adrian_otto has joined #openstack-kolla00:43
*** tonanhngo has quit IRC00:44
*** tonanhngo has joined #openstack-kolla00:44
*** zhurong has joined #openstack-kolla00:45
kfox1111inc0: you there?00:53
inc0yup00:53
inc0wassup?00:53
kfox1111to your mailing list question,00:53
inc0which one?:)00:53
kfox1111no, I think he wants to run arm based debian containers, on top of an arm based distro.00:53
inc0ah, debian00:53
kfox1111no other distro's really support arm very well.00:54
inc0but we would need debian support, gates and such00:54
kfox1111so no arm-centos for kolla.00:54
kfox1111right.00:54
kfox1111so if he's voluneering,00:54
kfox1111which it sounds like he's doing? he needs to do that stuff.00:54
*** cuongnv has joined #openstack-kolla00:54
inc0well, my point is - you want us to keep it, you take care of it00:55
kfox1111It sounded like hew was saying that too.00:55
inc0and first step would be write gate jobs00:55
inc0yeah, we'll talk about it tomorrow, we can just ask:)00:55
kfox1111oh. you responded to a second thread I think.00:55
kfox1111there was an arm one too at somepoint....00:55
inc0yeah, one on openstack dev and another on operators list00:56
inc0I saw it00:56
kfox1111yeah. sorry. got a bit confused.00:56
kfox1111the arm one makes a lot of sence.00:56
inc0no worries00:56
inc0well, yeah, it is use case00:56
inc0as long as we have someone to take care of it00:56
*** adrian_otto has quit IRC00:57
inc0we can keep it00:57
inc0but really...00:57
kfox1111the arm guy sounded like he might try post ptg.00:57
inc0I'd put requirement - you want debian, start with gates00:57
kfox1111I'd say, lets not kick it out and see if he follows through. if not, we drop it?00:58
kfox1111+100:58
inc0if he volunteers, we shouldn't kick it out just yet, we can always do it00:58
inc0but I want him saying clearly that "yes, I, the arm guy, hereby volunteer to write debian gates"00:58
kfox1111might double check with him to verify his intent was to volunteer.00:58
kfox1111yeah.00:58
kfox1111and spell out explicitly whats needed ahead of time.00:59
kfox1111"this is the work you need to do to keep it." "do you volunteer to do that?"00:59
inc0it's a bit rude to call him arm guy tho, I can't remember his nick here00:59
inc0yup agree01:00
inc0once we have gates, it's going to be whole lot easier, so that's a first step01:00
*** adrian_otto has joined #openstack-kolla01:02
Pavoso.... hows was everyone's day01:02
Pavomine was boring as f***01:03
*** schwicht has joined #openstack-kolla01:03
inc0Pavo, meetings 9am-2pm01:03
inc0mind is fried01:03
Pavoinc0 class over paloalto from 8am-4pm01:03
Pavoboring.............01:03
Pavolol01:03
Pavofirewall is a firewall is a firewall01:04
Pavolol01:04
inc0well, at least you're in sunny California;)01:04
Pavono matter the vendor01:04
Pavohell I wished01:04
Pavoin cold as hell Baltimore, MD01:04
inc0ahh....paloalto not as Palo Alto, CA01:04
PavoPaloalto firewalls lol01:04
inc0yeah, here in OR it's actually pretty nive01:05
Pavoits like 30 here01:05
jascott1https://www.youtube.com/watch?v=h9iwI3dbxqs01:05
inc0I think it's in 40s today here01:05
*** msimonin1 has quit IRC01:05
kfox1111inc0: sorry. no rudeness intended. just didn't have the email handy.01:06
inc0what this guy is doing to that poor guitar!?01:06
Pavolol01:06
kfox1111paloalto firewall.01:06
jascott1and the poor xylophone01:06
inc0kfox1111, let's hope he wont read this and hold it against us;) if he does: sorry :(01:06
kfox1111+1. sorry. no offence intended.01:07
*** eaguilar has quit IRC01:08
*** goldyfruit has quit IRC01:09
*** bpattewar has quit IRC01:10
*** adrian_otto has quit IRC01:14
*** adrian_otto has joined #openstack-kolla01:15
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds  https://review.openstack.org/41855001:16
*** adrian_otto has quit IRC01:16
*** sdake has quit IRC01:22
*** adrian_otto has joined #openstack-kolla01:26
*** inc0 has quit IRC01:26
kfox1111gota get this local repo patch done. :/01:26
*** lamt has joined #openstack-kolla01:31
*** adrian_otto has quit IRC01:32
*** sdake has joined #openstack-kolla01:35
*** sayantani01 has joined #openstack-kolla01:37
*** sayantani01 has quit IRC01:37
*** sayantani01 has joined #openstack-kolla01:37
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Capture lists of containers by type.  https://review.openstack.org/42492101:38
kfox1111and, we're goign to need a kubernetes rpm/deb repo mirror at some point too. :/01:41
*** tovin07_ has joined #openstack-kolla01:43
*** goldyfruit has joined #openstack-kolla01:43
*** schwicht has quit IRC01:45
openstackgerritJeffrey Zhang proposed openstack/kolla: Replace heka with fluentd in all profiles  https://review.openstack.org/42492801:55
*** trey_ has quit IRC02:01
*** jrobinson has quit IRC02:01
*** jrobinson has joined #openstack-kolla02:06
sbezverkkfox1111: could you ack this fix we talked about: https://review.openstack.org/#/c/424902/102:06
*** sdake has quit IRC02:06
openstackgerritzhubingbing proposed openstack/kolla: Change kolla_freezer_sudoers 640 to 440  https://review.openstack.org/42493402:07
jascott1kfox1111 qwang is going to work on local repo patch02:11
jascott1oh sorry you meant registry02:12
jascott1i was talking about helm local path charts02:12
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Capture lists of containers by type.  https://review.openstack.org/42492102:15
*** sacharya has joined #openstack-kolla02:15
*** zhubingbing has quit IRC02:15
kfox1111jascott1: awesome. :)02:15
kfox1111sbezverk: done. thanks.02:16
*** zhubingbing has joined #openstack-kolla02:20
*** schwicht has joined #openstack-kolla02:26
*** haplo37 has quit IRC02:34
*** eaguilar has joined #openstack-kolla02:39
*** haplo37 has joined #openstack-kolla02:42
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Capture lists of containers by type.  https://review.openstack.org/42492102:43
*** zhubingbing has quit IRC02:45
*** sudipto has joined #openstack-kolla02:47
*** sudipto_ has joined #openstack-kolla02:47
*** l4yerffeJ has quit IRC02:48
*** l4yerffeJ has joined #openstack-kolla02:49
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' to neccesary tasks (general roles)  https://review.openstack.org/39868202:53
*** prameswar has joined #openstack-kolla03:00
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' for only neccesary tasks (default roles)  https://review.openstack.org/39868403:06
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' for only neccesary tasks (all other roles)  https://review.openstack.org/39868503:08
openstackgerritsayantani proposed openstack/kolla-ansible: Modify config.yml playbook for bifrost role  https://review.openstack.org/41919603:12
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds  https://review.openstack.org/41855003:13
*** jrobinson_ has joined #openstack-kolla03:14
*** zhubingbing has joined #openstack-kolla03:14
*** jrobinson_ has quit IRC03:16
*** jrobinson has quit IRC03:17
*** jrobinson has joined #openstack-kolla03:17
*** sdake has joined #openstack-kolla03:20
openstackgerritDuong Ha-Quang proposed openstack/kolla: [WIP] Apply neutron database migration  https://review.openstack.org/40784003:22
*** sp___ has joined #openstack-kolla03:22
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Capture lists of containers by type.  https://review.openstack.org/42492103:23
*** zhubingbing has quit IRC03:29
*** hfu has joined #openstack-kolla03:32
sp___mandre: ping...03:34
sp___mandre:   If I am right you are *Martin André*03:34
sp___?03:34
openstackgerritDuong Ha-Quang proposed openstack/kolla: Apply neutron database migration  https://review.openstack.org/40784003:36
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible: Apply neutron database migration  https://review.openstack.org/40792203:36
SamYaplesp___: that is who mandre is, yes03:39
duonghqsp___, last few days you pinged me?03:42
sp___duonghq:  yes that was regarding   review for coverage BP, but that has been done by Jeffrey4l_ , pbourke and mandre.03:44
sp___SamYaple: thanks03:44
duonghqsp___, ack03:44
duonghqis it merged?03:44
*** eaguilar has quit IRC03:44
sp___duonghq: tx for reply03:44
duonghqI busied for awhile03:44
duonghqsorry03:44
sp___duonghq:  no, it is under progress after review03:45
sp___duonghq: np03:45
duonghqnice03:45
sp___duonghq:  is there any chance of extention of release date ?03:46
sp___duonghq:  today is the last date     right ?03:46
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds  https://review.openstack.org/41855003:47
duonghqsp___, for kolla and kolla-ansible: today is the last day03:47
openstackgerritZeyu Zhu proposed openstack/kolla-ansible: Modify the version_info in pbr  https://review.openstack.org/42496503:47
duonghqno chance for deadline extension03:47
sp___duonghq: thanks03:47
duonghqfor o-303:47
duonghqnp03:48
*** portdirect_away is now known as portdirect03:55
*** mdnadeem has joined #openstack-kolla03:59
*** l4yerffeJ has quit IRC04:01
*** l4yerffeJ has joined #openstack-kolla04:01
*** tovin07 has quit IRC04:02
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' to neccesary tasks (general roles)  https://review.openstack.org/39868204:03
sdakesp___ last day for what?04:04
sdakesp___ for which deliverable?04:04
sp___sdake: for kolla04:04
sdakesp___ kolla, kolla-ansible follow the release:trialing-cycle schedule04:05
sdakesp__ kolla-kubernetes follows the release:independent schedule (meaning we release when we want)04:05
sdakesp__ if your developing a bug fix, o3 is not the final deadline04:05
sdakesp__ if your developing a blueprint, o3 is the final deadline for blueprints04:05
sdakeduonghq there actually is a mechanism for extension - its called an FFE04:06
sdakeduonghq for release:trailing-cycle deliverables04:06
sdakefeature-freeze extension04:06
sdakeyou can send a mail to the mailing list and the PTL can decide to approve it or not04:06
sdakeif the community disagrees, then i'm not really sure what the process is as we typically bleed into the rcs04:07
sdakefor features04:07
sdakealthough I think that is bad practice especially with the PTG / summit schedules going  forward04:07
sp___sdake: ok, seem it would be completed today. I am working on coverage BP of kollla some more cases to add.04:07
sp___sdake: thanks for info04:07
sdakesp___ happy to help04:07
sdakeinc0 he did indicate he would do that04:08
sdakeoh no inc004:08
sdakeok - well gotta jet to take kids home bbl04:08
*** sdake has quit IRC04:08
*** jrobinson is now known as jrobinson-afk04:15
*** adrian_otto has joined #openstack-kolla04:16
*** adrian_otto has quit IRC04:16
*** skramaja has joined #openstack-kolla04:17
portdirecto/ morning kolla04:18
portdirectjascott1: delayed pong04:19
portdirectcheers for the head up about plugins04:19
duonghqmorning portdirect04:20
sbezverkportdirect: can you look at these two please: https://review.openstack.org/#/c/424697/ and https://review.openstack.org/#/c/424902/104:23
kfox1111There we go... trunk got past neutron now.04:28
kfox1111now looks like just ceph....04:28
kfox1111well, cinder.04:28
kfox1111last day for o3 kolla stuff?04:29
kfox1111can we please get https://review.openstack.org/#/c/422950/ in then?04:29
*** dave-mccowan has quit IRC04:30
*** lamt has quit IRC04:32
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder control package with kubernetes entrypoint  https://review.openstack.org/42469704:34
kfox1111ugg...04:35
kfox1111that means two more rebases. :/04:35
sbezverkkfox1111: will be done soon with services ;) no more rebases because of that..04:37
openstackgerritMerged openstack/kolla-kubernetes: Fixing location of dependency for vswitchd  https://review.openstack.org/42490204:38
kfox1111yeah. but am trying to get the gate more stable. :/04:38
kfox1111which is hard to do with checks failing and rebasing. :/04:38
kfox1111not your fault. just frustrating.04:38
*** jrobinson-afk is now known as jrobinson04:39
*** sdake_ has joined #openstack-kolla04:40
*** sdake_ has quit IRC04:40
*** sdake_ has joined #openstack-kolla04:40
kfox1111cute...04:43
kfox11112017-01-25 04:12:22.975 7 ERROR oslo_messaging.rpc.server libvirtError: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk1' could not be initialized04:43
SamYaplekfox1111: im pretty sure the static uid patch wasnt tested before it merged, fyi. so i think it might be related to that04:46
*** goldyfruit has quit IRC04:47
kfox1111SamYaple: oh. for 4.x containers?04:47
kfox1111interesting.04:47
SamYaplekfox1111: i tested a version of it on ubuntu. and i got it to pass the gate, but im 95% sure no one tested centos family beyond "it passed the gate"04:48
SamYaplebut i ran out of time to maintain the patch and abandonded it. someone else finished it04:48
kfox1111ah. ok.04:49
kfox1111sbezverk: yeah, I think I've seen several cases of gate failure due to openvswitch-agent coming up before vswitchd now.05:02
kfox1111http://logs.openstack.org/50/418550/55/check/gate-kolla-kubernetes-deploy-centos-binary-3-ceph-multi-nv/caf246b/console.html05:02
kfox1111SamYaple: libvirt logs here: http://logs.openstack.org/50/418550/55/check/gate-kolla-kubernetes-deploy-centos-binary-t-ceph-multi-nv/245adef/logs/openstack/kolla-nova-libvirt-p9376.txt they are rather large, so b ecareful.05:09
SamYaplekfox1111: oh thats ceph related05:10
SamYaplekfox1111: ceph probably isnt operational or it crashed05:11
kfox1111ceph's happy.05:11
kfox1111the vm was launched via glance.05:12
kfox1111the full ceph logs are here: http://logs.openstack.org/50/418550/55/check/gate-kolla-kubernetes-deploy-centos-binary-t-ceph-multi-nv/245adef/logs/ceph.log05:12
kfox1111I can't find anything in particular wrong in the logs. :/05:12
SamYapledontknow.jpg05:13
SamYaplebut im pretty sure thats ceph related05:13
SamYaplepossibly permissions related to ceph05:13
kfox1111last time I saw something like that, it was a bad rpm installed in the container. :/05:13
SamYaplesounds right05:13
SamYaplekraken just related btw05:14
SamYaplehopefully you arent accidentally pulling that in05:14
*** lamt has joined #openstack-kolla05:14
kfox1111SamYaple: cool.05:15
kfox1111looks like its pinned in trunk to jewel.05:15
SamYapleyea. but _still_05:16
SamYapledouble check :P05:16
kfox1111not easy... the tarball its pulling from is in docker repo format. :/05:16
kfox1111oh.. yeah. the ones portdirect is posting should be similar.05:19
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Optimize reconfigure action for nova  https://review.openstack.org/40741605:20
kfox1111gotta head out. nite all.05:23
kfox1111it is jewel btw.05:24
kfox1111just pulled and verified.05:24
*** huzhengchuan has quit IRC05:36
*** tonanhngo has quit IRC05:36
*** tonanhngo has joined #openstack-kolla05:37
*** tonanhngo has quit IRC05:41
openstackgerritSurya Prakash Singh proposed openstack/kolla: Code coverage improvement for kolla  https://review.openstack.org/42244405:44
openstackgerritMerged openstack/kolla-kubernetes: Add cinder control package with kubernetes entrypoint  https://review.openstack.org/42469705:46
*** sudswas__ has joined #openstack-kolla05:47
*** sudipto has quit IRC05:50
*** sudipto_ has quit IRC05:50
*** sudipto has joined #openstack-kolla05:50
*** sacharya has quit IRC05:51
jascott1portdirect top of the morning to ye06:00
portdirectword jascott1, hows it going dude?06:04
jascott1all good. will be seeing you in ATL krom willing06:04
jascott1portdirect how are things on the east side (of the atlantic)?06:06
*** zhubingbing has joined #openstack-kolla06:07
portdirectdamp06:08
*** skramaja has quit IRC06:09
jascott1damp here as well :/06:10
jascott1then again it is portland http://isitraininginportland.com/06:10
sdake_portdirect any chance you got the dev env to go06:10
jascott1hey sdake06:11
sdake_portdirect i got to the point of iscsi complaints06:11
sdake_sup jascott106:11
*** slagle has quit IRC06:15
*** prameswar has quit IRC06:15
*** jmontleon has quit IRC06:16
*** slagle has joined #openstack-kolla06:17
*** jmontleon has joined #openstack-kolla06:19
*** bjolo has quit IRC06:27
*** bmace_ has quit IRC06:30
*** zhubingbing has quit IRC06:34
*** Jeffrey4l_ has quit IRC06:48
*** zhubingbing has joined #openstack-kolla06:50
*** Jeffrey4l has joined #openstack-kolla06:53
*** trinaths has joined #openstack-kolla06:53
*** jrobinson has quit IRC06:54
*** tovin07 has joined #openstack-kolla06:54
openstackgerritAnh Tran proposed openstack/kolla-kubernetes: Typo fix: folowing => following  https://review.openstack.org/42500806:59
*** sdake_ has quit IRC07:01
*** zhubingbing has quit IRC07:05
*** bmace_ has joined #openstack-kolla07:07
*** zhubingbing has joined #openstack-kolla07:08
*** sudipto_ has joined #openstack-kolla07:13
*** sudipto has quit IRC07:14
*** sudswas__ has quit IRC07:14
*** sudipto has joined #openstack-kolla07:14
*** msimonin has joined #openstack-kolla07:16
*** matrohon has joined #openstack-kolla07:16
*** msimonin has quit IRC07:17
*** msimonin has joined #openstack-kolla07:17
*** msimonin has quit IRC07:18
*** msimonin has joined #openstack-kolla07:18
*** msimonin has quit IRC07:18
*** msimonin has joined #openstack-kolla07:19
*** msimonin has quit IRC07:19
*** msimonin has joined #openstack-kolla07:19
*** msimonin has quit IRC07:20
*** matrohon has quit IRC07:23
*** pcaruana has joined #openstack-kolla07:28
*** hfu has quit IRC07:29
*** tovin07 has quit IRC07:32
*** britthouser7 has joined #openstack-kolla07:34
*** zhubingbing has quit IRC07:35
*** klindgren_ has joined #openstack-kolla07:35
*** sayantan_ has joined #openstack-kolla07:36
*** klindgren has quit IRC07:38
*** sayantani01 has quit IRC07:38
*** britthouser has quit IRC07:38
*** mgoddard has joined #openstack-kolla07:39
*** prameswar has joined #openstack-kolla07:45
*** tovin07 has joined #openstack-kolla07:45
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for elasticsearch  https://review.openstack.org/42413907:48
openstackgerritSurya Prakash Singh proposed openstack/kolla: Docs updated to get coverage report  https://review.openstack.org/42503207:52
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Optimize reconfigure action for horizon  https://review.openstack.org/42271907:57
*** ingmar23 has joined #openstack-kolla07:59
*** bjolo has joined #openstack-kolla08:00
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for solum  https://review.openstack.org/42416708:04
*** saneax-_-|AFK is now known as saneax08:06
*** tovin07 has quit IRC08:13
*** shardy has joined #openstack-kolla08:19
*** huzhengchuan has joined #openstack-kolla08:23
*** tovin07 has joined #openstack-kolla08:29
*** Jeffrey4l_ has joined #openstack-kolla08:30
*** l4yerffeJ_ has joined #openstack-kolla08:30
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Run detect openstack_release variable play always  https://review.openstack.org/42505008:33
*** l4yerffeJ has quit IRC08:34
*** egonzalez has joined #openstack-kolla08:34
*** Jeffrey4l has quit IRC08:34
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for manila  https://review.openstack.org/42136508:35
*** msimonin has joined #openstack-kolla08:37
ingmar23exit08:37
*** ingmar23 has quit IRC08:37
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Custom init-runonce values  https://review.openstack.org/42468208:37
*** zhubingbing has joined #openstack-kolla08:37
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for aodh  https://review.openstack.org/42367008:41
*** hfu has joined #openstack-kolla08:42
openstackgerritPete Birley proposed openstack/kolla-kubernetes: Update docs to pin k8s version to 1.5.1 and remove helm pull workaround  https://review.openstack.org/42505908:44
openstackgerritPete Birley proposed openstack/kolla-kubernetes: Update docs to pin k8s version to 1.5.1 and remove helm pull workaround  https://review.openstack.org/42505908:47
openstackgerritPete Birley proposed openstack/kolla-kubernetes: Update docs to pin K8s at v1.5.1 and remove helm pull workaround  https://review.openstack.org/42505908:48
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Integrate designate with neutron  https://review.openstack.org/42467908:52
egonzalezportdirect: ping08:52
trinathshi, while setting up the kolla-kubernetes environment, the was struck at Ansible tasks  TASK [kube-join : join workers to the initialized kubernetes cluster] **********08:56
trinathsfrom a hour its struck there. Any clue to debug it ?08:56
sp_egonzalez: if you have sometime in this crunchy situation, would be great if you review these https://review.openstack.org/#/c/425000/ https://review.openstack.org/#/c/425032/ https://review.openstack.org/#/c/422444/08:58
sp_egonzalez: I think won't take much time. Please review08:59
*** hfu has quit IRC08:59
*** tovin07 has quit IRC09:00
openstackgerritsayantani proposed openstack/kolla: [WIP] Update Multinode deployment of kolla guide  https://review.openstack.org/40836409:01
egonzalezsp_: added a comment https://review.openstack.org/#/c/425000/209:02
sp_egonzalez: done09:04
sp_was a typo09:04
sp_thanks09:04
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for kuryr  https://review.openstack.org/42416109:05
*** sayantan_ has quit IRC09:07
*** newmember has joined #openstack-kolla09:08
*** hfu has joined #openstack-kolla09:08
*** Serlex has joined #openstack-kolla09:10
portdirectegonzalez: pong09:10
egonzalezis this bp ready to mark as implemented? https://blueprints.launchpad.net/kolla/+spec/add-kubernetes-images09:11
egonzalezportdirect: ^^09:11
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for octavia  https://review.openstack.org/42384609:11
portdirectegonzalez: yeah the kolla side of that's done09:12
egonzalezportdirect: thanks09:12
portdirectegonzalez: so its ok to mark as complete - cheers09:12
portdirecttrinaths: ping09:12
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for octavia  https://review.openstack.org/42384609:14
*** zhurong has quit IRC09:14
*** tovin07 has joined #openstack-kolla09:15
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Integrate designate with neutron  https://review.openstack.org/42467909:18
n0isyn0isehi kolla-team, i'm using kolla from the stable/newton branch, I am able to build images with kolla build but I am now trying to use the --build-args option with http_proxy and no_proxy but the no_proxy seems to not be handled, or i use it  badly - Maybe someone has an advice about its use ?09:18
*** portdirect is now known as portdirect_brb09:22
*** Mech422 has quit IRC09:23
*** Mech422 has joined #openstack-kolla09:24
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Custom init-runonce values  https://review.openstack.org/42468209:24
*** hfu_ has joined #openstack-kolla09:26
*** hfu has quit IRC09:29
trinathsportdirect_brb: pong09:30
trinathsportdirect_brb: i was struck at task, TASK [kube-join : join workers to the initialized kubernetes cluster]09:31
trinathsportdirect_brb: log here, http://paste.openstack.org/show/596394/09:32
*** portdirect_brb is now known as portdirect09:32
portdirectyeah i hit the same thing this morning09:32
portdirectgot a ps in to fix the docs and resovle the tiller issue with pulling images from gcr.io09:33
portdirectcan we destory that cluster?09:33
portdirectand then run the ./setup-halcyon.sh script again09:33
portdirectbut this time specify 'v1.5.1' as the k8s version09:34
portdirectcan then we can make a change to the ansible to man you dont need the image pull workaround09:34
portdirectin kube-deploy/roles/deploy-kube/tasks/centos.yml09:35
portdirectcan you make the changes from here:09:35
portdirecthttps://github.com/att-comdev/halcyon-kubernetes/pull/43/files09:35
*** mdnadeem has quit IRC09:37
*** mdnadeem has joined #openstack-kolla09:39
*** tovin07 has quit IRC09:41
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Optimize reconfiguration for tacker  https://review.openstack.org/42203609:41
*** sp___ has quit IRC09:42
trinathsportdirect: doing.09:45
portdirectif you can i'd also make another change09:45
portdirectin the config.rb file09:45
portdirectI'd strongly recommend upping the ram and vcpu given to each node09:46
trinathsportdirect: okay. please tell me. just destroyed the cluster09:46
trinathsoh i have workstation with 16GB RAM09:46
*** msimonin has quit IRC09:46
portdirectif you can up it to 3072 per node that would be great09:47
portdirectbut obviously the hardware you have available limits the scope there09:47
portdirectI'm running with 6GB ram and 4vcpu per node - but then have a bit more to play with on my workstation09:48
*** strigazi_AFK is now known as strigazi09:50
trinathsportdirect: ok09:51
*** athomas has joined #openstack-kolla09:53
*** tovin07_ has quit IRC09:55
*** mgoddard has quit IRC10:03
*** cuongnv has quit IRC10:03
*** duonghq has quit IRC10:05
Serlexthanks for the info portdirect. Have you come across " TASK [kube-init : check for bootstrap]" "The connection to the server localhost:8080 was refused"10:06
*** msimonin has joined #openstack-kolla10:12
*** sambetts|afk is now known as sambetts10:13
*** simon-AS559 has joined #openstack-kolla10:15
*** hfu_ has quit IRC10:16
*** openstackgerrit has quit IRC10:17
*** skramaja has joined #openstack-kolla10:18
*** mgoddard has joined #openstack-kolla10:19
*** egonzalez has quit IRC10:20
*** mdnadeem has quit IRC10:21
portdirectSerlex: :(10:23
portdirecti think i know what may be up there10:23
SerlexI missed a setup or a proxy issue10:24
portdirectcan you run the ./setup-halcyon.sh specifying 'v1.5.1' as the k8s version to use10:24
portdirectif you have a proxy then that could well be a cause10:24
Serlexnot setup...step*10:24
Serlexok sure, I will check10:24
portdirectand unfortunately my knowledge of that are is very small :(10:24
portdirect*area10:24
*** portdirect is now known as portdirect_away10:27
*** mdnadeem has joined #openstack-kolla10:29
*** pbourke has quit IRC10:29
*** pbourke has joined #openstack-kolla10:30
*** msimonin has quit IRC10:36
*** gfidente has joined #openstack-kolla10:40
*** gfidente has quit IRC10:40
*** gfidente has joined #openstack-kolla10:40
*** nrado has joined #openstack-kolla10:49
*** newmember has quit IRC11:00
*** egonzalez has joined #openstack-kolla11:02
*** nrado has quit IRC11:05
*** nrado has joined #openstack-kolla11:05
egonzalezzhubingbing: ping11:07
*** nrado has quit IRC11:08
*** nrado has joined #openstack-kolla11:08
*** hfu has joined #openstack-kolla11:15
*** papacz has joined #openstack-kolla11:16
*** openstackgerrit has joined #openstack-kolla11:20
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for solum  https://review.openstack.org/42416711:20
openstackgerritMerged openstack/kolla-ansible: Optimize reconfiguration for mistral  https://review.openstack.org/42378711:25
*** eaguilar has joined #openstack-kolla11:28
mliimamorning guys11:28
egonzalezmorning mliima , how things going?11:29
mliimafine egonzalez :)11:30
mliimaand you?11:30
openstackgerritMerged openstack/kolla-ansible: Optimize reconfiguration for watcher  https://review.openstack.org/42410511:38
openstackgerritMerged openstack/kolla-ansible: Optimize reconfiguration for sahara  https://review.openstack.org/42416511:38
*** schwicht has quit IRC11:39
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla: Reduce size of images built from git repos  https://review.openstack.org/42125211:43
*** mgoddard has quit IRC11:45
openstackgerritMauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for grafana  https://review.openstack.org/42415811:48
openstackgerritMauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for grafana  https://review.openstack.org/42415811:51
openstackgerritserlex proposed openstack/kolla-kubernetes: Added option to use --provide with vagrant  https://review.openstack.org/42514811:57
openstackgerritMauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for kuryr  https://review.openstack.org/42416111:59
openstackgerritNenad Radojevic proposed openstack/kolla-ansible: Use neutron user in neutron_lbaas.conf and overwrite defaults  https://review.openstack.org/42515011:59
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for kuryr  https://review.openstack.org/42416112:00
*** hfu has quit IRC12:00
*** hfu has joined #openstack-kolla12:00
*** hfu has quit IRC12:01
*** hfu has joined #openstack-kolla12:01
*** hfu has quit IRC12:01
openstackgerritMauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for collectd  https://review.openstack.org/42413312:02
*** portdirect_away is now known as portdirect12:02
*** hfu has joined #openstack-kolla12:02
*** hfu has quit IRC12:02
*** hfu has joined #openstack-kolla12:03
*** tonanhngo has joined #openstack-kolla12:03
*** hfu has quit IRC12:03
*** hfu has joined #openstack-kolla12:03
*** hfu has quit IRC12:04
openstackgerritMauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for influxdb  https://review.openstack.org/42415912:04
*** tonanhngo has quit IRC12:08
*** haplo37_ has quit IRC12:08
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for kuryr  https://review.openstack.org/42416112:09
*** haplo37_ has joined #openstack-kolla12:10
openstackgerritMauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for designate  https://review.openstack.org/42415512:12
*** trinaths has left #openstack-kolla12:14
portdirectSerlex: regarding  https://review.openstack.org/425148, did you need to use this option or is it just for info?12:17
SerlexHi, I had to use it and also a good info?12:19
openstackgerritMauricio Lima proposed openstack/kolla-ansible: Optimize reconfigure action for cinder  https://review.openstack.org/42238412:19
Serlexis that ok?12:19
*** shardy is now known as shardy_lunch12:20
portdirectyeah thats great12:20
portdirectgood to know - I've never needed to specify it manually myself12:20
Serlexphew...firrst commit :D12:21
Serlexprobably last12:21
portdirectlol - docs are the most important thing as well!12:21
portdirectdont say that :(12:21
SerlexI was been critical of myself :D12:22
portdirectalways better to undersell and over-deliver  :)12:24
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for karbor  https://review.openstack.org/42416012:24
egonzalezSerlex: I would use a lower case in the You ;)12:24
wznoinskhi all12:27
egonzalezwznoinsk: sup12:27
*** mgiles has joined #openstack-kolla12:27
wznoinskhi egonzalez , I'm wondering whether the problem I have a trivial one or not... I have nova_compute container that qemu-nbd mounts /dev/nbdS under /var/lib/nova/instances/UUID/rootfs, /var/lib/nova is a docker volume from the host but because nbd is mounted on top of it in nova_compute container the nova_libvirt container doesn't see it, anyone seen/fix that?12:29
*** eaguilar has quit IRC12:29
wznoinska minor fact: i'm using lxc virt_type of libvirt, but I guess the same problem would happen with any other virt_type if you're using raw type of image (lxc converts qcow2 -> raw before the above mounting as it needs that format)12:32
Jeffrey4l_egonzalez, could u review this https://review.openstack.org/424928  which unblock kolla's gate.12:33
*** rmart04 has joined #openstack-kolla12:33
egonzalezJeffrey4l_: done12:34
Jeffrey4l_thanks.12:35
Jeffrey4l_egonzalez, another big one https://review.openstack.org/416046 ;)12:35
*** eaguilar has joined #openstack-kolla12:36
egonzalezwznoinsk: never saw that befero, never tried lxc as virt_type, maybe need to install some dependency and other configuration. Can you try with qemu/kvm?12:36
Serlexegonzalez: it was suppose to be a new line12:37
SerlexBut I will change it now12:37
egonzalezJeffrey4l_: doing, gerrit is running so slow today :/12:38
*** sudipto has quit IRC12:39
*** sudipto_ has quit IRC12:39
Jeffrey4l_egonzalez, recommend to use gertty https://github.com/openstack/gertty12:39
openstackgerritMark Giles proposed openstack/kolla-kubernetes: Nova-control service deployment  https://review.openstack.org/42145712:39
*** sdake has joined #openstack-kolla12:45
sdakemorning peeps12:45
-openstackstatus- NOTICE: Gerrit is going to be restarted due to slow performance12:46
wznoinskegonzalez, qemu/kvm works fine as it's ok with using qcow2 file which is "dropped" to /var/lib/nova/instances/UUID , qemu-system-x86_64 has this as device -drive file=/var/lib/nova/instances/62bbb295-2df9-4f28-819e-b99df661384f/disk,format=qcow212:47
wznoinskegonzalez, lxc needs a filesystem 'unpacked' from the image, it does it via binding an image to /dev/nbdX device and mount that devices as  /var/lib/nova/instances/UUID/rootfs in nova_compute container (not seen by nova_libvirt) - hope the explanation is clear enough12:48
egonzalezwznoinsk: IIRC lxc need a nova-compute-lxc package installed12:49
-openstackstatus- NOTICE: Gerrit has been successfully restarted12:49
wznoinskegonzalez, it has, from nova point of view everything works fine12:50
egonzalezwznoinsk: as per your explanation, maybe need to add some mount points to the container12:50
openstackgerritserlex proposed openstack/kolla-kubernetes: Added option to use --provide with vagrant.  https://review.openstack.org/42514812:50
wznoinskegonzalez, yes, something around that, there's one mount per lxc instance spawned so not sure how to achieve that so the mount is seen by both containers12:51
wznoinskegonzalez, in short: I'm mounting something in containerA and I wanted containerB to see that as well12:52
wznoinskthis issue happens only when using Kolla, normally nova_compute/nova_libvirt have access to the same directory and mountpoint on the host hence not being a problem12:53
*** sp___ has joined #openstack-kolla12:53
*** nrado has quit IRC12:55
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for karbor  https://review.openstack.org/42416012:56
egonzalezwznoinsk: only this mount points are bind in libvirt and not in nova /sys/fs/cgroup /etc/libvirt/qemu12:59
wznoinskI see the same /var/lib/nova/instances in both containers, that's correct13:00
wznoinskqemu-nbd mounts lxc instance rootfs over a subdirectory of /var/lib/nova/instances in nova_compute container tho13:00
egonzalezwznoinsk: what changes you made in docker images to install lxc driver?13:02
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for karbor  https://review.openstack.org/42416013:04
*** hfu has joined #openstack-kolla13:05
sdakeegonzalez how are ou liking ibm these dyas13:07
*** eaguilar has quit IRC13:10
wznoinskegonzalez, very few, two additional mount points at the container start task and install of nova-compute-lxc13:10
*** hfu has quit IRC13:10
wznoinskhi sdake13:10
sp___hi... sdake13:11
*** zioproto has joined #openstack-kolla13:13
*** eaguilar has joined #openstack-kolla13:15
egonzalezsdake: all fine, today I had to give a presentation talk and evengelized kolla around IBMers hehe13:16
sdakeegonzalez nice!13:17
sdakesup wznoinsk sp___13:17
sdakeI have a 2 hour seminar on all things kolla next months13:17
sdakenext month in february13:17
sdake2 hours - goran - that will take some time13:17
sdakeegonzalez do you hae any material you prepped you can share that i can crib from13:17
sp___sdake:   there are some review items here13:18
sp___sdake: if you got some free time please review these13:18
sp___https://review.openstack.org/#/c/425000/     https://review.openstack.org/#/c/425032/    https://review.openstack.org/#/c/422444/1613:19
sdakesp___ actually now is the school hour13:19
sdakesp___ from around 6-9am every day13:19
mgilesAre the gates completely broken this morning?13:19
sdakesp___ (wife out of town so have no coverage)13:19
mgilesthis doesn't look normal: http://status.openstack.org/zuul/13:19
sdakemgiles - i will check that - which deliverable13:19
mgilesall of them13:19
sdakemgiles gerrit was restarted earlier13:19
sdakemgiles try a recheck13:19
sp___sdake: thank, np13:19
mgilesit's not that they're failing, they just aren't running13:20
mgilesI don't know if a recheck will fix that?  I can try13:20
sdakemgiles - a recheck should fix it13:21
sdakemgiles if gerrit restarts, it causes all kinds of chaos with the gates13:21
mgilesok.  thanks sdake13:21
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for karbor  https://review.openstack.org/42416013:25
mgilessdake  Still not seeing tests running13:25
*** shardy_lunch is now known as shardy13:26
*** v1k0d3n has quit IRC13:27
sdakemgiles openstack-infra is the place to go :)13:28
sdakemgiles join the channel and i can get hte conversation started13:28
openstackgerritMerged openstack/kolla: Replace heka with fluentd in all profiles  https://review.openstack.org/42492813:29
*** v1k0d3n has joined #openstack-kolla13:29
sdakemgiles I mean #openstack-infra on irc13:29
sdakemgiles note the gate jobs take a few minutes to actually show up in zuul13:30
mgilesyeah, just seeing a lot of stuff queued form a long time ago, but maybe it's all related to the gerrit reboot13:30
openstackgerritMauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for grafana  https://review.openstack.org/42415813:35
sdakemgiles nice you got he conversation started on your own :)13:35
*** williamcaban has joined #openstack-kolla13:37
*** lamt has quit IRC13:37
*** lamt has joined #openstack-kolla13:38
*** rhallisey has joined #openstack-kolla13:38
*** lamt has quit IRC13:42
openstackgerritMauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for influxdb  https://review.openstack.org/42415913:45
*** zhubingbing has quit IRC13:49
*** zhubingbing has joined #openstack-kolla13:52
*** mdnadeem has quit IRC13:53
*** ipsecguy has quit IRC13:53
*** mdnadeem has joined #openstack-kolla13:55
*** sp___ has quit IRC13:55
*** goldyfruit has joined #openstack-kolla13:59
*** ipsecguy has joined #openstack-kolla14:01
openstackgerritMerged openstack/kolla-ansible: Optimize reconfiguration for barbican  https://review.openstack.org/42372814:02
openstackgerritSaverio Proto proposed openstack/kolla: Use become: True when updating /etc/hosts  https://review.openstack.org/42519114:02
*** dmsimard|off is now known as dmsimard14:02
zioprotoBugfix for stable/newton are still accepted I hope ?14:02
egonzalezzioproto: yup, but need to be a backport from a patch from master14:05
zioprotoegonzalez: it is a cherry pick14:05
*** tonanhngo has joined #openstack-kolla14:07
egonzalezhrm, become line is in different order than master PS, order in this case does not matter but is not cherry-picked14:09
*** matrohon has joined #openstack-kolla14:15
*** dave-mccowan has joined #openstack-kolla14:16
*** jpstnh has joined #openstack-kolla14:16
*** skramaja has quit IRC14:19
*** hfu has joined #openstack-kolla14:20
*** zhubingbing has quit IRC14:21
*** zhubingbing has joined #openstack-kolla14:23
*** williamcaban has quit IRC14:23
*** jtriley has joined #openstack-kolla14:24
*** duonghq has joined #openstack-kolla14:25
*** huzhengchuan has quit IRC14:25
zioprotoegonzalez: cherry pick had a conflict that I had to resolve14:26
zioprotoI have a beginner question. I face yet another Become: true problem14:27
zioprotodo docker pull do I always need sudo ?14:27
zioprotoThis is not working for me: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/common/tasks/pull.yml#L214:28
zioprotowithout become True14:28
zioprotoI run the command14:29
zioprotokolla-ansible -i multinode pull14:29
zioprotoand I fail because of permission error14:29
*** japestinho_ has joined #openstack-kolla14:29
*** egonzalez has quit IRC14:30
zioprotoso anywhere I have a pull14:30
zioprotoI mean a docker pull14:31
zioprotoI have a permission error14:31
zioprotobut I installed docker with Kolla as well14:31
zioprotousing the bootstrap-servers kolla tool14:31
*** jpstnh has quit IRC14:33
*** fguillot has joined #openstack-kolla14:33
*** williamcaban has joined #openstack-kolla14:38
*** huzhengchuan has joined #openstack-kolla14:40
zioprotoanyone ?14:42
zioprotoshould I fill a bug ?14:42
rhalliseywhat's the error?14:42
rhalliseydocker giving you a perm error?14:43
*** williamcaban has quit IRC14:43
zioprotohttp://paste.openstack.org/show/596459/14:43
zioprotoIt is always a permission error14:43
zioprotoI can Become: true14:43
zioprotoand go ahead14:43
zioprotobut14:43
zioprotoI would like to understand if I got something wrong with the permissions14:43
zioprotoI am just following the Newton documentation14:44
zioprotois 'docker pull' a command that generally needs sudo rights ?14:44
rhalliseyya you generally need sudo14:45
rhalliseytest it by hand14:45
zioprotoof course14:45
zioprotothen I need to patch here14:45
zioprotohttps://github.com/openstack/kolla-ansible/blob/master/ansible/roles/common/tasks/pull.yml#L214:45
zioprotoall these pull actions14:45
zioprotothey should require Become: true14:46
zioprotoam I wrong ?14:46
zioprotoprobably who wrote this code was using only the root user ? is this possible ?14:46
*** TxGirlGeek has joined #openstack-kolla14:47
rhalliseyya I think you'll need a become: True14:47
rhalliseykolla points to docker docs that ask you to setup docker this way14:48
rhalliseyso if you don't do that, this task will fail without root14:48
openstackgerritSaverio Proto proposed openstack/kolla-ansible: Use become: True when pulling docker images  https://review.openstack.org/42521514:49
rhalliseyzioproto, I think you'll need that for all images14:50
rhalliseyI would also file a bug14:50
*** matrohon_ has joined #openstack-kolla14:50
sdakehey folks, I have a meeting conflict during our team meeting today - so won't be able to make it14:52
sdakeI'll be with you in spirit :)14:52
sdakesomeone woot for me plz :)14:52
sdake(and I'll read the logs today)14:52
*** matrohon has quit IRC14:53
zioprotorhallisey: https://bugs.launchpad.net/kolla-ansible/+bug/165931814:54
openstackLaunchpad bug 1659318 in kolla-ansible "Docker pull needs become: True" [Undecided,New]14:54
*** matrohon__ has joined #openstack-kolla14:55
openstackgerritSteven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm  https://review.openstack.org/42521914:55
sdakegroan14:56
*** matrohon_ has quit IRC14:57
openstackgerritSteven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm  https://review.openstack.org/40572014:57
sean-k-mooneyrhallisey: do you allways want to assume that we have not added the current use to the docker group though by adding become=true14:58
sdakemgiles bmace_ ^^ that last rev is where I get to with the iscsi error14:58
*** TxGirlGeek has quit IRC14:58
sdakesean-k-mooney i want to speak with you about how you setup outlook when I get a chance, but that wont be for  a few hours14:58
sdakesean-k-mooney will you be around then today - or will it be beer-o-clock for ya14:58
sean-k-mooneysdake: sure if i can help i will14:58
rhalliseysean-k-mooney, I mentioned that the kolla links to docs that tell you to do so14:58
*** TxGirlGeek has joined #openstack-kolla14:58
sdakesean-k-mooney there is some config option you are using that seems to allow inline responses14:59
sdakemy outlook 2016 doesn't break the blue bar in html posts14:59
rhalliseysean-k-mooney, I wasn't sure if they are still there14:59
sean-k-mooneyrhallisey: yes im pretty sure i implmented it in he kolla-host playbook too14:59
sdakesean-k-mooney I think perhaps you changed your outlook to post in non-html14:59
sdakesean-k-mooney I can't find a proper workaround via google14:59
sdakesean-k-mooney and microsoft has known about the problem for going on 2 years and not fixed it14:59
sean-k-mooneysdake: yes on my laptop not on my desktop14:59
rhalliseyzioproto, ^14:59
sean-k-mooneymailly becasue i was lazy15:00
*** williamcaban has joined #openstack-kolla15:02
sean-k-mooneysdake: ill take a few screen shots of my settings but yes i set it to add the > and also add a [Mooney,Sean K] tag automatically15:03
sean-k-mooneyi have outlook set to default to html for new mails but for replives it uses the same format as the message being replied too so rich/plain text for the mailing list15:04
*** sayantani01 has joined #openstack-kolla15:07
*** TxGirlGeek has quit IRC15:08
*** tonanhngo has quit IRC15:08
*** TxGirlGeek has joined #openstack-kolla15:09
*** tonanhngo has joined #openstack-kolla15:09
*** hfu has quit IRC15:11
*** williamcaban has quit IRC15:11
*** lrensing has joined #openstack-kolla15:11
*** TxGirlGeek has quit IRC15:13
*** tonanhngo has quit IRC15:13
openstackgerritSaverio Proto proposed openstack/kolla-ansible: Use become: True when pulling docker images  https://review.openstack.org/42521515:15
*** saneax is now known as saneax-_-|AFK15:15
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Refactor register.yml files  https://review.openstack.org/41604615:17
openstackgerritSaverio Proto proposed openstack/kolla-ansible: Use become: True when pulling docker images  https://review.openstack.org/42521515:17
*** zioproto has left #openstack-kolla15:17
*** zioproto has joined #openstack-kolla15:17
zioprotoSo I should add the ubuntu user to the docker group ?15:19
zioprotothis task should be done in the bootstrap-servers playbook ? we should add the ubuntu user to the docker group ?15:19
*** lamt has joined #openstack-kolla15:19
*** Mech422 has quit IRC15:21
*** zhubingbing has quit IRC15:25
*** williamcaban has joined #openstack-kolla15:27
*** zhubingbing has joined #openstack-kolla15:27
*** TxGirlGeek has joined #openstack-kolla15:28
openstackgerritMerged openstack/kolla-ansible: Optimize reconfiguration for ceilometer  https://review.openstack.org/42360615:30
openstackgerritMerged openstack/kolla-ansible: Optimize reconfigure action for gnocchi  https://review.openstack.org/42313315:30
*** srwilkers has joined #openstack-kolla15:31
*** srwilkers has quit IRC15:31
*** srwilkers has joined #openstack-kolla15:32
*** SridharP has joined #openstack-kolla15:32
*** williamcaban has quit IRC15:34
berendtI will join the team meeting a few minutes later. Hanging around on the highway...15:35
*** japestinho__ has joined #openstack-kolla15:35
zioprotothe meeting is on this channel ?15:36
*** rmart04 has quit IRC15:38
*** japestinho_ has quit IRC15:38
*** TxGirlGeek has quit IRC15:41
zioprotook I figured out that using the 'kolla' user you dont need sudo to make docker operations15:42
zioprotobut who is actually using this kolla user ?15:42
*** egonzalez has joined #openstack-kolla15:44
zioprotook I see now that the kolla user ssh keys are generated and put in /etc/kolla15:47
*** bjolo has quit IRC15:50
*** SridharP1 has joined #openstack-kolla15:50
zioprotohow can I use kolla_ssh_key when I run the kolla commands ?15:50
*** inc0 has joined #openstack-kolla15:51
*** SridharP has quit IRC15:52
inc0good morning15:52
SridharP1Hi15:52
*** SridharP1 has left #openstack-kolla15:52
*** SridharP1 has joined #openstack-kolla15:53
srwilkersmorning inc015:53
egonzalezJeffrey4l_: i'm testing gertty, have you ever saw this error "mapping values are not allowed here"15:53
*** sdake_ has joined #openstack-kolla15:53
Jeffrey4l_egonzalez, no..15:53
inc0Jeffrey4l_, did you get my message about gates?15:53
Jeffrey4l_inc0, yep. it is fixed by https://review.openstack.org/42492815:54
*** msimonin has joined #openstack-kolla15:55
*** sdake has quit IRC15:56
zhubingbinghey guys15:57
inc0meeting in 315:58
*** eaguilar has quit IRC15:58
zhubingbingkolla-meeting-3?15:58
inc0noo...openstack-meeting-4;) in 3 as in 3 minutes15:59
inc01 minute now15:59
zhubingbingok15:59
zhubingbing;)15:59
*** srwilkers has quit IRC16:05
*** srwilkers has joined #openstack-kolla16:05
openstackgerritSaverio Proto proposed openstack/kolla: Use become: True when updating /etc/hosts  https://review.openstack.org/42519116:06
*** mdnadeem has quit IRC16:13
*** RichWellum has joined #openstack-kolla16:13
openstackgerritSaverio Proto proposed openstack/kolla-ansible: use become: True to set sysctl values  https://review.openstack.org/42526316:17
* portdirect stands in the silence, and reflects on the serenity of a empty #kolla, birds start to sing in the distance, and a cool breeze blows.16:22
*** sacharya has joined #openstack-kolla16:24
*** sacharya has quit IRC16:24
*** TxGirlGeek has joined #openstack-kolla16:25
openstackgerritFarid Da Encarnacao proposed openstack/kolla-ansible: Fix typo heat.conf  https://review.openstack.org/42527016:26
*** williamcaban has joined #openstack-kolla16:27
*** RichWellum has quit IRC16:28
*** williamcaban has quit IRC16:33
*** SridharP has joined #openstack-kolla16:33
*** prameswar has quit IRC16:34
kfox1111morning.16:34
*** SridharP2 has joined #openstack-kolla16:35
*** SridharP1 has quit IRC16:35
kfox1111and.. rebase time.. :/16:35
*** SridharP has quit IRC16:38
* portdirect the silence is broken, a rabbit runs into a bush, clouds appear. somewhere a tractor is working the feilds16:40
*** williamcaban has joined #openstack-kolla16:43
*** nrado has joined #openstack-kolla16:44
*** zioproto has quit IRC16:45
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds  https://review.openstack.org/41855016:46
*** mdorman has joined #openstack-kolla16:49
mdormandoes the meeting start in 10 minutes, or did it start 50 minutes ago?16:50
*** retr0h has joined #openstack-kolla16:50
openstackgerritMerged openstack/kolla-ansible: Fix cleanup-containers script  https://review.openstack.org/42077516:53
egonzalezmdorman: 50 ago16:53
openstackgerritMerged openstack/kolla-ansible: Optimize reconfigure action for nova  https://review.openstack.org/40741616:54
*** zhubingbing has quit IRC16:55
mdormanthanks16:56
gemainc0: I am also adamant about gates16:59
*** athomas has quit IRC16:59
gemathat's why the 3rd party CI is high on my list so our images can be tested16:59
gemaand show up in the voting17:00
duonghqinc0, egonzalez I'll be back in about 8hr, and available for next17:00
inc0cool17:00
inc0thanks duonghq17:00
inc0gema, I think we should write regular gatejobs for infra too17:00
*** TxGirlGeek has quit IRC17:00
Jeffrey4l_sean-k-mooney, re qemu, no. in the gate, we are already in kvm. it will be very slow to use nested qemu.17:00
gemainc0: we don't have ARM64 on those gates yet17:01
inc0sean-k-mooney, we don't have control over how vms in infra are spawned17:01
duonghqnext 6-7hrs, and will "disappear" for couple of days due to "Tet holiday" in Vietnam (Lunar new year)17:01
gemaalso on my TODO but that is a bigger undertaking17:01
*** TxGirlGeek has joined #openstack-kolla17:01
duonghqnight guys17:01
SridharP2hello17:01
Jeffrey4l_aha, duonghq me too. :)17:01
*** jpstnh has joined #openstack-kolla17:01
SridharP2I am trying setup the kolla environment using QSG17:01
gemaberendt, inc0: thanks for your help17:02
inc0gema, so what we could do is you could write script like https://github.com/openstack/kolla/blob/master/tools/setup_gate.sh17:02
duonghqJeffrey4l_, nice17:02
inc0for arm17:02
*** duonghq has quit IRC17:02
*** Serlex has left #openstack-kolla17:02
sdake_hey peeps17:02
inc0or no, that doesn't make any sense17:02
inc0gema, we'll figure it out17:02
SridharP2It gets strucked @TASK [kube-join : join workers to the initialized kubernetes cluster] **********17:02
SridharP2skipping: [kube1]17:02
gemainc0: ack, thanks17:02
inc0gmorning sdake17:02
sdake_bmace_ i've nearly got the deployment script working (with help from friends)17:02
inc0ok afk, moving to office17:03
sdake_bmace_ icsid resets17:03
gemanight17:03
*** inc0 has quit IRC17:03
sdake_bmace_ rather enters a backoff crsh17:03
sean-k-mooneyJeffrey4l_: i was suggesting useing the image properties to request the kvm on the host to emulate an arm cpu17:03
sean-k-mooneyJeffrey4l_: well i was thinking that i did not say it till now17:03
gemasean-k-mooney: I haven't tried that, but it sounds slow :D17:04
*** japestinho__ has quit IRC17:04
gemabut it may work for testing, will think about it17:04
sean-k-mooneygema: well its not insanly slow but if it was supported it would mean your thridparty ci could be first party17:05
sean-k-mooneybut it would require a custom image from infra with the image properties set and they may not be able to provide that17:05
*** Jeffrey4l_ has quit IRC17:06
*** sp___ has joined #openstack-kolla17:07
*** tonanhngo has joined #openstack-kolla17:07
sbezverkSridharP2: are you trying to use kubeadm for bringing up your cluster?17:10
*** jemcevoy has quit IRC17:11
portdirectSridharP2: The issue you are having (if using halcyon) is that the docs refer to a k8s version that we cannot (yet) provide a controller manager image that supports ceph17:11
portdirectif you run the setup script with v1.5.1 instead of 1.5.2 and recreate the cluster you should be good to go17:12
SridharP2Quit17:12
*** SridharP2 has left #openstack-kolla17:12
*** newmember has joined #openstack-kolla17:13
sean-k-mooneyportdirect: is there a plan to provide a production way of deploying k8s for pike. be that in kolla-k8s or a recommedation that is gated on?17:13
berendtgema: you are welcome :-)17:14
portdirectsean-k-mooney: not that I'm aware of, deploying k8s for production is quite an involved activity and one I'd personally prefer to keep out of scope for kolla-k8s17:15
sean-k-mooneyit would be nice to have a tested recomendation for someone why is new to k8 that wants to use kolla-k8 to stand up openstack17:15
portdirectthough we will need to address quite a few issues on that front - especially regarding the kubernetes-entrypoint container and the distribution of access tokens17:16
sean-k-mooneypersonally there are several options discribed for dev but im not sure how to take the next step to production with kolla-k817:16
*** sp__ has joined #openstack-kolla17:16
* sean-k-mooney admitly im not sure how to take the first step to getting a dev setup with kolla-k8s working yet either17:17
portdirectsean-k-mooney: are you familiar with k8s?17:17
sean-k-mooneyportdirect: not partically17:17
* portdirect laughs and crys at the same time17:17
sean-k-mooneyi have been reading up on it in my spare time but have not spent enough time with it to be comfortable17:18
portdirectcool - in many ways you can view it (and I'll get shot for saying this) as a mini openstack17:18
sean-k-mooneyportdirect: well i am aware of what is should do and some of its concepts17:18
portdirectand so when it comes to a production cluster you have a similar number of viarbles to consider17:18
sean-k-mooneyi just have no hands on experince using it really yet17:18
*** williamcaban has quit IRC17:19
portdirectwe should definalty document some of them, ie PKI for api servers and etcd, and CNI networking17:19
sean-k-mooneyhehe ya i have noticed that k8 is perhaps the only tech stack i have found with = or more installers then openstack17:19
*** sp___ has quit IRC17:19
portdirectas well as ABAC for the api - which no openstack-on-k8s project has addressed to date17:19
sean-k-mooneyABAC account based acess control?17:20
sean-k-mooneyatribute?17:20
portdirectattribute17:20
*** srwilkers has quit IRC17:20
sean-k-mooneyah ok17:20
portdirectso you can understand why I'm a bit hesitant to go down that rabbit hole - your point above was right on the money, considering its age i think k8s has more installers than openstack17:21
*** l4yerffeJ_ has quit IRC17:21
*** l4yerffeJ_ has joined #openstack-kolla17:21
portdirectand that can turn into a higly opinionated area17:21
sean-k-mooneyit would be good to have a guide or a blog that explained how to transition form kolla-ansible to kolla-k8s. e.g. if you are already comfortable with kolla-ansibel what do you need to know about kolla-k8s to get started with it17:22
portdirectI personally have used both ansible and puppet (via foreman) for production clusters17:22
sean-k-mooneyhowever i am aware that kolla-k8s is changing alot with helm and charts and operators currently17:23
*** jheroux has joined #openstack-kolla17:24
portdirectI'd need to defer to someone more familiar with kolla-ansible on that front - I would like to see us move away from any dependency on kolla-ansible and so by that stage i think we will have two totally separate deliverables - and moving seamlessly from one to the other would be as involved as moving from say OSA to kolla-ansible17:24
*** Jeffrey4l has joined #openstack-kolla17:25
*** TxGirlGeek has quit IRC17:25
*** rushil has joined #openstack-kolla17:25
sean-k-mooneyoh yes i understand. i was thinking for developers as i would like to get to the point where i can contribute to feature in both17:26
sean-k-mooneye.g. i add a contaienr to kolla. i would like to know how to extend kolla-k8s the same way i would add an ansibel role.17:26
sean-k-mooneyi just dont know enough about how kolla-k8s works with k8s to make that conenction yet17:27
*** TxGirlGeek has joined #openstack-kolla17:27
bmace_sdake: that is great about iscsid.  i just found out what was wrong with my k8s cluster access that was failing so i am going to try to run again now and see how it goes.  i'm still working off a branch after my genconfig fixes rather than head since the fluentd changes seemed to have broken genconfig yesterday (at least from this centos container environment)17:27
portdirectoh - yeah - that makes sense - and i think that the work sdake_'s doing is the first step in that direction17:27
sdake_bmace_ - i'll push an update as soon as possible - ther  are many more problems in the script17:29
sdake_bmace_ i have a 30 minute meeting - then need to schlep the changes from a dev machine to my git  repo17:29
*** Jeffrey4l has quit IRC17:30
openstackgerritMerged openstack/kolla-kubernetes: Typo fix: folowing => following  https://review.openstack.org/42500817:30
*** msimonin has quit IRC17:30
*** sp__ has quit IRC17:30
portdirectsdake_: are you running 1.5.2 on your machines?17:31
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Implement Keystone zero-downtime upgrade  https://review.openstack.org/40521517:31
sdake_portdirect think so17:31
portdirectsdake_: att-comdev have not published a ceph enable controller manager image for that version which is whats caused a few people to get stuck during deployment17:32
portdirectany chance you could run kubectl version just to double check?17:32
sbezverkportdirect: is your setuo script uses dynamic pvc?17:34
sdake_yup17:34
sdake_portdirect server is 1.5.2, client is 1.5.117:34
portdirectsbezverk: no but it pulls in the image required to enable that17:34
sbezverkportdirect: for a bit I wa running 1.5.2 with 1.5.1 att controller manager17:35
sbezverkhave not noticed any issues17:35
portdirectsbezverk: yes - but the setup sets the version to be the same17:35
sbezverkso no really need to rollback to 1.5.1 for everything17:36
portdirectso doint it manually is fine - but is not the process we have documented17:36
sbezverkportdirect: sure sure17:36
portdirectsbezverk: https://github.com/att-comdev/halcyon-kubernetes/blob/master/kube-deploy/roles/kube-prep/tasks/patch-kube.yml#L1817:38
portdirectwithout a pullable version of that image the playbook will hang - as has been reported by three people on this channel today17:38
*** jmontleon has quit IRC17:39
mgileskfox1111 sbezverk portdirect When you have a chance, can you review nova-control service: https://review.openstack.org/#/c/42145717:39
mgilesNo hurry, but it's been sitting almost a week with no changes other than rebases and I wouldn't mind getting it closed out17:39
sbezverkmgiles: man this is 3rd time I +2 you ;-)17:40
mgilesI know - I don't think anybody else has looked at it to give it the second +217:40
sbezverkmgiles: please twist somebody's arm to workflow it17:40
mgilesThanks17:40
portdirectmgiles: even has reno17:40
* portdirect seraches for reason to be vindictive17:40
*** jmontleon has joined #openstack-kolla17:41
mgilesreal nice portdirect17:41
portdirectmgiles: i came up blank :( - nice work :)17:42
mgilesthanks portdirect.   I'll try to give you a reason next time. :)17:43
*** rushil has quit IRC17:45
*** RichWellum has joined #openstack-kolla17:47
*** jmontleon has quit IRC17:49
portdirectkfox1111/sbezverk : is this still required https://blueprints.launchpad.net/kolla/+spec/api-interface-bind-address-override17:50
kfox1111back17:53
*** tonanhngo has quit IRC17:53
kfox1111portdirect: yes/no?17:53
kfox1111portdirect: all of the existing services are complete. I'd expect other non tested services not to work though.17:54
kfox1111portdirect: but it may not matter depending on what we do with confimap generation going forward.17:54
*** jmontleon has joined #openstack-kolla17:54
kfox1111so, its either ongoing, or complete.17:54
portdirectso wa can mark it as complete? inc0 wanted me to get an answer17:56
* kfox1111 shrugs17:57
*** simon-AS559 has quit IRC17:57
kfox1111if we get to helm generated configmaps before we add any more services, yes.17:57
kfox1111if someone wants more services before that, maybe not.17:58
*** klindgren__ has joined #openstack-kolla17:58
kfox1111theres a third option, to split the bp and close it for ocata, and make anew one for pike17:58
portdirectright lets close it then - i don't think we should add any more services until we can actually run stuff :)17:58
openstackgerritMerged openstack/kolla-kubernetes: Nova-control service deployment  https://review.openstack.org/42145717:58
kfox1111mgiles: theres kubernetes_entrypoint: true set on a lot of things explicitly. this makes it hard to turn on/off as a set.17:59
kfox1111Though I know other ps's still have that in there too.17:59
mgileskfox1111 Do we have an alternative?18:00
kfox1111could you make another ps that removes the lower layer kubernetes_entrypoint: true  stuff from all the helm/services/*/values.yaml?18:00
mgilesAnd specify it globally?18:01
kfox1111mgiles: a global.kolla.all.kubernetes_entrypoint=true is enough to cover them all.18:01
*** klindgren_ has quit IRC18:01
kfox1111yeah.18:01
*** gfidente is now known as gfidente|afk18:01
mgilesThat means it will get run even for containers that don't need entrypoint?18:01
mgileslike the svcs18:01
kfox1111false should be needed only if you really dont want it.18:01
sbezverkportdirect: couple of services are not done18:01
kfox1111svc's don't look at the var.18:01
mgilesah ok18:02
mgilesmakes sense18:02
kfox1111portdirect: "actually running stuff" means on bare metal?18:02
kfox1111that should happen in the next week or two.18:02
*** egonzalez has quit IRC18:02
kfox1111I think we're close.18:02
kfox1111so adding more services may come realatively soon.18:02
*** TxGirlGeek has quit IRC18:03
sbezverkkfox1111: what about the horizon check fix? did it help?18:04
*** TxGirlGeek has joined #openstack-kolla18:04
sbezverkkfox1111: it seems horizon service ps is still hanging inprogress state18:05
sbezverkkfox1111: nvm18:06
kfox1111there are a few gate fixes still in review. :/18:07
kfox1111and I'm trying to get a cache going to help with the network/hub flakyness.18:08
kfox1111between the two, I'm hoping to get it stable again.18:08
sbezverkkfox1111: neutron in services bp shows as todo, but I think you completed it, right?18:09
openstackgerritsayantani proposed openstack/kolla: [WIP] Update Multinode deployment of kolla guide  https://review.openstack.org/40836418:09
*** TxGirlGeek has quit IRC18:10
*** matrohon__ has quit IRC18:10
bmace_sdake_: think i made it to your iscsid error.  you said you had a fix for it going up in not too long?18:10
*** TxGirlGeek has joined #openstack-kolla18:12
sbezverkbmace_: I worked a little bit with iscsi, could you share what exactly the problem with it?18:13
sayantani01Hi guys, can I have some eyes on https://review.openstack.org/#/c/424271/ ? :)18:14
bmace_sbezverk: i'm just running the dev_workflow.sh script near the end of the containerized dev environment setup and it is blowing up with: Unable to locate iscsid directory in search_paths=[/etc/kolla, /etc/kolla-kubernetes, /opt/kolla/etc/kolla, /opt/kolla-kubernetes/etc/kolla-kubernetes]18:16
bmace_sdake was hitting iscsi issues also and said he had something working18:16
bmace_i haven't dug too much into the script or why it might be failing yet.  if it isn't the same issue he was hitting I will.18:16
openstackgerritMark Giles proposed openstack/kolla-kubernetes: Improve handling of kubernetes_entrypoint variable in nova services  https://review.openstack.org/42532418:17
openstackgerritsayantani proposed openstack/kolla-ansible: Fix RabbitMQ server error in bifrost  https://review.openstack.org/42438418:17
sdake_i am grabbingbmyup - working on it18:17
sbezverkbmace_: ok got it, so it is not real iscsi issue :-), it looks more like a packaging issue18:17
sdake_bmace_ yup working on it18:17
sdake_bmace_ i have several fixes18:17
sdake_bmace_ you can diff the env and kee pthe ones you want or just pull a fresh patch18:17
sdake_note iscsi-tgtd now fails, but everything deploys propery18:18
sdake_iscsi-tgdt enters a backoff crash loop18:18
sdake_sbezverk ^^ so that may be a legitimate issue18:18
sdake_sbezverk i can provide you access if you like to take a quick peek18:18
sdake_sbezverk any diagnostics you could provide would be helpful - even if we are doing it wrong just stating that owuld be helfpul :)18:19
*** unicell has joined #openstack-kolla18:20
sbezverksdake_: most likley you hit a known bug, when host has iscsi-initiator-utils already installed, in the container iscsd fails to start18:20
sdake_sbezverk i am not running in a container18:20
sbezverkremvoing iscsi-utils from the host will resolve it18:20
sdake_sbezverk i am running on bare metal18:20
sbezverksdake_: then what goes into a crash loop?18:21
sdake_sbezverk i am sorting out workflow for real ops, not a dev workflow18:21
sdake_sbezverk its best if yu look at the machine i think :)18:21
sbezverkok, how I can access it?18:21
sdake_i'll sen you a pm with creds18:21
sdake_http://paste.fedoraproject.org/536506/14853686/ -> http://paste.fedoraproject.org/536506/1485368618:25
*** rushil has joined #openstack-kolla18:25
*** inc0 has joined #openstack-kolla18:29
*** sayantani01 has quit IRC18:30
kfox1111sbezverk: yeah.18:31
sbezverkkfox1111: so we can change it to done right?18:33
*** sambetts is now known as sambetts|afk18:34
*** inc0 has quit IRC18:34
sdake_i'll be pushing a patch in short order18:35
*** inc0 has joined #openstack-kolla18:35
sdake_openstack endpoint list seems busted18:35
sdake_bmace_ to get past the iscsi problem uninstall the iscsi initiator18:35
sdake_bmace_ on whatever node is running iscsi18:35
sdake_iscsi-inititaor is a default packgae in centos18:35
sbezverksdake_: not really, it gets installed only by command18:36
sdake_sbezverk ok - ill check int o that18:36
sdake_sbezverk what about this openstack endpoint list freezing up?18:36
sdake_sbezverk shouldn't that owrk?18:36
*** jmccarthy has joined #openstack-kolla18:36
sbezverksdake_: most likely something is not right with services18:36
bmace_sdake_: that is to fix the iscsi-tgtd issue you are hitting right, not the earlier issue that you have changes for?18:37
*** sdake_ has quit IRC18:39
*** sdake has joined #openstack-kolla18:40
*** krtaylor has quit IRC18:40
*** Jeffrey4l has joined #openstack-kolla18:41
sdakethis is like reliving a bad nightmare of kolla-ansible startup :)18:42
bmace_lol18:43
kfox1111sbezverk: yeah.18:44
*** williamcaban has joined #openstack-kolla18:44
kfox1111sdake: does catalog list work?18:44
bmace_sdake: using the halycon-vagrant stuff i don't have iscsi-utils on those nodes so i might be ok once i get past the earlier "unable to locate" iscsid error?18:44
kfox1111sdake: I found some of the keystone admin stuff doesn't work outside of k8s as part of the local repo patchset. I fix it there.18:45
sdakekfox1111 sbezverk is debuggin with me18:45
kfox1111sdake: openstack catalog list should work though. if not, its keystone nto working.18:45
*** Jeffrey4l has quit IRC18:46
*** rwellum has joined #openstack-kolla18:46
sdakekfox1111 ya - OS_AUTH_CONFIG is not correct at present - we are on it :)18:46
bmace_sdake: did you upload your changes? i see some from this morning around setup_dev_env and dev_workflow but the dev_workflow.sh change is size 0.18:46
*** sean-k-mooney has quit IRC18:47
kfox1111sdake: ok. cool.18:47
*** sean-k-mooney has joined #openstack-kolla18:48
sdakebmace_ i've got a lunch appointment and i will upload it after lucnh - after i debug this stuff a bit more18:48
kfox1111sdake: just wanted to mention the admin related keystone stuff not working outside of stuff running inside k8s in case you trip over that.18:48
sdakebmace_ i have a hard appointment in 20 mins so dont' have time to update the patc hyet18:48
bmace_sdake: kk18:48
sdakebmace_ after i get back from lucnh its first order of buviness18:49
sdakethe 0 byte change was to fix the problem with a script not being executable18:49
bmace_sdake: sounds great.  ah, yup, i fixed the same in my container.  maybe we can get this all working today!  nice start anyway.  lots of improvements we can make to it.18:50
sdakebmace_ it will be working today i am sure of i t18:50
bmace_i had to make a lot of local changes for proxy stuff which needs to be supported in a cleaner way in general.18:50
bmace_i suppose it can be just via build / run args also, so could even be just a doc thing18:51
*** rushil has quit IRC18:51
*** tonanhngo has joined #openstack-kolla18:53
*** pcaruana has quit IRC18:53
*** britthouser has joined #openstack-kolla18:54
*** inc0 has quit IRC18:54
kfox1111sbezverk: fyi, https://review.openstack.org/#/c/425331/18:55
*** inc0 has joined #openstack-kolla18:56
sbezverkkfox1111: what is it for? never had to deal with tha before18:57
*** tonanhngo has quit IRC18:57
sdakebmace_ - ok dude i think we are in good shape18:58
sdakebmace_ but i really gotta jet - or going to be late18:58
sdakebmace_ i have to tear down my env since i didn't setup kubeadm properly apparently18:58
sdakebmace_ however i'll push my changes up in a new review and you can use what you want18:58
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Create deliverables.yaml  https://review.openstack.org/42478418:58
bmace_sdake:  kk, fantastic, thanks18:58
*** shardy is now known as shardy_afk19:01
*** williamcaban has quit IRC19:01
*** rushil has joined #openstack-kolla19:04
britthouserSorry I missed meeting today.  My slack-irc server seems to have died.19:04
openstackgerritMark Giles proposed openstack/kolla-kubernetes: Improve handling of kubernetes_entrypoint variable  https://review.openstack.org/42535019:04
*** esmiurium has quit IRC19:05
mgileskfox1111 fyi, created another patch set to clean up the same issue with the entrypoint variable in the other services.  So now it should be consistent throughout19:05
kfox1111cool. thanks.19:06
kfox1111sbezverk: going to merge that with the work to gather docker container lists, so I can push them to tarballs.o.o and pull them back into the gate.19:07
kfox1111sbezverk: that way, we will see less breakages due to network connectivity and hub issues.19:07
*** MasterOfBugs has joined #openstack-kolla19:08
*** MasterOfBugs has quit IRC19:08
*** RichWellum has quit IRC19:09
*** MasterOfBugs has joined #openstack-kolla19:09
*** pramodrj07 has joined #openstack-kolla19:09
*** Jeffrey4l has joined #openstack-kolla19:10
*** britthouser7 has quit IRC19:10
openstackgerritMerged openstack/kolla-ansible: Fix typo heat.conf  https://review.openstack.org/42527019:11
*** esmiurium has joined #openstack-kolla19:12
sbezverkkfox1111: so docker hub's hip hops will not affect us :) I like it!!19:13
kfox1111and the same for google's container registry. and I think there was a third one in the list too....19:14
*** jmccarthy has quit IRC19:15
*** Jeffrey4l has quit IRC19:15
kfox1111http://logs.openstack.org/21/424921/4/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/1636f33/logs/docker_images.txt19:15
kfox1111yeah. quay too. :/19:15
kfox1111docker.io for calico. quay for calico. gcr for the rest of k8s, and docker.io for kolla.19:15
*** britthouser has quit IRC19:16
kfox1111so only the periodic jobs will pull from there, unless something changes between a periodic and a gate run.19:16
*** britthouser has joined #openstack-kolla19:16
kfox1111so, the gate shouldn't break in that case. will just switch to remove for those containers.19:17
kfox1111and then the next periodic run will fix it.19:17
openstackgerritEduardo Gonzalez proposed openstack/kolla: Implement Keystone zero-downtime upgrade  https://review.openstack.org/38854419:17
*** williamcaban has joined #openstack-kolla19:18
*** krtaylor has joined #openstack-kolla19:19
*** williamcaban has quit IRC19:22
*** msimonin has joined #openstack-kolla19:28
nradocan someone explain to me why I got -1 from jenkins https://review.openstack.org/#/c/422566/ ? How can i check it?19:29
*** mgoddard has joined #openstack-kolla19:29
qwangnrado: it's weird. try to recheck and see how it goes19:41
kfox1111and... another case of vswitchd out of order...19:41
kfox1111we really need a fix for that..19:41
*** sayantani01 has joined #openstack-kolla19:43
kfox1111sbezverk: uh, heres one that loks like a vswitchd issue,19:45
sbezverkkfox1111: do you mean neutron ovs agent getting ahaed of vswitchd?19:45
kfox1111but is with helm entrypoint: http://logs.openstack.org/24/425324/1/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/7464604/console.html19:45
*** britthouser has quit IRC19:45
kfox1111sbezverk: yeah.19:45
*** britthouser has joined #openstack-kolla19:45
kfox1111so for some reason, that race is hitting us a lot now.19:46
kfox1111we need to figure out a fix. :/19:46
kfox1111the problem is ovs isn't responsive yet, right?19:46
sbezverkkfox1111: I can try to workaround it by loading module in both containers ovsdb and vswitchd butthis is not a good solution19:46
kfox1111before neutron-openvswitch-agent starts, I mean.19:46
*** williamcaban has joined #openstack-kolla19:47
kfox1111what if we added an init container to neutron-openvswitch-agent that blocked until it could do an appropriate openvswitch command?19:47
sbezverkkfox1111: no vswitch has not loaded ovs kernel module yet19:47
kfox1111ovs-vsctl show or something.19:47
sbezverkkfox1111: lsmod | grep ovs19:47
*** rushil has quit IRC19:48
*** srwilkers has joined #openstack-kolla19:48
kfox1111are you sure its actually the kernel module its loking for?19:48
*** williamcaban has quit IRC19:48
kfox1111I would expect openvswitch-agentn not to talk to the kernel directly,19:48
kfox1111but to vswitchd.19:48
*** rushil has joined #openstack-kolla19:48
sbezverkkfox1111: right it does not19:49
kfox1111k.19:49
sbezverkkfox1111: but if the module is not loaded, then when it tries to get br-int datapath id, it comes back blank19:49
sbezverkkfox1111: I do not know exactly which system call it uses to get that info19:50
kfox1111can that happen before ovs-vsctl show works?19:50
kfox1111I think its talking through the unix socket to the db?19:50
sbezverkkfox1111: yes because for that ovsdb has already socket up and running19:50
kfox1111but the db's blocking waiting to talk to vswitchd and then neutron times out the connection?19:51
kfox1111this is going to be a hard one for me to test at the moment.19:51
sbezverkkfox1111: hard to confirm, do not know how neutron ovs agent requests bridge's dataplane id19:51
sbezverkkfox1111: easy solution at least for now19:52
sbezverkmoving openvswitch at the top of workflow19:52
kfox1111oh... just put in a barior...19:52
sbezverkit will give sufficient time for ovs to wake up19:52
kfox1111yeah. that would fix the gate. wouldn't fix the reboot issue.19:52
sbezverkkfox1111: we can use lsmod | grep openvswitch as a parameter for barrier19:53
*** williamcaban has joined #openstack-kolla19:53
kfox1111yeah, but not sure that will actually work or not, or if vswithc has to be up too.19:53
kfox1111theres a slight race there.19:53
kfox1111it would help for sure.19:53
kfox1111or, mayube we just do both.19:54
kfox1111check for kernel module. check for ovs-vsctl show working.19:54
kfox1111if both work, we're good to go either way.19:54
sbezverkkfox1111: we do not have ovs bits installed on the host19:54
kfox1111init-container.19:54
sbezverkso running ovs commands from host is problematic19:54
sbezverkah in init container19:54
sbezverkok then..19:54
kfox1111yeah. we just block neutron-openvswitch-agent entirely until ready.19:55
kfox1111I think I can take a quick stab at it...19:55
sbezverkkfox1111: sounds good19:55
sbezverk kfox1111: you will need a sample of output when module is not loaded19:56
sbezverkif I rememebr right it generates something, not an error though19:56
kfox1111lsmod | grep openvswitch | wc -l19:56
kfox1111should work.19:57
kfox1111if not 0, somethings loaded?19:57
*** Jeffrey4l has joined #openstack-kolla19:58
*** esmiurium has quit IRC19:59
*** zioproto has joined #openstack-kolla19:59
*** williamcaban has quit IRC20:00
openstackgerritMerged openstack/kolla-kubernetes: Improve handling of kubernetes_entrypoint variable  https://review.openstack.org/42535020:01
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch.  https://review.openstack.org/42537620:02
kfox1111sbezverk: have a look at that one.20:02
kfox1111ok. we have a gate change incoming. https://review.openstack.org/#/c/425331/20:03
kfox1111shouldn't have any effect, but just a heads up.20:03
sbezverkkfox1111: I think second check is redundant as ovs will always returns something20:04
*** williamcaban has joined #openstack-kolla20:04
sbezverkunless you analyze the exact output20:04
kfox1111sbezverk: you think the vsctl command only talks to the db, not the actual vswitchd?20:04
*** Jeffrey4l has quit IRC20:04
*** williamcaban has quit IRC20:04
nradocan i undo a recheck20:04
nrado??20:05
nradowas in the wrong window…20:05
sbezverkkfox1111: I think so, but not 100% sure20:06
sbezverkkfox1111: I think vswitchd is just needed to execute commands of neutron ovs agent20:07
kfox1111sbezverk: lets be extra sure then, to stablize the gate. we can always reevaluate later and remove it if we can prove it doesnt add value?20:07
sbezverkbut the actual ovs-vs commands talk to ovsdb directly20:07
kfox1111well, we may have  race still then...20:07
*** esmiurium has joined #openstack-kolla20:07
kfox1111how do we ensure vswitchd is actually up then?20:07
sbezverkkfox1111: sure thing I was not against it just a heads up ;)20:08
kfox1111yeah. no, its good. we need to figure out all the ways it can break.20:08
sbezverkkfox1111: the most relable way is to use entrypoint20:08
kfox1111is there a test command nwe can come up with that actually talks to the vswitchd.20:08
kfox1111we are using nentrypoint. but it fails on this case. :/20:08
sbezverkkfox1111: but we are not checking on POD20:09
sbezverkkfox1111: or daemonset20:09
sbezverkneutron ovs agent has no dependency confiugred for vswitchd20:09
kfox1111two isssues. I don't think you can check to see if a pod is running on the same host?20:09
kfox1111and 2, it makes for much worse scaling. :/20:10
kfox1111as its cheep to talk to something on the same server.20:10
kfox1111its much more expensive for all the things to talk back to the central server.20:10
kfox1111one other random note.20:10
kfox1111in the reboot tests, I have seen kubernetes state thigns are ready when they arnt'.20:10
kfox1111ie, if you pull the rug out from under it,20:11
kfox1111then start everythign back up.20:11
kfox1111etcd still has a record of all the pods being "ready", but they arent'.20:11
kfox1111then it goes and tries to relaunch everything, and updates the status as it goes.20:11
kfox1111kubernetes-entrypoint may malfunction in this state. not sure. :/20:11
sbezverkkfox1111: damonset can be queried for a state if opd20:11
sbezverkthat is what they need to use for this check20:12
kfox1111does it check locally only, or does it call back to the server?20:12
sbezverkkfox1111: it cannot check locally as locally no info exist.. I am positive they do it via api server20:13
sbezverkwell at least that is how I would approach it, to use single point of truth20:13
kfox1111yeah. and I really don't like that for scaling/security reasons. :/20:13
kfox1111theres got to be a better way for this one use case.20:14
kfox1111in another project, I just touch a file in one, and look for it in another pod. simple/scalable, if a little hacky looking.20:14
sbezverkkfox1111: what if that file gets removed accidently?20:15
sbezverkor will be left from the previous run?20:16
kfox1111it woudl only affect restarts on the container. so not too bad, but don't remove a file if needed...20:16
kfox1111hard reboot case could be handled by placing it in /dev/shm.20:16
kfox1111it will get whiped after reboot then, so won't exist.20:16
sbezverkI think api should be the central point of info requests20:16
kfox1111that requires A, extra security granted to containers. adn B, extra load on the api server.20:17
kfox1111both as an op make me nervious when you might have hundreds of nodes, that run user code.20:17
sbezverkkfox1111: container automatically gets token to request api server so no need any additional20:18
sbezverkinfo20:18
kfox1111sbezverk: yeah, a privileged one. and that really scares me. there is a ps in to disable that horible behavior. :/20:19
sbezverkkfox1111: well then it will break operator20:19
kfox1111the flag can be used selectively per pod/container. so only enabled where strictly needed.20:20
kfox1111so won't break operators.20:20
sbezverkas it uses this token to authenticate against api server for its quereis20:20
kfox1111yup. the idea is to lessen the attack surface to only where needed, not to disable it entirely everywhere.20:20
sbezverkkfox1111: ok if selective flag is available then should be ok20:20
kfox1111I just object to it being on 1000 pods when 10 pods would be sufficient. :)20:21
kfox1111I'm really glad they have the token feature though. very important.20:21
kfox1111still wish nova would figure that out. :/20:22
*** nrado has quit IRC20:22
sbezverkkfox1111: yep, really usefull, makes life much easier when talking to api from the container20:26
kfox1111sbezverk: uh... this looks a bit weird: http://logs.openstack.org/50/418550/56/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/aeed5cb/console.html20:28
kfox1111weird...20:30
kfox1111http://logs.openstack.org/50/418550/56/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/aeed5cb/logs/pods/kolla-cinder-create-keystone-service-9nr89-main.txt20:30
kfox1111its trying to contact 35357 on keystone-public?20:30
sbezverkhmm it should be 5000 on public20:32
kfox1111oh... because of this:20:32
kfox1111{{- $defaultPublicHost := printf "%s-public" $elementName }}20:32
kfox1111{{- $publicHost := .Values.public_host | default $defaultPublicHost }}20:32
kfox1111hmm...20:32
kfox1111a, this isn't using the global api. we missed it somehow...20:33
kfox1111and b, its registering a k8s internal dns name for external use....20:33
sbezverkkfox1111: how it was wokring before ?!?!?20:33
kfox1111no idea.... I'm pretty confused by that.20:33
sbezverkkfox1111: even today I saw sucessfull runs of entrypoint job20:34
kfox1111maybe we never ran any commands that relooked up the keystone endpoint from the endpoitns list.20:34
kfox1111its affecting this ps, as I'm trying to make admin endpoint external too.20:34
sbezverkkfox1111: it should be straight forward, you just need to external port for keystone-admin svc20:35
kfox1111not quite.20:36
sbezverkthen it will allocate cluster ip20:36
kfox1111the entrypoint in keystone has to point to the external address as well.20:36
*** Mech422 has joined #openstack-kolla20:36
kfox1111thats the problem I'm running into.20:36
sbezverkok20:36
kfox1111it was registered as "http://keystone-admin:...."20:36
kfox1111which doesn't work outside of k8s.20:36
sbezverkok right our approach does not allow overriding keystone-admin20:39
sbezverkby anything including ip address20:39
kfox1111I think I've got a ps kergiggered around.20:39
kfox1111added global api support, and renamed the var a little bit.20:39
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds  https://review.openstack.org/41855020:42
kfox1111sbezverk: the openvswitch folks recommended ovs-appctl version20:43
*** rwellum has quit IRC20:44
*** williamcaban has joined #openstack-kolla20:45
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch.  https://review.openstack.org/42537620:45
*** Jeffrey4l has joined #openstack-kolla20:45
*** saneax-_-|AFK has quit IRC20:46
sbezverkkfox1111: cool20:47
*** williamcaban has quit IRC20:49
*** mgiles has quit IRC20:58
*** TxGirlGeek has quit IRC21:02
*** jtriley has quit IRC21:02
*** shardy_afk has quit IRC21:03
openstackgerritsayantani proposed openstack/kolla-ansible: Modify config.yml playbook for bifrost role  https://review.openstack.org/41919621:03
*** shardy_afk has joined #openstack-kolla21:05
*** TxGirlGeek has joined #openstack-kolla21:05
*** shardy_afk is now known as shardy21:07
*** zioproto has quit IRC21:11
openstackgerritsayantani proposed openstack/kolla: Update Multinode deployment of kolla guide  https://review.openstack.org/40836421:16
*** bpattewar has joined #openstack-kolla21:16
openstackgerritsayantani proposed openstack/kolla: Update Multinode deployment of kolla guide  https://review.openstack.org/40836421:18
*** rushil has quit IRC21:19
*** erlon has quit IRC21:20
*** erlon has joined #openstack-kolla21:23
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Capture lists of containers by type and cache containers.  https://review.openstack.org/42492121:23
*** sayantani01 has quit IRC21:23
kfox1111sbezverk: theres part 2 of 3. :)21:24
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Support rolling upgrades of horizon  https://review.openstack.org/33399621:32
kfox1111inc0: https://review.openstack.org/#/c/422950/ ? :)21:32
*** l4yerffeJ has joined #openstack-kolla21:34
*** Jeffrey4l_ has joined #openstack-kolla21:34
*** l4yerffeJ_ has quit IRC21:35
sbezverkkfox1111: are you sure bzip exists in the base image used for vm?21:35
inc0there ya go kfox111121:35
kfox1111inc0: thanks.21:35
*** Jeffrey4l has quit IRC21:35
kfox1111sbezverk: oh. yeah. was going to force it in. but forgot.21:35
kfox1111guess we'll see if it is or not. :/21:35
*** sayantani01 has joined #openstack-kolla21:36
kfox1111interesting... the openvswitch commend is just blocking....21:36
kfox1111same behavior as neutron-openvswitch-agent it seems.21:37
kfox1111maybe just needs a timeout...21:37
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch.  https://review.openstack.org/42537621:38
sbezverkkfox1111: does it get stuck on second check?21:40
openstackgerritMerged openstack/kolla-kubernetes: Improve handling of kubernetes_entrypoint variable in nova services  https://review.openstack.org/42532421:40
kfox1111sbezverk: yeah. it seemed to. for 3 min. really weird.21:40
*** rushil has joined #openstack-kolla21:41
kfox1111rarg...21:42
sbezverkkfox1111: one thing could help us is timestapms when each container starts and finishes for the case of init containers.. then it would be easier to see possible racing things..21:42
kfox1111those should be in the pod/containername.txt file.21:43
kfox1111at the bottom.21:43
kfox1111its a kubectl describe pod <x>21:43
kfox1111which has the events.21:43
sbezverkah ok those..21:43
kfox1111the one thing I don't like, is its in minutes ago from now. :/21:43
*** shardy has quit IRC21:45
jascott1kfox1111 when you get time, what do you think about logical condition format? https://github.com/kubernetes/helm/issues/183721:47
sbezverkkfox1111: yeah we need msec timestamps21:47
sbezverkjascott1: would it be more yaml like, conditions: and then under list of charts with : enabled/disabled21:49
sbezverklike this conditions:21:49
sbezverk   subchart1: enabled21:49
sbezverk    subchart2: disabled21:49
*** gfidente|afk has quit IRC21:49
jascott1sbezverk yes like that21:49
jascott1but also we are talking about AND/OR/NOT21:49
jascott1sub1,sub2&sub321:50
jascott1sub1 OR sub2 AND sub321:50
jascott1sub1^sub221:51
jascott1sub1 AND NOT sub221:51
sbezverkjascott1: I see, that might be interesting task to parse all these21:51
jascott1sbezverk lol yeah Ive been thinking the same since I will probably be doing it :)21:51
jascott1there is also this idea of the format k8 labels use: {key: environment, operator: NotIn, values: [dev]}21:53
jascott1which is maybe more complex but already exists21:54
inc0sadpanda...my galera is broken21:54
inc0it doesn't get to listen on 3306 and I have no idea why21:54
*** v1k0d3n has quit IRC21:59
*** breitz has quit IRC22:00
openstackgerritsayantani proposed openstack/kolla-ansible: Update Bifrost playbooks  https://review.openstack.org/42543522:00
kfox1111jascott1: theres one weird thing there...22:00
*** breitz has joined #openstack-kolla22:00
*** v1k0d3n has joined #openstack-kolla22:01
kfox1111jascott1: , should probablyl be scoped lower then all the operators.22:01
kfox1111so, foo & bar,22:01
kfox1111foo could be "global.a,global.b"22:01
kfox1111whichever is first defined.22:01
jascott1so you are saying comma is only delimiter and not operator22:03
jascott1?22:03
*** Syffs has joined #openstack-kolla22:04
*** fguillot has quit IRC22:04
kfox1111well, its an operator of sorts,22:04
kfox1111but it happens before anythign else.22:04
sbezverkjascott1: imho you should use already existing format for something similar you try to achieve and not invent yet another option22:04
kfox1111or else ()'s are needed for grouping.22:04
kfox1111don't know I guess.22:05
kfox1111just thinkg, you might want:22:05
kfox1111(global.a,global.b)&(global.c)22:05
kfox1111first defined of a/b's boolean,22:05
kfox1111anded with value c.22:05
kfox1111this is kind of why I was suggesting just to use gotl.22:07
kfox1111it gets complicated fast. :/22:07
kfox1111though gotls lang has its own uglies. :/22:07
kfox1111but they are known and accepted already.22:08
*** lrensing has quit IRC22:09
openstackgerritSteven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm  https://review.openstack.org/40572022:09
sdakebmace_ give thtat a go22:09
bmace_kk, i'll take a look22:10
kfox1111sdake: the kolla namespace has to be explicitly created.22:11
sdakekfox1111 gotta pick up kids from school brb22:11
*** srwilkers has quit IRC22:13
bmace_sdake: not clear how anything in that change set would fix anything.. nothing changed in dev_workflow.sh where the failure is?22:13
*** goldyfruit has quit IRC22:16
*** rhallisey has quit IRC22:18
sbezverkkfox1111: can you print what gets returned by ovs command you used?22:19
kfox1111yeah.22:19
kfox1111wow. all green, first time. its been a while. :/22:21
kfox1111really need to get back there asap.22:22
sdakebmace_ which failure do you get22:22
kfox1111sdake: https://review.openstack.org/#/c/424921/ please.22:22
bmace_sdake:  from dev_workflow.sh run: Unable to locate iscsid directory in search_paths=[/etc/kolla, /etc/kolla-kubernetes, /opt/kolla/etc/kolla, /opt/kolla-kubernetes/etc/kolla-kubernetes]22:22
sdakebmace_ ya - that is because >> is coiming from ceph_all_in_one instead of iscsi_all_in_one22:23
sdakepretty sur thats in that path but maybe i fat fingered it22:23
kfox1111sdake: didn't see it in the ps.22:24
sdakeok will look - can't do 2 things at the exact same itme22:24
sdakeso pick which one you gys want first :)22:24
sdakelet me have a look at the dev workflow since bmace has been waiting on it all day22:24
kfox1111of corse, I will vote for my review. ;)22:24
kfox1111trying hard to get the gate stable again.22:25
kfox1111but dev env is important too.22:25
kfox1111k22:25
sbezverkkfox1111: please add this command for me22:28
sbezverkovs-dpctl show22:28
*** rwellum has joined #openstack-kolla22:29
sbezverkjust output no check or anything22:29
kfox1111in another loop? at the end?22:29
sbezverkkfox1111: before the loop22:29
sbezverknothing else22:29
openstackgerritSteven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm  https://review.openstack.org/40572022:30
bmace_kk, i see the change.  i'll make it in my container and try it out22:30
openstackgerritSteven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm  https://review.openstack.org/40572022:31
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch.  https://review.openstack.org/42537622:31
sdakebmace_ there is our final work22:31
sdakebmace_ or semi-final - that got us running22:31
kfox1111sdake: looks good to me.22:31
kfox1111sdake: did you get it fully working?22:31
sdakekfox1111 our env was misconfigured from kubeadm22:32
sdakekfox1111 so not certian although all containers were in an operational state22:32
bmace_sdake: i'm betting line 17 should be iscsi and not ceph also?22:32
sdakebmace_ i'm not sure the script needs more work22:32
sdakeit needs to be made idempotent for one22:32
kfox1111heh.22:32
kfox1111not sure thats a good idea.22:33
kfox1111well, idempotent is a good idea,22:33
* sdake has flashbacks of kolla pre compoe days22:33
sdakecompose days22:33
*** jheroux has quit IRC22:33
kfox1111but not trying to make the current workflow scripts that way. :/22:33
kfox1111yeah. it was intended just to get it workign once from scratch to working and give up if it fails.22:33
sbezverkkfox1111: I think line 89 is missing ';'22:33
kfox1111fault tolerance would be a huge pain.22:33
sdakekfox1111 i get it22:33
kfox1111sbezverk: doh. your right.22:34
sdakekfox1111 ok ie had a bit of a rough day so I gotta jet for a bit22:34
sdakebmace_ remember the key ther eis to uninstall iscsi-initiatoior on your host22:34
kfox1111k.22:34
kfox1111sdake: hope you didn't use my name in vain too often. ;)22:34
sdakenot a tall22:34
sdakethanks for getting it started ;)22:34
sdakethere was alot of confusion about why _ vs - in command namees22:35
kfox1111thanks for working on it. :)22:35
sdakeand why some things are in test and some in tools22:35
sdakeanyway - I'll be documenting how to do it without a script22:35
sdakeand we can tune from there22:35
kfox1111'why' implies perhaps some concious decision.22:35
bmace_sdake: see my comment about line 17? i can make it in the review also, but almost sure that should be iscsi rather than ceph now22:35
kfox1111the idea behind tests/tools was tests were scripts really intended just for the gate,22:35
* sdake wants to ptfo22:35
kfox1111while tools were more generally useful.22:36
kfox1111not sure it fell out that way in reality though.22:36
sdakebmace_ which file22:36
sdakebmace_ i'll look22:36
jascott1sbezverk +1 for reusing existing format (and pkg)22:36
bmace_the one you recently uploaded.  setup_config.sh22:36
kfox1111oh...22:37
openstackgerritMerged openstack/kolla-ansible: Fix bootstrap.yml in Bifrost  https://review.openstack.org/42427122:37
sdakegood call bmace22:37
kfox1111sbezverk: I think the ; thing might be why its not starting too.22:37
sdakealthough it works with ceph version :)22:37
kfox1111another missing ';'22:37
sdakebmace_ double check that file is in the repo22:37
openstackgerritMerged openstack/kolla: Add ceph fs containers  https://review.openstack.org/42295022:37
kfox1111inc0: thanks. :)22:38
sbezverkkfox1111: two more22:38
kfox1111yeah.22:38
sdakebmace_ indeed it is22:38
bmace_tests/bin/setup_config.sh?  it was part of your last change22:38
inc0np kfox111122:38
sdakebmace_ i didn't test with it that way22:38
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch.  https://review.openstack.org/42537622:38
bmace_sdake: well, it would be really odd to use the ceph all in one kubernetes config when using iscsid22:38
bmace_i'm assuming it is wrong, but i'll try it my way and make sure22:39
sdakebmace_ well it should be iscsi, what i'm getting at it I am out of steam to test it22:40
kfox1111yeah. I think iscsi's config shoudl be used.22:40
sdakebut I'll push an update if you test it that way22:40
bmace_kk, will take me a few to get my kubeX env back in shape, then i'll try it22:40
*** sayantani01 has quit IRC22:40
*** sayantani01 has joined #openstack-kolla22:41
openstackgerritSteven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm  https://review.openstack.org/40572022:41
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Implement Keystone zero-downtime upgrade  https://review.openstack.org/42544622:46
inc0hey, can someone deploy ansible-ubuntu-source?22:46
inc0my mariadb is acting up on fresh deploy and I'm scared by it22:46
kfox1111inc0: trigger a gate job test?22:48
inc0gates are fine, but gates aren't really multinode22:48
inc0not in ansible22:48
*** mgoddard has quit IRC22:51
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Implement Keystone zero-downtime upgrade  https://review.openstack.org/42544622:51
kfox1111ah. ok.22:52
*** dave-mcc_ has joined #openstack-kolla22:52
*** dave-mccowan has quit IRC22:54
inc0ok...that's probably local to me22:54
inc0vms aren't behaving consistently22:55
openstackgerritEduardo Gonzalez proposed openstack/kolla: Implement Keystone zero-downtime upgrade  https://review.openstack.org/38854422:55
*** vhosakot has joined #openstack-kolla22:57
*** huzhengchuan has quit IRC22:59
kfox1111oh my gosh yes!22:59
kfox1111https://blueprints.launchpad.net/horizon/+spec/reorganise-access-and-security22:59
kfox1111that crazy ui is finally going away22:59
* kfox1111 jumps up and down and dances22:59
*** l4yerffeJ_ has joined #openstack-kolla23:03
*** l4yerffeJ has quit IRC23:03
*** dave-mccowan has joined #openstack-kolla23:05
*** dave-mcc_ has quit IRC23:05
*** rushil has quit IRC23:07
*** esmiurium has quit IRC23:09
sdakegates are fine23:13
sdakefinedog ftw23:13
*** TxGirlGeek has quit IRC23:17
*** esmiurium has joined #openstack-kolla23:18
*** krtaylor has quit IRC23:18
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds  https://review.openstack.org/41855023:18
*** rushil has joined #openstack-kolla23:19
kfox1111sbezverk: http://logs.openstack.org/76/425376/5/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/77a2ba7/logs/pods/kolla-neutron-openvswitch-agent-network-f4h0q-deps-and-update-permissions.txt23:19
kfox1111interesting...23:20
sbezverkkfox1111: yeah it looks like it does not see pid file23:21
kfox1111 /var/run is mounted though.23:21
kfox1111looks like it may need privilege too.23:21
sbezverkkfox1111: we need to confirm if pid file gets created23:22
kfox1111do you have a running container we can poke at?23:22
sbezverkone sec, I recently killed my ovs, let me try to deploy service23:23
kfox1111on my bare metal host runnig ovs on the host, it does.23:23
kfox1111I'm guessing if its looking for the pid file, it will be looking up the pid too.23:24
kfox1111maybe we need to set the pid namespace on it to host too.23:24
kfox1111though thats a pod level thing. :/23:25
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch.  https://review.openstack.org/42537623:25
*** newmember has quit IRC23:25
sbezverkkfox1111: give me 5 minutes I think I will get ovs up and running23:25
kfox1111k23:25
*** newmember has joined #openstack-kolla23:25
kfox1111oh... interesting...23:26
kfox1111its getting the pid out of the pid file,23:26
kfox1111then opening /var/runn/openvswitch/ovs-vswitchd.<PID>.ctl socket.23:27
kfox1111so we shouldn't need pid namespace in that case....23:28
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch.  https://review.openstack.org/42537623:28
sdakerwellum http://docs.openstack.org/developer/kolla-kubernetes/development-environment.html23:28
*** vhosakot has quit IRC23:29
*** rushil has quit IRC23:33
rwellumsdake, thanks for the link!23:35
sdakerwellum np :)23:38
sdakeservice layer looking pretty good :)23:40
kfox1111sbezverk: https://review.openstack.org/#/c/424368/ should be ready23:41
kfox1111sdake: wait till you see it wrapped up in a computekit package. :)23:42
* inc0 is scared23:42
sbezverkkfox1111: docker exec -tu root -ti f0e12b3057b3 ls /var/run/openvswitch23:42
sbezverkbr-ex.mgmt  br-ex.snoop  db.sock  ovs-vswitchd.14.ctl  ovsdb-server.1.ctl23:42
*** lamt has quit IRC23:42
kfox1111sbezverk: weird. no pid file...23:43
sdakeinc0 the workflow is pretty good, although super pokey23:43
kfox1111maybe there is a missign arg?23:43
sdakealthough nobody cares about deploy speed, its operational performance that matters23:43
inc0sdake, scared because my kolla-ansible deploy is broken and I have no idea whats happening23:43
sbezverkkfox1111: 424368 ever succeeded for entrypoint?23:43
sdakeinc0 oh gotcha23:43
inc0can you try deploying master23:43
kfox1111I suspect the shell workflow will be faster then the helm one.23:43
inc0?23:43
sdakeinc0 i am leaving for dinner with my parents in 10 minutes - so not really23:44
sdakeinc0 possibly when i get back23:44
kfox1111as it won't bother with loading stuff that isn't needed at the time until later. we'll see though.23:44
inc0I'd appreciate it23:44
sdakeinc0 can you be more specific what is busted23:44
inc0mariadb doesn't bind to a port23:44
sdakeinc0 deploy - reconfigure - ugprade ?23:44
kfox1111sbezverk: yeah.23:44
sdakeinc0 3 nodes or aio?23:44
inc0and first error I'm getting is "connection reused"23:44
inc03 nodes23:44
sdakeiptables?23:44
sdakerefused?23:45
sdakeor reused :)23:45
inc0well it should bind to port23:45
inc0refused23:45
*** sayantani01 has quit IRC23:45
inc0ss -plant doesn't show it23:45
sdakei'v  seen that in days past23:45
inc0or rather, not consistently23:45
*** sayantani01 has joined #openstack-kolla23:45
kfox1111uh... weird...23:46
kfox1111grep vswitchd /usr/lib/systemd/ -r #returns nothing.23:46
kfox1111oh.. they are doing something quite different...23:47
kfox1111ovs-ctl.23:47
kfox1111so, we may need --pidfile=/var/run/openvswitch/ovs-vswitchd.pid23:52
kfox1111it says its default, but doesn't seem to be adding it.23:52
kfox1111and its explicit in redhat's running version.23:52
*** rushil has joined #openstack-kolla23:53
sbezverkkfox1111: I see another issue on my cluster :( entrypoint doesn ot see socker file23:55
kfox1111hmm?23:55
kfox1111the ovs one?23:55
kfox1111that is really strange, cause that seems to be working in the gate. and there's no conditional around it.23:56
sbezverkkfox1111: I can see the file exist23:56
kfox1111its blocking on it though?23:57
*** lamt has joined #openstack-kolla23:58
kfox1111hmm.. do we have reqdeps and initialize-ovs-vswitchd reversed in the vswitchd template?23:58
kfox1111shouldn't the socket exist before it tries to tweak the db?23:58
kfox1111maybe its just fail/retrying until it works. but less then ideal...23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!