Saturday, 2017-02-11

*** salv-orlando has quit IRC00:02
bmacesbezverk: you happen to know off the top of your head why one of the k8s pods would have an issue mounting /dev/rdb0?  i checked on kube2 and it is there, and i verified the pod was assigned to kube200:02
bmacesbezverk: i see in the describe pod output that it says the device doesn't exist :/00:03
sbezverkbmace: are you using dynamic pvc by any chance?00:03
bmacesbezverk: hrm... idk?  where do i look for that..00:03
sbezverkbmace: if you do then you must use att's kube controller-manager image00:04
bmacesbezverk: this was all working fine like 3 days ago :/00:04
sbezverkbmace: oh, I see00:04
sbezverkplease check on ceph cluster if there is no stale lock entries00:05
bmacesbezverk: then the gate changes happened and i haven't had a clean run since the merge.00:05
bmacesbezverk: any hints to where i should look?00:05
sbezverk rbd ls --pool volumes00:07
bmacesbezverk: guess i'll need to get some tools set up somewhere to help with ceph debugging.00:07
sbezverkand then check each volume00:07
sbezverkrbd lock list glance --pool volumes00:08
sbezverkbmace: you need to run on monitor node00:08
sbezverksorry --pool volumes you do not need to specify00:08
sbezverkI have to do it because  I do not use default pool00:09
bmacesbezverk: showmapped returns rabbitmq at rbd0 and mariadb at rbd100:14
bmacesbezverk: which is odd, since it seems like it was trying to mount rbd0 into mariadb-init00:14
*** crushil has quit IRC00:15
openstackgerritOpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements  https://review.openstack.org/43196100:16
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements  https://review.openstack.org/43196200:16
bmacesbezverk: ls for pool "kollavolumes" shows glance, helm-repo, mariadb and rabbitmq00:16
*** huzhengchuan has joined #openstack-kolla00:18
openstackgerritBorne Mace proposed openstack/kolla-kubernetes master: WIP: Added support for containerized dev bringup  https://review.openstack.org/42929500:18
sbezverkbmace: ok if volume is locked, then no new pods will be able to use it. You need to clear the lock manually00:19
*** sacharya has quit IRC00:20
bmacesbezverk: where would it show if it is locked?00:20
bmacesbezverk: i think i found it.. lock list00:20
bmacesbezverk: lock nuked...00:22
bmacesbezverk: as part of my process i always rebuild my ceph cluster from scratch so i don't know why anything odd would be happening now since the gate changes vs for the last couple of weeks when i never hit this issue00:24
kfox1111sbezverk: weird.00:26
kfox1111bmace: your review is looking really good to me.00:26
bmacekfox1111: thanks, other than the fact that it doesn't work anymore it is awesome :)00:27
kfox1111heh. yeah.00:27
kfox1111:/00:27
sbezverkbmace: not sure how gate change could change just your environment :(00:27
kfox1111thats the reason I wanted to figure out how to get everyting, or at least most of it into the gate.00:27
kfox1111to prevent future breakages of the dev env. :/00:27
sbezverktry to roll back and run without gate patch sets..00:27
bmacesbezverk: at this point i already merged and made some changes on top of the gate patch sets.00:28
bmaceand i feel pretty confident it isn't any personal env oddness since i re-build it every time and mark was hitting the same issue00:28
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Adding compute kit based on iscsi backend  https://review.openstack.org/43115600:29
bmaceit does rely on the ceph_workflow_service script, which was changed with the gate changes, though just from looking at things visually i don't see why it would cause breakage.. but once you remove every other possibility what you are left with must be the case.. so, i am just going through and removing possibilities at this point.00:30
sbezverkbmace: in this case we should probably get together on webex and try to figure out..00:30
sbezverkkfox1111: would you have some time tomorrow PM PST to troubleshoot ?00:32
bmacesbezverk: i certainly can.  if you or kfox1111 has a functioning halcyon-vagrant env it is crazy easy to bring up too.  with my latest upload even proxy settings happen automagically.00:32
kfox1111sbezverk: probably00:32
kfox1111I can't get vagrent to work.00:32
kfox1111I don't use virtualbox,00:33
sbezverkbmace: neither could I00:33
kfox1111and every time I try and use it with kvm, it makes me mad. :/00:33
*** fragatina has joined #openstack-kolla00:33
bmacei don't use vbox either, for this anyway, running it w/ libvirt00:33
kfox1111oh. nice. :)00:33
bmaceand mark is running w/ virtbox.. so we had the change set from like 3 days ago working all the way through with both00:33
*** crushil has joined #openstack-kolla00:34
kfox1111very nice. so, we're really close then. just gotta figure out why...00:36
kfox1111sbezverk: not seeing much in that change set that coudl break something...00:37
kfox1111unless I'm off by one.00:38
kfox1111looking at: git diff 7b5e5591fe7108117221eacfffe6e7edbd33ddfa...657fd6de1f1171eb2d2db7bab2839d7a4940b8eb00:38
kfox1111does 657fd6de1f1171eb2d2db7bab2839d7a4940b8eb work and the following patch fail, or is 657fd6de1f1171eb2d2db7bab2839d7a4940b8eb the one that breaks?00:38
*** Pavo has joined #openstack-kolla00:40
kfox1111the following patch is cloudkitty related. hmm...00:40
sbezverkthat is what bisect tells me00:40
sbezverkgit bisect good 6b37f73ace9f7aa3aa099308dcde280f63cc0e5a00:41
sbezverkBisecting: 0 revisions left to test after this (roughly 0 steps)00:41
sbezverk[657fd6de1f1171eb2d2db7bab2839d7a4940b8eb] Modify the openstack_release in globals.yml00:41
sbezverkkfox1111: does not work00:41
kfox1111hmm...00:43
sbezverkkfox1111: try for yourself, checkout 657 and see it fail00:43
kfox1111I believe you. just tyring to figure out why.00:44
kfox1111uh... 657fd6de1f1171eb2d2db7bab2839d7a4940b8eb says its from dec 23?00:45
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: DO NOT MERGE git bisect for b3 failure  https://review.openstack.org/43240800:45
kfox1111seems too long ago.00:45
*** fragatina has quit IRC00:46
bmacesbezverk: just trying to follow the gate right now to hunt down my issues.. i don't get setup_gate_seph.sh lines 137 / 139.  it looks like it is passing 3 args into those scripts but neither use the 3rd arg.. it is a problem because my change would introduce optional 3rd / 4th args00:47
kfox1111what were the start/endpoints you used?00:47
*** Pavo has quit IRC00:47
kfox1111bmace: it use to...00:49
kfox1111oh... maybe that was the trunk build ps that hasn't merged yet.00:49
bmacekfox1111: that uses the args, or it remves them? or you talking to sbezverk ? :)00:50
kfox1111I think the trunk ps I have in flight might use the 3rd arg.00:50
kfox1111I don't remember for sure though.00:51
bmacekfox1111: is there a review up for it?00:51
kfox1111https://review.openstack.org/#/c/418550/00:52
bmacekfox1111: that doesn't touch either of the ceph files, so i can't see how it would make the args useful in them.00:53
kfox1111ok.00:54
kfox1111I was misremembering the.00:54
kfox1111then.00:54
bmacethat is fine.  i'll put up a review to yank the args.. it is only more confusing to have them around00:55
kfox1111just the third arg?00:55
bmaceyes00:55
kfox1111k. sounds good.00:55
bmacei said args plural only because it is an extra arg to each script :)00:56
kfox1111ok. just was double checking. :) english is not the most precice of languages at times... :)00:56
bmaceagreed completely00:56
openstackgerritBorne Mace proposed openstack/kolla-kubernetes master: Remove no longer relevant argument to ceph_workflow*  https://review.openstack.org/43250900:59
*** adrian_otto has quit IRC01:00
*** sdake has joined #openstack-kolla01:02
kfox1111sbezverk: when you did the bisect, do you remember your start/end points?01:16
*** mannamne has quit IRC01:17
kfox1111sbezverk: 657fd6de1f1171eb2d2db7bab2839d7a4940b8eb seems too old.01:19
kfox1111its tagged fri dec 23.01:19
kfox1111but we unpinned the gate and had it working on feb5.01:19
*** ssurana has joined #openstack-kolla01:19
kfox1111its possible 657fd6de1f1171eb2d2db7bab2839d7a4940b8eb is broken for our use case, but I think the culprit is elsewhere.01:20
kfox1111 identified the issue on feb 8.01:20
kfox1111so we should be bisecting between those two dates.01:20
kfox1111I'd start at: e516ea509940113175cce24382c64d78a2fa42e3 and end at 6b47d8284b25bcbdc814878494a7ef005b66cffc01:21
sbezverkkfox1111: start was this one: 627719e66cae10d9673b786516b4a93ed65fc6cb01:22
sbezverkend was current master of kolla-ansible01:23
kfox1111ah. dec 1501:23
kfox1111yeah. so its possible they broke it, fixed it, and broke it again later. :/01:23
kfox1111I'll try bisecting too. maybe betwween the two of us we can find the smoking gun. :/01:26
kfox1111what a pain. :/01:26
*** MasterOfBugs has quit IRC01:26
*** pramodrj07 has quit IRC01:27
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: DONTMERGE: git bisecting a kolla-ansible issue.  https://review.openstack.org/43251201:28
*** sdake has quit IRC01:29
*** sdake has joined #openstack-kolla01:29
bmacesbezverk: i think i may have found it.. going to rebuild and try it out now.  it was some of the changes to the ceph workflow / ceph config stuff to break out common_ceph_config.sh into a new script.  no args were being passed so ceph-multi couldn't be set to override the hard coded addr01:31
bmacesbezverk: damn, actuall, don't think that is it :/01:31
kfox1111bmace: what behavior are you seeing? maybe I can help narrow it down.01:33
kfox1111(gotta leave in a few minutes though)01:33
bmacekfox1111: need to bring it up again, but it was an error mounting /dev/rdb0 in the mariadb-init pod.. though on the node the pod is spinning up on and it is there01:34
kfox1111hmm....01:34
kfox1111maybe an issue with the formattign of the rbd volume?01:34
*** sdake has quit IRC01:34
bmacekfox1111: the only odd thing i noticed was in my ceph image list it seemed to think rabbitmq should be rdb0 and mariadb should be rdb1 and mariadb-init was trying to mount the /dev/rdb001:35
bmacekfox1111: not sure why that would stop working after that all has been fine for like a week and a half though :/01:35
kfox1111which script are you calling?01:36
kfox1111ceph_workflow or ceph_workflow_services?01:36
kfox1111looks like the service varient.01:37
bmaceworkflow_service01:37
sbezverkgents I need to run, will check with you later01:37
kfox1111so... I warned sbezverk of this. looks like you hit it too...01:37
kfox1111there's nothing tieing a pv's name to a pvc's name.01:37
kfox1111pv's are ment to be a pool of stuff to pull randomly from,01:38
kfox1111and pvc's ask for "give me one".01:38
*** Pavo has joined #openstack-kolla01:38
kfox1111so in the case of ceph_workflow_service, it looks like it just throws up a bunch of services at the same time,01:38
bmacekfox1111: so you can get sort of randomly lucky / unlucky by naming them?01:38
bmaceyeah01:38
kfox1111possibly creating a two pv's at the same time, followed by 2 pvc requests.01:38
kfox1111and get them in random order. yeah.01:38
kfox1111it only works reliably if you have no other admins on the k8s system, no free pv's,01:39
kfox1111create the one pv, and then create the pvc right away.01:39
bmacekfox1111: very interesting... unexpected since it was working well and consistently for a long time01:39
kfox1111then the names will match.01:39
kfox1111its probably moved back and forth randomly, but you didn't look close enoguh to notice.01:40
kfox1111as it doesn't really have any effect but being puzzling when you look at what pv its actually bound to and finding a weeird name.01:40
kfox1111so, just a weird thing to ignore for now while debugging the real issue.01:41
kfox1111I really think we shoudl just drop the pv packages and put in the docs, "if you want to use non dynamic pvc creation, create some pv's like.... point to k8s docs"01:42
kfox1111gtg. have a good one01:43
*** unicell has quit IRC01:44
*** zhubingbing has joined #openstack-kolla01:45
*** Pavo has quit IRC01:51
bmacekfox1111: maybe that has something to do with this somehow... everything seems to be coming up ok this time.. so far.. fingers crossed.  mariadb is up anyway01:52
openstackgerritBorne Mace proposed openstack/kolla-kubernetes master: Added support for containerized dev bringup  https://review.openstack.org/42929501:54
*** rhallisey has quit IRC01:55
*** Pavo has joined #openstack-kolla01:58
*** qiliang27 has joined #openstack-kolla02:05
*** financeMan has quit IRC02:12
*** haplo37 has quit IRC02:13
*** sacharya has joined #openstack-kolla02:14
*** haplo37 has joined #openstack-kolla02:22
*** fragatina has joined #openstack-kolla02:24
*** qiliang27 has quit IRC02:27
*** sdake has joined #openstack-kolla02:28
*** sdake_ has joined #openstack-kolla02:29
sdake_evening folks02:31
*** sdake has quit IRC02:32
bmacesdake_: evening02:34
bmacesdake_: other than the missing doc write up my devenv review is ready for feedback.. i think there is some odd pv / pvc timing that can randomly cause it (and perhaps gate also) failures02:35
bmacesdake_: but most of the time it seems pretty happy :)02:35
*** fragatina has quit IRC02:39
*** fragatina has joined #openstack-kolla02:40
*** qiliang27 has joined #openstack-kolla02:41
*** sdake has joined #openstack-kolla02:44
*** saneax-_-|AFK is now known as saneax02:46
*** sdake_ has quit IRC02:48
*** Pavo has quit IRC02:54
*** crushil has quit IRC02:56
*** fragatina has quit IRC03:00
*** mgiles has quit IRC03:12
sdakebmace hotness03:15
sdakebmace sorry for delay - dental appointment today and struggling to focus :(03:16
sdakebmace if you have some basic instructions i'd be happy to test it on my gear03:16
*** sdake has quit IRC03:20
*** sdake has joined #openstack-kolla03:21
*** qiliang27 has quit IRC03:22
*** inc0 has quit IRC03:26
*** vcn has joined #openstack-kolla03:36
*** vcn has quit IRC03:37
bmacesdake: the basic instructions are in my last comment on the review.. just get a halcyon-vagrant env up and running.. i can toss up my config.rb on pastebin for you and my Vagrantfile since i tweaked it to get kube2 to have 4gb of memory..03:41
bmacesdake: then just run build_dev_image.sh.. then run_dev_image.sh.. and happiness :)03:41
sdakebmace cool03:41
sdakebmace i will try in about 20 minutes if you will still be around for support03:42
sdakeif not, enjoy your friday :)03:42
bmacesdake: http://paste.openstack.org/show/598475/  there are my diffs from base halcyon-vagrant-kubernetes03:42
bmacesdake: i'll be in and out.. feel free to toss any questions up and i'll look at them when i can03:43
bmacei'm running w/ libvirt.. if you are using virtbox your changes would be slightly different.03:43
*** mattmceuen has joined #openstack-kolla03:55
sdakei'm running with libvirt on centos 7.something04:07
sdakei'll let you know in 30 mins when I get the env setup from a fresh clean checkout04:07
sdakeif your sitll around - if not i'll see ya when you are :)04:07
*** fragatina has joined #openstack-kolla04:10
sdakebmace apparently i'll be interrupted slightly - be back in 20 mins to colelct food for a family member04:12
bmacesdake: kk, btw, i learned a lot about snapshotting qcow images w/ libvirt today and so once you get a good vagrant up finished you never have to do it again unless versions of libraries change and you want to start from scratch :)04:12
sdakebmace i like checking out master to make sure things are not broken04:12
sdakealso the version of kubernetes we use needs updates occasionallly04:12
*** fragatina has quit IRC04:15
*** salv-orlando has joined #openstack-kolla04:37
bmacesdake: i don't plan on using them permanently.. but this devenv testing had me doing a clean start like 5x / day... just resetting the disks is a lot better :)04:37
*** salv-orlando has quit IRC04:41
*** fragatina has joined #openstack-kolla04:42
*** fragatina has quit IRC04:43
*** fragatina has joined #openstack-kolla04:44
*** huzhengchuan has quit IRC04:55
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Adding compute kit based on iscsi backend  https://review.openstack.org/43115604:57
*** fragatina has quit IRC04:58
sbezverksdake: if you still around, can you point me to ps which fixed "no docker module" issue05:04
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: DO NOT MERGE git bisect for b3 failure  https://review.openstack.org/43240805:09
*** sdake_ has joined #openstack-kolla05:14
*** tvignaud has quit IRC05:17
*** sdake has quit IRC05:18
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Adding compute kit based on iscsi backend  https://review.openstack.org/43115605:35
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: DO NOT MERGE git bisect for b3 failure  https://review.openstack.org/43240805:38
*** sdake_ has quit IRC05:38
*** sacharya has quit IRC06:24
*** unicell has joined #openstack-kolla06:35
*** salv-orlando has joined #openstack-kolla06:39
*** kong has quit IRC06:39
*** crushil has joined #openstack-kolla06:40
*** salv-orlando has quit IRC06:43
*** unicell1 has joined #openstack-kolla06:45
*** unicell has quit IRC06:46
*** saneax is now known as saneax-_-|AFK06:47
*** crushil has quit IRC07:01
*** rwallner has joined #openstack-kolla07:25
*** rwallner has quit IRC07:29
*** unicell1 has quit IRC07:47
*** ssurana has quit IRC08:03
*** jmccarthy has joined #openstack-kolla08:04
*** sdake has joined #openstack-kolla08:04
*** rwallner has joined #openstack-kolla08:24
*** rwallner has quit IRC08:29
*** sdake has quit IRC08:30
*** tonanhngo has quit IRC08:31
*** sdake has joined #openstack-kolla08:38
*** salv-orlando has joined #openstack-kolla08:40
*** salv-orlando has quit IRC08:44
*** sdake has quit IRC08:50
*** mgoddard has joined #openstack-kolla08:51
*** mannamne has joined #openstack-kolla08:53
*** mgoddard has quit IRC09:00
*** hieulq_ has joined #openstack-kolla09:02
*** mgoddard has joined #openstack-kolla09:04
*** mgoddard has quit IRC09:09
*** jmccarthy has left #openstack-kolla09:14
*** mgoddard has joined #openstack-kolla09:14
*** mgoddard has quit IRC09:23
*** hieulq_ has quit IRC09:24
*** hrw has quit IRC09:38
*** hieulq_ has joined #openstack-kolla09:38
*** hieulq_ has quit IRC09:42
*** hrw has joined #openstack-kolla09:44
hrwhttps://review.openstack.org/#/c/430940 gets heavy testing. aarch64/debian went to 124 built, 83 failed. aarch64/ubuntu is in progress. x86-64/debian in progress (with non-debian repos disabled to compare with aarch64), x86-64/ubuntu in progress (also only ubuntu repos for comparison). anyone with spare review time?10:16
*** jmccarthy has joined #openstack-kolla10:29
*** tonanhngo has joined #openstack-kolla10:32
*** tonanhngo has quit IRC10:34
*** jrist has quit IRC10:42
*** skuda has joined #openstack-kolla10:49
*** Jeffrey4l_ has joined #openstack-kolla11:00
*** Jeffrey4l__ has quit IRC11:04
*** salv-orlando has joined #openstack-kolla11:15
*** salv-orlando has quit IRC11:20
openstackgerritSaverio Proto proposed openstack/kolla master: Add SNAP support when finding Docker images folder  https://review.openstack.org/42713911:29
*** sudipto has joined #openstack-kolla11:39
*** sudipto_ has joined #openstack-kolla11:39
*** mannamne has quit IRC11:44
*** sdake has joined #openstack-kolla11:57
sdakesbezverk i know of no docker module fix11:58
sdakeJeffrey4l_ are you aware of a kolla-ansile docker module fix as of late?11:58
*** pbourke has quit IRC12:08
*** pbourke has joined #openstack-kolla12:08
*** sudipto_ has quit IRC12:16
*** sudipto has quit IRC12:16
*** sdake_ has joined #openstack-kolla12:20
*** sdake_ has quit IRC12:21
*** sdake has quit IRC12:21
*** sdake_ has joined #openstack-kolla12:21
Jeffrey4l_sdake_, sorry? which docker module?12:31
sdake_Jeffrey4l_ sbezverk said there is some change to make docker.py work - i am not sure what he is tlaking about12:50
*** sdake_ has quit IRC12:50
Jeffrey4l_ok. i will sync with him.12:51
*** huzhengchuan has joined #openstack-kolla12:57
*** sdake has joined #openstack-kolla12:59
sdakeJeffrey4l_ anssible [05:50:28]  <sdake_>Jeffrey4l_ sbezverk said there is some change to make docker.py work - i am not sure what he is tlaking about13:00
*** jmccarthy has left #openstack-kolla13:22
*** sdake has quit IRC13:39
*** financeMan has joined #openstack-kolla13:48
*** skuda has quit IRC13:49
financeManHi As I stated yesterday I am installing kolla on a 64core 256 gb ram sever play computer I made for my family.  I have next to 0 background in openstack and come from software engineering background.13:49
*** jmccarthy has joined #openstack-kolla13:49
financeManIt was suggested to me that I follow the http://docs.openstack.org/developer/kolla/newton/quickstart.html to a T13:50
financeManSo my ignorance might help make it a little better ^_^13:50
financeManThe first thing it basically says is to run "kolla-ansible -i <<inventory file>> bootstrap-servers"13:50
financeManbut it doesn't say how to install kolla-ansible to start with...13:50
financeManI have a fresh bare-bones ubuntu 16.04 instance on the big server13:50
financeManI have never heard of kolla or ansible but I understand ansible is basically a fancy script system... But the very first section "Automatic Host Boostrap" section doesn't have much context and it just seems thrown in there13:52
financeManI cant tell how much it will do, will it do everything in the document? I am not really sure what running that command will do and if I have to do it before moving on in the doc13:52
financeManThe fact that later down in the document I see a "apt-get install ansible" would imply to me that that command is not something I run right away but rather a sneak peak of whats to come or something.13:53
*** Pavo has joined #openstack-kolla13:54
financeManI really want this "Automatic host bootstrap" section explained to me before I move on with the doc.  It really makes it sound like if I install ansible on the side and run "kolla-ansible -i <<inventory file>> bootstrap-servers" it will magically do everything in the document fo rme.13:54
*** sdake has joined #openstack-kolla13:57
*** sbezverk has quit IRC14:09
*** sdake has quit IRC14:21
*** sdake has joined #openstack-kolla14:21
*** eaguilar has joined #openstack-kolla14:26
*** sdake_ has joined #openstack-kolla14:27
*** Pavo has quit IRC14:28
*** sdake has quit IRC14:29
*** sdake_ has quit IRC14:32
*** jmccarthy has left #openstack-kolla14:32
*** sdake has joined #openstack-kolla14:33
*** sdake has quit IRC14:33
*** tonanhngo has joined #openstack-kolla14:34
financeManAlso, can you help me with this step -> http://pastebin.com/TQgA7xUW14:34
*** sdake has joined #openstack-kolla14:34
financeManI am not trying to 'guess' or even infer the slightest so as to make sure the guide is very straight forward without assumptions.14:35
*** tonanhngo has quit IRC14:35
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Adding compute kit based on iscsi backend  https://review.openstack.org/43115614:35
*** sbezverk has joined #openstack-kolla14:35
sbezverkJeffrey4l_: ping14:36
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: DO NOT MERGE git bisect for b3 failure  https://review.openstack.org/43240815:01
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Adding compute kit based on iscsi backend  https://review.openstack.org/43115615:16
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: DO NOT MERGE git bisect for b3 failure  https://review.openstack.org/43240815:39
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Adding compute kit based on iscsi backend  https://review.openstack.org/43115616:05
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Adding compute kit based on iscsi backend  https://review.openstack.org/43115616:06
*** abc__ has quit IRC16:15
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: DO NOT MERGE git bisect for b3 failure  https://review.openstack.org/43240816:24
*** Pavo has joined #openstack-kolla16:27
*** abc has joined #openstack-kolla16:27
*** eaguilar has quit IRC16:29
*** jmccarthy has joined #openstack-kolla16:37
*** kbyrne has quit IRC16:40
*** adrian_otto has joined #openstack-kolla16:51
*** negronjl has quit IRC16:52
*** klindgren_ has quit IRC16:52
*** adrian_otto has quit IRC16:52
*** Pavo has quit IRC16:55
*** sdake has quit IRC16:56
*** tonanhngo has joined #openstack-kolla17:01
*** sbezverk has quit IRC17:02
*** tonanhngo has quit IRC17:02
openstackgerritchen.xing proposed openstack/kolla-ansible master: [WIP] Add vitrage ansible role  https://review.openstack.org/43260817:03
*** japestinho_ has quit IRC17:04
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: DO NOT MERGE git bisect for b3 failure  https://review.openstack.org/43240817:12
*** kbyrne has joined #openstack-kolla17:12
openstackgerritchen.xing proposed openstack/kolla-ansible master: [WIP] Add vitrage ansible role  https://review.openstack.org/43260817:13
*** sbezverk has joined #openstack-kolla17:16
*** adrian_otto has joined #openstack-kolla17:39
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: DO NOT MERGE git bisect for b3 failure  https://review.openstack.org/43240817:43
*** adrian_otto has quit IRC17:46
*** g3ek has quit IRC17:49
*** rhallisey has joined #openstack-kolla17:50
*** Pavo has joined #openstack-kolla17:55
*** Pavo has quit IRC17:56
*** g3ek has joined #openstack-kolla17:58
*** tonanhngo has joined #openstack-kolla17:59
*** zhubingbing has quit IRC18:01
*** negronjl has joined #openstack-kolla18:03
*** mdorman has quit IRC18:09
*** mdorman has joined #openstack-kolla18:14
*** sdake has joined #openstack-kolla18:22
sbezverkkfox1111: Here is the commit that breaks b3 f31930ec92c1927db2de260fadd7083eb1c4b3d5 I proved it by running it and then one right before it which was success.18:23
sbezverkkfox1111: if you know how to revert it please let me know. Also compute_kit is know fully functional and ready for review.18:24
*** dave-mccowan has joined #openstack-kolla18:31
bmacesdake: did you ever get around to trying it out?18:38
*** tonanhngo has quit IRC18:49
*** sdake has quit IRC18:56
*** sdake has joined #openstack-kolla18:56
*** adrian_otto has joined #openstack-kolla19:11
*** adrian_otto has quit IRC19:16
*** dave-mccowan has quit IRC19:22
*** adrian_otto has joined #openstack-kolla19:30
*** adrian_otto has quit IRC19:38
*** sdake has quit IRC19:39
*** zhugaoxiao has quit IRC19:50
*** zhugaoxiao has joined #openstack-kolla19:51
*** fragatina has joined #openstack-kolla20:01
*** fragatina has joined #openstack-kolla20:02
*** sdake has joined #openstack-kolla20:04
sdakebmace sup dude20:04
sdakebmace i have not tried the dev env - went to a movie last night instead - have beer with friends in 1 hr20:04
sdakebmace i should be fully tanked and ready to code in about 5 hrs ;)20:05
sdakebmace in the meantime i'll take a look at this regression sbezverk has found20:05
sdakesbezverk that is a mege commit20:08
sdakesbezverk that merge commit is empty20:10
sdakesbezverk do you think it could possibly be some other commit?20:10
sdakesbezverk i'm pretty sure the bisect is not that specific commit20:14
bmacesdake: kk, let me know how it goes :)20:15
bmacesdake: the testing, not the drinking, lol20:15
jascott1sake let me know how the drinking goes20:21
sdakejascott1 if i can type i will20:22
jascott1anyone a rebase expert?20:22
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: DNM Kolla ansible broken  https://review.openstack.org/43269720:24
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: DNM Kolla ansible broken  https://review.openstack.org/43269720:29
sdakejascott1 yup what do you nee20:29
sdaked20:29
jascott1how to git properly properly i guess20:30
sdakescreenshare?20:30
jascott1so, i have feature branch, make commits rebase master and push to my fork.20:30
jascott1now I checkout that branch, do rebase master on it to get latest and then try to push it says rejected20:31
sdakei dont use feature branches - i always rebase on master - but can help out if i can see what you have going on20:31
jascott1its the helm PR20:31
jascott1one sec20:31
sdakewhich one :)20:31
jascott1https://github.com/jascott1/helm/tree/tags_conds20:31
jascott1thats my helm for and feature branch20:31
jascott1for/fork20:31
jascott1i am doing 'git checkout tags_cond' then 'git rebase master' then try to push20:32
jascott1and it gets rejected20:32
sdaketry git rebase --pull20:32
jascott1ah20:32
jascott1thanks will try20:33
sdakethe mysteries of branches elude me with git except for cherrypick operation ;-)20:33
sdakenot sure i have ever rebased a branch20:33
jascott1ya it can get... complicated20:33
sdakeya branches are for releases imo not for features :)20:34
sdakeis helm workflow based around branches?20:34
jascott1ya its standard github flow afaik20:34
sdakegtk20:34
jascott1hmm no pull in rebase20:34
sdakegit pull --rebase?20:35
jascott1it git pull --rebase20:35
sdakesorry :)20:35
jascott1oh np thx for the halp20:35
jascott1argh same results20:35
sdakejascott1 http://stackoverflow.com/questions/7929369/how-to-rebase-local-branch-with-remote-master20:35
jascott1I think my problem is i have already rebased once when I originally pushed PR20:36
sdake git pull --rebase origin master20:36
sdakerebase operation once completed shoudl be a done deal20:36
sdakerun git status and see if your in the middle of  a rebase20:36
sdakegit status will tell you if your in teh midst of an operation right in the first couple of lines of output20:37
sdakeaccording to the stackoverlfow wizard the  git pull --rebase origin master will do what you want20:37
jascott1ya someone also mentions having to do force to push back up20:38
sdakewizard/wizardry - assuming you are not in the midst of a rebase20:38
sdakerun gut status and paste output plz20:38
jascott1im actually decent with git but not too experience with rebasing over and over20:38
jascott1ya im not in rebase but wil paste20:38
sdakeok20:38
sdakecool try that git pull operation above20:39
sdakeno need for paste if git reports your not in a rebase20:39
sdakethat magic will rebase your origin branch (your local working copy) on top of the remote (which is master)20:39
*** bmace has quit IRC20:39
*** bmace has joined #openstack-kolla20:40
sdakei dislike force as a feature of git - however that may be necessary20:40
sdakeinstead of master, you may need kubernetes:master20:41
sdakegit workflow is -ETOOHARD imo20:41
sdakegithub that is20:41
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: DNM kolla-ansible should work here  https://review.openstack.org/43269820:43
jascott1sdake here is what I am trying http://paste.openstack.org/show/598519/20:46
*** sdake_ has joined #openstack-kolla20:47
*** sdake has quit IRC20:49
*** tonanhngo has joined #openstack-kolla20:49
jascott1sdake_ http://paste.openstack.org/show/598519/20:50
*** tonanhngo has quit IRC20:50
sdake_jascott1 what does git pull --rebase produce?20:52
sdake_(pasatewise)20:53
sdake_hint: Updates were rejected because the tip of your current branch is behind20:53
sdake_hint: its remote counterpart. Integrate the remote changes (e.g.20:53
jascott1it came out the same but I will run it20:54
sdake_jascott1 boy the github docs are terrible wrt rebasing and the github workflow20:57
sdake_jascott1 you would think for a co with a 90mil/yr burn rate, they could get that part right for all the money :)20:57
jascott1haha no doubt20:58
jascott1i think pull --rebase is shortcut since I get same output http://paste.openstack.org/show/598520/20:58
jascott1shortcut for pull master, then rebase master20:58
sdake_jascott1 there is always the hammer of --force20:59
jascott1ya20:59
sdake_jascott1 i'm not sure if that will result in a correct repo however20:59
jascott1i know20:59
jascott1last resort21:00
jascott1thinking thru, rebase rewrites history. it says my origin tags_conds tip is ahead of my local but my local has ALL those commits plus those from master21:01
sdake_jascott1 https://randyfay.com/content/rebase-workflow-git21:01
sdake_jascott1 give that a go21:01
sdake_your local is behind master by 30+ commits21:01
sdake_hence the reason you can't issue a  PR I think21:01
sdake_if your local has all the commits - force should work21:02
jascott1i just cant push21:02
jascott1the pr is already there21:02
jascott1yeah I migth just try that21:02
* sdake_ dies a little inside21:02
*** fragatina has quit IRC21:04
jascott1haha trying some help in #git21:04
jascott1if they dont know I will use the FINAL SOLUTION21:04
sdake_i think that blog post looks like a workable solution without the final solution ;-)21:06
sdake_although #git is a good resource if peopel are around21:06
*** ssurana has joined #openstack-kolla21:09
Davieysdake_ / jascott1: Why not just git pull --rebase from master?21:09
*** ssurana has quit IRC21:09
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: DNM Kolla ansible broken  https://review.openstack.org/43269721:13
jascott1one cat is saying force is ok in this scenario but there is a --force-with-lease that I am looking into, guess its safer or something21:13
sdake_Daviey i think we tried that operation however i am unclear if we did what you suggst (the from master part)21:13
sdake_boy bisect in the gate is a huge pita ^^21:13
jascott1i am pushing feature not master21:14
jascott1fyi dude is saying wrt force-with-lease: "it's the same as normal --force, but aborts when the remote has commits you don't have locally"21:15
jascott1so a little safer21:15
jascott1well that worked, we will see if the world exploded21:16
jascott1thanks guys21:16
jascott1^Daviey sdake21:16
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: DNM bisecting kolla-ansible to find regression  https://review.openstack.org/43269721:16
jascott1PR status isnt complaining about conflicts21:17
sdake_jascott1 cool thats a great feature21:17
sdake_TIL :)21:17
jascott1ha i was about to say TIL21:17
jascott1ya working on saturday21:18
jascott1yay21:18
sdake_more of a hobby for me21:18
sdake_I have a day job21:18
*** Pavo has joined #openstack-kolla21:18
jascott1helm release is tues and this is slated to go in21:18
jascott1its already been review and tested but was seeing intermittent test failure that I thought was my fault. supposedly its was an error in master21:19
sdake_bmace the dev env needs a rebase21:19
sdake_jascott1 does helm have any form of ci?21:19
jascott1yeah they use circle ci21:20
jascott1https://circleci.com/gh/kubernetes/helm/2718?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link21:20
jascott1and have local tests which were giving me the intermittent fail21:20
sdake_interesting approach21:21
sdake_i am tapped out on 50$/mo services :)21:21
jascott1ouch21:21
jascott1i hear ya21:21
sdake_i renogotiated my verizon bill from 720 to 36821:21
jascott1dang21:22
sdake_and my cable bill from 240 to 17121:22
sdake_with no loss of service or features21:22
jascott1you silver tongued devil you21:22
sdake_just gotta find the right person who cares enough to help21:22
jascott1true21:22
sdake_onee dude was like "there is nothing I can do"21:22
jascott1establish up front " I AM ALSO HUMAN"21:22
jascott1works with cops too (sometimes) :)21:23
sdake_i find calling on a tuesday at around 10-11 my time works best21:23
sdake_and to be friendly and kind rather then being a jerk :)21:23
jascott1ah right before lunch or?21:23
sdake_10 is when I call21:23
sdake_it usually takes 30-40 mins to get the job done21:23
sdake_if you use verizon, they have a "new verizon plan" which is far superior to their old data plan21:24
sdake_i had some crap on there for a wireless box i haven't used for 5-6 years21:24
sdake_20$mo data fee21:24
sdake_this was prior to phones allowing hotspots21:24
sdake_all ya gotta do is call :)21:25
sdake_i attempted renogiating my insurance however have a ticket from 2014 july which increases my yearly insurance bill by 1k21:25
* sdake_ groans21:25
sdake_after that I gave a go at lookingn at different insurance cos, and found out my insurance guy was really giving me the best deal21:26
sdake_i'll be back at him in aug to drop my bill 100$/mo21:26
sdake_if he doesn't someone else will :)21:26
sdake_of course I wont even have to call someone else, I'll just tell him straight up the ticket has fallen off and competitors are offering a complete insurance plan for 1k less21:27
sdake_that will get him into action :)21:27
*** adrian_otto has joined #openstack-kolla21:27
sdake_ everything is negotiatble21:27
financeManFollowing the guide for a clean install of ubuntu 16.04 I get this error -> http://pastebin.com/7fMSfGeR21:28
financeManSingle machine host for playing with kolla21:28
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: DNM bisecting kolla-ansible to find regression  https://review.openstack.org/43269721:29
*** adrian_otto has quit IRC21:29
sdake_financeMan its the weekend and most people are probably off doing other  things; I have no expertise in ubuntu however I think if you run ps -ef | grep libvirt and that shows no libvirt process, that error is safe to ignore21:30
financeManwhy arent people working 24/721:30
jascott1haha I like this person21:31
sdake_lol21:31
*** Pavo has quit IRC21:31
jascott1because we cant all be financeMan!21:31
sdake_gotta jet - beer-o-clock21:31
jascott1sdake_ save some brain cells for PTG evenings!21:32
jascott1to kill of course21:32
jascott1enough time in ATL and brain cells will line up for the suicide booth21:33
financeManunlimited money is achieved 24/7 level effort21:34
*** Jeffrey4l_ has quit IRC21:34
jascott1ah but time is priceless (unless thats all you have :/)21:34
*** Jeffrey4l_ has joined #openstack-kolla21:35
financeMansdake_ jascottl when the documentation states "By default kolla-build will build all containers using CentOS as the base image and binary installation as base installation method. "21:41
financeManIf I am installing on Ubuntu 16.0421:41
financeManis leaving it as centos default fine?21:42
jascott1financeMan your host is ubuntu? doesnt matter as the containers are just default to centos binary21:46
jascott1containers right, dont care about host os21:46
jascott1*usually*21:46
jascott1you can change to ubuntu in kolla-build.conf21:47
jascott1but dont have to21:47
jascott1i myself am more comfortable in debian flavored distros21:47
jascott1but the RHEL family seems to have their sh17 together more these days21:48
financeManjascottl I ran the build and finally got this final output21:54
financeManhttp://pastebin.com/UkacTMYe21:54
financeManis that ok?21:54
*** sbezverk has quit IRC22:06
financeMankfox1111 openstackgerrit berendt inc0 sdake_ just tried the build with literally 100% defaults and got "TASK [prechecks : Checking free port for Aodh API] ***************************** fatal: [localhost]: FAILED! => {"failed": true, "msg": "ERROR! 'dict object' has no attribute u'ansible_eth0'"}"22:06
financeManJust followed the quickstart guide to a T22:07
*** haplo37_ has quit IRC22:07
berendtfinanceMan: do you have an eth0 interface on your system?22:07
financeMan"enp6s0f0" lol22:08
financeManI just assumed it was22:08
financeMannever seen one like that22:08
financeMani'll update and try22:08
berendtnetwork_interface is eth0 by default, set it to enp...22:08
financeManfixed it, thanks22:08
*** sbezverk has joined #openstack-kolla22:09
financeManberendt ""msg": "Hostname has to resolve to IP address of api_interface""22:11
financeManI am not seeing immediately what I need to set in the config for this22:11
berendtadd the ip address you are using on enp... to /etc/hosts22:11
financeManMy host network ip is currently 192.168.1.112, should I not see this to that as well? "kolla_internal_vip_address: "192.168.1.112""22:12
berendtsome services use the hostname instead of the ip address22:12
berendti think this is the case for rabbitmq22:12
berendthttps://github.com/openstack/kolla-ansible/blob/master/ansible/roles/baremetal/tasks/pre-install.yml#L19-L2622:13
berendtbaremetal is a preparation role for all used nodes22:13
berendtthis will prepare your /etc/hosts file22:13
financeManhow do I run that22:16
*** haplo37_ has joined #openstack-kolla22:20
*** adrian_otto has joined #openstack-kolla22:29
financeManberendt Any idea why I am getting failed = true on this step ? "Checking if kolla_internal_vip_address and kolla_external_vip_address are not pingable from any node] ***"22:34
financeManit can ping it just fine22:35
berendtyeah, that's the problem :)22:35
berendtthose addresses should not be pingable when running the prechecks22:35
berendtthey will be managed by keepalived22:35
financeManwell it's all just a single computer where I have a single ethernet cable hooked into it22:37
financeManlocalhost and it has it's first ethernet port having the network ip of 192.168.1.11222:37
*** adrian_otto has quit IRC22:43
*** adrian_otto has joined #openstack-kolla22:45
*** adrian_otto has quit IRC22:46
financeManberendt what should I do if I'm just setting up a single box deploy with one nic?22:47
*** dave-mccowan has joined #openstack-kolla22:47
berendtjust use a second and thirst ip address from the same subnet22:48
financeManI'm not much of a network guy, do I just set those up and then how do I make them not pingable? like loopback addys or what22:48
financeManhow do I make them 'not pingable' by kolla standards berendt?22:58
berendtfinanceMan: do not configure them, kolla will do this for you inside the keepalived container22:59
financeManso just remove the ips?22:59
financeManberendt it says I need to have a "kolla_internal_vip_address"23:00
financeMancant not configure it23:00
berendtyes you set this address in the globals.yml file, but you do not configure the used addresses on the interfaces23:01
financeManOh HAH Eggcellent, thank you very much, worked.23:03
*** mannamne has joined #openstack-kolla23:05
*** dave-mccowan has quit IRC23:11
financeManberendt so "kolla-ansible deploy" seemed to work great but when I try to go to http://192.168.1.115 (vip addy), it just idles trying for a long time before timing out23:17
financeManthe guide said to run ". /etc/kolla/admin-openrc.sh" but that doesn't exist for me23:17
financeManEven after I ran 'kolla-ansible post-deploy"23:18
*** salv-orlando has joined #openstack-kolla23:21
berendtfinanceMan: excuse me, i am going to bed now, here it is after midnight23:21
*** salv-orlando has quit IRC23:26
*** mannamne has quit IRC23:27
*** eaguilar has joined #openstack-kolla23:35
*** tonanhngo has joined #openstack-kolla23:40
*** eaguilar has quit IRC23:43
*** goldyfruit has joined #openstack-kolla23:50
*** goldyfruit has quit IRC23:50
*** goldyfruit has joined #openstack-kolla23:52
*** huzhengchuan has quit IRC23:52
*** mannamne has joined #openstack-kolla23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!