Tuesday, 2018-05-29

*** ricolin has joined #openstack-meeting-500:08
*** roman_g has quit IRC02:02
*** yamamoto has quit IRC02:04
*** yamamoto has joined #openstack-meeting-502:04
*** notmyname has quit IRC02:44
*** spsurya has quit IRC02:44
*** persia has quit IRC02:44
*** portdirect has quit IRC02:44
*** skazi has quit IRC02:44
*** spsurya has joined #openstack-meeting-502:45
*** persia has joined #openstack-meeting-502:45
*** portdirect has joined #openstack-meeting-502:45
*** skazi has joined #openstack-meeting-502:45
*** notmyname has joined #openstack-meeting-502:46
*** yamamoto has quit IRC02:46
*** slaweq has quit IRC02:46
*** pcarver has quit IRC02:46
*** kazsh has quit IRC02:46
*** yamamoto has joined #openstack-meeting-502:48
*** slaweq has joined #openstack-meeting-502:48
*** pcarver has joined #openstack-meeting-502:48
*** kazsh has joined #openstack-meeting-502:48
*** Guest157` has quit IRC02:50
*** markvoelker has quit IRC02:50
*** njohnston has quit IRC02:50
*** ianychoi_ has quit IRC02:50
*** hoangcx has quit IRC02:50
*** Guest157` has joined #openstack-meeting-502:50
*** markvoelker has joined #openstack-meeting-502:50
*** njohnston has joined #openstack-meeting-502:50
*** ianychoi_ has joined #openstack-meeting-502:51
*** hoangcx has joined #openstack-meeting-502:51
*** hoangcx has left #openstack-meeting-504:05
*** markvoelker has quit IRC04:19
*** markvoelker has joined #openstack-meeting-504:22
*** kei-ichi has quit IRC04:59
*** radeks has joined #openstack-meeting-506:22
*** radek__ has joined #openstack-meeting-506:39
*** roman_g has joined #openstack-meeting-507:30
*** slaweq has quit IRC08:37
*** slaweq has joined #openstack-meeting-508:38
*** ianychoi_ is now known as ianychoi08:40
*** derekh has joined #openstack-meeting-508:45
*** ricolin has quit IRC09:04
*** markvoelker has quit IRC09:15
*** yamamoto has quit IRC10:10
*** markvoelker has joined #openstack-meeting-510:16
*** markvoelker has quit IRC10:51
*** yamamoto has joined #openstack-meeting-511:11
*** yamamoto has quit IRC11:16
*** yamamoto has joined #openstack-meeting-511:36
*** yamamoto has quit IRC11:47
*** markvoelker has joined #openstack-meeting-511:48
*** yamamoto has joined #openstack-meeting-511:48
*** markvoelker has quit IRC12:21
*** markvoelker has joined #openstack-meeting-513:17
*** markvoelker has quit IRC13:22
*** markvoelker has joined #openstack-meeting-513:33
*** markvoelker has quit IRC14:24
*** markvoelker has joined #openstack-meeting-514:26
*** markvoelker has quit IRC14:30
*** mjturek has joined #openstack-meeting-514:32
*** hongbin has joined #openstack-meeting-514:42
*** wxy| has joined #openstack-meeting-514:48
*** piotrrr has joined #openstack-meeting-514:48
*** tdoc_ has joined #openstack-meeting-514:50
*** felipemonteiro has joined #openstack-meeting-514:51
*** felipemonteiro_ has joined #openstack-meeting-514:54
*** felipemonteiro has quit IRC14:58
*** seaneagan has joined #openstack-meeting-514:59
*** felipemonteiro_ is now known as felipemonteiro14:59
mattmceuen#startmeeting openstack-helm15:00
openstackMeeting started Tue May 29 15:00:50 2018 UTC and is due to finish in 60 minutes.  The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: openstack-helm)"15:00
openstackThe meeting name has been set to 'openstack_helm'15:00
mattmceuen#topic rollcall15:00
*** openstack changes topic to "rollcall (Meeting topic: openstack-helm)"15:00
mattmceuenGM all!15:01
roman_ggood morning15:01
piotrrrhi15:01
lamto/15:01
tdoc_hello15:01
Guest157`hihi!15:01
mattmceuenHere's the agenda for today: https://etherpad.openstack.org/p/openstack-helm-meeting-2018-05-2915:01
mattmceuenWelcome15:01
Guest157`oops15:01
mattmceuenPlease add in anything else you'd like to discuss in there15:01
*** Guest157` is now known as d|k15:01
mattmceuenHey d|k :)  you have given up your anonymity!15:01
*** d|k is now known as Guest5746915:02
*** Guest57469 is now known as d|k`15:02
portdirecto/15:02
mattmceuenHey portdirect!  Aren't you on vacation?15:02
portdirectYup15:02
portdirectIn a mountain chalet on dodgy 3g :)15:02
roman_go/15:02
mattmceuenNice15:03
mattmceuenhey roman_g tdoc piotrrr lamt15:03
rwellumo/15:03
mattmceuenhey rwellum15:03
rwellumHey mattmceuen15:04
mattmceuenportdirect just make sure not to work too hard in this team meeting, just provide color commentary and lob virtual rotten tomatos15:04
mattmceuenFull disclosure:  I am brushing off all kinds of mental and technological cobwebs this morning following the long (USA) weekend, following the Vancouver summit15:05
mattmceuenFirst thing15:05
mattmceuen#topic Quick Summit Recap15:05
*** openstack changes topic to "Quick Summit Recap (Meeting topic: openstack-helm)"15:05
mattmceuenA lot of good things were set in motion for OSH in Vancouver15:06
mattmceuenMet a lot of folks f2f which was excellent as well, and a  number of imminent or potential OSH users15:06
tdoc_Props on all the sessions, I attended most of them, good stuff. (I would be one of those.)15:07
mattmceuenI won't give a rundown of them all here, since it's not all public - but I will be reaching out to them to get them pulled into our team15:07
mattmceuenAwesome - thanks for attending tdoc15:07
mattmceuenGlad to hear you thought they were valuable, since obviously I'm slightly biased :-D15:07
tdoc_Yeah, I think it gave us a good impression of where the project is and is heading...15:08
piotrrryup, good stuff indeed, +115:08
mattmceuenAlso there was a lot of discussion around Airship in the summit, which was great from an OSH perspective, as Airship is a consumer of OSH.  As well as one platform to run it on.15:10
roman_gsummit recap from youtube viewer: good (not great) presentations - listeners seemed to be a bit frustrated. But I hope that you attracted at least some attention to the Airship, since that one of goals. Overall, looking at other videos and comparing to previous years, it is seen that there is way less hype (which is good), and more work in direction of support of stabilization of the platform.15:10
portdirectThoughts on why people seemed frustrated?15:11
mattmceuenAgree with your assessment roman_g that there is overall less hype and more delivery focus in the presentations15:11
mattmceuenYes, was the frustration specific to the OSH presentations, or do you mean overall with the summit roman_g?15:11
roman_gOSH/Airship.15:12
roman_gThoughts on why people seemed frustrated? - I would say that not all people who were coming to your sessions were target audience.15:13
mattmceuenOk - I still have a few related sessions I need to catch up on via youtube, and I'll keep an ear out for that15:13
roman_gThat's what I understood from questions which were asked.15:13
portdirectSame, good info roman_g15:14
mattmceuenThe sessions I was at in person had a good vibe with a lot of Q&A that extended far past the end of the presentation, which was great to see15:14
roman_gthat's very good15:14
mattmceuenBut I'll re-watch those too in case I missed some things -- there were a lot of notes to be taken15:15
mattmceuenThanks roman_g15:15
mattmceuenAny other summit tidbits to share?15:15
roman_gwant to visit one of them )15:15
roman_gas a speaker15:15
rwellumRe the workshop I attended..15:16
mattmceuen+115:16
rwellumI could help out a bit with the folks sitting around me - most had no or very little k8s experience. I did think that running a set of scripts one after another was a bit counterintuitive - because on one hand it showed the power of osh on the other hand it didn't teach much about what was going on.15:17
portdirectAgree15:17
rwellumBut I don't have a good alternative tbh.15:18
mattmceuenYeah, that's a hard thing to balance15:18
mattmceuenWas wrestling with the same internal monologue rwellum15:18
portdirectUnfortunalty the scenarios I had slides for changed st the last minute (while on stage)15:18
rwellumYeah I noticed that portdirect15:18
portdirectSo had to freestyle the whole thing.15:18
rwellumYou had to pivot15:18
piotrrrMy impression is that some people (like me), might not necessarily care much about Airship, but do care about using OSH. And they want to use it to deploy OS on top of their, already existing, kubernetes clusters. So, I can imagine that such people might be a bit frustrated with yet one large box of moving parts being introduced into the picture (Airship). Sure, as far as I understand airship15:19
piotrrrwill not be a hard requirement for OSH, but I would say it would make sense to make a clearer distinction between the two, where possible.15:19
mattmceuenAirship has a nice four-line "stand up the stack" experience, which is awesome for showing that the thing exists and works, but really doesn't give hands-on at all.  OTOH you don't want to make it too deep or else newbies may get left behind15:19
mattmceuenSo I think the script approach is at least one good middle ground way to demo the product and peel back the curtain just a bit15:19
rwellumMany of the people around me were confused when we got to the stage where we had to run 'make'.15:20
rwellumFor example15:20
roman_gMotto: run `watch -n1 "kubectl -n xxxx get pods"` for each namespace in tmux screens, and provide more usefull info on what is happening during installation and other phases.15:20
mattmceuenpiotrrr yep, no requirement to use airship for OSH at all.15:20
portdirectrwellum: all the vms had the wrong things provisioned on them :(15:21
portdirectOh well15:21
rwellumroman_g: +115:21
roman_galso good to have some monitoring of docker pulling images, but it's impossible right now15:21
tdoc_rwellum: that was a bit unclear in the intructions, people where running the make in the wrong dir.15:21
rwellumtdoc_: me too - took a while to figure out15:21
rwellumportdirect: :(15:22
tdoc_it's because I had some prior knowledge that I was able to follow...15:22
portdirectYeah I'm really sorry15:22
rwellumOverall I think most people I spoke to were just very happy to play with a live demo. So that was good.15:22
tdoc_but unfortunaltely in my instances the db charts had troubles, so got stuck there.15:22
mattmceuenYes I will say that for having the wrong things provisioned on the VMs -- way to roll with the punches portdirect :)  made for a still valuable workshop15:22
portdirectI had to figure it out with over 100 eyes on me15:23
roman_g50 ppl15:23
rwellumtdoc_: I think although there were plenty of VM's - there were some latency issues maybe>15:23
mattmceuen300 eyes?15:23
rwellum?15:23
tdoc_yup15:23
*** MarkBaker has joined #openstack-meeting-515:23
piotrrrI think some people had issues with containers/pods getting stuck for some reason15:23
*** felipemonteiro has quit IRC15:23
tdoc_not complaining, given the conditions etc, it was still worthwhile..15:24
mattmceuengood15:24
*** felipemonteiro has joined #openstack-meeting-515:24
mattmceuenAlrighty - moving on to our next topic:15:24
mattmceuen#topic Storyboard15:24
*** openstack changes topic to "Storyboard (Meeting topic: openstack-helm)"15:24
mattmceuenSo we had reasons to move to Storyboard before15:25
mattmceuenBut now we are truly motivated15:25
mattmceuenI have been told that once we migrate to Storyboard, we can have15:25
mattmceuen...15:25
mattmceuenHoney Badger Stickers15:25
rwellum%^$*!!!15:25
mattmceuen(which is of course the OSH mascot)15:25
mattmceuenI know right15:25
lamtis that the condition for stickers?15:25
mattmceuenIt is15:26
lamtI will volunteer then - I want stickers15:26
mattmceuenslightly tounge-in-cheekly, but that's the agreement :)15:26
piotrrrWhat's the status of the migration?15:27
portdirectWho is leading this atm?15:27
mattmceuenThe biggest challenges with migration will be15:27
mattmceuen1) communicating it to everyone15:27
mattmceuen2) using the new storyboard-friendly git commit headers15:27
mattmceuenrwellum15:27
mattmceuenHe has done a POC of the migration that he shared a couple weeks back for feedback15:27
mattmceuenrwellum, have any concerns been raised?15:28
rwellumYeah the POC is still up for everyone to look at and play with15:28
rwellumNo not to me.15:28
*** gagehugo has joined #openstack-meeting-515:28
portdirectSo let's just pull the trigger?15:28
rwellumThere are other teams I spoke to at the Summit that are holding back for various reasons.15:28
mattmceuenIt sounds like the migration itself is small potatos, and the trigger can be pulled whenever we're ready15:28
rwellumYeah - it's all ready I think.15:28
portdirectrwellum: what like?15:28
mattmceuenhoney badger don't care about holding back15:28
mattmceuenDo we want to set a target of e.g. next monday so we can communicate?15:29
portdirectAlso can you update the docs, to point to storyboard etc?15:29
rwellumLike Cinder for example, they are so embedded in the old way and the sample migration took days to run and didn't complete.15:29
mattmceuenportdirect yup15:29
mattmceueninteresting rwellum15:29
rwellumBut I think for OSH - less worries as still new, you guys are writing the process newly.15:30
rwellumBad English but ykwim15:30
portdirectYeah, we dont have that much stuff to pack up and take over.15:30
rwellumYeah so if we target next monday, I'll contact the infra team and ask them to initiate the next step.15:31
mattmceuenGood.  Yeah, the biggest constructive criticism I've received re: OSH is that we could use a better commuity roadmap so it's easy to see where the project is going and to volunteer for work items.  We've been making good strides but the storyboard migration is a great opportunity to get that in good shape.15:31
portdirect++15:32
mattmceuenExcellent - let's do that, let me know if Monday turns out to be a bad day for any reason.  I'll plan on sending some comm out in the ML15:32
rwellumOk I will.15:32
tdoc_once you guys switch, will that mean new bugs can't be filed in launchpad?15:33
tdoc_ie will it be clear to end users at what point they should use which tracker?15:33
mattmceuenIt definitely means they /shouldn't/ be; will we be able to actually disable launchpad?15:33
rwellumThat's the idea tdoc_15:33
rwellumThe disable part - that's one of the things I want to check with infra15:34
rwellumI'll report back.15:34
mattmceuenexcellent15:34
rwellumAlso think it would be good to add a 'low hanging fruit' project group - for simple things to pass onto new users.15:34
mattmceuen+115:34
rwellumSimple bugs etc.15:34
roman_g+115:34
roman_gthat's important for me15:35
mattmceuenWe would like to cut our 1.0 release in the next couple months, and identification e.g. low-hanging doc updates as well would be good ones15:35
piotrrr+115:35
rwellumYup - that would be a great addition too15:35
rwellumimo15:35
portdirectrwellum: do you have the bandwidth to take a stab at getting a low hanging list up?15:35
rwellumYes I will do that.15:35
portdirectAwesome :D15:35
portdirectThx dude15:36
rwellumnp15:36
mattmceuenNext topic:15:36
mattmceuen#topic Creating a set of guidelines which would help contributors troubleshoot issues, e.g. stuck containers15:36
*** openstack changes topic to "Creating a set of guidelines which would help contributors troubleshoot issues, e.g. stuck containers (Meeting topic: openstack-helm)"15:36
mattmceuenpiotrrr, want to speak to this one?15:37
mattmceuenAdding some solid operational docs is definitely something we want to do as part of our 1.0 release15:38
piotrrryes, so we're just starting with OSH, and we're running into all kinds of different issues. Stuck containers/pods etc. We have no know-how on to troubleshoot those issues. If we're running into such problems, other contributors/operators might also be.15:39
mattmceuenIt would be good to capture a list of topics to speak to (and then create storyboard items for!) -- this is a good one15:39
mattmceuen+1 piotrrr15:39
tdoc_I think it would be nice to have. I've been at the point where I see a bunch of pods in init state and wondering what to do.15:39
piotrrrSo, my question would be whether the OSH community would like to collab on creating a doc with tips/hints for troubleshooting OSH and OS running on top of it15:39
portdirectYes!15:40
portdirectAnd this is where new users add huge value :)15:40
roman_gtdoc_: they were docker-pulling? )) you can't monitor progress of that unfortunatelly15:40
tdoc_It's often unclear to me which pod is waiting for which other pod to complete.15:41
mattmceuenWe have a troubleshooting doc already, we should all get into the habit a little bit more of adding things into it after we fix them!15:41
portdirectI know to go check the init containers, the eventually kublet logs - but this is prob not intuitive for new k8s users15:41
tdoc_+115:41
roman_g+1.15:41
mattmceuenIs there any good "general" kubernetes troubleshooting guide out there that we can refer to for good "technique"?15:42
portdirectJust the k8s docs that I'm aware of15:42
roman_ghaven't seen that15:42
portdirectThough they are quite thin15:42
roman_gbut having dashboard open helps a bit15:42
rwellumI have some debug steps from kolla-k8s - some would apply, I can look at the troubleshooting guide and see if any can help.15:43
portdirectmattmceuen: in the workshop, how many people were hung up on ceph ns activation?15:43
mattmceuenTip #1 :)  we have LMA user interfaces - good one roman_g15:43
mattmceuenI think at least 3-5 folks portdirect15:44
mattmceuenthat step was easy to miss for whatever reason15:44
piotrrrok, how do we want to start with? Maybe creating a etherpad where everyone from the OSH team could braindump their approaches/hints for troubleshooting. We can organize those into public  docs later on.15:44
roman_gapache airflow dag dashboard?15:44
portdirectroman_g: not airship15:44
portdirectThe actual lma stack from osh-infra.15:44
mattmceuenhttps://etherpad.openstack.org/p/openstack-helm-troubleshooting15:45
roman_gportdirect: ah, yep15:45
mattmceuen^ let's use that to jot down ideas as they come to us (and troubleshooting steps as we do them)15:45
roman_gmattmceuen: pin to channel topic here & in slack?15:45
mattmceuenThen we can turn them into storyboard things and doc updates at our convenience15:45
mattmceuenGood idea15:46
mattmceuenthanks piotrrr for bringing this up, let's revisit next week and see how the etherpadding is going15:47
rwellumhttps://github.com/openstack/kolla-kubernetes/blob/master/doc/source/deployment-guide.rst - look at the ts guide at the end. I did most of that.15:47
piotrrrsounds good, thanks15:47
rwellumI'll add to the etherpad15:48
piotrrr(I would be happy to help turn the notes into proper docs later on)15:48
mattmceuenThat would be awesome, thank you!15:48
mattmceuenrwellum there is a lot of good stuff in there that could be adapted to OSH15:49
rwellumYeah it's all k8s15:49
mattmceuenOk - we have another item tdoc_ wanted to bring up15:49
mattmceuen#topic Roundtable15:50
*** openstack changes topic to "Roundtable (Meeting topic: openstack-helm)"15:50
tdoc_yeah, so I brought this up the irc chan... Having some DNS issues with rabbitmq.15:50
*** MarkBaker has quit IRC15:50
tdoc_It seems in my case the rabbitmq-server won't start because it can't lookup its hostname.15:51
mattmceuenI haven't had a chance to catch up on the full conversation yet; you're still seeing the issue tdoc_?15:51
tdoc_That DNS record does not exist yet, because the readines/liveliness probes don't pass yet... So chicken-egg.15:51
portdirectIt's odd that the pod cannot resolve itself though15:52
tdoc_I need to add this service.alpha.kubernetes.io/tolerate-unready-endpoints to make it work15:52
mattmceuenI think you noted it's not an issue in the gates; do you know what the difference between your environment & the gates might be?15:52
mattmceuenWhat version of k8s are you running?15:53
tdoc_1.10.215:53
portdirectAnd you've now tried with both kube-dns and coredns?15:54
tdoc_I'm somewhat assuming it does not come up it the gates, but not familiar with that environment myself yet...15:54
tdoc_yup, tried both15:54
portdirectHmm, this is an odd outlier :/15:54
portdirectAre you on very slow machines?15:55
tdoc_As far as I've understood the docs though, k8s is not supposed to expose dns records for headless services until the probes indicate ready.15:55
portdirectCorrect15:55
portdirectThough the pod should be able to resolve itself15:55
tdoc_yeah, all my stuff is running in our local openstack cloud, so in VMs which might not be the fastest...15:55
tdoc_It complains about rabbitmq-rabbitmq-0.rabbitmq-dsv-7b1733.openstack.svc.cluster.local15:56
tdoc_Which I think is the record for the service15:56
portdirectFrom the 1st rabbit pod?15:56
tdoc_yup15:56
portdirectCan you paste the full logs?15:56
mattmceuenPlease paste them in the OSH chat15:57
mattmceuenSo we can keep this going since we're almost out of time15:57
tdoc_hmm, i'm not sure I have those handy right now15:57
mattmceuenNo worries, if you can share them when you have them handy that would be helpful15:57
tdoc_it's something like ERROR: epmd .... that hostname.... domain not found....15:57
tdoc_(sorry best I can do right now)15:57
mattmceuenWe'll get it figured out15:58
mattmceuenAlright, with two minutes left - any final discussion points?15:58
mattmceuenAlright - thanks for a great meeting all15:59
mattmceuenSee you in #openstack-helm15:59
mattmceuen#endmeeting15:59
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:59
openstackMeeting ended Tue May 29 15:59:54 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_helm/2018/openstack_helm.2018-05-29-15.00.html15:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_helm/2018/openstack_helm.2018-05-29-15.00.txt15:59
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_helm/2018/openstack_helm.2018-05-29-15.00.log.html15:59
*** d|k` is now known as d|k16:00
*** tdoc_ has quit IRC16:03
*** gagehugo has left #openstack-meeting-516:07
*** markvoelker has joined #openstack-meeting-516:52
*** markvoelker_ has joined #openstack-meeting-516:53
*** mjturek has quit IRC16:53
*** markvoelker has quit IRC16:57
*** derekh has quit IRC17:00
*** wxy| has quit IRC17:02
*** mjturek has joined #openstack-meeting-517:12
*** yamahata has joined #openstack-meeting-517:14
*** seaneagan has quit IRC17:57
*** iyamahat has joined #openstack-meeting-518:01
*** iyamahat has quit IRC18:05
*** iyamahat has joined #openstack-meeting-518:12
*** markvoelker_ has quit IRC18:33
*** felipemonteiro_ has joined #openstack-meeting-518:40
*** markvoelker has joined #openstack-meeting-518:42
*** radeks_ has joined #openstack-meeting-518:44
*** felipemonteiro has quit IRC18:44
*** felipemonteiro_ has quit IRC18:51
*** felipemonteiro_ has joined #openstack-meeting-518:51
*** radeks has quit IRC18:52
*** njohnston has quit IRC18:52
*** njohnston has joined #openstack-meeting-518:53
*** radeks__ has joined #openstack-meeting-519:07
*** radek__ has quit IRC19:10
*** radeks_ has quit IRC19:10
*** markvoelker has quit IRC19:18
*** markvoelker has joined #openstack-meeting-519:19
*** markvoelker has quit IRC19:22
*** felipemonteiro_ has quit IRC19:49
*** markvoelker has joined #openstack-meeting-519:49
*** markvoelker_ has joined #openstack-meeting-519:50
*** markvoelker has quit IRC19:54
*** radeks__ has quit IRC21:09
*** slaweq has quit IRC21:10
*** slaweq has joined #openstack-meeting-521:11
*** sshank has joined #openstack-meeting-521:44
*** hongbin has quit IRC23:00
*** mjturek has quit IRC23:32

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!