Thursday, 2018-09-06

*** spartakos has quit IRC00:30
*** jamesmcarthur has joined #openstack-operators00:33
*** rcernin has quit IRC00:50
*** rcernin has joined #openstack-operators00:51
*** gyee has quit IRC00:53
*** VW_ has joined #openstack-operators01:10
*** slaweq has joined #openstack-operators01:11
*** slaweq has quit IRC01:16
*** jamesmcarthur has quit IRC01:43
*** jamesmcarthur has joined #openstack-operators01:45
*** jamesmcarthur has quit IRC01:56
*** spartakos has joined #openstack-operators02:10
*** slaweq has joined #openstack-operators02:11
*** slaweq has quit IRC02:15
*** nicolasbock has quit IRC02:18
*** slaweq has joined #openstack-operators03:11
*** slaweq has quit IRC03:15
*** gkadam has joined #openstack-operators03:31
*** spartakos has quit IRC03:35
*** bauruine has quit IRC03:48
*** jamesmcarthur has joined #openstack-operators03:56
*** bauruine has joined #openstack-operators03:58
*** markvoelker has joined #openstack-operators04:06
*** markvoelker has quit IRC04:11
*** emccormickva has joined #openstack-operators04:17
*** emccormick has quit IRC04:20
*** marst has joined #openstack-operators04:40
*** Bhujay has joined #openstack-operators04:45
*** markvoelker has joined #openstack-operators05:07
*** slaweq has joined #openstack-operators05:11
*** slaweq has quit IRC05:15
*** r-mibu has joined #openstack-operators05:28
*** gkadam_ has joined #openstack-operators05:41
*** gkadam has quit IRC05:43
*** gkadam__ has joined #openstack-operators05:45
*** gkadam_ has quit IRC05:47
*** med_ has quit IRC05:54
*** Bhujay has quit IRC06:07
*** Bhujay has joined #openstack-operators06:18
*** pcaruana has joined #openstack-operators06:25
*** aojea has joined #openstack-operators06:40
*** markvoelker has quit IRC06:45
*** slaweq has joined #openstack-operators06:50
*** rcernin has quit IRC07:00
*** pcaruana has quit IRC07:36
*** markvoelker has joined #openstack-operators07:43
*** pcaruana has joined #openstack-operators07:48
*** electrofelix has joined #openstack-operators08:48
*** emcco__ has joined #openstack-operators09:05
*** emccormickva has quit IRC09:09
*** derekh has joined #openstack-operators09:21
*** dabukalam has joined #openstack-operators09:41
*** aojea has quit IRC09:54
*** nicolasbock has joined #openstack-operators10:31
*** med_ has joined #openstack-operators10:47
*** aojea has joined #openstack-operators11:26
*** markvoelker has quit IRC12:22
*** r-mibu has quit IRC12:47
*** jamesmcarthur has quit IRC12:52
*** jamesmcarthur has joined #openstack-operators12:52
*** jamesmcarthur has quit IRC13:03
*** mriedem has joined #openstack-operators13:08
*** dtrainor has quit IRC13:28
*** dtrainor has joined #openstack-operators13:31
*** mihalis68_ has joined #openstack-operators13:56
mihalis68_ping emccormick13:59
mihalis68_ping smcginnis13:59
smcginnisHey mihalis68_14:03
mihalis68_I'm tidying up the google doc a bit14:03
mihalis68_the one glaring defect is the ceph session overlaps upgrade sig14:03
smcginnisMaybe that's OK?14:03
mihalis68_erik was of the opinion we shouldn't overlap, but it seems like a big chunk to block out to me still14:04
mihalis68_there's a few moderators to find also14:05
mihalis68_I can't quite tell who proposed unified limits, for example14:05
smcginnisThat's still being worked on I thought. I can check with Lance from the keystone team that's done a lot there.14:06
mihalis68_yeah I just saw email about it from him14:07
smcginnisMight be too early to have a useful discussion on that. Unless they are looking for early input to be able to course correct on anything.14:07
mihalis68_I could put ceph in that slot then, since that seems more definitely useful and I could moderate that if Simon Leinen can;t14:08
smcginnisThat sounds like a good plan to me.14:09
mihalis68_are you ok moderating operator docs, Monday morning second session>14:09
mihalis68_?14:09
smcginnisYeah, that should be good.14:10
*** shubjero has joined #openstack-operators14:10
smcginnisI don't have the link handy. That's after the git/gerrit stuff, right?14:11
mihalis68_YES14:12
mihalis68_yes14:12
mihalis68_also https://docs.google.com/spreadsheets/d/1EUSYMs3GfglnD8yfFaAXWhLe0F5y9hCUKqCYe0Vp1oA/edit#gid=98152733614:12
smcginnisPerfect - thanks!14:13
mihalis68_one question: I copied over the template including "breakfast" for the start of the day. Do you know if PTG has breakfast?14:13
mihalis68_I don't want to imply food that's not going to turn up :)14:13
mihalis68_that would be like the biggest possible crime as an organiser!14:13
smcginnisIndeed!14:13
mihalis68_I have a feeling it's much more BYOB14:13
mihalis68_ping spotz14:14
mihalis68_Amy might know14:14
mihalis68_heck you've been to PTG before...14:14
mihalis68_any idea>14:14
smcginnisI know there will be coffee, but I can't recall if there were any snack kind of things.14:14
mihalis68_?14:14
smcginnisDefinitely not what I would categorize as breakfast if there is food.14:14
smcginnisMaybe change to Registration & Coffee to be safe.14:15
mihalis68_done14:15
mihalis68_I've described the slots on the monday afternoon which overlap with upgrades sig as (room available) since of course operators could adhoc sessions in there if they prefer instead of upgrade sig14:17
smcginnisGood idea.14:18
mihalis68_I pinged erik on signal in case I can get his attention14:18
mihalis68_I've put my name down to moderate wrap-up day 1, running old openstack (day 2) and feedback (day 2) and also backup ceph moderators. I'd like to shed one of those. It's not my event! :)14:19
spotzmihalis68_: here!14:19
mihalis68_we had "breakfast" down for the beginning of each day, but that's not true, right? just coffee>14:20
mihalis68_?14:20
spotzOpenStack Summit doesn't really do breakfast. Let's ping an expert though:) kendallwaters14:20
mihalis68_also I wrote that you're confirmed to moderate git/gerrit/osops right?14:20
mihalis68_Humph14:20
mihalis68_ops meetups had pretty good spreads14:20
spotzYeah I'll run a quick git and gerrit lunch and learn14:20
mihalis68_can't be that operators eat more than developers can it?14:20
mihalis68_:)14:20
emcco__hey guys. Sorry for the lateness14:21
mihalis68_we updated the first session to only mention coffee14:21
spotzOpen Source summit had breakfast no lunch but nothing I could eat:( And no ops never get more then dev:(14:21
mihalis68_can you cast your eye over https://docs.google.com/spreadsheets/d/1EUSYMs3GfglnD8yfFaAXWhLe0F5y9hCUKqCYe0Vp1oA/edit#gid=981527336 and yell out mistakes to me plz14:21
mihalis68_I wonder if I could get a large dunkin order on Bloomberg for day 114:22
emcco__oh hey it changed my name14:22
emcco__how interesting that it even let me stay on here14:22
spotzyou might have this one registered too14:22
mihalis68_you can do slash nick nickname and that works (for me), but at least via this client that changes how I try to auth and basically next time I can't get in14:23
*** emcco__ is now known as emccormick14:23
emccormickah14:23
emccormickthere we go14:23
mihalis68_yah I picked up a trailing _ also but I'm letting it go14:24
smcginnis:)14:24
mihalis68_can you see the google doc?14:24
emccormickpulling it up now14:24
spotzyep14:24
mihalis68_another change, erik, we moved unified limits to be listed with ? and alongside upgrade sig. This gets ceph into a better slot and it's because smcginnis thinks it's too soon to talk about that with ops14:25
spotzthere was a post on the dev list today about unified limits, seems not many folks working on it FYI14:25
mihalis68_yeah saw it14:25
mihalis68_even if I assume it's the best thing ever, does seem premature for ops to discuss, unless it's just "tell us of your pain when nova compute quota gets out of whack", but I think that's well known14:26
emccormicktoo soon I agree14:27
mihalis68_erik can you review the slots you're listed as moderating (on the right, may have to scroll sideways)14:28
emccormickso then that opens us up for upgrades14:28
emccormickI don't really know anything about running OS on Kube14:28
mihalis68_ok I removed unified limits14:28
emccormickand that's kinda overly specific I think14:29
emccormickCan we make that generically containers?14:29
mihalis68_yes absolutely. Have at it. I don't DO containers so far :| I am the last person to be devising that bit14:29
emccormickmake them both that way. Kube can be included14:29
emccormicks/k8s/containers/g14:29
smcginnisIf there's something unique about running containers on k8s, then that's good to discuss. But I don't believe there really is from my limited experience.14:30
emccormickother than that I would swap the 11:40 and 13:4014:30
emccormicklong slot for me *heh*14:31
mihalis68_ok I made k8s containers14:31
emccormickDefinitely no rstarmer14:31
emccormickswitch up OS on containers and Containers on OS14:32
mihalis68_done14:32
emccormickthe OS on containers is relevant to to the Should we Do it or Not discussion14:32
emccormicksuper14:32
mihalis68_I am wondering if we should really split over lunch14:32
mihalis68_11-12.30 is whether to containerize and then OS on containers. Could we make it 3 30 minute slots?14:33
emccormickAll of that noise will be a great segue to the upgrade discussions14:33
emccormickyeah sure14:33
emccormickLess time for me to be stuck in charge ;)14:34
*** Bhujay has quit IRC14:34
smcginnisHah14:34
mihalis68_done14:35
mihalis68_please review14:35
mihalis68_Now we have a hole after lunch14:35
emccormicklooks good to me14:36
mihalis68_not the worst thing ever. We could pull wrap-up forward (since we effectively adjourn and all go to upgrade sig )14:36
mihalis68_move running old openstack forward one slot, pull "wrap-up" forward from end-of-day and basically finish the scheduled operators stuff at that point?14:37
emccormicksounds like a winning plan14:37
smcginnisLet's see how the day goes and make a call if we should do our wrap up earlier, but you may be right.14:37
emccormickThe way we're doing it this time, this can all flex anyway14:38
emccormicktangents are totally ok ;)14:38
mihalis68_ok updated it14:38
smcginnisI only wonder if we might have a desire to regroup after the upgrade stuff if there's anything more we want to quick discuss. But I suppose that could be separate and after any wrap-up talk.14:38
mihalis68_yeah14:39
spotzAll the UC who are there will be at the UC sessions14:39
emccormickWe are also going to go for beer then, so we could start that early ;)14:39
mihalis68_that reminds me, does "feedback with UC" session on tuesday look like a good slot?14:39
smcginnisspotz: But they'll come over for the Feedback with UC sesion, right?14:40
spotzThat's what I meant:)14:40
emccormickOh, on that topic, I have not called the place yet but received positive review from a local on Wynkoop14:40
smcginnisOK, good.14:40
spotzWe put our stuff on Wednesday so that if any ops were still around and wanted to join they could14:40
mihalis68_I'm there all week and no plans at all Wed-Frid :|14:40
emccormickso as far as I'm concerned we plan on doing that14:40
spotzSounds good emccormick14:41
smcginnismihalis68_: I'm sure you'll find something to keep busy. ;)14:41
mihalis68_I promised to mail the mailing list with the schedule and mentioning wynkoop14:41
spotzI'm still trying to decide train or uber from.to airport14:41
smcginnisI did uber/lyft last time. Quick and easy.14:41
emccormickkendallwaters https://www.wynkoop.com/ That place after the ops meetup ends. Say..7pm since it's a cab ride away14:41
emccormicksince she wanted to send it out in email. :)14:42
smcginnisTotally unrelated, but since a lot of you use ceph, just want to make sure you've seen this - https://ceph.com/cephdays/ceph-day-berlin/14:42
emccormickI'm going to call and see if maybe they can block us off an area for 30 people or so without committing cash. It's a Monday, so maybe14:42
mihalis68_Ooh, sean that ceph day abuts openstack summit!14:43
mihalis68_i.e. is just before, I think14:43
emccormickOh super14:43
emccormickright before the summit14:43
mihalis68_yay14:43
emccormickRun by Mike Perez no less ;)14:44
smcginnisYep, day before the Summit officially starts.14:44
*** penick has quit IRC14:44
mihalis68_yeah mike has been in touch to see if Bloomberg might be able to host a ceph day in NYC14:44
mihalis68_good to see him back in action14:44
mihalis68_and talking of ceph one of our object stores is grumpy now14:44
mihalis68_5192 pgs stuck unclean14:45
mihalis68_Ye Gods14:45
smcginnisGet out your scrub brush.14:45
emccormickew14:45
mihalis68_it add 700M objects since aug 15th14:45
mihalis68_I so scare14:45
emccormickCall the exorcist14:45
mihalis68_so yeah... how is the schedule folks? If you think it's "presentable" I'll make a PDF and send it to the list and see if I can confirm the moderators14:46
smcginnisLooks good to me.14:46
mihalis68_and I will mention the rough consensus towards wynkoop, and that it's not sponsored currently14:46
emccormickcool14:47
smcginnisThanks for getting that together mihalis68_14:48
mihalis68_I'd say "my pleasure" except I'm like the least adept google docs guy and it causes actual physical pain14:48
mihalis68_maybe there are better tools14:48
mihalis68_(google docs spreadsheets specifically)14:49
smcginnisThere will always be pain. :D14:49
mihalis68_but yeah this coming together, I'm excited14:49
mihalis68_erik any final suggestions?14:49
mihalis68_spotz same Q.?14:49
mihalis68_sorry I'm mixing and matching first names and nicknames14:49
spotzmihalis68_: heheh14:49
mihalis68_look good enough to show the huddled masses?14:50
emccormickoh hey yes go for it14:50
spotzYep14:50
emccormickI think its good as is14:50
smcginnis+114:50
mihalis68_alright. Thanks for being here and providing feedback. It feels insane otherwise, just throwing words up and somehow thinking people will travel to denver and follow it14:50
mihalis68_oh yeah, one thing I forgot to mention. I am not going to put etherpad links in the google doc. I'm going to make a master etherpads ... etherpad14:51
mihalis68_because etherpad is more universally available14:51
emccormickmakes sense14:52
mihalis68_(I cannot touch this google doc from my corporate desktop, for example)14:52
mihalis68_I'll put the link to the master in this google doc, then make the pdf and then share with list14:52
smcginnisSounds good. That's often how the other PTG groups handle it.14:52
emccormickspiffy14:52
mihalis68_yeah, ok good. Think we're done here for now14:52
mihalis68_thx again14:52
emccormickoh, and smcginnis I found my problem with Cinder which was fixable with an update14:53
emccormickgotta love hitting a 3 year old bug14:54
smcginnisemccormick: Yeah? That's great it was fixed at least.14:54
smcginnisToo bad you had to hit it in the first place though.14:54
emccormickI don't often run openstack updates unless I have a reason14:55
emccormickI'm of the "if it ain't broke" mentality14:55
emccormickso I had ignored a backport14:55
smcginnisProbably a safe policy to have.14:55
smcginnisIn most cases at least.14:55
emccormickThis only got triggered under very specific and rare conditions. I just happened to get unlucky14:56
smcginnisYou don't happen to have the patch link handy yet, do you? I'm just curious, so no big deal.14:56
smcginnisAlways good to know if someone else comes along with a similar issue to have it somewhere in the back of your mind.14:57
mihalis68_am I insane? I  thought we had a link to an ops meetup etherpad on https://wiki.openstack.org/wiki/PTG/Queens/Etherpads14:57
smcginnismihalis68_: Do you mean https://wiki.openstack.org/wiki/PTG/Stein/Etherpads14:58
emccormickIt's somewhere in these 5000 browser tabs. I'll fire it over to you if I find it ;)14:58
mihalis68_ah yes. So, not insane, just unable to keep my denvers and OS releases straight :)14:58
mihalis68_thanks Sean!14:58
smcginnisOr are you thinking here? https://wiki.openstack.org/wiki/Operations/Meetups14:58
mihalis68_what sean linked14:59
smcginnisStop living in the past man!! :)14:59
mihalis68_hey we run mitaka14:59
emccormickI have this relic around still for a couple of edge cases.15:00
emccormickEverything else is Pike, and about to be Queens15:00
emccormickand when I get home from PTG, maybe to Rocky if I get adventurous15:00
emccormickThis Mitaka will live for a while tho15:01
mihalis68_we have a lot running now15:01
emccormickJust built the Queens containers yesterday. took 2 hours *heh*15:01
mihalis68_our new build will be Queens or later, but getting the apps to move will be a long process15:01
mihalis68_ah containers... maybe one day15:02
emccormickI did an Ocata->Pike upgrade with Kolla and it went fairly well.15:03
emccormickdoing a Pike->Queens soon. We'll see15:03
*** jamesmcarthur has joined #openstack-operators15:03
emccormickok, off to fight other fires. Ping me on Signal if you need anything15:04
mihalis68_will do, thanks again15:06
*** jamesmcarthur has quit IRC15:08
*** gyee has joined #openstack-operators15:11
kendallwatersemccormick: sorry i missed this meeting (forgot to put it on my calendar). saw your message about the ops evening event. i'll add to my email that i'm about to send to all registered PTG attendees! :)15:24
*** spartakos has joined #openstack-operators15:24
emccormickkendallwaters: Thanks! :). See you in Denver.15:28
*** Caterpillar2 has quit IRC15:41
*** macza has joined #openstack-operators15:42
*** pcaruana has quit IRC15:42
spotzThanks kendallwaters!15:45
*** jamesmcarthur has joined #openstack-operators15:49
*** pcaruana has joined #openstack-operators16:01
smcginnisI got confirmation if we want to reserve rooms via the PTGbot, we can do that now without it getting cleared out before the event starts.16:10
smcginnisSo if we need to reserve any of the reservable rooms for Wed-Fri or something, that can be done now.16:11
*** verdurin has quit IRC16:13
*** jamesmcarthur has quit IRC16:14
*** verdurin has joined #openstack-operators16:18
*** mriedem is now known as mriedem_away16:19
*** jamesmcarthur has joined #openstack-operators16:21
*** aojea has quit IRC16:37
*** jamesmcarthur has quit IRC16:37
*** jamesmcarthur has joined #openstack-operators16:39
*** derekh has quit IRC16:55
*** jamesmcarthur has quit IRC17:04
*** gkadam__ has quit IRC17:09
*** spartakos has quit IRC17:11
*** ChanServ sets mode: -rf 17:15
*** jamesmcarthur has joined #openstack-operators17:17
*** jamesmcarthur_ has joined #openstack-operators17:17
*** jamesmcarthur has quit IRC17:21
*** vijaykc4 has joined #openstack-operators17:22
*** pcaruana has quit IRC17:27
*** spartakos has joined #openstack-operators17:36
*** med_ has quit IRC17:38
*** jamesmcarthur_ has quit IRC17:39
*** mvenesio has joined #openstack-operators17:40
*** mriedem_away is now known as mriedem17:41
*** penick has joined #openstack-operators17:51
*** jamesmcarthur has joined #openstack-operators17:56
*** med_ has joined #openstack-operators17:59
*** electrofelix has quit IRC17:59
*** mrhillsman has joined #openstack-operators18:01
*** pcaruana has joined #openstack-operators18:02
*** mvenesio has quit IRC18:10
*** mvenesio has joined #openstack-operators18:11
*** spartakos has quit IRC18:15
*** melwitt is now known as melwitt_awaysick18:18
*** med_ has quit IRC18:34
*** jamesmcarthur has quit IRC18:40
*** jamesmcarthur has joined #openstack-operators18:50
*** med_ has joined #openstack-operators18:54
*** vijaykc4 has quit IRC19:08
mriedemhey guess what everyone, grenade is currently running queens -> master and skipping rocky,19:10
mriedemyou know what that means,19:10
mriedemskip-level upgrade mission accomplished19:10
*** spartakos has joined #openstack-operators19:30
*** pcaruana has quit IRC19:35
*** bollig has quit IRC19:37
*** bollig has joined #openstack-operators19:37
*** amrith has joined #openstack-operators20:17
*** jamesmcarthur has quit IRC20:27
*** nicolasbock has quit IRC20:34
*** nicolasbock has joined #openstack-operators20:54
*** dims has quit IRC20:55
mriedemgonna want ops to read this http://lists.openstack.org/pipermail/openstack-dev/2018-September/134395.html20:59
mriedemmnaser: ^20:59
mnasermriedem: you know a second that just came up reading that is that21:00
mnasermost deployment tools might not even report that failure21:00
mnaserso beyond grenade everything else will look ok until users upgrade and stuff breaks21:00
mnaserbecause the db is empty anyways when the service is first started21:00
mnaseri'll wait before biting21:01
mriedemwell,21:02
mriedemgrenade has a thing where it stores off some data from the rocky side, and verifies it's still on the stein side,21:03
mriedemdansmith added that last week21:03
mriedemso that's how we know in grenade if we f'ed up21:03
*** drewn3ss has joined #openstack-operators21:03
mriedemi would expect any other deployment project that is supporting upgrades and actually testing them to do something similar21:03
dansmithany other tool will have to handle this upgrade21:07
dansmithit's not going to be as easy as some other things21:07
dansmithusually we'd hope to do some post-upgrade sideways movement with a separate outage window,21:07
dansmithbut that's not going to be a thing here,21:07
dansmithso we'll need the tooling to know about this being a thing21:07
*** dims has joined #openstack-operators21:11
*** mvenesio has quit IRC21:18
*** mvenesio has joined #openstack-operators21:19
*** mvenesio has quit IRC21:25
*** med_ has quit IRC21:53
*** mriedem is now known as mriedem_away22:02
*** mriedem_away has quit IRC22:02
*** rcernin has joined #openstack-operators22:29
*** slaweq_ has joined #openstack-operators23:02
*** slaweq has quit IRC23:03
*** drewn3ss has quit IRC23:06
*** drewn3ss has joined #openstack-operators23:07
*** macza has quit IRC23:08
*** mihalis68_ has quit IRC23:11
*** med_ has joined #openstack-operators23:48

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!