| *** threestrands has quit IRC | 00:06 | |
| *** VW_ has joined #openstack-operators | 00:45 | |
| *** med_ has quit IRC | 01:05 | |
| *** med_ has joined #openstack-operators | 01:06 | |
| *** VW_ has quit IRC | 01:19 | |
| *** med_ has quit IRC | 01:21 | |
| *** rcernin has quit IRC | 01:24 | |
| *** rcernin has joined #openstack-operators | 01:24 | |
| *** med_ has joined #openstack-operators | 01:27 | |
| *** VW_ has joined #openstack-operators | 01:51 | |
| *** VW_ has quit IRC | 02:23 | |
| *** VW_ has joined #openstack-operators | 02:24 | |
| *** VW_ has quit IRC | 02:32 | |
| *** VW_ has joined #openstack-operators | 02:36 | |
| *** VW_ has quit IRC | 02:41 | |
| *** VW_ has joined #openstack-operators | 02:45 | |
| *** VW_ has quit IRC | 02:49 | |
| *** VW_ has joined #openstack-operators | 02:54 | |
| *** VW_ has quit IRC | 03:02 | |
| *** VW_ has joined #openstack-operators | 03:03 | |
| *** VW_ has quit IRC | 03:11 | |
| *** VW_ has joined #openstack-operators | 03:15 | |
| *** VW_ has quit IRC | 03:19 | |
| *** med_ has quit IRC | 04:06 | |
| *** Bhujay has joined #openstack-operators | 04:33 | |
| *** mugsie has quit IRC | 05:49 | |
| *** Bhujay has quit IRC | 06:16 | |
| *** rcernin has quit IRC | 06:33 | |
| *** Bhujay has joined #openstack-operators | 06:38 | |
| *** simon-AS559 has joined #openstack-operators | 06:59 | |
| *** simon-AS559 has quit IRC | 07:10 | |
| *** simon-AS559 has joined #openstack-operators | 07:39 | |
| *** simon-AS5591 has joined #openstack-operators | 07:41 | |
| *** simon-AS559 has quit IRC | 07:43 | |
| *** derekh has joined #openstack-operators | 08:28 | |
| *** electrofelix has joined #openstack-operators | 09:14 | |
| *** mugsie has joined #openstack-operators | 10:47 | |
| openstackgerrit | lei zhang proposed openstack/operations-guide master: Update common content for Rocky https://review.openstack.org/599563 | 10:56 |
|---|---|---|
| *** takamatsu has joined #openstack-operators | 11:00 | |
| *** dims has joined #openstack-operators | 11:56 | |
| *** med_ has joined #openstack-operators | 12:13 | |
| *** jamesmcarthur has joined #openstack-operators | 12:46 | |
| jamesmcarthur | Hello Operators! I am unfortunately unable to make the meeting again this week. If you all have any questions or concerns around the upcoming PTG, please shout at me. | 12:50 |
| jamesmcarthur | Very much looking forward to seeing everyone in Denver! | 12:50 |
| *** mriedem has joined #openstack-operators | 12:53 | |
| *** r-mibu has joined #openstack-operators | 13:01 | |
| *** jamesmcarthur has quit IRC | 13:05 | |
| *** jamesmcarthur has joined #openstack-operators | 13:45 | |
| *** aojea has joined #openstack-operators | 13:57 | |
| *** kendallwaters has joined #openstack-operators | 14:01 | |
| * smcginnis looks around | 14:01 | |
| spotz | take care jamesmcarthur | 14:01 |
| spotz | hey smcginnis | 14:01 |
| emccormick | o/ | 14:01 |
| *** mihalis68_ has joined #openstack-operators | 14:02 | |
| *** shintaro has joined #openstack-operators | 14:02 | |
| smcginnis | Do we have mihalis68_ or shintaro? | 14:02 |
| shintaro | hi | 14:02 |
| emccormick | good morning everyone. | 14:02 |
| emccormick | (they just joined) | 14:02 |
| mihalis68_ | I'm here | 14:02 |
| emccormick | afk for 2 minutes. Go ahead and start without me. | 14:02 |
| mihalis68_ | #startmeeting ops_meetup_team | 14:02 |
| openstack | Meeting started Tue Sep 4 14:02:58 2018 UTC and is due to finish in 60 minutes. The chair is mihalis68_. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:02 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:02 |
| *** openstack changes topic to " (Meeting topic: ops_meetup_team)" | 14:03 | |
| openstack | The meeting name has been set to 'ops_meetup_team' | 14:03 |
| mihalis68_ | I haven't updated the agenda yet, but I did take a first stab at laying out session suggestions from the etherpad on the agenda template on the google doc | 14:03 |
| mihalis68_ | We can obviously move and change individual items, however the biggest challenge I saw is that it's been suggested that all ops should just go to the upgrades sig | 14:05 |
| mihalis68_ | well when I checked the event calendar, that's the whole afternoon on the Monday | 14:05 |
| shintaro | looks great | 14:06 |
| mihalis68_ | which to me seems to vaporise a huge chunk of our day | 14:06 |
| mihalis68_ | I'd like to hear what people think about that. I've tentatively put containerisation in parallel with upgrades sig since my feeling is those who are containerised don't have the severe upgrade problems that those us who are non-containerised have. | 14:07 |
| spotz | link?:) | 14:07 |
| mihalis68_ | one sec, I have to switch to my laptop becuase of policy limitations here | 14:07 |
| shintaro | #link https://docs.google.com/spreadsheets/d/1EUSYMs3GfglnD8yfFaAXWhLe0F5y9hCUKqCYe0Vp1oA/edit#gid=981527336 | 14:07 |
| mihalis68_ | #link https://docs.google.com/spreadsheets/d/1EUSYMs3GfglnD8yfFaAXWhLe0F5y9hCUKqCYe0Vp1oA/edit#gid=981527336 | 14:07 |
| mihalis68_ | choose the DEN tab if it didn't land on that | 14:07 |
| emccormick | I disagree with that assertion. Containerized doesn't really eliminate upgrade pain on its own | 14:07 |
| emccormick | progress has been made in those projects to simplify it, and it's easier to bail out if things explode | 14:08 |
| mihalis68_ | but the problem is if we acccept a) upgrade sig is all afternon and b) ops should just go to that, that consumes 5 slots from the day | 14:08 |
| emccormick | but the Openstack-specific points are still in play | 14:08 |
| spotz | Glad I asked:) Ok SUSE person won't be there until Tuesday | 14:09 |
| emccormick | Even worse, I would note that you have less slots than you marked down I think? | 14:09 |
| emccormick | should end at 5 | 14:09 |
| mihalis68_ | I simply used the template | 14:10 |
| emccormick | at least based on the fact that the happy hour on Tuesday starts at 5 | 14:10 |
| mihalis68_ | why must we end at 5>? | 14:10 |
| emccormick | yes I know :) | 14:10 |
| shintaro | so here's upgrade-sig etherpad #link https://etherpad.openstack.org/p/upgrade-sig-ptg-stein | 14:10 |
| shintaro | do you think we all need to attend all of them? | 14:10 |
| shintaro | maybe aftger 15:15 | 14:11 |
| emccormick | Not really a lot going on there, huh? | 14:11 |
| mihalis68_ | Thanks for the link Shintaro, that's a smaller class it seems. | 14:11 |
| mihalis68_ | Shall we work from the top on this itinerary? | 14:11 |
| emccormick | we don't *have* to end at 5, but I think everyone else wraps up then, and afterstack things are planned to start then | 14:11 |
| mihalis68_ | I put gerrit/git first and the docs stuff. Reason being docs was listed twice and would be better if done after the gerrit intro so that ops feel they know how to contribute | 14:12 |
| mihalis68_ | any objection to these first two? | 14:13 |
| mihalis68_ | spotz are you up for moderating the very first session? :) | 14:13 |
| emccormick | Those seem fine to me | 14:13 |
| shintaro | +1 | 14:13 |
| emccormick | so long as spotz can come do it at that time :) | 14:13 |
| spotz | Yep, I'm trying to keep the frst 2 days for ops meetup | 14:13 |
| mihalis68_ | I've removed the "upgrades sig" from the first two slots as it hasn't started then | 14:14 |
| emccormick | let's flip running old openstack with the Suse thing | 14:14 |
| emccormick | since they can't do Tuesday, and we might get stuck on the old openstack topic longer than we think | 14:14 |
| mihalis68_ | done | 14:15 |
| * mnaser is lurking | 14:15 | |
| emccormick | and we can have our feedback session in the bar if needs be ;) | 14:15 |
| emccormick | seems appropriate to discuss Berlin over beer | 14:15 |
| emccormick | actually now that I look at it, going to 17:40 is fine. We can go longer that day since our evening thing that night is Ops | 14:16 |
| spotz | And if Nicolas ends up having a confict I'm sure we can put something together | 14:16 |
| spotz | Nicolas= suse:) | 14:17 |
| *** Bhujay has quit IRC | 14:17 | |
| mihalis68_ | Nicolas ... who? | 14:17 |
| mihalis68_ | I'm filling in moderators to the right of the agenda table | 14:17 |
| emccormick | heh. I can't edit | 14:18 |
| emccormick | You can put Julia Krieger in for bare metal | 14:18 |
| emccormick | she put that session in herself | 14:19 |
| mihalis68_ | done | 14:19 |
| spotz | mihalis68_: Nicolas Bock | 14:19 |
| emccormick | I guess you can give me the containerized stuff | 14:19 |
| mihalis68_ | Thanks! | 14:19 |
| emccormick | If we find someone else to do that, then great | 14:20 |
| emccormick | I'll ping Starmer and see if he's coming. | 14:20 |
| mihalis68_ | Yeah! Good call | 14:20 |
| mihalis68_ | #action emccormick to check with r.starmer re: ptg attendance and possible moderating container-related session | 14:21 |
| *** VW_ has joined #openstack-operators | 14:21 | |
| mihalis68_ | so we're making progress | 14:21 |
| mihalis68_ | we do still have something of a clash on day 1. After 4pm particularly, i pencilled in two sessions for ops, but upgrades sig should be running by then | 14:22 |
| mihalis68_ | I put myself down to moderate ceph again (that will be like 3rd time) | 14:22 |
| mihalis68_ | The ceph topic was fairly well fleshed out on the etherpad. Clearly at least two or more people want it had specific things to discuss | 14:23 |
| emccormick | spiffy. You can regale us with stories of broken RGW shards | 14:23 |
| emccormick | So for working in upgrades | 14:24 |
| mihalis68_ | I may cut and paste the Tokyo etherpad :) | 14:24 |
| emccormick | we have a huge gap at the top | 14:24 |
| shintaro | on the planning etherpad, Simon Leinen (simon-AS559) says he can do Ceph | 14:24 |
| emccormick | What say you to stripping the registration period down? | 14:24 |
| emccormick | Isn't that kind of built in to the beginning of PTG anyway prior to 9am? | 14:25 |
| mihalis68_ | do you know him, Shintaro? If he would like to then great | 14:25 |
| mihalis68_ | Ooh! I don't actually know. Can someone confirm? | 14:25 |
| shintaro | no I don't :( | 14:25 |
| mihalis68_ | that would help a lot | 14:25 |
| emccormick | spotz is there registration / coffee starting at 9, or are sessions starting at that time with registration prior? | 14:26 |
| mihalis68_ | I'll put him down with me as backup | 14:26 |
| spotz | emccormick: Registration is usually going earlier | 14:26 |
| shintaro | I can email him if he is coming and if he is still willing to moderate. | 14:27 |
| emccormick | so we can take that 90 minutes, shift everything before 15:05 up, and use that block to go talk upgrades | 14:27 |
| emccormick | Thoughts? | 14:27 |
| smcginnis | Just so there is a reference in the logs: | 14:28 |
| smcginnis | #link http://ptg.openstack.org/ptg.html | 14:28 |
| emccormick | We can have a meet and greet from 9-9:30 maybe, or 9:15 | 14:28 |
| smcginnis | That sounds good. | 14:28 |
| spotz | That would probably be a nice thing to do | 14:28 |
| smcginnis | Might be nice to have some time to meet everyone. | 14:29 |
| emccormick | We can do some yoga and coffee *hehe* | 14:30 |
| mihalis68_ | I've put something in the google doc about this | 14:30 |
| mihalis68_ | just pondering whether to attempt rejigging the whole thing during this meeting, might be a bad idea | 14:30 |
| emccormick | also lunch appears to be 12:30 - 13:30? | 14:31 |
| emccormick | so we gain another 30 minutes there | 14:31 |
| emccormick | mihalis68_ I can't think of a better time. We have everyone together, and I can't think of a more important topic. | 14:32 |
| emccormick | And after this I have to descend back into Rabbitmq / Cinder hell :D | 14:32 |
| mihalis68_ | I can rejig it later | 14:35 |
| mihalis68_ | we have a slot for lunch on monday and then coffee mid-afternoon | 14:36 |
| mihalis68_ | tuesday has coffee, lunch, coffee | 14:36 |
| *** jamesmcarthur has quit IRC | 14:36 | |
| mihalis68_ | those were specifically catered breaks in previous meetups. Does anyone know what the coffee arrangements are for PTG? | 14:36 |
| emccormick | lunch 12:30 - 1:30 | 14:37 |
| smcginnis | There's almost always coffee available. | 14:37 |
| emccormick | I think we just have coffee in our little anteroom | 14:37 |
| smcginnis | And lunch is all available on site with buffet stations and plenty of seating. | 14:37 |
| emccormick | so no need to take a long walk, but good for a bio break | 14:37 |
| smcginnis | Yep | 14:37 |
| mihalis68_ | should we insert coffee in day 1 morning? | 14:38 |
| mihalis68_ | weirdly it's not in the template | 14:38 |
| smcginnis | Might be good to plan a break. | 14:38 |
| mihalis68_ | as if we don't need morning coffee until day 2 | 14:38 |
| mihalis68_ | clearly false | 14:38 |
| spotz | Yeah even if folks don't need a physical break might need a mental one | 14:38 |
| emccormick | yeah that's fine. | 14:38 |
| emccormick | While Chris is editing, I wanted to get feedback on how far is too far to go for Monday night. | 14:39 |
| emccormick | There are a couple of options downtown (7+ miles away) that are interesting. There's one closeish that's about a 1.5 miles away | 14:40 |
| kendallwaters | FYI - there are not set coffee breaks but coffee will be served from 8:30am - 5pm every day | 14:40 |
| emccormick | meeting the criteria of activities other than boozing. | 14:40 |
| smcginnis | Thanks kendallwaters | 14:40 |
| emccormick | mmm | 14:40 |
| spotz | emccormick: Distance is dependent on how many people I think. It's easier to say get 2-3 Ubers then 10 | 14:40 |
| emccormick | We went quite a ways for Tokyo, and I didn't mind. I dunno if anyone else did though. | 14:41 |
| spotz | I'm game | 14:42 |
| mihalis68_ | I've pulled the operators-submitted items earlier, so there's only a partial clash with upgrades sig | 14:42 |
| *** dtrainor has joined #openstack-operators | 14:42 | |
| smcginnis | Might be nice to get away from the immediate area a litt.e | 14:43 |
| emccormick | I pondered this place here | 14:43 |
| emccormick | https://www.wynkoop.com/ | 14:43 |
| emccormick | They have an upstairs with free pool / darts / foozball | 14:43 |
| emccormick | and for me and those who like, delicious beeeeer | 14:43 |
| mihalis68_ | we have 15 minutes left. In addition to nailing down the social, will anyone have time to take one more run through the agenda a bit later this week? If so I'd like to pencil in a time | 14:44 |
| smcginnis | Looks cool emccormick | 14:44 |
| emccormick | it also looks enormous and it's a Monday so I doubt we'll have crowd problems | 14:44 |
| smcginnis | mihalis68_: Want to review on Thursday or Friday? | 14:44 |
| mihalis68_ | yes that would be good. | 14:45 |
| mihalis68_ | If there is consensus that the google doc is a good draft I can fine tune it | 14:45 |
| emccormick | I will continue looking at it and send suggestions. I can do either day to catch up on here | 14:45 |
| mihalis68_ | oh yes, another topic that occurred to me: the organisers have actually given the operators a room all week | 14:45 |
| smcginnis | I'll probably be around and able to spend a little time later in the week doing a final walk through. | 14:45 |
| mihalis68_ | wynkoop looks good to me | 14:46 |
| spotz | mihalis68_: I'll be about | 14:46 |
| mihalis68_ | so for finalising the agenda, I propose this: I email the operators mailing list with the link and ask for feedback emphasizing it's a draft | 14:46 |
| mihalis68_ | on that email chain we incorporate reasonable feedback and also pick out a time to finish it | 14:47 |
| emccormick | spotz You're ok going at 9:30 for tools stuff?. Don't want to take you away from other things if we don't have to. | 14:47 |
| mihalis68_ | then come back here to actually do that later in the week> | 14:47 |
| mihalis68_ | ? | 14:47 |
| spotz | mihalis68_: Maybe cut and paste into the email in case folks can't reach google | 14:47 |
| emccormick | +1 | 14:47 |
| mihalis68_ | I'll attach a PDF of it | 14:47 |
| emccormick | to both of you | 14:47 |
| mihalis68_ | is there any interest in having a 3rd day? Maybe a partial? Since we have the room? | 14:48 |
| spotz | emccormick: I should be good until Wed AM unless there's a UC/OSA/MEntoring/D&I/etc emergency:) | 14:48 |
| mihalis68_ | I know there's lot of other interesting stuff going on | 14:48 |
| emccormick | I'm definitely game for that | 14:48 |
| kendallwaters | FYI - ops only has a room Monday and Tuesday | 14:48 |
| *** aprice has joined #openstack-operators | 14:48 | |
| kendallwaters | There are rooms you can reserve if you wish to meet Wednesday - Friday | 14:48 |
| mihalis68_ | Ah ok, it seemed like we have it all week from what I saw | 14:48 |
| emccormick | kendallwaters We can take it to the bar :) | 14:48 |
| spotz | UC is planning on meeting Wednesday, so besides the one session we have if anyone needs to talk with us let us know | 14:49 |
| emccormick | and yeah the schedule lists us in it all week | 14:49 |
| emccormick | probably a misprint though. Just supposed to be empty I imagine | 14:49 |
| mihalis68_ | #action mihalis68 to share draft agenda with community. | 14:49 |
| kendallwaters | What schedule are you looking at? The one online just shows you Monday and Tuesday | 14:49 |
| emccormick | oh now it says it's empty | 14:49 |
| emccormick | I swear it wasn't like that just yesterday ;) | 14:49 |
| smcginnis | Both schedules I can find only show Mon-Tue. | 14:49 |
| *** VW_ has quit IRC | 14:50 | |
| emccormick | NM, I must have been hallucinating. Carry on | 14:50 |
| smcginnis | There's https://www.openstack.org/ptg/#tab_schedule and http://ptg.openstack.org/ptg.html | 14:50 |
| emccormick | The room is just unbooked thus far | 14:50 |
| emccormick | The latter is what I was looking at. I just must have done something strange with my tab clicking or something. I see it properly now. | 14:50 |
| mihalis68_ | Aspen is empty, Wed-Fri, yeah | 14:51 |
| emccormick | It's not currently assigned to anything, including us. | 14:51 |
| mihalis68_ | I could have sworn someone said here the room would be available to us all week, but now I'm not sure | 14:51 |
| kendallwaters | yes, it is empty and there won't be a screen and projector Wed-Fri (saving money that way) | 14:51 |
| emccormick | There's 6 empty rooms Wednesday so we'll assess interest in continuing when we get to it | 14:51 |
| emccormick | we can keep our paper pad right? :) | 14:52 |
| kendallwaters | haha!! :) | 14:52 |
| emccormick | or whatever it is that'll be in the outer room | 14:52 |
| emccormick | without it, how ever will we play pictionary? | 14:52 |
| kendallwaters | yes, there is a flip chart or whiteboard in every room so that is no problem if you plan to meet wednesday, thursday, or friday | 14:52 |
| mihalis68_ | should we tentatively agreed Erik's venue for the social? It seems to tick enough boxes | 14:53 |
| smcginnis | Good option if we need any kind of follow up. | 14:53 |
| mihalis68_ | that way I can mention it on the email to the mailing list | 14:53 |
| smcginnis | mihalis68_: +1 works for me. | 14:53 |
| mihalis68_ | I'll push the ceph talk to overflow day | 14:53 |
| mihalis68_ | and now we don't clash with upgrades sig | 14:53 |
| kendallwaters | you can easily reserves room via the ptgbot or you can visit me at the reg desk! :) | 14:53 |
| emccormick | I may srsly just buy something like this and start bringing it along | 14:54 |
| emccormick | https://www.amazon.com/Crenova-XPE496-Upgraded-Projector-Lumens/dp/B07CDD93KG/ref=sr_1_3_sspa?s=electronics&ie=UTF8&qid=1536072814&sr=1-3-spons&keywords=hdmi+projector&psc=1&smid=AGE8XQM095I52 | 14:54 |
| mihalis68_ | maybe we should make "overflow day" friday afternoon some time? Then it can be somehting of a recap | 14:54 |
| emccormick | $100? done deal | 14:54 |
| mihalis68_ | 3.68lbs | 14:54 |
| mihalis68_ | luggable | 14:54 |
| emccormick | indeed | 14:54 |
| mihalis68_ | that's less than one additional laptop | 14:55 |
| mihalis68_ | I took two to Vancouver the first time, so definitely doable | 14:55 |
| emccormick | shoot it up on the wall | 14:55 |
| emccormick | easy | 14:55 |
| mihalis68_ | I'll mention Wynkoop as the current suggestion then and see if people agree | 14:55 |
| emccormick | Let's mention the possibility of continued discussion in days after as a general thing then see how things play out when we get there. | 14:56 |
| mihalis68_ | we seem to be almost out of time and definitely out of steam. | 14:56 |
| spotz | Sounds good | 14:56 |
| emccormick | I don't know what % of people are staying all week, or even how many people we'll actually have | 14:56 |
| mihalis68_ | any more business? Sorry for the lack of agenda today | 14:56 |
| kendallwaters | i'll be sending out an email with a ton of PTG information, if y'all want me to include something about y'all's social event, let me know! | 14:57 |
| emccormick | No apologies. this has been productive | 14:57 |
| mihalis68_ | my boss scheduled status immediately before this meeting ... | 14:57 |
| emccormick | Do we want to nail down a regroup time? | 14:57 |
| mihalis68_ | kendallwaters that would be great | 14:57 |
| mihalis68_ | sean said thursday or friday, erik | 14:57 |
| emccormick | same time Thursday? | 14:57 |
| mihalis68_ | which works for me | 14:57 |
| smcginnis | Thursday morning should be good for me. | 14:57 |
| mihalis68_ | works for me | 14:58 |
| spotz | Just ping me in channel, I'll pop in:) | 14:58 |
| emccormick | ok, 10am EDT Thursday regroup | 14:58 |
| mihalis68_ | alright. I think we are done. Some of us will regroup here in two days at 10 AM ESRT | 14:58 |
| mihalis68_ | EST | 14:58 |
| mihalis68_ | other than that, see you in Denver! | 14:58 |
| mihalis68_ | thanks everyone | 14:58 |
| smcginnis | Looking forward to seeing everyone. | 14:58 |
| mihalis68_ | yeah! | 14:58 |
| emccormick | spiffy! | 14:58 |
| mihalis68_ | #endmeeting | 14:59 |
| *** openstack changes topic to "Discussion of topics related to operating Openstack infrastructures at scale | Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators | Logs: http://eavesdrop.openstack.org/irclogs/" | 14:59 | |
| openstack | Meeting ended Tue Sep 4 14:59:00 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:59 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2018/ops_meetup_team.2018-09-04-14.02.html | 14:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/ops_meetup_team/2018/ops_meetup_team.2018-09-04-14.02.txt | 14:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2018/ops_meetup_team.2018-09-04-14.02.log.html | 14:59 |
| *** jamesmcarthur has joined #openstack-operators | 14:59 | |
| emccormick | hey smcginnis you have a sec? Can you answer a stupid cinder question for me? | 15:00 |
| emccormick | it might be a stupid rabbit question, but it's Cinder-related ;) | 15:00 |
| *** shintaro has quit IRC | 15:00 | |
| smcginnis | I can try. What's happening? | 15:00 |
| emccormick | network outage last week has left cinder able to send attach / detach requests to nova-compute, and nova-compute processes them. However responses vaporize. There are oslo message timeouts. | 15:01 |
| emccormick | result: volumes show "attaching" or "detaching" eternally even though they are truly attached, detached. | 15:01 |
| emccormick | all things have been restarted, load balancers removed from the mix and Rabbit appears to process other things | 15:02 |
| emccormick | api queries work fine, create / delete works fine | 15:02 |
| emccormick | not sure of the workflow to know where it's dying. | 15:03 |
| emccormick | possibly some corruption in Rabbit? | 15:03 |
| smcginnis | That does sound like a message queue communication failure. | 15:04 |
| emccormick | all things show connected to it, so it at least things its working properly. It's single rabbit so no partitioned cluster problems | 15:04 |
| smcginnis | Anything in any of the log files? I would expect some errors and tracebacks. | 15:04 |
| emccormick | yes a couple. one sec | 15:06 |
| spotz | Sounds like you need to force detach and then re-attach emccormick | 15:07 |
| emccormick | this in api.log | 15:07 |
| emccormick | MessagingTimeout: Timed out waiting for a reply to message ID f05e37248d474810b020f6ee5c336ccc | 15:07 |
| emccormick | I've actually had to reset the detaching ones to in-use and try again | 15:07 |
| emccormick | but they go back to detaching | 15:07 |
| emccormick | very vague I know | 15:08 |
| emccormick | I also have a volume that someone attached after I had detached it. Cinder believed it to still have a device ID, and the OS of the instance did not. So now it's attached as /dev/vdg but Cinder thinks it's on /dev/vdk :D | 15:09 |
| smcginnis | That message would indicate a MQ communication failure. | 15:09 |
| emccormick | another problem, but I can sort that out | 15:09 |
| smcginnis | I would focus there and verify all your networking is working as expected to start. | 15:09 |
| emccormick | yeah, that's what I'm after. Just that the API process is on the same server as rabbit on this cluster and shows connected. Other things process messages. Very odd | 15:10 |
| smcginnis | Huh, that is odd. | 15:10 |
| emccormick | indeed | 15:11 |
| smcginnis | I'm sure it will end up being something trivial in the end, but it's not immediately obvious where the breakdown is. | 15:11 |
| emccormick | so basically the flow is after nova-compute processes the detach, it fires a message on the queue to cinder-api to let it know its done? | 15:11 |
| emccormick | or is there another stop in there? | 15:11 |
| emccormick | (keep in mind this is Mitaka) | 15:11 |
| emccormick | Oh I know. I'll kick myself for not seeing it when I finally find it I"m sure | 15:12 |
| emccormick | but I've been an idiot all weekend, so figured I'd bounce it off someone and see if they told me how I was being an idiot ;) | 15:12 |
| smcginnis | I believe the nova>cinder communication is via the REST API only. | 15:13 |
| emccormick | That kinda makes sense then. I see that API request hang I think | 15:14 |
| emccormick | so then I wonder what message it's waiting for a response to and from which process | 15:14 |
| smcginnis | And that api log message would make it look like it is getting a request and trying to call through the MQ to the scheduler but failing. | 15:14 |
| emccormick | so it goes cinder-api->cinder-scheduler->nova-compute->cinder-api? | 15:15 |
| emccormick | or cinder-scheduler goes to nova api | 15:16 |
| smcginnis | Are you doing the attach from the cinder side or from nova? | 15:16 |
| emccormick | nova | 15:16 |
| emccormick | openstack server add volume | 15:16 |
| smcginnis | Usually nova>cinder-api>cinder-scheduler or nova>cinder-api>cinder-vol, depending on the operation. | 15:17 |
| emccormick | which I think translates to nova attach-volume | 15:17 |
| smcginnis | I can't recall in mitaka if the attach call would go through the scheduler. | 15:17 |
| emccormick | This is before the time of global request ids so following it is a PITA ;) | 15:17 |
| smcginnis | Make sure all the cinder services are running. Maybe the c-vol service is down? | 15:17 |
| smcginnis | Heh, yeah. | 15:18 |
| emccormick | it's up and reporting normally | 15:18 |
| emccormick | I did find scheduler down early on, but that now seems to function, at least in its own mind | 15:18 |
| emccormick | ah well, I'll keep running scenarios and see what else comes out | 15:19 |
| smcginnis | OK, if I can spend some time later I will see if any other ideas come to mind. | 15:19 |
| smcginnis | But does appear to be a rabbit problem somewhere, whether with rabbit itself or getting to rabbit to communicate with the other services. | 15:19 |
| emccormick | been stripping out intermediary things. When the load balancer was in the way, I was getting a null response back to nova-compute from cinder-api but it turned out ot just be a timeout | 15:20 |
| emccormick | I may just try reinitializing all the queues | 15:20 |
| emccormick | let it rebuild itself from scratch and see if that makes it happier. | 15:20 |
| *** openstackgerrit has quit IRC | 15:20 | |
| emccormick | Thanks for taking the time. I appreciate it :) | 15:21 |
| emccormick | Haven't had a cinder problem like ever, but really it's just another Rabbit problem ;) | 15:21 |
| smcginnis | Heh, it is usually all good until something odd happens. Good luck! :) | 15:25 |
| *** jamesmcarthur has quit IRC | 15:43 | |
| *** jamesmcarthur has joined #openstack-operators | 15:43 | |
| *** aojea has quit IRC | 15:52 | |
| *** macza has joined #openstack-operators | 16:08 | |
| *** Bhujay has joined #openstack-operators | 16:17 | |
| *** simon-AS5591 has quit IRC | 16:18 | |
| *** gyee has joined #openstack-operators | 16:24 | |
| *** derekh has quit IRC | 17:01 | |
| *** r-mibu has quit IRC | 17:02 | |
| *** Adri2000 has quit IRC | 17:16 | |
| *** Adri2000 has joined #openstack-operators | 17:19 | |
| *** med_ has quit IRC | 17:21 | |
| *** med_ has joined #openstack-operators | 17:24 | |
| *** Bhujay has quit IRC | 17:31 | |
| *** electrofelix has quit IRC | 17:58 | |
| *** jamesmcarthur has quit IRC | 18:12 | |
| *** jamesmcarthur has joined #openstack-operators | 18:32 | |
| *** jamesmcarthur has quit IRC | 18:36 | |
| *** med_ has quit IRC | 18:49 | |
| *** jamesmcarthur has joined #openstack-operators | 19:11 | |
| *** jamesmcarthur has quit IRC | 19:15 | |
| *** jamesmcarthur_ has joined #openstack-operators | 19:15 | |
| *** jamesmcarthur_ has quit IRC | 19:33 | |
| *** med_ has joined #openstack-operators | 20:13 | |
| *** jamesmcarthur has joined #openstack-operators | 20:21 | |
| *** spartakos has joined #openstack-operators | 22:48 | |
| *** jamesmcarthur has quit IRC | 22:50 | |
| *** jamesmcarthur has joined #openstack-operators | 22:51 | |
| *** jamesmcarthur has quit IRC | 22:52 | |
| *** rcernin has joined #openstack-operators | 22:55 | |
| *** VW_ has joined #openstack-operators | 23:34 | |
| *** jamesmcarthur has joined #openstack-operators | 23:44 | |
| *** jamesmcarthur has quit IRC | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!