| *** AJaeger has quit IRC | 06:23 | |
| *** AJaeger has joined #openstack-infra-incident | 06:35 | |
| *** rosmaita has joined #openstack-infra-incident | 11:34 | |
| *** SteelyDan is now known as dansmith | 13:25 | |
| clarkb | mordred: corvus ianw fungi Just sent followup email re Cody's rax questions to you all. Can you take a look at that reasonably soonish? Seems like this issue has dragged on for ~6 weeks and if we can fix it quickly now that we've been escalated to that will hopefully make people appy | 15:34 |
|---|---|---|
| fungi | thanks, looking | 15:35 |
| fungi | cody ended up back at rackspace? neat! | 15:59 |
| clarkb | ya mordred fwiw I pulled cody off the recipient list so we can game plan a bit befor eresponding :) | 16:00 |
| mordred | oh. hah | 16:03 |
| mordred | clarkb: well - then ignore my hello to cody :) | 16:03 |
| clarkb | https://review.openstack.org/591446 is a related change | 16:09 |
| corvus | i guess we lost that when we switched to dib? | 16:10 |
| clarkb | I think when we switched off of puppet for dib | 16:10 |
| corvus | er yeah, that | 16:11 |
| corvus | cause, i mean, disabling password auth in the system as a whole was *literally* change #1 :) | 16:11 |
| corvus | https://review.openstack.org/#/c/1/ | 16:11 |
| clarkb | nice | 16:11 |
| fungi | that's a fun bit of history | 16:13 |
| clarkb | I'll reach out to cloudnull now and see if I can cc him on response to cody, then suggest that we clean up those servers and move on | 16:15 |
| clarkb | fungi: as for your checks, control plane is different because puppet applies sshd config | 16:15 |
| fungi | yep | 16:15 |
| fungi | i wrote that before i saw the comment in here about puppetless dib | 16:16 |
| clarkb | fungi: my hunch is that ianw was reproducing some test result on these nodes and they ran some service which ended up being compromised | 16:16 |
| fungi | entirely possible as well | 16:16 |
| clarkb | rather than ssh itself being at fault | 16:16 |
| fungi | plenty of potential backdoors after all | 16:16 |
| fungi | serves as a reminder to us all that we should delete temporary test servers when we're done with them | 16:16 |
| clarkb | ++ | 16:17 |
| clarkb | as for account contacts maybe we can put infra-root on there as well as PTL and jbryce? that should give us a decent spread? | 16:17 |
| *** pabelanger has joined #openstack-infra-incident | 16:18 | |
| fungi | yeah, but as i noted, none of them will get notified of problems unless someone in rackspace explicitly reaches out. trouble tickets they create will only end up mailing the internal sponsor, who may not be paying attention to them or may forget we need an explicit heads-up | 16:19 |
| clarkb | got it | 16:19 |
| fungi | it's not like we regularly log into all our cloud accounts and check for new tickets | 16:19 |
| clarkb | I've reached out to cloudnull will respond to Cody as soon as I have a response from Keven (as for CC or not) | 16:20 |
| fungi | yeah, i'm mildly worried they'll notice soon that mvw isn't working there | 16:22 |
| fungi | then again, it took them years to realize jbryce, mordred, pvo, et al were no longer there but were listed as internal sponsors on comped accounts ;) | 16:22 |
| clarkb | kevin says always feel free to CC him when doing rax thing s:) | 16:28 |
| clarkb | alright email sent, I'll wait for ianw to show up in a few hours before deleting thins just to double check (instance is off and suspended anyway) | 16:38 |
| *** rosmaita has quit IRC | 17:43 | |
| *** srwilkers_ has joined #openstack-infra-incident | 19:17 | |
| *** mgagne_ has joined #openstack-infra-incident | 19:24 | |
| *** srwilkers has quit IRC | 19:25 | |
| *** mgagne has quit IRC | 19:25 | |
| *** srwilkers_ is now known as srwilkers | 19:25 | |
| ianw | clarkb: hey, sorry, catching up here | 20:49 |
| ianw | this is the first i've heard of it, let me look at my notes | 20:49 |
| ianw | yeah, around then i was working on https://review.openstack.org/#/c/562004/ | 20:51 |
| clarkb | ianw: want ot follow up on that thread to let kevin and cody know we can delete the instance? (I'm happy for us to do it but unsure if rax wants to do more investigating) | 20:56 |
| ianw | clarkb: yes, just tapping away now, sorry about this | 20:59 |
| clarkb | no problem | 21:01 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!