| *** tetsuro has joined #openstack-infra | 00:22 | |
| *** dklyle has quit IRC | 00:55 | |
| *** david-lyle has joined #openstack-infra | 00:55 | |
| *** Goneri has quit IRC | 01:32 | |
| *** Topner has joined #openstack-infra | 01:48 | |
| *** Topner has quit IRC | 01:53 | |
| *** ysandeep|holiday is now known as ysandeep | 02:12 | |
| *** cyberpear has joined #openstack-infra | 02:46 | |
| *** tetsuro has quit IRC | 03:21 | |
| *** tetsuro has joined #openstack-infra | 03:22 | |
| *** tetsuro has quit IRC | 03:26 | |
| *** elod has quit IRC | 03:35 | |
| *** elod has joined #openstack-infra | 03:37 | |
| *** psachin has joined #openstack-infra | 03:39 | |
| *** Topner has joined #openstack-infra | 03:49 | |
| *** Topner has quit IRC | 03:54 | |
| *** tetsuro has joined #openstack-infra | 04:17 | |
| *** zzzeek has quit IRC | 04:21 | |
| *** ramishra has joined #openstack-infra | 04:22 | |
| *** zzzeek has joined #openstack-infra | 04:24 | |
| *** tetsuro_ has joined #openstack-infra | 04:24 | |
| *** tetsuro has quit IRC | 04:27 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-infra | 04:33 | |
| *** ykarel has joined #openstack-infra | 04:33 | |
| *** vishalmanchanda has joined #openstack-infra | 04:49 | |
| *** cyberpear has quit IRC | 04:55 | |
| *** matt_kosut has joined #openstack-infra | 04:57 | |
| *** abhishekk|away is now known as abhishekk | 05:20 | |
| *** zxiiro has quit IRC | 05:23 | |
| *** Topner has joined #openstack-infra | 05:50 | |
| *** Topner has quit IRC | 05:55 | |
| *** ricolin_ has joined #openstack-infra | 05:56 | |
| *** tkajinam has quit IRC | 06:13 | |
| *** tkajinam has joined #openstack-infra | 06:13 | |
| *** lmiccini has joined #openstack-infra | 06:15 | |
| *** ralonsoh has joined #openstack-infra | 06:16 | |
| *** rcernin has joined #openstack-infra | 06:16 | |
| *** rcernin has quit IRC | 06:22 | |
| *** ricolin_ has quit IRC | 06:23 | |
| *** ricolin_ has joined #openstack-infra | 06:23 | |
| *** eolivare has joined #openstack-infra | 06:40 | |
| *** jtomasek has joined #openstack-infra | 06:53 | |
| *** hashar has joined #openstack-infra | 06:56 | |
| *** slaweq has joined #openstack-infra | 07:03 | |
| *** jcapitao has joined #openstack-infra | 07:09 | |
| *** andrewbonney has joined #openstack-infra | 07:12 | |
| *** ykarel_ has joined #openstack-infra | 07:20 | |
| *** yolanda has quit IRC | 07:21 | |
| *** ykarel has quit IRC | 07:23 | |
| *** ykarel_ is now known as ykarel | 07:23 | |
| *** gfidente has joined #openstack-infra | 07:25 | |
| *** rpittau|afk is now known as rpittau | 07:29 | |
| *** jpena|off is now known as jpena | 07:32 | |
| *** yolanda has joined #openstack-infra | 07:37 | |
| *** tosky has joined #openstack-infra | 07:42 | |
| *** ykarel_ has joined #openstack-infra | 07:48 | |
| *** Topner has joined #openstack-infra | 07:51 | |
| *** ykarel has quit IRC | 07:51 | |
| *** Topner has quit IRC | 07:55 | |
| *** lucasagomes has joined #openstack-infra | 07:56 | |
| *** ociuhandu has joined #openstack-infra | 07:57 | |
| *** ykarel_ is now known as ykarel | 08:02 | |
| *** ricolin_ has quit IRC | 08:26 | |
| *** derekh has joined #openstack-infra | 08:27 | |
| *** dtantsur|afk is now known as dtantsur | 08:32 | |
| *** rcernin has joined #openstack-infra | 08:41 | |
| *** rcernin has quit IRC | 08:46 | |
| *** Ajohn has joined #openstack-infra | 08:47 | |
| *** tetsuro_ has quit IRC | 09:07 | |
| *** ykarel_ has joined #openstack-infra | 09:09 | |
| *** ykarel has quit IRC | 09:12 | |
| *** tetsuro has joined #openstack-infra | 09:14 | |
| *** tetsuro has quit IRC | 09:16 | |
| *** ykarel_ has quit IRC | 09:18 | |
| *** ykarel_ has joined #openstack-infra | 09:19 | |
| *** ykarel_ is now known as ykarel | 09:21 | |
| *** ykarel has quit IRC | 09:42 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:47 | |
| *** Topner has joined #openstack-infra | 09:52 | |
| *** ykarel has joined #openstack-infra | 09:54 | |
| *** Topner has quit IRC | 09:57 | |
| *** pmannidi has joined #openstack-infra | 09:57 | |
| *** david-lyle has quit IRC | 10:24 | |
| *** priteau has joined #openstack-infra | 10:44 | |
| *** jcapitao is now known as jcapitao_lunch | 10:46 | |
| *** jpena is now known as jpena|lunch | 11:33 | |
| *** priteau has quit IRC | 11:34 | |
| *** priteau has joined #openstack-infra | 11:37 | |
| *** Topner has joined #openstack-infra | 11:52 | |
| *** rlandy has joined #openstack-infra | 11:56 | |
| *** rlandy is now known as rlandy|rover | 11:56 | |
| *** Topner has quit IRC | 11:57 | |
| *** Topner has joined #openstack-infra | 12:00 | |
| *** rfolco has joined #openstack-infra | 12:06 | |
| *** Ajohn has quit IRC | 12:08 | |
| *** jcapitao_lunch is now known as jcapitao | 12:09 | |
| *** thiagop has joined #openstack-infra | 12:17 | |
| *** dhill has joined #openstack-infra | 12:21 | |
| *** haleyb has joined #openstack-infra | 12:25 | |
| *** jpena|lunch is now known as jpena | 12:29 | |
| *** rlandy|rover is now known as rlandy|rover|mtg | 12:36 | |
| *** lbragstad has quit IRC | 12:53 | |
| *** zxiiro has joined #openstack-infra | 13:01 | |
| *** Topner has quit IRC | 13:03 | |
| *** happyhemant has joined #openstack-infra | 13:05 | |
| *** lbragstad has joined #openstack-infra | 13:05 | |
| *** artom has joined #openstack-infra | 13:08 | |
| *** Topner has joined #openstack-infra | 13:13 | |
| *** rlandy|rover|mtg is now known as rlandy|rover | 13:16 | |
| *** redrobot has joined #openstack-infra | 13:31 | |
| *** slaweq has quit IRC | 13:33 | |
| *** slaweq has joined #openstack-infra | 13:39 | |
| openstackgerrit | Pierre Riteau proposed openstack/project-config master: Add Review-Priority flag for cloudkitty https://review.opendev.org/756072 | 13:40 |
|---|---|---|
| *** tkajinam has quit IRC | 13:49 | |
| *** tkajinam has joined #openstack-infra | 13:49 | |
| *** priteau has quit IRC | 13:52 | |
| *** priteau has joined #openstack-infra | 13:59 | |
| *** Goneri has joined #openstack-infra | 13:59 | |
| *** rfolco is now known as rfolco|brb | 14:03 | |
| *** vishalmanchanda has quit IRC | 14:19 | |
| *** derekh has quit IRC | 14:24 | |
| *** armax has joined #openstack-infra | 14:44 | |
| *** slaweq has quit IRC | 14:47 | |
| *** slaweq has joined #openstack-infra | 14:49 | |
| *** tonyb has quit IRC | 14:50 | |
| *** ysandeep is now known as ysandeep|away | 14:52 | |
| *** dklyle has joined #openstack-infra | 14:54 | |
| *** priteau has quit IRC | 15:11 | |
| *** ociuhandu_ has joined #openstack-infra | 15:13 | |
| *** ociuhandu has quit IRC | 15:15 | |
| *** ociuhandu_ has quit IRC | 15:17 | |
| *** vishalmanchanda has joined #openstack-infra | 15:25 | |
| *** zxiiro has quit IRC | 15:30 | |
| *** gyee has joined #openstack-infra | 15:34 | |
| *** happyhemant has quit IRC | 15:35 | |
| *** zxiiro has joined #openstack-infra | 15:45 | |
| *** hashar has quit IRC | 15:47 | |
| *** lucasagomes has quit IRC | 15:56 | |
| *** rfolco|brb is now known as rfolco | 16:03 | |
| *** rpittau is now known as rpittau|afk | 16:06 | |
| *** ykarel is now known as ykarel|away | 16:16 | |
| *** dtantsur is now known as dtantsur|afk | 16:21 | |
| *** tosky has quit IRC | 16:22 | |
| *** hamalq has joined #openstack-infra | 16:23 | |
| *** jtomasek has quit IRC | 16:24 | |
| *** jpena is now known as jpena|off | 16:33 | |
| *** jcapitao is now known as jcapitao_off | 16:33 | |
| *** ykarel|away has quit IRC | 16:34 | |
| *** gfidente is now known as gfidente|afk | 16:44 | |
| *** jcapitao_off has quit IRC | 17:01 | |
| *** hashar has joined #openstack-infra | 17:03 | |
| *** andrewbonney has quit IRC | 17:08 | |
| *** eolivare has quit IRC | 17:10 | |
| *** ociuhandu has joined #openstack-infra | 17:25 | |
| *** ociuhandu has quit IRC | 17:29 | |
| *** cyberpear has joined #openstack-infra | 17:32 | |
| *** vishalmanchanda has quit IRC | 18:15 | |
| *** psachin has quit IRC | 18:16 | |
| *** ralonsoh has quit IRC | 18:19 | |
| *** gfidente|afk has quit IRC | 18:38 | |
| *** _erlon_ has joined #openstack-infra | 18:54 | |
| *** viks____ has quit IRC | 19:02 | |
| *** Topner has quit IRC | 19:14 | |
| *** smarcet has joined #openstack-infra | 19:22 | |
| *** tosky has joined #openstack-infra | 19:29 | |
| *** zxiiro has quit IRC | 19:30 | |
| *** smarcet has quit IRC | 19:40 | |
| *** Topner has joined #openstack-infra | 19:43 | |
| *** Topner has quit IRC | 19:48 | |
| *** larsks has joined #openstack-infra | 19:49 | |
| larsks | Hey folks. Is something up with docs.openstack.org? I can't seem to load any pages from there. | 19:50 |
| fungi | checking | 19:50 |
| fungi | it's taking a while to load for me. i'll see if something's up (or down rather) with static.opendev.org or the afs backend | 19:50 |
| clarkb | dmesg says we lost connectivity with a fileserver | 19:51 |
| clarkb | afs02.dfw.openstack.org. specifically | 19:52 |
| fungi | looks like static.opendev.org is having trouble talking to a fileserver | 19:52 |
| fungi | afs: Lost contact with file server 23.253.73.143 in cell openstack.org (code -1) (all multi-homed ip addresses down for the server) | 19:52 |
| fungi | afs: file server 23.253.73.143 in cell openstack.org is back up (code 0) (multi-homed address; other same-host interfaces may still be down) | 19:52 |
| fungi | afs: Waiting for busy volume 536871075 () in cell openstack.org | 19:52 |
| fungi | repeated for multiple volumes | 19:52 |
| fungi | yeah, 23.253.73.143 is afs02.dfw | 19:53 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=6397&rra_id=all | 19:53 |
| fungi | which i can't ssh into | 19:53 |
| fungi | will check the oob console | 19:53 |
| clarkb | the extra cpu load seems to come along with some extra bw usage but that is before the fedora mirror would've been released I think | 19:53 |
| clarkb | unless fedora is primary on afs02? | 19:53 |
| clarkb | I confirm that I can't ssh, I get a host key and then not much further than that | 19:54 |
| fungi | hung kernel tasks reported on the console, and no login prompt after a carriage return | 19:55 |
| fungi | i think the vm hung | 19:55 |
| fungi | i'll give it a reboot via the nova api | 19:55 |
| clarkb | 2020-10-05 19:31:34 | Running vos release. <- thats the fedora mirror log which happens well after the problme starts and afs01 is primary | 19:55 |
| fungi | yeah, if we want to backtrack from lastlog (assuming it's not been thoroughly rotated away), the hung kernel tasks started 3961321 seconds after boot | 19:56 |
| fungi | in two bursts, a second appeared at 3961441 second | 19:57 |
| fungi | openstack server reboot --hard issued just now | 19:58 |
| *** lmiccini has quit IRC | 19:59 | |
| fungi | if those hung kernel tasks were logged today, then i expect to see them in the lastlog for 2020-08-20 | 19:59 |
| fungi | i worry that may be prior to wtmp.1 | 19:59 |
| clarkb | server seems up now | 20:00 |
| clarkb | and docs load | 20:01 |
| fungi | larsks: docs.o.o should be back up for you now? | 20:01 |
| fungi | yeah, just checked myself | 20:01 |
| fungi | once static.o.o realized the fileserver was down hard it switched to the redundant one, i thinl | 20:01 |
| clarkb | the fedora release detected failure and said it could not be completed | 20:02 |
| fungi | afs was not designed for our modern clouds where virtual machines just become hung and don't fall entirely offline | 20:02 |
| clarkb | I expect that means all the locks were returned and it will try again in a bit | 20:02 |
| fungi | yeah, hopefully we don't need to clean up any replicas which were mid-release, | 20:02 |
| fungi | but we should probably look to be sure | 20:03 |
| fungi | i can ssh into afs02.dwf now | 20:03 |
| fungi | wtmp.1 begins Thu Sep 3 21:10:10 2020 | 20:03 |
| fungi | fooey | 20:03 |
| fungi | anyway, past experience has indicated that if we see hung kernel tasks on the console, those tend to be irrecoverable and coincide with when the server ceased to be responsive | 20:04 |
| fungi | cacti places the time of death between 18:50 and 18:55 utc | 20:06 |
| fungi | #status log hard rebooted afs03.dfw.o.o to address a server hung condition which seems to have started at roughly 18:50 utc | 20:06 |
| openstackstatus | fungi: finished logging | 20:06 |
| * fungi sighs | 20:07 | |
| fungi | #status log CORRECTION: hard rebooted afs02.dfw.o.o to address a server hung condition which seems to have started at roughly 18:50 utc | 20:07 |
| openstackstatus | fungi: finished logging | 20:07 |
| *** Topner has joined #openstack-infra | 20:09 | |
| larsks | fungi: works for me now, yes. Thanks! | 20:20 |
| fungi | larsks: awesome. sorry about... clouds | 20:20 |
| larsks | fungi: also...wow, afs, haven't come across that in years and years... :) | 20:23 |
| fungi | you just don't run in the right circles, apparently ;) | 20:25 |
| *** mnasiadka has quit IRC | 20:25 | |
| *** jroll has quit IRC | 20:25 | |
| *** jroll has joined #openstack-infra | 20:27 | |
| *** slaweq has quit IRC | 20:50 | |
| *** matt_kosut has quit IRC | 21:04 | |
| *** rfolco has quit IRC | 21:04 | |
| *** hashar has quit IRC | 21:14 | |
| *** tonyb has joined #openstack-infra | 21:22 | |
| *** d34dh0r53 has quit IRC | 21:52 | |
| *** thiagop has quit IRC | 22:17 | |
| *** rcernin has joined #openstack-infra | 22:19 | |
| *** thiagop has joined #openstack-infra | 22:21 | |
| *** samueldmq has quit IRC | 22:22 | |
| *** PrinzElvis has quit IRC | 22:24 | |
| *** thedac has quit IRC | 22:24 | |
| *** thedac has joined #openstack-infra | 22:25 | |
| *** samueldmq has joined #openstack-infra | 22:26 | |
| *** PrinzElvis has joined #openstack-infra | 22:27 | |
| *** zzzeek has quit IRC | 22:46 | |
| *** zzzeek has joined #openstack-infra | 22:50 | |
| *** rlandy|rover has quit IRC | 23:01 | |
| *** tosky has quit IRC | 23:04 | |
| *** rfolco has joined #openstack-infra | 23:09 | |
| *** Topner has quit IRC | 23:33 | |
| *** thiagop has quit IRC | 23:36 | |
| *** rfolco has quit IRC | 23:42 | |
| *** dchen has joined #openstack-infra | 23:42 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!