| *** goldyfruit has quit IRC | 00:09 | |
| *** slaweq has joined #openstack-infra | 00:16 | |
| *** goldyfruit has joined #openstack-infra | 00:17 | |
| *** yamamoto has joined #openstack-infra | 00:18 | |
| *** jamesmcarthur has quit IRC | 00:18 | |
| *** lseki has quit IRC | 00:21 | |
| *** slaweq has quit IRC | 00:24 | |
| *** yamamoto has quit IRC | 00:25 | |
| *** markvoelker has joined #openstack-infra | 00:26 | |
| *** yamamoto has joined #openstack-infra | 00:29 | |
| *** harlowja has quit IRC | 00:29 | |
| *** yamamoto has quit IRC | 00:30 | |
| *** markvoelker has quit IRC | 00:31 | |
| *** georgk9 has quit IRC | 00:31 | |
| *** georgk9 has joined #openstack-infra | 00:31 | |
| *** diablo_rojo has quit IRC | 00:36 | |
| *** mattw4 has quit IRC | 00:40 | |
| *** yamamoto has joined #openstack-infra | 00:49 | |
| *** weifan has joined #openstack-infra | 00:50 | |
| *** weifan has quit IRC | 00:55 | |
| *** hwoarang has quit IRC | 00:56 | |
| *** weifan has joined #openstack-infra | 00:57 | |
| *** weifan has quit IRC | 00:57 | |
| *** hwoarang has joined #openstack-infra | 01:02 | |
| *** kopecmartin|off has quit IRC | 01:02 | |
| *** kopecmartin has joined #openstack-infra | 01:04 | |
| *** yamamoto has quit IRC | 01:05 | |
| *** jamesmcarthur has joined #openstack-infra | 01:08 | |
| *** slaweq has joined #openstack-infra | 01:13 | |
| *** jamesmcarthur has quit IRC | 01:14 | |
| *** slaweq has quit IRC | 01:24 | |
| *** markvoelker has joined #openstack-infra | 01:27 | |
| *** irclogbot_3 has quit IRC | 01:28 | |
| *** irclogbot_3 has joined #openstack-infra | 01:31 | |
| *** yamamoto has joined #openstack-infra | 01:34 | |
| *** ekultails has quit IRC | 01:37 | |
| *** markvoelker has quit IRC | 01:56 | |
| *** slaweq has joined #openstack-infra | 02:00 | |
| *** igordc has joined #openstack-infra | 02:07 | |
| *** mriedem has quit IRC | 02:07 | |
| *** slaweq has quit IRC | 02:10 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:10 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** cyberpear has quit IRC | 02:16 | |
| *** bhavikdbavishi has quit IRC | 02:19 | |
| *** yamamoto has quit IRC | 02:22 | |
| *** slaweq has quit IRC | 02:24 | |
| *** markvoelker has joined #openstack-infra | 02:53 | |
| *** goldyfruit has quit IRC | 02:54 | |
| *** mattw4 has joined #openstack-infra | 02:56 | |
| *** yamamoto has joined #openstack-infra | 02:57 | |
| *** markvoelker has quit IRC | 03:01 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:02 | |
| *** bobh has joined #openstack-infra | 03:04 | |
| *** bobh has quit IRC | 03:06 | |
| *** yamamoto has quit IRC | 03:08 | |
| *** slaweq has joined #openstack-infra | 03:12 | |
| *** goldyfruit has joined #openstack-infra | 03:22 | |
| *** slaweq has quit IRC | 03:25 | |
| *** persia has quit IRC | 03:26 | |
| *** persia has joined #openstack-infra | 03:26 | |
| *** goldyfruit has quit IRC | 03:27 | |
| *** spsurya has joined #openstack-infra | 03:27 | |
| *** weifan has joined #openstack-infra | 03:30 | |
| *** weifan has quit IRC | 03:34 | |
| *** michael-beaver has quit IRC | 03:37 | |
| *** bhavikdbavishi has quit IRC | 03:43 | |
| *** needssleep is now known as TheJulia | 03:48 | |
| *** yamamoto has joined #openstack-infra | 03:48 | |
| *** yamamoto has quit IRC | 03:50 | |
| *** bobh has joined #openstack-infra | 04:02 | |
| *** slaweq has joined #openstack-infra | 04:13 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:15 | |
| *** yamamoto has joined #openstack-infra | 04:20 | |
| *** weifan has joined #openstack-infra | 04:24 | |
| *** slaweq has quit IRC | 04:25 | |
| *** bhavikdbavishi has quit IRC | 04:26 | |
| *** kjackal has joined #openstack-infra | 04:27 | |
| *** weifan has quit IRC | 04:28 | |
| *** yamamoto has quit IRC | 04:29 | |
| *** igordc has quit IRC | 04:55 | |
| *** igordc has joined #openstack-infra | 04:55 | |
| *** markvoelker has joined #openstack-infra | 04:58 | |
| *** markvoelker has quit IRC | 05:02 | |
| *** yamamoto has joined #openstack-infra | 05:07 | |
| *** bobh has quit IRC | 05:19 | |
| *** yamamoto has quit IRC | 05:57 | |
| *** yamamoto has joined #openstack-infra | 05:57 | |
| *** markvoelker has joined #openstack-infra | 05:58 | |
| *** markvoelker has quit IRC | 06:03 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/665869 | 06:09 |
|---|---|---|
| *** bhavikdbavishi has joined #openstack-infra | 06:12 | |
| *** slaweq has joined #openstack-infra | 06:16 | |
| *** bhavikdbavishi has quit IRC | 06:22 | |
| *** lpetrut has joined #openstack-infra | 06:24 | |
| *** slaweq has quit IRC | 06:24 | |
| gmann | fungi: frickler can you check this - https://review.opendev.org/#/c/647912/ | 06:39 |
| *** igordc has quit IRC | 06:59 | |
| *** igordc has joined #openstack-infra | 06:59 | |
| *** markvoelker has joined #openstack-infra | 06:59 | |
| *** igordc has quit IRC | 07:03 | |
| *** jtomasek has joined #openstack-infra | 07:07 | |
| *** mattw4 has quit IRC | 07:10 | |
| *** pcaruana has joined #openstack-infra | 07:15 | |
| *** altlogbot_2 has quit IRC | 07:18 | |
| *** altlogbot_1 has joined #openstack-infra | 07:19 | |
| *** markvoelker has quit IRC | 07:26 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:41 | |
| *** elod has quit IRC | 07:45 | |
| *** elod has joined #openstack-infra | 07:47 | |
| *** bhavikdbavishi has quit IRC | 07:52 | |
| *** slaweq has joined #openstack-infra | 07:52 | |
| *** elod has quit IRC | 07:52 | |
| *** elod has joined #openstack-infra | 07:53 | |
| *** lpetrut has quit IRC | 08:20 | |
| *** yamamoto has quit IRC | 08:37 | |
| *** yamamoto has joined #openstack-infra | 08:37 | |
| *** imacdonn has quit IRC | 08:41 | |
| *** imacdonn has joined #openstack-infra | 08:42 | |
| *** kjackal has quit IRC | 08:42 | |
| *** kjackal_v2 has joined #openstack-infra | 08:42 | |
| openstackgerrit | Merged openstack/project-config master: Do not require a CLA for the sandbox https://review.opendev.org/666918 | 08:44 |
| *** rascasoft has quit IRC | 08:48 | |
| *** rascasoft has joined #openstack-infra | 08:48 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:56 | |
| *** ociuhandu has joined #openstack-infra | 09:19 | |
| *** whoami-rajat has joined #openstack-infra | 09:21 | |
| *** Lucas_Gray has joined #openstack-infra | 09:34 | |
| *** tosky has joined #openstack-infra | 09:39 | |
| *** ociuhandu has quit IRC | 09:53 | |
| *** markvoelker has joined #openstack-infra | 10:23 | |
| *** markvoelker has quit IRC | 10:28 | |
| *** jtomasek has quit IRC | 10:32 | |
| *** lathiat has quit IRC | 10:37 | |
| *** jtomasek has joined #openstack-infra | 10:38 | |
| *** Lucas_Gray has quit IRC | 10:38 | |
| *** Lucas_Gray has joined #openstack-infra | 10:41 | |
| *** yamamoto has quit IRC | 11:00 | |
| *** bhavikdbavishi has quit IRC | 11:08 | |
| *** lathiat has joined #openstack-infra | 11:21 | |
| *** markvoelker has joined #openstack-infra | 11:24 | |
| *** markvoelker has quit IRC | 11:29 | |
| *** ykarel|away has joined #openstack-infra | 11:29 | |
| *** yamamoto has joined #openstack-infra | 11:29 | |
| *** whoami-rajat has quit IRC | 11:31 | |
| *** Lucas_Gray has quit IRC | 11:44 | |
| *** rh-jelabarre has quit IRC | 11:48 | |
| *** jtomasek has quit IRC | 11:52 | |
| *** jamesdenton has joined #openstack-infra | 11:53 | |
| *** rh-jelabarre has joined #openstack-infra | 11:53 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:18 | |
| *** bgmccollum has quit IRC | 12:22 | |
| *** markvoelker has joined #openstack-infra | 12:25 | |
| *** bgmccollum has joined #openstack-infra | 12:28 | |
| *** markvoelker has quit IRC | 12:29 | |
| *** ykarel|away has quit IRC | 12:30 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 13:24 | |
| *** markvoelker has joined #openstack-infra | 13:26 | |
| *** bhavikdbavishi has quit IRC | 13:26 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 13:26 | |
| *** yamamoto has quit IRC | 13:26 | |
| *** yamamoto has joined #openstack-infra | 13:27 | |
| *** kjackal_v2 has quit IRC | 13:27 | |
| *** elod has quit IRC | 13:39 | |
| *** bhavikdbavishi has quit IRC | 13:41 | |
| *** markvoelker has quit IRC | 13:54 | |
| *** kjackal has joined #openstack-infra | 14:37 | |
| *** markvoelker has joined #openstack-infra | 14:51 | |
| *** markvoelker has quit IRC | 14:56 | |
| *** hwoarang has quit IRC | 15:12 | |
| *** hwoarang has joined #openstack-infra | 15:13 | |
| *** ykarel|away has joined #openstack-infra | 15:22 | |
| *** ralonsoh has joined #openstack-infra | 15:25 | |
| *** whoami-rajat has joined #openstack-infra | 15:30 | |
| *** ralonsoh has quit IRC | 15:32 | |
| *** ralonsoh has joined #openstack-infra | 15:32 | |
| *** tosky has quit IRC | 15:36 | |
| *** yamamoto has quit IRC | 15:37 | |
| *** yamamoto has joined #openstack-infra | 15:38 | |
| *** yamamoto has quit IRC | 15:40 | |
| *** ociuhandu has joined #openstack-infra | 15:42 | |
| *** yamamoto_ has joined #openstack-infra | 15:45 | |
| *** yamamoto_ has quit IRC | 15:45 | |
| *** yamamoto_ has joined #openstack-infra | 15:46 | |
| *** yamamoto_ has quit IRC | 15:48 | |
| *** yamamoto has joined #openstack-infra | 15:49 | |
| *** ykarel|away has quit IRC | 15:49 | |
| *** ociuhandu has quit IRC | 15:50 | |
| *** markvoelker has joined #openstack-infra | 15:52 | |
| *** yamamoto has quit IRC | 15:53 | |
| *** yamamoto has joined #openstack-infra | 15:54 | |
| *** yamamoto_ has joined #openstack-infra | 15:56 | |
| *** markvoelker has quit IRC | 15:56 | |
| *** icarusfactor has quit IRC | 15:58 | |
| *** yamamoto_ has quit IRC | 15:58 | |
| *** yamamoto has quit IRC | 16:00 | |
| *** yamamoto has joined #openstack-infra | 16:35 | |
| *** bhavikdbavishi has joined #openstack-infra | 16:41 | |
| *** yamamoto has quit IRC | 16:47 | |
| *** rh-jelabarre has quit IRC | 16:48 | |
| *** markvoelker has joined #openstack-infra | 16:53 | |
| *** rh-jelabarre has joined #openstack-infra | 16:53 | |
| *** markvoelker has quit IRC | 16:57 | |
| *** mattw4 has joined #openstack-infra | 17:06 | |
| *** bhavikdbavishi has quit IRC | 17:43 | |
| *** markvoelker has joined #openstack-infra | 17:53 | |
| *** bhavikdbavishi has joined #openstack-infra | 17:54 | |
| *** e0ne has joined #openstack-infra | 17:57 | |
| *** markvoelker has quit IRC | 17:58 | |
| *** e0ne has quit IRC | 18:15 | |
| ralonsoh | hi guys, I know it's saturday but let's see if I'm lucky | 18:44 |
| ralonsoh | 2019-06-22 18:22:45.732 | Reading package lists... | 18:44 |
| ralonsoh | 2019-06-22 18:22:45.767 | E: Failed to fetch http://mirror.regionone.limestone.openstack.org/ubuntu/dists/bionic/universe/binary-amd64/Packages 403 Forbidden [IP: 2607:ff68:100:54:f816:3eff:febd:b93 80] | 18:44 |
| ralonsoh | 2019-06-22 18:22:45.768 | E: Failed to fetch http://mirror.regionone.limestone.openstack.org/ubuntu/dists/bionic-updates/universe/binary-amd64/Packages 403 Forbidden [IP: 2607:ff68:100:54:f816:3eff:febd:b93 80] | 18:44 |
| ralonsoh | 2019-06-22 18:22:45.768 | E: Failed to fetch http://mirror.regionone.limestone.openstack.org/ubuntu/dists/bionic-backports/main/binary-amd64/Packages 403 Forbidden [IP: 2607:ff68:100:54:f816:3eff:febd:b93 80] | 18:44 |
| ralonsoh | 2019-06-22 18:22:45.768 | E: Failed to fetch http://mirror.regionone.limestone.openstack.org/ubuntu/dists/bionic-security/main/binary-amd64/Packages 403 Forbidden [IP: 2607:ff68:100:54:f816:3eff:febd:b93 80] | 18:44 |
| ralonsoh | do you know why are we receiving this error message? | 18:44 |
| *** spsurya has quit IRC | 18:46 | |
| pabelanger | ralonsoh: looks like a dns issues | 18:50 |
| pabelanger | or could be issue with regional mirror in limestone | 18:51 |
| pabelanger | I'm not in a position to look sadly | 18:51 |
| pabelanger | logan-: ^ | 18:51 |
| ralonsoh | pabelanger, I'm trying the ipv4 IP without success | 18:51 |
| pabelanger | http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=698&nodeid=node1_698&host_group_data= | 18:52 |
| pabelanger | that should be the mirror, I don't see anything odd with it | 18:53 |
| pabelanger | and it is responding to ping / ssh | 18:53 |
| pabelanger | ralonsoh: at this point, you'll have to wait until another infra-root comes online, but likey won't be until monday since it is the weekend | 18:53 |
| *** markvoelker has joined #openstack-infra | 18:54 | |
| ralonsoh | pabelanger, no problem, just to know and to try to bring up the neutron CI | 18:54 |
| fungi | taking a closer look | 18:56 |
| fungi | i can cat /afs/openstack.org/mirror/ubuntu/dists/bionic-security/main/binary-amd64/Packages locally on the mirror as an unprivileged user | 18:58 |
| fungi | i can also hit http://mirror.regionone.limestone.openstack.org/ubuntu/dists/bionic-security/main/binary-amd64/Packages with my browser just fine | 18:59 |
| fungi | over ipv6 too | 18:59 |
| fungi | ralonsoh: is it happening to multiple jobs/changes? | 19:00 |
| ralonsoh | fungi, let me check | 19:01 |
| fungi | a link to the job log at least would be helpful | 19:01 |
| ralonsoh | fungi, http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%20403%20%20Forbidden%20%5BIP%3A%202607%3Aff68%3A100%3A54%3Af816%3A3eff%3Afebd%3Ab93%2080%5D%5C%22 | 19:03 |
| ralonsoh | fungi, there you'll see several CI jobs with this error | 19:04 |
| fungi | looks like between 20:01:47 utc yesterday and 18:49:42 utc today apache logged a slew of "AH00037: Symbolic link not allowed or link target not accessible: /var/www/mirror/ubuntu" | 19:06 |
| fungi | the last one (so far) was ~15 minutes ago | 19:06 |
| fungi | the earliest hit i see ni logstash for message:"E: Failed to fetch http://mirror.regionone.limestone.openstack.org/ubuntu/dists/bionic-security/main/binary-amd64/Packages 403 Forbidden" was 20:01:47 utc | 19:10 |
| fungi | so that seems to correspond | 19:10 |
| fungi | and most recent in logstash is 18:49:42 utc, also a perfect match | 19:11 |
| fungi | oh, fun, my ssh session to the mirror server just hung | 19:11 |
| fungi | but yeah, since it seems connectivity to it is going in and out, i bet it's also having trouble reaching the afs servers | 19:12 |
| fungi | we may want to turn this region back down until logan- has a chance to take a look at possible network issues there | 19:14 |
| fungi | even diagnosing it is a challenge because my ssh session keeps halting for minutes at a time | 19:14 |
| ralonsoh | fungi, thanks for your quick review! | 19:16 |
| ralonsoh | and diagnostic | 19:16 |
| *** e0ne has joined #openstack-infra | 19:16 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Revert "Revert "Disable provider limestone"" https://review.opendev.org/666963 | 19:17 |
| fungi | infra-root: i'm self-approving that and may wind up bypassing ci if it can't pass jobs | 19:18 |
| ralonsoh | fungi, I've seen the patch. Thanks! | 19:18 |
| fungi | looks like the first log entry in dmesg after the most recent boot for the mirror is: | 19:19 |
| fungi | [Fri Jun 21 19:55:03 2019] afs: Lost contact with file server 104.130.138.161 in cell openstack.org (code -1) (all multi-homed ip addresses down for the server) | 19:19 |
| fungi | so that lines up as well | 19:19 |
| fungi | poking around in cacti, looks like network connectivity to it was stable up through the 19:50 snmp poll, and then at the 19:55 poll it didn't get a response | 19:22 |
| fungi | so our window looks like somewhere between 19:50-19:55 utc yesterday when the network began to be unreliable | 19:23 |
| fungi | logan-: ^ in case that helps narrow it down to any particular event on your side | 19:23 |
| *** markvoelker has quit IRC | 19:24 | |
| *** whoami-rajat has quit IRC | 19:29 | |
| fungi | ugh, already hit a retry_limit failure on one of the only two check jobs that change needs to pass... and yep, it got scheduled there (all three times apparently) | 19:31 |
| *** dave-mccowan has joined #openstack-infra | 19:31 | |
| ralonsoh | hahahaha | 19:32 |
| openstackgerrit | Merged openstack/project-config master: Revert "Revert "Disable provider limestone"" https://review.opendev.org/666963 | 19:35 |
| pabelanger | fungi: can confirm there seems to be a network issue with limestone, seeing same issue from zuul.a.c now | 19:35 |
| fungi | and it's not just ipv6, given it's also impacting afs (which is a v4-only protocol) | 19:36 |
| fungi | and it doesn't seem to be affecting any other providers, so pretty sure it's either in or near upstream from limestone | 19:37 |
| fungi | once the bridge rolls that change out we should quickly stop scheduling new jobs there | 19:39 |
| *** dave-mccowan has quit IRC | 19:42 | |
| *** Lucas_Gray has joined #openstack-infra | 19:43 | |
| *** jtomasek has joined #openstack-infra | 19:47 | |
| *** yamamoto has joined #openstack-infra | 19:48 | |
| *** e0ne has quit IRC | 19:52 | |
| *** yamamoto has quit IRC | 19:52 | |
| *** bhavikdbavishi has quit IRC | 19:56 | |
| *** yamamoto has joined #openstack-infra | 20:01 | |
| *** bhavikdbavishi has joined #openstack-infra | 20:05 | |
| *** ralonsoh has quit IRC | 20:06 | |
| *** igordc has joined #openstack-infra | 20:08 | |
| *** bhavikdbavishi has quit IRC | 20:09 | |
| *** jtomasek has quit IRC | 20:14 | |
| *** markvoelker has joined #openstack-infra | 20:21 | |
| *** Lucas_Gray has quit IRC | 20:27 | |
| *** yamamoto has quit IRC | 20:27 | |
| *** markvoelker has quit IRC | 20:30 | |
| *** kjackal has quit IRC | 20:37 | |
| *** lpetrut has joined #openstack-infra | 20:52 | |
| *** lpetrut has quit IRC | 20:53 | |
| *** lpetrut has joined #openstack-infra | 20:54 | |
| *** markvoelker has joined #openstack-infra | 20:55 | |
| *** pcaruana has quit IRC | 21:19 | |
| *** zbr|ruck has quit IRC | 21:21 | |
| *** slaweq has quit IRC | 21:23 | |
| fungi | last hit in logstash now is 19:27:59 | 21:25 |
| fungi | so two hours ago | 21:26 |
| fungi | connectivity seems to have smoothed out since 20:35, so nearly an hour ago | 21:28 |
| fungi | maybe whatever was impacting that environment has subsided, but will give it a while longer | 21:29 |
| *** zbr has joined #openstack-infra | 21:30 | |
| *** lpetrut has quit IRC | 21:49 | |
| *** lpetrut has joined #openstack-infra | 21:50 | |
| *** smarcet has joined #openstack-infra | 21:56 | |
| fungi | but possibility is it was running jobs there which impacted that environment | 21:57 |
| *** lpetrut has quit IRC | 22:00 | |
| *** smarcet has quit IRC | 22:01 | |
| *** slaweq has joined #openstack-infra | 22:12 | |
| *** zbr has quit IRC | 22:19 | |
| *** slaweq has quit IRC | 22:24 | |
| *** sthussey has quit IRC | 22:32 | |
| *** lpetrut has joined #openstack-infra | 22:35 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Stop bind mounting zuul dir into bwrap https://review.opendev.org/666972 | 22:37 |
| *** lpetrut has quit IRC | 22:50 | |
| *** rh-jelabarre has quit IRC | 22:53 | |
| *** rh-jelabarre has joined #openstack-infra | 22:58 | |
| *** markvoelker has quit IRC | 23:04 | |
| *** markvoelker has joined #openstack-infra | 23:05 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!