| *** zzzeek has quit IRC | 00:03 | |
| *** zzzeek has joined #openstack-infra | 00:05 | |
| *** lifeless has quit IRC | 00:26 | |
| *** ociuhandu has quit IRC | 00:27 | |
| *** lifeless has joined #openstack-infra | 00:28 | |
| *** zzzeek has quit IRC | 00:28 | |
| *** rcernin has quit IRC | 00:29 | |
| *** zzzeek has joined #openstack-infra | 00:30 | |
| *** zxiiro has quit IRC | 00:30 | |
| *** rcernin has joined #openstack-infra | 00:32 | |
| *** sorrison has joined #openstack-infra | 00:34 | |
| *** lifeless has quit IRC | 00:37 | |
| *** ociuhandu has joined #openstack-infra | 00:41 | |
| *** ociuhandu has quit IRC | 00:47 | |
| *** rcernin has quit IRC | 00:47 | |
| *** Goneri has quit IRC | 00:48 | |
| *** lifeless has joined #openstack-infra | 00:48 | |
| *** ociuhandu has joined #openstack-infra | 00:59 | |
| *** ociuhandu has quit IRC | 01:05 | |
| *** ociuhandu has joined #openstack-infra | 01:18 | |
| *** prometheanfire has joined #openstack-infra | 01:45 | |
| *** rcernin has joined #openstack-infra | 02:04 | |
| *** iurygregory has quit IRC | 02:07 | |
| *** takamatsu has quit IRC | 02:10 | |
| *** rcernin has quit IRC | 02:12 | |
| *** takamatsu has joined #openstack-infra | 02:43 | |
| *** dmsimard has quit IRC | 02:49 | |
| *** mmethot_ has joined #openstack-infra | 02:50 | |
| *** dmsimard has joined #openstack-infra | 02:50 | |
| *** mmethot has quit IRC | 02:53 | |
| *** rcernin has joined #openstack-infra | 03:17 | |
| *** zzzeek has quit IRC | 03:45 | |
| *** zzzeek has joined #openstack-infra | 03:47 | |
| *** rcernin has quit IRC | 03:57 | |
| *** reed has quit IRC | 04:00 | |
| *** reed has joined #openstack-infra | 04:01 | |
| *** zzzeek has quit IRC | 04:20 | |
| *** zzzeek has joined #openstack-infra | 04:20 | |
| *** reed has quit IRC | 04:46 | |
| *** reed has joined #openstack-infra | 04:46 | |
| *** rcernin has joined #openstack-infra | 05:15 | |
| *** rcernin has quit IRC | 05:21 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-infra | 05:33 | |
| *** dviroel has quit IRC | 05:40 | |
| *** rpioso has quit IRC | 05:40 | |
| *** rpioso has joined #openstack-infra | 05:40 | |
| *** dviroel has joined #openstack-infra | 05:40 | |
| *** rcernin has joined #openstack-infra | 05:47 | |
| *** matt_kosut has joined #openstack-infra | 05:50 | |
| *** rcernin has quit IRC | 05:56 | |
| *** reed has quit IRC | 05:56 | |
| *** reed has joined #openstack-infra | 05:57 | |
| *** ianychoi_ has joined #openstack-infra | 05:57 | |
| *** ianychoi has quit IRC | 06:00 | |
| *** rcernin has joined #openstack-infra | 06:26 | |
| *** ricolin has quit IRC | 06:31 | |
| *** rcernin has quit IRC | 06:31 | |
| *** rcernin has joined #openstack-infra | 06:36 | |
| *** lmiccini has joined #openstack-infra | 06:44 | |
| *** reed has quit IRC | 07:01 | |
| *** reed has joined #openstack-infra | 07:02 | |
| *** rcernin has quit IRC | 07:05 | |
| *** sboyron has joined #openstack-infra | 07:16 | |
| *** ricolin_ has joined #openstack-infra | 07:18 | |
| *** ricolin_ has quit IRC | 07:20 | |
| *** ricolin has joined #openstack-infra | 07:20 | |
| *** reed has quit IRC | 07:22 | |
| *** reed has joined #openstack-infra | 07:22 | |
| *** slaweq has joined #openstack-infra | 07:25 | |
| *** rpittau|afk is now known as rpittau | 07:26 | |
| *** iurygregory has joined #openstack-infra | 07:29 | |
| *** rcernin has joined #openstack-infra | 07:31 | |
| *** ralonsoh has joined #openstack-infra | 07:33 | |
| *** rcernin has quit IRC | 07:36 | |
| *** sboyron has quit IRC | 07:40 | |
| *** sshnaidm|off is now known as sshnaidm | 07:46 | |
| *** rcernin has joined #openstack-infra | 07:50 | |
| *** rpittau is now known as rpittau|bbl | 07:52 | |
| *** eolivare has joined #openstack-infra | 07:55 | |
| *** jcapitao has joined #openstack-infra | 07:55 | |
| *** rcernin has quit IRC | 07:55 | |
| *** rcernin has joined #openstack-infra | 07:56 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:03 | |
| *** hashar has joined #openstack-infra | 08:10 | |
| *** rcernin has quit IRC | 08:10 | |
| *** matt_kosut has quit IRC | 08:11 | |
| *** sboyron has joined #openstack-infra | 08:11 | |
| *** matt_kosut has joined #openstack-infra | 08:12 | |
| *** rcernin has joined #openstack-infra | 08:13 | |
| *** gfidente has joined #openstack-infra | 08:14 | |
| *** reed has quit IRC | 08:22 | |
| *** reed has joined #openstack-infra | 08:23 | |
| *** dchen has quit IRC | 08:24 | |
| *** xek has joined #openstack-infra | 08:25 | |
| *** dchen has joined #openstack-infra | 08:25 | |
| *** dchen has quit IRC | 08:26 | |
| *** rcernin has quit IRC | 08:37 | |
| *** reed has quit IRC | 08:43 | |
| *** reed has joined #openstack-infra | 08:43 | |
| *** ociuhandu has quit IRC | 08:46 | |
| *** andrewbonney has joined #openstack-infra | 08:49 | |
| *** tosky has joined #openstack-infra | 08:49 | |
| *** jpena|off is now known as jpena | 08:55 | |
| *** lucasagomes has joined #openstack-infra | 08:55 | |
| *** zzzeek has quit IRC | 08:57 | |
| *** zzzeek has joined #openstack-infra | 08:58 | |
| *** rcernin has joined #openstack-infra | 09:04 | |
| *** xek has quit IRC | 09:05 | |
| *** zzzeek has quit IRC | 09:06 | |
| *** zzzeek has joined #openstack-infra | 09:07 | |
| *** xek has joined #openstack-infra | 09:11 | |
| *** rcernin has quit IRC | 09:12 | |
| *** zzzeek has quit IRC | 09:24 | |
| *** zzzeek has joined #openstack-infra | 09:24 | |
| *** reed has quit IRC | 09:28 | |
| *** reed has joined #openstack-infra | 09:28 | |
| jpena | hi! I see the centos 7 mirror content on the AFS mirrors is outdated, and does not contain the 7.9 rpms | 09:48 |
|---|---|---|
| jpena | looking at https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update#L51, it looks like we're syncing from mirros.lstn.net | 09:48 |
| jpena | and it is out of sync, according to https://mirror-status.centos.org/ | 09:48 |
| *** reed has quit IRC | 09:48 | |
| jpena | are we using that server for a special reason, or could we switch to a different one? | 09:49 |
| *** rcernin has joined #openstack-infra | 09:49 | |
| *** reed has joined #openstack-infra | 09:49 | |
| *** xek has quit IRC | 09:52 | |
| *** xek has joined #openstack-infra | 09:55 | |
| *** rcernin has quit IRC | 10:03 | |
| *** reed has quit IRC | 10:04 | |
| *** reed has joined #openstack-infra | 10:05 | |
| *** sean-k-mooney2 has joined #openstack-infra | 10:05 | |
| *** sean-k-mooney1 has quit IRC | 10:08 | |
| *** reed has quit IRC | 10:10 | |
| *** reed has joined #openstack-infra | 10:10 | |
| *** xek has quit IRC | 10:11 | |
| frickler | jpena: iiuc we keep chasing for the least broken mirror server. if you know a better one, feel free to propose a patch | 10:13 |
| *** ociuhandu has joined #openstack-infra | 10:15 | |
| *** reed has quit IRC | 10:20 | |
| *** reed has joined #openstack-infra | 10:20 | |
| *** zzzeek has quit IRC | 10:23 | |
| *** zzzeek has joined #openstack-infra | 10:24 | |
| *** reed has quit IRC | 10:25 | |
| *** reed has joined #openstack-infra | 10:26 | |
| *** sean-k-mooney has joined #openstack-infra | 10:27 | |
| *** sean-k-mooney2 has quit IRC | 10:30 | |
| *** dtantsur|afk is now known as dtantsur | 10:30 | |
| *** sean-k-mooney1 has joined #openstack-infra | 10:32 | |
| *** sean-k-mooney has quit IRC | 10:35 | |
| *** zzzeek has quit IRC | 10:41 | |
| *** zzzeek has joined #openstack-infra | 10:42 | |
| *** reed has quit IRC | 10:46 | |
| *** reed has joined #openstack-infra | 10:46 | |
| openstackgerrit | Liam Young proposed openstack/project-config master: Add Magpie charm to OpenStack charms https://review.opendev.org/762819 | 10:59 |
| *** reed has quit IRC | 11:06 | |
| *** reed has joined #openstack-infra | 11:06 | |
| *** zzzeek has quit IRC | 11:12 | |
| *** zzzeek has joined #openstack-infra | 11:13 | |
| *** reed has quit IRC | 11:21 | |
| *** reed has joined #openstack-infra | 11:22 | |
| *** reed has quit IRC | 11:27 | |
| *** zzzeek has quit IRC | 11:27 | |
| *** sean-k-mooney1 has quit IRC | 11:27 | |
| *** reed has joined #openstack-infra | 11:27 | |
| *** sean-k-mooney1 has joined #openstack-infra | 11:27 | |
| *** sean-k-mooney1 is now known as sean-k-mooney | 11:28 | |
| *** rpittau|bbl is now known as rpittau | 11:30 | |
| *** zzzeek has joined #openstack-infra | 11:30 | |
| *** rcernin has joined #openstack-infra | 11:30 | |
| *** reed has quit IRC | 11:32 | |
| *** reed has joined #openstack-infra | 11:32 | |
| *** reed has quit IRC | 11:37 | |
| *** reed has joined #openstack-infra | 11:38 | |
| *** ociuhandu has quit IRC | 11:42 | |
| *** reed has quit IRC | 11:48 | |
| *** reed has joined #openstack-infra | 11:48 | |
| *** jcapitao is now known as jcapitao_lunch | 11:52 | |
| *** rcernin has quit IRC | 11:56 | |
| *** thiagop has joined #openstack-infra | 11:58 | |
| *** lpetrut has joined #openstack-infra | 12:00 | |
| *** rcernin has joined #openstack-infra | 12:02 | |
| *** ociuhandu has joined #openstack-infra | 12:13 | |
| *** zzzeek has quit IRC | 12:14 | |
| *** zzzeek has joined #openstack-infra | 12:16 | |
| *** rcernin has quit IRC | 12:19 | |
| *** rfolco has joined #openstack-infra | 12:19 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** rlandy has joined #openstack-infra | 12:33 | |
| *** rlandy is now known as rlandy|rover | 12:33 | |
| *** xek has joined #openstack-infra | 12:35 | |
| *** xek_ has joined #openstack-infra | 12:38 | |
| *** xek has quit IRC | 12:41 | |
| *** raissa has joined #openstack-infra | 12:48 | |
| *** jcapitao_lunch is now known as jcapitao | 12:50 | |
| *** raissa has quit IRC | 12:53 | |
| *** reed has quit IRC | 13:08 | |
| *** reed has joined #openstack-infra | 13:08 | |
| *** reed has quit IRC | 13:18 | |
| *** reed has joined #openstack-infra | 13:19 | |
| *** ociuhandu has quit IRC | 13:19 | |
| *** weshay|ruck has joined #openstack-infra | 13:22 | |
| *** ociuhandu has joined #openstack-infra | 13:22 | |
| *** Goneri has joined #openstack-infra | 13:27 | |
| *** jpena|lunch is now known as jpena | 13:29 | |
| *** zzzeek has quit IRC | 13:31 | |
| *** zzzeek has joined #openstack-infra | 13:34 | |
| *** d34dh0r53 has joined #openstack-infra | 13:45 | |
| *** slaweq has quit IRC | 13:47 | |
| *** tbarron has joined #openstack-infra | 13:47 | |
| *** slaweq has joined #openstack-infra | 13:50 | |
| weshay|ruck | ianw, have you seen https://review.opendev.org/#/c/762812 | 13:52 |
| weshay|ruck | zbr, hey | 13:54 |
| zbr | weshay|ruck: hi | 13:54 |
| *** zzzeek has quit IRC | 14:02 | |
| *** zzzeek has joined #openstack-infra | 14:07 | |
| *** zzzeek has quit IRC | 14:14 | |
| *** zzzeek has joined #openstack-infra | 14:14 | |
| *** zxiiro has joined #openstack-infra | 14:21 | |
| *** reed has quit IRC | 14:29 | |
| *** reed has joined #openstack-infra | 14:29 | |
| *** dtroyer has quit IRC | 14:31 | |
| zbr | fungi: clarkb: around to help with the mirroring swap? | 14:33 |
| *** zzzeek has quit IRC | 14:33 | |
| *** zzzeek has joined #openstack-infra | 14:36 | |
| fungi | did someone find a more current rsync mirror to switch to? every time we pick one, it eventually breaks or goes stale and we have to pick another | 14:39 |
| *** artom has joined #openstack-infra | 14:40 | |
| *** zzzeek has quit IRC | 14:41 | |
| *** zzzeek has joined #openstack-infra | 14:45 | |
| zbr | fungi: that is why CDNs were invented. | 14:50 |
| zbr | they are supposed to resolve the burden of maintaining list of mirrors,... if they work fine. | 14:51 |
| *** xek__ has joined #openstack-infra | 14:52 | |
| *** reed has quit IRC | 14:54 | |
| *** reed has joined #openstack-infra | 14:55 | |
| *** xek_ has quit IRC | 14:55 | |
| *** beekneemech is now known as bnemec | 15:00 | |
| *** reed has quit IRC | 15:05 | |
| *** reed has joined #openstack-infra | 15:05 | |
| *** lpetrut has quit IRC | 15:05 | |
| *** reed has quit IRC | 15:10 | |
| *** reed has joined #openstack-infra | 15:10 | |
| *** zzzeek has quit IRC | 15:11 | |
| *** zzzeek has joined #openstack-infra | 15:14 | |
| *** ociuhandu has quit IRC | 15:32 | |
| *** hashar has quit IRC | 15:32 | |
| *** ociuhandu has joined #openstack-infra | 15:32 | |
| *** ociuhandu has quit IRC | 15:33 | |
| *** ociuhandu has joined #openstack-infra | 15:33 | |
| *** dave-mccowan has joined #openstack-infra | 15:34 | |
| *** redrobot has quit IRC | 15:35 | |
| *** xek_ has joined #openstack-infra | 15:37 | |
| *** xek__ has quit IRC | 15:40 | |
| *** dave-mccowan has quit IRC | 15:45 | |
| *** hashar has joined #openstack-infra | 15:48 | |
| *** zzzeek has quit IRC | 15:49 | |
| *** zzzeek has joined #openstack-infra | 15:50 | |
| *** Guest64097 has joined #openstack-infra | 15:51 | |
| *** Guest64097 is now known as redrobot | 15:53 | |
| fungi | zbr: yes, point me to the rsync cdn then? | 15:55 |
| fungi | i'm sure it will solve our problems once and for all | 15:55 |
| fungi | logan-: if you're around, not sure if you have any insight into the mirror.lstn.net service but it's apparently got stale centos 7.x packages | 15:56 |
| zbr | logan was idea for the last 5 days. | 15:57 |
| zbr | idle | 15:57 |
| zbr | logan was idle | 15:57 |
| zbr | fungi: i am trying to understand how https://review.opendev.org/#/c/762812/ can make the mirroring worse than now. | 15:58 |
| *** zzzeek has quit IRC | 15:59 | |
| clarkb | zbr: rsync tries to connect and fails every time it attempts to connect because of connection limits | 16:00 |
| clarkb | then you end up stale because you're never syncing | 16:00 |
| *** zzzeek has joined #openstack-infra | 16:00 | |
| clarkb | right now we'll get updated if our upstream updates | 16:00 |
| clarkb | that is better | 16:00 |
| fungi | zbr: also i was unaware of 762812 until just now. i suppose that was some missing context from your question above | 16:03 |
| *** dklyle has joined #openstack-infra | 16:05 | |
| zbr | yep that is the change. I am not sure about the mentioned connection limits, maybe someone can explain them to me. | 16:06 |
| fungi | i'm not opposed to going back to trying mirrors.k.o, but as mentioned by others our past experience is that most of the time we try to connect we get an error saying the connection limit has been reached so we rarely manage to pull updates from there | 16:07 |
| clarkb | zbr: the kernel.org rsync mirrors limit the number of rsync connections to it. They are a very popular mirror so are basically always at the limit. When they are at the limit our rsync connections to them fail and we don't sync | 16:07 |
| fungi | especially if there's a major centos or similar update, we've seen it go weeks without an open slot for connections | 16:07 |
| clarkb | at least with the current situation when the current mirror does update we will update. If we switch to kernel.org we likely won't update reliably ever | 16:08 |
| zbr | how how load do these limits stand for? | 16:08 |
| fungi | and we try to be polite (probably unlike others pulling from there) so we don't aggressively reconnect we just try once every couple hours | 16:08 |
| zbr | are they implemented at DNS level or at specific edge server? | 16:08 |
| clarkb | you cannot implement connection limits in dns | 16:08 |
| fungi | zbr: they are global. x number of clients allowed to connect to the mirror | 16:08 |
| clarkb | its tcp level go away | 16:09 |
| fungi | so something like if 25 people are downloading from it at the same time it serves up an error to anyone else who connects until the connection count drops below their limit | 16:09 |
| zbr | ok but I do not expect it to continue to fail for a very long period of time. | 16:09 |
| fungi | (i don't recall the exact number) | 16:09 |
| clarkb | I think it was 50 | 16:10 |
| fungi | like i've said, we've seen it take weeks to pull an update from there in the past. maybe it's better now? i'm not opposed to trying again, just setting expectations | 16:10 |
| clarkb | why not use a different mirror that we don't expect to fail? | 16:10 |
| *** reed has quit IRC | 16:10 | |
| *** reed has joined #openstack-infra | 16:11 | |
| clarkb | I think that was the point of pabelanger's comment | 16:11 |
| clarkb | "kernel.org is expected to fail, let's find another alternative" | 16:11 |
| fungi | the counterargument is that other mirrors will eventually run into problems and cease updating for extended periods of time, while mirrors.k.o has someone grooming the round robin list and removing stale mirrors | 16:11 |
| zbr | fungi: yep, that was my point. that someone else is supposed to maintain the list of actively maintained mirror. | 16:12 |
| *** jamesdenton has quit IRC | 16:13 | |
| fungi | well, not exactly round-robin, looks like it relies on dns-based load distribution like 3dns or similar | 16:13 |
| * zbr is also aware that whatever we do is a gamble, but lets try to make it an informed gamble. | 16:13 | |
| *** jamesdenton has joined #openstack-infra | 16:13 | |
| clarkb | interesting that centos mirror list has kernel.org mirrors in california (they are in oregon) | 16:15 |
| fungi | for me it resolves to something that claims to be in/near san francisco | 16:16 |
| clarkb | not that important, and ya if we want to try it and see if it works then we can do that. But it would also probably be a good idea to look for other alternatives as well | 16:16 |
| clarkb | fungi: huh, I checked from my ovh node and home and they both point to oregon. I guess they don't exclusively host with osuosl then | 16:16 |
| clarkb | zbr: to be clear I'm not sure you assertion in the change is correct. We have seen it break perpetually for long periods in the past | 16:17 |
| *** zzzeek has quit IRC | 16:17 | |
| clarkb | its more like you succeeds once every few weeks if at all vs fail once every few weeks if at all | 16:17 |
| *** zzzeek has joined #openstack-infra | 16:20 | |
| fungi | looks like the last time we tried it for centos may have been before https://review.openstack.org/551318 switched to limestone a couple years ago | 16:20 |
| fungi | though we've more recently had to stop mirroring epel from there or opensuse for various reasons | 16:20 |
| zbr | almost 3 years. i guess reverting the change could allow us to see if situation improved. | 16:22 |
| zbr | if it does not, we can start benchmarking specific mirrors | 16:22 |
| *** reed has quit IRC | 16:23 | |
| fungi | https://review.openstack.org/553052 is when we moved off mirrors.k.o for epel, and was also around the same time | 16:24 |
| fungi | also cites "connection issues" so i assume that was the max clients thing as well | 16:25 |
| fungi | a year ago we moved opensuse mirroring around because mirrors.k.o and others had outdated content | 16:25 |
| clarkb | the big issue with rsync connection limits aiui is that connections tend to be held open for long periods of time when they are used | 16:25 |
| clarkb | which causes them to fill the pool quickly. | 16:25 |
| clarkb | the way around that is to authenticate and bypass the pool limits for anonymous access, but to do that you typically need to be a proper first level mirror and we're not interested in doing that as the requirements are much more stringent | 16:26 |
| clarkb | basically we want to mirror for our CI system not for everyone | 16:26 |
| clarkb | (fwiw I THink that is a reasonable requirement on the part of the mirroring systems out there. They have limited bandwidth so distributing to as wide an audience as possible first is a good idea) | 16:30 |
| zbr | clarkb: do you know were is the rsync mirroring documented? I was not able to find any link on hints on http://mirrors.kernel.org/ | 16:39 |
| clarkb | zbr: I don't. I know the rsync urls are listed on the mirror list though | 16:39 |
| zbr | i do see disclaimer but not a clue about how is setup, limitations,... eventually the tiers. | 16:40 |
| *** eolivare has quit IRC | 16:44 | |
| *** reed has joined #openstack-infra | 16:49 | |
| *** eolivare has joined #openstack-infra | 16:53 | |
| *** mgoddard has joined #openstack-infra | 16:55 | |
| *** lucasagomes has quit IRC | 17:03 | |
| *** amoralej|off has joined #openstack-infra | 17:03 | |
| *** amoralej|off has left #openstack-infra | 17:03 | |
| fungi | jpena: weshay|ruck: zbr: the mirror change has landed, it should hopefully be in place by the time the next rsync is attempted at 18:43 utc. you'll be able to see the logs at https://static.opendev.org/mirror/logs/rsync-mirrors/centos.log | 17:09 |
| zbr | thanks | 17:09 |
| weshay|ruck | fungi++ clarkb++ | 17:10 |
| weshay|ruck | thanks | 17:10 |
| weshay|ruck | rlandy|rover, ^^ | 17:10 |
| *** hamalq has joined #openstack-infra | 17:12 | |
| rlandy|rover | great - thanks | 17:12 |
| *** jcapitao has quit IRC | 17:14 | |
| *** kwazar has joined #openstack-infra | 17:35 | |
| *** zzzeek has quit IRC | 17:37 | |
| *** zzzeek has joined #openstack-infra | 17:38 | |
| *** ociuhandu_ has joined #openstack-infra | 17:38 | |
| *** rpittau is now known as rpittau|afk | 17:41 | |
| *** ociuhandu has quit IRC | 17:41 | |
| *** ociuhandu_ has quit IRC | 17:43 | |
| *** zzzeek has quit IRC | 17:45 | |
| *** zzzeek has joined #openstack-infra | 17:48 | |
| *** ociuhandu has joined #openstack-infra | 17:49 | |
| *** ociuhandu has quit IRC | 17:54 | |
| *** mgoddard has quit IRC | 18:05 | |
| *** jpena is now known as jpena|off | 18:10 | |
| faruqh | i could really really use some help - I am trying to install charmed openstack on three KVM from maas and I keep getting hung on the mysql-innodb-cluster. It tries and then it just gets stuck | 18:22 |
| clarkb | faruqh: we offer help with the developer infrastructure used to build the openstack software here. We don't really do much with the deployment side of openstack itself | 18:23 |
| clarkb | faruqh: I think there is an openstack charms channel though which might be a better place for this question. Let me see if I can find it | 18:23 |
| faruqh | iawesome clarkb thanks | 18:23 |
| clarkb | hrm no they have held meetings int he past but no separate channel that I see | 18:24 |
| clarkb | still looking though | 18:24 |
| faruqh | thank you - | 18:24 |
| clarkb | ah it may be #openstack-charms according to their docs | 18:24 |
| faruqh | clarkbthanks alot | 18:25 |
| clarkb | you're welcome | 18:25 |
| *** gfidente is now known as gfidente|afk | 18:37 | |
| *** eolivare has quit IRC | 18:53 | |
| *** zzzeek has quit IRC | 18:59 | |
| *** reed has quit IRC | 18:59 | |
| *** reed has joined #openstack-infra | 18:59 | |
| *** zzzeek has joined #openstack-infra | 19:00 | |
| *** andrewbonney has quit IRC | 19:10 | |
| fungi | jpena: weshay|ruck: zbr: rlandy|rover: the centos 7.x mirror has updated, see the end of the log for what was updated... https://static.opendev.org/mirror/logs/rsync-mirrors/centos.log | 19:30 |
| * weshay|ruck looks.. thanks fungi | 19:30 | |
| rlandy|rover | fungi: thank - job running to test centos-7 now | 19:31 |
| fungi | we'll want to keep an eye on it to make sure we don't start running into the rsync server being full on its client limit | 19:39 |
| fungi | since we've seen m.k.o do that for extended periods in the past | 19:40 |
| *** dciabrin has quit IRC | 19:52 | |
| *** dciabrin has joined #openstack-infra | 19:52 | |
| *** reed has quit IRC | 19:54 | |
| *** reed has joined #openstack-infra | 19:55 | |
| *** reed has quit IRC | 19:59 | |
| *** reed has joined #openstack-infra | 19:59 | |
| *** thiagop has quit IRC | 20:00 | |
| *** ralonsoh has quit IRC | 20:00 | |
| *** dtantsur is now known as dtantsur|afk | 20:02 | |
| *** reed_ has joined #openstack-infra | 20:11 | |
| *** reed has quit IRC | 20:12 | |
| *** reed has joined #openstack-infra | 20:12 | |
| *** faruqh has quit IRC | 20:14 | |
| *** hashar has quit IRC | 20:21 | |
| *** iurygregory has quit IRC | 20:40 | |
| *** iurygregory has joined #openstack-infra | 20:43 | |
| *** raissa has joined #openstack-infra | 20:44 | |
| *** ociuhandu has joined #openstack-infra | 20:55 | |
| *** ociuhandu has quit IRC | 20:57 | |
| *** arne_wiebalck has quit IRC | 21:13 | |
| *** arne_wiebalck has joined #openstack-infra | 21:13 | |
| *** lbragstad_ has joined #openstack-infra | 21:21 | |
| *** lbragstad has quit IRC | 21:24 | |
| *** rcernin has joined #openstack-infra | 21:50 | |
| *** rcernin has quit IRC | 21:51 | |
| *** rcernin has joined #openstack-infra | 21:51 | |
| *** matt_kosut has quit IRC | 21:57 | |
| *** rcernin has quit IRC | 22:02 | |
| *** matt_kosut has joined #openstack-infra | 22:04 | |
| *** rcernin has joined #openstack-infra | 22:06 | |
| *** matt_kosut has quit IRC | 22:08 | |
| *** sboyron has quit IRC | 22:08 | |
| *** sboyron has joined #openstack-infra | 22:09 | |
| *** rlandy|rover is now known as rlandy|rover|bbl | 22:28 | |
| *** xek_ has quit IRC | 22:29 | |
| *** lbragstad__ has joined #openstack-infra | 22:35 | |
| *** rledisez has quit IRC | 22:36 | |
| melwitt | I'm looking at an example of a TIMED_OUT job and wondering why don't I see the devstack and tempest steps (and their timings) in the console? I feel like they used to be there https://zuul.openstack.org/build/75d7b93f5385449589540c2657422841/console | 22:36 |
| *** rledisez has joined #openstack-infra | 22:36 | |
| clarkb | melwitt: it is likely due to the fact that when a job is timed out ansible is forceibly killed. And when that happens the json object that console renders from is not properly finished up | 22:36 |
| clarkb | melwitt: in those cases the console log should hae the info you need /me looks too | 22:37 |
| melwitt | ah ok. I can look at timings in the job-output.txt but it would be easier if it were in the console. just trying to identify any trends in why certain jobs time out more than others | 22:38 |
| *** lbragstad_ has quit IRC | 22:38 | |
| clarkb | ya the reason it isn't there is ansible was killed during the run phase due to a timeout | 22:39 |
| clarkb | and that means that we don't write out a properly formatted json file for the console to render | 22:39 |
| melwitt | I see | 22:39 |
| *** slaweq has quit IRC | 22:39 | |
| fungi | i wonder if we could do a better job of redirecting folks to the console log in those situations | 22:40 |
| clarkb | https://f75b41bb61072b844b31-b8d0e37c8407321b7ba0b13922acea87.ssl.cf1.rackcdn.com/735570/2/check/nova-ceph-multistore/75d7b93/job-output.json shows this as it goes pre pre pre post | 22:40 |
| melwitt | hm ok | 22:40 |
| melwitt | I could have sworn the devstack step was separate from the tempest step, but it must all fall under "run" I guess | 22:43 |
| clarkb | yes in those jobs I think they are both part of the run playbook | 22:43 |
| melwitt | gotcha, I understand now | 22:43 |
| clarkb | some jobs put devstack in pre because they are less concerned with whether or not a working cloud is produced and just want to assume one exists (sdk tests for example) | 22:43 |
| melwitt | I see | 22:45 |
| *** slaweq has joined #openstack-infra | 22:48 | |
| *** tosky_ has joined #openstack-infra | 22:52 | |
| *** tosky is now known as Guest41419 | 22:55 | |
| *** tosky_ is now known as tosky | 22:55 | |
| *** Guest41419 has quit IRC | 22:55 | |
| *** slaweq has quit IRC | 22:58 | |
| *** sboyron has quit IRC | 23:03 | |
| *** slaweq has joined #openstack-infra | 23:07 | |
| *** slaweq has quit IRC | 23:13 | |
| *** gibi has quit IRC | 23:20 | |
| *** gibi has joined #openstack-infra | 23:27 | |
| *** dchen has joined #openstack-infra | 23:41 | |
| *** raissa has quit IRC | 23:48 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!