| *** dviroel|out is now known as dviroel | 00:05 | |
| *** dviroel is now known as dviroel|out | 00:20 | |
| *** rlandy|bbl is now known as rlandy|out | 01:17 | |
| *** soniya29 is now known as soniya29|rover | 04:19 | |
| *** ysandeep|out is now known as ysandeep | 05:05 | |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: Add regex for g-api and c-vol timestamp https://review.opendev.org/c/openstack/ci-log-processing/+/835934 | 06:10 |
|---|---|---|
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: Add Opensearch configuration information; change README file; enable md https://review.opendev.org/c/openstack/ci-log-processing/+/832659 | 07:11 |
| *** ysandeep is now known as ysandeep|lunch | 07:23 | |
| opendevreview | chzhang8 proposed openstack/project-config master: register and bring back tricircle under x namespaces https://review.opendev.org/c/openstack/project-config/+/800442 | 07:34 |
| *** jpena|off is now known as jpena | 07:34 | |
| opendevreview | chzhang8 proposed openstack/project-config master: register and bring back tricircle under x namespaces https://review.opendev.org/c/openstack/project-config/+/800442 | 07:55 |
| opendevreview | chzhang8 proposed openstack/project-config master: create tricircle under x namespaces https://review.opendev.org/c/openstack/project-config/+/800442 | 08:28 |
| opendevreview | chzhang8 proposed openstack/project-config master: create tricircle under x namespaces https://review.opendev.org/c/openstack/project-config/+/800442 | 09:02 |
| *** ysandeep|lunch is now known as ysandeep | 10:07 | |
| *** soniya29|rover is now known as soniya29|rover|lunch | 10:08 | |
| *** rlandy|out is now known as rlandy | 10:25 | |
| *** soniya29|rover|lunch is now known as soniya29|rover | 10:27 | |
| opendevreview | Merged openstack/openstack-zuul-jobs master: Add stable/yoga to periodic-stable templates https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/835230 | 10:30 |
| *** bhagyashris_ is now known as bhagyashris | 11:09 | |
| *** soniya29|rover is now known as soniya29|rover|afk | 11:18 | |
| *** dviroel_ is now known as dviroel | 11:22 | |
| *** ysandeep is now known as ysandeep|brb | 11:49 | |
| *** ysandeep|brb is now known as ysandeep|afk | 12:05 | |
| *** soniya29 is now known as soniya29|rover | 12:07 | |
| sean-k-mooney | have there been any infra issues with opendev.org git repos lately? i was seeing very slow clone times as in singel to double digit KiB/s | 12:22 |
| sean-k-mooney | /times/rates/ | 12:23 |
| frickler | sean-k-mooney: we did a gitea upgrade recently, so that might be related. is it reproducible? via v4 or v6? specific repos or anything? | 12:31 |
| sean-k-mooney | it seamed ot be all repos i just exported GIT_BASE=https://github.com for now | 12:33 |
| sean-k-mooney | it was ipv4 i belive at least i got an ipv4 adress when i ping opendev.org | 12:34 |
| sean-k-mooney | i can try doing a verbose clone and see what its useing | 12:34 |
| sean-k-mooney | i noticed one of my devstack buildes spend 2600+ second in the git_timed section | 12:35 |
| sean-k-mooney | opendev.org. 3017 IN A 38.108.68.124 | 12:37 |
| sean-k-mooney | verbose clone does not show much extra but it seam to happen on all repos i clone. i was cloneing placment but os-vif is a nice small repo to test with | 12:39 |
| frickler | infra-root: ^^ I can confirm this at least on some attempts getting rates ~ 100 KB/s, while others run much faster, maybe someone can doublecheck who is not 200ms RTT away | 13:04 |
| frickler | fwiw according to SSL I'm landing on gitea03, but nothing obvious in cacti | 13:05 |
| fungi | sean-k-mooney: frickler: i haven't looked, but my money's on everyone doing openstack yoga upgrades from source | 13:07 |
| fungi | probably the servers are just overwhelmed | 13:07 |
| sean-k-mooney | ya could be | 13:08 |
| sean-k-mooney | i have updated my ansible role to use github for now. implementeing a repo and pip cache is on my todo list but wanted to keep it simple first | 13:10 |
| fungi | we saw our git server farm get slammed regularly in the weeks immediately following the past few major openstack releases, so i was sort of bracing for it to happen again | 13:14 |
| *** ysandeep|afk is now known as ysandeep | 13:14 | |
| fungi | day after the release is announced is a strong correlation, but i'll try to take a look at the resource consumption graphs in cacti to see if it pans out | 13:15 |
| fungi | i'm away most of today though, so probably not much help past that | 13:15 |
| sean-k-mooney | i assume you cant add cloudflare or one of the other cdn providers that has an free opensource hosting policy in front of it | 13:16 |
| sean-k-mooney | fungi: dont worry about it | 13:16 |
| sean-k-mooney | as i said i just used GIT_BASE to work around it by using github for now and gerrit seams fine | 13:16 |
| fungi | sean-k-mooney: i'm not sure caching git requests would work all that well without some custom protocol support for it, but if that's a possibility then we could probably implement it ourselves with squid or similar | 13:18 |
| sean-k-mooney | ya im not sure either i looked into it a few years ago ci ci reasons but then we just used zuul and it took care if it | 13:20 |
| fungi | right, the load problem has generally been that for clone operations and the like, the entire repository has to be read into memory. do that 4-5 times with the nova repo on the same backend, and it's oom'sville | 13:22 |
| sean-k-mooney | right so i wonder is there a way to cache that when its a full clone at least | 13:31 |
| sean-k-mooney | there might not be | 13:32 |
| sean-k-mooney | but ya nova is non trivial | 13:32 |
| frickler | really sad that we had to disable shallow cloning with gitea. testing from github that seems to make a huge impact both in term of time and data volume. like a factor of 5 for the nova repo for example | 13:39 |
| sean-k-mooney | is that just not supported by gitea or was there another reason it was disabled | 13:43 |
| sean-k-mooney | although if you really hate your self you shoudl try cloning nova form gerrit... with all the review data in the repo | 13:44 |
| sean-k-mooney | the gerrit version is over a gig on disk | 13:46 |
| frickler | sean-k-mooney: gitea has a bug that makes older git clients fail. https://github.com/go-gitea/gitea/issues/19118 | 13:52 |
| frickler | doing the same clone from github works fine | 13:52 |
| sean-k-mooney | ah how old is old | 13:53 |
| frickler | and yeah, cloning from gerrit would be even worse | 13:53 |
| frickler | sean-k-mooney: git in ubuntu focal is broken, impish works | 13:53 |
| sean-k-mooney | ack ya i tought you were going to say centos 8 or something | 13:54 |
| sean-k-mooney | but 20.04 is not that old | 13:54 |
| fungi | okay, so i did not get time to check the cacti graphs before leaving, but memory consumption and swap utilization driving system cpu and load average up are the symptoms to look for | 13:54 |
| fungi | when the server falls over, haproxy will temporarily remove it from the pool and then the problem load will shift to other servers, so you'll see sharp cliffs from that | 13:55 |
| *** dasm|off is now known as dasm | 14:00 | |
| clarkb | the gitea update was also just a bugfix update and those tend to be very small deltas | 15:29 |
| clarkb | I just got 1.65MBps whcih is fairly typical for me and the git cluster | 15:34 |
| *** ysandeep is now known as ysandeep|out | 15:40 | |
| *** dviroel_ is now known as dviroel | 15:54 | |
| opendevreview | Merged openstack/project-config master: Revert "[OVH/GRA1] Disable nodepool temporarily" https://review.opendev.org/c/openstack/project-config/+/835422 | 15:59 |
| *** rlandy is now known as rlandy|rover | 16:09 | |
| *** dviroel is now known as dviroel|lunch | 16:14 | |
| *** dasm is now known as dasm|ruck | 16:38 | |
| *** jpena is now known as jpena|off | 16:39 | |
| *** dviroel|lunch is now known as dviroel | 17:18 | |
| rezabojnordi | Hi guys | 19:27 |
| rezabojnordi | I have question about my architecture? | 19:27 |
| *** dviroel is now known as dviroel|afk | 20:24 | |
| opendevreview | Merged openstack/ci-log-processing master: Add regex for g-api and c-vol timestamp https://review.opendev.org/c/openstack/ci-log-processing/+/835934 | 20:43 |
| *** dasm|ruck is now known as dasm|off | 22:59 | |
| *** rlandy|rover is now known as rlandy|out | 23:08 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!