| *** ysandeep|out is now known as ysandeep | 00:14 | |
| *** rlandy is now known as rlandy|out | 00:28 | |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: update afs for centos 9 wheel volumes https://review.opendev.org/c/openstack/project-config/+/837076 | 01:54 |
|---|---|---|
| opendevreview | Merged openstack/project-config master: grafana: update afs for centos 9 wheel volumes https://review.opendev.org/c/openstack/project-config/+/837076 | 02:57 |
| *** chandankumar is now known as chkumar|ruck | 05:34 | |
| *** jpena|off is now known as jpena | 07:02 | |
| *** ysandeep is now known as ysandeep|lunch | 07:55 | |
| *** ysandeep|lunch is now known as ysandeep | 08:56 | |
| *** ysandeep is now known as ysandeep|afk | 09:58 | |
| *** ysandeep|afk is now known as ysandeep | 10:16 | |
| *** bhagyashris_ is now known as bhagyashris | 10:19 | |
| *** rlandy|out is now known as rlandy|rover | 10:21 | |
| *** dasm|off is now known as dasm | 11:16 | |
| *** dviroel|afk is now known as dviroel | 11:28 | |
| *** ysandeep is now known as ysandeep|afk | 12:06 | |
| *** ysandeep|afk is now known as ysandeep | 14:05 | |
| fzzf[m] | fungi: clarkb Hi, source-repositories how to use local image to build diskimage.like these https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/cache-devstack/source-repository-images .I want download cirrors, and use these. | 14:59 |
| clarkb | fzzf[m]: if yo uadd that cache-devstack element it will do it for you | 15:01 |
| clarkb | or you can add your own element with your own list of images like that | 15:01 |
| fzzf[m] | clarkb: I have add cache-devstack element, but I found every time build diskimage downloads cirrors from the network... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/HrTrDjfIEyfEHLUYkgtQKOav) | 15:04 |
| fungi | fzzf[m]: it should only download it if the file is not already in the dib cache on the nodepool builder | 15:04 |
| fungi | did it ever successfully download and cache the file? | 15:05 |
| fungi | or are you removing/not persisting that cache? | 15:05 |
| clarkb | ya there is nothing we can do about you not being able to download their images. YOu'll hvae to debug that on your side | 15:07 |
| clarkb | its possible they don't download for us but we juts added rocky images that build properly | 15:07 |
| fzzf[m] | fungi: how to check cache status. | 15:07 |
| clarkb | I just successfully downloaded the first image in the list | 15:07 |
| fungi | yeah, i tested downloading them at home when the question of whether the builder is behind a proxy came up | 15:09 |
| fzzf[m] | I can download images. but there was a failure during the build process | 15:10 |
| clarkb | fzzf[m]: the failure is downloading those images in the build process (at least that was what you linked previously and just indicated above) | 15:10 |
| fzzf[m] | clarkb: If I build successfully, nodepool will save the cache locally, right? | 15:11 |
| clarkb | yes | 15:12 |
| fzzf[m] | clarkb: okay, there are some other problems I will fix, then try to build. thanks :) | 15:13 |
| elodilles | hi infra team, i have a task in my release managements tasks list for this week: 'Coordinate with the Infrastructure team to swap out the previous cycle signing key and establish the new one for the starting cycle.' | 15:25 |
| elodilles | o:) | 15:25 |
| elodilles | so if you could help with this ^^^, that would be awesome :) | 15:25 |
| clarkb | I thought that happened but I'm trying to check now | 15:27 |
| clarkb | our docs don't seem to indicate where in our zuul configs we se tit | 15:27 |
| clarkb | I'm wrong it last rotated in october | 15:29 |
| clarkb | openstack/project-config/zuul.d/secrets.yaml | 15:29 |
| clarkb | I guess the change I remember was on the openstack/governance side updating key fingerprints and now we need to update hte actual key | 15:30 |
| fungi | clarkb: elodilles: the key blob needs to be exported, encoded and pushed, yes. it's on my to do list | 15:31 |
| fungi | and yeah, what was done earlier was publishing the upcoming public key to the releases repo/site | 15:32 |
| elodilles | clarkb: fungi: ack, thanks for dealing with it o/ | 15:35 |
| fungi | once the change to update the secret merges, then i'll push a separate change to update the use date ranges for the keys on the releases site | 15:36 |
| *** ysandeep is now known as ysandeep|out | 15:55 | |
| *** ysandeep|out is now known as ysandeep|PTO | 15:55 | |
| *** dviroel is now known as dviroel|lunch | 15:59 | |
| *** jpena is now known as jpena|off | 16:05 | |
| *** dviroel|lunch is now known as dviroel|mtg | 16:55 | |
| *** dviroel|mtg is now known as dviroel | 17:19 | |
| dansmith | clarkb: still around? | 18:06 |
| clarkb | dansmith: ya | 18:06 |
| dansmith | clarkb: so, let's say at the end of a tempest run I have generated a nice json dict of some stats (db queries, memory usage, etc) | 18:06 |
| dansmith | is there something we could plug into the log pipeline to digest that in some way, either just to compare that job to the average of recent runs? | 18:07 |
| clarkb | I think you could have elasticsearch ingest it and then from there you could use their tooling to graph it over time etc | 18:08 |
| dansmith | like, I wonder if we were to dump it into graphite on successful gate runs or something and use that to calculate an average (per job name) that we could compare against for a given proposed patch | 18:08 |
| clarkb | getting it into graphite is harder because of the way we firewall things off | 18:08 |
| clarkb | but it might be doable. I'd have to think about that | 18:08 |
| clarkb | grafana can graph data in ES though so maybe thats the easiest thing | 18:08 |
| dansmith | does elasticsearch tooling have graphing stuff? | 18:08 |
| dansmith | ah okay | 18:08 |
| clarkb | is have grafana talk to the ES as a secondary backend | 18:08 |
| fungi | keep in mind that the elasticsearch/opensearch retention is short, so graphing it *there* probably wouldn't make sense if you want a longer trend | 18:09 |
| dansmith | fungi: I'm more thinking "is this run's number higher than the average (of this job) for the past 7 days? | 18:09 |
| dansmith | (substantially higher of course) | 18:09 |
| fungi | for the past 7 days would almost certainly fit | 18:09 |
| fungi | if you want trending over months, you'd need to extract it to another location | 18:10 |
| dansmith | plotting over the course of a cycle or more would be something else for sure | 18:10 |
| dansmith | yeah | 18:10 |
| clarkb | ya I think starting there is straightforward then if we want to go longer we can think harder about getting the data in graphite | 18:10 |
| clarkb | and hopfeully you just change out the data source in the existing graphs and that is easy | 18:11 |
| clarkb | basically generating the data and making graphs for it can be constants then swap out backends if necessary | 18:11 |
| dansmith | but I guess this is something to talk to dpawlik about yeah? | 18:12 |
| clarkb | ya and or the opensearch folks they had talked about being interestd in specific use caess like this (at least to hear how we are trying to use it) | 18:12 |
| clarkb | I'm pretty sure that elasticsearch can store data like this out of the box though so should be good to go | 18:13 |
| dansmith | clarkb: https://termbin.com/suwq | 18:21 |
| clarkb | dansmith: https://www.elastic.co/blog/found-crash-elasticsearch#mapping-explosion indicates how to write your json to make es happier but that looks great otherwise | 18:27 |
| clarkb | then I think you ask dpawlik to ingest it as a json document (not logs) and you can then query/search it | 18:28 |
| dansmith | I'd expect it needs to be flat for something like es or graphite, I was mostly trying to make it readable for humans as a first go and then I can translate | 18:28 |
| dansmith | but yeah | 18:28 |
| dansmith | okay cool | 18:28 |
| *** dviroel is now known as dviroel|brb | 18:28 | |
| clarkb | dansmith: you might also want to capture privsep memory use as last I looked it was quite high | 18:30 |
| clarkb | and ist a bit more hidden | 18:30 |
| dansmith | I assume it's mixed inside one of those values currently | 18:31 |
| clarkb | oh as a child process. That may be | 18:31 |
| clarkb | dpawlik: for the ingstion of that I think you want ot do it into a different index that will make querying and pruning simpler | 18:32 |
| clarkb | and actually that may enable much longer retention time for the data since it should be fairly small | 18:32 |
| dansmith | well, I see it running for neutron on my devstack machine, but not sure it's under either of the systemd service trees, which seems weird | 18:32 |
| dansmith | I guess it double-forked to get out of that | 18:33 |
| dansmith | so yeah | 18:33 |
| clarkb | so ya my suggestion is json reformated to make ES happy, ingested as json doc to a dedicated index or set of indexes. Then when grafana queries it can query that specific index prefix insted. And if sizes are small as expected can keep more days of data around | 18:34 |
| * dansmith nods | 18:37 | |
| *** dviroel|brb is now known as dviroel | 18:41 | |
| dansmith | yeah, 82MB for just neutron's privsep helper... seems like a lot | 18:49 |
| *** akahat|rover is now known as akahat | 18:56 | |
| dansmith | clarkb: https://termbin.com/xger | 18:57 |
| dansmith | still pretty readable if that's flat enough | 18:57 |
| fungi | yeah, i think dpawlik said he expected to be around again next week, so probably good to sync up then on options | 19:02 |
| dansmith | I hope turning on the perf schema query logging won't slow things down too much | 19:05 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Replace old Yoga cycle signing key with Zed https://review.opendev.org/c/openstack/project-config/+/837174 | 19:21 |
| fungi | elodilles: once that ^ lands, we'll want to do a release-test tag to make sure things aren't somehow broken | 19:21 |
| fungi | and then i'll update the dates on the releases.o.o index page | 19:21 |
| elodilles | fungi: ack, noted | 20:00 |
| clarkb | dansmith: I doubt it will slow things down ES is really good at handlings docs like this aiui. Its the log use case it struggles with | 20:42 |
| *** dviroel is now known as dviroel|out | 20:42 | |
| clarkb | it still deals with random text like that better than a lot of options but it is really good at json doc storage aiui | 20:43 |
| clarkb | and ya I think that json looks like what it wants | 20:55 |
| dansmith | clarkb: I meant the mysql flags I'm having to turn on to get the per-db numbers | 21:06 |
| dansmith | but it looks like maybe it's okay based on the last run.. we'll see | 21:06 |
| clarkb | oh sorry. I expect simple counters aren't too bad. The analysis stuff is probably much more costly. But maybe also a good thing to run separately (specifically query analyzer against common queries) | 21:08 |
| dansmith | right, but I don't see anywhere I can get db query stats without full query logging | 21:08 |
| dansmith | but I haven't dug very deep.. once I get this to actually run I'll survey more | 21:08 |
| *** dasm is now known as dasm|off | 22:44 | |
| opendevreview | Clark Boylan proposed openstack/openstack-zuul-jobs master: Add build-wheel-cache jobs for CentOS Stream 9 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/836829 | 23:44 |
| clarkb | I don't think ^ will work but it helps capture what I learned and where I ended up | 23:44 |
| fungi | dansmith: probably goes without saying, but please make sure we don't collect mysql query logs from the jobs (there was an incident some years back with slowlog collection in devstack runs bloating logging massively) | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!