| opendevreview | Takashi Kajinami proposed openstack/project-config master: Fix wrong project removed https://review.opendev.org/c/openstack/project-config/+/875325 | 02:23 |
|---|---|---|
| tkajinam | ^^^ would be nice if we can merge this early. | 02:24 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Retire puppet-tacker - Step 1: End project Gating https://review.opendev.org/c/openstack/project-config/+/874539 | 02:24 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Retire puppet-tacker - Step 5: Remove Project https://review.opendev.org/c/openstack/project-config/+/875291 | 02:24 |
| fungi | tkajinam: i've already approved it, thanks! | 02:30 |
| tkajinam | fungi, thank you ! | 02:31 |
| fungi | well, i was also the one who approved the broken change :/ | 02:32 |
| tkajinam | all good as long as you help me fix the breakage :-) | 02:34 |
| opendevreview | Merged openstack/project-config master: Fix wrong project removed https://review.opendev.org/c/openstack/project-config/+/875325 | 02:50 |
| fungi | tkajinam: ^ it's in now. thanks again! | 02:55 |
| tkajinam | :-) | 02:56 |
| *** yadnesh|away is now known as yadnesh | 04:49 | |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Retire puppet-tacker - Step 5: Remove Project https://review.opendev.org/c/openstack/project-config/+/875291 | 07:47 |
| *** jpena|off is now known as jpena | 08:36 | |
| *** jpena is now known as jpena|off | 10:22 | |
| *** jpena|off is now known as jpena | 10:29 | |
| *** dviroel_ is now known as dviroel | 11:28 | |
| *** bhagyashris is now known as bhagyashris|ruck | 11:51 | |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Temporarily stop booting nodes in inmotion iad3 https://review.opendev.org/c/openstack/project-config/+/875488 | 12:47 |
| opendevreview | Merged openstack/project-config master: Temporarily stop booting nodes in inmotion iad3 https://review.opendev.org/c/openstack/project-config/+/875488 | 13:17 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Revert "Temporarily stop booting nodes in inmotion iad3" https://review.opendev.org/c/openstack/project-config/+/875495 | 13:43 |
| *** haleyb_ is now known as haleyb | 14:36 | |
| *** dansmith_ is now known as dansmith | 14:44 | |
| opendevreview | Merged openstack/project-config master: Revert "Temporarily stop booting nodes in inmotion iad3" https://review.opendev.org/c/openstack/project-config/+/875495 | 16:42 |
| *** jpena is now known as jpena|off | 17:42 | |
| dansmith | gerrit appears to be down.. getting a timeout for the last couple of minutes | 18:50 |
| fungi | dansmith: yep, clarkb is restarting it now in an attempt to resolve some strange overload condition, details in #opendev | 18:51 |
| dansmith | and it's back | 18:51 |
| dansmith | ack | 18:51 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org experienced severe performance degradation between 17:50 and 19:45 due to excessive API query activity; the addresses involved are now blocked but any changes missing job results from that timeframe should be rechecked | 19:55 | |
| JayF | I'm writing a blogpost on the enormity of OpenStack's CI system and how awesome it is; do we have any impressive metrics on how many total CI jobs we run over (time period) | 23:15 |
| JayF | I'm also happy to share a commentable google doc to the draft post if anyone would like to review | 23:15 |
| clarkb | JayF: I ususally start at https://grafana.opendev.org for pulling that sort of thing up | 23:16 |
| clarkb | there are graphs for zuul jobs and nodes used and so on there | 23:16 |
| * JayF is embarassingly bad at grafana | 23:16 | |
| JayF | but I'll take a stab :) | 23:16 |
| clarkb | I am too. Half the time I download the csv and then process locally instead of having their functions do it | 23:16 |
| clarkb | but thats where I go to look either way | 23:17 |
| JayF | yeah; I'll see if I can slice and dice it. I was hoping someone knew something off the top of their head (e.g. "we run over 10k vms a day in support of CI" or similar) | 23:17 |
| JayF | for now it gets a placeholder as I don't wanna leave writing-mode to go into yell-at-grafana mode :D | 23:17 |
| ade_lee_ | clarkb, fungi hey -- I'm trying to diagnose a failure to start up mysql when fips is enabled on ubuntu. | 23:21 |
| ade_lee_ | https://zuul.opendev.org/t/openstack/build/05be19edd1f041b18fef3dd88f1ad671/log/controller/logs/mysql/error_log.txt | 23:22 |
| ade_lee_ | clarkb, fungi there aren't a lot of logs here though, in fact, I can't even see the mysql.cnf | 23:22 |
| ade_lee_ | clarkb, fungi any way I can get a node held so that I can try to make it work? | 23:23 |
| clarkb | ade_lee_: yes we can do that, but also not having logs to debug failure to start a required service is often a indication the log collection could use updates | 23:23 |
| clarkb | following up with collection of logs that would've exposed the problem etc | 23:24 |
| ade_lee_ | clarkb, ack - though I'm not sure if log collection is failing because of some fips thing too :/ | 23:24 |
| ade_lee_ | clarkb, so yeah - if we hold the node, we can debug and then run whatever would have collected the logs to figure out why that is failing | 23:26 |
| clarkb | ade_lee_: ok if you recheck that change the next run of tempest-all-fips-focal that fails should be held | 23:26 |
| ade_lee_ | clarkb, thanks! | 23:26 |
| clarkb | I limited the hold to that change and job | 23:27 |
| ade_lee_ | clarkb, thanks - I think I've got to wait till it completes. I tried "recheck experimental" and nothing happened | 23:28 |
| JayF | clarkb: do I need some kind of special access to do direct querying? AFAICT I can only view existing dashboards, with no editing to make a custom dash | 23:29 |
| clarkb | ade_lee_: oh its already queued up. I can eject it and then we can recheck again | 23:29 |
| clarkb | JayF: no the databases are fronted by graphite at https://graphite.opendev.org | 23:29 |
| ade_lee_ | clarkb, cool - let me know when and I'll recheck | 23:30 |
| JayF | clarkb: oh good, this is more what I'm familiar with, ty | 23:30 |
| clarkb | ade_lee_: it dequeued | 23:31 |
| ade_lee_ | clarkb, thanks - its rerunning now | 23:32 |
| clarkb | ade_lee_: note its just `check experimental` iirc. Addin the `re` prefix causes it to also enqueue to check as a normal recheck | 23:33 |
| ade_lee_ | ok - thats what I ended up doing - "check experimental" | 23:33 |
| JayF | clarkb: sanity check -> https://graphite.opendev.org/?width=1107&height=874&target=summarize(sumSeries(group(stats_counts.zuul.tenant.openstack.pipeline.*.all_jobs))%2C%20%221d%22) indicates that I can say opendev runs over 10k CI jobs/day for OpenStack, yeah? | 23:36 |
| clarkb | JayF: yes I think so. WIth the small caveat that the openstack tenant in zuul still runs some things for opendev and starlingx etc | 23:37 |
| clarkb | but those are a tiny percentage of total time/resource use every time I look at those numbers | 23:38 |
| clarkb | openstack is regularly 95-98% of consumption so 95% * 12k should still be more than 10k | 23:38 |
| JayF | yeah I figured 10k made it safe | 23:38 |
| JayF | and it's a random blogpost; if people are citing it in a paper or something they'll be the ones doing it wrong :D | 23:39 |
| clarkb | I was going to message ade_lee the held node info but seems they just disconnected. | 23:53 |
| clarkb | ade_lee if you end up looking at logs its root@158.69.68.87 | 23:53 |
| clarkb | on our side thats np0033283253 in ovh bhs1 | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!