fungi | i've been noticing the hyperkitty/postorius/mailman ui is slowish lately... we're running on an 8gb flavor instance with 2gb swap, and all of the swap is showing used along with 95% of ram | 13:20 |
---|---|---|
fungi | the largest single consumer is mariadb, but most of the memory is consumed by uwsgi and python processes | 13:22 |
fungi | i'm going to do a quick apache restart to see how much that frees up, suspect crawlers have forced it to load a lot of content into process memory over time | 13:23 |
fungi | really only freed up about half a gig, so that's not it | 13:29 |
fungi | must be the python process memory for all the mailman daemons | 13:30 |
fungi | i'll do a quick mailman-web container restart next and see if it's django | 13:31 |
fungi | okay, that freed most of the swap and about 2.5gb of ram, so was at least half of the use | 13:37 |
fungi | i guess most of the rest is split between the mailman-core python processes and mariadb | 13:38 |
fungi | postorius ui is responding to me again | 13:41 |
fungi | and way more snappy now that it's not paging everything to/from disk each time it has to query the db | 13:42 |
fungi | looks like it's been about 6 months since the last restart of all the containers | 13:43 |
fungi | though the server's been up without a reboot for 901 days now | 13:44 |
*** dmellado62 is now known as dmellado6 | 14:13 | |
NicolaiRuckel | I'm working on a patch for Nova and I was wondering if there is a way to get a more readable output from the tests. If you just run `tox`, I get almost 40000 lines of output with the failed test cases and their stack traces somewhere in between. At the moment I just grep for "FAILED" but I was wondering if there's a smarter way. | 15:23 |
fungi | NicolaiRuckel: in our ci jobs we run subunit2html on the results to generate the "Unit Test Report" like you see linked at https://zuul.opendev.org/t/openstack/build/6eec7ca492054f18957763c8d7983ed1/artifacts | 15:26 |
stephenfin | you can also activate the virtualenv for the test environment you're using (e.g. `source .tox/py3/bin/activate`) and then use the 'stestr history show' command | 15:27 |
fungi | also things like `stestr run failing` i think, to just rerun previously failed tests? | 15:28 |
stephenfin | (You'll probably want the '--suppress-attachments' argument to hide all of the output) | 15:28 |
fungi | if you go the subunit2html route, here's a good example report with a failing test: https://d8e1d98dffdc278e1978-d367ceb2d3289c9a141d614bfa7b9335.ssl.cf1.rackcdn.com/openstack/3416a671afbc4b3ca0da99f0a9b8dc9e/testr_results.html | 15:28 |
stephenfin | Indeed. I often do e.g. 'tox -e py310 -- --failing', since most projects are set up to pass any additional positional arguments straight through to stestr | 15:28 |
NicolaiRuckel | Yeah, something like subunit2html is exactly what I was looking for. I don't understand how to get the subunit_stream though. | 15:52 |
NicolaiRuckel | `stestr run failing` looks super helpful too! | 15:52 |
NicolaiRuckel | thanks | 15:52 |
fungi | NicolaiRuckel: our jobs appear to execute `stestr last --subunit >> foo.subunit` after tox is done and then run subunit2html on that, but it could probably be done as a pipe instead | 15:56 |
Clark[m] | As a side note I personally think it is a bug that projects allow so much output in their tests. I like what zuul does which is to suppress out out unless there is a failure by default | 16:00 |
NicolaiRuckel | so I'd run tox, activate the venv and then run your commands? | 16:02 |
fungi | don't even have to activate, you could just `.tox/py312/bin/stestr last --subunit` for example | 16:02 |
fungi | python script entrypoints in venvs can typically be executed directly without needing to explicitly activate the venv | 16:03 |
NicolaiRuckel | oh, I didn't know that | 16:04 |
NicolaiRuckel | thanks | 16:04 |
fungi | i install almost all my python utilities into separate venvs and basically never activate a venv for them | 16:05 |
fungi | though in this case, i expect you could even get by with a completely separate stestr install, since all you're doing there is using it to read the local testrepository db and export a particular filtered subunit stream | 16:07 |
NicolaiRuckel | so something like `tox -e py3 && .tox/py3/bin/stestr last --subunit >> foo.subunit && .tox/py3/bin/subunit2html foo.subunit`? | 16:11 |
fungi | or possibly ... .tox/py3/bin/stestr last --subunit | .tox/py3/bin/subunit2html (assuming subunit2html is installed in that venv anyway) | 16:13 |
NicolaiRuckel | ah, right | 16:14 |
NicolaiRuckel | I'll give that a try | 16:14 |
fungi | again, a separate subunit2html install somewhere should work just as well, it's merely a utility to parse subunit streams and export html | 16:14 |
NicolaiRuckel | a separate install is probably a better idea, right | 16:15 |
fungi | note that it's os-testr which supplies the subunit2html utility, and you can probably install it from your linux distro even. it's packaged in debian and ubuntu at least (as python3-os-testr), but other distros likely carry a copy as well | 16:20 |
clarkb | no new cert expiration warning eamils so I think getting the base job happy was all that we needed there | 16:33 |
NicolaiRuckel | the pipe doesn't work for some reason but the other way works | 16:34 |
NicolaiRuckel | thanks again | 16:34 |
fungi | NicolaiRuckel: possible you need to pass "-" as the filename for it to read, but i haven't looked at its file handling to see if it treats that as an alias for stdin | 16:38 |
NicolaiRuckel | I'll give that a try after my current test run. | 16:39 |
opendevreview | Clark Boylan proposed opendev/infra-specs master: Add a less formal todo list link to specs https://review.opendev.org/c/opendev/infra-specs/+/956183 | 16:52 |
clarkb | infra-root ^ I think that is a straightforward chagne and I've moved the todo list to that new location | 16:52 |
clarkb | fungi: https://review.opendev.org/c/opendev/zone-opendev.org/+/956121 this should be safe to merge now right? its just teh system-config change we need to wait on? | 16:53 |
fungi | clarkb: presumably yes, i guess the concern is that if we merge the inventory addition then it will start up duplicate irc bot daemons? | 16:54 |
clarkb | fungi: yes | 16:54 |
clarkb | and start filling disk with logs where we don't want them. But mostly the ircbot identification fight that will ensue and potentially the matrix equivalent (though I think matrix will allow you to auth from multiple locations instead) | 16:54 |
clarkb | I +2'd both changes but didn't approve either. I think we can approve the dns change through if you want me to do that now | 16:55 |
fungi | yes, matrix does, i use that to my advantage | 16:55 |
fungi | thanks! | 16:55 |
clarkb | I've started putting pre ptg topic ideas on https://etherpad.opendev.org/p/opendev-preptg-october-2025 feel free to start adding your ideas too | 17:27 |
corvus | clarkb: i revised https://review.opendev.org/c/zuul/zuul/+/956119 Improve handling of unassigned ready nodes | 17:35 |
corvus | i think the periodic jobs weren't enough to dislodge those ready nodes, so i think our way out may actually be merging that and letting it work | 17:35 |
clarkb | corvus: ack I'll rereview | 17:37 |
* corvus tries to figure out if https://opendev.org/openstack/project-config/src/branch/master/nodepool/scripts/do-not-delete.txt is still a thing | 17:53 | |
clarkb | corvus: based ont he list of things in that file that were using that path I believe we can delete that | 17:54 |
clarkb | the amount of puppet that actually puppets is quite small and nothing that interacts with the CI systems anymore iirc | 17:54 |
corvus | yeah, i think so too, i'm just trying to affirm that | 17:55 |
corvus | https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2020-03-03.log.html#t2020-03-03T03:48:39 doesn't give much more of a clue about what places it might be embedded | 17:58 |
corvus | https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2020-03-03.log.html#t2020-03-03T06:14:50 suggests maybe just the "builders" | 17:59 |
corvus | nodepool.scripts doesn't show up in system-config. so i think i'm happy with "delete it" | 18:00 |
clarkb | corvus: ya I think its ok to delete | 18:00 |
corvus | it's mostly just that there is this file that says "things depend on this" but we didn't leave any record of what things, so it's hard to be positive we got all the dependencies (and apparently, we were surprised once before) | 18:01 |
opendevreview | James E. Blair proposed openstack/project-config master: Remove nodepool configuration/elements https://review.opendev.org/c/openstack/project-config/+/956184 | 18:02 |
clarkb | anyone else want to weigh in on https://review.opendev.org/c/opendev/infra-specs/+/956183 to give our informal todo list document a more permanent home and linked location? | 18:22 |
fungi | i don't think we need to apply our usual spec approval process for that change | 18:24 |
clarkb | agreed I've approved it now | 18:42 |
opendevreview | Merged opendev/infra-specs master: Add a less formal todo list link to specs https://review.opendev.org/c/opendev/infra-specs/+/956183 | 18:49 |
clarkb | that has deployed and is now available for perusal | 18:52 |
clarkb | I'm going to make the service coordinator election details official via email after lunch today (realized I meant to do that yesterday then didn't) | 18:55 |
clarkb | last call for input on that | 18:55 |
fungi | infra-root: we can now start tearing down refstack, i've un-wip'd https://review.opendev.org/947856 | 19:14 |
clarkb | fungi: looks like it needs a rebase to address merge conflicts | 19:51 |
fungi | yeah, it's been several months, so unsurprising. will get to that in a bit | 19:54 |
fungi | clarkb: as for eavesdrop02, should i just go ahead and update the apache proxy to point to it in the same change as the inventory addition? | 19:55 |
clarkb | fungi: ya the proxy updates in the same chagne seems reasonable | 20:00 |
clarkb | the deployment for inventory updates always runs all the jobs anyway so you may as well include that (its not like you saev time by excluding it) | 20:00 |
fungi | right, thinking back through the swap logistics, there's not a time where we'd want the proxy pointing at the old server while the bots and logs are on the new one | 20:15 |
clarkb | service coordinator election email has been sent | 20:17 |
opendevreview | Jeremy Stanley proposed opendev/system-config master: Replace eavesdrop01 with eavesdrop02 https://review.opendev.org/c/opendev/system-config/+/956122 | 21:43 |
opendevreview | Jeremy Stanley proposed opendev/system-config master: Wind down refstack.openstack.org https://review.opendev.org/c/opendev/system-config/+/947856 | 21:46 |
clarkb | fungi: one small but important detail on https://review.opendev.org/c/opendev/system-config/+/956122 | 21:56 |
fungi | d'oh! good catch | 21:59 |
opendevreview | Jeremy Stanley proposed opendev/system-config master: Replace eavesdrop01 with eavesdrop02 https://review.opendev.org/c/opendev/system-config/+/956122 | 22:00 |
fungi | pip 25.2 just dropped, eyes open for possible behavior changes as usual... https://discuss.python.org/t/100716 | 22:09 |
opendevreview | Jeremy Stanley proposed opendev/system-config master: Replace eavesdrop01 with eavesdrop02 https://review.opendev.org/c/opendev/system-config/+/956122 | 22:36 |
clarkb | fungi: you also need the le handler for that 99 name I think | 22:36 |
clarkb | https://review.opendev.org/c/opendev/system-config/+/956122/4/playbooks/roles/letsencrypt-create-certs/handlers/main.yaml this bit needs an additional handler | 22:37 |
fungi | oh, looking | 22:37 |
clarkb | I was just looking into the afiluer as you pushed the update | 22:37 |
fungi | ah, is what i added what you meant then? | 22:38 |
clarkb | fungi: what you added is half of what is needed. The other half goes in the hanlders file as you need a handler that matchse what you just added | 22:38 |
clarkb | the hanlder names are keyed off of the name on line 2 in https://review.opendev.org/c/opendev/system-config/+/956122/3..4/playbooks/zuul/templates/host_vars/eavesdrop99.opendev.org.yaml.j2#2 | 22:39 |
opendevreview | Jeremy Stanley proposed opendev/system-config master: Replace eavesdrop01 with eavesdrop02 https://review.opendev.org/c/opendev/system-config/+/956122 | 22:41 |
fungi | looks like we mix the production and test handlers there | 22:42 |
clarkb | there is a block under line 216 for gate testing hosts but I don't think it matters mcuh. Its basically a flattened dictionairy and order doesn't really matter | 22:43 |
opendevreview | Jeremy Stanley proposed opendev/system-config master: Replace eavesdrop01 with eavesdrop02 https://review.opendev.org/c/opendev/system-config/+/956122 | 22:47 |
fungi | moved it now | 22:47 |
corvus | i'm going to restart the launchers to get the fix that will presumably unwedge sjc3 | 23:32 |
corvus | i'm also going to restart schedulers/web so that we can see those ready nodes | 23:33 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!