Monday, 2025-01-27

fungiover the weekend i noticed another issue with mailman bounce processing on openstack-discuss14:26
fungisome muas (outlook in particular, i think?) emit over-long headers. exim accepts those messages (as does mailman, which turns them around unaltered), but then the exim remote smtp router rejects them with a "message has lines too long for transport" permanent error which is counting as a bounce for every recipient14:32
fungibased on package changelogs, it looks like this feature got implemented in exim 4.95 (jammy), and then workarounds were added to increase it from the default 998 bytes by a few orders of magnitude in ubuntu's 4.96-5 packages (so noble, not backported)14:35
fungii think we should duplicate the message_linelength_limit = 1G from noble into our configs, but am not quite clear how to go about that in playbooks/roles/base/exim/templates/exim4.conf.j2 since adding it on focal and earlier will result in a configuration parsing error14:37
fungii guess i can use an ansible platform conditional in the jinja?14:37
fungiaha, looks like i can, something along the same lines as https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/master-nameserver/templates/named.conf.j2#L13-L1514:42
opendevreviewJeremy Stanley proposed opendev/system-config master: Increase message_linelength_limit to 1G  https://review.opendev.org/c/opendev/system-config/+/94024814:59
fungithere's a stab at it ^14:59
corvusi'm finishing up the zuul restart manually15:07
corvusalso i'm going to re-restart the scheduler on zuul01 since zuul02 jumped ahead a change15:08
fungithanks corvus! i was worried it would come down to that15:09
fungibut didn't touch it because i wasn't sure if there might have been a cleaner way to resume the in-progress restart 15:10
corvusproabably not; problably just re-running the restart playbook would be the most automatic thing we could have done.  but this was near the end, so it's faster to do manually.  :)15:13
corvusthere's definitely room for improvement, but https://zuul.opendev.org/t/opendev/image/debian-bullseye exists now and we can see what zuul sees.15:20
fungiyay!15:22
fungihttps://zuul.opendev.org/t/openstack/build/058df70b33f04569a32d9f43ddd2250f is a new failure for me (timeout in install-kubectl : Download openshift client tarball)15:24
fungii'm able to fetch https://github.com/openshift/origin/releases/download/v3.11.0/openshift-origin-client-tools-v3.11.0-0cbc58b-linux-64bit.tar.gz here at home, so probably something temporary/random15:30
corvus++15:31
fungialso, looks like that's basically just cruft for the experimental kubernetes/rook gitea pod we're not using?15:31
corvusyeah i was thinking that's probably the only thing using it; could probably drop it if it becomes a problem.  and even if we restart that, we might not need to get that client that way anymore.15:35
Clark[m]Ya I think we can safely drop the fetch off that tool on bridge15:40
Clark[m]I'm having a bit of a slow start today, but I'd like to finish up the paste cleanup that stalled out Friday. Any concerns with deleting paste01 and it's volume at this point (reminder paste02 didn't get a volume)15:41
Clark[m]Then following up on bindep and PBR and pyproject.toml stuff as well as doing more grafana debugging15:41
clarkbmy monday morning system updates include a new bash package16:18
clarkband amd cpu ucode updates. This is a fun one16:19
fungiyeah, i guess that's the one that asus leaked in the rog bios downloads?16:20
clarkboh ya could be16:20
clarkbin any case I'll be rebooting shortly for that then finding breakfast then I can start digging in16:21
clarkblet me know if you think it is safe to proceed with paste01 and its volume cleanup despite the lack of volume on paste02 (frickler indicated this seemed fine on friday but I dont' think I heard from anyone else)16:22
fungiwhat were we using the volume for on the old server? database?16:22
clarkbyes the database16:22
clarkbit is a 75GB volume and we're only using 4.4GB of it iirc.16:23
clarkbthe relatively small size of the db is why paste02 is happy at the moment there is a bit of headroom on the server's / disk16:23
fungirootfs on new paste isn't even a quarter used, seems fine to just stick it there, i agree16:23
fungiwe've got backups, and also /opt ephemeral if we want to do that16:24
fungiso plenty of safety/options16:24
clarkbcool once I feel awake I'll actually proceed with deleting paste01 and its volume then16:24
clarkbfungi: and then for bindep/pbr/pyproject.toml stuff I think we're still waiting on a pbr beta release?16:25
clarkbI guess we can proceed with landing pyproject.toml in bindep since the release for bindep was made on Friday16:26
clarkbthen do followups that use teh beta release when available16:26
fungiyeah, on friday the openstack release team said something about approving the pbr beta release request today16:27
opendevreviewJeremy Stanley proposed opendev/system-config master: Increase message_linelength_limit to 1G  https://review.opendev.org/c/opendev/system-config/+/94024816:36
clarkbhttps://review.opendev.org/c/opendev/system-config/+/940166/ and parent are two improvements to grafana deployments to make testing more robust (the linked change will impact production too by setting up journald logging for the container)16:41
clarkbthose two should be safe to land at any time. Its the followup to upgrade that is not currently safe until we figure out the CORS stuff16:41
opendevreviewMerged opendev/zuul-jobs master: Compress raw/vhd images  https://review.opendev.org/c/opendev/zuul-jobs/+/93597016:52
clarkbok I have tea and have heard no objections to deleting paste01. I've quickly double checked and was just able to open a random paste from July. I'll proceed with server deletion and volume deletion now17:02
fungisounds good, thanks!17:04
clarkb#status log Deleted paste01.opendev.org (d2312252-2318-4b8e-a255-b8cb8c2069b6) and its associated data volume (e4b23816-b205-47fe-8372-1ce4f7650b0a)17:07
clarkbit should be safe to proceed with https://review.opendev.org/c/opendev/system-config/+/939128 now. That may trigger quite a few infra-prod jobs though as it modified inventory17:08
opendevstatusclarkb: finished logging17:08
clarkb(that requires paste01 backups on the smaller backup server. It is safe now because now that the server is gone it won't create new backups)17:08
corvusi logged into the zuul web ui and pressed the "Build Image" button on https://zuul.opendev.org/t/opendev/image/debian-bullseye -- that should build and upload a compressed raw image.17:09
corvuswe've already seen that in action in https://zuul.opendev.org/t/opendev/build/33a460e52e6b4c44ad524f97944dd7cc17:09
corvusthe build took 50m and the upload of both artifacts (raw and qcow) took 20m17:09
corvusthis next test will have the launcher download the compressed raw image, uncompress it, and upload it to vexxhost (we hope)17:10
opendevreviewClark Boylan proposed opendev/zone-opendev.org master: Remove paste01 dns records  https://review.opendev.org/c/opendev/zone-opendev.org/+/94025517:10
fungiclarkb: apparently the newline omitted by %- was necessary? https://zuul.opendev.org/t/openstack/build/89b96dcee0d241c1b063b9e6d0fa757617:10
fungithe job passed when it was just %17:11
clarkbhuh or maybe we need a trailing - instead of a leading -?17:11
clarkbthe leading - might remove the leading whitespace too? Its not super important to have I just thought avoiding blank lines would be a good idea17:11
clarkblanding 940255 and 939128 should largely conclude the paste replacement. I'm thinking doing one or two more services might be a good idea before doing review. Maybe codesearch and grafana or something along those lines17:13
clarkbcorvus: this largely exercises the use of ui and api and compression?17:14
fungiclarkb: looks like the %- removes the preceding newline (and not the indentation?): https://zuul.opendev.org/t/openstack/build/89b96dcee0d241c1b063b9e6d0fa7576/log/lists99.opendev.org/syslog.txt#141717:14
clarkboh interesting so ya we probably need the trailing - not the leading -17:14
clarkb-%} instead of {%-17:15
fungibut on the if not the endif, right?17:15
clarkbyes17:15
opendevreviewJeremy Stanley proposed opendev/system-config master: Increase message_linelength_limit to 1G  https://review.opendev.org/c/opendev/system-config/+/94024817:16
corvusclarkb: yes -- specifically, does the zuul-launcher processing of a compressed raw image work :)17:16
clarkbactually replacing some zuul servers might be a better exercise since we're a bit harder on zuul (from a usage and management perspective)17:17
clarkbI won't dive into that today (enough other stuff) but I'll think about that a bit17:17
opendevreviewMerged opendev/system-config master: Take screenshots of all grafana dashboards  https://review.opendev.org/c/opendev/system-config/+/94015417:35
clarkblooking at eavesdrop I see that ptg.opendev.org has a new cert from january 16th. We're still getting warnings about the cert expiring in 3 weeks though. Probably another case where apache isn't cycling worker processes quickly enough?17:42
opendevreviewMerged opendev/system-config master: Log grafana to /var/log/containers with journald/syslog  https://review.opendev.org/c/opendev/system-config/+/94016617:45
fungithat's what i've been expecting, yes17:45
clarkbI've just checked grafana and the container was restarted and we have a log file in /var/log/containers as expected17:47
clarkband the web ui appaers to be up and responding to me17:48
clarkbfungi: are you pushing an update to bindep to pull in the beta release of pbr?17:53
fungii will once we see the beta on pypi, yes17:54
clarkbgreatthanks! I need to pull up the latest state of the bindep stack anyway so will start catching up now17:54
fungibasically waiting for it to appear and then i've got the dnm change for bindep ready to push up on top of that stack17:58
clarkbfungi: the end of that stack is apparently in merge conflict17:59
clarkbso you might need to stack it on a change not at the end ofthe satck or rebase the whole thing or something17:59
fungiwill do. git didn't see any conflicts that needed resolving, but i guess gerrit is less flexible there18:00
opendevreviewJeremy Stanley proposed opendev/bindep master: Use PBR's pyproject.toml build-backend support  https://review.opendev.org/c/opendev/bindep/+/81674118:00
opendevreviewJeremy Stanley proposed opendev/bindep master: Evacuate most metadata out of setup.cfg  https://review.opendev.org/c/opendev/bindep/+/93852018:00
opendevreviewJeremy Stanley proposed opendev/bindep master: Drop support for Python 3.6  https://review.opendev.org/c/opendev/bindep/+/93856818:00
opendevreviewJeremy Stanley proposed opendev/bindep master: Drop requirements.txt  https://review.opendev.org/c/opendev/bindep/+/93857018:00
opendevreviewJeremy Stanley proposed opendev/bindep master: Drop tox.ini  https://review.opendev.org/c/opendev/bindep/+/94021918:01
opendevreviewJeremy Stanley proposed opendev/bindep master: DNM: Test PBR beta release  https://review.opendev.org/c/opendev/bindep/+/94025818:01
opendevreviewJeremy Stanley proposed opendev/bindep master: DNM: Test PBR beta release  https://review.opendev.org/c/opendev/bindep/+/94025818:03
fungiforgot to increase it in the dependencies list too18:03
fungijust for thoroughness18:03
clarkbya that mgiht be necessary for it to be considered as valid because it is a beta release?18:04
fungicorrect18:04
clarkbnot sure how thoroughly those different configs get checked when installing the deps18:04
fungione is for package build time, one is for runtime18:04
clarkbmakes sense18:04
fungiso we could end up with bindep built with the pbr beta but then run with the prior full release (which should still work fine of course)18:05
fungibut this way we get more thorough testing in both aspects18:06
fungialso when the constraints update proposal gets pushed, we should have some additional test feedback hopefully18:06
clarkbfungi: we should wip that change?18:06
fungii'm planning to do so the moment it exists18:07
clarkbthanks18:07
fungijust waiting for the propose-update-constraints job to start (it's still queued)18:07
fungiand stephenfin already has a full release of pbr proposed too, just wip for the moment, but once we're comfortable with these exercises we can go ahead with that too18:08
fungisome other what-if changes i want to add to that bindep stack are getting rid of test-requirements.txt and docs/requirements.txt (just distributing the appropriate test/docs dependencies into the necessary parts of the noxfile instead)18:10
clarkbbefore a full release happens we should ideally land the docs update (if that hasn't happend yet) and consider dropping the python3.12 restriction on setuptools as a requirement18:10
clarkbthough the setuptools change can also happe in a followup release18:10
clarkbif we want to tread slowly here18:11
fungiyeah, also stephenfin has a change up to add 3.13 testing (which seems to work once we ironed out the kink with 3.13.1t cpython from pyenv)18:11
clarkbfungi: the change to use the beta pbr release is after the python3.6 removal. DO we want to test it with python3.6 just to see if anything else other than setuptools support for editable wheels is a problem?18:19
fungisure, i can shuffle it up the stack once we have preliminary results18:19
fungihttps://review.opendev.org/940260 exists and is wip now18:24
fungihttps://zuul.opendev.org/t/opendev/buildset/86b76fa186a4487ebf62ddff9024d9df has the results from 940258,2 and i'll dig into the logs from them to double-check that we really did use the beta18:26
opendevreviewJeremy Stanley proposed opendev/bindep master: DNM: Test PBR beta release  https://review.opendev.org/c/opendev/bindep/+/94025818:26
fungiand that ^ checks it with 3.618:26
fungihttps://zuul.opendev.org/t/opendev/build/f005295b1c714381abe55c22efcb1341/console seems to have pulled "pbr-6.1.1.0b1-py2.py3-none-any.whl"18:30
fungi"Successfully installed [...] pbr-6.1.1.0b1"18:31
fungiunfortunately it doesn't log the version used for the build backend18:34
fungii'll test it locally and see if i can coax more details out18:41
fungithe 0.0.0 version in https://zuul.opendev.org/t/opendev/build/7490e13cb15e4b7c9fe3a6ded3900ca8/console#2/0/0/ubuntu-noble is mildly worrisome, wondering if that's because it still relies on directly invoking setup.py18:44
clarkbhrm ya that is not ideal18:47
fungiyeah, if i use `python3 setup.py sdist bdist_wheel` it fails to set the bindep version and ends up falling back on 0.0.018:47
fungiif i use https://pypi.org/p/build then i get the right version18:48
clarkbsetup.py should be effectively doing the same thing as the pep 440 path though since its all setuptools under the ood. I wonder where the disconnect is18:48
clarkbI guess we have the choice of updating openstack release builds to use `build` and just move on or we can try and debug and fix it18:49
clarkbhaving at least a small understanding of why it is happening is probably a good idea18:49
clarkbfungi: does python3 setup.py sdist bdist_wheel work with the current full release of pbr on python3.12?18:50
fungiyeah, if i roll back to bindep master branch, no pyproject.toml file, then `python3 setup.py sdist bdist_wheel` gets the correct version string from git when using the same python/setuptools versions18:50
fungithough i am testing with python 3.13. i'll repeat with 3.1218:50
clarkbfungi: I wonder if it has anything to do with the move of setup.cfg content18:51
clarkbsince moving all of that into pyproject.toml is going to be modern build tooling specific maybe?18:51
clarkbits possible we need to keep that stuff in setup.cfg for broad compatibiltiy?18:51
fungino, it's the mere existence of the pyproject.toml i think18:52
fungijust tested with python3.12/setuptools-75.8.0 on a checkout of 816741 and `python3 setup.py sdist bdist_wheel` builds bindep 0.0.018:53
fungicorrection, presence of the build-system section in pyproject.toml, if i simply create an empty pyproject.toml on a master branch checkout it's still working okay18:55
clarkbthat implies setuptools is changing its behavior based on pyproject.toml build-system even when it is invoked in the legacy manner. That feels like a bug but probably can't say until we know more18:56
fungii suspect this has always been the case, and if we want pbr.build to work as a pyproject.toml build-system.build-backend we need to use a pep-417 compliant build frontend like the build tool18:56
fungier, pep-51718:57
clarkbya that wouldn't surprise me and we might consider that a bug but upstream will say something like no this is used to force you to stop using the legacy sysetm or similar18:57
clarkband build should work with the pre pyproject.toml addition right? so we can simply update our build jobs and be compatibiltiy with or without pyproject.toml?18:57
fungii doubt they'd even get that detailed, and will more likely say that if you want to use pyproject.toml then stop directly invoking setup.py, with no further explanation18:58
clarkbya18:58
fungiand yes, if that build-python-release job invoked pyproject-build instead of setup.py i'm 99% sure it would generate the right version18:59
clarkbya I'm just worried about updating a global openstack job for the small number of things using pyproject.toml19:00
clarkbI guess if we want to be extra cautious we could switch behavior based on the presense (or lack) of a pyproject.toml file19:00
funginote that this isn't an openstack job, it's a job defined in zuul-jobs19:00
fungithough openstack is running it, of course19:00
clarkboh I thought this was the openstack specific job (which maybe inherits from this job)19:01
funginope, defined right in and used straight from zuul/zuul-jobs19:01
clarkbanyway seems like a good next step is updating the job to use `build` ? could be conditional based on pyproject.toml presence19:01
fungihttps://opendev.org/zuul/zuul-jobs/src/branch/master/roles/build-python-release/tasks/main.yaml#L219:02
fungiyeah, i'll work on a change for that19:02
clarkbsounds good. The kids don't have school today so we're going to try and grab brunch/lunch shortly. But I can review when I return19:03
fungihave fun!19:03
opendevreviewJeremy Stanley proposed opendev/bindep master: DNM: Test PBR beta release  https://review.opendev.org/c/opendev/bindep/+/94025819:58
opendevreviewJeremy Stanley proposed zuul/zuul-jobs master: Add ensure-pyproject-build role  https://review.opendev.org/c/zuul/zuul-jobs/+/94026720:01
corvuslooks like the vexxhost upload took about 12m and it's starting the raxflex upload now20:04
corvuswhen that's done... hopefully we'll see artifact c7b3bbdc25d0406f9ac6c27a95be3fab and upload 6e0a3ad8e88144ffa7406d7d5594b84f be deleted20:05
fungiclarkb: the latest iteration of the dnm change for bindep using the pbr beta version did pass for py36 as well. also it looks like the wip constraints bump of pbr to the beta version passed all jobs too (though i don't know how indicative that is of what to expect with a proper release)20:09
corvusit is done, but the old image wasn't automatically deleted.  that's not a huge problem in the scheme of things.20:10
fungithat's massive progress, indeed20:10
Clark[m]fungi: I'm guessing that PBR isn't pre installed so the contraint doesn't apply in most jobs? It may apply to devstack though as I think it may be preinstalled there20:16
fungiwell, also it tests runtime use where projects declare pbr as a dependency (install_requires)20:17
fungiClark[m]: the good news is that the release-openstack-python job is also just reusing the same build-python-release role from zuul-jobs, so if we fix it there we fix it everywhere20:20
fungidigging through codesearch's hits on any use of "setup.py sdist" or "setup.py bdist_wheel" turns up a lot, but nothing else critical that gets run cross-project. just some stuff locally in projects they'll need to address if they start doing build-system.build-backend in pyproject.toml20:28
opendevreviewAurelio Jargas proposed zuul/zuul-jobs master: Add ensure-uv role  https://review.opendev.org/c/zuul/zuul-jobs/+/94027120:46
clarkbok back. I'll review the zuul-jobs change now20:46
clarkbfungi: just to double check there isn't a change to update the build release job yet? Just a change to install build?20:56
clarkbfungi: also if you get a chance can you review https://review.opendev.org/c/opendev/zone-opendev.org/+/940255 and consider if we should single core approve https://review.opendev.org/c/opendev/system-config/+/939128 at this point?20:57
fungiclarkb: not yet, corvus suggested we get mordred's poetry change merged first since it touches the same role, but it's looking like it has new issues since the last time it was tested20:57
clarkboh I see in the zuul matrix room now20:58
fungiyeah, digging deeper into that, but i'm really not familiar with poetry at all since i've never touched it20:58
clarkbthats annoying that poetry doesn't respect requires python21:01
clarkbthis would be a complete non issue with pip...21:01
opendevreviewMerged opendev/zone-opendev.org master: Remove paste01 dns records  https://review.opendev.org/c/opendev/zone-opendev.org/+/94025521:01
clarkbfungi: 1.0.8 supports python3.821:02
clarkb1.1.0 supports python3.921:02
fungiyeah, but also it's actually the plugin that's erroring on a too-old python version, need to make sure it installs an old enough version of the plugin for the platform21:02
fungiit looks like poetry just wants to install the latest plugin version no matter what21:03
clarkbright poetry==1.1.0 should work on centos 9 and bullseye and 1.0.8 should work on focal21:03
fungibut will it still try to install the latest version of poetry-git-version-plugin?21:04
clarkboh sorry I meant the plugin at those versions21:04
fungiit seems like it's the `poetry self add poetry-git-version-plugin` pulling in the latest available version21:05
clarkbpoetry-git-version-plugin 1.1.0 should work with 3.9 and 1.0.8 should work with 3.821:05
clarkbhttps://python-poetry.org/docs/plugins/#with-pip this says you can install plugins with pip which should respect requires python21:05
fungimaybe `poetry self add ...` lets you specify plugin versions21:05
clarkbthat might be an easy option21:05
fungiah21:05
fungiagreed21:05
clarkbself add means install to the environment that poetry is installed to where as just add installs to the env for the project being managed by poetry21:06
clarkbso in theory if we just pip install to the venv poetry is in we can have pip figure out versions for us21:06
fungi{{ ensure_poetry_venv_path }}/bin/pip install poetry-git-version-plugin21:07
fungisomething like that i guess21:07
clarkb++21:07
opendevreviewJeremy Stanley proposed zuul/zuul-jobs master: Hook poetry into ensure-python and build-python-release  https://review.opendev.org/c/zuul/zuul-jobs/+/92309421:09
clarkband then I guess we should make pyproject-build the default and have an override to use setup.py directly and also the existing overrides for poetry?21:10
clarkbif use setup.py then setup.y; if poetry.lock present or poetry flag isn't turned off then poetry; else pyproject-build?21:11
fungiyeah, i was going to look for a "[build-system]" line in pyproject.toml, in order to keep the old behavior, but pyproject-build is supposed to also work with projects that don't have a pyproject.toml21:12
fungiso a use-the-old-stuff override projects can specify to get it back is maybe the best choice21:13
clarkbya I think so, its more forward looking. our default is future proof with an escape hatch for old stuff21:13
fungimaybe call the legacy override something like invoke_setup21:14
fungior even invoke_legacy_setup21:14
clarkb++21:15
opendevreviewClark Boylan proposed opendev/system-config master: Update grafana to 10.4.14  https://review.opendev.org/c/opendev/system-config/+/94007321:20
clarkbI'm putting a hold in place for ^21:20
fungiawesome21:21
clarkbas far as updates for the infra meeting agenda go I plan to recap the meetup content and point to our todo list, talk about converting to opendevmirror hosted container images, maybe touch on grafana stuff. Anything else to add?21:22
fungiexim line length change from earlier today could probably use some discussion, just so people are aware of yet another unexpected impact for mailman bounce processing21:29
clarkback added to my little note sheet21:31
fungithanks!21:32
opendevreviewMerged opendev/system-config master: Retire paste01 backups on the smaller backup server  https://review.opendev.org/c/opendev/system-config/+/93912821:59
opendevreviewJeremy Stanley proposed zuul/zuul-jobs master: build-python-release: pyproject-build by default  https://review.opendev.org/c/zuul/zuul-jobs/+/94027322:20
fungii'll get the dnm change build-depending on that ^ and see if it addresses the issue like i suspect it should22:21
clarkbsound sgood22:22
opendevreviewJeremy Stanley proposed opendev/bindep master: DNM: Test PBR beta release  https://review.opendev.org/c/opendev/bindep/+/94025822:23
clarkbfungi: will build emit the wheel and sdist files to the same location as setup.py?22:26
clarkbthat might be the other thing to check and address if not22:26
fungiyes, they still end up in dist/22:26
clarkbnice22:26
fungiit's really up to the backend i think22:26
funginot positive, but i suspect that's why it doesn't change when using setuptools directly vs using setuptools as a build backend22:27
fungiregardless, the build-python-release job should catch it if not22:28
clarkbmakes sense22:31
fungiokay, https://zuul.opendev.org/t/opendev/build/21e81d0f22bb4e86a07b57e24d5dd4ff is going to be challenging to troubleshoot22:33
fungiah, no, the dependency is borked, probably not even valid yaml or something22:34
opendevreviewJeremy Stanley proposed zuul/zuul-jobs master: build-python-release: pyproject-build by default  https://review.opendev.org/c/zuul/zuul-jobs/+/94027322:35
funginow it's looking somewhat better22:38
clarkbfungi: I tried to capture the exim bit in the agenda's wiki page if you have a chance to check it22:38
fungilgtm, thanks!22:39
fungiso this was without the depends-on: https://zuul.opendev.org/t/opendev/build/b7aa421f55fb444eb64c9bb8578fa582/log/job-output.txt#43122:42
fungiand this was with the depends-on: https://zuul.opendev.org/t/opendev/build/17f555729d56402da9906a86317b31c9/log/job-output.txt#166222:42
clarkbheld grafana is here: https://217.182.143.14/ and it seems to be working for me actually22:43
clarkbthere are warnings about deprecated panels that require angularbut those are just warnings22:43
clarkbI'll have to open every dashboard tosee if any of them fail22:43
fungimmm, one thing i'm noticing with the dnm change is that we're not getting bindep/tests included in the wheel, which i suspect is a problem with setuptools auto-discovery22:45
clarkbok the dib status dashboard appears to fail22:45
fungii'll test locally and see if i can bisect the stack to the point where they start showing up22:45
clarkbif anyone else looks that is probably a good place to start22:45
fungithe bindep/tests files that is22:45
clarkbfungi: I wonder if we're going to do src/bindep in our future22:45
funginah, it works later in the stack22:46
fungipretty sure22:46
clarkbthere is an __init__.py in the dir so not something silly like that file missing22:46
fungiright, it's directories without an __init_.py where setuptools by default doesn't find and include their contents22:47
fungiyeah, if i move to the end of the stack the wheel has bindep/tests/* included22:50
fungioh, wait, i was comparing them backwards. with the depends-on we get the expected wheel contents. without the depends-on not only is the version empty but the wheel is missing all the test scripts/fixtures22:55
clarkbfor grafana the problems appear to be related to the use of query in dashboards. Not all dashboards use queries those are fine22:55
clarkband the firefox trace shows a cors missing allowed origin error22:55
clarkbhttps://community.grafana.com/t/problem-setting-up-graphite-data-source/5558 indiciates that one solution for that may be to have grafana proxy the graphite request22:56
clarkbso now I'm trying to figure out have grafyaml configure sthe graphite datasource. Maybe we can switch that to proxy and everything will work22:56
opendevreviewClark Boylan proposed openstack/project-config master: Proxy Grafyaml requests to Graphite  https://review.opendev.org/c/openstack/project-config/+/94027623:02
opendevreviewClark Boylan proposed opendev/system-config master: Update grafana to 10.4.14  https://review.opendev.org/c/opendev/system-config/+/94007323:04
clarkbmaybe this will fix it for us23:04
clarkbya I think that may have fixed it23:36

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!