| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 05:32 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 05:57 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 06:14 |
| tkajinam | hmm. review.opendev.org responds too slowly to me. I've seen that ping to review.opendev.org (review03 actually) is not very stable. | 06:22 |
| tkajinam | the problem might be specific to my env, though | 06:22 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 06:50 |
| frickler | tkajinam: seems fine to me, so maybe some network issue? maybe send a traceroute and let me know (dm) your IP so I can check from the server? | 06:50 |
| tkajinam | frickler, thx but it works better now. I'll let you know in case something goes wrong again | 06:56 |
| tkajinam | probably the issue is specific to my local network | 06:56 |
| clarkb | mnasiadka: the current dib functests errors appear related to mounting the extracted image fs for the centos element. Why did we switch from the centos-minimal element? centos-minimal doesn't extract the fs from an upstream image and boot straps it from scratch so may be a better starting point (but having both checked by functests eventually does seem reasonable) | 15:37 |
| clarkb | msotly just wodnering if there was some specific issue with centos-minimal we were trying to work around there | 15:38 |
| mnasiadka | clarkb: I don’t think there’s an issue, I copy pasted centos9 (I think) - if you want to update it - feel free | 15:39 |
| mnasiadka | It’s friday after 5pm, not going to have a look earlier than monday :) | 15:39 |
| clarkb | enjoy your weekend. Its a holiday weekend here too | 15:39 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 15:40 |
| clarkb | for the failed mount I want to say at one point in the past the centos (or maybe fedora?) images used a filesystem with extremely new faetures enabled and the build nodes were not able to support it. That functest job uses bookworm which isn't ancient but also isn't brand new | 15:49 |
| clarkb | it might be worth trying noble to see if that helps and/or inspecting the image filesystem for problems like that | 15:49 |
| jrosser | has anyone seen an error like this before with git review? https://paste.opendev.org/show/boSDJqvjNEUlygqQTKKC/ | 15:56 |
| jrosser | i used rebase -i to squash the two commits in the relation chain here together https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/950697 | 15:57 |
| jrosser | and am struggling to git-review the result | 15:57 |
| clarkb | jrosser: yes one sec | 15:57 |
| fungi | jrosser: it could be you're hitting a rare packing mismatch between cgit and jgit, try adding --no-thin | 15:58 |
| clarkb | yes --no-thin is the flag I was trying to find | 15:58 |
| fungi | to the git review command, i mean | 15:58 |
| jrosser | oh thats it! thankyou :) | 15:59 |
| fungi | also sorry for the vague error message. short of making git-review catch that in the output somehow in order to be able to amend/replace it with something more helpful i don't know what we could do to make it less opaque | 16:00 |
| clarkb | basically under normal push situations the clietn and server try to figure out what the delta between them is in order to minimize the amount of data that needs to be transferred to complete the push. In some circumstances (that we are not entirely sure of) jgit and cgit do this negotiation and don't produce compatible results on both sides. When you pass --no-thin to git review it | 16:01 |
| clarkb | forward that to git push which does not do the optmized transfer and instead they share a lot more data (maybe all the adta?) | 16:01 |
| fungi | less efficient but apparently also more compatible with other git implementations | 16:01 |
| jrosser | now you mention this i think i do have some distant memory of --no-thin | 16:02 |
| jrosser | apparently too distant for friday afternoon | 16:02 |
| fungi | we only remember it because we get asked every time anyone encounters it | 16:04 |
| clarkb | whcih is like twice a yaer if that. It is very rare | 16:06 |
| clarkb | functests for centos-minimal 10-stream passes so ya I think this is generally working and now we have to sort out what has changed in the upstream image | 16:07 |
| clarkb | and honestly for a first pass saying centos doesn't support 10-stream but centos-minimal does is probably fine | 16:07 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Make scripts DCO compliant https://review.opendev.org/c/openstack/project-config/+/950770 | 16:09 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Comply with Policy for AI Generated Content https://review.opendev.org/c/openstack/project-config/+/950771 | 16:09 |
| fungi | elodilles: ^ some release scripts in there | 16:09 |
| fungi | i've added a "dco-signed-off-by" hashtag to all the dco compliance changes so far, and "generated-by" hashtag to all the generated content changes | 16:13 |
| fungi | oh, looks like i don't have permission to set hashtags on others changes in openstack/releases | 16:16 |
| elodilles | you mean topics? let me set them | 16:21 |
| elodilles | hmmm, you meant hashtag | 16:21 |
| elodilles | i haven't used that before, i've always used topic so far | 16:21 |
| corvus | i added a topic for the next infra meeting to discuss enabling hashtags globally | 16:22 |
| fungi | elodilles: yeah, changes can have a (reasonably) inlimited number of different hashtags set, while they can only ever have at most one topic. also gerrit upstream expects topics to be used for workflow-oriented purposes rather than being purely informational (e.g. they have features for considering changes with the same topic to be interdependent) | 16:24 |
| fungi | s/inlimited/unlimited/ | 16:25 |
| clarkb | https://bugzilla.redhat.com/show_bug.cgi?id=2013894 this is the old xfs can't be mounted problem | 16:32 |
| clarkb | I suspect we're hitting something similar this time around | 16:32 |
| fungi | too-new xfs mounted by a too-old kernel? | 16:33 |
| clarkb | that was the centos 9 stream image issue. Now we're facing a something similar in that the dib tests can't mount the filesystem from the centos 10 stream iamge on bookworm | 16:34 |
| clarkb | https://zuul.opendev.org/t/openstack/build/8c3d111bbe32439ca9c3bed3070515b7/log/logs/centos_10-stream-build-succeeds.FAIL.log#379 | 16:35 |
| clarkb | or wait maybe its a bug in dib? the mount command doesn't supply a mouint point so it is looking things up in fstab | 16:35 |
| clarkb | but maybe that happens beacuse we're unable to properly determine the fs details in the code above | 16:36 |
| clarkb | someone is probably going to need to grab the image and see how it is laid out and then figure out why dib isn't parsing how to mount the root device properly | 16:40 |
| clarkb | looks like ROOT_LOOPDEV is being unset somehow between when we call blkid against it and the mount call | 16:40 |
| clarkb | but I'm not immediately seeing how that get sunset | 16:40 |
| fungi | is it exported? new shell context? | 16:41 |
| fungi | failing to get passed through by sudo maybe? | 16:41 |
| clarkb | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/sysprep/bin/extract-image#L133-L153 roughly maps to https://zuul.opendev.org/t/openstack/build/8c3d111bbe32439ca9c3bed3070515b7/log/logs/centos_10-stream-build-succeeds.FAIL.log#363-378 | 16:42 |
| clarkb | It shouldn't need to be exported its used in the same context | 16:43 |
| clarkb | but also I don't feel a great need to debug this | 16:43 |
| clarkb | centos-minimal works and if people want to build 10-stream from upstream iamges they shoudl be able to debug this | 16:43 |
| fungi | fair enough | 16:44 |
| fungi | this is a big part of why we rely on the -minimal elements | 16:44 |
| clarkb | oh wait I'm blind there is no ROOT_LOOPDEV listed when getting the type and that is why we get all the fs types back | 16:47 |
| clarkb | ok so ya whoever wants to debug this needs to improve the root fs detection in the loop here: https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/sysprep/bin/extract-image#L99-L131 | 16:48 |
| clarkb | I think that /boot is also the / partition and the loop is determining it is /boot and then not allowing it to also be / | 16:53 |
| clarkb | if that is the case (and I haven't pulled the image to look) then I think that would be a bug in the image as the partition id is set for /boot/ and you're supposed to mount that partition to that location. But that partition has fstype xfs and label root too | 16:57 |
| clarkb | fungi: I just remembered to check mailman log rotation and I think it is working now | 18:24 |
| clarkb | and the mailing lists continue to operate as well | 18:24 |
| fungi | yeah, /var/lib/mailman/core/var/logs/ has a couple of rotations in it now, as expected | 18:49 |
| fungi | and the oldest one was compressed successfully in today's rotation, despite being massive from a year+ of being ignored | 18:50 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!