| *** dviroel|rover is now known as dviroel|out | 00:19 | |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 01:27 | |
| *** bhagyashris is now known as bhagyashris|out | 03:30 | |
| *** ysandeep|out is now known as ysandeep | 04:53 | |
| *** ykarel|away is now known as ykarel | 05:19 | |
| *** ykarel_ is now known as ykarel | 06:13 | |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack-gate master: Cap PyYAML to stay compatible with our code https://review.opendev.org/c/openstack/devstack-gate/+/813958 | 07:30 |
|---|---|---|
| *** jpena|off is now known as jpena | 07:32 | |
| *** ykarel is now known as ykarel|lunch | 08:14 | |
| *** ysandeep is now known as ysandeep|lunch | 08:35 | |
| *** ykarel|lunch is now known as ykarel|lunc | 09:27 | |
| *** ykarel|lunc is now known as ykarel | 09:27 | |
| *** ysandeep|lunch is now known as ysandeep | 09:39 | |
| -opendevstatus- NOTICE: zuul was stuck processing jobs and has been restarted. pending jobs will be re-enqueued | 10:02 | |
| *** jcapitao is now known as jcapitao_lunch | 10:26 | |
| *** rlandy is now known as rlandy|ruck | 10:37 | |
| sean-k-mooney | clarkb: sorry not that i know of with regrads to f34 | 11:04 |
| *** dviroel|out is now known as dviroel|rover | 11:09 | |
| *** jpena is now known as jpena|lunch | 11:27 | |
| *** akahat is now known as akahat|afk | 11:54 | |
| *** ysandeep is now known as ysandeep|afk | 11:58 | |
| fungi | ade_lee: i know you've been working some on fips testing, has that gotten to the point of trying tempest, or is it just unit and functional tests for now? | 12:08 |
| frickler | fungi: there were some wip patches, but not working yet iirc, let me check | 12:17 |
| fungi | we've got cloud providers starting to show an interest in having fips-compliant deployments, but getting tempest (via refstack) to work is apparently blocking the ability to actually claim they have interoperable services | 12:17 |
| fungi | lots of problems ssh'ing, generating certs, et cetera | 12:18 |
| frickler | https://review.opendev.org/c/openstack/neutron/+/797537 | 12:18 |
| fungi | thanks frickler! | 12:18 |
| frickler | fungi: sure, may I ask where you see those cloud providers showing interest? is that via the foundation? | 12:22 |
| fungi | frickler: yes, approaching the foundation about certifying their deployments as interoperable so they can use the trademarks | 12:22 |
| *** jpena|lunch is now known as jpena | 12:22 | |
| frickler | well they don't need to run tempest on a fips-enable machine to do that, or do they? | 12:23 |
| fungi | i'm only just hearing about the problem now, but the test architecture is certainly one of my first questions | 12:24 |
| frickler | might be worth discussing with the qa team once you have more details | 12:25 |
| fungi | yep, i'll also push to see if one of these companies can discuss their challenges with the community directly, since having an intermediary in such matters tends to just get in the way | 12:27 |
| frickler | +1 | 12:28 |
| fungi | i'm also going to try to track the fips testing work a little more actively from the security sig too, in case we can help drum up more people interested in working on it | 12:46 |
| fungi | (not that there are a bunch of people involved there any more either) | 12:47 |
| *** ysandeep|afk is now known as ysandeep | 12:55 | |
| *** ysandeep is now known as ysandeep|brb | 12:57 | |
| afaranha | Hi, we have a test timing out on the CI: https://zuul.opendev.org/t/openstack/build/c1c1a44dcb7d42e08d16f7e929e00378 | 12:57 |
| afaranha | it's from this FIPS patch https://review.opendev.org/c/openstack/swift/+/796057 | 12:57 |
| afaranha | When running the tests locally on a Centos8 server and FIPS enabled, it works. For the next step, can we have access to the node to troubleshoot the issue? or have a sosreport? | 12:57 |
| *** bhagyashris|out is now known as bhagyashris|mtg | 13:00 | |
| fungi | afaranha: great timing! we were just talking about some similar jobs in here. and yeah, since the run phase playbook timed out it seems we don't generate a summary of it due to incomplete ansible metadata | 13:01 |
| fungi | i made the mistake of trying to pull up the job log in my browser, so waiting for it to stop having a fit before i try to download a copy to go through locally instead | 13:02 |
| *** ysandeep|brb is now known as ysandeep | 13:03 | |
| afaranha | fungi, yea, it takes forever on the browser :P | 13:04 |
| afaranha | fungi, do you see more timing out jobs on swift? | 13:04 |
| fungi | afaranha: i downloaded the job-output.txt, decompressed it and looked at it locally. tox ran out of disk space on the test node, though i'm not certain that's why the playbook timed out it certainly wouldn't have succeeded after that | 13:06 |
| fungi | 2021-10-04 15:16:14.471609 | centos-8 | OSError: [Errno 28] No space left on device | 13:07 |
| afaranha | fungi, we would think that swift was retrying to create containers many times and that's why it was timing out | 13:09 |
| *** akahat|afk is now known as akahat | 13:10 | |
| afaranha | but it's passing on local server | 13:10 |
| fungi | afaranha: how much disk space do you have locally? more than the 60gb or so which tends to be available for file creation on our test nodes? | 13:10 |
| afaranha | do you know how can we troubleshoot this? how many containers it tried to create? | 13:11 |
| afaranha | fungi, 40GB, 2 cores 4GB RAM | 13:11 |
| afaranha | fungi, after the tests: 178M./.tox | 13:13 |
| fungi | i think swift's tests create files outside .tox though | 13:15 |
| fungi | looking at https://zuul.opendev.org/t/openstack/build/c1c1a44dcb7d42e08d16f7e929e00378/log/zuul-info/zuul-info.centos-8.txt#100 the job started with 25gb on its rootfs | 13:16 |
| fungi | normally, devstack would format and mount the ephemeral disk on the nodes in that provider at /opt, but i'm not seeing evidence of it doing that in the log | 13:17 |
| fungi | looking at the log, it creates a 1gb file and formats it xfs to mount on a loop device as a scratch space for xattr tests | 13:20 |
| fungi | oh! i think that may be where the "No space left on device" error is actually coming from, it seems like it may have run out of available space in that 1gb xfs filesystem it created | 13:21 |
| fungi | the exception is raised in swift/obj/diskfile.py from write_metadata calling to xattr | 13:22 |
| fungi | so i have a feeling that's where it was writing | 13:23 |
| ade_lee | fungi, frickler thats interesting to hear about other folks looking to have fips-compliant deployments | 13:28 |
| ade_lee | fungi, there are a bunch of patches out there right now for both unit and tempest tests | 13:29 |
| ade_lee | fungi, this is some of them -- https://review.opendev.org/q/topic:%22add_fips_job%22+(status:open%20OR%20status:merged) | 13:30 |
| afaranha | fungi, the fix would be to increase this file size? | 13:31 |
| ade_lee | fungi, the issue with tempest is that tempest uses paramiko and paramiko uses some md5 there. To get some results, I ended up patching paramiko to allow the md5, and so most of the tests there depend on having a hacked parmiko. But there are some other patches out there where I'm trying to replace paramiko in tempest with libssh | 13:32 |
| ade_lee | fungi, https://review.opendev.org/c/openstack/tempest/+/806274 | 13:33 |
| ade_lee | fungi, I think maybe I need to attend the security SIG session next week to discuss fips | 13:34 |
| fungi | ade_lee: that would be great! | 13:46 |
| ade_lee | afaranha, one thing I remember doing - but I might be misremembering - is also running the swift encryption job without fips enabled - and that job passed. We should probably do that again just to make sure. That would confirm that there really is something going on with fips and not vm sizes | 13:46 |
| fungi | afaranha: i don't know for sure that will fix it, the xfs filesystem running out of space could just be masking other problems as well, but it might at least fix whichever test are hanging | 13:46 |
| ade_lee | afaranha, that is running the encryption test using the centos nodeset | 13:47 |
| fungi | ade_lee: on closer inspection i don't think it's complaining about the vm size, but rather running out of space on the file-backed 1gb xfs loop fs the swift jobs create for the xattr tests | 13:47 |
| fungi | i'm not sure why that would only happen in the fips job, but maybe swift needs more space when using fips-compliant hashes? | 13:48 |
| afaranha | ade_lee, I recreate the env today, and re run without fips and with fips | 13:48 |
| afaranha | all passed | 13:48 |
| ade_lee | fungi, afaranha hmm .. so where is the file size defined in the test? we could try bumping to 1.5G .. | 13:50 |
| fungi | it's also possible filling up that xfs fs is merely a symptom of some other failure, causing a test to retry creating files in there over and over or something | 13:50 |
| fungi | i'm not really familiar with swift's jobs, i was mainly going by what i saw in the job-output.txt from the failing build, but can try to find where it's set | 13:51 |
| ade_lee | afaranha, lets do a run where we do the test on the centos-8-stream nodeset without fips enabled -- at least we can confirm then that there is something fips related. | 13:53 |
| afaranha | ade_lee, I actually don't know how to check this | 13:56 |
| ade_lee | afaranha, ok - let me take a quick look and we can sync | 13:56 |
| afaranha | ade_lee, ack, I will test without fips enabled again | 13:56 |
| afaranha | ade_lee, sorry, I'm going to the glance meeting in 3min, I need to talk about the bindep.txt | 13:57 |
| ade_lee | afaranha, sounds good | 13:57 |
| afaranha | ade_lee, btw, shouldn't we have some CIs running on centos upstream as well? | 13:58 |
| fungi | we do have jobs running on centos-7, centos-8, centos-8-stream, and there's centos-9-stream images in the works | 14:01 |
| ade_lee | afaranha, so I think there is already a swift job defined there -- swift-tox-func-encryption-py36-centos-8 which presumably passes | 14:02 |
| ade_lee | afaranha, actually we're running it in https://review.opendev.org/c/openstack/swift/+/796057 and we see it succeeds with no fips enabled and not when fips is enabbled | 14:03 |
| *** ykarel is now known as ykarel|away | 14:21 | |
| afaranha | ade_lee, fungi I was asking mostly because of the glance issue | 14:26 |
| fungi | which glance issue? | 14:26 |
| afaranha | fungi, https://review.opendev.org/c/openstack/glance/+/790536/28 | 14:27 |
| afaranha | the test was failing because it needs to install qemu-img to run the test, not fips related | 14:28 |
| fungi | ahh, okay | 14:28 |
| fungi | i guess qemu-img isn't packaged for centos? | 14:29 |
| afaranha | fungi, currently not. I'll check if the previous version had this issue, glance team want to backport it to victoria (I'm not sure yet if victoria uses centos7) | 14:30 |
| afaranha | https://bugs.launchpad.net/glance/+bug/1947146 | 14:30 |
| afaranha | fungi, for ubuntu it install qemu and qemu-utils package, but for centos there's nothing https://github.com/openstack/glance/blob/master/bindep.txt#L23-L24 | 14:31 |
| afaranha | s/ubuntu/dpkg/ s/centos/rpm/ | 14:31 |
| fungi | in nodepool we use qemu-img to perform image conversions (build raw, make qcow2 and vhd from that) so we can upload the "same" image to multiple providers who have different hyoervisors with their own image format requirements, but we use a forked qemu-img based on the ubuntu package | 14:32 |
| afaranha | fungi, what's the nodepool? | 14:37 |
| fungi | afaranha: a component of zuul: https://zuul-ci.org/docs/nodepool/ | 14:38 |
| fungi | it's the part of its service fleet which does resource management and assignment | 14:38 |
| fungi | (build/upload/replace virtual machine images, boot/delete test nodes) | 14:39 |
| fungi | though the use of qemu-img is via diskimage-builder | 14:40 |
| afaranha | fungi, that's a bit confusing | 14:43 |
| afaranha | the test was running qemu-img manually, maybe that's why we didn't see this issue on other tests... | 14:44 |
| fungi | sorry, i was just pointing out that while we do use qemu-img in production, we do so with a (forked) version of the ubuntu packages, not on centos | 14:45 |
| afaranha | fungi, https://github.com/openstack/glance/blob/master/glance/tests/functional/v2/test_images.py#L906-L912 | 14:45 |
| fungi | afaranha: yeah, if you're trying to call qemu-img create on a system where it's not installed, then you'll probably not have much luck: https://opendev.org/openstack/glance/src/branch/master/bindep.txt#L24 | 14:48 |
| fungi | the jobs will install it on dpkg-based platforms like ubuntu or debian based on that line in the bindep.txt file | 14:49 |
| fungi | you'd need to add a similar line for the name of whatever package in centos provides the qemu-img command (if there is one) | 14:49 |
| *** bhagyashris|mtg is now known as bhagyashris|away | 15:03 | |
| *** ysandeep is now known as ysandeep|dinner | 15:45 | |
| *** ysandeep|dinner is now known as ysandeep | 16:00 | |
| *** dpawlik5 is now known as dpawlik | 16:37 | |
| *** ysandeep is now known as ysandeep|out | 16:39 | |
| *** jpena is now known as jpena|off | 16:42 | |
| *** sboyron_ is now known as sboyron | 17:09 | |
| *** sboyron is now known as Guest2897 | 17:10 | |
| *** sboyron__ is now known as sboyron_ | 17:36 | |
| *** sboyron_ is now known as sboyron | 18:21 | |
| opendevreview | Merged openstack/devstack-gate master: Cap PyYAML to stay compatible with our code https://review.opendev.org/c/openstack/devstack-gate/+/813958 | 18:47 |
| clarkb | frickler: ^ thank you for jumping on that | 18:48 |
| fungi | oh, sorry i meant to review that earlier. but yes thanks! | 18:51 |
| frickler | yeah, that one seemed easy enough to do a quick fix instead of telling people to finally stop using ds-gate right now | 18:58 |
| frickler | also it was the patch that triggered me to look at the zuul issues when it didn't have results after 2h | 18:58 |
| fungi | right, people thankfully *have* stopped using devstack-gate for the latest openstack releases, but older stable branches of some projects are still using it | 19:00 |
| fungi | dropping current use of d-g took about 2 years longer than we had hoped it would, but i'm okay with where we're finally at with it | 19:01 |
| *** dviroel|rover is now known as dviroel|rover|afk | 21:30 | |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 22:20 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!