| -opendevstatus- NOTICE: The maintenance of the review.opendev.org Gerrit service is now complete and service has been restored. Please alert us in #opendev if you have any issues. Thank you | 03:28 | |
| *** akekane_ is now known as abhishekk | 07:07 | |
| opendevreview | Lukas Piwowarski proposed openstack/patrole master: Add cleanup for test_show_auto_allocated_topology test https://review.opendev.org/c/openstack/patrole/+/788316 | 07:12 |
|---|---|---|
| *** pojadhav- is now known as pojadhav | 07:41 | |
| opendevreview | Merged openstack/devstack stable/train: Fix is_fedora RHEL 8 detection https://review.opendev.org/c/openstack/devstack/+/756490 | 09:22 |
| *** rpittau|afk is now known as rpittau | 09:31 | |
| *** kopecmartin is now known as kopecmartin|pto | 09:45 | |
| lyarwood | gmann: https://review.opendev.org/c/openstack/devstack/+/798514 - would you mind hitting this re centos 8 stream support | 11:29 |
| opendevreview | Slawek Kaplonski proposed openstack/devstack master: Stop creating userrc_early https://review.opendev.org/c/openstack/devstack/+/780417 | 12:59 |
| opendevreview | Slawek Kaplonski proposed openstack/devstack master: Deploy Neutron with enforced new RBAC rules https://review.opendev.org/c/openstack/devstack/+/797450 | 12:59 |
| opendevreview | Ananya Banerjee proposed opendev/elastic-recheck master: Run elastic-recheck container https://review.opendev.org/c/opendev/elastic-recheck/+/729623 | 13:06 |
| opendevreview | Ananya proposed opendev/elastic-recheck master: Run elastic-recheck container https://review.opendev.org/c/opendev/elastic-recheck/+/729623 | 13:14 |
| opendevreview | Julia Kreger proposed openstack/devstack master: Provide a sane default glance limit https://review.opendev.org/c/openstack/devstack/+/801309 | 14:04 |
| TheJulia | I'd greatly appreciate reviews on ^^^ as ironic's gate is broken at the moment. | 14:05 |
| clarkb | that change seems fine to me. gmann and dansmith should probably take a look though | 15:32 |
| clarkb | dansmith: also if you have any large patch update stacks to push today I'd be curious to hear if they go any quicker than before | 15:32 |
| dansmith | clarkb: okay, we just split apart a big one last week, so not sure, but will let you know | 15:32 |
| dansmith | what does "quicker" mean and ... why? | 15:33 |
| dansmith | quicker like gerrit submission or test runtime? | 15:33 |
| clarkb | dansmith: gerrit submission. We migrated gerrit servers last night | 15:33 |
| dansmith | ah, cool, it's been painful lately, so that's great to hear | 15:34 |
| clarkb | to a bigger server to try and allevaite memory pressure | 15:34 |
| dansmith | fantastic | 15:36 |
| dansmith | TheJulia: why don't you just set GLANCE_ENABLE_QUOTAS=False in your job? | 16:32 |
| dansmith | there's no real need for you to run with those enabled, and you can merge that right now, yeah? | 16:33 |
| TheJulia | dansmith: We shouldn't have to change our jobs becasue of quota enforcement now being a default thing in the jobs | 16:33 |
| dansmith | but you want to change the default limit for everybody? I don't see the distinction... | 16:34 |
| TheJulia | and the larger question in my mind is, what if some person fires up devstack and tries to load centos in right now | 16:34 |
| TheJulia | I'm suggesting the default limit is wrong | 16:34 |
| dansmith | all of the disk size defaults we have are too small to do much real stuff right? | 16:35 |
| TheJulia | basically, yes | 16:35 |
| TheJulia | If someone was using ubuntu or debian, those images are stock about half the size | 16:36 |
| TheJulia | but anything customized is going to get decently sized quick if someone is doing basic testing | 16:36 |
| dansmith | right, but even with ubuntu disks being small, you can't fit two of them, or one and a snapshot, etc | 16:36 |
| * TheJulia shrugs | 16:36 | |
| TheJulia | dansmith: that is quite true | 16:36 |
| dansmith | I just don't think devstack needs to promise that the defaults will work for any system, any image, running a full parallel tempest | 16:37 |
| TheJulia | so are we saying we don't care about someone using centos stream + devstack? | 16:37 |
| dansmith | especially given that you can update the quota afterwards with a single command.. so if someone deploys a devstack, tries to upload a 1.2G image, they don't have to restack, they can just expand the limit | 16:38 |
| dansmith | TheJulia: now really, is that what I said? :) | 16:38 |
| TheJulia | dansmith: that is kind of how it comes off, to me at least if I was in the shoes of someone who didn't know waaaaay too much about all this stuff :) | 16:38 |
| TheJulia | basic development+testing should be able to be usable without having to have in depth knowledge of quotas | 16:39 |
| dansmith | would it help if I said I think the default should be 500M so that even the ubuntu images won't fit? | 16:39 |
| TheJulia | having to execute a command could also be viewed as a bug | 16:39 |
| dansmith | omg, having to execute a command? the horror! | 16:40 |
| TheJulia | i know right! | 16:40 |
| dansmith | I think most people do most testing with the cirros image, and if they're going to use real images, they're expecting to have to make some changes | 16:40 |
| TheJulia | seriously though, if we want to say 500, then that is fine, but we're going to brick basically anything that does custom images to test realistic scenarios | 16:40 |
| dansmith | *with defaults | 16:41 |
| TheJulia | yeah, we shouldn't have to modify defaults heavily to do basic scenario testing either | 16:41 |
| dansmith | I can't remember the last time I ran devstack without a local.conf | 16:41 |
| TheJulia | True, but the more and more options only known to developers that exist the harder it is to craft a config that works every time | 16:41 |
| * TheJulia wonders if this will also break third party CIs | 16:41 | |
| * TheJulia hasn't even checked them yet | 16:42 | |
| TheJulia | hmm 2 days ago | 16:44 |
| opendevreview | Pavan Kesava Rao proposed openstack/devstack stable/wallaby: Enable tempest tests for hostname sanitization https://review.opendev.org/c/openstack/devstack/+/801356 | 16:47 |
| TheJulia | dansmith: I've inquired at with at least one operator that should be around to reply, I suspect they are doing something funky but that they would be broken as they do scenario with whole images afaik | 16:48 |
| TheJulia | and they don't pull in ironic's zuul configs to run thier jobs :\ | 16:49 |
| dansmith | do they run with 100% default devstack? | 16:49 |
| dansmith | even the most basic of jobs that runs across almost everything has a pretty big config file delta from the devstack defaults: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4fd/801309/1/check/tempest-full-py3/4fd1de1/controller/logs/local_conf.txt | 16:53 |
| TheJulia | The change basically breaks anyone who uses any devstack with any larger images config that is not explicitly disabling it so they all need to now update their configs where previously the limit check didn't even exist. | 16:56 |
| dansmith | and a 5GB limit will just break fewer people but not avoid that entirely right? | 16:57 |
| dansmith | just to circle back to the pressing issue and away from the philosophical one, I don't really care that much what the default is. I'd like it enabled by default so we're doing the checking everywhere, but the tests don't care about devstack's default | 16:59 |
| TheJulia | dansmith: true, hopefully far fewer. | 16:59 |
| TheJulia | So, the largest 3rd party CI we've got in ironic doesn't hit this becasue they run ubuntu images for deployments *whew* | 17:00 |
| dansmith | to summarize my opinion (which doesn't matter at all) on the philosophical issue, I don't think that (a) devstack promises config file stability and (b) devstack defaults should be promised to work for anything other than a basic stack and use of the cirros image | 17:01 |
| TheJulia | dansmith: I'm totally cool with the default nature of it, and I personally agree. My concern is largely just being basically minimally usable without a bunch of extra work or extra special openstack contextual knowledge | 17:02 |
| TheJulia | dansmith: feels like we need a philosphy file someplace | 17:03 |
| dansmith | pretty sure most CI systems are having to configure neutron/network stuff to make guest checks work | 17:03 |
| dansmith | and the default is totally fine for the image that devstack configures for you by default | 17:04 |
| TheJulia | I think b is really bad for baremetal integration, but c'est la vie. | 17:04 |
| dansmith | you could even say that uploading a new image or configuring devstack to use a different one is "running a command" which puts it pretty much on the same footing as tweaking the image quota default, | 17:05 |
| dansmith | or adding more volumes to the quota, which I've had to do before because the default is 10 | 17:05 |
| TheJulia | so how are people going to find out about the setting? | 17:05 |
| TheJulia | just the admin through the logs? | 17:05 |
| TheJulia | I guess that doesn't really matter in the grand scheme of things | 17:06 |
| TheJulia | We can just document it or force override it in ironic's devstack plugin | 17:06 |
| TheJulia | oh wait... maybe not | 17:06 |
| dansmith | we've got a nice config doc with service details, | 17:06 |
| TheJulia | yeah, no, glance will exec and setup first | 17:07 |
| dansmith | I'll be glad to add something there | 17:07 |
| dansmith | in doc/source/configuration.rst | 17:07 |
| dansmith | TheJulia: you are totally able to do the override in your plugin, and/or disable it as I noted earlier, but you said you didn't want to | 17:07 |
| gmann | lyarwood: +A | 17:10 |
| TheJulia | dansmith: no, I said I shouldn't have to, there is a difference, and it is not just one job. Luckily most of it all draws from the in-repo ironic-base job, so two places should address it (devstack making it configurable, and then a change to accept the override (because I agree, it should still exist but just be at a reasonable number for test jobs to run where baremetal is concerned. Downstream CI folks will just have to | 17:11 |
| TheJulia | change stuff as they find the breakage I guess | 17:11 |
| dansmith | it's already disable-able | 17:11 |
| TheJulia | I don't think it should be wholesale disabled | 17:12 |
| opendevreview | Dan Smith proposed openstack/devstack master: Add configuration notes about glance limits https://review.opendev.org/c/openstack/devstack/+/801360 | 17:12 |
| dansmith | depends on your limit variable ^ | 17:12 |
| * TheJulia goes and revises her patch | 17:12 | |
| TheJulia | dansmith: awesome, thanks | 17:12 |
| dansmith | I mean human-depends, zuul doesn't know :) | 17:12 |
| TheJulia | i know :) | 17:13 |
| opendevreview | Pavan Kesava Rao proposed openstack/tempest master: Add test to verify FQDN hostname sanitization https://review.opendev.org/c/openstack/tempest/+/801362 | 17:16 |
| opendevreview | Julia Kreger proposed openstack/devstack master: Provide override for glance image size limit https://review.opendev.org/c/openstack/devstack/+/801309 | 17:17 |
| TheJulia | dansmith: revised ^ | 17:17 |
| gmann | kopecmartin: can you approve this quickly in case wallaby constraints start breaking gate https://review.opendev.org/c/openstack/tempest/+/801174/2 | 17:18 |
| dansmith | clarkb: I tell you, the web ui for gerrit certainly doesn't feel faster today :/ | 17:18 |
| TheJulia | I've had a couple UI slowdowns, but nothing like Thursday or Friday | 17:18 |
| TheJulia | "hold on, let me go make a sandwich" | 17:18 |
| gmann | dansmith: TheJulia so we are going with 1gb default but configurable right? | 17:19 |
| dansmith | gmann: sounds like it | 17:19 |
| clarkb | ya I think it is the hold on make a sandwich cases we expect to go away | 17:19 |
| opendevreview | Pavan Kesava Rao proposed openstack/tempest master: Add test to verify FQDN hostname sanitization https://review.opendev.org/c/openstack/tempest/+/795699 | 17:19 |
| dansmith | TheJulia: this morning (and just now) it was like 30s to load the page | 17:19 |
| clarkb | but general rtt and java is slow impacts not so much | 17:19 |
| dansmith | not quite sandwich territory, but long enough for me to get distracted | 17:19 |
| TheJulia | dansmith: yeouch | 17:19 |
| clarkb | dansmith: huh I haven't experienced that yet. What page was it? | 17:19 |
| dansmith | clarkb: TheJulia's link above just now | 17:20 |
| dansmith | voting went quicker, but the initial load was slowwww | 17:20 |
| gmann | dansmith: can you add default value also here https://review.opendev.org/c/openstack/devstack/+/801360 | 17:20 |
| clarkb | hrm that loaded really quickly for me just now. I wonder if now we're hitting cold cache issues | 17:20 |
| clarkb | that will be something to look into I guess | 17:20 |
| dansmith | clarkb: gmann's link loaded quick for me just now | 17:21 |
| clarkb | dansmith: ya I wonder if you loading TheJulia's change caused gerrit to go warm up the devsatck git caches | 17:21 |
| clarkb | and so you lost on the first load but subsequent loads are happier | 17:21 |
| clarkb | I think some cache coldness was expected because we didn't migrate the caches over from the old server | 17:21 |
| dansmith | maybe | 17:21 |
| opendevreview | Dan Smith proposed openstack/devstack master: Add configuration notes about glance limits https://review.opendev.org/c/openstack/devstack/+/801360 | 17:22 |
| dansmith | gmann: ^ rebased and default mentioned | 17:23 |
| dansmith | clarkb: only two spinner ticks when submitting this ^ definitely seems faster | 17:23 |
| clarkb | that is good news | 17:23 |
| gmann | dansmith: thanks +2 | 17:24 |
| clarkb | dansmith: one thing having additional memory allows us to do is to further tune our caches too (make them bigger). I don't think we'll dive into that next week but it is something I expect we'll be doing now that we have the ability to do so | 17:26 |
| clarkb | *I don't think we'll dive into that until at least next week | 17:26 |
| dansmith | well, cool.. sucks to have to increase the memory budget so much but.. glad we can, I guess | 17:27 |
| opendevreview | Pavan Kesava Rao proposed openstack/devstack master: Enable tempest tests for hostname sanitization https://review.opendev.org/c/openstack/devstack/+/801366 | 17:34 |
| *** rpittau is now known as rpittau|afk | 17:38 | |
| opendevreview | Pavan Kesava Rao proposed openstack/tempest master: Add test to verify FQDN hostname sanitization https://review.opendev.org/c/openstack/tempest/+/795699 | 17:39 |
| clarkb | I've approved the glance limits changes as it seems we are happy to go with the updates that dansmith made. | 17:42 |
| opendevreview | Merged openstack/devstack master: libvirt: Stop installing python bindings from pip https://review.opendev.org/c/openstack/devstack/+/798514 | 18:27 |
| clarkb | lyarwood: gmann yoctozepto ^ I think the real issue there is you are using centos 8 wheels on centos8 stream | 18:41 |
| clarkb | sounds like they need to be treated differently | 18:41 |
| yoctozepto | clarkb: then again, installing from sources may fail if platform's libvirt headers are incompatible | 18:42 |
| yoctozepto | been there, fixed that ;-) | 18:42 |
| yoctozepto | (w/ masakari) | 18:42 |
| clarkb | yoctozepto: they shouldn't be incompatible if you use teh correct wheel builds is my point | 18:43 |
| clarkb | the wheels we build for libvirt-python are distro release specific to avoid those problems | 18:43 |
| yoctozepto | clarkb: we would need to pin the version per distro | 18:43 |
| yoctozepto | it's easier to install from repo | 18:43 |
| clarkb | yoctozepto: no because newer libvirt-python is supposed to build against older libvirt. If that doesnt' work it is a bug upstream | 18:43 |
| yoctozepto | yeah, I was just saying that | 18:43 |
| yoctozepto | (i mean, writing a very similar message) | 18:44 |
| clarkb | the problem the above change addresses is using wheels built for one platform on another platform. We should not do that | 18:44 |
| yoctozepto | indeed | 18:44 |
| yoctozepto | I like the way we changed that in devstack | 18:44 |
| clarkb | Calling that out because other wheels could have similar issues over time | 18:44 |
| yoctozepto | as that is how production deployment projects do it | 18:44 |
| clarkb | changing libvirt-python to a distro package install doesn't fix the underlying issue | 18:45 |
| yoctozepto | yeah, wheels could always go wrong by not including the exact underlying libs versions | 18:45 |
| yoctozepto | but it's not easily fixable | 18:45 |
| yoctozepto | we haven't been hit hard before | 18:45 |
| yoctozepto | i.e., I am satisfied with the current workaround | 18:46 |
| clarkb | ok | 18:46 |
| dansmith | clarkb: just had another ~20s page load... Replying to something from 30m ago | 19:20 |
| dansmith | I dunno how long things stay in the cache, but I'm guessing less than 30m | 19:20 |
| dansmith | definitely faster submitting a 5-patch set | 19:38 |
| clarkb | dansmith: that is good to know. I'll have to double check gerrit cache thigns once stuff settles down. Its possible we can increase the ttls too | 19:58 |
| opendevreview | Merged openstack/devstack master: Provide override for glance image size limit https://review.opendev.org/c/openstack/devstack/+/801309 | 20:29 |
| opendevreview | Merged openstack/devstack master: Add configuration notes about glance limits https://review.opendev.org/c/openstack/devstack/+/801360 | 20:29 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!