| frickler | infra-root: docs.o.o seems non-responsive, possibly due to some crawler action | 04:53 |
|---|---|---|
| frickler | not 100% broken, but getting either connection timeout or connection refused most of the time | 05:00 |
| frickler | corvus: fyi, looks like that cache error has been happening consistently for all cron runs since sunday | 06:49 |
| *** liuxie is now known as liushy | 07:11 | |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 08:26 |
| mnasiadka | Hmm, anybody has an idea why in https://review.opendev.org/c/openstack/diskimage-builder/+/934045 Zuul says it can't find nodeset nested-virt-ubuntu-noble? | 08:31 |
| frickler | mnasiadka: because it doesn't exist? nodeset != label. see e.g. https://opendev.org/openstack/kolla-ansible/src/branch/master/zuul.d/nodesets.yaml#L62-L66 | 09:05 |
| mnasiadka | right, seems I didn't have my coffee yet ;-) | 09:06 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 09:12 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 09:13 |
| mnasiadka | tonyb: I think I’m done (sort of) with DIB and glean - would appreciate some reviews | 12:10 |
| opendevreview | Slawek Kaplonski proposed openstack/project-config master: Update neutron grafana dashboard with new FWaaS jobs https://review.opendev.org/c/openstack/project-config/+/950660 | 14:04 |
| clarkb | mnasiadka: tonyb looking at https://review.opendev.org/c/openstack/diskimage-builder/+/934045/ I think the glean depends on makes sense (and I'll be rereviewing that shortly), but I think we can/should drop the nodepool update and then we need to sort out how to run devstack? If we run the jobs on the nested virt label (something we are already doing) can we simply set host | 15:03 |
| clarkb | passthrough? | 15:03 |
| clarkb | anyway I sispect that we've gone from glean being the primary blocker to configuring the devstack test env as the main issue now so need to figure out what our answeris there | 15:04 |
| mnasiadka | That nodepool patch I think is only for some debug logs, unless I’m wrong and haven’t looked in depth into it :) | 15:04 |
| clarkb | mnasiadka: ya that is my impression too so we can probably drop it. Then the main concern is how do we run devstack in a way that can boot centos 10. | 15:05 |
| mnasiadka | I guess nested-virt labels are the way to go and host passthrough, shouldn’t be complex | 15:07 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 15:15 |
| mnasiadka | Removed the nodepool patch from deps | 15:16 |
| fungi | gonna grab a quick lunch, should be back by 16:30 utc | 15:37 |
| clarkb | mnasiadka: tonyb: ok I've posted a review to the glean change | 15:43 |
| clarkb | and I've posted a review to the dib change too | 15:58 |
| opendevreview | Merged openstack/diskimage-builder master: CI: Use opendevmirror/ubuntu for functests https://review.opendev.org/c/openstack/diskimage-builder/+/949897 | 16:03 |
| clarkb | mnasiadka: one small thing on https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 16:06 |
| clarkb | my neighbors are boring a new waterline under their driveway. Does not make pleasants sounds | 16:07 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds, labels and provider config https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 16:10 |
| corvus | clarkb: i approved mnasiadka 's change but its dep could use a review https://review.opendev.org/949697 | 16:32 |
| clarkb | oops I missed there was a deepnds on | 16:32 |
| mnasiadka | If Gerrit could highlight depends-on that would be a nice feature ;) | 16:34 |
| frickler | .oO("This change has an unapproved dependency, are you sure you want to W+1?" - "Yes, I really really mean it.") | 16:41 |
| mnasiadka | clarkb: seems the nodepool change is needed ;) | 17:35 |
| clarkb | do we know why? | 17:36 |
| mnasiadka | clarkb: externally-managed-environment https://zuul.opendev.org/t/openstack/build/73208d431af24024adb9fa5802a040b5 | 17:53 |
| mnasiadka | clarkb: this part here https://review.opendev.org/c/zuul/nodepool/+/934742/6/playbooks/nodepool-functional-container-openstack/pre.yaml | 17:54 |
| mnasiadka | clarkb: if tonyb’s rationale is ok for you I can remove the debug part of that change | 17:55 |
| clarkb | oh I see the chagne was doing debugging and ifxing an issue. Yes I think we should probably sdplit the fixup into a separate change so that it can be landed | 17:56 |
| clarkb | then if we need the debugging it can be a seprate change that we depends on for debugging purposes but probably don't need to approve/merge properly | 17:56 |
| clarkb | an alternative would be to use a virtualenv but I think that approach is fine for CI jobs | 17:57 |
| opendevreview | Merged zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 18:22 |
| mnasiadka | clarkb: https://review.opendev.org/c/zuul/nodepool/+/934742 - can you take it out from WIP? I’m not the owner so I can’t. | 18:27 |
| clarkb | mnasiadka: no this is why I still use -W | 18:28 |
| clarkb | I think I have to promtoe myself to admin to do that | 18:28 |
| clarkb | and then figure out the api call. Maybe fungi knows if there is an easier way | 18:28 |
| mnasiadka | In Kolla cores can do that, so maybe it’s a matter of acls? | 18:29 |
| clarkb | toggleWipState = group kolla-core | 18:30 |
| clarkb | ya seems to be ^ | 18:30 |
| clarkb | mnasiadka: what we could do is put the podman install fix in a new chnge and then keep the debugging in the wip change | 18:30 |
| mnasiadka | I don’t think we need the debugging anymore - but if we do - I’ll leave it around. | 18:31 |
| clarkb | mnasiadka: see the zuul room | 18:34 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 18:34 |
| clarkb | tonyb:already pushed an alternative that we can switch to | 18:34 |
| mnasiadka | clarkb: seen, switched | 18:34 |
| clarkb | cool | 18:34 |
| opendevreview | Michal Nasiadka proposed opendev/glean master: Add support for CentOS 10 keyfiles https://review.opendev.org/c/opendev/glean/+/941672 | 19:15 |
| fungi | yeah, administrative users can also unset wip if needed | 19:21 |
| fungi | though i've only done it elevating my normal account and using the webui | 19:22 |
| mnasiadka | clarkb: updated - there are two comments left, I left a question - but link-local is probably what we're after | 19:24 |
| tonyb | Do we have an exiting role for building disk-images suitable for use when switching from nodepool to devstack for DIB? | 19:26 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 19:37 |
| clarkb | tonyb: there is the role that zuul-providers uses from zuul-jobs | 19:50 |
| tonyb | clarkb: Thanks. I don't want to reinvent the wheel | 19:51 |
| clarkb | mnasiadka: for the glean comments I think we need to figure out what interface[type] = manual means when nova writes the config drive | 19:55 |
| clarkb | mnasiadka: I posted a response with more details, but I think it boils down to ^ figuring that out then making the decision that best matches that | 19:55 |
| mnasiadka | Sure, I’ll have a look tomorrow, and fix the cs10 after this round of zuul testing | 19:58 |
| clarkb | ya no rush. mnasiadka also it just occurred to me that we may need to have the dib functest run on noble nested virt labels too | 19:58 |
| clarkb | since its going to run centos 10 code within a chroot and t hat code may be compiled ith the expectation of running on newer harder | 19:59 |
| clarkb | s/harder/hardware/ | 19:59 |
| mnasiadka | Makes sense | 19:59 |
| corvus | tonyb: clarkb i think mnasiadka 's recently-merged change to ensure-dib will install dib from source if it's in the zuul project list, so it should even automatically do the right thing for a job like that. | 20:03 |
| tonyb | Ahh okay that makes things easier | 20:08 |
| clarkb | JayF: if you have a moment can you look at my recent comments on https://review.opendev.org/c/opendev/glean/+/941672 ? This change is trying to add EL10 support to glean and we're getting hung up on something that I suspect is related to bond interfaces and ironic expansion of network_data.json in config drives | 20:20 |
| clarkb | basically hoping you might be able to confirm this is indeed an ironic thing that is being handled there (since nova doesn't seem to produce config drives with those values) | 20:21 |
| JayF | Can you be more specific about file/location in that change? | 20:23 |
| clarkb | https://review.opendev.org/c/opendev/glean/+/941672/27..28/glean/cmd.py#b403 | 20:24 |
| clarkb | its that comment though I'm noticing that the diff itself renders weird there | 20:24 |
| clarkb | https://review.opendev.org/c/opendev/glean/+/941672/28/glean/cmd.py line 399 is maybe a bit more readable | 20:25 |
| clarkb | I suppose another option is that the nova schema is wrong | 20:28 |
| clarkb | `'type': 'ipv%d%s' % (version, net_type),` this appears to be how nova sets the type value and I don't think you can get 'manual' out of that so unlikely that nova's schema is wrong | 20:31 |
| clarkb | the boring machine is gone \o/ but now an electrician has shown up | 20:43 |
| fungi | is the electricial at least slightly less boring? | 20:46 |
| fungi | er, electrician | 20:46 |
| clarkb | much quieter which I apprecite. But I wonder if they put a hole through something they shouldn't have. Our electrical hookups are underground here... | 20:47 |
| clarkb | but it could also be low voltage (phone, coax, fiber) | 20:47 |
| dan_with | Hello. Sorry to drop in like this every time I need something. There is a block storage volume in SJC that needs moved to a different host. I cannot migrate it because it is 'in-use'. Can I have you take a look and migrate it sometime today or tomorrow, please? id: 2815af75-6058-41d1-abbf-6985b2777f49, name: mirror01.sjc3.raxflex.opendev.org/main02 | 20:57 |
| clarkb | dan_with: I assume the migration will be noticed by the host attaching that volume? | 20:58 |
| clarkb | if so we can turn off that region and then shutdown the node to avoid problems | 20:58 |
| fungi | yeah, that sounds like the least disruptive path. happy to get the change up to drop max-servers to 0 in nodepool for that region now | 20:59 |
| dan_with | Yes, you are correct. Shutting off that region and then the node will likely be beneficial to the process. | 20:59 |
| clarkb | fungi: dont' forget to do the zuul-providers side too | 20:59 |
| fungi | changeS thanks for the reminder! | 20:59 |
| dan_with | OHHHH! Hold off, I just realized that volume is AVAILABLE. It's not attached. So, I should be able to move it. | 21:03 |
| fungi | oh, neat, i wonder if we missed cleaning up an old volume? | 21:03 |
| clarkb | that could be let me look | 21:04 |
| clarkb | dan_with: mirror02.sjc3.raxflex.opendev.org/main01 this is the only volume I see from volume list in SCJ3 | 21:05 |
| clarkb | I wonder if we deleted it and it went away on the cloud side but has hung around in the storage system? | 21:05 |
| clarkb | dan_with: so I think you can proceed with doing whatever with it including cleaning it up | 21:05 |
| dan_with | I'm migrating it safely now. You can decide what to do with it later. | 21:07 |
| fungi | yeah, i'd be surprised if it shows up in the cinder api for anyone, since we don't see it | 21:08 |
| fungi | dan_with: doesn't sound like we could do anything with it, we don't even see that it exists in cinder at all | 21:08 |
| fungi | oh! unless it's somehow orphaned in the "old" project from before all the sjc3 projects got replaced | 21:10 |
| clarkb | right, openstack volume list does not show it to us | 21:10 |
| fungi | if i can remember how to authenticate to the old project in sjc3 i might be able to check that | 21:11 |
| clarkb | from memory we had mirror01.sjc3 in that region with a main/01 volume and then at some point that volume stopped working (we were told this happened around some volume encrpytion stuff). To quickly workaround that we created a new volume (main02?) and replaced main01 | 21:11 |
| clarkb | then later we replaced the mirror entirely with mirror02 | 21:11 |
| clarkb | fungi: oh yup that probably explains where it went | 21:11 |
| clarkb | if you do figure it out I think it can be safely cleaned up from our end | 21:12 |
| fungi | oh, actually eeasiest is probably to just jump into skyline and i think it lets you pick between the two projects | 21:13 |
| dan_with | The volume migrated successfully. You can do whatever you need to with it. Sorry for the false alarm | 21:22 |
| fungi | clarkb: dan_with: yep, that was it. looks like we deleted the server instance from our old 610275_Flex project before we moved to the new cross-region uuid project, but we missed cleaning up that volume. i'm deleting it now | 21:22 |
| fungi | Delete volume successfully, instance: mirror01.sjc3.raxflex.opendev.org/main02. | 21:23 |
| fungi | all gone now | 21:23 |
| dan_with | Hey, question for you all: I'm trying to migrate someone else's boot-from-volume instance volume. Cinder/openstack doesn't like that. What is the best way to handle a volume migration of a boot-from-volume? Take an image of the instance, delete the instance, migrate the volume, then rebuild the instance from the image? | 21:27 |
| fungi | dan_with: that might be a better question for the #openstack-cinder channel, but maybe mnaser has suggestions since vexxhost does mostly ceph-backed bfv | 21:30 |
| clarkb | dan_with: fungi: if the migration can be done with things shut off the user/customer may be willing to do that depending on how long it will take and what the workload is | 22:03 |
| clarkb | sometimes that is easier than rebuilding everything | 22:03 |
| dan_with | yeah, that's been the standard process, but it takes a lot of communication and planning when we have thousands of customers. I thought we would get away from some of this with Genestack, but storage is always going to be the foundation. Moving data around is challenging. Thanks for the prompt response. Appreciate it. | 22:05 |
| clarkb | with nova live migration I've often thought it would be cool to have a label/tag/note on instances indicating that a short non live migration is preferred to live migration as live migration can take a long time then fail and create weird behaviors sometimes and for many of our instances a 2 minute outage would just be better | 22:07 |
| clarkb | but as far as I know that doesn't exist | 22:07 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!