| *** EugenMayer43 is now known as EugenMayer4 | 01:26 | |
| opendevreview | John Garbutt proposed openstack/kolla-ansible master: Ensure we set dns_servers for OVN https://review.opendev.org/c/openstack/kolla-ansible/+/884460 | 08:22 |
|---|---|---|
| opendevreview | John Garbutt proposed openstack/kolla-ansible master: Ensure we set dns_servers for OVN https://review.opendev.org/c/openstack/kolla-ansible/+/884460 | 08:23 |
| opendevreview | John Garbutt proposed openstack/kolla-ansible master: Ensure we set dns_servers for OVN https://review.opendev.org/c/openstack/kolla-ansible/+/884460 | 08:36 |
| opendevreview | OpenStack Release Bot proposed openstack/ansible-collection-kolla stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/ansible-collection-kolla/+/884462 | 08:38 |
| opendevreview | OpenStack Release Bot proposed openstack/ansible-collection-kolla stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/ansible-collection-kolla/+/884463 | 08:38 |
| opendevreview | OpenStack Release Bot proposed openstack/ansible-collection-kolla master: Update master for stable/2023.1 https://review.opendev.org/c/openstack/ansible-collection-kolla/+/884464 | 08:38 |
| opendevreview | OpenStack Release Bot proposed openstack/kolla-ansible stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/kolla-ansible/+/884465 | 08:39 |
| opendevreview | OpenStack Release Bot proposed openstack/kolla-ansible stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/kolla-ansible/+/884466 | 08:39 |
| opendevreview | OpenStack Release Bot proposed openstack/kolla-ansible master: Update master for stable/2023.1 https://review.opendev.org/c/openstack/kolla-ansible/+/884467 | 08:39 |
| opendevreview | OpenStack Release Bot proposed openstack/kolla stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/kolla/+/884468 | 08:40 |
| opendevreview | OpenStack Release Bot proposed openstack/kolla stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/kolla/+/884469 | 08:40 |
| opendevreview | OpenStack Release Bot proposed openstack/kolla master: Update master for stable/2023.1 https://review.opendev.org/c/openstack/kolla/+/884470 | 08:40 |
| frickler | yay | 08:41 |
| SvenKieske | mgoddard: frickler: maybe you can take a look at https://review.opendev.org/c/openstack/kolla-ansible/+/884460 there is an older abandoned changeset linked where you where involved, maybe we can fix it better this time? | 08:41 |
| SvenKieske | I don't know enough about ovn dns to have a real opinion there | 08:42 |
| frickler | SvenKieske: thx for spotting this, commented | 08:48 |
| mnasiadka | frickler: should we try to merge Mark's patches around MariaDB to get out of the ansible-core 2.14.2 lockout? ;-) | 08:49 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla/+/884208 and https://review.opendev.org/c/openstack/kolla-ansible/+/884182 | 08:50 |
| frickler | mnasiadka: I didn't look at those yet, trying to finally get some downstream work done this week. also monday is a bank holiday here | 08:51 |
| frickler | now that we've branched anyway, I guess this can wait a couple of days? | 08:51 |
| frickler | like, we won't do the final release until end of next week earliest? | 08:51 |
| mnasiadka | In theory 1st of June is the release deadline (final releases) | 08:52 |
| mnasiadka | But it can wait until Tuesday I guess | 08:52 |
| SvenKieske | that's thursday; I had an initial look at the mariadb stuff yesterday, I had the impression it was still being worked on, did that change? | 08:52 |
| mnasiadka | It's marked as ready for review and looks like working - we can run a recheck to check if it doesn't fail again | 08:53 |
| SvenKieske | ah right, I see | 08:53 |
| mnasiadka | the kolla patch is trivial | 08:53 |
| SvenKieske | will look into it, but have also some meetings today | 08:53 |
| mnasiadka | so that one might get in and get backported easily | 08:53 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: ceilometer: Use assert on checks for readability https://review.opendev.org/c/openstack/kolla-ansible/+/869593 | 08:58 |
| opendevreview | Merged openstack/ansible-collection-kolla stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/ansible-collection-kolla/+/884462 | 08:58 |
| opendevreview | Merged openstack/ansible-collection-kolla stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/ansible-collection-kolla/+/884463 | 08:58 |
| opendevreview | Merged openstack/ansible-collection-kolla master: Update master for stable/2023.1 https://review.opendev.org/c/openstack/ansible-collection-kolla/+/884464 | 08:58 |
| opendevreview | John Garbutt proposed openstack/kolla-ansible master: Update docs about dns_servers for OVN https://review.opendev.org/c/openstack/kolla-ansible/+/884460 | 09:08 |
| opendevreview | Matt Crees proposed openstack/kayobe master: 2023.1 RC1 - Sync KA feature flags & inventories https://review.opendev.org/c/openstack/kayobe/+/884471 | 09:34 |
| grami[m] | Hi All, Sorry to be a pain any chance someone could help get this pushed through it's the cherrypick of an already merged in master. https://review.opendev.org/c/openstack/kolla-ansible/+/883915 | 09:44 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/kolla-ansible/+/884465 | 09:50 |
| opendevreview | Matt Crees proposed openstack/kayobe master: Use OpenStack 2023.1 release https://review.opendev.org/c/openstack/kayobe/+/884472 | 09:51 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/kolla-ansible/+/884466 | 09:54 |
| opendevreview | Merged openstack/kolla-ansible master: Update master for stable/2023.1 https://review.opendev.org/c/openstack/kolla-ansible/+/884467 | 09:54 |
| frickler | SvenKieske: https://review.opendev.org/c/openstack/kolla/+/882924 is a good example to see how a lot of jobs are unstable. would IMO be a good start to look into these failures and try to make things more reliable | 09:58 |
| SvenKieske | this reads like a call to action/work, will look into it :D | 09:59 |
| opendevreview | Matt Crees proposed openstack/kayobe-config master: 2023.1 RC1 - Sync defaults with Kayobe https://review.opendev.org/c/openstack/kayobe-config/+/884473 | 10:01 |
| opendevreview | Merged openstack/kolla stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/kolla/+/884468 | 10:02 |
| opendevreview | Merged openstack/kolla stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/kolla/+/884469 | 10:03 |
| opendevreview | Merged openstack/kolla master: Update master for stable/2023.1 https://review.opendev.org/c/openstack/kolla/+/884470 | 10:04 |
| mnasiadka | SvenKieske: rocky ironic inspector failure is a good puzzle | 10:17 |
| opendevreview | Matt Crees proposed openstack/kayobe-config-dev master: 2023.1 RC1 - Sync defaults with Kayobe https://review.opendev.org/c/openstack/kayobe-config-dev/+/884477 | 10:18 |
| opendevreview | Matt Crees proposed openstack/kayobe master: 2023.1 RC1 - Sync KA feature flags & inventories https://review.opendev.org/c/openstack/kayobe/+/884471 | 12:07 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: opensearch: move to yum/apt repos https://review.opendev.org/c/openstack/kolla/+/883716 | 12:18 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: CI: Fix centos9 builds and drop unnecessary zuul vars https://review.opendev.org/c/openstack/kolla/+/884487 | 12:27 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: CI: Fix centos9 builds and drop unnecessary zuul vars https://review.opendev.org/c/openstack/kolla/+/884487 | 12:28 |
| SvenKieske | so e.g. looking at this failed job: https://zuul.opendev.org/t/openstack/build/0cbba93348664af287208b79e1d3fc0c/log/primary/logs/ansible/test-core-openstack it fails during volume creation from image, after over > 240 seconds. I found this old cinder bug, which seems to indicate that volumes from images are created using a tmp file (wtf?): https://bugs.launchpad.net/nova/+bug/1272076 | 12:36 |
| SvenKieske | the test volume seems to be 1GB, I don't know the size of the cirros image from the top of my head, but it should be something below 500MB? that shouldn't run for several minutes, even including a tmp file, no? | 12:37 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: drop ceph jobs ulimit config for EL9 https://review.opendev.org/c/openstack/kolla-ansible/+/884488 | 12:37 |
| SvenKieske | frickler: do you happen to know what storage backend our CI uses? RBD? | 12:38 |
| mnasiadka | SvenKieske: cephadm multinode jobs use ceph, single node jobs == no cinder | 12:43 |
| mnasiadka | SvenKieske: ah, you mean nodepool VMs storage backend? :) | 12:43 |
| mnasiadka | SvenKieske: https://0668fd1a5cb7e14c0a62-6f4fdaa50c9ffb2ee70643e96aea629f.ssl.cf2.rackcdn.com/884182/5/check/kolla-ansible-ubuntu-cephadm/0cbba93/secondary1/logs/kolla/cinder/cinder-volume.txt - this log shows rbd timeout, so might be our ceph is not functioning properly, although ceph -s claims it's healthy | 12:47 |
| opendevreview | Matt Crees proposed openstack/kayobe-config-dev master: 2023.1 RC1 - Sync defaults with Kayobe https://review.opendev.org/c/openstack/kayobe-config-dev/+/884477 | 12:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm - set rbd application on pool creation https://review.opendev.org/c/openstack/kolla-ansible/+/884489 | 12:49 |
| mnasiadka | ah, that failure is volume from image | 12:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Enable cinder coordination for cephadm jobs https://review.opendev.org/c/openstack/kolla-ansible/+/884492 | 13:06 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: CI: Fix centos9 builds and drop unnecessary zuul vars https://review.opendev.org/c/openstack/kolla/+/884487 | 13:07 |
| SvenKieske | mnasiadka: thanks for the hints..is that ceph a baremetal install or virtualized itself? | 13:08 |
| mnasiadka | SvenKieske: we deploy ceph and openstack on three VMs that we get from OpenDev CI, we use cephadm for deploying ceph - you can see the role under roles/cephadm/ | 13:09 |
| SvenKieske | yeah, currently looking at the ceph logs and config | 13:11 |
| SvenKieske | there's at least a warning: https://zuul.opendev.org/t/openstack/build/0cbba93348664af287208b79e1d3fc0c/log/primary/logs/ceph/ceph_s.txt that 3 pools don't have an application enabled, depending on which this is, this can be harmless. | 13:12 |
| SvenKieske | there are some interesting errors in the cephadm log, drilling down further | 13:14 |
| mnasiadka | SvenKieske: I just posted three patches to get the cephadm multinode jobs a bit better | 13:24 |
| mnasiadka | well, maybe two | 13:25 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla-ansible/+/884489 and https://review.opendev.org/c/openstack/kolla-ansible/+/884492 | 13:25 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Enable cinder coordination for cephadm jobs https://review.opendev.org/c/openstack/kolla-ansible/+/884492 | 13:28 |
| SvenKieske | nice | 13:29 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: opensearch: move to yum/apt repos https://review.opendev.org/c/openstack/kolla/+/883716 | 13:31 |
| SvenKieske | In general, speaking from a very high level perspective, I observed that we tend to have a rare(?) failure pattern where we don't iterate over all necessary elements because the lists of these elements are very decentralized organized. so we had - afaik two bugs fixed this week where services where missing from certain ansible loops. | 13:32 |
| * SvenKieske wonders if we could reorganize the code somewhat to make this stuff more robust, via jinja templates, or if this adds another layer of abstraction which makes everything harder to reason about. | 13:33 | |
| SvenKieske | I'm not yet convinced we can form decent subgroups of services which could be referenced by lists; for this to be worthwhile there needs to be at least one subset of services over which we loop in different code locations. | 13:35 |
| SvenKieske | stuff like this: https://review.opendev.org/c/openstack/kolla-ansible/+/884492/2/tests/run.yml#148; it's of course easy to miss a service file or config dir if you have various lists of services to which different tasks are applied | 13:37 |
| mnasiadka | true, that file is not ideal - but we could include files for different scenarios, and that would help with visibility | 13:39 |
| mnasiadka | and I think we agreed on a goal to make the CI more documented - so that would help | 13:40 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: CI: Fix centos9 builds and drop unnecessary zuul vars https://review.opendev.org/c/openstack/kolla/+/884487 | 13:40 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: opensearch: move to yum/apt repos https://review.opendev.org/c/openstack/kolla/+/883716 | 13:40 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible master: remove unnecessary comments https://review.opendev.org/c/openstack/kolla-ansible/+/884503 | 13:43 |
| SvenKieske | mhm, I don't know this: would it be worthwhile to enable ceph telemetry in our CI ceph? that would depend on how long these clusters typicially live I suppose. if they get immediately deleted there's no point in reporting crashes etc. (the reporting takes some time, but is configurable) | 13:47 |
| mnasiadka | SvenKieske: they shouldn't live more than 3 hours :) | 13:54 |
| mnasiadka | well, maybe more on upgrades | 13:55 |
| guesswhat[m] | hello, question, is nested virtualization supported for nova vms ? | 14:00 |
| mmalchuk | guesswhat[m] supported if configured | 14:06 |
| guesswhat[m] | mmalchuk: can You point me to the specific config? tried to seach nested in globals.conf, but no luck. Thanks | 14:06 |
| mmalchuk | guesswhat[m] this is not the kolla configuration, but the system | 14:08 |
| mmalchuk | https://docs.openstack.org/devstack/latest/guides/devstack-with-nested-kvm.html | 14:08 |
| mmalchuk | as an example ^^^ | 14:09 |
| mmalchuk | https://docs.openstack.org/nova/latest/admin/cpu-models.html#host-passthrough | 14:11 |
| guesswhat[m] | oh, thanks, wondering if triple nested virtualization is nogo... have a lab in kvm where I running openstack node and controllel and the same time I want to use kata containers on some nodes in nova, that would be triple nested virtualization :X | 14:11 |
| mmalchuk | about 'Host passthrough' in nova | 14:11 |
| SvenKieske | not a no go, I guess, but should be terribly slow, maybe even unusable. first thing to check if you have nested-vt flags turned on in uefi, it's often disabled by default. it can also be a bit of a security risk; there where some CVE only affecting nested-vt in kvm / linux kernel. | 14:14 |
| mmalchuk | guesswhat[m] https://docs.openstack.org/nova/latest/admin/configuration/hypervisor-kvm.html#host-configuration | 14:14 |
| mmalchuk | about nova configuration ^^^ | 14:14 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: CI: Fix centos9 builds and drop unnecessary zuul vars https://review.opendev.org/c/openstack/kolla/+/884487 | 14:15 |
| mmalchuk | guesswhat[m] https://docs.openstack.org/nova/latest/admin/configuration/hypervisor-kvm.html#nova-configuration | 14:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: opensearch: move to yum/apt repos https://review.opendev.org/c/openstack/kolla/+/883716 | 14:15 |
| mmalchuk | host-passthrough configuration ^^^ | 14:15 |
| guesswhat[m] | Thanks :) | 14:17 |
| guesswhat[m] | Too bad there are not many Nova drivers available, like free esxi is limited to 8vCPU, hyper-v is nogo, xen is not supported | 14:18 |
| guesswhat[m] | I have 64c/128t CPU server, I am running Proxmox, then 3 VMs, vyos, controller and node ( nested virtualization ) | 14:19 |
| guesswhat[m] | But trying Kata containers in Kubernetes deployed on Openstack via ClusterAPI requires baremetal = virtualization, so that would 3rd level | 14:20 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: CI: Fix centos9 builds and drop unnecessary zuul vars https://review.opendev.org/c/openstack/kolla/+/884487 | 14:33 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: [DNM] Revert "CI: Enable bare metal testing for Ubuntu upgrades" https://review.opendev.org/c/openstack/kayobe/+/884525 | 16:23 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: [DNM] Revert "CI: Enable bare metal testing for Ubuntu upgrades" https://review.opendev.org/c/openstack/kayobe/+/884525 | 18:21 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Fix passwords.yml permissions https://review.opendev.org/c/openstack/kolla-ansible/+/882100 | 18:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!