| opendevreview | wangxiyuan proposed openstack/kolla-ansible master: Add openEuler Distro support https://review.opendev.org/c/openstack/kolla-ansible/+/830115 | 01:18 |
|---|---|---|
| jingvar | spatel: we have an addition tuninig option in ovn dbs like connection timeout | 05:15 |
| jingvar | but also we hava found another issue - high load cpu on DB and it cant'be multi threaded :) | 05:16 |
| jingvar | also we have to limit max builds | 05:18 |
| mnasiadka | morning | 07:12 |
| jingvar | morninig | 07:12 |
| mnasiadka | jingvar: check out those patches for python3-openvswitch, for getting OVN working better ;) | 07:12 |
| jingvar | mnasiadka: is it about db_timeout? | 07:13 |
| mnasiadka | what release are you running? xena? | 07:13 |
| jingvar | we've tested wallby and xena, current is xena | 07:14 |
| mnasiadka | so it should definitely help on timeouts/weird problems on Neutron side when connecting to NB DB | 07:14 |
| jingvar | we also tuned ovn-remote-probe-interval | 07:16 |
| jingvar | but it is not enough | 07:16 |
| jingvar | we have random failed tempest test with high concurency value | 07:17 |
| jingvar | And it correlated with raft mechanism | 07:17 |
| mnasiadka | how? | 07:21 |
| sri_ | morning | 07:22 |
| jingvar | we see that raft|INFO|received leadership transfer from | 07:22 |
| jingvar | always happens before failed tests | 07:22 |
| sri_ | when I enable octavia_network_type: "tenant" the deployment fails with "Destination directory /etc/dhcp does not exist" ! | 07:23 |
| mnasiadka | jingvar: OVN version? | 07:24 |
| jingvar | sorry I have a meet | 07:25 |
| jingvar | ovn-controller 21.06.1 | 07:28 |
| jingvar | ovn-nbctl 21.06.1 | 07:28 |
| mnasiadka | and OVS version? | 07:28 |
| jingvar | ovs-vsctl (Open vSwitch) 2.15.4 | 07:29 |
| opendevreview | Verification of a change to openstack/kolla stable/xena failed: Use python3-openvswitch from distro https://review.opendev.org/c/openstack/kolla/+/831416 | 07:48 |
| sri_ | mnasiadka: i've just find your comment on octavia_network_type: "tenant" : Should we add a note it only works when neutron_plugin_agent == 'ovs'? | 07:52 |
| sri_ | that's clarify the issue i am facing :) | 07:53 |
| mnasiadka | sri_: octavia_network_type: "tenant" should only be used for development or testing, please don't use it in production :) | 07:55 |
| mnasiadka | but yes, additional steps are needed to make this type of connection to work with OVN, so we should at least warn/fail that it works only when neutron_plugin_agent is ovs :) | 07:56 |
| frickler | zigo: mnasiadka: seems salsa is working again but the extrepo pages still 503 | 07:56 |
| mnasiadka | frickler: this is some mess... | 07:56 |
| sri_ | mnasiadka: Ack, this is a testing environment. | 08:09 |
| opendevreview | Merged openstack/kolla stable/wallaby: Use python3-openvswitch from distro https://review.opendev.org/c/openstack/kolla/+/831417 | 08:14 |
| zigo | frickler: <formorer> I will fix that as soon as I can, but not in the next hours | 08:19 |
| zigo | That's the current status ... | 08:20 |
| mnasiadka | sri_: just noticed your message with "destination directory does not exist" - that's not related to OVN for sure ;) | 08:20 |
| opendevreview | Merged openstack/kolla stable/victoria: Use python3-openvswitch from distro https://review.opendev.org/c/openstack/kolla/+/831418 | 08:23 |
| sri_ | mnasiadka: you're right, I am using neutron_plugin_agent: "openvswitch" | 08:23 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/ussuri: Use python3-openvswitch from distro https://review.opendev.org/c/openstack/kolla/+/831421 | 08:26 |
| sri_ | mnasiadka: logs https://paste.openstack.org/show/bLwkLtMdsTYf2WfHJ8OO/ | 08:26 |
| mnasiadka | sri_: it's rather a bug - it seems you don't have dhclient installed? | 08:27 |
| sri_ | I see, I am using centos 8 stream, dhclient is not installed | 08:31 |
| mnasiadka | sri_: anyway it's a bug, please raise it on bugs.launchpad.net/kolla-ansible | 09:02 |
| sri_ | mnasiadka: yes, installed the dhclient and deployment finished successfully | 09:03 |
| jingvar | Someone knows how to deploy ELK as external resource with itself keepalived, haproxy | 10:21 |
| jingvar | it is question | 10:22 |
| opendevreview | Mark Goddard proposed openstack/kayobe stable/victoria: Support configuring docker_registry network_mode https://review.opendev.org/c/openstack/kayobe/+/782703 | 10:26 |
| jingvar | there is a reference architecture where uses VirtualControlPlane | 10:29 |
| jingvar | it means OS serviced splited into several groups | 10:30 |
| jingvar | And they can work independently | 10:31 |
| hrw | Debian repos work again | 10:42 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/train: Build CentOS Stream 8 images https://review.opendev.org/c/openstack/kolla/+/831266 | 10:45 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/train: Build CentOS Stream 8 images https://review.opendev.org/c/openstack/kolla/+/831266 | 11:11 |
| opendevreview | Merged openstack/kayobe stable/victoria: Set correct gateway for the bifrost provision network https://review.opendev.org/c/openstack/kayobe/+/827745 | 11:20 |
| opendevreview | Merged openstack/kolla stable/xena: Use python3-openvswitch from distro https://review.opendev.org/c/openstack/kolla/+/831416 | 11:21 |
| opendevreview | Merged openstack/kolla stable/ussuri: Use python3-openvswitch from distro https://review.opendev.org/c/openstack/kolla/+/831421 | 11:22 |
| opendevreview | Merged openstack/kayobe stable/victoria: Set requirements branch for IPA build https://review.opendev.org/c/openstack/kayobe/+/828660 | 11:38 |
| opendevreview | Merged openstack/kolla stable/xena: Add qemu-img also in nova-libvirt image https://review.opendev.org/c/openstack/kolla/+/831411 | 11:52 |
| opendevreview | Merged openstack/kolla-ansible stable/xena: Fix hard coded OIDC response type https://review.opendev.org/c/openstack/kolla-ansible/+/831261 | 11:52 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Fix hard coded OIDC response type https://review.opendev.org/c/openstack/kolla-ansible/+/831262 | 11:52 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/train: Build CentOS Stream 8 images https://review.opendev.org/c/openstack/kolla/+/831266 | 12:35 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/train: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/831520 | 12:35 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/train: Build CentOS Stream 8 images https://review.opendev.org/c/openstack/kolla/+/831266 | 12:48 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla-ansible master: rabbitmq: add node parameter in rabbitmq_user call https://review.opendev.org/c/openstack/kolla-ansible/+/817190 | 13:01 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/train: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/831520 | 13:11 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/train: Build CentOS Stream 8 images https://review.opendev.org/c/openstack/kolla/+/831266 | 13:11 |
| zigo | frickler: extrepo should be fixed now, can you recheck ? | 13:12 |
| frickler | zigo: yes, I checked already earlier when hrw mentioned this, lgtm, thanks for the update | 13:24 |
| zigo | :) | 13:24 |
| mnasiadka | hrw: not that I anticipate breakage again - but can we think about having an infra mirror? | 13:35 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Stop using kolla-ansible bootstrap-servers https://review.opendev.org/c/openstack/kayobe/+/824998 | 13:40 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: CI: stop using zuul as kayobe_ansible_user in TLS jobs https://review.opendev.org/c/openstack/kayobe/+/831532 | 13:40 |
| spatel | jingvar if you around then just wanted to let you know neutron_ovn_metadata_agent issue has been resolved :) | 14:05 |
| spatel | as you said its related to timeout and solution is to add ovsdb_timeout=30 baed on https://bugs.launchpad.net/networking-ovn/+bug/1953591 | 14:06 |
| hrw | zigo: how much space debian-openstack repo takes? | 14:17 |
| SvenKieske | hey there, any feedback/review would be appreciated, I'm really eager to get this merged :) https://review.opendev.org/c/openstack/kolla-ansible/+/758576 thanks! | 14:27 |
| mnasiadka | #startmeeting kolla | 15:00 |
| opendevmeet | Meeting started Wed Mar 2 15:00:01 2022 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 15:00 |
| mnasiadka | #topic rollcall | 15:00 |
| mnasiadka | o/ | 15:00 |
| yoctozepto | o/ | 15:00 |
| hrw | /._.\ | 15:00 |
| yoctozepto | \._./ | 15:00 |
| yoctozepto | /*-*\ | 15:00 |
| yoctozepto | /^_*\ | 15:01 |
| mnasiadka | #topic agenda | 15:03 |
| mnasiadka | * Announcements | 15:03 |
| mnasiadka | * Review action items from the last meeting | 15:03 |
| mnasiadka | * CI status | 15:03 |
| mnasiadka | * Release tasks | 15:03 |
| mnasiadka | * Current cycle planning | 15:03 |
| mnasiadka | * Additional agenda (from whiteboard) | 15:03 |
| mnasiadka | * Open discussion | 15:03 |
| mnasiadka | #topic Announcements | 15:03 |
| mnasiadka | Cycle highlights merged | 15:03 |
| mgoddard | \o | 15:04 |
| mnasiadka | Feature freeze in two weeks | 15:04 |
| mnasiadka | #topic Review action items from the last meeting | 15:04 |
| mnasiadka | mnasiadka to triage security bugs and update them with resolution plan (if needed) | 15:05 |
| mnasiadka | hrw to discuss with pynacl upstream to release binary wheel of 1.4.0 for aarch64 | 15:05 |
| mnasiadka | mnasiadka to post patch for cycle highlights | 15:05 |
| mnasiadka | I've done cycle highlights, no progress on sec bugs | 15:05 |
| hrw | pynacl needs to be sorted other way | 15:05 |
| mnasiadka | #action mnasiadka to triage security bugs and update them with resolution plan (if needed) | 15:05 |
| hrw | I just lack time and motivation to work on it | 15:06 |
| mnasiadka | should we still have it as an action item? | 15:06 |
| mnasiadka | I guess it makes no sense to talk about this every week ;) | 15:07 |
| mnasiadka | #topic CI Status | 15:07 |
| hrw | sort of | 15:07 |
| mnasiadka | one more thing | 15:08 |
| zigo | hrw: Just one release, or the full of osbpo ? | 15:08 |
| hrw | zigo: both? | 15:08 |
| mnasiadka | hrw: do we have a bug raised or something? we could just tie it to Yoga release milestone | 15:08 |
| hrw | mnasiadka: it was Wallaby | 15:08 |
| hrw | mnasiadka: just found during Yoga cycle by someone | 15:09 |
| hrw | have to find details (but above) | 15:09 |
| mnasiadka | hrw: anyway, I guess we should have a bug | 15:09 |
| zigo | hrw: 16 GB for all releases since jessie-backports-liberty, not even 800 MB for a single release. | 15:09 |
| hrw | zigo: thanks | 15:09 |
| mnasiadka | so, what's the CI status? apart Debian - seems green | 15:09 |
| yoctozepto | yeah, green | 15:09 |
| mnasiadka | hrw: does that mean we could try to mirror it in infra? | 15:10 |
| zigo | hrw: If you need to have a local cache, there's rsync which is publicly opened. | 15:10 |
| hrw | mnasiadka: I prefer to have numbers when infra team asks | 15:10 |
| mnasiadka | sure | 15:10 |
| hrw | mnasiadka: but we do not publish debian/binary images so it is not so important to mirror imho | 15:11 |
| hrw | ok, back to meeting? | 15:11 |
| yoctozepto | but we also use it for source | 15:11 |
| mnasiadka | hrw: so tell me why debian-source is failing without extrepo | 15:11 |
| yoctozepto | I believe it's bumping, e.g., ovs for us | 15:11 |
| hrw | because some of python stuff | 15:11 |
| hrw | it is like with RDO/UCA - we get updated stuff from there | 15:11 |
| mnasiadka | well, just when it fails - and debian jobs are voting - we get a bit of a hell | 15:12 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/train: Build CentOS Stream 8 images https://review.opendev.org/c/openstack/kolla/+/831266 | 15:12 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/train: CI: Fix epel url override https://review.opendev.org/c/openstack/kolla/+/831546 | 15:12 |
| yoctozepto | I wonder if debian general backports are not enough to get "newer stuff" | 15:12 |
| hrw | I would prefer them to be | 15:13 |
| hrw | zigo: your opinion? | 15:13 |
| mnasiadka | anyway, before we get it back to voting - I would prefer we have a reliable mirror/other solution - so one server downtime doesn't stop us merging patches | 15:14 |
| frickler | also, osbpo was available all the time afaict, so mirroring it wouldn't have gotten us over the extrepo issue | 15:14 |
| frickler | the source of extrepo repo definitions is another spof | 15:14 |
| hrw | #action Enable osbpo in Debian APT sources, abandon extrepo command use then? | 15:15 |
| mnasiadka | probably makes sense | 15:15 |
| hrw | just like UCA which is always enabled | 15:15 |
| mnasiadka | anyway, let's move on | 15:15 |
| frickler | and then in addition we can set up a mirror in infra | 15:15 |
| mnasiadka | hrw: will you take that action, or is it an unassigned action waiting for better times? :-) | 15:16 |
| hrw | mnasiadka: this one I can take. Infra stuff may not | 15:16 |
| mnasiadka | yoctozepto/frickler: willing to help with infra stuff? | 15:17 |
| frickler | I'll look into the infra part | 15:17 |
| mnasiadka | great | 15:17 |
| yoctozepto | :-) | 15:18 |
| mnasiadka | #topic Release tasks | 15:18 |
| mnasiadka | Cycle highlights merged | 15:18 |
| mnasiadka | no release tasks for this week | 15:18 |
| mnasiadka | feature freeze in 2 weeks time if my math is correct | 15:19 |
| mnasiadka | #topic Current cycle planning | 15:19 |
| mnasiadka | let's look at the whiteboard | 15:20 |
| hrw | we should look at open patches and mark which ones to merge, which to postpone? | 15:20 |
| mnasiadka | most probably, let's see | 15:20 |
| mnasiadka | mgoddard/headphoneJames: Secure RBAC - is it done for this cycle? | 15:21 |
| mgoddard | no | 15:21 |
| mgoddard | I don't see keystone changing their default at this point in the cycle though | 15:22 |
| mnasiadka | none of the changes on the whiteboard merged | 15:22 |
| mnasiadka | do we mark them as RP+1? | 15:22 |
| mgoddard | in which case there would be no need for us to take action | 15:22 |
| mnasiadka | so basically we can wait with those patches until Z | 15:22 |
| mnasiadka | so no reason to do RP+1 | 15:22 |
| yoctozepto | thankfully | 15:22 |
| mnasiadka | the OpenSearch stuff is somewhat in progress, but I doubt we'll be able to fit it in this cycle - since only a WIP of the Kolla patch is posted | 15:23 |
| mgoddard | they're basically ready to merge, I'm just not convinced the community has consensus on how to do it just yet, so I'd prefer to wait | 15:24 |
| mnasiadka | Let's Encrypt - we've had some... misunderstandings and complications on last meeting | 15:24 |
| mgoddard | (re RBAC) | 15:24 |
| mnasiadka | I didn't have time to look into it | 15:24 |
| mgoddard | no update since then | 15:25 |
| mnasiadka | ok, I'll try to look into it and bring some thoughts to the next meeting | 15:25 |
| mnasiadka | what about Podman? | 15:25 |
| mgoddard | mnasiadka: I don't see LE merging this cycle. should we focus elsewhere in these last two weeks? | 15:26 |
| mnasiadka | mgoddard: I guess it makes sense | 15:26 |
| mnasiadka | (halomiva,hinermar) structure of podman testing and moving baremetal role to openstack collection | 15:26 |
| zigo | hrw: No, general backports aren't enough, you could setup the osbpo repos without extrepo though, but I don't think the salsa thingy that happened last 3 days will occur again. | 15:26 |
| hrw | zigo: ok | 15:27 |
| zigo | The solution *IS* reliable, it's just a one time missfortune. | 15:27 |
| zigo | I'm working on having extrepo to work offline with my colleague, so it wont have to download from salsa. | 15:27 |
| zigo | That's IMO the better solution. | 15:27 |
| mnasiadka | I don't think we have halomiva or hinermar | 15:27 |
| yoctozepto | yeah, we don't | 15:28 |
| yoctozepto | podman is unlikely to happen imho | 15:28 |
| yoctozepto | need to postpone | 15:28 |
| mnasiadka | ok then | 15:28 |
| mgoddard | how about systemd? | 15:28 |
| yoctozepto | no idea, I did not have time to follow | 15:28 |
| mnasiadka | would be nice to get sytemd merged, but it's so late in the cycle - we might not have enough time to test it thoroughly... | 15:29 |
| mgoddard | I agree it's too late for podman, (and said before that would be ambitious). A concerted effort could get systemd across the line, if we chose to do so | 15:29 |
| mnasiadka | how much of an effort is to get sytemd working? can somebody look into it and try to push it over the line? ;-) | 15:30 |
| mgoddard | I wonder if we could add another class that keeps the existing behaviour | 15:31 |
| mgoddard | then make systemd optional | 15:31 |
| frickler | is that just https://review.opendev.org/c/openstack/kolla-ansible/+/816724 or is there more to it? | 15:31 |
| mgoddard | frickler: that's "it", where it == open heart surgery | 15:31 |
| yoctozepto | indeed | 15:32 |
| mnasiadka | well, systemd as optional probably would be a wise choice for now | 15:32 |
| mnasiadka | if we can progress that in the next week or two, would be nice | 15:32 |
| mnasiadka | Rocky Linux - waiting for a new DIB release to get nodepool nodes running | 15:33 |
| mnasiadka | but now (thanks mgoddard!) I need to move some of the patch lines to kolla collection :) | 15:33 |
| mgoddard | ha | 15:34 |
| mnasiadka | mgoddard: how is the collection work going? | 15:34 |
| mgoddard | lots of patches ready for review | 15:34 |
| mgoddard | we have the collection installing for both kolla-ansible and kayobe now | 15:34 |
| mgoddard | but currently just using the unchanged baremetal role | 15:35 |
| mgoddard | next patches split it up into more sensible roles | 15:35 |
| mgoddard | and add some CI | 15:35 |
| mgoddard | this would be a good one: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/820907 | 15:35 |
| mnasiadka | I see yoctozepto has already reviewed | 15:36 |
| yoctozepto | yeah, bumped to +2 now | 15:36 |
| mnasiadka | so merging, looks ok | 15:36 |
| mnasiadka | so only two changes left from the whiteboard | 15:37 |
| mgoddard | check again :) | 15:37 |
| mnasiadka | mgoddard: do we need a review dashboard for kolla collection, or just include it in kolla-ansible one? | 15:37 |
| mgoddard | I've listed all patches required to merge this: https://review.opendev.org/c/openstack/kayobe/+/824998 | 15:38 |
| hrw | is git-review working for someone? | 15:38 |
| mgoddard | was earlier | 15:39 |
| yoctozepto | hrw: it worked today | 15:39 |
| mgoddard | mnasiadka: maybe in kolla-ansible? | 15:39 |
| hrw | thx | 15:39 |
| mnasiadka | mgoddard: I think that makes more sense, I'll update the dashboard | 15:39 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: Debian: always enable osbpo repositories https://review.opendev.org/c/openstack/kolla/+/831548 | 15:39 |
| hrw | finally | 15:39 |
| mnasiadka | #action mnasiadka to update kolla review dashboard with kolla collection | 15:39 |
| mnasiadka | ok, I'll make time to review all those patches, and post also one for Rocky Linux ;) | 15:40 |
| mnasiadka | what about libvirt on the host? | 15:40 |
| mnasiadka | two patches ready to review? | 15:41 |
| mgoddard | yup | 15:41 |
| mnasiadka | ok then, will also review those | 15:42 |
| mnasiadka | and try to go through Kayobe ones | 15:42 |
| mnasiadka | we need to notify fellow Kayobians that they should update all their patches that need to be merged in Y | 15:42 |
| mgoddard | UPDATE YOUR PATCHES | 15:42 |
| mgoddard | please | 15:42 |
| mnasiadka | ok | 15:43 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 15:43 |
| mnasiadka | (yoctozepto) keystone changes that will affect us | 15:44 |
| mnasiadka | will there be any? | 15:44 |
| mnasiadka | yoctozepto: wake up :) | 15:45 |
| yoctozepto | no | 15:45 |
| yoctozepto | seems not | 15:46 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: libvirt: deploy libvirt on the host https://review.opendev.org/c/openstack/kayobe/+/825359 | 15:46 |
| yoctozepto | next topic please | 15:46 |
| mnasiadka | (yoctozepto) ironic-(i)pxe renaming for sanity | 15:46 |
| yoctozepto | yeah, so we have the above profanity calling tftp - pxe, and http - ipxe | 15:46 |
| yoctozepto | I wonder if it's worth renaming the containers to reflect their actual role | 15:47 |
| yoctozepto | the only issue is we don't support renames - we would need to clean up old names | 15:47 |
| mgoddard | I see your point | 15:47 |
| mgoddard | unsure about cost/benefit | 15:48 |
| yoctozepto | me too | 15:48 |
| mgoddard | tftp/pxe - meh | 15:48 |
| mnasiadka | mgoddard: ansible-collection-kolla is not being synced to github? | 15:48 |
| yoctozepto | yeah, it's worse for ipxe tbh | 15:48 |
| mgoddard | ipxe/http - more of an issue since it's used for deployment | 15:48 |
| yoctozepto | precisely, it's for direct deploy | 15:48 |
| mgoddard | if someone wants to write the code I'll review it | 15:49 |
| yoctozepto | but when renaming it's similar cost for one as for two | 15:49 |
| yoctozepto | ok, I'll add to my own wishlist | 15:49 |
| mnasiadka | going through all that fuss just to rename, I'm not convinced - but if somebody wants to write the code - so be it | 15:49 |
| mnasiadka | #topic Open discussion | 15:50 |
| mnasiadka | Anything? Anyone? | 15:50 |
| yoctozepto | I'm almost finished with https://review.opendev.org/c/openstack/kolla-ansible/+/644271 | 15:50 |
| mgoddard | nice work on that yoctozepto | 15:51 |
| headphoneJames | Can we set a time to do a live discussion for let's encrypt? | 15:51 |
| mgoddard | now if you could get all of my other unfinished patches over the line that would be great | 15:51 |
| yoctozepto | xD | 15:52 |
| yoctozepto | we should merge this https://review.opendev.org/c/openstack/kolla-ansible/+/831055 | 15:52 |
| yoctozepto | as it's fixing the defaults | 15:52 |
| mgoddard | headphoneJames: let's discuss | 15:52 |
| yoctozepto | the tenks testing revealed this issue early | 15:52 |
| yoctozepto | I love it when testing actually catches bugs ;p | 15:52 |
| mgoddard | happens occasionally | 15:52 |
| mgoddard | headphoneJames: next week? | 15:54 |
| headphoneJames | Sure. This time or later works. | 15:55 |
| mgoddard | Same time as this meeting, on Monday? | 15:55 |
| mgoddard | How does that work for everyone? | 15:55 |
| headphoneJames | Works for me | 15:55 |
| mgoddard | Who wants to attend? | 15:55 |
| mgoddard | o/ | 15:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Add Rocky Linux support as Host OS https://review.opendev.org/c/openstack/kolla-ansible/+/815104 | 15:56 |
| mnasiadka | I can attend | 15:56 |
| yoctozepto | let me check | 15:56 |
| yoctozepto | works for me | 15:56 |
| mgoddard | ok then | 15:57 |
| mgoddard | I think we have quorum | 15:57 |
| headphoneJames | Great | 15:57 |
| headphoneJames | I'll post a zoom link on the channel on Monday 15utc | 15:57 |
| mnasiadka | frickler: you also had some opinion on the LE implementation? | 15:58 |
| mgoddard | I'll make a meeting invitation | 15:58 |
| headphoneJames | Thanks | 15:58 |
| frickler | as long as you talk about HTTP-01, I don't care much | 15:59 |
| yoctozepto | thanks mnasiadka for chairing | 16:00 |
| * yoctozepto off | 16:00 | |
| mnasiadka | We just need to implement HTTP-01 in a way, that DNS-01 can be added easily after. | 16:01 |
| mnasiadka | #endmeeting | 16:01 |
| opendevmeet | Meeting ended Wed Mar 2 16:01:09 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:01 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-03-02-15.00.html | 16:01 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-03-02-15.00.txt | 16:01 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-03-02-15.00.log.html | 16:01 |
| mnasiadka | thanks for the meeting! | 16:01 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/train: Build CentOS Stream 8 images https://review.opendev.org/c/openstack/kolla/+/831266 | 16:05 |
| jingvar | spatel: ovsdb_timeout=30 , It is only first issue with ovn :) | 16:35 |
| spatel | jingvar hey! it turn out it didn't resolve my issue :( | 16:35 |
| spatel | i have noticed some nodes are working but not all of them.. could be timing issue. so i am doing cronjob restart to bounce it periodically to hide my issue. | 16:36 |
| jingvar | there is a problem when starting db snapshoting | 16:37 |
| spatel | hmm what do you mean snapshotting ? | 16:38 |
| jingvar | db has limit 10Mb and a timer to do snapshot, when it happens, initialasing leader transfer | 16:39 |
| jingvar | and neuron agent don't like it | 16:39 |
| jingvar | we tuned some options | 16:41 |
| jingvar | and have Rally - NovaServers.boot_server_and_attach_interface failure_rateFailure rate criteria 0.00% <= 4.69% <= 0.00% - Failed | 16:42 |
| jingvar | 64 threads Iterations: 128 Failures: 6 | 16:43 |
| jingvar | with tempest the same behviauor | 16:43 |
| spatel | hmm very interesting | 16:50 |
| spatel | i have only 30 total compute nodes so and currently zero vms running on it so i doubt its under heavy load or anything | 16:51 |
| jingvar | I thhink 30 node with 1 vm per node and low user activity dosen;t bring a lot of issues :) | 16:55 |
| jingvar | we have set of test for cloud, we used OVS and just run it on OVN :) | 16:57 |
| jingvar | to feel the difference | 16:57 |
| jingvar | of course we raiesed api haprxy timeots | 16:58 |
| opendevreview | Merged openstack/ansible-collection-kolla master: CI: add initial deploy jobs https://review.opendev.org/c/openstack/ansible-collection-kolla/+/820907 | 17:00 |
| opendevreview | Merged openstack/ansible-collection-kolla master: baremetal: some small tidy ups https://review.opendev.org/c/openstack/ansible-collection-kolla/+/821003 | 17:00 |
| frickler | could someone mark https://blueprints.launchpad.net/kolla/+spec/libvirt-exporter as done? 642095 and 643568 are merged | 17:06 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/train: CI: Add CentOS Stream 8 jobs https://review.opendev.org/c/openstack/kolla-ansible/+/831265 | 17:07 |
| frickler | also, are those changes suitable for backporting? | 17:07 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/train: CI: Fix epel url override https://review.opendev.org/c/openstack/kolla/+/831546 | 17:07 |
| opendevreview | Mark Goddard proposed openstack/kolla stable/train: Build CentOS Stream 8 images https://review.opendev.org/c/openstack/kolla/+/831266 | 17:07 |
| mgoddard | frickler: done | 17:09 |
| jingvar | how can I check that node in monitring host group , I will use custom keepalived.conf | 17:16 |
| opendevreview | Merged openstack/kayobe master: Bump up Ansible supported versions https://review.opendev.org/c/openstack/kayobe/+/826745 | 17:34 |
| spatel | jingvar any other suggestion to fix this issue or any timeout etc.. i should adjust? | 18:00 |
| jingvar | https://www.openvswitch.org/support/ovscon2019/day1/1436-OVSCON-Nouman.pdf | 18:03 |
| jingvar | I have to collect current tune options that we used, I think tomorrow | 18:05 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Use Tenks in Ironic job https://review.opendev.org/c/openstack/kolla-ansible/+/644271 | 18:05 |
| jingvar | I'll split vip adresses into primary host groups - controllers, network, monitor etc | 18:08 |
| jingvar | services will have vip address from placed host group | 18:09 |
| jingvar | monitor separeted into mon and log nodes | 18:10 |
| jingvar | probably log nodes will be separated into weekly and long term | 18:11 |
| opendevreview | Merged openstack/kolla stable/train: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/831520 | 18:18 |
| yoctozepto | mnasiadka, mgoddard: in case you are not following #opendev - gitea was serving stale content for kolla-ansible repo | 18:40 |
| yoctozepto | :-( | 18:41 |
| admiyo | sudo mount | grep /run/openvswitch | wc -l | 19:19 |
| admiyo | 2047 | 19:19 |
| admiyo | is that normal? I am going to go out on a limb and say I think that is not normal | 19:19 |
| admiyo | so...running a kolla all in one (aarch64, built me own containers) is tripping over the toolbox container. | 19:20 |
| admiyo | quay.io/openstack.kolla/ubuntu-source-kolla-toolbox:master | 19:22 |
| admiyo | it keeps getting the amd64 not aarch64 container, I think | 19:22 |
| admiyo | ~$ sudo docker image inspect quay.io/openstack.kolla/ubuntu-source-kolla-toolbox:master | grep Arch | 19:23 |
| admiyo | "Architecture": "amd64", | 19:23 |
| admiyo | I thought docker manifest was supposed to show the architecture.... | 19:24 |
| admiyo | I built kolla/ubuntu-source-kolla-toolbox:13.0.1 but it is pulling from quay.io and running quay.io/openstack.kolla/ubuntu-source-kolla-toolbox:master | 19:30 |
| admiyo | what am I doing wrong? | 19:30 |
| admiyo | Do I need to set something in globals to tell it to use only containers I built myself? | 19:31 |
| admiyo | kolla keeps trying to run containers like quay.io/openstack.kolla/ubuntu-source-fluentd:13.0.1 instead of kolla/ubuntu-source-fluentd:13.0.1 what do I do to force it to look local | 19:58 |
| mmskv | Hi, I get an issue while deploying monasca. Monasca-ui horizon plugin does not work with the following error: AttributeError: 'Settings' object has no attribute 'DASHBOARDS'. How do I fix it? | 20:18 |
| priteau | admiyo: you would usually configure a docker registry and push your locally built image to it. So your image would be REGISTRY_IP:REGISTRY_PORT/kolla/ubuntu-source-fluentd:13.0.1 | 21:15 |
| admiyo | priteau, maybe, but the docs don't say that is necessary. I have done that on other occasions, and I will go to that if it turns out it is the only way to get the namespace right | 21:16 |
| priteau | if you have a local build on each host, maybe you can run this directly, but you may have to customise docker_registry and docker_namespace variables | 21:18 |
| admiyo | so, yeah, the goal is to set up artifactory for our lab and push all artifacts for the vm, baremetal image and container images there as well as other use cases. I was just tryingto not have to do that first in order to get this cluster up and running | 21:19 |
| priteau | you can set up a regular docker registry really easily. | 21:20 |
| admiyo | and this is an all-in-one for the control plane. I am not a fan of useing 30% of the compute power in a 10 node cluster as overhead.... | 21:20 |
| admiyo | yeah, I know, and I've done it before, but it is cutting corners unless you have a decent X509 mechanism in place...insecure | 21:20 |
| admiyo | which is just another piece I was hoping to skip | 21:21 |
| admiyo | the fact that docker does not confirm you need an aarch64 binary container image and sends an amd64 for toolbox and fluent really made my morning interesting | 21:21 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Use Tenks in Ironic job https://review.opendev.org/c/openstack/kolla-ansible/+/644271 | 22:02 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!