| admin1 | installing 25.0.0 on jammy, with osa + ceph, I get [ceph_client : Update Apt cache] 'http://download.ceph.com/debian-pacific jammy Release' .. while the docs did mention (Ubuntu 22.04 LTS (Jammy Jellyfish) (Experimental support in Yoga release) . | 07:13 |
|---|---|---|
| admin1 | what is the correct way to address that | 07:13 |
| noonedeadpunk | admin1: I think, you would need to set `ceph_pkg_source: distro` | 07:18 |
| admin1 | noonedeadpunk , thanks i will try | 07:19 |
| jrosser_ | morning | 07:41 |
| anskiy | jamesdenton: first one: https://bugs.launchpad.net/cloud-archive/+bug/1988270 | 07:56 |
| anskiy | that launchpad formatting made it look absolutely horrible :( | 07:56 |
| jrosser_ | andrewbonney: ^ you might want to read that too | 08:02 |
| andrewbonney | Ah, I think I've hit that, but only in a dev environment so far | 08:02 |
| anskiy | it didn't happen with libvirt 6, but now you get libvirt 8 from Yoga's UCA | 08:03 |
| noonedeadpunk | I wonder if we want to avoid having libvirt 8 in Yoga | 08:06 |
| noonedeadpunk | likely we can't do that though. As native I guess is 4.0.0? | 08:08 |
| noonedeadpunk | ah, no, it's 6.0.0 | 08:09 |
| noonedeadpunk | so we can fix that by forcing libvirt instalation of version 6.0.0 | 08:10 |
| noonedeadpunk | SO for nova MIN_LIBVIRT_VERSION is also 6.0.0 for Yoga | 08:12 |
| noonedeadpunk | with that I would likely propose to not install libvirt from UCA | 08:13 |
| anskiy | noonedeadpunk: yes, minimal is 6.0.0, but so it was for Xena too, with the same deprecation warning about removing support for everything below 7.0.0 for two releases straight :) | 08:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Bump OpenStack-Ansible Yoga https://review.opendev.org/c/openstack/openstack-ansible/+/855255 | 08:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Bump OpenStack-Ansible Yoga https://review.opendev.org/c/openstack/openstack-ansible/+/855255 | 08:16 |
| noonedeadpunk | From what I see, it's always 6.0.0 for Zed, so meh | 08:17 |
| noonedeadpunk | s/always/also | 08:17 |
| noonedeadpunk | I'm just not sure how to do that in a best way, needs some thinking through.... | 08:18 |
| noonedeadpunk | I wonder what we actually consume from UCA for source deployments... | 08:18 |
| anskiy | well, the other way around would be fixing the apparmor's profile: removing perfmon and bpf lines allow it to normally start. | 08:19 |
| anskiy | not sure about the consequences tho. I've googled some similar issues, totally unreleated to openstack, and it seems, that the problem is that this profile is intended to work with apparmors 3.x versions where support for those directives added (I might be wrong with my conclusions, as always, I'm just a selinux guy, so it's new for me :) ) | 08:21 |
| noonedeadpunk | I'm more concerned about consistency of deployments. As if you used 25.0.0 tag, you can get different libvirt versions for computes you deployed at the beginning comparing to later ones | 08:21 |
| noonedeadpunk | I bet that can influence live migrations and all sort of things | 08:22 |
| anskiy | by the ways, same goes for qemu versions, as it got bumped too, I guess | 08:22 |
| noonedeadpunk | yup, I believe it did | 08:22 |
| jrosser_ | do we need to speak to canoncial/uca people? they are in some of the openstack IRC/ML i think? | 08:22 |
| anskiy | and Nova has it's minimal version requirement for it too | 08:22 |
| noonedeadpunk | yeah, that's good idea | 08:24 |
| noonedeadpunk | though can't find uca teams IRC fast | 08:25 |
| anskiy | something makes me think, that UCAs are gonna either fix the profile and leaved the libvirt version as it is, or just file some other report for bumping apparmor's version | 08:25 |
| noonedeadpunk | fwiw UCA does not provide qemu | 08:29 |
| noonedeadpunk | at least according to https://openstack-ci-reports.ubuntu.com/reports/cloud-archive/yoga_versions.html | 08:29 |
| noonedeadpunk | So question - why except ovs/ovn bits do we set uca? | 08:29 |
| noonedeadpunk | I bet we needed newer libvirt some time ago, and that was motivation | 08:30 |
| noonedeadpunk | But as of today, we don't really need anything from there, except maybe fresh ovs versions | 08:31 |
| jrosser_ | yes i think that was it | 08:33 |
| jrosser_ | in the past (well, maybe as the ubuntu LTS release gets older and older) you need to make up the gap with UCA | 08:33 |
| jrosser_ | though it is kind of late now, as we have deployments already running Y | 08:34 |
| noonedeadpunk | and you're running libvirt 8.0.0? | 08:35 |
| noonedeadpunk | as if you still have 6.0.0 - then it's totally not too late :) | 08:35 |
| noonedeadpunk | ir maybe I'm terribly wrong and it was 8.0.0 from the beginning... And I'm just having misconception of what libvirt was in UCA at release | 08:37 |
| jrosser_ | i think ours is later because we ran into the vgpu mdev paths all being different | 08:39 |
| jrosser_ | `ii libvirt-daemon 8.0.0-1ubuntu7.1~cloud0` | 08:40 |
| jrosser_ | i think it is jamespage who deals with this stuff | 08:41 |
| jrosser_ | from canonical side | 08:41 |
| noonedeadpunk | oh, vgpu mdev - fun stuff.... | 08:44 |
| noonedeadpunk | btw Nvidia has _finally_ released docker images for their license server | 08:44 |
| noonedeadpunk | like 2 days ago or smth | 08:44 |
| admin1 | cephadm installs ceph on docker containers .. which is not accesible from ssh .. .. so our usual method of doing ssh to the mons don't work and get stuck on "ceph_client: verif ceph monitors are up" | 08:55 |
| jrosser_ | admin1: you don't have to use SSH to the mons | 08:59 |
| jrosser_ | see this https://docs.openstack.org/openstack-ansible-ceph_client/latest/config-from-file.html | 08:59 |
| jrosser_ | this would be the same situation as different teams managing openstack and ceph and there is no SSH allowed between them | 09:00 |
| jrosser_ | oh yes i saw the new licence server, not looked yet | 09:02 |
| noonedeadpunk | fwiw, I do think that it has a bug in it, as docker-compose up will stuck on first run. So does their qcow image. | 09:07 |
| noonedeadpunk | didn't dig too deep though, as jsut docker-compose stop/start proceeds and works | 09:07 |
| noonedeadpunk | qcow image now also jsut runs docker in it | 09:09 |
| jrosser_ | that makes some sense i guess | 09:20 |
| jrosser_ | i should take a look at this as i gave them a lot of grief about the original stuff | 09:20 |
| jrosser_ | anskiy: you might also be interested in this https://review.opendev.org/c/openstack/openstack-ansible/+/815284/1/inventory/group_vars/haproxy/keepalived.yml | 09:54 |
| jrosser_ | andrewbonney: ^ that is still WIP - not sure if we want to make progress on that? | 09:54 |
| jrosser_ | noonedeadpunk: we are running ipv4 + ipv6 external on keepalived here and have a ton of overrides needed for that | 09:58 |
| jrosser_ | might be worth looking at how we support that out of the box as you might want public ipv6 VIP even though the deployment is ipv4 internally | 09:59 |
| noonedeadpunk | We do same but almost no overrides | 10:00 |
| jrosser_ | interesting | 10:00 |
| noonedeadpunk | I think we do only define `extra_lb_tls_vip_addresses` | 10:01 |
| jrosser_ | we have independant check scripts and stuff as the v6 might work/break separately from the ipv45 | 10:01 |
| anskiy | jrosser_: do I need to somehow rework my change according to this one? | 10:01 |
| noonedeadpunk | ah, we don't do that.... As | 10:01 |
| jrosser_ | anskiy: no, that one is still work-in-progress, i just wondered if you were also aiming at doing ipv6 | 10:02 |
| noonedeadpunk | eventually, we add ipv6 to `vips_excluded:` | 10:02 |
| noonedeadpunk | under keepalived_instances | 10:02 |
| jrosser_ | as theres a few of us seem to be doing it but all differently | 10:02 |
| noonedeadpunk | I really can't recall why this is done. As trident was doing IPv6 and pushing some patches for it's support | 10:03 |
| anskiy | jrosser_: ah, no, not yet, I have many things, I've wanted to add even without it: `unicast_src_ip`, `unicast_peers`, `virtual_routes` and `virtual_rules`. I think, with my change, you can even add another instance if you want. | 10:04 |
| noonedeadpunk | and eventually anskiy patch would help a lot to reduce override to just add this vips_excluded key to keepalived_instances | 10:05 |
| noonedeadpunk | I would say that in our usecase if ipv6 is failing, then it will fail upstream, so keepalive can't help | 10:06 |
| *** dviroel|out is now known as dviroel | 11:23 | |
| jamesdenton | mornin' all | 12:24 |
| noonedeadpunk | \o/ | 12:26 |
| opendevreview | Merged openstack/openstack-ansible master: Add keepalived_instances_overrides for customizing keepalived_instances https://review.opendev.org/c/openstack/openstack-ansible/+/854370 | 12:33 |
| noonedeadpunk | jrosser_: I commented here https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/852588/2 | 14:00 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Add support to configure proxy-protocol-networks https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/854787 | 14:39 |
| prometheanfire | is aodh considered 'working' for the yoga release? venv can't install some package with version ===60.9, can't figure out which line that is in particular... | 14:40 |
| noonedeadpunk | prometheanfire: ah, good point. | 14:41 |
| noonedeadpunk | there's an easy workaround | 14:41 |
| * prometheanfire may be getting back into osa a bit more again, hopefully on a more permament basis | 14:42 | |
| noonedeadpunk | I've created a bug reprot to aodh for that but forgot to follow-up https://storyboard.openstack.org/#!/story/2010225 | 14:42 |
| noonedeadpunk | prometheanfire: eventually, I'd really loved to hear your opinion on that :D | 14:43 |
| prometheanfire | heh, I see tony commented :D | 14:43 |
| noonedeadpunk | ah, yes, see -2 now | 14:43 |
| noonedeadpunk | prometheanfire: as for workaround - just put `aodh_git_install_branch: 74eadfbd58359b7ebe9e1e40ae6b6ff245146bb8` to user_variables | 14:45 |
| prometheanfire | right | 14:45 |
| prometheanfire | I wonder if the problem is in adding the setuptools value | 14:45 |
| noonedeadpunk | the problem is that they added constrait to requirement | 14:46 |
| prometheanfire | ya | 14:46 |
| noonedeadpunk | ie `==` vs `===` imo | 14:46 |
| prometheanfire | that's what I mean | 14:46 |
| prometheanfire | setuptools===60.9.3;python_version=='3.9' | 14:46 |
| prometheanfire | https://github.com/openstack/aodh/compare/74eadfbd58359b7ebe9e1e40ae6b6ff245146bb8...539145cce1cbb1a862f135518f8b316a3fd0002a | 14:46 |
| noonedeadpunk | I will need to test though if `==` would be fine, but I guess it will | 14:47 |
| jamesdenton | 3 equal signs?? I have a hard enough time with = vs == | 14:48 |
| noonedeadpunk | prometheanfire: yeah, you indeed can use 849a0a0219e6f060efa1d0530afa2d6df2897501 which is later then the one I provided | 14:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Bump OpenStack-Ansible Yoga https://review.opendev.org/c/openstack/openstack-ansible/+/855255 | 14:53 |
| prometheanfire | that's a better one ya, the main change was the client | 14:54 |
| jrosser_ | kind of sad that https://review.opendev.org/c/openstack/aodh/+/842697 got approved by a single reviewer onto a stable branch | 14:57 |
| jrosser_ | prometheanfire: shouldnt aodh also be running some sort of requirements-check job to prevent this kind of error? | 14:57 |
| prometheanfire | lol | 14:59 |
| prometheanfire | yes, I should mention that, need to find the docs to point to for that too | 14:59 |
| prometheanfire | going to comment on the revert review | 14:59 |
| prometheanfire | https://review.opendev.org/c/openstack/aodh/+/852769 | 14:59 |
| prometheanfire | huh, aodh isn't in projects.txt | 15:05 |
| noonedeadpunk | pf.... | 15:06 |
| *** dviroel is now known as dviroel|lunch | 15:25 | |
| admin1 | is using letsencrypt as easy as setting the variable to true ? | 15:42 |
| anskiy | admin1: there is a doc for setting it up: https://docs.openstack.org/openstack-ansible/latest/user/security/ssl-certificates.html#certbot-certificates, it should work, but for some reason, I have patched haproxy_server role :) | 15:56 |
| jrosser_ | i wonder if that doc is up to date | 15:59 |
| jrosser_ | oh no its ok | 16:00 |
| anskiy | which I would actually need to discuss, but not today. | 16:01 |
| *** dviroel|lunch is now known as dviroel | 16:25 | |
| jrosser_ | what on earth does the "Which images do you use?" question mean in the user survey | 16:40 |
| jamesdenton | they're gonna need UUIDs sir | 16:42 |
| jrosser_ | no option for KVM (aarch64) weither | 16:47 |
| jrosser_ | *either | 16:47 |
| noonedeadpunk | is today last day of survey? | 17:22 |
| noonedeadpunk | As I clean forgot to fill it in | 17:23 |
| jrosser_ | yes today is the deadline | 17:24 |
| * noonedeadpunk grabs some beer from the freedge... | 17:34 | |
| noonedeadpunk | btw, backport of apt issue has merged to 2.13 | 17:38 |
| prometheanfire | is ovn considered the 'most fully supported option' for yoga plus? jamesdenton? | 18:16 |
| jamesdenton | probably not? | 18:16 |
| jamesdenton | it could definitely stand to be more tested IRL | 18:17 |
| prometheanfire | sounds about as expected | 18:17 |
| jamesdenton | we are not yet running any production workloads but there are some here who are | 18:17 |
| prometheanfire | doing a new cluster for work so building this out now, if it's good I'd like to use it | 18:18 |
| jamesdenton | it definitely works | 18:19 |
| jamesdenton | i don't know how "battle hardened" the OSA implementation is, though | 18:20 |
| jamesdenton | i am 90% through the docs for ML2/LXB->ML2/OVN migration | 18:21 |
| jrosser_ | imho OSA+OVN is good now for "lab kicking the tyres" | 18:43 |
| jrosser_ | and we need to do more of that to find whats working / broken | 18:43 |
| jrosser_ | jamesdenton: we have a lab with OVN + ASAP2 at the moment | 18:43 |
| jrosser_ | it some mixture of working and "special" | 18:43 |
| jamesdenton | oh neat | 18:43 |
| jrosser_ | for some reason the node thats supposed to be offloading is using more CPU than the one thats not | 18:44 |
| jamesdenton | i am most curious about the clustering | 18:44 |
| jamesdenton | how much more cpu? | 18:45 |
| jrosser_ | not sure really - grab andrewbonney when he's around for more info, i told him to take a look at your denver presentation for a baseline on what we should expect | 18:45 |
| jrosser_ | we also need to test multiple "network nodes" and see how that works for HA gateways | 18:46 |
| jamesdenton | not enough time to do all the things | 18:46 |
| jrosser_ | indeed | 18:46 |
| jrosser_ | we're focussing on multitenant ironic and OVN at the moment | 18:47 |
| jamesdenton | how's the ironic bit working out? were you able to iron out the NGS stuff? | 18:48 |
| jrosser_ | oh actually you might have an opinion on that | 18:49 |
| jrosser_ | this is unfortunate https://bugs.launchpad.net/openstack-ansible/+bug/1987405 | 18:50 |
| jrosser_ | we have a workaround of using a config override to put what should be in ml2_conf_genericswitch.ini into ml2_conf.ini so it doesnt matter that the NGS ini fine is not referenced | 18:52 |
| jrosser_ | theres also a bit of brokenness getting IPMI consoles wired into horizon as nova serial consoles but i think there is some progress fixing that | 18:53 |
| jamesdenton | ok, so when neutron-server is installed as uwsgi it uses a uwsgi role and there's no mechanism for custom ExecStart? | 18:55 |
| jamesdenton | i see, i guess i forgot about this: https://github.com/openstack/ansible-role-uwsgi | 18:56 |
| jrosser_ | yes, before we changed that there were some list of ini files based on what plugins you were using | 18:56 |
| jrosser_ | and thats not working any more | 18:56 |
| jrosser_ | well it's kind of more obtuse than that perhaps, because ExecStart is now about the uwsgi thing i think | 18:58 |
| jrosser_ | and i'm not sure i understand where the config file for the service is referenced any more | 18:59 |
| jamesdenton | This seems to imply that neutron-rpc-server actually loads them: https://docs.openstack.org/neutron/latest/admin/config-wsgi.html | 19:00 |
| jrosser_ | i wonder if there is no default for this https://opendev.org/openstack/networking-generic-switch/src/branch/master/networking_generic_switch/config.py#L47 | 19:06 |
| jrosser_ | so when unspecified it just doesnt load anything | 19:06 |
| jamesdenton | sounds reasonable. you are running a standard systemd neutron-rpc-server service, though, right? | 19:08 |
| jamesdenton | i would think the ExecStart would include generic switch ini | 19:08 |
| jrosser_ | rpc server is fine | 19:09 |
| jrosser_ | it's neutron-server thats not fine | 19:09 |
| jamesdenton | sure, i was just suggesting that the mechanism that load ml2_conf.ini might also be responsible for additional ml2_* | 19:10 |
| *** dviroel is now known as dviroel|afk | 19:47 | |
| *** dviroel|afk is now known as dviroel | 23:12 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!