| *** dviroel|ruck is now known as dviroel|out | 00:04 | |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Change location of ipaddr filter https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/831526 | 00:04 |
|---|---|---|
| *** aussielunix is now known as aussielunix_ | 01:20 | |
| *** aussielunix_ is now known as aussielunix | 01:23 | |
| *** anbanerj is now known as frenzyfriday | 07:38 | |
| *** arxcruz|off is now known as arxcruz | 07:47 | |
| Brace | I've lost VM networking on my openstack cluster, any idea what this error might mean - https://pastebin.com/EfW5WRsk | 09:08 |
| Brace | So far I've tried to restart the various neutron components and also rebooted (separately) all of my controller nodes | 09:08 |
| noonedeadpunk | Brace: have an idea | 09:12 |
| noonedeadpunk | Brace: try this out https://paste.openstack.org/show/bY235whPe5LKkFFzo6pn/ | 09:13 |
| noonedeadpunk | You might want to apply smth simmilar to cinder_volume_init_overrides and nova_compute_init_overrides as well | 09:14 |
| Brace | noonedeadpunk: I'll try that out, thank you so much! | 09:19 |
| *** arxcruz is now known as arxcruz|brb | 09:20 | |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_nova master: Add configuration option for heartbeat_in_pthread https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/833236 | 10:41 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_neutron master: Add configuration option for heartbeat_in_pthread https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/833237 | 10:42 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_cinder master: Add configuration option for heartbeat_in_pthread https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/833238 | 10:42 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_cinder master: Add configuration option for heartbeat_in_pthread https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/833238 | 11:00 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_nova master: Add configuration option for heartbeat_in_pthread https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/833236 | 11:00 |
| *** dviroel|out is now known as dviroel|ruck | 11:09 | |
| opendevreview | Merged openstack/openstack-ansible-tests master: Add ansible.utils collection requirement https://review.opendev.org/c/openstack/openstack-ansible-tests/+/833596 | 11:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Use separate tmp directory https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/831550 | 11:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Add galera_data_dir variable https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/831552 | 11:28 |
| gokhani | Hi folks, How can we use Mellanox infiniband ports with OSA? | 11:33 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_neutron master: Add parameters to limit the number of DHCP or L3 agents https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/833769 | 11:34 |
| gokhani | Hi folks, how can we use Mellanox İnfiniband ports with OSA? | 11:35 |
| jrosser | gokhani: that can mean several things but you could look at this https://satishdotpatel.github.io/HPC-on-openstack/ | 11:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update MariDB version to 10.6.7 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/833259 | 11:37 |
| noonedeadpunk | gokhani: yep, I did that and it was nasty :) | 11:39 |
| noonedeadpunk | gokhani: the main thing is that you will need to use sr-iov to pass ib devices inside containers | 11:39 |
| noonedeadpunk | since they can't be bridged | 11:39 |
| noonedeadpunk | if you're talking about IB for storage network as an example | 11:39 |
| gokhani | jrosser: thanks I will try | 11:41 |
| *** arxcruz|brb is now known as arxcruz | 11:42 | |
| gokhani | noonedeadpunk: yes firstly I tried to use IB for storage network | 11:42 |
| jrosser | gokhani: as noonedeadpunk says it depends what you want | 11:42 |
| jrosser | becasue that link i give you is nothing to do with storage, just VM<>VM MPI workloads | 11:42 |
| noonedeadpunk | hm, I think I need to document that :) | 11:43 |
| gokhani | noonedeadpunk: I need your documentation :) preliminary comments also will also help | 11:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add doc how to pass SR-IOV inside containers https://review.opendev.org/c/openstack/openstack-ansible/+/833775 | 12:11 |
| opendevreview | Merged openstack/openstack-ansible-lxc_container_create master: Change location of ipaddr filter https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/833118 | 12:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Add galera_data_dir variable https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/831552 | 12:27 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Change location of ipaddr filter https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/831530 | 12:42 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone stable/xena: add oauth support https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/833552 | 12:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Add reference_group support to provider_networks module https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/829741 | 13:19 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_neutron master: Update Mellanox ASAP^2 Documentation https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/833795 | 14:05 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Add reference_group support to provider_networks module https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/829741 | 14:15 |
| *** dviroel|ruck is now known as dviroel|ruck|mtg | 14:16 | |
| spatel | noonedeadpunk around? i have dumb question related kubernetes | 14:18 |
| spatel | I have deploy k8s cluster on my lab openstack using magnum. I don't have octavia lb. How do i expose my hello-world ngnix to outside world? | 14:20 |
| spatel | are there anyway or octavia is hard stop here? | 14:20 |
| gokhani | noonedeadpunk: one of my customer requested using zfs storage for their HPC clusters. Can I use zfs pool which shared with nfs for vm disks or glance images? | 14:22 |
| noonedeadpunk | um, yes, nfs can be used as backend for cinder and glance and nova | 14:24 |
| gokhani | spatel: https://docs.openstack.org/magnum/latest/user/#ingress_controller you can use also trafeik or nginx | 14:25 |
| noonedeadpunk | spatel: I never ran magnum without octavia :) I guess you can jsut not in HA manner | 14:25 |
| spatel | noonedeadpunk i am learning and trying to understand how kubernetes work if octavia not exist | 14:27 |
| gokhani | noonedeadpunk: thanks for https://review.opendev.org/c/openstack/openstack-ansible/+/833775 and ı think I need to also enable sriov | 14:27 |
| spatel | I can understand HA and redundancy won't be there if no octavia. This is just for learning | 14:27 |
| spatel | gokhani thanks | 14:29 |
| spatel | gokhani Are you doing HPC on openstack? because i am also doing it and would like to understand your setup also :) | 14:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update MariDB version to 10.6.7 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/833259 | 14:33 |
| gokhani | spatel: yes but I am newbie to HPC. I struggled to how can we achive run hpc cluster on openstack | 14:34 |
| gokhani | I am reading your blogpost :) | 14:34 |
| spatel | we can help each other :) | 14:34 |
| spatel | I am trying to use glusterfs with manila to provide shared space for MPI job application | 14:35 |
| gokhani | yes definitely :) My customer requested using zfs storage and they say it is must :) | 14:37 |
| noonedeadpunk | just in case - nfs is pretty bad idea to use as shared storage | 14:38 |
| noonedeadpunk | any network issue and you stuck with mount that can't be unmounted so you need to reboot all computes | 14:38 |
| gokhani | noonedeadpunk: you are right, I had a lof issues about nfs in some of my deployments, but I don't know how can I use zfs storage instead of nfs | 14:42 |
| spatel | noonedeadpunk nfs can be force umount with -l (lazy option) | 14:43 |
| noonedeadpunk | spatel: when it's passed to libvirt? | 14:45 |
| noonedeadpunk | because it's not local mount | 14:45 |
| noonedeadpunk | and connection is not re-initiated | 14:45 |
| spatel | you can use fuse command to kill attached pid | 14:45 |
| spatel | but that won't be clean | 14:45 |
| noonedeadpunk | and then you have VM without disk?:) | 14:46 |
| spatel | you are correct it won't be clean and neat | 14:46 |
| noonedeadpunk | so it's really all mess, corrupted databases and etc | 14:46 |
| noonedeadpunk | gokhani: and how you plan to distinguish userspace with zfs with openstack space? | 14:47 |
| noonedeadpunk | It sounds like you would need to have several storages anyway? | 14:47 |
| gokhani | spatel: my customer is running their applications on 4 gpu servers with 8 gpu cards (NVIDIA RTX A6000). I am planning tu use gpu passthrough. for gpu virtualization we need to nvidia vgpu license | 14:48 |
| spatel | My customer doesn't have license so we decided to use passthrough | 14:51 |
| spatel | vgpu is good if you have multiple folks trying to use infra. in my case customer itself using and he own everything in that case doesn't need vgpu | 14:52 |
| spatel | do you have infiniband network? | 14:52 |
| gokhani | spatel: yes I have infiniband network and I am planing use as storage network | 14:57 |
| jrosser | i think i also vote for mandating zfs being kind of orthoganal to using filesystems in openstack | 14:57 |
| spatel | In my case IB is just for mpi job and storage will be on 10G dedicated nic | 14:57 |
| spatel | jrosser ZFS is great but you need storage protocol like NFS or iSCSI | 15:00 |
| jrosser | yes thats what i mean | 15:00 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:01 |
| opendevmeet | Meeting started Tue Mar 15 15:01:21 2022 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:01 |
| *** dviroel|ruck|mtg is now known as dviroel|ruck | 15:01 | |
| noonedeadpunk | #topic rollcall | 15:01 |
| spatel | \o/ | 15:01 |
| NeilHanlon | o/ heya folks, hope everyone is doing well | 15:02 |
| mgariepy | \o | 15:04 |
| noonedeadpunk | #topic office hours | 15:10 |
| noonedeadpunk | sorry I bit distracted internally | 15:10 |
| noonedeadpunk | having some nesty issue for weeks now with mariadb that drops connection but thinks it alive so leaves lock on table until timeout for connection ends | 15:11 |
| damiandabrowski[m] | hey! (sorry for being late) | 15:13 |
| noonedeadpunk | We've got approval that https://bugs.launchpad.net/openstack-ansible/+bug/1955676 works nicely | 15:15 |
| noonedeadpunk | *solution for | 15:15 |
| noonedeadpunk | Which is https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/822860 | 15:15 |
| noonedeadpunk | So probably would be great to have it reviewed | 15:15 |
| NeilHanlon | been looking at the graylog role in the ops repo, seems fairly straightforward but was thinking of swapping out the current journal-to-graylog converter that's deprecated with https://github.com/parse-nl/SystemdJournal2Gelf, maybe? | 15:16 |
| noonedeadpunk | Another nasty thing we also hitted after X upgrade was https://review.opendev.org/q/topic:bug%252F1961603 so worth checking it as well | 15:16 |
| noonedeadpunk | NeilHanlon: well... the only tricky thing is that it requires go everywhere? | 15:17 |
| noonedeadpunk | which is.... meh... | 15:17 |
| NeilHanlon | probably would create deb/rpm packages for it | 15:18 |
| NeilHanlon | i'm not against updating the current python one either | 15:18 |
| noonedeadpunk | we just don't have anything that requires go atm, and we're trying to leave things as minimalistic as possible. | 15:19 |
| noonedeadpunk | but yeah. I saw that journal2gelf is not supported for quite a while which is quite sad | 15:20 |
| NeilHanlon | gotcha, makes sense. I'll look at updating the journal2gelf module. it didn't look too bad, just some py2 -> py3 stuff that needs fixing | 15:20 |
| NeilHanlon | mostly around bytes and strings and all that fun :) | 15:20 |
| spatel | NeilHanlon why don't we use journalbeat to push journal to graylog? | 15:20 |
| noonedeadpunk | well, https://github.com/nailgun/journal2gelf has 2.1.0 jsut in case. So it lloks like it jsut wasn't pushed to pypi? | 15:21 |
| NeilHanlon | spatel: i'm not familiar w/ journalbeat, but sounds like an interesting option, maybe? | 15:22 |
| spatel | NeilHanlon that is what i am using and works great | 15:22 |
| NeilHanlon | yeah I installed right from the master branch on a test box noonedeadpunk and it still seemed to have some issues | 15:22 |
| noonedeadpunk | but um, what point of having journalbeat with graylog?:) | 15:22 |
| noonedeadpunk | isn't this part of elk?:) | 15:22 |
| NeilHanlon | yeah that was sorta my next thing heh | 15:23 |
| noonedeadpunk | ah I see | 15:23 |
| spatel | I don't have ELK infra (I am using graylog outside OSA and beat to send logs) | 15:23 |
| noonedeadpunk | Next to that I tried to use system-scope stuff and keystone changes, but that has barely merged to openstack collection as well as openstacksdk, and ofc is broken in several places. Was trying to investigate but have quite a lot of internal issues I have to address first... | 15:24 |
| NeilHanlon | i think journalbeat is also golang, fwiw | 15:25 |
| noonedeadpunk | Then I'd prefer SystemdJournal2Gelf, since at least it's BSD licensed... | 15:26 |
| spatel | Yes but single RPM and i have not installed journalbeat on all containers but just on physical node like infra/compute. | 15:27 |
| spatel | i point Journalbeat to point /openstack/logs/ to push all container logs to graylog | 15:28 |
| noonedeadpunk | But journal from all containers is in /var/log/journal/ ? | 15:29 |
| spatel | damn it correct sorry my bad - https://paste.opendev.org/show/bYg0RxxDdBZjW2gRuIIg/ | 15:32 |
| *** dviroel|ruck is now known as dviroel|ruck|lunch | 15:35 | |
| jrosser | hello | 15:37 |
| spatel | hello | 15:38 |
| NeilHanlon | looking again with fresh eyes, it appears someone's already done some of the work for journal2gelf working https://github.com/nailgun/journal2gelf/pull/6/files | 15:38 |
| jrosser | i was just taking another look at the molecule stuff if we wanted to discuss role tests | 15:39 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-pki master: Refactor conditional generation of CA and certificates https://review.opendev.org/c/openstack/ansible-role-pki/+/830794 | 15:48 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_neutron master: Add parameters to limit the number of DHCP or L3 agents https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/833769 | 15:56 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Change location of ipaddr filter https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/831528 | 15:58 |
| noonedeadpunk | #endmeeting | 16:03 |
| opendevmeet | Meeting ended Tue Mar 15 16:03:18 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:03 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-03-15-15.01.html | 16:03 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-03-15-15.01.txt | 16:03 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-03-15-15.01.log.html | 16:03 |
| *** dviroel|ruck|lunch is now known as dviroel|ruck | 16:03 | |
| noonedeadpunk | jrosser: I had no time for functional tests despite I started them :( | 16:04 |
| jrosser | heh no problem - i have been away for ~2 weeks also | 16:04 |
| jrosser | though i don't want to waste my time on it either | 16:04 |
| noonedeadpunk | so the whole idea was to use integrated tests and jsut run tests/test.yml with tests/inventory.ini if present. and based on zuul var we know what project is that | 16:05 |
| noonedeadpunk | as alternative to molecule that should be easier to manage | 16:05 |
| noonedeadpunk | and then define such jobs and define vars for jobs if needed, that would end up in user_variables | 16:05 |
| noonedeadpunk | jobs can be defined in any project, jsut have same parent | 16:06 |
| jrosser | i would like something thats really useful for role development | 16:11 |
| jrosser | and quick turnaround teardown/re-run is really helpful for that | 16:11 |
| jrosser | and for a role unit test there might be more than one situation to test too | 16:12 |
| jrosser | "does it work with just role defaults" / "test things A" / "test things B" | 16:13 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Improve incremental backups rotation in mariabackup script https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/828170 | 16:15 |
| noonedeadpunk | yes, but we need to have depends-on working as tons of stuff rely on each other. And with that I was thinking to run jsut bootstrap-ansible without bootstrap-aio | 16:17 |
| noonedeadpunk | so it should be really fast | 16:17 |
| jrosser | do you think we concentrate too much on integration testing? | 16:17 |
| jrosser | there is almost no unit testing | 16:17 |
| noonedeadpunk | but why we can't launch unit testing same way? | 16:18 |
| jrosser | i was adding more stuff to the PKI role for the unfinished keystone patch and just don't know if i break something else | 16:18 |
| noonedeadpunk | I mean - it's same scenarious that will be launched, jsut not with molecule, but with zuul job? | 16:18 |
| jrosser | oh well right, so its a question of the test environment then i guess | 16:19 |
| noonedeadpunk | yup | 16:19 |
| jrosser | either VM + openstack-ansible + lxc + tests/test.yml | 16:19 |
| jrosser | or VM + moecule/role + docker/podman + many scenarios | 16:20 |
| noonedeadpunk | (we kind of don't need even LXC I guess) | 16:20 |
| jrosser | oh really we do | 16:20 |
| jrosser | because if you run the tests once then you can't guarantee to clean everything up properly if you fix things then want to try again | 16:20 |
| jrosser | really i am most concerned about local development being nice rather than zuul stuff | 16:21 |
| noonedeadpunk | I just see nightmare of managing molecule in terms of supported os, when we'd need to manually patch all roles, not working dependancies, ansible versions.... | 16:21 |
| noonedeadpunk | ah, local development is question indeed. | 16:21 |
| noonedeadpunk | I was thinking purely CI | 16:21 |
| jrosser | it takes really very long to boostrap some LXC with openstack-ansible | 16:22 |
| jrosser | and same again to tear down / re-create | 16:22 |
| noonedeadpunk | where we don't need to destroy lxc and re-run | 16:22 |
| jrosser | currently i work on the PKI role | 16:23 |
| noonedeadpunk | but I don't see how we can manage molecule either. It's even more messier then we had functional testing with tests repo | 16:23 |
| jrosser | and it's complex enough that i'm very much concerned that i break it | 16:23 |
| jrosser | are there specific things that need addressing? | 16:23 |
| opendevreview | Neil Hanlon proposed openstack/openstack-ansible master: Use the short form nodeset definition once it's merged upstream https://review.opendev.org/c/openstack/openstack-ansible/+/833877 | 16:24 |
| noonedeadpunk | I'd say whole https://review.opendev.org/c/openstack/ansible-role-pki/+/831236/3/molecule/molecule.yml ? | 16:25 |
| noonedeadpunk | like platforms, ansible-lint version and test-galaxy-requirements.yml as well | 16:26 |
| noonedeadpunk | not saying about ansible version itself... | 16:26 |
| jrosser | well, this is the very first patch i make | 16:27 |
| jrosser | with no attempt to make any of that proper | 16:27 |
| noonedeadpunk | I mean when we will have that everywhere, we would need to patch all roles each release kind of... | 16:27 |
| noonedeadpunk | and with stuff like jsut raised with netcommon.... | 16:27 |
| noonedeadpunk | we need to have env defined in one place | 16:28 |
| jrosser | i have removed the lint as it's duplicate | 16:28 |
| jrosser | i was going to copy the requirements file from openstack-ansible repo | 16:28 |
| jrosser | as in, copy at run-time | 16:29 |
| noonedeadpunk | and we can set image as variable as well? | 16:32 |
| noonedeadpunk | ok, then we will jsut don't have depends-on working which is likely fine if we don't ever depend on some other role | 16:33 |
| noonedeadpunk | which we shouldn't have anyway for unit testing | 16:33 |
| noonedeadpunk | and we need to somehow control molecule version | 16:34 |
| jrosser | we have a place for that sort of thing already https://github.com/openstack/openstack-ansible/blob/master/test-requirements.txt | 16:35 |
| jrosser | well anyway - sounds like i should leave this alone for a while | 16:36 |
| noonedeadpunk | I;'d say you have point here for sure.... | 16:37 |
| jrosser | some things we test quite thoroughly with infra scenario already, like DB cluster | 16:38 |
| jrosser | so i don't think there is any merit in refactoring things like that | 16:39 |
| jrosser | but kind of 'tool' roles like pki, keypairs, config_template and so on could do with a refresh particularly if we want to deprecate openstack-ansible-tests | 16:40 |
| jrosser | they are all pretty complex and need a test suite rather than just saying that AIO deploys OK | 16:40 |
| noonedeadpunk | yes, totally | 16:40 |
| noonedeadpunk | but indeed I haven't thought about local development when was thinking through my idea with jsut adding functional scenario into integrational testing | 16:41 |
| noonedeadpunk | I just thought that we have everything prepared and it's super easy instead of aio just run tests.yml in exact same env we run everywhere | 16:42 |
| noonedeadpunk | that would be super tricky indeed for local testing | 16:43 |
| jrosser | sounds like we had started thinking about this from opposite ends :) | 16:43 |
| noonedeadpunk | yeah, indeed) | 16:43 |
| gokhani | vb vx xv vgh<qvsxnnxvvsvsbsvzzzzzzzzzzbbbbbb x f jfze4munwök Plı*85 | 17:09 |
| noonedeadpunk | that looks like yubikey hash... | 17:18 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder master: Add configuration option for heartbeat_in_pthread https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/833238 | 17:28 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Add configuration option for heartbeat_in_pthread https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/833236 | 17:30 |
| *** arxcruz is now known as arxcruz|off | 17:55 | |
| tbarron | spatel: noonedeadpunk was reading backlog, don't see gokhani here now but manila has a zfs-on-linux driver. dunno if that'w what gokhani has or if it's oracle/solaris zfs. | 19:05 |
| noonedeadpunk | I guess it was linux yeah... | 19:05 |
| spatel | tbarron he left for the day. | 19:06 |
| noonedeadpunk | but dunno if he was looking for shared filesystem, felt like more for glance and cinder which is quite different layer anyway | 19:06 |
| noonedeadpunk | but actually I didn't know manila does support zfs! | 19:06 |
| tbarron | got it, ty. There zfs via nfs seems dicey as you say. | 19:07 |
| spatel | if we want to share ZFS then we need ganesha-nfs or iscsi protocol | 19:07 |
| tbarron | here's the manila zfs on linux driver doc: https://opendev.org/openstack/manila/src/branch/master/doc/source/admin/zfs_on_linux_driver.rst | 19:09 |
| spatel | so its NFS last mile correct? | 19:12 |
| tbarron | spatel: but not *ganesha* gatewayed. Others in #openstack-manila have done more with it than I have. But I think to share even native oracle zfs you use NFS or SMB, right? | 19:19 |
| spatel | I don't have much experience with ZFS but all i know we need server protocol to export filesystem (CIFS/SMB or NFS) | 19:20 |
| spatel | does ZFS has own client server protocol ? | 19:21 |
| tbarron | it leverages nfs or smb for network file sharing; manila driver uses nfs for zfs-backed shares (smb wasn't implemented for zfs for manila) | 19:26 |
| tbarron | so thereis no separate zfs-on-the wire client server protocol, but in theory | 19:27 |
| tbarron | you get cool back end features like replication, zfs snapshots, etc. | 19:28 |
| spatel | nice!! | 19:49 |
| spatel | does ZFS has clustering feature? not DAS based which is hardware based cluster. | 19:50 |
| jrosser | not really | 19:52 |
| jrosser | its a server plus as many block devices as you need, attached over nvme/sata/sas/FC/whatever | 19:54 |
| spatel | +1 | 20:01 |
| jrosser | i have a small zfs setup for cinder backup, so that the backups do not have a shared fate with the ceph cluster | 20:02 |
| jrosser | now if any users actually bother to use it is another thing...... | 20:02 |
| tbarron | jrosser: since you mentioned ceph, I just did an OSA AIO for the first time. Used | 20:10 |
| tbarron | "export SCENARIO='aio_manila'" before running bootstrap-aio.sh and | 20:11 |
| tbarron | it all seemed to "just work.". Nice, not a very big footprint and I don't have to | 20:11 |
| tbarron | rebuild kolla containers to test a code change. | 20:12 |
| tbarron | jrosser: But I'm led to ask: the upstream ceph community is deprecating | 20:12 |
| tbarron | ceph-ansible in favor of cephadm. | 20:13 |
| tbarron | Does OSA have plans to adapt to cephadm for future Ceph deployments? | 20:13 |
| jrosser | that scenario is exactly what we run when merging patches to our os_manila ansible role | 20:13 |
| jrosser | like here https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/827604 | 20:14 |
| tbarron | cool | 20:14 |
| tbarron | jrosser: manila (and devstack-plugin ceph, and tripleo) will be moving to the ceph | 20:15 |
| jrosser | tbarron: do you mean ceph-ansible itself is deprecated | 20:15 |
| tbarron | orchestrator backed by cephadm | 20:15 |
| jrosser | or just its use by redhat in their stuff is switching to cephadm | 20:15 |
| tbarron | sinde yes, ceph-ansible itself is being deprecated, and since | 20:15 |
| tbarron | the *upstream ceph community* will be only supporting new needed features like | 20:16 |
| tbarron | active-active ganesha (ceph-nfs daemon) with the orchestrator | 20:16 |
| tbarron | (which is either backed by cephadm or by rook (for k8s). | 20:17 |
| jrosser | imho ceph orchestrator intersects quite heavily with what tools such as ansible are trying to acheive | 20:17 |
| jrosser | and so if you want your deployment defined by your ansible code that this is going to be tricky | 20:17 |
| tbarron | jrosser: note that I am not here as a Red Hat guy (am actually independent now) and am | 20:17 |
| tbarron | just reporting. | 20:18 |
| tbarron | I liked my experience with OSA and manila and ceph and it would be cool if | 20:19 |
| *** dviroel|ruck is now known as dviroel|ruck|brb | 20:19 | |
| tbarron | OSA with manila and Ceph will be able to use the new stuff where Ganesha runs | 20:19 |
| tbarron | active-active inside the Ceph cluster with its own "ingress" (HA proxy) and | 20:20 |
| tbarron | looks to consumers like any other ceph daemon. | 20:20 |
| tbarron | jrosser: we can talk more about it later. devstack-plugin-ceph changes to do this are in progress. | 20:21 |
| tbarron | That plugin was pretty primitive, pre-ceph-ansible. | 20:22 |
| tbarron | But it did via shell commands what ceph-ansible does. | 20:22 |
| jrosser | yeah, though i think we maybe come from a deployer-first presepective | 20:22 |
| tbarron | jrosser: Yeah, that's the right perspective for OSA! | 20:22 |
| jrosser | we need battle hard, and safe not to burn your cluster when doing upgrades or scaling out | 20:23 |
| jrosser | ceph-ansible has been excellent in that regard | 20:23 |
| tbarron | note that I don't "have a dog in the fight" w.r.t. ceph-ansible vs cephadm, but | 20:23 |
| tbarron | I think my claim that upstream ceph community is, rightly or wrongly, dropping | 20:24 |
| tbarron | ceph-ansible in favor of cephadm is correct. Take it as a data claim to be checked. | 20:24 |
| jrosser | we are also not wedded to ceph-ansible for production | 20:25 |
| jrosser | in fact most havey users of OSA do not use the inbuilt ceph stuff at all and prefer to separate the concerns | 20:25 |
| jrosser | we provide hooks to integrate an external cluster | 20:25 |
| tbarron | That makes perfect sense. | 20:26 |
| jrosser | ceph-ansible gives us a quick solution for CI, and also for people who want a one-stop solution | 20:26 |
| tbarron | When I worked with TripleO it was pretty much the same. But when tripelO set | 20:26 |
| jrosser | but that integration causes major stress at upgrade time as the requirements of ceph(-ansible) and a particular openstack release are not necessarily co-incident | 20:27 |
| tbarron | up openstack to deploy the ceph cluster instead of just referenciing an externally deployed cluster, we tried to modify the tripleo triggered deployment to do it by the current best practice and look | 20:27 |
| tbarron | the same. | 20:27 |
| jrosser | btw i don't have manila in my deployments, it's always seemed too hard to get real multitenancy and performance at the same time | 20:28 |
| jrosser | but i would really like to | 20:28 |
| tbarron | jrosser: so pls. just take my remarks now as a "heads up". If I'm correct then | 20:28 |
| tbarron | external deployments of Ceph will move in time from ceph-ansible to cephadm. | 20:29 |
| tbarron | And at some point it make sense for OSA CI/dev/test deployments of Ceph to do the same. | 20:29 |
| tbarron | jrosser: your point about *real* multitenancy and performance tradeoff is fair. | 20:30 |
| jrosser | unless there is a way to not be docker/podman based i think it is highly unlikely you would see it used in OSA | 20:31 |
| tbarron | There was some anti-container sentiment within the Ceph community w.r.t. cephadm but | 20:32 |
| tbarron | so far I don't think it prevailed. | 20:32 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Change location of ipaddr filter https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/833119 | 20:33 |
| tbarron | My *rough* take on cephadm is that it aims to give one k8s-like declarative orchestration of a ceph cluster w/o actually using k8s. | 20:34 |
| jrosser | i was just reading the docs yes and it looks very much like that | 20:36 |
| tbarron | w.r.t. the multi-tenancy/performance tradeoff, cephadm *promises* to support | 20:40 |
| tbarron | dynamically spawning new nfs clusters (each a set of active-active nfs ganesha daemons) | 20:41 |
| tbarron | where each cluster can have its own ingress (HA proxy implemented, at least for now). | 20:42 |
| tbarron | So in theory manila could trigger these per-tenant, and set up per-tenant networking. | 20:42 |
| tbarron | nfs would still be in the data path from native CephFS | 20:43 |
| jrosser | how would you see that networking work? | 20:43 |
| tbarron | but there wouldn't be a single bottleneck. | 20:43 |
| tbarron | jrosser: I don't know if we'd need to take advantage of the BGP stuff now getting | 20:45 |
| tbarron | OpenStack support or not. Likely manila would need to inject return routes to the | 20:45 |
| tbarron | client network. So even if the ceph-nfs ingress is pingable from everywhere | 20:46 |
| tbarron | (and firewall/security rules could be set up by manila to restrict) | 20:46 |
| tbarron | packets could only return route in tenant-appropriate manner. | 20:47 |
| tbarron | Admittedly I am waving my hands. Originally the vision with Sage et. al. | 20:47 |
| tbarron | was to have k8s-managed ceph clusters and to use kuryr to tie the NFS ingresses | 20:48 |
| jrosser | so far the only thing that doesnt make me go ewwwwwww is future support for virtiofs | 20:48 |
| tbarron | back to tenant private neutron networks. So the goal would be something like that. | 20:48 |
| jrosser | as an example my storage network doesnt route anywhere | 20:48 |
| jrosser | absolutely nowhere near the tenants | 20:48 |
| tbarron | yeah that's the way I did it for the ceph daemons proper. But not for nfs :) and yeah, | 20:49 |
| tbarron | I get it, and that's why I pitched virtiofs to nova. | 20:49 |
| jrosser | i have another use case for that to pass intel RAPL power usage data into virtual machines | 20:50 |
| tbarron | They think they'll have something for Zed. | 20:50 |
| tbarron | But for bare metal compute instances we'd still need something like CephFS w/o | 20:50 |
| tbarron | virtiofs. | 20:50 |
| jrosser | we wrote an article for superuser here https://superuser.openstack.org/articles/environmental-reporting-dashboards-for-openstack-from-bbc-rd/ | 20:51 |
| jrosser | that describes a use case where having a static virtiofs mount from the hypervisor into the VM would allow per process energy accounting | 20:51 |
| tbarron | I hadn't seen that artticle, thank you! | 20:52 |
| jrosser | https://github.com/hubblo-org/scaphandre/issues/60 | 20:54 |
| tbarron | awesome | 20:55 |
| *** dviroel|ruck|brb is now known as dviroel|ruck | 20:57 | |
| spatel | jrosser is virtiofs ready for prod, i think not | 21:00 |
| jrosser | i do not think so | 21:00 |
| spatel | it would be awesome when ever its ready | 21:02 |
| spatel | tbarron blog out my manila+glusterfs setup here https://satishdotpatel.github.io/openstack-manila-integration-with-glusterfs/ | 21:02 |
| spatel | now i need to test how i can provide manila SRIOV interface for dedicated NFS mount point | 21:03 |
| spatel | tbarron by the way thank for the manila patch | 21:05 |
| *** dviroel|ruck is now known as dviroel|ruck|afk | 21:51 | |
| *** dviroel|ruck|afk is now known as dviroel|ruck | 23:55 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!