| kleini | with W I have the problem on some computes, that nova-compute runs into too many open files with oslo messaging. Is this a known issue? | 08:32 |
|---|---|---|
| andrewbonney | Aha, you'll be after https://bugs.launchpad.net/oslo.messaging/+bug/1949964 | 08:33 |
| andrewbonney | There's a partial fix in there, but I'm not sure it's 100% solved | 08:33 |
| kleini | oh, thanks very much. will try to get the fix deployed | 08:38 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-python_venv_build master: Add per-distro vars files https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/824180 | 08:50 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Fix modules location https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/824649 | 09:01 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Update provider_networks with latest changes https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/824646 | 09:02 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Move git_requirements to plugins collection https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/824563 | 09:03 |
| opendevreview | Merged openstack/openstack-ansible-tests stable/xena: Fix rich version for ansible-lint https://review.opendev.org/c/openstack/openstack-ansible-tests/+/824540 | 09:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_service master: Add ability to create templated services https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/816531 | 10:24 |
| noonedeadpunk | I'm working on journald-remote now and systemd templated service would be handy to define multiple destinations ^ | 10:25 |
| noonedeadpunk | Do we have any agreement if we want this feature or not because of complexity it brings? | 10:25 |
| noonedeadpunk | It always could be just different services indeed | 10:27 |
| noonedeadpunk | but it's somehow comfy to have same name and jsut different arguments... | 10:28 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Use common service setup tasks from a collection rather than in-role https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/820999 | 10:30 |
| jrosser__ | noonedeadpunk: the complexity is not out of line with other things we have | 10:31 |
| jrosser__ | it maybe lacks a link to the systemd documentation which describes what template services are | 10:32 |
| noonedeadpunk | I put it in reno:) https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/816531/4/releasenotes/notes/templated_service-f31e4515c2fd75ab.yaml | 10:32 |
| noonedeadpunk | sorry should have placed in commit msg as well I guess | 10:32 |
| noonedeadpunk | but it's really described in good manner there | 10:34 |
| jrosser__ | or even as a comment in the defaults file, as I had to go read to understand that it was not an ansible tenplating thing, but actually native to systemd | 10:34 |
| noonedeadpunk | ah, yes, fair | 10:35 |
| jrosser__ | does it need to also account for the “load” Boolean I just added? | 10:35 |
| noonedeadpunk | I was again suck in naming things... | 10:35 |
| noonedeadpunk | I just rebased in on top of your change | 10:35 |
| noonedeadpunk | L25 https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/816531/4/tasks/systemd_load.yml#25 | 10:36 |
| jrosser__ | ah yes I see | 10:37 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_service master: Add ability to create templated services https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/816531 | 10:37 |
| jrosser__ | I’m happy with it, so use it for the journal things | 10:38 |
| noonedeadpunk | Why I was talking about complexity is that with patch it becomes less obvious from output what service we're trying to run against | 10:38 |
| noonedeadpunk | because of double iteration | 10:38 |
| jrosser__ | can that be improved with two variables used in the task description | 10:39 |
| noonedeadpunk | Thus I added service_name to task to at least somehow cover that | 10:39 |
| jrosser__ | if I follow properly | 10:39 |
| noonedeadpunk | *to task name | 10:39 |
| noonedeadpunk | yeah, true | 10:39 |
| noonedeadpunk | But I wasn't able to make it for handlers for some reason... | 10:40 |
| noonedeadpunk | (but not sure now maybe I was :p) | 10:41 |
| noonedeadpunk | worth checking logs | 10:41 |
| *** arxcruz|ruck is now known as arxcruz | 11:13 | |
| *** dviroel|out is now known as dviroel | 11:21 | |
| opendevreview | Merged openstack/ansible-config_template master: Copy refactor of code quality issues https://review.opendev.org/c/openstack/ansible-config_template/+/824601 | 11:32 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-python_venv_build master: Split venv_rebuild functionality https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/773984 | 11:57 |
| *** anbanerj is now known as frenzyfriday|ruck | 12:02 | |
| opendevreview | Merged openstack/openstack-ansible-os_zun stable/xena: Remove testing on Centos-8 https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/824535 | 12:11 |
| jrosser__ | theres still two blocked things on here which i won't be able to look at today https://review.opendev.org/q/topic:%22osa%252Fremove-centos8%22+(status:open%20OR%20status:merged) | 12:13 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Use provider_networks from collection https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/824650 | 12:16 |
| opendevreview | Merged openstack/ansible-role-python_venv_build master: Add per-distro vars files https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/824180 | 13:14 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Move system_crontab_role to collection https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/824590 | 13:36 |
| opendevreview | Merged openstack/openstack-ansible-tests stable/ussuri: Remove opensuse jobs https://review.opendev.org/c/openstack/openstack-ansible-tests/+/824207 | 13:50 |
| opendevreview | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_murano stable/ussuri: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/824717 | 14:02 |
| opendevreview | Merged openstack/openstack-ansible master: Do not duplicate packages installed with the venv build role https://review.opendev.org/c/openstack/openstack-ansible/+/824179 | 14:37 |
| spatel | jamesdenton question how do i remove dead gateway chassis from OVN | 15:11 |
| jamesdenton | can you elaborate? | 15:18 |
| jamesdenton | (i am not sure) | 15:18 |
| spatel | I had 4 network node and one of node is dead, i have removed dead node but ovn still showing in ovn-nb db | 15:28 |
| *** dviroel is now known as dviroel|lunch | 15:33 | |
| jamesdenton | which command are you using to list those | 15:48 |
| spatel | jamesdenton let me DM you for security reason | 15:55 |
| opendevreview | Merged openstack/openstack-ansible-os_magnum master: Run service_setup only once https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/824526 | 16:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Convert infra-journal-remote playbook to role https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/824731 | 16:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Move infra-journal-remote logic to its role https://review.opendev.org/c/openstack/openstack-ansible/+/824734 | 16:41 |
| noonedeadpunk | damiandabrowski[m]: you might be interested in these 2 ^ | 16:41 |
| noonedeadpunk | for some reason I see systemd-journal-upload being stuck in my aio, but as log host I used container, so maybe that's why... | 16:42 |
| damiandabrowski[m] | ouh, so we already made a decision to put it into openstack-ansible-plugins | 16:42 |
| damiandabrowski[m] | thanks anyway | 16:42 |
| noonedeadpunk | I guess we did during last meeting? | 16:43 |
| noonedeadpunk | We agreed to re-evaluate that later if needed | 16:44 |
| noonedeadpunk | but place for now in plugins as "staging" place | 16:45 |
| damiandabrowski[m] | ouh ok, maybe i missed that :D it's ok then | 16:45 |
| *** dviroel|lunch is now known as dviroel| | 16:46 | |
| *** dviroel| is now known as dviroel | 16:46 | |
| noonedeadpunk | or we might all misunderstood each other | 16:46 |
| noonedeadpunk | I was kind of referrencing https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-01-11-15.00.log.html#l-93 | 16:47 |
| noonedeadpunk | but you're right, I must use `agreed` command more | 16:47 |
| noonedeadpunk | Will try fixing that in the future | 16:47 |
| damiandabrowski[m] | it's ok Dmitriy ;) | 16:47 |
| damiandabrowski[m] | if someone is interested: it may be a new beginning of infra-journal-remote role(i'm going to focus on that on February) | 16:48 |
| damiandabrowski[m] | https://github.com/citynetwork/role-journal-remote | 16:48 |
| noonedeadpunk | lol | 16:48 |
| noonedeadpunk | welll | 16:50 |
| noonedeadpunk | we're a big company :p | 16:50 |
| noonedeadpunk | I know that we were about to do some work on it but didn't know it has been already done | 16:53 |
| noonedeadpunk | I should have left that alone... | 16:54 |
| damiandabrowski[m] | ooops :D | 16:54 |
| damiandabrowski[m] | however, I haven't looked much into Erik's role yet, so at the end of the day, it may be easier to start with Your version | 16:57 |
| noonedeadpunk | they're very close... I kind of have a feeling that I just stole that work, but I swear I saw it just now :) | 16:58 |
| noonedeadpunk | I mean - even files named same | 16:59 |
| noonedeadpunk | well, looking https://github.com/citynetwork/role-journal-remote/blob/main/tasks/journal_remote_post_install.yaml#L40-L54 I realized that using systemd_service role to setup service might be indeed wrong | 17:03 |
| noonedeadpunk | as just realized that systemd-journal-remote.service shipped with package | 17:03 |
| noonedeadpunk | or, well, override_only should be used I believe | 17:05 |
| noonedeadpunk | or just mask default services :) | 17:05 |
| noonedeadpunk | but override_only sounds like nice option indeed. | 17:06 |
| noonedeadpunk | will sort this out anyway ) | 17:08 |
| damiandabrowski[m] | I'm not that familiar with ansible-role-systemd_service but looking at #816531, I think override_only may be a good idea ;) | 17:09 |
| jawad-axd | Hi all! I came across this paper http://seclab.cs.sunysb.edu/seclab/pubs/asiaccs16.pdf . A bit shocked to see Section 3.2 in the paper. Just want to know if that is really the case, when one compute node is compromised, to what extent rest of infrastructure is secure. Highly appreciate some comments on this. | 17:44 |
| mgariepy | in what case you can have a host compromised and be confident you are not screwed ? | 17:58 |
| noonedeadpunk | jawad-axd: um, for instance we create a rabbitmq users per vhost, each service reside on it's own vhost | 18:08 |
| jawad-axd | There are hypervisor exploitation stories around, and my boss is very concerned that if one compute host is compromised then others should not be affected. Sigh! But yeah, is there some answer to this question? | 18:09 |
| noonedeadpunk | And I don't think you can really send RPC call with nova rpc user that will make neutron to create port and without that you won't have instance | 18:10 |
| noonedeadpunk | If one is compromised this means there's a way to compromise it, which means that others likely have same door open. So even without RPC and other stuff mentioned you're likely screwed as mgariepy said | 18:11 |
| mgariepy | you can jump from compute to compute in case you have migration activated. | 18:12 |
| noonedeadpunk | well, also, I think good thing to do might be to use different users for cells... So even if they get rpc for cell, this doesn't mean it will be possible to screw whole rabbit | 18:13 |
| noonedeadpunk | for nova user - yes | 18:13 |
| mgariepy | if you use shared storage between your computes, you likely have access to the whole pool via the client access. | 18:13 |
| jawad-axd | From paper, they mentioned on compromised compute host , sniffing token, grabbing MQ credentials, and creating MG message with wild card, and getting authorized by API. According to them, there is no check after token is authorised from API, also about token permissions are not limited and so on. | 18:14 |
| mgariepy | for nova user yes but escalation is often possible but it depends on a lot of other stuff for sure.. | 18:15 |
| noonedeadpunk | hm, btw I wonder if things like https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/templates/nova.conf.j2#L135-L150 must be defined for computes.... | 18:19 |
| noonedeadpunk | because with that you don't need to sniff a thing:) | 18:19 |
| mgariepy | hmm. indeed. | 18:20 |
| jrosser__ | ultimately any host running software that needs to interact with another host must have some kind of credential | 18:21 |
| jrosser__ | and so if that credential is compromised then there is nothing you can do | 18:22 |
| mgariepy | even with vault or something like that it would not help a lot i think | 18:23 |
| noonedeadpunk | it will just make things a bit harder | 18:24 |
| mgariepy | yeah | 18:24 |
| noonedeadpunk | but until you have credentials to access vault... | 18:24 |
| noonedeadpunk | stored on same host | 18:24 |
| mgariepy | the idea is only to run faster than the other anyway no ? haha | 18:25 |
| mgariepy | so the lions doesn't get you ! | 18:25 |
| jawad-axd | Right. Seems like its very difficult to close all the holes from compromised compute, also there is shared storage between compute hosts.I got that. Is there any other OSA solution for compute security domains or maybe I am asking too much. | 18:25 |
| jawad-axd | ? | 18:26 |
| jawad-axd | Which might help in this kind of situation. | 18:26 |
| noonedeadpunk | So indeed, you don't need to sniff anything, as keystone admin credentials are stored in nova.conf on each compute | 18:30 |
| noonedeadpunk | they passed really extra mile there :P | 18:30 |
| noonedeadpunk | as nova-compute needs to talk to other services and have admin privileges or write specific policy, etc | 18:31 |
| mgariepy | locking all the holes is hard ofthen holes get spawned out of thin air like log4shell thing or heartbleed... | 18:31 |
| noonedeadpunk | good idea that was thrown in #openstack-nova - use application credentials per compute node. | 18:36 |
| noonedeadpunk | at least you can rotate that fast enough.... | 18:36 |
| noonedeadpunk | but still - extra users/projects could be created until that is done | 18:37 |
| jawad-axd | I ll look at app credentials with compute node. Thanks.Wondering if all openstack public clouds out there have some kind of mechansim for this secanrio or they are just trusting the hypervisor. | 18:41 |
| noonedeadpunk | jawad-axd: but that paper is stupid imo... `Each compute node stores its MQ credentials inside its OpenStack configuration files.` In addition to MQ credentials, all keystone user credentials are stored there as well, so they could just use them rather play around and sniff smth... | 18:42 |
| noonedeadpunk | If you can read nova.conf - you don't need to do all the rest stuff as you granted access to keystone | 18:43 |
| jawad-axd | Thats also true. @noonedeadpunk | 18:44 |
| noonedeadpunk | but eventually if you escaped libvirt domain - you should not be able to read nova.conf anyway... | 18:46 |
| noonedeadpunk | so it's only if you got root kind of... | 18:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Fix umask for /etc/nova directory https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/824774 | 18:54 |
| mgariepy | noonedeadpunk, small comment on this one ^^ | 19:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Fix umask for /etc/nova directory https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/824774 | 19:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Change default mode while creating directories https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/824774 | 19:08 |
| mgariepy | thanks | 19:13 |
| spatel | jamesdenton did you work on GPU virtualization | 19:53 |
| spatel | i have GPU nodes and trying to play :) | 19:53 |
| spatel | i am working on build HPC on openstack | 19:55 |
| spatel | My hardware is Tesla V100S PCIe 32GB | 19:56 |
| jrosser__ | noonedeadpunk: i expect we could do a lot more with this stuff https://docs.arbitrary.ch/security/systemd.html | 20:16 |
| spatel | is this check against any database to get this score? | 20:22 |
| spatel | how does it score ? | 20:22 |
| spatel | nevermind - https://itectec.com/ubuntu/ubuntu-how-to-address-results-of-systemd-analyze-security/ | 20:23 |
| spatel | hope this is not some like SELinux and soon folks stop paying attention and disable it | 20:24 |
| jamesdenton | spatel i did some GPU passthrough a while back, but nothing more than that | 20:26 |
| spatel | jamesdenton https://www.jimmdenton.com/gpu-offloading-openstack/ | 20:26 |
| jamesdenton | yep | 20:26 |
| spatel | jamesdenton - https://paste.opendev.org/show/812125/ | 20:34 |
| spatel | in my case should i black-list - nouveau, nvidia_drm | 20:34 |
| spatel | curious that you used vfio-pci but all other documents not talking about that | 20:52 |
| *** dviroel is now known as dviroel|out | 20:57 | |
| spatel | anyway i am following your blog to see how it goes :) | 21:08 |
| mgariepy | spatel, do you want to do passthrough or vgpus stuff? | 21:13 |
| spatel | passthrough | 21:13 |
| spatel | i want to expose my GPU to virtual machine | 21:13 |
| mgariepy | vgpus also do exposes some gpu to the vms | 21:14 |
| mgariepy | but you can usually split the gpu and share it between multiple vms. | 21:14 |
| spatel | what is the difference here? | 21:14 |
| mgariepy | and you need to pay a licence i think. | 21:15 |
| mgariepy | imo passthrough is much more simple. | 21:15 |
| spatel | what is the difference between Passthrough vs vgpu ? function and benefit point of view | 21:15 |
| mgariepy | you can split a gpu to share it between multiple vms | 21:16 |
| spatel | in vgpu deployment? | 21:17 |
| mgariepy | you can have 1 gpu splitted between 4 vms | 21:17 |
| mgariepy | (made up number as i don't currently have any that i run) | 21:18 |
| spatel | This is my first GPU compute nodes so no idea what i am doing :) | 21:18 |
| spatel | just 2 hour ago i got my GPU compute node and try to learn as much as possible | 21:19 |
| mgariepy | passthrough is more simple. | 21:19 |
| mgariepy | vgpus needs some licence and a match of driver version on the hosts and in vm. irrc. | 21:19 |
| spatel | lets do passthrough then and later vgpus | 21:20 |
| spatel | I have 10 GPU compute nodes where we are going to run simulation | 21:20 |
| mgariepy | ok | 21:21 |
| mgariepy | if you have 1 user i guess you don't need to bother with vgpus. | 21:21 |
| spatel | This is University openstack cluster for HPC style simulation and research | 21:22 |
| spatel | i am assuming multiple folks or student going to use | 21:22 |
| spatel | but let me first go with whatever easy | 21:23 |
| spatel | i am reading this doc and they are talking about vGPU - https://docs.openstack.org/nova/queens/admin/virtual-gpu.html | 21:23 |
| mgariepy | jamesdenton, nice post. i vaguely remember having to add the driver and pciid combinaison to the initramfs. | 21:23 |
| spatel | i am following his doc | 21:24 |
| mgariepy | let me know if it works. | 21:24 |
| spatel | I will blog that out too :) | 21:24 |
| spatel | i need to add entry in /etc/nova/nova.conf of compute node like - passthrough_whitelist: | 21:25 |
| spatel | but why do i need entry in nova-api for alias ? alias: { "vendor_id":"10de", "product_id":"1c30", "device_type":"type-PCI", "name":"quadro-p2000" } | 21:25 |
| mgariepy | so you can refer to the card bia the alias in the flavor. | 21:26 |
| spatel | what if i have multiple kind of GPU hardware then how do i handle in nova-api? | 21:26 |
| spatel | i have to add multiple key/value assuming | 21:26 |
| mgariepy | --property "pci_passthrough:alias"="quadro-p2000:1" | 21:27 |
| mgariepy | you can have multiple alias | 21:27 |
| spatel | ah! ok | 21:27 |
| spatel | let me give it a try | 21:27 |
| mgariepy | if you endup with gamers gpus that have usb ports and other stuff you will have fun :D | 21:28 |
| spatel | fun part is i am doing all this with kolla-ansible :) | 21:28 |
| mgariepy | get out ! :P | 21:28 |
| spatel | hehe | 21:28 |
| mgariepy | LOL | 21:28 |
| spatel | They have hard requirement to use kolla-ansible and that is why i am learning my way to use kolla | 21:28 |
| mgariepy | do you have real gpu or you have gamers ones? | 21:28 |
| spatel | real GPU :) | 21:29 |
| mgariepy | ok nice. | 21:29 |
| mgariepy | anyway. have a nice weekend. i'm, done for this week. | 21:29 |
| spatel | thanks for the help :) and have a great weekend with lots of wins and beers | 21:29 |
| spatel | wine* | 21:30 |
| spatel | i bought new keyborad and its messing with me | 21:30 |
| opendevreview | Merged openstack/openstack-ansible master: Move git_requirements plugin to collection https://review.opendev.org/c/openstack/openstack-ansible/+/824574 | 22:42 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!