| opendevreview | Ghanshyam proposed openstack/nova master: Server actions APIs scoped to project scope https://review.opendev.org/c/openstack/nova/+/824358 | 00:54 |
|---|---|---|
| gmann | seems like tempest-integrated-compute-centos-8-stream is failing consistently https://zuul.openstack.org/builds?job_name=tempest-integrated-compute-centos-8-stream | 01:50 |
| opendevreview | Ghanshyam proposed openstack/nova master: Server actions APIs scoped to project scope https://review.opendev.org/c/openstack/nova/+/824358 | 02:04 |
| melwitt | gmann: :( thanks, I wondered about it | 02:04 |
| *** artom__ is now known as artom | 02:07 | |
| gmann | melwitt: I am seeing this traceback in logs, not sure about root cause https://zuul.opendev.org/t/openstack/build/e0db6a9a7ba04e66b0781ba7d259357d/log/controller/logs/screen-q-svc.txt#32875 | 02:08 |
| gmann | pinged neutron folks about it in case they are aware or can find the rootcause | 02:10 |
| melwitt | ack, that's not something I've seen before | 02:12 |
| gmann | it seems it started failing since ~1.5 hrs https://zuul.openstack.org/builds?job_name=tempest-integrated-compute-centos-8-stream | 02:14 |
| melwitt | that's consistent with what I observed. I had been following that change and rechecking it and it had been working until not too long ago | 02:16 |
| *** EugenMayer3 is now known as EugenMayer | 04:05 | |
| *** hemna9 is now known as hemna | 07:38 | |
| *** songwenping_ is now known as songwenping | 08:09 | |
| bauzas | good morning Nova | 08:57 |
| gibi | good morning Nova | 08:57 |
| * bauzas will start his upstream day by using his -2 stick for specs | 08:57 | |
| gibi | I have a full day workshop downstream so I my presence will be spotty | 08:57 |
| bauzas | gibi: spotty ? fine | 08:59 |
| bauzas | use a headset to listen to it | 08:59 |
| gibi | yeah | 08:59 |
| bauzas | ok, my pun is terrible | 09:00 |
| gibi | video is mandatoryt | 09:00 |
| * bauzas hears crickets about his joke | 09:00 | |
| bauzas | http://www.reactiongifs.com/r/weeds.gif | 09:01 |
| gibi | sorry I'm distracted | 09:02 |
| * kashyap waves | 09:05 | |
| gibi | kashyap: o/ | 09:05 |
| kashyap | [Off-topic] If any of you use fancy ergonomic mechanical keyboards | 09:06 |
| *** Uggla|afk is now known as Uggla | 09:07 | |
| bauzas | hmmm, I forgot about this spec https://review.opendev.org/c/openstack/nova-specs/+/819510 | 09:12 |
| bauzas | sad it got a -2 | 09:12 |
| * bauzas wonders what happens with cyborg, haven't seen anything during the last weeks | 09:13 | |
| opendevreview | Federico Ressi proposed openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683 | 11:50 |
| *** dasm|off is now known as dasm | 13:32 | |
| *** tbachman_ is now known as tbachman | 15:09 | |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Reject AZ changes during aggregate add / remove host https://review.opendev.org/c/openstack/nova/+/821423 | 16:51 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: trigger nova-next with new tempest test https://review.opendev.org/c/openstack/nova/+/824607 | 16:58 |
| gmann | melwitt: bauzas gibi for centos8 stream job failing, I have filed the bug https://bugs.launchpad.net/neutron/+bug/1957941 | 17:12 |
| gmann | and making job as non voting to unblock the nova and tempest gate https://review.opendev.org/c/openstack/tempest/+/824740 | 17:12 |
| sean-k-mooney | gmann: oh i see the binding detail are failing a scope check | 17:30 |
| sean-k-mooney | and other fileds | 17:31 |
| sean-k-mooney | with the new deftionsing some of those shoudl be project_admin and the rest project member | 17:31 |
| sean-k-mooney | i.e. host_id woudl be project_admin | 17:31 |
| sean-k-mooney | as would the physical_network and segmentation_id i belive | 17:32 |
| sean-k-mooney | have neutron acidentlly started enforcing scope by default? | 17:33 |
| noonedeadpunk | hey! I was wondering - does keystone_authtoken/neutron sections should be defined for nova-compute? | 18:20 |
| noonedeadpunk | eventually does it needs interaction with keystone? I bet no as it only talk to conductor? | 18:20 |
| noonedeadpunk | likely only key_manager is required though | 18:22 |
| sean-k-mooney | nova-compute directly calls neutron | 18:25 |
| noonedeadpunk | aha | 18:25 |
| sean-k-mooney | nova-compute directly talks to most other services rest apis | 18:25 |
| sean-k-mooney | it only uses the condutor to talk to the scheduler, db and conductor itslef | 18:25 |
| noonedeadpunk | We in #openstack-ansible were just asked what can be done http://seclab.cs.sunysb.edu/seclab/pubs/asiaccs16.pdf and it feels what they write about RPC is a bit stupid since you have keystone admin credentials stored on each compute... | 18:26 |
| sean-k-mooney | you dont nessisarly have keystone admin creds | 18:27 |
| sean-k-mooney | in fact you shoudl not have keystone admin creds | 18:27 |
| noonedeadpunk | well, ok, service creds | 18:27 |
| noonedeadpunk | but it would still have kind of admin privileges? | 18:27 |
| sean-k-mooney | ok yes | 18:27 |
| sean-k-mooney | iddeally you woudl use application credentils | 18:27 |
| sean-k-mooney | am yes nova need to all some other services as admin | 18:28 |
| noonedeadpunk | so the idea there was - how much you fucked if one compute node is hacked :) | 18:28 |
| sean-k-mooney | but those service creditely shoudl only give admin on those spercices | 18:28 |
| sean-k-mooney | it depend on what you put there | 18:29 |
| sean-k-mooney | if you use app creds you can avoid using a password in your config | 18:29 |
| noonedeadpunk | I don't think it's still smth that keystone has out of the box? I mean even with all these scoped tokens you would still need to mess up with policies to get yourself covered? | 18:29 |
| sean-k-mooney | which makes revocation simpler | 18:29 |
| noonedeadpunk | like credential per compute... hm... | 18:30 |
| noonedeadpunk | that's interesting idea | 18:30 |
| sean-k-mooney | yep you can do cred per service per compute today | 18:30 |
| sean-k-mooney | if you really want too | 18:30 |
| sean-k-mooney | also if you are deploying in contiaenr you dont need the nova.conf to be visable to the nova_libvirt container | 18:31 |
| sean-k-mooney | so even if you break out of the vm you then have to also escap the container and selix and the file permisions to read the config file | 18:31 |
| sean-k-mooney | at least with ooo that is | 18:31 |
| sean-k-mooney | so for ooo the vms are runing itn the filesystem namesapce of nova_libvirt as the qemu user | 18:32 |
| sean-k-mooney | that contaienr does not have the nova.conf visable to it | 18:32 |
| sean-k-mooney | and selinux + file system permission woudl prevent the vm process form reading it | 18:32 |
| clarkb | wouldn't that be true for any setup running libvirt as not the nova user and basic file permissions? | 18:32 |
| clarkb | ya that | 18:33 |
| clarkb | however libvirt is privileged | 18:33 |
| noonedeadpunk | yeah, I guess in ubuntu apparmour does kind of same anyway | 18:33 |
| clarkb | so might be defeatable? | 18:33 |
| sean-k-mooney | yep even in a normal rpm/deb install selinux/apparmor + file system permisions can help | 18:33 |
| melwitt | thanks gmann++ | 18:33 |
| sean-k-mooney | clarkb: libvirt is but qemu does not run with the same user or group as libvirt | 18:33 |
| clarkb | aha got it | 18:34 |
| noonedeadpunk | but yeah, I got the idea:) At least I thought that we do smth stupid by placing access to keystone in nova-compute.conf but no, it's required) | 18:34 |
| opendevreview | Merged openstack/nova master: Add regression test for bug #1937084 https://review.opendev.org/c/openstack/nova/+/812126 | 18:34 |
| sean-k-mooney | so in general qemu shoudl not be part of the nova group so should not be able to read nova config files | 18:34 |
| sean-k-mooney | nova is often part of the qemu/libvirt group but that direction makes sense | 18:35 |
| sean-k-mooney | since nova need to be able to talk to libvirt at least and create files that qemu can read | 18:36 |
| melwitt | lyarwood: in case you didn't see, I addressed your comments on https://review.opendev.org/c/openstack/python-openstackclient/+/818306 | 18:37 |
| sean-k-mooney | noonedeadpunk: so ya i have not read that paper but im not sure how valid it is and if they have correctly deployed thigns | 18:38 |
| sean-k-mooney | noonedeadpunk: with devstack for example most thigns are owned by the stack user so the filesystem doesnt do much for you but opensack ansible, kolla or ooo should provide much more protection | 18:39 |
| noonedeadpunk | wasnt it reading in details, but yeah, it weird, as they read nova.conf, get rabbitmq creds, then were snifffing tokens (why when they could jsut take that from keystone_authotoken?) | 18:40 |
| noonedeadpunk | but yeah, I got idea, thanks sean-k-mooney a lot! | 18:40 |
| noonedeadpunk | as always, I got bunch of good ideas how to improve things) | 18:41 |
| *** tbachman_ is now known as tbachman | 18:43 | |
| sean-k-mooney | noonedeadpunk: looking at https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_pre_install.yml#L85-L121 by the way it looks like osa is already creating /etc/nova in the nova user and group | 18:43 |
| noonedeadpunk | yep, sure we do that! | 18:43 |
| sean-k-mooney | you likely dont want to use 755 for the mode | 18:43 |
| sean-k-mooney | https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_pre_install.yml#L92 | 18:43 |
| sean-k-mooney | since that is world readable | 18:44 |
| noonedeadpunk | good point | 18:44 |
| sean-k-mooney | say 750 | 18:44 |
| sean-k-mooney | or 650 for files | 18:44 |
| noonedeadpunk | we should totally review that asap... | 18:45 |
| noonedeadpunk | nova.conf is stored 0640 though | 18:45 |
| sean-k-mooney | ack then its likely fin already | 18:45 |
| sean-k-mooney | nova is part fo the libvirt group https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/drivers/kvm/nova_compute_kvm.yml#L56-L63 | 18:45 |
| noonedeadpunk | https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_post_install.yml#L58-L81 | 18:45 |
| sean-k-mooney | which is corrct | 18:45 |
| sean-k-mooney | and i dont see qemu or libvirt beign added to nova | 18:46 |
| sean-k-mooney | so for osa they should not be able to read the nova.conf | 18:46 |
| noonedeadpunk | yep, thanks a lot for checking that! We still should review directory permissions | 18:47 |
| sean-k-mooney | no worries, security is important | 18:54 |
| sean-k-mooney | eventually i hope ooo or some of the installer will start using https://docs.openstack.org/keystone/queens/user/application_credentials.html | 18:54 |
| sean-k-mooney | in the config | 18:54 |
| noonedeadpunk | I wish there was an ansible module ready for that... Shouldn't be that hard to write one though | 18:58 |
| noonedeadpunk | but we have huge backlog of missed features anyway now :( we still haven't managed to implement service_tokens https://docs.openstack.org/keystone/latest/admin/manage-services.html#configuring-service-tokens | 19:01 |
| noonedeadpunk | but the biggest issue I see with application credentials - is how to template config. So they should be stored somewhere after being created and managed... | 19:04 |
| noonedeadpunk | and kind of rotating of regular credentials is not _that_ hard - a matter of re-running playbook with specific tags.... | 19:15 |
| noonedeadpunk | I think the most challendging thing is to get aware that they got compromised | 19:15 |
| opendevreview | Ade Lee proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519 | 19:37 |
| *** tbachman_ is now known as tbachman | 19:50 | |
| *** tbachman_ is now known as tbachman | 20:32 | |
| opendevreview | Merged openstack/nova master: Make API fixture pass roles https://review.opendev.org/c/openstack/nova/+/819907 | 21:02 |
| *** tbachman_ is now known as tbachman | 21:22 | |
| *** dasm is now known as dasm|off | 21:46 | |
| *** tbachman_ is now known as tbachman | 22:24 | |
| opendevreview | Merged openstack/nova master: Update Interop doc https://review.opendev.org/c/openstack/nova/+/816980 | 22:27 |
| opendevreview | Merged openstack/nova master: api-ref: Adjust BFV rescue non-support note. https://review.opendev.org/c/openstack/nova/+/818823 | 22:28 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
