Tuesday, 2021-08-24

*** yoctozepto8 is now known as yoctozepto00:33
*** yoctozepto6 is now known as yoctozepto01:21
*** yoctozepto3 is now known as yoctozepto02:12
*** yoctozepto8 is now known as yoctozepto03:43
*** yoctozepto8 is now known as yoctozepto03:57
*** yoctozepto0 is now known as yoctozepto04:12
*** yoctozepto1 is now known as yoctozepto04:40
*** yoctozepto7 is now known as yoctozepto05:43
*** yoctozepto0 is now known as yoctozepto06:24
*** redrobot0 is now known as redrobot06:29
opendevreviewBalazs Gibizer proposed openstack/nova stable/wallaby: Avoid modifying the Mock class in test  https://review.opendev.org/c/openstack/nova/+/80575906:47
opendevreviewJohannes Kulik proposed openstack/nova master: api: Log actual number of UUIDs queried  https://review.opendev.org/c/openstack/nova/+/80578606:57
*** yoctozepto9 is now known as yoctozepto07:00
gibifungi: I re-opened the bug https://bugs.launchpad.net/nova/+bug/1927677 should I reopne the OSA part of it too?07:24
*** rpittau|afk is now known as rpittau07:27
*** yoctozepto1 is now known as yoctozepto07:52
admin1when i do a rescue, in horizon , it asks for pass .but that pas does not work for root or debian/ubuntu .. 08:19
admin1so how exactly is a rescue done 08:19
lyarwoodadmin1: so we attempt to inject the password during the rescue into the rescue image, either directly or via a configdrive and cloud-init08:25
lyarwoodadmin1: direct injection into the rescue image depends on the config in your env08:26
lyarwoodadmin1: and injection by configdrive depends on cloud-init being used by the rescue image08:26
lyarwoodhttps://github.com/openstack/nova/blob/2d2fb2debab9e1705231ebee1bd9305c5b22fa15/nova/virt/libvirt/driver.py#L4635-L4638 controls the direct injection08:27
admin1is there a good recommended resuce image that works ? 08:29
opendevreviewLee Yarwood proposed openstack/nova master: api: Introduce microversion 2.89 adjusting os-volume_attachments  https://review.opendev.org/c/openstack/nova/+/80427508:39
opendevreviewMerged openstack/nova master: address open redirect with 3 forward slashes  https://review.opendev.org/c/openstack/nova/+/80565408:50
*** akekane_ is now known as abhishekk08:53
lyarwoodadmin1: You can use pretty much anything, I tend to virt-customize the image with a password etc before use08:59
lyarwoodadmin1: but there are plenty of small rescue images made for this purpose with physical hosts like http://tinycorelinux.net/08:59
lyarwoodhttps://www.techradar.com/uk/best/best-linux-repair-and-rescue-distros has loads09:00
admin1lyarwood, thanks09:11
opendevreviewLee Yarwood proposed openstack/python-novaclient master: Microversion 2.89 - os-volume_attachments  https://review.opendev.org/c/openstack/python-novaclient/+/80502209:32
lyarwoodmelwitt: https://review.opendev.org/c/openstack/nova/+/805667 - would you mind hitting this when you're online to resolve a gate-failure09:51
opendevreviewLee Yarwood proposed openstack/nova master: api: Introduce microversion 2.89 adjusting os-volume_attachments  https://review.opendev.org/c/openstack/nova/+/80427510:44
gibilyarwood: I have doc questions in https://review.opendev.org/c/openstack/nova/+/804275 otherwise it looks good11:00
opendevreviewBalazs Gibizer proposed openstack/placement master: Bump os-traits to latest 2.6.0  https://review.opendev.org/c/openstack/placement/+/80583011:06
opendevreviewLee Yarwood proposed openstack/nova master: api: Introduce microversion 2.89 adjusting os-volume_attachments  https://review.opendev.org/c/openstack/nova/+/80427511:09
fungigibi: don't worry about the ossa task, i'll reopen it in a bit11:11
fungiand thanks!11:11
gibifungi: thanks you11:11
fungiunfortunately i have a number of them to catch up on after breakfast11:12
gibiis it a known fact that the metadata service will only have generic metadata while the config drive will have virt driver specific metadata? as per https://bugs.launchpad.net/nova/+bug/194089411:42
opendevreviewMerged openstack/nova master: docs: Drop references to non-filter scheduler drivers  https://review.opendev.org/c/openstack/nova/+/77364511:48
* lyarwood heads offline for a few hours, back this afternoon11:54
opendevreviewBalazs Gibizer proposed openstack/nova master: Fix documentation about cpu topologies  https://review.opendev.org/c/openstack/nova/+/80583311:55
gibistephenfin: an simple doc fix ^^11:55
sean-k-mooneygibi: actully that is only true if you are suing the new cpu_share_set and cpu dedicated set options12:53
sean-k-mooneyalthough you are not really chanign the content12:54
sean-k-mooneyso i gues your change is fine12:54
sean-k-mooneyyour main correct is with to without which indeed was a bug12:54
sean-k-mooneygibi: in case you were not aware if you use vcpu_pin_set and dont use the new options we still support the old cpu pinning behavior eventough that is deprecated12:55
sean-k-mooneystephenfin: was going to submit patchs to remove that but im not sure he had time to do that yet12:56
gibiack12:56
opendevreviewMerged openstack/nova master: api: Log actual number of UUIDs queried  https://review.opendev.org/c/openstack/nova/+/80578613:37
opendevreviewMerged openstack/nova master: Retry on vmware create_vm when it fails  https://review.opendev.org/c/openstack/nova/+/76458613:40
opendevreviewMerged openstack/nova master: Add missing [oslo_reports] options  https://review.opendev.org/c/openstack/nova/+/80456813:41
opendevreviewsean mooney proposed openstack/nova stable/wallaby: address open redirect with 3 forward slashes  https://review.opendev.org/c/openstack/nova/+/80581813:45
opendevreviewStephen Finucane proposed openstack/nova stable/wallaby: Avoid modifying the Mock class in test  https://review.opendev.org/c/openstack/nova/+/80575915:25
opendevreviewStephen Finucane proposed openstack/nova stable/victoria: Avoid modifying the Mock class in test  https://review.opendev.org/c/openstack/nova/+/80582315:26
gmannlyarwood: takashi comments are good to fix to correct api-ref. how you will like to do in follow up or in same patch ? https://review.opendev.org/c/openstack/nova/+/804275/1015:40
gibifyi, nova meeting starts in 15 minutes here in the channel15:44
gibi#startmeeting nova16:00
opendevmeetMeeting started Tue Aug 24 16:00:24 2021 UTC and is due to finish in 60 minutes.  The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
*** rpittau is now known as rpittau|afk16:00
gibio/16:00
elodilleso/16:03
gibithanks elodilles, I felt alone16:03
elodilles:]16:03
gibiis it only the two of us or other still listening quietly?16:04
gibiI won't do a full meeting just for the two of us16:05
gibilets see what is important from the agenda16:06
gibiwe have one critical bug  https://bugs.launchpad.net/nova/+bug/1940555 around SQLAlchemy and the fix is being discussed in: https://review.opendev.org/c/openstack/nova/+/80566316:06
gibiI guess it is a blocker for tripleoo16:07
gibioverall the nova gate is is an acceptable state. we are landing patches16:08
gibiabout the PTG: every info is in the PTG etherpad #link https://etherpad.opendev.org/p/nova-yoga-ptg16:08
gibiwe will have FF next week 16:09
gibirelease todos are being tracked in #link https://etherpad.opendev.org/p/nova-xena-rc-potential16:09
gibiif you see release critical bugs then please tag them with xena-rc-potential tag16:09
gibias per elodilles the stable gate also look healthy16:10
gibiis there any question before I close this?16:10
elodillesnope :X16:10
gibithen lets close this16:12
gibi#endmeeting16:12
opendevmeetMeeting ended Tue Aug 24 16:12:10 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:12
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2021/nova.2021-08-24-16.00.html16:12
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2021/nova.2021-08-24-16.00.txt16:12
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2021/nova.2021-08-24-16.00.log.html16:12
elodilleso/16:13
rmart04Hello All, I know this is a deployment question but its pretty nova specific, hopefully you someone might be able to help. When you perform live migration of an instance with a block device, and the disk is pre-live migrated using scp, is there anyway to configure that connection string to be via IP address rather than by hostname? For cases where ssh is only bound to a specific interface? That might not be16:37
rmart04 what DNS points at?16:37
sean-k-mooneygibi: elodilles  soory was working on something downstream so missed meeting16:39
sean-k-mooneybut ya the sql fix is on my todo list. we will see if we can get that fixed this week16:40
sean-k-mooneyelodilles: can you priortiese review of this backport https://review.opendev.org/c/openstack/nova/+/80581816:43
elodillessean-k-mooney: sure, looking16:46
rmart04@sean-k-mooney if you had any thoughts on the above that would be awesome :D 16:47
sean-k-mooneyrmart04: i think so one sec16:57
sean-k-mooneyrmart04: https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.live_migration_inbound_addr16:58
rmart04thanks for having a think, yes we have that set to the host IP that ssh listens on, and as far as I can tell that is working well for live migrations (with rbd backends). I'm wondering if the pre_migration requires an additional variable to be set?16:59
sean-k-mooneyill check if there is somthing for the scp specifcially17:00
sean-k-mooneyif not you could hardcode the hostname to ip mapping in /etc/hosts17:00
sean-k-mooneyso the two ways we have for copyting the block sotrage is scp or rsync https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/libvirt/volume/remotefs.py#L172-L22217:02
sean-k-mooneyi think this is how libvirt is calling it indirectly https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/libvirt/utils.py#L214-L25117:03
rmart04hmm, we did have hardcoded etc/hosts but we were trying to remove that dependency as we have a lot of hosts and updating this and restarting nova_compute on hundreds of boxes every time was getting tiresome (even in ansible)17:04
rmart04maybe we might have to look to re-implement that17:05
rmart04I guess I should mention we are still on stein too 17:05
rmart04_sad face_17:06
gansomelwitt, lyarwood, bauzas: If you have a spare moment could you please take a look at this backport that already has a +2? Keep in mind that the merge conflict is also proposed and also has a +2: https://review.opendev.org/c/openstack/nova/+/805304 Thanks in advance!17:08
sean-k-mooneyim not direclty seeing it. to be honest i tought for live migration libvirt actully did the copy and we only used scp for cold migration17:13
sean-k-mooneyah so e do this here https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/libvirt/driver.py#L9855-L987017:15
sean-k-mooneyrmart04: so we do the copy form the dest passing the source host name17:15
sean-k-mooneyalthough that is for the config drive17:15
rmart04the thing that flagged this up today was migrating amphora, unsure if octavia uses config drive by default... could be17:17
rmart04i'll check17:17
johnsomYes17:18
rmart04ahhhh.... yes, disk.config17:18
sean-k-mooneyyou could work around this by using vfat config drives17:20
sean-k-mooneyalso we dont need this anymore so we can remove it on master i think17:20
sean-k-mooneythe libvirt bug has been fixed17:20
sean-k-mooneyrmart04: so ya https://github.com/openstack/nova/blob/stable/stein/nova/conf/configdrive.py#L30 you can use vfat if that is supported by the guests instead if you need too17:22
opendevreviewStephen Finucane proposed openstack/nova master: db: Add migration to resolve shadow table discrepancies  https://review.opendev.org/c/openstack/nova/+/80573817:23
rmart04ok thanks for all the info, will see how i get on tomorrow17:27
opendevreviewLee Yarwood proposed openstack/nova master: fup: Fix os-volume_attachments api-ref parameters  https://review.opendev.org/c/openstack/nova/+/80587618:04
lyarwoodgmann: ack fixed in a fup on top of the api change to avoid the respin, apologies again!18:06
lyarwoodganso: ack I'll review this evening18:06
opendevreviewLee Yarwood proposed openstack/python-novaclient master: Microversion 2.89 - os-volume_attachments  https://review.opendev.org/c/openstack/python-novaclient/+/80502218:11
NobodyCamGood Morning Nova folks, was disk_available_least hypervisor stat removed in Ussuri?18:14
NobodyCamfor Ironic hypervisors?18:14
sean-k-mooneyi dont think so in ussuri but you should not be use the disk filter in ussuri for ironic or otherwise if that is why you are asking18:17
NobodyCamhey hey Sean, in queens that was a value I used to validate the node was ready for deployment. 18:19
sean-k-mooneywhy not use the node status in ironic for that18:21
sean-k-mooneyits ready for deployment when its in the avaiable state18:21
NobodyCamwith that we hit no valid hosts, I should add that when we provision we use force_node to deploy to specific nodes18:22
NobodyCamhere is a sample if what I was seeing:18:24
NobodyCamhttps://www.irccloud.com/pastebin/fuSfO22M/18:24
sean-k-mooneyNobodyCam: ironic stopped reporting resouces for cpu ram and disk several cycles ago18:25
NobodyCamack18:26
sean-k-mooneyi dont rember off the top of my head when we moved to trackign ironic nodes with custom resouce classes but im sure tha tlikely had an effect of os-hyperviors output eventually18:26
NobodyCamyep18:26
sean-k-mooneyin wally we have removed disk_avaiable_lset https://specs.openstack.org/openstack/nova-specs/specs/wallaby/implemented/modernize-os-hypervisors-api.html18:27
NobodyCam++ 18:27
sean-k-mooneyNobodyCam: what you could do is see if the node exist in os-hyperviors rather then check for that value specifically18:28
sean-k-mooneybut still the expected way to check this would be via ironic18:28
NobodyCamyea, let me try a few thing... yea we do check ironic18:28
sean-k-mooneyNobodyCam: i would proably check the state filed 18:29
sean-k-mooneyand mayby status18:29
sean-k-mooneyso check for sate:up status:enabled18:29
sean-k-mooneythat should mean its schduleable18:29
NobodyCam++ Awesome, let try some deploys with that... Thank you very much sean-k-mooney !!!!18:30
NobodyCamI'll let you know what I find18:30
sean-k-mooneycool18:30
sean-k-mooneythe other thing you could check woudl be placment to see that the node is there but i think os-hyperviors will only have the node after its in placment anyway18:31
opendevreviewMerged openstack/nova master: scheduler: Merge driver into manager  https://review.opendev.org/c/openstack/nova/+/77364418:31
opendevreviewMerged openstack/nova master: tests: Merge 'test_utils', 'test_scheduler_utils'  https://review.opendev.org/c/openstack/nova/+/77364618:31
opendevreviewMerged openstack/nova master: conf: Remove deprecated aliases  https://review.opendev.org/c/openstack/nova/+/77364718:31
lyarwoodmelwitt: https://review.opendev.org/c/openstack/placement/+/679441/ - if you're happy to look at the nits from me and Sean in a fup I'm happy to +W this now18:55
melwittlyarwood: sure thing, sorry I had only skimmed them so far18:57
lyarwoodmelwitt: no issues they are nits after all, I'll look at the AttributeCache change now18:58
melwittlyarwood: thanks, appreciate it. and I will look at the respin of the nova-manage volume patch today19:00
lyarwoodmelwitt: thanks, https://review.opendev.org/c/openstack/nova/+/805667 and https://review.opendev.org/c/openstack/nova/+/802317 could also use another core if you get time19:01
lyarwoodthe first fixing a gate-failure I introduced19:02
melwittwill do, added to the queue. thanks19:02
lyarwoodsean-k-mooney: are you about?19:30
sean-k-mooneyyes19:31
lyarwoodsean-k-mooney: https://zuul.opendev.org/t/openstack/build/ddb6cf53e65a404c89dac4dc4ca1cd51/log/controller/logs/screen-n-cpu.txt#13396 & https://e31f9be4742ab987838e-5d50e8ef24c23b6b3512cdd707b25e49.ssl.cf1.rackcdn.com/804286/4/gate/tempest-ipv6-only/ddb6cf5/testr_results.html - So I'm seeing a test failure dump out console logs suggesting that CirrOS is still trying and failing to load apic related stuff even with the feature disabled19:32
lyarwoodin the domain19:32
lyarwoodI wonder if we actually want <ioapic driver='qemu'/>19:35
sean-k-mooneyi dont think soe but we migtht want driver='none'19:36
sean-k-mooney wait ioapic19:36
sean-k-mooneythat is different from the apic19:37
lyarwoodyeah19:37
lyarwoodioapic19:37
lyarwoodTune the I/O APIC. Possible values for the driver attribute are: kvm (default for KVM domains) and qemu which puts I/O APIC in userspace which is also known as a split I/O APIC mode. Since 3.4.0 (QEMU/KVM only)19:37
lyarwoodhttps://libvirt.org/formatdomain.html#hypervisor-features19:37
sean-k-mooneylet me look at the kernel message again19:38
sean-k-mooneythe one i was fixing initally was a hagn related to the apic not the ioapic19:38
sean-k-mooney ok so that case ya is deffinetly the io apic19:39
sean-k-mooneyam i would still be tempted to say driver=none19:39
sean-k-mooneybut that is not documented19:39
sean-k-mooneylyarwood: i dont think changing it to qemu will help19:40
lyarwoodWhy given it's suggesting you should use that when using QEMU19:41
sean-k-mooneywell libvirt woudl do that automtically19:41
sean-k-mooneykvm is only the default for kvm domains19:42
sean-k-mooneyit wont use kvm on a qemu domain19:42
lyarwoodAre you sure libvirt will add this automatically to the domain?19:43
sean-k-mooneywell we are not requesting it if it provdie an api it will set the correct drvier im pretty sure19:43
sean-k-mooneyif no i would consider that a libvirt bug 19:44
sean-k-mooneyone that we likely should not fix in nova we could but that not how i would expect this to work19:44
sean-k-mooneywithout requesting an ioapic by the way i would not expect to get one19:44
sean-k-mooneyhttps://zuul.opendev.org/t/openstack/build/ddb6cf53e65a404c89dac4dc4ca1cd51/log/controller/logs/screen-n-cpu.txt#748919:44
sean-k-mooneywe are not requesting it 19:44
sean-k-mooneyperhaps its nto considerd a toggelable feature 19:45
sean-k-mooney"All features are listed within the features element, omitting a togglable feature tag turns it off."19:45
lyarwoodokay so we've never enabled the ioapic somehow?19:46
lyarwoodhttps://paste.centos.org/view/e7a2611c - we've only had three hits of this recently in our gate and I think two of these were before the latest workaround apic patch landed19:47
lyarwoodso maybe this can wait until kashyap is back19:47
sean-k-mooneywe have not enabeld it in the xml no19:49
sean-k-mooneyhttps://zuul.opendev.org/t/openstack/build/ddb6cf53e65a404c89dac4dc4ca1cd51/log/controller/logs/libvirt/libvirt/qemu/instance-00000001_log.txt19:51
sean-k-mooneywe also done seam to be enablit it on the qemu commandline19:51
sean-k-mooneyso either we need to explictly disable it which there seams to be no documented way to do or this is somethign qemu can disabel but libvirt cant19:52
opendevreviewArtom Lifshitz proposed openstack/nova master: Reproduce bug 1941005  https://review.opendev.org/c/openstack/nova/+/80588219:56
artomsean-k-mooney, ^^ confirmed, resize with extra specs is b0rked19:56
artomWe don't update the request spec19:56
sean-k-mooneystill on master?19:56
sean-k-mooneyit used to work in the past19:56
artomYep19:56
artomstephenfin's patch fixed the numa topology case, not the general case19:57
sean-k-mooneyeven before stephens patch19:57
sean-k-mooneywe regressed this at some point 19:57
sean-k-mooneyim sure we used to update it at least in memory19:57
artomIt's kinda tricky to see - *adding* stuff normally fails as expected if no hosts can provide it, because the resource claim uses the flavor and image directly19:57
sean-k-mooneywe may not have commited it back to the db19:57
sean-k-mooneybut we did use them at some point for scudliign and creatign the xml19:58
artomBut *removing* stuff can go wrong, because despite having hosts that can fit the new thing, the scheduler tries to schedule according to the old request spec19:58
sean-k-mooneyok well its broke now so we should fix an backport in anycase19:59
artomYar20:00
sean-k-mooneybut im pretty sure this used to work in icehouse as im pretty sure i tested resizeing to and from vms with pci deviecs back then20:00
artomPerhaps - icehouse was a while ago - and it's pretty specific to hit20:01
lyarwoodcuts to titanic 80 years gif20:01
sean-k-mooneyhehe20:02
sean-k-mooneyyes it was i just remember testing this in the past althogu hit was specific realted to cpu pinnign and pci passtough so i dont know20:02
sean-k-mooneyno one complained and the xmls seams to eb right althgu we know that there are several placese where this si broken for move operations in the past so im not that surprised20:03
sean-k-mooneyartom: dont we have whitebox test fo this by the way20:04
artomsean-k-mooney, we might...20:04
artomBut say you have two hosts with PCI devices - and you resize the instance from "PCI" to "no PCI"20:04
artomIt'll work fine - scheduling will pass with the bad old request spec because all hosts have PCI devices20:05
artomIt'll only bit you if you only have left hosts with *no* PCI devices20:05
artom*bite you20:05
artomBecause then the old request spec can't be scheduled anywhere20:05
sean-k-mooneyartom: we not if we were validating the pci claims in the test like we should be20:05
sean-k-mooneybut looking we dont have test for this20:05
artomI think claims use the flavor and image directly, so that part's fine20:06
artomBy pure luck, really20:06
artomBut it's fine :)20:06
sean-k-mooneywell we do20:06
sean-k-mooneyhttps://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/whitebox_tempest_plugin/api/compute/test_vpmu.py20:06
sean-k-mooneythe vpum tests20:06
sean-k-mooneyi know we added this recently for a feature20:06
sean-k-mooney*knew20:07
sean-k-mooneythe probel with https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/whitebox_tempest_plugin/api/compute/test_vpmu.py#L64-L7420:07
sean-k-mooneyis that we are not asserting anyting baout the request spec20:07
sean-k-mooneyjust the xml20:07
sean-k-mooneywhich means on the destination host we are correctly using the new flavor20:08
sean-k-mooneybut as you said the scudler is not nessisarly useing the correct extra specs20:08
sean-k-mooneyartom: so you have a repoducer do you know where the bug is?20:09
sean-k-mooneyor is that still TBD20:09
artomsean-k-mooney, well, I know vaguely where it is - we went through that code together to figure out the OSP 10 resize issue20:14
artomBut I don't have a fix yet, will need ot go through it again to figure out the best place for it20:14
artomThat code is full of potential races and side effects20:14
sean-k-mooneyack ya ok.20:15
opendevreviewMerged openstack/nova master: Fix documentation about cpu topologies  https://review.opendev.org/c/openstack/nova/+/80583320:38
lyarwoodelodilles: https://review.opendev.org/c/openstack/nova/+/795435 - would you mind taking a look at this if you have time?20:43
NobodyCamsean-k-mooney: FYI "resource provider / generation" for the win20:43
opendevreviewMerged openstack/nova stable/wallaby: Fix 1vcpu error with multiqueue and vif_type=tap  https://review.opendev.org/c/openstack/nova/+/80530420:52
*** tbachman is now known as Guest530120:57
*** tbachman is now known as Guest530221:01
opendevreviewMerged openstack/nova master: api: Introduce microversion 2.89 adjusting os-volume_attachments  https://review.opendev.org/c/openstack/nova/+/80427521:38
opendevreviewMerged openstack/nova master: Follow up from bp/pci-socket-affinity series  https://review.opendev.org/c/openstack/nova/+/77955622:08

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!