| opendevreview | melanie witt proposed openstack/nova stable/xena: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/821348 | 01:19 |
|---|---|---|
| opendevreview | melanie witt proposed openstack/nova stable/wallaby: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/821349 | 01:21 |
| opendevreview | melanie witt proposed openstack/nova stable/victoria: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/821350 | 01:26 |
| opendevreview | melanie witt proposed openstack/nova stable/ussuri: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/821351 | 01:28 |
| opendevreview | melanie witt proposed openstack/nova stable/train: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/821352 | 01:30 |
| opendevreview | Wenping Song proposed openstack/nova master: Fill the AcceleratorRequestBindingFailed exception msg info https://review.opendev.org/c/openstack/nova/+/817326 | 03:26 |
| opendevreview | Wenping Song proposed openstack/nova master: Generate cyborgclient when there is device_prfoile in instance https://review.opendev.org/c/openstack/nova/+/821368 | 05:45 |
| opendevreview | Wenping Song proposed openstack/nova master: Cleanup guest process when vm evacuate failed and then deleted https://review.opendev.org/c/openstack/nova/+/814746 | 08:00 |
| nautik | If anyone wants to review this MR you are welcome: https://review.opendev.org/c/openstack/nova/+/781076 | 08:14 |
| nautik | There are a few "build failed" messages but I think they are just temporary / testing infra issues | 08:15 |
| *** brinzhang_ is now known as brinzhang | 09:09 | |
| *** songwenping is now known as swp | 09:18 | |
| *** swp is now known as songwenping | 09:18 | |
| opendevreview | Sylvain Bauza proposed openstack/nova-specs master: Repropose boot a vm with unaddressed port https://review.opendev.org/c/openstack/nova-specs/+/821376 | 09:37 |
| opendevreview | Pierre Libeau proposed openstack/nova master: Nova resize don't extend disk in one specific case https://review.opendev.org/c/openstack/nova/+/820531 | 10:19 |
| opendevreview | Wenping Song proposed openstack/nova master: Fill the exception msg https://review.opendev.org/c/openstack/nova/+/821385 | 10:27 |
| opendevreview | Pierre-Samuel Le Stang proposed openstack/nova-specs master: Implements: blueprint soft-delete-instance-actions https://review.opendev.org/c/openstack/nova-specs/+/821387 | 10:43 |
| pslestang | hello! | 10:46 |
| gibi | o/ | 10:51 |
| kashyap | pslestang: Most ppl (at least in the West) might be off due to the year-end PTO. So expect delayed responses | 10:57 |
| *** sean-k-mooney1 is now known as sean-k-mooney | 11:31 | |
| gibi | fyi, it seems that the nova-ceph-multistore job is broken on master https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-multistore&project=openstack/nova I will file a bug | 11:31 |
| gibi | https://zuul.opendev.org/t/openstack/build/722c6caf8e454849b897a43bcf617dd2/log/job-output.txt#9419 | 11:31 |
| gibi | example ^^ | 11:31 |
| sean-k-mooney | ROR: Cannot uninstall 'logutils'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. | 11:32 |
| gibi | yepp that one | 11:32 |
| gibi | the last 4 build hit the same | 11:32 |
| sean-k-mooney | ya so that will proably break other things | 11:32 |
| sean-k-mooney | i guess we just need to update devstack | 11:32 |
| sean-k-mooney | to remove that package or mask it | 11:32 |
| sean-k-mooney | https://packages.ubuntu.com/source/focal/python-logutils | 11:33 |
| sean-k-mooney | looks like its packaged for openstack in debian | 11:34 |
| sean-k-mooney | so i guess it should be safe to remove | 11:35 |
| gibi | filed a bug https://bugs.launchpad.net/nova/+bug/1954427 | 11:37 |
| sean-k-mooney | https://review.opendev.org/c/openstack/devstack/+/821394 | 11:41 |
| sean-k-mooney | not sure if that will work but we can see if it fixes it | 11:41 |
| gibi | sean-k-mooney: awesome, I was still searching for the place in devstack to fix it | 11:41 |
| sean-k-mooney | that might be too early | 11:42 |
| sean-k-mooney | lookign at the logs its installed as part of installing ceph | 11:42 |
| sean-k-mooney | so i might need to fix this in the ceph plugin instead | 11:42 |
| sean-k-mooney | its pulled in by ++ functions-common:apt_get:1207 : sudo DEBIAN_FRONTEND=noninteractive http_proxy= https_proxy= no_proxy= apt-get --option Dpkg::Options::=--force-confold --assume-yes install ceph libnss3-tools python3-rados python3-rbd | 11:42 |
| gibi | that can explain why we only see this breaking the ceph job only | 11:43 |
| sean-k-mooney | ill quickly add the same to the ceph plugin | 11:43 |
| gibi | this is what I call teamwork :) | 11:43 |
| sean-k-mooney | im going to push a patch that may not work as it might unistalll ceph again but if that is what happens we can just remove the egg infor files to work aroudn it as we do in teh fixup fucntion | 11:49 |
| gibi | OK | 11:50 |
| sean-k-mooney | https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/821396 | 11:50 |
| sean-k-mooney | im hoping its just a recommended package rather then a required one for those packages | 11:52 |
| sean-k-mooney | actully no it looks like its required | 11:52 |
| sean-k-mooney | it snot in the suggested or recommeded section so ill have to do it the other way | 11:53 |
| sean-k-mooney | ok ill try and check back on that leater but hopfully that will fix it | 12:01 |
| sean-k-mooney | gibi: i pushed the first draft of the healthcheck spec yesterday evenign by the way https://review.opendev.org/c/openstack/nova-specs/+/821279 if you are interested. it defnitly need to be reworked since half of it i wrote 3 weeks ago and i just finished off the other section last night but most of the contrent is at least there | 12:03 |
| gibi | sean-k-mooney: ack, I will read it today | 12:03 |
| sean-k-mooney | cool im going to start POCing the code next week and ill use that to also help inform the design | 12:04 |
| opendevreview | mitya-eremeev-2 proposed openstack/nova master: Delete bogus attachments. https://review.opendev.org/c/openstack/nova/+/820935 | 12:54 |
| *** bhagyashris_ is now known as bhagyashris | 13:31 | |
| opendevreview | Pierre-Samuel Le Stang proposed openstack/nova-specs master: Implements: blueprint soft-delete-instance-actions https://review.opendev.org/c/openstack/nova-specs/+/821387 | 13:35 |
| AlexShishebarov | Hello. I wanted to ask a question about the bug I reported. | 13:40 |
| AlexShishebarov | https://bugs.launchpad.net/nova/+bug/1951623 This bug is that during live migration of a VM with a disabled port (to a neutron), the migration stopes with the error Timed out waiting for events “network-vif-plugged” (for disabled port). Will the change https://review.opendev.org/c/openstack/nova/+/767368 fix this problem? | 13:41 |
| sean-k-mooney | AlexShishebarov: this is currently being fixed | 14:06 |
| sean-k-mooney | AlexShishebarov: https://review.opendev.org/c/openstack/nova/+/819494 shoudl correct the issue | 14:06 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/767368 will not | 14:08 |
| AlexShishebarov | sean-k-mooney: According to the code does not resolve this bug (in my opinion). Nova will continue to wait for events from the neutron. But events from a neutron will not come for disabled port. https://review.opendev.org/c/openstack/nova/+/819494 nova/network/model.py(line 556) method get_live_migration_plug_time_events | 14:38 |
| sean-k-mooney | we shoudl be waiting for events for live migration | 14:43 |
| sean-k-mooney | for cold migration the even moved | 14:43 |
| sean-k-mooney | it went form bind time to plug tiem on revert | 14:44 |
| sean-k-mooney | you are correct that we shoudl skip disabeld ports | 14:45 |
| sean-k-mooney | well kind of | 14:46 |
| sean-k-mooney | it shoudl not actully be required as neuton shoudl be sending the event but they dont currntly | 14:46 |
| sean-k-mooney | we can skip them to work around the neutron bug but neutron shold be send network-vif-plugged event even for disabled ports | 14:46 |
| sean-k-mooney | for now we shoudl be checking the vif status here https://review.opendev.org/c/openstack/nova/+/819494/5/nova/network/model.py#494 | 14:48 |
| AlexShishebarov | I tried to solve the problem this way, but the nova saves the port state as "active" even if it is turned off to a neutron. https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L3183-L3185 | 14:53 |
| AlexShishebarov | in neutron | 14:55 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: functional: Add reproducer for #1907775 https://review.opendev.org/c/openstack/nova/+/766771 | 15:33 |
| gibi | stephenfin: I hope you are OK that I refreshed ^^ | 15:33 |
| gibi | I would like to add the rejection to the aggregate API to fix the bug | 15:33 |
| gibi | but I figured out that we have more cases to handle | 15:34 |
| gibi | so I added extra tests | 15:34 |
| opendevreview | sean mooney proposed openstack/nova-specs master: Add spec for external cpu power managment https://review.opendev.org/c/openstack/nova-specs/+/821419 | 15:51 |
| sean-k-mooney | gibi: it looks like https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/821396/ works | 17:06 |
| gibi | sean-k-mooney: awesome | 17:06 |
| sean-k-mooney | there are some test failerues in the non voting jobs but they are unrelated | 17:07 |
| sean-k-mooney | who normally reivews that repo? do you know? | 17:07 |
| sean-k-mooney | i might check on the qa channel | 17:08 |
| gibi | I don't. lyarwood might know but he is already on PTO | 17:10 |
| sean-k-mooney | ya i just left a message on #openstack-qa | 17:13 |
| sean-k-mooney | honestly it proably wont be seen until moday at this point | 17:14 |
| gibi | gmann: ^^ | 17:14 |
| gibi | gmann: I think you have +2 rights in devstack-plugin-ceph | 17:14 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: functional: Add reproducer for #1907775 https://review.opendev.org/c/openstack/nova/+/766771 | 17:14 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Reject AZ changes during aggregate add / remove host https://review.opendev.org/c/openstack/nova/+/821423 | 17:14 |
| sean-k-mooney | ya anyone in devstack-core or qa-release or cinder-core | 17:17 |
| sean-k-mooney | https://review.opendev.org/admin/groups/8e51d8f39e33d932a40f03060b861db57a3d25d4,members | 17:17 |
| sean-k-mooney | on one hand its a gate blocker on the ohter hand its friday evening and its not near a milestone so it can proably wait till monday without it being a major problem | 17:19 |
| sean-k-mooney | i can send a mail to the list just to keep everyone in the loop | 17:20 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: Debug whitebox RBD direct download test failures https://review.opendev.org/c/openstack/nova/+/821424 | 17:21 |
| gmann | sean-k-mooney: gibi checking | 17:22 |
| gibi | thanks! | 17:22 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org is being quickly restarted for a configuration adjustment, and should return momentarily | 17:27 | |
| gibi | sean-k-mooney: left initial feedback on the healtcheck spec | 17:32 |
| gibi | and that I will end my week | 17:32 |
| gibi | see you next week o/ | 17:33 |
| gmann | gibi: sean-k-mooney did not get why it started failing now? is there any recent change triggered this? | 17:35 |
| opendevreview | Ghanshyam proposed openstack/nova master: DNM: testing nova-ceph-multistore fix https://review.opendev.org/c/openstack/nova/+/821435 | 17:43 |
| sean-k-mooney | gmann: not that i know if i suspect either neutron started requrieing a new version of python3-logging or there was a cpeh package update upstream | 17:47 |
| gmann | i can see only in l-c requirements which should not be installed by devstack | 17:54 |
| sean-k-mooney | ya i was checking on codeasearch | 17:56 |
| sean-k-mooney | not sure what is pulling it in to be honest | 17:56 |
| sean-k-mooney | sean@cloud:~$ apt-cache rdepends python3-logutils | 17:59 |
| sean-k-mooney | python3-logutils | 17:59 |
| sean-k-mooney | Reverse Depends: | 17:59 |
| sean-k-mooney | python3-pecan | 17:59 |
| sean-k-mooney | changeme | 17:59 |
| sean-k-mooney | so its comiy for either python3-pecan or changeme | 17:59 |
| sean-k-mooney | likely python3-pecan | 17:59 |
| sean-k-mooney | ya so thye had a release 2 days ago | 18:02 |
| sean-k-mooney | https://pypi.org/project/pecan/ | 18:02 |
| sean-k-mooney | and its used by neturon | 18:02 |
| sean-k-mooney | gmann: and we udpated the upper constratints this morning https://github.com/openstack/requirements/commit/e74f91e250d127ff825dad5219527d62efaeb740 | 18:02 |
| gmann | humm, it was there in pecan 1.4.1 which constraints updated in requirement yesterday https://github.com/pecan/pecan/blob/1.4.1/requirements.txt | 18:03 |
| sean-k-mooney | yep | 18:03 |
| sean-k-mooney | so that is likely what broke it | 18:03 |
| gmann | yeah but there is no change in previous version of pecan on logutils | 18:03 |
| sean-k-mooney | https://github.com/pecan/pecan/blob/master/requirements.txt#L6 | 18:03 |
| sean-k-mooney | the require logutile >=3 | 18:03 |
| gmann | yeah that is there in old version too https://github.com/pecan/pecan/blob/1.3.3/requirements.txt | 18:03 |
| sean-k-mooney | ya 8 years | 18:04 |
| sean-k-mooney | im not sure but that is the best i can come up with the version that is shipped in ubuntu is 0.3.4 i think | 18:04 |
| sean-k-mooney | 0.3.3-7 | 18:05 |
| sean-k-mooney | so ghatt should work | 18:05 |
| gmann | I am trying to check the passing nova-ceph-multistore job and it was not pulled there somwhoe https://zuul.openstack.org/build/590c5996ca1b402486bfe1c7e1d08535/log/job-output.txt | 18:05 |
| sean-k-mooney | its likely a ubuntu or cpeh packageing change | 18:07 |
| sean-k-mooney | to we use the disto ceph or ceph form ceph.com | 18:07 |
| sean-k-mooney | oh | 18:08 |
| sean-k-mooney | the ceph pcakages changed | 18:08 |
| sean-k-mooney | CEPH_PACKAGES='ceph libnss3-tools python3-rados python3-rbd' | 18:08 |
| sean-k-mooney | on passin g | 18:08 |
| sean-k-mooney | actully they end up being the same | 18:10 |
| sean-k-mooney | CEPH_PACKAGES='ceph libnss3-tools python3-rados python3-rbd' | 18:10 |
| gmann | yeah its same | 18:10 |
| sean-k-mooney | et:20 https://download.ceph.com/debian-pacific focal/main amd64 python3-rbd amd64 16.2.7-1focal [386 kB] | 18:12 |
| sean-k-mooney | vs | 18:12 |
| sean-k-mooney | 021-12-10 10:37:05.265864 | controller | Get:48 https://download.ceph.com/debian-pacific focal/main amd64 python3-rbd amd64 16.2.7-1focal [386 kB] | 18:12 |
| sean-k-mooney | so i guess the same version of ceph too | 18:12 |
| gmann | sean-k-mooney: commented about commit msg to add these info and then i am +2 as it is gate blocker https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/821396/comment/a2721f12_ea844e6b/ | 18:21 |
| gmann | i cannot find root cause but it is clear it is pulling it recently so removing it is no issue | 18:22 |
| gmann | I can try to ping other core if they are around | 18:23 |
| sean-k-mooney | gibi: thanks yes ill spellcheck it with gramerly on monday | 18:32 |
| sean-k-mooney | gmann: am if tis ok with you ill update teh commit on monday i was just about to leave for the day | 18:33 |
| gmann | sean-k-mooney: ok, or i can update for you if it is ok? and try to get other core if they are around gouthamr or so. so that we can get gate unblocked | 18:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/nova stable/wallaby: Ensure MAC addresses characters are in the same case https://review.opendev.org/c/openstack/nova/+/816883 | 18:37 |
| * gouthamr is around | 18:37 | |
| * gouthamr looks | 18:38 | |
| jamesbenson | Can anyone help with refstack tests failing due to additional-properties? | 20:06 |
| jamesbenson | These are, for example this test: tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_get_instance_action | 20:10 |
| jamesbenson | Gives me a: jsonschema.exceptions.ValidationError: Additional properties are not allowed ('updated_at' was unexpected) | 20:11 |
| gmann | jamesbenson: hi | 20:12 |
| gmann | jamesbenson: is it with upstream code or modified API? | 20:13 |
| gmann | In tempest, we have strict validation on API response on additional-properties and t will fail if any API return any additional field in response without microversion or in downstream | 20:13 |
| jamesbenson | okay, so I need to add the microversions to my tempest.conf then? Currently I don't have those. | 20:15 |
| jamesbenson | It with a local deployment | 20:15 |
| jamesbenson | @gmann https://gitlab.com/utsa-ics/osias We do regular testing and only ran into this issue with Victoria on. Ussuri wasn't giving any issues. But we haven't merged code for Victoria, Wallaby, or Xena yet because of these issues. | 20:17 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org is being restarted again for a plugin change, and should be back shortly | 20:17 | |
| gmann | jamesbenson: i see, please cap the max version for victoria in tempest conf as 2.87 max_microversion https://github.com/openstack/tempest/blob/93a902072fd9986f2bb660166552f37d9eb5bdbb/tempest/config.py#L381 | 20:19 |
| jamesbenson | @gmann, Yeah, these are version locks we have: https://gitlab.com/utsa-ics/osias/-/blob/master/osias_variables.py#L67 | 20:20 |
| jamesbenson | We need to specify both min and max version, correct? | 20:21 |
| gmann | jamesbenson: min version can be None as Nova has not bumped the min version and its 2.1 only. min_microversion as None in tempest conf means nova 2.1 | 20:22 |
| gmann | jamesbenson: so you are getting error with these cap version right? | 20:22 |
| jamesbenson | @gmann, currently we don't have these caps used in our testing and are getting the errors. | 20:23 |
| gmann | jamesbenson: so basically with your current configuration, tempest will request nova microversion 2.80 for victoria | 20:24 |
| jamesbenson | We did use them at some point, but I think we were getting additional errors, so we stopped. But I'm running tests now, so I should know in a couple of hours. | 20:24 |
| gmann | jamesbenson: make min_microversion as None and max_microversion as you have currently in https://gitlab.com/utsa-ics/osias/-/blob/master/osias_variables.py#L72 | 20:24 |
| jamesbenson | Yes, these are what would get put into the tempest.conf file | 20:24 |
| jamesbenson | okay, will do! | 20:25 |
| gmann | jamesbenson: because tempest might fail with min_microversion as 2.80 because few tests might need modification on asserts/schema | 20:25 |
| jamesbenson | gotcha, should I set them all to "None"? | 20:26 |
| gmann | yeah | 20:26 |
| jamesbenson | Xena I have a max of: 2.90 | 20:26 |
| gmann | yeah, that's correct https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-xena | 20:26 |
| jamesbenson | Does the tempest version or other things matter? | 20:27 |
| jamesbenson | (And any thoughts on the project, feel free to share, we love input) | 20:27 |
| gmann | jamesbenson: for victoria no. Tempest master works for ussuri - current openstack master | 20:27 |
| gmann | that is what we test at upstream also, tempest master to test ussuri, victoria, wallaby, xena and current yoga | 20:28 |
| jamesbenson | okay, cool. I'll update tempest u->x to 29.1.0 | 20:28 |
| gmann | +1, please let me know if tempest fail in victoria as min_microversion=None and max_microversion='2.87' | 20:29 |
| jamesbenson | sure will do | 20:30 |
| jamesbenson | one other thing fails too | 20:31 |
| jamesbenson | we use cirros as our test image, and gocubsgo works, but then some tests use random passwords, and those fail | 20:31 |
| jamesbenson | Any ideas on that? | 20:31 |
| jamesbenson | microversion issue too? | 20:31 |
| gmann | jamesbenson: cirros is one we use in upstream too and password is configurable in tempest config. But tempest is not ready to run for all combination of micrversion. | 20:33 |
| gmann | I will say try with min_microversion=None and max_microversion='2.87' and then we can see if they still fail then its tempest issue | 20:33 |
| jamesbenson | ok, sounds good. These are our setups for tempest.conf | 20:34 |
| jamesbenson | https://gitlab.com/utsa-ics/osias/-/blob/master/test_setup.sh | 20:34 |
| gmann | yeah, that seems correct image_ssh_password but there might be some test issue, if you face issue ping me on #openstack-qa or file bug in tempest i can fix that | 20:36 |
| jamesbenson | you are a life saver! Thank you! | 20:36 |
| gmann | np! | 20:37 |
| jamesbenson | I do have one re-occuring bug, it always happens in ussuri up in the tearDownClass (tempest.api.compute.servers.test_multiple_create.MultipleCreateTestJSON). Details: {'type': 'SubnetInUse', 'message': 'Unable to complete operation on subnet 5be11612-2a84-4723-b683-3cf06e3567d4: One or more ports have an IP allocation from this subnet.', 'detail': ''} | 20:39 |
| jamesbenson | It can't destroy the subnet it creates.... | 20:40 |
| gmann | jamesbenson: can you please log bug in tempest for that, I will check on Monday | 20:41 |
| gmann | here https://bugs.launchpad.net/tempest | 20:42 |
| jamesbenson | sure thing | 20:42 |
| gmann | thanks | 20:42 |
| jamesbenson | Any special logs you need or just the output from Refstack on that? | 20:44 |
| gmann | you can add tempest log also | 20:44 |
| jamesbenson | https://bugs.launchpad.net/tempest/+bug/1954535 | 20:50 |
| *** mdbooth3 is now known as mdbooth | 21:44 | |
| jamesbenson | @gmann, 'None' had issues so I changed it to 2.1 for the min. I'm rerunning it now, I'll report back on Monday. Thanks again and have a great weekend! | 23:05 |
| gmann | jamesbenson: sure, you too. | 23:07 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!