| gmann | dansmith: this is enabling the nova rbac new defaults in devstack by default and nova change to add a new job to test old default please check https://review.opendev.org/c/openstack/devstack/+/883556 https://review.opendev.org/c/openstack/nova/+/883557 | 00:30 |
|---|---|---|
| dansmith | gmann: got it | 00:36 |
| gmann | thanks | 00:41 |
| *** dmellado90 is now known as dmellado9 | 05:09 | |
| *** EugenMayer45 is now known as EugenMayer4 | 06:17 | |
| opendevreview | Danylo Vodopianov proposed openstack/nova master: Packed virtqueue support was added. https://review.opendev.org/c/openstack/nova/+/876075 | 12:45 |
| opendevreview | Danylo Vodopianov proposed openstack/nova-specs master: Add support for Napatech LinkVirt SmartNICs https://review.opendev.org/c/openstack/nova-specs/+/859290 | 13:16 |
| opendevreview | Danylo Vodopianov proposed openstack/nova-specs master: Add support for Napatech LinkVirt SmartNICs https://review.opendev.org/c/openstack/nova-specs/+/859290 | 13:41 |
| opendevreview | Danylo Vodopianov proposed openstack/nova-specs master: Add support for Napatech LinkVirt SmartNICs https://review.opendev.org/c/openstack/nova-specs/+/859290 | 13:43 |
| opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015 | 13:44 |
| opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add healthcheck manager to manager base https://review.opendev.org/c/openstack/nova/+/827844 | 13:44 |
| opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add healthcheck tracker to nova context https://review.opendev.org/c/openstack/nova/+/829468 | 13:44 |
| opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add healthcheck utils and constants https://review.opendev.org/c/openstack/nova/+/829469 | 13:44 |
| opendevreview | Amit Uniyal proposed openstack/nova master: add healthcheck endpoint to proxy commands https://review.opendev.org/c/openstack/nova/+/830703 | 13:44 |
| dansmith | sean-k-mooney: this is the binding error I've been seeing quite a bit latey: https://a6fc37e91c861c55cf2e-59e8bddca242bc843b9f9be8c2ce73c4.ssl.cf2.rackcdn.com/879905/5/check/nova-live-migration/e3a0c46/testr_results.html | 15:01 |
| dansmith | it says to check neutron logs, so maybe it's just a neutron thing, but does that look familiar at all? | 15:01 |
| dansmith | actually I though it used to say something about os-vif in the message now that I think about it so maybe it's something different | 15:01 |
| opendevreview | yatin proposed openstack/nova master: Add config option to configure TB cache size https://review.opendev.org/c/openstack/nova/+/868419 | 15:09 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Re-order and parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | 15:10 |
| * artom waits for fireworks | 15:11 | |
| dansmith | artom: the GIL has nothing to do with your threading there | 15:17 |
| dansmith | what you mean, I think, is "as concurrently as the activities are green(ed)" | 15:18 |
| sean-k-mooney | dansmith: only because you linked it to me like two weeks ago | 15:20 |
| sean-k-mooney | ill tak a look at it again i didnt get around to it then | 15:20 |
| dansmith | sean-k-mooney: thanks | 15:20 |
| sean-k-mooney | Refusing to bind port e3308a61-39ff-4064-abb2-76de0d2139dc to dead agent: <neutron.plugins.ml2.drivers.ovn.agent.neutron_agent.ControllerAgent object at 0x7f6a7a6d2950> | 15:32 |
| dansmith | does that mean the agent on the compute died or something? | 15:32 |
| sean-k-mooney | i think that meanst the ovn metadtaa agent is dead since ovn its slef is agent less | 15:33 |
| sean-k-mooney | but yes neutorn think the agent is dead on the destination host | 15:34 |
| sean-k-mooney | it bound fine orgianly on the other host | 15:34 |
| artom | dansmith, right, I meant it in the sense of GIL only allowing one execution thread at a time, so we're counting to eventlet to do its yield thing | 15:34 |
| artom | I'll just remove it from the commit message :P | 15:35 |
| sean-k-mooney | ill see if i can figure out why | 15:35 |
| sean-k-mooney | dansmith: so it soudn like its hitting the code for https://github.com/openstack/neutron/commit/8a55f091925fd5e6742fb92783c524450843f5a0 | 15:48 |
| sean-k-mooney | hum so at the time of the port bidning | 15:50 |
| sean-k-mooney | there are no errro in the metadta aganet log but there are gaps for 3-6 seconds at a tiem and its interacting with both ovs and privsep | 15:51 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Re-order and parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | 16:12 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Call Neutron immediately upon _post_live_migration() start https://review.opendev.org/c/openstack/nova/+/883682 | 16:12 |
| sean-k-mooney | dansmith: so my best guess is its related to thsi change https://github.com/openstack/neutron/commit/628442aed7400251f12809a45605bd717f494c4e | 16:15 |
| sean-k-mooney | 7 mounts ago they started trying to spread the agent heatbeats | 16:16 |
| sean-k-mooney | im seeing logs to the effect fo delaying update to the cachs table for 10 seconds | 16:16 |
| sean-k-mooney | around when the agent prior to the agent being detected as dead | 16:17 |
| sean-k-mooney | my guess is if the agent is doign somthign like writing to the ovs db | 16:17 |
| sean-k-mooney | it can miss the heatbeat | 16:17 |
| sean-k-mooney | Delaying updating chassis table for 23 seconds {{(pid=38857) run /opt/stack/neutron/neutron/agent/ovn/metadata/agent.py:243}} | 16:18 |
| sean-k-mooney | im seeign quite a spread | 16:18 |
| dansmith | artom: ack I figured, probably better to make it accurate though yeah :) | 16:20 |
| dansmith | sean-k-mooney: ah, interesting | 16:20 |
| dansmith | sean-k-mooney: so like under heavy load they're missing some heartbeats maybe | 16:20 |
| sean-k-mooney | ya perhaps | 16:20 |
| sean-k-mooney | im goign to put up a tiny patch to change that form cfg.CONF.agent_down_time // 2 to cfg.CONF.agent_down_time // 3 | 16:21 |
| sean-k-mooney | that will make it heat beat a little more often | 16:21 |
| dansmith | ack cool | 16:22 |
| sean-k-mooney | that was recently done for rabbit 2 -> 3 for similar reasons | 16:22 |
| sean-k-mooney | oh its not merged yet https://review.opendev.org/c/openstack/oslo.messaging/+/875615 | 16:26 |
| sean-k-mooney | dansmith: i assume there isnt a bug currently | 16:32 |
| dansmith | sean-k-mooney: not that I've opened | 16:32 |
| sean-k-mooney | ok ill file one quickly with some of the errors i was seeing | 16:32 |
| sean-k-mooney | the logs are not super helpful | 16:33 |
| dansmith | sweet thanks | 16:40 |
| sean-k-mooney | https://bugs.launchpad.net/neutron/+bug/2020215 | 16:44 |
| sean-k-mooney | i will push a patch once i run the unit/functional tests and see what breaks | 16:45 |
| *** efried1 is now known as efried | 16:48 | |
| opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | 16:59 |
| *** melwitt_ is now known as melwitt | 17:11 | |
| sean-k-mooney | dansmith: i think https://review.opendev.org/c/openstack/neutron/+/883687 will help but its hard to tell if not then https://bugs.launchpad.net/neutron/+bug/2020215 might give the neutron folks another idea | 17:11 |
| dansmith | ack thanks for chasing that | 17:11 |
| sean-k-mooney | im going to finish there for today o/ | 17:11 |
| dansmith | thanks, enjoy the weekend | 17:11 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | 17:25 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | 20:06 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | 23:01 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!