| *** XinxinShen has joined #openstack-masakari | 00:53 | |
| *** suzhengwei has joined #openstack-masakari | 05:48 | |
| *** suzhengwei has quit IRC | 05:51 | |
| *** suzhengwei has joined #openstack-masakari | 05:52 | |
| *** XinxinShen_ has joined #openstack-masakari | 05:53 | |
| *** XinxinShen has quit IRC | 05:53 | |
| *** XinxinShen_ has left #openstack-masakari | 05:55 | |
| *** XinxinShen has joined #openstack-masakari | 05:55 | |
| *** jopdoro has joined #openstack-masakari | 06:01 | |
| jopdoro | join #openstack-masakari | 06:01 |
|---|---|---|
| jopdoro | Hi | 06:02 |
| *** jegor has joined #openstack-masakari | 06:02 | |
| jegor | Hi | 06:02 |
| yoctozepto | meeting in a moment | 06:03 |
| yoctozepto | dealing with tech issues | 06:03 |
| jegor | Ok | 06:03 |
| *** jopdoro has quit IRC | 06:04 | |
| *** jegor is now known as jopdorp | 06:04 | |
| suzhengwei | hi | 06:05 |
| yoctozepto | ok, sorry | 06:06 |
| yoctozepto | #startmeeting masakari | 06:06 |
| opendevmeet | Meeting started Tue Jun 1 06:06:14 2021 UTC and is due to finish in 60 minutes. The chair is yoctozepto. Information about MeetBot at http://wiki.debian.org/MeetBot. | 06:06 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 06:06 |
| opendevmeet | The meeting name has been set to 'masakari' | 06:06 |
| yoctozepto | #topic Roll-call | 06:07 |
| yoctozepto | bot broke | 06:07 |
| yoctozepto | \o/ | 06:07 |
| jopdorp | \o/ \O/ | 06:07 |
| yoctozepto | hi jopdorp | 06:07 |
| yoctozepto | hi suzhengwei | 06:07 |
| suzhengwei | 0/ | 06:07 |
| yoctozepto | glad to see you in the new network | 06:08 |
| jopdorp | oftc! | 06:08 |
| yoctozepto | #topic Agenda | 06:08 |
| yoctozepto | * Roll-call | 06:08 |
| yoctozepto | * Agenda | 06:08 |
| yoctozepto | * Announcements | 06:08 |
| yoctozepto | * Review action items from the last meeting | 06:08 |
| yoctozepto | * CI status | 06:08 |
| yoctozepto | * Backports pending reviews | 06:08 |
| yoctozepto | * Xena planning -> https://etherpad.opendev.org/p/masakari-xena-ptg | 06:08 |
| yoctozepto | * Open discussion | 06:08 |
| yoctozepto | #topic Announcements | 06:09 |
| yoctozepto | it's kind of obvious but we moved IRC channels to the new network - OFTC - that we are currently on | 06:09 |
| yoctozepto | this information probably makes more sense in the saved logs read externally because you clearly know it being here ;-) | 06:10 |
| yoctozepto | #topic Review action items from the last meeting | 06:10 |
| yoctozepto | there were none | 06:10 |
| yoctozepto | #topic CI status | 06:10 |
| yoctozepto | I saw it green recently | 06:10 |
| yoctozepto | #topic Backports pending reviews | 06:11 |
| yoctozepto | none | 06:11 |
| yoctozepto | #topic Xena planning -> https://etherpad.opendev.org/p/masakari-xena-ptg | 06:11 |
| yoctozepto | ok, this is the interesting part | 06:11 |
| yoctozepto | have we got more progress? | 06:11 |
| jopdorp | not from my side | 06:13 |
| yoctozepto | ok | 06:13 |
| suzhengwei | update https://review.opendev.org/c/openstack/masakari/+/788382 | 06:15 |
| yoctozepto | I saw suzhengwei commented on my comment | 06:15 |
| yoctozepto | oh, right, that one | 06:16 |
| yoctozepto | I have not read it yet | 06:16 |
| suzhengwei | no hurry | 06:17 |
| yoctozepto | read it | 06:17 |
| yoctozepto | but I'm not sure if you agree with me or disagree | 06:18 |
| yoctozepto | was there any issue with the flow I presented in my comment? | 06:18 |
| suzhengwei | My opinion, no need wat for the result if it disable or force-down nova-compute service. | 06:19 |
| suzhengwei | There is no asynchronous call in nova inside. | 06:20 |
| yoctozepto | well, if we check if it's up and it is, then we should try checking for it going down; if nova still sees the host, then we could say either abort or force it down and continue (this could be up to the user's choice) | 06:21 |
| suzhengwei | The return of calls to nova-api can show whether the status/state has changed sucessefully. | 06:21 |
| suzhengwei | we check if it's up just because we are not sure it has been fenced. | 06:23 |
| suzhengwei | the nova-compute diable or not will not stop evacuation. but down or not matters. | 06:25 |
| yoctozepto | suzhengwei: yes, we don't know if it's down precisely ;-) | 06:26 |
| yoctozepto | also | 06:26 |
| yoctozepto | if nova thinks it's up and it's actually down, then it will fail disabling the service as well | 06:26 |
| yoctozepto | I think disabling the service is an extra such that the host does not come into play if it gets back on and needs operator's intervention | 06:27 |
| yoctozepto | the order of actions is unfortunate though, I pointed you at the relevant bug reports | 06:27 |
| suzhengwei | yep, we can disable the compute firstly, but I don't think need to wait. the enable_disable_service call returns show something. | 06:31 |
| suzhengwei | It the call failed, it will raise exception. | 06:31 |
| yoctozepto | suzhengwei: but it waits | 06:32 |
| yoctozepto | last time I checked, there was some serious delay | 06:32 |
| yoctozepto | because the disable wanted to speak via mq to the nova-compute service | 06:32 |
| suzhengwei | but it is synchronous calls in nova inside. think that if nova-compute already down, which one to disable or force-down the compute service. | 06:37 |
| suzhengwei | not nova-compute, all in the nova-api. | 06:38 |
| suzhengwei | if it dellay, the call would respone dellay too. | 06:38 |
| suzhengwei | delay | 06:39 |
| yoctozepto | suzhengwei: yeah, it's synchronously waiting for an answer on mq | 06:40 |
| yoctozepto | if nova-api thinks nova-compute is up | 06:40 |
| yoctozepto | but it's actually not | 06:40 |
| yoctozepto | then disabling it is going to timeout | 06:40 |
| yoctozepto | try it out locally | 06:41 |
| yoctozepto | just without masakari, ensure nova-compute has just been confirmed to be up, then firewall that host away and try disabling the service in nova | 06:41 |
| suzhengwei | I am very sure. | 06:41 |
| suzhengwei | Even the compute node down. I can still disable or force-down it freely. | 06:42 |
| yoctozepto | if nova knows it's down, then yes | 06:43 |
| yoctozepto | if nova thinks it's up but it's not, timeout | 06:43 |
| yoctozepto | unless they changed that in recent series | 06:43 |
| yoctozepto | because that's what it was in ussuri for sure | 06:43 |
| yoctozepto | I suggest you check and I will check as well | 06:46 |
| suzhengwei | the code in nova project. service_update_by_host_and_binary function in nova/compute/api.py | 06:49 |
| suzhengwei | It just changes db in the nova-api process. | 06:50 |
| yoctozepto | suzhengwei: but it finally calls https://github.com/openstack/nova/blob/5cf06bf33d8f187d444f812177946e134e4c9932/nova/compute/api.py#L5863 | 06:53 |
| yoctozepto | which has the unfortunate self.rpcapi.set_host_enabled | 06:53 |
| yoctozepto | and it usually just timeouts | 06:54 |
| suzhengwei | oh, i see. It need synchronous with pacement. But it save status first, then rpcapi.set_host_enabled. | 06:57 |
| suzhengwei | placement | 06:57 |
| yoctozepto | suzhengwei: yeah, I would say it's an edge situation on nova's side | 06:59 |
| yoctozepto | ok, the point is we are disabling the service as a service for the operator and not because we require it | 06:59 |
| yoctozepto | and we have to end the meeting | 06:59 |
| yoctozepto | thanks for the discussion, that was a fruitful meeting | 07:00 |
| yoctozepto | see you next time | 07:00 |
| yoctozepto | #endmeeting | 07:00 |
| opendevmeet | Meeting ended Tue Jun 1 07:00:10 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 07:00 |
| opendevmeet | Minutes: http://eavesdrop.openstack.org/meetings/masakari/2021/masakari.2021-06-01-06.06.html | 07:00 |
| opendevmeet | Minutes (text): http://eavesdrop.openstack.org/meetings/masakari/2021/masakari.2021-06-01-06.06.txt | 07:00 |
| opendevmeet | Log: http://eavesdrop.openstack.org/meetings/masakari/2021/masakari.2021-06-01-06.06.log.html | 07:00 |
| jopdorp | thanks! | 07:02 |
| jopdorp | see you next time | 07:02 |
| *** XinxinShen has quit IRC | 07:07 | |
| *** suzhengwei has quit IRC | 09:35 | |
| *** suzhengwei has joined #openstack-masakari | 09:57 | |
| *** suzhengwei has quit IRC | 10:58 | |
| *** suzhengwei has joined #openstack-masakari | 11:14 | |
| *** suzhengwei has quit IRC | 12:40 | |
| *** opendevreview has quit IRC | 14:44 | |
| *** suzhengwei has joined #openstack-masakari | 15:32 | |
| *** suzhengwei has quit IRC | 15:42 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!