*** ykado has joined #openstack-masakari | 03:52 | |
*** vishalmanchanda has joined #openstack-masakari | 04:33 | |
ykado | Hi | 04:36 |
---|---|---|
ykado | Will there be a weekly meeting? | 04:36 |
*** suzhengwei has joined #openstack-masakari | 05:55 | |
yoctozepto | ykado: yes, it's every two weeks though and it's today, will start in 40 minutes | 06:20 |
suzhengwei | Is it the first meeting since meeting time changed? | 06:22 |
ykado | yoctozepto: thank you! | 06:26 |
jopdorp | I think it's the second, but I missed last one | 06:33 |
yoctozepto | suzhengwei: second | 06:44 |
yoctozepto | ykado: you are welcome | 06:44 |
yoctozepto | suzhengwei, jopdorp: past meeting: http://eavesdrop.openstack.org/meetings/masakari/2020/masakari.2020-09-01-07.01.html http://eavesdrop.openstack.org/meetings/masakari/2020/masakari.2020-09-01-07.01.log.html | 06:45 |
yoctozepto | place for agenda: https://wiki.openstack.org/wiki/Meetings/Masakari | 06:55 |
yoctozepto | I also encourage you to add your nicks to the "10 minute warning" - it's not a guaranteed service (for that a usual calendar app is better) but can be helpful | 06:56 |
yoctozepto | a'ight, time to start the party | 07:00 |
yoctozepto | #startmeeting masakari | 07:01 |
openstack | Meeting started Tue Sep 15 07:01:00 2020 UTC and is due to finish in 60 minutes. The chair is yoctozepto. Information about MeetBot at http://wiki.debian.org/MeetBot. | 07:01 |
openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 07:01 |
*** openstack changes topic to " (Meeting topic: masakari)" | 07:01 | |
openstack | The meeting name has been set to 'masakari' | 07:01 |
yoctozepto | #topic Roll-call | 07:01 |
*** openstack changes topic to "Roll-call (Meeting topic: masakari)" | 07:01 | |
yoctozepto | o/ | 07:01 |
jopdorp | here we go | 07:02 |
ykado | o/ | 07:02 |
suzhengwei | hi | 07:02 |
noonedeadpunk | o/ | 07:02 |
suzhengwei | o/ | 07:02 |
jopdorp | o/ | 07:02 |
yoctozepto | #topic Agenda | 07:03 |
*** openstack changes topic to "Agenda (Meeting topic: masakari)" | 07:03 | |
yoctozepto | * Roll-call | 07:03 |
yoctozepto | * Agenda | 07:03 |
yoctozepto | * Announcements | 07:03 |
yoctozepto | ** gates have been fixed and are running Focal now | 07:03 |
yoctozepto | ** python-masakariclient 6.1.1 released for Victoria (branched): https://releases.openstack.org/victoria/index.html#victoria-python-masakariclient | 07:03 |
yoctozepto | ** we are in Victoria feature freeze now / RC1 next week: https://releases.openstack.org/victoria/schedule.html | 07:03 |
yoctozepto | * Review action items from the last meeting | 07:03 |
yoctozepto | * CI status | 07:03 |
yoctozepto | * Critical Bugs and Patches | 07:03 |
yoctozepto | * Victoria release planning | 07:03 |
yoctozepto | * Open discussion | 07:03 |
yoctozepto | #topic Announcements | 07:03 |
*** openstack changes topic to "Announcements (Meeting topic: masakari)" | 07:03 | |
yoctozepto | #info gates have been fixed and are running Focal now | 07:04 |
yoctozepto | a part of the Victoria community goals was to migrate the testing to Focal which runs Py38 | 07:04 |
yoctozepto | it has happened now | 07:04 |
yoctozepto | I had to self-merge a bunch of patches (due to deadlines) to make it happen but we are there | 07:05 |
jopdorp | good | 07:05 |
yoctozepto | all merged patches are visible in gerrit so you can post-review them still and raise any issues; there should really be none though due to the kind of those patches | 07:05 |
jopdorp | I remember a patch that removed py37 tests | 07:06 |
yoctozepto | many thanks to gmann for helping with the migration and driving the goal openstack-wise | 07:06 |
yoctozepto | jopdorp: yes | 07:06 |
yoctozepto | do note py37 is *not* a target platform for openstack now | 07:06 |
jopdorp | cool | 07:06 |
yoctozepto | https://governance.openstack.org/tc/reference/runtimes/victoria.html | 07:07 |
jopdorp | thanks | 07:07 |
yoctozepto | the spoken-of migration has happened so py37 is no longer relevant | 07:07 |
yoctozepto | it's unlikely it breaks now that we test py36 and py38 but who knows :-) | 07:07 |
jopdorp | agreed | 07:08 |
yoctozepto | #info https://governance.openstack.org/tc/reference/runtimes/victoria.html | 07:08 |
yoctozepto | (in case you are wondering why I'm doing the # stuff - it's to get these entries into the summary) | 07:08 |
jopdorp | I was indeed wondering | 07:08 |
yoctozepto | #info python-masakariclient 6.1.1 released for Victoria (branched): https://releases.openstack.org/victoria/index.html#victoria-python-masakariclient | 07:09 |
yoctozepto | python-masakariclient has a stable/victoria branch now | 07:09 |
yoctozepto | this week all client libraries got their releases | 07:09 |
yoctozepto | and there should be no feature changes whatsoever in them now that they are officially "stable" | 07:09 |
yoctozepto | #info we are in Victoria feature freeze now / RC1 next week: https://releases.openstack.org/victoria/schedule.html | 07:10 |
noonedeadpunk | that makes me sad kind of | 07:10 |
yoctozepto | noonedeadpunk: me too, unfortunately | 07:10 |
suzhengwei | me, too | 07:10 |
jopdorp | this one didn't make it https://review.opendev.org/#/c/740777/ | 07:11 |
yoctozepto | don't worry, we'll make wallaby the best masakari release ever :-) | 07:11 |
jopdorp | haha | 07:11 |
jopdorp | yes | 07:11 |
yoctozepto | #info https://review.opendev.org/740777 | 07:11 |
yoctozepto | as a matter of fact, due to low gate activity and low complexity of that patch, I suggest we put it in the exceptions and just merge it | 07:11 |
yoctozepto | then we have at least one feature in victoria | 07:11 |
yoctozepto | suzhengwei: what do you think? | 07:12 |
yoctozepto | jopdorp has already +2 | 07:12 |
suzhengwei | agree | 07:12 |
yoctozepto | I find it uncomfortable to +2 my own proposals | 07:12 |
yoctozepto | unless they fix the gates | 07:12 |
yoctozepto | suzhengwei: thanks, please review and hopefully leave a +2 today/tomorrow :-) | 07:13 |
jopdorp | nice | 07:13 |
yoctozepto | #info Review action items from the last meeting | 07:13 |
yoctozepto | aaand there were none! | 07:13 |
jopdorp | lol | 07:13 |
yoctozepto | oopsie | 07:13 |
yoctozepto | #undo | 07:14 |
openstack | Removing item from minutes: #info Review action items from the last meeting | 07:14 |
yoctozepto | #topic Review action items from the last meeting | 07:14 |
*** openstack changes topic to "Review action items from the last meeting (Meeting topic: masakari)" | 07:14 | |
yoctozepto | #info there were none | 07:14 |
yoctozepto | #topic CI status | 07:14 |
*** openstack changes topic to "CI status (Meeting topic: masakari)" | 07:14 | |
yoctozepto | the master CI is green, I did not check others | 07:14 |
yoctozepto | master+Victoria considering the client has already branched | 07:15 |
yoctozepto | we could use some better organization in this regard, I'll try to spin up some CI glory page like we have for Kolla | 07:15 |
yoctozepto | #action yoctozepto to bring some visibility to Masakari CI status | 07:15 |
jopdorp | I don't really know where to look for the ci stuff | 07:16 |
jopdorp | only the results in gerrit | 07:16 |
jopdorp | but I don't know where they get configured | 07:16 |
jopdorp | or where the results for the main branches are visible | 07:16 |
yoctozepto | no problem, I know most of you are new to this so that's why I just assigned this task to myself | 07:17 |
yoctozepto | in general one can look at https://zuul.opendev.org/t/openstack/builds | 07:17 |
yoctozepto | with filters obviously | 07:17 |
yoctozepto | or https://zuul.opendev.org/t/openstack/status | 07:17 |
yoctozepto | for currently running | 07:17 |
jopdorp | is that also where they are configured? | 07:17 |
yoctozepto | I don't remember if masakari runs relevant periodics at the moment | 07:18 |
yoctozepto | jopdorp: all CI stuff is configured in the repo nowadays | 07:18 |
yoctozepto | the CI/CD system that openstack uses is Zuul | 07:18 |
yoctozepto | in this setup we are considered users of zuul so these docs hold: https://zuul-ci.org/docs/zuul/reference/user.html | 07:18 |
yoctozepto | it's usually either zuul.yaml or zuul.d with some more yamls inside | 07:19 |
yoctozepto | it can be a hidden file/dir so .zuul.yaml .zuul.d respectively | 07:19 |
yoctozepto | zuul is driven using yaml and ansible (which still uses yaml) | 07:19 |
yoctozepto | for all the other details please just review the files in repo and the docs :-) | 07:20 |
jopdorp | thanks, I'll dive into that | 07:20 |
yoctozepto | #topic Critical Bugs and Patches | 07:20 |
*** openstack changes topic to "Critical Bugs and Patches (Meeting topic: masakari)" | 07:20 | |
yoctozepto | #info none so far | 07:21 |
ykado | hi, I wanted to raise about this review. https://review.opendev.org/#/c/720623/ | 07:21 |
yoctozepto | but it could be that they have not been triaged | 07:21 |
ykado | sorry, probably not the good timing yet? | 07:21 |
yoctozepto | ykado: well, it's a fix to some bug but not necessarily critical I guess? let's postpone for the open discussion | 07:22 |
ykado | ok | 07:22 |
yoctozepto | if you know of breaking/fugly bugs then please report/triage them | 07:22 |
jopdorp | we encountered something that I'm not entirely sure is a masakari bug | 07:22 |
yoctozepto | please speak up | 07:23 |
jopdorp | but we weren't able yet to get failovers of instances with LUKS encrypted volumes tow ork | 07:23 |
jopdorp | they get a keymanager error | 07:23 |
yoctozepto | hmm, that does not sound like something masakari could go wrong about | 07:23 |
jopdorp | barbican right related | 07:23 |
jopdorp | rights | 07:23 |
yoctozepto | masakari essentially runs evacuations against instances | 07:23 |
yoctozepto | try plain evacuation and it might be failing | 07:24 |
jopdorp | yeah | 07:24 |
yoctozepto | I *think* I saw someone reporting this issue against cinder+barbican | 07:24 |
jopdorp | I think it's more configuration related | 07:24 |
yoctozepto | could | 07:24 |
yoctozepto | well then, let's not wander offtopic too much :-) | 07:24 |
jopdorp | probably the place would be @openstack-kolla | 07:24 |
jopdorp | # | 07:24 |
yoctozepto | jopdorp: yeah | 07:25 |
yoctozepto | #topic Victoria release planning | 07:25 |
*** openstack changes topic to "Victoria release planning (Meeting topic: masakari)" | 07:25 | |
yoctozepto | we already know it's frozen (freezing? :-) ) and we can only really squeeze that one patch of mine I mentioned | 07:25 |
yoctozepto | (plus obviously any bug fixes) | 07:25 |
yoctozepto | (noonedeadpunk triggered) | 07:26 |
yoctozepto | next week is RC1 | 07:26 |
yoctozepto | so all the other repos will branch stable/victoria as well | 07:26 |
yoctozepto | RC1 is R-3 | 07:26 |
yoctozepto | so then it's a matter of 3 weeks to polish eventual issues | 07:26 |
yoctozepto | #topic Open discussion | 07:27 |
*** openstack changes topic to "Open discussion (Meeting topic: masakari)" | 07:27 | |
yoctozepto | ykado: now it's the time | 07:27 |
yoctozepto | what about that commit | 07:27 |
ykado | yoctozepto: thanks | 07:27 |
yoctozepto | https://review.opendev.org/720623 | 07:28 |
ykado | I was wondering how this can progress. | 07:28 |
yoctozepto | the Radosław guy is me so my opinion on how that should progress is in that comment there | 07:28 |
yoctozepto | the "better design" part is surely about wallaby now | 07:29 |
yoctozepto | but the rest holds | 07:29 |
yoctozepto | suzhengwei: could you comment on that? | 07:30 |
yoctozepto | the part that needs dealing is "I guess we could still mix the two ideas and check on init while considering the timeout" | 07:30 |
suzhengwei | the current design is the simplest | 07:31 |
yoctozepto | so it's actually a mix of your (suzhengwei) and tpatil's ideas | 07:31 |
suzhengwei | no, I didn't get tpatil's thought. | 07:32 |
ykado | I see. it is true, that there is no real reproducible way, if I understand it correctly. | 07:32 |
ykado | I only could reproduce this by forcefully powering off all the compute nodes or by stopping the masakari-engine services | 07:32 |
ykado | however, without this patch there is no way to recover the compute-nodes that got resolved, unless you update the database manually | 07:32 |
suzhengwei | yes, it is a big use problem for product. | 07:33 |
yoctozepto | hmm, maybe what we need is an easier but manual way to achieve that | 07:33 |
yoctozepto | I'm worried tpatil is right that this could be too aggressive and result in more masakari surprises | 07:34 |
yoctozepto | i.e. simple but backstabbing :-) | 07:34 |
suzhengwei | It give a expired time for user to config. and that looks reasonable. | 07:35 |
ykado | I agree. sorry, I'm still new to Masakari. but what are the potential issue that relates with "running" statuses, if this get removed(although the default timeout value is quite long as suzhengwei mentioned) ? | 07:36 |
suzhengwei | If it can't recovery one failure host in a short time, the HA is useless. | 07:36 |
yoctozepto | suzhengwei: true that | 07:38 |
suzhengwei | I think to turn it into failure is OK | 07:38 |
yoctozepto | one thing is some notifications are host-level so for a large host this could take a while | 07:38 |
*** priteau has joined #openstack-masakari | 07:40 | |
suzhengwei | so I leave the expired time configrable. | 07:40 |
yoctozepto | aye, it's set to 24 hours | 07:40 |
yoctozepto | by default | 07:41 |
yoctozepto | ok, there is one edge case that this deals with because of RUNNING and generated_time | 07:42 |
yoctozepto | it could be that the engine picks up a notification to run and self-sabotages itself | 07:42 |
yoctozepto | imagine a situation where the engine was down too long | 07:43 |
yoctozepto | or maybe not | 07:43 |
yoctozepto | because rpc call will surely expire by this point | 07:43 |
suzhengwei | I think it is an controller node issue. | 07:44 |
suzhengwei | controller HA issue. | 07:44 |
yoctozepto | yeah, masakari does not do a great job of self-HA | 07:45 |
suzhengwei | other service also suffer | 07:45 |
yoctozepto | yeah, but it's no consolation considering masakari is THE HA project :D | 07:46 |
suzhengwei | controller HA and instance/compute HA is diffrent issue. They have their own solution. | 07:48 |
yoctozepto | true that but still sad | 07:49 |
suzhengwei | we can't avoid all controller HA problem influence. | 07:49 |
yoctozepto | well, we could mitigate more though but it needs some redesign to happen | 07:50 |
yoctozepto | your approach seems to be dealing with the reported issue | 07:51 |
suzhengwei | Doing in a better way is better than doing nothing. | 07:51 |
suzhengwei | https://review.opendev.org/#/c/732477/ | 07:53 |
suzhengwei | this is a long term spec, it gives a solution-degrade retry. | 07:54 |
yoctozepto | looks promising | 07:56 |
yoctozepto | ok, I'll re-review suzhengwei's patch; I just need to delve into the masakari code more to be more confident about it | 07:57 |
yoctozepto | anyone else up to the review task? | 07:57 |
yoctozepto | oh my, I have completely forgotten - the virtual PTG is coming - do we want a session for masakari? do you have any time preferences? please let me know via mail - I'll spin up a thread on openstack-discuss | 07:58 |
yoctozepto | I hope you are all subscribed | 07:58 |
yoctozepto | #action yoctozepto to spin up a Masakari Wallaby vPTG thread on openstack-discuss mailing list | 07:59 |
yoctozepto | please suzhengwei remember to review the only-feature-patch-that-we-can-get-it | 07:59 |
yoctozepto | thank you | 07:59 |
yoctozepto | and thank you all for attending | 07:59 |
ykado | thank you! | 07:59 |
yoctozepto | #endmeeting | 08:00 |
*** openstack changes topic to "#openstack-masakari" | 08:00 | |
openstack | Meeting ended Tue Sep 15 08:00:03 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 08:00 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/masakari/2020/masakari.2020-09-15-07.01.html | 08:00 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/masakari/2020/masakari.2020-09-15-07.01.txt | 08:00 |
openstack | Log: http://eavesdrop.openstack.org/meetings/masakari/2020/masakari.2020-09-15-07.01.log.html | 08:00 |
* yoctozepto afk | 08:00 | |
openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors https://review.opendev.org/728629 | 09:40 |
openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors https://review.opendev.org/728629 | 09:44 |
openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors https://review.opendev.org/728629 | 09:44 |
openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors https://review.opendev.org/728629 | 09:44 |
*** ykado has quit IRC | 10:10 | |
*** suzhengwei has quit IRC | 11:17 | |
*** priteau has quit IRC | 12:42 | |
*** priteau has joined #openstack-masakari | 12:55 | |
*** priteau has quit IRC | 13:15 | |
*** uneek_ has joined #openstack-masakari | 13:57 | |
*** gmann_ has joined #openstack-masakari | 13:57 | |
*** priteau has joined #openstack-masakari | 13:57 | |
*** uneek has quit IRC | 14:00 | |
*** gmann has quit IRC | 14:00 | |
*** coreycb has quit IRC | 14:00 | |
*** uneek_ is now known as uneek | 14:00 | |
*** coreycb_ has joined #openstack-masakari | 14:00 | |
*** gmann_ is now known as gmann | 14:00 | |
*** coreycb_ is now known as coreycb | 14:00 | |
openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors https://review.opendev.org/728629 | 14:22 |
openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors https://review.opendev.org/728629 | 15:10 |
yoctozepto | noonedeadpunk: sorry for being unable to address all issues at once but this should be the last comment and we are merging 🙈 | 15:21 |
noonedeadpunk | yoctozepto: tbh I'm not so good in unit testing:( so not sure I got how to achive that. I thought that TestHypers has to be class object, and that was the only way that came to me how to do that | 15:41 |
yoctozepto | noonedeadpunk: aye, no problem - I can fix that for you - just tell me if you need that | 15:42 |
noonedeadpunk | would be great tbh:) | 15:42 |
yoctozepto | noonedeadpunk: ok | 15:42 |
noonedeadpunk | or at least some reference to some sample :p | 15:42 |
yoctozepto | hah, I don't know where would be the best one though | 15:43 |
noonedeadpunk | (I guess fixing that might be faster for you) | 15:43 |
yoctozepto | exactly :D | 15:43 |
noonedeadpunk | yeah, so go on:) | 15:43 |
yoctozepto | the CI is globally fried | 15:43 |
yoctozepto | good lord of testing | 15:43 |
noonedeadpunk | yeah all because of oslo.service :( | 15:44 |
yoctozepto | noonedeadpunk: because of many oslos | 15:45 |
yoctozepto | because of pypi failing mirror | 15:45 |
yoctozepto | so all oslo. things fail to download randomly | 15:46 |
openstackgerrit | Radosław Piliszek proposed openstack/masakari master: Search in nova services instead of hypervisors https://review.opendev.org/728629 | 16:01 |
yoctozepto | noonedeadpunk: ^ | 16:01 |
noonedeadpunk | hm. why I though it will fail.... | 16:02 |
noonedeadpunk | ah, maybe it's going to pass now as we don't ask api attributes! | 16:03 |
noonedeadpunk | as it was failing I guess on `compute.host` request for py36 | 16:04 |
yoctozepto | noonedeadpunk: fwiw, there is no positive test that tests whether this ever succeeds | 16:05 |
noonedeadpunk | I noticed that as well. | 16:05 |
noonedeadpunk | but thought that we do test it with some other test | 16:06 |
noonedeadpunk | I mean in case we can't add host to db, smth should defenitely fail... | 16:07 |
yoctozepto | noonedeadpunk: yes, there is something weird about these tests still | 16:09 |
yoctozepto | because we are essentially seeding the list so they actually should *not* raise | 16:09 |
-openstackstatus- NOTICE: Our PyPI caching proxies are serving stale package indexes for some packages. We think because PyPI's CDN is serving stale package indexes. We are sorting out how we can either fix or workaround that. In the meantime updating requirements is likely the wrong option. | 16:10 | |
noonedeadpunk | why they should not raise? | 16:19 |
noonedeadpunk | ` if len(computes) == 0: raise` | 16:19 |
noonedeadpunk | https://review.opendev.org/#/c/728629/21/masakari/compute/nova.py | 16:20 |
noonedeadpunk | so since list is empty we raise issue | 16:20 |
noonedeadpunk | ah, I think I got what you mean | 16:22 |
yoctozepto | mock_novaclient.services.list.return_value = [self.FAKE_NOVA_SERVICE] | 16:22 |
yoctozepto | yeah | 16:22 |
noonedeadpunk | the return result of `mock_novaclient.services.list` | 16:22 |
noonedeadpunk | hm, yeah, that's weird, but probably that's because it's filtered afterwards? | 16:23 |
noonedeadpunk | yeah and I can make this test fail actually... | 16:28 |
noonedeadpunk | *cant | 16:28 |
noonedeadpunk | oh, btw, that is smth that we probably want here https://opendev.org/openstack/masakari/src/branch/master/masakari/tests/unit/compute/test_nova.py#L184 | 16:31 |
noonedeadpunk | yeah | 16:36 |
noonedeadpunk | let me try to patch that | 16:36 |
openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors https://review.opendev.org/728629 | 16:43 |
openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors https://review.opendev.org/728629 | 16:44 |
openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors https://review.opendev.org/728629 | 16:45 |
noonedeadpunk | sorry for that... | 16:45 |
noonedeadpunk | had bad head locally | 16:46 |
noonedeadpunk | yoctozepto: I think now should be ok | 16:47 |
noonedeadpunk | but it's not ;( | 16:50 |
yoctozepto | noonedeadpunk: we'll see to it tomorrow | 17:01 |
noonedeadpunk | I mean it's not failed when list is not empty | 17:01 |
noonedeadpunk | for test_find_compute_service_non_existing_host_name | 17:02 |
*** vishalmanchanda has quit IRC | 18:13 | |
*** priteau has quit IRC | 19:16 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!