Tuesday, 2020-09-15

*** ykado has joined #openstack-masakari03:52
*** vishalmanchanda has joined #openstack-masakari04:33
ykadoHi04:36
ykadoWill there be a weekly meeting?04:36
*** suzhengwei has joined #openstack-masakari05:55
yoctozeptoykado: yes, it's every two weeks though and it's today, will start in 40 minutes06:20
suzhengweiIs it the first meeting since meeting time changed?06:22
ykadoyoctozepto: thank you!06:26
jopdorpI think it's the second, but I missed last one06:33
yoctozeptosuzhengwei: second06:44
yoctozeptoykado: you are welcome06:44
yoctozeptosuzhengwei, jopdorp: past meeting: http://eavesdrop.openstack.org/meetings/masakari/2020/masakari.2020-09-01-07.01.html http://eavesdrop.openstack.org/meetings/masakari/2020/masakari.2020-09-01-07.01.log.html06:45
yoctozeptoplace for agenda: https://wiki.openstack.org/wiki/Meetings/Masakari06:55
yoctozeptoI also encourage you to add your nicks to the "10 minute warning" - it's not a guaranteed service (for that a usual calendar app is better) but can be helpful06:56
yoctozeptoa'ight, time to start the party07:00
yoctozepto#startmeeting masakari07:01
openstackMeeting started Tue Sep 15 07:01:00 2020 UTC and is due to finish in 60 minutes.  The chair is yoctozepto. Information about MeetBot at http://wiki.debian.org/MeetBot.07:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.07:01
*** openstack changes topic to " (Meeting topic: masakari)"07:01
openstackThe meeting name has been set to 'masakari'07:01
yoctozepto#topic Roll-call07:01
*** openstack changes topic to "Roll-call (Meeting topic: masakari)"07:01
yoctozeptoo/07:01
jopdorphere we go07:02
ykadoo/07:02
suzhengweihi07:02
noonedeadpunko/07:02
suzhengweio/07:02
jopdorpo/07:02
yoctozepto#topic Agenda07:03
*** openstack changes topic to "Agenda (Meeting topic: masakari)"07:03
yoctozepto* Roll-call07:03
yoctozepto* Agenda07:03
yoctozepto* Announcements07:03
yoctozepto  ** gates have been fixed and are running Focal now07:03
yoctozepto  ** python-masakariclient 6.1.1 released for Victoria (branched): https://releases.openstack.org/victoria/index.html#victoria-python-masakariclient07:03
yoctozepto  ** we are in Victoria feature freeze now / RC1 next week: https://releases.openstack.org/victoria/schedule.html07:03
yoctozepto* Review action items from the last meeting07:03
yoctozepto* CI status07:03
yoctozepto* Critical Bugs and Patches07:03
yoctozepto* Victoria release planning07:03
yoctozepto* Open discussion07:03
yoctozepto#topic Announcements07:03
*** openstack changes topic to "Announcements (Meeting topic: masakari)"07:03
yoctozepto#info gates have been fixed and are running Focal now07:04
yoctozeptoa part of the Victoria community goals was to migrate the testing to Focal which runs Py3807:04
yoctozeptoit has happened now07:04
yoctozeptoI had to self-merge a bunch of patches (due to deadlines) to make it happen but we are there07:05
jopdorpgood07:05
yoctozeptoall merged patches are visible in gerrit so you can post-review them still and raise any issues; there should really be none though due to the kind of those patches07:05
jopdorpI remember a patch that removed py37 tests07:06
yoctozeptomany thanks to gmann for helping with the migration and driving the goal openstack-wise07:06
yoctozeptojopdorp: yes07:06
yoctozeptodo note py37 is *not* a target platform for openstack now07:06
jopdorpcool07:06
yoctozeptohttps://governance.openstack.org/tc/reference/runtimes/victoria.html07:07
jopdorpthanks07:07
yoctozeptothe spoken-of migration has happened so py37 is no longer relevant07:07
yoctozeptoit's unlikely it breaks now that we test py36 and py38 but who knows :-)07:07
jopdorpagreed07:08
yoctozepto#info https://governance.openstack.org/tc/reference/runtimes/victoria.html07:08
yoctozepto(in case you are wondering why I'm doing the # stuff - it's to get these entries into the summary)07:08
jopdorpI was indeed wondering07:08
yoctozepto#info python-masakariclient 6.1.1 released for Victoria (branched): https://releases.openstack.org/victoria/index.html#victoria-python-masakariclient07:09
yoctozeptopython-masakariclient has a stable/victoria branch now07:09
yoctozeptothis week all client libraries got their releases07:09
yoctozeptoand there should be no feature changes whatsoever in them now that they are officially "stable"07:09
yoctozepto#info we are in Victoria feature freeze now / RC1 next week: https://releases.openstack.org/victoria/schedule.html07:10
noonedeadpunkthat makes me sad kind of07:10
yoctozeptonoonedeadpunk: me too, unfortunately07:10
suzhengweime, too07:10
jopdorpthis one didn't make it https://review.opendev.org/#/c/740777/07:11
yoctozeptodon't worry, we'll make wallaby the best masakari release ever :-)07:11
jopdorphaha07:11
jopdorpyes07:11
yoctozepto#info https://review.opendev.org/74077707:11
yoctozeptoas a matter of fact, due to low gate activity and low complexity of that patch, I suggest we put it in the exceptions and just merge it07:11
yoctozeptothen we have at least one feature in victoria07:11
yoctozeptosuzhengwei: what do you think?07:12
yoctozeptojopdorp has already +207:12
suzhengweiagree07:12
yoctozeptoI find it uncomfortable to +2 my own proposals07:12
yoctozeptounless they fix the gates07:12
yoctozeptosuzhengwei: thanks, please review and hopefully leave a +2 today/tomorrow :-)07:13
jopdorpnice07:13
yoctozepto#info Review action items from the last meeting07:13
yoctozeptoaaand there were none!07:13
jopdorplol07:13
yoctozeptooopsie07:13
yoctozepto#undo07:14
openstackRemoving item from minutes: #info Review action items from the last meeting07:14
yoctozepto#topic Review action items from the last meeting07:14
*** openstack changes topic to "Review action items from the last meeting (Meeting topic: masakari)"07:14
yoctozepto#info there were none07:14
yoctozepto#topic CI status07:14
*** openstack changes topic to "CI status (Meeting topic: masakari)"07:14
yoctozeptothe master CI is green, I did not check others07:14
yoctozeptomaster+Victoria considering the client has already branched07:15
yoctozeptowe could use some better organization in this regard, I'll try to spin up some CI glory page like we have for Kolla07:15
yoctozepto#action yoctozepto to bring some visibility to Masakari CI status07:15
jopdorpI don't really know where to look for the ci stuff07:16
jopdorponly the results in gerrit07:16
jopdorpbut I don't know where they get configured07:16
jopdorpor where the results for the main branches are visible07:16
yoctozeptono problem, I know most of you are new to this so that's why I just assigned this task to myself07:17
yoctozeptoin general one can look at https://zuul.opendev.org/t/openstack/builds07:17
yoctozeptowith filters obviously07:17
yoctozeptoor https://zuul.opendev.org/t/openstack/status07:17
yoctozeptofor currently running07:17
jopdorpis that also where they are configured?07:17
yoctozeptoI don't remember if masakari runs relevant periodics at the moment07:18
yoctozeptojopdorp: all CI stuff is configured in the repo nowadays07:18
yoctozeptothe CI/CD system that openstack uses is Zuul07:18
yoctozeptoin this setup we are considered users of zuul so these docs hold: https://zuul-ci.org/docs/zuul/reference/user.html07:18
yoctozeptoit's usually either zuul.yaml or zuul.d with some more yamls inside07:19
yoctozeptoit can be a hidden file/dir so .zuul.yaml .zuul.d respectively07:19
yoctozeptozuul is driven using yaml and ansible (which still uses yaml)07:19
yoctozeptofor all the other details please just review the files in repo and the docs :-)07:20
jopdorpthanks, I'll dive into that07:20
yoctozepto#topic Critical Bugs and Patches07:20
*** openstack changes topic to "Critical Bugs and Patches (Meeting topic: masakari)"07:20
yoctozepto#info none so far07:21
ykadohi, I wanted to raise about this review. https://review.opendev.org/#/c/720623/07:21
yoctozeptobut it could be that they have not been triaged07:21
ykadosorry, probably not the good timing yet?07:21
yoctozeptoykado: well, it's a fix to some bug but not necessarily critical I guess? let's postpone for the open discussion07:22
ykadook07:22
yoctozeptoif you know of breaking/fugly bugs then please report/triage them07:22
jopdorpwe encountered something that I'm not entirely sure is a masakari bug07:22
yoctozeptoplease speak up07:23
jopdorpbut we weren't able yet to get failovers of instances with LUKS encrypted volumes tow ork07:23
jopdorpthey get a keymanager error07:23
yoctozeptohmm, that does not sound like something masakari could go wrong about07:23
jopdorpbarbican right related07:23
jopdorprights07:23
yoctozeptomasakari essentially runs evacuations against instances07:23
yoctozeptotry plain evacuation and it might be failing07:24
jopdorpyeah07:24
yoctozeptoI *think* I saw someone reporting this issue against cinder+barbican07:24
jopdorpI think it's more configuration related07:24
yoctozeptocould07:24
yoctozeptowell then, let's not wander offtopic too much :-)07:24
jopdorpprobably the place would be @openstack-kolla07:24
jopdorp#07:24
yoctozeptojopdorp: yeah07:25
yoctozepto#topic Victoria release planning07:25
*** openstack changes topic to "Victoria release planning (Meeting topic: masakari)"07:25
yoctozeptowe already know it's frozen (freezing? :-) ) and we can only really squeeze that one patch of mine I mentioned07:25
yoctozepto(plus obviously any bug fixes)07:25
yoctozepto(noonedeadpunk triggered)07:26
yoctozeptonext week is RC107:26
yoctozeptoso all the other repos will branch stable/victoria as well07:26
yoctozeptoRC1 is R-307:26
yoctozeptoso then it's a matter of 3 weeks to polish eventual issues07:26
yoctozepto#topic Open discussion07:27
*** openstack changes topic to "Open discussion (Meeting topic: masakari)"07:27
yoctozeptoykado: now it's the time07:27
yoctozeptowhat about that commit07:27
ykadoyoctozepto: thanks07:27
yoctozeptohttps://review.opendev.org/72062307:28
ykadoI was wondering how this can progress.07:28
yoctozeptothe Radosław guy is me so my opinion on how that should progress is in that comment there07:28
yoctozeptothe "better design" part is surely about wallaby now07:29
yoctozeptobut the rest holds07:29
yoctozeptosuzhengwei: could you comment on that?07:30
yoctozeptothe part that needs dealing is "I guess we could still mix the two ideas and check on init while considering the timeout"07:30
suzhengweithe current design is the simplest07:31
yoctozeptoso it's actually a mix of your (suzhengwei) and tpatil's ideas07:31
suzhengweino, I didn't get tpatil's thought.07:32
ykadoI see. it is true, that there is no real reproducible way, if I understand it correctly.07:32
ykadoI only could reproduce this by forcefully powering off all the compute nodes or by stopping the masakari-engine services07:32
ykadohowever, without this patch there is no way to recover the compute-nodes that got resolved, unless you update the database manually07:32
suzhengweiyes, it is a big use problem for product.07:33
yoctozeptohmm, maybe what we need is an easier but manual way to achieve that07:33
yoctozeptoI'm worried tpatil is right that this could be too aggressive and result in more masakari surprises07:34
yoctozeptoi.e. simple but backstabbing :-)07:34
suzhengweiIt give a expired time for user to config. and that looks reasonable.07:35
ykadoI agree. sorry, I'm still new to Masakari. but what are the potential issue that relates with "running" statuses, if this get removed(although the default timeout value is quite long as suzhengwei mentioned) ?07:36
suzhengweiIf it can't recovery one failure host in a short time, the HA is useless.07:36
yoctozeptosuzhengwei: true that07:38
suzhengweiI think to turn it into failure is OK07:38
yoctozeptoone thing is some notifications are host-level so for a large host this could take a while07:38
*** priteau has joined #openstack-masakari07:40
suzhengweiso I leave the expired time configrable.07:40
yoctozeptoaye, it's set to 24 hours07:40
yoctozeptoby default07:41
yoctozeptook, there is one edge case that this deals with because of RUNNING and generated_time07:42
yoctozeptoit could be that the engine picks up a notification to run and self-sabotages itself07:42
yoctozeptoimagine a situation where the engine was down too long07:43
yoctozeptoor maybe not07:43
yoctozeptobecause rpc call will surely expire by this point07:43
suzhengweiI think it is an controller node issue.07:44
suzhengweicontroller HA issue.07:44
yoctozeptoyeah, masakari does not do a great job of self-HA07:45
suzhengweiother service also suffer07:45
yoctozeptoyeah, but it's no consolation considering masakari is THE HA project :D07:46
suzhengweicontroller HA and instance/compute HA is diffrent issue. They have their own solution.07:48
yoctozeptotrue that but still sad07:49
suzhengweiwe can't avoid all controller HA problem influence.07:49
yoctozeptowell, we could mitigate more though but it needs some redesign to happen07:50
yoctozeptoyour approach seems to be dealing with the reported issue07:51
suzhengweiDoing in a better way is better than doing nothing.07:51
suzhengweihttps://review.opendev.org/#/c/732477/07:53
suzhengweithis is a long term spec, it gives a solution-degrade retry.07:54
yoctozeptolooks promising07:56
yoctozeptook, I'll re-review suzhengwei's patch; I just need to delve into the masakari code more to be more confident about it07:57
yoctozeptoanyone else up to the review task?07:57
yoctozeptooh my, I have completely forgotten - the virtual PTG is coming - do we want a session for masakari? do you have any time preferences? please let me know via mail - I'll spin up a thread on openstack-discuss07:58
yoctozeptoI hope you are all subscribed07:58
yoctozepto#action yoctozepto to spin up a Masakari Wallaby vPTG thread on openstack-discuss mailing list07:59
yoctozeptoplease suzhengwei remember to review the only-feature-patch-that-we-can-get-it07:59
yoctozeptothank you07:59
yoctozeptoand thank you all for attending07:59
ykadothank you!07:59
yoctozepto#endmeeting08:00
*** openstack changes topic to "#openstack-masakari"08:00
openstackMeeting ended Tue Sep 15 08:00:03 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)08:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/masakari/2020/masakari.2020-09-15-07.01.html08:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/masakari/2020/masakari.2020-09-15-07.01.txt08:00
openstackLog:            http://eavesdrop.openstack.org/meetings/masakari/2020/masakari.2020-09-15-07.01.log.html08:00
* yoctozepto afk08:00
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors  https://review.opendev.org/72862909:40
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors  https://review.opendev.org/72862909:44
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors  https://review.opendev.org/72862909:44
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors  https://review.opendev.org/72862909:44
*** ykado has quit IRC10:10
*** suzhengwei has quit IRC11:17
*** priteau has quit IRC12:42
*** priteau has joined #openstack-masakari12:55
*** priteau has quit IRC13:15
*** uneek_ has joined #openstack-masakari13:57
*** gmann_ has joined #openstack-masakari13:57
*** priteau has joined #openstack-masakari13:57
*** uneek has quit IRC14:00
*** gmann has quit IRC14:00
*** coreycb has quit IRC14:00
*** uneek_ is now known as uneek14:00
*** coreycb_ has joined #openstack-masakari14:00
*** gmann_ is now known as gmann14:00
*** coreycb_ is now known as coreycb14:00
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors  https://review.opendev.org/72862914:22
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors  https://review.opendev.org/72862915:10
yoctozeptonoonedeadpunk: sorry for being unable to address all issues at once but this should be the last comment and we are merging 🙈15:21
noonedeadpunkyoctozepto: tbh I'm not so good in unit testing:( so not sure I got how to achive that. I thought that TestHypers has to be class object, and that was the only way that came to me how to do that15:41
yoctozeptonoonedeadpunk: aye, no problem - I can fix that for you - just tell me if you need that15:42
noonedeadpunkwould be great tbh:)15:42
yoctozeptonoonedeadpunk: ok15:42
noonedeadpunkor at least some reference to some sample :p15:42
yoctozeptohah, I don't know where would be the best one though15:43
noonedeadpunk(I guess fixing that might be faster for you)15:43
yoctozeptoexactly :D15:43
noonedeadpunkyeah, so go on:)15:43
yoctozeptothe CI is globally fried15:43
yoctozeptogood lord of testing15:43
noonedeadpunkyeah all because of oslo.service :(15:44
yoctozeptonoonedeadpunk: because of many oslos15:45
yoctozeptobecause of pypi failing mirror15:45
yoctozeptoso all oslo. things fail to download randomly15:46
openstackgerritRadosÅ‚aw Piliszek proposed openstack/masakari master: Search in nova services instead of hypervisors  https://review.opendev.org/72862916:01
yoctozeptonoonedeadpunk: ^16:01
noonedeadpunkhm. why I though it will fail....16:02
noonedeadpunkah, maybe it's going to pass now as we don't ask api attributes!16:03
noonedeadpunkas it was failing I guess on `compute.host` request for py3616:04
yoctozeptonoonedeadpunk: fwiw, there is no positive test that tests whether this ever succeeds16:05
noonedeadpunkI noticed that as well.16:05
noonedeadpunkbut thought that we do test it with some other test16:06
noonedeadpunkI mean in case we can't add host to db, smth should defenitely fail...16:07
yoctozeptonoonedeadpunk: yes, there is something weird about these tests still16:09
yoctozeptobecause we are essentially seeding the list so they actually should *not* raise16:09
-openstackstatus- NOTICE: Our PyPI caching proxies are serving stale package indexes for some packages. We think because PyPI's CDN is serving stale package indexes. We are sorting out how we can either fix or workaround that. In the meantime updating requirements is likely the wrong option.16:10
noonedeadpunkwhy they should not raise?16:19
noonedeadpunk` if len(computes) == 0: raise`16:19
noonedeadpunkhttps://review.opendev.org/#/c/728629/21/masakari/compute/nova.py16:20
noonedeadpunkso since list is empty we raise issue16:20
noonedeadpunkah, I think I got what you mean16:22
yoctozeptomock_novaclient.services.list.return_value = [self.FAKE_NOVA_SERVICE]16:22
yoctozeptoyeah16:22
noonedeadpunkthe return result of `mock_novaclient.services.list`16:22
noonedeadpunkhm, yeah, that's weird, but probably that's because it's filtered afterwards?16:23
noonedeadpunkyeah and I can make this test fail actually...16:28
noonedeadpunk*cant16:28
noonedeadpunkoh, btw, that is smth that we probably want here https://opendev.org/openstack/masakari/src/branch/master/masakari/tests/unit/compute/test_nova.py#L18416:31
noonedeadpunkyeah16:36
noonedeadpunklet me try to patch that16:36
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors  https://review.opendev.org/72862916:43
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors  https://review.opendev.org/72862916:44
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/masakari master: Search in nova services instead of hypervisors  https://review.opendev.org/72862916:45
noonedeadpunksorry for that...16:45
noonedeadpunkhad bad head locally16:46
noonedeadpunkyoctozepto: I think now should be ok16:47
noonedeadpunkbut it's not ;(16:50
yoctozeptonoonedeadpunk: we'll see to it tomorrow17:01
noonedeadpunkI mean it's not failed when list is not empty17:01
noonedeadpunkfor test_find_compute_service_non_existing_host_name17:02
*** vishalmanchanda has quit IRC18:13
*** priteau has quit IRC19:16

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!