| opendevreview | David proposed openstack/watcher master: Disable real metrics on devstack injected data jobs https://review.opendev.org/c/openstack/watcher/+/955281 | 11:36 |
|---|---|---|
| opendevreview | Merged openstack/watcher-specs master: Add status_message field to the Audits https://review.opendev.org/c/openstack/watcher-specs/+/954718 | 11:54 |
| chandankumar | who all are here | 12:03 |
| rlandy | o/ | 12:03 |
| chandankumar | time for irc meeting | 12:03 |
| chandankumar | #startmeeting | 12:03 |
| opendevmeet | chandankumar: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee' | 12:03 |
| dviroel | o/ | 12:04 |
| chandankumar | #startmeeting watcher | 12:04 |
| opendevmeet | Meeting started Thu Jul 31 12:04:30 2025 UTC and is due to finish in 60 minutes. The chair is chandankumar. Information about MeetBot at http://wiki.debian.org/MeetBot. | 12:04 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 12:04 |
| opendevmeet | The meeting name has been set to 'watcher' | 12:04 |
| dviroel | the name is required, only the name is enough | 12:04 |
| dviroel | :) | 12:04 |
| chandankumar | courtesy ping: sean-k-mooney chandankumar morenod rlandy | 12:04 |
| sean-k-mooney | o/ | 12:04 |
| rlandy | I'm here :) | 12:04 |
| chandankumar | o/ | 12:05 |
| morenod | o/ | 12:05 |
| chandankumar | let's start with today's meeting agenda | 12:05 |
| chandankumar | #link https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L21 (Meeting agenda) | 12:05 |
| chandankumar | feel free to add your own topics to the agenda | 12:05 |
| chandankumar | Starting with the first one | 12:05 |
| chandankumar | #topic Eventlet Removal | 12:06 |
| dviroel | o/ | 12:06 |
| dviroel | as usual, the etherpad link | 12:06 |
| dviroel | #link https://etherpad.opendev.org/p/watcher-eventlet-removal (watcher evenlet removal etherpad) | 12:06 |
| chandankumar | #link https://etherpad.opendev.org/p/watcher-eventlet-removal (watcher evenlet removal etherpad) | 12:06 |
| dviroel | :) | 12:06 |
| dviroel | some minor changes this week | 12:06 |
| dviroel | i removed the depends-on changes from the main dec-engine patch | 12:07 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/952257 (Extend decision engine to support threading mode) | 12:07 |
| dviroel | the devstack one merged, the other one was the tempest-plugin change, which is not required to merge the main one | 12:07 |
| dviroel | but there is another DNM change just to test the new continous audit test: | 12:08 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/956199 | 12:08 |
| opendevreview | David proposed openstack/watcher master: Disable real metrics on devstack injected data jobs https://review.opendev.org/c/openstack/watcher/+/955281 | 12:09 |
| dviroel | not that we discussed about replacing te continuous audit test wit a unit or functional test | 12:09 |
| sean-k-mooney | yep devstack change merged yesterday so that unblocks that patch | 12:09 |
| sean-k-mooney | we can have both | 12:10 |
| dviroel | it turns that I couldn't find a way yet of mocking everything needed to simulate the bahavior found with continuous audit thread | 12:10 |
| sean-k-mooney | ack | 12:10 |
| dviroel | I updated instead the tempest-plugin change | 12:10 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/954264 | 12:10 |
| dviroel | to use only one audit as Alfredo suggested | 12:11 |
| dviroel | and turns that I hit another bug | 12:11 |
| dviroel | one from zone_migration that I filed in the past | 12:11 |
| * dviroel find the link | 12:12 | |
| dviroel | #link https://bugs.launchpad.net/watcher/+bug/2098984 | 12:12 |
| dviroel | so i started to hit this issue with continuous audit, with a 10s interval | 12:12 |
| dviroel | CI also hit that issue | 12:12 |
| sean-k-mooney | that the isse with not sharing the same model? | 12:13 |
| dviroel | not, another one | 12:13 |
| sean-k-mooney | oh ok | 12:13 |
| dviroel | zone_migration gets instances/volumes from nova/cinder but while they aren't yet in the model | 12:13 |
| dviroel | it raises an exception, since it is not properly handled | 12:13 |
| sean-k-mooney | oh didnt we fix that before | 12:13 |
| sean-k-mooney | for other stragies | 12:13 |
| sean-k-mooney | we added a polling loop or somethign liek that to make sure the model was synced | 12:14 |
| dviroel | this is specific for zone_migration implementation, not all strategies use clients to get info about instances/volumes | 12:14 |
| dviroel | the proposed fix: | 12:14 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/956198/1/watcher/decision_engine/strategy/strategies/zone_migration.py | 12:14 |
| sean-k-mooney | oh i see | 12:15 |
| sean-k-mooney | your fixing this from the watcher size not the test side | 12:15 |
| dviroel | another patch to add a unit test for this scenario: | 12:15 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/956197 | 12:15 |
| sean-k-mooney | ya this feels like it a real watcher bug | 12:15 |
| sean-k-mooney | hum | 12:16 |
| sean-k-mooney | so you are plannign to fix this by filtering to only the ones in the model | 12:16 |
| dviroel | sean-k-mooney: it was doing this already | 12:16 |
| dviroel | but not treating the exception | 12:16 |
| sean-k-mooney | ah your right | 12:16 |
| sean-k-mooney | so the ohter way to adress this is to updte teh model with the missing isntance | 12:17 |
| sean-k-mooney | i guess we can consider that as a latter enhancment and fix the expction handelign first | 12:17 |
| sean-k-mooney | ok i think just handelign the excption is more backportable anyway | 12:18 |
| dviroel | right, we can further discuss that, even if strategies should be getting info directly from the services.. | 12:18 |
| dviroel | but yes, we should backport this one | 12:19 |
| dviroel | in the etherpad there is a link to the error in CI, if someone wants to take a look | 12:19 |
| dviroel | alright, this bug is not eventlet related | 12:20 |
| dviroel | but one change take to another | 12:20 |
| dviroel | and I ended fixing this bug | 12:20 |
| dviroel | interesting that the continous audit test was useful for cathing it | 12:20 |
| sean-k-mooney | ya so we wont backport any of the eventlet change bu tthis is a ligitmate bug in its own right | 12:20 |
| sean-k-mooney | and we likely shoudl backprot that | 12:21 |
| dviroel | +1 | 12:21 |
| sean-k-mooney | so thatnk for filing a seperate tracker and spliting it out | 12:21 |
| chandankumar | #link https://bugs.launchpad.net/watcher/+bug/2098984 | 12:21 |
| dviroel | sure np | 12:21 |
| chandankumar | and fix https://review.opendev.org/c/openstack/watcher/+/956198 | 12:21 |
| sean-k-mooney | chandankumar: yep dviroel linked thosse above | 12:22 |
| chandankumar | yup | 12:22 |
| dviroel | alright, if nobody has any questions, that's cover my eventlet part | 12:22 |
| sean-k-mooney | one | 12:23 |
| chandankumar | thank you dviroel for sharing the update :-) | 12:23 |
| sean-k-mooney | but slightly unerelated | 12:23 |
| sean-k-mooney | the content provider job failed to build https://softwarefactory-project.io/zuul/t/rdoproject.org/build/6a8fe1f8aa174887803d784ec9cebdc4 | 12:23 |
| chandankumar | sean-k-mooney: the fix merged, few hours back | 12:23 |
| dviroel | yeah, it is failing in lot of jobs, but I still didn't start the investigation | 12:23 |
| sean-k-mooney | have we seen that on other patches ro do folks knwo why | 12:23 |
| dviroel | chandankumar: oh, good to know, i was about to ask you | 12:24 |
| sean-k-mooney | oh cool | 12:24 |
| sean-k-mooney | all good then | 12:24 |
| chandankumar | thanks sean-k-mooney for bringing that one | 12:24 |
| dviroel | i will recheck the patches afterwards then | 12:24 |
| sean-k-mooney | """ The task includes an option with an undefined variable. The error was: {{ ansible_user }}: 'ansible_user' is undefined. 'ansible_user' is undefined. {{ ansible_user }}: 'ansible_user' is undefined. 'ansible_user' is undefined""" | 12:24 |
| sean-k-mooney | i think perhaps ansible_user was missing :) | 12:25 |
| sean-k-mooney | ansibel can be a bit verbose ocationally | 12:25 |
| chandankumar | https://github.com/openstack-k8s-operators/ci-framework/commit/225d9d2f4b38a8d8e7e56bd431bb056462aab8c6 | 12:25 |
| dviroel | yeah right, it was podman role | 12:25 |
| rlandy | showed up late yesterday | 12:26 |
| rlandy | chandankumar, fixed it today | 12:26 |
| dviroel | chandankumar++ | 12:26 |
| chandankumar | Since no further question, moving now to next topic | 12:26 |
| dviroel | chandankumar: we can move to next topic | 12:26 |
| chandankumar | #topic Croniter swap with appscheduler | 12:26 |
| chandankumar | I was working on above topic and we had a long discussion for the same here https://review.opendev.org/c/openstack/watcher/+/955459/5#message-191158289ed45d4824525724dc38d247c0e8d4bc | 12:27 |
| chandankumar | #link https://review.opendev.org/c/openstack/watcher/+/955459/5#message-191158289ed45d4824525724dc38d247c0e8d4bc | 12:27 |
| chandankumar | I tried to summarize notes here https://etherpad.opendev.org/p/watcher-croniter-swap, But I will drop here also | 12:27 |
| chandankumar | The review discussed about migrating from croniter to appscheduler crontigger library. | 12:28 |
| chandankumar | Croniter supports 7 field format (with years and seconds as optional field) while appscheduler supports 5 field format. | 12:28 |
| chandankumar | The watcher continous audit specs does not provide any info about supporting 5 or 7 field format. | 12:28 |
| chandankumar | Since we are going to swap croniter usage with appscheduler. We saw few issues/concerns. | 12:28 |
| chandankumar | Upgrade Impact: Existing scheduled jobs (continuous audits) using croniter-specific syntax(which becomes invalid format) will fail after the migration. | 12:28 |
| chandankumar | Critical Failure: ongoing continuous audit created after the "bad-formatted" one, will also fail to schedule next runs as the worker responsible of scheduling fails with uncatched exception. | 12:28 |
| chandankumar | Thank you sean-k-mooney and Alfredo for actively reviewing and providing feedback on this | 12:29 |
| chandankumar | In order to mitigate these whole issues, the following plan is suggested: | 12:29 |
| chandankumar | 1. We need add watcher status check to detect if any audits are using an incomparable interval format. | 12:29 |
| chandankumar | 2. we need to deprecate the use of 6/7 column format and log a warning when its used. we can do that by trying to use aspschduler then fallback to using cronitoer if apscheduler cannot parse it. | 12:29 |
| chandankumar | 3. do the migration automatically on load from the db. | 12:30 |
| chandankumar | 4. provide a CLI tool to do an online migration of the data via watcher-manage to convert from 6/7 format to 5 format | 12:30 |
| chandankumar | 5. document a manual procedure to do the conversation via the api | 12:30 |
| chandankumar | 6. Finally by 2026.2 we will drop the fallback and only use apscheduler. | 12:30 |
| chandankumar | we also need to add proper exception handling and api validaitons for these formats. | 12:31 |
| dviroel | so we will call the 6/7 format as invalid already? we will just accept its input and do the conversion | 12:31 |
| chandankumar | The main thing we wanted to discuss about support 5 field or 7 field format | 12:31 |
| sean-k-mooney | the api validation can basiclly just be "parse it with aspchdluer or cronitor" | 12:31 |
| sean-k-mooney | dviroel: so i coudl not find anything to say it was ever offically supproted | 12:31 |
| dviroel | ack, we can justify that was never supported | 12:32 |
| sean-k-mooney | the plan above is the most conservitive option | 12:32 |
| dviroel | and will be an invalid input the future releases | 12:32 |
| dviroel | yeah | 12:32 |
| chandankumar | we went over code and specs, there is no mention of formats | 12:32 |
| chandankumar | the test uses 5 field format | 12:32 |
| dviroel | yeah, I saw your comments about specs/releasenotes | 12:32 |
| sean-k-mooney | the agressive option is say no it was never supprote we only supprot 5 colume format. but even if we did that i think the watcher-status command and posibly a helper command to do the converton woudl be good to have | 12:33 |
| chandankumar | yup | 12:33 |
| sean-k-mooney | given someone has taken over maintance of it again | 12:33 |
| dviroel | yes, since there wasn't anything blocking it before | 12:33 |
| sean-k-mooney | i think we are ok to take the concerviitve one | 12:33 |
| chandankumar | ok | 12:34 |
| dviroel | yeah, looks a good approach | 12:34 |
| chandankumar | one more question, since we have a plan in place, Do we want to document the plan in spec or existing bug would be fine to track? | 12:35 |
| sean-k-mooney | we have one other option by the way, we could vendor a 7 colum parser in watcher. i woudl prefer not to but that is an option if we relaly need that in the future. | 12:35 |
| sean-k-mooney | that a good question | 12:36 |
| sean-k-mooney | i think we can use the exisitng bug | 12:36 |
| sean-k-mooney | we may want to have a bluepirnt or a seocnd bug to track the followup work | 12:37 |
| dviroel | or even create more bugs, like the missing API validation, or for the missing doc | 12:37 |
| dviroel | etc | 12:37 |
| sean-k-mooney | for next cycel and the one after. this does nto feel like it need a spec but im not oppsoed. ya the validation exctra can be tracked seperatly | 12:38 |
| chandankumar | more bugs sounds good. | 12:38 |
| chandankumar | I will add these info the bugs and will update the review based on the plan. | 12:39 |
| dviroel | ack chandankumar | 12:40 |
| chandankumar | That's it I wanted to discuss on croniter swap. | 12:40 |
| chandankumar | Any questions or concerns on this topic before moving to next one. | 12:40 |
| dviroel | tks chandankumar | 12:40 |
| dviroel | we can move, lot to cover yet | 12:41 |
| chandankumar | thank you sean-k-mooney dviroel for the discussion! | 12:41 |
| chandankumar | #topic Open Reviews | 12:41 |
| chandankumar | #link https://review.opendev.org/c/openstack/watcher/+/955711 (Fix api-ref doc for GET /infra-optim/v1/data_model) | 12:41 |
| dviroel | i have a few to request attention | 12:41 |
| dviroel | not going to spend too much time on them | 12:41 |
| chandankumar | dviroel: go ahead | 12:42 |
| dviroel | there is a doc update, pls check the related bug | 12:42 |
| dviroel | #link https://bugs.launchpad.net/watcher/+bug/2117726 | 12:42 |
| dviroel | we can further discuss in the bug | 12:42 |
| dviroel | but the api-ref wasn't reflecting all the fields | 12:43 |
| dviroel | and looking at the code, it seems that they were they since the beginning | 12:43 |
| dviroel | I also added a few unit tests to validate the response: | 12:44 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/955820 | 12:44 |
| dviroel | maybe not the best way to do that, but I accept reviews or proposals for enhancements | 12:44 |
| dviroel | and finally, a small update in the extend compute model attributes spec | 12:45 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher-specs/+/955921 | 12:45 |
| sean-k-mooney | we have api sample tests | 12:45 |
| sean-k-mooney | so we may want to enhace those too | 12:45 |
| dviroel | to also incluse the flavor extra_specs in compute model | 12:45 |
| dviroel | sean-k-mooney: right | 12:46 |
| sean-k-mooney | you still have that last one marked as WIP in geerit | 12:47 |
| sean-k-mooney | most project dont use that feature form my expeirnce but is there a specific reason? | 12:47 |
| dviroel | sean-k-mooney: you are talking about: | 12:47 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/955827 ? | 12:47 |
| dviroel | I found a issue and marked as WIP again, but I can W-1 too | 12:48 |
| sean-k-mooney | ack normally we use -w instaead | 12:48 |
| dviroel | yep | 12:49 |
| sean-k-mooney | part of the reason i prefer that | 12:49 |
| sean-k-mooney | other then avoidign change :) | 12:49 |
| sean-k-mooney | is i likel to leave a commetn why | 12:49 |
| dviroel | done | 12:49 |
| sean-k-mooney | i.e so reviewers knwo what the issue you found is if its not obvious | 12:49 |
| dviroel | yeah, i can will add more details about it in a few | 12:50 |
| dviroel | tks | 12:50 |
| sean-k-mooney | no worreis you mentioend it was an issue with notificatons | 12:50 |
| sean-k-mooney | that basiclly enough to let ohter know "oh this will get revised again" | 12:50 |
| dviroel | ++ | 12:50 |
| chandankumar | there are few more reviews from quangngo in the bottom I am going to cover in this section. If ok? | 12:51 |
| dviroel | chandankumar: sure, pls go ahead, i will get back to extend-compute-model next week | 12:51 |
| chandankumar | Reviews related to Add options to disable migration in host maintenance | 12:51 |
| chandankumar | #link https://review.opendev.org/c/openstack/watcher/+/952538 | 12:52 |
| chandankumar | #link Add tests for disable migration in host maintenance https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/954214 | 12:52 |
| chandankumar | Please take a look at these reviews. | 12:52 |
| sean-k-mooney | that was getting pretty close i think. i looked at much fo the code but not the etsting in detail | 12:52 |
| chandankumar | there are some questions from author on etherpad, let me bring one by one | 12:52 |
| dviroel | i still own reviews there, but it is on my list | 12:53 |
| chandankumar | Is it possible for this feature to appear in 2025.02 release? | 12:53 |
| dviroel | 2025.2 yes right | 12:53 |
| dviroel | we are 4 weeks from the feature freeze | 12:54 |
| sean-k-mooney | yes this will likely be in 2025.2 | 12:54 |
| sean-k-mooney | ubutnu are freee to backport this downstream only to thre distro | 12:54 |
| dviroel | but if the question was 2025.1, that's a no | 12:54 |
| sean-k-mooney | but we wont be backproting this upstream | 12:54 |
| chandankumar | there was one follow up questions also Question for Ubuntu SRU: backportability this feature to any current stable branches? (A no expected, Ubuntu SRU decision just requires upstream confirmation) | 12:54 |
| sean-k-mooney | we also are unlikely to backprot this to our donstream | 12:54 |
| sean-k-mooney | feature are not allowed to be backpaorted understable policy | 12:55 |
| sean-k-mooney | so this was never a backport candiate | 12:55 |
| chandankumar | quangngo: I hope it answers the your queries. | 12:56 |
| sean-k-mooney | https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes | 12:56 |
| dviroel | ++ | 12:56 |
| quangngo | yes, we expect that, ack! | 12:56 |
| dviroel | quangngo: tks for proposing the patches, I will take a look on those | 12:56 |
| chandankumar | Since we have 4 mins left. I am going to move over to next topic | 12:57 |
| dviroel | sure | 12:57 |
| sean-k-mooney | quangngo: in this particalar case canonical likely coudl backprot that enhancement downstream safely | 12:57 |
| sean-k-mooney | but its more risk then we woudl normally take upstream | 12:57 |
| chandankumar | #topic monasca retirement and sdk adoption | 12:58 |
| sean-k-mooney | ya so i added that | 12:58 |
| sean-k-mooney | tl;dr | 12:58 |
| sean-k-mooney | the tc has resolved to continue with the retirement process for monsasca | 12:58 |
| sean-k-mooney | son in the next few weeks the git repos will be retired and there will be no future releases of monasca | 12:59 |
| dviroel | rip monasca | 12:59 |
| sean-k-mooney | 5 months ago we deprecated support | 12:59 |
| sean-k-mooney | and we had planned to remove it in 2026.2 | 12:59 |
| sean-k-mooney | to mitigate the impact of the retirement | 12:59 |
| sean-k-mooney | i plan to work on some targeted patches to make it an optional depency for this cycle | 12:59 |
| sean-k-mooney | we can dicusss for next cycle if we want to acclerate the removal | 13:00 |
| dviroel | +1 | 13:00 |
| sean-k-mooney | or not | 13:00 |
| sean-k-mooney | we have no tempest test or docs so iw as going to propsoe droping it at the start of 2026.1 | 13:00 |
| dviroel | make the conditional import would be great | 13:00 |
| sean-k-mooney | so the follow up to that is we shoudl do the same with all the datasocue and openstack project clients | 13:01 |
| sean-k-mooney | and ideally replace the proejct client with the openstack sdk | 13:01 |
| dviroel | +1 | 13:01 |
| chandankumar | +1 | 13:01 |
| sean-k-mooney | that is work for next cycle | 13:01 |
| chandankumar | thank you sean-k-mooney for bring that up. | 13:01 |
| sean-k-mooney | i will likely draw up a propsoal for that prior to the ptg and either create a spec or blueprint | 13:01 |
| sean-k-mooney | that basicly all i had. | 13:02 |
| chandankumar | Since we are running out of time, I will go with last topic | 13:02 |
| dviroel | sean-k-mooney: thanks for that | 13:02 |
| chandankumar | #topic volunteer to chair for next week meeting | 13:02 |
| chandankumar | Anyone would like to take it? | 13:03 |
| dviroel | i can chair, since I will be out on 14th | 13:03 |
| chandankumar | thanks dviroel | 13:03 |
| chandankumar | time to wrap up | 13:03 |
| dviroel | :) | 13:03 |
| chandankumar | thank you all for attending | 13:03 |
| chandankumar | #endmeeting | 13:03 |
| opendevmeet | Meeting ended Thu Jul 31 13:03:54 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:03 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/watcher/2025/watcher.2025-07-31-12.04.html | 13:03 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/watcher/2025/watcher.2025-07-31-12.04.txt | 13:03 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/watcher/2025/watcher.2025-07-31-12.04.log.html | 13:03 |
| dviroel | thanks chandankumar++ | 13:03 |
| morenod | chandankumar++ | 13:04 |
| sean-k-mooney | dviroel: how do you feel about me tracking making monasca optional as an rfe bug? | 13:49 |
| dviroel | sean-k-mooney: looks fine, there isn't too much details to track as bp or spec I think | 13:50 |
| sean-k-mooney | ack ill file one then to track this and give a breif summary in the descripition | 13:51 |
| sean-k-mooney | the client -> sdk change shoudl likely be a spec or something more formal | 13:52 |
| sean-k-mooney | but we can dicuss that in the future | 13:52 |
| dviroel | agree, the sdk change will be a bigger effort | 13:53 |
| opendevreview | David proposed openstack/watcher-tempest-plugin master: Add custom flavor and dynamic threshold to workload_balance tests https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/953853 | 14:05 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!