Monday, 2019-02-18

rakhmerovhi all06:46
rakhmerovPlease let me know if you plan to attend the PTG in Denver following after the summit06:47
rakhmerovwe need to decided if it makes sense to ask for a dedicated room for Mistral06:48
rakhmerovif there are at least 3 people who could come I'll ask for a room06:48
*** jtomasek has joined #openstack-mistral06:49
*** apetrich has joined #openstack-mistral07:19
*** jtomasek has quit IRC07:27
*** jtomasek has joined #openstack-mistral07:28
*** pgaxatte has joined #openstack-mistral07:40
apetrichmorning07:47
*** chkumar|ruck has joined #openstack-mistral07:57
apetrichrakhmerov, you there?08:01
*** gkadam has joined #openstack-mistral08:06
*** gkadam has quit IRC08:09
*** akovi has joined #openstack-mistral08:12
apetrichd0ugal, ping?08:39
rakhmerovapetrich: yep08:43
rakhmerovI'm here now08:43
apetrichrakhmerov, cheers, so looking at the proposed revert08:43
rakhmerovI've abandoned it08:43
apetrichrakhmerov, is due to this bug https://bugs.launchpad.net/tripleo/+bug/181602608:43
openstackLaunchpad bug 1816026 in tripleo "multinode promotion jobs timing out at overcloud deploy" [Critical,In progress] - Assigned to chandan kumar (chkumar246)08:43
apetrichyeah08:43
apetrichI know08:43
rakhmerovok08:43
rakhmerovhe wrote that it wasn't the cause08:44
apetrichbut comment #808:44
rakhmerovand it really couldn't be IMO08:44
rakhmerovok, looking08:44
apetrichbasically  <% $.running_config_download_workflows != [] %> is returning true always08:44
rakhmerovapetrich: I don't understand his comments in the patch08:49
rakhmerovhe wrote:08:49
apetrichI know :)08:49
rakhmerov“In both jobs, https://github.com/openstack/mistral/commit/58d6634702014eb3c2bf70dbe487fe5983cef8e9 commit is used,08:49
rakhmerovcheck https://bugs.launchpad.net/tripleo/+bug/1816026/comments/8”08:49
openstackLaunchpad bug 1816026 in tripleo "multinode promotion jobs timing out at overcloud deploy" [Critical,In progress] - Assigned to chandan kumar (chkumar246)08:49
apetrichthat's the ci problems and tempest08:49
rakhmerovthe commit that he pointed to is a different one and has nothing to do with the one that's being reverted08:49
apetrichbut what it relates to us is that somehow08:49
apetrichthis  <% $.running_config_download_workflows != [] %> is returning true always08:50
rakhmerovso where exactly should I look at?08:50
apetrichand I'm trying to figure that out08:50
rakhmerovapetrich: where do I find this line?08:50
apetrichhttps://github.com/openstack/tripleo-common/blob/master/workbooks/deployment.yaml#L40408:50
rakhmerovok08:51
apetrichoh sorry. I meant comment #708:51
apetrichnot 808:51
rakhmerovhow is this related?08:51
apetrichmy bad08:51
rakhmerovok08:51
apetrichI'm trying to figure out why. if that somehow has something to do with the string conversion08:52
rakhmerov"str()" is not used here08:52
apetrichif the [] is being somehow converted to "[]" and failing the test08:52
apetrichinternally not explicitly08:52
rakhmerovyes, but I see no reason why it can be converted08:53
apetrichon the other hand looking at comment #9 proposed change it might have nothing to do with us.08:54
apetrichso let ignore that for now and wait for that proposed change go through ci and if it fails and relates to us we can talk again08:54
d0ugalMorning08:55
rakhmerovmorning d0ugal08:55
apetricho/08:55
rakhmerovapetrich: even if it's somehow magically related (I fail to see how though), we need to understand the details08:56
rakhmerovand might want to provide a real fix for this problem08:57
rakhmerovif any08:57
apetrichrakhmerov, aye.08:57
rakhmerovapetrich: I'll clarify. [] cannot be converted into "[]". That patch has nothing to do with that for sure. All it does is pre-processing a data context for08:58
rakhmerovnot an expression08:58
rakhmerov[] here is a part of an expression08:59
rakhmerovthis patch only disables tuplifying lists in a data context (as it's supposed to be), that's it08:59
akoviit would be good to know what times out09:03
rakhmerovyes09:04
rakhmerovd0ugal: did you happen to find out anything new about that dependency problem?09:04
d0ugalrakhmerov: No, not really09:05
rakhmerovok09:05
d0ugalrakhmerov: the cap was removed from oslo.cache tho09:05
d0ugalso maybe that'll help09:05
rakhmerovwhen?09:06
d0ugalToday09:06
rakhmerovI checked in the morning (my morning) and it was still there09:06
d0ugalhttps://review.openstack.org/#/c/636037/09:06
*** vgvoleg has quit IRC09:06
rakhmerovaah, yes09:06
openstackgerritDougal Matthews proposed openstack/mistral master: Update dogpile.cache to match global requirements  https://review.openstack.org/63714509:06
*** vgvoleg has joined #openstack-mistral09:11
rakhmerovd0ugal: your patch is still failing09:14
rakhmerovI guess it's not been enough time yet09:14
d0ugalWe might even need a oslo.cache release, but I am not sure09:15
rakhmerovyeah, I think we need09:16
rakhmerovI guess only released dependencies are pulled in, right?09:17
d0ugalYeah, I think so09:20
openstackgerritQuique Llorente proposed openstack/mistral master: WIP: Test empty list comparation  https://review.openstack.org/63750709:43
d0ugalrakhmerov, akovi: ^09:47
rakhmerovyes09:47
akoviyes09:49
rakhmerovd0ugal: do we need to ask oslo team to release oslo.cache?09:54
d0ugalI think so, you can propose it but they will need the PTL/release liaison to approve it anyway09:54
d0ugalLocally for me that test fails, but if I revert a39db2d3dc21afbe8b9e1d6f8dd870da272b9671 then it passes09:56
d0ugalSo I don't understand in, but it seems there is indeed a regression in that patch09:57
rakhmerovyeah, I see...09:57
rakhmerovweird09:57
rakhmerovok, then we'll have to revert that patch and I'll resend the initial patch when I modify it09:58
d0ugalrakhmerov: quiquell in #tripleo is trying to find a fix. I'll see if they can join here09:59
rakhmerovd0ugal: or we can try to fix it quickly in Mistral09:59
rakhmerovok09:59
d0ugalThat is what I mean, they are looking for a fix to go with that test10:00
rakhmerovok10:00
rakhmerovlet me also check..10:00
d0ugalThanks10:00
*** quiquell|rover has joined #openstack-mistral10:01
quiquell|roverrakhmerov: o/10:01
rakhmerovquiquell|rover: hi, print what you get in $.my_list10:01
rakhmerovboth value and type10:01
quiquell|roverrakhmerov: ack10:02
quiquell|roverrakhmerov: give me a sec10:02
apetrichrakhmerov, I did that10:03
apetrichlist10:03
apetrichnot tuple or string10:03
rakhmerovempty list?10:04
rakhmerov[] ?10:04
apetrichaye10:04
rakhmerovhm...10:04
rakhmerovmystery10:04
d0ugal:)10:04
apetrichso I guess it the != implementation :questionmark10:04
d0ugalapetrich: but isn't that in YAQL itself? Why would it change with the change in mistral?10:05
rakhmerovd0ugal: that's what surprises me most10:06
quiquell|roverok10:06
quiquell|roverhave a quickfix10:07
rakhmerovquiquell|rover: what fix?10:07
rakhmerovI assume that we just actually revealed a hidden bug10:07
openstackgerritQuique Llorente proposed openstack/mistral master: WIP: Test empty list comparation  https://review.openstack.org/63750710:08
quiquell|rover^10:08
quiquell|roverNow it passes10:08
rakhmerovwhat I mean is that this expression could have always been False but it wasn't right10:08
apetrichbool($.my_list) is false as expected10:09
rakhmerovquiquell|rover: do you understand how this fix helped?10:10
quiquell|roverrakhmerov: not really :-)10:10
quiquell|roverI mean looks like empty list is converted into empty tuple =10:10
quiquell|roversince an empty list is not a string not a tuple10:10
quiquell|roverit goes to the wrong if statement10:10
rakhmerovno, it can't be converted into a tuple10:11
rakhmerovif I print $.my_list it's a list10:11
quiquell|rovermake sense10:12
rakhmerovquiquell|rover: let me investigate this quickly..10:13
quiquell|roverrakhmerov: is a promotion blocker for us10:13
quiquell|roverrakhmerov: the quickfix is good enough ?10:13
quiquell|roverrakhmerov: or better go with the revert ?10:13
rakhmerovquiquell|rover: it's not a good idea to merge something that we don't fully understand10:14
rakhmerovthis is a very sensitive area10:15
quiquell|roverrakhmerov: Maybe we can merge the revert but with the unit test ?10:15
quiquell|roverrakhmerov: we mistral is covered ?10:16
rakhmerovquiquell|rover, d0ugal, apetrich, akovi: let's just merge a revert as is for now10:16
d0ugalOkay, so I see the difference10:16
rakhmerovd0ugal: ?10:16
d0ugalOne sec10:17
d0ugalTrying to figure out how to share it10:17
d0ugalhttps://github.com/openstack/mistral/blob/master/mistral/utils/expression_utils.py#L5910:17
d0ugalOn this line, the "outer" list is never passed through yaql_utils.convert_input_data10:17
d0ugalI believe this would be more correct:10:18
quiquell|roverd0ugal: nop, is "reconstructed"10:18
d0ugalreturn yaql_utils.convert_input_data([rec(t, rec) for t in obj], rec)10:18
d0ugaland it seems that yaql_utils.convert_input_data actually converts it into a tuple10:18
quiquell|roverand printing types now10:19
quiquell|rover   ERROR [mistral.utils.expression_utils] []10:19
quiquell|rover   ERROR [mistral.utils.expression_utils] ()10:19
quiquell|rover   ERROR [mistral.utils.expression_utils] []10:19
quiquell|rover   ERROR [mistral.utils.expression_utils] ()10:19
quiquell|roverdamn sorry10:19
quiquell|rover        LOG.error(list(rec(t, rec) for t in obj))10:19
d0ugalIt is hard to know what that means :)10:19
quiquell|rover        LOG.error(yaql_utils.convert_input_data(obj, rec))10:19
d0ugalRight10:20
quiquell|roveryep my last paste was wrong10:20
quiquell|roverso it passes if we convert that into a tuple10:20
quiquell|roverd0ugal: your approach should work10:20
d0ugalquiquell|rover: I think it makes more sense, rather than a special case for an empty list10:21
quiquell|roverd0ugal: le tme refactor it a little10:21
rakhmerovd0ugal: please provide a fix then10:21
d0ugalrakhmerov: I think quiquell|rover is going to do it10:21
rakhmerovok10:21
rakhmerovhonestly, I'm not sure I understand everything you said10:22
d0ugalquiquell|rover: http://paste.openstack.org/show/745250/10:22
rakhmerovwhat do you mean by outer list?10:22
d0ugalrakhmerov: the list()10:22
d0ugalrakhmerov: See that paste, that is the update I made to quiquell|rover's patch10:22
d0ugalrakhmerov: https://github.com/openstack/mistral/blob/master/mistral/utils/expression_utils.py#L5910:22
apetrichd0ugal, sounds like a better solution10:22
d0ugalrakhmerov: that returns a list10:22
openstackgerritQuique Llorente proposed openstack/mistral master: WIP: Test empty list comparation  https://review.openstack.org/63750710:23
quiquell|roverd0ugal: with your approach ^10:23
quiquell|roveris passing10:23
rakhmerovd0ugal: yes10:23
d0ugalbut it should return the result of yaql_utils.convert_input_data, which is a tuple10:23
rakhmerovwhy should it return a tuple?10:23
rakhmerovI don't understand..10:23
d0ugalrakhmerov: because it does everywhere else10:23
rakhmerovwhere?10:23
d0ugalthe yaql language utils converts to a tuple10:23
d0ugalbut that line we don't call it10:23
rakhmerovwait10:23
quiquell|roveryaql thing maybe ?10:23
d0ugalhttps://github.com/openstack/yaql/blob/master/yaql/language/utils.py#L7110:23
rakhmerovwait a second...10:24
quiquell|roverhumm it breaks the other test10:24
rakhmerovd0ugal: the whole idea of my patch was that we stop converting lists into tuples everything recursively10:24
rakhmerovquiquell|rover: sure it does10:24
rakhmerov... everything -> everywhere ...10:24
d0ugalThen we can't use the YAQL convert_input_data function, because it converts to tuples10:25
d0ugal:)10:25
rakhmerovwe don't use it after my patch10:25
rakhmerovpay attention to "rec"10:25
rakhmerovit stands for "recursive"10:25
d0ugalbut we must somewhere, or why do we have a tuple in one place and a list in the other? Which is why the expression fails10:26
rakhmerovin the very beginning of my function we do rec = _convert_yaql_input_data10:26
rakhmerovd0ugal: where do we have a tuple?10:26
d0ugalCould YAQL convert a listeral [] into a tuple?10:26
rakhmerovd0ugal: I'm printing right in the code and it's always a list10:26
rakhmerovnot a tuple10:26
rakhmerovd0ugal: no, I just checked that10:27
rakhmerovit's always a list10:27
d0ugalSo why does the comparison change :)10:27
rakhmerovI don't know yet10:27
rakhmerov:)10:27
d0ugal"Default #list function implementation in standard library produces a list (tuple) comprised of given elements. However, the host might decide to give it a different implementation."10:27
d0ugalFrom the YAQL docs10:27
d0ugalhttps://yaql.readthedocs.io/en/latest/language_reference.html#list-expressions10:28
d0ugalIt seems to consider lists and tuples as the same thing?10:28
rakhmerovd0ugal: like I said, we might have just revealed the other bug10:28
rakhmerovthat expression in the TripleO workflow could just always return False10:28
rakhmerovbut it wasn't right10:29
d0ugalquiquell|rover: as a workaround in tripleo we could check the lengh is 010:29
d0ugallength10:29
rakhmerovyes10:29
rakhmerovd0ugal: it is more reliable10:29
d0ugalSomething is wrong here, but that should unblock it10:29
quiquell|rovergive me sec10:31
rakhmerovd0ugal, quiquell|rover: please let's not hurry with quick workarounds in Mistral10:31
quiquell|roverrakhmerov: sure no problem10:31
rakhmerovif you can change a condition in TripleO, it'd be good10:31
rakhmerovin the meantime I'll investigate what's going on10:32
quiquell|roverrakhmerov: just one to know what is the best a approach here10:32
rakhmerovok10:32
rakhmerovthis place is sensitive and pretty tricky, we need to be careful10:32
quiquell|roverrakhmerov, d0ugal: so we check length or we revert ?10:32
quiquell|roverunit test is legit though10:32
d0ugalquiquell|rover: I would suggest checking the length. That should work either way10:32
d0ugalquiquell|rover: agreeded, we need a fix in mistral but we also need to not break the other test :)10:33
d0ugalquiquell|rover: <% $.running_config_download_workflows.len() > 0 %>10:34
d0ugalquiquell|rover: <% $.running_config_download_workflows.len() = 0 %>10:34
d0ugalquiquell|rover: I guess that is the two we need.10:35
quiquell|roverd0ugal: let me add them10:35
rakhmerovd0ugal: just checked again in debugger, conversion works as expected, it's a list10:35
rakhmerovok, I'll look at it more..10:35
d0ugalFWIW, it seems that YAQL strongly suggests we use tuples everywhere10:44
rakhmerovno10:44
rakhmerov:)10:44
d0ugalyes10:44
d0ugal"Strongly prefer immutable data structures over mutable ones. Use tuple`s rather than `list`s, `frozenset instead of set. Python does not have a built-in immutable dictionary class so yaql provides one on its own - yaql.language.utils.FrozenDict."10:44
d0ugalhttps://yaql.readthedocs.io/en/latest/extending_yaql.html?highlight=converttuplestolists#function-development-hints10:44
quiquell|roverI am openning a pandora Âbox here ?10:45
d0ugalyup :)10:45
rakhmerovd0ugal: I know about it, yes10:45
d0ugalok10:45
rakhmerovbut the reason is just to use immutable types10:45
rakhmerovlist itself is totally ok10:45
d0ugalWhich seems like a good reason10:45
d0ugalok10:46
rakhmerovyes, but we know how we use YAQL in Mistral10:46
rakhmerovwe always make sure that the data context is not modified while YAQL is doing something with it10:46
openstackgerritQuique Llorente proposed openstack/mistral master: WIP: Test empty list comparation  https://review.openstack.org/63750710:46
quiquell|roverrakhmero, d0ugal: with len ^10:46
rakhmerovyes10:46
rakhmerovd0ugal: just hold on please, I'll investigate :)10:46
rakhmerovHave to run now though, will do it a little later10:46
d0ugalI'm not doing anything10:46
d0ugalI am just reading10:47
rakhmerovok10:47
quiquell|roverrakhmerov: we can merge the revert though ?10:47
quiquell|roverrakhmerov: while we do proper solution ?10:47
quiquell|roverrakhmerov: or len is enouogh ?10:47
d0ugalquiquell|rover: hmm, we don't need the len in mistral10:47
d0ugalquiquell|rover: we should add that to the workflow in tripleo-common10:47
d0ugaland then we shouldn't need a change in Mistral10:47
quiquell|roverd0ugal: you mean we should compare with len instead of [] at workflow ?10:48
d0ugalquiquell|rover: yup10:48
quiquell|roverd0ugal: so we don't have to change mistral to fix stuff ?10:48
rakhmerovquiquell|rover: let's not merge the revert10:48
d0ugalexactly10:48
d0ugaland we can leave mistral broken for now lol10:48
rakhmerovthe patch fixed another compatibility problem10:48
quiquell|roverd0ugal: let me see the impact though10:48
rakhmerovfor a different user10:48
rakhmerovd0ugal: we're not sure exactly that it's now broken )10:49
rakhmerovthe investigation is not over10:49
d0ugalrakhmerov: I agree, but it doesn't seem correct either10:49
d0ugalThat patch shouldn't have changed the behaviour of the YAQL expression10:49
d0ugalso I'm pretty certain it was broken10:50
openstackgerritQuique Llorente proposed openstack/mistral master: Return empty list as tuple  https://review.openstack.org/63750710:50
quiquell|roverUpdateed commit message10:50
rakhmerovlet's not argue anymore :)10:50
quiquell|roverGoing to look at workflows10:50
d0ugalsure10:50
d0ugalI just don't know what needs to change to fix it10:50
rakhmerovd0ugal: again, think about it. The expression could have been always evaluating in a wrong way :)10:50
rakhmerovcan you assume that too? :)10:51
rakhmerovmay be [] can't be really used as well10:52
rakhmerovin expressions10:52
rakhmerovI'm not sure about that either10:52
quiquell|roverrakhmerov, d0ugal: Only those two places use this comparation at tripleo-common10:52
d0ugalrakhmerov: but the unit test result was different when reverting the patch - so I'm not sure how it could always have been that way10:52
rakhmerovd0ugal: hm.. yes.. ok10:53
rakhmerovright10:53
rakhmerovdamn.. ok10:53
rakhmerovok, I'm gone10:53
d0ugalSure10:53
d0ugalThanks for the help10:53
rakhmerovwill be back later10:53
d0ugalquiquell|rover: Yeah, I checked that too. We use len in a couple of places10:53
quiquell|roverd0ugal: so sure it was a known issue10:54
d0ugalquiquell|rover: I don't think so, I think I have used len because it seems more logical to me10:54
quiquell|roverd0ugal: two in one :-)10:55
quiquell|roverd0ugal: this is it ? https://review.openstack.org/63752210:55
quiquell|roverd0ugal: well previous mistral version was good though10:55
d0ugalAgreed10:55
quiquell|roverso mistral is no good there10:55
d0ugallol10:56
quiquell|roverOr it cannot drop support for [] comparation10:56
quiquell|rovers/casnnot/can/10:56
quiquell|roverchkumar|ruck: ^10:57
d0ugalquiquell|rover: Sure, lets not worry about that now. I think the change in approach will solve the problems in tripleo10:57
chkumar|ruckquiquell|rover: yes looks, good it will help to unblock the ci10:57
d0ugalbbiab10:58
* chkumar|ruck is going to test it right now10:58
quiquell|rovercool thank sa lot rakhmerov, d0ugal10:58
quiquell|roverchkumar|ruck: go go go go10:58
*** apetrich has quit IRC12:11
*** apetrich has joined #openstack-mistral12:25
*** apetrich has quit IRC12:30
*** apetrich has joined #openstack-mistral12:48
*** quiquell|rover is now known as quique|rover|eat13:28
openstackgerritAdriano Petrich proposed openstack/mistral master: Return empty list as tuple  https://review.openstack.org/63750713:40
*** akovi has quit IRC13:47
*** akovi has joined #openstack-mistral13:48
*** quique|rover|eat is now known as quiquell|rover14:09
*** chkumar|ruck is now known as chandankumar14:28
*** quiquell|rover is now known as quiquell|off15:11
rakhmerovd0ugal: [] is internally represented as a tuple15:43
d0ugalrakhmerov: Yeah, that is what I thought15:43
rakhmerovthat's why the comparison fails15:43
rakhmerovyeah..15:43
rakhmerovI don't like all this stuff )15:43
rakhmerovok, I'll try to come up with some idea15:44
d0ugaland I guess that is why they suggest using tuple15:44
d0ugalIt is a bit messy15:44
rakhmerovd0ugal: the interesting thing is that we used to use YAQL w/o any context pre-processing at all15:45
rakhmerovand most of the things worked15:45
rakhmerovexcept list of dicts15:45
rakhmerovsorry, sets of dicts15:45
d0ugalInteresting15:45
rakhmerovyes, because dict is an unhashable type15:46
rakhmerovit can't be put into a set15:46
d0ugalRight, that makes sense15:46
d0ugalI have hit that before :)15:46
rakhmerovthat's why we started using yaql_utils.convert_input_data in the first place15:46
rakhmerovto replace sets with something else15:46
rakhmerovit is a mess, indeed15:47
*** akovi has quit IRC15:50
rakhmerovd0ugal, apetrich: btw, the special handling of [] is not going to help because we can also use a non-empty string literal in an expression16:14
rakhmerovlike [1,2,3]16:14
rakhmerovso16:14
rakhmerovmost likely we'll have to revert my patch16:14
rakhmerov:)16:14
rakhmerovbut give me some time, I want to debug YAQL a little bit more16:15
apetrich:(16:15
*** pgaxatte has quit IRC16:57
d0ugalrakhmerov: Agreed, that is why I added a -1 :)18:14
d0ugalLet us know when you have something ready to review18:14
*** jtomasek has quit IRC21:40

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!