| *** pvaneck has quit IRC | 00:01 | |
| *** markvoelker has joined #refstack | 00:29 | |
| *** markvoelker has quit IRC | 00:34 | |
| *** Rockyg has quit IRC | 00:44 | |
| *** markvoelker has joined #refstack | 00:54 | |
| *** clee has quit IRC | 05:10 | |
| *** clee has joined #refstack | 05:11 | |
| *** pilgrimstack has quit IRC | 06:50 | |
| *** pilgrimstack has joined #refstack | 06:53 | |
| *** markvoelker has quit IRC | 07:24 | |
| *** pilgrimstack1 has joined #refstack | 07:34 | |
| *** pilgrimstack has quit IRC | 07:34 | |
| *** pilgrimstack has joined #refstack | 07:57 | |
| *** pilgrimstack1 has quit IRC | 07:57 | |
| *** markvoelker has joined #refstack | 08:24 | |
| *** markvoelker has quit IRC | 08:29 | |
| *** markvoelker has joined #refstack | 10:26 | |
| *** markvoelker has quit IRC | 10:31 | |
| openstackgerrit | Ghe Rivero proposed openstack/refstack-client: Fix regex to find test id https://review.openstack.org/381232 | 10:53 |
|---|---|---|
| *** markvoelker has joined #refstack | 11:27 | |
| *** markvoelker has quit IRC | 11:32 | |
| *** edmondsw has joined #refstack | 12:18 | |
| *** markvoelker has joined #refstack | 12:33 | |
| *** pilgrimstack has quit IRC | 13:36 | |
| *** pilgrimstack1 has joined #refstack | 13:49 | |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted to address performance degradation and should return momentarily | 14:02 | |
| *** openstackgerrit has quit IRC | 16:26 | |
| *** openstackgerrit has joined #refstack | 16:28 | |
| *** openstackgerrit has quit IRC | 16:28 | |
| *** openstackgerrit has joined #refstack | 16:29 | |
| *** openstackgerrit has quit IRC | 16:30 | |
| *** openstackgerrit has joined #refstack | 16:31 | |
| *** selena_092 has joined #refstack | 17:22 | |
| selena_092 | hello, i have question related 2016.08 list of tests...is there a way to retrieve list of tests that are for newton release ? https://refstack.openstack.org/api/v1/guidelines/2016.08/ | 17:23 |
| selena_092 | its very confusing as to which tempest tag should be used for test because lot of tests im getting "not found in Tempest list." | 17:24 |
| selena_092 | thats because tests name have been changing over the time and i guess this is refstack way of ensuring it covers a test for all tempest releases(i.e. its tags) | 17:25 |
| hogepodge | selena_092: can you send us a list of the tests that have changed names? | 17:25 |
| hogepodge | selena_092: we update the current guidelines with new and old test names, and refstack server and client use that to do the right thing | 17:26 |
| selena_092 | for instance "tempest.api.network.test_networks.NetworksTestJSON.test_list_networks NOT FOUND " and thats because of this commit https://github.com/openstack/tempest/commit/e4184ea28e21b86edc28c9f3c02ec28d0d2fb476 , as a consequence if i want to run this i will have to use tempest tag prior to tag 6, however tag prior 12 is not even supported for n | 17:27 |
| selena_092 | ewton | 17:27 |
| selena_092 | hogepodge: sure | 17:27 |
| hogepodge | selena_092: even just a paste will help, we can get a patch in pretty quickly | 17:28 |
| selena_092 | hogepodge: thanks im on it..im pasting on paste.openstack.org one by one..thanks a lot for quick followup | 17:29 |
| hogepodge | selena_092: you're the second person who has mentioned it today, I just want to get a patch in so folks aren't tripped up by it any more | 17:29 |
| selena_092 | hogepodge: heres the list http://paste.openstack.org/show/584270/ | 17:32 |
| selena_092 | this was when i ran tempest master of yesterday with refstack and used the 2016.08 list of tests | 17:33 |
| selena_092 | and my deployment was newton | 17:33 |
| hogepodge | selena_092: thanks | 17:33 |
| hogepodge | selena_092: line 13 was just fixed by this patch: https://review.openstack.org/#/c/373987/4 but the others need to be done too. I'll get on it. | 17:35 |
| selena_092 | hogepodge: thanks so much..for passing the defcore criteria of 2016.08. i should retrieve list of tests like this ? https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=platform&type=required&alias=true&flag=false | 17:37 |
| hogepodge | selena_092: yeah | 17:38 |
| *** pilgrimstack1 has quit IRC | 17:39 | |
| selena_092 | hogepodge: there is a new test i see tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_pagination[id-af55e775-8e4b-4feb-8719-215c43b0238c] thats now aded and also marked as required however i want to mention here this test fails with HTTPS based endpoints of cinder as i found in our environment , i raised a bug here htt | 17:40 |
| selena_092 | ps://bugs.launchpad.net/bugs/1532116 for tempest team, they confirmed its because lack of X-Forwarded-For HTTP header | 17:40 |
| selena_092 | hogepodge: just to reconfirm, after the alias fix , refstack wont complain about "NOT FOUND" error for thoslist of tests ..right? | 17:45 |
| hogepodge | selena_092: it will still complain. from what I understand refstack builds the list of tests to run and includes the aliases | 17:53 |
| hogepodge | selena_092: so the one that doesn't exist will throw that error, but the one that does exist will still run and be reported appropriately | 17:53 |
| hogepodge | selena_092: I'm looking through the list you provided and so far it looks like everything but the one I mentioned already has an alias associated with it | 17:54 |
| hogepodge | catherineD: should be able to help you more on how it works | 17:54 |
| selena_092 | hogepodge: so when submitting the 0.json in .testrepository of test results to refstack API, we will not get failure of tests that had "NOT FOUND " .. | 17:54 |
| selena_092 | catherineD: ^^ | 17:55 |
| hogepodge | selena_092: if the alias was run and passed, it should not be reported as failure | 17:55 |
| selena_092 | hogepodge: ok that makes sense now..please let me know once you push a patch for alias of those lists of list above :) | 17:56 |
| selena_092 | hogepodge: what about https://bugs.launchpad.net/tempest/+bug/1532116 and its impact on tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_pagination ? can you guys mark it flagged? this is a test that passes only if we have HTTP based endpoints but fails for HTTPS based cinder endpoints .. | 17:58 |
| openstack | Launchpad bug 1532116 in tempest "Tempest volume v2 test_volume_list_pagination fails for https based endpoints" [Undecided,New] - Assigned to Daryl Walleck (dwalleck) | 17:58 |
| hogepodge | selena_092: all of the tests have aliases | 18:12 |
| *** pvaneck has joined #refstack | 18:13 | |
| hogepodge | selena_092: can you submit a flag patch to the defcore repository so we can discuss it during out next meeting (which happens tomorrow) | 18:13 |
| *** pilgrimstack has joined #refstack | 18:13 | |
| selena_092 | hogepodge: thanks will do it. | 18:28 |
| hogepodge | selena_092: thanks! flags always carry more weight when it's vendor reported and backed by a bug, so I don't think it will be a problem. | 18:40 |
| *** Rockyg has joined #refstack | 18:43 | |
| selena_092 | hogepodge: can you please share your patch for the aliases of tests i reported ? | 18:49 |
| selena_092 | or they aliases already ? | 18:49 |
| selena_092 | lol i cannot search on review.openstack.org by owner:hogepodge :/ | 18:52 |
| openstackgerrit | Merged openstack/refstack: Fix typo: remove redundant 'the' https://review.openstack.org/380937 | 18:53 |
| hogepodge | selena_092: user owner chris@openstack.org | 18:58 |
| selena_092 | sorry i found it..i was looking if you merged any changes for aliases of those tests i reported to you..? | 18:59 |
| openstackgerrit | Merged openstack/refstack-client: Fix missing dependency in refstack-client https://review.openstack.org/361189 | 19:51 |
| catherineD | luzC: ping | 19:54 |
| luzC | yes | 19:55 |
| luzC | my understanding is that test cases in tempest have a idempotent_id which is unique... | 19:57 |
| luzC | her hand... while we use our implementation - docs are up to date) | 19:57 |
| luzC | <catherine_d|1> there is no doc update from our side because the implementation is a black box to RefStack users | 19:57 |
| luzC | ups wrong copy paste | 19:57 |
| luzC | http://docs.openstack.org/developer/tempest/HACKING.html#test-identification-with-idempotent-id | 19:57 |
| catherineD | see this bug report about uniqueness of the idempotent_id https://bugs.launchpad.net/tempest/+bug/1433700 | 19:59 |
| openstack | Launchpad bug 1433700 in tempest "Multiple test cases associated with same test UUID" [Undecided,Invalid] - Assigned to Sergey Slipushenko (sslypushenko) | 19:59 |
| catherineD | this is reason why even today the guideline are still using fully qualified test name (with aliases) as the testcases name else a list of idempotent_id would ahve been easier | 20:00 |
| luzC | ohhh | 20:02 |
| luzC | I see | 20:02 |
| luzC | then current behavior is wrong... | 20:03 |
| luzC | from ostestr perspective, we still need to pass a whitelist which is a file containing a regex on each line... | 20:06 |
| luzC | sintaxis such as "tempest.api.test2[id-3,smoke]" won't work for ostestr... it will not find coincidences... | 20:08 |
| luzC | we have to still create the withelist using the fully qualified test name such as "tempest.api.test2" | 20:09 |
| *** pilgrimstack has quit IRC | 20:11 | |
| luzC | am I making any sense? | 20:18 |
| catherineD | luzC:what is the format of the whitelist produces by ostestr .. | 20:19 |
| catherineD | selena_092: when retrieving testlist using this command https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=platform&type=required&alias=true&flag=false | 20:21 |
| selena_092 | catherineD: thanks..so even if refstack complains "test NOT FOUND" it wont effect as failure when submiitting to refstack API ..am i right? | 20:22 |
| catherineD | selena_092: this indicates that all aliases of a testcase will be returned. Of the list of aliases only one will match the testcase name of your current tempest version thus the rest will show not found | 20:23 |
| catherineD | correct | 20:23 |
| catherineD | because from refstack point of view we really do not knoiw which version the user is using | 20:23 |
| selena_092 | catherineD: are you guys planning to release 2017.01 soon? | 20:24 |
| selena_092 | thanks | 20:24 |
| catherineD | so they way I will do is to send the data up to refstack and concentrate on debug the failed list | 20:24 |
| catherineD | can you locate the failed list on the web site? | 20:25 |
| selena_092 | i havent submitted yet..just ran and saw that it was complaining about missing tests.. | 20:25 |
| catherineD | I do not have the time line in mind .. usually the new guideline is 3 months lag of the OpenStack release | 20:26 |
| selena_092 | catherineD: this is list of tests that refstack reported as MISSING when i ran latest tempest yesterday with refstack http://paste.openstack.org/show/584270/ | 20:26 |
| catherineD | yea we should have document to descirbe that ... just do not have time yet | 20:26 |
| luzC | @CatherineD the format for whitelist is # Whitelist File | 20:27 |
| luzC | ^regex1 # Include these tests | 20:27 |
| luzC | .*regex2 # include those tests | 20:27 |
| catherineD | yup I have cheked that listI can say all have them have alias soi you should be find | 20:27 |
| selena_092 | catherineD: thanks so much.. | 20:28 |
| catherineD | selena_092: when you submit results pls submit result with signatiure ,... that way yiu can delete the result that you don't wnat and keep our system clean | 20:28 |
| catherineD | selena_092: are you a memeber of openstack? | 20:29 |
| catherineD | we use openstack openid for authentication ... therefore you need to be an openstack member to login as a user at refstack | 20:29 |
| selena_092 | catherineD: yes our team member already is | 20:30 |
| selena_092 | and we submitted and got signed successfully for mitaka :) | 20:30 |
| catherineD | then definitely upload the data with siganture | 20:30 |
| catherineD | selena_092: congrats !!! | 20:31 |
| catherineD | so you are all set? | 20:31 |
| selena_092 | catherineD: yes , thank you so much for the help :) | 20:31 |
| catherineD | one of the thing we really would like the community to do is to test all API (of course after you test the small list of must pass test for ease of de bug ) that way it give us datato have a meaningful defined list for the next guideline like 2017.01 | 20:33 |
| catherineD | luzC: then it is a bit difference then what refstack-client whitelist goal is then? | 20:34 |
| catherineD | seems like we still need refstack-own whitelist | 20:34 |
| catherineD | selena_092: yw | 20:35 |
| luzC | @catherineD ostestr and "tempest run" whitelist are regexes to run the test cases... refstack-client normalization creates a list that matches the names from the guideline to the actual existing names on the tempest installation... | 20:40 |
| luzC | @catherineD I'm going to dig more into the whitelist to fix it (stop using ids on the whitelist)... or try to reuse the full normalized TC list | 20:44 |
| *** selena_092 has quit IRC | 20:44 | |
| *** pilgrimstack has joined #refstack | 20:59 | |
| *** pilgrimstack has quit IRC | 21:04 | |
| luzC | @CatherineD i was taking a look at the bug again... I noticed that ids refer to the same test case, sometimes it uses available protocols or versions... at the end... valid TCs from the guideline should pass right? | 21:22 |
| *** Rockyg has quit IRC | 21:55 | |
| catherineD | luzC: pass fail checking is done strictly using fully qualified testcase (TC) name (with aliases) so if refstack shows test passed that means the TC name matched one of the aliases in the test list | 22:07 |
| *** edmondsw has quit IRC | 22:10 | |
| luzC | catherineD I don't see the harm on running TCs via de id. Worst case scenario will be the same test case running for 2 versions: V1 and V2, even when only one is evaluated against the guideline... | 22:13 |
| luzC | I would recommend keeping the whitelist with idempotent ids... | 22:16 |
| luzC | the other option would be to remove the whitelist and pass normalized list with "--load-list" which os-testr bypass directly to testr | 22:20 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!