| opendevreview | Goutham Pacha Ravi proposed openstack/python-manilaclient master: Refactor code from oslo_incubator https://review.opendev.org/c/openstack/python-manilaclient/+/803421 | 06:08 |
|---|---|---|
| opendevreview | Goutham Pacha Ravi proposed openstack/python-manilaclient master: Refactor code from oslo_incubator https://review.opendev.org/c/openstack/python-manilaclient/+/803421 | 06:12 |
| opendevreview | Archana Kumari proposed openstack/python-manilaclient master: [OSC] Implement Share Group Commands https://review.opendev.org/c/openstack/python-manilaclient/+/801740 | 06:47 |
| opendevreview | Goutham Pacha Ravi proposed openstack/python-manilaclient master: WIP: Metadata for all user facing resources https://review.opendev.org/c/openstack/python-manilaclient/+/803424 | 07:37 |
| opendevreview | Archana Kumari proposed openstack/python-manilaclient master: [OSC] Implement Share Group Commands https://review.opendev.org/c/openstack/python-manilaclient/+/801740 | 08:04 |
| fzzf | Hi, I'm doing tempest test on driver, I have encountered problem. https://paste.opendev.org/show/807879/ The first two are 'tempest run' show. In the end is m-api log. this occur when tearDownClass .But when I execute it alone, it all success. I have modify rest server performance params, but it doesn't work. I adjust requests method with timeout=None is no useless. | 09:29 |
| gouthamr | fzzf: when you see "Target share type still in use"; its usually an indication that a prior test failed, and we're unable to delete the share from that test failure - and re-attempts have failed as well; do you have the full tempest log? | 09:40 |
| fzzf | gouthamr: How to export tempest logs | 09:42 |
| vkmc | fzzf, o/ how are you running tempest? | 09:43 |
| vkmc | redirecting the output of the command to a file should be enough | 09:43 |
| fzzf | vkmc: I run tempest run -r manila_tempest_tests.tests.api | 09:44 |
| vkmc | fzzf, oki, you can do "tempest run -r manila_tempest_tests.tests.api > tempest-output" and then paste the content of that file in paste.openstack.org | 09:45 |
| vkmc | so we get the full log | 09:45 |
| fzzf | vkmc:umm,I save screen as file. it's ok | 09:47 |
| vkmc | fzzf, not sure if I follow, what do you mean with save screen? | 09:49 |
| fzzf | vkmc: I can't paste it on paste.opendev.org. it show Could not submit your paste because your paste contains spam. | 09:52 |
| vkmc | fzzf, maybe that's because of the length | 09:53 |
| vkmc | try centos pastebin https://pastebin.centos.org/ | 09:53 |
| fzzf | vkmc: ok. I paste it. https://paste.centos.org/view/22425b6a | 09:55 |
| vkmc | awesome, thx | 09:55 |
| vkmc | fzzf, sorry, got in a meeting | 10:43 |
| vkmc | fzzf, so I see several timeouts, which might be related to 1. an unhealthy environment (check m-shr, m-sch and m-api logs) 2. not enough resources | 10:44 |
| vkmc | probably 1. | 10:44 |
| vkmc | could you check those logs? | 10:44 |
| opendevreview | Goutham Pacha Ravi proposed openstack/python-manilaclient master: WIP: Metadata for all user facing resources https://review.opendev.org/c/openstack/python-manilaclient/+/803424 | 10:45 |
| fzzf | vkmc: it's fine. I check logs.there are error correspond to tempest log.https://paste.opendev.org/show/807882/ . But I have no idea to fix it. what is 2. mean? | 10:59 |
| *** dviroel|out is now known as dviroel | 11:16 | |
| opendevreview | Archana Kumari proposed openstack/python-manilaclient master: [OSC] Implement Share Group Commands https://review.opendev.org/c/openstack/python-manilaclient/+/801740 | 11:32 |
| vkmc | fzzf, would need more information... from the logs | 12:02 |
| vkmc | fzzf, try running a subset of tests | 12:02 |
| vkmc | fzzf, what is the output of "tempest run -r manila_tempest_tests.tests.api.test_shares_actions"? | 12:02 |
| fzzf | vkmc: when I run subset of tests. it all success exclude some skipped | 12:11 |
| fzzf | https://paste.opendev.org/show/807884/ | 12:12 |
| vkmc | fzzf, ok, so let's try "tempest run -r --concurrency 2 manila_tempest_tests.tests.api" | 12:27 |
| vkmc | if that doesn't work, I guess we can try running test serially... it might take more time, but should take less resources | 12:29 |
| fzzf | vkmc: is this? tempest run -r --concurrency 2 manila_tempest_tests.tests.api | 12:32 |
| fzzf | tempest run: error: argument --regex/-r: expected one argument | 12:32 |
| fzzf | when i run | 12:32 |
| vkmc | fzzf, ah, sorry, "tempest run --concurrency 2 -r manila_tempest_tests.tests.api" | 12:32 |
| vkmc | there | 12:32 |
| fzzf | ok | 12:33 |
| fzzf | what is this params using | 12:34 |
| fzzf | limit thread? | 12:34 |
| vkmc | fzzf, https://docs.openstack.org/tempest/latest/run.html#test-execution | 12:35 |
| vkmc | yep | 12:35 |
| fzzf | vkmc:not enough resources is refer to manila's server or backend server | 12:42 |
| vkmc | fzzf, hmm, I'd try playing with those params, maybe trying the serial approach... seems it's a problem with resources you have, because the environment otherwise is healthy | 12:43 |
| fzzf | vkmc: I don't understand. My tempest plugin is installed on the same computer as manila.Is this right? resources problem is refer to this computer or backend device computer? | 12:50 |
| vkmc | fzzf, is this a development environment? | 12:55 |
| fzzf | vkmc:yes | 12:56 |
| vkmc | fzzf, I assume it is a virtual machine? | 12:56 |
| fzzf | vkmc:no,it's a physical server | 12:57 |
| vkmc | fzzf, the deployment is fine, I mean, having tempest in the same place as you have manila | 12:58 |
| vkmc | fzzf, but if we analyze a bit things, the deployment seems to be healthy if it's the one you are using and you don't see any error on the logs... running a subset of the tests succeed because it doesn't require so much compute power to run... running the full suite, though, starts throwing timeouts | 13:00 |
| vkmc | so that makes me think there might be some resources constrains | 13:01 |
| vkmc | I might need more info to understand the real problem | 13:01 |
| vkmc | so that's why I advise you to try running the integration suite with lower concurrency/serially... since that might require less resources | 13:02 |
| vkmc | sorry I cannot help more | 13:02 |
| vkmc | maybe someone else has a better idea though ^ | 13:02 |
| fzzf | vkmc:yes,it seem like computer resource is not enough. | 13:03 |
| vkmc | fzzf, tempest is quite resource demanding :( | 13:03 |
| vkmc | we sometimes hit similar issues in our CI | 13:04 |
| fzzf | I have not installed CI.Should CI be installed in the same place as manila? | 13:05 |
| vkmc | fzzf, oh no, I don't think you should have that in your development environment | 13:06 |
| fzzf | vkcm: I have finsh tempest. it success | 13:06 |
| vkmc | \o/ | 13:07 |
| fzzf | vkmc:sorry. it's a testing environment acutally | 13:07 |
| vkmc | awesome! | 13:07 |
| vkmc | with concurrency set to 2? | 13:07 |
| fzzf | vkmc: yes | 13:07 |
| vkmc | sweet! | 13:07 |
| vkmc | fzzf, so maybe you can try increasing concurrency a bit so next run doesn't take so much for you... I believe default is 8, so you can try with 4 or 6 | 13:08 |
| fzzf | vkmc: What do you mean by development environment? this environment is built to complete manila's driver test. | 13:10 |
| vkmc | fzzf, I mean an environment you/your team is using for bug fixes and/or feature development | 13:11 |
| vkmc | or for testing some specific functionality | 13:11 |
| vkmc | I get you are testing a manila driver | 13:11 |
| fzzf | vkmc: I'm doing new dirver test. I have asked you before If you remember. Should the CI environment and tempest be installed together? | 13:17 |
| fzzf | Now I only have installed manila and tempest in one server. | 13:18 |
| vkmc | fzzf, so... if you are developing a new driver then you need to submit a third party CI for your driver | 13:20 |
| vkmc | maybe carloss can give more context on this ^ | 13:20 |
| fzzf | vkmc: umm,yes,I have asked before. I just don't understand Should CI be installed in a new environment? | 13:21 |
| vkmc | fzzf, no, you don't need to install CI in a new environment | 13:22 |
| fzzf | I mean the environment don't have manila and tempest | 13:22 |
| carloss | iirc tempest will try to use all cores available in your machine, that's why we usually set a limit for it. The NetApp CI usually runs tempest using only 2 cores (--concurrency=2). It makes the execution last longer | 13:28 |
| carloss | gathering more information about third party CI, just a sec... | 13:32 |
| carloss | :) | 13:32 |
| vkmc | thanks carloss :) | 13:32 |
| vkmc | I don't know much about third party CI, I thought we have a bit more of docs about that | 13:32 |
| vkmc | maybe I'm looking on the wrong place | 13:33 |
| fzzf | carloss: thanks. I'm recommend use Software Factory as the base for new CI system.https://wiki.openstack.org/wiki/CinderXenaPTGSummary#Using_Software_Factory_for_Cinder_Third_Party_CI | 13:35 |
| carloss | fzzf: here's some more information about the third party CI: https://docs.opendev.org/opendev/system-config/latest/third_party.html | 14:24 |
| carloss | Yes, the software factory to deploy the third party CI is a great tool and people are using it a lot - it makes lives easier when deploying zuul, gerrit and other tools. It's higly recommended and Pure recently used it and document some steps, so it's worth reading their documentations and watching their presentation in the previous ptg | 14:24 |
| carloss | One of the requirements to have your driver merged is that you have a third party CI running the tempest tests using your driver and your backend, so we can certify that the operations that were implemented are working properly. Also, the third party CI must be voting in the openstack/manila changes (not only the one that introduces the new driver - we require that because there are lots of changes happening at the same | 14:24 |
| carloss | time in manila and the third party CI output is a good thermometer to check if such changes are not going to break the drivers). | 14:24 |
| carloss | Given that, here is a high level (very high level :p) overview of how the CI works for NetApp as an example | 14:24 |
| carloss | 1. We keep listening to the upstream gerrit. As soon as we detected that zuul voted +1 in a change, we go to phase 2. | 14:24 |
| carloss | 2. We get this change and we launch a new node (node = new VM in nodepool) or get one that was waiting to be used. The node is created using an operational system image we have configured previously | 14:24 |
| carloss | 3. We configure our backends - we have a set of storages dedicated to running tests, so we get one device and add it in the local.conf of the devstack, and manila can do operations there | 14:24 |
| carloss | 4. Set up devstack (we enable tempest in the installation) | 14:24 |
| carloss | 5. When the setup is done, we run the tempest tests, and the backends that are going to be used to do so are NetApp backends, since they are the ones we kept in enabled_share_backends | 14:24 |
| carloss | 6. We wait for the tests to be completed, get the logs and save them in a place where is available for everyone to see (people should be able to check the logs of the tests execution) - here's an example of a test result output: https://logs.openstack.netapp.com/logs/41/734041/15/upstream-check/manila-cDOT-ss/d9cc6a7/ | 14:24 |
| carloss | 7. Post the vote and the logs in the upstream gerrit | 14:24 |
| carloss | There might be other ways to do it as well... In the way we do, it works quite okay, because whenever our zuul realizes a new change as added, we just launch a new vm and when tests are done, the vms are immediately dropped. | 14:24 |
| carloss | So in a brief summary: we have the CI system, it launches nodes (VMs), we install devstack with tempest in those nodes, we collect the results of the tests, make them available and vote in the change | 14:24 |
| carloss | s/(node = new VM in nodepool)/(node = VM managed by nodepool, but created in a hypervisor dedicated to setup vms) | 14:27 |
| carloss | zuul/nodepool will automate a lot of this things for you :) | 14:28 |
| *** dviroel is now known as dviroel|out | 20:38 | |
| opendevreview | Ashley Rodriguez proposed openstack/manila master: WIP:Add Share Snapshot Metadata Resource https://review.opendev.org/c/openstack/manila/+/801372 | 22:44 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!