| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add even more debug logs. https://review.openstack.org/49458 | 00:15 |
|---|---|---|
| *** matsuhashi has joined #openstack-qa | 00:32 | |
| openstackgerrit | A change was merged to openstack-dev/devstack: Change DATABASE_HOST default to 127.0.0.1 https://review.openstack.org/49409 | 00:34 |
| *** openstackrecheck has quit IRC | 01:01 | |
| *** openstackrecheck has joined #openstack-qa | 01:01 | |
| *** nosnos has joined #openstack-qa | 01:04 | |
| *** jaypipes has quit IRC | 01:11 | |
| *** sthaha has joined #openstack-qa | 01:34 | |
| dims | jog0, Am going to run rechecks on this review - https://review.openstack.org/#/c/49403/ (for the bogus fd issue) | 01:37 |
| openstackgerrit | A change was merged to openstack-dev/devstack: xenapi: set dhcp timeout on VM installation https://review.openstack.org/48630 | 01:42 |
| openstackgerrit | Clark Boylan proposed a change to openstack/tempest: Double test processes. https://review.openstack.org/49466 | 01:49 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/49233 failed tempest with an unrecognized error | 02:04 |
| *** mlavalle has joined #openstack-qa | 02:07 | |
| *** mlavalle has quit IRC | 02:08 | |
| dkranz | mtreinish: Sorry, I had to leave a little early today. | 02:15 |
| openstackgerrit | Clark Boylan proposed a change to openstack/tempest: Double test processes https://review.openstack.org/49466 | 02:28 |
| openstackgerrit | A change was merged to openstack/tempest: Heat: use first_address instead of first_private_address https://review.openstack.org/48868 | 02:51 |
| openstackgerrit | Masayuki Igawa proposed a change to openstack/tempest: Fix to use proper random values https://review.openstack.org/46886 | 03:04 |
| *** mtreinish has quit IRC | 04:28 | |
| *** anteaya has quit IRC | 04:30 | |
| *** Madan has joined #openstack-qa | 04:33 | |
| *** mtreinish has joined #openstack-qa | 04:42 | |
| *** raies has joined #openstack-qa | 04:48 | |
| openstackgerrit | Lingxian Kong proposed a change to openstack/tempest: add server suspend/resume negative tests https://review.openstack.org/48814 | 04:49 |
| *** kong has joined #openstack-qa | 04:49 | |
| openstackgerrit | Lingxian Kong proposed a change to openstack/tempest: add server suspend/resume negative tests https://review.openstack.org/48814 | 04:52 |
| *** SergeyLukjanov has joined #openstack-qa | 04:56 | |
| openstackgerrit | Lingxian Kong proposed a change to openstack/tempest: add server delete/unpause tests https://review.openstack.org/48796 | 04:59 |
| *** jaypipes has joined #openstack-qa | 05:39 | |
| *** ChanServ sets mode: +o jaypipes | 05:39 | |
| *** lynnkong has joined #openstack-qa | 05:49 | |
| openstackgerrit | Mh Raies proposed a change to openstack/tempest: Adding testcase for dhcp agent scheduler https://review.openstack.org/48632 | 05:51 |
| *** kong has quit IRC | 05:53 | |
| raies | jaypipes: hi, if you have time please have a look on https://review.openstack.org/#/c/47339/ | 05:53 |
| *** prekarat has quit IRC | 05:54 | |
| *** SergeyLukjanov has quit IRC | 06:17 | |
| *** mkoderer_ is now known as mkoderer | 06:26 | |
| openstackrecheck | openstack/tempest change: https://review.openstack.org/48632 failed tempest with an unrecognized error | 06:33 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/49211 failed tempest with an unrecognized error | 06:38 |
| *** SergeyLukjanov has joined #openstack-qa | 06:41 | |
| *** SergeyLukjanov has quit IRC | 06:42 | |
| *** shilpa_ has joined #openstack-qa | 06:43 | |
| *** bashok has joined #openstack-qa | 06:45 | |
| *** ko_ has joined #openstack-qa | 06:46 | |
| *** jlibosva has joined #openstack-qa | 06:51 | |
| *** yeylon_ has joined #openstack-qa | 06:53 | |
| *** afazekas has joined #openstack-qa | 07:11 | |
| *** bashok_ has joined #openstack-qa | 07:12 | |
| *** bashok has quit IRC | 07:15 | |
| *** prekarat has joined #openstack-qa | 07:21 | |
| *** prekarat has quit IRC | 07:26 | |
| *** giulivo has joined #openstack-qa | 07:28 | |
| *** prekarat has joined #openstack-qa | 07:31 | |
| *** ko_ has quit IRC | 07:34 | |
| openstackgerrit | A change was merged to openstack/tempest: Unskip test_tokens and update expected status to 404 from 401 https://review.openstack.org/49035 | 07:42 |
| *** fbo_away is now known as fbo | 07:55 | |
| *** SergeyLukjanov has joined #openstack-qa | 08:04 | |
| *** tkammer has joined #openstack-qa | 08:05 | |
| openstackgerrit | Lingxian Kong proposed a change to openstack/tempest: add some negative tests for force_delete/restore https://review.openstack.org/49488 | 08:20 |
| *** yassine has joined #openstack-qa | 08:23 | |
| *** thomasbiege1 has joined #openstack-qa | 08:27 | |
| *** thomasbiege1 has quit IRC | 08:36 | |
| openstackrecheck | openstack/tempest change: https://review.openstack.org/49488 failed tempest with an unrecognized error | 08:51 |
| *** tkammer has quit IRC | 08:56 | |
| *** matsuhashi has quit IRC | 09:01 | |
| *** matsuhashi has joined #openstack-qa | 09:09 | |
| raies | afazekas: hi | 09:14 |
| raies | I need your help | 09:14 |
| *** thomasbiege1 has joined #openstack-qa | 09:16 | |
| *** prekarat has quit IRC | 09:16 | |
| *** prekarat has joined #openstack-qa | 09:17 | |
| raies | afazekas: I need router xml support for vpnaas xml support | 09:17 |
| raies | I can implement these but you have done already | 09:17 |
| raies | so tell me how could I use it ? | 09:18 |
| afazekas | I'll update that change, and you can rebase a change to another persons change as dependency | 09:21 |
| openstackgerrit | Lingxian Kong proposed a change to openstack/tempest: add some negative tests for force_delete/restore https://review.openstack.org/49488 | 09:21 |
| *** mkollaro has joined #openstack-qa | 09:22 | |
| *** thomasbiege2 has joined #openstack-qa | 09:23 | |
| *** thomasbiege3 has joined #openstack-qa | 09:25 | |
| *** thomasbiege1 has quit IRC | 09:26 | |
| openstackgerrit | afazekas proposed a change to openstack/tempest: Add xml support to the floating ip and router https://review.openstack.org/46050 | 09:26 |
| *** thomasbiege2 has quit IRC | 09:27 | |
| afazekas | mkdorer: I have moved the method to a common place | 09:27 |
| afazekas | mkoderer: I have moved the method to a common place | 09:27 |
| raies | afazekas: | 09:27 |
| raies | its ok but vpnaas xml support is dependent on one of my code | 09:28 |
| raies | then how can I make two dependency of same code | 09:28 |
| openstackrecheck | openstack/cinder change: https://review.openstack.org/48938 failed tempest with an unrecognized error | 09:29 |
| afazekas | my guess would be depending on a merge change, but I never tried it | 09:29 |
| afazekas | BTW: this got -1 from jenkins, and looks like it has related exception https://review.openstack.org/#/c/47339/ | 09:30 |
| *** jhenner has joined #openstack-qa | 09:31 | |
| openstackgerrit | Lingxian Kong proposed a change to openstack/tempest: add some negative tests for attach_interfaces https://review.openstack.org/49231 | 09:32 |
| raies | afazekas: https://review.openstack.org/#/c/47339/ and its dependents are not being approved for too long time :) | 09:32 |
| raies | please look into the mater | 09:32 |
| afazekas | It does not have any pending dependency now | 09:32 |
| raies | no no I mean the patches which are dependent of https://review.openstack.org/#/c/47339/ | 09:33 |
| raies | such as https://review.openstack.org/#/c/47348/ | 09:34 |
| mkoderer | afazekas: ok thx | 09:34 |
| mkoderer | afazekas: I am working on some more debug messages for the stress test | 09:35 |
| mkoderer | to know why it fails sometimes | 09:35 |
| *** thomasbiege3 has quit IRC | 09:35 | |
| afazekas | raies: the dependent patches cannot be merged before the dependency, even if they are approved, they will just wait until the decency merged | 09:35 |
| * afazekas new default plugin https://review.openstack.org/#/c/47837/ | 09:37 | |
| raies | afazekas: yes I know, I think you mean me wrong. I wanted to tell that https://review.openstack.org/#/c/47339/ approval is must. If possible please look for https://review.openstack.org/#/c/47339/ | 09:37 |
| afazekas | raies: the first failure is: http://www.fpaste.org/44008/93123138/ | 09:39 |
| raies | afazekas: I am implementing vpnaas xml support. for this I need your code https://review.openstack.org/#/c/46050/ for router xml support. Please do for https://review.openstack.org/#/c/46050/ also so that I can commit my vpnaas xml support | 09:39 |
| afazekas | you can try to recheck it | 09:39 |
| raies | you mean recheck no bug ? | 09:39 |
| raies | ?? | 09:40 |
| *** sthaha has quit IRC | 09:49 | |
| *** sthaha has joined #openstack-qa | 09:50 | |
| *** sthaha has quit IRC | 09:50 | |
| *** sthaha has joined #openstack-qa | 09:50 | |
| afazekas | raies: yes | 09:50 |
| raies | thanks I did :) | 09:51 |
| *** jhenner has quit IRC | 09:54 | |
| *** jhenner has joined #openstack-qa | 09:55 | |
| openstackgerrit | Lingxian Kong proposed a change to openstack/tempest: add two negative tests for flavor-access https://review.openstack.org/49501 | 10:01 |
| *** lynnkong has quit IRC | 10:01 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/devstack-gate: Ensure devstack is launched with proper tenant isolation setting https://review.openstack.org/49504 | 10:05 |
| *** jhenner has quit IRC | 10:09 | |
| *** jhenner has joined #openstack-qa | 10:10 | |
| openstackgerrit | Masayuki Igawa proposed a change to openstack/tempest: Fix to use proper random values https://review.openstack.org/46886 | 10:13 |
| raies | afazekas: hi Please have a look on https://review.openstack.org/#/c/48632/ | 10:15 |
| openstackgerrit | afazekas proposed a change to openstack/tempest: Make the call command as root helper configure able https://review.openstack.org/49506 | 10:16 |
| afazekas | before the raising the skipExceptions the isolated credentials should be cleaned up | 10:17 |
| mkoderer | github is down :( | 10:20 |
| openstackrecheck | openstack/tempest change: https://review.openstack.org/47339 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 10:21 |
| afazekas | raies: looks like the ext network does not have dhcp | 10:24 |
| raies | afazekas: where are you talking about in code ? | 10:24 |
| raies | ?? | 10:26 |
| afazekas | https://review.openstack.org/#/c/48632/3/tempest/api/network/test_dhcp_agent_scheduler.py L43 is it return with non empty list ? | 10:26 |
| afazekas | usually the dhcp on the external net is disabled | 10:27 |
| raies | when I had checked that an empty list was returned at that time | 10:28 |
| raies | afazekas: thats why I was unable to check deep in body | 10:28 |
| raies | and did only with response code | 10:29 |
| afazekas | in the config file the public_network_id should contain the external network id | 10:29 |
| afazekas | so listing the networks is not required you can just read the config | 10:30 |
| raies | in config file I checked only one line saying public_network_id = {$PUBLIC_NETWORK_ID} | 10:31 |
| raies | ok I should read the config file | 10:32 |
| afazekas | I guess any network with a DHCP enabled subnet, should be related to dhcp agent which is also visible in the agent-list query. | 10:33 |
| raies | self.network_cfg.public_network_id | 10:34 |
| raies | yes you are right | 10:35 |
| raies | afazekas: then here is two amendmet required right 1. isolated credentials should be cleaned up 2. use self.network_cfg.public_network_id isntead fetching from list ? | 10:38 |
| afazekas | yes | 10:40 |
| afazekas | And you should try to create test case where the list_dhcp_agents_hosting_network results are not empty | 10:41 |
| raies | afazekas: I will try that also. But after approval of this as I have to study a bit more on this | 10:42 |
| raies | afazekas: I will be away for 40 min. In mean time please tace care of https://review.openstack.org/#/c/47339/ | 10:44 |
| afazekas | I can change the boto tests to do not cry bacuse if the leaking network, but I guess the ml2 change is related to the vpn issue | 10:45 |
| afazekas | can you try it with and without this change: https://review.openstack.org/#/c/47837/ | 10:46 |
| openstackgerrit | afazekas proposed a change to openstack/tempest: TEMPEST_PY26_NOSE_COMPAT in py26 jobs https://review.openstack.org/49511 | 10:53 |
| *** nosnos has quit IRC | 10:56 | |
| *** matsuhashi has quit IRC | 10:57 | |
| *** weshay has joined #openstack-qa | 10:59 | |
| openstackgerrit | afazekas proposed a change to openstack-dev/devstack: rootwrap tempest https://review.openstack.org/49512 | 11:08 |
| *** ko_ has joined #openstack-qa | 11:09 | |
| raies | afazekas: what is this https://review.openstack.org/#/c/47837/ | 11:15 |
| afazekas | A new neutron plugin | 11:23 |
| afazekas | https://wiki.openstack.org/wiki/Neutron/ML2 | 11:23 |
| *** jhenner has quit IRC | 11:23 | |
| *** adalbas has joined #openstack-qa | 11:25 | |
| *** jhenner has joined #openstack-qa | 11:40 | |
| raies | afazekas: how could I try with or without it ? | 11:44 |
| openstackgerrit | afazekas proposed a change to openstack-dev/devstack: rootwrap tempest https://review.openstack.org/49512 | 11:45 |
| afazekas | raies: checkout an older and newer devstack version | 11:46 |
| afazekas | is the vpn change working on your system with the latest everything ? | 11:46 |
| raies | yes nosetest and tox both were successful | 11:46 |
| afazekas | and how old is your neutron and devstack repo ? | 11:47 |
| raies | drwxr-xr-x 11 openstack openstack 4096 Sep 20 11:54 neutron/ | 11:48 |
| raies | and devstack is of 18 september | 11:49 |
| afazekas | https://github.com/openstack-infra/devstack-gate/commit/71baa816918d8ed29e1891edb3b511e3d9532280 | 11:50 |
| raies | it mean q-vpn service is not enabled currently ? | 11:51 |
| raies | then now what I should do ? | 11:52 |
| afazekas | sdague: ^^ | 11:53 |
| sdague | afazekas: what is the question? | 11:54 |
| afazekas | It might solve the agent update issues https://review.openstack.org/#/c/49439/ | 11:54 |
| afazekas | sdague: now the q-vpn is disabled , can it be re-enabled ? How to proceed with q-vpn tests? | 11:55 |
| sdague | I think that until the other neutron races quiet down, we can't | 11:56 |
| sdague | it was really affecting other project's ability to get to rc | 11:56 |
| sdague | this really needs to land first - https://review.openstack.org/#/c/49504/ to make sure we're actually exercising neutron on neutron patches | 11:57 |
| *** adalbas has quit IRC | 11:58 | |
| *** bashok_ has quit IRC | 11:59 | |
| *** ko_ has quit IRC | 12:02 | |
| *** ken1ohmichi has joined #openstack-qa | 12:03 | |
| *** Madan has quit IRC | 12:04 | |
| afazekas | sdague: the 'tempest' user is not in the 'stack' group and might cause difficulties | 12:08 |
| afazekas | If you explicitly define the commands in the gate script, you need to add every new command there which leads to two changes | 12:09 |
| sdague | afazekas: if so, we need to sort that out. tempest should be able to run without being in that group | 12:09 |
| sdague | afazekas: the point is, we basically shouldn't need to be doing this | 12:10 |
| sdague | jeblair correctly wanted to be very explicit with what we let tempest run as root | 12:10 |
| sdague | and that we need to work away from it over time, and get neutron itself to implement a "debug dump" api | 12:11 |
| afazekas | http://logs.openstack.org/01/42101/2/check/gate-tempest-devstack-vm-postgres-full/dc0c6bb/console.html.gz | 12:11 |
| afazekas | 'tempest.egg-info/requires.txt' was owned by stack user | 12:12 |
| sdague | ok, so we need to fix it | 12:12 |
| afazekas | why is better to create new user, instead of using the Jenkins user ? | 12:12 |
| sdague | that might be an option, however this is where we are headed | 12:14 |
| sdague | but realize, we talked about it in infra the other day, and tempest can't be given blanket root | 12:15 |
| afazekas | it's main negative effect some file might be owned by root instead of 'stack' or 'jenkins' users | 12:17 |
| afazekas | /opt/stack/new/tempest needs to be owned by new user | 12:19 |
| afazekas | 'iniset /opt/stack/new/tempest/etc/tempest.conf DEFAULT lock_path /opt/stack/data/tempest' | 12:19 |
| afazekas | needs to owned by the new user | 12:19 |
| afazekas | I hope it is enough | 12:20 |
| sdague | currious that we set lockpath not in /tmp somewhere | 12:20 |
| sdague | so it's not clear to me why the tempest dir needs to be owned by tempest | 12:21 |
| sdague | it shouldn't be modifying anything except the locks dir | 12:21 |
| afazekas | tox | 12:22 |
| afazekas | tox venv creation modifies the tempst dir | 12:22 |
| sdague | tox is supposed to be running with site-packages | 12:22 |
| afazekas | the log file in the tempest dir | 12:22 |
| sdague | and not create a venv | 12:22 |
| sdague | ok, log file location, I'll accept that one | 12:22 |
| afazekas | even with pep8? | 12:22 |
| sdague | we shouldn't be running pep8 in devstack gate | 12:23 |
| sdague | a nova change shouldn't be running tempest pep8, so if we need to separate that out, we should | 12:23 |
| *** ken1ohmichi has quit IRC | 12:38 | |
| *** sourabh has quit IRC | 12:39 | |
| *** jhenner1 has joined #openstack-qa | 12:41 | |
| *** jhenner has quit IRC | 12:43 | |
| *** salv-orlando has joined #openstack-qa | 12:51 | |
| *** dkranz has quit IRC | 12:54 | |
| *** rfolco has joined #openstack-qa | 12:54 | |
| psedlak|afk | giulivo: did you get any info why the with-neutron-check fails for your stable patch? | 12:56 |
| *** psedlak|afk is now known as psedlak | 12:56 | |
| psedlak | giulivo: i mean that py26 skip exc https://review.openstack.org/#/c/49491/ | 12:56 |
| giulivo | nah I don't think it relates to the patch | 12:57 |
| giulivo | it doesn't get to the point where tempest is started | 12:58 |
| psedlak | giulivo: it seems as https://bugs.launchpad.net/openstack-ci/+bug/1234181 | 12:58 |
| giulivo | yep logs seem to be similar | 12:59 |
| *** weshay has quit IRC | 13:08 | |
| *** mriedem has joined #openstack-qa | 13:08 | |
| *** shilpa_ has quit IRC | 13:15 | |
| *** fbo is now known as fbo_away | 13:16 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack/tempest: Handle test_list_servers_by_changes_since without tenant isolation https://review.openstack.org/49447 | 13:19 |
| *** fbo_away is now known as fbo | 13:25 | |
| *** kong has joined #openstack-qa | 13:28 | |
| *** prekarat has quit IRC | 13:31 | |
| mriedem | since tempest doesn't really "release" anything, does it do a milestone-proposed branch at any point? | 13:35 |
| mriedem | or does it just kind of freeze/manage itself while the core projects are going through release candidates | 13:35 |
| sdague | mriedem: at some point we just cut a stable branch when it seems prudent | 13:36 |
| sdague | after all the projects cut their stable branches | 13:36 |
| mriedem | sdague: that's what i thought. has there ever been a case where a blueprint has landed shortly after master opens up for a new release and someone tries to bring in tempest tests to cover it? | 13:37 |
| sdague | there basically ends up being a day of creating the stable/havana bitrot job accross devstack, tempest, devstack-gate, grenade | 13:37 |
| sdague | mriedem: yep, that's what triggered when we did stable last time | 13:37 |
| sdague | when someone was trying to land a patch that didn't work on stable/grizzly | 13:37 |
| mriedem | ok, someone had their ducks in a row :) | 13:37 |
| *** adalbas has joined #openstack-qa | 13:40 | |
| afazekas | sdague: https://review.openstack.org/#/c/48388/ can we enable the ssh test just only the nova net jobs until the neutron version is not complete ? | 13:43 |
| sdague | afazekas: if someone proposes the patch we can consider it | 13:44 |
| sdague | I just don't want to add any more tests that aren't running | 13:44 |
| *** weshay has joined #openstack-qa | 13:56 | |
| *** dkranz has joined #openstack-qa | 14:02 | |
| *** yeylon_ has quit IRC | 14:05 | |
| *** DennyZhang has joined #openstack-qa | 14:07 | |
| *** sileht has quit IRC | 14:14 | |
| mtreinish | sdague: it looks like the neutron tenant isolation jobs stopped running in tenant isolation: http://logs.openstack.org/03/49403/3/check/check-tempest-devstack-vm-neutron-pg-isolated/fea5663/logs/devstacklog.txt.gz#_2013-10-03_13_25_37 (probably still need to search for allow_tenant_) | 14:14 |
| mtreinish | sdague: I think I found the bug already: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml#L286 | 14:15 |
| mtreinish | will using 1 work instead of True? | 14:15 |
| *** sthaha has quit IRC | 14:16 | |
| *** sileht has joined #openstack-qa | 14:16 | |
| *** jhenner1 has quit IRC | 14:17 | |
| sdague | mtreinish: oh, good point | 14:17 |
| sdague | we aren't running that through trueorfalse | 14:17 |
| mtreinish | sdague: actually I'm not sure that's the issue necessarily. Looking at devstack gate I'm not sure the env is the same when it runs stack.sh | 14:17 |
| *** jhenner has joined #openstack-qa | 14:18 | |
| mtreinish | I'm thinking we just add another if to devstack-gate and add tenat isolation to the localrc | 14:18 |
| sdague | so that's actually this review - https://review.openstack.org/#/c/49504/ | 14:19 |
| sdague | which is even more worrying that neutron was failing those jobs when they weren't tenant issolation jobs yet | 14:19 |
| mtreinish | sdague: well salv-orlando was the one who noticed that things started passing again on those jobs | 14:20 |
| mtreinish | and that tenant isolation wasn't working | 14:20 |
| mtreinish | sdague: yeah that is a bit surprising | 14:20 |
| mtreinish | because there shouldn't have been a difference between the isolation and regular jobs | 14:20 |
| sdague | there isn't | 14:20 |
| sdague | so it's just the 20% fails are causing blocks | 14:21 |
| mtreinish | so it was just the increased number of runs | 14:22 |
| mtreinish | makes sense | 14:22 |
| sdague | right | 14:24 |
| sdague | so we should turn isolation back on | 14:24 |
| sdague | but even just the multi job thing is good | 14:24 |
| mtreinish | yes I agree, we also should probably push https://bugs.launchpad.net/neutron/+bug/1224001 back to critical too | 14:25 |
| sdague | ok agenda for today's meeting is up - https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Weekly_QA_Team_meeting | 14:25 |
| sdague | I also started this for planning - https://etherpad.openstack.org/icehouse-qa-session-planning as it's easier to discuss some things in etherpad | 14:26 |
| mtreinish | what is elastic-recheck top issues? Issues with the elastic-recheck code or things it found? | 14:26 |
| sdague | ? | 14:26 |
| sdague | oh, top issues found | 14:26 |
| mtreinish | ok | 14:27 |
| *** yeylon_ has joined #openstack-qa | 14:27 | |
| *** anteaya has joined #openstack-qa | 14:29 | |
| salv-orlando | mtreinish, sdague: I will move the patch out of WIP soon - had some infrastructure issues on my setup | 14:30 |
| salv-orlando | btw, tenant isolation jobs are a bit different in that they also create network stuff when the credentials are retrieved | 14:31 |
| salv-orlando | apart from that nothing changes. And so far I am unable yet to explain why in the failures related to bug 1224001 it just seems that the l3 agent receives notification but does not process them. | 14:32 |
| sdague | afazekas: we are not adding sudo ALL | 14:36 |
| sdague | end of story, that will not fly | 14:36 |
| mtreinish | salv-orlando: actually I was refering to the fact that tenant_isolation isn't being used on the tenat_islolation jobs | 14:37 |
| mtreinish | right now | 14:37 |
| mtreinish | salv-orlando: yeah my guess is the extra load on neutron from tenant_isolation is what was causing that bug | 14:37 |
| salv-orlando | mtreinish: I am quite aware of that - I have been trying to repro bug 1224001 to no avail for about 10 hours ;) | 14:38 |
| salv-orlando | the issue is that the extra load, is just a router, a subnet, and a network. Peanuts. | 14:39 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: run tempest tests under tempest user https://review.openstack.org/49021 | 14:40 |
| afazekas | sdague: What are the befits of not doing 'ALL' ? | 14:41 |
| sdague | afazekas: because ALL means tempest can call anything as root | 14:42 |
| sdague | which it should not be able to | 14:42 |
| *** shilpa_ has joined #openstack-qa | 14:42 | |
| sdague | we have to be super explicit about times we are violating the premiss that tempest is running as a normal user | 14:43 |
| afazekas | sdague: it is explicit in one location, why to guard it on another location in a different repo? | 14:44 |
| sdague | afazekas: the answer is no, you can go read infra scrollback if you want | 14:45 |
| sdague | but the change will never be allowed into devstack gate if it's ALL | 14:45 |
| afazekas | when was it discussed on infra ? | 14:47 |
| sdague | earlier this week | 14:49 |
| mtreinish | mordred: is it ok if I rip out the tox update from https://review.openstack.org/#/c/45303 as a separate change. I'd like to get that part at least merged while you sort out the build_sphinx stuff | 14:54 |
| mordred | mtreinish: yes please | 14:56 |
| mtreinish | mordred: ok cool | 14:58 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Update to latest tox https://review.openstack.org/49559 | 15:00 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/devstack-gate: Ensure devstack has proper tenant isolation setting https://review.openstack.org/49504 | 15:04 |
| salv-orlando | mtreinish, sdague: https://review.openstack.org/#/c/49560/ and https://review.openstack.org/49504 should ensure the isolation jobs actually enable isolation | 15:05 |
| sdague | salv-orlando: yep, looks good | 15:05 |
| rahmu | Hello everyone. I'm trying to send a request to the root of the swift URL (for instance http://...swift...:8080/healthcheck or http://swift:8080/crossdomain.xml) using the tempest Swift client | 15:13 |
| rahmu | no matter how I try, the account_client will append the AUTH_$(tenant_id) to the url | 15:13 |
| rahmu | any idea how I should proceed? | 15:13 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Split out unit tests as separate tox jobs https://review.openstack.org/49562 | 15:25 |
| *** mlavalle has joined #openstack-qa | 15:30 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Fix zero-fill logic in graphs https://review.openstack.org/49566 | 15:31 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/devstack-gate: Ensure devstack has proper tenant isolation setting https://review.openstack.org/49504 | 15:42 |
| salv-orlando | mtreinish, jeblair: ^^ I hope I got the approach right this time | 15:43 |
| mtreinish | salv-orlando: looks good to me | 15:47 |
| salv-orlando | mtreinish: thanks. | 15:48 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: run tempest tests under tempest user https://review.openstack.org/49021 | 15:49 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Split out unit tests as separate tox jobs https://review.openstack.org/49562 | 15:50 |
| mtreinish | sdague: do we have a bug for this failure: http://logs.openstack.org/59/49559/1/check/check-grenade-devstack-vm/a8e62d7/console.html ? | 15:51 |
| mtreinish | I've seen it a couple of times | 15:51 |
| sdague | mtreinish: nope... and weird | 15:52 |
| sdague | that's a missing package dep | 15:52 |
| openstackgerrit | Lingxian Kong proposed a change to openstack/tempest: add a negative test for flavor_extra_specs https://review.openstack.org/49577 | 15:53 |
| *** yassine has quit IRC | 15:53 | |
| *** marun has quit IRC | 16:01 | |
| *** marun has joined #openstack-qa | 16:02 | |
| *** afazekas has quit IRC | 16:12 | |
| *** SergeyLukjanov has quit IRC | 16:22 | |
| jog0 | dims: 3 out of 13 failures from gate not directly related to your patch. but thats a 23% fail rate and always neutron ... sigh | 16:23 |
| dims | jog0, good news is sign of bogus fd issues since yday - "Second simultaneous read" OR "Cannot understand JSON" | 16:24 |
| jog0 | dims: for https://review.openstack.org/#/c/49403/ if we think it may help and we don't think it makes things worse I think go fir it | 16:24 |
| jog0 | dims: whoa no 'second simultaneous read' after 10-2 20:00 | 16:25 |
| jog0 | not bad! | 16:25 |
| dims | jog0, the prev patch is holding up, so unless we see "Second simultaneous read" OR "Cannot understand JSON" today, we should leave things as is | 16:25 |
| jog0 | awesome!! | 16:25 |
| dims | not sure why we saw stuff from noon to about 4 pm | 16:25 |
| dims | something cached? | 16:25 |
| jog0 | dims: yeah that would be my guess | 16:26 |
| clarkb | check queue possibly | 16:26 |
| clarkb | jog0: https://review.openstack.org/#/c/49466/ | 16:26 |
| jog0 | clarkb: that make sense | 16:26 |
| clarkb | jog0: also https://review.openstack.org/#/c/49458/ | 16:27 |
| *** afazekas has joined #openstack-qa | 16:27 | |
| jog0 | clarkb: for the multi thread you need more rechecks | 16:28 |
| jog0 | to get a better dataset | 16:28 |
| jog0 | also do we have graphite run times? | 16:28 |
| jog0 | we can say for sure that 49466 isn't making things slower at least. and possibly faster | 16:28 |
| clarkb | jog0: I am not sure if we have graphite run times. May be a good thing to add if we don't already have them (zuul has the info and can pass that to graphite) | 16:29 |
| jog0 | clarkb: for the second patch I agree with you, new patch coming in a little bit | 16:29 |
| clarkb | jog0: yeah definitely need more data. but initial results are good :) | 16:29 |
| jog0 | clarkb: yeah | 16:29 |
| *** kong has quit IRC | 16:30 | |
| *** jlibosva has quit IRC | 16:33 | |
| *** afazekas_ has joined #openstack-qa | 16:36 | |
| *** DennyZhang has quit IRC | 16:38 | |
| openstackgerrit | Lingxian Kong proposed a change to openstack/tempest: add some tests for flavor https://review.openstack.org/49588 | 16:38 |
| *** DennyZhang has joined #openstack-qa | 16:38 | |
| afazekas | http://logs.openstack.org/51/48551/2/check/check-tempest-devstack-vm-neutron/c0f6bbb/console.html is it possible the network created by the ServersTestJSON still exists when the ServersTestManualDisk running ? | 16:42 |
| *** weshay has quit IRC | 16:43 | |
| afazekas | Does tempest ever creates a shared net ? | 16:46 |
| *** weshay has joined #openstack-qa | 16:48 | |
| *** jlibosva has joined #openstack-qa | 16:50 | |
| Anju | afazekas: ping | 16:59 |
| afazekas | Anju: pong | 16:59 |
| Anju | afazekas: a flavor name can eceed the limit of 255? | 16:59 |
| Anju | afazekas: a flavor name can not exceed the limit of 255? | 16:59 |
| afazekas | probably it is limited by the field size used in DB creation | 17:00 |
| Anju | afazekas: SOME NEW features are are in v2 api | 17:01 |
| sdague | dkranz afazekas: qa meeting | 17:01 |
| Anju | like tag -upadte and deletion | 17:01 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Fix zero-fill logic in graphs https://review.openstack.org/49566 | 17:04 |
| *** bdperkin has quit IRC | 17:16 | |
| *** SergeyLukjanov has joined #openstack-qa | 17:23 | |
| openstackgerrit | A change was merged to openstack/tempest: add server delete/unpause tests https://review.openstack.org/48796 | 17:23 |
| afazekas | Anju: varchar(255) with mariadb | 17:30 |
| openstackgerrit | afazekas proposed a change to openstack/tempest: TEMPEST_PY26_NOSE_COMPAT in py26 jobs https://review.openstack.org/49511 | 17:36 |
| *** jlibosva1 has joined #openstack-qa | 17:39 | |
| *** jlibosva has quit IRC | 17:43 | |
| *** afazekas_ has quit IRC | 17:49 | |
| *** afazekas has quit IRC | 17:49 | |
| openstackgerrit | Nejc Saje proposed a change to openstack/tempest: Adds client and CRUD tests for Ceilometer alarms https://review.openstack.org/39237 | 17:54 |
| *** jlibosva1 is now known as jlibosva | 17:56 | |
| *** jhenner has quit IRC | 17:56 | |
| *** thomasbiege1 has joined #openstack-qa | 17:59 | |
| psedlak | giulivo, sdague, dkranz: so about that neutron-vm failing for stable/grizzly ... what we can do to get more info why it fails? | 17:59 |
| giulivo | psedlak, so I think we should figure the setup about the neutron vm | 18:00 |
| dkranz | psedlak: Does it fail every time? | 18:00 |
| giulivo | and try to reproduce that and eventually fix it, it's not even when it gets to tempest that it fails | 18:00 |
| psedlak | dkranz: seems so - there aren't many reviews for stable | 18:00 |
| dkranz | psedlak: Well we can kick it again. | 18:01 |
| psedlak | and as pointed out by giulivo - http://logs.openstack.org/91/49491/2/check/check-tempest-devstack-vm-neutron/1fbbb16/console.html | 18:01 |
| dkranz | psedlak: I guess it has to be reproduced locally, or determined that it cannot be. | 18:01 |
| psedlak | it silently 'dies' in the devstack/libs/neutron | 18:01 |
| giulivo | dkranz, yeah and I can't figure anything in the logs | 18:01 |
| dkranz | Have you tried to reproduce by running devstack locally? | 18:02 |
| dkranz | Using the localrc from the failed run? | 18:02 |
| giulivo | dkranz, not yet but that is the plan yes | 18:04 |
| psedlak | dkranz, giulivo: me neither but i'm more concernd how it happens we don't have anything in the logs for such issue | 18:05 |
| psedlak | but that can be probably figured out localy too | 18:05 |
| dkranz | psedlak: Let's hope so. Not sure what else we can do. | 18:06 |
| *** mlavalle has quit IRC | 18:07 | |
| giulivo | I'll work on this later or tomorrow anyway | 18:07 |
| dkranz | psedlak: I filed https://bugs.launchpad.net/devstack/+bug/1232283 last week after talking to infra | 18:07 |
| dkranz | psedlak: Same sort of issue but not reproducible | 18:07 |
| psedlak | dkranz: yeah, really bad stuff :/ | 18:09 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: run tempest tests under tempest user https://review.openstack.org/49021 | 18:09 |
| psedlak | dkranz: btw, your bp about gating clients with stable ... i'd like that implke | 18:10 |
| *** jlibosva has quit IRC | 18:10 | |
| psedlak | *implemented, but not sure about what has to be done ... could you give me more info? | 18:10 |
| dkranz | psedlak: implke??? | 18:10 |
| psedlak | :) | 18:10 |
| dkranz | psedlak: Monty Taylor has already done it | 18:11 |
| dkranz | psedlak: But not merged yet. Here is the patch: | 18:11 |
| psedlak | dkranz: ah, it's already done? so wrong bp state? or it shoudld be reassigned? | 18:11 |
| *** shilpa_ has quit IRC | 18:11 | |
| dkranz | psedlak: https://review.openstack.org/#/c/41931/ | 18:11 |
| *** thomasbiege1 has quit IRC | 18:12 | |
| dkranz | psedlak: I had discussed this with him before creating the blueprint. I am just recording his progress as a tempest blueprint | 18:12 |
| dkranz | psedlak: As soon as that patch merges I will close the bp. I discussed it with him this morning. | 18:13 |
| psedlak | dkranz: cool, thanks for update :) | 18:13 |
| dkranz | psedlak: np | 18:14 |
| *** DennyZhang has quit IRC | 19:01 | |
| *** rfolco has quit IRC | 19:13 | |
| *** mlavalle has joined #openstack-qa | 19:16 | |
| *** marun has quit IRC | 19:21 | |
| dims | jog0, sdague - is this the pile to look at? http://status.openstack.org/elastic-recheck/ | 19:24 |
| *** marun has joined #openstack-qa | 19:32 | |
| *** mkollaro has quit IRC | 19:35 | |
| *** mkollaro has joined #openstack-qa | 19:36 | |
| *** marun has quit IRC | 19:41 | |
| *** marun has joined #openstack-qa | 19:50 | |
| *** mkollaro has quit IRC | 19:59 | |
| *** adalbas has quit IRC | 20:03 | |
| *** mkollaro has joined #openstack-qa | 20:08 | |
| *** SergeyLukjanov has quit IRC | 20:09 | |
| openstackgerrit | Shiv Haris proposed a change to openstack/tempest: Debugging Jenkins failure issues https://review.openstack.org/49628 | 20:23 |
| giulivo | ping cyeoh | 20:28 |
| *** marun has quit IRC | 20:29 | |
| openstackgerrit | A change was merged to openstack/tempest: TEMPEST_PY26_NOSE_COMPAT in py26 jobs https://review.openstack.org/49511 | 20:40 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Cleanup tests https://review.openstack.org/48764 | 20:42 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Make test_required_files.py real unit tests https://review.openstack.org/49265 | 20:42 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Reorganize tests into unit and functional tests https://review.openstack.org/49391 | 20:46 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: run tempest tests under tempest user https://review.openstack.org/49021 | 20:46 |
| sdague | dkranz: https://review.openstack.org/#/c/49447/ can you take a look at that and +2 if you like | 20:48 |
| dkranz | sdague: Just a sec... | 20:49 |
| dkranz | sdague: Hmm. The previous code was quite suspect. | 20:50 |
| dkranz | sdague: Have a minute for the error log issue? | 20:50 |
| sdague | yes | 20:54 |
| sdague | so the idea is that we'd have another tool in tempest that would analyze logs, in general | 20:54 |
| sdague | so that we could just call it at the end of the run | 20:55 |
| sdague | and point it at either a set of files, or urls, it would slurp them in, pull out the traces, compare them to a white list, error if anything new was in there | 20:55 |
| sdague | most importanly, it should live in tempest/tools | 20:55 |
| *** jhenner has joined #openstack-qa | 21:04 | |
| dkranz | sdague: Here's what I was thinking https://etherpad.openstack.org/iqZOBI5HZz | 21:05 |
| dkranz | sdague: Which is pretty much what you said | 21:05 |
| *** rfolco has joined #openstack-qa | 21:05 | |
| *** dims has quit IRC | 21:08 | |
| *** dims has joined #openstack-qa | 21:09 | |
| giulivo | sdague, dkranz this https://review.openstack.org/#/c/48147 I think the guy wasn't passing stop_on_error | 21:10 |
| openstackgerrit | A change was merged to openstack-dev/devstack: ensure tgtd is running in debug mode https://review.openstack.org/48626 | 21:10 |
| giulivo | I wanted to triple check that but that's why I didn't +1 | 21:10 |
| openstackgerrit | A change was merged to openstack-dev/devstack: xenapi: increase default memory to 3G https://review.openstack.org/49401 | 21:10 |
| openstackgerrit | A change was merged to openstack-dev/devstack: Show where files are copied for sudo switch to stack user https://review.openstack.org/49213 | 21:10 |
| *** rfolco has quit IRC | 21:11 | |
| openstackgerrit | A change was merged to openstack-dev/devstack: Allow openrc to be loaded in zsh https://review.openstack.org/48922 | 21:12 |
| openstackgerrit | A change was merged to openstack-dev/devstack: Don't bail when setting up horizon on openSUSE https://review.openstack.org/49219 | 21:12 |
| openstackgerrit | Steve Baker proposed a change to openstack/tempest: Allow _status_timeout to be used for non-nova polling https://review.openstack.org/48869 | 21:12 |
| openstackgerrit | Steve Baker proposed a change to openstack/tempest: Orchestration autoscaling improve status polling https://review.openstack.org/48870 | 21:12 |
| openstackgerrit | A change was merged to openstack-dev/devstack: Uniquify unsupported distro error message https://review.openstack.org/49218 | 21:12 |
| openstackgerrit | A change was merged to openstack-dev/devstack: Ensure SSL CA certificates are installed for curl https://review.openstack.org/49217 | 21:12 |
| dkranz | sdague: See my comment at the end of the pad | 21:14 |
| *** marun has joined #openstack-qa | 21:14 | |
| sdague | yep, responding | 21:15 |
| dkranz | sdague: Works for me. | 21:15 |
| openstackgerrit | A change was merged to openstack/tempest: Handle test_list_servers_by_changes_since without tenant isolation https://review.openstack.org/49447 | 21:22 |
| sdague | giulivo: oops, if I screwed that one up a fix would be appreciated :) | 21:24 |
| giulivo | yeah I knew you were going to say that | 21:25 |
| sdague | giulivo: if you think something is weird in the future, definitely throw a -1 on it | 21:25 |
| sdague | as I won't +2 something with a -1 without a really good reason | 21:25 |
| giulivo | yeah no problems I wasn't sure about the findings and wanted to double check that | 21:25 |
| sdague | yeh, -1s are easy to reverse later | 21:26 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Ensure devstack has proper tenant isolation setting https://review.openstack.org/49504 | 21:37 |
| openstackgerrit | A change was merged to openstack/tempest: add server suspend/resume negative tests https://review.openstack.org/48814 | 21:41 |
| jog0 | dims: the pile to look at? | 21:41 |
| openstackgerrit | A change was merged to openstack/tempest: _error_in_logs function of driver.py shall check all nodes https://review.openstack.org/48147 | 21:42 |
| *** mriedem has quit IRC | 21:45 | |
| *** fbo is now known as fbo_away | 21:49 | |
| *** dkranz has quit IRC | 21:56 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add even more debug logs. https://review.openstack.org/49458 | 22:05 |
| dims | jog0, top defects that needs attention | 22:16 |
| jog0 | dims: not exactly :/ | 22:18 |
| jog0 | that only shows the known issues that we have queries to classify | 22:18 |
| jog0 | so its a subset of issues that need attention | 22:18 |
| jog0 | dims: we still need to monitor http://status.openstack.org/rechecks/ (and mark duplicates so they dedup on that page and find classificatiosn for them) | 22:19 |
| dims | jog0, cool thx | 22:21 |
| jog0 | mtreinish: when I run 'tox -epy27 in e-r it fails | 22:21 |
| mtreinish | jog0: with 49391 or just on HEAD? | 22:22 |
| jog0 | HEAD | 22:23 |
| jog0 | just an FYI | 22:23 |
| mtreinish | hmm, I tested the changes I made to required files do you have a fail log I can look at? | 22:24 |
| jog0 | mtreinish: pastebinit ftw http://paste.openstack.org/show/47937/ | 22:26 |
| jog0 | from .testrepository | 22:26 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add even more debug logs. https://review.openstack.org/49458 | 22:27 |
| mtreinish | wow there are a bunch of failures | 22:28 |
| mtreinish | the first one in test_queries is pretty obvious: 12327480 isn't a valid bug | 22:28 |
| mtreinish | the second one is that I forgot to update the gerrit tests after adding support for multiple failures | 22:30 |
| mtreinish | jog0: I've never seen the fail for elastic_recheck.tests.test_required_files.TestRequiredFiles.test_files_at_url_pass before | 22:30 |
| mtreinish | looks like something with mox | 22:30 |
| jog0 | yeah ... wonder if you can reproduce it | 22:31 |
| mtreinish | jog0: I'm about to head home for today. I'll take a give it a shot tomorrow morning | 22:32 |
| *** openstackrecheck has quit IRC | 22:33 | |
| *** openstackrecheck has joined #openstack-qa | 22:33 | |
| clarkb | jog0: I just restarted elastic-recheck to pick up the new log stuff | 22:33 |
| jog0 | mtreinish: cool there is no rush on this. it was jsut a heads up | 22:33 |
| jog0 | clarkb: thanks | 22:33 |
| jog0 | clarkb: running a local copy to compare to | 22:33 |
| jog0 | nothing failing yet | 22:33 |
| jog0 | tempest wise | 22:33 |
| *** weshay has quit IRC | 22:36 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/49626 failed tempest with an unrecognized error | 22:38 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/49641 failed tempest with an unrecognized error | 22:42 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/49627 failed tempest with an unrecognized error | 22:42 |
| jog0 | clarkb: looks like its working ^ | 22:57 |
| clarkb | cool | 22:57 |
| clarkb | I suppose we wait and see if it dies then go looking in logs | 22:57 |
| *** rfolco has joined #openstack-qa | 23:02 | |
| *** masayukig has joined #openstack-qa | 23:02 | |
| *** jhenner has quit IRC | 23:04 | |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/49644 failed tempest because of: https://bugs.launchpad.net/bugs/1224001 | 23:05 |
| *** dims has quit IRC | 23:06 | |
| *** mlavalle has quit IRC | 23:09 | |
| *** sthaha has joined #openstack-qa | 23:15 | |
| *** sthaha has joined #openstack-qa | 23:15 | |
| *** dims has joined #openstack-qa | 23:23 | |
| *** mkollaro has quit IRC | 23:29 | |
| jog0 | clarkb: looks like it died | 23:46 |
| jog0 | I just picked up https://review.openstack.org/#/c/45386/5 due to https://bugs.launchpad.net/bugs/1224001 | 23:47 |
| clarkb | jog0: ok grabbing logs. how far back do you want? | 23:49 |
| jog0 | since 49644? | 23:50 |
| jog0 | 23:08? | 23:50 |
| jog0 | TBH not sure what to look for | 23:50 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!