Thursday, 2019-03-21

*** fnaval has quit IRC00:04
*** lbragstad has quit IRC00:14
*** cshen has joined #openstack-qa00:27
*** cshen has quit IRC00:31
*** fnaval has joined #openstack-qa00:34
*** fnaval has quit IRC00:45
*** mriedem has quit IRC01:26
*** lbragstad has joined #openstack-qa01:35
*** mattw4 has quit IRC01:42
*** cshen has joined #openstack-qa02:27
*** cshen has quit IRC02:32
*** markvoelker has joined #openstack-qa02:43
*** yamamoto has joined #openstack-qa03:01
*** rh-jelabarre has quit IRC03:21
*** yamamoto has quit IRC04:03
*** whoami-rajat has joined #openstack-qa04:10
openstackgerritMerged openstack/tempest master: Remove unused and duplicate nodeset defined in Tempest  https://review.openstack.org/64173504:41
*** sridharg has joined #openstack-qa05:39
*** gcheresh has joined #openstack-qa06:05
*** lbragstad has quit IRC06:06
*** e0ne has joined #openstack-qa06:06
*** e0ne has quit IRC06:07
*** pcaruana has joined #openstack-qa06:11
*** brinzhang has joined #openstack-qa06:28
*** Luzi has joined #openstack-qa06:44
*** slaweq has quit IRC06:45
*** sridharg has quit IRC06:50
*** sridharg has joined #openstack-qa06:53
*** ccamposr has joined #openstack-qa07:08
*** sridharg has quit IRC07:21
*** sridharg has joined #openstack-qa07:23
*** rcernin has quit IRC07:24
*** cshen has joined #openstack-qa07:26
*** slaweq has joined #openstack-qa07:41
*** e0ne has joined #openstack-qa07:45
*** yamamoto has joined #openstack-qa08:01
*** yamamoto has quit IRC08:05
*** openstackgerrit has quit IRC08:17
*** kopecmartin|off is now known as kopecmartin08:18
*** kopecmartin is now known as randarah08:19
*** randarah is now known as kopecmartin08:19
*** gfidente has joined #openstack-qa08:24
*** dtantsur|afk is now known as dtantsur08:24
*** tosky has joined #openstack-qa08:27
*** gkadam has joined #openstack-qa08:33
*** gkadam has quit IRC08:33
*** slaweq_ has joined #openstack-qa08:36
*** slaweq has quit IRC08:38
*** jpena|off is now known as jpena08:45
*** slaweq_ has quit IRC08:47
*** e0ne has quit IRC08:48
*** e0ne has joined #openstack-qa08:51
*** whoami-rajat has quit IRC09:10
*** whoami-rajat has joined #openstack-qa09:18
*** sridharg has quit IRC09:19
*** sridharg has joined #openstack-qa09:27
*** pcaruana has quit IRC09:45
*** pcaruana has joined #openstack-qa09:46
*** cshen has quit IRC09:57
*** slaweq_ has joined #openstack-qa10:03
*** zbr has quit IRC10:08
*** zbr has joined #openstack-qa10:10
*** cshen has joined #openstack-qa10:11
*** e0ne has quit IRC10:14
*** zbr has quit IRC10:16
*** e0ne has joined #openstack-qa10:17
*** slaweq_ is now known as slaweq10:18
*** openstackgerrit has joined #openstack-qa10:54
openstackgerritLuigi Toscano proposed openstack-dev/grenade master: Add GRENADE_DEVSTACK_EXTERNAL to skip part of the setup  https://review.openstack.org/63726510:54
openstackgerritLuigi Toscano proposed openstack-dev/grenade master: WIP Setup grenade-base, grenade, grenade-{py3,postgresql}  https://review.openstack.org/54893610:54
openstackgerritLuigi Toscano proposed openstack-dev/grenade master: DNM - grenade-heat-test job  https://review.openstack.org/63977410:54
*** e0ne has quit IRC10:58
*** e0ne has joined #openstack-qa10:59
openstackgerritPavlo Shchelokovskyy proposed openstack-dev/devstack stable/pike: Remove apache tls-proxy sites when stopping  https://review.openstack.org/64512511:01
openstackgerritPavlo Shchelokovskyy proposed openstack-dev/devstack stable/ocata: Remove apache tls-proxy sites when stopping  https://review.openstack.org/64512711:09
toskyfrickler: any additional concerns about https://review.openstack.org/#/c/639374/ ?11:13
fricklertosky: lgtm, let's wait on the results for your grenade patches, then I'm fine with merging it11:27
*** whoami-rajat has quit IRC11:30
*** whoami-rajat has joined #openstack-qa11:34
*** zbr has joined #openstack-qa11:38
*** dave-mccowan has joined #openstack-qa11:50
*** rh-jelabarre has joined #openstack-qa11:51
*** raildo has joined #openstack-qa11:53
*** dave-mccowan has quit IRC11:53
*** cshen has quit IRC11:53
*** jpena is now known as jpena|lunch12:01
*** rf0lc0 is now known as rfolco12:16
*** cshen has joined #openstack-qa12:17
*** markvoelker has quit IRC12:22
*** rh-jelabarre has quit IRC12:34
*** mriedem has joined #openstack-qa12:38
openstackgerritLuigi Toscano proposed openstack-dev/grenade master: WIP Setup grenade-base, grenade, grenade-{py3,postgresql}  https://review.openstack.org/54893612:40
openstackgerritLuigi Toscano proposed openstack-dev/grenade master: DNM - grenade-heat-test job  https://review.openstack.org/63977412:40
*** yamamoto has joined #openstack-qa12:42
*** yamamoto has quit IRC12:43
*** e0ne has quit IRC12:51
*** yamamoto has joined #openstack-qa12:51
*** dtantsur is now known as dtantsur|brb12:52
*** lbragstad has joined #openstack-qa12:54
*** yamamoto has quit IRC13:05
*** yamamoto has joined #openstack-qa13:05
*** yamamoto has quit IRC13:06
*** irclogbot_0 has quit IRC13:07
*** irclogbot_2 has joined #openstack-qa13:09
*** brinzhang has quit IRC13:11
openstackgerritMerged openstack/devstack-plugin-ceph master: Disable multiattach feature for stable branches before stein  https://review.openstack.org/64340213:19
*** jpena|lunch is now known as jpena13:20
*** awaugama has joined #openstack-qa13:22
*** altlogbot_3 has quit IRC13:23
*** e0ne has joined #openstack-qa13:23
*** altlogbot_2 has joined #openstack-qa13:24
*** yamamoto has joined #openstack-qa13:30
*** brinzhang has joined #openstack-qa13:32
*** brinzhang has quit IRC13:33
*** e0ne has quit IRC13:37
*** altlogbot_2 has quit IRC13:39
*** whoami-rajat has quit IRC13:40
*** altlogbot_3 has joined #openstack-qa13:41
*** irclogbot_2 has quit IRC13:45
*** e0ne has joined #openstack-qa13:45
*** irclogbot_1 has joined #openstack-qa13:47
*** yamamoto has quit IRC13:48
openstackgerritMerged openstack-infra/elastic-recheck master: Remove old queries for stale and/or fixed bugs  https://review.openstack.org/64504013:49
*** yamamoto has joined #openstack-qa13:50
*** rh-jelabarre has joined #openstack-qa13:50
*** mjturek has joined #openstack-qa14:02
*** whoami-rajat has joined #openstack-qa14:23
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck master: Add query for neutron functional ansible bug 1820993  https://review.openstack.org/64519314:26
openstackbug 1820993 in neutron "Functional and fullstack tests job definition is broken" [Critical,In progress] https://launchpad.net/bugs/1820993 - Assigned to Slawek Kaplonski (slaweq)14:26
*** mjturek has quit IRC14:33
*** gfidente has quit IRC14:33
*** altlogbot_3 has quit IRC14:35
*** yamamoto has quit IRC14:36
*** altlogbot_1 has joined #openstack-qa14:37
*** dtantsur|brb is now known as dtantsur14:37
*** yamamoto has joined #openstack-qa14:37
*** irclogbot_1 has quit IRC14:38
*** gfidente has joined #openstack-qa14:38
*** irclogbot_2 has joined #openstack-qa14:39
*** mjturek has joined #openstack-qa14:39
*** yamamoto has quit IRC14:42
*** gcheresh has quit IRC14:50
openstackgerritMerged openstack-infra/elastic-recheck master: Add query for neutron functional ansible bug 1820993  https://review.openstack.org/64519314:54
openstackbug 1820993 in neutron "Functional and fullstack tests job definition is broken" [Critical,In progress] https://launchpad.net/bugs/1820993 - Assigned to Slawek Kaplonski (slaweq)14:54
*** raildo has quit IRC14:56
*** raildo has joined #openstack-qa14:58
*** fnaval has joined #openstack-qa15:21
*** Luzi has quit IRC15:25
*** mattw4 has joined #openstack-qa15:36
*** mattw4 has quit IRC15:44
*** mattw4 has joined #openstack-qa15:48
*** cshen has quit IRC15:53
gmannoffice hour time16:01
gmann#startmeeting qa16:02
openstackMeeting started Thu Mar 21 16:02:01 2019 UTC and is due to finish in 60 minutes.  The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot.16:02
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:02
*** openstack changes topic to " (Meeting topic: qa)"16:02
openstackThe meeting name has been set to 'qa'16:02
gmannwho all here today16:02
gmannI will be here and checking few priority bugs fix which we should land soon.16:03
gmannthis is for tempest plugin installation issues16:06
gmann#link https://review.openstack.org/#/c/643333/16:06
gmannand its backport to stable - https://review.openstack.org/#/q/Ia1ae6869a8fede2af5cd7c875e0946b6a75eb51816:06
gmannother fix is for installing the pckg on py3 fallback things - #link https://review.openstack.org/#/c/644638/16:07
gmannfrickler: ianw if you can have your eyes on these. need another +216:07
*** e0ne has quit IRC16:10
gmanni do not see anything critical from tempest side,16:11
gmannwe merged the zuul job failure  - #link https://review.openstack.org/#/c/644699/16:12
gmannalso opened a bug for that - https://bugs.launchpad.net/tempest/+bug/182107116:13
openstackLaunchpad bug 1821071 in tempest "All devstack-tempest jobs are failing due to 'gabbi_tempest_path' is undefined" [Undecided,New]16:13
gmannother part is about PTG planning.16:14
gmann#link https://etherpad.openstack.org/p/qa-train-ptg16:14
gmannyou can write your PTG topic in ^^ etherpad16:14
*** mjturek has quit IRC16:16
*** mattw4 has quit IRC16:21
*** mjturek has joined #openstack-qa16:24
toskyuh, sorry, I need to update my calendar16:25
toskythe gabbi_tempest_path issue also affected my under-testing grenade jobs16:27
toskytalking about them, I'm trying to fix the logging (for real) but I'm puzzled because in the last iteration some logs are not compressed, and they have been compressed in the past runs, and I'm not sure how it did(n't) happen16:28
toskybetween patch 75 and 77 of https://review.openstack.org/#/c/548936/, files like grenade.sh_log.txt are not compressed anymore16:29
gmanntosky: on gabbi_tempest_path, do you have bug reported on that ?16:30
gmanni am thinking to move that var into gabbi-tempest plugin itself.16:30
toskygmann: no, I didn't - it's about a work-in-progress job and I fixed it16:31
toskyI think that it was put outside the plugin for a specific reason, but I forgot the content of the discussion16:31
gmanni think reason was not to have playbooks for one var only16:31
gmannthat's what i remember during that path review16:32
gmannplaybooks on gabbit-tempest16:32
* afazekas o/16:34
gmanntosky: https://review.openstack.org/#/c/607507/2/playbooks/devstack-tempest.yaml@1116:35
gmannafazekas: hi16:35
*** cshen has joined #openstack-qa16:37
toskygmann: so it's between duplicated playbooks and duplicated variable which can break16:39
gmanni will propose patch or check with chris if any issue now to move that to gabbi-tempest as it can create and fail all jobs16:39
gmanntosky: means? we will remove it from devstack-tempest once moved to gabbit-tempest playbook16:40
*** cshen has quit IRC16:42
toskygmann: I'm fine with that as well, I was just pointing out that the reason for not having a gabbi-tempest playbook was exactly to not rewrite the playbook16:46
*** mvkr has quit IRC16:48
gmanntosky: ok, let me check how much work and any difficulties in duplicate things16:50
gmanntosky: re: on log compressed issue . no idea anything on zuul side happen16:53
toskyeheh, it's done by role, but I can't see the difference between the two runs16:53
tosky(!)16:53
gmannclarkb: fungi any idea on logs (grenade.sh_log.txt) are not compressed anymore in this http://logs.openstack.org/36/548936/76/check/grenade/9643c68/controller/logs/16:54
gmannPS75, ir was compressed16:54
fungiare you sure it was compressed at upload and not later?16:54
*** rpittau is now known as rpittau|afk16:55
gmanntosky: it is only not compressed in PS76 and working fine in PS77 right ?16:55
fungiwe have a cron job which goes through the logserver compressing things which look safe to compress but jobs failed to do themselves16:55
toskygmann: the other way16:55
toskyfungi: oh16:55
toskyoh16:55
toskyso I'm not going crazy as I though16:55
gmannah checked it,16:55
toskythought*16:55
toskybecause I remember that I've seen the logs not compressed, then I rechecked and they were compressed16:56
toskyand I was scratching my head16:56
fungiso just because something is compressed now from an earlier run doesn't mean it was compressed when it was originally uploaded. our compression cron may have compressed it hours or even days later16:56
toskyI see, thanks16:56
toskyso it never worked \o/16:56
gmannohk16:57
gmanngrenade_localrc are compressed16:59
gmanntime up for office hour, i will close the office hour and we can continue the discussion.17:00
gmann#endmeeting17:00
*** openstack changes topic to "Discussion of OpenStack Quality Assurance Projects (tempest, devstack, grenade, and friends) | docs https://wiki.openstack.org/wiki/QA | specs https://review.openstack.org/#/q/project:openstack/qa-specs | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-qa/"17:00
openstackMeeting ended Thu Mar 21 17:00:48 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/qa/2019/qa.2019-03-21-16.02.html17:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/qa/2019/qa.2019-03-21-16.02.txt17:00
toskyyes; also, the files which are not compressed are originally symlinks iirc, but I have seen other originally-symlinks being compressed17:00
openstackLog:            http://eavesdrop.openstack.org/meetings/qa/2019/qa.2019-03-21-16.02.log.html17:00
gmanntosky: humm but did you check they never compressed? i mean we might have always checked them after upload only17:01
toskymaybe, but maybe it's just a path issue17:03
toskysee here: http://logs.openstack.org/36/548936/77/check/grenade/3a4e817/job-output.txt.gz#_2019-03-21_13_35_19_07791717:04
toskyI see that same "No such file or directory" also for some tempest jobs17:05
toskyuhm17:05
toskyI suspect a bug in the stage-output role, which seems to not compress all the files which are not in {{ stage_dir }}17:08
openstackgerritLuigi Toscano proposed openstack-dev/grenade master: WIP Setup grenade-base, grenade, grenade-{py3,postgresql}  https://review.openstack.org/54893617:13
openstackgerritLuigi Toscano proposed openstack-dev/grenade master: DNM - grenade-heat-test job  https://review.openstack.org/63977417:13
gmannbut on tempest job it does compress and we use same role there..17:15
*** mriedem is now known as mriedem_grooming17:16
*** mvkr has joined #openstack-qa17:16
*** iurygregory has quit IRC17:25
*** dtantsur is now known as dtantsur|afk17:30
*** mattw4 has joined #openstack-qa17:32
*** gcheresh has joined #openstack-qa17:32
*** ccamposr has quit IRC17:36
openstackgerritVladislav Kuzmin proposed openstack-dev/devstack master: Enable certificate check for glance_store+swift  https://review.openstack.org/64527917:40
openstackgerritVladislav Kuzmin proposed openstack-dev/devstack master: Enable certificate check for glance_store+swift  https://review.openstack.org/64455817:42
*** gmann is now known as gmann_afk17:43
*** mvkr has quit IRC17:46
*** cshen has joined #openstack-qa17:48
*** cshen has quit IRC17:52
*** gcheresh has quit IRC17:58
*** e0ne has joined #openstack-qa18:05
*** e0ne has quit IRC18:12
openstackgerritBrian Haley proposed openstack-dev/devstack master: Use L3RouterPlugin alias  https://review.openstack.org/64528918:13
*** gfidente has quit IRC18:14
*** jpena is now known as jpena|off18:22
*** whoami-rajat has quit IRC18:23
*** yamamoto has joined #openstack-qa18:27
*** yamamoto has quit IRC18:31
*** cshen has joined #openstack-qa18:32
*** cshen has quit IRC18:37
*** gmann_afk is now known as gmann18:40
*** kopecmartin is now known as kopecmartin|off18:43
*** mriedem_grooming is now known as mriedem18:53
*** sshnaidm|rover is now known as sshnaidm|off19:03
toskygmann: no, on tempest job the compression does not happen19:05
toskygmann: this is a fresh run of a tempest job: http://logs.openstack.org/55/590055/25/check/sahara-tests-tempest/3499a01/controller/logs/19:05
toskyso the magic compresson hasn't been executed yet19:05
toskyfiles like ._localrc_auto.txt, ._stackend.txt, local_conf.txt and tempest_conf.txt are not compressed19:06
gmanntosky: humm, devstacklog.txt is compressed19:12
*** mvkr has joined #openstack-qa19:13
toskygmann: the uncompressed files are basically the ones for which stage-ouput reported an error here: http://logs.openstack.org/55/590055/25/check/sahara-tests-tempest/3499a01/job-output.txt.gz#_2019-03-21_17_34_27_69309719:20
toskyI suspect a mismatch between the path of the files before the copy (see all_sources: https://opendev.org/openstack-infra/zuul-jobs/src/branch/master/roles/stage-output/tasks/main.yaml#L44 )19:23
toskyand the path of the files after the copy to the staging folder (https://opendev.org/openstack-infra/zuul-jobs/src/branch/master/roles/stage-output/tasks/main.yaml#L59)19:23
toskythe compression step works inside stage_dir, but it uses the paths from all_sources19:24
*** whoami-rajat has joined #openstack-qa19:26
*** e0ne has joined #openstack-qa19:30
*** sridharg has quit IRC19:46
*** e0ne has quit IRC19:49
*** e0ne has joined #openstack-qa19:52
gmannyeah it does  - with_items: "{{ all_sources }}"20:01
*** e0ne has quit IRC20:04
* gmann it does not log what all copied or not - http://logs.openstack.org/55/590055/25/check/sahara-tests-tempest/3499a01/job-output.txt.gz#_2019-03-21_17_33_44_26856020:06
*** awaugama has quit IRC20:20
*** e0ne has joined #openstack-qa20:28
toskyI created a debug job but the gates are a bit busy20:29
*** cshen has joined #openstack-qa20:33
*** cshen has quit IRC20:38
*** rh-jelabarre has quit IRC21:04
*** mjturek has quit IRC21:06
toskyfrickler: the grenade job based on the tempest_plugins patch passed :)21:09
*** raildo has quit IRC21:12
*** e0ne has quit IRC21:17
*** pcaruana has quit IRC21:33
openstackgerritMerged openstack/tempest master: Checking config file actually exist or not in tempest run  https://review.openstack.org/62516221:38
openstackgerritMerged openstack-dev/devstack master: Fix installing tempest plugins  https://review.openstack.org/64333321:38
*** bdperkin has quit IRC21:44
*** whoami-rajat has quit IRC21:45
toskygmann: the loop *is* correct, as it uses item.dest, but the problem is that some of the items have been given the .txt extension (https://opendev.org/openstack-infra/zuul-jobs/src/branch/master/roles/stage-output/tasks/main.yaml#L83)22:01
toskyso the list is not accurate anymore22:01
gmannohhk22:03
gmanncan we add it with .log as those are most of our log files?22:04
toskythere are few different ways to fix this22:04
tosky- when archiving, check if, instead of filename.ext, filename_ext.txt exists, and compress it22:04
tosky- put the rename and archiving logic in the same loop which goes through all files (less ansibly, but we are using it for shell commands anyway)22:05
tosky- implement some bigger restructuring22:05
*** irclogbot_2 has quit IRC22:05
gmannwe can maintain standard ext for log files but we cannot control that on derived jobs22:06
toskythis logic should take care of the .txt files, but something is failing anyway https://opendev.org/openstack-infra/zuul-jobs/src/branch/master/roles/stage-output/tasks/main.yaml#L3522:07
gmann2nd one looks more appropriate22:07
toskyI will think about it tomorrow :)22:07
toskybut at least I'm happy that I found the issue and that I was not going crazy with those compressed/uncompressed files22:08
gmann+1.22:08
*** rcernin has joined #openstack-qa22:19
*** fnaval has quit IRC22:23
*** cshen has joined #openstack-qa22:33
*** cshen has quit IRC22:37
*** mriedem is now known as mriedem_afk22:50
*** tosky has quit IRC22:59
*** mattw4 has quit IRC23:10
*** fnaval has joined #openstack-qa23:10
*** bdperkin has joined #openstack-qa23:16
*** lbragstad has quit IRC23:46

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!