| *** tosky has quit IRC | 00:12 | |
| *** brault has joined #oooq | 00:48 | |
| *** brault has quit IRC | 00:52 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 01:02 |
|---|---|---|
| *** rlandy has joined #oooq | 01:35 | |
| rlandy | weshay: still around? any idea why this combination does not trigger the fs053 test? https://review.openstack.org/#/c/581488/ https://review.rdoproject.org/zuul3/status.html | 01:37 |
| rlandy | what did I miss? | 01:37 |
| weshay | rlandy, /me looks | 01:37 |
| rlandy | https://review.rdoproject.org/r/#/c/14772/ | 01:37 |
| rlandy | should define the job triggers | 01:37 |
| rlandy | should trigger on changes to tq/tqe/tripleo-ci as well as browbeat | 01:38 |
| weshay | rlandy, oh I think we learned that the zuul config in rdo sf sucks in that it does not trigger unitil merged | 01:40 |
| weshay | rlandy, however I wonder if that is just for triggers on repos outside of sf | 01:41 |
| rlandy | hmmm | 01:41 |
| rlandy | what's a catch 22 | 01:41 |
| weshay | rlandy, so I wonder if we set it up to say.. trigger on ci-config | 01:42 |
| weshay | which does exist there.. | 01:42 |
| weshay | test it.. w/ that | 01:42 |
| rlandy | I'll have to test is with a reproducer | 01:42 |
| weshay | rlandy, /me looks at this | 01:42 |
| rlandy | weshay: the alternative is the to merge it triggering just one file in browbeat so we can see what it does | 01:43 |
| weshay | rlandy, aye.. do you have core? | 01:43 |
| rlandy | weshay: yes - do you? | 01:44 |
| weshay | rlandy, no | 01:44 |
| rlandy | hmmm | 01:44 |
| weshay | rlandy, we can fix that.. so I'll just resubmit under a diff commit id | 01:45 |
| weshay | rlandy, sec | 01:45 |
| weshay | rlandy, I think we should try to trigger w/ ci-config though too | 01:45 |
| weshay | fwiw | 01:45 |
| rlandy | I don;t know what 'trigger w/ ci-config' is | 01:45 |
| rlandy | the ci-config doesn't trigger that test | 01:46 |
| rlandy | ok - let me put in another commit | 01:46 |
| rlandy | I don't want to merge this tonight w/o anyone's approval | 01:48 |
| weshay | rlandy, I think... https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/projects.yaml#L4833 | 01:51 |
| rlandy | oh - I see | 01:52 |
| weshay | rlandy, and then we add a test patch to ci-config w/ a depends on ur zuul change | 01:52 |
| weshay | rlandy, not sure | 01:52 |
| weshay | but I think it's worth a shot | 01:52 |
| rlandy | weshay: actually I think putting in a temp patch to trigger on only one file is a good idea | 01:53 |
| rlandy | but I 'll ask for merge tomorrow | 01:53 |
| weshay | ok.. that is fine too | 01:53 |
| rlandy | so I am not the one pushing through my own code | 01:53 |
| rlandy | then we can merge the job definition and one file trigger and work from there | 01:54 |
| rlandy | thanks for your help | 01:54 |
| weshay | k | 01:55 |
| rlandy | https://review.rdoproject.org/r/#/c/14880/ | 01:59 |
| *** rlandy has quit IRC | 02:02 | |
| *** jaganathan has joined #oooq | 02:37 | |
| *** gkadam has joined #oooq | 02:55 | |
| *** gkadam has quit IRC | 02:59 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 03:02 |
| *** skramaja has joined #oooq | 03:17 | |
| *** skramaja_ has joined #oooq | 03:21 | |
| *** skramaja has quit IRC | 03:21 | |
| *** udesale has joined #oooq | 03:24 | |
| *** ykarel has joined #oooq | 04:24 | |
| *** trown has quit IRC | 04:24 | |
| *** trown|brb has joined #oooq | 04:34 | |
| *** holser_ has joined #oooq | 04:54 | |
| *** udesale has quit IRC | 04:55 | |
| *** udesale has joined #oooq | 04:55 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode- (1 more message) | 05:02 |
| *** lucasagomes_ has joined #oooq | 05:12 | |
| *** links has joined #oooq | 05:15 | |
| *** hamzy has quit IRC | 05:16 | |
| *** lucasagomes has quit IRC | 05:16 | |
| *** Tengu has quit IRC | 05:16 | |
| *** chkumar|ruck has quit IRC | 05:28 | |
| *** chandankumar has joined #oooq | 05:29 | |
| *** chandankumar is now known as chkumar|ruck | 05:29 | |
| *** hamzy has joined #oooq | 05:35 | |
| *** Tengu has joined #oooq | 05:35 | |
| *** ratailor has joined #oooq | 05:35 | |
| *** quiquell|off is now known as quiquell | 05:39 | |
| quiquell | sshnaidm|rover, chkumar|ruck: new rrcockpit url http://38.145.34.131/d/7q6lisOik/cockpit?orgId=1 | 05:42 |
| quiquell | Tell me if you are missing somet hing | 05:43 |
| *** sanjayu_ has joined #oooq | 06:05 | |
| chkumar|ruck | quiquell: I wants me to login | 06:06 |
| ykarel | chkumar|ruck, join #rhos-ops, all ovb jobs affected by rdo cloud issue | 06:06 |
| *** jfrancoa has joined #oooq | 06:07 | |
| ykarel | chkumar|ruck, i have asked csastri to look | 06:07 |
| chkumar|ruck | ykarel: thanks :-) | 06:12 |
| chkumar|ruck | %gatestatus | 06:15 |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009 (1 more message) | 06:15 |
| chkumar|ruck | quiquell: which credentials to use? | 06:17 |
| *** holser_ has quit IRC | 06:24 | |
| quiquell | chkumar|ruck: Forgot to activate anonymous access, wait | 06:30 |
| *** kopecmartin has joined #oooq | 06:34 | |
| quiquell | chkumar|ruck: need another spin to activate it, give me a few, ok ? | 06:46 |
| chkumar|ruck | quiquell: yup totally ok | 06:47 |
| *** bogdando has joined #oooq | 06:49 | |
| *** florianf has joined #oooq | 06:50 | |
| *** holser_ has joined #oooq | 06:56 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode- (1 more message) | 07:02 |
| *** amoralej|off has joined #oooq | 07:06 | |
| *** amoralej|off is now known as amoralej | 07:06 | |
| *** gkadam has joined #oooq | 07:08 | |
| *** ccamacho has joined #oooq | 07:08 | |
| *** brault has joined #oooq | 07:17 | |
| *** tesseract has joined #oooq | 07:18 | |
| *** brault has quit IRC | 07:19 | |
| *** brault has joined #oooq | 07:19 | |
| *** ykarel is now known as ykarel|lunch | 07:19 | |
| quiquell | chkumar|ruck: Ok now anonymous works | 07:32 |
| *** zoli|gone is now known as zoli | 07:44 | |
| *** jtomasek has joined #oooq | 07:50 | |
| *** jtomasek has quit IRC | 07:50 | |
| *** jtomasek has joined #oooq | 07:50 | |
| *** skramaja_ is now known as skramaja | 08:05 | |
| *** tosky has joined #oooq | 08:06 | |
| *** sanjayu_ is now known as saneax | 08:14 | |
| *** ykarel|lunch is now known as ykarel | 08:15 | |
| *** lucasagomes_ is now known as lucasagomes | 08:15 | |
| *** holser__ has joined #oooq | 08:20 | |
| *** holser_ has quit IRC | 08:21 | |
| *** amoralej_ has joined #oooq | 08:22 | |
| *** amoralej has quit IRC | 08:22 | |
| chkumar|ruck | quiquell: thanks :-) | 08:27 |
| quiquell | marios, sshnaidm|rover: fs as strings needed rebase https://review.openstack.org/#/c/583022 | 08:36 |
| marios | quiquell: ack | 08:38 |
| jfrancoa | anybody else has lost connectivity with the internal irc channels? quiquell, marios ? | 08:58 |
| chkumar|ruck | RDO cloud outage has just started | 08:59 |
| marios | jfrancoa: checking | 08:59 |
| marios | jfrancoa: no i'm still on internal | 08:59 |
| marios | jfrancoa: maybe try re/connect vpn | 09:00 |
| jfrancoa | marios: yes, I'll try it. thanks | 09:00 |
| quiquell | Will check | 09:00 |
| quiquell | I am good there now | 09:00 |
| jfrancoa | quiquell: marios: restarting helped, as usual :-D thanks to both | 09:01 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 09:02 |
| *** amoralej has joined #oooq | 09:08 | |
| *** amoralej has quit IRC | 09:09 | |
| *** amoralej has joined #oooq | 09:09 | |
| *** amoralej_ has quit IRC | 09:10 | |
| *** dtantsur|afk is now known as dtantsur | 09:20 | |
| *** zoli is now known as zoli|lunch | 09:38 | |
| *** brault has quit IRC | 09:44 | |
| *** brault has joined #oooq | 09:45 | |
| *** Tengu has quit IRC | 09:48 | |
| sshnaidm|rover | chkumar|ruck, seems like we hit it again: http://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e548c5a/job-output.txt.gz#_2018-07-18_07_44_08_936878 | 09:48 |
| sshnaidm|rover | http://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/6232ced/job-output.txt.gz#_2018-07-18_07_49_30_459692 | 09:48 |
| sshnaidm|rover | chkumar|ruck, is the bug still opened? | 09:48 |
| bogdando | sshnaidm|rover: hi, https://review.openstack.org/#/c/465047/ looks good to go | 09:48 |
| bogdando | at least for -2 removing :) | 09:49 |
| sshnaidm|rover | bogdando, ack, will remove | 09:49 |
| sshnaidm|rover | bogdando, btw, in which job this code runs? | 09:49 |
| bogdando | sshnaidm|rover: tested in https://review.openstack.org/#/c/583515/ | 09:50 |
| sshnaidm|rover | bogdando, and job name? | 09:50 |
| bogdando | sshnaidm|rover: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/zuul.d/multinode-jobs.yaml#n395 | 09:51 |
| bogdando | jfrancoa: ^^ am I right> | 09:51 |
| sshnaidm|rover | bogdando, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades ? | 09:52 |
| bogdando | yes, should be it | 09:52 |
| jfrancoa | bogdando: right. I triggered the job a while ago, so the results should start appearing soon | 09:56 |
| bogdando | thanks | 09:56 |
| *** jaganathan has quit IRC | 10:07 | |
| chkumar|ruck | sshnaidm|rover: https://bugs.launchpad.net/tripleo/+bug/1781871 | 10:08 |
| openstack | Launchpad bug 1781871 in tripleo "RDO cloud jobs failing with SSH Error: data could not be sent to remote host \"38.145.32.100\". Make sure this host can be reached over ssh" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 10:08 |
| chkumar|ruck | sshnaidm|rover: but it is coming on openstack servers na not on rdo cloud | 10:09 |
| sshnaidm|rover | chkumar|ruck, of course | 10:10 |
| sshnaidm|rover | chkumar|ruck, it happens everywhere | 10:10 |
| *** ykarel is now known as ykarel|afk | 10:13 | |
| *** atoth has quit IRC | 10:14 | |
| *** hubbot` has quit IRC | 10:23 | |
| quiquell | sshnaidm|rover: What is the extra_node_keypair used for ? | 10:26 |
| sshnaidm|rover | quiquell, will be used for a specific ovb job "all tls" | 10:28 |
| sshnaidm|rover | quiquell, anyway, it's not a place to remove it, every commit should have a scope | 10:29 |
| *** hubbot has joined #oooq | 10:29 | |
| quiquell | sshnaidm|rover: I have just added, but then I need to put a extra_key.pub somewhere to test this | 10:29 |
| quiquell | sshnaidm|rover: Don't understand why it's needed in infra pieces like promoter or rrcockpit | 10:30 |
| sshnaidm|rover | quiquell, not sure I follow.. | 10:32 |
| sshnaidm|rover | quiquell, I think we need to sync about this patch a little | 10:35 |
| quiquell | sshnaidm|rover: Yep, when ever you have some time we can bj on it | 10:35 |
| sshnaidm|rover | quiquell, cool, will ping you | 10:36 |
| quiquell | sshnaidm|rover: ack | 10:36 |
| *** dmellado has quit IRC | 10:36 | |
| *** zoli|lunch is now known as zoli | 10:40 | |
| *** dmellado has joined #oooq | 10:42 | |
| quiquell | Basic question, I see conditionals at master.yaml or featureset051.yaml files | 10:46 |
| quiquell | they are treated like jinja templates ? | 10:46 |
| sshnaidm|rover | quiquell, jinja templates are only in variable definition | 11:01 |
| sshnaidm|rover | quiquell, the files themselves are yaml format | 11:01 |
| quiquell | sshnaidm|rove: Found that we can have conditionals for the contents of variables | 11:02 |
| sshnaidm|rover | quiquell, but ansible evaluates jinja lang inside vars | 11:02 |
| quiquell | sshnaidm|rover: IThat's going to be soo helpful | 11:02 |
| sshnaidm|rover | quiquell, yeah, there is a limited flexibility | 11:02 |
| quiquell | sshnaidm|rover: Can you take a look at this sprint review ? | 11:03 |
| quiquell | sshnaidm|rover: https://review.openstack.org/#/c/582885/, is not very big | 11:03 |
| *** atoth has joined #oooq | 11:03 | |
| quiquell | sshnaidm|rover: Have check the conditional manually | 11:03 |
| chkumar|ruck | sshnaidm|rover: I need some help on this bug https://bugs.launchpad.net/tripleo/+bug/1782317 | 11:04 |
| openstack | Launchpad bug 1782317 in tripleo "[master] scenario008 multinode job failing at undercloud giving Invalid local_interface specified. br-ex is not available." [Critical,Triaged] | 11:04 |
| quiquell | rfolco: Check this https://review.openstack.org/#/c/582885/20/playbooks/tripleo-ci/vars/common.yaml | 11:04 |
| chkumar|ruck | where can i look for local_interface | 11:04 |
| quiquell | rfolco: That's what I mean by moving conditionals to the .yaml include vars | 11:05 |
| quiquell | rfolco: Like releases files, ej. master.yaml | 11:05 |
| sshnaidm|rover | quiquell, if tripleo_root and workspace are the same, why to use both? | 11:08 |
| quiquell | sshnaidm|rover: playbooks are expecting tripleo_root variable, but the scripts where working with WORKSPACE | 11:09 |
| quiquell | sshnaidm|rover: Just want to keep the naming | 11:09 |
| quiquell | sshnaidm|rover: We can also create diferent directories for both | 11:09 |
| sshnaidm|rover | chkumar|ruck, br-ex is not available - I suspect it's leftovers from last CI sprint | 11:10 |
| sshnaidm|rover | quiquell, tripleo_root was for folder which includes all repos, if we save them in workspace, then no need to keep it | 11:10 |
| chkumar|ruck | sshnaidm|rover: scenario 008 is broken from last 2 weeks | 11:11 |
| quiquell | sshnaidm|rover: a -e tripleo_root=workspace will be needed for the underlying playbooks | 11:11 |
| chkumar|ruck | so today I filed a bug, whom can i bug to get it fixed? | 11:11 |
| sshnaidm|rover | chkumar|ruck, yeah, it's what I'm talking about | 11:11 |
| sshnaidm|rover | chkumar|ruck, I'll handle it.. | 11:12 |
| chkumar|ruck | sshnaidm|rover: Thanks :-) | 11:12 |
| sshnaidm|rover | quiquell, I see.. then I don't understand why we copy repos to workspace | 11:13 |
| quiquell | sshnaidm|rover: All the underlying playbooks expectr to be at one root directory | 11:13 |
| sshnaidm|rover | quiquell, and how exactly to define where to use tripleo_root and where workspace, if they are the same | 11:14 |
| quiquell | sshnaidm|rover: wait... I have find in idea | 11:14 |
| quiquell | sshnaidm|rover: Playbooks are only using tripleo_root ansible var | 11:15 |
| quiquell | sshnaidm|rover: And from them access to the different repos | 11:15 |
| quiquell | sshnaidm|rover: workspace is for the script, but we can make the scripts use tripleo_root too | 11:16 |
| quiquell | sshnaidm|rover: We cannot set tripleo_root to /home/zuul/src/git.openstack.org/openstack/ | 11:16 |
| quiquell | because we have /home/zuul/src/git.openstack.org/openstack-infra/tripleo-ci/ | 11:16 |
| quiquell | tripleo-ci is at openstack-infra not openstack | 11:16 |
| sshnaidm|rover | quiquell, it's easily solved with symlink | 11:17 |
| quiquell | sshnaidm|rover: That's alreay don't at run-v3.yaml | 11:17 |
| quiquell | with hardlinks | 11:17 |
| quiquell | previous sprint did that | 11:18 |
| quiquell | so, the bash scripts use workspace and the playbooks use tripleo_root | 11:18 |
| quiquell | I can remove workspace and make the script use tripleo_root | 11:18 |
| quiquell | What do you think ? | 11:18 |
| sshnaidm|rover | quiquell, so we will continue to copy repos to workspace, I don't think it's good idea. Workspace should be place for our scripts, logs, images, other artifacts. tripleo_root/anything should be root folder for repositories | 11:20 |
| sshnaidm|rover | quiquell, actually tripleo_root should be /home/zuul/src/git.openstack.org/openstack | 11:20 |
| quiquell | sshnaidm|rover: Cannot be there | 11:20 |
| quiquell | tripleo-ci is a openstack-infra | 11:20 |
| quiquell | This is the main issue | 11:20 |
| sshnaidm|rover | quiquell, ln -s /home/zuul/src/git.openstack.org/openstack-infra/tripleo-ci /home/zuul/src/git.openstack.org/openstack/tripleo-ci | 11:20 |
| quiquell | We don't have permissions there | 11:21 |
| sshnaidm|rover | quiquell, what do you mean? | 11:21 |
| quiquell | We can write only at zuul.project.src_dir | 11:21 |
| quiquell | sshnaidm|rover: But let me try | 11:21 |
| sshnaidm|rover | quiquell, I think I had a patch that does it, and everything worked | 11:22 |
| quiquell | sshnaidm|rover: Let me check | 11:22 |
| quiquell | sshnaidm|rover: Would be nice to reduce links | 11:22 |
| quiquell | sshnaidm|rover: Still we use tripleo_root for generated stuff | 11:22 |
| sshnaidm|rover | quiquell, I will start process of moving tripleo-ci to openstack namespace also | 11:22 |
| quiquell | I don't know if /home/zuul/src/git.openstack.org/openstack/ is a good place for it | 11:22 |
| sshnaidm|rover | so maybe it will be temporary hack with symlink | 11:23 |
| sshnaidm|rover | quiquell, generated stuff? | 11:23 |
| sshnaidm|rover | quiquell, do you mean jinja templates? | 11:23 |
| quiquell | wait wait | 11:23 |
| *** ykarel|afk is now known as ykarel | 11:24 | |
| quiquell | pufff we have at least a playbook that uses the env var WORKSPACE :-/ | 11:26 |
| quiquell | but non try to write a tripleo_root | 11:26 |
| quiquell | sshnaidm|rover: So is a good thing to have both and point tripleo_root to openstack | 11:27 |
| quiquell | sshnaidm|rover: We can move tripleo-ci to openstack namespace ? | 11:27 |
| quiquell | sshnaidm|rover: It would take years ? | 11:27 |
| sshnaidm|rover | quiquell, hopefully not | 11:27 |
| chkumar|ruck | weshay: sshnaidm|rover: https://bugs.launchpad.net/tripleo/+bug/1782267 this bug does not have logs, is it related to the https://bugs.launchpad.net/tripleo/+bug/1782211 comments? | 11:34 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 11:34 |
| openstack | Launchpad bug 1782088 in tripleo "duplicate for #1782211 [master][promotion]undercloud install failed by giving ImportError: cannot import name utils at Starting OpenStack Neutron Destroy Patch Ports" [Critical,Triaged] | 11:34 |
| quiquell | sshnaidm|rover: Added the changes to the review, let's see how it works | 11:34 |
| sshnaidm|rover | chkumar|ruck, you can see that last is dup of first | 11:35 |
| sshnaidm|rover | chkumar|ruck, it's not from job, it's from local run | 11:35 |
| quiquell | marios, sshnaidm|rover: This one is ready for merge https://review.openstack.org/#/c/583022/ | 11:35 |
| quiquell | marios, sshnaidm|rover: featureset numbers as strings | 11:35 |
| *** ratailor has quit IRC | 11:38 | |
| *** jaganathan has joined #oooq | 11:44 | |
| quiquell | sshnaidm|rover: Linking of just tripleo-ci at openstack namespace is working | 11:49 |
| sshnaidm|rover | quiquell, left additional comments in https://review.openstack.org/#/c/582885/ | 11:51 |
| quiquell | sshnaidm|rover: Thanks, very helpful, good for an ansible newbie like myself | 11:58 |
| *** trown|brb is now known as trown | 12:00 | |
| *** skramaja_ has joined #oooq | 12:05 | |
| weshay | chkumar|ruck, ya.. no worries | 12:06 |
| weshay | chkumar|ruck, I see Alan's comments | 12:06 |
| weshay | chkumar|ruck, are you interested in giving status on the program call? | 12:07 |
| chkumar|ruck | weshay: sure I can update that | 12:08 |
| *** skramaja has quit IRC | 12:08 | |
| weshay | chkumar|ruck, k.. join https://bluejeans.com/7253947007/ | 12:10 |
| weshay | chkumar|ruck, search for your name, "Chandan" in https://docs.google.com/document/d/13StMviF7pnXve42F2vigeZoopoofEtqOyZj8CxVpxko/edit# | 12:10 |
| weshay | chkumar|ruck, review the status I added for July 18th | 12:11 |
| weshay | that is all you need to say :) | 12:11 |
| weshay | chkumar|ruck, make sense? | 12:11 |
| chkumar|ruck | weshay: yup correct | 12:12 |
| *** amoralej is now known as amoralej|lunch | 12:18 | |
| weshay | chkumar|ruck, almost up :) | 12:25 |
| weshay | chkumar|ruck, well done | 12:26 |
| chkumar|ruck | weshay: thanks boss :-) | 12:26 |
| * chkumar|ruck was fearing while speaking | 12:26 | |
| *** tcw has quit IRC | 12:27 | |
| *** tcw has joined #oooq | 12:27 | |
| *** tcw1 has joined #oooq | 12:28 | |
| *** tcw has quit IRC | 12:28 | |
| rfolco | quiquell, are you going to rebase https://review.openstack.org/#/c/582885 later on top of my patch ? | 12:28 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo- (1 more message) | 12:29 |
| ssbarnea1 | is there a trick i can use to be able to bundle several CRs on gerrit? like 5-10 of them, unrelated. | 12:30 |
| quiquell | ssbarnea1: topic maybe | 12:34 |
| quiquell | rfolco: This is the one that can be in the top https://review.openstack.org/#/c/582466/ | 12:35 |
| *** rlandy has joined #oooq | 12:35 | |
| quiquell | rfolco: But they are not releated | 12:35 |
| quiquell | rfolco: This one https://review.openstack.org/#/c/582885/ "common.yaml" can be at t he top of yours | 12:35 |
| quiquell | rfolco: so you can use "workspace" and "tripleo_root" and "periodic" variable | 12:35 |
| weshay | ssbarnea1, ping | 12:36 |
| quiquell | rfolco: After your patch with featureset numbers as string I think we have to merge the common.yaml | 12:36 |
| rfolco | quiquell, so things are not like we discussed yesterday | 12:37 |
| quiquell | rfolco: So we can move to cmmon.yaml stuff rom toci_gate_test and toci_quickstart | 12:37 |
| weshay | ssbarnea1, have a sec to join early? | 12:37 |
| weshay | quiquell, since today is actually wednesday.. I'm ignoring you :) | 12:37 |
| rasca | sshnaidm|rover, hi, so this https://review.openstack.org/#/c/582932/ passed with fs 10, can we proceed for the merge? | 12:37 |
| weshay | sshnaidm|rover, chkumar|ruck 21 gate failures? | 12:38 |
| quiquell | weshay: You are nog ignoring me, as you are saying that you are ignoring me | 12:38 |
| weshay | is that accurate | 12:38 |
| weshay | quiquell, SOB!!!! | 12:38 |
| quiquell | rfolco: As I remember it was to move in top of your patches this https://review.openstack.org/#/c/582466/ | 12:38 |
| quiquell | rfolco: But also we talk about moving this other as the base https://review.openstack.org/#/c/582885/ | 12:39 |
| quiquell | rfolco: I can be wrong totally | 12:39 |
| chkumar|ruck | weshay: 21 gates failures all the ovb jobs failed in the morning, due to little hiccup in http on controller node | 12:39 |
| chkumar|ruck | weshay: https://etherpad.openstack.org/p/8jz2JhpZWc | 12:39 |
| weshay | ssbarnea1, https://bluejeans.com/4113567798 | 12:39 |
| rfolco | quiquell, that sounds reasonable. I'll make the changes to reflect this. | 12:40 |
| chkumar|ruck | weshay: or you are referring to something else? | 12:40 |
| quiquell | rfolco: I am not going to add more stuff at common.yaml | 12:40 |
| quiquell | rfolco: So we can start to think about merge it, | 12:40 |
| weshay | chkumar|ruck, http://38.145.34.131/d/7q6lisOik/cockpit?orgId=1 | 12:40 |
| * weshay looking at the gate jobs | 12:40 | |
| quiquell | so it's safe to reparent now | 12:40 |
| rlandy | sshnaidm|rover: panda|off: rfolco: I need to start testing the browbeat job in in CI. I have tried 'Depends-On' but the fs035 jobs does not trigger. Can we merge https://review.rdoproject.org/r/#/c/14808/ and https://review.rdoproject.org/r/#/c/14880/ so that fs053 job will trigger on just one file change so we can test that way? | 12:40 |
| weshay | chkumar|ruck, ovb jobs can not be gate | 12:40 |
| weshay | rlandy, good morning.. thanks for your help yesterday | 12:41 |
| chkumar|ruck | weshay: others are related to thisobne https://bugs.launchpad.net/tripleo/+bug/1781871 | 12:41 |
| openstack | Launchpad bug 1781871 in tripleo "RDO cloud jobs failing with SSH Error: data could not be sent to remote host \"38.145.32.100\". Make sure this host can be reached over ssh" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 12:41 |
| rlandy | we need to watch timing and access | 12:41 |
| weshay | rlandy, the ovb introspection error is fixed https://bugs.launchpad.net/tripleo/+bug/1782268 | 12:41 |
| openstack | Launchpad bug 1782088 in tripleo "duplicate for #1782268 [master][promotion]undercloud install failed by giving ImportError: cannot import name utils at Starting OpenStack Neutron Destroy Patch Ports" [Critical,Fix released] - Assigned to yatin (yatinkarel) | 12:41 |
| *** quiquell is now known as quiquell|lunch | 12:41 | |
| chkumar|ruck | weshay: ssh issue happening in openstack jobs | 12:42 |
| *** agopi|off has quit IRC | 12:42 | |
| rlandy | weshay: lol - what would we do w/o ykarel??? | 12:42 |
| bogdando | weshay, sshnaidm|rover: https://review.openstack.org/#/c/465047/13 worked | 12:42 |
| weshay | chkumar|ruck, what issue? | 12:42 |
| bogdando | in https://review.openstack.org/#/c/583515/, it passed the needed upgrade job | 12:42 |
| sshnaidm|rover | weshay, it's infra problem with ssh | 12:42 |
| bogdando | tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades | 12:42 |
| chkumar|ruck | weshay: http://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/e548c5a/job-output.txt.gz#_2018-07-18_07_44_08_936878 | 12:42 |
| weshay | chkumar|ruck, sshnaidm|rover thanks | 12:43 |
| weshay | chkumar|ruck, sshnaidm|rover they haven't posted anything to their wiki or twitter | 12:44 |
| chkumar|ruck | switched primary address for openstackci pypi account from review@o.o to infra-root@o.o so that it doesn't get mixed in with gerrit backscatter (we can switch to a dedicated alias later if needed) | 12:44 |
| chkumar|ruck | lastmessage not seen | 12:44 |
| weshay | chkumar|ruck, sshnaidm|rover in these cases.. imho it would be informative to describe the issue and error and add a tracking bug to tripleo w/ the alert tag so that folks can get the status w/o having to bug you guys :) | 12:45 |
| rlandy | also .. | 12:45 |
| bogdando | sshnaidm|rover: commented | 12:45 |
| bogdando | we can't do by script name | 12:45 |
| rlandy | weshay: sshnaidm|rover: marios: https://review.openstack.org/#/c/581484/ - I think this can merge w/o causing issues | 12:45 |
| rfolco | rlandy, sorry if I miss something.... why do you need browbeat in all featuresets if you only run browbeat on fs053 ? | 12:46 |
| rlandy | rfolco: I wasn't sure I do | 12:47 |
| rlandy | but if you look all the jobs have upgrades included | 12:47 |
| rlandy | even though not all jobs run them | 12:47 |
| chkumar|ruck | weshay: sshnaidm|rover https://review.rdoproject.org/r/#/c/14870/ is applied in the rdo afs mirror machine after pypi caching change by apevec, if anything odd is seen, feel free to report | 12:47 |
| rlandy | also there was error a few runs ago | 12:47 |
| rlandy | saying the playbook was not available ... see | 12:48 |
| *** quiquell|lunch is now known as quiquell | 12:48 | |
| rlandy | https://logs.rdoproject.org/88/581488/8/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch/27bdb3f/job-output.txt.gz#_2018-07-17_01_12_49_452444 | 12:48 |
| rlandy | Could not find or access '/home/zuul/workspace/.quickstart/playbooks/browbeat-minimal.yml' | 12:48 |
| rlandy | even though that fs did not use the playbook | 12:48 |
| rfolco | rlandy, oh I see all those jobs have been translated by script... there is much duplication. | 12:48 |
| rlandy | sshnaidm|rover: ^^ am I right about that? | 12:49 |
| rlandy | rfolco: it's a complicated now | 12:49 |
| quiquell | rfolco: Added some comments to your patches | 12:49 |
| rlandy | and really I am guessing my way through this transformed code | 12:49 |
| *** ykarel is now known as ykarel|away | 12:49 | |
| rlandy | but we need to start testing now | 12:50 |
| rfolco | rlandy, this .quickstart dir works now ? | 12:50 |
| rfolco | it seems odd to me | 12:50 |
| rlandy | rfolco: we have passing tests now | 12:50 |
| rlandy | me too | 12:50 |
| rlandy | feel free to explain this to me :) | 12:50 |
| rfolco | rlandy, still trying to understand where this playbook is defined | 12:51 |
| rfolco | :) | 12:51 |
| chkumar|ruck | arxcruz: https://review.rdoproject.org/r/#/c/14886/ | 12:51 |
| chkumar|ruck | ajo: fs020 is already in experimental, will i replace fs020 tofs021 in experimental? | 12:52 |
| weshay | ssbarnea1, https://ci.centos.org/view/rdo/view/tripleo-gate/ | 12:52 |
| arxcruz | weshay: ^ | 12:52 |
| chkumar|ruck | ajo: sorry | 12:53 |
| ajo | chkumar|ruck: not sure if that was to me, what's fs021? :) | 12:53 |
| ajo | hehe | 12:53 |
| chkumar|ruck | arxcruz: fs020 is already in experimental, will i replace fs020 tofs021 in experimental? | 12:53 |
| ajo | ack | 12:53 |
| chkumar|ruck | ajo: featureset021 == fs021 | 12:53 |
| rlandy | rfolco: this playbook - which one? there are a bunch in the mix | 12:54 |
| bogdando | sshnaidm|rover: updated by your comments | 12:55 |
| arxcruz | chkumar|ruck: yes, it's the same, except for the skip list, so i think fs021 should also be in experimental, as non voting | 12:55 |
| arxcruz | the idea of this is to see how much we are covering on tempest and see what we can remove from skip list during time | 12:55 |
| chkumar|ruck | arxcruz: ack | 12:56 |
| marios | rlandy: ack | 12:56 |
| rfolco | rlandy, so browbeat is not a playbook you call from the job definition. Featreset053 runs toci_gate which calls browbeat under .quickstart dir (only if browbeat-minimal.yml is modified --> ansible/oooq/browbeat-minimal.yml | 12:59 |
| rlandy | under the workspace | 12:59 |
| rlandy | but yes | 12:59 |
| rfolco | rlandy, I did not look at toci_gate... but... if toci_gate clones browbeat, you don't need it in required-projects | 13:00 |
| rlandy | rfolco: then why the error? | 13:00 |
| *** hubbot has quit IRC | 13:00 | |
| rlandy | I'd happily take it out | 13:00 |
| rfolco | rlandy, the error you pointed me last time was that you were trying to use browbeat from /home/zuul/git.openstack.org/openstack/browbeat .... use = copy to workspace | 13:01 |
| rfolco | rlandy, required-projects makes zuul to clone the repo into /home/zuul/src/git.... | 13:01 |
| rfolco | but if this job is already done by zuul-cloner or quickstart, you don't need it | 13:02 |
| rfolco | rlandy, cannot find browbeat in tq tqe or tripleo-ci... where is it? | 13:03 |
| rlandy | rfolco: it's in its own repo | 13:04 |
| *** hubbot has joined #oooq | 13:04 | |
| rlandy | https://github.com/openstack/browbeat | 13:04 |
| rfolco | rlandy, but how toci_gate fs053 runs it ? | 13:04 |
| rfolco | the run playbook in https://review.rdoproject.org/r/#/c/14808/7/playbooks/legacy/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master/run.yaml runs toci_gate | 13:04 |
| rlandy | https://review.openstack.org/#/c/581488/11/playbooks/baremetal-quickstart-extras.yml | 13:04 |
| *** skramaja_ has quit IRC | 13:05 | |
| rfolco | aaaah | 13:05 |
| rfolco | :) | 13:05 |
| rlandy | sshnaidm|rover merged it | 13:05 |
| rlandy | I am happy to change it in another patch | 13:05 |
| sshnaidm|rover | rlandy, it's fine to fix it, it doesn't run anywhere for now.. | 13:06 |
| rlandy | sshnaidm|rover: thanks - yeah, we just need a test space | 13:06 |
| sshnaidm|rover | rlandy, sorry if I merged it before | 13:06 |
| rlandy | no - thank you | 13:06 |
| rlandy | we are missing some settings | 13:07 |
| rlandy | waiting for agopi/ rook to be available to add them somewhere | 13:08 |
| rlandy | then we can run | 13:08 |
| weshay | ssbarnea1, http://cistatus.tripleo.org/gates/ | 13:08 |
| *** agopi|off has joined #oooq | 13:10 | |
| *** agopi|off is now known as agopi | 13:10 | |
| rlandy | agpoi: hi | 13:12 |
| sshnaidm|rover | arxcruz, chkumar|ruck what is credentials to tempestvm? is it in our infra-setup group? | 13:12 |
| rlandy | agpoi: can we talk about the settings you took out and where to put them now? | 13:12 |
| rook | rlandy: what settings are m,issing | 13:13 |
| arxcruz | sshnaidm|rover: tempestmail ? not yet, i still need to create the playbook for that | 13:13 |
| arxcruz | sshnaidm|rover: gimme your creds, i'll add it | 13:13 |
| agopi | rlandy, hi | 13:13 |
| chkumar|ruck | sshnaidm|rover: i donot have | 13:14 |
| *** ykarel|away has quit IRC | 13:14 | |
| sshnaidm|rover | arxcruz, https://github.com/sshnaidm.keys | 13:14 |
| rlandy | agopi: rook: https://review.openstack.org/#/c/581484/5..6/config/general_config/featureset053.yml | 13:14 |
| agopi | rook, i can create a config file hardcoding most of the variables | 13:14 |
| agopi | but some can't be put in config file | 13:15 |
| rlandy | agopi: rook: we can keep the enable/disable settings | 13:15 |
| agopi | like the benchmark var itself | 13:15 |
| rlandy | and the config file | 13:15 |
| rlandy | the cloud_name can move to rdocloud settings | 13:15 |
| rlandy | hosts should move | 13:16 |
| rlandy | nothing env specific should be in this file | 13:16 |
| *** vinaykns has joined #oooq | 13:16 | |
| rlandy | make sense? | 13:16 |
| agopi | okay got it, those hosts are basically random urls because config file needs hosts in there. | 13:17 |
| agopi | ack rlandy | 13:17 |
| chkumar|ruck | weshay: sshnaidm|rover I need some help on adding fs021 in 24 periodic queue https://review.rdoproject.org/r/#/c/14887/ | 13:18 |
| chkumar|ruck | please have a look | 13:18 |
| rlandy | agopi: rook; ok - let's move those to rdocloud config | 13:18 |
| rlandy | weshay: ^^ agree? | 13:18 |
| rook | ok, so the browbeat specific config will be rdocloud? | 13:18 |
| * chkumar|ruck moves to home | 13:18 | |
| rlandy | rook: well cloud_name: rdo_cloud | 13:18 |
| rook | why not just out the entire browbeat_rdo cloud in the browbeat dir | 13:19 |
| rook | why segment it? | 13:19 |
| rlandy | the hosts will be different per env | 13:19 |
| agopi | rlandy, i can hardcode elastic_enabled, elastic_host, grafana_enabled and grafana_host so that we dont have to worry about it now | 13:19 |
| rook | it will still be rdo_cloud though, right? | 13:19 |
| agopi | those hosts are just for the sake of being there | 13:19 |
| rlandy | agopi: the enabled stuff can stay | 13:20 |
| sshnaidm|rover | bogdando, I didn't understand your argument about script though, not critical just curious | 13:20 |
| arxcruz | chkumar|ruck: https://review.openstack.org/#/c/573220/ \o/ | 13:20 |
| rfolco | rlandy, I think you can test without browbeat as required-project... If I am not missing anything, it is installed as a requirement here https://review.openstack.org/#/c/581484/6/quickstart-extras-requirements.txt | 13:20 |
| sshnaidm|rover | chkumar|ruck, firstly let's decide which branches you want to run it on | 13:21 |
| rlandy | agopi: rook: my objection is the rdo_cloud thing mostly | 13:21 |
| sshnaidm|rover | chkumar|ruck, 24 hr is mostly for pike and ocata | 13:22 |
| rook | rlandy: can you show me where? | 13:22 |
| agopi | i put cloud_name: rdo_cloud rook | 13:22 |
| rlandy | one sec - let's step back | 13:22 |
| rook | that is literally just a string... it has no meaning to anything really. | 13:22 |
| rook | in the grand scheme of things | 13:22 |
| rlandy | let's talk about where these settings **should** be | 13:22 |
| rook | it could be "bobs carwash" | 13:23 |
| rfolco | rlandy, if you are not using it from /home/zuul/src/git.openstack.org/openstack/browbeat, then its a signal you don't need it... I'll comment your patch if you don't mind | 13:23 |
| *** quiquell is now known as quiquell|launch | 13:23 | |
| *** quiquell|launch is now known as quiquell|lunch | 13:23 | |
| sshnaidm|rover | chkumar|ruck, also you need a periodic job, not regular, please see how other periodic jobs are defined | 13:23 |
| rlandy | rfoloc; one moment - will gat back to you | 13:23 |
| rook | if we were using elasticsearch it becomes important. | 13:23 |
| *** vinaykns has left #oooq | 13:23 | |
| rook | elastic + collectd/grafana | 13:23 |
| rlandy | rook: agopi: the goal is you should be able to use this featureset as well | 13:24 |
| rlandy | so, | 13:24 |
| rook | since we are not, it has no significance | 13:24 |
| sshnaidm|rover | chkumar|ruck, what is purpose of running it in periodic? | 13:24 |
| rlandy | before we move anything ... | 13:24 |
| rlandy | this is a browbeat fs | 13:24 |
| rlandy | so we can have browbeat specific settings in there | 13:25 |
| rlandy | just not env settings | 13:25 |
| rlandy | if, in CI where this fs is used | 13:25 |
| rlandy | the hosts are just fake, that is fine | 13:25 |
| rlandy | I am just trying to keep this uniform and env-agnostic | 13:26 |
| rlandy | rook; agopi: pls see https://github.com/openstack-infra/tripleo-ci/blob/master/toci-quickstart/config/testenv/ovb-rdocloud.yml | 13:26 |
| rlandy | as an example | 13:26 |
| marios | weshay: posted this btw https://review.openstack.org/#/c/583547/ for that bug you pointed me at yesterday https://bugs.launchpad.net/openstack-infra/+bug/1781255 | 13:26 |
| openstack | Launchpad bug 1781255 in tripleo ""Error: centos-release-ceph-luminous conflicts with centos-release-ceph-jewel-1.0-1.el7.centos.noarch" [Undecided,New] | 13:26 |
| marios | weshay: even though we are gonna stop using it we should probably land that fix for now anyway /first | 13:27 |
| rlandy | we would keep your cloud_name in here | 13:27 |
| weshay | marios++ | 13:28 |
| hubbot | weshay: marios's karma is now 3 | 13:28 |
| rlandy | maybe the rest can stay | 13:28 |
| marios | weshay: ha :) thanks was an easy fix and i was already staring at that code | 13:29 |
| rlandy | for sure the enable can | 13:29 |
| rlandy | alright | 13:29 |
| rlandy | I'll try a test run | 13:29 |
| agopi | ack rlandy | 13:29 |
| rlandy | agopi: is the cloud name important? | 13:30 |
| rlandy | could I switch it to rdocloud? | 13:30 |
| rlandy | no dash? | 13:30 |
| agopi | because we arent collecting data its not imp | 13:30 |
| rlandy | ok | 13:30 |
| rlandy | let me try and we will discuss from there | 13:30 |
| agopi | okay | 13:31 |
| rook | so we need to template out that from the browbeat config | 13:31 |
| agopi | yes rook as i mentioned above i can hardcode them | 13:31 |
| agopi | https://github.com/openstack/browbeat/blob/master/ansible/oooq/roles/template-configs/templates/browbeat-minimal-ci.yaml.j2 create a template specific to this job | 13:32 |
| rook | i was under the impression rlandy is suggesting to store cloud_name here : ovb-rdocloud.yml -- so, i don't get the sense hardcoding it is what we want. | 13:34 |
| rlandy | rook: yep - no hardcoding | 13:34 |
| rlandy | just env settings where they belong | 13:34 |
| rlandy | I am moving the cloud_name and the hosts - that is all | 13:34 |
| agopi | oh gotcha. | 13:35 |
| rook | so in the grand scheme of things, I think what rlandy is asking is everything under workloads: we keep... things under browbeat: ealsticsearch: grafana: they keep | 13:35 |
| rook | which gets me to another point about our browbeat config | 13:35 |
| rook | i want to break it up | 13:35 |
| rook | browbeat-infra.yaml | 13:36 |
| *** myoung|off is now known as myoung | 13:36 | |
| rook | which will have that shit, and browbeat-workloads.yaml which will be the workload definition | 13:36 |
| rook | agopi: ^ | 13:36 |
| *** tcw1 is now known as tcw | 13:36 | |
| rlandy | I am just trying to keep your testing uniform with upstream, that is all | 13:36 |
| rook | yeah | 13:36 |
| rook | understood rlandy... I think I am tracking | 13:36 |
| *** amoralej|lunch is now known as amoralej | 13:37 | |
| agopi | makes sense rook, rlandy. i now get what you're saying. | 13:38 |
| rlandy | patches coming up | 13:38 |
| *** Tengu has joined #oooq | 13:44 | |
| *** tcw has quit IRC | 13:46 | |
| *** tcw has joined #oooq | 13:47 | |
| *** quiquell|lunch is now known as quiquell | 13:54 | |
| rlandy | sshnaidm|rover: https://review.rdoproject.org/r/#/c/14880/ what is missing here for merge? | 13:58 |
| rlandy | Needs Verified | 13:58 |
| rlandy | Verified | 13:58 |
| rlandy | +1 Zuul CI | 13:58 |
| rlandy | oh nvm | 13:58 |
| rlandy | no - wait - depends-on is merged | 13:59 |
| myoung | o/ happy weds! | 14:00 |
| rlandy | still? | 14:00 |
| sshnaidm|rover | rlandy, seems zuul queue was reloaded | 14:01 |
| rlandy | sshnaidm|rover: I rechecked | 14:01 |
| sshnaidm|rover | rlandy, no need, gate jobs started | 14:01 |
| arxcruz | sshnaidm|rover: weshay rlandy trown could you guys please, please, pretty please review https://review.openstack.org/#/c/573220/ ? only this is missing to close a card :D | 14:01 |
| arxcruz | <3 | 14:01 |
| rlandy | cool | 14:03 |
| sshnaidm|rover | arxcruz, where does this code execute? | 14:03 |
| arxcruz | sshnaidm|rover: scenario004 | 14:03 |
| rlandy | yay!!! | 14:05 |
| rlandy | we have test kick | 14:05 |
| rlandy | legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master | 14:05 |
| sshnaidm|rover | arxcruz, did you test it on different from master branch? | 14:05 |
| agopi | rlandy++ | 14:06 |
| hubbot | agopi: rlandy's karma is now 14 | 14:06 |
| rlandy | rfolco: as soon as I see one clear run, I'll remove browbeat from required projects | 14:06 |
| rlandy | then we can see how that works | 14:06 |
| arxcruz | sshnaidm|rover: other branches are not affectred, only master and rocky | 14:07 |
| rfolco | rlandy, sounds like a plan | 14:07 |
| rlandy | agopi: rook: weshay: we have a queued job in https://review.rdoproject.org/zuul/status.html | 14:07 |
| rlandy | let's see what shakes out | 14:07 |
| arxcruz | tempestconf on other branches also doesn't have the patch, so we are good | 14:07 |
| weshay | rlandy, which review? | 14:08 |
| rlandy | openstack/browbeat | 14:08 |
| rlandy | 583581,2 | 14:08 |
| sshnaidm|rover | arxcruz, yeah, i just wasn't sure about such jinja construct | 14:08 |
| weshay | rlandy+++ | 14:09 |
| *** quiquell is now known as quiquell|off | 14:12 | |
| *** dtrainor has quit IRC | 14:20 | |
| weshay | rlandy, sshnaidm|rover fyi.. so the introspection bug on libvirt deployments is most likely caused by a baseos / libvirt change https://bugzilla.redhat.com/show_bug.cgi?id=1576464 | 14:25 |
| openstack | bugzilla.redhat.com bug 1576464 in libvirt "Hash operation not allowed during iteration" [High,Verified] - Assigned to mprivozn | 14:25 |
| weshay | fyi | 14:25 |
| * weshay getting into it | 14:25 | |
| *** udesale_ has joined #oooq | 14:26 | |
| *** udesale has quit IRC | 14:27 | |
| rlandy | well | 14:28 |
| rlandy | interesting find | 14:29 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo- (1 more message) | 14:29 |
| sshnaidm|rover | weshay, so waiting for libvirt-4.3.0-1 | 14:32 |
| myoung | panda|off: If you are available for a quick sync after training today, please ping. I need 3-5 min. | 14:32 |
| weshay | sshnaidm|rover, why would this work in ci.centos? | 14:35 |
| chkumar|ruck | sshnaidm|rover: we wanted a 24 periodic job for all branches running fs21 so that we can clear the skip list at regular interval and fix the job | 14:35 |
| weshay | my brain hurts | 14:35 |
| chkumar|ruck | sshnaidm|rover: it should not be part of promotion | 14:35 |
| weshay | sshnaidm|rover, https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-375/undercloud/var/log/extra/rpm-list.txt.gz | 14:40 |
| weshay | I don't get it | 14:40 |
| sshnaidm|rover | weshay, chkumar|ruck one by one, please :) | 14:41 |
| weshay | https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-375/172.19.2.142/var/log/extra/rpm-list.txt.gz | 14:41 |
| weshay | lolz | 14:41 |
| weshay | sshnaidm|rover, I'm just fyi | 14:41 |
| weshay | 'ing you | 14:41 |
| weshay | chkumar|ruck, it can be in the 24hr queue next to the ocata/pike jobs | 14:42 |
| weshay | chkumar|ruck, we just wont add it to https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/dlrnapi_promoter/config | 14:42 |
| weshay | chkumar|ruck, make sense? | 14:42 |
| chkumar|ruck | weshay: yup make sense, lots of new things to grasp in last 2 weels | 14:43 |
| sshnaidm|rover | chkumar|ruck, part of promotion or not - decides criteria files like this https://github.com/rdo-infra/ci-config/blob/614e886f17e146c9327b77d57009cd82394649f7/ci-scripts/dlrnapi_promoter/config/queens.ini#L1 | 14:43 |
| sshnaidm|rover | chkumar|ruck, we can run it with all other periodic jobs, no problem | 14:44 |
| chkumar|ruck | sure, I will create the zuulv3 job for fs21 for stable branches | 14:44 |
| sshnaidm|rover | chkumar|ruck, but: 1) it should be periodic 2) better to have it for all branches and run it according to branch run policy | 14:44 |
| *** dtrainor has joined #oooq | 14:45 | |
| sshnaidm|rover | chkumar|ruck, like 24 hr for pike and ocata, openstack-periodic for master queens | 14:45 |
| sshnaidm|rover | chkumar|ruck, just look at repo how other periodic jobs are defined | 14:45 |
| chkumar|ruck | sshnaidm|rover: do we need to keep for ocata, it is already EOL? | 14:46 |
| sshnaidm|rover | chkumar|ruck, is it? | 14:46 |
| chkumar|ruck | i mean downstream RHOS-11 EOLed already | 14:46 |
| chkumar|ruck | we have kept it for fast forward upgrades | 14:46 |
| sshnaidm|rover | chkumar|ruck, well, when it will be EOLed we'll delete just everything, including 012 job | 14:49 |
| sshnaidm|rover | chkumar|ruck, it won't add us additional work | 14:50 |
| sshnaidm|rover | s/012/021/ | 14:50 |
| sshnaidm|rover | chkumar|ruck, but up to you | 14:50 |
| chkumar|ruck | sshnaidm|rover: sure, | 14:50 |
| rasca | chkumar|ruck, can you have a look at this https://review.openstack.org/#/c/573255/ and maybe drive the +2 +workflow so we can merge? | 14:56 |
| *** kopecmartin has quit IRC | 14:56 | |
| weshay | sshnaidm|rover, +1 correct on the direction w/ that job | 14:58 |
| weshay | chkumar|ruck, sshnaidm|rover I'm not terribly concerned about ocata, it's not EOL upstream | 14:58 |
| weshay | chkumar|ruck, sshnaidm|rover let's focus on master/rocky, queens, pike | 14:58 |
| weshay | chkumar|ruck, especially queens and pike the skip list is terrible | 14:59 |
| weshay | sshnaidm|rover, hrm.. I don't see a fix in centos | 15:03 |
| weshay | https://errata.devel.redhat.com/advisory/34155/builds | 15:03 |
| weshay | https://cbs.centos.org/koji/packageinfo?packageID=130 | 15:03 |
| sshnaidm|rover | weshay, last update is form 2018-05-09 | 15:05 |
| weshay | ya | 15:05 |
| sshnaidm|rover | weshay, doesn't seems like people hurry to build it.. | 15:05 |
| weshay | sshnaidm|rover, we could rebuild from src to test | 15:05 |
| weshay | sshnaidm|rover, btw.. I still can't get master rdo2 fs20 to get passed the undercloud | 15:05 |
| sshnaidm|rover | weshay, maybe something with slave..? | 15:07 |
| weshay | naw | 15:07 |
| sshnaidm|rover | weshay, if it fetches the right code and uses old, seems like it has the old somewhere | 15:08 |
| sshnaidm|rover | weshay, cache, temp dirs.. | 15:09 |
| weshay | sshnaidm|rover, I rm -Rf /home/rhos-ci/jenkins/workspace | 15:09 |
| weshay | sshnaidm|rover, could be a temp dir I suppose | 15:09 |
| weshay | sshnaidm|rover, maybe the image? | 15:09 |
| weshay | sshnaidm|rover, maybe /var/cache/tripleo-quickstart? | 15:09 |
| arxcruz | holly shit! weshay i FOUND THE PROBLEM WITH SCENARIO002 | 15:12 |
| weshay | arxcruz, oh my | 15:13 |
| weshay | ? | 15:13 |
| weshay | arxcruz, so in the endpoint url? | 15:13 |
| arxcruz | weshay: no | 15:13 |
| arxcruz | weshay: for some reason, tempest.conf is returning the region as RegionOne, so it's getting different endpoint | 15:14 |
| arxcruz | the correct is regionOne | 15:14 |
| sshnaidm|rover | weshay, I'd terminate the slave totally.. | 15:14 |
| arxcruz | I ALWAYS told this regionOne was a terrible idea | 15:14 |
| weshay | sshnaidm|rover, ? | 15:15 |
| weshay | sshnaidm|rover, who are you Arnold | 15:15 |
| weshay | ? | 15:15 |
| weshay | arxcruz, oh man | 15:16 |
| chkumar|ruck | arxcruz: camelcasing was always a problem another example was SwiftOperator | 15:16 |
| weshay | arxcruz, it's such a nice moment when you find that shit | 15:16 |
| weshay | and so frustrating all at the same time | 15:16 |
| sshnaidm|rover | weshay, annihilate it | 15:17 |
| weshay | sshnaidm|rover, I basically did man | 15:17 |
| weshay | sshnaidm|rover, the next thing to try would be to clean the virthosts | 15:17 |
| arxcruz | chkumar|ruck: weshay tosky so, we have a bug in tempestconf | 15:17 |
| arxcruz | http://logs.openstack.org/96/582996/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/4e48c7b/logs/undercloud/home/zuul/tempest_container.sh.txt.gz | 15:17 |
| weshay | sshnaidm|rover, we could pin that job to a particular host | 15:17 |
| arxcruz | it's passing the tempest-deployer-input.conf that has the regionOne set, but on tempest.conf is not there http://logs.openstack.org/96/582996/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/4e48c7b/logs/undercloud/home/zuul/tempest/etc/tempest.conf.txt.gz | 15:18 |
| chkumar|ruck | arxcruz: https://github.com/openstack/tempest/blob/master/tempest/config.py#L132 | 15:19 |
| chkumar|ruck | may be it is taking the default one | 15:19 |
| tosky | arxcruz: so it's a regression? It used to work, and also it didn't show in other jobs | 15:20 |
| tosky | we merged patches until few hours ago | 15:20 |
| arxcruz | tosky: checking other jobs to see if that is also the case in other jobs | 15:20 |
| chkumar|ruck | oh it is capital R | 15:20 |
| arxcruz | tosky: so, it's only on scenario002, let's see why | 15:21 |
| weshay | marios, https://review.openstack.org/#/c/583547/3 | 15:22 |
| tosky | arxcruz: what is the difference? Is it containerized and the others are not? | 15:23 |
| tosky | arxcruz: maybe the file is not mounted in the container correctly (see http://logs.openstack.org/96/582996/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/4e48c7b/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-07-18_13_01_36 ? ) | 15:23 |
| arxcruz | tosky: checking right now | 15:23 |
| rasca | weshay, chkumar|ruck, so the error I get in rdocloud deploying master seems totally reproducible -> "Unable to find any of pip2, pip to use. pip needs to be installed." | 15:23 |
| rasca | weshay, chkumar|ruck, is some sort of known issue? | 15:23 |
| marios | weshay: checking | 15:23 |
| tosky | arxcruz: that's it - -v is for volumes, not files | 15:25 |
| chkumar|ruck | rasca: nope not seen | 15:25 |
| tosky | arxcruz: the file is not passed to the container | 15:25 |
| arxcruz | tosky: do you know the option for file ? | 15:25 |
| rasca | rlandy, maybe you can give me a hint here (look above please) | 15:25 |
| chkumar|ruck | tosky: arxcruz but whitelist or skipfile are getting passed there | 15:25 |
| tosky | arxcruz: I don't think you can pass files | 15:26 |
| tosky | chkumar|ruck: sure? | 15:26 |
| rlandy | rasca: using the reproducer or quickstart.sh directly? | 15:27 |
| chkumar|ruck | tosky: http://logs.openstack.org/96/582996/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/4e48c7b/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-07-18_13_01_36 | 15:27 |
| tosky | chkumar|ruck: how do you know that they are passed and working? | 15:27 |
| weshay | marios, think you can just assume to nuke ceph from orbit | 15:27 |
| rasca | rlandy, quickstart at the moment | 15:27 |
| rlandy | we merged a patch yesterday that weshay wrote to expose pip errors | 15:27 |
| chkumar|ruck | tosky: check test results | 15:27 |
| tosky | chkumar|ruck: no - that's the command line which says that you are passing them, it does not mean that it works | 15:27 |
| rlandy | looking for it | 15:27 |
| marios | weshay: ack locking on | 15:27 |
| rasca | rlandy, consider about the reproducer that I'm running this locally on my rdocloud tenant, so how can I obtain a reproducer? | 15:28 |
| rlandy | rasca: you can take one from another job and edit the changes to incorporate | 15:29 |
| rasca | rlandy, I did it once but I think that by this time I'll need to change it again | 15:29 |
| rasca | rlandy, can you give me a link as a reference? | 15:29 |
| tosky | chkumar|ruck, arxcruz: but anyway, regardless of whether the files are passed or not, how is it working if the region is incorrect? | 15:30 |
| rlandy | rasca: wget this file: https://logs.rdoproject.org/84/581484/6/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/0118a75/logs/reproducer-quickstart.sh | 15:30 |
| rlandy | ^^ fs001 | 15:30 |
| rlandy | you can change that | 15:30 |
| rlandy | ${TOCI_JOBTYPE:="ovb-3ctlr_1comp-featureset001"} | 15:30 |
| arxcruz | tosky: chkumar|ruck give me a few minutes, i'm debugging | 15:31 |
| rlandy | : ${ZUUL_CHANGES:="openstack/tripleo-quickstart:master:refs/changes/84/581484/6"} | 15:31 |
| rlandy | ^^ edit that | 15:31 |
| rlandy | will give you an ovb job | 15:31 |
| *** jaganathan has quit IRC | 15:31 | |
| rasca | rlandy, trying it at once thanks | 15:34 |
| marios | weshay: https://review.openstack.org/#/c/583547/4/scripts/tripleo.sh | 15:37 |
| marios | gf | 15:39 |
| sshnaidm|rover | weshay, look much more green now | 15:40 |
| weshay | sshnaidm|rover, I can't forget but we need to build jobs w/ centos fasttrack | 15:45 |
| weshay | to help sniff out issues w/ minor releases | 15:45 |
| weshay | I'll get that going | 15:45 |
| sshnaidm|rover | weshay, what is fasttrack | 15:46 |
| sshnaidm|rover | ? | 15:46 |
| weshay | sshnaidm|rover, they start to drop in centos 7.6 updates | 15:46 |
| weshay | so we can flag issues before another minor update of cenots | 15:47 |
| weshay | centos | 15:47 |
| * weshay has a card on it some where | 15:47 | |
| rasca | rlandy, just one last clarification, TOCI_JOBTYPE is not clear to me | 15:47 |
| sshnaidm|rover | weshay, is it about testing new centos release or just issues? | 15:47 |
| rasca | rlandy, the featureset part, yes, it is | 15:47 |
| rasca | rlandy, but the previous, not | 15:47 |
| weshay | sshnaidm|rover, fasttrack is only about the $next release | 15:47 |
| arxcruz | chkumar|ruck: how can i run tempest and get into the docker container? | 15:47 |
| weshay | sshnaidm|rover, so imho.. master jobs only.. | 15:48 |
| rasca | rlandy, take : ${TOCI_JOBTYPE:="ovb-3ctlr_1comp-featureset041"} how am I supposed to change "ovb-3ctlr_1comp"? | 15:48 |
| weshay | in a weekly or something pipeline | 15:48 |
| sshnaidm|rover | weshay, but these changes are introduced in rhel firstly, isn't it? | 15:48 |
| weshay | sshnaidm|rover, of course | 15:48 |
| weshay | sshnaidm|rover, we don't have any rdo on rhel jobs atm | 15:48 |
| sshnaidm|rover | weshay, and before that they are introduced into fedora afaik | 15:48 |
| weshay | not until we get them going in the internal sf | 15:48 |
| rlandy | rasca: yep - needs to change | 15:48 |
| weshay | sshnaidm|rover, and the sky is blue :) | 15:49 |
| rlandy | but we have done a lot of refactoring | 15:49 |
| rlandy | so you may hit other problems | 15:49 |
| sshnaidm|rover | weshay, so maybe worth to catch them in fedora, sooner is better | 15:49 |
| weshay | sshnaidm|rover, tripleo doesn't support fedora | 15:49 |
| sshnaidm|rover | weshay, aren't we gonna test it on python3? | 15:50 |
| sshnaidm|rover | weshay, on fedora raw | 15:50 |
| sshnaidm|rover | weshay, iirc | 15:51 |
| rasca | rlandy, yes but my problem is that I don't know what to change! | 15:51 |
| chkumar|ruck | arxcruz: http://logs.openstack.org/96/582996/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/4e48c7b/logs/undercloud/home/zuul/tempest-setup.sh | 15:51 |
| arxcruz | chkumar|ruck: but it will execute tempest as well | 15:51 |
| sshnaidm|rover | weshay, well, anyway, if we have these repos anywhere, worth to test them of course, need a new release file | 15:52 |
| arxcruz | i want to stay inside the docker container to check some settings | 15:52 |
| arxcruz | chkumar|ruck: i remember you did something | 15:52 |
| chkumar|ruck | arxcruz: in this script ttp://paste.openstack.org/show/726211/ | 15:52 |
| chkumar|ruck | http://paste.openstack.org/show/726211/ | 15:52 |
| chkumar|ruck | s/-i/-it and replace last script part | 15:53 |
| arxcruz | chkumar|ruck: k | 15:53 |
| *** agopi has left #oooq | 15:53 | |
| *** agopi has joined #oooq | 15:53 | |
| *** links has quit IRC | 15:55 | |
| *** jfrancoa has quit IRC | 15:56 | |
| chkumar|ruck | arxcruz: does it worked? | 15:58 |
| *** gkadam is now known as gkadam-afk | 15:59 | |
| arxcruz | chkumar|ruck: no, it's stucked | 16:01 |
| *** bogdando has quit IRC | 16:04 | |
| chkumar|ruck | arxcruz: can I get into that system? | 16:06 |
| *** zoli is now known as zoli|PTO | 16:06 | |
| *** zoli|PTO is now known as zoli | 16:06 | |
| arxcruz | chkumar|ruck: your keys ? | 16:06 |
| *** saneax has quit IRC | 16:07 | |
| chkumar|ruck | arxcruz: http://paste.openstack.org/show/726213/ | 16:07 |
| rlandy | rasca: 41 is your fs? | 16:09 |
| rasca | rlandy, yes it is | 16:09 |
| rlandy | {TOCI_JOBTYPE:="ovb-3ctlr_1comp-featureset041"} is correct then | 16:10 |
| rlandy | if you're running on ovb | 16:11 |
| rlandy | you are correct | 16:11 |
| *** sshnaidm|rover is now known as sshnaidm|bbl | 16:12 | |
| *** udesale_ has quit IRC | 16:13 | |
| *** udesale has joined #oooq | 16:13 | |
| rasca | rlandy, yeah I get to the same conclusion :) I'm completing the installation on the undercloud now | 16:16 |
| rlandy | rasca: apologies I misunderstood your original question | 16:17 |
| rlandy | happy to hear you are moving along | 16:17 |
| *** udesale has quit IRC | 16:18 | |
| *** gkadam-afk is now known as gkadam | 16:23 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container- (1 more message) | 16:29 |
| *** florianf has quit IRC | 16:31 | |
| rfolco | does anybody know why containers-multinode is timing out ? | 16:35 |
| rfolco | chkumar|ruck, ^ | 16:36 |
| chkumar|ruck | rfolco: https://bugs.launchpad.net/tripleo/+bug/1781888 | 16:37 |
| openstack | Launchpad bug 1781888 in tripleo "[stable/queens]tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades job timing out on stable/queens noop jobs " [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 16:37 |
| chkumar|ruck | you mean this one? | 16:37 |
| chkumar|ruck | rfolco: https://bugs.launchpad.net/tripleo/+bug/1782102 | 16:38 |
| openstack | Launchpad bug 1782102 in tripleo "[master] Multinode periodic RDO promotion jobs are timing out on FS035, 001, 002" [Critical,Triaged] | 16:38 |
| rfolco | chkumar|ruck, http://logs.openstack.org/22/583022/2/gate/tripleo-ci-centos-7-containers-multinode/20bde14/ | 16:38 |
| chkumar|ruck | rfolco: it is a known issue | 16:39 |
| chkumar|ruck | rfolco: one min | 16:39 |
| chkumar|ruck | rfolco: https://bugs.launchpad.net/tripleo/+bug/1781871 | 16:40 |
| openstack | Launchpad bug 1781871 in tripleo "RDO cloud jobs failing with SSH Error: data could not be sent to remote host \"38.145.32.100\". Make sure this host can be reached over ssh" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 16:40 |
| chkumar|ruck | rfolco: paul is working on that | 16:40 |
| chkumar|ruck | it tracks for both rdo and openstack | 16:40 |
| rfolco | chkumar|ruck, thanks | 16:46 |
| *** myoung is now known as myoung|lunch | 16:50 | |
| * chkumar|ruck out now | 16:55 | |
| *** atoth has quit IRC | 16:58 | |
| *** atoth has joined #oooq | 16:58 | |
| *** trown is now known as trown|lunch | 17:05 | |
| weshay | marios, doesn't the bootstrap role have to be in tripleo-ci? | 17:15 |
| weshay | https://review.openstack.org/#/c/583195/2/playbooks/tripleo-ci/run-v3.yaml | 17:15 |
| weshay | does the pre playbook have access to the tqe roles at that point? | 17:16 |
| *** dtantsur is now known as dtantsur|afk | 17:26 | |
| *** tesseract has quit IRC | 17:35 | |
| *** gkadam has quit IRC | 17:38 | |
| *** amoralej is now known as amoralej|off | 17:39 | |
| *** atoth has quit IRC | 17:39 | |
| *** atoth has joined #oooq | 17:40 | |
| weshay | rlandy, so you just have the job up.. browbeat is not integrated yet right? | 17:45 |
| rlandy | weshay: I missed changing https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L132 | 17:45 |
| rlandy | I changed baremetal-full-deploy.yml | 17:46 |
| rlandy | I need to add another patch to trigger that playbook | 17:46 |
| rlandy | and some way to not run it if var is not set | 17:46 |
| rasca | rlandy, weshay, so the job is in the deploy overcloud phase, which is really good. I'd say that we can consider the patches good. | 17:46 |
| rlandy | rasca: good news | 17:47 |
| rasca | rlandy, about baremetal-full-deploy.yml remember that inside one of the pending reviews there's a modification on that file | 17:47 |
| rlandy | weshay: kind of confusing with this playbook override | 17:47 |
| rlandy | " $QUICKSTART_SH_JOBS " =~ " $TOCI_JOBTYPE " | 17:47 |
| rlandy | need to check that condition | 17:47 |
| rasca | rlandy, but in any case I would like to have a clearer idea about how the reproducer thing works | 17:47 |
| rlandy | rasca: we can go through that tomorrow | 17:48 |
| rasca | rlandy, yes, please it would be great. | 17:48 |
| *** jtomasek has quit IRC | 17:50 | |
| *** dsneddon has quit IRC | 17:51 | |
| rlandy | weshay: adding another patch and will rerun | 17:52 |
| rfolco | rlandy, " $QUICKSTART_SH_JOBS " =~ " $TOCI_JOBTYPE --> I replace this with 2 simple ifs... one for ovb, another for multinode... https://review.openstack.org/#/c/582385/6/playbooks/tripleo-ci/templates/toci_gate_test.sh.j2 | 17:56 |
| rfolco | as part of toci_jobtype replacement | 17:57 |
| rlandy | rfolco: sounds good - working with what is there atm | 17:57 |
| rlandy | anyone know if it is possible to block include playbooks? | 18:04 |
| weshay | rlandy, weird.. now power on is working.. but introspection still times.. just fyi'ing / complaining not looking for help | 18:07 |
| rlandy | weshay: while we are complaining ... the old way of including playbooks is very wasteful | 18:08 |
| weshay | ? | 18:08 |
| rlandy | all things considered, would be way easier to add an extra playbook to the run on fs053 | 18:08 |
| rlandy | at some point | 18:08 |
| *** myoung|lunch is now known as myoung | 18:09 | |
| rlandy | I need to conditionally include a playbook in toci_gate_test | 18:09 |
| weshay | rlandy, you mean specify the extra playbook in the fs config? | 18:09 |
| rlandy | the playbook specification is set out in toci_gate_test | 18:10 |
| rlandy | so ... for the moment ... | 18:10 |
| rlandy | I am looking if it is possible to block include playbooks | 18:10 |
| weshay | rlandy, you mean adding another section like.. https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L173 | 18:11 |
| weshay | rlandy, browbeat) | 18:11 |
| weshay | ? | 18:11 |
| rlandy | no - I was going to always include it | 18:12 |
| myoung | I've finally completed excavating email/trello/things from the weeks away. With remaining time this sprint per guidance would like to jump in and help QA cards. Are there specific cards that need more QA attention more than others at this point? | 18:12 |
| rlandy | and block: when: the playbook itself | 18:12 |
| weshay | rlandy, and block? | 18:13 |
| rlandy | block: | 18:13 |
| rlandy | when: | 18:13 |
| rlandy | - include | 18:13 |
| rlandy | but I don;t think that is possible with playbooks | 18:14 |
| rlandy | just tasks | 18:14 |
| weshay | rlandy, for JOB_TYPE_PART in $(sed 's/-/ /g' <<< "${TOCI_JOBTYPE:-}") ; do | 18:14 |
| weshay | so currently.. as things stand the old way | 18:14 |
| rlandy | yaeh - but I don't want a whole section | 18:14 |
| weshay | rlandy, if browbeat is in the job_type string.. | 18:14 |
| weshay | you can case on it | 18:14 |
| rlandy | just to checf or fs053 | 18:14 |
| weshay | rlandy, hrm.. | 18:14 |
| rlandy | and then add another playbook | 18:14 |
| rlandy | which I can do | 18:14 |
| rlandy | I think | 18:14 |
| weshay | rlandy, or you could add some logic in the ovb section | 18:15 |
| weshay | to check the fs file | 18:16 |
| weshay | as well | 18:16 |
| weshay | and set the playbooks there | 18:16 |
| weshay | this is the problem with software | 18:16 |
| weshay | there are multiple ways of doing things | 18:16 |
| weshay | always | 18:16 |
| weshay | rlandy, so forward looking | 18:16 |
| rlandy | things will be much better | 18:16 |
| weshay | rlandy, what happens when we need n+1 browbeat jobs | 18:16 |
| rlandy | with the new configuration | 18:17 |
| rlandy | weshay: if we want more browbeat jobs, then we have to do this: | 18:17 |
| rlandy | always include the browbeat playbook for ovb | 18:17 |
| rlandy | and switch case within the playbook | 18:18 |
| rlandy | as I was trying to do at the start | 18:18 |
| rlandy | to go back to my original question then ... | 18:19 |
| rlandy | how does one use blocks to include playbooks? | 18:19 |
| rlandy | idk if that is possible | 18:19 |
| *** dsneddon has joined #oooq | 18:28 | |
| weshay | rlandy, bah | 18:28 |
| rlandy | https://paste.fedoraproject.org/paste/kNPAcCI5MVozPqdnX9lMuw | 18:28 |
| rlandy | weshay: ^^ is that legitimate ansible | 18:28 |
| rlandy | for a playbook?? | 18:28 |
| * rlandy hopes against the odds | 18:28 | |
| rlandy | hack, hack, hack | 18:29 |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container- (1 more message) | 18:29 |
| rlandy | I could just include when: on every line | 18:30 |
| weshay | rlandy, bah | 18:30 |
| weshay | why | 18:30 |
| weshay | why | 18:30 |
| weshay | why | 18:30 |
| rlandy | why do I want to do that? | 18:30 |
| * weshay is debating between | 18:30 | |
| weshay | rlandy, let's blue | 18:31 |
| weshay | sorry | 18:31 |
| rlandy | k - joining | 18:31 |
| *** atoth has quit IRC | 18:39 | |
| weshay | https://review.openstack.org/583700 | 18:40 |
| weshay | rlandy, ^ | 18:40 |
| Tengu | weshay: heya! did you see my comment on your review for the validations in CI? | 18:56 |
| weshay | aye | 19:12 |
| weshay | https://review.openstack.org/#/c/583275/ | 19:13 |
| weshay | I swear | 19:20 |
| weshay | 2018-07-18 16:33:50 | "fatal: [localhost]: FAILED! => {\"changed\": true, \"cmd\": [\"ntpdate\", \"-u\", \"pool.ntp.org\"], \"delta\": \"0:00:08.711899\", \"end\": \"2018-07-18 16:33:47.729579\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2018-07-18 16:33:39.017680\", \"stderr\": \"18 Jul 16:33:47 ntpdate[6200]: no server suitable for synchronization found\", \"stderr_lines\": [\"18 Jul 16:33:47 ntpdate[6200]: | 19:20 |
| weshay | no server suitable for synchronization found\"], \"stdout\": \"\", \"stdout_lines\": []}", | 19:20 |
| weshay | I'm going to burn jenkins down | 19:20 |
| myoung | weshay: got a link to that job? will add to the IT ticket I opened last night | 19:39 |
| weshay | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/config/environments/oooq-internal.yml/*view*/ | 19:40 |
| weshay | vs. | 19:41 |
| weshay | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/environments/oooq-internal.yml | 19:41 |
| weshay | myoung, I want to try this w/o the ci-script though | 19:41 |
| *** atoth has joined #oooq | 19:43 | |
| * myoung looks | 19:43 | |
| weshay | https://code.engineering.redhat.com/gerrit/#/c/144346/ | 19:45 |
| rlandy | agpoi: hi - how hard is it to get a change merged in browbeat? | 19:50 |
| agopi | rlandy, | 19:51 |
| agopi | easy enough | 19:51 |
| agopi | rook, | 19:51 |
| agopi | is it this one rlandy ? | 19:51 |
| agopi | https://review.openstack.org/#/c/583581/ | 19:51 |
| rlandy | I need to edit it - sec | 19:51 |
| agopi | okay, please ping rook when ready to be merged. | 19:52 |
| rook | agopi: one sec. | 19:52 |
| agopi | okay rook | 19:52 |
| rlandy | agopi: rook: https://review.openstack.org/583581 Add conditional to browbeat minimal test | 19:53 |
| rlandy | I may need to undo that at some point but it would help if that was in atm | 19:54 |
| rook | agopi: load that page :) | 19:54 |
| agopi | aww | 19:54 |
| agopi | rook++ | 19:54 |
| hubbot | agopi: rook's karma is now 1 | 19:54 |
| agopi | <3 | 19:54 |
| rook | agopi++ | 19:54 |
| hubbot | rook: agopi's karma is now 1 | 19:54 |
| *** atoth has quit IRC | 19:59 | |
| *** atoth has joined #oooq | 19:59 | |
| weshay | myoung, you can close the ticket | 20:02 |
| weshay | it's something in our tooling | 20:02 |
| weshay | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/tripleo-environments/config/environments/oooq-internal.yml/*view*/ | 20:02 |
| myoung | weshay: i've been backtracking thru the past few days of deltas, why don't they match? was about to hop on that node | 20:02 |
| rlandy | rook: agopi: we are including the minimal playbook in the list of default CI playbooks, hence this review will protect other featuresets from kicking browbeat: https://review.openstack.org/#/c/583581/ | 20:03 |
| myoung | sagi's changes are not reflected, but it looks from the sha that we're pulling down tripleo-env at the right revision to include it | 20:03 |
| weshay | myoung, vs.. https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/config/environments/oooq-internal.yml/*view*/ | 20:03 |
| weshay | myoung, is there any place where we pin tripleo-environments? | 20:03 |
| rlandy | ty | 20:03 |
| agopi | oh okay understood rlandy, and i understand this makes easier to kick browbeat off on other tests too right? | 20:03 |
| myoung | not that i'm aware of...or at least that i recall. looking... | 20:04 |
| myoung | or do we have a problem with workspace clean actually cleaning? | 20:04 |
| * myoung looks | 20:04 | |
| rlandy | agopi: ack | 20:05 |
| rlandy | more generic | 20:05 |
| agopi | sounds good rlandy | 20:06 |
| myoung | weshay: /ws/tripleo-environments/config/environments/oooq-internal.yml is the one from git, and if jenkins is to be trusted it's wonky that ws/config/environments/oooq-internal.yml (the one copied by unpacking of the egg) is stale | 20:06 |
| myoung | can i hope on that node? what's the timestamp delta? seems like the old oooq-internal in the destination location isn't being overridden | 20:07 |
| myoung | hop* | 20:07 |
| myoung | weshay: ahh you just wiped it ;) | 20:09 |
| weshay | I did previously as well | 20:09 |
| weshay | but this time disconnected the slave | 20:09 |
| myoung | ya i saw, i'm on it too | 20:10 |
| * myoung won't change anything...just poking it | 20:10 | |
| agopi | rlandy, not sure how it happened but if you checkc https://review.openstack.org/#/c/583717/1/ansible/oooq/browbeat-minimal.yml it only shows your comment as the diff | 20:10 |
| weshay | myoung, https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/config/environments/oooq-internal.yml/*view*/ | 20:10 |
| weshay | still a problem | 20:10 |
| agopi | it shouldn't as your change hasn't been merged yet | 20:10 |
| agopi | https://github.com/openstack/browbeat/blob/master/ansible/oooq/browbeat-minimal.yml | 20:10 |
| myoung | wtf where is that coming from | 20:10 |
| agopi | or am i missing something | 20:11 |
| myoung | weshay: /me is digging thru https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/113/console | 20:11 |
| rlandy | agopi, that is correct | 20:11 |
| rlandy | built off the change I asked to merge | 20:11 |
| agopi | okay gotcha | 20:12 |
| *** sshnaidm|bbl is now known as sshnaidm|rover | 20:18 | |
| rfolco | sshnaidm|rover, any ideas why gate containers-multinode fails? https://review.openstack.org/#/c/583022/ | 20:21 |
| rfolco | timing out :( | 20:21 |
| sshnaidm|rover | rfolco, yes, known infra problem, recheck | 20:22 |
| rfolco | sshnaidm|rover, recheck won't run gate again. | 20:22 |
| sshnaidm|rover | rfolco, what do you mean? | 20:22 |
| rfolco | sshnaidm|rover, nm, will recheck | 20:23 |
| rfolco | thx | 20:24 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo- (1 more message) | 20:30 |
| myoung | weshay: i think it's coming from the pip cache somehow...debugging it | 20:30 |
| myoung | weshay: i think would make sense to obliterate /home/rhos-ci/.cache/ and see what happens | 20:31 |
| weshay | ya.. just did that on https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/computer/rdo-manager-slave_rdo-ci-fx2-01-s2/log | 20:31 |
| weshay | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/config/environments/oooq-internal.yml/*view*/ | 20:33 |
| weshay | myoung, worked | 20:33 |
| weshay | myoung, so maybe the ticket to support is.. what is the proper way to disable jenkins pip cache per slave | 20:34 |
| myoung | hrm...wonder if the --no-cache-dir added https://github.com/openstack/tripleo-quickstart/commit/df55d3908d053b5b3a33d0dfa22ed7cbe65ea1b6 a couple weeks ago would make this work better....just run with OPT_CLEAN | 20:35 |
| weshay | that's a straight up jenkins bug | 20:39 |
| weshay | myoung, oh in the full-deploy.sh script | 20:39 |
| myoung | hrm...i have a theory | 20:39 |
| weshay | adding --clean | 20:40 |
| weshay | ya | 20:40 |
| myoung | i wonder if the pip running in the context of quickstart.sh isn't honoring the change we put in to handle the concurrency pip bug, so it's looking in the OLD pip cache location (the default one) ~/.cache/pip, instead of the location overridden by the jenkins global env variable that makes it ~/.cache/{executor_number}/pip (in this case ~/.cache/0/pip | 20:41 |
| weshay | myoung, in fact.. --bootstrap should be replaced w/ --clean | 20:41 |
| myoung | that would make it find an old version of the file | 20:41 |
| weshay | --bootstrap just installs the rpms | 20:41 |
| weshay | https://review.openstack.org/583736 | 20:44 |
| myoung | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/115/injectedEnvVars/ has XDG_CACHE_HOME=/home/rhos-ci/.cache/$EXECUTOR_NUMBER (correct), but on the slave i was on if somehow quickstart.sh's pip install wasn't seeing/using that override, it would abs()-ly find the older verion of oooq-internal.yml | 20:44 |
| myoung | weshay: that patch makes sense, however in this case (internal schtuff) I think the egg is being unpacked here: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/ci-scripts/prep-internal-rhel.sh?id=39976b947b01b1fb92e14a91a347aea50711f12c#n39 | 20:45 |
| *** gkadam has joined #oooq | 20:45 | |
| weshay | myoung, ya.. internal def sets it up first | 20:46 |
| myoung | kind of want an option to have quickstart.sh bletch out all the env vars (with a --verbose flag) it knows about - to validate the XDG_CACHE_HOME theory | 20:47 |
| myoung | i guess we could nuke the old cache location and see if it populates it on the offline slave. care if I give it a whack? | 20:47 |
| myoung | (it shoulnd't...it's set env wide) | 20:47 |
| weshay | myoung, sure | 20:48 |
| *** agopi is now known as agopi|brb | 20:53 | |
| *** agopi|brb has quit IRC | 20:57 | |
| *** trown|lunch is now known as trown|outtypewww | 20:59 | |
| *** gkadam has quit IRC | 21:04 | |
| myoung | weshay: ruled out that theory. it's not ignoring XDG_CACHE_HOME, is using the right pip cache, but it does appear to be pulling it from the cache and picking up the older version. nuking the cache or --clean should be the fix. | 21:28 |
| * myoung has to dash to fetch a wee one | 21:29 | |
| weshay | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/config/environments/oooq-internal.yml/*view*/ | 21:29 |
| weshay | should work | 21:29 |
| * myoung looks quickly | 21:29 | |
| * myoung nods | 21:30 | |
| myoung | weshay: the --force-reinstall just forces a uninstall then reinstall (from cache). we need the --no-cache-dir 4 sure. | 21:31 |
| myoung | weshay: heh pip -vvv is spammy but made that clear | 21:31 |
| myoung | so "quickstart.sh --clean $theRest FTW" | 21:32 |
| *** myoung is now known as myoung|afk | 21:32 | |
| ssbarnea1 | pip cache isolation workaround works only in some cases, downstream i had several failures due to tox.ini not passing the custom cache to venvs. i did check the ticket but i got the impression that they already fix it by now in pip. | 22:10 |
| ssbarnea1 | but also cleaning the cache every week was another way to lower the risks. | 22:11 |
| panda|off | rfolco: merged | 22:19 |
| *** chem has quit IRC | 22:26 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci- (1 more message) | 22:30 |
| *** holser__ has quit IRC | 22:39 | |
| *** brault has quit IRC | 22:46 | |
| *** rlandy has quit IRC | 22:57 | |
| *** honza has quit IRC | 22:59 | |
| *** myoung|afk has quit IRC | 23:03 | |
| *** faceman has quit IRC | 23:03 | |
| *** myoung has joined #oooq | 23:03 | |
| *** faceman has joined #oooq | 23:05 | |
| *** tosky has quit IRC | 23:20 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!