Friday, 2018-06-22

*** vvaldez has joined #oooq00:48
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/56722400:49
*** tcw has quit IRC02:33
*** tcw has joined #oooq02:37
hubbotFAILING CHECK JOBS on master: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/56722402:50
*** skramaja has joined #oooq02:59
*** vvaldez has quit IRC03:07
*** ykarel|away has joined #oooq03:35
*** udesale has joined #oooq03:45
*** ykarel|away is now known as ykarel04:02
*** ratailor has joined #oooq04:28
*** jaganathan has joined #oooq04:28
*** pgadiya has joined #oooq04:46
*** pgadiya has quit IRC04:46
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, master: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/56044504:50
*** ratailor has quit IRC04:54
*** ratailor_ has joined #oooq04:54
*** quiquell|off is now known as quiquell|rover05:37
*** bogdando has joined #oooq05:48
*** brault has quit IRC06:15
*** brault has joined #oooq06:23
quiquell|roverykarel: Do you know what do we use opstools for ?06:29
*** sanjayu_ has joined #oooq06:31
*** ykarel_ has joined #oooq06:34
*** ykarel has quit IRC06:36
quiquell|roversshnaidm: Do you know what do we use opstools for ?06:38
ykarel_quiquell|rover, no06:38
*** ykarel_ is now known as ykarel06:38
*** holser_ has joined #oooq06:48
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, master: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/56044506:50
*** quiquell|rover is now known as quique|rover|afk06:57
*** Goneri has joined #oooq07:05
*** ccamacho has joined #oooq07:06
*** tesseract has joined #oooq07:11
*** tosky has joined #oooq07:18
*** amoralej|off is now known as amoralej07:39
*** quique|rover|afk is now known as quiquell|rover07:42
*** zoli|gone is now known as zoli07:53
*** kopecmartin has joined #oooq07:55
*** d0ugal_ has quit IRC08:03
*** d0ugal has joined #oooq08:04
sshnaidmquiquell|rover, we install collectd from there, maybe something else08:05
*** sshnaidm is now known as sshnaidm|off08:05
*** ykarel is now known as ykarel|lunch08:23
*** skramaja has quit IRC08:33
*** dtantsur|afk is now known as dtantsur08:39
*** anande has joined #oooq08:42
hubbotFAILING CHECK JOBS on master: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/56722408:50
quiquell|roverchandankumar, tosky: Do you know about this issue ? http://logs.openstack.org/32/576632/2/gate/tripleo-ci-centos-7-scenario002-multinode-oooq-container/e50b49b/logs/undercloud/home/zuul/tempest/tempest.html.gz08:50
quiquell|roverIt's a rediecting loop08:50
*** ratailor__ has joined #oooq08:53
*** ratailor_ has quit IRC08:57
*** zoli is now known as zoli|lunch09:03
*** ykarel|lunch is now known as ykarel09:07
*** ykarel has quit IRC09:14
*** anande has quit IRC09:24
*** ykarel has joined #oooq09:49
toskyquiquell|rover: uhm, no; can you pinpoint when it started? Maybe it's easy to find the commit that changed the behavior09:54
quiquell|rovertosky: Sure09:54
*** d0ugal has quit IRC09:55
*** d0ugal has joined #oooq09:57
quiquell|rovertosky: I think it was transitory, now I see it passing at zuul's gate09:57
toskyoh, oki09:57
toskythere were no recent changes in the telemetry_tempest_plugin09:57
quiquell|rovertosky: The redirection loop could be some reouting problem09:57
quiquell|rovertosky: That got fixed at infra or similar09:57
toskyright09:57
quiquell|rovertosky: thanks btw09:58
toskynp (when there is no effort is easy :)10:02
*** jaganathan has quit IRC10:41
*** zoli|lunch is now known as zoli10:49
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/56722410:50
*** jaganathan has joined #oooq10:51
*** honza has quit IRC10:55
*** jaganathan has quit IRC10:58
*** honza has joined #oooq11:00
*** honza is now known as Guest7176311:00
*** quiquell|rover has quit IRC11:17
*** jaganathan has joined #oooq11:30
*** jaganathan has quit IRC11:30
*** jaganathan has joined #oooq11:31
*** udesale_ has joined #oooq11:32
*** udesale_ has quit IRC11:33
*** udesale_ has joined #oooq11:34
*** udesale has quit IRC11:34
*** quiquell has joined #oooq11:42
*** udesale_ has quit IRC11:42
*** quiquell is now known as quiquell|rover11:42
*** jaganathan has quit IRC11:44
*** jaganathan has joined #oooq11:45
*** Goneri has quit IRC11:50
*** panda|off is now known as panda11:56
*** trown|outtypewww is now known as trown12:00
pandaquiquell|rover: are failures in undercloud install known ? http://logs.openstack.org/79/576879/17/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/6acdf7c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-06-22_10_37_3612:05
quiquell|roverpanda: Yep, we have identified the issue also12:06
pandaquiquell|rover: ok, thanks12:06
quiquell|roverhttps://bugs.launchpad.net/tripleo/+bug/177820112:06
openstackLaunchpad bug 1778201 in tripleo "os-refresh-config undercloud install Error: Evaluation Error: Error while evaluating a Function Call, pick(): must receive at least one non empty" [Critical,In progress] - Assigned to Quique Llorente (quiquell)12:06
*** amoralej is now known as amoralej|lunch12:07
pandatrown: line 91 of the etherpad, the playbook is trying to resolve the hosts: all, peers, children, hosts12:20
pandazuuul inventory may not work as it is.12:21
*** moguimar has quit IRC12:21
trownpanda: not sure I understand... I have used that inventory successfully12:22
trownpanda: trying to figure out your missing overcloud group thing... in my libvirt env that is there12:22
*** moguimar has joined #oooq12:23
pandatrown: mmh, what version of ansible are you using ?12:23
trownansible-2.4.0.0-5.el7.noarch12:24
pandamore or less the same ... whay does it not understand the inventory then ..12:25
trownpanda: what error are you hitting specifically... maybe some yaml formatting thing?12:25
pandafatal: [all:]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname all:: Name or service not known\r\n", "unreachable": true}12:25
pandafatal: [peers:]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname peers:: Name or service not known\r\n", "unreachable": true}12:25
trownthough I have been copy pasting from etherpad...12:25
pandafatal: [children:]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname children:: Name or service not known\r\n", "unreachable": true}12:26
pandafatal: [hosts:]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname hosts:: Name or service not known\r\n", "unreachable": true}12:26
pandaduring facts gathering12:26
*** jaganathan has quit IRC12:27
trownpanda: hmm... mine is definitely only trying to gather facts from primary and secondary12:28
trownpanda: could you paste your inventory?12:28
*** rlandy has joined #oooq12:28
pandatrown: http://paste.openstack.org/show/724108/12:29
quiquell|roverrlandy: Hello12:30
rlandyquiquell|rover: hello12:31
quiquell|roverrlandy: Did you say something about OVB create ?12:31
rlandyquiquell|rover: I have been working on an doff on getting the last pieces of the rhos-13 gate going ...12:31
*** jaganathan has joined #oooq12:32
quiquell|roverrlandy: I cleaned up central ci there were resources problems12:32
rlandyI noticed last night that the stack create was in continuous create and never completed12:32
trownpanda: weird... if I diff your file and mine every line is different... but I cant tell what is different...12:32
rlandyI worked in the stage tenant12:32
rlandyso I wanted to know if you saw that in the regular tenant12:32
rlandyI see you have some passing jobs today so maybe it's fine now12:33
rlandyI'll recheck12:33
pandatrown: debugging ansible, my version is missing the yaml plugin12:33
quiquell|roverrlandy: Ok12:33
pandatrown: WARNING]:  * Failed to parse /tmp/tmp.71Du0ufqEt/.quickstart/zuul-hosts.yaml with ini plugin: /tmp/tmp.71Du0ufqEt/.quickstart/zuul-hosts.yaml:5: Expected key=value host variable assignment, got: null12:33
quiquell|roverrlandy: Let me know if it works now for you12:33
rlandyI was worried the upgrade may have impacted it12:33
rlandybut it look slike your stuff is rolling along12:34
pandatrown: so the inventoryis probably ok, I'll try to understand what's happening with my system12:34
trownpanda: oh, that would do it... parsing that file as .ini would not work :P12:34
trownnot sure how yaml plugin is installed... mine "just worked"12:34
quiquell|roverrlandy: The periodic-master-rdo_trunk-ovb-minimal-pacemaker-multiple-nics is super weird12:34
rlandyquiquell|rover: lol - that is one I did not set up :)12:35
rlandyyou still having trouble there>12:35
quiquell|roverrlandy: I mean the issue, the job is alright12:35
quiquell|roverrlandy: I have two runs now, one is taking the image correctly the next not12:35
quiquell|roverrlandy: I am starting to thing that other jobs running in the same slave are affecting it12:35
rlandyquiquell|rover: check which save each job is running on12:35
rlandythere are two slaves12:36
rlandypossible diff12:36
quiquell|roverrlandy: Two consecutives in the same slave12:36
trownpanda: btw overcloud group is being added to the inventory very early in my reproducer env... before/during undercloud-setup ... trying to figure out why that did not happen in CI12:36
rlandyquiquell|rover: we need to make sure the workspace is deleted properly between the runs12:36
rlandyand that there is no memory from one job to the other12:36
*** kopecmartin has quit IRC12:39
*** jaganathan has quit IRC12:41
*** jaganathan has joined #oooq12:41
pandatrown: ok, got some other hiccups with yaml and neede to recopy and paste, but works now12:41
trownpanda: in your test job something is making the subnode fail to get added to the inventory at all http://logs.openstack.org/79/576879/16/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/53638a7/logs/quickstart_files/hosts.txt.gz   ...    grep "Add subnode to ansible inventory"  http://logs.openstack.org/79/576879/16/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/53638a7/logs/quickstart_install.log  ... there12:46
trown is no "changed" entry for it12:46
trownpanda: ah found it https://github.com/openstack/tripleo-quickstart/blob/master/roles/tripleo-inventory/tasks/inventory.yml#L4512:47
pandatrown: running toci script, I'll check for that error12:47
pandatrown: also, the upstream patch fails because there is not overcloud group, added it on the inventory in nodeset, but there's an error in undercloud install and I can't make progress there12:48
trownpanda: ya I think the lack of overcloud group is not solved by your patch... that is what I was looking at12:49
trownpanda: because the zuul inventory is not used for our ansible runs from quickstart12:49
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/56722412:50
trownpanda: issue is that the subnode does not get added to the inventory via our quickstart inventory role ... something is not working with populating the /etc/nodepool/sub_nodes_private file in your legacy playbook12:50
pandammmhh12:51
pandatrown: the playbook was copied directly from the upstream legacy/pre.yaml playbook12:51
trownpanda: commented in the review... I think it might be the groups['subnodes'] in that task12:53
*** ratailor__ has quit IRC12:53
trownpanda: ya we have no subnodes group in our custom nodeset12:54
*** ratailor__ has joined #oooq12:54
trownpanda: so that task that sets /etc/nodepool stuff doesnt run12:54
pandatrown: ok, so I should add subnodes: hosts: -secondary ?12:54
pandatrown: or both12:55
trownpanda: looking for the the legacy nodeset to see12:55
trownpanda: just secondary https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/zuul.d/nodesets.yaml#L184-L18612:56
pandaok, replacing also zuul-cloner with the cp commands directly12:57
rlandyhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-13-ci-rhos-ovb-minimal-pacemaker-public-bond/ is half the trouble12:59
rlandyquiquell|rover: ^^ disabling12:59
rlandytaking resources12:59
quiquell|roverrlandy: Ahh ok13:00
quiquell|roverrlandy: Thanks13:00
*** skramaja has joined #oooq13:12
pandatrown: undercloud install failed in libvirt reproducer as expected ...13:13
pandatrown: the rest worked, with some adjustments to the system (addded at the top of the etherpad)13:15
trownpanda: ya my undercloud install failed too13:21
*** myoung|off is now known as myoung|ruck13:22
*** amoralej|lunch is now known as amoralej13:22
trownpanda: how did you enable the ansible yaml plugin13:23
myoung|rucko/  good morning13:24
pandatrown: in /etc/ansible/ansible.cfg, section [inventory] uncomment the plugins13:24
trownpanda: hmm in my ansible config it has yaml in the default inventory plugins actually... but good to add that to the etherpad as possible issue and resolution13:25
pandatrown: fresh centos install13:25
pandatrown: fresh ansible install too, started more or less from scratch13:25
myoung|ruckquiquell|rover: not sure (yet) about context re: opstools, but it's https://github.com/centos-opstools a set of tools to configure monitoring and such.  THere are some tripleo specific integrations as well to set up log collection and forwarding, sensu / monitoring, healthchecks, and more recently afaik skydive project was added as well.    I haven't played with it recently but I always thought http://opstools-ansible.readthedocs.io/en/13:27
myoung|rucklatest/tripleo_integration.html looked like a good target for future CI sprints vs. how we do log collection now (e.g. set up a node for the deployment and fwd logs to a micro elk stack)13:27
*** vvaldez has joined #oooq13:36
myoung|ruckrasca: ^^ maybe we could prototype this on your BMU job, would solve nicely the problem you were seeing yesterday w.r.t. how we collect logs (instantly searchable vs. "have to know what you're looking for, or download everything slowly"13:36
quiquell|roverWe need some +2 to unblock CI https://review.rdoproject.org/r/#/c/14432/13:38
quiquell|roverIt's pinning puppet stdlib to good version13:39
myoung|ruckquiquell|rover: note 17 mins ago https://github.com/puppetlabs/puppetlabs-stdlib/pull/890 they might revert as well13:39
*** ykarel has quit IRC13:40
*** ykarel has joined #oooq13:40
quiquell|rovermyoung|ruck: Cool, we can unpin after revert is merged13:41
quiquell|rovermyoung|ruck: Now CI is stuck there13:42
*** vvaldez has quit IRC13:43
*** tcw has quit IRC13:45
*** vvaldez has joined #oooq13:51
*** bogdando has quit IRC13:51
*** agopi has joined #oooq13:54
myoung|ruckquiquell|rover: o/ back and avail if you are13:54
*** tcw has joined #oooq13:55
quiquell|roverok calling your bj13:56
*** vinaykns has joined #oooq13:56
myoung|ruck%gatestatus13:57
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/56722413:57
quiquell|roverhubbot: Is lying13:57
hubbotquiquell|rover: Error: "Is" is not a valid command.13:57
*** moguimar has quit IRC14:03
*** moguimar has joined #oooq14:04
*** ratailor__ has quit IRC14:11
*** ykarel is now known as ykarel|away14:14
*** ykarel|away has quit IRC14:16
*** udesale has joined #oooq14:21
*** skramaja has quit IRC14:34
*** jaganathan has quit IRC14:39
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/56722414:50
*** vvaldez has quit IRC14:52
*** vvaldez has joined #oooq14:52
*** moguimar has quit IRC14:53
*** vvaldez has quit IRC14:56
*** quiquell|rover is now known as quiquell|off15:01
*** sanjayu_ has quit IRC15:06
*** sanjay__u has joined #oooq15:11
*** ykarel|away has joined #oooq15:35
*** panda is now known as panda|off15:59
*** tesseract has quit IRC16:02
rlandyrasca: helo - ha question for you16:14
rlandyhello16:14
*** holser_ has quit IRC16:17
*** ykarel|away has quit IRC16:25
*** zoli is now known as zoli|gone16:29
*** dtantsur is now known as dtantsur|afk16:37
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/56722416:50
*** trown is now known as trown|lunch17:03
*** amoralej is now known as amoralej|off17:33
*** agopi has quit IRC17:48
*** agopi has joined #oooq17:49
*** trown|lunch is now known as trown18:04
hubbotFAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-scenario003-multinode-oooq-container,  (1 more message)18:50
*** yolanda__ has joined #oooq19:12
*** yolanda_ has quit IRC19:16
*** yolanda__ has quit IRC19:18
*** udesale has quit IRC19:35
*** atoth has quit IRC19:42
*** udesale has joined #oooq19:49
*** udesale has quit IRC19:58
hubbotFAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-scenario003-multinode-oooq-container,  (1 more message)20:50
*** rfolco_ has quit IRC21:00
*** trown is now known as trown|outtypewww21:07
*** holser_ has joined #oooq21:10
*** yolanda has joined #oooq21:15
*** holser_ has quit IRC21:37
*** hamzy_ has joined #oooq22:06
*** hamzy has quit IRC22:08
hubbotFAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-scenario003-multinode-oooq-container,  (1 more message)22:50
*** myoung has joined #oooq22:53
*** pliu_ has joined #oooq22:54
*** rnoriega_ has joined #oooq22:54
*** jjoyce has joined #oooq22:54
*** _atoth has joined #oooq22:54
*** rasca- has joined #oooq22:54
*** lhinds|out has joined #oooq22:55
*** lhinds has quit IRC22:56
*** weshay_PTO has quit IRC22:57
*** weshay has joined #oooq22:57
*** pliu has quit IRC22:57
*** rasca has quit IRC22:57
*** faceman has quit IRC22:57
*** fuzzball81 has quit IRC22:57
*** rnoriega has quit IRC22:58
*** jschluet has joined #oooq22:58
*** jschlueter has quit IRC22:58
*** myoung|ruck has quit IRC22:58
*** myoung is now known as myoung|off23:03
*** rlandy has quit IRC23:05
*** vinaykns has quit IRC23:12
*** vvaldez has joined #oooq23:27
*** tosky has quit IRC23:41

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!