Tuesday, 2018-10-23

*** saneax has quit IRC00:13
*** saneax has joined #oooq00:14
*** ssbarnea has joined #oooq00:22
*** gouthamr has joined #oooq00:47
*** dmellado has joined #oooq00:51
*** matbu has quit IRC00:57
*** matbu has joined #oooq00:58
*** hubbot1 has joined #oooq00:59
*** rlandy has quit IRC01:27
*** saneax has quit IRC01:52
*** saneax has joined #oooq01:53
*** sanjayu_ has joined #oooq01:54
*** saneax has quit IRC01:55
*** sanjayu_ has quit IRC01:58
*** sanjayu__ has joined #oooq01:58
*** sanjayu__ has quit IRC02:00
*** saneax has joined #oooq02:01
*** saneax has quit IRC02:06
*** saneax has joined #oooq02:06
*** saneax has quit IRC02:12
*** sanjayu_ has joined #oooq02:12
*** sanjayu_ has quit IRC02:25
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb- (3 more messages)02:35
*** saneax has joined #oooq03:28
*** sanjayu_ has joined #oooq03:30
*** saneax has quit IRC03:33
*** udesale has joined #oooq03:58
*** dsneddon has quit IRC04:11
*** rfolco|rover has quit IRC04:30
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb- (3 more messages)04:35
*** jfrancoa has joined #oooq05:08
*** ratailor has joined #oooq05:18
*** quiquell|off is now known as quiquell05:40
*** dsneddon has joined #oooq06:25
*** holser_ has joined #oooq06:27
*** holser_ has quit IRC06:28
*** holser_ has joined #oooq06:29
*** ccamacho has joined #oooq06:31
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb- (3 more messages)06:35
*** jaosorior has quit IRC07:02
*** jaosorior has joined #oooq07:05
jaosorioranybody else getting this issue in quickstart http://paste.openstack.org/show/732687/ ?07:14
*** amoralej|off is now known as amoralej07:20
quiquelljaosorior: could be releated to the python3 migration07:33
quiquelljaosorior: https://review.openstack.org/#/c/612289/07:34
quiquelljaosorior: ssbarnea is fixing it07:34
jaosoriorgot it07:38
jaosorioralthough I'm running quickstart from fedora2807:38
quiquelljaosorior: We are now preparing stuff to start hacking around for the new fedora28 job to work07:39
*** sshnaidm|afk is now known as sshnaidm|pto07:41
*** bogdando has joined #oooq07:42
*** tosky has joined #oooq07:57
*** dtantsur|afk is now known as dtantsur08:22
ssbarneaquiquell jaosorior marios: this is supposed to fix the py3 detection fault in qs.sh08:26
ssbarneahttps://review.openstack.org/#/c/612471/08:26
jaosoriorssbarnea: commented.08:29
*** hubbot1 has quit IRC08:30
*** gouthamr has quit IRC08:32
ssbarneais rdo cloud down again, I am no longer able to access my f28 machine08:33
quiquellssbarnea: same issue here08:34
quiquellssbarnea: commented too08:34
*** dmellado has quit IRC08:34
quiquellssbarnea: Will try it08:34
ssbarneai think that soon I will do something nasty, I will add a monitor bot for my rdo machines, when one goes down, it will start complaining on irc every 30mins.08:39
ssbarneabtw, can someone explain me the selinux python bindings issue? i am curious why we. need the cp step.08:40
quiquellssbarnea: I imaging your house full of goldberg machines doing all kind of stuff :-)08:40
quiquellssbarnea: At fedora28 with selinux activated found impossible to run for example ansible copy module stuff08:40
quiquellssbarnea: It complains about python-libselinux not installed but installing it does not help08:41
quiquellssbarnea: this is running ansible over python3 with selinux activated08:41
ssbarneaquiquell: you are right, i think i am about to reach >100 MAC addresses in my home lan.08:41
quiquellssbarnea: I think the selinux was kind of mix of ansible-fedora28 bug or the like08:42
ssbarneaquiquell: isn't this caused by creation of venv without --site-packages?08:42
quiquellssbarnea: In fact I found your name at a github issue around this08:42
quiquellssbarnea: Talking about normal running without virtualenv08:42
chandankumararxcruz: Hello08:49
chandankumararxcruz: you are looking into this card https://trello.com/c/qiKrS3Nc/783-cixlp1797931tripleociproa-tempest-pip-install-fails-on-rocky-minimal-config ?08:50
arxcruzchandankumar: hello, yes08:50
chandankumararxcruz: ack, thanks :-)08:50
ssbarneaquiquell: if you can find the bug it would be great, i do not remember it and I wonder if we cannot find a better solution for it. current version is not really portable.08:55
*** dsneddon has quit IRC08:55
*** dsneddon has joined #oooq08:58
quiquellssbarnea: This is it https://github.com/ansible/ansible/issues/3434008:58
*** dtrainor has quit IRC08:59
quiquellssbarnea: This is on the gates https://review.openstack.org/#/c/612289/09:04
quiquellssbarnea: Trying the os test with docker images of all the distros09:07
*** gouthamr has joined #oooq09:11
*** dmellado has joined #oooq09:18
*** tosky has quit IRC09:24
*** tosky has joined #oooq09:24
*** hubbot1 has joined #oooq09:33
arxcruzchandankumar: https://tree.taiga.io/project/tripleo-ci-board/issue/244 created yesterday :)09:44
*** quiquell is now known as quiquell|brb09:50
*** udesale has quit IRC09:51
*** udesale has joined #oooq09:52
jaosoriorssbarnea: the "correct python3 detection logic" patch didn't work for m10:03
jaosoriorme10:03
jaosoriorjust tested it in fedora 2810:03
jaosoriorI still get the "Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!" error10:03
jaosoriorthis patch https://review.openstack.org/#/c/612289/ seems to be the solution to the issue, even though it means reverting that patch10:04
*** quiquell|brb is now known as quiquell10:08
*** rfolco has joined #oooq10:11
*** ssbarnea_ has joined #oooq10:16
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb- (3 more messages)10:36
*** apetrich has quit IRC10:38
*** dtrainor has joined #oooq10:48
*** apetrich has joined #oooq10:54
ssbarneadid anyone create an issue about numpy (14mb wheel) being installed by qs by default? no wonder why out builds are slow.10:55
rfolcolaunchpad sucks11:00
rfolcoan our filling a bug and... crashes11:01
rfolcohour*11:01
ssbarneaquiquell: regarding selinux it was what I was expecting: we do create virtualenv withouyt --system-site-packages switch by default so no reason to wonder why is not there.11:08
ssbarneawith  --system-site-packages it works without any dirty hacks11:08
quiquellssbarnea: And I see the issue without creating virtualenv11:09
quiquellssbarnea: But that's just at my laptop11:10
ssbarneaquiquell: probably you do not have the bindings installed, which is another issue.11:10
quiquellssbarnea: Yep I installed it but nah don't worry is not a big deal11:10
ssbarneanow the big question is if we have other reasons for not using site packages by default?11:11
*** chem has quit IRC11:11
quiquellssbarnea:11:11
*** chem has joined #oooq11:12
quiquellssbarnea: https://tree.taiga.io/project/tripleo-ci-board/task/248?kanban-status=144727611:12
quiquellssbarnea: openstack infra add a exclude at dnf.conf11:12
quiquellssbarnea: for setuptools, virtualenv and pip11:12
quiquellssbarnea: they are supose to be already in the image so we don't have to care about them11:13
quiquellssbarnea: But don't worry this is part of the f28 job stuff I think11:13
ssbarneaquiquell i will adapt the selinux hack for the moment, but we still have to fix it correctly.11:14
*** udesale has quit IRC11:14
quiquellssbarnea: ack11:15
*** tosky has quit IRC11:18
*** tosky has joined #oooq11:18
*** rfolco is now known as rfolco|rucker11:23
*** panda is now known as panda|lunch11:27
*** ratailor has quit IRC11:29
ssbarneabrowbeat as req is a huge PITA: browbeat 0.0.1.dev1283 has requirement ansible==2.4.1, but you'll have ansible 2.5.7 which is incompatible.11:41
ssbarneathis is the 2nd issue caused by browbeat, after the numpy one.11:41
*** amoralej is now known as amoralej|lunch12:09
rfolco|ruckerssbarnea, would you please have a look at https://bugs.launchpad.net/tripleo/+bug/1799431 ? I have the impression the pkg abstraction patch you wrote might a cause. Wanted to hear from you.12:12
openstackLaunchpad bug 1799431 in tripleo "quickstart venv: IOError: [Errno 26] Text file busy" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco)12:12
quiquellrfolco|rucker: we have a fix for that in previous rr sessions12:14
quiquellhttps://review.openstack.org/#/c/602347/12:14
quiquellUps looks different12:15
rfolco|ruckerlooks the same!!!12:15
rfolco|ruckerthis is what I was looking for yesterday12:16
ssbarnearfolco|rucker: looking now.12:16
rfolco|ruckerah yeah chandankumar filed that bug12:17
chandankumarsorry which bug?12:18
rfolco|ruckerhttps://bugs.launchpad.net/tripleo/+bug/179237512:18
openstackLaunchpad bug 1792375 in tripleo "[master] multinode noop job failure giving IOError: [Errno 26] Text file busy: '/home/zuul/workspace/.quickstart/bin/python'" [Critical,In progress] - Assigned to chandan kumar (chkumar246)12:18
rascasofthey folks, can we make this move https://review.openstack.org/#/c/610087/ ? Can someone workflow it?12:18
chandankumarrfolco|rucker: https://review.openstack.org/60234712:18
chandankumarfix is there12:18
rfolco|ruckerthank you chandankumar12:18
chandankumarrfolco|rucker: but i didnot get the panda comment12:19
chandankumarrfolco|rucker: feel free to take it12:19
rfolco|ruckernobody understands what panda|lunch says12:20
quiquellrfolco|rucker: :_D12:21
rfolco|ruckerchandankumar, I believe he means "print something indicating that venv is being reused"12:21
rfolco|ruckermore verbose..12:21
ssbarnearfolco|rucker replied on the bug, likely CI/Jenkins issue.12:21
chandankumarrfolco|rucker: updating the patvhc12:22
rfolco|ruckerssbarnea, thanks for the update, yes, this is about reusing venv --> https://review.openstack.org/60234712:23
ssbarnearfolco|rucker and why the... we didn't merge it? it is two month old.12:27
quiquellssbarnea: Yep... openstack latencies :-(12:27
quiquellssbarnea: ansible question12:28
ssbarneai am going to implement panda remark as he downvoted, but i will lose the votes.12:28
quiquellssbarnea: using 'shell' module, rewrites PATH variable ?12:28
quiquellssbarnea: You have to explicitly say environment... ?12:28
rfolco|ruckerquiquell, what tentant runs for this job ?12:29
rfolco|ruckerhttps://logs.rdoproject.org/51/611551/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/961ac9b/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-10-23_08_23_4312:29
rfolco|ruckergot this question on rhos-ops12:29
*** rlandy has joined #oooq12:29
quiquellrfolco|rucker: https://logs.rdoproject.org/51/611551/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/961ac9b/zuul-info/inventory.yaml12:29
quiquellrdo-cloud-tripleo12:29
quiquell38.145.35.7912:30
ssbarneaquiquell: shell runs isolated from your calling env, for obvious reasons, ones of them being that "localhost" is not a sub-shell, as it may even use SSH to connect to your own localhost.12:30
rlandymarios: hello12:30
ssbarneaso, with ansible you do not inherit anything unless you do it manually. some would consider it a bug, in this case I consider it a feature12:30
quiquellssbarnea: ack12:30
rfolco|ruckerquiquell, thx12:30
quiquellssbarnea: Yep better explicit12:31
*** udesale has joined #oooq12:31
mariosrlandy: o/12:32
rlandymarios: what's the latest on the containerized upgrades job?12:33
chandankumarssbarnea: quiquell please have a look at https://review.openstack.org/#/c/612618/1/roles/validate-tempest/tasks/stackviz.yml@45 particular line I am doing something wrong with status code12:33
ssbarneasshnaidm|pto marios rlandy : please revote on https://review.openstack.org/#/c/602347 so we don't rediscover it in another two months from now.12:33
quiquellssbarnea: I think we have an issue with python3-openstack12:35
ssbarneaquiquell only one?12:35
quiquellssbarnea: Was expecting it...12:35
quiquell:-)12:35
quiquellssbarnea: They just provide openstack-3 command not openstack and we don't want to use pip version of the o penstackclient I suppose12:36
mariosrlandy: nothing i spent ages staring at logs/but couldn't find some rational explanation12:36
quiquellssbarnea: We want to use DLRN stuff12:36
quiquellssbarnea: Going to ask at infra12:36
mariosrlandy: i tried investigating whether the depends-on the original patch was the issue but tripleo-common is being updated in bot cases pass/fail12:36
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb- (3 more messages)12:36
rlandymarios : so, I don't know much about how these upgrades jobs are working12:37
mariosrlandy: if we can't come up with anything we can try to merge the revert revert but it is not cool that we don't know what is happening12:37
rlandymarios: did my notes make any sense?12:37
rlandymarios: I am not sure how the containers get used in the upgrade12:37
rlandybut I can see that the ones that are there in /home/zuul are not referenced12:38
chandankumarssbarnea: https://src.fedoraproject.org/rpms/python-openstackclient/blob/f28/f/python-openstackclient.spec#_224 only openstack openstack-2 or openstack-python-version is found for f28 no openstack-312:38
rlandyI can tell how the change gets into the other jobs12:38
chandankumarsorry there is openstack-3 https://src.fedoraproject.org/rpms/python-openstackclient/blob/f28/f/python-openstackclient.spec#_24012:39
mariosrlandy: i saw your comment about the containers but i wasn't sure what the connection with the error is ... the problem was the templates/generated ansible playbooks are referencing something which is 'not defined' (but it should be via tripleo-common)12:39
rlandymarios: I was comparing tripleo-ci-centos-7-undercloud-containers12:40
rlandymarios: I think what we are trying to solve is why the change was not included and therefore the tests passed12:41
rlandynot why there was a failure when the change was actually picked up12:41
mariosrlandy: but the change _was_ incldued in the sense that that is why we get the error.12:41
rlandy^^ is that correct?12:41
mariosrlandy: the error was literally "foo not defined " where foo was being refereced in the change12:41
rlandymarios: my understanding was that if there was a failure, it should have happened in the change job ... https://review.openstack.org/#/c/611800/12:43
rlandywhich it did not12:43
rlandyit shows up once the change is merged12:43
rlandymarios: ^^ if that is not the correct understanding, can we bj for 5 mins to clarify?12:45
mariosrlandy: sure lets do that. what i meant was in the original job the change was being included since you can find ironic_api_short_bootstrap_node_name in http://logs.openstack.org/18/610018/9/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/2042382/logs/undercloud/home/zuul/undercloud-ansible-A9y0CL/Undercloud/upgrade_tasks.yaml.txt.gz and that causes the error. the missing thing is the depends-on in tripleo-common, apparently12:47
*** trown|outtypewww is now known as trown12:47
mariosrlandy: my bluejeans: https://redhat.bluejeans.com/766192537312:47
*** amoralej|lunch is now known as amoralej12:50
ssbarneaquiquell : if you can help me test https://review.openstack.org/#/c/612677/ it would be great.12:56
quiquellssbarnea: Sure12:57
quiquellssbarnea: I am the QE12:57
ssbarneai also want to swap the use of bash function python_cmd with a variable, anyone against?12:57
ssbarneait does not make sense for me to run that code every time, in the end we only need python executable, same remark applies to pkg_mgr one.12:57
quiquellssbarnea: Yep, and function are easier to mock just to test stuff12:58
rfolco|ruckerssbarnea, I saw your patch failed to merge13:03
rfolco|ruckerhttp://logs.openstack.org/91/610491/9/gate/tripleo-ci-centos-7-containers-multinode/f54eb7e/job-output.txt.gz#_2018-10-23_03_33_30_24492613:03
rfolco|ruckerwondering if you know why13:04
mariosrlandy: https://review.openstack.org/#/c/611526/13:06
ssbarnearfolco|rucker : this looked like a broken mirror that didn't had the pbr on it, PITA. I wonder if pip can use multiple mirrors with fallbacks so we don't get a broken build just because one mirror is messed.13:07
ssbarnearfolco|rucker or at least this is what I seen on th ebuild log.13:07
rfolco|ruckeryeah, I am just thinking what could be alternative mirror por pypi13:08
rfolco|ruckerthe package is there http://mirror.bhs1.ovh.openstack.org/pypi/simple/pbr/13:09
rfolco|ruckerso it looks like intermittent network issue13:09
rfolco|ruckeror mirror itself13:09
chandankumartime to go home, /me connect from there13:13
*** quiquell is now known as quiquell|lunch13:13
rfolco|ruckerquick question marios... https://bugs.launchpad.net/tripleo/+bug/1797918 --> is this something we can fix or infra?13:35
openstackLaunchpad bug 1797918 in tripleo "teclient returns failures when attempting to provision a stack in rdo-cloud" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)13:35
rfolco|ruckerI am not familiar with teclient13:35
mariosrfolco|rucker: we don't know for sure yet ... it _was_ happening when rdo cloud was down/issues but weshay also posted some debug patches sec lemme find13:36
mariostracked here  https://bugs.launchpad.net/tripleo/+bug/1797526 rfolco|rucker you might want to link those two bugs then for now?13:38
openstackLaunchpad bug 1797526 in tripleo "Failed to get power state for node FS01/02" [Critical,Triaged]13:38
*** agopi has quit IRC13:39
rfolco|ruckermarios, link=mention, or link=dup13:42
*** jfrancoa has quit IRC13:44
mariosrfolco|rucker: probably mention13:45
rfolco|ruckermarios, sweet13:47
*** jfrancoa has joined #oooq13:50
ssbarneado we have a way to find how often we got a message on our logs?13:51
ssbarneaI am curious how often we see something like distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse.... in our builds.13:52
*** sanjayu_ has quit IRC14:01
*** sanjayu_ has joined #oooq14:02
*** panda|lunch is now known as panda14:04
*** sanjayu_ has quit IRC14:09
*** quiquell|lunch is now known as quiquell14:10
quiquellssbarnea: centos7+python3 should be enough ?14:16
ssbarneaquiquell for my patch, probably better to test with more than this.14:17
quiquellssbarnea: ack14:17
*** huynq has joined #oooq14:18
quiquellssbarnea: Will test it in the morning14:20
*** jfrancoa has quit IRC14:26
*** quiquell is now known as quiquell|off14:29
ssbarnearfolco|rucker can i get a +W on https://review.openstack.org/#/c/610491/ ? it failed to merge due to a random pip bug.14:33
ssbarnea(pypi to be correct)14:33
rfolco|rucker[siri] ssbarnea did you mean 'rlandy' ? :)14:34
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb- (3 more messages)14:36
mariossorry wasn't sure you were around rfolco|rucker ;) (tripleo)14:39
rfolco|ruckernp thx14:39
rfolco|ruckertriple reminder14:39
rfolco|ruckerI Community Meeting starts now at https://bluejeans.com/5878458097 - agenda: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting ~L3214:40
rfolco|ruckerping call folks14:40
mariosrlandy: wana join?14:42
rfolco|ruckerpanda, ^14:43
rlandyjoining14:47
*** dsneddon has quit IRC14:56
rfolco|ruckermarios, sorry for communication problems in the call, thanks for covering it14:57
*** ccamacho has quit IRC14:57
mariosrfolco|rucker: np didn't do anything mate14:57
rlandymarios: https://review.openstack.org/#/c/612724/15:00
rlandywe can only hope15:01
mariosrlandy: ack updated the commit msg15:04
mariosrlandy: and shardy also just revoted15:04
*** hamzy has quit IRC15:04
rlandymarios: thanks15:05
ssbarneainteresting discovery: f28 has ansible 2.7 as rpm which means that we need to bump to it to assure a consistent version. this task by itself is not trivial.15:09
*** ccamacho has joined #oooq15:17
*** ccamacho has quit IRC15:17
*** huynq has quit IRC15:25
*** dtantsur is now known as dtantsur|brb15:32
*** kopecmartin is now known as kopecmartin|off15:33
*** hamzy has joined #oooq15:38
*** dsneddon has joined #oooq15:39
rlandyssbarnea: https://review.openstack.org/#/c/591540/ is that ready to merge?15:41
rlandyalex thinks so15:41
rlandyyou're the committer so checking15:42
*** agopi has joined #oooq15:48
*** rfolco|rucker is now known as rfolco|brb16:01
*** udesale has quit IRC16:06
*** agopi has quit IRC16:24
ssbarnearlandy yep, should be safe to merge.16:31
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb- (3 more messages)16:36
*** dsneddon has quit IRC16:37
rlandyok16:49
*** bogdando has quit IRC16:50
rlandyrfolco|brb: I have been out of the loop on ovb status16:54
rlandywhat's the latest? still te-broker issues?16:54
ssbarneai need help testing https://review.openstack.org/#/c/612677/ -- anyone up to to test quickstart.sh changes?17:00
*** holser_ has quit IRC17:05
rlandyssbarnea: they are tested in tripleo-quickstart-gate17:07
rlandyssbarnea: ^^ is that the fix for the python virtualenv problem?17:07
rlandyssbarnea: also bm uses quickstart.sh17:08
rlandycan app,y the patch there if it helps17:08
ssbarnearlandy: yep, this is supposed to fix the previously wrong use of python3 on centos.17:10
ssbarneait was merged and reverted because it used the onliner python3 detection which was incorrect.17:10
ssbarneanow we have 50 line version which seems to work, based on my tests on various versions of centos/fedora/rhel/...17:11
rlandyssbarnea; ok - I'll apply it to my bm job17:11
ssbarnearlandy but if you could try to run qs yourself (probably with -b and without, it would clearly help validating the logic)17:12
*** hamzy has quit IRC17:12
ssbarnearlandy if you want I can explain each line17:12
*** hamzy has joined #oooq17:12
*** dtantsur|brb is now known as dtantsur17:13
ssbarneafor example: checking for "virtualenv" command is not the portable (newer exec is virtualenv-3)17:13
rlandyhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-baremetal-dell_fc430_envB-single_nic_vlans/17:13
rlandy^^ failed before so if it works now, we will know17:13
rlandyadding your patch17:13
ssbarneaso i decided to run install_deps all the time (also fixes the bug where was failing to clone due to missing git)17:13
*** dsneddon has joined #oooq17:16
*** rfolco|brb is now known as rfolco|rucker17:16
rlandyssbarnea: see results running now in https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-baremetal-dell_fc430_envB-single_nic_vlans/14/console17:17
rlandylooking better17:17
rlandylet it go past the repo changes and then I'll vote17:17
rfolco|ruckerrlandy, yes te-broker bug still unresolved17:20
rlandy17:19:37 sudo: no tty present and no askpass program specified17:21
rlandyssbarnea: hmmm ... guess that is required now17:21
rlandyrfolco|rucker: is anything being done about it?17:21
rlandycan you post bug link?17:21
rfolco|ruckerrlandy, are we talking about same bug ? https://bugs.launchpad.net/tripleo/+bug/179791817:22
openstackLaunchpad bug 1797918 in tripleo "teclient returns failures when attempting to provision a stack in rdo-cloud" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)17:22
ssbarnearlandy: what causes the askpass msg? it was not there before?17:23
rlandyI know 0 will look into it17:24
ssbarneaok thanks17:24
rfolco|ruckerrlandy, I don't if this is related but when asking about the ntp issue [1] to rhos-ops I got the following response [2]17:28
rfolco|rucker[1] https://logs.rdoproject.org/51/611551/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/961ac9b/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-10-23_08_23_4317:28
rfolco|rucker[2] <kforde> rf0lc0, I had to remove the false ha router from the other controller17:29
*** dtantsur is now known as dtantsur|afk17:29
rfolco|ruckerrlandy, need you expertise here to come up with ideas on how to reproduce / test this issue17:29
rlandysec - phone17:31
*** agopi has joined #oooq17:34
*** agopi has quit IRC17:39
*** trown is now known as trown|lunch17:42
rlandyrfolco|rucker: sorry - was on a call - just looking into the failure from ssbarnea test and then I'll looking into te-broker17:44
rlandywhat happened to all our cores???17:44
rlandyand who let weshay go on pto anyways?17:45
rlandyssbarnea: I think it's the sudo install of deps17:46
rlandy17:19:37 Error: This command has to be run under the root user.17:46
rlandy^^ I don;t think that's workable17:47
ssbarnearlandy: qs needs to be able install deps17:49
ssbarnearlandy should I conclude that we never run install_deps on ci?17:50
ssbarnearlandy: yes, i can make it avoid sudo if deps are already installed, *still* our CI needs to also cover the case of those not being installed, right?17:51
rlandyssbarnea: so, depends on what the expectation is17:51
rlandyI can make the slave work with sudo17:52
rlandybut is that really a requirement for running qs?17:52
rlandyit can install deps in virtualenv17:52
ssbarneathis would be great, meanwhile I will make it not to need sudo unless it detects missing components17:52
ssbarneaso it will ask for sudo if you don't have git for example17:53
ssbarneafor stuff installable without sudo I will do it without17:53
rlandysudo true || $(package_manager) install sudo17:53
rlandyso if sudo is false17:53
rlandyyum/dnf install sudo17:53
rlandywhich does require root17:53
ssbarnearlandy: did you know that qs.sh -b was failing on machines without git? ... even if we have code inside for installing git.17:54
rlandyssbarnea: I think I did fall over that at some point17:54
ssbarneabut that code was not run17:54
rlandyso, git and sudo should be availble17:54
rlandyssbarnea: I can definitely get around this17:54
rlandybut we should determine what the requirement is17:55
* rlandy checks what's on the slave17:55
*** hamzy has quit IRC17:59
*** hamzy has joined #oooq18:00
rlandyssbarnea: sudo is installed18:02
rlandy[rhos-ci@dhcp150-212 ~]$ which sudo18:02
rlandy[rhos-ci@dhcp150-212 ~]$ sudo true18:02
rlandyWe trust you have received the usual lecture from the local System18:02
rlandyAdministrator. It usually boils down to these three things:18:02
rlandy    #1) Respect the privacy of others.18:02
rlandy    #2) Think before you type.18:02
rlandy    #3) With great power comes great responsibility.18:02
rlandyssbarnea: ^^ it's the check that causing the error18:02
rlandyit's not that sudo is not installed18:03
rlandythe user is not set up for it18:03
*** agopi has joined #oooq18:06
*** apetrich has quit IRC18:16
*** apetrich has joined #oooq18:17
*** amoralej is now known as amoralej|off18:19
*** hamzy has quit IRC18:23
*** hamzy has joined #oooq18:24
rlandyrfolco|rucker: ok - looking at the te-broker stuff now18:24
*** vinaykns has joined #oooq18:32
*** irclogbot_0 has joined #oooq18:35
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-multinode-1ctlr-featureset010, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb- (4 more messages)18:36
*** chandankumar is now known as chkumar|off18:37
*** weshay has joined #oooq18:38
rlandyhello weshay!18:41
rlandywoohoo more cores18:41
weshayrlandy, hey there :)18:41
weshayrlandy, send em my way :)18:41
rlandyweshay, upgrades18:41
weshayrlandy, we were in teaneck18:41
rlandyweshay: wow - nice orthodox neighborhood :)18:42
weshayrlandy, oh ya.. shabbat was like halloween in my neighborhood18:42
weshayeveryone out and about.. walking18:43
rlandyyep - here everything closes on saturday - even the public library18:43
ssbarneadoes anyone knows if the "venv" tox environment from tq is used anywhere?18:44
rlandyhmmm - not off hand18:44
rlandywe cab test it18:44
rlandycan18:44
rlandyssbarnea: you can use my test job if you like18:44
ssbarneanot wanting to remove it, just to partially repurpose it for easy testing18:44
rlandyit's red anyways18:44
ssbarneain few minutes I should have an update on my change, got lots of improvements.18:45
*** panda has quit IRC18:45
rlandyok - ping me to rerun18:45
rlandyweshay, : chem has some reviews - half of which18:45
*** panda has joined #oooq18:45
rlandyi have not gotten to18:46
rlandyhttps://review.openstack.org/#/c/604706/18:46
rlandyhttps://review.openstack.org/#/c/604736/18:46
weshayok.. /me gets some coffee and will dig in18:46
rlandywait ... if you call now ... there's more18:47
rlandyhttps://review.openstack.org/#/c/606996/ - needs w18:49
rlandyweshay: ^^ has two +2 ... and then there is this ... https://review.openstack.org/#/c/591540/ - I would like to see ci run - but if you're ok  with it, I'll vote18:50
weshayk19:00
rlandynvm - found a reason to -1 anyways19:00
* rlandy is in the -1 mode today19:00
rlandyweshay: do you know what is happening with https://review.openstack.org/#/c/612724/ and https://bugs.launchpad.net/tripleo/+bug/1798525/19:01
openstackLaunchpad bug 1798525 in tripleo "The undercloud-upgrades job is failing during the upgrade with "error was: 'ironic_api_short_bootstrap_node_name' is undefined" " [Undecided,Triaged] - Assigned to Marios Andreou (marios-b)19:01
rlandythere has been a revert and a revert revert19:02
rlandykind of confused why the correct version of tht was not picked up19:02
weshayoh ya..19:02
rlandymarios and I worked on that all yesterday19:02
weshayI'd have to look again but ya.. I remember that issue19:02
rlandywe decided to move the job out of gate19:02
rlandyand back to non-voting19:02
rlandymerge and see19:02
rlandyless damage19:03
rlandyok?19:03
*** agopi has quit IRC19:03
weshayya.. that is fine19:03
* rlandy breathes again19:03
rlandy2018-10-23 15:31:36.431454 | primary |   Could not find a version that satisfies the requirement netaddr>=0.7.18 (from -r requirements.txt (line 5)) (from versions: )19:13
rlandy2018-10-23 15:31:36.973380 | primary | No matching distribution found for netaddr>=0.7.18 (from -r requirements.txt (line 5))19:13
rlandyrfolco|rucker: ^^ latest error ovb is picking up19:14
rlandyseems to be creating the stack ok19:14
rlandyteclient returns failures when attempting to provision a stack in rdo-cloud logs a different error19:15
weshayrlandy, so reading through https://review.openstack.org/#/c/606996/11/scripts/emit_releases_file/emit_releases_file.py19:15
* rlandy isn;t even sure these days what we are debugging here19:15
weshayIt's not clear to me yet why dlrn-current is needed19:15
* rlandy looks19:15
weshayDEPLOY_NEWEST_HASH19:16
rlandymarios asked that question19:16
rlandyin comment on Patch Set 9:19:16
rlandy Line 50:19:17
rlandywhy current and not consistent at least where we have had a build for all packages19:17
rlandychem answers him19:17
ssbarneaweshay: is it normal for "qs.sh -b" to fail with "ERROR: You must specify a target machine." ? while  "qs.sh --install-deps" can freely endup with success.19:17
ssbarneainstall_deps is run by bootstrap, but bootstrap contains much more (and critical code)19:18
weshayssbarnea, ya19:18
ssbarneai want to test it, doing only bootstrapping, but the current behaviour prevents me from using its exit code.19:18
weshaystill want nuke a lot of that script19:19
rlandyssbarnea: you need a $VIRTHOST19:19
weshayssbarnea, I see ur point --bootstrap should probably have the same user experience as --install-deps19:19
ssbarneai just want to run the boostrap() part to test it, I do not care (for the moment) about the other part.19:20
ssbarneabased on terminology, seems resonable to belive the -b switch is supposed to run bootstrap() function and not much more than this.19:20
weshayrfolco|rucker, you around?19:22
rfolco|ruckerweshay, yes19:23
rfolco|ruckero/19:23
weshayrfolco|rucker, any updates on a master promotion?19:24
weshaywant to jump on blue?19:24
rfolco|ruckerok19:24
rfolco|ruckerweshay, in your room19:25
ssbarnearlandy: got https://review.openstack.org/#/c/612677/ updated, if you can retest it would be great.19:27
rlandyssbarnea: sure - let me rekick19:27
rlandyssbarnea: job 1519:28
weshayrfolco|rucker, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/617ef64/logs/19:31
weshayrfolco|rucker, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/9d9c06b/job-output.txt.gz#_2018-10-23_12_40_45_15896719:32
weshayhttps://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/9d9c06b/ara-report/result/d46fadb8-0f8f-46d0-bd07-5e4ed344167e/19:34
rlandyhmmm19:34
rlandy OSError: [Errno 17] File exists19:35
rlandyssbarnea,:^^19:36
rlandymaybe because we ran before to set up?19:36
*** trown|lunch is now known as trown19:36
ssbarnearlandy i suspect this is caused by a recursive symlink issue, i seen it recently and I suspect is caused by someone that did symlink python -> python3 or something similar.19:42
ssbarneawhen this happens virtualenv may get confused and the virtualenv created is a mess.19:43
rlandyssbarnea: would it help if I clean cache?19:43
ssbarneasure, please do.19:43
rlandyI can try that and run again19:43
ssbarneai do have the command and i am trying to replicate it locally, i suspect that the slave is a centos-7?19:44
ssbarneai not really clear from the execution. maybe I should add a banner to qs.sh that displays clearly detected platform.19:45
rlandyfedora19:45
rlandyslave probably needs an update19:45
ssbarnearlandy : somehow seems very similar to https://review.openstack.org/#/c/602347/ which I hope to merge soon.19:47
rlandyssbarnea: ok - if this run fails, I'll include that patch19:48
ssbarneano need, i already included it in mine.19:48
ssbarneabut this usually happens if you try to create or activate a venv from inside a venv.19:48
ssbarneaanyway, I need to go,... dinner and a *drink*, too much for today.19:49
*** agopi has joined #oooq19:50
rlandyhmm ... same failure19:57
rlandyout of luck19:57
*** ssbarnea_ has quit IRC20:36
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-multinode-1ctlr-featureset010, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb- (4 more messages)20:36
*** hamzy has quit IRC20:40
*** hamzy has joined #oooq20:40
weshayrlandy, https://review.openstack.org/#/c/610087/20:40
rlandylooking20:41
*** agopi has quit IRC20:44
weshayrlandy, did the patch to the te-broker merge?20:44
rlandyweshay: which patch20:44
rlandythere are lots of those these days :)20:45
rlandyweshay: speaking of which - why would we need a br-ex interface for standalone as in https://review.openstack.org/#/c/604736/16/roles/standalone-upgrade/defaults/main.yml?20:45
weshayrlandy, ya.. it did https://review.openstack.org/#/c/610113/120:45
weshayrlandy, that is the second interface in upstream jobs no?20:46
rlandyok - so then I un -1 your other patch20:46
rlandyweshay: well for multinode yes20:48
rlandywhen we do actually need a bridge20:48
weshayrlandy, so check out the standalone doc20:50
weshaythe interface used locally is eth120:51
weshayand that's a small diff in the reproducer as well20:51
rlandyweshay: ok - getting back to ovb on rdocloud,21:00
weshayrlandy, meh..21:00
rlandythe bugs relate to the stack creation21:01
weshayrlandy, if we had upshift and real bm results that would be ++21:01
rlandybut we are getting by that in a lot of jobs here21:01
rlandyupshift and bm stuck behind quickstart.sh failures21:01
weshayrlandy, https://docs.google.com/spreadsheets/u/1/d/1Rz6qxoqQ8Mmm2GokjaKrj6inr7RJ0R75WuqFdi8RThc/edit?usp=drive_web&ouid=10886484440192456634121:02
weshayrlandy, need help w/ upshift?21:02
rlandyweshay: two upshift blockers are of now ...21:02
rlandyquickstart.sh21:02
rlandythat ssbarnea is working on21:02
rlandyand21:03
rlandyissue of ara install21:03
rlandythat I think rfolco|rucker put in a patch yesterday for21:03
weshayssbarnea, pretty sure we need to just turn ara off for now21:03
rlandyI think that is what the patch did21:03
rlandyfor ara21:03
weshayah good21:03
rlandyquickstart still falling over21:03
* rlandy gets patch21:03
weshaythat's right.. I talked to rfolco|rucker about that before I left21:03
rlandyyeah - it was not installing in a virtualenv21:04
rlandyand rhel minimal does not have python-pip21:04
rlandyso no pip install working21:04
rlandyhence this card ...21:04
rlandyhttps://tree.taiga.io/project/tripleo-ci-board/us/226?no-milestone=121:04
rlandyhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/upshift-rhos-14-ci-rhos-ovb-featureset001/181/console21:05
rlandywe're stack crating now21:05
rlandycreating now21:05
rlandythat is good21:05
rlandyweshay: https://review.openstack.org/#/c/611717/21:06
rlandymerged21:06
weshayah ya https://review.openstack.org/#/c/611717/21:06
weshaynice21:06
weshayrlandy, https://tree.taiga.io/project/tripleo-ci-board/us/226?no-milestone=121:07
weshaydoes not make sense to me21:07
rlandyyou missed the joke :(21:08
weshayon RHEL we've always had to yum install easy-install21:08
rlandyyou had to be there21:08
weshayk21:08
rlandyno python-pip21:08
*** trown is now known as trown|outtypewww21:08
rlandyconfirmed with apevec21:08
weshayin 821:08
weshayor 721:08
rlandywe're still 721:08
rlandyatm21:08
weshayso.. how is it working in our osp jobs?21:09
rlandyugh - I probably should have rebase the patches on that job ...21:09
*** vinaykns has quit IRC21:35
*** agopi has joined #oooq21:48
rlandyhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/upshift-rhos-14-ci-rhos-ovb-featureset001/181/console22:04
rlandyha22:04
rlandywe got to introspection on upshift22:04
rlandyweshay: https://thirdparty.logs.rdoproject.org/jenkins-upshift-rhos-14-ci-rhos-ovb-featureset001-181/undercloud/home/stack/overcloud_prep_images.log.txt.gz#_2018-10-23_18_03_5922:19
rlandyupshift concurs22:19
rlandyhttps://bugs.launchpad.net/tripleo/+bug/179752622:20
openstackLaunchpad bug 1797526 in tripleo "Failed to get power state for node FS01/02" [Critical,Triaged]22:20
rlandythat's rhos-1422:20
rlandybut quickstart/q-e master22:20
* rlandy runs again22:21
rlandyI still have the stack22:21
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-multinode-1ctlr-featureset010, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb- (4 more messages)22:37
*** hamzy has quit IRC22:59
*** tosky has quit IRC23:02
*** sanjayu_ has joined #oooq23:05
*** rlandy is now known as rlandy|bbl23:17

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!