Tuesday, 2018-06-05

« Monday, 2018-06-04 Index Wednesday, 2018-06-06 »
rlandy|roverweshay: back in about 1 1/2 hrs - will re-enable the cron then - unless you object00:10
*** rlandy|rover is now known as rlandy|rover|bbl00:11
*** dtrainor has quit IRC00:16
*** dtrainor has joined #oooq00:16
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.00:30
*** rlandy|rover|bbl is now known as rlandy|rover01:35
rlandy|roverhmmm - queens is still busy going01:44
rlandy|roveroh - it just keep going in a loop02:22
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.02:30
rlandy|roverweshay: arxcruz|ruck: cron uncommented02:31
rlandy|roverocata just kicked02:33
*** myoung|off is now known as myoung03:08
myoungrlandy|rover: o/  all ok w/ promoter?03:08
*** udesale has joined #oooq03:44
*** jaganathan has joined #oooq04:07
*** links has joined #oooq04:22
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.04:30
*** saneax has joined #oooq04:57
*** ccamacho has quit IRC05:07
*** rlandy|rover has quit IRC05:22
*** quiquell has joined #oooq05:32
*** skramaja has joined #oooq05:45
*** ccamacho has joined #oooq05:46
quiquellarxcruz|ruck: You there ?05:48
*** yolanda_ is now known as yolanda05:48
*** myoung is now known as myoung|zzz05:49
*** ccamacho has quit IRC05:49
*** pgadiya has joined #oooq05:50
*** pgadiya has quit IRC05:50
*** bogdando has joined #oooq05:53
*** links has quit IRC06:08
*** skramaja has quit IRC06:14
*** jaosorior has joined #oooq06:14
*** skramaja has joined #oooq06:14
*** ccamacho has joined #oooq06:19
chandankumararxcruz|ruck: dmsiard06:20
*** links has joined #oooq06:26
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.06:30
*** sshnaidm has joined #oooq06:53
sshnaidmtrown|outtypewww, panda please review https://review.openstack.org/#/c/572107/06:55
*** jbadiapa has joined #oooq07:01
*** gkadam has quit IRC07:02
*** sshnaidm is now known as sshnaidm|brq07:05
*** ratailor has joined #oooq07:14
*** ykarel has joined #oooq07:16
*** ykarel is now known as ykarel|training07:17
*** kopecmartin has joined #oooq07:30
*** jbadiapa has quit IRC07:32
*** jbadiapa has joined #oooq07:33
*** amoralej|off is now known as amoralej07:37
quiquellarxcruz|ruck: I am going to put promoter in squencial mode07:41
quiquellIt's failing at queens07:41
bogdandojust sent an update for the zuul jobs dependencies http://lists.openstack.org/pipermail/openstack-dev/2018-June/131144.html07:45
bogdandojfyi07:45
bogdandolet's merge it and see how it works for us07:45
*** holser__ has joined #oooq07:51
quiquellbogdando: Do you have a minute for a zuul question ?08:07
*** tosky has joined #oooq08:15
*** gkadam has joined #oooq08:21
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.08:30
bogdandoquiquell: sure08:32
quiquellbogdando: We are activating scenario000 (keystone only) job for stable/queens08:33
quiquellbogdando: Do we have to add the job to all the stable/queens branches around in tripleo ?08:33
quiquellbogdando: I have identified tht, tripleo-common, puppet-tripleo and python-tripleoclient08:34
bogdandoquiquell: I cannot answer that, but I would check for another job name around08:34
bogdandoas an example08:34
bogdandosee http://codesearch.openstack.org/08:34
sshnaidm|brqquiquell, do you look for repos to enable this job?08:34
quiquellsshnaidm|brq: yep, exactly that08:35
quiquellsshnaidm|brq: Has do a zuul + gerrit search08:35
quiquellBu don't know if I missing some08:35
quiquellsshnaidm|brq: job name is tripleo-ci-centos-7-scenario000-multinode-oooq-container08:35
sshnaidm|brqquiquell, wasn't it decided in sprint..? I don't remember. But we need ti on tripleo-upgrade definitely08:35
quiquellsshnaidm|brq: We don't have to check every change that already goes to the job in master ?08:36
quiquellsshnaidm|brq: Going to add some reviews and see what the teams says08:36
sshnaidm|brqquiquell, not sure I understand, do you enable job for queens branch or you run queens job for master patches..?08:37
sshnaidm|brqquiquell, ya, seems like a good plan08:37
quiquellsshnaidm|brq: Have just found the projects that already run tripleo-ci-centos-7-scenario000-multinode-oooq-container08:38
quiquellsshnaidm|brq: And for those I am checking if they have an stable/queens branch08:38
quiquellsshnaidm|brq: If so, then add to the branch the job to zuul.d08:38
*** ratailor has quit IRC08:44
*** ratailor has joined #oooq08:45
arxcruz|ruckpanda: around ?08:50
quiquellarxcruz|ruck: I think he is in PTO this week08:53
arxcruz|ruck:/08:53
quiquellarxcruz|ruck: I have put the promoter in sequencial mode, it was failing in queens at the lock08:54
arxcruz|ruckquiquell: the promoter is already in the promoter server or still in myoung server ?08:54
quiquellarxcruz|ruck: It was running in the promoter server with the crontab08:55
arxcruz|ruckok08:55
arxcruz|ruckjust saw rlandy comment from yesterday08:55
quiquellarxcruz|ruck: What did she say (my raspberry has die, so I don't have history)08:55
arxcruz|ruckthat she uncoment crontab in the promote server08:55
quiquellarxcruz|ruck: Ok, this is problematic with the current state of promoter08:56
quiquellarxcruz|ruck: Now it's slow but steady08:56
arxcruz|ruckquiquell: the dns problem is fixed afaik09:00
quiquellarxcruz|ruck: what DNS problem ?09:03
*** jfrancoa has joined #oooq09:11
*** jfrancoa has quit IRC09:19
quiquellarxcruz|ruck: the DNS problem is this ? Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 127.0.0.1:53: no such host"09:24
arxcruz|ruckquiquell: no, it was a degradation09:35
quiquellarxcruz|ruck: Do you know about the stuff I pasted ?09:36
arxcruz|ruckquiquell: no... but might be dns, why is it looking into 127.0.0.1 ?09:36
quiquellarxcruz|ruck: dnsmasq is running locally I suppose09:37
sshnaidm|brqquiquell, arxcruz|ruck which host is it?09:47
arxcruz|rucksshnaidm|brq: the promoter script09:47
quiquellarxcruz|ruck: I am not talking about the promoter09:47
quiquellarxcruz|ruck: Talking about a random job of mine09:48
arxcruz|ruckquiquell: okay, can you point me the logs ?09:48
quiquellarxcruz|ruck: Sure http://logs.openstack.org/96/572096/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/0958661/09:49
quiquellsshnaidm|brq: The host is https://registry-1.docker.io/v2/:09:49
arxcruz|ruckquiquell: link to the error :)09:50
quiquellarxcruz|ruck: Ups sorry, http://logs.openstack.org/96/572096/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/0958661/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz09:51
*** skramaja has quit IRC09:55
sshnaidm|brqquiquell, weird.. http://logs.openstack.org/96/572096/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/0958661/logs/undercloud/var/lib/unbound/unbound.log.txt.gz09:56
sshnaidm|brqquiquell, unbound resolved it, maybe took time, it's impossible to see without timestampe09:56
quiquellsshnaidm|brq: Impossible to discover  the duration of the query09:58
quiquellsshnaidm|brq: Let's see the results after recheck09:58
quiquellsshnaidm|brq, arxcruz|ruck: ocata phase 2 has wrong numbers here http://rhos-release.virt.bos.redhat.com:3030/rhosp10:00
quiquellChecking https://dashboards.rdoproject.org/ocata, it has an old promotion10:00
arxcruz|ruckquiquell: didn't get it, rhos-release is wrong is that what you mean ?10:01
ssbarneahi guys! i need few hints regarding 3o heat templates. see https://github.com/openstack/tripleo-heat-templates/blob/c24cec0fa70370d548c055c5977e8a2da1f2531b/environments/storage/cinder-netapp-config.yaml10:02
arxcruz|rucki don't know how this works, i think it takes a while to update10:02
Tengussbarnea: hello! you might want to /join #tripleo instead :).10:02
quiquellarxcruz|ruck: Ok, let's wait.10:02
ssbarneaok, will do. bit weird as this file does not exist and is not clear who is supposed to create it10:03
*** zoli is now known as zoli|lunch10:07
quiquellsshnaidm|brq: We need a +1w here https://review.rdoproject.org/r/#/c/1360410:25
quiquellsshnaidm|brq:10:26
quiquellpromoter: rewrite hash exclusion critera10:26
sshnaidm|brqquiquell, yeah, need to review it before..10:27
quiquellsshnaidm|brq: Sure, I am doing it also, maybe we can add soime unit tests there10:27
sshnaidm|brqwould be great10:27
*** moguimar has quit IRC10:29
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.10:30
*** skramaja has joined #oooq10:31
sshnaidm|brqand need to improve this scripts logging, I'm breaking eyes trying to understand what happens there..10:33
quiquellsshnaidm|brq: Yep, myoung|zzz has total refactoring idea, but this will take time10:37
quiquellsshnaidm|brq: Let's test what we have10:37
*** skramaja has quit IRC10:38
*** skramaja has joined #oooq10:39
quiquellsshnaidm|brq: https://review.openstack.org/#/c/572096/ Some cleanpu at toci10:43
*** saneax has quit IRC10:47
*** pliu has quit IRC10:50
*** lhinds has quit IRC10:50
*** tosky__ has joined #oooq10:51
*** lhinds has joined #oooq10:51
*** pliu has joined #oooq10:51
*** tosky has quit IRC10:52
quiquellsshnaidm|brq: Found a problem if there is no hashes fetched.10:57
quiquellrunning unit test10:57
*** myoung|zzz has quit IRC10:57
*** jjoyce has quit IRC10:57
*** lhinds has quit IRC10:57
*** jschluet has quit IRC10:57
*** rnoriega has quit IRC10:57
*** weshay has quit IRC10:57
*** rasca has quit IRC10:58
*** myoung has joined #oooq10:58
*** weshay has joined #oooq10:58
*** rnoriega has joined #oooq10:58
*** jjoyce has joined #oooq10:58
*** lhinds has joined #oooq10:59
*** rasca has joined #oooq11:01
*** jfrancoa has joined #oooq11:01
*** jschlueter has joined #oooq11:02
*** jfrancoa has quit IRC11:07
*** links has quit IRC11:09
*** moguimar has joined #oooq11:12
*** atoth has joined #oooq11:16
*** gkadam has quit IRC11:17
*** gkadam has joined #oooq11:17
*** udesale has quit IRC11:17
*** udesale has joined #oooq11:17
*** udesale has quit IRC11:23
*** links has joined #oooq11:25
*** zoli|lunch is now known as zoli11:27
*** matbu has quit IRC11:29
*** quiquell is now known as quiquell|lunch11:29
*** matbu has joined #oooq11:30
sshnaidm|brqquiquell|lunch, hmm.. does it mean that release_args just weren't used before??11:33
weshaysshnaidm|brq, any idea what I may be missing? re: https://review.openstack.org/#/c/572217/5/toci_gate_test-oooq.sh http://logs.openstack.org/17/572217/5/check/tripleo-ci-centos-7-standalone/79a89f9/job-output.txt.gz#_2018-06-05_06_29_49_29151811:41
*** dtantsur is now known as dtantsur|brb11:41
sshnaidm|brqweshay, seems like zuul failure somewhere11:52
*** tcw has quit IRC11:53
*** tcw has joined #oooq11:54
*** amoralej is now known as amoralej|lunch11:56
weshaysshnaidm|brq, I believe it's failing in toci_gate_test_oooq.sh12:04
sshnaidm|brqweshay, no, it's before12:04
weshaysshnaidm|brq, you sure.. seems like this line https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-ci/run.yaml#L5712:05
weshayah no.. that is just the def12:06
*** quiquell|lunch is now known as quiquell12:13
*** saneax has joined #oooq12:14
*** trown|outtypewww is now known as trown12:16
*** tosky__ is now known as tosky12:23
sshnaidm|brqweshay, need your powers please:  https://review.openstack.org/#/c/572107/12:23
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.12:30
weshayrfolco_, ping quiquell re: reviews and cards12:31
*** rlandy has joined #oooq12:31
*** rlandy is now known as rlandy|rover12:32
rfolco_weshay, yep was about to ping quiquell12:32
rlandy|roverarxcruz|ruck: hello12:32
rlandy|roverarxcruz|ruck: how are we doing today?12:32
rlandy|roverI re-enabled the cron12:33
rfolco_quiquell, any card needing help or qe ?12:33
quiquellrfolco_: Set you as QE for one of them12:34
quiquellfs037 on queens12:34
rlandy|roverocta phase2 not so ok - will look12:34
rlandy|roverquiquell: hi there12:34
rlandy|roverquiquell: was sort of half listening at yesterday's scrum - need any info wrt container updates?12:35
rfolco_quiquell, thx12:35
quiquellrlandy|rover: give a min12:41
quiquellrlandy|rover: I have put promoter back on sequencial, queens was failing12:41
rlandy|roverquiquell: so you killed the cron and started promoter.sh?12:42
rlandy|roverI see the cron is commented out again12:42
rfolco_quiquell, quick question... fs037 doesn't need to be run against tq and tqe ?12:44
rlandy|roverquiquell: did you start promoter.sh from a tux window by any chance?12:44
rlandy|roverso we can kill it if need be?12:44
rlandy|roverwell I see your PID12:45
rlandy|roverit's just helpful to see where the script is atm - akthough I guess we can tell from docker commands12:46
*** brault has quit IRC12:46
* rlandy|rover feels like I am talking to myself12:46
rfolco_nm quiquell ignore my question. This is about stable branch :)12:47
rlandy|roveranyways need to put a patch in for the infra change12:47
*** amoralej|lunch is now known as amoralej12:51
rlandy|roverarxcruz|ruck: pls see comments on https://review.rdoproject.org/r/#/c/14060 - if you need help with the suggested other solution, can help there as we used it for libvirt12:51
quiquellrlandy|rover: I started a promoter.sh at tmux12:52
weshayquiquell, https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/tasks/main.yml#L2912:53
arxcruz|ruckrlandy|rover: checking12:53
rlandy|roverquiquell; thanks - I see it12:53
rlandy|roverwe need to change the cron to run sequentially then12:53
quiquellrlandy|rover: is more of a daemon with systemd12:54
quiquellI have this12:54
weshayhttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/tasks/main.yml#L173-L18012:54
rlandy|roverquiquell: you have this. meaning?12:55
quiquellhttps://review.rdoproject.org/r/#/c/13622/12:56
arxcruz|ruckrlandy|rover: thanks, i'll do that, however i have a question, how can i get the private ipv4 from dns-server ? i'm not seeing where this fact is being set :/12:56
rfolco_weshay, how do I recheck rdo-3rd-party only?12:56
arxcruz|ruckshall I use os_server_facts ?12:56
quiquellrlandy|rover: Sorry, my brain has go to the clouds :-)12:56
rlandy|roverI see the review - checking12:56
rlandy|roverarxcruz|ruck: getting12:57
rlandy|roverarxcruz|ruck: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/tenant_vars/common.yml12:59
rlandy|roverinterestingly enough that is not what it is now12:59
weshayquiquell, http://status.openstack.org/openstack-health/#/?searchProject=tripleo&groupKey=build_name12:59
myoungquiquell, rlandy|rover, sshnaidm|brq, as a stopgap / low hanging fruit approach to stopping eye bleed in the promoter logs (without breaking quiquell's log parsing in the short term), and until we can do something a little cleaner, what if we just had the logs contain formatted (pretty printed) json output, instead of the massive wrapped unformattted json output?12:59
myoung^^ would be a tiny change to the promoter's logging funcs12:59
myoungquiquell: what did you mean about rhos-release dashboards being wrong?  are you looking at the # days diff reported there vs. date of promotion events?13:00
rlandy|rovermyoung: I have no problem with that - but the logs aren't our biggest issues right now - just imho13:00
myoungquiquell: if so, that's expected...rhos-release dashboards report based on repo creation time, not promotion event time13:00
myoungrlandy|rover: I concur, just parsing the comments from while I was sleeping ;)  My view would be to do as little as possible to the promoter until we have time to address it properly...but if eyes are bleeding I could propose a one-liner that will at least make humans happier13:01
* myoung also thinks we have bigger fish to fry13:01
rlandy|roverquiquell: I like what you have done - pls see comments in line - we can try it out13:08
myoungquiquell: ditto13:09
rlandy|roverrasca: hi - wrt https://review.openstack.org/#/c/572155/ - didn't get to update it yesterday - will work on that today 0 just to correct lint errors etc.13:09
myoungquiquell: comments in patch13:09
quiquellmyoung, rlandy|rover: mtg finishing13:10
*** dtantsur|brb is now known as dtantsur13:10
myoungbrb13:10
*** matbu has quit IRC13:11
*** matbu has joined #oooq13:13
arxcruz|ruckrlandy|rover: that's why i was asking... :/13:14
arxcruz|ruckrlandy|rover: i think we can't reserve the private ip right ?13:14
arxcruz|ruckso that tenant_vars is wrong13:14
arxcruz|ruckwell, i'll update the tenant_vars in my patch13:14
rlandy|roverarxcruz|ruck: none of the internal ips are what we set13:15
rlandy|roverwe can reserve13:15
rlandy|roverbecause it's our own tenant and own tenant internal network13:16
rlandy|roverif not, we could never pick internal ips at all13:16
arxcruz|ruckrlandy|rover: ok, i'll update the ips for all the servers13:18
*** matbu has quit IRC13:19
rlandy|roverarxcruz|ruck: I am not sure that's the right thing ... let's think about this ...13:20
rlandy|roverif we tear down all the infra - we have a clean tenant13:20
arxcruz|ruckyes13:20
rlandy|roverwhat happens on reboot, reconnect?13:20
arxcruz|ruckrlandy|rover: the ip stays the same13:21
rlandy|roverquestion is why/when the private ip might change13:21
arxcruz|ruckboth private and public, i just restart twice the tempestmail to test13:21
rlandy|roverthe whole infra setup is based on the fact that we can determine out ips13:21
arxcruz|ruckrlandy|rover: i don't think it can change without we want it, but i don't know if we can reserve13:21
arxcruz|rucklike the public ip13:21
quiquellrlandy|rover: I am back do you want to talk about promoter ?13:22
quiquellmyoung: ^13:22
rlandy|roverquiquell: we left comments13:22
rlandy|roverwe can try it out - not sure of we an explicit cleanup13:23
myoungsure13:23
myoungi'm here13:23
rlandy|roveror id stopping with systectl is good enough13:23
myoungpop into my room?13:23
quiquellmyoung: going there13:23
rlandy|rovernot sure of we need it? more to discuss than what is in comments?13:23
rlandy|rovermyoung; ^^ quiquell ^^?13:24
*** matbu has joined #oooq13:24
myoungditto...just making self available13:25
quiquellrlandy|rover: yep is enough in the review13:25
quiquellmyoung: I am adding some unit test to the panda's change13:26
quiquellBut I don't know how urgent the change is needed13:26
rlandy|roverquiquell: not urgent13:26
quiquellrlandy|rover: Then I will try to add them, the logic there is big enough to do a good use of some unit testing13:27
rlandy|roverquiquell: we have something going atm13:27
rlandy|roverquiquell: we can pick it up later13:27
quiquellrlandy|rover: You mean the panda's change13:27
quiquellNot the sequancial change13:28
rlandy|roverquiquell: not sure what you mean about panda's change - on sprint?13:29
rlandy|roveralso - why was queens breaking using the cron?13:29
rlandy|roverdid lock not work?13:29
*** jbadiapa has quit IRC13:29
rlandy|roverit was working when I enabled it originally so I want know what is going wrong with the current implementation13:29
quiquellrlandy|rover: https://review.rdoproject.org/r/#/c/1360413:29
quiquellpromoter: rewrite hash exclusion critera13:29
quiquellI am adding unit test for those13:29
weshaytrown, fyi please read this card.. we found a design bug https://trello.com/c/6tcD7ilr/778-injecting-zuul-changes-at-various-points-in-the-job-workflow13:30
rlandy|roverquiquell: ok - I see panda's change you are working on - can we return to why queens was failing13:30
rlandy|roverwe did this before13:30
rlandy|roverimplement a new solution w/o debugging the problem13:31
rlandy|roverso I'd just like to know what happened that we had to go back to sequential13:31
quiquellrlandy|rover: There were no timeout or retry mechanisme in the lock and octata promotions was interfering with queens13:31
quiquellrlandy|rover: Becouse we have a one big lock, that we don't need btw.13:32
quiquellrlandy|rover: That's why we move it to run it sequancially, is slower but is good enough13:32
arxcruz|ruckrlandy|rover: i think now is okay13:32
rlandy|roverarxcruz|ruck: the review? or the promoter running?13:33
arxcruz|ruckrlandy|rover: the review lol13:33
rlandy|roverarxcruz|ruck: ok - I'll look in a sec13:33
weshayquiquell, myoung can you guys fix this card https://trello.com/c/pfQ867XP/779-differentiate-the-same-featureset-to-do-the-two-begin-release-end-release-combinations13:34
weshayquiquell, the reviews in the card are yours.. so who is the owner vs. qe here?13:34
quiquellweshay: There were from the design phase, kind of POC13:34
quiquellweshay: Going to move them to the comments13:35
rlandy|roverweshay: when you have a moment, can you weigh in on what we should be doing now - wrt -re-disbaled cron, promoter.sh running in tmux and merging changes to make the promoter run via starting quiquell's review13:35
rlandy|roverall are possible - what is the right path at this point?13:35
* weshay looks13:36
weshaywe're refering to https://review.rdoproject.org/r/#/c/1360413:36
weshay?13:36
rlandy|roverhttps://review.rdoproject.org/r/#/c/13622/13:36
rlandy|roverweshay: ^^13:36
myounghttps://review.rdoproject.org/r/#/c/13622/13:36
myoungrlandy|rover: whoops...concurrent pastes :)13:37
quiquellweshay, myoung: have move the POCs to the comments at card https://trello.com/c/pfQ867XP/779-differentiate-the-same-featureset-to-do-the-two-begin-release-end-release-combinations13:37
rlandy|roverweshay: the cron was disabled again this morning13:37
weshayok.. FOLKS13:37
weshayonly arxcruz|ruck and rlandy|rover should be messing around in the promoter ATM13:37
myoungweshay: re https://trello.com/c/pfQ867XP/779-differentiate-the-same-featureset-to-do-the-two-begin-release-end-release-combinations, I picked this up recently and I'm getting my head around it.  This is another card we didn't really get a detailed design for as part of planning.  working on it today13:37
rlandy|roverthat work was done originally on may 3rd :)13:38
weshayfair13:38
rlandy|roverweshay: I just don't want to fall into another situation like last time13:39
rlandy|roverwhere we implement something temp13:39
rlandy|roverthe promoter.sh running in tmux13:39
rlandy|roverand leave the issue at hand13:39
weshayquiquell, rlandy|rover arxcruz|ruck I would not change the promoter until panda can review the change13:39
rlandy|roverweshay: fine by me - ...13:39
rlandy|roveras long as you are ok with our running the promoter13:40
* myoung thinks as well that https://review.rdoproject.org/r/#/c/13622 needs to be disentangled, there are 3 changes there13:40
weshayhe's put more work into the promoter than anyone else on the team13:40
quiquellweshay: promoter fails if we don't do so13:40
rlandy|roversequentially - kicked by hand atm13:40
rlandy|roverquiquell: true - but we have a stop gap13:40
rlandy|roverthe sequential run kicked by hand - as you did13:40
rlandy|roverfor a few days - if weshay agrees13:40
quiquellrlandy|rover: Ok, let's leave the creature alone13:40
rlandy|roverI agree in the long term your solution is right13:40
weshayquiquell, rlandy|rover I suspect re-enabling cron was not the right thing to do.. there is a lot of tribal knowledge here13:41
weshayrun it via the script as it was before it was taken down13:41
*** jbadiapa has joined #oooq13:41
weshayI only want to see rlandy|rover and arxcruz|ruck discussing and working on the promoter at this time13:41
rlandy|roverweshay: ack - that is just not a sustainable solution ... looking when panda is back13:41
weshayrlandy|rover, the concern is that the script is not git or something else?13:42
rlandy|roverweshay: my concern is that the script is kicked in a non-public way13:42
rlandy|roverok it is in tmux13:42
weshayrlandy|rover, sure.. I think we'll be fine for the week13:43
rlandy|roverweshay; quiquell's patch does add the script to git13:43
weshayI see that13:43
rlandy|roverweshay: ack, ok - will pick it up with panda and quiquell  and you when panda is back13:43
weshaythanks all13:44
quiquellrlandy|rover, weshay: I have put some alarms in the promoter machine, if it goes down we will see13:44
weshayquiquell, can you monitor drive free space?13:44
quiquellweshay: I think, let me check13:45
rlandy|roverwe should be able to13:45
rlandy|roverin /var/lib/docker13:45
rfolco_https://review.openstack.org/#/c/572187/ review begging weshay myoung trown rlandy|rover quiquell :)13:47
weshayrfolco_, where is tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades defined?13:47
* myoung isn't logged into the promoter and backs away slowly13:48
weshayah I see it13:48
weshay- job:13:48
weshay    name: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades13:48
weshay    parent: tripleo-ci-dsvm-multinode13:48
rlandy|roversorry arxcruz|ruck - looking at your patch now13:49
myoungrfolco_: looking13:49
rfolco_weshay, quiquell's patch merged --> https://review.openstack.org/#/c/558425/18/zuul.d/multinode-jobs.yaml13:49
rfolco_weshay, I put a reference in the commit message to his patch :)13:50
weshayrfolco_, ya.. I see it now thanks13:50
weshayrfolco_, is this the job log? http://logs.openstack.org/87/572187/2/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/ca1ee14/job-output.txt.gz13:50
weshayrfolco_, join my blue13:51
rfolco_k13:51
arxcruz|ruckrlandy|rover: weshay so, regarding changing tqe and tq gate jobs to rhos-13, the rhos-13 jobs was already defined, but not enabled, i enable it and i'll see if the changes will triger it13:51
rlandy|roverarxcruz|ruck: careful there ...13:52
rlandy|roveruses the same resources13:52
weshay:)13:52
rlandy|roverI would not mass enable it13:52
rlandy|roverwe should kick one job13:52
arxcruz|ruckrlandy|rover: i just enable tqe job13:52
rlandy|roverand see what shakes out13:52
rlandy|roverarxcruz|ruck: let it run one job and disable again13:52
rlandy|roverjust imho13:52
arxcruz|ruckrlandy|rover: ack13:52
arxcruz|ruckquiquell: see, the board is now updated for ocata phase 113:52
arxcruz|ruckit was a matter of time :)13:52
arxcruz|ruckoh, nevermind, it was phase 2 right ?13:53
rlandy|roverarxcruz|ruck: will be enough for us to debug13:53
quiquellweshay: arxcruz|ruck yep13:53
quiquell:-)13:53
arxcruz|ruckmyoung: i think you change in http://rhos-release.virt.bos.redhat.com:3030/rhosp is missing something, because ocata is 4d with last hash imported on 05-31 but https://dashboards.rdoproject.org/ocata shows repo from 06-0113:54
weshayquiquell--13:55
hubbotweshay: quiquell's karma is now -113:55
weshayquiquell, you need to review changes more in depth13:55
quiquellweshay: wich one ?13:56
weshay57218713:56
quiquellholy sh... yep sorry, I will, can we make tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades voting ?13:57
weshayquiquell, no.. but the upgrade job is not running13:58
myoungarxcruz|ruck: there's existing work on the rhos-13 stuff as well, let me find the bug with details.  we talked about 2 of them at triage yesterday13:58
myoungarxcruz|ruck: regarding the image ID's, duplicate images, etc.  If you need I can show you how to kick just one job to test things13:58
quiquellweshay: Ejecuted the recheck this morning, Ãlet's see13:58
myoungarxcruz|ruck: using jenkins gerrit plugin and/or just using jenkins13:58
arxcruz|ruckmyoung: actually, this is just to update the tq and tqe job from rhos-12 to rhos-1313:58
myoungarxcruz|ruck: i understand what it is...  afaik we're not ready to flip the switch to 13 until we are sure it works13:59
arxcruz|ruckmyoung: got it13:59
weshayrfolco_, http://logs.openstack.org/46/571446/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/a00fd6e/job-output.txt.gz#_2018-06-05_10_01_50_31994714:01
myoungarxcruz|ruck: i need to go nose down on sprint tasks, let me know if you want a 5 min rundown on rhos-13 tq/tqe gates work.  There's a lot of detail in existing LP's and the etherpad from last sprint.  Let me know if you need links to LP's etc.14:02
arxcruz|ruckmyoung: sure, i do need :)14:02
rlandy|rovermyoung; arxcruz|ruck: which is why suggested we run one job - to see what the situation is atm - wrt rhos-1314:02
arxcruz|ruckwhen you can14:02
rascarlandy|rover, I'm working on it14:03
arxcruz|ruckrlandy|rover: sure :) just wanted to do something, being ruck i have the feel i'm doing nothing usefull hehehe14:03
myoungarxcruz|ruck: we can schedule a 5-10 after community meeting.  it woudl help if you could catch up on existing work / LP documentation.  There's some good low hanging fruit for ruck/rover tasks there if you have time14:03
rlandy|roverrasca: cool - thanks14:03
arxcruz|ruckmyoung: where's this documentation? in the ruckrover etherpad ?14:03
rlandy|rovertripleo meeting14:03
myoungfolks #tripleo meeting is starting.  cya there14:03
rlandy|roverarxcruz|ruck: you joining?14:04
rlandy|roverthere we go - lots of ruck work to do :)14:04
arxcruz|ruckrlandy|rover: ooo meeting ?14:04
rlandy|roveryep14:04
rlandy|roverI see you there, then I will just lurk14:04
quiquellrfolco_: Don't understand the job error14:05
rfolco_quiquell, I am debugging with weshay14:05
quiquellrfolco_: ok ok14:05
quiquellweshay: I think the upgrade is working without changing build-test-package, the RPMs are ok after it finished14:08
* myoung is in CI meeting room early, if anyone want's to pretend we work in human proximity feel free to join :)14:09
quiquellmyoung: What's the ooo ci community meeting about ?14:10
*** ccamacho has quit IRC14:12
*** ccamacho1 has joined #oooq14:12
rlandy|roverquiquell: the community meeting is open to devs outside the Ci team who have questions/want support for Ci-related work14:14
quiquellrlandy|rover: ci ask me everything :-)14:15
quiquellhave to drop now14:15
*** quiquell is now known as quiquell|off14:15
weshaymyoung, tripleo mtg <EmilienM> please take some time to update the etherpads14:16
myoungweshay, arxcruz|ruck was in middle of typing same.14:17
arxcruz|ruckme?14:17
myoungarxcruz|ruck: ya, I can do it, but this week it's super easy...we're all green :)14:18
myoungarxcruz|ruck: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting @L3514:18
myoungarxcruz|ruck: how often does ruck get to say "nothing to see here" :)14:18
* myoung is happy to do it...14:18
arxcruz|ruckmyoung: never lol14:19
* rlandy|rover doesn't think there is nothing to see - we have some stuff we're covering up :(14:19
myoungCI squad: note that https://bugs.launchpad.net/tripleo/+bug/1775190 was just created, which is tracking the discussion around CI coverage for openshift.  Please subscribe/follow - this is the topic I've beem mentioning over the past few CI meetings as something that we as a team need to both be aware of and thinking about when we look to future14:20
openstackLaunchpad bug 1775190 in tripleo "Add CI coverage for openshift deployment" [Medium,Triaged]14:20
quiquell|offasdfafafda14:23
myoungarxcruz|ruck: nvrmind.  status is updated14:24
myoungrlandy|rover: add additional status items at will, I just filled in values from source of truth (dashboard)14:25
myoungarxcruz|ruck: ^^14:25
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.14:31
myoungthe meeting in #tripleo just ended, CI meeting starts now folks show up.  will call it if no agenda items in first 5-1014:31
rlandy|rovermyoung: sorry - additional item where?14:32
myoungrlandy|rover: optional -  https://etherpad.openstack.org/p/tripleo-ci-squad-meeting @L35, just CI status14:32
myoungrlandy|rover: more a ruck task14:32
myoungrlandy|rover: just to your point that everything isn't totally green/smooth - if we want to communicate anything we can add in etherpad above/beyond basic # days for pomotions from various phases14:33
rlandy|rovermyoung: wrt removing promotion criteria14:34
weshayrfolco_, #rhos-upgrades14:39
rascahey rlandy|rover do we know if rdo cloud is broken somehow? Because I fixed all the stuff, but it's failing the rdo gates: https://logs.rdoproject.org/55/572155/2/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/Z4771a7ecff1b48d5937ce0bb555b8332/console.txt.gz but the error seems not related to what we touched14:39
rascarlandy|rover, it's "cp: cannot stat ‘/opt/stack/new/python-tripleoclient’: No such file or directory"14:40
rlandy|roverrasca: sorry- been rover busy - will take a look14:41
rlandy|roverhmmm ....build-teat-package issue14:43
rlandy|roverrasca: I don't think it's you14:47
rlandy|roverarxcruz|ruck: https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-containers-build/977/console14:53
rlandy|roverwe got a containers build failure14:54
rlandy|rover12:22:54 TASK [undercloud-deploy : Install the undercloud14:54
arxcruz|ruckrlandy|rover: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-centos-7-master-containers-build/14ef616/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2018-06-05_12_35_0314:55
*** alee_ has joined #oooq14:55
arxcruz|ruck2018-06-05 12:35:03 | APIException: no such option collect_timing in group [keystone_authtoken]14:55
arxcruz|ruckrlandy|rover: open lp14:55
rlandy|roverarxcruz|ruck: you see any build-test-packages errors?14:56
alee_weshay, trown , sshnaidm|brq please review https://review.openstack.org/#/c/571805/14:58
rlandy|roverarxcruz|ruck: you're opening a bug or should I? sorry - not clear from your response above14:58
arxcruz|ruckrlandy|rover: no14:58
rlandy|roverno - which one?14:58
arxcruz|ruckrlandy|rover: sorry, bad english here, I'll open the bug :)14:58
rlandy|roverarxcruz|ruck: k14:58
arxcruz|rucklol, you type too fast :)14:58
arxcruz|ruckso, no, i'm not seeing build-test-packages errors https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-centos-7-master-containers-build/14ef616/ara_oooq/14:59
arxcruz|ruckand yes, i'll open the launchpad14:59
rlandy|roverarxcruz|ruck: ye - just don't want to take your work - since you are feeling w/o tasks :)14:59
arxcruz|ruckhahaha, don't worry14:59
arxcruz|ruckrlandy|rover: https://bugs.launchpad.net/tripleo/+bug/177519915:01
openstackLaunchpad bug 1775199 in tripleo "Failure in tripleo undercloud install no such option collect_timing in keystone_authtoken group" [Critical,Triaged]15:01
rlandy|roverarxcruz|ruck: cool - thanks15:03
arxcruz|ruckmyoung: you leave ?15:03
weshayhrm.. rfolco_ sova is up to date15:03
weshay:(15:03
weshayre: upgrade jobs15:03
rfolco_weshay, what do you mean15:04
myoungarxcruz|ruck: i need a few minutes15:04
weshayrfolco_, http://cistatus.tripleo.org/promotion/15:04
weshaycomparing that to.. the rdo results in https://review.openstack.org/#/c/570776/15:04
arxcruz|ruckrlandy|rover: marked as critical and promotion blocker15:04
arxcruz|ruckmyoung: k, i'll grab a coffee15:04
rfolco_weshay, I did not find a bug for docker inspect command: Error while fetching server API version: ('Connection aborted.', error(13, 'Permission denied'))15:05
rfolco_weshay, upgrade jobs are all broken, is that what you mean?15:05
rfolco_arxcruz|ruck, just in case... have you seen this before ? http://logs.openstack.org/87/572187/2/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/ca1ee14/logs/undercloud/home/zuul/upgrade_overcloud_prep_containers.log.txt.gz#_2018-06-05_12_21_0815:06
weshayrfolco_, pretty fucking much15:07
arxcruz|ruckrfolco_: nope15:07
myoungarxcruz|ruck: great.  please ping me when you've reviewed https://bugs.launchpad.net/tripleo/+bug/1773569 and https://bugs.launchpad.net/tripleo/+bug/177357215:08
openstackLaunchpad bug 1773569 in tripleo "TQE: ovb-create-stack.yml role overrides are not working as expected" [High,New] - Assigned to Matt Young (halcyondude)15:08
openstackLaunchpad bug 1773572 in tripleo "rdo2: periodic ovb job (master) isn't handling hashes correctly" [High,Triaged]15:08
arxcruz|ruckmyoung: i am aware of this two bugs, we review it yesterday15:08
myoungarxcruz|ruck: ack.  also notes from handoff https://review.rdoproject.org/etherpad/p/ruckrover-sprint14 @L13015:09
weshayarxcruz|ruck, you my ruck15:09
weshayrfolco_, http://logs.openstack.org/41/566241/11/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/ee28a30/job-output.txt.gz#_2018-06-05_13_21_19_96073915:09
weshayarxcruz|ruck, please open a critical bug re: any failing job15:10
weshayis no longer logging the output from gate_test_oooq.sh15:10
weshayit's not just our jobs :)15:10
weshayarxcruz|ruck, w/ tag alert15:10
arxcruz|ruckweshay: bug being backed15:11
weshaybacked?15:11
myoungarxcruz|ruck: back in 5 mins, available after that.  need to time-box.  All - if anyone wants/needs to (briefly) see how to kick TQ/TQE gates for debug/diagnosis on internal rhos-jenkins, feel free to join in 5 @ https://bluejeans.com/705085945515:11
arxcruz|ruckweshay: do you have a console log sample to put in the bug ?15:13
arxcruz|ruckweshay: as in baked* hehehe, as in bread15:13
* myoung thought he was just going crazy 15:13
weshayarxcruz|ruck, pick any red job http://tripleo.org/cistatus.html15:13
weshayarxcruz|ruck, search for hown -hR zuul /opt/git15:14
myounghttp://logs.openstack.org/41/566241/11/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/ee28a30/job-output.txt.gz#_2018-06-05_13_21_19_960739 isn't getting the exact same error but same result15:14
myoung(^^ found while looking at results from patches for https://trello.com/c/vJwmd6SD/771-indexing-upgrade-logs-in-logstash)15:15
weshayarxcruz|ruck, note.. the logs under logs for the run are actually there15:15
myoungarxcruz|ruck: let's chat after get critical ruck things covered first15:15
arxcruz|ruckweshay: rlandy|rover https://bugs.launchpad.net/tripleo/+bug/177520515:16
openstackLaunchpad bug 1775205 in tripleo "Failing jobs are no longer getting the output from gate_test_oooq.sh script" [Critical,Triaged]15:16
weshayarxcruz|ruck, also the ara logs are there w/ the playbook execution15:16
weshayarxcruz|ruck, dude15:16
weshayarxcruz|ruck, it's not a promotion blocker15:16
weshayarxcruz|ruck, alert15:16
arxcruz|ruckweshay: sorry, fixed15:17
weshayarxcruz|ruck, /me going to update it a bit15:17
weshayarxcruz|ruck, rlandy|rover fyi.. https://bugs.launchpad.net/tripleo/+bug/177520515:19
openstackLaunchpad bug 1775205 in tripleo "Failing jobs are no longer getting the output from gate_test_oooq.sh script" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)15:19
*** ccamacho1 has quit IRC15:19
weshayarxcruz|ruck, rlandy|rover can you guys investigate this and possibly work w/ #openstack-infra15:19
weshaynot sure why this is happening15:19
rlandy|roverweshay: yes - assigned the bug to me15:20
weshayperfect.. thanks all15:20
rlandy|roverchecking rdocloud logs15:21
rlandy|roverweshay: arxcruz|ruck: rdocloud logs are still good - https://logs.rdoproject.org/69/568869/10/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch/Zd0607eec00cc4648aab182726616b861/console.txt.gz15:21
rlandy|roverthere was a zuul update - maybe something changed - will check15:22
arxcruz|ruckrlandy|rover: my guess is the ansible config in upstream15:22
rlandy|roveransible config?15:22
weshayrlandy|rover, arxcruz|ruck see my latest update in the bug15:22
arxcruz|rucki mean, this jobs are now ansible right ?15:22
arxcruz|ruckat the end zuul calls some ansible code that run our script15:23
*** udesale has joined #oooq15:24
weshayrfolco_, so I don't see a bug on the container prep for upgrades15:24
weshayrfolco_, https://bugs.launchpad.net/tripleo/+bugs?field.searchtext=containers&search=Search&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.assignee=&field.bug_reporter=&field.omit_dupes=on&field.has_patch=&field.has_no_package=&orderby=-datecreated15:24
weshay&start=015:24
weshayrfolco_, let's open one up on the error.. fill out the details a bit.. then start an email thread w/ interested parties15:25
rfolco_weshay, yep, I also did the same here... no f* similar bugs15:25
rfolco_weshay, who do I include in the email thread15:25
rlandy|roverweshay: thanks - see your update - just checking a few more details - then will ping on openstack-infra15:25
weshayrlandy|rover, arxcruz|ruck when approaching the infra guys.. be as respectful and gracious as possible15:26
*** udesale has quit IRC15:26
arxcruz|ruckweshay: i'm a gentleman :)15:27
weshayarxcruz|ruck, you are sir.. truly15:27
rlandy|roverweshay: ack - which is why I don't want to approach them until we have full info15:27
weshayrlandy|rover++15:27
hubbotweshay: rlandy|rover's karma is now 215:27
weshayrlandy|rover++15:27
hubbotweshay: rlandy|rover's karma is now 315:27
rlandy|rover.me does not enjoy sounding like an idiot15:27
*** udesale has joined #oooq15:27
weshayrfolco_, probably worth scanning through https://github.com/openstack/tripleo-quickstart-extras/commits/master/roles/overcloud-prep-containers15:28
weshayand https://github.com/openstack/python-tripleoclient/commits/master15:28
rlandy|roverrasca: ^^ may have hit this as well15:28
*** udesale has quit IRC15:28
rlandy|roverarxcruz|ruck: is this only upgrades?15:29
rlandy|rovernvm15:29
arxcruz|ruckany failing job15:29
rlandy|rover PROJECTS='openstack/tripleo-upgrade openstack/tripleo-quickstart-extras openstack/tripleo-quickstart '15:29
*** udesale has joined #oooq15:29
*** jbadiapa has quit IRC15:30
*** udesale has quit IRC15:30
weshayrlandy|rover, arxcruz|ruck fyi.. I first noticed this last night15:31
rlandy|rovernot just failing jobs - all jobs15:31
weshaynot sure how far back it goes15:31
weshayrlandy|rover, all jobs too?15:31
weshayrlandy|rover, arxcruz|ruck hrm.. check any commits to tripleo-ci15:32
rlandy|rover2018-06-05 13:21:20.783196 | RUN END RESULT_NORMAL: [untrusted : git.openstack.org/openstack-infra/tripleo-ci/playbooks/tripleo-ci/run.yaml@master]15:33
*** udesale has joined #oooq15:33
*** sshnaidm|brq has quit IRC15:33
*** udesale has quit IRC15:34
*** ratailor has quit IRC15:34
rlandy|roverweshay: arxcruz|ruck; so this one is ok ... http://logs.openstack.org/94/571694/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/891e548/job-output.txt.gz#_2018-06-05_13_20_31_52961115:36
rfolco_weshay, https://bugs.launchpad.net/tripleo/+bug/177520915:37
openstackLaunchpad bug 1775209 in tripleo "skopeo permission denied on upgrade prep containers" [Critical,New]15:37
rfolco_lunch brb15:37
*** bogdando has quit IRC15:38
rlandy|roverhere we have a full log ... http://logs.openstack.org/24/567224/16/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/9b40aef/job-output.txt.gz15:39
*** ykarel|training has quit IRC15:41
arxcruz|ruckrlandy|rover: i think it's only when job fails15:47
arxcruz|ruckmyoung: whenever you want15:47
rlandy|roverarxcruz|ruck; no - commenting bug15:48
arxcruz|ruckok15:48
*** yolanda has quit IRC15:50
*** skramaja has quit IRC15:50
weshayanyone run a recreate lately?15:50
rlandy|roversee last comment15:51
rlandy|roveron bug15:51
rlandy|rover          sudo chown -hR $(whoami) /opt/git15:57
rlandy|rover          function gate_hook {15:57
rlandy|rover              bash -xe /opt/stack/new/tripleo-ci/toci_gate_test.sh15:57
rlandy|rover          }15:57
rlandy|roverweshay: arxcruz|ruck: no changes in tripleo-ci during that time16:01
rlandy|roverchange happened sometime late 06/0416:02
weshayk16:02
rlandy|roverarxcruz|ruck: what were you mentioning about ansible config upstream?16:03
rlandy|roverlooks like we have less verbosity16:03
rlandy|rover2018-06-05 00:26:16.072916 | PLAY RECAP16:03
rlandy|rover2018-06-05 00:26:16.073661 | primary | ok: 3 changed: 3 unreachable: 0 failed: 016:03
arxcruz|ruckrlandy|rover: i was just saying that our code is called by some ansible upstream in zuul16:03
*** links has quit IRC16:03
arxcruz|ruckmaybe the verbosity was reduced16:04
arxcruz|ruckrlandy|rover: because in ara you can see the output16:04
arxcruz|ruckbut not in the job-output16:04
rlandy|roverha - where is that set?16:05
*** yolanda has joined #oooq16:05
rlandy|roverweshay: ^^?16:05
arxcruz|ruckrlandy|rover: I'm guessing here, dunno if that what happens16:05
rlandy|roverwhere is here?16:05
arxcruz|ruckbut logically seems to be the case16:05
arxcruz|rucki'm guessing about the verbosity16:06
rlandy|rover"I'm guessing here" - where is the here you are referring to?16:06
myoungarxcruz|ruck: can now if you like.16:06
rlandy|roversee the diff in out put betn ...16:06
myoungarxcruz|ruck: time-boxed to 10 min16:06
arxcruz|ruckrlandy|rover: it was just an expression, i'm guessing here, as here in this case16:06
myoungarxcruz|ruck: won't take that long16:07
arxcruz|ruckmyoung: ok, joining16:07
*** zoli is now known as zoli|gone16:13
*** alee_ has quit IRC16:20
*** brault has joined #oooq16:22
*** jaganathan has quit IRC16:24
rlandy|roverweshay: do we have another contact on infra I should be pinging?16:25
*** brault_ has joined #oooq16:27
weshayrlandy|rover, your ping directly to paul will probably be ignored by the other infra folks16:28
weshayrlandy|rover, although I don't think it's a verbosity issue16:29
weshayor an ansible verbosity issue anyway16:29
*** jaganathan has joined #oooq16:29
*** brault has quit IRC16:29
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.16:31
arxcruz|ruckrlandy|rover: there are some stacks in central ci with 5 days old, i'm deleting ok ?16:33
rlandy|roverarxcruz|ruck: yep16:33
*** holser__ has quit IRC16:33
*** jfrancoa has joined #oooq16:35
*** jfrancoa has quit IRC16:37
EmilienMI would need someone from CI squad to help me on https://review.openstack.org/#/c/572215/ - this is not urgent so no worries16:40
arxcruz|ruckrlandy|rover: when you say 06/04 you mean mm/dd or dd/mm ?16:40
rlandy|roverarxcruz|ruck: I'm an American now - they write the date with month first :)16:41
rlandy|rover06/04 is june 4th16:41
arxcruz|ruckok, just to check how far should i search16:41
rlandy|roverbut I'll more explicit  in future16:41
rlandy|roverarxcruz|ruck,  the two logs I linked are where the switch happens16:42
arxcruz|ruckrlandy|rover: my confusion was exactly because you're not american native :)16:42
rlandy|roverarxcruz|ruck: I adapt :)16:42
rlandy|roverarxcruz|ruck: see infra channel16:42
rlandy|roverthey are working on it there16:42
weshayrfolco_, another use for sf https://trello.com/c/9CSqm0bw/796-epic-python316:42
*** myoung is now known as myoung|lunch16:43
arxcruz|ruckrlandy|rover: cool, so the problem was in zuul indeed :)16:43
*** brault has joined #oooq16:43
rlandy|roverarxcruz|ruck: I think so - watching conversation there16:43
weshayrlandy|rover, another data point16:46
weshayhttp://zuul.openstack.org/stream.html?uuid=43a892e9a8c247078e61753bb5304605&logfile=console.log16:46
weshaytotally blank16:46
*** jbadiapa has joined #oooq16:46
rlandy|roveroh how nice16:46
weshayrlandy|rover, probably related16:46
rlandy|roverweshay: infa is on it16:46
weshayoh /me looks16:46
rlandy|roversee updated bug16:47
rlandy|roverand conversation on channel16:47
*** brault_ has quit IRC16:47
weshayrlandy|rover++16:48
hubbotweshay: rlandy|rover's karma is now 416:48
weshaythanks16:48
rlandy|roverweshay: lol - does that mean I was sufficiently polite and gracious?16:49
weshayrlandy|rover, man.. I dread asking anything in that channel16:49
rlandy|roveroh whatever, I am without shame at this point16:49
weshayrlandy|rover, well.. the relationship was VERY tense about a year ago16:50
weshaywhen we were filling their log server and using all their nodes16:50
rlandy|roverweshay: k - will keep that in mind16:52
weshayrlandy|rover, we've done a really great job improving the gate jobs to near 100% pass rates16:52
rfolco_EmilienM, what can I do for you on https://review.openstack.org/#/c/572215/ ? A depends-on change to test those fsXX jobs ?16:52
weshayand reducing the logs16:52
*** brault has quit IRC16:53
*** kopecmartin has quit IRC16:53
*** alee_ has joined #oooq16:56
*** brault has joined #oooq16:57
*** jaosorior has quit IRC16:58
*** brault has quit IRC16:59
weshaytrown, congrats on your 5 years at RH :)17:03
trownweshay: is it today?17:05
weshayno.. but in a few17:05
trownweshay: I knew it was coming up... basically my daughters age +117:05
*** tosky__ has joined #oooq17:05
*** tosky has quit IRC17:05
*** tosky__ is now known as tosky17:05
rlandy|rovertrown: congrats!17:06
arxcruz|rucktrown: yay! congrats!!!17:14
arxcruz|ruckboth 5 years and your daughters+1 :D17:15
rlandy|roverok - back to rasca's failing job17:20
*** jaganathan has quit IRC17:22
*** dtantsur is now known as dtantsur|afk17:24
*** amoralej is now known as amoralej|off17:28
rfolco_trown, congrats! :)17:29
*** apetrich has quit IRC17:29
*** apetrich has joined #oooq17:30
myoung|lunchtrown: happy pending pentaversary!17:32
trownhaha17:34
*** trown is now known as trown|lunch17:38
*** myoung|lunch is now known as myoung17:45
*** ccamacho has joined #oooq17:46
*** gkadam has quit IRC17:52
*** myoung is now known as myoung|biaf17:59
*** ccamacho has quit IRC18:00
rlandy|roverrfolco_: weshay: I am looking at a failure in 'build-test-packages : Run DLRN' - did I see you guys discussing a similar failure earlier?18:02
* rlandy|rover thought I saw fly by about build-test-packages18:02
rlandy|roverhttps://logs.rdoproject.org/55/572155/2/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/Z4771a7ecff1b48d5937ce0bb555b8332/console.txt.gz#_2018-06-05_14_27_57_25518:02
rfolco_rlandy|rover, I've seen this in SF... missing deps18:03
rlandy|roverrfolco_: was there a fix? this is fs001 rdocloud18:03
rfolco_not sure this applies here, because there I had rhel7 / osp18:03
rfolco_missing deps were mock, python2-{future,stestr}18:04
rlandy|roverhmmm - probably diff18:04
rlandy|roverrfolco_: thanks - will invesitgate further18:04
rfolco_rlandy|rover, it does not look the same issue, sorry18:05
rfolco_rlandy|rover, previous step failed too18:07
rfolco_TASK [build-test-packages : Copy from ready directories if configured]18:07
weshaystandalone installer is working :)18:23
weshayhttp://logs.openstack.org/17/572217/6/check/tripleo-ci-centos-7-standalone/b2eb1b7/logs/ara_oooq/result/bb49965e-4fb7-43ea-a9e3-c227702c17de/18:24
rfolco_nice18:27
weshayrfolco_, that's another good candidate for internal sf18:27
rfolco_weshay, noted18:28
*** myoung|biaf is now known as myoung18:30
hubbotFAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/56044518:31
myoungstandalone...cool!  saw that review posted for that role and was looking at it last night18:32
*** trown|lunch is now known as trown18:38
EmilienMrfolco_: figure out why fs037 job is failing on https://review.openstack.org/#/c/572215/ would help me a lot18:41
*** EmilienM is now known as EmilienM|PTO18:43
weshayhrybacki, ping.. let me know when you have a minute to chat18:48
*** yolanda has quit IRC18:49
*** dtrainor has quit IRC18:50
weshayrlandy|rover, ^18:54
weshayrlandy|rover, arxcruz|ruck http://cistatus.tripleo.org/ --> tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates18:57
weshaybunch of these from 6/5 | Reason was NOT FOUND. Please investigate. |18:57
weshay%gatestatus18:57
rlandy|roverlooking18:57
hubbotFAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/56044518:57
*** links has joined #oooq18:59
weshayrlandy|rover, post_failure https://review.openstack.org/#/c/572215/19:00
weshayis what we're looking for19:00
rlandy|rover2018-06-05 10:58:22 | command: overcloud deploy -> tripleoclient.v1.overcloud_deploy.DeployOvercloud (auth=True)19:00
rlandy|rover2018-06-05 10:58:22 | Using auth plugin: password19:00
rlandy|rover2018-06-05 10:58:24 | No stack found, will be doing a stack create19:00
rlandy|rover2018-06-05 11:00:25 | Action tripleo.parameters.update execution failed: Error validating environment for plan overcloud: ERROR: Internal Error19:00
rlandy|roverwhat I see19:00
rlandy|rovererror in deploy itseld19:01
weshayhrm19:01
weshayhrm.. this is weird19:01
weshayprobably again related to infra19:01
weshayhttp://logs.openstack.org/15/572215/2/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/ef5c4b8/19:01
weshayhas no logs19:01
weshayEmilienM|PTO, ^19:01
rlandy|roverlol - now EmilienM|PTO is going on PTO - weshay and EmilienM|PTO must be one person - when one is here, the other one goes on PTO19:02
rlandy|roverthe no logs thing is infra19:03
rlandy|roverthere are a couple patches in flight to deal with that19:03
weshayk19:03
weshaythank you19:03
rlandy|roverthe failing post is something else19:03
weshayrlandy|rover, I suspect our hubbot jobs will catch this soon19:03
rlandy|roverweshay: depends19:04
rlandy|roverI am looking through a few more19:04
rlandy|roverusing ara19:04
rlandy|roverweshay: if you include the successes, the failures are not that much more common than usual but I suspect the lack of logs results in them all being reported as 'Reason NOT found'19:06
weshayya19:07
rlandy|roverstill confirming19:07
rlandy|roverto see if we have a common issue19:07
EmilienM|PTOrlandy|rover: one day and he tired me too much. I Need PTO :P19:09
*** jbadiapa has quit IRC19:10
rlandy|roverthis is the same review - different version ... http://logs.openstack.org/15/572215/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/041fda0/logs/undercloud/home/zuul/overcloud_deploy_post.log.txt.gz19:13
rlandy|roverthe rest don't show post deploy failures19:13
weshayEmilienM|PTO,19:13
weshayJe suis une petite fille qui aime les caniches roses19:13
EmilienM|PTOyou like dogs ? :-o19:14
weshaylolz19:14
rlandy|roverweshay: don;t see  much consistency in the failures - will see going forward19:20
weshaythanks19:20
rlandy|roverlooking at tripleo-quickstart-extras-gate-newton-delorean-full-minimal19:23
*** yolanda has joined #oooq19:29
rlandy|roverweshay: ^^ the last time that job passed was once on may 9th19:30
rlandy|roverchecking why19:31
*** links has quit IRC19:46
*** sshnaidm|brq has joined #oooq19:53
rlandy|roverweshay: https://bugs.launchpad.net/tripleo/+bug/1773445 is to deal with tripleo-quickstart-extras-gate-newton-delorean-full-minimal failure19:56
openstackLaunchpad bug 1773445 in tripleo "tripleo-quickstart-extras-gate-newton-delorean-full-minimal fails to install undercloud - Access denied for user 'heat'@'192.168.24.1" [High,Triaged] - Assigned to Arx Cruz (arxcruz)19:56
rlandy|roverassigned to arx19:57
weshayrlandy|rover, k.. thanks20:20
hubbotFAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56722420:31
*** holser__ has joined #oooq20:35
rlandy|roverweird -  tripleo-ci-centos-7-undercloud-upgrades looks green20:38
rlandy|roverone failure on stable/queens20:38
rlandy|roverweshay: hello - backport question for you20:38
rlandy|roverhttps://review.openstack.org/#/c/572155/20:39
* weshay looks20:40
rlandy|roverhttp://logs.openstack.org/55/572155/2/check/openstack-tox-py27/9c630a7/job-output.txt.gz#_2018-06-05_15_41_49_87691620:40
rlandy|roverweshay: ^^ the tox error is not in a file we changed20:40
rlandy|roverbugger problem20:41
rlandy|roverbigger20:41
rlandy|roversee build-test-packages error ...20:41
weshayrlandy|rover, ya.. but those are all check jobs20:41
weshayrlandy|rover, not sure what you are after here20:41
rlandy|roverweshay: yes - but we have a build error20:41
rlandy|rovergetting20:42
*** brault has joined #oooq20:42
rlandy|roverhttps://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/981/console20:42
weshayrlandy|rover, oh20:43
weshayk20:43
rlandy|roverhttps://logs.rdoproject.org/55/572155/2/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/Z310db5aeb6dd4f5586cfba0c2e7fb425/console.txt.gz#_2018-06-05_16_47_39_74420:43
weshayso /me checks the upstream job20:43
weshaythey will need to fix that20:43
rlandy|roverweshay: we need this backport to get rid of a ci error20:43
rlandy|roverbut we missed something20:43
rlandy|roverthis is stable/queens20:43
weshayhttp://logs.openstack.org/55/572155/2/check/tripleo-ci-centos-7-containers-multinode/7b33b5a/logs/delorean_logs/07/fb/07fbbe2a9448b9c6ab8449f1a6c7f8777a06dce4_dev/build.log.txt.gz20:44
weshayis that the same error /me checks20:44
weshayrlandy|rover, what did we miss?20:44
weshayrlandy|rover, what is indicating to you this is a ci/quickstart issue?20:45
rlandy|roverlooking20:45
*** trown is now known as trown|outtypewww20:46
rlandy|roverweshay; what concerns me is the drln build error20:46
weshayrlandy|rover, that happens from time to tome20:46
weshaytime20:46
rlandy|rover["cp: cannot stat ‘/opt/stack/new/python-tripleoclient’: No such file or directory"]20:46
rlandy|rover^^ that really?20:46
rlandy|roverso I can retry and hope for better days?20:47
rlandy|roverthe check errors on tox must be unrelated20:47
weshayrlandy|rover, hrm.. so we obviously can not log that dir.. but it would be nice to get a ls -la from it logged20:47
rlandy|roverI guess I'll retry - yep - we need better logging20:48
weshayrlandy|rover, looking what we have20:49
rlandy|roveryeah - me too20:49
weshayrlandy|rover, what line is the "no such file on20:50
weshayrlandy|rover, bluejeans me in20:50
weshayI want to see what you are seeing20:50
rlandy|roverok20:51
weshayrlandy|rover, my blue20:51
*** holser___ has joined #oooq21:02
*** holser___ has quit IRC21:03
*** holser__ has quit IRC21:06
*** myoung is now known as myoung|off21:08
*** sshnaidm|brq has quit IRC21:58
*** brault has quit IRC22:03
*** atoth has quit IRC22:03
hubbotFAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/56722422:31
rlandy|roverweshay: confirmed rasca's change is still failing the unit test on build - but now we know what it is - will pick it up with him tomorrow23:17
rlandy|roverweshay: need to step away ... bbl23:22
*** rlandy|rover is now known as rlandy|rover|bbl23:22
*** tosky has quit IRC23:30
« Monday, 2018-06-04 Index Wednesday, 2018-06-06 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!