Thursday, 2018-06-07

hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.00:32
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.02:32
*** rlandy|rover|bbl is now known as rlandy|rover02:35
*** rlandy|rover has quit IRC02:39
*** d0ugal_ has joined #oooq02:50
*** d0ugal has quit IRC02:51
*** udesale has joined #oooq04:02
*** jaganathan has quit IRC04:05
*** jaganathan has joined #oooq04:05
*** ccamacho has quit IRC04:05
*** links has joined #oooq04:28
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.04:33
*** brault has joined #oooq04:44
*** brault has quit IRC04:48
*** pgadiya has joined #oooq05:08
*** pgadiya has quit IRC05:08
*** ykarel|training has joined #oooq05:17
*** skramaja has joined #oooq05:20
*** alee has quit IRC05:41
*** saneax has joined #oooq05:56
*** ykarel|training has quit IRC06:02
*** ykarel|training has joined #oooq06:05
*** ykarel|training has quit IRC06:14
*** quiquell|off is now known as quiquell06:22
*** holser__ has joined #oooq06:30
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.06:33
*** zoli is now known as zoli|wfh06:38
*** zoli|wfh is now known as zoli06:38
*** ccamacho has joined #oooq06:52
*** kopecmartin has joined #oooq07:04
*** gkadam has joined #oooq07:07
*** amoralej|off is now known as amoralej07:14
*** tosky has joined #oooq07:26
*** brault has joined #oooq07:47
*** jaosorior has joined #oooq07:52
*** tcw has quit IRC08:05
*** jaosorior has quit IRC08:10
*** tcw has joined #oooq08:13
*** ykarel|training has joined #oooq08:26
*** ratailor has joined #oooq08:28
*** d0ugal_ has quit IRC08:29
*** d0ugal has joined #oooq08:29
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.08:33
*** jaosorior has joined #oooq08:54
*** dtantsur|afk is now known as dtantsur09:09
*** links has quit IRC09:48
*** links has joined #oooq10:05
*** ykarel|training has quit IRC10:30
quiquellarxcruz|ruck: Another two legit alarms about master and queens promotions10:33
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.10:33
arxcruz|ruckquiquell: where?10:33
quiquell#tripleo-ci10:33
quiquellarxcruz|ruck: Have check on http://rhos-release.virt.bos.redhat.com:3030/rhosp and it's legit10:33
quiquellarxcruz|ruck: but is master not pike, the alarm is wrong10:34
arxcruz|ruckquiquell: there's an issue with master, hrybacki is working on that10:35
quiquellarxcruz|ruck: Ok, was just checking the alarm system, thanks10:35
* quiquell give another beer to arxcruz|ruck10:35
arxcruz|ruckquiquell: also a problem with queens10:35
arxcruz|ruckboth are known issues so, we are "good"10:36
arxcruz|rucki check this morning ;)10:36
quiquellarxcruz|ruck: nice10:36
quiquellarxcruz|ruck: sorry for the noise10:36
arxcruz|rucknp10:36
quiquellarxcruz|ruck: pike is ok ?10:38
quiquellhttps://dashboards.rdoproject.org/pike two days too10:39
*** links has quit IRC10:39
quiquellhttp://rhos-release.virt.bos.redhat.com:3030/rhosp is not accurate there10:39
*** apetrich has quit IRC10:40
arxcruz|ruckquiquell: i think it just not run https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-pike-current-tripleo/10:42
arxcruz|ruckquiquell: seems to be ok, last run was june 510:42
quiquellarxcruz|ruck: 4 days was the alarming threshold ?10:43
arxcruz|ruckquiquell: 510:44
arxcruz|ruckquiquell: it seems there's no hash to promote10:44
arxcruz|ruckhttp://38.145.34.55/pike.log10:44
arxcruz|ruck2018-06-07 10:40:37,576 629 DEBUG    promoter Remaining hashes after removing already promoted ones: []10:44
quiquellarxcruz|ruck: ok, the alarm is legit too, thanks10:45
* quiquell gives directly the beer keg to arxcruz|ruck10:46
*** links has joined #oooq10:56
*** udesale has quit IRC11:07
*** ykarel|training has joined #oooq11:38
*** brault has quit IRC11:42
*** tcw has quit IRC11:55
*** tcw has joined #oooq11:55
*** amoralej is now known as amoralej|lunch11:59
*** jbadiapa has joined #oooq12:12
*** brault has joined #oooq12:13
weshayquiquell, arxcruz|ruck you discussing promotion_blocker bugs on the dashboard?12:22
arxcruz|ruckweshay: no, i just notice your message there regarding the keystone one12:23
weshayquiquell, btw.. came up w/ a new design for the build-test-packages work we discussed earlier12:23
weshayarxcruz|ruck, oh.. ya.. I removed it.. I don't want to freak out hrybacki :)12:23
arxcruz|rucklol12:23
arxcruz|ruckno pressure hrybacki, but the world needs you12:23
weshaybut I do want to make it clear to jbuchta that one bug.. can set the branch to red12:24
quiquellweshay: Good morning, reimplementing it, sorry about my previous fucked up patchset12:24
weshayquiquell, there was nothing that bad about your patchset12:24
weshayand it implemented the design we discussed12:24
quiquellweshay: btw, to test it i have use this https://review.openstack.org/#/c/571435/12:24
quiquellweshay: harcoded n -> n + 1, trown was asking12:25
weshaysometimes it just takes a couple days and review to come up w/ a better design12:25
*** apetrich has joined #oooq12:30
arxcruz|ruckquiquell: was you who saw this error before? 2018-06-07 09:33:51 | time="2018-06-07T09:33:51Z" level=fatal msg="Error determining repository tags: Get https://registry-1.docker.io/v2/tripleomaster/centos-binary-heat-engine/tags/list: EOF"12:30
quiquellarxcruz|ruck: Similar, could not resolve registry-1.docker.io host12:32
quiquellThis one is more about docker.io returning an empty file12:32
weshayrasca, ping https://bugs.launchpad.net/tripleo/+bug/177383712:32
openstackLaunchpad bug 1773837 in tripleo "Overcloud deployment failed with node unreachable even if the node is available" [Critical,Triaged]12:32
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.12:33
rascaweshay, hey12:33
weshayrasca, you have a second?12:34
rascaweshay, always for you12:34
weshayrasca, oh baby.. /me puts on my dress https://bluejeans.com/u/whayutin/12:34
arxcruz|ruckquiquell: how can i check on your r&r stuff how many times a particular job fails ?12:37
quiquellarxcruz|ruck: what job ?12:38
arxcruz|ruckquiquell: tripleo-ci-centos-7-undercloud-containers12:38
rfolcocores could please push this forward (fs037 queens on check) ? https://review.openstack.org/#/c/570902/12:38
quiquellarxcruz|ruck: two days ?12:41
*** trown|outtypewww is now known as trown12:41
quiquellrfolco: also the other ones are needed12:41
arxcruz|ruckquiquell: two days failing?12:41
quiquellarxcruz|ruck: You want to see how many times it's failing for how long, 2 days ?12:41
rfolcoquiquell, aren't those fake patches ?12:42
arxcruz|ruckquiquell: yeah, 2 days is okay12:42
*** jbadiapa has quit IRC12:44
quiquellrfolco: nope, they serve two purposes, trigger fs037 on master and also, specify files that trigger it12:44
*** bogdando has joined #oooq12:45
*** bogdando has quit IRC12:45
quiquellArx http://38.145.34.131:3000/d/_6Qu9TIik/arx?panelId=2&fullscreen&orgId=112:48
weshaytrown, 1-113:01
weshayhttps://bluejeans.com/4113567798/13:01
weshayquiquell, arxcruz|ruck what is your opinion re: listing out the lp's that are triaged in tripleo w/ promotion_blocker13:02
arxcruz|ruckchandankumar: the job you asked failed only twice, perhaps was some network issue when trying to connect to docker.io13:03
arxcruz|ruckweshay: looks okay to me13:04
quiquellweshay: That's a good one, now in the r&r we have the urgents only13:04
quiquellweshay: Every stuff that is in the r&r doc and can be put in the dashboard will help13:05
*** myoung|off is now known as myoung13:05
quiquellmyoung: Good morning, sorry I was not here for you yesterday13:05
myoungquiquell: morning!13:06
myoungquiquell: timezones and schedules are real constructs, no worries.13:06
*** skramaja has quit IRC13:06
quiquellmyoung: Do you want some help with https://trello.com/c/pfQ867XP/779-differentiate-the-same-featureset-to-do-the-two-begin-release-end-release-combinations ?13:07
* myoung joins #tripleo-ci, he didn't know it was a thing :)13:09
*** udesale has joined #oooq13:09
myoungquiquell: catching up on scrollback and cards.  Yesterday I went thru trown's patch to the python script (including the base script itself created last sprint), and had digested your changes and created some patches to invoke the workflow where we actually build new RPM's for THT.  That was the background work to understand context for what we'll need to change in TOCI to enable leveraging all these deltas.  I don't think the redesign proposed13:12
myoungyesterday by weshay/trown changes that.13:12
*** ratailor has quit IRC13:13
*** apetrich has quit IRC13:13
myoungregarding differentiate, I think the DoD shoudl probably include cleaving and/or adding jobs that use the (to be added) variable/input (both ways)13:13
quiquellmyoung: Will try to copy the DoD from the inject at the end it's similar13:14
myoungquiquell: I think the small POC / patches you posted last week are sane/rational to support this.  happy to sync on this and your card in a few mins, I'm checking on the results of last night's patches to THT to start covering the base cases (1 patch, 2 patches, multiple branches, etc).  I think they will need to be rebased and/or re-run, using the jobs that ran last night as starting point / baseline13:15
myoungtrown: regarding the QE tasks for your modifications to the python script, I'm not sure what's left there, you included UT's for your changes that look complete.  I'm planning today/tomorrow to just run those UT's locally and test via inspection, until we have all this wired up end to end I'm not sure what else there is to QE  for that card13:16
myoungtrown: will +1 your patches when that's done (this afternoon)13:17
*** atoth has joined #oooq13:19
myoungquiquell: ahh... https://review.openstack.org/#/c/572736 has PS 3,4,5,6 :)  /me looks...13:21
myoungtime-shifted team is amazing.  go to bed, wake up, nearly 5 hours of dev occurs hah.  eat.13:21
myoungneat.13:21
quiquellmyoung: coding time machine13:22
*** apetrich has joined #oooq13:26
*** ykarel|training has quit IRC13:27
myoungquiquell: i find it exhilerating / racing, and sometimes challenging as we don't really have designs for things that go more than a few days into the future.  So can spend time understanding/grokking an approach, kick off tests/jobs, wake up and iteration has already happened.  It's super cool/fast, but requires some degree of thrash in the absense of more concrete planning.  Will talk in retro...  To be clear...IMHO we're running really fast,13:28
myoungand that's awesome.13:28
quiquellmyoung: you can not have it all13:29
* myoung is happy to talk after scrums, he needs a quiet 30 to catch up / reviews.13:29
* myoung smiles at quiquell and nods in agreement.13:29
myoungquiquell: The flip side is also worth discussing in retrospective.  I think it might be interesting if you / arxcruz|ruck / panda or others that wake up while the Americas are sleeping and are solo on the early side shared "a day in the life" - don't want to belabor here/now.  TL;DR I would listen with interest experience(s) around this topic.13:32
quiquellmyoung: playing well we can benefict from the follow the sun approach13:34
hrybackiarxcruz|ruck: running reproducer w/ changes now :)13:38
*** rlandy has joined #oooq13:46
*** rlandy is now known as rlandy|rover13:47
rlandy|roverarxcruz|ruck: hi there - sorry about late arrival - how are things?13:47
arxcruz|ruckrlandy|rover: hey, no problem, got your message, everything is same as yesterday13:48
arxcruz|ruckno big news13:48
rlandy|roverarxcruz|ruck: cool - thanks13:48
myoungquiquell: left comments circa PS6, but I see you're still iterating, catch up after scrums if you have time today13:51
*** brault has quit IRC13:52
weshayarxcruz|ruck, how you doing w/ the get_nodes issue?13:52
arxcruz|ruckweshay: working on it submited a new patchset13:53
arxcruz|rucknow the issue is that the client isn't get authenticated13:53
arxcruz|ruckbut i'm investigating13:53
quiquellmyoung: Yep, testing in the RDO cloud and uploading changes13:53
quiquellmyoung: I can wait for it to pass to upload to gerrit13:53
myoungCI squad: o/  morning!  scrum in 5 min13:55
quiquellmyoung: Have tech difficulties13:57
hrybackiarxcruz|ruck: looks like mistral side has a fix in. I'm testing the puppet-keystone fix now13:57
arxcruz|ruckhrybacki: cool :)13:58
weshayarxcruz|ruck++13:58
hubbotweshay: arxcruz|ruck's karma is now 313:58
weshayarxcruz|ruck, re: doc creation13:59
arxcruz|ruckoh, forgot about that one, let me check13:59
arxcruz|ruckweshay: https://review.openstack.org/#/c/568783/ need a +2 +w13:59
weshayarxcruz|ruck, /me is going to +2 and merge as I see things working.. I don't see tempest doc in your patch.. which I suspect is just a depends-on issue13:59
weshayw/ chandankumar's patch13:59
weshaybut it's good to merge13:59
* weshay looking at14:00
weshayhttps://logs.rdoproject.org/83/568783/5/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Zac4551e603da4f7083dbf1049b596103/docs/build/14:00
weshayhttps://thirdparty.logs.rdoproject.org/jenkins-tqe-gate-rhos-12-ci-rhos-ovb-minimal-pacemaker-public-bond-1201/docs/build/overcloud-validate.html14:00
quiquellmyoung: pass me the blue jeans link please14:00
arxcruz|ruckweshay: https://review.openstack.org/#/c/566527/14:00
arxcruz|ruckalso +2 +w14:00
myoungCI Squad: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting, https://bluejeans.com/705085945514:00
weshayarxcruz|ruck, chandankumar ++++=14:00
myoung^^ (or bluejeans.com/matyoung)14:00
rlandy|roverarxcruz|ruck: weshay: here's a first since April 13 - https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo-delorean-promote-newton/14:02
myoungweshay, (rlandy|rover || arxcruz|ruck), panda - CI scrum?14:02
arxcruz|ruckmyoung: ruck and rover don't participate on ci scrum14:03
arxcruz|ruckafaik14:03
weshayruck does not14:03
weshayrover does14:03
rlandy|roverI am there14:03
arxcruz|ruckbut if you want, just a sec, i'll join you14:03
rlandy|roverarxcruz|ruck: np - I am covering us14:03
*** gkadam has quit IRC14:03
*** udesale has quit IRC14:03
myoungthx...is just the first few mins14:03
*** gkadam has joined #oooq14:04
*** udesale has joined #oooq14:04
*** dtrainor has joined #oooq14:06
*** amoralej|lunch is now known as amoralej14:07
*** links has quit IRC14:23
trownquiquell: can I quickly bug you about comments on UT for https://review.openstack.org/#/c/572420/14:27
trownquiquell: I think I understand the confusion from your last comment now... those unittests do not test get_dlrn_hash, we have other unittests to test get_dlrn_hash14:28
trownquiquell: ie we are not "excercising target_hash = get_dlrn_hash(target_release, CURRENT_HASH_NAME)" we are "excercising assert (compose_releases_dictionary(stable_release, featureset,  upgrade_from) == expected_releases)"14:29
trownquiquell: we specifically mock get_dlrn_hash entirely, because we are not testing it there14:30
trownquiquell: thanks for your reviews btw, the 'false' vs False issue is legit14:31
trownI will fix that up14:31
quiquelltrown: we mock get_dlrn_hash what I mean is that we are exercising the use of it14:32
weshayarxcruz|ruck, no luck re: get_nodes :(14:33
weshayarxcruz|ruck, I wonder if we're looking at the right thing14:33
quiquelltrown: the use have to be to pass A y and get B, to test it we A and B have to be different.14:33
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.14:33
weshayarxcruz|ruck, wonder if you should spend some time to better understand if/when the task is executed vs skipped14:33
quiquelltrown: If you want we can check on bj14:33
trownquiquell: sure14:35
trownquiquell: lets talk on bluejeans14:35
arxcruz|ruckweshay: my bad... :(14:35
weshayarxcruz|ruck, no worries mate.. just trying to help14:36
quiquelltrown: give a minute to set up blue jeans (have a fresh fedora now)14:38
trownquiquell: your example of the code breaking will also happen if we change the mock side_effect to ['foo', 'bar'] and change all the expected releases to match...14:39
quiquelltrown: I have to be missing something14:39
quiquelltrown: calling14:41
*** ccamacho has quit IRC14:42
rlandy|roverchandankumar: https://review.openstack.org/#/c/572858/? Doesn't look like it picked up tempest14:43
trownquiquell: on your bluejeans?14:44
quiquelltrown: I am at your bj14:44
rascatrown, hey man, can I have your eyes on this https://review.openstack.org/#/c/573255/1/roles/collect-logs/tasks/collect.yml ? myoung suggestion feels really good to me, but what we prefer in general for oooq? Is it better a default in defaults/main.yml or is an addition to the docs enough and so we can compact like matt suggested?14:44
myoungtrown, rasca, updated with a +114:45
trownquiquell: pinche bluejeans... some day I will get there14:45
myoungfine with either approach, both have merit14:45
myoungtrown, quiquell, mind if I listen in ?14:45
quiquellmyoung: no problem from my side14:46
myoungquiquell: link/14:46
myoung?14:46
quiquellmyoung: trowns room14:46
trownbluejeans is completely failing to load for me :(14:46
quiquelltrown: redjeans !!!14:47
myoungtrown: has been happening to me too...hence the phone useage as a backup when I run meetings14:48
*** apetrich has quit IRC14:48
*** saneax has quit IRC14:48
myoungtrown: what's your meeting id#?  sometimes having someone else join it kicks it over on server side (join existing room vs. first create)14:48
trownmyoung: u/jtrowbri14:49
myoungwow...that crashes for me too14:50
myoungheh BJ: "Oops... we'll be right back"14:50
myoungmy room is open/working for another 10 mins...or we can use quiquell's room14:51
myoungtrown: maybe something with your room specifically14:51
myoungand it's working14:51
rlandy|roverarxcruz|ruck: hi - looking at the newton failures ... is this different to the one you are working on?https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-newton-delorean-minimal-954/undercloud/home/stack/undercloud_install.log.gz14:55
rlandy|rover2018-06-07 10:17:49 | 2018-06-07 10:17:49 - [mNotice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'192.168.24.1' (using password: YES)")[0m14:56
arxcruz|ruckrlandy|rover: i think the one i was working on is a dead end, the guys from certmonger puppet doesn''t want to merge my patch14:56
rlandy|roverarxcruz|ruck: :(14:56
arxcruz|ruckrlandy|rover: yes, this is the problem all because of dbus, so keystone doesn''t create the credentials14:56
rlandy|roverarxcruz|ruck: hmmm ... we need to decide what to do here ultimately14:57
rlandy|roverdo you have the bug handy?14:57
rlandy|roverlet's add these failures as well so they are tracked14:57
rlandy|rovernow that we actually get to run this testing14:57
arxcruz|ruckrlandy|rover: i need to check the bug14:58
myoungchandankumar, kopecmartin, tempest squad scrum15:01
*** saneax has joined #oooq15:04
*** apetrich has joined #oooq15:04
rlandy|roverarxcruz|ruck: requesting your review - to add to promotion https://review.rdoproject.org/r/#/c/14093/15:04
*** quiquell is now known as quiquell|off15:04
*** d0ugal has quit IRC15:07
rlandy|roverarxcruz|ruck: weshay:doing cleanup of openstack-nodepool tenant - 7 stacks that should go - nothing major15:07
*** zoli is now known as zoli|gone15:07
*** zoli|gone is now known as zoli15:07
rlandy|rovermake that 8 stacks15:09
rlandy|roverwhen is panda coming back?15:15
weshayhttps://bugzilla.redhat.com/buglist.cgi?action=wrap&bug_status=NEW&bug_status=ASSIGNED&bug_status=POST&bug_status=MODIFIED&bug_status=ON_DEV&bug_status=ON_QA&bug_status=VERIFIED&bug_status=RELEASE_PENDING&chfield=%5BBug%20creation%5D&chfieldto=-9d&f1=product&f10=flagtypes.name&f11=CP&f12=cf_internal_whiteboard&f2=cf_internal_whiteboard&f3=component&f4=cf_conditional_nak&f5=cf_qe_conditional_nak&f6=OP&f7=keywords&f8=priority&f9=bug_severity&ke15:15
weshayywords=FutureFeature%2C%20Tracking%2C%20Documentation%2C%20&keywords_type=nowords&list_id=8949811&n6=1&o1=equals&o10=substring&o12=equals&o2=substring&o3=notsubstring&o4=isempty&o5=isempty&o7=substring&o8=notsubstring&o9=notsubstring&order=changeddate%2Copendate%2Cbug_id&query_format=advanced&saved_report_id=1949&v1=Red%20Hat%20OpenStack&v10=rhos&v12=DFG%3ARDOCI&v2=DFG%3A&v3=doc&v7=Triaged&v8=unspecified&v9=unspecified15:15
rlandy|roverthat was not the answer I was expecting :)15:16
*** saneax has quit IRC15:17
*** apetrich has quit IRC15:22
-openstackstatus- NOTICE: Zuul update for Ansible 2.5 in progress. Scheduler crashed as unexpected side effect of pip upgrade. Will be back and running shortly.15:29
rlandy|roverarxcruz|ruck: ^^ happy times15:40
arxcruz|ruckrlandy|rover: (╯°□°)╯︵ ┻━┻15:40
arxcruz|ruckwe are done for today then :P15:41
rlandy|roverarxcruz|ruck: lol - not quite - I have phase 2 work15:41
arxcruz|ruckrlandy|rover: actually, i'm almost done for today, for real, it's almost 6pm here :P15:41
rlandy|roverarxcruz|ruck; ack - will watch for any disasters on return15:42
rlandy|roverhave a good night15:42
arxcruz|ruckrlandy|rover: but of course, if you need  me for something just ping, i'm always closer to my computer :P15:42
rlandy|roverk - hope not to have to disturb you late at night15:43
*** holser__ has quit IRC15:46
chandankumararxcruz|ruck: https://review.openstack.org/#/q/topic:tempestconf_200+(status:open+OR+status:merged)15:50
arxcruz|ruckchandankumar: what's the job that uses featureset10 ?16:00
-openstackstatus- NOTICE: The zuul upgrade to ansible 2.5 is complete and zuul is running again. Changes uploaded or approved between 15:25 and 15:45 will need to be rechecked. Please report any problems in #openstack-infra16:09
rlandy|rover%gatestatus16:15
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.16:15
chandankumararxcruz|ruck++16:22
hubbotchandankumar: arxcruz|ruck's karma is now 416:22
*** myoung is now known as myoung|biaf16:23
*** myoung|biaf is now known as myoung|lunch16:32
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.16:33
chandankumarmyoung|lunch: weshay can we get a push on this https://bugzilla.redhat.com/show_bug.cgi?id=157493116:45
openstackbugzilla.redhat.com bug 1574931 in documentation "Update the tempest documentation for RHOS-13" [High,New] - Assigned to rhos-docs16:45
weshaychandankumar, a push?16:46
weshaymeaning to next release?16:46
chandankumarweshay: nope, getting it done16:46
*** tbarron has quit IRC16:46
chandankumarweshay: with rhos-13 release16:46
weshay14ok?16:47
weshayshould be16:47
chandankumarweshay: As last discussed we have kept tempest container as tech preview16:47
chandankumarweshay: so we need documentation update16:47
weshaydone16:48
chandankumarweshay: thanks16:48
*** tbarron has joined #oooq16:50
*** ccamacho has joined #oooq16:57
*** jaganathan has quit IRC16:59
weshaymyoung|lunch, re: https://review.openstack.org/#/c/544696/13/tasks/upgrade/main.yml I suggest removing the log commands from the scripts in question and using the established pattern we have in ci17:01
*** dtantsur is now known as dtantsur|afk17:04
*** kopecmartin has quit IRC17:07
*** ccamacho has quit IRC17:08
*** ccamacho has joined #oooq17:08
*** ccamacho has quit IRC17:09
*** ccamacho has joined #oooq17:09
*** gkadam has quit IRC17:14
*** ccamacho has quit IRC17:21
*** ccamacho has joined #oooq17:21
*** trown is now known as trown|lunch17:25
*** udesale has quit IRC17:28
*** atoth has quit IRC17:30
*** myoung|lunch is now known as myoung17:32
*** amoralej is now known as amoralej|off17:33
myoungweshay: ack17:35
weshayarxcruz|ruck, rlandy|rover you guys familiar w/ the cix board?17:39
weshayarxcruz|ruck, rlandy|rover fyi.. if we opened a promotion_blocker it will land in New/Triage.  We need to move to "Critical PChain Outage" or "CIX Failing Jobs"17:40
weshayand set the appropriate trello tag for the release17:40
weshayquiquell|off, would be interesting to add to the dashboard.. any card in https://trello.com/b/j4IcIomh/production-chain-escalation in the new/Triage column that contains [tripleoci]17:41
*** atoth has joined #oooq17:42
*** atoth has quit IRC17:46
rlandy|roverweshay: yes ... moving that card17:47
weshayrlandy|rover, thanks17:47
weshayrlandy|rover, we could do this automatically.. w/ a keyword in the bug name.. or a tag17:48
weshayprobably is annoying at this point17:48
rlandy|roverweshay: well, is there a reason the card does not land in Critical PChain Outage automatically?17:49
rlandy|roverwe always move it there17:49
rlandy|roveraddthe bug17:49
weshayrlandy|rover, the process and design is have a human do it.. but meh17:49
weshayrlandy|rover, I have to rewrite the code that does this anyway for storyboard17:49
rlandy|roverit just seems we always follow the same process17:50
weshayso.. I'll take that as an RFE17:50
weshayit's not 100%, but close17:50
rlandy|roverweshay: well, we may need to keep updating the cards17:50
rlandy|roveras we add people and assign RCA17:50
*** atoth has joined #oooq17:50
rlandy|roverit would be super nice if it did that automatically17:51
weshayk k17:56
weshayrlandy|rover, fyi.. the undercloud install bug looks fixed for master https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-containers-build/986/console17:57
rlandy|roverooo - good17:58
rlandy|roverweshay: now to see what lurks behind that17:59
rlandy|roverno job runs for a few days17:59
weshayrlandy|rover, ya18:00
*** saneax has joined #oooq18:09
*** apetrich has joined #oooq18:11
rlandy|rovertrown|lunch: myoung: weshay: I had to revert this change because I switched the order of playbook arguments and it broke periodic. I resubmitted keeping playbook args in tact. Checked with periodic reproducer and we get the correct containers (rdoproject registry) projects ones now. Please review. we should not merge this on a fruday again - in case I killed something else :(18:16
weshayrlandy|rover, which review?18:16
rlandy|roveroh sorry18:17
rlandy|roverhttps://review.openstack.org/#/c/570694/18:17
rlandy|roverkind of an important detail18:17
weshayheh.. /me looks18:22
weshayrlandy|rover, fyi https://review.openstack.org/#/c/553465/518:22
myoungweshay: thanks for feedback, and good idea.  I've implemented your feedback, and also tried to make this a little more clear here: https://review.openstack.org/#/c/544696/13..1518:26
myoungrlandy|rover, weshay: regarding automatically moving to critical outage for all cards, if I recall correctly from setting up process, some folks felt strongly that cards are not moved to critical outage until (specifically) a human puts it there, as there are SLA and update implications (update every 4 hours was a goal talked about) for things in that column.  Pragmatically / realistically I agree, and we nearly always indeed move it there....18:29
myoungbut IMHO there might be some parties that disagree with the autotriage --> pchain outage idea.  {shrug}18:29
myoungrlandy|rover: looking18:30
myoung@review18:30
weshaymyoung, have you double checked that we haven't mixed update/upgrade w/ template and log names? https://review.openstack.org/#/c/544696/15/tasks/update/create-undercloud-update-scripts.yaml18:32
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.18:33
myoungweshay: in looking at the logging, we are today doing 2 things that were confusing...  1.) using undercloud_upgrade.sh.j2 for *both* update and upgrade workflows.  2.) had 2 seperate copies of undercloud_upgrade.sh.j2 - one used for both (normal) upgrade+update, and a second (different yet identically named) template for FFU upgrades.  I've renamed the latter with an ffu_ prefix.18:37
*** trown|lunch is now known as trown18:38
myoungfor #1 I created 2 different variables that point to the same file, opting not to rename it for fear of making too many changes at once (described in commit message at end), and for #2 the rename18:38
* myoung looks at it again anywahy18:38
rlandy|roverweshay: fyi ... I made this adjustment for env E when running fs001 ... http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/hardware_environments/hp_dl360_envE/network_configs/single_nic_vlans/env_settings.yml#n4118:40
rlandy|roverI am not sure if that will be the same on all envs - probably we can fix it18:40
rlandy|roverI am duplicating it in the review for the other envs for testing - I can adjust if you think it's no good18:41
weshayrlandy|rover, line 4118:42
weshayis probably best left for the fs ya?18:42
rlandy|roverweshay: it's in the fs18:42
rlandy|roverbut if we use fs001 as is, it failed that step18:42
rlandy|roverI took it out to re-enable the env for master18:42
weshayoh18:42
rlandy|roverbut now that we are redoing the whole lot, we should think about that more18:43
weshayso the deployment fails w/ ssl overcloud18:43
weshayhuh18:43
rlandy|roverit did in my initial tests18:43
rlandy|roverbut that was rather quick18:43
rlandy|roverI did what we needed to get the env back18:43
weshayya18:43
weshayunderstood18:43
rlandy|roverok - just something to review in new set18:43
rlandy|roverI will test on another env with ssl enabled - let's see18:44
weshaynice containers in master/queens were built successfully18:46
rlandy|roverand osp gate is not looking good - will check that out later18:46
rlandy|roverupstream zuul looks still functional - that's good18:47
*** gkadam has joined #oooq18:51
myoungtrown, rfolco, quiquell|off, panda, weshay: regarding https://trello.com/c/vJwmd6SD/771-indexing-upgrade-logs-in-logstash, effectively this is my patch (that have just reworked)...I'm happy to be QE + author for this (minor) card, but with TC hat on I see QE is patch author.  If anyone wants to have a look once CI on it is done might make sense.  Probably not a big deal either way.18:53
myoung(well it is my patch lol)18:54
myoungs/effectively//18:54
weshaymarios, fyi ^18:54
myoungahh sorry marios...muscle memory :)18:54
weshaymyoung, he'd be a good person to review it18:57
weshayrlandy|rover, oh btw.. I asked arxcruz|ruck to double check attila was removed from any rdo cloud tenants and instances18:57
weshaynot sure if that happened, but probably good to check w/ him tomorrow18:57
myoungweshay: ack, indeed.18:58
rlandy|roverok - I'll check with him tomorrow18:58
rlandy|roverI think adarazs is still on a lot of our access points18:58
rlandy|roverhe's probably having too good a time in the south of Spain (?) to access anything though18:59
myoungmarios: I've added you btw just now to the QE custom field dropdown that's on our cards.  You can pick your own color! :)18:59
myoungrlandy|rover:  sangria yields "don't care about ssh keys" {grin}19:00
*** Goneri has joined #oooq19:11
weshaymyoung, marios k k.. was just looking for review on gerrit change as he's not officially on the team yet19:12
weshaymyoung, marios however he probably won't mind either19:13
weshaymyoung, you can still QE sagi's change19:13
*** atoth has quit IRC19:22
*** tcw has quit IRC19:23
*** holser__ has joined #oooq19:23
myoungweshay: ack, i didn't put him as QE on the card, I just added him to the available choices for the drop-down, as part of onboarding (meetings, boards, etc)19:24
weshayah roger19:24
weshaythanks19:24
rlandy|roverwe got some promotion failures19:26
rlandy|roverInstall the undercloud19:26
*** tcw has joined #oooq19:26
rlandy|roverFailed to establish a new connection: [Errno 111] Connection refused',))\\n\", \"deploy_status_code\": 1}",19:27
*** d0ugal has joined #oooq19:29
weshay  "+ openstack quota set --cores -1 --instances -1 --ram -1 5e38b3f0fccb448386cdb39ba3d45d8b",19:29
weshay2018-06-07 19:12:14 |                 "Unknown Error (HTTP 503)"19:29
weshayrlandy|rover, ya.. that's worth running a recreate on19:30
weshayrlandy|rover, need help?19:30
rlandy|roverweshay: I am looking through the jobs19:30
rlandy|roverwe got 2 failures19:30
rlandy|roverso the far the other jobs are getting by that19:30
rlandy|roverthat was the error on fs01619:30
rlandy|rovercomparing fs001 error19:30
rlandy|rover2018-06-07 19:11:59 |         "stderr: Unknown Error (HTTP 503)",19:31
rlandy|rover2018-06-07 19:11:59 |         "Unknown Error (HTTP 503)"19:31
rlandy|rover2018-06-07 19:11:59 |     ]19:31
weshayUnable to establish connection to http://192.168.24.1:8774/v2.1/os-quota-sets/e68b399367024f39849bf1deaccb67fe: HTTPConnectionPool(host='192.168.24.1', port=8774): Max retries exceeded with url: /v2.1/os-quota-sets/e68b399367024f39849bf1deaccb67fe (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7ff21a83dd10>: Failed to establish a new connection: [Errno 111] Connection refused',))",19:31
rlandy|rover"stderr: /usr/lib/python2.7/site-packages/openstack/_meta/connection.py:122: ImportWarning: Could not import data-processing service filter: No module named data_processing.data_processing_service",19:33
rlandy|roverweshay: going to run reproducer on my tenant19:34
rlandy|roveryour tenant may be the better reproducer if we are looking at quotas19:34
weshayrlandy|rover let's use ovb obviously for fs001 and libvirt for fs01619:35
weshayrlandy|rover, want me to take fs001?19:35
weshaynot seeing much here https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/cc496dd/undercloud/var/log/extra/errors.txt.gz19:35
rlandy|roverweshay: sure - just want to lock down the original eror19:36
rlandy|rovererror19:36
rlandy|roverthere are a few spewing out19:36
weshayrlandy|rover, http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/extraconfig/post_deploy/undercloud_post.sh#n6919:37
rlandy|roverweshay: nobody else reporting quota issues on rhos-ops atm ( not that that means anything unto itself)19:37
rlandy|roveryes - but that's not the original error, at least I don't think19:37
rlandy|roversomething happens before that19:37
rlandy|roverweshay: ok - let's kick the reproducers and see where we go19:38
rlandy|roveryou on fs00119:38
rlandy|roverthen I'll do fs01619:38
weshayif [ "$(hiera nova_api_enabled)" = "true" ]; then19:38
rlandy|roveroh no - we're all red now19:38
weshay    # Disable nova quotas19:38
weshay    openstack quota set --cores -1 --instances -1 --ram -1 $(openstack project show admin | awk '$2=="id" {print $4}'19:38
rlandy|roverweshay: mass red incoming19:39
rlandy|roverthis is the first error:19:40
weshayk.. one at a time19:40
rlandy|rover2018-06-07 19:25:46 |         "Found 1 unmapped computes in cell: 212b1b3a-2c67-4935-8380-0297f480f136",19:40
rlandy|rover2018-06-07 19:25:46 |         "stderr: Unable to establish connection to http://192.168.24.1:8774/v2.1/os-services: HTTPConnectionPool(host='192.168.24.1', port=8774): Max retries exceeded with url: /v2.1/os-services (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fae48b486d0>: Failed to establish a new connection: [Errno 111] Connection refused',))",19:40
rlandy|roverthen we get19:41
rlandy|roverMax retries exceeded with url: /v2.1/os-quota-sets/0e15632013534926a3b02f9f2cacead4 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f3ec7a0cb90>: Failed to establish a new connection: [Errno 111] Connection refused',))\\n\", \"deploy_status_code\": 1}\n[2018-06-07 19:25:59,291] (heat-config) [DEBUG] [2018-06-07 19:25:54,777] (heat-config) [INFO]19:41
rlandy|rover TASK [Output for UndercloudPostDeployment19:41
rlandy|roverweshay: checking queens19:42
rlandy|roverqueens is good19:42
rlandy|roverso probably a real master problem19:43
weshayrlandy|rover, what do you think of19:43
weshay2018-06-07 19:11:59 |         "Status: Downloaded newer image for trunk.registry.rdoproject.org/tripleomaster/centos-binary-nova-compute-ironic:d596a4ea9f0b0d9255f21bbd4c3757eec6017669_ceffe88f",19:43
weshayhttps://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/60b4155/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2018-06-07_19_11_5919:43
weshayrlandy|rover, seems like nova is def not working here19:43
rlandy|rovertiming on upload/download?19:43
rlandy|roverweshay: ok to log a bug and edit the description/details as we go?19:44
rlandy|roversomewhere we were can keep our thoughts outside of irc19:44
weshayyes please.. but be specific.. I think the deployment of the undercloud stops here http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/extraconfig/post_deploy/undercloud_post.sh#n6919:45
weshayalthough the error is before that19:45
rlandy|roverwe will adjust - let's just start somewhere - bug incoming19:45
weshayrlandy|rover, bluejeans for a minute19:49
weshayhttps://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/0830d9b/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2018-06-05_05_55_0319:49
weshaycompare that good log19:49
rlandy|roverk19:49
weshayrlandy|rover, search for repository trunk.registry.rdoproject.org/tripleomaster/centos-binary-nova-compute-ironic19:49
weshayrlandy|rover, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/60b4155/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2018-06-07_19_11_5919:51
*** dtrainor has quit IRC20:04
weshayrlandy|rover, v20:06
weshayhttps://github.com/openstack/nova/commit/79304c1d15742ba04db19204a7c4aa87faaaa2f520:06
*** dtrainor has joined #oooq20:06
*** gkadam has quit IRC20:11
weshayrlandy|rover, http://paste.openstack.org/show/722923/20:14
*** d0ugal has quit IRC20:21
weshayrlandy|rover, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/cc496dd/undercloud/var/log/extra/nova_list.txt.gz20:28
weshayrlandy|rover, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/0830d9b/undercloud/var/log/extra/nova_list.txt.gz20:29
weshayrlandy|rover, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/0830d9b/undercloud/var/log/extra/network.txt.gz20:33
arxcruz|ruckweshay:20:33
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.20:33
arxcruz|ruck16:39:27 TASK [tripleo-inventory : fetch overcloud node names and IPs] ******************20:33
arxcruz|ruck16:39:27 Thursday 07 June 2018  16:39:27 +0000 (0:00:04.056)       1:28:56.812 *********20:33
arxcruz|ruck16:39:31 changed: [undercloud]20:34
arxcruz|ruckweshay: https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/2241/console20:34
arxcruz|ruckweshay: https://review.openstack.org/#/c/572709/20:34
arxcruz|ruck\o/20:34
weshayarxcruz|ruck, you are the rockinist20:34
weshayarxcruz++20:34
weshayrlandy|rover, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/cc496dd/undercloud/var/log/extra/network.txt.gz20:34
arxcruz|ruckrlandy|rover: weshay so, if you guys can +2+w so next periodic run we have it tomorrow would be nice :)20:35
*** brault has joined #oooq20:35
arxcruz|rucknow i'm out :)20:35
weshayarxcruz|ruck, thanks brotha20:37
rlandy|roverarxcruz|ruck: reviewed and +2 w+120:37
rlandy|roverhttps://github.com/openstack/tripleo-heat-templates/blob/master/environments/docker.yaml20:40
*** brault has quit IRC20:40
*** trown is now known as trown|outtypewww20:52
*** Goneri has quit IRC21:01
*** myoung is now known as myoung|off21:14
*** holser__ has quit IRC21:26
*** brault has joined #oooq21:27
*** hamzy has quit IRC22:04
*** hamzy has joined #oooq22:05
weshayrfolco, https://review.openstack.org/57345722:32
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.22:34
*** Goneri has joined #oooq23:00
*** Goneri has quit IRC23:27
rlandy|rovercontainerized undercloud deployment failed.\23:42
rlandy|roverError (HTTP 503)\\n\", \"deploy_status_code\": 1}",23:44
rlandy|roverweshay: ^^ reproducer reproduced23:44
rlandy|roveras I see so did yours23:44
rlandy|roveroh and it kicked again23:45

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!