Thursday, 2018-08-16

« Wednesday, 2018-08-15 Index Friday, 2018-08-17 »
*** dalvarez has quit IRC02:42
*** rlandy|afk is now known as rlandy02:44
*** rlandy has quit IRC02:44
*** radez has quit IRC02:46
*** amoralej|off has quit IRC02:47
*** ykarel has joined #oooq03:27
*** udesale has joined #oooq03:36
*** saneax has joined #oooq04:09
*** skramaja has joined #oooq04:37
*** jaganathan has joined #oooq04:37
*** apetrich has joined #oooq05:06
*** ratailor has joined #oooq05:27
*** jfrancoa has joined #oooq06:13
*** ykarel is now known as ykarel|afk06:30
*** ykarel|afk has quit IRC06:34
*** ratailor has quit IRC06:43
*** ratailor has joined #oooq06:47
*** gkadam has joined #oooq06:48
*** ykarel|afk has joined #oooq06:53
*** ykarel|afk is now known as ykarel06:54
*** amoralej has joined #oooq06:58
*** kopecmartin has joined #oooq07:11
*** bogdando has joined #oooq07:26
*** chem has joined #oooq07:49
*** dalvarez has joined #oooq08:24
*** panda|off is now known as panda08:56
*** jaosorior has quit IRC09:10
*** d0ugal has quit IRC09:14
*** d0ugal has joined #oooq09:18
pandamarios: change your nick09:22
pandassbarnea: you there ?09:22
*** marios|rover has joined #oooq09:25
jfrancoapanda: Could you give me a hand please? do you know where can I find the job definition for legacy-tripleo-ci-centos-7-container-to-container-upgrades-queens?09:28
jfrancoapanda: I'm debugging https://bugs.launchpad.net/tripleo/+bug/178503909:28
openstackLaunchpad bug 1785039 in tripleo "[Upgrades] legacy-tripleo-ci-centos-7-container-to-container-upgrades-master failing during deployment" [High,Triaged] - Assigned to Gabriele Cerami (gcerami)09:28
jfrancoapanda: it used to be in the config project, but I saw that zuul/upstream.yml was removed and I can't find it now09:29
pandajfrancoa: https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=blob;f=zuul.d/zuul-legacy-jobs.yaml;h=3957f4e667932828d9c237be9674e56717aff9fc;hb=refs/heads/master#l133309:31
pandajfrancoa: it's currently a mess up there, we are trying to fix all this in the next sprint09:31
jfrancoapanda: thanks, I wasn't aware of this repo09:31
pandajfrancoa: but for now, all the jobs are defined there, and they have a playbook per job, duplicated from the original09:32
jfrancoapanda: cool, thanks for the explanation09:32
jfrancoapanda: by the way, are you aware if I need to ask for any permissions to be able to access the rdoproject.org gerrit? I used to have rights in here, but lately I can't see any gerrit patch/project09:34
jfrancoapanda: nah, some cookies issue with mozilla09:40
pandajfrancoa: yep, just need github credentials09:50
*** ratailor has quit IRC09:57
*** ratailor has joined #oooq09:57
*** ykarel is now known as ykarel|lunch10:01
jfrancoapanda: problem solved https://review.rdoproject.org/r/#/c/15557/ , do you know any way to test the upgrades jobs with this patch?10:02
jfrancoapanda: or will the rdo zuul trigger the affected jobs automatically?10:02
*** d0ugal has quit IRC10:07
*** d0ugal has joined #oooq10:13
*** rf0lc0 has quit IRC10:15
*** rf0lc0 has joined #oooq10:16
*** skramaja has quit IRC10:19
*** skramaja has joined #oooq10:25
chandankumararxcruz: myoung weshay when you are around https://trello.com/c/CcrOAWdI/903-tempest-squad-sprint-18-prepare-for-rocky-release we have added the cards for upcomign sprint10:26
*** jaosorior has joined #oooq10:28
*** skramaja has quit IRC10:32
*** panda is now known as panda|lunch11:02
*** amoralej is now known as amoralej|lunch11:09
*** ykarel|lunch is now known as ykarel11:09
*** udesale has quit IRC11:30
*** ratailor has quit IRC12:07
*** amoralej|lunch is now known as amoralej12:15
weshayssbarnea, howdy12:17
weshayssbarnea, is still pto I think12:17
*** skramaja has joined #oooq12:22
*** agopi has quit IRC12:22
*** jaosorior has quit IRC12:25
amoralejsome jobs for rocky are failing with error12:25
amoralej2018-08-16 11:53:38.057117 | primary | emit_releases_file.py: error: argument --stable-release: invalid choice: 'rocky' (choose from 'newton', 'ocata', 'pike', 'queens', 'master')12:25
*** jaosorior has joined #oooq12:25
amoralejhttps://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-rocky/149ffa0/job-output.txt.gz12:25
ykarelrocky missing here:- https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/emit_releases_file/emit_releases_file.py#L1012:29
ykarelmarios|rover, ^^12:38
ykarelno ruck today?12:38
*** rlandy has joined #oooq12:41
rlandyjschlueter: hello - running into an undercloud install error on rhos-13 - https://thirdparty.logs.rdoproject.org/jenkins-tq-gate-rhos-13-ci-rhos-ovb-featureset001-129/undercloud/home/stack/undercloud_install.log.txt.gz#_2018-08-15_18_29_2312:50
*** agopi has joined #oooq12:50
rlandyfamiliar to you?12:50
rlandywe should be using passed_phase112:50
rlandyweshay: ^^ fyi12:51
*** agopi_ has joined #oooq12:51
*** agopi has quit IRC12:55
*** agopi_ is now known as agopi12:56
myoungo/ sprint planning for CI Squad s18 starts shortly12:56
mariosthanks ykarel will have to check later in calls right now13:01
mariosykarel: ruck is away weshay is filling in i am rovering and we are all in a call right now13:01
ykarelmarios, ack. btw sshnaidm proposed fix for that issue13:01
weshayrlandy, hrm.. osp-1313:02
weshayrlandy, Duplicate key name 'uniq_ports0standard_attr_id'13:03
weshayrlandy, maybe try passed phase 2?13:03
weshayrlandy, /me looks at builds13:03
weshayrlandy, http://paste.openstack.org/show/728194/13:04
weshay2018-08-15 18:19:02 | 2018-08-15 18:19:02,408 INFO: [1;31mError: Command exceeded timeout[0m13:04
rlandyweshay: should not be exceeding timeout though13:05
rlandyI can increase the timeout13:05
rlandybut I wanted to be sure it was not a legit error13:05
rlandyweshay: admittedly it's slow on this platform13:05
weshayrlandy, https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP13/13:07
rlandyweshay: so switch to passed phase 2?13:08
rlandyand increase timeout?13:08
*** udesale has joined #oooq13:12
*** jaosorior has quit IRC13:14
*** skramaja has quit IRC13:14
*** jaosorior has joined #oooq13:15
*** jaosorior has quit IRC13:20
rlandyweshay: possibly we are running into the 'lower quality hardware' issue here ... https://review.openstack.org/#/c/428843/13:23
*** jfrancoa has quit IRC13:23
weshayrlandy, ya.. sorry was the job ovb or virt?13:23
rlandyweshay: ovb13:24
rlandytrying to revive that job13:24
weshayright k13:24
weshayrlandy, so maybe bump up the cpu and memory as a test?13:24
*** jfrancoa has joined #oooq13:25
rlandyweshay: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/environments/ci-rhos.yml#n34 - xlarge - pretty extensive13:26
weshayrlandy, that is rhos-ci cloud right?13:38
weshaycirhos13:38
weshayya13:38
rlandyweshay: yep13:38
* weshay looks at that13:38
rlandyweshay: I increased the volume to 8013:39
weshayk13:39
rlandyif you look at the job config - we deploy with attached volumes13:39
rlandysince we could not get bigger flavors13:39
weshayrlandy, go big once w/ the undercloud m5.large13:40
weshay:)13:40
rlandygo big or go home13:41
* rlandy looks13:41
*** matbu has joined #oooq13:42
weshayrlandy, :)13:44
*** radez has joined #oooq13:45
*** saneax has quit IRC13:57
amoralejit seems we need a file for rocky in https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/validate-tempest/vars14:13
amoralejjobs are failing because of that14:13
weshayamoralej, k.. chandankumar just submitted a patch14:28
weshayamoralej, https://review.openstack.org/#/c/592489/114:28
amoralejcool14:29
ykarelweshay, also add to the checklist so it's can be avoided for stein14:29
weshayykarel, aye.. thanks14:29
amoralejchandankumar, there is no tag for tempestconf for rocky yet?14:29
chandankumaramoralej: will come next week14:30
amoralejack14:30
chandankumaramoralej: waiting for tosky to come14:30
*** jaosorior has joined #oooq14:32
*** d0ugal has quit IRC14:39
*** d0ugal has joined #oooq14:41
*** marios|rover has quit IRC14:50
*** ratailor has joined #oooq14:59
weshaywas there a decision?15:04
weshayI see more options15:04
*** ratailor has quit IRC15:05
*** ratailor has joined #oooq15:06
*** marios|rover has joined #oooq15:07
chandankumarweshay: we have few patches pending15:07
chandankumarbut i need some feedback from tosky so waiting till monday15:08
*** kopecmartin has quit IRC15:09
*** ykarel is now known as ykarel|away15:09
weshaychandankumar, k15:12
*** jfrancoa has quit IRC15:32
*** vinaykns has joined #oooq15:32
*** verdurin has quit IRC15:36
*** verdurin has joined #oooq15:38
*** bogdando has quit IRC15:44
*** d0ugal has quit IRC15:47
*** ykarel|away has quit IRC15:51
*** d0ugal has joined #oooq16:00
chandankumarmyoung: Hello16:04
chandankumarmyoung: https://trello.com/c/CcrOAWdI/903-tempest-squad-sprint-18-prepare-for-rocky-release I have added sprint 18 cards reviewd with mkopec please let's know if you need anything else16:05
*** udesale has quit IRC16:05
*** ratailor has quit IRC16:09
*** amoralej is now known as amoralej|off16:22
*** panda|lunch is now known as panda|off16:28
* myoung walks over to the transmogrifier and looks for the "matt" position on the selector lever16:28
*** myoung is now known as myoung|lunch16:28
mariosweshay: myoung panda|off sshnaidm rf0lc0 ** have a great day folks bbyl16:28
mariosttyl evel :)16:28
myoung|lunchthanks everyone for patience, endurance, and a productive discussion and design session16:28
marioseven (k time to go now)16:29
weshayl8r16:29
rf0lc0evil?16:29
rf0lc0:)16:29
mariosrf0lc0: yes :D it must be i mean 208 mins of meeting what else16:29
* rf0lc0 thinks we should rename this meeting to marathon planning 16:30
myoung|lunchA NEW RECORD.16:31
myoung|luncheveryone go outside and breathe some fresh air.  let your craniums unmelt :)16:31
* myoung|lunch walks away from keyboard to do ^^16:31
panda|offwhat planning ?16:31
panda|offwe didn't even talk about the card16:31
mariosmyoung|lunch: teling ya 3 hours + is dangerous territory ... mass hysteria and we are locked in the call until someone external looks for us16:32
mariosit could be ages16:32
mariosthank heavens weshay joined to break the lock16:32
myoung|lunchok i was about to go...and going after this...but yeah, that might be a TC fail.  my bad lol16:32
* myoung|lunch is still proud of us though...we did some rational design discussion and ship navigation16:33
myoung|lunchapologies for the length16:33
*** rlandy is now known as rlandy|brb16:33
*** sshnaidm is now known as sshnaidm|bbl16:35
*** rf0lc0 is now known as rfolco16:39
*** gkadam has quit IRC17:03
*** rlandy|brb is now known as rlandy17:11
*** weshay is now known as weshay|ruck17:12
weshay|ruckrlandy, any luck w/ the UBER Flavor?17:13
rlandyweshay|ruck: trying now - was caught up in the planning fun this morning17:13
weshay|ruckrlandy, https://review.openstack.org/59259117:19
*** gkadam has joined #oooq17:29
rlandyweshay|ruck: arxcruz is on PTO :(17:29
rlandyso much for my coaching :(17:30
rlandyweshay|ruck: question ... the comment says "mistral team is investigating"17:33
rlandywhen does the tempest team investigate vs, when does that team ask the product team to?17:34
rlandychandankumar: ^^?17:34
*** myoung|lunch is now known as myoung17:44
chandankumarrlandy: it is related to mistral tests, I donot have much knowledge, let's have an ETA, till then add it to ski list?17:44
chandankumar*skip17:44
chandankumarweshay|ruck: ^^17:44
chandankumartempest team takes care of tempest tooling and the framework not the tests17:45
chandankumarwe are here to help team how they can enable and run tempest specific tests not on if the tests failed17:46
chandankumarbecause those are component specific needs deeper knowlege of the servie how they are behaving17:46
weshay|ruckchandankumar, certainly.. if there is a failing tempest test in the upstream job.. WE ALL ARE HERE to help debug it17:52
weshay|ruckchandankumar, if needed we can bring in others to help17:52
weshay|ruckchandankumar, w/ fs20 being the exception17:52
weshay|ruckchandankumar, there is a pretty limited scope of tempest tests being used in upstream check / gate jobs17:52
weshay|ruckthat we should be able to debug those down and call in help when needed17:53
* chandankumar will look at this tomorrow17:53
weshay|ruckapetrich, or toure can help too17:54
weshay|ruckrlandy, apetrich w/ https://bugs.launchpad.net/tripleo/+bug/173695017:54
openstackLaunchpad bug 1736950 in tripleo "CI: mistral testmistral_tempest_tests.tests.api.v2.test_actions.ActionTestsV2.test_get_list_actions_not_in_list_filter fails in gate scenario003 containers" [Critical,Triaged]17:54
rlandythanks18:03
apetrichrlandy, weshay|ruck looking18:03
rlandyapetrich: hello!18:03
weshay|ruckchandankumar, arxcruz so rlandy is going to get familiar w/ setting up a tempest debug env18:04
apetrichrlandy, hiya18:04
weshay|ruckarxcruz, I've watched you do this.. chandankumar if you can help rlandy or point her at doc that would be very helpful18:04
chandankumarweshay|ruck: time to catch d0ugal also :-)18:05
apetrichweshay|ruck, isn't arxcruz fighting for the horde at the moment?18:05
weshay|ruckya.. he's pto18:05
chandankumarhe will be available tomorrow during planning18:05
apetrichbest excuse for a pto ever18:06
apetrichoh that test18:06
apetrichrlandy, yeah I think I know what the issue is18:07
rlandychandankumar: thanks - I'l catch arxcruz when he is back18:07
rlandyapetrich: great18:07
* chandankumar knows fedora is good for application development but not good for openstack18:08
* chandankumar find very hard when updating openstack related fedora dependencies because they donot care about us at all18:09
rlandymyoung: where was that tempest doc to review?18:15
rlandyweshay|ruck: https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-13-ci-rhos-ovb-featureset001/130/console - running with UBER flavor on the undercloud - will see if it helps18:15
myoungrlandy: https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset053.yml#L14518:15
myoungrlandy: http://logs.openstack.org/42/589742/3/check/build-openstack-sphinx-docs/2a676c0/html18:16
rlandythank you18:16
weshay|ruckchandankumar, oh ya..18:16
myoungrlandy: ack.  i'm reading it later on as well18:16
weshay|ruckfedora sucsk18:16
weshay|rucksucks18:16
* weshay|ruck votes for fedora lts18:17
rlandyI read it a while back but time to refresh18:17
weshay|ruckchandankumar, I have a fedora guest image that's working for me up to a point18:17
weshay|ruckchandankumar, on standalone18:17
weshay|ruckchandankumar, we need to get tempest moving on standalone18:17
weshay|ruckchandankumar, $next sprint18:17
weshay|ruckshould be easy18:18
weshay|ruckI had some patches18:18
apetrichrlandy, spinning an env to test it but I will have to close the lid soon. I can have a look at that early in the morning tomorrow18:18
rlandyapetrich: k, thanks18:18
*** rlandy has quit IRC18:32
*** myoung is now known as myoung|brb18:34
*** rlandy_ has joined #oooq18:35
*** gkadam_ has joined #oooq18:38
*** gkadam has quit IRC18:42
*** sshnaidm|bbl is now known as sshnaidm18:43
chandankumarweshay|ruck: sure sir18:53
chandankumarweshay|ruck: https://trello.com/c/CcrOAWdI/903-tempest-squad-sprint-18-prepare-for-rocky-release feel free to take a look at sprint 18 topics since we did not get a chance to spea about that18:54
weshay|ruckthanks chandankumar will catch you tomorrow18:56
weshay|ruck:)18:56
*** gkadam_ has quit IRC19:00
rlandy_panda|off: myoung|brb: I'll trying a POC review for the reparent task19:05
rlandy_rfolco: ^^19:05
rfolcorlandy_, ack, I'm checking a better way to include custom playbooks and tempest skiplist and custom fs vars and ...19:07
*** apetrich has quit IRC19:10
weshay|ruckbah.. data here is old19:19
weshay|ruckhttp://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&from=now-6h&to=now19:19
weshay|ruck:(19:19
*** myoung|brb is now known as myoung19:19
myoungrlandy_, rfolco, I'm doing some thinking around what an ontology/namespace of job layers would like like...e.g....what logical tree shape makes sense, as well as putting into words/doc format some of the things we talked about today. Next week I'm in training so trying to get some thoughts/ideas out prior to monday.19:27
rfolcoack young man myoung19:30
* myoung will be "forever Young"19:31
* myoung groans at his own terrible pun19:31
rfolco;)19:31
myoungI think my brain has recently rebounded from this morning.19:31
myounglol19:31
weshay|ruckrlandy_, that ovb job on 13 is using the fs001 config?19:35
rlandy_weshay|ruck: yes19:39
*** rlandy_ is now known as rlandy19:40
weshay|ruckrlandy, something is funky there19:40
weshay|ruckrlandy, can we get on the undercloud?19:40
*** jrist has quit IRC19:40
rlandyweshay|ruck: I am on the undercloud now19:40
weshay|rucktmate19:40
rlandywatching the tail on undercloud install19:40
weshay|ruckrlandy, docker ps19:40
weshay|ruckwhat's up w/ that19:40
weshay|ruckoh wait..19:40
weshay|ruckthis is the undercloud in queens19:41
weshay|ruckno docker19:41
weshay|ruckwth is taking so long19:41
weshay|ruckanything funky on the networking?19:41
rlandyweshay|ruck: this is the point where trouble starts19:49
weshay|ruckya19:50
weshay|ruckhaproxy19:50
weshay|ruckhrm19:51
weshay|ruckthat seems like a puppet bug to me19:51
weshay|ruckthat should be something that rasca should be able to kill for us19:52
weshay|ruckbut let's see what happens now19:52
weshay|ruckya19:52
weshay|ruckrlandy, weird19:53
weshay|rucksuch a odd thing to fail on for a GA product19:53
rlandyweshay|ruck: something is killing it19:53
weshay|ruckmaybe there was a regression19:53
rlandyfailed in the same place yesterday19:53
weshay|ruckya19:53
rlandyweird19:53
rlandythat we should find it19:54
rlandysurely tested elsewhere19:54
rlandypoor haproxy19:54
rlandyweshay|ruck: which is hwy I asked this morning19:54
rlandynot passing undercloud install is pretty basic19:54
rlandyunless it's env specific19:55
weshay|ruckhttps://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP13/job/phase1-13_director-rhel-7.5-virthost-1cont_1comp_1ceph-ipv4-vxlan-ceph-containers/lastBuild/console19:55
weshay|ruckit's going to pass now19:55
rlandybecause you restarted haproxy19:56
rlandywhich should NOT be required19:57
rlandyto state the obvious19:57
weshay|ruckthis one passed for QE19:58
weshay|ruckhttp://cougar11.scl.lab.tlv.redhat.com/phase1-13_director-rhel-7.5-virthost-1cont_1comp_1ceph-ipv4-vxlan-ceph-containers/168/undercloud-0.tar.gz?undercloud-0/etc/yum.repos.d/rhos-release-13.repo19:58
weshay|ruckrlandy, oh ya.. totally19:58
weshay|ruckthat was because I started it19:58
rlandyweshay|ruck: can we check hapoxy status now?19:58
weshay|ruckbut puppet should have retried the start instead of just polling that status19:58
rfolcodocker hub down19:58
rfolconm sat 25th19:59
rlandyweshay|ruck: ok - so it passed for QE but fails for us ... meaning???20:00
rlandymay still fail now20:00
weshay|ruckrlandy, no.. I'm just finding a known working build20:00
weshay|ruckfrom the recent past20:00
weshay|ruckrlandy, I don't know..20:00
weshay|ruckI don't know what it means :)20:00
weshay|ruckrlandy, maybe we need to try http://rhos-qe-mirror-tlv.usersys.redhat.com/rcm-guest/puddles/OpenStack/13.0-RHEL-7/7.5-z1/20:01
weshay|ruckwhich is the 13.1 release20:01
weshay|ruckrlandy, all we're trying to do is to make sure tq/tqe is working with it right20:02
rlandyweshay|ruck: we didn't pay off the correct OSP mafia boss this month?20:02
weshay|ruckso we should find the MOST reliable build they have20:02
weshay|ruckbah haha20:02
weshay|ruckrlandy, it could be that there are workarounds20:02
* weshay|ruck runs20:02
rlandyweshay|ruck: no - come back - we are not don yet!!!!20:03
rlandydone20:03
* weshay|ruck looks20:03
rlandylet's just see if this passes with restart20:03
weshay|ruckit's setting up glance20:03
rlandyie: now20:04
weshay|ruckrlandy, ya20:04
rlandyweshay|ruck: "MOST reliable build they have" so we should be using passed phase 220:04
weshay|ruckrlandy, I would say.. something we shipped20:06
weshay|ruckhttp://rhos-qe-mirror-tlv.usersys.redhat.com/rcm-guest/puddles/OpenStack/13.0-RHEL-7/7.5-z1/20:06
weshay|ruckwell.. from a boston mirror20:06
weshay|ruckdam this slow20:06
rlandylegit timeout?20:08
rlandyweshay|ruck: ok - will try a boston mirror when this is officially dead20:09
rfolcorlandy, what is the multinode fs for browbeat ? fs053 is ovb...20:22
rlandyrfolco: there is no multinode for browbeat20:24
rlandyit's a simulation of real hw only20:24
weshay|ruckrlandy, https://redhat.bluejeans.com/u/whayutin/20:24
rfolcorlandy, oh I see thx20:24
rlandyhttp://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-13.yml20:30
myoungrfolco: the actual browbeat (perf ci) jobs run with large numbers of actual nodes, afaik they can go up to 24 nodes per bench without having to reconfig...but not sure what # of nodes are running in rdo2 now...  agopi would know20:32
rfolcomyoung, thx ... my question has much simpler reason than browbeat itself20:34
agopimyoung, 6 nodes20:34
agopi3 controllers 2 computes ^20:34
rfolcogood to know, thank you agopi ;)20:34
agopinp rfolco :)20:35
myoungagopi: were it possible (via configuration) to easily up the node #, is that still a goal?  that was something jkilpatr used to talk about...20:39
*** jrist has joined #oooq20:40
agopiit should be, and given that we've got a hardware crunch i don't see it happen anytime soon. We usually do larger scale deployment in scalelab but they're not LTA.20:40
myounglta?20:40
agopilong term availability my bad20:41
*** ssbarnea has quit IRC20:45
*** apetrich has joined #oooq20:46
rlandyweshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/147299/20:53
rlandymyoung: ^^ pls check if I did that correctly20:56
myounglooking now, missing one bit20:56
myoungcommenting there20:56
weshay|ruckwhat's missing?20:57
rlandyoh sorry20:57
rlandyI will resubmit20:57
weshay|ruck?20:58
rlandymyoung:  did you comment?20:59
myoungjust now...i guess review window was 5 min lol20:59
rlandymyoung: sorry, I'm not understanding21:00
rlandywhat needs to change?21:00
rlandythis is the test for tq tqe21:00
rlandyis there something else that needs to change on this file?21:01
rlandyyou use this file for other jobs?21:01
rlandyso I messed that up?21:01
rlandyshould I create a separate file?21:02
myoungi see21:02
myoungno it's fine...it's just that the auto triggers for normal osp13 jobs are looking at a new phase 1 passed puddle21:02
myoungwhat you have for the tq/tqe jobs that use that configfile (in terms of args to rhos-release) is fine21:03
myoungand will work21:03
rlandymyoung: I can create a new file if thta work work better21:03
myoungbut just some concern that the non tq/tqe gate jobs (that are running osp passed_phase_1) will trigger on a new puddle still, but then the config file causes to call rhos-release with z1 puddle21:04
* myoung needs to review/audit those jobs to be sure...and wants to purge them all with fire and use zuul config downstream instead and chuckles21:04
rlandymyoung: ok - let me know  - can change it if need be21:05
rlandysorry for the quick merge21:06
myoungrlandy: sok...by the time i looked was already in heh21:06
* myoung reloads osp0 state from memory banks21:06
rlandymyoung: we can create a rhos-13-z1 file21:07
rlandyif we need to21:07
myoungrlandy: ack, i'm looking here: https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/osp-director-promote-13-puddle21:07
myoungthat seems to have been red since the late may21:08
myoungwell june for https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/osp-rhos-13-promote-puddle-featureset020-1ctlr_1comp_64gb/21:08
myoungand rasca's job is green21:08
rlandybut rasca's job works21:09
rlandyidk21:09
myoungright...so https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/oooq-rhos-13-puddle-bmu-ha-lab-phoenix-float_nic_with_vlans/3/ is the one that matters21:09
rlandywill have to check with him when he is on line21:10
rlandyhow does that work??21:10
myoungso the parent job uses the trigger script to look for new puddles, and set current_build, which is passed to all jobs (https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/oooq-rhos-13-puddle-bmu-ha-lab-phoenix-float_nic_with_vlans/3/parameters)21:11
myoung^^ that's at passed_phase_121:11
myoungso looking now to see if he uses the changed file (that now uses z1 vs. passed_phase_1) - as far as i recall his stuff is pulling the puddle from the current_build variable, so should be fine but wanted to make sure21:12
myoungso we don't get a silent puddle mismatch21:12
myoungrlandy: i think it's fine21:13
myounghe does pass that yaml (00:01:33.725 + ansible-playbook -vvvv /home/rhos-ci/jenkins/workspace/oooq-rhos-13-puddle-bmu-ha-lab-phoenix-float_nic_with_vlans/playbooks/baremetal-undercloud-validate-ha.yml -e @/home/rhos-ci/jenkins/workspace/oooq-rhos-13-puddle-bmu-ha-lab-phoenix-float_nic_with_vlans/config/release/rhos-13.yml )21:14
myoungbut21:14
myoungcurrent_build is defined so it just uses the specifc puddle # passed to it21:14
myoungsorry...wanted to make sure as he's on PTO and those jobs have recently been stabilized21:14
*** agopi is now known as agopi|out21:20
*** agopi|out has quit IRC21:24
*** jrist has quit IRC21:55
*** apetrich has quit IRC22:01
*** vinaykns has quit IRC22:06
*** sshnaidm is now known as sshnaidm|off22:45
weshay|ruckrlandy, :( no nodes captured in the logs https://thirdparty.logs.rdoproject.org/jenkins-tq-gate-rhos-13-ci-rhos-ovb-featureset001-130/22:56
rlandyweshay|ruck: the create stacks and access ahve been failing22:56
rlandyI switched to the regular tenant22:56
rlandynot looking good22:56
weshay|ruckrlandy, ya. but that is the undercloud install we were watching22:57
rlandyit's not getting there22:57
weshay|ruckk k22:57
weshay|ruckwhat evs22:57
rlandyI am watching the regular tenant now22:58
weshay|ruckrlandy,  k22:58
weshay|ruckrlandy, https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-13-ci-rhos-ovb-featureset001/134/console22:58
weshay|ruckthat one?22:58
rlandyfor the moment22:58
weshay|ruckk22:58
rlandy| 02c2ed16-39fe-49d6-9c29-ee51a83d6a60 | bmc-14813         | BUILD  |                                                                                                                                                                                                          | bmc-base | m1.small  |22:59
rlandy| f5218bbb-f809-4f9d-af75-ffbbf6f5ccde | undercloud-14813  | BUILD  |                                                                                                                                                                                                          |          | m1.xlarge |22:59
rlandy| 02958638-df42-443a-9e9c-5753e763e4ff | baremetal-14813_0 | ACTIVE | overcloud_tenant-14813=172.16.0.5; public-14813=10.0.0.7; overcloud_internal-14813=172.17.0.8; overcloud_storage-14813=172.18.0.2; provision-14813=192.168.24.7; overcloud_storage_mgmt-14813=172.19.0.6 |          | m1.large  |22:59
rlandy| d2e58e58-e6c9-41d1-ab9d-e84c7eeff063 | baremetal-14813_1 | ACTIVE | overcloud_tenant-14813=172.16.0.9; public-14813=10.0.0.9; overcloud_internal-14813=172.17.0.5; overcloud_storage-14813=172.18.0.1; provision-14813=192.168.24.5; overcloud_storage_mgmt-14813=172.19.0.7 |          | m1.large  |22:59
rlandy^^ sitting there22:59
weshay|ruckrlandy, are your jobs using this?22:59
weshay|ruckhttp://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/jenkins/jobs/tripleo-quickstart/scripts/trigger-getbuild.sh#n11222:59
rlandyno23:01
rlandytriggering off changes23:01
weshay|ruckrlandy, :)23:01
weshay|ruckya... looks good23:01
weshay|ruckwell done23:01
*** jrist has joined #oooq23:01
rlandycrazy slow or broken, I don't know which atm23:04
rlandycrazy slow23:04
rlandyone just completed23:05
rlandymay deploy before I officially retire23:05
weshay|rucklolz23:06
weshay|ruckya.. maybe rdo-cloud isn't so bad23:06
weshay|rucklolz23:06
rlandyoh my gosh - I LOVE rdocloud23:06
rlandyand the admins23:07
rlandythey respond23:07
weshay|ruckrlandy, maybe we should send those guys a redhat reward23:09
rlandyweshay|ruck: yeah - I have sent in the past23:09
rlandymaybe it's time again23:09
weshay|ruckya.. I'll see what I have as well23:09
rlandyweshay|ruck: I can do it - then you can approve23:10
weshay|ruckrlandy, it's their manager, not me23:10
rlandyhmmm... I thought it was both managers23:10
rlandyanyways23:10
rlandyI have points if you need23:10
weshay|ruckI'll do it, save urs..23:11
weshay|ruckfyi http://eavesdrop.openstack.org/irclogs/%23oooq/23:21
weshay|ruckrlandy, oh btw.. I requested an upshift account today, I put us as admins23:21
weshay|ruckthe compute nodes of rdo cloud may be moved to upshift23:22
weshay|ruckso ..23:22
weshay|ruckthere's that23:22
weshay|ruckrlandy, we have several delete_failed in ci-rhos23:24
rlandyweshay|ruck: nice on the upshift ...23:24
* weshay|ruck remvoing23:24
rlandy I know about the delete failed23:24
rlandythey are not taking resources23:24
weshay|ruckk23:25
rlandyI removed all the created failed ones23:25
rlandywe do need to get them removed but they should not be holding anything up23:25
weshay|ruckrlandy, did I get this fixed correctly https://review.rdoproject.org/r/#/c/14482/23:25
rlandyweshay|ruck: looks ok - I +1'ed23:28
rlandywill have to try it out23:28
rlandyunless you have and can post results23:28
rlandyweshay|ruck: this internal cloud has some how slowed down even more23:28
weshay|ruckman.. 10.0.0.823:29
rlandyI know I can put  in a ticket23:29
rlandybut d we have a direct contact?23:29
weshay|ruckrlandy, I wish we had some production jobs here.. to light them up w/ an escalation23:29
rlandyit's unusable now23:29
weshay|ruckrlandy, undercloud is 10.0.0.8 ya?23:29
rlandyssh root@10.8.241.12623:29
weshay|ruckwhayutin•~/OPENSTACK_API/ci-rhos ᐅ  ping 10.8.241.126                                                                                                                                           thinkdoe ⌚ 19:30:1123:30
weshay|ruckPING 10.8.241.126 (10.8.241.126) 56(84) bytes of data.23:30
weshay|ruck^C23:30
weshay|ruck--- 10.8.241.126 ping statistics ---23:30
weshay|ruck25 packets transmitted, 0 received, 100% packet loss, time 24602ms23:30
rlandythere is something wrong here23:32
* rlandy checks horizon23:32
rlandyhttps://ci-rhos.centralci.eng.rdu2.redhat.com/dashboard/project/instances/f5218bbb-f809-4f9d-af75-ffbbf6f5ccde/23:36
weshay|ruckrlandy, https://docs.engineering.redhat.com/display/CentralCI/Central+CI+Home23:37
* weshay|ruck opening a ticket23:37
rlandyweshay|ruck: thank you23:38
rlandyI think it's getting slower and slower23:38
rlandyif that i spossible23:38
rlandymaybe they can reboot it?23:39
weshay|ruckrlandy, https://redhat.service-now.com/surl.do?n=PNT033278523:47
weshay|ruckyou should be added to the ticket23:47
rlandyweshay|ruck: thanks - I can access it23:48
rlandyadding to the ticket23:48
weshay|ruckrlandy, I think we should take a virthost offline23:49
rlandyweshay|ruck: the slave?23:49
weshay|rucktomorrow maybe I'll try that tomorrow23:49
rlandyok but why?23:49
weshay|ruckya.. just to see if we can get osp-13 to work23:49
rlandyah ok23:50
weshay|ruckI guess rasca has done that though?23:50
rlandyyes23:50
weshay|ruckis it a waste?23:50
weshay|ruckok23:50
* rlandy gets23:50
rlandyhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/oooq-rhos-13-puddle-bmu-ha-lab-phoenix-float_nic_with_vlans/23:51
weshay|ruckah cool23:51
rlandyweshay|ruck: it's not a waste if we want that for a reason23:51
rlandyrasca is not deploying fs 00123:51
weshay|ruckya.. only we were to run it nightly23:51
weshay|ruckor weekly as you were suggesting23:51
rlandyweshay|ruck: we could also use one of our bm23:52
rlandyif ovb becomes useless23:52
rlandywe know it works with queens23:52
* rlandy needs to run out ... will pick this up later23:53
*** rlandy is now known as rlandy|bbl23:53
« Wednesday, 2018-08-15 Index Friday, 2018-08-17 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!