Tuesday, 2020-06-23

« Monday, 2020-06-22 Index Wednesday, 2020-06-24 »
*** rfolco has quit IRC00:32
*** jmasud has quit IRC01:15
*** Tengu has quit IRC03:07
*** Tengu has joined #oooq03:09
*** rlandy|ruck|bbl is now known as rlandy03:10
*** Goneri has quit IRC03:11
*** sanjayu_ has joined #oooq03:14
*** dmellado has quit IRC03:59
*** ykarel|away is now known as ykarel04:10
*** skramaja has joined #oooq04:14
*** aakarsh has quit IRC04:25
*** aakarsh has joined #oooq04:25
*** jmasud has joined #oooq04:33
*** udesale has joined #oooq04:41
*** ratailor has joined #oooq05:08
*** jmasud has quit IRC05:10
*** jmasud has joined #oooq05:11
*** dtantsur|afk has quit IRC05:14
*** jmasud has quit IRC05:17
*** ysandeep|PTO is now known as ysandeep05:18
*** jmasud has joined #oooq05:19
*** soniya29|afk is now known as soniya29|rover05:26
*** marios has joined #oooq05:31
*** pojadhav|afk is now known as pojadhav05:37
*** soniya29|rover is now known as soniya29|afk05:40
*** jtomasek has joined #oooq07:01
*** ccamacho has joined #oooq07:02
*** jbadiapa has joined #oooq07:06
*** jmasud_ has joined #oooq07:15
*** jmasud has quit IRC07:17
*** jfrancoa has joined #oooq07:19
*** soniya29|afk is now known as soniya29|rover07:35
*** tosky has joined #oooq07:35
*** amoralej|off is now known as amoralej07:36
mariosneeds reviews please https://review.rdoproject.org/r/#/c/28161 https://review.rdoproject.org/r/#/c/28162 test at https://review.rdoproject.org/r/#/c/2810607:49
*** jpena|off is now known as jpena07:54
*** ykarel is now known as ykarel|lunch08:23
zbrmorning everyone!08:31
zbrpanda: ping me when ready to start the joy08:31
marioslol zbr08:32
zbrmarios: fyi, joy = upgrading the promoter08:32
mariosyeah i imagined it was something related08:32
mariosdoes anyone know off-hand if ansible lets you configure itself, like lineinfile to ansible.cfg via an ansible task? :D08:33
zbrbut i seen the change reverted, due to possible issue (not really convinced).08:33
mariosi vaguely recall someone trying that and there is some protection against it but i am going to try it08:33
mariosbecause i have nothing else left to try08:34
zbrmarios: you can edit ansible.cfg but it will reload the file when another instance is started, not the same runtime08:34
zbrobviously that this does not apply to zuul at all.08:34
marioshmm... zbr yeah i was considering this. i wonder though does it apply to 2 different playbook executions... perhaps they still count as the same runtime likely08:34
marioszbr: i.e. i am configuring in pre.yaml and want to use it in run-v3.yaml08:35
zbri do not know the details but my guess is that you are using it wrong if you need that08:35
marioszbr: i suspect part of our problems stems from no/little collections support before ansible 2.908:35
zbrand i would be against such practice, mainly because ansible works best if you call it only once (like zuul).08:35
marioszbr: e.g. in the default ansible.cfg there is no collections_paths options for ansible 2.8.1208:36
zbrmarios: don't try anything collection before 2.9, it does not worth the efforth08:36
marioszbr: k thanks08:36
marioszbr: i08:36
zbrthey are broken badly in 2.8, zuul will bump default ansible to 2.9 quite soon, faster than the usual cadence.08:36
marioszbr: i'm still gonna try edit ansible.cfg by ansible task in pre, so i can say i tried it and i'm really curious to see what will happen :D08:36
*** ysandeep is now known as ysandeep|lunch08:37
zbri would support any move to switch minimal ansible to 2.908:37
*** pojadhav is now known as pojadhav|lunch08:40
*** ysandeep|lunch is now known as ysandeep08:50
amoralejhi, i've pushed a minor update of rabbitmq-server, 3.8.2 -> 3.8.3 to centos mirror, i've tested it and it should be fine09:04
amoralejlet me know if you see anything abnormal09:05
*** dtantsur has joined #oooq09:05
*** owalsh has quit IRC09:08
*** owalsh_ has joined #oooq09:08
*** jmasud_ has quit IRC09:11
*** sshnaidm|afk is now known as sshnaidm|ruck09:11
*** jmasud has joined #oooq09:14
*** pojadhav|lunch is now known as pojadhav09:29
*** derekh has joined #oooq09:30
*** udesale_ has joined #oooq09:46
*** udesale has quit IRC09:49
*** ratailor_ has joined #oooq09:51
*** ratailor has quit IRC09:54
*** marios has quit IRC09:54
*** holser_ has joined #oooq10:39
arxcruzchandankumar: kopecmartin please review https://review.opendev.org/#/c/720434/10:40
*** holser has quit IRC10:41
*** marios has joined #oooq10:46
*** ykarel|lunch is now known as ykarel10:58
*** pojadhav is now known as pojadhav|afk11:30
*** jpena is now known as jpena|lunch11:31
soniya29|roversshnaidm|ruck, tripleo-ci-centos-8-standalone job is healthy and the patch https://review.opendev.org/#/c/737001/ has also got merged, still it is shown as failure in cockpit.11:49
sshnaidm|rucksoniya29|rover, where is the failure?11:50
*** amoralej is now known as amoralej|lunch11:52
soniya29|roversshnaidm|ruck, according to cockpit the above job is failing in gate, but I have looked at patch, the patch got merged by zuul11:52
*** ysandeep is now known as ysandeep|afk12:00
*** rlandy has joined #oooq12:00
*** rlandy is now known as rlandy|ruck12:01
*** rfolco has joined #oooq12:01
rlandy|rucksshnaidm|ruck: soniya29|rover: hey ...12:03
soniya29|roverrlandy|ruck, hello12:03
rlandy|rucksshnaidm|ruck: soniya29|rover: I think we should promote queens12:03
soniya29|roverrlandy|ruck, 7 jobs are failing in upstream gate12:04
sshnaidm|ruckrlandy|ruck, how did it finish?12:04
rlandy|rucksshnaidm|ruck: how did what finish?12:04
sshnaidm|ruckrlandy|ruck, queens12:05
rlandy|rucksshnaidm|ruck: it's 21 days out ... wes was just looking for the best match results and editing the criteria to promote12:06
rlandy|rucksshnaidm|ruck: you and I have access to the promoter12:06
rlandy|ruckpromoter server  - to do that12:06
rlandy|ruckhttp://38.102.83.109/centos7_queens.log12:06
rlandy|ruck2020-06-23 11:52:27,667 7090 INFO     promoter Skipping promotion of centos7-queens from tripleo-ci-testing to current-tripleo, missing successful jobs: [u'periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens']12:07
rlandy|ruck^^ only one missing job12:07
rlandy|ruck2020-06-23 11:52:27,666 7090 DEBUG    promoter Successful jobs for commit: 73823d6340f694992933b7853274362a21486c91, distro: 1f2be4ddcc2ef43e98f16d5d3d32309050f6a9f2, timestamp=159202832212:08
* rlandy|ruck checks criteria12:08
sshnaidm|ruckrlandy|ruck, it's from 13 Jun12:09
rlandy|rucksshnaidm|ruck: better than 21 days12:10
sshnaidm|ruckrlandy|ruck, how about 7e4577fd10d1579d72037343607f9d66b7ed194812:11
sshnaidm|ruckrlandy|ruck, it's from 17 Jun12:11
rlandy|ruck2020-06-23 11:52:27,025 7090 INFO     promoter Checking hash commit: 73823d6340f694992933b7853274362a21486c91, distro: 1f2be4ddcc2ef43e98f16d5d3d32309050f6a9f2,12:12
rlandy|ruckthat's ok12:12
rlandy|rucksshnaidm|ruck: the promoter should pick the latest hash - we just need to make sure the failed jobs are excluded12:13
rlandy|rucksshnaidm|ruck: ie: in this case fs020?12:13
* rlandy|ruck gets on promoter and checks12:13
sshnaidm|ruckrlandy|ruck, well, your 73823d6340f694992933b7853274362a21486c91 is better12:13
sshnaidm|ruckrlandy|ruck, yep12:14
sshnaidm|ruckrlandy|ruck, preparing a patch?12:14
rlandy|rucksshnaidm|ruck: no - checking promotion criteria on promoter - tmating you in12:14
chandankumarrlandy|ruck, Hello, good morning12:15
chandankumarrlandy|ruck, here I have tried the container build https://review.rdoproject.org/r/#/c/28194/12:15
chandankumarrlandy|ruck, and here is the base job https://review.rdoproject.org/r/28195 please have a look12:16
rlandy|ruckchandankumar: hey - one sec - busy with promoter - will ping in a few12:16
chandankumarsure12:16
rlandy|rucksshnaidm|ruck: soniya29|rover: wrt ipa - I think we can merge https://review.opendev.org/#/c/737058/12:19
rlandy|ruckat leats try get rid of one of the issues12:19
mariosneeds reviews please https://review.rdoproject.org/r/#/c/28161 https://review.rdoproject.org/r/#/c/28162 test at https://review.rdoproject.org/r/#/c/2810612:21
soniya29|roverrlandy|ruck, yeah, but RDO Third party CI is failing badly, so will it be suitable in such case to get it merged?12:21
rlandy|rucksoniya29|rover: it shouldn't impact third party at all12:22
rlandy|ruckbut you are welcome to recheck it12:22
soniya29|roverrlandy|ruck, If its not gonna impact then we can merge since we need it now12:23
sshnaidm|rucksoniya29|rover, 3party is failing because escalated card of ovb, that's fine12:23
rlandy|rucksshnaidm|ruck: ovb still failing though12:23
sshnaidm|rucksoniya29|rover, I mean it's not fine, but..12:23
rlandy|rucksoniya29|rover: ipa jobs are failing gates12:23
rlandy|ruckso I12:23
rlandy|ruck'm trying to get that fixed12:24
soniya29|roversshnaidm|ruck, rlandy|ruck, yeah, lets get that merged12:24
rlandy|rucksshnaidm|ruck: soniya29|rover: taking another shot at the other patch12:24
sshnaidm|rucksoniya29|rover, rlandy|ruck, https://bugs.launchpad.net/tripleo/+bug/1884518 - Harald is on this12:24
openstackLaunchpad bug 1884518 in tripleo "OVB metalsmith deployment fails: Failed to attach VIF ... to bare metal node, Node ... is locked by host undercloud" [Critical,Triaged]12:24
rlandy|rucksshnaidm|ruck: yeah - good12:24
rlandy|ruckthat's a crazy failure to debug12:24
rlandy|rucksshnaidm|ruck: lastly - I re-uploaded the ipxe-boot image for downstream12:25
rlandy|rucklooks the same to me12:25
sshnaidm|ruckrlandy|ruck, yeah?12:25
rlandy|rucksshnaidm|ruck: maybe it's not shutting down12:25
sshnaidm|ruckrlandy|ruck, what I saw is ironic fails to set pxe boot on node12:25
rlandy|ruckbut the last job failed on an unrelated undercloud install12:25
rlandy|ruckwill try again later today12:26
sshnaidm|ruckrlandy|ruck, but I was able to set boot pxe manually12:26
sshnaidm|ruckrlandy|ruck, and rerun introspection, and then node shut down :D12:26
rlandy|rucksshnaidm|ruck: in time?12:26
soniya29|roverrlandy|ruck, we need workflow +1 for https://review.opendev.org/#/c/737058/12:26
sshnaidm|ruckrlandy|ruck, I think it just was removed by nodepool then12:26
rlandy|rucksoniya29|rover: done - thanks12:26
rlandy|rucksshnaidm|ruck: let me kick that again12:27
rlandy|ruckI really hope this works now12:27
rlandy|ruckI am so done with downstream and ovb12:27
sshnaidm|ruckrlandy|ruck, btw, that works pretty good: https://review.opendev.org/#/c/737289/12:27
sshnaidm|ruckrlandy|ruck, you can see example of failure in comments12:27
rlandy|rucksshnaidm|ruck: yes - I have that included in the job12:27
rlandy|rucksee depends-on12:28
* rlandy|ruck votes12:28
*** ysandeep|afk is now known as ysandeep12:28
rlandy|rucksshnaidm|ruck: here we go again - hopefully that works12:33
*** pojadhav|afk is now known as pojadhav12:34
*** jpena|lunch is now known as jpena12:35
rlandy|rucksshnaidm|ruck: panda: pls see http://38.102.83.109/centos7_queens.log12:36
rlandy|ruckerror in log12:37
rlandy|ruckFAIL\nERROR ========== centos-binary-haproxy IS NOT BUILT! FIX THIS ASAP!12:37
rlandy|ruckwhat is that?12:37
*** derekh has quit IRC12:40
*** derekh has joined #oooq12:40
sshnaidm|ruckrlandy|ruck, it didn't find containers12:40
rlandy|rucksshnaidm|ruck: we may have to punt until tomorrow when that pipeline runs12:41
rlandy|ruckand then rerun failed jobs12:41
rlandy|ruckpromote our best bet then12:41
rlandy|ruckthoughts?12:41
rlandy|rucksoniya29|rover: https://meet.google.com/jng-htpw-avm12:42
ykarelit's likely old container images deleted in rdo registry as not protected by a named symlink12:43
*** ratailor_ has quit IRC12:44
rlandy|ruckykarel: probably - from June 17?12:45
ykarelrlandy|ruck, iirc only 3 days are kept12:45
rlandy|ruckwell that explains things12:45
ykarelif it's not in tripleo-ci-testing, current-tripleo etc12:45
rlandy|ruckthen we can't promote older hashed12:45
rlandy|ruckhashes12:45
sshnaidm|ruckrlandy|ruck, damn, again12:46
rlandy|rucksshnaidm|ruck: ^^ well I guess we'll just hope for a good run tomorrow12:46
rlandy|ruckor we can edit the pipeline to run today12:46
sshnaidm|ruckykarel, it IS in tripleo-ci-testing though12:46
ykarelsshnaidm|ruck, current tripleo-ci-testing12:46
ykarelhttps://trunk.rdoproject.org/centos7-queens/tripleo-ci-testing/delorean.repo12:47
sshnaidm|ruckykarel, we should keep the previous too12:47
sshnaidm|ruckykarel, oh, no, it's previous-current-tripleo12:48
sshnaidm|ruckwell, that explains12:48
rlandy|rucktomorrow - otherwise we're out of the water here12:51
ykarelrlandy|ruck, sshnaidm|ruck yes it's three days https://review.rdoproject.org/r/gitweb?p=rdo-infra/rdo-infra-playbooks.git;a=blob;f=roles/rdo-infra/registry-image-pruning/defaults/main.yml#l3112:51
ykarelif not in whitelist https://review.rdoproject.org/r/gitweb?p=rdo-infra/rdo-infra-playbooks.git;a=blob;f=roles/rdo-infra/registry-image-pruning/defaults/main.yml#l212:51
sshnaidm|ruckykarel, ack, asking if it can be depending on release12:52
sshnaidm|ruckno need to delete every 3 days if containers are built once in 4 days12:52
ykarelsshnaidm|ruck, for now it's 3 days, yes can be changed12:53
sshnaidm|ruckykarel, I think it should be not by days amount12:54
sshnaidm|ruckbut on tags amount12:54
marioschandankumar: thanks for review i replied at https://review.rdoproject.org/r/#/c/28162/5/zuul.d/jobs-promoter-centos-7.yaml when you next have time thank you12:56
ykarelsshnaidm|ruck, ack, u can propose and it can be evaluated12:57
ykareliirc earlier it used to be 10 days12:57
ykarelbut yes 3 days in this case is low12:58
ykarelwhen there are no regular promotions12:58
ykarelis there known blocker in queens?12:58
ykarelif not a fresh run can be promoted for now to clear12:58
sshnaidm|ruckykarel, no blockers afaik, but it runs twice a week only12:59
ykarelthrice12:59
sshnaidm|ruckykarel, I think today is next time13:00
rlandy|ruckchandankumar: let's chat after community meeting13:00
ykareltomorrow, wednesday and weekend13:00
chandankumarrlandy|ruck, sure13:00
ykarelsshnaidm|ruck, last failure should be fixed by your patches13:00
ykareldocker_namespace one13:01
ykarelso i think good to trigger it and see if all clear13:01
sshnaidm|ruckyeah, I see periodic runs fine13:02
sshnaidm|ruckso should be ok13:02
*** amoralej|lunch is now known as amoralej13:05
sshnaidm|ruckrlandy|ruck, introspection fails on ussuri for 1 node of 413:27
mariosarxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar community call in 3 mins add agenda items https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both13:27
sshnaidm|ruckrlandy|ruck, did you try larger timeout for introspection on psi?13:27
rlandy|rucksshnaidm|ruck: I did13:27
rlandy|ruckstill never shut down13:28
rlandy|ruckbut that was before I uploaded the newer image13:28
rlandy|rucklet's see what this run does13:28
sshnaidm|ruckrlandy|ruck, ok, let's hold a node also13:28
rlandy|rucksshnaidm|ruck: introspection always fails on one node for ussuri?13:28
sshnaidm|ruckI'd like to experiment13:28
rlandy|rucksshnaidm|ruck: oh - I can do that now13:28
rlandy|ruckI have a token13:28
sshnaidm|ruckrlandy|ruck, from some jobs that I saw13:28
* rlandy|ruck has to try it out13:28
rlandy|rucksshnaidm|ruck I didn;t see any easy way to up the timeout13:29
rlandy|ruckI landed up editing the introspection playbook13:29
sshnaidm|ruckrlandy|ruck, isn't it a parameter??13:29
sshnaidm|ruckrlandy|ruck, lemme check.13:29
mariosarxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar13:32
marioscommunity call13:32
zbro/13:32
mariosE.g. the test there https://review.rdoproject.org/r/#/c/28190 is trying to use https://review.rdoproject.org/r/#/c/28189/2/ci-scripts/infra-setup/roles/get_hash/tasks/get_hash.yaml but it is ignored - no ‘sanity check’ in logs https://logserver.rdoproject.org/90/28190/2/check/periodic-tripleo-centos-8-train-component-baremetal-promote-consistent-to-component-ci-testing/bfcb93b/job-output.txt13:37
mariossshnaidm|ruck: ^^13:38
mariosrlandy|ruck: ^13:40
mariosDepends-On: https://review.rdoproject.org/r/2818913:41
chandankumarrlandy|ruck, want to discuss container build stuff right now?13:49
rlandy|ruckchandankumar: ack - 5 mins - just need to hold a node13:51
*** sshnaidm|ruck is now known as sshnaidm|afk13:51
chandankumarrlandy|ruck, sure, let me know when ready13:51
rlandy|ruckchandankumar: k - let's meet14:04
chandankumarrlandy|ruck, https://meet.google.com/ymj-yetp-muv?pli=1&authuser=014:04
rlandy|ruckjoining14:05
chandankumarrlandy|ruck, https://review.rdoproject.org/r/#/c/28195/14:09
mariosneeds reviews please https://review.rdoproject.org/r/#/c/28161 https://review.rdoproject.org/r/#/c/28162 test at https://review.rdoproject.org/r/#/c/2810614:13
mariospromoter staging/molecule periodics https://tree.taiga.io/project/tripleo-ci-board/task/180114:13
mariosthank you14:13
rlandy|rucksshnaidm|afk: introspection coming up14:14
chandankumarrlandy|ruck, https://logserver.rdoproject.org/94/28194/1/check/periodic-tripleo-build-containers-ubi-8/4f642b7/logs/containers-successfully-built.log14:17
chandankumarrlandy|ruck, https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/config/standalone.py#L18214:24
*** TrevorV has joined #oooq14:24
rlandy|rucksoniya29|rover: let's touch base out failing scenario 001, 00 10 in gates before you leave14:29
soniya29|roverrlandy|ruck, okay14:32
chandankumarhttps://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/templates/build.sh.j2#L6114:33
rlandy|ruckchandankumar: https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/3rd_party.html14:36
*** skramaja has quit IRC14:38
*** sshnaidm|afk is now known as sshnaidm|ruck14:39
*** ysandeep is now known as ysandeep|afk14:39
soniya29|roverrlandy|ruck, will be back in 1 hr14:43
*** ysandeep|afk is now known as ysandeep14:46
*** Goneri has joined #oooq14:50
rlandy|rucksshnaidm|ruck: soniya29|rover: gate is a disaster14:52
mariosplease add to your review queue https://review.rdoproject.org/r/#/c/28161 https://review.rdoproject.org/r/#/c/28162 test at https://review.rdoproject.org/r/#/c/28106 promoter staging/molecule periodics https://tree.taiga.io/project/tripleo-ci-board/task/180114:53
sshnaidm|ruckmarios, https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-base-promote-hash/run.yaml#L7:L714:54
sshnaidm|ruckmarios, so it's called from trusted repo14:54
sshnaidm|ruckmarios, bad luck14:54
mariossshnaidm|ruck: right14:55
mariossshnaidm|ruck: wow14:55
mariossshnaidm|ruck: ack thanks for that14:55
*** marios is now known as marios|call14:59
rlandy|ruckmving here15:03
rlandy|rucksshnaidm|ruck: what do you want to do about promotions that are blocked by OVB?15:04
rlandy|ruckjust leave ussuri train etc until OVB is fixed?15:04
sshnaidm|ruckrlandy|ruck, need to ignore ovb for now15:04
sshnaidm|ruckrlandy|ruck, nope, let's promote15:04
rlandy|rucksshnaidm|ruck: so we can edit criteria for ussuri15:05
sshnaidm|ruckrlandy|ruck, we already had a big delay because of wrong containers config15:05
rlandy|ruckbut I think that is the old promoter?15:05
sshnaidm|ruckrlandy|ruck, all except master in vexx promoter15:05
sshnaidm|ruckat least was recently15:05
sshnaidm|ruckrlandy|ruck, well, not sure about ussuri15:06
rlandy|rucksshnaidm|ruck: weshay_pto  said only one can deal with componentized promotions15:06
rlandy|ruckI think that would be rdocloud then15:06
rlandy|rucksame as master15:06
sshnaidm|ruckk15:06
rlandy|ruckwant to jump on that one and edit?15:06
* rlandy|ruck can tmate15:06
sshnaidm|ruckrlandy|ruck, sure15:07
rlandy|ruckk - sec15:07
*** owalsh_ is now known as owalsh15:07
rlandy|rucksshnaidm|ruck: side note - introspection failed again on internal - node should be held for when you want to look at it15:10
rlandy|rucksshnaidm|ruck: let's do this - on tamte on rdocloud promoter15:12
rlandy|rucktamte15:12
chandankumarrlandy|ruck, we got the testingmaster namespace for testing container push15:13
rlandy|ruckchandankumar: cool - go wild15:14
rlandy|ruckwith --authfile15:14
chandankumarlet me put patches authfile15:14
chandankumarpatch coming soon15:14
rlandy|ruckwill be fine if we just don't collect it15:14
rlandy|ruckawesome15:14
rlandy|ruckanybody's guess what authfile looks like15:15
*** ykarel is now known as ykarel|afk15:33
rlandy|rucksshnaidm|ruck: ugh - so my node didn't hold - will kick again -but it look like the nodes shut down when introspection fails15:35
rlandy|ruckat the 20 min mark15:35
rlandy|ruckwhen I increased the timeout, it just shutdown later15:35
*** ysandeep is now known as ysandeep|away15:36
sshnaidm|ruckrlandy|ruck, I think by default we retry to introspect if some node failed it15:36
rlandy|ruckreq-cc625c1d-4d47-4fdc-921e-f08db1d6907b Stop June 23, 2020, 2:35 p.m. 08e376a648608779c7d381265e42c2f2633c7fba38987122547abc8e582bfb10 -15:36
rlandy|ruckreq-ede9728d-67cc-48ac-8bcf-732b8182e3aa Start June 23, 2020, 2:15 p.m. 08e376a648608779c7d381265e42c2f2633c7fba38987122547abc8e582bfb1015:36
sshnaidm|ruckrlandy|ruck, I'd like to see if it really retries..15:36
rlandy|ruck^^ see the shutdown at exactly 20 mins15:37
rlandy|ruckok - let me kick again15:38
rlandy|ruckI'll ditch these nodes15:38
sshnaidm|ruckrlandy|ruck, do we have logs from the last job?15:40
rlandy|rucksshnaidm|ruck: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/7d77ca91d9674f69a3f2402bb738ae7f15:40
*** ykarel|afk is now known as ykarel15:55
*** marios|call is now known as marios|out15:59
sshnaidm|ruckrlandy|ruck, where you edit timeout, can you set retries to 0? and make timeout like an hour for example15:59
rlandy|rucksshnaidm|ruck: in my previous try of that, the nodes just shutdown later - when introspection dies - but we'll see16:00
sshnaidm|ruckrlandy|ruck, I'd blame retries there16:03
rlandy|rucksshnaidm|ruck: how so?16:03
sshnaidm|ruckI think it doesn't wait enough before retrying, I saw only 10 seconds16:04
rlandy|ruckinteresting16:04
* rlandy|ruck tries to edit playbook16:10
rlandy|rucksshnaidm|ruck: I edited the playbook ...16:13
rlandy|ruck    max_retries: 016:13
rlandy|ruck    node_timeout: 180016:13
rlandy|rucklet's see what happens16:13
*** marios|out has quit IRC16:13
sshnaidm|ruckgreat16:14
rlandy|ruckif I ste retries to 0, I don;t think retry_delay will work16:26
*** dtantsur is now known as dtantsur|afk16:29
*** amoralej is now known as amoralej|off16:30
*** ykarel is now known as ykarel|away16:30
sshnaidm|ruckrlandy|ruck, yeah, let's keep it 016:32
rlandy|ruckk16:32
sshnaidm|ruckrlandy|ruck, anyway need to work on delays, not the optimal now16:33
rlandy|ruckyeah ok16:35
sshnaidm|ruckrlandy|ruck, interesting, I see ussuri ovb passed here: https://review.opendev.org/#/c/737289/16:37
rlandy|rucksshnaidm|ruck: ha - nice change :)16:38
rlandy|ruckmaybe all nodes need some extra time16:38
sshnaidm|ruckrlandy|ruck, yeah, and maybe need to set it as a parameter, may be bigger on real bm16:42
rlandy|rucksshnaidm|ruck: funnily enough real bm does ok16:42
sshnaidm|ruckok)16:42
soniya29|roverrlandy|ruck, gate is horrible17:03
rlandy|rucksoniya29|rover: no worries- taken care o17:04
rlandy|ruckof17:04
rlandy|ruckinfra17:04
rlandy|ruckthere is one last job with the yellow post failures17:05
*** udesale_ has quit IRC17:05
rlandy|rucksshnaidm|ruck: introspection in action zuul@10.0.111.11717:09
sshnaidm|ruckrlandy|ruck, watching17:11
*** jpena is now known as jpena|off17:11
rlandy|rucktimeout set to 18017:14
rlandy|ruck180017:14
*** derekh has quit IRC17:15
*** sshnaidm|ruck is now known as sshnaidm|afk17:24
chandankumarrlandy|ruck, around?17:39
chandankumarrlandy|ruck, when we do podman login -u -p17:39
rlandy|ruckchandankumar: yes- what's up?17:39
chandankumarthen it generates auth.json file17:39
rlandy|ruckchandankumar: before we push'17:39
chandankumarwhich lives under cat /run/user/1000/containers/auth.json17:39
chandankumaryes before push17:39
chandankumarrlandy|ruck, we can reuse that path during push17:39
rlandy|ruckchandankumar: so downstream I auth on every push17:39
rlandy|ruckI think you can17:39
chandankumarlet me propose the patches17:40
chandankumarand we are done17:40
chandankumar${XDG_RUNTIME_DIR}/containers/auth.json is the default path17:40
chandankumarand will take a look tomorrow17:40
rlandy|ruckok17:44
rlandy|ruckchandankumar: hey - can you look at a tempest test for a sec?17:49
rlandy|ruckhttps://d23c12b0aad63e0703e8-47c67d6f96b63324a4b586b546351bb6.ssl.cf5.rackcdn.com/729465/43/check/tripleo-ci-centos-8-scenario010-standalone/ae53ac9/logs/undercloud/var/log/tempest/stestr_results.html17:49
rlandy|rucktiming?17:49
*** jtomasek has quit IRC20:06
*** jfrancoa has quit IRC20:43
*** TrevorV has quit IRC21:35
*** jbadiapa has quit IRC21:38
*** Goneri has quit IRC21:46
*** rlandy|ruck is now known as rlandy|ruck|bbl22:32
*** jschlueter has quit IRC22:37
*** tosky has quit IRC22:42
*** chem has quit IRC22:53
« Monday, 2020-06-22 Index Wednesday, 2020-06-24 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!