| rlandy | periodic-tripleo-ci-rhel-8-bm_envE-3ctlr_1comp-featureset001-baremetal-rhos-17openstack/tripleo-ci-internal-jobsmastercheck201352,164 hrs 44 mins 57 secs2020-06-16 19:22:16SUCCESS | 00:09 |
|---|---|---|
| rlandy | yay! | 00:09 |
| rlandy | weshay|ruck: ^^ small celebration | 00:09 |
| rlandy | sshnaidm: introspection patch is under test on BM ... https://code.engineering.redhat.com/gerrit/#/c/201352/ | 00:11 |
| *** jmasud has joined #oooq | 00:20 | |
| *** Goneri has joined #oooq | 00:54 | |
| *** holser has quit IRC | 01:02 | |
| *** holser has joined #oooq | 01:03 | |
| *** Goneri has quit IRC | 01:48 | |
| *** jmasud has quit IRC | 02:18 | |
| *** jmasud has joined #oooq | 02:37 | |
| *** apetrich has quit IRC | 02:42 | |
| *** jmasud has quit IRC | 03:24 | |
| *** jmasud has joined #oooq | 03:25 | |
| *** ysandeep|away is now known as ysandeep | 03:55 | |
| ysandeep | pojadhav|ruck, Hi good morning, Do we have a bug for these intermittent timeouts - https://4c474afa65738b153370-596781338ef27878bb1588affd1dfa8a.ssl.cf2.rackcdn.com/735428/4/check/tripleo-ci-centos-8-scenario007-multinode-oooq-container/36084fe/job-output.txt | 04:02 |
| ysandeep | 3 job failed with similiar error and I am also facing similiar issue in downstream for multinode jobs- https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario007-multinode-oooq-container | 04:02 |
| ysandeep | ykarel|away, fyi.. Similiar timeout issue in upstream too - https://4c474afa65738b153370-596781338ef27878bb1588affd1dfa8a.ssl.cf2.rackcdn.com/735428/4/check/tripleo-ci-centos-8-scenario007-multinode-oooq-container/36084fe/job-output.txt | 04:03 |
| *** jmasud has quit IRC | 04:05 | |
| *** jmasud has joined #oooq | 04:06 | |
| *** ykarel|away is now known as ykarel | 04:08 | |
| ykarel | ysandeep, cool | 04:08 |
| ykarel | can u try to reproduce again in downstream | 04:08 |
| ykarel | and get node held | 04:08 |
| ysandeep | ykarel, yes will try, will request node held if it stuck at that point.. after stuck it stay on that point for another 1 hour.. | 04:09 |
| ykarel | ysandeep, i think node held request is still there, as it passed yesterday | 04:10 |
| ysandeep | ykarel, ohhh.. good then | 04:11 |
| ysandeep | ykarel, posted recheck https://code.engineering.redhat.com/gerrit/#/c/202941/ .. will try to access node in few minutes as soon as I get IP info from job-output.txt | 04:12 |
| ykarel | ysandeep, ack, but keys needs to be put there | 04:15 |
| ysandeep | ykarel, okay i thought we pass keys during node held request. no worries - will request some in #rhos-ops if it get stuck | 04:17 |
| ykarel | ack | 04:18 |
| *** jmasud has quit IRC | 04:40 | |
| *** jmasud has joined #oooq | 05:03 | |
| *** jfrancoa has joined #oooq | 05:06 | |
| *** ratailor has joined #oooq | 05:10 | |
| *** jmasud has quit IRC | 05:16 | |
| *** jmasud has joined #oooq | 05:19 | |
| *** marios has joined #oooq | 05:20 | |
| *** udesale has joined #oooq | 05:33 | |
| *** skramaja has joined #oooq | 05:35 | |
| pojadhav|ruck | ysandeep, nope.. we dont have bug.. let me check and then will file a bug. | 05:46 |
| ysandeep | pojadhav|ruck, thanks! | 05:46 |
| *** amoralej|off is now known as amoralej | 07:07 | |
| *** dtantsur has joined #oooq | 07:07 | |
| *** jmasud has quit IRC | 07:08 | |
| *** apetrich has joined #oooq | 07:11 | |
| *** ccamacho has joined #oooq | 07:12 | |
| *** tosky has joined #oooq | 07:36 | |
| *** jpena|off is now known as jpena | 07:55 | |
| *** ykarel is now known as ykarel|lunch | 08:13 | |
| *** jmasud has joined #oooq | 08:38 | |
| *** pojadhav|ruck is now known as pojadhav|lunch | 08:53 | |
| *** ykarel|lunch is now known as ykarel | 09:03 | |
| *** apetrich has quit IRC | 09:17 | |
| *** apetrich has joined #oooq | 09:26 | |
| *** ysandeep is now known as ysandeep|afk | 09:26 | |
| *** holser has quit IRC | 09:35 | |
| *** holser has joined #oooq | 09:36 | |
| *** ysandeep|afk is now known as ysandeep | 09:54 | |
| *** derekh has joined #oooq | 10:17 | |
| sshnaidm | pojadhav|lunch, hey, do you know if multinode ussuri job is broken? | 10:26 |
| *** jmasud has quit IRC | 10:39 | |
| chandankumar | I will be away for an hour | 10:44 |
| *** derekh has quit IRC | 11:00 | |
| *** derekh has joined #oooq | 11:00 | |
| pojadhav|lunch | sshnaidm, https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released&job_name=periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri talking about this..? | 11:07 |
| *** pojadhav|lunch is now known as pojadhav|ruck | 11:07 | |
| sshnaidm | pojadhav|ruck, tripleo-ci-centos-8-containers-multinode-ussuri | 11:20 |
| sshnaidm | pojadhav|ruck, https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-containers-multinode-ussuri# | 11:21 |
| pojadhav|ruck | sshnaidm, https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released&job_name=tripleo-ci-centos-8-containers-multinode-ussuri | 11:21 |
| pojadhav|ruck | sshnaidm, let me check | 11:21 |
| sshnaidm | pojadhav|ruck, it's upstream job, not third party | 11:21 |
| pojadhav|ruck | sshnaidm, yup | 11:21 |
| *** jpena is now known as jpena|lunch | 11:30 | |
| *** sshnaidm_ has joined #oooq | 11:32 | |
| *** sshnaidm has quit IRC | 11:33 | |
| *** sshnaidm_ is now known as sshnaidm | 11:35 | |
| *** rfolco|rover has joined #oooq | 11:36 | |
| *** jschlueter has quit IRC | 11:44 | |
| rlandy | ysandeep: hi - there were two successful runs on env E yesterday | 11:51 |
| rlandy | I ma going to +2 the combined patch | 11:51 |
| rlandy | I need to remove the extra files on my review to test sshnaidm patch - but then that job can go as well into the baremetal component line | 11:52 |
| rlandy | ysandeep: env E runs faster so we are less likely to get the timeout env A gets | 11:52 |
| ysandeep | rlandy, upstream ci is busted for some jobs on the combined patch | 11:52 |
| rlandy | ysandeep: yep - upstrea ci is messed up in general ;( | 11:52 |
| rlandy | no worries, I'll watch it in my afternoon | 11:52 |
| rlandy | ysandeep: I ma going to try increase the tenant timeout so we can up the timeout in your env A job | 11:53 |
| ysandeep | rlandy, found why it was timeouting sometimes on envA - https://code.engineering.redhat.com/gerrit/#/c/203408/ will fix that | 11:53 |
| rlandy | ok | 11:53 |
| ysandeep | rlandy, conflicting range b/w tempest and vip | 11:54 |
| rlandy | ysandeep: ok - so we need one job in tripleo line and one in baremetal | 11:54 |
| ysandeep | rlandy, which environemt on which line? if you have something in mind already | 11:54 |
| ykarel | chandankumar, rlandy can u check https://review.opendev.org/#/c/735210/ +W missing | 11:55 |
| rlandy | ysandeep: it doesn't matter really | 11:55 |
| weshay|ruck | sshnaidm, rlandy rdo cloud has a spike in old stacks.. anyone deal w/ that fun lately? I haven't seen that in a long time.. /me assumes we still have scripts cleaning it out automatically right? | 11:55 |
| rlandy | ykarel: +w 'ed | 11:55 |
| sshnaidm | weshay|ruck, I'll look into it, I disabled cleaning intentionally | 11:55 |
| sshnaidm | to check where can we leak stacks | 11:56 |
| ykarel | rlandy, Thanks | 11:56 |
| *** tosky has quit IRC | 11:56 | |
| rlandy | weshay|ruck: ha - I need to put that script into downstream | 11:56 |
| rlandy | thanks for reminder | 11:56 |
| rlandy | ysandeep: merging https://code.engineering.redhat.com/gerrit/#/c/203408 | 11:56 |
| rlandy | ysandeep: it makes no diff which env we put in which line | 11:57 |
| rlandy | let me check the reviews - sec | 11:57 |
| ysandeep | rlandy, thanks! | 11:58 |
| rlandy | ysandeep: ok - here's my suggestion ... | 11:58 |
| rlandy | https://code.engineering.redhat.com/gerrit/#/c/201352/18/zuul.d/component-jobs.yaml | 11:58 |
| *** tosky has joined #oooq | 11:59 | |
| rlandy | ysandeep: I think you used the same job defintions as I did for https://code.engineering.redhat.com/gerrit/#/c/201352/18/zuul.d/baremetal-jobs.yaml? | 11:59 |
| rlandy | right? | 11:59 |
| rlandy | those should be consistent | 11:59 |
| rlandy | ysandeep: if so, ... | 11:59 |
| rlandy | my suggestion is that we both define jobs for the triple and baremetal lines | 11:59 |
| sshnaidm | seems like something happened on rdo-cloud in 4.00 UTC | 11:59 |
| rlandy | so that we can switch out if need be | 12:00 |
| ysandeep | rlandy, yes i kept them consistent with yours | 12:00 |
| rlandy | ysandeep: ok - so then I think let's both put in component line definitions for both tripleo and baremetal | 12:00 |
| rlandy | we can merge those | 12:00 |
| rlandy | once the depends on review is merged in the upstream, | 12:01 |
| rlandy | since env E runs faster, I think we should add that one to the tripleo line | 12:01 |
| rlandy | and env A to baremetal - since that one starts first | 12:01 |
| rlandy | ysandeep: thoughts? | 12:01 |
| rlandy | but we can merge the job definitions for both | 12:02 |
| ysandeep | rlandy, I agree with your suggesstions | 12:02 |
| rlandy | ysandeep: cool | 12:02 |
| rlandy | marios: thanks for your help yesterday - we have uploaded dockerfiles | 12:03 |
| rlandy | weshay|ruck: ^^ fyi | 12:03 |
| marios | rlandy: ack thanks (didn't do anything but I'll take it ;)) | 12:04 |
| rfolco|rover | retro call | 12:04 |
| rfolco|rover | ping | 12:04 |
| rfolco|rover | arxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar | 12:04 |
| panda | oh | 12:05 |
| panda | ouch | 12:05 |
| chandankumar | rfolco|rover, there is no invite | 12:05 |
| panda | chandankumar: I have it. | 12:06 |
| chandankumar | panda, ah good, gmeet link please | 12:06 |
| panda | chandankumar: ems-focw-fex | 12:07 |
| *** amoralej is now known as amoralej|lunch | 12:09 | |
| *** ratailor has quit IRC | 12:11 | |
| rlandy | chandankumar: marios: ysandeep: https://code.engineering.redhat.com/gerrit/#/c/200707/ ... re: 'springing free of compose" will merge after meeting | 12:24 |
| *** jpena|lunch is now known as jpena | 12:37 | |
| *** Tengu has quit IRC | 12:39 | |
| *** Tengu has joined #oooq | 12:40 | |
| jfrancoa | rfolco|rover: hey, as you are the rover, are you aware of this error: http://pastebin.test.redhat.com/876107 ? its happening in all our master tripleo-upgrade patches and we can't merge anything: https://review.opendev.org/#/c/698526/ | 13:05 |
| chandankumar | weshay|ruck, want to talk right now? | 13:07 |
| rfolco|rover | jfrancoa, not sure, this looks that virtualenv issue removed from upstream image | 13:07 |
| *** amoralej|lunch is now known as amoralej | 13:07 | |
| jfrancoa | rfolco|rover: yep, it doesn't look related to the repository itself...looks like more an environmental issue | 13:08 |
| chandankumar | rfolco|rover, weshay|ruck we need to find ruck rover also for next sprint | 13:09 |
| rfolco|rover | jfrancoa, which job you referring to? tripleo-upgrade-centos-8-molecule RETRY_LIMIT in 4m 01s | 13:09 |
| rfolco|rover | ? | 13:09 |
| jfrancoa | rfolco|rover: exactly | 13:09 |
| jfrancoa | rfolco|rover: and there are more patches where the same thing is happening: https://review.opendev.org/#/c/734783/ | 13:09 |
| rfolco|rover | jfrancoa, ok gime a few and we'll find the fix | 13:13 |
| rlandy | chandankumar: marios: ysandeep: merging https://code.engineering.redhat.com/gerrit/#/c/200707/ - tested on standalone. if your jobs break, we can revert | 13:13 |
| ysandeep | rlandy, ack o/ | 13:14 |
| chandankumar | rlandy, +1 | 13:14 |
| jfrancoa | rfolco|rover: cool, thanks! I've checked other repos and tripleo-ansible is having the same blockage : https://review.opendev.org/#/c/736024/ | 13:14 |
| rlandy | ysandeep: chandankumar: great - thanks | 13:14 |
| jfrancoa | rfolco|rover: do you need a lp bug? | 13:14 |
| rfolco|rover | I am looking for it.... I have other gate failures to open bugs, so bare with me | 13:15 |
| rlandy | ysandeep: fyi ... https://code.engineering.redhat.com/gerrit/#/c/203392/1/hardware_environments/hp_dl360_envE/network_configs/single_nic_vlans/env_settings.yml | 13:15 |
| rlandy | I think I am going to merge that | 13:15 |
| rlandy | to use sshnaidm's change https://review.opendev.org/#/c/734798 | 13:15 |
| marios | rlandy: ack thanks | 13:15 |
| ysandeep | rlandy, looking | 13:16 |
| rlandy | ysandeep: you just get extra debug | 13:16 |
| rlandy | ysandeep: +2'ed https://review.opendev.org/#/c/734654 | 13:17 |
| ysandeep | rlandy, fine with me - just want to confirm do you want to do that on envE only or envA also? | 13:17 |
| rlandy | will try get that through this afternoon | 13:17 |
| rlandy | ysandeep: well, as long as it doesn't add to your job time, it's good to have on env A | 13:18 |
| rlandy | extra debug can be helpful | 13:18 |
| rlandy | ysandeep: I'm abandoning https://review.opendev.org/#/c/729415/ for the combined patch | 13:19 |
| ysandeep | rlandy, I am just afraid of pcap file size - incase of some issue during introspection | 13:19 |
| ysandeep | but lets see - we can only know by trying | 13:20 |
| rlandy | ysandeep: my last run on env E tested it - but we can watch and see how things go on env E | 13:20 |
| rlandy | and yo can add to env A later | 13:20 |
| rlandy | I am going to merge my job definitions (baremetal and tripleo pipelines) | 13:21 |
| ysandeep | yup, i think it would be good to kill pcap after some time 10-15 min max | 13:21 |
| rlandy | ok - let's see how it goes | 13:21 |
| pojadhav|ruck | rfolco|rover, jfrancoa : https://bugs.launchpad.net/tripleo/+bug/1883909 | 13:25 |
| openstack | Launchpad bug 1883909 in tripleo "Failed to find required executable virtualenv in paths: /home/zuul/.local/bin:/home/zuul/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin" [High,Triaged] | 13:25 |
| rfolco|rover | thanks pojadhav|ruck | 13:25 |
| rlandy | ysandeep: pls review https://code.engineering.redhat.com/gerrit/#/c/201352/ | 13:26 |
| *** udesale_ has joined #oooq | 13:26 | |
| jfrancoa | pojadhav|ruck: thanks! | 13:26 |
| rfolco|rover | pojadhav|ruck, can you please search for the venv fix we had few days ago and see if that can be also fix this one ? | 13:26 |
| pojadhav|ruck | rfolco|rover, jfrancoa : issue will be fix by this patch https://review.opendev.org/#/c/736183/ | 13:26 |
| ysandeep | rlandy, ack | 13:26 |
| rfolco|rover | pojadhav|ruck, nice it just merged | 13:27 |
| rfolco|rover | thanks zbr ;) | 13:27 |
| *** udesale has quit IRC | 13:28 | |
| weshay|ruck | panda, https://hackmd.io/4jAfexu4TBivhareCb4MhQ | 13:29 |
| ysandeep | rlandy, looks perfect to me | 13:29 |
| ysandeep | rlandy, i will clean my BM patch and will share in sometime | 13:30 |
| rlandy | ysandeep: great | 13:31 |
| rlandy | merging so you can rebase | 13:31 |
| *** TrevorV has joined #oooq | 13:37 | |
| *** rlandy is now known as rlandy|mtg | 13:41 | |
| rfolco|rover | sshnaidm, please revote | 13:58 |
| rfolco|rover | https://review.opendev.org/#/c/736196/4 | 13:58 |
| rfolco|rover | weshay|ruck, ^ | 13:58 |
| rfolco|rover | a promotion would fix it, but our ovb jobs are timing out in ussuri... https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released | 13:59 |
| ykarel | rfolco|rover, instead of ^^ may be u can look on promotion, that can clear it more quickly on both master and stable/ussuri | 13:59 |
| rfolco|rover | ykarel, meanwhile looking at ussuri jobs to promote | 13:59 |
| ykarel | rfolco|rover, ovb is failing in check too both master/ussuri, i think sshnaidm is debugging that only | 13:59 |
| sshnaidm | ykarel, rfolco|rover preparing a bug now.. | 14:00 |
| ykarel | sshnaidm, ack | 14:00 |
| rfolco|rover | weshay|ruck, think we can relax ussuri criteria ? https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released | 14:00 |
| rfolco|rover | full-tempest is inconsistent, fails a different test every run | 14:01 |
| rfolco|rover | and ovb jobs are timing out | 14:01 |
| rfolco|rover | weshay|ruck, we depend on a ussuri promotion to clear gate issues, thats the context ^ | 14:02 |
| sshnaidm | https://bugs.launchpad.net/tripleo/+bug/1883913 | 14:05 |
| openstack | Launchpad bug 1883913 in tripleo "Introspection fails in OVB, bmc error: openstackbmc: struct.error: unpack requires a string argument of length 2" [Critical,Triaged] | 14:05 |
| sshnaidm | rfolco|rover, weshay|ruck ^ | 14:05 |
| *** ysandeep is now known as ysandeep|brb | 14:07 | |
| *** pojadhav|ruck is now known as pojadhav|afk | 14:11 | |
| rlandy|mtg | panda: fyi .. the internal promoter will need another step | 14:16 |
| rlandy|mtg | same as images ... we need to promote dockerfiles | 14:16 |
| rlandy|mtg | putting link in pvt | 14:17 |
| panda | rlandy|mtg: .... | 14:19 |
| ykarel | sshnaidm, in rdo cloud and vexxhost bmc image is same or different? | 14:20 |
| ykarel | as a quick check can u see what pyghmi version is there | 14:20 |
| *** rlandy|mtg is now known as rlandy | 14:39 | |
| rlandy | panda: ? | 14:40 |
| rlandy | did you post a response somewhere I missed? | 14:40 |
| rlandy | chandankumar: hi - just checking up on the build-test packages error " ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt" | 14:48 |
| rlandy | anything I can do to help there? | 14:48 |
| rlandy | I know your patch is failing in gates | 14:49 |
| chandankumar | rlandy, dlrn_pre_installed: true to false | 14:49 |
| chandankumar | till we get the build test packages patch lands | 14:50 |
| rlandy | chandankumar: where would I set that? | 14:50 |
| chandankumar | rlandy, rhos-17 release file | 14:50 |
| rlandy | in the release file? | 14:50 |
| rlandy | ah ok | 14:50 |
| rlandy | chandankumar: putting up patch | 14:50 |
| rlandy | chandankumar: https://code.engineering.redhat.com/gerrit/203483 Disable dlrn_pre_installed while waiting on patch fix | 14:55 |
| rlandy | marios: ysandeep|brb: ^^ fyi | 14:56 |
| rlandy | to get downstream jobs running again | 14:56 |
| *** Goneri has joined #oooq | 14:57 | |
| weshay|ruck | chandankumar, "selfLink": "/oapi/v1/namespaces/tripleotraincentos8/imagestreamtags/" | 14:59 |
| *** pojadhav|afk is now known as pojadhav|ruck | 14:59 | |
| marios | rlandy: ack i tried to +1 just now but it told me the change was closed ;) | 15:02 |
| rlandy | marios: sorry - merged | 15:02 |
| marios | rlandy: i mean, i had to go look that up anyway so http://codesearch.openstack.org/?q=dlrn_pre_installed | 15:02 |
| rlandy | need to get jobs back up and running | 15:02 |
| rlandy | going to revert soon anyways | 15:02 |
| marios | rlandy: ack yeah though chkumar's fix for build test packages is nearly in gate | 15:02 |
| marios | rlandy: but sure it could be a whiel if it hits issues merging | 15:03 |
| rlandy | marios: yep but the gate s giving him headaches | 15:03 |
| rlandy | atm we have all our jobs failing | 15:03 |
| rlandy | anyways ... getting back to OVB downstream | 15:04 |
| rlandy | time to tackle another monster | 15:04 |
| *** ysandeep|brb is now known as ysandeep | 15:04 | |
| ysandeep | rlandy, ack - https://code.engineering.redhat.com/gerrit/203483 sry was afk | 15:05 |
| *** ratailor has joined #oooq | 15:05 | |
| sshnaidm | ykarel, added to bug | 15:09 |
| * ykarel check | 15:09 | |
| ykarel | sshnaidm, ack the pyghmi version is not good | 15:10 |
| panda | rlandy: so they are in the same place as overcloud images, and behave like overcloud images ? so promotion is just switching a couple of links ? | 15:11 |
| ykarel | needs >= 1.5.13 | 15:11 |
| rlandy | panda: ack - we will need that for the new container workflow | 15:12 |
| rlandy | ykarel: sshnaidm: are you updating the bmc image? | 15:12 |
| rlandy | need to port that to downstream | 15:13 |
| ykarel | https://review.rdoproject.org/r/#/c/27663/ | 15:13 |
| ykarel | rlandy, /me not doing it, but i think that needs to be updated to clear issues | 15:13 |
| rlandy | ykarel: there was a BZ on this ... | 15:13 |
| chandankumar | rlandy, looks good thanks! | 15:13 |
| rlandy | the resolution said that python-pyghmi was installed via pip and closed it | 15:14 |
| ykarel | rlandy, yes see above patch | 15:14 |
| ykarel | for bz | 15:15 |
| sshnaidm | rlandy, seems like need to update, me can get to it tomorrow only | 15:15 |
| rlandy | ykarel: yep - that BZ :) | 15:15 |
| rlandy | sshnaidm: ok - np - when I see your update, I'll port it to downstream | 15:16 |
| *** sshnaidm is now known as sshnaidm|bbl | 15:17 | |
| rlandy | ykarel: is that only a train/osp-16 problem? afaict downstream in candidate deps, the version is still 1.2.x | 15:18 |
| rlandy | in osp-17 | 15:18 |
| ykarel | rlandy, it's rhel 8.2/centos8.2 bug | 15:18 |
| ykarel | so whatever release on these, should hit it | 15:18 |
| rlandy | ykarel: good - thanks | 15:19 |
| *** jschlueter has joined #oooq | 15:19 | |
| ysandeep | rlandy, https://code.engineering.redhat.com/gerrit/#/c/201630/ i have rebased my patch | 15:27 |
| *** chem has quit IRC | 15:30 | |
| *** chem has joined #oooq | 15:32 | |
| weshay|ruck | rfolco|rover, easy change https://review.opendev.org/#/c/733676/ | 15:32 |
| *** skramaja has quit IRC | 15:46 | |
| marios | panda: can you please check those when you have some time https://review.rdoproject.org/r/28083 https://review.rdoproject.org/r/28123 & test @ https://review.rdoproject.org/r/28106 - the centos7 ones are at least green with the minor fix. for centos8 looks like deps issues but i didn't dig much there (i don't think those are expected to work currently, right?) | 15:51 |
| panda | marios: enqueued | 15:52 |
| marios | panda: thanks | 15:52 |
| *** udesale_ has quit IRC | 15:53 | |
| *** ratailor has quit IRC | 16:04 | |
| *** dtantsur is now known as dtantsur|afk | 16:12 | |
| *** jmasud has joined #oooq | 16:20 | |
| panda | rlandy: I added the change in the internal promoter for the dockerfile | 16:23 |
| *** marios is now known as marios|out | 16:23 | |
| panda | rlandy: but there is no dir with the hash that promoted yesterday | 16:23 |
| panda | do you want to promote again ? | 16:24 |
| rlandy | panda: thanks - dockerfile promotions will only be required when we switch to the new container workflow | 16:24 |
| rlandy | ie; next sprint | 16:24 |
| panda | rlandy: ok | 16:24 |
| rlandy | the dockerfile was a test run I did | 16:24 |
| rlandy | panda: sorry - I should have mentioned that | 16:25 |
| panda | rlandy: is this for upstream too ? | 16:25 |
| rlandy | panda: no we won't upload dockerfiles upstream | 16:25 |
| rlandy | it's only for reldel import | 16:25 |
| panda | rlandy: ok | 16:25 |
| *** ykarel is now known as ykarel|away | 16:33 | |
| rfolco|rover | weshay|ruck, ovb jobs for ussuri timing out, do you want me to testproject those and try a promotion ? https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released | 16:34 |
| ykarel|away | rfolco|rover, ^^ likely same https://bugs.launchpad.net/tripleo/+bug/1883913, so testproject shouldn't help | 16:36 |
| openstack | Launchpad bug 1883913 in tripleo "Introspection fails in OVB, bmc error: openstackbmc: struct.error: unpack requires a string argument of length 2" [Critical,Triaged] | 16:36 |
| *** ysandeep is now known as ysandeep|away | 16:36 | |
| ykarel|away | would need a promotion ignoring those, as it's failing in check too | 16:36 |
| rfolco|rover | ykarel|away, ironic-conductor log is fine, are you sure its the same issue ? | 16:37 |
| rfolco|rover | https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset030-ussuri/327a15a/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz | 16:37 |
| rfolco|rover | weshay|ruck, may day may day | 16:38 |
| weshay|ruck | rfolco|rover, ykarel|away I have to promote ussuri now | 16:41 |
| weshay|ruck | there is a pacemaker mismatch again between current-tripleo containers and baseos | 16:41 |
| weshay|ruck | 3rd party will be down https://trunk.rdoproject.org/api-centos8-ussuri/api/civotes_agg_detail.html?ref_hash=bbbf23b9a9df141aa05aaa7a184411f9 | 16:41 |
| rfolco|rover | weshay|ruck, yes, that's what I am asking :) | 16:42 |
| weshay|ruck | rfolco|rover, do we have a read on the introspection error? | 16:42 |
| rfolco|rover | weshay|ruck, I am not sure this is the same reported by sshnaidm|bbl, but its happening across all ovb jobs | 16:43 |
| weshay|ruck | awesome :( | 16:44 |
| rfolco|rover | looking at the ironic-conductor logs, its not what the bug has | 16:44 |
| rfolco|rover | this is the error | 16:49 |
| rfolco|rover | https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-ussuri/7682746/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz | 16:49 |
| rfolco|rover | ykarel|away is right weshay|ruck, maybe ironic-conductor log is not exactly the same, but the error is the same | 16:52 |
| rfolco|rover | No nodes are manageable at this time. | 16:52 |
| weshay|ruck | rfolco|rover, ykarel|away k.. it's not an ironic issue | 16:53 |
| weshay|ruck | nothing changed there | 16:53 |
| weshay|ruck | last update to ironic was 6/12 | 16:53 |
| weshay|ruck | this started on 6/15 | 16:54 |
| weshay|ruck | https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset001&job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001&branch=stable%2Fussuri | 16:55 |
| weshay|ruck | most likely centos8.2 | 16:55 |
| ykarel|away | yes it's due to itk, needs update for pyghmi in bmc node | 16:55 |
| ykarel|away | that should atleast move it forward | 16:55 |
| *** amoralej is now known as amoralej|off | 16:55 | |
| weshay|ruck | k | 16:56 |
| weshay|ruck | this one just worked :) https://review.rdoproject.org/zuul/build/240a69b6b5f449ae8007cdcba79e4aa6 | 16:57 |
| ykarel|away | :( no clue than | 16:59 |
| *** derekh has quit IRC | 17:01 | |
| *** jbadiapa has quit IRC | 17:05 | |
| *** jpena is now known as jpena|off | 17:06 | |
| rlandy | ysandeep|away: merging https://code.engineering.redhat.com/gerrit/#/c/201630/ | 17:07 |
| *** marios|out has quit IRC | 17:35 | |
| *** jmasud has quit IRC | 17:44 | |
| rlandy | weshay|ruck: do we update containers in undercloud install on ovb? | 18:00 |
| rlandy | is that a thing? | 18:00 |
| weshay|ruck | yes | 18:01 |
| rlandy | trying o follow these new logs is a challenge | 18:01 |
| weshay|ruck | heh | 18:01 |
| rlandy | weshay|ruck: where would I see that update log? | 18:01 |
| weshay|ruck | ya.. so let me qualify that answer | 18:01 |
| weshay|ruck | shit changes.. but afaik yes | 18:01 |
| weshay|ruck | rlandy, it's in the undercloud ansible log I think | 18:01 |
| weshay|ruck | rlandy, point me at the url | 18:02 |
| weshay|ruck | we'll find it | 18:02 |
| rlandy | it was' | 18:02 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/36/200436/15/check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal/5b9b22e/logs/undercloud/home/zuul/ansible.log.txt.gz | 18:02 |
| rlandy | that is all I see in the ansible log right now | 18:02 |
| rlandy | weshay|ruck: ok - I think it is in logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 18:05 |
| rlandy | although I can't fine the task | 18:05 |
| rlandy | find | 18:05 |
| weshay|ruck | sec | 18:06 |
| rlandy | ugh - I hate these new logs | 18:06 |
| weshay|ruck | rlandy, 2020-06-11 18:11:37,935 p=83632 u=zuul n=ansible | 2020-06-11 18:11:37.934135 | fa163e0c-5ee1-f8c7-c518-00000000000c | FATAL | Notice | localhost | error={"changed": false, "msg": "No nodes are manageable at this time."} | 18:19 |
| weshay|ruck | so that's the error? | 18:20 |
| weshay|ruck | rlandy, in the undercloud log | 18:21 |
| weshay|ruck | 2020-06-11 17:46:14.973544 | fa163e0c-5ee1-2db9-d758-00000000010f | TIMING | tripleo_container_image_prepare : Run tripleo_container_image_prepare logged to: /var/log/tripleo-container-image-prepare.log | 0:03:13.658 | 27.32s | 18:21 |
| weshay|ruck | but I don't see any updates logged there.. | 18:22 |
| weshay|ruck | see some 404 | 18:22 |
| *** jmasud has joined #oooq | 18:36 | |
| rlandy | weshay|ruck: that's an older log | 18:44 |
| rlandy | I want to update the pygmi version | 18:44 |
| weshay|ruck | on a container? | 18:45 |
| rlandy | yes | 18:45 |
| rlandy | in fact ofr the component pieplien for ovb | 18:45 |
| rlandy | we should be updating | 18:46 |
| rlandy | https://review.rdoproject.org/zuul/build/9a3ca14ebe2b490996ed12785860a7d7 | 18:46 |
| rlandy | let's look at that | 18:47 |
| weshay|ruck | rlandy, k k.. so just to establish what I'm looking for.... container updates for the undercloud | 18:47 |
| rlandy | for example | 18:47 |
| weshay|ruck | k | 18:47 |
| rlandy | ugh - not standalone | 18:47 |
| rlandy | I need ovb | 18:47 |
| rlandy | stupid me | 18:47 |
| rlandy | https://review.rdoproject.org/zuul/build/6b2f9463315d4d31bc659cbf830de1dc | 18:48 |
| weshay|ruck | rlandy, no updates to install | 18:48 |
| weshay|ruck | rlandy, https://trunk.rdoproject.org/centos8-ussuri/component/baremetal/ | 18:48 |
| weshay|ruck | look at the dates | 18:48 |
| rlandy | https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/6b2f946/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 18:49 |
| rlandy | weshay|ruck: where do you see ' no updates to install' | 18:50 |
| weshay|ruck | rlandy, ur chasing a ghost | 18:50 |
| rlandy | and how do we know what repos are being updated? | 18:50 |
| weshay|ruck | rlandy, there are no rpm updates | 18:50 |
| weshay|ruck | let's chat | 18:50 |
| weshay|ruck | https://meet.google.com/aar-rjwc-mee | 18:51 |
| rlandy | update_repo: "delorean-current,baremetal " | 18:51 |
| rlandy | good | 18:51 |
| rlandy | that's overcloud | 18:51 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/36/200436/15/check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal/5b9b22e/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz | 18:52 |
| rlandy | weshay|ruck: ^^ that is not updating | 18:52 |
| rlandy | so need to fix downstream job | 18:52 |
| weshay|ruck | there is nothing to update | 18:52 |
| weshay|ruck | but yes | 18:52 |
| rlandy | upstream is fine | 18:52 |
| weshay|ruck | that param file is wrong | 18:53 |
| rlandy | I am just comparing | 18:53 |
| rlandy | corrcet | 18:53 |
| weshay|ruck | rlandy, but also look at https://trunk.rdoproject.org/centos8-ussuri/component/baremetal/ | 18:53 |
| rlandy | weshay|ruck: sum total- OVB is not running introspection - we need the new pygmi version | 18:54 |
| rlandy | lon is updating the candidate repo | 18:55 |
| weshay|ruck | k | 18:55 |
| rlandy | I wanted to update containers | 18:55 |
| rlandy | rather than rebuild then all | 18:55 |
| rlandy | atm | 18:55 |
| rlandy | that is all | 18:55 |
| rlandy | weshay|ruck: but I have some idea now about OVB downstream | 18:55 |
| rlandy | getting there | 18:55 |
| weshay|ruck | :) | 18:57 |
| *** jmasud has quit IRC | 19:00 | |
| *** rlandy is now known as rlandy|brb | 19:01 | |
| rfolco|rover | zbr, around? | 19:07 |
| *** sshnaidm|bbl is now known as sshnaidm | 19:09 | |
| *** jmasud has joined #oooq | 19:18 | |
| *** rlandy|brb is now known as rlandy | 19:26 | |
| *** jmasud has quit IRC | 19:36 | |
| rlandy | periodic-tripleo-ci-rhel-8-standalone-on-multinode-ipa-security-rhos-17 success - yay | 19:37 |
| rlandy | we're back | 19:37 |
| weshay|ruck | :) | 19:37 |
| rlandy | weshay|ruck: https://review.opendev.org/#/c/734654/ - worth recheck that or not yet? | 19:38 |
| rlandy | upstream CI stability | 19:38 |
| * weshay|ruck looks | 19:38 | |
| weshay|ruck | ussuri may work now.. | 19:39 |
| weshay|ruck | 3rd party will fail | 19:39 |
| rlandy | ok - rechecking | 19:39 |
| *** jmasud has joined #oooq | 19:41 | |
| *** rlandy_ has joined #oooq | 20:09 | |
| *** tosky_ has joined #oooq | 20:12 | |
| *** tosky has quit IRC | 20:13 | |
| *** tosky_ is now known as tosky | 20:13 | |
| *** irclogbot_3 has quit IRC | 20:14 | |
| *** jrist- has joined #oooq | 20:16 | |
| *** jfrancoa has quit IRC | 20:18 | |
| *** rlandy has quit IRC | 20:18 | |
| *** beagles has quit IRC | 20:18 | |
| *** jrist has quit IRC | 20:18 | |
| *** rlandy__ has joined #oooq | 20:18 | |
| *** rlandy_ has quit IRC | 20:19 | |
| *** irclogbot_3 has joined #oooq | 20:20 | |
| *** jmasud has quit IRC | 20:22 | |
| *** rlandy__ is now known as rlandy | 20:39 | |
| *** jmasud has joined #oooq | 20:53 | |
| *** rfolco|rover has quit IRC | 21:12 | |
| *** jmasud has quit IRC | 21:23 | |
| *** jmasud has joined #oooq | 21:29 | |
| rlandy | ysandeep|away: weshay|ruck: https://code.engineering.redhat.com/gerrit/203511 Add component baremetal jobs to tripleo and baremetal lines | 21:50 |
| rlandy | for when depends-on merges | 21:50 |
| *** sanjayu_ has quit IRC | 21:52 | |
| *** tosky has quit IRC | 21:56 | |
| *** tosky has joined #oooq | 21:56 | |
| *** rlandy is now known as rlandy|bbl | 21:59 | |
| *** jmasud has quit IRC | 22:23 | |
| *** jmasud has joined #oooq | 22:23 | |
| *** TrevorV has quit IRC | 22:36 | |
| *** jmasud has quit IRC | 23:03 | |
| *** tosky has quit IRC | 23:09 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!