Monday, 2020-06-22

*** jmasud has joined #oooq03:07
*** jmasud has quit IRC03:41
*** jmasud has joined #oooq04:12
*** jmasud has quit IRC04:16
*** ratailor has joined #oooq04:16
*** jtomasek has joined #oooq05:29
*** udesale has joined #oooq05:33
*** jmasud has joined #oooq05:34
*** marios has joined #oooq05:35
*** saneax has joined #oooq06:10
*** jmasud has quit IRC06:25
*** jmasud has joined #oooq06:29
*** soniya29|rover is now known as soniya29|ruck06:33
*** jmasud has quit IRC06:48
*** jmasud has joined #oooq06:50
*** jmasud has quit IRC07:01
*** ChanServ sets mode: +o arxcruz07:26
*** bhagyashris is now known as bhagyashris|lunc07:27
*** tosky has joined #oooq07:29
*** sshnaidm|off is now known as sshnaidm|ruck07:29
sshnaidm|rucksoniya29|ruck, hi07:30
*** amoralej|off is now known as amoralej07:35
*** jfrancoa has joined #oooq07:41
*** jmasud has joined #oooq07:46
*** jpena|off is now known as jpena07:56
*** raukadah is now known as chandankumar07:58
*** soniya29|ruck is now known as soniya29|rover08:01
soniya29|roversshnaidm, hello08:02
*** jmasud has quit IRC08:03
soniya29|roversshnaidm, multinode fs-10-stein job is failing due to Image not found exception, here is the bug link:- https://bugs.launchpad.net/tripleo/+bug/188448708:03
openstackLaunchpad bug 1884487 in tripleo "fs10-stein job is consitently failing with ImageNotFoundException: Not found image: docker://docker.io/tripleostein/centos-binary-cron:645e66121aedd5ffe7d55deccad8905c20ae0a4d_e4d61247" [Critical,Triaged]08:03
*** dtantsur|afk is now known as dtantsur08:10
zbrnote: today me and panda are out (not changing nick as few channels have a policy against that)08:16
*** dpawlik6 has quit IRC08:18
ykarelsshnaidm|ruck, soniya29|rover, ^^ is duplicate of https://bugs.launchpad.net/tripleo/+bug/188437108:22
openstackLaunchpad bug 1884371 in tripleo "periodic master - queens jobs using docker.io in container prep." [Critical,In progress] - Assigned to wes hayutin (weshayutin)08:22
ykareland is caused by tripleo-ci change08:22
ykarelhttps://review.opendev.org/#/c/729824/08:23
*** ykarel is now known as ykarel|lunch08:25
sshnaidm|ruckykarel|lunch, ack08:31
*** bhagyashris|lunc is now known as bhagyashris08:34
*** dpawlik6 has joined #oooq08:36
*** ykarel|lunch is now known as ykarel08:58
soniya29|roverykarel, but it is for queens job so it is applicable for stein as well08:59
soniya29|roversshnaidm, ^^ ?09:00
ykarelsoniya29|rover, it's common for releases09:00
sshnaidm|rucksoniya29|rover, the bug means from queens to master09:04
*** jmasud has joined #oooq09:06
soniya29|roverykarel, sshnaidm, okay, I wasn't aware it is common for all releases.09:07
*** ykarel is now known as ykarel|mtg09:07
chandankumarmarios, sshnaidm|ruck, zbr please have a look at these patches https://review.opendev.org/#/q/topic:mirror-centos+(status:open+OR+status:merged) thanks!09:13
soniya29|roversshnaidm|ruck, same with fs-30 stein09:19
sshnaidm|rucksoniya29|rover, same - what do you mean?09:20
soniya29|roverfs-30 stein job also facing the same issue as discussed above, fails due to same issue09:21
sshnaidm|rucksoniya29|rover, it's for all periodic jobs09:23
soniya29|roversshnaidm|ruck, okay09:25
marioschandankumar: adding to queue09:30
*** apetrich has joined #oooq09:31
sshnaidm|ruckis today some holiday?09:32
mariossshnaidm|ruck: don't think so but 11:16 < zbr> note: today me and panda are out (not changing nick as few channels have a policy against that)09:36
sshnaidm|ruckmarios, I suppose it's just a monday09:37
*** ykarel|mtg is now known as ykarel09:42
arxcruzzbr: any idea https://957fa00c03fc9b0ae44c-39a519f1e914655bdb6fac4a25e72862.ssl.cf5.rackcdn.com/720434/19/check/openstack-tempest-skiplist-validate/40198a0/job-output.txt ?10:01
zbrarxcruz: read https://stackoverflow.com/questions/7446187/no-module-named-pkg-resources -- pkg_resources is part of setuptools10:08
zbrthat is usually added whent he venv is created, but if if you have a too old python, you may have a version that does not have it.10:09
zbrprobably you are using py27 instead of py3610:09
arxcruzzbr: hmmmm i'll take a look10:09
arxcruzit was working a few days ago10:09
chandankumarmarios, ysandeep|away can we meet sometime today to divide the train work? or want to do during scrum?10:13
marioschandankumar: half hour before scrum ? we can use scrum line10:13
chandankumarmarios, yes works from me, sending an invite10:13
marioschandankumar: ack10:13
*** jmasud has quit IRC10:22
*** jfrancoa has quit IRC10:31
*** jfrancoa has joined #oooq10:34
*** derekh has joined #oooq11:08
*** pojadhav is now known as pojadhav|brb11:17
*** jpena is now known as jpena|lunch11:29
*** rlandy has joined #oooq11:49
*** rlandy is now known as rlandy|ruck11:50
*** soniya29|rover is now known as soniya29|rover|a11:52
*** soniya29|rover|a is now known as soniya29|afk11:52
rlandy|rucksshnaidm|ruck: soniya29|afk: hi - how's it going today?11:53
rlandy|rucksshnaidm|ruck: are you going to attend escalation meeting or should I do that?11:53
sshnaidm|ruckrlandy|ruck, I'll join the meeting11:54
chandankumarrlandy|ruck, you are looking at upstream containers build task https://tree.taiga.io/project/tripleo-ci-board/us/1821?milestone=267741 ?11:54
sshnaidm|ruckrlandy|ruck, going pretty well, all periodic are broken, so everything is calm11:54
rlandy|rucksshnaidm|ruck: we updated the bmc image on rdocloud as you requested11:55
rlandy|ruckbut master is failing deploy11:55
rlandy|ruckchandankumar: I can do - I don;t know if anyone else has picked it up?11:55
chandankumarrlandy|ruck, right now not yet, weshay_pto is on pto11:56
rlandy|ruckchandankumar: depends how ruck/rover goes11:56
chandankumarrlandy|ruck, I will take a look11:56
rlandy|ruckchandankumar: will join the scrum to discuss11:56
sshnaidm|ruckrlandy|ruck, yeah, submitted for that https://bugs.launchpad.net/tripleo/+bug/188451811:58
openstackLaunchpad bug 1884518 in tripleo "OVB metalsmith deployment fails: Failed to attach VIF ... to bare metal node, Node ... is locked by host undercloud" [Critical,Triaged]11:58
rlandy|rucksshnaidm|ruck: ok - let me know what you need me to pick up and when you are eod12:00
rlandy|rucksshnaidm|ruck: is queens promotion also blocked by container settings?12:01
rlandy|ruckif not, can look at getting that through12:01
*** amoralej is now known as amoralej|lunch12:05
*** rfolco has joined #oooq12:05
sshnaidm|ruckrlandy|ruck, according to bug all of them12:07
chandankumarsshnaidm|ruck, please have a look at this review https://review.opendev.org/#/c/737179/ when free12:11
*** pojadhav|brb is now known as pojadhav12:12
rlandy|ruckmarios: ysandeep|away: looking at the downstream container pull issue12:16
rlandy|rucklooks like we are trying to pull containers from the mirror12:16
*** jbadiapa has joined #oooq12:19
mariosrlandy|ruck: ack - not been looking there today, let me know ifyou need reviews/etc12:20
rlandy|ruckmarios: I just have no idea why it's pulling containers from the mirror12:20
rlandy|rucktrying weshay_pto's patch12:21
*** ratailor has quit IRC12:21
rlandy|ruckhttps://review.opendev.org/#/c/737132/12:21
*** udesale_ has joined #oooq12:25
rlandy|ruckDockerRegistryMirror]12:26
*** udesale has quit IRC12:27
*** skramaja has joined #oooq12:28
sshnaidm|ruckrlandy|ruck, do you know what is about ipa server failure in escalation board?12:29
rlandy|rucksshnaidm|ruck: I updated that12:29
rlandy|ruckthere is a patch I am spinning on12:30
rlandy|ruckthe issue is the auto-reverse option12:30
rlandy|ruckif we get clean passes, I12:30
rlandy|ruckll merge that12:30
sshnaidm|ruckrlandy|ruck, cool12:30
rlandy|ruckyou can leave the card a few days - I'll update it12:30
chandankumarysandeep|away, c8 meeting joining12:30
*** jpena|lunch is now known as jpena12:32
*** marios is now known as marios|call12:32
ysandeep|awaychandankumar, sry i was on planned pto today, joining to get updates12:36
chandankumarmarios|call, https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/component-jobs.yaml#L21912:36
soniya29|afksshnaidm|ruck, yeah, its good :)12:37
*** soniya29|afk is now known as soniya2912:37
*** soniya29 is now known as soniya29|rover12:37
*** ysandeep|away is now known as ysandeep|PTO12:38
chandankumarmarios|call, https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-train-component-baremetal-promote-consistent-to-component-ci-testing/30ef5f9/job-output.txt12:43
chandankumarmarios|call, https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-train/c2ef46c/job-output.txt12:44
chandankumarmarios|call, https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-train/c2ef46c/job-output.txt12:44
marios|callhttp://mirror.regionone.rdo-cloud-tripleo.rdoproject.org:8080/rdo/centos8-train/current/delorean.repo.md512:45
*** jfrancoa has quit IRC12:46
sshnaidm|ruckrlandy|ruck, ipa has the same error: https://07b92d57b5f1a612a74b-4cebd80e32fd7e7586df4103c9478cce.ssl.cf5.rackcdn.com/737058/1/check/tripleo-ci-centos-8-standalone-on-multinode-ipa/c7d95cc/job-output.txt12:47
*** migi has joined #oooq12:47
*** jfrancoa has joined #oooq12:47
sshnaidm|ruckrlandy|ruck, seems like no such package in repos12:48
chandankumarpojadhav, https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/project-templates-components.yaml and https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/component-jobs.yaml12:48
rlandy|rucksshnaidm|ruck: so there were two problems with ipa12:50
rlandy|ruckthe package issue shows up later12:51
rlandy|ruckhttps://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-on-multinode-ipa12:51
rlandy|ruckjob results are mixed12:52
rlandy|ruckbut there is some problem with that package12:52
rlandy|rucksshnaidm|ruck: was chatting with ade about that package on friday - will check in with him again12:53
sshnaidm|ruckrlandy|ruck, ack12:53
rfolcoping scrum arxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar13:00
rfolcochandankumar, waiting for you13:02
rfolcochandankumar, run13:02
*** amoralej|lunch is now known as amoralej13:10
*** rlandy|ruck is now known as rlandy|ruck|mtg13:12
*** TrevorV has joined #oooq13:30
rlandy|ruck|mtgchandankumar: ping me when you want to chat about container builds13:31
*** rlandy|ruck|mtg is now known as rlandy|ruck13:31
chandankumarrlandy|ruck, yes sure13:31
chandankumarrlandy|ruck, after 15 mins,13:31
rlandy|ruckchandankumar: sure - no rush13:32
*** pojadhav is now known as pojadhav|brb13:33
*** marios|call is now known as marios13:34
rlandy|ruckmarios: ysandeep|PTO: re: container pull issue downstream ... testing now - but looking like https://review.opendev.org/#/c/737235/ will fix it13:37
mariosrlandy|ruck: thanks13:38
rlandy|rucksshnaidm|ruck: ok if I w+ https://review.opendev.org/#/c/737234/?13:46
chandankumarrlandy|ruck, https://meet.google.com/wyi-efgk-jux13:48
rlandy|ruckjoining13:49
*** rlandy|ruck is now known as rlandy|ruck|mtg13:49
sshnaidm|ruckrlandy|ruck|mtg, sure13:53
rlandy|ruck|mtgdone13:53
*** pojadhav|brb is now known as pojadhav13:56
rlandy|ruck|mtgchandankumar: https://code.engineering.redhat.com/gerrit/#/c/203269/13:58
rlandy|ruck|mtgchandankumar: http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/zuul.d/tripleo-push-containers.yaml13:58
*** rlandy|ruck|mtg is now known as rlandy|ruck14:01
chandankumarrlandy|ruck, thanks!14:02
rlandy|rucksshnaidm|ruck: W+ing https://review.opendev.org/#/c/737235/ as well14:05
sshnaidm|ruckrlandy|ruck, great14:06
soniya29|roverrlandy|ruck, tripleo-ci-centos-8-standalone-on-multinode-ipa job is failing in https://review.opendev.org/#/c/736521/14:10
rlandy|rucksoniya29|rover: yep - I have a LP and a patch for that ...14:13
rlandy|ruckhttps://review.opendev.org/#/c/73705814:14
soniya29|roverrlandy|ruck, sure14:14
*** pojadhav is now known as pojadhav|fk14:36
*** pojadhav|fk is now known as pojadhav|afk14:36
*** soniya29|rover is now known as soniya29|afk14:36
rlandy|rucksshnaidm|ruck: hey ...14:37
sshnaidm|ruckrlandy|ruck, hi14:37
rlandy|rucksshnaidm|ruck: I took the new bmc-template into downstream ...14:37
sshnaidm|ruckrlandy|ruck, how is it?14:37
* rlandy|ruck puts links in pvt14:37
rlandy|rucksshnaidm|ruck: the nodes says it's taking the new template14:38
rlandy|ruckhowever if you look at the log I pasted in pvt14:38
rlandy|ruckand search for pyghmi, how do you know what version you got?14:39
rlandy|ruckpip install pyghmi14:39
sshnaidm|ruckrlandy|ruck, hmm, apparently you can't from logs14:40
sshnaidm|ruckrlandy|ruck, but just "pip freeze | grep pyghmi" in image14:41
rlandy|rucksshnaidm|ruck: downstream still looks like it's suffering from the bmc issue14:41
sshnaidm|ruckrlandy|ruck, did you take one from vexx?14:41
rlandy|rucksshnaidm|ruck: rdocloud14:41
sshnaidm|ruckrlandy|ruck, where do you see this? In this logs it looks ifne14:41
sshnaidm|ruckrlandy|ruck, take one from vexx14:41
rlandy|rucksshnaidm|ruck: I don't - the nodes is still up14:41
sshnaidm|ruckrlandy|ruck, so, it seems there is no "struct" errors in bmc logs which were caused by old pyghmi14:43
rlandy|rucksshnaidm|ruck: yep - still looks like old one14:43
rlandy|rucksshnaidm|ruck: the one inrdocloud seems fine now14:43
* rlandy|ruck tries image again14:44
sshnaidm|ruckrlandy|ruck, sorry, I don't follow14:44
rlandy|rucksshnaidm|ruck: will get and upload image again14:44
*** ykarel is now known as ykarel|away14:46
*** saneax has quit IRC14:52
*** saneax has joined #oooq14:53
*** chem is now known as chem|reboot14:53
*** saneax has quit IRC14:54
*** saneax has joined #oooq14:55
*** sanjayu_ has joined #oooq14:57
*** saneax has quit IRC15:00
derekh bash15:13
*** chem has joined #oooq15:21
*** skramaja has quit IRC15:34
chandankumararxcruz, Hello15:59
chandankumararxcruz, you are hitting this issue Unable to find any of pip3 to use.  pip needs to be installed.?15:59
chandankumaror something else15:59
chandankumarakahat, commented on the review16:02
sshnaidm|ruckrlandy|ruck, how is it going with a new image?16:07
rlandy|rucksshnaidm|ruck: we haven't gotten to introspection yet16:09
rlandy|rucksshnaidm|ruck: can confirm that the ironic conductor container should have new pyghmi though16:09
rfolcochandankumar, I think* the problem is that your don't have python3-setuptools installed, I am out of context so ignore if this is non-sense.16:09
rlandy|ruckad did the old run though ... https://sf.hosted.upshift.rdu2.redhat.com/logs/36/200436/22/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-baremetal-rhos-17/bb4beb6/logs/undercloud/var/log/extra/podman/containers/ironic_conductor/podman_info.log.txt.gz16:10
chandankumarrfolco, I am waiting for arx to see the issue16:10
*** dtantsur is now known as dtantsur|afk16:10
rfolcochandankumar, k16:10
*** sshnaidm|ruck is now known as sshnaidm|afk16:13
*** Goneri has joined #oooq16:14
*** udesale_ has quit IRC16:21
sshnaidm|afkrlandy|ruck, did you see such error in ipa job? happened in gates now: https://027cb57af33b4dad80bb-9476297ff63d6e8659419c3d29e9b66c.ssl.cf1.rackcdn.com/735815/1/gate/tripleo-ci-centos-8-standalone-on-multinode-ipa/a717cc9/job-output.txt16:23
rlandy|rucksshnaidm|afk: yep - there are two bugs open16:24
rlandy|ruckthe fix to ^^ is in https://review.opendev.org/#/c/737058/16:24
rlandy|rucksshnaidm|afk: ^^ getting that fix through is blocked by https://bugs.launchpad.net/tripleo/+bug/188457016:25
openstackLaunchpad bug 1884570 in tripleo "tripleo-ci-centos-8-standalone-on-multinode-ipa is failing sporadically on 'install tls dependencies'" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)16:25
rlandy|rucksshnaidm|afk: ade_lee is out - pinged hrybacki on that16:25
*** marios has quit IRC16:26
sshnaidm|afkrlandy|ruck, I see, thought it's something new if in gates16:30
rlandy|rucksshnaidm|afk: nah - been chasing it for a few days16:30
rlandy|rucksshnaidm|afk: still around?16:42
sshnaidm|afkrlandy|ruck, ya16:42
rlandy|rucksshnaidm|afk: introspection is running now16:43
rlandy|ruckcan you log on to 10.0.110.6 and take a look?16:43
sshnaidm|afkrlandy|ruck, yep16:43
rlandy|rucksshnaidm|afk: note nothing much gets reported in overcloud_prep_images_introspection_debug.pcap.log16:43
rlandy|ruckmaybe blocked?16:44
sshnaidm|afkrlandy|ruck, or maybe debug is false16:44
rlandy|rucksshnaidm|afk: are iptables rules correct?16:47
rlandy|ruckmaybe 67/68/69 blocked?16:47
rlandy|ruckports16:47
*** amoralej is now known as amoralej|off16:47
rlandy|ruckor we could turn off iptables16:48
sshnaidm|afkrlandy|ruck, I don't see it blocked16:49
sshnaidm|afkrlandy|ruck, and what should be a problem?16:49
rlandy|rucksshnaidm|afk: idk - but intospection is just hanging there16:50
rlandy|ruckthe bcm console fef054b6-f7d1-4c64-895a-914fbbb8e399/console looks ok though16:51
*** jmasud has joined #oooq16:52
rlandy|rucksshnaidm|afk: basically iiuc, the nodes should power off at this point16:53
sshnaidm|afkrlandy|ruck, ok, will take a look later a bit, gotta run16:55
rlandy|rucksshnaidm|afk: k - will see if we can hold this node16:55
*** jtomasek has quit IRC16:58
*** jtomasek has joined #oooq17:01
*** derekh has quit IRC17:05
*** jmasud has quit IRC17:15
*** jpena is now known as jpena|off17:23
*** jmasud has joined #oooq17:24
*** aakarsh has joined #oooq17:29
*** rlandy|ruck is now known as rlandy|ruck|mtg17:59
*** chem has quit IRC18:05
*** chem has joined #oooq18:06
sshnaidm|afkrlandy|ruck|mtg, maybe worth to update ipxe-boot image too from rdo-cloud/18:13
sshnaidm|afk?18:13
*** rlandy|ruck|mtg is now known as rlandy|ruck18:26
rlandy|rucksshnaidm|afk: I can try that18:26
rlandy|ruckI think it's the same image18:26
rlandy|ruckI updated it a couple weeks ago18:26
*** jtomasek has quit IRC18:35
*** jmasud has quit IRC19:16
*** jmasud has joined #oooq19:18
*** sanjayu_ has quit IRC19:56
*** jfrancoa has quit IRC20:10
*** jmasud has quit IRC20:26
*** rfolco has quit IRC20:59
*** jmasud has joined #oooq20:59
*** jbadiapa has quit IRC21:05
*** rascasoft has joined #oooq21:47
*** TrevorV has quit IRC22:16
*** jmasud has quit IRC22:19
*** jmasud has joined #oooq22:40
*** tosky has quit IRC22:59
*** rlandy|ruck is now known as rlandy|ruck|bbl23:05
*** rfolco has joined #oooq23:18
*** rascasoft has quit IRC23:30
*** jmasud has quit IRC23:43
*** jmasud has joined #oooq23:55

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!