Friday, 2021-07-16

« Thursday, 2021-07-15 Index Saturday, 2021-07-17 »
*** rlandy|bbl is now known as rlandy01:49
*** ykarel|away is now known as ykarel04:31
sshnaidm|afkzbr, fyi https://bugs.launchpad.net/tripleo/+bug/193645905:00
*** chandankumar is now known as chkumar|rover05:21
chkumar|roverbhagyashris_: https://bugs.launchpad.net/tripleo/+bug/193646506:24
*** amoralej|away is now known as amoralej07:02
zbrsshnaidm|afk: from its description it has nothing to do with molecule itself07:35
*** dmellado_ is now known as dmellado07:41
amoralejchkumar|rover, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34419 that's moving phase2 jobs from ci.centos.org to new jenkins server08:01
bhagyashris_marios, fyi i have reported bug https://bugs.launchpad.net/tripleo/+bug/193647608:01
amoralejyou see any issue from your side?08:02
bhagyashris_chkumar|rover, ^08:02
*** bhagyashris_ is now known as bhagyashris|ruck08:24
chkumar|roveramoralej: checking08:32
chkumar|roveramoralej: done08:39
chkumar|roverchkumar|rover: I will take care of cockpit changes08:39
amoralejthanks chkumar|rover 08:40
zbrmarios: dviroel|out: shouldn't we better avoid repeating tripleo for the submodule name?  triple_repos.get_hash or tripleo_repos.yum_config seem more logic to me than triple_repos.tripleo_get_hash ....08:58
marioszbr: yeah it makes sense08:58
marioszbr: will we sync before this afternoon call? 08:59
zbrif you have few minutes it may prove wise, i kinda want to share my thoughs with you.08:59
marioszbr: yes09:00
marioszbr: we said we would talk 09:00
zbrbjs-zfgr-qwb09:01
marioszbr: plugging headphones09:01
*** ykarel is now known as ykarel|lunch09:06
marios34567 :) what do i win https://review.rdoproject.org/r/c/testproject/+/34567 10:08
zbrmarios: there is a problem with get_hash assumption that it can install the the config file, which prevents use of module unless is installed as root10:51
zbri think we should havea a fallback to embedded config file, so it would load the config from within the module when system one is missing10:51
marioszbr: k we can discuss that so you mean the config file used by get-hash referenced in data_files of setup.cfg10:54
zbryeah, i get some test failures, and i think i spot an error inside _resolve_local_config_path which was supposed to work.10:54
zbri just discovered that we used tripleo-repos was using python3.8 when running `tox-py39` job, due to a simple `basepython=python3` inside tox.ini, tox warning nobody acted on.11:02
*** ykarel|lunch is now known as ykarel11:11
*** dviroel|out is now known as dviroel11:19
akahatchkumar|rover, bhagyashris|ruck hey master promotion got failed. ppc image is missing. containers got updated but dlrn and qcow images are not. I'm trying to promote them manually now. latest hash: 3313dbb8684c4b1a22827a94f756519311:51
akahatI'll turn off master promotion service for some time.11:51
akahatlogs: http://10.0.148.74/promoter_logs/centos8_master_2021-07-16T11:07.log11:52
akahatmaster promoted: http://10.0.148.74/promoter_logs/centos8_master_2021-07-16T11:54.log11:57
bhagyashris|ruckakahat, do you have conatianers promotion logs12:02
*** amoralej is now known as amoralej|lunch12:05
akahatbhagyashris|ruck, http://10.0.148.74/promoter_logs/container-push/20210716-101216.log12:05
bhagyashris|ruckgot it12:05
bhagyashris|ruckit's not taking exclude list12:06
bhagyashris|ruckakahat, here we have exclude ppc containers list https://review.opendev.org/c/openstack/tripleo-ci/+/792782/2/roles/build-containers/vars/main.yaml#1512:11
bhagyashris|ruckwhich contains ironic-pxe container12:11
akahatbhagyashris|ruck, okay. some python side code issues.12:15
bhagyashris|ruckakahat, i got the issue12:19
bhagyashris|ruckit's failing here https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33815/41/ci-scripts/container-push/roles/containers-promote/tasks/check_set_ppc_manifests.yml last task12:20
bhagyashris|ruckdue to this change it's failing https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33815/35..41/ci-scripts/container-push/roles/containers-promote/tasks/check_set_ppc_manifests.yml12:23
bhagyashris|ruckother wise left hand side change was working 12:23
bhagyashris|ruckakahat, ^12:23
bhagyashris|ruckakahat, hey you around ?12:30
akahatbhagyashris|ruck, yes.12:33
bhagyashris|ruckakahat, fyi testing here https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34570 12:34
akahatlooking12:35
bhagyashris|rucki have reverted back your change that you made in the playbook https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33815/38..39/ci-scripts/container-push/roles/containers-promote/tasks/check_set_ppc_manifests.yml12:35
bhagyashris|ruckthis change https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33815/38..39/ci-scripts/container-push/roles/containers-promote/tasks/check_set_ppc_manifests.yml broke the ppc container manifest push 12:36
*** marios_ is now known as marios12:40
bhagyashris|rucklooks like this https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33815/39/ci-scripts/container-push/roles/containers-promote/tasks/check_set_ppc_manifests.yml#42 will randomly pull the containers image the same way we were doing before 12:41
jpodivinweshay|ruck: hi. Do you have a moment. I would like to ask about potential fix for the https://bugs.launchpad.net/tripleo/+bug/1936218 . I'm not really sure if I got it right. 12:42
bhagyashris|ruckakahat, quick call?12:43
jpodivinPatching in the oooq-extras12:43
weshay|ruckjpodivin, I don't have for a few hours.. no meetings on friday.. lolz12:45
weshay|ruckI wish12:45
weshay|ruckchkumar|rover, can you help jpodivin 12:45
chkumar|roverweshay|ruck: checking12:49
jpodivingreat12:49
jpodivinchkumar|rover: we can talk it over on irc or I can start up mee.t12:49
chkumar|roverI am reading the bug12:50
jpodivinchkumar|rover: yep12:50
chkumar|roverjpodivin: so based on the logs, bug and patch, tripleo-ansible-inventory.yaml is missing?12:54
jpodivinnot exactly12:55
jpodivinIt was never supposed to be there in a first place :D12:55
jpodivinBut yeah, it's now in a subdirectory12:55
jpodivinthe patch I've submitted, it's in the LP, is reconfiguring our jobs to look for it there 12:55
jpodivinAt least it should. As far as I can tell. 12:56
chkumar|roverhttps://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/80102612:56
chkumar|roverlet me do one thing, 12:56
jpodivinsure12:56
chkumar|roverlet me fire a testproject patch with above patch and test your change and get the node hold12:56
chkumar|roverso that you can directly debug what is going on there12:57
jpodivinchkumar|rover: that would be great, thank you. 12:57
jpodivinI'm personally a bit concerned about the failure on tripleo-ci-centos-8-scenario007-multinode-oooq-container. It's the only one that had problem during the check, and to make things worse it was an issue durring overcloud deploy. 12:58
jpodivinNot sure if my patch is to blame or.12:58
jpodivinhttps://d99c47d463700dbafce6-6ac4a54b3a9261cc3d1dc9362057083e.ssl.cf2.rackcdn.com/801026/1/check/tripleo-ci-centos-8-scenario007-multinode-oooq-container/da8b0e0/logs/undercloud/home/zuul/overcloud_deploy.log12:58
chkumar|roverjpodivin: can you share your pub key?13:03
bhagyashris|ruckakahat, replied your comments https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3457013:03
bhagyashris|ruckand zuul is +1 i think we should merge this patch https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3457013:04
jpodivinchkumar|rover: sure. a moment13:05
rlandymarios: hi .. I rekicked the downstream testproject check on tripleo-get-hash13:05
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/95/200295/123/check/periodic-tripleo-ci-rhel-8-standalone-rhos-17/2dff100/job-output.txt13:05
rlandyjust fyi ^^ debugging this before I comment on patch13:05
rlandyunless you know off hand what itis13:05
chkumar|roverykarel: can you hold the node for this review https://review.rdoproject.org/r/c/testproject/+/34571 and add pub key https://github.com/raukadah.keys 13:08
chkumar|rover?13:08
chkumar|roverjpodivin: testproject review https://review.rdoproject.org/r/c/testproject/+/3457113:08
ykarelchkumar|rover, ack doing13:09
jpodivinchkumar|rover: thank you13:09
mariosrlandy: in call rigth now will check in bit? 13:10
*** marios is now known as marios|call13:10
rlandyno rush13:10
ykarelchkumar|rover, ssh zuul@38.102.83.3613:11
* bhagyashris|ruck back in some time13:13
*** amoralej|lunch is now known as amoralej13:13
akahatbhagyashris|ruck, hey.. it was my dog.. sorry.. yes we can have quick call.13:25
*** marios|call is now known as marios13:34
mariosrlandy: no doesn't look familiar/didn't come across that with the other testing for get hash (2021-07-16 08:50:46.158644 | primary | TASK [standalone : Parse container params] ************************************* 13:35
rlandymarios: probably something with the way we use internal settings13:35
rlandyleft some notes on the patch13:36
rlandyrelated to tag13:36
rlandylooking through tripleo-environments13:36
mariosrlandy: looks like it used the module though "Get DLRN hash tripleo-ci-testing using tripleo_get_hash ok: [undercloud]13:36
rlandywill post the rest when I find it13:36
mariosrlandy: thank you13:36
mariosrlandy: will have a closer look ... have put a workflow -1 on the wireup patch for now anyway to prevent accidental merge13:37
weshay|ruckmarios, chat?13:44
weshay|ruckre: release gov13:44
mariosweshay|ruck: sure 13:47
weshay|ruckmarios, https://meet.google.com/npq-brtx-xfn13:48
chkumar|roverykarel: thanks!13:49
bhagyashris|ruckakahat, i am back14:16
chkumar|roverjpodivin: Hello, https://review.rdoproject.org/r/c/testproject/+/34571 and with this patch https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/80102614:22
chkumar|roverperiodic-tripleo-ci-centos-8-standalone-network-master-validation SUCCESS 1h 17m 32s14:22
chkumar|roverhttps://logserver.rdoproject.org/71/34571/1/check/periodic-tripleo-ci-centos-8-standalone-network-master-validation/57c1bf6/logs/undercloud/home/zuul/tripleo-deploy/tripleo-ansible-inventory.yaml.txt.gz14:22
jpodivinJust saw it. 14:22
jpodivinI suppose that this means we have only one problem left with the patch https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/80102614:23
chkumar|rovernow checking tripleo-ci-centos-8-scenario007-multinode-oooq-container job failure14:23
jpodivinchkumar|rover: exactly, I tried to figure out what went wrong.14:24
jpodivinchkumar|rover: but apart from it having something to do with overcloud-deploy...14:24
jpodivinchkumar|rover: the final part of the overcloud-deploy.log with the stack trace http://pastebin.test.redhat.com/98010614:27
jpodivinthe deploy-steps playbook experienced some sort of issue. 14:28
chkumar|roverrlandy: weshay|ruck during overcloud deploy have you seen this msg undercloud still blocked14:29
rlandynot that I recall14:29
chkumar|roverrlandy: weshay|ruck https://d99c47d463700dbafce6-6ac4a54b3a9261cc3d1dc9362057083e.ssl.cf2.rackcdn.com/801026/1/check/tripleo-ci-centos-8-scenario007-multinode-oooq-container/da8b0e0/logs/undercloud/home/zuul/overcloud-deploy/overcloud/config-download/stdout - 14:29
chkumar|roverCheck Keystone user status | undercloud | error={"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}14:29
chkumar|roverjpodivin: ^^14:29
chkumar|roverjpodivin: due to no_log we are not sure what went wrong there14:30
chkumar|roverLet me do a recheck14:30
jpodivinchkumar|rover: so we are all on the same page then :D14:30
chkumar|roveryes14:32
jpodivindeploy_steps_playbook.yaml just imports other playbooks.14:32
jpodivinone of those is deploy_steps_network_validation_playbook.yaml14:32
mariosrlandy: weshay|ruck: when you get a sec please https://review.rdoproject.org/r/c/rdo-jobs/+/34569 (i missed it in weshay review https://review.rdoproject.org/r/c/rdo-jobs/+/34498/3#message-a34391f538833a31af7038b40f9087d33112a91b )14:36
rlandychkumar|rover: haven't seen that14:37
rlandyrunning 17 now - don't see that issue14:37
chkumar|roverrlandy: ok, not sure what is going wrong there14:38
chkumar|rovermay be weshay|ruck have some insight14:38
chkumar|roverweshay|ruck: rlandy please have a look at this patch https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34419 when free, thansk!14:43
bhagyashris|ruckakahat, can you plz join14:45
chkumar|roverrlandy: weshay|ruck logging out now, keep an eye on this review https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/800965 and https://review.opendev.org/c/openstack/python-tripleoclient/+/801019 14:45
chkumar|roverrest are all good14:45
akahatbhagyashris|ruck, yes. 14:45
rlandychkumar|rover: ack - have a good weekend14:45
chkumar|roverrlandy: you too thanks :-)14:45
*** ykarel is now known as ykarel|away14:46
rlandyweshay|ruck: promoting 17 ... http://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel8-osp17/api/civotes_agg_detail.html?ref_hash=8d844d87e22605da8a7cb8899a60310915:10
mariosrlandy: when you get a chance fyi i replied to your comment there https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/793331/14#message-d900d179e29b78a2855b3cdedf15907770686911 (& sanity check @ https://review.rdoproject.org/r/c/testproject/+/34567)15:14
rlandymarios: sure - just promoting 17 atm 15:15
mariosrlandy: yeah no rush just mentioned so i don't forget 15:15
mariosrlandy: obviously waiting on the d/stream testing results from the other patch first anyway 15:16
zbrdviroel: marios: https://review.opendev.org/c/openstack/tripleo-repos/+/801029/ is ready for review, i addressed all reported bits and zuul will give green on it in few minutes, already passed most jobs.15:18
marioszbr: ack next week though at this point noting for monday reviews 15:19
dviroelzbr: tks will take a look soon15:20
* dviroel lunch15:23
rlandymarios: ok ... https://review.opendev.org/c/openstack/tripleo-quickstart/+/791486 - a bunch of comments on where the internal failure comes from15:35
rlandyecho "--extra-vars @$release_file -e dlrn_hash=$release_hash -e get_build_command=$release_hash ${newest_release_hash:+-e dlrn_hash_newest=$newest_release_hash}"15:35
rlandyhttps://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/oooq_common_functions.sh.j2#L16515:35
mariosrlandy: ack noted that on the review so i can revisit on monday 15:36
mariosrlandy: thank you for testing that and for the info 15:37
rlandymarios: wrt https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/793331 - ack on container builds15:37
rlandychecking image builds15:37
rlandywill comment15:37
mariosrlandy: ack thanks15:37
rlandyotherwise +115:37
rlandymarios: wrt 17 failure - let's discuss on monday if downstream still needs that workflow15:38
rlandyit may not15:38
mariosrlandy: k 15:38
mariosweshay|ruck: o/ fyi had a go through it and reworked a bit https://docs.google.com/document/d/11FsXWfkX-CpLbdPEm9hVS2gHHWBBsFjxg6dCt7W1xg4/edit?disco=AAAAMnXLlEg 16:04
mariosweshay|ruck: will check and send out on monday 16:04
*** marios is now known as marios|out16:35
*** amoralej is now known as amoralej|off16:45
rlandylunch - biab16:45
weshay|ruckrlandy, can sync anytime17:17
rlandyweshay|ruck: now ok?17:21
weshay|ruckrlandy, aye17:21
weshay|ruckrlandy, meet.google.com/sks-kbgm-zow17:22
rlandyweshay|ruck: https://review.rdoproject.org/r/c/testproject/+/3116517:30
weshay|ruckrlandy, gri project-merged  --project_name tripleo --age 117:55
rlandyweshay|ruck: ok - got ruck_rover.py running19:37
weshay|ruckrlandy, :)20:15
weshay|rucksorry.. my basement flooded a little last night.. was cleaning20:15
rlandyoh gosh20:15
rlandyjust before your vaca20:15
weshay|ruckya20:15
weshay|ruckfun20:15
rlandyweshay|ruck: can we merge https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/80096520:26
rlandyoh wait rabi responded20:26
rlandyor we can merge that and I change another review20:27
rlandysince it will spring train free20:27
rlandyasync task did not complete within the requested time - 5700s - scenario00020:51
rlandyweshay|ruck: promoting 16.2 http://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel8-osp16-2/api/civotes_agg_detail.html?ref_hash=5af1f4f3d7b3ae52bd0c268481025fac21:58
rlandyBM and fs020 passed21:58
weshay|ruckwoot21:58
weshay|rucknice21:58
rlandyweshay|ruck: logging off - have a good vacation22:01
rlandydon't worry about us :)22:01
« Thursday, 2021-07-15 Index Saturday, 2021-07-17 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!