Wednesday, 2020-07-15

*** ccamacho has quit IRC00:01
*** jbadiapa has quit IRC00:29
*** saneax has quit IRC00:42
*** rlandy|ruck has quit IRC00:46
*** jmasud has quit IRC01:15
*** Tengu has quit IRC01:31
*** Tengu has joined #oooq01:33
*** Goneri has quit IRC01:43
*** apetrich has quit IRC02:14
*** rfolco has quit IRC03:14
*** jmasud has joined #oooq03:52
*** rfolco has joined #oooq03:52
*** rfolco has quit IRC03:56
*** skramaja has joined #oooq04:19
*** ysandeep|away is now known as ysandeep|rover04:32
*** ykarel has joined #oooq04:34
*** ratailor has joined #oooq04:47
*** udesale has joined #oooq05:22
*** ratailor has quit IRC05:27
*** ratailor has joined #oooq05:27
*** marios has joined #oooq05:31
*** saneax has joined #oooq05:50
zbrchandankumar: glad to see that you did not ask about pylint stuff, it will clearly help us write better code05:59
*** udesale has quit IRC06:07
chandankumarzbr:  :-)06:07
zbrfor the first week or so, it involves lot of swearing, but after that one (slowly) learns to enjoy it06:10
*** udesale has joined #oooq06:20
zbrchandankumar: another one https://review.rdoproject.org/r/#/c/28535/06:40
ysandeep|roverzbr|ruck, Stein periodic jobs https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable3  didn't trigger because periodic-tripleo-centos-7-stein-containers-build-push failed with NODE_FAILURE06:41
ysandeep|roverzbr|ruck,  and good morning :)06:42
zbrmorning!06:42
ysandeep|roverhmmm.. Next run of stein is on weekend, i am wondering if i should run a test project to get stein promotion, we are not doing great on promotion apart from ussuri06:45
akahatchandankumar, arxcruz cgoncalves if everything looks good then we can merge this: https://review.opendev.org/#/c/731501/06:59
*** zbr|ruck has quit IRC07:02
*** dsneddon has quit IRC07:02
*** soniya29 has joined #oooq07:02
*** zbr|ruck has joined #oooq07:03
*** holser has joined #oooq07:04
*** zbr has quit IRC07:06
ysandeep|roverzbr|ruck, fyi.. i have posted https://review.rdoproject.org/r/#/c/28537/ testproject to rerun stein pipeline inorder to get promotion.07:17
*** amoralej|off is now known as amoralej07:30
*** zbr has joined #oooq07:41
*** matbu has quit IRC07:45
*** dtantsur|afk is now known as dtantsur07:45
*** tosky has joined #oooq07:46
*** udesale has quit IRC07:47
arxcruzakahat: from my side it's okay, depending now on tosky, kopecmartin and chandankumar07:47
*** udesale has joined #oooq07:47
*** matbu has joined #oooq07:48
toskyarxcruz, akahat : and cgoncalves, who raised some issues, so I wouldn't merge without his approval07:50
arxcruztosky: you joined the channel before the message from akahat he's already pinging cgoncalves as well :)07:53
chandankumarzbr: marios please retake a look on this https://review.rdoproject.org/r/#/c/28521/ thanks!08:02
zbrchandankumar: marios: https://review.rdoproject.org/r/#/c/28454/ please, quite simple.08:04
marioschandankumar: zbr: will do but not right now just done a bunch of reviews need to do other stuff08:05
zbras get_hash was never tested on c7, we can as well remove the c7 if you prefer.08:05
zbrno worry, for this one we do not need two reviewers, is not touching prod code.08:06
holsermarios is reproducer working?08:15
mariosholser: should be but haven't run it in a long time it had some issues... sshnaidm is the most recent to use it. as far as i know it should be in good shape now08:19
mariosholser: are you having some issue?08:19
holserI am having issues with https://review.opendev.org/#/c/730325/11/tripleoclient/v1/overcloud_upgrade.py08:20
holserit's simple code and locally it works08:21
holserin the job "openstack overcloud update prepare --help" doesn't work08:22
holserso PREPARE_ANSWER is empty08:23
holserso it fails on "openstack overcloud update prepare" as it's waiting for "--yes" which is PREPARE_ANSWER08:24
mariosholser: yeah i recall.. i don't know ... you are hoping to run the update job locally then?08:24
holseryeah08:24
holserand cannot run reproducer ... it fails all the time with dlrn_hashes08:25
mariosholser: fwiw i have some similar case where the tests are green locally but failing on py27 jobs there https://review.opendev.org/#/c/740019/08:25
mariosholser: again really simple code08:25
mariosholser: so08:25
mariosholser: what is the error you have?08:25
holserhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_55c/730325/11/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/55cb03f/logs/undercloud/home/zuul/overcloud_update_prepare.log08:26
*** bogdando has joined #oooq08:26
holsermarios look at 2020-07-14 00:23:09 | START with options: overcloud update prepare --templates08:26
mariosholser: no that is the --yes error08:27
mariosholser: i mean wrt reproducer 11:25 < holser> and cannot run reproducer ... it fails all the time with dlrn_hashes08:27
bogdandomarios: hi, so we have "Failed to parse dlrn hash"08:28
bogdandowith a fresh reproducer created for a job08:28
mariosbogdando: sounds like an issue with get_hash maybe perhaps there is a new bug there :/08:28
bogdandolikely08:29
mariossshnaidm: are you aware of somethign with reproducer re 11:28 < bogdando> marios: hi, so we have "Failed to parse dlrn hash"08:29
mariosbogdando: otherwise i think your best bet is to file it so we can see some trace and try to understand & fix it08:29
mariosbogdando: i think you had success relatively lately with reproducer right? it must be a recent issue then08:29
bogdandowe deploy with mirror_path: mirror.regionone.rdo-cloud.rdoproject.org08:30
mariosbogdando: like the task trace which is failing08:30
bogdandothis points to http://mirror.regionone.rdo-cloud.rdoproject.org:8080/rdo/centos7-master/08:30
bogdandobut apparently the further URI leads to 40408:30
bogdandothe hash is from this jog https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4b5/730325/11/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/4b55c9b/logs/reproducer-quickstart/reproducer-zuul-based-quickstart.tar08:31
bogdandoe961461fc3de7c29860ddacfcf697aaf08:32
bogdandomarios: not really recently, more a 3 month ago  :)08:33
bogdandothan*08:33
bogdandomarios, sshnaidm: ^^08:34
bogdandoI don't have more infor to submit for a bug, mind taking it over?08:34
mariosbogdando: it would be nice to have a trace08:34
mariosbogdando: of the error08:34
mariosbogdando: perhaps holser has one?08:34
bogdandoI'll check08:34
*** derekh has joined #oooq08:36
bogdandomarios, sshnaidm: https://bugs.launchpad.net/tripleo/+bug/188763708:39
openstackLaunchpad bug 1887637 in tripleo "CI reproducer fails to parse dlrn hash tag" [High,New]08:39
holsermarios concerning https://review.opendev.org/#/c/740019/608:39
holseryou need to fix tests08:39
bogdandoplease check the trace, anything else to add?08:39
holser2020-07-14 11:09:12.357779 | ubuntu-bionic | Expected: run_command(['sudo', 'dnf', 'upgrade', '-y', 'python2-tripleoclient', 'openstack-tripleo-common', 'openstack-tripleo-heat-templates', 'openstack-tripleo-validations', 'tripleo-ansible'], name='Update extra packages')08:40
holser25562020-07-14 11:09:12.357787 | ubuntu-bionic | Actual: run_command(['sudo', 'yum', 'upgrade', '-y', 'python2-tripleoclient', 'openstack-tripleo-common', 'openstack-tripleo-heat-templates', 'openstack-tripleo-validations', 'tripleo-ansible'], name='Update extra packages')08:40
holser25572020-07-14 11:09:12.357795 | ubuntu-bionic |08:40
holserhttps://zuul.opendev.org/t/openstack/build/8c585e3ab3354ca2b4e9931bc073ad01/log/job-output.txt#2555-255608:40
bogdandoI'll add quickstart log as well08:41
mariosbogdando: thank you08:41
mariosholser: but locally they are green that is my point08:41
bogdandoadded it08:41
mariosholser: because i switch from sys returning 2 or 3 and it works .. i don't know why it fails in gerrit08:42
mariosholser: its why it reminded me of your problem i.e. for you the code works fine but in the job it fails08:42
*** bhagyashris is now known as bhagyashris|lunc08:43
mariosholser: python -m unittest tripleoclient/tests/v1/undercloud/test_install_upgrade.py --> Ran 13 tests in 0.222s08:43
mariosholser: and i sanity checked by 'breaking' the test and rerunning and it failed fine08:43
mariosholser: so locally its ok08:43
bogdandomarios: where from pop usps  that 3371c543036a0f9f7b485489c490de4bd8cb52d1_f935e212 I have no idea :/08:44
holsermarios the problem is how tests are written08:44
holserinstead of mocking they are requiring dnf/yum08:44
holserand in your env - everything is ok08:44
mariosholser: yeah ack i will consider this when i iterate today thank you08:45
holseryw08:45
*** ccamacho has joined #oooq08:47
toskyakahat, arxcruz, chandankumar, kopecmartin: I added a comment a comment, there is a stylistic change I don't like and I should have reported before, but it's not a -1 and I'm not going to do anything else with that review08:48
arxcruztosky: i agree with you, akahat please change the " to ' and we will merge it08:52
bogdandosshnaidm, marios: The hash e961461fc3de7c29860ddacfcf697aaf written in the launcher playbook is faily one, the one calculated by quickstart (3371c543036a0f9f7b485489c490de4bd8cb52d1_f935e212) is OK. So that should be some issue with the former one stored in the reproducer tarball.08:55
*** ysandeep|rover is now known as ysandeep|lunch08:57
holsermarios http://pastebin.test.redhat.com/88438709:05
holsernew issue09:06
holsermarios nvm09:07
mariosholser: talking in rhos-upgrades09:09
mariosholser: where is the paste from ? reproducer?09:09
holsermarios reproducer installed centos709:10
holserand I need centos809:10
holsermarios if we test master https://review.opendev.org/#/c/730325/11 so reproducer should tell to use centos8 images at https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_55c/730325/11/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/55cb03f/logs/README-reproducer.html but it's nt09:12
*** apetrich has joined #oooq09:12
sshnaidmbogdando, can you please paste into bug how you run reproducer?09:14
*** ysandeep|lunch is now known as ysandeep09:17
*** soniya29 has quit IRC09:23
*** ysandeep is now known as ysandeep|rover09:24
*** jbadiapa has joined #oooq09:25
ysandeep|roverfolks c7 train promotion is failing - http://38.102.83.109/centos7_train.log  while creating manifests for "neutron-metadata-agent-ovn" - stderr_lines": ["unexpected end of JSON input", "unexpected end of JSON input"] , I am unable to reproduce in my local env - http://paste.openstack.org/show/795933/ , Just checking in case you seen similiar error in the past?09:29
holsersshnaidm where can I get centos8 qcow2 with python3 for reporducer09:33
sshnaidmholser, not sure I understand09:43
sshnaidmholser, bare centos8 qcow2 you can download from centos site09:44
sshnaidmholser, or you mean CI image..?09:47
holserCI image09:48
*** jbadiapa has quit IRC09:57
sshnaidmholser, you can download it from rdo cloud for example, id is dc21cb56-d4ca-4a16-9973-315e903f9a4310:02
holsersshnaidm ok10:02
holserthanks10:02
holsersshnaidm Can you explain https://paste.centos.org/view/471af96010:06
holserWhy do we need centos-release-openstack-queens for master joubs?10:06
holsersshnaidm Can we review https://review.rdoproject.org/r/#/c/27643/3/playbooks/prepare.yaml really quick?10:08
ykarelbogdando, looks like u using centos7 image in centos8 job10:08
ykarelwrt above failure10:09
holserykarel Could you be so kind to review https://review.rdoproject.org/r/#/c/27643/10:10
ykarelholser, looking10:10
holsersshnaidm ^^^10:10
ykarelholser, without it works for both older and new releases?10:11
holserit does10:11
ykareliirc it was likely added for openvswitch10:11
holserykarel queens has gone10:11
ykarelwhere?10:11
ykarelgone?10:11
holserI mean if we need centos-release-openstack10:11
ykarelcentos-release-openstack-train can be used for both c7 and c8, that's the only release supporting both10:12
holserwe need to put proper version centos-release-openstack-[queens|rocky|master]10:12
holserbut openvswitch works without it10:12
holserykarel it was introduced by https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/commit/665c73c63562d5d272470a62b6645ef7dbff14dd10:14
sshnaidmholser, let me see ci results10:15
holsersshnaidm thanks10:15
ykarelholser, hmm it was added for openvswitch initially https://review.opendev.org/#/c/580518/410:15
ykarelbut iirc it's being installed from master deps repo now a days, so should be fine to remvoe it10:17
*** jbadiapa has joined #oooq10:18
*** jbadiapa has quit IRC10:19
*** jbadiapa has joined #oooq10:19
holserykarel yep10:19
ykarelyes https://review.opendev.org/#/c/708430/10:20
ykarelfor 7 we still using queens repo but it should be fine10:21
sshnaidmbogdando, holser fyi, tar reproducer may not work out of box, I'd recommend to run it different way - to setup ci system and then to submit patch manually, we describe it in manual: https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/README.md#install10:23
*** sshnaidm is now known as sshnaidm|afk10:24
*** bhagyashris|lunc is now known as bhagyashris10:37
*** Tengu has quit IRC10:46
*** Tengu has joined #oooq10:48
pojadhavHi folks, I need some inputs on this bug https://bugs.launchpad.net/tripleo/+bug/188763310:59
openstackLaunchpad bug 1887633 in tripleo "periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-train job fails with pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] EHOSTUNREACH)")" [Undecided,Triaged]10:59
*** tosky has quit IRC11:05
*** tosky has joined #oooq11:10
bhagyashrismarios, hi have you ever faced this issue https://bugs.launchpad.net/tripleo/+bug/1876893 during upgrade/update in c8 train11:10
openstackLaunchpad bug 1876893 in tripleo "Error: error removing container - device or resource busy" [Critical,In progress] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)11:10
mariosbhagyashris: don't think i'v seen that one11:13
*** saneax has quit IRC11:13
bhagyashrismarios, ok11:14
*** saneax has joined #oooq11:14
ysandeep|rovertrying my luck again:-11:14
ysandeep|roverfolks c7 train promotion is failing - http://38.102.83.109/centos7_train.log  while creating manifests for "neutron-metadata-agent-ovn" - stderr_lines": ["unexpected end of JSON input", "unexpected end of JSON input"] , I am unable to reproduce in my local env - http://paste.openstack.org/show/795933/ , Just checking in case you seen similiar error in the past?11:14
ysandeep|roverzbr|ruck, marios chandankumar ^^11:14
mariosysandeep|rover: not seen... but if there is a problem pushing manifests we might just turn that off i don't think anyone is using it?11:16
mariosysandeep|rover: i thought it was already off11:16
chandankumarysandeep|rover: nope, manifest needs to be off there11:17
mariosysandeep|rover: wierd that that is th eonly one that fails though11:18
marioschanged: [localhost] => (item=neutron-server-ovn)11:18
mariosfailed: [localhost] (item=neutron-metadata-agent-ovn) => {"ansible_index_var": "index", "ansible_loop_var": "item", "changed": true, "cmd": "docker manifest create  docker.io/tripleotrain/centos-binary-neutron-metadata-agent-ovn:7d0406b1a2bb054f42b198e9494ddc54372e7285_6e7a0112_manifest11:18
mariosdocker.io/tripleotrain/centos-binary-neutron-metadata-agent-ovn:7d0406b1a2bb054f42b198e9494ddc54372e7285_6e7a0112_x86_64\ndocker manifest annotate --arch amd64 docker.io/tripleotrain/centos-binary-neutron-metadata-agent-ovn:7d0406b1a2bb054f42b198e9494ddc54372e7285_6e7a0112_manifest docker.io/tripleotrain/centos-binary-neutron-metadata-agent-ovn:7d0406b1a2bb054f42b198e9494ddc54372e7285_6e7a0112_x86_64\n",11:18
ysandeep|rovermarios, yes only 1 container is failing11:18
marios"delta": "0:00:00.126053", "end": "2020-07-15 10:23:11.218634", "index": 60, "item": "neutron-metadata-agent-ovn", "msg": "non-zero return code", "rc": 1, "start": "2020-07-15 10:23:11.092581", "stderr": "unexpected end of JSON input\nunexpected end of JSON input", "stderr_lines": ["unexpected end of JSON input", "unexpected end of JSON input"], "stdout": "", "stdout_lines": []}11:18
marioschanged: [localhost] => (item=neutron-server)11:18
chandankumarmarios: ysandeep|rover from config manifest _push for c7 and c8 train is enabled11:19
chandankumarhttps://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/train.ini#L11 and https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-8/train.ini#L1111:19
ysandeep|roverchandankumar, marios safe to turn it off? we don't utilize any additional info from container's manifest11:20
ysandeep|rover?11:20
mariosysandeep|rover: manifests are only for ppc containers tagged but they aren't available so we have it off11:21
mariosysandeep|rover: i think it is already off on master/ussuri11:21
mariosysandeep|rover: once they push the ppc containers then we can turn it back on11:21
ysandeep|rovermarios, sry but what's ppc?11:22
mariosysandeep|rover: i mean power pc arch ... lik containers built for ppc instaed of x8611:23
mariosppc64le ysandeep|rover11:23
ysandeep|rovermarios, okay got it thanks!11:24
ysandeep|roverchandankumar, yes you are right its already off for master/ussuri - https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-8/ussuri.ini#L1111:24
chandankumarysandeep|rover: propose a change in ci-config repo and then we can cherry -pick changes to promoter11:25
ysandeep|roverchandankumar, ack i will push a patch to turn if off for c7/c8 train11:25
ysandeep|roverwriting a bz for reference11:26
chandankumarysandeep|rover:  sure11:26
chandankumarthanks!11:26
ysandeep|roverchandankumar, thank you :)11:26
*** ccamacho has quit IRC11:29
*** udesale_ has joined #oooq11:31
*** pojadhav is now known as pojadhav|afk11:33
*** udesale has quit IRC11:34
ysandeep|rovermarios, chandankumar fyi.. https://review.rdoproject.org/r/#/c/28540/ and thank you for helping me on this :)11:50
mariosack ysandeep|rover11:52
ysandeep|roverpojadhav|afk, I am still looking at your bz for train ipa issue, will let you know when find something concrete11:54
akahatcgoncalves, hi, need to ask about review: https://review.opendev.org/#/c/740314/11:55
akahatcgoncalves, how to enable ovn provider?11:55
akahatfew tests are failing because of it.11:55
*** rfolco has joined #oooq11:56
*** rlandy has joined #oooq11:57
chandankumarysandeep|rover: this job https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario010-ovn-provider-standalone&result=SUCCESS was passing long time ago, we broke again11:57
cgoncalvesakahat, https://review.opendev.org/#/c/714639/11:57
chandankumarplease open a bug11:57
*** rlandy is now known as rlandy|ruck11:58
* akahat looking...11:58
rlandy|ruckysandeep|rover: chandankumar: thanks for picking up the train 7 container push failure11:58
* chandankumar does not nothing there11:59
chandankumarysandeep|rover: may be https://review.opendev.org/#/c/714639/ will fix sc10-ovn job11:59
ysandeep|roverchandankumar, i saw this but since its failing from may end i thought may be some story behind this.. will check previous RR notes11:59
ysandeep|roverchandankumar, checking11:59
rlandy|ruckzbr:  where are we with the master change?11:59
rlandy|ruckysandeep|rover: ^^?11:59
rlandy|ruckjoining program call11:59
*** saneax has quit IRC12:00
ysandeep|roverrlandy|ruck, chandankumar was refining his patch - https://review.rdoproject.org/r/#/c/28521/12:00
*** saneax has joined #oooq12:01
chandankumarrlandy|ruck: ysandeep|rover https://review.rdoproject.org/r/#/c/28521/ is ready from myside, waiting on zbr and marios to approve12:01
rlandy|ruckgreat - thanks12:01
*** dsneddon has joined #oooq12:03
rlandy|ruckysandeep|rover: chandankumar: do you have access to the promoter server to add https://review.rdoproject.org/r/#/c/28540/?12:04
rlandy|ruckthe stable promoter server12:04
ysandeep|roverrlandy|ruck, i don't have12:04
rlandy|ruckysandeep|rover: ok - will get you a tmate12:04
rlandy|ruckafter finish talking at program call12:04
ysandeep|roverrlandy|ruck, i am in prgm call too as observer12:05
rlandy|ruckysandeep|rover: we'll speak for like 1 minute :)12:05
mariosrlandy|ruck: at approximatly half passed ;)12:06
ysandeep|roverrlandy|ruck, :) hahahs i wish we were first in the list12:06
rlandy|ruckwe need to listen to all this12:07
rlandy|ruckmarios :)12:07
* rlandy|ruck sets timer12:07
*** dsneddon has quit IRC12:11
rlandy|ruckysandeep|rover: sent you tmate12:11
*** amoralej is now known as amoralej|lunch12:11
ysandeep|roverrlandy|ruck, ack12:12
rlandy|ruckysandeep|rover: do you see may change?12:12
rlandy|ruckset manifest_push: false12:12
rlandy|ruckok?12:12
ysandeep|roverlooks good12:13
rlandy|ruckwill save and restart promoter service12:13
ysandeep|roverrlandy|ruck, you making similiar changes for c8 train ?12:14
rlandy|ruckysandeep|rover: yep12:14
mariosbiab errand12:14
*** marios has quit IRC12:14
ysandeep|roverrlandy|ruck, okay great12:14
rlandy|ruckysandeep|rover: manifest_push: false12:15
rlandy|ruck - was there already on train centos812:15
ysandeep|roverrlandy|ruck, okay12:15
ysandeep|rovermaybe some manually changed earlier.. i had to make that change here - https://review.rdoproject.org/r/#/c/28540/12:16
rlandy|ruckysandeep|rover: stein failed container push in the pipeline12:18
rlandy|ruckwill testproject that line12:18
rlandy|ruckchecking queens, rocky12:18
ysandeep|roverrlandy|ruck, already did - https://review.rdoproject.org/r/#/c/28537/12:18
ysandeep|roverrlandy|ruck, its running but some ovb jobs failed there12:19
ysandeep|roverpending results from others12:19
rlandy|ruckysandeep|rover: ok - I'll check whether those are crteria12:19
rlandy|ruckwe need fs00212:19
rlandy|ruck1 and 212:19
rlandy|ruck on ovb12:19
ysandeep|roverthose failed :)12:20
rlandy|ruckysandeep|rover: yep - I'll rerun12:20
rlandy|ruckjust checking if 39 is in criteria12:20
ysandeep|roverrlandy|ruck, fyi.. queen/rocky running right now too in penstack-periodic-integration-stable4-5 pipeline12:21
rlandy|ruckysandeep|rover: yep12:21
rlandy|ruckfs001 is out ot criteria as is 3912:21
rlandy|ruckbut I'll rerun 1 and 212:21
ysandeep|roverokay12:22
ysandeep|roveri am dropping from tmate.. will keep an eye on logs via url12:23
rlandy|ruckysandeep|rover: submitted testproject12:29
ysandeep|roverack12:30
*** pojadhav|afk is now known as pojadhav12:32
pojadhavysandeep|rover, sure thanks12:32
pojadhavrlandy|ruck, Do you have any inputs on this https://bugs.launchpad.net/tripleo/+bug/1887633 ?12:33
openstackLaunchpad bug 1887633 in tripleo "periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-train job fails with pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] EHOSTUNREACH)")" [Undecided,Triaged]12:33
rlandy|ruckpojadhav: will look after call12:34
pojadhavrlandy|ruck, sure thanks :)12:34
*** ratailor has quit IRC12:38
zbrrlandy|ruck: i gave my +2 on chandan patch.12:42
rlandy|ruckzbr: ok - then we need to get change on the server itself12:42
zbrwf and open a new tmate. my key is not on it or i have wrong ip12:43
rlandy|ruckzbr: mol-promoterFAILURE in 5m 07s (non-voting)12:47
rlandy|ruck^^ ok? before I workflow12:47
rlandy|ruckzbr: sending you the tmate12:48
*** soniya29 has joined #oooq12:48
*** sshnaidm|afk is now known as sshnaidm12:53
ysandeep|roverchandankumar, fyi.. https://bugs.launchpad.net/tripleo/+bug/188766612:55
openstackLaunchpad bug 1887666 in tripleo "tripleo-ci-centos-8-scenario010-ovn-provider-standalone is failing on tempest tests with error Details: {'faultcode': 'Client', 'faultstring': "Provider 'ovn' is not enabled.", 'debuginfo': None}" [High,Triaged]12:55
*** ccamacho has joined #oooq12:55
chandankumarysandeep|rover: I have passed the patch12:56
*** marios has joined #oooq12:56
ysandeep|roverchandankumar, yes.. jobs are running lets wait for results12:56
*** ccamacho has quit IRC12:56
*** ccamacho has joined #oooq12:57
*** ccamacho has quit IRC12:58
*** saneax has quit IRC12:59
chandankumararxcruz: soniya29 https://review.rdoproject.org/r/#/c/28513/ skip list jobs13:04
chandankumartestproject coming13:04
ysandeep|roverrlandy|ruck, in downstream periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17 failing, is that already known?13:04
rlandy|ruckysandeep|rover: we re-enabled octavia conatiner build13:05
rlandy|ruckit might be from there13:05
rlandy|ruckI haven't looked yet13:05
rlandy|ruckysandeep|rover: what is failing is BM13:05
rlandy|ruckthere is a fail on flavors13:05
rlandy|ruckif you have time, you can check into that13:05
* rlandy|ruck fixing upstream container build - sec13:05
ysandeep|roverrlandy|ruck, Okay if I will take a look tomorrow's morning.. leaving for the day now13:06
rlandy|ruckysandeep|rover: sure ...13:06
rlandy|ruckthat's the problem ...13:06
ysandeep|roverrlandy|ruck, thanks!13:07
rlandy|ruckwhen we're on upstream13:07
rlandy|rucknobody to keep an eye on downsteeam :(13:07
ysandeep|roverrlandy|ruck, yes i came to know when i was testing a new integration job, but all integration job were skipping because periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17 failed13:08
*** skramaja has quit IRC13:08
rlandy|ruckysandeep|rover: I'll check that later13:08
rlandy|ruckI was expecting a possible failure due to the change13:08
ysandeep|roveri took a quick look this afternoon and 1 container failed to build.. so i think you are right it may be about that ocatavia thing13:09
*** ysandeep|rover is now known as ysandeep|away13:10
*** amoralej|lunch is now known as amoralej13:12
ysandeep|awayand we got c7 train promotion ...  SUCCESS promoting centos7-train tripleo-ci-testing as current-tripleo :)13:14
arxcruzchandankumar: why not have a base job, and the others inherit from the base?13:14
rlandy|ruckzbr: ok - ready13:15
rlandy|ruckjoining gm13:15
arxcruzchandankumar: nevermind, i see what you did13:15
*** rlandy|ruck is now known as rlandy|ruck|mtg13:17
chandankumararxcruz: if it https://review.rdoproject.org/r/#/c/28543/ works, we can move it to weekend pipeline13:28
arxcruzchandankumar: ack, it will fail, we just need to check if fail on tempest13:29
arxcruz:)13:29
chandankumaryes13:29
arxcruzwaiting the results13:29
chandankumarrlandy|ruck|mtg: did we patched the promoter with new container fix code?13:30
rlandy|ruck|mtgchandankumar: on tmate with zbr doing that now13:30
chandankumarrlandy|ruck|mtg: cool, thanks!13:30
rlandy|ruck|mtgzbr: http://38.145.34.55/centos8_ussuri.log13:35
rlandy|ruck|mtgzbr: https://hub.docker.com/repository/registry-1.docker.io/tripleomaster/centos-binary-openstack-base/tags?page=113:36
chandankumarrlandy|ruck|mtg: there is no container centos-binary-openstack-base, it is now centos-binary-base13:43
zbrchandankumar: can you join me and rlandy|ruck|mtg to debug promoter? nmh-wxgh-tdg13:48
zbrhttps://sbarnea.com/ss/Screen-Shot-2020-07-15-14-50-11.70.png13:50
chandankumarrlandy|ruck|mtg:  https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config.py#L4813:57
mariospanda: will you join us14:01
mariospanda: (we have scheduled call now )14:01
mariospanda: re Promoter Status sync14:01
*** Goneri has joined #oooq14:11
holsersshnaidm no package screen in centos814:13
holserShould we use tmux instead?14:13
sshnaidmholser, yeah14:18
sshnaidmholser, I was sure we replaced it..14:18
holserok let me fix14:18
holsergive me 2 sec14:18
*** bogdando has quit IRC14:20
holsersshnaidm https://review.rdoproject.org/r/#/c/28544/14:26
sshnaidmholser, +2, thanks14:26
holsersshnaidm yw14:36
zbrrfolco: any updates on mol-promoter job? if not let me know if I can try to update it.14:42
rfolcozbr, more or less14:42
rfolcozbr, I was able to get to the end of the job, except the stage setup part14:43
zbrrfolco: upload your last changes and ping send me a msg when you leave for the day, so I can continue tomorrow.14:44
chandankumarrlandy|ruck|mtg: I am putting a patch for exclude list14:48
rlandy|ruck|mtgchandankumar: cool14:48
rlandy|ruck|mtgchandankumar: maybe we can patch the promoter server?14:48
zbrchandankumar: i need to first remove old periodic on https://review.rdoproject.org/r/#/c/28455/14:49
zbri will add them back with new names14:50
rlandy|ruck|mtgzbr: chandankumar: marios: ysandeep|away: rfolco: bhagyashris: pojadhav: akahat: soniya29: panda: note vexxhost upgrade today14:52
rlandy|ruck|mtgwe will need to check if promotion jobs restart or not14:53
chandankumarrlandy|ruck|mtg: marios https://review.opendev.org/741233 patch up14:56
chandankumarplease review14:58
mariosbrb14:59
rfolcozbr, provisioning molecule scenario just passed locally15:03
rfolcozbr, will submit the change, probably will still fail in zuul for a while15:03
rfolcowe can break down into smaller patches and fix zuul failures if any15:04
zbrrfolco: upload15:04
mariosack chandankumar will do15:06
zbrpanda: you did not lint after rebasing https://review.rdoproject.org/r/#/c/27626/ -- it will fail15:11
zbrimport order is wrong15:11
rfolcozbr, https://review.rdoproject.org/r/#/c/28439/15:12
rfolcozbr, wip so I will try to split into smaller changes and fix zuul failures if any by tomorrow15:12
rfolcomarios, fyi ^15:13
*** ykarel is now known as ykarel|away15:13
rfolcothe good news is that the molecule scenario now works locally at least15:13
rfolcothe bad news is that I changed too much, even unnecessary places15:13
*** TrevorV has joined #oooq15:14
mariosrfolco: thanks15:16
pandazbr: thanks for moving the target a bit further every time that I'm close to the goal.15:17
zbrpanda: that is why i send you the ping yesterday, i knew that pylint would break your change15:23
*** dsneddon has joined #oooq15:25
*** rlandy|ruck|mtg is now known as rlandy|ruck15:26
pandazbr: you want to kill me, I always keep pprint imported even if I don't use it, so I can use it for small debugging15:26
*** bogdando has joined #oooq15:27
zbrpanda: bad practice. if you want, i can repair it for you.15:27
pandazbr: nobody is going to be abkle to repair you once I've done with you.15:28
zbrpanda: tbh, these imports where only one 1/58 type of issues identified by pylint, i had to put skips on all.15:28
zbrlucky me in am miles away15:28
pandazbr: I don't thin isort is working correctly ...15:29
pandaimport unittest15:30
pandaimport pytest15:30
pandafrom common import PromotionError15:30
pandait adds a space between unittest and pytest15:30
zbrconmon! or conman ;)15:30
pandaand pytest at the bottom ....15:30
pandaunless it's ordering by standard /non-standard library15:30
zbrbasic section rules: stdlib, 3rdparty, 1st party15:31
zbrthere are few others, but not so common.15:31
pandazbr: I'll make a demo for the all hands DFG next time15:31
zbrisort will do the work for you15:31
panda"I much I hate sorin"15:31
*** ykarel|away has quit IRC15:43
chandankumarzbr: rlandy|ruck https://review.rdoproject.org/r/#/c/28549/15:43
zbrchandankumar: why? not sure why it breaks on master.15:46
rlandy|ruckchandankumar: let me know what you think about comment on https://review.rdoproject.org/r/#/c/28549/15:47
rlandy|ruckin case we backport new container build15:47
chandankumarzbr: because those container does not exist15:48
chandankumarzbr: once tmp container list ggets created, it again append with openstack-base and os container15:48
zbr|rucki am trying to understand why on master we do not want to have a "base" container listed but we want on other branches. in the end all containers are based on a "base" one, regardless the branch.15:50
zbr|ruckthat is why i am wondering why different logic for master15:50
chandankumarzbr|ruck: let me pass the build tree for new container buidl15:51
toskyarxcruz: do you want to do the final push on the octavia patch?15:52
chandankumarzbr|ruck: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push/4bf5e62/logs/buildah-builds/container-builds/562e9548-2d0d-45b8-883e-c2ceb8091b93/build-tree.yaml15:53
chandankumarzbr|ruck: base container is ubi15:54
chandankumarzbr|ruck: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push/4bf5e62/logs/buildah-builds/container-builds/562e9548-2d0d-45b8-883e-c2ceb8091b93/base/Dockerfile so may we want to add ubi as a explicit list15:55
chandankumar?15:55
chandankumarrlandy|ruck: zbr|ruck https://review.opendev.org/#/c/740314  please merge in your time15:56
rlandy|ruckchandankumar: we need the depends merged there15:57
rlandy|ruckhttps://review.opendev.org/#/c/73150115:57
*** Tengu has quit IRC15:57
rlandy|ruckchandankumar: ^^ I don't have +2 on osf/python-tempestconf15:57
chandankumarrlandy|ruck: arxcruz will take care of that15:58
rlandy|ruckok15:58
*** Tengu has joined #oooq15:58
*** marios is now known as marios|out16:00
rlandy|ruckand .. another promoter error16:01
rlandy|ruck2020-07-15 15:57:14,196 12849 ERROR    promoter failed: [localhost] (item=openstack-base) => {"ansible_loop_var": "item", "changed": false, "item": "openstack-base", "msg": "Error pulling image trunk.registry.rdoproject.org/tripleomaster/centos-binary-openstack-base:53d2c73068becf11b7eb9f3662e666ee - 404 Client Error: Not Found (\"manifest for trunk.registry.rdoproject.org/tripleomaster/centos-binary-openstack-base:53d2c7306816:01
rlandy|ruckbecf11b7eb9f3662e666ee not found: manifest unknown: manifest unknown\")"}16:01
rlandy|ruckI added this to excludes16:01
chandankumarmay be that task is appending that16:02
rlandy|ruckoh - I excluded openstack-base16:03
*** marios|out has quit IRC16:03
rlandy|rucklet me try exclude both names16:04
chandankumarok16:04
* rlandy|ruck puts ALL containers in exclude list and runs away16:04
*** soniya29 has quit IRC16:09
rlandy|ruckzbr: we have a fs002 deploy problem16:17
toskychandankumar: is it so urgent that we can't wait for a final vote from arxcruz ?16:21
chandankumartosky: nope, it is not urgent16:22
*** TrevorV has quit IRC16:23
*** TrevorV has joined #oooq16:28
rlandy|ruckzbr|ruck: no dice on promoter16:33
*** ykarel|away has joined #oooq16:35
*** Tengu has quit IRC16:37
*** udesale_ has quit IRC16:38
*** Tengu has joined #oooq16:38
rlandy|ruckchandankumar: one more before you go ...16:41
rlandy|ruckhttps://review.rdoproject.org/r/#/c/28542/16:42
rlandy|ruckupdated per ykarel|away's comment16:42
arxcruzhey, tosky chandankumar which patch?16:44
*** bogdando has quit IRC16:45
toskyarxcruz: THAT patch! but you already found it16:48
arxcruztosky: :)16:50
ykarel|awayrlandy|ruck, sorry, just noticed container-login playbook is used in other jobs as well so need to add additional var instead of relying on registry_login_enabled16:50
ykarel|awayhttps://review.rdoproject.org/codesearch/?q=container-login.yaml&i=nope&files=&repos=16:51
rlandy|ruckok - will fix that later16:54
*** amoralej is now known as amoralej|off16:55
*** derekh has quit IRC17:00
rlandy|ruckykarel|away: tripleo_podman_buildah_login: "{{ push_containers|default(false)|bool }}"?17:06
*** dtantsur is now known as dtantsur|afk17:18
ykarel|awayrlandy|ruck, yes looks good17:18
rlandy|ruckthanks17:19
*** Tengu has quit IRC17:20
*** Tengu has joined #oooq17:22
*** ykarel|away has quit IRC17:25
*** Tengu has quit IRC17:28
*** Tengu has joined #oooq17:30
*** Tengu has quit IRC17:37
rlandy|ruckzbr: still around?17:38
*** Tengu has joined #oooq17:39
*** Tengu has quit IRC18:17
zbr|ruckrlandy|ruck: not really18:18
rlandy|ruckzbr|ruck: no worries - we're pushing to dockerhub now18:18
*** Tengu has joined #oooq18:19
zbr|ruckrlandy|ruck: rfolco: i want to merge https://review.rdoproject.org/r/#/c/28455/ -- even if marios has some doubts.18:21
rfolco+1 zbr|ruck18:22
zbr|ruckis not about removing the job permanently, is about adding its newer versio.18:23
rfolcozbr|ruck, I can't find what I changed that instance is not recognized anymore https://logserver.rdoproject.org/39/28439/11/check/mol-promoter/ccc006e/tox/reports.html18:23
*** tosky has quit IRC18:25
rlandy|ruckzbr|ruck: is it urgent?18:25
*** tosky has joined #oooq18:25
zbr|rucknot really18:25
rlandy|ruckif not, then I'd rather let marios have his say18:25
zbr|ruckhe did not know what me and folco were doing regarding mol-* jobs.18:26
rlandy|ruckzbr|ruck: I get that18:26
rlandy|ruckzbr|ruck: up to you18:26
zbr|ruckbasically we are going to scrap all molecule jobs and replace them with mol-* ones.18:27
zbr|ruckwhich use a well defined base job.18:27
zbr|ruckmol-promoter was the first one, followed by mol-get_hash, which is blocked by that removal.18:27
zbr|ruckbasically because periodic jobs were defined inside rdo-jobs (which is bit unclear to me)18:28
rlandy|ruckok by me18:28
rlandy|ruckyou have +2, right?18:28
zbr|ruckAFAIK, we can defined periodics anywhere.18:28
zbr|ruckyeah, i do.18:28
rlandy|ruckok - so marios can fight with you :)18:28
zbr|ruckjust wanted to explain it18:28
* rlandy|ruck stays out of this one18:29
zbr|ruckis not really a fight, he asked for some paperwork.18:29
zbr|ruckin his right to do it :D18:29
zbr|ruckin case it was not clear taiga/hackmd = paperwork18:30
rlandy|ruckzbr|ruck: ok - I'm editing https://review.rdoproject.org/r/#/c/28549/18:30
rlandy|ruck^^ change is already on the promoter18:30
rlandy|ruckie: to get rid of openstack-base18:30
zbr|ruckrfolco: i will look tomorrow, too tired now. Basically in mol-pre we add instance to /etc/hosts so our job is able to connect to "instance" --- which in zuul is 127.0.0118:32
rfolcoaaaaah pre.yml18:32
rfolcogot it18:33
rfolcothx zbr|ruck18:33
zbr|ruckthat is where you should look18:33
zbr|ruckthat is the trick we do with zuul, very similar with what you do for local testing, finding a way to resolve "instance".18:33
zbr|rucki will takeover tomorrow morning from where you leave it, maybe we can speed it up.18:34
zbr|rucknow I have "bigger" problems at home, stuggle to find a good ANSI TKL mechanical keyboard in UK,...18:35
rfolcothat is weird18:36
rlandy|ruckHOLY COW - we promoted master19:01
rlandy|ruckzbr|ruck: ^^19:01
*** Tengu has quit IRC19:01
rlandy|ruckwoohoo and images19:02
rlandy|ruckhttps://images.rdoproject.org/centos8/master/rdo_trunk/current-tripleo/19:02
* rlandy|ruck does happy dance19:02
rlandy|ruckchandankumar: ^^19:03
*** Tengu has joined #oooq19:03
*** Tengu has quit IRC19:41
*** Tengu has joined #oooq19:43
*** Tengu has quit IRC20:01
*** Tengu has joined #oooq20:03
arxcruzakahat: congrats on https://review.opendev.org/#/c/731501/ :)20:56
*** Tengu has quit IRC21:14
*** Tengu has joined #oooq21:16
*** TrevorV has quit IRC21:18
*** Tengu has quit IRC21:36
*** Tengu has joined #oooq21:38
*** rlandy|ruck is now known as rlandy|ruck|bbl22:34
*** tosky has quit IRC22:48
*** Tengu has quit IRC22:54
*** Tengu has joined #oooq22:56
*** Tengu has quit IRC23:16
*** Tengu has joined #oooq23:17
*** ysandeep|away is now known as ysandeep23:54
ysandeeprlandy|ruck|bbl, :) we got master promotion \o/ yay23:55
*** rlandy|ruck|bbl is now known as rlandy|ruck23:55
rlandy|ruckysandeep: yep - ...23:55
rlandy|ruckand ci.centos passed23:56
rlandy|ruckso that should promote next23:56
rlandy|ruckvexxhost issues23:56
rlandy|ruckstein will promote next23:56
ysandeeprlandy|ruck, you re-running stein jobs via testproject?23:57
rlandy|ruckyep23:57
ysandeepokay23:57
rlandy|ruckysandeep: with two queens jobs I tried rdocloud23:57
rlandy|ruckysandeep: I haven't tackled downstream yet23:58
rlandy|ruckwill look at that in a bit23:58
ysandeeprlandy|ruck, i will take a look at bm failure today23:58
rlandy|ruckcool23:58
ysandeepplease check that https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17/bcb2416/job-output.txt if you have time.. I have less idea about this job..23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!