| *** rlandy is now known as rlandy|out | 00:53 | |
| *** ysandeep|PTO is now known as ysandeep | 05:07 | |
| marios | o/ | 05:17 |
|---|---|---|
| ysandeep | \o good morning everyone | 05:20 |
| marios | bhagyashris|ruck: o/ can you reply to the support ticket sending you pvt now | 05:52 |
| marios | bhagyashris|ruck: (better if only one person is replying there) | 05:52 |
| bhagyashris|ruck | marios, ack | 05:57 |
| marios | thanks bhagyashris|ruck | 06:04 |
| marios | jm1[m]: is the ansible openstack modules session @ ptg today? | 06:09 |
| marios | jm1[m]: i see os-ansible-modules at 1300 utc https://ptg.opendev.org/ptg.html | 06:09 |
| *** amoralej|off is now known as amoralej | 06:19 | |
| * bhagyashris|ruck lunch brb | 06:56 | |
| jm1 | o/ | 07:29 |
| jm1 | marios: yes, aoc ptg is at 1300 utc today :) | 07:29 |
| marios | o/ | 07:34 |
| marios | thx jm1 | 07:35 |
| marios | bhagyashris|ruck: frenzyfriday|rover: o/ do you want to sync today? if you do ping me/send me invite please? (e.g. in 25 mins as usual is a good time?) | 07:37 |
| * marios fetch cofffee | 07:37 | |
| frenzyfriday|rover | marios, upstream looks good. Master promoted finally | 07:37 |
| marios | \o/ | 07:37 |
| frenzyfriday|rover | I have to check the cix, I'll ping you if I need help on them | 07:37 |
| frenzyfriday|rover | looks like everything promoted :D thats suspicious | 07:38 |
| jm1 | frenzyfriday|rover: master promoted? oh yeah🥳 | 07:39 |
| frenzyfriday|rover | jm1, yep finally! I'll check on your card and add back to criteria if everything works | 07:39 |
| marios | thanks frenzyfriday|rover | 07:45 |
| marios | 10:38 < frenzyfriday|rover> looks like everything promoted :D thats suspicious | 07:45 |
| marios | yup you've been in this team long enough :D ^^^ | 07:45 |
| frenzyfriday|rover | XD | 07:46 |
| bhagyashris|ruck | marios, hey nothing from downstream side we are still blocked due to dns issue | 07:47 |
| marios | bhagyashris|ruck: ack | 07:48 |
| marios | arxcruz: o/ looks good now https://quay.io/organization/tripleozedcentos9 | 08:25 |
| marios | arxcruz: after the log fix patch merges we should be OK for the copy script? (those were manual?^^ ) | 08:25 |
| arxcruz | marios yes, it will update the toolbox automatically and keep copying | 08:26 |
| marios | thx for help arxcruz | 08:27 |
| marios | need more reviews please when you have time o/ https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45641 Add zed ovb periodic integration jobs for zed criteria | 08:43 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 08:43 |
| *** ysandeep is now known as ysandeep|lunch | 09:08 | |
| *** amoralej` is now known as amoralej | 09:09 | |
| jpodivin | bhagyashris|ruck: Hi. I'm running into problems with one of our component pipeline jobs. Really odd, we are cleaning a log directory before running our tests. Because we make assertions on the contents. But for some reason, even when the task seems to have passed, the original contents are still there. https://logserver.rdoproject.org/openstack-component-validation/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_ | 09:40 |
| jpodivin | 1comp-featureset001-component-master-validation/80cc670/job-output.txt | 09:40 |
| marios | jpodivin: yes! | 09:41 |
| marios | jpodivin: i was just digging there https://opendev.org/openstack/validations-common/src/commit/b02d478d513a2b35b969ef96f766923714c4a20a/roles/validations/tasks/list_validation_history.yaml#L27 | 09:41 |
| marios | jpodivin: i thought it was zed specific (i see it in some results @ https://review.rdoproject.org/r/c/testproject/+/45451/5#message-6030179ed278fc32a111e88cf407a385454dc295 | 09:42 |
| marios | jpodivin: is it intermittent? it seems some of those jobs were previously green. perhaps a recent commit (last week or so?) | 09:42 |
| jpodivin | marios: It may be but we have just merged https://review.opendev.org/c/openstack/validations-common/+/861716 and it's still happening. | 09:42 |
| jpodivin | what I don't understand is: How can something be removed and yet be still there. | 09:43 |
| jpodivin | I mean, unless the module call is wrong or something. | 09:43 |
| jpodivin | But it does seem straight forward https://docs.ansible.com/ansible/latest/collections/ansible/builtin/file_module.html | 09:44 |
| marios | jpodivin: for https://review.opendev.org/c/openstack/validations-common/+/861716 then we'll need promotion to get it into the periodic jobs | 09:45 |
| marios | jpodivin: promotion of https://trunk.rdoproject.org/centos9-master/component/validation/ | 09:45 |
| jpodivin | marios: see I would think that as well but when I look at the most recent failures ... I see that task running | 09:45 |
| jpodivin | and returning "Changed" | 09:45 |
| jpodivin | https://logserver.rdoproject.org/openstack-component-validation/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation/80cc670/job-output.txt | 09:46 |
| jpodivin | see here ^ | 09:46 |
| jpodivin | 2022-10-19 21:52:06.613349 | 09:46 |
| jpodivin | At that point there should be two files in the log dir, one for undercloud-disabled-services, other for undercloud-disk-space. | 09:47 |
| jpodivin | As you can see here: https://logserver.rdoproject.org/openstack-component-validation/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation/80cc670/logs/undercloud/home/zuul/ansible.log.txt.gz | 09:48 |
| jpodivin | These validations are executed some time before the log dir is removed. | 09:48 |
| marios | jpodivin: yeah but that is validation component job | 09:49 |
| jm1 | frenzyfriday|rover, marios: do we have any pending reviews for zed cockpit stuff? | 09:49 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 09:49 |
| marios | jpodivin: so it fetches component-ci-testing for the validation bits (validation repo is under test here so ci-testing for that and curren-tripleo for other stuff) | 09:49 |
| marios | https://logserver.rdoproject.org/openstack-component-validation/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation/80cc670/logs/undercloud/etc/yum.repos.d/validation-component.repo.txt.gz | 09:49 |
| marios | jpodivin: ^^^ so it must be new enough to have the recently merged thing | 09:50 |
| jpodivin | it is yes. That's why I don't understand I don't see any results. | 09:50 |
| marios | jpodivin: yeah b02d478d513a2b35b969ef96f766923714c4a20a via https://trunk.rdoproject.org/centos9-master/component/validation/component-ci-testing/versions.csv | 09:51 |
| jpodivin | marios: logs show something is going on ... task is executed and returning "changed" | 09:51 |
| marios | jpodivin: k so... this hits all branches i guess? well, wallaby was green when i checked eg https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby | 09:51 |
| marios | jpodivin: so it hits only master? (and zed in my test jobs) | 09:51 |
| marios | jm1: looking, i know there was one thing pojadhav was looking at | 09:52 |
| marios | jm1: https://issues.redhat.com/browse/TRIPLEOCI-1249 | 09:52 |
| marios | pojadhav: can you add review please ^^ | 09:52 |
| marios | frenzyfriday|rover: fyi discussion with jpodivin here we have new blocker for master (and zed) components | 09:53 |
| jpodivin | marios: it used to hit only master. But VF is essentially putting the same code in master and wallaby ... | 09:53 |
| jpodivin | marios: I feel like if we could figure out why my mitigation isn't working we would be able to resolve this. | 09:54 |
| marios | frenzyfriday|rover: do you have time to file that please ? otherwise i can let me know | 09:54 |
| jm1 | marios: ack, thanks! | 09:54 |
| marios | frenzyfriday|rover: giving you some logs in pvt so easier to track | 09:54 |
| * frenzyfriday|rover checks | 09:54 | |
| jpodivin | I have a bug right here https://bugs.launchpad.net/tripleo/+bug/1993262 | 09:54 |
| jpodivin | frenzyfriday|rover: ^ | 09:54 |
| marios | ah great frenzyfriday|rover ^^^ | 09:54 |
| marios | thanks jpodivin | 09:54 |
| marios | that is cix as it blocks components | 09:55 |
| frenzyfriday|rover | of awesome! thanks | 09:55 |
| marios | jpodivin: should we disable until we can work it out? is there some smaller hammer than disabling all of it? https://github.com/rdo-infra/rdo-jobs/blob/a1789557651febb6b386a11bfbaabd1199da19cd/zuul.d/component-jobs-master-centos9.yaml#L638 | 09:56 |
| marios | jpodivin: s/should we/we should ;) | 09:56 |
| frenzyfriday|rover | marios, jpodivin there is already a fix for that? I see fix released in the Bug | 09:56 |
| jpodivin | marios: wait .... I'm having a feeling these are two different things | 09:56 |
| marios | frenzyfriday|rover: i think they were hoping https://review.opendev.org/c/openstack/validations-common/+/861716 would help | 09:56 |
| marios | jpodivin: can you please update with this info ^^^ in the bug? | 09:56 |
| frenzyfriday|rover | ah, okay | 09:56 |
| jpodivin | marios: I believe the patch is linked automatically | 09:57 |
| jpodivin | marios, frenzyfriday|rover: yep, I think I'm right. These are two different issues. | 09:57 |
| frenzyfriday|rover | I added promotion blocker tag, if we later decide to set the bug back to in progress it will show up on cix | 09:57 |
| marios | jpodivin: https://bugs.launchpad.net/tripleo/+bug/1993262 looks like the thing i am hitting in zed though. what is the 'other issue' you are referring to? | 09:59 |
| jpodivin | frenzyfriday|rover: I think we should file a new bug actually. I've checked logs from jobs marios run, and it doesn't seem to be the same error. | 09:59 |
| marios | jpodivin: you mean the proposed/merged fix is unrelated to the bug? | 09:59 |
| marios | 12:10:22.112916 | primary | fatal: [undercloud]: FAILED! => {"changed": false, "msg": "The history output length 5 doesn't match the number of | 09:59 |
| marios | expected validations runs 3.\n"} | 09:59 |
| marios | looks same jpodivin ? ^ | 09:59 |
| frenzyfriday|rover | jpodivin, ack, /me files a bug | 09:59 |
| marios | frenzyfriday|rover: wait | 09:59 |
| marios | jpodivin: please clarify? | 09:59 |
| frenzyfriday|rover | https://logserver.rdoproject.org/51/45451/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-zed-validation/eccce54/logs/undercloud/home/zuul/full_validation_history.log.txt.gz looks the same as https://logserver.rdoproject.org/91/43591/11/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation/8bf10e0/logs/undercloud/home/zuul/full_validation_his | 10:01 |
| frenzyfriday|rover | tory.log.txt.gz | 10:01 |
| jpodivin | marios: yes, that is true for some, but periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-zed fails on deploy | 10:01 |
| jpodivin | https://review.rdoproject.org/zuul/build/5c7645e36e4d46ef84b90d6e7ae10842 | 10:01 |
| marios | jpodivin: ? i know only some of those hit the issue. | 10:01 |
| marios | jpodivin: so it is the same issue then | 10:01 |
| marios | frenzyfriday|rover: no need for another bug | 10:01 |
| marios | jpodivin: i have 3 there | 10:01 |
| marios | * https://logserver.rdoproject.org/51/45451/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-compute-zed/b7ce58b/job-output.txt | 10:01 |
| marios | * https://logserver.rdoproject.org/51/45451/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-zed/c9f0335/job-output.txt | 10:01 |
| marios | * https://logserver.rdoproject.org/51/45451/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-zed-validation/eccce54/job-output.txt | 10:02 |
| jpodivin | ok those are the same thing. | 10:02 |
| marios | * https://logserver.rdoproject.org/51/45451/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-zed/3d29db4/job-output.txt | 10:02 |
| marios | k | 10:02 |
| marios | four in fact | 10:02 |
| jpodivin | So coming back to what we were talking about. Can you think of any reason why would the task be ignored? | 10:02 |
| marios | jpodivin: you mean in the validation component? it says 'changed'? maybe it is the wrong fix/soemthing more needed? what do you mean task is ignored | 10:03 |
| jpodivin | marios: that's what I mean. It shows up, but nothing really happens. | 10:03 |
| marios | jpodivin: frenzyfriday|rover: i am going to put in a patch to disable validations | 10:04 |
| marios | jpodivin: is there some way to disable just this one instead? ^^ | 10:04 |
| jpodivin | marios: what do you mean? | 10:04 |
| marios | jpodivin: i mean just this validation instead of all of them | 10:05 |
| marios | jpodivin: dont think so just looking at the task | 10:05 |
| jpodivin | marios: well, there is no one validation doing this. This is an issue of the tests we run on our framework. | 10:06 |
| marios | jpodivin: ack so i guess enable_validation: false is the way for now sounds like | 10:06 |
| marios | jpodivin: until we find the actual problem/fix | 10:07 |
| jpodivin | marios: well we know what the problem is ... directory is not getting removed . | 10:07 |
| marios | :) | 10:08 |
| marios | k then the fix | 10:08 |
| jpodivin | and that's the problem. I have no idea how to fix ansible ignoring my orders. :D | 10:08 |
| marios | jpodivin: k but either way it will be easier to work it out without having the pressure of blocking all component lines | 10:09 |
| marios | jpodivin: frenzyfriday|rover: added info @ https://bugs.launchpad.net/tripleo/+bug/1993262/comments/4 | 10:11 |
| frenzyfriday|rover | jm1, hey, did we have a tempest skiplist patch for https://bugs.launchpad.net/tripleo/+bug/1992668? The sc10 kvm internal is passing now. | 10:14 |
| frenzyfriday|rover | jm1, oh, I see the job itself was out of criteria. | 10:15 |
| frenzyfriday|rover | move sc10 kvm back to criteria - https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45707 | 10:18 |
| *** rlandy|out is now known as rlandy | 10:29 | |
| rlandy | bhagyashris|ruck: frenzyfriday|rover: hey | 10:30 |
| bhagyashris|ruck | rlandy, Hi, | 10:30 |
| rlandy | bhagyashris|ruck: frenzyfriday|rover: want to sync? | 10:30 |
| rlandy | any progress with OVB on internal? | 10:30 |
| bhagyashris|ruck | created service-now ticket for container build issue https://redhat.service-now.com/help?id=rh_ticket&is_new_order=true&table=incident&sys_id=e14a38db872e999807c9ed3c8bbb35c8 | 10:30 |
| rlandy | bhagyashris|ruck: when was the last container push failure? | 10:31 |
| rlandy | frenzyfriday|rover'; W+'ed your patch | 10:31 |
| bhagyashris|ruck | yesterday on rhos17.1 on rhel9 | 10:32 |
| bhagyashris|ruck | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17.1/a69a965/job-output.txt | 10:32 |
| rlandy | bhagyashris|ruck: so no failures today? | 10:32 |
| rlandy | bhagyashris|ruck: I rerun 17.1 on 8 yesterday | 10:33 |
| rlandy | when things started to come back | 10:33 |
| rlandy | worked ok | 10:33 |
| bhagyashris|ruck | rlandy, today rhos17 on rhel9 ran and container build pass there | 10:33 |
| bhagyashris|ruck | looks like situation is improving | 10:34 |
| rlandy | bhagyashris|ruck: and 16.2 | 10:34 |
| rlandy | so I suspect it may be better | 10:34 |
| bhagyashris|ruck | rlandy, and jfyi if we see the recent run of rhos17 o rhel9 | 10:34 |
| bhagyashris|ruck | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/f87ef31d7a2049b8aad0f6c5c99e4bed | 10:34 |
| bhagyashris|ruck | only the ovb jobs are failing all the other jobs passed | 10:34 |
| bhagyashris|ruck | rlandy, yes looks better now | 10:35 |
| rlandy | bhagyashris|ruck: I expect baremetal fails as well | 10:35 |
| rlandy | we still have settings to change there | 10:35 |
| bhagyashris|ruck | rlandy, yeah | 10:35 |
| bhagyashris|ruck | and 16-2 is also looking better now https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/0245f621c18443ae8e735582f49cd9fd | 10:36 |
| rlandy | bhagyashris|ruck: ok - so I am pinging dpawlik on https://redhat.service-now.com/help?id=rh_ticket&table=sc_req_item&sys_id=36aeadc78726d9d0d5cc642c8bbb3518&view=ess | 10:37 |
| rlandy | no action there | 10:37 |
| rlandy | in the mean time | 10:37 |
| rlandy | there are two things to investigate/fix | 10:37 |
| rlandy | bhagyashris|ruck: the baremetal settings | 10:37 |
| rlandy | and why scenario010 is still failing | 10:37 |
| rlandy | frenzyfriday|rover: you around? | 10:38 |
| frenzyfriday|rover | rlandy, yeah, reading back | 10:38 |
| rlandy | frenzyfriday|rover: bhagyashris|ruck: let's sync - quicker | 10:38 |
| bhagyashris|ruck | rlandy, fine. but IMO we should keep that as open for few days just to check the results and if it's consistently passing then we can close the ticket | 10:38 |
| frenzyfriday|rover | rlandy, ack | 10:38 |
| rlandy | https://meet.google.com/ngu-joiv-crb?pli=1&authuser=0 | 10:39 |
| rlandy | frenzyfriday|rover: bhagyashris|ruck: ^^ | 10:39 |
| marios | jpodivin: frenzyfriday|rover: https://review.rdoproject.org/r/c/rdo-jobs/+/45708 | 10:41 |
| marios | rlandy: fyi ^^ | 10:41 |
| *** ysandeep|lunch is now known as ysandeep | 10:46 | |
| pojadhav | marios, ack | 10:49 |
| jpodivin | marios: I think I've got it. It's wrong dir. | 10:51 |
| * jpodivin facepal | 10:51 | |
| * jpodivin m | 10:51 | |
| marios | jpodivin: ok if you have fix happy to abandon that one so.. race ;) | 10:52 |
| rlandy | bhagyashris|ruck: remove nodeset: https://code.engineering.redhat.com/gerrit/gitweb?p=openstack/tripleo-ci-internal-jobs.git;a=blob;f=zuul.d/integration-pipeline-rhos-16.2.yaml;h=23faa51783a4f27ca2f4a4e99d8b21e41adda10d;hb=HEAD | 10:53 |
| rlandy | bhagyashris|ruck: and here: https://code.engineering.redhat.com/gerrit/gitweb?p=openstack/tripleo-ci-internal-jobs.git;a=blob;f=zuul.d/project-templates-components.yaml;h=d57bd37428caab60202829777425dd5e1f9afca3;hb=HEAD | 10:59 |
| marios | jpodivin: thinking about it i think we still need the disable. you'll have to merge the fix, then we'll need it promoted through validation component in order to be available to the rest of the components | 11:04 |
| marios | jpodivin: that will take at least a day so.. | 11:04 |
| jpodivin | yeah, that makes sense. My fix is up, tests are running here https://review.rdoproject.org/r/c/testproject/+/43591 | 11:06 |
| marios | jpodivin: thanks just saw... can you please ... ah :D was going to ask for test thanks! | 11:07 |
| marios | well at least i know it is not a zed specific thing so unblocking https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45611/2#message-fc804672407c139e2288ee7bd71b3757534af006 | 11:08 |
| jpodivin | marios: thanks for the help. :) | 11:09 |
| marios | needs reviews please https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45611 Adds rest of component jobs to zed component criteria | 11:11 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 11:11 |
| marios | thanks for jumping on that jpodivin | 11:11 |
| rlandy | arxcruz: hello - pls update this card: https://trello.com/c/3p8i2YdZ/2639-cixlp1982874tripleociproa-testcreateobjectwithtransferencoding-is-failing-on-tripleo-jobs | 11:11 |
| arxcruz | rlandy i will, i'm bumping tempest version on master and doing tests to make sure it is working | 11:12 |
| arxcruz | let me grab the reviews for you | 11:12 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 11:12 |
| arxcruz | once i have it i'll update the card | 11:12 |
| arxcruz | rlandy testing here https://review.rdoproject.org/r/c/testproject/+/45706 and bumping version here https://review.rdoproject.org/r/c/rdoinfo/+/45711 | 11:14 |
| rlandy | arxcruz: thanks - pls add those comments to the card | 11:16 |
| arxcruz | ok | 11:16 |
| marios | pojadhav: i don't see the review yet though https://issues.redhat.com/browse/TRIPLEOCI-1249 please add? | 11:16 |
| pojadhav | marios, I am working on it | 11:16 |
| marios | pojadhav: ah sorry, i thought i already reviewd that ? | 11:17 |
| marios | pojadhav: ah maybe i confused it with your 'victoria removal' one. | 11:17 |
| pojadhav | marios, the issue was when my victoria patch got marged, promotions data were seen, so patch got reverted. so I also waiting to take help from dasm for cockpit changes for both victoria removal and addition of zed release. | 11:18 |
| marios | pojadhav: i thought you already posted... sorry ... so jm1 just fyi there is no patch yet pojadhav working on it | 11:18 |
| pojadhav | promotions data were not seen when victoria patch got merged. | 11:18 |
| marios | yeah i recall that i just thought i had also seen the zed one my bad | 11:19 |
| pojadhav | marios, still I am preparing patch for the same. will push in some time. | 11:19 |
| marios | k | 11:20 |
| marios | thanks pojadhav i did not intend to pressure you to finish it quickly. i really thought you already had it and was pressing for you to put into the jira task (i couldn't find it in the reviews ;) now i know why) | 11:20 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 11:20 |
| pojadhav | marios, its fine :) | 11:21 |
| ysandeep | chandankumar, o/ Do we have a green run after updating ctlplane ip? for this patch: https://review.opendev.org/c/openstack/tripleo-quickstart/+/861748 | 11:22 |
| chandankumar | ysandeep: yes | 11:22 |
| ysandeep | chandankumar, could you please pass me those logs, I want to check nova-compute logs on compute node | 11:23 |
| chandankumar | ysandeep: I have forced failed the latest run https://review.rdoproject.org/r/c/testproject/+/45547/20/.zuul.yaml | 11:23 |
| chandankumar | I have hold the node also | 11:23 |
| chandankumar | https://logserver.rdoproject.org/47/45547/20/check/tripleo-ci-centos-9-standalone-external-compute-target-host1/5a777a8/job-output.txt | 11:23 |
| ysandeep | ahh, I was checking this testproject but only checked last green run - https://review.rdoproject.org/r/c/testproject/+/45547/20#message-7595b526a061716cfadf00082cda7b30944dc201 | 11:24 |
| chandankumar | https://logserver.rdoproject.org/47/45547/20/check/tripleo-ci-centos-9-standalone-external-compute-target-host1/5a777a8/logs/subnode-1/var/log/containers/nova/nova-compute.log.txt.gz | 11:24 |
| chandankumar | ysandeep: everything is working fine | 11:24 |
| chandankumar | except controller to compute communication | 11:24 |
| chandankumar | will fix later on | 11:24 |
| *** dviroel|out is now known as dviroel | 11:25 | |
| ysandeep | chandankumar, controller to compute communication - I thought changing ctlplane IP was going to solve that issue only | 11:26 |
| chandankumar | ysandeep: so https://logserver.rdoproject.org/47/45547/19/check/tripleo-ci-centos-9-standalone-external-compute-target-host1/913dc66/logs/subnode-1/var/log/containers/nova/nova-compute.log.txt.gz is gone now | 11:26 |
| chandankumar | we are seeing new issue https://logserver.rdoproject.org/47/45547/20/check/tripleo-ci-centos-9-standalone-external-compute-target-host1/5a777a8/logs/subnode-1/var/log/containers/nova/nova-compute.log.txt.gz now | 11:26 |
| chandankumar | somethign wrong on control plane networking side | 11:27 |
| chandankumar | I have asked yatin to take a look | 11:27 |
| bhagyashris|ruck | rlandy, chandankumar marios scenario010 patch https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/431979 Remove single-rhel-8-4-node nodeset for scenario010 jobs | 11:27 |
| bhagyashris|ruck | scenario010 fix^ | 11:27 |
| ysandeep | chandankumar, ack | 11:27 |
| chandankumar | ysandeep: we need this patch https://review.opendev.org/c/openstack/tripleo-quickstart/+/861748 | 11:27 |
| ysandeep | I would say lets hold https://review.opendev.org/c/openstack/tripleo-quickstart/+/861748 , till we completely fix compute to standalone communication or atleast know what the issue is | 11:29 |
| marios | bhagyashris|ruck: ack but please add related-bug>? | 11:29 |
| rlandy | bhagyashris|ruck; thanks merging | 11:31 |
| marios | rlandy: can we add bug before merge? | 11:32 |
| rlandy | we can - I merged as you +2'ed it | 11:32 |
| marios | must be th is one https://bugzilla.redhat.com/2105408 | 11:32 |
| rlandy | yep | 11:33 |
| rlandy | that's the right one | 11:33 |
| rlandy | ysandeep: hey | 11:33 |
| marios | rlandy: ok whatever its done now. next time i should -1 then ;) (I did add a comment in my defense but point taken) | 11:33 |
| ysandeep | rlandy, hello o/ | 11:33 |
| rlandy | marios: I fixed the commit message with the added bug number - all done | 11:36 |
| marios | rlandy: :) thank you | 11:36 |
| rlandy | ysandeep: hey - have a few minutes to talk about rr tool and alternative jobs? | 11:36 |
| rlandy | I am thinking how to represent that | 11:37 |
| ysandeep | rlandy: sure, let me grab my earphones | 11:37 |
| rlandy | chandankumar: IBM cloud scheduling is soooo slow | 11:37 |
| rlandy | chandankumar: have you noticed that? | 11:38 |
| rlandy | ysandeep: https://meet.google.com/cwg-jdmw-bxe?pli=1&authuser=0 | 11:38 |
| marios | rlandy: bhagyashris|ruck: frenzyfriday|rover: jpodivin: going to merge that FYI https://review.rdoproject.org/r/c/rdo-jobs/+/45708 (disable validations for related-bug) | 11:38 |
| rlandy | marios: ok | 11:39 |
| chandankumar | rlandy: yes, so many jobs are queued | 11:41 |
| chandankumar | dpawlik: hello | 11:41 |
| chandankumar | dpawlik: so many jobs on ibm cloud in queued state, is it expected? job scheduling is slow | 11:41 |
| marios | pojadhav: i think https://issues.redhat.com/browse/TRIPLEOCI-1247 done there? if you agree please move it done when you have a minute thanks | 11:43 |
| pojadhav | marios, yes its done | 11:44 |
| pojadhav | i will move it to done | 11:44 |
| pojadhav | marios, moved to done 1247 task | 11:45 |
| marios | frenzyfriday|rover: assigned this to you https://issues.redhat.com/browse/TRIPLEOCI-1250 and moved it done ;) | 11:46 |
| marios | thanks pojadhav | 11:46 |
| bhagyashris|ruck | rlandy, dns nameserver update patch https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/431983 Update dns_nameservers ip | 11:50 |
| * bhagyashris|ruck tea brb | 11:50 | |
| rlandy | bhagyashris|ruck: thanks - will check | 11:50 |
| rlandy | ysandeep: ^^ can you take a look - and then will merge | 11:51 |
| rlandy | bhagyashris|ruck: once that patch is merged - pls testproject the 16.2 bm job | 11:52 |
| rlandy | and let's see if we get by | 11:52 |
| ysandeep | I remember jakob corrected these values last week, looking at related bug on what's the reasoning behind change | 11:54 |
| dpawlik | chandankumar: dunno, will check in few min | 11:54 |
| dpawlik | chandankumar: https://softwarefactory-project.io/grafana/d/CJAaWS3nz/provider-ibm-bm2-nodepool?orgId=1 | 11:55 |
| dpawlik | 9 is in use | 11:55 |
| rlandy | ysandeep: jm1 corrected some of them | 11:57 |
| rlandy | the change is needed | 11:57 |
| rlandy | old nameservers do not work | 11:57 |
| marios | oooci please add to your queue please: criteria patches https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45611 https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/429994 | 11:57 |
| ysandeep | rlandy, bhagyashris|ruck https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/431983/1#message-5c99cf3ea082bab965e9b60355b3e2e7b36635df comment for discussion | 12:04 |
| ysandeep | bhagyashris|ruck, are bm jobs broken currently, If yes, I can login and crosscheck the dns setting | 12:05 |
| rlandy | ssec | 12:06 |
| ysandeep | there are not clear info in the page i have attached - it is mentioned to use first entry for PSI, LAB's, Enginnering ,and second entry for RDU | 12:07 |
| chandankumar | dpawlik: ah ok then it makes sense | 12:08 |
| chandankumar | dpawlik: thank you :-) | 12:08 |
| chandankumar | rlandy: based on this https://softwarefactory-project.io/grafana/d/CJAaWS3nz/provider-ibm-bm2-nodepool?orgId=1 it appears to be slow | 12:08 |
| frenzyfriday|rover | marios, cool, thanks | 12:08 |
| ysandeep | rlandy, bhagyashris|ruck ^^ my above comment.. As these BM machines are in RDU but these are LAB- I wonder if we should go ahead with first entry or second entry | 12:09 |
| chandankumar | rlandy: on IBM cloud, in one go, at max 10 instance gets spanned | 12:09 |
| rlandy | chandankumar: so just loading? | 12:11 |
| *** amoralej is now known as amoralej|lunch | 12:13 | |
| rlandy | ysandeep: sorry - need a few | 12:13 |
| rlandy | chandankumar: https://review.opendev.org/c/openstack/tripleo-docs/+/861962 - fallout from PTg? | 12:14 |
| *** ysandeep is now known as ysandeep|mtg | 12:15 | |
| chandankumar | rlandy: yes | 12:15 |
| rlandy | chandankumar: yes to which question? | 12:31 |
| chandankumar | rlandy: ovb tag comes from ptg discussion | 12:31 |
| chandankumar | rlandy: regarding ibm, we need to wait instances to available then new nodes to get assigned | 12:34 |
| rlandy | ack ok | 12:35 |
| bhagyashris|ruck | ysandeep|mtg, yes bm jobs are broken | 12:37 |
| rlandy | ysandeep|mtg: bhagyashris|ruck: looking at dns patch | 12:40 |
| rlandy | let me confirm why they are broken | 12:40 |
| rlandy | 2022-10-19 23:21:22.650522 | primary | TASK [fetch-images : Cache image by checksum] ********************************** | 12:41 |
| rlandy | 2022-10-19 23:21:22.650532 | primary | Wednesday 19 October 2022 23:21:22 -0400 (0:00:00.088) 1:47:24.299 ***** | 12:41 |
| rlandy | 2022-10-19 23:22:07.639191 | primary | fatal: [undercloud]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange", "unreachable": true} | 12:41 |
| rlandy | 2022-10-19 00:51:30.689350 | primary | TASK [overcloud-ssl : Create overcloud-create-ssl-cert.sh] ********************* | 12:42 |
| rlandy | 2022-10-19 00:51:30.689358 | primary | Wednesday 19 October 2022 00:51:30 -0400 (0:00:07.088) 0:11:22.300 ***** | 12:42 |
| rlandy | 2022-10-19 00:52:14.879502 | primary | fatal: [undercloud]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange", "unreachable": true} | 12:42 |
| rlandy | ysandeep|mtg: bhagyashris|ruck: so ysandeep|mtg is correct here | 12:43 |
| rlandy | dns is not the issue | 12:43 |
| rlandy | bhagyashris|ruck: sorry for the work ... there is another problem | 12:44 |
| bhagyashris|ruck | rlandy, np | 12:49 |
| rlandy | bhagyashris|ruck: I'm going to try rekick both baremetals | 12:49 |
| rlandy | let's see if we can get a consistent failure here or nor | 12:49 |
| rlandy | not | 12:49 |
| bhagyashris|ruck | rlandy, ack | 12:50 |
| frenzyfriday|rover | hey, pls add to your review lists https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45747 (master sc10 kvm vexx in criteria) | 12:57 |
| reviewbot | I have added your review to the Review list | 12:57 |
| *** amoralej|lunch is now known as amoralej | 13:09 | |
| *** dasm|off is now known as dasm | 13:13 | |
| dasm | o/ | 13:13 |
| bhagyashris|ruck | marios, dasm hey should we meet for rr handover at 15 UTC (after PTG) | 13:16 |
| bhagyashris|ruck | rlandy, ^ | 13:16 |
| marios | bhagyashris|ruck: fine for me | 13:18 |
| marios | bhagyashris|ruck: if we finish in time - otherwise we can do it tomorrow morning | 13:18 |
| bhagyashris|ruck | marios, ok | 13:20 |
| marios | bhagyashris|ruck: there is company call in 1.5 hours ish | 13:22 |
| dasm | bhagyashris|ruck: ack | 13:26 |
| dasm | marios: right, company meeting | 13:26 |
| dasm | arxcruz: hey! I'm sorry, I didn't want to offend you. I appologize for my comment. Marios hit the nail on the head though: you're the only one who's actually involved into containers code, hence no one else is able to help you with changes. | 13:27 |
| arxcruz | das | 13:28 |
| arxcruz | dasm still, it's not the kind of comment i would expect in gerrit, but i would like to move on of this subject so, all good | 13:29 |
| dasm | ack | 13:29 |
| *** ysandeep|mtg is now known as ysandeep | 13:44 | |
| chandankumar | ysandeep: ykarel thank you :-) | 13:45 |
| rlandy | bhagyashris|ruck: frenzyfriday|rover; marios: by some miracle OVB stacks are starting now on downstream | 14:00 |
| marios | rlandy: finally thanks | 14:00 |
| rlandy | idk how - the ticket was never updated | 14:00 |
| *** dviroel is now known as dviroel|ptg | 14:08 | |
| marios | rlandy: well perhaps the dns issue was resolved? but the container thing was also not consistent so maybe we got lucky and have containers now ... lets see how far ovb gets :) | 14:10 |
| rlandy | container push failed on 17.1 | 14:10 |
| rlandy | I am rerunning 16.2 jobs that failed | 14:11 |
| rlandy | then will rekick 17.1 | 14:11 |
| rlandy | trying to see what shakes out today | 14:11 |
| rlandy | but if we can clean up before rr take over, that would be nice | 14:11 |
| rlandy | container push may still be an issue | 14:12 |
| marios | rlandy: yeah that conatiner issue was on/off not consistent (remember you closed the ticket at one point and then we saw it again) | 14:12 |
| *** ysandeep is now known as ysandeep|dinner | 14:29 | |
| rlandy | frenzyfriday|rover: https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/2bcd4ce/job-output.txt | 14:34 |
| rlandy | same error showed up on internal | 14:34 |
| marios | rlandy: frenzyfriday|rover: is that a new thing? looking at the error wonder if this is related (is it only on wallaby?) https://review.opendev.org/c/openstack/tripleo-image-elements/+/855840 | 14:36 |
| marios | rlandy: i merged that yesterday ^^ | 14:36 |
| rlandy | marios: let me check | 14:38 |
| marios | think it might be | 14:38 |
| rlandy | I saw it on the downstream jobs | 14:38 |
| rlandy | and then checked upstream | 14:38 |
| rlandy | I think I saw it on wallaby c9 yesterday | 14:39 |
| rlandy | I didn;t check today | 14:39 |
| frenzyfriday|rover | I can see it from 2022-10-19 17:21:57 | 14:40 |
| rlandy | marios: it's in c9 ... https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/15798ef/job-output.txt | 14:40 |
| rlandy | master was fine yesterday | 14:40 |
| rlandy | can we revert the merge? | 14:40 |
| marios | rlandy: yeah the master for that patch merged in august, that is why i thought it was 'safe' and didn't request tests ... :/ silly me like we haven't learnt our lesson with these kinds of patches | 14:41 |
| rlandy | marios: well - maybe we are missing a companion patch to make it work | 14:41 |
| marios | frenzyfriday|rover: want to do it or should I (click revert on the patch and you can try a testproject? ) | 14:41 |
| marios | rlandy: yeah maybe | 14:41 |
| rlandy | can you link the patch? | 14:41 |
| marios | rlandy: 17:36 < marios> rlandy: frenzyfriday|rover: is that a new thing? looking at the error wonder if this is related (is it only on wallaby?) https://review.opendev.org/c/openstack/tripleo-image-elements/+/855840 | 14:41 |
| rlandy | oh | 14:42 |
| marios | merged Yesterday at 1:03 PM | 14:42 |
| rlandy | https://review.opendev.org/q/topic:wallaby-bootpart - al merged | 14:42 |
| marios | so it seems a likely candidate | 14:42 |
| frenzyfriday|rover | marios, lemme try a tp with the revert | 14:42 |
| rlandy | ok - let me try a revert | 14:42 |
| marios | frenzyfriday|rover: thanks | 14:42 |
| rlandy | frenzyfriday|rover: oh you're already on that? | 14:43 |
| frenzyfriday|rover | rlandy, yeah, revert is created , trying a tp now | 14:44 |
| rlandy | frenzyfriday|rover++ | 14:44 |
| rlandy | frenzyfriday|rover: are you on the hardware prov chat? | 14:44 |
| rlandy | can you ping steve there | 14:44 |
| rlandy | I'll try catch him in my afternoom | 14:44 |
| bhagyashris|ruck | rlandy, ohh thanks :) | 14:47 |
| frenzyfriday|rover | rlandy, the google spaces thing right? yep | 14:48 |
| marios | rlandy: maybe you can sync with dasm a bit during your day? then i will just pickup in the morning and sync with bhagyashris|ruck & frenzyfriday|rover as needed sound good? dasm ok for you? | 14:49 |
| marios | we don't have time for a sync now i think and after the company call is past working day for APAC/EU | 14:49 |
| dasm | marios: sounds good. | 14:49 |
| marios | rlandy: dasm: bhagyashris|ruck: frenzyfriday|rover: sound good? ^^ | 14:49 |
| frenzyfriday|rover | marios, yep, good for me | 14:49 |
| marios | berb | 14:50 |
| marios | brb | 14:50 |
| frenzyfriday|rover | rlandy, revert: https://review.opendev.org/c/openstack/tripleo-image-elements/+/862112 | 14:50 |
| frenzyfriday|rover | i am fixing the merge conflict, else it will not let me put depends on | 14:50 |
| rlandy | frenzyfriday|rover: ack | 14:54 |
| rlandy | marios: sure | 14:54 |
| rlandy | dasm: let's touch base in the afternoon | 14:54 |
| rlandy | marios: dasm: do you want us to start the new hackmd or you want to sort that in the morning? | 14:55 |
| *** ysandeep|dinner is now known as ysandeep | 14:55 | |
| marios | rlandy: dasm: up to you if you start it i will just add there | 14:56 |
| marios | rlandy: i'll check https://hackmd.io/2hB-P772SqyqDs0KKZzZEQ?view | 14:56 |
| marios | (the index) | 14:56 |
| rlandy | ok | 14:56 |
| rlandy | marios: downstream is still not in amazing shape | 14:57 |
| rlandy | trying to clean up a bit | 14:57 |
| rlandy | dasm: we'll have to take stock of that and check in with nhicher later | 14:58 |
| dasm | marios: i'm reviewing your https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45611 it seems to be pretty reddish over there. Some of jobs failing due to "The history output length" | 14:58 |
| marios | thanks rlandy its been bad ~ a week now so lets hope we are on the tail end of the infra issues | 14:58 |
| rlandy | yep - onwards an dupwards | 14:58 |
| dasm | I recall you talked today's morning with someone wrt similar issue. Is it still the case? | 14:59 |
| marios | dasm: ack thanks for looking. so with my sales hat on: the history output length thing is known/existing (commented @ https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45611/2#message-fc804672407c139e2288ee7bd71b3757534af006 ) | 14:59 |
| dasm | checking | 14:59 |
| marios | dasm: so we don't have to block on that (not zed specific) | 14:59 |
| bhagyashris|ruck | rlandy, marios dasm frenzyfriday|rover ok for me | 14:59 |
| marios | dasm: for all the jobs there is at least one passing example, except fs1/network which is failing on that validations bug | 14:59 |
| dasm | marios: ack, thx. I'm gonna add the comment | 15:00 |
| marios | dasm: thx | 15:00 |
| * bhagyashris|ruck leaving for the day ... | 15:00 | |
| dasm | bhagyashris|ruck: o/ | 15:01 |
| rlandy | bhagyashris|ruck: have a good night | 15:01 |
| frenzyfriday|rover | Run build-images.sh is failing in the components, checking | 15:01 |
| frenzyfriday|rover | rlandy, the tp with image mount's revert: https://review.rdoproject.org/r/c/testproject/+/45405 i will check back in some time | 15:20 |
| rlandy | frenzyfriday|rover: thanks - will watch that | 15:20 |
| rlandy | frenzyfriday|rover: marios_: https://review.rdoproject.org/r/c/testproject/+/45405 failed | 15:49 |
| rlandy | rebuild images as well? | 15:49 |
| rlandy | overcloud-hardened-uefi-full.raw | 15:49 |
| * rlandy tries | 15:50 | |
| *** marios_ is now known as marios | 15:52 | |
| marios | yeah you'll need to build the image for the ovb jobs rlandy frenzyfriday|rover | 15:53 |
| rlandy | ack - adding that | 15:53 |
| chandankumar | see ya people! | 16:03 |
| dasm | chandankumar: o/ | 16:03 |
| marios | me off too o/ | 16:03 |
| frenzyfriday|rover | ohh, rlandy thanks for updating the tp | 16:05 |
| *** marios is now known as marios|out | 16:05 | |
| rlandy | dasm: marios|out: frenzyfriday|rover: nice ... https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45747 | 16:06 |
| *** dviroel|ptg is now known as dviroel|lunch\ | 16:06 | |
| *** amoralej is now known as amoralej|off | 16:07 | |
| dasm | yup | 16:15 |
| * ysandeep out, Have a great rest of your day everyone o/ | 16:20 | |
| *** ysandeep is now known as ysandeep|out | 16:20 | |
| dasm | ysandeep|out: o/ | 16:27 |
| *** jpena is now known as jpena|off | 16:39 | |
| *** dviroel|lunch\ is now known as dviroel | 16:53 | |
| * frenzyfriday|rover is leaving for the day | 17:30 | |
| frenzyfriday|rover | rlandy, the component jobs are failing Run build-images.sh I will check and file a bug tomorow | 17:31 |
| rlandy | frenzyfriday|rover: https://review.opendev.org/c/openstack/tripleo-image-elements/+/862112 fails | 17:58 |
| * rlandy check what else was w+'ed | 17:58 | |
| rlandy | we may have to revert the whole series | 18:01 |
| * rlandy creates bug | 18:01 | |
| rlandy | dasm: frenzyfriday|rover: ^^ | 18:01 |
| rlandy | dasm: you around? | 18:28 |
| dasm | rlandy: 'sup? | 18:28 |
| rlandy | dasm: let's rr sync | 18:28 |
| dasm | k | 18:28 |
| rlandy | dasm: https://meet.google.com/hhb-smxy-mix?pli=1&authuser=0 | 18:29 |
| rlandy | dasm: https://hackmd.io/0Lev7RRlRDCj9hiNUKN1zw | 18:30 |
| rlandy | rcastillo: hey - when you have a moment - pls review dasm's patches on review list | 19:08 |
| rlandy | infra changes | 19:08 |
| rcastillo | rlandy: ack I'll have a look | 19:20 |
| rlandy | rcastillo: thanks - dasm will also talk about these patches a tuesday's community call | 19:20 |
| dasm | thx | 19:24 |
| * jm1 out for today, have a nice evening folks :) | 19:55 | |
| rlandy | dasm: making some progress downstream | 22:23 |
| dasm | nice | 22:23 |
| rlandy | 16.2 all passed expect baremtal | 22:23 |
| rlandy | rerunning 17.2 ovb jobs | 22:23 |
| dasm | i passed couple components so we're pretty clear | 22:23 |
| dasm | cs8 train network is almost done | 22:23 |
| dasm | cs8 wallaby is fully clear | 22:24 |
| rlandy | nice | 22:24 |
| rlandy | pls leave notes for marios | 22:24 |
| dasm | i left some already | 22:24 |
| dasm | i'm having issues with cert @downstream with rr script, so that's gonna be my next thing tomorrow | 22:24 |
| rlandy | dasm: sending you pct msg | 22:29 |
| rlandy | ok - it's been 12 hours | 22:30 |
| rlandy | will bbl to check on OVB | 22:30 |
| *** rlandy is now known as rlandy|bbl | 22:30 | |
| dasm | ack | 22:30 |
| rlandy|bbl | will leave notes on hackmd | 22:31 |
| dasm | bhagyashris|ruck: frenzyfriday|rover i created new rr status: https://hackmd.io/wtT4lbOSSeuLcRS2aPTQAQ if you're gonna update anything, it would be nice to do it over there. Thanks | 22:46 |
| * dasm over and out | 22:46 | |
| *** dasm is now known as dasm|off | 22:46 | |
| *** dviroel is now known as dviroel|out | 22:47 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!