| *** rlandy|ruck|bbl is now known as rlandy|out | 00:15 | |
| *** akahat|ruck is now known as akahat | 04:12 | |
| *** yadnesh|away is now known as yadnesh | 04:37 | |
| *** pojadhav|out is now known as pojadhav | 04:48 | |
| *** ysandeep|out is now known as ysandeep|ruck | 05:00 | |
| ysandeep|ruck | akahat, hey o/ do you know if we still need this job where we push to rdo registry | 05:39 |
|---|---|---|
| ysandeep|ruck | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-9-push-master/3708e35/logs/containers-built.log | 05:39 |
| akahat | ysandeep|ruck, no. We don't need this job. We are in process of removing it. | 05:40 |
| ysandeep|ruck | ack, thanks! | 05:40 |
| ysandeep|ruck | As you know container builds are broken currently and rabi proposed a patch https://review.rdoproject.org/r/c/config/+/46139 that will fix ubi-quay one, so I wanted to cross check do we need to fix -centos-9-push-master job or not | 05:42 |
| ysandeep|ruck | akahat, containers on trunk.rdo.registry is not used by promoter/copy to quay script/ or anything right? | 05:43 |
| akahat | ysandeep|ruck, yes. we are using it. | 05:51 |
| ysandeep|ruck | akahat, where? | 05:51 |
| akahat | ysandeep|ruck, https://review.rdoproject.org/r/q/topic:rdo_registry_decomission | 05:52 |
| akahat | ysandeep|ruck, specifically in promoter https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45898/4/ci-scripts/container-push/roles/containers-promote/defaults/main.yml | 05:52 |
| ysandeep|ruck | akahat, okay so till this is fixed we need to fix periodic-tripleo-ci-build-containers-centos-9-push-master | 05:53 |
| akahat | ysandeep|ruck, yes. | 05:54 |
| ysandeep|ruck | ahh depends-on https://review.rdoproject.org/r/c/rdo-jobs/+/45921 have a -1, and needs to be fixed before we can merge things. | 05:54 |
| akahat | ysandeep|ruck, yeah.. will update those patches in some time. | 05:57 |
| ysandeep|ruck | akahat, thanks! | 05:57 |
| akahat | ysandeep|ruck, if we merge promoter and copy-quay patch then we don't need to maintain it. Promoter will not get blocked. | 06:01 |
| akahat | and we also removed those from criteria. | 06:02 |
| akahat | promotion criteria * | 06:02 |
| ysandeep|ruck | akahat, thanks, I will add similiar fix for periodic-tripleo-ci-build-containers-centos-9-push-master for now and so that update your patches in your own time. | 06:03 |
| ysandeep|ruck | so that you can* update | 06:04 |
| akahat | ysandeep|ruck, ok | 06:05 |
| marios | \o | 06:06 |
| ysandeep|ruck | marios, good morning o/ | 06:06 |
| ysandeep|ruck | marios, once you settle down, can we meet for 5 mins for container build issue, we can sync later in our day for other issues. | 06:08 |
| marios | ysandeep|ruck: sure gimme few mins | 06:12 |
| *** soniya29|rover is now known as soniya29 | 06:52 | |
| marios | ysandeep|ruck: looks like it was added by yatin there https://opendev.org/openstack/tripleo-ci/commit/a5f3a2554623ebc71f24e2d1deff1f3a9e57a70d fwiw | 06:56 |
| ykarel | did that broke something now :)? | 06:58 |
| ysandeep|ruck | marios, looks like it was always from source and ykarel added option to install from package keeping default as source. | 07:00 |
| marios | ykarel: o/ another change updated python_interpreter and we have a bug https://bugs.launchpad.net/tripleo/+bug/1996874 | 07:00 |
| ykarel | mmm but libselinux-python is for python2.7 iirc | 07:02 |
| * ykarel reads bug | 07:02 | |
| marios | ykarel: we are unblocking for now with disable_venv if we decide we really want to keep that for container builds we can work it out without the fire | 07:03 |
| marios | ykarel: https://review.rdoproject.org/r/c/config/+/46139/1#message-8dd7b02ced8ab3efb1f2af5ee28c6dfeb850f17f | 07:04 |
| marios | ysandeep|ruck: which were the other reviews you showed me (for the other container-push jobs?) looking for it (can you add those to bug for context please when you get a sec) | 07:05 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 07:05 |
| marios | https://review.rdoproject.org/r/c/rdo-jobs/+/46141 | 07:05 |
| marios | got it ysandeep|ruck ^ | 07:05 |
| ysandeep|ruck | marios, yes that one, its on same topic and already on bug | 07:06 |
| marios | ah sorry missed it then ysandeep|ruck | 07:07 |
| ykarel | marios, ok, if DLRN is being used in those jobs then it's totally fine to use packages | 07:07 |
| marios | ysandeep|ruck: one unkown for me... that patch from rabi merged 16th and doesn't seem to be backported yet (you mentioned rabi wants it on zed)... but then how did the bug hit https://logserver.rdoproject.org/69/41469/36/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-zed/48b2722/logs/build.log :/ | 07:09 |
| ysandeep|ruck | looking | 07:10 |
| marios | thnks for looking ykarel | 07:11 |
| ysandeep|ruck | marios, https://logserver.rdoproject.org/69/41469/36/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-zed/48b2722/zuul-info/inventory.yaml | 07:20 |
| ysandeep|ruck | looks like we are not checking our correct branches for source | 07:20 |
| ysandeep|ruck | marios, http://pastebin.test.redhat.com/1081875 | 07:20 |
| ysandeep|ruck | ~~~ | 07:20 |
| ysandeep|ruck | canonical_name: opendev.org/openstack/python-tripleoclient | 07:20 |
| ysandeep|ruck | checkout: master | 07:20 |
| ysandeep|ruck | ~~~ | 07:20 |
| marios | ysandeep|ruck: yeah looking | 07:21 |
| marios | forgot overrid-checkout ysandeep|ruck | 07:23 |
| marios | adding | 07:23 |
| marios | wow | 07:23 |
| ysandeep|ruck | :) | 07:23 |
| ysandeep|ruck | another reason to use packages then source :D | 07:23 |
| ysandeep|ruck | ykarel, could you please +w this https://review.rdoproject.org/r/c/rdo-jobs/+/46141, i think chandankumar is on PTO and we don't have any second core in this timezone. | 07:26 |
| marios | ysandeep|ruck: https://review.rdoproject.org/r/c/rdo-jobs/+/46142 | 07:27 |
| marios | ykarel: and that one too please :) ^^ | 07:28 |
| marios | ykarel: and are you available to be ruck next week? | 07:28 |
| marios | joke :D | 07:28 |
| ysandeep|ruck | marios, +2ed | 07:29 |
| *** amoralej|off is now known as amoralej | 07:31 | |
| ykarel | marios, ysandeep|ruck done | 07:32 |
| ysandeep|ruck | ykarel++ thanks! | 07:32 |
| marios | thnks ykarel ysandeep|ruck | 07:32 |
| ysandeep|ruck | another bug solved on itself: https://bugs.launchpad.net/tripleo/+bug/1996933 :) - will circle back to find the root cause. | 07:46 |
| marios | best kind of bug | 07:50 |
| ysandeep|ruck | yeah :) I am checking downstream container build issue now, if we can crack that it will be a very good Friday o/ | 07:59 |
| jm1 | happy friday folks :) | 08:19 |
| ysandeep|ruck | hey jm1 happy friday o/ | 08:19 |
| jm1 | chandankumar, frenzy_friday, marios, ysandeep|ruck: i am going to apply changes to our infra this morning. please do not merge patches for rdo-infra/ci-config repo, they wont be applied anyway. thank you! | 08:20 |
| jm1 | frenzy_friday: do you work on incockpit or rrcockpit? | 08:21 |
| ysandeep|ruck | >> please do not merge patches for rdo-infra/ci-config repo - Any patches or just dashboard related? | 08:22 |
| jm1 | ysandeep|ruck: any because all patches will be pulled by our infra | 08:23 |
| ysandeep|ruck | ack, I will ping you if we really needed anything for promotion criteria skip and promote. | 08:25 |
| jm1 | ysandeep|ruck: sure | 08:25 |
| marios | jm1: o/ can you check https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/46047/3#message-6c69e6c5d729a236230abd743291a1586b248a29 when you have a chance | 08:25 |
| marios | i think they have the 'old' data format there | 08:25 |
| jm1 | marios: once i am done with infra, it will be much easier to test. gimme some time | 08:26 |
| marios | ack no fire its on the review list in your own time | 08:26 |
| marios | jm1: ^ | 08:26 |
| jm1 | ysandeep|ruck, marios: do you know what happened to promoter? i am not able to login | 08:44 |
| ysandeep|ruck | I haven't logged in today, let me check | 08:46 |
| ysandeep|ruck | atleast the web interface is working: http://promoter.rdoproject.org/ | 08:47 |
| ysandeep|ruck | jm1, login works for me | 08:48 |
| ysandeep|ruck | jm1, are you using right username, its your kerb | 08:49 |
| jm1 | ysandeep|ruck: are you able to connect via ssh? | 08:50 |
| ysandeep|ruck | yes | 08:50 |
| jm1 | ysandeep|ruck: f2f? | 08:50 |
| marios | jm1: i am also in | 08:52 |
| frenzy_friday | jm1, ack. I am not working on the cockpits | 08:53 |
| * marios back in ~1 hour | 08:54 | |
| jm1 | akahat: hello :) here? | 09:00 |
| jm1 | ysandeep|ruck: thank you ysandeep|ruck :) | 09:08 |
| jm1 | ysandeep|ruck, dasm|off: looks like our promoter is heavily outdated, is not running our ansible playbooks and hasnt received any package update since at least 8 month.. | 09:08 |
| ysandeep|ruck | jm1: ack, maybe we should discuss with akahat and bhagyashris before updating it, or maybe create a new promoter server with ansible enabled. | 09:10 |
| ysandeep|ruck | so that old one can keep running till we have new working fully. | 09:11 |
| jm1 | ysandeep|ruck: no worry, i wont touch that system. it is not in sync with our playbooks since at least 8 month | 09:11 |
| ysandeep|ruck | thanks! you know its friday :D | 09:12 |
| jm1 | ysandeep|ruck: will talk with rlandy about it. keeping it running is a security risk | 09:12 |
| ysandeep|ruck | +1 with your concerns, all our tools should be deployed by our infra playbook and always be in sync with ansible running | 09:13 |
| akahat | jm1, o/ | 10:00 |
| akahat | jm1, ysandeep|ruck I agree with jm1 on updating promoter server. | 10:01 |
| akahat | we need to keep it down at least for few hours to update it. as It's friday we will not going to do it today. :D | 10:01 |
| jm1 | akahat: no worry, hopefully we will have a different path soon | 10:01 |
| jm1 | akahat: do you know if any config on promoter has been altered from our infra playbooks? | 10:02 |
| jm1 | akahat: or is it the same as in the playbooks but just not updated since install time? | 10:02 |
| akahat | jm1, promoter configs are not updated since we use playbooks to configure and deploy. Not updated since installed times. | 10:04 |
| jm1 | akahat: this is good news :) | 10:04 |
| akahat | so we need to test all in one stage server. | 10:05 |
| jm1 | akahat: that is what we are working on right now :D | 10:05 |
| ysandeep|ruck | marios, good news, we figured out the downstream container build issue as well.. it was container-selinux- version which got leaked from rhel9.1 content via osptrunk-deps | 10:06 |
| ysandeep|ruck | we can discuss a possible workaround once rlandy|out is in | 10:06 |
| jm1 | akahat: so if we deploy promoter somewhere else with all credentials, then we can shutdown vexxhost promoter? | 10:06 |
| akahat | jm1, cool. please let me know if you need me. | 10:07 |
| jm1 | akahat: that was a question ;D | 10:07 |
| akahat | jm1, no. We will not going to shutdown promoter server. | 10:07 |
| akahat | on vexxhost. | 10:07 |
| jm1 | akahat: why? if playbooks are all in sync, then spinning up a second instance should be fine (except for different ip) | 10:08 |
| akahat | all the infra changes are not going to affect running promoter server. | 10:08 |
| jm1 | akahat: ? | 10:09 |
| akahat | jm1, call? | 10:09 |
| jm1 | akahat: sure | 10:09 |
| akahat | jm1, https://meet.google.com/deh-yuxv-hcu | 10:10 |
| marios | ysandeep|ruck: nice thanks for update | 10:13 |
| * ysandeep|ruck afk for few mins to take a walk | 10:22 | |
| *** ysandeep|ruck is now known as ysandeep|ruck|afk | 10:22 | |
| jm1 | akahat: thanks for the update on promoter! :) | 10:44 |
| *** ysandeep|ruck|afk is now known as ysandeep|ruck | 10:45 | |
| fdiazbra | hi team! I tried to stop using hiera command and use puppet lookup, but since version 3 of hiera.yaml has been deprecated, I get some warnings and with these warnings, my change cannot pass the gate. Can you give me some advice? | 10:52 |
| marios | fdiazbra: o/ worth asking tkajinam on #tripleo he worked on hiera removal from tripleo/puppet-tripleo stuff | 10:56 |
| marios | fdiazbra: which change? | 10:56 |
| fdiazbra | marios: The change is: https://review.opendev.org/c/openstack/tripleo-ansible/+/861966 | 10:57 |
| marios | fdiazbra: for the undercloud-upgrade job, there was an issue but it seems to be transient (job is green again https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade ) | 10:59 |
| marios | fdiazbra: i see now 2022-11-17 10:40:29.459555 | centos-9-stream | stderr: |- | 11:00 |
| marios | 2022-11-17 10:40:29.459568 | centos-9-stream | [1;33mWarning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 | 11:00 |
| marios | fdiazbra: in molecule job 2022-11-17 10:40:29.459555 | centos-9-stream | stderr: |- | 11:00 |
| fdiazbra | marios: yes this is the warning that makes the fail in the molecule job | 11:01 |
| marios | fdiazbra: ack i pinged tkajinam for you in tripleo | 11:02 |
| fdiazbra | marios: thanks! | 11:02 |
| marios | fdiazbra: he is in apac so kind of end of day might reply later | 11:02 |
| *** rlandy|out is now known as rlandy|rover | 11:02 | |
| rlandy|rover | ysandeep|ruck: hi - want to sync now? | 11:03 |
| ysandeep|ruck | sure | 11:03 |
| ysandeep|ruck | meet.google.com/gqq-axqq-dtw | 11:04 |
| marios | ysandeep|ruck: rlandy|rover: let me know if you need me to join? otherwise i'll carry on | 11:04 |
| frenzy_friday | hey folks, which is the downstream repo where the nodeset labels are defined? I want to check if we have a 9.1 stream label | 11:10 |
| ysandeep|ruck | marios: thanks, we will ping you if needed | 11:11 |
| ysandeep|ruck | rlandy|rover, https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/435653 | 11:14 |
| * marios dropped from reviews call - ping me if you want to discuss reviews | 11:16 | |
| chandankumar | most of the reviews are already reviewed | 11:17 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 11:17 |
| chandankumar | akahat: when you get some time, can you update the rdo trunk removal reviews | 11:17 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 11:17 |
| marios | ack chandankumar | 11:17 |
| akahat | chandankumar, ack | 11:17 |
| *** dviroel|out is now known as dviroel | 11:21 | |
| * dviroel o/ | 11:26 | |
| rlandy|rover | pojadhav: hi - you around | 11:28 |
| rlandy|rover | frenzy_friday: hi | 11:34 |
| rlandy|rover | frenzy_friday: wanted to chat with you about getting going on the rhel 9.1 move | 11:34 |
| pojadhav | rlandy|rover, hey | 11:35 |
| pojadhav | rlandy|rover, yes i am around.. | 11:35 |
| rlandy|rover | pojadhav: we have our 1-1 later but maybe better to meet now? | 11:35 |
| pojadhav | rlandy|rover, sure | 11:35 |
| rlandy|rover | joining 1-1 meeting | 11:35 |
| frenzy_friday | rlandy|rover, yep sure. I am looking for the repo where the labels are defined | 11:40 |
| rlandy|rover | frenzy_friday: sec - talking with pojadhav | 11:41 |
| * pojadhav away for half hour | 12:13 | |
| dviroel | chandankumar: hey, around? | 12:18 |
| chandankumar | dviroel: hello, yes | 12:19 |
| chandankumar | dviroel: anything I can help on ibm cloud | 12:19 |
| chandankumar | dviroel: is the ibm3 creds on bitwarden? | 12:19 |
| dviroel | chandankumar: yes, should be everything there | 12:19 |
| dviroel | chandankumar: yes, some help with ibm3 | 12:19 |
| dviroel | chandankumar: so we merged config changes, and stable4 line run fine | 12:20 |
| chandankumar | ok | 12:20 |
| dviroel | chandankumar: now I see errors in ovb-manage when running on ibm3 | 12:20 |
| dviroel | chandankumar: https://logserver.rdoproject.org/46/46046/3/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train-bm3/2e6d8e3/job-output.txt | 12:21 |
| chandankumar | dviroel: Did we uploaded all the required images on ibm3? | 12:21 |
| dviroel | chandankumar: hum, i don't think so | 12:21 |
| dviroel | chandankumar: we will get there I think, the error seems to happen before that | 12:22 |
| dviroel | chandankumar: " keystoneauth1.exceptions.catalog.EndpointNotFound: public endpoint for orchestration service in regionOne region not found" | 12:22 |
| dviroel | should be one of the first thing to do on the cloud | 12:22 |
| chandankumar | let me add the creds in clouds.yaml and see | 12:22 |
| dviroel | so I suspect on an error on clouds.yaml | 12:22 |
| dviroel | yeah, i can list catalog from my machine | 12:23 |
| ysandeep|ruck | rlandy|rover, https://code.engineering.redhat.com/gerrit/c/testproject/+/211643/168#message-24329242e8ca912c5888b19c9c162eb7b31cf94b all container builds are green if we exclude osptrunk-deps | 12:23 |
| ysandeep|ruck | rlandy|rover, I have update commit msg: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/435653 , ready to go as soon as ci reports back | 12:24 |
| ysandeep|ruck | updated* the commit msg | 12:24 |
| chandankumar | dviroel: heat service is not installed there | 12:28 |
| chandankumar | dviroel: check orchestration is missing from there | 12:28 |
| ysandeep|ruck | reviewbot, please add this to reviewlist: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/435653 | 12:29 |
| reviewbot | I have added your review to the Review list | 12:29 |
| dviroel | chandankumar: oh, you are right | 12:30 |
| dviroel | as the error message is saying :) | 12:30 |
| soniya29 | rlandy|rover, meeting time? | 12:30 |
| dviroel | chandankumar: also missing some images there, bmc and boot | 12:31 |
| *** frenzy_friday is now known as frenzy_friday|lunch | 12:31 | |
| chandankumar | dviroel: you can download the images from ibm2 cloud and upload it there | 12:31 |
| dviroel | chandankumar: ack | 12:31 |
| rlandy|rover | soniya29: give me 5 pls | 12:33 |
| soniya29 | rlandy|rover, sure | 12:33 |
| rlandy|rover | ysandeep|ruck: sorry on 1-1s | 12:34 |
| rlandy|rover | reviewing your patch | 12:34 |
| chandankumar | dviroel: they might missed few patches in the deployment https://github.com/shiftstack/dev-install/pulls | 12:34 |
| chandankumar | dviroel: nicholas might be knowing about that | 12:35 |
| ysandeep|ruck | nw, please take your time, we need to wait for zuul to report back.. but I think we should remove those 16.2 from check in downstream and replace with 17 jobs | 12:35 |
| rlandy|rover | ysandeep|ruck: added my +2 so you can merge when ready | 12:35 |
| ysandeep|ruck | rlandy|rover, thanks! | 12:36 |
| rlandy|rover | soniya29: joining | 12:38 |
| dviroel | chandankumar: ack, will ping nhicher - a little bit earlier there | 12:40 |
| marios | bhagyashris: how is the new 8 on 9 d/stream line looking? | 12:40 |
| dviroel | chandankumar: /me downloading and uploading images | 12:40 |
| marios | bhagyashris: if we are good we can start removing the 8 ? | 12:40 |
| bhagyashris | marios, sure | 12:41 |
| bhagyashris | checking... | 12:41 |
| bhagyashris | marios, removal of 8 patches merged https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/435458 https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/435450 | 12:44 |
| marios | bhagyashris: ok .. should have sanity checked first but lets do it now just in case | 12:45 |
| *** ysandeep|ruck is now known as ysandeep|ruck|afk | 12:46 | |
| bhagyashris | marios, currently 8 on 9 job is running in integration line 17.1 on rhel9 | 12:46 |
| marios | bhagyashris: obviously i know this | 12:47 |
| marios | bhagyashris: we should have checked that this new job/8 on 9 line is OK before we removed the 8 line though | 12:48 |
| marios | bhagyashris: we cannot verify yet | 12:48 |
| marios | bhagyashris: because we don't have a green run from it yet | 12:48 |
| marios | bhagyashris: patch merged 16th | 12:48 |
| marios | bhagyashris: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1&project=openstack/tripleo-ci | 12:48 |
| marios | bhagyashris: and on 16th the job is failingon a known bug. next run should be OK | 12:49 |
| bhagyashris | marios, yeah if it failed i will revert the patches because now the changes merged so better i will monitor the line and if it failed i will revert the change | 12:49 |
| marios | bhagyashris: the 17 skipped cos containers failed | 12:49 |
| marios | bhagyashris: NO STOP | 12:49 |
| marios | bhagyashris: do not revert | 12:49 |
| marios | bhagyashris: i am saying we | 12:49 |
| marios | phone? | 12:49 |
| marios | bhagyashris: starting meet | 12:49 |
| bhagyashris | marios, https://meet.google.com/ywr-jeat-cpa | 12:50 |
| *** dviroel_ is now known as dviroel | 12:57 | |
| marios | bhagyashris: https://code.engineering.redhat.com/gerrit/c/testproject/+/434544/7#message-92758ebba3105cf896ffc92d0c380a80a9a09b5e | 12:57 |
| jm1 | marios: is the dev who submitted this dashboard patch here on irc? https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/46047 | 13:04 |
| rlandy|rover | pojadhav: ok - let's check ... | 13:05 |
| rlandy|rover | joining 1-1 time again | 13:05 |
| marios | jm1: i don't know who they are | 13:06 |
| jm1 | marios: ok thanks, maybe frenzy_friday|lunch know | 13:07 |
| jm1 | *s | 13:07 |
| chandankumar | rlandy|rover: dviroel ysandeep|ruck|afk please have a look https://github.com/tripleo-ci/release/pull/1 if everything looks ok, then I can send the pr to openshift/release | 13:08 |
| dviroel | chandankumar: ack, will do | 13:09 |
| *** amoralej is now known as amoralej|lunch | 13:11 | |
| jm1 | akahat, dasm|off, frenzy_friday|lunch, marios, ysandeep|ruck|afk: infra patches have been merged and infra is still up and running :D https://review.rdoproject.org/r/q/topic:ansible-inventory | 13:12 |
| jm1 | akahat, dasm|off, frenzy_friday|lunch, marios, ysandeep|ruck|afk: this means we can now deploy staging envs | 13:12 |
| rlandy|rover | pojadhav: ok - sarah added me as admin to the osp planning board, I can create sprints nwow | 13:13 |
| pojadhav | rlandy|rover, ack | 13:13 |
| *** yadnesh is now known as yadnesh|away | 13:13 | |
| rlandy|rover | I asked her to add you and dariusz | 13:14 |
| rlandy|rover | not sure if she will | 13:14 |
| rlandy|rover | let's see | 13:14 |
| pojadhav | rlandy|rover, thanks | 13:14 |
| rlandy|rover | pojadhav: ok - now you and dasm|off both have permissions | 13:16 |
| pojadhav | rlandy|rover, great thanks! | 13:16 |
| marios | pojadhav: dasm|off: rlandy|rover: was just asked by migarcia re jira for 17.1/rhel9.1 - pooja are you going to creat this? | 13:21 |
| marios | in our scrum board i mean pojadhav | 13:21 |
| pojadhav | rlandy|rover, dasm|off : I have rolled the new sprint 71. | 13:21 |
| marios | or rlandy|rover do we want to creat that with the 'new ' process | 13:21 |
| marios | rlandy|rover: probably this ^^ | 13:21 |
| rlandy|rover | marios: pojadhav or dasm|off will create the card | 13:22 |
| marios | pojadhav: rlandy|rover: i told migarcia to ask again next week anyway when we create the story | 13:22 |
| rlandy|rover | frenzy_friday|lunch is already working on it | 13:22 |
| pojadhav | rlandy|rover, marios : new sprint created now | 13:22 |
| * rlandy|rover will talk with migarcia | 13:22 | |
| marios | thx rlandy|rover | 13:25 |
| *** ysandeep|ruck|afk is now known as ysandeep|ruck | 13:30 | |
| jm1 | frenzy_friday|lunch: wanna sync on dev env when you are back? | 13:30 |
| rlandy|rover | ysandeep|ruck: patch is finished testing - o to merge? | 13:30 |
| ysandeep|ruck | rlandy|rover, yes | 13:31 |
| rlandy|rover | ok - wil do | 13:31 |
| ysandeep|ruck | rlandy|rover, podified mtg | 13:31 |
| *** frenzy_friday|lunch is now known as frenzy_friday | 13:32 | |
| frenzy_friday | rlandy|rover, marios : hackmd for 9.1: https://hackmd.io/0ZqodGQsTACAG7ptetSrIg I'll put it on the jira card | 13:32 |
| marios | thx frenzy_friday | 13:33 |
| pojadhav | rlandy|rover, dasm|off : I have created new column "BLOCKED" to the sprint board. | 13:53 |
| rlandy|rover | pojadhav: can't open our board now | 13:53 |
| rlandy|rover | can you remove that | 13:53 |
| rlandy|rover | we're editing the epics | 13:53 |
| pojadhav | ok | 13:53 |
| pojadhav | rlandy|rover, removed | 13:54 |
| *** pojadhav is now known as pojadhav|dinner | 13:57 | |
| *** dasm|off is now known as dasm | 14:09 | |
| dasm | o/ | 14:09 |
| *** pojadhav|dinner is now known as pojadhav | 14:11 | |
| marios | dasm: \o welcome to friday | 14:12 |
| dasm | marios: \o/ | 14:13 |
| marios | dasm: i know we have a call scheduled for upgrades... i'd like to also discuss https://trello.com/c/IryyjbGs i told alan we'd try and test the new package.. looks like you have some history there can we go through together (found your patches) | 14:13 |
| dasm | marios: first friday of this week! | 14:13 |
| marios | dasm: either we can do in our call or if you are available we can meet earlier too in 15 mins or more if you need | 14:13 |
| dasm | marios: atm following podified meeting. it should be done in 15. we could meet then | 14:13 |
| marios | dasm: always my favourite 16:13 < dasm> marios: first friday of this week! | 14:13 |
| marios | dasm: ok cool ping me when available thanks (but also fine if its early you want to catchup and we talk in normal call in 1 hour ish) | 14:14 |
| *** amoralej|lunch is now known as amoralej | 14:18 | |
| dasm | marios: we can do it earlier | 14:20 |
| dasm | marios: ping | 14:29 |
| dasm | marios: https://meet.google.com/zpb-uqdv-mmu?authuser=0 | 14:30 |
| marios | dasm: joining gimme 1 mins | 14:31 |
| rlandy|rover | ysandeep|ruck: your ds patch merged | 14:32 |
| rlandy|rover | going to rekick 17.1 line | 14:32 |
| dasm | rlandy|rover: i had to add "Workflow" to newly created subtask ( dviroel's one) | 14:32 |
| dasm | to make it visible | 14:32 |
| rlandy|rover | dasm: looking | 14:33 |
| dasm | rlandy|rover: now Task OSP-20297 has subtask visible on the board | 14:33 |
| dasm | rlandy|rover: this option is preseit under Edit/Contributors/Workstream | 14:33 |
| rlandy|rover | dasm: workflow or workstream? | 14:34 |
| rlandy|rover | workstream - agreed | 14:34 |
| rlandy|rover | I though tripleoCI was added | 14:34 |
| rlandy|rover | but if that's what it takes, great | 14:34 |
| rlandy|rover | dasm++ | 14:34 |
| ysandeep|ruck | rlandy|rover, thanks! | 14:34 |
| ysandeep|ruck | rlandy|rover, I am out soon, the only pending bug is one from Cedric on which he is still working. | 14:35 |
| rlandy|rover | anyays - cool that works | 14:35 |
| rlandy|rover | ysandeep|ruck++ awesome | 14:35 |
| rlandy|rover | will work on promoting some lines | 14:35 |
| rlandy|rover | ysandeep|ruck: can you check components when you come in on monday | 14:35 |
| ysandeep|ruck | rlandy|rover, yes I will | 14:35 |
| rlandy|rover | ysandeep|ruck: just fyi thursday is a holiday here in USA - but I will work that day to complete the shift | 14:35 |
| dviroel | dasm: workstream, ack, thanks. probably because the story also have that set | 14:36 |
| dasm | yes | 14:36 |
| ysandeep|ruck | rlandy|rover, :(, I would say take a day off, I can stretch for few more hours to cover for few more hours and if something is really burning I can probably request doug/dariusz/rafael to take care of that. | 14:41 |
| rlandy|rover | ysandeep|ruck: let's see how wednesday looks | 14:41 |
| rlandy|rover | ysandeep|ruck: have a good weekend | 14:41 |
| ysandeep|ruck | you too | 14:41 |
| *** ysandeep|ruck is now known as ysandeep|out | 14:42 | |
| * jm1 bbl | 14:43 | |
| frenzy_friday | jm1 thanks for going through and debugging the infra set up! | 14:43 |
| frenzy_friday | jm1, dasm : patch for newer nginx: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/46063 | 14:44 |
| rlandy|rover | frenzy_friday: sorry didn't get a chance to chat with you earlier | 14:49 |
| rlandy|rover | frenzy_friday: you all set with rhel 9.1 stuff? | 14:49 |
| rlandy|rover | ping if you need help getting started | 14:50 |
| frenzy_friday | rlandy|rover, hey, can we have a quick call? | 14:50 |
| frenzy_friday | https://meet.google.com/sij-eqhm-pnm | 14:50 |
| rlandy|rover | sure | 14:53 |
| rlandy|rover | joining | 14:53 |
| frenzy_friday | rlandy|rover, https://code.engineering.redhat.com/gerrit/q/topic:rhel-17.1-on-9.1 | 14:54 |
| dasm | https://bugs.launchpad.net/tripleo/+bug/1960355/ | 15:04 |
| marios | arxcruz: o/ please help we are looking at https://trello.com/c/IryyjbGs - question... where can i install python-oauthlib into tempest venv before execution we need to test a specific package for a fix | 15:05 |
| marios | arxcruz: can you please let me or dasm know | 15:05 |
| *** dviroel is now known as dviroel|lunch | 15:07 | |
| arxcruz | marios: i'll take a look, but i don't think we use venv to run tempest | 15:44 |
| marios | arxcruz: ok then where would be the right place if there is one? | 15:44 |
| arxcruz | https://trello.com/c/IryyjbGs/2344-cixlp1960355tripleociproa-tempest-tests-testoauth1tokensoauth1tokenstest-are-failing-on-tripleo-ci-centos-9-ovb-3ctlr1comp-featu#comment-63725919bfee4e007181ec39 | 15:45 |
| arxcruz | need to make a test with this rpm | 15:45 |
| arxcruz | maybe run a testproject, hold node, install it and run manually | 15:45 |
| marios | arxcruz: yeah that was going to be my plan for monday ^^ dasm | 15:46 |
| marios | arxcruz: was hoping there was a 'tempest_packages' type of thing we could use | 15:46 |
| arxcruz | marios: i think there is, let me check | 15:46 |
| arxcruz | marios: you can use the tempest.yml and add a install task to test it | 15:47 |
| arxcruz | marios: anywhere before https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/tasks/tempest.yml#L275 | 15:47 |
| marios | arxcruz: ack thanks... the complication was i assumed it was venv and i had to go dig at tempest but yeah if it is host packages then ^^^ should do it | 15:48 |
| rlandy|rover | ha - no tempest tests are running on zed | 15:48 |
| marios | rlandy|rover: ? where i mean pretty sure we've seen them on zed before | 15:49 |
| marios | rlandy|rover: is it everywhere? | 15:49 |
| marios | rlandy|rover: someting happen? | 15:49 |
| rlandy|rover | marios: sc002 | 15:49 |
| rlandy|rover | others do have some | 15:49 |
| rlandy|rover | https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-zed/34afb21/logs/undercloud/var/log/tempest/stestr_results.html.gz | 15:50 |
| rlandy|rover | failed but ran some | 15:50 |
| marios | rlandy|rover: maybe something in tempest skiplist? | 15:50 |
| rlandy|rover | yeah | 15:50 |
| rlandy|rover | arxcruz: ^^? | 15:50 |
| rlandy|rover | are we missing a release in the allow list? | 15:50 |
| marios | dasm: if you have time to try a explicit install task @ 17:47 < arxcruz> marios: anywhere before https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/tasks/tempest.yml#L275 | 15:51 |
| marios | dasm: otherwise i'll be checking on monday - maybe drop me a note of you do so i don't duplicate? | 15:51 |
| arxcruz | rlandy|rover: this seems to be an issue with some service | 15:51 |
| marios | please thanks | 15:51 |
| arxcruz | rlandy|rover: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.24.3', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd0f62cf8b0>: Failed to establish a new connection: [Errno 113] No route to host')) | 15:51 |
| marios | arxcruz: no did something merge recently in skiplist that you know of which caused tests to stop running for fs2 is what rlandy|rover and i were talking about mainly | 15:52 |
| marios | err scenario2 | 15:52 |
| marios | rlandy|rover: right? ^ | 15:52 |
| arxcruz | marios: do you have logs ? | 15:52 |
| marios | i do not rlandy|rover was asking about that | 15:52 |
| rlandy|rover | sorry - posting | 15:53 |
| rlandy|rover | https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario002-standalone-zed/4828a7b/logs/undercloud/var/log/tempest/stestr_results.html.gz | 15:53 |
| marios | internal server error? https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario002-standalone-zed/4828a7b/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz | 15:54 |
| arxcruz | checking | 15:54 |
| arxcruz | marios: rlandy|rover so, i can confirm it's nothing related to skiplist, since we have the skipped and allowed list working properly | 15:55 |
| arxcruz | having said that, it seems to be related to service failure | 15:56 |
| arxcruz | checking | 15:56 |
| arxcruz | https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario002-standalone-zed/4828a7b/logs/undercloud/home/zuul/tempest/etc/ | 15:56 |
| arxcruz | excludelist and includelist is there, so not related to skiplist | 15:56 |
| marios | arxcruz: yeah internal server error wouldn't be skiplist.. i was wondering that before i saw those logs | 15:56 |
| arxcruz | marios: seems to be related to mysql | 15:58 |
| arxcruz | or the connection between keystone - nova - mysql | 15:59 |
| arxcruz | marios: rlandy|rover https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario002-standalone-zed/4828a7b/logs/undercloud/var/log/containers/keystone/keystone.log.txt.gz | 15:59 |
| arxcruz | 2022-11-18 14:54:30.794 23 ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: nova. | 15:59 |
| arxcruz | and if you check nova logs: | 15:59 |
| rlandy|rover | hmmm | 16:00 |
| arxcruz | 2022-11-18 15:09:53.505 2 ERROR oslo_service.periodic_task oslo_messaging.rpc.client.RemoteError: Remote error: DBConnectionError (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') | 16:00 |
| arxcruz | https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario002-standalone-zed/4828a7b/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz | 16:00 |
| arxcruz | i mean, actually, not related to nova it self, i see keystone wasn't able to find other services as well | 16:01 |
| arxcruz | i'm not seeing errors in mysql itself | 16:01 |
| arxcruz | so it might be the connection | 16:01 |
| arxcruz | haproxy | 16:02 |
| arxcruz | although i'm not seeing haproxy errors | 16:02 |
| arxcruz | so, the error is because keystone are not being able to communicate/find the other services, and that's because the services can't talk with mysql | 16:03 |
| rlandy|rover | arxcruz: ok - there are a few other zed failures | 16:05 |
| rlandy|rover | looking at those | 16:06 |
| dasm | marios: sure. will see what i can do | 16:09 |
| *** dviroel|lunch is now known as dviroel | 16:16 | |
| dasm | marios: wrt https://bugs.launchpad.net/tripleo/+bug/1960355 | 16:22 |
| dasm | i'm not sure if we can install it for tempest, because the error was inside containers/httpd | 16:22 |
| dasm | which sounds like we're installing that as a part of httpd container | 16:22 |
| marios | ack dasm thanks for looking ... ill revisit monday | 16:25 |
| * marios off in few mins | 16:26 | |
| *** marios is now known as marios|out | 16:35 | |
| dasm | marios|out: i left you comment on CIX card | 16:40 |
| rlandy|rover | lunch - brb | 17:01 |
| dviroel | dasm: previous podified story is now moved to done. Had to create new sub tasks on OSP project, since Jira doesn't allow change parents between projects | 17:25 |
| dviroel | dasm: created a new story for ibm cloud | 17:26 |
| dviroel | still need to add more sub-tasks for ibm | 17:26 |
| dasm | dviroel++ | 17:29 |
| *** amoralej is now known as amoralej|off | 17:36 | |
| jm1 | dasm: hello :) time for a chat? | 18:21 |
| dasm | jm1: if you want to | 18:21 |
| jm1 | dasm: no :P | 18:21 |
| dasm | jm1: let's do that Monday then | 18:21 |
| jm1 | dasm: you dont even want a status update in text form? | 18:22 |
| dasm | jm1: i've seen your patches. i've assumed everything went fine | 18:22 |
| jm1 | dasm: partially. but ok we can talk on monday | 18:29 |
| dasm | jm1: just enjoy your weekend. | 18:29 |
| rlandy|rover | dasm: hey - do we need to meet re: the board? or we all set? | 18:43 |
| rlandy|rover | dasm: if you have a few we should meet quickly re: tc/ua | 18:43 |
| dasm | rlandy|rover: we might if you want to. | 18:43 |
| rlandy|rover | ok | 18:44 |
| rlandy|rover | dasm: https://meet.google.com/nqw-xpkv-twe?pli=1&authuser=0 | 18:45 |
| * dviroel going afk | 20:10 | |
| dviroel | have a great weekend team | 20:11 |
| rlandy|rover | enjoy | 20:11 |
| *** dviroel is now known as dviroel|out | 20:13 | |
| jm1 | rlandy|rover, dasm: incockpit is now also in our staging env. its needs some more love like all other hosts as well but its a start :D https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/46150 | 20:27 |
| dasm | jm1: sweet! | 20:28 |
| jm1 | rlandy|rover: once frenzy_friday has fixed nginx and we requested and got assigned a dns hostname for one of those machines at rhos-dev-stage we can actually shutdown our downstream cockpit on bm | 20:29 |
| rlandy|rover | jm1: nice | 20:43 |
| rlandy|rover | awesome work | 20:43 |
| rlandy|rover | jm1: great to hear we shutdown the bm | 20:43 |
| jm1 | rlandy|rover: not yet! first we need to work out some issues with nginx, outdated packages and get that dns hostname | 20:44 |
| jm1 | rlandy|rover: but it will definitely be possible in q1 | 20:45 |
| rlandy|rover | perfect | 20:45 |
| rlandy|rover | thanks for picking that up | 20:45 |
| jm1 | i am out for today :) | 20:46 |
| * jm1 have a nice evening #oooq :) | 20:46 | |
| jm1 | ah, and weekend 🙈 | 20:46 |
| rlandy|rover | have a good weekend!! | 20:48 |
| dasm | jm1[m]: o/ | 20:58 |
| * dasm => brb | 20:58 | |
| rlandy|rover | I'm out - see you monday | 21:03 |
| dasm | bye | 22:27 |
| *** dasm is now known as dasm|off | 22:49 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!