| * rcastillo signing out for the day | 01:33 | |
| *** pojadhav- is now known as pojadhav | 05:39 | |
| *** ysandeep|out is now known as ysandeep|rover | 06:04 | |
| *** jm1|ruck is now known as jm1|rover | 06:48 | |
| Tengu | hello there! is there any known issue with the CI and package installation from centos mirrors? I'm hitting "Curl error (6): Couldn't resolve host name for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [getaddrinfo() thread failed to start]" | 07:02 |
|---|---|---|
| Tengu | in, at least: https://zuul.opendev.org/t/openstack/build/1561481e9fb24bf6b06518ce4f5abd9b | 07:03 |
| Tengu | hit it yesterday already, and the recheck I launched today seems to fail as well. sounds weird though, since another molecule job is working just fine. | 07:03 |
| Tengu | though it's not launched within a container, apparently. | 07:03 |
| ysandeep|rover | Tengu, https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http worked for me locally | 07:12 |
| Tengu | ysandeep|rover: same here... | 07:12 |
| Tengu | weird. let's wait for the actual fail and re-retry | 07:12 |
| ysandeep|rover | Tengu, in tripleo-ci based jobs.. instead of using centos.org mirror we use the local mirror where instance is spawned | 07:13 |
| ysandeep|rover | for ex. https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1e4/842250/1/check/tripleo-ci-centos-9-content-provider/1e42f95/logs/undercloud/etc/yum.repos.d/quickstart-centos-base.repo | 07:13 |
| Tengu | no idea where that repo comes from though | 07:13 |
| Tengu | it's in molecule | 07:13 |
| ysandeep|rover | I wonder if we can do same in molecule | 07:14 |
| Tengu | more than probably | 07:14 |
| Tengu | is that repo file available from some network location? | 07:14 |
| Tengu | if so, we may modify the Dockerfile to actually fetch it. | 07:15 |
| Tengu | though it seems generated by oooq - but there are params for that I guess | 07:15 |
| ysandeep|rover | https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-9/promotion-testing-hash-master.yml#L137 | 07:16 |
| ysandeep|rover | baseurl: ${NODEPOOL_CENTOS_MIRROR}/9-stream/BaseOS/x86_64/os/ | 07:16 |
| Tengu | hmmm ok. | 07:16 |
| Tengu | though..... that must be an env var passed to the Dockerfile during the image build, for molecule. | 07:17 |
| Tengu | guess we may be able to do something. | 07:17 |
| ysandeep|rover | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/nodepool-setup/templates/mirror_info.sh.j2#L54 | 07:18 |
| ysandeep|rover | export NODEPOOL_CENTOS_MIRROR=${NODEPOOL_CENTOS_MIRROR:-http://$NODEPOOL_MIRROR_HOST/centos} | 07:19 |
| Tengu | yep | 07:19 |
| Tengu | and export NODEPOOL_MIRROR_HOST=${NODEPOOL_MIRROR_HOST:-mirror.mtl01.inap.opendev.org} | 07:19 |
| Tengu | that sets the default. | 07:19 |
| ysandeep|rover | yeah, +1 to the general idea of using local mirror where molecule job is spawned - will avoid this kind of network/mirror related issues. | 07:21 |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 07:24 | |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 08:38 | |
| ysandeep|rover | jm1: hey fyi o/ if you just want to rerun rdo third party jobs you can post check-rdo instead of recheck | 08:40 |
| jm1 | ysandeep|rover: ack, thx :) | 08:41 |
| ysandeep|rover | jm1, left a comment: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/842218/1#message-ab7ed550a2dff941405cc544725e5b652181f2f2 | 08:48 |
| jm1 | ysandeep|rover: i am not sure where to define undercloud_undercloud_timezone. maybe in tripleo-environments/config/environments/*? | 08:52 |
| ysandeep|rover | config/environments/ovb-common.yml - should have helped - from file name looks like it should be used by all ovb jobs.. /me can take a look later | 08:57 |
| ysandeep|rover | jm1: https://logserver.rdoproject.org/18/842218/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/5331c69/logs/undercloud/home/zuul/workspace/logs/playbook_executions.log.txt.gz quickstart was triggered in following way: https://logserver.rdoproject.org/18/842218/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/5331c69/logs/undercloud/home/zuul/workspace/logs/playbook_executions.log.txt.gz | 09:10 |
| ysandeep|rover | so i.e add in one of those files.. or maybe in this case we can just add in roles/undercloud-deploy defaults | 09:12 |
| jbadiapa | Hi, I need a little bit of help with a molecule job. | 09:13 |
| jbadiapa | I added a molecule job at https://review.opendev.org/c/openstack/tripleo-ansible/+/836845 but it seems that it's not been executed and I can not find why | 09:14 |
| ysandeep|rover | jbadiapa, hey o/ , let me tag chandankumar, he was working lately with molecule jobs and might know the issue from top of his head. | 09:17 |
| ysandeep|rover | chandankumar, ^^ if you can take a look please | 09:18 |
| jbadiapa | ysandeep|rover, chandankumar: thx | 09:18 |
| chandankumar | on it | 09:18 |
| chandankumar | jbadiapa: few nits, rest looks good | 09:25 |
| jm1 | ysandeep|rover: ack, added it to role defaults since it is helpful for all ci jobs. testproject is rerunning | 09:30 |
| jbadiapa | chandankumar: thx a lot | 09:30 |
| jm1 | ysandeep|rover: i am working on ci quality degraded cards right now. anything else i should have a look at? | 09:31 |
| ysandeep|rover | jm1: thanks, I think config/environments/ovb-common.yml and config/environments/rdocloud.yml can be dropped now - now that its in role default. | 09:31 |
| jm1 | ysandeep|rover: ack | 09:32 |
| ysandeep|rover | jm1: fyi.. we have a new bug we will need i.e security or df on that.. https://bugs.launchpad.net/tripleo/+bug/1973863 | 09:33 |
| ysandeep|rover | Tengu: https://bugs.launchpad.net/tripleo/+bug/1973863 is df or security? | 09:34 |
| Tengu | ysandeep|rover: I'd go first with Security, that's their service - and DF may help | 09:37 |
| ysandeep|rover | Tengu, ack.. I was wondering if recent https://github.com/openstack/tripleo-heat-templates/commit/ccf8e0c33db2b36d6711d26e8f3abf038dcb866b is anyhow related. | 09:37 |
| Tengu | I don't think it is | 09:37 |
| ysandeep|rover | Tengu, ack | 09:37 |
| Tengu | I may be wrong of course. | 09:38 |
| Tengu | but... after looking at the change, I don't think it's related. | 09:38 |
| Tengu | oh. wait. hmmm. | 09:38 |
| jbadiapa | chandankumar: do you might know why one of the molecule jobs is not executed? | 09:39 |
| Tengu | tripleo_keystone_enable_internal_tls: {get_param: EnableInternalTLS} | 09:39 |
| Tengu | wondering if that one isn't a single string, not a list. | 09:39 |
| Tengu | ah, wait. that's a bool. | 09:40 |
| ysandeep|rover | only fs035/64 failing - where we test with ssl enabled | 09:40 |
| Tengu | keystone_volumes + tripleo_keystone_internal_tls_ca_file guess we want to append. | 09:40 |
| Tengu | not concat. | 09:40 |
| Tengu | ysandeep|rover: yeah. that's more than probably it | 09:41 |
| Tengu | tripleo_ansible/roles/tripleo_keystone/defaults/main.yml:tripleo_keystone_internal_tls_ca_file: /etc/ipa/ca.crt | 09:41 |
| Tengu | bingo | 09:41 |
| Tengu | now.... shall we accept a list, or not, that's a question for Security. | 09:42 |
| Tengu | but that's the issue. Not directly the patch you linked, but an actual bug in the standalone role :) | 09:42 |
| ysandeep|rover | Tengu, Is it possible to left a comment on bug, I can then ping someone from security. | 09:43 |
| Tengu | ysandeep|rover: lemme ping them | 09:43 |
| Tengu | on it! | 09:43 |
| ysandeep|rover | Tengu, great - directly is better | 09:43 |
| *** jpena|off is now known as jpena | 09:46 | |
| ysandeep|rover | folks hey o/ security component failed because of tripleo component bug, because we include delorean-current - I wonder if its a good idea to include delorean-current in component line jobs. | 09:46 |
| jm1 | ysandeep|rover: should we mark lp bug #1960661 as a duplicate of lp bug #1960310? both are failing on the same tempest test | 09:47 |
| ysandeep|rover | imho... Atleast in component line jobs we should not include delorean-current | 09:47 |
| ysandeep|rover | jm1: looking | 09:47 |
| chandankumar | jbadiapa: can you try changing the verifier to testinfra https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/backup_and_restore/molecule/default/molecule.yml#L28 | 09:48 |
| chandankumar | that is the only difference I am seeing with other molecule file | 09:49 |
| * chandankumar reads more about the verifier | 09:49 | |
| ysandeep|rover | jm1, ussuri is eol and gone for us.. I would just close ussuri bugs and cix cards if any | 09:49 |
| Tengu | ysandeep|rover: added a comment in the meanwhile, apparently it should be a string. Getting some insurance, and I'll push a patch. | 09:49 |
| jm1 | ysandeep|rover: it affects train and victoria as well | 09:50 |
| ysandeep|rover | yeah, I should have read the whole bug first.. in 1st comment only ussuri was mentioned | 09:50 |
| jm1 | ysandeep|rover: i will just add a comment on cix card and on lp bug #1960661 that it might be a duplicate | 09:51 |
| ysandeep|rover | jm1: sounds good, thanks! | 09:54 |
| Tengu | LP should be under control shortly. | 09:54 |
| Tengu | sorry, I didn't really pay attention when you mentioned it on the internal channel, ysandeep|rover. I was, again, neck-deep in nftables. | 09:55 |
| ysandeep|rover | Tengu: thanks! let me run a testproject to confirm it solves the issue | 09:55 |
| Tengu | ysandeep|rover: sounds good! | 09:55 |
| Tengu | you have the link on #tripleo, but: https://review.opendev.org/c/openstack/tripleo-ansible/+/842303 | 09:55 |
| ysandeep|rover | Tengu++ | 09:55 |
| jbadiapa | chandaumar, i'll do that thx | 09:59 |
| chandankumar | jbadiapa: if that does not work, then will ping cloudnull on #tripleo , he will be around in few hours | 10:00 |
| *** rlandy|out is now known as rlandy | 10:22 | |
| rlandy | ysandeep|rover: jm1: hi - how are things today? | 10:25 |
| rlandy | want to sync? | 10:25 |
| jm1 | rlandy: ack | 10:26 |
| rlandy | jm1: ysandeep|rover: when you are ready ... https://meet.google.com/yht-emav-bva?pli=1&authuser=0 | 10:27 |
| pojadhav | rlandy, ysandeep|rover, jm1 : is this known issue https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider-wallaby&job_name=tripleo-ci-centos-9-content-provider&skip=0 | 10:48 |
| pojadhav | my patch is blocked due to some jobs failure : https://review.opendev.org/c/openstack/tripleo-ci/+/835460/26#message-8fc0023bc17ffd2cbbbb3eae44202ec379827679 | 10:49 |
| ysandeep|rover | pojadhav, you are not failing on content-provider but on ipa job | 10:51 |
| ysandeep|rover | https://review.opendev.org/c/openstack/tripleo-ci/+/835460/26#message-fb6e48b4f8d7107710fccfc6a357eeb69e5d705f | 10:51 |
| pojadhav | ysandeep|rover, but c9 content provider also failed with retry limit and rest c9 scenarios got skipped. | 10:52 |
| pojadhav | ahh sorry | 10:53 |
| ysandeep|rover | pojadhav, the latest run passed on content-provider and ipa failure is known | 10:53 |
| pojadhav | ysandeep|rover, oops my bad.. looked at it several times so posted previous url. | 10:54 |
| pojadhav | ysandeep|rover, ack thank you! | 10:54 |
| chandankumar | ysandeep|rover: what is the tempest issue and which patch is not getting built? | 11:01 |
| ysandeep|rover | chandankumar, thanks.. I meant rdoinfo don't work as depends-on in testproject | 11:04 |
| ysandeep|rover | chandankumar, testing your way: https://review.rdoproject.org/r/c/rdoinfo/+/42939 | 11:04 |
| Tengu | ysandeep|rover: how does the testproject look now? all green? :) | 11:04 |
| Tengu | ah. the red isn't on the patch test. gosh, I was affraid. | 11:04 |
| * chandankumar thinking | 11:06 | |
| chandankumar | jbadiapa: this time https://770d2c4845d1b6e3f37a-370e010525c6da286e6aa54793058fb2.ssl.cf5.rackcdn.com/836845/12/check/tripleo-ansible-centos-8-molecule-backup_and_restore/e51ddf0/reports.html | 11:08 |
| chandankumar | set basic home fact got skipped | 11:08 |
| chandankumar | oh sorry that is from default one | 11:09 |
| chandankumar | weired it is still not running | 11:10 |
| jbadiapa | chandankumar, I might have a typo or something | 11:10 |
| jbadiapa | otherwise I dont understand | 11:10 |
| jbadiapa | maybe the name of the directory | 11:12 |
| chandankumar | ysandeep|rover: which upstream tempest patch fixes the issue? | 11:13 |
| chandankumar | ysandeep|rover: via https://review.rdoproject.org/r/c/rdoinfo/+/42939 it will build the package from master and test it with the deppyment | 11:14 |
| chandankumar | jbadiapa: may be yaml vs yml? | 11:14 |
| chandankumar | jbadiapa: https://review.opendev.org/c/openstack/tripleo-ansible/+/836845 | 11:15 |
| chandankumar | https://review.opendev.org/c/openstack/tripleo-ansible/+/836845/12/tripleo_ansible/roles/backup_and_restore/molecule/default/molecule.yml is ending with yml | 11:15 |
| chandankumar | can we also rename this to https://review.opendev.org/c/openstack/tripleo-ansible/+/836845/12/tripleo_ansible/roles/backup_and_restore/molecule/cli-undercloud-backup-db/molecule.yaml and other files to yml | 11:16 |
| ysandeep|rover | chandankumar, I was pinged from alan that pin needs to be removed for wallaby - https://trello.com/c/V3XK8eEK/2133-cixbz2012096ospphase2osp17openstack-novafailed-to-detach-device-from-the-live-domain-config-when-attaching-already-attached-volu | 11:16 |
| jbadiapa | chandankumar, ok, I'm going to rename the directory and then the extension | 11:16 |
| jbadiapa | let's see where is the thing | 11:17 |
| ysandeep|rover | chandankumar, https://review.opendev.org/c/openstack/tempest/+/840112 | 11:17 |
| chandankumar | ysandeep|rover: ok then my way will take care of that | 11:17 |
| chandankumar | ysandeep|rover: if that fixes the issue, then we can do a new release of tempest | 11:18 |
| chandankumar | ysandeep|rover: does the test is in skiplist or unskipped? | 11:18 |
| ysandeep|rover | chandankumar, so for wallaby - instead of unpinning. we need to do a new release of tempest and then bump the tag.. | 11:19 |
| chandankumar | ysandeep|rover: let's fix test the stuff whether the issue is fixed or not | 11:19 |
| chandankumar | then do the release | 11:19 |
| *** dviroel_ is now known as dviroel | 11:19 | |
| ysandeep|rover | https://trello.com/c/V3XK8eEK/2133-cixbz2012096ospphase2osp17openstack-novafailed-to-detach-device-from-the-live-domain-config-when-attaching-already-attached-volu was not reported by us, I think we are just testing new tempest don't break anything in wallaby | 11:20 |
| chandankumar | jbadiapa: can you rename these files molecule.yaml to molecule.yml | 11:21 |
| chandankumar | same for coverage and preapre for new dir | 11:21 |
| jbadiapa | chandakumar, yes, that's going to be my next test | 11:21 |
| chandankumar | https://molecule.readthedocs.io/en/latest/configuration.html | 11:22 |
| chandankumar | there is no mention of molecule.yaml | 11:22 |
| jbadiapa | Oh! | 11:22 |
| jbadiapa | chandakumar++ | 11:23 |
| jbadiapa | thanks a lot | 11:23 |
| chandankumar | ysandeep|rover: https://trello.com/c/V3XK8eEK/2133-cixbz2012096ospphase2osp17openstack-novafailed-to-detach-device-from-the-live-domain-config-when-attaching-already-attached-volu and https://trello.com/c/EsgceuAN/2342-cixlp1960310tripleociproa-compute-tests-eg-testrebuildserverwithvolumeattached-failing-with-volume-id-failed-to-reach-available are similar ? | 11:25 |
| ysandeep|rover | chandankumar: I think so from quick look, jm1 is looking at degraded cards may be he can confirm ^^ | 11:28 |
| chandankumar | jbadiapa: \o https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_877/836845/14/check/tripleo-ansible-centos-8-molecule-backup_and_restore/87733b4/job-output.txt | 11:51 |
| chandankumar | it is working now, but tests are failing, you can fix it | 11:51 |
| dviroel | oh, my 'periodic-promote-ceph-daemon' pipeline wasn't trigerred :( | 11:54 |
| dviroel | rlandy: chandankumar ^ any ideas why? | 11:54 |
| jbadiapa | chandakumar: that's the error I was trying to fix, thx | 11:54 |
| jbadiapa | chandankumar^ | 11:54 |
| rlandy | dviroel: looking | 11:55 |
| rlandy | dviroel: does your line show up on rdo zuul? | 11:55 |
| rlandy | if not, you may need a zuul restart to add that | 11:56 |
| dviroel | rlandy: yes | 11:56 |
| rlandy | periodic-promote-ceph-daemon - ok - there it is | 11:56 |
| * rlandy checks templates | 11:57 | |
| dviroel | rlandy: missing on tripleo-ci templates, I think | 11:58 |
| rlandy | yep | 11:58 |
| rlandy | dviroel: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo.yaml#L12 | 11:58 |
| dviroel | rlandy: https://review.rdoproject.org/r/c/config/+/42940 | 12:00 |
| dviroel | here ^ we use the project-template name IIUC | 12:02 |
| chandankumar | dviroel: can this https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/ci-config-molecule-python-pipeline.yaml#L5 be periodic-promote-ceph-daemon ? | 12:06 |
| chandankumar | base on this https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-pipeline-main.yaml#L5 | 12:07 |
| dviroel | chandankumar: sorry - didn't get your point | 12:09 |
| chandankumar | dviroel: for example train one https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-pipeline-train.yaml#L5 | 12:09 |
| chandankumar | openstack-periodic-integration-stable4 is the pipeline we use | 12:10 |
| chandankumar | dviroel: We need to do similar for ceph pipeline | 12:10 |
| chandankumar | https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/ci-config-molecule-python-pipeline.yaml#L5 periodic-ci-config-molecule-python needs to be replaced with periodic-promote-ceph-daemon | 12:10 |
| chandankumar | and then it will work | 12:11 |
| dviroel | chandankumar: mine is this one https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/ceph-jobs.yaml#L21 | 12:11 |
| chandankumar | oh sorry | 12:11 |
| jm1 | chandankumar: i think there are actually at least three cix cards which have the same root cause, you quoted two of them, here is the third: https://trello.com/c/KZ2NsQh0/2356-cixlp1960661tripleociproa-ussuri-jobs-failing-on-tempestapicomputeserverstestserveractionsserveractionstestjson | 12:11 |
| jm1 | ysandeep|rover: wrt https://trello.com/c/V3XK8eEK/2133-cixbz2012096ospphase2osp17openstack-novafailed-to-detach-device-from-the-live-domain-config-when-attaching-already-attached-volu#comment-6284db6e62c23b8479c13587 | 12:12 |
| chandankumar | dviroel: then https://review.rdoproject.org/r/c/config/+/42940 will fix the issue, thanks rlandy :-) | 12:12 |
| jm1 | ysandeep|rover: why do we need a new release? | 12:13 |
| jm1 | ysandeep|rover: we can pin to a git hash | 12:13 |
| dviroel | chandankumar: tks, will manually test as soon its get merged | 12:13 |
| ysandeep|rover | chandankumar, ^^ | 12:13 |
| dviroel | just kicked | 12:14 |
| ysandeep|rover | jm1: check chatter above, chandankumar mentioned we need a new release of tempest | 12:15 |
| dviroel | rlandy: we you have some time, no rush, can you create a wallaby folder here https://images.rdoproject.org/ceph/daemon/ ? | 12:15 |
| ysandeep|rover | jm1, but +1 to pin using git hash, lets see what chandankumar suggests | 12:16 |
| * dviroel 'periodic-promote-ceph-daemon' now running | 12:16 | |
| chandankumar | ysandeep|rover: if current merged patches from master fixes the issue, then we can pin | 12:16 |
| ysandeep|rover | chandankumar: using git hash? | 12:17 |
| chandankumar | once all the patches of https://review.opendev.org/q/topic:wait_until_sshable_pingable merges, we can do a new release | 12:17 |
| chandankumar | ysandeep|rover: yes | 12:17 |
| rlandy | dviroel: done ... | 12:18 |
| chandankumar | ysandeep|rover: https://review.rdoproject.org/r/c/rdoinfo/+/40228 | 12:18 |
| rlandy | drwxrwxr-x. 2 uploader uploader 81 May 13 17:36 master | 12:18 |
| rlandy | drwxrwxr-x. 2 uploader uploader 6 May 18 12:17 wallaby | 12:18 |
| chandankumar | ysandeep|rover: make sure we also update xena tag | 12:18 |
| dviroel | rlandy: thanks | 12:18 |
| dviroel | rlandy: yes, it is there | 12:18 |
| ysandeep|rover | chandankumar, yeah good point | 12:19 |
| soniya29 | chandankumar, arxcruz_ rlandy, pojadhav, ysandeep|rover, kopecmartin, please add/edit today's agenda for tempest meeting - https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ | 12:20 |
| kopecmartin | soniya29: i can't attend, i'm interviewing interns today | 12:21 |
| kopecmartin | i thought i declined the meeting already, seems like the calendar is disobeying me | 12:22 |
| jm1 | chandankumar, ysandeep|rover: can we safely bump tempest in all RDO releases? this issue apparently affects all | 12:23 |
| jm1 | kopecmartin: ^ | 12:23 |
| chandankumar | jm1: nope | 12:23 |
| chandankumar | jm1: for train, we need to carefully backport the patches which fixes the issue | 12:24 |
| chandankumar | each bump brings new functionality and might break existing deployments | 12:24 |
| chandankumar | for older releases | 12:24 |
| rlandy | chandankumar: wrt train per the prod chain meeting - no more skiplist | 12:24 |
| rlandy | will explain at tempest call | 12:24 |
| chandankumar | ok | 12:25 |
| rlandy | dasm|off: are you still working on https://trello.com/c/IryyjbGs/2344-cixlp1960355tripleociproa-tempest-tests-testoauth1tokensoauth1tokenstest-are-failing-on-tripleo-ci-centos-9-ovb-3ctlr1comp-featu? | 12:25 |
| rlandy | pls update the CIX | 12:25 |
| rlandy | asked jm1 to look at clearing that column | 12:26 |
| rlandy | bhagyashris: pls add a tempest skiplist cleanup to the jira sprint board | 12:26 |
| rlandy | akahat is set o start work on that | 12:27 |
| rlandy | arxcruz_: ^^ we will need your brain/advice here | 12:27 |
| jm1 | rlandy: which tempest meeting? i might want to join in order to get an idea how we handle this tempest issue on older releases | 12:27 |
| rlandy | soniya29: ^^ pls invite jm1 to meeting today | 12:28 |
| rlandy | soniya29: nvm added jm1 | 12:28 |
| rlandy | jm1: invite sent | 12:29 |
| jm1 | rlandy: thanks! | 12:31 |
| arxcruz_ | rlandy form what I see, it's related to public url doesn't have the right permissions | 12:32 |
| rlandy | arxcruz_: sorry - your input on the skiplist work | 12:34 |
| *** arxcruz_ is now known as arxcruz | 12:34 | |
| arxcruz | rlandy oh, ack | 12:35 |
| ysandeep|rover | Tengu, fyi.. failed with your patch as well.. we share the logs soon | 12:38 |
| ysandeep|rover | we will* share the logs soon | 12:38 |
| ysandeep|rover | Tengu, sharing pastebin in the meantime: http://pastebin.test.redhat.com/1052592 | 12:39 |
| ysandeep|rover | Error: ['/etc/pki/tls/certs/httpd:/etc/pki/tls/certs/httpd:ro', '/etc/pki/tls/private/httpd:/etc/pki/tls/private/httpd:ro']: incorrect volume format, should be [host-dir:]ctr-dir[:option] | 12:39 |
| bhagyashris | rlandy, ack | 12:43 |
| bhagyashris | done https://issues.redhat.com/browse/TRIPLEOCI-1045 | 12:45 |
| ysandeep|rover | rlandy, jm1 fyi.. heads up sent to openstack-discuss about ipa blocking check/gate | 12:45 |
| ysandeep|rover | [TripleO] Gate blocker on Master - standalone-on-multinode-ipa failing | 12:45 |
| Tengu | ysandeep|rover: sounds like another error | 12:47 |
| Tengu | ysandeep|rover: unless the thing eating the volumes list isn't supporting a list... | 12:47 |
| Tengu | ysandeep|rover: my git repo is locked for some more time, I'm deploying a lab with my nft things. | 12:48 |
| ysandeep|rover | Tengu, I wonder if we should just revert james patch to unblock check/gate and fix this properly in a followup. | 12:48 |
| Tengu | ysandeep|rover: care to check with him? | 12:48 |
| ysandeep|rover | sure | 12:49 |
| Tengu | apparently, he missed some testproject before switching to the new role. | 12:49 |
| Tengu | reverting sounds OK, and he (we) can work with some testproject on the follow-up. | 12:49 |
| rlandy | ysandeep|rover: thank you | 12:52 |
| soniya29 | rlandy, okay | 12:52 |
| soniya29 | kopecmartin, haha :) | 12:53 |
| rlandy | ysandeep|rover: which diff code does the rr tool use to diff the rpms? | 12:53 |
| ysandeep|rover | rlandy: do you remember wes wrote some code earlier diffing the CSVs | 12:56 |
| rlandy | def get_diff(control_tag, file1, test_tag, file2): | 12:57 |
| rlandy | ysandeep|rover: yeah - same code I used in diff control test | 12:57 |
| rlandy | ok - then I got it | 12:57 |
| ysandeep|rover | yeah its same code.. just rewritten in rr | 12:57 |
| soniya29 | rlandy, chandankumar, jm1, arxcruz, tempest meeting? | 13:00 |
| soniya29 | chandankumar, ^^? | 13:01 |
| jm1 | ysandeep|rover: timezone is now picked up in ci, see latest comment in https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/842218 | 13:16 |
| ysandeep|rover | jm1, nice .. do we need same changes in config/environments/rdocloud-ci.yml.example ? | 13:31 |
| jm1 | ysandeep|rover: kept it there because it is an *.example file. can remove the change there if you want | 13:32 |
| *** dasm|off is now known as dasm | 13:52 | |
| dasm | o/ | 13:52 |
| dasm | rlandy: i did not look into OAUTH failures for weeks | 13:53 |
| dasm | i can respin the patch and check it agaie | 13:53 |
| dasm | *again | 13:53 |
| *** pojadhav is now known as pojadhav|afk | 14:05 | |
| rlandy | jm1: ^^ | 14:11 |
| jm1 | rlandy, dasm: cool! could you link your testproject in lp bug? | 14:12 |
| jm1 | rlandy, ysandeep|rover, chandankumar: where does downstream have their tempest skiplist (or whatever they use to select tests) | 14:13 |
| rlandy | ysandeep|rover: chandankumar: so on psi ... | 14:13 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-internal-wallaby&project=testproject&skip=0 | 14:13 |
| rlandy | we were doing pretty well | 14:13 |
| rlandy | jm1: excellent question - I asked | 14:14 |
| rlandy | apparently in multiple places | 14:14 |
| dasm | jm1: it's last comment by bhagyashris | 14:14 |
| jm1 | dasm: ok, thx | 14:15 |
| jm1 | rlandy: reason i ask is that i want to check why downstream did not hit the tempest bug in train | 14:15 |
| dasm | makes sense | 14:15 |
| ysandeep|rover | rlandy, the issue which we are hitting seems to be because of resource crunch in vexx | 14:17 |
| ysandeep|rover | rlandy, still seeing same issue today - I have pinged mnaser on vexx channel to get today's loadaverage on compute nodes. | 14:18 |
| rlandy | ysandeep|rover: " issue which we are hitting" ie: the new random tempest failures? | 14:19 |
| ysandeep|rover | rlandy, yeah | 14:19 |
| rlandy | ysandeep|rover: hmmm ... maybe we start using psi :) | 14:19 |
| ysandeep|rover | rechecking https://code.engineering.redhat.com/gerrit/c/testproject/+/209874 to have more data | 14:21 |
| ysandeep|rover | jm1, ++ nice "May 18 10:12:14 undercloud.localdomain python3[70991]: ansible-timezone Invoked with name=UTC hwclock=None" | 14:28 |
| jm1 | ysandeep|rover: do you have a testproject or something for https://review.rdoproject.org/r/c/rdoinfo/+/42939? | 14:39 |
| ysandeep|rover | jm1, testing in patch itself: https://review.rdoproject.org/r/c/rdoinfo/+/42939/2/.zuul.yaml | 14:40 |
| jm1 | ysandeep|rover: these jobs dont use tempest skiplist? | 14:42 |
| ysandeep|rover | jm1: they will if in tempest-skiplist if we are skipping with jobs:[] | 14:45 |
| jm1 | ysandeep|rover: i would like to find out if your new wallaby tempest pkgs could also help with other bugs. will it be enough to add depends-on your patch here (and add wallaby jobs)? https://review.rdoproject.org/r/c/testproject/+/42916 | 14:47 |
| ysandeep|rover | jm1, lets chat for few mins after cix - we can stay back | 14:48 |
| jm1 | ysandeep|rover: ack | 14:48 |
| ysandeep|rover | jm1, ex: https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml#L1025 | 14:52 |
| jm1 | ysandeep|rover: https://review.rdoproject.org/r/c/testproject/+/42916 | 14:54 |
| jm1 | ysandeep|rover: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/842074 | 14:54 |
| * jm1 bbl | 15:01 | |
| *** ysandeep|rover is now known as ysandeep|dinner | 15:17 | |
| rlandy | ysandeep|dinner: jm1: going into another meeting - afterwhic you'll probably be EoD - anything you want me to watch/do? | 15:21 |
| rlandy | periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-16.2testprojectmastercheck200295,261 | 15:28 |
| rlandy | 1 hr 17 mins 51 secs2022-05-18 13:46:36SUCCESS | 15:28 |
| rlandy | ha worked | 15:28 |
| rlandy | ysandeep|dinner: jm1: ^^ going to try test out a few tests with these new containers | 15:29 |
| *** rlandy is now known as rlandy|mtg | 15:31 | |
| *** dviroel is now known as dviroel|lunch | 15:39 | |
| *** ysandeep|dinner is now known as ysandeep | 15:52 | |
| ysandeep | rlandy|mtg, thanks, just one thing - please see this get through gate: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/842320 | 15:52 |
| ysandeep | I have requested james and cedric to +w this, we already have two +2 | 15:53 |
| ysandeep | we have the +w now as well | 15:54 |
| ysandeep | dasm, hey o/ regarding legacy boot - during the review time - we discussd about modifying one of the existing featureset for wallaby to test bios instead of creating new job | 15:55 |
| ysandeep | may be fs020.. | 15:56 |
| ysandeep | we just want to test node boots up with legacy bios as well | 15:56 |
| *** ysandeep is now known as ysandeep|out | 15:57 | |
| * ysandeep|out out see you all tomorrow o/ | 15:57 | |
| dasm | ysandeep|out: hey, i noticed your comment. I asked for clarification | 16:07 |
| *** dviroel|lunch is now known as dviroel | 16:31 | |
| rlandy|mtg | ysandeep|out: ok Tengu w+'ed https://review.opendev.org/c/openstack/tripleo-heat-templates/+/842320 | 16:32 |
| *** rlandy|mtg is now known as rlandy | 16:35 | |
| Tengu | rlandy: I'll work tomorrow on a proper fix. | 16:41 |
| Tengu | or try to. | 16:41 |
| Tengu | if James doesn't push anything first. | 16:41 |
| rlandy | thanks | 16:43 |
| *** jpena is now known as jpena|off | 16:52 | |
| rlandy | lunch - brb | 16:56 |
| rlandy | dviroel: pls see forwarded email and meeting invite for monday | 17:47 |
| dviroel | rlandy: great, thanks | 17:51 |
| rlandy | dviroel: we're a bit behind ... | 17:51 |
| rlandy | actually ... let's chat few minutes | 17:51 |
| rlandy | dviroel: https://meet.google.com/fhj-yskd-ges | 17:52 |
| *** rlandy is now known as rlandy|mtg | 18:28 | |
| rlandy|mtg | dviroel: git a green pass on the container builds with multiple registry logins | 18:29 |
| rlandy|mtg | git | 18:29 |
| rlandy|mtg | got | 18:29 |
| rlandy|mtg | I am running some tests with those new containers | 18:29 |
| rlandy|mtg | if they pass, would be ready to use the playbook with the duplicated login until the role is ready | 18:30 |
| dviroel | ack - already have a role working in my machine | 18:30 |
| dviroel | should be ready soon | 18:30 |
| jm1 | rlandy|mtg: fyi https://trello.com/c/EsgceuAN/2342-cixlp1960310tripleociproa-compute-tests-eg-testrebuildserverwithvolumeattached-failing-with-volume-id-failed-to-reach-available#comment-6285433e1809176d890d274f | 19:05 |
| *** rlandy|mtg is now known as rlandy | 19:08 | |
| rlandy | jm1: thanks - looking - sorry - just rolling off calls | 19:08 |
| rlandy | jm1: fyi - also added you to prodified control plan meeting if you have an interest | 19:09 |
| rlandy | rcastillo: dasm: ^^ it's early for you | 19:09 |
| rlandy | but will discuss at tomorrow's scrum if anyone else has an interest | 19:09 |
| rlandy | jm1: so we don;t have conclusive results? | 19:11 |
| jm1 | rlandy: thanks for adding me to the meetings :) | 19:12 |
| jm1 | rlandy: "so we don;t have conclusive results" > what do you mean? | 19:13 |
| dasm | k | 19:13 |
| rlandy | you are all welcome | 19:14 |
| rlandy | juts don't want to overwhelm everyone | 19:15 |
| rlandy | jm1: what I get from your comments .... | 19:16 |
| rlandy | QE would not see all the failures as they only run some tests | 19:16 |
| rcastillo | lunch brb | 19:16 |
| jm1 | rlandy: ah yes. they wont catch all errors we see because they do not run all tempest tests | 19:17 |
| rlandy | jm1: ysandeep|out: fyi ... merged patches ... | 19:17 |
| rlandy | <opendevreview> Merged openstack/tripleo-heat-templates stable/wallaby: Add RabbitFIPS parameter https://review.opendev.org/c/openstack/tripleo-heat-templates/+/828145 | 19:17 |
| rlandy | <opendevreview> Merged openstack/tripleo-heat-templates master: Revert "Use tripleo_keystone role" https://review.opendev.org/c/openstack/tripleo-heat-templates/+/842320 | 19:17 |
| rlandy | jm1: understood - ok | 19:17 |
| rlandy | jm1: but we don't know yet if the fix to tempest worked? | 19:17 |
| rlandy | if that skiplist under test somewhere? | 19:17 |
| * rlandy checks if master has new version of tempest | 19:21 | |
| jm1 | rlandy: just added another comment. c9 wallaby still fails https://trello.com/c/EsgceuAN/2342-cixlp1960310tripleociproa-compute-tests-eg-testrebuildserverwithvolumeattached-failing-with-volume-id-failed-to-reach-available#comment-628547cf0f7f7365af16bfa0 | 19:24 |
| jm1 | rlandy: this time its using the latest tempest package, but it still does not pass | 19:25 |
| jm1 | rlandy: lets see what ysandeep|out's testproject reports tomorrow | 19:25 |
| rlandy | jm1: ok - that is what I was asking | 19:25 |
| rlandy | his project reported | 19:26 |
| rlandy | jm1: sorry - https://review.rdoproject.org/r/c/rdoinfo/+/42952 - running but some failed | 19:27 |
| rlandy | penstack-tempest-30.1.1-0.20220517112907.569c7a8.el9.src.rpm2022-05-17 11:32 1.1M | 19:28 |
| rlandy | in tripleo-ci-testing | 19:28 |
| rlandy | so it's in the line now | 19:28 |
| jm1 | rlandy: this one is mine, this one is ysandeep|out: https://review.rdoproject.org/r/c/rdoinfo/+/42939 | 19:28 |
| jm1 | rlandy: we will have a detailed look tomorrow | 19:31 |
| rlandy | jm1: ok - I am interested in seeing if master (tripleo-ci-testing) passes with skip - I'll testproject that - no worries | 19:31 |
| rlandy | have a good night | 19:31 |
| jm1 | rlandy: already have a testproject | 19:32 |
| jm1 | rlandy: you could just retrigger | 19:32 |
| rlandy | great - link? | 19:32 |
| jm1 | rlandy: https://review.rdoproject.org/r/c/testproject/+/41826 | 19:32 |
| jm1 | rlandy: all is linked in lp bug report | 19:32 |
| rlandy | perfect - thanks | 19:33 |
| * jm1 out for today | 19:33 | |
| *** dviroel is now known as dviroel|out | 20:28 | |
| rcastillo | rlandy: at what time is the podified control plane meeting? | 21:48 |
| rlandy | rcastillo: 8: 30am my time - I'll send you the invite but it's way early for you | 22:07 |
| rlandy | needed to catch the QE folks in Israel | 22:07 |
| rlandy | this is just the first meeting | 22:07 |
| *** rlandy is now known as rlandy|bbl | 22:11 | |
| * dasm signs off | 22:32 | |
| *** dasm is now known as dasm|off | 22:32 | |
| * rcastillo leaving as well | 22:48 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!