| *** ysandeep|out is now known as ysandeep|rover | 05:04 | |
| ysandeep|rover | hello everyone o/ have a wonderful morning | 05:08 |
|---|---|---|
| marios | \o | 05:09 |
| chandankumar | o/ | 05:16 |
| soniya29 | ysandeep|rover, same to you :) | 05:19 |
| pojadhav | morning \o/ | 05:52 |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 06:15 | |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 06:48 | |
| akahat | chandankumar, marios OVB jobs failing with net-config-undercloud.yaml file not found. I've found two references with it https://codesearch.opendev.org/?q=net-config-undercloud.yaml&i=nope&literal=nope&files=&excludeFiles=&repos= | 07:10 |
| akahat | and this file is not present in THT repo. Is it used anywhere. | 07:10 |
| akahat | https://logserver.rdoproject.org/27/41827/8/check/periodic-tripleo-ci-centos-8-ovb-1ctlr_1cellctrl_1comp-featureset063-master/0d03242/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 07:10 |
| akahat | https://logserver.rdoproject.org/27/41827/8/check/tripleo-ci-centos-8-ovb-1ctlr_1cellctrl_1comp-featureset063/1d32c90/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 07:11 |
| jm1 | akahat, Tengu: any advice on what to test additionally for this revert? https://review.opendev.org/c/openstack/tripleo-ci/+/841992 | 07:19 |
| Tengu | jm1: did you check the reasons of the RDO failures= | 07:21 |
| Tengu | ? | 07:21 |
| jm1 | Tengu: why rdo ci failed? yes, added comments | 07:22 |
| Tengu | just saw it. | 07:22 |
| Tengu | guess that should be OK, with the running testproject. | 07:22 |
| Tengu | jm1: though you may want to get anyone from HrdProv on that. Harald or Steve. | 07:23 |
| Tengu | especially Steve actually. | 07:23 |
| jm1 | Tengu: ack, will ping him. thanks! | 07:24 |
| Tengu | jm1: voted +1 while adding Harald, Steve and Julia | 07:24 |
| Tengu | jm1: you may wait to -w it for now, in the wait of testproject. | 07:24 |
| Tengu | just to avoid any unwanted merge in-between | 07:25 |
| jm1 | Tengu: thx, wip'ed it | 07:25 |
| jm1 | Tengu: testproject actually finished for wallaby already. i just added two new jobs for ussuri and train to really cover all jobs from the lp bug | 07:26 |
| Tengu | 'k | 07:27 |
| marios | akahat: o/ | 07:29 |
| marios | akahat: you mean in your patch ? will look in a minute just finishing some reviews | 07:29 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 07:29 |
| *** jpena|off is now known as jpena | 07:31 | |
| akahat | marios, ack | 07:35 |
| marios | akahat: looks like it was removed i found it on older branches, so must be some recent commit in tht | 07:44 |
| marios | akahat: https://opendev.org/openstack/tripleo-heat-templates/src/commit/847982a51a20bcdc53df0a29785faf66c8ae18e0/net-config-undercloud.j2.yaml#L3 | 07:45 |
| marios | akahat: i would expect this to be seen more widely then fyi ysandeep|rover ^^^ akahat is hitting some ovb issue on his master jobs @ 10:10 < akahat> https://logserver.rdoproject.org/27/41827/8/check/periodic-tripleo-ci-centos-8-ovb-1ctlr_1cellctrl_1comp-featureset063-master/0d03242/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 07:45 |
| ysandeep|rover | marios, why c8 job for master? | 07:46 |
| ysandeep|rover | master jobs should be on c9 only, right? | 07:47 |
| akahat | ysandeep|rover, I ran it for wallaby https://review.rdoproject.org/r/c/testproject/+/41827 | 07:47 |
| marios | ysandeep|rover: yes master should be 9 only | 07:47 |
| akahat | ysandeep|rover, oh.. it should be -wallaby not -master | 07:49 |
| marios | akahat: k false alarm then? i didn't even check your testproject i just went on the logs | 07:51 |
| akahat | marios, yes. that was false alarm.. sorry... and thanks ysandeep|rover. | 07:51 |
| ysandeep|rover | marios, akahat i thought we decided to only keep fs001 and fs035 for wallaby c8 , those are the only jobs we run in periodic now. | 07:51 |
| ysandeep|rover | akahat, I wonder do we even run fs063 anymore for c8 wallaby.. | 07:52 |
| akahat | ysandeep|rover, we don't have those job runs since long itm.e | 07:53 |
| akahat | ysandeep|rover, if those are used should i go ahead and remove it? | 07:53 |
| ysandeep|rover | akahat, should i read this as "if those are not* used" | 07:54 |
| akahat | ysandeep|rover, :) | 07:54 |
| akahat | yes. | 07:54 |
| ysandeep|rover | akahat, +1 to remove ovb c8 master jobs and also the featureset which don't run anymore for c8 wallaby. | 07:55 |
| akahat | ysandeep|rover, okay. I'll take care of it. | 07:55 |
| ysandeep|rover | akahat++ marios++ | 07:55 |
| marios | np akahat | 07:56 |
| marios | ysandeep|rover: thanks | 07:56 |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 08:00 | |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 09:21 | |
| ysandeep|rover | reviewbot: please add in review list: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42914 | 09:57 |
| reviewbot | I have added your review to the Review list | 09:57 |
| ysandeep|rover | frenzy_friday, ^^ fyi.. | 09:57 |
| frenzy_friday | ysandeep|rover, cool, thanks! | 10:01 |
| jm1 | ysandeep|rover: do we need force_periodic for testproject with job periodic-tripleo-ci-centos-8-standalone-full-tempest-api-victoria? | 10:10 |
| ysandeep|rover | jm1, that's a integration line job so yes | 10:10 |
| ysandeep|rover | an* | 10:11 |
| jm1 | ysandeep|rover: we dont need dlrn hash, if i just want to run it again? | 10:11 |
| jm1 | *with new code | 10:11 |
| ysandeep|rover | jm1: if you don't pass dlrn hash, it will use the latest tripleo-ci-testing hash | 10:11 |
| jm1 | ysandeep|rover: ack, that is fine | 10:12 |
| frenzy_friday | hey ysandeep|rover, commented on your patch. Bot pls add to review list https://review.rdoproject.org/r/c/rdo-jobs/+/42917 | 10:20 |
| reviewbot | I have added your review to the Review list | 10:20 |
| jm1 | ysandeep|rover: went through ci quality degraded cards in trello and left comments where appropriate | 10:23 |
| jm1 | ysandeep|rover: atm wallaby c8 compute comp is the only non-waiting, open issue? | 10:25 |
| *** rlandy|out is now known as rlandy | 10:26 | |
| * jm1 lunch | 10:26 | |
| ysandeep|rover | jm1, which bug? | 10:27 |
| rlandy | ysandeep|rover: jm1: how are things? | 10:27 |
| rlandy | jm1: pls nick | 10:28 |
| ysandeep|rover | rlandy: no new bug, same tempest random fun | 10:28 |
| jm1 | ysandeep|rover: what do you mean with bug? wallaby c8 compute comp? got it from your rr notes | 10:28 |
| rlandy | ysandeep|rover: any nes on the neutron fix? | 10:28 |
| rlandy | news | 10:28 |
| *** jm1|ruck is now known as jm1|rover | 10:28 | |
| pojadhav | chandankumar, hello. could you please review again, patch went in merge conflict due to soniya's patch. https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/839588 | 10:30 |
| pojadhav | arxcruz, ^^ | 10:30 |
| ysandeep|rover | rlandy: nope but not all the tempest tests are failing on neutron bug - may be 1/10.. rest are just random | 10:31 |
| ysandeep|rover | rlandy, random issues may be related with performance on infra | 10:31 |
| pojadhav | we will make patches dependant to avoid such conflicts again and again. | 10:31 |
| rlandy | ysandeep|rover: jm1|rover: let's sync for a few - https://meet.google.com/nhf-cqto-efe?pli=1&authuser=0 | 10:31 |
| ysandeep|rover | rlandy, I see chandankumar have success with ovb in downstream - Lets kick more jobs there and compare results with vexx | 10:31 |
| * ysandeep|rover fetching headphone | 10:32 | |
| rlandy | bhagyashris: hi - how is your reparening work? | 10:32 |
| rlandy | bhagyashris: do you have the two patterns to review? | 10:32 |
| chandankumar | ysandeep|rover: is train fs01 still in trouble state? | 10:32 |
| bhagyashris | rlandy, not yet discussing with akahat working on it | 10:33 |
| pojadhav | rlandy, https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/839588 when free thanks! | 10:33 |
| bhagyashris | soon will post patch... | 10:33 |
| chandankumar | rlandy: ysandeep|rover let's get ovb patches merged then https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/409010 | 10:35 |
| ysandeep|rover | chandankumar, thanks.. will check soon | 10:36 |
| pojadhav | thanks chandankumar rlandy for approval | 10:36 |
| arxcruz | pojadhav sorry, didn't saw your message chandankumar already +w | 10:37 |
| pojadhav | arxcruz, np | 10:37 |
| chandankumar | ysandeep|rover: rlandy Does Downstream still have limitted resources? | 10:37 |
| rlandy | chandankumar: no - should be fine now on resources | 10:38 |
| rlandy | we are on rr sync | 10:38 |
| ysandeep|rover | chandankumar, no.. we got additional resources | 10:38 |
| rlandy | will check in a few your revuew | 10:38 |
| chandankumar | https://sf.hosted.upshift.rdu2.redhat.com/logs/35/409835/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-internal-wallaby/abc35c2/job-output.txt | 10:38 |
| *** pojadhav is now known as pojadhav|afk | 10:49 | |
| rlandy | chandankumar: one error: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/409010 | 10:51 |
| rlandy | then we can merge | 10:51 |
| chandankumar | nice catch | 10:52 |
| chandankumar | rlandy: updated | 10:53 |
| rlandy | chandankumar: merging | 10:54 |
| rlandy | chandankumar: ysandeep|rover is going to send a testproject to run those jobs | 10:54 |
| chandankumar | rlandy: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/409834 is also good to go | 10:54 |
| rlandy | ok - will check afterwards | 10:55 |
| * jm1 lunch | 10:55 | |
| rlandy | frenzy_friday: pls join us on https://meet.google.com/nhf-cqto-efe?pli=1&authuser=0 | 10:59 |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 11:18 | |
| *** pojadhav|afk is now known as pojadhav | 11:26 | |
| *** dviroel|out is now known as dviroel | 11:28 | |
| frenzy_friday | rlandy, ysandeep|rover|brb dpawlik : the push failure 'might' have been becaus of 2 jobs running at the same time or the user getting updated in between the jobs. I am rerunning the jobs in a patch to see what happens now that there is only a single user. https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-centos-9-quay-master (the user change merged around 9:00 utc) | 11:38 |
| rlandy | frenzy_friday: ok - pls let us know the result | 11:39 |
| rlandy | ysandeep|rover|brb also did a rerun before | 11:39 |
| frenzy_friday | ok, this one also failed with 401 unauthorized during push: https://logserver.rdoproject.org/28/41528/16/check/periodic-tripleo-build-containers-base-c9-quay-master/4a23493/logs/build.log Checking | 11:40 |
| rlandy | marios: chandankumar: ysandeep|rover|brb: dviroel: available to meet for 5? https://meet.google.com/mac-bmmz-mij?pli=1&authuser=0 | 11:49 |
| rlandy | if not, will try touch base with you all tomorrow | 11:50 |
| rlandy | re: next gen | 11:50 |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 11:51 | |
| chandankumar | joining | 11:51 |
| ysandeep|rover | rlandy, joining | 11:51 |
| dviroel | joinning | 11:51 |
| bhagyashris | rlandy, hey here is the one solution https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/399683/23/zuul.d/component-jobs.yaml#817 and testproject patch https://code.engineering.redhat.com/gerrit/c/testproject/+/400570 - this is the easy and less complicated solution | 11:52 |
| bhagyashris | working on second solution | 11:52 |
| bhagyashris | create standard job for each solution | 11:53 |
| rlandy | marios: around? if not will touch base with you later | 11:53 |
| bhagyashris | s/create standard job for each solution/create standard job for each scenario | 11:53 |
| rlandy | bhagyashris: thanks - rolling on to a bunch of meetings | 12:01 |
| rlandy | will review afterwards | 12:01 |
| marios | rlandy: o/ | 12:01 |
| rlandy | marios: eek ... let's join for 2 mins | 12:01 |
| rlandy | sam chat | 12:01 |
| marios | rlandy: joining now | 12:01 |
| rlandy | https://meet.google.com/mac-bmmz-mij?pli=1&authuser=0 | 12:02 |
| ysandeep|rover | jm1, sry missed to follow up on wallaby c8 compute -> job passed in rerun - Its missing in criteria - /me proposing a patch to add that job in criteria | 12:03 |
| jm1 | ysandeep|rover: ack, thanks! | 12:07 |
| ysandeep|rover | jm1: ahh, fs001 don't run in component line anymore: that poped up in rr script because you ran a testproject for that job and which failed: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-compute-wallaby | 12:12 |
| ysandeep|rover | for c8 compute component. | 12:13 |
| bhagyashris | rlandy, ack thanks | 12:15 |
| ysandeep|rover | rlandy, jm1 fyi.. 17/8 promoted | 12:18 |
| ysandeep|rover | reviewbot: please add in review list: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/409923 | 12:29 |
| reviewbot | I have added your review to the Review list | 12:29 |
| ysandeep|rover | rlandy, ^^ fyi.. bm in criteria for 17/9 | 12:29 |
| ysandeep|rover | frenzy_friday, hey o/ container build failed in your current run as well | 12:44 |
| ysandeep|rover | https://review.rdoproject.org/zuul/status#41528 | 12:44 |
| frenzy_friday | ysandeep|rover, yep, issues with the cirobot user permission. Daniel and I are checking | 12:45 |
| ysandeep|rover | frenzy_friday++ dpawlik++ thanks! | 12:45 |
| dpawlik | Robot user got permissions. We can recheck the jobs | 12:55 |
| dpawlik | btw, can we delete organizations: testchandan tripleomaster tripleotraincentos7 ? | 12:56 |
| chandankumar | dpawlik: we can testchandan | 12:57 |
| chandankumar | not sure about others | 12:57 |
| dpawlik | k, deleted | 13:04 |
| dpawlik | I guess tripleomaster without centos8/9 suffix can be also deleted | 13:05 |
| dpawlik | but maybe frenzy_friday can clean it up ;) | 13:05 |
| ysandeep|rover | dpawlik, mentioned in some upstream code still https://opendev.org/openstack/tripleo-operator-ansible/src/branch/master/playbooks/container-build.yaml#L15 | 13:06 |
| dpawlik | ah | 13:06 |
| ysandeep|rover | we will need to do cleanup in upstream code first | 13:06 |
| frenzy_friday | dpawlik, aye, will clean up the ones not used | 13:06 |
| ysandeep|rover | frenzy_friday, could you please also make relavant changes in upstream/rdo code first. | 13:08 |
| frenzy_friday | ysandeep|rover, which changes? The ones we were discussing in the call? | 13:08 |
| ysandeep|rover | frenzy_friday, yeah there are still mention of tripleomaster at many places for ex. https://codesearch.opendev.org/?q=tripleomaster | 13:09 |
| frenzy_friday | oh. yep, I'll check and clean | 13:09 |
| ysandeep|rover | frenzy_friday++ thanks! | 13:10 |
| bhagyashris | arxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj, dviroel, rcastillo, dasm, jm1, marios, ysandeep|rover | 13:22 |
| bhagyashris | Tripleo CI community call | 13:22 |
| bhagyashris | https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2022-05-17-Community-Call | 13:23 |
| bhagyashris | @ line 32 | 13:23 |
| ysandeep|rover | rlandy, fyi.. patch to run ovb jobs in psi for comparison - https://code.engineering.redhat.com/gerrit/c/testproject/+/209874 | 13:23 |
| rlandy | thank upi | 13:23 |
| rlandy | you | 13:23 |
| bhagyashris | https://meet.google.com/bqx-xwht-wky | 13:23 |
| bhagyashris | akahat, chandankumar marios rlandy | 13:31 |
| bhagyashris | rcastillo, jm1 dviroel | 13:31 |
| bhagyashris | ^ | 13:32 |
| ysandeep|rover | rlandy, crosscheck ntp seems in sync.. | 13:55 |
| ysandeep|rover | https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/3e5d277/logs/overcloud-controller-0/var/log/extra/chrony.txt.gz | 13:55 |
| ysandeep|rover | https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/3e5d277/logs/overcloud-novacompute-0/var/log/extra/chrony.txt.gz | 13:55 |
| ysandeep|rover | and on undercloud https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/3e5d277/logs/undercloud/var/log/extra/chrony.txt.gz | 13:55 |
| rlandy | ysandeep|rover: on the undercloud | 13:55 |
| rlandy | iiuc | 13:56 |
| rlandy | does not show GTM | 13:56 |
| rlandy | is the sync correct there? | 13:56 |
| *** dasm|off is now known as dasm | 13:56 | |
| dasm | o/ | 13:56 |
| rlandy | ah last link | 13:56 |
| ysandeep|rover | looks okay on undercloud as well - Ref time (UTC) : Tue May 17 07:47:13 2022 | 13:57 |
| rlandy | ysandeep|rover: what do you think of this ... https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/93e97f1/logs/undercloud/var/log/containers/mysql/mariadb.log.txt.gz | 14:00 |
| rlandy | vs | 14:00 |
| rlandy | https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/93e97f1/logs/overcloud-controller-0/var/log/containers/mysql/mysqld.log.txt.gz | 14:00 |
| rlandy | 5 hours later | 14:00 |
| rlandy | the ref time is the same | 14:02 |
| ysandeep|rover | rlandy, i think timezone difference - job only ran for 1 hr 7 mins | 14:02 |
| rlandy | ysandeep|rover: undercloud and overcloud should match | 14:03 |
| rlandy | var/log/extra/chrony.txt.gz | 14:03 |
| rlandy | Ref time (UTC) : Tue May 17 12:38:45 2022 | 14:03 |
| rlandy | both show ref time | 14:03 |
| rlandy | but idk if that means actual time | 14:04 |
| ysandeep|rover | rlandy, mariadb.log vs mysqld.log - may be they report in different way - but not 100% sure without checking code | 14:05 |
| rlandy | Reference ID : C3BA0164 (bwntpo.bluewin.ch) | 14:07 |
| rlandy | Reference ID : D9B4D1D6 (ntp1a.versadns.com) | 14:07 |
| rlandy | us vs europe | 14:07 |
| rlandy | ysandeep|rover: in downstream we explicitly set ntp | 14:09 |
| * rlandy wonders if we should try that in upstream | 14:10 | |
| rlandy | in downstream the ref match | 14:10 |
| dasm | marios: where can i find overcloud logs on this job? | 14:10 |
| ysandeep|rover | dasm, https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/93e97f1/logs/overcloud-controller-0/ | 14:10 |
| marios | dasm: that job didn't get far enough for overcloud logs but yeah ysandeep|rover has an example | 14:11 |
| dasm | ysandeep|rover: thanks | 14:11 |
| rlandy | frenzy_friday: fyi quay push failing everywhere | 14:11 |
| marios | dasm: as i wrote on the gchat you need to go to logs/nodename/var/log for the logs on each | 14:12 |
| frenzy_friday | rlandy, master should be fixed now | 14:12 |
| dasm | marios: ack | 14:12 |
| frenzy_friday | https://review.rdoproject.org/r/c/rdo-jobs/+/41528 | 14:12 |
| frenzy_friday | rlandy, the cirobot user didnt have enough rights, dpawlik fixed it | 14:13 |
| rlandy | frenzy_friday: k- thanks | 14:16 |
| jm1 | arxcruz++ thank you, very helpful! :) | 14:16 |
| chandankumar | #oooq please remember the tempest homework. | 14:16 |
| rlandy | ysandeep|rover: jm1: sync for a few? | 14:16 |
| chandankumar | thanks arxcruz++++ nicely done. | 14:16 |
| rlandy | https://meet.google.com/qfp-gnad-xzq | 14:16 |
| arxcruz | chandankumar :* | 14:16 |
| rlandy | arxcruz++ - very helpful | 14:16 |
| bhagyashris | arxcruz, chandankumar ++ | 14:17 |
| * bhagyashris dinner brb | 14:17 | |
| dasm | arxcruz: marios: i'm sorry. i didn't want to sound harsh when was asking about logs | 14:25 |
| jm1 | https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/93e97f1/logs/undercloud/var/log/containers/stdouts/neutron_api.log.txt.gz | 14:25 |
| jm1 | https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/93e97f1/logs/overcloud-controller-0/var/log/containers/stdouts/neutron_api.log.txt.gz | 14:25 |
| arxcruz | dasm I didn't take it as being harsh | 14:26 |
| arxcruz | my knowledge is different from yours, when there's something related to network for example, i'm very bad at it | 14:26 |
| marios | dasm: didn't at all :D | 14:27 |
| marios | (you didn't sound harsh, i didn't take it as harsh) | 14:27 |
| jm1 | Tengu, marios: Any idea where undercloud gets its timezone from? Do we actually configure the tz when setting up undercloud nodes? | 14:30 |
| marios | jm1: quick grep says yes but would have to dig at that ./overcloud-resource-registry-puppet.j2.yaml: OS::TripleO::Services::Timezone: deployment/time/timezone-baremetal-ansible.yaml | 14:31 |
| marios | jm1: (tripleo-heat-templates ^^) | 14:31 |
| jm1 | marios: does this apply to undercloud as well? | 14:32 |
| marios | jm1: not sure without digging there but the one i linked to is specific to the overcloud | 14:32 |
| ysandeep|rover | https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/roles/Undercloud.yaml#L59 - undercloud have that service as well | 14:34 |
| marios | jm1: looks like it does | 14:34 |
| marios | ysandeep|rover: i came here to post that :) ./roles/Undercloud.yaml:88: - OS::TripleO::Services::Timezone | 14:34 |
| jm1 | ysandeep|rover, marios: ack, thanks! | 14:34 |
| marios | jm1: that is implemented in tripleo-ansible looks like fyi | 14:35 |
| dasm | arxcruz: marios thanks :) | 14:35 |
| ysandeep|rover | default is UTC: https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/time/timezone-baremetal-ansible.yaml#L30 | 14:35 |
| marios | jm1: fyi there https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_timezone | 14:35 |
| marios | jm1: (tripleo-heat-templates in that file ysandeep|rover points to is calling that ^^ tripleo-ansible role) | 14:36 |
| jm1 | ysandeep|rover, marios: no idea if this is still used but if it is, i guess it will use localtime https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/v1/undercloud_config.py#L502 | 14:40 |
| ysandeep|rover | rlandy, I spoke with luca from pidone... he says its the same issue because of slowness | 14:44 |
| ysandeep|rover | ~~~ | 14:44 |
| ysandeep|rover | <lmiccini> ysandeep|rover: seems the same issue with podman being slow | 14:44 |
| ysandeep|rover | <lmiccini> "warning: Unexpected result (error: Resource agent did not complete in time) was recorded for monitor of haproxy-bundle-podman-0 on overcloud-controller-0 at May 17 07:39:17 2022 | rc=1 id=haproxy-bundle-podman-0_last_failure_0" | 14:44 |
| ysandeep|rover | ~~~ | 14:44 |
| ysandeep|rover | jm1, this is still in play: https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/v1/undercloud_config.py#L502 | 14:46 |
| jm1 | ysandeep|rover: looks like undercloud uses localtime, not utc. "ansible-timezone Invoked with name=America/New_York hwclock=None" https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/93e97f1/logs/undercloud/var/log/extra/journal.txt.gz | 14:46 |
| jm1 | ysandeep|rover: but should be easy to fix, right? we could just add "undercloud_timezone: 'UTC'" to https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/v1/undercloud_config.py#L502 | 14:48 |
| jm1 | ysandeep|rover: sorry, here is correct link https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset001.yml | 14:48 |
| jm1 | ysandeep|rover: shall i create a patch which does this for all featuresets? | 14:49 |
| marios | jm1: yes this sounds right, or we could consider updating the tripleoclient too i mean the default in undercloud config | 14:50 |
| marios | jm1: since there is disparity between undercloud default and overcloud defaults | 14:50 |
| ysandeep|rover | jm1: looking at code - If we want to set timezone in undercloud - we can just mentioned that in undercloud.conf | 14:50 |
| ysandeep|rover | marios, https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/v1/undercloud_config.py#L502-L503 | 14:51 |
| jm1 | ysandeep|rover, marios: reason for putting it into config/general_config/featureset*.yml was that these files have "undercloud_ntp_servers:" as well | 14:51 |
| ysandeep|rover | looks like it read undercloud.conf or use the localtimezone.. we create our own undercloud.conf in ci - we can add undercloud_timezone in undercloud.conf | 14:53 |
| ysandeep|rover | but yeah if overcloud default is UTC.. we can change tripleoclient to default to utc as well - https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/time/timezone-baremetal-ansible.yaml#L30 | 14:54 |
| marios | jm1: k i just feel like it is a valid thing to propose i.e. we should align the default timezon on under/overcloud so setting the default in client would be valid | 14:54 |
| marios | jm1: if there is pushback for whatever reason we haven't thought of, then we can do featureset* change | 14:54 |
| ysandeep|rover | +1 to make changes in tripleoclient instead , here - https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/v1/undercloud_config.py#L503 | 14:55 |
| jm1 | marios, ysandeep|rover: i am fine with all of course. | 14:55 |
| jm1 | marios, ysandeep|rover: we completely change the undercloud anyway, dont we? | 14:56 |
| jm1 | marios, ysandeep|rover: what i mean is, since we change the undercloud node anyway, it should be safe to change tz to utc. operators would expect changes to undercloud nodes anyway | 14:59 |
| ysandeep|rover | jm1, marios the existing behavior on undercloud seems valid as per docs | 14:59 |
| ysandeep|rover | https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html-single/director_installation_and_usage/index | 14:59 |
| ysandeep|rover | ~~~ | 14:59 |
| ysandeep|rover | undercloud_timezone | 14:59 |
| ysandeep|rover | Host timezone for the undercloud. If you do not specify a timezone, director uses the existing timezone configuration. | 14:59 |
| ysandeep|rover | ~~~ | 14:59 |
| jm1 | ysandeep|rover: oh nice. so we better dont change the defaults ;) | 15:00 |
| * jm1 bbl | 15:02 | |
| ysandeep|rover | jm1: In this case i would just add undercloud_timezone in https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-deploy/templates/undercloud.conf.j2 for ci usecase. | 15:03 |
| jm1 | ysandeep|rover: will this only affect ci? or will customers be affected by this as well? | 15:04 |
| ysandeep|rover | jm1, this file is only used by us in ci / manual tripleo-quickstart user (only few developers) | 15:04 |
| jm1 | ysandeep|rover: ack, will have a break now and propose a patch later. or go ahead and do it, you decide ;) | 15:05 |
| ysandeep|rover | jm1, all yours I am almost eod :D | 15:07 |
| jm1 | ysandeep|rover: ack, then have a nice evening :) | 15:08 |
| rlandy | dviroel: will touch base with you re: multiple container reg logins | 15:14 |
| rlandy | no joy last night ... "assertion": "\"invalid username/password\" in registry_login_podman.results.0.stderr", | 15:14 |
| ysandeep|rover | rlandy, hopefully things will be better after vexx cpu overcommit reduction | 15:14 |
| rlandy | ysandeep|rover: there is always hope | 15:15 |
| * rlandy is tired of this | 15:15 | |
| dviroel | rlandy: ack, after all-hands i will go lunch, after that, i will be available to chat | 15:15 |
| rlandy | dviroel: have meeting with mike burns then - will ping afterwards | 15:16 |
| rlandy | let's try a single approach here | 15:16 |
| rlandy | ideally a role | 15:16 |
| rlandy | after we tested in downstream | 15:16 |
| dviroel | rlandy: ack | 15:20 |
| *** ysandeep|rover is now known as ysandeep|out | 15:24 | |
| frenzy_friday | hey ysandeep|out rlandy , quay jobs should be fixed now https://review.rdoproject.org/r/c/rdo-jobs/+/42773 , https://review.rdoproject.org/r/c/rdo-jobs/+/41528 | 15:34 |
| rlandy | frenzy_friday++ thanks | 15:34 |
| ysandeep|out | frenzy_friday++ thanks for working on that | 15:34 |
| frenzy_friday | dpawlik++ thanks for fixing the write permissions | 15:35 |
| ysandeep|out | rlandy, /me shutdown sequence need anything? | 15:37 |
| dasm | ysandeep|out: fyi, i'll be around on 05/20 for rr related stuff. | 15:37 |
| ysandeep|out | dasm: on recharge day ? | 15:37 |
| dasm | ysandeep|out: yes. i'm swapping the day with 05/31 | 15:38 |
| rlandy | ysandeep|out: no - need me to do anything other than spin on ovb in downstream | 15:38 |
| ysandeep|out | dasm: thanks! | 15:38 |
| dviroel | rlandy: chandankumar marios can you review these changes https://review.rdoproject.org/r/c/rdo-jobs/+/42792 and https://review.rdoproject.org/r/c/config/+/42892 - would be good to have it running this week, we should have a new ceph stable tag soon | 15:38 |
| marios | dviroel: sure adding to my list | 15:38 |
| ysandeep|out | dasm, we will invite you to rr handover sync in that case on thursday. | 15:39 |
| dviroel | marios: tks, you reviewed already actually, ;) | 15:39 |
| dasm | k | 15:39 |
| ysandeep|out | rlandy: no thanks, that's all | 15:39 |
| rlandy | ysandeep|out: thanks - have a good night | 15:39 |
| * ysandeep|out disappears | 15:40 | |
| rlandy | dviroel: done ... you can merge https://review.rdoproject.org/r/c/config/+/42892 when ready | 15:41 |
| rlandy | dviroel: your pipeline is called periodic-promote-ceph-daemon | 15:42 |
| rlandy | and your project template ceph-daemon-promotion-pipeline | 15:43 |
| marios | dviroel: ack revoted | 15:43 |
| rlandy | dviroel: correct? | 15:43 |
| rlandy | periodic-promote-ceph-daemon | 15:43 |
| dviroel | rlandy: double checking | 15:44 |
| rlandy | oh nvm - see it underneath | 15:44 |
| rlandy | dviroel: ^^ | 15:44 |
| rlandy | dviroel: should be ok | 15:45 |
| dviroel | rlandy: yes, following the other triggers pattern | 15:45 |
| dviroel | zuul magic | 15:45 |
| rlandy | yep - missed that line - it's fine | 15:45 |
| dviroel | rlandy++: great presentation btw | 16:02 |
| * dviroel lunch | 16:02 | |
| *** dviroel is now known as dviroel|lunch | 16:02 | |
| *** marios is now known as marios|out | 16:03 | |
| rlandy | dviroel|lunch: thanks | 16:07 |
| rlandy | lunch - brb | 16:07 |
| ade_lee | dviroel_, dasm , rlandy hey - can you gys remind me again where the code is where we do a dns lookup to make sure unbound is up -- I think for the ipa job iirc | 16:43 |
| dasm | not sure from top of my head | 16:45 |
| *** dviroel_ is now known as dviroel | 16:52 | |
| dviroel | ade_lee: hey, not sure which task you search for, there is 2 different roles, one for multinode https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/ipa-multinode | 16:58 |
| dviroel | ade_lee: another one to install freeipa, used on ovb jobs https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/freeipa-setup | 16:59 |
| dviroel | ade_lee: check the installation scripts | 16:59 |
| rlandy | one is standalone and one is ovb | 17:00 |
| rlandy | dviroel: ade_lee: ^^ | 17:00 |
| ade_lee | rlandy, dviroel dasm thanks | 17:04 |
| dviroel | rlandy: want to sync about container login | 17:40 |
| dviroel | ? | 17:40 |
| rlandy | dviroel: will ping in 5 | 18:04 |
| *** jpena is now known as jpena|off | 18:08 | |
| rlandy | rcastillo: pls join #rhos-delivery | 18:10 |
| rcastillo | rlandy: sec | 18:12 |
| rlandy | dviroel: sorry - will be with you in a few | 18:12 |
| rlandy | just sorting delivery | 18:12 |
| rcastillo | rlandy: vpn disconnected, I'm on there now | 18:13 |
| rlandy | dviroel: https://meet.google.com/hqi-sbcp-kmm?pli=1&authuser=0 | 18:16 |
| rlandy | dviroel: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/410062/1/playbooks/tripleo-rdo-base/container-login-test.yaml | 18:48 |
| rlandy | dviroel: dasm: review time | 19:00 |
| *** jm1|ruck is now known as jm1|rover | 19:10 | |
| dasm | https://review.rdoproject.org/r/c/config/+/42619 | 19:21 |
| dasm | https://review.rdoproject.org/r/c/rdo-jobs/+/42621 | 19:25 |
| dasm | https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/408117 | 19:33 |
| dasm | oh, i killed wrong window :/ | 19:39 |
| jm1 | rlandy: fyi result of discussion with sandeep https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/842218 | 19:39 |
| rlandy | jm1: thanks - hope that wasn't a rat hole discussion | 19:42 |
| dasm | rlandy: dviroel rcastillo pushed an update: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/408117 | 19:42 |
| rlandy | thanks - voted | 19:44 |
| dasm | thanks | 19:44 |
| rlandy | dviroel++++++++++++++++++++++++++++++++ | 19:44 |
| rlandy | you rock!!!! | 19:44 |
| rlandy | only hardcoding working | 19:45 |
| rlandy | finally | 19:45 |
| rlandy | login worked | 19:45 |
| dviroel | \o/ | 19:46 |
| rlandy | 17th time's a charm | 19:46 |
| dasm | dviroel++ | 19:46 |
| * jm1 out for today, have a nice evening :) | 19:47 | |
| rcastillo | jm1 | 19:48 |
| rcastillo | o/ | 19:48 |
| rcastillo | lunch brb | 20:01 |
| dasm | dviroel: you're right with commit msg. I updated it (just noticed that few minutes ago) | 20:10 |
| rlandy | jm1; ysandeep|out: just fyi - I am rebuilding the 16.2 containers with a new base image | 20:17 |
| rlandy | in case there are failures | 20:17 |
| rlandy | will leave the patch info | 20:17 |
| dviroel | dasm: thanks :) | 20:17 |
| dasm | uefi across the (upstream) board is merged | 20:29 |
| dasm | \( ﾟヮﾟ)/ | 20:29 |
| dviroel | (⊙‿⊙) | 20:30 |
| rlandy | jm1; ysandeep|out: program doc updated per now | 20:34 |
| dasm | rlandy: answering your question wrt bios-legacy jobs: https://review.rdoproject.org/r/c/testproject/+/42319 | 20:34 |
| dasm | that's testproject testing those | 20:35 |
| rlandy | k | 20:35 |
| dasm | i also took down cs9-master from it | 20:35 |
| rlandy | great | 20:35 |
| dasm | so we're having only cs8 & cs9 wallaby jobs | 20:35 |
| * rlandy needs to run out for errand - will review when biab | 20:35 | |
| dasm | o/ | 20:35 |
| *** rlandy is now known as rlandy|biab | 20:35 | |
| rcastillo | back | 20:58 |
| *** rlandy|biab is now known as rlandy | 21:07 | |
| rlandy | bhagyashris: reviewed patch - looks goo - minus the linter error | 21:19 |
| * dasm leaves | 21:49 | |
| dasm | o/ | 21:49 |
| *** dasm is now known as dasm|off | 21:49 | |
| *** rlandy is now known as rlandy|out | 22:02 | |
| *** dviroel is now known as dviroe|out | 22:12 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!