| *** ysandeep|away is now known as ysandeep | 05:39 | |
| zbr | vote on https://review.opendev.org/c/openstack/project-config/+/806012 | 06:38 |
|---|---|---|
| *** amoralej|off is now known as amoralej | 07:02 | |
| *** jpena|off is now known as jpena | 07:37 | |
| *** ykarel is now known as ykarel|lunch | 08:28 | |
| frenzy_friday | Hey zbr, https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/787502/ is failing in the linters but is passing locally. Do you have an idea why? | 09:17 |
| *** sshnaidm|afk is now known as sshnaidm | 09:38 | |
| *** ykarel|lunch is now known as ykarel | 10:18 | |
| chandankumar | sshnaidm: ysandeep arxcruz for interview coordination, how we want to coordinate? | 10:19 |
| chandankumar | Are we planning to time box it? | 10:19 |
| sshnaidm | chandankumar, I think better in that channel | 10:19 |
| sshnaidm | arxcruz, ysandeep ^^ | 10:21 |
| *** rlandy is now known as rlandy|rover | 10:38 | |
| *** ysandeep is now known as ysandeep|mtg | 10:57 | |
| *** rlandy is now known as rlandy_testbox | 10:58 | |
| *** dviroel|out is now known as dviroel|ruck | 11:14 | |
| *** jpena is now known as jpena|lunch | 11:42 | |
| * dviroel|ruck brb in a couple of mins | 11:58 | |
| *** dviroel|ruck is now known as dviroel|ruck|brb | 11:58 | |
| *** dviroel|ruck|brb is now known as dviroel|ruck | 12:20 | |
| *** jpena|lunch is now known as jpena | 12:38 | |
| arxcruz | bhagyashris: i need to take my daughter at school, i'll be late for the scrum | 12:47 |
| bhagyashris | arxcruz, ack | 12:47 |
| chandankumar | rlandy|rover: weshay|ruck just wanted to update on container build https://review.rdoproject.org/zuul/builds?job_name=tripleo-build-containers-stream9-development&project=testproject without ceph | 12:49 |
| chandankumar | trying now with ceph | 12:49 |
| weshay|ruck | chandankumar, w/ ceph.. re: a deploy or image build... we don't build ceph | 12:54 |
| chandankumar | *ceph-common | 12:56 |
| *** amoralej is now known as amoralej|lunch | 12:59 | |
| bhagyashris | akahat, chandankumar rlandy_testbox zbr sshnaidm scrum time | 13:00 |
| bhagyashris | rlandy|rover, ^ | 13:01 |
| bhagyashris | Review list | 13:04 |
| bhagyashris | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34706/ | 13:04 |
| bhagyashris | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34995 | 13:04 |
| bhagyashris | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/35008 | 13:05 |
| rlandy|rover | https://review.opendev.org/c/openstack/tripleo-quickstart/+/804399 | 13:06 |
| frenzy_friday | https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/787502/ - need some help on the linetrs - it passes on local | 13:09 |
| bhagyashris | https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/787502 | 13:11 |
| bhagyashris | https://review.opendev.org/c/openstack/tripleo-quickstart/+/791486 | 13:26 |
| *** ysandeep|mtg is now known as ysandeep | 13:29 | |
| zbr | is "Failed to store expired repos cache" a redherring? | 13:31 |
| sshnaidm | probably | 13:31 |
| ysandeep | akahat, hey o/ do you have fs001 results for train branch for your operator patches? I want to compare them with 16.2 results | 13:49 |
| *** amoralej|lunch is now known as amoralej | 13:50 | |
| ysandeep | akahat: found it! https://logserver.rdoproject.org/17/804117/7/openstack-experimental/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train-branch/bb5c657/job-output.txt | 13:52 |
| akahat | ysandeep, i've for fs02 and fs35: https://review.rdoproject.org/r/c/testproject/+/31612 | 13:57 |
| akahat | ysandeep, well network roles are skipped in fs01-train. | 13:58 |
| ysandeep | akahat: thanks! i got now - network_provision is w+ and baremetal_provision is ussuri+ | 13:58 |
| ysandeep | https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset001.yml#L216-L225 | 13:58 |
| akahat | :) | 13:59 |
| ysandeep | akahat: you have a suggestion from sagi on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/804117 | 14:01 |
| rlandy|rover | ysandeep; anything required for downstream today? | 14:06 |
| rlandy|rover | if not will just look into the couple failing on 17 line | 14:06 |
| rlandy|rover | I know Pooja is out for a bit | 14:07 |
| rlandy|rover | dviroel|ruck: woohoo master promotion | 14:07 |
| dviroel|ruck | yes :) | 14:08 |
| rlandy|rover | dviroel|ruck: we should check the component line - I can split that task with you | 14:08 |
| rlandy|rover | dviroel|ruck: know how to check the dates on those lines? | 14:09 |
| dviroel|ruck | rlandy|rover: ok, yeah, i think so, was just looking glance errors here https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario001-standalone-glance-master | 14:09 |
| rlandy|rover | dviroel|ruck: k - I'll take train, ussuri, victoria | 14:10 |
| chandankumar | rlandy|rover: arxcruz kopecmartin we need to bring back the tempest container for operator work | 14:10 |
| rlandy|rover | you good with master/wallaby? | 14:10 |
| dviroel|ruck | rlandy|rover: yes | 14:10 |
| rlandy|rover | cool | 14:10 |
| chandankumar | rlandy|rover: arxcruz kopecmartin https://review.opendev.org/c/openstack/tripleo-common/+/801987 | 14:11 |
| ysandeep | rlandy|rover: Didn't get a chance to ruck/rove downstream today, I was neck down trying to figure out bm deployment templates, which I finally did today.. | 14:11 |
| ysandeep | rlandy|rover, fyi:- made some improvements | 14:12 |
| ysandeep | * While switching 17 jobs to use networkdatav2, Noticed we are not deleting redis/ovndb ports during "openstack overcloud delete overcloud":- | 14:12 |
| ysandeep | * Spoke with rabi, he cherrypicked the fix to wallaby https://review.opendev.org/c/openstack/tripleo-ansible/+/806083, I tested the change in my environemnt it works. | 14:12 |
| ysandeep | * Backported a feature for wallaby "node unprovisioning during overcloud delete in wallaby" https://review.opendev.org/c/openstack/python-tripleoclient/+/806079 | 14:12 |
| weshay|ruck | ysandeep++ | 14:15 |
| rlandy|rover | sec | 14:15 |
| kopecmartin | chandankumar: then we need to figure out who's gonna maintain it, how is it tested, built etc .. because at this moment i don't know anything about it | 14:15 |
| arxcruz | chandankumar: why god... why? | 14:18 |
| chandankumar | arxcruz: it is required for operator work | 14:19 |
| chandankumar | to run tempest | 14:19 |
| chandankumar | openstack deployment using https://github.com/openstack-k8s-operators/osp-director-operator | 14:19 |
| chandankumar | it is totally new work | 14:19 |
| rlandy|rover | ysandeep: looks good - will check it out | 14:28 |
| rlandy|rover | chandankumar: k - we will be going down the path of testing that soon enough | 14:36 |
| dviroel|ruck | rlandy|rover: will testproject all components that run scenario001, they have been failing a lot, for different reasons. It is possible to see the same error in different components too, but not more than once in the component. | 14:51 |
| rlandy|rover | dviroel|ruck: didn't we skip some scenario001 tests? | 14:54 |
| rlandy|rover | we may need to change the jobs to [] | 14:55 |
| rlandy|rover | to include all | 14:55 |
| rlandy|rover | in skiplist | 14:55 |
| rlandy|rover | rather than having you add every component test | 14:55 |
| rlandy|rover | if it's the same failure | 14:55 |
| dviroel|ruck | isn't the same failure. actually, there is differents failures happening on scenario001, we just skip some tempest test. | 14:56 |
| sshnaidm | zbr, I don't see install_ansible_collections functions is executed anywhere, I think it's just defined, but not executed | 14:57 |
| sshnaidm | zbr, in install-deps.sh | 14:57 |
| dviroel|ruck | will continue investigate after lunch, brb | 14:58 |
| * dviroel|ruck lunch | 14:58 | |
| zbr | install deps itself is sourced, it only has functions inside not direct actions, is like a bash library. | 15:02 |
| zbr | sshnaidm: installation of collections happens inside qs boostrap(), which is executed only when qs is called with --boostrap. | 15:04 |
| zbr | we should either ensure we do this, or call install_ansible_collections from another appropiate place. | 15:05 |
| sshnaidm | zbr, I don't think we use quickstart.sh in ovb jobs, only install_deps.sh, so it's not executed | 15:06 |
| rlandy|rover | dviroel|ruck: k - ping if you need help with components | 15:06 |
| sshnaidm | zbr, can you see in logs where it's executed? Maybe worth to add -v to ansible-galaxy | 15:07 |
| zbr | any problem adding install_ansible_collections at the end after install_package_deps_via_bindep? it should work unless ansible is missing. | 15:07 |
| *** amoralej is now known as amoralej|off | 15:10 | |
| chandankumar | rlandy|rover: ysandeep meeting time | 15:29 |
| ysandeep | chandankumar, joining in a minute, sry jumping from another mtg | 15:31 |
| *** jpena is now known as jpena|off | 15:39 | |
| chandankumar | ysandeep: rlandy|rover frenzy_friday https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/promoter/templates/dlrn-promoter-service.sh.j2 | 15:46 |
| * dviroel|ruck back | 15:48 | |
| rlandy|rover | https://zuul.opendev.org/t/openstack/builds?job_name=puppet-openstack-unit-6.21-centos-8-stream | 16:13 |
| rlandy|rover | weshay|ruck: ^^ we just ping on #tripleo for this? | 16:13 |
| rlandy|rover | puppet-tripleo failures in gate | 16:13 |
| weshay|ruck | can email takashi | 16:15 |
| rlandy|rover | train component line is good | 16:16 |
| rlandy|rover | 6 days agotripleopromoted-components9 hours ago - trouble | 16:33 |
| ysandeep | rlandy|rover: with 16.2 bm envd - looks like there is some hardware issue, not because of operator work | 16:38 |
| ysandeep | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/5231268/logs/undercloud/home/zuul/overcloud_import_nodes.log | 16:38 |
| rlandy|rover | k - thanks | 16:38 |
| ysandeep | I will look tomorrow o/ | 16:39 |
| *** ysandeep is now known as ysandeep|away | 16:39 | |
| chandankumar | rlandy|rover: please use the older one promoter | 16:39 |
| chandankumar | rlandy|rover: it needs some fixes | 16:39 |
| rlandy|rover | chandankumar: np - thanks | 16:42 |
| rlandy|rover | chandankumar: fyi - I am out tomorrow | 16:42 |
| rlandy|rover | so no rush | 16:42 |
| dviroel|ruck | rlandy|rover: scenario001 seems to be a real issue in master components | 16:45 |
| dviroel|ruck | rlandy|rover: but not all issues are the same | 16:46 |
| rlandy|rover | dviroel|ruck: k | 16:53 |
| rlandy|rover | lunch - brb | 16:53 |
| dviroel|ruck | rlandy|rover: here https://review.rdoproject.org/r/c/testproject/+/34983, common component seems to have a similar failure to https://bugs.launchpad.net/tripleo/+bug/1940866 | 17:46 |
| dviroel|ruck | adding as a comment | 17:46 |
| rlandy|rover | ok | 17:47 |
| rlandy|rover | so add to skiplist? | 17:47 |
| dviroel|ruck | yeah, probably. There are other runs that says that failed in tempest execution, but there is no log outputs: https://logserver.rdoproject.org/83/34983/8/check/periodic-tripleo-ci-centos-8-scenario001-standalone-cinder-master/3c12df2/logs/ | 17:56 |
| dviroel|ruck | maybe if we skip those tests, we might see these other ones passing too, maybe | 17:56 |
| *** rlandy is now known as rlandy_testbox | 17:57 | |
| dviroel|ruck | rlandy|rover: adding components to skiplist https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/806261 | 18:39 |
| rlandy|rover | dviroel|ruck: sorry - getting a lesson on cockpit editing | 18:45 |
| * rlandy|rover review | 18:45 | |
| rlandy|rover | dviroel|ruck: you'll probably need to add all of wallaby | 18:45 |
| rlandy|rover | if it's too much of a pain you can skip all jobs with [] | 18:46 |
| rlandy|rover | but approving this for now | 18:46 |
| rlandy|rover | let's merge and see if it clears the lines | 18:46 |
| rlandy|rover | thanks | 18:46 |
| dviroel|ruck | ok :) | 18:51 |
| rlandy|rover | https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/268195 Add promotions in descending order | 19:17 |
| rlandy|rover | weshay|ruck: ^^ | 19:18 |
| *** sshnaidm is now known as sshnaidm|afk | 19:32 | |
| rlandy|rover | dviroel|ruck: hi - fyi - I am PTO tomorrow | 20:46 |
| rlandy|rover | weshay|ruck will be around though | 20:47 |
| weshay|ruck | :) | 20:47 |
| weshay|ruck | enjoy it | 20:47 |
| dviroel|ruck | rlandy|rover: ok | 20:47 |
| rlandy|rover | thanks - mom says hi | 20:47 |
| dviroel|ruck | hi o/ | 20:48 |
| dviroel|ruck | rlandy|rover: https://logserver.rdoproject.org/83/34983/9/check/periodic-tripleo-ci-centos-8-scenario001-standalone-cloudops-master/0e503f9/logs/ | 20:50 |
| dviroel|ruck | seen this mysql errors more often | 20:50 |
| dviroel|ruck | seems to be another issue with scenario001 | 20:51 |
| rlandy|rover | wow scenario001 is a true winner | 20:52 |
| rlandy|rover | 69ca146fc05d 192.168.24.1:8787/tripleomaster/openstack-mariadb:64e0281eb4c358d842dfdd238d9486dd-updated-20210826192259 /container_puppet... 27 minutes ago Exited (6) 17 minutes ago mysql_wait_bundle | 20:53 |
| rlandy|rover | yeah we were tracking that somewhere | 20:54 |
| rlandy|rover | looking at a tripleo line failure | 20:54 |
| rlandy|rover | compares | 20:54 |
| rlandy|rover | 22 days agotraincurrent-tripleocentos-7 - will also check that out | 20:55 |
| rlandy|rover | 6 days agotripleopromoted-components14 hours ago | 20:56 |
| weshay|ruck | rlandy|rover, I think c7-train was all passing when we looked earlier in the week | 20:56 |
| rlandy|rover | weshay|ruck: it was | 20:56 |
| rlandy|rover | which is why its strange it did not promote | 20:57 |
| rlandy|rover | that was wednesday | 20:57 |
| rlandy|rover | weshay|ruck; line is green, green, green https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable4-centos7 | 20:58 |
| rlandy|rover | must be criteria mismatch | 20:58 |
| rlandy|rover | or the promoter for c7 not running | 20:58 |
| rlandy|rover | weshay|ruck: if 10.0.148.59 is still valid as the c7 promoter, I can't access it | 21:01 |
| rlandy|rover | doesn't ping | 21:02 |
| weshay|ruck | rlandy|rover, it's not valid.. c7 should be on promoter.rdo | 21:02 |
| rlandy|rover | k - checking there | 21:02 |
| weshay|ruck | rlandy|rover, probably not even configured to run | 21:03 |
| weshay|ruck | lolz | 21:03 |
| rlandy|rover | it's nowhere | 21:03 |
| * rlandy|rover gets on promoter | 21:04 | |
| weshay|ruck | ya.. see if the script is kicking it off | 21:04 |
| rlandy|rover | let's see what's supposed to run there | 21:04 |
| rlandy|rover | 22 long days | 21:04 |
| rlandy|rover | with bright green runs :) | 21:04 |
| rlandy|rover | weshay|ruck: yep not in ci-scripts/dlrnapi_promoter/dlrn-promoter.sh on the actual promoter box | 21:09 |
| rlandy|rover | adding and adding config and restarting | 21:09 |
| rlandy|rover | dviroel|ruck: weshay|ruck; added train c7 to rdo promoter and restarted | 21:14 |
| rlandy|rover | in case you see any issues tomorrow | 21:14 |
| weshay|ruck | k | 21:14 |
| weshay|ruck | thanks | 21:14 |
| weshay|ruck | would love c7 to die | 21:14 |
| dviroel|ruck | ok | 21:14 |
| rlandy|rover | ye - should not require much love at this point - hopefully it just runs | 21:15 |
| rlandy|rover | criteria seemed up to date | 21:15 |
| rlandy|rover | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-ussuri&pipeline=openstack-component-tripleo | 21:16 |
| rlandy|rover | ha ok - that's why it's not promoting | 21:17 |
| rlandy|rover | 2021-08-21 21:19:43.949813 | primary | TASK [os_tempest : Ping router ip address] ************************************* | 21:18 |
| rlandy|rover | interesting - we had this issue in 17 | 21:18 |
| rlandy|rover | checking if it's the same | 21:18 |
| rlandy|rover | openvswitch ovn/ovs | 21:18 |
| dviroel|ruck | some weird outputs here: https://logserver.rdoproject.org/83/34983/9/check/periodic-tripleo-ci-centos-8-scenario001-standalone-common-master/1a28102/logs/undercloud/var/log/containers/stdouts/rabbitmq-bundle.log.txt.gz | 21:42 |
| dviroel|ruck | Looks like this is not happening in jobs that succeeded ^ | 21:49 |
| rlandy|rover | dviroel|ruck: going to log a bug for failing periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-ussuri | 21:52 |
| rlandy|rover | looks similar to a bugzilla we raised | 21:53 |
| rlandy|rover | can you ping ykarel with that bug | 21:53 |
| rlandy|rover | I think he updated OVN | 21:53 |
| dviroel|ruck | ok, sure | 21:53 |
| rlandy|rover | thanks | 21:53 |
| rlandy|rover | will post bug here - juts collecting info | 21:53 |
| dviroel|ruck | ok, I will ping tomorrow | 21:54 |
| rlandy|rover | dviroel|ruck: https://bugs.launchpad.net/tripleo/+bug/1941802 - we fixed the bugzilla referenced by upgrading openvswitch | 22:11 |
| dviroel|ruck | ack | 22:13 |
| * dviroel|ruck dinner and out | 22:18 | |
| *** dviroel|ruck is now known as dviroel|out | 22:18 | |
| *** rlandy|rover is now known as rlandy|rover|bbl | 22:19 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!