| *** jmasud has quit IRC | 00:05 | |
| *** jmasud has joined #oooq | 00:07 | |
| *** jmasud has quit IRC | 00:09 | |
| *** rlandy has quit IRC | 00:28 | |
| *** valleedelisle is now known as _dvd | 01:14 | |
| *** jmasud has joined #oooq | 01:24 | |
| *** udesale has joined #oooq | 01:42 | |
| *** apetrich has quit IRC | 02:09 | |
| *** jmasud has quit IRC | 03:03 | |
| *** ykarel|away has joined #oooq | 03:12 | |
| *** ratailor has joined #oooq | 04:35 | |
| *** jmasud has joined #oooq | 04:37 | |
| *** skramaja has joined #oooq | 05:04 | |
| *** pojadhav|away is now known as pojadhav|rover | 05:11 | |
| *** jmasud has quit IRC | 05:19 | |
| *** jmasud has joined #oooq | 05:28 | |
| *** jmasud has quit IRC | 05:33 | |
| *** marios has joined #oooq | 05:36 | |
| *** jfrancoa has joined #oooq | 05:36 | |
| *** slaweq_ has joined #oooq | 05:47 | |
| *** jmasud has joined #oooq | 05:53 | |
| *** jpodivin has joined #oooq | 05:55 | |
| *** skramaja_ has joined #oooq | 06:06 | |
| *** skramaja has quit IRC | 06:06 | |
| *** amoralej has joined #oooq | 06:08 | |
| *** pojadhav|rover is now known as pojadhav|dentist | 06:22 | |
| *** ysandeep|away is now known as ysandeep | 06:28 | |
| *** slaweq_ has quit IRC | 06:28 | |
| *** apetrich has joined #oooq | 06:34 | |
| *** dsneddon has quit IRC | 06:41 | |
| *** ykarel|away is now known as ykarel | 06:43 | |
| *** dsneddon has joined #oooq | 06:44 | |
| *** dsneddon has quit IRC | 06:55 | |
| *** dsneddon has joined #oooq | 06:57 | |
| *** pojadhav|dentist is now known as pojadhav|rover | 07:06 | |
| *** ratailor_ has joined #oooq | 07:39 | |
| *** ratailor has quit IRC | 07:40 | |
| akahat|ruck | arxcruz, chandankumar o/ | 07:44 |
|---|---|---|
| akahat|ruck | arxcruz, chandankumar periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby job have some tempest test failures | 07:44 |
| *** tosky has joined #oooq | 07:44 | |
| akahat|ruck | but those are not consistent.. https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby | 07:45 |
| *** pojadhav|rover is now known as pojadhav|lunch | 07:45 | |
| akahat|ruck | different tests are failing in each job | 07:45 |
| *** jpenag is now known as jpena | 07:51 | |
| *** jbadiapa has joined #oooq | 07:52 | |
| *** ysandeep is now known as ysandeep|lunch | 07:57 | |
| *** slaweq_ has joined #oooq | 08:00 | |
| *** saneax has joined #oooq | 08:02 | |
| *** holser has joined #oooq | 08:09 | |
| *** derekh has joined #oooq | 08:10 | |
| *** slaweq_ has quit IRC | 08:12 | |
| *** ysandeep|lunch is now known as ysandeep | 08:24 | |
| *** jmasud has quit IRC | 08:26 | |
| *** pojadhav|lunch is now known as pojadhav|rover | 08:38 | |
| *** derekh has quit IRC | 08:41 | |
| *** slaweq_ has joined #oooq | 08:42 | |
| *** slaweq_ is now known as slawe | 08:42 | |
| *** slawe is now known as slaweq | 08:42 | |
| *** ratailor__ has joined #oooq | 08:56 | |
| *** ratailor_ has quit IRC | 08:58 | |
| *** slaweq has quit IRC | 09:10 | |
| akahat|ruck | sshnaidm, ping | 09:33 |
| akahat|ruck | sshnaidm, we are seeing some Connection time out error while pushing images to the undercloud | 09:33 |
| akahat|ruck | sshnaidm, https://2319feb14e831f87c231-4854ed941a7816d1225c43ae9b456d0e.ssl.cf1.rackcdn.com/788372/1/gate/tripleo-ci-centos-8-undercloud-upgrade-ussuri/913ab62/logs/undercloud/home/zuul/undercloud_install.log | 09:34 |
| akahat|ruck | https://2319feb14e831f87c231-4854ed941a7816d1225c43ae9b456d0e.ssl.cf1.rackcdn.com/788372/1/gate/tripleo-ci-centos-8-undercloud-upgrade-ussuri/913ab62/logs/undercloud/var/log/tripleo-container-image-prepare.log | 09:35 |
| akahat|ruck | pojadhav|rover, is it known bug? ^^ | 09:35 |
| pojadhav|rover | akahat|ruck, I haven't seen this before | 09:37 |
| akahat|ruck | marios, o/ around? | 09:45 |
| *** dpawlik has quit IRC | 09:48 | |
| *** dpawlik has joined #oooq | 09:48 | |
| akahat|ruck | pojadhav|rover, fyi: https://bugs.launchpad.net/tripleo/+bug/1927075 | 09:55 |
| openstack | Launchpad bug 1927075 in tripleo "Undercloud: Failed to push containers to undercloud registry" [Critical,New] | 09:55 |
| pojadhav|rover | akahat|ruck, ack | 09:55 |
| marios | suip akahat|ruck | 10:02 |
| marios | (sup) | 10:02 |
| akahat|ruck | marios, hey.. undercloud-upgrad-ussuri job is failing to push containers to local registry. | 10:04 |
| akahat|ruck | i thought you might know about it. I've open bz for it ^^ | 10:04 |
| sshnaidm | akahat|ruck, looking | 10:21 |
| sshnaidm | akahat|ruck, it's not pushing to local registry, it's pulling from content provider | 10:23 |
| sshnaidm | DEBUG:urllib3.connectionpool:Starting new HTTPS connection (4): 10.4.70.121:5001\nDEBUG:urllib3.util.retry | 10:23 |
| sshnaidm | 10.4.70.121 is not a local registry, it's "train" content provider: https://2319feb14e831f87c231-4854ed941a7816d1225c43ae9b456d0e.ssl.cf1.rackcdn.com/788372/1/gate/tripleo-ci-centos-8-undercloud-upgrade-ussuri/913ab62/zuul-info/inventory.yaml | 10:24 |
| sshnaidm | registry_ip_address_branch: | 10:24 |
| sshnaidm | train: 10.4.70.121 | 10:24 |
| sshnaidm | ussuri: 198.72.124.44 | 10:24 |
| marios | akahat|ruck: https://bugs.launchpad.net/tripleo/+bug/1927075/comments/1 | 10:24 |
| openstack | Launchpad bug 1927075 in tripleo "Undercloud: Failed to push containers to undercloud registry" [Critical,New] | 10:24 |
| sshnaidm | akahat|ruck, marios is right, it's an infra problem. Look at IP of this host: hosts: | 10:25 |
| sshnaidm | primary: | 10:25 |
| sshnaidm | ansible_connection: ssh | 10:25 |
| sshnaidm | ansible_host: 198.72.124.41 | 10:25 |
| sshnaidm | you see that ussuri provider and this host are from the same network: 198.72.124.41 | 10:26 |
| sshnaidm | and train provider is from different network: 10.4.70.121 | 10:26 |
| sshnaidm | also with internal IP | 10:26 |
| sshnaidm | it shouldn't be so, parent and child jobs in infra should be on the same cloud | 10:27 |
| sshnaidm | but when retry limit happens, than one of hosts can be spawned on a different cloud, it's known issue in infra and I think they had a patch to fix that, not sure if it's merged | 10:27 |
| *** amoralej is now known as amoralej|afk | 10:28 | |
| akahat|ruck | sshnaidm, marios thank you for clarification. | 10:29 |
| akahat|ruck | I'll keep open this bug for while if i've seen any more issues related to this. | 10:30 |
| akahat|ruck | sshnaidm++ marios++ | 10:30 |
| sshnaidm | akahat|ruck, for clarification, open info in both providers. train provider: "cloud: limestone", ussuri provider: "cloud: inap" | 10:34 |
| sshnaidm | it's better to raise this to infra | 10:34 |
| akahat|ruck | sshnaidm, okay | 10:35 |
| sshnaidm | akahat|ruck, is "TestNetworkBasicOps.test_hotplug_nic" failures a known issue? | 10:39 |
| sshnaidm | I saw it in different patches | 10:39 |
| akahat|ruck | sshnaidm, yes. | 10:40 |
| akahat|ruck | sshnaidm, for now we added it to skip list and there is active cix for it: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/788818 | 10:41 |
| sshnaidm | akahat|ruck, but this skiplist is for periodic jobs only, isn't it? | 10:42 |
| sshnaidm | I see it failing in patches today | 10:42 |
| sshnaidm | akahat|ruck, https://018fea8e11afc40d13ea-4306e1de58933cdc6e67f822d23a8204.ssl.cf2.rackcdn.com/788922/1/check/tripleo-ci-centos-8-containers-multinode/4d9c666/logs/stackviz/index.html#/testrepository.subunit | 10:43 |
| *** amoralej|afk is now known as amoralej | 10:48 | |
| akahat|ruck | i'm really not sure it is only for periodic | 10:48 |
| *** dsneddon has quit IRC | 10:57 | |
| *** dsneddon has joined #oooq | 10:57 | |
| *** holser has quit IRC | 11:15 | |
| pojadhav|rover | sshnaidm, akahat|ruck : we can add all the failed jobs for "TestNetworkBasicOps.test_hotplug_nic" to https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml#L1822-L1838 | 11:18 |
| *** slaweq has joined #oooq | 11:20 | |
| sshnaidm | I think TestNetworkBasicOps.test_port_security_macspoofing_port started to fail too | 11:20 |
| sshnaidm | for now I see it fails on wallaby: tripleo-ci-centos-8-containers-multinode-wallaby | 11:21 |
| sshnaidm | or here for example, wallaby patch: https://review.opendev.org/c/openstack/python-tripleoclient/+/788922 | 11:22 |
| sshnaidm | akahat|ruck, pojadhav|rover I think we need a bug for it too | 11:23 |
| pojadhav|rover | sshnaidm, ack | 11:23 |
| akahat|ruck | sshnaidm, pojadhav|rover already added by weshay|ruck https://opendev.org/openstack/openstack-tempest-skiplist/commit/f746510689fde3ea46b661890fca50a616cc1b83 | 11:25 |
| sshnaidm | releases: | 11:25 |
| sshnaidm | - name: master | 11:25 |
| sshnaidm | it should be also wallaby | 11:25 |
| akahat|ruck | sshnaidm, ah.. okay. i'll add it. | 11:27 |
| pojadhav|rover | sshnaidm, akahat|ruck : I will add wallaby release. | 11:27 |
| akahat|ruck | pojadhav|rover, okay. please go ahead. | 11:27 |
| *** slaweq has quit IRC | 11:31 | |
| *** jpena is now known as jpena|lunch | 11:33 | |
| *** derekh has joined #oooq | 11:35 | |
| *** holser has joined #oooq | 11:37 | |
| *** holser has quit IRC | 11:40 | |
| *** holser has joined #oooq | 11:40 | |
| *** rlandy has joined #oooq | 11:41 | |
| *** dsneddon has quit IRC | 11:41 | |
| *** dsneddon has joined #oooq | 11:42 | |
| rlandy | pojadhav|rover: akahat|ruck: hey how are things ruck/rover wise | 11:48 |
| pojadhav|rover | rlandy, FS01, FS20 and FS35 (integration main line) confusing.. consistent failures but having inconsistent results :( | 11:49 |
| pojadhav|rover | for FS01, one failed with https://review.rdoproject.org/zuul/build/f10d7c2ef89744c9b8fa7bb44bdb8368/log/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 11:50 |
| pojadhav|rover | other failed with https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/e13f0cd/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz | 11:51 |
| rlandy | pojadhav|rover: let me check | 11:52 |
| pojadhav|rover | yep | 11:52 |
| *** dviroel is now known as dviroel|nho | 11:54 | |
| * pojadhav|rover brb | 11:55 | |
| *** tosky_ has joined #oooq | 11:56 | |
| *** tosky has quit IRC | 11:57 | |
| rlandy | 2021-05-03 20:03:47 | 2021-05-03 20:03:47.730147 | fa163e77-477e-6934-40e9-00000000003d | FATAL | Wait for connection to become available | 192.168.24.29 | error={"changed": false, "elapsed": 2419, "msg": "timed out waiting for ping module test success: Data could not be sent to remote host \"192.168.24.29\". Make sure this host can be reached over ssh: ssh: connect to host 192.168.24.29 port 22: No route to host\r\n"} | 11:58 |
| rlandy | ^^ error first one | 11:58 |
| rlandy | the second is tempest error | 11:58 |
| rlandy | which maens it must have gotten through deploy | 11:59 |
| rlandy | weshay|ruck added a bunch of tests to the skiplist yesterday | 11:59 |
| * akahat|ruck back in 1 hr | 12:02 | |
| rlandy | tempest.api.compute.admin.test_volumes_negative.VolumesAdminNegativeTest.test_update_attached_volume_with_nonexistent_volume_in_body [309.056798s] ... FAILED | 12:03 |
| rlandy | so yep - you have two diff failures | 12:03 |
| *** tosky_ is now known as tosky | 12:04 | |
| rlandy | pojadhav|rover: so far - think the deploy failure is sporadic so far | 12:06 |
| rlandy | network | 12:06 |
| rlandy | tempest is legit | 12:06 |
| rlandy | Failed: 19 | 12:08 |
| rlandy | ^^ that's where the issue is | 12:08 |
| rlandy | tempest.api.compute.admin.test_volumes_negative.VolumesAdminNegativeTest.test_update_attached_volume_with_nonexistent_volume_in_body [311.898480s] ... FAILED | 12:08 |
| rlandy | pojadhav|rover: so yeah ... forget the deploy failure - unless you see it again - tempest is the problem | 12:11 |
| rlandy | 3} tempest.api.compute.admin.test_volumes_negative.VolumesAdminNegativeTest.test_update_attached_volume_with_nonexistent_volume_in_body [310.966125s] ... FAILED | 12:14 |
| pojadhav|rover | rlandy, weshay|ruck.. so should we skip this test " tempest.api.compute.admin.test_volumes_negative.VolumesAdminNegativeTest.test_update_attached_volume" too..? | 12:15 |
| *** derekh has quit IRC | 12:16 | |
| rlandy | I don't see any LPs with these tests | 12:17 |
| rlandy | checking the skiplist | 12:17 |
| weshay|ruck | pojadhav|rover, in periodic.. just the periodic.. that is fine | 12:19 |
| weshay|ruck | 3rd party looks back online https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset001&job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 | 12:20 |
| pojadhav|rover | yeah | 12:21 |
| rlandy | https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml | 12:21 |
| rlandy | nothing in skiplist atm | 12:21 |
| *** amoralej is now known as amoralej|lunch | 12:22 | |
| rlandy | ysandeep: are we ready to merge the cirros changes ofr downstream - reviewed tham | 12:26 |
| rlandy | them | 12:26 |
| ysandeep | rlandy, we have to merge them coz sc010 and sc012 are complaining because we merged similiar changes in tqe.. | 12:27 |
| ysandeep | just wondering if that kernel panic bug for new cirros image is solved? | 12:28 |
| weshay|ruck | akahat|ruck, you have a handle on the cirros issue? | 12:29 |
| ysandeep | weshay|ruck, rlandy fyi.. https://bugzilla.redhat.com/show_bug.cgi?id=1956758 , opening another one for https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-full-tempest-scenario-rhos-16.2/06615e7/logs/undercloud/var/log/tempest/stestr_results.html | 12:30 |
| openstack | bugzilla.redhat.com bug 1956758 in openstack-neutron "periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-rhos-16.2 failing on networking related tempest tests, Neutron server have ERROR ovsdbapp.event neutron.plugins.ml2.common.exceptions.MechanismDriverError" [Urgent,New] - Assigned to rhos-maint | 12:30 |
| rlandy | ysandeep: ack - was holding on to those until we stabilize cirros 5 upstream | 12:31 |
| rlandy | but at least we would be in line | 12:31 |
| *** jpena|lunch is now known as jpena | 12:32 | |
| rlandy | ysandeep: weshay|ruck: let's merge those | 12:32 |
| rlandy | at least we will be in sync with upstream | 12:32 |
| ysandeep | rlandy, I agree with you | 12:32 |
| rlandy | done | 12:32 |
| arxcruz | weshay|ruck: rlandy bhagyashris akahat|ruck https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32938 | 12:37 |
| arxcruz | take a look and review, this role is to keep building and copying the script to /usr/local/bin to be available on path | 12:38 |
| rlandy | arxcruz: thanks - a lot there - will look later | 12:38 |
| bhagyashris | arxcruz, ack | 12:42 |
| ysandeep | rlandy, weshay|ruck fyi.. https://bugzilla.redhat.com/show_bug.cgi?id=1956801 cixing it. | 12:43 |
| openstack | bugzilla.redhat.com bug 1956801 in openstack-neutron "16.2 tempest test: neutron_tempest_plugin.scenario.test_internal_dns.InternalDNSTest failed with MismatchError: 'openstackgate.local' not in 'nameserver 10.5.30.160" [Urgent,New] - Assigned to rhos-maint | 12:43 |
| rlandy | ok | 12:44 |
| *** ratailor__ has quit IRC | 12:47 | |
| arxcruz | bhagyashris: weshay|ruck please use this on the next sprint retrospective :) https://i.imgur.com/3dUh9oz.jpg | 13:05 |
| bhagyashris | arxcruz, you are the owner of retro for next sprint :D | 13:09 |
| weshay|ruck | arxcruz, that's awesome | 13:10 |
| marios | thanks arxcruz :) | 13:12 |
| * marios looking forward to it | 13:12 | |
| * marios must teach my kid about star wars he is nine today :) | 13:13 | |
| marios | may the 4th be with you arxcruz | 13:13 |
| arxcruz | marios: may the 4th be with you too :) | 13:13 |
| *** jlarriba has quit IRC | 13:15 | |
| *** amoralej|lunch is now known as amoralej | 13:16 | |
| *** jlarriba has joined #oooq | 13:19 | |
| weshay|ruck | pojadhav|rover, {0} tempest.api.compute.admin.test_volumes_negative.VolumesAdminNegativeTest.test_update_attached_volume_with_nonexistent_volume_in_body [309.056798s] ... FAILED | 13:21 |
| weshay|ruck | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/e13f0cd/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz | 13:21 |
| *** jlarriba has quit IRC | 13:21 | |
| weshay|ruck | pojadhav|rover, please write that up | 13:21 |
| weshay|ruck | and skip | 13:21 |
| *** jlarriba has joined #oooq | 13:21 | |
| *** jlarriba has quit IRC | 13:21 | |
| pojadhav|rover | weshay|ruck, yup | 13:21 |
| *** jlarriba has joined #oooq | 13:22 | |
| weshay|ruck | pojadhav|rover, let's tackle .. https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master as well | 13:23 |
| pojadhav|rover | yes, once FS01 done.. will focus on FS035 | 13:24 |
| weshay|ruck | pojadhav|rover, meh.. I think 35 will be fixed w/ rabi's patch.. but we may need an additional | 13:25 |
| weshay|ruck | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master | 13:25 |
| weshay|ruck | same thing | 13:25 |
| weshay|ruck | "Trying to pull quay.io/tripleomaster/openstack-cinder-volume:current-tripleo...", " Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 38.102.83.187:53: dial udp 38.102.83.187:53: connect: network is unreachable", "Error: Error initializing source | 13:25 |
| weshay|ruck | akahat|ruck, ping | 13:25 |
| weshay|ruck | pojadhav|rover, ya.. 35 should be fixed w/ https://review.opendev.org/c/openstack/python-tripleoclient/+/789131 and it's finally in the gate | 13:27 |
| weshay|ruck | thanks for the recheck | 13:27 |
| pojadhav|rover | :) | 13:27 |
| *** jfrancoa has quit IRC | 13:29 | |
| bhagyashris | arxcruz, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, weshay, chandankumar, frenzy_friday | 13:29 |
| bhagyashris | TripleO CI Community Meeting in 1 min | 13:29 |
| bhagyashris | akahat|ruck, pojadhav|rover weshay|ruck ^ | 13:29 |
| frenzy_friday | thanks bhagyashris | 13:29 |
| weshay|ruck | arxcruz, can you please poke at this https://bugs.launchpad.net/tripleo/+bug/1927001 | 13:29 |
| openstack | Launchpad bug 1927001 in tripleo "cirros retry: TypeError: 'Retrying' object is not iterable" [Undecided,Triaged] - Assigned to Amol Kahat (amolkahat) | 13:29 |
| bhagyashris | https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2021-05-04-Community-call | 13:29 |
| arxcruz | weshay|ruck: this was fixed by https://review.opendev.org/c/osf/python-tempestconf/+/787975 | 13:30 |
| weshay|ruck | arxcruz, hrmm.. | 13:33 |
| arxcruz | maybe it requires a promotion on tempestconf | 13:33 |
| arxcruz | but this happened a few days ago with ysandeep and I was able to run this code and it worked | 13:33 |
| weshay|ruck | arxcruz, a promotion | 13:34 |
| arxcruz | weshay|ruck: ? | 13:34 |
| weshay|ruck | arxcruz, please go get this fixed.. https://trunk.rdoproject.org/centos8-train/component/tempest/consistent/ | 13:36 |
| weshay|ruck | arxcruz, the rdoinfo pin must be holding train back | 13:36 |
| weshay|ruck | arxcruz, https://github.com/redhat-openstack/rdoinfo/blob/master/tags/train.yml#L1142-L1145 | 13:38 |
| weshay|ruck | arxcruz, chandankumar is that even the right thing? tempestconf doesn't line up w/ https://github.com/openstack-archive/python-tempestconf/tags | 13:39 |
| arxcruz | weshay|ruck: https://opendev.org/osf/python-tempestconf | 13:39 |
| arxcruz | it's under osf now | 13:39 |
| *** jfrancoa has joined #oooq | 13:40 | |
| arxcruz | kopecmartin: do we need to have tempestconf pinned on train ? | 13:40 |
| weshay|ruck | arxcruz, get on the call please | 13:42 |
| marios | kopecmartin: we need you to please release new python-tempestconf | 13:47 |
| *** skramaja_ has quit IRC | 13:48 | |
| marios | kopecmartin: (i.e. new tag) and then once merged we need to add new pin for it in rdoinfo https://github.com/redhat-openstack/rdoinfo/blob/master/tags/train.yml#L1142-L1145 | 13:49 |
| marios | kopecmartin: please ping me or rlandy or weshay|ruck or arxcruz about this if it isn't clear please? | 13:49 |
| *** slaweq has joined #oooq | 13:55 | |
| akahat|ruck | weshay|ruck, ping | 13:56 |
| marios | kopecmartin: whenever you next can don't worry if you can't today arxcruz told us you weren't feeling very well so get well first ;) ! | 13:57 |
| ysandeep | rlandy, weshay|ruck review please whenever time permits - https://code.engineering.redhat.com/gerrit/#/c/239634/ - Readd tripleo/BM infrared job in criteria | 13:59 |
| akahat|ruck | weshay|ruck, https://ff10a8307f2205dbdea8-b3fafd56f0033b14a62b64b1ec80f6e2.ssl.cf1.rackcdn.com/788371/1/gate/tripleo-ci-centos-8-standalone-upgrade-victoria/dc02c3c/logs/undercloud/var/log/containers/stdouts/neutron_db_sync.log | 14:01 |
| weshay|ruck | akahat|ruck, https://bugs.launchpad.net/tripleo/+bug/1927035 | 14:01 |
| openstack | Launchpad bug 1927035 in tripleo "/usr/bin/neutron-db-manage", db-sync fails w/ unexpected keyword argument 'sync_backref'" [Critical,Triaged] | 14:01 |
| rlandy | ysandeep: nice to see that passing again | 14:08 |
| arxcruz | weshay|ruck: marios kopecmartin just confirmed that tempestconf is branchless and should work on all releases, i'm checking with him why it is pinned on queens and train | 14:13 |
| pojadhav|rover | weshay|ruck, posted test patch for FS35 : https://review.rdoproject.org/r/c/testproject/+/33569 | 14:14 |
| pojadhav|rover | as rabi's patch got merged, this should pass.. | 14:15 |
| rlandy | pojadhav|rover: weshay|ruck: akahat|ruck: when you do plan to next promote master? | 14:20 |
| rlandy | spinning on the ipa job | 14:20 |
| rlandy | and suspect that current-tripleo does not always get updated there | 14:20 |
| rlandy | so currently the job has been passing consistently | 14:21 |
| rlandy | since Thursday afternoon | 14:21 |
| pojadhav|rover | rlandy, tomorrow morning | 14:21 |
| pojadhav|rover | weshay|ruck said | 14:21 |
| weshay|ruck | pojadhav|rover, maybe we want to depends on rabi's patch that is about to land | 14:21 |
| pojadhav|rover | weshay|ruck, that patch got merged | 14:22 |
| weshay|ruck | oh great | 14:22 |
| rlandy | pojadhav|rover: weshay|ruck: k - good - then we will know if the updated is the problem | 14:22 |
| rlandy | thanks | 14:22 |
| pojadhav|rover | weshay|ruck, https://review.opendev.org/c/openstack/python-tripleoclient/+/789131 | 14:22 |
| weshay|ruck | pojadhav|rover, woot | 14:22 |
| pojadhav|rover | weshay|ruck, rlandy : please approve https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/789570 | 14:24 |
| pojadhav|rover | this is too https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/789552 | 14:24 |
| weshay|ruck | pojadhav|rover++ if you get some time.. see if you can do a little root cause analysis on that.. | 14:26 |
| pojadhav|rover | weshay|ruck, yup | 14:27 |
| rlandy | Job puppet-openstack-integration-6-scenario001-tempest-centos-8-stream not defined | 14:27 |
| pojadhav|rover | I will take break now.. | 14:27 |
| rlandy | weshay|ruck: ^^ shoot - going to have to do some repo adding to RDO | 14:27 |
| *** pojadhav|rover is now known as pojadhav|dinner | 14:28 | |
| weshay|ruck | pojadhav|rover, k.. thanks | 14:28 |
| rlandy | things are never simple | 14:28 |
| weshay|ruck | arxcruz, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, weshay, chandankumar, frenzy_friday, dviroel FYI... Soniya is presenting at the all hands today | 14:30 |
| weshay|ruck | please try to attend if possible to cheer her on! | 14:31 |
| rlandy | nice | 14:31 |
| frenzy_friday | cool! | 14:31 |
| ysandeep | great! soniya29++ All the best! | 14:31 |
| marios | thanks for reminder weshay|ruck good luck soniya29 | 14:31 |
| soniya29 | marios, ysandeep, rlandy, frenzy_friday thanks :) | 14:32 |
| *** ysandeep is now known as ysandeep|dinner | 14:33 | |
| bhagyashris | soniya29, All the best! :) | 14:34 |
| kopecmartin | arxcruz: i'm here | 14:37 |
| kopecmartin | for a little bit | 14:37 |
| pojadhav|dinner | soniya29, good luck ;) | 14:39 |
| kopecmartin | marios: now i'm fine, got vaccinated 2 hours ago (what's gonna happen tomorrow we'll see :D ) ... /me reading what's going on | 14:41 |
| kopecmartin | marios: arxcruz so master tempestconf contains changes that you need? | 14:42 |
| arxcruz | kopecmartin: hey man | 14:42 |
| arxcruz | kopecmartin: yes | 14:42 |
| arxcruz | kopecmartin: i wondering why queens and train was pinned | 14:42 |
| arxcruz | since you told me tempestconf is branchless | 14:42 |
| kopecmartin | is is, how's that related? i'm trying to keep all supported releases on the same tag (while possible) | 14:43 |
| arxcruz | kopecmartin: queens tag is 2. something | 14:47 |
| arxcruz | iirc | 14:47 |
| arxcruz | just wondering if remmove tag from queens branch will have some issue | 14:47 |
| arxcruz | not sure if there's new code that won't work on queens | 14:48 |
| soniya29 | bhagyashris, pojadhav|dinner thanks :) | 14:48 |
| kopecmartin | arxcruz: i don't see lots of new changes in master after 3.2.0 .. this one is the one needed? https://opendev.org/osf/python-tempestconf/commit/2e09b75839487a67ed994244e0323fd175fbe987 | 14:51 |
| arxcruz | kopecmartin: yup | 14:53 |
| arxcruz | but we might need it on queens as well | 14:53 |
| kopecmartin | arxcruz: ok, let me quickly create a new tag and you can try to bump it for queens as well, not sure if that's gonna work | 14:56 |
| marios | arxcruz: kopecmartin: as discussed lets focus on train first which is the immediate problem weshay|ruck wanted to solve | 14:58 |
| *** ysandeep|dinner is now known as ysandeep | 14:59 | |
| marios | kopecmartin: so just make it bugfix release like 3.2.1 or something | 14:59 |
| kopecmartin | marios: (y) | 14:59 |
| *** amoralej is now known as amoralej|off | 15:06 | |
| kopecmartin | arxcruz: marios i pushed a new tag - 3.2.1 | 15:09 |
| kopecmartin | hopefully it works, I need to drive to brno now | 15:12 |
| marios | thanks kopecmartin | 15:15 |
| marios | kopecmartin: link? please? | 15:15 |
| marios | weshay|ruck: ^^^ fyi arxcruz | 15:15 |
| *** dviroel|nho is now known as dviroel | 15:17 | |
| *** jmasud has joined #oooq | 15:21 | |
| *** holser has quit IRC | 15:33 | |
| *** holser has joined #oooq | 15:34 | |
| weshay|ruck | soniya29++ | 15:37 |
| weshay|ruck | well done! | 15:37 |
| soniya29 | weshay|ruck, thank you | 15:37 |
| bhagyashris | soniya29, Great job ++ :) | 15:37 |
| akahat|ruck | nice presetation soniya29 | 15:37 |
| akahat|ruck | soniya29++ | 15:37 |
| frenzy_friday | soniya29, great presentation! | 15:37 |
| pojadhav|dinner | soniya29++ | 15:37 |
| dviroel | soniya29: great job! ++ | 15:37 |
| soniya29 | bhagyashris, akahat|ruck, frenzy_friday , pojadhav|dinner, dviroel thanks :) | 15:39 |
| marios | well done soniya29 | 15:41 |
| weshay|ruck | ysandeep, you have a skip-list change coming for https://trello.com/c/C1yvkhYD/1930-cixbz1956758osp162rhel8rhos-162periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-rhos-162-failing-on-networking-related-tem ? | 15:42 |
| ysandeep | weshay|ruck, yes for this one as well https://trello.com/c/ManqYzs3/1931-cixbz1956801osp162rhel8rhos-162162-tempest-test-neutrontempestpluginscenariotestinternaldnsinternaldnstest-failed-with-mismatche | 15:43 |
| weshay|ruck | k.. thanks | 15:43 |
| sshnaidm | soniya29++ | 15:49 |
| *** chem has quit IRC | 15:52 | |
| ysandeep | weshay|ruck, should I add all these test in skiplist? https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-rhos-16.2/751cc5c/logs/undercloud/var/log/tempest/stestr_results.html | 15:56 |
| weshay|ruck | ysandeep, so this is where it's hard when it's multiple | 15:56 |
| weshay|ruck | try your best to have one bug per root cause | 15:57 |
| weshay|ruck | but time cap the root cause analysis | 15:57 |
| weshay|ruck | ysandeep, is that an ok answer or need more detail? | 15:57 |
| ysandeep | ^^ for this one, so far i have opened single bug as all seems networking related(with tracebacks in logs)... Will deep dive if tomorrow to see if root cause is different | 15:59 |
| ysandeep | Will deep dive* tomorrow | 15:59 |
| weshay|ruck | ysandeep, thanks | 15:59 |
| *** sshnaidm is now known as sshnaidm|afk | 15:59 | |
| *** dviroel is now known as dviroel|lunch | 15:59 | |
| *** marios is now known as marios|out | 16:03 | |
| *** ykarel has quit IRC | 16:04 | |
| frenzy_friday | hey all, need a little help on converting strings to escaped regex https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/780282/22/src/sova-converter.py#43 | 16:07 |
| ysandeep | weshay|ruck, rlandy something is wrong.. tests which were earlier skipped in rhos-16.2 are now running | 16:10 |
| *** marios|out has quit IRC | 16:10 | |
| ysandeep | for ex: now https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-full-tempest-scenario-rhos-16.2/96d184b/logs/undercloud/var/log/tempest/stestr_results.html | 16:10 |
| ysandeep | earlier: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-full-tempest-scenario-rhos-16.2/dda768e/logs/undercloud/var/log/tempest/stestr_results.html | 16:10 |
| ysandeep | neutron_tempest_plugin.scenario.test_internal_dns.InternalDNSTest ^^ was not running earlier | 16:10 |
| ysandeep | May be because of https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/788002/1/playbooks/tasks/tempest.yml ? | 16:10 |
| ysandeep | weshay|ruck, rlandy this task was in skip list for train.. i think with ^^ we have to readd for 16.2.. .. https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/789597 | 16:11 |
| weshay|ruck | ysandeep, hrm.. rlandy | 16:12 |
| weshay|ruck | so w/o the osp release in https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml#L71-L97 | 16:12 |
| weshay|ruck | I can see why now that is skipped | 16:12 |
| weshay|ruck | er.. isn't skipped :) | 16:13 |
| weshay|ruck | ysandeep, so either we revert https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/788002/1/playbooks/tasks/tempest.yml | 16:13 |
| weshay|ruck | ysandeep, rlandy or we go through the list and add osp releases to what we need skipped | 16:13 |
| ysandeep | weshay|ruck, rlandy if we revert.. we loose ability to only skip in 16.2 https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/789597 | 16:14 |
| ysandeep | weshay|ruck, rlandy sry this one https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/788013 | 16:14 |
| ysandeep | weshay|ruck, rlandy want to chat for few mins, before I go to sleep.. :) | 16:15 |
| weshay|ruck | ysandeep, https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/789597/1/roles/validate-tempest/vars/tempest_skip.yml would still skip for train / 16.2 | 16:15 |
| weshay|ruck | at least it should | 16:16 |
| weshay|ruck | imho | 16:16 |
| ysandeep | weshay|ruck, rlandy meet.google.com/pcd-nnvk-anx if you have free slot now | 16:16 |
| weshay|ruck | ysandeep, the relase var for 16.2 is train | 16:16 |
| rlandy | ok | 16:20 |
| weshay|ruck | rlandy, https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/789450 | 16:21 |
| *** pojadhav|dinner is now known as pojadhav|away | 16:22 | |
| weshay|ruck | rlandy, https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/789451 | 16:51 |
| rlandy | weshay|ruck: pojadhav|away: https://review.rdoproject.org/r/c/rdo-jobs/+/33496 - thsi job keeps failing master scenario001 | 16:59 |
| rlandy | it's listed as a wallaby branching change | 16:59 |
| rlandy | but there are multiple changes in that patch | 16:59 |
| rlandy | since the pacific (pineapple) release | 17:00 |
| rlandy | Thanks for adding this. I think we just want to change the Ceph version since Wallaby is on Pacific. | 17:01 |
| rlandy | comment from john | 17:01 |
| *** jmasud has quit IRC | 17:01 | |
| *** dviroel|lunch is now known as dviroel | 17:10 | |
| *** jmasud has joined #oooq | 17:13 | |
| rlandy | weshay|ruck: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/789451 merged | 17:15 |
| weshay|ruck | rlandy, https://review.opendev.org/c/openstack/tripleo-ansible/+/789548 | 17:15 |
| akahat|ruck | weshay|ruck, https://review.rdoproject.org/r/c/testproject/+/33575 tripleo-ci-centos-8-standalone-upgrade-victoria testproject job. | 17:18 |
| akahat|ruck | weshay|ruck, rlandy i'm stuck with this job mol-promote_images. https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32679 | 17:19 |
| akahat|ruck | not sure why it is not connecting to the host. | 17:19 |
| weshay|ruck | akahat|ruck, thanks | 17:20 |
| akahat|ruck | please take a look. | 17:20 |
| rlandy | weshay|ruck: w+'ed https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/789450 | 17:20 |
| rlandy | akahat|ruck: possibly infra? | 17:22 |
| * rlandy tries recheck to confirm | 17:22 | |
| akahat|ruck | rlandy, no it's not infra. | 17:23 |
| rlandy | k - checking | 17:25 |
| rlandy | lunch ... brb | 17:29 |
| *** jpena is now known as jpena|off | 17:37 | |
| *** jpodivin has quit IRC | 18:20 | |
| *** jbadiapa has quit IRC | 18:27 | |
| weshay|ruck | need a review https://review.opendev.org/c/openstack/tripleo-ci/+/789616 | 18:47 |
| rlandy | done | 19:05 |
| rlandy | https://logserver.rdoproject.org/96/33496/7/check/tripleo-ceph-integration-centos-8-scenario001-standalone-wallaby/bfe41ee/job-output.txt | 19:06 |
| rlandy | weshay|ruck: ^^ | 19:07 |
| rlandy | seen that in wallaby? | 19:07 |
| rlandy | 2021-05-04 17:38:32.789988 | primary | ERROR: Cannot install -r quickstart-extras-requirements.txt (line 13) because these package versions have conflicting dependencies. | 19:07 |
| rlandy | 2021-05-04 17:38:32.790098 | primary | | 19:07 |
| rlandy | 2021-05-04 17:38:32.790107 | primary | The conflict is caused by: | 19:07 |
| rlandy | 2021-05-04 17:38:32.790112 | primary | tripleo-ansible 3.0.1.dev250 depends on tripleo-common>=16.0.0 | 19:07 |
| rlandy | 2021-05-04 17:38:32.790116 | primary | The user requested (constraint) tripleo-common===15.0.0 | 19:07 |
| rlandy | maybe only the ceph jobs? | 19:07 |
| weshay|ruck | doubt it.. we need to update upper-constraints | 19:08 |
| *** radez has joined #oooq | 19:09 | |
| * weshay|ruck poking | 19:11 | |
| weshay|ruck | rlandy, https://opendev.org/openstack/requirements/commit/a99e8294bf7ae95029375a946f26955dab19e238 | 19:19 |
| rlandy | oh gee | 19:20 |
| weshay|ruck | so it's some kind of cache issue | 19:20 |
| rlandy | recheck? | 19:20 |
| weshay|ruck | rlandy, ya.. | 19:21 |
| weshay|ruck | rlandy, https://review.rdoproject.org/r/c/rdo-jobs/+/33576 | 19:21 |
| rlandy | voted | 19:21 |
| weshay|ruck | rlandy, https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/789625 | 19:39 |
| rlandy | ok | 19:41 |
| rlandy | will post hackmd | 19:41 |
| *** dviroel is now known as dviroel|brb | 19:57 | |
| rlandy | weshay|ruck: where does puppet-openstack-integration-6-scenario001-tempest-centos-8-stream run? | 20:27 |
| rlandy | https://zuul.openstack.org/builds?job_name=+puppet-openstack-integration-6-scenario001-tempest-centos-8-stream | 20:28 |
| rlandy | no trace of it | 20:28 |
| *** slaweq has quit IRC | 20:40 | |
| weshay|ruck | rlandy, look at the tripleo-puppet and all the puppet repos | 20:42 |
| rlandy | puppet-openstack-integration-5-scenario001-tempest-centos-8-stream has a trace on stable/ussuri | 20:43 |
| weshay|ruck | rlandy, what are you saying a trace? there's a bug? | 20:49 |
| rlandy | no - english expression | 20:49 |
| rlandy | no sign on them having been run lately | 20:49 |
| weshay|ruck | rlandy, sorry.. you want to look at https://opendev.org/openstack/puppet-openstack-integration/src/branch/master/zuul.d/layout.yaml#L6 | 20:51 |
| weshay|ruck | https://opendev.org/openstack/puppet-openstack-integration/src/branch/master/zuul.d/layout.yaml#L59-L68 | 20:51 |
| weshay|ruck | rlandy, in the puppet projects | 20:52 |
| weshay|ruck | the name is overridden | 20:52 |
| rlandy | https://review.rdoproject.org/r/c/testproject/+/31586 | 20:53 |
| rlandy | noop | 20:53 |
| rlandy | which means I am missing something to have these jobs kick | 20:53 |
| weshay|ruck | tomorrow | 20:54 |
| rlandy | weshay|ruck: ^^ see what I mean | 20:54 |
| rlandy | scratch head | 20:54 |
| rlandy | change puppet file maybe | 20:54 |
| *** dviroel|brb is now known as dviroel | 21:20 | |
| *** saneax has quit IRC | 22:04 | |
| *** saneax has joined #oooq | 22:05 | |
| *** udesale has quit IRC | 22:44 | |
| *** holser has quit IRC | 22:56 | |
| *** tosky has quit IRC | 23:28 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
