| *** amoralej|off is now known as amoralej | 06:03 | |
| *** ykarel_ is now known as ykarel | 06:07 | |
| ykarel | bhagyashris|rover, is failure in phase 1 master known one? | 06:07 |
|---|---|---|
| ykarel | https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-35/undercloud/home/stack/overcloud_deploy.log | 06:07 |
| ykarel | related to ephemeral heat | 06:08 |
| bhagyashris|rover | ykarel, not sure, Didn't check before | 06:09 |
| ykarel | ack can check now | 06:10 |
| *** ysandeep|out is now known as ysandeep | 06:36 | |
| *** jpodivin is now known as jpodivin|ruck | 06:36 | |
| chandankumar | ysandeep: ack, thanks :-) | 06:42 |
| ysandeep | chandankumar++ marios++ thanks for reviews on those patches! | 06:43 |
| chandankumar | bhagyashris|rover: any reviews needs attension | 06:44 |
| chandankumar | for promotion | 06:44 |
| chandankumar | ? | 06:44 |
| bhagyashris|rover | chandankumar, nope | 06:54 |
| *** jpena|off is now known as jpena | 07:00 | |
| bhagyashris|rover | slaweq, hi | 07:01 |
| slaweq | bhagyashris|rover: hi | 07:02 |
| bhagyashris|rover | i have reported separate bug for victoria https://bugs.launchpad.net/tripleo/+bug/1945443 | 07:02 |
| slaweq | bhagyashris|rover: thx | 07:02 |
| slaweq | bhagyashris|rover: I will try to investigate that | 07:02 |
| bhagyashris|rover | but we are still facing same issue this one https://bugs.launchpad.net/tripleo/+bug/1943708/ on master | 07:02 |
| slaweq | is it happening all the time or is it intermittent? | 07:02 |
| bhagyashris|rover | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master | 07:03 |
| slaweq | bhagyashris|rover: regarding the issue on master, I just replied in https://trello.com/c/nMmHG0WW | 07:03 |
| slaweq | can You check it? | 07:03 |
| slaweq | it seems for me that fix for that issue wasn't included in the neutron package instelled in the neutron-api container there | 07:03 |
| bhagyashris|rover | victoria one failed two time with same error and passed on testproject so we can say victoria one is intermittent | 07:04 |
| bhagyashris|rover | but master one is consistent | 07:04 |
| slaweq | bhagyashris|rover: ok, for victoria issue I think that the same issue we have in u/s ci from time to time | 07:05 |
| bhagyashris|rover | slaweq, ok i will check that, but currently we add that test in the tempest skiplist | 07:05 |
| slaweq | I will try to analyze logs to maybe find out something | 07:05 |
| bhagyashris|rover | slaweq, ok thanks :) | 07:05 |
| slaweq | bhagyashris|rover: please check if that patch is included there, I was checking it with testproject and was running fine | 07:05 |
| bhagyashris|rover | slaweq, ack | 07:17 |
| erbot | Required files not ready after 301.6659679412842s for distgit-cbs-validate-centos8-stream 35892,1,2360db5e41b3434b879e2f3f52bdb553 | 07:44 |
| bhagyashris|rover | slaweq, that change came yesterday in the integration line | 07:44 |
| bhagyashris|rover | i will revert my skip list patch | 07:44 |
| bhagyashris|rover | openstack-neutron change regarding master failure | 07:45 |
| slaweq | bhagyashris|rover++ thx | 07:51 |
| bhagyashris|rover | chandankumar, arxcruz|ruck we can merge this patch https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/811785 thanks | 07:55 |
| arxcruz|ruck | bhagyashris|rover: waiting chandankumar | 08:06 |
| bhagyashris|rover | arxcruz|ruck, okay | 08:07 |
| chandankumar | marios: \o, Thanks for filing cs9 bug https://bugs.launchpad.net/tripleo/+bug/1945589, please add promotion blocker tag to them for cix, when free :-) | 08:40 |
| marios | chandankumar: sanity check that one is a mistake right https://review.opendev.org/c/openstack/tripleo-ci/+/808177/4#message-6e91b487744f306c05e2eb2be39461560f6adf9e | 08:42 |
| marios | chandankumar: wasnt sure about filing that one ... i think it is related to the bigger network manager vs scripts in centos9 issue | 08:42 |
| marios | chandankumar: for now we have a workaround so not blocked but not sure if the workaround is the end of it (hence i filed the bug) | 08:43 |
| marios | chandankumar: so we want cix for all c9 issues? | 08:43 |
| chandankumar | marios: yes | 08:44 |
| marios | chandankumar: k adding | 08:45 |
| chandankumar | marios: as per apevec for el9 status call, we need to cix bugs which is blocking the work or workaround so that we get attension | 08:45 |
| erbot | Required files not ready after 301.50063729286194s for distgit-cbs-validate-centos8-stream 35570,1,5f49273e378f47488022a1a461333817 | 08:49 |
| * bhagyashris|rover lunch | 08:56 | |
| *** ykarel is now known as ykarel|lunch | 09:13 | |
| marios | 2021-09-30 09:15:53.640 | Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/centos9.repo; Configuration: OptionBinding with id "failovermethod" does not exist | 09:55 |
| marios | ah red herring | 09:55 |
| ysandeep | bhagyashris|rover looks like content-provider have some issues in upstream | 10:00 |
| ysandeep | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-content-provider | 10:00 |
| ysandeep | Incase not already known. | 10:00 |
| ysandeep | error pushing image "quay.io/ceph/daemon:v6.0.4-stable-6.0-pacific-centos-8-x86_64" to "docker://127.0.0.1:5001/tripleomaster/daemon:v6.0.4-stable-6.0-pacific-centos-8-x86_64": error locating image with name "quay.io/ceph/daemon:v6.0.4-stable-6.0-pacific-centos-8-x86_64" ([quay.io/ceph/daemon:v6.0.4-stable-6.0-pacific-centos-8-x86_64]): image not knownerror pushing image "quay.io/ceph/daemon:v6.0.4-stable-6.0-pacific-centos-8-x86_64" to | 10:01 |
| ysandeep | "docker://127.0.0.1:5001/tripleomaster/daemon:v6.0.4-stable-6.0-pacific-centos-8-x86_64": error locating image with name "quay.io/ceph/daemon:v6.0.4-stable-6.0-pacific-centos-8-x86_64" ([quay.io/ceph/daemon:v6.0.4-stable-6.0-pacific-centos-8-x86_64]): image not known | 10:01 |
| ysandeep | arxcruz|ruck, ^^ | 10:01 |
| *** ykarel|lunch is now known as ykarel | 10:05 | |
| arxcruz|ruck | ysandeep: quay.io/ceph ? | 10:13 |
| arxcruz|ruck | should be ceph.quay.io no ? | 10:13 |
| arxcruz|ruck | let me check | 10:13 |
| marios | arxcruz|ruck: ysandeep: there was an issue with quay earlier today dns issues | 10:14 |
| marios | https://status.quay.io/ https://status.quay.io/incidents/bt8yqm82069z | 10:14 |
| arxcruz|ruck | marios: ysandeep I'm able to pull the image now | 10:15 |
| arxcruz|ruck | ysandeep: marios seems to be the case | 10:15 |
| marios | arxcruz|ruck: yes it has been resolved a little while ago | 10:15 |
| ysandeep | yeah.. looks like resolved now.. thanks marios | 10:15 |
| bhagyashris|rover | ysandeep, still an issue? | 10:23 |
| ysandeep | bhagyashris|rover: yes as per arxcruz|ruck manual image pull test. | 10:24 |
| ysandeep | bhagyashris|rover, I am working now | 10:24 |
| ysandeep | I mean* | 10:24 |
| ysandeep | bhagyashris|rover, Let's wait for one green run in check/gate than we can confirm all green | 10:24 |
| bhagyashris|rover | ysandeep, ack thanks :) | 10:25 |
| *** rlandy is now known as rlandy|ruck | 10:39 | |
| rlandy|ruck | arxcruz|ruck: bhagyashris|rover: hey - how are things today? | 10:40 |
| arxcruz|ruck | rlandy|ruck: bhagyashris|rover solved everything :) | 10:41 |
| rlandy|ruck | everything? | 10:41 |
| bhagyashris|rover | rlandy|ruck, going good, we had promotion for all the branches yesterday | 10:42 |
| arxcruz|ruck | rlandy|ruck: lol well, bhagyashris|rover is the woman :) | 10:42 |
| bhagyashris|rover | rlandy|ruck, just the cockpit is showing the wrong info regarding the promotions | 10:42 |
| rlandy|ruck | bhagyashris|rover++ | 10:42 |
| rlandy|ruck | bhagyashris|rover: ack - I pinged pooja about it | 10:43 |
| rlandy|ruck | when she incorporated her last change, weshay's change came in and broke the cockpit | 10:43 |
| bhagyashris|rover | rlandy|ruck, i think pooja is on PTO today and tomorrow | 10:43 |
| rlandy|ruck | arxcruz|ruck: I need your python expertise | 10:44 |
| rlandy|ruck | for ^^ fix | 10:44 |
| * bhagyashris|rover tea brb | 10:44 | |
| rlandy|ruck | I left some notes on the tripleo-ci chat | 10:44 |
| arxcruz|ruck | rlandy|ruck: yeah, i notice that | 10:44 |
| arxcruz|ruck | i'll work on that | 10:44 |
| rlandy|ruck | arxcruz|ruck: join me on the cockpit | 10:44 |
| rlandy|ruck | arxcruz|ruck: https://meet.google.com/bmz-mhox-evz?pli=1&authuser=0 | 10:45 |
| rlandy|ruck | ysandeep: hey - are we still blocked on 16.2? | 10:58 |
| rlandy|ruck | and 17? | 10:58 |
| ysandeep | rlandy|ruck: 16.2 is a legit bug.. neutron team working on it. | 10:59 |
| ysandeep | rlandy|ruck, Regarding 17 - that tempest patch merged.. but tempest is pinned in rdoinfo.. | 10:59 |
| rlandy|ruck | ysandeep: yep .. I know but we're holding the whole release on it | 11:00 |
| ysandeep | for 17: we are testing with downstream cherrypick of tempest patch: https://code.engineering.redhat.com/gerrit/c/testproject/+/190672 | 11:01 |
| ysandeep | if it works.. we can unpin tempest in rdoinfo.. | 11:01 |
| ysandeep | rlandy|ruck: imho.. for 16.2 lets hold.. because that's a legit bug.. otherwise compose will the same issue.. | 11:02 |
| rlandy|ruck | ysandeep: I agree we should not promote broken code | 11:02 |
| chandankumar | rlandy|ruck: arxcruz|ruck https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/35888 | 11:03 |
| rlandy|ruck | chandankumar: will join review meeting in a sec | 11:03 |
| rlandy|ruck | chandankumar: thanks | 11:04 |
| arxcruz|ruck | chandankumar: :* | 11:04 |
| ysandeep | slaweq: I understand ralonsoh is working on that 16.2 bug.. but if you guys need any help from us like a reproducer environment to speed up the investigation, Please let us know.. We haven't promoted 16.2 in more than 20 days so that's becoming urgent for us. | 11:04 |
| slaweq | ysandeep: sure, I will ask ralonsoh about it | 11:05 |
| ysandeep | slaweq: thanks! | 11:05 |
| rlandy|ruck | slaweq: thanks - we are holding all of 16.2 on this | 11:07 |
| rlandy|ruck | and the promotion is almost a month old | 11:07 |
| rlandy|ruck | chandankumar: joining review time? | 11:08 |
| chandankumar | rlandy|ruck: yes | 11:08 |
| chandankumar | bhagyashris|rover: hello, DO we need to merge this https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/811834 ? | 11:13 |
| chandankumar | rlandy|ruck: ^^ | 11:13 |
| bhagyashris|rover | chandankumar, i see in the recent run fs020 passed | 11:14 |
| bhagyashris|rover | looks like that test is failing intermittently | 11:14 |
| rlandy|ruck | chandankumar: tempest component is broken with this | 11:14 |
| bhagyashris|rover | rlandy|ruck, ^ | 11:14 |
| rlandy|ruck | bhagyashris|rover: and victoria | 11:14 |
| rlandy|ruck | master | 11:14 |
| bhagyashris|rover | rlandy|ruck, victoria one | 11:15 |
| bhagyashris|rover | rlandy|ruck, master one got fixed and it's passing now | 11:15 |
| rlandy|ruck | bhagyashris|rover: and victoria? | 11:15 |
| rlandy|ruck | failed fs020 | 11:15 |
| bhagyashris|rover | rlandy|ruck, victoria one failed two times and in the recent run it passed so looks like it's failing intermittently | 11:16 |
| chandankumar | https://review.rdoproject.org/zuul/builds?pipeline=openstack-component-tempest | 11:16 |
| rlandy|ruck | bhagyashris|rover: train and master | 11:17 |
| rlandy|ruck | both failing | 11:17 |
| bhagyashris|rover | rlandy|ruck, ok let me check | 11:17 |
| rlandy|ruck | bhagyashris|rover: train one is just a once off | 11:17 |
| rlandy|ruck | bhagyashris|rover: so keep the skiplist - juts change the test to the component line test | 11:17 |
| rlandy|ruck | we will remove the entry when it clears the component line | 11:18 |
| bhagyashris|rover | rlandy|ruck, ack | 11:19 |
| chandankumar | bhagyashris|rover: https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-tempest-master/2817ea7/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz | 11:19 |
| chandankumar | log link | 11:19 |
| *** dviroel|out is now known as dviroel | 11:21 | |
| rlandy|ruck | bhagyashris|rover: arxcruz|ruck: rechecking https://review.rdoproject.org/r/c/testproject/+/35777 | 11:25 |
| rlandy|ruck | to see if https://review.opendev.org/c/openstack/tripleo-quickstart/+/801409 is ok to merge | 11:25 |
| bhagyashris|rover | rlandy|ruck, chandankumar done https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/811834 | 11:25 |
| *** jpena is now known as jpena|lunch | 11:25 | |
| bhagyashris|rover | ack | 11:25 |
| ysandeep | rlandy|ruck: Please review https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/276927 when you have time(we can merge the job definitions), I have separated this from adding in rhos-release check patch, Hit some issues with that.. Send details in an email | 11:46 |
| chandankumar | marios: ysandeep arxcruz|ruck can we meet for interview call planning in another 30 mins? | 11:58 |
| arxcruz|ruck | chandankumar: no | 11:58 |
| arxcruz|ruck | chandankumar: just kidding :) | 11:58 |
| marios | chandankumar: k | 11:58 |
| ysandeep | chandankumar: sure, works for me | 11:58 |
| chandankumar | marios: ysandeep arxcruz|ruck will pass the link in another 30 mins | 11:59 |
| ysandeep | ack | 12:00 |
| rlandy|ruck | bhagyashris: hey ... https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/811834 | 12:02 |
| rlandy|ruck | - should be master | 12:02 |
| *** amoralej is now known as amoralej|lunch | 12:15 | |
| *** jpena|lunch is now known as jpena | 12:16 | |
| rlandy|ruck | arxcruz|ruck: hey - https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/35888 merged | 12:22 |
| rlandy|ruck | how far out are you on the abandoned status cockpit fix? | 12:23 |
| rlandy|ruck | otherwise will pull this change to see if it works | 12:23 |
| rlandy|ruck | arxcruz|ruck: are you on the cockpit? | 12:23 |
| kopecmartin | chandankumar: thank you for the notes in regards of os_tempest, i'm just going through them | 12:27 |
| rlandy|ruck | chandankumar: marios: arxcruz|ruck: https://meet.google.com/jfx-pwrz-meb?authuser=0 | 12:30 |
| rlandy|ruck | ysandeep: ^^ | 12:31 |
| erbot | Required files not ready after 300.9513714313507s for periodic-tripleo-ci-centos-8-standalone-baremetal-master-validation 35899,3,cac7b7fa28264f79a1d7782daa4d727d | 12:38 |
| rlandy|ruck | frenzy_friday++ | 12:41 |
| rlandy|ruck | arxcruz|ruck: joining meeting again | 12:41 |
| arxcruz|ruck | rlandy|ruck: https://meet.google.com/bmz-mhox-evz?pli=1&authuser=1 ? | 12:43 |
| *** bhagyashris is now known as bhagyashris|rover | 12:43 | |
| * dviroel looks with angry face to technician ISP fixing cables | 12:57 | |
| ysandeep | marios, you are on mute if you are speaking | 13:09 |
| rlandy|ruck | ysandeep: merged | 13:16 |
| ysandeep | thanks! | 13:16 |
| *** pojadhav- is now known as pojadhav | 13:22 | |
| *** amoralej|lunch is now known as amoralej | 13:34 | |
| *** pojadhav- is now known as pojadhav | 13:54 | |
| rlandy|ruck | bhagyashris|rover: hey - do you want to meet for your 1-on-1? | 13:59 |
| rlandy|ruck | arxcruz|ruck: let's take one more cut at the cockpit | 14:05 |
| rlandy|ruck | making some manual changes in the mean time | 14:05 |
| arxcruz|ruck | rlandy|ruck: just a sec, i'll check what's going wrong | 14:16 |
| rlandy|ruck | arxcruz|ruck: I updated the ABORTED stuff manually | 14:17 |
| rlandy|ruck | arxcruz|ruck: nothing is wring as such | 14:17 |
| arxcruz|ruck | rlandy|ruck: ok, i'll update it in the json file | 14:17 |
| rlandy|ruck | arxcruz|ruck: k- then we can try build again :) fun times | 14:18 |
| rlandy|ruck | maybe tomorrow | 14:18 |
| rlandy|ruck | arxcruz|ruck: if you are not slammed, you can join the interview to at its start | 14:18 |
| bhagyashris_ | rlandy|ruck, nothing much to say will meet next week | 14:19 |
| *** bhagyashris_ is now known as bhagyashris|rover | 14:19 | |
| bhagyashris|rover | rlandy|ruck, sorry there was some connection issue | 14:19 |
| rlandy|ruck | bhagyashris|rover: sure - will chat then | 14:19 |
| rlandy|ruck | no worries :) | 14:19 |
| rlandy|ruck | all good | 14:19 |
| bhagyashris|rover | yup | 14:21 |
| dviroel | ade_lee: hi o/ | 14:22 |
| dviroel | ade_lee: fyi https://bugs.launchpad.net/tripleo/+bug/1945655 | 14:22 |
| dviroel | the issue that you hit yesterday | 14:23 |
| ade_lee | dviroel, ah great! | 14:30 |
| erbot | Required files not ready after 301.2821626663208s for deps-cbs-validate 35904,1,6fe61575e5aa4e3f8c3af9f4a0d41f13 | 14:39 |
| ade_lee | dviroel, guss | 14:43 |
| ade_lee | dviroel, guess we just need to add the fix as a dependency then | 14:43 |
| erbot | Required files not ready after 301.29233264923096s for periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train 32107,21,a8e4ffa8868046229146d2875a5fcf9d | 14:45 |
| ade_lee | dviroel, trying to figure out how to proceed further on https://review.opendev.org/c/openstack/tripleo-ci/+/810786 | 14:45 |
| ade_lee | dviroel, we're seeing the same error - so I'm not entirely sure we're getting the right library (libssh) | 14:46 |
| ade_lee | dviroel, I guess we can add something to print out the ssh_type | 14:47 |
| dviroel | ade_lee: yeah, we can add a debug message to see which ssh_type is being used | 14:47 |
| ade_lee | dviroel, I do plan to set up a couple of fips enabled machines to reproduce the issue with both libraries and see what happens. should be able to use a stripped down version of the playbooks to reproduce what we're seeing | 14:48 |
| erbot | Required files not ready after 406.4150080680847s for periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master 35777,2,6f666234bd7543ae8e9e8c74e038607f | 14:50 |
| dviroel | ade_lee: yeah, that should be a good approach atm | 14:52 |
| *** amoralej is now known as amoralej|off | 14:58 | |
| *** chandankumar is now known as raukadah | 15:21 | |
| erbot | Required files not ready after 301.2610352039337s for DLRN-rpmbuild-centos8 35902,2,21b88f32fa2d44b9ad7546b6fedb7a50 | 15:31 |
| * dviroel lunch | 15:35 | |
| *** jpena is now known as jpena|off | 15:52 | |
| *** marios is now known as marios|out | 16:09 | |
| *** ysandeep is now known as ysandeep|dinner | 16:10 | |
| rlandy|ruck | arxcruz|ruck: just debugging vexx ovb failures | 16:11 |
| rlandy|ruck | let's touch base on cockpit later/tomorrow | 16:11 |
| rlandy|ruck | let me know when you are EoD | 16:11 |
| arxcruz|ruck | rlandy|ruck: i'm putting the patch for the cockpit today or tomorrow morning | 16:11 |
| rlandy|ruck | arxcruz|ruck: sure | 16:11 |
| arxcruz|ruck | rlandy|ruck: and I'll add the test failing in the tempest component pipeline | 16:11 |
| rlandy|ruck | arxcruz|ruck: no worries | 16:12 |
| rlandy|ruck | that will kick shortly | 16:12 |
| rlandy|ruck | arxcruz|ruck: bhagyashris|rover: OVB is down | 16:13 |
| arxcruz|ruck | rlandy|ruck: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/35908 the cockpit fix | 16:20 |
| rlandy|ruck | arxcruz|ruck: ci-scripts/infra-setup/roles/rrcockpit/files/grafana/rdo.admins.alert-notification.json | 16:22 |
| rlandy|ruck | was only expecting a change to ci-scripts/infra-setup/roles/rrcockpit/files/grafana/cockpit.dashboard.json | 16:22 |
| arxcruz|ruck | rlandy|ruck: well, i did the export, and it comes with these changes | 16:23 |
| arxcruz|ruck | then I added | 16:23 |
| arxcruz|ruck | but i only touch in the cockpit | 16:23 |
| rlandy|ruck | hmmm .. | 16:23 |
| rlandy|ruck | arxcruz|ruck: k - let's look at it tomorrow | 16:24 |
| rlandy|ruck | I have made the manual changes in the mean time | 16:24 |
| erbot | Required files not ready after 301.0610842704773s for tox-linters 35908,1,845f310aaffc43418fefe53aa94cb88b | 16:29 |
| arxcruz|ruck | rlandy|ruck: the sibling error seems to be on the infra side right? I'm checking the code | 17:05 |
| rlandy|ruck | arxcruz|ruck: zuul restarted now | 17:05 |
| rlandy|ruck | let's see what happens | 17:05 |
| arxcruz|ruck | ok | 17:05 |
| rlandy|ruck | OVB is also on infra side | 17:05 |
| rlandy|ruck | got a ping out to infra and vexxhost | 17:06 |
| rlandy|ruck | arxcruz|ruck: it's never our fault | 17:06 |
| arxcruz|ruck | rlandy|ruck: naturally :) | 17:06 |
| rlandy|ruck | we're perfect | 17:06 |
| rlandy|ruck | arxcruz|ruck++ for help on cockpit | 17:06 |
| *** ysandeep|dinner is now known as ysandeep|out | 17:09 | |
| arxcruz|ruck | rlandy|ruck: i did nothing :) | 17:14 |
| rlandy|ruck | arxcruz|ruck: og course not - we're perfect | 17:38 |
| rlandy|ruck | arxcruz|ruck: hey - can you open your window on the cockpit? | 18:11 |
| rlandy|ruck | the tmux is too narrow | 18:11 |
| arxcruz|ruck | rlandy|ruck: done | 18:12 |
| rlandy|ruck | thank you | 18:12 |
| rlandy|ruck | arxcruz|ruck: looking into why the cockpit is not picking up current-tripleo-rdo promotions | 18:13 |
| arxcruz|ruck | rlandy|ruck: checking | 18:20 |
| rlandy|ruck | arxcruz|ruck: pooja added a review earlier ... | 18:22 |
| rlandy|ruck | it looks like there may have been a manual edit to pick up lone promotion | 18:22 |
| * rlandy|ruck gets | 18:23 | |
| rlandy|ruck | tox fix merged | 18:26 |
| rlandy|ruck | those jobs should clear now | 18:26 |
| rlandy|ruck | arxcruz|ruck: rechecked https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/35908 - same tox failure | 18:27 |
| rlandy|ruck | arxcruz|ruck: https://github.com/rdo-infra/ci-config/commit/6a38890e2140448cc34b3eeb158ae2cc299cc67b - pooja's change to pick up jobs | 18:28 |
| arxcruz|ruck | rlandy|ruck: needs to create a view for current-tripleo-rdo ? | 18:35 |
| rlandy|ruck | arxcruz|ruck: it has one | 18:35 |
| rlandy|ruck | on the right of upstream promotions | 18:35 |
| arxcruz|ruck | rlandy|ruck: where these information are comming from ? | 18:36 |
| arxcruz|ruck | i'm not seeing in the cockpit code current-tripleo-rdo info | 18:36 |
| rlandy|ruck | arxcruz|ruck: /me will show you | 18:36 |
| rlandy|ruck | https://meet.google.com/rdn-cnbf-zsu?pli=1&authuser=0 | 18:37 |
| rlandy|ruck | arxcruz|ruck: ^^ | 18:37 |
| arxcruz|ruck | rlandy|ruck: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/35908 done | 19:03 |
| rlandy|ruck | arxcruz|ruck: great | 19:03 |
| rlandy|ruck | going to merge that | 19:03 |
| rlandy|ruck | we can try the pull tomorrow morning | 19:03 |
| rlandy|ruck | arxcruz|ruck: hmmm ... some lines have result != ABORTED twice | 19:07 |
| arxcruz|ruck | rlandy|ruck: notice now, i'll check | 19:09 |
| rlandy|ruck | arxcruz|ruck: k- no emergency | 19:09 |
| rlandy|ruck | when you have time | 19:09 |
| dviroel | rlandy|ruck: o/ | 21:10 |
| dviroel | is there any c7 job failing atm? with get_hash? | 21:10 |
| dviroel | my patch is failing on molecule job, using c7 instance | 21:11 |
| dviroel | https://zuul.opendev.org/t/openstack/build/833f2fed093542eeb46bd1308e3de3b8/log/job-output.txt#1004 | 21:11 |
| rlandy|ruck | dviroel: hi ... | 21:11 |
| * rlandy|ruck looks | 21:11 | |
| dviroel | ssl cert issue | 21:12 |
| rlandy|ruck | Invalid response received from the delorean server. Queried URL: https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md5. Response code: -1. Response text: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)>. Failed to create TripleOHashInfo object.", "msg": "Error something went wrong fetching hash info", "success": false} | 21:12 |
| rlandy|ruck | dviroel: OVB jobs are all down due to ssl issues | 21:13 |
| rlandy|ruck | that may be something different | 21:13 |
| rlandy|ruck | is that a consistent ox failure? | 21:13 |
| rlandy|ruck | dviroel: tox jobs were failing all over the place due to zuul-jobs change | 21:15 |
| rlandy|ruck | which just got fixed | 21:15 |
| rlandy|ruck | can I recheck your job? | 21:15 |
| dviroel | only saw that happening in molecule jobs | 21:15 |
| dviroel | let me se if i can reproduce without tox/molecule | 21:16 |
| rlandy|ruck | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-tox-molecule&project=openstack%2Ftripleo-repos | 21:16 |
| rlandy|ruck | previous failure was different issue | 21:18 |
| * rlandy|ruck check other dlrn operations | 21:18 | |
| rlandy|ruck | dviroel: (unrelated) could you vote on https://review.opendev.org/c/openstack/tripleo-ci/+/811973/ - just so I get a second vote before merge? | 21:19 |
| dviroel | rlandy|ruck: http://pastebin.test.redhat.com/997976 | 21:20 |
| rlandy|ruck | dviroel: maybe you're hitting what OVB is https://bugs.launchpad.net/tripleo/+bug/1945678/comments/2 | 21:22 |
| rlandy|ruck | Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)) | 21:22 |
| dviroel | rlandy|ruck: reviewed, 'tripleo_ansible/roles/tripleo_lvmfilter' is correct. +1 | 21:23 |
| rlandy|ruck | thanks | 21:23 |
| dviroel | rlandy|ruck: hum, yeah, maybe, also py2.7 | 21:23 |
| dviroel | the same thing is not happening with py3 | 21:23 |
| rlandy|ruck | interesting | 21:24 |
| rlandy|ruck | the OVB error is in the BMC node | 21:24 |
| rlandy|ruck | so neither | 21:24 |
| * rlandy|ruck look sgain | 21:24 | |
| rlandy|ruck | dviroel: and the exact same command with python3 passes? | 21:25 |
| dviroel | rlandy|ruck: different lib | 21:25 |
| dviroel | rlandy|ruck: https://github.com/openstack/tripleo-repos/blob/master/plugins/module_utils/tripleo_repos/get_hash/tripleo_hash_info.py#L28-L59 | 21:26 |
| rlandy|ruck | let's check other c7 | 21:27 |
| * dviroel checks py3 again | 21:27 | |
| dviroel | it is passing for py3, same url | 21:30 |
| rlandy|ruck | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-content-provider | 21:30 |
| rlandy|ruck | c7 content providers are ok | 21:30 |
| rlandy|ruck | dviroel: can only guess that the python2 code path is not running through auth correctly | 21:32 |
| rlandy|ruck | don't have other good traces of it | 21:32 |
| rlandy|ruck | otherwise we would have gate failures | 21:32 |
| rlandy|ruck | that path must be hit on all c7 jobs | 21:34 |
| rlandy|ruck | granted we don;t run many of them | 21:34 |
| rlandy|ruck | dviroel: so marios changed that code again for the fs039 failures | 21:38 |
| rlandy|ruck | so this may be a legit error | 21:38 |
| dviroel | rlandy|ruck: the same code was working yesterday | 21:39 |
| rlandy|ruck | dviroel: so then either we have an issue with rdo auth | 21:40 |
| rlandy|ruck | which is possible | 21:40 |
| rlandy|ruck | they made a change to the DNS on one network | 21:40 |
| rlandy|ruck | so it's possible that is hitting you as well | 21:40 |
| rlandy|ruck | just on the c7 path | 21:40 |
| rlandy|ruck | we have no other similar jon running and passing since then | 21:41 |
| rlandy|ruck | so you have a legit issue | 21:41 |
| rlandy|ruck | just depends where the change is | 21:42 |
| rlandy|ruck | <dpawlik> Today we added a second DNS entry to the public network subnets | 21:46 |
| rlandy|ruck | http://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp16-2/current-tripleo/delorean.repo.md5 | 21:48 |
| rlandy|ruck | dviroel: ^^ does this give you the same error? | 21:48 |
| dviroel | oh, no, it works with this link | 21:49 |
| dviroel | https://www.irccloud.com/pastebin/FNUL8Bj6/ | 21:50 |
| dviroel | ah, but this is not a https link | 21:51 |
| dviroel | "https://bugs.launchpad.net/tripleo/" for instance also works | 21:53 |
| rlandy|ruck | dviroel: so I guess then it might be the vexx chnage | 21:54 |
| rlandy|ruck | dviroel: I am waiting on a response from vexx about that | 21:54 |
| rlandy|ruck | <dpawlik|afk> rlandy|ruck: I guess it is related to letsencrypt cert :P | 21:54 |
| rlandy|ruck | dviroel: ping me again tomorrow | 21:54 |
| rlandy|ruck | will update you on vexx status | 21:54 |
| rlandy|ruck | and you can try again | 21:54 |
| dviroel | rlandy|ruck: ok, thanks | 21:55 |
| rlandy|ruck | if other https works | 21:55 |
| dviroel | rlandy|ruck: going out for the day | 21:56 |
| rlandy|ruck | dviroel: have a good night | 21:56 |
| dviroel | rlandy|ruck: you too | 21:56 |
| *** dviroel is now known as dviroel|out | 21:56 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!