| *** dviroel|rover is now known as dviroel|out | 01:09 | |
| dviroel|out | o/ | 01:09 |
|---|---|---|
| *** rlandy is now known as rlandy|out | 01:27 | |
| *** ysandeep|out is now known as ysandeep | 05:17 | |
| ysandeep | soniya29|ruck, fyi.. We are seeing some jobs failure today with retry_limit and no logs: https://zuul.openstack.org/builds?result=RETRY_LIMIT&result=RETRY&skip=0 | 05:19 |
| ysandeep | I have pinged on #opendev | 05:19 |
| akahat | Good Morning!! | 05:38 |
| * soniya29|ruck looking | 05:54 | |
| soniya29|ruck | ysandeep, ack | 05:55 |
| *** pojadhav|out is now known as pojadhav | 05:57 | |
| jm1 | o/ | 06:49 |
| *** ysandeep is now known as ysandeep|brb | 07:06 | |
| *** jpena|off is now known as jpena | 07:40 | |
| * pojadhav brb | 07:59 | |
| *** ysandeep|brb is now known as ysandeep | 07:59 | |
| ysandeep | bhagyashris, please check chat on internal pcci channel | 08:56 |
| bhagyashris | ysandeep, yes | 09:15 |
| frenzyfriday | jm1, hey, do you know this error KeyError: 'volumes is not found. openstack.compute.v2.server.Server objects do not support setting arbitrary keys through the dict interface ? | 09:19 |
| frenzyfriday | I am trying to run https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/server_provision/tasks/main.yml#L25 | 09:19 |
| jm1 | are you using aoc 1.x.x with openstacksdk >=0.99.0? | 09:20 |
| jm1 | frenzyfriday: ^ | 09:20 |
| frenzyfriday | jm1, openstacksdk is 0.101.1.dev4, checking the aoc version | 09:25 |
| frenzyfriday | yep, latest master aoc | 09:26 |
| *** ysandeep is now known as ysandeep|lunch | 09:47 | |
| *** ysandeep|lunch is now known as ysandeep | 10:27 | |
| *** rlandy_ is now known as rlandy | 10:37 | |
| rlandy | soniya29|ruck: hello | 10:40 |
| soniya29|ruck | rlandy, hello | 10:40 |
| rlandy | let's sync to get things up to date before the next ruck rovers | 10:40 |
| soniya29|ruck | rlandy, sure | 10:41 |
| rlandy | soniya29|ruck: https://meet.google.com/ohi-raiq-cxh?pli=1&authuser=0 | 10:42 |
| jm1 | rlandy: are you able to connect to vexxhost? | 10:44 |
| rlandy | jm1; connect in what way? | 10:45 |
| rlandy | ssh to nodes? | 10:45 |
| jm1 | rlandy: open the webinterface | 10:45 |
| jm1 | rlandy: https://rdo.vexxhost.ca/ | 10:45 |
| jm1 | rlandy: and then login | 10:45 |
| rlandy | rdo.vexxhost.ca uses an invalid security certificate. | 10:46 |
| rlandy | jm1: ^^ that? | 10:46 |
| frenzyfriday | even after that I am getting default backend - 404 | 10:46 |
| chandankumar | jm1: switch to https://identity.rdo.mtl2.vexxhost.net/v3 | 10:46 |
| chandankumar | we fixed certiticate issue by switching to this | 10:47 |
| rlandy | jm1: did you report to #vexxhost? | 10:47 |
| chandankumar | oh you need web interface | 10:47 |
| rlandy | jm1: pls ping guilhermesp on #vexxhost to report it | 10:48 |
| soniya29|ruck | rlandy, https://logserver.rdoproject.org/83/40083/20/check/periodic-tripleo-ci-centos-9-scenario001-standalone-master/2f66046/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 10:49 |
| soniya29|ruck | rlandy, "Error: can only create exec sessions on running containers: container state improper", "stderr_lines": ["Error: can only create exec sessions on running containers: container state improper"] | 10:50 |
| jm1 | rlandy: ack, pinged him | 10:54 |
| jm1 | chandankumar: right, the horizon dashboard has issues. no idea if this is a new or known issue, but reported it on #vexxhost | 10:55 |
| soniya29|ruck | rlandy, https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/c945d2a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 10:57 |
| rlandy | frenzyfriday:hi - how's the downstream dashboard? | 11:14 |
| rlandy | frenzyfriday: pojadhav: ^^ pls make this your top priority to get sorted | 11:14 |
| rlandy | if the PSI one exists, can you post the ip address? | 11:14 |
| jm1 | rlandy: we are working on it atm ;) | 11:20 |
| jm1 | rlandy: frenzyfriday is, i am just watching ^^ | 11:20 |
| rlandy | frenzyfriday++ | 11:20 |
| rlandy | jm1 thanks for supervising | 11:20 |
| * soniya29|ruck tea break | 11:23 | |
| *** soniya29|ruck is now known as soniya29|ruck|afk | 11:23 | |
| *** dviroel|out is now known as dviroel|rover | 11:23 | |
| dviroel|rover | o/ | 11:25 |
| * bhagyashris brb | 11:28 | |
| rlandy | dviroel|rover: hi | 11:36 |
| rlandy | dviroel|rover: did a quick meet with soniya29|ruck|afk earlier | 11:37 |
| rlandy | few issues on master we should discuss | 11:37 |
| rlandy | giving soniya29|ruck|afk a few to return | 11:37 |
| rlandy | dviroel|rover: soniya29|ruck|afk added two new master bugs to the rr hackmd | 11:38 |
| rlandy | jm1: can you update bitwarden with https://dashboard.rdo.mtl2.vexxhost.net/? | 11:38 |
| dviroel|rover | rlandy: ack, i will take a look then | 11:39 |
| rlandy | dviroel|rover: let's sync when soniya29|ruck|afk is back | 11:39 |
| rlandy | jm1: nvm - I'm on bw - will update it | 11:39 |
| frenzyfriday | folk, if the promoter downstream has any issues with nameserver/unable to resolve internal hostnames pls let us know. The infra scripts probably changes the nameserver in the private network of psi | 11:40 |
| rlandy | updated | 11:42 |
| rlandy | frenzyfriday: still http://tripleo-cockpit.lab4.eng.bos.redhat.com/? | 11:43 |
| rlandy | frenzyfriday: or is there a new link? | 11:44 |
| rlandy | ohhhh promoter | 11:44 |
| rlandy | sorry | 11:44 |
| rlandy | promoter worked 37 mins ago | 11:44 |
| frenzyfriday | rlandy, incockpit is still not deployed. jm1 and me are still working - the setup scripts are old and needs a lot of update | 11:44 |
| rlandy | yeah - sorry - read it wrong | 11:44 |
| frenzyfriday | yeah, we might have accidentally changed nameserber in the private nw of psi - so if promoter breaks it might be due to this | 11:45 |
| jm1 | rlandy, chandankumar: has downstream promoter been set up with ci-config/infra-setup playbooks? or manually? | 11:45 |
| rlandy | also playbboks | 11:46 |
| rlandy | jm1: chandankumar is traveling - you acn ask akahat | 11:47 |
| rlandy | bhagyashris: welcome back | 11:47 |
| rlandy | bhagyashris: how is 17.1 going | 11:47 |
| akahat | jm1, All promoter servers are setup using infra-setup playbooks. | 11:48 |
| dviroel|rover | rlandy: soniya29|ruck|afk: https://opendev.org/openstack/tripleo-heat-templates/commit/298463b8104665b7cd7afea70c6d2952384d6b89 | 11:50 |
| dviroel|rover | code that is failing on scn01 | 11:50 |
| dviroel|rover | added as comment | 11:50 |
| jm1 | akahat: isn't downstream promoter on psi? | 11:51 |
| akahat | jm1, I don't know where it is hosted. :( But yes i configured it. | 11:52 |
| rlandy | dviroel|rover: revert? | 11:52 |
| rlandy | or at least test a revert - blocking promotions | 11:52 |
| rlandy | that and the failure on fs035 | 11:53 |
| jm1 | akahat: our ci-config/infra-setup is missing the psi config part completely | 11:53 |
| rlandy | jm1: it's internal - can't put that in public repo | 11:53 |
| frenzyfriday | do we have an internal repo where we have the set up/server provision scripts for psi? | 11:54 |
| rlandy | it's in rrcockpit | 11:54 |
| rlandy | on codeng | 11:54 |
| dviroel|rover | rlandy: yeah, i will give a try | 11:54 |
| rlandy | dviroel|rover: if soniya29|ruck|afk is not back in 5 - let's sync | 11:54 |
| rlandy | there is another bug to add | 11:54 |
| dviroel|rover | ok | 11:55 |
| jm1 | rlandy: rrcockpit on codeeng only has config info about telegraf etc. but it is missing the provisioning parts such as networks, subnets etc. | 11:57 |
| rlandy | that stuff is manual | 11:58 |
| jm1 | rlandy: it does not have to be. our vexxhost based infra is completely automated as well. | 11:59 |
| rlandy | jm1: feel free to fix it :) | 12:01 |
| rlandy | it should be the same as vexx with diff settings | 12:02 |
| frenzyfriday | where do we get a cert to connect to https://code.engineering.redhat.com from the psi servers? | 12:03 |
| bhagyashris | rlandy, hey 17.1 is blocked due to this issue https://bugzilla.redhat.com/show_bug.cgi?id=2121333 | 12:04 |
| *** soniya29|ruck|afk is now known as soniya29|ruck | 12:05 | |
| soniya29|ruck | rlandy, dviroel|rover, i am back | 12:05 |
| dviroel|rover | https://review.rdoproject.org/r/c/testproject/+/44652 - testing revert here | 12:06 |
| rlandy | bhagyashris: do you have reviews to start the line? | 12:06 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 12:06 |
| dviroel|rover | soniya29|ruck: ^ | 12:06 |
| rlandy | with image and container builds? | 12:06 |
| rlandy | let's get those in and started | 12:06 |
| rlandy | so the hashes are moving | 12:06 |
| rlandy | dviroel|rover: soniya29|ruck: https://meet.google.com/qob-bizk-qek?pli=1&authuser=0 | 12:07 |
| bhagyashris | rlandy, all jobs definitions are in one patch | 12:08 |
| rlandy | bhagyashris: we can merge definitions | 12:08 |
| bhagyashris | rlandy, i will separate out the passing jobs in separate patch | 12:08 |
| rlandy | and just not put them in the lines | 12:08 |
| rlandy | yeah - that's fine | 12:08 |
| rlandy | let's get something moving | 12:08 |
| bhagyashris | rlandy, ok | 12:09 |
| bhagyashris | here is the patch https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/424293 | 12:09 |
| bhagyashris | rlandy, ack | 12:09 |
| rlandy | bhagyashris: possible to separate? | 12:10 |
| rlandy | just add what is working now? | 12:10 |
| rlandy | image and container builds in the line | 12:10 |
| bhagyashris | rlandy, sure | 12:10 |
| bhagyashris | np | 12:10 |
| rlandy | bhagyashris: you can even merge the definitions if wanted | 12:10 |
| rlandy | just keep the line with what works | 12:10 |
| bhagyashris | rlandy, i think just merged the patch having passing jobs | 12:11 |
| rlandy | bhagyashris++ | 12:11 |
| bhagyashris | give me min i will create patch | 12:11 |
| bhagyashris | rlandy, here is tha patch https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/426066 | 12:16 |
| *** tosky_ is now known as tosky | 12:25 | |
| *** ysandeep is now known as ysandeep|brb | 12:47 | |
| rlandy | pojadhav: hi - the retro was supposed to be half hour later - same time as wed | 12:53 |
| rlandy | dasm|off is not here yet | 12:54 |
| pojadhav | rlandy, ohh okay I thought same time as the scrum was on thurs. | 12:54 |
| pojadhav | willl do it now | 12:54 |
| rlandy | pojadhav: thank you - same time as the retro was on wednesday | 12:54 |
| rlandy | that was our compromise to the india/california time split | 12:55 |
| pojadhav | yep | 12:55 |
| jm1 | rlandy: automating the psi config takes time. atm we do not have docs or code for our infra on psi. imho we should not do or allow any manual provisioning for our production infra. lets talk about this in our scrum or retro. | 12:56 |
| rlandy | jm1: yep - internal cockpit was never on psi | 12:58 |
| rlandy | so any time there was a deployment there for dev env etc. - I guess it was ad hoc | 12:58 |
| rlandy | this is our first attempt to put production cockpit on psi | 12:58 |
| rcastillo | o/ | 12:59 |
| rlandy | I don't think it's a good idea to keep anything production on old, unsupported hardware | 12:59 |
| rlandy | so yeah, we are taking a hit now | 12:59 |
| rlandy | but otherwise we will take the hit later | 12:59 |
| jm1 | rlandy: i am pointing at downstream promoter here. the issue we have is that we have no code or docs and thus dont know what was done. we might have changed dns config on internal promoter subnet accidentially but we have no way to check. so expect outage once dhclient on psi infra has refreshed | 12:59 |
| jm1 | *dhclient inside vms on psi infra | 13:00 |
| dpawlik | hey, some progress in moving from registry.rdoproject.org to quay.rdoproject.org ? | 13:01 |
| jm1 | akahat: do you still have some docs or memory on what dns settings you set on the downstream provider subnet? | 13:02 |
| jm1 | akahat: it is 1.1.1.1 and 8.8.8.8 now which is very likely wrong since it wont resolve rh internal hosts | 13:03 |
| rlandy | bhagyashris: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/426066/1/zuul.d/rhos17.1-rhel8-jobs.yaml#79 | 13:04 |
| rlandy | override-checkout: rhos-17.0-trunk-patches - should be 17.1 | 13:05 |
| rlandy | bhagyashris: pls update that and I'll merge | 13:05 |
| akahat | jm1, it's nameserver 8.8.8.8 in /etc/resolve.conf on promoter server | 13:10 |
| bhagyashris | rlandy, ack | 13:25 |
| pojadhav | retro in 3 mins : arxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29|ruck , pojadhav, akahat, chandankumar, frenzy_friday, dviroel|rover , rcastillo, jm1, rlandy, dasm | 13:27 |
| soniya29|ruck | pojadhav, ack | 13:27 |
| pojadhav | miro board : https://miro.com/app/board/uXjVPdFzXJI=/ | 13:28 |
| bhagyashris | rlandy, Done | 13:29 |
| *** dasm|off is now known as dasm | 13:31 | |
| dasm | o/ | 13:31 |
| pojadhav | bhagyashris, ^^ | 13:31 |
| rlandy | frenzyfriday: jm1: yep - http://10.0.110.143/promoter_logs/container-push/ promoter is stopped | 14:04 |
| frenzyfriday | rlandy, do you know what are the correct nameservers for downstream? | 14:05 |
| jm1 | rlandy, akahat: dhclient on downstream promoter has updated /etc/resolv.conf to 8.8.8.8, so we have to update its subnet's nameservers | 14:07 |
| jm1 | rlandy, akahat, frenzyfriday: i have no access to downstream promoter, it looks like it does not accept our ssh keys | 14:08 |
| rlandy | frenzyfriday: sending in pvt | 14:08 |
| rlandy | jm1: akahat is correct | 14:08 |
| rlandy | that will not work downstream | 14:08 |
| rlandy | jm1: ^^ sent them to frenzyfriday | 14:10 |
| rlandy | what the downstream ones are | 14:10 |
| * frenzyfriday changing back the nameservers and hoping the promoter works | 14:10 | |
| frenzyfriday | nameserver is updated. Can we restart the promoter? Or maybe it will work on its own? | 14:12 |
| jm1 | frenzyfriday: simply wait till it updates itself | 14:12 |
| jm1 | rlandy: "jm1: akahat is correct" ? was there any discussion i missed? | 14:15 |
| *** ysandeep|brb is now known as ysandeep | 14:28 | |
| dviroel|rover | rcastillo++ | 14:33 |
| *** dviroel is now known as dviroel|rover | 15:02 | |
| *** tosky is now known as Guest1129 | 15:12 | |
| *** tosky_ is now known as tosky | 15:12 | |
| frenzyfriday | where do we get a cert to connect to https://code.engineering.redhat.com from the psi servers? we tried the crt mentioned on the Source but that didnt work | 15:15 |
| rlandy | frenzyfriday: sending to you in pvt | 15:32 |
| ysandeep | dasm: sorry for the noise, I misunderstood that telegraf container image build is hosted in Upstream dashboard. | 15:39 |
| ysandeep | good to hear its only downstream issue | 15:40 |
| dasm | ysandeep: no worries. i didn't really know which instance it was. that's why i raised the issue with finding the right instance during our retrospective meeting. | 15:40 |
| ysandeep | dasm, ack, its a valid concern.. we should do some better tracking | 15:42 |
| frenzyfriday | https://drive.google.com/drive/folders/10e3mmICAdjJaKuF3e7J8he0adlC1mXSC - meet transcript | 15:44 |
| ysandeep | frenzyfriday, need access :) | 15:44 |
| dviroel|rover | jm1: pojadhav: I will add a summary on rr hackmd by my eod | 15:44 |
| pojadhav | dviroel|rover, sure thank you!! | 15:45 |
| * dviroel|rover lunch | 15:45 | |
| *** dviroel|rover is now known as dviroel|rover|lunch | 15:45 | |
| jm1 | dviroel|rover|lunch: ok, thank you! | 15:47 |
| * pojadhav leaving for the day | 15:49 | |
| *** pojadhav is now known as pojadhav|out | 15:49 | |
| * jm1 bbl | 15:50 | |
| rlandy | 16.2 promoted | 15:51 |
| rlandy | dviroel|rover|lunch: ^^ | 15:53 |
| rlandy | dviroel|rover|lunch: have fs001 wallaby c8 in rerun, train still going | 15:53 |
| * rlandy goes back into meeting | 15:57 | |
| dasm | rlandy: good luck! :) | 15:58 |
| ysandeep | rlandy, we need to learn from you on how to deal with so many meetings :) | 15:58 |
| *** ysandeep is now known as ysandeep|out | 15:58 | |
| rlandy | ha - do less useful work | 15:59 |
| rlandy | == life as a manager - become a useless engineer | 16:00 |
| ysandeep|out | reviewbot: please add in review list https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/426159 and https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/426166 | 16:02 |
| reviewbot | I could not add the review to Review List | 16:02 |
| reviewbot | I could not add the review to Review List | 16:02 |
| *** jpena is now known as jpena|off | 16:41 | |
| rlandy | lunch - biab | 17:00 |
| *** dviroel|rover|lunch is now known as dviroel|rover | 17:10 | |
| dviroel|rover | dasm: O/ | 17:27 |
| dviroel|rover | dasm: anything that we can do to get this PR merged? https://github.com/oauthlib/oauthlib/pull/818 | 17:27 |
| dviroel|rover | it seems that they want a new ci to test that, but don't want to start the work | 17:28 |
| dviroel|rover | lol | 17:28 |
| dviroel|rover | they only have linter job | 17:30 |
| dviroel|rover | rlandy: need your review/vote here https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/854665 | 17:45 |
| dviroel|rover | ^ this is failing for a long time on internal | 17:46 |
| rlandy | not resoloving | 17:57 |
| rlandy | dviroel|rover: voted | 17:58 |
| rlandy | you can merge | 17:58 |
| rlandy | 17.1 promoted current-tripleo/2022-08-25 | 17:59 |
| rlandy | as did 16.2 | 17:59 |
| dviroel|rover | nice | 18:01 |
| dasm | dviroel|rover: i'm asking them every couple weeks. it's still: "we're working on CI" | 18:01 |
| dasm | wrt oauthlib | 18:01 |
| dviroel|rover | dasm: =/ ok tks | 18:02 |
| dasm | yeah. i even asked if they need help with setting things up... no answer | 18:02 |
| dviroel|rover | dasm: were you able to test with fs035? | 18:02 |
| dasm | no. can we pull changes from github and incorporate them into test? | 18:03 |
| dviroel|rover | dasm: i do think so, but never tested | 18:03 |
| dasm | hmm | 18:03 |
| dviroel|rover | dasm: there is a code that do that, but never tried | 18:04 |
| dviroel|rover | fetch from PR | 18:04 |
| dasm | oh, there is? i know we can include pr in requirements (i think) | 18:04 |
| dviroel|rover | dasm: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/tasks/main.yml#L264 | 18:08 |
| dasm | neat. i'll give it a try | 18:11 |
| dviroel|rover | dasm: yeah, probably won't work | 18:12 |
| dviroel|rover | dasm: i was created for specific projects: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/defaults/main.yml#L87 | 18:12 |
| dasm | hmm | 18:12 |
| dviroel|rover | dasm: you could try to provide this code in a repo? | 18:12 |
| dasm | what do you mean by "provide in a repo"? | 18:13 |
| dasm | afair, we're installing oauthlib from pip | 18:13 |
| dviroel|rover | dasm: yeah, maybe you could build and push it to pip with a different name, and install your modified version? | 18:15 |
| dviroel|rover | dasm: just ideas | 18:15 |
| dasm | i'll think about it and i'll see what i can do. thanks dviroel|rover++ | 18:16 |
| dviroel|rover | dasm++ for proposing that fix | 18:18 |
| rlandy | dviroel|rover: all clear on downstream promos | 18:21 |
| dviroel|rover | ~o~ | 18:23 |
| dviroel|rover | S2 downstream | 18:23 |
| rlandy | dviroel|rover: did the revert work for sc01? | 18:26 |
| dviroel|rover | rlandy: yes | 18:27 |
| rlandy | dviroel|rover: should we push to merge? | 18:27 |
| dviroel|rover | rlandy: i have faith that downstream will give us fs035 and fs064 | 18:29 |
| dviroel|rover | so we will miss only scenario001 | 18:29 |
| rlandy | dviroel|rover: I'm all about faith | 18:30 |
| rlandy | then we should push to get that revert in | 18:30 |
| rlandy | and clear the line | 18:30 |
| rlandy | rerunning w c8 fa001 | 18:30 |
| rlandy | only one missing there | 18:30 |
| dviroel|rover | rlandy: I'm adding internal 039 and 064 jobs, they never run on internal | 18:30 |
| dviroel|rover | fs035 didn't pass because of skiplist | 18:31 |
| * dviroel|rover because was missing on skiplist, not skiplist fault | 18:31 | |
| * dviroel|rover biab\ | 18:32 | |
| * dviroel|rover back | 18:47 | |
| dviroel|rover | rlandy: ah, revert test results is here | 18:53 |
| dviroel|rover | https://review.rdoproject.org/r/c/testproject/+/44652/2#message-f1e828a0bb573bce701664f7af2dce96090ae868 | 18:53 |
| rlandy | bhagyashris: merged container builds and image build lines | 18:55 |
| rlandy | now we need config patch to get that line started again | 18:55 |
| rlandy | dviroel|rover: if mrunge is still around - ask him to vote | 18:56 |
| rlandy | then we can merge to clear the line | 18:56 |
| *** kopecmartin is now known as kopecmartin|pto | 18:57 | |
| dviroel|rover | rlandy: ack, I will ping him | 19:08 |
| dviroel|rover | rlandy: wondering why this is not failing on check | 19:10 |
| dviroel|rover | new bits on integration lines that triggers that? | 19:10 |
| dviroel|rover | can be related to infra? | 19:10 |
| rlandy | dviroel|rover: good question - master is old in current-tripleo? | 19:10 |
| rlandy | you could test that | 19:11 |
| rlandy | run the check job in vexx | 19:11 |
| dviroel|rover | yeah | 19:11 |
| dviroel|rover | oh, missing CentOS-Stream-GenericCloud-9-20211216 on interna; | 19:34 |
| * jm1 out for today, see you tomorrow :) | 19:46 | |
| dviroel|rover | rlandy: so, for master, our best hash is tripleo-ci-testing/89/fb/89fbcbce79c4a970207d7f5952937f1c from 2 days ago | 19:47 |
| dviroel|rover | missing only: periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master | 19:47 |
| rlandy | dviroel|rover: does that hash include the fix for fs039? | 19:49 |
| rlandy | promoted up from the component?> | 19:49 |
| rlandy | if yes, you can skip promote | 19:49 |
| dviroel|rover | let me check | 19:49 |
| rlandy | we can make a wallaby and train call at your EoD | 19:51 |
| dviroel|rover | rlandy: yes, it include that fix | 19:53 |
| dviroel|rover | https://trunk.rdoproject.org/centos9-master/component/security/e3/af/e3afdf14b6569d04c02481c4c5aeb11e854b5043_5237618a | 19:54 |
| dviroel|rover | component ^ | 19:54 |
| dviroel|rover | https://opendev.org/x/tripleo-ipa/commit/e3afdf14b6569d04c02481c4c5aeb11e854b5043 | 19:54 |
| dviroel|rover | fix ^ | 19:54 |
| dviroel|rover | https://trunk.rdoproject.org/centos9-master/tripleo-ci-testing/89/fb/89fbcbce79c4a970207d7f5952937f1c/delorean.repo | 19:54 |
| dviroel|rover | ^ aggregate hash | 19:54 |
| rlandy | dviroel|rover: and fs039 - tempest failures? | 19:57 |
| rlandy | not repeatable - go ahead and skip promote | 19:57 |
| dviroel|rover | let me get results, but were different failures | 19:58 |
| dviroel|rover | rlandy: all were deployment failures, overcloud deploy, or freeipa install errors | 20:03 |
| dviroel|rover | no tempest | 20:03 |
| dviroel|rover | maybe if we try again | 20:03 |
| rlandy | lol | 20:04 |
| rlandy | dviroel|rover: one try then go agead | 20:05 |
| dviroel|rover | (╯°□°)╯︵ ┻━┻ | 20:05 |
| rlandy | 7 days old master is more of a risk | 20:05 |
| dviroel|rover | rlandy: also, i will promote security wallaby, lets see how it goes on integration | 20:05 |
| dviroel|rover | rlandy: i am using this workaround to avoid ipa_setup failure: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/854180 | 20:09 |
| dviroel|rover | it works, but it is ugly | 20:09 |
| rlandy | dviroel|rover: lol | 20:12 |
| rlandy | not the prettiest - but ok | 20:12 |
| rlandy | dviroel|rover: only thing - I would keep the package_manager var | 20:13 |
| dviroel|rover | rlandy: yeah, i was worried about ansible replacing it with those quotes that I added, but I can easily validate on my machine too] | 20:16 |
| dviroel|rover | * not properly replacing the var | 20:16 |
| dviroel|rover | rlandy: uploaded a new CentOS Image on internal, needed for ovb extra node | 20:17 |
| dviroel|rover | will try to run 039 there | 20:17 |
| dviroel|rover | was failing on ovn | 20:17 |
| dviroel|rover | ovb* | 20:18 |
| rlandy | ok | 20:42 |
| rlandy | dviroel|rover: train still missing fs039 and fs035 | 20:43 |
| rlandy | looking at if it's safe to skip or not | 20:43 |
| rlandy | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train&skip=0 | 20:43 |
| rlandy | tempest.api.compute.admin.test_volumes_negative.VolumesAdminNegativeTest - passed on - failed the other | 20:45 |
| dviroel|rover | i think I just saw that on skiplist | 20:46 |
| rlandy | dviroel|rover: so ... comparing https://logserver.rdoproject.org/54/36254/153/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/6c55629/logs/undercloud/var/log/tempest/stestr_results.html.gz and https://logserver.rdoproject.org/54/36254/152/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/a2c8873/logs/undercloud/var/log/tempest/stestr_results.html.gz | 20:46 |
| rlandy | we could go with fs035 failing | 20:47 |
| rlandy | checking fs039 | 20:47 |
| dviroel|rover | tempest.api.compute.admin.test_volumes_negative.VolumesAdminNegativeTest | 20:47 |
| dviroel|rover | skipped for https://bugs.launchpad.net/tripleo/+bug/1933999 | 20:47 |
| rlandy | still watching fs039 | 20:50 |
| rlandy | fs01 wallany as well | 20:50 |
| dviroel|rover | ack | 20:52 |
| rlandy | dviroel|rover: updated hackmd for downstream stuff | 21:09 |
| dviroel|rover | rlandy: tyvm | 21:11 |
| dviroel|rover | this was a bad rover shift at the end | 21:31 |
| dviroel|rover | dviroel|rover-- | 21:31 |
| dviroel|rover | vexxhost-- | 21:31 |
| rlandy | dviroel|rover: not your fault | 21:36 |
| rlandy | vexxhost stuff | 21:37 |
| rlandy | dviroel|rover: need to go out for about an hour - pls leave me any reviews to merge | 21:38 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 21:38 |
| rlandy | will be back | 21:38 |
| dviroel|rover | rlandy: ok | 21:38 |
| *** rlandy is now known as rlandy|bbl | 21:38 | |
| dasm | hmmm... i think i might've found the last valid cs9 release, which boots up via ipxe. | 21:46 |
| dasm | seems like something has changed between CentOS-Stream-GenericCloud-9-20220606.0 and CentOS-Stream-GenericCloud-9-20220621.1 impacting the way how ipxe starts. | 21:46 |
| dasm | one interesting thing: both releases are next to each other, so there is a 2 weeks gap. Other releases seem to be every week. | 21:47 |
| dviroel|rover | rlandy|bbl: there is no other way https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44654 | 22:02 |
| * rcastillo|rover leaving for today | 22:02 | |
| *** tosky_ is now known as tosky | 22:06 | |
| dviroel|rover | rlandy|bbl: fs001-wallaby-c8 failed on tempest tests, waiting for logs | 22:30 |
| *** dasm is now known as dasm|off | 22:32 | |
| * dasm|off => offline | 22:32 | |
| dasm|off | o/ | 22:32 |
| dviroel|rover | why rcastillo|rover is still rover? lol | 22:33 |
| dasm|off | :) | 22:36 |
| dviroel|rover | rlandy|bbl: https://logserver.rdoproject.org/55/36255/109/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/3509e75/logs/undercloud/var/log/tempest/stestr_results.html.gz | 22:38 |
| dviroel|rover | lots of tests failing with 500, this looks like our old infra issues :/ | 22:38 |
| *** rlandy|bbl is now known as rlandy | 22:47 | |
| rlandy | dviroel|rover: looking | 22:47 |
| rlandy | dviroel|rover: merging https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44654 | 22:48 |
| rlandy | pls leave note for rr to revert it | 22:48 |
| rlandy | dviroel|rover: we can promote this train hash- only missing fs035 | 22:49 |
| rlandy | I'll rerun wallaby c8 later tonight | 22:49 |
| * rlandy puts in train patch | 22:49 | |
| dviroel|rover | ack | 22:50 |
| * dviroel|rover dinner | 22:50 | |
| rlandy | dviroel|rover: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44655 Temp skip train fs035 | 22:51 |
| dviroel|rover | we could improve our promoter to accept a OR condition | 23:18 |
| dviroel|rover | like fs035 or fs035-internal :) | 23:18 |
| dviroel|rover | how are we with wallaby c9? | 23:33 |
| dviroel|rover | i have this hash here 2779b7691127687f9beeb36a4e686d3c | 23:33 |
| dviroel|rover | trying that again | 23:38 |
| rlandy | ok | 23:54 |
| rlandy | dviroel|rover: good idea on the promoter | 23:55 |
| rlandy | giving wallaby c8 one last try | 23:57 |
| rlandy | dviroel|rover: late for you | 23:57 |
| * rlandy will update rr page after promotions | 23:57 | |
| rlandy | going to grocery store - biab | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!