Friday, 2022-07-15

« Thursday, 2022-07-14 Index Saturday, 2022-07-16 »
*** rlandy|bbl is now known as rlandy|out01:10
*** dviroel|rover|afk is now known as dviroel|rover01:17
*** ysandeep|out is now known as ysandeep01:37
ysandeephello oooci o/ morning 01:38
*** ysandeep is now known as ysandeep|afk03:45
mariosfridaaaaayyyy \o/05:25
*** ysandeep|afk is now known as ysandeep06:16
ysandeephappy friday marios and all o/06:20
marios\o06:24
jm1o/06:46
marioschandankumar: o/ please vote when you get a chance (the two should go together) https://review.opendev.org/c/openstack/tripleo-ci/+/849883 06:53
*** chandankumar is now known as chkumar|rover06:59
chkumar|roverchkumar|rover: done07:02
chkumar|roverjm1: \o07:03
chkumar|roverjm1: please address the rlandy and doug suggestion on this review https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/847419 when free, thanks!07:03
jm1chkumar|rover: working on it..07:04
marioshttps://logserver.rdoproject.org/58/43558/13/check/tripleo-ci-centos-8-9-multinode-mixed-os/e621742/logs/undercloud/var/log/tempest/stestr_results.html.gz \o/07:43
* marios collapses07:43
jm1marios: woohooo 🥳09:04
mariosjm1: ;)09:09
chkumar|rovermarios: awesome \o/09:13
marioschkumar|rover: \o/09:19
bhagyashrismarios, woohoo \o/09:21
marios:) 09:22
ysandeepmarios, so in this job we have centos9 controller and centos8 compute right?09:22
mariosysandeep: why do i feel like i am not going to like your next statement :)09:23
mariosysandeep: yes09:23
mariosysandeep: ... did you spot something?09:26
ysandeepmarios: cool thanks.. nah all good, just wondering will we support instance migration from c8 compute to c9 compute and vice versa.09:26
mariosysandeep: not sure about that and don't know if it has been fully decided. not just instance migration, but generally there will be a 'degraded' support when in this state (this will happen during an upgrade so the deployment is still 'in the upgrade state' the whole time until all the 8 computes are done)09:27
mariosysandeep: so don't know if will be supported or what will be supported09:27
mariosysandeep: like upgrades/compute/df have to agree and sign off on what they can support for day2 operations in that state09:28
marios(scale out etc)09:28
ysandeepah okay09:29
ysandeepso we deploying ctlr stack first and then overcloud deploy09:30
ysandeepnice we have logs for both https://logserver.rdoproject.org/58/43558/13/check/tripleo-ci-centos-8-9-multinode-mixed-os/e621742/logs/undercloud/home/zuul/ctrl_overcloud-deploy.sh.txt.gz and https://logserver.rdoproject.org/58/43558/13/check/tripleo-ci-centos-8-9-multinode-mixed-os/e621742/logs/undercloud/home/zuul/overcloud-deploy.sh.txt.gz09:31
mariosyeah i actually just kept the default name (as i didn't need to change it/didn't give a value) so i just have 'overcloud' for both. but yes, first deployment of control then compute09:31
mariosysandeep: yeah i just kept some of those logs but we can revisit/add whatever09:31
mariosysandeep: for eaxmple maybe using different stack name instaed of 'overcloud' for both will give us better log files? perhaps that is a value09:32
mariosbut anyway there will be lots to fix and improve but first would like to start getting the base merged https://review.opendev.org/q/topic:oooci_mixed_rhel which will take a few days at least i think09:32
chkumar|rovermarios: I have left some initial comments, I am not sure you saw that09:38
chkumar|roveron mixed rhel patches09:38
marioschkumar|rover: thank you I saw but was working on the final bug for tempest first ... will start to work on those09:39
chkumar|rovermarios: thank you :-)09:39
ysandeepmarios, very creative of you to think out of the box here, deploying single controller in one stack and in second stack use controller of first stack.. I know about multi stack / DCN extended to edge , looks like you mixed both DCN and multi stack in a single approach for this CI job.09:41
ysandeepafair.. In DCN deployment we use single overcloud stack09:42
ysandeepah.. no my memory failed me - IN dcn we use multiple stacks: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/distributed_compute_node_and_storage_deployment/assembly_deploy-edge-without-storage 09:43
ysandeepso basically this job is DCN deployment in CI with different base OS in central and edge site09:44
mariosysandeep: 'creative' ;) yeah i was happy to find the multi stack stuff otherwise not sure how we would do this09:45
ysandeepso we covered.. controller son 9 and computes on 8 scenario with this job.. Do we also need to cover 1 controller on 9 and other on 8 scenario :)?09:50
ysandeepcontroller on* 909:50
mariosysandeep: in future.. for mvp the scenario is specifically 9 control 8 compute09:52
ysandeepokay, thanks for info o/ marios++ great work09:53
mariosthanks ysandeep 09:54
ysandeepintesting.. https://logserver.rdoproject.org/58/43558/13/check/tripleo-ci-centos-8-9-multinode-mixed-os/e621742/logs/subnode-1/etc/hosts.txt.gz don't have compute node entry but still everything worked.. not sure how it resolved compute node to its ip.. normally controller have entry for compute nodes in /etc/hosts.10:04
ysandeepbut may be things work differently in DCN(I don't have exp with dcn deployments)10:05
mariosysandeep: well not quite 'everything worked' - we had to run nova cell discover_hosts after compute dpeloyment, for controller (nova_api) to discover the compute node and be able to finally boot vms10:06
mariosysandeep: there https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/841764/29/playbooks/multinode-overcloud-mixed-os-deploy-compute.yml#6710:07
ysandeepack, I see we have mentioned similiar steps here to connect to remote compute node10:20
ysandeephttps://opendev.org/openstack/tripleo-docs/src/branch/master/deploy-guide/source/deployment/standalone.rst#discover-the-remote-compute-node-from-the-central-controller-node10:20
mariosysandeep: ha i wish i found that first :) i just saw what the 'normal' deployment would do with https://opendev.org/openstack/tripleo-heat-templates/src/commit/48deb4cbb53d187454c7de82e7125900e93926d1/deployment/nova/nova-compute-common-container-puppet.yaml#L59-L78 and recreated it10:21
mariosysandeep: (https://bugs.launchpad.net/tripleo/+bug/1981459 )10:21
ysandeep:) I wonder if we had to do similiar step in dcn deployment but I don't find the same workaround in docs for dcn.. anyway +1 (I will revisit again)10:28
*** rlandy|out is now known as rlandy10:31
rlandychkumar|rover: hey - how are things?10:32
chkumar|roverrlandy: everything seems to be normal10:32
rlandywant to sync or wait for dasm to come line?10:33
chkumar|roverrlandy: rhos-16.2 container image update looks good10:33
chkumar|rovernothing seen in the logs10:33
rlandychkumar|rover: yay10:33
chkumar|roverrlandy: let's wait for dasm|off to come online then we can sync10:33
rlandychkumar|rover: going to rerun the two failed jobs in 16.210:34
rlandyso we can promote the new base image10:34
chkumar|roverrlandy: already re-runned those10:34
chkumar|roverrlandy: https://hackmd.io/KrGAxws2QiaHuJmj8TH6Jg?view10:34
chkumar|roverfor all release failed jobs which seems not good have testproject running10:35
rlandychkumar|rover: https://code.engineering.redhat.com/gerrit/c/testproject/+/419956/2/.zuul.yaml are 17 reruns10:36
rlandywe need 16.210:36
rlandythe same testproject is linked in 16.2 and 17 lines10:37
rlandyis there another one?10:37
rlandyRH8 RHOS-16.2 Last promotion: 12th july10:37
rlandy    Re-running failed job: https://code.engineering.redhat.com/gerrit/c/testproject/+/41995610:37
rlandyRH8 RHOS-17 Last promotion: 13th july10:37
rlandy    Re-running failed job: https://code.engineering.redhat.com/gerrit/c/testproject/+/41995610:37
rlandysame job10:37
chkumar|roveroh sorry10:40
chkumar|roveri updated the same patchset10:40
chkumar|roverlet me re0run 16.2 one10:40
rlandygreat  - thanks10:41
rlandychkumar|rover: dviroel|rover said using the bigger flavor for train did not help10:46
chkumar|roverrlandy: I was looking at master fs03510:46
rlandyit may help for master if you want to try that out10:46
chkumar|roverhttps://paste.centos.org/view/raw/09698f7510:46
chkumar|roverI picked one ipv6 test which is failing in 3 last runs10:46
chkumar|rovertrying to find out the issue10:46
rlandyyep - would be good to try see what is going on10:47
rlandyfrenzy_friday: hello11:00
rlandyfrenzy_friday: how are things with 17.1?11:00
rlandychkumar|rover: +2'ed https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/419819 - but needs rebase11:01
rlandyfrenzy_friday: reviewed your jobs11:01
rlandycan we run them now?11:01
rlandyfrenzy_friday: all merged11:02
rlandylet's run those today and get the hash moved along11:03
rlandyping if you need help11:03
chkumar|roverrlandy: fixed merged conflict https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/419819 needs your vote again11:07
rlandychkumar|rover: done11:10
chkumar|roverrlandy: I am marking this bug https://bugs.launchpad.net/tripleo/+bug/1981634 as a duplicate of https://bugs.launchpad.net/tripleo/+bug/198146711:24
rlandywill look in a sec - on review time11:24
frenzy_fridayrlandy, yep, running them now11:25
rlandyfrenzy_friday: thanks11:25
*** dviroel|rover is now known as dviroel11:28
* dviroel not rover anymore11:29
rlandyfrenzy_friday: need to fix ...11:47
frenzy_fridayrlandy, which one?11:47
rlandyfrenzy_friday: see error on job11:47
rlandygetting11:47
frenzy_fridayoh, checking11:48
mariosfood biab11:48
rlandyThe error was: {{ dlrnapi_protocol }}://{{ dlrnapi_host }}/{{ dlrn_baseurl[distro][osp_release|default(release)] }}: 'dict object' has no attribute 'rhos-17.1'\n\nThe error appears to be in '/var/lib/zuul/builds/7eaac3717da54e87a87529efe91eb758/trusted/project_4/review.rdoproject.org/rdo-infra/ci-config/ci-scripts/infra-setup/roles/get_hash/tasks/get_hash.yaml': line 16, column 3, but may\nbe elsewhere in the file depending on 11:48
rlandythe exact syntax problem.\n\nThe offending line appears to be:\n\n# commit/distro hashes for component pipeline (an unique commit hash for each component)\n- name: discover the hashes url (component)\n  ^ here\n"11:48
rlandywe need to add this option11:49
rlandyfrenzy_friday: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/get_hash/defaults/main.yml11:50
rlandyyou need to add 17.1 options there11:50
frenzy_fridayoohh, ack, adding11:52
*** ysandeep is now known as ysandeep|afk11:56
chkumar|roverdviroel: it is just tracking the overcloud deployment failures https://bugs.launchpad.net/tripleo/+bug/1981478 na?11:57
dviroelchkumar|rover: hi o/ - no, happens also in tempest tests, when you look into controller's errors, lots of db connections appears11:59
chkumar|roverdviroel: ok, thanks I will track it here12:00
dviroelchkumar|rover: yeah, if we identify a specific issue on one of the these jobs, we can create another one12:08
dviroelbut so far, everything seems related to mysql/network issues12:08
frenzy_fridaychkumar|rover, I did not get your comment on https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44022 . RHOS 17 on RHEL8 is already added right?12:09
frenzy_fridayAre we running rhosp 17.1 on RHEL8 as well?12:10
chkumar|roverfrenzy_friday: sorry wanted to add the comment for rhos-17.112:11
chkumar|roverrlandy: in future is there any plan to run rhos-17.1 on rhel-8 also?12:11
* bhagyashris stepping out for bit12:26
*** amoralej is now known as amoralej|lunch12:31
jm1dviroel: thanks for your comments here :) https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/84994512:38
jm1chkumar|rover: may want to have a look here? edited as requested ;) https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/84994512:38
dviroeljm1: thanks for trying the other approach12:42
chkumar|roverjm1: thanks ;-)12:43
mariosinteresting discussion in tc meet about stream/stability fyi https://meetings.opendev.org/meetings/tc/2022/tc.2022-07-14-15.00.log.html#l-31 12:44
marios15:21:00 <gmann> #agree to make centos stream jobs testing in periodic way but keep it in testing runtime. monitor, debug, and report the failure to centos stream team12:44
jm1marios: this one has 3x +2, maybe you could give it a +w? 🥺 https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/84994512:48
mariosjm1: sure but i think it has enough votes you don't need more perhaps one of the folks that +2 can +w it ? 12:50
mariosdone12:50
mariosjm1: when you have time please review https://review.opendev.org/q/topic:oooci_mixed_rhel 12:51
jm1marios: getting to your +2/+w is like reaching boss level 😉 thanks for merging it, now we can work on porting tripleo to lastest aoc :)12:56
mariosjm1: hah why i am probably the easiest person to give +2 !12:56
marioslike here 12:56
mariosplease +w *this*12:56
* marios OK!12:56
marios;D12:57
mariosjm1: but seriously i expect payment this is not free, so please add to your reviews list 15:51 < marios> jm1: when you have time please review https://review.opendev.org/q/topic:oooci_mixed_rhel 12:57
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.12:57
mariosjm1: otherwise you'll be hearing from my lawyer12:57
mariosgood day sir12:57
jm1marios: yes, sir! 💂12:58
chkumar|roverrlandy: dasm|off train cs8 promoted13:11
rlandychkumar|rover: wow13:14
rlandychkumar|rover: did you skip?13:14
chkumar|roverrlandy: nothing13:14
rlandyoh - we used the internal13:14
rlandyno - we didn't https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config_environments/rdo/CentOS-8/train.yaml13:15
rlandywow13:15
chkumar|roverI just re-runned the failed jobs, fs035 0113:16
chkumar|roverit passed13:16
rlandyttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train&pipeline=openstack-periodic-integration-stable4&skip=013:16
rlandywow13:16
mariosthank you for review chkumar|rover++13:16
rlandynever passed in a while13:16
mariosreview*s*13:16
jm1marios: your patch series will take a while to review, it's massive 😅13:20
rlandychkumar|rover: wow13:20
jm1marios: working on in13:20
jm1*it13:20
rlandydviroel: ^^ https://review.rdoproject.org/r/c/testproject/+/44018 - check that out13:20
rlandypassing train fs00113:20
rlandyfrenzy_friday: hey - looking better on the promote jobs?13:20
frenzy_fridayrlandy, are we planning to run rhosp 17.1 on rhel 8 in the future?13:21
dviroelrlandy: chkumar|rover: this is impossible13:22
dviroellol13:22
*** amoralej|lunch is now known as amoralej13:22
chkumar|roverit's the mercy of CI13:24
dviroelchkumar|rover: the ci respects you13:25
rlandyfrenzy_friday: ack13:25
* rlandy bows to chkumar|rover 13:25
rlandydviroel: we don't have the magic :(13:25
rlandyfrenzy_friday: pls add both options13:25
* chkumar|rover bows back13:26
dviroelrlandy: no, we don't, we are lucky to have chkumar|rover with us13:26
frenzy_fridayack, rlandy, chkumar|rover  updated https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4402213:27
mariosjm1: yeah no worries and i understand a lot of it will be unkown but feel free to add questions there so it can be useful for you as learning (not about mypatch, but about our ci code/mechanisms more generally)13:27
mariosjm1: and you can always "just review the ansible" kind of thing maybe you can spot something to fix 13:28
mariosthanks13:28
rlandyfrenzy_friday: looks good - pls try testproject with that depends on13:29
rlandylet's see if we are missing anything else13:29
*** dasm|off is now known as dasm|ruck13:33
dasm|rucko/13:33
chkumar|roverdasm|ruck: \o good morning :-)13:33
dviroel\o13:33
dasm|ruckchkumar|rover: o/ anything important you're fighting with right now?13:34
chkumar|roverdasm|ruck: nothing looking at fs03513:35
dasm|ruckack13:35
chkumar|roverhttps://review.opendev.org/c/openstack/tripleo-quickstart/+/849988 test results 13:35
dasm|ruckchkumar|rover: cs8 wallaby looks like promoted too13:37
dasm|ruckneat13:37
chkumar|roverdasm|ruck: yes and train also13:37
chkumar|roverjust promoted right now13:37
dasm|ruck++13:37
chkumar|roverI think we can focus on master13:38
dasm|ruckk13:38
rlandychkumar|rover: rekicked the rhos-17 patch13:39
rlandyadded failed envD for 16.213:40
rlandydasm|ruck: ^^13:40
dasm|ruckrlandy: "failed envD for 16.2" what do you mean?13:41
dasm|ruckchkumar|rover: do you have cs9 master testproject running?13:41
chkumar|roverit just failed13:41
dasm|ruckchkumar|rover: if not, i'm gonna kick off one now for ebad96608af28084b7aee40526cf476013:41
chkumar|roverI have rechecked soniya tp13:41
dasm|ruckah13:42
chkumar|roverdasm|ruck: I think we can rekick a fresh tp13:42
dasm|ruckthis https://review.rdoproject.org/r/c/testproject/+/42692 ?13:42
dasm|ruckk, i'm gonna start a new one13:42
chkumar|roverhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-master13:42
chkumar|roverdasm|ruck: give me 5 mins13:42
chkumar|roverI think I can fix this job13:42
dasm|ruckchkumar|rover: https://review.rdoproject.org/r/c/testproject/+/4237413:43
dviroelchkumar|rover: dasm|ruck: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43689 - feel free to decide on merging or not, based on current results :P13:44
dasm|ruckok, updated the patch one more time -- it's gonna stop the test13:44
rlandydasm|ruck: https://code.engineering.redhat.com/gerrit/c/testproject/+/41995613:44
dasm|ruckrlandy: ack13:44
dasm|ruckdviroel: ack13:44
chkumar|roverbhagyashris: hello13:52
chkumar|roverbhagyashris: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-master is failing13:52
chkumar|roverbhagyashris: I think https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml#L269 is wrong13:53
chkumar|roverwe need to add tags here https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L724 just like this13:53
chkumar|roverDo you know about this one?13:53
chkumar|roverrlandy: dasm|ruck dviroel marios https://review.rdoproject.org/r/c/rdo-jobs/+/44024 please have a look when free, thanks!14:01
bhagyashrischkumar|rover, hey when i tested it was passing https://review.rdoproject.org/r/c/testproject/+/39932/33#message-57afa997db1286cece3def5fa6c6e5830aaa4d0514:02
dasm|ruckchkumar|rover: we have separate trunks for those?14:02
dasm|ruckyeah, seems like it14:03
bhagyashrischkumar|rover, so looks like it may be different issue otherwise it should should that time only 14:03
chkumar|roverbhagyashris: let me send a testproject to confirm that14:04
bhagyashrischkumar|rover, and sc010-ovn-provider is inheritated from sc010-standalone job and that job contains the tag here https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml#L21114:05
marioschkumar|rover: ok but do we realy need it? added a comment14:06
marioschkumar|rover: i mean it should have tags there via inheritance already? 14:06
chkumar|roverbhagyashris: I think you are correct14:06
chkumar|roverI missed that line14:06
chkumar|rovermarios: bhagyashris yes14:07
marioschkumar|rover: k14:07
dasm|ruckchkumar|rover: in the meantime, do you want me to chase a new hash for promition?14:10
chkumar|roverdasm|ruck: yes please14:11
dasm|ruckon it14:12
mariosrlandy: first green runs https://zuul.opendev.org/t/openstack/build/a3fbbdfba721457d8301e105164be2da (1 hr 54 mins 35 secs) & https://review.rdoproject.org/zuul/build/4205031ebc834dabbfb05757108143f5 (2 hrs 21 mins 3 secs) \o/ 14:12
mariosthat upstream is well within timeout but still gonna be our longer upstream job we have to be careful with files: or otherwise restrict it somehow14:13
mariostimes in brackets wrong way aroudn but you get the jist ;)14:13
mariosinteresting its like half an hour faster in rdo/vexx env 14:14
rlandymarios++ very nice14:15
chkumar|roverdasm|ruck: we need a seperate tp for this periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master14:16
chkumar|roverin downstream14:16
*** ysandeep|afk is now known as ysandeep14:16
ysandeepchkumar|rover, rlandy marios dviroel mtg time14:16
dasm|ruckchkumar|rover: started: https://code.engineering.redhat.com/gerrit/c/testproject/+/40811914:17
chkumar|roverdasm|ruck: cool14:17
jm1rcastillo, rlandy, chkumar|rover: quick! stop this merge!!! https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/84741914:25
jm1ysandeep, marios: ^14:25
rlandyjm1; abondon the patch14:25
jm1rlandy: thanks!14:25
rlandyjm1; and w-1 it if you don;t want it merged14:26
jm1rlandy: i dont have permissions14:26
dasm|ruckit's rcastillo's patch. rcastillo needs to -w it14:26
rlandyany core can14:26
jm1rlandy: please abandon, i am not core there14:27
rlandyjm1: done14:28
jm1rlandy: thank you very much. that was close 😅14:28
frenzy_fridayhttps://code.engineering.redhat.com/gerrit/c/testproject/+/420019 failed again with the same error14:33
frenzy_fridayMaybe the depends on didnt work14:33
mariosysandeep: reminder you're sharing screen #14:37
mariosysandeep: don't share you email secrets14:37
ysandeepno secrets gere, finding one draw.io from chandan14:38
marios;)14:38
dasm|ruckbtw centos breaking openstack: https://lists.openstack.org/pipermail/openstack-discuss/2022-July/029468.html cc rlandy 14:40
mariosjm1: sorry in call but i see it is abandoned now 14:51
jm1marios: no worry, rlandy abandoned it. i just pinged all because it had to be done quick14:52
rlandyfrenzy_friday: ERROR: Got error 404, probably endpoint https://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel9-osp17-1 is not available14:53
rlandyfrenzy_friday: pls check with release -delivery on that14:54
frenzy_fridayrlandy, ack15:05
rlandyfrenzy_friday: jon should be able to confirm that 15:05
frenzy_fridayrlandy, I see https://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel9-osp17 returns 404 as well. Am I using a wrong url?15:09
rlandyfrenzy_friday: doens't mean that the endpoint is working15:10
rlandycorrect15:10
rlandybut two diff things15:10
frenzy_fridayno, I mean foe 16.2, 17 - for all of them I am getting 40415:10
rlandychecking15:11
chkumar|roverdasm|ruck: https://bugs.launchpad.net/tripleo/+bug/198182915:12
chkumar|roverdasm|ruck: added it to cix, if you get more info about this bug please add15:12
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-rhos-17-component-network-promote-consistent-to-component-ci-testing/f05645b/job-output.txt15:12
rlandyfrenzy_friday: ^^ works there15:12
dasm|ruckchkumar|rover: ack15:12
frenzy_fridayrlandy, are you able to access https://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel8-osp17 from the browser?15:13
rlandyfrenzy_friday: try hit with with api15:14
chkumar|rovergthiemonge: hello, please have a look at this bug https://bugs.launchpad.net/tripleo/+bug/1981829 when around, I have added as a promotion blocker15:14
chkumar|roverthanks!15:14
chkumar|roverdasm|ruck: rlandy everything is on hackmd, see ya on Monday!15:15
rlandyfrenzy_friday: they are fixing it15:15
dasm|ruckchkumar|rover: thanks, see ya15:15
dviroelchkumar|rover: happy fridayo/15:16
* dviroel lunch15:16
frenzy_fridayyep15:16
rlandychkumar|rover: have a good weekend15:16
*** dviroel is now known as dviroel|lunch15:16
rlandyfrenzy_friday: pls try again15:16
frenzy_fridayI am waiting for the patch from amoralej to merge15:17
rlandyack - thanks15:20
*** ysandeep is now known as ysandeep|out15:28
* rlandy needs to pick up package - brb15:42
*** rlandy is now known as rlandy|brb15:42
marioshappy weekend tripleo-ci o/15:57
*** marios is now known as marios|out15:57
dasm|ruckmarios|out: o/16:04
*** rlandy|brb is now known as rlandy16:05
rlandydasm|ruck: you all set?16:05
dasm|ruckyup, chasing cs9 master16:05
dasm|ruckseems like we might have a valid kvm issue. but still verifying16:06
*** dviroel|lunch is now known as dviroel16:32
*** amoralej is now known as amoralej|off16:53
rlandylunch brb16:59
dasm|ruckk17:01
dasm|ruckwe have two issues with periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master17:10
dasm|ruck> octavia_tempest_plugin.tests.scenario.v2.test_pool.PoolScenarioTest)testtools.testresult17:10
dasm|ruck> (octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest)testtools.testresult17:10
dasm|ruckchecking notes if we've noticed  that already17:11
dasm|ruckhmm... seems that we can have something already: https://bugs.launchpad.net/tripleo/+bug/197954617:12
dviroeldasm|ruck: there is a CIX for this issue17:12
dviroeldasm|ruck: the KVM error one17:13
dviroel" show_loadbalancer provisioning_status updated to an invalid state of ERROR"17:13
dasm|rucklast time it passed 2022-07-05 and it's continuously failing since then17:13
dasm|ruckdviroel: i believe that might be either nova or kvm issue, not octavia17:13
dviroeldasm|ruck: yes, there are more info in the BZs17:14
dasm|ruckk17:14
dasm|ruck> 2022-07-15 16:34:00.420+0000: 191452: info : qemuMonitorJSONIOProcessLine:218 : QEMU_MONITOR_RECV_REPLY: mon=0x7fa7800852f0 reply={"id": "libvirt-48", "error": {"class": "GenericError", "desc": "SEV is not enabled in KVM"}}17:14
dasm|ruckhmm17:15
dasm|ruckrlandy: when did you enable security enabled VMs for sbaker?17:15
dasm|ruckdviroel: but you're talking about your 16.2 bz?17:16
dviroeldasm|ruck: it was logged against it17:17
dviroelcheck instance log on logs/undercloud/var/log/containers/libvirt/qemu/17:17
dviroel"KVM: entry failed, hardware error 0x8"17:17
dviroelsame errkr17:17
dasm|ruckack17:18
dasm|rucki'm checking this: https://sf.hosted.upshift.rdu2.redhat.com/logs/19/408119/11/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/ca7e7c3/logs/undercloud/var/log/containers/libvirt/virtqemud.log17:18
dasm|ruckyeah, i see the KVM error17:18
dasm|ruckwait a sec... it's cs9 sc10 kvm master 17:20
dasm|ruckit's not rhos at all, it just runs on internal infra.17:20
rlandydasm|ruck: I uploaded the image a few days ago17:38
dasm|ruckack17:41
dasm|rucki added this: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4402917:41
dasm|ruckrlandy: dviroel can you check? ^17:41
dviroeldasm|ruck: ack, there is no other way, just kvm missing?17:44
dasm|ruckdviroel: it's not the only one atm, but it's definitely gonna fail until KVM bug is fixed17:45
dasm|rucki'm checking other missing jobs17:45
dviroelack17:45
dasm|ruckthat's the test run for cs9 master: https://review.rdoproject.org/r/c/testproject/+/4237417:45
rlandylooking17:46
dasm|rucki suu cs9 main integration is running right now17:47
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master&skip=017:47
dasm|ruckyes17:47
rlandydasm|ruck: is that is the only job missing - ok17:48
rlandypls revert after promo17:48
rlandydone17:49
dasm|ruckk, thx17:49
dasm|ruckok... i'm shocked: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/e0f2f5fee5df461c8b5aa3f67706ecd6 is green18:06
dasm|ruckit's so inconclusive...18:06
dviroellol18:21
dasm|ruckit is not funny :P18:21
rlandynot as funny as train fs001 passing18:24
rlandydasm|ruck: ask chkumar|rover to run it  - he has the magic touch18:24
rlandydviroel and I are no good18:24
dviroelexactly18:24
dasm|ruckhmm... maybe it means noisy neighbors?18:25
dasm|ruckif it's working early India TZ?18:25
dasm|ruckidk, just speculating :)18:25
dviroelhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train18:25
dviroelsonyia was trying the same testproject in india tz18:25
dasm|ruckhmm18:26
dviroelworked only with chandan18:26
rlandyhe spins some charm18:40
dasm|ruckcs8 train & wallaby are nicely promoted. cs9 wallaby and master aren't so nice18:44
dviroelnice18:54
rlandyfrenzy_friday: you're good to go now 18:54
rlandy17.1 endpoint is functional18:54
rlandypromote jobs are running18:55
dasm|rucknice18:55
rlandydasm|ruck: we need to start tracking the OVB investigation18:55
rlandyneed to deal with the ceph and cinder issue on internal now18:56
rlandybut we need to start collating all this to stake steps one way or another18:56
dasm|rucki haven't started doing anything yet.18:56
dasm|ruckrlandy: i've never integrated anything. is it something similar to enabling "nodepool"?18:57
rlandyintegrated?19:11
dasm|ruckenabled new cloud19:22
jm1have a nice weekend, team 🍻19:43
dviroelo/19:43
dasm|ruckjm1: o/19:47
rlandydasm|ruck: you need help with ibm cloud?20:07
rlandydasm|ruck: master is not promoting20:08
* dviroel biab20:09
rlandywhat hash were you chasing20:09
*** dviroel is now known as dviroel|biab20:09
dasm|ruckrlandy: it's having its issue20:19
dasm|ruckrlandy: https://review.rdoproject.org/r/c/testproject/+/4237420:20
dasm|ruck55921623476163598a5dba0658e34aa320:20
dasm|ruckoh, pipeline must've finished20:20
dasm|ruckoh no. it just moved in different place20:21
dasm|ruckthat's one of issues with cs9 master: https://bugs.launchpad.net/tripleo/+bug/198182920:22
dasm|ruckcs9 wallaby also is not happy: https://review.rdoproject.org/r/c/testproject/+/4136720:24
* rlandy looks20:25
rlandydasm|ruck: you said it was the only missing job20:26
rlandywhat hash did you check for that???:20:26
rlandymaster ran again20:26
dasm|ruckno, i said it's one job which was definitely gonna fail, because of ongoin kvm problem20:26
rlandywhat was the previous hash?20:26
dasm|ruckprevious: ebad96608af28084b7aee40526cf476020:27
rlandyso no - then we can't skip until we know it's a good hash for everything else20:27
dasm|ruckhmm20:27
dasm|ruckeven if kvm won't work at all?20:27
dasm|rucki didn't want to skip it just for promo20:27
dasm|ruckmy intention was to disable it from promo for time being until there's gonna be fix20:28
rlandywell then we need to keep skipping and unskipping20:28
rlandyno20:28
dasm|rucki didn't want to go back and forth with enabling/disabling20:28
rlandywe need to keep that file clean20:28
rlandyand just skip and unskip as needed20:28
dasm|ruckit's broken by design. 20:29
rlandy<rlandy> dasm|ruck: is that is the only job missing - ok20:29
rlandy<rlandy> pls revert after promo20:29
dasm|ruckrlandy: and i responded "no", it was not the only one missing20:29
dasm|ruckrlandy: revert: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4369020:30
rlandymaybe the current hash will be better20:30
rlandyrunning now20:30
dasm|ruckcurrent hash is still computing. promo line is in progresss20:31
rlandyack20:32
rlandystill working20:32
dasm|ruckrlandy: this can be reverted: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4368920:38
rlandydone20:39
dasm|ruckk, thx20:39
rlandysorry - just need to get the ceph patches in for 1720:43
dasm|ruckno worries20:45
rlandydasm|ruck: ok- new master hash looks better21:06
rlandylet's see if we can get a pass on rerun of those failures21:06
dasm|ruckstill 5 broken, including ovn.21:06
dasm|rucki'm gonna rerun them in a sec, just about to wrap up one, currently running hash21:06
dasm|ruckhttps://review.rdoproject.org/zuul/status/change/42374,2721:07
dasm|ruckit passed, just waiting for results to being posted21:07
*** dviroel|biab is now known as dviroel21:08
dasm|ruckchkumar|rover: sc010 ovn across most (if not all) releases is being affected by KVM issue. I left note in the hackmd21:20
rlandyk - one less to rerun21:20
dasm|ruckrlandy: kvm is such a pain right now ^21:21
dasm|ruckor, more detailed: sc01021:21
rlandyhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-master&skip=021:21
rlandydasm|ruck'; ^^ new failure21:21
rlandywas working cleanly a few days ago21:21
dasm|rucknot new21:21
dasm|ruckond one21:22
rlandyreal issue21:22
dasm|ruck*old21:22
dasm|ruckthere is a bug on that21:22
dasm|ruckhttps://bugs.launchpad.net/tripleo/+bug/198182921:22
rlandyyeah 0 but the last 3 days?21:22
rlandyseems weird21:22
rlandywas clean for two weeks before that21:22
dasm|ruckunder the hood kvm is failing21:23
dasm|ruckoh wait. i'm not sure if i'm not mixing two things right now21:23
rlandyDetails: {'faultcode': 'Client', 'faultstring': 'Policy does not allow this request to be performed.', 'debuginfo': None}21:23
rlandy^^ different error21:23
dasm|ruckright. i was thinking about kvm one21:24
rlandyperiodic-tripleo-ci-centos-9-scenario010-kvm-standalone-masteropenstack/tripleo-cimasteropenstack-periodic-integration-main1 hr 41 mins 54 secs2022-07-15 17:17:58SUCCESS21:24
dasm|ruckbut it's submitted already21:24
rlandydasm|ruck: ^^ passes cleanly21:24
dasm|ruckthe one on psi does not21:25
dasm|ruckchkumar|rover asked me to kick off one there, and it failed21:25
rlandyack21:25
rlandydasm: but the error is not a poicy issue21:25
rlandylooks different to me21:25
dasm|ruckthis: https://bugs.launchpad.net/tripleo/+bug/198182921:25
dasm|ruckyou're right. both are different. one is for kvm, another for policy21:26
rlandyDetails: (TrafficOperationsScenarioTest:setUpClass) show_server status failed to update to ACTIVE within the required time 300. Current status of show_server: BUILD21:26
rlandydasm|ruck: pls log a bug for the policy one21:27
rlandyit's new21:27
rlandythe old bug is the active state one21:27
dasm|ruck21:22:22        dasm|ruck | https://bugs.launchpad.net/tripleo/+bug/1981829 21:27
dasm|ruckit's already logged21:27
dasm|ruckwhat do you mean by "active state one"?21:27
dasm|ruckoh, lovely21:30
dasm|ruck> periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-clients-rhos-17 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/ec7883b754ba499688896269fddd75aa : NODE_FAILURE in 0s21:30
rlandydasm|ruck: the previous hash is in rerun21:32
rlandylet's see if that one passes21:32
dasm|ruckk21:32
rlandyat tempest test stage 21:32
rlandyfor fs001 17 on 821:32
rlandydasm|ruck: I rekicked the jenkins job for 16.221:33
rlandymissing from promo21:33
rlandy17  on 9 - no new hash21:33
dasm|ruck17 on 9 is up to date21:33
rlandytrying to figure out the new ceph stuff21:33
dasm|rucki'm slowly disconnecting. mixing cases.21:34
dasm|rucki'm calling it a day.21:34
dviroelhave a great weekend team21:34
dviroelo/21:34
dasm|ruckchkumar|rover: i tried to clear components for all outstanding releases. Links are added in the hackmd21:34
dasm|ruckdviroel: take care o/21:34
dasm|ruckchkumar|rover: the only outliers are cs9 wallaby and master. it's mainly due to kvm job. i'm rekicking promo hash21:35
dasm|ruckchkumar|rover: i hope it's gonna be clearer by Morning21:36
* dasm|ruck => offline21:36
dasm|ruckhave a good one!21:36
dasm|ruckrlandy have a good weekend!21:37
*** dasm|ruck is now known as dasm|off21:37
*** dviroel is now known as dviroel|out21:37
rlandydasm|off: you too21:38
rcastilloleaving as well22:28
rcastillohave a good weekend22:28
rlandybye all22:50
« Thursday, 2022-07-14 Index Saturday, 2022-07-16 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!