| *** pojadhav|out is now known as pojadhav|ruck | 02:46 | |
| *** ysandeep|out is now known as ysandeep | 05:06 | |
| chandankumar | dpawlik: hello | 05:07 |
|---|---|---|
| chandankumar | dpawlik: I am running cs9 container multinode job here https://review.rdoproject.org/r/c/testproject/+/34926, during Remove the build SSH key from all nodes task, connection to node is timing out https://review.rdoproject.org/r/c/rdo-jobs/+/36460/1#message-78690e2d2839ac5d536d040b33125a191e52b097 please have a look, when free , thanks ! | 05:09 |
| ysandeep | chandankumar, thanks for workflowing c9 standalone in integration line, I am also adding standalone in all components and component promotion criteria so that components keep promoting in c9. | 05:11 |
| chandankumar | ysandeep: awesome :-) | 05:11 |
| chandankumar | ++ | 05:11 |
| chandankumar | sc1 to 4 is also in good shape | 05:12 |
| chandankumar | https://review.rdoproject.org/r/c/rdo-jobs/+/36464 | 05:12 |
| chandankumar | sorry not in good shape taking a look at failure https://review.rdoproject.org/r/c/testproject/+/36465 | 05:13 |
| chandankumar | frenzy_friday: \o morning | 05:13 |
| frenzy_friday | chandankumar, hey, good morning | 05:13 |
| frenzy_friday | c9 scenario001 and scenario004 failed finally | 05:13 |
| chandankumar | "Not found image: https://trunk.registry.rdoproject.org/v2/tripleomastercentos9/openstack-collectd/manifests/336a068c9e6198e5f6a04848dc579371", "msg": "Error running container image prepare: Not found image: https://trunk.registry.rdoproject.org/v2/tripleomastercentos9/openstack-collectd/manifests/336a068c9e6198e5f6a04848dc579371", "params": {}, "success": false} | 05:14 |
| chandankumar | 2021-10-27 23:40:33.263973 | fa163ecb-4646-d937-e815-00000000166b | TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:05:16.464123 | 114.27s | 05:14 |
| chandankumar | frenzy_friday: collectd container is not built which is expected | 05:15 |
| frenzy_friday | yep, Non-2xx: id b6c292f414b36a95a611d60d4628a022a756892a, status 404, reason Not Found | 05:15 |
| frenzy_friday | but scenario002 and 003 do not need collectd? | 05:16 |
| chandankumar | frenzy_friday: nope | 05:16 |
| frenzy_friday | ok | 05:16 |
| chandankumar | frenzy_friday: https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/ci/environments/scenario001-standalone.yaml#L15 | 05:16 |
| frenzy_friday | chandankumar, oh, ok. thanks | 05:17 |
| chandankumar | frenzy_friday: sc4 have different issue https://logserver.rdoproject.org/65/36465/1/check/periodic-tripleo-ci-centos-9-scenario004-standalone-master/1aadd0d/logs/undercloud/home/zuul-worker/standalone_deploy.log | 05:17 |
| chandankumar | FATAL | set tripleo-ansible ceph dashboard vars | undercloud | error={"changed": false, "msg": "No key/value pairs provided, at least one is required for this action to succeed"} | 05:17 |
| chandankumar | frenzy_friday: please open a bug for sc4 and cix it | 05:18 |
| ysandeep | chandankumar, frenzy_friday I see other scenarios standlone tasks in mini integration story.. imho.. those should go to "build out full integration line" story. If you agree we can move them under that story. | 05:18 |
| chandankumar | ysandeep: yes, we can close out the mini integration line, and move remaining tasks in full integration line | 05:19 |
| ysandeep | iirc.. mini integration line is .. container/image build and vanilla stanalone only.. | 05:19 |
| chandankumar | ysandeep: for image build, we need to add the jobs to the integration line | 05:19 |
| chandankumar | https://review.rdoproject.org/r/c/config/+/36234 | 05:19 |
| ysandeep | chandankumar, ack, I can propose a patch to add in the integration line | 05:21 |
| chandankumar | ysandeep: awesome ++ | 05:21 |
| * ysandeep updating board | 05:21 | |
| *** ykarel|away is now known as ykarel | 05:46 | |
| ysandeep | chandankumar, for component jobs we need first current-tripleo promotion for c9 master. | 05:52 |
| chandankumar | ysandeep: yes | 05:53 |
| chandankumar | ysandeep: may be we can add full tempest api, scenario and vanila stnadalone to promotion criteria | 05:53 |
| chandankumar | it will create current tripleo | 05:53 |
| chandankumar | But we merged the cs9 promoter config changes | 05:54 |
| chandankumar | let me take a look at promoter config | 05:54 |
| ysandeep | ack, sound good.. for now i think just first promotion is enough, I don't want c9 master hashes to be ahead of c8 master hashes... (last time in wallaby branching we hit some issues in wallaby integration which were hidden in master component lines) | 05:57 |
| chandankumar | ysandeep: something is not good there http://promoter.rdoproject.org/promoter_logs/centos9_master.log | 05:57 |
| chandankumar | http://promoter.rdoproject.org/config/CentOS-9/master.yaml | 05:58 |
| ysandeep | hmm.. promote-promoted-components should not be in criteria.. | 06:00 |
| chandankumar | yes, | 06:00 |
| ysandeep | chandankumar, let me put a patch to add standalone and remove promote-promoted-components | 06:01 |
| ysandeep | full tempest api, scenario - do we have patches to add job definition? otherwise I can put them | 06:02 |
| chandankumar | let me find out those patches | 06:02 |
| chandankumar | ysandeep: https://review.rdoproject.org/r/c/rdo-jobs/+/36363 and https://review.rdoproject.org/r/c/testproject/+/36364 | 06:02 |
| chandankumar | ysandeep: for api there is no review for job definition | 06:04 |
| chandankumar | https://review.rdoproject.org/r/c/testproject/+/35229/29/.zuul.yaml | 06:04 |
| ysandeep | chandankumar, ack i can add that | 06:04 |
| ysandeep | chandankumar, ^^ bhagyashri patch is in merge conflict.. let me add api in same patch as well | 06:07 |
| chandankumar | ok | 06:07 |
| akahat | chandankumar, why we are using centos8 jobas a parent in c9 jobs? | 06:09 |
| akahat | like it's in the above patch: https://review.rdoproject.org/r/c/rdo-jobs/+/36363/2/zuul.d/integration-jobs-cs9-master.yaml#14 | 06:10 |
| chandankumar | akahat: good question, for initial mini integration line, we have started using cs8 job as a parent and we have not created cs9 base job as upstream nodes were not available | 06:11 |
| chandankumar | akahat: we will revisit that once we have nodes available | 06:11 |
| chandankumar | and then replace the parent | 06:12 |
| chandankumar | akahat: if you want to replace those parent from upstream to downstream, Please go ahead and propose the patches | 06:13 |
| chandankumar | let me add a card | 06:13 |
| akahat | okay. thanks for clarification. | 06:13 |
| akahat | sure. | 06:13 |
| chandankumar | another to take into account | 06:13 |
| chandankumar | currently we are using virt customized nodepool image in rdo side it will change to dib some day | 06:14 |
| chandankumar | *another thing | 06:14 |
| ysandeep | chandankumar, fyi.. https://review.rdoproject.org/r/c/rdo-jobs/+/36363 updated for api/scenario.. rerunning testproject | 06:36 |
| ysandeep | frenzy_friday, ^^ fyi.. | 06:37 |
| ysandeep | chandankumar, frenzy_friday i have moved the tasks from mini-integration story to full-integration line story (those should belong there) | 06:42 |
| chandankumar | ysandeep: frenzy_friday marios I have added a tech debt card to remove all workaround in future https://issues.redhat.com/browse/TRIPLEOCI-705 | 06:46 |
| chandankumar | please do add there | 06:46 |
| chandankumar | if you find one | 06:46 |
| frenzy_friday | ack | 06:46 |
| marios | chandankumar: k | 06:47 |
| ysandeep | chandankumar: ok, thanks! | 06:50 |
| *** jpena|off is now known as jpena | 06:54 | |
| *** amoralej|off is now known as amoralej | 07:06 | |
| * frenzy_friday lunch | 07:37 | |
| ysandeep | marios: hello o/ I see you have already added periodic image build jobs definitions, thanks for that. Can I wire them up so that they can run in integration line, chandankumar and me are trying to wire things up before first current-tripleo promotion. | 07:38 |
| marios | ysandeep: sure go ahead | 07:39 |
| ysandeep | marios++ thanks! | 07:39 |
| chandankumar | pojadhav|ruck: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/285577/3#message-c1392ecdf0ba6790024347fc3bd56c0998d79a3c please have a look when free, thansk@ | 07:40 |
| ysandeep | marios, chandankumar fyi.. https://review.rdoproject.org/r/c/rdo-jobs/+/36479 | 07:43 |
| ysandeep | marios: one thing i want to confirm, do we not need overcloud-hardened-full , I only see overcloud-hardened-uefi-full in job def patch... I recall some discussion in the past abour replacing overcloud-hardened-full with overcloud-hardened-uefi-full.. If that's the case I think we are good. | 07:44 |
| marios | ysandeep: yeah the uefi-full is the new 'hardened' we wont need a standalone hardened job any more | 07:44 |
| ysandeep | marios: ack, thanks for confirming | 07:45 |
| pojadhav|ruck | chandankumar, yup working on comments.. thanks for review..!! | 07:46 |
| *** ysandeep is now known as ysandeep|lunch | 07:48 | |
| *** pojadhav is now known as pojadhav|ruck | 07:51 | |
| * pojadhav|ruck lunch | 07:51 | |
| * frenzy_friday back | 08:09 | |
| *** ykarel is now known as ykarel|lunch | 08:34 | |
| * pojadhav|ruck back | 08:34 | |
| *** ysandeep|lunch is now known as ysandeep | 08:45 | |
| ysandeep | periodic overcloud-full failed on running image sanity: https://logserver.rdoproject.org/54/32054/48/check/periodic-tripleo-centos-9-buildimage-overcloud-full-master/4bc12fb/job-output.txt | 09:17 |
| ysandeep | I see the testproject have image_sanity: false https://review.rdoproject.org/r/c/testproject/+/36267/7/.zuul.yaml#35 , do we want to add image_sanity: false in periodic job def as well for the timebeing? | 09:18 |
| marios | ysandeep: yes we don't know if it is an issue with the image or something to do with virt-tools/image sanity for cs9 yet | 09:19 |
| marios | ysandeep: this is why i'm chasing fs1/ovb to test if the image is ok | 09:19 |
| marios | ysandeep: https://review.opendev.org/c/openstack/tripleo-ci/+/808177/16..18/playbooks/tripleo-buildimages/pre9.yaml#8 | 09:19 |
| ysandeep | marios: ack | 09:22 |
| ysandeep | chandankumar, Update criteria file for c9: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36482 | 09:26 |
| *** ysandeep is now known as ysandeep|afk | 09:46 | |
| *** ykarel|lunch is now known as ykarel | 10:04 | |
| *** rlandy is now known as rlandy|ruck | 10:31 | |
| rlandy|ruck | marios: hey - want to talk about PSI request | 10:37 |
| marios | rlandy|ruck: sure | 10:37 |
| *** ysandeep|afk is now known as ysandeep | 10:37 | |
| rlandy|ruck | https://meet.google.com/dxc-nihv-tzx?pli=1&authuser=0 | 10:37 |
| rlandy|ruck | marios: ^^ | 10:37 |
| marios | rlandy|ruck: ack | 10:37 |
| chandankumar | marios: can you vote on this one https://review.opendev.org/c/openstack/releases/+/814827 as a tripleo liason (I think)? | 10:47 |
| chandankumar | Thanks! | 10:47 |
| chandankumar | When free | 10:47 |
| marios | chandankumar: has to be the ptl | 10:47 |
| *** dviroel|rover|afk is now known as dviroel|rover | 10:50 | |
| chandankumar | pojadhav|ruck: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/815777/3#message-ba7b649de89961767bc1461345fbb4b6f1b74ac6 | 10:51 |
| marios | chandankumar: sec in call will check | 10:52 |
| pojadhav|ruck | chandankumar, checking | 10:53 |
| pojadhav|ruck | chandankumar, will check with wallaby as well. thanks | 10:55 |
| chandankumar | dviroel|rover: \o. please have a look at this review https://review.opendev.org/c/openstack/tripleo-ci/+/815537 | 10:58 |
| dviroel|rover | chandankumar: hi | 10:59 |
| dviroel|rover | chandankumar: doesn't seems to be helping us | 10:59 |
| chandankumar | rlandy|ruck: dviroel|rover please have a look at this one https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/814641 when free, thansk! | 10:59 |
| marios | chandankumar: voted on there but it needs the ptl to vote for the ptl-approved label | 11:00 |
| chandankumar | ysandeep: please have a look at this https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/797452 | 11:01 |
| chandankumar | marios: aye, Thank you :-) | 11:02 |
| ysandeep | chandankumar, ack, will check in my next round of reviews | 11:03 |
| chandankumar | ysandeep: aye, thanks :-) | 11:05 |
| *** ykarel is now known as ykarel|afk | 11:15 | |
| rlandy|ruck | arxcruz: hey ... https://review.opendev.org/c/openstack/tripleo-ci/+/813629 | 11:19 |
| rlandy|ruck | is there a testproject patch with the update/upgrade jobs we can check before merging? | 11:19 |
| *** jpena is now known as jpena|lunch | 11:23 | |
| rlandy|ruck | jbadiapa: hi - we're looking at https://bugs.launchpad.net/tripleo/+bug/1949070 | 11:24 |
| rlandy|ruck | we merged https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/806284 yesterday | 11:24 |
| rlandy|ruck | is that related to your issue? | 11:24 |
| rlandy|ruck | should we revert that? | 11:24 |
| ysandeep | rlandy|ruck, i think jbadiapa reported that to track issue as suggested by chandankumar https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/797452 | 11:26 |
| jbadiapa | rlandy|ruck: hi, yes, ysandeep is right... that bug is to match the VMs with the flavors | 11:26 |
| rlandy|ruck | ysandeep: ack - I am just looking for the cause | 11:26 |
| rlandy|ruck | jbadiapa: ack - just looking for the patch that caused this | 11:26 |
| rlandy|ruck | if it was related to what we merged yesterday | 11:27 |
| jbadiapa | rlandy|ruk, it's been at least for a long time....even several releases | 11:27 |
| rlandy|ruck | jbadiapa: ok - we're looking to merge this patch https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/809487/ | 11:28 |
| rlandy|ruck | but we won;t do it if there is already an issue with the patch below | 11:28 |
| * ysandeep confused on how these two things are related | 11:29 | |
| jbadiapa | ysandeep, me too | 11:29 |
| jbadiapa | rlandy|ruck: maybe we are talking about https://bugs.launchpad.net/tripleo/+bug/1948001 | 11:30 |
| ysandeep | yeah, but even that's atleast 1 - 2 weeks older | 11:31 |
| ysandeep | not started yesterday | 11:32 |
| rlandy|ruck | jbadiapa: hi - we need to send that to hardware prov | 11:32 |
| ysandeep | rlandy|ruck, I am taking at look at that bug right now.. I will check with hardware prov if needed | 11:32 |
| jbadiapa | nothing regarding yesterday from my side. The bug I created today was due to the fact that chandankumar asked me on the patch submitted | 11:33 |
| jbadiapa | as ysandeep said before | 11:33 |
| rlandy|ruck | jbadiapa: k - just fyi - if you want a bug to be noticed, you can ping us with it - also triage with the current release - yoga-1 and mark it triaged in status and high/critical | 11:34 |
| rlandy|ruck | ysandeep: do we need to raise that as a CIX? | 11:34 |
| ysandeep | rlandy|ruck: no its only impact manual deployment and we have a workaround whole_disk=false | 11:34 |
| *** ysandeep is now known as ysandeep|mtg | 11:35 | |
| rlandy|ruck | ysandeep: ok - we need to doc that somewhere | 11:35 |
| rlandy|ruck | we need to out stevebaker on those as a FYI | 11:35 |
| jbadiapa | randly|ruck, ysandeep: the point is with the whole_disk_images=false is using the old way instead the new hardened.qcow2 image | 11:35 |
| rlandy|ruck | yeah | 11:38 |
| rlandy|ruck | so we need to raise these to stevebaker | 11:38 |
| jbadiapa | rlandy|ruck: ok, I will ping you next time. thx | 11:38 |
| jbadiapa | I haven't seen him on the irc or dont know his nickname | 11:39 |
| ysandeep|mtg | jbadiapa, irc: stevebaker , but he is in newzealand | 11:41 |
| jbadiapa | oh, I see, thx. | 11:42 |
| rlandy|ruck | jbadiapa: pining harold on #tripleo | 11:42 |
| rlandy|ruck | on where to raise these bugs | 11:42 |
| * ysandeep|mtg trying to reproduce your issue in my local env, I will let you know if I find anything. | 11:42 | |
| jbadiapa | rlandy|ruck: btw, I would like to talk to you regarding creating a periodic job for testing backup and restore with ironic integration. is it possible? | 11:43 |
| jbadiapa | ysandeep|mtg, thx | 11:43 |
| rlandy|ruck | jbadiapa: sure - I have some time open in my calendar tomorrow - feel free to book some time | 11:43 |
| jbadiapa | rlandy|ruck: great thx, I'll do | 11:44 |
| marios | sshnaidm: o/ around? do you have a few mins to help me debug some issues with ovb - trying to get the c9 fs1 job working there https://review.rdoproject.org/r/c/testproject/+/36267 - main problem/confusion now is that the bmc_console shows it booting c7 | 11:44 |
| marios | https://logserver.rdoproject.org/67/36267/7/check/tripleo-stream9-development-centos-9-ovb-3ctlr_1comp-featureset001-master/7d6afee/logs/bmc_7_59942-console.log (even though we merged that https://review.rdoproject.org/r/c/config/+/36457 ) | 11:45 |
| marios | sshnaidm: doesn't have to be now we can schedule a time that works for you | 11:45 |
| sshnaidm | marios, looking | 11:46 |
| frenzy_friday | Hey rlandy|ruck Can we merge this one https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/815823 ? This has a real bug with it which can be CIXed. So we can check the changes in the trello script as well | 11:46 |
| rlandy|ruck | frenzy_friday: cool - looking | 11:47 |
| frenzy_friday | rlandy|ruck, also, did I miss any tag in this bug https://bugs.launchpad.net/tripleo/+bug/1949033 ? It does not show up on cix trello board | 11:47 |
| rlandy|ruck | frenzy_friday: let me check the server | 11:48 |
| marios | sshnaidm: stack params look OK https://paste.opendev.org/raw/810248/ (undercloud_image is set to the genericcloud-9 which we added with https://review.rdoproject.org/r/c/config/+/36457 - the job is failing right now cos it cannot attach instance to provision OVB network (No server with a name or ID exists) | 11:48 |
| frenzy_friday | thanks | 11:48 |
| marios | sshnaidm: thanks | 11:48 |
| * pojadhav|ruck stepping out for dr appointment | 11:48 | |
| rlandy|ruck | frenzy_friday: it's there | 11:49 |
| rlandy|ruck | in new/triaged | 11:49 |
| rlandy|ruck | merging your change | 11:49 |
| *** ysandeep|mtg is now known as ysandeep | 11:51 | |
| sshnaidm | marios, weird, seems like it either detects undercloud ID wrong, or trying to connect to a different cloud where this ID doesn't exist | 11:53 |
| frenzy_friday | rlandy|ruck, oh ok, I see it now. So when https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/815823 merges we should hopefully see a health link in the desc | 11:54 |
| jbadiapa | rlandy|ruck: I've already sent the invitation. I aslo added jfrancoa (wants to know about it) | 11:54 |
| jbadiapa | thx | 11:54 |
| marios | sshnaidm: yeah... one thing i noticed is that it fails to retrieve the undercloud ID from the file (there is a trace there grep "Find out UUID of instance with metadata file") and fetches from the metadata service | 11:54 |
| marios | sshnaidm: compared to a 'good' c8 job where it gets it from the file | 11:54 |
| rlandy|ruck | jbadiapa: pls add ysandeep as well | 11:55 |
| sshnaidm | yeah, it's possible from both usually, we switched to file to not abuse metadata server | 11:55 |
| sshnaidm | marios, let's hold a job to see | 11:55 |
| rlandy|ruck | jbadiapa: and marios | 11:55 |
| jbadiapa | sure | 11:55 |
| marios | sshnaidm: ack ok i'll try get a hold on it and will ping again thanks for checking for now anyway | 11:55 |
| ysandeep | jbadiapa: my email: sandyada@redhat.com | 11:56 |
| jbadiapa | thx | 11:56 |
| sshnaidm | marios, you can add following to keep ovb stack too: https://paste.opendev.org/show/810250/ | 11:56 |
| marios | sshnaidm: k will do | 11:57 |
| jbadiapa | let me know if the time does not work for you | 11:57 |
| marios | amoralej: o/ hello are you the right person to ask about holding a node? or is there someone 'on duty' i can go ask instead ;) | 12:00 |
| amoralej | i can do it | 12:00 |
| amoralej | marios, which job and repo? | 12:01 |
| marios | amoralej: thank you sir... please there https://review.rdoproject.org/r/c/testproject/+/36267 for job tripleo-stream9-development-centos-9-ovb-3ctlr_1comp-featureset001-master | 12:01 |
| marios | amoralej: i am about to update it so it will re-run in a bit | 12:01 |
| amoralej | ok | 12:01 |
| marios | amoralej: thank you very much | 12:01 |
| *** jpena|lunch is now known as jpena | 12:22 | |
| *** amoralej is now known as amoralej|lunch | 12:28 | |
| chandankumar | dpawlik: pojadhav|ruck is there a bug report for these failure https://logserver.rdoproject.org/82/815782/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/6ea5950/job-output.txt | 12:39 |
| chandankumar | /bin/tripleo-mount-image: line 210: [: missing `]' | 12:39 |
| chandankumar | sorry dpawlik it was for dviroel|rover | 12:40 |
| dviroel|rover | chandankumar: don't think so, checking | 12:45 |
| dviroel|rover | chandankumar: no, we don't have a LP, i can see three failures here https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 | 12:51 |
| dviroel|rover | but doesn't always happen | 12:51 |
| dviroel|rover | fails around 45m | 12:52 |
| ysandeep | marios: Congratulation on new headphone :D | 13:01 |
| chandankumar | yes it's anew one | 13:01 |
| rlandy|ruck | dviroel|rover: scrum | 13:02 |
| dviroel|rover | ack | 13:02 |
| akahat | looks like new headphone do not have light. :) | 13:02 |
| sshnaidm | marios, as it looks - this job is not running on vexxhost | 13:02 |
| sshnaidm | marios, it runs on PSI.. can it be? | 13:03 |
| marios | https://logserver.rdoproject.org/67/36267/7/check/tripleo-stream9-development-centos-9-ovb-3ctlr_1comp-featureset001-master/7d6afee/zuul-info/inventory.yaml | 13:13 |
| marios | https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/defaults/main.yml sshnaidm | 13:16 |
| *** ykarel|afk is now known as ykarel | 13:19 | |
| rlandy|ruck | review list: | 13:19 |
| rlandy|ruck | https://review.rdoproject.org/r/c/rdo-jobs/+/36483 | 13:19 |
| rlandy|ruck | https://review.rdoproject.org/zuul/build/ef0801c8a7284270a7407328e8a9a041, https://review.rdoproject.org/zuul/build/687d8c1feebd427c8fdfcd556a5804a4 | 13:20 |
| *** amoralej|lunch is now known as amoralej | 13:20 | |
| frenzy_friday | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36484 - 002 003 in criteria | 13:21 |
| sshnaidm | marios, well, seems like it IS on vexxhost.. | 13:22 |
| akahat | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36477 | 13:25 |
| ykarel | sshnaidm, i think ^ happening as it's not in tripleo nodepool tenant | 13:29 |
| ykarel | Provider: vexxhost-nodepool-sf from job logs | 13:30 |
| ykarel | should be tripleo one to get creds working | 13:30 |
| sshnaidm | ykarel, so it's not our usual tenant? | 13:31 |
| ykarel | sshnaidm, yes for tripleo it's vexxhost-nodepool-tripleo | 13:31 |
| ykarel | may be can check other jobs | 13:31 |
| sshnaidm | ykarel, oh, right | 13:31 |
| sshnaidm | dpawlik, I think we found ^^ | 13:32 |
| rlandy|ruck | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36477/1/ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py | 13:35 |
| rlandy|ruck | marios: ^^ | 13:35 |
| marios | thanks | 13:35 |
| arxcruz | rlandy|ruck: https://review.opendev.org/c/openstack/tripleo-ci/+/813629 | 13:45 |
| ysandeep | chandankumar, full-tempest-api is rerunning.. lets wait for current run to finish before adjusting the criteria.. | 13:45 |
| chandankumar | ysandeep: proposing a patch to modify the criteria | 13:45 |
| chandankumar | ysandeep: ok | 13:46 |
| ysandeep | last 2 run failed on different tempest test.. I think it may pass | 13:46 |
| chandankumar | sshnaidm: on ovb patch, updated with depends on | 13:56 |
| chandankumar | nodeset | 13:56 |
| frenzy_friday | tripleo critical bugs script is adding health links if present on cix cards now \O/: https://trello.com/c/YeTRbuU5/2204-cixlp1949033tripleociproa-no-key-value-pairs-provided-at-least-one-is-required-for-this-action-to-succeed | 13:58 |
| marios | thank you amoralej | 13:58 |
| marios | amoralej: also rlandy pointed me to https://sf.hosted.upshift.rdu2.redhat.com/etherpad/p/zuul_admin_HOWTO so hopefully next time i can do it without bothering you (famous last words) | 13:59 |
| dviroel|rover | rlandy|ruck: ysandeep pojadhav|ruck arxcruz https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36471 | 14:00 |
| dviroel|rover | ^ fixind rdo promotions panel | 14:01 |
| ysandeep | dviroel|rover, ack | 14:01 |
| dviroel|rover | snapshot in the comments, thanks | 14:01 |
| ysandeep | thanks! checking.. I will vote soon | 14:01 |
| dviroel|rover | tks! | 14:02 |
| ysandeep | dviroel|rover, the grafana snapshot you shared is from production server or do you have a development environment ? | 14:06 |
| arxcruz | dviroel|rover: i was about to send similar patch | 14:08 |
| arxcruz | dviroel|rover: i think for upstream we need the docker-compose host variable | 14:08 |
| arxcruz | i know local it fails if we have that | 14:08 |
| akahat | rlandy|ruck, o/ re: sc10 jobs.. we are good with the all branch results, except ussuri check jobs are failing. | 14:09 |
| rlandy|ruck | akahat: yeah - let's touch base on that | 14:09 |
| rlandy|ruck | give me a few | 14:09 |
| akahat | okay | 14:10 |
| rlandy|ruck | dviroel|rover: thanks - will look in a bit | 14:11 |
| rlandy|ruck | just fighting missing laptop for new hire | 14:12 |
| rlandy|ruck | marios: ^^ fyi - will approve when done with this battle :) | 14:12 |
| marios | rlandy|ruck: ack thanks | 14:14 |
| dviroel|rover | arxcruz: hi o/ | 14:25 |
| arxcruz | dviroel|rover: oi sumido xD | 14:25 |
| dviroel|rover | arxcruz: hum, but doesn't make sense to enable host network and expose a port | 14:25 |
| dviroel|rover | arxcruz: all other containers don't use host networking too | 14:26 |
| arxcruz | dviroel|rover: we can try, if it fails, we know who to blame :P | 14:26 |
| * dviroel|rover runs | 14:26 | |
| dviroel|rover | ysandeep: my dev environment | 14:27 |
| arxcruz | ade_lee_: hi, what can i do for you regarding the collect_logs and your fips test | 14:28 |
| rlandy|ruck | ade_lee_: ^^ this is your man on collect logs issues | 14:28 |
| rlandy|ruck | pls post the failure log we looked at yesterday | 14:28 |
| rlandy|ruck | dviroel|rover: pojadhav|ruck: fyi https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/view/phase-1-pipelines/ | 14:42 |
| rlandy|ruck | phase 1 jobs have gone missing | 14:42 |
| rlandy|ruck | rdo guys are trying to restore them | 14:42 |
| pojadhav|ruck | rlandy|ruck, oops | 14:42 |
| rlandy|ruck | yeah right | 14:42 |
| dviroel|rover | ´･_･` | 14:42 |
| ade_lee_ | rlandy|ruck, arxcruz sorry in another conversation | 14:46 |
| arxcruz | np | 14:46 |
| ade_lee_ | rlandy|ruck, incidentally what we tried yesterday failed - it still looks like the env file was not added | 14:47 |
| rlandy|ruck | hmmm ... | 14:51 |
| *** ykarel is now known as ykarel|away | 14:51 | |
| rlandy|ruck | sshnaidm: hey ...to add a new file to env standalone deploy ... | 14:52 |
| rlandy|ruck | https://review.opendev.org/c/openstack/tripleo-quickstart/+/815750/3/config/general_config/featureset052.yml | 14:52 |
| rlandy|ruck | standalone_custom_env_files | 14:52 |
| rlandy|ruck | is the right var? | 14:52 |
| rlandy|ruck | is there another one we are using or overriding these days? | 14:52 |
| rlandy|ruck | ade_lee_: ^^ checking | 14:53 |
| ade_lee_ | rlandy|ruck, so maybe I need to override standalone_custom_env_files in my job -- maybe its not using featureset 52? | 14:58 |
| ade_lee_ | sshnaidm, ^^ | 14:58 |
| rlandy|ruck | it should be | 14:58 |
| rlandy|ruck | ade_lee_: where is the testproject run | 14:58 |
| * rlandy|ruck wants to check the variable | 14:58 | |
| rlandy|ruck | akahat: let's chat after DF call | 14:59 |
| ade_lee_ | I don't think we did one recently | 14:59 |
| sshnaidm | rlandy|ruck, ade_lee_ we are talking about standalone ipa job, right? | 15:00 |
| rlandy|ruck | ade_lee_: it's in the deploy file on your job - afaict | 15:01 |
| ade_lee_ | sshnaidm, for one .. but also scenario 2 .. | 15:01 |
| rlandy|ruck | http://pastebin.test.redhat.com/1004641 | 15:01 |
| ade_lee_ | rlandy|ruck, really -- looking? | 15:01 |
| rlandy|ruck | https://6ce576c3d9483acba2ed-75a25eb118ff9373239bb3443dc0d448.ssl.cf1.rackcdn.com/815750/3/check/tripleo-ci-centos-8-standalone/1a9255a/logs/undercloud/home/zuul/index.html | 15:01 |
| rlandy|ruck | FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/tripleo-deploy/tripleo-heat-installer-templates/environments/fips.yaml' | 15:02 |
| rlandy|ruck | None | 15:02 |
| rlandy|ruck | [Errno 2] No such file or directory: '/home/zuul/tripleo-deploy/tripleo-heat-installer-templates/environments/fips.yaml' | 15:02 |
| rlandy|ruck | ade_lee_: it tried to include it | 15:02 |
| rlandy|ruck | but didn't find the file | 15:02 |
| sshnaidm | ade_lee_, rlandy|ruck maybe here? https://github.com/openstack/tripleo-ci/blob/master/zuul.d/standalone-jobs.yaml#L852-L856 | 15:02 |
| ysandeep | chandankumar, full-tempest-api passed https://review.rdoproject.org/r/c/testproject/+/36364 | 15:03 |
| *** ysandeep is now known as ysandeep|mtg | 15:03 | |
| rlandy|ruck | sshnaidm: sorry - I think we are ok | 15:03 |
| rlandy|ruck | it did try include the added file | 15:03 |
| chandankumar | sshnaidm: ysandeep|mtg marios https://review.rdoproject.org/r/c/rdo-jobs/+/36486 | 15:03 |
| chandankumar | and https://softwarefactory-project.io/r/c/config/+/23007 | 15:03 |
| ade_lee_ | rlandy|ruck, I think I'm confused - let me look again | 15:05 |
| rlandy|ruck | ade_lee_: I checked the standalone run on that job | 15:05 |
| rlandy|ruck | -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/fips.yaml | 15:06 |
| chandankumar | ysandeep|mtg: ok, cool, hope this time promoter do cs9 promotion | 15:06 |
| chandankumar | ysandeep|mtg: if nodeset change broke anything let me know tomorrow | 15:07 |
| ade_lee_ | rlandy|ruck, I was looking at scenario4-standalone | 15:08 |
| *** pojadhav|ruck is now known as pojadhav|dinner | 15:08 | |
| * rlandy|ruck looks | 15:08 | |
| chandankumar | rlandy|ruck: fyi https://review.rdoproject.org/r/c/rdo-jobs/+/36486 | 15:08 |
| ade_lee_ | rlandy|ruck, yeah - so its not there as far as I can tell | 15:09 |
| rlandy|ruck | ade_lee_: https://review.opendev.org/c/openstack/tripleo-quickstart/+/815750 - there is no scneraio 04 on this patch | 15:10 |
| rlandy|ruck | what am I missing??? | 15:10 |
| ade_lee_ | rlandy|ruck, https://review.opendev.org/c/openstack/tripleo-ci/+/808215 | 15:10 |
| chandankumar | marios: is it ok if I include fs02 job in this also https://review.rdoproject.org/r/c/testproject/+/36267 ? | 15:10 |
| chandankumar | want to see how it behaves when the image gets built with in the job | 15:11 |
| ade_lee_ | rlandy|ruck, it helps if we're looking at the right patches together :) | 15:11 |
| ade_lee_ | rlandy|ruck, and yes - its in the standalone build here -- I think scenario4 is maybe using a different config file | 15:12 |
| rlandy|ruck | ade_lee_: https://github.com/openstack/tripleo-ci/blob/master/zuul.d/standalone-jobs.yaml#L454 | 15:13 |
| rlandy|ruck | gets overridden on scenario004 | 15:13 |
| marios | chandankumar: sure go ahead | 15:13 |
| ade_lee_ | yeah | 15:13 |
| rlandy|ruck | you'll need to add your file there | 15:13 |
| rlandy|ruck | https://github.com/openstack/tripleo-ci/blob/master/zuul.d/standalone-jobs.yaml#L432 | 15:13 |
| rlandy|ruck | scn003 is the same | 15:13 |
| ade_lee_ | ack - ok let me check the rest too | 15:14 |
| rlandy|ruck | ade_lee_: so vanilla change fs052 | 15:14 |
| rlandy|ruck | all the scenario - add in the job defs | 15:14 |
| arxcruz | rlandy|ruck: ade_lee_ so, which file is not being collected ? | 15:14 |
| ade_lee_ | arxcruz, so we're looking at this job .. https://review.opendev.org/c/openstack/tripleo-ci/+/808215 scenario 2 | 15:15 |
| ade_lee_ | https://zuul.opendev.org/t/openstack/build/ea87e0b364ab43c6b31ddc753bc32274 | 15:15 |
| rlandy|ruck | ade_lee_: vanilla standalone deploy on https://review.opendev.org/c/openstack/tripleo-ci/+/808215 has your change | 15:15 |
| rlandy|ruck | so yeah - multiple changes | 15:15 |
| ade_lee_ | arxcruz, consistently, for this job, log collection fails | 15:15 |
| ade_lee_ | rlandy|ruck, ack - its never easy :/ | 15:16 |
| * dviroel|rover lunch | 15:17 | |
| arxcruz | i see | 15:18 |
| ade_lee_ | arxcruz, which is weird because it isn't failing for any other job | 15:19 |
| arxcruz | failed: [undercloud] (item=cpuinfo) => { | 15:19 |
| ade_lee_ | arxcruz, and when we tried to do this in rdo - log collection succeeds | 15:20 |
| ade_lee_ | arxcruz, I think whatever you pasted got truncated | 15:20 |
| arxcruz | ade_lee_: https://83d0aab0d56e8ab1cd96-cf2e6d1d128a778849f5c48f17b8a34a.ssl.cf2.rackcdn.com/808215/46/check/tripleo-ci-centos-8-scenario002-standalone/ea87e0b/logs/quickstart_collect_logs.log | 15:20 |
| arxcruz | failed: [undercloud] (item=cpuinfo) | 15:20 |
| arxcruz | ade_lee_: rlandy|ruck there a bunch of <127.0.0.2> Failed to connect to the host via ssh: OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021 | 15:22 |
| ade_lee_ | arxcruz, that could make sense -- if I'd expect anything to fail under fips it would be doing an md5 or an ssh connection -- but why does it only fail on this job? | 15:24 |
| ade_lee_ | and only on upstream and not rdo | 15:24 |
| arxcruz | ade_lee_: maybe the node you're using ? | 15:24 |
| marios | chandankumar: NODE_FAILURE at https://review.rdoproject.org/r/c/testproject/+/36267 i am guessing the depends-on(s) needs to merge for the new label first ? | 15:24 |
| arxcruz | i saw in the patch that you're using multinode-fips | 15:25 |
| chandankumar | marios: depends on should work | 15:25 |
| chandankumar | marios: mhu is looking into that | 15:25 |
| ade_lee_ | arxcruz, right fips is enabled | 15:25 |
| marios | chandankumar: thanks switchign chans | 15:25 |
| ade_lee_ | arxcruz, but thats true for all the jobs in that patch (except the content provider ones) | 15:26 |
| arxcruz | ade_lee_: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_59b/808215/46/check/tripleo-ci-centos-8-scenario001-standalone/59b66e3/logs/index.html is also not collectiong the logs | 15:26 |
| arxcruz | but it passes | 15:26 |
| arxcruz | ade_lee_: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_59b/808215/46/check/tripleo-ci-centos-8-scenario001-standalone/59b66e3/logs/quickstart_collect_logs.log | 15:27 |
| arxcruz | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_59b/808215/46/check/tripleo-ci-centos-8-scenario001-standalone/59b66e3/logs/quickstart_collect_logs.log | 15:27 |
| ade_lee_ | arxcruz, scenario 3 is collecting the logs as is scenario 4 | 15:28 |
| marios | amoralej: thanks for help - i deleted the hold request please check make sure all good | 15:29 |
| ade_lee_ | arxcruz, where is the code that tells the log collection to do an ssh? | 15:31 |
| ade_lee_ | arxcruz, maybe we're in a situation where its failing because it can only do ssh with rsa and sha1 signatures for some reason -- that would fail under fips. we might be able to get it to use ecdsa keys instead as a test | 15:32 |
| rlandy|ruck | akahat: hey - sorry :) - still around? | 15:47 |
| *** ysandeep|mtg is now known as ysandeep | 15:49 | |
| * ysandeep shutdown sequence .. | 15:49 | |
| *** amoralej is now known as amoralej|off | 15:52 | |
| akahat | rlandy|ruck, yes.. i'm here | 15:53 |
| *** ysandeep is now known as ysandeep|out | 15:53 | |
| marios | thanks for chasing it sshnaidm and chandankumar looks like they have another patch up (https://softwarefactory-project.io/r/c/config/+/23008) ... will continue tomorrow | 15:55 |
| marios | sshnaidm: at least feels like forward progress and not just me staring at logs trying to understand wtf is going on with the bmc console... | 15:55 |
| rlandy|ruck | akahat: hey | 15:57 |
| arxcruz | ade_lee_: that's really weird, i'm not seeing any difference between the missing logs one and the normal one, they are even running in the same node | 15:57 |
| rlandy|ruck | akahat: https://meet.google.com/qzf-zogb-fhu?pli=1&authuser=0 | 15:57 |
| rlandy|ruck | let's talk scenario010 | 15:57 |
| ade_lee_ | arxcruz, yeah - definitely weird | 15:58 |
| arxcruz | rlandy|ruck: ade_lee_ i think i found the problem... | 15:59 |
| rlandy|ruck | frenzy_friday: hey - what's the status of https://bugs.launchpad.net/tripleo/+bug/1944416? | 16:01 |
| rlandy|ruck | do we still need it? | 16:01 |
| arxcruz | rlandy|ruck: ade_lee_ i think it's timeout, the two jobs that are missing logs, run the collect logs in 14:55 and 15:09 | 16:01 |
| arxcruz | while the others in less than that | 16:02 |
| rlandy|ruck | marios: can we close https://bugs.launchpad.net/tripleo/+bug/1943968 out? | 16:02 |
| arxcruz | it might be the time that collect logs have to publish it | 16:02 |
| ade_lee_ | arxcruz, huh, I wonder why its taking so long .. | 16:02 |
| ade_lee_ | arxcruz, can we increase the timeout? | 16:02 |
| arxcruz | well, im not sure because one passing is 14:27 | 16:02 |
| sshnaidm | marios, sure, it's infra problem, would impact all c9 jobs anyway | 16:02 |
| rlandy|ruck | akahat: pls ping when you are back or we can talk tomorrow | 16:03 |
| arxcruz | not sure if these 30 seconds is the key | 16:03 |
| chandankumar | rlandy|ruck: dviroel|rover if https://review.rdoproject.org/r/c/testproject/+/36480 passes then please get the depends on merged | 16:05 |
| chandankumar | to patch sc4 | 16:05 |
| * rlandy|ruck looks | 16:05 | |
| *** jpena is now known as jpena|off | 16:05 | |
| rlandy|ruck | https://review.rdoproject.org/r/c/rdo-jobs/+/36464 in merge conflict | 16:06 |
| chandankumar | rlandy|ruck: dviroel|rover and also keep an eye on this one https://review.rdoproject.org/r/c/testproject/+/36267 | 16:06 |
| rlandy|ruck | https://review.rdoproject.org/r/c/rdo-jobs/+/36464 has a -1 | 16:06 |
| rlandy|ruck | chandankumar: ^^ | 16:06 |
| rlandy|ruck | there are a bunch | 16:06 |
| rlandy|ruck | not in status to merge | 16:06 |
| marios | rlandy|ruck: yes done | 16:07 |
| rlandy|ruck | maybe https://review.opendev.org/c/openstack/tripleo-heat-templates/+/815782 | 16:07 |
| rlandy|ruck | marios: thanks | 16:07 |
| rlandy|ruck | watching the image build jobs | 16:08 |
| dviroel|rover | chandankumar: ok, i'll watch those | 16:09 |
| ade_lee_ | arxcruz, are you thinking this is a timeout for the whole job, or for that specific task? | 16:09 |
| akahat | rlandy|ruck, sorry.. i'm back | 16:09 |
| ade_lee_ | arxcruz, coz the job itself didn't time out | 16:10 |
| rlandy|ruck | akahat: https://meet.google.com/qzf-zogb-fhu?pli=1&authuser=0 | 16:10 |
| chandankumar | dviroel|rover: https://review.rdoproject.org/zuul/status#36480 and https://review.rdoproject.org/r/c/testproject/+/36267 this also | 16:11 |
| chandankumar | will catch up tomorrow morning | 16:11 |
| chandankumar | see ya! | 16:13 |
| dviroel|rover | o/ | 16:13 |
| rlandy|ruck | akahat: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-ussuri&job_name=periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-victoria&job_name=periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-master&job_name=periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-wallaby&job_name=periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-train | 16:13 |
| *** marios is now known as marios|out | 16:14 | |
| rlandy|ruck | akahat: scenario010 branch results | 16:15 |
| rlandy|ruck | all scnario010 jobs are passing in periodic | 16:16 |
| rlandy|ruck | akahat: https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario010-standalone | 16:17 |
| rlandy|ruck | akahat: https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario010-standalone&job_name=https%3A%2F%2Fzuul.opendev.org%2Ft%2Fopenstack%2Fbuilds%3Fjob_name%3Dtripleo-ci-centos-8-scenario010-standalone-wallaby&job_name=https%3A%2F%2Fzuul.opendev.org%2Ft%2Fopenstack%2Fbuilds%3Fjob_name%3Dtripleo-ci-centos-8-scenario010-standalone-victoria&job_name=https%3A%2F%2Fzuul.opendev.org%2Ft%2Fopenstack%2Fbuilds%3 | 16:17 |
| rlandy|ruck | Fjob_name%3Dtripleo-ci-centos-8-scenario010-ussuri&job_name=https%3A%2F%2Fzuul.opendev.org%2Ft%2Fopenstack%2Fbuilds%3Fjob_name%3Dtripleo-ci-centos-8-scenario010-standalone-train | 16:17 |
| rlandy|ruck | akahat: https://trello.com/c/tx4SwUQp/2109-cixlp1895248tripleociproa-master-loadbalancerscenariotesttestloadbalanceripv4crud-showloadbalancer-provisioningstatus-updated-to | 16:18 |
| rlandy|ruck | akahat: https://bugs.launchpad.net/tripleo/+bug/1944978 | 16:19 |
| rlandy|ruck | akahat: psi - only good until dec- working with infra - for another solution | 16:20 |
| rlandy|ruck | akahat: next step on check | 16:20 |
| *** pojadhav|dinner is now known as pojadhav|out | 16:43 | |
| dviroel|rover | rlandy|ruck: anything needing attention? | 16:59 |
| rlandy|ruck | dviroel|rover: not at the moment ... unless you want to look into the scenario010 CIX | 17:00 |
| rlandy|ruck | akahat cleared the periodics on that | 17:00 |
| rlandy|ruck | have a 15 min meeting | 17:00 |
| rlandy|ruck | will link afterwards | 17:00 |
| rlandy|ruck | ussuri should promote | 17:00 |
| rlandy|ruck | downstream is dead | 17:01 |
| rlandy|ruck | gate is ok | 17:01 |
| dviroel|rover | rip downstream | 17:01 |
| rlandy|ruck | dviroel|rover: you can go back to some yum config work | 17:01 |
| rlandy|ruck | we will pic up some upgrade stuff tomorro | 17:01 |
| rlandy|ruck | tomorrow | 17:01 |
| dviroel|rover | rlandy|ruck: yeah, thinking about that too | 17:02 |
| dviroel|rover | already chasing some promotions too | 17:02 |
| *** derekh is now known as derekh_afk | 17:04 | |
| akahat | https://review.rdoproject.org/r/q/topic:%22scenario010-branchful%22+(status:open%20OR%20status:merged) | 17:22 |
| akahat | rlandy|ruck, dviroel|rover ^^ | 17:22 |
| rlandy|ruck | dviroel|rover: k - yeah - so you are welcome to go back to yum config | 17:30 |
| rlandy|ruck | whatever works for you | 17:30 |
| rlandy|ruck | akahat: https://review.rdoproject.org/r/c/rdo-jobs/+/36490 - I see what you did there :) | 17:31 |
| rlandy|ruck | and I am totally fine with using tags | 17:32 |
| rlandy|ruck | just don't want to conflate the two | 17:32 |
| rlandy|ruck | can you put in one patch to add kvm | 17:32 |
| dviroel|rover | yeah, splitting in two will be better | 17:32 |
| rlandy|ruck | and then you are welcome to put in a second patch with tags | 17:32 |
| rlandy|ruck | and we'll put it up for vote | 17:32 |
| akahat | rlandy|ruck, dviroel|rover okay. | 17:33 |
| rlandy|ruck | akahat: thanks - just want to separate the scenario010 specific stuff | 17:33 |
| rlandy|ruck | because we are sharing it with nate's team | 17:33 |
| rlandy|ruck | don't want to confuse the issue | 17:33 |
| rlandy|ruck | thank you | 17:33 |
| akahat | rlandy|ruck, dviroel|rover https://review.rdoproject.org/r/c/rdo-jobs/+/36492 :) | 17:39 |
| akahat | I'll update the old one. | 17:39 |
| dviroel|rover | tks | 17:41 |
| rlandy|ruck | akahat: perfect - thanks | 17:45 |
| rlandy|ruck | merging | 17:46 |
| dviroel|rover | rlandy|ruck: gah, seems that everything is failing due to 'tripleo-mount-image' | 17:47 |
| dviroel|rover | everything == ovb* | 17:48 |
| rlandy|ruck | dviroel|rover: ah - looking | 17:48 |
| rlandy|ruck | dviroel|rover: k - let's revert that | 17:49 |
| dviroel|rover | rlandy|ruck: need a bug for it? | 17:50 |
| *** sshnaidm is now known as sshnaidm|afk | 17:50 | |
| rlandy|ruck | dviroel|rover: ack | 17:50 |
| rlandy|ruck | dviroel|rover: this last one I assume : https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/809487 | 17:51 |
| dviroel|rover | yes | 17:52 |
| rlandy|ruck | k - revert with bug | 17:52 |
| rlandy|ruck | bug does not need a ton of details - just the failures pls | 17:52 |
| rlandy|ruck | thank you | 17:52 |
| rlandy|ruck | weird sandeep tested that - but ok | 17:53 |
| dviroel|rover | ok | 17:54 |
| rlandy|ruck | ping me with review and I'll w+ it | 17:56 |
| rlandy|ruck | can get the fix later | 17:56 |
| rlandy|ruck | let's just get the failure out the queue | 17:56 |
| rlandy|ruck | akahat: will merge https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36491 when wallaby has run once with that job | 17:59 |
| rlandy|ruck | thanks | 17:59 |
| akahat | rlandy|ruck, okay.. adding it to the testproject. | 18:01 |
| rlandy|ruck | great | 18:01 |
| rlandy|ruck | dviroel|rover: I assume he may be missing a release switch on that | 18:04 |
| rlandy|ruck | dviroel|rover: do you have a testproject on the revert? | 18:04 |
| rlandy|ruck | if not I will add one | 18:04 |
| dviroel|rover | rlandy|ruck: no, i don't have | 18:05 |
| dviroel|rover | rlandy|ruck: yes, seems to pass only on master | 18:05 |
| rlandy|ruck | dviroel|rover: probably on wallaby as well | 18:05 |
| dviroel|rover | ack | 18:05 |
| rlandy|ruck | dviroel|rover: adding testproject | 18:05 |
| rlandy|ruck | will send it back to stevebaker | 18:05 |
| dviroel|rover | rlandy|ruck: also failing here: https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 but with a different error | 18:09 |
| dviroel|rover | rlandy|ruck: chandan pointed this error earlier "/bin/tripleo-mount-image: line 210: [: missing `]'" | 18:10 |
| rlandy|ruck | dviroel|rover - there were two patches | 18:10 |
| rlandy|ruck | let's see how many we need to get rid of | 18:11 |
| rlandy|ruck | https://meet.google.com/uzn-wzcf-bsm?pli=1&authuser=0 | 18:11 |
| rlandy|ruck | dviroel|rover: ^^ | 18:11 |
| dviroel|rover | rlandy|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/806284 | 18:12 |
| -opendevstatus- NOTICE: mirror.bhs1.ovh.opendev.org filled its disk around 17:25 UTC. We have corrected this issue around 18:25 UTC and jobs that failed due to this mirror can be rechecked. | 18:43 | |
| rlandy|ruck | akahat++ nice email | 18:47 |
| rlandy|ruck | dviroel|rover: I think one revert is ok | 19:03 |
| * dviroel|rover looks | 19:05 | |
| dviroel|rover | rlandy|ruck: unexpected for me, cause of the error that we have "/bin/sh: line 1: type: tripleo-mount-image: not found" | 19:09 |
| dviroel|rover | rlandy|ruck: that was added in the first patch | 19:09 |
| dviroel|rover | rlandy|ruck: ah, search for "tripleo-mount-image\n/bin/sh" | 19:12 |
| dviroel|rover | in the running job | 19:12 |
| dviroel|rover | the error is there, didn't fail yet | 19:13 |
| rlandy|ruck | let's see | 19:14 |
| rlandy|ruck | will ping steve when he is in | 19:15 |
| rlandy|ruck | hmmm - overcloud image broke | 19:59 |
| dviroel|rover | (╯°□°)╯︵ ┻━┻ | 20:00 |
| dviroel|rover | train? | 20:02 |
| rlandy|ruck | rerunning to check | 20:02 |
| rlandy|ruck | 404 for http://mirror.centos.org/centos/8-stream/HighAvailability/x86_64/os/Packages/resource-agents-4.9.0-5.el8.x86_64.rpm (IP: 67.222.128.245) | 20:02 |
| rlandy|ruck | 2021-10-28 18:35:55.823 | [FAILED] resource-agents-4.9.0-5.el8.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success | 20:03 |
| rlandy|ruck | 2021-10-28 18:35:55.837 | Error: Error downloading packages: | 20:03 |
| rlandy|ruck | 2021-10-28 18:35:55.837 | Cannot download Packages/resource-agents-4.9.0-5.el8.x86_64.rpm: All mirrors were tried | 20:03 |
| rlandy|ruck | could be random or repeatable trying | 20:03 |
| dviroel|rover | yep | 20:04 |
| rlandy|ruck | dviroel|rover: ha ... https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 | 20:44 |
| rlandy|ruck | fs001 working on multiple patches now | 20:45 |
| dviroel|rover | rlandy|ruck: it is intermittent | 20:47 |
| rlandy|ruck | ok | 20:47 |
| dviroel|rover | rlandy|ruck: search for '815782,2' | 20:47 |
| rlandy|ruck | let's merge the revert | 20:47 |
| rlandy|ruck | and we can revert revert again | 20:47 |
| dviroel|rover | yep | 20:48 |
| dviroel|rover | might have another variable that triggers the issue | 20:48 |
| rlandy|ruck | https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/815881 | 20:49 |
| rlandy|ruck | dviroel|rover: ^^ pls vote anyways | 20:49 |
| rlandy|ruck | I don;t think we need the second revert | 20:49 |
| dviroel|rover | voted in both | 20:50 |
| dviroel|rover | i think that we'll need, lets wait for more results | 20:50 |
| dviroel|rover | with 1 revert, the error is still there, but the job didn't fail :P | 20:51 |
| rlandy|ruck | the first one for sure we need | 20:51 |
| rlandy|ruck | so let's start there | 20:51 |
| dviroel|rover | yep | 20:51 |
| rlandy|ruck | https://review.rdoproject.org/zuul/stream/cccd695750654705857ea900e5463a7d?logfile=console.log | 20:51 |
| rlandy|ruck | think train build image is doing better now | 20:51 |
| rlandy|ruck | hmmm | 21:00 |
| rlandy|ruck | idk | 21:00 |
| rlandy|ruck | not sure abut either atm | 21:00 |
| rlandy|ruck | ussuri is fine | 21:00 |
| dviroel|rover | updated the bug title with the real issue: https://bugs.launchpad.net/tripleo/+bug/1949107 | 21:15 |
| dviroel|rover | in the code, he is testing if 'tripleo-mount-image' exists, so he is ok not testing agains different branches | 21:17 |
| dviroel|rover | but it is failing in the 'mount' | 21:17 |
| dviroel|rover | https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/806284/18/roles/modify-image/tasks/manual.yml#53 | 21:17 |
| dviroel|rover | going afk | 21:33 |
| *** dviroel|rover is now known as dviroel|rover|afk | 21:33 | |
| rlandy|ruck | ok | 21:38 |
| rlandy|ruck | thanks | 21:38 |
| rlandy|ruck | waiting for victoria tests to complete | 21:40 |
| *** rlandy|ruck is now known as rlandy|ruck|afk | 22:21 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!