| *** jmasud has joined #oooq | 00:15 | |
| *** rlandy|bbl is now known as rlandy | 01:45 | |
| *** jfrancoa has quit IRC | 01:52 | |
| *** radez has quit IRC | 01:52 | |
| *** amoralej|off has quit IRC | 01:52 | |
| *** Tengu has quit IRC | 01:52 | |
| *** tristanC has quit IRC | 01:52 | |
| *** pojadhav|away has quit IRC | 01:52 | |
| *** chandankumar has quit IRC | 01:52 | |
| *** radez has joined #oooq | 01:54 | |
| *** jfrancoa has joined #oooq | 01:54 | |
| *** amoralej|off has joined #oooq | 01:54 | |
| *** Tengu has joined #oooq | 01:54 | |
| *** tristanC has joined #oooq | 01:54 | |
| *** pojadhav|away has joined #oooq | 01:54 | |
| *** chandankumar has joined #oooq | 01:54 | |
| *** jmasud has quit IRC | 01:59 | |
| *** pojadhav|away is now known as pojadhav|rover | 02:07 | |
| *** apetrich has quit IRC | 02:10 | |
| *** jmasud has joined #oooq | 04:10 | |
| *** ykarel has joined #oooq | 04:10 | |
| *** ratailor has joined #oooq | 04:32 | |
| *** udesale has joined #oooq | 04:40 | |
| *** saneax has joined #oooq | 05:06 | |
| *** marios has joined #oooq | 05:38 | |
| *** udesale has quit IRC | 06:12 | |
| pojadhav|rover | chandankumar, 0/ | 06:12 |
|---|---|---|
| *** slaweq has joined #oooq | 06:15 | |
| *** ysandeep|away is now known as ysandeep | 06:28 | |
| *** jpodivin has joined #oooq | 06:42 | |
| *** jpena|off is now known as jpena | 07:01 | |
| akahat | pojadhav|rover, o/ | 07:01 |
| akahat | pojadhav|rover, https://bugs.launchpad.net/tripleo/+bug/1927314 | 07:02 |
| openstack | Launchpad bug 1927314 in tripleo "standalone_hostname is undefined" [High,Triaged] | 07:02 |
| pojadhav|rover | akahat, ack | 07:02 |
| *** amoralej|off is now known as amoralej | 07:11 | |
| chandankumar | pojadhav|rover: o/ | 07:12 |
| pojadhav|rover | chandankumar, I have seen all wallaby jobs got failed https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable1 but ysandeep helped me to find out issue and its related bug and patch.. | 07:15 |
| pojadhav|rover | akahat, fyi https://review.opendev.org/c/openstack/ansible-role-tripleo-modify-image/+/789999/ | 07:16 |
| pojadhav|rover | akahat, fyi https://bugs.launchpad.net/tripleo/+bug/1927302 | 07:16 |
| openstack | Launchpad bug 1927302 in tripleo "rdo jobs broken with stat /etc/yum/vars: no such file or directory"" [Critical,Triaged] | 07:16 |
| chandankumar | pojadhav|rover: sorry which bug or patch? | 07:16 |
| *** chem has joined #oooq | 07:16 | |
| pojadhav|rover | chandankumar, https://review.opendev.org/c/openstack/ansible-role-tripleo-modify-image/+/789999/ and https://bugs.launchpad.net/tripleo/+bug/1927302 | 07:16 |
| *** jmasud has quit IRC | 07:28 | |
| *** apetrich has joined #oooq | 07:34 | |
| *** tosky has joined #oooq | 07:47 | |
| *** derekh has joined #oooq | 08:00 | |
| *** ykarel is now known as ykarel|lunch | 08:05 | |
| jlarriba | weshay|ruck I am having issues with the RDO build in my change https://review.opendev.org/c/openstack/tripleo-ansible/+/789855 that do not seem related to my changes | 08:29 |
| jlarriba | apparently, tempest is failing as it tries to delete a network while it still has ports | 08:30 |
| jlarriba | there is some known issue ongoing with RDO CI? | 08:30 |
| jlarriba | pojadhav|rover ^^ | 08:31 |
| pojadhav|rover | jlarriba, yes this known issue.. https://bugs.launchpad.net/tripleo/+bug/1927302 | 08:43 |
| openstack | Launchpad bug 1927302 in tripleo "rdo jobs broken with stat /etc/yum/vars: no such file or directory"" [Critical,Triaged] | 08:43 |
| *** ysandeep is now known as ysandeep|lunch | 08:47 | |
| *** pojadhav|rover is now known as pojadhav|lunch | 08:48 | |
| jlarriba | pojadhav|lunch, ack, thanks! | 08:49 |
| *** holser has joined #oooq | 09:21 | |
| *** ysandeep|lunch is now known as ysandeep | 09:39 | |
| arxcruz | kopecmartin: around ? | 09:43 |
| kopecmartin | arxcruz: yes | 09:43 |
| *** pojadhav|lunch is now known as pojadhav|rover | 09:44 | |
| arxcruz | kopecmartin: regarding https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/784885 | 09:44 |
| arxcruz | i know you're working on rpm build for train | 09:44 |
| arxcruz | do you have any update? need help on something ? | 09:45 |
| *** matbu has quit IRC | 09:47 | |
| kopecmartin | arxcruz: well, the review is ready https://review.rdoproject.org/r/c/rdoinfo/+/33420 however I couldn't verify the bump on centos-7 jobs because they are broken .. | 09:50 |
| kopecmartin | if we can verify in other way that it will work on centos-7 , we can merge the review right away | 09:51 |
| arxcruz | pojadhav|rover: are the centos-7 train jobs failing ? | 09:54 |
| *** matbu has joined #oooq | 09:58 | |
| *** ykarel|lunch is now known as ykarel | 10:02 | |
| chandankumar | kopecmartin: if the tempest package builds successfully on c7 we can merge it | 10:18 |
| kopecmartin | chandankumar: rdoinfo-DLRN-check passed, not sure if it's on centos7 | 10:18 |
| pojadhav|rover | arxcruz, after 1 May CentOS-7 train jobs not run in integration line. https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable4-centos7 | 10:19 |
| arxcruz | pojadhav|rover: i did not get it, sorry, can you explain? | 10:20 |
| pojadhav|rover | arxcruz, I mean to say, I am not seeing CentOS-7 train jobs are not running recently in integration line.. | 10:21 |
| pojadhav|rover | arxcruz, there is issue in some ovb jobs and standalone upgrade job.. rest jobs are green as i can see here https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable4-centos7 | 10:25 |
| marios | zbr: o/ can you please check comments there when you have some time https://review.opendev.org/c/openstack/tripleo-repos/+/789535/4/tox.ini#17 | 10:26 |
| pojadhav|rover | arxcruz, yes those are running they run over weekend and wed only. | 10:28 |
| arxcruz | marios: zbr is on pto until may 12 | 10:29 |
| arxcruz | chandankumar: what do you think ? | 10:29 |
| marios | ah k thanks arxcruz | 10:29 |
| arxcruz | marios: he should put an off on his nickname though | 10:30 |
| chandankumar | arxcruz: sorry you mean about C7 train? | 10:30 |
| arxcruz | chandankumar: yes, shall we bump and try it on os_tempest ? | 10:30 |
| chandankumar | kopecmartin: yes rdoinfo-DLRN-check for c7 | 10:31 |
| chandankumar | kopecmartin: let's rebase all the patches and merge it | 10:32 |
| kopecmartin | chandankumar: ok | 10:33 |
| chandankumar | arxcruz: for os_tempest side, on c7 side, we can pin os_tempest here https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base.yaml#L31 but we need to create a seperate required projects here with https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base.yaml#L39 | 10:34 |
| chandankumar | so that it will not break stein and queens os_Tempest jobs | 10:35 |
| arxcruz | chandankumar: i have a dnm removing that | 10:35 |
| arxcruz | let me check the results | 10:35 |
| arxcruz | chandankumar: https://review.opendev.org/c/openstack/tripleo-ci/+/789276 | 10:37 |
| arxcruz | chandankumar: unfortunately there is no c7 job lol | 10:37 |
| chandankumar | arxcruz: we can use testproject | 10:37 |
| arxcruz | chandankumar: yup | 10:38 |
| chandankumar | ysandeep: I am proposing a patch to fix downstream packaging work | 10:48 |
| ysandeep | chandankumar, ack o/ i think you already saw.. we are hiting this failure https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/3e785cf6b98d459ab9988dff74d56553 | 10:50 |
| ysandeep | https://sf.hosted.upshift.rdu2.redhat.com/logs/43/211643/40/check/periodic-tripleo-ci-rhel-8-standalone-rhos-16.2/3e785cf/job-output.txt | 10:50 |
| ysandeep | chandankumar, you can use this testproject for testing if you wish.. https://code.engineering.redhat.com/gerrit/211643 | 10:52 |
| chandankumar | yes on it | 10:53 |
| ysandeep | chandankumar++ thanks! | 10:53 |
| kopecmartin | chandankumar: done, they need to be force merged (one needs the other .. classic when we do this magic with reverts ..) https://review.rdoproject.org/r/c/rdoinfo/+/33420 https://review.rdoproject.org/r/c/openstack/tempest-distgit/+/33468 | 11:01 |
| *** ratailor_ has joined #oooq | 11:14 | |
| *** ratailor has quit IRC | 11:16 | |
| chandankumar | pojadhav|rover: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/789552 commented back | 11:17 |
| pojadhav|rover | chandankumar, ack | 11:19 |
| chandankumar | arxcruz: tempest patches approved | 11:21 |
| pojadhav|rover | chandankumar, pls review : https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/789552 | 11:21 |
| *** jpena is now known as jpena|lunch | 11:32 | |
| akahat | chandankumar, this is ready to merge: https://review.rdoproject.org/r/c/rdo-jobs/+/33519 | 11:38 |
| akahat | chandankumar, wait.. just saw some comments from marios | 11:39 |
| akahat | :) | 11:39 |
| marios | akahat: o/ hey feel free to ignore i added +2 not blockers | 11:40 |
| marios | akahat: if you decide to update i will revote | 11:40 |
| *** rlandy has joined #oooq | 11:46 | |
| akahat | marios, chandankumar please review i've updated it https://review.rdoproject.org/r/c/rdo-jobs/+/33519 | 11:49 |
| marios | ack akahat | 11:50 |
| * pojadhav|rover brb in sometime | 11:50 | |
| *** jfrancoa has quit IRC | 11:52 | |
| *** dviroel is now known as dviroel|nho | 12:00 | |
| *** jfrancoa has joined #oooq | 12:00 | |
| marios | rlandy: weshay|ruck: o/ call? | 12:01 |
| *** amoralej is now known as amoralej|lunch | 12:08 | |
| *** derekh has quit IRC | 12:17 | |
| chandankumar | ysandeep: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/3ed5384b6d6d4e0995e7cf0d5f4c2dc6?logfile=console.log it is working now | 12:21 |
| ysandeep | chandankumar++ thank you! | 12:21 |
| chandankumar | ysandeep: https://review.opendev.org/q/topic:%22downstream_packaging%22+(status:open%20OR%20status:merged) | 12:22 |
| *** jpena|lunch is now known as jpena | 12:26 | |
| *** ykarel_ has joined #oooq | 12:30 | |
| chandankumar | weshay|ruck: rlandy https://review.opendev.org/q/topic:%22downstream_packaging%22+(status:open%20OR%20status:merged) when free, thansk! | 12:31 |
| *** ykarel has quit IRC | 12:32 | |
| *** marios is now known as marios|call | 12:36 | |
| *** ratailor_ has quit IRC | 12:47 | |
| *** marios|call is now known as marios | 12:49 | |
| rlandy | weshay|ruck: ysandeep: bhagyashris is out now - per email... who is running scrum? | 12:51 |
| ysandeep | rlandy, i can o/ | 12:51 |
| rlandy | thanks | 12:54 |
| slaweq | chandankumar: hi, can You help me find in logs why job https://review.rdoproject.org/zuul/build/4cb7d57e6bc94574bedc12bad484339e/logs failed? | 12:56 |
| slaweq | I just see "Playbook run of multinode-standalone.yml failed" | 12:56 |
| slaweq | but I have no idea where to find logs from that multinode-standalone.yml playbook execution and what error it was | 12:57 |
| chandankumar | slaweq: checking | 12:57 |
| slaweq | chandankumar: thx | 12:57 |
| weshay|ruck | slaweq, chandankumar looks like it timed out | 12:58 |
| weshay|ruck | chandankumar, slaweq note.. the Timeout file https://logserver.rdoproject.org/57/32157/6/check/periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-master/4cb7d57/logs/ | 12:58 |
| weshay|ruck | going to have to get on the system | 12:58 |
| weshay|ruck | timed out while working w/ the octavia image | 12:59 |
| rlandy | ysandeep: pls review https://code.engineering.redhat.com/gerrit/#/c/240044/ | 12:59 |
| slaweq | weshay|ruck: but that job has got timeout set to 22000 seconds: https://review.rdoproject.org/r/c/testproject/+/32157/6/zuul.yaml#7 | 13:00 |
| slaweq | so why it was stopped after 2 hours? | 13:00 |
| rlandy | need ed to fix errors from puppet-openstack-integration | 13:00 |
| weshay|ruck | slaweq, there is also a tripleo timeout on the deploy | 13:00 |
| weshay|ruck | slaweq, 2021-05-05 13:31:48.558003 | primary | "DEPLOY_TIMEOUT_ARG": 90 | 13:01 |
| weshay|ruck | slaweq, see https://logserver.rdoproject.org/57/32157/6/check/periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-master/4cb7d57/job-output.txt | 13:01 |
| *** amoralej|lunch is now known as amoralej | 13:01 | |
| ysandeep | weshay|ruck, chandankumar rlandy scrum | 13:01 |
| slaweq | weshay|ruck: ok, thx | 13:01 |
| slaweq | so it is weird | 13:01 |
| marios | ysandeep: ping cos weshay|ruck said | 13:01 |
| slaweq | as it runs for 2h and can't even get to tempest run with cpu_model=EPYC | 13:02 |
| slaweq | and without that setting in nova-compute it was running (and failing) tempest in about 1h30m | 13:02 |
| *** ykarel_ is now known as ykarel | 13:02 | |
| slaweq | weshay|ruck++ chandankumar++ thx a lot for help | 13:02 |
| slaweq | I will need to ask compute dfg folks to take a look at that one again | 13:03 |
| chandankumar | slaweq: https://logserver.rdoproject.org/57/32157/6/check/periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-master/4cb7d57/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 13:05 |
| chandankumar | it timed out while configuring octavia on overcloud | 13:05 |
| chandankumar | https://logserver.rdoproject.org/57/32157/6/check/periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-master/4cb7d57/logs/undercloud/home/zuul/standalone-ansible-5m46o8e2/octavia-ansible/octavia-ansible.log.txt.gz | 13:06 |
| cgoncalves | there's a long-standing LP for this where ceph-ansible was also being impacted | 13:07 |
| *** udesale has joined #oooq | 13:08 | |
| slaweq | cgoncalves: but the same job was running better last week, see https://review.rdoproject.org/zuul/build/cf88f4fcd8bf448ab42e05955eeb13a6 | 13:08 |
| slaweq | it was deploying everything correctly and then tempest was failing | 13:08 |
| slaweq | this week I just changed cpu_models in nova-compute's config there | 13:09 |
| slaweq | and it can't even get to run tempest | 13:09 |
| cgoncalves | slaweq, yes. the issue I am talking about is a *random* one at overcloud deployment where it gets stuck while configuring octavia | 13:09 |
| slaweq | cgoncalves: ahh, ok | 13:10 |
| slaweq | so I will try to recheck couple more times :) | 13:10 |
| slaweq | thx | 13:10 |
| ysandeep | chandankumar, fyi.. you are on mute for us | 13:10 |
| chandankumar | ysandeep: we can move both the cards to done | 13:10 |
| chandankumar | ysandeep: some issue with mike | 13:10 |
| ysandeep | chandankumar, ack thanks! | 13:10 |
| cgoncalves | chandankumar, weshay|ruck, slaweq: https://bugs.launchpad.net/tripleo/+bug/1881087 | 13:11 |
| openstack | Launchpad bug 1881087 in tripleo "tripleo-ci-centos-8-scenario010-standalone timing out frequently" [High,Triaged] | 13:11 |
| slaweq | cgoncalves: thx | 13:18 |
| weshay|ruck | cgoncalves, slaweq and chandan are looking at that timeout right now | 13:19 |
| chandankumar | rlandy: weshay|ruck marios sshnaidm setting a meeting on tripleo-repos on monday? | 13:31 |
| chandankumar | * for monday | 13:31 |
| marios | chandankumar: ack thanks | 13:31 |
| sshnaidm | chandankumar, I have a conflict, maybe 30 minutes later? | 13:37 |
| sshnaidm | chandankumar, or before scrum | 13:37 |
| *** jmasud has joined #oooq | 13:44 | |
| *** jmasud has quit IRC | 13:46 | |
| marios | ysandeep: good job ;) | 13:49 |
| chandankumar | ysandeep: thanks! | 13:50 |
| rlandy | marios: https://review.opendev.org/c/openstack/releases/+/790053 are we approved here? | 13:50 |
| weshay|ruck | chandankumar, you have any notes on your yum module work thus far? | 13:50 |
| ysandeep | marios, chandankumar thanks! | 13:50 |
| weshay|ruck | hackmd | 13:50 |
| * weshay|ruck looks | 13:50 | |
| rlandy | short a w+ | 13:51 |
| ysandeep | rlandy, looking at https://code.engineering.redhat.com/gerrit/#/c/240044/ now | 13:56 |
| ysandeep | rlandy, lgtm | 13:59 |
| chandankumar | weshay|ruck: https://hackmd.io/mU1te45ASEWbnLqpSexB_Q?view | 13:59 |
| weshay|ruck | akahat, pojadhav|rover we need 3rd party ovb fs001 to hit the repo here: https://review.opendev.org/c/openstack/ansible-role-tripleo-modify-image/+/789999 | 13:59 |
| weshay|ruck | ansible-role-tripleo-modify-image | 14:00 |
| weshay|ruck | who has time to add that job to ensure it's triggering on changes to that repo? | 14:00 |
| weshay|ruck | re: https://bugs.launchpad.net/tripleo/+bug/1927302 | 14:00 |
| openstack | Launchpad bug 1927302 in tripleo "rdo jobs broken with stat /etc/yum/vars: no such file or directory"" [Critical,Triaged] | 14:00 |
| chandankumar | weshay|ruck: job is added | 14:00 |
| chandankumar | weshay|ruck: https://review.rdoproject.org/r/c/config/+/33602 | 14:01 |
| weshay|ruck | chandankumar++ ykarel++ | 14:01 |
| weshay|ruck | thanks | 14:01 |
| rlandy | ysandeep: thanks merging | 14:07 |
| *** akahat is now known as akahat|ruck | 14:07 | |
| weshay|ruck | chandankumar, slaweq https://bugs.launchpad.net/tripleo/+bug/1881087 https://bugs.launchpad.net/tripleo/+bug/1919312 may be related to what you guys were seeing | 14:08 |
| openstack | Launchpad bug 1881087 in tripleo "tripleo-ci-centos-8-scenario010-standalone timing out frequently" [High,Triaged] | 14:08 |
| openstack | Launchpad bug 1919312 in tripleo "Octavia fails in scenario010 - sometimes" [Critical,Triaged] | 14:08 |
| weshay|ruck | marios, rlandy fyi.. we have wf :) https://review.opendev.org/c/openstack/releases/+/790053 | 14:14 |
| weshay|ruck | oh no we don't | 14:14 |
| weshay|ruck | sorry | 14:14 |
| weshay|ruck | tired | 14:14 |
| rlandy | Workflow no votes | 14:15 |
| marios | weshay|ruck: yeah but we have +2 from elod which is good ;) | 14:15 |
| rlandy | weshay|ruck: what's it going to cost us??? | 14:15 |
| * rlandy starts 'go fund me' campaign | 14:15 | |
| marios | weshay|ruck: just ping herve in tripleo too lets see | 14:16 |
| weshay|ruck | rock on | 14:16 |
| ysandeep | rlandy, could you please review https://code.engineering.redhat.com/gerrit/#/c/239858/ , I would like this to be merged as this is in config project, so that I can do testing with testproject.. right now I have to login on each test run and fix this.. | 14:18 |
| rlandy | ysandeep: voted | 14:20 |
| rlandy | ysandeep: do you have w+ there? | 14:20 |
| ysandeep | rlandy, for ex: you will see DockerInsecureRegistryAddress twice here https://sf.hosted.upshift.rdu2.redhat.com/logs/88/232088/15/check/tripleo-ci-rhel-8-standalone-rhos-16.2/877b1fc/logs/undercloud/home/zuul/standalone_parameters.yaml | 14:20 |
| rlandy | k | 14:21 |
| ysandeep | rlandy, thanks! I will +w it.. Please feel free to revert if if causes any issue.. I will keep an eye on next run. | 14:21 |
| rlandy | ysandeep: sure | 14:21 |
| rlandy | ysandeep: I am putting in changes to clear a bunch of the errors from downstream zuul | 14:22 |
| ysandeep | rlandy, could you please check https://code.engineering.redhat.com/gerrit/#/c/232088/ .. failure seems related to nodeset | 14:22 |
| sshnaidm | akahat|ruck, pojadhav|rover weshay|ruck is ovb failure know? https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 | 14:23 |
| sshnaidm | some problem with modifying image | 14:23 |
| rlandy | ysandeep: hmmm ... | 14:23 |
| rlandy | latest merge should have fixed that | 14:24 |
| rlandy | ysandeep: can you rebase? | 14:24 |
| ysandeep | rlandy, ack, my job was already running when this change was merged, will rebase my patches | 14:24 |
| rlandy | ysandeep: also https://code.engineering.redhat.com/gerrit/240241 Remove horzion jobs from tripleo-ci-internal | 14:26 |
| rlandy | part of error clean up | 14:26 |
| ysandeep | rlandy, lgtm | 14:26 |
| *** ykarel_ has joined #oooq | 14:27 | |
| pojadhav|rover | sshnaidm, yes https://bugs.launchpad.net/tripleo/+bug/1927302 | 14:28 |
| openstack | Launchpad bug 1927302 in tripleo "rdo jobs broken with stat /etc/yum/vars: no such file or directory"" [Critical,Triaged] | 14:28 |
| pojadhav|rover | sshnaidm, https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-wallaby/79a3c62/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 14:28 |
| sshnaidm | pojadhav|rover, ack | 14:28 |
| sshnaidm | weshay|ruck, probably this needs a big cleanup: https://bugs.launchpad.net/tripleo/+bugs?field.tag=promotion-blocker | 14:29 |
| *** ykarel has quit IRC | 14:30 | |
| rlandy | ysandeep: still have the nodeset error? | 14:30 |
| weshay|ruck | sshnaidm, we have scripts that set bugs to incomplete at a certain age... marios you want to do that or should I? | 14:30 |
| sshnaidm | weshay|ruck, and probably to remove promotion-blocker tag? | 14:31 |
| weshay|ruck | meh | 14:31 |
| weshay|ruck | sshnaidm, I have a filter for what's important.. https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=0&fvid=1778061576 | 14:32 |
| sshnaidm | weshay|ruck, nice | 14:32 |
| weshay|ruck | sshnaidm, data -> filter -> active | 14:32 |
| sshnaidm | but spreadsheets are for accountants | 14:33 |
| ysandeep | sshnaidm lol! | 14:33 |
| rlandy | ysandeep: ok - we are down from 550 to 157 errors | 14:33 |
| weshay|ruck | sshnaidm, https://gitlab.cee.redhat.com/whayutin/bug_tracking | 14:34 |
| ysandeep | rlandy, I have rebased my patch, jobs are running: I think we are good with nodeset error.. | 14:37 |
| rlandy | ok | 14:38 |
| ysandeep | rlandy, weshay|ruck fyi.. I am running 16.2 tempest api/scenario in testproject: https://code.engineering.redhat.com/gerrit/#/c/211494/ with osp_release revert as depends-on, So that we can promote if today's run is good.. we are behind in promotion ~2 weeks. | 14:45 |
| marios | weshay|ruck: rlandy: nope https://review.opendev.org/c/openstack/releases/+/790053/1#message-bbe5f4fb45f56d5dd6a48fa4c282cd44ec8da792 | 14:45 |
| marios | weshay|ruck: is not legal to fetch a new 'from master' hash | 14:45 |
| weshay|ruck | marios, k.. let's discuss at the DF mtg | 14:46 |
| marios | weshay|ruck: so this means we have to go with backports... lets discuss with DF in a bit | 14:46 |
| marios | ack | 14:46 |
| weshay|ruck | ysandeep, I'm going to promote 17 components to get your patch https://review.opendev.org/c/openstack/tripleo-common/+/789563 | 14:50 |
| weshay|ruck | rlandy, ysandeep internal promoter .. can you ping it? | 14:51 |
| ysandeep | weshay|ruck, we have branched tripleo-common, i think we need cherry-pick on wallaby https://review.opendev.org/c/openstack/tripleo-common/+/789809 | 14:52 |
| rlandy | weshay|ruck: checking | 14:52 |
| *** pojadhav|rover is now known as pojadhav|away | 14:52 | |
| ysandeep | i can't ssh.. checking console | 14:52 |
| rlandy | weshay|ruck: no can't ping | 14:53 |
| weshay|ruck | marios, while ur moving bugs.. can you run it to set anything over 365 that is not triaged or fixed.. to incomplete | 14:53 |
| marios | weshay|ruck: abandoned https://review.opendev.org/c/openstack/releases/+/790053/1#message-4efb55546da328991f5a5c53de33eb47507b969f | 14:53 |
| weshay|ruck | k | 14:53 |
| marios | weshay|ruck: not today i will do that next | 14:54 |
| weshay|ruck | k.. thanks | 14:54 |
| marios | weshay|ruck: i mean the stale bugs thing | 14:54 |
| weshay|ruck | ya | 14:54 |
| weshay|ruck | and to sshnaidm's point. we should probably run it against bugs w/ promotion-blocker that are older than .. maybe a year or something | 14:54 |
| marios | weshay|ruck: so at least we established it isn't a case of 'update hashes before release' no such thing (you can update to a newer hash in stable/wallaby like i did) but can't fetch from master | 14:55 |
| marios | weshay|ruck: so the only solution is to not use client/common | 14:55 |
| marios | weshay|ruck: or use some other repo like we spoke about with rlandy earlier | 14:55 |
| ysandeep | rlandy, weshay|ruck https://paste.opensuse.org/42320886 .. looks like issue with promoter OS | 14:55 |
| ysandeep | rlandy, weshay|ruck, want to try reboot? | 14:56 |
| *** dviroel|nho is now known as dviroel | 14:56 | |
| weshay|ruck | ysandeep, ya.. go 4 it | 14:56 |
| ysandeep | aye o/ | 14:56 |
| rlandy | ysandeep: ack - nothing to lose | 14:56 |
| ysandeep | i wish we have taken a snapshot earlier.. next time i will do it | 14:57 |
| ysandeep | IMHO.. we should have a snapshot for our IMP server | 14:58 |
| marios | weshay|ruck: o/ joining df? | 15:00 |
| *** ykarel_ is now known as ykarel|away | 15:01 | |
| rlandy | marios: we got a -1 now? | 15:01 |
| marios | rlandy: yeah it is illegal like we suspected | 15:01 |
| marios | rlandy: cos we were fetching from master | 15:01 |
| marios | cant do that | 15:01 |
| marios | rlandy: gonna discuss on df call now if you wana join | 15:01 |
| rlandy | marios: ok | 15:01 |
| rlandy | will eavesdrop | 15:02 |
| rlandy | frenzy_friday: can you point me to the doc again? | 15:05 |
| rlandy | tripleo-health | 15:05 |
| ysandeep | rlandy, weshay|ruck err.. our internal promoter vm is in rescue mode: http://pastebin.test.redhat.com/961855 | 15:10 |
| rlandy | ysandeep: weshay|ruck: we may need to redeploy this one | 15:10 |
| rlandy | chandankumar: ^^ how far out are we from the deploying the internal promoter? | 15:11 |
| ysandeep | its booting so fast.. I can't go to grub | 15:11 |
| rlandy | can we snapshot it? | 15:12 |
| ysandeep | We can but at this stage.. dont't think snapshot will be of much value | 15:13 |
| weshay|ruck | rlandy, we saved those scripts right? I think I have a patch or something | 15:16 |
| frenzy_friday | rlandy, https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/788785 | 15:19 |
| rlandy | weshay|ruck: I dearly hope so :) | 15:22 |
| rlandy | weshay|ruck: the image and dockerfile ones are on the server | 15:23 |
| rlandy | weshay|ruck: why don't we just try deploy the downstream promoter | 15:23 |
| rlandy | once we are down and out | 15:23 |
| rlandy | marios: pls ping if you need help with backports | 15:23 |
| rlandy | had to drop for another meeting | 15:24 |
| rlandy | frenzy_friday: thanks - found it from a previous message :) | 15:24 |
| marios | rlandy: so slagle said he will post those so we should try and help with the reviews | 15:25 |
| marios | rlandy: i'll get on those in my morning he will have posted them by then | 15:25 |
| rlandy | ack | 15:25 |
| *** ykarel|away has quit IRC | 15:25 | |
| *** jpodivin has quit IRC | 15:32 | |
| ysandeep | rlandy, weshay|ruck i am trying to rescue that instance.. I have created a snapshot promoter-server-before-troubleshooting before touching anything.. its late for me.. if it takes time.. i will continue tomorrow | 15:33 |
| rlandy | ysandeep: np ... have a good night | 15:34 |
| rlandy | weshay|ruck and I will pick it up | 15:34 |
| *** sshnaidm is now known as sshnaidm|afk | 15:44 | |
| weshay|ruck | ysandeep, rlandy which tenant is that in? | 15:57 |
| ysandeep | weshay|ruck, rhos-dev-ci | 15:58 |
| rlandy | weshay|ruck: rdo-dev-ci | 15:58 |
| rlandy | sec ... working with frenzy_friday | 16:00 |
| *** jpena is now known as jpena|away | 16:07 | |
| *** marios is now known as marios|out | 16:09 | |
| *** jmasud has joined #oooq | 16:18 | |
| rlandy | weshay|ruck: you sill working on the downstream promoter? | 16:26 |
| ysandeep | hard luck.. instance is boot from volume.. nova have "nova rescue" but not cinder...afaik.. | 16:26 |
| rlandy | ysandeep: we will have to start again there | 16:27 |
| rlandy | it's late | 16:27 |
| rlandy | we can pick this up with you tomorrow | 16:27 |
| rlandy | will check it out | 16:27 |
| ysandeep | Just need to remove "/dev/mapper/loop0p1 /var/lib/docker" from fstab.. But i can't get into rw mode .. will continue checking tomorrow | 16:28 |
| * ysandeep out | 16:28 | |
| rlandy | not worth it | 16:28 |
| *** ysandeep is now known as ysandeep|away | 16:28 | |
| rlandy | lunch ... brb | 16:33 |
| *** marios|out has quit IRC | 16:36 | |
| *** amoralej is now known as amoralej|off | 16:37 | |
| *** jpena|away is now known as jpena|off | 16:47 | |
| weshay|ruck | ysandeep|away, you in the fstab? | 17:04 |
| weshay|ruck | rlandy, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33150/1/ci-scripts/infra-setup/roles/promoter/templates/force_promote_component.sh.j2 :) | 17:06 |
| *** udesale has quit IRC | 17:06 | |
| weshay|ruck | rlandy, looking for the image stuff | 17:07 |
| rlandy | weshay|ruck++ good man! | 17:07 |
| rlandy | weshay|ruck: should create a new instance in the in the other tenant | 17:08 |
| rlandy | will do that | 17:08 |
| rlandy | to reset this up | 17:08 |
| rlandy | weshay|ruck: pls ping when back | 19:13 |
| rlandy | put together the 16.2 promote script | 19:13 |
| *** saneax has quit IRC | 19:30 | |
| *** jmasud has quit IRC | 19:39 | |
| *** dviroel is now known as dviroel|brb | 19:51 | |
| weshay|ruck | rlandy, back | 20:26 |
| *** dviroel|brb is now known as dviroel | 20:33 | |
| rlandy | if it works | 20:35 |
| rlandy | and I didn't mess anything up | 20:35 |
| *** slaweq has quit IRC | 20:38 | |
| *** jmasud has joined #oooq | 21:02 | |
| *** rlandy is now known as rlandy|bbl | 22:24 | |
| *** jmasud has quit IRC | 22:27 | |
| *** tosky has quit IRC | 23:17 | |
| *** jmasud has joined #oooq | 23:42 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!