| *** rlandy|ruck|bbl is now known as rlandy|ruck | 00:20 | |
| *** ysandeep|out is now known as ysandeep | 04:04 | |
| ysandeep | chandankumar, fyi.. https://bugs.launchpad.net/tripleo/+bug/1946822 we need to cix c9 blocker, right? | 04:12 |
|---|---|---|
| ysandeep | ykarel|away: hey good morning o/ I am working on c9, I don't see extras repos here: https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/extras/x86_64/os/ , do you know if we have moved content of that repo to other repos? | 04:22 |
| *** ykarel|away is now known as ykarel | 04:32 | |
| ykarel | ysandeep, it will be there when c9 get's released | 04:32 |
| ykarel | as that contains release rpms for SIG | 04:33 |
| ysandeep | ykarel, ack, thanks! | 04:33 |
| ysandeep | ykarel, fyi.. we are hitting this dependency issue https://bugs.launchpad.net/tripleo/+bug/1946822 | 04:33 |
| ykarel | ysandeep, ack will check in some time, currently in a meeting | 04:34 |
| ysandeep | ykarel: sure, no hurry | 04:34 |
| chandankumar | ysandeep: yes, all cs9 issues needs to be cixed | 04:42 |
| ysandeep | chandankumar: ack, thanks! adding promotion-blocker flag | 04:44 |
| chandankumar | ysandeep: you can find few centos repos here also https://trunk.rdoproject.org/centos9-master/delorean-deps.repo coming from deps | 04:45 |
| ysandeep | chandankumar: ack | 04:46 |
| chandankumar | ysandeep: container-tools module comes from appstream | 04:47 |
| chandankumar | so we do not need powertools repos | 04:47 |
| chandankumar | in future there will be no container-tools modules for cs9 also | 04:47 |
| ysandeep | chandankumar, I didn't find powertools here: https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/ but I found it here https://composes.centos.org/latest-CentOS-Stream-8/compose/PowerTools/x86_64/os/ , If we don't need that, I can remove it . | 04:50 |
| chandankumar | ysandeep: https://wiki.centos.org/AdditionalResources/Repositories | 04:51 |
| chandankumar | PowerTools - Available only for CentOS8, the PowerTools repository provides most of the developer tools. Disabled by default. | 04:51 |
| chandankumar | but I will confirm with amoralej|off regarding powertools (if in future needed for any package) | 04:52 |
| *** bhagyashris is now known as bhagyashris|rover | 05:21 | |
| chandankumar | ysandeep: you can find the container build logs here https://logserver.rdoproject.org/53/18953/106/check/tripleo-build-containers-stream9-development/75f400e/logs/build.log | 05:31 |
| ysandeep | chandankumar, I was talking about individual container build log, it was missing in the failing job, i think that was failing at a earlier stage, I can see logs in good run https://logserver.rdoproject.org/53/18953/106/check/tripleo-build-containers-stream9-development/9379f76/logs/container-builds/83cef063-7a2b-4fe7-b27d-29863f349dcd/base/os/os-build.log | 05:34 |
| chandankumar | ysandeep: yes, it was failing in the earlier stage while evaluating the jinja | 05:35 |
| chandankumar | thanks for checking in by the way :-) | 05:36 |
| ysandeep | my pleasure :D | 05:42 |
| marios | bhagyashris|rover: arxcruz|ruck: o/ any fallout observed from https://review.opendev.org/c/openstack/tripleo-ci/+/810261 ? (is 3rd party/rdo check/gate broken?) | 06:10 |
| *** amoralej|off is now known as amoralej | 06:19 | |
| amoralej | ysandeep, chandankumar in centos9 the equivalent to powertools is CRB | 06:20 |
| amoralej | we need powertools in CS8 and CRB in CS9 | 06:21 |
| ysandeep | amoralej, ack thanks and congratulations! | 06:21 |
| amoralej | thanks ysandeep :) | 06:21 |
| ysandeep | amoralej, fyi.. we are hitting this issue https://bugs.launchpad.net/tripleo/+bug/1946822 because of missing genisoimage package | 06:22 |
| amoralej | mmm we are waiting for a new release to merge the review, iirc | 06:23 |
| amoralej | https://review.rdoproject.org/r/c/openstack/ironicclient-distgit/+/35671 | 06:23 |
| ysandeep | amoralej, thanks! added myself as a follower on that review | 06:25 |
| bhagyashris|rover | marios, not yet. will check | 06:37 |
| marios | bhagyashris|rover: thanks i looked around a bit i saw some running 3rd party jobs after that merged . | 06:38 |
| marios | bhagyashris|rover: if you see something ping we will need to revert immediately | 06:38 |
| marios | bhagyashris|rover: or find some other solution anyway | 06:38 |
| bhagyashris|rover | marios, ack thanks | 06:44 |
| marios | bhagyashris|rover: arxcruz|ruck: o/ whats the story on tox molecule job https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-molecule&project=openstack/ansible-role-collect-logs | 07:22 |
| marios | seems to be broken for a while since september | 07:23 |
| marios | ah i see https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/809979 | 07:25 |
| bhagyashris|rover | marios, ysandeep chandankumar kindly add in your review list fix mol-get-hash-centos7 job https://review.opendev.org/c/zuul/zuul-jobs/+/813749 thanks | 07:26 |
| bhagyashris|rover | here iit's tested https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36231 | 07:28 |
| *** jpena|off is now known as jpena | 07:29 | |
| *** poojajadhav is now known as pojadhav | 07:29 | |
| marios | bhagyashris|rover: ack but you will have to find some zuul-jobs cores | 07:29 |
| bhagyashris|rover | yup | 07:30 |
| marios | bhagyashris|rover: https://review.opendev.org/admin/groups/339d2fc70a5268571c130371e7501193d9ce7e86,members | 07:30 |
| bhagyashris|rover | marios, thanks | 07:31 |
| bhagyashris|rover | added | 07:32 |
| Tengu | tosky, marios https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/813777 | 08:17 |
| marios | Tengu: ack thanks adding to reviews | 08:22 |
| Tengu | :) | 08:23 |
| Tengu | trying my best | 08:23 |
| arxcruz | bhagyashris|rover: ping | 08:47 |
| arxcruz | ok, i'm back | 08:47 |
| *** arxcruz is now known as arxcruz|rover | 08:48 | |
| Tengu | arxcruz|rover: I see you're trying to correct the molecule tests on the collect-logs role - is this one already known? https://6bccc71dcd109068b189-1635a47266c72ea76f240926bd5c4cc0.ssl.cf2.rackcdn.com/813777/3/check/openstack-tox-molecule/6d68348/tox/reports.html | 09:21 |
| Tengu | I think I've already seen something related to that error... | 09:22 |
| *** ykarel is now known as ykarel|lunch | 09:23 | |
| *** ysandeep is now known as ysandeep|mtg | 09:25 | |
| arxcruz|rover | Tengu: oh, is this back ? | 09:25 |
| Tengu | arxcruz|rover: apparently? I'm up-to-date against master I think | 09:25 |
| arxcruz|rover | frenzy_friday: did you change the elastic again ? | 09:25 |
| Tengu | yep. up-to-date against master. | 09:25 |
| frenzy_friday | arxcruz|rover, the nova mag? Nope | 09:26 |
| arxcruz|rover | Tengu: yeah, this is a bit hard to properly fix, and i'm ruck rovering | 09:26 |
| * frenzy_friday reading ^ | 09:26 | |
| arxcruz|rover | frenzy_friday: i think the molecule job is getting a different nova error | 09:26 |
| Tengu | yay :) | 09:27 |
| arxcruz|rover | Tengu: i mean, it's simulating a nova error, not a "real" error :) | 09:27 |
| Tengu | arxcruz|rover: ah :). better! | 09:28 |
| Tengu | but since it's not working..... | 09:28 |
| frenzy_friday | checking if there are any more "No_valid_host_was_found" in the queries | 09:28 |
| frenzy_friday | arxcruz|rover, I thought we already changed this https://opendev.org/openstack/tripleo-ci-health-queries/src/branch/master/src/data/queries.yml#L578 in patch https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/809577/2/src/data/queries.yml#578 | 09:33 |
| frenzy_friday | not sure what happened, adding again | 09:33 |
| arxcruz|rover | frenzy_friday: https://opendev.org/openstack/tripleo-ci-health-queries/src/branch/master/src/data/queries.yml#L33 | 09:33 |
| arxcruz|rover | there's more | 09:34 |
| arxcruz|rover | not sure if it's related, need to check | 09:34 |
| arxcruz|rover | after the meeting today i can take a look | 09:34 |
| frenzy_friday | yes, but what happened to patch 809577 that merged? | 09:34 |
| arxcruz|rover | i have deutches klass jetzt | 09:34 |
| arxcruz|rover | frenzy_friday: no idea | 09:34 |
| arxcruz|rover | frenzy_friday: ci works in misterious ways | 09:34 |
| Tengu | Schroedinger Patch? :) | 09:37 |
| frenzy_friday | ^ lol | 09:40 |
| Tengu | ^^ | 09:41 |
| frenzy_friday | https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/813791 should fix | 09:42 |
| Tengu | frenzy_friday: hmmmm the name seems different? _No_valid_host_was_found.log is wanted in the molecule run for my patch... Or is it not linked? | 09:43 |
| Tengu | i.e. there seem to be a space prefix, and a capital N.. | 09:44 |
| frenzy_friday | checking | 09:44 |
| soniya29 | chandankumar, kopecmartin, arxcruz|rover, ysandeep|mtg, please add/edit today's agenda for the tempest meeting - https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ | 09:51 |
| Tengu | frenzy_friday: can I depends-on my patch on yours? | 09:52 |
| frenzy_friday | Tengu, no, sova/ansible collect logs directly downloads the file https://opendev.org/openstack/tripleo-ci-health-queries/src/branch/master/output/sova-pattern-generated.json - so depends on doent work right now | 09:54 |
| Tengu | frenzy_friday: ok! no problem | 09:54 |
| frenzy_friday | That is what arx was trying to fix before his ruck/rover sprint | 09:54 |
| Tengu | ok | 09:54 |
| Tengu | so no actual way to test things then | 09:54 |
| frenzy_friday | Tengu, right now I am comparing https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/813791/2/output/sova-pattern-generated.json and https://opendev.org/openstack/tripleo-ci-health-queries/src/branch/master/output/sova-pattern-generated.json | 09:55 |
| frenzy_friday | Checking if the script adds _ prefix and capitalises the forst letter or that has to be present | 09:56 |
| Tengu | ok | 09:56 |
| frenzy_friday | Hey dpawlik, is https://review.rdoproject.org/analytics/app/ down? | 09:58 |
| dpawlik | >.< yes.... | 10:04 |
| dpawlik | frenzy_friday: restarting... | 10:08 |
| frenzy_friday | thanks | 10:08 |
| dpawlik | frenzy_friday: could you make a notes after which query the elastic is dying ? | 10:09 |
| dpawlik | frenzy_friday: we increse xmx and xms flag twice from default vaules that was running few months | 10:09 |
| dpawlik | frenzy_friday: and it's still dying | 10:09 |
| dpawlik | frenzy_friday: maybe I can try to help with query or set some limitation...? | 10:10 |
| dpawlik | if it needs to be like it is, I will set the current values as default | 10:11 |
| frenzy_friday | dpawlik, yes sure, lemme rerun the last query that I ran and see if it goes down again | 10:11 |
| dpawlik | thanks frenzy_friday ! | 10:13 |
| frenzy_friday | ok, its running now | 10:15 |
| frenzy_friday | dpawlik, when a patch is submitted to this repo the check job validates the query files (https://opendev.org/openstack/tripleo-ci-health-queries/src/branch/master/output/elastic-recheck) be querying ES with each of them | 10:16 |
| frenzy_friday | ^ this was the last thing I ran | 10:17 |
| frenzy_friday | once on my local and once by zuul | 10:17 |
| dpawlik | frenzy_friday: hmm, could we add some time limitation like "last 7 days"? so far we have retention set to 14 days AFAIR | 10:19 |
| frenzy_friday | dpawlik, ack, lemme change it to 7 days in elastic recheck | 10:20 |
| dpawlik | it would be huge improvement | 10:20 |
| dpawlik | <elasticsearch says thank you :) > | 10:20 |
| akahat | ysandeep|mtg, chandankumar, arxcruz|rover bhagyashris|rover ping, I'm stopping downstream promoter server.. i'll let you know once issue is fixed. | 10:20 |
| bhagyashris|rover | akahat, ack | 10:21 |
| frenzy_friday | Tengu, arxcruz|rover looks like there were some mismatch btw the original sova file and the ones we generate - https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/813791 should fix it hopefully | 10:28 |
| *** pojadhav is now known as pojadhav|brb | 10:31 | |
| *** rlandy is now known as rlandy|ruck | 10:31 | |
| rlandy|ruck | arxcruz|rover: bhagyashris|rover: hi - how are things today? | 10:35 |
| arxcruz|rover | rlandy|ruck: a few failures | 10:37 |
| * arxcruz|rover needs 30 min | 10:37 | |
| *** ykarel|lunch is now known as ykarel | 10:37 | |
| rlandy|ruck | arxcruz|rover: there are conent-provider failures due to ceph related containers | 10:38 |
| rlandy|ruck | did you have a patch for those issues? | 10:39 |
| rlandy|ruck | arxcruz|rover: other failures you are looking at? | 10:39 |
| arxcruz|rover | rlandy|ruck: i thought chandankumar was working on that | 10:40 |
| rlandy|ruck | 2021-10-12 21:29:56.758376 | primary | "cmd": "buildah images; buildah push --format=v2s2 --tls-verify=False --log-level debug quay.ceph.io/prometheus/node-exporter:v0.17.0 docker://127.0.0.1:5001/tripleomaster/node-exporter:v0.17.0;\n", | 10:40 |
| rlandy|ruck | chandankumar: ^^ do we have a bug for this | 10:40 |
| chandankumar | rlandy|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/782362 | 10:40 |
| chandankumar | arxcruz|rover: I got busy with cs9 work sorry for that | 10:40 |
| chandankumar | please take over the review | 10:40 |
| rlandy|ruck | chandankumar: thank you | 10:41 |
| rlandy|ruck | 10:41 | |
| rlandy|ruck | Fix Released | 10:41 |
| rlandy|ruck | bug is ^^ | 10:41 |
| bhagyashris|rover | rlandy|ruck, the issue you pointed got resolved | 10:41 |
| bhagyashris|rover | promoting the first hash one | 10:42 |
| rlandy|ruck | akahat: hi - any update on internal promoter? | 10:42 |
| rlandy|ruck | bhagyashris|rover: on rdo promotions? | 10:42 |
| bhagyashris|rover | rlandy|ruck, yes, | 10:42 |
| akahat | rlandy|ruck, hello.. i'm looking in to it. | 10:42 |
| rlandy|ruck | bhagyashris|rover: also - thank you for posting a patch for mol-7 | 10:42 |
| bhagyashris|rover | rlandy|ruck, :) | 10:43 |
| rlandy|ruck | akahat: should we promote 17 with the old promoter? | 10:43 |
| rlandy|ruck | or do you want to use this to test the current one? | 10:43 |
| akahat | rlandy|ruck, i want to use the current one for testing. so we can have smooth promotions. | 10:44 |
| rlandy|ruck | akahat: ok | 10:44 |
| akahat | rlandy|ruck, for internal promoter issue with name resolution. Paramiko is not able to resolve the hostname. Replacing it with ip is working. | 10:49 |
| rlandy|ruck | akahat: ok - has 17 promoted? | 10:50 |
| rlandy|ruck | is there an associated patch? | 10:50 |
| akahat | rlandy|ruck, no. not yet. I'm trying to promote it now. | 10:50 |
| rlandy|ruck | akahat: great - thanks | 10:51 |
| rlandy|ruck | bhagyashris|rover: can you look into https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/782362 | 10:51 |
| bhagyashris|rover | rlandy|ruck, will check | 10:54 |
| rlandy|ruck | bhagyashris|rover: thanks - see yatin's comment | 10:54 |
| akahat | rlandy|ruck, bhagyashris|rover arxcruz|rover ysandeep|mtg RHOS-17 is promoted: http://10.0.110.143/promoter_logs/redhat8_osp17.log | 10:56 |
| akahat | sending patch to fix it. | 10:57 |
| ysandeep|mtg | akahat++ nice, thank you! | 10:57 |
| *** ysandeep|mtg is now known as ysandeep | 10:57 | |
| rlandy|ruck | akahat: good news- thanks for fixing that | 10:58 |
| Tengu | frenzy_friday: great, thanks! | 11:04 |
| frenzy_friday | Tengu, arxcruz|rover when you get some time pls add to your review list https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/813791 | 11:05 |
| Tengu | frenzy_friday: on it! | 11:05 |
| Tengu | though... is "Nova_failure_no_valid_host_was_found" correct in https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/813791/4/output/sova-pattern-generated.json#695 ? I'm not used to sova tool, so that's probably a stupid question | 11:05 |
| Tengu | https://logserver.rdoproject.org/77/813777/3/openstack-check/tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset001/1f31dcc/logs/undercloud/var/log/extra/selinux_consolidated_avc.txt.gz \o/ | 11:12 |
| Tengu | leading to: https://logserver.rdoproject.org/77/813777/3/openstack-check/tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset001/1f31dcc/logs/undercloud/var/log/extra/selinux_denials_detail.txt.gz | 11:13 |
| Tengu | hmmm. wait. I should also use the "what" in the matcher. | 11:13 |
| frenzy_friday | Tengu, I think so, because in the original sova file we have https://opendev.org/openstack/tripleo-ci-health-queries/src/branch/master/output/sova-patterns.json#L982 -> "ResourceInError: resources.NovaCompute: Went to statu..." | 11:15 |
| frenzy_friday | In the generated one https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/813791/4/output/sova-pattern-generated.json#695 -> "id:Nova_failure_no_valid_host_was_found" -> https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/813791/4/output/sova-pattern-generated.json#1281 | 11:16 |
| frenzy_friday | So "ResourceInError: resources.NovaCompute: Went to status ERROR due to \"Message: No valid host was found. , Code: 500" in both files are called "Nova_failure_no_valid_host_was_found" | 11:16 |
| rlandy|ruck | akahat: hey - chandankumar and I would like to merge the log rotation patch - pls can you look at marios commemts on https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36130/3/ci-scripts/infra-setup/roles/promoter/tasks/main.yml | 11:16 |
| akahat | rlandy|ruck, ysandeep https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/281644 | 11:17 |
| *** dviroel|out is now known as dviroel | 11:17 | |
| akahat | rlandy|ruck, okay looking in it. | 11:17 |
| frenzy_friday | Tengu, and msg:"No valid host was found." in the original file corresponds to patterns "No valid host was found. There are not enough hosts" and "Went to status ERROR due to \"Message: No valid host was found" (lines 1108 and 172. | 11:19 |
| rlandy|ruck | akahat: hi - https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/281644/1/ci-scripts/dlrnapi_promoter/config_environments/rdo/RedHat-8/rhos-16.2.yaml | 11:19 |
| rlandy|ruck | pls see question in ^^ | 11:19 |
| rlandy|ruck | images is in there twice | 11:19 |
| rlandy|ruck | ok to use ip if we really need to | 11:19 |
| rlandy|ruck | let me know and will merge | 11:19 |
| frenzy_friday | In the generated file msg:"No valid host was found." -> https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/813791/4/output/sova-pattern-generated.json#65 and https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/813791/4/output/sova-pattern-generated.json#806 | 11:20 |
| ysandeep | akahat, rlandy|ruck instead of replacing host with ip, I think would be good if we add internal dns in resolc.conf | 11:20 |
| ysandeep | resolv.conf* | 11:20 |
| rlandy|ruck | ysandeep: ack | 11:20 |
| rlandy|ruck | it should resolve | 11:20 |
| akahat | ysandeep, yes that will also work. | 11:21 |
| rlandy|ruck | ysandeep: https://review.opendev.org/c/openstack/tripleo-quickstart/+/813624 - has that been tested with downstream? | 11:22 |
| rlandy|ruck | if no problems there, we can merge | 11:22 |
| Tengu | frenzy_friday: ok. well. let's get it in then! | 11:22 |
| ysandeep | rlandy|ruck, chandankumar left a comment there.. I need to look at that. | 11:23 |
| ysandeep | + test with downstream + c7 jobs | 11:23 |
| rlandy|ruck | ysandeep: ok - pls ping when we have that confirmation and we'll merge | 11:23 |
| ysandeep | ack | 11:24 |
| *** jpena is now known as jpena|lunch | 11:24 | |
| frenzy_friday | *fingers crossed* | 11:24 |
| akahat | rlandy|ruck, marios chandankumar there is one more question related to logrotate: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36130/3/ci-scripts/infra-setup/roles/promoter/tasks/main.yml | 11:26 |
| rlandy|ruck | ok - let's close that out tomorrow | 11:26 |
| rlandy|ruck | finish the discussion | 11:26 |
| rlandy|ruck | frenzy_friday: https://review.opendev.org/c/opendev/elastic-recheck/+/805638 - ready for review? | 11:27 |
| rlandy|ruck | zuul -1 on that | 11:27 |
| frenzy_friday | rlandy|ruck, yes, fixing the tox and addressing a comment from Sagi | 11:28 |
| *** pojadhav|brb is now known as pojadhav | 11:29 | |
| rlandy|ruck | frenzy_friday: k - np- just checking | 11:31 |
| *** ysandeep is now known as ysandeep|afk | 11:31 | |
| marios | akahat: i think you misunderstood my comment the rotate 60 means that after it is rotated 60 times it will be removed | 11:33 |
| marios | akahat: at least according to the docs | 11:33 |
| * marios food biab | 11:35 | |
| rlandy|ruck | arxcruz|rover: hey - do you have a patch for https://trello.com/c/ptIv41kA/2139-cixlp1946659tripleociproa-update-upgrades-jobs-are-failing-in-check-gate-after-promotions-not-found-image | 11:37 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1946659 | 11:37 |
| rlandy|ruck | to report to CIX | 11:37 |
| arxcruz|rover | rlandy|ruck: yes, just a sec | 11:37 |
| rlandy|ruck | chandankumar: can we close this out? https://trello.com/c/uxNdFLUp/2134-cixlp1946461tripleociproa-centos-stream-9-missing-packages-tracker-for-container-build | 11:37 |
| chandankumar | rlandy|ruck: nope, we need to wait for qrouted and collectd container | 11:38 |
| akahat | marios, okay. i got your point. I think we don't need maxage. | 11:38 |
| Tengu | cool. just that molecule thing, and we're good with my selinux consolidation script for sealert! | 11:39 |
| arxcruz|rover | rlandy|ruck: https://review.opendev.org/c/openstack/tripleo-ci/+/813629 i update the card on cix | 11:39 |
| akahat | marios, so we should reduce the value to 15 ? | 11:39 |
| rlandy|ruck | arxcruz|rover: great - thank you | 11:39 |
| marios | akahat: yeah maybe, or even 30 i guess we should discuss that but 60 days seems high | 11:56 |
| sshnaidm | arxcruz|rover, do we still use tempest-sendmail.tripleo.org ? | 11:59 |
| arxcruz|rover | sshnaidm: nope | 11:59 |
| soniya29 | chandankumar, kopecmartin, rlandy|ruck, ysandeep|afk, tempest meeting? | 12:00 |
| *** ysandeep|afk is now known as ysandeep | 12:01 | |
| soniya29 | chandankumar, ^^ | 12:01 |
| *** amoralej is now known as amoralej|lunch | 12:02 | |
| rlandy|ruck | soniya29: ^^ on program call | 12:02 |
| bhagyashris|rover | dviroel, hey i have updated the invite by providing the miro board link | 12:04 |
| dviroel | bhagyashris|rover: ah, thanks :) | 12:04 |
| marios | lol dviroel i love the happy/sad icons for the retro | 12:14 |
| marios | :D | 12:14 |
| dviroel | marios: \o/ | 12:15 |
| soniya29 | rlandy|ruck, no problem | 12:15 |
| rlandy|ruck | Check Default IPv4 Gateway availability | overcloud-controller-0 | error={"changed": false, "cmd": ["ping", "-w", "10", "-c", "1", "10.0.0.1"], "delta": "0:00:03.078052", "end": "2021-10-13 10:32:10.263464", "msg": "non-zero return code", "rc": 1, "start": "2021-10-13 10:32:07.185412", "stderr": "", "stderr_lines": [], "stdout": "PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.\nFrom 10.0.0.48 icmp_seq=1 Destination Host | 12:17 |
| rlandy|ruck | Unreachable\nFrom 10.0.0.48 icmp_seq=2 Destination Host Unreachable\nFrom 10.0.0.48 icmp_seq=3 Destination Host Unreachable\n\n--- 10.0.0.1 ping statistics ---\n3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2047ms\npipe 3", "stdout_lines": ["PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.", "From 10.0.0.48 icmp_seq=1 Destination Host Unreachable", "From 10.0.0.48 icmp_seq=2 Destination Host | 12:17 |
| rlandy|ruck | Unreachable", "From 10.0.0.48 icmp_seq=3 Destination Host Unreachable", "", "--- 10.0.0.1 ping statistics ---", "3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2047ms", "pipe 3"]} | 12:17 |
| rlandy|ruck | oops | 12:17 |
| rlandy|ruck | bhagyashris|rover: arxcruz|rover: ^^ fyi - OVB failures | 12:17 |
| rlandy|ruck | trying rerun to check | 12:19 |
| bhagyashris|rover | rlandy|ruck, ack | 12:19 |
| rlandy|ruck | ysandeep: akahat: merging https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/281644 | 12:20 |
| akahat | rlandy|ruck, ack | 12:20 |
| *** jpena|lunch is now known as jpena | 12:24 | |
| rlandy|ruck | ysandeep: chandankumar: https://review.rdoproject.org/r/c/rdo-jobs/+/35912 | 12:24 |
| rlandy|ruck | going to merge that | 12:24 |
| rlandy|ruck | ok to go? | 12:24 |
| ysandeep | rlandy|ruck, yes | 12:24 |
| ysandeep | I already voted there, Only have +1 on rdo-jobs repo | 12:25 |
| dviroel | akahat: bhagyashris|rover arxcruz|rover frenzy_friday marios rlandy|ruck soniya29 chandankumar pojadhav sshnaidm ysandeep retro meeting will start in 5 min | 12:25 |
| arxcruz|rover | dviroel: still stucked in the program call meeting | 12:25 |
| arxcruz|rover | i'll be late | 12:25 |
| rlandy|ruck | yep | 12:26 |
| dviroel | ok | 12:26 |
| rlandy|ruck | same | 12:26 |
| chandankumar | rlandy|ruck: good to go | 12:26 |
| rlandy|ruck | arxcruz|rover: you reporting on the call? | 12:26 |
| arxcruz|rover | rlandy|ruck: yes | 12:26 |
| dviroel | board link: https://miro.com/app/board/o9J_lrP1x4k=/?invite_link_id=182833317619 | 12:26 |
| rlandy|ruck | arxcruz|rover: dropped call - you can as well | 12:28 |
| rlandy|ruck | dviroel; very cute board | 12:28 |
| dviroel | :) | 12:28 |
| *** ykarel__ is now known as ykarel | 12:37 | |
| arxcruz|rover | dviroel: bota ai o sertanejo | 12:41 |
| arxcruz|rover | xD | 12:42 |
| dviroel | lolz | 12:42 |
| *** amoralej|lunch is now known as amoralej | 13:04 | |
| pojadhav | Review Request : https://review.rdoproject.org/r/c/rdo-jobs/+/36134 and https://review.rdoproject.org/r/c/rdo-jobs/+/36133 | 13:12 |
| chandankumar | dviroel: thank you for making awesome retro :-) | 13:32 |
| *** pojadhav is now known as pojadhav|ruck | 13:32 | |
| dviroel | chandankumar: \o/ thank, and thank you all for joining and participating | 13:33 |
| *** dviroel is now known as dviroel|rover | 13:33 | |
| bhagyashris|rover | dviroel, thanks for the retro :) | 13:33 |
| frenzy_friday | rlandy|ruck, arxcruz|rover can we pls merge https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/813791/ ? Hope it fixes the collect logs failure | 13:34 |
| pojadhav|ruck | bhagyashris|rover, arxcruz|rover, rlandy|ruck : when we are planning for RR hand off.. ? | 13:34 |
| bhagyashris|rover | dviroel|rover, pojadhav|ruck tell your free time... | 13:34 |
| rlandy|ruck | marios: ysandeep: forwarding the doodle to you | 13:34 |
| rlandy|ruck | if you want to join | 13:34 |
| bhagyashris|rover | it would be great if we do today | 13:35 |
| dviroel|rover | bhagyashris|rover: i'm available in the next 90 min, after that lunch | 13:35 |
| ysandeep | rlandy|ruck, thanks! | 13:36 |
| bhagyashris|rover | dviroel|rover, ack | 13:36 |
| bhagyashris|rover | rlandy|ruck, dviroel|rover arxcruz|rover pojadhav|ruck fyi i will take day off tomorrow for some personal work... but i will be available for planning meeting | 13:37 |
| *** arxcruz|rover is now known as arxcruz | 13:37 | |
| bhagyashris|rover | so that's why it would be great if we do it today | 13:37 |
| arxcruz | bhagyashris|rover: you're the boss, just say the time :) | 13:38 |
| rlandy|ruck | dviroel|rover: will touch base with you this afternoon to start ruck/rover | 13:38 |
| dviroel|rover | rlandy|ruck: ok | 13:38 |
| rlandy|ruck | pojadhav|ruck: pls touch base with arxcruz | 13:38 |
| rlandy|ruck | as what to pick up | 13:38 |
| pojadhav|ruck | rlandy|ruck, ack | 13:39 |
| bhagyashris|rover | rlandy|ruck, or else we will meet now and in case if we missed some thing then dviroel|rover will ask you | 13:39 |
| *** bhagyashris|rover is now known as bhagyashris | 13:39 | |
| rlandy|ruck | arxcruz: are you done with the updates/upgrades hash work or you want to hand that on? | 13:39 |
| rlandy|ruck | bhagyashris: I am on prod chain - feel free to meet with other ruck/rovers now w/o me | 13:40 |
| arxcruz | rlandy|ruck: i'm still testing, the job variable wasn't available, not sure if it's because it fails on the content-provider job or not | 13:40 |
| bhagyashris | rlandy|ruck, sure thanks | 13:40 |
| arxcruz | rlandy|ruck: i would like to continue with that at least until monday | 13:40 |
| rlandy|ruck | arxcruz: k- pls do | 13:40 |
| bhagyashris | pojadhav|ruck, dviroel|rover arxcruz let's meet https://meet.google.com/xnf-tvdh-pmk?authuser=0 | 13:40 |
| bhagyashris | pojadhav|ruck, dviroel|rover https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ansible-centos-8-molecule-tripleo_sshd | 13:46 |
| Tengu | tosky, marios still working a bit on the regexp - I have to make it a bit more lose, the content of an AVC line might change and include (or not) some details. | 13:58 |
| marios | Tengu: thanks i have it in my list for tomorrow morning reviews | 14:00 |
| Tengu | marios: trying to get it stable before my long weekend | 14:01 |
| Tengu | there's a good job example directly in the project CI, just perfect. | 14:02 |
| rlandy|ruck | bhagyashris: hey - just following up on https://review.opendev.org/c/zuul/zuul-jobs/+/813749 | 14:11 |
| rlandy|ruck | bhagyashris: have you pinged anyone in openstack-infra to review it? | 14:12 |
| rlandy|ruck | dviroel|rover: pojadhav|ruck: you all set? | 14:22 |
| dviroel|rover | rlandy|ruck: yep, w | 14:25 |
| dviroel|rover | will follow up next cix call | 14:25 |
| Tengu | frenzy_friday: is there anything to do/wait now that the "no valid host" patch merged? | 14:27 |
| Tengu | frenzy_friday: I apparently still hit it now: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c9f/813777/5/check/openstack-tox-molecule/c9f6a8d/tox/reports.html - but maybe there's some automated things that will kick later? | 14:27 |
| rlandy|ruck | ok | 14:27 |
| rlandy|ruck | dviroel|rover: just testproject'ed a bunch of failing promotion tests | 14:28 |
| rlandy|ruck | all except master | 14:28 |
| rlandy|ruck | OVB was a bit funky - we need to keep an eye on that | 14:28 |
| dviroel|rover | met me check master then | 14:29 |
| dviroel|rover | ack | 14:29 |
| pojadhav|ruck | rlandy|ruck, yup | 14:48 |
| pojadhav|ruck | rlandy|ruck, leaving for the day, pls left some notes for me if I need to keep eye on anything. will catch up from tomorrow morning. | 14:51 |
| marios | chandankumar: rlandy|ruck: ** tripleo-ci please add to reviews list https://review.opendev.org/c/openstack/tripleo-ci/+/808177 | 14:52 |
| ysandeep | marios, ack | 14:54 |
| *** ysandeep is now known as ysandeep|dinner | 14:55 | |
| frenzy_friday | Tengu, hm.. looks like something else is wrong in the sova file. It should have worked right after merging. Lemme check | 14:58 |
| * dviroel|rover lunch, brbr | 15:01 | |
| chandankumar | dviroel|rover: rlandy|ruck please get these two merged https://review.opendev.org/c/openstack/tripleo-common/+/800580/68 and https://review.opendev.org/c/openstack/tripleo-common/+/813745 merged | 15:05 |
| chandankumar | marios: may be tomorrow | 15:06 |
| chandankumar | see ya! | 15:06 |
| marios | chandankumar: ack adding to list | 15:08 |
| *** ykarel is now known as ykarel|away | 15:12 | |
| bhagyashris | rlandy|ruck, nope i have just added the infra core in the reviewers list | 15:19 |
| rlandy|ruck | bhagyashris: np - I pinged fungi | 15:20 |
| bhagyashris | rlandy|ruck, ack thanks | 15:20 |
| rlandy|ruck | marios: is https://review.opendev.org/c/openstack/diskimage-builder/+/806819 ready for merge again? | 15:21 |
| rlandy|ruck | if so, will ping ianw and stevebaker | 15:22 |
| rlandy|ruck | when they come on line | 15:22 |
| marios | rlandy|ruck: looking | 15:24 |
| marios | ah i didn't revisit after this morning | 15:24 |
| marios | rlandy|ruck: job is failing i men the one that ianw wanted to check | 15:24 |
| rlandy|ruck | marios: no worries - it's just a depends_on on the patch you wanted merge | 15:25 |
| rlandy|ruck | k | 15:25 |
| marios | rlandy|ruck: right | 15:25 |
| rlandy|ruck | marios: story of our lives, right :) | 15:25 |
| marios | rlandy|ruck: ack i'll check it again in the morning see whats up | 15:26 |
| rlandy|ruck | marios: cool - thanks | 15:26 |
| frenzy_friday | Tengu, found the issue: The file it looks for is generated by sova as "_No_valid_host_was_found__pip_conflicting_dependencies.log" This is probably because in the files that sova looks into (https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/roles/collect_logs/tasks/sova.yml#L17-L21) there is indeed an error related to pip | 15:31 |
| *** marios is now known as marios|out | 15:33 | |
| frenzy_friday | ^ looking for a way to solve it https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/813853 | 15:40 |
| rlandy|ruck | ysandeep|dinner: chandankumar: fyi ... breakages on downstream with new container build code https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-16.2/e12144f/logs/build.log | 15:46 |
| rlandy|ruck | openstack tripleo container image build: error: unrecognized arguments: --tcib-extra tcib_release=8 --tcib-extra tcib_python_version=3.6 | 15:46 |
| rlandy|ruck | best fix there? | 15:46 |
| rlandy|ruck | dviroel|rover: pojadhav|ruck: ^^ fyi | 15:47 |
| rlandy|ruck | wallaby still failing network provision | 15:50 |
| rlandy|ruck | Network configuration file does not exist: /usr/share/openstack-tripleo-heat-templates/network-data-samples/default-network-isolation.yaml | 15:50 |
| rlandy|ruck | jenkins jobs | 15:50 |
| frenzy_friday | rlandy|ruck, http://ci-health.tripleo.org/ | 16:07 |
| frenzy_friday | http://ci-health.tripleo.org/:5001 | 16:07 |
| *** ysandeep|dinner is now known as ysandeep | 16:10 | |
| ysandeep | rlandy|ruck, reading back | 16:10 |
| rlandy|ruck | in meeting | 16:10 |
| rlandy|ruck | pls see logs | 16:10 |
| * ysandeep looking | 16:13 | |
| ysandeep | rlandy|ruck: proposing fix | 16:17 |
| *** amoralej is now known as amoralej|off | 16:22 | |
| ysandeep | chandankumar, rlandy|ruck fix is up: https://review.opendev.org/c/openstack/tripleo-ci/+/813863 | 16:28 |
| ysandeep | i had tested 17 container build with chandan patch here.. https://code.engineering.redhat.com/gerrit/c/testproject/+/211643 .. I should have included 16.2 as well in my test | 16:29 |
| Tengu | frenzy_friday: ah cool, hopefully things will be good soon then :) | 16:30 |
| *** jpena is now known as jpena|off | 16:30 | |
| ysandeep | rlandy|ruck, when you get free, what's the issue in wallaby, which job is failing? | 16:31 |
| rlandy|ruck | ysandeep: Network configuration file does not exist: /usr/share/openstack-tripleo-heat-templates/network-data-samples/default-network-isolation.yaml | 16:31 |
| rlandy|ruck | will ping when off meeting | 16:31 |
| ysandeep | ack | 16:31 |
| * ysandeep running testproject in 16.2 in the meantime to test the fix | 16:31 | |
| rlandy|ruck | ysandeep: hey - off meeting | 16:37 |
| ysandeep | rlandy|ruck, which job failing for wallaby, minimal? | 16:37 |
| rlandy|ruck | ysandeep: ack | 16:38 |
| rlandy|ruck | I haven't checked yet if master is failing the same way | 16:38 |
| rlandy|ruck | looking now | 16:38 |
| rlandy|ruck | + /tmp/workspace/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/bin/cico node get --arch x86_64 --release 8-stream --count 1 --retry-count 6 --retry-interval 60 -f csv | 16:40 |
| rlandy|ruck | + sed 1d | 16:40 |
| rlandy|ruck | string indices must be integers | 16:40 |
| rlandy|ruck | master may be infra failure | 16:40 |
| rlandy|ruck | + echo 'FATAL: no nodes were provisioned' | 16:41 |
| rlandy|ruck | FATAL: no nodes were provisioned | 16:41 |
| rlandy|ruck | + exit 1 | 16:41 |
| ysandeep | rlandy|ruck, testing 16.2 container build fix here: https://code.engineering.redhat.com/gerrit/c/testproject/+/190672 | 16:43 |
| rlandy|ruck | ysandeep: master looks like infra stuff | 16:43 |
| * ysandeep looking at minimal jobs now | 16:43 | |
| rlandy|ruck | pinged on rdo | 16:43 |
| rlandy|ruck | ysandeep: for minimal ... | 16:44 |
| rlandy|ruck | https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/network-data-samples/default-network-isolation.yaml | 16:45 |
| rlandy|ruck | should be there | 16:45 |
| rlandy|ruck | https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-wallaby-current-tripleo-delorean-minimal-60/undercloud/usr/share/openstack-tripleo-heat-templates/ | 16:46 |
| rlandy|ruck | ha | 16:46 |
| rlandy|ruck | not | 16:46 |
| rlandy|ruck | ysandeep: ^^ not there | 16:46 |
| rlandy|ruck | correct | 16:46 |
| ysandeep | rlandy|ruck, should be there: https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/network-data-samples/default-network-isolation.yaml | 16:47 |
| rlandy|ruck | ysandeep: not installed | 16:48 |
| ysandeep | let me check what version of tht is insalled there | 16:48 |
| rlandy|ruck | ysandeep: installed in master jobs | 16:52 |
| rlandy|ruck | https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-50/undercloud/usr/share/openstack-tripleo-heat-templates/ | 16:52 |
| ysandeep | installed tht on wallaby is quite recent: nstack-tripleo-heat-templates.noarch 14.3.1-0.20211010015303.f556f24.el8 @delorean-component-tripleo | 16:53 |
| ysandeep | openstack-tripleo-heat-templates.noarch 14.3.1-0.20211010015303.f556f24.el8 @delorean-component-tripleo | 16:53 |
| rlandy|ruck | right - but it's not copying that dir | 16:53 |
| ysandeep | yeah... and patch was merged in march: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/765218 | 16:54 |
| ysandeep | let me compare that dir from one of our wallaby job | 16:55 |
| rlandy|ruck | ysandeep: we don't collect /usr/share there :( | 16:56 |
| rlandy|ruck | I looked | 16:57 |
| rlandy|ruck | ysandeep: whatever is doing the copying is not copying that folder | 16:57 |
| rlandy|ruck | ysandeep: creating bug - need to roll to next meeting | 16:58 |
| ysandeep | rlandy|ruck, let me troubleshoot for a while.. i will write a bug if i cannot figure out | 16:58 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1947015 | 17:03 |
| rlandy|ruck | ysandeep: ^^ to track | 17:03 |
| ysandeep | rlandy|ruck, thanks! I extracted the rpm locally.. indeed that dir is not present in rpm itself | 17:03 |
| ysandeep | rlandy|ruck, fixed merged in wallaby 2 days ago https://review.rdoproject.org/r/c/openstack/tripleo-heat-templates-distgit/+/35974 | 17:18 |
| ysandeep | rpm distgit issue... that directory was not packaged | 17:18 |
| ysandeep | confirmed issue is now resolved in latest tht rpm(tripleo component currently at consistent hash) | 17:22 |
| rlandy|ruck | ysandeep: nice | 17:25 |
| rlandy|ruck | we should that resolved with next run then | 17:25 |
| ysandeep | yes | 17:25 |
| dviroel|rover | nice debug :) | 17:29 |
| rlandy|ruck | dviroel|rover: leaving https://bugs.launchpad.net/tripleo/+bug/1947015 open | 17:31 |
| rlandy|ruck | until we see a clean run | 17:31 |
| dviroel|rover | ack | 17:31 |
| rlandy|ruck | I didn't mark it promotion-blocker | 17:32 |
| rlandy|ruck | as it may resolve in next run | 17:32 |
| rlandy|ruck | dviroel|rover: https://review.opendev.org/c/zuul/zuul-jobs/+/813749 got w+'ed so that's good | 17:50 |
| rlandy|ruck | we can revert the patch to make the c7 jobs non-voting after that merges | 17:50 |
| dviroel|rover | great | 17:52 |
| rlandy|ruck | dviroel|rover: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/782362 | 17:52 |
| rlandy|ruck | ^^ we need to get that working | 17:52 |
| rlandy|ruck | for the content-provider failures pulling non-tripleo containers | 17:53 |
| * dviroel|rover looking | 17:56 | |
| * ysandeep out for the day, see you tomorrow o/ | 18:18 | |
| *** ysandeep is now known as ysandeep|out | 18:19 | |
| * rlandy|ruck lunch - brb | 18:19 | |
| dviroel|rover | rlandy|ruck: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/35975 | 19:18 |
| rlandy|ruck | dviroel|rover: nice - merging | 19:18 |
| dviroel|rover | rlandy|ruck: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train | 19:27 |
| dviroel|rover | failed twice in "openstack overcloud node introspect" | 19:27 |
| dviroel|rover | but in the end, introspection was sucess https://logserver.rdoproject.org/95/24995/118/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/c61f37b/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz | 19:28 |
| rlandy|ruck | https://logserver.rdoproject.org/95/24995/118/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/c61f37b/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz | 19:29 |
| dviroel|rover | job that passes https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/a3f0078/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz | 19:29 |
| rlandy|ruck | Introspection of node completed:de85b2cd-8ba3-4769-a76f-76466b612142. Status:FAILED. Errors:Introspection timeout | 19:31 |
| rlandy|ruck | right ^^ | 19:31 |
| rlandy|ruck | so - you're asking if we should write that up yet as a bug? | 19:32 |
| rlandy|ruck | or debug/fix it? | 19:32 |
| rlandy|ruck | dviroel|rover: ^^? | 19:32 |
| rlandy|ruck | it's running again now in testproject | 19:33 |
| rlandy|ruck | and in the line | 19:33 |
| rlandy|ruck | if either of them fails, yes | 19:33 |
| dviroel|rover | yes, if we should give another try | 19:33 |
| dviroel|rover | yeah, there is an extra 'timeout' error in the failing job, we should try again | 19:35 |
| rlandy|ruck | dviroel|rover: yeah if current runs fail - we'll bug and investigate | 19:52 |
| rlandy|ruck | dviroel|rover: pls vote on https://review.opendev.org/c/openstack/tripleo-ci/+/813863 | 20:25 |
| rlandy|ruck | and then we can merge that | 20:25 |
| dviroel|rover | done | 20:37 |
| dviroel|rover | rlandy|ruck: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master | 20:49 |
| dviroel|rover | lots of tempest tests failing | 20:50 |
| rlandy|ruck | one sec - just resubmitting patch | 20:51 |
| dviroel|rover | ah, mysqld daemon is restarting | 20:52 |
| dviroel|rover | https://logserver.rdoproject.org/32/35132/11/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/94ee759/logs/undercloud/var/log/containers/mysql/mysqld.log.txt.gz | 20:54 |
| dviroel|rover | 2021-10-13 16:36:02 3038 [ERROR] InnoDB: WSREP: referenced FK check fail: Lock wait index `PRIMARY` table `ovs_neutron`.`securitygroupportbindings` | 20:54 |
| dviroel|rover | 211013 16:42:17 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql | 20:54 |
| rlandy|ruck | dviroel|rover: looking | 20:55 |
| dviroel|rover | https://logserver.rdoproject.org/32/35132/11/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/94ee759/logs/undercloud/var/log/containers/keystone/keystone.log.txt.gz | 20:56 |
| dviroel|rover | 2021-10-13 16:40:11.186 210 ERROR keystone.server.flask.request_processing.middleware.auth_context [req-17ded4ab-4d62-46a8-b09f-6a83afa5fe9a 8a0a462af25248b99f6b44cf33b2c205 08812bde6bc74d3ba52eb6d5ff8b46ef - default default] (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') | 20:56 |
| rlandy|ruck | dviroel|rover: oh wow - we got some failures there | 20:57 |
| rlandy|ruck | yep - let's get a bug on that | 20:57 |
| dviroel|rover | \o/ | 20:57 |
| rlandy|ruck | you can mark it promotion-blocker | 20:58 |
| dviroel|rover | ack | 20:58 |
| rlandy|ruck | we'll close it if the current run does not show that | 20:58 |
| rlandy|ruck | set up issue | 20:58 |
| * rlandy|ruck checking previous runs | 20:59 | |
| rlandy|ruck | hmmm ... | 21:00 |
| rlandy|ruck | checking if we have diff tests failing here | 21:00 |
| rlandy|ruck | dviroel|rover: so the interesting thing here is .... | 21:01 |
| rlandy|ruck | we have diff setup tests failing each time: | 21:02 |
| rlandy|ruck | setUpClass (tempest.api.compute.servers.test_instance_actions | 21:02 |
| rlandy|ruck | setUpClass (keystone_tempest_plugin.tests.api.identity.v3.test_mapping_rules | 21:02 |
| rlandy|ruck | setUpClass (cinder_tempest_plugin.api.volume.test_create_from_image | 21:02 |
| rlandy|ruck | in each of the three failing runs | 21:02 |
| rlandy|ruck | never the less, let's get a tracking bug on this | 21:03 |
| dviroel|rover | rlandy|ruck: https://logserver.rdoproject.org/32/35132/11/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/94ee759/logs/undercloud/var/log/containers/mysql/mysqld.log.txt.gz | 21:03 |
| dviroel|rover | i think that mysqld error are the same | 21:03 |
| dviroel|rover | neutron related | 21:03 |
| rlandy|ruck | yeah possible it hits a different test | 21:04 |
| * dviroel|rover not sure if I checked all 3 jobs | 21:04 | |
| rlandy|ruck | either way | 21:04 |
| dviroel|rover | we need new milestones in LP | 21:10 |
| dviroel|rover | rlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/1947050 | 21:13 |
| rlandy|ruck | dviroel|rover: ack - you are right | 21:14 |
| rlandy|ruck | asking slagle | 21:14 |
| * dviroel|rover going out in min, but will be around | 21:19 | |
| rlandy|ruck | dviroel|rover; nice catch! | 21:33 |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 22:28 | |
| -opendevstatus- NOTICE: Both Gerrit and Zuul services are being restarted briefly for minor updates, and should return to service momentarily; all previously running builds will be reenqueued once Zuul is fully started again | 22:49 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!