| *** rlandy is now known as rlandy|out | 01:27 | |
| *** akahat is now known as akahat|ruck | 04:37 | |
| akahat|ruck | Good morning! O/ | 04:39 |
|---|---|---|
| *** ysandeep|away is now known as ysandeep | 04:49 | |
| ysandeep | akahat|ruck, good morning o/ | 04:50 |
| bhagyashris | akahat|ruck, Hi, | 05:37 |
| marios | morning bhagyashris akahat|ruck o/ | 05:45 |
| bhagyashris | o/ | 05:53 |
| marios | akahat|ruck: when you have time in a bit lets do a quick sync call please re ruck|rover stuffs | 05:56 |
| akahat|ruck | marios, we can sync now. | 05:59 |
| marios | akahat|ruck: k sec plugging headphones | 06:00 |
| akahat|ruck | marios, https://meet.google.com/cyb-sbxd-aga | 06:00 |
| jm1 | good morning folks :) | 06:40 |
| *** amoralej|off is now known as amoralej | 06:53 | |
| jm1 | rlandy|out, akahat|ruck: we had a c9 master promotion yesterday 🥳 looks like rlandy and promotion enforcer are a great team 😁 | 07:02 |
| akahat|ruck | marios, bhagyashris all the internal jobs going to fail.. because of certificate expire. New certificate needs to install in the server.. looks like we need to create CIx for It team. | 07:07 |
| akahat|ruck | re: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-periodic-integration-rhos-17.1-rhel9&skip=0 | 07:07 |
| marios | akahat|ruck: already expired :/ 2022-09-18 08:44:45.673439 | primary | 2022-09-18 08:44:45 | curl: (60) SSL certificate problem: certificate has expired | 07:12 |
| marios | akahat|ruck: ack so that's 2 cix for d/stream one for RETRY and the other for this cert issue | 07:12 |
| akahat|ruck | yes. I've posted issue on #rhos-ops | 07:12 |
| marios | thnks akahat|ruck | 07:12 |
| akahat|ruck | marios, we are seeing certificate issue in retry / retry_limit | 07:13 |
| akahat|ruck | jobs * | 07:13 |
| marios | akahat|ruck: do you have logs that bhagyashris passed from friday? check if they are the same cert issue | 07:24 |
| marios | akahat|ruck: otherwise you'll have 2 different ones | 07:24 |
| marios | akahat|ruck: (from friday re RETRY/dstream issue) | 07:24 |
| marios | akahat|ruck: they are 2 different issues | 07:25 |
| akahat|ruck | marios, above link is same which shared by bhagyashris. | 07:25 |
| akahat|ruck | aoh. | 07:25 |
| marios | akahat|ruck: see https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/6d417b38ea624c63a3668cf763c2c25c RETRY from friday | 07:25 |
| marios | akahat|ruck: 2022-09-16 13:39:24.173529 | primary | "msg": "Stack baremetal_84070 did not deploy successfully. See the stack status message above." | 07:26 |
| marios | akahat|ruck: so cert one for sure and may be another issue on those RETRY so please chekc | 07:26 |
| akahat|ruck | marios, yes.. thanks for the above logs. | 07:27 |
| *** jpena|off is now known as jpena | 07:34 | |
| *** ysandeep is now known as ysandeep|lunch | 08:22 | |
| akahat|ruck | marios, hey.. i think we don't need to create CIX for the stack creation failure. Those failed jobs re-run and passed. | 08:39 |
| akahat|ruck | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-rhos-17.1&project=openstack/tripleo-ci | 08:39 |
| akahat|ruck | this is reported bug: https://bugzilla.redhat.com/show_bug.cgi?id=2127840 | 08:40 |
| akahat|ruck | bhagyashris, ^ | 08:40 |
| bhagyashris | akahat|ruck, ack | 08:41 |
| marios | akahat|ruck: k thanks for checking | 08:56 |
| *** ysandeep|lunch is now known as ysandeep | 09:00 | |
| marios | akahat|ruck: see pvt re the cert issue for workaround | 09:05 |
| arxcruz | marios akahat|ruck ysandeep https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/858266 please review, it's a promotion blocker :) | 09:18 |
| marios | arxcruz: ack will check | 09:26 |
| akahat|ruck | arxcruz, thanks. | 09:26 |
| akahat|ruck | arxcruz, testing here: https://review.rdoproject.org/r/c/testproject/+/41469 | 09:26 |
| arxcruz | akahat|ruck no need to test, if you run this: tempest-skip list-allowed --file roles/validate-tempest/vars/tempest_allow.yml --release train --group featureset062_periodic --job periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train | 09:27 |
| arxcruz | you will see that without the patch, it returns empty | 09:27 |
| arxcruz | if you run with the patch, it will return the tests | 09:27 |
| arxcruz | that was the error on the scen010 job | 09:27 |
| arxcruz | it was returning empty | 09:27 |
| akahat|ruck | yup.. it was empty. | 09:28 |
| akahat|ruck | https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/428868 | 09:37 |
| akahat|ruck | marios, arxcruz chandankumar bhagyashris ^^ | 09:37 |
| marios | akahat|ruck: k | 09:45 |
| marios | akahat|ruck: ack but please add TODO comment pointing to the bug we should not keep this | 09:46 |
| akahat|ruck | marios, ack | 09:47 |
| *** pojadhav- is now known as pojadhav | 09:47 | |
| *** pojadhav- is now known as pojadhav | 09:54 | |
| *** rlandy|out is now known as rlandy|rover | 10:20 | |
| rlandy|rover | akahat|ruck: marios: will sync in a few | 10:23 |
| rlandy|rover | amoralej: hi - I added https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/858209 | 10:23 |
| rlandy|rover | but the mirror issue starts before extras gets into the picture | 10:24 |
| amoralej | i think syntax is wrong | 10:25 |
| amoralej | lemme check | 10:25 |
| rlandy|rover | amoralej: does the var need to be added here: https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/base/pre.yaml | 10:25 |
| marios | rlandy|rover: o/ | 10:25 |
| rlandy|rover | akahat|ruck: let's sync ... marios ... you can join is you can | 10:26 |
| rlandy|rover | https://meet.google.com/bjh-yjyu-ybi?pli=1&authuser=0 | 10:26 |
| *** anbanerj is now known as frenzyfriday | 10:26 | |
| rlandy|rover | https://hackmd.io/a92u74hDQhe5afE_LUtbRw has no new info | 10:26 |
| amoralej | rlandy|rover, i left a comment | 10:29 |
| rlandy|rover | amoralej: ok - I see - but I think this is too late | 10:30 |
| rlandy|rover | it won;t fix anything earlier | 10:30 |
| rlandy|rover | ie: the failure is before this | 10:30 |
| rlandy|rover | in https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/base/pre.yaml | 10:30 |
| rlandy|rover | do we not need the var there???? | 10:30 |
| amoralej | ah, correct | 10:30 |
| amoralej | it should be in base job | 10:30 |
| amoralej | i think | 10:30 |
| amoralej | but i think not in the playbook | 10:31 |
| amoralej | but in the job variables? | 10:31 |
| * ysandeep stepping out for sometime | 10:33 | |
| *** ysandeep is now known as ysandeep|afk | 10:34 | |
| rlandy|rover | amoralej: then is would have to be where this is called | 10:37 |
| marios | rlandy|rover: https://bugzilla.redhat.com/show_bug.cgi?id=2127828 | 10:38 |
| rlandy|rover | ykarel: are you still hitting https://bugs.launchpad.net/tripleo/+bug/1989606? | 10:51 |
| ykarel | rlandy|rover, i have not checked new logs and atleast i don't see the same in my patches | 10:52 |
| rlandy|rover | ok - thanks | 10:52 |
| ykarel | rlandy|rover, do we have logstash having logs from rdo jobs? | 10:52 |
| ykarel | if yes then can check there | 10:52 |
| rlandy|rover | frenzyfriday: ^^ | 10:52 |
| frenzyfriday | checking | 10:53 |
| amoralej | rlandy|rover, what i meant is to add the vars in parent jobs as in https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/multinode-jobs.yaml#L49-L56 | 11:07 |
| akahat|ruck | rlandy|rover, https://review.rdoproject.org/r/c/testproject/+/41469 | 11:08 |
| frenzyfriday | rlandy|rover, nope, I do not see it in logstash https://review.rdoproject.org/analytics/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15d,to:now))&_a=(columns:!(_source),filters:!(),index:logstash,interval:auto,query:(language:kuery,query:'build_name:%22tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001%22%20AND%20build_status:%22FAILURE%22%20AND%20message:%22that%20name%20is%20alre | 11:09 |
| frenzyfriday | ady%20in%20use%22'),sort:!()) The job is tracked though | 11:09 |
| frenzyfriday | lemme check with Daniel | 11:10 |
| marios | anyone for reviews | 11:16 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 11:16 |
| rlandy|rover | amoralej: ok - trying that | 11:35 |
| jm1 | rlandy|rover: no need to recheck rrr jobs, they will be rerun after max. 30-45 minutes ;) | 11:39 |
| rlandy|rover | jm1: automatically? | 11:41 |
| jm1 | rlandy|rover: yes | 11:41 |
| rlandy|rover | jm1: can we talk about this? | 11:41 |
| rlandy|rover | some jobs are rerunning and we know they won't pass | 11:41 |
| rlandy|rover | like the train kvm job | 11:41 |
| rlandy|rover | it will only pass now | 11:41 |
| rlandy|rover | after fix merges | 11:41 |
| jm1 | rlandy|rover: i can stop rrr if you want | 11:42 |
| rlandy|rover | jm1: pls - lets talk about usage at meeting | 11:42 |
| rlandy|rover | jm1: can be very useful | 11:42 |
| rlandy|rover | but I think let's just see if it's the right thing for all cases | 11:42 |
| jm1 | rlandy|rover: lets talk about that tomorrow in community meeting? i dont want to bore all others with rrr 🙈 | 11:43 |
| jm1 | rlandy|rover: i will stop it for now | 11:43 |
| rlandy|rover | jm1: I think everyone needs to see it - community call is good | 11:43 |
| rlandy|rover | I think there is definite value -in rerun one - not every job every 45 mins | 11:44 |
| jm1 | rlandy|rover: it is only rerunning failing jobs in criteria which are not running currently. rerunning specific jobs requires another script. maybe worth creating a jira card for that | 11:47 |
| jm1 | rlandy|rover: anyway, stopped promotion enforcer for now. running jobs will finish but no new jobs will be scheduled | 11:48 |
| rlandy|rover | jm1: yeah - so I think there is a lot of potential here - so let's juts talk about cadence and usage etc. | 11:48 |
| rlandy|rover | we are on limited resources | 11:48 |
| rlandy|rover | so we need to balance this | 11:48 |
| rlandy|rover | jm1: thank you for working on this - let's see what is going to be the right balance after community call | 11:49 |
| marios | rlandy|rover: akahat|ruck: fyi https://review.opendev.org/c/openstack/tripleo-ci/+/857778/2#message-9812d7397af7eb57810490ab60ddc0cd58a98c90 seems related to https://lists.openstack.org/pipermail/openstack-discuss/2022-September/030505.html - added fix in v2 https://review.opendev.org/c/openstack/tripleo-ci/+/857778/2#message-6dbc8e38a9f343e7e55e299f9d62fd731d341103 | 12:01 |
| marios | akahat|ruck: broken tripleo-ci gate ^^^ but should be fixed with tripleo-ci/+/857778/2 | 12:02 |
| ykarel | frenzyfriday, are the logs collected/stored there and for how many days? | 12:02 |
| ykarel | for ex: i see that issue 5 days ago https://logserver.rdoproject.org/82/857182/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/7ea04b3/logs/undercloud/var/log/extra/logstash.txt.gz | 12:02 |
| ykarel | but not in kibana | 12:02 |
| frenzyfriday | ykarel, 15 days. But the overcloud file was not in the list of collected files. I have a patch for it (for the upstream logstash) - https://review.opendev.org/c/openstack/ci-log-processing/+/858306 | 12:03 |
| *** ysandeep|afk is now known as ysandeep | 12:03 | |
| marios | chandankumar: ysandeep: akahat|ruck: rlandy|rover: please vote when you have time https://review.opendev.org/c/openstack/tripleo-ci/+/857778 | 12:04 |
| ykarel | frenzyfriday, the error is also logstash file logs/undercloud/var/log/extra/logstash.txt and seems that's included in the list you shared above | 12:05 |
| ykarel | https://review.opendev.org/c/openstack/ci-log-processing/+/858306/1/logscraper/download-list.yaml.sample#180 | 12:06 |
| ysandeep | marios, looking | 12:07 |
| frenzyfriday | ykarel, in the rdo for some reason I see only job-output.txt is pulled into logstash. dpawlik is helping to identify where I can add more files (like the #tripleo section in the upstream file) | 12:08 |
| ykarel | frenzyfriday, okk Thanks | 12:08 |
| rlandy|rover | amoralej: to check - they should all be set to true? | 12:08 |
| amoralej | not all | 12:08 |
| amoralej | similar to defaults | 12:09 |
| amoralej | but just change extras-common to true | 12:09 |
| rlandy|rover | k | 12:09 |
| rlandy|rover | amoralej: k - trying with multinode - if that works will add to standalone, and all the others | 12:09 |
| amoralej | ack | 12:09 |
| *** amoralej is now known as amoralej|lunch | 12:24 | |
| soniya | marios, i have updated this patch now - https://review.opendev.org/c/openstack/tripleo-heat-templates/+/852844, can you go through it whenever time permits? | 12:33 |
| akahat|ruck | marios, ack. will take a look. | 12:34 |
| marios | soniya: ack noting | 12:42 |
| soniya | marios, thanks | 12:54 |
| soniya | rlandy|rover, pojadhav, i need to step out for 15-20 mins | 13:11 |
| soniya | i may join scrum a bit late | 13:11 |
| *** soniya is now known as soniya|afk | 13:11 | |
| arxcruz | dpawlik hey, please take a look at https://review.opendev.org/c/openstack/ci-log-processing/+/858373 | 13:27 |
| arxcruz | let's see if we can start to collect these data | 13:27 |
| *** Guest772 is now known as dasm | 13:30 | |
| pojadhav | arxcruz, rlandy|rover, soniya|afk, akahat|ruck, rcastillo|rover, jm1 : scrum | 13:31 |
| chandankumar | hello #oooq, if you make any change in tripleo-ci/zuul.d files you might hit zuul syntx issue | 13:31 |
| chandankumar | fix is here: https://review.opendev.org/c/openstack/tripleo-ci/+/858380 | 13:31 |
| dasm | o/ | 13:31 |
| pojadhav | chandankumar, : scrum | 13:31 |
| dpawlik | arxcruz: it goes in the right direction. Good job | 13:36 |
| arxcruz | dpawlik what are the next steps? | 13:36 |
| arxcruz | i think you need to setup something on the ES side right? | 13:37 |
| -opendevstatus- NOTICE: As of the weekend, Zuul only supports queue declarations at the project level; if expected jobs aren't running, see this announcement: https://lists.opendev.org/pipermail/service-announce/2022-September/000044.html | 13:37 | |
| dpawlik | arxcruz: so it would be good to add a test to the ansible playbook that validate functionality | 13:38 |
| dpawlik | + add the new file to the https://opendev.org/openstack/ci-log-processing/src/branch/master/ansible/roles/logscraper/templates/download-list.yaml.j2 | 13:39 |
| arxcruz | dpawlik the file it's there | 13:39 |
| dpawlik | add a param to this https://opendev.org/openstack/ci-log-processing/src/branch/master/ansible/playbooks/check-services-sender.yml | 13:39 |
| arxcruz | dpawlik can you comment there so lukas can work on that? | 13:40 |
| dpawlik | and maybe simple query to validate that the new index is there https://opendev.org/openstack/ci-log-processing/src/branch/master/ansible/roles/check-services/tasks/download.yml#L113 | 13:40 |
| dpawlik | <need to think, little busy right now> | 13:40 |
| dpawlik | sure | 13:40 |
| dpawlik | arxcruz done | 13:44 |
| marios | thank you pojadhav | 13:45 |
| dasm | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44889 https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44890 | 13:47 |
| marios | akahat|ruck: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44933/1#message-58448ab9a3496e636a696fed983e689489448214 | 14:04 |
| marios | akahat|ruck: we still need to merge that (probably update to have only 9 in criteria for now) per that ^^ | 14:05 |
| marios | akahat|ruck: pojadhav: moving back to ready review so we don't forget https://issues.redhat.com/browse/TRIPLEOCI-1162 | 14:05 |
| akahat|ruck | marios, ack. once lp got resolved we can merge it. | 14:06 |
| rlandy|rover | https://review.rdoproject.org/r/c/config/+/45114 - tested in job in https://review.rdoproject.org/r/c/testproject/+/36255/142/.zuul.yaml | 14:10 |
| rlandy|rover | amoralej|lunch: marios: akahat|ruck: ^^ | 14:10 |
| rlandy|rover | that is a config patch | 14:11 |
| *** amoralej|lunch is now known as amoralej | 14:11 | |
| amoralej | forgot to renick, sorry | 14:11 |
| rlandy|rover | so it's a merge and hope situation | 14:11 |
| marios | - periodic-tripleo-centos-9-zed-component-baremetal-promote-consistent-to-component-ci-testing: &force_periodic | 14:11 |
| marios | pojadhav: https://review.opendev.org/c/openstack/tripleo-ci/+/857778/1#message-9812d7397af7eb57810490ab60ddc0cd58a98c90 | 14:11 |
| *** rcastillo|rover is now known as rcastillo | 14:11 | |
| marios | pojadhav: https://lists.openstack.org/pipermail/openstack-discuss/2022-September/030505.html | 14:12 |
| pojadhav | marios, please have a look when you free https://review.opendev.org/c/openstack/tripleo-ci/+/856051/4#message-7b2f8bf186c78d27d8a521aa5105458c58524b08 | 14:16 |
| pojadhav | marios, https://review.opendev.org/q/topic:tripleo_victoria_eol+status:open | 14:16 |
| marios | k pojadhav | 14:18 |
| chandankumar | marios: rlandy|rover please have a look at this review https://review.opendev.org/c/openstack/tripleo-quickstart/+/858262 when free, thank you :-) | 14:45 |
| Tengu | «retrying is life, retrying is all» | 14:46 |
| marios | chandankumar: ack will do adding to my reviews | 14:51 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 14:51 |
| jm1 | Tengu: oh yeah 🤠 promotion enforcer will safe you 😁 | 14:57 |
| * jm1 bbl | 14:58 | |
| *** ysandeep is now known as ysandeep|dinner | 14:58 | |
| chandankumar | see ya people! | 15:00 |
| dasm | chandankumar: o/ | 15:03 |
| *** marios is now known as marios|out | 15:28 | |
| rlandy|rover | ysandeep|dinner: your thoughts on https://review.opendev.org/c/openstack/tripleo-quickstart/+/858262? | 15:46 |
| rlandy|rover | any concerns about merging that? | 15:46 |
| rlandy|rover | akahat|ruck: hmm so wrt wallaby c9 ... periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-internal-wallaby passed | 15:48 |
| rlandy|rover | if current rerun does not pass ... will change criteria to use internal to promote | 15:48 |
| *** ysandeep|dinner is now known as ysandeep | 16:02 | |
| ysandeep | rlandy|rover, we often hit 502 error in CI - this change lgtm.. | 16:02 |
| ysandeep | voted | 16:02 |
| rlandy|rover | k- let's try it | 16:03 |
| * ysandeep out, see everyone tomorrow | 16:03 | |
| *** ysandeep is now known as ysandeep|out | 16:03 | |
| rlandy|rover | akahat|ruck: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45132 Temp use internal fs001 to promote wallaby | 16:19 |
| rlandy|rover | ugh - master line is not kicking | 16:20 |
| rlandy|rover | lunch brb | 16:24 |
| *** jpena is now known as jpena|off | 16:28 | |
| akahat|ruck | rlandy|rover, that's nice. | 16:31 |
| rlandy|rover | just failed mol-promote_images | 16:31 |
| akahat|ruck | rlandy|rover, voted +1. | 16:35 |
| rlandy|rover | akahat|ruck: thanks - abandoned/restored | 16:36 |
| rlandy|rover | hopefully will pass this time | 16:36 |
| rlandy|rover | akahat|ruck: also testing new namservers | 16:36 |
| dasm | rlandy|rover: o/ afair you mentioned we have a docker hub pro account. am i recalling it right? | 16:36 |
| rlandy|rover | dasm: ack - pls check bitwarden | 16:37 |
| dasm | ack | 16:37 |
| rlandy|rover | I pay $7 a month for that :) | 16:37 |
| dasm | i'm gonna look into using quay in the future, but for now i'm struggling with setting up cockpit :/ it relies on grafana, nginx and mariadb stored at Docker Hub :( | 16:38 |
| * akahat|ruck See you tomorrow.. | 16:51 | |
| rlandy|rover | akahat|ruck: have a good night | 17:02 |
| *** amoralej is now known as amoralej|off | 17:35 | |
| rlandy|rover | akahat|ruck: merged the revert: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/428413 | 18:30 |
| rlandy|rover | also reverting https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44763 | 18:32 |
| rlandy|rover | wallaby promoted | 18:32 |
| * jm1 out for today, have a nice evening #oooq | 19:02 | |
| rcastillo | heading out, I'll bbl | 21:29 |
| * dasm => offline | 22:00 | |
| *** dasm is now known as dasm|off | 22:00 | |
| *** rlandy|rover is now known as rlandy|out | 22:50 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!