| *** dviroel|afk is now known as dviroel | 00:41 | |
| *** dviroel is now known as dviroel|out | 00:50 | |
| *** rlandy is now known as rlandy|out | 01:58 | |
| *** ysandeep|out is now known as ysandeep | 04:42 | |
| ysandeep | good morning ci o/ | 04:45 |
|---|---|---|
| bhagyashris|ruck | good morning all | 04:49 |
| *** ysandeep is now known as ysandeep|brb | 05:02 | |
| *** ysandeep|brb is now known as ysandeep | 05:35 | |
| * bhagyashris|ruck lunch afk | 05:52 | |
| ysandeep | jm1[m], bhagyashris|ruck, I am seeing mirror related issue on periodic jobs is it known? | 06:23 |
| ysandeep | https://logserver.rdoproject.org/54/31954/89/check/periodic-tripleo-ci-centos-9-scenario007-multinode-oooq-container-master/9a2c631/job-output.txt | 06:23 |
| ysandeep | ~~~ | 06:23 |
| ysandeep | 2022-09-13 06:09:53.963695 | primary | Errors during downloading metadata for repository 'extras-common': | 06:23 |
| ysandeep | 2022-09-13 06:09:53.963727 | primary | - Curl error (7): Couldn't connect to server for https://mirrors.centos.org/metalink?repo=centos-extras-sig-extras-common-9-stream&arch=x86_64&protocol=https,http [Failed to connect to mirrors.centos.org port 443: No route to host] | 06:23 |
| ysandeep | ~~~ | 06:23 |
| marios | i saw that at least once yesterday ysandeep (is the job in RETRY fail?) | 06:31 |
| ysandeep | marios, yes | 06:31 |
| marios | ysandeep: you remember we had a card against vexx for that but it was moved done. not sure if this is the same thing or something new now though | 06:33 |
| marios | ysandeep: but not consistent so retry may help you if you are waiting for some specific run | 06:33 |
| marios | tempest failed in the gate but not sure if this is transient yet (unrelated to the above discussion) | 06:35 |
| marios | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_732/854140/1/gate/tripleo-ci-centos-8-standalone/7320f7d/logs/undercloud/var/log/tempest/stestr_results.html | 06:35 |
| ysandeep | marios, thanks! this time failure is during pulling content from mirrors.centos.org, afair.. that card was for pulling content from vexx mirrors, I have faced this issue twice, let me try again | 06:36 |
| marios | ysandeep: right yeah should be a different thing then | 06:37 |
| marios | ysandeep: but since it was seen yesterday and still today then we should have a cix on that | 06:37 |
| ysandeep | found the previous bug: https://bugs.launchpad.net/tripleo/+bug/1983817 , yes that was for vexx mirror.. looks different | 06:38 |
| marios | ack ysandeep thanks for finding | 06:39 |
| ysandeep | marios, thanks! I will let bhagyashris|ruck and jm1[m] report and investigate once they are in. | 06:39 |
| marios | ysandeep: ack sounds good | 06:41 |
| marios | jm1[m]: fyi hopefully transient so just noted them there for now https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854140/1#message-436dc7bef64a98a5462c2703e5fbb0603e2373e4 | 06:45 |
| bhagyashris|ruck | ysandeep, ack | 07:27 |
| abregman|afk | bhagyashris|ruck, ysandeep, mario: hey, can we merge this one? https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/427727 | 07:31 |
| abregman|afk | marios: ^ | 07:31 |
| *** jm1|ruck is now known as jm1|rover | 07:33 | |
| jm1 | moin 😴 | 07:33 |
| marios | abregman|afk: looking | 07:35 |
| *** jpena|off is now known as jpena | 07:37 | |
| chem | marios: which one is the "usual" one, ~/.zuul.yaml or zuul.d/layout.yaml ? | 07:40 |
| chem | marios: hum zuul.d/layout.yaml seems to be the one | 07:41 |
| marios | chem: we have zuul.d/layout in tripleo land | 07:42 |
| marios | chem: but .zuul.yaml is popular elsewhere | 07:42 |
| jm1 | bhagyashris|ruck, marios, rlandy|out: i have moved todays rr notes to a new hackmd because we are are hitting chars limit soon https://hackmd.io/dKeK6zo9R66heikGyCb4NA | 07:56 |
| marios | jm1: ack and thanks for updating the index too | 07:57 |
| *** abregman|afk is now known as abregman | 07:57 | |
| jm1 | bhagyashris|ruck: you are editing the old doc.. | 07:59 |
| abregman | bhagyashris|ruck, ysandeep, jm1, marios: and this one please https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/427877 | 08:05 |
| marios | abregman: looking | 08:15 |
| marios | abregman: lets hold on that for a bit commented | 08:17 |
| marios | please add https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/427877 to the review list | 08:18 |
| marios | frenzyfriday: no bot? ^ | 08:18 |
| abregman | marios: so we should we ping tomorrow/later this week or you will simply merge this at some point? | 08:18 |
| marios | abregman: please add your reviews to https://hackmd.io/FGMoCiRfSNa8puA1BpTQ-Q?edit | 08:18 |
| marios | abregman: we only ever check today/yesterday so you'll need to re-add if not merged tomorrow | 08:19 |
| marios | abregman: usually adding to review list and if needed you are welcome to join the reviews call to present something about the change | 08:20 |
| marios | abregman: so usually adding to review list is enough but feel free to ping as needed ;) | 08:20 |
| frenzyfriday | marios, bot is down for maintainence. I'll get it back in a few hrs | 08:21 |
| frenzyfriday | hackmd has suddenly started returning 403, I am checking why | 08:22 |
| abregman | marios: got it, thanks | 08:24 |
| frenzyfriday | add to reviewlist https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/427877 | 08:28 |
| marios | frenzyfriday: thanks | 08:29 |
| marios | abregman: there is a bot you can use here abregman fyi frenzyfriday owns it and it should be back soon fyi | 08:29 |
| abregman | marios, frenzyfriday: what's the syntax? :) | 08:30 |
| marios | abregman: like that 11:28 < frenzyfriday> add to reviewlist | 08:30 |
| marios | https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/427877 | 08:30 |
| frenzyfriday | bot add to reviewlist https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/427877 | 08:32 |
| reviewbot | I have added your review to the Review list | 08:32 |
| frenzyfriday | now it works. /me putting up a patch | 08:33 |
| marios | thank you frenzyfriday | 08:33 |
| marios | welcome back reviewbot | 08:33 |
| marios | :( | 08:33 |
| abregman | add to reviewlist https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/427710 | 08:34 |
| frenzyfriday | pls wait for the patch marios :D | 08:34 |
| marios | :) | 08:35 |
| frenzyfriday | here is the patch, pls add to your review lists https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44911 | 08:36 |
| reviewbot | I have added your review to the Review list | 08:36 |
| frenzyfriday | meanwhile you can use the bot, it is running on my local | 08:36 |
| marios | ack frenzyfriday but how was it working before then | 08:37 |
| marios | ysandeep: chandankumar: please merge it when you have a minute https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44911 | 08:37 |
| frenzyfriday | looks like hackmd has a bug (or feature?) There are 2 types of notes - personal notes for which the api patch is ...notes/<note_id> and team notes for which patch is .../teams/<team_id>/notes/<note_id> The bot used to use the syntax /notes/ to update the team notes. It worked well for so many months. Since a few weeks this api call returns 403 but still updates the actual note somehow! So the reviews get added to | 08:39 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 08:39 |
| frenzyfriday | the list but the bot thinks 403 means it couldnt add reviews to the list | 08:39 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 08:39 |
| marios | ack thanks frenzyfriday | 08:40 |
| mciecierski | Hi, I would like to ask if it is possible to run tripleo-quickstart against PSI cloud, instead of using libvirt guest. I see in docs this https://docs.openstack.org/tripleo-quickstart/latest/configuration.html#consuming-openstack-hosted-vm-instances-as-overcloud-undercloud-nodes, but it seems experimental feature. | 08:41 |
| frenzyfriday | jm1, hey 0/ I have a question regarding the incockpit ansible pull thingy. A new patch https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44860 which changes the docker compose yaml merged yesterday. Will it be automatically pulled and used in the incockpit deployment (the one on c7 vm on baremetal) or should I manually do stuff? | 08:47 |
| jm1 | frenzyfriday: you edited the docker-compose.yml, hence docker-compose (called from ansible-pull) automatically picked up the stuff. downstream dockpit is already running with influx 1.8 | 08:52 |
| frenzyfriday | jm1, awesome , thanks! | 08:53 |
| marios | mciecierski: should be able to do it but your mileage may vary/you may need to tweak things (feel free to ask here). | 08:53 |
| * frenzyfriday likes ansible pull now | 08:53 | |
| marios | mciecierski: we have some config there fwiw @ https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/testenv/multinode-psi.yml https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/testenv/ovb-psi.yml | 08:53 |
| marios | mciecierski: but that config is for use in our jobs so only some might apply | 08:54 |
| bhagyashris|ruck | jm1, plz | 08:54 |
| bhagyashris|ruck | send me the new hackmd link | 08:54 |
| bhagyashris|ruck | got it | 08:57 |
| abregman | marios: can you please send me an invite for today's review meeting? | 09:01 |
| mciecierski | marios: After doing some tweaks, should I pass this config as such `bash quickstart.sh --nodes toci-quickstart/config/testenv/multinode-psi.yml to use it ? | 09:02 |
| mciecierski | Or nodes need to be provisioned in advanced and https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/testenv/multinode-psi.yml is `Feature Configuration`? | 09:05 |
| ysandeep | frenzyfriday, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44911 failed on lint | 09:06 |
| *** ysandeep is now known as ysandeep|lunch | 09:06 | |
| marios | abregman: sure sec | 09:06 |
| marios | abregman: tomrrow (today's one is much later you may not want to join) | 09:06 |
| marios | sending both | 09:07 |
| marios | abregman: see private please | 09:07 |
| jm1 | ysandeep: regarding the mirror issue thingy: checked todays c9 master failures and we had not had this reason once. we had it yesterday though | 09:07 |
| marios | mciecierski: not sure about that to be honest it is not something i've tried for a while. in ci we get nodes from nodepool and then run quickstart and plays on those you can see some examples of how we use it at https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_quickstart.sh.j2 | 09:15 |
| marios | mciecierski: and yeah the multinode-psi is more about feature config | 09:16 |
| marios | abregman: added you for todays call too | 09:17 |
| mciecierski | marios: ack, thank you | 09:21 |
| marios | folks anyone tried quickstart.sh with psi lately? I recall someone on one of our calls saying... maybe you can help mciecierski who is trying | 09:22 |
| abregman | marios: thank you | 09:28 |
| abregman | in the meantime, can we merge this one? https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/427710 | 09:28 |
| abregman | testproject won't pass and there is CIX but I understand from Ronelle we can merge in the meantime, simply not the criteria change | 09:28 |
| frenzyfriday | ysandeep|lunch, thanks, updated https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44911/ | 09:32 |
| marios | amoralej|off: o/ ah maybe you're out today | 09:38 |
| jm1 | ysandeep|lunch: oh i actually got this centos mirror issue just now! https://logserver.rdoproject.org/08/44908/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/cd2b20a/job-output.txt | 10:01 |
| jm1 | rlandy|out: look at that! https://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_agg_detail.html?ref_hash=f6749f9cda54d017021ab56bf4ec0958 🥳 | 10:06 |
| marios | jm1: we have both... the "current" issue for Failed to connect to mirrors.centos.org port 443: No route to host another example there https://review.rdoproject.org/zuul/build/045fb41162a24832ab1ee61f327db3a4 | 10:06 |
| marios | jm1: but also we are still seeing that one https://bugs.launchpad.net/tripleo/+bug/1983817 (old closed bug) Could not resolve host: mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org] example there https://review.rdoproject.org/zuul/build/417881f0cfad4159936e9d65ce59e765 | 10:07 |
| jm1 | rlandy|out: we had to rerun fs64 only a couple of dozen times and voila it passes | 10:07 |
| marios | jm1: either resurrect +bug/1983817 (and its cix) or probably cleaner file a new one since we are mainly seeing the mirrors.centos.org issue now i think? | 10:07 |
| jm1 | marios: why? it is intermittent and not even very often compared to the other failures | 10:08 |
| marios | jm1: we still track intermittent issues as cix so they might be addressed. intermittent != transient | 10:08 |
| marios | jm1: we've had that since at least yesterday, and apparently it is still ongoing | 10:09 |
| jm1 | marios: we have ~6 other intermittent failures which are coming up dozens of times a week, hence we should report those first | 10:09 |
| jm1 | marios: rr notes has a list of intermittent bugs which i am seeing every day | 10:10 |
| jm1 | marios: when i get some time i will create bugs | 10:11 |
| marios | jm1: this seems like a serious issue - first one i checked master buildset see that https://review.rdoproject.org/zuul/buildset/372d55679278445da538e008b4ac3018 | 10:11 |
| marios | i see 10 jobs fail on RETRY in that ONE buildset | 10:12 |
| marios | jm1: so yeah i think this one is worth a CIX asap | 10:12 |
| marios | jm1: ack on 13:11 < jm1> marios: when i get some time i will create bugs | 10:12 |
| marios | jm1: ping if you want me to file something | 10:13 |
| jm1 | marios: i completely agree with you that our situation is bad. we have so many intermittent issues that i had to rekick c9 wallaby fs64 dozens of times until it passes. its just i focus on the most outdated components etc. first | 10:17 |
| jm1 | marios: i would really appreciate if you could file a bug for that one | 10:18 |
| jm1 | rlandy|out: c9 wallaby is promoting.. lets hope all jobs still work.. http://promoter.rdoproject.org/promoter_logs/centos9_wallaby.log | 10:22 |
| marios | jm1: ok i will file that no problem doing | 10:22 |
| jm1 | marios: shall we file bugs for all other intermittent errors as well? | 10:23 |
| jm1 | marios: we will be flooded with intermittent issues. even the ones we currently have are not handled | 10:24 |
| marios | jm1: i don't know jakob i haven't seen them but i can tell you this one is killing all the lines | 10:24 |
| marios | jm1: i have seen it for master wallaby 9 8 .. not on train but wont be surprised | 10:24 |
| marios | this is just chekcing the last runs for those ^^^ sec will add links in bug | 10:25 |
| marios | jm1: general rule is if you see something more than once (even 2 or 3 examples) it is enough to file a bug | 10:25 |
| jm1 | marios: that was really meant as a open question. i dont know what the "right" way is here which is why i am asking :D | 10:25 |
| jm1 | marios: ack | 10:26 |
| marios | jm1: it could be transient ... and fixed by end of day but then you still have a timestamp... during this day promotions were blocked by this thing | 10:26 |
| marios | jm1: for the RETRY/mirror thing we have seen it since at least yesterday might be older | 10:26 |
| marios | jm1: we should establish asap that we are blocked on this thing | 10:26 |
| *** rlandy|out is now known as rlandy | 10:29 | |
| jm1 | marios: but how can we be blocked on that thing if jobs pass after rerun? | 10:31 |
| rlandy | marios: chandankumar: ysandeep|lunch: jm1: hey ... wanted to talk to you about OVB in general ... pls ping when around | 10:31 |
| marios | jm1: there https://bugs.launchpad.net/tripleo/+bug/1989452 | 10:31 |
| jm1 | rlandy: around ;) | 10:31 |
| rlandy | ie: I don't want to keep doing this rerun dance | 10:31 |
| rlandy | jm1: hey ... how's the network component | 10:31 |
| rlandy | we get by there? | 10:31 |
| jm1 | rlandy: sync? | 10:32 |
| rlandy | jm1: yeah - we can | 10:32 |
| rlandy | bhagyashris|ruck" ^^ want to join? | 10:32 |
| rlandy | https://meet.google.com/soo-vrjn-kgt?pli=1&authuser=0 | 10:32 |
| marios | jm1: we are blocked because it kills your promotion lines 13:31 < jm1> marios: but how can we be blocked on that thing if jobs pass after rerun? | 10:33 |
| rlandy | jm1: bhagyashris|ruck: marios: ^^ if you want to join | 10:33 |
| marios | jm1: and you have to go chase things that should not be failing | 10:33 |
| marios | rlandy: will join in a couple mins brb | 10:33 |
| rlandy | k | 10:33 |
| *** ysandeep|lunch is now known as ysandeep | 10:43 | |
| marios | jm1: as discussed i thought you meant they were passing in testproject not on actual rerun in the same buildset. i have removed the flags https://bugs.launchpad.net/tripleo/+bug/1989452/comments/2 | 10:45 |
| bhagyashris|ruck | rlandy, https://logserver.rdoproject.org/74/44874/2/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/c38d70b/logs/ | 11:07 |
| abregman | hey. can someone review and possibly merge? https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/427710 | 11:09 |
| abregman | I know there is the reviews meeting. but after every merge we need to rebase other changes, so we can't simply excepts to merge or review everything on the meeting (assuming it goes this way) | 11:10 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 11:10 |
| rlandy | jm1: chandankumar: ysandeep: marios; sen invite to discuss OVB | 11:17 |
| ysandeep | rlandy, ack | 11:17 |
| rlandy | we will need chandankumar | 11:17 |
| chandankumar | rlandy: ack | 11:18 |
| rlandy | so if he is not around, will recshedule | 11:18 |
| rlandy | oh there he is | 11:18 |
| rlandy | abregman: ack - looking at patches | 11:18 |
| abregman | expect* | 11:19 |
| abregman | thanks | 11:19 |
| rlandy | abregman: frenzyfriday: so sc001 and 002 is in | 11:19 |
| rlandy | checking of we have criteria for those | 11:19 |
| * jm1 lunch | 11:19 | |
| abregman | rlandy: sc004 is different - it didn't pass testing (there is CIX) and so criteria is not relevant atm for it | 11:19 |
| abregman | but yesterday you said we can get it in, simply without criteria. so was hoping to do that now | 11:20 |
| * rlandy checking criteria for 001 and 002 | 11:20 | |
| abregman | and move to the next scenario | 11:20 |
| rlandy | abregman: ack | 11:20 |
| * pojadhav brb | 11:20 | |
| rlandy | criteria for 001 is in | 11:21 |
| rlandy | ok - O see mario +1'ed here | 11:23 |
| rlandy | marios | 11:23 |
| rlandy | going to rework that order | 11:23 |
| marios | ack rlandy thanks got the invite lgtm | 11:23 |
| rlandy | could we get a second vote on https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/427710? | 11:24 |
| rlandy | any cores? ^^ | 11:24 |
| rlandy | test has a CIX so we expect failures | 11:25 |
| rlandy | but also a fix | 11:25 |
| rlandy | so ok with adding this now | 11:25 |
| marios | ack rlandy wf | 11:29 |
| rlandy | marios: ty | 11:32 |
| rlandy | abregman: ^^ | 11:33 |
| rlandy | going to edit criteria patch | 11:33 |
| rlandy | then will get that merged later today | 11:33 |
| *** dviroel|out is now known as dviroel | 11:36 | |
| abregman | thank you! | 11:46 |
| abregman | moving to the next one | 11:46 |
| chandankumar | rlandy: jm1 ovb meeting | 12:01 |
| frenzyfriday | afuscoar, hey, I have moved grafana to 8.3.11 here: http://10.0.111.235/?orgId=1 Lemme know if you find anything broken | 12:21 |
| frenzyfriday | here is the patch, pls add to review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44950 | 12:21 |
| reviewbot | I have added your review to the Review list | 12:21 |
| jsanemet | hello | 12:22 |
| jsanemet | rlandy marios: could I please get a review on this change request?: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/427754 | 12:22 |
| jsanemet | it is the followup to abregman's one | 12:22 |
| *** frenzyfriday is now known as frenzyfriday|lunch | 12:32 | |
| marios | jsanemet: i'll add it to my next reviews | 12:34 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 12:34 |
| marios | jsanemet: you can add things to our review list with the reviewbot here or directly at https://hackmd.io/FGMoCiRfSNa8puA1BpTQ-Q?edit FYI | 12:35 |
| reviewbot | I have added your review to the Review list | 12:35 |
| jsanemet | marios: awesome, thank you very much | 12:38 |
| marios | np | 12:41 |
| jm1 | marios: btw thanks for pointing out the difference between transient and intermittent failures in tripleo ci. feel free to explain terms when you feel i have a different/wrong understanding of terms, because that is probably true XD | 12:44 |
| jm1 | rlandy: skiplist patch to help c9 master network component https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/857422 | 12:45 |
| rlandy | jm1: thanks - merging that | 12:45 |
| jm1 | rlandy: btw ykarel has a patch up to fix this issue but it has not been merged yet https://bugs.launchpad.net/tripleo/+bug/1989197/comments/3 | 12:45 |
| rlandy | yep | 12:45 |
| marios | thank you jm1 np and you were right about that retry error anyway | 12:47 |
| jm1 | rlandy: our c9 wallaby promotion has jobs that fail so i guess the whole promotion will fail? what do we do about that? | 12:48 |
| rlandy | failed? | 12:49 |
| * rlandy looks | 12:49 | |
| jm1 | rlandy: its still wip, just looking at openstack-periodic-integration-stable1 | 12:49 |
| rlandy | jm1: not sure what you mean | 12:50 |
| rlandy | openstack-periodic-integration-stable1 has failures ack | 12:51 |
| rlandy | last wallaby promotion happened 2 hours ago | 12:52 |
| rlandy | its still wip - what is? | 12:52 |
| jm1 | rlandy: ah the one in progress has a different hash. ok i got it. | 12:53 |
| jm1 | rlandy: i was expecting to see something in the log but could not find anything about promotion of f6749f9cda54d017021ab56bf4ec0958 http://promoter.rdoproject.org/promoter_logs/centos9_wallaby.log | 12:57 |
| rlandy | jm1: http://promoter.rdoproject.org/promoter_logs/container-push/ | 12:57 |
| rlandy | you can see what go to container push there | 12:57 |
| jm1 | rlandy: ack thx! | 12:59 |
| rlandy | http://promoter.rdoproject.org/promoter_logs/centos9_wallaby_2022-09-13T11:06.log | 13:00 |
| rlandy | jm1: ^^ shows you promoted 1 hash | 13:00 |
| jm1 | rlandy: this was the logfile i was looking for. so it had been rotated already | 13:01 |
| rlandy | yep | 13:01 |
| *** dasm|off is now known as dasm | 13:21 | |
| dasm | o/ | 13:21 |
| *** frenzyfriday|lunch is now known as frenzyfriday | 13:26 | |
| pojadhav | community call : arxcruz, rlandy, marios, ysandeep, bhagyashris|ruck , svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj, dviroel, rcastillo, dasm, jm1|rover | 13:29 |
| pojadhav | in 1 min | 13:29 |
| pojadhav | https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg | 13:29 |
| jm1 | rlandy: c9 master promotion is missing only sc10 internal kvm (failed on both internal and vexxhost), fs64 and fs35. so our usual suspects | 13:44 |
| rlandy | jm1: give it a few | 13:45 |
| rlandy | fs064 and fs035 running now | 13:45 |
| rlandy | fs035 internal passed | 13:45 |
| jm1 | rlandy: both are running several times atm ;) | 13:45 |
| rlandy | will likely skip sc010 kvm | 13:45 |
| rlandy | will see after this meeting | 13:45 |
| jm1 | rlandy: will have aoc mtg after community call, then have to go to city. but will be back in evening. rr doc is up to date. | 13:46 |
| jm1 | rlandy: ah, rekicked c9 master network comp. job as well. running atm | 13:47 |
| rlandy | jm1: k - np - I'll take care of the promotion | 13:47 |
| rlandy | fs064 juts passed | 13:47 |
| jm1 | rlandy: spraying with machine guns helped :D | 13:48 |
| ibernal | Hello everyone, I have a quick question, if I want to ssh into a machine that is running a job in component pipeline, but my job is passing, can I still use autohold feature in zuul? | 14:11 |
| ibernal | Or do we have a keyword we can use in testproject to hold that machine? | 14:11 |
| ibernal | Thanks! | 14:11 |
| * jm1 hopping on a bike, cycling to tailer. bbl | 14:38 | |
| ysandeep | ibernal, zuul only hold a hold if job fails, but i think there is a var to force fail the job.. i will find the var and share with you | 14:41 |
| ysandeep | zuul only hold a job* | 14:41 |
| ibernal | awesome, thank you | 14:41 |
| ysandeep | ibernal, force_job_failure: true | 14:42 |
| ibernal | and I should use it on testproject or the job definition itself? | 14:42 |
| ysandeep | ibernal, but you need to request someone from infra/ci team to add you node on hold and add your keys. | 14:42 |
| ysandeep | ibernal, yes under job vars | 14:42 |
| ibernal | I requestec access earlier today so I can hold the job myself | 14:43 |
| ibernal | ysandeep: thank you for the quick response | 14:44 |
| dviroel | rlandy: fips enabled image still failing to boot - I have a guess that is something related to vexxhost. Because we also have a problem there when we try to enable fips with the reboot process. | 14:52 |
| dviroel | rlandy: so I have an idea of trying on ibm cloud instead | 14:52 |
| dviroel | to see the results | 14:52 |
| *** ykarel is now known as ykarel|afk | 14:54 | |
| ysandeep | rlandy, pojadhav dasm since you are planning rr , just a headup that 06-12 nov, doug/me/chandan will be in next gen mtg | 15:01 |
| dasm | ysandeep: ack | 15:02 |
| *** eliadcohen__ is now known as eliadcohen | 15:04 | |
| *** ykarel|afk is now known as ykarel | 15:10 | |
| dasm | frenzyfriday: o/ qq about elastic recheck. are you actively involved in that right now? can it be put on back burner for the next sprint? | 15:11 |
| pojadhav | ysandeep, yep considered that already | 15:11 |
| frenzyfriday | dasm, yeah, i am working on ER when I get time. It is not on priority | 15:12 |
| rlandy | pojadhav: marios: ok - rr schedule sorted | 15:14 |
| pojadhav | rlandy, great :) | 15:14 |
| pojadhav | thanks !! | 15:14 |
| marios | someone light the chimney with white smoke | 15:15 |
| pojadhav | marios, :D | 15:15 |
| rlandy | pojadhav: dasm: pls ping the team to look at the rest of the year and check if they are on pto on any weeks assigned to them | 15:15 |
| pojadhav | rlandy, yes will ping | 15:15 |
| rlandy | dasm: ok to remove the proposal section so it does not confuse anyone? | 15:16 |
| dasm | rlandy: removed the proposal section | 15:17 |
| rlandy | ty | 15:18 |
| dasm | frenzyfriday: ack. should we keep it in active sprint then? or can we move it to backlog? if it's not continuous, ongoing effort | 15:18 |
| frenzyfriday | dasm we can move it to backlog. | 15:19 |
| pojadhav | hello all : arxcruz, rlandy, marios, ysandeep, bhagyashris|ruck , svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj, dviroel, rcastillo, dasm, jm1|rover | 15:19 |
| pojadhav | please check RR schedule and switch based on your PTO plans. | 15:19 |
| marios | thanks pojadhav will do | 15:21 |
| dviroel | pojadhav: thanks | 15:22 |
| bhagyashris|ruck | pojadhav, ack | 15:23 |
| rlandy | jm1: marios: bhagyashris|ruck: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44964 Skip kvm internal job to promote master | 15:27 |
| dviroel | jm1: btw, internal kvm master still failing - https://sf.hosted.upshift.rdu2.redhat.com/logs/32/425432/11/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master-1/1125beb/logs/undercloud/var/log/extra/errors.txt | 15:31 |
| dviroel | jm1: "Skylake is not correct, or your host CPU arch does not support this model." | 15:31 |
| chandankumar | see ya people! | 15:33 |
| abregman | rlandy, dviroel, bhagyashris|ruck: can we merge this? https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/427754 | 15:37 |
| *** marios is now known as marios|out | 15:49 | |
| *** dviroel is now known as dviroel|lunch | 15:51 | |
| *** ysandeep is now known as ysandeep|out | 15:57 | |
| frenzyfriday | hey folks, whet is the right way to push a patch to gerrit using git commands (not git review)? I tried something like git push origin HEAD:refs/changes/65/44965/1 but that does not work | 16:27 |
| dasm | frenzyfriday: "git commit --amend" to change the commit content and "git review" | 16:28 |
| dasm | it should update it | 16:28 |
| frenzyfriday | no, I mean using only usual git commands, not git review | 16:29 |
| frenzyfriday | I am trying to do it through python, using GitPython which I think supports only the usual git commands | 16:29 |
| dasm | i don't know about that one. | 16:29 |
| dasm | git-review is a python package | 16:29 |
| dasm | hmm. it says it could work, frenzyfriday: https://gerrit-review.googlesource.com/Documentation/user-upload.html#_git_push | 16:32 |
| frenzyfriday | I cant figure out how do I get this part ssh://sshusername@hostname:29418/projectname from my python code :/ | 16:33 |
| *** jpena is now known as jpena|off | 16:35 | |
| frenzyfriday | arxcruz, jm1 follow up on our auto testproject discussion: pls lemme know if you have some ideas on this when you get some time: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44966/1/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf_py3/git_utils.py#17 | 16:35 |
| rlandy | abregman: w+'ed | 16:36 |
| rlandy | dviroel|lunch: you could give psi a shot | 16:37 |
| rlandy | ibm cloud is fine as well | 16:37 |
| dasm | frenzyfriday: wouldn't it be easier to run bash script to do that? | 16:37 |
| dasm | frenzyfriday: i'd prefer avoiding more options to rr script | 16:38 |
| frenzyfriday | dasm yeah probably that will be a better option. But that means we need to parse the RR script's output and get the testproject again through the bash script whereas in rr script we alreay get it in a variable | 16:39 |
| dasm | frenzyfriday: or we can dump rr output to a file. it should be straighforward with jinja templates | 16:40 |
| dasm | then, bash script picks up the file and based on user's config -- sends it to review | 16:40 |
| dasm | frenzyfriday: i don't know what was the initial discussion here. is it going to be automatic, or does it require manual intervention from the user? | 16:41 |
| frenzyfriday | dasm, it is the same discussion from the retro - "something" to automate the number of times we are running testproject to reckick failed jobs (nothing more concrete) | 16:43 |
| dasm | frenzyfriday: that's a good starting point, but probably wrong direction. | 16:43 |
| frenzyfriday | I think your approach is better - we add one flag to RR script - which makes it write the testprojs to a temp file | 16:43 |
| dasm | frenzyfriday: i would be more inclined towards gathering some stats about number of rechecks to see what is costing us the most | 16:43 |
| dasm | frenzyfriday: we can always create a temp file which can be used, or ignored | 16:44 |
| abregman | can we get review for scenario 010? https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/428022 | 16:52 |
| *** dviroel|lunch is now known as dviroel | 16:57 | |
| abregman | rlandy, dasm: ^ | 18:24 |
| rlandy | yep - in a bit | 18:24 |
| abregman | tnx | 18:25 |
| * jm1 3-year-old little girl comes to her father holding a floppy disk in her hand. She says: “Daddy, Daddy, somebody 3D-printed the save icon.” 😆 https://eyeondesign.aiga.org/we-spoke-with-the-last-person-standing-in-the-floppy-disk-business/ | 18:27 | |
| rlandy | bhagyashris|ruck' hi ... https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/428003 - reviewing that | 18:30 |
| rlandy | can you comment of we need the args? | 18:30 |
| jm1 | dviroel: which cpu module are you testing with? "Skylake"? how about the generic ones i linked yesterday? | 18:32 |
| jm1 | rlandy: c9 master has promoted, fs35 passed in one of my reruns | 18:34 |
| rlandy | jm1: ack - only skipped kvm job | 18:36 |
| rlandy | here is the revert on that: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44758 | 18:36 |
| rlandy | I'm rechecking the network jobs | 18:36 |
| dviroel | jm1: i can try that too - i was not expecting Skylake failing on Skylake | 18:37 |
| jm1 | rlandy, bhagyashris|ruck, pojadhav, dasm and team: can we please postpone the rr shifts discussion to our retro or planning meeting? having rlandy as a ruck or rover is ridiculous | 18:37 |
| rlandy | jm1: we just spent a whole hour on that | 18:38 |
| jm1 | dviroel: actually to me it looks like this cpu models in l2 does not work at all. but we know for sure after you have tested the generic profile | 18:39 |
| rlandy | pojadhav: pls bring your victoria tear down patches to wed review meeting | 18:39 |
| rlandy | I would think our easiest path here is to get PSI to assign us the right nodes | 18:40 |
| dviroel | jm1: ack - will try that in a bit | 18:41 |
| jm1 | dviroel: and btw, thanks for testing all that and retriggering the job thousands of times. i know it sucks and i am really glad you help us with that! | 18:41 |
| dviroel | jm1: np, I know that rrs have lot of things to do, so at least I can help with something | 18:42 |
| jm1 | rlandy: i rekicked c9 master network as well, just to be increase the chance that one of those jobs passes ;) | 18:43 |
| rlandy | jm1: need to see why those are failing rather than just a random rekick | 18:51 |
| jm1 | rlandy: ^ is why i had to create another rr doc. failure reasons in rr docs take a looot of space | 18:53 |
| rlandy | I get it | 18:54 |
| rlandy | jm1: https://hackmd.io/ulFAL5DBRoGQB6eUuOpC1Q | 18:54 |
| rlandy | going to try get some order to the madness | 18:54 |
| rlandy | this can't carry on | 18:54 |
| rlandy | I know it's bad | 18:54 |
| jm1 | rlandy: for fs35 and fs64 i check the logs 2-3 times and if it is always an intermittent/transient failure, then i scale up the number of "parallel" jobs and do not look at each logs any more. it just takes too much time. i would still be checking the errors of todays runs. how many reruns did i do today? 50? | 18:56 |
| jm1 | rlandy: but usually i would not do that. i did it because we wanted to get promotions ;) | 18:57 |
| rlandy | jm1: it's ok | 18:59 |
| rlandy | you have the promos you need | 18:59 |
| rlandy | you can take a day off that | 18:59 |
| rlandy | I would like to start splitting where we are running jobs | 18:59 |
| rlandy | jm1: for fs035 | 19:00 |
| rlandy | if it ran and passed on internal | 19:00 |
| rlandy | ok to ski[ | 19:00 |
| rlandy | skip | 19:00 |
| rlandy | dasm: dviroel: rcastillo: review time? | 19:02 |
| jm1 | rlandy: but is moving jobs etc. the right way? to me it looks as if tripleo or rhos in general is very sensitive to load or latency (cpu? network? disk?) of the underlying system. we need help from dfgs to make rdo/rhos more robust. | 19:02 |
| rlandy | jm1: on review time - sec | 19:03 |
| rlandy | feel free to add notes to hackmd | 19:03 |
| jm1 | rlandy: ack | 19:03 |
| rlandy | dasm: doing your reviews + docker compose updates | 19:15 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 19:15 |
| rlandy | dasm: can you review pls: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44950 | 19:16 |
| jm1 | rlandy: ovb status hackmd is now in ruck_rover section | 19:17 |
| jm1 | rlandy: also added some more content to the intro section in case we want to show it to someone outside our team | 19:18 |
| abregman | add to review list https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/428286 | 19:22 |
| reviewbot | I could not add the review to Review List | 19:22 |
| abregman | :( | 19:26 |
| rlandy | dasm: hey - when you are back we can look at your series of patches | 19:30 |
| dasm | rlandy: sorry, i had an emergency. everything is fine, but i was afk | 19:40 |
| rlandy | dasm: np - we were just struggling to understand your reviews | 19:47 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 19:47 |
| dasm | ack | 19:48 |
| abregman | can we merge this one? https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/428286 | 19:48 |
| rlandy | dasm: we should merge these if they help | 19:52 |
| dasm | rlandy: the intend is to lower number of queries to zuul. they *should* help. similar to our 3 or 4 other attempts from th past | 19:54 |
| rlandy | dasm: can we test that? | 19:55 |
| rlandy | prove it works? | 19:55 |
| rlandy | won't break anything else? | 19:55 |
| rlandy | rr script is high impact | 19:55 |
| dasm | rlandy: i did diff comparison on cs9 master branch for integration and all components (influx) | 19:56 |
| dasm | there is no difference. i can provide it again | 19:56 |
| jm1 | rlandy: looks like nothing is urgent on rr front for now. c9 wallaby promoted today, c9 master promoted today and it took only a liiiiiiitle bit of cheating. ok and thousands of reruns.. anyway. only c9 master network is way out but this is in rerun. good results for a single rr day 😆 with that i can sleep well tonight. eod now | 19:56 |
| rlandy | jm1: thanks - have a good night | 19:56 |
| dasm | rlandy: also, there are unittests in place to ensure backcompat | 19:56 |
| jm1 | rlandy: thanks to you :D | 19:56 |
| dasm | jm1: take care, good night | 19:57 |
| * jm1 out for today, have a good night #oooq | 19:57 | |
| rcastillo | o/ jm1 | 19:57 |
| rlandy | dasm: ok - so what's thebets way here: you want us to get back on a call? | 19:57 |
| dasm | rlandy: if you need additional explanations we can talk f2f | 19:57 |
| rlandy | dviroel: rcastillo: ^^? | 19:58 |
| dasm | if there are some missing pieces which can be left as a review, i can address those too | 19:58 |
| rlandy | dasm: yeah - let's see if we can get the others back | 19:59 |
| rlandy | so your reviews don't lag | 19:59 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 19:59 |
| dasm | k, i'm back | 19:59 |
| rcastillo | I'm going through the script changes atm | 19:59 |
| rlandy | ok | 19:59 |
| rlandy | rcastillo++ | 19:59 |
| rcastillo | dasm: I'll let you know if I have any questions | 19:59 |
| rlandy | dasm: ok - so ping if you don't see good reviews/movement there | 19:59 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 19:59 |
| dasm | ack | 19:59 |
| rlandy | dasm: also -can your review ananya's patch on grafana? | 20:00 |
| rlandy | you would know that better | 20:00 |
| dasm | sure | 20:00 |
| rlandy | ty | 20:00 |
| *** dviroel is now known as dviroel|brb | 20:10 | |
| abregman | add to review list https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/427816 | 20:14 |
| reviewbot | I could not add the review to Review List | 20:14 |
| abregman | :'( | 20:14 |
| rlandy | abregman: re: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/427816 - pls follow the same order as 17 | 20:19 |
| abregman | rlandy: done | 20:21 |
| rcastillo | dasm: patches LGTM, see comments about tests | 20:43 |
| dasm | rcastillo: ack, thx. | 20:44 |
| dasm | rcastillo: i'm gonna respond to you on the patch, but you asked about track_component and track_integration inside "class TestInfluxDBMeasurements(unittest.TestCase):" | 20:45 |
| dasm | there is no influx for them anymore | 20:46 |
| arxcruz | reviewbot please add https://review.opendev.org/c/openstack/tripleo-quickstart/+/839725 to review | 20:46 |
| arxcruz | reviewbot help | 20:46 |
| rcastillo | right but there's no test for the non-influx case | 20:46 |
| arxcruz | i never know the damn pattern | 20:46 |
| rcastillo | afaict | 20:46 |
| dasm | rcastillo: because i never wrote it | 20:46 |
| dasm | rcastillo: i can't test rendering tables | 20:47 |
| arxcruz | reviewbot add to review list https://review.opendev.org/c/openstack/tripleo-quickstart/+/839725 | 20:47 |
| reviewbot | I could not add the review to Review List | 20:47 |
| arxcruz | ... | 20:47 |
| rcastillo | dasm: ah, ack | 20:47 |
| rcastillo | oh well | 20:47 |
| dasm | rcastillo: i'm slowly working towards smaller, chunk-sized functions, so I'm gonna be able (hopefully) to write some tests | 20:47 |
| rcastillo | yeah, I could see that intention in the patches | 20:48 |
| dasm | for now, we're just using manual testing (by "we" i mean ysandeep|out and myself) to compare outputs | 20:48 |
| rcastillo | yeah it's ok since that path is just used by us locally | 20:48 |
| dasm | rcastillo: i left similar note on the review, to keep the history in one place | 20:50 |
| dasm | rcastillo: thank you for taking time to review those | 20:50 |
| rcastillo | thanks for giving some attention to the script | 20:51 |
| dasm | rcastillo: otherwise dpawlik would come back to us more often, saying we're killing zuul :) | 20:51 |
| dasm | (actually, it might be just me ;P ) | 20:52 |
| rlandy | abregman: thanks - voted | 20:57 |
| rlandy | dviroel|brb: need help with the fips image? | 21:03 |
| rlandy | waiting for network tests - back later | 21:27 |
| *** rlandy is now known as rlandy|bbl | 21:27 | |
| * dasm leaves | 22:28 | |
| *** dasm is now known as dasm|off | 22:28 | |
| dasm|off | o/ | 22:28 |
| * rcastillo out as well | 22:28 | |
| rcastillo | o/ | 22:28 |
| *** dasm|off is now known as Guest305 | 23:03 | |
| *** rlandy|bbl is now known as rlandy | 23:27 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!