| *** ysandeep|out is now known as ysandeep | 00:29 | |
| ysandeep | good morning ci o/ | 00:30 |
|---|---|---|
| dasm | ysandeep: is it morning already? | 00:30 |
| dasm | good morning! | 00:31 |
| ysandeep | yes, 06:01 am here | 00:31 |
| dasm | 17:31 (5:31PM) | 00:32 |
| ysandeep | :) we have 11:30 hr difference wow | 00:32 |
| dasm | yup | 00:33 |
| * dasm is done for tonight | 00:37 | |
| dasm | o/ | 00:37 |
| *** dasm is now known as dasm|off | 00:37 | |
| ysandeep | dasm|off, o/ see you tomorrow, have a great evening! | 00:40 |
| tonyb | Hi all, it looks like we merged a criteria patch ( https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/428718 ) without the job/pipeline patch ( https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/428693 ) Can we merge the latter (jobs patch) ? | 02:10 |
| ysandeep | tonyb: hey o/ looking | 02:26 |
| tonyb | ysandeep: Thanks :) | 02:26 |
| ysandeep | nw :) +wed , for future will be good to use depends-on to avoid similiar issues. | 02:31 |
| tonyb | ysandeep: Thanks, and I totally agree about the Depends-On | 02:36 |
| *** pojadhav|PTO is now known as pojadhav | 03:08 | |
| *** ysandeep is now known as ysandeep|afk | 03:31 | |
| akahat|ruck | Good morning o/ | 04:24 |
| marios | o/ | 05:57 |
| *** ysandeep|afk is now known as ysandeep | 05:58 | |
| *** amoralej|off is now known as amoralej | 07:03 | |
| marios | akahat|ruck: under control? lets talk about the prog call doc/update when you have time | 07:12 |
| *** jpena|off is now known as jpena | 07:38 | |
| marios | akahat|ruck: d/stream pipelines didn't run - like there is some problem you mean? | 07:52 |
| marios | 10:52 <akahat|ruck> pipeline_integration-pcci-17.1_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv6-geneve-ceph_rcj | 07:52 |
| marios | 10:52 <akahat|ruck> this is the job | 07:52 |
| marios | akahat|ruck: we have that in criteria? ^ | 07:52 |
| akahat|ruck | marios, pipeline_integration-pcci-17.1_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph_rcj | 07:53 |
| jm1 | hello hello folks :) /me got lost in my mails and almost forgot to log in to irc 😅 | 07:53 |
| akahat|ruck | yes. | 07:53 |
| akahat|ruck | those are not triggered. | 07:53 |
| marios | akahat|ruck: k lets discuss on phone in a bit | 07:53 |
| akahat|ruck | marios, yeah. | 07:53 |
| marios | o/ jm1 | 07:53 |
| jm1 | marios: o/ :D | 07:53 |
| akahat|ruck | jm1, o/ | 07:53 |
| jm1 | akahat|ruck: o/ how is rr? | 07:54 |
| akahat|ruck | jm1, it's good for now. :D | 07:54 |
| jm1 | akahat|ruck: hehe, good :) | 07:55 |
| dpawlik | marios: "are you able to create 'zed' directory for us at http://images.rdoproject.org/centos9/ please? (doesn't have to be immediately but when you have time if you can please?)" -> ping kkula, amoralej or jcapitao | 08:02 |
| marios | thanks dpawlik | 08:12 |
| akahat|ruck | marios, are you free can we sync now? | 08:18 |
| marios | akahat|ruck: lets do in 12 mins as you suggested going afk for 10 | 08:18 |
| akahat|ruck | marios, ok. | 08:18 |
| akahat|ruck | https://meet.google.com/cna-jguu-xqb | 08:18 |
| *** pojadhav is now known as pojadhav|afk | 08:22 | |
| marios | akahat|ruck: let me know when you're back just dropped | 08:30 |
| akahat|ruck | marios, i'm back | 08:32 |
| marios | akahat|ruck: please rejoin cant hear you | 08:33 |
| amoralej | dpawlik, done | 08:34 |
| marios | thank you amoralej o/ | 08:35 |
| marios | akahat|ruck: let me know when to join :D | 08:35 |
| marios | amoralej: can you also please create rdo_trunk inside zed? | 08:36 |
| marios | amoralej: then we should be able to write into there | 08:36 |
| amoralej | done | 08:36 |
| marios | thank you amoralej++ | 08:37 |
| marios | frenzyfriday: o/ do you have some time today please to talk about new secret needed for zed container push? | 08:37 |
| frenzyfriday | marios, yep sure | 08:37 |
| * frenzyfriday getting my notes | 08:38 | |
| marios | frenzyfriday: thanks - lets schedule it... at 1200 CET (in 1hr/20mins time) OK for you? | 08:38 |
| marios | frenzyfriday: if ok i'll send invite now | 08:38 |
| frenzyfriday | marios, yep, sounds good | 08:39 |
| marios | ack frenzyfriday | 08:39 |
| marios | akahat|ruck: o/ should we join? | 08:40 |
| akahat|ruck | marios, yes | 08:41 |
| akahat|ruck | waiting for you | 08:41 |
| marios | akahat|ruck: ? | 08:42 |
| frenzyfriday | hey dpawlik , in which repo did we add the registry names last time so that you quay tool will pick it up and create the namespaces/robot users wih proper permissions? | 08:57 |
| dpawlik | frenzyfriday: oh, don't remember | 08:58 |
| dpawlik | frenzyfriday: should I help with creating new namespace + users + etc ? | 08:58 |
| frenzyfriday | dpawlik, no, I can create them manually on quay.rdo - but I thought last time there was like a quay tool which did all these and all we had to do was add the registry namespace | 09:01 |
| dpawlik | indeed | 09:04 |
| dpawlik | https://softwarefactory-project.io/r/plugins/gitiles/software-factory/python-quay-tool | 09:04 |
| dpawlik | hope all is covered in README | 09:04 |
| dpawlik | let me know if you need help | 09:04 |
| dpawlik | "Basic workflow how to setup new organziation" | 09:05 |
| frenzyfriday | dpawlik, thanks, checking | 09:06 |
| marios | thank you frenzyfriday dpawlik much appreciated | 09:12 |
| akahat|ruck | marios, ysandeep please reivew https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/429155 | 09:16 |
| ysandeep | akahat|ruck, do we have a bug why jenkins jobs are failing? | 09:18 |
| akahat|ruck | ysandeep, we do not found those jobs defined in the downstream jenkins | 09:19 |
| ysandeep | akahat|ruck, let me ping you on internal channel but those jobs are green | 09:20 |
| akahat|ruck | ysandeep, ack | 09:20 |
| marios | ack akahat|ruck we are discussing with afazekas now | 09:21 |
| marios | i think we should go with it though | 09:21 |
| arxcruz | soniya29 i don't have anything for today, we might cancel the meeting, wdyt? | 09:22 |
| marios | ysandeep: please add your vote and i'll workflow https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/429155 | 09:24 |
| ysandeep | marios, its a -1 for me | 09:24 |
| ysandeep | voting | 09:24 |
| ysandeep | no bug | 09:24 |
| marios | ysandeep: akahat|ruck: please https://meet.google.com/kkz-wwhx-fgq | 09:25 |
| marios | easeier | 09:25 |
| ysandeep | joining | 09:26 |
| soniya29 | arxcruz, yeah | 09:27 |
| marios | frenzyfriday: https://quay.rdoproject.org/organization/tripleozedcentos9 | 10:04 |
| *** ysandeep is now known as ysandeep|lunch | 10:06 | |
| *** rlandy|out is now known as rlandy|ruck | 10:29 | |
| rlandy|ruck | akahat|ruck, marios: want to sync? | 10:31 |
| akahat|ruck | rlandy|ruck, yes. | 10:31 |
| marios | rlandy|ruck: eating, will catch you in a bit? (already in sync with akahat|ruck ) | 10:32 |
| soniya29 | kopecmartin, rlandy|ruck, ysandeep|lunch, arxcruz, we are cancelling today's in the lack of agenda | 10:32 |
| arxcruz | soniya29 ack | 10:32 |
| rlandy|ruck | marios: you can skip | 10:32 |
| rlandy|ruck | soniya29: arxcruz: won't be bal eto attend | 10:32 |
| rlandy|ruck | dealing with rr | 10:32 |
| rlandy|ruck | arxcruz: ^^ pls work this out | 10:32 |
| rlandy|ruck | akahat|ruck: https://meet.google.com/wuu-sepa-dva?pli=1&authuser=0 | 10:33 |
| arxcruz | rlandy|ruck bal eto? | 10:33 |
| frenzyfriday | pls add to review list https://review.rdoproject.org/r/c/config/+/45188 (quay secrets) | 10:34 |
| reviewbot | I could not add the review to Review List | 10:34 |
| frenzyfriday | nice, again! :( | 10:35 |
| marios | frenzyfriday: thank you! | 10:35 |
| * frenzyfriday will check later what happened to reviewbot | 10:36 | |
| frenzyfriday | ^ this time it really didnt add the review | 10:36 |
| frenzyfriday | marios, can you pls send me your container push patch with the jobs def? | 10:39 |
| marios | frenzyfriday: https://review.rdoproject.org/r/c/testproject/+/45037 | 10:52 |
| marios | frenzyfriday: sorry in call | 10:52 |
| frenzyfriday | marios, np, thanks, checking | 10:52 |
| *** ysandeep|lunch is now known as ysandeep | 10:58 | |
| *** pojadhav|afk is now known as pojadhav | 11:14 | |
| frenzyfriday | hey folks, review time | 11:18 |
| rlandy|ruck | bhagyashris: chandankumar: what do you want to do about https://trello.com/c/kzzOd5by/2710-cixbz2127111osp16-2rhel8stderr-error-reading-multiple-stages-arg-requires-exactly-one-argument-definition-n? | 11:28 |
| rlandy|ruck | keep it for the real fix> | 11:28 |
| bhagyashris | rlandy|ruck, i think we will need real fix this is temporary fix | 11:34 |
| bhagyashris | i am not sure what's in chandankumar's mind | 11:34 |
| rlandy|ruck | arxcruz: soniya29: https://review.opendev.org/c/openstack/tempest/+/857333 - still open - for open CIX - can you discuss at meeting? | 11:38 |
| rlandy|ruck | https://trello.com/c/sfZJuWff/2723-cixbz2128494osp16-2rhel8unhandled-nooptionerror-in-python-tempestconf | 11:38 |
| rlandy|ruck | ^^ other open CIX of tempest | 11:38 |
| arxcruz | rlandy|ruck i think kopecmartin is working with afazekas about this | 11:38 |
| rlandy|ruck | yep - juts an fyi | 11:38 |
| arxcruz | and regarding https://review.opendev.org/c/openstack/tempest/+/857333 i asked gmann and kopecmartin to review it yesterday on qa meeting | 11:39 |
| soniya29 | rlandy|ruck, ack | 11:43 |
| rlandy|ruck | ysandeep: have a clashing meeting with most of today's operator testing meeting | 11:43 |
| rlandy|ruck | can join foe the first 15 mins | 11:43 |
| frenzyfriday | marios, I have updated bitwarden entry for rdo quay | 11:44 |
| *** frenzyfriday is now known as frenzyfriday|lunch | 11:44 | |
| ysandeep | rlandy|ruck, ack.. I will give status update on where we are. | 11:44 |
| marios | frenzyfriday|lunch: thank you very much for your fast response on this | 11:47 |
| chandankumar | rlandy|ruck: checking | 11:51 |
| marios | amoralej: when you have a sec can you please sanity check permissions rsync: recv_generator: mkdir "/var/www/html/images/centos9/zed/rdo_trunk/66210b4cc52f6fd4cd50781d0c8724dc" failed: Permission denied @@ https://logserver.rdoproject.org/37/45037/4/check/periodic-tripleo-centos-9-buildimage-overcloud-hardened-uefi-full-zed/0373c61/job-output.txt | 11:53 |
| marios | amoralej: for the newly created zed/rdo_trunk dirs please | 11:53 |
| amoralej | ah, sorry | 11:53 |
| marios | should be same as the wallaby ones for reference | 11:53 |
| amoralej | i think i know what happened | 11:53 |
| marios | thanks amoralej np | 11:53 |
| amoralej | marios, it should work now | 11:55 |
| marios | amoralej: also any idea why i have recheck roulette for this Request failed: <urlopen error [Errno 113] No route to host for https://trunk.rdoproject.org/centos9-zed/promoted-components/delorean.repo.md5 at https://logserver.rdoproject.org/37/45037/4/check/periodic-tripleo-centos-9-zed-promote-promoted-components-to-tripleo-ci-testing/ff3a711/job-output.txt and also in | 11:55 |
| marios | https://review.rdoproject.org/r/c/testproject/+/45115/2#message-4c18156ffcd4209f5e9167bed526d96656ddf594 | 11:55 |
| marios | amoralej: i'll file a bug for that but mainly first checking in case rings some bells thanks amoralej | 11:55 |
| marios | thank you 14:55 < amoralej> marios, it should work now | 11:55 |
| marios | amoralej: i'll rerun in a bit ^ | 11:56 |
| amoralej | marios, looks like infra issue ? | 11:56 |
| chandankumar | rlandy|ruck: which osp release is based on stable/zed? | 11:56 |
| marios | amoralej: not sure yet. nodeset is the same as the wallaby/9 and master jobs | 11:56 |
| marios | chandankumar: 18 | 11:56 |
| amoralej | error is consistently happening? | 11:57 |
| marios | amoralej: no but pretty often like i have 3/4 rechecks before i get a green run | 11:57 |
| chandankumar | marios: ok, thanks :-) | 11:57 |
| marios | chandankumar: np | 11:57 |
| marios | k thanks amoralej as i said just mentioned in case will file a bug in a bit anyway ... right now focusing on bootstrap the pipeline | 11:58 |
| chandankumar | rlandy|ruck: marios we can move this card https://trello.com/c/kzzOd5by/2710-cixbz2127111osp16-2rhel8stderr-error-reading-multiple-stages-arg-requires-exactly-one-argument-definition-n to Done | 12:02 |
| chandankumar | akahat|ruck: How is the c8 promotion looks like? | 12:03 |
| chandankumar | Are we seeing any IBM cloud related issues there? | 12:03 |
| chandankumar | from cockpit cs8 wallaby and train promoted yesterday | 12:04 |
| akahat|ruck | chandankumar, I didn't seen any issues related to it. | 12:04 |
| chandankumar | akahat|ruck: cool! | 12:05 |
| chandankumar | akahat|ruck: thanks! | 12:06 |
| *** amoralej is now known as amoralej|lunch | 12:10 | |
| rlandy|ruck | akahat|ruck: I'm testproject running the skipped jobs from wallaby line | 12:10 |
| akahat|ruck | rlandy|ruck, ack. will keep eye on it. | 12:11 |
| rlandy|ruck | master looks like it has a deploy issue | 12:12 |
| rlandy|ruck | akahat|ruck: wrt component line - pls concentrate on master components | 12:21 |
| akahat|ruck | rlandy|ruck, yeah.. there are failures.. but not consistent.. :( | 12:22 |
| *** ysandeep is now known as ysandeep|afk | 12:26 | |
| chandankumar | rlandy|ruck: marios akahat|ruck https://review.rdoproject.org/r/q/topic:dib-cs9-label please have a look when free :-) | 12:35 |
| chandankumar | it is going to impact cs9 wallaby & master pipeline jobs | 12:35 |
| *** ysandeep|afk is now known as ysandeep | 12:39 | |
| *** frenzyfriday|lunch is now known as frenzyfriday | 12:46 | |
| marios | chandankumar: k adding to my reviews list | 12:49 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 12:49 |
| rlandy|ruck | all: open forum now | 13:01 |
| rlandy|ruck | chandankumar: nice - still testing?? | 13:13 |
| rlandy|ruck | https://review.rdoproject.org/r/c/testproject/+/44982 | 13:13 |
| rlandy|ruck | chandankumar: jm1: need your opinions pls ... | 13:14 |
| rlandy|ruck | https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/tasks/install_openstack_deps.yml#L25 | 13:14 |
| rlandy|ruck | failing a lot as in https://logserver.rdoproject.org/54/31954/96/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/49c1b69/job-output.txt | 13:14 |
| rlandy|ruck | increase the retries? | 13:15 |
| jm1 | rlandy|ruck: never touched this part of tripleo. network seems to be unreachable, so i guess increasing the retries wont help? either network config is broken or pip has rate limiting? | 13:22 |
| jm1 | rlandy|ruck: oh wait it has issues connecting to github | 13:23 |
| *** amoralej|lunch is now known as amoralej | 13:26 | |
| chandankumar | rlandy|ruck: yes, it has issues with connecting to github | 13:29 |
| rlandy|ruck | OMG we got a mention | 13:30 |
| jm1 | chandankumar, rlandy|ruck: how about adding openstack/requirements to job.required-projects and then using upper-constraints.txt from there instead of pulling it from github? | 13:30 |
| jm1 | chandankumar, rlandy|ruck: url in constraint_file would have to be replaced https://codesearch.rdoproject.org/codesearch/?q=constraint_file | 13:32 |
| chandankumar | jm1: then we need to change this https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/defaults/main.yml#L140 | 13:32 |
| * rlandy|ruck reads back | 13:32 | |
| jm1 | chandankumar: we have to replace other locations as well, see above ;) | 13:32 |
| rlandy|ruck | chandankumar: jm1: hmmm ... will have to think if it's worth it | 13:33 |
| rlandy|ruck | will check the retries on that | 13:33 |
| chandankumar | rlandy|ruck: switching to release specific constriant_file may or may not work | 13:34 |
| chandankumar | because of these deps https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/defaults/main.yml#L133 | 13:34 |
| rlandy|ruck | chandankumar: jm1: https://review.rdoproject.org/zuul/builds?result=RETRY&skip=0 - those in 25 min range | 13:36 |
| rlandy|ruck | rest are; "Failed to download metadata for repo 'RDO-CentOS9-stream-deps': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", | 13:38 |
| chandankumar | rlandy|ruck: let's use https://opendev.org/openstack/requirements/raw/branch/stable/wallaby/upper-constraints.txt url for constriants | 13:39 |
| marios | rlandy|ruck: o/ fyi chandankumar has agreed to swap ruck shift with me (he is on in ~1 month normally) | 13:48 |
| marios | rlandy|ruck: so i can keep chasing the zed bootstrap at least to end of this sprint | 13:48 |
| marios | thank you chandankumar o/ | 13:48 |
| marios | \o/ | 13:48 |
| rlandy|ruck | chandankumar++ | 13:48 |
| rlandy|ruck | sure | 13:49 |
| marios | fyi pojadhav dasm|off ^^^ | 13:49 |
| pojadhav | marios, ack | 13:50 |
| rlandy|ruck | 17.1 on rhel-8 promoted | 13:50 |
| rlandy|ruck | marios: can help you from friday if needed | 13:52 |
| rlandy|ruck | chandankumar: hmm - upstream nodes patch failing OVB - consistently | 13:53 |
| rlandy|ruck | https://review.rdoproject.org/r/c/rdo-jobs/+/44968 | 13:53 |
| marios | rlandy|ruck: ack | 13:58 |
| chandankumar | rlandy|ruck: checking | 14:04 |
| *** dasm|off is now known as dasm | 14:05 | |
| dasm | o/ | 14:05 |
| Tengu | folks, care to add https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/857096 and https://review.opendev.org/c/openstack/tripleo-quickstart/+/857128 to your review queue? | 14:11 |
| reviewbot | I could not add the review to Review List | 14:12 |
| reviewbot | I could not add the review to Review List | 14:12 |
| Tengu | uho | 14:13 |
| Tengu | that's not nice, reviewbot ;_; | 14:13 |
| ysandeep | Tengu, something wrong with bot, I will manually add to review list | 14:13 |
| Tengu | ysandeep: thanks! | 14:13 |
| Tengu | maybe that's because bot doesn't know me | 14:14 |
| Tengu | "who's that tengu here, really, nah, I don't know him, bleh" | 14:14 |
| Tengu | hmm I can't really vote on the oooq-extras patch, since I kind of took over your change.. | 14:15 |
| ysandeep | yes, good to have second pair of eyes + also for awareness so that ci folks knows what's going there | 14:16 |
| Tengu | yuo | 14:16 |
| Tengu | *yup | 14:16 |
| rlandy|ruck | pojadhav: hi | 14:16 |
| * ysandeep out, see everyone tomorrow | 14:16 | |
| *** ysandeep is now known as ysandeep|out | 14:16 | |
| rlandy|ruck | pojadhav: pls see marios email and update rr sheet | 14:16 |
| marios | please review when you have some time oooci empty criteria & promoter releases for zed bootstrap easy one https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45202 thanks | 14:52 |
| marios | amoralej: any idea when we can expect this to start populating http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos9-zed/component/tempest/b3/9b/ | 15:04 |
| marios | amoralej: it fails the container build because no package openstack-tempest-all @ https://logserver.rdoproject.org/37/45037/4/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-zed/b3dbe0d/logs/container-builds/006f1004-3603-4ae9-a186-f63829e4c848/base/os/tempest/tempest-build.log | 15:04 |
| amoralej | mmmm | 15:04 |
| amoralej | that's reverse proxy | 15:04 |
| amoralej | i'd say it shold just work | 15:04 |
| amoralej | ah Error: Unable to find a match: openstack-tempest-all | 15:05 |
| amoralej | that's bootstraping issue | 15:05 |
| amoralej | we should be able to fix it | 15:05 |
| amoralej | gimme a while | 15:05 |
| marios | amoralej: yeah i think the problem is empty repo @ http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos9-zed/component/tempest/b3/9b/ | 15:07 |
| marios | amoralej: thank you for your help | 15:07 |
| marios | amoralej: (image push worked thanks for fixing permissions) | 15:08 |
| * jm1 bbl | 15:10 | |
| marios | ah not empty amoralej but should have only one build there? | 15:12 |
| amoralej | http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos9-zed/component/tempest/b3/9b/b39bbcc0bc5be39cc4d9e3cf2addf1924c8df49a_e68950ea/ | 15:13 |
| amoralej | marios it's just that openstack-tempest-all is not built | 15:13 |
| rlandy|ruck | akahat|ruck: do we have any bug related to "Lost connection to MySQL server during query" seems to be happening a lot | 15:14 |
| marios | amoralej: well only for that one hash though the other 2 at http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos9-zed/component/tempest/b3/9b/ have the build | 15:15 |
| rlandy|ruck | jm1: ^^ see that one during your rr shift? | 15:15 |
| chandankumar | see ya people! | 15:15 |
| marios | o/ chandankumar | 15:15 |
| amoralej | marios, yes, that's because we have to disable to build it in CBS to break cyclic dep | 15:15 |
| marios | wait for me chandankumar first round on me ;) | 15:15 |
| marios | k thanks amoralej i'll stop harrassing you thakns for looking let me know if i need to do sthing? | 15:16 |
| akahat|ruck | rlandy|ruck, no. | 15:28 |
| * marios off | 15:29 | |
| *** marios is now known as marios|out | 15:31 | |
| pojadhav | rlandy|ruck, updated rr sheet | 16:04 |
| rlandy|ruck | pojadhav: thanks | 16:07 |
| rlandy|ruck | akahat|ruck: https://bugs.launchpad.net/tripleo/+bug/1990415 - we have a real bug on master/wallaby | 16:08 |
| afuscoar | Hi jm1: Can u please take a look to this one? https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44950 Thx. | 16:17 |
| dasm | rlandy|ruck: o/ i'm running podman containers on a new cs9 instance for cockpit. They're dying miserably after about 45 minutes, and I'm trying to figure out what's going on. | 16:22 |
| dasm | rlandy|ruck: do you know someone who can help me? | 16:22 |
| rlandy|ruck | dasm: jm1 is probably your friend here | 16:36 |
| rlandy|ruck | or frenzyfriday | 16:36 |
| frenzyfriday | rlandy|ruck, we have discussed this in today's review call. Need more +2 from either jm1 or dasm as it is an infra patch | 16:37 |
| rcastillo | dasm maybe make sure you're running system containers and not session containers, since those will get killed | 16:37 |
| frenzyfriday | okay, we have vote from dasm , now we need a +wf. I dont have wf rights | 16:38 |
| rlandy|ruck | frenzyfriday: meaning? | 16:38 |
| rlandy|ruck | dasm needs helping debugging | 16:38 |
| frenzyfriday | of sorry, I thought this is about Adrian's patch | 16:38 |
| frenzyfriday | looking | 16:38 |
| dasm | rcastillo: hmm... that might be it. i'm using ansible to spin them up. | 16:40 |
| *** jpena is now known as jpena|off | 16:40 | |
| rcastillo | dasm: try becoming root for those tasks, see if it helps | 16:40 |
| dasm | rcastillo: k, thanks for that. i'll see if that changes anything | 16:41 |
| dasm | rlandy|ruck: wrt Adrian's patch i don't have +2 but I tested it and it works. | 16:41 |
| rlandy|ruck | dasm: pls pass patch | 16:42 |
| dasm | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44950 | 16:42 |
| rlandy|ruck | I just gave jm1 merge rights there | 16:42 |
| rlandy|ruck | dasm: do we need that merged asap? | 16:43 |
| rlandy|ruck | jm1 might show up again to merge | 16:43 |
| rlandy|ruck | if he does not, I can do so | 16:43 |
| dasm | afuscoar: do you need grafana 9.1.5 asap? | 16:44 |
| rlandy|ruck | oh - this is ci config | 16:44 |
| dasm | afuscoar: ^^ | 16:44 |
| rlandy|ruck | ok - so I may need to merge it | 16:44 |
| dasm | rcastillo: i'm gonna keep an open session to see if that's the issue. if it won't get killed soon, you might be onto something | 16:45 |
| dasm | i'd prefer not running as a root, but if that's the only thing, we might do that | 16:45 |
| rcastillo | dasm lmk how that goes | 16:46 |
| dasm | ofc | 16:46 |
| rcastillo | I agree about running as root. We were running as root already with docker so we're not regressing at least ;) | 16:47 |
| dasm | but we're not progressing too ;) | 16:47 |
| rcastillo | docker -> podman | 16:47 |
| rcastillo | it's something | 16:47 |
| dasm | xD | 16:47 |
| rlandy|ruck | lunch - brb | 16:50 |
| dasm | rcastillo: journalctl logs are corresponding with this: user@1000.service: Deactivated successfully. | 16:52 |
| dasm | huh. seems like it's ongoing problem: https://lists.podman.io/archives/list/podman@lists.podman.io/thread/CQTZAORZXW6UWDNAV5TAWAV6PTDJFW6V/ | 16:54 |
| dasm | > "loginctl enable-linger" | 16:54 |
| dasm | fixed it | 16:54 |
| dasm | maybe? | 16:54 |
| dasm | https://github.com/containers/podman/blob/main/troubleshooting.md#21-a-rootless-container-running-in-detached-mode-is-closed-at-logout | 16:55 |
| rcastillo | dasm: oh, nice, that might do it | 16:56 |
| dasm | yeah, thanks for mentioning that rcastillo++ | 16:56 |
| dasm | i'm gonna try this | 16:56 |
| rcastillo | happy to help | 17:05 |
| afuscoar | dasm: sorry for not answering, we need it mainly to start the ruck & rover | 17:06 |
| afuscoar | In this case we need both https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44950/ and https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/428675 | 17:07 |
| dasm | ack | 17:08 |
| dasm | hmm... i'm having issues with code.eng: Proxy Error | 17:09 |
| dasm | i'm gonna give it a try later | 17:09 |
| afuscoar | Yeah it doesn't work right now | 17:10 |
| *** amoralej is now known as amoralej|off | 17:14 | |
| * akahat|ruck offline | 18:02 | |
| jm1 | frenzyfriday, afuscoar, rlandy|ruck: grafana patch is +w'ed | 19:01 |
| jm1 | frenzyfriday: grafana 9.1.5 looks so much cooler :D | 19:02 |
| afuscoar | Thank you very much, jm1 | 19:05 |
| rlandy|ruck | thanks | 19:06 |
| afuscoar | Now it's submitted. Perfect. Can you also check this one? https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/428675 | 19:09 |
| jm1 | afuscoar: proxy error when accessing the patch.. server issues? | 19:11 |
| afuscoar | Oh, it's still happening | 19:12 |
| afuscoar | It has been like this since a lot | 19:12 |
| afuscoar | is there a way to report it? | 19:12 |
| afuscoar | Idk the procedure in this case | 19:12 |
| jm1 | rlandy|ruck: marked https://bugs.launchpad.net/tripleo/+bug/1990415 as a duplicate of https://bugs.launchpad.net/tripleo/+bug/1987092 | 19:17 |
| rlandy|ruck | will look in a sec | 19:18 |
| jm1 | afuscoar: looks like #rhos-ops is complaining about code.eng as well ;) | 19:18 |
| rlandy|ruck | yep - down | 19:35 |
| rlandy|ruck | jm1: that's annoying failure | 19:35 |
| rlandy|ruck | we need to get the bottom of it | 19:35 |
| jm1 | rlandy|ruck: its on cix board so it will be taken care of... XD | 19:36 |
| rlandy|ruck | jm1: yeah - need to ping damien tomorrow | 19:36 |
| rlandy|ruck | and tell him I'm desperate | 19:36 |
| rlandy|ruck | for his fix | 19:37 |
| jm1 | rlandy|ruck: but agree, a looooooooot of errors are related to mysql (although not all are galera errors) | 19:37 |
| rlandy|ruck | yeah - so I wasn't sure if it was all the same | 19:37 |
| rlandy|ruck | also fs064 used to fail a lot earlier | 19:37 |
| rlandy|ruck | so now we are seeing it consistently get to deploy | 19:37 |
| rlandy|ruck | and fail a lot there | 19:37 |
| rlandy|ruck | fs039 passes | 19:38 |
| rlandy|ruck | though | 19:38 |
| rlandy|ruck | interesting | 19:38 |
| jm1 | rlandy|ruck: fs39 passes consistently? | 19:38 |
| jm1 | rlandy|ruck: like always? | 19:38 |
| rlandy|ruck | consistently is pushing it | 19:38 |
| jm1 | rlandy|ruck: 😂 | 19:38 |
| jm1 | rlandy|ruck: 😠| 19:38 |
| rlandy|ruck | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master&skip=0 | 19:39 |
| rlandy|ruck | the three failures on top are the hithub access | 19:39 |
| rlandy|ruck | github | 19:39 |
| rlandy|ruck | I mentioned earlier | 19:39 |
| jm1 | rlandy|ruck: using the "right" filters (only periodic jobs) and voila it looks much worse ^^ https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master&pipeline=openstack-periodic-integration-main&skip=0 | 19:40 |
| rlandy|ruck | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby&skip=0 getting better | 19:40 |
| rlandy|ruck | those retries are my next mission | 19:41 |
| jm1 | rlandy|ruck: wow! fs39 on wallaby is reaaly unstable https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby&pipeline=openstack-periodic-integration-stable1&skip=0 | 19:41 |
| rlandy|ruck | the image build failed | 19:42 |
| rlandy|ruck | so the job had to get testproject rerun | 19:42 |
| rlandy|ruck | twice | 19:42 |
| jm1 | rlandy|ruck: i am really looking forward to what frenzyfriday's log analysis tools will reveal | 19:43 |
| jm1 | rlandy|ruck: hopefully this will give us some visibilities about intermittent failures | 19:43 |
| rlandy|ruck | I'm hoping to get the failures down deploy/tempest | 19:43 |
| rlandy|ruck | and then work on thoes | 19:43 |
| rlandy|ruck | get rid of the install ones | 19:43 |
| rlandy|ruck | we don't see the IPA install failures anymore | 19:44 |
| rlandy|ruck | which is telling | 19:44 |
| jm1 | rlandy|ruck: interesting how you got all the time to work on the bugs and actually solve them. i was always rerunning and reading logs.. | 19:44 |
| rlandy|ruck | because gate failures were minimal | 19:44 |
| rlandy|ruck | lucky week | 19:44 |
| jm1 | rlandy|ruck: no ipa errors? did you dns patch solve that? | 19:44 |
| rlandy|ruck | hopeful it did | 19:46 |
| jm1 | rlandy|ruck: how did you get the idea that the reason could be dns? | 19:51 |
| rlandy|ruck | jm1: ade lee told me | 19:51 |
| jm1 | rlandy|ruck: its not that these ipa errors were really telling us anything except that dnf could not install packages for whatever reason | 19:51 |
| rlandy|ruck | yeah - wouldn't have known either | 19:52 |
| rlandy|ruck | but he did | 19:52 |
| rlandy|ruck | I guess that is why he is a security expert | 19:52 |
| jm1 | rlandy|ruck: if this was really dns related then it did not only solve the ipa errors but also a bunch of other bugs which we have since at least march https://bugs.launchpad.net/tripleo/+bug/1965124 | 19:53 |
| jm1 | rlandy|ruck: we will see on my next rr shift :D | 19:53 |
| jm1 | rlandy|ruck: so in three weeks.. | 19:54 |
| jm1 | anyway, i am out for today | 19:55 |
| * jm1 have a nice evening #oooq :) | 19:55 | |
| afuscoar1 | Good evening to everyone! | 20:05 |
| rlandy|ruck | bye | 20:11 |
| dasm | rcastillo: 2h later and podman is still running. seems like you hit the jackpot :) | 21:15 |
| rcastillo | nice dasm++ | 21:16 |
| rcastillo | I've run into that issue in the past with rootless | 21:16 |
| dasm | good to know. | 21:16 |
| *** rlandy|ruck is now known as rlandy|bbl | 22:05 | |
| * rcastillo out for today | 23:12 | |
| dasm | rcastillo: o/ | 23:28 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!