| *** dviroel|rover is now known as dviroel|out | 00:19 | |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 01:27 | |
| *** bhagyashris is now known as bhagyashris|out | 03:30 | |
| * bhagyashris|out out for the day will back for planning meeting | 03:34 | |
| *** pojadhav is now known as pojadhav|ruck | 04:19 | |
| *** ysandeep|out is now known as ysandeep | 04:53 | |
| *** ykarel|away is now known as ykarel | 05:19 | |
| chandankumar | dpawlik: hello | 05:38 |
|---|---|---|
| chandankumar | dpawlik: can we upload the CentOS stream 9 cloud image on image server and give a generic name? | 05:38 |
| *** pojadhav is now known as pojadhav|ruck | 06:06 | |
| *** ykarel_ is now known as ykarel | 06:13 | |
| *** amoralej|off is now known as amoralej | 06:37 | |
| dpawlik | chandankumar: sure | 06:38 |
| dpawlik | chandankumar: CentOS-Stream-GenericCloud-9-20211013.3.x86_64.qcow2 | 06:39 |
| dpawlik | is ok? | 06:39 |
| dpawlik | chandankumar: done | 06:43 |
| chandankumar | dpawlik: http://images.rdoproject.org/CentOS-9-Stream-x86_64-GenericCloud.qcow2 seems ok | 06:46 |
| *** pojadhav|ruck is now known as pojadhav|afk | 07:11 | |
| *** jpena|off is now known as jpena | 07:32 | |
| chandankumar | sshnaidm: \o, regarding this https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/813812 where we can pass the ansible version while building spec file | 08:03 |
| chandankumar | ? | 08:04 |
| sshnaidm | chandankumar, we don't pass it, why? | 08:04 |
| sshnaidm | chandankumar, whatever it will be - it will work | 08:04 |
| chandankumar | sshnaidm: reason I am asking because since it is a jinja template I am not sure how the https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/813812/1/roles/build-test-packages/templates/specs/ansible-netcommon-build.spec.j2#13 will convert to BuildRequires: <to particular ansible version> | 08:09 |
| chandankumar | Do we have a testproject with ansible-core? | 08:09 |
| sshnaidm | chandankumar, BuildRequires: accepts boolean, like here: https://src.fedoraproject.org/rpms/ansible-collection-community-general/blob/rawhide/f/ansible-collection-community-general.spec#_17 | 08:10 |
| chandankumar | sshnaidm: ah, awesome that i was not knowing that, thanks! | 08:11 |
| sshnaidm | chandankumar, sure, it's a new format for collections we'll use now | 08:11 |
| *** ykarel is now known as ykarel|lunch | 08:14 | |
| *** ysandeep is now known as ysandeep|lunch | 08:35 | |
| frenzy_friday | hey arxcruz, sshnaidm Is there a way to override variables of a role in a task where we use include_role? I was trying like this https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/813853/2/roles/collect_logs/molecule/sova/converge.yml but that is not correct | 08:57 |
| sshnaidm | frenzy_friday, this should work | 09:02 |
| frenzy_friday | oh, my bad sorry, there was a typo | 09:16 |
| *** ykarel|lunch is now known as ykarel|lunc | 09:27 | |
| *** ykarel|lunc is now known as ykarel | 09:27 | |
| *** ysandeep|lunch is now known as ysandeep | 09:39 | |
| chandankumar | Any more reviews needs attension? | 09:53 |
| chandankumar | ysandeep: can we get a downstream run using this patch https://review.opendev.org/c/openstack/tripleo-quickstart/+/813624 ? | 09:57 |
| ysandeep | chandankumar, ack I will trigger a testproject | 09:58 |
| chandankumar | ysandeep: can you take a look at these two also https://review.opendev.org/c/openstack/tripleo-operator-ansible/+/813138 and https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/813137 when free | 10:00 |
| -opendevstatus- NOTICE: zuul was stuck processing jobs and has been restarted. pending jobs will be re-enqueued | 10:01 | |
| ysandeep | chandankumar: ack | 10:01 |
| chandankumar | Tengu: can we remove -1 vote on this https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/809987 ? | 10:06 |
| ysandeep | chandankumar, Tengu is on PTO | 10:07 |
| chandankumar | arxcruz: sshnaidm please have a look https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/809987 when free, thanks! | 10:07 |
| chandankumar | ysandeep: ah, thanks for the info. | 10:07 |
| chandankumar | ysandeep: pojadhav|afk please test this image build change for downstream https://review.opendev.org/c/openstack/tripleo-ci/+/808177 (I am sure it will not break downstream), just for sanity, thanks! | 10:08 |
| sshnaidm | chandankumar, done | 10:10 |
| chandankumar | sshnaidm: thanks ! | 10:10 |
| *** rlandy is now known as rlandy|ruck | 10:37 | |
| rlandy|ruck | soniya29: hello - I don;t see any request | 10:39 |
| rlandy|ruck | soniya29: did pojadhav|afk say where she sent it? | 10:39 |
| rlandy|ruck | dviroel|out: pojadhav|afk: hi - let's touch base when you get back in | 10:42 |
| ysandeep | chandankumar, testprojects: https://code.engineering.redhat.com/gerrit/c/testproject/+/190672 and https://code.engineering.redhat.com/gerrit/c/testproject/+/211643 | 10:44 |
| chandankumar | ysandeep: thanks!, will take a look | 10:50 |
| chandankumar | rlandy|ruck: dviroel|out pojadhav|afk has passed following test projects https://review.rdoproject.org/r/c/testproject/+/33791 https://review.rdoproject.org/r/c/testproject/+/33582 | 10:52 |
| chandankumar | and https://review.rdoproject.org/r/c/testproject/+/35132 | 10:53 |
| chandankumar | when she went afk | 10:54 |
| rlandy|ruck | chandankumar: ok - thanks | 10:54 |
| rlandy|ruck | I am looking at the ipa and fs039 failures | 10:54 |
| chandankumar | I have added notes in testproject comments | 10:54 |
| rlandy|ruck | dviroel|out: pojadhav|afk: https://bugs.launchpad.net/tripleo/+bug/1947144 | 11:01 |
| rlandy|ruck | fyi - we can talk about the suggested patch | 11:01 |
| *** pojadhav|afk is now known as pojadhav | 11:08 | |
| pojadhav | rlandy|ruck, actually before taking belongings related to laptop, we need approval on mail from manager, thats why we put a mail. | 11:09 |
| *** dviroel|out is now known as dviroel|rover | 11:09 | |
| *** pojadhav is now known as pojadhav|ruck | 11:09 | |
| rlandy|ruck | pojadhav: approval sent in email | 11:10 |
| pojadhav|ruck | rlandy|ruck, yeah.. thank you! | 11:13 |
| rlandy|ruck | ysandeep: pls review https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/813137 | 11:14 |
| pojadhav|ruck | rlandy|ruck, dviroel|rover : give me some time, will grab a coffee then will sync up | 11:14 |
| ysandeep | rlandy|ruck, ack, its on my review list | 11:15 |
| rlandy|ruck | ysandeep: w-1'ed it until we have the right test results there | 11:15 |
| marios | brb | 11:20 |
| jpodivin | I was wondering, if it came to that, would would the best way to add a broader version of https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/validations.yml Broader as in targeting more hosts than just undercloud. | 11:23 |
| jpodivin | s/would would/what would be/ | 11:23 |
| chandankumar | dpawlik: thanks for the images. | 11:24 |
| dpawlik | np chandankumar | 11:26 |
| *** jpena is now known as jpena|lunch | 11:27 | |
| rlandy|ruck | ysandeep: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/813137 - any downstream concerns here? | 11:30 |
| rlandy|ruck | pojadhav|ruck: dviroel|rover: need 5 more minutes - will ping you for sync shortly | 11:32 |
| dviroel|rover | ok | 11:32 |
| ysandeep | rlandy|ruck, just that till python-tripleo patch reaches tripleo-ci-testing our ovb jobs will fail on 17, but its seems safe to merge... we recently promoted.. | 11:32 |
| rlandy|ruck | ysandeep - ok | 11:33 |
| ysandeep | rlandy|ruck, i think we can merge this one | 11:33 |
| rlandy|ruck | done | 11:33 |
| ysandeep | thanks! | 11:34 |
| pojadhav|ruck | rlandy|ruck, dviroel|rover : i am back.. can we sync up..? | 11:34 |
| pojadhav|ruck | rlandy|ruck, ack.. sorry I didnt read your before msg.. | 11:35 |
| chandankumar | pojadhav|ruck: please have a look https://review.rdoproject.org/r/c/rdo-jobs/+/36134 when free, thanks! | 11:36 |
| pojadhav|ruck | chandankumar, sure | 11:36 |
| ysandeep | chandankumar, downstream image build patch is happy https://code.engineering.redhat.com/gerrit/c/testproject/+/190672 | 11:39 |
| chandankumar | ysandeep: awesome, thansk! | 11:39 |
| rlandy|ruck | dviroel|rover: pojadhav|ruck: https://meet.google.com/rtz-kedr-qdi?pli=1&authuser=0 | 11:43 |
| rlandy|ruck | dviroel|rover: pojadhav|ruck: https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/ipa-multinode/tasks | 11:50 |
| rlandy|ruck | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master/955ad83/logs/38.102.83.221/var/log/ipaserver-install.log.txt.gz | 11:52 |
| *** akahat is now known as akahat|afk | 11:54 | |
| jfrancoa | hey marios! let me know if you have a moment to ask you a question about https://review.opendev.org/c/openstack/tripleo-ansible/+/812948 | 11:55 |
| marios | jfrancoa: o/ | 11:56 |
| marios | jfrancoa: wana gmeet? | 11:56 |
| jfrancoa | marios: sure! why not! | 11:56 |
| marios | https://meet.google.com/mpv-azho-ngq jfrancoa | 11:57 |
| *** ysandeep is now known as ysandeep|afk | 11:58 | |
| *** amoralej is now known as amoralej|lunch | 11:59 | |
| dviroel|rover | pojadhav|ruck: will testproject wallaby and train now, after that i'll focus on the logs to search for issues | 12:02 |
| dviroel|rover | pojadhav|ruck: s/wallaby/victoria ;) | 12:02 |
| pojadhav|ruck | dviroel|rover, ack share me those reviews as well. | 12:03 |
| pojadhav|ruck | :) | 12:03 |
| frenzy_friday | hey arxcruz, can we merge this one https://review.opendev.org/c/opendev/elastic-recheck/+/813795 ? | 12:11 |
| *** jpena|lunch is now known as jpena | 12:22 | |
| chandankumar | rlandy|ruck: meeting? | 12:31 |
| rlandy|ruck | yep - joining | 12:32 |
| rlandy|ruck | just go off recruiting call | 12:32 |
| pojadhav|ruck | chandankumar, rlandy|ruck, ysandeep|afk, marios : kindly review when free https://review.rdoproject.org/r/c/rdo-jobs/+/36134 thanks! | 12:35 |
| *** amoralej|lunch is now known as amoralej | 12:49 | |
| *** ysandeep|afk is now known as ysandeep | 12:55 | |
| *** ysandeep is now known as ysandeep|brb | 12:57 | |
| *** bhagyashris|out is now known as bhagyashris|mtg | 13:00 | |
| bhagyashris|mtg | akahat|afk, chandankumar dviroel|rover rlandy|ruck planning meeting time | 13:00 |
| arxcruz | frenzy_friday: i thought i have +2 rights there, but i don't | 13:01 |
| bhagyashris|mtg | rlandy|ruck, chandankumar around? | 13:02 |
| bhagyashris|mtg | akahat|afk, ^ | 13:02 |
| *** ysandeep|brb is now known as ysandeep | 13:03 | |
| *** akahat|afk is now known as akahat | 13:10 | |
| marios | dviroel|rover: o/ did you see that https://review.opendev.org/c/openstack/tripleo-repos/+/813975 - kinda bad that the packaging is broken just before we demo it ;) | 13:22 |
| marios | dviroel|rover: right now with setup.py install i need sudo to use the commands | 13:22 |
| dviroel|rover | marios: yeah, i saw your patch. i wasn't pay attention to that setup part | 13:27 |
| marios | dviroel|rover: same | 13:28 |
| ade_lee | jpena, hey - when you get a chance, I think more work is needed on the fips enabled image. | 13:36 |
| ade_lee | jpena, as far as I can tell, the image does not have fips=1 on the kernel command line - which is what is needed. | 13:36 |
| ade_lee | jpena, you can see what I did for instance. when I was testing infrared -- https://review.gerrithub.io/c/redhat-openstack/infrared/+/515913/36/plugins/virsh/tasks/vms_1_create_disk.yml | 13:38 |
| jpena | ade_lee: in a meeting, will check in ~1h (sorry!) | 14:16 |
| ade_lee | jpena, ack no worries - I'm about to start my meeting-a-rama now too | 14:16 |
| *** ykarel is now known as ykarel|away | 14:21 | |
| dviroel|rover | marios: https://review.opendev.org/c/openstack/tripleo-quickstart/+/812672/2#message-cea064acca069b3fb505e21f1425c2e1e38c1563 | 14:26 |
| chandankumar | marios: are you around tomorrow? | 14:33 |
| dviroel|rover | pojadhav|ruck: https://review.rdoproject.org/zuul/status#35174 | 14:33 |
| dviroel|rover | looks like it is going well so far | 14:34 |
| pojadhav|ruck | dviroel|rover, yeah.. thanks! | 14:34 |
| chandankumar | rlandy|ruck: dviroel|rover pojadhav|ruck https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/782362/17#message-ca1e069403c513e5832f94bfda10db85b69db5d0 | 14:35 |
| marios | chandankumar: yes | 14:35 |
| rlandy|ruck | chandankumar: looks like it worked! | 14:36 |
| chandankumar | marios: I will be around tomorrow | 14:36 |
| rlandy|ruck | chandankumar: any possible impact in other places? ok to merge? | 14:37 |
| chandankumar | rlandy|ruck: no impact | 14:37 |
| chandankumar | rlandy|ruck: good news now the fallback registry will work | 14:37 |
| rlandy|ruck | chandankumar:pls revot eto +2 and we can merge | 14:37 |
| chandankumar | if quay.ceph.io is also goes down | 14:37 |
| dviroel|rover | chandankumar: rlandy|ruck ++ | 14:38 |
| rlandy|ruck | dviroel|rove: pls vote | 14:38 |
| jpena | ade_lee: https://softwarefactory-project.io/r/c/config/+/22902 (oops) | 14:39 |
| rlandy|ruck | jpena: thanks | 14:40 |
| ade_lee | jpena, thanks | 14:41 |
| rlandy|ruck | chandankumar: woohoo https://review.opendev.org/c/openstack/tripleo-common/+/813745 w+ | 14:41 |
| rlandy|ruck | chandankumar: any other patches need reviews/work/watching? | 14:42 |
| chandankumar | rlandy|ruck: nope | 14:42 |
| chandankumar | rlandy|ruck: arxcruz we need to make sure we have all the ceph related containers on rdo registry from train onwards | 14:42 |
| chandankumar | rest will be fine | 14:42 |
| rlandy|ruck | arxcruz: ^^ do we have those already? | 14:43 |
| frenzy_friday | hey chandankumar ysandeep marios are you guys already working on c9 undercloud and multinode jobs? Or can I take those? | 14:44 |
| chandankumar | frenzy_friday: I have not picked it | 14:45 |
| ysandeep | frenzy_friday, sure you can pick them, but would be good if we can get first pass on standalone first | 14:45 |
| rlandy|ruck | frenzy_friday: are you all set with patches to fix collect-logs? | 14:46 |
| rlandy|ruck | anything you need reviewed? | 14:46 |
| chandankumar | frenzy_friday: I have a work for you if interested? in CS9 rdo nodepool image, the user is zuul-worker, we need to change it to zuul | 14:46 |
| chandankumar | frenzy_friday: rlandy|ruck can help you on that. | 14:46 |
| frenzy_friday | chandankumar, yep | 14:48 |
| frenzy_friday | rlandy|ruck, yes sort of - I checked in https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/813853 It looked like the job is fixed, I tried to abandon and create a clean patch https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/813988 with the exact same content - it failed. So restoring the abandoned patch now | 14:49 |
| rlandy|ruck | frenzy_friday: k - ping when you want reviews there | 14:50 |
| marios | frenzy_friday: i am not | 14:50 |
| marios | ack dviroel|rover will check | 14:51 |
| arxcruz | rlandy|ruck: chandankumar not yet, but i | 14:52 |
| arxcruz | but i'm working on that | 14:52 |
| chandankumar | arxcruz: awesome ++ | 14:53 |
| rlandy|ruck | arxcruz: we merged the patch | 14:53 |
| rlandy|ruck | well we workflowed it | 14:53 |
| arxcruz | rlandy|ruck: which one? the one from chandankumar ? | 14:53 |
| rlandy|ruck | yeah | 14:53 |
| rlandy|ruck | should be hold on that? | 14:53 |
| rlandy|ruck | arxcruz: chandankumar: ^^ until arxcruz is done with the copy? | 14:53 |
| chandankumar | rlandy|ruck: ok for me | 14:54 |
| arxcruz | rlandy|ruck: ykarel|away -1 | 14:54 |
| arxcruz | chandankumar: rlandy|ruck sorry, i thought it was another problem, you want me to copy ceph containers to rdo-registry right? | 14:55 |
| arxcruz | just rdo-registry or also quay.io ? on which namespace? the same as tripleo$release ? | 14:55 |
| chandankumar | arxcruz: yes, we just want to make sure all the non-tripleo containers with ceph namespace exists there | 14:55 |
| chandankumar | arxcruz: yes and also in quay | 14:55 |
| chandankumar | rdo registry namespace -> ceph | 14:56 |
| arxcruz | chandankumar: yes, but in which namespace? shall we have a trunk.registry/ceph/$ceph-containers | 14:56 |
| arxcruz | chandankumar: no problem for rdo, but i don | 14:56 |
| arxcruz | chandankumar: no problem for rdo, but i dont think we own quay.io/ceph | 14:56 |
| frenzy_friday | chandankumar, which repo is CS9 rdo nodepool image ? | 14:56 |
| chandankumar | rlandy|ruck: arxcruz https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/container-build/tasks/non_tripleo_containers.yml#L55 | 14:57 |
| rlandy|ruck | chandankumar: marios: https://review.opendev.org/c/openstack/tripleo-ci/+/808177 - +1'ed because of few comments in line - ok b me to merge this (when depends stuff is ready) and fix afterwards | 14:57 |
| frenzy_friday | rdoproject config repo? | 14:57 |
| chandankumar | frenzy_friday: can you check internal conversation | 14:58 |
| chandankumar | on #rhos-opes | 14:59 |
| rlandy|ruck | jpena: ade_lee: pld review https://code.engineering.redhat.com/gerrit/c/openstack/sf-config/+/281397 | 14:59 |
| rlandy|ruck | anything more needed here? | 15:00 |
| *** bhagyashris|mtg is now known as bhagyashris|away | 15:03 | |
| jpena | rlandy|ruck: just added a comment about a missing package | 15:03 |
| marios | rlandy|ruck: ack we need the depends-on first... yes to merging. there is still one outstanding issue with image_sanity for the overcloud-full job but we should merge this and i can iterate if there is more needed | 15:03 |
| ade_lee | rlandy|ruck, jpena ack thanks - will check soon - gotta continue my meeting-a-rama | 15:03 |
| rlandy|ruck | jpena: thanks - will update | 15:04 |
| arxcruz | chandankumar: any chance you have the list of ceph containers ? | 15:07 |
| chandankumar | arxcruz: https://opendev.org/openstack/tripleo-common/src/branch/master/container-images/tripleo_containers.yaml#L166 | 15:08 |
| chandankumar | arxcruz: just change the branch till train | 15:08 |
| chandankumar | the version varies | 15:08 |
| *** amoralej is now known as amoralej|off | 15:09 | |
| * pojadhav|ruck leaving for the day. | 15:20 | |
| *** pojadhav|ruck is now known as pojadhav|out | 15:20 | |
| chandankumar | see ya tomorrow! | 15:24 |
| arxcruz | chandankumar: https://paste.opendev.org/show/810012/ | 15:24 |
| arxcruz | rlandy|ruck: ^ | 15:24 |
| rlandy|ruck | arxcruz: cool | 15:28 |
| * dviroel|rover lunch | 15:29 | |
| rlandy|ruck | arxcruz: what's the story on the quay side? | 15:29 |
| sshnaidm | frenzy_friday, is molecule job broken on collect-logs? | 15:29 |
| frenzy_friday | sshnaidm, yes, this happened: https://bugs.launchpad.net/tripleo/+bug/1947133 | 15:29 |
| arxcruz | rlandy|ruck: quay already have these on ceph namespace | 15:30 |
| arxcruz | i copy from there | 15:30 |
| arxcruz | but some of those were missing on rdo.registry | 15:30 |
| rlandy|ruck | arxcruz: great - thanks | 15:30 |
| frenzy_friday | sshnaidm, I am trying to fix it here https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/813853 - it passed after patchset4 but the same stuf broke here https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/813988 | 15:30 |
| sshnaidm | frenzy_friday, so which patch should solve it? | 15:31 |
| frenzy_friday | sshnaidm, https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/813988 should iseally, but it is failing | 15:34 |
| sshnaidm | frenzy_friday, it complains about wrong var name | 15:36 |
| sshnaidm | frenzy_friday, you can just exclude this rule from ansible-lint check | 15:36 |
| frenzy_friday | sshnaidm, what is the right way to override https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/roles/collect_logs/tasks/sova.yml#L19 this variable in https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/roles/collect_logs/molecule/sova/converge.yml#L5 ? | 15:38 |
| sshnaidm | frenzy_friday, idk, but overriding it not correct, you put it in flat while this should be a "files: " dictionary: https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/roles/collect_logs/tasks/sova.yml#L16 | 15:40 |
| sshnaidm | that's why linter complains as well | 15:40 |
| sshnaidm | frenzy_friday, I'm not sure it helps tbh, what is the failure in this job? | 15:41 |
| frenzy_friday | sshnaidm, right!! I missed files! | 15:42 |
| sshnaidm | frenzy_friday, but it's just a parameter name to sova module | 15:43 |
| sshnaidm | frenzy_friday, it won't replace anything | 15:43 |
| sshnaidm | frenzy_friday, I think I know what is this.. | 15:45 |
| *** ysandeep is now known as ysandeep|dinner | 15:45 | |
| frenzy_friday | so in this job sova looks into all of console/errors/ironic...files for error pattern. And we are introducing a particular error in the console file | 15:45 |
| frenzy_friday | But if there are other genuine errors in the job those regexes will also get picked up by sova and the filename will not be the same as the one it is trying to assert | 15:46 |
| frenzy_friday | By changing the variables i am trying to make sova look only into the one file where we have injected our specific error | 15:46 |
| frenzy_friday | so that https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/roles/collect_logs/molecule/sova/verify.yml#L34 file will be of the same name | 15:47 |
| *** ysandeep|dinner is now known as ysandeep | 16:00 | |
| ysandeep | rlandy|ruck, dviroel|rover what's the current milestone ? | 16:02 |
| rlandy|ruck | ysandeep: yoga-1 | 16:03 |
| ysandeep | rlandy|ruck, thanks! | 16:03 |
| ysandeep | rlandy|ruck, can we delete the xena milestones or do we still need them? | 16:04 |
| rlandy|ruck | ysandeep: slagle ie the PTL - his choice | 16:04 |
| rlandy|ruck | we don't afaik | 16:05 |
| ysandeep | ack o/ | 16:05 |
| ysandeep | I will chat with him. | 16:05 |
| *** marios is now known as marios|out | 16:13 | |
| * rlandy|ruck lunch - brb | 16:29 | |
| dviroel|rover | cool, victoria is going to promote | 16:30 |
| Tengu | chandankumar: that patch was the one setting a timeout for sealert? if so, please, no, don't merge...... there's a better alternative -.- | 16:32 |
| *** dpawlik5 is now known as dpawlik | 16:37 | |
| *** ysandeep is now known as ysandeep|out | 16:39 | |
| * ysandeep|out out for the day, see you tomorrow o/ | 16:40 | |
| *** jpena is now known as jpena|off | 16:42 | |
| dviroel|rover | rlandy|ruck: victoria promoted | 17:25 |
| rlandy|ruck | dviroel|rover: nice | 17:25 |
| rlandy|ruck | dviroel|rover: want to chat about fs039 multinode-ipa | 17:25 |
| rlandy|ruck | or are you ok there? | 17:25 |
| dviroel|rover | for train: periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train is missing now | 17:25 |
| dviroel|rover | rlandy|ruck: i don't see any dns errors anymore | 17:25 |
| rlandy|ruck | dviroel|rover: sporadic | 17:26 |
| rlandy|ruck | per 1.1.1.1's moods | 17:26 |
| dviroel|rover | rlandy|ruck: but node tripleo.operator.tripleo_overcloud_node_introspect is also failing | 17:26 |
| rlandy|ruck | dviroel|rover: so that is fs035 train | 17:26 |
| rlandy|ruck | or any ovb on train | 17:26 |
| * dviroel|rover checks | 17:27 | |
| dviroel|rover | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train | 17:27 |
| dviroel|rover | yes, it is happening on fs039 too | 17:28 |
| dviroel|rover | there is one testproject job stuck on introspesction right now https://review.rdoproject.org/zuul/stream/47dd615909204f0498a50bbd2c63eb07?logfile=console.log | 17:28 |
| dviroel|rover | for fs039 train | 17:28 |
| * rlandy|ruck checks | 17:28 | |
| rlandy|ruck | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train | 17:29 |
| rlandy|ruck | hmm spotty | 17:29 |
| dviroel|rover | nomadic issue | 17:31 |
| rlandy|ruck | checking changes | 17:32 |
| rlandy|ruck | dviroel|rover: searching https://trunk.rdoproject.org/centos8-train/report.html for possible impact changes | 17:34 |
| rlandy|ruck | around 10/09 | 17:34 |
| rlandy|ruck | hard to tell as we don;t get a consistent failure | 17:37 |
| rlandy|ruck | dviroel|rover: wonder if this impacts ... https://github.com/openstack/tripleo-quickstart-extras/commit/32953e32e12c9bca14f5d21d33d72d3d833f02de | 17:39 |
| rlandy|ruck | maybe we should try that with just train | 17:40 |
| rlandy|ruck | or that may be after | 17:40 |
| rlandy|ruck | forget thay | 17:42 |
| rlandy|ruck | that | 17:42 |
| rlandy|ruck | mostly these error are infra related | 17:43 |
| rlandy|ruck | but we don't see them outside of train | 17:43 |
| rlandy|ruck | Introspection of node completed:67d8f525-0270-44cd-af9b-c700370e04fb. Status:FAILED. Errors:Introspection timeout | 17:47 |
| rlandy|ruck | Introspection of node completed:cf25eef3-71ba-4af3-9a42-5b982cb0a234. Status:FAILED. Errors:Introspection timeo | 17:47 |
| rlandy|ruck | only two of 4 nodes timed out | 17:47 |
| dviroel|rover | humm | 17:47 |
| dviroel|rover | tricky one | 17:50 |
| rlandy|ruck | `tripleo_overcloud_node_import_timeout`: (Integer) Time in seconds to wait for the command to complete. Default: 900 | 17:53 |
| rlandy|ruck | dviroel|rover: seeing of we can increase the timeout | 17:53 |
| rlandy|ruck | check that for introspect | 17:54 |
| rlandy|ruck | tripleo_overcloud_node_introspect_node_timeout: (Integer) Maximum timeout for node introspection. | 17:55 |
| rlandy|ruck | tripleo_overcloud_node_introspect_max_retries: (Integer) Maximum introspection retries. | 17:55 |
| rlandy|ruck | tripleo_overcloud_node_introspect_retry_timeout: (Integer) Maximum timeout between introspection retries. | 17:55 |
| dviroel|rover | looking at fs035 jobs, Friday 08 October introspection node was taking ~ 25 min | 18:00 |
| dviroel|rover | Thursday 14 October we have a successful run, but took 1h4m to complete the instrospection role | 18:01 |
| rlandy|ruck | I gather these are the defaults: | 18:03 |
| rlandy|ruck | 227 tripleo_overcloud_node_introspect_node_timeout: 60 | 18:03 |
| rlandy|ruck | 228 tripleo_overcloud_node_introspect_max_retries: 2 | 18:03 |
| rlandy|ruck | 229 tripleo_overcloud_node_introspect_retry_timeout: 120 | 18:03 |
| * rlandy|ruck tries | 18:03 | |
| dviroel|rover | btw, https://review.rdoproject.org/zuul/stream/47dd615909204f0498a50bbd2c63eb07?logfile=console.log that is running now finished the introspection with success, but took more than 1h | 18:03 |
| rlandy|ruck | dviroel|rover: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/814044 WIP: Test increasing the introspection timeout | 18:08 |
| rlandy|ruck | no idea if that will help | 18:09 |
| rlandy|ruck | we can testproject spin all the train OVB jobs with that | 18:09 |
| rlandy|ruck | and see if it helps | 18:09 |
| dviroel|rover | rlandy|ruck: we should try, yes, and dig why is taking so long now | 18:11 |
| rlandy|ruck | submitting test project | 18:11 |
| rlandy|ruck | dviroel|rover: https://review.rdoproject.org/r/c/testproject/+/36254 Test train OVB jobs | 18:12 |
| rlandy|ruck | let's see fallout there | 18:12 |
| * rlandy|ruck edits fs039 patch | 18:13 | |
| rlandy|ruck | https://logserver.rdoproject.org/32/35132/13/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/e5aba41/logs/undercloud/etc/resolv.conf.txt.gz | 18:15 |
| rlandy|ruck | nameserver 127.0.0.1 | 18:15 |
| rlandy|ruck | running unbound | 18:15 |
| rlandy|ruck | https://logserver.rdoproject.org/32/35132/13/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/e5aba41/logs/undercloud/etc/unbound/forwarding.conf.txt.gz | 18:16 |
| rlandy|ruck | dviroel|rover: and fs039 with more forwarders: https://review.rdoproject.org/r/c/testproject/+/36255 Test fs039 with added DNS forwarders | 18:23 |
| dviroel|rover | great, periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master failed on DNS again | 18:25 |
| rlandy|ruck | dviroel|rover: we need to change the dns vars there as well | 18:26 |
| rlandy|ruck | coming up | 18:26 |
| dviroel|rover | rlandy|ruck: found this https://opendev.org/openstack/tripleo-common/commit/8a5a07677684a06ec75248e818c403d2d1a67ba3 | 18:29 |
| dviroel|rover | but seems harmless | 18:30 |
| rlandy|ruck | looks like they hit the same issue | 18:31 |
| rlandy|ruck | dviroel|rover: one more ... https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/814047 | 18:45 |
| dviroel|rover | ++ | 18:47 |
| rlandy|ruck | dviroel|rover: https://review.rdoproject.org/r/c/testproject/+/36256 Test multinode-ipa with additional forwarders [NEW] | 18:48 |
| rlandy|ruck | that's all - I promise!! | 18:48 |
| * rlandy|ruck goes back to checing image reviews | 18:48 | |
| rlandy|ruck | dviroel|rover; pojadhav|out: fyi - the security component may start getting old due to multinode-ipa and fa039 | 18:54 |
| dviroel|rover | rlandy|ruck: ah, we have one green on fs039 train, 558abab3dd4bf7adff9a601aca52a916 will be promoted | 19:06 |
| rlandy|ruck | well on the upside, it will only break ovb - which is non-voting | 19:07 |
| rlandy|ruck | might hose the component lines | 19:07 |
| rlandy|ruck | which are hosed already | 19:07 |
| dviroel|rover | lets wait for DNS outputs, but looking at the logs, seems that if one of the DNS fails to answer the EDNS query, it fails | 19:16 |
| dviroel|rover | https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/07697ce/logs/supplemental/var/log/ipaserver-install.log.txt.gz | 19:16 |
| dviroel|rover | it fails == ipaserver install fails | 19:17 |
| rlandy|ruck | yeah | 19:31 |
| rlandy|ruck | I took that one out of the multinode-ipa test | 19:31 |
| rlandy|ruck | removing that one here | 19:31 |
| rlandy|ruck | updated | 19:32 |
| rlandy|ruck | retesting | 19:32 |
| dviroel|rover | rlandy|ruck: re https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/782362/ | 20:19 |
| dviroel|rover | https://f12c37c8a25912360378-40001c60cc2cc6a7bddcc7547f0d5486.ssl.cf5.rackcdn.com/782362/17/check/tripleo-ci-centos-8-content-provider/510b501/logs/undercloud/home/zuul/non_tripleo_containers.log | 20:20 |
| dviroel|rover | still empty | 20:20 |
| rlandy|ruck | dviroel|rover: yeah | 20:20 |
| dviroel|rover | maybe just copy everything from results | 20:20 |
| dviroel|rover | https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/782362/17/roles/container-build/tasks/non_tripleo_containers.yml#124 | 20:20 |
| dviroel|rover | wdyt? | 20:21 |
| dviroel|rover | everything from {{ non_tripleo }} | 20:21 |
| dviroel|rover | like was before | 20:21 |
| rlandy|ruck | dviroel|rover: sure - try it | 20:22 |
| rlandy|ruck | the var ... | 20:22 |
| dviroel|rover | makes more sense since we also want stderr just in case of failures | 20:22 |
| rlandy|ruck | non_tripleo.results | 20:23 |
| rlandy|ruck | yeah - o | 20:23 |
| rlandy|ruck | let's see if that fixes it | 20:23 |
| dviroel|rover | rlandy|ruck: https://logserver.rdoproject.org/55/36255/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-victoria/c9da12f/logs/supplemental/var/log/ipaserver-install.log.txt.gz | 21:16 |
| dviroel|rover | still failing on 1.1.1.1 | 21:16 |
| dviroel|rover | we might need to try openDns only | 21:17 |
| rlandy|ruck | dviroel|rover: yeah - go4it | 21:18 |
| rlandy|ruck | whatever works at this point | 21:19 |
| dviroel|rover | ok | 21:19 |
| rlandy|ruck | we used to use 8.8.8.8 | 21:20 |
| rlandy|ruck | but it rate limits | 21:20 |
| rlandy|ruck | and we get kicked off | 21:20 |
| dviroel|rover | oh, hope that opendns works for us | 21:30 |
| * dviroel|rover will be afk waiting for some job to run | 21:30 | |
| * dviroel|rover s/job/jobs | 21:30 | |
| *** dviroel|rover is now known as dviroel|rover|afk | 21:30 | |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 22:20 | |
| *** chem is now known as Guest2925 | 22:30 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!