| *** ysandeep is now known as ysandeep|afk | 01:33 | |
| *** ysandeep|afk is now known as ysandeep | 02:19 | |
| *** ysandeep is now known as ysandeep|afk | 04:15 | |
| *** soniya29 is now known as soniya29|ruck | 04:32 | |
| marios | o/ | 04:59 |
|---|---|---|
| *** ysandeep|afk is now known as ysandeep | 05:04 | |
| ysandeep | marios, o/ happy Friday | 05:04 |
| marios | \o/ | 05:08 |
| soniya29|ruck | marios, can we sync after 1 hr or so? | 05:23 |
| marios | soniya29|ruck: yeah sounds good and we should wait for frenzyfriday|ruck too | 05:24 |
| soniya29|ruck | okay..lets wait for frenzyfriday|ruck and then we can sync | 05:25 |
| * soniya29|ruck will be back in 15-20 mins | 05:26 | |
| *** soniya29|ruck is now known as soniya29|ruck|afk | 05:26 | |
| *** soniya29|ruck|afk is now known as soniya29|ruck | 06:00 | |
| jm1 | #oooq: happy friday folks :D | 06:12 |
| marios | soniya29|ruck: did you already create a new hackmd? | 06:37 |
| soniya29|ruck | marios, yeah | 06:37 |
| marios | soniya29|ruck: k please add to https://hackmd.io/2hB-P772SqyqDs0KKZzZEQ?view | 06:37 |
| *** amoralej|off is now known as amoralej | 06:38 | |
| soniya29|ruck | marios, done | 06:40 |
| marios | thx | 06:40 |
| *** ysandeep is now known as ysandeep|away | 07:01 | |
| marios | soniya29|ruck: lets sync in 25 mins and frenzyfriday|ruck can catch us or you can sync with her after as needed | 07:05 |
| soniya29|ruck | marios, ack | 07:05 |
| frenzyfriday|ruck | hey marios soniya29|ruck , sorry I forgot to create the hackmd yesterday. I'll be online in ~5 mins | 07:06 |
| marios | o/ frenzyfriday|ruck np soniya29|ruck started a new one and its in the rr index | 07:06 |
| soniya29|ruck | frenzyfriday|ruck, i have created new hackmd | 07:06 |
| frenzyfriday|ruck | ack, thanks | 07:07 |
| *** frenzyfriday|ruck is now known as frenzyfriday|rover | 07:07 | |
| marios | frenzyfriday|rover: soniya29|ruck: sent invite for ~20 mins time | 07:07 |
| soniya29|ruck | marios, ack | 07:08 |
| *** jpena|off is now known as jpena | 07:17 | |
| marios | please add to review list https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/859846 | 07:29 |
| frenzyfriday|rover | sorry, the bit disconnected again | 07:55 |
| frenzyfriday|rover | bot please add to review list https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/859846 | 07:56 |
| reviewbot | I have added your review to the Review list | 07:56 |
| marios | :) thanks | 07:57 |
| marios | reviewbot: sudo make me a sandwich | 07:58 |
| marios | :( | 07:58 |
| *** ysandeep|away is now known as ysandeep|lunch | 08:10 | |
| marios | arxcruz: thanks for https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/859846 | 08:31 |
| marios | arxcruz: but i think we still have a problem/needs further work | 08:31 |
| marios | arxcruz: basically all the 'master' skipped tests will need to be duplicated for zed | 08:31 |
| arxcruz | marios yes, i'm working on a script to do it | 08:32 |
| marios | arxcruz: eg https://logserver.rdoproject.org/37/45037/6/check/periodic-tripleo-ci-centos-9-scenario012-standalone-zed/9b35a00/logs/undercloud/var/log/tempest/stestr_results.html.gz --> https://opendev.org/openstack/openstack-tempest-skiplist/src/commit/c26ed9998964b4897e88a603211b45142bbcf4e1/roles/validate-tempest/vars/tempest_skip.yml#L649 | 08:32 |
| marios | arxcruz: ah great ... that was my next question do we have some way to do it cos the sed/grep with that will be hairy :D | 08:33 |
| marios | arxcruz: thank you! | 08:33 |
| * soniya29|ruck lunch break | 08:45 | |
| *** soniya29|ruck is now known as soniya29|ruck|lunch | 08:46 | |
| *** soniya29|ruck|lunch is now known as soniya29|ruck | 09:36 | |
| *** ysandeep|lunch is now known as ysandeep | 09:40 | |
| marios | food biab | 09:53 |
| * soniya29|ruck tea | 10:24 | |
| *** soniya29|ruck is now known as soniya29|ruck|afk | 10:24 | |
| *** rlandy|out is now known as rlandy | 10:34 | |
| marios | code.eng down ? | 10:36 |
| *** soniya29|ruck|afk is now known as soniya29|ruck | 10:36 | |
| soniya29|ruck | marios, yes | 10:36 |
| rlandy | frenzyfriday|rover: soniya29|ruck|afk: marios: guess you already did a sync | 10:36 |
| rlandy | frenzyfriday|rover: soniya29|ruck|afk: marios: would just want to go over CIX and 16.2 line | 10:36 |
| rlandy | which is behind | 10:36 |
| rlandy | marios: you can skip | 10:37 |
| marios | \o rlandy yes we had a call earlier | 10:37 |
| marios | rlandy: i can join np (we're waiting for the ceph fix to merge for one of those it was wf this morning) | 10:37 |
| soniya29|ruck | code.eng has scheduled outage today | 10:37 |
| marios | k seems back now | 10:37 |
| rlandy | working now | 10:38 |
| rlandy | RHEL9 RHOS 16.2 - promoted on 23 sep | 10:38 |
| rlandy | tp:- https://code.engineering.redhat.com/gerrit/c/testproject/+/407599 | 10:38 |
| rlandy | that tp is running 17 | 10:39 |
| soniya29|ruck | rlandy, nope, patches were moving to merge_Conflict state coz of outage | 10:40 |
| rlandy | https://review.opendev.org/c/openstack/tripleo-heat-templates/+/859444 still has to merge | 10:40 |
| rlandy | k just waiting for frenzyfriday|rover to return | 10:40 |
| rlandy | then we can do CIX | 10:41 |
| frenzyfriday|rover | rlandy, hey | 10:41 |
| rlandy | hi | 10:41 |
| rlandy | frenzyfriday|rover: soniya29|ruck: marios(optional): https://meet.google.com/pjm-ohvh-jsb?pli=1&authuser=0 | 10:41 |
| frenzyfriday|rover | I am checking fs20 wallaby tempest thing | 10:41 |
| rlandy | to go trough CIX | 10:41 |
| marios | ack rlandy ok i'll skip and ping me if you want me to join | 10:42 |
| *** ysandeep is now known as ysandeep|brb | 10:45 | |
| rlandy | marios: https://trello.com/c/6QZA1XT2/2739-cixlp1991240tripleociproa-failed-loading-etc-yumreposd-deloreanrepo - you commented here that the fix was a revert | 10:46 |
| rlandy | we still need to fix victoria | 10:46 |
| rlandy | pojadhav: ^^ | 10:46 |
| rlandy | let's discuss at review time | 10:46 |
| rlandy | frenzyfriday|rover: arxcruz: added you to card: https://trello.com/c/NHPWa2I2/2740-cixlp1991232tripleociproa-insert-headers-is-not-a-valid-option-for-a-tcp-protocol-listener | 10:50 |
| owalsh_ | frenzyfriday|rover: hey, is something broken in the wallaby ffu job? | 10:58 |
| owalsh_ | https://zuul.opendev.org/t/openstack/build/6f3c4e9f98cb4bbd9c5d96732b49165c | 10:58 |
| marios | owalsh_: yeah but should be fixed now (we reverted something yesterday ) | 10:58 |
| owalsh_ | marios: ack, thanks | 10:58 |
| marios | owalsh_: top results green now at https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-undercloud-ffu-wallaby | 10:59 |
| marios | owalsh_: np | 10:59 |
| frenzyfriday|rover | rlandy, were you talking about https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/849113/1/roles/validate-tempest/vars/tempest_skip.yml ? I'll check and add this again to skiplist | 11:06 |
| *** ysandeep|brb is now known as ysandeep | 11:13 | |
| marios | anyone want to discuss reviews now is the time :) | 11:15 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 11:15 |
| soniya29|ruck | rlandy, still not able to rerun the jobs in code.eng..may be outage effect | 11:23 |
| pojadhav | rlandy, ack | 11:28 |
| ysandeep | marios, https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/427471 +wed, lgtm | 11:30 |
| pojadhav | rlandy, marios : have you guys discussed what things needs to be done to have emit release changes wrt to removal of victoria/ussuri ?? sorry was afk for some important work. | 11:31 |
| marios | thank you ysandeep | 11:33 |
| marios | pojadhav: no not yet we'll have to revisit the removal again | 11:33 |
| marios | pojadhav: and make sure we test the ffu job this time (my bad i should have suggested it really... but ... hindsight ;)) | 11:33 |
| pojadhav | marios, sure will do this time | 11:34 |
| rlandy | marios: w+ed https://review.rdoproject.org/r/c/rdo-jobs/+/45248 | 11:34 |
| rlandy | pojadhav: marios: I left notes on the bug where the error was | 11:34 |
| rlandy | ie: what was being picked up incorrectly | 11:34 |
| pojadhav | rlandy, ack | 11:34 |
| marios | great thanks rlandy | 11:35 |
| rlandy | ysandeep: dviroel: I have a conflicting meeting at 1:30 UTC | 11:40 |
| rlandy | ysandeep: dviroel: is there anything we need to discuss? want me to move to meeting or do you want to meet w/o me? | 11:40 |
| ysandeep | dviroel, I am continuing my work to add github action job, do you want to discuss anything? | 11:42 |
| dviroel | rlandy: ysandeep and I are in sync already. We can skip and share new things in another moment | 11:43 |
| rlandy | ok | 11:45 |
| rlandy | soniya29|ruck; hi - pls get the 17.1 on rhel-9 reruns going so we can see the results | 11:49 |
| * dviroel need to go afk a bit - quick med appt | 11:49 | |
| soniya29|ruck | rlandy, ack | 11:49 |
| *** dviroel is now known as dviroel|afk | 11:50 | |
| soniya29|ruck | rlandy: code.eng is complaining - this change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository | 11:54 |
| rlandy | soniya29|ruck: it's a testproject change? | 12:00 |
| soniya29|ruck | yup | 12:00 |
| rlandy | soniya29|ruck: if so, create a new testproject patch | 12:00 |
| rlandy | start from scratch | 12:00 |
| rlandy | the repo changed | 12:00 |
| rlandy | there is a no-op job in the default now | 12:00 |
| soniya29|ruck | okay | 12:00 |
| *** amoralej is now known as amoralej|lunch | 12:27 | |
| soniya29|ruck | rlandy, https://code.engineering.redhat.com/gerrit/c/testproject/+/430167 | 12:31 |
| rlandy | soniya29|ruck: thank you | 12:41 |
| arxcruz | rlandy i update https://trello.com/c/NHPWa2I2/2740-cixlp1991232tripleociproa-insert-headers-is-not-a-valid-option-for-a-tcp-protocol-listener with my findings took me a while, i can also be on the next cix to explain better what's happening | 12:49 |
| rlandy | arxcruz: thanks ... frenzyfriday|rover: ^^ pls review | 12:53 |
| soniya29|ruck | rlandy, i have updated the bug - https://bugs.launchpad.net/tripleo/+bug/1991093 with my findings | 13:05 |
| *** dviroel|afk is now known as dviroel | 13:06 | |
| rlandy | frenzyfriday|rover: can you check the master tempest and compute components? | 13:22 |
| frenzyfriday|rover | rlandy, ack, checking | 13:24 |
| *** amoralej|lunch is now known as amoralej | 13:31 | |
| * soniya29|ruck dinner | 13:38 | |
| *** soniya29|ruck is now known as soniya29|ruck|dinner | 13:38 | |
| *** dasm|off is now known as dasm | 13:39 | |
| dasm | o/ | 13:39 |
| frenzyfriday|rover | master compute has tempest tests failing - /me creating skiplist | 13:40 |
| frenzyfriday|rover | dasm, \o | 13:41 |
| rlandy | dviroel: hello! ... can you add your patch to test the larger node flavor to https://issues.redhat.com/secure/RapidBoard.jspa?rapidView=11751&projectKey=TRIPLEOCI&view=detail&selectedIssue=TRIPLEOCI-1234 | 13:45 |
| rlandy | thanks | 13:45 |
| frenzyfriday|rover | rlandy, https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/859993 - the bug is already filed. Added compute to the skiplist | 13:48 |
| frenzyfriday|rover | i have a tp for the rest of the components - taking a look soon | 13:48 |
| * frenzyfriday|rover food | 13:48 | |
| rlandy | frenzyfriday|rover: ah ok | 13:49 |
| rlandy | merging | 13:49 |
| dviroel | rlandy: ok, doing now | 13:51 |
| *** soniya29|ruck|dinner is now known as soniya29|ruck | 14:04 | |
| rlandy | dviroel: thanks - trying that approach on master - can you check my patch? https://review.rdoproject.org/r/c/testproject/+/43883/41/.zuul.yaml | 14:23 |
| dviroel | rlandy: yeah, looks correct | 14:27 |
| rlandy | k - let's see what it does | 14:29 |
| rlandy | ysandeep: 1-1? | 14:32 |
| ysandeep | rlandy, sry.. coming | 14:33 |
| jpodivin | frenzyfriday|rover: I'm seeing couple of failures on tripleo content provider https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider&skip=0 issues with package repo during dlrn build | 14:33 |
| jpodivin | could be a flake though | 14:33 |
| soniya29|ruck | ykarel, c8 fs35 train failing with compute and network tempest tests failing - https://bugs.launchpad.net/tripleo/+bug/1991093 | 14:34 |
| frenzyfriday|rover | jpodivin, ack, lemme check | 14:34 |
| jpodivin | frenzyfriday|rover: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_77a/858882/13/check/tripleo-ci-centos-9-content-provider/77a9ae3/logs/undercloud/home/zuul/DLRN/data/repos/component/tripleo/be/2f/be2f14890fc6a922e3650b7274299ebe55701a19_dev/rpmbuild.log | 14:35 |
| jpodivin | here you go | 14:35 |
| jpodivin | last line quickstart-centos-base repo | 14:35 |
| soniya29|ruck | ykarel, neutron openvswitch-agent not starting, not sure wether it is a correct data point | 14:35 |
| jpodivin | btw I'm rechecking the patch atm so we'll know how reproducible it is in a bit. | 14:36 |
| soniya29|ruck | ykarel, https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/e8133d7/logs/undercloud/var/log/extra/podman/containers/neutron_ovs_agent/stdout.log.txt.gz | 14:36 |
| frenzyfriday|rover | jpodivin, hm.. looks like a mirror issue maybe? Do you see this often? | 14:36 |
| jpodivin | frenzyfriday|rover: as I said. it could be a flake. I saw it just once so far, although last handful of jobs were aborted before completion. | 14:38 |
| frenzyfriday|rover | cool :) I'll keep an eye on the job | 14:38 |
| jpodivin | mostly I just wanted to know if it's not an already known issue, so I wouldn't tax CI with recheck | 14:38 |
| dviroel | rlandy: http://pastebin.test.redhat.com/1075843 | 14:41 |
| rlandy | dviroel: so it's operational? | 14:55 |
| rlandy | soniya29|ruck: hi | 14:57 |
| rlandy | soniya29|ruck: need help with anything? | 14:58 |
| dviroel | rlandy: yeah, we can also try to increase extra node if needed | 15:02 |
| dviroel | depending on the failure, extra node uses ci.m1.small | 15:03 |
| rlandy | dviroel: my best guess is that it's a networking thing | 15:03 |
| rlandy | but we;ll see if this does anything | 15:03 |
| rlandy | dviroel: also looking at the promoter code | 15:04 |
| rlandy | to accept an upstream OR internal job for promo | 15:04 |
| dviroel | ack | 15:07 |
| soniya29|ruck | rlandy, nope | 15:08 |
| soniya29|ruck | thanks | 15:08 |
| * ysandeep out, happy weekend everyone :) see you on Monday o/ fyi.. I will be out on Monday's first half for an appointment. | 15:11 | |
| dviroel | ysandeep: o/ | 15:11 |
| *** ysandeep is now known as ysandeep|out | 15:11 | |
| *** dviroel is now known as dviroel|lunch | 15:12 | |
| marios | o/ ysandeep|out have a good w/e | 15:14 |
| *** marios is now known as marios|out | 15:31 | |
| marios|out | dasm: o/ just saw your email lets talk next week | 15:34 |
| marios|out | dasm: (upgrades ) | 15:34 |
| dasm | marios|out: sure! | 15:44 |
| rlandy | fatal: [supplemental]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host 10.0.1.239 port 22: Connection timed out", | 15:57 |
| rlandy | interesting | 15:57 |
| rlandy | with one bigger node? | 15:57 |
| *** dviroel|lunch is now known as dviroel | 16:05 | |
| frenzyfriday|rover | hey, what is the way to find out which version of tempest is installed? Example, in here https://logserver.rdoproject.org/52/45352/1/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/b1f2fb9/ | 16:06 |
| frenzyfriday|rover | oh, got it | 16:07 |
| *** amoralej is now known as amoralej|off | 16:11 | |
| *** jpena is now known as jpena|off | 16:17 | |
| frenzyfriday|rover | rlandy, https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/860003 for fs20 wallaby - same error as before. | 16:19 |
| rlandy | k - wil look in a few | 16:20 |
| frenzyfriday|rover | soniya29|ruck, rlandy I am leaving for the day in ~30 min. tp which are running (master component, train etc) are on hackmd. fs20 skiplist is up (it passed once on the current hash though) | 16:33 |
| rlandy | frenzyfriday|rover: thaks | 16:33 |
| rlandy | thanks | 16:33 |
| soniya29|ruck | rlandy, i am also leaving for the day | 16:34 |
| rlandy | ok | 16:34 |
| frenzyfriday|rover | patches on hackmg are marked to watch and to merge :P | 16:36 |
| ykarel | soniya no idea can check on Tuesday if not resolved till then, Monday i will mostly be out | 16:49 |
| ykarel | but from what i see ovs agent is running | 16:49 |
| ykarel | anyway the logs u shared are from undercloud so not related | 16:57 |
| rlandy | frenzyfriday|rover: 17.1 promoted and 16.2 tripleo- kicked 16.2 integration line | 18:00 |
| rlandy | rerunning fs001 train to promo | 18:01 |
| frenzyfriday|rover | rlandy, thanks. I reran the still failing components in master | 18:23 |
| rlandy | still experimenting with OVB :( | 18:24 |
| dviroel | rlandy: fyi, prow jobs are properly claiming cluster from hive, which can be created in advance by the hive tool. We now claim the cluster, install an operator (mariadb) and check if it is working. | 19:05 |
| dviroel | https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/32731/rehearse-32731-pull-ci-openstack-k8s-operators-mariadb-operator-master-mariadb-operator-e2e/1575921717379862528 | 19:05 |
| dviroel | test log: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_release/32731/rehearse-32731-pull-ci-openstack-k8s-operators-mariadb-operator-master-mariadb-operator-e2e/1575921717379862528/artifacts/mariadb-operator-e2e/openstack-k8s-operators-mariadb-e2e/build-log.txt | 19:06 |
| dviroel | ^ operator being installed | 19:06 |
| rlandy | dviroel: wow - nice | 19:11 |
| rlandy | dviroel: hmm ... https://review.rdoproject.org/zuul/stream/c8baf1077d1045f5a8caeb830fce7fb5?logfile=console.log | 19:12 |
| rlandy | fails ipa connection | 19:12 |
| rlandy | is there another param to specify | 19:12 |
| dviroel | only fails in testproject? | 19:18 |
| dviroel | maybe something changed or missing another param, let me check | 19:19 |
| rlandy | yeah | 19:31 |
| rlandy | I think sp | 19:31 |
| rlandy | dviroel: maybe I am missing some network? | 19:32 |
| * jm1 out for today. have a nice weekend #oooq :D | 19:40 | |
| dasm | jm1[m]: o/ | 19:43 |
| dviroel | rlandy: the rest should take the default I think, maybe we are setting too much details, and we should try set only: baremetal_flavor: ci.m1.xxlarge | 19:55 |
| dviroel | rlandy: waiting the fs064 to start extra-node stack creation, to compare | 20:08 |
| dviroel | rlandy: the only difference that I see, is baremetal_image and undercloud_image, they are set to "CentOS-Stream-GenericCloud-9-20211216" on job running on integration line | 20:13 |
| dviroel | in the testproject, we are setting to "ipxe-uefi" | 20:14 |
| *** dviroel is now known as dviroel|afk | 20:28 | |
| frenzyfriday|rover | rlandy, train hash 9d5764744bef97b9ab0bc614da917f3d is out only on fs001 - I have 2 runs of fs001 with different tempest fails - we can skip and promote | 20:38 |
| frenzyfriday|rover | https://logserver.rdoproject.org/07/45407/2/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/887224b/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz and https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/9564a97/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz | 20:38 |
| *** frenzyfriday|rover is now known as frenzyfriday | 20:39 | |
| rlandy | dviroel|afk: thanks | 20:44 |
| rlandy | frenzyfriday: ack | 20:44 |
| rlandy | frenzyfriday: go4it | 20:44 |
| rlandy | frenzyfriday: you can put in the patch | 20:44 |
| frenzyfriday | rlandy, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45424 | 20:48 |
| * frenzyfriday leaving for the day | 20:48 | |
| dasm | frenzyfriday: o/ | 20:48 |
| frenzyfriday | happy weekend \o/ | 20:48 |
| rlandy | frenzyfriday: thanks | 20:49 |
| * dasm => offline | 21:17 | |
| dasm | o/ | 21:17 |
| *** dasm is now known as dasm|off | 21:17 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!