| *** ykarel|away is now known as ykarel | 05:23 | |
| *** marios is now known as marios|ruck | 06:02 | |
| anbanerj|rover | marios|ruck, O/ | 07:03 |
|---|---|---|
| marios|ruck | o/ anbanerj|rover | 07:03 |
| anbanerj|rover | https://bugs.launchpad.net/tripleo/+bug/1932344 - https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/796954 - TestMinimumBasicScenario.test_minimum_basic_scenario is in skiplist now | 07:03 |
| marios|ruck | thank you anbanerj|rover | 07:03 |
| marios|ruck | anbanerj|rover: oh really? | 07:03 |
| marios|ruck | no | 07:03 |
| marios|ruck | sec | 07:03 |
| anbanerj|rover | I see a patch from Wes | 07:03 |
| marios|ruck | 3123 19:23 < marios|ruck> anbanerj|rover: please bear in mind/note on the bug that 18:31 < marios|ruck> sshnaidm : anbanerj|rover: we can't add to skips or it makes some periodics fail like fs2 see https://review.opendev .org/c/openstack/openstack-tempest-skiplist/+/795742 $ | 07:05 |
| marios|ruck | anbanerj|rover: see https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/795742 | 07:05 |
| marios|ruck | anbanerj|rover: ah it is a different test? | 07:06 |
| marios|ruck | minimum_basic_scenario vs test_network_basic_ops | 07:06 |
| marios|ruck | anbanerj|rover: then it is ok sorry | 07:06 |
| anbanerj|rover | yes. There was already a bug https://bugs.launchpad.net/tripleo/+bug/1923194 for fs020 failures. TestMinimumBasicScenario.test_minimum_basic_scenario was skipped for a select few jobs then | 07:07 |
| *** jpena|off is now known as jpena | 07:18 | |
| anbanerj|rover | marios|ruck, we have 2 patches that failed gate with job status LOST. Can I recheck those or do those have some other meaning? | 07:37 |
| marios|ruck | anbanerj|rover: don't know what that is might ask in opendev but i suspect infra folks aren't around yet | 07:41 |
| marios|ruck | anbanerj|rover: just recheck i think though if there are only 2 | 07:42 |
| marios|ruck | we can investigate if we see more | 07:42 |
| anbanerj|rover | ok | 07:42 |
| chandankumar | Tengu, marios|ruck, Good morning, Please have a look at this: https://review.opendev.org/c/openstack/tripleo-operator-ansible/+/791645 | 07:43 |
| chandankumar | thanks! | 07:43 |
| Tengu | lemme have a look :) | 07:43 |
| Tengu | chandankumar: hmmmmmmm tripleo-operator-ansible is branchless....? | 07:44 |
| Tengu | chandankumar: not sure to understand why other tarballs are needed. | 07:45 |
| chandankumar | Tengu, yes | 07:45 |
| chandankumar | Tengu, we need tagged tarballs on tarballs.opendev.org so that we can use it in rdo side for building packages for different branches | 07:45 |
| Tengu | chandankumar: so it will copy the -master.tar to -train.tar, -ussuri.tar and so on? (+ signature apparently) | 07:46 |
| Tengu | or will it re-build the tarball from scratch for each release, while using the same content (since branchless)? | 07:47 |
| Tengu | just wanting to understand the whole thing, sorry if those are stupid questions ^^' | 07:47 |
| chandankumar | Tengu, https://tarballs.opendev.org/openstack/tripleo-operator-ansible/ | 07:47 |
| Tengu | yeah, I've seen it already, it's linked in the commit message | 07:48 |
| chandankumar | Tengu, sorry | 07:48 |
| Tengu | still... since the project is branchless, there's only master | 07:48 |
| chandankumar | Tengu, https://review.opendev.org/c/openstack/tripleo-operator-ansible/+/791645 once it gets merged | 07:48 |
| marios|ruck | chandankumar: will do | 07:48 |
| chandankumar | we will cut a new tag from master | 07:48 |
| chandankumar | then then job will generate a tarballs with the new tag | 07:49 |
| Tengu | chandankumar: and we'll get -stein.tar.gz, -train.tar.gz, ... up to master.tar.gz? | 07:49 |
| chandankumar | then we will update the tag on rdo side in tripleo-operator-ansible spec file for each release | 07:49 |
| chandankumar | then it will get built | 07:49 |
| chandankumar | the packages for each release | 07:49 |
| Tengu | ok. so it's just a "hack" because RDO doesn't understand branchless project with only a "master" tarball | 07:49 |
| Tengu | I think I get it | 07:50 |
| chandankumar | Yes | 07:50 |
| chandankumar | Tengu, https://review.opendev.org/q/topic:%22remove_uneeded_tq_requirements%22+(status:open%20OR%20status:merged) | 07:50 |
| Tengu | 'k, so some cleaning and consolidations. Lovely | 07:50 |
| chandankumar | Tengu, and https://review.rdoproject.org/r/q/topic:%22remove_uneeded_tq_requirements%22+(status:open%20OR%20status:merged) to clear confusion | 07:51 |
| Tengu | sounds good! thanks for the explanations :) | 07:52 |
| chandankumar | marios|ruck, this job is failing https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7eb/782362/15/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/7ebe540/logs/undercloud/home/zuul/overcloud_update_converge.log with "msg": "Failed containers: keystone_bootstrap" | 08:20 |
| chandankumar | is it a known issue? | 08:20 |
| marios|ruck | chandankumar: yes | 08:21 |
| marios|ruck | chandankumar: https://bugs.launchpad.net/tripleo/+bug/1932261 | 08:22 |
| marios|ruck | chandankumar: * https://review.opendev.org/c/openstack/tripleo-ansible/+/796857 | 08:22 |
| marios|ruck | chandankumar: the patch merged this morning so recheck should clear for you | 08:23 |
| chandankumar | marios|ruck, thanks :-) | 08:25 |
| *** ykarel is now known as ykarel|lunch | 09:00 | |
| *** raukadah is now known as chandankumar | 09:26 | |
| marios|ruck | anbanerj|rover: o/ hey this one https://bugs.launchpad.net/tripleo/+bug/1881087 is the one we were discussing with ronelle yesterday afternoon? | 09:41 |
| marios|ruck | anbanerj|rover: meant to be fixed by that patch but isn't? | 09:41 |
| marios|ruck | anbanerj|rover: 'that patch' the octavia one /me tries to find | 09:42 |
| marios|ruck | anbanerj|rover: yeah that one i think https://review.opendev.org/c/openstack/networking-ovn/+/796517 | 09:43 |
| anbanerj|rover | marios|ruck, yep https://bugs.launchpad.net/tripleo/+bug/1881087 this one. | 09:46 |
| marios|ruck | anbanerj|rover: ack and looks like we didn't promote train yet though since it merged | 09:47 |
| marios|ruck | anbanerj|rover: i think ronelle was chasing that promotion | 09:47 |
| anbanerj|rover | marios|ruck, It is failing for some other branches too. lemme check | 09:47 |
| marios|ruck | anbanerj|rover: k, i'll try to update the bug in a bit | 09:48 |
| marios|ruck | anbanerj|rover: but really my question was 'how do we know it didn't work' are we still seeing that error in the promotion jobs? | 09:49 |
| anbanerj|rover | marios|ruck, https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-train&job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train&job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-master&job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-ussuri&job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-victoria | 09:49 |
| anbanerj|rover | periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train and ussuri is failing | 09:49 |
| anbanerj|rover | victoria and master are passing | 09:49 |
| marios|ruck | anbanerj|rover: thanks | 09:50 |
| anbanerj|rover | marios|ruck, and upstream https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario010-standalone&job_name=tripleo-ci-centos-7-scenario010-multinode-oooq-container&job_name=tripleo-ci-centos-7-scenario010-ovn-provider-standalone&job_name=tripleo-ci-centos-8-scenario010-ovn-provider-standalone&job_name=tripleo-ci-centos-7-scenario010-standalone | 09:52 |
| marios|ruck | anbanerj|rover: upstream is to be expected if we didn't promote yet | 09:52 |
| marios|ruck | anbanerj|rover: that why i was asking about the promotion lines | 09:53 |
| anbanerj|rover | marios|ruck, ah, ok | 09:53 |
| marios|ruck | anbanerj|rover: this change? https://review.opendev.org/c/openstack/networking-ovn/+/796517 or was there another one that you know about? | 09:57 |
| anbanerj|rover | only this one ^\ | 09:58 |
| marios|ruck | anbanerj|rover: k i am confused then that change points to a different (train only) bug there https://bugs.launchpad.net/neutron/+bug/1932093 :/ | 09:58 |
| anbanerj|rover | marios|ruck, true.. There are no fix proposed on https://bugs.launchpad.net/tripleo/+bug/1881087 The only octavia patch that was mentioned was ^ | 09:59 |
| marios|ruck | anbanerj|rover: but who/where told us that patch is related i don't see it on the bug or the cix (only from you on the cix) | 10:00 |
| marios|ruck | anbanerj|rover: maybe we just heard 'octavia' and assumed it was related | 10:00 |
| marios|ruck | anbanerj|rover: it matters because i was going to write something like 'the patch merged but the problem is ongoing' and it is a d/stream osp blocker | 10:01 |
| anbanerj|rover | I heard in the program call that there was an octavia patch and kind of hoped it would fix all the issues at the same time | 10:01 |
| marios|ruck | anbanerj|rover: but it seems that patch might have been about another fix not this one | 10:01 |
| marios|ruck | anbanerj|rover: yeah OK thats fine just trying to understand the status am putting note on the bug and cix now | 10:01 |
| anbanerj|rover | no, we might need a different patch | 10:02 |
| marios|ruck | anbanerj|rover: so in fact this one is ongoing and i don't think we have any fix for it yet | 10:02 |
| anbanerj|rover | yes | 10:02 |
| marios|ruck | anbanerj|rover: this one == timeout | 10:02 |
| marios|ruck | anbanerj|rover: ack | 10:02 |
| *** ykarel|lunch is now known as ykarel | 10:13 | |
| zbr | marios|ruck: chandankumar: is anything stopping https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/796278 ? | 11:16 |
| marios|ruck | zbr: will check in bit but going afk few mins now | 11:23 |
| marios|ruck | anbanerj|rover: going afk for a bit | 11:30 |
| anbanerj|rover | sure np | 11:31 |
| marios|ruck | anbanerj|rover: thank you | 11:31 |
| *** jpena is now known as jpena|lunch | 11:41 | |
| arxcruz | it's too hot | 11:46 |
| arxcruz | i'm melting | 11:46 |
| *** bhagyashris_ is now known as bhagyashris | 11:50 | |
| *** pojadhav- is now known as pojadhav | 11:51 | |
| *** ysandeep is now known as ysandeep|brb | 11:58 | |
| *** amoralej is now known as amoralej|lunch | 12:04 | |
| *** whayutin is now known as weshay | 12:07 | |
| anbanerj|rover | rlandy, weshay I have rights to change bugs toi triaged now, thanks! | 12:13 |
| rlandy | great | 12:14 |
| weshay | anbanerj|rover, added a section https://docs.google.com/document/d/1bnhfx7F8Ue4-sLdIjF6fju3tWBg7hFH3r2Kad-IaXD0/edit#heading=h.iqzqr0bo38be | 12:16 |
| weshay | dviroel, ^ make sure you hit that bit | 12:16 |
| anbanerj|rover | thanks | 12:17 |
| weshay | anbanerj|rover, marios|ruck need anything or to hand off anything? | 12:17 |
| anbanerj|rover | I had 3 open bugs with new status, checking if they are still relevant or can be closed | 12:20 |
| marios|ruck | weshay: not in particular | 12:20 |
| marios|ruck | weshay: will do before i leave later | 12:21 |
| marios|ruck | weshay: main thing is we should get some promotions, master for https://trello.com/c/sJDrMKrT/1989-cixlp1930575tripleociproa-periodic-tripleo-ci-centos-8-standalone-upgrade-master-failing-tripleo-upgrade-not-identifed-as-an-ope#comment-60cc73ef39b89023e120c533 | 12:21 |
| anbanerj|rover | https://bugs.launchpad.net/tripleo/+bug/1931842 marios|ruck shall I close this one - we are not hitting this any more | 12:22 |
| marios|ruck | anbanerj|rover: yes | 12:22 |
| dviroel | weshay: ack, just requested to join the team in launchpad | 12:24 |
| anbanerj|rover | marios|ruck, not sure how it ^ was fixed, but we hit it for a while for sure. Should I change it to invalid or fix released? | 12:26 |
| marios|ruck | anbanerj|rover: doesn't matter too much but with fix released people expect a pointer to the fix | 12:27 |
| marios|ruck | anbanerj|rover: so maybe invalid or incomplete | 12:28 |
| anbanerj|rover | marios|ruck, ok thanks | 12:28 |
| *** amoralej|lunch is now known as amoralej | 12:30 | |
| dviroel | weshay: tks! | 12:31 |
| weshay | marios|ruck, qq.. re: emphemeral heat.. https://review.opendev.org/c/openstack/tripleo-quickstart/+/792248/2/config/general_config/featureset037.yml, https://review.opendev.org/c/openstack/tripleo-quickstart/+/777108/31/config/general_config/featureset010.yml https://review.opendev.org/q/topic:%22ephemeral-heat%22+(status:open%20OR%20status:merged) | 12:36 |
| weshay | marios|ruck, is that coming to wallaby soon or should it be exlcuded? | 12:36 |
| weshay | until the patches are closer | 12:36 |
| *** jpena|lunch is now known as jpena | 12:38 | |
| weshay | arxcruz, thanks for pushing on https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/796818/ | 12:44 |
| weshay | merged 61 patches in the last 24 hours :) | 12:45 |
| marios|ruck | weshay: slagle plans to bring it to wallaby but i don't think he has done it yet (or at least not all) | 12:46 |
| marios|ruck | weshay: if that is the question | 12:46 |
| weshay | marios|ruck, it's the tq featureset config.. | 12:47 |
| weshay | wallaby makes the call unless I'm reading it incorrectly | 12:47 |
| marios|ruck | weshay: yes that is the intention | 12:47 |
| weshay | but the timing? | 12:47 |
| marios|ruck | weshay: is/was the intention of the review | 12:47 |
| * weshay checks jobs | 12:47 | |
| marios|ruck | weshay: just asked slagle in tripleo | 12:48 |
| marios|ruck | weshay: gotcha i get what you mean now yeah we should exclude | 12:48 |
| marios|ruck | weshay: i am having this conversation with you in two channels | 12:48 |
| marios|ruck | :) | 12:48 |
| weshay | marios|ruck, this is working fine.. https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-containers-multinode-wallaby and includes wallaby.. https://review.opendev.org/c/openstack/tripleo-quickstart/+/777108 | 12:49 |
| anbanerj|rover | marios|ruck, https://bugs.launchpad.net/tripleo/+bug/1931694 - we are still hitting this , shall I move to triaged? If I find more details I'll put in comments | 12:50 |
| weshay | anbanerj|rover, always move to triaged. | 12:50 |
| anbanerj|rover | ok | 12:50 |
| weshay | anbanerj|rover, there is a little pop up when you open a bug w/ instructions | 12:50 |
| marios|ruck | weshay: so not sure i still feel like we should exclude wallaby there until they merge | 12:51 |
| marios|ruck | but until it breaks i guess we just ignore it ;) | 12:51 |
| marios|ruck | weshay: ? | 12:51 |
| weshay | anbanerj|rover, dviroel http://pastebin.test.redhat.com/973067 read through that | 12:52 |
| weshay | marios|ruck, ya.. don't know :) | 12:53 |
| weshay | if it's not failing.. I guess it's fine | 12:53 |
| weshay | perhaps heat is running in two places in the interim | 12:53 |
| weshay | would have to look | 12:53 |
| *** ysandeep|brb is now known as ysandeep | 12:54 | |
| marios|ruck | anbanerj|rover: sure but pretty low priority to be honest stein/queens EOL proposal openstack-discuss @ http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023115.html | 12:56 |
| marios|ruck | weshay: did you see that ^^^ i added it in the google doc | 12:56 |
| anbanerj|rover | marios|ruck, ok, marking as low | 12:57 |
| weshay | marios|ruck, ya.. I think I responded to the email on opendev | 12:57 |
| weshay | yup.. I did | 12:58 |
| marios|ruck | weshay: oh todya? i missed it checking | 12:58 |
| weshay | last night | 13:01 |
| anbanerj|rover | marios|ruck, should I tag https://bugs.launchpad.net/tripleo/+bug/1931601 promotion blocker? (periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train times out frequently) | 13:01 |
| marios|ruck | anbanerj|rover: anything more recent? | 13:02 |
| marios|ruck | anbanerj|rover: those logs are from 9/10 june from https://bugs.launchpad.net/tripleo/+bug/1931601 | 13:03 |
| marios|ruck | anbanerj|rover: if it is still happening then yes since periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train is in the criteria | 13:03 |
| anbanerj|rover | marios|ruck, yes found some from yesterday. Adding tag. Updating logs | 13:03 |
| weshay | ysandeep, you want to poke at grafana or you good? | 13:05 |
| ysandeep | weshay: lets chat for few mins | 13:06 |
| weshay | k | 13:06 |
| weshay | ysandeep, meet.google.com/gqt-rokq-qxb | 13:07 |
| weshay | anbanerj|rover, can you look at https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-undercloud-containers-master/e7e2e12/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 13:07 |
| weshay | Ansible execution failed. playbook: deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2 | 13:08 |
| weshay | Could not clean up: 'ClientManager' object has no attribute 'sdk_connection' | 13:08 |
| anbanerj|rover | checking | 13:09 |
| raukadah | weshay, please remove -1 from this change https://review.opendev.org/c/openstack/tripleo-quickstart/+/790701 | 13:11 |
| raukadah | weshay, tripleo-operator-ansible is available from stein onwards | 13:12 |
| raukadah | it is built by dlrn now | 13:12 |
| *** raukadah is now known as chandankumar | 13:13 | |
| weshay | k.. sec | 13:13 |
| anbanerj|rover | weshay, there is a similar bug https://bugs.launchpad.net/tripleo/+bug/1930811 but for different jobs | 13:16 |
| weshay | zbr, 1min | 13:31 |
| * ysandeep leaving for the day, Have a great weekend, See you on Monday! | 13:37 | |
| *** ysandeep is now known as ysandeep|away | 13:37 | |
| weshay | zbr, anbanerj|rover got this working :) http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario | 13:41 |
| anbanerj|rover | weshay, thanks. Should we revert this then https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/796954 ? | 13:43 |
| weshay | anbanerj|rover, will have to look | 13:43 |
| weshay | anbanerj|rover, what's the port for the tripleo-ci-health hitting rdo? | 13:43 |
| anbanerj|rover | our health dashboard? It is not set up with rdo yet | 13:44 |
| weshay | zbr, https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/796007 | 13:47 |
| * anbanerj|rover going to get food | 13:49 | |
| weshay | marios|ruck, dviroel atm... zbr has not yet started looking at replacing python setup.py for tqe_requirments.. does that block you folks? | 13:54 |
| weshay | or is python setup.py ok enough for now? | 13:54 |
| marios|ruck | weshay: i am not looking there this sprint really so no | 13:54 |
| weshay | ++ | 13:55 |
| marios|ruck | weshay: my patch is still using python setup.py | 13:55 |
| dviroel | weshay: same for my patch | 13:55 |
| weshay | reviews on https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/796278 | 13:57 |
| arxcruz | No happy friday today ? | 14:00 |
| raukadah | zbr, dviroel left comments on your patches, feel free to ping back if not clear, thanks :-) | 14:07 |
| arxcruz | ysandeep|away: where are you? | 14:08 |
| arxcruz | raukadah: hey man happy friday, where are you ? | 14:08 |
| weshay | raukadah, HEY | 14:08 |
| weshay | social hour | 14:09 |
| raukadah | arxcruz, hiding in the cave, | 14:09 |
| raukadah | enjoying the rain | 14:09 |
| weshay | raukadah, you already have a MAN CAVE? | 14:09 |
| weshay | that was quick | 14:09 |
| raukadah | mike issue | 14:13 |
| pojadhav | folks. due to heavy rain, having n/w issues | 14:18 |
| anbanerj|rover | anbanerj|rover, is back | 14:23 |
| weshay | anbanerj|rover, how's lunch? | 14:23 |
| marios|ruck | brb | 14:30 |
| *** marios is now known as marios|ruck | 14:34 | |
| *** jpena is now known as jpena|out | 14:48 | |
| *** pojadhav- is now known as pojadhav | 14:48 | |
| weshay | pojadhav, qq if ur still around | 15:02 |
| *** amoralej is now known as amoralej|off | 15:02 | |
| weshay | marios|ruck, pojadhav periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset037-updates-wallaby is in criteria for wallaby but not running atm | 15:05 |
| weshay | looking at jira I don't see work in progress atm.. | 15:05 |
| weshay | shall I rename master and wallaby to the proper upstream name.. and get it running and add to criteria when ready? or hold off | 15:05 |
| weshay | ah.. it's not upstream yet | 15:08 |
| marios|ruck | weshay: looks like it is commented out | 15:09 |
| marios|ruck | weshay: giving link in pvt | 15:09 |
| weshay | ah.. I wonder if our new ruck/rover tool is reading commented out job | 15:09 |
| marios|ruck | weshay: but agree that it should be in the perioridc it isnt currnetly? https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset037-updates-wallaby | 15:10 |
| marios|ruck | i see only check/testproject there | 15:10 |
| weshay | OH.. I bet the ruck / rover tool is pulling criteria from the old promotion server | 15:10 |
| weshay | things always in flux :) | 15:10 |
| marios|ruck | somuch fun | 15:11 |
| weshay | ysandeep|away, fyi https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34060 comment re: urls | 15:14 |
| weshay | k.. wallaby should promote today | 15:17 |
| marios|ruck | weshay: master no good candidate :/ (need for https://trello.com/c/sJDrMKrT/1989-cixlp1930575tripleociproa-periodic-tripleo-ci-centos-8-standalone-upgrade-master-failing-tripleo-upgrade-not-identifed-as-an-ope#comment-60cc73ef39b89023e120c533 ) | 15:20 |
| marios|ruck | weshay: and train :/ nope https://review.rdoproject.org/zuul/buildset/3d050dd844cf4e5986bd5e5dbad38925 even image build fails there but only once so far not sure if real or not * https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-train/343db01/build-err.log | 15:21 |
| weshay | ++ I'll kick master now | 15:21 |
| marios|ruck | weshay: but the candidate is bad | 15:22 |
| marios|ruck | weshay: missing jobs | 15:22 |
| weshay | train or master? | 15:22 |
| marios|ruck | weshay: both | 15:22 |
| marios|ruck | all of the things | 15:22 |
| marios|ruck | weshay: master there https://review.rdoproject.org/zuul/buildset/00a1bcfc9d2c4306bcda8910f16ddbf5 | 15:23 |
| marios|ruck | weshay: train there https://review.rdoproject.org/zuul/buildset/3d050dd844cf4e5986bd5e5dbad38925 | 15:23 |
| weshay | master is has more failing than train | 15:23 |
| weshay | marios|ruck, more jobs will pass over the weekend when rdo isn't busy | 15:27 |
| marios|ruck | weshay: ack hope so | 15:27 |
| marios|ruck | weshay: apart from that bug bandini was asking me about some other tripleo-common patch and 'why isn't it in the current-tripleo yet we already promoted' | 15:28 |
| marios|ruck | weshay: its because tripleo component *hadn't* promoted for a few days. chased and got that so we need the integration now | 15:28 |
| weshay | master? | 15:29 |
| marios|ruck | weshay: yea | 15:29 |
| marios|ruck | weshay: that https://trello.com/c/sJDrMKrT/1989-cixlp1930575tripleociproa-periodic-tripleo-ci-centos-8-standalone-upgrade-master-failing-tripleo-upgrade-not-identifed-as-an-ope#comment-60cc73ef39b89023e120c533 | 15:29 |
| weshay | tripleo promoted yesterday | 15:29 |
| zbr | weshay: very good idea to test pip-compile on tripleo-repos! -- it opens few questions and we can decide how we want to use it. | 15:29 |
| weshay | zbr, cool | 15:29 |
| weshay | illustrative purposes.. lolz | 15:31 |
| marios|ruck | thanks weshay | 15:31 |
| zbr | see https://review.opendev.org/c/openstack/tripleo-repos/+/797125 -- i am starting to believe that maybe it would be better to name the output file constraints.txt instead of requirements.txt | 15:32 |
| zbr | and apparently using upper constraints does also work (need to double check but the --pipargs should do the trick) | 15:33 |
| zbr | my initial impression is that we should build only one consolidated constraints.txt per repo which is used as -c... by various tox jobs, keeping the input files like they were already. | 15:34 |
| zbr | is like having an in-repo super constrained file, one based on upper one. | 15:35 |
| zbr | now the tricky bit is that updating upper constraint file no longer have any effect on the code from the repo, at least until we run "tox -e deps" manually. | 15:36 |
| zbr | is that a problem or not? | 15:36 |
| weshay | zbr, it's def NOT a problem for the repos we've lined up | 15:36 |
| weshay | it may impact packaging decisions on tripleo proper repos | 15:36 |
| weshay | zbr, make sense? | 15:37 |
| zbr | yept. | 15:38 |
| zbr | do you also think is better to name the output constraints.txt ? | 15:38 |
| weshay | marios|ruck, the 000 wallaby updates job is running in the wallaby tripleo component and green :) so that's good | 15:38 |
| weshay | zbr, magic 8-ball says.. I'm fresh out of decisions for the morning.. and still needs to read your doc in depth | 15:39 |
| marios|ruck | weshay: thanks | 15:40 |
| rlandy | weshay: marios|ruck: anbanerj|rover: ok - here's another 17 -> wallaby discovery | 15:41 |
| rlandy | compare these two: | 15:41 |
| rlandy | https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/596773b/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz | 15:41 |
| rlandy | https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-wallaby/a0bdb52/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz | 15:41 |
| rlandy | note that the OVB ceph parameters are missing some of what the standalone jobs have | 15:42 |
| rlandy | that's a bug, right? | 15:42 |
| marios|ruck | rlandy: not sure my brain can handle that right now | 15:43 |
| rlandy | marios|ruck: don't worry about it - you're almost at weekend | 15:44 |
| * rlandy chasing it down | 15:44 | |
| marios|ruck | rlandy: will try having a look in a bit when doing some exercise (after i'm afk here i mean) | 15:45 |
| rlandy | marios|ruck: no worries | 15:45 |
| rlandy | just an FYI | 15:45 |
| rlandy | I'm debugging | 15:45 |
| weshay | rlandy, https://github.com/openstack/tripleo-heat-templates/blob/master/README.rst#service-testing-matrix | 15:45 |
| *** dviroel is now known as dviroel|busy | 15:46 | |
| rlandy | so we don't bring down those containers for OVB | 15:46 |
| rlandy | still seems wrong | 15:47 |
| weshay | rlandy, ng command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=192.168.24.1:8787/ceph-ci/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64 -e NODE_NAME=standalone.localdomain -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4b5c8c0a-ff60-454b-a1b4-9747aa737d19:/var/log/ceph:z -v /var/lib/ceph/4b5c8c0a-ff60-454b-a1b4-9747aa737d19/mon.standalone.localdomain:/var/lib/ceph/mon/ | 15:47 |
| weshay | ceph-standalone.localdomain:z -v /tmp/ceph-tmpoc004qkt:/tmp/keyring:z -v /tmp/ceph-tmpmwll2wp7:/tmp/monmap:z 192.168.24.1:8787/ceph-ci/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64 --mkfs -i standalone.localdomain --fsid 4b5c8c0a-ff60-454b-a1b4-9747aa737d19 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debu | 15:47 |
| weshay | g --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true | 15:47 |
| weshay | https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-wallaby/a0bdb52/logs/undercloud/var/log/ceph/cephadm.log.txt.gz | 15:47 |
| rlandy | scenario001 is right | 15:47 |
| rlandy | OVB is missing | 15:47 |
| weshay | rlandy, https://logserver.rdoproject.org/89/795889/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/ec5190b/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz | 15:49 |
| weshay | ussuri ^ | 15:49 |
| weshay | rlandy, what's wrong? | 15:50 |
| weshay | that ceph is not running? | 15:50 |
| rlandy | I am not sure - pulling wrong container | 15:52 |
| rlandy | 2021-06-18 09:53:41 | 2021-06-18 09:53:41.546985 | 0058bcc2-63a5-28d7-8ae0-000000003429 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Unable to authenticate. This may indicate missing registry credentials or the provided container or namespace does not exist. 401 Client Error: Unauthorized for url: https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-r | 15:52 |
| rlandy | egistry&scope=repository%3Aopenshift4%2Fose-prometheus-alertmanager%3Apull", "msg": "Error running container image prepare: Unable to authenticate. This may indicate missing registry credentials or the provided container or namespace does not exist. 401 Client Error: Unauthorized for url: https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&scope=repository%3Aopenshift4%2Fose-prometheu | 15:52 |
| rlandy | s-alertmanager%3Apull", "params": {}, "success": false} | 15:52 |
| rlandy | pulling the right ceph one | 15:52 |
| rlandy | so we are missing those settings | 15:52 |
| weshay | rlandy, we are missing them where? | 15:53 |
| weshay | 17? or fs001 wallaby | 15:53 |
| rlandy | well both I think | 15:53 |
| rlandy | hold on | 15:53 |
| rlandy | hold on checking | 15:53 |
| weshay | ya.. ceph is not in play in fs001 | 15:53 |
| rlandy | 'imagename': 'quay.ceph.io/prometheus/prometheus:v2.7.2' | 15:54 |
| rlandy | 2021-06-18 12:39:02,859 348604 DEBUG urllib3.connectionpool [ ] https://quay.ceph.io:443 "GET /v2/prometheus/node-exporter/manifests/v0.17.0 HTTP/1.1" 200 6725 | 15:54 |
| rlandy | still gets pulled | 15:55 |
| rlandy | but upstream default settings work | 15:55 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/standalone/tasks/containers.yml#L159 | 15:56 |
| rlandy | weshay: ^^ we get all the settings in standalone | 15:56 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/standalone/tasks/containers.yml#L150 | 15:56 |
| rlandy | alertmanager | 15:56 |
| rlandy | but not in overcloud | 15:56 |
| rlandy | docker_openshift_prometheus_alertmanager_tag | 15:57 |
| weshay | k.. so ur looking at https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-17/207f39b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 15:57 |
| weshay | promethius is not ceph | 15:57 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2 is short of options | 15:58 |
| rlandy | ceph_alertmanager | 15:58 |
| rlandy | not tere | 15:58 |
| rlandy | there | 15:58 |
| * rlandy fixes after quick lunch | 15:59 | |
| *** marios|ruck is now known as marios|out | 16:02 | |
| weshay | rlandy, it shouldn't be trying to deploy it | 16:05 |
| rlandy | it's pulling it | 16:07 |
| weshay | ya I see that | 16:07 |
| weshay | rlandy, compare to https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/596773b/logs/undercloud/var/log/extra/podman/podman_allinfo.log.txt.gz | 16:08 |
| weshay | rlandy, https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-manila/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario004-standalone-manila-rhos-17/53201bc/logs/undercloud/var/log/extra/podman/podman_allinfo.log | 16:16 |
| weshay | ceph_alertmanager_image: openshift-ose-prometheus-alertmanager | 16:18 |
| weshay | ceph_alertmanager_namespace: registry-proxy.engineering.redhat.com/rh-osbs | 16:18 |
| weshay | ceph_alertmanager_tag: v4.1 | 16:18 |
| weshay | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-rhos-17/8678e6e/logs/undercloud/home/zuul/containers-prepare-parameters.yaml | 16:18 |
| *** jpena|out is now known as jpena | 16:22 | |
| anbanerj|rover | Where can I find deploy_steps_playbook.yaml ? Which repo is that? | 16:23 |
| rlandy | back | 16:25 |
| weshay | anbanerj|rover, look at the config-download dir in the /home/zull logs | 16:25 |
| rlandy | standalone is right | 16:25 |
| weshay | rlandy, so I see config for ceph in wallaby.. but ceph is not on the controllers.. https://logserver.rdoproject.org/07/797007/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/cebf110/logs/overcloud-controller-0/var/log/extra/podman/podman_allinfo.log.txt.gz | 16:25 |
| weshay | rlandy, ya.. standalone is right | 16:26 |
| rlandy | those options just need to be added to overcloud | 16:26 |
| weshay | anbanerj|rover, you'll find the step playbooks and config | 16:26 |
| weshay | rlandy, call me | 16:26 |
| rlandy | maybe | 16:26 |
| weshay | rlandy, https://meet.google.com/ram-vxpt-wvt?authuser=1 | 16:27 |
| anbanerj|rover | weshay, after this which dir https://832c4623d9dde3d2f5ac-0f289cd40f565f3b373c9082881efda3.ssl.cf1.rackcdn.com/796403/1/gate/tripleo-ci-centos-8-scenario001-standalone/9c3c784/logs/undercloud/home/zuul/index.html ? | 16:30 |
| anbanerj|rover | i'll check on monday | 16:41 |
| *** jpena is now known as jpena|off | 16:48 | |
| weshay | rlandy, this is wallaby https://logserver.rdoproject.org/07/797007/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/cebf110/logs/undercloud/home/zuul/config-download/overcloud/external_deploy_steps_tasks_step1.yaml.txt.gz | 16:52 |
| weshay | rlandy, this is train https://logserver.rdoproject.org/52/795252/6/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/36004cf/logs/undercloud/var/lib/mistral/overcloud/external_deploy_steps_tasks_step1.yaml.txt.gz | 16:52 |
| rlandy | ugh 17 just kicked | 18:01 |
| rlandy | will wait until containers are built | 18:01 |
| weshay | rlandy, to see if the issue recreates? | 19:46 |
| rlandy | weshay: yep | 19:47 |
| rlandy | containers finished | 19:47 |
| rlandy | rekicking with param changes | 19:47 |
| rlandy | can't run BM together | 19:49 |
| rlandy | but kicked another OVB | 19:50 |
| weshay | what's weird about this one.. https://review.opendev.org/c/openstack/tripleo-quickstart/+/796888/1/config/general_config/featureset010.yml w/ or w/o ephemeral_heat set it works upstream.. | 19:51 |
| weshay | rlandy, nothing cherry-picked to wallaby yet https://review.opendev.org/q/topic:%22ephemeral-heat%22+(status:open%20OR%20status:merged) | 19:52 |
| weshay | or maybe it got in before we branched wallaby | 19:52 |
| weshay | that's probably it | 19:52 |
| weshay | tripleo-ci-centos-8-containers-multinode-wallabySUCCESS | 19:53 |
| weshay | oh.. k.. ya.. w/ or w/o | 19:53 |
| weshay | but blows up 17 | 19:53 |
| weshay | fun | 19:53 |
| rlandy | so upstream will merrily carry on | 19:56 |
| rlandy | downstream gets all uppity with auth | 19:57 |
| rlandy | doing a rerun with multinode and OVB | 19:57 |
| weshay | rlandy, oy.. https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/796047 | 19:58 |
| rlandy | weshay: I think that's supposed to be just a test patch | 20:01 |
| rlandy | let's w-1 | 20:01 |
| rlandy | stopped that | 20:01 |
| weshay | thanks.. added a comment | 20:04 |
| weshay | fixing irc bouncer | 20:13 |
| weshay|ruck | ah.. bouncer :) | 20:14 |
| rlandy | you back with us? | 20:22 |
| rlandy | can't tell with those multiple personalities | 20:22 |
| weshay|ruck | rlandy, I"M BACK!!! | 20:24 |
| rlandy | woohoo - welcome back weshay|ruck! | 20:25 |
| rlandy | need to drop something off - brb | 20:41 |
| *** rlandy is now known as rlandy|brb | 20:42 | |
| *** rlandy|brb is now known as rlandy | 20:57 | |
| rlandy | OVB is such a pain downstream | 21:44 |
| rlandy | never kick until the 3rd try | 21:44 |
| weshay|ruck | rlandy, seeems like they take 3 attemps | 21:51 |
| weshay|ruck | to get the stack right always | 21:51 |
| weshay|ruck | rlandy, go have a weekend please.. 010 passed :) | 21:51 |
| rlandy | weshay|ruck: yeah - checking out - have a good shabbos | 21:53 |
| rlandy | thanks for the help today | 21:53 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!