| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Revert "Revert "Move CA cert injection to host_prep_tasks"" https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846126 | 00:24 |
|---|---|---|
| opendevreview | Rafael Castillo proposed openstack/tripleo-quickstart master: DNM: Test building latest sdk in dependency https://review.opendev.org/c/openstack/tripleo-quickstart/+/845375 | 00:25 |
| opendevreview | Takashi Kajinami proposed openstack/python-tripleoclient master: overcloud deploy: Remove validation based on existing stack https://review.opendev.org/c/openstack/python-tripleoclient/+/846239 | 01:38 |
| opendevreview | Brendan Shephard proposed openstack/python-tripleoclient master: Fix config-download playbook name https://review.opendev.org/c/openstack/python-tripleoclient/+/846241 | 01:56 |
| opendevreview | Brendan Shephard proposed openstack/python-tripleoclient master: Remove overcloud config download command https://review.opendev.org/c/openstack/python-tripleoclient/+/846243 | 02:58 |
| opendevreview | Takashi Kajinami proposed openstack/python-tripleoclient master: overcloud deploy: Remove validation based on existing stack https://review.opendev.org/c/openstack/python-tripleoclient/+/846239 | 03:55 |
| opendevreview | Rafael Castillo proposed openstack/tripleo-quickstart master: DNM: Test building latest sdk in dependency https://review.opendev.org/c/openstack/tripleo-quickstart/+/845375 | 04:12 |
| opendevreview | yatin proposed openstack/tripleo-quickstart-extras master: Only deploy ceph before the overcloud in master https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159 | 04:25 |
| *** pojadhav is now known as pojadhav|ruck | 04:25 | |
| *** chkumar|rover is now known as chandankumar | 05:10 | |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Neutron: Deprecate unmaintained features https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846249 | 05:18 |
| chandankumar | tkajinam: hello | 05:20 |
| chandankumar | tkajinam: please have a look at this issue https://bugs.launchpad.net/tripleo/+bug/1978997 when free, thanks! | 05:20 |
| tkajinam | chandankumar, I've left a comment but that looks like a pacemaker issue | 05:28 |
| tkajinam | pacemaker does never promote redis | 05:28 |
| tkajinam | might need someone form pidone team to check it. I suspect there were some change made in pacemaker but I'm not quite sure atm | 05:29 |
| chandankumar | tkajinam: thanks, will ask dciabrin to take a look https://bugs.launchpad.net/tripleo/+bug/1978997 when he is around, :-) | 05:29 |
| opendevreview | Pranali Deore proposed openstack/tripleo-heat-templates master: Enabling ``use_keystone_limits`` in glance to use keystone limits https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846099 | 05:30 |
| tkajinam | chandankumar, can you compare pacemaker version ? current one and the one used in the last successful run | 05:30 |
| chandankumar | sure | 05:30 |
| chandankumar | tkajinam: https://bugs.launchpad.net/tripleo/+bug/1978997/comments/2 | 05:35 |
| chandankumar | new pacemaker got added there in failing job | 05:36 |
| tkajinam | chandankumar, hmm. though the change is supposed to be minor | 05:47 |
| mrunge | chandankumar, tkajinam I believe the redis/pacemaker issue is not the root cause | 05:51 |
| mrunge | gnocchi is aborting requests to send data, my gut feeling is, it doesn't work with the version of sqlalchemy | 05:52 |
| mrunge | https://ac8676a2e4bb22cb45b3-6ec72dc5947dcd41834023b354511109.ssl.cf5.rackcdn.com/842370/6/gate/tripleo-ci-centos-9-scenario001-standalone/93b1333/logs/undercloud/var/log/containers/gnocchi/app.log | 05:52 |
| tkajinam | mrunge, hmm, that might be an issue but according to pacemaker log, it never promoted redis and kept it unpromoted status | 05:56 |
| mrunge | tkajinam, so, there are two issues then? | 05:56 |
| tkajinam | mrunge, I guess haproxy is disconnecting because of no badkends available | 05:56 |
| mrunge | the connection between gnocchi and redis is unreliable, gnocchi will just retry | 05:57 |
| tkajinam | mrunge, If you are talking about that SAWarning then read this https://review.opendev.org/c/openstack/heat/+/845866 | 05:58 |
| tkajinam | I think that is just a warning. I can propose a patch to get rid of that warning | 05:58 |
| mrunge | but it is a problem (over time), when gnocchi can never connect to redis | 05:58 |
| tkajinam | yeah | 05:58 |
| tkajinam | that's what I understood from the error output | 05:59 |
| mrunge | tkajinam, the sqlalchemy warning may be another issue. fact is, that one can not send metrics to gnocchi, gnocchi does not accept new metrics | 05:59 |
| mrunge | I could debug that down to gnocchi trying to recreate an existing archive policy, which then fails | 05:59 |
| mrunge | question is, why the archive policy is not found in the first place | 06:00 |
| tkajinam | ahh, ok | 06:00 |
| tkajinam | I think we can look into the redis problem first and once that is sorted then that gnocchi issue | 06:00 |
| mrunge | gnocchi not being able to stand up as quickly as usual may be causing issues too | 06:00 |
| mrunge | I wonder if the gnocchi not accepting metrics is a fall-out from srbac we found only now | 06:04 |
| mrunge | and just by coincidence | 06:04 |
| tkajinam | I think it should not be affected by srbac now because we still enable legacy rules and has never enforce scope. | 06:05 |
| mrunge | there goes my theory of the user in gnocchi not being allowed to list archive-policies for whatever reason | 06:06 |
| tkajinam | hmm | 06:11 |
| tkajinam | might need to check the policy file deployed. let me check | 06:11 |
| tkajinam | I think the policy-in-code implementation which I proposed is not getting released. the feature presents only in master afaik | 06:11 |
| mrunge | gnocchi in pip is version 4.4. which is not supported by gnocchi upstream anymore | 06:12 |
| mrunge | 4.5 is not released yet | 06:12 |
| tkajinam | mrunge, I vaguely guess I broke something by https://github.com/rdo-packages/gnocchi-distgit/commit/80fb0cb9a3ef595af9e894ffebb7ad0a247b2d32 | 06:20 |
| tkajinam | https://github.com/gnocchixyz/gnocchi/commit/9d14a3183a15baa8e30cd47fb061159820b6e2fe | 06:20 |
| tkajinam | in the past we generate yaml from json during configuration but we are now using the yaml file included in the gnocchi repo | 06:20 |
| tkajinam | I suspect the policy rule provided by that file has something wrong. though it's not quite clear what is wrong. | 06:21 |
| tkajinam | mrunge, that SAWarning should be fixed by this, https://github.com/kvesteri/sqlalchemy-utils/commit/2f4058b20332b5fed06a437ebd34551025504b70 | 06:22 |
| tkajinam | sqlalchemy-utils >= 0.38.1 is required | 06:22 |
| mrunge | tkajinam, if the issue of gnocchi not accepting metrics is unrelated to sqlalchemy versions, it should exist in earlier openstack releases too | 06:29 |
| tkajinam | mrunge, ok I noticed you pointed a different warning | 06:30 |
| tkajinam | yeah | 06:30 |
| *** amoralej|off is now known as amoralej | 06:31 | |
| dciabrin | morning o/ | 06:54 |
| dciabrin | tkajinam, mrunge i just saw redis in the chat backlog? I seem to rememeber we disable redis by default in master nowadays, maybe this job is not expecting it? I'm checking the logs as i'm writing that, so maybe it's unrelated | 06:55 |
| tkajinam | dciabrin, redis is enabled in that job and the container is started but somehow promotion (removing the invalid master) is not working for some reason | 06:58 |
| dciabrin | oh | 06:59 |
| *** jpena|off is now known as jpena | 07:01 | |
| *** arxcruz is now known as arxcruz|rover | 07:08 | |
| tkajinam | hello. does anybody from the core team care to merge this victoria backport ? https://review.opendev.org/c/openstack/os-net-config/+/843384 | 07:36 |
| tkajinam | train backport is also awaiting for 2nd +2 https://review.opendev.org/c/openstack/os-net-config/+/843385 | 07:36 |
| opendevreview | Merged openstack/python-tripleoclient stable/victoria: Remove deprecation warnings for the role command https://review.opendev.org/c/openstack/python-tripleoclient/+/804714 | 07:42 |
| opendevreview | Merged openstack/tripleo-quickstart master: Temp downgrade of libvirt 8.4.x - failing tempest https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194 | 07:42 |
| opendevreview | Merged openstack/tripleo-quickstart-extras master: Revert "Temporarily freeze openssl" https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846125 | 07:42 |
| *** ysandeep|out is now known as ysandeep|afk | 08:10 | |
| dciabrin | tkajinam, marios is wondering if the victoria backport is required? | 08:21 |
| tkajinam | dciabrin, it can be skipped in general as we are EOLing the branch | 08:22 |
| tkajinam | but for above case I prefer moving that victoria backport forward because if we abandone that then I need to update train backport to remove the commit id, which wipes existing +2 | 08:22 |
| dciabrin | ah got you. | 08:22 |
| dciabrin | i +a'd it | 08:23 |
| tkajinam | dciabrin, thx ! | 08:23 |
| tkajinam | if it gets stuck in gate then I'll abandon it anyway instead of try fixing CI now | 08:24 |
| dciabrin | ack | 08:24 |
| chandankumar | dciabrin: tkajinam: mrunge anything we can do currently to temp fix the sc1 and unblock the gates? | 08:25 |
| marios | dciabrin: o/ which patch ? | 08:27 |
| dciabrin | chandankumar, I see that the failing job has resource-agents-4.10.0-17.el9.x86_64 instead of resource-agents-4.10.0-15.el9.x86_64 but i don't know yet if there's any difference in the redis resource agent | 08:27 |
| marios | dciabrin: tkajinam: k that one must be 10:36 < tkajinam> hello. does anybody from the core team care to merge this victoria backport ? https://review.opendev.org/c/openstack/os-net-config/+/843384 | 08:27 |
| dciabrin | marios, yes. but tkajinam cleared my question. sorry for the noise, you heard nothing *jedi trick* | 08:28 |
| marios | what was i doing again? | 08:28 |
| marios | ah yes coffe | 08:28 |
| marios | brb | 08:28 |
| marios | tkajinam: ack i'll check next week anyway before i redo the releases thing ... no problem if we have to wait if there are any more issues like that | 08:28 |
| dciabrin | lol | 08:29 |
| marios | ;) | 08:29 |
| tkajinam | marios, ack, though that would not become a blocker :-) | 08:29 |
| tkajinam | chandankumar, immediate workaround might be deploy redis as a standalone service, or maybe try downgrading pacemaker packages + resource agents | 08:30 |
| marios | chandankumar: so https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194 didn't help or this is new issue now :/ | 08:31 |
| marios | chandankumar: sorry i see it now https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159/4..5#message-2e6b6aed542fc6906dab6eb0585bb3bd899a4931 | 08:31 |
| dciabrin | chandankumar, can you easily pin pacemaker* + resource-agents to the prior versions? | 08:42 |
| chandankumar | dciabrin: let me try that | 08:43 |
| chandankumar | marios: ipa job got fixed, sc1 failure investigation is still going on | 08:44 |
| tkajinam | 2022-06-09 - Oyvind Albrigtsen <oalbrigt@redhat.com> - 4.10.0-17 | 08:48 |
| tkajinam | - lvmlockd: fail when use_lvmlockd has not been set | 08:48 |
| tkajinam | - aws-vpc-move-ip: add interface label support | 08:48 |
| tkajinam | Resolves: rhbz#2094828 | 08:48 |
| tkajinam | Resolves: rhbz#2093213 | 08:48 |
| tkajinam | does not look related | 08:48 |
| dciabrin | tkajinam, i have to figure out what the redis resource agent is doing, I'm surprised I see no mention of this master-redis flag being set in the logs | 08:54 |
| tkajinam | yeah | 08:58 |
| tkajinam | dciabrin, I tend to suspect something wrong within core pacemaker because it never attempt to promote, but there might be something wrong with resource agent causing that. not familiar with the actual logic | 08:59 |
| tkajinam | Jun 17 07:05:18.749 standalone.localdomain pacemaker-schedulerd[61286] (unpack_rsc_op_failure) warning: Unexpected result (error: Resource agent did not complete within 20s) was recorded for monitor of redis-bundle-podman-0 on standalone at Jun 17 07:05:02 2022 | rc=1 id=redis-bundle-podman-0_last_failure_0 | 08:59 |
| tkajinam | this sounds storage but relationship is not quite clear, atm | 08:59 |
| * tkajinam is checking pacemaker log | 08:59 | |
| dciabrin | tkajinam, the theory should be that at some point a monitoring operation should trigger a promotion, hence my surprise | 09:02 |
| dciabrin | as monitors happen all the time, even after one monitor op has failed | 09:02 |
| opendevreview | chandan kumar proposed openstack/tripleo-common master: [DNM] Downgrade pacemaker and resource-agents https://review.opendev.org/c/openstack/tripleo-common/+/846287 | 09:22 |
| chandankumar | dciabrin: ^^ there is no way to pin the pacemaker, so I am downgrading it | 09:23 |
| dciabrin | chandankumar, oh ok | 09:23 |
| chandankumar | content provider will build the new container and test it in sc1 job | 09:23 |
| chandankumar | https://zuul.opendev.org/t/openstack/status#846287 | 09:23 |
| tkajinam | I checked changes recently merged but could not find clearly suspicious change... | 09:31 |
| * tkajinam needs to be away for a while | 09:31 | |
| opendevreview | chandan kumar proposed openstack/tripleo-ci master: Switch to ubi9 containers_base_image image https://review.opendev.org/c/openstack/tripleo-ci/+/843787 | 09:35 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Neutron: Remove NSX plugin support https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846289 | 09:45 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: Neutron: Remove NSX plugin support https://review.opendev.org/c/openstack/puppet-tripleo/+/846290 | 09:48 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-common master: Remove unused oslo.vmware from Neutron image https://review.opendev.org/c/openstack/tripleo-common/+/846291 | 09:50 |
| *** rlandy|out is now known as rlandy | 10:30 | |
| opendevreview | Rajesh Tailor proposed openstack/tripleo-ansible master: Remove - from endfor statement https://review.opendev.org/c/openstack/tripleo-ansible/+/845333 | 10:40 |
| opendevreview | Gorka Eguileor proposed openstack/tripleo-common master: Reduce memory consumption in Cinder services https://review.opendev.org/c/openstack/tripleo-common/+/845807 | 10:43 |
| opendevreview | Brent Eagles proposed openstack/puppet-tripleo stable/wallaby: Redis required for designate central service https://review.opendev.org/c/openstack/puppet-tripleo/+/846338 | 11:01 |
| opendevreview | Francesco Pantano proposed openstack/tripleo-ansible master: Add ssl spec parameter to rgw https://review.opendev.org/c/openstack/tripleo-ansible/+/846340 | 11:05 |
| *** ysandeep|afk is now known as ysandeep | 11:09 | |
| opendevreview | Marios Andreou proposed openstack/tripleo-quickstart-extras master: WIP add some new plays for mixed mode https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/841764 | 11:39 |
| *** ysandeep is now known as ysandeep|afk | 11:47 | |
| fultonj | mornings | 11:49 |
| opendevreview | Manojkatari proposed openstack/puppet-tripleo master: Support to configure cinder backend_defaults https://review.opendev.org/c/openstack/puppet-tripleo/+/845854 | 12:08 |
| fmount | fultonj: o/ thanks for your comment in https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 | 12:15 |
| fultonj | fmount: yeah sure | 12:15 |
| fultonj | it's the longest commit message for 1 line of code i've done :/ | 12:15 |
| fmount | ahahah yeah | 12:15 |
| fultonj | but ther's a lot of subtly | 12:15 |
| fmount | I was actually a bit confused by the last part of the comment | 12:16 |
| fmount | but now I think I get it | 12:16 |
| fmount | you use tripleo operator to deploy ceph | 12:16 |
| fultonj | the push_destination is tricky | 12:16 |
| fmount | but it calls tripleoclient right? | 12:16 |
| fultonj | yes | 12:16 |
| fultonj | tripleo-ansible-operator | 12:16 |
| fultonj | not to be confused with go-operators or people who use tripleo :) | 12:17 |
| fmount | so the ceph_ns is actually the extravar that tripleoclient sets | 12:17 |
| fmount | tripleo-ansible-operator yes | 12:17 |
| fmount | I meant ^^ | 12:17 |
| fultonj | yeah lots of work to set an extra_vars when tripleo_cephadm role is called | 12:17 |
| fultonj | https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v2/overcloud_ceph.py#L564-L566 | 12:18 |
| *** amoralej is now known as amoralej|lunch | 12:19 | |
| fultonj | fmount: ^ extra_vars['tripleo_cephadm_container_ns'] will use what came out of parsing container image prepare (but the parsing sneaks in undercloud registry) | 12:19 |
| fultonj | UNLESS parsed_args.container_namespace is there | 12:19 |
| fultonj | tripleo-ansible-operator sneaks that in | 12:20 |
| fultonj | standalone_container_ceph_updates only when is true | 12:20 |
| fultonj | else it's omit | 12:20 |
| fultonj | fmount: does it makes sense? | 12:21 |
| fmount | yeah fultonj | 12:21 |
| fmount | makes sense thanks :D | 12:22 |
| fultonj | more importantly, i want to see what the ci did with it | 12:25 |
| rlandy | https://review.opendev.org/c/openstack/tripleo-common/+/846287/ - got a zuul +1 | 12:26 |
| rlandy | can we remove the DNM and go with this? | 12:26 |
| fultonj | rlandy: sounds good me me | 12:26 |
| rlandy | chandankumar: fmount: dciabrin: ^^? | 12:26 |
| rlandy | fultonj: two fixes merged - two to go | 12:27 |
| fultonj | yay progress | 12:27 |
| fmount | wow nice :D | 12:28 |
| fmount | let's move forward then | 12:28 |
| rlandy | k - removing DNM - updating commit message | 12:28 |
| fmount | ++ rlandy thanks | 12:28 |
| dciabrin | rlandy, i'll +2 it right after, thanks | 12:28 |
| dciabrin | tkajinam, ramishra, https://paste.opendev.org/show/b3MbIknb2FT4jRB5GutT/ i'm trying to deploy a standalone locally, but it's erroring out on me in a way I never saw before. Does that ring a bell to you maybe? | 12:30 |
| chandan_kumar | dciabrin pacemaker downgrade worked | 12:33 |
| opendevreview | Ronelle Landy proposed openstack/tripleo-common master: Downgrade pacemaker and resource-agents https://review.opendev.org/c/openstack/tripleo-common/+/846287 | 12:33 |
| dciabrin | chandan_kumar, yup, I just +2'd it | 12:33 |
| dciabrin | thx! | 12:33 |
| chandan_kumar | CentOS stream updates brought these damages | 12:34 |
| rlandy | https://review.opendev.org/c/openstack/tripleo-common/+/846287 should be good to go | 12:34 |
| rlandy | dciabrin: thanks for review | 12:35 |
| rlandy | fmount: ^^ if you want to add your vote | 12:35 |
| rlandy | will merge when zuul revotes there | 12:35 |
| rlandy | arxcruz|rover: pojadhav|ruck: ^^ fyi | 12:35 |
| fmount | rlandy: done! | 12:36 |
| rlandy | great | 12:36 |
| rlandy | getting there after yesterday's disaster | 12:36 |
| tkajinam | dciabrin, hmmmm, no. | 12:37 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-common master: wip: testing downgrading resource agents https://review.opendev.org/c/openstack/tripleo-common/+/846351 | 12:39 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-common master: wip: testing downgrading pacemaker/pacemaker-remote https://review.opendev.org/c/openstack/tripleo-common/+/846352 | 12:40 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-common master: wip: Revert "Downgrade pacemaker and resource-agents" https://review.opendev.org/c/openstack/tripleo-common/+/846353 | 12:41 |
| tkajinam | ^^^ this would allows us to know which is causing the problem | 12:41 |
| * tkajinam is abusing ci resource for testing :-P | 12:42 | |
| tkajinam | wait | 12:43 |
| tkajinam | https://review.opendev.org/c/openstack/tripleo-common/+/846291 | 12:43 |
| tkajinam | tripleo-ci-centos-9-standalone passed | 12:43 |
| tkajinam | ahh, not that one | 12:44 |
| tkajinam | need to check scenario001 | 12:44 |
| *** dviroel|out is now known as dviroel | 13:00 | |
| *** amoralej|lunch is now known as amoralej | 13:03 | |
| opendevreview | Francesco Pantano proposed openstack/tripleo-common stable/wallaby: Split off Ceph related container images https://review.opendev.org/c/openstack/tripleo-common/+/846356 | 13:06 |
| opendevreview | Francesco Pantano proposed openstack/tripleo-common stable/wallaby: Split off Ceph related container images https://review.opendev.org/c/openstack/tripleo-common/+/846356 | 13:07 |
| opendevreview | Francesco Pantano proposed openstack/tripleo-common stable/wallaby: Split off Ceph related container images https://review.opendev.org/c/openstack/tripleo-common/+/846356 | 13:08 |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Only deploy ceph before the overcloud in master https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159 | 13:13 |
| rlandy | arxcruz|rover: pojadhav|ruck: fultonj: ^^ added the depends-on triple-common patch | 13:13 |
| rlandy | let's see if that clears zuul now | 13:13 |
| arxcruz|rover | ok | 13:14 |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Only deploy ceph before the overcloud in master https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159 | 13:14 |
| opendevreview | Francesco Pantano proposed openstack/tripleo-common stable/wallaby: Split off Ceph related container images https://review.opendev.org/c/openstack/tripleo-common/+/846356 | 13:15 |
| *** ysandeep|afk is now known as ysandeep | 13:16 | |
| pojadhav|ruck | rlandy, ack | 13:18 |
| *** pojadhav|ruck is now known as pojadhav|dinner | 13:18 | |
| fultonj | thanks rlandy | 13:30 |
| *** pojadhav|dinner is now known as pojadhav|ruck | 13:40 | |
| *** dviroel is now known as dviroel|pto | 14:03 | |
| *** ysandeep is now known as ysandeep|out | 14:04 | |
| opendevreview | Rabi Mishra proposed openstack/tripleo-heat-templates stable/wallaby: Remove legacy network-isolation env files https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846255 | 14:04 |
| fultonj | rlandy: FYI i'll be AFK for the next 2 hours but back in the afternoon | 14:09 |
| fultonj | fmount: fyi ^ | 14:09 |
| rlandy | fultonj: k - np - do you just want to w+ the tripleo-common patch? | 14:10 |
| rlandy | I'll take care of w+ on the t-q-e one | 14:11 |
| fultonj | https://review.opendev.org/c/openstack/tripleo-common/+/846287 | 14:11 |
| fultonj | +2 +w | 14:11 |
| rlandy | cool - thanks | 14:12 |
| fmount | fultonj: np, rlandy feel free to ping me if we need any other review! | 14:12 |
| fultonj | i guess we want to know next why https://review.rdoproject.org/r/c/testproject/+/36256 which tested https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 failed | 14:15 |
| fultonj | i'll try to answer that later | 14:15 |
| fmount | fultonj: https://logserver.rdoproject.org/56/36256/94/check/periodic-tripleo-ci-centos-9-scenario001-standalone-master/444a82e/logs/quickstart_install.log | 14:18 |
| fmount | "openstack overcloud ceph deploy: error: unrecognized arguments: ternary('quay.rdoproject.org/tripleomastercentos9', omit)"] | 14:19 |
| fmount | I suspect we need to update https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 | 14:19 |
| opendevreview | Francesco Pantano proposed openstack/tripleo-quickstart-extras master: Override Ceph --container-namespace for periodic jobs https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 | 14:22 |
| fmount | rlandy: /me trying to push an update ^^ | 14:22 |
| rlandy | ok - let's see | 14:24 |
| opendevreview | Francesco Pantano proposed openstack/tripleo-quickstart-extras master: Override Ceph --container-namespace for periodic jobs https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 | 14:25 |
| *** pojadhav|ruck is now known as pojadhav|out | 14:26 | |
| opendevreview | Merged openstack/tripleo-ci master: Added standalone-full-tempest-api & -scenario -options jobs https://review.opendev.org/c/openstack/tripleo-ci/+/843268 | 14:51 |
| opendevreview | Merged openstack/tripleo-ansible master: Set current ceph dashboard mgr backend fact https://review.opendev.org/c/openstack/tripleo-ansible/+/845115 | 15:08 |
| fmount | rlandy: following up on the last failure related to: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 | 15:19 |
| fmount | is https://review.rdoproject.org/r/c/testproject/+/36256/95/.zuul.yaml a valid change? | 15:19 |
| fmount | rlandy: this variable -> standalone_container_ceph_updates is false by default, unless explicitly defined in the job as true | 15:20 |
| fmount | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/standalone/defaults/main.yml | 15:20 |
| fmount | ^^ so I want to check first that fultonj's theory about overriding the container namespace works as expected | 15:21 |
| rlandy | looking | 15:21 |
| rlandy | standalone_container_ceph_updates: "{{ job.standalone_container_ceph_updates|default(false)|bool }}" | 15:22 |
| rlandy | that would work | 15:22 |
| rlandy | because it references job.standalone_container_ceph_updates | 15:22 |
| fmount | ++ perfect | 15:22 |
| fmount | I just wanted to confirm | 15:22 |
| rlandy | and vars | 15:23 |
| rlandy | will give you a job. variable | 15:23 |
| fmount | ok let's see the next iteration of https://review.rdoproject.org/r/c/testproject/+/36256 | 15:23 |
| *** marios is now known as marios|out | 15:25 | |
| opendevreview | Francesco Pantano proposed openstack/tripleo-ansible stable/wallaby: Set current ceph dashboard mgr backend fact https://review.opendev.org/c/openstack/tripleo-ansible/+/846256 | 15:25 |
| fmount | rlandy: nice, we have progress here -> https://review.rdoproject.org/zuul/stream/7e501842c5174f219cf523688a17041b?logfile=console.log | 15:39 |
| fmount | and "CONTAINER_NAMESPACE": "quay.rdoproject.org/tripleomastercentos9" is set as expected | 15:39 |
| fmount | I have to update the quickstart patch one last time but I'm waiting for the testproject to finish | 15:40 |
| fmount | fultonj: ^ | 15:40 |
| fmount | :D | 15:40 |
| opendevreview | Merged openstack/tripleo-ci master: Revert "Install Python38 on CentOS-8" https://review.opendev.org/c/openstack/tripleo-ci/+/846113 | 15:45 |
| *** amoralej is now known as amoralej|off | 15:51 | |
| rlandy | fmount: awesome | 15:55 |
| rlandy | lunch quickly - wil check back in a few | 15:55 |
| fmount | rlandy: take your time! ceph is done on testproject, just waiting for results before updating the quickstart change :D | 15:57 |
| fmount | so good news ^^ :D | 15:57 |
| opendevreview | Sofer Athlan-Guyot proposed openstack/tripleo-upgrade stable/train: Fix validation inventory setup. https://review.opendev.org/c/openstack/tripleo-upgrade/+/846134 | 16:08 |
| opendevreview | Francesco Pantano proposed openstack/tripleo-common stable/train: Split off Ceph related container images https://review.opendev.org/c/openstack/tripleo-common/+/846421 | 16:14 |
| *** jpena is now known as jpena|off | 16:15 | |
| opendevreview | Merged openstack/tripleo-ansible master: Add ceph_config_set before applying the spec https://review.opendev.org/c/openstack/tripleo-ansible/+/845111 | 16:27 |
| rlandy | fmount: hmmm ... https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159/ failing scenario001 | 16:27 |
| rlandy | do I need your change? | 16:27 |
| fmount | I think so ^ | 16:28 |
| * fmount double checking the failure | 16:28 | |
| fmount | rlandy: nono this seems different | 16:29 |
| fmount | ceph passed there | 16:29 |
| fmount | I mean deployed ceph | 16:29 |
| rlandy | ok - checking | 16:30 |
| fmount | and also | 16:30 |
| fmount | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2f7/846159/7/check/tripleo-ci-centos-9-scenario001-standalone/2f7f159/logs/undercloud/home/zuul/tripleo-deploy/standalone-ansible-6arj_t2v/cephadm/cephadm_command.log | 16:30 |
| fmount | during the OC deployment ceph looks good ^^ | 16:30 |
| fmount | so I guess it's unrelated | 16:30 |
| fmount | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2f7/846159/7/check/tripleo-ci-centos-9-scenario001-standalone/2f7f159/logs/undercloud/home/zuul/ansible.log | 16:31 |
| fmount | edis.exceptions.ConnectionError: Connection closed by server. | 16:31 |
| fmount | ^^^ I guess we know what's going on there ^^ | 16:31 |
| opendevreview | Francesco Pantano proposed openstack/tripleo-quickstart-extras master: Override Ceph --container-namespace for periodic jobs https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 | 16:41 |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Only deploy ceph before the overcloud in master https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159 | 16:42 |
| rlandy | fmount: my fault - I depended on wrong patch | 16:43 |
| fmount | rlandy: np, so both zuul and testproject should be good on that change | 16:46 |
| fmount | I updated the patch to put a default() on the bool value, and rechecked the testproject | 16:46 |
| opendevreview | Francesco Pantano proposed openstack/tripleo-ansible stable/wallaby: Add ceph_config_set before applying the spec https://review.opendev.org/c/openstack/tripleo-ansible/+/846258 | 16:48 |
| fultonj | o/ | 16:53 |
| fmount | rlandy: I'm going afk, but fultonj is back :D | 16:53 |
| fultonj | thanks fmount | 16:53 |
| rlandy | k - have a good weekend | 16:53 |
| rlandy | on 1-1 | 16:53 |
| rlandy | will check back | 16:53 |
| fultonj | anything important i should see? | 16:54 |
| * fultonj starts reading backlog | 16:54 | |
| fmount | fultonj: you can check the update on the quickstart patch | 16:54 |
| fultonj | i saw you added default(false) which is good | 16:54 |
| fmount | I pushed an update, CI looks good now (for ceph) and failing on the gnocchi/redis stuff | 16:54 |
| fmount | now let's see the next iteration | 16:55 |
| fmount | fultonj: testproject -> https://review.rdoproject.org/zuul/stream/90a61424885a44508e29e8b79ed91d8c?logfile=console.log | 16:55 |
| fultonj | thx | 16:55 |
| fmount | have a nice weekend! :D | 16:56 |
| fmount | ttyM ! | 16:56 |
| fultonj | https://review.rdoproject.org/zuul/stream/90a61424885a44508e29e8b79ed91d8c?logfile=console.log | 17:00 |
| fultonj | thansk fmount | 17:00 |
| fultonj | ceph is now beingdepoyed in the console with the desired CONTAINER_NAMESPACE | 17:00 |
| fultonj | and https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 is confirmed to fix the bug ^ | 17:03 |
| fultonj | now waiting on overcloud deploy | 17:03 |
| opendevreview | Bogdan Dobrelya proposed openstack/tripleo-ansible master: Add nova-compute standalone role https://review.opendev.org/c/openstack/tripleo-ansible/+/843659 | 17:13 |
| opendevreview | Bogdan Dobrelya proposed openstack/tripleo-ansible master: Example hiera bindings fixes based on tht data https://review.opendev.org/c/openstack/tripleo-ansible/+/846431 | 17:13 |
| fultonj | rlandy: https://review.rdoproject.org/r/c/testproject/+/36256 should depends-on both https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 AND https://review.opendev.org/c/openstack/tripleo-common/+/846287 because it failed on the gnocchi issue | 18:21 |
| fultonj | i have made the update | 18:26 |
| fultonj | etherpad updated in both ceph active bugs we're awaiting results | 18:31 |
| rlandy | ok | 18:43 |
| rlandy | thank you | 18:44 |
| rlandy | will try get them all through before monday | 18:44 |
| opendevreview | Merged openstack/tripleo-common master: Downgrade pacemaker and resource-agents https://review.opendev.org/c/openstack/tripleo-common/+/846287 | 19:28 |
| opendevreview | John Fulton proposed openstack/python-tripleoclient master: Fall back to alt path for ceph client data during export https://review.opendev.org/c/openstack/python-tripleoclient/+/846441 | 20:12 |
| opendevreview | John Fulton proposed openstack/tripleo-ansible master: Set tripleo_ceph_client_vars consistently with THT https://review.opendev.org/c/openstack/tripleo-ansible/+/846443 | 20:31 |
| fultonj | rlandy: https://review.rdoproject.org/zuul/build/003c9d67c22540c9893a9825b1b60aff failed for the intermittent user creation issue ; a recheck should get it past though I'll open a bug to track that | 20:38 |
| fultonj | https://bugs.launchpad.net/tripleo/+bug/1979093 | 20:49 |
| rlandy | fultonj: thanks - also had to recheck | 21:25 |
| rlandy | will try again n sunday if this run fails | 21:26 |
| opendevreview | Francesco Pantano proposed openstack/tripleo-quickstart-extras master: Override Ceph --container-namespace for periodic jobs https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 | 21:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!