| opendevreview | Merged openstack/tripleo-common master: Remove /etc/rabbitmq/rabbitmq.conf https://review.opendev.org/c/openstack/tripleo-common/+/809207 | 00:40 |
|---|---|---|
| *** odyssey4me is now known as Guest104 | 00:45 | |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates stable/victoria: Wallaby and before: Fix wrong puppet parameter name https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809493 | 00:46 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates stable/ussuri: Wallaby and before: Fix wrong puppet parameter name https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809494 | 00:46 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates stable/train: Wallaby and before: Fix wrong puppet parameter name https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809495 | 00:47 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo stable/wallaby: Replace deprecated ceilometer::agent::auth https://review.opendev.org/c/openstack/puppet-tripleo/+/809496 | 00:51 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates stable/wallaby: Remove hieratada for ceilometer::agent::auth https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809497 | 00:52 |
| *** odyssey4me is now known as Guest105 | 00:52 | |
| *** pmannidi is now known as pmannidi|AFK | 00:59 | |
| *** pojadhav|out is now known as pojadhav | 03:54 | |
| *** odyssey4me is now known as Guest124 | 04:02 | |
| opendevreview | Merged openstack/tripleo-quickstart master: Add defaults overcloud_image as overcloud-full https://review.opendev.org/c/openstack/tripleo-quickstart/+/809367 | 04:20 |
| opendevreview | Rabi Mishra proposed openstack/tripleo-heat-templates master: Revert "Configure ironic to do UEFI boot by default" https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809311 | 05:14 |
| opendevreview | Rabi Mishra proposed openstack/tripleo-heat-templates master: Deploy standalone ironic and neutron in undercloud https://review.opendev.org/c/openstack/tripleo-heat-templates/+/794912 | 05:14 |
| opendevreview | Rabi Mishra proposed openstack/tripleo-heat-templates master: Add environment to enable keystone in undercloud https://review.opendev.org/c/openstack/tripleo-heat-templates/+/799408 | 05:14 |
| *** ykarel|away is now known as ykarel | 05:39 | |
| opendevreview | Michele Baldessari proposed openstack/tripleo-common stable/wallaby: Remove /etc/rabbitmq/rabbitmq.conf https://review.opendev.org/c/openstack/tripleo-common/+/809498 | 05:42 |
| opendevreview | Bhagyashri Shewale proposed openstack/tripleo-ci master: [WIP] Remove jobs corresponds to obselete featuresets https://review.opendev.org/c/openstack/tripleo-ci/+/808059 | 06:25 |
| opendevreview | Cedric Jeanneret proposed openstack/tripleo-ansible master: Allow to disable GSSAPI authentication https://review.opendev.org/c/openstack/tripleo-ansible/+/759057 | 06:49 |
| *** amoralej|off is now known as amoralej | 06:54 | |
| *** ysandeep|out is now known as ysandeep | 07:20 | |
| opendevreview | Bhagyashri Shewale proposed openstack/tripleo-ci master: [WIP] Remove jobs corresponds to obselete featuresets https://review.opendev.org/c/openstack/tripleo-ci/+/808059 | 07:22 |
| *** jpena|off is now known as jpena | 07:26 | |
| *** arxcruz|pto is now known as arxcruz | 07:40 | |
| opendevreview | Arx Cruz proposed openstack/ansible-role-collect-logs master: DNM - Dummy patch to test molecule https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/809535 | 07:44 |
| opendevreview | Slawek Kaplonski proposed openstack/openstack-tempest-skiplist master: DNM Revert "Adding full-tempest-scenario-master to neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle" https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/809507 | 07:53 |
| opendevreview | Bhagyashri Shewale proposed openstack/tripleo-ci master: [WIP] Remove jobs corresponds to obselete featuresets https://review.opendev.org/c/openstack/tripleo-ci/+/808059 | 07:54 |
| *** odyssey4me is now known as Guest139 | 08:03 | |
| Tengu | folks, care to merge this small change? https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809427 | 08:04 |
| Tengu | bogdando, ramishra if you have a small cycle :) -^^ thanks! | 08:04 |
| Tengu | thanks :) | 08:08 |
| opendevreview | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Fix kill-script for cgroups v2 and podman https://review.opendev.org/c/openstack/tripleo-heat-templates/+/807609 | 08:13 |
| *** odyssey4me is now known as Guest140 | 08:27 | |
| opendevreview | Rabi Mishra proposed openstack/tripleo-heat-templates master: WIP Don't use *-dist.conf files https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809540 | 08:28 |
| bogdando | Tengu: urw | 08:35 |
| Tengu | bogdando: need some help for testing your kill-script patch? | 08:36 |
| bogdando | Tengu: I pupshed an update that should work for both cases now | 08:37 |
| bogdando | v1 and v2 cgroups | 08:37 |
| Tengu | cool. yeah, better ensuring support for both. | 08:37 |
| opendevreview | Ananya proposed openstack/tripleo-ci-health-queries master: removing "Nova failure" from no valid host was found description https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/809577 | 08:55 |
| opendevreview | Vijayalakshmi proposed openstack/os-net-config master: Execution of os-net-config-sriov service is done before driverctl and network-pre services https://review.opendev.org/c/openstack/os-net-config/+/809578 | 08:57 |
| *** ykarel_ is now known as ykarel | 09:14 | |
| opendevreview | Bogdan Dobrelya proposed openstack/puppet-tripleo master: Fix front/back-end options https://review.opendev.org/c/openstack/puppet-tripleo/+/809411 | 09:19 |
| Tengu | bogdando: regarding https://bugs.launchpad.net/tripleo/+bug/1943170 - I've pushed a patch on molecule (see last comment) that should hopefully solve the issue with the pluggy dependency. | 09:29 |
| bogdando | thanks | 09:30 |
| Tengu | Sorin is on it, so we can expect it to be merged fast. | 09:30 |
| bogdando | Tengu: but can we pin it yet? | 09:30 |
| bogdando | since gates blocked | 09:30 |
| Tengu | pin what? | 09:31 |
| Tengu | https://bugs.launchpad.net/tripleo/+bug/1943170/comments/5 the issue is clearly molecule depending on a too old pluggy compared to the other constraints. | 09:32 |
| bogdando | Tengu: molecule, to not wait for that upstream fix | 09:44 |
| Tengu | bogdando: that wouldn't do anything. | 09:44 |
| bogdando | why? | 09:44 |
| Tengu | bogdando: the issue is: molecule requires a too old pluggy version | 09:44 |
| bogdando | well, there had to be some regression, so we could pin | 09:45 |
| Tengu | all other deps are requesting pluggy==1.0 when molecule currently requires <1.0 | 09:45 |
| bogdando | it used to work before | 09:45 |
| Tengu | well, afaik it comes from higher level (openstack requirements)... not sure we can actually modify it "just like that". | 09:46 |
| opendevreview | Bhagyashri Shewale proposed openstack/tripleo-ci master: [WIP] Remove jobs corresponds to obselete featuresets https://review.opendev.org/c/openstack/tripleo-ci/+/808059 | 09:47 |
| Tengu | " The user requested (constraint) pluggy===1.0.0" this constraint doesn't come from the tripleo-ansible repository, for instance. | 09:47 |
| Tengu | but from the global openstack constraint | 09:47 |
| Tengu | good luck changing it.. but you can try I guess. | 09:47 |
| Tengu | (my molecule PR has 2 approvals already - we might expect it to merge once their CI is done) | 09:48 |
| Tengu | #food. | 09:51 |
| *** ykarel is now known as ykarel|lunch | 10:05 | |
| opendevreview | Bhagyashri Shewale proposed openstack/puppet-tripleo stable/ussuri: Run TripleO jobs on CentOS8 instead of CentOS7 https://review.opendev.org/c/openstack/puppet-tripleo/+/809627 | 10:21 |
| opendevreview | Bhagyashri Shewale proposed openstack/puppet-tripleo stable/train: Run TripleO jobs on CentOS8 instead of CentOS7 https://review.opendev.org/c/openstack/puppet-tripleo/+/809560 | 10:22 |
| opendevreview | Bhagyashri Shewale proposed openstack/puppet-tripleo stable/train: Run TripleO jobs on CentOS8 instead of CentOS7 https://review.opendev.org/c/openstack/puppet-tripleo/+/809560 | 10:23 |
| opendevreview | Bhagyashri Shewale proposed openstack/tripleo-ci master: [WIP] Remove jobs corresponds to obselete featuresets https://review.opendev.org/c/openstack/tripleo-ci/+/808059 | 10:26 |
| opendevreview | Marios Andreou proposed openstack/tripleo-repos master: Add centos7 molecule instance for train/c7/py2 use-case https://review.opendev.org/c/openstack/tripleo-repos/+/809628 | 10:31 |
| opendevreview | Juan Badia Payno proposed openstack/tripleo-quickstart-extras master: Remove --standalone argument from upgrade script https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/809416 | 10:41 |
| opendevreview | Merged openstack/tripleo-heat-templates stable/ussuri: Fix ansible var https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809378 | 10:45 |
| *** ykarel|lunch is now known as ykarel | 10:49 | |
| opendevreview | Merged openstack/tripleo-heat-templates stable/ussuri: dpdk_telemetry is not used https://review.opendev.org/c/openstack/tripleo-heat-templates/+/805780 | 10:53 |
| opendevreview | Merged openstack/python-tripleoclient master: Return "overcloud status" output without debug https://review.opendev.org/c/openstack/python-tripleoclient/+/809442 | 10:53 |
| opendevreview | Sandeep Yadav proposed openstack/python-tripleoclient stable/wallaby: Return "overcloud status" output without debug https://review.opendev.org/c/openstack/python-tripleoclient/+/809561 | 10:55 |
| opendevreview | Sandeep Yadav proposed openstack/python-tripleoclient stable/victoria: Return "overcloud status" output without debug https://review.opendev.org/c/openstack/python-tripleoclient/+/809562 | 10:55 |
| opendevreview | Matthias Runge proposed openstack/tripleo-heat-templates stable/train: dpdk_telemetry is not used https://review.opendev.org/c/openstack/tripleo-heat-templates/+/805758 | 11:07 |
| *** dviroel|out is now known as dviroel | 11:09 | |
| *** rlandy is now known as rlandy|rover | 11:13 | |
| *** pojadhav- is now known as pojadhav|brb | 11:31 | |
| *** jpena is now known as jpena|lunch | 11:32 | |
| *** bhagyashris_ is now known as bhagyashris | 11:43 | |
| Tengu | bogdando: we should get a new molecule with the fix already (3.5.1 apparently) | 11:44 |
| *** rlandy|rover is now known as rlandy|rover|mtg | 11:50 | |
| ratailor | #tripleo did anyone get this error while using oooq for deployment | 11:55 |
| ratailor | 2021-09-17 11:12:37 | 2021-09-17 11:12:37.680 107469 ERROR openstack [-] ERROR: Protected resource registry overrides detected! These entries are used in internal environments and should not be overridden in the user environment. Please remove these overrides from the environment files. | 11:55 |
| ratailor | 2021-09-17 11:12:37 | Conflict detected for resource_registry entry: OS::TripleO::DeployedServer::ControlPlanePort. | 11:55 |
| ratailor | 2021-09-17 11:12:37 | User environment: /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml. | 11:55 |
| ratailor | 2021-09-17 11:12:37 | Internal environment: ['/home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/baremetal-deployed.yaml'] | 11:55 |
| *** ykarel is now known as ykarel|afk | 12:00 | |
| Tengu | hjensas: -^^ that's for you I think :) | 12:01 |
| Tengu | meh... ok. new molecule, but this apparently creates new issues now.... | 12:02 |
| opendevreview | Cedric Jeanneret proposed openstack/tripleo-ansible master: Allow to disable GSSAPI authentication https://review.opendev.org/c/openstack/tripleo-ansible/+/759057 | 12:06 |
| opendevreview | Bhagyashri Shewale proposed openstack/tripleo-ci master: [WIP] Remove jobs corresponds to obselete featuresets https://review.opendev.org/c/openstack/tripleo-ci/+/808059 | 12:19 |
| opendevreview | Damien Ciabrini proposed openstack/tripleo-heat-templates stable/wallaby: WIP drop unused openstack DB users in mysql https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809639 | 12:29 |
| *** jpena|lunch is now known as jpena | 12:29 | |
| *** pojadhav|brb is now known as pojadhav | 12:35 | |
| opendevreview | Damien Ciabrini proposed openstack/tripleo-heat-templates stable/wallaby: WIP drop unused openstack DB users in mysql https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809639 | 12:38 |
| opendevreview | Rajesh Tailor proposed openstack/tripleo-heat-templates master: Enable support for Libvirt modular daemons https://review.opendev.org/c/openstack/tripleo-heat-templates/+/803915 | 12:39 |
| opendevreview | Bhagyashri Shewale proposed openstack/tripleo-ci master: [WIP] Remove jobs corresponds to obselete featuresets https://review.opendev.org/c/openstack/tripleo-ci/+/808059 | 12:40 |
| opendevreview | Kevin Carter proposed openstack/tripleo-ansible master: fix galaxy meta requirements https://review.opendev.org/c/openstack/tripleo-ansible/+/809643 | 12:41 |
| opendevreview | Cedric Jeanneret proposed openstack/tripleo-ansible master: Allow to disable GSSAPI authentication https://review.opendev.org/c/openstack/tripleo-ansible/+/759057 | 12:46 |
| *** amoralej is now known as amoralej|lunch | 12:48 | |
| *** rlandy|rover|mtg is now known as rlandy|rover | 12:50 | |
| bogdando | slaweq: hi PTAL the neutron and OVN parts https://review.opendev.org/c/openstack/puppet-tripleo/+/809411 | 13:00 |
| bogdando | mschuppert: ^^ for the nova part please | 13:00 |
| slaweq | bogdando: sure | 13:01 |
| mschuppert | bogdando done thanks | 13:03 |
| opendevreview | Marios Andreou proposed openstack/tripleo-quickstart master: Partial revert for tripleo-quickstart/+/791486 for centos7 https://review.opendev.org/c/openstack/tripleo-quickstart/+/809670 | 13:12 |
| marios | weshay|ruck: rlandy|rover: anbanerj|ruck: ^^^ fyi to unblock train for now | 13:12 |
| marios | weshay|ruck: to buy us time for a proper fix | 13:13 |
| rlandy|rover | marios: tested? | 13:19 |
| opendevreview | Sandeep Yadav proposed openstack/python-tripleoclient stable/victoria: Return "overcloud status" output without debug https://review.opendev.org/c/openstack/python-tripleoclient/+/809562 | 13:20 |
| *** ykarel|afk is now known as ykarel | 13:20 | |
| opendevreview | Sandeep Yadav proposed openstack/python-tripleoclient stable/victoria: Return "overcloud status" output without debug https://review.opendev.org/c/openstack/python-tripleoclient/+/809562 | 13:21 |
| marios | rlandy|rover: ack will post in a sec | 13:34 |
| rlandy|rover | marios: thanks - will vote when ^^ posted | 13:34 |
| *** amoralej|lunch is now known as amoralej | 13:34 | |
| marios | rlandy|rover: should be ok as it is just paste of what we had there before (doesn't touch new code) but ack for sanity check | 13:34 |
| hjensas | Tengu: see https://review.opendev.org/c/openstack/tripleo-quickstart/+/807703, and https://review.opendev.org/c/openstack/tripleo-quickstart/+/807704 and https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/806562 | 13:38 |
| hjensas | Tengu: there may be other places that need the protection disabled until the other patches merge. | 13:39 |
| hjensas | ratailor is not around any longer it seems? | 13:39 |
| opendevreview | Slawek Kaplonski proposed openstack/tripleo-heat-templates master: Added support for Neutron loggings service plugin configuration https://review.opendev.org/c/openstack/tripleo-heat-templates/+/804223 | 13:40 |
| ykarel | hjensas, i shared with ratailor to try out similar to https://review.opendev.org/c/openstack/tripleo-quickstart/+/807703 until other fixes merged | 13:41 |
| hjensas | ykarel: thanks! | 13:41 |
| opendevreview | Marios Andreou proposed openstack/tripleo-heat-templates stable/train: DNM testing for tripleo-quickstart/+/809670 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809687 | 13:41 |
| marios | rlandy|rover: ^ | 13:42 |
| marios | rlandy|rover: oops forgot depends ;) | 13:42 |
| opendevreview | Marios Andreou proposed openstack/tripleo-heat-templates stable/train: DNM testing for tripleo-quickstart/+/809670 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809687 | 13:42 |
| rlandy|rover | marios: thanks - watching ^^ | 13:48 |
| marios | rlandy|rover: thanks | 13:48 |
| ade_lee | weshay|ruck, dviroel hey --- I was hoping you guys would be able to help me figure out the failing jobs on https://review.opendev.org/c/openstack/tripleo-ci/+/808215 | 14:08 |
| weshay|ruck | anbanerj|ruck, ^ | 14:09 |
| ade_lee | weshay|ruck, dviroel one of the failures has few logs, the others seem to have failed some tempest tests but I have no idea which | 14:09 |
| dviroel | ade_lee: is this patch that has a manila test failing | 14:09 |
| dviroel | ? | 14:09 |
| dviroel | that you mention before | 14:09 |
| ade_lee | dviroel, it might -- I see there are some manila tests being run and some failures | 14:10 |
| ade_lee | just not sure if the failures are expected or not | 14:10 |
| ade_lee | is there a nice output somewhere that shows all the failing tests? | 14:10 |
| dviroel | ah ok, scenario004 is failing, but manila tests is passing, but swift is failing | 14:11 |
| gouthamr | there used to be a html render of the test results on these jobs | 14:12 |
| dviroel | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e46/808215/14/check/tripleo-ci-centos-8-scenario004-standalone/e46581f/logs/undercloud/var/log/tempest/stestr_results.html | 14:12 |
| dviroel | scenario004 failures ^ | 14:12 |
| gouthamr | dviroel++ | 14:12 |
| gouthamr | thank you, i was looking under home/zuul | 14:12 |
| ade_lee | ah great -- I was wondering where those results were | 14:13 |
| opendevreview | Alex Schultz proposed openstack/puppet-tripleo master: Remove mistral https://review.opendev.org/c/openstack/puppet-tripleo/+/809466 | 14:13 |
| opendevreview | Alex Schultz proposed openstack/puppet-tripleo master: Remove zaqar https://review.opendev.org/c/openstack/puppet-tripleo/+/809467 | 14:14 |
| gouthamr | o/ ade_lee: we (manila folks) are just hearing about FIPS testing and starting to pay attention - like dviroel, i work on manila; and i wanted to get in touch with you about this stuff | 14:14 |
| gouthamr | ade_lee: is this the failure you called out recently, or is there another one? | 14:15 |
| gouthamr | cc: vhari | 14:15 |
| ade_lee | gouthamr, this is where I thought there might be maila failures - but it turns out (fortunately) that I was wrong. | 14:16 |
| ade_lee | gouthamr, that said, though, I spoke with vhari about setting up a gate job to test fips within manila | 14:17 |
| ade_lee | similar to the other projects | 14:17 |
| ade_lee | because we may not catch all the failures here | 14:17 |
| gouthamr | ade_lee++ yes, totally - thanks! we just run one sanity test in scenario004 | 14:18 |
| gouthamr | ade_lee: i'm interested in the upstream testing you've been doing with the integrated gate elsewhere so we can identify the compliance stuff earlier in the chain | 14:19 |
| ade_lee | gouthamr, vhari said he was going to start looking into it and get back to me with any questions. I would ask that when you create a review with fips tests that you use the topic add_fips_job so we can track it | 14:19 |
| ade_lee | gouthamr, vhari https://review.opendev.org/q/topic:%22add_fips_job%22+(status:open%20OR%20status:merged) | 14:20 |
| gouthamr | ade_lee: ack! thanks for sharing you'll have some more representation at the office hours this week, and possibly questions | 14:20 |
| ade_lee | excellent :) | 14:21 |
| gouthamr | ashrodri carloss ^ | 14:21 |
| ade_lee | gouthamr, because of the stuff we've added in zuul-jobs etc. it should be quite easy to set up a zuul job to test fips | 14:21 |
| *** dviroel is now known as dviroel|mtg | 14:21 | |
| opendevreview | Ananya proposed openstack/tripleo-ci master: Overriding gearman server and port for tripleo-ci-centos-8-containers-multinode https://review.opendev.org/c/openstack/tripleo-ci/+/809739 | 14:21 |
| ade_lee | check the octavia, nova, barbican , glance .. | 14:22 |
| ade_lee | dviroel|mtg, looks like scenario 10 has octavia failures, any idea whats going on in scenario 2? | 14:24 |
| ashrodri | hello! Ill take a look at those tests as well, thanks for the info ade_lee | 14:24 |
| dviroel|mtg | scenario002 is tricky, no logs to help us there | 14:25 |
| ade_lee | dviroel|mtg, yeah - anything I can do to enable more logs? | 14:25 |
| * dviroel|mtg looking | 14:26 | |
| ade_lee | dviroel|mtg, presumably there are supposed to be logs, right? so something broke in creating them? | 14:26 |
| dviroel|mtg | ade_lee: might be, cause it failed on tempest https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_758/808215/13/check/tripleo-ci-centos-8-scenario002-standalone/758090b/logs/quickstart_install.log | 14:27 |
| carloss | i'll take a look too... thanks ade_lee! | 14:30 |
| ade_lee | dviroel|mtg, not sure if anything can be gleaned from https://zuul.opendev.org/t/openstack/build/8bfe1aff12a24c15818ea0fc7503c4ca/log/logs/quickstart_collect_logs.log | 14:34 |
| opendevreview | Lance Bragstad proposed openstack/tripleo-quickstart master: Update featureset 10 to deploy with secure RBAC's project personas https://review.opendev.org/c/openstack/tripleo-quickstart/+/809749 | 14:37 |
| *** dviroel|mtg is now known as dviroel | 14:39 | |
| opendevreview | Lance Bragstad proposed openstack/tripleo-quickstart master: Update featureset 10 to deploy with secure RBAC's project personas https://review.opendev.org/c/openstack/tripleo-quickstart/+/809749 | 14:42 |
| ade_lee | johnsom, hey - can you take a look at https://review.opendev.org/c/openstack/tripleo-ci/+/808215 scenario 10 . There appear to be octavia test failures | 14:43 |
| johnsom | ade_lee Sure, give me a minute | 14:44 |
| ade_lee | johnsom, thanks ! | 14:44 |
| johnsom | ade_lee looks like nova is throwing a 504 when we are asking it to attach a neutron port. I'm going to have to dig deeper on this one. | 14:51 |
| johnsom | nova client usually gives us more of an exception to go on... | 14:52 |
| ade_lee | johnsom, ack thanks - let me know if there is anything I can do to help | 14:53 |
| johnsom | ade_lee | 14:56 |
| johnsom | https://www.irccloud.com/pastebin/Ps4NYTDi/ | 14:56 |
| johnsom | So HAProxy shows there are no functioning nova API instances | 14:56 |
| johnsom | That is the 504 Octavia received and caused the LB create to fail | 14:57 |
| ade_lee | huh -- looking at nova logs .. | 14:58 |
| johnsom | Here is a direct link to the haproxy logs in case it is helpful: https://zuul.opendev.org/t/openstack/build/af88c0d00ebb48509a51f6eb941c65c6/log/logs/undercloud/var/log/containers/haproxy/haproxy.log#1276 | 14:58 |
| ade_lee | johnsom, you sure it just wasn't down temporarily during setup - cause from that haproxy log, it looks like calls were being made .. | 15:01 |
| *** ykarel is now known as ykarel|away | 15:02 | |
| johnsom | ade_lee Line three in my snippet is the call from Octavia being rejected by haproxy. I would assume it wouldn't try to create an LB until nova was finished installing. | 15:03 |
| johnsom | ade_lee But I don't know that test scenario well | 15:03 |
| johnsom | The haproxy log just says the nova api endpoints are failing the health checks. | 15:04 |
| johnsom | ade_lee https://zuul.opendev.org/t/openstack/build/af88c0d00ebb48509a51f6eb941c65c6/log/logs/undercloud/var/log/containers/nova/nova-api.log#2445 | 15:05 |
| johnsom | There appears to be an exception in the nova api log around that time | 15:05 |
| ade_lee | yeah - the test in that scenario seems to start at 21:25:37 and end at 21:36:29, and yeah, haproxy seems to report things being up again at 21:36:17 | 15:08 |
| ade_lee | but yeah - things are definitely awry in nova | 15:08 |
| ade_lee | johnsom, thanks - following up with nova folks | 15:23 |
| johnsom | +1 | 15:23 |
| rlandy|rover | ade_lee: dviroel: correct - scneraio002 has no log as failure in collect_logs: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_758/808215/13/check/tripleo-ci-centos-8-scenario002-standalone/758090b/logs/quickstart_collect_logs.log | 15:36 |
| ade_lee | rlandy|rover, unfortunately - that log doesn't tell me why it failed :/ | 15:37 |
| rlandy|rover | ade_lee: looking at the job | 15:38 |
| *** ysandeep is now known as ysandeep|mtg | 15:42 | |
| rlandy|rover | ade_lee: so it looks like only test that is not collecting logs is scenraio002 | 15:44 |
| rlandy|rover | you have logs on the rets | 15:44 |
| rlandy|rover | rest | 15:44 |
| rlandy|rover | and I see this happened at least twice | 15:45 |
| rlandy|rover | and scneraio002 has logs on that changes | 15:46 |
| rlandy|rover | so we have a legit issue | 15:46 |
| ade_lee | rlandy|rover, yup -- I'm doing a recheck to see if the same errors persist on the others, but scenario 2 does seem to be consistently not providing logs | 15:46 |
| rlandy|rover | ade_lee: yep - on the last three runs - no logs | 15:46 |
| * rlandy|rover looks at error | 15:46 | |
| rlandy|rover | ade_lee: so here are the scenario differencesL | 15:51 |
| rlandy|rover | https://github.com/openstack/tripleo-heat-templates#service-testing-matrix | 15:51 |
| ade_lee | rlandy|rover, interesting - so what makes this different in terms of the script that generates logs? | 15:54 |
| rlandy|rover | nothing | 15:54 |
| rlandy|rover | it's just the difference of the services that each scenario would use | 15:55 |
| rlandy|rover | swift in this case | 15:55 |
| ade_lee | rlandy|rover, right -- it looks like the log collection script is eventually timing out while collecting the container logs? | 15:57 |
| rlandy|rover | right | 15:57 |
| rlandy|rover | so we have two options | 15:57 |
| rlandy|rover | DNM patch to avoid collecting container logs | 15:58 |
| rlandy|rover | and see if it can carry on | 15:58 |
| rlandy|rover | or | 15:58 |
| ade_lee | but its going to 127.0.0.2 -- so like localhost? | 15:58 |
| rlandy|rover | try run this on rdo and hold the node and see what is going on there | 15:58 |
| ade_lee | rlandy|rover, I like option 2 -- I suspect something is broken and without collecting logs , we wont know what | 15:58 |
| ade_lee | in fact, we know something went wrong in tempest test | 15:59 |
| rlandy|rover | yes | 16:00 |
| rlandy|rover | which could be related | 16:00 |
| ade_lee | plus we need to figure out why the log script is broken | 16:00 |
| ade_lee | yup | 16:00 |
| rlandy|rover | let's see if we can kick scenario002 in rdo | 16:00 |
| rlandy|rover | with a depends-on your patch | 16:00 |
| ade_lee | rlandy|rover, cool thanks! | 16:01 |
| rlandy|rover | https://github.com/openstack/ansible-role-collect-logs/search?q=containers | 16:01 |
| rlandy|rover | ^^ offending code | 16:01 |
| * rlandy|rover tries | 16:02 | |
| *** marios is now known as marios|out | 16:11 | |
| rlandy|rover | ade_lee: running https://review.rdoproject.org/r/c/testproject/+/35121 with tripleo-ci-centos-8-scenario002-standalone - will hold node | 16:15 |
| ade_lee | rlandy|rover, thanks! | 16:16 |
| rlandy|rover | let's see if it reproduces the same error | 16:16 |
| opendevreview | Lance Bragstad proposed openstack/tripleo-quickstart master: Update featureset 20 to use secure RBAC project personas https://review.opendev.org/c/openstack/tripleo-quickstart/+/809811 | 16:17 |
| dviroel | ysandeep|mtg: fyi ^ | 16:27 |
| opendevreview | Merged openstack/tripleo-heat-templates master: Revert "Configure ironic to do UEFI boot by default" https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809311 | 16:34 |
| opendevreview | Merged openstack/tripleo-common stable/wallaby: Remove /etc/rabbitmq/rabbitmq.conf https://review.opendev.org/c/openstack/tripleo-common/+/809498 | 16:34 |
| *** jpena is now known as jpena|off | 16:49 | |
| ykarel|away | mwhahaha, wrt fs035 CIX, is memcached part already considered? that job is working fine since switch to use ips for memcache | 16:50 |
| mwhahaha | ykarel|away: probably not, i'd recommend pointing that out in the card | 16:51 |
| ykarel|away | mwhahaha, ok i will comment | 16:51 |
| mwhahaha | what's likely happening was memcache failures were silently adding 10-20 seconds | 16:51 |
| mwhahaha | which blcoks | 16:51 |
| mwhahaha | because pymemcache is terrible | 16:51 |
| ykarel|away | yes | 16:51 |
| lbragstad | dviroel typo'd https://zuul.opendev.org/t/openstack/build/bfd4075e8baa4a9cb2dd56c7ad7a0bd4/log/logs/undercloud/home/zuul/overcloud_deploy.log | 16:54 |
| lbragstad | respinning that patch | 16:54 |
| *** ysandeep|mtg is now known as ysandeep | 16:54 | |
| dviroel | lbragstad: damn :P ok, tks | 16:54 |
| ysandeep | dviroel: thanks | 16:54 |
| *** ysandeep is now known as ysandeep|dinner | 16:54 | |
| *** amoralej is now known as amoralej|off | 16:56 | |
| opendevreview | Lance Bragstad proposed openstack/tripleo-quickstart master: Update featureset 10 to deploy with secure RBAC's project personas https://review.opendev.org/c/openstack/tripleo-quickstart/+/809749 | 17:01 |
| opendevreview | Lance Bragstad proposed openstack/tripleo-quickstart master: Update featureset 20 to use secure RBAC project personas https://review.opendev.org/c/openstack/tripleo-quickstart/+/809811 | 17:01 |
| opendevreview | Michele Baldessari proposed openstack/tripleo-heat-templates stable/wallaby: Simplify mysql users creation https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809310 | 17:01 |
| *** ysandeep|dinner is now known as ysandeep | 17:08 | |
| opendevreview | Merged openstack/tripleo-quickstart master: Fix quoting of task name {{ overcloud_image }} https://review.opendev.org/c/openstack/tripleo-quickstart/+/809484 | 17:29 |
| *** rlandy|rover is now known as rlandy|rover|biab | 17:35 | |
| opendevreview | Douglas Viroel proposed openstack/tripleo-quickstart master: WIP - Replace generic repo setup with yum_config module https://review.opendev.org/c/openstack/tripleo-quickstart/+/808408 | 17:55 |
| *** ysandeep is now known as ysandeep|away | 18:03 | |
| *** rlandy|rover|biab is now known as rlandy|rover | 18:13 | |
| opendevreview | Merged openstack/tripleo-quickstart-extras master: Remove --standalone argument from upgrade script https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/809416 | 18:29 |
| rlandy|rover | ade_lee: hmmm https://logserver.rdoproject.org/21/35121/21/check/periodic-tripleo-ci-centos-8-scenario002-standalone-master/281cd77/logs/undercloud/home/zuul/dlrn.log.txt.gz failing build-test packages for tempest | 19:29 |
| rlandy|rover | might need to exclude that | 19:29 |
| rlandy|rover | or may explain error | 19:29 |
| opendevreview | Douglas Viroel proposed openstack/tripleo-repos master: Add module to generate CentOS Compose repos https://review.opendev.org/c/openstack/tripleo-repos/+/804306 | 20:20 |
| opendevreview | Douglas Viroel proposed openstack/tripleo-repos master: Add module to generate CentOS Compose repos https://review.opendev.org/c/openstack/tripleo-repos/+/804306 | 20:27 |
| *** dviroel is now known as dviroel|out | 20:31 | |
| ade_lee | rlandy|rover, hmm interesting | 20:36 |
| ade_lee | rlandy|rover, so there is a tempest patch in there. we need it to make sure we're using ecc keys | 20:37 |
| ade_lee | rlandy|rover, but this build works in the other builds .. | 20:38 |
| ade_lee | rlandy|rover, so somthing is different on rdo .. | 20:38 |
| ade_lee | rlandy|rover, ah .. I see it | 20:40 |
| ade_lee | rlandy|rover, https://logserver.rdoproject.org/21/35121/21/check/periodic-tripleo-ci-centos-8-scenario002-standalone-master/281cd77/logs/undercloud/home/zuul/DLRN/data/repos/component/tempest/db/16/db16109411e1088c35f566c77bcd115c294a635a_dev/build.log.txt.gz | 20:40 |
| rlandy|rover | ade_lee: sorry - just got off meeting | 20:40 |
| rlandy|rover | ade_lee: yeah - could be something in rdo build-test packages | 20:41 |
| ade_lee | so yeah -- in the upstream gates we disable fips for the jobs where we build the packages | 20:41 |
| rlandy|rover | ade_lee: ah | 20:41 |
| ade_lee | this is because there is a problem with the sphinx version | 20:41 |
| ade_lee | rlandy|rover, so in rdo, is the build step separate like upstream? | 20:42 |
| rlandy|rover | ade_lee: if I removed that tempest patch from the testproject | 20:42 |
| rlandy|rover | would that impact | 20:42 |
| ade_lee | then you'd run into other tempest failures | 20:42 |
| rlandy|rover | I cam set build-test packages to false | 20:42 |
| ade_lee | but then you won't get the tempest change | 20:43 |
| ade_lee | and you'll run into other tempest failures | 20:43 |
| ade_lee | rlandy|rover, https://review.opendev.org/c/openstack/tripleo-ci/+/808215/14/zuul.d/content-provider.yaml | 20:43 |
| rlandy|rover | can we copy the rpm from somewhere? | 20:43 |
| rlandy|rover | and not build it in the job? | 20:43 |
| ade_lee | thats where we set fips to false in the content-provider jobs | 20:44 |
| ade_lee | so when we run upstream, we don't enable fips on the contentprovider jobs ie. the builds | 20:44 |
| ade_lee | is there a similar job on rdo? | 20:44 |
| rlandy|rover | yeah | 20:45 |
| rlandy|rover | we do that for rhos-release in the downstream | 20:45 |
| * rlandy|rover gets | 20:45 | |
| ade_lee | rlandy|rover, ok - then we just need to set the same variable | 20:46 |
| rlandy|rover | ade_lee: k- this is going to need a little work | 20:46 |
| ade_lee | an we should be good to go | 20:46 |
| rlandy|rover | ok if we pick this up on monday? | 20:46 |
| rlandy|rover | and we can build the dependent job | 20:46 |
| ade_lee | rlandy|rover, no worries - I have to head out anywauys -- | 20:46 |
| ade_lee | soccer practice for kid | 20:46 |
| ade_lee | have a good weekend! | 20:47 |
| rlandy|rover | ade_lee: example: http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/zuul.d/rhos-release.yaml | 20:47 |
| ade_lee | we can pick up on monday | 20:47 |
| rlandy|rover | have a good weekend | 20:47 |
| lbragstad | rlandy|rover dviroel|out https://review.opendev.org/c/openstack/tripleo-quickstart/+/809749/3 resulted in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/809749/3/check/tripleo-ci-centos-8-containers-multinode/d3b70ce/logs/subnode-1/var/lib/config-data/puppet-generated/keystone/etc/keystone/policy.yaml | 21:09 |
| lbragstad | looks like the overrides worked | 21:09 |
| dviroel|out | ++ | 21:10 |
| rlandy|rover | lbragstad: good start | 21:10 |
| lbragstad | looking for the tempest logs for that run now | 21:10 |
| rlandy|rover | lbragstad: you got a zuul +1 | 21:11 |
| rlandy|rover | look at job tripleo-ci-centos-8-containers-multinode | 21:11 |
| rlandy|rover | 2021-09-17 18:32:09.663931 | primary | + export TOCI_JOBTYPE=multinode-1ctlr-featureset010 | 21:12 |
| rlandy|rover | lbragstad: ttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/809749/3/check/tripleo-ci-centos-8-containers-multinode/d3b70ce/logs/undercloud/var/log/tempest/stestr_results.html | 21:12 |
| rlandy|rover | not a lot of tests | 21:12 |
| rlandy|rover | you can enable more if need be | 21:12 |
| lbragstad | nice | 21:12 |
| lbragstad | ok - cool, maybe next week we can try stacking another change on there that includes more tests | 21:14 |
| rlandy|rover | lbragstad: ack - let's pick this up on monday | 21:32 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!