openstackgerrit | Steve Baker proposed openstack/tripleo-quickstart master: DNM switch off nova/glance in featureset001 https://review.opendev.org/711364 | 00:04 |
---|---|---|
*** sgw has quit IRC | 00:09 | |
openstackgerrit | Merged openstack/tripleo-heat-templates master: nova-compute-container: add missing condition for ksmdisabled https://review.opendev.org/711007 | 00:11 |
openstackgerrit | Merged openstack/puppet-tripleo master: Revert "Add replication_probe_interval for ovsdbs" https://review.opendev.org/711201 | 00:11 |
openstackgerrit | Merged openstack/tripleo-heat-templates master: Use command to set pythonintepreter https://review.opendev.org/710980 | 00:11 |
*** slaweq has joined #tripleo | 00:11 | |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: deployment: set DeployIdentifier to '' if --skip-deploy-identifier is used https://review.opendev.org/711366 | 00:13 |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/train: nova-compute-container: add missing condition for ksmdisabled https://review.opendev.org/711368 | 00:14 |
openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: overcloud: set DeployIdentifier to '' if --skip-deploy-identifier is used https://review.opendev.org/711369 | 00:15 |
*** Tengu has quit IRC | 00:20 | |
*** Tengu has joined #tripleo | 00:22 | |
*** mcornea has quit IRC | 00:26 | |
*** slaweq has quit IRC | 00:27 | |
*** sgw has joined #tripleo | 00:28 | |
EmilienM | sshnaidm|afk, weshay|ruck: have we seen errors on scenario001-centos8? | 00:33 |
EmilienM | rlandy|bbl: ^ | 00:33 |
EmilienM | https://c6bc260fb8a50c93c7b6-115c1089095738e3e088969e8724f0ca.ssl.cf2.rackcdn.com/711230/1/check/tripleo-ci-centos-8-scenario001-standalone/2ad61f9/logs/undercloud/var/log/containers/stdouts/gnocchi_db_sync.log | 00:33 |
EmilienM | cradox.ObjectNotFound: [errno 2] error calling conf_read_file | 00:33 |
weshay|ruck | EmilienM, yes.. we need a promotion | 00:33 |
weshay|ruck | hrm.. not sure if thats the same error we saw earlier | 00:33 |
EmilienM | weshay|ruck: you found the same error? just to make sure we don't miss another issue | 00:33 |
* weshay|ruck checks | 00:33 | |
weshay|ruck | ya.. sec | 00:33 |
weshay|ruck | checking | 00:34 |
weshay|ruck | EmilienM, different | 00:34 |
weshay|ruck | also busted due to https://review.opendev.org/#/c/709701/ | 00:34 |
EmilienM | it's in the context of https://review.opendev.org/#/c/704443/ | 00:34 |
EmilienM | but I don't think it's related to it ^ | 00:34 |
EmilienM | gnocchi is configured by puppet | 00:34 |
weshay|ruck | ya.. a puppet change could have busted it up | 00:35 |
weshay|ruck | EmilienM, https://review.opendev.org/#/c/711320/ | 00:35 |
EmilienM | weshay|ruck: why don't we replace | 00:36 |
EmilienM | instead of adding | 00:36 |
weshay|ruck | EmilienM, we're not adding | 00:36 |
weshay|ruck | look at the branch excludes | 00:36 |
EmilienM | ah | 00:37 |
EmilienM | right | 00:37 |
weshay|ruck | :) | 00:37 |
EmilienM | also | 00:37 |
EmilienM | getting this on centos8: | 00:37 |
EmilienM | Error: error checking path "/run/libvirt": stat /run/libvirt: no such file or directory | 00:37 |
weshay|ruck | where? | 00:37 |
EmilienM | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_688/711230/1/check/tripleo-ci-centos-7-scenario012-standalone/6882c20/logs/undercloud/home/zuul/standalone_deploy.log | 00:38 |
weshay|ruck | EmilienM, btw.. I need to explain this to you at some point http://dashboard-ci.tripleo.org/d/UDA4H3aZk/component-pipeline?orgId=1 | 00:38 |
weshay|ruck | will keep bugs off df's queue | 00:38 |
EmilienM | I need to eat | 00:38 |
EmilienM | or she'll kill me | 00:38 |
EmilienM | :/ | 00:38 |
weshay|ruck | bu by | 00:39 |
EmilienM | I'll come back eventually | 00:39 |
EmilienM | baie | 00:39 |
EmilienM | I have 10 tabs of issues | 00:39 |
EmilienM | you can all find them in https://review.opendev.org/#/c/711230/ | 00:39 |
EmilienM | (check zuul.o.o) | 00:39 |
weshay|ruck | scen012 is f..'d | 00:39 |
weshay|ruck | has been | 00:39 |
EmilienM | k but still | 00:39 |
EmilienM | this is libvirt | 00:39 |
EmilienM | anyway | 00:39 |
EmilienM | aurevoir | 00:39 |
weshay|ruck | bu by | 00:39 |
weshay|ruck | mf.. didn't even vote | 00:40 |
*** dsneddon has quit IRC | 00:40 | |
*** dsneddon has joined #tripleo | 00:45 | |
*** jmasud has joined #tripleo | 00:47 | |
*** jmasud has quit IRC | 00:54 | |
*** jmasud has joined #tripleo | 00:56 | |
*** slaweq has joined #tripleo | 01:11 | |
*** slaweq has quit IRC | 01:16 | |
*** lbragstad_ has joined #tripleo | 01:34 | |
*** lbragstad has quit IRC | 01:36 | |
*** nhicher has quit IRC | 01:38 | |
*** nhicher has joined #tripleo | 01:38 | |
*** tkajinam has quit IRC | 01:49 | |
*** tkajinam has joined #tripleo | 01:50 | |
openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates master: Do not install client packages in overcloud on octavia deploy https://review.opendev.org/688958 | 01:55 |
*** fultonj has joined #tripleo | 01:58 | |
*** avivgt has quit IRC | 02:02 | |
*** Goneri has quit IRC | 02:24 | |
openstackgerrit | Merged openstack/tripleo-ci master: Handle cloudsig releases https://review.opendev.org/710910 | 02:31 |
openstackgerrit | Merged openstack/python-tripleoclient master: Remove the mistral from the check_deprecated_parameters workflow https://review.opendev.org/709141 | 02:31 |
*** sanjayu_ has joined #tripleo | 02:42 | |
openstackgerrit | Merged openstack/python-tripleoclient master: Remove mistral when creating the overcloudrc file. https://review.opendev.org/694138 | 02:44 |
openstackgerrit | Merged openstack/paunch master: Update zuul layout to reflect recent changes in TripleO https://review.opendev.org/710824 | 02:44 |
openstackgerrit | Merged openstack/ansible-role-collect-logs master: Add pattern for image prepare error https://review.opendev.org/711341 | 02:44 |
openstackgerrit | Merged openstack/tripleo-ci master: Use centos8 built images https://review.opendev.org/711280 | 02:44 |
*** psachin has joined #tripleo | 02:51 | |
*** sanjayu_ has quit IRC | 02:53 | |
*** raukadah is now known as chandankumar | 03:02 | |
*** lbragstad_ has quit IRC | 03:09 | |
*** slaweq has joined #tripleo | 03:11 | |
*** slaweq has quit IRC | 03:16 | |
openstackgerrit | Merged openstack/tripleo-ansible master: Fix typo in logs https://review.opendev.org/711335 | 03:21 |
openstackgerrit | Merged openstack/tripleo-ci master: Add c8 multinode containers job https://review.opendev.org/710179 | 03:46 |
*** dsneddon has quit IRC | 03:51 | |
*** apetrich has quit IRC | 04:00 | |
*** udesale has joined #tripleo | 04:09 | |
*** slaweq has joined #tripleo | 04:11 | |
*** slaweq has quit IRC | 04:16 | |
openstackgerrit | Merged openstack/tripleo-heat-templates master: Cleanup tasks for container-puppet.py https://review.opendev.org/710506 | 04:16 |
*** social has quit IRC | 04:20 | |
*** rlandy|bbl is now known as rlandy | 04:29 | |
*** skramaja has joined #tripleo | 04:29 | |
openstackgerrit | Ronelle Landy proposed openstack/tripleo-ci master: Add centos-8 scenario010 standalone jobs. https://review.opendev.org/711287 | 04:32 |
openstackgerrit | Merged openstack/tripleo-heat-templates master: Fixed package names for CentOS-8 octavia deployment https://review.opendev.org/710913 | 04:40 |
*** jdandrea has joined #tripleo | 04:41 | |
openstackgerrit | wes hayutin proposed openstack/tripleo-ci master: DNM: Add centos-8 scenario000-multinode update job https://review.opendev.org/710876 | 04:45 |
*** rlandy has quit IRC | 04:53 | |
*** ykarel|away is now known as ykarel | 04:55 | |
*** ade_lee has quit IRC | 04:58 | |
*** ade_lee has joined #tripleo | 04:59 | |
*** slaweq has joined #tripleo | 05:11 | |
*** udesale has quit IRC | 05:31 | |
*** evrardjp has quit IRC | 05:35 | |
*** evrardjp has joined #tripleo | 05:35 | |
*** slaweq has quit IRC | 05:38 | |
*** udesale has joined #tripleo | 05:39 | |
*** jdandrea has quit IRC | 05:40 | |
openstackgerrit | Merged openstack/tripleo-ci master: Improve build images role for CentOS8 https://review.opendev.org/708599 | 05:54 |
*** mburned is now known as mburned_out | 06:03 | |
*** mburned_out is now known as mburned | 06:03 | |
*** dsneddon has joined #tripleo | 06:12 | |
*** sri_ has joined #tripleo | 06:21 | |
*** mburned is now known as mburned_out | 06:27 | |
*** mburned_out is now known as mburned | 06:27 | |
*** marios has joined #tripleo | 06:28 | |
*** bandini has joined #tripleo | 06:35 | |
*** weshay|ruck has quit IRC | 06:36 | |
*** saneax has joined #tripleo | 06:36 | |
*** weshay has joined #tripleo | 06:36 | |
*** weshay has quit IRC | 06:40 | |
*** weshay has joined #tripleo | 06:40 | |
*** otherwiseguy has quit IRC | 06:41 | |
*** rnoriega_ has quit IRC | 06:41 | |
*** otherwiseguy has joined #tripleo | 06:42 | |
*** waleedm has joined #tripleo | 06:45 | |
*** rnoriega_ has joined #tripleo | 06:46 | |
*** threestrands has quit IRC | 06:48 | |
*** jfrancoa has joined #tripleo | 06:52 | |
openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Add NeutronDhcpAgentDnsmasqEnableAddr6List param https://review.opendev.org/710342 | 06:55 |
openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: Add the certificate specs in ceph_mgr service https://review.opendev.org/702846 | 06:55 |
*** dwilde has quit IRC | 06:56 | |
*** dsneddon has quit IRC | 07:01 | |
*** ccamacho has quit IRC | 07:02 | |
*** abdysn has joined #tripleo | 07:10 | |
*** weshay has quit IRC | 07:13 | |
*** weshay_ has joined #tripleo | 07:16 | |
*** jcapitao_off has joined #tripleo | 07:20 | |
openstackgerrit | yatin proposed openstack/ansible-role-tripleo-modify-image master: Remove unavailable repos from update https://review.opendev.org/710738 | 07:20 |
openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-ci master: Adds tripleo-ci-centos-8-containers-undercloud-minion https://review.opendev.org/710961 | 07:20 |
*** jcapitao_off is now known as jcapitao | 07:22 | |
openstackgerrit | Rabi Mishra proposed openstack/python-tripleoclient master: Skip network check with stack env when validations disabled https://review.opendev.org/711392 | 07:22 |
*** pcaruana has joined #tripleo | 07:24 | |
*** jbadiapa has joined #tripleo | 07:24 | |
ykarel | chandankumar, looks like https://review.opendev.org/#/c/709141/7/tripleoclient/workflows/parameters.py@130 has caused the multinode failures | 07:27 |
ykarel | merged few hours back, i noticed failure in c8 job upstream too | 07:27 |
*** hjensas has quit IRC | 07:28 | |
ykarel | looks like something else is wrong, as that is list, but error says dict | 07:33 |
ykarel | me seeing https://2714a4271cb20642317a-491b37cd909c997f4e833a309b30267e.ssl.cf1.rackcdn.com/711287/4/check/tripleo-ci-centos-8-containers-multinode/6a2bec0/logs/undercloud/home/zuul/overcloud_deploy.log | 07:34 |
*** otherwiseguy has quit IRC | 07:34 | |
openstackgerrit | Francesco Pantano proposed openstack/tripleo-ansible master: Add a new set of tasks to build the openstack pool and key list https://review.opendev.org/709083 | 07:34 |
*** weshay_ has quit IRC | 07:34 | |
*** weshay_ has joined #tripleo | 07:34 | |
*** otherwiseguy has joined #tripleo | 07:35 | |
*** ykarel is now known as ykarel|lunch | 07:35 | |
*** dmacpher__ has quit IRC | 07:36 | |
*** lhinds has quit IRC | 07:38 | |
*** rnoriega_ has quit IRC | 07:39 | |
*** mburned has quit IRC | 07:39 | |
*** weshay_ has quit IRC | 07:39 | |
*** haleyb|away has quit IRC | 07:39 | |
*** otherwiseguy has quit IRC | 07:40 | |
*** abdysn has quit IRC | 07:50 | |
*** waleedm has quit IRC | 07:50 | |
*** waleedm has joined #tripleo | 07:51 | |
*** abdysn has joined #tripleo | 07:51 | |
*** tesseract has joined #tripleo | 07:52 | |
*** tremble has joined #tripleo | 07:53 | |
*** rnoriega_ has joined #tripleo | 07:55 | |
*** otherwiseguy has joined #tripleo | 07:56 | |
*** weshay_ has joined #tripleo | 07:56 | |
*** iurygregory has joined #tripleo | 07:56 | |
*** lhinds has joined #tripleo | 07:58 | |
*** cylopez has joined #tripleo | 08:02 | |
openstackgerrit | Merged openstack/tripleo-common stable/train: Fix memcached healthcheck regression https://review.opendev.org/711340 | 08:03 |
openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Remove unnecessary yum calls https://review.opendev.org/710982 | 08:03 |
*** hjensas has joined #tripleo | 08:08 | |
*** ccamacho has joined #tripleo | 08:11 | |
*** tkajinam has quit IRC | 08:16 | |
*** weshay_ has quit IRC | 08:24 | |
*** rnoriega_ has quit IRC | 08:25 | |
*** weshay has joined #tripleo | 08:25 | |
*** jpena|off is now known as jpena | 08:26 | |
*** rnoriega_ has joined #tripleo | 08:28 | |
*** saneax has quit IRC | 08:29 | |
*** saneax has joined #tripleo | 08:30 | |
*** ykarel|lunch is now known as ykarel | 08:32 | |
*** rainleander has joined #tripleo | 08:32 | |
jbadiapa | good morning! | 08:33 |
*** leanderthal has quit IRC | 08:35 | |
*** amoralej|off is now known as amoralej | 08:45 | |
ykarel | chandankumar, commented on the patch wrt ^^ issue | 08:45 |
chandankumar | ykarel, looking | 08:45 |
*** rpittau|afk is now known as rpittau | 08:46 | |
*** bogdando has joined #tripleo | 08:46 | |
*** marios has quit IRC | 08:56 | |
*** pojadhav is now known as pojadhav|lunch | 09:01 | |
*** jpich has joined #tripleo | 09:01 | |
*** pojadhav|lunch is now known as pojadhav | 09:01 | |
*** lucasagomes has joined #tripleo | 09:02 | |
*** ade_lee has quit IRC | 09:07 | |
*** sai_p___ has quit IRC | 09:07 | |
*** mwhahaha has quit IRC | 09:07 | |
*** mmagr has quit IRC | 09:08 | |
*** abishop has quit IRC | 09:08 | |
*** stevebaker has quit IRC | 09:08 | |
*** mrunge has quit IRC | 09:08 | |
*** StevenK has quit IRC | 09:08 | |
openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: WIP enforce firewall rules before pacemaker-auth https://review.opendev.org/711405 | 09:17 |
*** cylopez has quit IRC | 09:18 | |
*** cylopez has joined #tripleo | 09:18 | |
*** apetrich has joined #tripleo | 09:24 | |
*** gfidente|afk is now known as gfidente | 09:25 | |
*** tosky has joined #tripleo | 09:25 | |
*** suuuper has joined #tripleo | 09:28 | |
*** holser has joined #tripleo | 09:28 | |
openstackgerrit | yatin proposed openstack/tripleo-quickstart master: Fix convert image package list https://review.opendev.org/711408 | 09:29 |
ykarel | chandankumar, weshay issue caused by last patch ^^ | 09:29 |
ykarel | with https://review.opendev.org/#/c/711078/1 | 09:30 |
chandankumar | nit pick above ^^ | 09:34 |
ykarel | looking | 09:34 |
ykarel | chandankumar, any specific advantage of using that | 09:35 |
ykarel | is version thing | 09:35 |
chandankumar | good to use version compare | 09:35 |
chandankumar | https://docs.ansible.com/ansible/latest/user_guide/playbooks_tests.html#version-comparison | 09:36 |
*** sshnaidm|afk is now known as sshnaidm | 09:37 | |
*** alexmcleod has joined #tripleo | 09:37 | |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DNM: fail before introspection https://review.opendev.org/695975 | 09:39 |
openstackgerrit | yatin proposed openstack/tripleo-quickstart master: Fix convert image package list https://review.opendev.org/711408 | 09:41 |
*** ade_lee has joined #tripleo | 09:43 | |
*** sai_p___ has joined #tripleo | 09:43 | |
*** mwhahaha has joined #tripleo | 09:43 | |
*** mmagr has joined #tripleo | 09:43 | |
*** abishop has joined #tripleo | 09:43 | |
*** stevebaker has joined #tripleo | 09:43 | |
*** mrunge has joined #tripleo | 09:43 | |
*** StevenK has joined #tripleo | 09:43 | |
*** orwell.freenode.net sets mode: +ov mwhahaha mwhahaha | 09:43 | |
sshnaidm | cores, please review https://review.opendev.org/#/c/711253/ fix for flaky test | 09:43 |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ansible master: WIP: try podman driver https://review.opendev.org/708851 | 09:44 |
*** openstackstatus has quit IRC | 09:45 | |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DNM: clean nodes on OVB https://review.opendev.org/709816 | 09:46 |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: DNM: use vexxhost registry for promotion jobs https://review.opendev.org/710135 | 09:48 |
*** bogdando has quit IRC | 09:55 | |
*** bogdando has joined #tripleo | 09:56 | |
*** sri_ has quit IRC | 10:03 | |
openstackgerrit | Christophe Fontaine proposed openstack/tripleo-ansible master: 'tripleo_ovs_dpdk_lcore_list' should be optional https://review.opendev.org/711360 | 10:11 |
openstackgerrit | Christophe Fontaine proposed openstack/tripleo-ansible master: CI: mount /lib/modules https://review.opendev.org/711420 | 10:11 |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ansible master: WIP: try podman driver https://review.opendev.org/708851 | 10:13 |
*** derekh has joined #tripleo | 10:13 | |
openstackgerrit | Merged openstack/tripleo-ci master: add the upstream centos-8 undercloud job https://review.opendev.org/710856 | 10:16 |
*** marios has joined #tripleo | 10:17 | |
openstackgerrit | Juan Badia Payno proposed openstack/tripleo-ansible master: WIP: BaR backup /var/lib/ceph ceph mon and mgr https://review.opendev.org/711423 | 10:19 |
*** jcapitao has quit IRC | 10:20 | |
*** jcapitao has joined #tripleo | 10:21 | |
*** psachin has quit IRC | 10:23 | |
*** dtantsur|afk is now known as dtantsur | 10:24 | |
openstackgerrit | Merged openstack/tripleo-ci master: Add centos-8 scenario007 standalone jobs. https://review.opendev.org/711330 | 10:25 |
openstackgerrit | yatin proposed openstack/ansible-role-tripleo-modify-image master: Remove unavailable repos from update https://review.opendev.org/710738 | 10:28 |
*** hjensas has quit IRC | 10:31 | |
*** jcapitao has quit IRC | 10:32 | |
*** jcapitao has joined #tripleo | 10:33 | |
openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/paunch stable/stein: Cleanup containers in the same loop as they are created https://review.opendev.org/711426 | 10:37 |
*** alexmcleod has quit IRC | 10:43 | |
*** hjensas has joined #tripleo | 10:44 | |
*** udesale has quit IRC | 10:49 | |
*** alexmcleod has joined #tripleo | 10:50 | |
openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-quickstart-extras master: Add job.repos to container updates for component jobs https://review.opendev.org/711156 | 10:51 |
*** alexmcleod has quit IRC | 10:55 | |
*** alexmcleod has joined #tripleo | 10:56 | |
*** cylopez has quit IRC | 10:59 | |
openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/paunch stable/queens: Cleanup containers in the same loop as they are created https://review.opendev.org/711432 | 10:59 |
*** jmasud has quit IRC | 11:04 | |
*** jmasud has joined #tripleo | 11:04 | |
openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/paunch stable/queens: Cleanup containers in the same loop as they are created https://review.opendev.org/711432 | 11:05 |
openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-ci master: Add centos-8 scenario010 standalone jobs. https://review.opendev.org/711287 | 11:06 |
openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/paunch stable/stein: Cleanup containers in the same loop as they are created https://review.opendev.org/711426 | 11:07 |
openstackgerrit | Rabi Mishra proposed openstack/tripleo-common master: Move RotateFernetKeysAction functionality to utils https://review.opendev.org/711434 | 11:11 |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/queens: Enable objects versioning for Ceph RadosGW https://review.opendev.org/710202 | 11:14 |
openstackgerrit | Rabi Mishra proposed openstack/tripleo-ansible master: WIP Use only ansible for fernet key rotation https://review.opendev.org/711436 | 11:18 |
*** slaweq has joined #tripleo | 11:18 | |
openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-ci master: lint: enable black https://review.opendev.org/703980 | 11:19 |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ansible master: WIP: try podman driver https://review.opendev.org/708851 | 11:21 |
*** rpittau is now known as rpittau|bbl | 11:27 | |
*** chem has quit IRC | 11:35 | |
*** tesseract-RH has joined #tripleo | 11:40 | |
openstackgerrit | Francesco Pantano proposed openstack/tripleo-ansible master: Add a new set of tasks to build the openstack pool and key list https://review.opendev.org/709083 | 11:41 |
openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: Add CephBasePoolVars and CephKeyVars structures https://review.opendev.org/709297 | 11:42 |
*** tesseract has quit IRC | 11:43 | |
openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates stable/train: WIP: Move Stein upgrade tasks to ffwd tasks. https://review.opendev.org/706107 | 11:50 |
*** hjensas has quit IRC | 11:54 | |
*** jcapitao is now known as jcapitao_lunch | 11:54 | |
*** yprokule has joined #tripleo | 11:56 | |
*** marios is now known as marios|call | 12:01 | |
*** rlandy has joined #tripleo | 12:02 | |
*** amoralej is now known as amoralej|lunch | 12:06 | |
*** raildo has joined #tripleo | 12:08 | |
*** jpena is now known as jpena|lunch | 12:09 | |
*** dpawlik has quit IRC | 12:14 | |
*** cylopez has joined #tripleo | 12:15 | |
*** dpawlik has joined #tripleo | 12:15 | |
openstackgerrit | Luca Miccini proposed openstack/puppet-tripleo stable/queens: set x-forwarded-port header for proxied requests https://review.opendev.org/711442 | 12:16 |
*** slaweq has quit IRC | 12:18 | |
*** jcapitao_lunch has quit IRC | 12:19 | |
openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade master: Remove Congress service from roles_data. https://review.opendev.org/710242 | 12:20 |
*** jcapitao_lunch has joined #tripleo | 12:21 | |
*** pcaruana has quit IRC | 12:22 | |
*** hjensas has joined #tripleo | 12:22 | |
*** dpawlik has quit IRC | 12:22 | |
openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add always tags for hieradata render in external upgrade. https://review.opendev.org/710320 | 12:26 |
*** cylopez has quit IRC | 12:30 | |
gfidente | cloudnull I think I need help understanding what is going on with https://review.opendev.org/#/c/697990/ | 12:30 |
openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-ci master: Add centos-8 scenario010 standalone jobs. https://review.opendev.org/711287 | 12:30 |
gfidente | the skeleton_test fails, but not the role test itself | 12:30 |
gfidente | so I can't reproduce it locally | 12:31 |
gfidente | and the failure seems just a file not found :( | 12:31 |
gfidente | can you help understand what is happening there and why it fails? | 12:31 |
cgoncalves | gfidente, hey! you might be able to help me with a TripleO Ceph related question :) I have a patch that adds a new upgrade job where Octavia and Ceph are enabled. problem is the job times out in what appears to be while upgrading Ceph. would you have any hints? | 12:33 |
cgoncalves | the upgrade process appears to halt here: https://zuul.opendev.org/t/openstack/build/125e42ce57724b67b368a3f8f15f4c21/log/logs/undercloud/home/zuul/standalone_upgrade.log#8673 | 12:33 |
openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Enforce firewall rules before pacemaker-auth https://review.opendev.org/711405 | 12:34 |
*** yprokule has quit IRC | 12:34 | |
gfidente | cgoncalves hi | 12:34 |
*** pcaruana has joined #tripleo | 12:35 | |
gfidente | cgoncalves so you don't have the logs location, cause we have a bug in the task name | 12:35 |
gfidente | but I suspect this is just refusing to run because we have a single osd | 12:35 |
*** dpawlik has joined #tripleo | 12:36 | |
gfidente | let me find the logs | 12:36 |
cgoncalves | this? https://zuul.opendev.org/t/openstack/build/125e42ce57724b67b368a3f8f15f4c21/log/logs/undercloud/home/zuul/standalone-ansible-T9r6F_/ceph-ansible/ceph_ansible_command.log | 12:37 |
cgoncalves | FAILED - RETRYING: waiting for the monitor(s) to form the quorum... (10 retries left). | 12:37 |
gfidente | yeah that https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_125/710473/2/check/tripleo-ci-centos-7-scenario010-standalone-upgrade/125e42c/logs/undercloud/home/zuul/standalone-ansible-T9r6F_/ceph-ansible/ceph_ansible_command.log | 12:39 |
*** ansmith_ has quit IRC | 12:40 | |
*** jpich has quit IRC | 12:40 | |
cgoncalves | gfidente, thanks. so two questions: why does it fail on single osd only during upgrade? and what would be your suggestion to make the job work? | 12:41 |
*** jpich has joined #tripleo | 12:41 | |
gfidente | cgoncalves it should be running rolling_update though, not site-container | 12:41 |
gfidente | cgoncalves the command.sh says it was running site-container | 12:41 |
gfidente | is this the initial deployment or the actual upgrade attempt? | 12:41 |
cgoncalves | gfidente, upgrade | 12:42 |
*** cylopez has joined #tripleo | 12:42 | |
gfidente | so this is external-run --tags ceph ? | 12:42 |
cgoncalves | I'm not familiar with that site-container thingy | 12:42 |
gfidente | I mean at which step of the job is this failing> | 12:43 |
gfidente | so site-container playbook should work with a single mon/osd | 12:43 |
gfidente | rolling_update won't because you can't keep the cluster up and this is a requirement from ceph folks | 12:43 |
gfidente | so site-container itself must be failing for a different reason, what is the change? | 12:44 |
cgoncalves | gfidente, I think it fails at external deployment step 2 | 12:45 |
cgoncalves | https://zuul.opendev.org/t/openstack/build/125e42ce57724b67b368a3f8f15f4c21/log/logs/undercloud/home/zuul/standalone_upgrade.log#8199 | 12:45 |
cgoncalves | gfidente, this is the change: https://review.opendev.org/#/c/710473/ | 12:45 |
cgoncalves | the associated feature set is https://review.opendev.org/#/c/710556/1/config/general_config/featureset064.yml | 12:45 |
*** morazi has joined #tripleo | 12:46 | |
cgoncalves | and the associated scenario is https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario010-standalone.yaml | 12:46 |
gfidente | but that works already with ceph | 12:50 |
gfidente | in other jobs it seems | 12:50 |
gfidente | scenario010 I mean | 12:50 |
cgoncalves | correct. the problem manifests itself during standalone upgrade | 12:50 |
*** bfournie has quit IRC | 12:53 | |
gfidente | cgoncalves so looking at the ceph daemon logs | 12:54 |
gfidente | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_125/710473/2/check/tripleo-ci-centos-7-scenario010-standalone-upgrade/125e42c/logs/undercloud/var/log/extra/podman/containers/ | 12:54 |
gfidente | it did deploythe initial overcloud standalone | 12:54 |
openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Remove yum calls from images prepare script https://review.opendev.org/711273 | 12:54 |
cgoncalves | gfidente, +1. it even passed tempest tests | 12:56 |
openstackgerrit | Merged openstack/tripleo-upgrade stable/train: Change roles_data/update_serial without using shell https://review.opendev.org/711264 | 12:57 |
openstackgerrit | Merged openstack/tripleo-upgrade stable/rocky: Allow to switch from parallel to serial update. https://review.opendev.org/710724 | 12:57 |
*** mburned has joined #tripleo | 12:58 | |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ci master: DNM use tripleo-ci-testing images for c8 https://review.opendev.org/711451 | 12:59 |
openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates master: Do not install client packages in overcloud on octavia deploy https://review.opendev.org/688958 | 13:00 |
*** ahosam has joined #tripleo | 13:01 | |
*** rainleander is now known as leanderthal | 13:02 | |
*** hjensas has quit IRC | 13:07 | |
*** rh-jelabarre has joined #tripleo | 13:13 | |
*** udesale has joined #tripleo | 13:14 | |
*** jcapitao_lunch is now known as jcapitao | 13:15 | |
*** rh-jelabarre has quit IRC | 13:21 | |
*** rh-jelabarre has joined #tripleo | 13:21 | |
*** hjensas has joined #tripleo | 13:27 | |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-common stable/train: deployment: set DeployIdentifier to '' if --skip-deploy-identifier is used https://review.opendev.org/711454 | 13:29 |
*** ansmith_ has joined #tripleo | 13:36 | |
*** ansmith has joined #tripleo | 13:38 | |
*** avivgt has joined #tripleo | 13:38 | |
*** bfournie has joined #tripleo | 13:39 | |
*** ansmith_ has quit IRC | 13:40 | |
*** cgoncalves has quit IRC | 13:41 | |
*** mcornea has joined #tripleo | 13:41 | |
*** cgoncalves has joined #tripleo | 13:45 | |
*** rpittau|bbl is now known as rpittau | 13:45 | |
*** jdandrea has joined #tripleo | 13:46 | |
*** fmount has quit IRC | 13:47 | |
openstackgerrit | Juan Badia Payno proposed openstack/tripleo-ansible master: WIP:BaR Always do MySQL dumps even the services wont be stopped https://review.opendev.org/711458 | 13:47 |
*** fmount has joined #tripleo | 13:48 | |
*** avivgt has quit IRC | 13:52 | |
*** jcoufal has joined #tripleo | 13:54 | |
*** jcoufal has quit IRC | 13:54 | |
openstackgerrit | Merged openstack/tripleo-ci master: Adds tripleo-ci-centos-8-containers-undercloud-minion https://review.opendev.org/710961 | 13:55 |
*** jcoufal has joined #tripleo | 13:55 | |
openstackgerrit | Lukas Bezdicka proposed openstack/tripleo-heat-templates master: [Q->T] Introduce Queens to Train Upgrade https://review.opendev.org/711459 | 14:02 |
openstackgerrit | Christophe Fontaine proposed openstack/tripleo-ansible master: 'tripleo_ovs_dpdk_lcore_list' should be optional https://review.opendev.org/711360 | 14:02 |
*** jdandrea has quit IRC | 14:03 | |
*** avivgt has joined #tripleo | 14:03 | |
*** jdandrea_ has joined #tripleo | 14:04 | |
cloudnull | gfidente will take a look shortly | 14:09 |
gfidente | cloudnull thanks | 14:09 |
cloudnull | gfidente i actually have no idea what that test is doing, looks like its trying to exercise a role that doesn't exist | 14:12 |
*** lbragstad has joined #tripleo | 14:13 | |
cloudnull | and i dont see that in the tripleo-ansible zuul config ... | 14:13 |
cloudnull | weshay ^ is that something that was added to tripleo-ci ? | 14:14 |
cloudnull | https://zuul.opendev.org/t/openstack/build/99187eafaac24323b0b331dbb1900080 | 14:14 |
cloudnull | https://zuul.opendev.org/t/openstack/job/tripleo-ansible-centos-7-molecule-skeleton_test | 14:15 |
cloudnull | looks like that job is now gone ? | 14:15 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Add tripleo-lvmfilter role to restrict visible block devices for LVM https://review.opendev.org/697990 | 14:15 |
cloudnull | gfidente I rebased it just to see if it was something from an old run, sadly it looks like that test is back | 14:16 |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Remove all ignore_errors to avoid confusion when debugging https://review.opendev.org/711059 | 14:18 |
EmilienM | bogdando: hey, is it better you think? https://review.opendev.org/#/c/711059/3..4/deployment/undercloud/undercloud-upgrade.yaml | 14:18 |
bogdando | EmilienM: yeah, checking for rc should work better | 14:18 |
cloudnull | gfidente ah - https://review.opendev.org/#/c/697990/24/zuul.d/molecule.yaml@118 | 14:19 |
bogdando | EmilienM: but for that particular case, it should be ==0 | 14:19 |
cloudnull | gfidente is that something maybe you had in a local checkout ? | 14:19 |
bogdando | or != 0 for the failed check | 14:19 |
EmilienM | bogdando: oh right | 14:19 |
bogdando | EmilienM: for the idempotency Gods you can also add changed_when: false sometimes :) | 14:19 |
cloudnull | gfidente it looks like your lvmfilter change is adding the "skeleton_test" role to the zuul config | 14:20 |
bogdando | may be in followup | 14:20 |
EmilienM | bogdando: yeah | 14:20 |
*** avivgt has quit IRC | 14:20 | |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Remove all ignore_errors to avoid confusion when debugging https://review.opendev.org/711059 | 14:22 |
EmilienM | bogdando: I can't change the failed_when, or the task will fail if OVS upgrade isn't needed | 14:22 |
EmilienM | bogdando: however I can add changed_when, in a followup | 14:22 |
bogdando | sure, that works | 14:22 |
gfidente | cloudnull aaaaaah that's because I run the newrole playbook to create it | 14:23 |
openstackgerrit | Christophe Fontaine proposed openstack/tripleo-ansible master: 'tripleo_ovs_dpdk_lcore_list' should be optional https://review.opendev.org/711360 | 14:24 |
cloudnull | did the new role playbook generate that role test ? | 14:24 |
*** marios has joined #tripleo | 14:25 | |
cloudnull | or did you run that playbook with the option `role_name=skeleton_test`? | 14:25 |
*** marios|call has quit IRC | 14:25 | |
weshay | cloudnull, need a few | 14:25 |
cloudnull | weshay disregard I think we found the issue | 14:26 |
*** ykarel is now known as ykarel|away | 14:27 | |
*** ekultails has joined #tripleo | 14:29 | |
*** jpena|lunch is now known as jpena | 14:29 | |
*** amoralej|lunch is now known as amoralej | 14:31 | |
*** Goneri has joined #tripleo | 14:31 | |
openstackgerrit | Merged openstack/tripleo-upgrade master: Initital ffwd2 patch to automate OS upgrade. https://review.opendev.org/704905 | 14:32 |
openstackgerrit | Merged openstack/paunch master: Cleanup containers in the same loop as they are created https://review.opendev.org/707408 | 14:32 |
*** TrevorV has joined #tripleo | 14:32 | |
gfidente | cloudnull I don't think I ran it with name=skeleton_test no | 14:32 |
gfidente | cloudnull so shall I just go ahead and remove it from zuul in an update? | 14:33 |
*** chem has joined #tripleo | 14:33 | |
*** sgw has quit IRC | 14:33 | |
openstackgerrit | Lukas Bezdicka proposed openstack/tripleo-common master: [Q->T] Introduce Queens to Train Upgrade https://review.opendev.org/711468 | 14:33 |
*** haleyb has joined #tripleo | 14:36 | |
*** ahosam has quit IRC | 14:38 | |
*** cgoncalves has quit IRC | 14:39 | |
*** cgoncalves has joined #tripleo | 14:40 | |
gfidente | cloudnull I think I might have pulled it in rebasing one of the old patchsets then carried it forward into the new ones | 14:41 |
gfidente | cloudnull so I'll clear it no | 14:41 |
gfidente | *now | 14:41 |
*** jcapitao has quit IRC | 14:42 | |
*** waleedm has quit IRC | 14:42 | |
*** jcapitao has joined #tripleo | 14:44 | |
openstackgerrit | Giulio Fidente proposed openstack/tripleo-ansible master: Add tripleo-lvmfilter role to restrict visible block devices for LVM https://review.opendev.org/697990 | 14:45 |
openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: Adding molecule test for fetch-images role https://review.opendev.org/681211 | 14:48 |
*** sgw has joined #tripleo | 14:49 | |
*** haleyb is now known as haleyb|away | 14:50 | |
cloudnull | gfidente that last review looks good ! | 14:54 |
gfidente | the -modules job got moved around probably for the same reason | 14:54 |
gfidente | I can put it back to clear the gerrit change | 14:55 |
cloudnull | ++ the add role playbook will resort the jobs | 14:55 |
*** tremble has quit IRC | 14:55 | |
cloudnull | so you will see movement when its run, that's normal | 14:55 |
gfidente | I insisted on running the tests locally | 14:56 |
gfidente | didn't even figure it was a different module | 14:56 |
gfidente | skeleton_thingie | 14:56 |
*** abdysn has quit IRC | 14:57 | |
gfidente | thanks again | 14:57 |
cloudnull | anytime, sorry for the confusion | 14:57 |
openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: update tht standalone scenario jobs for centos-8 https://review.opendev.org/710811 | 14:59 |
openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/paunch stable/queens: Cleanup containers in the same loop as they are created https://review.opendev.org/711432 | 15:01 |
openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: update tht standalone scenario jobs for centos-8 https://review.opendev.org/710811 | 15:02 |
openstackgerrit | wes hayutin proposed openstack/tripleo-common master: replace centos-7 with centos-8 in scenario jobs https://review.opendev.org/710821 | 15:02 |
*** d0ugal has joined #tripleo | 15:07 | |
*** gfidente has quit IRC | 15:08 | |
*** tremble has joined #tripleo | 15:11 | |
*** jaosorior has quit IRC | 15:20 | |
*** ramishra has quit IRC | 15:21 | |
*** ramishra has joined #tripleo | 15:23 | |
weshay | cloudnull++ d0ugal++ | 15:24 |
* d0ugal looks around curiously | 15:25 | |
openstackgerrit | Arx Cruz proposed openstack/python-tripleoclient master: Fix KeyError exception on check_deprecated_parameters https://review.opendev.org/711480 | 15:29 |
weshay | anyone seen this in centos-8-containers-multinode | 15:39 |
weshay | 2020-03-05 12:21:07 | heat_resource_tree_params = heat_resource_tree['parameters'] | 15:39 |
weshay | 2020-03-05 12:21:07 | KeyError: 'parameters' | 15:39 |
weshay | 2020-03-05 12:21:07 | 'parameters' | 15:39 |
weshay | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_879/710961/3/gate/tripleo-ci-centos-8-containers-multinode/879bf81/logs/undercloud/home/zuul/overcloud_deploy.log | 15:39 |
weshay | https://bugs.launchpad.net/tripleo/+bug/1866184 | 15:40 |
openstack | Launchpad bug 1866184 in tripleo "centos-8 multinode overcloud deploy failed with heat_resource_tree_params = heat_resource_tree['parameters'] KeyError: 'parameters'" [Critical,In progress] - Assigned to Arx Cruz (arxcruz) | 15:40 |
weshay | nrmind | 15:40 |
weshay | arxcruz|rover++ | 15:40 |
weshay | EmilienM, cloudnull please review asap containers-multinode is down https://review.opendev.org/#/c/711480/ | 15:40 |
cloudnull | weshay - https://review.opendev.org/#/c/711291/ | 15:41 |
weshay | cloudnull, or should I double check w/ centos-8 | 15:42 |
weshay | just looking for confidence level | 15:42 |
openstackgerrit | wes hayutin proposed openstack/tripleo-ci master: make centos-8 containers-multinode voting/gating https://review.opendev.org/711488 | 15:46 |
arxcruz|rover | weshay: not sure if it will work because of other issue with dict not being iterable | 15:46 |
ykarel|away | cloudnull, will ^^ also fixes unhashable dict error at 'user_defined': i in user_params.keys() | 15:47 |
arxcruz|rover | nevermind, it will work because i in user_params.keys will return true :) | 15:47 |
sshnaidm | weshay, another one: https://bugs.launchpad.net/tripleo/+bug/1866204 | 15:47 |
openstack | Launchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged] | 15:47 |
ykarel|away | https://review.opendev.org/#/c/709141/7/tripleoclient/workflows/parameters.py@128 | 15:47 |
ykarel|away | cloudnull, ^^ | 15:47 |
sshnaidm | weshay, actually it can be a promotion blocker | 15:47 |
*** ramishra has quit IRC | 15:48 | |
cloudnull | ykarel|away weshay EmilienM I'm happy to abandon my review and go with https://review.opendev.org/#/c/711480 | 15:48 |
openstackgerrit | wes hayutin proposed openstack/python-tripleoclient master: update layout for centos-8 containers-multinode https://review.opendev.org/711489 | 15:49 |
ykarel|away | cloudnull, me not sure if this will fix unhashable dict issue | 15:49 |
sshnaidm | cloudnull, apetrich fyi: https://bugs.launchpad.net/tripleo/+bug/1866204 I think cleaning node by default was introduced in one of latest mistral_to_ansible patches. We don't need to clean nodes in CI, need to make sure it's not by default when running nodes "provide" | 15:49 |
openstack | Launchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged] | 15:49 |
ykarel|away | if u are sure any will work | 15:49 |
weshay | cloudnull, ykarel|away let me know.. once it's green.. it needs be returned to voting asap | 15:50 |
weshay | sshnaidm, is centos-8 stuck on the same thing? | 15:51 |
sshnaidm | weshay, yeah | 15:51 |
cloudnull | tripleo-ci-centos-7-containers-multinode passed https://review.opendev.org/#/c/711291 are we seeing an issue with "unhashable dict error at 'user_defined': i in user_params.keys()" ? | 15:51 |
weshay | sshnaidm, https://review.rdoproject.org/r/#/c/25332/ | 15:51 |
weshay | k | 15:51 |
cloudnull | sshnaidm i dont think that the cleaning patches merged just yet ? | 15:51 |
ykarel|away | cloudnull, issue is in centos8 | 15:51 |
ykarel|away | with python3 | 15:51 |
ykarel|away | python2 jobs are fine | 15:51 |
cloudnull | oic | 15:51 |
cloudnull | then lets go with arxcruz|rover's review w | 15:51 |
sshnaidm | cloudnull, maybe register patch..? | 15:51 |
cloudnull | i'll abandon mine | 15:51 |
ykarel|away | cloudnull, so that review will fix unhashable dict error? | 15:52 |
weshay | ekultails, cix https://bugs.launchpad.net/tripleo/+bug/1866204 | 15:52 |
openstack | Launchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged] | 15:52 |
sshnaidm | weshay, it's here: https://review.rdoproject.org/r/#/c/25666/ | 15:52 |
sshnaidm | weshay, the job you pointed to is waiting for image building fixes, chandankumar is on it | 15:53 |
arxcruz|rover | cloudnull: sorry, didn't knew you were working on it, just saw the bug filled :) | 15:53 |
cloudnull | its all good | 15:53 |
cloudnull | ykarel|away it looks like it | 15:53 |
cloudnull | https://pasted.tech/pastes/fab38cc0c5109a888bf1b385e1e086c867cfa7f2 | 15:53 |
weshay | so which patch we going w/? | 15:53 |
cloudnull | arxcruz|rover 's review will ensure both default to a dict | 15:54 |
ykarel|away | cloudnull, instead of 'i' try a dict | 15:54 |
ykarel|away | i think that's the issue | 15:54 |
ykarel|away | in python2 that will pass | 15:54 |
ykarel|away | with py3 fails | 15:54 |
cloudnull | i just tested it with py37 | 15:54 |
ykarel|away | cloudnull, but u tested string | 15:54 |
arxcruz|rover | ykarel|away: but i should not be a dict... | 15:54 |
ykarel|away | test dict | 15:55 |
ykarel|away | arxcruz|rover, somehow it's coming as dict | 15:55 |
ykarel|away | that's why the issue imo | 15:55 |
ykarel|away | python3 -c "x={1:1};{} in x.keys()" | 15:56 |
*** jcapitao is now known as jcapitao_afk | 15:56 | |
ykarel|away | vs python2 -c "x={1:1};{} in x.keys()" | 15:56 |
arxcruz|rover | ykarel|away: yeah you right, let me test something | 15:57 |
cloudnull | interesting. TIL | 15:57 |
weshay | arxcruz|rover++ ykarel|away++ | 15:57 |
cloudnull | probably need to use a sentinel for py3, might need to handle that loop two ways depending on py version | 15:58 |
openstackgerrit | wes hayutin proposed openstack/tripleo-ci master: make centos-8 containers-multinode voting/gating https://review.opendev.org/711488 | 15:58 |
openstackgerrit | wes hayutin proposed openstack/python-tripleoclient master: update layout for centos-8 containers-multinode https://review.opendev.org/711489 | 15:59 |
weshay | mwhahaha, https://review.opendev.org/#/c/711320/ | 16:00 |
cloudnull | sshnaidm the register patch hasn't merged yet either - https://review.opendev.org/#/c/709583/6 | 16:00 |
*** dsneddon has joined #tripleo | 16:01 | |
*** Trevor_V has joined #tripleo | 16:01 | |
openstackgerrit | wes hayutin proposed openstack/tripleo-ansible master: update containers-multinode to centos-8 https://review.opendev.org/711494 | 16:04 |
*** TrevorV has quit IRC | 16:05 | |
openstackgerrit | wes hayutin proposed openstack/tripleo-common master: update containers-multinode to centos-8 https://review.opendev.org/711496 | 16:06 |
sshnaidm | cloudnull, I see, will look more.. | 16:10 |
*** jcapitao_afk is now known as jcapitao | 16:10 | |
sshnaidm | rlandy, do you remember if we run nodes clean by default in ovb? | 16:10 |
*** jcapitao has quit IRC | 16:12 | |
rlandy | sshnaidm: I disabled that on baremetal because it was taking forever - by default on ovb, I think yes - will have to check that | 16:12 |
rlandy | getting | 16:12 |
openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Remove topic suffix from default value https://review.opendev.org/710123 | 16:12 |
*** morazi has quit IRC | 16:13 | |
rlandy | sshnaidm: so checking the settings for OVB - yes - we clean nodes in releases after pike (other than the vexxhost settings you have) | 16:16 |
sshnaidm | rlandy, where is it defined? | 16:16 |
rlandy | sshnaidm: search for undercloud_clean_nodes | 16:16 |
rlandy | it's in undercloud.conf | 16:17 |
rlandy | we set that in some of the featuresets | 16:17 |
rlandy | and then you have a vexxhost setting in tripleo-ci | 16:17 |
rlandy | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/hardware_environments/dell_fc430_envA/network_configs/single_nic_vlans/env_settings.yml#n30 - that is where we disable for bm - downstream only | 16:18 |
*** jcapitao has joined #tripleo | 16:18 | |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove the mistral from the deploy_plan workflow https://review.opendev.org/710479 | 16:19 |
sshnaidm | rlandy, hmm.. I see it's disabled in vexx | 16:20 |
sshnaidm | rlandy, didn't think it's done in undercloud part | 16:20 |
rlandy | sshnaidm: yeah - not sure why that would be | 16:20 |
rlandy | it's set that | 16:20 |
rlandy | there | 16:20 |
rlandy | the only reason to disable it would be timing or issues with the overcloud nodes disks | 16:21 |
openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: update containers-multinode to centos-8 https://review.opendev.org/711499 | 16:23 |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ci master: Sync vexxhost settings for OVB https://review.opendev.org/711500 | 16:24 |
sshnaidm | rlandy, let's see ^^ | 16:24 |
rlandy | sshnaidm: for some reason it's also set in multinode on vexxhost? https://github.com/openstack/tripleo-ci/blob/master/toci-quickstart/config/testenv/multinode-vexxhost.yml#L57 | 16:25 |
openstackgerrit | Arx Cruz proposed openstack/python-tripleoclient master: Fix KeyError exception on check_deprecated_parameters https://review.opendev.org/711480 | 16:26 |
arxcruz|rover | ykarel|away: now it will work :) ^^ | 16:26 |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ci master: Sync vexxhost and rdo cloud settings https://review.opendev.org/711500 | 16:28 |
sshnaidm | rlandy, ack, changed it too | 16:28 |
rlandy | k - let's see the damage | 16:28 |
*** jcoufal has quit IRC | 16:29 | |
arxcruz|rover | weshay: ^ | 16:30 |
arxcruz|rover | weshay: heading home, ping me if you need something :) | 16:30 |
openstackgerrit | wes hayutin proposed openstack/python-tripleoclient master: update containers-multinode to centos-8 https://review.opendev.org/711503 | 16:32 |
weshay | arxcruz|rover, thanks | 16:32 |
weshay | arxcruz|rover, confident in that last revision? | 16:32 |
*** ramishra has joined #tripleo | 16:33 | |
openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-ci master: Add centos8 support for emit_releases_script https://review.opendev.org/710441 | 16:33 |
*** aedc_ has joined #tripleo | 16:34 | |
openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-quickstart-extras master: Revert "Set container build hash based on dlrn md5 checksum" https://review.opendev.org/711305 | 16:34 |
chandankumar | rlandy, ^^ | 16:34 |
rlandy | chandankumar: yeah - better - the other way around would be a disaster :( | 16:35 |
chandankumar | rlandy, rechecked here https://review.rdoproject.org/r/#/c/25694/ | 16:35 |
rlandy | chandankumar: ack - thanks - will keep an eye on that | 16:36 |
*** aedc has quit IRC | 16:36 | |
openstackgerrit | Merged openstack/python-tripleoclient master: Avoid TZ issues in test_image test https://review.opendev.org/711253 | 16:38 |
openstackgerrit | Marios Andreou proposed openstack/tripleo-ci master: WIP Add centos-8-scenario010-ovn-provider-standalone job definition https://review.opendev.org/711507 | 16:40 |
rlandy | marios: did you see this at all in the last few days ... Data could not be sent to remote host "127.0.0.2". Make sure this host can be reached over ssh: kex_exchange_identification: Connection closed by remote host | 16:43 |
openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Enforce firewall rules before pacemaker-auth https://review.opendev.org/711509 | 16:43 |
rlandy | hitting the component pipeline in a few places | 16:43 |
rlandy | after the tempest run | 16:44 |
rlandy | arxcruz|rover: ^^? | 16:44 |
rlandy | rechecking those jobs | 16:44 |
*** udesale has quit IRC | 16:44 | |
*** cfontain has joined #tripleo | 16:46 | |
openstackgerrit | wes hayutin proposed openstack/tripleo-ci master: update periodics for centos-8 https://review.opendev.org/711510 | 16:46 |
*** morazi has joined #tripleo | 16:46 | |
*** Trevor_V has quit IRC | 16:47 | |
*** ccamacho has quit IRC | 16:48 | |
openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade master: Adapt FFWD2 Undercloud upgrade workflow. https://review.opendev.org/706238 | 16:48 |
*** aedc_ has quit IRC | 16:49 | |
*** TrevorV has joined #tripleo | 16:50 | |
openstackgerrit | wes hayutin proposed openstack/python-tripleoclient master: update containers-multinode, minion to centos-8 https://review.opendev.org/711503 | 16:51 |
*** aedc_ has joined #tripleo | 16:51 | |
*** cfontain has quit IRC | 16:52 | |
openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade master: Adapt FFWD2 Undercloud upgrade workflow. https://review.opendev.org/706238 | 16:53 |
*** rcernin has quit IRC | 16:55 | |
openstackgerrit | wes hayutin proposed openstack/tripleo-ansible master: update more jobs to centos-8 https://review.opendev.org/711494 | 16:56 |
*** aedc_ has quit IRC | 16:56 | |
*** marios is now known as marios|out | 16:59 | |
*** cylopez has quit IRC | 17:00 | |
openstackgerrit | Merged openstack/tripleo-ansible stable/train: Tune-up package install https://review.opendev.org/711349 | 17:01 |
*** tesseract-RH has quit IRC | 17:01 | |
*** lucasagomes has quit IRC | 17:02 | |
*** bogdando has quit IRC | 17:06 | |
openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: update containers-multinode, minion to centos-8 https://review.opendev.org/711499 | 17:07 |
openstackgerrit | wes hayutin proposed openstack/tripleo-ci master: make centos-8 undercloud-containers voting https://review.opendev.org/711516 | 17:10 |
*** suuuper has quit IRC | 17:15 | |
*** jpich has quit IRC | 17:22 | |
*** rpittau is now known as rpittau|afk | 17:25 | |
openstackgerrit | Merged openstack/tripleo-ansible stable/train: Creates container_config_data module https://review.opendev.org/711347 | 17:29 |
openstackgerrit | Merged openstack/tripleo-ansible master: Only add host entries for role_networks https://review.opendev.org/710574 | 17:32 |
openstackgerrit | wes hayutin proposed openstack/tripleo-common master: update jobs to centos-8 https://review.opendev.org/711496 | 17:32 |
*** evrardjp has quit IRC | 17:35 | |
*** marios|out has quit IRC | 17:35 | |
*** evrardjp has joined #tripleo | 17:35 | |
*** holser has quit IRC | 17:36 | |
weshay | ekultails, sorry for the confusion.. you have the right log and review now https://logserver.rdoproject.org/66/25666/7/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/4fc1241/job-output.txt | 17:37 |
weshay | thanks for the help | 17:37 |
*** dtantsur is now known as dtantsur|afk | 17:40 | |
ekultails | weshay: No worries. Thanks for the update. :-) | 17:46 |
openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ci master: lint: enable black https://review.opendev.org/703980 | 17:49 |
openstackgerrit | Merged openstack/tripleo-ansible master: update tripleo-ansible ci for centos-8 https://review.opendev.org/710810 | 17:59 |
*** derekh has quit IRC | 18:00 | |
openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Make sure we create stonith resources before stonith levels https://review.opendev.org/711527 | 18:06 |
*** bandini has quit IRC | 18:06 | |
*** jcapitao is now known as jcapitao_off | 18:10 | |
*** slaweq has joined #tripleo | 18:11 | |
openstackgerrit | Arx Cruz proposed openstack/python-tripleoclient master: Fix KeyError exception on check_deprecated_parameters https://review.opendev.org/711480 | 18:13 |
*** jcapitao_off has quit IRC | 18:15 | |
*** chandankumar is now known as raukadah | 18:18 | |
*** ccamacho has joined #tripleo | 18:20 | |
*** Goneri has quit IRC | 18:23 | |
*** jbadiapa has quit IRC | 18:23 | |
*** jpena is now known as jpena|off | 18:26 | |
*** Goneri has joined #tripleo | 18:27 | |
openstackgerrit | Merged openstack/tripleo-heat-templates stable/train: nova-compute-container: add missing condition for ksmdisabled https://review.opendev.org/711368 | 18:31 |
openstackgerrit | Merged openstack/tripleo-common master: deployment: set DeployIdentifier to '' if --skip-deploy-identifier is used https://review.opendev.org/711366 | 18:31 |
openstackgerrit | Merged openstack/python-tripleoclient master: overcloud: set DeployIdentifier to '' if --skip-deploy-identifier is used https://review.opendev.org/711369 | 18:31 |
openstackgerrit | Merged openstack/python-tripleoclient master: Skip network check with stack env when validations disabled https://review.opendev.org/711392 | 18:31 |
sshnaidm | ekultails, centos8 bug is here https://bugs.launchpad.net/tripleo/+bug/1866202 | 18:32 |
openstack | Launchpad bug 1866202 in tripleo "OVB on centos8 fails because of networking failures" [Critical,Triaged] | 18:32 |
sshnaidm | ekultails, the patch that runs centos8 is https://review.rdoproject.org/r/#/c/25666/ | 18:33 |
ekultails | sshnaidm: Awesome, thanks for sharing. Yeah, I was just looking at the NIC configs on one of these broken CI jobs. | 18:35 |
*** amoralej is now known as amoralej|off | 18:35 | |
sshnaidm | ekultails, br-ex is a red herring, the problem starts before that.. | 18:35 |
sshnaidm | ekultails, maybe changes with network manager, a lot of logs from it | 18:35 |
sshnaidm | ekultails, centos8 btw has renamed interfaces to "ensX", not sure how it's in images, need to check | 18:36 |
ekultails | sshanidm: https://logserver.rdoproject.org/66/25666/7/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/4fc1241/logs/overcloud-controller-0/etc/sysconfig/network-scripts/ | 18:37 |
ekultails | I was wondering if that was the problem. There's both ifcfg-eth* and a ifcfg-ens3. | 18:37 |
ekultails | sshnaidm: This is my latest update/comment from the CI bug I've been looking into: https://bugs.launchpad.net/tripleo/+bug/1866204/comments/7 | 18:38 |
openstack | Launchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged] | 18:38 |
ekultails | I think we're on the right track. It might just bee something in TripleO Quickstart that needs to be updated. | 18:39 |
weshay | sshnaidm, ekultails we can't really fix the ifnames issue from ci.. would have to be in the dib | 18:41 |
sshnaidm | ekultails, this is a different bug | 18:41 |
sshnaidm | ekultails, if you're talking about "yum", it's wip, it's a wrong patch | 18:41 |
sshnaidm | ekultails, just updated bugs with comments, please take a look | 18:41 |
sshnaidm | weshay, I just checked, and seems like interfaces are ok in ovb | 18:42 |
weshay | sshnaidm, ik | 18:42 |
weshay | ok | 18:42 |
sshnaidm | weshay, they have ethX | 18:42 |
weshay | sshnaidm, can you set the default to download and not build images? | 18:42 |
openstackgerrit | Alex Schultz proposed openstack/tripleo-common stable/stein: Use config for layers https://review.opendev.org/710705 | 18:42 |
weshay | maybe in the release file | 18:42 |
weshay | for tripleo-ci/CentOS-8/master.yml | 18:43 |
sshnaidm | weshay, it is, but periodic job usually build and test them | 18:43 |
weshay | sshnaidm, ya.. not needed though | 18:43 |
ekultails | sshnaidm: The yum comment can be disregarded. I guess the OVB cleaning is probably different than the NIC configuration issue, right? | 18:43 |
weshay | because of the job that actually builds them | 18:43 |
weshay | better to skip the build in ovb jobs for now | 18:43 |
sshnaidm | weshay, anyway, raukadah should make this script working for c8, it's wip in https://review.opendev.org/#/c/711081/ | 18:44 |
weshay | until we kill the tqe role | 18:44 |
sshnaidm | weshay, if we have building jobs in periodic we can configure them | 18:44 |
weshay | sshnaidm, we have centos-8 overcloud-full and ipa landed | 18:44 |
sshnaidm | weshay, because it was failing before when didn't find tripleo-ci-testing images.. | 18:44 |
weshay | k.. we will now | 18:44 |
weshay | sshnaidm, next run we should see it | 18:45 |
sshnaidm | weshay, ok, then it's fine to run it with "to_build: false" | 18:45 |
weshay | ++ | 18:45 |
weshay | yes | 18:45 |
*** slaweq has quit IRC | 18:47 | |
*** rmccabe has joined #tripleo | 18:53 | |
rmccabe | anyone know if there's a known good procedure for deploying from upstream master? trying to do some testing and i've not been able to get a good deploy | 18:54 |
*** jaosorior has joined #tripleo | 19:08 | |
sshnaidm | ekultails, can you please move your comments to c8 bug? It will help to concentrate the all info in a right place | 19:17 |
*** mcornea has quit IRC | 19:18 | |
openstackgerrit | Brent Eagles proposed openstack/tripleo-ansible master: octavia move openstack api calls to undercloud https://review.opendev.org/711094 | 19:23 |
ekultails | sshnaidm: Will do. | 19:27 |
openstackgerrit | wes hayutin proposed openstack/tripleo-ansible master: WIP: here we go.. centos-7 -> 8 for molecule https://review.opendev.org/711535 | 19:28 |
weshay | cloudnull, should be interesting ^ | 19:28 |
openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Add job.repos to container updates for component jobs https://review.opendev.org/711156 | 19:29 |
openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Drop resource registry override in ip-from-pool https://review.opendev.org/711537 | 19:33 |
openstackgerrit | Harald Jensås proposed openstack/tripleo-docs master: Predictable IPs, no resource_registry override https://review.opendev.org/711539 | 19:37 |
cloudnull | weshay nice! The roles should work on RHEL / cent 8. Most are already testing in cent8 containers. | 19:40 |
weshay | ok.. cloudnull we were wondering if the upgrade roles need to run on centos-7 still | 19:40 |
weshay | not quite sure what to make of that was going to ping those guys | 19:40 |
weshay | matbu, ^ may be able to provide some insight | 19:40 |
openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Drop resource registry override in ip-from-pool https://review.opendev.org/711537 | 19:41 |
openstackgerrit | Ryan McCabe proposed openstack/tripleo-heat-templates master: Add STF environment https://review.opendev.org/694623 | 19:42 |
sshnaidm | rlandy, seems like cleaning fails on vexx: https://logserver.rdoproject.org/00/711500/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/1324126/logs/undercloud/var/log/extra/baremetal_list.txt.gz | 19:42 |
*** saneax has quit IRC | 19:42 | |
sshnaidm | rlandy, need to fix that.. | 19:42 |
sshnaidm | weshay, ^^ | 19:42 |
*** pcaruana has quit IRC | 19:42 | |
rlandy | sshnaidm: is there a difference in the nodes between vexx and rdocloud? | 19:43 |
rlandy | looking at error | 19:43 |
sshnaidm | rlandy, should not be, except of openstack version if the cloud | 19:43 |
openstackgerrit | wes hayutin proposed openstack/python-tripleoclient master: Fix KeyError exception on check_deprecated_parameters https://review.opendev.org/711480 | 19:43 |
*** cfontain has joined #tripleo | 19:43 | |
sshnaidm | rlandy, same settings, and I think we disabled cleaning before because it was failing.. | 19:44 |
rlandy | sshnaidm: you probably had a reason to make cleaning false | 19:44 |
weshay | sshnaidm, can't we turn cleaning off | 19:44 |
sshnaidm | rlandy, yeah, and forgot about it later | 19:44 |
sshnaidm | weshay, it decreases our coverage | 19:44 |
sshnaidm | weshay, somehow it works on rdo cloud | 19:45 |
rlandy | had some problems in baremetal - but the overcloud nodes there are not clean when we start | 19:45 |
sshnaidm | rlandy, vms should be clean, just created | 19:45 |
rlandy | right - here you should not have that issue | 19:46 |
mmagr | Hey guys, is there anybody able to find time for review of https://review.opendev.org/#/c/704080/ or https://review.opendev.org/#/c/709106/ please? | 19:46 |
rlandy | introspection does ok | 19:46 |
sshnaidm | rlandy, introspection didn't start, there are no manageable nodes | 19:46 |
sshnaidm | rlandy, actually I think maybe we need to fail it in this case.. | 19:47 |
cloudnull | weshay no idea | 19:47 |
rlandy | 2020-03-05 17:44:23 | msg: No nodes are manageable at this time. 2020-03-05 17:47:58 | 4 node(s) successfully moved to the "available" state. | 19:47 |
rlandy | that should fail | 19:48 |
rlandy | sshnaidm: ^^ yeah definitely | 19:48 |
rlandy | then we would catch it there | 19:48 |
mwhahaha | cloudnull: openstack overcloud node introspect --all-manageable --provide is broken | 19:49 |
*** cfontain has quit IRC | 19:49 | |
mwhahaha | cloudnull: http://paste.openstack.org/show/790358/ | 19:49 |
cloudnull | I dont think the provide patch merged yet ? | 19:50 |
mwhahaha | we shouldn't have removed the ansible bits then | 19:51 |
cloudnull | https://review.opendev.org/#/c/710308/ | 19:51 |
mwhahaha | using a uuid doesn't work either | 19:51 |
mwhahaha | http://pastebin.test.redhat.com/842422 | 19:51 |
cloudnull | looks like its concatenating the uuids - ebc0d52f-1c93-4c72-8b93-5b9e052ddc60 0cacc47d-7bf4-447c-a6d1-ef637ed05dbd 0b3f9b16-3027-49f5-994a-bdf830f8837f | 19:52 |
rlandy | sshnaidm: have you checked with the vexx guys about the node configs. We usually see this failure when there is a disk issue in cleaning | 19:53 |
mwhahaha | cloudnull: no it errors with one node too | 19:53 |
cloudnull | mwhahaha https://review.opendev.org/#/c/710308/5/tripleoclient/workflows/baremetal.py - we're not using the provide playbook yet, not until that patch is merged | 19:54 |
mwhahaha | this seems to indicate otherwise? | 19:54 |
* mwhahaha shrugs | 19:54 | |
mwhahaha | this is the same workflow i've been using for years | 19:54 |
mwhahaha | so kinda bad it no longer works | 19:54 |
* mwhahaha files a bug | 19:54 | |
sshnaidm | mwhahaha, we have a problem with this recently | 19:55 |
sshnaidm | mwhahaha, I think it's related to bug that was opened today | 19:55 |
sshnaidm | mwhahaha, but in here http://paste.openstack.org/show/790358/ nodes are in "available" state | 19:55 |
sshnaidm | not in "managealbe" | 19:56 |
mwhahaha | right that's what --provide is supposed to do | 19:56 |
mwhahaha | if i remember correctly | 19:56 |
mwhahaha | so that's the missing piece | 19:57 |
mwhahaha | --provide has no effect | 19:57 |
mwhahaha | wait ---provide moves them to available | 19:57 |
sshnaidm | mwhahaha, yep | 19:58 |
sshnaidm | mwhahaha, provide does move to available | 19:58 |
sshnaidm | mwhahaha, registration should move to manageable, introspection just introspects, cleaning leaves nodes in manageable, and then "provide" make them available | 19:59 |
sshnaidm | iirc | 19:59 |
cloudnull | so we're good ? | 20:01 |
mwhahaha | i don't think that's what happens | 20:01 |
sshnaidm | mwhahaha, but maybe you hit https://bugs.launchpad.net/tripleo/+bug/1866204 | 20:01 |
openstack | Launchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged] | 20:01 |
mwhahaha | no the registration was successful | 20:02 |
sshnaidm | it's not clear yet what happens, but definitely some problem is there | 20:02 |
openstackgerrit | Emilien Macchi proposed openstack/paunch stable/train: Cleanup containers in the same loop as they are created https://review.opendev.org/709760 | 20:02 |
sshnaidm | mwhahaha, what are nodes state right after registration? | 20:02 |
mwhahaha | let me nuke them and see | 20:03 |
mwhahaha | i was also going to try osp16 as well | 20:03 |
sshnaidm | something should call "manage" for nodes to make them manageable, and maybe it was part of registration in mistral.. there are so much duplicates in these calls | 20:04 |
mwhahaha | ok so import makes them manageable | 20:04 |
sshnaidm | aha | 20:04 |
sshnaidm | mwhahaha, and then you run introspection? | 20:05 |
mwhahaha | yea | 20:05 |
mwhahaha | i usually run introspect --all-manageable --provide | 20:05 |
sshnaidm | mwhahaha, and it doesn't find manageabler nodes? | 20:05 |
mwhahaha | ok so it's --provide | 20:06 |
mwhahaha | it's running before introspection | 20:06 |
mwhahaha | if i do introspect <uuids> it's running | 20:06 |
sshnaidm | cloudnull, maybe we switched the order? ^ | 20:07 |
cloudnull | of all managable and provide ? | 20:08 |
mwhahaha | --provide | 20:08 |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/train: Cleanup tasks for container-puppet.py https://review.opendev.org/711542 | 20:08 |
mwhahaha | --all-manageable just doesn't work | 20:08 |
mwhahaha | unless --provide is running prior to --all-manageable lookups | 20:09 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Revert "Modify the introspection playbook to simplify client interactions" https://review.opendev.org/711543 | 20:09 |
mwhahaha | i'll check that next | 20:09 |
cloudnull | ^ that should fix the all-managable issue | 20:09 |
mwhahaha | so i had successful introspection w/o --provide | 20:09 |
cloudnull | https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v2/overcloud_node.py#L89-L98 - it looks like provide is still using the old mistral functions still, | 20:11 |
sshnaidm | cloudnull, oh, I see it remove all-manageable option | 20:11 |
mwhahaha | i think it still ran when the introspect failed | 20:11 |
mwhahaha | so that might have been the problem | 20:12 |
cloudnull | the provide playbook isn't yet afaict | 20:12 |
mwhahaha | --all-manageable was a noop, and we then ran provide | 20:12 |
cloudnull | ****isn't used yet afaict | 20:12 |
sshnaidm | cloudnull, so it might solve also https://bugs.launchpad.net/tripleo/+bug/1866204 | 20:14 |
openstack | Launchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged] | 20:14 |
sshnaidm | cloudnull, I'd be glad to be added to reviews for any introspection part | 20:14 |
sshnaidm | cloudnull, anyway I deal with it in OVB jobs | 20:14 |
cloudnull | if we can get https://review.opendev.org/#/c/707348 in, then that revert wouldn't be needed. | 20:15 |
openstackgerrit | Alex Schultz proposed openstack/tripleo-ansible master: Revert "Modify the introspection playbook to simplify client interactions" https://review.opendev.org/711543 | 20:15 |
mwhahaha | added closes bug | 20:15 |
mwhahaha | i think that solves that bug | 20:15 |
cloudnull | ++ | 20:15 |
mwhahaha | because what happened is that --all-manageable basically did nothing but the provide did run | 20:15 |
mwhahaha | which makes subsequent runs bad | 20:15 |
mwhahaha | because it's already been moved to available even without introspection | 20:16 |
cloudnull | makes sense. | 20:16 |
mwhahaha | seems like we shouldn't be able to provide a node if it hasn't been introspected | 20:16 |
mwhahaha | that seems like a bug in ironic? | 20:16 |
mwhahaha | unless it's valid to be able to skip introspection? | 20:16 |
openstackgerrit | Ronelle Landy proposed openstack/tripleo-ci master: DNM: Add centos-8 scenario000-multinode update job https://review.opendev.org/710876 | 20:18 |
*** cylopez has joined #tripleo | 20:23 | |
*** sgw has quit IRC | 20:23 | |
weshay | how the heck did it pass two ovb jobs to begin w/ | 20:24 |
sshnaidm | mwhahaha, we can skip introspection totally | 20:25 |
sshnaidm | mwhahaha, and cleaning | 20:25 |
lbragstad | hey folks, i have an environment using RDO master and i verified https://review.opendev.org/#/c/702031/13 and https://review.opendev.org/#/c/705041/3 exist on the undercloud | 20:25 |
sshnaidm | mwhahaha, cloudnull but I think maybe worth to fail a playbook if no node was introspected? | 20:25 |
mwhahaha | for that command, yea | 20:26 |
cloudnull | probably a good idea | 20:26 |
mwhahaha | we have a seperate provide command | 20:26 |
lbragstad | my overcloud nodes have a processor count of 4 (using lscpu), but only two keystone processes are running in the keystone container | 20:26 |
sshnaidm | mwhahaha, and maybe worth to separate these commands, it's too much dups | 20:26 |
mwhahaha | lbragstad: bet you don't have those in overcloud-full | 20:26 |
lbragstad | am i missing something or should there be at least 4? | 20:26 |
mwhahaha | sshnaidm: i perfer the single shot command but that's me | 20:27 |
cloudnull | sshnaidm once we revert the revert, the playbook will require that the node uuids be provided. | 20:27 |
cloudnull | so that should guard against no nodes being introspected ? | 20:27 |
sshnaidm | cloudnull, we still need --all-manageable | 20:27 |
cloudnull | for sure, that will be handled in the client | 20:27 |
sshnaidm | cloudnull, and if it doesn't find manageable nodes, need to fail | 20:27 |
lbragstad | mwhahaha overcloud-full? for additional context, i'm using a deployed-server environment | 20:28 |
mwhahaha | lbragstad: ok, same diff. do the overcloud nodes have the patches | 20:28 |
cloudnull | sshnaidm https://review.opendev.org/#/c/707348/9/tripleoclient/workflows/baremetal.py@169 | 20:28 |
sshnaidm | cloudnull, oh, I recall that you want to sort manageable nodes in client, right? | 20:28 |
lbragstad | mwhahaha checking | 20:28 |
cloudnull | that playbook should fail if the list is empty | 20:28 |
mwhahaha | lbragstad: also do you have the THT that switches it to be os_workers_keystone, i think it's os_workers by default in THT | 20:29 |
cloudnull | though we could add a specific fail task to ensure | 20:29 |
sshnaidm | cloudnull, yeah, that seems totally fine | 20:29 |
sshnaidm | cloudnull, and this as well | 20:29 |
cloudnull | but for now the revert seems like the best course of action | 20:29 |
cloudnull | we have a bunch of changes to get through before https://review.opendev.org/#/c/707348 is in | 20:30 |
mwhahaha | lbragstad: https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/keystone/keystone-container-puppet.yaml#L112 needs to be updated | 20:30 |
sshnaidm | cloudnull, maybe it will solve problems with ovb failing there.. | 20:30 |
cloudnull | potentially | 20:30 |
sshnaidm | cloudnull, let's rerun check-rdo on all of them after this is merged | 20:30 |
lbragstad | mwhahaha that explains it | 20:30 |
lbragstad | mwhahaha i deployed by overriding KeystoneWorkers and it worked... i'll get a patch up | 20:31 |
*** cfontain has joined #tripleo | 20:31 | |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Fix delete plan https://review.opendev.org/711548 | 20:32 |
mwhahaha | hrm loading local files seems broken with overcloud deploy command | 20:33 |
mwhahaha | -e from the local path doesn't seem to be found | 20:33 |
mwhahaha | i don't recall any patches around our template stuff lately, does anyone know of recent changes? | 20:33 |
mwhahaha | i take that back | 20:35 |
fultonj | mwhahaha: i don't. that sounds scary. how would such a break pass any of the ci jobs | 20:35 |
mwhahaha | EmilienM: we might want to revert https://review.opendev.org/#/c/642032/15 | 20:35 |
openstackgerrit | Lance Bragstad proposed openstack/tripleo-heat-templates master: Update the number of keystone workers https://review.opendev.org/711549 | 20:35 |
lbragstad | mwhahaha done ^ | 20:35 |
EmilienM | yeah? | 20:35 |
mwhahaha | fun now i get heat_resource_tree_params = heat_resource_tree['parameters'] | 20:35 |
* mwhahaha flips tables | 20:35 | |
mwhahaha | and openstack overcloud pland elete is broken | 20:36 |
mwhahaha | fjdkhlfhdksjfhdasljfdshjkf | 20:36 |
EmilienM | faeioujefaoiaefiaefio | 20:36 |
mwhahaha | AttributeError: 'ClientManager' object has no attribute 'create_mistral_context' | 20:36 |
EmilienM | oops | 20:36 |
EmilienM | https://review.opendev.org/#/c/705678/ ? | 20:36 |
mwhahaha | yes that doesn't work | 20:37 |
openstackgerrit | wes hayutin proposed openstack/paunch master: move centos-7 jobs to centos-8 where possible https://review.opendev.org/710816 | 20:37 |
mwhahaha | so glad we're testing things | 20:37 |
EmilienM | apparently not | 20:37 |
EmilienM | d0ugal: ^ fyi should we revert as well? | 20:37 |
cloudnull | mwhahaha Fix delete plan https://review.opendev.org/711548 | 20:38 |
* mwhahaha files bug | 20:38 | |
openstackgerrit | wes hayutin proposed openstack/paunch master: move centos-7 jobs to centos-8 where possible https://review.opendev.org/710816 | 20:38 |
EmilienM | nice | 20:38 |
EmilienM | cloudnull reads minds | 20:38 |
* cloudnull just ran into that issue today | 20:39 | |
mwhahaha | ok so now i can't deploy | 20:39 |
mwhahaha | heat_resource_tree_params = heat_resource_tree['parameters'] | 20:39 |
mwhahaha | is failing on parameters | 20:39 |
mwhahaha | that's teh check deprecate parameters function | 20:39 |
*** sgw has joined #tripleo | 20:39 | |
mwhahaha | i vaguely recall a cix about that | 20:39 |
cloudnull | mwhahaha arxcruz|rover has a fix for that - https://review.opendev.org/#/c/711480/ | 20:40 |
mwhahaha | why don't we have this key anymore | 20:41 |
mwhahaha | wonder if it's related to that previous heat patch | 20:41 |
openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Revert "Modify roles to remove unused services" https://review.opendev.org/711551 | 20:42 |
* mwhahaha tests | 20:42 | |
cloudnull | it looks like the key isn't there on first run? | 20:42 |
mwhahaha | but there was | 20:42 |
mwhahaha | like this just came up after it wasn't finding my files in the local dir that were not fully qualified | 20:42 |
* mwhahaha shrugs | 20:42 | |
cloudnull | I ran into that issue the other day as well, and posted https://review.opendev.org/#/c/711291/ but arxcruz|rover review was better. | 20:43 |
mwhahaha | something weird is going on | 20:43 |
mwhahaha | i guess i should deploy an overcloud by hand more often | 20:44 |
weshay | hand built overcloud's a delicious treat | 20:44 |
mwhahaha | it's like getting punched in the face every command run | 20:44 |
*** TrevorV has quit IRC | 20:44 | |
fultonj | artisanal overclouds | 20:44 |
mwhahaha | ok so that unused service thing is not related to teh parameters | 20:45 |
*** sshnaidm is now known as sshnaidm|afk | 20:48 | |
*** slaweq has joined #tripleo | 20:48 | |
weshay | that's some sneaky shit right there https://review.opendev.org/#/c/711543/ | 20:49 |
mwhahaha | ok so we have 0 coverage on including a file from the current working directory | 20:49 |
weshay | lolz | 20:49 |
weshay | ha ha | 20:50 |
mwhahaha | so who knows when this broke | 20:50 |
weshay | just saw that too | 20:50 |
* weshay throws things at cloudnull | 20:50 | |
weshay | ah.. I wonder if rdo triggered | 20:50 |
openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade stable/train: Initital ffwd2 patch to automate OS upgrade. https://review.opendev.org/711552 | 20:50 |
weshay | but is overloaded | 20:50 |
weshay | mwhahaha, https://review.opendev.org/#/c/710356/ | 20:50 |
weshay | because that had two jobs run | 20:51 |
mwhahaha | weshay: i don't think we use --all-manageable in ci | 20:51 |
* mwhahaha shrugs | 20:51 | |
weshay | mwhahaha, ya.. it's running | 20:51 |
weshay | https://review.rdoproject.org/zuul/status | 20:51 |
cloudnull | I think there are differences due to 001 being novaless ? | 20:51 |
mwhahaha | https://bugs.launchpad.net/tripleo/+bug/1866249 | 20:53 |
openstack | Launchpad bug 1866249 in tripleo "using an environment file from the current working directory no longer works for overcloud deploy" [High,Triaged] | 20:53 |
* mwhahaha goes digging | 20:53 | |
*** cgoncalves has quit IRC | 20:53 | |
*** rcernin has joined #tripleo | 20:54 | |
openstackgerrit | Merged openstack/tripleo-ansible master: Revert "Modify the introspection playbook to simplify client interactions" https://review.opendev.org/711543 | 20:54 |
*** cgoncalves has joined #tripleo | 20:56 | |
*** jfrancoa has quit IRC | 20:57 | |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Revert "Revert "Modify the introspection playbook to simplify client interactions"" https://review.opendev.org/711553 | 20:57 |
* mwhahaha throws things at cloudnull | 20:59 | |
cloudnull | what did i do now ? | 21:00 |
mwhahaha | i think youre cleanup of the os.makedirs in https://review.opendev.org/#/c/671878/44/tripleoclient/v1/overcloud_deploy.py broke the local file loading | 21:00 |
* mwhahaha is testing theory | 21:00 | |
cloudnull | but that was months ago ? | 21:01 |
cloudnull | we're just seeing that now ? | 21:01 |
mwhahaha | yea cause we don't test non-fullyqualified files | 21:01 |
mwhahaha | in CI | 21:01 |
* mwhahaha flips tables | 21:01 | |
cloudnull | fun | 21:01 |
openstackgerrit | Merged openstack/tripleo-docs master: Fix a typo https://review.opendev.org/711346 | 21:02 |
*** ansmith has quit IRC | 21:02 | |
mwhahaha | this is why i always deploy by hand :/ | 21:02 |
mwhahaha | uncovers so many of these regressions | 21:02 |
mwhahaha | too bad i hadn't done it in some time | 21:03 |
cloudnull | something easy to fix ? | 21:04 |
*** cylopez has quit IRC | 21:04 | |
mwhahaha | yea revert all the changes in that file | 21:06 |
mwhahaha | specifically i think it's the _Deploy_tripleo_heat_templates_tmpdir part | 21:07 |
*** xek_ has quit IRC | 21:07 | |
mwhahaha | cause reverting that function seems to have fixed it | 21:07 |
mwhahaha | an hour and 20mins later i'm finally deploying an overcloud | 21:08 |
*** cgoncalves has quit IRC | 21:10 | |
*** cgoncalves has joined #tripleo | 21:11 | |
cloudnull | mwhahaha do you have a bug handy for the relative path thing ? | 21:16 |
*** alexmcleod has quit IRC | 21:17 | |
mwhahaha | i got it | 21:17 |
*** rcernin has quit IRC | 21:18 | |
*** dpawlik has quit IRC | 21:18 | |
openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Partial Revert "Replace ansible shell with python runner" https://review.opendev.org/711556 | 21:19 |
mwhahaha | cloudnull: -^ | 21:19 |
cloudnull | ++ | 21:20 |
openstackgerrit | Alex Schultz proposed openstack/tripleo-common stable/stein: Use config for layers https://review.opendev.org/710705 | 21:24 |
openstackgerrit | Flavio Fernandes proposed openstack/tripleo-heat-templates master: Add Octavia OVN Provider configuration https://review.opendev.org/707695 | 21:30 |
openstackgerrit | Steve Baker proposed openstack/tripleo-ansible master: module tripleo_baremetal_expand_roles https://review.opendev.org/706417 | 21:30 |
openstackgerrit | Steve Baker proposed openstack/tripleo-ansible master: module tripleo_baremetal_check_existing https://review.opendev.org/707043 | 21:30 |
openstackgerrit | Steve Baker proposed openstack/tripleo-ansible master: module tripleo_baremetal_populate_environment https://review.opendev.org/707295 | 21:30 |
openstackgerrit | Steve Baker proposed openstack/tripleo-ansible master: module metalsmith_instances https://review.opendev.org/706418 | 21:30 |
openstackgerrit | Steve Baker proposed openstack/tripleo-ansible master: Playbooks for overcloud node (un)provision https://review.opendev.org/707044 | 21:30 |
openstackgerrit | Steve Baker proposed openstack/tripleo-ansible master: Auto-detect default overcloud-full image https://review.opendev.org/711359 | 21:30 |
*** slaweq has quit IRC | 21:33 | |
openstackgerrit | Steve Baker proposed openstack/tripleo-quickstart-extras master: Do a local image copy if there is no glance https://review.opendev.org/711354 | 21:37 |
cloudnull | stevebaker - i gave your reviews another spin last night, at least for provisioning, it seemed to be working great. | 21:42 |
ekultails | stevebaker: Hey, good morning Steve. I was wondering if you've encountered anything similar to this / what your thoughts on this bug are (see my last comment): https://bugs.launchpad.net/tripleo/+bug/1866204 | 21:42 |
openstack | Launchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,Triaged] | 21:42 |
*** raildo has quit IRC | 21:47 | |
*** dsneddon has quit IRC | 21:48 | |
*** bfournie has quit IRC | 21:50 | |
*** cfontain_ has joined #tripleo | 21:58 | |
*** cfontain has quit IRC | 22:01 | |
*** cfontain_ has quit IRC | 22:03 | |
*** ansmith has joined #tripleo | 22:10 | |
openstackgerrit | wes hayutin proposed openstack/python-tripleoclient master: update layout for centos-8 containers-multinode https://review.opendev.org/711489 | 22:18 |
*** iurygregory has quit IRC | 22:18 | |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Remove invalid mistral envocation and references https://review.opendev.org/711562 | 22:24 |
*** jmasud has quit IRC | 22:26 | |
stevebaker | cloudnull: cool, thanks | 22:35 |
*** bfournie has joined #tripleo | 22:39 | |
stevebaker | ekultails: my dev environment is OVB, and I occasionally have node clean failures/timeouts which could be what you're seeing. I haven't looked into it though, I just manually "manage" -> "provide" to trigger clean again. | 22:41 |
stevebaker | ekultails: if you have any leads I could investigate the next time it happens | 22:41 |
*** tkajinam has joined #tripleo | 22:45 | |
ekultails | stevebaker: Cool, thanks. It's interesting that you occasionally run into it, too. I normally use pre-deployed servers so the error isn't something I've experienced before. I don't think we have much to go on besides the one CI failure. I'll get an e-mail thread going with you (and maybe some network folks?) once I can find out some more details. | 22:50 |
*** morazi has quit IRC | 22:50 | |
*** rh-jelabarre has quit IRC | 23:11 | |
*** cfontain_ has joined #tripleo | 23:18 | |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/train: Remove comment about tripleo_container_manage being experimental https://review.opendev.org/711563 | 23:21 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral when running the validate_nodes https://review.opendev.org/709552 | 23:21 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral when running the register_or_update workflow https://review.opendev.org/709583 | 23:21 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral from the undeploy_roles workflow https://review.opendev.org/709614 | 23:21 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral from the configure workflow https://review.opendev.org/709616 | 23:21 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral from the configure_manageable_nodes workflow https://review.opendev.org/709633 | 23:21 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove the mistral from the clean_.*_nodes workflow https://review.opendev.org/709755 | 23:21 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove the mistral from the create_raid_configuration workflow https://review.opendev.org/709858 | 23:21 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral from the bios_configuration workflows https://review.opendev.org/709861 | 23:21 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Client code to do node import with ansible instead of mistral https://review.opendev.org/704561 | 23:21 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral from the introspection workflows https://review.opendev.org/707348 | 23:21 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral when running the provide workflows https://review.opendev.org/710308 | 23:21 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove the mistral from the deploy_plan workflow https://review.opendev.org/710479 | 23:22 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove the mistral from the create_plan workflow https://review.opendev.org/711564 | 23:22 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Remove mistral when running the update_plan workflow https://review.opendev.org/711565 | 23:22 |
EmilienM | poor CI | 23:23 |
cloudnull | sadly had to rebase after some of the changes that went in today :( | 23:24 |
openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Ensure the os-cloud flag is used for CLI commands https://review.opendev.org/711260 | 23:29 |
openstackgerrit | Christophe Fontaine proposed openstack/tripleo-ansible master: 'tripleo_ovs_dpdk_lcore_list' should be optional https://review.opendev.org/711360 | 23:37 |
EmilienM | ade_lee, weshay https://review.opendev.org/#/c/711114/ | 23:45 |
stevebaker | cloudnull, EmilienM: hey what is the plan for ansiblizing the overcloud deploy command. baremetal (un)provision are standalone commands but are also optionally integrated into the overcloud deploy command. I'd like to just swap out the workflow call with a playbook call but that would violate our .v1 .v2 pattern | 23:49 |
EmilienM | stevebaker: you mean the tripleo-operator-ansible role for overcloud deploy? | 23:50 |
stevebaker | EmilienM: I don't know! | 23:51 |
openstackgerrit | Merged openstack/tripleo-upgrade stable/train: Replace Docker service by Podman in roles_data. https://review.opendev.org/710955 | 23:55 |
stevebaker | EmilienM: I mean what is the plan for "openstack overcloud deploy" not to call mistral? It looks like tripleo-operator-ansible/roles/tripleo_overcloud_deploy still calls it | 23:59 |
EmilienM | oh | 23:59 |
cloudnull | stevebaker for deploy_plan I just converted the workflow - https://review.opendev.org/#/c/710479/ - its now just executed all through the client, in a future iteration I'd like to make that, and others, a playbook, but my first priority was to get rid of mistral. | 23:59 |
EmilienM | yeah what cloudnull said | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!