| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO https://review.openstack.org/567224 | 00:00 |
|---|---|---|
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/560445 | 00:00 |
| *** artom has joined #tripleo | 00:00 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 00:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO https://review.openstack.org/604293 | 00:01 |
| *** mcornea has quit IRC | 00:02 | |
| *** wolverineav has joined #tripleo | 00:06 | |
| *** vinaykns has quit IRC | 00:09 | |
| *** wolverineav has quit IRC | 00:10 | |
| *** rlandy has quit IRC | 00:16 | |
| *** rascasoft has joined #tripleo | 00:17 | |
| *** jobewan has quit IRC | 00:18 | |
| *** wolverineav has joined #tripleo | 00:19 | |
| *** itlinux has joined #tripleo | 00:21 | |
| *** wolverineav has quit IRC | 00:23 | |
| *** rascasoft has quit IRC | 00:24 | |
| *** khyr0n has quit IRC | 00:31 | |
| *** gbarros has joined #tripleo | 00:36 | |
| *** wolverineav has joined #tripleo | 00:37 | |
| *** gbarros has quit IRC | 00:39 | |
| chandankumar | EmilienM: mwhahaha https://review.openstack.org/#/c/637838/ to unblock os_tempest gate | 00:45 |
| *** gbarros has joined #tripleo | 00:49 | |
| *** ade_lee_ has quit IRC | 00:53 | |
| *** sdake has joined #tripleo | 00:58 | |
| *** wolverineav has quit IRC | 01:03 | |
| openstackgerrit | Steve Baker proposed openstack/ansible-role-tripleo-modify-image master: Speed up execution time of yum_update.sh https://review.openstack.org/635398 | 01:05 |
| *** wolverineav has joined #tripleo | 01:07 | |
| *** wolverineav has quit IRC | 01:12 | |
| openstackgerrit | Rafael Folco proposed openstack-infra/tripleo-ci master: [DNM] Add py3/f28 support to tripleo container build https://review.openstack.org/636160 | 01:17 |
| *** _fragatina has quit IRC | 01:21 | |
| mwhahaha | chandankumar: commented | 01:26 |
| *** wolverineav has joined #tripleo | 01:27 | |
| *** thrash is now known as thrash|g0ne | 01:28 | |
| *** itlinux has quit IRC | 01:28 | |
| *** wolverineav has quit IRC | 01:32 | |
| chandankumar | mwhahaha: done! | 01:33 |
| *** wolverineav has joined #tripleo | 01:40 | |
| *** wolverineav has quit IRC | 01:44 | |
| *** rascasoft has joined #tripleo | 02:04 | |
| *** itlinux has joined #tripleo | 02:09 | |
| *** rascasoft has quit IRC | 02:12 | |
| *** hamzy has joined #tripleo | 02:13 | |
| *** cfontain has joined #tripleo | 02:14 | |
| *** wolverineav has joined #tripleo | 02:17 | |
| *** sai_p_ has quit IRC | 02:19 | |
| *** wolverineav has quit IRC | 02:22 | |
| *** mschuppert has quit IRC | 02:22 | |
| *** wolverineav has joined #tripleo | 02:25 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Removes all pacemaker from scenario-standalone 1-3 environments https://review.openstack.org/636563 | 02:30 |
| *** wolverineav has quit IRC | 02:30 | |
| *** sdake has quit IRC | 02:36 | |
| openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/tripleo-common stable/rocky: Stops growth of massive dentry cache growth https://review.openstack.org/638051 | 02:38 |
| openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/tripleo-common stable/queens: Stops growth of massive dentry cache growth https://review.openstack.org/636660 | 02:43 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Only include neutron::designate class in neutron server https://review.openstack.org/629645 | 02:44 |
| *** cfontain has quit IRC | 02:47 | |
| openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/tripleo-heat-templates stable/queens: Remove ties between ceilometer and panko https://review.openstack.org/636657 | 02:56 |
| *** tzumainn has quit IRC | 02:58 | |
| *** psachin has joined #tripleo | 03:05 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: container image build: introduce --use-buildah https://review.openstack.org/634322 | 03:07 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Remove host-config-and-reboot interface https://review.openstack.org/637936 | 03:12 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add missing RoleParameters and ServiceNames https://review.openstack.org/637569 | 03:12 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Add fedora-28 support to container-build job on fedora-28 with Buildah https://review.openstack.org/635991 | 03:13 |
| openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/tripleo-heat-templates stable/queens: Remove ties between ceilometer and panko https://review.openstack.org/636657 | 03:14 |
| *** apetrich has quit IRC | 03:14 | |
| *** agopi has joined #tripleo | 03:16 | |
| *** cfontain has joined #tripleo | 03:17 | |
| *** hamzy has quit IRC | 03:20 | |
| openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/tripleo-heat-templates stable/rocky: Remove ties between ceilometer and panko https://review.openstack.org/638068 | 03:28 |
| *** skramaja has joined #tripleo | 03:43 | |
| *** rascasoft has joined #tripleo | 03:45 | |
| *** itlinux has quit IRC | 03:45 | |
| *** skramaja_ has joined #tripleo | 03:48 | |
| *** skramaja has quit IRC | 03:48 | |
| *** cfontain has quit IRC | 03:50 | |
| *** rascasoft has quit IRC | 03:53 | |
| *** udesale has joined #tripleo | 03:56 | |
| *** itlinux has joined #tripleo | 04:02 | |
| *** gbarros has quit IRC | 04:06 | |
| *** ykarel|away has joined #tripleo | 04:08 | |
| *** ykarel|away is now known as ykarel | 04:09 | |
| *** itlinux has quit IRC | 04:10 | |
| *** itlinux has joined #tripleo | 04:13 | |
| *** wolverineav has joined #tripleo | 04:15 | |
| *** hamzy has joined #tripleo | 04:16 | |
| *** itlinux has quit IRC | 04:18 | |
| *** janki has joined #tripleo | 04:22 | |
| *** itlinux has joined #tripleo | 04:23 | |
| *** ramishra has joined #tripleo | 04:23 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Revert "Remove host-config-and-reboot interface" https://review.openstack.org/638074 | 04:26 |
| *** itlinux has quit IRC | 04:28 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Revert "Remove host-config-and-reboot interface" https://review.openstack.org/638074 | 04:29 |
| *** itlinux has joined #tripleo | 04:33 | |
| *** ade_lee has joined #tripleo | 04:46 | |
| *** ykarel has quit IRC | 04:48 | |
| *** ade_lee has quit IRC | 04:51 | |
| *** ade_lee has joined #tripleo | 04:51 | |
| *** owalsh_ has joined #tripleo | 04:54 | |
| *** wolverineav has quit IRC | 04:55 | |
| *** owalsh has quit IRC | 04:57 | |
| *** ykarel has joined #tripleo | 05:05 | |
| *** shyamb has joined #tripleo | 05:07 | |
| *** shyamb has quit IRC | 05:12 | |
| *** itlinux_ has joined #tripleo | 05:25 | |
| *** itlinux has quit IRC | 05:27 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Remove unused resources of PreNetworkConfig for NFV https://review.openstack.org/638081 | 05:29 |
| *** rascasoft has joined #tripleo | 05:30 | |
| *** shyamb has joined #tripleo | 05:34 | |
| ykarel | stevebaker, Thanks for updating, i forgot adding --provides | 05:37 |
| ykarel | it seems working | 05:37 |
| *** rascasoft has quit IRC | 05:38 | |
| *** itlinux has joined #tripleo | 05:40 | |
| *** itlinux_ has quit IRC | 05:43 | |
| *** _fragatina has joined #tripleo | 05:44 | |
| *** _fragatina has quit IRC | 05:45 | |
| *** _fragatina has joined #tripleo | 05:46 | |
| *** cfontain has joined #tripleo | 05:48 | |
| *** itlinux has quit IRC | 05:56 | |
| *** tkajinam_ has joined #tripleo | 05:57 | |
| shyamb | Hi | 05:58 |
| *** pdeore has joined #tripleo | 05:58 | |
| shyamb | ykarel: After some fixes, I am able to get undercloud deployed successfully | 05:58 |
| shyamb | thanks a lot | 05:58 |
| ykarel | shyamb, cool | 05:58 |
| *** tkajinam has quit IRC | 05:59 | |
| *** dmacpher_ has quit IRC | 06:00 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: TLS everywhere: Set post-save command for httpd https://review.openstack.org/636874 | 06:01 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: TLS everywhere: Set post-save command for RabbitMQ https://review.openstack.org/636875 | 06:01 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: TLS everywhere: Set post-save command for redis https://review.openstack.org/636876 | 06:01 |
| *** ratailor has joined #tripleo | 06:05 | |
| *** shyamb has quit IRC | 06:05 | |
| *** shyamb has joined #tripleo | 06:06 | |
| *** wolverineav has joined #tripleo | 06:07 | |
| *** rascasoft has joined #tripleo | 06:09 | |
| *** wolverineav has quit IRC | 06:11 | |
| *** rascasoft has quit IRC | 06:17 | |
| *** cfontain has quit IRC | 06:20 | |
| *** itlinux has joined #tripleo | 06:23 | |
| Tengu | hello there | 06:24 |
| *** sdake has joined #tripleo | 06:26 | |
| *** sdake has quit IRC | 06:27 | |
| *** jtomasek has joined #tripleo | 06:28 | |
| *** itlinux has quit IRC | 06:28 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: WIP Be able to know when we are running inside a minor update workflow https://review.openstack.org/635725 | 06:29 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: WIP Do not restart bundles during a minor update https://review.openstack.org/635731 | 06:29 |
| *** saneax has joined #tripleo | 06:31 | |
| *** itlinux has joined #tripleo | 06:33 | |
| *** yprokule has joined #tripleo | 06:40 | |
| *** ksambor has joined #tripleo | 06:42 | |
| *** jfrancoa has joined #tripleo | 06:46 | |
| *** hjensas has quit IRC | 06:48 | |
| *** quiquell|off is now known as quiquell|rover | 06:49 | |
| *** itlinux has quit IRC | 06:50 | |
| *** cfontain has joined #tripleo | 06:52 | |
| *** _fragatina has quit IRC | 06:53 | |
| *** rascasoft has joined #tripleo | 06:54 | |
| *** _fragatina has joined #tripleo | 06:54 | |
| *** shyamb has quit IRC | 06:58 | |
| *** rascasoft has quit IRC | 07:00 | |
| *** _fragatina has quit IRC | 07:00 | |
| *** itlinux has joined #tripleo | 07:03 | |
| *** shyamb has joined #tripleo | 07:06 | |
| *** itlinux has quit IRC | 07:07 | |
| *** mschuppert has joined #tripleo | 07:09 | |
| *** rascasoft has joined #tripleo | 07:09 | |
| *** avivgta has joined #tripleo | 07:13 | |
| *** aufi has joined #tripleo | 07:13 | |
| shyamb | by default novajoin-ipa-setup command is not available on undercloud | 07:14 |
| shyamb | anyidea that which rpm package provides this command? | 07:14 |
| *** itlinux has joined #tripleo | 07:25 | |
| openstackgerrit | waleed mousa proposed openstack/os-net-config master: Allow adding sriov_pf over ovs_bridge, linux_bond and ovs_bond https://review.openstack.org/634648 | 07:27 |
| openstackgerrit | Hamdy Khader proposed openstack/tripleo-heat-templates master: Add Mellanox SDN ML2 template for a dockerized service https://review.openstack.org/637829 | 07:27 |
| quiquell|rover | EmilienM: Still MANIFEST_BLOB_UNKNOWN at gates http://logs.openstack.org/38/637838/2/gate/tripleo-ci-centos-7-standalone/f26aa60/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 07:27 |
| *** ccamacho has quit IRC | 07:29 | |
| *** ccamacho has joined #tripleo | 07:29 | |
| *** udesale has quit IRC | 07:34 | |
| *** udesale has joined #tripleo | 07:35 | |
| *** quiquell|rover is now known as quique|rover|brb | 07:36 | |
| *** itlinux has quit IRC | 07:36 | |
| *** apetrich has joined #tripleo | 07:37 | |
| quique|rover|brb | EmilienM: I see the docker-distribution replacement merged https://review.openstack.org/#/c/624830/ | 07:37 |
| *** ratailor_ has joined #tripleo | 07:40 | |
| *** hjensas has joined #tripleo | 07:41 | |
| *** ratailor has quit IRC | 07:43 | |
| *** cfontain has quit IRC | 07:45 | |
| *** cfontain has joined #tripleo | 07:45 | |
| *** cfontain has joined #tripleo | 07:46 | |
| *** ykarel is now known as ykarel|lunch | 07:49 | |
| *** shyamb has quit IRC | 07:50 | |
| *** cfontain has quit IRC | 07:51 | |
| *** hberaud|gone is now known as hberaud | 07:54 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Add functionality to manage KSM on compute nodes per role https://review.openstack.org/631021 | 07:55 |
| *** itlinux has joined #tripleo | 07:56 | |
| *** paramite has joined #tripleo | 07:58 | |
| *** chandankumar is now known as chkumar|ruck | 07:59 | |
| *** pcaruana has joined #tripleo | 07:59 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 08:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO https://review.openstack.org/604293 | 08:00 |
| *** itlinux has quit IRC | 08:01 | |
| *** itlinux has joined #tripleo | 08:03 | |
| *** morazi has joined #tripleo | 08:04 | |
| *** quique|rover|brb is now known as quiquell|rover | 08:04 | |
| *** kopecmartin|off is now known as kopecmartin | 08:09 | |
| *** jchhatbar has joined #tripleo | 08:09 | |
| *** jchhatbar has quit IRC | 08:10 | |
| *** jchhatbar has joined #tripleo | 08:11 | |
| *** janki has quit IRC | 08:11 | |
| *** tkajinam_ has quit IRC | 08:14 | |
| *** itlinux has quit IRC | 08:16 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-common master: Run keypair creating as root https://review.openstack.org/636844 | 08:20 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-common master: Run keypair creating as root https://review.openstack.org/636844 | 08:21 |
| *** itlinux has joined #tripleo | 08:23 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-heat-templates master: Use tripleo-admin as octavia groups_vars owner https://review.openstack.org/636571 | 08:23 |
| *** cschwede has joined #tripleo | 08:23 | |
| Tengu | chkumar|ruck: heya! do you want me to take over your patch making the CI fail on failed services|containers ? I now have a pretty good idea for the latter (commented your change) | 08:24 |
| *** jpich has joined #tripleo | 08:24 | |
| *** cylopez has joined #tripleo | 08:24 | |
| chkumar|ruck | Tengu: sure, feel free to take it | 08:24 |
| Tengu | chkumar|ruck: ok :) | 08:25 |
| openstackgerrit | waleed mousa proposed openstack/puppet-tripleo master: Adding a udev rule to save sriov interface name before configuring switchdev https://review.openstack.org/638103 | 08:25 |
| *** itlinux has quit IRC | 08:28 | |
| *** waleedm has joined #tripleo | 08:28 | |
| *** skramaja_ is now known as skramaja | 08:28 | |
| *** rcernin has quit IRC | 08:29 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-common master: Run keypair creating as root https://review.openstack.org/636844 | 08:31 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-heat-templates master: Use tripleo-admin as octavia groups_vars owner https://review.openstack.org/636571 | 08:31 |
| *** dtantsur|afk is now known as dtantsur | 08:34 | |
| *** avivgta has quit IRC | 08:38 | |
| quiquell|rover | EmilienM: MANIFEST_BLOB_UNKNOWN massive here https://review.openstack.org/#/c/636232/ | 08:41 |
| quiquell|rover | EmilienM: all standalone scenarios jobs at gates :-( | 08:41 |
| quiquell|rover | EmilienM: I am going to open a bug to track that and try to logstash it | 08:42 |
| *** tosky has joined #tripleo | 08:43 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-quickstart-extras master: Raise an error if a service or container is failed https://review.openstack.org/637729 | 08:45 |
| Tengu | chkumar|ruck: -^^ | 08:46 |
| Tengu | let's see how it goes in the CI. | 08:46 |
| Tengu | damn. the assert was wrong -.-'. | 08:47 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-quickstart-extras master: Raise an error if a service or container is failed https://review.openstack.org/637729 | 08:47 |
| *** owalsh_ is now known as owalsh | 08:48 | |
| *** holser_ has joined #tripleo | 08:50 | |
| *** ykarel|lunch is now known as ykarel | 08:51 | |
| *** shyamb has joined #tripleo | 08:52 | |
| *** holser_ has quit IRC | 08:53 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Change build-container jobs definition to allow reuse https://review.openstack.org/638108 | 08:54 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Change build-container jobs definition to allow reuse https://review.openstack.org/638108 | 08:55 |
| chkumar|ruck | Tengu: thanks! | 08:56 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Change build-container jobs definition to allow reuse https://review.openstack.org/638108 | 08:57 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Change build-container jobs definition to allow reuse https://review.openstack.org/638108 | 08:59 |
| *** holser_ has joined #tripleo | 09:00 | |
| *** jpena|off is now known as jpena | 09:00 | |
| *** gfidente has joined #tripleo | 09:01 | |
| dalvarez | larsks: ping re. baremetal nodes + metadata access | 09:01 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Change build-container jobs definition to allow reuse https://review.openstack.org/638108 | 09:02 |
| dalvarez | larsks: lucasagomes ideally, instances should reach 169.254.169.254 through the OVN metadata port and never reach qdhcp namespace. If you have a setup we can take a look...I sent a job yesterday and seems to be working: https://review.openstack.org/#/c/637863/ | 09:04 |
| *** saneax has quit IRC | 09:04 | |
| dalvarez | larsks: lucasagomes ^ i checked ovn metadata agent logs in tempest and requests arrive to it and are properly answered: http://logs.openstack.org/63/637863/2/check/networking-ovn-tempest-dsvm-ovs-release/3d204d3/controller/logs/screen-networking-ovn-metadata-agent.txt.gz#_Feb_19_16_48_53_162769 | 09:05 |
| *** saneax has joined #tripleo | 09:05 | |
| dalvarez | larsks: this patchset i sent spawned neutron dhcp agent alongside ovn services... ideally instances should have a static route to reach 169.254.169.254 through *ovn* metadata port | 09:06 |
| dalvarez | larsks: but i think that without that patch, metadata should be broken if neutron dhcp agent is deployed | 09:06 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Change build-container jobs definition to allow reuse https://review.openstack.org/638108 | 09:07 |
| *** bogdando has joined #tripleo | 09:08 | |
| *** cfontain has joined #tripleo | 09:11 | |
| *** ykarel is now known as ykarel|afk | 09:13 | |
| openstackgerrit | Natal Ngétal proposed openstack/python-tripleoclient master: Add support of extra variables. https://review.openstack.org/637530 | 09:13 |
| *** ykarel|afk is now known as ykarel | 09:17 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: [DNM] testing telemetry tests https://review.openstack.org/638112 | 09:27 |
| *** moshele has joined #tripleo | 09:30 | |
| *** pbandark has joined #tripleo | 09:32 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Migrating Undercloud playbooks to their own Ansible Role https://review.openstack.org/636958 | 09:33 |
| *** avivgta has joined #tripleo | 09:36 | |
| *** Skyraven has joined #tripleo | 09:36 | |
| Skyraven | hi there, I have a funky question...and no clear answer:( if using Tripleo+OSP13, I have a NodeDataLookup section which for each Ceph node specifies the /dev/path...to the disks | 09:37 |
| *** kopecmartin is now known as kopecmartin|afk | 09:37 | |
| Skyraven | and then for ceph-ansible I see it needs defining lvm_volumes... how do I define there? | 09:37 |
| Skyraven | all examples there are with /dev/sda...but then it would never use the mapping from NodeDataLookup | 09:38 |
| Skyraven | how are these 2 things put together | 09:38 |
| *** derekh has joined #tripleo | 09:45 | |
| *** panda|off is now known as panda | 09:46 | |
| *** shyamb has quit IRC | 09:46 | |
| Tengu | quiquell|rover: heya! is it a know issue? http://logs.openstack.org/29/637729/4/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/896374b/logs/undercloud/home/zuul/undercloud_install.log.gz | 09:47 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Make build-containers job definition reusable https://review.openstack.org/638108 | 09:47 |
| quiquell|rover | Tengu: undercloud_hostname ? | 09:47 |
| Tengu | so OK, non-voting, but still | 09:47 |
| Tengu | quiquell|rover: yeah, not-fully-qualified hostname | 09:47 |
| quiquell|rover | Tengu: yep we are fixing it | 09:47 |
| quiquell|rover | Tengu: missing var at fs050 | 09:48 |
| Tengu | cool :) | 09:48 |
| moshele | skramaja: hi | 09:48 |
| quiquell|rover | jfrancoa: ^ Tengu affected too | 09:49 |
| Tengu | quiquell|rover: btw: https://review.openstack.org/#/c/637729/ I've done some modifications in order to make it fail in case of failed service/container. | 09:49 |
| Tengu | quiquell|rover: guess we'll covera wide area with that. | 09:49 |
| jfrancoa | Tengu: I'm going to propose a patch to fix it, we are missing +undercloud_undercloud_hostname in some featuresets | 09:50 |
| Tengu | jfrancoa++ :) | 09:50 |
| quiquell|rover | Tengu: implemented or with an and :-) | 09:52 |
| *** shyamb has joined #tripleo | 09:53 | |
| *** zaneb has quit IRC | 09:53 | |
| quiquell|rover | Tengu: well at least is great tool for ruck/rover | 09:53 |
| quiquell|rover | +2 | 09:53 |
| * Tengu doesn't like boolean algebra ;) | 09:53 | |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Append qemu-img to Mistral executor package list https://review.openstack.org/637515 | 09:54 |
| quiquell|rover | Tengu: well boolean algebra + ansible is not a good combination | 09:54 |
| Tengu | but yeah. it's funky. I didn't know about the "assert" | 09:54 |
| Tengu | hehe | 09:54 |
| Tengu | yeah. the list is an "and" - so you need to tweak terms. | 09:54 |
| Tengu | guess I did it right. | 09:54 |
| quiquell|rover | Yep :-) | 09:55 |
| Tengu | grep returns 1 in case it does NOT match - wc -l just returns the number of line. so if non-0 AND 0, we're good. | 09:55 |
| quiquell|rover | Maybe we can skip - and do normal or just for readability | 09:55 |
| quiquell|rover | And wrap lines | 09:55 |
| Tengu | well... readability is a matter of perception in that case in fact. | 09:56 |
| Tengu | I find it more readable with the "and" :D | 09:56 |
| Tengu | but I'm probably deviate. | 09:56 |
| Tengu | or crazy. | 09:56 |
| quiquell|rover | ack na just saying all good I put +2 there | 09:56 |
| Tengu | hopefully CI will be happy as well. RDO doesn't look that happy right now :(. | 09:56 |
| openstackgerrit | Jakub Libosvar proposed openstack/tripleo-heat-templates master: ml2-ansible: Set api_max_retries when net-ansible is used https://review.openstack.org/638119 | 09:57 |
| shyamb | Hi | 09:59 |
| shyamb | undercloud is failing at kerberose keystab ste | 09:59 |
| shyamb | http://paste.openstack.org/show/745450/ | 09:59 |
| quiquell|rover | Tengu: well is ci.centos jobs | 10:00 |
| shyamb | Do I need to set service_principle parameter in undercloud conf? | 10:01 |
| *** suuuper has joined #tripleo | 10:01 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Make build-containers job definition reusable https://review.openstack.org/638108 | 10:02 |
| *** shyamb has quit IRC | 10:06 | |
| *** shyamb has joined #tripleo | 10:06 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart master: Add missing undercloud_conf parameters. https://review.openstack.org/638122 | 10:06 |
| *** xek has joined #tripleo | 10:08 | |
| *** shyamb has quit IRC | 10:12 | |
| jaosorior | bandini: got a chance to review these https://review.openstack.org/#/q/topic:certmonger_post_save+status:open ? | 10:13 |
| xek | mwhahaha, bandini, morning :) can you +2 https://review.openstack.org/636877 and the following 2 patches? | 10:13 |
| jaosorior | skramaja: ^^ | 10:13 |
| *** matbu has quit IRC | 10:15 | |
| *** matbu has joined #tripleo | 10:19 | |
| *** pdeore has quit IRC | 10:27 | |
| *** moshele has quit IRC | 10:28 | |
| *** wolverineav has joined #tripleo | 10:29 | |
| *** aufi has quit IRC | 10:31 | |
| *** wolverineav has quit IRC | 10:33 | |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: Enable OVN baremetal by default with Ironic service https://review.openstack.org/637871 | 10:37 |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-docs master: Correct environment path for installing Ironic https://review.openstack.org/638126 | 10:37 |
| quiquell|rover | chkumar|ruck: this rings any bell ? http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/home/zuul/tempest.log.txt.gz | 10:44 |
| *** pdeore has joined #tripleo | 10:49 | |
| chkumar|ruck | quiquell|rover: appears new | 10:52 |
| chkumar|ruck | quiquell|rover: another more issue there http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/var/log/extra/errors.txt.gz#_2019-02-20_09_55_28_236 | 10:52 |
| quiquell|rover | chkumar|ruck: also http://logs.openstack.org/45/560445/255/check/tripleo-ci-centos-7-standalone/3a6595c/logs/undercloud/var/log/extra/errors.txt.gz | 10:53 |
| quiquell|rover | chkumar|ruck: KeyError: 'neutron:ovn-metadata-sb-cfg' | 10:53 |
| quiquell|rover | chkumar|ruck: this is related to the change from flat to geneva ? | 10:53 |
| *** shyamb has joined #tripleo | 10:53 | |
| chkumar|ruck | quiquell|rover: we have not changed anything in validate-tempest role | 10:53 |
| *** aufi has joined #tripleo | 10:54 | |
| chkumar|ruck | quiquell|rover: as they are only creating public network and private network is handled by tempestconf | 10:54 |
| chkumar|ruck | quiquell|rover: nope | 10:54 |
| *** ccamacho has quit IRC | 10:54 | |
| quiquell|rover | chkumar|ruck: I have si this duplicate option: barbican at a passing one | 10:54 |
| chkumar|ruck | quiquell|rover: it is also an issue, we need to fix that | 10:55 |
| quiquell|rover | ack | 10:55 |
| chkumar|ruck | quiquell|rover: let me find more info related to that subnet | 10:55 |
| Tengu | hmmmm. | 10:56 |
| Tengu | quiquell|rover: apparently my patch isn't working for some reason. | 10:56 |
| quiquell|rover | chkumar|ruck: in the standalone job issue I see timeouts at nova compite | 10:56 |
| Tengu | quiquell|rover: i.e. http://logs.openstack.org/29/637729/4/check/tripleo-ci-centos-7-undercloud-containers/d9fa103/logs/undercloud/var/log/extra/failed_containers.log.gz lists bad containers, while it's a success. | 10:57 |
| jaosorior | bandini: thanks | 10:57 |
| bandini | np | 10:57 |
| Tengu | chkumar|ruck: quiquell|rover damn... errors in the log collection aren't fatal for the CI: http://logs.openstack.org/29/637729/4/check/tripleo-ci-centos-7-undercloud-containers/d9fa103/logs/quickstart_collect_logs.log | 10:58 |
| Tengu | we have to move that somewhere else. | 10:58 |
| quiquell|rover | Tengu: you will need to create new POST for that | 10:58 |
| Tengu | quiquell|rover: care to guide me? | 10:59 |
| *** Skyraven has quit IRC | 10:59 | |
| aedc | Hi, is there a way to separately define DNS and IdM in undercloud.conf for TLS everywhere ? | 10:59 |
| quiquell|rover | But it's wair let me check why the failure is not propagated | 10:59 |
| Tengu | quiquell|rover: ack :) | 11:00 |
| quiquell|rover | Tengu: ahh i see | 11:00 |
| quiquell|rover | http://logs.openstack.org/29/637729/4/check/tripleo-ci-centos-7-undercloud-containers/d9fa103/logs/collect_logs.sh | 11:00 |
| quiquell|rover | echo "WARNING: quickstart collect-logs failed, check quickstart_collectlogs.log for details" | 11:00 |
| quiquell|rover | Tengu: maybe you can put the check out of the role and inside this script | 11:01 |
| *** moshele has joined #tripleo | 11:01 | |
| quiquell|rover | At the end or something | 11:01 |
| quiquell|rover | chkumar|ruck: do you have time for sync ? | 11:02 |
| Tengu | quiquell|rover: hmm ok. | 11:02 |
| chkumar|ruck | quiquell|rover: yup sure | 11:02 |
| chkumar|ruck | quiquell|rover: do we need right now | 11:02 |
| Tengu | quiquell|rover: any idea where that script comes from? | 11:02 |
| quiquell|rover | chkumar|ruck: Was just looking at the wiki, just let me know why you have an idle cycle | 11:03 |
| quiquell|rover | Tengu: Maybe better at collect logs you just collect the container logs as you have already do | 11:05 |
| chkumar|ruck | quiquell|rover: sorry which wiki, idle cycle did not get that | 11:05 |
| quiquell|rover | Tengu: and put new post that runs after collect logs that fail | 11:05 |
| quiquell|rover | chkumar|ruck: rr wiki I mean | 11:05 |
| Tengu | quiquell|rover: I'd love to, but that's out of my current knowledge. Will need some help for that :). | 11:05 |
| quiquell|rover | Tengu: https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-ci/post.yaml | 11:06 |
| Tengu | jrist: heya! I think I won't be able to join the mtg for UI - have another one with my team, it happens once per sprint.... | 11:06 |
| Tengu | quiquell|rover: ok, so the assert part shoud be moved in that post.yaml right? | 11:06 |
| quiquell|rover | Tengu: we can try | 11:07 |
| quiquell|rover | Tengu: but it's all about collect logs | 11:07 |
| quiquell|rover | Tengu: maybe different post | 11:07 |
| Tengu | hmm. | 11:07 |
| quiquell|rover | Tengu: but then we have to change job definitions | 11:07 |
| chkumar|ruck | quiquell|rover: ok, so I am done with telemetry one | 11:07 |
| quiquell|rover | :-/ | 11:07 |
| Tengu | well, lunch time, I'll check that a bit later. | 11:07 |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates stable/rocky: Fail to live migration if instance has NUMA topology https://review.openstack.org/638131 | 11:07 |
| Tengu | #hungry ;). | 11:07 |
| Tengu | and working while hungry is bad. | 11:08 |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates stable/queens: Fail to live migration if instance has NUMA topology https://review.openstack.org/638132 | 11:08 |
| quiquell|rover | chkumar|ruck: so, what new stuff do we have ? I see timeout at compute at noop | 11:08 |
| chkumar|ruck | quiquell|rover: only one bug left https://bugs.launchpad.net/tripleo/+bug/1813224 left thinking about looking into it | 11:08 |
| openstack | Launchpad bug 1813224 in tripleo "fedora28 standalone failing on tempest" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 11:08 |
| quiquell|rover | chkumar|ruck: and about new stuff appearing now ? | 11:08 |
| chkumar|ruck | quiquell|rover: looking the cockpit right now | 11:10 |
| quiquell|rover | chkumar|ruck: we have failrues at gates and noop | 11:10 |
| quiquell|rover | noop has not finished yet | 11:10 |
| *** xek_ has joined #tripleo | 11:11 | |
| *** cfontain has quit IRC | 11:11 | |
| *** ccamacho has joined #tripleo | 11:12 | |
| chkumar|ruck | quiquell|rover: [ERROR] stdout: ', u'Determining IP information for eth2... failed.', u' -> http://logs.rdoproject.org/45/560445/255/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/2ec66f8/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 11:12 |
| *** xek_ has quit IRC | 11:12 | |
| *** xek_ has joined #tripleo | 11:12 | |
| *** xek has quit IRC | 11:13 | |
| chkumar|ruck | quiquell|rover: did we get the reason of tempest failure for noop standalone? | 11:13 |
| quiquell|rover | chkumar|ruck: ^ http://logs.openstack.org/45/560445/255/check/tripleo-ci-centos-7-standalone/3a6595c/logs/undercloud/var/log/extra/errors.txt.gz#_2019-02-20_09_27_28_832 | 11:13 |
| *** udesale has quit IRC | 11:14 | |
| quiquell|rover | cannot get change ports at neutron I thin | 11:14 |
| chkumar|ruck | quiquell|rover: looking for more info | 11:16 |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: Enable OVN baremetal by default with Ironic service https://review.openstack.org/637871 | 11:16 |
| *** galaxyyahoo has quit IRC | 11:16 | |
| *** galaxyyahoo has joined #tripleo | 11:16 | |
| *** xek_ has quit IRC | 11:17 | |
| chkumar|ruck | quiquell|rover: df92f96-c871-49aa-844d-c90b511df609 is not seen in tempest.log | 11:18 |
| *** xek has joined #tripleo | 11:18 | |
| chkumar|ruck | * df92f96-c871-49aa-844d-c90b511df609 port | 11:18 |
| *** holser_ is now known as holser|lunch | 11:19 | |
| chkumar|ruck | quiquell|rover: got it,you are correct Build of instance ef70b6b4-eba3-452a-ba1e-f1f795eecc18 was re-scheduled: Request to http://192.168.24.1:9696/v2.0/ports/9df92f96-c871-49aa-844d-c90b511df609 timed out\n' | 11:19 |
| chkumar|ruck | slaweq: Hello | 11:19 |
| *** florianf has joined #tripleo | 11:21 | |
| jaosorior | sshnaidm: hey, how's it going? | 11:22 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Remove tasks that stop and disable Horizon services. https://review.openstack.org/638134 | 11:22 |
| sshnaidm | jaosorior, passing! \o/ | 11:22 |
| jaosorior | sshnaidm: whaaaa | 11:22 |
| jaosorior | where? how? | 11:22 |
| slaweq | chkumar|ruck: hi | 11:22 |
| sshnaidm | jaosorior, look at that: https://review.openstack.org/#/c/637800/ | 11:22 |
| chkumar|ruck | slaweq: can you take a look at this tempest error: http://logs.openstack.org/45/560445/255/check/tripleo-ci-centos-7-standalone/3a6595c/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-02-20_09_27_58 and | 11:22 |
| jaosorior | sshnaidm: dude! nice!!! | 11:23 |
| jaosorior | sshnaidm: what's next? | 11:23 |
| sshnaidm | jaosorior, I'm configuring it to run on all tht patches, and ovb as well.. but rdo ci doesn't feel good right now | 11:23 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Remove tasks that stop and disable Horizon services. https://review.openstack.org/638134 | 11:23 |
| jaosorior | sshnaidm: :( | 11:23 |
| sshnaidm | jaosorior, when it will recover I merge all of them | 11:23 |
| chkumar|ruck | slaweq: here is the logs of that server http://logs.openstack.org/45/560445/255/check/tripleo-ci-centos-7-standalone/3a6595c/logs/undercloud/var/log/extra/errors.txt.gz#_2019-02-20_09_27_28_850 | 11:23 |
| *** ratailor_ has quit IRC | 11:23 | |
| jaosorior | sshnaidm: any patches you want me to +2 ? | 11:24 |
| sshnaidm | jaosorior, no, everything is ready atm | 11:24 |
| sshnaidm | jaosorior, any other repo except of tht to run it? | 11:24 |
| jaosorior | sshnaidm: puppet-tripleo | 11:24 |
| sshnaidm | jaosorior, ok | 11:24 |
| jaosorior | sshnaidm: and the novajoin repo | 11:24 |
| sshnaidm | jaosorior, cool, will do | 11:25 |
| jaosorior | sshnaidm: thanks a lot dude! | 11:25 |
| sshnaidm | jaosorior, later we'll talk what to do next - tempest tests, other topology, etc, etc | 11:25 |
| sshnaidm | jaosorior, sure, np | 11:25 |
| jaosorior | sshnaidm: so, it'll be scenario039, right? | 11:25 |
| sshnaidm | jaosorior, yep | 11:25 |
| jaosorior | cool! | 11:25 |
| sshnaidm | jaosorior, it'll we be first ptg when we don't do tls job.. it breaks a tradition :( | 11:26 |
| jaosorior | lol | 11:26 |
| sshnaidm | jaosorior, but I'm sure we'll find something.. | 11:26 |
| jaosorior | sshnaidm: well, I also won't be there next PTG :( | 11:26 |
| sshnaidm | jaosorior, really? | 11:27 |
| jaosorior | yep | 11:27 |
| sshnaidm | jaosorior, then how will you provide me a beer for tls job? o_O | 11:28 |
| jaosorior | sshnaidm: I'll outsource the beer getting to one of my team mates | 11:28 |
| jaosorior | sshnaidm: don't worry, your beers are secured :P | 11:28 |
| sshnaidm | jaosorior, ok, deal :D | 11:28 |
| chkumar|ruck | quiquell|rover: regarding this fs01 failure https://logs.rdoproject.org/45/560445/255/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/2ec66f8/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-02-20_09_32_29 | 11:34 |
| chkumar|ruck | quiquell|rover: 2019-02-20 09:32:29 | [2019/02/20 09:29:42 AM] [ERROR] stdout: ERROR : [/etc/sysconfig/network-scripts/ifup-eth] Error, some other host (76:D2:05:65:88:D6) already uses address 172.17.0.66. | 11:34 |
| quiquell|rover | chkumar|ruck: the "Only volume-backed servers are allowed for flavors with zero disk" is as promotion-blcoker | 11:34 |
| quiquell|rover | http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/854df3d/logs/tempest.html | 11:35 |
| chkumar|ruck | quiquell|rover: it might be related to rdo cloud, I grepped for this IP, but the port associted does not exist might got deted | 11:35 |
| chkumar|ruck | quiquell|rover: above issue is fixed already with https://review.openstack.org/#/c/637679/ | 11:35 |
| quiquell|rover | chkumar|ruck: 24 hours ago... | 11:36 |
| quiquell|rover | chkumar|ruck: We have being unlucky we have to wait for next promotion spin | 11:37 |
| *** wolverineav has joined #tripleo | 11:37 | |
| chkumar|ruck | quiquell|rover: http://python2-tempestconf-2.0.1-0.20190115181339.f9b3c05.el7.noarch/ is the older rpm | 11:37 |
| quiquell|rover | chkumar|ruck: ack | 11:37 |
| chkumar|ruck | quiquell|rover: regarding fs01 I have added the info on etherpad | 11:37 |
| *** xek has quit IRC | 11:38 | |
| quiquell|rover | chkumar|ruck: flatten has fail again | 11:41 |
| quiquell|rover | damn | 11:41 |
| *** wolverineav has quit IRC | 11:41 | |
| chkumar|ruck | it has multiple job failures | 11:41 |
| quiquell|rover | chkumar|ruck: It's unrelated | 11:42 |
| quiquell|rover | chkumar|ruck: another timeout here http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/var/log/extra/errors.txt.gz | 11:44 |
| quiquell|rover | chkumar|ruck: this is from the flatten review we need to close bug ^ | 11:44 |
| chkumar|ruck | quiquell|rover: Tengu + podman stats --all --no-stream | 11:44 |
| chkumar|ruck | Error: GetWinsize: inappropriate ioctl for device | 11:44 |
| chkumar|ruck | ID NAME CPU % MEM USAGE / LIMIT MEM % NET IO BLOCK IO PIDS | 11:44 |
| chkumar|ruck | Y | 11:44 |
| *** shyamb has quit IRC | 11:44 | |
| chkumar|ruck | Tengu: quiquell|rover: http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-scenario001-standalone/d001eed/logs/undercloud/var/log/extra/podman/podman_allinfo.log.txt.gz | 11:44 |
| quiquell|rover | chkumar|ruck: have being always like that I think | 11:44 |
| quiquell|rover | the GetWinsize I have see it always | 11:45 |
| chkumar|ruck | quiquell|rover: ok | 11:45 |
| quiquell|rover | chkumar|ruck: I am starting to see timeouts at neutron, can be related to OVN ? | 11:45 |
| quiquell|rover | chkumar|ruck: upstream | 11:45 |
| quiquell|rover | chkumar|ruck: I see tripleo-ci-centos-7-undercloud-containers is working fine elsewhere | 11:47 |
| chkumar|ruck | quiquell|rover: so current timed out is standalone scenario 2 na? | 11:47 |
| quiquell|rover | chkumar|ruck: and without sceanrio | 11:48 |
| chkumar|ruck | quiquell|rover: where you are seeing | 11:49 |
| chkumar|ruck | quiquell|rover: http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&from=now%2Fd&to=now%2Fd&var-launchpad_tags=alert&var-promotion_names=current-tripleo&var-promotion_names=current-tripleo-rdo&var-promotion_names=current-tripleo-rdo-testing&var-releases=master&var-releases=rocky&var-releases=queens&var-releases=pike&var-influxdb_filter=result%7C%3D%7CTIMED_OUT | 11:49 |
| quiquell|rover | chkumar|ruck: I mean neuntron timeout at "errors.txt" | 11:50 |
| quiquell|rover | chkumar|ruck: I am just checking noop review | 11:50 |
| chkumar|ruck | quiquell|rover: on neutron timedout I have pinged slaweq | 11:51 |
| chkumar|ruck | quiquell|rover: i think he is looking into that | 11:52 |
| *** shyamb has joined #tripleo | 11:52 | |
| quiquell|rover | chkumar|ruck: ack | 11:52 |
| *** moguimar has joined #tripleo | 11:52 | |
| chkumar|ruck | quiquell|rover: fs01 failure on noop, might be related to rdo cloud | 11:52 |
| quiquell|rover | chkumar|ruck: ack, is the port cleaning still going on ? | 11:53 |
| chkumar|ruck | quiquell|rover: nope the port cleaning script stopped 1 hour ago as there is no down ports whose age is greater than 5 hrs | 11:54 |
| chkumar|ruck | quiquell|rover: and current down port list is 240 | 11:54 |
| quiquell|rover | chkumar|ruck: let's talk with kforde | 11:55 |
| *** dmacpher has joined #tripleo | 11:57 | |
| chkumar|ruck | dalvarez: lucasagomes can you take a look at this error http://logs.openstack.org/45/560445/255/check/tripleo-ci-centos-7-standalone/3a6595c/logs/undercloud/var/log/extra/errors.txt.gz#_2019-02-20_09_27_28_850 is it related to ovn | 11:58 |
| chkumar|ruck | quiquell|rover: we need to ignore these error KeyError: 'neutron:ovn-metadata-sb-cfg' | 11:58 |
| quiquell|rover | chkumar|ruck: ack, I am looking at the issue that prevent merge the collectd flattne | 11:58 |
| quiquell|rover | chkumar|ruck: I think it's OVN related too | 11:58 |
| slaweq | chkumar|ruck: I will check it a bit later | 11:58 |
| slaweq | now I'm on call | 11:59 |
| dalvarez | chkumar|ruck: this one " ERROR /var/log/extra/errors.txt: ERROR /var/log/containers/neutron/server.log: 33 ERROR ovsdbapp.event LOG.debug("Clearing up external_mac of port %s", port_id, mac)" is the one we talked about, we can ignore it but we're close to merge the fix | 11:59 |
| chkumar|ruck | slaweq: sure thanks :-) | 11:59 |
| chkumar|ruck | dalvarez: can you take a look at above timedout message | 11:59 |
| lucasagomes | chkumar|ruck, hi there yeah we've fixed that at https://review.openstack.org/#/c/637175/ (it's on it's way in the gate | 11:59 |
| lucasagomes | )* | 11:59 |
| dalvarez | chkumar|ruck: you mean the nova one then? | 11:59 |
| chkumar|ruck | dalvarez: yes | 11:59 |
| lucasagomes | the mac one is being fixed at https://review.openstack.org/#/c/637172/ | 12:00 |
| dalvarez | chkumar|ruck: : ConnectTimeout: Request to http://192.168.24.1:9696/v2.0/ports/9df92f96-c871-49aa-844d-c90b511df609 timed out 9696 is neutron server right? | 12:00 |
| quiquell|rover | dalvarez: yep | 12:00 |
| chkumar|ruck | dalvarez: yup | 12:00 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart master: Doc: how to pass variables to for a final job configuration https://review.openstack.org/636600 | 12:02 |
| dalvarez | chkumar|ruck: quiquell|rover i cant see any error in neutron server apart from the ovsdbapp one... i can see lots of rabbit issues before | 12:02 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart master: Doc: Overview of configuration variables in quickstart and CI workflow https://review.openstack.org/636600 | 12:02 |
| quiquell|rover | dalvarez: ack | 12:04 |
| chkumar|ruck | dalvarez: apart from neutron server any other place we can look? | 12:04 |
| dalvarez | chkumar|ruck: not sure, can we reproduce it and connect? not sure if the IP endpoint is correct... neutron server seems ok | 12:05 |
| * dalvarez brb | 12:05 | |
| quiquell|rover | dalvarez: I don't see this one at working job http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/var/log/extra/errors.txt.gz#_2019-02-20_10_03_18_032 | 12:06 |
| quiquell|rover | dalvarez: na rabbitmq | 12:07 |
| *** kopecmartin|afk is now known as kopecmartin | 12:07 | |
| quiquell|rover | chkumar|ruck: error Datapath Invalid 143017134675269 | 12:08 |
| *** ansmith has quit IRC | 12:08 | |
| quiquell|rover | chkumar|ruck: I see errors at OVS here | 12:11 |
| quiquell|rover | chkumar|ruck: they are not present at passing ones | 12:11 |
| quiquell|rover | http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/var/log/openvswitch/ovs-vswitchd.log.txt.gz | 12:11 |
| *** skramaja_ has joined #tripleo | 12:12 | |
| *** jpena is now known as jpena|lunch | 12:12 | |
| *** skramaja has quit IRC | 12:12 | |
| chkumar|ruck | quiquell|rover: 2019-02-20T10:02:10.381Z|00106|rconn|ERR|br-ctlplane<->tcp:127.0.0.1:6633: no response to inactivity probe after 5 seconds, disconnecting you mean this? | 12:12 |
| quiquell|rover | chkumar|ruck: yep is not at pssing one http://logs.openstack.org/30/637530/7/check/tripleo-ci-centos-7-undercloud-containers/0851278/logs/undercloud/var/log/openvswitch/ovsdb-server.log.txt.gz | 12:13 |
| Tengu | back. sooo. | 12:13 |
| chkumar|ruck | quiquell|rover: then I think only slaweq can help | 12:13 |
| quiquell|rover | Tengu: now I leave! | 12:13 |
| quiquell|rover | chkumar|ruck: ack | 12:13 |
| Tengu | quiquell|rover: naaaa | 12:14 |
| Tengu | quiquell|rover: shouldn't the new tests be added to https://github.com/openstack-infra/tripleo-ci/blob/master/roles/run-test/tasks/main.yaml ? | 12:14 |
| *** janki has joined #tripleo | 12:14 | |
| Tengu | looks like it's probably better. | 12:14 |
| quiquell|rover | Tengu: ideally you put some validation at tripleo-quickstart | 12:14 |
| quiquell|rover | Tengu: like tempest validations we have | 12:15 |
| Tengu | ah. lemme check then | 12:15 |
| quiquell|rover | Tengu: so it' works for anyone | 12:15 |
| quiquell|rover | Tengu: but maybe is not that easy | 12:15 |
| *** jchhatbar has quit IRC | 12:15 | |
| Tengu | lemme check. Apparently it's in quickstart-extras | 12:16 |
| Tengu | guess I can do something neat. | 12:16 |
| Tengu | there are a couple of validate-* roles | 12:17 |
| quiquell|rover | Tengu: yep something like that | 12:17 |
| Tengu | so either I can extend an existing one, or create a new with doc and include it. | 12:17 |
| quiquell|rover | Mayeb you can put a validate-containers one | 12:17 |
| Tengu | validate-services would be best, since we want to check services (systemctl) and containers (podman|docker). | 12:17 |
| Tengu | lemme check that. | 12:17 |
| quiquell|rover | agree | 12:17 |
| *** agurenko has joined #tripleo | 12:18 | |
| quiquell|rover | chkumar|ruck: this does not look good either http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/var/log/journal.txt.gz#_Feb_20_10_02_12 | 12:20 |
| quiquell|rover | After ovs disconnecting br-int all goes to hell | 12:21 |
| chkumar|ruck | quiquell|rover: file a bug better, we need to find the neutron guys | 12:22 |
| *** pdeore has quit IRC | 12:23 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: [FFU] Make sure group access work correctly with ansible 2.6. https://review.openstack.org/637542 | 12:23 |
| chkumar|ruck | quiquell|rover: or will it? | 12:23 |
| chkumar|ruck | I? | 12:23 |
| quiquell|rover | will do | 12:24 |
| quiquell|rover | chkumar|ruck: looking at logstash | 12:26 |
| quiquell|rover | It has start 20th | 12:26 |
| *** skramaja has joined #tripleo | 12:27 | |
| *** skramaja_ has quit IRC | 12:28 | |
| *** raildo has joined #tripleo | 12:30 | |
| *** shyamb has quit IRC | 12:31 | |
| *** agurenko has quit IRC | 12:33 | |
| chkumar|ruck | quiquell|rover: is something merged on neutron side which caused it or as ovs is getting installed from packages | 12:33 |
| quiquell|rover | chkumar|ruck: https://bugs.launchpad.net/tripleo/+bug/1816750 | 12:35 |
| openstack | Launchpad bug 1816750 in tripleo "ovs-vswitchd errors at br-ctlplane and br-int" [High,Triaged] | 12:35 |
| quiquell|rover | chkumar|ruck: we don't have promotions | 12:35 |
| quiquell|rover | chkumar|ruck: so nothing new can be | 12:35 |
| quiquell|rover | chkumar|ruck: maybe ovs we don't take ovs from promotions | 12:36 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-quickstart-extras master: Raise an error if a service or container is failed https://review.openstack.org/637729 | 12:36 |
| Tengu | chkumar|ruck: quiquell|rover -^^ | 12:36 |
| quiquell|rover | chkumar|ruck: what was the other jobs with OVN issues ? | 12:36 |
| Tengu | crossing fingers. | 12:36 |
| Tengu | never done that, soooo... it has a fairly high probability of failing. | 12:36 |
| EmilienM | quiquell|rover: yes because scenarios aren't switched to image-serve yet | 12:37 |
| EmilienM | quiquell|rover: I suspect the problem is gone without docker registry | 12:37 |
| quiquell|rover | EmilienM: ack then this is it | 12:38 |
| EmilienM | quiquell|rover: wait, let me investigate the failures. I just started my day | 12:38 |
| quiquell|rover | EmilienM: no problem, have paste stuff at the bug | 12:38 |
| chkumar|ruck | quiquell|rover: currntly no ovn issue, to fix two ovs errors these two patches https://review.openstack.org/#/c/637175/ | 12:39 |
| chkumar|ruck | https://review.openstack.org/#/c/637172/ | 12:39 |
| EmilienM | quiquell|rover: what's the bug url please? | 12:41 |
| quiquell|rover | EmilienM: https://bugs.launchpad.net/tripleo/+bug/1815576 | 12:41 |
| openstack | Launchpad bug 1815576 in tripleo "Intermitent failures at tripleo-container-image-prepare at gates with MANIFEST_BLOB_UNKNOWN" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 12:41 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Migrating Undercloud playbooks to their own Ansible Role https://review.openstack.org/636958 | 12:42 |
| *** thrash|g0ne is now known as thrash | 12:43 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart-extras master: Add note to zuul reproducer readme about the ssh key (-m PEM) https://review.openstack.org/638140 | 12:44 |
| EmilienM | quiquell|rover: I seem to have the same issue with image-serve | 12:44 |
| quiquell|rover | :-((( | 12:45 |
| EmilienM | I think there is still an issue in the uploader | 12:45 |
| quiquell|rover | EmilienM: I have being checking code a little | 12:45 |
| quiquell|rover | EmilienM: The retries are before the issue | 12:45 |
| quiquell|rover | EmilienM: I mean after the issue | 12:45 |
| quiquell|rover | But not sure, maybe we can run it at debug level until we found the problem | 12:46 |
| EmilienM | quiquell|rover: I have a patch for that | 12:47 |
| EmilienM | https://review.openstack.org/#/c/636960/ | 12:47 |
| quiquell|rover | EmilienM: let's just merge, it's going to have problem with performance or something ? | 12:47 |
| EmilienM | jaosorior: ^ can you review it please | 12:47 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Adding scenario012 standalone job https://review.openstack.org/634725 | 12:47 |
| EmilienM | jaosorior: and https://review.openstack.org/#/c/638050/ | 12:48 |
| EmilienM | and https://review.openstack.org/#/c/637859/ | 12:48 |
| openstackgerrit | Kamil Sambor proposed openstack/puppet-tripleo master: Add OVN to enabled provider drivers in Octavia https://review.openstack.org/638141 | 12:48 |
| quiquell|rover | EmilienM: If we add some time to the logs maybe we can logstash it | 12:49 |
| EmilienM | quiquell|rover: see my last comment on the bug | 12:50 |
| quiquell|rover | EmilienM: you have to remove override at jobs here https://review.openstack.org/#/c/637859 | 12:50 |
| quiquell|rover | EmilienM: we are overriding cli with docker at job definition for scenarios have to be removed | 12:50 |
| EmilienM | quiquell|rover: I know, we talked about it yesterday | 12:50 |
| quiquell|rover | EmilienM: ack | 12:50 |
| EmilienM | quiquell|rover: the order makes it so podman is deployed with my patch | 12:51 |
| EmilienM | http://logs.openstack.org/59/637859/3/check/tripleo-ci-centos-7-scenario001-standalone/b43a7fd/logs/undercloud/home/zuul/standalone.sh.txt.gz | 12:51 |
| EmilienM | check it out ^ | 12:51 |
| EmilienM | you can see the result: http://logs.openstack.org/59/637859/3/check/tripleo-ci-centos-7-scenario001-standalone/b43a7fd/logs/undercloud/var/log/extra/podman/podman_allinfo.log.txt.gz | 12:51 |
| slaweq | chkumar|ruck: are You asking about those errors in http://logs.openstack.org/45/560445/255/check/tripleo-ci-centos-7-standalone/3a6595c/logs/undercloud/var/log/extra/errors.txt.gz#_2019-02-20_09_27_28_850 ? | 12:51 |
| quiquell|rover | EmilienM: perfect | 12:51 |
| chkumar|ruck | slaweq: https://bugs.launchpad.net/tripleo/+bug/1816750 | 12:52 |
| openstack | Launchpad bug 1816750 in tripleo "ovs-vswitchd errors at br-ctlplane and br-int" [High,Triaged] | 12:52 |
| *** cfontain has joined #tripleo | 12:52 | |
| chkumar|ruck | slaweq: yes | 12:52 |
| jaosorior | EmilienM: sure, will do | 12:53 |
| *** agurenko has joined #tripleo | 12:54 | |
| dalvarez | quiquell|rover: the error you linked before it's from Neutron OVS agent which shouldnt even be running outside scenario007 as it's for ML2/OVS and we shifted to ML2/OVN in tripleo | 12:55 |
| dalvarez | quiquell|rover: talking about http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/var/log/extra/errors.txt.gz#_2019-02-20_10_03_18_032 | 12:55 |
| *** holser|lunch is now known as holser_ | 12:55 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Add fedora-28 support to container-build job on fedora-28 with Buildah https://review.openstack.org/635991 | 12:56 |
| quiquell|rover | dalvarez: LP -> https://bugs.launchpad.net/tripleo/+bug/1816750 | 12:56 |
| openstack | Launchpad bug 1816750 in tripleo "ovs-vswitchd errors at br-ctlplane and br-int" [High,Triaged] | 12:56 |
| slaweq | chkumar|ruck: in neutron-server log: http://logs.openstack.org/45/560445/255/check/tripleo-ci-centos-7-standalone/3a6595c/logs/undercloud/var/log/containers/neutron/server.log.1.gz | 12:56 |
| *** cfontain has quit IRC | 12:56 | |
| slaweq | chkumar|ruck: I can see only some errors related to ovn, like e.g.: http://paste.openstack.org/show/745466/ | 12:56 |
| chkumar|ruck | dalvarez: https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset027.yml is used | 12:57 |
| slaweq | dalvarez: can You check ^^ if that may be related to errors with neutron response timeouts and issues like that? | 12:57 |
| * dalvarez sighs | 12:57 | |
| dalvarez | :D | 12:57 |
| slaweq | thx dalvarez | 12:58 |
| slaweq | dalvarez++ | 12:58 |
| dalvarez | slaweq: chkumar|ruck quiquell|rover the error here http://paste.openstack.org/show/745466/ is unrelated to the failures | 12:58 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Migrating Undercloud playbooks to their own Ansible Role https://review.openstack.org/636958 | 12:58 |
| chkumar|ruck | sorry worng fs | 12:59 |
| chkumar|ruck | slaweq: sorry | 12:59 |
| quiquell|rover | dalvarez: There is a review to fix that already I think | 13:00 |
| chkumar|ruck | slaweq: you are looking at the worng job | 13:00 |
| openstackgerrit | Juan Badia Payno proposed openstack/ansible-role-openstack-operations master: WIP - ansible-pacemaker fix https://review.openstack.org/638143 | 13:00 |
| *** rlandy has joined #tripleo | 13:00 | |
| dalvarez | quiquell|rover: yeah i sent it :p | 13:00 |
| chkumar|ruck | slaweq: the failure is in containerized -undercloud deployment | 13:00 |
| quiquell|rover | yep | 13:00 |
| chkumar|ruck | slaweq: http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-02-20_10_06_36 | 13:00 |
| chkumar|ruck | slaweq: the tests were passing earlier | 13:00 |
| *** psachin has quit IRC | 13:01 | |
| slaweq | chkumar|ruck: so the timeouts in http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-02-20_10_06_36 comes from undercloud or overcloud neutron? | 13:01 |
| gfidente | mwhahaha I think I am not getting https://review.openstack.org/#/c/637984 | 13:01 |
| *** ansmith has joined #tripleo | 13:01 | |
| chkumar|ruck | slaweq: it is from undercloud there is no overcloud there | 13:01 |
| gfidente | can't we force-lower both the dmidecode output and the user input | 13:01 |
| gfidente | ? | 13:01 |
| slaweq | chkumar|ruck: ok, and this http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/var/log/containers/neutron/server.log.txt.gz is neutron-server log for this undercloud, right? | 13:02 |
| dalvarez | chkumar|ruck: no overcloud? i thought is standalone | 13:02 |
| dalvarez | chkumar|ruck: slaweq quiquell|rover if it's undercloud then it's unrelated to OVN, we're not using OVN in the undercloud at all | 13:02 |
| dalvarez | this is why neutron dhcp agent was running there, i got confused thinking it was standalone | 13:02 |
| dalvarez | ksambor++ | 13:03 |
| chkumar|ruck | dalvarez: it is undercloud deployment | 13:03 |
| dalvarez | ok ok | 13:03 |
| ksambor | np | 13:03 |
| quiquell|rover | dalvarez: maybe we have update ovs or something | 13:03 |
| dalvarez | hmm this is weird | 13:03 |
| chkumar|ruck | it is related to ovs | 13:03 |
| quiquell|rover | Let me compare RPMs | 13:03 |
| dalvarez | ksambor: but in neutron server log you can see some OVN stuff | 13:03 |
| quiquell|rover | yep | 13:03 |
| dalvarez | ah no | 13:03 |
| dalvarez | 2019-02-20 09:46:49.254 8 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['openvswitch', 'baremetal'] | 13:04 |
| chkumar|ruck | quiquell|rover: i donot think there is a change in rpm as there is no promotion | 13:04 |
| quiquell|rover | chkumar|ruck: ovs is not from promotions | 13:05 |
| *** psachin has joined #tripleo | 13:05 | |
| mwhahaha | gfidente: we'd have to find the specific heat keys and lower.case them in a dictionary. Heat can't do that | 13:05 |
| chkumar|ruck | ah ok | 13:05 |
| dalvarez | chkumar|ruck: http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/var/log/containers/neutron/server.log.txt.gz <. -i dont see any ovn errors here and it makes sense because this is undercloud, hence ML2/OVS. Are we talking about diffrent errors than before? | 13:05 |
| gfidente | mwhahaha I did with yaql for ceph-ansible | 13:05 |
| mwhahaha | gfidente: so no, we can't. Also it depends on the base os | 13:05 |
| chkumar|ruck | dalvarez: in error logs there are timedout and ovn errors | 13:06 |
| mwhahaha | gfidente: yaql is way more complicated than that patch. No thanks | 13:06 |
| chkumar|ruck | dalvarez: you passed the fixes for ovn ones | 13:06 |
| slaweq | chkumar|ruck: it was very slow in some cases: http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/var/log/containers/neutron/server.log.txt.gz#_2019-02-20_10_03_23_925 | 13:06 |
| dalvarez | chkumar|ruck: but it doesnt make sense to see ovn errors in undercloud as ovn is only used in overcloud, can you show me again (not in errors file) | 13:07 |
| dalvarez | slaweq: theres' bunch of these: http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/var/log/containers/neutron/server.log.txt.gz#_2019-02-20_10_04_24_461 | 13:07 |
| mwhahaha | gfidente: also this needs to go back to at least Newton | 13:07 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-docs master: Add notes for DCN operational availability modes https://review.openstack.org/632089 | 13:07 |
| gfidente | mwhahaha you reviewed it too https://review.openstack.org/#/c/634220/3/docker/services/ceph-ansible/ceph-base.yaml | 13:07 |
| gfidente | mwhahaha agreed on the backports | 13:07 |
| chkumar|ruck | dalvarez: but the actual timedout issue is related to neutron/ovs | 13:07 |
| quiquell|rover | but not ovn | 13:08 |
| dalvarez | chkumar|ruck: right, got it ... i still think that we should make sure that no ovn services are running int he UC | 13:08 |
| quiquell|rover | I think | 13:08 |
| dalvarez | at all | 13:08 |
| *** rh-jelabarre has joined #tripleo | 13:08 | |
| quiquell|rover | I mean the timeout not the errors | 13:09 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Apply stop period for paunch-managed stateful svcs https://review.openstack.org/635161 | 13:09 |
| quiquell|rover | chkumar|ruck: we have also a standalone job with similar timeouts ? | 13:09 |
| slaweq | chkumar|ruck: looks like neutron-server was overloaded | 13:09 |
| slaweq | chkumar|ruck: for example in dhcp agent logs in similar time I see some rpc messaging timeouts: http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/var/log/containers/neutron/dhcp-agent.log.txt.gz#_2019-02-20_10_03_25_032 | 13:09 |
| slaweq | chkumar|ruck: also, there is a lot of errors like http://logs.openstack.org/32/636232/5/check/tripleo-ci-centos-7-undercloud-containers/0f5c399/logs/undercloud/var/log/containers/neutron/dhcp-agent.log.txt.gz#_2019-02-20_10_03_30_403 | 13:10 |
| slaweq | which I related to containers I think | 13:10 |
| openstackgerrit | Juan Badia Payno proposed openstack/ansible-role-openstack-operations master: WIP - ansible-pacemaker fix https://review.openstack.org/638143 | 13:10 |
| quiquell|rover | slaweq: the dhcp is present at same job passing | 13:10 |
| *** saneax has quit IRC | 13:10 | |
| *** saneax has joined #tripleo | 13:11 | |
| *** dprince has joined #tripleo | 13:11 | |
| slaweq | quiquell|rover: yes, but if it is doing full sync all the time, then it may overload neutron server and db | 13:12 |
| slaweq | and because of that other api calls, like create-subnet may be very slow | 13:12 |
| quiquell|rover | slaweq: so we have to remove all OVN stuff from those jobs | 13:12 |
| quiquell|rover | slaweq: also OVS stuff here http://logs.openstack.org/45/560445/255/check/tripleo-ci-centos-7-standalone/3a6595c/logs/undercloud/var/log/openvswitch/ovsdb-server.log.txt.gz | 13:13 |
| quiquell|rover | slaweq: that's a standalone job | 13:13 |
| quiquell|rover | failint with timeouts too | 13:13 |
| dalvarez | gosh | 13:13 |
| dalvarez | quiquell|rover: why do you wanna remove the ovn stuff from the jobs? undercloud is not supposed to run any ovn thing | 13:14 |
| slaweq | quiquell|rover: TBH I don't think that those ovsdb timeouts are directly related to neutron api slow down | 13:14 |
| chkumar|ruck | one min there two failure from two different jobs | 13:15 |
| slaweq | quiquell|rover: what I think is that maybe node was a bit slow, dhcp agent was doing "full sync" all the time because of those errors which I pointed and that caused big load on db | 13:15 |
| chkumar|ruck | 1. is containerized-undercloiud and 2. from standalone | 13:15 |
| chkumar|ruck | both are different | 13:15 |
| slaweq | because of load on db, it was very slowly processing e.g. saving new subnet | 13:15 |
| slaweq | and that caused timeouts | 13:15 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates master: DNM: test jobs https://review.openstack.org/638147 | 13:15 |
| quiquell|rover | chkumar|ruck: both show ovs issue | 13:16 |
| jrist | someone ping me here? | 13:16 |
| quiquell|rover | slaweq: ack | 13:16 |
| chkumar|ruck | in standalone connection to port is timing out | 13:16 |
| openstackgerrit | Sagi Shnaidman proposed openstack/puppet-tripleo master: DNM: test jobs in pt https://review.openstack.org/638148 | 13:16 |
| chkumar|ruck | ykarel: http://logs.openstack.org/45/560445/255/check/tripleo-ci-centos-7-standalone/3a6595c/logs/undercloud/var/log/extra/errors.txt.gz#_2019-02-20_09_27_28_832 | 13:16 |
| quiquell|rover | slaweq: misunderstood it | 13:17 |
| ykarel | chkumar|ruck, ack | 13:17 |
| *** quiquell|rover is now known as quique|rover|mtg | 13:20 | |
| openstackgerrit | Kamil Sambor proposed openstack/puppet-tripleo master: Add OVN to enabled provider drivers in Octavia https://review.openstack.org/638141 | 13:24 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Set tempest_private_net_provider_type to geneve for os_tempest https://review.openstack.org/637838 | 13:26 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Remove upgrade_tasks added during nova services flattening. https://review.openstack.org/637925 | 13:26 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Set hostname for containerized undercloud-upgrade https://review.openstack.org/637938 | 13:26 |
| openstackgerrit | Natal Ngétal proposed openstack/python-tripleoclient master: Add support of extra variables. https://review.openstack.org/637530 | 13:27 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Apply stop period for paunch-managed stateful svcs https://review.openstack.org/635161 | 13:29 |
| quique|rover|mtg | chkumar|ruck: Humm I don't see dhcp errors in the standalone job with OVS issues http://logs.openstack.org/45/560445/255/check/tripleo-ci-centos-7-standalone/3a6595c/logs/undercloud/var/log/extra/errors.txt.gz | 13:31 |
| quique|rover|mtg | slaweq: ^ | 13:31 |
| *** trown is now known as trown|outtypewww | 13:31 | |
| chkumar|ruck | quique|rover|mtg: it is not ovs issue, standalone uses ivn | 13:32 |
| chkumar|ruck | *ovn | 13:32 |
| quique|rover|mtg | chkumar|ruck: looks like same thing here http://logs.openstack.org/45/560445/255/check/tripleo-ci-centos-7-standalone/3a6595c/logs/undercloud/var/log/openvswitch/ovsdb-server.log.txt.gz | 13:33 |
| quique|rover|mtg | chkumar|ruck: nah is different | 13:34 |
| *** jpena|lunch is now known as jpena | 13:34 | |
| slaweq | quique|rover|mtg: yes, and logs in this case looks "better" but this one specific call took 35 seconds: 2019-02-20 09:27:29.856 30 INFO neutron.wsgi [req-f05aabe6-71b3-4540-b858-71c204489d63 5955a592c9ae44b69a1fed4c42572356 7f98e3fd15804a4d911573a84264d093 - default default] 192.168.24.1 "PUT /v2.0/ports/9df92f96-c871-49aa-844d-c90b511df609 HTTP/1.1" status: 200 len: 1103 time: 35.8992271 | 13:35 |
| *** agopi has quit IRC | 13:35 | |
| chkumar|ruck | quique|rover|mtg: that error looks similar | 13:36 |
| quique|rover|mtg | chkumar|ruck: at journal so is some contxt around -> http://logs.openstack.org/45/560445/255/check/tripleo-ci-centos-7-standalone/3a6595c/logs/undercloud/var/log/journal.txt.gz#_Feb_20_09_51_22 | 13:37 |
| quique|rover|mtg | slaweq: so creating port ends but takes a looot of time and it times out in the other side ? | 13:37 |
| slaweq | quique|rover|mtg: it was update port call | 13:38 |
| slaweq | and yes, it was fine but veeeeery slow | 13:38 |
| quique|rover|mtg | slaweq: so it can be just infra issue with network or nodes | 13:38 |
| slaweq | quique|rover|mtg: and if You look at logs for this request in neutron logs, it looks that biggest "gap" is between 2 calls: | 13:39 |
| slaweq | 2019-02-20 09:26:56.605 30 DEBUG neutron.notifiers.nova [req-f05aabe6-71b3-4540-b858-71c204489d63 5955a592c9ae44b69a1fed4c42572356 7f98e3fd15804a4d911573a84264d093 - default default] Ignoring state change previous_port_status: DOWN current_port_status: DOWN port_id 9df92f96-c871-49aa-844d-c90b511df609 record_port_status_changed /usr/lib/python2.7/site-packages/neutron/notifiers/nova.py:212 | 13:39 |
| slaweq | and | 13:39 |
| slaweq | 2019-02-20 09:27:26.006 30 DEBUG neutron_lib.callbacks.manager [req-f05aabe6-71b3-4540-b858-71c204489d63 5955a592c9ae44b69a1fed4c42572356 7f98e3fd15804a4d911573a84264d093 - default default] Notify callbacks ['neutron.services.qos.qos_plugin.QoSPlugin._validate_update_port_callback--9223372036854009775'] for port, precommit_update _notify_loop | 13:39 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: container image build: introduce --use-buildah https://review.openstack.org/634322 | 13:39 |
| slaweq | /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:193 | 13:39 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Add fedora-28 support to container-build job on fedora-28 with Buildah https://review.openstack.org/635991 | 13:39 |
| slaweq | after this second one was logged, all other was done in about 2-3 seconds | 13:39 |
| *** gbarros has joined #tripleo | 13:39 | |
| openstackgerrit | Juan Badia Payno proposed openstack/ansible-role-openstack-operations master: WIP - ansible-pacemaker fix https://review.openstack.org/638143 | 13:40 |
| quique|rover|mtg | slaweq: let's just monitor to see if it's gone | 13:41 |
| quique|rover|mtg | slaweq: thanks mate | 13:41 |
| slaweq | quique|rover|mtg: ok | 13:41 |
| slaweq | yw | 13:41 |
| quique|rover|mtg | chkumar|ruck: have you see Feb 20 09:51:39 standalone.localdomain podman[103791]: /usr/share/openstack-tripleo-common/healthcheck/common.sh: line 64: [: 1550655999: unary operator expected | 13:42 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Create deployment/deprecated directory https://review.openstack.org/637890 | 13:42 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Deprecate Docker https://review.openstack.org/637915 | 13:42 |
| chkumar|ruck | quique|rover|mtg: we donot run dstat in standalone na? I will also give a fair idea when load was high | 13:43 |
| quique|rover|mtg | humm didn't sshnaidm added it ? | 13:43 |
| sshnaidm | quique|rover|mtg, I added atop | 13:43 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: standalone: switch container registry to ImageServce https://review.openstack.org/637861 | 13:43 |
| *** gbarros has quit IRC | 13:44 | |
| sshnaidm | quique|rover|mtg, but we ran dstat previously too, so worth to check | 13:44 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Add external_resource_vip_id property to network_data.yaml https://review.openstack.org/638002 | 13:44 |
| sshnaidm | chkumar|ruck, ^^ | 13:44 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: image-serve: redirect root to /v2 https://review.openstack.org/637404 | 13:44 |
| *** gbarros has joined #tripleo | 13:44 | |
| quique|rover|mtg | sshnaidm: maybe atop can help us too | 13:44 |
| *** spsurya has joined #tripleo | 13:44 | |
| *** florianf has quit IRC | 13:45 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Add external_resource_vip_id property to network_data.yaml https://review.openstack.org/638002 | 13:45 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: TLS everywhere: Set post-save command for nova-vnc-proxy https://review.openstack.org/636877 | 13:46 |
| chkumar|ruck | quique|rover|mtg: sshnaidm ok, let me enable dstat and see | 13:46 |
| chkumar|ruck | quique|rover|mtg: regarding above podman error from where have you grepped from standalone? | 13:47 |
| quique|rover|mtg | chkumar|ruck: yep standalone | 13:48 |
| quique|rover|mtg | chkumar|ruck: I am going to clos this as invalid https://bugs.launchpad.net/tripleo/+bug/1816750 | 13:49 |
| openstack | Launchpad bug 1816750 in tripleo "ovs-vswitchd errors at br-ctlplane and br-int" [High,Triaged] | 13:49 |
| quique|rover|mtg | chkumar|ruck: or do you want to use it for the dstat stuff ? | 13:49 |
| chkumar|ruck | quique|rover|mtg: yes | 13:49 |
| quique|rover|mtg | invalid or dstat ? | 13:49 |
| dtantsur | slagle: hi! "novacompute" in the example in http://tripleo.org/install/advanced_deployment/deployed_server.html#deployed-server-with-config-download, where does it come from? it's not a role name, is it? | 13:49 |
| chkumar|ruck | quique|rover|mtg: sorry, mark it in valid | 13:50 |
| quique|rover|mtg | ack | 13:50 |
| slagle | dtantsur: it comes from the hostnameformat on the role | 13:50 |
| chkumar|ruck | quique|rover|mtg: for standalone I am adding dstat to check if above tests fails what is the load | 13:51 |
| slagle | dtantsur: HostnameFormatDefault rather | 13:51 |
| quique|rover|mtg | chkumar|ruck: that will help us a lot | 13:51 |
| dtantsur | slagle: oh, so it's pretty random, not just role.lower().. | 13:51 |
| slagle | dtantsur: no. there's nothing random about it | 13:52 |
| dtantsur | then why novacompute for Compute role? :) | 13:52 |
| openstackgerrit | Juan Badia Payno proposed openstack/ansible-role-openstack-operations master: WIP - ansible-pacemaker fix https://review.openstack.org/638143 | 13:52 |
| *** jchhatbar has joined #tripleo | 13:52 | |
| slagle | dtantsur: some backwards compatibility issue | 13:53 |
| dtantsur | sigh | 13:53 |
| slagle | dang users! | 13:53 |
| slagle | such a burden | 13:53 |
| *** jchhatbar has quit IRC | 13:53 | |
| dtantsur | slagle: which means that for the metalsmith work (to generate the expected hostnames) I'll have to fetch the roles from <somewhere> and check their HostNameDefault, right? | 13:53 |
| *** jchhatbar has joined #tripleo | 13:54 | |
| *** janki has quit IRC | 13:54 | |
| slagle | dtantsur: no. you'll want to actually check HostnameFormat from the stack environment. the default property will just set that default value, but it could in fact be any value | 13:54 |
| *** jcoufal has joined #tripleo | 13:54 | |
| dtantsur | slagle: do you have an example handy on how to do it from tripleo-common? | 13:55 |
| *** mcornea has joined #tripleo | 13:55 | |
| slagle | dtantsur: not handy. i'd have to check | 13:55 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Run dstat prior deploying to standalone https://review.openstack.org/638159 | 13:56 |
| chkumar|ruck | sshnaidm: quique|rover|mtg weshay ^^ | 13:56 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: TLS everywhere: Set post-save command for neutron dhcpd https://review.openstack.org/636878 | 13:56 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: TLS everywhere: Mount the whole /etc/pki/libvirt/ directory in libvirt https://review.openstack.org/636879 | 13:56 |
| slagle | dtantsur: utils/play.py:get_env i think might get it for you | 13:58 |
| dtantsur | thnx! | 13:58 |
| slagle | utils/plan.py that is | 13:58 |
| larsks | dalvarez: good morning! Re: your comment earlier, the route to the metadata service received by the baremetal nodes is via the qdhcp namespace. I can hook you up with a tmate session if you'd like to poke around. | 13:58 |
| *** hjensas has quit IRC | 13:58 | |
| slagle | dtantsur: should have a parameter_defaults key | 13:58 |
| dalvarez | larsks: yeah we can do that | 13:59 |
| dalvarez | larsks: those baremetal nodes were provisioned by ironic in the overcloud, right? | 13:59 |
| dalvarez | larsks: good morning! :) | 13:59 |
| larsks | dalvarez: this is a standalone tripleo, there is no undercloud/overcloud. | 13:59 |
| dalvarez | ack | 13:59 |
| larsks | But they were provisioned by ironic, yes. | 13:59 |
| dalvarez | got it | 14:00 |
| dtantsur | slagle: so, it looks like get_env(swift, 'overcloud')['parameter_defaults']['HostnameFormat'] (or HostnameFormatDefault?)? | 14:00 |
| larsks | Also, we'll need to wait about 20 minutes for that tmate session, because things got a bit horked earlier when docker got stuck trying to remove a container. | 14:00 |
| larsks | Just rebooted and re-running the deploy right now. | 14:00 |
| * dtantsur wonders how to pull down the plan to look at it | 14:00 | |
| dtantsur | oh | 14:00 |
| dtantsur | slagle: but if we do that, we cannot follow the approach in the spec with deploying bare metals before creating the heat stack.. | 14:01 |
| *** ykarel is now known as ykarel|pto | 14:02 | |
| *** jtomasek has quit IRC | 14:02 | |
| *** jtomasek has joined #tripleo | 14:04 | |
| dtantsur | I guess I can go the other way and return an environment file that overrides HostnameFormatDefault for Roles.. or can I? | 14:04 |
| chkumar|ruck | Tengu: swp file got added https://review.openstack.org/#/c/637729/5/roles/validate-services/tasks/.containers.yaml.swp,unified | 14:06 |
| dtantsur | slagle: do I still need HostnameMap in http://tripleo.org/install/advanced_deployment/deployed_server.html#deployed-server-with-config-download if both sides match? | 14:07 |
| *** vinaykns has joined #tripleo | 14:08 | |
| mwhahaha | gfidente: also your yaql was the content to the file, the problem here is it affects $::uuid in puppet so we have to match both upper/lower depending on version. It's the copying of the hieradata file is probably the most simple solution with the least amount of change | 14:08 |
| *** sdake has joined #tripleo | 14:10 | |
| *** agopi has joined #tripleo | 14:10 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common master: nova-less-deploy: implement deploy_roles workflow https://review.openstack.org/637191 | 14:12 |
| *** _fragatina has joined #tripleo | 14:13 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/rocky: Add missing RoleParameters and ServiceNames https://review.openstack.org/638163 | 14:14 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/queens: Add missing RoleParameters and ServiceNames https://review.openstack.org/638164 | 14:14 |
| slagle | dtantsur: are you asking if you need the HostnameMap even if the actual server hostnames match the heat assigned values? | 14:14 |
| Tengu | chkumar|ruck: erf.... can't we have a common .gitignore that actually DROP that kind of things? --. | 14:14 |
| dtantsur | slagle: yep | 14:14 |
| chkumar|ruck | Tengu: I think it has let me check | 14:15 |
| chkumar|ruck | Tengu: .swp is missing https://github.com/openstack/tripleo-quickstart-extras/blob/master/.gitignore | 14:15 |
| Tengu | chkumar|ruck: let's finish the zuul run | 14:16 |
| openstackgerrit | Natal Ngétal proposed openstack/python-tripleoclient master: Add support of extra variables. https://review.openstack.org/637530 | 14:16 |
| Tengu | chkumar|ruck: I'll do another patch in order to drop .*.swp | 14:16 |
| chkumar|ruck | Tengu: let me push a patch for .swp | 14:16 |
| Tengu | chkumar|ruck: ok | 14:16 |
| chkumar|ruck | Tengu: sure go ahead | 14:16 |
| quique|rover|mtg | chkumar|ruck: for this https://bugs.launchpad.net/tripleo/+bug/1816414 | 14:16 |
| openstack | Launchpad bug 1816414 in tripleo "[queens][fs017] telemetry tempest plugin tests failed in multinode promotion pipeline" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 14:16 |
| chkumar|ruck | quique|rover|mtg: yes | 14:17 |
| quique|rover|mtg | chkumar|ruck: we need https://github.com/gnocchixyz/gnocchi/pull/1019 ? | 14:17 |
| chkumar|ruck | quique|rover|mtg: yes | 14:17 |
| quique|rover|mtg | ack | 14:17 |
| chkumar|ruck | quique|rover|mtg: and https://review.rdoproject.org/r/#/c/18917/ this one | 14:17 |
| quique|rover|mtg | chkumar|ruck: and htis ? https://review.openstack.org/#/c/638118/ | 14:17 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-ipsec master: Migrate README to rst https://review.openstack.org/637869 | 14:18 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-ipsec master: Fix ansible-lint issues https://review.openstack.org/638167 | 14:18 |
| chkumar|ruck | quique|rover|mtg: it is for rokcy | 14:18 |
| chkumar|ruck | quique|rover|mtg: those failure might pop up in rocky also | 14:18 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-quickstart-extras master: Ignore swp files from vim and emacs. https://review.openstack.org/638168 | 14:18 |
| Tengu | chkumar|ruck: -^^ | 14:18 |
| chkumar|ruck | quique|rover|mtg: I need to go through the logs | 14:18 |
| slagle | dtantsur: i guess you may not. haven't tried it tbh | 14:18 |
| *** sdake has quit IRC | 14:18 | |
| quique|rover|mtg | chkumar|ruck: ack thanks man | 14:19 |
| chkumar|ruck | quique|rover|mtg: rdo patch and pr will solve queens issue | 14:19 |
| slagle | dtantsur: almost every deployment i've helped with, whether deployed-server or not, uses HostnameMap | 14:19 |
| chkumar|ruck | quique|rover|mtg: once we have promotion, https://review.openstack.org/#/c/604311/ we have to merge this to close master bug | 14:20 |
| quique|rover|mtg | chkumar|ruck: we have to skip that at master ? | 14:22 |
| chkumar|ruck | quique|rover|mtg: no no we have to unskip it | 14:22 |
| chkumar|ruck | quique|rover|mtg: it was added to skip list I think 6 months back | 14:22 |
| quique|rover|mtg | chkumar|ruck: I have read the gerrit backward :-) | 14:23 |
| openstackgerrit | Sylvain Afchain proposed openstack/tripleo-heat-templates master: Allow embedded Skydive etcd port https://review.openstack.org/637492 | 14:28 |
| *** srini__ has joined #tripleo | 14:28 | |
| dprince | bogdando: I think mandre is saying here that he is removing previous -1 https://review.openstack.org/#/c/620062/ | 14:29 |
| *** ykarel|pto has quit IRC | 14:29 | |
| dprince | bogdando: I'm not sure we need to block it now fwiw. | 14:29 |
| vinaykns | mwhahaha, EmilienM: need reviews for https://review.openstack.org/#/c/636657/ | 14:30 |
| *** srini_ has quit IRC | 14:30 | |
| dprince | EmilienM: I think mandre's comment here gives us the status update we need to continue the wrapper scripts: https://review.openstack.org/#/c/620062/ | 14:30 |
| chkumar|ruck | Tengu: thanks :-) | 14:32 |
| bogdando | dprince: I think I got the message for no notion of namespaces wrong then | 14:32 |
| bogdando | like expecting it to be expanded in the spec | 14:32 |
| bogdando | will not block then | 14:32 |
| vinaykns | mwhahaha: could you review this as well https://review.openstack.org/#/c/636660/ | 14:33 |
| vinaykns | thank you.! | 14:33 |
| *** sdake has joined #tripleo | 14:34 | |
| mandre | dprince, bogdando: yeah I think this namespace requirement for neutron could be clarified in the spec, as I said in the comments without this requirement we can simplify quite a bit to have less files involved | 14:34 |
| bogdando | dprince: hmm, so that is why I found the spec as incomplete | 14:34 |
| bogdando | and wanted to wait for the clarification until it's merged as is | 14:35 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-quickstart-extras master: Raise an error if a service or container is failed https://review.openstack.org/637729 | 14:35 |
| *** chkumar|ruck is now known as kmrchdn | 14:35 | |
| mandre | my poc only had 3 files: the service file, the path file, and the arg file (that gets deleted once the service finish running) | 14:35 |
| *** mcornea has quit IRC | 14:35 | |
| Tengu | kmrchdn: so apparently I need to poke around in the different playbooks in order to find the right places to inject the new role :). | 14:35 |
| dprince | mandre, bogdando: okay. we can add that bit I guess. -2 for clarification it is | 14:37 |
| bogdando | yea, thanks | 14:37 |
| bogdando | -2 was for preventing its getting merged so early | 14:38 |
| bogdando | as there was a plenty of +1s and a single +2 | 14:38 |
| openstackgerrit | Dan Prince proposed openstack/ansible-role-tripleo-modify-image master: Fix dev install (yum -y...) https://review.openstack.org/638174 | 14:38 |
| dprince | bogdando: workflow perhaps for clarity? | 14:39 |
| dprince | bogdando: I'm fine with your -2... so long as you are committing to reviewing it ASAP when I fix it :) | 14:39 |
| bogdando | well, I understand that differently, but may be that's just me. I can only think of -1 wf as "hold on *I'm* working on it" | 14:39 |
| bogdando | but I cant tell for the authors if they do | 14:39 |
| *** Vorrtex has joined #tripleo | 14:40 | |
| *** avivgta has quit IRC | 14:40 | |
| bogdando | dprince: ack | 14:41 |
| bogdando | I trust you! :) | 14:41 |
| openstackgerrit | Dan Prince proposed openstack/ansible-role-tripleo-modify-image master: Fix dev install (yum -y...) https://review.openstack.org/638174 | 14:41 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-ipsec master: Migrate README to rst https://review.openstack.org/637869 | 14:42 |
| openstackgerrit | Merged openstack/tripleo-ipsec master: Fix ansible-lint issues https://review.openstack.org/638167 | 14:43 |
| *** mcornea has joined #tripleo | 14:47 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Be able to know when we are running inside a minor update workflow https://review.openstack.org/635725 | 14:49 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Do not restart bundles during a minor update https://review.openstack.org/635731 | 14:49 |
| *** srini__ has quit IRC | 14:50 | |
| *** ykarel|pto has joined #tripleo | 14:50 | |
| *** tzumainn has joined #tripleo | 14:51 | |
| *** jobewan has joined #tripleo | 14:52 | |
| *** avivgt has joined #tripleo | 14:57 | |
| *** sdake has quit IRC | 14:58 | |
| openstackgerrit | Merged openstack/tripleo-ipsec master: Migrate README to rst https://review.openstack.org/637869 | 15:02 |
| *** quique|rover|mtg is now known as quiquell | 15:06 | |
| *** aedc has quit IRC | 15:06 | |
| *** mjturek has joined #tripleo | 15:07 | |
| *** wolverineav has joined #tripleo | 15:13 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Make build-containers job definition reusable https://review.openstack.org/638108 | 15:15 |
| *** florianf has joined #tripleo | 15:15 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/tripleo-ci master: Make build-containers job definition reusable https://review.openstack.org/638108 | 15:17 |
| *** wolverineav has quit IRC | 15:18 | |
| *** jchhatbar has quit IRC | 15:19 | |
| *** ekultails has joined #tripleo | 15:20 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-quickstart-extras master: Raise an error if a service or container is failed https://review.openstack.org/637729 | 15:21 |
| *** moshele has quit IRC | 15:21 | |
| *** waleedm has quit IRC | 15:27 | |
| *** saneax has quit IRC | 15:33 | |
| *** skramaja has quit IRC | 15:43 | |
| openstackgerrit | Merged openstack/python-tripleoclient master: undercloud: configure ContainerImagePrepareDebug based on undercloud_debug https://review.openstack.org/636960 | 15:43 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: nova_libvirt_init_secret: add net=host https://review.openstack.org/638050 | 15:43 |
| *** hjensas has joined #tripleo | 15:43 | |
| *** sai_p_ has joined #tripleo | 15:46 | |
| *** kmrchdn is now known as chandankumar | 15:48 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Apply stop period for paunch-managed stateful svcs https://review.openstack.org/635161 | 15:52 |
| *** agurenko has quit IRC | 15:53 | |
| *** quiquell is now known as quiquell|off | 15:56 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/560445 | 16:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 16:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO https://review.openstack.org/604293 | 16:00 |
| *** yprokule has quit IRC | 16:01 | |
| *** aedc has joined #tripleo | 16:03 | |
| *** mcornea has quit IRC | 16:06 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Add fedora-28 support to container-build job on fedora-28 with Buildah https://review.openstack.org/635991 | 16:07 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Add fedora-28 support to container-build job on fedora-28 with Buildah https://review.openstack.org/635991 | 16:08 |
| *** mcornea has joined #tripleo | 16:10 | |
| *** jfrancoa has quit IRC | 16:15 | |
| *** wolverineav has joined #tripleo | 16:29 | |
| openstackgerrit | Luca Miccini proposed openstack/tripleo-common master: Adds redfish support to 'overcloud generate fencing'. https://review.openstack.org/638204 | 16:30 |
| *** xek has joined #tripleo | 16:31 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Switch scenario001-002-003 to use Podman https://review.openstack.org/637859 | 16:33 |
| *** aufi has quit IRC | 16:35 | |
| *** avivgt has quit IRC | 16:37 | |
| *** aedc has quit IRC | 16:37 | |
| openstackgerrit | Kamil Sambor proposed openstack/puppet-tripleo master: Add OVN to enabled provider drivers in Octavia https://review.openstack.org/638141 | 16:37 |
| openstackgerrit | Luke Short proposed openstack/tripleo-common stable/queens: Workflow and action for deployment failures https://review.openstack.org/619133 | 16:38 |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Check proxy env vars https://review.openstack.org/636772 | 16:44 |
| *** wolverineav has quit IRC | 16:49 | |
| *** wolverin_ has joined #tripleo | 16:49 | |
| *** wolverin_ has quit IRC | 16:53 | |
| openstackgerrit | Brent Eagles proposed openstack/puppet-tripleo master: Run octavia-api under apache https://review.openstack.org/636379 | 16:54 |
| *** suuuper has quit IRC | 16:55 | |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: Remove the container cli override for standalone scenario 1-3 https://review.openstack.org/638212 | 16:57 |
| marios | EmilienM: ^ | 16:57 |
| EmilienM | marios: thanks | 16:58 |
| *** psachin has quit IRC | 17:04 | |
| *** moshele has joined #tripleo | 17:05 | |
| fultonj | EmilienM: you mentioned net=host issues last night. were they for ceph containers? FYI ceph-ansible versions earlier than 4 had that issue. the ceph-ansible 4 beta rpm does not have this issue https://buildlogs.centos.org/centos/7/storage/x86_64/ceph-nautilus/ | 17:05 |
| *** hjensas has quit IRC | 17:05 | |
| fultonj | https://review.openstack.org/#/c/562213/ | 17:05 |
| EmilienM | fultonj: nova | 17:05 |
| fultonj | EmilienM: ^ i got scen004 to deploy ceph correctly using ceph-ansible 4beta rpm + ceph-container nautilus beta | 17:06 |
| fultonj | http://logs.openstack.org/13/562213/16/check/tripleo-ci-centos-7-scenario001-standalone/47b03f1/logs/undercloud/home/zuul/undercloud-ansible-3JHTto/ceph-ansible/ceph_ansible_command.log.txt.gz#_2019-02-15_22_03_22_833 | 17:06 |
| fultonj | the deployment timed out before i could see openstack excercise ceph however | 17:07 |
| *** ksambor has quit IRC | 17:07 | |
| fultonj | i recheck'd lat night and 001/004 both failed before they could even run the ceph deploy | 17:08 |
| openstackgerrit | David Peacock proposed openstack/tripleo-heat-templates master: flatten database service MySQL Server https://review.openstack.org/633980 | 17:08 |
| *** cylopez has quit IRC | 17:08 | |
| EmilienM | fultonj: awesome! | 17:08 |
| fultonj | thanks EmilienM. have you see 001 and 004 fail in 6 minutes like this before? https://review.openstack.org/#/c/562213 | 17:11 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: kolla_builder: don't build template mapping when not needed https://review.openstack.org/637652 | 17:11 |
| EmilienM | fultonj: I haven't | 17:11 |
| openstackgerrit | David Peacock proposed openstack/tripleo-heat-templates master: flatten database service MySQL Server https://review.openstack.org/633980 | 17:11 |
| EmilienM | fultonj: I checked the logs and could not find yet | 17:11 |
| *** ramishra has quit IRC | 17:12 | |
| fultonj | in this job the depends-on tree is tht --> ooq and tht --> tripleo-ci | 17:12 |
| fultonj | i assume there's no issue with that however | 17:12 |
| *** pcaruana has quit IRC | 17:13 | |
| EmilienM | it should be good | 17:13 |
| mwhahaha | http://logs.openstack.org/13/562213/16/check/tripleo-ci-centos-7-containers-multinode/be44bfc/job-output.txt.gz#_2019-02-19_20_33_43_513580 | 17:13 |
| mwhahaha | parser error | 17:13 |
| fultonj | mwhahaha: thanks | 17:14 |
| *** dtantsur is now known as dtantsur|afk | 17:14 | |
| openstackgerrit | David Peacock proposed openstack/tripleo-heat-templates master: flatten database service Redis https://review.openstack.org/635930 | 17:14 |
| *** _fragatina has quit IRC | 17:14 | |
| *** moshele has quit IRC | 17:17 | |
| *** bogdando has quit IRC | 17:18 | |
| *** chandankumar is now known as kmrchdn | 17:19 | |
| *** florianf has quit IRC | 17:19 | |
| *** dsneddon has quit IRC | 17:25 | |
| *** dciabrin has quit IRC | 17:25 | |
| *** jpich has quit IRC | 17:25 | |
| *** dciabrin has joined #tripleo | 17:26 | |
| *** dsneddon has joined #tripleo | 17:27 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Adding support of glance cinder store settings https://review.openstack.org/635429 | 17:29 |
| *** moguimar has quit IRC | 17:29 | |
| fultonj | gfidente: can you review fmount's https://review.openstack.org/#/c/637537 when you get a chance? | 17:41 |
| weshay | mwhahaha d0ugal fyi https://review.openstack.org/#/c/638154/ | 17:42 |
| *** hberaud is now known as hberaud|gone | 17:43 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-common master: Use Octavia's project for lb security group ops https://review.openstack.org/634286 | 17:45 |
| jaosorior | sshnaidm: lol, yep, it broke | 17:48 |
| jaosorior | vinaykns: ^^ | 17:48 |
| jaosorior | gotta figure out how to fix it up | 17:48 |
| *** jaosorior has quit IRC | 17:49 | |
| *** jaosorior has joined #tripleo | 17:49 | |
| vinaykns | jaosorior: yeah.! | 17:49 |
| *** derekh has quit IRC | 17:50 | |
| vinaykns | jaosorior: looks like even connecting to external freeipa also giving tough time | 17:51 |
| vinaykns | jaosorior: need to look into it | 17:51 |
| sshnaidm | jaosorior, ok.. will look then, can you paste a full quickstart command you run? | 17:52 |
| jaosorior | ./quickstart.sh --no-clone --teardown all --clean -p quickstart-extras.yml -N config/nodes/1ctlr_1comp_1supp.yml -c config/general_config/ipa.yml -R master-tripleo-ci -w /home/josorior/local_quickstart --tags all $VIRTHOST | 17:53 |
| jaosorior | sshnaidm: ^^ | 17:53 |
| *** thrash is now known as thrash|biab | 17:54 | |
| sshnaidm | jaosorior, ack | 17:57 |
| *** panda is now known as panda|off | 17:59 | |
| *** wolverineav has joined #tripleo | 17:59 | |
| *** dciabrin has quit IRC | 18:01 | |
| *** sdake has joined #tripleo | 18:03 | |
| *** wolverineav has quit IRC | 18:04 | |
| openstackgerrit | Jon Schlueter proposed openstack/tripleo-common master: drop python-crypto changes from overrides https://review.openstack.org/634465 | 18:05 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-quickstart-extras master: Raise an error if a service or container is failed https://review.openstack.org/637729 | 18:11 |
| *** jpena is now known as jpena|off | 18:11 | |
| EmilienM | jaosorior: can you look https://review.openstack.org/#/c/637890/ please? | 18:12 |
| EmilienM | (I had to fix some YAML but same as before) | 18:12 |
| *** _fragatina has joined #tripleo | 18:12 | |
| *** wolverineav has joined #tripleo | 18:12 | |
| *** morazi has quit IRC | 18:13 | |
| gfidente | fultonj ack | 18:14 |
| *** kopecmartin is now known as kopecmartin|off | 18:15 | |
| *** wolverineav has quit IRC | 18:15 | |
| *** wolverineav has joined #tripleo | 18:15 | |
| *** wolverineav has quit IRC | 18:16 | |
| *** wolverineav has joined #tripleo | 18:17 | |
| *** wolverineav has quit IRC | 18:18 | |
| *** wolverineav has joined #tripleo | 18:18 | |
| *** wolverineav has quit IRC | 18:20 | |
| *** ykarel|pto has quit IRC | 18:22 | |
| *** wolverineav has joined #tripleo | 18:25 | |
| *** holser_ has quit IRC | 18:28 | |
| *** thrash|biab is now known as thrash | 18:28 | |
| *** wolverineav has quit IRC | 18:30 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: standalone: enable debug on container image build https://review.openstack.org/638234 | 18:32 |
| *** gfidente is now known as gfidente|afk | 18:32 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/636370 | 18:33 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/636370 | 18:33 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/636370 | 18:34 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test $î - never merge https://review.openstack.org/636371 | 18:34 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test $î - never merge https://review.openstack.org/636372 | 18:34 |
| *** wolverineav has joined #tripleo | 18:34 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test $î - never merge https://review.openstack.org/636373 | 18:34 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test $î - never merge https://review.openstack.org/636374 | 18:34 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test $î - never merge https://review.openstack.org/636375 | 18:35 |
| EmilienM | quiquell|off, stevebaker^ trying to reproduce https://bugs.launchpad.net/tripleo/+bug/1815576 again in CI with more logs this time | 18:35 |
| openstack | Launchpad bug 1815576 in tripleo "Intermitent failures at tripleo-container-image-prepare at gates with MANIFEST_BLOB_UNKNOWN" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 18:35 |
| *** tzumainn has quit IRC | 18:36 | |
| *** dciabrin has joined #tripleo | 18:42 | |
| *** xek_ has joined #tripleo | 18:42 | |
| *** xek has quit IRC | 18:45 | |
| mwhahaha | abuser of CI | 18:48 |
| EmilienM | i paid for it | 18:49 |
| *** waleedm has joined #tripleo | 18:55 | |
| *** moshele has joined #tripleo | 18:56 | |
| *** sdake has quit IRC | 18:58 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: image_uploader: conditionally import docker https://review.openstack.org/638238 | 19:01 |
| *** mcornea has quit IRC | 19:02 | |
| *** mcornea has joined #tripleo | 19:02 | |
| *** dciabrin has quit IRC | 19:03 | |
| *** dciabrin has joined #tripleo | 19:04 | |
| *** sdake has joined #tripleo | 19:13 | |
| *** wolverineav has quit IRC | 19:15 | |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Deprecate docker config options https://review.openstack.org/638243 | 19:18 |
| mwhahaha | EmilienM: -^ | 19:18 |
| *** hjensas has joined #tripleo | 19:18 | |
| *** jaosorior has quit IRC | 19:18 | |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Deprecate docker config options https://review.openstack.org/638243 | 19:19 |
| *** xek_ has quit IRC | 19:19 | |
| *** xek_ has joined #tripleo | 19:19 | |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Deprecate docker config options https://review.openstack.org/638243 | 19:20 |
| *** xek_ has quit IRC | 19:21 | |
| *** xek has joined #tripleo | 19:22 | |
| EmilienM | mwhahaha: ack | 19:22 |
| *** tzumainn has joined #tripleo | 19:22 | |
| *** galaxyyahoo has quit IRC | 19:22 | |
| *** galaxyyahoo has joined #tripleo | 19:23 | |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Check proxy env vars https://review.openstack.org/636772 | 19:23 |
| *** xek_ has joined #tripleo | 19:26 | |
| *** jcoufal has quit IRC | 19:27 | |
| *** xek has quit IRC | 19:27 | |
| *** jaosorior has joined #tripleo | 19:28 | |
| *** saneax has joined #tripleo | 19:28 | |
| *** xek has joined #tripleo | 19:31 | |
| *** xek_ has quit IRC | 19:31 | |
| *** rfolco has quit IRC | 19:32 | |
| *** xek has quit IRC | 19:34 | |
| *** xek has joined #tripleo | 19:35 | |
| *** rfolco has joined #tripleo | 19:39 | |
| *** dciabrin has quit IRC | 19:41 | |
| *** dciabrin has joined #tripleo | 19:42 | |
| sshnaidm | jaosorior, tls job fails on rocky, seems like it requires backport: http://logs.rdoproject.org/68/638068/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/401ef12/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 19:45 |
| sshnaidm | jaosorior, do you want to limit it for specific branches? | 19:45 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Podman support in haproxy-public-tls-inject https://review.openstack.org/638246 | 19:46 |
| EmilienM | jaosorior: ^ | 19:46 |
| *** wolverineav has joined #tripleo | 19:46 | |
| *** jcoufal has joined #tripleo | 19:47 | |
| *** sdake has quit IRC | 19:50 | |
| stevebaker | EmilienM: it sounds like fetching a remote layer sometimes fails in a way which we don't catch and retry | 19:50 |
| EmilienM | yeah that's what I thought, I haven't found yet what we don't retry | 19:51 |
| *** wolverineav has quit IRC | 19:51 | |
| *** wolverineav has joined #tripleo | 19:51 | |
| *** saneax has quit IRC | 19:53 | |
| EmilienM | maybe the _copy_manifest_config_to_registry | 19:55 |
| EmilienM | it doesn't seem to happen during the tar-split | 19:56 |
| EmilienM | actually it could be in _copy_stream_to_registry as well | 19:56 |
| EmilienM | well, _copy_layer_registry_to_registry (with a retry) returns cls._copy_stream_to_registry, so it should be fine | 19:58 |
| *** wolverineav has quit IRC | 19:59 | |
| stevebaker | EmilienM: the problem is missing layers, so its most likely to be in _copy_layer_registry_to_registry | 19:59 |
| EmilienM | which has a retry | 20:00 |
| stevebaker | only on equests.exceptions.RequestException, maybe we should just retry on everything | 20:00 |
| *** wolverineav has joined #tripleo | 20:01 | |
| EmilienM | ah yes indeed | 20:01 |
| stevebaker | EmilienM: or put in some try/except logging in there to find out what else we should retry on. Its probably an IO error or somesuch. Actually we should do 2 things: | 20:02 |
| mwhahaha | beagles: https://bugs.launchpad.net/tripleo/+bug/1816847 | 20:03 |
| openstack | Launchpad bug 1816847 in tripleo "default NeutronNetworkType is still set to vxlan,vlan but we don't support that with ovn by default" [Critical,Triaged] | 20:03 |
| stevebaker | EmilienM: 1. add a job.exception() check in _copy_registry_to_registry so we get logging when layers fail, then | 20:03 |
| beagles | mwhahaha, ack thanks will fix | 20:03 |
| stevebaker | 2. add the exceptions discovered by 1. in the retry for _copy_layer_registry_to_registry | 20:04 |
| EmilienM | yeah it's a good idea | 20:04 |
| stevebaker | EmilienM: shall I do that right now? | 20:04 |
| EmilienM | stevebaker: you're already on it ! | 20:04 |
| *** wolverineav has quit IRC | 20:05 | |
| EmilienM | I prefer this idea than retrying *all* the times | 20:05 |
| *** wolverineav has joined #tripleo | 20:06 | |
| *** waleedm has quit IRC | 20:06 | |
| *** wolverineav has quit IRC | 20:11 | |
| *** paramite has quit IRC | 20:13 | |
| mwhahaha | beagles: also for whatever reason it's failing with resources.ServiceChain: Property error: resources[46].properties: Property DockerOvnMetadataImage not assigned by default too | 20:16 |
| beagles | mwhahaha, ack - not sure why that is yet | 20:18 |
| *** dciabrin has quit IRC | 20:20 | |
| *** khyr0n has joined #tripleo | 20:20 | |
| *** dciabrin has joined #tripleo | 20:21 | |
| *** dciabrin has quit IRC | 20:21 | |
| *** ekultails has quit IRC | 20:22 | |
| beagles | mwhahaha, do you know if there is a reason we are setting NeutronNetworkType in network environment files? | 20:22 |
| beagles | seems like it shouldn't be necessary | 20:22 |
| * mwhahaha shrugs | 20:22 | |
| mwhahaha | history? | 20:22 |
| mwhahaha | :D | 20:22 |
| beagles | yeah | 20:22 |
| *** waleedm has joined #tripleo | 20:23 | |
| beagles | do you mind if this patch removes them? | 20:23 |
| *** dciabrin has joined #tripleo | 20:23 | |
| * mwhahaha shrugs | 20:23 | |
| mwhahaha | what's the default it would use if not set there? | 20:23 |
| mwhahaha | geneve,vlan? | 20:23 |
| beagles | just geneve as it happens | 20:23 |
| *** ekultails has joined #tripleo | 20:23 | |
| mwhahaha | so i think vlan was historically done for network isolation | 20:23 |
| * mwhahaha sees if there's any git log history | 20:24 | |
| beagles | oh right, this is for the overcloud network | 20:24 |
| mwhahaha | https://github.com/openstack/tripleo-heat-templates/commit/53ff813d95f464999d6b37b00572fe1ad58bcebb | 20:24 |
| mwhahaha | from 2016 | 20:24 |
| beagles | well actually wait.. | 20:24 |
| mwhahaha | just some "sane" defaults | 20:24 |
| beagles | ah | 20:25 |
| beagles | mmm | 20:25 |
| mwhahaha | stevebaker: so we need to have the ovn container bits on by default, where did we bury that code that excluded/include it base on the neutron mechanismdrivers? cause it doesn't seem to be working anymore | 20:27 |
| beagles | mwhahaha, I'll leave the vlan in for now because people might actually be relying on it being there | 20:28 |
| beagles | (without realizing even) | 20:28 |
| stevebaker | mwhahaha: it sounds like the logic in this function needs to change if ovn is now the default https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/kolla_builder.py#L108 | 20:28 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates master: Change vxlan to geneve for network environment files https://review.openstack.org/638253 | 20:29 |
| mwhahaha | stevebaker: we set NeutronMechanismDrivers: ovn in the overcloud-resource-registry tho | 20:29 |
| mwhahaha | doesn't seem to be honoring that | 20:29 |
| tonyb | I have a problem with current RDO where after undercloud install adds the ctrlplane bridge it isn't adding the default gw which means tings fail (like time sync) | 20:30 |
| tonyb | my undercloud.conf worked on rocky did anythign change in there that may have caused this and I need to update? | 20:31 |
| *** dprince has quit IRC | 20:31 | |
| stevebaker | mwhahaha: yeah, that should be fixed. Here is another place where default behaviour might need to be updated https://github.com/openstack/tripleo-common/blob/master/container-images/overcloud_containers.yaml.j2#L528-L562 | 20:31 |
| mwhahaha | tonyb: we don't manage the default gw in the udnercloud install (never did?) | 20:32 |
| tonyb | mwhahaha: Hmm interesting, I would have thought you needed to when you add IPs etc to the bridge | 20:32 |
| tonyb | I'll poke at it when I get another system to test on | 20:33 |
| mwhahaha | tonyb: by default we assume the provisioning interface is the secondary interface | 20:33 |
| mwhahaha | tonyb: so we dont' manage the default route | 20:33 |
| mwhahaha | tonyb: if you are using only a single nic you'd need an advance network config | 20:33 |
| mwhahaha | tonyb: we add a bridge to the 2nd ip (eth1) and add the undercloud ip to that | 20:34 |
| *** dprince has joined #tripleo | 20:36 | |
| tonyb | mwhahaha: Hmm okay, can you point me at the docs for the advance network config? | 20:37 |
| *** jtomasek has quit IRC | 20:37 | |
| mwhahaha | not really? | 20:37 |
| mwhahaha | there's some info in the sample undercloud.conf | 20:37 |
| tonyb | mwhahaha: like I said this setup worked on rocky, I'm happy to adjust | 20:37 |
| tonyb | mwhahaha: okay | 20:37 |
| tonyb | I'll go looking | 20:37 |
| mwhahaha | tonyb: what is your setup? 1 nic or two | 20:37 |
| mwhahaha | the udnercloud should be the same between rocky/master | 20:38 |
| *** pbandark has quit IRC | 20:38 | |
| mwhahaha | unless you were testing with instack-undercloud | 20:38 |
| tonyb | mwhahaha: 2 NICS eth0 is provisioniing (and default gateway) eth1 is management | 20:38 |
| mwhahaha | that shouldn't have been workin in rocky, that's the problem | 20:38 |
| tonyb | mwhahaha: nope it was config-download in my rocky testing | 20:38 |
| mwhahaha | weird | 20:38 |
| tonyb | mwhahaha: Okay so I've just been lucky until now ;P | 20:39 |
| mwhahaha | probably | 20:39 |
| mwhahaha | we assume the provisioning network isn't the default route | 20:39 |
| mwhahaha | i think it's always been that way | 20:39 |
| *** dprince has quit IRC | 20:39 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Remove ties between ceilometer and panko https://review.openstack.org/638068 | 20:39 |
| mwhahaha | but prior to rocky we didn't completely redo the networking like we do with the containerized bits | 20:39 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Remove ties between ceilometer and panko https://review.openstack.org/636657 | 20:39 |
| openstackgerrit | Merged openstack/tripleo-common stable/rocky: Stops growth of massive dentry cache growth https://review.openstack.org/638051 | 20:39 |
| openstackgerrit | Merged openstack/tripleo-common stable/queens: Stops growth of massive dentry cache growth https://review.openstack.org/636660 | 20:39 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: flatten collectd service configuration https://review.openstack.org/636232 | 20:39 |
| mwhahaha | tonyb: it's just an os-net-config config, https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/config/standalone.py#L22 | 20:40 |
| mwhahaha | see net_config_override | 20:40 |
| mwhahaha | you'd need to pass the route | 20:40 |
| mwhahaha | we have an example with the standalone | 20:40 |
| * mwhahaha digs | 20:40 | |
| tonyb | mwhahaha: Thanks | 20:41 |
| mwhahaha | oh you could cheat with ControlPlaneStaticRoutes via a custome_env_file | 20:41 |
| *** wolverineav has joined #tripleo | 20:42 | |
| mwhahaha | tonyb: see 1 nic example from step 6 in https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/standalone.html | 20:42 |
| mwhahaha | ControlPlaneStaticRoutes would add a default gw there | 20:42 |
| *** spsurya has quit IRC | 20:42 | |
| mwhahaha | tonyb: you may try pinging hjenas as he wrote the advanced config bits | 20:42 |
| tonyb | mwhahaha: Thanks I | 20:44 |
| tonyb | 'll try that next time I rebuild | 20:45 |
| fultonj | tonyb: so when you deploy your overcloud it fails because it has no gateway to the NTP server? | 20:45 |
| mwhahaha | no there's no gateway | 20:45 |
| tonyb | fultonj: correct | 20:45 |
| mwhahaha | thats because we use os-net-config to completely reconfigure stuff (so the default gw is removed) | 20:45 |
| tonyb | fultonj: it 'hangs' trying to time sync | 20:45 |
| tonyb | fultonj: if I ssh in and rum ip route add default ... the task carries on and life is happy | 20:46 |
| *** wolverineav has quit IRC | 20:46 | |
| fultonj | and 'ip route' returns no default route on the OC node | 20:47 |
| fultonj | until you add one | 20:47 |
| mwhahaha | we've historically assumes the management network was routable and was eth0 | 20:48 |
| mwhahaha | and that the provisioning network wasn't the one providing the default gw | 20:48 |
| mwhahaha | i think we just lucked out with instack-undercloud not touching the default route | 20:48 |
| tonyb | fultonj: correct | 20:49 |
| tonyb | mwhahaha: I think it's more than luck as something *must* have added the route when it created the bridge but I'll poke and see what I can find | 20:50 |
| mwhahaha | the only way it wouldn't have been luck is if we had a gateway config item and id on't believe we ever did | 20:50 |
| mwhahaha | oh yea we do | 20:51 |
| mwhahaha | tonyb: what's your undercloud.conf | 20:51 |
| mwhahaha | tonyb: gateway under the [ctlplane-subnet] should create it | 20:51 |
| mwhahaha | if you have gateway but it's not under ctlplane-subnet it's probably not getting applied | 20:51 |
| tonyb | mwhahaha: openstack-config --set undercloud.conf ctlplane-subnet cidr 192.168.0.0/24 | 20:52 |
| tonyb | openstack-config --set undercloud.conf ctlplane-subnet masquerade true | 20:52 |
| tonyb | openstack-config --set undercloud.conf ctlplane-subnet dhcp_start 192.168.0.64 | 20:52 |
| tonyb | openstack-config --set undercloud.conf ctlplane-subnet dhcp_end 192.168.0.127 | 20:52 |
| tonyb | openstack-config --set undercloud.conf ctlplane-subnet inspection_iprange 192.168.0.128,192.168.0.191 | 20:52 |
| tonyb | openstack-config --set undercloud.conf ctlplane-subnet gateway 192.168.0.254 | 20:52 |
| *** raildo has quit IRC | 20:52 | |
| mwhahaha | so that should work i believe | 20:53 |
| mwhahaha | i'd file a bug and ping hjenas | 20:53 |
| tonyb | mwhahaha: okay cool | 20:53 |
| *** wolverineav has joined #tripleo | 20:54 | |
| *** raildo has joined #tripleo | 20:54 | |
| *** dciabrin has quit IRC | 21:00 | |
| *** dciabrin has joined #tripleo | 21:01 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Improve retry behaviour of _copy_layer_registry_to_registry https://review.openstack.org/638266 | 21:02 |
| *** xek_ has joined #tripleo | 21:03 | |
| *** raildo has quit IRC | 21:04 | |
| *** xek has quit IRC | 21:04 | |
| mwhahaha | stevebaker: so for the overcloud deploy where is the tripleo initial container images prepare launched if not provided via cli? is that in mistral? | 21:07 |
| stevebaker | mwhahaha: for overcloud prepare is run twice. Once in mistral to populate the heat image parameters, and once as an external deploy task to transfer the images. Let me dig up links | 21:08 |
| mwhahaha | i think i need the first one | 21:09 |
| mwhahaha | trying to track down why the NuetronMechanismDrivers from the plan is not being correctly honored | 21:09 |
| mwhahaha | (also we clearly do not test the "default" case in CI) | 21:09 |
| stevebaker | mwhahaha: the action https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/container_images.py#L136 | 21:09 |
| * mwhahaha blame EmilienM | 21:09 | |
| stevebaker | mwhahaha: the workflow https://github.com/openstack/tripleo-common/blob/master/workbooks/deployment.yaml#L168 | 21:10 |
| * mwhahaha shakes fist at mistral | 21:11 | |
| *** sdake has joined #tripleo | 21:12 | |
| jaosorior | sshnaidm: I'll check it out tomorrow. | 21:12 |
| *** waleedm has quit IRC | 21:13 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: standalone: switch container registry to ImageServce https://review.openstack.org/637861 | 21:16 |
| *** xek_ has quit IRC | 21:19 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add base job for TripleO CI baremetal jobs https://review.openstack.org/636763 | 21:21 |
| *** mjturek has quit IRC | 21:26 | |
| openstackgerrit | Jill Rouleau proposed openstack/tripleo-heat-templates master: step4: flatten nova service configuration https://review.openstack.org/637230 | 21:29 |
| *** ansmith has quit IRC | 21:30 | |
| *** wolverineav has quit IRC | 21:34 | |
| *** wolverineav has joined #tripleo | 21:35 | |
| *** Vorrtex has quit IRC | 21:36 | |
| *** toure is now known as toure|gpne | 21:37 | |
| *** toure|gpne is now known as toure|gone | 21:37 | |
| *** sdake has quit IRC | 21:38 | |
| *** wolverineav has quit IRC | 21:39 | |
| *** sdake has joined #tripleo | 21:40 | |
| *** dsneddon has quit IRC | 21:40 | |
| *** dsneddon has joined #tripleo | 21:42 | |
| *** wolverineav has joined #tripleo | 21:46 | |
| *** sdake has quit IRC | 21:48 | |
| *** sdake has joined #tripleo | 21:50 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Create deployment/deprecated directory https://review.openstack.org/637890 | 21:58 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Deprecate Docker https://review.openstack.org/637915 | 21:58 |
| *** sdake has quit IRC | 21:59 | |
| *** sdake has joined #tripleo | 22:01 | |
| *** dciabrin has quit IRC | 22:05 | |
| openstackgerrit | Merged openstack/ansible-role-tripleo-modify-image master: Fix dev install (yum -y...) https://review.openstack.org/638174 | 22:10 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Add fedora-28 support to container-build job on fedora-28 with Buildah https://review.openstack.org/635991 | 22:16 |
| EmilienM | stevebaker: thanks for the patch | 22:16 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/636370 | 22:18 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/636370 | 22:18 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test $î - never merge https://review.openstack.org/636371 | 22:18 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test $î - never merge https://review.openstack.org/636372 | 22:18 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test $î - never merge https://review.openstack.org/636373 | 22:18 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test $î - never merge https://review.openstack.org/636374 | 22:18 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test $î - never merge https://review.openstack.org/636375 | 22:18 |
| EmilienM | I added your patch in Depends-On, let's see if it helped | 22:18 |
| *** wolverineav has quit IRC | 22:19 | |
| *** artom has quit IRC | 22:19 | |
| *** wolverineav has joined #tripleo | 22:20 | |
| openstackgerrit | Juan Badia Payno proposed openstack/ansible-role-openstack-operations master: WIP - ansible-pacemaker fix https://review.openstack.org/638143 | 22:20 |
| stevebaker | EmilienM: cool | 22:20 |
| *** mcornea has quit IRC | 22:20 | |
| *** wolverineav has quit IRC | 22:21 | |
| *** ekultails has quit IRC | 22:22 | |
| *** wolverineav has joined #tripleo | 22:23 | |
| mwhahaha | stevebaker: so we don't bother doing the lookup for ovn/odl unless you have a ContainerImagePrepare set | 22:23 |
| mwhahaha | (this is likely also larsks's problem) | 22:23 |
| *** artom has joined #tripleo | 22:24 | |
| *** moshele has quit IRC | 22:24 | |
| stevebaker | mwhahaha: ah, that could be fixable. But he should be setting a ContainerImagePrepare | 22:25 |
| mwhahaha | stevebaker: by default we return here, https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/kolla_builder.py#L140-L142 | 22:25 |
| mwhahaha | so we never get to the set_neutron_Drive ron L154 | 22:25 |
| mwhahaha | if we're expecting containerimageprepare to exist we should fail hard | 22:26 |
| mwhahaha | i thought it was optional because it would just use the defaults if we don't provide one | 22:27 |
| *** rcernin has joined #tripleo | 22:27 | |
| *** wolverineav has quit IRC | 22:28 | |
| stevebaker | mwhahaha: so really the problem is that populating the defaults doesn't take NeutronMechanismDrivers into account | 22:29 |
| stevebaker | there is no defaults which will just work because we have conditional logic in our overcloud_containers.yaml.j2 | 22:30 |
| *** moshele has joined #tripleo | 22:31 | |
| mwhahaha | so perhaps we should just fail if ContainerImagePrepare is not set with a note about it being required and to go run image prepare default | 22:31 |
| *** dciabrin has joined #tripleo | 22:31 | |
| mwhahaha | i think is case is that he's setting an array and we don't check for an array | 22:32 |
| stevebaker | mwhahaha: so if OVN is the default now, our generated defaults should reflect that. As for larsks's specific situation, he should follow the docs for generating a file which contains a ContainerImagePrepare | 22:32 |
| mwhahaha | looks like he is setting a ContainerImagePrepare | 22:33 |
| larsks | stevebaker: I think I may already have followed those instructinos, but cip was explicitly setting ovn to null because the service was set to os::heat::none until I included that specific ovn environment file. | 22:33 |
| mwhahaha | maybe his problem is the array for NeutronMechanismDrivers | 22:33 |
| openstackgerrit | Juan Badia Payno proposed openstack/ansible-role-openstack-operations master: Fix the gate ansible-pacemaker (ansible module) installed as a module some minors fix at restore_galera.yml https://review.openstack.org/638143 | 22:34 |
| stevebaker | mwhahaha: hey, aren't we dropping ODL support and making OVN mandatory? If so we can eventually remove all neutron_driver things | 22:34 |
| openstackgerrit | Juan Badia Payno proposed openstack/ansible-role-openstack-operations master: Fix the gate https://review.openstack.org/638143 | 22:34 |
| mwhahaha | I don't know that ovn is mandatory tho | 22:35 |
| mwhahaha | it is the default | 22:35 |
| EmilienM | it's not mandatory | 22:35 |
| EmilienM | it's just replacing ML2 OVS for deafult | 22:35 |
| larsks | stevebaker: the ovn services were being explicilty rejected by the generated service filter, created by https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/kolla_builder.py#L81 | 22:35 |
| stevebaker | larsks: interesting, maybe this block needs some scrutiny to ensure params and services matches what is in tripleo-heat-templates https://github.com/openstack/tripleo-common/blob/master/container-images/overcloud_containers.yaml.j2#L705-L736 | 22:37 |
| larsks | stevebaker: I wasn't sure what was supposed to be enabled by default (or not). That is, I'm not sure I was hitting a bug of if I including the ovn environment file was in fact the correct solution. | 22:37 |
| * larsks has to go make dinner | 22:38 | |
| mwhahaha | larsks: we've switched to ovn by default on master as of last week so there's likely some fallout for this change. I was just hitting something similar but i was incorrectly not including any ContainerImagePrepare for an overcloud deploy | 22:38 |
| mwhahaha | but the standalone deploy instructions tells you to do that byd efault | 22:39 |
| stevebaker | larsks: ok, I'll go through your email reply and see what I find | 22:39 |
| mwhahaha | the missing ContainerImagePrepare appears to have been my problem | 22:40 |
| *** wolverineav has joined #tripleo | 22:41 | |
| mwhahaha | i still think we need to fail differently | 22:41 |
| openstackgerrit | John Fulton proposed openstack-infra/tripleo-ci master: Add LVM option for ceph-loop-device role https://review.openstack.org/637196 | 22:42 |
| stevebaker | mwhahaha: yeah, failing hard is a option, but at the time I wanted the JustWorks user experience since that is what we had pre-containers. The early image prepare approach was mandatory and the feedback was ... not good | 22:43 |
| mwhahaha | well if it's not provided there can we default to what is returned via the image prepare default? | 22:43 |
| mwhahaha | cause that seems to be working fine | 22:43 |
| mwhahaha | it's just that we don't bother doing the rest of the code to lookup the ovn stuff and destinations | 22:44 |
| mwhahaha | i'll poke at an option later | 22:44 |
| jidar | does anybody know what the deploy based options are I need to enable to get multipath to work on my controllers? | 22:44 |
| stevebaker | yes I think that is reasonable | 22:44 |
| *** wolverineav has quit IRC | 22:47 | |
| *** wolverineav has joined #tripleo | 22:47 | |
| *** wolverineav has quit IRC | 22:48 | |
| *** wolverineav has joined #tripleo | 22:48 | |
| *** ansmith has joined #tripleo | 22:49 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Use the default ContainerImagePrepare if not provided https://review.openstack.org/638286 | 22:51 |
| mwhahaha | stevebaker: i think -^ that's all we have to do but i'll test it out later | 22:51 |
| *** moshele has quit IRC | 22:53 | |
| *** tkajinam has joined #tripleo | 22:55 | |
| *** wolverineav has quit IRC | 23:00 | |
| *** wolverineav has joined #tripleo | 23:03 | |
| *** rcernin has quit IRC | 23:06 | |
| *** rcernin has joined #tripleo | 23:08 | |
| jidar | https://bugzilla.redhat.com/show_bug.cgi?id=1584225 oh :( | 23:09 |
| openstack | bugzilla.redhat.com bug 1584225 in openstack-tripleo-heat-templates "Adding Support for Multipath Deployments" [Unspecified,New] - Assigned to cinder-bugs | 23:09 |
| *** wolverineav has quit IRC | 23:09 | |
| *** wolverineav has joined #tripleo | 23:09 | |
| jidar | I guess I'm going to have to write some NodeExtraConfigPost | 23:10 |
| *** rascasoft has quit IRC | 23:12 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-quickstart-extras master: WIP/DNM: Add split-controlplane-standalone.yml playbook https://review.openstack.org/617368 | 23:12 |
| openstackgerrit | John Fulton proposed openstack/tripleo-quickstart master: WIP/DNM: Add featureset 59 for split controlplane https://review.openstack.org/615987 | 23:12 |
| openstackgerrit | John Fulton proposed openstack-infra/tripleo-ci master: WIP/DNM: Add split-controlplane standalone non-voting job definition https://review.openstack.org/615988 | 23:12 |
| *** rlandy is now known as rlandy|bbl | 23:14 | |
| *** khyr0n has quit IRC | 23:17 | |
| *** agopi has quit IRC | 23:17 | |
| *** wolverineav has quit IRC | 23:24 | |
| *** vinaykns has quit IRC | 23:27 | |
| *** wolverineav has joined #tripleo | 23:29 | |
| *** wolverineav has quit IRC | 23:34 | |
| *** wolverineav has joined #tripleo | 23:34 | |
| *** wolverineav has quit IRC | 23:46 | |
| *** wolverineav has joined #tripleo | 23:47 | |
| *** gfidente|afk has quit IRC | 23:47 | |
| *** wolverineav has quit IRC | 23:48 | |
| *** wolverineav has joined #tripleo | 23:48 | |
| *** jobewan has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!