| *** vnogin has joined #openstack-ansible | 00:06 | |
| logan- | jralbert: set your external vip address to the hostname | 00:07 |
|---|---|---|
| jralbert | really? The docs specify IPs everywhere, so I kinda assumed that was a requirement | 00:07 |
| *** vnogin has quit IRC | 00:10 | |
| jralbert | @logan- thanks for that! | 00:12 |
| jralbert | Okay, a separate question: is anybody running multiple Horizons for different domains in a multi-domain environment? We'd like to segregate some clients from others without making them aware of other domains. | 00:34 |
| *** jamesdenton has quit IRC | 00:36 | |
| *** macza has joined #openstack-ansible | 00:47 | |
| *** macza has quit IRC | 00:52 | |
| *** mmercer has quit IRC | 00:52 | |
| *** vnogin has joined #openstack-ansible | 01:06 | |
| *** vnogin has quit IRC | 01:11 | |
| *** mino_ has joined #openstack-ansible | 01:29 | |
| *** jamesdenton has joined #openstack-ansible | 01:46 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: Revert "Revert "Work around nested virt issues on OVH test nodes"" https://review.openstack.org/604719 | 01:47 |
| *** jralbert has quit IRC | 01:56 | |
| *** macza has joined #openstack-ansible | 02:18 | |
| *** spatel has joined #openstack-ansible | 02:20 | |
| spatel | cloudnull: ^^ | 02:21 |
| spatel | do you guys disable KSM on production compute node? echo 0 >/sys/kernel/mm/ksm/run | 02:21 |
| *** macza has quit IRC | 02:22 | |
| * prometheanfire has noticed perf inconsistancies with KSM | 02:23 | |
| prometheanfire | haven't used it in a while though | 02:23 |
| logan- | spatel: just so you know, newer versions of OSA disable ksm by default https://github.com/openstack/openstack-ansible-os_nova/blob/1c222d60b35189db90506554a80f24d881451c37/defaults/main.yml#L192-L193 | 02:35 |
| spatel | logan-: good to know!! and glad to see you! | 02:36 |
| *** vnogin has joined #openstack-ansible | 02:37 | |
| spatel | I was reading somewhere disabling KSM cause as not merging the pages can cause increase in memory usage and lead to swapping; which impacts the performance | 02:37 |
| logan- | it can impact performance both ways. like you said, more free memory means less swapping and more cache space. but ksm is not a free operation because it has to scan the memory chunks and dedupe them | 02:40 |
| logan- | and enabling it comes with some security implications also | 02:41 |
| *** vnogin has quit IRC | 02:42 | |
| spatel | hmm! i think disable would be idle option | 02:47 |
| spatel | logan-: while you here can you tell me how to calculate HugePage for 64GB host memory? | 02:47 |
| spatel | If i have 64GB compute node and i want to enable HugePage in that case i have two choice 2M and 1G so in that case how any pages i should pick? | 02:48 |
| spatel | i want to run apache web servers as guest VM on these compute node | 02:49 |
| *** Bhujay has joined #openstack-ansible | 02:49 | |
| spatel | if i go with 1G pagesize then should i go with 64 pages to cover all memory | 02:50 |
| logan- | not sure. i haven't had to deal with hugepages much | 02:50 |
| spatel | There isn't any good info on internet also | 02:50 |
| openstackgerrit | bhujay kumar proposed openstack/openstack-ansible-os_ironic stable/rocky: Adding [service_catalog] in ironic.conf (50 character) https://review.openstack.org/605311 | 02:57 |
| openstackgerrit | bhujay kumar proposed openstack/openstack-ansible-os_ironic stable/queens: Adding [service_catalog] in ironic.conf (50 character) https://review.openstack.org/605313 | 02:57 |
| *** Bhujay has quit IRC | 03:07 | |
| *** dave-mccowan has quit IRC | 03:44 | |
| *** spatel has quit IRC | 03:45 | |
| *** dave-mccowan has joined #openstack-ansible | 03:46 | |
| *** vnogin has joined #openstack-ansible | 03:48 | |
| *** vnogin has quit IRC | 03:53 | |
| *** udesale has joined #openstack-ansible | 03:53 | |
| *** vnogin has joined #openstack-ansible | 04:08 | |
| *** macza has joined #openstack-ansible | 04:09 | |
| *** vnogin has quit IRC | 04:13 | |
| *** pcaruana has joined #openstack-ansible | 04:14 | |
| *** shyamb has joined #openstack-ansible | 04:26 | |
| *** shyamb has quit IRC | 04:31 | |
| *** shyamb has joined #openstack-ansible | 04:34 | |
| *** pcaruana has quit IRC | 04:38 | |
| *** maddtux has joined #openstack-ansible | 04:59 | |
| *** dxiri has joined #openstack-ansible | 05:00 | |
| *** dxiri has quit IRC | 05:22 | |
| *** maddtux has quit IRC | 05:22 | |
| *** maddtux_ has joined #openstack-ansible | 05:24 | |
| *** maddtux_ is now known as maddtux | 05:25 | |
| *** Bhujay has joined #openstack-ansible | 05:26 | |
| *** macza_ has joined #openstack-ansible | 05:28 | |
| *** shyamb has quit IRC | 05:28 | |
| *** macza has quit IRC | 05:32 | |
| *** Bhujay has quit IRC | 05:32 | |
| *** macza_ has quit IRC | 05:33 | |
| *** shyamb has joined #openstack-ansible | 05:36 | |
| *** dave-mccowan has quit IRC | 05:39 | |
| *** pcaruana has joined #openstack-ansible | 05:43 | |
| *** Bhujay has joined #openstack-ansible | 05:49 | |
| *** shyamb has quit IRC | 06:09 | |
| *** DanyC has joined #openstack-ansible | 06:10 | |
| *** shyamb has joined #openstack-ansible | 06:11 | |
| *** DanyC has quit IRC | 06:31 | |
| *** shyamb has quit IRC | 06:38 | |
| *** phasespace has joined #openstack-ansible | 06:39 | |
| *** chkumar|off is now known as chkumar|ruck | 06:44 | |
| *** vnogin has joined #openstack-ansible | 06:49 | |
| *** vnogin has quit IRC | 06:55 | |
| *** shyamb has joined #openstack-ansible | 06:58 | |
| *** olivierb has joined #openstack-ansible | 07:13 | |
| *** olivierb has quit IRC | 07:17 | |
| *** olivierb has joined #openstack-ansible | 07:17 | |
| *** vnogin has joined #openstack-ansible | 07:19 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_hosts master: tasks: lxc_install_zypper: Use the 'state' option for external repos https://review.openstack.org/605102 | 07:23 |
| *** shardy has joined #openstack-ansible | 07:23 | |
| *** vnogin has quit IRC | 07:25 | |
| *** shyamb has quit IRC | 07:29 | |
| *** shyamb has joined #openstack-ansible | 07:35 | |
| *** mino_ has quit IRC | 07:43 | |
| *** shyamb has quit IRC | 07:45 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_blazar stable/rocky: Fix functional tests for Blazar https://review.openstack.org/604813 | 07:50 |
| openstackgerrit | Merged openstack/openstack-ansible-os_blazar stable/rocky: use include_tasks instead of include https://review.openstack.org/604814 | 07:52 |
| openstackgerrit | Merged openstack/openstack-ansible-os_blazar stable/rocky: Add the MQ vhost/user creation and configuration https://review.openstack.org/604815 | 07:52 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer master: Trivial: Fix the pep8 warning https://review.openstack.org/589033 | 07:57 |
| *** vnogin has joined #openstack-ansible | 08:00 | |
| *** vnogin has quit IRC | 08:05 | |
| *** mino_ has joined #openstack-ansible | 08:11 | |
| *** mino__ has joined #openstack-ansible | 08:13 | |
| *** mino_ has quit IRC | 08:15 | |
| *** gkadam has joined #openstack-ansible | 08:16 | |
| *** Emine has joined #openstack-ansible | 08:17 | |
| *** egonzalez has joined #openstack-ansible | 08:20 | |
| egonzalez | hi, anyone able to talk about ansible plugins license https://github.com/openstack/openstack-ansible-plugins | 08:21 |
| egonzalez | So, as first item, plugins are GPL license and not compatible with openstack | 08:22 |
| egonzalez | I see in OSA plugins repository many licensed as apache | 08:22 |
| *** hamzaachi has joined #openstack-ansible | 08:23 | |
| *** cshen has joined #openstack-ansible | 08:24 | |
| egonzalez | Is there a reason? Some plugins are compatible? Import this is possible under apache license https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/action/__init__.py? | 08:24 |
| *** DanyC has joined #openstack-ansible | 08:25 | |
| *** shyamb has joined #openstack-ansible | 08:26 | |
| *** shardy has quit IRC | 08:35 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: Update messaging notification configuration https://review.openstack.org/604789 | 08:35 |
| *** shardy has joined #openstack-ansible | 08:36 | |
| *** olivier__ has joined #openstack-ansible | 08:46 | |
| *** olivierb has quit IRC | 08:46 | |
| *** tosky has joined #openstack-ansible | 08:48 | |
| jrosser | egonzalez: this may be helpful for you https://github.com/ansible/ansible/issues/8864 | 08:50 |
| jrosser | that addresses modules specifically, you'd have to look at the imported code for plugins? | 08:51 |
| *** vnogin has joined #openstack-ansible | 08:53 | |
| *** vnogin has quit IRC | 08:54 | |
| *** chkumar|ruck has quit IRC | 08:58 | |
| *** hamzaachi_ has joined #openstack-ansible | 09:01 | |
| *** hamzaachi has quit IRC | 09:02 | |
| *** vnogin has joined #openstack-ansible | 09:03 | |
| *** vnogin has quit IRC | 09:08 | |
| ioni | hello guys | 09:11 |
| ioni | until what release of openstack ansible you will guys support ubuntu 16.04? | 09:12 |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: Revert "Revert "Work around nested virt issues on OVH test nodes"" https://review.openstack.org/604719 | 09:16 |
| *** hamzaachi_ has quit IRC | 09:18 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_hosts master: tasks: lxc_install_zypper: Use the 'state' option for external repos https://review.openstack.org/605102 | 09:19 |
| egonzalez | jrosser: thanks a lot for the link | 09:20 |
| odyssey4me | ioni: rocky will be the transition release that supports both xenial and bionic, but xenial support will be removed in stein | 09:23 |
| ioni | odyssey4me, thanks | 09:24 |
| *** electrofelix has joined #openstack-ansible | 09:26 | |
| Bhujay | irosser : not sure how it got added during the commit , but i had overlooked those words in the commit message | 09:34 |
| Bhujay | jrosser , ^ | 09:39 |
| *** priteau has joined #openstack-ansible | 09:41 | |
| *** gameon has quit IRC | 09:46 | |
| *** shardy is now known as shardy_mtg | 09:48 | |
| *** shyamb has quit IRC | 09:51 | |
| jrosser | Bhujay: can you edit those out (either in the gerrit UI or push a fixed version)? | 09:51 |
| *** vnogin has joined #openstack-ansible | 09:53 | |
| Bhujay | jrosser, thanks for pointing out the error and your reply . Can you suggest what should i do on the original commit that has been merged in master already ? | 09:54 |
| jrosser | Bhujay: it's too late for the master one now, ideally it would have been caught in the reviews. As it's really a cosmetic thing rather than being incorrect it can probably stay, unless anyone else feels strongly about it | 09:57 |
| *** vnogin has quit IRC | 09:58 | |
| Bhujay | jrosser , ok , I will wait for other reviewers comment . The other option could be correct the master with another patch and then backport | 10:02 |
| *** shyamb has joined #openstack-ansible | 10:04 | |
| Bhujay | unless there is some other gerrit magic you can advise that can solve this easily | 10:04 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible master: [DO NOT MERGE] SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/604366 | 10:09 |
| *** shardy_mtg has quit IRC | 10:11 | |
| *** shyamb has quit IRC | 10:16 | |
| *** mino__ has quit IRC | 10:19 | |
| *** hamzaachi has joined #openstack-ansible | 10:20 | |
| hwoarang | i would appreciate some reviews on https://review.openstack.org/#/q/status:open+branch:master+topic:osa-add-leap-150 (nova is passing too) and https://review.openstack.org/#/c/604473/ to make some progress with leap 15. | 10:21 |
| *** vnogin has joined #openstack-ansible | 10:22 | |
| *** jonher has quit IRC | 10:26 | |
| *** hamzaachi has quit IRC | 10:26 | |
| *** vnogin has quit IRC | 10:27 | |
| *** hamzaachi has joined #openstack-ansible | 10:30 | |
| *** hamzaachi has quit IRC | 10:35 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_heat master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/605389 | 10:46 |
| *** cshen has quit IRC | 10:48 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_horizon master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/605390 | 10:48 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_designate master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/605391 | 10:52 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_aodh master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/605392 | 10:53 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_barbican master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/605393 | 10:54 |
| *** shyamb has joined #openstack-ansible | 10:55 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_gnocchi master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/605394 | 10:56 |
| *** cshen has joined #openstack-ansible | 10:56 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_ceilometer master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/605395 | 10:57 |
| odyssey4me | hwoarang: jrosser made a good observation in https://review.openstack.org/604695 | 11:01 |
| jrosser | i checked the build log and it is indeed installing the packages over and over | 11:01 |
| hwoarang | odyssey4me: jrosser: state 'present' should ensure that they are only being installed once | 11:02 |
| hwoarang | also this happens because both containers have the same physical host | 11:02 |
| hwoarang | if they are on different hosts, then it should do the right thing no? | 11:03 |
| odyssey4me | hwoarang: the thing is that if there are two containers, then that task will delegate twice to the same host at the same time, and that will deadlock the package manager on at least one of those executions. | 11:03 |
| hwoarang | how do you propose to solve that? | 11:04 |
| hwoarang | serial: 1 is per play | 11:04 |
| hwoarang | not per task | 11:04 |
| odyssey4me | unfortunately run_once won't work there - because it needs to run once per physical host | 11:04 |
| odyssey4me | I guess there'd need to be a set fact task which puts together a list of the first container for each physical host, then only execute that task when the container name is in that list. | 11:06 |
| hwoarang | i could just install the apparmor package on lxc_host directly? | 11:07 |
| odyssey4me | The set_fact could run_once, and delegate to localhost. | 11:07 |
| hwoarang | lxc_host role i mean | 11:07 |
| odyssey4me | Well, yes - except that the packages need to be on the host that's going to hold the neutron agents, right? | 11:07 |
| hwoarang | for now | 11:07 |
| odyssey4me | And that's not guaranteed to be a LXC host. | 11:08 |
| hwoarang | right | 11:08 |
| hwoarang | ok | 11:08 |
| *** udesale has quit IRC | 11:12 | |
| *** pcaruana has quit IRC | 11:15 | |
| *** phasespace has quit IRC | 11:21 | |
| *** olivier__ has quit IRC | 11:28 | |
| *** olivierb has joined #openstack-ansible | 11:28 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: Remove mnaio_hosts group and use vm_hosts only https://review.openstack.org/605401 | 11:30 |
| *** oanson has quit IRC | 11:35 | |
| *** vnogin has joined #openstack-ansible | 11:51 | |
| *** mma has joined #openstack-ansible | 11:54 | |
| *** jonher has joined #openstack-ansible | 11:54 | |
| *** shyamb has quit IRC | 11:55 | |
| *** vnogin has quit IRC | 11:58 | |
| *** shyamb has joined #openstack-ansible | 12:00 | |
| *** strattao has joined #openstack-ansible | 12:12 | |
| logan- | why do we need to delegate to physical host | 12:15 |
| logan- | either apparmor is running because neutron is running on metal | 12:15 |
| logan- | or it is not running because its containerized | 12:15 |
| logan- | if its not running, then the broken profile has no effect right? | 12:15 |
| hwoarang | logan-: apparmor can run on physical host and neutron can be in container | 12:16 |
| hwoarang | if apparmor is running on physical host then it also affects processes within the container | 12:16 |
| logan- | the physical host will load profiles from the container? | 12:16 |
| hwoarang | no but the loaded profiles on the physical host affect the container | 12:17 |
| odyssey4me | we could simplify this by hard stopping a deployment if the agents are in a container, then we only support tunning them outside a container | 12:17 |
| odyssey4me | *running | 12:17 |
| *** cshen has quit IRC | 12:18 | |
| logan- | if something installs haproxy on the host and does not disable the profile, that is not os_neutron's problem? | 12:18 |
| logan- | the only other place we isntall haproxy is haproxy_server, and if that is installed on metal, it will disable the profile on the metal host | 12:18 |
| hwoarang | haproxy could be shipped in the image for example | 12:18 |
| hwoarang | in our tests, some neutron things are on localhost, some other on containers | 12:18 |
| *** vollman has joined #openstack-ansible | 12:18 | |
| hwoarang | those who are on localhost, install haproxy, and the loaded haproxy profile affects the container | 12:19 |
| *** shardy has joined #openstack-ansible | 12:19 | |
| hwoarang | but it's not just haproxy | 12:20 |
| hwoarang | dnsmasq ships by default with apparmor so that's loaded too | 12:20 |
| hwoarang | so you need to manage it on the physical host | 12:20 |
| nicolasbock | > ok maybe [nicolasbock](https://matrix.to/#/@nicolasbock:matrix.org) can help. i am not familiar with this particular suse ci instance | 12:20 |
| nicolasbock | hwo | 12:20 |
| *** cshen has joined #openstack-ansible | 12:22 | |
| hwoarang | nicolasbock: what?^ | 12:22 |
| logan- | i get your point that some neutron agents are containerized, and some not, but that just shows my point... there is no need to complicate anything with delegation. the agents that install on metal will disable the apparmor haproxy profile on metal, the ones in containers do not need to delegate | 12:24 |
| nicolasbock | Sorry hwoarang I am using matrix.io now to chat here and apparently I am still used to using hexchat ;) | 12:24 |
| nicolasbock | All I was trying to say was that I am looking into the SUSE CI thing | 12:24 |
| mgariepy | morning everyone | 12:25 |
| hwoarang | logan-: but you assume that something installs on metal. what if everything is in a container and the physical OS has apparmor running? | 12:25 |
| nicolasbock | Speaking of which, I don't understand why Zuul is flagging https://review.openstack.org/#/c/598143/ as failed on SUSE CI. When I look at https://build.opensuse.org/project/show/home:suse-cloud-ci:rpm-packaging-sles12-Master-b9014e0e0bb67dc40cdd1c84e89157f39df3a17a all builds are marked as succeeded. | 12:25 |
| nicolasbock | hwoarang: Could you explain to me why we are fixing the version of rabbitmq-server to the one in packagecloud.io and not use the one that's in `Cloud:OpenStack:Master` in OBS? | 12:27 |
| hwoarang | nicolasbock: we are not. we use the distro one. why do you think we use the packagecloud? | 12:29 |
| *** chkumar|ruck has joined #openstack-ansible | 12:30 | |
| *** udesale has joined #openstack-ansible | 12:30 | |
| nicolasbock | Maybe I am just confused, but I am seeing this error: fatal: [aio1_rabbit_mq_container-1d51e303]: FAILED! => {"changed": false, "msg": "To install a new major/minor version of RabbitMQ set '-e rabbitmq_upgrade=true'."} | 12:30 |
| hwoarang | i need more context :) | 12:30 |
| logan- | hwoarang: in that case there is nothing to do because the profile installed by os_neutron is not loaded | 12:30 |
| hwoarang | logan-: i don't follow :( | 12:31 |
| nicolasbock | In the rabbitmq_upgrade_check.yml playbook there is: - not installed_rabbitmq.stdout | search(rabbitmq_package_version) | 12:32 |
| nicolasbock | And the package version seems to be the one from packagecloud.io | 12:32 |
| nicolasbock | But then again, I might be confused here ;) | 12:32 |
| logan- | If it is all containerized then there is no profile present on the host (installed by os_neutron) that can be disabled | 12:32 |
| hwoarang | logan-: but apparmor could be part of the base OS image. so even if neutron just does not install appamor* stuff, then the base OS maybe already have that installed, up and running and loaded | 12:33 |
| hwoarang | nicolasbock: it's hard to follow :) maybe we are using the packagecloud version for the tests so we can try upgrades. COM only provides a single version. I need again more context (version, what are you trying to do, possibly logs) otherwise if you think there is a problem please open a bug so i can look at it :) | 12:34 |
| *** maddtux has quit IRC | 12:47 | |
| *** shyamb has quit IRC | 12:49 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_neutron master: AppArmor: Handle required profiles for neutron services https://review.openstack.org/604695 | 12:51 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_neutron master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/604286 | 12:51 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_heat master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/605389 | 12:58 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Remove mnaio_hosts group and use vm_hosts only https://review.openstack.org/605401 | 13:01 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Fix file-backed VM deployment https://review.openstack.org/605413 | 13:01 |
| nicolasbock | hwoarang: I am re-running setup-hosts.yml in an AIO on master and am getting this error | 13:05 |
| nicolasbock | After adding a debug statement right before the failed task I am getting: rabbitmq_package_version is set to rabbitmq-server-3.7.5-1.suse.noarch | 13:06 |
| hwoarang | nicolasbock: do you override the rabbitmq_install_method variable? | 13:06 |
| nicolasbock | But the container has version rabbitmq-server-3.6.16-111.4.x86_64 | 13:06 |
| nicolasbock | Not that I am aware of :) | 13:06 |
| nicolasbock | Where is it? | 13:06 |
| hwoarang | that's defined in the rabbitmq_server role | 13:07 |
| hwoarang | hmm i see what's going on | 13:08 |
| nicolasbock | No I am not overriding it | 13:08 |
| hwoarang | one sec | 13:08 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-rabbitmq_server master: rabbitmq_upgrade_check: Do not complains for upgrades on distro packages https://review.openstack.org/605417 | 13:12 |
| hwoarang | nicolasbock: maybe this will fix it ^. It's a bogus check i think | 13:12 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-rabbitmq_server master: rabbitmq_upgrade_check: Do not complain for upgrades on distro packages https://review.openstack.org/605417 | 13:13 |
| nicolasbock | I'll give it a try hwoarang :) | 13:19 |
| nicolasbock | Thanks! | 13:19 |
| ioni | does somebody have a fresh instalation of queens ? | 13:20 |
| ioni | when i updated from pike to queens I ended up with 3 containers for heat | 13:21 |
| ioni | ctrl1_heat_api_container-2441b13d ctrl1_heat_apis_container-5bc187b8 ctrl1_heat_engine_container-bfed23a2 | 13:21 |
| ioni | which ones are ok? | 13:22 |
| ioni | api or apis? | 13:22 |
| *** chkumar|ruck is now known as chandankumar | 13:26 | |
| logan- | ioni: a lot of container juggling during the queens upgrade. see the 'cleanup' playbooks https://github.com/openstack/openstack-ansible/tree/stable/queens/scripts/upgrade-utilities/playbooks | 13:31 |
| ioni | logan-, pretty sure there is no such thing in the manual | 13:33 |
| ioni | https://docs.openstack.org/openstack-ansible/queens/admin/upgrades/major-upgrades.html | 13:33 |
| logan- | https://docs.openstack.org/openstack-ansible/queens/admin/upgrades/major-upgrades.html#clean-up-unnecessary-containers | 13:33 |
| logan- | at the bottom :) | 13:33 |
| ioni | logan-, right, i didn't reach the end yet :D | 13:34 |
| ioni | i through they are the beginning | 13:34 |
| logan- | no, the cleanup happens at the end to ensure that the replacement containers are fully deployed first | 13:34 |
| *** hamzaachi has joined #openstack-ansible | 13:36 | |
| ioni | logan-, ok | 13:37 |
| ioni | thanks | 13:37 |
| *** hamzaachi has quit IRC | 13:37 | |
| ioni | currently testing the upgrade to rocky | 13:37 |
| ioni | :D | 13:37 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Fix file-backed VM deployment https://review.openstack.org/605413 | 13:38 |
| logan- | ioni: nice! will be interested to hear how your bionic upgrade goes | 13:41 |
| ioni | logan-, well, bionic is not a priority for me now | 13:41 |
| ioni | i'll run it on xenial with bionic kernel | 13:42 |
| logan- | ah | 13:42 |
| *** lbragstad has quit IRC | 13:45 | |
| arxcruz | odyssey4me: so, stackviz is working, now the problem is that the files are being gzipped on logs, and so it's not rendering anything | 13:49 |
| arxcruz | but the data is being collected properly | 13:49 |
| *** lbragstad has joined #openstack-ansible | 13:50 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Fix file-backed VM deployment https://review.openstack.org/605413 | 13:51 |
| openstackgerrit | Taseer Ahmed proposed openstack/openstack-ansible master: Integrate Blazar with OpenStack Ansible https://review.openstack.org/549956 | 13:51 |
| openstackgerrit | Marc Gariépy (mgariepy) proposed openstack/openstack-ansible-ceph_client master: Fix apt pinning for ceph https://review.openstack.org/605429 | 13:55 |
| mgariepy | logan-, https://review.openstack.org/605429 | 13:55 |
| odyssey4me | arxcruz: ah, we can fix that in the tests/integrated repo | 13:56 |
| odyssey4me | arxcruz: adjust this find to exclude that folder: https://github.com/openstack/openstack-ansible-tests/blob/master/test-log-collect.sh#L161-L168 | 13:56 |
| arxcruz | odyssey4me: yup, i will :) | 13:56 |
| logan- | mgariepy: ugh they changed it again :( | 13:56 |
| odyssey4me | arxcruz: do the same for this one: https://github.com/openstack/openstack-ansible/blob/master/scripts/scripts-library.sh#L130-L136 | 13:57 |
| mgariepy | fun times with upgrading haha | 13:58 |
| arxcruz | odyssey4me: so, it would be easier to just gunzip the stackviz dir at the end no? because stackviz static html files has js, html, css, and fonts | 13:59 |
| arxcruz | and the find command find everything except those types | 13:59 |
| odyssey4me | arxcruz: well, I'd prefer to just exclude that folder from the find if possible | 13:59 |
| arxcruz | odyssey4me: good for me | 13:59 |
| odyssey4me | rather than exclude the file type, exclude the folder | 13:59 |
| *** mma has quit IRC | 13:59 | |
| arxcruz | make sense | 14:01 |
| mgariepy | logan-, are you using uca or ceph.com ? | 14:04 |
| *** spatel has joined #openstack-ansible | 14:05 | |
| *** vnogin has joined #openstack-ansible | 14:12 | |
| logan- | ceph.com | 14:13 |
| *** udesale has quit IRC | 14:15 | |
| *** vnogin has quit IRC | 14:17 | |
| *** olivierb has quit IRC | 14:18 | |
| *** spatel has quit IRC | 14:19 | |
| *** spatel has joined #openstack-ansible | 14:19 | |
| spatel | jamesdenton: morning! | 14:20 |
| spatel | i am going to put SR-IOV VM in production to see how it doing.. | 14:20 |
| jamesdenton | sounds good! | 14:20 |
| *** olivierb has joined #openstack-ansible | 14:23 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove glance CORS overrides https://review.openstack.org/605437 | 14:26 |
| ioni | spatel, what cards do you have? | 14:27 |
| spatel | 03:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10) | 14:28 |
| *** dxiri has joined #openstack-ansible | 14:28 | |
| spatel | Qlogic | 14:28 |
| ioni | spatel, cool. i choose for my next region to use solarflare cards | 14:29 |
| ioni | spatel, but i'm a bit nervous to see how they are working | 14:30 |
| spatel | first time i am putting something on SR-IOV | 14:30 |
| ioni | spatel, i'll ask you tomorrow how is going | 14:30 |
| spatel | why tomorrow i am putting load on it in 30 min | 14:31 |
| spatel | if you around you will get to know | 14:31 |
| ioni | cool | 14:31 |
| spatel | also my customer load is low at present but still too high for VM | 14:31 |
| spatel | lets see | 14:31 |
| ioni | how much was the card? | 14:32 |
| spatel | i don't but i check on ebay few month back price was around $250 | 14:32 |
| ioni | ok, it's 10G only? | 14:32 |
| spatel | yes 10G | 14:33 |
| ioni | ok | 14:33 |
| ioni | mine was around 350$ but is dual 40G | 14:33 |
| spatel | we had packet loss on tap interface so today i am going to move that traffic on SR-IOV | 14:33 |
| ioni | right | 14:34 |
| openstackgerrit | Olivier Bourdon proposed openstack/openstack-ansible master: Fix healthcheck-hosts to work when behind a proxy https://review.openstack.org/605440 | 14:34 |
| spatel | soon i am going to test on DPDK and compare performance between all | 14:35 |
| ioni | spatel, i had once this problem it was due the linux bridge couldn't handle the traffic | 14:35 |
| kaiokmo | hwoarang: did you face any troubles running tests in leap-15? I had to "disable_gpg_check" to a couple of zypper tasks | 14:35 |
| spatel | even OVS can't handle traffic | 14:35 |
| ioni | spatel, DPDK cannot be used with SR-IOV? | 14:35 |
| ioni | it's DPDK or SR-IOV? | 14:36 |
| spatel | Yes DPDK + OVS | 14:36 |
| *** strattao has quit IRC | 14:38 | |
| *** strattao has joined #openstack-ansible | 14:42 | |
| hwoarang | kaiokmo: no but you probablu need to use the gpg_import_keys or something | 14:43 |
| hwoarang | for zypper_repository module | 14:43 |
| hwoarang | but i need more specific information/logs in order for me to look at it | 14:43 |
| *** Bhujay has quit IRC | 14:44 | |
| *** hamzaachi has joined #openstack-ansible | 14:44 | |
| *** pcaruana has joined #openstack-ansible | 15:01 | |
| *** mma has joined #openstack-ansible | 15:06 | |
| *** mma has quit IRC | 15:10 | |
| spatel | I put customer traffic on SR-IOV and so far so good, they will be peak again tomorrow 9pm so they would be fun to watch | 15:10 |
| *** cshen has quit IRC | 15:10 | |
| spatel | ioni: jamesdenton ^^ | 15:10 |
| jamesdenton | great! thanks for the update | 15:11 |
| spatel | current udp rate is 45kpps | 15:11 |
| *** vnogin has joined #openstack-ansible | 15:11 | |
| spatel | sorry they will be peak tomorrow 9:00AM EST morning | 15:11 |
| spatel | will give you more update tomorrow | 15:12 |
| jamesdenton | k | 15:12 |
| spatel | Let me put that server IP in smokeping so i can catch packetloss | 15:12 |
| spatel | jamesdenton: i am reading you book and love it fresh contents... i will post my review wherever possible! | 15:13 |
| ioni | spatel, what book? :D | 15:14 |
| jamesdenton | oh cool! i appreciate any feedback you can give me | 15:14 |
| spatel | you don't know :O | 15:14 |
| ioni | i don't know | 15:14 |
| ioni | care to share the name of the book? | 15:15 |
| spatel | let me do that hold on | 15:15 |
| spatel | https://www.amazon.com/Learning-OpenStack-Networking-technologies-OpenStack-based-ebook/dp/B07FDHXZYG?keywords=neutron+openstack&qid=1537974903&sr=8-8&ref=sr_1_8 | 15:15 |
| spatel | Must have book!! if you care about Openstack networking!!! | 15:15 |
| ioni | spatel, ok, i have openstack cloud computing cookbook | 15:16 |
| *** vnogin has quit IRC | 15:16 | |
| spatel | I have that one too, but Openstack networking has lots of meat :) | 15:16 |
| ioni | it didn't help me much that particular book | 15:16 |
| ioni | i'll order that one then | 15:17 |
| spatel | Openstack cloud computing give you good start with OSA | 15:17 |
| nicolasbock | Purely hypothetically, let's say I deployed an AIO and then added designate and re-ran the hosts, infrastructure, and openstack playbooks. | 15:18 |
| nicolasbock | Then I find that the python-designateclient was not install on the host | 15:18 |
| nicolasbock | Do I need to run anything else for that to happen? | 15:18 |
| jamesdenton | is it not installed in the utility container? | 15:18 |
| ioni | spatel, i'm using ASO since mitaka :) | 15:18 |
| noonedeadpunk | nicolasbock: I think you should look for clients inside utility contianer | 15:19 |
| nicolasbock | Let me check jamesdenton | 15:19 |
| spatel | I have two cloud one build by packstack and other one build by OSA | 15:19 |
| nicolasbock | Yes, awesome, it's in there! | 15:20 |
| nicolasbock | Maybe I am just misunderstanding how to drive an AIO | 15:20 |
| nicolasbock | Do I run the openstack client from within the utility container? | 15:20 |
| jamesdenton | Interfacing with openstack via clients should be done in the Utility container | 15:20 |
| nicolasbock | I was running it on the host so far | 15:20 |
| jamesdenton | yep | 15:20 |
| nicolasbock | Ah, ok, my bad then | 15:20 |
| nicolasbock | Thanks for the help! | 15:20 |
| jamesdenton | it's all good. you can do it from the host, too, but that may require some extra steps. Utility is the way to go | 15:21 |
| noonedeadpunk | nicolasbock: and clients aren't deployed to the deployment host itself. And it's not only about AIO, but regular deployment as well | 15:21 |
| *** chandankumar is now known as chkumar|off | 15:26 | |
| nicolasbock | Thanks noonedeadpunk | 15:30 |
| mnaser | i am so over rabbitmq | 15:35 |
| *** kstev has joined #openstack-ansible | 15:40 | |
| *** vnogin has joined #openstack-ansible | 15:41 | |
| nicolasbock | Another more general question: Can I run only the rabbitmq-server role? And if so how? | 15:41 |
| odyssey4me | nicolasbock: within an integrated build, or for some other purpose? | 15:42 |
| nicolasbock | In the AIO. For some reason the apparmor haproxy is still enforced and I would like to debug it | 15:43 |
| odyssey4me | nicolasbock: yeah, the utility container is meant to be used for administration purposes | 15:43 |
| odyssey4me | nicolasbock: ok, you can just run the rabbitmq-install.yml playbook | 15:43 |
| odyssey4me | nicolasbock: if you look at the setup-* playbooks, all they do is include other playbooks - any one of those can be run individually | 15:43 |
| nicolasbock | Awesome, thanks! | 15:44 |
| odyssey4me | noonedeadpunk: well, for rocky onwards we actually do install some clients on the deploy host - but you have to activate the ansible-runtime venv to use them | 15:44 |
| *** vnogin has quit IRC | 15:46 | |
| *** macza has joined #openstack-ansible | 15:46 | |
| *** Bhujay has joined #openstack-ansible | 15:46 | |
| mgariepy | 2018-09-26 09:46:39.141 27929 ERROR nova.objects.keypair [req-1e90793a-09bd-471a-a0b0-46a9ef599805 - - - - -] Some instances are still missing keypair information. Unable to run keypair migration at this time. | 15:46 |
| *** goldenfri has joined #openstack-ansible | 15:47 | |
| odyssey4me | mgariepy: that looks like a nice gift :p | 15:50 |
| mgariepy | any idea how to fix that ? | 15:50 |
| mgariepy | lol | 15:50 |
| mgariepy | just loop on this over an over. | 15:51 |
| spotz | Could you bring the instance down as an image insert it and then reupload it? | 15:51 |
| *** macza has quit IRC | 15:51 | |
| mgariepy | TASK [Perform online data migrations] ******************************************************************** | 15:52 |
| *** macza has joined #openstack-ansible | 15:52 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_horizon master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/605390 | 15:53 |
| odyssey4me | mgariepy: well, it's possible that some sort of work had to be done that hasn't been done - maybe try and do a nova status-check and see what it returns? | 15:53 |
| noonedeadpunk | odyssey4me: oh, didn't know that - I'm waiting for the release to start preparations for the upgrade | 15:53 |
| *** dave-mccowan has joined #openstack-ansible | 15:54 | |
| noonedeadpunk | so haven't enough time yet for looking into changelist for rocky:( | 15:55 |
| *** eglute has joined #openstack-ansible | 15:56 | |
| *** dave-mccowan has quit IRC | 15:57 | |
| openstackgerrit | bhujay kumar proposed openstack/openstack-ansible-os_ironic stable/rocky: Adding [service_catalog] in ironic.conf https://review.openstack.org/605311 | 15:57 |
| *** dxiri has quit IRC | 15:58 | |
| openstackgerrit | bhujay kumar proposed openstack/openstack-ansible-os_ironic stable/queens: Adding [service_catalog] in ironic.conf https://review.openstack.org/605313 | 15:59 |
| mgariepy | nova-status upgrade check seems ok | 16:00 |
| odyssey4me | mgariepy: perhaps best to ask the folks in #openstack-nova then | 16:08 |
| *** Leo_m has joined #openstack-ansible | 16:10 | |
| *** vnogin has joined #openstack-ansible | 16:10 | |
| *** vnogin has quit IRC | 16:15 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add additional API tests to confirm functionality https://review.openstack.org/605194 | 16:16 |
| *** Emine has quit IRC | 16:16 | |
| mgariepy | select count(instance_uuid) from instance_extra where keypairs is NULL; << return a few instances.. | 16:17 |
| hwoarang | mnaser: hello. what do you think about https://review.openstack.org/#/c/599425/ ? i have done many rechecks and the timings are close to ubuntu and with the recent changes it's even less than 2h | 16:21 |
| hwoarang | i meant, that the timeouts are similar to ubuntu. when ubuntu timeouts, opensuse timeouts too but overall seems fine :/ | 16:22 |
| *** gkadam has quit IRC | 16:25 | |
| *** Bhujay has quit IRC | 16:29 | |
| *** dxiri has joined #openstack-ansible | 16:35 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_horizon master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/605390 | 16:37 |
| *** hamzaachi has quit IRC | 16:40 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Only adjust user_variables.yml if pre_config_osa is true https://review.openstack.org/605484 | 16:41 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts master: tasks: lxc_install_zypper: Use the 'state' option for external repos https://review.openstack.org/605102 | 16:43 |
| mgariepy | after some updates in the DB, it seems that i should survive another day.. | 16:50 |
| *** evrardjp has quit IRC | 16:50 | |
| *** olivierb has quit IRC | 16:54 | |
| nicolasbock | Hi, another question of a more general nature: I rebooted an AIO and now can't talk to Keystone. To be more precise, I can curl the management bridge IP on port 5000 and am getting a proper response | 17:07 |
| nicolasbock | But when I curl the v3/tokens URI I am getting a 404 not found error | 17:07 |
| nicolasbock | Any ideas of what could be broken? | 17:08 |
| cloudnull | nicolasbock does the curl to 5000 return a 503 ? | 17:08 |
| nicolasbock | Oh wait, I finally realized that the keystone logs are in journalctl in the keystone container | 17:08 |
| nicolasbock | I am getting a DB error | 17:09 |
| nicolasbock | so maybe mariadb is not up | 17:09 |
| nicolasbock | Let me check that | 17:09 |
| nicolasbock | No, the database is showing that it's up and running | 17:12 |
| nicolasbock | Keystone is throwing: SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') | 17:14 |
| cloudnull | can you connect to the database from the keystone container | 17:16 |
| cloudnull | ? | 17:16 |
| cloudnull | direct from keystone to galera and then try keystone to galera via haproxy ? | 17:17 |
| nicolasbock | How do I do that? | 17:17 |
| nicolasbock | I mean from keystone to galera? | 17:17 |
| cloudnull | run `mysql -h $GALERA_IP -u keystone -p` | 17:18 |
| cloudnull | you'll have to get the password from the keystone config | 17:18 |
| nicolasbock | Ok, I have to install the mysql client first it seems | 17:19 |
| *** shardy has quit IRC | 17:20 | |
| nicolasbock | It works directly but fails via haproxy | 17:22 |
| cloudnull | is the the xinetd issue ? | 17:23 |
| mgariepy | is keepalived started ? | 17:24 |
| cloudnull | nicolasbock do you have this pr https://review.openstack.org/#/c/601468/ | 17:24 |
| odyssey4me | Folks, we have a pretty major bug for master/rocky when building a multi-node. A patch will be incoming shortly to fix it. | 17:25 |
| nicolasbock | cloudnull: Yes, that patch I have. I pulled from master last night | 17:25 |
| nicolasbock | And mgariepy no, keepalived is not running | 17:26 |
| nicolasbock | It's not enabled in systemd | 17:27 |
| mgariepy | if it's aio, i think it's not relevant sorry | 17:27 |
| odyssey4me | cloudnull: could you review https://review.openstack.org/604308 & https://review.openstack.org/#/q/I33f7994a50ba5fcea7fca171dfca6b4d32011900 | 17:27 |
| nicolasbock | No problem mgariepy | 17:27 |
| *** strattao has quit IRC | 17:30 | |
| *** strattao has joined #openstack-ansible | 17:31 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove keystone_admin-back load balancer manipulation https://review.openstack.org/605494 | 17:33 |
| odyssey4me | cores - please review https://review.openstack.org/605494 - it fixes a release blocker for rocky, so I'd like to get it merged asap | 17:34 |
| odyssey4me | meanwhile, I'll verify whether the same problem exists for rocky and work out a similar fix (in rocky we do have both back-ends) | 17:34 |
| nicolasbock | For some reason haproxy is not logging anything on the AIO host | 17:47 |
| nicolasbock | The directory /var/log/haproxy is empty | 17:47 |
| odyssey4me | nicolasbock: yep, that's a known thing - I think there is a patch up for that, but I haven't tried it | 17:48 |
| nicolasbock | Ok, but you don't think it's critical in a sense of that haproxy might not be working correctly? | 17:48 |
| odyssey4me | nicolasbock: https://review.openstack.org/587687 | 17:49 |
| odyssey4me | easiest way to check state for haproxy is to run 'hatop -s /var/run/haproxy.stat' on the load balancer host | 17:49 |
| odyssey4me | but yeah, t'd be nice to have haproxy logs | 17:49 |
| nicolasbock | Ah! | 17:51 |
| nicolasbock | The galera-back is down | 17:51 |
| nicolasbock | That explains it | 17:51 |
| nicolasbock | Now can I get it to tell me why? | 17:52 |
| *** cshen has joined #openstack-ansible | 17:53 | |
| odyssey4me | nicolasbock: well, it does a http check IIRC against the xinetd port exposed from the container to verify health - you could do the same and see what happens? | 17:53 |
| nicolasbock | Is the xinetd service supposed to be running? | 17:54 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible master: Add qdrouterd role for oslo.messaging backend deployment https://review.openstack.org/570261 | 17:54 |
| *** electrofelix has quit IRC | 17:56 | |
| *** DanyC has quit IRC | 17:56 | |
| *** vnogin has joined #openstack-ansible | 17:59 | |
| nicolasbock | The xinetd service wasn't running in the galera container | 18:01 |
| nicolasbock | I manually started it and now I can openstack token issue | 18:02 |
| nicolasbock | The systemd service is not enabled by the way | 18:02 |
| nicolasbock | That seems wrong, no? | 18:02 |
| *** vnogin has quit IRC | 18:03 | |
| openstackgerrit | Nicolas Bock proposed openstack/openstack-ansible-galera_server master: Enable the xinetd service for reboot https://review.openstack.org/605500 | 18:22 |
| nicolasbock | odyssey4me: Could you review ^^? | 18:23 |
| nicolasbock | Thanks! | 18:23 |
| nicolasbock | The AIO is rebooting much better now | 18:23 |
| *** pcaruana has quit IRC | 18:24 | |
| openstackgerrit | Nicolas Bock proposed openstack/openstack-ansible-galera_server master: Enable the xinetd service for reboot https://review.openstack.org/605500 | 18:24 |
| *** lbragstad has quit IRC | 18:24 | |
| *** lbragstad has joined #openstack-ansible | 18:25 | |
| *** vnogin has joined #openstack-ansible | 18:28 | |
| *** vnogin has quit IRC | 18:33 | |
| *** cshen has quit IRC | 18:34 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove keystone_admin-back load balancer manipulation https://review.openstack.org/605494 | 18:34 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Use loop_control for haproxy keystone back-end enablement https://review.openstack.org/605504 | 18:40 |
| odyssey4me | logan-: you broke it ;) | 18:41 |
| odyssey4me | https://review.openstack.org/#/q/I0d7e4804bdc92ffe7a679060a686e684c01fcd1b will fix it again, and https://review.openstack.org/605504 will need to be ported back to queens/pike/ocata | 18:41 |
| logan- | ah yep good catch thanks | 18:47 |
| odyssey4me | nicolasbock: good find, although I've asked for an adjustment in https://review.openstack.org/#/c/605500/2 | 18:49 |
| nicolasbock | Thanks | 18:50 |
| nicolasbock | I have replied :) | 18:50 |
| nicolasbock | I am looking into your suggestion now and why that didn't work for me when I tried it | 18:50 |
| odyssey4me | nicolasbock: ok, so the handler will only get called when that file changes - which is going to be a bit of an issue, because it won't - maybe your approach is best... | 18:51 |
| nicolasbock | Ok, I don't think I am qualified to answer whether it's best where I put it now 🙂 But it's working where i put it ;) | 18:52 |
| odyssey4me | nicolasbock: I've asked for a TODO note, and for it to be added to the handler as well. This task will only be to cater for upgrades from anything before the patch. | 18:53 |
| odyssey4me | This patch will also need to be ported back - probably all the way to ocata I think. | 18:54 |
| odyssey4me | yep, confirmed | 18:54 |
| odyssey4me | but we get it merged in master first | 18:54 |
| *** vnogin has joined #openstack-ansible | 18:58 | |
| openstackgerrit | Nicolas Bock proposed openstack/openstack-ansible-galera_server master: Enable the xinetd service for reboot https://review.openstack.org/605500 | 18:58 |
| nicolasbock | I'll backport it once it's merged in master | 18:59 |
| odyssey4me | perfect, thanks nicolasbock | 19:00 |
| *** vnogin has quit IRC | 19:02 | |
| *** mmercer has joined #openstack-ansible | 19:07 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_neutron master: AppArmor: Handle required profiles for neutron services https://review.openstack.org/604695 | 19:21 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_neutron master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/604286 | 19:21 |
| openstackgerrit | Merged openstack/openstack-ansible stable/queens: Remove broken uptime tests during upgrade https://review.openstack.org/605152 | 19:21 |
| *** Emine has joined #openstack-ansible | 19:26 | |
| odyssey4me | hmm, that's odd - I'm suddenly starting to get 'Exception: ERROR: The 'make' utility is missing from PATH' when building the ansible runtime venv | 19:28 |
| odyssey4me | this doesn't happen in the gate because we use pre-built wheels | 19:28 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: MNAIO: Remove mnaio_hosts group and use vm_hosts only https://review.openstack.org/605401 | 19:29 |
| ioni | i removed in rocky the repo container and trying to create it again with setup hosts and it fails to generate the network config | 19:33 |
| odyssey4me | ioni: did you remove the cached facts for thhe container first? | 19:34 |
| ioni | odyssey4me, hmm no. let me retry :D | 19:34 |
| ioni | i removed the container because repo install failed to molteniron git | 19:35 |
| ioni | but there was nowhere in rocky | 19:35 |
| ioni | it's an upgrade from queens to rocky | 19:35 |
| *** hamzaachi has joined #openstack-ansible | 19:35 | |
| ioni | and it tried to get the repo for molteniron stable/queens into rocky | 19:35 |
| ioni | so i wiped the container to start clean :) | 19:36 |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: Remove broken uptime tests during upgrade https://review.openstack.org/605153 | 19:43 |
| odyssey4me | ioni: molteniron has been removed from the integrated build - it's not an official project, and to be honest, we didn't think anyone was using it! | 19:45 |
| ioni | odyssey4me, i'm not using it | 19:46 |
| ioni | it just failed at repo build | 19:46 |
| ioni | but i don't know why | 19:46 |
| odyssey4me | oh, I think I know what happened - the old role is still there - we'll have to implement a cleanup task in the upgrade script | 19:46 |
| odyssey4me | could you register a bug for that? | 19:46 |
| ioni | odyssey4me, https://paste.xinu.at/J9Jzu/#n24 | 19:47 |
| ioni | right, so i have to delete from /etc/ansible/roles | 19:48 |
| ioni | that was the next step | 19:48 |
| odyssey4me | ioni: yep, rm -rf /etc/ansible/roles/os_molteniron | 19:49 |
| odyssey4me | if you could please register a bug with the symptoms, then it won't get forgotten | 19:49 |
| ioni | odyssey4me, https://bugs.launchpad.net/openstack-ansible/+bug/1794599 | 19:51 |
| openstack | Launchpad bug 1794599 in openstack-ansible "Rocky repo build fails when upgrading from queens" [Undecided,New] | 19:51 |
| odyssey4me | perfect, thank you :) | 19:51 |
| ioni | odyssey4me, i'm going to wipe everything from roles :D | 19:51 |
| ioni | just to be sure | 19:52 |
| ioni | this time i'm starting to upgrade testing early | 19:52 |
| *** cshen has joined #openstack-ansible | 19:54 | |
| odyssey4me | ioni: fantastic - it's really good to have early testing done | 19:54 |
| ioni | on the other hand, this is complely offtopic but maybe you guys know, i'm searching for cloud providers that can give bgp so i can i use my own network into vm | 19:55 |
| odyssey4me | ioni: FYI this will be needed if your testing is multi-node: https://review.openstack.org/605504 | 19:55 |
| ioni | i know VEXXHOST | 19:55 |
| ioni | odyssey4me, sadly no. I use a simple setup with a single keystone | 19:56 |
| odyssey4me | ok, then you should be fine | 19:56 |
| ioni | odyssey4me, any other ideas regarding networking for repo container? I removed the facts from /etc/openstack_deploy/ansible_facts | 20:01 |
| ioni | should I remove from inventory as well? | 20:01 |
| ioni | https://paste.xinu.at/TxM/ | 20:01 |
| mnaser | cloudnull: are you around? | 20:01 |
| cloudnull | kinda | 20:02 |
| cloudnull | lurking atm been in meetings all day | 20:02 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Use loop_control for haproxy keystone back-end enablement https://review.openstack.org/605517 | 20:03 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: Use loop_control for haproxy keystone back-end enablement https://review.openstack.org/605518 | 20:05 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Use loop_control for haproxy keystone back-end enablement https://review.openstack.org/605519 | 20:06 |
| odyssey4me | folks, given how badly broken the stable branches are without https://review.openstack.org/#/q/I0d7e4804bdc92ffe7a679060a686e684c01fcd1b I've done early backports to try and get them in asap before it hits too many deployments | 20:07 |
| cloudnull | mnaser need anything? | 20:09 |
| mnaser | cloudnull: sorry, didnt see reply, for elk, im thinking what do you think about instead of having all those checks | 20:09 |
| mnaser | we just 'enable' all those pipelnies and make variables to disable them by default if someone needs | 20:09 |
| mnaser | i dont think there's much harm in that... is there? | 20:09 |
| cloudnull | that might be a better (faster) way to do it | 20:10 |
| mnaser | there's bugs around it (i.e. nova doesnt create /var/log/nova because we dont put anything on metal) | 20:10 |
| cloudnull | the checks may be too conservative | 20:10 |
| mnaser | so control plane logs dont get shipped | 20:10 |
| mnaser | but neutron control plane do because we have /var/log/neutron on the host | 20:10 |
| cloudnull | jrosser ^ | 20:10 |
| mnaser | so im thinking of making that change and moving it to a group var | 20:11 |
| mnaser | enable * and allow disabling to be opt in | 20:11 |
| cloudnull | Id be good with taht | 20:11 |
| mnaser | at the same time i'll template it out because we repeat so many things in there | 20:11 |
| *** evrardjp has joined #openstack-ansible | 20:11 | |
| cloudnull | sweet! | 20:12 |
| jrosser | which is this? filebeat? | 20:12 |
| mnaser | jrosser: yes | 20:13 |
| *** strattao has left #openstack-ansible | 20:19 | |
| jrosser | sounds reasonable to enable it all so long as filebeat doesnt create it's own log spam for things it can't find | 20:19 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Automate the removal of the molteniron role https://review.openstack.org/605522 | 20:22 |
| jrosser | mnaser: theres also improvement in fact gathering could be done.... it's done twice, once at the start of each play and then straight away again in elastic_data_hosts, for me it's very slow | 20:22 |
| mnaser | jrosser: lets see what the warnings look like | 20:22 |
| odyssey4me | ioni: perhaps you could give https://review.openstack.org/605522 a try :) | 20:22 |
| mnaser | if its just on start up it might be ok | 20:22 |
| mnaser | i mean it does matching on globs | 20:23 |
| mnaser | so technically it just .. matches nothing | 20:23 |
| ioni | odyssey4me, well for now, i cannot create repo container :) | 20:23 |
| odyssey4me | mnaser: you probably want to restrict the fact gathering subset to exclude ohai/facter | 20:23 |
| ioni | odyssey4me, i don't think is even trying to create the network file | 20:23 |
| odyssey4me | https://github.com/openstack/openstack-ansible/blob/master/scripts/openstack-ansible.rc#L33 | 20:23 |
| ioni | i need to look in tasks | 20:24 |
| ioni | TASK [lxc_container_create : Remove legacy network config for eth0] | 20:24 |
| ioni | TASK [lxc_container_create : Create and start the container] | 20:24 |
| jrosser | odyssey4me: ahha i bet folks with rocky don't see the slowness i do on queens, its openstack-ansible vs. ELK embedded ansible | 20:24 |
| ioni | TASK [lxc_container_create : Gather container facts] | 20:24 |
| ioni | TASK [lxc_container_create : Drop container setup script] | 20:25 |
| jrosser | the latter wont be picking up that rc file | 20:25 |
| ioni | TASK [lxc_container_create : Drop container first run script] | 20:25 |
| ioni | TASK [lxc_container_create : Execute first script] | 20:25 |
| ioni | no network file creation | 20:25 |
| ioni | the last task fail for new container because it doesn't have network | 20:25 |
| ioni | and wiring is done after lxc_container_create : Execute first script | 20:26 |
| ioni | but is not ran for repo container | 20:26 |
| ioni | because is in error state | 20:26 |
| ioni | i think there is an ordering task bug | 20:27 |
| ioni | odyssey4me, ^ | 20:27 |
| odyssey4me | ioni: it should have networking? at the very least the eth0 should be there? | 20:28 |
| odyssey4me | could you take a look inside the vm? the networking should be setup using systemd | 20:28 |
| ioni | https://paste.xinu.at/TxM/ | 20:28 |
| ioni | this is the files for container | 20:28 |
| ioni | odyssey4me, https://paste.xinu.at/8dRC/ | 20:29 |
| ioni | no eth0 | 20:29 |
| ioni | just lo | 20:29 |
| ioni | i'll copy it from other container | 20:29 |
| odyssey4me | ioni: could you register a bug with steps to reproduce the issue? | 20:30 |
| *** cshen has quit IRC | 20:31 | |
| ioni | ok | 20:31 |
| odyssey4me | thanks, that'll be really helpful to work out what went wrong and fix it | 20:32 |
| *** cshen has joined #openstack-ansible | 20:32 | |
| *** priteau has quit IRC | 20:33 | |
| ioni | odyssey4me, https://bugs.launchpad.net/openstack-ansible/+bug/1794613 | 20:36 |
| openstack | Launchpad bug 1794613 in openstack-ansible "Rocky setup hosts fails to recreate destroyed container" [Undecided,New] | 20:36 |
| odyssey4me | ioni: great, thank you | 20:36 |
| ioni | odyssey4me, simple steps | 20:36 |
| ioni | destroy, recreate | 20:36 |
| ioni | i've done this multiple times with pike and queens | 20:37 |
| odyssey4me | I'm not going to be able to pick that up right now - I have a few too many irons in the fire. Hopefully someone else will pick it up soon. | 20:37 |
| odyssey4me | Yeah, we did a bunch of reworking in rocky, so it's entirely plausible that this is a bug that's crept in. | 20:37 |
| *** hamzaachi has quit IRC | 20:38 | |
| *** sawblade6 has joined #openstack-ansible | 20:48 | |
| *** spatel has quit IRC | 20:53 | |
| *** hamzaachi has joined #openstack-ansible | 20:54 | |
| jrosser | ioni: did container-first-run.sh make any helpful output? | 20:57 |
| ioni | jrosser, want the output? | 20:58 |
| jrosser | if theres something useful, add it to the bug report | 20:58 |
| ioni | jrosser, https://paste.xinu.at/z1bLsI/ | 20:59 |
| ioni | it just tries to run apt-get update and fails | 20:59 |
| *** sawblade6 has quit IRC | 21:03 | |
| *** nsmeds_ has joined #openstack-ansible | 21:08 | |
| *** DanyC has joined #openstack-ansible | 21:12 | |
| *** cshen has quit IRC | 21:12 | |
| *** hamzaachi has quit IRC | 21:13 | |
| *** nsmeds_ has quit IRC | 21:21 | |
| *** kstev has quit IRC | 21:32 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: MNAIO: Fix file-backed VM deployment https://review.openstack.org/605413 | 21:43 |
| *** mgariepy has quit IRC | 21:59 | |
| logan- | odyssey4me: which package were you seeing the 'make' error on? | 22:03 |
| odyssey4me | pyacl | 22:03 |
| odyssey4me | it was just transient though - a recheck worked just fine, I suspect my jobs hit a bad mirror or something | 22:04 |
| logan- | interesting, i wonder if there was an upstream change. i am seeing it in some non-osa jobs installing netbox | 22:04 |
| logan- | http://paste.openstack.org/raw/730969/ | 22:04 |
| odyssey4me | yup, that's exactly the same | 22:04 |
| odyssey4me | the thing is, they publish wheels, so you should never need to compile it | 22:05 |
| logan- | they released today | 22:05 |
| logan- | 1.3.0 2018-09-26 | 22:05 |
| odyssey4me | we're pinned to an earlier release though | 22:05 |
| odyssey4me | for OSA at least | 22:05 |
| logan- | are you sure? :) | 22:06 |
| odyssey4me | I see that you were downloading PyNaCl-1.3.0 | 22:06 |
| logan- | yeah | 22:06 |
| *** mmercer has quit IRC | 22:06 | |
| odyssey4me | we're pinned: https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L368 | 22:06 |
| *** mgariepy has joined #openstack-ansible | 22:13 | |
| *** cshen has joined #openstack-ansible | 22:24 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: Update all SHAs for 16.0.20 https://review.openstack.org/604624 | 22:29 |
| *** cshen has quit IRC | 22:29 | |
| *** tosky has quit IRC | 22:37 | |
| *** spatel has joined #openstack-ansible | 22:43 | |
| *** Leo_m has quit IRC | 22:47 | |
| *** spatel has quit IRC | 22:47 | |
| *** macza has quit IRC | 23:09 | |
| *** macza has joined #openstack-ansible | 23:10 | |
| *** macza has quit IRC | 23:14 | |
| *** DanyC has quit IRC | 23:19 | |
| *** macza has joined #openstack-ansible | 23:34 | |
| *** dxiri has quit IRC | 23:35 | |
| *** macza has quit IRC | 23:39 | |
| *** openstackgerrit has quit IRC | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!