| *** brad[] has quit IRC | 00:00 | |
| *** brad- has joined #openstack-ansible | 00:01 | |
| *** ansmith_ has quit IRC | 00:01 | |
| *** ansmith_ has joined #openstack-ansible | 00:31 | |
| *** goldyfruit has quit IRC | 00:42 | |
| *** goldyfruit has joined #openstack-ansible | 00:44 | |
| *** cloudnull has quit IRC | 01:04 | |
| *** cloudnull has joined #openstack-ansible | 01:05 | |
| *** ansmith_ has quit IRC | 01:09 | |
| *** spatel has joined #openstack-ansible | 01:13 | |
| *** openstackstatus has joined #openstack-ansible | 01:15 | |
| *** ChanServ sets mode: +v openstackstatus | 01:15 | |
| *** spatel has quit IRC | 01:18 | |
| *** udesale has joined #openstack-ansible | 04:09 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-ansible | 05:34 | |
| *** raukadah is now known as chkumar|rover | 05:44 | |
| *** errr has quit IRC | 05:55 | |
| *** errr has joined #openstack-ansible | 05:56 | |
| *** shyamb has joined #openstack-ansible | 06:16 | |
| *** pcaruana has joined #openstack-ansible | 06:25 | |
| *** Neurognostic_ has joined #openstack-ansible | 06:38 | |
| *** Neurognostic has quit IRC | 06:42 | |
| *** Neurognostic__ has joined #openstack-ansible | 06:44 | |
| *** Neurognostic has joined #openstack-ansible | 06:47 | |
| *** Neurognostic_ has quit IRC | 06:47 | |
| *** Neurognostic__ has quit IRC | 06:49 | |
| *** Neurognostic has quit IRC | 06:52 | |
| *** Neurognostic has joined #openstack-ansible | 06:52 | |
| *** Neurognostic has quit IRC | 06:53 | |
| *** Neurognostic has joined #openstack-ansible | 06:54 | |
| *** shyamb has quit IRC | 06:56 | |
| *** Neurognostic_ has joined #openstack-ansible | 07:01 | |
| *** Neurognostic has quit IRC | 07:04 | |
| *** miloa has joined #openstack-ansible | 07:05 | |
| *** Neurognostic_ has quit IRC | 07:07 | |
| *** elenalindq has joined #openstack-ansible | 07:11 | |
| *** shyamb has joined #openstack-ansible | 07:14 | |
| *** tosky has joined #openstack-ansible | 07:35 | |
| *** shyamb has quit IRC | 08:01 | |
| *** pcaruana has quit IRC | 08:35 | |
| *** ivve has joined #openstack-ansible | 08:36 | |
| *** cshen has joined #openstack-ansible | 08:40 | |
| *** shyamb has joined #openstack-ansible | 08:40 | |
| *** rpittau|afk is now known as rpittau | 08:41 | |
| *** maharg101 has joined #openstack-ansible | 08:51 | |
| *** DanyC has joined #openstack-ansible | 08:52 | |
| *** pcaruana has joined #openstack-ansible | 08:57 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: manila: stop including all hosts in haproxy https://review.opendev.org/706757 | 08:59 |
|---|---|---|
| janno | I am currently having a problem with the repo containers. Every request of a file "hangs", while navigating through directories is no problem. does this sound familiar to anyone? | 09:00 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: Add manila_service_region https://review.opendev.org/704957 | 09:01 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_manila master: cleanup: config values https://review.opendev.org/704951 | 09:03 |
| mnaser | oh for gods sake | 09:14 |
| mnaser | opensuse breaks $world once again | 09:14 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_nova master: pre-install: use path and drop dest https://review.opendev.org/705892 | 09:19 |
| *** sshnaidm has quit IRC | 09:41 | |
| *** sshnaidm has joined #openstack-ansible | 09:42 | |
| *** rgogunskiy has joined #openstack-ansible | 09:43 | |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Allow user override of beat install locations https://review.opendev.org/706528 | 09:59 |
| jrosser | mnaser: perhaps we should re-evaulate the suse support | 10:00 |
| jrosser | afaik we don't have any actual real world deployments? | 10:00 |
| jrosser | we're carrying n-v jobs all over the place too | 10:01 |
| *** shyamb has quit IRC | 10:04 | |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Expose options for Kibana load balancing https://review.opendev.org/706781 | 10:15 |
| *** shyamb has joined #openstack-ansible | 10:19 | |
| mnaser | jrosser: i dont know who runs openstack on suse | 10:31 |
| mnaser | let alone osa tbh | 10:31 |
| *** udesale has quit IRC | 10:48 | |
| *** shyamb has quit IRC | 10:58 | |
| *** dirk has joined #openstack-ansible | 11:14 | |
| *** ansmith_ has joined #openstack-ansible | 11:26 | |
| *** shyamb has joined #openstack-ansible | 11:28 | |
| *** Neurognostic has joined #openstack-ansible | 11:35 | |
| *** ansmith_ has quit IRC | 11:38 | |
| *** Neurognostic_ has joined #openstack-ansible | 11:38 | |
| *** Neurognostic_ has quit IRC | 11:39 | |
| *** Neurognostic_ has joined #openstack-ansible | 11:40 | |
| *** Neurognostic has quit IRC | 11:42 | |
| *** rgogunskiy has quit IRC | 11:48 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Add manila tempest testing https://review.opendev.org/674653 | 11:58 |
| *** nicolasbock has joined #openstack-ansible | 12:01 | |
| *** rgogunskiy has joined #openstack-ansible | 12:05 | |
| *** rpittau is now known as rpittau|bbl | 12:08 | |
| *** Neurognostic_ has quit IRC | 12:20 | |
| *** shyamb has quit IRC | 12:21 | |
| *** udesale has joined #openstack-ansible | 12:28 | |
| *** electrofelix has joined #openstack-ansible | 12:44 | |
| *** mgariepy has joined #openstack-ansible | 12:45 | |
| *** shyamb has joined #openstack-ansible | 12:59 | |
| openstackgerrit | Chenjun Shen proposed openstack/openstack-ansible master: Fix idempotency for modifying lxc config https://review.opendev.org/706823 | 13:06 |
| *** shyamb has quit IRC | 13:07 | |
| cshen | jrosser: my company used to run openstack on suse, but now only with osa. | 13:08 |
| mnaser | cshen: what os do you use? | 13:10 |
| cshen | Ubuntu only | 13:11 |
| *** rpittau|bbl is now known as rpittau | 13:12 | |
| *** vesper11 has quit IRC | 13:15 | |
| *** vesper has joined #openstack-ansible | 13:15 | |
| *** rh-jelabarre has joined #openstack-ansible | 13:17 | |
| *** aedc_ has joined #openstack-ansible | 13:20 | |
| *** aedc has quit IRC | 13:23 | |
| *** gary_perkins has joined #openstack-ansible | 13:26 | |
| mnaser | cshen: ah cool | 13:30 |
| *** aedc_ has quit IRC | 13:30 | |
| admin0 | as long as the price is good, suse will run anything :) | 13:31 |
| *** bhyrted has joined #openstack-ansible | 13:31 | |
| *** aedc_ has joined #openstack-ansible | 13:33 | |
| *** pcaruana has quit IRC | 13:39 | |
| *** ansmith_ has joined #openstack-ansible | 13:40 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_manila master: Add image download for manila https://review.opendev.org/704972 | 13:46 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_manila master: Add image download for manila https://review.opendev.org/704972 | 13:49 |
| *** mgariepy has quit IRC | 13:51 | |
| *** mgariepy has joined #openstack-ansible | 13:52 | |
| *** itandops has joined #openstack-ansible | 13:57 | |
| *** partlycloudy has quit IRC | 14:00 | |
| *** mjwales has joined #openstack-ansible | 14:05 | |
| *** rgogunskiy has quit IRC | 14:06 | |
| mjwales | On a fresh Rocky install keepalived is consuming large amounts of memory and causing other processes to then fail as the nodes runs out of memory. It looks like each keepalived process consumes 80MB RAM and there are ~1500 processes. Is this normal? | 14:06 |
| *** pcaruana has joined #openstack-ansible | 14:10 | |
| *** aedc_ has quit IRC | 14:24 | |
| admin0 | mjwales, not normal | 14:28 |
| admin0 | 1500 keepalive processes ? | 14:28 |
| admin0 | the max i see when i do ps ax | grep keepalive is 3 | 14:29 |
| admin0 | and this cluster is up for almost a year now | 14:30 |
| mjwales | Across the three control nodes keepalived has 267, 261 and 329. We rebooted/restarted each control node around 2 hours ago | 14:31 |
| *** mjwales has quit IRC | 14:31 | |
| *** mjwales has joined #openstack-ansible | 14:32 | |
| mjwales | admin0: all our tenant networks are configured with HA if that makes a difference | 14:33 |
| admin0 | is that keepalive for tenants ? | 14:33 |
| admin0 | or for the openstack APIs ? | 14:34 |
| admin0 | if for the openstack API endpoints, 1500 is still very high | 14:34 |
| mjwales | For the tenants (I think) an example is: keepalived -P -f /var/lib/neutron/ha_confs/393446c4-4e2d-473b-8129-8660abd74736/keepalived.conf -p /var/lib/neutron/ha_confs/393446c4-4e2d-473b-8129-8660abd74736.pid -r /var/lib/neutron/ha_confs/393446c4-4e2d-473b-8129-8660abd74736.pid-vrrp | 14:34 |
| admin0 | ok .. for the tenants | 14:35 |
| admin0 | then i guess you simply need to upgrade the RAM :) | 14:35 |
| cshen | it depends on how many tenant networks you have. | 14:35 |
| admin0 | or reduce the tenant networks | 14:35 |
| admin0 | if it needs ram, it needs ram | 14:35 |
| mjwales | There are currently ~200 tenant networks deployed. This worked fine on a 18.1.8 cluster but having issues on 18.1.17 | 14:35 |
| admin0 | haproxy version the same ? | 14:36 |
| mjwales | No longer have the old cluster for comparison after a failed minor version upgrade. This is a redeployment from scratch | 14:37 |
| mjwales | The three control nodes have 92GB RAM each | 14:38 |
| *** mjwales has quit IRC | 14:38 | |
| *** mjwales has joined #openstack-ansible | 14:39 | |
| admin0 | mjwales, i felt 92 gb of ram is not enough .. so i run 128gb each .. but to not have networking overload, i also have a provider network that provides direct DHCP addresses .. so there is no need to create floating ips unless needed .. and i make my computes ( not on scheduler) also my network nodes | 14:41 |
| admin0 | so i have not faced this type of issue of controllers running out of ram | 14:41 |
| *** peeejayz has joined #openstack-ansible | 14:42 | |
| mjwales | admin0, okay maybe more RAM would help. What I don't understand is why so many keepalived processes are running where there are only 200 tenant networks | 14:44 |
| admin0 | 200 x number of tenant networks right | 14:44 |
| admin0 | i mean replication | 14:44 |
| admin0 | mjwales, take one router, and then from your admin rc run neutron l3-agent-list-hosting-router $router-uuid and see how many routers are there | 14:46 |
| mjwales | Three routers per network | 14:47 |
| jamesdenton | IIRC there is a keepalive instance for every router using L3 HA, on every agent node involved. Check ps and see try to cross-reference the IDs to actual routers. And see what the outliers are | 14:47 |
| mjwales | What about neutron-keepalived-state-change? | 14:47 |
| *** spatel has joined #openstack-ansible | 14:49 | |
| jamesdenton | don't recall, how many do you see? | 14:49 |
| *** ivve has quit IRC | 14:51 | |
| mjwales | 216, 216, 814 across the three controllers respectively | 14:52 |
| *** mathlin has quit IRC | 14:54 | |
| jamesdenton | same, i guess 1 per router | 14:54 |
| jamesdenton | check this out: https://bugs.launchpad.net/neutron/+bug/1857422 | 14:55 |
| openstack | Launchpad bug 1857422 in neutron "neutron-keepalived-state-change and keeplived cannot be cleanup for those routers which is deleted during l3-agent died" [Undecided,New] | 14:55 |
| jamesdenton | i would try to see many of those processes are stale and tied to routers that may no longer exist | 14:56 |
| jamesdenton | also, check the l3-agent logs to see if there are any errors related to those processes | 14:56 |
| mjwales | Thanks. Making a guess should there be a single directory in /var/lib/neutron/ha_confs for each router? | 14:57 |
| jamesdenton | yes | 14:59 |
| mjwales | Okay so I have 838 so clearly something a miss as "openstack router list" only returns 215 | 14:59 |
| mjwales | Will cross reference to see which actually exist | 15:00 |
| *** djhankb has quit IRC | 15:00 | |
| jamesdenton | any tracebacks in the logs? | 15:02 |
| admin0 | 215 routers x times 3 = 645 of those already | 15:03 |
| admin0 | that could be active based on replication x 3 . plus all those other ones that are not cleared, deleted or stale | 15:03 |
| admin0 | you can sql query the active routers vs ip netns output to check for differences in the two | 15:04 |
| openstackgerrit | Merged openstack/openstack-ansible-ceph_client master: Be more specific about python binary to use https://review.opendev.org/706308 | 15:04 |
| openstackgerrit | Merged openstack/openstack-ansible master: Add manila_service_region https://review.opendev.org/704957 | 15:04 |
| openstackgerrit | Merged openstack/openstack-ansible stable/train: Install appropriate tempest releases on stable branches https://review.opendev.org/703572 | 15:04 |
| jrosser | oh my goodness it finally merged 8-O | 15:06 |
| mjwales | jamesdenton: trying to look through the logs now but they are huge! | 15:06 |
| *** KeithMnemonic has joined #openstack-ansible | 15:06 | |
| mjwales | admin0: its the stat routers that are causing the issue all 600 of them | 15:06 |
| mjwales | jamesdenton: nothing sticks out before the node runs out of memory | 15:09 |
| jamesdenton | odd, but ok. i would also check syslog while youre at it. But ultimately, gotta find out how many of those keepalived processes are tied to routers that no longer exist | 15:10 |
| openstackgerrit | yedhu proposed openstack/openstack-ansible master: Added regexp to avoid multiple entry of lxc.start.order in os-lxc-container-setup.yml https://review.opendev.org/706874 | 15:11 |
| mjwales | Is there a defined process for removing these routers and associated files in ha_confs given that they don't exist in OpenStack? | 15:12 |
| jamesdenton | Well, ideally the agent would clean them up | 15:12 |
| jamesdenton | once the router is deleted, all respective processes should go, too. | 15:12 |
| openstackgerrit | Merged openstack/openstack-ansible-os_manila master: Add manila CI https://review.opendev.org/675934 | 15:13 |
| *** mgariepy has quit IRC | 15:14 | |
| jrosser | jamesdenton: didnt we have issues with rootwrap config and the keepalived stuff? | 15:15 |
| itandops | Hi everyone, | 15:15 |
| jamesdenton | yeeeeeah that sounds familiar | 15:16 |
| mjwales | jamesdenton: That seems to be the problem. Many routers don't exist yet they still have a folder in ha_confs | 15:16 |
| jamesdenton | 18.1.17 you say? | 15:17 |
| mjwales | Yes, fresh deployment | 15:17 |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server master: Add NODENAME in rabbitmq env https://review.opendev.org/703577 | 15:18 |
| *** mgariepy has joined #openstack-ansible | 15:18 | |
| jamesdenton | jrosser I see your bug here: https://bugs.launchpad.net/openstack-ansible/+bug/1780733 | 15:18 |
| openstack | Launchpad bug 1780733 in openstack-ansible "Neutron rootwrap error" [High,Triaged] - Assigned to David Wilde (dave-wilde) | 15:18 |
| jrosser | yeah that was a looong time ago | 15:19 |
| mjwales | The only warnings that show up in the neutron log are: Bad interface line: ad_partner_oper_port_state 63: IndexError: list index out of range | 15:22 |
| mjwales | neutron-linuxbridge-agent.log to be exact | 15:22 |
| jamesdenton | yeah. you can ignore that | 15:23 |
| mjwales | Is there anyway to force the agent to resync? | 15:25 |
| jamesdenton | you can restart he agent, but it would likely not mess with any existing (stale) processes, since it doesn't know anything about them anymore | 15:26 |
| jamesdenton | it might be helpful to create a new router, and delete that router to see if things hang around and if there's any associated error | 15:26 |
| mjwales | Creating a router creates a directory in each of the control nodes (ha_confs) with a file state (each contains backup). Deleting the router removes the state file but leaves the directory behind | 15:31 |
| jamesdenton | is there still a process? | 15:32 |
| itandops | any suggestion to my questions guys ? | 15:33 |
| jamesdenton | itandops looks like your questions didnt come thru | 15:33 |
| mjwales | jamesdenton: no process | 15:34 |
| jamesdenton | mjwales that's good, i guess? I would have to check a similar environment to see if the directory is left behind. have to find one, first. Is this a prod env? | 15:35 |
| mjwales | This is prod | 15:36 |
| mjwales | The question is how to clean up all the routers that have been left behind that no longer exist in OpenStack? | 15:36 |
| mjwales | Otherwise the nodes are going to consume all their memory again | 15:37 |
| jamesdenton | for now you could try killing the stale processes. If you have a way to replicate the behavior, with steps, processes, etc. then i recommend opening a bug | 15:38 |
| mjwales | jamesdenton: thanks! Will do some cleaning up and see if the problem reoccurs | 15:38 |
| jamesdenton | having those keepalived processes hanging around after the router has been deleted is not (or should not be) expected behavior | 15:38 |
| itandops | jamesdenton My question is in case where we deploy behind a physical load balancer. | 15:47 |
| jamesdenton | did you have a specific question? If | 15:52 |
| jamesdenton | *if you did, i didnt see it | 15:52 |
| admin0 | mjwales, you can also say your compute nodes are also network nodes ( if networking is the same ) | 15:53 |
| admin0 | then your controllers will not get overloaded | 15:54 |
| admin0 | while you delete and add again | 15:54 |
| mjwales | admin0: will bear that in mind the infrastructure cannot be touch now for 3-4 months as it is used for teaching | 15:55 |
| admin0 | mjwales, do you have an extra node that you can make temporarily a network node ? | 15:55 |
| admin0 | just to make sure you do not run out of memory | 15:55 |
| admin0 | and distribute the routers to that node | 15:56 |
| mjwales | Not at the moment :( only the three controller nodes have an uplink to the outside world | 15:57 |
| admin0 | well, I thought of some options. A. add more ram to the controllers ( one at a time) B. reduce the HA mode to non HA .. C. if HA is required, maybe write a script and run via cron so that instead of 3, keep only 2 running D. buy a network card .. add it to one of the compute nodes, make it as network node, redistribute the routers so that full ram is not used E. rethink the networking .. maybe use direct DHCP so that you do not even | 15:59 |
| admin0 | have to create routers in the first place | 15:59 |
| admin0 | can't think of other ways on how i would approach this | 16:00 |
| mjwales | admin0: Thanks for the ideas! More RAM is definitely the easiest to do so will take a look down that route first :) | 16:01 |
| openstackgerrit | Merged openstack/openstack-ansible-os_manila master: refactor: service group names https://review.opendev.org/704945 | 16:03 |
| openstackgerrit | Merged openstack/openstack-ansible-os_manila master: cleanup: config values https://review.opendev.org/704951 | 16:03 |
| admin0 | we have a few OSA clusters serving 800+ people .. i never run network on my controllers .. i distribute them to compute nodes which are out of schedulers and where i manually migrate either big or very long running workloads .. that way, i know my controllers and network node keeps running fine | 16:03 |
| admin0 | and in other,which has almost no SLA, all compute nodes are also network nodes .. so there is never a big impact or a total impact | 16:03 |
| admin0 | best part is the use of provider network that is not floating ip but direct dhcp . this removes the need of creation of routers and namespaces and keeps the networking resources down | 16:05 |
| cshen | admin0: you meant that in your env, you don't use openstack router? | 16:06 |
| admin0 | it is used, but only by 20% of the people | 16:07 |
| admin0 | rest are happy to select corp-dhcp as network provider , get an IP they can ssh and get on with their workload | 16:07 |
| cshen | I see, the provider network is based on vlan? | 16:08 |
| admin0 | yep | 16:08 |
| admin0 | https://www.openstackfaq.com/openstack-add-direct-attached-dhcp-ip/ | 16:08 |
| cshen | thanks for the link. | 16:08 |
| admin0 | the .1 gateway is a vyos router that then has routing rules with the rest of the organizations, ipsec tunnels etc | 16:09 |
| *** partlycloudy has joined #openstack-ansible | 16:09 | |
| cshen | Ok | 16:09 |
| admin0 | if all that your users need is connectivity, then floating ip, network, router is an extra unnecessary step for them to do .. as well as extra resources for you to manage and maintain | 16:10 |
| admin0 | downside .. they delete the VM, the ip also go | 16:11 |
| admin0 | but we run designate, so when they create a vm say web, it becomes web.username.domain.name and they can just ping/ssh to that name instead of thinking about ips | 16:11 |
| admin0 | which has worked beautifully | 16:11 |
| *** chkumar|rover is now known as raukadah | 16:15 | |
| *** cshen has quit IRC | 16:18 | |
| *** gyee has joined #openstack-ansible | 16:18 | |
| *** udesale has quit IRC | 16:24 | |
| *** rh-jelabarre has quit IRC | 16:26 | |
| *** rh-jelabarre has joined #openstack-ansible | 16:32 | |
| *** itandops has quit IRC | 16:37 | |
| *** mjwales has quit IRC | 16:46 | |
| *** tosky has quit IRC | 16:48 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Increase logstash output connection timeout for beats https://review.opendev.org/706230 | 16:57 |
| *** miloa has quit IRC | 17:06 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Expose Elasticsearch data path configuration https://review.opendev.org/704770 | 17:07 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Exposed config for logstash elasticsearch endpoints https://review.opendev.org/705483 | 17:07 |
| *** rpittau is now known as rpittau|afk | 17:07 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Explicitly use Elasticsearch data node for ILM index update https://review.opendev.org/706014 | 17:09 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Update ILM policy if non-existent https://review.opendev.org/706015 | 17:14 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Propagated ILM changes to auditbeat install https://review.opendev.org/706069 | 17:14 |
| *** peeejayz has quit IRC | 17:17 | |
| *** pcaruana has quit IRC | 17:18 | |
| *** pcaruana has joined #openstack-ansible | 17:30 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_manila master: Add image download for manila https://review.opendev.org/704972 | 17:31 |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-ansible | 17:34 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:35 | |
| *** pcaruana has quit IRC | 17:42 | |
| *** electrofelix has quit IRC | 17:49 | |
| jrosser | \o/ looks like we are getting there for train fixes https://review.opendev.org/#/c/705336/ | 17:53 |
| jrosser | ^ reviews please before something else breaks :) | 17:53 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest stable/train: Revert "Mark ubuntu distro and debian stretch & centos as non voting" https://review.opendev.org/706944 | 17:54 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest stable/train: Revert "Mark ubuntu distro and debian stretch & centos as non voting" https://review.opendev.org/706944 | 17:54 |
| *** errr has quit IRC | 17:55 | |
| *** openstackstatus has quit IRC | 17:57 | |
| *** openstack has joined #openstack-ansible | 18:01 | |
| *** ChanServ sets mode: +o openstack | 18:01 | |
| openstackgerrit | kourosh vivan proposed openstack/openstack-ansible-rabbitmq_server stable/stein: Add NODENAME in rabbitmq env https://review.opendev.org/706948 | 18:13 |
| openstackgerrit | kourosh vivan proposed openstack/openstack-ansible-rabbitmq_server stable/train: Add NODENAME in rabbitmq env https://review.opendev.org/706949 | 18:14 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Exposed Kibana elastic endpoint https://review.opendev.org/706950 | 18:30 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Update journalbeat name to use hostname https://review.opendev.org/706951 | 18:32 |
| openstackgerrit | Sam Choraria proposed openstack/openstack-ansible-ops master: Allow beat processors to be defined through configuration data https://review.opendev.org/705965 | 18:36 |
| *** hwoarang has quit IRC | 18:49 | |
| *** hwoarang has joined #openstack-ansible | 18:52 | |
| *** ansmith_ has quit IRC | 19:11 | |
| *** ansmith_ has joined #openstack-ansible | 19:11 | |
| gillesMo | Hello. Python modules uWSGI and scrypt (at least) are not build with the correct libssl dependancy on Ubuntu 16.04 (while having a mixed repo nodes during OS upgrades). | 19:43 |
| openstackgerrit | Shannon Mitchell proposed openstack/openstack-ansible-os_designate master: Fix paging link protocol when behind haproxy https://review.opendev.org/706090 | 19:47 |
| jrosser | gillesMo: do you think that there are different versions of libssl on the 16.04 targets compared to the 16.04 repo server? | 19:47 |
| gillesMo | jrosser: No, I checked. It seems that ubuntu 16 venvs have benn built on ubuntu 18 repo container, But logs are purged, I'm not sure | 19:48 |
| openstackgerrit | Shannon Mitchell proposed openstack/openstack-ansible-os_designate master: Fix paging link protocol when behind haproxy https://review.opendev.org/706090 | 19:55 |
| jrosser | gillesMo: i guess it all comes down to this group being defined properly https://github.com/openstack/openstack-ansible/blob/stable/rocky/playbooks/repo-build.yml#L47 | 19:55 |
| jrosser | if i were in your position i would be prioritising getting everything to 18.04 to remove the mixed OS setup | 19:57 |
| mgariepy | gillesMo, wich tag ? | 20:12 |
| *** NewJorg has quit IRC | 20:14 | |
| *** NewJorg has joined #openstack-ansible | 20:14 | |
| mgariepy | gillesMo, i have a couple setup with mixed env. and i don't have/haven't seen issues so far. | 20:15 |
| *** DanyC has joined #openstack-ansible | 20:19 | |
| *** maharg101 has quit IRC | 20:34 | |
| gillesMo | jrosser: Yes, but... I have 7 compute nodes, 3 with local storage, not easy to do just now | 20:35 |
| gillesMo | mgariepy: 18.1.18 | 20:36 |
| gillesMo | I had not before, just when I do the mistake of upgrading from 18.1.11 to 18.1.18 | 20:37 |
| mgariepy | i'm on 18.1.11 | 20:38 |
| *** waxfire has quit IRC | 20:55 | |
| *** Darcidride has quit IRC | 21:28 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest stable/train: Always use virtualenv and pip to install tempest on Ubuntu https://review.opendev.org/705336 | 21:36 |
| *** elenalindq has quit IRC | 21:42 | |
| *** ansmith_ has quit IRC | 21:47 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible stable/train: Revert non voting jobs from "Install appropriate tempest releases on stable branches" https://review.opendev.org/706983 | 21:47 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible stable/train: Revert non voting jobs from "Install appropriate tempest releases on stable branches" https://review.opendev.org/706983 | 21:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: pre-install: use path and drop dest https://review.opendev.org/705892 | 22:13 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest stable/train: Revert "Mark ubuntu distro and debian stretch & centos as non voting" https://review.opendev.org/706944 | 22:17 |
| *** dmsimard is now known as dmsimard|off | 22:17 | |
| *** waxfire has joined #openstack-ansible | 22:23 | |
| *** ansmith_ has joined #openstack-ansible | 22:24 | |
| *** waxfire has quit IRC | 22:25 | |
| *** nsmeds has joined #openstack-ansible | 22:29 | |
| *** waxfire has joined #openstack-ansible | 22:29 | |
| *** waxfire has quit IRC | 22:30 | |
| *** CeeMac has joined #openstack-ansible | 22:34 | |
| *** waxfire has joined #openstack-ansible | 22:36 | |
| *** openstackgerrit has quit IRC | 22:46 | |
| *** spatel has quit IRC | 22:50 | |
| *** waxfire has quit IRC | 22:52 | |
| *** waxfire has joined #openstack-ansible | 22:52 | |
| *** waxfire has joined #openstack-ansible | 22:53 | |
| *** waxfire has quit IRC | 22:55 | |
| *** waxfire has joined #openstack-ansible | 22:56 | |
| *** waxfire has quit IRC | 23:08 | |
| *** waxfire has joined #openstack-ansible | 23:08 | |
| *** waxfire has quit IRC | 23:19 | |
| *** nicolasbock has quit IRC | 23:26 | |
| *** nicolasbock has joined #openstack-ansible | 23:35 | |
| *** rh-jelabarre has quit IRC | 23:37 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!