| hamidlotfi | Hi friends, I set up a multi-node environment with stable/ZED on OSA, after deploying an instance, the instance can not fetch metadata and all retries failed. | 05:56 |
|---|---|---|
| jrosser | hamidlotfi: can I check, is this the first time you’ve reported that here? | 06:45 |
| hamidlotfi | actually this is a first time I've deployed OSA. (note: I didn't set any option in the config file,did I?) | 06:50 |
| hamidlotfi | Yes, you can. what did you need to check.I can send any log file. | 06:52 |
| hamidlotfi | jrosser: ^ | 06:52 |
| hamidlotfi | If I any change in the YAML file such as `os-neutron-install.yml`, it is enough to run playbook.yml or anything else? | 06:59 |
| hamidlotfi | I mean adding new options in the `user_variables.yml`and then deploying `os-neutron-install.yml`, is it correct? | 07:03 |
| hamidlotfi | wont it be messed up if I deployed again? | 07:04 |
| jrosser | hamidlotfi: you can make changes just to neutron yes by running only that playbook | 07:50 |
| jrosser | there are also ansible tags that will skip everything except deploying config file changes too in order to speed things up | 07:50 |
| jrosser | you should be able to make any changes you like to user_variables and run the playbooks as you need, that won’t mess up the deployment | 07:52 |
| opendevreview | Andrew Bonney proposed openstack/ansible-role-systemd_networkd master: Handle omitted variables which appear as empty strings https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/869736 | 08:09 |
| hamidlotfi | jrosser: Thanks; I don't know what was the origin of the metadata problem and I don't know what does`neutron_vxlan_group: "239.0.0.1"`do, but now instances gets metadata stuff after 4 failed tries (https://pastebin.com/uuyYZKJH) after adding that option to the user_variables and deploying the neutron playbook. Do you have any idea? If it's the broadcasting neutron OVN IP, why it's not hard-coded by default?! | 08:13 |
| jrosser | hamidlotfi: is this OVN? kinuxbridge? | 08:15 |
| hamidlotfi | Yes OVN. | 08:15 |
| jrosser | hmm well in OVN each compute should be providing metadata locally | 08:18 |
| hamidlotfi | In making the next servers, it doesn't even give a few limited errors anymore and builds quickly | 08:19 |
| jrosser | is that on the same compute host? | 08:20 |
| hamidlotfi | I have two compute and all instance on them is OK, now. | 08:23 |
| hamidlotfi | (Note: I'm using vxlan for the self-service network in the openstack_user_config) | 08:26 |
| jrosser | so your question about neutron_vxlan_group | 08:27 |
| jrosser | broadcast traffic in vxlan is usually converted to multicast so it is sent to all tunnel endpoints, just like a L2 broadcast packet would be on a normal network | 08:28 |
| jrosser | and that variable is specifying the multicast group that should be used | 08:28 |
| noonedeadpunk | mornings | 09:11 |
| noonedeadpunk | more OVN stuff coming in... Quite time to review docs then :D | 09:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Restore dynamic_inventory unit testing https://review.opendev.org/c/openstack/openstack-ansible/+/869776 | 09:24 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-haproxy_server master: Fix dict object key error when haproxy interfaces not defined https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/869799 | 09:27 |
| hamidlotfi | >that variable is specifying the multicast group that should be used | 09:35 |
| noonedeadpunk | andrewbonney: that patch is weird... There always should be an interface in the object and be at least an empty string (I guess) | 09:36 |
| noonedeadpunk | Oh... Maybe not for extra_lb_tls_vip_addresses | 09:36 |
| andrewbonney | Yeah, I think that was the issue | 09:36 |
| noonedeadpunk | But maybe jsut adjust this then https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/vars/main.yml#L22 ? | 09:37 |
| noonedeadpunk | nah, forget it, you;'re right | 09:37 |
| hamidlotfi | jrosser: I deleted neutron_vxlan_group: "239.0.0.1" from the user_variables file and deployed again; Now there's no error related to metadata. Isn't it odd? Maybe it needs to run one time even you override deployment! | 09:38 |
| noonedeadpunk | we shouldn't require to supply key if it's not required | 09:38 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Restore dynamic_inventory unit testing https://review.opendev.org/c/openstack/openstack-ansible/+/869776 | 09:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Restore dynamic_inventory unit testing https://review.opendev.org/c/openstack/openstack-ansible/+/869776 | 10:14 |
| jrosser | looks like centos mirror wierdness again | 10:26 |
| jrosser | hamidlotfi: well i'm not sure, making a config change to neutron will have the side effect of restarting some serivces | 10:26 |
| jrosser | so it could easily be we have some ordering issue with starting / restarting neutron and ovn things during deployment | 10:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Allow empty records for container_skel https://review.opendev.org/c/openstack/openstack-ansible/+/869762 | 10:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Prevent bootstrap failure when all roles/collections are overriden https://review.opendev.org/c/openstack/openstack-ansible/+/869459 | 10:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Sync ZFS pool names https://review.opendev.org/c/openstack/openstack-ansible/+/869634 | 10:56 |
| moha7 | Hey | 11:02 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-haproxy_server master: Fix dict object key error when haproxy interfaces not defined https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/869799 | 11:04 |
| moha7 | My issue with the provider (external) network: The router gateway is not pingable from outside! Here is the provider network config used for deployment: http://ix.io/4kSN and this is what has been deployed: http://ix.io/4kSQ | 11:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Bump OpenStack-Ansible Xena https://review.opendev.org/c/openstack/openstack-ansible/+/869810 | 11:09 |
| noonedeadpunk | moha7: So the thing is that not everyone here is aware about OVN specifics, so I can hardly answer if anything is wrong and how to debug OVN. So worth pinging someone with OVN expertise, like jamesdenton or mgariepy | 11:13 |
| noonedeadpunk | or spatel | 11:13 |
| noonedeadpunk | or you can try out ovs driver in which more community members have expertise :-) | 11:17 |
| moha7 | +1 | 11:18 |
| moha7 | jamesdenton, mgariepy: ^ | 11:18 |
| moha7 | https://www.irccloud.com/pastebin/fqevsdkm/ | 11:29 |
| moha7 | noonedeadpunk: Indeed, destiny led me to OVN! The first time I installed Zed, I realized that I'm facing something called OVN and I got suggestion to continue with it since this is the future path. | 11:29 |
| moha7 | The problem with OVS is that it is not documented anywhere. I will have to write the OVS configuration all by myself, which I can't handle! | 11:30 |
| jrosser | moha7: there is an OVS scenario in the AIO - we test this | 11:30 |
| noonedeadpunk | And I bet there're docs for ovs as well | 11:35 |
| noonedeadpunk | moha7: have you checked that? https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-openvswitch.html | 11:35 |
| moha7 | Oh, Thanks; I'll give it a try | 12:13 |
| noonedeadpunk | if you're spinning up aio, then you can use SCENARIO=aio_ovs or smth like that | 12:26 |
| moha7 | Is it necessary to have openvswitch-switch installed on the hosts? At moment, it's not installed on controllers. | 12:49 |
| mgariepy | moha7, your ip is not pignable do you know where it's hosted ? | 13:06 |
| mgariepy | if you have a bunch of network nodes and computes and do use the magical ovn dvr thing. step 1 is to look where it's scheduled. | 13:06 |
| mgariepy | moha7, usually with this you will see the gateway chassis it's hosted on `ovn-nbctl show` | 13:08 |
| moha7 | I was not using `neutron_plugin_base: - ovn-router` in the user_variables; Now I run another deploy to see what happens for the provider network. | 13:08 |
| mgariepy | anyhow, when you know where the ip is scheduled you can check if you see the packet on the physical interface and so on | 13:17 |
| mgariepy | https://blog.russellbryant.net/2016/11/11/ovn-logical-flows-and-ovn-trace/ | 13:17 |
| moha7 | mgariepy: This document: https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/targethosts.html does not explains how to create bridges on the hosts; For the OVN/OVS scenarios, should them be created by Open vSwitch, or this Ubuntu netplan configuration is ok: http://ix.io/4kTn ? | 13:20 |
| mgariepy | ovs or ovn ? also if ovs what firewall driver are you using ? | 13:21 |
| moha7 | mgariepy: I don't understand this phrases: "where the IP is hosted / where the ip is scheduled"! | 13:21 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Remove "warn" paramter from command module https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/869658 | 13:21 |
| moha7 | I have issue with the external IPs. The hand of routers that is in the provider network is not pingable. | 13:23 |
| mgariepy | what's your deployment ? aio ? or on a couple of machines? | 13:23 |
| moha7 | I mean, when I set a floating IP on an instance, I can not see it from outside of the OpenStack. | 13:24 |
| moha7 | Deployment: 3 Controller nodes and 2 Compute nodes | 13:24 |
| moha7 | Os: Ubuntu 22.04 | 13:24 |
| mgariepy | for ovn some network can be created via ansible playbook here: https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/setup_ovs_ovn.yml#L55-L83 | 13:25 |
| mgariepy | same for ovs here: https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/ovs_config.yml#L16-L42 | 13:25 |
| mgariepy | ok on ovn-northd container/host can you run `ovn-nbctl show` ? | 13:25 |
| mgariepy | from this page : https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html#useful-open-virtual-network-ovn-commands | 13:27 |
| mgariepy | if you paste the output of `ovn-nbctl show` i can help you understand it. | 13:29 |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: Increase thread/process to 2 for keystone https://review.opendev.org/c/openstack/openstack-ansible/+/869642 | 13:30 |
| moha7 | mgariepy: I run a deploy; I'm waiting to be finished! I'll send you the output ASAP. | 13:34 |
| mgariepy | ok no worries. | 13:34 |
| opendevreview | Merged openstack/openstack-ansible stable/zed: Block unauthenticated Ironic API endpoints from untrusted networks https://review.opendev.org/c/openstack/openstack-ansible/+/869641 | 13:47 |
| jamesdenton | good morning. i have a call right now, moha7, but happy to help after | 13:51 |
| jamesdenton | FYI - OVS and OVN playbooks can create the ovs bridges for you, and connect them, there are some directives on that | 13:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Increase thread/process to 2 for keystone https://review.opendev.org/c/openstack/openstack-ansible/+/869830 | 13:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Increase thread/process to 2 for keystone https://review.opendev.org/c/openstack/openstack-ansible/+/869830 | 13:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Bump OpenStack-Ansible Xena https://review.opendev.org/c/openstack/openstack-ansible/+/869810 | 13:59 |
| moha7 | I finally found an Open vSwitch configuration for Ubuntu netplan: https://github.com/canonical/netplan/blob/main/examples/openvswitch.yaml | 14:02 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder master: Remove rsync requirement for cinder https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/869725 | 14:03 |
| moha7 | jamesdenton: "can create the ovs bridges"; On which config files and how I should tell those playbooks to creates bridges? | 14:06 |
| jamesdenton | sure | 14:06 |
| jamesdenton | which docs are you following for the install? | 14:07 |
| moha7 | https://docs.openstack.org/openstack-ansible-os_neutron/zed/app-ovn.html | 14:07 |
| moha7 | and: https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/targethosts.html | 14:08 |
| jamesdenton | perfect, let's look at the first one | 14:08 |
| jamesdenton | https://docs.openstack.org/openstack-ansible-os_neutron/zed/app-ovn.html#openstack-ansible-user-variables | 14:08 |
| jamesdenton | I assume you have the `neutron_provider_networks` key defined? | 14:09 |
| moha7 | jamesdenton: the config you introduced me, this one: `network-northd_hosts`, is not used in the first document. | 14:11 |
| jamesdenton | right - i was expecting it to not be necessary, but if it is, then i will need to update docs and/or make it automatic | 14:12 |
| jamesdenton | so, keep it there for now in your config, and i'll see what we need to do to make it not necessary to keep | 14:12 |
| moha7 | jamesdenton: http://ix.io/4kU1 <-- `neutron_provider_networks` | 14:13 |
| moha7 | Ah, I'm not sure if it's necessary! | 14:14 |
| jamesdenton | ok, couple of things | 14:14 |
| jamesdenton | network_mappings: "vlan:br-provider" --> with this, you are creating a provider label named 'vlan' and associating it with an OVS bridge named 'br-provider'. That's OK. In a lot of upstream documentation, the provider label is named 'physnet1'. The 'vlan' label is OSA specific and is a legacy identifier. You can name it whatever you want, and when you create provider networks, you will refer to the label name | 14:17 |
| jamesdenton | network_interface_mappings: "br-provider:br-vlan" --> with this, you are directing OSA playbooks to create an OVS bridge named 'br-provider' and attach an interface named 'br-vlan' to that bridge. The 'br-vlan' linux bridge (probably defined in outdated, yet current, docs), is probably not the way I would go. Instead, define the interface you would've connected to br-vlan (ie. eth3 or bond1) | 14:18 |
| jamesdenton | Example: network_interface_mappings: "br-provider:bond1" --> That will instruct OSA playbooks to create OVS bridge named br-provider and connect bond1 | 14:18 |
| jamesdenton | network_vlan_ranges: "vlan:1:4095" --> with this, you are directing OSA to configure the Neutron config files to allow tenant networks on the 'vlan' provider label to allocate any VLAN ID between 1 and 4095. Probably way too large of a range, especially for automatic tenant network allocation | 14:20 |
| jamesdenton | If you look at the example neutron_provider_networks on that page, there should be some comments outlining why the settings are the way they are | 14:21 |
| moha7 | `network_mappings: "vlan:br-provider" --> Can I use star `*` instaead of the vlan in this syntax? | 14:21 |
| moha7 | `network_mappings: "vlan:br-provider"` --> Can I use star `*` instaead of the vlan in this syntax? | 14:21 |
| moha7 | Thank you jamesdenton for the complete explanation you provided; I'll consider them | 14:24 |
| jamesdenton | for network mappings? no, you cannot use a star. The mapping of 'label' to 'bridge' is required | 14:24 |
| jamesdenton | that's a Neutron requirement, not OSA | 14:25 |
| moha7 | to have differenet external network, then this would be the syntax: `"ext1:br-provider, ext2:br-provider"`, right? | 14:26 |
| jamesdenton | almost; you need a separate bridge for each provider 'label' | 14:26 |
| moha7 | Aha | 14:26 |
| jamesdenton | ext1:br-provider, ext2:br-provider2 | 14:26 |
| jamesdenton | network_interface_mappings: "br-provider:eth1,br-provider2:eth2" | 14:27 |
| jamesdenton | for example, would be the corresponding interface mappings | 14:27 |
| jamesdenton | network_mappings: "ext1:br-provider,ext2:br-provider2", for example, if your label to bridge mapping | 14:27 |
| jamesdenton | the playbooks will automatically create the OVS bridges and connect the interfaces. That config is stored in the OVS DB and you do *not* need to configure netplan or ifupdown for that | 14:28 |
| moha7 | In the future, when we need an extra external network, should be deployed by OSA, or we should create it (by ovs-vsctl) and add it to the ml2/openvswitch config file? | 14:28 |
| jamesdenton | ideally, you would create it within OSA configs and rerun the appropriate playbooks | 14:28 |
| jamesdenton | in a pinch, we have been known to do both - update config files and then do it by hand, to avoid running playbooks in certain environments. But, that's a risk. | 14:29 |
| jrosser | for extra external networks it might help to think about them all as vlans on a single interface if you can | 14:31 |
| jrosser | then there is nothing to do when adding a new one except to issue the relevant openstack commands to create the network in neutron | 14:32 |
| jamesdenton | ^^^ exactly. | 14:32 |
| jamesdenton | a single provider bridge with an interface configured as a trunk allows Neutron to handle the VLAN tagging | 14:32 |
| jrosser | if you have to add a new phyiscal interface each time you want a new external interface then you are in for a bad time | 14:32 |
| jrosser | like just don't ever use flat type networks | 14:33 |
| jrosser | even if you only have one, make it vlan type and specify the vlan id | 14:33 |
| spatel | aio default use OVN correct but what about manual deployment i believe it still use lxb correct? | 14:40 |
| spatel | I meant manual deployment of zed OSA on 3 node cluster | 14:40 |
| spatel | I meant manual deployment of zed OSA on 3 node cluster | 14:41 |
| mgariepy | spatel, https://github.com/openstack/openstack-ansible/commit/79fc2a604d6bdd9f5fc17bd170829f85c93a5239 | 14:43 |
| spatel | Just asking because i noticed it didn't install deploy ovn northd so assuming it still LXB (Doesn't it confuse people when they go to production?) | 14:43 |
| spatel | mgariepy i am confused now :D | 14:45 |
| mgariepy | maybe we should update the examople configs in etc/openstack_deploy/** | 14:45 |
| spatel | I deployed AIO and it used OVN then i deploy 3 node cluster using zed and its using LXB :( | 14:46 |
| mgariepy | you did copy example openstack_user_config from the repo ? | 14:46 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible master: Add Glance tempest plugin repo to testing SHA pins list https://review.opendev.org/c/openstack/openstack-ansible/+/869850 | 14:47 |
| spatel | No... I didn't copy anything all default (i assumed it will use OVN default driver) | 14:47 |
| spatel | This is what i have - https://paste.opendev.org/show/b1B7uM9XS4KpuzzrtrL5/ | 14:49 |
| spatel | In short i have to manually tel ml2.ovn in user_variables.yml to deply ovn in Zed (3 node cluster scenario) | 14:50 |
| spatel | or i missed something in notes.. | 14:50 |
| spatel | I have to leave now.. I will check again but agreed we need to make doc little more clear about Zed and OVN relationship :D | 14:51 |
| jamesdenton | @spatel i think there's a bug with northd group not being defined automatically. i need to look at it | 15:35 |
| noonedeadpunk | Um, northd won't appear on it's own - it needs to be defined in openstack_user_config for sure | 15:52 |
| noonedeadpunk | there's no magic there | 15:52 |
| jamesdenton | ok then, i wasn't sure if it was something we could (or should) associate with the network hosts automatically | 15:53 |
| jamesdenton | i'll update the docs | 15:54 |
| jrosser | is northd a network hosts thing or an infra hosts thing? | 15:56 |
| jrosser | this sounds a bit like bfore when we had neutron-api running on network nodes which didnt make total sense | 15:56 |
| jamesdenton | well, if i had to pick, i'd prob say infra | 15:56 |
| jrosser | sounds right | 15:58 |
| mgariepy | infra also here :p | 15:59 |
| mgariepy | controller is the only network one haha | 15:59 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_neutron master: Update OVN northd group documentation https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/869856 | 16:12 |
| opendevreview | Merged openstack/openstack-ansible-ceph_client master: Define libvirt secrets from keyring files in ceph_extra_confs https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/867970 | 16:15 |
| opendevreview | Merged openstack/openstack-ansible master: Fix comment typo in nova install playbook https://review.opendev.org/c/openstack/openstack-ansible/+/869686 | 16:24 |
| NeilHanlon | we haven't seen any CI failures for rocky, have we ? someone mentioned it wasn't working from the instructions... something to do this afternoon | 16:25 |
| opendevreview | Merged openstack/openstack-ansible master: Bump pip and wheel to latest versions https://review.opendev.org/c/openstack/openstack-ansible/+/869715 | 16:26 |
| noonedeadpunk | no, I don't think we have. I've seen epel being out sync for short time, but that's it | 16:27 |
| noonedeadpunk | I'd say it's likely config thing then OS-related | 16:28 |
| NeilHanlon | epel does be like that sometimes, doesn't it.. | 16:33 |
| noonedeadpunk | oh yes, no kidding it does | 16:41 |
| spatel | I am back folks.. | 17:01 |
| spatel | What is the default ML2 driver for zed? | 17:02 |
| spatel | jamesdenton ^^ | 17:04 |
| spatel | https://docs.openstack.org/releasenotes/openstack-ansible/zed.html | 17:05 |
| prometheanfire | spatel: ovn https://github.com/openstack/openstack-ansible-os_neutron/blob/stable/zed/defaults/main.yml#L361 | 17:05 |
| spatel | Its saying OVN but when i deployed on 3 node cluster why it didn't created ovn-northd container? | 17:05 |
| noonedeadpunk | because `network-northd_hosts` must be mentioned in openstack_user_config | 17:06 |
| spatel | https://paste.opendev.org/show/b1hx5pxtT76kPyQ2liKi/ | 17:06 |
| noonedeadpunk | see https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/869856/1/doc/source/app-ovn.rst#81 | 17:07 |
| spatel | oh that was missing in my case.. | 17:07 |
| noonedeadpunk | spatel: and what has you defined in your openstack_user_config? | 17:07 |
| spatel | i don't have network-northd_hosts in config. | 17:08 |
| spatel | Let me add and run playbook | 17:08 |
| prometheanfire | setup-hosts/infra first, forgot that myself | 17:09 |
| spatel | where are the example files? | 17:10 |
| spatel | i didn't see that in any example files.. | 17:10 |
| noonedeadpunk | Well, we're trying to catch up with docs | 17:10 |
| noonedeadpunk | I've sent you a patch that does add it | 17:11 |
| spatel | I think we should rollout example file because that was my issue... | 17:11 |
| noonedeadpunk | yeah, sure... | 17:12 |
| noonedeadpunk | we should update tons of things to be frank | 17:12 |
| spatel | +1 | 17:12 |
| prometheanfire | yep, always more work :D patches welcome | 17:13 |
| spatel | we should create network-northd_hosts by default if didn't mention.. just put it next to neutron-server container | 17:13 |
| spatel | that is essential component of OVN | 17:14 |
| spatel | if user want to move it anywhere else then he can... | 17:14 |
| spatel | let me add and run... and verify functionality.. | 17:14 |
| jrosser | i wonder if this is helpful really https://opendev.org/openstack/openstack-ansible/src/branch/master/etc/openstack_deploy/conf.d/neutron.yml.aio | 17:19 |
| prometheanfire | ya, that's what tipped me off to that section being missing | 17:19 |
| jrosser | and i guess what i mean by helpful / unhelpful is why is that not in here https://opendev.org/openstack/openstack-ansible/src/branch/master/etc/openstack_deploy/openstack_user_config.yml.aio.j2 | 17:20 |
| jrosser | i see the point of the conf.d/*.aio files for peripheral services but maybe not so much for core things | 17:21 |
| spatel | jrosser yes.. first thing i do delete all aio file when i deploy in production and that is what i did :( | 17:22 |
| jrosser | for example identity_hosts is defined in the AIO template but also in keystone.yml.aio | 17:22 |
| spatel | so i lost that example.. | 17:22 |
| jrosser | jamesdenton: what do you think about that ^ | 17:22 |
| jrosser | is the neutron.yml.aio being separate counterproductive? | 17:23 |
| spatel | I would say we should make some stuff default in this example network-northd_hosts: (if not define then install in infra* nodes) it will reduce typing and error | 17:23 |
| jrosser | do we want to backport these ceph_client fixes? | 17:33 |
| spatel | noonedeadpunk oh boy.. i have add this but still it didn't create ovn_northd container - https://paste.opendev.org/show/b05Zx6h5RPAkRfvLk0Jg/ | 17:45 |
| spatel | I will debug later because its consuming my lots of time :( | 17:45 |
| jamesdenton | thats the wronggroup name | 17:46 |
| noonedeadpunk | jrosser: well, I'd say that aio.j2 is usually not refferenced as example | 17:46 |
| noonedeadpunk | But openstack_user_config.yml.aio can and should contain network-northd_hosts for instance | 17:47 |
| jamesdenton | jrosser lemme get back to you shortly | 17:47 |
| jamesdenton | you may have put it in env.d | 17:47 |
| spatel | wthhh | 17:48 |
| noonedeadpunk | is it wrong? | 17:48 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/env.d/neutron.yml#L130 | 17:48 |
| spatel | damn it... neutron_ovn_northd vs network-northd_hosts | 17:49 |
| jrosser | imho aio.j2 is the most tested thing we have as it's used for all role tests | 17:49 |
| noonedeadpunk | But it's not inlcuded into docs anywhere | 17:49 |
| jrosser | hmm | 17:49 |
| spatel | its easy to miss _ and - :( | 17:49 |
| noonedeadpunk | While many of others openstack_user_config.yml.* are | 17:50 |
| noonedeadpunk | Like for AIO we suggest copying stuff from conf.d at all https://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html#bootstrap-the-aio-configuration | 17:51 |
| jrosser | hah i never read that before | 17:52 |
| jrosser | read/remembered | 17:52 |
| jrosser | i always set SCENARIO | 17:53 |
| noonedeadpunk | For production example we do literal include | 17:53 |
| noonedeadpunk | https://github.com/openstack/openstack-ansible/blob/master/doc/source/user/prod/example.rst#deployment-configuration | 17:53 |
| noonedeadpunk | So I'd say we should adjust these first, and j2 that we use in CI doesn't matter much as for documentation purposes | 17:54 |
| noonedeadpunk | also we actually need northd conditionally, which is the main issue... | 17:54 |
| noonedeadpunk | Otherwise we could add it to `network-infra_containers` | 17:55 |
| jamesdenton | https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/conf.d/neutron.yml.aio | 17:55 |
| noonedeadpunk | but then northd will be spawned for lxb/ovs as well which is wrong... | 17:55 |
| jamesdenton | that's what i was referring to, conf.d, sorry | 17:55 |
| jamesdenton | spatel it's not just dash vs underscore, but prefix is network- not neutron- | 17:56 |
| spatel | yes.. just fixed that | 17:56 |
| jamesdenton | cool, that should do it. | 17:56 |
| spatel | jamesdenton we should default set that on network_hosts: instead of specify. | 17:57 |
| spatel | If user want to run somewhere else then he/she can use override | 17:58 |
| spatel | it will make life easier | 17:58 |
| jamesdenton | jrosser i don't see it being counter productive, but could be confusing if overlooked. | 17:58 |
| jamesdenton | or, just configure the group w/ alias and move on | 17:58 |
| noonedeadpunk | spatel: we can't do that | 18:01 |
| spatel | uhu | 18:02 |
| noonedeadpunk | as I said earlier - this will result in creation of northd for lxb/ovs as well | 18:02 |
| noonedeadpunk | while it won't be utilized - contaier will be created | 18:02 |
| jamesdenton | those are things i sometimes overlook, since aren't using lxc | 18:04 |
| spatel | why if ml2.ovn is set then merge group with network_hosts: | 18:07 |
| spatel | or may be there factor complicate it.. | 18:07 |
| BobZAnnapolis | sorry folks, sorry to bug ya (again)...follow-up to yesterday's 'we rebooted our controllers and we're having issues' issue - we got the controllers back in sync, all the lxc-* containers on the controllers back in sync as well (we think) except now we can't create instances or volumes, we're having cinder issues :-( on the controllers, the cinder-api containers are logging. . .but. . .nothing is being logged i | 18:10 |
| spatel | It must be mysql or rabbitmq out of cluster, i would check that first BobZAnnapolis | 18:12 |
| BobZAnnapolis | lxc-ls -f says all containers are up & running on all 3 controllers, rabbitmq clusterctl cmds indicate it is working fine - we're going to recheck keepalived & ha-proxy (again) but at last check, they were indicating "I'm ok" | 18:12 |
| BobZAnnapolis | spatel : thanks, will try that too | 18:12 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Update OVN northd group documentation https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/869856 | 18:48 |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: Sync ZFS pool names https://review.opendev.org/c/openstack/openstack-ansible/+/869634 | 19:18 |
| opendevreview | Merged openstack/ansible-role-systemd_networkd master: Fix static routes to use Destination rather than Source key https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/869733 | 19:52 |
| opendevreview | Merged openstack/ansible-role-systemd_networkd master: Handle omitted variables which appear as empty strings https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/869736 | 19:55 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Fix dict object key error when haproxy interfaces not defined https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/869799 | 20:27 |
| BobZAnnapolis | another silly question, after sourcing openrc, i'm executing several CLI "openstack. . .list" commands - just "list" commands to retrieve running information, i was just now informed that "The account is locked for . . . ." (admin) ? What causes this condition, and is there some default upper limit # of cmds an admin user is allowed to execute ? tia | 20:40 |
| mgariepy | not in default install no. | 20:41 |
| mgariepy | not sure it's possible to rate-limit commands like that | 20:41 |
| BobZAnnapolis | yeah it's a weird one, we have over a handful of systems up & running, i've been retrieving info from all of them - the other 4-5 are having no issues - just this 1 says that 'admin' a/c is now locked :-( | 20:45 |
| mgariepy | it's possible to have some config in keystone for lockout_failures but it's not default for sure. | 20:46 |
| BobZAnnapolis | thx, looks like there is a "20 minute timeout when locked" condition in place somewhere as well - i was just able to get back in - i'll check keystone - | 20:51 |
| BobZAnnapolis | fyi, keystone.conf "lockout_duration = 900" | 20:57 |
| mgariepy | anyone else uses that in keystone ? | 20:58 |
| opendevreview | Merged openstack/openstack-ansible master: Add Glance tempest plugin repo to testing SHA pins list https://review.opendev.org/c/openstack/openstack-ansible/+/869850 | 21:12 |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: Prevent bootstrap failure when all roles/collections are overriden https://review.opendev.org/c/openstack/openstack-ansible/+/869459 | 21:12 |
| opendevreview | Merged openstack/openstack-ansible stable/xena: Increase thread/process to 2 for keystone https://review.opendev.org/c/openstack/openstack-ansible/+/869830 | 21:12 |
| spatel | I never heard and had issue with lockout in keystone | 21:15 |
| spatel | BobZAnnapolis What was the issue with controller, did you manage to find it? | 21:16 |
| BobZAnnapolis | spatel : Sort of, not entirely tho. We have multiple separate systems running, the controller reboots worked flawlessly on 1 but when we repeated the procedure on another system, dumpster fire :-) we used existing troubleshooting docs,online docs, local knowledge to get the controllers back up - controllers looked ok but then we started having neutron & cinder issues, had to go down that rabbit hole (wink wink) | 21:36 |
| mgariepy | the dev systems always work flawlessly :P | 21:40 |
| BobZAnnapolis | mgariepy : :-) close, 1 dev, 1 production worked flawlessly, 1 production went belly up | 21:44 |
| mgariepy | haha :) | 21:44 |
| mgariepy | just when you are super confident it will work. boom | 21:44 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron stable/zed: Update OVN northd group documentation https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/869834 | 22:15 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-systemd_networkd stable/zed: Fix static routes to use Destination rather than Source key https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/869835 | 22:15 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-systemd_networkd stable/yoga: Fix static routes to use Destination rather than Source key https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/869836 | 22:15 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-systemd_networkd stable/xena: Fix static routes to use Destination rather than Source key https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/869837 | 22:15 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-systemd_networkd stable/wallaby: Fix static routes to use Destination rather than Source key https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/869838 | 22:15 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-systemd_networkd stable/zed: Handle omitted variables which appear as empty strings https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/869839 | 22:16 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-systemd_networkd stable/yoga: Handle omitted variables which appear as empty strings https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/869840 | 22:16 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-systemd_networkd stable/xena: Handle omitted variables which appear as empty strings https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/869841 | 22:16 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-systemd_networkd stable/wallaby: Handle omitted variables which appear as empty strings https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/869842 | 22:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-haproxy_server stable/zed: Fix dict object key error when haproxy interfaces not defined https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/869843 | 22:17 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!