| soltanedare | Hi | 07:34 |
|---|---|---|
| noonedeadpunk | o/ | 07:48 |
| damiandabrowski | hey folks | 09:22 |
| damiandabrowski | FYI: my separated haproxy config changes passed gating over the weekend | 09:22 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible/+/871189/ | 09:22 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/871188 | 09:22 |
| damiandabrowski | i fixed a lot of your suggestions there and added a summary of recent changes in a comment | 09:23 |
| noonedeadpunk | will try to have a look during the day or tomorrow morning at worst | 09:24 |
| admin1 | \o | 10:48 |
| *** dviroel_ is now known as dviroel | 12:15 | |
| damiandabrowski | so our 'linear' strategy plugin really breaks loops, i was able to reproduce it on a fresh AIO environment | 12:30 |
| damiandabrowski | https://bugs.launchpad.net/openstack-ansible/+bug/2007849 | 12:30 |
| noonedeadpunk | well, that's quite sad | 12:33 |
| jrosser | https://github.com/openstack/openstack-ansible-plugins/blob/master/plugins/strategy/linear.py#L97 | 12:33 |
| noonedeadpunk | It's actually good question if it's still needed. It used to fix flaws in default linear plugin when talking about containets | 12:34 |
| noonedeadpunk | as it was somehow helping out with running against hosts and containers at the same time to prevent race conditions or smth like that... | 12:34 |
| noonedeadpunk | But that was soooo long ago | 12:34 |
| noonedeadpunk | cloudnull: maybe you recall details about it? | 12:35 |
| jrosser | it probably needs to detect with* and loop | 12:37 |
| jrosser | or just be deleted entirely | 12:37 |
| noonedeadpunk | that;s the commit that added strategy https://github.com/openstack/openstack-ansible-plugins/commit/cb01efef6657fce5003f099e5209b7086a0cd469 | 12:41 |
| noonedeadpunk | So yeah, it was added with SSH plugin and still imports it from what I see. Or well - attempts to import it, but I'm not sure if import works... | 12:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-plugins master: [DNM] Check if everything works fine without linear plugin https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/874425 | 14:21 |
| noonedeadpunk | damiandabrowski: ssh connection plugin is needed for sure... | 14:45 |
| noonedeadpunk | as we don't start ssh inside containers with some exceptions, like keystone | 14:45 |
| noonedeadpunk | well.... | 14:45 |
| noonedeadpunk | maybe with ssl certs we can start ssh inside lxc, but then jrosser for example will need to pass extra network to containers | 14:45 |
| noonedeadpunk | Which is smth I bet he don't want to do | 14:46 |
| noonedeadpunk | As containers are isolation layer and software inside lxc don't really have access to SSH | 14:46 |
| noonedeadpunk | *ssh network | 14:46 |
| noonedeadpunk | or do they? | 14:46 |
| admin1 | so you mean we cannot do ssh util from deploy .. but always have to login to c* and then lxc-attach ? | 14:58 |
| admin1 | any thoughts on moving to lxd as well ? | 14:58 |
| noonedeadpunk | admin1: as soon as it will be possible to avoid using snap with lxd.... | 15:09 |
| noonedeadpunk | and yes, that's exactly how it works now | 15:09 |
| admin1 | got it .. | 15:23 |
| Mohaa7 | Hi | 15:28 |
| Mohaa7 | If I'm not going to use the modules which are listed after the Horizon in the setup-openstack.yml, should I comment them in the yml file? | 15:29 |
| noonedeadpunk | Mohaa7: um, I would not change playbooks to be frank. If you want to - you can use your own playbooks of course. But the thing is - that out of all playbooks only roles that have some hosts in inventory will be ran | 16:00 |
| noonedeadpunk | In other words - if you have not defined any host for trove in openstack_user_config - trove role will be ignored even playbook is included | 16:01 |
| Mohaa7 | +1, If they are fully skipped, while not having hosts in the openstack_user_config, yeah, it's ok | 16:19 |
| Mohaa7 | https://usercontent.irccloud-cdn.com/file/Ob7fmy4D/error-but-working.png | 16:20 |
| Mohaa7 | My recent deployments, all are results to a working environment, but on all of them I get a failure on compute01, as you see in the above image. It's weird! | 16:21 |
| Mohaa7 | resulted* | 16:21 |
| noonedeadpunk | Mohaa7: um, well, you'll need to find task that's failing so that we could help you | 16:23 |
| Mohaa7 | both on VMS and bare metals with different networks. | 16:23 |
| Mohaa7 | It's a huge log file! It's not saying on which part there's an issue! | 16:24 |
| Mohaa7 | Ah, I found the position of error, here: https://paste.opendev.org/show/bMLvOPRo628mi2RZ6J6c/ | 16:27 |
| noonedeadpunk | hm, can you check if there's some error also a bit above? | 16:30 |
| noonedeadpunk | As it basically says that neutron-ovn-metadata-agent service does not exist on compute01 for some reason | 16:31 |
| Mohaa7 | it's happened twice: https://usercontent.irccloud-cdn.com/file/3C5bI5fi/image.png | 16:31 |
| Mohaa7 | yes, I'll check now | 16:31 |
| noonedeadpunk | well, it's both in handlers, and smth could happen also before handlers | 16:31 |
| noonedeadpunk | handlers will be triggered regardless | 16:32 |
| Mohaa7 | There's another error before them: https://usercontent.irccloud-cdn.com/file/bLi8kSEG/image.png | 16:33 |
| noonedeadpunk | that is interesting | 16:34 |
| noonedeadpunk | I can recall seing that some time ago, but can't recall about what caused that | 16:35 |
| noonedeadpunk | Mohaa7: you have cutted screenshot in a bit unfortunate way :) Can you kindly also include output of the previous task? | 16:37 |
| Mohaa7 | I'm checking this error on two different deployed environment, and same error on Compute01 in the logs of bothe env! | 16:37 |
| noonedeadpunk | ` Fetch override files`? | 16:37 |
| Mohaa7 | I didn't get it! | 16:38 |
| Mohaa7 | what do you mean by "Fetch override files"? | 16:39 |
| noonedeadpunk | ok, and what the content do you have in /openstack/venvs/neutron-26.0.1/etc/neutron? | 16:39 |
| noonedeadpunk | I kind of wonder if for some reason you don't have neutron installed on compute01 for some reason | 16:41 |
| Mohaa7 | noonedeadpunk /openstack/venvs/neutron-26.0.1/etc/neutron: dnsmasq-neutron.conf with these two options: `dhcp-ignore=tag:!known` and `user=neutron` | 16:42 |
| noonedeadpunk | hm... it should contain more.... | 16:43 |
| noonedeadpunk | And is content on another compute is same? | 16:43 |
| Mohaa7 | noonedeadpunk: lins above/under that screenshot lines: http://sprunge.us/tFeADB --> ctrl+f f or `failed` | 16:52 |
| noonedeadpunk | what a weird set of hosts btw, why compute01 is executed along with infra but not compute02... | 16:54 |
| noonedeadpunk | it's really weird output | 16:54 |
| Mohaa7 | Oh, it was from infra01; Here is the list of files in `/openstack/venvs/neutron-26.0.1/etc/neutron`: | 16:54 |
| Mohaa7 | https://usercontent.irccloud-cdn.com/file/tJAnvyX6/image.png | 16:54 |
| Mohaa7 | Compute02: https://usercontent.irccloud-cdn.com/file/4Omkh3Bw/image.png | 16:55 |
| Mohaa7 | some items are missed on compute01 in that path, but why? | 16:55 |
| noonedeadpunk | that is really good question | 16:55 |
| Mohaa7 | indeed, on file is missed: neutron_ovn_metadata_agent.ini | 16:57 |
| Mohaa7 | one* | 16:57 |
| noonedeadpunk | I'd say it's due to https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/tasks/neutron_post_install.yml#L108-L115 not being executed for compute01 for some reason | 16:58 |
| noonedeadpunk | That kind of smells like a bug... OR smth is off with groups definition in your envs | 16:59 |
| Mohaa7 | I have metadata hosts set on infra nodes: https://usercontent.irccloud-cdn.com/file/a7kd0YDi/image.png | 17:01 |
| noonedeadpunk | So neutron-ovn-metadata-agent should be launched only on neutron_ovn_controller group | 17:02 |
| noonedeadpunk | and not on computes | 17:02 |
| noonedeadpunk | Which is network-gateway_hosts | 17:03 |
| noonedeadpunk | Well, depending on what you've defined as network-gateway_hosts ofc | 17:03 |
| noonedeadpunk | neutron_metadata_agent shouldn't be defined at all for OVN scenario iirc | 17:04 |
| noonedeadpunk | can you share your definitions from openstack_user_config? | 17:04 |
| Mohaa7 | has been set in this way: `network-gateway_hosts: *compute_hosts` | 17:04 |
| Mohaa7 | yes, w8 please | 17:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Update documentation for LXC/metal and LXB/OVS/OVN https://review.opendev.org/c/openstack/openstack-ansible/+/867577 | 17:06 |
| Mohaa7 | noonedeadpunk: the config file is here: https://paste.ubuntu.com/p/Bbj7sMkXjg/ | 17:09 |
| noonedeadpunk | Mohaa7: I don't think you should define network_hosts though I'm also not sure it should cause issues like that | 17:15 |
| noonedeadpunk | as network_hosts is valid for ovs/lxb scenarios but not ovn | 17:15 |
| Mohaa7 | Mohaa7: I added t because you are using it in the AIO | 17:16 |
| Mohaa7 | it* | 17:16 |
| noonedeadpunk | so you should have neutron_ovn_metadata_agent on computes but should not have neutron_metadata_agent on infra | 17:16 |
| noonedeadpunk | yeah, but network-infra_hosts should be more the enough | 17:18 |
| noonedeadpunk | still I don't think it's the root cause... | 17:18 |
| noonedeadpunk | I'm quite suspicios about https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/tasks/neutron_post_install.yml#L108-L115 as it should have ran but it's not in fact | 17:19 |
| noonedeadpunk | And likely run_once is the reason why.... | 17:19 |
| Mohaa7 | What if I run the os-neutron-install.yml again after removing network_hosts from the config file? | 17:20 |
| noonedeadpunk | Well, they're already in openstack_inventory.json, which then should also be carefully cleaned up. | 17:20 |
| Mohaa7 | Oops! | 17:20 |
| noonedeadpunk | Mohaa7: to be frank - I assume if you will jsut run `openstack-ansible playbooks/os-neutron-isntall.yml --limit compute01` - it will finish without errors | 17:21 |
| noonedeadpunk | If the problem is in the place I'm thinking about | 17:21 |
| Mohaa7 | Then, I keep `network_hosts` there and run yours to see if it creates the metadata file in `/openstack/venvs/neutron-26.0.1/etc/neutron` or not | 17:22 |
| noonedeadpunk | As this part from your output looks like a bug to me https://paste.openstack.org/show/bGn3zicwfTGsES4MyHDg/ | 17:22 |
| noonedeadpunk | yeah, let's try this out | 17:23 |
| Mohaa7 | I also try the whole process on a new lab, this time without `network_hosts`, to know if it's the cause | 17:24 |
| noonedeadpunk | That would be great if you have a place for multi-node lab as it would take time for me to spawn one | 17:28 |
| Mohaa7 | noonedeadpunk: I run it on comute01; result: success; And now there's a `neutron_ovn_metadata_agent.ini` in /openstack/venvs/neutron-26.0.1/etc/neutron/ on compute01 | 17:30 |
| noonedeadpunk | So, would be great if you could submit a bug so that we won't loose it | 17:31 |
| Mohaa7 | Sure, but let me try for a new env without network_hosts that takes ~4 hours | 17:32 |
| noonedeadpunk | oh, yes, totally | 17:32 |
| noonedeadpunk | that is super interesting to check | 17:35 |
| Mohaa7 | (: | 17:35 |
| Mohaa7 | Thanks for your help | 17:35 |
| Mohaa7 | I finally deployed OSA on bare metals (our acceptance env) successfully. | 17:37 |
| Mohaa7 | Thank you so much to all of you in this channel | 17:37 |
| noonedeadpunk | well, there's at least 1 bug you're affected with... | 17:43 |
| Mohaa7 | yeah | 17:53 |
| Mohaa7 | Here: https://docs.openstack.org/openstack-ansible-haproxy_server/latest/configure-haproxy.html it's said that `haproxy_keepalived_external_vip_cidr: 192.168.0.4/25` as a sample. I'm confused as someone here mentioned that it definitely should be set as `<ip>/32` regardless of the subnet. But in the link it's using /25! | 17:56 |
| noonedeadpunk | yes thta must be /32 | 17:58 |
| noonedeadpunk | Mohaa7: do you want to push a change?:) | 17:59 |
| Mohaa7 | Yes, now I have some space to work on the docs. Let me read https://docs.openstack.org/doc-contrib-guide/ at first | 18:02 |
| noonedeadpunk | Mohaa7: In short - you need to have an account in Ubuntu One which will allow you to configure gerrit account. There you will need to fill in your nickname and upload ssh key | 18:05 |
| noonedeadpunk | then ensure you have git-review plugin for git. | 18:05 |
| Mohaa7 | +1 | 18:06 |
| noonedeadpunk | Then it's like that - clone repo, make commit, do `git review`. If you need to make a change - amend existing commit rather then adding new one (that's biggest difference with github/gitlab) | 18:06 |
| noonedeadpunk | As gerrit identifies changes not based on branch, but pased on `Change-Id` that is part of the commit message | 18:07 |
| noonedeadpunk | If you want to donwload some patch locally to adjust it - do `git review -d <id>`, do changes, `git add .`, `git commit --amend`, `git review` | 18:09 |
| noonedeadpunk | you can add `-f` to last command if you want to delete branch that was created by downloading patch | 18:09 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: [DNM] Check if everything works fine without linear plugin https://review.opendev.org/c/openstack/openstack-ansible/+/874482 | 18:13 |
| Mohaa7 | I do; If something in this regard is unclear to me, I will raise it here (: | 18:15 |
| *** gmann is now known as gmann_afk | 18:37 | |
| *** gmann_afk is now known as gmann | 18:51 | |
| admin1 | the VIP can be set to anything. i have /32 in some, i have /24 in some, /22 in some | 19:06 |
| admin1 | i have not seen an issue on it so far | 19:06 |
| admin1 | but it also depends on your specific use case | 19:06 |
| jrwr_ | doing my first setup, putting all containers on one host, I'm getting a lockup without error at [lxc_container_create : Write default container config] for setup-hosts.yml, running with -vvv shows a OK for lxc.apparmor.profile=unconfined on manila_container and stops here every time. there are no SSH sessions active when this locks up | 22:03 |
| jrwr_ | for debug, I've ran openstack-ansible lxc-containers-destroy.yml to try and clear out that host | 22:07 |
| jrwr_ | removed manila and metering-compute out of user config and moved on, (I didn't want those services /anyway/) | 22:15 |
| jrosser | jrwr_: it's always useful to build an all-in-one as a reference to compare first attempts with https://docs.openstack.org/openstack-ansible/zed/user/aio/quickstart.html | 22:21 |
| jrosser | thats auto-configured with a pretty minimal set of services | 22:22 |
| Mohaa7 | admin1: then it's meaningless to define the subnet alongside the IP in `haproxy_keepalived_external_vip_cidr: 192.168.0.4/25`. It would be only enough to set an IP itself: `haproxy_keepalived_external_vip_cidr: 192.168.0.4`, wouldn't it? | 22:34 |
| *** jrwr_ is now known as jrwr | 22:35 | |
| Mohaa7 | In my case I'm choosing an IP, for example 172.20.21.22, from a range of `/24`. But I set it as `172.20.21.22/32` in the user_variables.yml file; And it works well. If it's also going to work by `172.20.21.22/24`, so what's the purpose of defining its subnet there? | 22:40 |
| admin1 | Mohaa7, the subnet there defines how you route . if you just use 10.0.0.1 for example and it added /8 by itself, then the interface where this is added will hold the route for 10.0.0.x and your other interfaces where any 10x might not work | 22:48 |
| admin1 | so netmask has its place there | 22:48 |
| admin1 | as you are adding an ip, the netmask defines the arp/broadcast domain and how linux will add that to its route | 22:49 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
