| *** cshen has quit IRC | 00:35 | |
| -openstackstatus- NOTICE: The Gerrit service on review.opendev.org is being quickly restarted to apply a new security patch | 00:55 | |
| *** spatel has joined #openstack-ansible | 01:00 | |
| *** spatel has quit IRC | 01:04 | |
| *** gyee has quit IRC | 01:09 | |
| *** maharg101 has joined #openstack-ansible | 01:47 | |
| *** maharg101 has quit IRC | 01:52 | |
| *** macz_ has quit IRC | 02:14 | |
| *** cshen has joined #openstack-ansible | 02:31 | |
| *** cshen has quit IRC | 02:36 | |
| *** cshen has joined #openstack-ansible | 02:45 | |
| *** cshen has quit IRC | 02:49 | |
| *** irclogbot_2 has quit IRC | 03:27 | |
| *** irclogbot_3 has joined #openstack-ansible | 04:15 | |
| *** irclogbot_3 has quit IRC | 04:20 | |
| *** jfan has quit IRC | 04:26 | |
| *** lemko2 has joined #openstack-ansible | 04:28 | |
| *** lemko2 has quit IRC | 04:29 | |
| *** lemko0 has joined #openstack-ansible | 04:29 | |
| *** lemko has quit IRC | 04:30 | |
| *** lemko0 is now known as lemko | 04:30 | |
| *** irclogbot_2 has joined #openstack-ansible | 04:40 | |
| *** cshen has joined #openstack-ansible | 04:45 | |
| *** cshen has quit IRC | 04:49 | |
| *** irclogbot_2 has quit IRC | 04:54 | |
| *** irclogbot_2 has joined #openstack-ansible | 04:58 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:33 | |
| *** sc has quit IRC | 05:36 | |
| *** priteau has quit IRC | 05:47 | |
| *** sc has joined #openstack-ansible | 05:48 | |
| *** yasemind has joined #openstack-ansible | 05:48 | |
| *** maharg101 has joined #openstack-ansible | 05:49 | |
| *** hamzaachi has joined #openstack-ansible | 05:53 | |
| *** maharg101 has quit IRC | 05:53 | |
| *** spatel has joined #openstack-ansible | 05:58 | |
| *** spatel has quit IRC | 05:58 | |
| *** cshen has joined #openstack-ansible | 06:00 | |
| *** hamzaachi has quit IRC | 06:03 | |
| *** cshen has quit IRC | 06:04 | |
| *** cshen has joined #openstack-ansible | 06:34 | |
| *** cshen has quit IRC | 06:40 | |
| *** yasemind has quit IRC | 06:44 | |
| *** macz_ has joined #openstack-ansible | 06:56 | |
| *** macz_ has quit IRC | 07:01 | |
| *** miloa has joined #openstack-ansible | 07:21 | |
| *** hamzaachi has joined #openstack-ansible | 07:28 | |
| *** macz_ has joined #openstack-ansible | 07:49 | |
| *** maharg101 has joined #openstack-ansible | 07:50 | |
| *** macz_ has quit IRC | 07:53 | |
| *** maharg101 has quit IRC | 07:55 | |
| *** klamath_atx has joined #openstack-ansible | 08:05 | |
| *** poopcat has quit IRC | 08:08 | |
| *** cshen has joined #openstack-ansible | 08:08 | |
| *** SiavashSardari has joined #openstack-ansible | 08:10 | |
| *** poopcat has joined #openstack-ansible | 08:11 | |
| *** klamath_atx has quit IRC | 08:24 | |
| *** rpittau|afk is now known as rpittau | 08:25 | |
| *** andrewbonney has joined #openstack-ansible | 08:27 | |
| *** maharg101 has joined #openstack-ansible | 08:31 | |
| *** gillesMo has quit IRC | 08:50 | |
| *** gillesMo has joined #openstack-ansible | 08:50 | |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-galera_server stable/victoria: Bring db setup vars in line with other roles https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/772550 | 08:51 |
|---|---|---|
| *** priteau has joined #openstack-ansible | 08:56 | |
| *** tosky has joined #openstack-ansible | 09:02 | |
| *** jbadiapa has joined #openstack-ansible | 09:17 | |
| jrosser | morning | 09:18 |
| admin0 | \o morning | 09:21 |
| SiavashSardari | morning | 09:24 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Bump ansible-base to 2.10.5 https://review.opendev.org/c/openstack/openstack-ansible-tests/+/773467 | 09:27 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Increase git clone depth from 10 to 20 https://review.opendev.org/c/openstack/openstack-ansible/+/773352 | 09:34 |
| *** d34dh0r53 has quit IRC | 09:39 | |
| *** d34dh0r53 has joined #openstack-ansible | 09:39 | |
| SiavashSardari | jrosser yesterday you mentioned log dir in containers mounts on physical hosts and I remember using that journalctl --merge on physical hosts (this was my train release setup and was at least month ago) anyways, I forgot about that and installed vector as log shipper on all containers. today I wanted to redesign my log stuff and while the mounting | 09:47 |
| SiavashSardari | of log dirs happened, but the logs are in /run/log/journal/ dir. did I miss something here? I don't remember changing journal.conf last time I used that in OSA | 09:47 |
| *** d34dh0r53 has quit IRC | 09:48 | |
| *** d34dh0r53 has joined #openstack-ansible | 09:49 | |
| jrosser | SiavashSardari: on my pyhsical hosts i see the journals all mounted to /var/log/journal | 09:51 |
| jrosser | i don't think OSA has ever touched journal.conf | 09:52 |
| SiavashSardari | jrosser in my setup I have mounted directories in /var/log/journal but in containers journal files are in /run/log/journal/ directory. | 09:59 |
| SiavashSardari | maybe somtething changed in systemd-journald?? | 10:00 |
| jrosser | perhaps it depends on the OS, mine are ubuntu | 10:00 |
| *** jbadiapa has quit IRC | 10:11 | |
| *** jbadiapa has joined #openstack-ansible | 10:11 | |
| *** avagi has quit IRC | 10:13 | |
| SiavashSardari | jrosser mine are Ubuntu too, but now focal in previous setup I mentioned it was bionic | 10:21 |
| *** avagi has joined #openstack-ansible | 10:22 | |
| jrosser | SiavashSardari: could you check if you think that this needs updating to use OS-specific paths for the journals? https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/master/tasks/lxc_container_config.yml#L261-L291 | 10:28 |
| *** tosky has quit IRC | 10:33 | |
| *** tosky has joined #openstack-ansible | 10:34 | |
| SiavashSardari | jrosser will do. but just to make sure, your setup is bionic? | 10:37 |
| jrosser | yes it's currently bionic | 10:37 |
| SiavashSardari | any one here has a setup with focal? and if you have, could you please check if your journal files are in /var/log/journal or in /run/log/journal? | 10:39 |
| SiavashSardari | I will dig a bit on systemd versions maybe I can find something there | 10:39 |
| *** ierdem has joined #openstack-ansible | 10:48 | |
| ierdem | Hello everyone, I encountered a problem about keepalived. I am using OSA-Stable/Ussuri on Ubuntu 18.04LTS. Keepalived seems works fine but I can not connect external lbvip address, oddly I can connect internal one. When I stop keepalived on infra hosts and attach external IP manually with "ip addr add 172.30.22.200/32 dev br-openstack" this | 10:51 |
| ierdem | command, I can connect and it works | 10:51 |
| ierdem | keepalived version is 1.3.9. Do you have any idea about this problem? Thanks | 10:53 |
| jrosser | SiavashSardari: turns out i have a focal AIO here, but it's without containers | 10:55 |
| *** macz_ has joined #openstack-ansible | 10:55 | |
| jrosser | the system journal is in /var/log/journal/<machine-id> and /run/log/journal directory is present but empty | 10:56 |
| *** macz_ has quit IRC | 11:00 | |
| SiavashSardari | jrosser interestingly enough I have some logs in /var/log/journal/<machine-id> but they are from Dec 21. | 11:00 |
| jrosser | container logs? | 11:01 |
| SiavashSardari | yep | 11:01 |
| jrosser | check for this if the logs don't look current https://review.opendev.org/c/openstack/openstack-ansible/+/771205 | 11:01 |
| SiavashSardari | it is their boot logs I think | 11:01 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Fix linter errors https://review.opendev.org/c/openstack/ansible-hardening/+/771481 | 11:05 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Fix linter errors https://review.opendev.org/c/openstack/ansible-hardening/+/771481 | 11:11 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Make possible to avoid aide installation https://review.opendev.org/c/openstack/ansible-hardening/+/772561 | 11:16 |
| *** SiavashSardari has quit IRC | 11:26 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 11:38 | |
| *** SiavashSardari has joined #openstack-ansible | 11:44 | |
| ierdem | I am repeating my problem, can you please help me if you know the solution; I encountered a problem about keepalived. I am using OSA-Stable/Ussuri on Ubuntu 18.04LTS. Keepalived seems works fine but I can not connect external lbvip address, oddly I can connect internal one. When I stop keepalived on infra hosts and attach external IP manually with | 11:46 |
| ierdem | "ip addr add 172.30.22.200/32 dev br-openstack" with this command, I can connect and it works. keepalived version is 1.3.9. Do you have any idea about this problem? Thanks | 11:46 |
| jrosser | ierdem: the answer probably lies in the keepalived logs | 11:47 |
| jrosser | if you are able to find anything suspicious looking there about keepalived binding to br-openstack then you can paste it at paste.openstack.org | 11:49 |
| jrosser | also check that the contents of the keepalived config file looks sensible | 11:49 |
| SiavashSardari | jrosser sorry, was at a meeting. so the bug you send happens when adding new container to an existing setup?? | 12:01 |
| jrosser | it's a bug https://bugs.launchpad.net/openstack-ansible/+bug/1895533 | 12:02 |
| openstack | Launchpad bug 1895533 in openstack-ansible "/var/log bind mount overshadows /var/log/journal bindmount in lxc container setup" [High,Fix released] - Assigned to Dmitriy Rabotyagov (noonedeadpunk) | 12:02 |
| jrosser | theres quite a good description there | 12:03 |
| SiavashSardari | yeah I already read that | 12:04 |
| ierdem | jrosser I checked keepalived logs, there is nothing suspicious in there, also I checked confs and they are ok too. My problem is keepalived attaches External IP correctly but I cannot ping or ssh. If I attach this IP manually it works fine. I could not find the root of problem | 12:04 |
| jrosser | ierdem: can you not even ping locally from the host it has attached the IP to? | 12:04 |
| ierdem | I can not | 12:05 |
| jrosser | i am wondering if the /32 is correct | 12:07 |
| ierdem | I tried first with /24 and it did not work | 12:07 |
| jrosser | when it's assigned with keepalived what do you see with ip addr | grep br-openstack | 12:08 |
| ierdem | 8: br-openstack: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 | 12:11 |
| ierdem | inet 172.30.22.40/24 brd 172.30.22.255 scope global br-openstack | 12:11 |
| ierdem | inet 170.30.22.200/32 scope global br-openstack | 12:11 |
| ierdem | 15: bond0.50@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-openstack state UP group default qlen 1000 | 12:11 |
| ierdem | sorry I should copy that to paste.openstack.org, | 12:12 |
| jrosser | and what error do you get with ping 170.30.22.200 | 12:15 |
| ierdem | http://paste.openstack.org/show/802229/ | 12:16 |
| jrosser | and what is 172.30.22.42 ? | 12:17 |
| jrosser | (i am assuming that you are testing the ping from the host with 172.30.22.40 on br-openstack) | 12:18 |
| ierdem | it is same on infra01, http://paste.openstack.org/show/802230/. We are accesing our physical hosts (infra and computes) via br-openstack interface | 12:21 |
| ierdem | 172.30.22.[40-42] -> infra0[1-3] , 172.30.22.[43-50] -> compute[1-10] | 12:21 |
| admin0 | ierdem, you should add the full /subnet and not /32 | 12:23 |
| admin0 | can u show your config lines ? | 12:23 |
| ierdem | ok, I am changing the keepalived confs now, I will share the ping outputs and confs after this | 12:24 |
| admin0 | don't share the ping outputs :) .. share the haproxy/keeplive config from user_variables, run the haproxy playbook and then share the ip -4 a output from the 3 controllers | 12:24 |
| admin0 | also ip route show also helps | 12:25 |
| admin0 | in case your ip is overlapping with some other routes | 12:25 |
| jrosser | it was wondering why the 'destination unreachable' was coming from .42 when the host we are on is .40 | 12:27 |
| *** SiavashSardari has quit IRC | 12:27 | |
| ierdem | admin0 Now I got it, while I am doing this changes may you please give me advice about that Keepalived attaches both of external and internal lbvip addresses and I can reach internal lbvip but cannot external. Also I tried to attach different IPs to external lbvip from the same network. Even in this case my IP | 12:29 |
| ierdem | ... Even in this case may my IP overlaps? | 12:29 |
| admin0 | show me config first please.. coz i cannot make sense of that statement :) | 12:30 |
| ierdem | jrosser I tried to ping 22.200 from the infra3 (.22.42) host first http://paste.openstack.org/show/802229/, after you said I also tried it from infra1 (.22.40) http://paste.openstack.org/show/802230/ | 12:31 |
| jrosser | ok | 12:31 |
| admin0 | why are you not pasting ip -4 and route -n output and only failed pings ? | 12:32 |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 12:36 | |
| ierdem | admin0 user_variables file http://paste.openstack.org/show/802233/, ip -4 a http://paste.openstack.org/show/802235/, route -n http://paste.openstack.org/show/802236/ for 3 hosts | 12:37 |
| ierdem | admin0 I realized that route -n command shows 2 br-openstack interface | 12:38 |
| ierdem | on infra1 hosts | 12:38 |
| frickler | ierdem: do you really want 170.30.22.200 or is this a typo? | 12:39 |
| ierdem | We will use this IP, so yes I want it, its correct | 12:40 |
| jrosser | omg how did i miss that | 12:41 |
| frickler | from your other configuration it looks like you'd want 172.30.22.200 | 12:42 |
| jrosser | ierdem: 172.30 vs. 170.30? really? | 12:42 |
| admin0 | :) | 12:43 |
| ierdem | OMG :( | 12:43 |
| jrosser | we've all been there :) | 12:43 |
| admin0 | yep . now you know why config/ip/route helps to see | 12:43 |
| admin0 | ierdem, was there a reason to not use 172.29.x -- the openstack ranges ? | 12:44 |
| admin0 | as in the config ? | 12:45 |
| ierdem | Thank you so much, probably I could not find that if I check for hours. Thank you for your time, admin0 jrosser frickler | 12:45 |
| jrosser | no worries | 12:45 |
| ierdem | admin0 our clients ararnged this ranges for us, it was not our choice | 12:47 |
| admin0 | ok | 12:47 |
| *** gokhani has joined #openstack-ansible | 12:47 | |
| jrosser | 172.16.0.0/12 is quite likely already in use inside company networks as its regular rfc1918 address space | 12:49 |
| jrosser | certainly the case here for me so i can't use the default OSA cidr as they're already in use for something else | 12:49 |
| admin0 | in my case, i always ask for 3 vlans to use those and only the external is the one that office can access .. this way, i try to make sure anyone in the team can read the osa docs and exactly know how the cloud is built | 12:50 |
| jrosser | here reuse of address space is forbidden | 12:55 |
| *** SmearedBeard has joined #openstack-ansible | 13:12 | |
| *** cshen_ has joined #openstack-ansible | 13:13 | |
| *** cshen has quit IRC | 13:16 | |
| *** cshen_ has quit IRC | 13:54 | |
| admin0 | i set metering_hosts: *infrastructure_hosts .. but gnocchi api container is not being created | 13:59 |
| admin0 | isnt this line enough to create those ? | 13:59 |
| admin0 | https://pastebin.com/wwhft8Ht -- this is what I have | 14:00 |
| mgariepy | noonedeadpunk, you found the ansible-hardening issue i guess | 14:05 |
| noonedeadpunk | yeah | 14:10 |
| noonedeadpunk | it was so stupid.... | 14:10 |
| noonedeadpunk | that functional test was running with check.... | 14:10 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Fix linter errors https://review.opendev.org/c/openstack/ansible-hardening/+/771481 | 14:11 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Make possible to avoid aide installation https://review.opendev.org/c/openstack/ansible-hardening/+/772561 | 14:12 |
| mgariepy | yep saw the diff | 14:14 |
| mgariepy | computer are hard. | 14:14 |
| noonedeadpunk | indeed | 14:19 |
| mgariepy | i also found an issue on neutron ;) https://review.opendev.org/c/openstack/neutron/+/773165 | 14:23 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-lxc_container_create master: Avoid delegation to the target container https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/773690 | 14:25 |
| *** gshippey has joined #openstack-ansible | 14:45 | |
| *** spatel has joined #openstack-ansible | 14:50 | |
| *** gokhani has quit IRC | 14:51 | |
| *** cshen has joined #openstack-ansible | 14:51 | |
| *** spatel has quit IRC | 14:54 | |
| *** spatel has joined #openstack-ansible | 14:55 | |
| *** pcaruana has quit IRC | 15:03 | |
| spatel | jrosser hey, finally journalbeat is working but look like its sending too much information, I have used some drop field to trim it down. I am trying to find if somehow i can use container name in logs statement or filter because all current logs coming from infra* nodes and hard to know what container they are associated | 15:18 |
| *** miloa has quit IRC | 15:20 | |
| *** ilush has joined #openstack-ansible | 15:21 | |
| *** pcaruana has joined #openstack-ansible | 15:25 | |
| jrosser | spatel: the journal entries should all have info about where they come from i think..... | 15:28 |
| spatel | hmm! i am not able to find that information on my graylog server, may be i need parser to filter out that particular information | 15:30 |
| spatel | Let me play and see if i missing something. | 15:30 |
| jrosser | spatel: it's elasticsearch backed? i don't know about how graylog is set up but at this point i'd be using kibana to look at the raw data thats been collected | 15:42 |
| spatel | Graylog use elasticsearch to store all logs | 15:43 |
| spatel | graylog is just fancy GUI with enterprise level of feature but behind the scenes its elasticsearch storage | 15:46 |
| spatel | jrosser i can hookup kibana and look into raw data. | 15:50 |
| noonedeadpunk | yeah, and it's way simpler that elk stack, though, it has more interesting design (at least for alerts imo) | 15:51 |
| spatel | are you doing any log parsing before pumping data into ES? | 15:51 |
| jrosser | yes lots with logstash | 15:51 |
| noonedeadpunk | since graylog uses input chacne and proceed all alerts before writing to elastic | 15:51 |
| spatel | may be that is the key.. | 15:51 |
| noonedeadpunk | instead of reading from it all new stuff and then alert... | 15:51 |
| jrosser | see the pipeline in https://github.com/openstack/openstack-ansible-ops/blob/master/elk_metrics_7x/templates/logstash-pipelines.yml.j2 | 15:52 |
| jrosser | does things like unify the timestamps and make sense of the special fields that oslo.log is putting into the journal | 15:52 |
| jrosser | like the req_id i think | 15:52 |
| spatel | that is possible graylog not filtering proper field and just doing all standard stuff. | 15:52 |
| jrosser | thing is that the journal is not just a list of log text lines | 15:53 |
| jrosser | there are many other fields that you don't normally see | 15:53 |
| noonedeadpunk | there's an gelf input and journalctl module that puts stuff into correct fields | 15:53 |
| jrosser | ok cool | 15:53 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/graylog/graylog-forward-logs.yml | 15:54 |
| spatel | jrosser oh i can see logstash is doing all work there. maybe i need to look into that | 15:54 |
| jrosser | i can't say really - we don't have a graylog here so i can't compare what you might want to do | 15:54 |
| spatel | noonedeadpunk can journalbeat support gelf ? | 15:54 |
| noonedeadpunk | nah, I don't think so. gelf is graylog specific format | 15:55 |
| noonedeadpunk | but there's another tooling for elk iirc | 15:55 |
| spatel | gelf need some golang stuff etc.. i hate to install all those stuff if they are not useful enough | 15:55 |
| noonedeadpunk | actually I think it's journalbeat | 15:56 |
| spatel | let me keep playing to get more ideas. Thank you | 15:57 |
| gshippey | o/ | 15:58 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 16:00 |
| openstack | Meeting started Tue Feb 2 16:00:14 2021 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)" | 16:00 | |
| openstack | The meeting name has been set to 'openstack_ansible_meeting' | 16:00 |
| noonedeadpunk | # topic office hours | 16:00 |
| noonedeadpunk | #topic office hours | 16:00 |
| *** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)" | 16:00 | |
| noonedeadpunk | \o/ | 16:00 |
| *** ilush has quit IRC | 16:01 | |
| noonedeadpunk | So, with release of amqp 5.0.5 it seems we need to speed up with SSL topic | 16:02 |
| noonedeadpunk | I tried to cover some of the comments raised for https://review.opendev.org/c/openstack/openstack-ansible-specs/+/758805 | 16:02 |
| noonedeadpunk | but maybe let's discuss them? | 16:02 |
| noonedeadpunk | `What to do when certs expire (including the root cert) and how that should be managed, including their default lifetimes` | 16:03 |
| noonedeadpunk | I'd say that we should just have a flag that will force role to renew certificate or root ca | 16:03 |
| *** macz_ has joined #openstack-ansible | 16:04 | |
| *** SmearedBeard has quit IRC | 16:05 | |
| noonedeadpunk | I don't feel like we should be watching after expiration dates atm out of the box. Considering we should be able to work with user provided certificates and let's encrypt where applicable | 16:05 |
| *** SmearedBeard has joined #openstack-ansible | 16:05 | |
| noonedeadpunk | But yes, we totally need to have a valid mechanism of root ca update without making cluster stuck because of that update | 16:05 |
| noonedeadpunk | also imo cert revocation systems is kind of overkill at the moment as well | 16:06 |
| noonedeadpunk | it's probably nice to have feature, but where we are at the moment and what needs to be done overall is kind of... | 16:07 |
| noonedeadpunk | Btw I already asked for repo creation https://review.opendev.org/q/topic:%22osa%252Fpki%22+(status:open%20OR%20status:merged) | 16:07 |
| jrosser | o/ hello | 16:09 |
| andrewbonney | For expiry I guess the main detail is making sure the expiring one remains trusted whilst rollover happens | 16:10 |
| jrosser | the root CA would be extremely long lived and in intermediate is the more likley thing to need rotating? | 16:10 |
| jrosser | so probably two different things, rotate root CA (very very infrequent unless some security incident with it) | 16:11 |
| noonedeadpunk | yes, totally, 2 different flags | 16:11 |
| jrosser | re-issue service cert + intermediate bundle against a new intermediate, and that should be much much easier than rotating a root | 16:12 |
| noonedeadpunk | but again with remaining root ca trusted even when new one is in place | 16:12 |
| noonedeadpunk | because otherwise I don't see how to update root. It was super clever suggestion I was not aware of | 16:13 |
| noonedeadpunk | I mean https://tools.ietf.org/html/rfc4210#section-4.4 | 16:13 |
| noonedeadpunk | hm I guess I'm a bit lost in terminology ( So intermediate is root CA and "root" in private key, right? | 16:14 |
| noonedeadpunk | or you're talking about something extra? As intermediate I guess is addition to CA one? | 16:15 |
| jrosser | generally certs for services are not signed directly with the private key of the root CA | 16:16 |
| jrosser | the only thing you use that for is to generate an intermediate CA cert/key, and you can have as many of those as you like | 16:16 |
| noonedeadpunk | and you issue certificates with intermediate ones? | 16:16 |
| jrosser | which is good, because you can revoke/change an intermediate whenever you like without affecting the trust of stuff signed from a different intermediate | 16:17 |
| jrosser | it's like a tree | 16:17 |
| noonedeadpunk | I just never was digging deep in how certs are issued on provider side | 16:17 |
| jrosser | thats why generally you make the root CA valid for a very very long time | 16:17 |
| jrosser | but you can make the lifetime of the intermediates shorter, and the pain of rolling them is really much smaller than if you wanted to roll the entire root CA | 16:18 |
| noonedeadpunk | I'm not sure if you can put CA in trust store... I guess you can? | 16:18 |
| jrosser | oh absolutely, thats pretty much what it contains | 16:18 |
| noonedeadpunk | Just in terms that we won't need to define intermediate chan to the services since they will be trusted system wide? | 16:19 |
| noonedeadpunk | ok, I guess I got the idea. Need to read more anyway | 16:19 |
| jrosser | sure well i think we should write more and maybe test some of this | 16:20 |
| jrosser | there doesnt seem to be anything too major we have missed from the comments | 16:22 |
| noonedeadpunk | yeah, I guess so | 16:23 |
| noonedeadpunk | but all comments were really valid though | 16:24 |
| noonedeadpunk | regarding hardening - it seems I got role unstuck https://review.opendev.org/c/openstack/ansible-hardening/+/771481 | 16:25 |
| noonedeadpunk | but I'm not sure in 1 thing there, which makes role compatible with ansible 2.10 and later only | 16:26 |
| noonedeadpunk | which is `truthy(convert_bool=True)` filter | 16:26 |
| jrosser | for master/osa thats fine, not sure how much use we get beyond that? | 16:26 |
| noonedeadpunk | I guess we can use for V as well? | 16:32 |
| noonedeadpunk | the main concern is that role was used not only by OSA I guess | 16:33 |
| noonedeadpunk | it has been used even outside of the openstack... | 16:33 |
| jrosser | i expect you used the new 2.10 keyword for a good reason? | 16:34 |
| noonedeadpunk | good question... I used to replace https://opendev.org/openstack/ansible-hardening/src/branch/master/tasks/rhel7stig/accounts.yml#L147 to fix linters | 16:34 |
| noonedeadpunk | but... item.value here might be either int or bool or string | 16:35 |
| noonedeadpunk | and I'm out of good ideas how to test them except comparing to empty string or with truthy test... | 16:35 |
| noonedeadpunk | because bool for string will be false and you can't check legth of int or bool... | 16:36 |
| noonedeadpunk | we can leave it as is and add noqa here | 16:36 |
| jrosser | sounds reasonable as its a difficult test to do properly | 16:39 |
| jrosser | are there bugs to look at? | 16:40 |
| noonedeadpunk | there were no new ones. But there were some untriaged left from last year | 16:41 |
| noonedeadpunk | #topic bug triage | 16:41 |
| *** openstack changes topic to "bug triage (Meeting topic: openstack_ansible_meeting)" | 16:41 | |
| noonedeadpunk | I guess I just found extra one:) | 16:41 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/templates/nova.conf.j2#L241 - this will be always false right? | 16:41 |
| noonedeadpunk | good place to use new trythy filter as well? | 16:42 |
| jrosser | thats just broken now? | 16:43 |
| jrosser | string -> false | 16:43 |
| noonedeadpunk | yeah... | 16:43 |
| noonedeadpunk | just faced it | 16:43 |
| * noonedeadpunk upgrading T->V directly | 16:44 | |
| jrosser | how does that even work at all then | 16:44 |
| jrosser | V ceph job for example | 16:44 |
| noonedeadpunk | we have a lot of diskspace :p | 16:44 |
| noonedeadpunk | so nova uses local storage for ephemeral drives | 16:45 |
| jrosser | andrewbonney: ^ one to add to the list! :) | 16:45 |
| noonedeadpunk | and nova_rbd_inuse is defined not correctly as well... doh | 16:46 |
| jrosser | hmm seems like we need a LP bug for this | 16:47 |
| noonedeadpunk | yeah, will spawn some | 16:47 |
| * noonedeadpunk fixing environment | 16:47 | |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server stable/victoria: Bring db setup vars in line with other roles https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/772550 | 16:48 |
| noonedeadpunk | oh, btw, what about galera issue we're facing... | 16:49 |
| noonedeadpunk | I think it might be worth reaching galera folks for some help with that? | 16:49 |
| noonedeadpunk | oh, and https://bugs.launchpad.net/openstack-ansible/+bug/1908703 | 16:50 |
| openstack | Launchpad bug 1908703 in openstack-ansible "federation domain not configured correct" [Undecided,New] | 16:50 |
| jrosser | yeah, i made a paste with the journal when galera had not started properly, that should be useful | 16:51 |
| jrosser | gshippey: are you around? | 16:52 |
| gshippey | I am | 16:52 |
| jrosser | the federation bug just mentioned before, does it look like the example mapping we give in the docs is missing some things for the default domain? | 16:52 |
| gshippey | Just had a quick look at the docs, and the domain_id on the trusted_idp is there. Give me a sec, need to find some old patches of mine | 16:57 |
| noonedeadpunk | annoying thing is that gerrit now not linked to LP | 16:58 |
| gshippey | if anything looking at the keystone_sp structure in https://docs.openstack.org/openstack-ansible-os_keystone/latest/ the federated_identities should be pulling the domain from the idp rather than the other way around | 16:58 |
| jrosser | hmm looks like pertoft is not here in irc? | 16:59 |
| jrosser | gshippey: if you would be able to follow up to the reply on that bug it would be awesome | 17:00 |
| jrosser | noonedeadpunk: we are encountering this in our upgrade work https://github.com/ansible/ansible/issues/72776 | 17:01 |
| noonedeadpunk | yeah I saw patch from andrewbonney, but didn't have time to read bug carefully | 17:03 |
| gshippey | I will do, essentially I don't think the domain of the idp functionally matters and to maintain backwards compatibility specifying the domain of the idp has to be optional. | 17:03 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Fix nova_libvirt_images_rbd_pool check https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/773732 | 17:03 |
| jrosser | noonedeadpunk: i am not sure if it is triggered by something specific in our environment | 17:03 |
| andrewbonney | I'm still investigating at the moment but I think in our case it's because our deploy host doesn't have name resolution for the container hosts | 17:03 |
| jrosser | to do with the way hosts vs. IPs in the inventory | 17:04 |
| andrewbonney | Or rather that's why it doesn't show up elsewhere | 17:04 |
| jrosser | there was discussion recently if the OSA things should be adding entries to the deploy host /etc/hosts | 17:05 |
| jrosser | becasue the behavour currently will be different if infra1 is the deploy host vs. some dedicated deploy host | 17:05 |
| noonedeadpunk | and in your case deploy host is placed on infra? | 17:06 |
| jrosser | and we would never see this sort of thing in CI jobs because deploy==infra host | 17:06 |
| jrosser | no it's seperate | 17:06 |
| noonedeadpunk | it's also separate for me... | 17:07 |
| noonedeadpunk | but anyway I see nothing wrong in setting hosts file to the deploy host as well | 17:07 |
| noonedeadpunk | except it's not so easy to achieve I guess) | 17:07 |
| noonedeadpunk | as we don't want to run whole openstack_hosts against deploy | 17:07 |
| jrosser | no we do not want to do that | 17:08 |
| jrosser | i think we let andrewbonney dig into this and see what the root cause is | 17:08 |
| jrosser | there is a further instance of it beyond the patch today which cannot be fixed in a straightforward way | 17:09 |
| noonedeadpunk | Btw I'm thinking if we should release 22.0.1 now (once all V backports will land) and 22.1.0 after that I guess? | 17:09 |
| noonedeadpunk | as point release used to mark that it's pretty safe to upgrade?:) | 17:09 |
| jrosser | sounds like we are both working though V upgrades on prod environments and catching a few things | 17:10 |
| jrosser | so yes a 22.1.0 when all that is settled would be good | 17:11 |
| noonedeadpunk | k | 17:11 |
| noonedeadpunk | #endmeeting | 17:11 |
| *** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/osa-review-board-v3" | 17:11 | |
| openstack | Meeting ended Tue Feb 2 17:11:11 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:11 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-02-02-16.00.html | 17:11 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-02-02-16.00.txt | 17:11 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-02-02-16.00.log.html | 17:11 |
| *** hamzaachi has quit IRC | 17:16 | |
| *** macz_ has quit IRC | 17:16 | |
| gshippey | https://bugs.launchpad.net/openstack-ansible/+bug/1890492 and https://bugs.launchpad.net/openstack-ansible/+bug/1720535 should be set to wont fix? We don't use those nova filters in newer versions do we? | 17:28 |
| openstack | Launchpad bug 1890492 in openstack-ansible "Rocky Deployments Fail During repo_build" [Undecided,New] | 17:28 |
| openstack | Launchpad bug 1720535 in openstack-ansible "lxd.filters out of date" [Medium,Confirmed] | 17:28 |
| noonedeadpunk | gshippey: yeah, I guess so | 17:33 |
| *** d34dh0r53 has quit IRC | 17:42 | |
| *** jbadiapa has quit IRC | 17:42 | |
| *** spatel has quit IRC | 17:44 | |
| *** spatel has joined #openstack-ansible | 17:46 | |
| gshippey | also @jrosser didn't you fix this?? https://bugs.launchpad.net/openstack-ansible/+bug/1900808 | 17:47 |
| openstack | Launchpad bug 1900808 in openstack-ansible "keystone SAML2 federation installation error " [Undecided,New] | 17:47 |
| *** d34dh0r53 has joined #openstack-ansible | 17:50 | |
| jrosser | gshippey: yes though I thing because the launchpad/Herriot integration no longer works that’s not been automatically marked as resolved | 17:52 |
| jrosser | *Gerrit | 17:53 |
| noonedeadpunk | and it's not a prio for infra btw to fix integration | 17:53 |
| noonedeadpunk | "fix when convenient" | 17:54 |
| jrosser | I maybe even forgot to put the header in the patch :( | 17:54 |
| *** maharg101 has quit IRC | 18:02 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Fix lxc_hosts_container_image_url condition https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/773781 | 18:07 |
| *** SmearedBeard has quit IRC | 18:10 | |
| *** macz_ has joined #openstack-ansible | 18:12 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 18:34 | |
| *** rpittau is now known as rpittau|afk | 18:39 | |
| *** andrewbonney has quit IRC | 19:15 | |
| *** ierdem has quit IRC | 19:23 | |
| *** ierdem has joined #openstack-ansible | 19:26 | |
| *** evrardjp has quit IRC | 19:51 | |
| *** maharg101 has joined #openstack-ansible | 19:59 | |
| *** maharg101 has quit IRC | 20:04 | |
| *** maharg101 has joined #openstack-ansible | 20:08 | |
| *** evrardjp has joined #openstack-ansible | 20:18 | |
| *** evrardjp has quit IRC | 20:26 | |
| *** evrardjp has joined #openstack-ansible | 20:27 | |
| *** maharg101 has quit IRC | 20:41 | |
| *** ierdem has quit IRC | 21:07 | |
| *** spatel has quit IRC | 21:09 | |
| *** cshen has quit IRC | 21:37 | |
| *** cshen has joined #openstack-ansible | 21:39 | |
| *** cshen has quit IRC | 21:44 | |
| *** cshen has joined #openstack-ansible | 21:53 | |
| *** cshen has quit IRC | 22:05 | |
| *** cshen has joined #openstack-ansible | 22:17 | |
| *** ChiTo has joined #openstack-ansible | 22:24 | |
| ChiTo | Hi OSA team | 22:24 |
| ChiTo | I remember there is a variable I can pass as -e "" to force a reinstallation on a specific playbook, but I don't recall it, by chance can you assist me? | 22:25 |
| ChiTo | I would like to make sure that I can reinstall all the packages for a specific service (in my case Barbican) | 22:25 |
| *** gshippey has quit IRC | 22:36 | |
| admin0 | -e build_venv=true ? | 22:37 |
| *** maharg101 has joined #openstack-ansible | 22:39 | |
| *** maharg101 has quit IRC | 22:43 | |
| *** spatel has joined #openstack-ansible | 22:46 | |
| ChiTo | admin0: Thanks admin0 that is the good one | 22:46 |
| ChiTo | by chance do you know if OSA is able to install the barbican UI? | 22:47 |
| ChiTo | I noticed that there are no variables associated to horizon_enable_barbican_ui | 22:47 |
| *** spatel has quit IRC | 23:13 | |
| jrosser | ChiTo: unless there is a specific reason to use venv_rebuild=true then i would be cautious with it | 23:22 |
| jrosser | it would be much better to submit a bug report for the reason that you think you need that flag | 23:22 |
| ChiTo | jrosser: 100% agreed, since my deployment was on VMs treated as bare metal sometimes it is difficult to me to remove the venvs | 23:23 |
| jrosser | admin0: you should read this https://bugs.launchpad.net/openstack-ansible/+bug/1914301 | 23:23 |
| openstack | Launchpad bug 1914301 in openstack-ansible "passing venv_rebuild=true leaves repo server in unusable state" [Undecided,New] | 23:23 |
| jrosser | you can just delete them | 23:23 |
| *** spatel has joined #openstack-ansible | 23:23 | |
| jrosser | like with rm | 23:23 |
| jrosser | or if it is an lxc deployment delete/recreate the containers | 23:25 |
| ChiTo | jrosser: got it, thanks for the hint, I think that would be the best approach to go. Next regions I will deploy will be on LXC to avoid this situation | 23:25 |
| jrosser | ok well also do check out that bug i just linked | 23:25 |
| jrosser | not sure if you will have a repo host or not for a metal deploy, so this may not be an issue | 23:26 |
| ChiTo | jrosser: btw I noticed that the redhat7 yaml variables file required a ksmtuned package, but on rhel 7 or centos7 it is not a pkg but a kernel module, when I was setting some specific filters I found that the playbook failed because the ksmtuned was not found, not sure if I have to open a ticket due this package exists only in Ubuntu | 23:26 |
| ChiTo | jrosser: Agreed thanks for the clarification, definitely I will deploy on LXC | 23:26 |
| jrosser | ChiTo: you mean this? https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/vars/redhat.yml#L82-L83 | 23:28 |
| ChiTo | jrosser: that is correct | 23:29 |
| openstackgerrit | Merged openstack/openstack-ansible master: Increase git clone depth from 10 to 20 https://review.opendev.org/c/openstack/openstack-ansible/+/773352 | 23:29 |
| jrosser | oh hmm "The ksm service is included in the qemu-kvm package." | 23:29 |
| jrosser | from here https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/chap-ksm | 23:29 |
| jrosser | yes so that looks like a bug | 23:29 |
| jrosser | does the kernel module get loaded properly? | 23:29 |
| ChiTo | I just got: paste.openstack.org/show/802264 | 23:30 |
| ChiTo | jrosser: Yes it was loaded properly that i why I just commented it out | 23:31 |
| ChiTo | to avoid to look for the package | 23:31 |
| jrosser | ok so it looks like a simple fix | 23:31 |
| ChiTo | agreed | 23:31 |
| jrosser | if you are using Centos note that we don't yet have a good answer for what to do woth centos8-stream | 23:32 |
| jrosser | the things we need from EPEL are not working with stream | 23:33 |
| ChiTo | jrosser: got it, I am aware of it, I think eventually I will go with Ubuntu | 23:33 |
| ChiTo | jrosser: In the meantime I will continue only with CentOS 7.X and then to see what happens with a repo for Stream, but in my case it is very likely I will go to Ubuntu | 23:33 |
| *** spatel has quit IRC | 23:34 | |
| jrosser | i need to take another look at it, things may have changed | 23:34 |
| ChiTo | thx jrosser for your recommendations | 23:35 |
| *** ChiTo has quit IRC | 23:45 | |
| *** spatel has joined #openstack-ansible | 23:50 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!