| opendevreview | Merged openstack/openstack-ansible-os_designate stable/yoga: Fix race condition during designate setup https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/867013 | 00:05 |
|---|---|---|
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Separate OVN gateway functions from ovn-controllers https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/760647 | 00:26 |
| *** chandankumar is now known as chkumar|ruck | 04:41 | |
| noonedeadpunk | hm, what's going to be installed on northd? | 09:02 |
| noonedeadpunk | As I think we don't have neutron to be installed there? | 09:03 |
| jrosser | i was just looking at my remianing ironic patches and something looks broken | 09:03 |
| noonedeadpunk | yup because of https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/863546 | 09:04 |
| noonedeadpunk | As we still copy neutron configs when they're not needed | 09:04 |
| noonedeadpunk | and smart_sources rely on it being around | 09:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Do not provision neutron config when not needed https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/867087 | 09:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Update AIO to use OVN-style provider network for Octavia https://review.opendev.org/c/openstack/openstack-ansible/+/867052 | 09:11 |
| noonedeadpunk | Hope this will do it | 09:12 |
| jrosser | oh i think i saw that the image module now supports tags | 09:24 |
| jrosser | we should be able to get rid of a command: here https://github.com/openstack/openstack-ansible-os_octavia/blob/master/tasks/octavia_amp_image.yml | 09:24 |
| noonedeadpunk | Yup they should now :) | 09:43 |
| noonedeadpunk | I wonder wtf is OPNFV CI | 09:43 |
| noonedeadpunk | jrosser: TBH I still think we should create some openstack_resources role in plugins repo that can be called from roles to setup stuff, like images/flavors/networks/aggregates/etc | 09:46 |
| noonedeadpunk | and maybe move service_setup in there | 09:46 |
| jrosser | i was also thinking about trying to write a module for wheel build | 09:47 |
| jrosser | but i first also wanted to look at how many tasks we run for each role in total | 09:48 |
| jrosser | to see where it might be worth putting work in to reduce the number of tasks | 09:48 |
| noonedeadpunk | Um, it's a bit different I guess? As it was unification vs reducing. As we have different code to upload images for magnum/octavia/trove/smth else | 09:49 |
| noonedeadpunk | Or well, I have more narrow use-case, which is represented by this patch: https://review.opendev.org/c/openstack/openstack-ansible/+/854235 | 09:51 |
| jrosser | the trouble with a generic resources role would be maybe introducing many many skipped tasks for all the things that are not wanted in some other role | 09:52 |
| jrosser | i think we suffer from this with systemd_service that has so many special things it does | 09:52 |
| noonedeadpunk | The problem with systemd_service is that we don't have proper structure for it... | 09:52 |
| noonedeadpunk | As we should likely be splitting these cases into different task sets | 09:53 |
| noonedeadpunk | so that include could be skipped if not defined | 09:53 |
| noonedeadpunk | Skipping inlcude is fast (I have impression it's faster then skipping task even, but haven't tested that) | 09:54 |
| noonedeadpunk | Also we could do tasks_from | 09:54 |
| jrosser | i think we have a special case for skipping in the connection plugin too | 09:54 |
| jrosser | like early bail-out in some situations | 09:55 |
| noonedeadpunk | So my usecase with host aggregates is to automatically add compute from az1 to it's aggregate as part of it's setup. | 09:58 |
| noonedeadpunk | And I struggle to see good approach for that.... | 09:58 |
| noonedeadpunk | Like add to end of os-nova-install playbook... | 09:59 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Remove support for calico ml2 driver. https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/866123 | 11:02 |
| *** dviroel|afk is now known as dviroel|rover | 11:06 | |
| noonedeadpunk | So https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/867087 looks good according to https://review.opendev.org/c/openstack/openstack-ansible/+/867052 results | 11:33 |
| noonedeadpunk | I think we should figure out wtf happening in CI with keystone during tempest tests... | 11:39 |
| noonedeadpunk | That's super annoying and cause a lot of CI resources waste | 11:39 |
| *** frenzy_friday is now known as frenzy_friday|food | 12:38 | |
| opendevreview | Merged openstack/openstack-ansible-os_ironic master: Tidy definition of http dir for inspector https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/866132 | 12:44 |
| mgariepy | https://zuul.opendev.org/t/openstack/build/d143a14c71564135bd6adfdef8af9f4a/log/logs/host/keystone-wsgi-public.service.journal-10-38-14.log.txt#15059-15084 | 13:27 |
| mgariepy | that's probably not too good. | 13:27 |
| jamesdenton | mornin | 13:29 |
| mgariepy | https://zuul.opendev.org/t/openstack/build/d143a14c71564135bd6adfdef8af9f4a/log/logs/host/keystone-wsgi-public.service.journal-10-38-14.log.txt#25670-25721 | 13:29 |
| mgariepy | morning jamesdenton | 13:33 |
| noonedeadpunk | o/ | 13:38 |
| jamesdenton | hi hi hi | 13:39 |
| mgariepy | keystone it's not 100% clear why it went down :/ | 13:39 |
| mgariepy | L7 down, means that the app didn't respond in time i guess. | 13:42 |
| mgariepy | maybe it got saturated here ? https://zuul.opendev.org/t/openstack/build/d143a14c71564135bd6adfdef8af9f4a/log/logs/etc/host/uwsgi/keystone-wsgi-public.ini.txt#10 | 14:02 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible master: Increase thread/process to 2 for keystone https://review.opendev.org/c/openstack/openstack-ansible/+/867113 | 14:14 |
| noonedeadpunk | Yeah, might be | 14:15 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible master: Update AIO to use OVN-style provider network for Octavia https://review.opendev.org/c/openstack/openstack-ansible/+/867052 | 14:17 |
| noonedeadpunk | I thought for some reason it doesn't matter much as Apache is main headache but worth trying | 14:17 |
| mgariepy | oops. | 14:17 |
| mgariepy | lol should have rebased lol | 14:17 |
| noonedeadpunk | doesn't matter much | 14:18 |
| noonedeadpunk | should work anyway | 14:18 |
| mgariepy | yeah i know | 14:18 |
| mgariepy | it's kinda obscure why haproxy disconnect the keystone backend. | 14:19 |
| mgariepy | but if apache is waiting on the data from uswgi. it might explain the timeout haproxy sees. | 14:20 |
| noonedeadpunk | haproxy is supposed to recheck kind of.... | 14:20 |
| noonedeadpunk | and it's down just for a second according to log | 14:21 |
| noonedeadpunk | but yeah | 14:21 |
| mgariepy | maybe we could set fall `2 or 3` and rise 1 ? https://zuul.opendev.org/t/openstack/build/d143a14c71564135bd6adfdef8af9f4a/log/logs/etc/host/haproxy/conf.d/keystone_service.txt#32 | 14:23 |
| mgariepy | once haproxy thinks the backend is dead. it disconnect it and no one can connect back | 14:23 |
| noonedeadpunk | oh, why in the world it's like that | 14:31 |
| noonedeadpunk | I haven't spotted that | 14:31 |
| noonedeadpunk | IMO it shouldn't be like fall 1 | 14:31 |
| mgariepy | it should fall either. | 14:32 |
| mgariepy | shouldn't** | 14:32 |
| noonedeadpunk | oh, we have fall 1 raise 1 for everything? | 14:33 |
| noonedeadpunk | hm | 14:33 |
| noonedeadpunk | We override that in CI? | 14:33 |
| noonedeadpunk | as default is 3 | 14:34 |
| noonedeadpunk | hm, these 2 variables are never used https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/defaults/main.yml#L25-L26 | 14:35 |
| noonedeadpunk | wtf the logic is that https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/templates/service.j2#L142 | 14:35 |
| noonedeadpunk | o_O | 14:36 |
| *** dviroel|rover is now known as dviroel|rover|afk | 14:40 | |
| mgariepy | well. | 14:40 |
| mgariepy | did infra job failed less on that issue ? | 14:40 |
| mgariepy | hrm.. https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/829899 | 14:42 |
| mgariepy | lol | 14:42 |
| mgariepy | do we need to clean openstack-ansible repo from calico as well ? | 14:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Make use of haproxy_rise and haproxy_fall variables https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/867116 | 14:44 |
| noonedeadpunk | mgariepy: yup. And there was patch for that | 14:45 |
| noonedeadpunk | I jsut think that we need to wait for neutron fix to land first | 14:45 |
| mgariepy | where is the patch ? | 14:45 |
| mgariepy | i did look but didn't found it. | 14:46 |
| mgariepy | only saw the os_neutron on. | 14:46 |
| mgariepy | one* | 14:46 |
| noonedeadpunk | sec | 14:46 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible/+/866119 | 14:46 |
| noonedeadpunk | Well, it would still require recheck... | 14:46 |
| *** frenzy_friday|food is now known as frenzy_friday | 14:50 | |
| noonedeadpunk | damn, https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/867087 is failing in gates :( | 14:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Do not provision neutron config when not needed https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/867087 | 14:52 |
| noonedeadpunk | with `ERROR neutron OSError: write error` huh | 14:53 |
| mgariepy | neutron went down on haproxy also. | 14:55 |
| noonedeadpunk | Yeah.... | 14:56 |
| noonedeadpunk | wait? | 14:56 |
| noonedeadpunk | for what patch? | 14:56 |
| mgariepy | the one that fails gates. | 14:57 |
| noonedeadpunk | yeah, but it sounds more like underlying FS issue then haproxy or smth... | 14:58 |
| noonedeadpunk | as neutron-server died with write error | 14:58 |
| spatel | mgariepy quick question, are you using snap for container for deployment host or lxc ? | 15:40 |
| noonedeadpunk | hm, it would be fun to pack osa in snap package and do bootstrap in install.... | 15:49 |
| noonedeadpunk | also fun for auto-updates | 15:49 |
| spatel | I wish we can put small doc related how to spin up target node in LXD | 15:51 |
| noonedeadpunk | I wish there would be option for LXD that is not snap.... | 15:52 |
| spatel | you can install lxc instead of lxd without snap | 15:52 |
| jrosser | i think that debian bookworm will have lxd5.0 as apt packages rather than snap | 15:52 |
| noonedeadpunk | that is exactly what we're doing for osa? | 15:53 |
| noonedeadpunk | yes, I just found that | 15:53 |
| jrosser | hopefully that will make its way back into ubuntu :) | 15:53 |
| noonedeadpunk | Like someone jsut re-build it and publish in ppa... | 15:53 |
| spatel | yes.. but why not snap/lxd ? are they not good? | 15:53 |
| jrosser | snap is really (imho) meant for client / desktoppy things | 15:54 |
| jrosser | and as such it auto-updates in a way you cannot prevent unless you do horrible hacks | 15:54 |
| spatel | ouch! | 15:55 |
| noonedeadpunk | I've installed thunderbird with snap jsut to get version with Matrix client inside and still regret that | 15:55 |
| jrosser | so to have what is essentially a complete hypervisor randomly able to update / restart itself outside of any planned maintainance is pretty gross | 15:55 |
| noonedeadpunk | It asks to restart Thunderbird like weekly. But the problem is that restart and even laptop reboot didn't help to get rid of the message that restart is required :D | 15:56 |
| jrosser | we do run a bunch of stuff in snap/lxd here | 15:56 |
| noonedeadpunk | Well, I've heard that LXD snap has merged some workaround to prevent LXC restart on snap update or smth | 15:56 |
| jrosser | but we have to do a horrible hack to prevent the updates | 15:56 |
| noonedeadpunk | It's actually not that horrible last time you've shown it :D | 15:57 |
| noonedeadpunk | but yes, totally not the approach that should be mainstreamed | 15:57 |
| jrosser | one time it updated to a version with a bug and took down a bunch of supposedly H/A things because they all updated mostly at the same time | 15:57 |
| jrosser | like nameservers :( | 15:58 |
| noonedeadpunk | FUN | 15:58 |
| noonedeadpunk | (Fucked Up Nameservers) | 15:58 |
| jrosser | i would totally use a ppa instead of a snap if someone rebuilt the debian one | 15:58 |
| noonedeadpunk | I wonder how hard that would be | 15:59 |
| noonedeadpunk | For centos there's already CORP that exists | 15:59 |
| spatel | on ubuntu i am trying to install lxc like OSA use then what package i need to install | 16:00 |
| spatel | by default when i did apt install lxd (it install snap :( ) | 16:00 |
| noonedeadpunk | Lol, we have "create snap package" button on our OSA launchpad page.... | 16:01 |
| jrosser | spatel: for a deploy host its really fine | 16:02 |
| noonedeadpunk | damn, installing osa with snap sounds fun in my head | 16:02 |
| spatel | it would be much quicker | 16:02 |
| spatel | I am playing with kolla-ansible and i found kolla is very fast because image is already and all you need drop config and start container | 16:03 |
| spatel | wish osa has something like that where container is ready and we can drop config and start service | 16:03 |
| ElDuderino | same here. kolla has been pretty cool. | 16:03 |
| noonedeadpunk | you loose a lot of flexability with that approach | 16:04 |
| spatel | I did upgrade kolla in 10 minute Vs OSA took few hours :( | 16:04 |
| noonedeadpunk | also fwiw kolla does install only released tags of openstack from pypi | 16:05 |
| ElDuderino | we run OSA in prod, and I'm a huge fan of it, but kolla was pretty cool. | 16:05 |
| noonedeadpunk | so good luck with local forks | 16:05 |
| noonedeadpunk | or some extra code that you want in | 16:05 |
| spatel | In that case we can create own images right? | 16:05 |
| noonedeadpunk | Likely? | 16:06 |
| spatel | Yes, that is what we doing. when we need to patch neutron we do create image in docker with patch and rollout | 16:06 |
| mgariepy | spatel, i use snap lxd | 16:07 |
| noonedeadpunk | Why you need kolla then if you build images on your own... | 16:07 |
| spatel | kolla create image with software re-compiled in it so all you need spin up docker and ansible just drop neutron.conf file and done | 16:07 |
| noonedeadpunk | to drop neutron.conf? | 16:08 |
| spatel | Image building process is one time | 16:08 |
| spatel | it use overlay filesystem so kind of quick | 16:08 |
| noonedeadpunk | I was thinking about building in code in lxc images, but I kind of failed to see how that would speed things up | 16:09 |
| spatel | I am not saying kolla is best and osa not but just saying this is what i noticed. whenever i want to rollout in kolla is super quick and fast | 16:09 |
| noonedeadpunk | except storing them somewhere on repo_container | 16:09 |
| noonedeadpunk | but that covers only 3 hosts, and osa spends most time on deployment of computes and net nodes anyway | 16:09 |
| noonedeadpunk | that are still bare metal | 16:09 |
| noonedeadpunk | like deploying heat or magnum takes couple of minutes as well | 16:10 |
| noonedeadpunk | and upgrading them same time | 16:11 |
| noonedeadpunk | also install from pypi is super fast.... | 16:12 |
| noonedeadpunk | Maybe we should also get an option to jsut install from pypi.... | 16:13 |
| noonedeadpunk | and get our releases to tags instead of SHAs | 16:13 |
| spatel | whatever we can do in advance and pre-build so next time just rolling out code.. | 16:14 |
| spatel | mgariepy how do you copy file in lxd container? | 16:21 |
| spatel | like in OSA lxc we can browse folder inside /var/lib/lxc/<container/rootfs | 16:21 |
| mgariepy | you can do a lxd snapshot and send it over. | 16:22 |
| mgariepy | but it's not live. it doesn't update automatically either. | 16:23 |
| spatel | all i want copy bunch of files inside container | 16:23 |
| spatel | i can use scp but wonder if simple way to copy file in container with browse filesystem from host machine | 16:24 |
| mgariepy | you can use the lxd client to copy files around | 16:26 |
| mgariepy | it won't be easier i don't think | 16:26 |
| spatel | there is a way - ls -l /var/snap/lxd/common/lxd/storage-pools/lxd/containers/mycontainer/rootfs/ | 16:27 |
| spatel | I am trying to create osa deployment node from existing on.. | 16:35 |
| spatel | Just created lxd container and checked out repo and my production tag git checkout 23.3.0 | 16:35 |
| spatel | now copying /etc/openstack_deploy inside my lxd container | 16:36 |
| spatel | then going to run scripts/bootstrap-ansible.sh | 16:36 |
| spatel | and copy /etc/hosts + ssh keys | 16:36 |
| jrosser | spatel: `lxc file push <blah>` | 16:38 |
| spatel | does that work with directory also? | 16:38 |
| jrosser | --recursive --create-dirs apparently | 16:39 |
| spatel | nice | 16:41 |
| jrosser | you can use cloud-init with lxd so you can use that to insert your ssh key | 16:45 |
| jrosser | then just treat it like a regular server | 16:45 |
| spatel | I am playing with lxd to get more familiar | 16:47 |
| spatel | jrosser should i run scripts/bootstrap-ansible.sh or better copy all files from original deployment host like /etc/ansible directory etc. | 16:48 |
| jrosser | well, if everything is done right only two things matter, the sha you check out the openstack-ansible repo to | 16:49 |
| jrosser | and the contents of /etc/openstack-deploy | 16:50 |
| mgariepy | ssh-keys and maybe ansible-vault stuff. | 16:50 |
| spatel | vault ? | 16:50 |
| jrosser | like if you patch any roles then hopefully you've already got those as forks on github or something | 16:51 |
| spatel | in my case i didn't do any patching | 16:51 |
| spatel | let me run scripts/bootstrap-ansible.sh and see, technically it should download all roles etc.. | 16:52 |
| mgariepy | depending on how you encrypt your secret with ansible-vault | 16:52 |
| jrosser | i think also in the past there were some files in ~root - maybe octavia CA or something | 16:52 |
| jrosser | but that was an oversight | 16:52 |
| mgariepy | on this i need to go. i got a sick kid here.. | 16:53 |
| mgariepy | have a nice weekend | 16:53 |
| spatel | have a good weekend!! thanks you.. its flu time so make sure nothing serious | 16:53 |
| noonedeadpunk | wait, I have recalled - we have distro install for those who prefer deployment speed ! | 16:54 |
| noonedeadpunk | as that is really fast as well as with upgrades | 16:55 |
| noonedeadpunk | spatel: the thing is that we pre-build wheels and rolling them out net time is way faster then building them | 16:57 |
| spatel | noonedeadpunk you are correct but in kolla there is a option for source and it works as fast as distro | 16:57 |
| noonedeadpunk | but building wheels itself take quite some time - installing from pypi like kolla do is way faster | 16:57 |
| noonedeadpunk | it's not source - it's pypi | 16:57 |
| noonedeadpunk | it's huge difference in process and result | 16:57 |
| spatel | yes.. its pypi but they called it method: source | 16:58 |
| noonedeadpunk | While you can do re-install anything with pre-built wheels without external connectivity - in kolla you would need to mirror pypi which is tough thing to do | 16:58 |
| noonedeadpunk | Installation from pypi whould be much faster I believe.... | 16:59 |
| noonedeadpunk | especially if we could cache things | 16:59 |
| spatel | yeppp | 16:59 |
| spatel | jrosser i did run scripts/bootstrap-ansible.sh which download /etc/ansible/role etc.. | 17:00 |
| spatel | But when i compare size of folder its different then original. Old deployment node has 91MB size of role folder and in new deployment has 63MB | 17:01 |
| spatel | does that indicating something wrong? | 17:01 |
| jrosser | i don't know | 17:11 |
| spatel | I think its just filesystem block size issue.. | 17:11 |
| jrosser | perhaps over time we deprecate some roles too | 17:11 |
| spatel | i have compared files side by side and didn't see any difference | 17:11 |
| spatel | now running playbook to see if its going to change anything on one of my compute node or not :) | 17:12 |
| noonedeadpunk | I think new shallow_since does reduce amount of content that's being fetched from git | 17:18 |
| spatel | what is the tags to distribute nova keys ? | 17:18 |
| noonedeadpunk | on which version? as on Y it's not needed anymore | 17:18 |
| spatel | Wallaby | 17:18 |
| noonedeadpunk | nova-key | 17:19 |
| spatel | cool but in newer version it should do itself correct | 17:20 |
| noonedeadpunk | yup | 17:21 |
| noonedeadpunk | it's ssh certs instead of rsa keys | 17:21 |
| noonedeadpunk | which are used only for offline migration.... | 17:22 |
| spatel | yes.. we don't have ceph so using block migration | 17:23 |
| spatel | is this correct command openstack-ansible os-nova-install.yml --tags nova-key --limit compute_hosts | 17:23 |
| *** dviroel|rover|afk is now known as dviroel|rover | 18:05 | |
| noonedeadpunk | looks fair | 18:53 |
| noonedeadpunk | spatel: there's proof it's good :D https://opendev.org/openstack/openstack-ansible/src/branch/stable/xena/scripts/add-compute.sh#L37 | 18:54 |
| spatel | Thank you!! so we have script to add compute nodes :) | 19:02 |
| spatel | i didn't know that | 19:02 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Do not provision neutron config when not needed https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/867087 | 19:10 |
| *** dviroel|rover is now known as dviroel|out | 21:20 | |
| *** tosky is now known as Guest1527 | 22:34 | |
| *** tosky_ is now known as tosky | 22:34 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!