| prometheanfire | coolcool | 00:02 |
|---|---|---|
| *** arxcruz is now known as arxcruz|ruck | 07:54 | |
| moha7 | jamesdenton: Hi; Why we use the type 'bridges' on the hosts for the OSA, but we use interfaces themselves during the manual installation of OpenStack? | 08:52 |
| moha7 | In document: "Install the kernel extra package if you have one for your kernel version" --> Shouldn't it be "... if you have NOT one ..."? (https://docs.openstack.org/project-deploy-guide/openstack-ansible/zed/targethosts.html+) | 09:12 |
| noonedeadpunk | moha7: well, I guess depends. I assume there were cases when there're no extra modules for the installed kernel | 09:25 |
| noonedeadpunk | And bridges/interfaces as we're in the process of updating documentations. On bare metal hosts bridges does not make much sense and were originally used to align naming through the docs | 09:26 |
| noonedeadpunk | And James pushed couple of commits to change that but it's big work | 09:27 |
| noonedeadpunk | and we all have bunch of other duties | 09:27 |
| moha7 | Ah, "_cases, when there're no extra modules_" -,> Then it's my misunderstanding, you're right. | 09:40 |
| moha7 | noonedeadpunk: So you mean I can bypass the step of setting bridges up in the Ubuntu Netplan configuration, either lxb bridges or ovs bridges, and it works by just giving the name of interfaces, as in ens160, ens192, etc directly in the openstack_user_config.yml file instead of br-mgmt and br-vxlan, right? | 09:46 |
| opendevreview | Andrew Bonney proposed openstack/ansible-role-zookeeper master: Add configuration option for native Prometheus exporter https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/870049 | 09:48 |
| noonedeadpunk | moha7: um, well, not always :D There's a bit more then that. If we're talkign about bare metal deployments (without LXC) - then yes, bridges are not needed. If we're talking about compute or net nodes - bridges are not needed there either. But they are needed for LXC hosts (your controllers/infra nodes) | 09:49 |
| noonedeadpunk | But even then br-vlan or br-vxlan can be just interfaces isntead of bridges | 09:50 |
| noonedeadpunk | Again - it's about what can be done, but docs with bridges work just fine | 09:50 |
| moha7 | noonedeadpunk: I think I get the point; but what do you mean by "bare metal deployments (without LXC)"? Do you mean other deployment solutions as in manual or Kolla? | 09:57 |
| moha7 | got* | 09:57 |
| noonedeadpunk | So in OSA you can either use LXC containers to separate services or deploy everything on node directly without any containers | 09:58 |
| noonedeadpunk | https://docs.openstack.org/openstack-ansible/latest/reference/inventory/configure-inventory.html#deploying-directly-on-hosts | 09:59 |
| moha7 | OMG! I didn't knew it | 10:06 |
| noonedeadpunk | unfortunatelly, we have quite a few things that are not obviously documented, as there's never time for docs :( | 10:08 |
| noonedeadpunk | but contributions are always very welcome | 10:10 |
| noonedeadpunk | As it's also hard to spot what's missing when dealing with things on daily basis, as you have blurred eye and hardly check docs as you know how to do things | 10:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Prevent mariadbcheck.socket to wait for network.target https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/870071 | 10:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Remove "warn" parameter from command module https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/869656 | 10:25 |
| noonedeadpunk | NeilHanlon: ah, I guess I found what you meant about rocky changes - https://review.opendev.org/q/topic:rocky-9-virtqemud | 10:51 |
| jrosser | noonedeadpunk: how far do you think we are from making 26.1.0? | 10:56 |
| jrosser | we have the things andrewbonney found this week and OVN / ironic docs | 10:57 |
| noonedeadpunk | Well, I was thinking to make at least 26.0.1 weeks ago... But now it was actually qustion I was going to ask you given found things | 10:57 |
| jrosser | well - we have completed an upgrade now with 26.0.0 + tons of patches | 10:57 |
| jrosser | but i think that if we merge everything that is proposed then our local patches pretty much all disappear | 10:58 |
| jrosser | andrewbonney: can confirm this ^ | 10:58 |
| andrewbonney | Yes, I think everything is merged to Zed now apart from https://review.opendev.org/c/openstack/openstack-ansible/+/869974 | 10:58 |
| jrosser | i had a change to uwsgi to bind to multiple interface s thats not backported | 10:58 |
| jrosser | but thats kind of feature rather than bugfix and it landed late | 10:59 |
| noonedeadpunk | I can recall also couple of bugfixes now on master only | 11:00 |
| jrosser | also https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/867547 | 11:01 |
| jrosser | and https://review.opendev.org/c/openstack/openstack-ansible/+/867577 | 11:01 |
| jrosser | oh was that the systemd service loop? | 11:01 |
| jrosser | *dependancy loop | 11:01 |
| noonedeadpunk | yeah | 11:01 |
| noonedeadpunk | both for mount and mariadb | 11:01 |
| noonedeadpunk | well docs do not block releases as they're not tighten to them | 11:02 |
| jrosser | anyway i think we are close | 11:02 |
| noonedeadpunk | I also feel nerveus about https://review.opendev.org/q/topic:rocky-9-virtqemud as that sounds like huge PITA | 11:05 |
| noonedeadpunk | or well, https://fedoraproject.org/wiki/Changes/LibvirtModularDaemons | 11:06 |
| jrosser | tbh I am not sure why the jobs still pass | 11:10 |
| noonedeadpunk | yeah. eventually I think we're kind of ready | 11:10 |
| jrosser | particularly if there are conflicting packages somehow | 11:10 |
| noonedeadpunk | well, maybe there're some meta stuff for transition | 11:10 |
| noonedeadpunk | But I would expect that kind of migration happening only with new major libvirt version | 11:11 |
| noonedeadpunk | Which I wouldn't expect to see at least on Rocky... | 11:11 |
| noonedeadpunk | well, it might be coming from RDO as well | 11:11 |
| jrosser | I was unsure of the patches too | 11:11 |
| jrosser | like not converting daemon name to list on all OS | 11:12 |
| jrosser | but I only looked at them super quickly | 11:12 |
| noonedeadpunk | yeah, how that passed I have no idea | 11:12 |
| noonedeadpunk | as iterating over string should have fail dramatically | 11:12 |
| noonedeadpunk | unless new ansible does some check nowadays for string/list (which would be quite weird) | 11:15 |
| noonedeadpunk | (not weirder then re-implementing package resolver for apt module though) | 11:16 |
| noonedeadpunk | Btw I have that thingy that I'd love to discuss: https://review.opendev.org/c/openstack/openstack-ansible/+/869762 Maybe worth saving it for meeting... But I'm going to rely on smth like that in a new deployment as it's a nightmare otherwise | 11:17 |
| amarao | If I want to test a newer version of role, what is the easiest way to do so? Is editing ansible-role-requirements.yml enough? And when should I patch it, before running bootstrap script or after? | 12:07 |
| noonedeadpunk | amarao: easiest way to create /etc/openstack_deploy/user_role_requirements.yml file that will contain override of the role | 12:10 |
| noonedeadpunk | and after that run bootstrap-ansible.sh | 12:10 |
| noonedeadpunk | or you can go to /etc/ansible/roles/role_name and just `git pull; git checkout` - but that's if only testing and better not doing like that | 12:11 |
| amarao | Oh, thanks. I didn't know about user_role_requirements.yml. | 12:12 |
| noonedeadpunk | there're couple of bugs related to that that were patched jsut now - like if you override literally every role there was an issue. And it was a bit trickier with collection-requirements until patch where we have names for all colelctions in the list | 12:14 |
| amarao | Is user_role_requirements.yml have the format as ansible-role-requirements.yml? | 12:19 |
| opendevreview | Merged openstack/ansible-role-systemd_mount stable/zed: Fix mount's systemd unit dependency logic https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/869937 | 12:26 |
| jrosser | amarao: https://opendev.org/openstack/openstack-ansible/src/branch/master/doc/source/reference/configuration/extending-osa.rst#maintaining-local-forks-of-ansible-roles | 12:30 |
| noonedeadpunk | amarao: sorry, indeed it's user-role-requirements.yml. and yes, same format | 12:34 |
| amarao | It worked, thank you! My last question, and I feel silly about it, if I want to test changes from review (before merge), where can I get git remote? F.e. https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/870071/1. I really git around and I can't find git url for proposed changes to clone... | 12:53 |
| jrosser | amarao: press the "three spots" top right | 12:57 |
| jrosser | choose "download patch" | 12:58 |
| jrosser | then i usually cherry-pick onto my local repo | 12:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-pki master: Create relative links to root instead of absolute https://review.opendev.org/c/openstack/ansible-role-pki/+/870089 | 12:58 |
| amarao | Huh... I hoped for git URL to put into user-role-requirements.yml... There isn't git accessible repo with pre-merge changes, is it? | 13:00 |
| jrosser | well it's a patch not really merged to a repo | 13:00 |
| jrosser | so clone doesnt really make much sense | 13:00 |
| amarao | So I need to apply patch after running ansible bootstrap..., ok, I got it. | 13:01 |
| amarao | (I do not have local repos, as I use separate 'deployment server' for whole ansible thing, and I manage it with my in-house playbook). | 13:01 |
| amarao | Оh, no, it is! Three dots, download patch, and there is checkout url: https://review.opendev.org/openstack/openstack-ansible-galera_server refs/changes/71/870071/1 | 13:04 |
| jrosser | I feel that’s slightly ambiguous about exactly what point if the tree that refers to | 13:07 |
| noonedeadpunk | Yeah, the problem is that our script at the moment is not capable of dealing with refs | 13:09 |
| noonedeadpunk | it's likely possible to fix, but time is always a problem | 13:10 |
| noonedeadpunk | (at least for our parallel git clone script) | 13:10 |
| noonedeadpunk | damn, I'm so fed up dealing with our dynamic_inventory script... No fun debugging it at all... | 13:14 |
| noonedeadpunk | Also I'm afraid of consequences once we fix https://bugs.launchpad.net/openstack-ansible/+bug/2002645/comments/6 | 13:16 |
| noonedeadpunk | as now behaviour is quite unpredictable I'd say | 13:16 |
| amarao | We've give up on dynamic inventory at all and just put our own inventory instead without invoking dynamic inventory at all. It's much more verbose, but at least, very debuggable. | 13:22 |
| noonedeadpunk | well, in case of bare metla deployments that does make sense indeed. | 13:22 |
| noonedeadpunk | For lxc - not much | 13:22 |
| jrosser | amarao: for if you're doing a lab or just tests then yes you can cherry-pick after bootstrap | 13:23 |
| jrosser | if you want to make that more deterministic for production deployment then i'd recommend forking the repo on github and making your own branch to keep outstanding patches, then refer to that in ansible-role-requirements.yml | 13:24 |
| noonedeadpunk | I find dynamic inventory handy and generally working until it;s not :D | 13:24 |
| amarao | It's slightly more convoluted. I have set of playbooks to reinstall BM, clone, bootstrap, configure, run openstack-ansible playbooks, etc. It's not in production yet,but it's all 'auto'. So I now trying to squeeze patch mode into existing playbooks... | 13:24 |
| amarao | (I also run remote ansible via local ansible and dump logs after that... crazy thing). | 13:24 |
| noonedeadpunk | any reason to not use ara? | 13:25 |
| amarao | (and pinning all deps, removing internet from install time are my next targets) | 13:25 |
| jrosser | well if you keep your /etc/openstack_deploy in a git repo then you'll have user-role-requirements.yml under version control, referring to your forked repo | 13:25 |
| noonedeadpunk | As osa capable of installing ara, not sure if we've ever implemented configuration to map it with remote server, but should be easy to do | 13:25 |
| amarao | I plan to remove intermediate ansible at the end and just have common inventory between osa playbooks and my playbooks. The main "odd" thing we have is that inventory will be generated by our own orchestration code (the one to configure BM servers, switches, etc). So 'ansible-in-ansible' is temporary. Also, it's surprisingly usable: https://paste.openstack.org/show/bRWWON76mcgAbGPtI0Ug/ (note the Logs task at the end with /dev | 13:28 |
| amarao | /tty hack). | 13:28 |
| jrosser | amarao: the whole reason that user-role-requirements is there is to address your issue of needing to manage unmerged patches or ones from master | 13:29 |
| jrosser | we tried, but it is too difficult to automate pulling in a list of patches from gerrit when building the deployment server | 13:29 |
| jrosser | it is much easier to maintain forks of the relevant repos on github or a mirror and refer to those as overrides | 13:30 |
| amarao | I still hadn't found the way to convert pre-merged ref to compatible format. It fails with " No commits selected for shallow request" but I work on it. | 13:30 |
| jrosser | imho it is better to make a fork | 13:30 |
| amarao | I want to make it usable for future cases. | 13:31 |
| jrosser | maybe i'm not explaining properly | 13:31 |
| jrosser | here is what we do to carry an outstanding patch to the uwsgi role for example https://github.com/bbc/ansible-role-uwsgi/tree/bbc-zed-26.0.0 | 13:32 |
| jrosser | then just refer to that in user-role-requirements | 13:33 |
| amarao | Ok, I give up. git clone can't clone a specific refspec, which is unexpected to me. I'll go your way, thanks. | 13:56 |
| jamesdenton | moha7 the use of bridges, like br-vxlan, br-vlan, br-mgmt, etc. are related to the use of LXC containers running the actual services. Those bridges allow network communication through the host, since they are bridging the physical interface to the container(s). Some of the bridges, like br-vlan, were used when Neutron agents lived in LXC containers, too, but are no longer necessary since the agents live on metal now. If | 14:12 |
| jamesdenton | you are not using LXC, then the bridges aren't technically as important, but the installation still assumes they are used. It also provides some consistency between hosts, especially when physical interface names can vary (unless you rename them) | 14:12 |
| moha7 | Is it matter the type of bridges, lxb bridges or ovs bridges? | 14:14 |
| jamesdenton | both should be supported, but the default is lxb | 14:15 |
| moha7 | jamesdenton: If I run the OSA in the state without LXC, then considering that "the installation still assumes [those bridges] are used", then should I build the bridges or not? | 14:18 |
| jamesdenton | I run on "metal", which is non-LXC, and I use br-mgmt and br-overlay (renamed from br-vxlan) | 14:19 |
| jamesdenton | the management IP of my host is configured on br-mgmt | 14:19 |
| jamesdenton | and br-overlay has the TEP (VTEP) address for Geneve/VXLAN | 14:19 |
| jamesdenton | happy to share my netplan config | 14:19 |
| moha7 | Is there any benchmark, or experience, indicating which method is better, seperating services with LXC containers or running in bare metal mode without LXC all stuff on the nodes? | 14:21 |
| moha7 | jamesdenton: > happy to share my netplan config | 14:22 |
| moha7 | I am very much looking forward to it ((: | 14:22 |
| noonedeadpunk | The only tricky part regarding metal are day2 operations | 14:22 |
| jamesdenton | Well, operationally, it probably makes more sense to segment the services via containers, and many here are doing just that. There is a speed penalty during deployment/upgrades, since you have to touch the containers vs metal | 14:22 |
| noonedeadpunk | Especially when different services rely on same distro-provided binaries | 14:23 |
| noonedeadpunk | Ceph is quite good example of how things might go wrong if both glance and cinder use rbd as a backend | 14:23 |
| jamesdenton | moha7 booting up my infra VMs now; migrated to a new hypervisor | 14:24 |
| * NeilHanlon 's lab is in complete disarray | 14:24 | |
| noonedeadpunk | as then when upgrading glance you might also upgrade ceph, which is used also by cinder, which might not be ready for such upgrade | 14:24 |
| jamesdenton | i have managed to keep mine relatively functional | 14:24 |
| noonedeadpunk | Well, in case you're storing images in swift - that's not an issue. Or have cinder-volumes elsewhere | 14:25 |
| noonedeadpunk | so depends | 14:25 |
| jrosser | also the lxc is only the controllers | 14:25 |
| noonedeadpunk | I would say metal is also quite popular. So might be matter of personal preferences as well | 14:26 |
| jrosser | when you have lots of computes the potential overhead of a containerised control plane is not really large | 14:26 |
| NeilHanlon | "it depends", unfortunately | 14:26 |
| jamesdenton | jrosser spot on | 14:26 |
| jrosser | but if you want to rinse-repeat a multinode lab lots of times, then it will certainly be quicker with a metal deploy | 14:27 |
| jamesdenton | i have forgotton the caveats of lxc and 18.04->20.04 or 20.04->22.04, anything to worry about there? | 14:27 |
| jrosser | i think that now we build the base images from scratch rather than all that nonsense with downloading one, it seems pretty much a non event | 14:28 |
| jamesdenton | moha7 https://paste.opendev.org/show/b4rV83uoshha3RTxRT75/ | 14:28 |
| jrosser | i guess the thing to note there with jamesdenton network config for a metal deploy is that its pretty identical to what you'd have for an lxc one | 14:29 |
| jamesdenton | yeah, actually, this is a hybrid | 14:29 |
| jamesdenton | slow migration from LXC -> metal; my rabbitmq and galera services are still containerized | 14:29 |
| jamesdenton | so, i get the "best of both worlds" when things break :D | 14:30 |
| noonedeadpunk | jamesdenton: yeah, well, we have not really working cross-OS support due to lsyncd and repo containers, that were fixed only for Yoga | 14:30 |
| noonedeadpunk | So quite a mess with disabling proper repo backend on a haproxy and stopping lsyncd | 14:30 |
| jamesdenton | ahh ok | 14:30 |
| moha7 | jamesdenton: Thanks; Your vlan20 has mtu 1500, while others are all set on 9000; Is there something special with br-mgmt having of MTU1500? | 14:30 |
| noonedeadpunk | But I'd say we have quite fair docs of the process | 14:30 |
| jamesdenton | moha7 yes, vlan20 is my management interface to a default gateway (Cisco Firepower). Want to keep 1500 MTU there to avoid issues out to the worls | 14:31 |
| jamesdenton | *world | 14:31 |
| noonedeadpunk | moha7: I don't think it's being special, rather then no need in jumbo frames there | 14:31 |
| jamesdenton | but vlan21 is Geneve/VXLAN and vlan22 is storage, so 9000 there | 14:31 |
| noonedeadpunk | Since there's no data flowing and mostly small packets there | 14:31 |
| * NeilHanlon twitches at the words Cisco Firepower | 14:31 | |
| noonedeadpunk | :D | 14:31 |
| jamesdenton | noonedeadpunk do you recall the issue with /var/run dirs not being created? Do we have a bug for that? | 14:32 |
| jamesdenton | NeilHanlon you will be happy to know it runs ASA code. | 14:32 |
| noonedeadpunk | NeilHanlon: so, any idea about switching to this https://fedoraproject.org/wiki/Changes/LibvirtModularDaemons on R9? | 14:32 |
| NeilHanlon | jamesdenton: thanks I hate it :P | 14:32 |
| jamesdenton | me too. me too. | 14:33 |
| NeilHanlon | noonedeadpunk: yeah i've been doing some digging. it's definitely switched to for RHEL 9, but i'm not sure why we're only seeing this pop up now | 14:33 |
| NeilHanlon | https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/considerations_in_adopting_rhel_9/assembly_virtualization_considerations-in-adopting-rhel-9 | 14:33 |
| noonedeadpunk | I'm jsut a bit surprised that it's done with same libvirt version. In case it's the same | 14:33 |
| NeilHanlon | as far as I can tell, the RHEL/Rocky libvirt package has shipped these modular daemons since the beginning | 14:34 |
| noonedeadpunk | aha | 14:34 |
| noonedeadpunk | but old way works somehow I assume? | 14:34 |
| NeilHanlon | The guidance from RHEL suggests that it is indeed possible to use the old, monolithic daemon(s). | 14:35 |
| NeilHanlon | https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/configuring_and_managing_virtualization/optimizing-virtual-machine-performance-in-rhel_configuring-and-managing-virtualization#proc_enabling-modular-libvirt-daemons_assembly_optimizing-libvirt-daemons | 14:35 |
| noonedeadpunk | As I'm mostly conserned about migration of tcp/tls sockers | 14:36 |
| NeilHanlon | IMO, the path of least astonishment for users would be to keep using the monolithic daemon until other OSes switch (?) | 14:36 |
| NeilHanlon | though i'm worried now we'll need to account for both scenarios going forward for current users | 14:36 |
| opendevreview | Merged openstack/ansible-role-systemd_networkd stable/xena: Fix static routes to use Destination rather than Source key https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/869837 | 14:38 |
| noonedeadpunk | Yeah, that's my main concerns as well actually. As using different approaches can be quite messy. Not saying that a pre-requirement is to stop all VMs | 14:38 |
| noonedeadpunk | And that's kind of a deal breaker here | 14:39 |
| jrosser | i guess what we don't know is the provenance of the install that prometheanfire had, like when/how it was done | 14:39 |
| jrosser | vs. what happens if you start from fresh today | 14:40 |
| jrosser | theres quite a bunch of it here https://codesearch.opendev.org/?q=virtqemud | 14:42 |
| moha7 | noonedeadpunk: "day2 operations"? Is it an idiom? | 14:43 |
| jrosser | moha7: well once you have deployed (day 1) you then are on to day 2 and everything beyond | 14:44 |
| noonedeadpunk | yah, puppet has merged that year ago... | 14:44 |
| jrosser | when you need to monitor / maintain / upgrade | 14:44 |
| noonedeadpunk | So sounds like high time for us as well.... | 14:45 |
| jrosser | and hopefully do so without any of your users noticing you do that | 14:45 |
| *** tosky_ is now known as tosky | 14:47 | |
| noonedeadpunk | Hm, I don't see neither debian bullseye nor focal having virtproxyd/virtqemud | 14:47 |
| noonedeadpunk | don't have jammy handy | 14:48 |
| NeilHanlon | i'm kinda baffled this change was shoehorned into RHEL 9 like this | 14:49 |
| NeilHanlon | i wouldn't have expected it until RHEL 10 | 14:49 |
| jrosser | jammy has a man page https://packages.ubuntu.com/search?suite=jammy&arch=any&mode=filename&searchon=contents&keywords=virtproxyd | 14:50 |
| moha7 | jrosser: day2, cool | 14:50 |
| noonedeadpunk | which is mostly puppet code ? o_O | 14:53 |
| NeilHanlon | in the future all systems will be configured by just writing ruby | 14:53 |
| noonedeadpunk | or python like charms do | 14:54 |
| noonedeadpunk | and nothing in kinetic/lunar either. | 14:55 |
| noonedeadpunk | So sounds like it's a rhel thingy for the timebeing in which time should be invested to find out migration plan and timeline for it | 14:55 |
| noonedeadpunk | or keep monolithic until it's possible | 14:56 |
| noonedeadpunk | sounds like discussion for PTG actually | 14:56 |
| NeilHanlon | i'm going to spend some time this afternoon if I can trying to recreate prometheanfire's breakage, possibly upgrading from 9.0 to see if that has something to do with it | 14:57 |
| NeilHanlon | but agreed probably a good discussion for PTG | 14:58 |
| noonedeadpunk | tbh, I'd assume that our CI would catch some obvious breakages... | 15:01 |
| noonedeadpunk | but maybe it's caveats for multinode, like live migrations or smth | 15:01 |
| NeilHanlon | prometheanfire: if you can also let us know a bit more about the provenance of the system as jrosser said. How did you do the initial deploy? When? What OS version was installed initially? etc | 15:02 |
| damnthem | Hi, guys. Is it possible to create a separate Ceph cluster for each Cinder AZ, using OSA-integrated ceph-ansible playbook? | 15:14 |
| noonedeadpunk | damnthem: well. partially at least :) I managed to do that, but there were some non-trivial failures in ceph-ansible down the road which I had to shortcut through, as that was POC deployment | 15:20 |
| noonedeadpunk | I can't really recall details, but I think the issue was when you're trying to run ceph-ansible across all AZs at once | 15:21 |
| noonedeadpunk | When running with limits it was working fine | 15:21 |
| noonedeadpunk | Also I'd suggest using cluster_names | 15:21 |
| noonedeadpunk | Wait a sec, I still might have config for AZs somewhere | 15:22 |
| damnthem | Yes, actually also thought about that (cluster names). And judjing by the structure of ceph ansible playbook i got couple ideas about why u have problems with running installation across all AZs) | 15:24 |
| noonedeadpunk | damnthem: these were user_variables regarding ceph/cinder https://paste.openstack.org/show/bBIcVS2faINLR1Erh5pX/ | 15:24 |
| damnthem | That's would be helpful for sure., thank you | 15:25 |
| * NeilHanlon saves | 15:25 | |
| noonedeadpunk | these were openstack_user_config https://paste.openstack.org/show/bgPT2VqXSpoe4jeCyGpB/ | 15:26 |
| noonedeadpunk | And I also created this env.d/ceph.yml as I wanted to host mons/mgrs in LXC containers: https://paste.openstack.org/show/b6JiwHSUsI11365DeHtA/ | 15:28 |
| noonedeadpunk | As I said - that was sandbox I've played with, so not everything 100% accurate but should give some idea | 15:28 |
| noonedeadpunk | Ah, and I've also used this patch for inventory to generate handy AZ groups https://review.opendev.org/c/openstack/openstack-ansible/+/869762 | 15:29 |
| noonedeadpunk | There're quite some nuances down the road, but overall quite doable, yes | 15:29 |
| noonedeadpunk | likely I forgot to paste env.d/cinder.yml for cinder_volumes... here it goes https://paste.openstack.org/show/bDRoc5EqmAI4IE5nWHWi/ | 15:31 |
| noonedeadpunk | I wanted to write proper docs about AZ path, but lack time and still in progress of deploying it for real | 15:35 |
| damnthem | Thanks again. Will look into that | 15:44 |
| prometheanfire | noonedeadpunk: pre-req to stop VMs, that's harsh | 15:45 |
| prometheanfire | NeilHanlon: I didn't do the initial deploy, started with rocky-9, my deploy has ceph as well | 15:45 |
| prometheanfire | the error I was seeing was trying to run virsh secret-create (iirc) from the ceph_client osa role | 15:46 |
| mgariepy | jamesdenton, around ? | 15:48 |
| prometheanfire | that seems to trigger a systemd socket which then starts virtsecretd, virtsecretd has a conflicts= line on libvirtd, so kills libvirtd when it comes up | 15:48 |
| NeilHanlon | i think we should mask the modular daemons | 15:50 |
| prometheanfire | the playbook starts libvirtd, which kills the virtsecretd socket/service. it then runs the virsh secret-create, which either fails because of the missing socket/service or starts the virtsecretd socket/service; either way the secret-create fails | 15:50 |
| prometheanfire | changing to the modular daemons requires vm restarts? | 15:51 |
| MrR | Hi all, speaking of haproxy, I'm hoping someone can give me some guidance on it and letsencrypt, i'm having trouble pulling a certificate from LE which i think just be my own mistake but help would be appreciated. | 16:01 |
| MrR | I have a port forward pointing to my external vip ip (have and can try others again) and port 8888, the vip/domain is set in my router and i can ping it (and all nodes/related ips) from the entire network. | 16:01 |
| MrR | I can see when running setup-infrastructure it is trying to pull the cert but fails, it always runs on my second node which has an ip of 236.55 and i can see in the journal that it does in fact come up (i've tried a few ips such as for the nodes on the flat and mgmt network so i could be mistaken that it worked on the external vip ip but it definatly came up for at least one of the ips) but still fails. | 16:01 |
| MrR | If i try manually pulling a cert via certbot i can do so by a record on my domain with a forward from the router to the device/domain pulling the cert (this is on a none openstack machine in case it matters), so in my troubleshooting i used nc -l as a listener on a node, used an online port checker and it states closed, trying again with the non openstack machine, port reports opened. I'm guestimating i'm | 16:01 |
| MrR | getting confused somewhere along the way as to what ips i should be pointing these to. | 16:01 |
| MrR | So, i have the following set: | 16:01 |
| MrR | haproxy_keepalived_external_vip_cidr: "192.168.1.62/24" | 16:01 |
| MrR | haproxy_keepalived_internal_vip_cidr: "{{internal_lb_vip_address}}/22" | 16:01 |
| MrR | haproxy_keepalived_external_interface: br-flat | 16:01 |
| MrR | haproxy_keepalived_internal_interface: br-mgmt | 16:01 |
| MrR | haproxy_ssl_letsencrypt_enable: True | 16:01 |
| MrR | haproxy_ssl_letsencrypt_install_method: "distro" | 16:01 |
| MrR | haproxy_ssl_letsencrypt_email: "" | 16:01 |
| MrR | haproxy_interval: 2000 | 16:01 |
| MrR | openstack_service_adminuri_proto: https | 16:01 |
| MrR | openstack_service_internaluri_proto: https | 16:01 |
| MrR | haproxy_ssl: True | 16:01 |
| MrR | haproxy_ssl_all_vips: true | 16:01 |
| MrR | I also have internal_vip set as 192.168.1.61, external_vip set as openstack.domain.com which as i said is resolvable via my router accross the entire network. | 16:01 |
| MrR | The nodes have the relevant networks including flat which each node has an ip on thjem (in case of node 2 192.168.1.55 for flat). I have tried setting haproxy_ssl_letsencrypt_certbot_bind_address to multiple different ips (node, external_vip, internal_vip, mgmt, flat etc with the relevant forwards etc) to still have it fail. What am i missing? | 16:02 |
| MrR | I can run/test anything required if help can be offered in figuring this out. Any help would be welcomed | 16:02 |
| MrR | sorry, i did not know it would paste like that, new to irc | 16:02 |
| noonedeadpunk | oh, so we basically don't test ceph scenario for rhel | 16:12 |
| noonedeadpunk | that's good imput on how to reproduce prometheanfire | 16:13 |
| noonedeadpunk | NeilHanlon: masking daemons is easy peasy :D | 16:13 |
| noonedeadpunk | easy way to reproduce should be `git clone https://opendev.org/openstack/openstack-ansible; cd openstack-ansible; ./scripts/gate-check-commit.sh aio_metal_ceph` | 16:14 |
| noonedeadpunk | prometheanfire: well, that's what said in rhel docs: `Your virtual machines are shut down.` in Prerequisites of Enabling modular libvirt daemons section here https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/configuring_and_managing_virtualization/optimizing-virtual-machine-performance-in-rhel_configuring-and-managing-virtualization#proc_enabling-modular-libvirt-daemons_assembly_optimizing-libvirt-daemons | 16:15 |
| noonedeadpunk | MrR: for pasting we use https://paste.openstack.org/ in general | 16:16 |
| MrR | ahhh, thanks for that, I'll use that in future | 16:16 |
| prometheanfire | noonedeadpunk: heh, fun migration then | 16:17 |
| noonedeadpunk | Well, I haven't tested that, jsut saw that in docs jsut now to which Neil has pointed me | 16:17 |
| noonedeadpunk | damn, when I will stop making typos in word `just` | 16:19 |
| mgariepy | when you take the time to setup auto-correct for it :P | 16:20 |
| noonedeadpunk | MrR: I assume issue might be in port forwarding? As for let's encrypt we're doing some tricks and it's not really port 8888 IIRC | 16:20 |
| prometheanfire | I switched back and forth on a test system with running VMs, I'm sure it's not supported but it's 'fine' | 16:20 |
| noonedeadpunk | prometheanfire: well, anyway I think these patches do need a bit more love. As they're not covering conversion of tcp/tls sockets at very least | 16:21 |
| prometheanfire | yep, I'm waiting for direction from NeilHanlon first (mask or not) | 16:22 |
| noonedeadpunk | MrR: we use 8888 just for the backend, and then access it through haproxy that listens on 443 I guess | 16:22 |
| noonedeadpunk | I'm quite burried with stuff right now, and likely next week as well :( | 16:23 |
| jrosser | MrR: port 80 | 16:23 |
| jrosser | you need 80 and 443 to work externally for LE to work | 16:24 |
| MrR | <noonedeadpunk, ok i'll try forwarding 443 also. Should i say set the haproxy_ssl_letsencrypt_certbot_bind_address to the node ip on the flat network or should the vip address work or should i just leave it out? | 16:25 |
| jrosser | dont change haproxy_ssl_letsencrypt_certbot_bind_address | 16:25 |
| MrR | jrosser, i had tried 443 also but i may not have matched ips, i'll fire it up and give it a go now | 16:26 |
| noonedeadpunk | MrR: and eventually these overrides: https://docs.openstack.org/openstack-ansible/latest/user/security/index.html#certbot-certificates | 16:26 |
| jrosser | MrR: the important thing to know is that the LE servers will make a request to your virtual IP | 16:26 |
| jrosser | MrR: and haproxy directs that to whichever of your infra nodes is making the certbot request | 16:26 |
| MrR | jrossr, so leave haproxy_ssl_letsencrypt_certbot_bind_address out, forward 80 and 443 to the external_vip which is attached to the domain and give it a go | 16:28 |
| jrosser | that would be a good start | 16:28 |
| jrosser | certbot needs to bind to a local IP on the infra node | 16:28 |
| jrosser | haproxy then uses that as a backend | 16:28 |
| jrosser | this is complex becasue there are N infra nodes behind a loadbalancer | 16:29 |
| MrR | so the flat ip should work if i'm understanding that right, only 3 nodes right now, all running haproxy | 16:29 |
| jrosser | and trickery is required to make sure that the request from the LE servers to collect the token from certbot goes to the correct infra service | 16:30 |
| jrosser | well, put simply you make a request for a certificate for example.com | 16:30 |
| jrosser | whatever IP example.com resolves to must serve the token back at the well-known url | 16:30 |
| jrosser | MrR: i don't really know what you mean by the flat IP | 16:32 |
| MrR | sorry, i mean an ip on the br-flat network | 16:33 |
| jrosser | thats kind of not important | 16:33 |
| jrosser | external_lb_vip_address must be set to the FDQN that resolves to your haproxy external IP | 16:33 |
| jrosser | then whatever port forward you have must allow 80/443 for that IP | 16:35 |
| opendevreview | Merged openstack/ansible-role-systemd_mount stable/yoga: Fix mount's systemd unit dependency logic https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/869938 | 16:38 |
| jrosser | MrR: you are also doing NAT? | 16:38 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Prevent mariadbcheck.socket to wait for network.target https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/870071 | 16:48 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server stable/zed: Prevent mariadbcheck.socket to wait for network.target https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/870056 | 16:48 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server stable/yoga: Prevent mariadbcheck.socket to wait for network.target https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/870057 | 16:48 |
| jrosser | MrR: "in my troubleshooting i used nc -l as a listener on a node" | 16:49 |
| jrosser | the backend to haproxy is expected to be http https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy/haproxy.yml#L290 | 16:51 |
| jrosser | so haproxy will not respond to just a port being opened | 16:51 |
| jrosser | see how we "prime" the loadbalancer to ensure that it directs traffic with the correct infra node before running certbot with a fake http server https://github.com/openstack/openstack-ansible-haproxy_server/blob/master/templates/letsencrypt_pre_hook_certbot_distro.j2#L4 | 16:52 |
| MrR | yes, using NAT, sorry was just firing them up and changing a few things to run it again | 16:54 |
| jrosser | if you try that python command on an infra node you should be able to see the haproxy status change | 16:55 |
| jrosser | and then your external port check should work, or even wget the fake server | 16:55 |
| jrosser | noonedeadpunk: ^ i thought we should look at what directory the temporary LE web server actually serves.... | 16:57 |
| MrR | already started setup-infra so if it fails i'll run that and report back, usually fails ssl in about 15 mins after starting | 17:08 |
| MrR | ran with -vvvv so i can share the output if it helps | 17:09 |
| jrosser | you ca run the haproxy playbook on its own | 17:10 |
| prometheanfire | for self managed ceph that we want to use for object storage (swift), I imagine we can't use the built in ceph-rgw stuff and have to create the tenant / endpoint / etc manually? | 17:10 |
| jrosser | look inside setup-infra.yml, it just calls a string of other playbooks | 17:10 |
| jrosser | prometheanfire: I think that the rgw setup stuff is in its own playbook to enable that | 17:11 |
| prometheanfire | ya, ceph-rgw-keystone-setup is a thing | 17:12 |
| prometheanfire | rgw would need to be configured manually on the ceph side since osa doesn't touch that still, but could work | 17:12 |
| jrosser | so given the right set of vars it should set those things up | 17:12 |
| prometheanfire | ya, figuring out those vars now :P | 17:12 |
| prometheanfire | I ceph_rgws is not going to be defined so need to short circuit some stuff (the ceph-rgw-install playbook entirely), looks like the keystone playbook doesn't block based on ceph_rgws at least | 17:15 |
| jrosser | prometheanfire: pay careful attention to https://github.com/openstack/openstack-ansible/blob/master/playbooks/ceph-rgw-install.yml#L17 | 17:17 |
| prometheanfire | yep, which is why I have to call it directly | 17:18 |
| jrosser | that is equivalent to (rgw deployed by OSA) or (list of external rgw) | 17:18 |
| jrosser | you shouldnt have to | 17:18 |
| jrosser | if this is an external ceph cluster then defining a list of ceph_rgws just like you supply a list of external mons should be whats needed | 17:19 |
| jrosser | that list being length > 0 will make the service/endpoint setup stuff happen | 17:19 |
| prometheanfire | if ceph_rgws is set then osa will install ceph-rgw on those hosts (the ceph-server) | 17:20 |
| jrosser | no | 17:20 |
| jrosser | well yes | 17:20 |
| jrosser | but ceph-rgw != ceph_rgw | 17:20 |
| jrosser | small but important difference in group name vs. var name there | 17:21 |
| jrosser | define and populate the group ceph-rgw and OSA will deploy | 17:21 |
| jrosser | define the list ceph_rgw of some external hosts and OSA will not deploy, but make reference to in haproxy setup | 17:21 |
| prometheanfire | ok, I'll try with ceph_rgws set to the mons (where I have them deployed) | 17:22 |
| prometheanfire | cool | 17:22 |
| jrosser | here https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/ceph.yml#L43-L48 | 17:22 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Fix scheduler track_instance_changes option https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/870023 | 17:24 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_nova stable/zed: Fix scheduler track_instance_changes option https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/870058 | 17:25 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_nova stable/yoga: Fix scheduler track_instance_changes option https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/870059 | 17:25 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_nova stable/xena: Fix scheduler track_instance_changes option https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/870060 | 17:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add container_ip option for metal hosts https://review.opendev.org/c/openstack/openstack-ansible/+/870113 | 17:28 |
| noonedeadpunk | jrosser: andrewbonney you might be interested in this one ^ | 17:28 |
| noonedeadpunk | I assume it should be safe for existing environments. If apply container_ip to o_u_c container_address will be replaced with valid data. And according to test that will be reversed | 17:29 |
| jrosser | i think it will be interesting to look why so far we did not need that | 17:30 |
| jrosser | noonedeadpunk: did you have some specific thing leading to that - interesting to understand what you've used it for | 17:31 |
| noonedeadpunk | you're not running on metal and applied bunch of overrides that are still needed until we replace ansible_host with management_address | 17:31 |
| noonedeadpunk | well, eventually it's andrewbonney that lead me to look into this direction. As replacing ansible_host with container_address should have worked here https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/galera_all.yml#L46 in my opinion | 17:32 |
| noonedeadpunk | but it didn't as ips were wrong | 17:33 |
| jrosser | ah right ok | 17:33 |
| noonedeadpunk | We don't have environment with ssh!=mgmt yet - jsut deploying it, but I kind of unsure about what happens for my_ip on neutron and nova | 17:34 |
| jrosser | oh yes i think we did initially have live migration on the ssh interface | 17:34 |
| noonedeadpunk | As that will resolve to mess https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/common-playbooks/nova.yml#L157 for example | 17:34 |
| jrosser | i do not remember why or what we did, can look at that next week but we had to change something to ensure it was on mgmt network | 17:35 |
| noonedeadpunk | I believe change nova_management_address | 17:35 |
| noonedeadpunk | But I can imagine plenty of things that can go wrong | 17:35 |
| noonedeadpunk | And even worse for bare metal scenario | 17:35 |
| noonedeadpunk | Where services will likely listen on SSH network | 17:36 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Allow to create OVS bridge for lxcbr0 https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/868603 | 17:36 |
| jrosser | we should be generally removing ansible_host then from the configs | 17:37 |
| noonedeadpunk | as https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/all.yml#L35 | 17:37 |
| noonedeadpunk | I wonder why this was never raised tbh... Though we never said it's possible to split SSH from MGTM | 17:38 |
| jrosser | hmm, well we did it from that start | 17:38 |
| noonedeadpunk | yeah. well. If take galera example, I'm not sure that ansible_host is incorrect here https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/galera_all.yml#L36 | 17:38 |
| jrosser | but totally possible we have a bunch of overrides | 17:39 |
| noonedeadpunk | but well, on lxc it's not _that_ painful | 17:39 |
| noonedeadpunk | so regarding galera - I can assume that monitoring in general will be performed through SSH net | 17:40 |
| noonedeadpunk | But I'm not sure, as I'm not there yet :D | 17:40 |
| jrosser | right, in our case the SSH net has all the nodes and a pair of bastions | 17:40 |
| jrosser | so is like a DMZ | 17:40 |
| jrosser | and also on the bastions are elasticsearch / prometheus gateways to gather monitoring | 17:41 |
| noonedeadpunk | yeah, exactly, so monitoring worth to be allowed from ansible_host | 17:41 |
| noonedeadpunk | I was thinking about same architecture | 17:41 |
| opendevreview | Merged openstack/ansible-role-pki master: Create relative links to root instead of absolute https://review.opendev.org/c/openstack/ansible-role-pki/+/870089 | 17:41 |
| opendevreview | Merged openstack/ansible-role-pki master: Use loop labels to suppress output https://review.opendev.org/c/openstack/ansible-role-pki/+/870015 | 17:41 |
| jrosser | well it's worked nicely | 17:41 |
| jrosser | gives a good structure as well when considering a compromised node | 17:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-pki stable/zed: Create relative links to root instead of absolute https://review.opendev.org/c/openstack/ansible-role-pki/+/870065 | 17:42 |
| jrosser | and connections always going from more trusted -> less trusted zones for SSH and monitoring | 17:42 |
| noonedeadpunk | yep, right | 17:44 |
| jrosser | i mean if you really go to town then you can do private vlan on the switches too | 17:45 |
| jrosser | as there is no legitimate traffic except basion<>nodes, nothing nodes<>nodes | 17:45 |
| noonedeadpunk | um, any migrations go nodes<>nodes? or? | 17:46 |
| jrosser | for us thats on br-mgmt | 17:46 |
| jrosser | we made the ssh network not particularly resilient, like no bonds or anything | 17:46 |
| noonedeadpunk | oh, yes, sure | 17:46 |
| jrosser | but br-mgmt is all bonds and blah blah | 17:46 |
| jrosser | and nothing except monitoring should break if we lose the ssh network | 17:46 |
| noonedeadpunk | I'm not sure how it all ends up at the end in terms of networking design, as I don't have enough clearance for all details of the deployment | 17:47 |
| noonedeadpunk | but presumably it will be nuts | 17:47 |
| jrosser | hah :) | 17:47 |
| jrosser | we were thinking about how to do L3-to-the-host and ECMP rather than L2 bonds | 17:48 |
| jrosser | that is maybe the most fragile / largest blast radius failure we have right now | 17:49 |
| noonedeadpunk | Yeah, that has been raised as well | 17:49 |
| noonedeadpunk | But more pain point is likely CD-ing changes | 17:49 |
| jrosser | firmware upgrades on VPC pair switches was terrifying and when it goes wrong you lose a whole load of stuff | 17:49 |
| noonedeadpunk | at least for me, as any SSH access should be minimized | 17:50 |
| noonedeadpunk | So - unattended opensatck upgrades :D | 17:50 |
| jrosser | i think the bulk of our use of SSH is for debugging wtf is going on when things break | 17:51 |
| noonedeadpunk | yeah, exactly. | 17:51 |
| noonedeadpunk | but I think idea is to get rid of deploy host somehow and jsut use CI/CD for all management | 17:52 |
| noonedeadpunk | Which I still can't fully turn my head around implementation | 17:52 |
| jrosser | its perhaps not so hard to have the deploy host create-able from state all stored in git | 17:53 |
| jrosser | but there does seem to need to be a lot of human-in-the-loop to deploy changes | 17:54 |
| noonedeadpunk | As I feel it being too futuristic/utopic | 17:54 |
| noonedeadpunk | Well, bootstraping deploy host with Zuul is already done - wasn't that hard indeed | 17:54 |
| noonedeadpunk | But getting rid of human-in-the-loop is ugh.... | 17:55 |
| noonedeadpunk | Simple operations like adding new node or rolling out specific service change is doable | 17:55 |
| noonedeadpunk | But still not sure what to do with ad-hocs or debugging | 17:56 |
| noonedeadpunk | Btw have some sweet test for the inventory, likely need to share it in ops repo or smth... | 17:58 |
| noonedeadpunk | That verifies no mistake was made and groups are not intersecting (like no l3 agent on compute hosts), that amount of containers is correct and not missing smth (like 2 galera ones), and that every compute is part ovsagent as well. So basically intersections of different groups and checking their content | 18:00 |
| noonedeadpunk | Fully ansible to be native zuul job | 18:01 |
| noonedeadpunk | a bit generalized tox version: https://paste.openstack.org/show/bNhsa7uWomhDfoYtUlDg/ | 18:05 |
| noonedeadpunk | order of elements in collocated_groups is critical just in case | 18:06 |
| noonedeadpunk | yeah, nothing too fancy, but protects from generic human mistakes | 18:07 |
| noonedeadpunk | ok, will check out for weekends now. Will have a look on stable branches merge status though to propose bumps | 18:08 |
| mgariepy | have a nice one noonedeadpunk | 18:08 |
| mgariepy | take care | 18:08 |
| opendevreview | Merged openstack/openstack-ansible master: Restore dynamic_inventory unit testing https://review.opendev.org/c/openstack/openstack-ansible/+/869776 | 18:20 |
| jrosser | have a good weekend | 18:32 |
| noonedeadpunk | 963!rg91w | 19:18 |
| noonedeadpunk | sorry, kiddo playing with keyboard :D | 19:18 |
| mgariepy | ;) | 19:36 |
| jamesdenton | mgariepy i am around now, sorry | 20:24 |
| mgariepy | no worries. | 20:25 |
| mgariepy | was only looking at the ovn png again and i was wondering why it wasn't containing the kvm ;) | 20:25 |
| jamesdenton | as in, the tap? or something else? | 20:26 |
| mgariepy | like it done for ovs and lxb | 20:26 |
| jamesdenton | oh, ok. good question and prob an oversight | 20:27 |
| jamesdenton | if you wanna leave a comment i will prob revisit that commit this weekend | 20:28 |
| mgariepy | done thanks | 20:29 |
| jamesdenton | thanks, i see what you mean. complete oversight | 20:29 |
| jamesdenton | i think you had another comment, too. the "br-tunnel" thing. i may remove or at least comment on that | 20:30 |
| mgariepy | well yeah it depends on some implementation also. | 20:32 |
| mgariepy | added the comment for br-tunnel | 20:34 |
| jamesdenton | thanks | 20:36 |
| mgariepy | i'd like to document a bit the ovn commands a bit. | 20:37 |
| mgariepy | on different hosts and so on. how to map the different UUID | 20:38 |
| mgariepy | https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html#useful-open-virtual-network-ovn-commands | 20:38 |
| mgariepy | to add to that a bit. | 20:38 |
| jamesdenton | yeah, any tips you have would be good | 20:45 |
| mgariepy | yep i'll check to start something next week | 20:47 |
| mgariepy | but i don't have mnaio install. so documenting the output of the command would be kinda hard for me. | 20:48 |
| jamesdenton | If you have an openstack cloud to deploy against, i've been working on a MNAIOv2 that will do an 8-node VM-based deploy w/ OVN or OVS | 20:50 |
| mgariepy | i do have a couple of clouds to deploy against. | 20:50 |
| mgariepy | where is your v2 code ? | 20:50 |
| jamesdenton | i need to clean it up a little bit, but it's here: https://github.com/busterswt/mnaiov2 | 20:50 |
| jamesdenton | the readme needs some love, i will update that in a few. but uses terraform to deploy the VMs, sets up some routers, etc. Needs a routed external network so your terraform/ansible host can hit the VMs | 20:52 |
| mgariepy | i'll take a look with a collegue next week for sure. | 20:54 |
| jamesdenton | cool, any feedback is appreciated. | 20:54 |
| mgariepy | sure | 20:56 |
| mgariepy | now it's time to go shovel some snow. | 20:58 |
| mgariepy | have a nice weekend | 20:59 |
| jamesdenton | sounds like fun, don't hurt yourself! take care | 20:59 |
| mgariepy | not super fun.. ;p haha i need to do it so if i need to leave i don't get stuck | 21:01 |
| prometheanfire | I know this isn't the ceph channel but that one's dead, so... when using `ceph orch` to deploy radosgw, is the only way to configure it via the `ceph config set` command (and not via the hosts's ceph.conf)? | 21:04 |
| jamesdenton | sorry, can't help ya | 21:06 |
| opendevreview | Merged openstack/openstack-ansible-galera_server stable/zed: Prevent mariadbcheck.socket to wait for network.target https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/870056 | 21:24 |
| MrR | noonedeadpunk jrosser thank you, that python string really helped (shortened to timeout 5 python3 -m http.server 8888 --bind IP), after trail and mostly error I have figured that the ip to forward to is the internal_lb_vip, the ONE ip i hadn't tried which is absolutely typical. I don't know if this is how it's supposed to be but if it is i found no documentation stating so, now on to the next problem haha | 21:25 |
| MrR | prometheanfire yes only via config set, but stringing them together with && makes it easier | 21:26 |
| prometheanfire | kk, thanks. upstream docs still seem to point to ceph.conf config by default | 21:27 |
| MrR | no worries, yeah alot of the docs refer to past ways, try and stick to the cephadm section on the ceph site, if its not under the ceph header it can take some piecing together to find the new command | 21:28 |
| MrR | *cephadm header | 21:28 |
| prometheanfire | yepyep | 21:29 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/zed: Fix scheduler track_instance_changes option https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/870058 | 21:29 |
| opendevreview | Merged openstack/ansible-role-pki stable/zed: Create relative links to root instead of absolute https://review.opendev.org/c/openstack/ansible-role-pki/+/870065 | 21:30 |
| jrosser | MrR: it should not be the internal_lb_vip you need to forward, it really should be the external one | 21:45 |
| jrosser | if you think about a cloud without a NAT like you have, but with API endpoint internet accessible, the mgmt network (internal vip) will not be routable from the outside where the LE servers are (rfc1918 address), but the external VIP should be (proper external IP) | 21:47 |
| jrosser | MrR: just FYI here is how it works https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy/haproxy.yml#L224-L241 | 21:50 |
| jrosser | port 443 is horizon as https, and haproxy is set up to redirect port 80 to port 443 | 21:50 |
| MrR | those were my thought's which is why I hadn't thought to try it, for now, it's progress, when i finally have a running cloud that i can tear down and bring up easily i'll go back to it and fix it, this is my forth way of deploying (tried packages, trippleo and kolla before this) and the first time i've had such an issue | 21:50 |
| MrR | Thanks, bookmarked for when i look into it | 21:51 |
| jrosser | but there is also an haproxy acl set up (haproxy_ssl_letsencrypt_acl) which redirects any requests to .well-known/acme-challenge to the letsencrypt loadbalancer backend rather than horizon | 21:51 |
| MrR | Found my next problem, qdrouterd is looking for a file called debian-11.yml that doesnt seem to exist | 21:52 |
| MrR | Think i'll just remove it as it was just an enable everything and see what fails option haha | 21:52 |
| jrosser | hmm i am surprised that is even being deployed tbh | 21:52 |
| MrR | well that reply tells me i should get rid of it haha | 21:53 |
| jrosser | anyway it's good to hear you've got LE working, the setup for that is really quite subtle in order to make it support H/A | 21:54 |
| jrosser | MrR: i think that you'll find OSA is more of a 'toolbox' approach compared to some of the other deploy methods | 21:55 |
| jrosser | you can have really whatever architecture you want, metal or containers, tons of choice on networking, everything configurable | 21:56 |
| jrosser | we provide a set of "sensible defaults" but really that is just a reference to start from | 21:56 |
| MrR | quick question, is there a better way of removing services that i've missed? The way i have done it is to use the inventory-manage script, remove all references and destroy the related containers | 21:57 |
| jrosser | downside is that the learning curve can be very steep, and there are many possbilties to break things | 21:57 |
| jrosser | inventory-manage is the 'official' tool we provide to do that | 21:58 |
| MrR | yeah my issues with some of the other methods was documentation and troubleshooting, this tells me where i can find the problem, the others you needed a crystal ball sometimes | 21:58 |
| jrosser | it would be really useful to get any feedback on things like this | 21:58 |
| MrR | great thanks, i'm doing something right haha | 21:58 |
| jrosser | heh no worries | 21:58 |
| jrosser | most people active here are actual operators | 21:59 |
| jrosser | not just developers working on $tool for their employers product, so theres a lot of real-world experience here | 21:59 |
| prometheanfire | MrR: yep, got it working | 22:04 |
| mgariepy | jamesdenton, it's done. 1h of snowblower and a shovel do the trick. 200 ft by 15ft is kinda long for snow removal | 22:04 |
| * prometheanfire is waiting for the snow to stop, lake effect stuff here | 22:06 | |
| mgariepy | here the snow is full of water. it's a bit too hot for snow to be ligth and fluffy haha | 22:09 |
| mgariepy | there was a good 4-5 inch of snow to remove. | 22:09 |
| prometheanfire | ah, nothing bad here, just 2ish inches | 22:12 |
| MrR | jrosser I can admit now I am familiar with it this is the best method, the documentation is much better but i have found a few things i have had to go on a deep dive for and i'm still not sure if i can override files such as /etc/ansible/roles/os_trove/defaults/main.yml in /etc/openstack_deploy and it took a while to find i even needed to edit that file. Which reminds me, when setting up independant rabbitmq | 22:16 |
| MrR | for trove, in group_vars/trove_rabbitmq.yml it states to put rabbitmq_monitoring_password: <password>, my question is, do i replace <password> or does it pull it from user_secrets.yml? I've looked and still dont know as i havent got that far | 22:16 |
| MrR | and if it doesnt pull it from user_secrets.yml it really should | 22:17 |
| jrosser | so as far as everything in /etc/ansible/roles/* goes, the key thing is that for all of them defaults/main.yml is the "external interface" to the roles | 22:17 |
| jrosser | that should be your first port of call and essentially the documentation for the external interface to those roles | 22:18 |
| jrosser | regarding trove, honestly i can say thats one of our least used roles | 22:19 |
| jrosser | MrR: could you give me a link to group_vars/trove_rabbitmq.yml ? | 22:20 |
| MrR | it's not something i need and won't be in my final deployment, just testing for the sake of it | 22:20 |
| MrR | yeah let me just find it | 22:21 |
| jrosser | the need to set up an extra rabbitmq is tricky | 22:21 |
| jrosser | oh hold on thats not trove is it.... | 22:21 |
| jrosser | :) | 22:21 |
| jamesdenton | mgariepy yeah, 200ft, oh boy | 22:21 |
| MrR | https://docs.openstack.org/openstack-ansible-os_trove/zed/configure-trove.html | 22:21 |
| jamesdenton | no snow for me since moving back from Ohio. | 22:21 |
| opendevreview | Merged openstack/openstack-ansible master: Skip haproxy with setup-infrastructure for upgrades https://review.opendev.org/c/openstack/openstack-ansible/+/869974 | 22:22 |
| jrosser | MrR: i think that documentation is probably wrong | 22:24 |
| jrosser | because rabbitmq_monitoring_password and rabbitmq_cookie_token are defined in user_secrets.yml, and the way those vars are loaded is at the highest possible preference in ansible | 22:25 |
| mgariepy | jamesdenton, i have a 45inch walk behind snowblower | 22:26 |
| mgariepy | i'm too cheap to buy a tractor ;) haha | 22:26 |
| jrosser | so those will always override anything set in /etc/openstack_deploy/group_vars/trove_rabbitmq.yml | 22:26 |
| MrR | an old value then, suppose if it succeeds i can check the password/cookie and see if it matches, if not i'll just remove it as i don't actually need it, i have a strange need to find out what does and doesn't work, swear i like making more work for myself | 22:28 |
| jrosser | i think that that is going to happen is that the trove rabbitmq will end up with the same var settings as the main one | 22:30 |
| jrosser | and thats a quite interesting bug! | 22:31 |
| jamesdenton | i had a snow thrower, but didn't end up using it as we had 2 mild winters before i left. ended up giving it to the neighbor | 22:31 |
| MrR | yeah as it completely defies the point of a separate rabbitmq haha, i'll let you know | 22:31 |
| MrR | or file a bug, which would probably be a better idea | 22:32 |
| jrosser | well perhaps - all being equal the trove vm communicate over a dedicated br-dbaas network and can only contact the dedicated rabbitmq | 22:32 |
| jrosser | if they can get to the mgmt network you have bigger problems | 22:32 |
| MrR | true, forgot I had to set a network for it | 22:33 |
| jrosser | i think you might be in for a rough time trying to just enable everything | 22:33 |
| jrosser | telemetry stack can be challenging | 22:33 |
| prometheanfire | last I knew it was fairly orphaned | 22:45 |
| MrR | jrosser, well I'm about to find out setup-infra has just finished without error, so on to setup-openstack, I did go through as much documentation as I could find to set all variables but then again i'm also working on at least 4 projects and running a household so i won't be too surprised if i've missed something, actually not sure if i set up the ceph rgw on cephs side, set up everything ceph apart from taht | 23:04 |
| MrR | so best check | 23:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Skip haproxy with setup-infrastructure for upgrades https://review.opendev.org/c/openstack/openstack-ansible/+/870068 | 23:40 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!