| opendevreview | Takashi Kajinami proposed openstack/ansible-config_template master: Remove TripleO jobs https://review.opendev.org/c/openstack/ansible-config_template/+/877486 | 02:09 |
|---|---|---|
| opendevreview | Takashi Kajinami proposed openstack/ansible-config_template master: Replace deprecated whitelist_externals https://review.opendev.org/c/openstack/ansible-config_template/+/877570 | 02:09 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use a map file to select haproxy horizon backend from the base frontend https://review.opendev.org/c/openstack/openstack-ansible/+/876851 | 09:03 |
| Losraio | Hello everyone | 09:05 |
| Losraio | I have the following error when running the setup-infrastructure.yml | 09:05 |
| Losraio | https://paste.openstack.org/show/bR08qz7GKQAx5QJuXXRF/ | 09:05 |
| Losraio | Any help would be much appreciated | 09:06 |
| Losraio | FYI, yes, the 10.1.0.12 IP is reachable and SSH-able from the deployment host | 09:06 |
| jrosser | Losraio: is 10.1.0.12 your internal VIP? | 09:18 |
| Losraio | its the management network IP for the controller node | 09:19 |
| Losraio | and the internal VIP | 09:19 |
| Losraio | Tbh I think it has something to do with the system resources not being enough to run all these containers | 09:19 |
| jrosser | so you should have haproxy bound to that IP | 09:19 |
| Losraio | Oh, I think I omitted that in the user_config.yaml | 09:20 |
| dokeeffe85 | Sorry all, I was dragged away yesterday. Thanks for all the answers :) | 09:20 |
| Losraio | But won't it be a problem if both the haproxy_hosts and interal_lp_vip have the same IP address? | 09:20 |
| jrosser | Losraio: you mean you have no `haproxy_hosts` defined? | 09:21 |
| Losraio | Nope I don't | 09:22 |
| Losraio | Since it mentions that it's optional | 09:22 |
| Losraio | And I'm only testing right now | 09:22 |
| jrosser | haproxy can't really be optional | 09:23 |
| Losraio | But I have to define it, then? | 09:23 |
| jrosser | do you remember where it says it is optional? | 09:23 |
| jrosser | well i mean it kind of can be optional, in that people have sometimes used a hardware F5 loadbalancer instead of haproxy, so in that sense it can be optional | 09:24 |
| jrosser | but there must be a loadbalancer of some kind | 09:24 |
| Losraio | Yes, it says so on the openstack_user_config.yml.example | 09:25 |
| Losraio | So... I guess I must declare it in the user_config then. But will there be a problem if it has the same IP as the internal_lb_vip? | 09:26 |
| jrosser | hmm well it says `Recommend at least one target host for this service if hardware load balancers are not being used.` | 09:26 |
| jrosser | so i think it is that a loadbalancer is expected, it's just that its your choice to use haproxy or something else | 09:26 |
| Losraio | Right | 09:27 |
| jrosser | and i would advise that you put an IP on br-mgmt of each controller | 09:27 |
| Losraio | I do so already | 09:28 |
| jrosser | and also a completely separate IP for the loadbalancer VIP, just so its completely obvious whats going on | 09:28 |
| Losraio | OK, thanks | 09:28 |
| jrosser | if you have H/A controllers then thats a requirement anyway | 09:29 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_nova master: Stop installing qemu-system on debian variants https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/877604 | 09:58 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Remove deprecated support for cisco ucs and cims ironic drivers. https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/877606 | 10:16 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use a map file to select haproxy horizon backend from the base frontend https://review.opendev.org/c/openstack/openstack-ansible/+/876851 | 10:26 |
| jrosser | damiandabrowski: sorry my comment about enabling TLS backends was on the wrong patch | 10:39 |
| jrosser | i think i meant that for the 'big patch' instead | 10:40 |
| damiandabrowski | np, i didn't work on it yet(just rebased it on top of your changes) | 10:40 |
| damiandabrowski | but i get your point | 10:40 |
| damiandabrowski | and you were probably right about temporary_service_defintions, we probably don't need them anymore | 10:41 |
| jrosser | hopefully moving the haproxy vars to group_vars should be a pretty simple move now | 10:41 |
| jrosser | os_ironic | 10:43 |
| jrosser | arg | 10:43 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-haproxy_server master: Prepare haproxy role for separated haproxy config https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/875779 | 10:51 |
| noonedeadpunk | damiandabrowski: can you kindly review these 2 things? https://review.opendev.org/q/topic:bump_osa+status:open | 10:59 |
| damiandabrowski | done, i also have few simple patches that would appreciate reviews :D | 11:03 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/877429 | 11:03 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/871188 | 11:03 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible/+/876851 | 11:04 |
| noonedeadpunk | damiandabrowski: have you played with https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/876749 ? | 11:11 |
| damiandabrowski | did a quick test on my aio and worked fine | 11:11 |
| noonedeadpunk | ++ | 11:12 |
| opendevreview | Merged openstack/openstack-ansible master: Split haproxy horizon config into 'base' frontend and 'horizon' backend https://review.opendev.org/c/openstack/openstack-ansible/+/876160 | 11:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Install socat and configure ipmtool-socat console interface https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/877618 | 11:18 |
| jrosser | damiandabrowski: are https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/871188 and https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/876749 a merge confict? | 11:24 |
| jrosser | the maps patch is written on top of the original code i think with the item.service key still present | 11:24 |
| damiandabrowski | ahh IMO it is, wonder why gerrit does not show "merge conflicts" section | 11:26 |
| damiandabrowski | what do you suggest? put them in the same relation chain? | 11:27 |
| damiandabrowski | or maybe i can just rebase 871188 after map patch is merged | 11:29 |
| jrosser | given that 876749 is in the gate already i think 876749 needs rebasing on top of it now, and the new map code adjusting also to remove the extra .service | 11:29 |
| jrosser | arg | 11:29 |
| jrosser | given that 876749 is in the gate already i think 871188 needs rebasing on top of it now, and the new map code adjusting also to remove the extra .service | 11:30 |
| jrosser | ^ better :) | 11:30 |
| damiandabrowski | ack | 11:30 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Enable TLS frontend for repo_server by default https://review.opendev.org/c/openstack/openstack-ansible/+/876426 | 11:38 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Rename idrac interfaces to idrac-wsman https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/877627 | 12:14 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Enable raid interface implementations for ironic hardware drivers https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/877628 | 12:14 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Add a no_driver ironic driver type https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/877629 | 12:14 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-haproxy_server master: Add support for haproxy map files https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/876749 | 12:27 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-haproxy_server master: Simplify haproxy_service_configs structure https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/871188 | 12:27 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-haproxy_server master: Prepare haproxy role for separated haproxy config https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/875779 | 12:27 |
| jrosser | argh 876749 would have been close to merging /o\ | 12:30 |
| damiandabrowski | yeah, i still don't understand what happened :| i just wanted to rebase 871188 on top of 876749 | 12:33 |
| damiandabrowski | but git review did something with 876749's commit message | 12:34 |
| jrosser | it's like theres a missing info on git review | 12:35 |
| jrosser | it's "push all these - are you sure" and some of the time only some of them change | 12:35 |
| jrosser | and other times a bunch can change in a surprising way | 12:35 |
| noonedeadpunk | oh, yes, that indeed does happen sometime | 12:39 |
| noonedeadpunk | I have no idea why though | 12:39 |
| opendevreview | Merged openstack/openstack-ansible-repo_server master: Turn off absolute_redirect for nginx https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/877429 | 13:05 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-haproxy_server master: Simplify haproxy_service_configs structure https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/871188 | 13:19 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-haproxy_server master: Prepare haproxy role for separated haproxy config https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/875779 | 13:20 |
| opendevreview | Merged openstack/openstack-ansible stable/xena: Bump OpenStack-Ansible Xena https://review.opendev.org/c/openstack/openstack-ansible/+/877488 | 13:31 |
| dokeeffe85 | Hi all, quick one, can I change the keystone public endpoint for an IP address to a DNS without breaking everything? | 13:49 |
| dokeeffe85 | It's ok, sorted :) | 13:53 |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: Bump OpenStack-Ansible Yoga https://review.opendev.org/c/openstack/openstack-ansible/+/876982 | 13:55 |
| opendevreview | Merged openstack/openstack-ansible master: Deploy step-ca when 'stepca' is part of the deployment scenario. https://review.opendev.org/c/openstack/openstack-ansible/+/876637 | 13:55 |
| jrosser | dokeeffe85: you'll have to run (parts) of all the service playbooks to get the service catalog entries updated (or manually adjust them) | 13:57 |
| jrosser | ^ if you want to make the public endpoint for all the services be the FQDN | 13:58 |
| dokeeffe85 | Thanks jrosser | 14:00 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Add a no_driver ironic driver type https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/877629 | 14:01 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Add `acl` package to all hosts and containers https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/877665 | 14:46 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-config_template master: Remove TripleO jobs https://review.opendev.org/c/openstack/ansible-config_template/+/877486 | 14:46 |
| Losraio | Hey all | 15:00 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-hardening master: Disable UsePriviledgeSeparation directive for sshd https://review.opendev.org/c/openstack/ansible-hardening/+/877666 | 15:00 |
| Losraio | Could someone please enlighten me on this eror? | 15:00 |
| Losraio | https://pasteboard.co/mqEfDxzXn3aO.png | 15:00 |
| Losraio | Sorry for the screenshot instead of the paste | 15:00 |
| Losraio | Should I perhaps try to get rid of the iscsi_ip declaration on the user_config.yml?# | 15:08 |
| noonedeadpunk | o/ | 15:15 |
| noonedeadpunk | Losraio: do you have anything in cinder-api logs? | 15:17 |
| Losraio | Hmm let me check | 15:18 |
| Losraio | Where should these logs be? | 15:18 |
| noonedeadpunk | As eventually what this task is trying to do - jsut execute command `/openstack/venvs/utility-26.0.1/bin/openstack volume type create --property volume_backend_name=LVM_SCSI lvm` from utility container | 15:18 |
| noonedeadpunk | But API does not respond in time and looks like it gets stuck processing the call | 15:19 |
| noonedeadpunk | you can check inside cinder-api containers using journalctl -u cinder-api | 15:19 |
| Losraio | I'm thinking whether it has something to do with my user config | 15:19 |
| noonedeadpunk | It totally can | 15:19 |
| Losraio | Because the IP address I have assigned to iscsi_ip is the management network address for the storage node | 15:20 |
| Losraio | Should this cause a problem? | 15:20 |
| noonedeadpunk | Well, cinder should be able to reach it. But it might also depend on IP storage node listening for connections | 15:21 |
| Losraio | What makes me worry is the error about 2 positional arguments though | 15:21 |
| Losraio | Let me try to run the playbook again | 15:22 |
| noonedeadpunk | You can try running this command I provided | 15:22 |
| Losraio | I sure will | 15:22 |
| noonedeadpunk | It's exactly same | 15:22 |
| Losraio | How do I gain access to the container from the infra node? | 15:23 |
| Losraio | I haven't worked with lxc before so I'm completely clueless | 15:23 |
| noonedeadpunk | lxc-attach -n <container_name> | 15:23 |
| Losraio | Oh | 15:23 |
| noonedeadpunk | lxc-ls --active to list them all | 15:23 |
| noonedeadpunk | *all running | 15:23 |
| spatel | jamesdenton look like my company can arrange some budget for summit.. | 15:25 |
| spatel | looking for hotel option etc.. | 15:25 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Enable raid interface implementations for ironic hardware drivers https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/877628 | 15:26 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Add a no_driver ironic driver type https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/877629 | 15:26 |
| noonedeadpunk | spatel: sweet. I hope I wil lget my visa in time :D | 15:29 |
| spatel | I hope!!! you will | 15:29 |
| spatel | Did you guys book hotel? | 15:29 |
| spatel | admin1 what about you? | 15:30 |
| noonedeadpunk | Um, I guess so, but I have no idea which one | 15:30 |
| spatel | Registration is $799 for summit.. | 15:30 |
| noonedeadpunk | spatel: don't you have ATC code??? | 15:30 |
| spatel | I need to reach-out to someone for discount promo | 15:30 |
| spatel | No | 15:30 |
| noonedeadpunk | I bet you've contributed? | 15:30 |
| spatel | How do i find that? | 15:31 |
| noonedeadpunk | https://www.stackalytics.io/?release=zed&user_id=satish-txt&metric=commits | 15:31 |
| noonedeadpunk | well, depending on how they were counting | 15:31 |
| noonedeadpunk | Don't you have email with topic `OpenInfra Summit Vancouver 2023 Registration Promo Code`? | 15:32 |
| spatel | No i don't have that email.. let me search again | 15:33 |
| Losraio | noonedeadpunk so, I did not get to run your suggested command just yet, but I had started the playbook execution from before and the error did not pop up... | 15:37 |
| Losraio | Let's see how that goes | 15:37 |
| noonedeadpunk | oh, huh, ok | 15:39 |
| noonedeadpunk | maybe it needed more time then timeout configured... | 15:40 |
| Losraio | That's what I'm thinking, because the infra node is incredibly stressed right now | 15:40 |
| Losraio | As in RAM and CPU usage | 15:40 |
| opendevreview | Merged openstack/openstack-ansible master: Add a /etc/hosts entry for the external IP of an AIO https://review.opendev.org/c/openstack/openstack-ansible/+/876638 | 16:16 |
| Losraio | noonedeadpunk: Yeah, I can't even SSH into the infra node anymore, so I guess I should allocate much more resources to the VM before trying out anything else | 16:17 |
| noonedeadpunk | Well, having 12-16 GB and 4 CPU cores should be enough for the VM | 16:18 |
| noonedeadpunk | In case of aio | 16:18 |
| Losraio | I'm following the test example from the documentation | 16:20 |
| Losraio | And the sole infra node has got 8GB of ram and 8 vCPUS | 16:20 |
| Losraio | But, for some reason, just from running the playbooks the RAM usage is capped at at least 95% usage | 16:20 |
| Losraio | And the CPU won't go lower than 70% | 16:20 |
| noonedeadpunk | well, 8gb is what we're using in CI and it's not really enough IMO. | 16:21 |
| Losraio | Yeah I can totally understand why :D | 16:21 |
| noonedeadpunk | 8 cpus should be more then okay to be frank. I wonder what does consume CPU as it sound a bit off | 16:22 |
| noonedeadpunk | Or well | 16:22 |
| noonedeadpunk | In AIO we do limit amount of threads for services | 16:22 |
| Losraio | I'm suspecting that the proxmox cluster on which my vms are running is simply inadequate | 16:22 |
| noonedeadpunk | You can check all these `_wsgi_threads` `_wsgi_processes` variables we override here: https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2 | 16:23 |
| noonedeadpunk | It will save you a lot of resources if you're not doing AIO but trying to reproduce deployment manually | 16:23 |
| Losraio | hmm | 16:24 |
| Losraio | I see | 16:26 |
| Losraio | You mostly use a single thread and a single process | 16:27 |
| noonedeadpunk | Yeah, kinda. But again for POC on a small VM you don't expect concurrency anyway | 16:28 |
| Losraio | True | 16:28 |
| noonedeadpunk | jrosser: huh, have you seen this uploaded patch? https://bugs.launchpad.net/openstack-ansible/+bug/2009834 | 16:30 |
| noonedeadpunk | I'm not sure it's correct though | 16:31 |
| jrosser | i don't think it is correct | 16:33 |
| jrosser | the thing is the whole logic with those vars is to decide if to restart control plane services at the point that an upgrade is "complete" | 16:33 |
| noonedeadpunk | yeah, it kind of breaks the point | 16:33 |
| jrosser | and that is all totally not relevant for when adding a compute node | 16:33 |
| jrosser | i did try to split out the code that deploys the control plane from the code that deploys a compute node | 16:34 |
| jrosser | *but* the thing is you need to call the whole setup-openstack.yml even when adding a compute node as there is no idea what services need installing, you can't just isolate nova compute | 16:34 |
| noonedeadpunk | I wonder if we should just gather facts for nova_all somewhere there... | 16:34 |
| jrosser | even so it's not necessarily right - idk what any of this does when there is a node down for example | 16:35 |
| jrosser | so you cant gather facts for that, then it iterates over nova_all and -> fail | 16:35 |
| noonedeadpunk | well. to be fair if you're upgrading when compute is down, or upgrading through couple of releases - nova will fail on itself | 16:36 |
| noonedeadpunk | but yes, adding such limitation for adding compute is not good | 16:36 |
| jrosser | right - though somehow adding a compute node should be a relatively lightweight things | 16:36 |
| noonedeadpunk | we can add --skip-tags for example... But it's nasty | 16:37 |
| jrosser | maybe one of the original ideas to add a `-e nova_add_compute_node=True` is perhaps not so bad after all | 16:37 |
| jrosser | so that the whole business of restarting the control plane stuff is just skipped | 16:37 |
| noonedeadpunk | even if imagine dropping this all out and re-factoring logic - it will be totally not backportable | 16:42 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Add support for haproxy map files https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/876749 | 16:58 |
| opendevreview | Merged openstack/openstack-ansible master: Use certbot to generate SSL cert for the external VIP in 'stepca' scenario https://review.opendev.org/c/openstack/openstack-ansible/+/876639 | 17:00 |
| jrosser | noonedeadpunk: do i remeber you saying you had people trying ceph-immutable-object-cache ? | 17:07 |
| noonedeadpunk | yeah, folks played with that | 17:09 |
| noonedeadpunk | But I have no idea about details. But likely can ask/connect you | 17:09 |
| jrosser | i was just trying to make it work - and it runs, but pretty unclear what to do next when libvirt+librbd does not appear to use it | 17:09 |
| jrosser | (this is the image read cache btw, not the write cache) | 17:10 |
| noonedeadpunk | ah. no, we were playing with write cache | 17:10 |
| noonedeadpunk | As they were trying to reduce latency for writes | 17:10 |
| noonedeadpunk | But I'd expect same logic to be abpplied... But I think they were leveraging ceph.conf that is being used by nova for rbd connection | 17:12 |
| jrosser | which should be /etc/ceph/ceph.conf - or is there another? | 17:12 |
| noonedeadpunk | Yeah, this one, unless you've defined another path for ceph.conf | 17:12 |
| noonedeadpunk | So caching worked transparently for libvirt | 17:13 |
| jrosser | ^ write? | 17:14 |
| noonedeadpunk | yeah, it was write I believe | 17:18 |
| noonedeadpunk | as all fuss was about commit/apply latencies | 17:23 |
| jrosser | wierd thing is there is almost nothing on the internet about using ceph-immutable-object-cach | 17:29 |
| jrosser | apart from the ceph docs | 17:30 |
| admin1 | spatel, not in this one | 20:01 |
| admin1 | maybe in the next one | 20:01 |
| spatel | ohh okie | 20:02 |
| spatel | Next will be in US i think | 20:02 |
| admin1 | i have finally been able to use magnum to launch k8s cluser and use the octavia lb ingress .. next in the list is to use cinder for volumes | 20:04 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Add `acl` package to all hosts and containers https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/877665 | 20:44 |
| jrosser | does anyone have a example of using ceph-immutable-object-cache with libvirt/rbd - the docs are fine for setting it up but a reproducible example of how it's expected to work would be great to see | 20:54 |
| jrosser | arg -ECHAN | 20:54 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
