| HN | Hello everyone, I need help on how to do Customization Script (Cloudinit) on Windows VM | 01:26 |
|---|---|---|
| HN | I've problem to inject Administrator password on it | 01:26 |
| HN | here the pastebin : https://pastebin.com/gmrH8qyR | 01:36 |
| opendevreview | Andrew Bonney proposed openstack/ansible-role-python_venv_build stable/wallaby: Add distro/arch to requirements file path https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/804432 | 08:08 |
| admin1 | morning \o | 08:11 |
| admin1 | so i have a platform up .. and then i added gnocchi .. but haproxy is not adding the gnocchi endpoints .. how to force haproxy to do this ? | 08:12 |
| admin1 | rerunning haproxy playbook did not helped | 08:12 |
| opendevreview | Andrew Bonney proposed openstack/ansible-role-python_venv_build stable/victoria: Add distro/arch to requirements file path https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/804493 | 08:24 |
| admin1 | https://docs.openstack.org/openstack-ansible/victoria/user/prod/gnocchi_redis.html -- for this, is it recommended to create custom lxc containers .. or do it directly int he controllers ? | 08:24 |
| DK4 | jrosser: thanks for the hints and advice. i found a mistake in my config files that caused the issue and fixed it | 08:45 |
| anskiy | admin1: do you have metering-infra_hosts and metrics_hosts (not sure if this one needed for gnocchi) in openstack_user_config.yml? | 08:48 |
| admin1 | i have those | 08:50 |
| admin1 | metrics_hosts: metering-compute_hosts: metering-alarm_hosts: and metering-infra_hosts: | 08:51 |
| admin1 | if i use the inventory manage to list., i do see the 3x gnocchi_api containers, the containers are already created | 08:51 |
| admin1 | gnocchi roles run fine without issues . | 08:52 |
| admin1 | just that the endpoint in haproxy seems to be missing,, which blocks the celilometer role to run i think | 08:52 |
| admin1 | endpoint list has added gnocchi on internal http://172.29.236.9:8041 . but that 8041 is no where to be found in haproxy | 08:53 |
| jnamdar | hi guys, someone else prolly mentioned it but are the docs down just for me? :D | 08:59 |
| jnamdar | on https://docs.openstack.org/openstack-ansible/latest/ | 08:59 |
| jnamdar | only happening on /latest it seems | 09:00 |
| admin1 | https://docs.openstack.org/openstack-ansible/wallaby/ works | 09:48 |
| admin1 | maybe a softlink issue :) | 09:48 |
| kleini_ | Can I utilize OSA somehow to keep Bionic packages up to date? All my hosts and containers show a long list of possible package upgrades. And is it possible to upgrade the ubuntu-bionic-amd64 container, so new service containers get up to date packages? | 10:21 |
| *** kleini_ is now known as kleini | 10:22 | |
| DK4 | so ive managed to get past the first two ansible playbooks and iam now using the deploy-openstack runbook at the task "Create Database for service" im failing tough | 10:30 |
| DK4 | https://pastebin.com/MzQM3st8 it sas connection refused to host, but iam unsure which hosts wants to connect to which, ive deployed some public keys between the hosts and ssh should actually work. any ideas? | 10:30 |
| admin1 | DK4, try with -vvvv | 10:37 |
| admin1 | before you run the opnstack playbook, you can try to a few random containers, expecially the galera and rabbitq and ping others | 10:38 |
| admin1 | to ensure that every other container can reach rabbit and galera | 10:38 |
| admin1 | or | 10:39 |
| admin1 | that your deploy host cannot directly ssh to the management container IP | 10:39 |
| admin1 | its good for it to be in the same layer2 . or at least routed | 10:39 |
| admin1 | i have 2 custom haproxy questions .. 1. how to override variables in haproxy extra conf, so that it binds only to the internal VIP and does not create an external frontend .. the docs are not clear in this part .. and 2. how to specify that the bind should be on a specific VIP .. like for ceph object storage, instead of region.cloud.com:8080, i | 10:51 |
| admin1 | want to bind to s3.region.cloud.com 443 | 10:51 |
| jrosser | admin1: for (1) look at how internal only services like galera are setup now in the haproxy vars and use the same approach for your internal only service | 11:54 |
| jrosser | for (2) if those two fqdn resolve to the same external IP in dns then there is nothing to do. if you need an additional external VIP for the s3 look in the haproxy role defaults at the vars available for defining extra VIP | 11:56 |
| jrosser | DK4: you should find the task in /etc/ansible/os_keystone which fails and set no_log to false, you will then get the underlying error shown | 11:57 |
| jrosser | I would also check that your galera cluster is peperly formed (see our cluster maintainable docs for this) andyou can also check that the mysql cli client in the utility container can interact with the db properly | 11:58 |
| admin1 | his erorr is: Failed to connect to the host via ssh | 11:59 |
| admin1 | looks like the deploy does not connect or have access to the internal mgmt range | 11:59 |
| jrosser | it doesn’t ssh to the utilit container | 11:59 |
| jrosser | it ssh to infra1 and then lxc exec <foo> | 12:00 |
| jrosser | anyway, whatever is hidden behind no_log is usually helpful | 12:02 |
| jrosser | I’d guess that ssh to infra1 is kind of ok otherwise it wouldn’t have got as far as keystone setup | 12:03 |
| admin1 | hi jrosser, do you know to force a haproxy rebuild or check why its not adding gnocchi endpoints | 12:09 |
| admin1 | i also rm -rf the ansible_facts thinking if something was cached | 12:10 |
| jrosser | don’t rebuild it, that’s not helpful | 12:10 |
| jrosser | you need to look at all the fragments of config files that are templates out | 12:11 |
| admin1 | "look at all the fragments of config files that are templates out" - went above my head :D | 12:11 |
| jrosser | then then all get assembled into the final config file | 12:11 |
| jrosser | well, you need to dig into how the role works | 12:12 |
| jrosser | it makes a small piece of the config file in /etc/haproxy for each service | 12:12 |
| jrosser | then are then all glued together | 12:12 |
| jrosser | so it either is / is not making the piece for gnocchi | 12:13 |
| jrosser | if it’s not then figure out why | 12:13 |
| admin1 | ok .. | 12:14 |
| jrosser | each service you’ve defined makes it’s own little file | 12:14 |
| jrosser | and as the role runs you’ll see them all generated in turn in the ansible output | 12:14 |
| admin1 | but the haproxy runs before the os-* roles are run | 12:30 |
| admin1 | and the endpoints are there saying backend is not connected as they are not up yet | 12:30 |
| admin1 | i added 3 things later .. gnocchi, ceilometer and aodh .. i do't see those little pieces inside the conf.d | 12:35 |
| admin1 | i will check how they are being created | 12:35 |
| jrosser | I think there are conditionals based on ansible groups in the data that defines the backends | 12:37 |
| admin1 | jrosser, isn't haproxy configuring the files first, https://github.com/openstack/openstack-ansible-haproxy_server/blob/master/tasks/haproxy_service_config.yml#L16 and then doing the master file | 13:07 |
| admin1 | if i rm -rf the /etc/haproxy folder and do a haproxy install -l $host, it re-generates the conf itself | 13:10 |
| admin1 | hmm | 13:10 |
| admin1 | i am not sure now :D | 13:10 |
| admin1 | so each role will generate the file, and in just gets appended | 13:13 |
| jrosser | no, haproxy role generates the whole haproxy config | 13:14 |
| jrosser | the service roles are not involved | 13:14 |
| jrosser | there is a var something like haproxy_services which defines the whole possible set of backends | 13:16 |
| admin1 | which is empty in the config | 13:16 |
| admin1 | https://github.com/openstack/openstack-ansible-haproxy_server/blob/master/defaults/main.yml#L47 | 13:16 |
| jrosser | it’s in openstack-ansible group_vars/haproxy | 13:16 |
| jrosser | that’s the role default | 13:16 |
| jrosser | lots of these are set to specific things in the openstack-ansible repo | 13:17 |
| admin1 | i see aodh and gnocchi there | 13:31 |
| admin1 | let me paste my config and variables | 13:31 |
| admin1 | jrosser https://gist.githubusercontent.com/a1git/b421c9c30c36622a5ecfe0aace490bb4/raw/7735cd82a113979b2f64f200b46db820525bd90a/gistfile1.txt | 13:37 |
| admin1 | if this helps | 13:37 |
| jrosser | not really, you need to do some more active debugging yourself | 13:40 |
| jrosser | start here https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy/haproxy.yml#L160 | 13:40 |
| jrosser | follow the code, add debug: tasks to the ansible where necessary | 13:40 |
| admin1 | shouldn't gnocchi_all be gnocchi_api in the groups ? | 13:46 |
| admin1 | inventory has gnocchi_all with children gnocchi_api and gnocchi_metricd | 13:47 |
| spatel | jrosser this is not going to sail - https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/803990 | 13:47 |
| spatel | its keep coming every neutron CI patch | 13:48 |
| ierdem | Hi, is there anyone had tried to use Skyline dashboard before? I have some problems about it: skyline error log https://paste.openstack.org/show/808059/, skyline.yaml https://paste.openstack.org/show/808060/. I am trying it on devstack/victoria | 14:04 |
| spatel | ierdem i never used that but i would like to try, is this free or paied? | 14:08 |
| ierdem | it is free, https://opendev.org/skyline, also you can find demo videos on youtube | 14:09 |
| spatel | i saw that demo | 14:09 |
| spatel | i will give it a try and let you know if its going to work or not :) | 14:10 |
| spatel | what is console vs apiserver? | 14:11 |
| ierdem | spatel, thanks, I will also keep trying | 14:12 |
| ierdem | I think console is UI, apiserver is backend | 14:12 |
| spatel | hmm | 14:12 |
| ierdem | I am using docker, both inside in container | 14:12 |
| canori01 | hey guys, I have a ussuri deployment with 1600 ports. I've noticed every time I do an upgrade, it takes about an hour for all the routers that the L3 agent handles to come online. (90 routers over 3 physical agents). Are there any tunables I could tweak to speed that up? Or prevent routers from going down altogether? I notice that when all neutron-server daemons are restarted, it also triggers that behavior | 14:13 |
| spatel | ierdem wise idea.. if it work then we should add into OSA :) lxc to have optional UI | 14:14 |
| andrewbonney | canori01: which version are you on at the moment? this sounds similar to something we fixed not so long ago | 14:14 |
| spatel | horizon is horrible :( | 14:14 |
| andrewbonney | canori01: I'm assuming you have a HA router setup rather than a single instance of the l3 agent? | 14:14 |
| canori01 | andrewbonney: That is correct - HA setup. I'm on ussuri currentyl | 14:18 |
| andrewbonney | canori01: I've just had a look an it seems we only backported as far as victoria. I'll just prep a patch you can take a look at | 14:19 |
| mgariepy | ierdem, the ui seems 100 time better than horizon. | 14:20 |
| canori01 | andrewbonney: I don't mind upgrading to victoria. I would like to take a look at the fix that went in though | 14:21 |
| andrewbonney | canori01: sure. it's https://github.com/openstack/openstack-ansible-os_neutron/commit/d12a8aa6c5f7af293e6b127795a13becd3d223d0 with a follow up in https://github.com/openstack/openstack-ansible-os_neutron/commit/af9f3b2257b5284a169360d8c92044dd6a53a1c3 to correct an error | 14:22 |
| canori01 | andrewbonney: awesome. Is that in the stable/victoria version of the playbook? Cause if so, I'll go ahead and pull that to try it out | 14:27 |
| andrewbonney | It's in the stable/victoria neutron role, although I can't say whether any other changes in the files on that branch may be incompatible with ussuri | 14:28 |
| ierdem | spatel, mgariepy . It looks like an enterprise paid version of horizon and it is so cool | 14:29 |
| canori01 | andrewbonney: That's fine. I'll just roll that whole deployment to victoria. It's a test one anyway | 14:29 |
| canori01 | andrewbonney: although, do you know what tunables I could tweak in the event that I needed to actually take down a physical node that runs an l3 agents or like if the node crashed and I suddenly needed a lot of routers to come back online? Does osad provide an override for that? I was thinking maybe things like rpc workers or rpc timeouts. | 14:31 |
| andrewbonney | In theory if one node crashed the other(s) should take over very quickly (a few seconds) as that's mostly keepalived's job. The issue those patches solve is if you take all of the network nodes down in quick succession, which prevents the usual recovery mode | 14:36 |
| canori01 | Fair enough. It sounds like that patch is addresses the behavior that I'm seeing | 14:39 |
| spatel | jrosser we are removing this patch but doesn't this cause issue who are doing upgrade from V->W and they have 10.5.6 running (in my case) ? https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/804371 | 15:32 |
| admin1 | ierdem, got a screenshot of it ? | 15:33 |
| jrosser | spatel: look at "relation chain" ON THE RIGHT O | 15:33 |
| jrosser | argh | 15:33 |
| admin1 | skyline dashboard was first created by cloudvps -- when i was working on it around 2017 .. looks like another UI got the same name and got uploaded a week back | 15:33 |
| jrosser | on the right of the screen | 15:33 |
| admin1 | i want to check to see if its the same or just the name matches | 15:33 |
| spatel | admin1 - https://www.youtube.com/watch?v=pFAJLwzxv0A&ab_channel=%E5%90%B4%E6%96%87%E7%9B%B8 | 15:34 |
| admin1 | this is the original skyline ( 2014 ) https://www.youtube.com/watch?v=Wo8YqIDBMxg | 15:35 |
| ierdem | admin1, you are right, i saw cloudvps's dashboard too and it seems the same. You can check on this video https://www.youtube.com/watch?v=pFAJLwzxv0A | 15:35 |
| admin1 | this looks new and fresh :D | 15:36 |
| spatel | jrosser hmm so this patch doesn't need if i upgrade from 10.5.6 -> 10.5.12 or something.. right? | 15:36 |
| ierdem | admin1, oops, i changed my mind, these are not the same but similar | 15:36 |
| jrosser | spatel: W is released with 10.5.9, thats how it is | 15:37 |
| jrosser | on master, as usual, we will upgrade during the dev cycle for the X release | 15:37 |
| jrosser | the versions pretty much get fixed at release, and should be bumped up early in the next cycle | 15:38 |
| jrosser | thats what those patches are | 15:38 |
| spatel | okay! currently i am running 22.1.0 so i will first move to 22.2.0 and then do 23.1.0 (assuming in week we will have that) | 15:39 |
| jrosser | if you want to upgrade galera to some other version to avoid the bug in 10.5.9 thats fine, just override the version vars | 15:40 |
| spatel | oh! so i can directly go to 10.5.6 -> 10.5.12 rght? | 15:40 |
| jrosser | i would think so, +/- testing it | 15:41 |
| spatel | let me setup AIO and give it a try | 15:41 |
| jrosser | we were just super unlucky with 10.5.8 and 10.5.9, plus it took longer than the W release for 10.5.10 which we never tried | 15:42 |
| spatel | hmm, so when are we going to cut 23.1.0 (i know couple neutron patch is stuck but lets move quickly, i want to deploy my OVN on bigger lab to run some benchmarking ) | 15:44 |
| jrosser | i cant merge my own patches :) | 15:46 |
| spatel | calico is painful.. i would say set it NV because its going to hurt every single neutron patch in future (until we know what is going on) | 15:46 |
| jrosser | needs reviews | 15:46 |
| jrosser | someone needs to fix the calico NV patch now it's on focal | 15:46 |
| spatel | make me reviewer :) | 15:46 |
| spatel | I did deploy calico in lab and hit that bug but when second time i run tempest it works so very odd.. (no logs saying something is wrong, its just nova not able to bind port first time run but next run it was) | 15:47 |
| spatel | so very flaky | 15:48 |
| spatel | I gotta go.. will see you online in 30 min | 15:49 |
| admin1 | hi jrosser, do you see any variable missing for haproxy here: https://pastebin.com/CUyqd3Lt .. i find it not including the services | 17:12 |
| admin1 | stange find | 17:20 |
| admin1 | if i remove my overrides, it works | 17:20 |
| jrosser | I’m confused, you put “ERROR” in your paste but there’s no error there | 17:20 |
| admin1 | when i removed my rgw and redis override, it now regenerated all the endpoints | 17:21 |
| admin1 | i meant the missing ones - gnocchi, etc | 17:21 |
| jrosser | what did you override? | 17:22 |
| admin1 | when i #haproxy_service_configs -- this whole block, then it finally generated the configs for the added services | 17:22 |
| jrosser | haproxy_service_configs? | 17:23 |
| admin1 | yeah | 17:23 |
| admin1 | which is just 2 additional service i added .. for ceph rgw and redis | 17:24 |
| admin1 | this is prod .. i need to replicate this in test and observe tis again | 17:25 |
| jrosser | did you look at how haproxy_service_configs is used? | 17:26 |
| admin1 | setup with x servics, add a haproxy_service_config block, ad new services, see if haproxy adds those new services or not .. if it does not add, # the haproxy_service_config block and see if it discovers it .. | 17:26 |
| jrosser | which release? | 17:27 |
| admin1 | 22.2.0 | 17:27 |
| jrosser | victoria? | 17:27 |
| admin1 | yea | 17:27 |
| jrosser | you looked in the haproxy_server defaults/main.yml then overrode haproxy_service_configs | 17:28 |
| jrosser | my question really is did you look how haproxy_service_configs is used in the openstack-ansible repo? | 17:28 |
| jrosser | specifically this https://github.com/openstack/openstack-ansible/blob/bfc14c5d54cc80952fe1564644849a37ec86bb13/playbooks/haproxy-install.yml#L57-L58 | 17:29 |
| admin1 | it takes me a while to understand these .. | 17:29 |
| jrosser | sure - but this is kind of important to use OSA effectively | 17:30 |
| jrosser | to know how the wiring up of the roles works | 17:30 |
| jrosser | if you override haproxy_service_configs then you delete all the standard haproxy endpoint config from here https://github.com/openstack/openstack-ansible/blob/stable/victoria/inventory/group_vars/haproxy/haproxy.yml#L549 | 17:31 |
| jrosser | and you also delete anything in haproxy_extra_services from being deployed | 17:31 |
| jrosser | so that totally explains why you never got the gnocchi / ceilometer endpoints | 17:31 |
| jrosser | becasue there were no longer in the data passed to the haproxy role, only redis and rgw | 17:32 |
| jrosser | to do this properly you let OSA define all the standard builtin endpoints with haproxy_default_services, you supply your own extra ones in haproxy_extra_services | 17:33 |
| jrosser | those two vars get combined in the playbook into one var haproxy_service_configs, which the role then deploys | 17:33 |
| admin1 | so instead of using haproxy_service_configs variable ( which overrode the default) i should use the haproxy_extra_services: | 17:34 |
| jrosser | to supply your own additional endpoints, yes | 17:34 |
| admin1 | do you know how i can bind my own address .. that i never could find | 17:35 |
| jrosser | you have an extra external IP on the haproxy node? | 17:36 |
| admin1 | i can add one | 17:36 |
| admin1 | oh .. i was thinking SNI | 17:36 |
| admin1 | and not this extra ip way | 17:36 |
| jrosser | right, i figured this wasnt a clear question | 17:36 |
| admin1 | do i need to add another external ip again on the vip .. haproxy does support SNI .. so i supposed it can bind on the same IP , but on 2 different external domains | 17:37 |
| jrosser | yes so haproxy can do all these things | 17:37 |
| jrosser | haproxy_server template for the config file maybe can, maybe can't | 17:38 |
| jrosser | i don't really have an answer for you | 17:39 |
| admin1 | no problem | 17:39 |
| jrosser | becasue haproxy is very flexible | 17:39 |
| jrosser | the haproxy template is very flexible | 17:39 |
| admin1 | i will try to figure it out .. | 17:39 |
| jrosser | you have choices of another IP, or SNI or whatever | 17:39 |
| jrosser | so find the intersection of what you want with what the haproxy template can generate | 17:39 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!