Monday, 2023-04-24

noonedeadpunkpsymin[m]: I think this paste https://paste.opendev.org/show/bXyDEtCmnmpUpwsRBejt/ is result of failed/skipped utility-install.yml playbook07:33
noonedeadpunkSo I'd suggest re-running it to see if everything is good there07:34
noonedeadpunkmornings07:34
noonedeadpunkdoh... spice5-html gone from github....07:41
jrossergood morning07:57
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Switch spice-html5 source to freedesktop gitlab  https://review.opendev.org/c/openstack/openstack-ansible/+/88133607:58
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Switch spice-html5 source to freedesktop gitlab  https://review.opendev.org/c/openstack/openstack-ansible/+/88133707:59
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Switch spice-html5 source to freedesktop gitlab  https://review.opendev.org/c/openstack/openstack-ansible/+/88131708:00
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Switch spice-html5 source to freedesktop gitlab  https://review.opendev.org/c/openstack/openstack-ansible/+/88131808:00
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Bump OpenStack-Ansible Zed  https://review.opendev.org/c/openstack/openstack-ansible/+/88133808:02
jrosseris this a temporary thing https://review.opendev.org/c/openstack/openstack-ansible/+/88004708:19
jrosseri.e we need to do more work later to fix up the resource creation08:20
noonedeadpunkjrosser: well. We should move to this common role for resource creation I'd say08:22
noonedeadpunkBut not sure if it will solve a chicken-egg we have today08:23
jrosserthe cluster should work without the FIP - just not accessible from outside08:23
noonedeadpunkit's only for aio setup that's mainly for CI as well08:27
noonedeadpunkAlso not sure it was working previously to be frank08:27
noonedeadpunkIt sounds like collection was buggy enough to do smth stupid that gerenally worked, maybe same way...08:28
hux[m]Hallo, I'm looking into patching Openstack to a new release however I worry a bit regarding octavia and it's cert, it looks like the CA are regenerated every time? https://github.com/elastx/openstack-ansible-os_octavia/blob/stable/xena/tasks/octavia_certs.yml#L61-L77 is this by design or do I miss anything?10:08
hux[m]s/elastx/openstack/10:09
hux[m]For extra info i found the following bug report that seems to indicate that this breaks existing loadbalancers: https://bugzilla.redhat.com/show_bug.cgi?id=164553610:11
damiandabrowskihux: openssl_privatekey should not override already existing file(until you pass force=true)10:14
damiandabrowskihttps://docs.ansible.com/ansible/latest/collections/community/crypto/openssl_privatekey_module.html#parameter-force10:14
hux[m]Ahh that makes sense, thanks alot for your swift response :)10:15
damiandabrowskinp ;) there's one more thing worth mentioning10:18
damiandabrowskiif private key doesn't have 600 permissions, it will be overridden by openssl_privatekey module10:18
damiandabrowskiit may happen if you store private key in git, as git doesn't preserve permissions10:19
damiandabrowskiwe fixed that but probably only in PKI role: https://review.opendev.org/c/openstack/ansible-role-pki/+/86755310:19
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_blazar master: Add uWSGI support to blazar  https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/88065110:48
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_blazar master: Add TLS support to blazar backends  https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/88065210:48
opendevreviewMerged openstack/openstack-ansible master: Disable floating IP usage in magnum_cluster_templates  https://review.opendev.org/c/openstack/openstack-ansible/+/88004710:53
jrosserhux[m]: you have the Xena branch there - if you are upgrading to a later release then you should look carefully at how the certificate handling has been changed10:54
jrossernoonedeadpunk: this is why my network node disks are filling https://review.opendev.org/c/openstack/neutron/+/87127411:17
jrosserno idea why this only started to fill the disk in Zed - it's clearly visible on our monitoring that was the point it started11:18
jrosserbut that code has been there for loooong time11:18
noonedeadpunkWell, if it used syslog as specified, it would end up in journald...11:19
jrosserwell the sad thing is, right now it is in it's own log file, in syslog and also in the journal11:21
jrosserso 3x11:22
*** dviroel__ is now known as dviroel12:04
* NeilHanlon sighs13:21
NeilHanlonkeystone not liking me this morning13:21
NeilHanlonhttps://rpa.st/YI5A13:21
NeilHanlonbtw psymin[m]: https://git.shrug.pw/linuxadminbooks/AdvancedLinuxAdmin/src/branch/main/ansible/playbooks/tasks/setup-network.yml https://git.shrug.pw/linuxadminbooks/AdvancedLinuxAdmin/src/branch/main/ansible/playbooks/vars/common-network.yml13:24
NeilHanlonthese might be helpful to you13:24
mgariepylooking at this i do prefer renaming my interface :D so the mgmt bridge can be on the same interface everywhere :D no matter what is the pci slot or the drive used. 13:28
NeilHanlonyeah, my lab is in the cloud so sometimes they get assigned... oddly13:29
opendevreviewMerged openstack/openstack-ansible stable/zed: Disable floating IP usage in magnum_cluster_templates  https://review.opendev.org/c/openstack/openstack-ansible/+/88108913:29
NeilHanlonthat or my terraform code is bad, whhich is also likely13:29
mgariepyhehe :D13:29
NeilHanlonwhen you have real servers and can just plug the right darn cable in, that's easier ;) 13:30
mgariepyyeah but different servers model have different layout sometimes 13:31
NeilHanlonthat's true, yeah. i was lucky at my last job that we had standard racks and everything worked nicely lol13:32
opendevreviewMerged openstack/openstack-ansible stable/yoga: Disable floating IP usage in magnum_cluster_templates  https://review.opendev.org/c/openstack/openstack-ansible/+/88108813:38
cloudnull☕ ⏲️14:05
noonedeadpunko/14:12
psymin\o14:13
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Switch default provider to amphorav2  https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/88138214:16
NeilHanlonnoonedeadpunk: ever seen this? https://rpa.st/YI5A14:16
noonedeadpunkNeilHanlon: not really...14:17
noonedeadpunkworth dropping no_log to check on that...14:17
noonedeadpunkOr try to execute command manually14:18
NeilHanlonyea, that was my next step...14:20
NeilHanlondebating just tearing it down and starting over to see if it goes away :P 14:20
psyminI assume my target container here is just having issues talking to pypy.org?  https://paste.opendev.org/show/btGxQS7tFllCVrClMgbL/14:21
psyminNeilHanlon, setting no_log to false would probably give some useful insight14:22
noonedeadpunkNeilHanlon: well, I'd guess that smth could be wrong with DB for example...14:22
NeilHanlonnoonedeadpunk: that was my thought as well so I tried recreating the lxc containers 14:23
NeilHanlonw/ lxc_container_recreate 14:23
noonedeadpunkLike haproxy marking all backends as down due to weird src ip towards mgmt network14:23
noonedeadpunk(like when VIP is added as /24 instead of /32)14:23
NeilHanlonokay that leads to my next question which is possibly related... i thought in the past, OSA took care of adding the internal/external VIPs to the system14:24
NeilHanlonbut that didn't happen for this install 14:24
* NeilHanlon needs to look at logs14:24
NeilHanlonheh14:26
NeilHanlonthis was my fault 🙃14:26
NeilHanlonThe task includes an option with an undefined variable. The error was: {{ keystone_service_publicuri_proto }}://{{ external_lb_vip_address }}:{{ keystone_service_port }}: {{ bootstrap_host_public_address | default(ansible_facts['default_ipv4']['address']) }}14:26
psyminhmm, the baremetal OS can curl https://pypi.org/simple/pip/ but myplacement container can't15:06
NeilHanlonthe repo container is supposed to handle that communication, I believe15:07
psymincool, repo container can curl it15:10
NeilHanlonyeah, so in OSA, the repo container handles downloading and compiling stuff, and serving it to other containers15:10
psyminnice15:12
jrossereth0 on all the containers is the default route15:19
jrosserso if you can't do normal things like use the package manager or curl in the containers then something might be broken15:19
jrosserpsymin: ^^15:19
psyminI can use the package manager, I can tcptraceroute to the endpoint on 443.  But can't curl the url from that container.15:21
jrosserthe repo container is only used for the things that are not downloaded, like python wheels built for specific git SHA, or where prebuilt wheels are not available from pypi that satisfy what the constraints file wants15:21
jrosserthen i think that needs some debugging, there is dnsmasq running on the host that organises all that15:21
psyminI assume that the placement container should be attempting to update its version of pip?15:22
jrosserevery time a python virtualenv is built the version of pip/setuptools/wheel is made to be a specific version15:23
psyminit looks like the pip version on this container is "pip 22.0.2" and pip is complaining that "The user requested (constraint) pip==22.1.2"15:25
psyminmy assumption is that it is attempting to update pip to 22.1.2 via pypi.org and failing?15:26
jrosserit looks like that15:26
jrosseryou could do some manual debugging in one of the containers15:26
jrossercreate a virtualenv and the command line, activate it and try `pip install pip==22.1.2`15:27
jrosserthe thing is that the version of pip you see on the container is the one that comes with the operating system package manager15:28
jrosserand the only thing thats really used for is to boostrap things sufficiently that a virtualenv can be created and then the actual versions required to be installed15:28
jrosserit is very important not to mix up python things that apt/dnf install with things that pip installs15:29
psyminwhat might cause placement to not be happy with the pip rule, but repo to be fine with it?15:34
psyminrule == url, typo15:34
noonedeadpunkpsymin: can you share `ip a; ip r` output?15:35
psyminfrom the containers?15:35
noonedeadpunkyep15:36
psyminhttps://paste.opendev.org/show/bGYGhUpPWxXUukioU2Jv/15:38
BobZAnnapolisAny known issues using the latest "stable" version of OA and setting up multi-node (evaluation) installation on AWS ? tia15:40
noonedeadpunkand what's in `iptables -L -t nat` from metal host?15:40
noonedeadpunkBobZAnnapolis: um, probably not?15:41
psyminmy mtu kludge wasn't on an interface, fixed and it works now.  Looking forward to having upstream not need it.  Will redeploy after this "works"15:41
psymin(by upstream I mean the router I don't have access to at the moment)15:42
noonedeadpunkpsymin: you can define MTU in `container_networks`15:43
noonedeadpunkie https://opendev.org/openstack/openstack-ansible/src/branch/master/etc/openstack_deploy/openstack_user_config.yml.example#L270-L28015:44
noonedeadpunkAh, though you need for eth015:44
noonedeadpunkpsymin: I think you can use `lxc_container_default_mtu`15:48
jrossera good tool to understand what/where that is would be https://codesearch.opendev.org/?q=lxc_container_default_mtu15:49
psyminthanks!15:51
opendevreviewMerged openstack/openstack-ansible-os_blazar master: Fix blazar_service_type  https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/88065016:22
opendevreviewMerged openstack/openstack-ansible-os_glance master: Move _glance_available_stores to defaults  https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/88087217:19
psyminI removed my containers and went through the playbooks again.  The first two succeed, but setup-openstack.yml fails again like yesterday.  https://paste.opendev.org/show/bDnJmUxzSMF0ad9pxbPR/  Yesterday I put in a kludge to manually set ansible_python_interpreter to "/usr/bin/python3" in "/etc/ansible/ansible_collections/openstack/osa/roles/db_setup/tasks/main.yml"  is there a better way to do this?17:25
psyminUbuntu 22.04.2 / Zed17:27
psymincripes, might be user error yet again :)17:27
psyminyep, I'm past it :D17:29
noonedeadpunkok, good to hear :)17:43
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Reduce amount of task that are executed  https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/88139717:47
noonedeadpunkWonder if that has any potential to pass ^17:48
noonedeadpunkalso, we might want to review if we need to have local facts on venv installation at all17:48
noonedeadpunkas we might set them but never use17:48
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Delegate compute wait tasks to service_setup_host  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/88013917:51
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Use include instead of import for conditional tasks  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/87494717:52
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Add way to periodically trim Cinder DB  https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/88097517:53
psymindefinitely making progresss :) https://paste.opendev.org/show/bDQTHomcNdg7jW3neh8l/18:16
noonedeadpunkpsymin: Is gluster healthy on repo hosts?18:23
noonedeadpunkalso wonder why wheels were not built18:24
noonedeadpunkor they were and it's indeed a gluster thingy18:24
noonedeadpunkbtw, cores, I've hold on release patches to land this https://review.opendev.org/c/openstack/openstack-ansible/+/88133618:25
noonedeadpunk(and backports)18:25
psymingluster is running on repo container, not sure how to test if it is healthy yet18:27
psyminlooks like it is active and running, I do see an error in the log18:28
jrosserrelease schdule looks wrong for antelope cycle trailing projects https://releases.openstack.org/bobcat/schedule.html18:31
jrosserantelope/bobcat typo there i think18:31
psyminthe path /var/www/repo/os-releases/26.1.0 exists on the repo container, but not "ubuntu-22.04-x86_64/requirements/neutron-26.1.0-constraints.txt"  hmm, I also don't have an ubuntu container like I did before wiping out the containers and redeploying18:37
NeilHanlonannoying re: spice, noonedeadpunk.. but, +1 from me18:38
jrosserpsymin: the netutron wheel build should have been earlier in the log18:42
psyminjrosser, https://paste.opendev.org/show/blMvNJISw4uIKj944Lad/ 18:53
psyminlooks like it skipped the "Upgrade pip/setuptools/wheel to the versions we want" section this time for infra1_neutron_server_container-a831628d18:54
jrosserwell it's skipped the whole wheel build entirely18:55
psyminwhere should I be digging to find the cause?18:56
jrosserprobably here https://github.com/openstack/ansible-role-python_venv_build/blob/stable/zed/vars/main.yml#L8018:57
psyminlooks like it is a boolean19:02
jrosseryes, you have a kind of small deployment where that is going to be false a bunch of the time19:03
jrosserand for some reason when neutron is deployed to the compute node there is an assumption that the wheels are available19:03
jrosseryou can try putting `venv_wheel_build_enable: true` in your variables19:04
jrosserbut tbh i am a little too tired to know why it's doing this19:04
opendevreviewMerged openstack/openstack-ansible master: Switch spice-html5 source to freedesktop gitlab  https://review.opendev.org/c/openstack/openstack-ansible/+/88133621:14
psymindefinitely making progress, I need to find which variable is undefined :) https://paste.opendev.org/show/bQAWjUUFW84ReaWKqowI/21:41
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_adjutant master: Add TLS support to adjutant backends  https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/88114721:57
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_adjutant master: Add TLS support to adjutant backends  https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/88114721:58
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_aodh master: Add TLS support to aodh backends  https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/88039521:58
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_barbican master: Add TLS support to barbican backends  https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/87991721:59
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_blazar master: Add TLS support to blazar backends  https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/88065222:00
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_cinder master: Add TLS support to cinder backends  https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/87496622:01
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_cloudkitty master: Add TLS support to cloudkitty backends  https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/88054822:01
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_designate master: Add TLS support to designate backends  https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/88039722:01
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_glance master: Add TLS support to glance backends  https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/82101122:01
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_gnocchi master: Add TLS support to gnocchi backends  https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/88054922:01
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_heat master: Add TLS support to heat backends  https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/87991622:02
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_horizon master: Add PKI support to horizon backends  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/87951722:02
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_ironic master: Add TLS support to ironic backends  https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/88039422:03
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_magnum master: Add TLS support to magnum backends  https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/87991822:04
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_manila master: Add TLS support to manila backends  https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/88055122:04
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_masakari master: Add TLS support to masakari backends  https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/88039822:04
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_mistral master: Add TLS support to mistral backends  https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/88055022:05
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_murano master: Add TLS support to murano backends  https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/88064622:05
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_neutron master: Add TLS support to neutron_server backends  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/87365422:05
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_octavia master: Add TLS support to octavia backends  https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/88068022:05
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_placement master: Add TLS support to placement backends  https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/87938022:05
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_sahara master: Add TLS support to sahara backends  https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/88064322:06
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_senlin master: Add TLS support to senlin backends  https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/88055422:06
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_swift master: Add TLS support to swift backends  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/88039622:06
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_tacker master: Add TLS support to tacker backends  https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/88055522:06
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_trove master: Add TLS support to trove backends  https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/88055222:06
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_zun master: Add TLS support to zun backends  https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/88114022:07
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-repo_server master: Add TLS support to repo_server backends  https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/87642922:08
NeilHanlonpsymin: this is the relevant code you're reaching. I suspect your openstack_user_config.yml for bridge mappings needs adjustment https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/tasks/providers/setup_ovs_ovn.yml#L55-L6522:48
psyminNeilHanlon, I've added some configuration in /etc/ansible/roles/os_neutron/defaults/main.yml for neutron_provider_networks and added one line to Setup Network Provider Bridges "- bridge_mapping.split(':') | length >= 2"  We'll see if I've really messed things up shortly :D22:49
NeilHanlonthat shouldn't be needed.. all of that is passed through from your openstack_user_config.yml22:50
psyminthe one line was added to /etc/ansible/roles/os_neutron/tasks/providers/setup_ovs_ovn.yml22:50
psyminoh good, I'll move stuff like that back to the openstack_user_config.html after this fails22:51
NeilHanlonyeah, the network_mappings, for example, come from global_overrides: provider_networks:, iirc22:54
psyminI suspect the bridge_mapping.split issue could be related to not having the neutron_provider_networks: stuff set22:55
psyminis openstack_user_config.yml where I should be setting lxc_container_default_mtu: "1464" ?22:59
psyminas well as  _venv_wheel_build_enable: "true" ?22:59
psymin[m]whoa,  "[Playbook execution success]"23:04
NeilHanlonpsymin: check out https://docs.openstack.org/openstack-ansible/latest/reference/configuration/using-overrides.html#user-overrides23:05
NeilHanlonespecially the 'user variables' section23:06
psyminexcited to have the playbook complete at least, horizon interface allows me to log in but then throws an error "Something went wrong!" .. progress! :D23:08
NeilHanlonprogress indeed :) haproxy logs will help you there23:09

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!