| *** ysandeep|out is now known as ysandeep | 01:50 | |
| *** ysandeep is now known as ysandeep|afk | 02:57 | |
| *** ysandeep|afk is now known as ysandeep | 03:10 | |
| *** ysandeep is now known as ysandeep|away | 03:25 | |
| Banister1640 | I'm having trouble running the setup-openstack playbook, I'm running into: fatal: [aio1_aodh_container-ed8abc4e]: FAILED! => {"changed": false, "msg": "The python packages have failed to install, please check the log file located at /var/log/python_venv_build.log for more information.\n"} | 03:54 |
|---|---|---|
| Banister1640 | the log I am getting is https://pastebin.com/dgiZn0sF | 03:54 |
| Banister1640 | if anybody has any help or pointers it would be much appreciated | 03:55 |
| snadge | ive seen this same error actually too.. you might find the people who can help are in a timezone that may respond several hours from now (europe) | 04:05 |
| snadge | i have just ignored it for now and am trying to fix an unrelated cinder issue.. i've never used NetAPP iscsi before, and ive plugged the right information in and its authenticating.. but openstack is apparently just sending GET / to the api server on port 443 which returns 400 bad request | 04:05 |
| snadge | so im hoping and praying that someone else happens to use a netapp with iscsi and knows the missing magic ingredient that i haven't been able to find | 04:11 |
| snadge | the specific hardware is DM5100F | 04:12 |
| *** ysandeep|away is now known as ysandeep | 04:13 | |
| Banister1640 | Ah of course, time zones. I have some netapp hardware too I hope to use later, although it's a lot less nice, and a lot older. I'm setting this up for myself in my homelab. First time using openstack | 04:17 |
| snadge | i have faith that a solution will be found as it is incredibly close to just working, it authenticates correctly etc you can expose volumes with curl from the command line | 05:02 |
| snadge | curl -X GET -u admin:notmypassword -k 'https://10.150.25.someip/api/storage/volumes' | 05:07 |
| snadge | password and last quad substituted but you get the idea | 05:07 |
| snadge | cinder is just sending GET / .. which causes the 400 to come back, instead of something meaningful like /api/etc | 05:10 |
| *** ysandeep is now known as ysandeep|afk | 06:04 | |
| *** ysandeep|afk is now known as ysandeep | 06:45 | |
| snadge | i wish i could just patch python to ignore the ssl certificate because i dont have time to set it up properly | 06:46 |
| snadge | ive at least got it to a point now where it refuses to touch the self signed cert, its too much documentation to fix that properly now and i want to come back to it later | 06:47 |
| noonedeadpunk | Banister1640: sounds like quite valid bug for aodh | 07:31 |
| snadge | turns out its actually easier to turn on http api.. and that worked, other admin will sort out certificate authority etc later | 07:32 |
| snadge | so now it is just this aodh bug that is remaining yes | 07:33 |
| noonedeadpunk | I would need to spawn AIO to catch the issue to see what's actually wrong | 07:33 |
| noonedeadpunk | as not sure where this requirement does come from | 07:34 |
| snadge | i have the time to assist and debug this issue potentially.. since im at a point where i have a failing aodh playbook.. so i could be remote hands or whatever, i typically just poke at ansible.log like a caveman and make grunting sounds | 07:34 |
| noonedeadpunk | ok, I think I've catched the error | 07:50 |
| noonedeadpunk | and I have by far no idea wtf | 07:57 |
| jrosser_ | morning | 07:57 |
| noonedeadpunk | seems it's aodh itself that's broken with our sha | 08:00 |
| noonedeadpunk | Banister1640: snadge: try defining `aodh_git_install_branch: 14.0.0` | 08:07 |
| noonedeadpunk | seems they've backported smth breaking to yoga | 08:08 |
| noonedeadpunk | and feel free to fil a bug to aodh about that :) | 08:10 |
| snadge | if there is an offtopic place where i could discuss why im sadly not super motivated to do that, i would | 08:12 |
| snadge | for 3 years i was basically the openstack guy, which is funny given how some of the basic level of things i've needed help with | 08:16 |
| snadge | but sadly my contract has come to an end, not for performance, but purely due to financial crunch reasons | 08:16 |
| snadge | so this yoga install is one last gift to them | 08:17 |
| snadge | i made sure the full timer understands all this stuff though and literally sat with him to do this last install, so it is in good hands | 08:19 |
| snadge | so i will be honest.. i am a linux systems admin, who now has experience installing and using openstack.. and i thank you guys sincerely for that privilege it has been a pleasure | 08:20 |
| *** ysandeep is now known as ysandeep|lunch | 08:21 | |
| noonedeadpunk | well, it's never late to become openstack engineer for linux admin guy ;) | 08:21 |
| snadge | thats right i understand what all those words mean which puts me way ahead of just other linux guys right | 08:21 |
| snadge | and i've run all those playbooks before many many times.. you guys know how it goes | 08:22 |
| snadge | without the help i've received i may have had many sleepless nights profusely sweating | 08:23 |
| snadge | but instead they have been really happy with the first two installs ive done and been using it solid for that whole time | 08:26 |
| snadge | im glad i got the experience with aio, then a vsphere test cluster and a single controller multi compute cluster install | 08:27 |
| snadge | i can then build on that with more complicated multi controller vastly more split out larger installations with seperate logging servers etc | 08:29 |
| jrosser_ | good to hear it's worked out | 08:33 |
| jrosser_ | noonedeadpunk: https://docs.openstack.org/neutron/latest/admin/config-experimental-framework.html | 09:02 |
| jrosser_ | i wonder if we need some of that | 09:02 |
| noonedeadpunk | it;s likely time for us to switch somewhere as default.... | 09:04 |
| noonedeadpunk | but yes, we do need that indeed | 09:04 |
| snadge | i have a british passport, so if there is openstack work going i wont say no ;) | 09:31 |
| snadge | i can be linux guy with openstack experience keen to gain more, happy to receive intermediate pay | 09:33 |
| *** tosky_ is now known as tosky | 09:33 | |
| *** ysandeep|lunch is now known as ysandeep | 10:12 | |
| *** ysandeep is now known as ysandeep|afk | 10:55 | |
| *** dviroel|out is now known as dviroel | 11:31 | |
| *** ysandeep|afk is now known as ysandeep | 11:45 | |
| jrosser_ | noonedeadpunk: do you set `public_endpoint` in your keystone.conf? | 13:49 |
| noonedeadpunk | snadge: you can check out https://cleura.uhigher.com/en/jobs?v=28 :) | 13:55 |
| noonedeadpunk | jrosser_: um, I don't have that in overrides at least | 13:55 |
| noonedeadpunk | let me check | 13:56 |
| jrosser_ | when i visit https://my-url:5000/v3 it returns an http:// location for keystone rather than https:// | 13:57 |
| jrosser_ | which was surprising - and it appears to be driven from here https://github.com/openstack/keystone/blob/master/keystone/server/flask/request_processing/middleware/auth_context.py#L73-L95 | 13:58 |
| noonedeadpunk | I do recall smth like that actually | 14:01 |
| noonedeadpunk | but I don't think it was any issue for quite a while | 14:02 |
| noonedeadpunk | or well, I totally do recall magnum or smth was not respecting that | 14:03 |
| noonedeadpunk | but keystone is okeyish for quite a while | 14:04 |
| jrosser_ | yes there was something with magnum | 14:04 |
| noonedeadpunk | answering your question - no I don't have public_endpoint in keystone.conf | 14:06 |
| jrosser_ | my master aio is doing the same, returning http://ip-addr:5000/v3 | 14:06 |
| noonedeadpunk | I think, it depends on haproxy? | 14:07 |
| jrosser_ | we had a strange case with some terraform trying to use http url and this is the only place i can find it | 14:07 |
| jrosser_ | not sure, as if i go in the keystone container and wget the service it's the same response there | 14:07 |
| jrosser_ | i beleive this is all generated in keystone | 14:08 |
| jrosser_ | its from here https://github.com/openstack/keystone/blob/master/keystone/api/discovery.py#L60 | 14:08 |
| noonedeadpunk | So, I have xena sandbox I'm working on, it's quite default and curl is correct | 14:08 |
| jrosser_ | it gives https:// ? | 14:09 |
| noonedeadpunk | https://paste.openstack.org/show/bOUs54ReElh0mmrpwipe/ | 14:09 |
| jrosser_ | thats really interesting as we did yoga upgrade over the last couple of days | 14:09 |
| jrosser_ | exactly when this terraform then blew up | 14:09 |
| noonedeadpunk | well, we don't have yoga anywhere yet.... | 14:10 |
| noonedeadpunk | I bet we already patched smth like that before.... | 14:10 |
| jrosser_ | andrewbonney: ^ another breadcrumb pointing to something changing X->Y | 14:11 |
| spatel | folks, i am upgrading openstack from wallaby to Xena and encounter here during rabbitmq/galera upgrade step - https://paste.opendev.org/show/bDBsAnLkn6y37K2bA0Si/ | 14:11 |
| spatel | I am in meeting so will look into it later but just wanted to mention here | 14:11 |
| noonedeadpunk | jrosser_: I bet it was based on some headers passed to the wsgi by haproxy or smth like that. At least when there was magnum story or smth | 14:14 |
| jrosser_ | i think with magnum it took the internal URL and passed that out the the cluster nodes, when they really needed the external url | 14:14 |
| noonedeadpunk | so it was taken smth like http refferer ort smth like that | 14:15 |
| noonedeadpunk | question - was you upgrading also ubuntu to 22.04? | 14:15 |
| noonedeadpunk | as haproxy version could influence then | 14:16 |
| jrosser_ | no, we're on focal still | 14:16 |
| noonedeadpunk | well, on yoga aio I indeed see http instead of https | 14:17 |
| noonedeadpunk | like it's not respecting X-Forwarded-Proto anymore | 14:17 |
| noonedeadpunk | And I bet we defined it for this specific reason | 14:17 |
| jrosser_ | https://github.com/openstack/openstack-ansible-os_keystone/commit/6fae2bdade459c85dba55aae64c9f6f4e485a782 | 14:20 |
| jrosser_ | {% if (keystone_ssl | bool) and (keystone_external_ssl | bool) %} | 14:20 |
| jrosser_ | ^ "and", or "or" ? | 14:20 |
| noonedeadpunk | I can recally ou already asked that | 14:21 |
| jrosser_ | yes i think i did | 14:21 |
| jrosser_ | it was broken for when i tried out skyline | 14:21 |
| noonedeadpunk | I totally can recall that these variables were not what you think they are | 14:22 |
| noonedeadpunk | So idea was to set RequestHeader only when we terminate on haproxy and connection between haproxy and keystone is not encrypted | 14:23 |
| noonedeadpunk | or smth like that | 14:23 |
| noonedeadpunk | because keystone_ssl was meaning if keystone backend is serving SSL itself | 14:23 |
| noonedeadpunk | and if it does, you don't need to set keystone_secure_proxy_ssl_header as it's already https | 14:24 |
| noonedeadpunk | So yes, I think logic is incorrect indeed | 14:26 |
| noonedeadpunk | Sorry, I need to leave now, if you won;t sort it out I will take a deeper look later today | 14:29 |
| jrosser_ | no problem | 14:29 |
| jrosser_ | we can take a look | 14:30 |
| noonedeadpunk | though it won't explain IP insteasd of fqdn | 14:30 |
| jrosser_ | oh thats from an AIO though? so expected | 14:30 |
| noonedeadpunk | ah, ok | 14:31 |
| jrosser_ | anyway would be great to have a second opinion on it if you have a moment later too | 14:31 |
| opendevreview | Jean-Philippe Evrard proposed openstack/openstack-ansible-openstack_hosts master: Define coherent safe default for package state https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/852567 | 14:49 |
| mrf | Hi! | 14:54 |
| mrf | Designate in yml is designate_hosts: ? | 14:54 |
| jrosser_ | mrf: https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/conf.d/designate.yml.example | 14:59 |
| jrosser_ | lots of example config fragments in that directory | 14:59 |
| mrf | and them i can run the setup-everything --limit="dnsaas_hosts" ?? | 15:01 |
| jrosser_ | not quite | 15:04 |
| jrosser_ | you'd first create the containers, and you can --limit to something like "infra_all,localhost" perhaps there | 15:05 |
| jrosser_ | then run playbooks/haproxy_install.yml | 15:05 |
| jrosser_ | then playbooks/os-designate-install.yml | 15:05 |
| jrosser_ | then playbooks/os-horizon-install.yml to get the dashboard enabled | 15:05 |
| jrosser_ | of course you can run the playbooks/setup-<hosts/openstack>.yml but that will take really a very long time | 15:06 |
| mrf | running ! :) | 15:09 |
| mrf | lets check how designate works | 15:09 |
| mrf | and got sense for my deployment | 15:10 |
| jrosser_ | you'll need your own authoritiative dns servers | 15:15 |
| jrosser_ | bind9 or something set up so that designate can push records in | 15:15 |
| mrf | yes we got it but i want to check how customer see the dns record etc... | 15:20 |
| mrf | we got powerdns servers | 15:20 |
| *** ysandeep is now known as ysandeep|dinner | 15:48 | |
| spatel | noonedeadpunk jrosser_ https://paste.opendev.org/show/bzbzBDX8Wc0PxGlsnVLF/ | 15:51 |
| spatel | is keepalived password issue is a bug? | 15:51 |
| spatel | it doesn't like long password | 15:51 |
| spatel | if i set 8 char then it works | 15:53 |
| spatel | Keepalived v2.0.19 (10/19,2019) | 15:56 |
| *** dviroel is now known as dviroel|lunch | 16:04 | |
| *** ysandeep|dinner is now known as ysandeep|out | 16:18 | |
| jrosser_ | spatel: i blame evrardjp :) | 16:45 |
| jrosser_ | there is a change to the keepalived ansible role that we use https://github.com/evrardjp/ansible-keepalived/commit/6871592aa1e5839af671871ba9ddf5bc225e5a94 | 16:47 |
| jrosser_ | that makes it now validate the keepalived config, and as you've seen the passwords being longer than 8 chars is enough to make the validation fail | 16:47 |
| mgariepy | well if you don't validate it will only ignore the exeeding char.. just like some BMC's... | 16:49 |
| jrosser_ | sadly there is no override that we can set to disable the validation in the keepalived role | 16:51 |
| jrosser_ | spatel: the best thing you can do is edit /etc/openstack_deploy/user_secrets.yml and supply just the first 8 chars | 16:51 |
| *** dviroel|lunch is now known as dviroel | 16:55 | |
| spatel | sorry i was in meeting | 18:14 |
| spatel | why not we remove validation? | 18:14 |
| *** tosky_ is now known as tosky | 18:16 | |
| jrosser_ | becasue it is not an OSA role, it is one external maintained by evrardjp | 18:23 |
| jrosser_ | so this is now a problem for several of our stable branches, unfortunatley | 18:24 |
| *** dviroel is now known as dviroel|afk | 19:07 | |
| noonedeadpunk | damn.... We should likely rollback and fix keepalived role version for stable branches to avoid failure | 19:32 |
| noonedeadpunk | I did that at least for wallaby :) https://opendev.org/openstack/openstack-ansible/src/branch/stable/wallaby/ansible-role-requirements.yml#L44 | 19:32 |
| noonedeadpunk | clean forgot for xena bump :( | 19:33 |
| jrosser_ | we never run that role in ci either do we? | 19:33 |
| noonedeadpunk | I think we don't | 19:34 |
| noonedeadpunk | sorry, I will check keystone thing tomorrow though - just returned and quite exchausted | 19:34 |
| jrosser_ | i made a bug in LP btw, i'm not going to be able to look at it | 19:34 |
| jrosser_ | *today | 19:34 |
| jrosser_ | the VRRP key is just wrong length anywat | 19:34 |
| noonedeadpunk | ok, great, the we won't loose it | 19:34 |
| jrosser_ | we need to update the pw-gen script to make 8-char keys | 19:35 |
| noonedeadpunk | well, there're more breaking changes n the role from what I can recall | 19:35 |
| noonedeadpunk | we can slice string and pass only haproxy_keepalived_authentication_password[0:7] :D | 19:37 |
| noonedeadpunk | we still need to trim it for upgrades. Or just drop it. As password functionality in keepalived is quite broken and not working properly, and dropped at all in vrrp 3 | 19:38 |
| mgariepy | lol. so when the other guy see the password in user-secrets it's not a small 8 char one.. | 19:38 |
| mgariepy | hey i'll be on vacation for the next 2 weeks. | 19:39 |
| jrosser_ | yes i noticed we had more validation errors for vrrp3 | 19:43 |
| noonedeadpunk | yep, actually I'm too on vacation for next 2 weeks | 19:46 |
| mgariepy | nice :D take care | 19:46 |
| mgariepy | i'll be disconected from irc and work related stuff ;) haha | 19:47 |
| noonedeadpunk | presumably I wiil need to drive 4000km during them... so thanks ! :) | 19:47 |
| mgariepy | ouch | 19:47 |
| mgariepy | what are you doing the second week ? | 19:47 |
| noonedeadpunk | lol :D | 19:47 |
| jrosser_ | driving back? | 19:47 |
| noonedeadpunk | I guess trying to find car service | 19:47 |
| mgariepy | lol | 19:47 |
| jrosser_ | centos-9 is still broken, looks lie facebook mirror is corrupt | 19:49 |
| jrosser_ | i looked at kolla jobs and they just dont bother with anything except ubuntu | 19:49 |
| noonedeadpunk | Triplo is in connec with infra regarding it | 19:54 |
| noonedeadpunk | saw a ml | 19:54 |
| noonedeadpunk | I wonder if ubuntu will release 22.04.1 today or postpone for another week | 19:57 |
| snadge | that aodh workaround worked, cool | 21:23 |
| snadge | im glad they postponed because i had a bunch of really important stuff to do unrelated to ubuntu update breakage | 21:24 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!