Friday, 2020-09-04

*** cloudnull has joined #openstack-ansible00:02
*** jamesden_ has quit IRC00:23
*** jamesdenton has joined #openstack-ansible00:25
*** djhankb has quit IRC00:27
*** djhankb has joined #openstack-ansible00:27
*** jamesden_ has joined #openstack-ansible00:45
*** jamesdenton has quit IRC00:47
*** cshen has joined #openstack-ansible00:49
*** cshen has quit IRC00:53
*** gyee has quit IRC01:32
*** yolanda has quit IRC02:07
*** djhankb has quit IRC02:28
*** djhankb has joined #openstack-ansible02:29
*** cshen has joined #openstack-ansible02:49
*** cshen has quit IRC02:54
*** cshen has joined #openstack-ansible04:29
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-ansible04:33
*** cshen has quit IRC04:33
*** dave-mccowan has quit IRC04:42
*** shyamb has joined #openstack-ansible04:52
*** shyamb has quit IRC05:14
*** shyamb has joined #openstack-ansible05:21
*** shyam89 has joined #openstack-ansible05:24
*** shyamb has quit IRC05:27
BlackFXI am about ready to give up lol05:30
*** shyam89 has quit IRC05:49
jrosserBlackFX: there is a playbook playbooks/lxc-containers-destroy.yml05:49
jrosserBlackFX: did you try an all-in-one in the end?05:52
jrossernoonedeadpunk: senlin has unusual properties, like heavy use of openstacksdk inside the service itself, like here https://opendev.org/openstack/senlin/src/branch/master/senlin/drivers/sdk.py#L109-L12605:56
jrosserthere does not appear to be a way to put connection parameters for openstacksdk into the config file, so things we might usually template into a config file can't be set05:57
*** cshen has joined #openstack-ansible06:00
BlackFXYeah I did, that's how I worked out the indentation error.06:02
*** pcaruana has joined #openstack-ansible06:04
BlackFXI am gonna clean up all the containers and try a fresh run, I can't really see why it'd be upset. The only thing I am doing that isn't in one of the seplates is haveing 2 LVM backends for cinder.06:04
*** cshen has quit IRC06:05
BlackFX*templates06:05
*** pcaruana has quit IRC06:06
*** pcaruana has joined #openstack-ansible06:06
jrosserbeing able to completely clean the nodes is very useful for testing06:14
jrosserotherwise it's never clear if something is residual from previous things06:14
BlackFXdo I need to do anything other than the container destroy to clean up?06:15
jrosserthat will remove the containers06:15
*** pcaruana has quit IRC06:15
jrosserbut there is very much config done to the host itself06:15
jrosseri'm not sure you've shared the probem you currently have?06:15
BlackFXStill stuck with the galera cluster member issue when running setup-infrastructure06:16
jrossersorry i cannot find it in the scollback06:18
noonedeadpunkjrosser: I'm not sure about this tbh... as eventually it's passing creds https://opendev.org/openstack/senlin/src/branch/master/senlin/drivers/os/keystone_v3.py#L93-L9706:18
noonedeadpunkand creds are taken from config06:19
jrossernoonedeadpunk: ok sure, but endpoint, insecure flag......06:19
jrosseri found it all goes wrong in an AIO06:19
noonedeadpunkhm, that's true....06:20
jrosserstandard failure pattern of defaulting to public endpoint then failing on cert verify06:20
jrosserand no way apparently to configure that06:20
noonedeadpunklet's suggest patch to senlin?:)06:20
noonedeadpunkas you can pass endpoint to openstacksdk connect06:21
noonedeadpunk(not 100% about insecure)06:21
BlackFXTASK [galera_server : Fail when the host is not in galera_cluster_members] **********************************************************************************************06:21
BlackFXfatal: [infra1_utility_container-8d88b1ab]: FAILED! => {"changed": false, "msg": "The host infra1_utility_container-8d88b1ab must be in galera_cluster_members."}06:21
jrossernoonedeadpunk: in create_connection i hacked in     params['identity_interface']='internal'06:21
jrosserand then it started working with cli from utility container06:22
jrosseri was wondering if the normal OS_<...> environment vars in the systemd unit could be used, if the openstacksdk library reads those....06:23
noonedeadpunkwe also can kind of add `'interface': cfg.CONF.authentication.interface` here https://opendev.org/openstack/senlin/src/branch/master/senlin/drivers/os/keystone_v3.py#L12006:23
noonedeadpunkbut need to check if they're probably passing smth with kwargs...06:23
jrosserBlackFX: so that is the same problem from before, somehow the utility container is targetted as one of the galera hosts06:24
BlackFXYes, so I am just going to try destroying and regenerating the containers06:25
jrosserwell06:25
BlackFXbut I can see no reason for it to be behaving that way06:25
jrossercan you paste more of the context, just the error message itself is difficult to spot things06:26
jrosserplaybooks/galera_insall.yml and just paste the whole output to paste.openstack.org /06:26
BlackFXok06:27
jrosseri would suspect the deleting / re-creating the containers will behave the same, becasue the inventory information about which tasks run against which hosts, and ansible groups is all coming from outside those06:27
*** pcaruana has joined #openstack-ansible06:28
noonedeadpunkand verify can go there as well....06:28
jrosserthe whole code is interesting - not even sure it is using the normal keystone authtoken section at all right now06:30
noonedeadpunkit's using authenticate section I guess06:30
noonedeadpunkso yeah06:30
noonedeadpunkbut I like using sdk tbh06:30
jrosserfirst time i see the sdk used like this in a service, which is kind of cool06:31
jrosseryes06:31
noonedeadpunkif you was not going to, I'll try to make patch to senlin to cover this06:32
jrosserplease - i probably take short notice holiday next week so wont have opportunity to chase it06:32
jrosserwanted to get a test in with an AIO before that to look for issues - i already had someone ask for this kind of function here so very interested06:33
jrosserspatel patches in an AIO will deploy fine but then 'openstack cluster build info' in utility container will fail straight away with a stack trace06:34
jrosserthe template for the [authentication] section is currently wrong but i put comments on the review06:35
*** shyamb has joined #openstack-ansible06:35
BlackFXDoing that has given totally different errors: http://paste.openstack.org/show/797450/06:40
jrosserok so tht suggests ansible cannot connect to your loadblanacer node at 192.168.2.206:43
jrosserhmm maybe not quite....06:45
jrosserBlackFX: here is the task that fails https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/common-tasks/haproxy-endpoint-manage.yml#L1606:48
jrosserand it looks like 'failed to connect' is maybe the haproxy control socket06:48
jrosserso perhaps haproxy is failing to start?06:48
noonedeadpunkjrosser: I think this should do the trick https://review.opendev.org/74987406:49
BlackFXHmm - okay - I will dig in that direction :)06:49
jrosserBlackFX: maybe check out what is actually inside all the playbooks/setup-<....>.yml06:49
jrosserit's just lists of other playbooks, you can call them individually just fine06:49
jrosserwork methodically through06:49
BlackFXOkay so playbooks/galera_insall.yml now completes cleanly. setup-infrastructure still fails in exactly the same way06:53
jrossernoonedeadpunk: also this https://opendev.org/openstack/senlin/src/branch/master/senlin/drivers/os/keystone_v3.py#L15106:56
noonedeadpunkah damn it06:56
noonedeadpunkyeah06:56
jrossernoonedeadpunk: i expect this is actually involved in the functionality more http://codesearch.openstack.org/?q=get_senlin_endpoint&i=nope&files=&repos=06:57
jrosserso not really sure about what that actually does06:57
jrossermaybe tempest shows for that?06:57
jrosserpublic may be appropriate if it needs to find it's own endpoint from the perspective of VM06:58
noonedeadpunkI'm wondering about keystone_authtoken section as it's not mentioned anywhere except docs06:59
jrosseryes, it was kind of confusing06:59
*** shyamb has quit IRC07:10
jrosserBlackFX: i have to head off now but context for any error messages is really important07:10
BlackFXOkay - thanks07:10
jrosserthis is so unusual "The host infra1_utility_container-8d88b1ab must be in galera_cluster_members" that i think folk will struggle to understand/help without the output from the play in a paste07:11
BlackFXit's failing in utility-install07:11
BlackFXrather than the galera install...07:11
BlackFXI'll see where I get to07:11
jrosserok, important info :)07:11
jrosserwhich branch/tag is this?07:12
BlackFXOn branch stable/ussuri07:13
jrosserhere in the utlility playbook is where it should install the galera *client* https://github.com/openstack/openstack-ansible/blob/stable/ussuri/playbooks/utility-install.yml#L64-L6707:14
*** shyamb has joined #openstack-ansible07:14
noonedeadpunkmaybe it's just for keystonemiddleware to verify tokens....07:14
jrosseryour error suggests that maybe the galera server instead is being installed into the utility container, which is very wrong07:15
BlackFXYeah thats what I am seeing locally too07:15
jrosserBlackFX: paste the utility play output and maybe noonedeadpunk can make a suggestion, really i must go now07:15
*** shyamb has quit IRC07:16
*** shyamb has joined #openstack-ansible07:16
BlackFXOkay thanks for your help07:16
BlackFXhttp://paste.openstack.org/show/797452/ this is the output if your keen @noonedeadpunk :)07:18
*** cshen has joined #openstack-ansible07:18
*** shyamb has quit IRC07:18
*** shyamb has joined #openstack-ansible07:19
noonedeadpunkiirc you don't have utility container in galera group?07:21
*** tosky has joined #openstack-ansible07:25
BlackFXfor some reason it's attempting to install the server role on the utility container. Cant see any reason why it is, the playbook says not to07:26
*** shyamb has quit IRC07:29
*** andrewbonney has joined #openstack-ansible07:40
noonedeadpunkhaven't you override galera_install_server?07:42
noonedeadpunkie in user_variables or somewhere?07:42
ebbexnoonedeadpunk: could you cast a vote on this https://review.opendev.org/#/c/749282/ ?07:45
*** shyamb has joined #openstack-ansible07:47
*** shyamb has quit IRC07:48
*** shyamb has joined #openstack-ansible07:48
*** shyam89 has joined #openstack-ansible07:49
noonedeadpunkebbex: hm... have some concerns...07:53
*** shyamb has quit IRC07:53
*** mmethot has joined #openstack-ansible07:54
*** mmethot_ has quit IRC07:56
noonedeadpunk(commented)08:11
BlackFXHmmmm - what listens on port 8181 - MaxRetryError: HTTPConnectionPool(host='192.168.2.2', port=8181)08:40
*** shyamb has joined #openstack-ansible08:40
noonedeadpunkused to be repo_server iirc08:42
noonedeadpunkbut nothing nowadays08:42
*** shyam89 has quit IRC08:42
BlackFXthis step in utility-install.yml is using it : TASK [python_venv_build : Upgrade pip/setuptools/wheel to the versions we want] *08:43
noonedeadpunkyou should check that /root/.pip.conf is absent08:44
*** SecOpsNinja has joined #openstack-ansible08:50
BlackFXnope, no sign of it.09:02
SecOpsNinjaregaring lxc containers, how is the network configured in them? the host has default route but the containers don't (even if i try to recreate them). this happend after i tried to create the external flat network in haproxy and replaced the haproxy configuration09:02
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_glance master: Fix native service path  https://review.opendev.org/74990709:12
noonedeadpunkSecOpsNinja: so lxcbr should have srcnat from the host09:12
noonedeadpunkBlackFX: hm, weird. maybe smth in /etc/pip.conf or dunno.. I think that package source is overriden, but we don't have such thing since queens or rocky I guess...09:14
jrosserport 8181 is still the repo server (via the loadbalancer) for wheels and venv build stuff09:17
noonedeadpunkindeed, sory...09:20
jrosserand missing default route is probably broken dnsmasq on the host for lxcbr009:22
*** aedc has joined #openstack-ansible09:23
SecOpsNinjanoonedeadpunk, i was checking the containers and they dont have the default rule and because of that i can install software in them. is this definied in any tasks? what is normally the default network of them eth0 (lxbrc0) or eth1 (br-mgmt)?09:27
jrosserSecOpsNinja: eth0 is normally the default route09:28
jrosserwhich is NAT from the host by dnsmasq running on the host09:28
SecOpsNinjayep fro some reason the eth0 doesn have any ip in the containers in eth0 interface09:29
SecOpsNinjaeven trying to recreate it...09:29
noonedeadpunkhm, dnsmasq on the host should care about ips09:29
SecOpsNinjathe eth0 is connected to lxcbr0 right?09:29
jrosserthats all configured here https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/defaults/main.yml#L101-L12109:29
jrosserin the lxc_hosts role09:30
BlackFX haproxy was sick, thanks for the pointer09:30
SecOpsNinjahow can i force the recreating of lxcbr0 in the infra host?09:35
SecOpsNinjabecuase im seasing the that brctl only shows 1 container eth0  in lxcbr009:36
jrosserSecOpsNinja: https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/files/lxc-veth-check.sh09:37
SecOpsNinjajrosser,  thanks09:38
jrosserthat script should be installed by lxc_hosts role09:39
SecOpsNinjayes a lot of them is reporting has a broken veth d2e27537_eth0 and an automated fix can not be found09:40
SecOpsNinjayep and even in the recreated container thathave its eth0 in lxcbr0 is not able to resolve names (it doesn't have anything in /etc/resolv.conf) so i supose the prolem should be in dnsmasq?09:44
*** shyamb has quit IRC09:54
BlackFXSo close lol, now failing setting up keystone when adding the service project09:59
BlackFXI think that will be a task for tomorrow09:59
*** shyamb has joined #openstack-ansible10:01
BlackFXhttp://paste.openstack.org/show/797456/10:23
*** djhankb has quit IRC10:28
*** djhankb has joined #openstack-ansible10:29
openstackgerritErik Berg proposed openstack/openstack-ansible-ceph_client stable/ussuri: Remove trailing '/' from ceph_apt_repo_url  https://review.opendev.org/74928210:57
ebbexnoonedeadpunk, jrosser : would this be better? Removing the repo with trailing slash line before we add a new ceph-repo (which someone could possibly override to add back with a trailing slash)11:04
*** aedc has quit IRC11:04
noonedeadpunkperwsonally I like it more that way)11:08
noonedeadpunkworth adding note that we can remove that task in wallaby release11:08
jrosserebbex: we should probably revisit the master patch?11:46
*** djhankb has quit IRC11:53
*** lkoranda has joined #openstack-ansible11:54
*** djhankb has joined #openstack-ansible11:54
*** omegapoint has joined #openstack-ansible11:56
*** dave-mccowan has joined #openstack-ansible11:59
mgariepycan we ask for centos8 repos to be mirrored in CI ?12:03
mgariepyho.. different fail on the other patch.12:05
mgariepyho. not really, same error only not at the same place.12:06
mgariepy2020-09-04 07:33:01.800559 | centos-8 | fatal: [aio1_glance_container-d3d2f29b]: FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'epel-networkd': Yum repo downloading error: Downloading error(s): repodata/c984028c6055d55d0d54c2f4324c00445148b90451a2e0bd4c545447c10c4a01-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/2094cdfc0278a8e6fb7dd86f308943f3751496c0bbba3245b12:06
mgariepydb2c48366166636-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/e2c3195544a9b831b711df514ee9f0558ca6c2455e8e144b3c0b2d61aed1e624-prestodelta.xml.xz - Cannot download, all mirrors were already tried without success", "rc": 1, "results": []}12:06
*** redrobot has quit IRC12:10
noonedeadpunkeah, epel instanly brokem :(12:15
noonedeadpunkhave no idea how to work around it12:16
noonedeadpunkNot sure if mirrioring will do the trick as it seems the issue is that during runtime they change these uids12:17
noonedeadpunkwe had the same for centos 7 rdo repos and were doing smth complicated stuff there12:17
noonedeadpunk(trying to avoid uid usage)12:17
noonedeadpunkbut I didn't find the way for epel....12:18
mgariepynot sure why ppl like centos..12:19
*** cshen has quit IRC12:21
*** csmart has quit IRC12:21
*** fridtjof[m] has quit IRC12:21
*** masterpe has quit IRC12:22
*** ioni has quit IRC12:22
*** shyamb has quit IRC12:24
watersjcentos == old =~ stable12:30
*** masterpe has joined #openstack-ansible12:31
mgariepywell. sure it can keep running but when you do use centos you always need some extra unstable repos12:34
*** mmethot has quit IRC13:01
*** fridtjof[m] has joined #openstack-ansible13:03
*** ioni has joined #openstack-ansible13:03
*** csmart has joined #openstack-ansible13:03
*** mmethot has joined #openstack-ansible13:03
*** mmethot has quit IRC13:08
*** spatel has joined #openstack-ansible13:08
*** mmethot has joined #openstack-ansible13:08
*** spatel has quit IRC13:12
*** lkoranda has quit IRC13:18
openstackgerritJames Denton proposed openstack/openstack-ansible-os_ironic master: [WIP] Add iPXE support to Ironic Conductor  https://review.opendev.org/73633613:28
openstackgerritJames Denton proposed openstack/openstack-ansible-os_ironic master: Add iPXE support to Ironic Conductor  https://review.opendev.org/73633613:31
*** cshen has joined #openstack-ansible13:32
ebbexjrosser: if https://review.opendev.org/#/c/748412/1 goes through, it's almost like the other patch in master never happened.13:37
jrosseroh of course!13:38
noonedeadpunkebbex: are you sure it needs depends-on?13:40
ebbexnoonedeadpunk: not anymore, as it's already in master. they were submitted together13:46
ebbexthe cleanup-logic was mostly for convenience to cherry-pick down to stable branches.13:47
noonedeadpunkit's worth I think to specify not patch id, but link to the patch. as now we're waiting for ussuri patch to merge to merge master...13:47
omegapointhey, i'm running into this bug trying to upgrade from pike to queens, how can i fix this? https://review.opendev.org/#/c/605601/2/tasks/ceilometer_db_setup.yml (the parameter doesn't exist so my playbook run fails, but i don't know where to find the file that parameterizes this locally)13:48
ebbexnoonedeadpunk: I can resubmit it without the depends-on. gimme two minutes :)13:48
openstackgerritErik Berg proposed openstack/openstack-ansible-ceph_client master: Remove cleanup logic for trailing '/'  https://review.opendev.org/74841213:51
*** d34dh0r53 has joined #openstack-ansible14:03
*** dave-mccowan has quit IRC14:08
openstackgerritSatish Patel proposed openstack/openstack-ansible-os_senlin master: Adjusted senlin_pip_packages: and changed senlin_container_mysql_password to senlin_galera_password  https://review.opendev.org/74936514:21
*** dave-mccowan has joined #openstack-ansible14:24
*** cshen has quit IRC14:34
*** cshen has joined #openstack-ansible15:01
*** rpittau is now known as rpittau|afk15:18
omegapointis it possible that the docs for using the calico neutron plugin is incomplete? https://docs.openstack.org/openstack-ansible-os_neutron/queens/app-calico.html i performed the steps in the docs but it appears to yield an invalid neutron-server config when switching from linuxbridge to calico (it still contains vlan and vxlan type drivers but they aren't available).15:21
jamesden_it is possible, yes.15:31
*** jamesden_ is now known as jamesdenton15:31
jamesdentonwhat is valid there? flat?15:32
jamesdentonif ml2.calico is set as the neutron_plugin_type (not neutron_plugin_types), then this should be invoked: https://github.com/openstack/openstack-ansible-os_neutron/blob/stable/queens/vars/main.yml#L203-L20815:33
omegapointyeah i did set neutron_plugin_type to ml2.calico but it didn't render it as "flat,local" as described in that file... strange15:38
omegapointi just resorted to setting neutron_ml2_drivers_type to "flat,local", which should have the same effect i guess?15:39
jamesdentonit should, yes15:44
*** gyee has joined #openstack-ansible15:44
jamesdentondid the mechanism_drivers get set to calico in ml2_conf.ini?15:45
omegapointyep15:47
*** cshen has quit IRC15:48
jamesdentoncan you share the ml2 conf file?15:49
omegapointhttp://paste.openstack.org/show/797481/15:54
jamesdentonok, so your type drivers got set OK15:57
jamesdentonbut yeah, tenant network types need to be adjusted accordingly15:57
jamesdentonand that might be coming from your provider network definition in openstack_user_config15:58
jamesdentonor user_variables if you're overriding that15:58
NobodyCamGood morning OSA folks!16:08
*** MickyMan77 has joined #openstack-ansible16:11
NobodyCamOff the wall question, I ended up with a IP conflict between a compute host and a lxc-container it is there a way to edit the Ip in the inventory, I attempted to remove the offending host with:16:13
NobodyCamhttps://www.irccloud.com/pastebin/8PJUsZUJ/16:13
NobodyCambut it is finding the lxc-container16:14
MickyMan77I gettings this kind of error on one of the controller nodes, all nodes is installed with same hardware/software, the other two nodes is installed without any problem.--------------------fatal: [controller03-for_utility_container-13eded48]: FAILED! => {"changed": false, "checksum": "4c93a25d01ca2ab694e06c7d7697df48b47fdbf2", "msg": "Destination16:14
MickyMan77directory /etc/sudoers.d does not exist"}controller03-for_utility_container-13eded48 : ok=63   changed=3    unreachable=0    failed=1    skipped=3    rescued=0    ignored=0------------------openstack-ansible version 21.0.0.0rc1-108-g72a39d652CentOS 8.216:14
MickyMan77i will repost that.16:15
MickyMan77I gettings this kind of error on one of the controller nodes, all nodes is installed with same hardware/software, the other two nodes is installed without any problem.16:15
MickyMan77fatal: [controller03-for_utility_container-13eded48]: FAILED! => {"changed": false, "checksum": "4c93a25d01ca2ab694e06c7d7697df48b47fdbf2", "msg": "Destination directory /etc/sudoers.d does not exist"}16:15
MickyMan77controller03-for_utility_container-13eded48 : ok=63   changed=3    unreachable=0    failed=1    skipped=3    rescued=0    ignored=016:15
MickyMan77---16:16
MickyMan77openstack-ansible version 21.0.0.0rc1-108-g72a39d652CentOS 8.216:16
ebbexMickyMan77: do you know which step this in the playbooks setup-hosts, setup-infrastructure, setup-openstack?16:19
MickyMan77setup-hosts16:19
*** cshen has joined #openstack-ansible16:19
*** cshen has quit IRC16:24
ebbexI can't really tell where this goes wrong, what are the tasks preceeding the failure?16:31
*** cshen has joined #openstack-ansible17:20
*** SecOpsNinja has left #openstack-ansible17:38
*** omegapoint has quit IRC17:49
openstackgerritMerged openstack/openstack-ansible-ceph_client stable/ussuri: Remove trailing '/' from ceph_apt_repo_url  https://review.opendev.org/74928218:07
*** andrewbonney has quit IRC18:22
*** cshen has quit IRC18:29
*** MickyMan77 has quit IRC18:33
jrosserNobodyCam: if you have your /etc/openstack_deploy in git then you could try editing and references to the IP of the compute host in the inventory json18:44
jrosserjust make sure you have a way to back out if something goes bad18:45
NobodyCamAh thank you18:45
jrossermoving the compute host IP is probably less work than fixing an lxc container IP18:46
*** dave-mccowan has quit IRC18:46
NobodyCamYea that’s what I’m attempting18:49
*** cshen has joined #openstack-ansible18:59
*** alvinstarr has quit IRC19:10
*** cshen has quit IRC19:40
*** spatel has joined #openstack-ansible19:52
*** spatel has quit IRC19:56
*** cshen has joined #openstack-ansible20:03
*** cshen has quit IRC20:07
*** renich has joined #openstack-ansible20:22
*** renich has quit IRC21:21
*** rh-jelabarre has quit IRC21:29
BlackFXDoes anyone know what ansible is doing during the "os_keystone: Add service project" step? It is failing but the debug output doesn't show what it's actually running through python on the host. http://paste.openstack.org/show/797456/21:44
BlackFXHmm now it's changed to a 504 error http://paste.openstack.org/show/797490/21:53
*** cshen has joined #openstack-ansible22:04
*** cshen has quit IRC22:08
BlackFXAnyone know where the keystone container sends it's logs?22:14
BlackFXUFW!!!!!22:39
*** tosky has quit IRC23:07

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!