Wednesday, 2018-11-28

*** tosky has quit IRC00:13
jamesdentonthanks cloudnull and jrosser!00:23
*** weezS has quit IRC00:34
*** amab has quit IRC01:04
*** cshen has quit IRC01:37
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_placement master: [WIP] Create base files to install placement  https://review.openstack.org/61882001:56
pabelangerhow do lxc containers get their IP addressing?03:00
pabelangerI'm trying to bootstrap a new controller, to confirm my deployment process works, but this time around, not of the containers can be SSH into03:00
pabelangerseem IP address haven't been setup03:00
*** maddtux has joined #openstack-ansible03:01
pabelangerhttps://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/logs/e9/e9d0ce46f58e74abefc15558e6c70bff84e9bbe7/post/packet-ci-cloud-deploy/6975fbe/job-output.html#l324603:02
pabelangershow ansible unable to gather facts03:02
*** chhagarw has joined #openstack-ansible03:59
*** ram5391 has joined #openstack-ansible04:53
*** udesale has joined #openstack-ansible04:54
*** pcaruana has joined #openstack-ansible05:09
*** mma has joined #openstack-ansible05:11
*** ram5391 has quit IRC05:11
*** mma has quit IRC05:15
*** lemouchon has quit IRC05:48
*** cshen has joined #openstack-ansible06:04
*** cshen has quit IRC06:09
*** gyee has quit IRC06:20
*** cshen has joined #openstack-ansible06:22
*** aedc has quit IRC06:25
*** cshen has quit IRC06:27
*** ahosam has joined #openstack-ansible06:29
*** udesale has quit IRC06:43
*** udesale has joined #openstack-ansible06:44
*** udesale has quit IRC06:45
*** udesale has joined #openstack-ansible06:46
*** udesale has quit IRC06:57
*** chkumar|away is now known as chkumar|ruck06:59
*** gkadam has joined #openstack-ansible07:00
*** radeks has joined #openstack-ansible07:07
jrosserpabelanger: OSA does not SSH directly to the containers, there is a connection plugin which works out what physical_host is for each container, ssh to there then uses a local technique to execute the ansible against the correct container07:08
jrosserso "no route to host" is maybe more to do with the physical host in the case than the container07:09
jrosserpabelanger: i also see that mitogen is installed - is ansible configured to use that in your setup?07:11
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_nova master: Ensure that UEFI firmware is present  https://review.openstack.org/62002607:12
*** rgogunskiy has quit IRC07:28
*** dcdamien has joined #openstack-ansible07:34
*** hamzaachi has joined #openstack-ansible07:45
*** DanyC has joined #openstack-ansible07:51
*** udesale has joined #openstack-ansible07:53
*** DanyC has quit IRC07:56
*** mma has joined #openstack-ansible07:57
*** udesale has quit IRC07:58
*** rpittau has joined #openstack-ansible08:01
jonherMorning, could we get some +workflow on https://review.openstack.org/619092/ and https://review.openstack.org/581248/ Thank you :)08:04
openstackgerritMerged openstack/openstack-ansible-haproxy_server stable/rocky: Add 'absent' service state  https://review.openstack.org/62033708:06
*** aedc has joined #openstack-ansible08:08
Miouge-Good morning everyone08:09
*** jawad_axd has joined #openstack-ansible08:10
*** udesale has joined #openstack-ansible08:11
jrosserMorning08:16
*** hamzaachi has quit IRC08:19
*** rpittau_ has joined #openstack-ansible08:31
*** ahosam has quit IRC08:31
*** hamzaachi has joined #openstack-ansible08:31
*** ahosam has joined #openstack-ansible08:31
*** hamzaachi has quit IRC08:32
*** hamzaachi has joined #openstack-ansible08:32
*** hamzaachi has quit IRC08:33
*** rpittau has quit IRC08:33
*** hamzaachi has joined #openstack-ansible08:34
*** rpittau_ is now known as rpittau08:35
*** rpittau_ has joined #openstack-ansible08:40
*** rpittau has quit IRC08:40
*** rgogunskiy has joined #openstack-ansible08:41
*** rgogunskiy has quit IRC08:42
*** rgogunskiy has joined #openstack-ansible08:44
*** hamzaachi has quit IRC08:45
*** hamzaachi has joined #openstack-ansible08:46
*** rpittau_ is now known as rpittau08:49
*** tosky has joined #openstack-ansible08:51
*** aedc has quit IRC09:02
*** shardy has joined #openstack-ansible09:10
*** ahosam has quit IRC09:10
*** ahosam has joined #openstack-ansible09:10
*** rgogunskiy has quit IRC09:11
openstackgerritAndriy Shevchenko proposed openstack/openstack-ansible-os_barbican master: fix tox python3 overrides  https://review.openstack.org/60682509:14
*** dcdamien has quit IRC09:20
evrardjpmorning everyone09:21
jonhermorning \o09:21
*** DanyC has joined #openstack-ansible09:30
*** DanyC has quit IRC09:30
*** DanyC has joined #openstack-ansible09:31
*** rpittau has quit IRC09:34
jrosserwould be good to get some reviews on this to improve centos gate reliability https://review.openstack.org/#/c/620313/09:49
*** ahosam has quit IRC09:49
Miouge-evrardjp: I saw your comment on the LetsEncrypt patch (586774) “If someone wants to implement a different way that's not killing haproxy connections with a lua backend, up to him/her :)”. I am not familiar with the HAProxy Lua client, but can't we redirect the /.well-known/acme-challenge to the certbot standalone webserver in a new LXC container? (idea inspired from https://serversforhackers.com/c/letsencrypt-with-haproxy )10:00
*** cshen has joined #openstack-ansible10:05
*** dcdamien has joined #openstack-ansible10:05
*** hamzaachi has quit IRC10:08
*** hamzaachi has joined #openstack-ansible10:09
*** tobias-urdin has quit IRC10:10
*** cshen has quit IRC10:10
*** cshen has joined #openstack-ansible10:12
evrardjpMiouge-: we can10:17
evrardjpMiouge-: But we can probably use haproxy's internal system to deal with that10:17
evrardjpassuming real passthrough is not used ofc10:18
*** ahosam has joined #openstack-ansible10:19
noonedeadpunkmorning folks10:22
openstackgerritMerged openstack/openstack-ansible-ops master: MNAIO: Add ability to deploy Ceph  https://review.openstack.org/60733810:26
evrardjpmorning noonedeadpunk10:26
noonedeadpunkI've got a bit offtopic question (not directly related to OSA). Does anybody knows/remeber how to make nova work with libvirt 4? It's kinda P->Q upgrade question....10:27
noonedeadpunkIt seems, that nova should generate config for VMs without serial, but it still includes it, which results in "'serial' is deprecated, please use the corresponding option of '-device' instead"10:30
odyssey4menoonedeadpunk I don't know if there's a solution - you might need to ask in #openstack-nova10:31
odyssey4meA workaround would be to create a fork of nova which does what's needed, then use that fork. We give you the tools to do that after all. ;)10:31
noonedeadpunkYeah, I've just thought that's it's something, that you faced with (and probably fixed) during P->Q upgrade (as Ubuntu in P uses libvirt 3.6, and in Q it's already 4.0)10:32
odyssey4menoonedeadpunk maybe there's a setting somewhere? or maybe this is a nova bug, actually10:33
Miouge-evrardjp: it doesn’t look like the haproxy-lua thing works with several haproxy servers, it would need to share the challenge, keys and certificates with the other haproxy servers10:34
*** hamzaachi has quit IRC10:35
evrardjpMiouge-: you probably want to share the certificates anyway, as they are sharing the same endpoints10:36
noonedeadpunkodyssey4me may be:) anyway thanks, at least now I know, that it's not smth common, solution to which I just wasn't able to find10:36
evrardjpbut yeah I'd be surprised on how to rotate certificates on sub nodes.10:37
evrardjpwhether with a certbot in a container or not10:37
Miouge-evrardjp: Yep, it needs a mechanism to copy challenge/key/cert to other haproxy servers. AFAIK haproxy servers are not really aware of each other10:38
evrardjplsyncd should be enough10:38
*** electrofelix has joined #openstack-ansible10:39
evrardjpMiouge-: I haven't digged deep into what it takes to reload a certificate from haproxy. That's why I left that comment10:40
evrardjpyeah the 'with a lua backend' should have been '(with a lua backend, for example)'10:40
Miouge-evrardjp: I think I start to see how this could work: ACME client (lua/certbot/whatever) + lsyncd + something to reload. A little similar to ingress-nginx in Kubernetes10:45
*** Miouge- has left #openstack-ansible10:46
*** Miouge- has joined #openstack-ansible10:46
*** vnogin has joined #openstack-ansible10:47
*** ahosam has quit IRC10:57
*** hamzaachi has joined #openstack-ansible10:57
*** vakuznet has quit IRC11:19
*** vnogin has quit IRC11:21
openstackgerritDebo Zhang proposed openstack/openstack-ansible master: Fix Chinese quotes  https://review.openstack.org/62056611:22
*** chason has quit IRC11:31
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Update playbook to newer syntax.  https://review.openstack.org/62056911:32
odyssey4mehwoarang evrardjp so ceph on stable/rocky for suse is broken, and we have https://review.openstack.org/619796 proposed to resolve that because the stable-3.2 branch has the right fixes... there was recentlya patch to move stable/rocky to use stable-3.1 because stable-3.2 is still a release candidate... so we need to decide - do we accept a release candidate for our rocky release, or do we make ceph/suse non voting while a fix is11:35
odyssey4me implemented upstream in stable-3.111:35
odyssey4me implemented upstream in stable-3.111:35
odyssey4methe revert to use stable-3.1 is here: https://review.openstack.org/61648111:36
*** mma has quit IRC11:40
*** dave-mccowan has joined #openstack-ansible11:44
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone master: Ensure that LDAP config is deployed on all keystone hosts  https://review.openstack.org/62057411:45
*** cshen has quit IRC11:47
*** mma has joined #openstack-ansible11:48
*** hamzaachi has quit IRC11:50
*** mma has quit IRC11:54
openstackgerritVieri proposed openstack/openstack-ansible-os_cloudkitty master: fix tox python3 overrides  https://review.openstack.org/60681911:54
*** cshen has joined #openstack-ansible12:14
*** hamzaachi has joined #openstack-ansible12:14
*** ansmith has quit IRC12:17
*** cshen has quit IRC12:18
*** cshen has joined #openstack-ansible12:20
jamesdentonhow does one rerun a gate job that failed?12:21
kaiokmojamesdenton: recheck?12:24
jamesdentonis that sufficient for the +2 as well?12:25
*** cshen has quit IRC12:25
*** cshen has joined #openstack-ansible12:28
kaiokmoeven if you recheck it will probably mantain +2cr and +1w IIRC, and it should be enough12:28
kaiokmomaintain*12:30
*** ahosam has joined #openstack-ansible12:33
*** maddtux has quit IRC12:53
*** fghaas has joined #openstack-ansible12:56
*** hamzaachi has quit IRC13:06
*** ahosam has quit IRC13:13
mnaserjamesdenton: its not possible to rerun gate jobs only13:17
mnasera recheck will go through both check and gate13:17
*** strattao has joined #openstack-ansible13:17
jamesdentonbummer, but OK. thanks mnaser. and kaiokmo.13:17
mnaserjamesdenton: #makejobsstableagain13:18
jamesdentonno kidding13:18
mnaseranything you're struggling with jamesdenton ?13:18
jamesdentonnot at the moment. stupid gate timeout13:18
jamesdentontoday is the day it's gonna work, right?13:19
jrosserevrardjp: rechecking https://review.openstack.org/#/c/618822/ isnt going to work - we need to fix ceph/suse first?13:19
evrardjpI thought this was a temp faire13:19
evrardjpfailure13:19
jrosserno, it is brok13:19
evrardjpas aio distro suse worked for this branch13:20
evrardjpI see13:20
jrossersee the comments earlier about https://review.openstack.org/61979613:20
evrardjphttp://logs.openstack.org/22/618822/1/check/openstack-ansible-deploy-distro_ceph-opensuse-423/cb90b16/job-output.txt.gz#_2018-11-26_17_28_49_95223713:20
evrardjpok13:21
jrosseryes that - ceph-ansible 3.1 does not set that var for suse at all13:21
mnaserah centos was the culprit13:21
mnaserand opensuse-15013:21
jamesdentonfor me? yes13:22
mnaserhttp://logs.openstack.org/69/584069/14/gate/openstack-ansible-deploy-aio_lxc-centos-7/aa06a7e/logs/ara-report/13:22
mnaserdoesnt seem like anything that took much longer in there13:23
evrardjpjrosser: I must say I love this kind of comments: https://github.com/ceph/ceph-ansible/issues/307813:23
mnaserwait what13:24
mnaserceph-ansible is not maintained?13:24
evrardjptaht's what I read.13:24
*** hamzaachi has joined #openstack-ansible13:24
mnaseri think maybe13:24
mnaserthey meant to say13:24
mnasernot maintained for coreos13:24
mnaser?13:24
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_placement master: [WIP] Create base files to install placement  https://review.openstack.org/61882013:25
evrardjpI sure hope so13:25
evrardjpI hope it's not moving away to be container only13:25
kaiokmoREADME does not says that it is not maintained13:25
kaiokmoand there is a lot of recent commits, so13:25
mnaserhttps://github.com/ceph/ceph-ansible/commit/d749dc9f365643f0be171efb7f934be7d763582f13:26
mnaseri think this is what they mean13:26
evrardjpkaiokmo: neither does the docs13:26
kaiokmomnaser: yeah probably13:27
evrardjpmnaser: makes sense13:27
evrardjpstill need the ceph_uid there, and we should probably make it -nv until it's resolved13:27
evrardjpI don't have the cycles right now13:27
mnaserim curious more on how13:27
mnaserit broke.13:27
evrardjpmnaser: we updated the shas13:27
mnaserso did ceph-ansible go from *not* having it to *having* it?13:28
jrosserWe accidentally were using master13:28
evrardjpthis ceph_uid var? yes13:28
jrosserFixed that13:28
mnaserah13:28
mnaserokay13:28
jrosserSo it went away13:28
evrardjpceph_uid var is a 'new var' for us, that is well handled in ubuntu/centos. It appeared during the change of sha (move to track 3.2 vs master)13:29
jrosserAnd when it got rolled back to 3.1 I don’t think 3.2 was a proper thing, still not quite sure about that13:29
evrardjpsorry for me using half sentences sometimes13:29
evrardjpoh it's maybe 3.1 I can't remember13:29
evrardjpanyway13:29
*** fghaas has quit IRC13:30
pabelangerjrosser: yah, after I feel a sleep I figured I should retest without mitogen and see if that was the problem. Looks like it is, I'll keep it disabled for now. But strange, I thought the containers did get an IP address.13:30
*** fghaas has joined #openstack-ansible13:32
*** strattao has quit IRC13:33
jrosserpabelanger: they do indeed get an IP on the mgmt network, but thats now how ansible connects to them in this case13:33
*** chason has joined #openstack-ansible13:33
pabelangerack13:34
jrosserpabelanger: i have done tests with mitogen in an AIO and it sort of works, but nothing yet on multiple nodes - some thought would be needed about passing physical_host into mitogen_via13:34
pabelangerjrosser: yup, I think there is some weirdness going on too. I've just disabled it for now, as I was just testing how it worked13:36
jrosserdelegation is broken really, and it will eventually blow up here https://github.com/dw/mitogen/issues/41413:39
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Fix warning for guestfish workaround task  https://review.openstack.org/62059913:43
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: If there is no secondary data disk, use the root disk  https://review.openstack.org/62060013:43
odyssey4memnaser evrardjp the job to make the suse job voting was created before the switch of the ceph-ansible sha, and it merged after the switch of the ceph-ansible sha13:45
odyssey4mezuul clearly didn't do a rebase in that case13:45
odyssey4memnaser evrardjp jrosser so the question I posed earlier stands - are we prepared to use the not-yet-release stable-3.2 branch for rocky to ensure suse support works... or do we make suse support non-voting until an upstream patch is done and merged13:47
mnaserodyssey4me: i think its opensuse 150 which is broken right? not 423?13:48
evrardjpI don't have the time to fix that today, so I suggest we move to nv, add a bug, and maybe assign that to me13:48
evrardjpmnaser: 42.313:48
evrardjpis broken13:48
evrardjponly for ceph with distro packages13:48
evrardjphttps://review.openstack.org/#/c/618822/13:48
mnaseri would rather move one distro to nv rather than potentially risk the others13:49
jrosseragreed - it's also not very clear from the ceph-ansible repo is 3.2 is released or not13:49
odyssey4meok, if you're happy with that, so am I13:49
jrossertheres a branch an a release note13:49
odyssey4me3.2 has not been tagged - only RC tags13:49
evrardjpI'd rather stay on 3.1 too13:49
jrosseraahhhhh13:49
odyssey4meok, I'll work up the patch to switch to non-voting then and do the bug for evrardjp - thanks evrardjp13:50
evrardjpwell if I had more time I'd fix that now, but I have plenty on my plate today13:50
odyssey4memnaser jrosser I'm still looking for a review for https://review.openstack.org/55179113:51
openstackgerritJames Denton proposed openstack/openstack-ansible-os_neutron master: Modify OVS hostname value to match server's hostname  https://review.openstack.org/62060413:51
mnaserah dang odyssey4me i got distracted13:52
mnasergoing over it now13:52
mnaserodyssey4me: lgtm. :)13:56
*** cshen_ has joined #openstack-ansible14:04
mgariepysomeone have better eyes than me? http://logs.openstack.org/40/620140/1/check/openstack-ansible-linters/0e3897a/job-output.txt.gz#_2018-11-27_14_45_39_25439314:04
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Set SUSE ceph-distro job to non-voting  https://review.openstack.org/62060714:05
*** cshen has quit IRC14:06
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Remove unnecessary octavia scenario AIO bootstrap  https://review.openstack.org/61979214:10
*** vakuznet has joined #openstack-ansible14:10
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Ensure AIO container_tech/install_method vars are namespaced  https://review.openstack.org/62027814:10
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Implement documentation changes for translations  https://review.openstack.org/62027914:10
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: [docs] Clean up the AIO user story  https://review.openstack.org/62028014:10
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Update all SHAs for 18.1.1  https://review.openstack.org/61882214:10
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Update playbook to newer syntax.  https://review.openstack.org/62056914:11
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Add extra volume types to AIO  https://review.openstack.org/61705314:12
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Move ARA install to end of bootstrap  https://review.openstack.org/61778514:12
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: haproxy: remove repo_cache service  https://review.openstack.org/62033614:12
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Add MariaDB infrastructure mirrors  https://review.openstack.org/61971414:12
openstackgerritMarc Gariépy (mgariepy) proposed openstack/openstack-ansible stable/rocky: Add openstackclient bash completion  https://review.openstack.org/61935414:13
odyssey4memgariepy when they're in the same repo - no need to use depends-on, better to just rebase on top of the patch it depends on14:13
odyssey4medepends-on is actually for cross-repo depenencies14:14
odyssey4meno matter - it still works :)14:14
mgariepyok14:14
mgariepynext time :D14:15
*** ansmith has joined #openstack-ansible14:16
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_placement master: [WIP] Create base files to install placement  https://review.openstack.org/61882014:18
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: haproxy: remove repo_cache service  https://review.openstack.org/62033614:18
odyssey4mejrosser could you cast your eyes on https://review.openstack.org/#/c/614194/ - it's been sitting a while14:21
odyssey4meand it needs your ceph expertise14:21
odyssey4meor logan- if you're available14:22
*** udesale has quit IRC14:24
*** udesale has joined #openstack-ansible14:25
pabelangerlogan-: I seem to be having some routing issue getting to keystone from my nodepool node: https://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/logs/47/474cf5544baa48eb04a6309395ba8c8c5194a083/post/packet-ci-cloud-deploy/f23881f/job-output.html#l633714:33
pabelangerlogan-: curious if you had this issue or I missed something14:34
*** strattao has joined #openstack-ansible14:36
mnaserpabelanger: looks like you've deployed keystone on the internal ip14:40
mnaserwhich is unreachable14:40
mnaserthat job is delegated to localhsot14:41
mnaserso services (aka keystone) must be reachable14:41
pabelangermnaser: if I am reading right, localhost is trying to reach the network, and you are right that range is only exposed via SSH proxyhost14:41
pabelangerbut, I think logan- is doing the same thing14:42
*** fghaas has quit IRC14:42
logan-im also running pike which does not have the new delegation setup yet afaik14:43
*** vnogin has joined #openstack-ansible14:43
pabelangerOh14:43
pabelangerah14:43
logan-but i think you could just delegate that task to the utility container instead14:43
logan-iirc there is a var to configure the delegation14:43
pabelangerYah, I suspect I need to setup an HTTP proxy for that network14:44
pabelangerif the deployment host needs to do HTTP things with it14:44
pabelangerlogan-: any ideas where I make that change?14:44
logan-it shouldn't need that ability, you should be able to delegate the task so the http request comes from the utility container14:44
logan-not sure, checking14:45
pabelangerokay, same looking to see how it works14:45
logan-keystone_service_setup_host14:47
*** cshen_ has quit IRC14:47
*** cshen has joined #openstack-ansible14:47
logan-so user_variables.yml, set keystone_service_setup_host: "{{ groups['utility_all'][0] }}" and see if that works14:47
logan-actually youll hit this problem with all services, so set openstack_service_setup_host instead14:48
logan-keystone_service_setup_host inherits that var14:48
logan-heh even found a reno :) https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/tree/releasenotes/notes/keystone-service-setup-host-cd3ee3346af823e6.yaml14:48
pabelangerokay, creating PR now14:49
pabelangerlogan-: is keystone the only service that does this now, or will I have to do it for every openstack service14:50
pabelangersorry, not familiar with the change14:50
logan-if you set openstack_service_setup_host it should fix it for all openstack services14:51
pabelangerkk14:51
pabelangerhttps://git.openstack.org/cgit/openstack/openstack-ansible/tree/releasenotes/notes/openstack-service-setup-host-f38d655eed285f57.yaml14:53
antonymodyssey4me: so this sshd version tag fun... do we have a process for backport a fix to eol versions?  it's breaking the leapfrog code because it bootstraps each version as it goes, i guess the other alternative would be just to fork all of the branches to another repo, fix them and point leapfrog to that14:53
odyssey4meantonym pike is done, as it ocata - anything else is EOL and will require a fork14:54
antonymok, so i'll put up something to the leap code to make the osa repo location a variable then14:55
odyssey4meantonym yeah, not so much fun - but c'est la vie14:56
antonymyeah, my main concern it would break other users trying the leap code and that don't have the fixed repo14:56
antonymwill probably have to include the forked repo with the fixes in the notes then14:56
odyssey4meantonym another option could be to implement something in the leap code to modify the a-r-r after cloning and before using it to bootstrap14:57
odyssey4meat least from ocata onwards we have the extended maintenance policy to keep the branches alive longer14:58
antonymyeah, i thought about that too but it just seems a bit messier, especially if these type of repos break over time14:58
odyssey4meyeah, unfortunately we can't bring those branches back as far as I know - so perhaps for those which are eol the mess is necessary15:00
pabelangerlogan-: okay, PR merged, post job running15:01
pabelangeronce this works, the next step is to drop the need for jumphost to br-mgmt network15:02
pabelangerand just use the public IPs directly15:02
*** vnogin has quit IRC15:12
*** chkumar|ruck is now known as chkumar|away15:17
*** mgariepy has quit IRC15:18
*** mgariepy has joined #openstack-ansible15:19
Miouge-I have a 10 nodes, single region OpenStack Fuel deployment (Mitaka I think) that I would love to convert to OSA. What is the general procedure? Any ideas where I can find info about this kind of thing? any talk on this out there?15:21
*** ivve has joined #openstack-ansible15:21
odyssey4meMiouge- some folks have done it, but I'm not aware of any notes anywhere15:21
odyssey4meI know mnaser and his crew have done it, as well as the city cloud folks.15:22
*** vakuznet has quit IRC15:22
openstackgerritMerged openstack/openstack-ansible master: Ensure that a consistent mirror is used for RDO  https://review.openstack.org/62031315:23
mgariepyi did switch from fuel to osa a while ago, but didn't keep the vms.15:25
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Ensure that a consistent mirror is used for RDO  https://review.openstack.org/62062915:26
mgariepyi kept the volumes and network (-ish) ovs > linux bridge, gre > vxlan15:26
Miouge-mgariepy: Ceph for Cinder?15:26
mgariepyyep15:26
jrosserodyssey4me: you going to edit that for rocky?15:26
Miouge-mgariepy: ceph-ansible “took over” the Ceph cluster? maybe that could be a first step15:27
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Ensure that a consistent mirror is used for RDO  https://review.openstack.org/62062915:27
jrossercool15:28
mgariepymy ceph cluster is only 8 nodes, i didn't bother with it, only moved the mons from controllers to the ceph cluster nodes.15:28
mgariepybut yeah, probably move out the ceph cluster, from fuel, then rabbit and galera i guess.15:29
mgariepymaybe it has changed but in the old days, fuel wasn't really managing stuff, it was only a run once thing15:30
*** jawad_ax_ has joined #openstack-ansible15:31
pabelangerHmm15:31
pabelangershade is required for this module: https://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/logs/b2/b2cc94a1fd0e2c169ece8d343d39c8f50d2c941f/post/packet-ci-cloud-deploy/12a69c7/job-output.html#l601715:32
Miouge-mgariepy: I don’t follow, did you make a cresh Ceph cluster or added nodes to the existing one and used the autoheal to migrate data?15:32
Miouge-s/cresh/fresh15:32
mgariepynop, i just moved the ceph mons on other nodes.15:32
mgariepythe mons where on the controllers.15:33
*** jawad_axd has quit IRC15:33
*** jawad_a__ has joined #openstack-ansible15:33
mgariepyand moved them on the ceph nodes.15:33
mgariepydo you need to keep the vms running ?15:34
Miouge-I can reboot them or turn off for a week end or so15:35
*** gkadam has quit IRC15:35
pabelangerlogan-: odyssey4me: looks like a bug with https://review.openstack.org/568146/ seems the utility container doesn't have shade installed15:36
pabelangersee log above15:36
*** jawad_ax_ has quit IRC15:36
logan-ahh makes sense15:36
mgariepywhat os is on the fuel nodes ?15:37
mgariepyis it xenial ?15:37
pabelangernot sure how best to solve this, delegate shade dependencies to op openstack_service_setup_host too?15:37
Miouge-mgariepy: Fuel node is Centos VM but all OpenStack nodes are Ubuntu 14.04 I think15:38
pabelangerhow is shade installed on to deployment host? I am guessing from pip?15:41
pabelangerif so, I think I can just add shade to utility_pip_packages15:42
*** udesale has quit IRC15:43
*** udesale has joined #openstack-ansible15:43
odyssey4mepabelanger it's already there, but in a venv - just chaging that var won't help because it assumes whatever the service setup host is has the right packages in the system15:45
odyssey4mewe may have to try and work out something nicer to make that easier15:45
odyssey4meinstalling python packages into the system is something we're actively trying to avoid15:45
odyssey4me(via pip I mean)15:45
pabelangerodyssey4me: okay, will defer to you on how best to fix then :)15:46
mgariepyMiouge-, start by moving ceph-mon and rgw somewhere else15:46
mgariepyMiouge-, watch for : https://bugs.launchpad.net/nova/+bug/145264115:46
openstackLaunchpad bug 1452641 in nova (Ubuntu) "Static Ceph mon IP addresses in connection_info can prevent VM startup" [Medium,In progress] - Assigned to Corey Bryant (corey.bryant)15:46
pabelangerodyssey4me: when you say system, you mean outside of the venv?15:47
odyssey4mepabelanger we may have to include shade (rocky)/openstacksdk (stein) in the utility_pip_packages for just this purpose15:48
mnaserodyssey4me: i had a huge upgrade fire yesterday caused by the galera upgrade playbooks15:48
mnaserwe somehow restart all of them at some point? one by one? without ever checking that they have fully sync'd up..15:49
odyssey4mepabelanger yes, unfortunately I don't see an easy way around that - thankfully it's a small number of packages15:50
mnaserand also, somehow, haproxy was routing things to one of the galera servers that was *not* ready to serve traffic yet15:50
mnaserresulting in a bunch of WSREP not responding15:50
pabelangerodyssey4me: if it was pulled in via apt / yum, is that the better way to deal with the package?15:50
odyssey4memnaser run-upgrade does a serialised upgrade IIRC - and the playbooks to deploy do so too... I think we should actually do it in the normal deploy playbook, not have a different process for both15:50
mnaserodyssey4me: but we call "Restart mysql (All)" for some reason15:51
mnaserat least the log show that15:51
odyssey4mepabelanger yeah, that's a decent option15:51
odyssey4memnaser it's a bit of a context switch for me right now, and I don't remember the details off-hand15:51
mnasersorry15:52
mnaseri'll dig into it more15:52
odyssey4mepabelanger let me take a peek through what's going on there to see what'll work best15:52
mnaserbut yeah, it was pretty tragic for the cloud, i will dig more15:52
pabelangerodyssey4me: okay, thanks. will hold off on utility_pip_packages for the moment15:52
mnaseralso to figure out why haproxy was sending traffic to a non-ready server15:52
*** cshen has quit IRC15:53
mnaserton of: 2018-11-26 21:40:28.906 109358 ERROR sqlalchemy.pool.QueuePool InternalError: (1047, u'WSREP has not yet prepared node for application use')15:53
odyssey4mepabelanger ok, as I thought - https://github.com/openstack/openstack-ansible/blob/master/playbooks/utility-install.yml#L103-L112 installs utility_pip_packages into the venv15:53
pabelangerodyssey4me: and that is bad, in this case?15:54
*** cshen has joined #openstack-ansible15:54
*** vakuznet has joined #openstack-ansible15:54
odyssey4mepabelanger no, it means that we'll ned to try and use a distro package - so add shade to https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/utility_all.yml#L29-L3215:54
odyssey4meexcept that the package name is different across distributions, so this may be a little fun :)15:54
pabelangerodyssey4me: yah, okay for now I can add it to my user_variables file and see if that fixes the issue15:55
odyssey4mean option is to allow the resulting python executable in each role to be overridden, but default back to the system15:56
pabelangerodyssey4me: yah, that is what I do with my zuul / nodepool roles15:56
pabelangerI think that is a good option also15:56
odyssey4meso in https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/keystone_service_update.yml#L30 change ansible_python['executable'] to be something like keystone_service_setup_host_python_executable and have that be set in defaults as keystone_service_setup_host_python_executable: "{{ openstack_service_setup_host_python_executable ¬ default(ansible_python['executable']) }}"15:58
Miouge-mgariepy: I am thinking to do: fuel Mitaka => OSA Mitaka => OSA Newton => ... => OSA Rocky15:58
odyssey4methen in the utility group_vars we can set that var by default so that everything works like magic15:59
odyssey4meassuming that works, I think that's the best option15:59
pabelangerodyssey4me: actually, we already install some openstack client libraries using utility_distro_openstack_clients_packages maybe just see about adding shade there by default too?15:59
*** weezS has joined #openstack-ansible15:59
pabelangerhttps://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/utility_all.yml#L3415:59
odyssey4mepabelanger that list is for distro (not source) builds16:00
mgariepyMiouge-, so, fuel Mitaka => OSA Mitaka > osa newton 14.04 > osa newton 16.04 > osa o p q r < osa R 180416:00
mgariepyMiouge-, lots of works ahead :)16:01
pabelangerodyssey4me: okay and we prefer source build for shade?16:01
odyssey4mepabelanger distro/source is a setting for the build as a whole16:02
odyssey4methe distro install of openstack is new, and only really fully working for suse right now16:02
pabelangerah, okay16:02
odyssey4meI'll do a couple of patches now which you can try out.16:02
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone master: Enable overriding the service setup host python interpreter  https://review.openstack.org/62064416:09
odyssey4mebother, now it gets complicated :/16:20
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Set the utility container service setup interprter automatically  https://review.openstack.org/62065116:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Set the utility container service setup interpreter automatically  https://review.openstack.org/62065116:26
*** gyee has joined #openstack-ansible16:27
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Set the utility container service setup interpreter automatically  https://review.openstack.org/62065116:27
odyssey4mepabelanger something like that - the patch done for keystone would have to be propogated to other roles too16:27
*** hamzaachi has quit IRC16:27
Miouge-mgariepy: yes, it won’t happen overnight. I need to do this in stages16:29
*** udesale has quit IRC16:34
*** jawad_a__ has quit IRC16:35
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: Update all SHAs for 16.0.23  https://review.openstack.org/61884016:35
*** jawad_axd has joined #openstack-ansible16:36
mgariepyMiouge-, when you get there, here are the notes for newton 1404 > 1604 upgrades: https://etherpad.openstack.org/p/osa-newton-xenial-upgrade16:37
*** jawad_axd has quit IRC16:41
cloudnullmornings all16:42
Miouge-mgariepy: Thanks! That’s the kind of thing I am looking for16:43
*** hamerins has joined #openstack-ansible16:43
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: Remove non-existant 'functional' test  https://review.openstack.org/62065516:44
*** cshen has quit IRC16:46
*** hamerins has quit IRC16:47
*** hamerins has joined #openstack-ansible16:48
*** fghaas has joined #openstack-ansible16:49
pabelangerodyssey4me: oh noes: To utilize this module, the installed version ofthe shade library MUST be >=1.8.016:56
pabelangerlooking to see what version was installed vi apip16:57
pabelangerthat is from utility_distro_packages16:58
*** lbragstad has quit IRC17:02
*** pcaruana has quit IRC17:03
*** lbragstad has joined #openstack-ansible17:05
*** KeithMnemonic has quit IRC17:14
*** strattao has quit IRC17:20
openstackgerritAntony Messerli proposed openstack/openstack-ansible-ops master: Allow the option to specify OSA git repo URL  https://review.openstack.org/62066217:24
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_placement master: [WIP] Create base files to install placement  https://review.openstack.org/61882017:26
*** hamerins has quit IRC17:28
*** jackivanov has quit IRC17:28
*** hamerins has joined #openstack-ansible17:29
odyssey4mepabelanger yep, using a distro package sucks for this where we need more precise control - which is why the venv is better17:35
* cloudnull is a general hater of disto packaging 17:39
odyssey4mepabelanger if you could try out those two patches to verify the concept then we can get that propogated17:40
cloudnullodyssey4me -re: https://review.openstack.org/#/c/607808 - im totally confused.17:40
pabelangerodyssey4me: sure, let me see how to apply them to my OSA deployment host17:40
cloudnullwouldnt that loop iterate over all nodes, which would include the first one ?17:41
odyssey4mecloudnull the issue I'm trying to get resolved is that the sshd must be enabled as soon as possible17:41
odyssey4meit's all nice and dandy if they all eventually get sshd enabled, but it's much better if it's enabled right from the get-go17:41
cloudnullso should I order the list?17:41
odyssey4mecloudnull nope, just make the task execute when the ansible_host is the first ansible_play_host17:42
cloudnulllike sorted by playbook host and inventory hostname17:42
odyssey4mecloudnull nope, like https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/main.yml#L13117:43
cloudnullthen wouldn't it run on only the one host ?17:43
odyssey4meit only need to, because it loops through delegation to all the others, right?17:43
odyssey4meso run on host 1, but loop through the group and delegate to them all - exactly like https://github.com/openstack/openstack-ansible-os_keystone/commit/aefb58080337633e6d4c10bb2709c8d73ab9564d17:44
cloudnullit needs to run on all the hosts that are responsible for the ring.17:44
odyssey4meit's doing run_once now, right?17:45
cloudnulloh so you want run_once to be changed to that when ?17:45
odyssey4meyes, exactly17:45
odyssey4meso the host it runs on is deterministically the first in the play every time17:45
cloudnullOK.17:46
* cloudnull was being stupid 17:46
odyssey4medoesn't that make sense?17:46
*** fghaas has quit IRC17:48
*** hamzaachi has joined #openstack-ansible17:48
cloudnullyea!17:50
cloudnullI was thinking we were debating on what needs to run and where. however we're just making sure where we start the iteration is deterministic17:51
cloudnullwhich makes total sense17:51
* cloudnull updating that change17:51
jamesdentonCan I get some eyes on https://review.openstack.org/#/c/620604/ when someone has a sec? Simple change.17:52
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-os_swift master: Add variable for the ssh service and ensure its enabled  https://review.openstack.org/60780817:52
cloudnullI went with the list intersection version of that as that should skip the task entirely should a limit be used that doesn't include the swift_proxy group17:56
cloudnullanyone around able to review https://review.openstack.org/#/c/62039917:58
*** hamerins has quit IRC18:01
*** hamerins has joined #openstack-ansible18:02
cloudnullalso https://review.openstack.org/#/c/620386/ would be a great cherr-pick to get into rocky18:03
*** dcdamien has quit IRC18:05
cloudnullalso https://review.openstack.org/#/c/607814 - should i ever get cent to stop timing out18:05
*** shardy has quit IRC18:11
*** hamerins has quit IRC18:12
*** hamerins has joined #openstack-ansible18:12
pabelangerodyssey4me: comment on https://review.openstack.org/62065118:22
openstackgerritMerged openstack/ansible-role-systemd_service stable/rocky: Allow user/group to be overriden in unit section  https://review.openstack.org/62038618:25
pabelangerodyssey4me: and a 2nd comment, to fix syntax error18:25
*** cshen has joined #openstack-ansible18:26
pabelangerodyssey4me: those 2 changes seemed to have made os_keystone work18:29
pabelangertrying to see why glance failed now18:29
*** vollman has joined #openstack-ansible18:31
*** chhagarw has quit IRC18:41
pabelangerodyssey4me: ah, we'll need to same patch against all OS roles, since they too run on the utility container18:43
*** electrofelix has quit IRC18:45
*** DanyC_ has joined #openstack-ansible18:52
*** DanyC has quit IRC18:56
*** DanyC_ has quit IRC18:57
cloudnullhey all - https://bugs.launchpad.net/openstack-ansible/+bug/1805239 - regarding the issue, seems we have some gaps with rsysog and centos (RHEL). so the question is, do we fill in the logrotate gaps or do we refactor some of that code and try and not override the distro specific config files?18:58
openstackLaunchpad bug 1805239 in openstack-ansible "missing logrotate of the syslog, daemon.log " [Medium,Confirmed] - Assigned to Vadim Kuznetsov (vakuznet)18:58
*** hamerins has quit IRC19:00
*** hamerins has joined #openstack-ansible19:01
cloudnullseems like an important issue to get resolved, especially in stable for folks running centos19:01
*** preece has joined #openstack-ansible19:03
*** gisak has joined #openstack-ansible19:05
gisakhello guys19:06
gisakis it possible to put glance on controller node?19:06
gisakif yes, I guess in this case I need a br-storage for controller as well, right ?19:07
*** lbragstad has quit IRC19:08
jrossergisak: I think that is the default layout, you want br-storage on your controllers anyway I think19:10
*** lbragstad has joined #openstack-ansible19:12
gisakok, thanks )19:12
*** lbragstad has quit IRC19:12
*** lbragstad has joined #openstack-ansible19:14
*** lbragstad has quit IRC19:14
*** lbragstad has joined #openstack-ansible19:16
*** lbragstad has quit IRC19:16
*** vakuznet has quit IRC19:18
jrossercloudnull: a quick skim of the rsyslog client role the code looks quite blunt19:21
jrosserAt the very least it requires a ton of comments adding :/19:21
*** fghaas has joined #openstack-ansible19:23
*** radeks_ has joined #openstack-ansible19:25
*** gisak has quit IRC19:25
*** radeks has quit IRC19:26
*** jawad_axd has joined #openstack-ansible19:34
*** vakuznet has joined #openstack-ansible19:35
*** strattao has joined #openstack-ansible19:37
cloudnulljrosser totally agree. its largely been unchanged since it was originally created.19:41
jrosserI can pretty much not understand the intent there without studying it much harder - perhaps that points to a refactor?19:42
*** cshen has quit IRC19:45
*** mrhillsman is now known as mrhillsman|lunch19:46
*** strattao has quit IRC19:47
cloudnull++ its due for sure.19:47
cloudnullthat said, theres been work to decommission our use of rsyslog in favor of journald19:48
cloudnullso it may just be something of benefit for stable19:48
cloudnullwhich then begs the question, is it worth doing?19:49
*** jawad_axd has quit IRC19:49
*** jawad_axd has joined #openstack-ansible19:52
*** cshen has joined #openstack-ansible19:52
*** radeks has joined #openstack-ansible19:53
*** radeks_ has quit IRC19:55
*** jawad_axd has quit IRC20:15
*** cshen has quit IRC20:16
*** cshen has joined #openstack-ansible20:28
*** lbragstad has joined #openstack-ansible20:30
*** hamzaachi has quit IRC20:31
*** mrhillsman|lunch is now known as mrhillsman20:35
*** openstackgerrit has quit IRC20:36
*** dcdamien has joined #openstack-ansible20:39
*** ahosam has joined #openstack-ansible20:48
*** openstackgerrit has joined #openstack-ansible21:13
openstackgerritMerged openstack/openstack-ansible-ceph_client master: SUSE: Fix ceph packages for Leap 15  https://review.openstack.org/61553521:13
*** fghaas has quit IRC21:16
*** klamath has quit IRC21:26
*** klamath has joined #openstack-ansible21:26
*** vollman has quit IRC21:27
*** radeks has quit IRC21:32
*** lbragstad has quit IRC21:37
*** lbragstad has joined #openstack-ansible21:37
*** ansmith has quit IRC21:59
jamesdentonnoonedeadpunk Just FYI (open resolver issue): https://review.openstack.org/#/c/333829/22:10
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Include Swift AUTH_%(tenant_id)s suffix in rgw Keystone endpoint  https://review.openstack.org/61419422:27
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Implement AIO changes to allow test usage by roles  https://review.openstack.org/55179122:29
*** strattao has joined #openstack-ansible22:37
*** strattao has quit IRC22:39
*** ahosam has quit IRC22:51
*** ktims has joined #openstack-ansible23:23
ktimshi folks i wonder if someone can help me with openstack+ceph deployment with openstack-ansible. currently i'm stuck at lxc_hosts cache preparation. it appears that the container is getting neither ca-certificates nor the APT key for the ceph repository and i can't find any ansible variables that apply to this.23:25
*** cshen has quit IRC23:28
*** xjra has joined #openstack-ansible23:41
xjraWe find ourselves with a requirement to run multiple Horizon instances for clients in different security contexts; is it possible to have OSA deploy multiple Horizons with different configurations?23:43
xjrahttp://paste.openstack.org/show/736340/23:46
xjraOops, unrelated paste23:46

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!