Wednesday, 2018-12-12

*** tosky has quit IRC00:01
*** macza has quit IRC00:23
*** weezS has quit IRC00:44
*** jawad_axd has joined #openstack-ansible00:47
*** jawad_axd has quit IRC00:51
*** dave-mccowan has joined #openstack-ansible01:15
*** ansmith has joined #openstack-ansible01:38
*** markvoelker has quit IRC01:41
*** hwoarang has quit IRC01:48
*** jawad_axd has joined #openstack-ansible01:49
*** ansmith has quit IRC01:52
*** jawad_axd has quit IRC01:53
*** hwoarang has joined #openstack-ansible01:53
*** cshen has quit IRC02:09
*** jawad_axd has joined #openstack-ansible02:51
*** jawad_axd has quit IRC02:55
*** dave-mccowan has quit IRC03:52
*** hwoarang has quit IRC03:56
*** hwoarang has joined #openstack-ansible04:03
*** udesale has joined #openstack-ansible04:17
*** dhellmann has quit IRC05:19
*** dhellmann has joined #openstack-ansible05:20
*** radeks has joined #openstack-ansible05:20
*** cshen has joined #openstack-ansible05:50
*** hamzaachi has quit IRC05:58
*** ThiagoCMC has quit IRC05:59
*** nurdie has joined #openstack-ansible06:05
*** hamzaachi has joined #openstack-ansible06:05
nurdieI have an Openstack Ocata deployment that was bootstrapped with OSA and all controller services containerized (python envs, etc). It was just upgraded to Pike with yum. Is there any way to come back from that?06:06
*** hwoarang has quit IRC06:16
*** hwoarang has joined #openstack-ansible06:22
*** cshen has quit IRC06:25
*** hamzaachi has quit IRC06:34
chandan_kumarodyssey4me: jrosser evrardjp Hello06:47
chandan_kumarodyssey4me: jrosser evrardjp can we get reviews on this patch https://review.openstack.org/622999 ?06:47
chandan_kumararxcruz: kopecmartin|off odyssey4me: jrosser evrardjp will i merge this two patch https://review.openstack.org/623187 and https://review.openstack.org/623413 in a single patch?06:48
*** aedc has joined #openstack-ansible07:01
*** cshen has joined #openstack-ansible07:11
*** aedc has quit IRC07:14
*** cshen_ has joined #openstack-ansible07:17
*** cshen has quit IRC07:19
*** openstackgerrit has quit IRC07:29
*** ivve has quit IRC07:30
*** jawad_axd has joined #openstack-ansible07:34
*** sum12 has quit IRC07:35
jrosserodyssey4me: they require each other?07:36
*** sum12 has joined #openstack-ansible07:37
*** hamzaachi has joined #openstack-ansible07:48
*** kopecmartin|off is now known as kopecmartin07:48
*** trident has quit IRC07:58
*** trident has joined #openstack-ansible08:00
*** ahosam has joined #openstack-ansible08:01
*** ahosam has quit IRC08:01
*** ahosam has joined #openstack-ansible08:01
*** ahosam has quit IRC08:02
*** shardy has joined #openstack-ansible08:05
evrardjpchandan_kumar: having a look now08:11
*** shardy has quit IRC08:21
*** shardy has joined #openstack-ansible08:22
jrosserOh, hmm ignore me, irc phone client confusion, I’m busy till late afternoon today08:32
*** markvoelker has joined #openstack-ansible08:44
*** tosky has joined #openstack-ansible08:46
*** markvoelker has quit IRC08:49
*** jonher_ has joined #openstack-ansible08:57
*** fresta_ has joined #openstack-ansible08:57
*** jonher has quit IRC09:00
*** jonher_ is now known as jonher09:00
*** fresta has quit IRC09:01
*** ivve has joined #openstack-ansible09:04
*** DanyC has joined #openstack-ansible09:19
admin0\o09:26
admin0morning all09:27
admin0nurdie, you meant an OSA deployment got (automatically) upgraded via yum upgrade ?09:28
admin0did it work :D ?09:28
*** openstackgerrit has joined #openstack-ansible09:37
openstackgerritArx Cruz proposed openstack/openstack-ansible-os_tempest master: Better tempest black and whitelist management  https://review.openstack.org/62160509:37
*** markvoelker has joined #openstack-ansible09:39
openstackgerritChristian Zunker proposed openstack/openstack-ansible-os_cloudkitty master: Add missing libxml2 packages  https://review.openstack.org/62461809:52
*** ivve has quit IRC09:52
*** sm806 has quit IRC09:54
*** sm806 has joined #openstack-ansible09:55
*** sm806 has quit IRC09:55
*** sm806 has joined #openstack-ansible09:55
*** electrofelix has joined #openstack-ansible10:04
*** gkadam has joined #openstack-ansible10:05
*** gkadam has quit IRC10:05
nurdieadmin0: No it definitely didn't work, lol. It's still broken10:08
nurdieHaving to weed through all of the old Ocata bugs to even get the OSA play to run cleanly. It broke the network, services, alll of it10:09
admin0someone accidently nuked the galera cluster of an osa cluster which had paying customers .. not sure if your work is more or mine fixing this mess :)10:14
admin0so the instances are running .. rest all = gone10:14
admin0will make a nice blog post on how to re-add the base image as image and copy the disk delta to shift the workload from now defunt to new cluster10:15
nurdieadmin0: ouch! We do nightly backups of our galera cluster10:19
admin0in mine, i do hourly10:20
admin0but i retain only 24 hours back10:20
admin0nurdie, why are you stil in ocata ?10:24
admin0what i have seen is ..  during the told you hold to not upgrade, something changes in the OS  (apt/yum update) that brings in new libs .. so later when you try to upgrade, it starts giving issues which are not documented because those use-changes or changes were not there when the new release was done .. so holding back for too long = more issues10:25
admin0during the period*10:25
admin0so now give or take its 1.5+ years .. if you say nothing has changed in your system during this time, then the same osa/ocata might play well ( with the original issues ) .. but if things have changed, you might be facing those  issues which only you will face10:28
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server stable/queens: Fix SSL support  https://review.openstack.org/62463310:30
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server stable/queens: Fix Galera self-signed SSL functionality  https://review.openstack.org/62448210:31
*** DanyC has quit IRC10:33
odyssey4menurdie the only thing that really matters in the openstack deployment if openstack is hosed is the database - if that's safe, you can recover it10:34
odyssey4menurdie so you have a choice - but personally I'd recommend upgrading from ocata to pike, then pike to rocky asap... rocky really is the first stable centos deployment we've had10:35
nurdieodyssey4me: If I have a copy of the the database after all VMs were shutdown, can I redeploy and use that DB? It has a Ceph backend, and I'm terrified of losing all of the VMs10:36
odyssey4menurdie you may hit trouble if your repositories were upgraded to the new centos 7.6 or even 7.5 - we might have to port back some fixes for that to work in ocata or pike10:37
odyssey4meif they're on 7.0 then I think you're ok, but I don't use centos so I can't fully vouch for it10:37
odyssey4menurdie ok, so are all the vm's running and they still have network connectivity?10:38
nurdieodyssey4me: Everything is down except Ceph10:38
nurdieWe turned off all VMs before upgrade attempt, then backed up the DB, again, when we realized it was going south10:39
nurdieMultiple compute hosts, 3 controllers (all running containerized services)10:39
odyssey4menurdie ok, and is ceph deployed with the OSA deployment or seperately?10:40
*** DanyC has joined #openstack-ansible10:42
nurdieseparately10:42
odyssey4mensmeds I noticed that https://review.openstack.org/624482 brought in part of https://review.openstack.org/572939, so I've ported that back ( https://review.openstack.org/624633 ) and done a clean pick of your patch on top of it10:43
odyssey4menurdie well, if you have some way of testing the process to iron out the kinks and to understand what you're in for then I'd encourage you to do that first - otherwise you're in for a risky and wild ride and could lose things10:46
admin0when does 18.1.1 be released .. so that i do not have to cherry pick that ovs patch again and again10:46
admin0plus there might be more changes10:47
odyssey4meadmin0 whenever https://review.openstack.org/624406 is approved, 18.1.1 will be released10:47
odyssey4meonce that's done, then https://review.openstack.org/624407 will be clear to merge10:47
odyssey4menurdie what exactly happened - you did 'yum upgrade' from 7.0 to 7.6 and everything fell apart?10:48
nurdieyum install centos-openstack-pike; yum update10:50
nurdielol10:50
nurdieOn an OSA deploy*10:50
odyssey4menurdie oh dear10:50
odyssey4menurdie was that done on all the hosts, or only part of them?10:50
nurdieAll of them10:50
odyssey4mewell, that was silly :p10:51
nurdieMhmm. Wasn't my idea10:51
odyssey4meI don't know what 'yum install centos-openstack-pike' actually installs, but is it possible to remove it?10:51
odyssey4meAnd then does 'yum update' do an upgrade to 7.6, or what does it do exactly?10:52
nurdieBut I learned a lot about OSA today lol10:52
nurdieYes, yum update pulled in the openstack pike updates + the centos updates10:53
odyssey4menurdie ok, so as long as your /etc/openstack_deploy folder is intact, and your database is intact, the control plane can pretty much be rebuilt from scratch - if you rebuild the hosts using the same MAC/IP's then it should be fine10:54
odyssey4mefrom there your concern is the compute hosts10:55
odyssey4methere you should be able to uninstall the openstack packages and make sure you purge all their config, then run the OSA deployment over the top of them to get them running again10:55
nurdieI still need to go through the upgrade process, because of DB schemas, etc, right? Which means I still need a working OS deployment lol10:56
nurdieJe pense, but je ne connais pas10:56
odyssey4meI would recommend just going with Pike - OSA will handle the database upgrades and all that.10:56
nurdiehmm10:56
odyssey4meYou'll probably have to handle the OSA changes yourself by hand though.10:56
odyssey4methese parts: https://docs.openstack.org/openstack-ansible/pike/user/manual-upgrade.html10:57
nurdieHaha! I just saw a page like that a little while ago that had a disclaimer about the upgrade script being not production safe10:58
nurdieSafer than a yum update on a CentOS box? :p10:58
odyssey4methe upgrade process is a one size fits all, so it's not guaranteed for every environment which is why we had that disclaimer10:58
odyssey4meand, quite honestly, I don't know if it works on CentOS - it's quite possible that one or two things will fall apart10:59
odyssey4mebut in your case if you're wiping the env, then there's only a small part of that process which matters...11:00
odyssey4mehttps://docs.openstack.org/openstack-ansible/pike/user/manual-upgrade.html#clean-up-old-facts11:00
odyssey4mehttps://docs.openstack.org/openstack-ansible/pike/user/manual-upgrade.html#update-configuration-and-environment-files11:00
odyssey4mehttps://docs.openstack.org/openstack-ansible/pike/user/manual-upgrade.html#update-user-secrets-file11:00
odyssey4mehttps://docs.openstack.org/openstack-ansible/pike/user/manual-upgrade.html#clean-up-the-ceph-ansible-galaxy-namespaced-roles11:00
odyssey4mewith that your /etc/openstack_deploy will be prepped11:01
odyssey4meeverything else is just the same as a standard deployment - except in your case you'll be stopping after setup-hosts.yml to ensure that you restore your database to the galera containersbefore continuing with setup-infrasructure/setup-openstack11:02
*** rodolof has joined #openstack-ansible11:02
*** udesale has quit IRC11:02
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance stable/rocky: Fix permissions on /var/lib/glance/cache  https://review.openstack.org/62466211:03
*** udesale has joined #openstack-ansible11:03
odyssey4menurdie I would definitely recommend putting together a few VM's for a test environment to work through the process to make sure you're familiar with it.11:04
nurdieMuch gratitude for the tips, odyssey4me! We may stab that route, because the Ocata OSA repos seem like a giant rabbit hole11:05
odyssey4menurdie yeah, it's old11:05
odyssey4menurdie as admin0 recommended earlier, it's better to keep up with releases as quickly as possible - and especially when something like a new centos releases, make sure you're up to date before using it... best to actually maintain your own centos mirrors and only update when you're ready for it11:06
nurdieWe have DB copies of the Galera cluster and critical infra VMs before the upgrade. Our stack is already essentially a giant test environment now o_011:07
admin0nurdie, i still find my cluster with a nuked db more fun :)11:07
odyssey4meadmin0 that, sir, is a mess11:07
nurdieadmin0: hahahaha you have fun with that, and your hourly rates :p11:08
odyssey4mehehehe, yeah - when paid by the hour in that situation, as tough as it is, there is a silver lining11:08
odyssey4memnaser jrosser I wonder whether it'd be prudent for us to backport https://review.openstack.org/623515 so that centos builds using pike/queens also work with centos 7.611:09
odyssey4melet me just do it11:10
admin0the silver lining here is that  you can re-upload the base file as an image, create a new instance with the same name and just copy the disk11:10
*** jawad_axd has quit IRC11:10
admin0now i am inclined to create a test cluster with ceph, nuke the db and see how far we can go11:10
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts stable/queens: prep: remove old machinectl workarounds  https://review.openstack.org/62466511:11
odyssey4meadmin0 hopefully the networks in the projects aren't too complex11:12
odyssey4mebut still - restoring the VM is the easy part... getting all the networking back to what it was, and all the security groups, etc is only really possible if they have some sort of documentation, which they always do - right?! :p11:13
*** sum12 has quit IRC11:18
jrosserodyssey4me: i think it would be good yes - we're on 7.6 hosts for everything arent we now?11:18
*** sum12 has joined #openstack-ansible11:19
*** sum12 has quit IRC11:19
*** sum12 has joined #openstack-ansible11:20
admin0lots of scripting .. to dumpxml every running instance, to get the username project and image and then map the security groups using iptables11:20
admin0a lot of work, but turned out not to be scary .. as all the client instances are running11:21
openstackgerritlei zhang proposed openstack/openstack-ansible master: Add a note about stale env.d files  https://review.openstack.org/62466711:22
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for running tempestconf from packages  https://review.openstack.org/62299911:29
*** markvoelker has quit IRC11:30
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts stable/pike: prep: remove old machinectl workarounds  https://review.openstack.org/62467411:30
*** ansmith has joined #openstack-ansible11:50
*** jamesdenton has quit IRC11:56
*** jamesdenton has joined #openstack-ansible12:00
*** markvoelker has joined #openstack-ansible12:05
*** ansmith has quit IRC12:05
gundalowhttps://github.com/ansible/ansible/pull/37113 `Added support for missing options capabilities and root_device in properties of os_ironic.py ansible module` Could I get a final review from someone here? Thanks in advance.12:06
odyssey4megundalow looking, although I may not have enough understanding to know whether it's the right thing to do12:30
gundalowThanks :)12:31
odyssey4meah, I see that julia has verified it which works as a SME review for me12:31
odyssey4megundalow I see you've done some feedback, do you want a revision before merging? the functionality seems right to me12:32
gundalowOK, I'll edit that and merge. Thanks :)12:33
jamesdentonodyssey4me Quick chat about https://review.openstack.org/#/c/454450/ if you have a sec. I could not find the referenced (upcoming) documentation for post-upgrade (P->Q), specifically how to migrate from container to bare metal. Seems to me like nothing changes w/r/t existing containers, but NEW network_hosts will have agents deployed on BM. This sort of conflicts with the language of the release notes IMO. ->12:35
jamesdenton https://docs.openstack.org/releasenotes/openstack-ansible/queens.html#relnotes-17-0-0-stable-queens-upgrade-notes12:35
odyssey4mejamesdenton yeah, unfortunately cloudnull never put anything together as a guide - and it would be really nice to have something in docs to help people12:37
odyssey4meas I understand it, there will - after the upgrade, be agents on bare metal and in containers - although when I was working on the patch to remove the old containers I did not see that12:38
odyssey4meI'm not all that familiar with neutron, so I decided to leave it to others to figure out.12:38
odyssey4mein theory you should already have new agents and can work through scheduling (using the at&t tool perhaps, or native tooling) all routers/networks/etc to the ones on bare metal, disable the ones in containers, then once they're all moved over you can wipe the old container12:39
jamesdentonRIGHT. My experience so far is that the old containers remained (expected) and nothing changed in the inventory, so agents were not deployed on BM. I did add a new network_host, though, and agents were deployed on BM to that. I suspect that some inventory munging will need to be done to force BM deploy to existing infra nodes during the upgrade, and then the container removal exercise post-upgrade12:39
odyssey4meit should be totally automatable, but it needs someone with the time and expertise to work that out12:40
jamesdentonthe router and network shuffle, as you say12:40
*** dave-mccowan has joined #openstack-ansible12:40
jamesdentonwell, i'll give the docs a workup. and the automation may need to be left to more capable hands12:41
odyssey4meaha, ok - that sort of makes sense... so the issue is that there is already an inventory entry for that network_host... so it keeps deploying to the same one12:41
odyssey4mehmm, I wonder if there's a way we can do it online somehow12:41
jamesdentonI'm trying to get some additional info from this bug report to see if/how they deviated from the upgrade notes: https://bugs.launchpad.net/openstack-ansible/+bug/180477012:42
openstackLaunchpad bug 1804770 in openstack-ansible "Pike -> Queens Upgrade: Documentation for OVS setup and neutron agent rebalancing missing" [High,Confirmed] - Assigned to James Denton (james-denton)12:42
odyssey4meIt seems that it'd be possible to remove the container from the inventory, and with the new env.d when re-running ansible it won't recreate the inventory entry for the container - but then we lose any ways of running ansible against it. :/12:42
jamesdentonso you think removing the neutron_agents_containers from inventory won't result in a new entry for the respective BM host?12:44
odyssey4mejamesdenton I don't think it will - but the trick is that we need to ensure that the new agents are on the BM host before we remove the cotainer.12:45
pabelangermorning! Do you think we can approve https://review.openstack.org/624407/ today, to update SHAs for stable/rocky for 18.1.2? I guess that needs a release patch first?12:45
odyssey4mepabelanger that won't go into gate until https://review.openstack.org/624406 merges12:45
odyssey4meand it's likely that will only happen once tonyb has seen it, which is likely late tonight for you12:46
odyssey4meearly morning for him :)12:46
odyssey4mejamesdenton that's a nice bug report :)12:48
pabelangerodyssey4me: ack, thanks12:49
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Recovery the galera cluster with no gvwstate.dat file  https://review.openstack.org/62468712:53
openstackgerritMerged openstack/openstack-ansible stable/rocky: nspawn: correct task names  https://review.openstack.org/62218312:53
*** vakuznet has joined #openstack-ansible13:07
*** hamzaachi has quit IRC13:19
*** hamzaachi has joined #openstack-ansible13:20
*** markvoelker has quit IRC13:28
*** vollman has joined #openstack-ansible13:30
openstackgerritMerged openstack/openstack-ansible-ops master: MNAIO: Recovery the galera cluster with no gvwstate.dat file  https://review.openstack.org/62468713:31
*** rjgibson has joined #openstack-ansible13:44
*** ansmith has joined #openstack-ansible13:45
*** markvoelker has joined #openstack-ansible14:01
arxcruzodyssey4me: evrardjp do you guys mind take a look at https://review.openstack.org/#/c/621605/ ?14:13
arxcruzwhen you have time14:13
odyssey4mearxcruz um, what the heck? I get the impression that you're not aware of how to make use of group_vars/host_vars/extra_vars ...14:14
odyssey4mearxcruz also, part of the point of the joint working on this role is to have a common black/whitelist rather than managing two14:15
*** fresta has joined #openstack-ansible14:17
arxcruzodyssey4me: okay... sorry...14:17
*** jonher_ has joined #openstack-ansible14:17
arxcruzwe can have a common black/whitelist14:18
*** maharg101 has joined #openstack-ansible14:19
*** rodolof has quit IRC14:20
*** jonher has quit IRC14:21
*** jonher_ is now known as jonher14:21
*** rodolof has joined #openstack-ansible14:21
*** fresta_ has quit IRC14:21
openstackgerritAndy Smith proposed openstack/openstack-ansible master: Add qdrouterd role for rpc messaging backend deployment  https://review.openstack.org/62418414:22
maharg101hi, just reading https://bugs.launchpad.net/openstack-ansible/+bug/1657518 regarding firewalld, has anyone built up a list of firewall requirements for openstack that could be shared ?14:32
openstackLaunchpad bug 1657518 in openstack-ansible "Hosts role should disable or remove firewalld" [High,Fix released] - Assigned to Major Hayden (rackerhacker)14:32
*** irclogbot_0 has quit IRC14:36
*** irclogbot_0 has joined #openstack-ansible14:51
*** cshen_ has quit IRC14:54
*** Miouge has quit IRC14:55
*** Miouge has joined #openstack-ansible14:57
*** DanyC has quit IRC15:07
*** DanyC has joined #openstack-ansible15:07
*** DanyC has quit IRC15:11
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove the log container in the AIO config  https://review.openstack.org/61952615:14
noonedeadpunkI've got a bunch of small patches for review, so if someone has time for it - you're welcome https://goo.gl/8QpssC15:14
*** markvoelker has quit IRC15:18
odyssey4menoonedeadpunk I'll look through them now.15:18
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_masakari stable/rocky: Add missing docs/releasenotes tests  https://review.openstack.org/61663515:23
*** sum12 has quit IRC15:28
*** sum12 has joined #openstack-ansible15:28
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Adds ability to enable domain dropdown list on login page  https://review.openstack.org/62225615:32
openstackgerritkourosh vivan proposed openstack/openstack-ansible-os_trove stable/rocky: Add segmentation option  https://review.openstack.org/62474115:33
*** DanyC has joined #openstack-ansible15:33
*** DanyC has quit IRC15:34
*** DanyC has joined #openstack-ansible15:35
*** rodolof has quit IRC15:38
*** rodolof has joined #openstack-ansible15:39
noonedeadpunkodyssey4me: regarding your comment here https://review.openstack.org/#/c/623643/ - I've faced with the problem during upgrade to the Rocky, and this workaround worked for me, and didn't brought any problems. However offered option will be defenitely safer.15:39
odyssey4menoonedeadpunk ok, I can approve it - but if you think my suggestion is safer then could you push up a patch to change it in master15:39
noonedeadpunkSo, should I abandon this cherry-pick, place offered patch for the master, and then squash them and backport to rocky?15:39
noonedeadpunkyep, sure:)15:40
openstackgerritcaoyuan proposed openstack/openstack-ansible-os_zaqar stable/rocky: Add description section  https://review.openstack.org/62474315:41
odyssey4menoonedeadpunk after testing that it works for you, of course :)15:41
*** vollman has quit IRC15:52
*** kopecmartin is now known as kopecmartin|off16:05
*** cshen has joined #openstack-ansible16:05
*** gyee has joined #openstack-ansible16:19
*** udesale has quit IRC16:19
nsmedsodyssey4me: saw your change - understood. thanks for helping16:20
noonedeadpunkodyssey4me: I've tried offered option (and several common as well), and every ends up with this error: http://paste.openstack.org/show/737147/16:24
odyssey4menoonedeadpunk argh.. must be a bug in it somewhere - if you're concerned enough to figure it out, please do - I don't mind either way16:26
spotzabout if needed16:27
spotzhad 2 days where the channel was broke for me:(16:27
noonedeadpunkActually, I ended up with these quotes, as wasn't able to find more proper solution. But I'll do another try16:28
*** cshen has quit IRC16:30
noonedeadpunkI think, the point is, that when jinja tries to get an address from dict, ansible_ib1.87d2 should be quoted (or dot somehow escaped) that it was interpreted as a single key16:32
*** macza has joined #openstack-ansible16:34
*** cshen has joined #openstack-ansible16:46
*** radeks has quit IRC16:49
*** cshen has quit IRC16:51
jrossermnaser: one for you here https://review.openstack.org/#/c/624336/16:52
*** electrofelix has quit IRC16:53
*** cshen has joined #openstack-ansible16:56
mnaserjrosser: thats perfect thanks17:18
jrosserthat should unlock lxc_container_create tests i think17:19
*** DanyC has quit IRC17:22
openstackgerritMerged openstack/openstack-ansible-os_masakari stable/rocky: Add missing docs/releasenotes tests  https://review.openstack.org/61663517:23
openstackgerritMerged openstack/openstack-ansible-os_cloudkitty master: Add missing libxml2 packages  https://review.openstack.org/62461817:26
mnaserjrosser: it will17:27
mnaseri think it will just unlock the patch that removes that tho17:27
openstackgerritMerged openstack/openstack-ansible-os_horizon master: Adds ability to enable domain dropdown list on login page  https://review.openstack.org/62225617:29
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/rocky: Ensure that LDAP config is deployed on all keystone hosts  https://review.openstack.org/62443017:30
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: Ensure that systemd-resolved is present in the Centos container  https://review.openstack.org/62433617:37
*** spatel has joined #openstack-ansible17:38
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts stable/rocky: Ensure that systemd-resolved is present in the Centos container  https://review.openstack.org/62477117:38
spateljamesdenton: morning!17:38
*** ianychoi has joined #openstack-ansible17:38
spateli have neutron VLAN question if you around17:39
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/queens: Ensure that LDAP config is deployed on all keystone hosts  https://review.openstack.org/62443317:40
spatelI should say afternoon :)17:40
openstackgerritMerged openstack/openstack-ansible-os_cinder master: Fix ansible deprecation warnings  https://review.openstack.org/62165617:41
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Adds ability to set no-resolv for dnsmasq.  https://review.openstack.org/62007617:44
openstackgerritMerged openstack/openstack-ansible-lxc_hosts stable/queens: prep: remove old machinectl workarounds  https://review.openstack.org/62466517:44
jamesdentonhello spatel. whats up?17:44
jrossermnaser: http://logs.openstack.org/34/624434/1/gate/openstack-ansible-functional-centos-7/1dc28e7/logs/host/lxc-cache-prep-commands.log.txt.gz17:45
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ceilometer master: Add ability to override meters.yaml and event_definitions.yaml  https://review.openstack.org/62323917:47
spateljamesdenton: I have vlan_200 on my openstack for public and its near to full and i am working with ISP to get new public pool, so is it possible i can add new public subnet to existing vlan ?17:49
spatelIf not that i have to create new VLAN on all physical switch ( I have 200 switches in network :( )17:49
jamesdentonyou can. the upstream router/gateway for that network needs to be able to handle multiple L3 subnets on a single interface, though.17:50
jamesdentonmost routers can do that. firewall devices (like Cisco ASA) maybe not so much17:50
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Clean-up and simplify the major upgrade  https://review.openstack.org/62477317:51
*** hamzaachi has quit IRC17:51
noonedeadpunkfolks, does anyone know if ceilometer and telemetry projects are alive? As nobody cares about reviews and, actually, filled bugs there...17:51
spatelYes. new subnet i have to setup on my gateway switches..17:51
jamesdentonspatel so then it's just a matter of 'openstack subnet create --network EXISTING_VLAN_NET ...'17:52
spatelThis is what i did to create my first VLAN network and subnet17:52
spatelneutron subnet-create net_pub_vlan_200 71.xxx.xxx.0/24 --name sub_pub_vlan_200 --allocation-pool start=71.xxx.xxx.2,end=71.xxx.xxx.254 --dns-nameservers 8.8.8.8 --gateway=71.xxx.xxx.117:52
spatelCan i use same command to add 72.xxx.xxx.0/24 subnet?17:52
jamesdentonyes. just give it a different name17:53
jamesdentongateway will be different, start/end, etc.17:53
spatelDifferent name for subnet right but keep "net_pub_vlan_200" same ?17:53
jamesdentonright17:53
spatelsomething like this..17:54
spatelneutron subnet-create net_pub_vlan_200 71.xxx.xxx.0/24 --name sub_new_vlan_200 --allocation-pool start=72.xxx.xxx.2,end=72.xxx.xxx.254 --dns-nameservers 8.8.8.8 --gateway=72.xxx.xxx.117:54
openstackgerritMerged openstack/openstack-ansible-os_trove master: Add segmentation option  https://review.openstack.org/62353317:54
jamesdentonbe mindful of the cidr you're passing, that would need to be 72.xxx also17:54
spateljamesdenton: typo..... :)17:55
spatelyes it should be 72....17:55
spateljamesdenton: thank you!!!17:55
jamesdentonlooks good to me. sure17:55
spatelI have almost 175 compute node in openstack :)  how do i find out what will be the limit of cloud?17:56
*** rodolof has quit IRC17:56
jamesdentonpush it till it breaks, then N-117:56
spateltouch wood everything looks good so far.. load on infra nodes around 2 to 3%..17:56
jrosserspatel: you doing any vrrp or anything for the gateway of those subnets? allocating .2 to .254 leaves you little wiggle room for that sort of thing17:57
spatelI have place order for 100 more compute nodes... so wondering should i build new cloud or keep adding in existing one :)17:57
jamesdentongood catch jrosser17:57
jrosseri tend to leave .1 to .9 for such things and have a rule that .5 is vrrp gateway in all /24's17:58
spateljrosser:  1, 2, 3 first ip will be used by HRSP or VRRP..17:58
jrosserthen you have room for two routers, and spare to add/change if you need to upgrade or stuff like that17:58
noonedeadpunkodyssey4me: It seems, that 'ansible_{{ hostvars[inventory_hostname] | json_query(find_bridge) | replace('-','_') }}' has to be quoted, as the problem occurs inside json_query(), when it see extra dot inside of the interface name (like ansible_ib1.87d2.ipv4.address), while the query should be kinda "'ansible_ib1.87d2'.ipv4.address"17:58
jrosseri guess in this case i'm a little old-school in assigning actual ip to the router interfaces in addition to the vrrp vip18:00
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/ocata: Ensure that LDAP config is deployed on all keystone hosts  https://review.openstack.org/62443618:00
spateljrosser: is that VRRP for DHCP?18:01
jamesdentonspatel your physical gateway device18:02
jamesdentonis it HA?18:02
jrosserno, to make the subnet gateway H/A18:02
spateli am using physical router gateway so i don't need any VRRP for routing..18:02
openstackgerritMerged openstack/openstack-ansible-os_keystone master: Force force-tlsv12 only  https://review.openstack.org/62324018:03
spatelI don't have any vRouter on openstack..18:03
jrosserno this is on your hardware18:03
jrosserif you have two routers you really should arrange with something like VRRP that the gw ip will fail over between them18:03
spatelYes that is true... floating ip ( I have HSRP setup )18:04
spatelbut you only need 3 ip address to setup VRRP or HSRP18:04
spateljust trying to understand i am on same page.. or i am missing details here18:05
jamesdentonyeah, what he's implying is that make sure your dhcp allocation range doesn't overlap with those IPs18:05
jamesdentonotherwise neutron may have out an IP configured on the router itself18:05
spateljamesdenton: :) got it..18:05
jamesdenton*hand18:05
spatelThat command i posted was just example.. i didn't realized .2 to .254 range i specified there18:06
jamesdentonahh ok18:06
spateljamesdenton: also DHCP instance also take 3 IPs right from public pool?18:06
spatelin 3 node cluster18:06
jamesdentonif you have 3 DHCP agents, yes18:07
spatelperfect! that is what i thought18:07
spateljamesdenton: how do i measure openstack limit ?  any components i should keep eye on it.. before i say *NOT MORE* compute node18:08
spatelright now i am keep adding compute node every weeks...18:08
jamesdentonothers may be better equipped to answer that. if you're not using l3 agent, l2 pop, etc. then the number would be higher. i would say watch DB and Rabbit18:09
jamesdentoni've always heard ~250 is a good spot18:09
spatelhmm! okay.. currently i have 175 so i will keep eye..18:10
spateljamesdenton: at some point i want to migrate my existing LinuxBridge to OVS (to use dpdk )18:11
spateldon't know how hard its going to be but.. i have to go that path.. SR-IOV is just extra pain with performance :)18:11
jamesdentoni think we're a ways out from that.18:11
jamesdentonhave you kicked the tires on OVS+DPDK to see if you get performance close to that of SR-IOV?18:12
spatelNo i didn't18:12
spateli will soon but.. 2019 goal18:13
*** maharg101 has quit IRC18:14
*** maharg101 has joined #openstack-ansible18:17
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron stable/rocky: Adds ability to set no-resolv for dnsmasq.  https://review.openstack.org/62477918:23
openstackgerritMatthew Thode proposed openstack/openstack-ansible-ops master: Chage cpu model to allow for nested virt to work  https://review.openstack.org/62478018:23
prometheanfireodyssey4me: if around, this should fix things... https://review.openstack.org/62478018:24
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron stable/queens: Adds ability to set no-resolv for dnsmasq.  https://review.openstack.org/62478118:29
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron stable/queens: Adds ability to set no-resolv for dnsmasq.  https://review.openstack.org/62478118:31
*** shardy has quit IRC18:35
*** goldenfri has joined #openstack-ansible18:44
openstackgerritMerged openstack/openstack-ansible master: Added masakari-dashboard repo to openstack_services  https://review.openstack.org/61906218:55
jrossernoonedeadpunk: I don’t think it’s true that no one cares about the ceilometer roles, it’s just they’ve been in a bad place for so long that core reviewers like me don’t necessarily have it running in our labs, so it’s hard to get up to speed and be able to make a judgement on patches18:58
jrosserFWIW I think it’s really great that they are getting some attention18:59
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron stable/pike: Adds ability to set no-resolv for dnsmasq.  https://review.openstack.org/62478619:01
openstackgerritMerged openstack/openstack-ansible stable/pike: Update all SHAs for 16.0.24  https://review.openstack.org/62442719:05
openstackgerritMerged openstack/openstack-ansible stable/queens: Update all SHAs for 17.1.6  https://review.openstack.org/62441419:05
openstackgerritMerged openstack/openstack-ansible stable/rocky: Add retries for utility container distro package installs  https://review.openstack.org/62402419:05
openstackgerritMerged openstack/openstack-ansible stable/rocky: Zuul: Simplify the integrated test playbooks  https://review.openstack.org/62438519:05
noonedeadpunkjrosser: I meant about ceilometer itself to be honest:) I will never say that anyone here doesn't care about smth19:05
*** spatel has quit IRC19:07
noonedeadpunkIRC #openstack-telemetry is deserted, bugs are not reviewed for month, even if they have related fixes in gerrit...19:08
jrosserOh that’s sad19:08
noonedeadpunkSo I'm just became worried about the destiny of the project, as we're building billing based on it right now.19:09
jrosserI would like to have more of this working in my cloud19:09
jamesdentonhttps://julien.danjou.info/lessons-from-openstack-telemetry-deflation/19:09
jamesdentoni wouldn't put your eggs in that basket19:11
noonedeadpunkjamesdenton: That's really sad...19:12
jamesdentonagreed19:12
mnasernoonedeadpunk: you should start getting more involved with the project and take the lead :D19:13
*** AB2019 has joined #openstack-ansible19:14
jamesdentoncheers for noonedeadpunk - OpenStack Telemetry PTL19:15
noonedeadpunkI'm not so perfect at Python to take a leadership:) I'd wish I had19:16
jrosserYou might find even small involvement surprisingly fruitful, particularly with as a deployer with a use case, and a foot here in a deployment project. A lot could get fixed quickly....19:17
noonedeadpunkYep, probably you're right19:20
noonedeadpunkOh, I've read the book written by this guy - never knew that he's a telemetry PTL19:21
*** spatel has joined #openstack-ansible19:24
spatelFolks... i have quick question, i want to change something in /etc/rsyslog.d/50-default.conf file so how do i rollout my change?19:25
spatelI am seeing this file is static file and not using any template variables19:25
spatel# /etc/ansible/roles/rsyslog_client/files/50-default.conf19:26
spatelwhat are the options i have to edit this file and rollout to make sure next update won't wipe out my changes?19:26
jamesdentonwhat are you needing to change?19:27
spatel*.*;local7,auth,authpriv,cron,daemon,mail,news.none  -/var/log/syslog19:28
spatelCentOS using /var/log/messages  ( my monitoring agent reading this files )19:29
spateli want to change /var/log/syslog to /var/log/messages19:29
*** vollman has joined #openstack-ansible19:29
spatel i want to change /var/log/syslog to /var/log/messages19:29
jamesdentonHmm. symlink?19:29
-spatel- [root@ostack-compute-75 ~]# ls -l /var/log/syslog19:30
-spatel- ls: cannot access /var/log/syslog: No such file or directory19:30
spatelSomething is broken in syslog deployment of OSA19:30
spatelits not logging any logs..19:30
spatelThis is what i found if i remove "$PrivDropToUser syslog" & "$PrivDropToGroup syslog" from /etc/rsyslog.conf file then it start logging..19:32
spatelWe have all centOS folks and when they see no /var/log/messages they freaked out :(19:33
jamesdentonmay be an oversight for centos deploys on our part.19:34
spatelwhy don't in OSA make simple variable to symlink it or change path of file..19:34
jamesdentonhttps://bugs.launchpad.net/openstack-ansible/+filebug19:34
*** maharg101 has quit IRC19:35
spatelI have find /etc/ansible/roles/rsyslog_client/templates/rsyslog.conf.j2  where i can change that but no variable :(19:35
spateljamesdenton: i did file bug last week but no activity.. yet19:36
*** maharg101 has joined #openstack-ansible19:38
jamesdentonwho is the owner of /var/log/messages?19:41
jamesdentonroot:root?19:41
vakuznetspatel related bug https://bugs.launchpad.net/openstack-ansible/+bug/1805239 what bug did you file?19:42
openstackLaunchpad bug 1805239 in openstack-ansible "missing logrotate of the syslog, daemon.log " [Medium,Confirmed] - Assigned to Vadim Kuznetsov (vakuznet)19:42
jamesdentonhttps://bugs.launchpad.net/openstack-ansible/+bug/180726819:42
openstackLaunchpad bug 1807268 in openstack-ansible "CentOS rsyslog bug" [Undecided,New]19:42
spatelvakuznet: looking..19:48
spateljamesdenton:19:48
-spatel- [root@ostack-osa templates(keystone_admin)]# ls -l /var/log/messages19:48
-spatel- -rw------- 1 root root 710528 Dec 12 14:40 /var/log/messages19:48
jrosserWasn’t there a patch for that?19:50
spatelhttps://bugs.launchpad.net/openstack-ansible/+bug/180726819:53
openstackLaunchpad bug 1807268 in openstack-ansible "CentOS rsyslog bug" [Undecided,New]19:53
spatelHere you go vakuznet19:53
redkriegDoes anyone know if this is still the best way to do multiple regions?  I'd be fine with only having one keystone and pointing the other region to it if that's practical, I just don't see a good way to do it.  https://developer.rackspace.com/blog/keystone-to-keystone-federation-with-openstack-ansible/19:53
*** Adri2000 has quit IRC19:54
*** Adri2000 has joined #openstack-ansible19:56
*** rodolof has joined #openstack-ansible19:56
spateljamesdenton: you got it19:57
spatelwhat are the option i have now ?19:57
spatelOr should i write custom ansible module to roll out my change outside OSA ?19:58
spatelit will be ugly!19:58
*** nurdie has quit IRC20:07
*** cshen has quit IRC20:34
*** cshen has joined #openstack-ansible20:36
openstackgerritJames Denton proposed openstack/openstack-ansible-rsyslog_client master: [WIP] Templatize rsyslog configuration files  https://review.openstack.org/62480520:36
jamesdentonspatel You can try that: https://review.openstack.org/62480520:37
jamesdentonbut i can't guarantee it's the right approach, or there may be something in the works20:37
*** nurdie has joined #openstack-ansible20:38
spateloh! i got it what you did, looks good20:40
spatelshould i do that by hand until patch get merge ?20:40
jrossermnaser: i'm sure there was some discussion previously about this rsyslog/centos stuff - was there a patch?20:40
spatelall your changes..20:40
jamesdentonspatel there may already be something in the works by mnaser so it may be in vain at this point.20:41
spatelhmm!20:42
jrosseri've not found anything so might have been imagining it - but it was certainly discussed - maybe in the meeting bug triage actually20:43
*** nurdie has quit IRC20:43
jamesdentonFor you, today, you might need to roll changes by hand until something official comes down. I think you can create a symlink for /var/log/syslog, and it would appear in /var/log/messages20:43
spatelI am writing quick ansible module to roll out those changes..20:46
spateljamesdenton: thanks again!20:47
jamesdentonsure. lemme know!20:47
spateljamesdenton: this is what i did.. and rollout to all node http://paste.openstack.org/show/737166/21:04
jamesdentonafter adding the vars?21:06
jamesdentonor that was your playbook after downloading the patch21:07
spatelI did quick hack.. manually rollout those file.. i didn't use your patch21:10
spateli thought may be something in on the way so waiting for final merge21:10
jamesdentonok so you changed the values in the files then21:10
spatelin short create ansible module outside OSA and rollout two files and restart rsyslog21:11
jamesdentoncool21:11
spatelYes!! i did21:11
spateleverything working now21:11
spatellogs are flowing now21:11
jamesdentonsubsequent playbook runs may overwrite that, just FYI21:11
spatelYes i know that :) so for now i made those changes in role/rsyslog_client module21:13
spatelso new box will get correct fix.. until we have something in upstream21:14
jamesdentongood deal21:14
spotznoonedeadpunk: I thought ceilometer was dead, at least the API was removed. How long have things been sitting? Folks may be on holiday21:15
spatelI have internal git to check-in those changes so later i can review what changes i mad overtime..21:15
*** vakuznet has quit IRC21:21
*** vnogin has joined #openstack-ansible21:25
*** hw_wutianwei_ has quit IRC21:25
*** vnogin has quit IRC21:30
*** rodolof has quit IRC21:32
*** rodolof has joined #openstack-ansible21:32
*** ansmith has quit IRC21:33
*** spatel has quit IRC21:53
*** DanyC has joined #openstack-ansible22:02
*** dcdamien has joined #openstack-ansible22:02
*** mmercer has quit IRC22:22
*** AB2019_ has joined #openstack-ansible22:22
*** DanyC_ has joined #openstack-ansible22:25
openstackgerritMerged openstack/openstack-ansible stable/rocky: Fixed ability of usage interfaces/bridges with dots (aliases)  https://review.openstack.org/62364322:27
*** AB2019 has quit IRC22:29
*** andreykurilin has quit IRC22:29
*** eumel8 has quit IRC22:29
*** logan- has quit IRC22:29
*** AB2019_ is now known as AB201922:29
*** DanyC has quit IRC22:29
*** logan- has joined #openstack-ansible22:31
*** trident has quit IRC22:32
*** trident has joined #openstack-ansible22:34
*** cshen has quit IRC22:40
openstackgerritAntony Messerli proposed openstack/openstack-ansible-ops master: Better handling for resuming leapfrog upgrade  https://review.openstack.org/62445523:14
*** dave-mccowan has quit IRC23:19
*** rodolof has quit IRC23:21
*** rodolof has joined #openstack-ansible23:21
*** DanyC_ has quit IRC23:24
*** mmercer has joined #openstack-ansible23:35
*** rodolof has quit IRC23:35
*** rodolof has joined #openstack-ansible23:35
*** rodolof has quit IRC23:37
*** rodolof has joined #openstack-ansible23:38
*** DanyC has joined #openstack-ansible23:40
*** DanyC has quit IRC23:44
*** dcdamien has quit IRC23:47
*** mmercer is now known as Daemoen23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!