Friday, 2019-07-12

*** weifan has joined #openstack-ansible00:25
*** weifan has quit IRC00:25
*** gyee has quit IRC00:28
*** dmsimard4 is now known as dmsimard00:32
*** irclogbot_1 has joined #openstack-ansible00:54
*** irclogbot_1 has quit IRC01:00
*** kplant has quit IRC01:15
*** irclogbot_2 has joined #openstack-ansible01:24
*** irclogbot_2 has quit IRC01:34
*** irclogbot_3 has joined #openstack-ansible02:24
*** irclogbot_3 has quit IRC02:30
*** altlogbot_0 has joined #openstack-ansible02:46
*** altlogbot_0 has quit IRC02:52
*** irclogbot_1 has joined #openstack-ansible03:20
*** BjoernT has joined #openstack-ansible03:21
*** irclogbot_1 has quit IRC03:26
*** BjoernT has quit IRC03:26
*** BjoernT has joined #openstack-ansible03:27
*** irclogbot_1 has joined #openstack-ansible03:50
*** irclogbot_1 has quit IRC03:56
*** BjoernT has quit IRC04:02
*** udesale has joined #openstack-ansible04:05
*** poopcat has quit IRC04:52
*** irclogbot_1 has joined #openstack-ansible04:58
*** pcaruana has joined #openstack-ansible05:13
*** irclogbot_1 has quit IRC05:16
*** aedc has quit IRC05:47
*** jbadiapa has quit IRC05:55
*** cshen has joined #openstack-ansible05:59
*** cshen has quit IRC06:03
*** cshen has joined #openstack-ansible06:07
*** altlogbot_3 has joined #openstack-ansible06:08
*** altlogbot_3 has quit IRC06:13
*** luksky11 has joined #openstack-ansible06:14
*** Namrata has joined #openstack-ansible06:32
*** irclogbot_2 has joined #openstack-ansible06:34
*** irclogbot_2 has quit IRC06:40
*** cshen has quit IRC06:41
*** ivve has joined #openstack-ansible06:46
*** aedc has joined #openstack-ansible06:58
*** miloa has joined #openstack-ansible07:02
miloaMorning07:02
*** cshen has joined #openstack-ansible07:03
*** ygk_12345 has joined #openstack-ansible07:06
ygk_12345hi all07:06
ygk_12345i facing an issue with my rocky setup07:07
ygk_12345the le agent is taking a lot of time to add subnet interfaces to the router07:07
ygk_12345*l3 agent07:07
*** Namrata has quit IRC07:13
*** cshen has quit IRC07:14
*** cshen has joined #openstack-ansible07:15
*** rpittau|afk is now known as rpittau07:18
*** tosky has joined #openstack-ansible07:19
*** jbadiapa has joined #openstack-ansible07:22
*** jbadiapa has quit IRC07:27
*** Namrata has joined #openstack-ansible07:29
*** cshen has quit IRC07:30
*** cshen has joined #openstack-ansible07:30
*** ygk_12345 has quit IRC07:31
*** cshen has quit IRC07:43
*** aedc has quit IRC07:48
*** shyamb has joined #openstack-ansible07:49
*** aedc has joined #openstack-ansible07:54
*** cshen has joined #openstack-ansible07:55
*** shyamb has quit IRC07:59
*** altlogbot_1 has joined #openstack-ansible08:00
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron master: service_setup: refactor service setup to a single file  https://review.opendev.org/67046008:01
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron master: service_setup: refactor service setup to a single file  https://review.opendev.org/67046008:03
*** altlogbot_1 has quit IRC08:04
*** yolanda has quit IRC08:08
*** yolanda has joined #openstack-ansible08:09
*** cshen has quit IRC08:10
*** altlogbot_2 has joined #openstack-ansible08:10
*** gkadam has joined #openstack-ansible08:15
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_gnocchi master: service_setup: refactor service setup to a single file  https://review.opendev.org/67047108:15
*** altlogbot_2 has quit IRC08:16
*** altlogbot_2 has joined #openstack-ansible08:16
*** gkadam has quit IRC08:17
*** cshen has joined #openstack-ansible08:20
*** altlogbot_2 has quit IRC08:22
*** altlogbot_1 has joined #openstack-ansible08:22
NamrataHi folks while upgrading from Rocky to Stein the upgrade script fails on openstack-ansible setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true'it fails at http://paste.openstack.org/show/754340/ and what I can see is there is no  galera_cluster_name variable in /etc/openstack_deploy/user_*.yml but there is wsrep_cluster_name08:24
Namrata in running galera cluster as the failure message suggests.08:24
Namratahow to solve this failure08:24
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ceilometer master: service_setup: refactor service setup to a single file  https://review.opendev.org/67047308:25
*** altlogbot_1 has quit IRC08:28
*** cshen has quit IRC08:33
*** heikkine has joined #openstack-ansible08:39
*** cshen has joined #openstack-ansible08:42
*** shyamb has joined #openstack-ansible08:46
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Differentiate AIO /etc/hosts entries for localhost and aio1  https://review.opendev.org/67039208:46
*** cshen has quit IRC08:46
*** aedc has quit IRC08:48
jrosserNamrata: going from rocky to stein changes the wsrep method for galera in a way that requires a manual intervention during the upgrade08:48
Namratajrosser what changes should I make08:49
Namrataand the manual intervention is for time being till the stein release is not prepared ?08:50
jrosserNamrata: i copy this for you off our internal wiki http://paste.openstack.org/show/754342/08:50
jrosserthe issue is that the upgrade tooling does not handle this properly08:51
Namratajrosser thanks a lot08:51
*** altlogbot_0 has joined #openstack-ansible08:52
Namratajrosser can you address the other two issues http://paste.openstack.org/show/754202/ http://paste.openstack.org/show/754194/08:54
Namratathis also needs manual intervention08:54
Namratawhile running upgarde scripy08:54
Namrata*script08:54
jrosseras far as i know passing the --skip-tags worked for me for the first one08:55
jrosserand the second one you need to manually checkout the right sha - i've not found the root cause of why the ansible git module does that when the equivalent cli command does not seem to08:55
Namrata--skip-tags works for me too but that should be not the ideal case or not I wanted to confirm08:56
Namrataand second one i  also checked out the right sha i am also not able to identify why there is an issue08:57
*** altlogbot_0 has quit IRC08:58
*** irclogbot_2 has joined #openstack-ansible08:58
jrosserNamrata: this git issue always happens with the same set of repos so i imagine it's a sublte bug in ansible which is only triggered by somthing unusual in those repos, like a missing tag or a historical force-push having made the history slightly inconsistent09:02
*** irclogbot_2 has quit IRC09:04
Namrata@jrosser thanks for addressing the issues. thanks for your time.09:04
*** luksky11 has quit IRC09:09
CeeMacmorning09:18
*** luksky11 has joined #openstack-ansible09:32
*** vnogin has joined #openstack-ansible09:44
*** shyamb has quit IRC10:05
*** irclogbot_0 has joined #openstack-ansible10:30
*** yolanda has quit IRC10:31
*** yolanda has joined #openstack-ansible10:32
*** irclogbot_0 has quit IRC10:38
*** shyamb has joined #openstack-ansible10:40
*** irclogbot_0 has joined #openstack-ansible10:41
*** irclogbot_0 has quit IRC10:44
*** ivve has quit IRC11:01
*** ivve has joined #openstack-ansible11:01
*** altlogbot_1 has joined #openstack-ansible11:03
*** miloa has quit IRC11:05
*** shyam89 has joined #openstack-ansible11:08
*** altlogbot_1 has quit IRC11:08
*** shyamb has quit IRC11:08
*** altlogbot_2 has joined #openstack-ansible11:11
*** kplant has joined #openstack-ansible11:14
*** altlogbot_2 has quit IRC11:16
*** cshen has joined #openstack-ansible11:25
*** cshen has quit IRC11:34
*** cshen has joined #openstack-ansible11:39
*** shyam89 has quit IRC11:39
*** shyam89 has joined #openstack-ansible11:48
*** vnogin has quit IRC11:49
*** cshen has quit IRC11:53
*** aedc has joined #openstack-ansible11:54
*** udesale has quit IRC11:59
*** udesale has joined #openstack-ansible12:00
*** cshen has joined #openstack-ansible12:04
*** altlogbot_3 has joined #openstack-ansible12:07
*** altlogbot_3 has quit IRC12:08
*** goldyfruit has quit IRC12:22
*** cshen has quit IRC12:22
*** electrofelix has joined #openstack-ansible12:23
*** aedc has quit IRC12:27
*** cshen has joined #openstack-ansible12:37
evrardjpmorning CeeMac12:37
*** Namrata has quit IRC12:42
*** markvoelker has quit IRC12:45
*** viks___ has quit IRC12:46
*** aedc has joined #openstack-ansible13:00
CeeMacAfternoon surely :)13:10
*** aedc has quit IRC13:12
*** gtarnaras has joined #openstack-ansible13:13
*** cshen has quit IRC13:17
*** luksky11 has quit IRC13:22
*** cshen has joined #openstack-ansible13:24
guilhermespmornings!13:31
*** BjoernT has joined #openstack-ansible13:32
*** shyam89 has quit IRC13:33
*** cshen has quit IRC13:34
*** irclogbot_3 has joined #openstack-ansible13:35
*** irclogbot_3 has quit IRC13:38
*** goldyfruit has joined #openstack-ansible13:49
*** cshen has joined #openstack-ansible13:50
*** cshen has quit IRC13:54
NobodyCamGood Morning OSA folks, just wanted to ask a dumb question:14:04
NobodyCamto add a new cinder back end to a existing cloud one would:14:06
NobodyCammodify openstack_user_config.yml adding the new backend info14:06
NobodyCamthen run openstack-ansible os-cinder-install.yml ?14:06
*** irclogbot_2 has joined #openstack-ansible14:09
*** altlogbot_1 has joined #openstack-ansible14:13
*** ivve has quit IRC14:14
*** chandankumar is now known as raukadah14:14
openstackgerritMohammed Naser proposed openstack/openstack-ansible master: masakari: add missing is_metal field  https://review.opendev.org/67056814:15
openstackgerritMohammed Naser proposed openstack/openstack-ansible master: inventory: don't crash with missing global overrides  https://review.opendev.org/67056914:16
*** cshen has joined #openstack-ansible14:16
*** spatel has joined #openstack-ansible14:17
mnaserhmm14:20
mnaseranyone ever ran osa from a non-root user?14:20
mnaserlooks like its trying to connect as $USER instead or 'root'14:21
*** cshen has quit IRC14:38
*** rpittau is now known as elfosardo14:52
*** markvoelker has joined #openstack-ansible14:53
jrossermnaser: have you seen that the utility playbook doesn’t update the sym links to the clip tools correctly?14:55
jrosserclip/cli14:55
mnaserjrosser: this is for updates right?14:56
jrosserAnything that already is linked doesn’t get relinked to point to the binaries in a new venv when we do upgrades14:56
mnasernot fresh deploys14:56
*** markvoelker has quit IRC14:56
mnaseroof14:56
mnaserwhy is that14:56
mnaserdoes ansible not relink things14:56
jrosserThe task is not smart enough14:56
jrosserIt’s using shell/command rather than a more native ansible-ism14:57
jrosserAnd I could really see a good reason for that14:57
jrosserCouldnt14:57
mnaserjrosser: i think its the black magic around regex stuff with python-whatever14:58
mnaserbut yes this seems like something that should be refactored asily14:58
mnaseri think you can just do the clean up in _openstack_client_list14:58
*** ansmith has quit IRC15:23
*** cshen has joined #openstack-ansible15:44
goldenfriHi everyone. So nova alias requires multiple repeating alias entries for multiple aliases, but OSA will only take the last alias entry. Any ideas how I can work around this?15:45
*** cshen has quit IRC15:48
*** gtarnaras has quit IRC15:49
*** ansmith has joined #openstack-ansible15:57
*** adrianreza_ has joined #openstack-ansible16:01
*** gyee has joined #openstack-ansible16:01
evrardjpjrosser: I am validating this whole chain to be consistent, but else I don't think we should do this. I think we should be using the config file overrides for it.16:02
jrosserevrardjp: is like an example of what you think would acceptable in that case16:05
evrardjpI have started to merge stuff to still be consistent16:05
evrardjpI prefer consistency first16:05
* mnaser is trying to get the context for ^16:05
evrardjpit's an improvement already to do this16:05
evrardjpI mean we could have in user_variables some kind of override16:05
evrardjpit would mean one per service ofc, but they will probably be all the same, therefore a reuse of yaml anchors could be a win16:06
evrardjpbut that's already a good improvement to me16:06
jrossermnaser: I asked evrardjp to take a look at https://review.opendev.org/#/q/topic:bind-to-mgmt+(status:open+OR+status:merged)16:06
evrardjpmnaser: it's about running services not on 0.0.0.016:07
jrosserevrardjp: I see this as not just about the AIO16:07
jrosserI’d like to be binding to the mgmt network by default n all deploys16:08
evrardjpyup I guessed that, because I saw you changed the groupvars16:08
mnaserin terms of security yes, this pretty much values what jrosser needs/wants in terms of network segration16:08
mnaserand also once again, doing a 3 node ctl plane with haproxy+keepalive on metal is broken in our current system16:08
mnaserwhich imho is a pretty common use case.. 3 controllers, N computes .. it's what tripleo does too16:09
evrardjpyeah it's common16:09
jrossermnaser: how far off fixing that is my patch set?16:09
evrardjpjrosser: I like the fact we're providing this for everyone in fact16:10
evrardjpand I think your way is probably the most elegant way to provide that for everyone16:10
evrardjpI am maybe too careful in terms of group vars16:10
evrardjp:D16:10
evrardjpI have to run16:11
evrardjpttyl!16:11
mnaserjrosser: i think its pretty much most of it, i dunno if guilhermesp had a chance to share our configs but thats pretty much mostly it16:12
mnaserthe neat thing is just add haproxy and watch it all blow up in flames :P16:12
jrossermnaser: I think if that goes on .101 as the vip we might get lucky16:13
jrosserGive or take horizon or something16:14
jrosserI need to do a bit more on Galera as some tasks assume 127.0.0.1 right now but that looks fairly ok to fix16:15
mnaserjrosser: deploying an all metal galera cluster as we speak16:15
mnaserill give the group_vars when done16:15
*** elfosardo is now known as rpittau|afk16:27
jrossermnaser: i think this is what needs dealing with https://github.com/openstack/openstack-ansible-galera_server/search?q=127.0.0.1&unscoped_q=127.0.0.116:34
jrosserwe've talked a lot about moving from group vars to config overrides, and it would be good to see a worked example of that16:35
jrosseras you get one chance only to define the overrides for a particular config file, and we need to feed the "default wiring" in as well as anything else the user wants to add16:36
openstackgerritMohammed Naser proposed openstack/openstack-ansible-galera_server master: cleanup: refactor into galera_mariadb_service_name  https://review.opendev.org/67059916:42
mnaserjrosser: pushing fixes shortly16:52
openstackgerritMohammed Naser proposed openstack/openstack-ansible-galera_server master: setup: don't try to login to create users  https://review.opendev.org/67060116:52
openstackgerritMohammed Naser proposed openstack/openstack-ansible-galera_server master: setup: don't try to login to create users and upgrade  https://review.opendev.org/67060116:54
guilhermespmnaser jrosser evrardjp I think we have an opportunity now to test the bind-br-mgmt16:55
guilhermespwe have a couple of deploy ahead all on metal16:55
guilhermespI think more things will come up through the process16:55
guilhermespthe vars I have are all like specific for an ipv6 deployment.16:55
mnaserfwiw those patches + http://paste.openstack.org/show/754349/ = a succesful galera deploy here (with bonus of ipv6 too)16:56
guilhermespthat would be useful for us to work in the future btw16:56
mnaseryeah its a lot of |ipwrap-ing for ipv616:56
guilhermespyeah the ipwarp for example16:56
mnaseri just also noticed we create users N times in galera16:57
guilhermespit is somethign that we can have wihtout affecting ipv4 deplomnts16:57
mnaser++16:57
jrosserCan we take a second pass across all this for ipv6?16:57
guilhermesp@mnaser is this using the db-refactor?16:57
guilhermespno?16:57
mnaserjrosser: i think it would be good after we land this yeah16:58
mnaserguilhermesp: i dont think we applied that logic there16:58
guilhermespjrosser: I think I can find some time yeah, just need to finish some priorities on our side16:58
jrosserteamwork++ :)16:58
* mnaser would be happy to see us run ipv6 only jobs in gate16:58
jrosserI’d be totally happy with that in my deployments too16:59
*** udesale has quit IRC16:59
openstackgerritMerged openstack/openstack-ansible-os_ironic master: Fix CentOS installtion  https://review.opendev.org/67029417:09
openstackgerritMerged openstack/openstack-ansible-os_ironic master: Add integrated coverage  https://review.opendev.org/66008317:09
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server master: Fix detection of rabbitmq restart  https://review.opendev.org/67027417:27
*** nicolasbock has joined #openstack-ansible17:29
jrossermnaser: will this https://review.opendev.org/#/c/670599/ leave the service with the old name running at upgrade time?17:40
*** poopcat has joined #openstack-ansible17:54
*** kplant has quit IRC18:04
openstackgerritMerged openstack/openstack-ansible-os_nova master: Add default settings for nova metadata and service bind addresses  https://review.opendev.org/67004618:10
*** electrofelix has quit IRC18:18
mnaserjrosser: not sure actually..18:20
openstackgerritMohammed Naser proposed openstack/openstack-ansible stable/stein: ceph: stop adding repos  https://review.opendev.org/67061718:39
*** irclogbot_2 has quit IRC18:49
*** luksky11 has joined #openstack-ansible18:50
*** irclogbot_1 has joined #openstack-ansible18:51
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: [WIP] Bind services to mgmt network addresses  https://review.opendev.org/67005119:27
*** kplant has joined #openstack-ansible20:32
*** goldyfruit has quit IRC20:35
redkriegis there any supported way to ensure openstack service users get the same uid/gid between computes?  I've got some computes where nova is 998 and others where it's 999, need them to be the same for network share permissions20:44
jrosserredkrieg: nova_system_user_uid / gid20:48
redkriegjrosser: thanks much.  I've got ceph-mds working btw, I'll probably need some tips on getting a patch together to submit upstream next week20:49
redkriegnow to reinstall all my computes...20:49
jrossergood news on ceph, what are you going to use cephfs for?20:49
redkriegif you promise not to yell at me...  /var/lib/nova/instances/snapshots and /var/lib/nova/instances/_base20:50
redkriegtrust me, I /really/ don't want to be using it for those20:50
jrosser:)20:50
spatelwhile talking about ceph, is it ok to add different size of OSD in ceph or better to keep same size?20:52
redkriegI know next to nothing about the inner workings of ceph, I've basically crash-coursed getting it set up this week20:54
kplantredkrieg: you can use ceph as a backend for nova without using cephfs20:55
kplantif that's what you're looking for20:55
redkriegI know, I'm using it for glance and cinder, using local storage for nova instance disks, but using cephfs for _base and snapshots because they can be very large and I don't have the local storage on every node for them (for instance, if my node is at 90% capacity and someone who takes up 25% of it wants to take a snapshot)20:56
kplantah, gottcha20:57
kplanti didn't want to deal with that so i just force everyone to use cinder volumes20:57
kplantlazy :]20:57
*** pcaruana has quit IRC20:59
redkriegmy clients will be more performance hungry than cinder volumes can support for me :)20:59
spatelredkrieg: you are saying cephfs provide more performance compare to cinder volumes?21:00
*** TxGirlGeek has joined #openstack-ansible21:00
kplanti think redkrieg is referring to the nova disks for the instances vs. cinder volumes21:00
kplanton drives local to the compute nodes21:00
redkriegyeah, cephfs is being used for image cache and temporary snapshot data in my setup21:01
kplantredkrieg: i'll probably run into that soon and need to start delivering compute instances with local nvme drives :-|21:01
kplantfor now they're more than happy with our ceph cluster which is all nvme21:01
kplantbut ceph doesn't have the best latency21:01
redkriegyou're probably going to want my ceph-mds patch then :D21:01
kplantoh?21:04
redkriegif you plan to offload some things to cephfs, yeah21:04
kplantoh, no21:05
kplanti'll just say "these are you local disks with no redundancy, have fun"21:05
kplantno snapshots21:05
redkrieghah, that works too21:05
kplantif they care about their data it should be on a cinder volume21:05
*** weifan has joined #openstack-ansible21:07
*** weifan has quit IRC21:18
mnaserfyi ceph for image cache might be problematic because i think with nova the image cache isnt actually a cache but actually where the base images are stored if you're using qcow221:22
*** weifan has joined #openstack-ansible21:22
jrossermnaser: less success now we bind galera address https://review.opendev.org/#/c/670051/21:22
jrosserthe jobs that are working are happy accident because of this https://github.com/openstack/openstack-ansible/blob/dc5729ad6f10b7e6083d60c2270f75090cf3d5f4/inventory/group_vars/all/infra.yml#L3521:23
mnaserjrosser: nice!21:23
mnaserand oh21:24
redkriegmnaser: we configured it so _base is only temporarily needed during instance creation.  it creates a copy of the image for nova to actually use as the disk.21:24
mnaseryeah that'll do it lol21:24
mnaseroh so you're not doing cow21:24
jrosseri think they fail becasue the mysql_db ansible module fails as it defaults to 'localhost', so the db_setup fails21:24
mnaserwell im seeing rabbit fail to start up, not galera jrosser21:24
mnaserunless i looked at the wrong job21:24
jrosserprobably multiple errors21:24
jrosseri think i need to nuke my local AIO and start again21:25
redkriegmnaser: yeah that was it.      use_cow_images: False21:25
redkriegtakes about 7 seconds to spin up an ubuntu server instance with this setup, so not too bad21:25
jrossermnaser: this not too happy either http://logs.openstack.org/51/670051/12/check/openstack-ansible-deploy-aio_metal-opensuse-15/9824a47/logs/openstack/aio1-mysql_logs/galera_server_error.log.txt.gz21:26
mnaserweird21:30
jrosserslightly annoyingly the metal-bionic job passes21:30
jrosserand i'm not sure why there is a distro dependance here21:30
*** spatel has quit IRC21:35
*** weifan has quit IRC21:38
*** weifan has joined #openstack-ansible21:40
*** weifan has quit IRC21:41
*** weifan has joined #openstack-ansible21:42
*** weifan has quit IRC21:47
*** BjoernT has quit IRC21:47
*** luksky11 has quit IRC21:48
*** ianychoi has quit IRC21:54
*** KeithMnemonic has quit IRC22:02
openstackgerritMerged openstack/ansible-config_template master: Add option to use a remote source  https://review.opendev.org/66975822:32
openstackgerritMohammed Naser proposed openstack/openstack-ansible-tests master: sync: add db_setup.yml  https://review.opendev.org/67067222:43
mnaserguilhermesp: when you have some spare time, if you can do fix up the 4 roles so we can finish up that work :D22:46
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Add neutron_api_bind_address default variable  https://review.opendev.org/67028522:52
openstackgerritMerged openstack/openstack-ansible-openstack_hosts stable/queens: use ansible_hostname to construct canonical_hostname in /etc/hosts  https://review.opendev.org/62168823:17
*** tosky has quit IRC23:20
*** nicolasbock has quit IRC23:22
*** weifan has joined #openstack-ansible23:23
*** weifan has quit IRC23:28

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!