Friday, 2017-03-17

*** LinStatSDR has joined #openstack-ansible00:11
cloudnullxdfil: I don't believe so. should be ok to randomize and go.00:15
openstackgerritMerged openstack/openstack-ansible-os_cinder master: Add support for cinder v3 api  https://review.openstack.org/44650300:16
*** LinStatSDR has left #openstack-ansible00:18
*** woodard has quit IRC00:21
*** woodard has joined #openstack-ansible00:21
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: define the default kernel statically  https://review.openstack.org/44679000:24
*** Jeffrey4l has quit IRC00:25
*** jrobinson has quit IRC00:27
*** weezS has quit IRC00:35
*** sukesh has quit IRC00:38
*** gouthamr has joined #openstack-ansible00:38
*** deadnull has quit IRC00:48
*** acormier has joined #openstack-ansible00:53
*** SerenaFeng has joined #openstack-ansible00:59
*** cuongnv has joined #openstack-ansible01:00
*** MasterOfBugs has quit IRC01:01
*** acormier has quit IRC01:02
*** acormier has joined #openstack-ansible01:02
*** jrobinson has joined #openstack-ansible01:03
*** acormier has quit IRC01:08
*** weezS has joined #openstack-ansible01:19
*** schwicht has joined #openstack-ansible01:22
*** galstrom_zzz is now known as galstrom01:26
*** SerenaFeng has quit IRC01:26
*** SerenaFeng has joined #openstack-ansible01:37
*** schwicht has quit IRC01:44
*** schwicht has joined #openstack-ansible01:45
*** schwicht_at_work has joined #openstack-ansible01:47
*** schwicht has quit IRC01:50
*** Jeffrey4l has joined #openstack-ansible01:53
*** jamielennox is now known as jamielennox|away01:56
*** jamielennox|away is now known as jamielennox02:01
*** eki__ has joined #openstack-ansible02:02
eki__someone interested in trying to help me figure out why I can't ping my openStack routers from my aio host?02:04
*** schwicht_at_work has quit IRC02:12
*** jwitko has quit IRC02:13
*** Jeffrey4l has quit IRC02:14
*** Jeffrey4l has joined #openstack-ansible02:14
*** cuongnv has quit IRC02:29
*** cuongnv has joined #openstack-ansible02:32
*** sanfern has joined #openstack-ansible02:33
*** winggundamth has joined #openstack-ansible02:37
*** sanfern has quit IRC02:39
*** Jeffrey4l has quit IRC02:41
*** Jeffrey4l has joined #openstack-ansible02:41
*** cathrich_ has joined #openstack-ansible02:47
*** cathrichardson has quit IRC02:47
*** Mahe has quit IRC02:48
*** Mahe has joined #openstack-ansible02:50
*** gouthamr has quit IRC03:01
*** weezS has quit IRC03:02
*** acormier has joined #openstack-ansible03:04
*** weezS has joined #openstack-ansible03:04
*** jrobinson has quit IRC03:08
*** galstrom is now known as galstrom_zzz03:10
*** crushil has joined #openstack-ansible03:20
*** weezS has quit IRC03:22
*** SerenaFeng has quit IRC03:24
*** SerenaFeng has joined #openstack-ansible03:24
*** acormier has quit IRC03:27
*** galstrom_zzz is now known as galstrom03:28
*** SerenaFeng has quit IRC03:29
*** LinStatSDR has joined #openstack-ansible03:29
*** weezS has joined #openstack-ansible03:33
*** markvoelker_ has quit IRC03:53
*** udesale has joined #openstack-ansible04:00
*** LinStatSDR has quit IRC04:05
*** galstrom is now known as galstrom_zzz04:07
openstackgerritRavi Kumar Boyapati proposed openstack/openstack-ansible-rsyslog_client master: Fix remote logging template  https://review.openstack.org/44683804:14
eki__I am just going to write the problem, if anyone has any advice it would be awesome.05:06
eki__I am installing openstack aoi on a machine with single interface (eth0). The installation works perfectly, everything seems fine. I can create instances, networks,routers and access horizon. The problem is I can't figure out how to configure the routers so the instances can access internet.05:06
eki__What should I configure as external network? Is it the br-vxlan or br-vlan networks or my real physical network? There must be few ways to do this, but what would be the correct way of doing this with aoi, or does it simply depend on what I have configured in openstack_user_config.yml, or can I use the default configuration?05:06
eki__I have been reading the documentation for a bit but can't figure it out05:08
eki__the main problem seems to be that I can't connect to the routers I create05:10
*** sanfern has joined #openstack-ansible05:12
*** crushil has quit IRC05:12
*** Jack_Iv has joined #openstack-ansible05:24
*** sanfern has quit IRC05:28
*** sanfern has joined #openstack-ansible05:28
xdfileki__ what is your eth0 config ?05:31
eki__first tried with static ip05:32
xdfilcan i see it?05:32
eki__paste here?05:33
eki__just nuked the whole install too05:33
xdfilidk if that will work05:33
xdfilppl use those paste services05:33
*** shausy has joined #openstack-ansible05:34
eki__http://pastebin.com/NFRgVEtp05:34
xdfillogan- I fixed the storwize issue, been working on bgp05:34
eki__that is what I have now since I nuked05:34
eki__I did have promisc + broadcast before too05:34
xdfillogan- trying to figure out how to make the container bgp iface ip static05:35
eki__I also tried moving the ip settings to a bridge but I currently don't have real understing what I am supposed to do05:35
eki__is it supposed to work as it is on clean install?05:36
*** NikhilS has joined #openstack-ansible05:36
xdfileki__ make a bridge interface and put the IP from eth0 on the bridge interface instead05:36
eki__will do05:36
xdfilthen you should be able to specify the bridge interface as a flat network in neutron05:37
eki__should I name the bridge to something that openstack expects or separete?05:37
xdfilbr-provider is common05:37
xdfilI hate all the bridge names in openstack05:39
xdfili hate wahat the openstack comunity names things05:39
xdfileki__ full disclosure: ive never done an AIO install and I've given people bad advice in the past05:41
eki__ok05:41
xdfilI don't know what an AIO looks like05:41
eki__should be fine :D05:41
eki__http://pastebin.com/4b72fktL05:41
xdfili'm already confused by the fact you have no brdiges for trhe containers?05:41
eki__I have been hitting my head to wall for a day now05:41
eki__xdfil: there will be once I run the installer05:42
eki__"nuked the whole thing, and in process to build it from start"05:42
eki__sorry shoudl have specified that05:42
xdfilohhhh05:42
eki__just running the bootstrap-aio.sh05:44
eki__have to take look at the openstack_user_config.ym before running the all the other playbooks05:45
eki__xdfil: thanks for helping out :D05:46
*** weezS has quit IRC05:48
eki__http://pastebin.com/feZ7Z25x that is the default openstack_user_config.yml  with aio05:48
eki__or what was genereated for me05:48
eki__there is no mention of eth0 nor br-provider, should there be?05:49
eki__and state of my interfaces http://pastebin.com/37R15JGm05:51
eki__in my /etc/network/interfaces.d/osa_interfaces.cfg <- automatically generated (http://pastebin.com/pUQUwVmG) there is this line05:55
eki__"    # To provide internet connectivity to instances"05:55
eki__which is on br-vxlan05:55
eki__makes me think I am supposed to configure router with external interface on the same network as the br-vxlan ( in my case 172.29.240/22)05:56
eki__aand those exact rules are configure for eth005:57
eki__shoudln't those be br-provider05:57
*** SerenaFeng has joined #openstack-ansible05:57
eki__will give this idea a try05:58
xdfiljust do br-mgmt and br-vlan05:58
xdfilbut the "flat" onew05:59
xdfilbut the "flat" one05:59
xdfilnot the "vlan" one05:59
eki__remove others?05:59
xdfilyes05:59
xdfilyou don't need vxlan storage06:00
*** SerenaFeng has quit IRC06:00
xdfilactually06:00
xdfilkeep both vlan bridges06:00
eki__ok06:00
eki__so no removing of bridges06:01
eki__or just keep the vlan  ones06:01
eki__aa06:01
xdfili think i see whats going on they share out the br-vlan as vlan 1 to do a provider network06:01
eki__brmgmt and br-vlan and br-vlanx06:01
xdfilvlans and mgmt06:02
xdfilno vlanx06:02
xdfilwell wait06:02
eki__*vxlan06:02
xdfilit doesnt mattee06:02
*** Jeffrey4l has quit IRC06:02
xdfilif you want to play with vlanx do it06:02
xdfilbut you dont need it to do the basic stuff06:03
eki__for now I want to figure out how to connect to instance from the host06:03
xdfillike connect instances to the internet06:03
xdfilogh06:03
eki__and internet06:03
xdfilconnecting to the host can be anissue06:03
xdfilIt might not be possible06:04
eki__hmmm06:04
eki__connecting from the host --> instance?06:04
*** SerenaFeng has joined #openstack-ansible06:04
xdfilyes its a common issue in virtulization06:05
eki__how would you connect to it then?06:05
eki__from other computer in external network?06:05
xdfilyes short answer06:05
*** jascott1- is now known as jascott106:06
xdfilidk i'm sure if you messed with it it could work, I dont thing the linux bridge is the issue but maybe LXC networking stuff or apparmor would get in your way06:06
xdfilmaybe what i;m thinking only applys to macvtap06:07
eki__hmmm06:07
eki__in short no removing of stuff?06:09
eki__I will build with the defaults , and this time make the router have ip from external group and try pingin from different host06:13
eki__didn't think that would be a problem06:13
*** Jeffrey4l has joined #openstack-ansible06:17
*** Amit82 has joined #openstack-ansible06:34
Amit82Hi All06:36
eki__hi06:36
Amit82muxdeamon: Yesterday, you shared your openstack_user_config.yml and /etc/network/interfaces files with me.06:37
Amit82could you please tell that in "external_lb_vip_address: openstack.willsher.systems" openstack.willsher.systems corresponds to IP on which interface06:38
Amit82?06:38
*** sanfern has quit IRC06:38
Amit82does it bind to ip on en0?06:38
*** sanfern has joined #openstack-ansible06:39
*** yolanda has joined #openstack-ansible06:45
*** McMurlock1 has joined #openstack-ansible06:54
*** fxpester has joined #openstack-ansible07:03
*** basilAB_ has quit IRC07:18
*** SerenaFeng has quit IRC07:19
*** aetaric has quit IRC07:19
*** mrhillsman has quit IRC07:20
*** vaishali has joined #openstack-ansible07:23
*** basilAB has quit IRC07:23
*** aetaric has joined #openstack-ansible07:24
*** basilAB has joined #openstack-ansible07:26
*** SerenaFeng has joined #openstack-ansible07:30
*** mrhillsman has joined #openstack-ansible07:33
*** manheim has joined #openstack-ansible07:35
*** manheim has quit IRC07:37
*** manheim has joined #openstack-ansible07:38
*** manheim has joined #openstack-ansible07:38
*** pcaruana has joined #openstack-ansible07:43
*** Jack_Iv has quit IRC07:52
*** Jack_Iv has joined #openstack-ansible07:52
*** Jack_Iv has quit IRC07:56
*** sunjon_ has quit IRC07:58
*** Jack_Iv has joined #openstack-ansible08:01
*** sanfern has quit IRC08:03
*** pmannidi has quit IRC08:04
*** sanfern has joined #openstack-ansible08:04
*** DimGR has joined #openstack-ansible08:06
*** Jack_Iv has quit IRC08:22
*** sanfern has quit IRC08:23
*** Jack_Iv has joined #openstack-ansible08:23
*** Jack_Iv has quit IRC08:27
*** shausy has quit IRC08:33
*** shausy has joined #openstack-ansible08:34
*** udesale has quit IRC08:41
*** vnogin has joined #openstack-ansible08:41
*** udesale has joined #openstack-ansible08:41
admin0morning all \o08:45
eki__morning08:54
eki__xdfil: no luck pinging from outside09:00
eki__:/09:00
*** shardy has joined #openstack-ansible09:01
eki__is there some security/firewall settings that need to be changed to be able to ping routers?09:02
*** sanfern has joined #openstack-ansible09:05
*** david-lyle_ has joined #openstack-ansible09:18
*** david-lyle has quit IRC09:18
admin0check routing :)09:20
*** lwiecek has joined #openstack-ansible09:20
*** karimb has joined #openstack-ansible09:24
*** qiliang27 has quit IRC09:33
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/mitaka: Update all SHAs for 13.3.17  https://review.openstack.org/44646809:38
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Update all SHAs for Pike 2017-03-16  https://review.openstack.org/44639709:40
*** hamza has joined #openstack-ansible09:43
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Revert to setuptools 33.1.1  https://review.openstack.org/44692209:44
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/newton: Revert to setuptools 33.1.1  https://review.openstack.org/44692309:45
odyssey4meandymccr unfortunately the new setuptools just got blocked too - I guess we should hold off those changes indefinitely until we see a month of no changes upstream09:45
andymccrodyssey4me: sounds good09:46
openstackgerritMerged openstack/openstack-ansible-repo_build stable/newton: Updates to support ignored packages and external indexes  https://review.openstack.org/44671909:48
odyssey4meandymccr argh: http://logs.openstack.org/periodic/periodic-openstack-ansible-upgrade-aio-master-ubuntu-xenial/573a6b7/console.html#_2017-03-17_08_06_22_46457209:48
odyssey4methe swift deployment failed in the upgrade due to the min_part_hours09:48
andymccrodyssey4me: there is a var you can set to get around that - we do that in the swift gate09:48
odyssey4meyou've worked around that in the role tests right? all we do is add an override var for the AIO config I guess?09:48
andymccrswift_pretend_min_part_hours_passed: True09:49
DimGRwhen doing in Ocata  15.0 lxc-container-create.yml --limit neutron_all it complains about http://paste.openstack.org/show/603020/           if i run it without the --limit it finishes fine . is this  a user error or bug ?09:51
*** SerenaFeng has quit IRC09:57
*** vnogin has quit IRC10:03
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Bypass min_part_hours check for the AIO  https://review.openstack.org/44693210:04
odyssey4meandymccr ^10:04
andymccrodyssey4me: btw do you still have the failed gate? i think there may be a bug there that can be fixed10:05
odyssey4meandymccr what do you mean?10:05
odyssey4meDimGR if you want to limit for that playbook you must also include lxc_hosts as the playbook uses host delegation and needs to gather facts from them10:06
odyssey4meie lxc-container-create.yml --limit lxc_hosts,neutron_all10:06
andymccrodyssey4me: well there should be no change in the ring so it shouldnt try a rebalance, i think it tries a rebalance and doesnt recognize that there is no change (possibly)10:07
odyssey4meandymccr hmm, this is after an upgrade so surely it should need to anyway due to the new code?10:08
andymccrodyssey4me: the ring shouldnt change if the hosts/disks haven't afaik, but yeah i'll take a look - i can recreate that pretty easily10:08
odyssey4meandymccr that set of task doesn't seem to execute on any conditional except for being the primary in the ring: https://github.com/openstack/openstack-ansible-os_swift/blob/master/tasks/swift_rings.yml#L2610:10
odyssey4mehttps://github.com/openstack/openstack-ansible-os_swift/blob/master/tasks/swift_rings_build.yml#L4610:10
odyssey4methere's no conditional to stop it building10:11
odyssey4meI'm guessing that you might have meant for the swift_rings_check to produce a result, then for the build to happen if that result was that it was needed?10:12
*** cuongnv has quit IRC10:15
*** shausy has quit IRC10:16
*** shausy has joined #openstack-ansible10:16
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/newton: Updates to support ignored packages and external indexes  https://review.openstack.org/44671810:18
andymccrodyssey4me: yeah it'll happen in the execution of the script itself (so swift_rings.py shouldnt try the rebalance so it wouldnt then fail) but that can be fixed :)10:20
odyssey4meah10:21
odyssey4meI would think that it'd be better if the script exited with a different return code instead of failing in that condition.10:21
odyssey4meie 0 = success, 1 = fail, 2 = did nothing10:22
odyssey4meor something like that10:22
andymccrodyssey4me: yeah i thinkso - although i think the problem is we call the ringbuilder swift bits directly now and that will error 1 if you try rebalance and it cant rebalance10:23
andymccrso we will need some logic in there but it should be easy enough10:23
odyssey4methe alternative would be to add a '--check' parameter which is read only and informs the automation what needs doing10:23
andymccrthats a new change in Pike (for our swift role) so it should be fine in Ocata/Newton still10:23
odyssey4meperhaps we can get something like that built into the swift tooling so that everyone can benefit from it10:24
odyssey4me(similar to what we arranged with keystone)10:24
*** deadnull has joined #openstack-ansible10:30
*** deadnull has quit IRC10:30
odyssey4mebbiab - off to run some errands10:32
andymccri need to run some errands this afternoon so i probably wont be back online today (or at least much later).10:37
*** udesale has quit IRC10:48
*** vnogin has joined #openstack-ansible10:51
*** smatzek has joined #openstack-ansible10:52
*** vnogin has quit IRC10:52
*** vnogin has joined #openstack-ansible10:52
*** Andrew_jedi has joined #openstack-ansible10:56
*** rcarrillocruz has quit IRC11:00
*** karimb has quit IRC11:04
*** karimb has joined #openstack-ansible11:06
*** hamza has quit IRC11:38
*** Jack_Iv has joined #openstack-ansible11:40
*** Jack_Iv has quit IRC11:40
*** Jack_Iv has joined #openstack-ansible11:40
*** winggundamth has quit IRC11:40
*** shausy has quit IRC11:51
*** shausy has joined #openstack-ansible11:51
bwwHi Amit82: Any luck on AIO external connectivity?11:51
*** shardy is now known as shardy_lunch12:04
*** Andrew_jedi has quit IRC12:07
*** Andrew_jedi has joined #openstack-ansible12:12
*** LinStatSDR has joined #openstack-ansible12:14
*** Jack_Iv_ has joined #openstack-ansible12:15
Amit82bww: we are able to have external connectivity12:17
Amit82but we are not using AIO12:17
Amit82we are having two separate machines acting as Controller and Compute node respectively12:18
*** Jack_Iv has quit IRC12:18
bwwok, can you share your deployment configuration and any guides on how you deplyed?12:20
bwwdeployed12:20
Amit82I followed the the newton deployment guide12:20
Amit82as I have installed Newton release using 14.0.8 tag12:21
bwwok, did you have to make any modifications to networking?12:21
*** LinStatSDR has left #openstack-ansible12:21
*** woodard has quit IRC12:24
*** Oku_OS is now known as Oku_OS-away12:24
*** Oku_OS-away is now known as Oku_OS12:24
*** woodard has joined #openstack-ansible12:24
Amit82bww: I have pasted /etc/network/interfaces from both the nodes and openstack_user_config.yml here: http://paste.openstack.org/show/603118/12:26
Amit82hope it helps12:26
Amit82We are still facing some issue of loosing connectivity to Compute and Controller node, if we use two NICs12:27
Amit82but if we use one extra NIC (eth2) in addition to eth1, for accessing lab n/w, things are working fine12:28
bwwcool thanks...I am fairly new to deployiong Openstack via Ansible, but getting the hang of it12:29
*** Jack_Iv_ has quit IRC12:33
*** markvoelker has joined #openstack-ansible12:35
*** fandi has joined #openstack-ansible12:39
*** rpittau has quit IRC12:41
bwwAmit8212:45
bwwIs your setup with two servers and the nova is a physical server while the rest of the services like storage, etc are runing on the controller node as LXC containers?12:46
*** shausy has quit IRC12:48
Amit82bww: I am having this example setup: https://docs.openstack.org/project-deploy-guide/openstack-ansible/newton/app-config-test.html#test-environment-config12:50
Amit82"Block Storage Host" is not there in my case12:50
bwwok cool, so your not running a seprate storage node12:52
*** Jack_Iv has joined #openstack-ansible12:52
*** acormier has joined #openstack-ansible12:52
*** acormier has quit IRC12:52
*** acormier has joined #openstack-ansible12:52
*** schwicht has joined #openstack-ansible12:54
mgariepycloudnull, 3.10.0-514.2.2.el7.x86_6412:54
mgariepycloudnull, starting with this image, yum update -y, https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1612.raw.tar.gz12:55
*** acormier has quit IRC12:55
*** Amit82 has quit IRC12:55
*** acormier has joined #openstack-ansible12:55
*** askb has quit IRC12:56
*** manheim has quit IRC12:56
*** manheim has joined #openstack-ansible12:56
mgariepycloudnull, http://paste.openstack.org/show/603127/12:57
*** acormier has quit IRC12:57
*** muxdaemon has quit IRC12:59
*** schwicht has quit IRC13:03
*** schwicht has joined #openstack-ansible13:05
*** hamza has joined #openstack-ansible13:06
*** karimb has quit IRC13:09
*** esberglu has joined #openstack-ansible13:13
*** esberglu has left #openstack-ansible13:13
*** esberglu has joined #openstack-ansible13:13
*** karimb has joined #openstack-ansible13:18
openstackgerritMerged openstack/openstack-ansible stable/newton: Revert to setuptools 33.1.1  https://review.openstack.org/44692313:18
*** schwicht has quit IRC13:21
*** gouthamr has joined #openstack-ansible13:23
*** muxdaemon has joined #openstack-ansible13:25
*** hamza has quit IRC13:29
Andrew_jedimgariepy: Hi, quick question, If i want to look for neutron code in CentOS7 then i should like in "/usr/share/nova" directory ?13:30
Andrew_jedis/like/look13:30
*** muxdaemon has quit IRC13:32
*** smatzek has quit IRC13:32
odyssey4meAndrew_jedi our deployment on CentOS is the same as on Ubuntu for the most part13:33
odyssey4mecertainly for the venvs13:33
odyssey4meie you'll find the venvs in /openstack/venvs13:33
*** cathrichardson has joined #openstack-ansible13:34
*** LinStatSDR has joined #openstack-ansible13:34
Andrew_jediodyssey4me: Ohh, yes, i forgot about the venvs. Actually this particular installation is done via ansible but not using OSA for now. I was looking for the location of the actual code to debug an issue.13:34
odyssey4meah ok13:34
*** cathrich_ has quit IRC13:36
openstackgerritMerged openstack/openstack-ansible stable/ocata: Revert to setuptools 33.1.1  https://review.openstack.org/44692213:37
admin0cannot get telemetry to work on AIO13:37
admin0:(13:37
*** LinStatSDR has left #openstack-ansible13:38
odyssey4meadmin0 do you have a bug report for it?13:40
odyssey4mehave you figured out anything particular wrong?13:40
admin0i am working on gathering all the data13:41
odyssey4methe telemetry stuff is a bit weird13:41
Andrew_jediadmin0: It was working for me, what is the problem that you are facing ?13:42
admin0i think first issue i face is “how to verify its working” :D13:42
Andrew_jediadmin0: Check metrics via ceilometer client, create an autoscaling template and test alarms13:43
admin0what API commands would verify that its working ..  like i do for openstack server list ; openstack image list ; openstack volume list etc13:44
*** cpuga has joined #openstack-ansible13:45
*** schwicht has joined #openstack-ansible13:45
mgariepyAndrew_jedi, if you are installing from pkg you can list pkg with `rpm -qa` and file with `rpm -qf <pkg>`13:47
mgariepyoops, 'rpm -ql <pkg>'13:48
mgariepyto list files. -qf can tell you which package a file belong to.13:48
Andrew_jedimgariepy: Awesome, thanks!13:48
mgariepyit's impressive what breaks when you checkout cinder role in place of nova...13:52
*** crushil has joined #openstack-ansible13:53
*** FrankZhang has joined #openstack-ansible13:53
*** hamza has joined #openstack-ansible13:55
odyssey4melol13:55
*** shardy_lunch is now known as shardy13:56
*** smatzek has joined #openstack-ansible13:57
*** woodard has quit IRC13:58
*** FrankZhang has quit IRC13:58
*** FrankZhang has joined #openstack-ansible13:58
Andrew_jedimgariepy: just found mitaka code in newton package. :p14:03
mgariepyAndrew_jedi, where the pakcage comes from ?14:04
Andrew_jediRDO repo, i am looking for a word in english language to express extreme surprise ...14:05
*** fabg has joined #openstack-ansible14:13
*** lwiecek has quit IRC14:14
Andrew_jedimgariepy: Correction. Packages are fine, somebody decided to modify one of package file, figured it out via "rpm --verify".14:16
mgariepyhehe14:16
openstackgerritMerged openstack/openstack-ansible-ops master: define the default kernel statically  https://review.openstack.org/44679014:16
Andrew_jedimgariepy: :p14:24
*** cjloader has joined #openstack-ansible14:24
fabgodyssey4me: hi, did you have time to see https://bugs.launchpad.net/openstack-ansible/+bug/1670632. I know you have more to do :-) Or did someone encounter the same issue on 14.1.1 ?14:24
openstackLaunchpad bug 1670632 in openstack-ansible "ceilometer error because gnocchiclient > 3.0 for stable/newton " [Undecided,New] - Assigned to Jesse Pretorius (jesse-pretorius)14:24
admin0Andrew_jedi:….. for a word in english language to express extreme surprise   =>  Massively surprized14:25
Andrew_jediadmin0: try "flabbergasted".14:26
admin0and i thought it was german word :D14:26
admin0or nordic14:26
odyssey4mefabg yeah, sorry - I've been trying to confirm it but have been side-tracked with other issues14:27
*** chris_hultin|AWA is now known as chris_hultin14:30
fabgodyssey4me: i understand ;-) I follow the osa community activity and i see the mass of work you slaughter :-)14:30
*** foutatoro has joined #openstack-ansible14:33
foutatorohi, I try to ping an instance in project network from the namespace of dhcp but I get the error "ping: error while loading shared libraries: libcap.so.2: cannot stat shared object: Permission denied"14:35
foutatorodid someboby face in OSA with issue before ?14:36
*** rpittau has joined #openstack-ansible14:38
*** cathrichardson has quit IRC14:39
*** cathrichardson has joined #openstack-ansible14:39
*** jmckind has joined #openstack-ansible14:42
*** agrebennikov has joined #openstack-ansible14:46
*** sc68cal has quit IRC14:47
*** kstev has joined #openstack-ansible14:48
*** vnogin has quit IRC14:48
*** acormier has joined #openstack-ansible14:50
*** fandi has quit IRC14:53
*** looking_around has joined #openstack-ansible14:54
*** looking_around has left #openstack-ansible14:55
*** Dinesh_Bhor has quit IRC14:56
*** manheim has quit IRC14:57
*** marst has quit IRC14:58
foutatoroany suggestion ? is it a bug with xenial ? ..14:58
strattaofor the provider networks specified in the openstack_user_config.yml, do I have to specify a “flat” network?15:03
*** marst has joined #openstack-ansible15:04
strattaoor can I just use only a vlan provider network for the br-vlan container bridge connection?15:04
*** ansibleRhino has joined #openstack-ansible15:04
*** marst has quit IRC15:05
admin0 strattao . is it post-install question ?15:05
*** marst has joined #openstack-ansible15:05
*** fxpester has quit IRC15:05
strattaoyes - I get an error saying that the flat network is not defined15:05
strattaobut I don’t want a flat network…15:05
*** aludwar has quit IRC15:05
*** aludwar has joined #openstack-ansible15:06
*** sanfern has quit IRC15:06
strattaoand can’t figure out what is expecting to use a flat network15:06
*** fabg has quit IRC15:07
*** sanfern has joined #openstack-ansible15:07
*** messy has quit IRC15:08
*** NikhilS has quit IRC15:08
*** hamza has quit IRC15:10
admin0strattao: you don’t have to use flat network .. you can use vlan to add exsternal network —  http://www.openstackfaq.com/openstack-add-floating-ips/15:12
*** manheim has joined #openstack-ansible15:15
*** Jack_Iv has quit IRC15:20
*** Jack_Iv has joined #openstack-ansible15:21
*** manheim has quit IRC15:22
*** acormier has quit IRC15:28
*** vnogin has joined #openstack-ansible15:30
*** foutatoro has quit IRC15:31
*** vnogin has quit IRC15:34
*** vnogin has joined #openstack-ansible15:34
openstackgerritTom Jose Kalapura proposed openstack/openstack-ansible-rsyslog_client stable/newton: Fix remote logging template  https://review.openstack.org/44703615:39
agrebennikovseems st patrick's day killed the channel for today :D15:41
spotzagrebennikov: It's not helping it no:)15:42
spotzI think our fearless leader may still be in Milan or travelling back15:43
*** galstrom_zzz is now known as galstrom15:43
agrebennikovbut usually there is pretty active discussion happening by this time during the week... even on fridays ;)15:44
spotzJust blame St. Paddy's:)15:44
*** crushil has quit IRC15:45
*** Jack_Iv has quit IRC15:49
strattaothanks admin015:50
*** hamza has joined #openstack-ansible16:02
*** xinli has joined #openstack-ansible16:02
*** vishwanathj has joined #openstack-ansible16:03
admin0strattao: were yu able to add the IPs ?16:04
*** Oku_OS is now known as Oku_OS-away16:05
*** chris_hultin is now known as chris_hultin|AWA16:07
*** muxdaemon has joined #openstack-ansible16:12
*** MasterOfBugs has joined #openstack-ansible16:17
*** vishwanathj has quit IRC16:17
*** shananigans has quit IRC16:23
*** zz_pwnall1337 is now known as pwnall133716:25
*** marst has quit IRC16:28
*** muxdaemo_ has joined #openstack-ansible16:33
*** Andrew_jedi has quit IRC16:36
*** muxdaemon has quit IRC16:37
*** muxdaemo_ has quit IRC16:38
*** muxdaemon has joined #openstack-ansible16:39
*** crushil has joined #openstack-ansible16:40
*** cmart has joined #openstack-ansible16:42
*** Jack_Iv has joined #openstack-ansible16:50
*** ansibleRhino has left #openstack-ansible16:50
*** vnogin has quit IRC16:52
*** Jack_Iv has quit IRC16:53
*** acormier has joined #openstack-ansible16:57
*** Andrew_jedi has joined #openstack-ansible16:59
cmartis anyone here using nova libvirt password/key injection? it seems that libguestfs may be broken for OSA Newton.17:05
odyssey4mecsmart not as far as I know? although I'd like to understand how it's an OSA issue and not a nova/libguestfs issue?17:07
odyssey4meperhaps a config problem?17:07
odyssey4meby that I mean nova.conf?17:07
csmartodyssey4me: cmart ^17:08
cmartodyssey4me, nova.conf looks good, I have "inject_partition = -1", "inject_password = True", "inject_key = True"17:08
csmartMan, our nics are a bit too similar...17:09
cmartwhen I launch an instance I get the following in nova-compute.log: "Ignoring error injecting data into image <LocalFileImage:{'path': '/var/lib/nova/instances/0c32099c-c6da-4505-a8a0-69bd26b2bb5f/disk', 'format': 'qcow2'}> (libguestfs installed but not usable (/usr/bin/supermin exited with error status 1."17:09
odyssey4melol17:09
* csmart goes back to sleep :-)17:09
csmart\o17:09
odyssey4mehmm, tell me that whatever is trying to do this is running on bare metal - not in a container?17:10
odyssey4mewhoops, sorry csmart17:10
*** hamza has quit IRC17:10
cmartodyseey4me, yes, this is on a compute host17:10
odyssey4meas far as I recall, that function should be happening on the compute hosts - and libguestfs can only build the supermin on a bare metal host17:10
odyssey4mehmm, ok, so either we're missing a package, some config or something is broken in ubuntu?17:11
odyssey4mecan you validate with someone who has done this at all that you have the right config in nova.conf and the right packages - it might be worth pinging the openstack-operators list about it17:12
csmartodyssey4me: all good, even I do a double take when cmart talks :-) later17:12
odyssey4meof course, one could hope that someone in here has a working setup and can help troubleshoot17:13
odyssey4menot sure if logan- or jmccrory make use of that and can help troubleshoot17:13
odyssey4meor agrebennikov17:14
cmartodyssey4me could do, but the libvirt section of nova.conf looks good according to the various blog entries I've read. I believe this is the code that Nova executes to produce the error: https://github.com/openstack/nova/blob/c6cb5cf1ba0b1483951f3d236c53ff7924f3d7af/nova/virt/disk/vfs/guestfs.py#L7617:15
cmartAlso, below that error is this: "To see full error messages you may need to enable debugging. Do: export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 and run the command again."17:16
cmart(I'm not sure how to pass environment variables to the running Nova service)17:17
cmartSo I'm unsure if this is an OSA issue at all, just curious if others have gotten this working.17:17
*** pcaruana has quit IRC17:18
cmartoverall I'd rather not do it this way and just use cloud-init to inject the SSH key, but we have some years-old images without cloud-init, that we still need to support.17:23
odyssey4meyeah, I think the majority of people use cloud-init17:26
agrebennikovcmart, sorry, I guess I'm missing the context a little bit... how may I help having baremetals?17:28
*** muxdaemon has quit IRC17:29
agrebennikovodyssey4me, is that still valid?17:29
cmartagrebennikov just having a trouble with libvirt key injection on the compute host, which uses libguestfs.17:30
agrebennikovconfig drive you mean?17:30
odyssey4meno, the libvirt key injection actually modifies the image on the compute host using libguestfs instead of using cloud-init (which uses config drive) IIRC17:31
agrebennikovah, yeah, sorry... so... we use cloud-init everywhere, but if I only need to change one compute node configuration - I can test it out17:32
*** woodard has joined #openstack-ansible17:34
cmartagrebennikov, only if you feel like trying it out. i'm also continuing to troubleshoot. All I did was set the following three YAML variables:17:34
cmart"nova_libvirt_inject_key: True", "nova_libvirt_inject_partition: -1", "nova_libvirt_inject_password: True"17:35
*** chris_hultin|AWA is now known as chris_hultin17:35
cmartI don't advocate that others go this route if you already use cloud-init and it's working for you! This is only so we can continue supporting some older, cloud-init-less images maintained by our community17:36
odyssey4mefreebsd?17:37
xdfilodyssey4me: What's the appropriate way to give a container a static IP?17:38
*** chris_hultin is now known as chris_hultin|AWA17:38
cmartodyssey4me nope, customized builds of older Ubuntu and CentOS, which from a security perspective should be phased out, but people have used them for scientific work17:38
*** muxdaemon has joined #openstack-ansible17:39
cmartodyssey4me for example if you want to try to reproduce a genomics study done 3 years ago, we let you launch the exact same image that was used to run the original analysis, so you can hopefully see how the results were obtained. it's a data provenance thing.17:40
odyssey4mexdfil you don't, unless you want to hack your inventory.json once the inventory item has already been generated17:40
*** SerenaFeng has joined #openstack-ansible17:41
xdfilI have to for BGP speaker. the physical routers need to peer with the IP address of the dragent17:41
odyssey4mecmart ah, yeah - I tried it a few years ago and think I got it right for freebsd17:42
odyssey4mebut that was back for grizzly, and I don't have access ot that environment any more17:42
*** Jack_Iv has joined #openstack-ansible17:42
xdfilwont the inventory.json get overwritten later on?17:42
odyssey4mexdfil once an item is generated it will stay there for the lifetime of the container17:42
xdfilok cool thanks17:43
odyssey4mexdfil if you really need a static IP you can set that service to run on_metal and dedicate a host (or hosts) to it instead of using containers17:43
odyssey4methat's something useful to do with the network_hosts in large environments17:43
xdfilahh right17:44
xdfilya know what I like that better17:44
xdfilway better actually i dont need a br-bgp network anymore17:45
xdfilwait... that would mean all the neutron agents would be on metel17:45
agrebennikovso cmart if I do that on one of the computes - there still will be cloud-init workflow in place... how do I know if the stuff you need works?17:46
agrebennikovor you want to give me an image?17:46
cmartagrebennikov it's probably best if you try launching an image with cloud-init disabled or absent. happy to give you one.17:50
agrebennikovsure17:50
agrebennikovcan I just manually change the config of nova on one of the computes?17:51
agrebennikovor you want me to do it via osa?17:51
*** chris_hultin|AWA is now known as chris_hultin17:51
agrebennikov(sure - meant to say "sure, give it to me" ;)17:51
odyssey4mexdfil yep, you'd need to dedicate at least two hosts (to allow router migration if one goes down) and all the agents would live on there17:51
odyssey4meagrebennikov I don't think it matters - what cmart needs is a working configuration to compare against. How it got there is immaterial.17:52
agrebennikovthat works...17:53
odyssey4meEven if it's some other deployment tool, built using packages or whatever.17:53
cmartyes, I did it via OSA but you may not need to. I think all OSA does is template out the vars in nova.conf. how can I send you a 2 GB qcow2 image?17:53
*** shananigans has joined #openstack-ansible17:53
agrebennikovbut I'll only do it if you vote for https://review.openstack.org/#/c/444491/2 and https://review.openstack.org/#/c/425997/18 :D17:53
agrebennikovcmart, you probably can....17:53
agrebennikovif you for example upload to gdrive17:54
agrebennikovand send me the link17:54
cmartyep, or Amazon S3. give me a few minutes :)17:54
*** crushil has quit IRC17:55
*** muxdaemon has quit IRC17:55
odyssey4meagrebennikov you asked for it ;)18:02
odyssey4mehopefully my review makes sense18:02
agrebennikovso essentially you want me to populate the cert back to the deployment host from the galera cert once it is created?18:04
agrebennikov*from the galera host18:04
agrebennikovodyssey4me,18:06
odyssey4meyeah, everything is massively simpler and more reliable when the deployment host is used as a place to pull to and push from18:08
DimGRit took me 30 days to figure that out ^^18:08
odyssey4methen even self generated certs are 'user configurated', so there's only one way it ever gets there18:08
odyssey4mewe have a bunch of distribution mechanisms that use delegation which we should ideally get rid of and replace with this simpler, more reliable mechanism18:09
odyssey4meideally every ssl cert generation and placement fits there, and perhaps other things18:09
odyssey4mewe're ideally hoping to move towards a place where we don't need a ssh service in the containers - it's far more secure18:10
odyssey4meright now it's only the container to container delegation that's holding that back as far as I know18:10
*** cjloader has quit IRC18:11
odyssey4mein case you're not aware we already connect to containers via the host, rather than directly via ssh18:11
*** cjloader has joined #openstack-ansible18:11
xdfilodyssey4me: I'm thinking of making the container/infra hosts ( i have 3 ) be bare metal for neutron_agents18:12
xdfilyou forsee any issues with that?18:13
*** cjloader has quit IRC18:13
xdfilOSA wise18:13
odyssey4mexdfil you mean having your infra hosts with all the containers on them also host the agents, but on bare metal instead of in containers?18:13
xdfilyes18:13
*** cjloader has joined #openstack-ansible18:13
odyssey4mewell, it depends on your use-case18:13
*** SerenaFeng has quit IRC18:14
odyssey4mein that situation if something breaks on your network agent hosts, you also lose a third of your api infrastructure when you need to rebuild the hosts18:14
odyssey4meif you're doing that, better to containerise the agents18:14
odyssey4methe containers are easy to replace, the host is not18:14
odyssey4meif it's a seperate host, then you only affect that one smaller set of services if everything goes belly up18:15
xdfilhmmm good points18:15
odyssey4methe whole reason we've designed it to be so flexible in terms of service-host allocation is so that you really can spread and scale easily18:16
odyssey4mebut you can compress if your budget can't handle it, but that compression should be done with services in containers18:16
*** poopcat has joined #openstack-ansible18:16
xdfilin my case I have 8 nodes that are way too beefy to be anything other than compute hosts18:17
xdfil3 nodes that are less-beefy more suited for controller work18:17
xdfilI think it makes sense, to do bare metal neutron agents on the 3 controller nodes18:19
odyssey4mesounds like you should stick to the basic 3 controller model then18:19
xdfilI'm just concerned about running into an issue with the OSA runs18:19
odyssey4mewell, that's up to you - I'd not advise it operationally, but it's your environment :)18:20
odyssey4methe other concern is security18:20
odyssey4mecontainers segregate the kernel name spaces for all the services18:20
odyssey4meI don't know what effect it will have if your host is holding those services and all the containers with the possibility of namespace clashes or overlap.18:21
xdfilso if agents running on metal, and the agent gets compromised it can own all the containers is the security concern18:21
*** acormier has quit IRC18:23
*** crushil has joined #openstack-ansible18:23
*** poopcat has quit IRC18:24
xdfilodyssey4me if I configured all the hosts pre-deploy with OVS bridges instead of linux bridges, and specified those bridges in user_settings.yml in place of the br-mgmt/br-vlan would OSA be able to attach containers to them?18:26
xdfilslightly off topic, but I'm curious about that18:27
odyssey4meI honestly have no clue18:27
xdfil:)18:27
*** jmckind has quit IRC18:28
*** poopcat has joined #openstack-ansible18:32
xdfilodyssey4me another thing I am getting hung up on. If an OVS bridge (say br-tun) is connected to a linux bridge ( say br-vxlan ) which interface do I configure the IP on that neutron uses for local_ip18:32
odyssey4mexdfil I honestly have no clue. Networking is something of a dark art to me.18:37
xdfilmy network instincts tell me that putting an IP on an interface causes it to become a layer 3 interface and it will nto forward layer 2 frames18:37
odyssey4meI figure things like that out by trying, failing, then trying again.18:37
xdfilodyssey4me: I know I'm just trying to embaras you :)18:37
xdfilthanks though18:37
odyssey4meThe best thing you can do is figure out what you think you need, then try it. Then dig into any problems that arise and try to break it and hack it.18:38
odyssey4meOnce you've learned, wipe it all and try again.18:38
odyssey4meFor a new cloud environment, do this many, many times over.18:38
odyssey4me.For an existing production cloud - leave it and make sure you have a sufficient lab to test with.18:39
*** vnogin has joined #openstack-ansible18:39
xdfilyeah, i'm time constrained. I'm trying to solve problems without stumbling into unexpected dragons18:39
odyssey4meThen once you;ve worked out where you want to be, work out how you get there18:39
*** poopcat has quit IRC18:40
odyssey4meI used to have an old server setup with a bunch of VM's and automation to build, rebuild and do it again and again.18:40
odyssey4meSomething similar to the multi-node-aio in the ops repo18:40
odyssey4meIt's a vital tool if you're designing/deploying/supporting/maintaining a cloud.18:40
*** david-lyle_ is now known as david-lyle18:43
xdfilodyssey4me I was looking at inventory.json and the containers don't say DHCP they have addresses. So every time they reboot they will get that same address via DHCP?18:44
xdfilbut if i destroy/create it will change or stay the same?18:44
odyssey4mexdfil the container eth0 is DHCP and NATted through the host, which is why we don't use them... the addresses you're seeing are the br-mgmt and others which are static for the lifetime of the container18:45
odyssey4meif you destroy/recreate it will stay the same18:45
odyssey4methe only way to change it is to remove it from the inventory, or edit it18:45
xdfilhmmm ok mine changed at some point but I dont rememeber what i did18:46
xdfilthanks18:46
odyssey4meit will only change if you remove it, or remove the inventory entirely18:46
*** galstrom is now known as galstrom_zzz18:48
*** poopcat has joined #openstack-ansible18:52
xdfilodyssey4me I just noticed OSA didn't connect the agents container to the bridge I defined in user-settings18:56
*** retreved has joined #openstack-ansible18:57
xdfilveth get created during lxc-containers-create?18:57
odyssey4meyep19:00
*** galstrom_zzz is now known as galstrom19:01
agrebennikovodyssey4me, can you please make this https://review.openstack.org/#/c/446754/1 happen? I got votes for master, but evrardjp seems to be off today :(19:08
*** foutatoro has joined #openstack-ansible19:09
odyssey4meagrebennikov sure, looks about right - can you add a release note (or edit the existing one) to specify that 'haproxy_bufsize' has been removed in master19:11
agrebennikovoh yeah, sorry about that19:11
odyssey4meit should be an 'upgrade' release note, and should say that it's been removed, and it should be replaced by using the new tuning option19:12
agrebennikovyou mean, for backports it will be both "features" and "upgrades"?19:13
openstackgerritMerged openstack/openstack-ansible-haproxy_server stable/ocata: Added Haproxy global tunables  https://review.openstack.org/44675519:14
openstackgerritMerged openstack/openstack-ansible-haproxy_server stable/newton: Added Haproxy global tunables  https://review.openstack.org/44675419:14
agrebennikovoh, wait...19:15
agrebennikovwhy?...19:15
agrebennikovodyssey4me,19:15
agrebennikovI was about to add the change :)19:15
agrebennikovsp is it going to be a separate commit now?19:17
*** aludwar has quit IRC19:18
*** aludwar has joined #openstack-ansible19:19
*** Jack_Iv has quit IRC19:19
odyssey4meagrebennikov this should be an edit to the release note in master19:19
odyssey4meit'll be a seperate commit19:20
agrebennikovaahh19:20
odyssey4methe backports are fine as-is as nothing has been removed, only added19:20
odyssey4meso in the master patch you've made the feature note which is fine as-is19:20
odyssey4meadd another portion to the same release note with the heading 'upgrade: '19:21
agrebennikovyep19:21
odyssey4mewith text as described above19:21
*** Andrew_jedi has quit IRC19:22
*** Jack_Iv has joined #openstack-ansible19:23
openstackgerritAndrey Grebennikov proposed openstack/openstack-ansible-haproxy_server master: Mentioned haproxy_bufsize option removal  https://review.openstack.org/44710419:24
agrebennikovodyssey4me, like this? https://review.openstack.org/44710419:25
odyssey4mea few tweaks - can I make them for you?19:28
agrebennikovsure, thanks you very much!19:29
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-haproxy_server master: Mentioned haproxy_bufsize option removal  https://review.openstack.org/44710419:30
odyssey4methere we go19:30
odyssey4menow when we release pike, anyone who previously tuned with that setting will know to change it19:31
agrebennikovexactly19:31
*** Andrew_jedi has joined #openstack-ansible19:32
agrebennikovsorry for my bad language (not native) :P19:32
agrebennikovcloudnull, can you please vote here? https://review.openstack.org/44710419:33
*** rboyapat has joined #openstack-ansible19:35
rboyapatcore members: can you please review https://review.openstack.org/#/c/446838/19:35
*** xinli has quit IRC19:35
*** manheim has joined #openstack-ansible19:41
openstackgerritMerged openstack/openstack-ansible-haproxy_server master: Mentioned haproxy_bufsize option removal  https://review.openstack.org/44710419:42
*** shardy has quit IRC19:44
*** galstrom is now known as galstrom_zzz20:13
openstackgerritKyle L. Henderson proposed openstack/openstack-ansible stable/newton: Provide example of using veth pairs for br-vlan  https://review.openstack.org/44555620:17
*** foutatoro has quit IRC20:24
*** yolanda has quit IRC20:26
*** manheim has quit IRC20:26
*** yolanda has joined #openstack-ansible20:26
*** xinli has joined #openstack-ansible20:27
jrosser_odyssey4me: i see earlier you talked about removing ssh from the containers20:28
*** cpuga has quit IRC20:29
*** cpuga has joined #openstack-ansible20:29
jrosser_is that in progress somewhere, as it would make stuff significantly easier in our situation20:29
*** cpuga has quit IRC20:30
odyssey4mejrosser_ it's not really a planned activity, primarily because no-one has really raised it as a priority20:30
*** cpuga has joined #openstack-ansible20:30
*** retreved has quit IRC20:31
odyssey4mewe did a significant amount of work in Newton to make it possible, but the rest will require someone taking the time to figure out what breaks when openssh is removed from the containers and then it'll take a bit of collective thought to figure out what to do about those things20:31
odyssey4meif it's of interest to you, then perhaps you have resources to commit to doing the initial investigation?20:32
odyssey4mewe could probably collaborate on figuring this stuff out on an etherpad, then work up a spec20:32
*** crushil has quit IRC20:32
DimGRi had major issues too with ssh into containers from not local host20:33
odyssey4mebut most of us are already tied up with other activities, so we'll definitely need more hands20:33
*** smatzek has quit IRC20:34
*** Jack_Iv has quit IRC20:34
*** cpuga has quit IRC20:35
*** chris_hultin is now known as chris_hultin|AWA20:36
*** cjloader has quit IRC20:40
*** crushil has joined #openstack-ansible20:42
*** Andrew_jedi has quit IRC20:44
jrosser_odyssey4me: can i ask about the ssh stuff then, currently the deployment host never ssh to the containers?20:46
*** Jack_Iv has joined #openstack-ansible20:47
jrosser_but the containers need to ssh between themselves to distribute certs etc?20:47
odyssey4mecurrently the deployment host connects through the hosts to get to the containers for almost everything20:50
odyssey4mewe have a connection plugin which makes Ansible connect to the host, then lxc-attach into the container to execute things20:51
odyssey4metherefore it does not need openssh running in the container20:51
odyssey4mehowever, we have delegated tasks - for example the os-glance-install.yml playbook targets the glance containers, but delegates the database creation to the galera containers and the rabbitmq vhost creation to the rabbitmq containers20:52
odyssey4mewhen that happens, it requires ssh connectivity between the containers20:52
jrosser_ok20:52
odyssey4meto prevent that, we would have to change the pattern of how we do these things20:52
jrosser_currently we have a service network with provisions all the bare metal, and has the deploment host on it20:53
jrosser_this is not the mgmt network though20:53
jrosser_and so we get into difficulty20:53
*** Andrew_jedi has joined #openstack-ansible20:54
jrosser_but knowing that the service net does not need to ssh to the containers simplifies things20:54
jrosser_it only has to go to all the hosts20:55
odyssey4meyeah20:55
odyssey4meI'm not sure what would fall out if you do this, but you could also add a different bridge and apply these settings on it instead of on br-mgmt: https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.prod.example#L34-L3520:56
jrosser_yes we could20:56
jrosser_but the point of the service net is it is more trusted than the mgmt net20:56
odyssey4methere's a fair chance that we've made a lot of assumptions about br-mgmt being the network ansible connects to, but that is stuff we should actually clean up20:56
jrosser_and very deliberatley isolated from it20:56
odyssey4meyep, fair point20:57
jrosser_i could never get the deployment host being on mgmt past a security review20:57
odyssey4mebr-mgmt is meant to be an isolated network with only outgoing internet access anyway20:58
odyssey4meit's probably a badly named network20:58
jrosser_indeed :)20:58
jrosser_im also trying not to have mgmt having outgoing internet access20:59
odyssey4mewith or without a proxy20:59
jrosser_proxy is ok20:59
jrosser_nat to internet is not21:00
odyssey4meok, that should work already21:00
jrosser_yes21:00
odyssey4meit's a bit of a PITA to get right in the beginning, but it works eventually21:00
jrosser_we are just crawling from multinode aio to something like that21:00
odyssey4meI'm doing a bunch of work to try and make a fully air-gapped deployment work.21:01
jrosser_but as you say it is a lot of work to get the environment right21:01
jrosser_oh ++ on that21:01
jrosser_happy to test things21:01
odyssey4meyeah, once we have the whole story worked out and broader testing started in our group we'll start to figure out how to upstream the tooling21:01
odyssey4mea lot of it right now is a bit of a hack job21:02
jrosser_ideally my service net provides all proxies and packeges etc21:02
*** xinli has quit IRC21:02
jrosser_and mgmt becomes purely api chatter21:03
*** crushil has quit IRC21:03
jrosser_but at the same time i dont want the service net in the containers21:03
odyssey4mebut the basic stuff we're doing it preparing all packages, repo data, etc in a pipeline - then tooling the deployment to stage either offline or online, then execute the normal build using the staged data21:03
jrosser_so thats a bit chicken/egg atm21:03
*** rboyapat has quit IRC21:04
odyssey4meyeah, it'd be nice to figure that out - I expect it can be done with what's already there - perhaps with a few patches to ensure we're looking the right things up the right way21:04
*** rboyapat has joined #openstack-ansible21:04
*** McMurlock1 has quit IRC21:06
*** rboyapat has quit IRC21:07
jrosser_i also need to think about the implications of things like bare metal network nodes where the service net and provider nets come together with little isolation21:08
jrosser_thats feeling a little less separated than regualar infra nodes currently21:08
*** foutatoro has joined #openstack-ansible21:19
*** cjloader has joined #openstack-ansible21:24
*** cjloader has quit IRC21:28
odyssey4meyeah, unfortunately ironic doesn't really separate things properly just yet as far as I know - it puts all projects on the same provisioning network, so you'd have to do a ton of ACL's to secure it21:29
*** Andrew_jedi has quit IRC21:32
*** kstev has quit IRC21:32
*** gouthamr has quit IRC21:33
*** Jack_Iv has quit IRC21:34
*** Jack_Iv has joined #openstack-ansible21:35
*** Jeffrey4l has quit IRC21:35
*** schwicht has quit IRC21:36
*** Jeffrey4l has joined #openstack-ansible21:36
*** Jack_Iv has quit IRC21:39
*** esberglu has quit IRC21:41
*** esberglu has joined #openstack-ansible21:41
*** esberglu has quit IRC21:46
*** karimb has quit IRC21:51
*** gouthamr has joined #openstack-ansible21:57
openstackgerritGerman Eichberger proposed openstack/openstack-ansible-os_octavia master: [WIP] Adds iptables rules to protect octavia server container  https://review.openstack.org/44715122:03
dankolbrsHi all, I submitted https://bugs.launchpad.net/openstack-ansible/+bug/1673889 . Feel free to yell at me here or in there if I missed anything or left anything out22:05
openstackLaunchpad bug 1673889 in openstack-ansible "Nova services do not restart on N->O upgrade" [Undecided,New]22:05
cmartodyssey4me and agrebennikov, I solved the libvirt key injection issue. in Ubuntu the kernel is non-readable to non-root users by default, so nova could not read the kernel and launch the supermin!22:21
*** schwicht has joined #openstack-ansible22:21
odyssey4mecmart interesting - so how did you solve it? does it need a rootwrap edit?22:21
cmartwell just now I solved it with `sudo chmod 0644 /boot/vmlinuz*`22:22
odyssey4meif so, then it's actually a nova bug which you can patch :)22:22
odyssey4mehmm, I expect that nova may be interested in it - if you could put together a bug report to describe the error output, the config bits and the packages/platform then we could perhaps engage with some of the nova crew to see what to do about it22:23
odyssey4mejohnthetubaguy may have some insight22:23
*** foutatoro has quit IRC22:25
cmartok. it is an Ubuntu-specific bug. something needs to make sure that the user running libguestfs (in our case nova) can read the compressed kernel files in /boot. it sounds like there have been years of handwringing over this between libguestfs, Ubuntu, and various other projects22:25
odyssey4mecmart sure, but that's exactly the purpose of the rootwrap files22:25
odyssey4methey're effectively files which implement sudoer capabilities22:26
cmartis rootwrap a nova-ism or an OSA-ism?22:26
odyssey4mehttps://github.com/openstack/nova/blob/master/etc/nova/rootwrap.d/compute.filters22:26
cmartaha22:26
odyssey4merootwrap itself is a python wrapper which is implemented for priveleged commands - it reads that file to figure out what it's allowed to do, or not22:27
odyssey4mebut it's basically a sudoers file22:27
cmartok. and if the config for "what can run as root" is maintained in nova, i'll write up the bug against nova. should be a simple fix. Thanks odyssey4me and agrebennikov for helping me triage!22:29
odyssey4mecmart note this though: https://github.com/openstack/nova/blob/424972e2f03c42b76d47775b016d3ec2d001632f/nova/conf/libvirt.py#L147-L15322:29
cmartya - what about?22:30
odyssey4meactually, more importantly https://github.com/openstack/nova/blob/424972e2f03c42b76d47775b016d3ec2d001632f/nova/conf/libvirt.py#L169-L18122:30
odyssey4melibguestfs appears to be broken, so there's that22:30
odyssey4mebut if libguestfs is not there, nbd will be used22:30
odyssey4methe long term fix is to get the libguestfs fix into nova22:31
odyssey4methe short term fix may be to make sure libguestfs is not there22:31
agrebennikovcmart, nice stuff :)22:31
odyssey4meif OSA puts it there, we can remove that quickly22:31
odyssey4meany patches into nova will take time, and are not likely to be backported22:31
odyssey4meof course we'd like you to validate that removing it helps :)22:32
cmartunfortunately nbd as a fallback doesn't help you if you set inject_partition to -1, meaning libguestfs must be available to find the correct partition to mount https://github.com/openstack/nova/blob/424972e2f03c42b76d47775b016d3ec2d001632f/nova/conf/libvirt.py#L185-L19822:32
agrebennikovodyssey4me, "patches into nova will take time" should be said like "patches into nova will take forever"22:32
agrebennikov:P22:32
odyssey4me:) I'm trying to be polite agrebennikov - the nova crew have the hardest job.22:33
cmart(and that's how I'm doing it!)22:33
cmartok. thanks guys. I'll get this written up later tonight. gotta scoot and get my taxes done.22:33
odyssey4mecmart if anything, you may get advise for alternative ways of achieving the same thing22:34
agrebennikovcmart, have fun with taxes ;)22:34
odyssey4mehappy ot have helped - have a great w/end!22:34
agrebennikovodyssey4me, before you leave - can you please look at http://paste.openstack.org/show/603198/ and tell me if it can be accepted as an extention to nova.conf?22:35
agrebennikovor there is easier way for doing it22:36
agrebennikovit is regarding https://bugs.launchpad.net/openstack-ansible/+bug/167357022:36
openstackLaunchpad bug 1673570 in openstack-ansible "With more than one ceph cluster as the backend nova has secret_uuid hardcoded" [Undecided,New]22:36
agrebennikov(only in case you are familiar with that part)22:36
agrebennikovif not - lets go home22:37
*** vnogin has quit IRC22:38
odyssey4meargh, the whole usage of ceph needs a rewrite22:40
odyssey4meunfortunately I don't understand that code, and certainly won't right now - I'm quite tired22:40
odyssey4mebut now that we have everything integrated we can rewrite all that nonsense which hasn't changed much since kilo22:41
odyssey4meit's time we do that22:41
odyssey4meto fairly review this I'd have to try and work out the spider web22:41
odyssey4menext week we can try and see if logan- or mattt are available to help puzzle it out - they understand it better than most22:42
logan-i'll read thru it tonight22:42
odyssey4melogan- actually uses it in production, so that helps22:42
odyssey4melol, a logan- lurks in the bushes22:42
logan-odyssey4me  ;) someone said ceph22:42
odyssey4mehahaha22:43
odyssey4mehow's your upgrade to newton going?22:43
odyssey4mesorry, the upgrade from trusty to xenial22:43
logan-100% done now :)22:43
logan-no more trusty anywhere22:43
odyssey4meoh awesome :) good news, well worth a celebration22:44
agrebennikovall right, logan- can I bug you on monday morning then?22:44
odyssey4metime to look forward again22:44
odyssey4mewell - probably time to take a break, actually22:44
logan-no kidding! great to have it behind. really cool work cloudnull did this week with the repo stuff. I was just reading some of that today22:44
logan-agrebennikov: i've got a question for you, 1 sec let me find this older review22:45
agrebennikovabsolutely22:45
agrebennikovtopic?22:45
agrebennikovI have all mine opened22:45
logan-https://review.openstack.org/#/c/409353/ is your bug/patch similar to this one?22:46
agrebennikovsoooooo not actually :)22:46
odyssey4mehmm, yeah - it was michaelgugino who was working on ceph multi back-end things22:46
agrebennikovin fact it ia all messed up, and I'd say on the nova side22:46
odyssey4meunfortunately we haven't seen him for a while22:46
logan-i haven't really read thru your bug yet but i was just curious if its the same/similar thing22:46
agrebennikovthe main idea - with multi-ceph AND ephemerals in ceph nova can't work22:47
odyssey4meyeah, I think it's time to w/end y'all22:47
agrebennikovand my past above should cover that22:47
agrebennikovyeah, that makes perfect sense22:47
logan-ttyl odyssey4me22:47
logan-agrebennikov: why doesn't it work? missing libvirt secrets?22:48
agrebennikovok, long story short - with volumes in ceph secret uuid and ceph user can be delivered via rpc from cinder22:48
logan-nevermind.. i'll read the bug and patch(es) first and bug you with questions afterwards :P22:49
agrebennikovwith ephemerals in ceph these params Must be hardcoded in nova.conf on the compute22:49
logan-right22:49
agrebennikovI have to submit a couple of new params into nova though22:49
agrebennikovbut it is a separate story22:49
agrebennikovlet's chat on monday22:49
logan-ok sounds good22:50
logan-have a good weekend22:50
agrebennikovnice. cu then! have a great green-beer evening!22:50
logan-yes indeed22:50
agrebennikov(if you are here in the us)22:50
*** schwicht has quit IRC22:51
logan-dallas22:51
logan-:)22:51
*** agrebennikov has quit IRC22:55
openstackgerritMerged openstack/openstack-ansible-rsyslog_client master: Fix remote logging template  https://review.openstack.org/44683822:56
*** jbadiapa has quit IRC23:01
*** vnogin has joined #openstack-ansible23:02
*** jamesdenton has joined #openstack-ansible23:12
*** jamesden_ has joined #openstack-ansible23:16
*** schwicht has joined #openstack-ansible23:17
*** jamesdenton has quit IRC23:19
*** markvoelker has quit IRC23:26
*** DimGR has quit IRC23:43
*** vnogin has quit IRC23:47
*** marst has joined #openstack-ansible23:50
*** crushil has joined #openstack-ansible23:52
*** acormier has joined #openstack-ansible23:53
*** acormier has quit IRC23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!