Tuesday, 2015-12-01

*** sacharya has joined #openstack-ansible00:12
*** rebase has quit IRC00:15
*** sacharya has quit IRC00:17
*** mss has quit IRC00:21
*** openstackstatus has quit IRC00:24
*** openstackstatus has joined #openstack-ansible00:24
*** ChanServ sets mode: +v openstackstatus00:24
*** KLevenstein has quit IRC00:26
*** rebase has joined #openstack-ansible00:34
*** eil397 has quit IRC00:43
*** sdake has joined #openstack-ansible00:46
*** rebase has quit IRC00:51
*** galstrom is now known as galstrom_zzz00:55
*** sdake has quit IRC01:06
*** gtt116__ has joined #openstack-ansible01:06
*** sdake has joined #openstack-ansible01:19
*** karimb has quit IRC01:20
*** openstackgerrit has quit IRC01:22
*** openstackgerrit has joined #openstack-ansible01:22
*** sdake has quit IRC01:32
*** markvoelker has quit IRC01:46
*** coolj has joined #openstack-ansible01:52
cooljcloudnull: do we use older python-openstackclient versions for a reason? http://rpc-slushee.rackspace.com/os-releases/11.0.0/python_openstackclient-1.0.2-py2.py3-none-any.whl01:55
coolji need at least 1.6.0 for OS_ENDPOINT_TYPE support https://github.com/openstack/python-openstackclient/commit/ee64c2fa6b9eb6f8d628e8899df39a4ee01c2bb301:55
cloudnullkysse: So far the only other way that I can cause this issue, is to have a misconfiguration in neutron or to to remove a nic from a network or compute node01:58
kyssearhm. So I only use two infra nodes, and those nodes are also network nodes01:58
cloudnullso i was curious if you have all of the br-vlan, br-vxlan, br-mgmt interfaces on both your network and compute nodes01:59
kysseneutron containers run on those two machines01:59
cloudnullcoolj:  it looks like openstackclient it pinned for kilo02:00
cloudnullhttps://github.com/openstack/requirements/blob/stable/kilo/global-requirements.txt#L13302:00
cloudnulland we have http://rpc-repo.rackspace.com/os-releases/11.2.5/python_openstackclient-1.0.4-py2.py3-none-any.whl02:00
cloudnullkysse: so those bridges are on your infra boxes, but are they also on your compute node?02:00
kyssehttp://paste.nerv.fi/45555345.txt02:01
cloudnullok02:04
kysseI'm worried about interfaces 'inside' lxc containers.02:05
cloudnullwhys that ?02:05
kysseis it possible that those interfaces are not specified correctly?02:05
cooljcloudnull: yuck, but okay thanks02:05
kyssejust saying, cuz I dont fully understand that eth11..12 naming inside them.02:06
cloudnullI mean in the grand scheme of things everything is possible.02:06
cloudnullyou could destroy the neutron containers and recreate them02:06
cloudnulljust to be sure02:07
kysseand what host_bind_override is supposed to do?02:07
cloudnullopenstack-ansible lxc-container-destory.yml --limit neutron_agent; openstack-ansible lxc-container-create.yml --limit neutron_agent02:08
cloudnullthe host_bind_override is to give a network a specific interface02:08
kysseomg. destroyer.02:08
kyssehmh.02:09
cloudnullbut only a specific interface on a given machine02:09
kyssehttp://paste.nerv.fi/91658287.txt If I have specified things like that02:09
cloudnullhowever you've removed the flat network from the openstack_user_config.yml, correct?02:09
kysseyes02:09
cloudnullif you have that entry02:10
cloudnullyou would need to create the eth12 device on your compute nodes as well as your network nodes.02:10
kysseah sorry, looked at aio file..02:10
cloudnulland neutron would bind to the specified interface instead of the known bridge02:11
kysseI'm pretty much trying to avoid physical interface name mappings02:11
kyssesoo let's destroy those containers..02:12
cloudnulllol02:13
cloudnullonce done you'll need to rerun the os-neutron-install.yml play to resetup neutron within the containers.02:13
kysseyep02:15
openstackgerritMerged openstack/openstack-ansible: Add container-reconciler and object-expirer to os-swift  https://review.openstack.org/24729002:16
openstackgerritMerged openstack/openstack-ansible: Adjust default nova_force_config_drive to be False  https://review.openstack.org/25123202:16
openstackgerritMerged openstack/openstack-ansible: Remove ssh args to ensure pipelining is working  https://review.openstack.org/25008802:16
cloudnullkysse: also open up the /etc/openstack_deploy/openstack_inventory.json file and see if you have an entry like http://cdn.pasteraw.com/dfgl9qlvs4l02tup39qdgnyld6xsf6f02:18
cloudnullnote, i have eth12 for a flat network02:18
cloudnullhowever you should not02:18
kysseok02:19
kyssehttp://sprunge.us/jUPc02:28
*** sacharya has joined #openstack-ansible02:36
kysseso..02:37
kyssesame error again, I did everything. Hmm.02:38
kyssePortBindingFailed: Binding failed for port d408b88e-eb5f-4757-be94-28f8e05750a1, please check neutron logs for more information.02:38
kyssewhat is that UID?02:38
kysseI tried to search, but I could not find it from mysql02:38
kysseor neutron any-list etc.02:38
cloudnullthat looks right02:42
*** sacharya_ has joined #openstack-ansible02:43
cloudnullwhat about the /var/log/neutron/neutron-linuxbridge-agent.log file from within the agent containers02:45
cloudnulldo you have a logging node02:45
cloudnullif so you should be able to easily get all logs as they're shipped02:45
*** sacharya has quit IRC02:46
kysseyep02:47
openstackgerritMerged openstack/openstack-ansible: Fix logrotate config for horizon and keystone.  https://review.openstack.org/24632702:49
kysselets see02:52
kyssehttp://sprunge.us/DDBN02:57
kyssehttp://sprunge.us/LQdR02:57
kyssehttp://sprunge.us/QMKL02:57
kyssenothing interesting..02:57
*** fawadkhaliq has joined #openstack-ansible02:57
kysseoh gosh. from within the agent containers? HMM02:58
kyssethere are no such log!02:58
kysseonly compute1(su9d1), infra1,infra202:59
*** sacharya_ has quit IRC03:28
*** sacharya has joined #openstack-ansible03:29
*** baker has joined #openstack-ansible03:36
*** baker has quit IRC03:51
*** sacharya has quit IRC04:37
*** sacharya has joined #openstack-ansible04:38
*** jbweber has joined #openstack-ansible04:44
*** tlian2 has quit IRC04:53
*** sdake has joined #openstack-ansible05:00
*** sdake_ has joined #openstack-ansible05:03
*** sdake has quit IRC05:04
*** sdake_ has quit IRC05:10
iceyao_hi05:30
*** fawadkhaliq has quit IRC05:36
*** sacharya_ has joined #openstack-ansible05:38
*** sacharya has quit IRC05:40
*** alkari has joined #openstack-ansible05:49
*** fawadkhaliq has joined #openstack-ansible05:58
*** fawadkhaliq has quit IRC05:58
*** fawadkhaliq has joined #openstack-ansible05:59
*** sdake has joined #openstack-ansible06:12
*** rminmin has joined #openstack-ansible06:12
*** javeriak has joined #openstack-ansible06:14
*** sacharya_ has quit IRC06:28
*** alkari has quit IRC06:33
*** phiche has joined #openstack-ansible06:34
*** javeriak has quit IRC06:36
*** rminmin has quit IRC06:37
*** sdake has quit IRC06:38
*** mss has joined #openstack-ansible06:43
*** mss has quit IRC06:52
*** phiche has quit IRC07:02
*** phiche has joined #openstack-ansible07:15
*** fawadkhaliq has quit IRC07:52
*** fawadkhaliq has joined #openstack-ansible07:53
*** fawadkhaliq has quit IRC07:53
*** fawadkhaliq has joined #openstack-ansible07:54
*** fawadkhaliq has quit IRC07:57
*** fawadkhaliq has joined #openstack-ansible07:57
*** fawadkhaliq has quit IRC07:57
*** fawadkhaliq has joined #openstack-ansible07:58
*** fawadkhaliq has quit IRC08:02
*** fawadkhaliq has joined #openstack-ansible08:03
*** fawadkhaliq has quit IRC08:03
*** fawadkhaliq has joined #openstack-ansible08:04
*** javeriak has joined #openstack-ansible08:19
*** adac has joined #openstack-ansible08:20
adacI'm trying to recover the galera cluster  (All machines have 'seqno:   -1' ) with "mysqld --wsrep-recover" however the sequence number is exact the same as also 'ansible galera_container -m shell -a "cat /var/lib/mysql/grastate.dat"' shows on all three nodes08:21
adacstill the procedure described in http://docs.openstack.org/developer/openstack-ansible/install-guide/ops-galera-start.html cannot start up the cluster anymore08:21
adacany ideas what I might can do about this?08:21
*** permalac has joined #openstack-ansible08:25
adacIt seems there are two warnings when starting the recovery: https://gist.github.com/anonymous/314d8cbcf8b57e981f1108:27
*** mpavone has joined #openstack-ansible08:30
*** tiagogomes has joined #openstack-ansible08:39
odyssey4mekysse cloudnull the keystone_authtoken settings are right as they are, the admin_ options were all deprecated in kilo - the user-name deprecation error is incorrect as user-name is an inappropriate option for any ini files... I think you had a red herring there08:43
odyssey4mecloudnull also, 'auth_plugin = password' is the new default in most services for liberty08:44
odyssey4meadac, so all your nodes have seq -1 ?08:45
adacodyssey4me, exactly yes08:46
odyssey4meadac from /opt/openstack-ansible/playbooks execute: openstack-ansible -e galera_ignore_cluster_state=true galera-install.yml08:47
*** javeriak has quit IRC08:47
odyssey4meevrardjp morning - can you please backport https://review.openstack.org/246503 and https://review.openstack.org/248819 to the liberty branch?08:48
odyssey4memorning iceyao_ :)08:48
*** karimb has joined #openstack-ansible08:48
adacodyssey4me, would this delete the data in the mysql database? I mean in my testing case it is ok, just asking not that I execute this command at some point in production as well  if its harmful ;-)08:53
odyssey4meadac ideally this should not be run in production, but when your whole cluster is at seq -1 then I don't think there's harm in it - all it's going to do is initiate the cluster on one node and then start the service on the others08:56
odyssey4meit's be the equivalent of running: ansible galera_all[0] -m shell -a '/etc/init.d/mysql start --wsrep-new-cluster'08:57
adacodyssey4me, I see. thanks a lot!08:58
hughsaundersodyssey4me: galera:!galera[0]08:59
odyssey4methen ansible galera_all:!galera_all[0] -m shell -a '/etc/init.d/mysql start'09:00
odyssey4methanks hughsaunders :)09:00
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Allow ramdisk_id, kernel_id to be null on schema  https://review.openstack.org/25172009:01
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Add container-reconciler and object-expirer to os-swift  https://review.openstack.org/25172109:01
*** mpavone has quit IRC09:02
adacodyssey4me, I now did "openstack-ansible -e galera_ignore_cluster_state=true galera-install.yml" but "ansible galera_container -m shell -a "cat /var/lib/mysql/grastate.dat"" still shows "seqno -1" Do I still have to restart them manually?09:02
*** fawadkhaliq has quit IRC09:03
*** fawadkhaliq has joined #openstack-ansible09:03
*** karimb_ has joined #openstack-ansible09:05
odyssey4meevrardjp lol, never mind - I see that the horizon branding one is already done09:05
odyssey4meadac it sounds to me like your cluster is utterly broken - you'll have to dig into what went wrong using mariadb/galera documentation09:05
odyssey4meif this is a test setup, then you can simply blow away the containers and rebuild them09:06
*** iceyao_ has quit IRC09:06
odyssey4methe data will still be there anyway09:06
*** karimb has quit IRC09:07
*** xek has joined #openstack-ansible09:08
adacodyssey4me, exactly make everything new09:10
adacodyssey4me, may I ask which command/playbook I have to run for to re-setup the galera cluster?09:11
*** javeriak has joined #openstack-ansible09:16
*** mpavone has joined #openstack-ansible09:16
odyssey4meadac I think it's along the lines of: openstack-ansible lxc-containers-destroy.yml --limit galera_all09:16
odyssey4meadac then you'll need to re-run: openstack-ansible setup-hosts.yml09:20
odyssey4me(to recreate the containers)09:20
odyssey4methen openstack-ansible galera-install.yml09:20
odyssey4me(ro reinstall galera)09:20
adacok thank you!09:21
*** iceyao has joined #openstack-ansible09:21
*** javeriak_ has joined #openstack-ansible09:35
*** javeriak has quit IRC09:38
*** javeriak has joined #openstack-ansible09:39
*** javeriak_ has quit IRC09:40
*** javeriak_ has joined #openstack-ansible09:46
*** javeriak has quit IRC09:49
*** javeriak has joined #openstack-ansible09:57
*** javeriak_ has quit IRC09:58
permalacif I would like to deploy an openstack instance with ansible, where I should start reading?10:15
andymccrpermalac: there is an install guide available here http://docs.openstack.org/developer/openstack-ansible/install-guide/index.html10:19
andymccroh wait instance10:19
andymccryou mean just deploy an instance on an existing openstack cloud using ansible?10:19
permalacsorry, my bad. I mean prepare an openstack installation from scratch.10:20
andymccrahh excellent yes, tehn the above link will work :)10:20
permalacthe install guide is good.10:20
permalacthanks10:20
andymccr*then10:20
andymccrno problem!10:20
andymccrif you run into any problems feel free to drop questions in here - theres usually somebody about who will be able to help10:21
adacodyssey4me,  When reinstalling galera, I get the following error: https://gist.github.com/anonymous/d6ea8af57ebd78cf9ee5 I have no luck with galera it seems (:10:33
odyssey4mepermalac you can kick the tyres with a single VM too, using http://docs.openstack.org/developer/openstack-ansible/developer-docs/quickstart-aio.html10:33
odyssey4meadac if you do openstack-ansible -e galera_ignore_cluster_state=true galera-install.yml now?10:34
adacodyssey4me, trying that out10:35
*** timrc has quit IRC10:47
*** timrc has joined #openstack-ansible10:49
adacodyssey4me, it just finished, it says still the following: https://gist.github.com/anonymous/51cf4702ee80ef0b2ba611:00
adachoweer this installation seems to be fine, there was no error in it11:00
adac*however11:00
*** marekd has joined #openstack-ansible11:04
*** mpavone has quit IRC11:09
adacI destry everything now and rebuild it11:12
adac:)11:13
odyssey4me:)11:19
*** mpavone has joined #openstack-ansible11:21
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-specs: Added spec for policy-files-distribution This explains/opens the discussion to component policy file distribution with the help of openstack-ansible.  https://review.openstack.org/24922711:33
*** misc has quit IRC11:34
*** mss has joined #openstack-ansible11:52
*** misc has joined #openstack-ansible11:54
*** mss has quit IRC11:56
*** javeriak has quit IRC11:57
odyssey4mehughsaunders mattt did you see cloudnull's response in https://review.openstack.org/241483 ?12:05
matttodyssey4me: updated12:08
*** javeriak has joined #openstack-ansible12:21
*** jaypipes has joined #openstack-ansible12:22
*** jaypipes has quit IRC12:49
mhaydenmorning12:49
*** jaypipes has joined #openstack-ansible12:54
*** fawadkhaliq has quit IRC13:00
Sam-I-Ammornings13:00
*** fawadkhaliq has joined #openstack-ansible13:01
*** fawadkhaliq has quit IRC13:02
mgariepygood morning everyone13:02
*** fawadkhaliq has joined #openstack-ansible13:03
*** fawadkhaliq has quit IRC13:03
odyssey4meo/ all13:03
Sam-I-Amyo13:03
*** javeriak has quit IRC13:04
*** fawadkhaliq has joined #openstack-ansible13:04
*** gparaskevas has joined #openstack-ansible13:06
gparaskevashello you guys(cartman voice)13:10
mattthehe13:10
gparaskevascan i have the same variable name in more than one ansible role for the same playbook?13:11
gparaskevasi have a playbooks that uses 3 roles lets say and each role has a variable named game_name:13:11
gparaskevasevery role has the same variable name with diferent value13:12
gparaskevasthe variable file i use is in defaults13:12
gparaskevason every roles13:12
gparaskevasrol*13:12
gparaskevasrole*13:12
matttgparaskevas: is each role getting called at the same time ?13:14
matttsounds like a bad idea personally, but i'm not an ansible expert13:15
gparaskevasbasicaly they ar called like this13:15
gparaskevashttp://paste.openstack.org/show/480494/13:17
odyssey4megparaskevas if you want to use the same name in multiple role defaults or role vars you can, but if you don't namespace them individually then you'll end up with confusion if you ever run two roles together in a playbook with a playbook var or an override var13:17
odyssey4methat's why the best practise is to namespace them13:17
*** persia has joined #openstack-ansible13:18
gparaskevashow do i namespace them?13:18
*** fawadkhaliq has quit IRC13:18
gparaskevaslet me check13:18
*** fawadkhaliq has joined #openstack-ansible13:19
gparaskevasso what i do is wrong?13:19
matttgparaskevas: you gonna have a bad time right there13:20
matttgparaskevas: do role1_game_name, role2_game_name, etc.13:20
odyssey4megparaskevas if you have a role called 'role1' and a var called 'var1', then a role called 'role2' and a var in role2 called 'var1' then if you call both roles with group_vars/playbook vars you can only set the var in group_vars/playbook vars for both to the same value - never different13:20
odyssey4meso to namespace them is to simply call them, in each role, role1_var1 and role2_var113:21
gparaskevasOK this is what i am doing now. let make a meme about that as well13:22
matttgparaskevas: https://cessnachick.files.wordpress.com/2015/08/youre-going-to-have-a-bad-time.png13:23
gparaskevashttp://cdn.meme.am/instances2/500x/3125581.jpg13:23
gparaskevas#proper13:24
gparaskevas:P13:24
gparaskevashahah13:24
mattt:D13:24
gparaskevasthanks both of you guys!13:25
gparaskevasski instructors*13:25
matttgparaskevas: have you been drinking?13:27
mattt:)13:27
gparaskevasloll because of the happiness? nah13:28
gparaskevas:P13:28
gparaskevasmaybe its the cafeine effect :P13:28
*** sdake has joined #openstack-ansible13:34
*** adac has quit IRC13:34
*** sdake has quit IRC13:42
*** permalac has quit IRC13:44
openstackgerritMajor Hayden proposed openstack/openstack-ansible-security: Updating tests for openstack-ansible-security  https://review.openstack.org/25143013:45
cloudnullMorning13:48
kyssegood morning13:49
*** markvoelker_ has joined #openstack-ansible13:49
mhaydenodyssey4me: thanks for the tips on https://review.openstack.org/#/c/251430/ -- i've got it working now13:49
*** sdake has joined #openstack-ansible13:50
*** tlian has joined #openstack-ansible13:51
cloudnullKysse did you make it all happy yet?13:51
cloudnullOr is it still mad?13:52
*** mss has joined #openstack-ansible13:52
kysseshe is angry.13:52
cloudnullGparaskevas rofl13:53
*** sdake_ has joined #openstack-ansible13:53
gparaskevascloudnull: about the meme :P hehe13:53
cloudnullKysse same error? Vif binding?13:54
*** sdake has quit IRC13:54
cloudnullYup the meme made me lol13:54
gparaskevascloudnull: hah thanks13:54
*** mss has quit IRC13:57
kyssesame error13:57
cloudnull... :'(13:59
cloudnullAnd in all we've not seen anything of merit in the neuron agent logs, right?14:00
cloudnullLinuxbridge,l3,server on computer node and neuron containers14:01
kyssenothing. Linux-bridge logs are empty @neutron containers14:05
cloudnullNeutron_agent containers right?14:06
cloudnullI'd the neutron-linuxbridge-agent running?14:06
cloudnull*is14:07
matttkysse: do a nova service-list14:15
kysseI got 2 jump on the bus, I'm traveling from Turku Finland to Pori finland now, so lets check those things in 2h.14:17
*** targon has joined #openstack-ansible14:18
openstackgerritMajor Hayden proposed openstack/openstack-ansible-security: Updating tests for openstack-ansible-security  https://review.openstack.org/25143014:19
openstackgerritMajor Hayden proposed openstack/openstack-ansible-security: Check mode compatibility for security role  https://review.openstack.org/24581314:19
cloudnullKysse have a nice trip14:22
openstackgerritMajor Hayden proposed openstack/openstack-ansible-security: Check mode compatibility for security role  https://review.openstack.org/24581314:24
openstackgerritMajor Hayden proposed openstack/openstack-ansible-security: Updating tests for openstack-ansible-security  https://review.openstack.org/25143014:24
mhaydengerrit dependencies are wild sometimes14:24
*** ctina has joined #openstack-ansible14:26
openstackgerritDarren Birkett proposed openstack/openstack-ansible: Add new alarm for the galera checks  https://review.openstack.org/25189014:27
*** KLevenstein has joined #openstack-ansible14:31
*** markvoelker has joined #openstack-ansible14:34
*** markvoelker_ has quit IRC14:35
*** daneyon has joined #openstack-ansible14:53
*** daneyon has quit IRC14:55
*** targon has quit IRC14:57
matttcloudnull: quick question for you15:04
*** linggao has joined #openstack-ansible15:05
linggaoGood morning.15:05
*** spotz_zzz is now known as spotz15:06
linggaoI have installed all-in-on using ansible. But somehow I could not find the container for nova_compute. And nova hypervisor-list shows no hypervisors.15:06
matttlinggao: nova-compute will run on the host itself15:07
linggaomattt, thanks. Good to know. But why there is no hypervisor listed?15:07
linggao nova hypervisor-list15:09
linggao+----+---------------------+-------+--------+15:09
linggao| ID | Hypervisor hostname | State | Status |15:09
linggao+----+---------------------+-------+--------+15:09
linggao+----+---------------------+-------+--------+15:09
bgmccollumwell, seems nova-compute isn't running...15:09
bgmccollumor isn't checking in15:09
bgmccollumcheck /var/log/nova/nova-compute.log for clues15:10
*** daneyon has joined #openstack-ansible15:10
cloudnullMattt what's up?15:14
matttcloudnull: hey!  https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-nova-install.yml#L79-L9315:14
matttcloudnull: we reference nova_virt_type there but that var isn't available in the pre_tasks from what i can tell ... any way around that short of moving the task into the os_nova role?15:15
*** linggao_ has joined #openstack-ansible15:15
*** linggao has quit IRC15:15
*** linggao_ has quit IRC15:16
*** linggao has joined #openstack-ansible15:17
cloudnullmattt the shell task ?15:17
matttcloudnull: yah, the when: clause references nova_virt_type15:18
matttbut unless you explicitly set that it will cause os-nova-install.yml to fail15:18
matttor do we mandate that you always set that in your env ?15:18
*** Mudpuppy has joined #openstack-ansible15:18
matttlinggao: did you see bgmccollum's response ?15:19
cloudnullah i see15:19
cloudnulli think we could change https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-nova-install.yml#L90 to - 'nova_virt_type is defined and nova_virt_type==kvm'15:19
*** baker has joined #openstack-ansible15:20
matttcloudnull: but that would skip the task when someone is actually using kvm but didn't set an override :)15:20
cloudnullrather - "nova_virt_type is defined and nova_virt_type=='kvm'"15:20
cloudnullcorrect.15:20
cloudnullbut that task is only used when nova is in a container15:20
matttwhich is what i'm testing now :P15:21
linggaomattt, yes. I am trying to bring up the nova-compute by hand on host.  Got the following error. http://pastebin.com/77HEEsQs15:21
cloudnullso then I would say that the variable would need to be defined when using kvm and running nova in a container15:21
cloudnullbecause the role has no clue what a container is15:21
matttcloudnull: k15:23
matttcloudnull: i was hoping you could vars_files os_nova's defaults file but that doesn't even work15:23
matttat least not for pre_tasks15:23
cloudnullto that end, I believe in the liberty release we've removed everything container releated from the various roles and put it all in the tasks.15:24
cloudnullfor the plays that is15:24
cloudnullmattt:  i think you can use straight "vars" in the play15:24
matttcloudnull: i see, let me have a look ... i'm actually fiddling w/ kilo here specifically15:25
*** fawadkhaliq has quit IRC15:25
cloudnullso vars: "{{ _virt_type | default('kvm') }}"15:25
cloudnullthen in the task use _virt_type as the check variable15:25
matttcloudnull: ah i see what you mean, not ideal because we now decouple from defaults in os_nova ... but yeah it's either that or we bail if someone doesn't specify it15:26
matttwhat is worse do you think?15:26
*** fawadkhaliq has joined #openstack-ansible15:26
*** markvoelker_ has joined #openstack-ansible15:28
cloudnulli think its safe to assume that KVM will be used when running nova-compute in an LXC container15:30
cloudnullmaybe add a check before hand to see if "/dev/kvm /dev/kvm" are available15:31
*** markvoelker has quit IRC15:32
cloudnullhowever this begs the larger question, should nova compute run in a container? but Ill leave that to the philosophers.15:32
*** markvoelker_ has quit IRC15:33
*** gtt116_ has joined #openstack-ansible15:34
*** sdake_ has quit IRC15:36
*** gtt116__ has quit IRC15:38
linggaomattt, bgmccollum   I got hte following error when starting nova-compute.  2015-12-01 09:36:27.571 4038 WARNING nova.virt.libvirt.driver [req-f7b2861f-dba5-40b3-b489-fe11794f6bcf - - - - -] Cannot update service status on host "ctl2b" since it is not registered.15:39
matttlinggao: yeah probably due to the libvirt error in your paste15:41
matttlinggao: what are you running this on ?15:41
linggaomattt, I am running it on ubuntu 14.04.3 LTS15:41
matttlinggao: you running kvm in kvm or something?15:42
linggaomattt, the host is a vm. I believe it is xen.15:43
matttlinggao: try setting virt_type = qemu in /etc/nova/nova.conf and restart nova-compute15:44
linggaook15:44
matttlinggao: if that works, you can add an override to /etc/openstack_deploy/user_variables.yml (nova_virt_type: qemu) on your host where you run ansible15:45
bgmccollumlinggao: can you paste /var/log/libvirt/libvirtd.log too...?15:46
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Convert AIO bootstrap from bash to Ansible  https://review.openstack.org/23952515:46
linggaomattt the virt_type is already qumu in nova.conf. It is default from ansible.15:46
matttbummer :(15:46
linggaobgmccollum, http://pastebin.com/GY35KKKz15:47
bgmccollumlinggao: i know how to fix this...15:48
bgmccollumodyssey4me: we have a live example of an unidentifiable cpu like i talked about last week...!15:48
matttbgmccollum: yay !15:48
odyssey4meheh bgmccollum15:49
bgmccollumlinggao: ok, you need to replace your /usr/share/libvirt/cpu_map.xml with this -- https://raw.githubusercontent.com/cloud-training/rpc-heat-ansible/master/playbooks/files/usr/share/libvirt/cpu_map.xml15:49
odyssey4mebgmccollum I think the right solution for this is to get it upstream into the libvirt codebase - but we may also have to work around it in some way...15:49
bgmccollumthen restart libvirt-bin, then restart nova-compute15:50
linggaobgmccollum, ok. Let me try.....15:50
odyssey4memhayden alright - let's see if the updated https://review.openstack.org/239525 passes - if it does, do you have some time to work through it tomorrow?15:50
mhaydeni should15:50
bgmccollumodyssey4me: can we also grab /var/log/libvirt/libvirtd.log in the gate?15:51
odyssey4memhayden see if you can review it today already and have any questions/comments15:51
mhaydenodyssey4me: it seems like there are a bunch of deletions in there, but i think that's from rebasing15:52
odyssey4mebgmccollum we should - I think we'd need a patch to collect it though - I take it that it needs to come from the compute host?15:52
mhaydenyeah, rebasing15:52
mhaydenokay nevermind -- got spooked there at first when i diff'ed 63/6415:52
bgmccollumodyssey4me: yes compute host...15:52
bgmccollumodyssey4me: im guess thats lives somewhere in infra?15:53
odyssey4memhayden it's quite a large restructure. but a lot of what you did was used... I just managed to cut down a lot of the shell stuff by using modules15:53
mhaydenthanks for hacking through that, odyssey4me15:53
odyssey4mebgmccollum the log collection has to be set by us - right now I don't think we have anything that tells us to collect that log15:53
mhaydencould anyone give sharma some advice on the openstack-dev thread?15:54
mhaydeni have some ideas but someone a little more experienced might be better15:54
odyssey4mebgmccollum I'll look into that once we have this updated bootstrap and gate check tested to be working. It's in my TODO list. :)15:54
linggaobgmccollum, it does not work.  same error.15:55
matttderp.15:55
*** javeriak has joined #openstack-ansible15:55
bgmccollumlinggao: is this your own xen host?15:56
linggaono, I rented this from softlayer.15:56
odyssey4memhayden the good news is that both the voting and non voting checks have gone through the bootstrap already :)15:56
*** javeriak has quit IRC15:57
*** javeriak_ has joined #openstack-ansible15:57
bgmccollumlinggao: that cpu_map.xml was specifically crafted to match the cpu flags passed down to rackspace public cloud instances on xen hosts...15:57
bgmccollumlinggao: what you're going to need to do is customize it to match the CPU flags your host passes to your guests15:57
odyssey4mebgmccollum are you sure that this issue can't be worked around rather simply using the cpu_type nova.conf setting?15:57
bgmccollumi was hoping it would work out of the box...15:58
bgmccollumodyssey4me: maybe...can you specify a make / model?15:58
linggaobgmccollum, thanks. I will give it a try.15:58
odyssey4mebgmccollum yes, I think so - it's been a while since I checked15:58
odyssey4meany volunteers to run the bug triage? my brain is fried15:58
odyssey4mecloudnull ^15:59
cloudnullI can15:59
odyssey4methanks cloudnull - I need to step outside for a few minutes, brb16:00
-cloudnull- bug triage time cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, dolphm, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, mhayden, scarlisle, luckyinva, ntt, javeriak16:00
mattt\o16:00
*** markvoelker has joined #openstack-ansible16:00
cloudnullopps. i think i sent that twice ...16:00
cloudnulleither way o/16:00
cloudnullshould be faily quick , we only have two new issues16:01
cloudnullill let people filter in for a min or two16:02
* mhayden woots16:03
* andymccr filters16:04
palendaeo/16:04
cloudnullso first up16:05
cloudnullhttps://bugs.launchpad.net/openstack-ansible/+bug/150628516:05
openstackLaunchpad bug 1506285 in python-openstackclient "11.2.1 : openstack client with V3 auth causes usability issues" [Undecided,New]16:05
Sam-I-Ami gotta sit this one out16:06
cloudnullwhich leads to https://bugs.launchpad.net/openstack-ansible/+bug/1518351 and this patch16:08
openstackLaunchpad bug 1518351 in openstack-ansible trunk "Playbooks stop when enabling keystone ldap backend " [Medium,In progress] - Assigned to Kevin Carter (kevin-carter)16:08
cloudnullhttps://review.openstack.org/#/c/248900/16:08
cloudnullwhich seems to need a reabase16:08
cloudnulland this patch to OSC https://review.openstack.org/#/c/251597/16:09
cloudnullthe last patch is interesting . it essentially fixes the client by assuming the domain is "default"16:10
*** sdake has joined #openstack-ansible16:11
cloudnullbut Keystone PTL is against that, so id assume that would not go in16:11
stevelleis https://review.openstack.org/#/c/238509/ worth mentioning as related too16:12
cloudnullfor the bug itself, i think https://review.openstack.org/#/c/248900/ resolve it for us to make sure specific ldap commands are not run when the service users are in LDAP.16:12
cloudnullstevelle: ++16:12
*** mss has joined #openstack-ansible16:12
cloudnullhttps://review.openstack.org/#/c/238509/ is also related to the various ldap fixes.16:12
matttdidn't we triage these same issues last week?  :)16:13
cloudnullnow if we only had an LDAP gate... wink wink, nudge nudge (somebody)16:13
cloudnullmattt:  likely16:13
cloudnullidk though16:13
odyssey4mecloudnull yeah, working on that soon16:14
cloudnulli wasnt mentioning names, :p16:14
matttcloudnull: i could be mistaken!16:14
cloudnullso for this issue https://bugs.launchpad.net/openstack-ansible/+bug/150628516:14
openstackLaunchpad bug 1506285 in python-openstackclient "11.2.1 : openstack client with V3 auth causes usability issues" [Undecided,New]16:14
cloudnullit was marked invalid, I tend to agree that the specific client issue is invalid for our case.16:15
cloudnulland that the other two PRs in flight are fixing the core problem for us16:15
*** linggao has quit IRC16:15
*** linggao has joined #openstack-ansible16:15
odyssey4meagreed - the source issue is being dealt with in the ldap patches... the usability issue has to be dealt with upstream16:15
cloudnulltiagogomes: worked on https://review.openstack.org/#/c/238509/ and it would be nice to land that16:16
cloudnulland i added a check to skip tasks when ldap is used https://review.openstack.org/#/c/248900/16:16
*** phiche has quit IRC16:18
cloudnulldone16:18
cloudnullnext16:18
cloudnullhttps://bugs.launchpad.net/openstack-ansible/+bug/152123316:18
openstackLaunchpad bug 1521233 in openstack-ansible "Adjust commonly failing tasks in openstack-ansible-security" [Undecided,New] - Assigned to Major Hayden (rackerhacker)16:18
cloudnullmhayden: what say you?16:19
* mhayden winks16:19
matttnotbug16:19
openstackgerritDarren Birkett proposed openstack/openstack-ansible: Update pinned lxc version  https://review.openstack.org/25194516:19
mhaydenthis one blocks proper gate checks16:19
mhaydeni'm almost done with the dev work on 152123316:19
mhaydeni'm gradually turning those fail: module uses into actual changes with variables to toggle them on/off16:20
cloudnullwishlist, triaged, assigned . done16:20
mhaydenthanks sir ;)16:20
cloudnull:)16:20
cloudnullso thats the end of the new ones.16:20
cloudnullanything that we need to address otherwise?16:20
* mhayden has one, searching16:21
mhaydenhttps://bugs.launchpad.net/openstack-ansible/+bug/151614216:22
openstackLaunchpad bug 1516142 in openstack-ansible "openstack-ansible-security needs a check mode for auditing" [Wishlist,In progress] - Assigned to Major Hayden (rackerhacker)16:22
mhaydenthat one is blocking some of my future work there, especially gating16:22
mhaydenif anyone has time to put eyeballs on it, i'd be much obliged16:22
mhaydenodyssey4me gave some feedback already16:22
* cloudnull looking16:22
*** jwagner has joined #openstack-ansible16:23
cloudnulli can look that over in a bit16:23
mhaydenrelevant review -> https://review.openstack.org/#/c/245813/16:23
*** jwagner has quit IRC16:23
mhaydenit didn't get linked to the ticket for some reason :/16:23
*** jwagner has joined #openstack-ansible16:24
cloudnullits all good.16:24
cloudnullit happens16:24
*** daneyon has quit IRC16:24
mhaydenonce the gate checks are ready for check/functional, i should be able to get this thing looking more like a finisheed product16:24
mhaydenit might even impress mattt... just maybe16:24
mhayden;)16:25
odyssey4memhayden fyi it didn't get linked because your first commit didn't have the closes-bug line in the commit message :)16:25
mhaydeni figured it was my error :(16:25
mhaydeni added to the bug ticket16:25
cloudnullwhat else we got?16:26
palendaemhayden: Yeah, has to be the first commit. Subsequent ones won't do it16:26
odyssey4mesubsequent additions to the commit message will only reflect in the bug when it merges16:26
openstackgerritKevin Carter proposed openstack/openstack-ansible: Updated the py_pkgs lookup plugin for multi source  https://review.openstack.org/24148316:29
cloudnullmattt hughsaunders ^16:29
cloudnullhughsaunders:  i made the changes on your box , you should be able to clean that one up now16:30
cloudnullif you want that is16:30
*** jwagner has quit IRC16:30
*** phiche has joined #openstack-ansible16:30
*** sdake has quit IRC16:30
*** mpavone has quit IRC16:31
hughsaunderscloudnull: I'll reuse it for testing your next patchset :)16:31
cloudnullits currently checked out on your box16:31
bgmccollumanyone know the difference between libvirt cpu_modes `host-model` and `host-passthrought`...?16:32
bgmccollumfor reference -- https://github.com/openstack/nova/blob/78d07cce136467328eeabeeba90a8e914635ec85/nova/virt/libvirt/driver.py#L219-L23016:32
*** jwagner has joined #openstack-ansible16:33
*** jwagner has quit IRC16:33
cloudnullbgmccollum: https://wiki.openstack.org/wiki/LibvirtXMLCPUModel16:34
bgmccollumyeah stumbled upon that...nova16:35
bgmccollumnova's default is host-model...but it says..."Beware, due to the way libvirt detects host CPU and due to the fact libvirt does not talk to QEMU/KVM when creating the CPU model, CPU configuration created using host-model may not work as expected."16:35
bgmccollumgreat16:35
odyssey4meyeah, host-model is meant to be the safer, more compatible one for situations where you want to live migrate between compute hosts that are different16:36
odyssey4methe most safe way is to specify a custom model and keep that consistent across all hosts, or at least across a host aggregate16:37
cloudnullbgmccollum:  i think you can blanket OpenStack with that statement. . . "Beware, due to the way OpenStack detects bits and due to the fact OpenStack does not talk to some bits when creating things, configurations using somethings may not work as expected."16:37
* cloudnull a helper16:37
bgmccollumlulz16:38
bgmccollumcpu_mode: custom16:39
bgmccollumcpu_model: 48616:39
bgmccollumenjoy your cloud16:39
odyssey4melol16:39
bgmccollumthat shit will work everywhere16:39
cloudnullcpu_model: 486dx2 # for performance16:39
*** sacharya has joined #openstack-ansible16:39
bgmccollumcpu_model: ppc6416:40
*** gparaskevas has quit IRC16:40
bgmccollumcpu_model: 808816:40
bgmccollumblinkenlights16:40
*** jwagner has joined #openstack-ansible16:41
mhaydenodyssey4me: did you want me to whip out a scalpel and pull out unneeded stuff from the scripts library?16:41
odyssey4memhayden I think for now a thorough review is more important... let's find all the bugs and clean them up16:42
mhaydenah, so did you want me to take action on your comment for now?16:42
* mhayden is waiting on jenkins to finish16:43
odyssey4memhayden nope - just test it, review how it's done, see if you find any issues16:43
mhaydencan do16:43
odyssey4melet's mark all the issues in this review, then work through the patch in detail tomorrow (I need a fresh head)16:43
mhaydenhah i can imagine16:44
mhaydeni think i lost brain cells working through these patches16:44
mhaydenalso, the ssh errors randomly popping up along the way didn't help :P16:45
odyssey4meeven without the scripts library removal, that's over 500 lines of bash code removed :)16:45
mhaydenwoot16:45
mhaydennot that i dislike bash or anything, but i like seeing more ansible than bash :P16:45
odyssey4mego on mhayden - do us a meme for that16:45
mhaydenWAT16:45
*** mgoddard_ has joined #openstack-ansible16:46
odyssey4mewell, I suppose I can take you through it now if you like?16:46
odyssey4memy brain seems to be enjoying some caffiene16:46
bgmccollumanyone noticing a difference in SSH failure?16:46
mhaydenodyssey4me: i've gotta scoot to meet my wife for lunch16:46
bgmccollumhopefully a reduction16:46
odyssey4mebgmccollum haven't seen one in a while now16:46
bgmccollumnod...disk prep for HP too right?16:47
odyssey4mebgmccollum yeah, the combo of both has made master/liberty work nicely - https://review.openstack.org/239525 has converted the bash prep scripts to ansible, which is a lot cleaner and easier to read16:48
odyssey4meincluding a pre-flight check :) https://review.openstack.org/#/c/239525/64/tests/roles/bootstrap-host/tasks/check-requirements.yml,cm16:48
bgmccollumnice...16:48
*** mgoddard has quit IRC16:49
*** fawadkhaliq has quit IRC16:50
spotzbgmccollum Are you talking recently? They were awful during the handon session at summit:(16:51
bgmccollumspotz: yes, specifically SSH failure in gate checks16:52
bgmccollumnot SSH failures in general ;)16:52
*** daneyon has joined #openstack-ansible16:53
spotzahh yeah I haven't submitted anything for you guys in a bit and that was a doc fix. So.. no comment:)16:55
*** sdake has joined #openstack-ansible16:57
odyssey4memhayden are you ok with me pulling out the red hat support for now with the view that it gets added in a subsequent patch that can properly test it?17:01
*** mgoddard has joined #openstack-ansible17:01
*** mgoddard_ has quit IRC17:01
bgmccollumlinggao: glad you got it working...!17:02
linggaothanks bgmccollum!17:03
bgmccollumlinggao: do you mind trying something else?17:04
*** daneyon_ has joined #openstack-ansible17:05
bgmccollumlinggao: instead of letting libvirt try to identiy the CPU make / model, i want to try specifying it explicitly...17:05
linggaobgmccollum, yes. Please let me know how.17:06
*** daneyon has quit IRC17:08
bgmccollumlinggao: in your /etc/nova/nova.conf, under the [libvirt] section...set cpu_mode to custom, and cpu_model to Haswell...restart libvirt-bin and nova-compute...check your nova-compute.log and libvirtd.log for errors...17:08
bgmccollumif it doesnt like that model...then you can step back one generation at a time...SandyBridge, Westmere, Nehalem, Penryn, Conroe etc...17:11
linggaobgmccollum, should I change the cpu_map.xml to original or use the one I just modified?17:13
bgmccollumto be scientific, you should backup the working cpu_map.xml, and revert to the original...17:13
linggaook17:14
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Convert AIO bootstrap from bash to Ansible  https://review.openstack.org/23952517:15
*** linggao has quit IRC17:16
*** linggao has joined #openstack-ansible17:16
bgmccollumlinggao: if none of that works, we have 1 more thing to try...17:18
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Convert AIO bootstrap from bash to Ansible  https://review.openstack.org/23952517:20
*** fawadkhaliq has joined #openstack-ansible17:22
*** phiche has quit IRC17:24
*** phiche has joined #openstack-ansible17:25
*** baker has quit IRC17:31
*** alkari has joined #openstack-ansible17:34
linggaobgmccollum, I have tried a few. Haswell, SandyBridge, Westmere, Nehalem, Penryn, Conroe. None of them worked.17:36
bgmccollumlinggao: ok...one last thing to try...17:36
bgmccollumlinggao: can you comment out the cpu_model, and set cpu_mode to host-passthrough?17:37
*** karimb_ has quit IRC17:37
bgmccollumrestart and tail logs as before...17:37
bgmccollumand if that doesnt work either...then its back to cpu_mode host-model, and your custom cpu_map.xml... :/17:38
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Add new alarm for the galera checks  https://review.openstack.org/25189017:38
linggaobgmccollum, I did not work. Same errors in both logs.17:40
*** baker has joined #openstack-ansible17:40
bgmccollumdang...ok. thanks for checking. sounds like those options won't work for us either...cc odyssey4me17:40
*** gtt116__ has joined #openstack-ansible17:42
bgmccollumlinggao: so for now...put your custom cpu_map.xml back in place, set cpu_mode = host-model, remove cpu_model option, and restart the bits...17:42
*** gtt116_ has quit IRC17:45
bgmccollumfood...17:50
mhaydenodyssey4me: sounds okay with me17:51
linggaobgmccollum, sure. thanks a lot!17:51
linggaofood too....17:51
odyssey4memhayden I found some leftover bits, and cleaned up the scripts library a bit too :)17:51
mhaydengracias17:52
odyssey4memhayden there does appear to be an issue with some of the new providers in the gate check - the updated patch has failed in the voting gate already, but is waiting for the RAX check to complete.17:52
mhaydendarn17:53
odyssey4meit would seem that this new provider either doesn't have a second disk, or the bash to find the largest disk is wonky17:53
odyssey4methis rendered a blank result: lsblk -brndo NAME,TYPE,RO,SIZE | awk '/d[b-z]+ disk 0/{ if ($4>m){m=$4; d=$1}}; END{print d}'17:54
cloudnullany chance i can get some love on https://review.openstack.org/#/c/241483/ ?17:54
odyssey4mecloudnull I'll ask mattt and hughsaunders to re-test/re-review tomorrow, and I'll also re-test.17:56
prometheanfirehave people been seeing can't connect ssh issues since https://github.com/openstack/openstack-ansible/commit/fa0772aa1b0d036e8b7e490a37cd37a65d6ee50a ?17:57
odyssey4meit'd be nice to get some other reviews though, perhaps sigmavirus24_awa or stevelle have some time to look through it?17:57
odyssey4meprometheanfire nope, that was committed to resolve ongoing ssh connection issues17:57
prometheanfirecool17:57
* mhayden is in git branching hell17:58
prometheanfirejust told qe to do that to fix that17:58
prometheanfiremhayden: why?17:58
mhaydenjust shuffling some things around and got my branches mixed up :)17:58
mhaydentrying to bring in odyssey4me's new stuff17:58
prometheanfirefail17:58
*** dmsimard is now known as dmsimard|afk18:00
mhaydenodyssey4me: i think bootstrap_host_data_disk_device comes back as defined, but empty18:01
*** sdake has quit IRC18:01
odyssey4memhayden which frighteningly may mean there is only one disk - we'll have to wait for the whole job to complete to review all the logs18:01
mhaydenodyssey4me: the builds on raxcloud won't have a data disk18:02
odyssey4memhayden they do ;)18:02
mhaydenreally?18:02
odyssey4memhayden  the flavor used by infra, yes18:02
mhaydenthey didn't when i was hacking through the bootstrap stuff, unless i totally missed it18:03
odyssey4memhayden see this example: http://logs.openstack.org/90/247290/4/check/gate-openstack-ansible-commit-nv/70d7c0b/logs/instance-info/host_info_1448450817.log18:03
odyssey4menotice xvde18:03
mhaydenweird18:04
mhaydeni wonder if that device has a partition table when the instance first comes up18:04
odyssey4memhayden nope: http://logs.openstack.org/90/247290/4/check/gate-openstack-ansible-commit-nv/70d7c0b/logs/instance-info/host_info_1448447420.log18:04
mhaydenimma wander back to the security stuff until jenkins finishes doing its thing ;)18:04
mhaydenodyssey4me: wait, block devices section shows xvde + xvde118:05
odyssey4meyeah, so initially it's partitioned but not mounted18:05
mhaydenoh i was talking about the dsvm, not the nv18:05
odyssey4mewe trash and make it however we want it18:05
odyssey4methe nv build is always rax right now :)18:06
odyssey4methe dsvm build is all over the place18:06
odyssey4mehere's a rax dsvm build: http://logs.openstack.org/90/247290/4/check/gate-openstack-ansible-dsvm-commit/6550631/logs/instance-info/host_info_1448447570.log18:06
*** sdake has joined #openstack-ansible18:06
odyssey4mesee under block devices - xvde is still there18:07
odyssey4methe builds are using the same flavor, just a different image18:07
mhaydenweird, okay18:07
*** alkari1 has joined #openstack-ansible18:09
*** alkari has quit IRC18:10
*** eil397 has joined #openstack-ansible18:14
*** linggao_ has joined #openstack-ansible18:16
*** linggao has quit IRC18:16
*** iceyao has quit IRC18:16
*** sdake has quit IRC18:16
*** iceyao has joined #openstack-ansible18:22
*** alkari has joined #openstack-ansible18:22
*** phiche has quit IRC18:23
*** alkari1 has quit IRC18:23
*** mgoddard has quit IRC18:24
*** mancdaz has quit IRC18:26
odyssey4meOMG, the blocker to LB going voting in the gate is finally voted through: https://review.openstack.org/19348518:27
*** mancdaz has joined #openstack-ansible18:27
openstackgerritMajor Hayden proposed openstack/openstack-ansible-security: Adjusting commonly failing tasks  https://review.openstack.org/25200918:27
*** galstrom_zzz is now known as galstrom18:41
*** baker has quit IRC18:42
openstackgerritKevin Carter proposed openstack/openstack-ansible: Functional backport for the new repo-build process  https://review.openstack.org/24421518:45
kyssemancdaz: all running and up18:47
cloudnullstevelle: logan- mgariepy https://review.openstack.org/#/c/244215/ was updated to rebase off the head of kilo and to pull in the changes from master. if you all can revote on that it would be appreciated.18:47
cloudnullkysse: working ?18:47
kyssenope18:48
kysse 16:15       mattt| kysse: do a nova service-list18:48
kysseI'm thinking about downgrading ansible playbook18:48
cloudnullwhat about `neutron agent-list`18:48
d9k:-)18:49
kysseall true and :-)18:49
cloudnullkysse:  so i did an install last night of 10 nodes off the head of liberty and I cant recreate the issue ...18:49
kyssehngh.18:49
kysseand compute node should not have any lxc containers?18:49
cloudnulland these were 10 physical nodes18:49
cloudnullcorrect18:50
kyssethis is killing me18:50
logan-cloudnull: ill do a build an revote in a few thanks!18:50
cloudnullthank you !18:50
cloudnullon the neutron agent container we should be seeing traffic and logs indicating bridges and namespaces are being created.18:50
cloudnullkysse:  your saying that the neutron agent container has no logs18:51
kyssewell, it does now. But no errors.18:51
cloudnulldo you mind clearing the log file, restarting the service, building a vm and posting that data ?18:51
kysseok18:52
kyssecont1 http://sprunge.us/KMVg cont2 http://sprunge.us/KMVg18:55
kysseplugin rpc error appears when restarting service18:56
*** phiche has joined #openstack-ansible18:56
kyssearght. cont1 //sprunge.us/QcVZ cont2 http://sprunge.us/KMVg18:56
cloudnullso that error brings up https://bugs.launchpad.net/neutron/+bug/147058418:59
openstackLaunchpad bug 1470584 in neutron "Linux bridge agent fails badly with physical_interface_mappings issue" [Undecided,Fix released] - Assigned to Kevin Benton (kevinbenton)18:59
cloudnullwhich was raised by Sam-I-Am18:59
cloudnulland references at the bottom "on Liberty for specifying a wrong interface."18:59
*** baker has joined #openstack-ansible19:00
cloudnullall of this leads me to believe the interface mapping is still wrong19:03
cloudnullkysse:  whats the `physical_interface_mappings` say in the neutron agent container and the compute node ?19:05
cloudnullprometheanfire: looks like you ran into this a bit ago too http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2015-10-22.log.html ?19:07
prometheanfirewhich line?19:08
cloudnullRE: CRITICAL neutron [-] AttributeError: 'LinuxBridgeNeutronAgentRPC' object has no attribute 'plugin_rpc'19:09
d9kvlan:br-vlan : compute, vlan:eth11 : infra1_agentcont19:10
prometheanfireoh19:10
openstackgerritMerged openstack/openstack-ansible: Update pinned lxc version  https://review.openstack.org/25194519:10
prometheanfireit db_upgrade didn't make an entry into a table19:10
cloudnullwas that for security group related ?19:11
prometheanfirehttp://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2015-10-22.log.html#t2015-10-22T19:30:1919:11
prometheanfireinsert into networksecuritybindings values ('34d13f35-d93d-4baf-b493-4b079518c789', 't');19:12
cloudnulld9k:  on compute br-vlan exists,  on infra1_agentcont eth11 exists ?19:12
prometheanfirepostgres syntax19:12
d9kyes19:12
*** erikmwilson has quit IRC19:15
*** erikmwilson has joined #openstack-ansible19:16
*** linggao_ has quit IRC19:16
*** linggao_ has joined #openstack-ansible19:16
cloudnulld9k: can you check your DB to see if the "networksecuritybindings" table exists? and that it has entries for your networks in it ?19:17
cloudnulld9k: kysse you shouls see something like http://cdn.pasteraw.com/lopivmwhlukzkwkirgrz1jzscfal7qr19:17
cloudnullif not we can add it like prometheanfire had to and see if theres some random issue causing that to fail in upstream neutron19:18
d9khttp://sprunge.us/AfcM19:21
prometheanfirethat looks right to me19:22
cloudnullok well thats not it , it would seem19:22
cloudnulld9k:  do you have any istances still in error mode ?19:26
cloudnullcan you do a nova show <ID>19:26
d9khttp://sprunge.us/WTid19:27
cloudnullalso do you have the commands used when the networks were created ?19:27
cloudnulloh, nevermind i think they were done in horizon, right?19:28
d9ky19:28
cloudnullwell thats showing "no valid host" do you have a lot of error mode VMs ?19:29
kysse1 or two19:29
cloudnullSam-I-Am: ^ is that still what nova will show even if theres a vif issue?19:29
kysseit always says no valid host19:31
kysseI thought it was cuz the "no binding plaaplaa" error19:31
kysseor what it was.. port binding failed or something19:31
*** sdake has joined #openstack-ansible19:36
*** baker has quit IRC19:42
*** dmsimard|afk is now known as dmsimard19:45
*** karimb has joined #openstack-ansible19:46
Sam-I-Amcloudnull: yeah thats the generic error message19:48
*** karimb has quit IRC19:51
cloudnullmaybe this is all related to the way the networks were created at this point ?19:52
cloudnullkysse, Sam-I-Am, d9k ^19:52
kyssemaybe19:53
*** sdake_ has joined #openstack-ansible19:55
*** sdake has quit IRC19:55
kysseso using neutron cli then..20:00
*** sdake_ is now known as sdake20:03
Sam-I-Amis the network available on the compute hosts?20:06
kysse.. how?20:07
*** mss_ has joined #openstack-ansible20:08
*** fawadkhaliq has quit IRC20:09
*** mss has quit IRC20:11
Sam-I-Amkysse: what kind of network are you trying to use?20:12
kyssevlan20:12
Sam-I-Amprovider or private?20:12
kysseprovider20:14
Sam-I-Amis the interface (or bridge) defined in interface_mappings available on your infra and compute nodes?20:16
*** linggao_ has quit IRC20:16
*** linggao_ has joined #openstack-ansible20:16
kysseyes it is.20:16
kyssehmh.20:16
kysseI created a network with neutron cli20:16
kyssenetwork's ports show status down20:16
Sam-I-Amthats normal until you plug it into something20:18
*** phiche1 has joined #openstack-ansible20:19
*** phiche has quit IRC20:20
linggao_Hi all, I got all-in-one installed and one instance deployed. But cannot ping the instance. I can get into the instance from the console on horizon.20:31
linggao_any hint?20:31
*** errr_ is now known as errr20:32
linggao_I am using the image from installed by os-tempest-install.yaml20:32
bgmccollumlinggao: security groups?20:33
linggao_bgmccollum, I did not specify security groups with nova boot command.20:35
bgmccollumyou can change them after booting...20:35
linggao_ok. let me try...20:36
cloudnullkysse: anything when creating the network by hand ?20:45
linggao_bgmccollum, it works now. thanks again :-)20:57
bgmccollumlinggao_ great...!20:57
kyssenope not really21:00
*** sdake has quit IRC21:06
*** sdake has joined #openstack-ansible21:08
*** phiche has joined #openstack-ansible21:09
*** phiche1 has quit IRC21:09
cloudnullkysse:  i really dont know what the blocker is at this point.21:10
kysseyep21:12
kyssewhat version of openstack-ansible you suggest?21:12
*** sdake has quit IRC21:12
*** jbweber has quit IRC21:13
*** daneyon_ has quit IRC21:13
*** sdake has joined #openstack-ansible21:14
*** jbweber has joined #openstack-ansible21:15
*** linggao_ has quit IRC21:16
*** linggao_ has joined #openstack-ansible21:17
cloudnullkysse: liberty is working for me off the head of the liberty branch. you could give 11.2.5 a go which'll deploy kilo21:20
*** javeriak_ has quit IRC21:21
kysseI'm just wordless cuz I dont get _any_ good log error message for this problem. I'd hate to run something old.21:21
kyssemaybe I just have to create virtual enviroment for this deployment stuff. It's not fun to reainstall all those machines and configure network every time21:22
kysse:p21:22
*** ctina_ has joined #openstack-ansible21:22
kyssewhat you had to do to repeat that error message?21:25
*** ctina has quit IRC21:26
*** ctina_ has quit IRC21:27
*** harlowja has quit IRC21:27
*** harlowja has joined #openstack-ansible21:28
openstackgerritKevin Carter proposed openstack/openstack-ansible: Fixes playbook runtime issues with ldap  https://review.openstack.org/24890021:28
cloudnullkysse:  the only way I could get the vif error was to either take a network away from one of my compute nodes21:29
kysseok21:29
cloudnullor to mess with the provider networks such that it created a broken physical map21:30
cloudnulllike having the flat network bound to the same interface21:30
cloudnulletc21:30
*** phiche has quit IRC21:30
*** sdake has quit IRC21:30
kysseyepyep21:31
*** mancdaz has quit IRC21:31
*** mancdaz has joined #openstack-ansible21:32
*** sdake has joined #openstack-ansible21:34
cloudnullso im once again rekicking my lab will try to reproduce the issue21:35
kysseheh21:36
mhaydenodyssey4me: i did a build of the ansible bootstrap stuff on rax with an instance that has extra storage -- it worked just fine21:36
mhaydenit picked up xvde21:36
*** sdake has quit IRC21:41
cloudnullkysse: its quite perplexing why you're env is not happy but im intent on trying to get to the bottom of i t21:50
kysseindeed21:50
kyssewe are doing hard debugging also.21:50
*** baker has joined #openstack-ansible22:06
*** Mudpuppy has quit IRC22:06
*** ysm has joined #openstack-ansible22:13
*** linggao__ has joined #openstack-ansible22:16
*** linggao_ has quit IRC22:16
*** sacharya has quit IRC22:18
*** harlowja has quit IRC22:19
*** harlowja has joined #openstack-ansible22:21
*** galstrom is now known as galstrom_zzz22:31
*** daneyon has joined #openstack-ansible22:37
*** mcarden_ has joined #openstack-ansible22:45
*** mcarden has quit IRC22:45
*** mcarden_ is now known as mcarden22:46
*** daneyon has quit IRC22:50
*** harlowja has quit IRC22:57
cloudnullkysse: still around ?22:58
kysseyes22:59
kysseI'm running it again now, so I cant test.23:00
cloudnullwhen your run is complete can you set "neutron_l2_population: true" in your user variables file and rerun the os-neutron-install.yml play23:00
kyssesure but whe deviced to run kilo first, then we can upgrade and test that.23:01
kyssewe decided*23:01
*** harlowja has joined #openstack-ansible23:01
cloudnullI'm attempting to recreate your exact env w/ the config details you've provided and in my tests so far that is one thing that has differed between my installs and yours23:02
kyssethis is good training for us too.23:02
kyssemmh ok23:02
cloudnulli have it set, you do not, because the playbook assumes the use of l3ha when more than one network node is present23:02
cloudnullhttps://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_neutron/tasks/neutron_post_install.yml#L16-L3423:02
kyssebtw, is it possible to set flat network point to a bridge?23:03
cloudnulll2 population enabled is the default in kilo23:03
kysseok23:03
cloudnullyes23:03
cloudnullyou can23:03
kyssecool.23:03
cloudnullhowever you should hang a veth off of the bridge23:03
cloudnulland set the host bind override to the name of the veth interface23:03
kysseoh I see.23:04
cloudnullwhich is what we do here23:04
cloudnullhttps://github.com/openstack/openstack-ansible/blob/master/etc/network/interfaces.d/aio_interfaces.cfg#L52-L5923:04
cloudnullthen vlan, flat, even vxlan could all go off of the same bridge if you wanted it to23:04
*** ysm has quit IRC23:06
cloudnullFYI here are my logs for the neutron lxb agent on the compute node: http://cdn.pasteraw.com/9bv8qknv72x4so83ih1wg0rdsha9ljs23:07
cloudnullim not seeing that stacktrace in the logs when i restart the service23:08
kyssehmh. yep yep23:10
kyssethanks23:10
cloudnullso at this moment im thinking that the root cause is that l2 pop needs to be enabled.23:10
cloudnullbut im testing more23:10
cloudnullso more soon23:10
cloudnullhe might not be around however mattt i think ran into something similar a while back23:13
cloudnullif i remember right23:14
*** daneyon has joined #openstack-ansible23:15
*** sdake has joined #openstack-ansible23:16
*** linggao__ has quit IRC23:16
*** linggao__ has joined #openstack-ansible23:16
*** linggao__ has quit IRC23:16
cloudnullkysse: w/ l2pop disabled http://cdn.pasteraw.com/dg3bmy773jz1n7lfxl1wgge3aml6xt23:18
kyssehah..23:19
kysseso you solved it, I think :-)23:19
cloudnullmaybe23:19
cloudnullre-enabling and will rerun the test23:19
cloudnullkysse: are you on launchpad ?23:20
kysseI think I'll stop installing kilo, and test that with liberty23:20
kysseno23:20
cloudnullok23:20
kysseI think I can register.23:20
*** KLevenstein has quit IRC23:22
kysseville.vuorinen@multim.fi23:22
cloudnullconfirmed http://cdn.pasteraw.com/nbz6csr3wf3xjh154b9p29uwwm28bob23:25
cloudnullthe 2 active tests are w/ l2pop enabled23:25
cloudnullthe 5 errors are w/ l2pop disabled23:25
kyssecool, cool cool cool.23:25
*** darrenc is now known as darrenc_afk23:31
cloudnullkysse https://bugs.launchpad.net/openstack-ansible/+bug/152179323:32
openstackLaunchpad bug 1521793 in openstack-ansible "Master/Liberty w/ L2pop disabled breaks neutron" [High,Triaged] - Assigned to Kevin Carter (kevin-carter)23:32
*** spotz is now known as spotz_zzz23:34
kyssethanks23:34
*** baker has quit IRC23:34
*** sdake has quit IRC23:36
openstackgerritKevin Carter proposed openstack/openstack-ansible: Fix neutron issue w/ l2pop  https://review.openstack.org/25210023:37
cloudnull^ fix for the issue23:37
*** networkninja has joined #openstack-ansible23:38
cloudnullkysse: in your env when you reconfigure neutron you'll need to restart the neutron lxb service on your compute node and likely restart the neutron agent containers after the play has been completed.23:39
*** sdake has joined #openstack-ansible23:39
kysseok23:39
cloudnullim off for a bit , but let me know how it goes23:41
*** dstanek has quit IRC23:41
*** dstanek has joined #openstack-ansible23:41
*** daneyon has quit IRC23:42
*** daneyon has joined #openstack-ansible23:47
*** daneyon has quit IRC23:48
*** darrenc_afk is now known as darrenc23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!