Thursday, 2015-10-29

*** markvoelker has quit IRC00:00
*** openstackgerrit has quit IRC00:01
*** openstackgerrit has joined #openstack-ansible00:02
*** daneyon_ has quit IRC00:05
*** harlowja has joined #openstack-ansible00:08
*** gouthamr has joined #openstack-ansible00:09
*** markvoelker has joined #openstack-ansible00:11
*** markvoelker has quit IRC00:11
*** markvoelker has joined #openstack-ansible00:13
*** markvoelker has quit IRC00:13
*** woodard has quit IRC00:18
*** spotz_zzz is now known as spotz00:24
*** harlowja has quit IRC00:47
*** alejandrito has joined #openstack-ansible00:55
*** rebase_ has joined #openstack-ansible00:58
*** abitha has quit IRC01:02
*** sdake has joined #openstack-ansible01:28
*** rebase_ has quit IRC01:32
*** spotz is now known as spotz_zzz01:35
*** markvoelker has joined #openstack-ansible01:35
*** alejandrito has quit IRC01:36
*** harlowja has joined #openstack-ansible01:40
*** sdake has quit IRC01:46
*** sdake has joined #openstack-ansible01:56
*** elo has joined #openstack-ansible01:57
*** markvoelker has quit IRC01:57
*** markvoelker has joined #openstack-ansible01:58
*** markvoelker has quit IRC01:58
*** markvoelker has joined #openstack-ansible02:01
*** markvoelker has quit IRC02:01
*** harlowja has quit IRC02:02
*** elo has quit IRC02:05
*** markvoelker has joined #openstack-ansible02:06
*** harlowja has joined #openstack-ansible02:09
*** markvoelker has quit IRC02:11
*** elo has joined #openstack-ansible02:13
*** timrc has quit IRC02:18
*** timrc has joined #openstack-ansible02:19
*** fawadkhaliq has joined #openstack-ansible02:19
*** sdake has quit IRC02:28
*** spotz_zzz is now known as spotz02:31
*** alejandrito has joined #openstack-ansible02:33
*** sdake has joined #openstack-ansible02:33
*** fawadkhaliq has quit IRC02:40
*** fawadkhaliq has joined #openstack-ansible02:41
*** fawadkhaliq has quit IRC02:43
*** harlowja has quit IRC02:44
*** harlowja has joined #openstack-ansible02:49
*** markvoelker has joined #openstack-ansible02:50
*** harlowja has quit IRC02:51
*** fawadkhaliq has joined #openstack-ansible02:52
*** sawangpongm has joined #openstack-ansible03:04
*** spotz is now known as spotz_zzz03:07
*** gouthamr_ has joined #openstack-ansible03:08
*** gouthamr has quit IRC03:11
*** harlowja has joined #openstack-ansible03:11
*** alejandrito has quit IRC03:14
*** sawangpongm has left #openstack-ansible03:18
*** fawadkhaliq has quit IRC03:19
*** markvoelker has quit IRC03:29
*** harlowja has quit IRC03:32
*** tlian has quit IRC03:33
*** sdake has quit IRC03:36
*** sdake has joined #openstack-ansible03:42
*** elo has quit IRC03:44
*** subscope has joined #openstack-ansible03:57
*** abitha has joined #openstack-ansible04:03
*** mgoddard_ has joined #openstack-ansible04:05
*** elo has joined #openstack-ansible04:08
*** mgoddard has quit IRC04:09
*** g3rms_ has left #openstack-ansible04:09
*** daneyon has joined #openstack-ansible04:15
*** markvoelker has joined #openstack-ansible04:16
*** sdake has quit IRC04:21
*** gouthamr_ has quit IRC04:22
*** alejandrito has joined #openstack-ansible04:23
*** sdake has joined #openstack-ansible04:24
*** daneyon has quit IRC04:28
*** daneyon has joined #openstack-ansible04:32
*** sawangpongm has joined #openstack-ansible04:32
*** abitha has quit IRC04:34
*** abitha has joined #openstack-ansible04:40
*** alejandrito has quit IRC04:41
*** sawangpongm has quit IRC04:43
*** sdake has quit IRC04:44
*** sawangpongm has joined #openstack-ansible04:44
*** Mudpuppy has joined #openstack-ansible04:45
*** sdake has joined #openstack-ansible04:45
*** harlowja has joined #openstack-ansible04:48
*** markvoelker has quit IRC04:49
*** Mudpuppy has quit IRC04:49
*** javeriak has joined #openstack-ansible04:54
*** sdake has quit IRC04:54
*** sdake has joined #openstack-ansible04:55
*** elo has quit IRC05:01
*** sdake has quit IRC05:04
*** sawangpongm has quit IRC05:04
*** sdake has joined #openstack-ansible05:05
*** alejandrito has joined #openstack-ansible05:13
*** daneyon has quit IRC05:16
*** daneyon has joined #openstack-ansible05:16
*** harlowja has quit IRC05:17
*** elo has joined #openstack-ansible05:17
*** elo has quit IRC05:21
*** sdake has quit IRC05:21
*** javeriak_ has joined #openstack-ansible05:21
*** sdake has joined #openstack-ansible05:22
*** shausy has joined #openstack-ansible05:22
*** javeriak has quit IRC05:23
*** elo has joined #openstack-ansible05:29
*** sdake has quit IRC05:33
*** sawangpongm has joined #openstack-ansible05:38
*** tnarg has joined #openstack-ansible05:40
*** sawangpongm has quit IRC05:42
*** sawangpongm has joined #openstack-ansible05:46
*** abitha has quit IRC05:53
*** sdake has joined #openstack-ansible05:59
*** sdake has quit IRC06:04
*** sdake has joined #openstack-ansible06:06
*** spotz_zzz is now known as spotz06:09
*** sdake has quit IRC06:12
*** sawangpongm has quit IRC06:13
*** rebase has joined #openstack-ansible06:15
*** sawangpongm has joined #openstack-ansible06:17
*** sdake has joined #openstack-ansible06:19
*** tnarg has quit IRC06:19
*** sdake has quit IRC06:20
*** tnarg has joined #openstack-ansible06:26
*** subscope has quit IRC06:29
*** sawangpongm has quit IRC06:30
*** subscope has joined #openstack-ansible06:32
*** markvoelker has joined #openstack-ansible06:34
*** sdake has joined #openstack-ansible06:36
*** kawaja has quit IRC06:39
*** subscope has quit IRC06:42
*** sdake has quit IRC06:43
*** sdake has joined #openstack-ansible06:49
odyssey4meoh nice, openstack has developed an automated release notes generator: http://docs.openstack.org/developer/reno/06:49
*** sawangpongm has joined #openstack-ansible06:50
*** sdake_ has joined #openstack-ansible06:51
*** markvoelker has quit IRC06:51
*** sdake has quit IRC06:54
*** tnarg has quit IRC06:58
*** sdake_ has quit IRC06:58
*** timrc has quit IRC07:03
*** timrc has joined #openstack-ansible07:03
*** sawangpongm has quit IRC07:11
*** rebase has quit IRC07:12
*** jcannava has quit IRC07:13
*** hughsaunders has quit IRC07:15
*** rebase has joined #openstack-ansible07:19
*** jcannava has joined #openstack-ansible07:19
*** spotz is now known as spotz_zzz07:20
*** elo has quit IRC07:20
*** sawangpongm has joined #openstack-ansible07:21
*** fawadkhaliq has joined #openstack-ansible07:23
*** rebase has quit IRC07:26
*** mpavone has joined #openstack-ansible07:29
*** javeriak has joined #openstack-ansible07:30
*** javeriak_ has quit IRC07:33
*** spotz_zzz is now known as spotz07:36
*** coolj has quit IRC07:46
*** javeriak_ has joined #openstack-ansible07:48
*** javeria__ has joined #openstack-ansible07:50
*** javeriak has quit IRC07:51
*** javeriak_ has quit IRC07:53
*** GheRivero has joined #openstack-ansible08:10
*** karimb has joined #openstack-ansible08:12
*** spotz is now known as spotz_zzz08:12
*** karimb has quit IRC08:13
*** karimb has joined #openstack-ansible08:14
*** karimb has quit IRC08:15
*** karimb has joined #openstack-ansible08:15
*** persia has quit IRC08:24
*** daneyon has quit IRC08:25
*** persia has joined #openstack-ansible08:27
*** karimb has quit IRC08:28
*** spotz_zzz is now known as spotz08:29
*** tnarg has joined #openstack-ansible08:33
*** tnarg has quit IRC08:38
*** javeriak has joined #openstack-ansible08:47
*** karimb has joined #openstack-ansible08:48
*** gparaskevas has joined #openstack-ansible08:49
*** javeria__ has quit IRC08:50
*** fawadkhaliq has quit IRC08:52
*** javeriak_ has joined #openstack-ansible08:52
*** javeriak has quit IRC08:56
*** spotz is now known as spotz_zzz08:57
*** ashishb has joined #openstack-ansible08:58
*** alejandrito has quit IRC09:03
*** sawangpongm has quit IRC09:08
*** hughsaunders has joined #openstack-ansible09:14
*** javeriak has joined #openstack-ansible09:34
*** tnarg has joined #openstack-ansible09:35
*** javeriak_ has quit IRC09:37
*** tnarg has quit IRC09:40
*** alejandrito has joined #openstack-ansible09:40
*** alejandrito has quit IRC09:45
*** javeriak_ has joined #openstack-ansible09:47
*** javeriak has quit IRC09:47
*** javeriak has joined #openstack-ansible09:52
*** javeriak_ has quit IRC09:52
*** ashishb has quit IRC10:07
*** ashishb has joined #openstack-ansible10:08
*** javeriak has quit IRC10:16
*** gparaskevas has quit IRC10:28
*** tnarg has joined #openstack-ansible10:36
*** javeriak has joined #openstack-ansible10:37
*** tnarg has quit IRC10:40
*** javeriak_ has joined #openstack-ansible10:41
*** javeriak has quit IRC10:42
*** Mudpuppy has joined #openstack-ansible10:47
*** ashishb has quit IRC10:49
*** tnarg has joined #openstack-ansible10:51
*** Mudpuppy has quit IRC10:51
*** sawangpongm has joined #openstack-ansible11:15
*** javeriak_ has quit IRC11:28
*** javeriak has joined #openstack-ansible11:31
*** Mudpuppy has joined #openstack-ansible11:31
*** sawangpongm has left #openstack-ansible11:31
*** sawangpongm has joined #openstack-ansible11:31
*** sawangpongm has left #openstack-ansible11:32
*** slotti has joined #openstack-ansible12:06
*** javeriak has quit IRC12:13
*** slotti has quit IRC12:16
mhaydenmorning12:21
mhaydenodyssey4me: that's nifty12:22
mhaydencould any available core reviewers give me a tiny nudge on 3 reviews? https://review.openstack.org/#/q/status:open+project:openstack/openstack-ansible-security,n,z12:22
mhaydeni can provide candy12:22
mhayden:)12:23
*** ashishb has joined #openstack-ansible12:31
*** shausy has quit IRC12:32
*** neilus has joined #openstack-ansible12:36
*** woodard has joined #openstack-ansible12:38
*** woodard has quit IRC12:38
*** woodard has joined #openstack-ansible12:39
tiagogomesI don't understand how to configure cinder to use ceph as backend. Shoudn't the cinder configuration allow you to set rbd_secret_uuid somewhere12:50
*** slotti has joined #openstack-ansible12:50
*** tlian has joined #openstack-ansible12:56
*** javeriak has joined #openstack-ansible12:57
*** karimb has quit IRC12:57
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952513:03
mhaydendoes openstack's jenkins support color output?13:05
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952513:06
mhaydenmattt: added some debug stuff to test ssh connectivity13:07
mhaydenlooks like ssh connectivity is okay https://jenkins02.openstack.org/job/gate-openstack-ansible-commit-nv/219/console13:09
mhaydenmattt: what's the difference between the dvsm and nv jobs?13:09
mhaydenlooks like i might have an ssh key issue13:10
matttmhayden: the nv one is non-voting, can't remember what is different about it -- may be the base image used?13:10
mhaydenbut it only affects the dvsm job13:10
mhaydenit offered keys but they weren't accepted13:11
mhaydenperhaps i missed a permitrootlogin13:12
mhaydenOH SON OF A13:12
mhaydeni didn't bounce ssh13:12
mhayden:|13:12
* mhayden goes back in the corner13:12
matttbra srsly13:12
mhaydenrazzafrazzakhsdfdsfkjshgkhfds13:12
mhayden:)13:12
matttmhayden: btw, you can inject your key into instances if you ever need to test stuff13:13
matttmhayden: we've had to do that a few times to debug some tricky situations13:13
mhaydenoh wait, i am restarting ssh but i don't think i'm adding the key to authorized keys perhaps13:14
*** javeriak_ has joined #openstack-ansible13:14
mhaydenhmm so ssh is getting restarted, and the key is being placed into authorized_keys13:15
mhaydenmattt: do the gate jobs run as the root user by default?13:15
matttyeah root13:16
mhaydenweird13:16
mhaydeni wonder if something strange is in the sshd_config in the gate job13:16
matttmhayden: where is it being placed in authorized_keys ?13:17
mhaydennot sure13:17
mhaydenthe ordering shouldn't matter, right?13:17
matttno, but is it actually happening?13:18
*** javeriak has quit IRC13:18
*** Mudpuppy has quit IRC13:18
matttmhayden: not seeing it browsing the review, but perhaps i'm overlooking it13:18
mhaydenyes, the ansible task places it13:18
mhaydenwhen it generates it13:18
mhaydenit's automatically added to authorized_hosts13:19
mhaydenerr _keys13:19
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952513:19
mhaydenadded in debug lines to display sshd_config and some syslog13:19
matttthat doesn't sound right13:20
matttso you create a key and it automatically adds it to your host's authorized_keys?13:20
mhaydenwell this run should show if the key is missing13:20
mhaydenhttps://jenkins06.openstack.org/job/gate-openstack-ansible-commit-nv/169/console13:20
mattthttp://docs.ansible.com/ansible/authorized_key_module.html13:21
matttmaybe just drop it explicitly13:21
mhaydeni'm not using that module13:21
mhaydenit's weird that the nv job works every time13:21
matttyeah super odd13:22
mhaydenwell i forced both to fail this go around13:22
matttpackage not getting installed?13:22
mhaydenbecause i wanted some logging13:22
mhayden2015-10-29 13:23:24.672 | Oct 29 13:23:24 devstack-trusty-rax-iad-5620616 sshd[9282]: ROOT LOGIN REFUSED FROM 172.29.236.100 [preauth]13:23
matttso13:23
mhaydenPermitRootLogin no13:23
mhaydenWTF13:23
mhaydeni adjusted that in the playbook13:23
mhaydenoh the regex didn't match13:24
matttalso, id_rsa.pub is some image key13:24
mhaydengot tacked onto the end of the key13:24
matttinjected in the image13:24
matttis that what you want?13:24
mhaydenend of the file rather13:24
*** skamithi13 has joined #openstack-ansible13:24
mhaydenyeah, so permitrootlogin is my problem13:25
matttmhayden: https://github.com/openstack/openstack-ansible/blob/master/scripts/scripts-library.sh#L135_L13913:25
matttsee13:26
matttbefore if both priv and pub keys didn't exist we blew them away and recreated13:26
mattti don't think your custom key is even getting created13:26
matttbecuaes one exists it's skipping it13:26
mhaydenokay, i'll look at that next13:26
*** Mudpuppy has joined #openstack-ansible13:27
*** Mudpuppy has quit IRC13:27
*** gouthamr has joined #openstack-ansible13:30
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952513:33
*** metral_zzz has joined #openstack-ansible13:33
*** metral_zzz is now known as metral13:33
*** erikmwilson has quit IRC13:36
*** woodard has quit IRC13:38
*** woodard has joined #openstack-ansible13:43
*** javeriak_ has quit IRC13:45
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952513:46
*** CheKoLyN has joined #openstack-ansible13:48
openstackgerritMatt Riedemann proposed openstack/openstack-ansible: Cleanup the os_neutron role readme  https://review.openstack.org/24023513:52
mhaydenwow, i'm totally baffled as to why my permitrootlogin isn't working13:52
*** ysm has joined #openstack-ansible13:53
mhaydenmattt: imma put in a fat sleep and add my key :|13:54
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952513:58
*** gouthamr has quit IRC14:00
*** gouthamr has joined #openstack-ansible14:00
*** k_stev has joined #openstack-ansible14:01
*** harlowja has joined #openstack-ansible14:05
matttmhayden: get it ?14:06
mhaydenstill wrestling14:06
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952514:17
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952514:18
*** karimb has joined #openstack-ansible14:21
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952514:22
*** karimb has quit IRC14:22
*** karimb has joined #openstack-ansible14:23
*** Mudpuppy has joined #openstack-ansible14:23
*** Mudpuppy has quit IRC14:24
*** Mudpuppy has joined #openstack-ansible14:24
*** sigmavirus24_awa is now known as sigmavirus2414:25
mhaydenmattt: got the ssh key thing sorted out14:26
mhaydenneeded to 1) remove existing keys 2) and toss the pub key into authorized_keys14:26
matttisn't that what i said :(14:29
matttbut glad it's sorted!14:29
matttso is that role done now?14:29
*** mgoddard__ has joined #openstack-ansible14:30
mhaydengate check is running . . . . . .14:31
mhaydenit blew up on container creation :/14:31
mhaydenit makes a bunch of containers and then fails on the last 5-614:31
mhaydenperhaps running out of ram?14:31
mhaydenbut the lxc method doesn't put anything in stderr14:32
*** mgoddard_ has quit IRC14:33
*** ysm has quit IRC14:34
mhaydenbut the containers start in the nv, not the dvsm14:34
Mudpuppymahyden check ram and lxc volume size free available space14:36
mhaydenlooks like swap memory isn't getting provisioned correctly14:37
*** jimchou has joined #openstack-ansible14:38
mhaydenthe tasks are running but the swap isn't actually online14:39
mhaydenthat's peculiar14:39
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952514:44
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952514:48
*** mgoddard has joined #openstack-ansible14:53
*** mgoddard__ has quit IRC14:54
mhaydenmattt: how do i access the dvsm VM?14:56
mhaydenit's on 10.x address14:56
mhaydenoh nvm14:56
* mhayden is watching RAM usage like a hawk14:57
matttmhayden: icanhazip :P14:57
mhaydeni heard that site is just for hackers14:58
*** ysm has joined #openstack-ansible14:58
*** harlowja has quit IRC14:58
mhaydenokay, so the nv build is doing repo builds already14:58
* mhayden crosses his fingers14:58
*** hyperfocus has joined #openstack-ansible14:59
*** is_null has joined #openstack-ansible15:00
hyperfocusIs this mature enough to use? https://github.com/openstack/openstack-ansible-security15:00
hyperfocusNot that I have to redeploy in a month time because of new commits15:00
mhaydenhyperfocus: currently in alpha-ish state15:02
mhaydenabout three reviews left, then it will need some thorough gate testing15:02
mhaydenbut if you find bugs/issues, please let me know so i can get them fixed up15:02
mattthyperfocus: to use on a production install?15:02
mattti'd say no :x15:02
mhaydenright, but it's not terribly far off ;)15:03
*** tnarg has quit IRC15:03
mhaydenthe big things that remain are 1) some code cleanup, 2) auditing to ensure we got all the stigs done, 3) auditing to ensure we didn't break any openstack-isms15:03
*** tnarg has joined #openstack-ansible15:03
mhaydenmattt: <315:05
hyperfocusThank you for your work guys! ;)15:10
hyperfocusAnd yes I'll try it out and report bugs once I find them15:10
hyperfocusIs it really openstack specific or also for general purpose hardening?15:12
openstackgerritMajor Hayden proposed openstack/openstack-ansible-security: V-3857{4,6,7}: Password hashing algorithms  https://review.openstack.org/23307115:13
*** jwagner_away is now known as jwagner15:15
openstackgerritMajor Hayden proposed openstack/openstack-ansible-security: V-3857{4,6,7}: Password hashing algorithms  https://review.openstack.org/23307115:18
mhaydenhah, the nv check died due to ubuntu's keyserver being unrespnsive15:21
* mhayden can't win15:21
*** neilus has quit IRC15:22
sigmavirus24mhayden: you're welcome15:30
*** richoid has quit IRC15:33
Mudpuppyubuntu is the future15:39
*** woodard has quit IRC15:40
*** javeriak has joined #openstack-ansible15:42
*** sdake has joined #openstack-ansible15:42
*** richoid has joined #openstack-ansible15:43
*** sdake_ has joined #openstack-ansible15:43
*** alop has joined #openstack-ansible15:46
*** sdake has quit IRC15:47
*** javeriak has quit IRC15:47
*** ashishb has quit IRC15:48
mhaydenis there a reason why the debug information is stripped from gate check runs?15:54
mhaydenit would be helpful to see it as the job runs in jenkins15:54
*** Bjoern_ has joined #openstack-ansible15:56
*** sdake_ has quit IRC15:57
mhaydeni guess it could be a bit cluttered15:59
sigmavirus24mhayden: the ansible logs are kept separate16:00
sigmavirus24the debug information is there but the gate was stripped otu to make it easier to quickly spot the problem16:00
mhaydenyeah, i've seen that part16:00
*** mgoddard has quit IRC16:00
mhaydeni guess it makes sense16:00
*** mgoddard has joined #openstack-ansible16:00
mhaydeni think my brain is fried from trying to get these tests to pass, please ignore me :P16:00
mhaydenhughsaunders: i think i have corrected all of the uglygreps -> https://review.openstack.org/#/c/233071/16:03
*** woodard has joined #openstack-ansible16:03
hughsaundersmhayden: :)16:03
*** neilus has joined #openstack-ansible16:04
*** hyperfocus has left #openstack-ansible16:10
tiagogomeshi, I am stuck trying to understand the task is being skipped here: http://paste.openstack.org/show/477657/16:13
*** mpavone has quit IRC16:13
mattttiagogomes: which playbook are you running?16:15
tiagogomesos-nova-install with the ceph-client tag16:15
Bjoern_you need to add the tag ceph-auth-nova-libvirt-secret16:17
Bjoern_this task does not cary ceph-client tags and the host needs to be inside the nova_compute group16:17
tiagogomesI tried that tag as well16:17
tiagogomesyes, but is skipping for my compute hosts as well (devhw7 and devhw8)16:18
mattttiagogomes: do you have nova_libvirt_images_rbd_pool set ?16:18
mattt99% sure that stuff is skipped if that var isn't set16:18
tiagogomesmattt I have not, but I have a cinder_backend set to ceph, which I think it was enough16:18
mattttiagogomes: https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/ceph_client/tasks/main.yml#L24-L2616:19
Bjoern_mattt: don't see this dependency on this task16:19
*** Bjoern_ is now known as BjoernT16:19
tiagogomesmattt I have a cinder ceph backend sio cinder_backends_rbd_inuse will evaluate to true16:20
BjoernTHe's already inside the ceph_client role so it obviously runs ceph_auth.yml16:20
tiagogomesright, ceph_auth.yml is being included, but that tasks skipped16:21
BjoernTdo a scripts/inventory_manage.py -f /etc/openstack_deploy/openstack_invenory.json -l |grep devhw8 please16:21
BjoernTscripts/inventory-manage.py -f /etc/openstack_deploy/openstack_inventory.json -l |grep devhw816:23
BjoernTwithout typos now16:23
tiagogomesBjoernT,  http://paste.openstack.org/show/477659/16:24
BjoernTyepp devhw8 is not a compute node16:24
BjoernTat least according the inventory16:25
*** neilus has quit IRC16:25
matttBjoernT: http://logs.openstack.org/31/239631/1/check/gate-openstack-ansible-dsvm-commit/4df5c05/console.html#_2015-10-27_16_51_15_21016:25
tiagogomeswell, it is, it is runnign nova-compute and I launched some VMs16:25
BjoernTdid you use the exact name inside the /etc/openstack_deploy/openstack_user_config.yml under compute_hosts like compute_hosts: devhw8: ip:xxxx16:26
matttBjoernT: ansible still goes into ceph-auth even if all the tasks are skipped16:26
BjoernTso it'a a all in one build ? the groups it is in seems to be infra nodes16:27
BjoernTmattt: yepp as I just said16:27
tiagogomescompute_hosts:16:27
tiagogomes  devhw7:16:27
tiagogomes    ip: 10.24.1.6716:27
tiagogomes  devhw8:16:27
tiagogomes    ip: 10.24.1.8516:27
BjoernTThat could be an inventory-manager bug16:27
tiagogomesis there a way to fix this without redeploying?16:27
tiagogomes:q16:28
BjoernTcan you pastebin  the whole user_config.yml ?16:28
BjoernTstrip IPs before pasting16:28
BjoernTat least any public information16:28
tiagogomesthey are private16:28
BjoernTok16:29
mattttiagogomes: did cinder deploy w/ ceph support ?16:29
tiagogomesBjoernT, http://paste.openstack.org/show/477660/16:29
mattttiagogomes: ie. cinder type-list should return something ceph-related16:29
tiagogomesok what I did was this. I deployed the cloud with success (launching VMs, etc). Then I thought it would be nice to add ceph support16:29
tiagogomesSo I added two new hosts to openstack_user_config and ran the setup-hosts playbook again16:30
BjoernTthats on kilo ?16:30
tiagogomesyep kilo16:30
tiagogomesAfter setup-hosts, I run ceph-client for cinder, which setup everything correctly (I can create volumes on the ceph storage)16:31
tiagogomesI tried to the same for Nova (required to attach volumes to the VMs) but it is skipping the ceph-auth tasks16:31
mattti don't know why, but you could try setting nova_libvirt_images_rbd_pool: vms16:32
matttbut that isn't actually required, more interesting to see if it works or not16:32
* tiagogomes obeys16:32
BjoernTtiagogomes: http://paste.openstack.org/show/5E31NBEcXamNWxUfaSR6/ my generated inventory seems to be fine but I did not add ceph yet16:33
tiagogomeswtf!16:34
BjoernTI bet in when you look inside the inventory backup /etc/openstack_deploy/backup_openstack_inventory.tar you see the it was inside the nova_compute group16:34
BjoernTyour current nova_compute group seems to be empty now :     "nova_compute": {16:35
BjoernT        "children": [],16:35
BjoernT        "hosts": [ ]16:35
BjoernT    },16:35
tiagogomesmattt BjoernT adding `nova_libvirt_images_rbd_pool: vms`! ( at least the tasks were not skipped)16:37
BjoernTdevhw7 and devhw8 are also listed as share-infra_hosts nor sure16:37
BjoernTif you had to do it to get the osd_hosts group working16:38
* tiagogomes doesn't understand adding that variable worked16:39
BjoernTdid you inventory change ?16:39
BjoernTyour16:39
tiagogomesBjoernT I don't think so16:40
BjoernT../scripts/inventory-manage.py  -l -f  /etc/openstack_deploy/openstack_inventory.json | grep devhw[7-8]16:40
BjoernTthat would make no sense16:40
matttif adding that vairable worked then you have compute nodes in your inventory16:40
BjoernTthat adding nova_libvirt_images_rbd_pool would fix this particular task16:40
*** neilus has joined #openstack-ansible16:41
matttso only reason it may not have worked is that your cinder backend was not setup correctly16:41
tiagogomesBjoernT, I know that it does not make sense http://paste.openstack.org/show/477664/16:41
BjoernTmattt: we talk only about https://github.com/openstack/openstack-ansible/blob/kilo/playbooks/roles/ceph_client/tasks/ceph_auth.yml#L118-L12616:42
tiagogomesmattt if it wasn't ceph-auth wouldn't be included no?16:42
BjoernTnot ceph in general16:42
*** woodard has quit IRC16:42
BjoernTtiagogames: grep 'nova_compute":' -A5  /etc/openstack_deploy/openstack_inventory.json16:43
BjoernTI'm curious what your inventory looks like16:44
tiagogomesI removed that variable and it is skipping again!16:44
tiagogomes    "nova_compute": {16:44
tiagogomes        "children": [],16:44
tiagogomes        "hosts": [16:44
tiagogomes            "devhw8",16:44
tiagogomes            "devhw7"16:44
tiagogomes        ]16:44
BjoernToh so it is just a display problem of the inventory-manage.py script16:45
tiagogomeslooks like so16:45
tiagogomeshold on a sec, setting nova_libvirt_images_rbd_pool as the side-effect of storing the ephemeral disks on ceph as well no16:48
matttyessir16:49
tiagogomesnp, I was planning to do that anyway16:49
BjoernTHmm, I personally would never store ephemeral disks on replicated storage like ceph16:50
tiagogomesI want that, to live migrate VMs16:50
*** woodard has joined #openstack-ansible16:50
tiagogomesCurrently I set /var/lib/nova as NFS mountpoint, which is meh16:50
BjoernTbut now it seems you don't even have the choice if you want to store the root disk inside ceph16:51
BjoernTnova can live migrate instances with ephermeral storage but it would take longer16:51
BjoernTin the end you kinda defeat the purpose of ephemeral storage where expect fast IO and not traversing network. Just image you put swap  on ephemeral storage16:52
tiagogomesthat's true16:53
BjoernTnow you ceph vms group is nicely writing any swap IO onto multiple osds because you replicate16:53
BjoernTthat's how I killed my data center on netapp16:53
BjoernTeverything was nicely swapping on netapp and then you even snapshot this stuff , OMG16:53
mhaydenthanks, hughsaunders ;)16:54
*** ashishb has joined #openstack-ansible16:54
matttBjoernT: yeah not necessarily sure that is the way forward, i just wanted to ensure doing that would cause those tasks to run16:56
tiagogomesBjoernT how long does it take to live migrate without distributed storage?16:57
tiagogomesAlso, looking at the code I have no idea why nova_libvirt_images_rbd_pool is needed16:57
tiagogomesok, I figure it out. `cinder_backends_rbd_inuse` should be `cinder_backend_rbd_inuse` in ceph_all.yml16:59
mattttiagogomes: ah weird17:00
hughsaundersmhayden: so does rvar have a value or not? https://gist.github.com/hughsaunders/93a067fab8ffc8dda49e17:04
*** jwagner is now known as jwagner_away17:09
*** woodard has quit IRC17:11
mattttiagogomes: not sure that is true17:13
mattttiagogomes: https://github.com/openstack/openstack-ansible/blob/master/playbooks/inventory/group_vars/hosts.yml#L196-L20717:13
matttcinder_backend_rbd_inuse would be wrong in this context17:13
mhaydenhughsaunders: that's unusual17:14
hughsaundersmhayden: I think when + register is a potentially hazardous17:15
hughsaunderss/a//17:15
*** neilus has quit IRC17:16
*** greg_a has joined #openstack-ansible17:17
tiagogomesmattt right, that didn't fix the problem I am list17:18
tiagogomess/list/lost17:18
hughsaundersmhayden: the apparent inconsistency in that gist is due to over indentation of the when clause on the first debug - my mistake not ansible's17:22
*** greg_a has quit IRC17:23
hughsaundershowever it still stands that when + register could cause problems17:23
*** abitha has joined #openstack-ansible17:24
mhaydenindeed17:26
mhaydenweird, when i try the ansible aio playbook thing, i end up with a repo container that isn't on mgmt net17:28
mhaydenit's like that interface is missing17:28
tiagogomesmattt interesting both cinder_backend_rbd_inuse are inder_backends_rbd_inuse are set to false17:31
mattttiagogomes: well there's yoru problem17:33
tiagogomesso auto-detection is not working17:34
mattttiagogomes: can you paste a 'cinder type-list' for me ?17:34
tiagogomes+--------------------------------------+------+17:35
tiagogomes|                  ID                  | Name |17:35
tiagogomes+--------------------------------------+------+17:35
tiagogomes| 4926b6be-4c70-45b6-9c8b-06cba187cfc3 | ceph |17:35
tiagogomes+--------------------------------------+------+17:35
mattttiagogomes: can you also paste your conf.d/ceph.yml ?17:35
matttor where-ever you've configured the ceph cinder backend17:37
*** neilus has joined #openstack-ansible17:37
tiagogomesmattt http://paste.openstack.org/show/477665/17:37
matttpaste.openstack.org is so slow :(17:37
*** javeriak_ has joined #openstack-ansible17:37
*** jwagner_away is now known as jwagner17:38
mattttiagogomes: don't spot any issues with that, hmmmph17:39
mattttiagogomes: you going to be around tomorrow?  i have to head out shortly17:39
tiagogomesmattt me too :)17:40
tiagogomesI'll be here tomorrow yes17:40
hughsaundersmhayden: approved it anyway, decided  your current version is probably better than removing the when and using failed_when: false.17:40
mattttiagogomes: cool let's have another crack tomorrow17:40
tiagogomesmattt ok :) see you tomorrow17:41
openstackgerritMerged openstack/openstack-ansible-security: V-38660: SNMPv3  https://review.openstack.org/23322617:41
mhaydenhughsaunders: thanks -- now if we can convince mattt to push that review over the edge ;)17:42
*** woodard has joined #openstack-ansible17:47
mhaydenoh, i forgot to flush iptables17:48
mhaydenaaaaaaah17:48
* mhayden applies head to desk17:49
*** woodard has quit IRC17:52
*** ashishb has quit IRC17:52
*** javeriak_ has quit IRC17:58
*** tiagogomes has quit IRC17:58
*** javeriak has joined #openstack-ansible17:59
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952518:07
*** javeriak_ has joined #openstack-ansible18:18
Sam-I-Amnot sure if this is supposed to happen, but get pip in the aio more or less catted the output of pip to my screen18:19
*** ashishb has joined #openstack-ansible18:19
*** javeriak has quit IRC18:22
*** javeriak has joined #openstack-ansible18:22
*** javeriak_ has quit IRC18:25
mhaydenSam-I-Am: i saw that too18:32
mhaydenin the shell script, right?18:32
*** karimb has quit IRC18:34
Sam-I-Amyeah18:36
*** woodard has joined #openstack-ansible18:38
*** neilus has quit IRC18:42
*** greg_a has joined #openstack-ansible18:51
*** greg_a has quit IRC18:55
*** neilus has joined #openstack-ansible19:01
*** neilus has quit IRC19:01
*** woodard has quit IRC19:10
*** skamithi14 has joined #openstack-ansible19:13
*** skamithi13 has quit IRC19:15
mhaydenomg stop the presses -- tests are passing for me19:21
sigmavirus24mhayden: *stops Jenkins*19:22
sigmavirus24Is that what you meant?19:22
mhaydenlawl19:22
Sam-I-Ammhayden: what did you fix?19:23
mhaydenoh nvm, the environment variable didn't exported and my code wasn't used19:25
mhaydenwonderful19:25
sigmavirus24mhayden: hah19:26
mhaydeni may swear off computers today19:26
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952519:30
mhaydenat least the testing infra isn't terribly busy today19:31
*** woodard has joined #openstack-ansible19:31
sigmavirus24mhayden: this is the small reward you get for working during the summit19:33
mhaydeni'm also learning that HP doesn't give much disk space in the root volume19:39
mhaydensigh19:39
mhaydenthat explains some other failed tests19:39
*** woodard has quit IRC19:53
*** skamithi14 has quit IRC19:55
*** skamithi13 has joined #openstack-ansible19:55
*** ashishb has quit IRC19:56
*** woodard has joined #openstack-ansible19:58
*** karimb has joined #openstack-ansible20:13
*** ysm has quit IRC20:20
*** ysm has joined #openstack-ansible20:24
*** rady has joined #openstack-ansible20:35
*** CheKoLyN has quit IRC20:54
*** javeriak has quit IRC20:58
mhaydenso i see volume groups being created here: https://github.com/openstack/openstack-ansible/blob/master/scripts/scripts-library.sh#L9321:00
mhaydenbut what makes the logical volume on that lxc volume group?21:00
mhaydeni can't find that anywhere in the code21:01
Sam-I-Amcinder itself makes lvs21:02
Sam-I-Amerr, lxc?21:02
mhaydengot that part, but the LXC one is confusing me21:02
Sam-I-Ami would assume lxc does21:02
mhaydenHP's disk is ~ 30GB but they offer that big ol' 470GB disk, too21:02
mhaydenbut i can't find anything that makes the logical volume for lxc21:03
Sam-I-Ami was sort of under the impression that we only use lvm for lxc if the vg already exists21:03
*** woodard has quit IRC21:03
* mhayden is digging in LXC roles21:03
mhaydenah, i found it21:04
mhaydenin container_create.yml21:04
mhaydenthere's a check to see if the vg is present21:04
mhaydenotherwise it uses the directory instead21:04
Sam-I-Amyarp21:04
mhaydenokay, that answers that21:04
*** coolj has joined #openstack-ansible21:06
mhaydenit's been one of those "let's learn how the sausage is made" kinda says21:07
mhaydens/says/days/21:07
Sam-I-Amisnt that every day?21:07
Sam-I-Amsausage causes cancer or something21:07
mhaydenmattt: <321:08
* mattt fistbumps mhayden21:08
mhaydenthere's only one left! :)21:08
mhaydenthe last security hardening review -> https://review.openstack.org/#/c/233284/21:08
mhayden(until i find tons of bugs)21:08
matttreviewed that one already :(21:09
openstackgerritMerged openstack/openstack-ansible-security: V-3857{4,6,7}: Password hashing algorithms  https://review.openstack.org/23307121:11
*** gouthamr has quit IRC21:12
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952521:13
* mhayden woots21:13
matttho21:13
mattt*how's that going mhayden ?21:13
mhaydenhad to go back and deal with that disk space mess after my build blew up at hp21:14
mhaydenand Apsu said i could ignore the mess... but alas, i cannot21:14
mhaydenhah, and i do all that work and my build ends up at rax again21:15
Sam-I-Amyw21:16
openstackgerritMajor Hayden proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952521:21
Sam-I-Ammhayden: patch set 35621:21
mhaydenhah yeah, works fine when i run it but the gate hates me21:22
mhaydeni wish i could get access to the actual image they use21:22
Sam-I-Ami'm sure its somewhere21:22
mhaydenarg, now the ssh issue is back21:24
mhaydenand i didn't change anything21:24
Sam-I-Ami had that locally a bit ago21:24
mhaydenit gets to the first playbook and then can't ssh21:24
Sam-I-Amoh, mine was dying later than that21:25
mhaydenstill need to figure out the difference between the nv + dvsm jobs21:26
Sam-I-Ami think jesse said its 2 different images21:26
Sam-I-Amdvsm is the same one that devstack runs on21:26
*** ysm has quit IRC21:27
mhaydenSam-I-Am: ah, thanks for that21:30
mhaydenhelps me understand more about that21:30
*** rady has quit IRC21:32
mhaydenwell i'm throwing in the towel for today21:32
*** Mudpuppy has quit IRC21:36
*** k_stev has quit IRC22:00
Apsumhayden: Like I said, pretend it's not there. It's for your own good.22:05
*** rady has joined #openstack-ansible22:14
*** BjoernT has quit IRC22:25
*** Bjoern_ has joined #openstack-ansible22:25
sigmavirus24mhayden: if ignored for long enough the gate stops voting22:27
openstackgerritSteve Lewis proposed openstack/openstack-ansible: Replace invalid references to Glance  https://review.openstack.org/24034022:29
*** sdake has joined #openstack-ansible22:37
*** sigmavirus24 is now known as sigmavirus24_awa22:37
*** sdake_ has joined #openstack-ansible22:39
*** jimchou has quit IRC22:39
*** sdake has quit IRC22:42
*** andyhky` is now known as andyhky22:43
*** sdake_ has quit IRC22:44
*** sdake has joined #openstack-ansible22:49
*** jwagner is now known as jwagner_away22:50
*** Bjoern_ has quit IRC23:06
*** gouthamr has joined #openstack-ansible23:09
*** daneyon has joined #openstack-ansible23:15
openstackgerritMerged openstack/openstack-ansible-security: V-51337: Use an LSM at boot  https://review.openstack.org/23328423:15
*** alop has quit IRC23:20
*** daneyon_ has joined #openstack-ansible23:22
*** daneyon has quit IRC23:25
*** spotz_zzz is now known as spotz23:28
*** Mudpuppy has joined #openstack-ansible23:30
*** slotti has quit IRC23:37
*** Mudpuppy has quit IRC23:38
*** spotz is now known as spotz_zzz23:39
*** rady has quit IRC23:47
*** daneyon_ has quit IRC23:51
*** daneyon has joined #openstack-ansible23:52
*** daneyon has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!