| odyssey4me | so that task needs to be in the barbican role | 00:01 |
|---|---|---|
| odyssey4me | I'm suggesting that it's added to your existing patch | 00:02 |
| odyssey4me | obviously the keystone var ref needs to be adjusted to be barbican related | 00:02 |
| odyssey4me | it suits the existing patch considering that it's applicable to enabling centos deployment for barbican :) | 00:03 |
| SmearedBeard | yeah, I added and rerun, failed at the same part.. gonna destroy container and try again | 00:05 |
| odyssey4me | yeah, it'll only activate if the venv changed | 00:08 |
| odyssey4me | you don't have to destroy the container - you can just remove the venv and the venv tarball, but destroying the container is probably simpler | 00:09 |
| openstackgerrit | Merged openstack/openstack-ansible stable/newton: Retry apt cache update https://review.openstack.org/521807 | 00:16 |
| SmearedBeard | Wahoo odyssey4me that all worked, ill update my patch as well | 00:17 |
| odyssey4me | SmearedBeard excellent :) | 00:17 |
| odyssey4me | thanks for persisting! | 00:17 |
| SmearedBeard | thank you for all the help and guidance | 00:18 |
| odyssey4me | I suppose the lesson learned from all that is to lead with the actual symptom first, then discuss what you've looked at... we spent a good while thinking it was a repo build issue there. ;) | 00:19 |
| odyssey4me | haha, I was just wondering whether we have bug report guidance... and: https://docs.openstack.org/openstack-ansible/latest/contributor/contribute.html#reporting-bugs | 00:21 |
| openstackgerrit | Aaron Mehar proposed openstack/openstack-ansible-os_barbican master: Add CentOS 7 deployment support https://review.openstack.org/521772 | 00:22 |
| odyssey4me | the trouble with having documentation is that it reaches the stage where figuring out how to lay it out in a useful and intuitive way becomes hard | 00:23 |
| SmearedBeard | yeah, I find troubleshooting is a real skill to have and learn that many people neglect :) | 00:23 |
| odyssey4me | yeah, finding the right audience is one of the hardest parts of doing docs | 00:24 |
| SmearedBeard | yeah, this is my first contribution to something and it I found myself going from page to page, site to site to look at guides, and bugs etc to make sure I was doing it right | 00:24 |
| odyssey4me | on the one hand you have people who can't seem to understand how to work through the steps, but on the other the steps are not always obvious | 00:25 |
| odyssey4me | on the one hand we want to make this a system where you really have a lot of choice - you can implement just about any architecture | 00:26 |
| odyssey4me | but on the other hand, with that choice comes a matrix of possibilities which we can't possibly test the entire matrix for... so things fall through the cracks and we have to rely on humans to report issues | 00:27 |
| odyssey4me | adding more test paths adds more tests to maintain... and unfortunately the number of people who can triage the issue, much less submit a suggested fix, is limited | 00:28 |
| odyssey4me | so thank you for stepping up - you had an itch, and you figured out how to scratch it and shared it :) | 00:29 |
| SmearedBeard | I minimal and common use testing is fine IMO, and anything beyond can be done perhaps on a case-by-case or individually.. | 00:30 |
| SmearedBeard | no worries, that is what opensource is all about :) | 00:30 |
| SmearedBeard | it's past midnight, Im off to bed | 00:30 |
| *** chyka_ has quit IRC | 00:31 | |
| *** chyka has joined #openstack-ansible | 00:31 | |
| odyssey4me | yeah, ultimately as a project we try to help out where we can - but we also have to manage our own tech debt and realise that it's those that contribute materially that drive the way the project goes | 00:31 |
| odyssey4me | it is past midnight, but my patch just worked... holy moley | 00:32 |
| odyssey4me | haha, I'm rambling | 00:33 |
| odyssey4me | have a good night's rest SmearedBeard - chat again tomorrow... thanks again | 00:33 |
| *** chyka has quit IRC | 00:36 | |
| *** SmearedBeard has quit IRC | 00:39 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: Make pw-token-gen.py more random https://review.openstack.org/520714 | 00:46 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: [DOCS] Correct secret uuid in Ceph example https://review.openstack.org/522039 | 00:49 |
| *** Chealion has quit IRC | 00:57 | |
| *** Chealion has joined #openstack-ansible | 00:58 | |
| *** hw_wutianwei has joined #openstack-ansible | 01:02 | |
| *** nshetty has joined #openstack-ansible | 01:09 | |
| *** huxinhui_ has joined #openstack-ansible | 01:15 | |
| *** tnarg has quit IRC | 01:19 | |
| *** hw_wutianwei has quit IRC | 01:30 | |
| *** nshetty_ has joined #openstack-ansible | 01:44 | |
| *** nshetty has quit IRC | 01:46 | |
| *** kukacz has quit IRC | 02:00 | |
| *** kukacz_ has joined #openstack-ansible | 02:01 | |
| *** chyka has joined #openstack-ansible | 02:02 | |
| *** hw_wutianwei has joined #openstack-ansible | 02:06 | |
| *** chyka has quit IRC | 02:07 | |
| *** nshetty_ has quit IRC | 02:20 | |
| cloudnull | evenings | 02:21 |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: [DOCS] Correct secret uuid in Ceph example https://review.openstack.org/522039 | 02:26 |
| *** woodard has quit IRC | 02:29 | |
| *** woodard has joined #openstack-ansible | 02:30 | |
| *** mrtenio-afk has quit IRC | 02:56 | |
| *** mrtenio-afk has joined #openstack-ansible | 02:56 | |
| openstackgerrit | Cuong Nguyen proposed openstack/openstack-ansible-os_nova master: Use group_names to check a host belongs to group https://review.openstack.org/520944 | 02:58 |
| *** albertcard1 is now known as poopcat | 02:58 | |
| *** cuongnv has joined #openstack-ansible | 03:00 | |
| cuongnv | cloudnull: can you pls help to review this https://review.openstack.org/#/c/520989? | 03:00 |
| cuongnv | odyssey4me: ^ thanks for your review and recheck | 03:01 |
| *** poopcat has quit IRC | 03:04 | |
| *** poopcat has joined #openstack-ansible | 03:04 | |
| openstackgerrit | Cuong Nguyen proposed openstack/openstack-ansible-os_glance master: Use glance_services to get group name https://review.openstack.org/520989 | 03:12 |
| cloudnull | cuongnv: looking | 03:15 |
| *** m|y|k has joined #openstack-ansible | 03:15 | |
| cloudnull | cuongnv: where is glance_services being defined? | 03:16 |
| cuongnv | cloudnull: defaults/main.yml | 03:16 |
| cloudnull | ah i see it now. | 03:21 |
| cloudnull | nicely done! | 03:21 |
| cuongnv | cloudnull: thanks | 03:26 |
| *** m|y|k has quit IRC | 03:59 | |
| *** m|y|k has joined #openstack-ansible | 04:00 | |
| *** john51 has quit IRC | 04:02 | |
| *** john51 has joined #openstack-ansible | 04:02 | |
| *** nshetty has joined #openstack-ansible | 04:41 | |
| *** pmannidi has quit IRC | 04:45 | |
| *** poopcat has quit IRC | 05:02 | |
| *** udesale has joined #openstack-ansible | 05:06 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Add dynamic table for our tested Scenarios https://review.openstack.org/520294 | 05:07 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_nova stable/ocata: Update filters to support over-commit https://review.openstack.org/522113 | 05:10 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Update all SHAs for master https://review.openstack.org/517520 | 05:11 |
| *** hybridpollo has quit IRC | 05:34 | |
| *** pmannidi has joined #openstack-ansible | 05:34 | |
| *** chyka has joined #openstack-ansible | 05:47 | |
| *** lihi has quit IRC | 05:51 | |
| *** oanson_ has joined #openstack-ansible | 05:52 | |
| *** oanson has quit IRC | 05:52 | |
| *** oanson_ is now known as oanson | 05:52 | |
| *** chyka has quit IRC | 05:52 | |
| *** lihi has joined #openstack-ansible | 05:53 | |
| *** markvoelker has quit IRC | 05:53 | |
| *** udesale has quit IRC | 06:00 | |
| *** udesale has joined #openstack-ansible | 06:00 | |
| *** taseer1 has joined #openstack-ansible | 06:00 | |
| *** Taseer has quit IRC | 06:01 | |
| *** taseer2 has joined #openstack-ansible | 06:01 | |
| *** taseer1 has quit IRC | 06:05 | |
| *** gouthamr has quit IRC | 06:10 | |
| *** indistylo has joined #openstack-ansible | 06:22 | |
| *** indistylo has quit IRC | 06:26 | |
| *** openstackgerrit has quit IRC | 06:33 | |
| *** hw_wutianwei1 has joined #openstack-ansible | 06:45 | |
| *** hw_wutianwei has quit IRC | 06:48 | |
| *** hw_wutianwei1 is now known as hw_wutianwei | 06:48 | |
| *** SmearedBeard has joined #openstack-ansible | 06:52 | |
| *** markvoelker has joined #openstack-ansible | 06:54 | |
| *** arbrandes has joined #openstack-ansible | 07:01 | |
| *** arbrandes1 has quit IRC | 07:03 | |
| *** SmearedBeard has quit IRC | 07:04 | |
| *** PaulusNL has joined #openstack-ansible | 07:15 | |
| *** nshetty is now known as nshetty|lunch | 07:22 | |
| *** openstackgerrit has joined #openstack-ansible | 07:25 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Converge neutron agents onto Baremetal https://review.openstack.org/454450 | 07:25 |
| *** jvidal has joined #openstack-ansible | 07:30 | |
| *** pmannidi has quit IRC | 07:37 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_nova stable/pike: Set the nova-lxd lib to use PIKE https://review.openstack.org/522155 | 07:41 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_nova stable/pike: Update filters to support over-commit https://review.openstack.org/516429 | 07:42 |
| *** threestrands has quit IRC | 07:47 | |
| *** pcaruana has joined #openstack-ansible | 07:49 | |
| *** mbuil has joined #openstack-ansible | 07:49 | |
| *** mrch has joined #openstack-ansible | 07:51 | |
| prometheanfire | go to bed cloudnull | 07:54 |
| openstackgerrit | Merged openstack/openstack-ansible-plugins stable/ocata: Add missing domain parameter to _get_group_roles https://review.openstack.org/521600 | 07:56 |
| openstackgerrit | Merged openstack/openstack-ansible-ceph_client stable/ocata: Remove one layer of includes https://review.openstack.org/521568 | 07:58 |
| *** SmearedBeard has joined #openstack-ansible | 07:58 | |
| *** shardy has joined #openstack-ansible | 07:59 | |
| *** SmearedBeard has quit IRC | 08:05 | |
| *** chyka has joined #openstack-ansible | 08:19 | |
| PaulusNL | goodmorning all, I've got a new openstack-ansible test environment in a personal VM environment. I don't know why but the only thing that is not working is Horizon webinterface on the extermal vip address | 08:19 |
| PaulusNL | (this would also be eventually be a start for out production) | 08:20 |
| PaulusNL | how can I troubleshoot this best? | 08:20 |
| *** gkadam has joined #openstack-ansible | 08:20 | |
| PaulusNL | if I connect to the utility-container I can see users with the command openstack user list for example | 08:20 |
| *** chyka has quit IRC | 08:23 | |
| *** holmsten has joined #openstack-ansible | 08:31 | |
| *** nshetty|lunch is now known as nshetty | 08:38 | |
| evrardjp | good morning everyone | 08:42 |
| evrardjp | PaulusNL: so external IP and internal IP are different? | 08:42 |
| evrardjp | if yes, did you check whether haproxy binds on this external IP, port 443? | 08:43 |
| *** sxc731 has joined #openstack-ansible | 08:43 | |
| PaulusNL | evrardjp: thank you for your quick response | 08:46 |
| PaulusNL | yes they are different but maybe I need to redo this with the same ip | 08:46 |
| PaulusNL | and I will check my HAproxy | 08:47 |
| evrardjp | keep those different | 08:49 |
| evrardjp | it's easier | 08:49 |
| PaulusNL | ok | 08:49 |
| PaulusNL | I will do this | 08:49 |
| evrardjp | this way you can isolate traffic too. | 08:49 |
| PaulusNL | ok | 08:49 |
| evrardjp | was it the same IP? | 08:49 |
| PaulusNL | nope | 08:49 |
| PaulusNL | two different | 08:49 |
| PaulusNL | internal and external | 08:50 |
| PaulusNL | but after I rebooted my deployment server (that also includes a few containers) that also has the utility_container is not started automatically | 08:50 |
| PaulusNL | so I think I have more problems with this test enviroment | 08:51 |
| evrardjp | I don't know why you have so much trouble. | 08:54 |
| evrardjp | maybe if you show your openstack user config and your user_*.yml files, we could help you. It shouldn't be that hard on clean machines. | 08:55 |
| evrardjp | Is your deploy server an infra node? | 08:56 |
| PaulusNL | ok | 08:56 |
| PaulusNL | jep | 08:56 |
| PaulusNL | shall I paste it | 08:56 |
| evrardjp | lxc-ls -f ? | 08:56 |
| *** udesale has quit IRC | 08:57 | |
| PaulusNL | evrardjp: did a pm | 08:58 |
| evrardjp | I see | 08:58 |
| PaulusNL | I also don't know why I'm running in so much trouble but I also did not get time for doing more about this installation | 08:58 |
| evrardjp | it runs on boot | 08:58 |
| evrardjp | you just need to wait a little after a reboot | 08:58 |
| PaulusNL | from today I have time from my boss :) | 08:58 |
| PaulusNL | ok | 08:58 |
| evrardjp | it takes a while to load all those containers when you have a single machine doing everything | 08:59 |
| PaulusNL | ok clear | 08:59 |
| PaulusNL | need to talk to someone... I will leave it for a while like this :) | 09:00 |
| PaulusNL | thank again evrardjp | 09:00 |
| PaulusNL | I can also paste the config I used | 09:01 |
| PaulusNL | if that is helpfull? | 09:01 |
| evrardjp | generally it is, we can understand the context a little more | 09:01 |
| evrardjp | please paste a link here for everyone to help you with | 09:01 |
| PaulusNL | ok | 09:01 |
| evrardjp | just remove what you consider confidential (public ips/names) | 09:02 |
| PaulusNL | that is for this no problem... it's my personall test env | 09:02 |
| PaulusNL | pm with the link to the config | 09:03 |
| evrardjp | could you write it here? This way anyone can help? | 09:04 |
| evrardjp | I have too much on my plate | 09:04 |
| PaulusNL | yes I can | 09:04 |
| PaulusNL | http://paste.openstack.org/show/627045/ | 09:05 |
| PaulusNL | thanks evrardjp | 09:06 |
| evrardjp | odyssey4me: hwoarang are you aware of a releasenotes problem? | 09:10 |
| evrardjp | or linters, or tempest... | 09:12 |
| *** pbandark has joined #openstack-ansible | 09:14 | |
| *** taseer2 is now known as Taseer | 09:23 | |
| hwoarang | evrardjp: good morning. no clue | 09:40 |
| *** pester has joined #openstack-ansible | 09:48 | |
| *** fxpester has quit IRC | 09:49 | |
| * bndzor breaks his ceph cluster on purpose to remove raid60, switching to raid 0 | 10:01 | |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/ocata: Retry apt cache update https://review.openstack.org/521803 | 10:03 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-ceph_client stable/pike: Remove one layer of includes https://review.openstack.org/521566 | 10:03 |
| evrardjp | hwoarang: thanks :) | 10:04 |
| openstackgerrit | Merged openstack/openstack-ansible-os_gnocchi stable/ocata: Deprecate auth_plugin option https://review.openstack.org/521816 | 10:05 |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh stable/ocata: Change auth config to keystone v3 according to openstack ocata aodh docs[1], we modified auth config under the "keystone_authtoken" and "service_credentials" groups. [1] https://docs.openstack.org/project-install-guide/telemetry-alarming/ \ ocata/install- https://review.openstack.org/521832 | 10:13 |
| *** cuongnv has quit IRC | 10:13 | |
| *** electrofelix has joined #openstack-ansible | 10:25 | |
| *** hw_wutianwei has quit IRC | 10:28 | |
| *** stuartgr has joined #openstack-ansible | 10:30 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_ceilometer stable/newton: Initial OSA zuul v3 role jobs https://review.openstack.org/513999 | 10:50 |
| andymccr | random question - does anybody remember where we setup UCA in mitaka? | 11:01 |
| *** jwitko has quit IRC | 11:05 | |
| odyssey4me | andymccr we didn't | 11:12 |
| andymccr | ahh ok | 11:12 |
| odyssey4me | we only started using UCA in newton | 11:12 |
| andymccr | well that would explain it :) | 11:14 |
| andymccr | i wonder if default ubuntu backports comes with the libec packages for swift | 11:14 |
| andymccr | must do since we install them as part of mitaka | 11:14 |
| odyssey4me | andymccr for trusty it was in backports, yes | 11:15 |
| odyssey4me | evrardjp I know that infra has been modifying the tests/builds for releasenotes and docs over the last week or so, and it has resulted in breakage. Over the w/end it was totally busted. | 11:20 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Add CentOS 7 deployment support https://review.openstack.org/521772 | 11:22 |
| *** sxc731 has quit IRC | 11:22 | |
| evrardjp | odyssey4me: I've pinged Andreas | 11:23 |
| evrardjp | I've seen the issue appearing after a patch of his | 11:23 |
| odyssey4me | hwoarang it'd be nice to round out the barbican role with suse support too ;) | 11:23 |
| odyssey4me | evrardjp I see a comment in #openstack-infra saying that releasenotes are working again - the comment was at 06:33 this morning | 11:25 |
| hwoarang | odyssey4me: sure :) | 11:33 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_gnocchi stable/newton: Use numpy 1.11.1 https://review.openstack.org/522224 | 11:56 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_gnocchi stable/newton: Use numpy version specified in upper constraints https://review.openstack.org/522224 | 11:57 |
| hwoarang | odyssey4me: do you know why we require postgresql-devel libraries in various os_* roles? | 11:59 |
| odyssey4me | hwoarang no idea, at some point I think they may have become a build requirement... but I don't know | 11:59 |
| hwoarang | right | 12:00 |
| *** ianychoi has quit IRC | 12:00 | |
| *** ianychoi has joined #openstack-ansible | 12:00 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_gnocchi stable/ocata: Use numpy version specified in upper constraints https://review.openstack.org/522226 | 12:01 |
| *** huxinhui_ has quit IRC | 12:04 | |
| *** shardy has quit IRC | 12:04 | |
| *** shardy has joined #openstack-ansible | 12:06 | |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Use package module to install lvm2 https://review.openstack.org/521953 | 12:10 |
| *** shardy has quit IRC | 12:15 | |
| *** shardy has joined #openstack-ansible | 12:16 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova stable/ocata: Update filters to support over-commit https://review.openstack.org/522113 | 12:18 |
| *** kukacz_ is now known as kukacz | 12:20 | |
| *** shardy has quit IRC | 12:23 | |
| *** SmearedBeard has joined #openstack-ansible | 12:26 | |
| *** shardy has joined #openstack-ansible | 12:28 | |
| *** pester has quit IRC | 12:36 | |
| *** fxpester has joined #openstack-ansible | 12:36 | |
| *** mrch has quit IRC | 12:38 | |
| odyssey4me | hwoarang evrardjp could you take a peek at https://review.openstack.org/#/q/Id9636b1160f84d069e1e231996f19773c81402fd asap please | 12:40 |
| evrardjp | it rings me a bell that if we remove that, we have a 1.11.x that gives problem | 12:42 |
| evrardjp | https://etherpad.openstack.org/p/osa-telemetry-stack-fixing | 12:42 |
| FoXMaN | hello | 12:43 |
| odyssey4me | https://github.com/openstack/openstack-ansible-os_gnocchi/commit/7fa88e5dec2f2828e3e0f44353932188eb9888ce is where it got added | 12:43 |
| odyssey4me | it happens to only be panndas that needs constraining, because it's not in u-c | 12:43 |
| evrardjp | https://github.com/openstack/openstack-ansible-os_gnocchi/commit/7fa88e5dec2f2828e3e0f44353932188eb9888ce | 12:43 |
| evrardjp | ok | 12:43 |
| evrardjp | let's try this | 12:44 |
| evrardjp | gokhan: FYI ^ | 12:44 |
| evrardjp | FoXMaN: hello | 12:44 |
| FoXMaN | any advice on how to debug slow horizon/api responses? i narrowed down problem to nova-api-os-compute container which takes 20-55 seconds to respond the request | 12:45 |
| FoXMaN | machines running containers have 6 cores and 32GB of ram. No sign of i/o related problems | 12:45 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_gnocchi stable/ocata: Use numpy version specified in upper constraints https://review.openstack.org/522226 | 12:45 |
| FoXMaN | but everything is slow | 12:45 |
| evrardjp | FoXMaN: nothing in the logs? dns resolving is fine? | 12:46 |
| odyssey4me | evrardjp hwoarang updated the ocata one to reflect that the version is the same in u-c | 12:46 |
| *** jvidal has quit IRC | 12:46 | |
| odyssey4me | the newton one was set to that version by me when trying to make the gates work again, so I think the newton one is good | 12:46 |
| evrardjp | done | 12:47 |
| odyssey4me | thanks | 12:47 |
| evrardjp | gokhan: please note the change ^ | 12:47 |
| FoXMaN | evrardjp: i haven't seen anything relevant. except horizon logs stating that request cannot be sent (due to haproxy timeout i belive) | 12:48 |
| gokhan | hello odyssey4me evrardjp | 12:48 |
| evrardjp | FoXMaN: did you try to see in a cli debug mode, to find which call is generally wrong? | 12:48 |
| evrardjp | FoXMaN: could you check also your haproxy 's backends state? | 12:48 |
| odyssey4me | gokhan howdy - I took the version spec for numpy out of the gnocchi role as it's already covered by upper-constraints | 12:48 |
| evrardjp | FoXMaN: is that a new installation or something old? | 12:49 |
| flaper87 | Hey folks, quick q re zuul: how are the jobs referenced in your zuul definition? I noticed you have openstack-ansible-linters in some repos but I don't see any requirement on openstack-ansible-tests | 12:49 |
| flaper87 | is that done automagically by zuul? | 12:49 |
| odyssey4me | flaper87 requirement? | 12:49 |
| FoXMaN | evrardjp: this is 2 months old installation. 20 compute hosts | 12:49 |
| evrardjp | flaper87: https://github.com/openstack/openstack-ansible-tests/blob/master/zuul.d/jobs.yaml#L17 ? | 12:50 |
| FoXMaN | evrardjp: by cli debug mode you mean client debug or service ? | 12:50 |
| gokhan | odyssey4me, yep you are right. ın fact pandas is enough | 12:50 |
| flaper87 | lemme rephrase | 12:50 |
| FoXMaN | evrardjp: haproxy times out after 50 seconds of waiting for backend response so it returns gateway timeout | 12:50 |
| evrardjp | client debug... to see all the calls, finding which one is slow | 12:50 |
| flaper87 | When I patch for https://github.com/openstack/openstack-ansible-plugins is posted | 12:50 |
| flaper87 | it triggers these jobs: https://github.com/openstack/openstack-ansible-plugins/blob/master/zuul.d/project.yaml#L18-L30 | 12:51 |
| odyssey4me | flaper87 yup | 12:51 |
| flaper87 | but, openstack-ansible-linters is not defined in that repo but openstack-ansible-tests | 12:51 |
| odyssey4me | correct | 12:51 |
| flaper87 | how does zuul find the definition for those jobs? | 12:51 |
| evrardjp | yes, we all share a funny loving global scope | 12:51 |
| evrardjp | flaper87: zuul is kinda awesome and scary, that's your answer. It knows ALL the jobs. | 12:52 |
| flaper87 | evrardjp: ok, so, it's done by zuul | 12:52 |
| flaper87 | coolio | 12:52 |
| odyssey4me | flaper87 it loads jobs into a single namespace from all repositories in an order that's specified in project-config... and all that has to happen is that the repo holding the tests needs to be before the repo using the tests, so we pushes our tests repo high up: https://review.openstack.org/512678 | 12:53 |
| *** yifei has quit IRC | 12:53 | |
| flaper87 | a-ha, that's the piece I was missing | 12:54 |
| flaper87 | gotcha, thanks for the details | 12:54 |
| odyssey4me | sure, no problem :) | 12:54 |
| FoXMaN | evrardjp: client debug indicates that GET http://172.23.0.20:8774/v2.1/f2321282f3824f99868879b6afe3368f/servers/detail?all_tenants=True took 20 of 22 total request time | 12:54 |
| evrardjp | I definitely don't know what this means like that. | 12:55 |
| odyssey4me | evrardjp mind giving this a nudge? https://review.openstack.org/521923 | 12:55 |
| evrardjp | FoXMaN: so it's just nova_api_os_compute that is slow? | 12:56 |
| evrardjp | odyssey4me: that's so ugly. | 12:57 |
| gokhan | evrardjp, https://review.openstack.org/#/c/521818/ what is reason failure. I can not understand clearly. error is ERROR: functional: commands failed | 12:57 |
| evrardjp | odyssey4me: you're doing me a lot of pain recently. | 12:57 |
| *** cjloader has joined #openstack-ansible | 12:57 | |
| evrardjp | But ok, voted. | 12:57 |
| odyssey4me | evrardjp how so? | 12:57 |
| FoXMaN | evrardjp: yes | 12:58 |
| evrardjp | so many ways to fix the linting, just skipping it ISN'T the solution :p | 12:58 |
| evrardjp | but you know it is a backport, and we need to advance. | 12:58 |
| odyssey4me | evrardjp oh sure, but better to get the gates moving sometimes than to get all creative while the patches pile up | 12:59 |
| evrardjp | vote is there. | 12:59 |
| odyssey4me | in most cases these test implementations are so horribly hacky they frighten me | 12:59 |
| evrardjp | gokhan: if you check on the log | 13:00 |
| odyssey4me | trying to make them more elegant starts feeling a bit like this: https://media.giphy.com/media/wotqBCVmRVn2w/giphy.gif | 13:00 |
| evrardjp | you can see on this line the reason of the failure http://logs.openstack.org/18/521818/1/gate/openstack-ansible-functional-centos-7/f83d560/job-output.txt.gz#_2017-11-22_09_24_19_595485 | 13:00 |
| evrardjp | odyssey4me: lol | 13:01 |
| *** pradiprwt has joined #openstack-ansible | 13:04 | |
| gokhan | evrardjp ok thanks. And also on nova-console and aodh haproxy config gives http 503 error . Service is unavailable beacause of 'http-check expect status 401'. If we remove this , services are working succesfully. | 13:05 |
| evrardjp | gokhan: you can change that in haproxy config | 13:05 |
| evrardjp | gokhan: https://github.com/openstack/openstack-ansible/blob/master/group_vars/haproxy_all/haproxy.yml | 13:05 |
| pradiprwt | Hi Everyone, How to disable audh, ceilometer projects in installation ? | 13:06 |
| evrardjp | it's not by default pradiprwt | 13:06 |
| pradiprwt | evrardjp: I remember last time I have disabled it by deleting few files "rm /etc/openstack_deploy/conf.d/{ceilometer,gnocchi,aodh}.yml" but now files names are different something like audh.yml.aio | 13:08 |
| pradiprwt | do I need to delete the .aio file ? | 13:09 |
| odyssey4me | pradiprwt the .aio files are not loaded by default, only .yml files | 13:11 |
| odyssey4me | so to enable, you rename to .yml | 13:11 |
| *** cjloader has quit IRC | 13:12 | |
| pradiprwt | odyssey4me : I want to disable it, default audh and ceilometer enabled | 13:13 |
| odyssey4me | no, by default it is not enabled | 13:13 |
| hwoarang | evrardjp: can we get this in https://review.openstack.org/#/c/521790/ so we dont miss breakages on pike? thank you | 13:13 |
| pradiprwt | While istallation I can see the containers with these names | 13:14 |
| odyssey4me | pradiprwt the default aio build renames these file in order to enable the services: https://github.com/openstack/openstack-ansible/blob/master/tests/bootstrap-aio.yml#L29-L38 | 13:14 |
| odyssey4me | pradiprwt then you must be using a *really* old build, or you've put in config yourself to enable them | 13:14 |
| evrardjp | hwoarang: I like this dynamism we have with the jobs. We should continue that way if a problem occurs. | 13:15 |
| hwoarang | :) | 13:15 |
| odyssey4me | evrardjp yeah, it's nice | 13:15 |
| evrardjp | while still fixing/hoping it shouldn't happen. | 13:15 |
| evrardjp | I am wondering what we can do for copr though. | 13:15 |
| pradiprwt | odyssey4me: got it , I am installing multi-node with HA, I think I need to rename the files | 13:16 |
| gokhan | evrardjp aodh-api haproxy config is fixed on master. But on pike and ocata it must also be fixed because of this line for pike https://github.com/openstack/openstack-ansible/blob/stable/pike/group_vars/all/haproxy.yml#L215 | 13:16 |
| evrardjp | because right now copr deserves to be called ncopr | 13:16 |
| openstackgerrit | Merged openstack/openstack-ansible-os_gnocchi stable/newton: Use numpy version specified in upper constraints https://review.openstack.org/522224 | 13:16 |
| evrardjp | ncapr* | 13:16 |
| hwoarang | evrardjp: it's very convenient we can tweak jobs from a single repo | 13:16 |
| evrardjp | yeah | 13:16 |
| odyssey4me | evrardjp yeah, we should implement an infra mirror for it | 13:16 |
| evrardjp | that we manage | 13:16 |
| hwoarang | corp stuff is annoying indeed :/ | 13:16 |
| evrardjp | not sure we can | 13:16 |
| odyssey4me | why not? either mirror or reverse proxy | 13:17 |
| odyssey4me | we did a for mariadb, and a reverse proxy for images.linuxcontainers | 13:17 |
| evrardjp | reverse proxy maybe but mirror ... couldn't there be a problem with the licensing? | 13:17 |
| odyssey4me | I dunno. rev proxy is fine then | 13:17 |
| odyssey4me | I'd just like to see less centos fails | 13:18 |
| evrardjp | tell me about it! | 13:18 |
| evrardjp | :p | 13:18 |
| evrardjp | or having a fast centos. | 13:18 |
| evrardjp | we are one month too early to ask for Xmas. | 13:18 |
| evrardjp | let's continue with what we have! | 13:19 |
| gokhan | evrardjp, also for nova-console it must be fixed. https://github.com/openstack/openstack-ansible/blob/master/group_vars/haproxy_all/haproxy.yml#L192 . this line must be removed. İf you confirm, ı can send patch for it. | 13:19 |
| evrardjp | gokhan: I can't confirm ..., because ppl don't seem to complain about it... I'd say we need to properly triage this if that's a bug | 13:20 |
| evrardjp | could you file a bug? | 13:20 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/newton: Update os_gnocchi role SHA for numpy version fix https://review.openstack.org/522240 | 13:25 |
| FoXMaN | evrardjp: after restarting container hosts,memcached did not start inside the containers. So we had no memcached alive | 13:26 |
| evrardjp | that sounds bad :p | 13:27 |
| odyssey4me | evrardjp hwoarang could you please review https://review.openstack.org/522240 asap | 13:28 |
| evrardjp | it was already done! | 13:28 |
| odyssey4me | thanks | 13:28 |
| *** pradiprwt has quit IRC | 13:29 | |
| odyssey4me | the ocata one can wait for the next role sha bump as it's the same version | 13:29 |
| *** yifei has joined #openstack-ansible | 13:29 | |
| *** yifei has quit IRC | 13:33 | |
| *** hw_wutianwei has joined #openstack-ansible | 13:36 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: Retry apt cache update https://review.openstack.org/521799 | 13:38 |
| *** dave-mccowan has joined #openstack-ansible | 13:41 | |
| hwoarang | evrardjp: is this outdated https://specs.openstack.org/openstack/openstack-ansible-specs/ or am i looking at the wrong thing? | 13:42 |
| odyssey4me | hwoarang yeah, for some reason the specs repo isn't publishing properly | 13:42 |
| odyssey4me | I haven't had a chance to look into it. | 13:42 |
| hwoarang | oh ;/ | 13:42 |
| hwoarang | ok | 13:42 |
| odyssey4me | it appears to only be ours, and I see other service specs are being published | 13:43 |
| odyssey4me | if you're able to look into it I'd appreciate that! | 13:43 |
| evrardjp | yeah we don't publish | 13:44 |
| *** dave-mcc_ has joined #openstack-ansible | 13:44 | |
| odyssey4me | we are supposed to | 13:44 |
| evrardjp | I don't think it matters. | 13:44 |
| evrardjp | does it? | 13:44 |
| odyssey4me | it matters - either we publish, or that content must be removed | 13:44 |
| evrardjp | I mean: are we really supposed to? | 13:44 |
| odyssey4me | because right now it looks like we aren't doing anything with specs | 13:45 |
| odyssey4me | yes, we are supposed to - we've been publishing them since kilo when we created that repo | 13:45 |
| *** dave-mccowan has quit IRC | 13:45 | |
| hwoarang | it's convenient to know what specs are there i think | 13:46 |
| evrardjp | doing it in the past doesn't mean we are forced to by governance | 13:46 |
| hwoarang | well it's misleading if you keep outdated info there :) | 13:46 |
| *** SerenaFeng has joined #openstack-ansible | 13:46 | |
| odyssey4me | no, we're not forced to by governance | 13:46 |
| evrardjp | hwoarang: I'd rather make that part of our documentation. | 13:46 |
| *** SerenaFeng has quit IRC | 13:46 | |
| odyssey4me | but we *chose* to implement the specs process | 13:46 |
| odyssey4me | and if we have specs, we should publish them just like we do all our other docs | 13:47 |
| odyssey4me | the published specs are indexed better by search engines than the git source | 13:48 |
| evrardjp | I am all in for open design. How the sausage is made doesn't matter to me. | 13:48 |
| evrardjp | I am thinking of removing blueprints too. | 13:49 |
| *** yifei has joined #openstack-ansible | 13:50 | |
| odyssey4me | sure, if you like | 13:51 |
| *** SerenaFeng has joined #openstack-ansible | 13:51 | |
| evrardjp | but I agree this spec thing isn't clear. | 13:51 |
| odyssey4me | personally I find that specs are useful as a record of design decisions including the background that led up to them | 13:51 |
| odyssey4me | reviews alone don't cover that, and there isn't really any other facility that provides something similar at this stage | 13:52 |
| odyssey4me | if we choose to implement something else to replace it that makes better sense for our workflow, then I'm all in | 13:52 |
| odyssey4me | It could perhaps be useful to rather publish them in our docs so that they're in-context. | 13:53 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Fix ansible-lint test failures https://review.openstack.org/522029 | 13:58 |
| gokhan | ok evrardjp I will file a bug | 13:58 |
| evrardjp | odyssey4me: I don't mean get rid of the specs. The specs are interesting. I think it should be closer to the project, part of our documentation maybe. Will talk with Docs PTL what he thinks about that. | 14:01 |
| evrardjp | I've updated the wiki to point to our specs too with the cgit. | 14:01 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible-specs master: Fix linter issue https://review.openstack.org/522257 | 14:01 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron stable/pike: Resolve ansible-lint test failures https://review.openstack.org/521923 | 14:02 |
| logan- | i think that might fix the specs publishing issue ^ | 14:08 |
| evrardjp | logan-: if this is the only cause, that was a quick fix :) | 14:11 |
| logan- | i don't understand how that got merged | 14:12 |
| logan- | so who knows | 14:12 |
| logan- | lol | 14:12 |
| openstackgerrit | Merged openstack/openstack-ansible-os_gnocchi stable/ocata: Use numpy version specified in upper constraints https://review.openstack.org/522226 | 14:14 |
| hwoarang | hmm something regressed really bad on centos https://review.openstack.org/518611 opensuse finishes in less than 1h but centos timesout in tempest :/ | 14:14 |
| logan- | actually it doesn't look like the ci job runs tox at all, so doc8 isn't running there | 14:14 |
| logan- | i guess it probably isn't the issue then | 14:14 |
| evrardjp | maybe someone should file a bug | 14:15 |
| evrardjp | this way we register the issue and we can properly triage it, this way ppl can say what they think about the criticallity | 14:15 |
| evrardjp | -l | 14:15 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: Use group_names to check a host belongs to group https://review.openstack.org/520944 | 14:16 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_rally stable/ocata: Initial Zuul v3 role jobs https://review.openstack.org/514292 | 14:16 |
| openstackgerrit | Merged openstack/openstack-ansible-specs master: Fix linter issue https://review.openstack.org/522257 | 14:18 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/pike: zuul.d: jobs.yaml: Restore voting status of openSUSE jobs https://review.openstack.org/521790 | 14:20 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally stable/ocata: Updated from OpenStack Ansible Tests https://review.openstack.org/520139 | 14:21 |
| *** PaulusNL has quit IRC | 14:24 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_rally stable/ocata: Initial Zuul v3 role jobs https://review.openstack.org/514292 | 14:26 |
| odyssey4me | logan- evrardjp I suspect that the job is incomplete, because I didn't see a job go into post for the publishing | 14:26 |
| odyssey4me | the job definition I mean | 14:26 |
| logan- | ah | 14:26 |
| logan- | yeah | 14:26 |
| odyssey4me | it's odd though, because when I last checked we were defined the same way as other specs repositories | 14:27 |
| odyssey4me | admittedly that was a long time ago though, and I wasn't looking very hard because all the other zuul v3 stuff was happening | 14:27 |
| evrardjp | file a bug, triage it, fix it depending on the urgency :p | 14:30 |
| evrardjp | it's a process :) | 14:30 |
| *** SerenaFeng has quit IRC | 14:31 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh stable/ocata: Deprecate auth_plugin option https://review.openstack.org/521818 | 14:31 |
| *** esberglu has joined #openstack-ansible | 14:34 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: Update all SHAs for 16.0.4 https://review.openstack.org/518829 | 14:36 |
| gun1x | i got approval for hardware, to create a POC | 14:37 |
| gun1x | i need to get the list of hardware now. was thinking of: 2 firewalls/routers, 2 switches, 3 controller nodes, 3 compute+ceph nodes | 14:37 |
| gun1x | i still need to figure the best vendor for network devices | 14:38 |
| *** nshetty has quit IRC | 14:45 | |
| cloudnull | mornings | 14:45 |
| *** portante has quit IRC | 14:49 | |
| cloudnull | gun1x: We use a lot of F5 of Cisco and/or Arista. Most of the nics we use are Intel (some mellanox) and we've recently be testing Q'Logic (I've heard it looks good so far). | 14:55 |
| cloudnull | ** F5, Cisco and/or Arista... | 14:56 |
| gun1x | cloudnull: all nics 10GB i guess | 14:56 |
| cloudnull | yup. | 14:56 |
| gun1x | and classic 4 nics / node? 2 bonds, 1 bond mgmt + storage, 1 bond vxlan + vlan ? | 14:56 |
| cloudnull | we'll use the onboard nics if we need 1GB | 14:56 |
| gun1x | why would you "need" lesser power | 14:57 |
| gun1x | ? | 14:57 |
| cloudnull | depends on the setup. | 14:58 |
| cloudnull | and the port density | 14:58 |
| jrosser | 25G nics are the very similar price to 10G these days and same for switches | 14:58 |
| cloudnull | ^ | 14:58 |
| gun1x | nice | 14:58 |
| evrardjp | Did anyone already get issues with http://docs.ansible.com/ansible/latest/git_module.html that gets fixed by removing the depth: argument? | 14:58 |
| evrardjp | It's not the first time | 14:59 |
| evrardjp | for me | 14:59 |
| evrardjp | udner ansible 2.1 | 14:59 |
| gun1x | are you running compute and ceph OSD on the same nodes? and ceph mon on the controllers? | 14:59 |
| evrardjp | and good morning gun1x jrosser cloudnull | 14:59 |
| odyssey4me | evrardjp you're seeing issues if depth is set? | 14:59 |
| evrardjp | yes | 14:59 |
| gun1x | good morning, it's 5 pm in eastern europe :D | 14:59 |
| odyssey4me | might it be an old git version? | 14:59 |
| evrardjp | no, git 2. | 15:00 |
| evrardjp | not 1.9 something | 15:00 |
| odyssey4me | oh, interesting - I've not seen it, but I can totally believe nonsense happening with ansible 2.1.x | 15:00 |
| evrardjp | so it should be totally alright | 15:00 |
| odyssey4me | ansible 2.2 did so many things better - 2.1 was a bit of a fail | 15:00 |
| evrardjp | it's quite transient | 15:00 |
| evrardjp | the worse thing | 15:00 |
| cloudnull | o/ evrardjp | 15:00 |
| evrardjp | is that this module is probably calling module.warn which doesn't work (let me check on that one) | 15:01 |
| odyssey4me | evrardjp perhaps we should just remove all uses of git depth for newton then? | 15:01 |
| evrardjp | cloudnull: how was the bug triage yesterday? | 15:01 |
| evrardjp | N and O I'd say. | 15:01 |
| odyssey4me | it'll just slow it down horribly, which is not good | 15:01 |
| evrardjp | I've seen this happen in O too. | 15:01 |
| evrardjp | O2. Hahah. | 15:01 |
| cloudnull | evrardjp: it was the best ! | 15:01 |
| evrardjp | cloudnull: great! | 15:01 |
| evrardjp | I have seen no logs of it. | 15:02 |
| cloudnull | we triaged like you wouldn't believe | 15:02 |
| evrardjp | That's right! | 15:02 |
| cloudnull | kinda like a secret santa | 15:02 |
| evrardjp | LD | 15:02 |
| evrardjp | :D | 15:02 |
| * cloudnull forgot he was supposed to run the triage | 15:02 | |
| * cloudnull is a bad person | 15:02 | |
| evrardjp | Maybe we should organise one this week? Next week is community meeting | 15:02 |
| cloudnull | IDK, i'm off tomorrow and friday | 15:03 |
| evrardjp | with the zuul, summit, holidays, meetings, it was quite a few weeks. | 15:03 |
| cloudnull | I suspect most of the US will be too | 15:03 |
| evrardjp | ok | 15:03 |
| cloudnull | but I'd be happy to join in should we have a meeting | 15:03 |
| evrardjp | let's do it next week, another day of the week, probably thursday | 15:03 |
| *** portante has joined #openstack-ansible | 15:03 | |
| cloudnull | gun1x: re:ceph - I believe we deploy using stand alone ceph cluster to keep the separation of concerns however others may have thoughts on / real data on what works. | 15:05 |
| * cloudnull is no ceph expert | 15:05 | |
| *** cjloader has joined #openstack-ansible | 15:06 | |
| andymccr | i think it depends on your use-case gun1x - e.g. if you dont have massive workloads and/or you have limited hardware you "could" put osds on compute. im not a huge fan of that approach personally. | 15:07 |
| andymccr | mon hosts could go on infra, but again if you have large load they'll probably need to be split off at some point | 15:07 |
| gun1x | andymccr: for a Proof of Concept i think it should be enough | 15:08 |
| andymccr | gun1x: yeah definitely, for a PoC it'd work fine | 15:09 |
| gun1x | andymccr: i am looking through https://github.com/ceph/ceph-ansible/blob/master/group_vars/all.yml.sample and can't find the variables to configure object storage to run on hdd and block storage to run on ssd | 15:10 |
| gun1x | logan- said this is possible but i can't remember wher eto find the docu on how to write the playbook | 15:11 |
| andymccr | gun1x: that's not really a ceph-ansible thing, thats how you configure your drives and pools | 15:11 |
| gun1x | andymccr: so that is not configure via ansible and is done via ceph commands? | 15:12 |
| *** gouthamr has joined #openstack-ansible | 15:12 | |
| andymccr | gun1x: yeah take a look at this article for example: https://www.sebastien-han.fr/blog/2014/08/25/ceph-mix-sata-and-ssd-within-the-same-box/ | 15:12 |
| andymccr | you'd then create the 2 pools and for your cinder backend you'd use the pool with only SSDs, and for radosGW you'd use the sata pool name | 15:13 |
| gun1x | andymccr: doesn't OSA automatically create the pools? | 15:13 |
| jrosser | imho you might want to do ceph yourself using ceph-ansible | 15:14 |
| andymccr | gun1x: i think we create a pool to get it set-up as part of an AIO | 15:14 |
| jrosser | you will then learn all about it rather than fight ceph-ansible indirectly through OSA | 15:14 |
| jrosser | ans in luminous it will auto allocate SSD and HDD into the relevant pools for you, no need to fiddle the crush map | 15:15 |
| andymccr | jrosser: ahh sweet. did not know that had been added to luminous - how does it know which is ssd vs sata? | 15:15 |
| gun1x | jrosser: AFAIK support for ceph was added in pike in OSA | 15:16 |
| jrosser | i think "spinning media" is a flag that can be read from a drive | 15:16 |
| andymccr | jrosser: oh that is pretty awesome | 15:16 |
| andymccr | https://blog-fromsomedude.rhcloud.com/2017/05/16/Luminous-series-CRUSH-devices-class/ | 15:17 |
| andymccr | gives some details gun1x - pretty decent read | 15:17 |
| jrosser | gun1x: yeah i know, just sharing that even though thats there we've chosen to separate them | 15:17 |
| jrosser | but it's up to you :) felt like too much to tangle together i guess in our environment | 15:18 |
| andymccr | jrosser: i think its nice in some ways - e.g. the inventory integration, but from a learning perspective i'd agree, deploy ceph yourself with ceph-ansible and give that a go | 15:18 |
| gun1x | ok i will go through the documentation | 15:21 |
| evrardjp | oh interesting, we are not the only ones: https://github.com/openstack/os-faults/blob/master/os_faults/ansible/executor.py#L19-L22 | 15:22 |
| gun1x | thank you for the advice, andymccr jrosser | 15:26 |
| jrosser | andymccr: cat /sys/block/sda/queue/rotational | 15:33 |
| jrosser | also lsblk -d -o name,rota | 15:34 |
| *** marst has joined #openstack-ansible | 15:36 | |
| *** mbuil has quit IRC | 15:51 | |
| *** mbuil has joined #openstack-ansible | 15:52 | |
| logan- | for converged vs non converged osds, ive not had great experiences when the servers have a lot of drives. there is too much contention between vms and the osds for cpu and ram. but converged has worked great for me on blades where i am only running 1 or 2 osds per node | 15:52 |
| *** holmsten has quit IRC | 15:56 | |
| *** yifei has quit IRC | 15:57 | |
| *** michelv has joined #openstack-ansible | 16:08 | |
| odyssey4me | I'm clearly being dumb here. Is there a condition I can use to test if a host exists in the inventory. | 16:17 |
| odyssey4me | I'm using the openstack dynamic inventory, and need to only do something against a host if it's already there.... if not, then the creation will happen and life moves on swimmingly. | 16:18 |
| odyssey4me | jrosser logan- evrardjp thoughts? | 16:18 |
| logan- | if hostvars[host] is defined | 16:19 |
| logan- | if host in groups['all'] | 16:19 |
| logan- | one of those maybe? | 16:19 |
| odyssey4me | I suppose I could use... when: "'server-name' in groups['all']" | 16:19 |
| odyssey4me | heh, same-same | 16:19 |
| odyssey4me | thanks | 16:19 |
| logan- | if host in hostvars.keys() | 16:20 |
| odyssey4me | I think I like that one best | 16:21 |
| evrardjp | technically the best would be if member of groups['all'] | 16:22 |
| odyssey4me | oh? ok | 16:22 |
| evrardjp | but most likely if host in hostvars.keys() would work | 16:22 |
| cloudnull | odyssey4me: looking for a known host in inventory or a host of a particular group? | 16:24 |
| odyssey4me | in the inventory as a whole | 16:26 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible master: [WIP] Implement iptables management https://review.openstack.org/512099 | 16:33 |
| odyssey4me | yeah, this works nicely: when: "'server-name' in groups['all']" | 16:35 |
| *** germs has joined #openstack-ansible | 16:41 | |
| SmearedBeard | hey all.. quick question I understand this to divide the number of cores by 2 to the nearest whole number, but not sure what that 1 means at the end ;[ansible_processor_vcpus|default(2) // 2, 1; | 16:44 |
| SmearedBeard | if that a minimum acceptable value? | 16:44 |
| odyssey4me | SmearedBeard paste the whole line | 16:44 |
| SmearedBeard | heat_api_threads: "{{ [[ansible_processor_vcpus|default(2) // 2, 1] | max, heat_api_threads_max] | min }}" | 16:45 |
| odyssey4me | IIRC it takes the smallest value between half the cores (an integer result) and the max processes | 16:45 |
| logan- | hwoarang: is the build hanging at the end of the run still happening? im guessing you didnt find the cause yet. i think i might have just stumbled on something | 16:45 |
| odyssey4me | yeah, so the array is [ half cores, 1 ] | max -> ie the largest number | 16:45 |
| *** hw_wutianwei has quit IRC | 16:46 | |
| odyssey4me | then that is used in another array - [ that largest number, the max threads ] | min | 16:46 |
| odyssey4me | it's there to prevent masses of processes from being created in very hihgh end systems like POWER where there are often 128+ cores in a system | 16:46 |
| hwoarang | logan-: yeah it does. i have a strange feeling that ansible can't kill processes properly or something and this https://review.openstack.org/#/c/520615/ seems to fix it | 16:46 |
| hwoarang | logan-: as you see the opensuse job failed but because of tempest not because of timeout. the playbook finishes correctly | 16:47 |
| hwoarang | i have no clue... | 16:47 |
| logan- | yep i was just suspecting dstat isnt being killled | 16:47 |
| logan- | http://osa-ci.objects-us-dfw-1.cloud.lstn.net/516448/10/927/logs/console.log <- last 3 lines of output here | 16:47 |
| logan- | log collcetion function didn't run in that build and you can see jenkins alert about the hung file descriptors | 16:48 |
| hwoarang | yeah but i would expect that parent should kill everything upon exit | 16:48 |
| *** woodard has quit IRC | 16:48 | |
| logan- | instead of running "dstat <args> &" maybe we should run it in nohup or something to decouple it from the gate-check-commit descriptors | 16:48 |
| hwoarang | yeah :/ | 16:48 |
| hwoarang | sneaky bug | 16:48 |
| odyssey4me | logan- hwoarang I noticed when working on the tests to verify api uptime during upgrades that there are problems trying to kill processes from ansible tasks... I think it might have to do with something along the lines of one thing can't kill another subshell's process or something like that | 16:49 |
| logan- | yup i see the only other thing we're launching in a similar fashion is some stuff in the rolling upgrade tests | 16:49 |
| odyssey4me | yup, and that's where I saw the issue | 16:49 |
| odyssey4me | the one in the integrated repo still isn't working right as far as I know | 16:50 |
| SmearedBeard | odyssey4me that makes sense and what I understood, I was making notes and released I had 16 threads, but my processor is an i7 so the I should have been assigned 4 threads.. | 16:50 |
| odyssey4me | the keystone one is working though | 16:50 |
| odyssey4me | SmearedBeard I don't think it's cores though, it's vCPU's... so it's likely cores + hyperthreading | 16:50 |
| odyssey4me | you may have 8 cores, but the host will think it's 16 cpu's... and therefore ansible will too | 16:51 |
| logan- | ok thanks ill take a look at that | 16:51 |
| *** savvas has quit IRC | 16:51 | |
| *** savvas_ has joined #openstack-ansible | 16:51 | |
| SmearedBeard | odyssey4me it's an i7 4 core, so 8 with hyperthreading - I am running grep -c processor /proc/cpuinfo and getting 8.. | 16:52 |
| odyssey4me | SmearedBeard oh, that's interesting | 16:52 |
| odyssey4me | SmearedBeard if you do this what's shown: ansible -m debug -a var=ansible_processor_vcpus <host> | 16:53 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible master: Fix automatic log copying in Zuul runs https://review.openstack.org/516448 | 17:02 |
| *** michelv has quit IRC | 17:07 | |
| *** chyka has joined #openstack-ansible | 17:07 | |
| mbuil | can somebody confirm me that we can use "with_first_found" syntax with an "include" statement like here ==> https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/main.yml#L73? | 17:08 |
| mbuil | It is not working for me :( | 17:08 |
| *** chyka has joined #openstack-ansible | 17:10 | |
| cloudnull | mbuil: yup that should work. | 17:12 |
| cloudnull | what issue are you seeing ? | 17:12 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_rally stable/ocata: Initial Zuul v3 role jobs https://review.openstack.org/514292 | 17:13 |
| mbuil | cloudnull: ok. I don't see anything being executed from the include playbook. However, if I include it doing (in this case neutron_plugin_type = ml2.opendaylight) - include: opendaylght_config.yml, then I can see the playbook being executed | 17:15 |
| cloudnull | and the task file is in the providers directory ? | 17:16 |
| cloudnull | providers/opendaylght_config.yml | 17:16 |
| mbuil | cloudnull: https://hastebin.com/bucurecove.md. The Include provider specific config(s) is supposed to execute that part, but you can observe that nothing is executed and it jumps to the next line | 17:16 |
| mbuil | cloudnull: yes, it is there ==> https://github.com/openstack/openstack-ansible-os_neutron/tree/master/tasks/providers | 17:17 |
| cloudnull | ah so it's literally not running this file https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/opendaylight_config.yml | 17:18 |
| cloudnull | I thought I saw a patch for that fly by the other day | 17:18 |
| * cloudnull looking | 17:18 | |
| *** pcaruana has quit IRC | 17:18 | |
| mbuil | cloudnull: exactly. Perhaps it gets the name incorrectly... I am investigating too | 17:19 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest stable/pike: Fix ansible-lint test failures https://review.openstack.org/522304 | 17:22 |
| mbuil | cloudnull: I found the issue, my mistake! There was a typo in the neutron_plugin_type | 17:23 |
| mbuil | cloudnull: it works now :) | 17:24 |
| cloudnull | nice! | 17:24 |
| mbuil | cloudnull: thanks! | 17:24 |
| * cloudnull is glad it wasn't something we broke :D | 17:24 | |
| odyssey4me | those with_first_found tasks do result in some obscure errors | 17:24 |
| *** gkadam has quit IRC | 17:25 | |
| odyssey4me | I've wasted time on one of those at least once, going down rabbit holes of trying to figure out why ansible's looking in the wrong places... eventually to realise that the error was far, far simpler. | 17:26 |
| odyssey4me | jmccrory evrardjp how's that inventory conversion script going? | 17:26 |
| odyssey4me | also, in other news, I'm kinda warming to the idea of having our 'openstack-ansible' become something a little less of a bash-hack and quite a bit more of a real client | 17:28 |
| andymccr | with_first_found is the devil | 17:28 |
| odyssey4me | it could allow us to have commands like 'openstack-ansible bootstrap roles' and perhaps replace the inventory-manage script with 'openstack-ansible inventory ...' | 17:28 |
| evrardjp | odyssey4me: good to hear :) | 17:29 |
| evrardjp | odyssey4me: the click framework is quite great on those topics, very simple to use, and to extend | 17:29 |
| evrardjp | I didn't work on the inventory conversion for a while, but it was working for me | 17:30 |
| odyssey4me | but the click framework is not what openstack uses unfortunately, so if we use it we become a snowflake | 17:30 |
| evrardjp | yes, but it's widely used in the python community, so I think we are good. | 17:30 |
| odyssey4me | I wonder why the openstack client didn't use it. | 17:30 |
| evrardjp | https://github.com/pallets | 17:31 |
| odyssey4me | I vaguely remember a ML discussion about it years ago. | 17:31 |
| evrardjp | I think we are independent enough | 17:31 |
| evrardjp | I didn't want to learn something more complex than something I already knew how to do :p | 17:31 |
| evrardjp | we are using jinja anyway so we are already using part of pallets work? :) | 17:32 |
| odyssey4me | yeah, I'm good with that | 17:32 |
| evrardjp | anyway | 17:32 |
| odyssey4me | not sure if we'd be violating some form of governance if we used it | 17:32 |
| evrardjp | I'd say that if we do we'd be happy to change things later. | 17:33 |
| evrardjp | but I think the prio is not on that | 17:33 |
| evrardjp | the prio right now is on ansible 2.4 | 17:33 |
| evrardjp | bringing the same features as what we have, but with ansible 2.4 | 17:33 |
| bndzor | soon(TM) | 17:34 |
| evrardjp | so using our wrapper with an empty inventory file to be able to load userspace group vars / host vars + the docs that goes with it. | 17:34 |
| evrardjp | that and the usual burning fires. | 17:34 |
| odyssey4me | I'm pretty close to making the repo server entirely optional in a deployment, and also making it optionally a cache for wheels instead of having to build them. | 17:36 |
| odyssey4me | I should have some more patches up soon for that. | 17:36 |
| *** savvas has joined #openstack-ansible | 17:40 | |
| *** savvas_ has quit IRC | 17:40 | |
| cloudnull | ^ prometheanfire - re: using click, does that have any implications in requirements? | 17:41 |
| bndzor | fml, the moment i need the console and it says 503 Service Unavailable | 17:44 |
| evrardjp | odyssey4me: cloudnull could you vote on this? it works now: https://review.openstack.org/#/c/514292/ | 17:45 |
| evrardjp | bndzor: sorry to hear that :p | 17:45 |
| evrardjp | looks like you were the only one to have that recently | 17:45 |
| evrardjp | I heard that today too | 17:45 |
| evrardjp | wonder what changed | 17:46 |
| cloudnull | bndzor: https://i.imgflip.com/1ztyzp.jpg | 17:46 |
| bndzor | :( | 17:46 |
| odyssey4me | evrardjp awesome to see https://review.openstack.org/#/c/514292/6 working - happy to merge it? | 17:48 |
| evrardjp | yeah | 17:48 |
| evrardjp | will fix rally everywhere, this way I want to show a benchmark team that deploying on metal rocks. | 17:48 |
| evrardjp | I am listening too much metal too, doesn't help. | 17:49 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_rally master: Constraint ansible installation https://review.openstack.org/522314 | 17:51 |
| bndzor | hmm | 17:53 |
| bndzor | what keeps overwriting my /etc/resolv.conf | 17:53 |
| bndzor | dpkg -P cloudinit, still happening.. dhcd ? | 17:53 |
| bndzor | dhcpd*? | 17:53 |
| *** dmsimard is now known as dmsimard|afk | 17:53 | |
| prometheanfire | cloudnull: what's using click? | 17:53 |
| evrardjp | me | 17:53 |
| evrardjp | I am using it alot! | 17:54 |
| prometheanfire | in what, where? | 17:54 |
| odyssey4me | bndzor usually resolvconf will do that based on network interface config | 17:54 |
| evrardjp | in osa, and almost all the cli tooling I write | 17:54 |
| evrardjp | prometheanfire: why would that be a problem? | 17:54 |
| cloudnull | prometheanfire: its a py lib https://pypi.python.org/pypi/click | 17:54 |
| prometheanfire | it shouldn't | 17:54 |
| evrardjp | k | 17:54 |
| prometheanfire | it'd be a problem for co-installability | 17:54 |
| evrardjp | what do you mean? | 17:55 |
| prometheanfire | I don't think osa is listed in projects.txt | 17:55 |
| prometheanfire | looking | 17:55 |
| odyssey4me | yes | 17:55 |
| cloudnull | we participate in requirements so if we add this lib, its added everywhere. right? | 17:55 |
| odyssey4me | but that will only be an issue if we expect it to be in the global requirements | 17:55 |
| prometheanfire | https://github.com/openstack/requirements/blob/master/projects.txt#L127 | 17:55 |
| odyssey4me | if we have it as a requirement through some other means (like we do with ansible), then it won't be affected by that | 17:55 |
| prometheanfire | it should be added, we track the deps of projects that are in reqs | 17:56 |
| evrardjp | not sure what you meant there odyssey4me? | 17:56 |
| odyssey4me | *but* if we plan to build a package and publish it to pypi, then we'd have to have it in requirements.txt and therefore we'd have to have it in global requirements | 17:56 |
| prometheanfire | it's installed through ansible, not reqs.txt? | 17:56 |
| evrardjp | prometheanfire: at the moment nothing is listed in reqs. and click isn't installed through ansible | 17:57 |
| odyssey4me | evrardjp any package in requirements.txt or test-requirements.txt has to be registered in global requirements in the requirements repo - there's a process and set of requirements for that | 17:57 |
| odyssey4me | if we get it in, then it'll get put into u-c and all those nice things | 17:57 |
| evrardjp | ok, and would that be a problem? | 17:57 |
| prometheanfire | well, I don't think there'd be a problem with adding it to the reqs repo | 17:57 |
| evrardjp | ok | 17:58 |
| evrardjp | then we are good. | 17:58 |
| odyssey4me | *but* one of the criteria is that there isn't something in the list that already does what it does | 17:58 |
| prometheanfire | true, is there? | 17:58 |
| odyssey4me | and so we'd have to justify why we aren't using the things the openstackclient is using | 17:58 |
| evrardjp | because we are not openstackclient | 17:59 |
| evrardjp | :p | 17:59 |
| prometheanfire | that's not a valid reason | 18:00 |
| prometheanfire | the best angle to take would be to find something click can do that the other libs can't | 18:00 |
| prometheanfire | I also don't see tests in https://github.com/pallets/click | 18:00 |
| evrardjp | but there is no lib that does that? | 18:00 |
| evrardjp | https://github.com/pallets/click/tree/master/tests ? | 18:01 |
| cloudnull | we use click in monitorstack https://github.com/openstack/monitorstack/blob/master/requirements.txt#L1 | 18:01 |
| prometheanfire | maybe https://github.com/pallets/click/blob/master/click/testing.py | 18:01 |
| cloudnull | but its not part of g-r-r | 18:01 |
| prometheanfire | lol | 18:01 |
| evrardjp | https://github.com/pallets/click/blob/master/tox.ini#L12 | 18:01 |
| prometheanfire | ya, missed that entire folder | 18:01 |
| odyssey4me | cloudnull yeah, but I don't think monitorstack is subscribed to the global requirements management | 18:01 |
| cloudnull | it is not | 18:01 |
| evrardjp | but when you think about it, this tooling could be completely optional | 18:02 |
| evrardjp | and completely separated | 18:02 |
| odyssey4me | I'm not saying that click can't be used, but I am just saying that don't think of it as a given that you will be able to. Your review to put it into g-r will need to justify why your lib is better to use for what we're doing. | 18:02 |
| odyssey4me | if the reasons are good, I find that the requirements team (hey prometheanfire!) are usually happy to roll with it | 18:03 |
| prometheanfire | yep, beyond that it looks good | 18:03 |
| evrardjp | well click allows me to define in a very clean way with just a few decorators a context for my cli functions. | 18:04 |
| evrardjp | Also convenient way to validate arguments without the need of argparse | 18:04 |
| prometheanfire | don't tell me here, tell me in the review :P | 18:05 |
| evrardjp | I mean | 18:05 |
| evrardjp | yeah. | 18:05 |
| evrardjp | "I am just lazy" should be a good thing, right? | 18:05 |
| prometheanfire | in that case you should make everyone lazy, and switch out lib_x to for click | 18:06 |
| prometheanfire | in all of openstack :P | 18:06 |
| odyssey4me | so, I just asked mordred about it - to his knowledge click wasn't considered for OSC... cliff was used perhaps because it was the best around at the time that met the needs of OSC... and it seems that click wasn't around then | 18:07 |
| *** mbuil has quit IRC | 18:07 | |
| odyssey4me | osc started in 2012, click released 1.0 in 2014 it seems | 18:07 |
| evrardjp | oh yeah that would make sense | 18:07 |
| odyssey4me | armed with all that info, I think you have a proposal ;) | 18:08 |
| evrardjp | well I will remember all this conversation :) | 18:09 |
| prometheanfire | so, since we have another lib that does it all already it's gonna be a hard sell | 18:11 |
| prometheanfire | atm the only reason to allow it in reqs would be if cliff was fully switched out for click | 18:11 |
| odyssey4me | well, that could be an interesting project ;) | 18:12 |
| prometheanfire | yep | 18:12 |
| prometheanfire | basically, find a feature that's NEEDED that click provides that cliff doesn't and it will be easier | 18:13 |
| evrardjp | decorators! | 18:13 |
| evrardjp | I mean proper decorators | 18:13 |
| evrardjp | damn let's ignore this whole thing | 18:14 |
| prometheanfire | lol | 18:14 |
| evrardjp | I will just do like monitorstack | 18:14 |
| prometheanfire | iirc, the requirements bot may not like that | 18:15 |
| evrardjp | ? | 18:15 |
| evrardjp | monitorstack is just a different project, so we can do something like that, no? | 18:15 |
| odyssey4me | yeah, if we add a requirement to requirements.txt that isn't in g-t then we'll fail the requirements checks | 18:15 |
| prometheanfire | having something that's not tracked | 18:15 |
| odyssey4me | we can, but then we have to unsubscribe from the g-r process and go back to managing it ourselves | 18:16 |
| prometheanfire | monitorstack isn't in projects.txt | 18:16 |
| evrardjp | no I mean having a different project | 18:16 |
| evrardjp | yeah | 18:16 |
| evrardjp | openstack-ansible-cli | 18:16 |
| evrardjp | boom | 18:16 |
| prometheanfire | oh, that's possible | 18:16 |
| evrardjp | odyssey4me: are the linters fixed? I thought everything was fixed now | 18:17 |
| prometheanfire | it may come back to bite you if click (or any other lib not tracked) starts to not be co-installable with the rest of openstack (lib versions supported) | 18:17 |
| evrardjp | odyssey4me: ignore this. me short memory. | 18:17 |
| *** m|y|k has quit IRC | 18:17 | |
| odyssey4me | prometheanfire for us, today, that's less of an issue - but it would become an issue if we ever got packaged by the distributions | 18:18 |
| odyssey4me | there's the other factor of us effectively splitting from the community direction if we did it, which doesn't sit well | 18:18 |
| odyssey4me | isn't it amazing how a simple technology choice can be the cause of so much trouble ;) | 18:19 |
| prometheanfire | not totally true, it'd be installed in the global namespace, iirc nova-compute is run there as well | 18:19 |
| prometheanfire | :D | 18:19 |
| prometheanfire | if it was installed in a safe place I wouldn't care as much | 18:21 |
| prometheanfire | isn't ansible itself installed in a venv? | 18:21 |
| odyssey4me | and I guess we can never expect our users to do that ;) | 18:21 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-ceph_client stable/pike: Fix linters https://review.openstack.org/522322 | 18:21 |
| odyssey4me | yes, our tooling puts it into a venv and most of our users, as far as we know, use that tooling | 18:21 |
| odyssey4me | but it is plausible that someone may not | 18:22 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-ceph_client stable/pike: Remove one layer of includes https://review.openstack.org/521566 | 18:22 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-ceph_client stable/pike: Remove one layer of includes https://review.openstack.org/521566 | 18:22 |
| prometheanfire | ya, that makes it hard | 18:22 |
| prometheanfire | how much work would it be to use cliff instead? | 18:22 |
| *** TxGirlGeek has joined #openstack-ansible | 18:23 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally stable/ocata: Initial Zuul v3 role jobs https://review.openstack.org/514292 | 18:23 |
| prometheanfire | because I think that's what I'd tell any other project :( | 18:24 |
| evrardjp | I checked, cliff isn't that easy to review and read. So I'd say it would involve writing quite a series of code. I'd rather write everything using argparse instead, and multiply the amount of code we have by 3. | 18:24 |
| prometheanfire | you can bug tonyb and dirk about it as well | 18:24 |
| prometheanfire | maybe message -dev about fully switching to it? | 18:24 |
| evrardjp | Nah it's fine we already spent too much time discussing it. | 18:24 |
| evrardjp | lol no | 18:25 |
| odyssey4me | if we have the right reasons, and it doesn't interfere, and it's co-installable, then it might still be accepted into g-r given the right justification | 18:25 |
| evrardjp | I'd rather re-write with argparse than start the politics fight. I'll get results faster. | 18:25 |
| evrardjp | could anyone vote on https://review.openstack.org/#/c/522322/1 ? | 18:26 |
| prometheanfire | ya, the ease of use can be an argument, I'm just not sure it's enough :( | 18:26 |
| odyssey4me | done | 18:26 |
| evrardjp | closer to python community :) | 18:26 |
| evrardjp | thanks | 18:26 |
| evrardjp | zaqar passes on my machine! | 18:27 |
| evrardjp | woot | 18:27 |
| evrardjp | cloudnull: could you give me a nice fast vote? https://review.openstack.org/#/c/522322 | 18:27 |
| odyssey4me | wooho | 18:28 |
| bndzor | if i want to deploy more compute nodes, is it enough to run os-nova ? | 18:28 |
| prometheanfire | dynamic-inventory needs to be run right? | 18:29 |
| prometheanfire | https://docs.openstack.org/openstack-ansible/latest/admin/maintenance-tasks/scale-environment.html | 18:29 |
| odyssey4me | bndzor nova and neutron | 18:29 |
| bndzor | odyssey4me: cool, in that order ? | 18:30 |
| bndzor | ah cool, thanks for that link prometheanfire | 18:30 |
| odyssey4me | it appears so https://github.com/openstack/openstack-ansible/blob/master/playbooks/setup-openstack.yml#L20-L21 | 18:30 |
| odyssey4me | openstack-ansible os-nova-install.yml os-neutron-install.yml | 18:31 |
| prometheanfire | odyssey4me: does dynamic-inventory not need to be run anymore? | 18:31 |
| odyssey4me | prometheanfire ansible runs it, and it's still there right now - but we are trying to move away from it for ansible 2.4 | 18:33 |
| odyssey4me | unless I've misunderstood your question? | 18:33 |
| prometheanfire | oh, neat | 18:34 |
| evrardjp | yeah then ansible tower! | 18:34 |
| prometheanfire | the docs say to run it manually | 18:34 |
| prometheanfire | lol | 18:34 |
| odyssey4me | huh? where? | 18:34 |
| odyssey4me | as far as I know it does say so as a diagnostic or verification only | 18:35 |
| prometheanfire | oh | 18:35 |
| evrardjp | yeah ansible runs it automatically? | 18:35 |
| odyssey4me | oh I see in the page you linked | 18:35 |
| odyssey4me | that's a pretty strange sequence to advise | 18:35 |
| odyssey4me | those warnings are all pre-newton | 18:36 |
| prometheanfire | yarp | 18:36 |
| odyssey4me | oh, I see what's being done there | 18:37 |
| evrardjp | ? | 18:37 |
| odyssey4me | that's an interesting way of catering to a control plane host's containers | 18:37 |
| odyssey4me | so basically you don't have to type out all the container names in your limit | 18:37 |
| odyssey4me | that tag to update the hosts is very old too | 18:38 |
| odyssey4me | wow, it would be so nice to have an army of operators to keep these docs up to date | 18:38 |
| odyssey4me | ok, I'm out for the night - cheers all! | 18:40 |
| prometheanfire | nn | 18:40 |
| evrardjp | good night odyssey4me! | 18:40 |
| bndzor | ah btw, i have a strange question guys. So in my old env, im running xen. Is there anyway that if i dd and create an image of one of my old machines, can push it into openstack somehow ? | 18:42 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-ceph_client stable/ocata: Fix linters https://review.openstack.org/522346 | 18:43 |
| bndzor | or, i guess i could setup one vm, attach a temporary volume, dd it back there, then detach it, and create a new image and attach that as the volume | 18:46 |
| *** hachi_ has joined #openstack-ansible | 18:47 | |
| *** viktor_ has joined #openstack-ansible | 18:48 | |
| prometheanfire | should have asked for reviews on https://review.openstack.org/521696 | 18:48 |
| openstackgerrit | Merged openstack/openstack-ansible-ceph_client stable/pike: Fix linters https://review.openstack.org/522322 | 18:48 |
| *** shardy has quit IRC | 18:51 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/newton: Update os_gnocchi role SHA for numpy version fix https://review.openstack.org/522240 | 18:53 |
| *** poopcat has joined #openstack-ansible | 19:02 | |
| *** albertcard1 has joined #openstack-ansible | 19:08 | |
| *** poopcat has quit IRC | 19:08 | |
| cloudnull | bndzor: I wrote a tool to something like that a while back | 19:16 |
| cloudnull | https://github.com/cloudnull/InstanceSync | 19:16 |
| bndzor | nice! | 19:16 |
| cloudnull | its really just fancy rsync | 19:16 |
| *** cjloader has quit IRC | 19:17 | |
| cloudnull | its likely a bit old and will need to be updated for more modern instance | 19:17 |
| cloudnull | however it could help you get the job done. | 19:17 |
| bndzor | im doing a test now with dd first to see if it even works | 19:18 |
| cloudnull | Aug 20, 2012 ... its been a minute since i looked at this | 19:19 |
| cloudnull | dd should work just fine. | 19:19 |
| cloudnull | you might need to install cloud-init | 19:19 |
| cloudnull | then dd | 19:19 |
| *** electrofelix has quit IRC | 19:19 | |
| cloudnull | if cloud-init wasn't already there. | 19:20 |
| bndzor | hehe | 19:21 |
| bndzor | lets see what happens without | 19:21 |
| bndzor | the thing is, im also moving over a legacy environment into os | 19:21 |
| bndzor | i dont have time to reinstall and do everything fancy right now so this will have to do | 19:22 |
| bndzor | ive done the most ghetto crazy setup ever with a ethernet bridge between the two environments etc | 19:22 |
| bndzor | but hey, it works :) | 19:22 |
| *** stuartgr has quit IRC | 19:22 | |
| logan- | bndzor: i don't know if you have any windows instances, but i helped a client move windows vms from xen to openstack recently and wrote some stuff on it https://logan.protiumit.com/2017/04/13/xen-to-kvm.html | 19:22 |
| bndzor | yuck windows! | 19:23 |
| bndzor | odyssey4me: i can confirm its not enough to run thoose two playbooks | 19:23 |
| bndzor | fatal: [appserver7]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Warning: Permanently added '172.29.236.57' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: /usr/bin/python: not found\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 127} | 19:23 |
| bndzor | fatal: [appserver4]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Warning: Permanently added '172.29.236.54' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: /usr/bin/python: not found\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 127} | 19:23 |
| bndzor | fatal: [appserver6]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Warning: Permanently added '172.29.236.56' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: /usr/bin/python: not found\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 127} | 19:23 |
| bndzor | fatal: [appserver5]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Warning: Permanently added '172.29.236.55' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: /usr/bin/python: not found\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 127} | 19:23 |
| bndzor | fatal: [appserver3]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Warning: Permanently added '172.29.236.53' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: /usr/bin/python: not found\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 127} | 19:23 |
| bndzor | il just run setup-everything later and head out to grab some food | 19:24 |
| cloudnull | logan-: nice! | 19:24 |
| * cloudnull has not had to do anything windows related (professionally) in a very long time. | 19:25 | |
| bndzor | i only use windows to play some games | 19:25 |
| bndzor | when ever that happens.. | 19:25 |
| bndzor | been working so hard latley that i am expecting a calls from chineese sewing factoriess coming soon | 19:26 |
| cloudnull | lol | 19:26 |
| cloudnull | logan-: that driver issue looks like a tun of fun | 19:28 |
| bndzor | only windows can have a message that tells you a system disk could not be found booting from it | 19:29 |
| bndzor | :-D | 19:29 |
| logan- | cloudnull: yeah windows cloud image prep is a blast. its not limited to xen conversions either... i think i get mild ptsd anytime someone says "just slipstream these drivers" | 19:30 |
| logan- | cause you just know its gonna be a week of fighting bsod | 19:30 |
| bndzor | that feeling when you get ~150mbit/s on a 20ge uplink and the other side has the same :( | 19:31 |
| logan- | haha | 19:31 |
| cloudnull | ha! | 19:31 |
| bndzor | never, ever, ovh again | 19:31 |
| cloudnull | oye logan- do you guys sell those discontinued servers out right? | 19:36 |
| cloudnull | Like If i wanted to buy a couple and colo? | 19:36 |
| logan- | yes | 19:36 |
| bndzor | if you want cheap ass servers check out kimsufi.. but its very lowend and ovh ;( | 19:36 |
| cloudnull | I'm contemplating moving all of my things off "cloud" and into dedicated gear | 19:37 |
| cloudnull | I've got a couple whitebox servers at home . | 19:38 |
| SmearedBeard | and soyoustart, it's the middle brand | 19:38 |
| cloudnull | but I'm thinking about getting a half rack | 19:38 |
| SmearedBeard | so I found that trove does not support CentOS 7 either :( so ill be testing this this evening and sending a patch up for this as well | 19:39 |
| *** hachi_ has quit IRC | 19:40 | |
| cloudnull | ++ | 19:40 |
| cloudnull | sadly a couple of our roles have haven't had the same multi-os love as others. | 19:41 |
| logan- | cloudnull seems to be a popular thing lately. lots of ppl doing colo term deal with either old hardware bundled into the deal or we lease new hardware to fill the space. | 19:41 |
| cloudnull | I want to say all of them support opensuse. | 19:41 |
| cloudnull | ^ thanks hwoarang :) | 19:41 |
| SmearedBeard | cloudnull meh not big deal, it's even enough to figure it and add | 19:41 |
| SmearedBeard | and they do not.. barbican and trove have not got opensuse | 19:41 |
| cloudnull | ah. | 19:42 |
| hwoarang | yeah not all of them | 19:43 |
| hwoarang | i am doing barbican these days so stay tuned | 19:43 |
| cloudnull | logan-: are the prices for purchase on the https://www.limestonenetworks.com ? | 19:43 |
| SmearedBeard | cool.. I did start looking at Barbican but opensuse package names looked weird, so I went back my testing my deployment | 19:44 |
| *** hachi_ has joined #openstack-ansible | 19:44 | |
| bgmccollum_ | If I add a provider_network after completing a deployment, i know i need to re-run the neutron playbook, but I think I need to run something else so the corresponding interface gets created in the neutron_agents container...yes? | 19:44 |
| cloudnull | bgmccollum_: lxc-container-create | 19:44 |
| hwoarang | SmearedBeard: normally they are pretty similar to Centos. and the package list appears to be a subset of those listed in os_horizon | 19:44 |
| bgmccollum_ | do i need to destroy before? or it'll fix it up? | 19:44 |
| bgmccollum_ | cloudnull thx | 19:45 |
| cloudnull | you can chain the two together `openstack-ansible lxc-container-create.yml os-neutron-install.yml --limit neutron_all` | 19:45 |
| *** bgmccollum_ is now known as bgmccollum | 19:45 | |
| bgmccollum | perfect | 19:45 |
| cloudnull | nothing to do before hand | 19:45 |
| cloudnull | it'll add the interfaces into the container config and restart it | 19:45 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible master: scripts: scripts-library.sh: Use pgrep -f to find the dstat process https://review.openstack.org/520615 | 19:45 |
| SmearedBeard | cloudnull ill take a look and and test once vagrant is done running for trove/centos | 19:46 |
| cloudnull | sweet! | 19:46 |
| logan- | cloudnull: no not really. Most of the stuff on the site is bare metal lease not colo. Ping me sometime what you want specs wise. We have been basically giving away e3v1 and dual 5600s in most of the deals lately and also free months on longer term deals | 19:48 |
| SmearedBeard | cloudnull speaking of --limit.. my deploy failed, ansible has the ability to restart from the trove play right? | 19:48 |
| logan- | cloudnull: all that old hw doesn’t do much good sitting in the stock room so if we can get someone into colo on it, we make it cheap :p | 19:49 |
| *** hachi_ has quit IRC | 19:50 | |
| *** hachi_ has joined #openstack-ansible | 19:53 | |
| cloudnull | SmearedBeard: yes. | 19:56 |
| cloudnull | you can just rerun the os-trove-install playbook | 19:56 |
| cloudnull | without having to do all of the setup-openstack plays | 19:56 |
| cloudnull | assuming it failed within the trove role. | 19:56 |
| *** esberglu has quit IRC | 20:00 | |
| *** esberglu has joined #openstack-ansible | 20:00 | |
| *** esberglu has quit IRC | 20:00 | |
| SmearedBeard | cloudnull yeah, failed at ' Gather variables for each operating system'.. what about things after trove? octavia is LBaaSv2 right? | 20:01 |
| *** esberglu has joined #openstack-ansible | 20:01 | |
| cloudnull | that's likely a fact issue or can be related to the operating system variable file not existing | 20:02 |
| cloudnull | https://github.com/openstack/openstack-ansible-os_neutron/tree/master/vars | 20:02 |
| cloudnull | it tries to do a lookup of the variable file by name using facts. | 20:03 |
| cloudnull | so if the facts are bad / missing or if the variable file is not found it'll throw that error | 20:03 |
| cloudnull | as for octavia, yes that is lbaasv2 | 20:03 |
| cloudnull | ^ xgerman_ | 20:03 |
| cloudnull | ^ has been working on that role. | 20:03 |
| SmearedBeard | cloudnull I fixed the issue and just testing before sending a patch, I just wanted to avoid runnig setup-openstack.yml again, as it's long | 20:05 |
| cloudnull | ++ | 20:05 |
| *** esberglu has quit IRC | 20:05 | |
| gun1x | logan-: andymccr: https://www.sebastien-han.fr/blog/2014/11/27/ceph-recover-osds-after-ssd-journal-failure/ | 20:06 |
| gun1x | did you ever try anything like this to recover a journal? | 20:06 |
| bndzor | # ceph health | 20:12 |
| bndzor | HEALTH_OK | 20:12 |
| * bndzor goes back to beeing not productive | 20:13 | |
| cloudnull | hahaha | 20:13 |
| *** hybridpollo has joined #openstack-ansible | 20:15 | |
| *** gouthamr has quit IRC | 20:20 | |
| *** albertcard1 is now known as poopcat | 20:21 | |
| *** gouthamr has joined #openstack-ansible | 20:22 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_rally master: Constraint ansible installation https://review.openstack.org/522314 | 20:23 |
| andymccr | gun1x: not recently, im not sure if there is a better process now | 20:23 |
| gun1x | andymccr: ok so that actually works | 20:23 |
| andymccr | gun1x: if only the journal has failed hten should do yeah. assuming you set the right values there - that command uses like 20GB journals so if youre using something diff it wont | 20:25 |
| andymccr | and device names are correct etc :) | 20:25 |
| andymccr | but that article is pretty old, so perhaps they have improved that | 20:25 |
| gun1x | andymccr: thanks! | 20:26 |
| *** hachi_ has quit IRC | 20:26 | |
| cloudnull | where's leseb when you need him :P | 20:27 |
| *** hachi_ has joined #openstack-ansible | 20:28 | |
| *** hachi_ has quit IRC | 20:28 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_zaqar stable/ocata: Initial Zuul v3 role jobs https://review.openstack.org/514299 | 20:28 |
| *** hamza has joined #openstack-ansible | 20:29 | |
| evrardjp | cloudnull: in france | 20:30 |
| evrardjp | :p | 20:30 |
| evrardjp | cloudnull: andymccr because you seem to be there: wanna vote on this one? https://review.openstack.org/#/c/522346/ | 20:31 |
| evrardjp | it's about ceph! | 20:31 |
| cloudnull | done | 20:32 |
| evrardjp | at the same time you could also vote for more ceph things: https://review.openstack.org/#/c/521566/ | 20:32 |
| logan- | gun1x: nope. havent had a situation where ive ever needed to recover a journal | 20:32 |
| bndzor | knock in wood | 20:34 |
| hamza | Hello, | 20:35 |
| hamza | can you please tell me what is the expected time for this to get merged ? https://review.openstack.org/#/c/477017 i mean the idea of isolating using systemd-nspawn | 20:36 |
| bndzor | thats a large commit | 20:39 |
| bndzor | but i like that idea | 20:39 |
| *** pbandark has quit IRC | 20:40 | |
| openstackgerrit | Merged openstack/openstack-ansible-ceph_client stable/ocata: Fix linters https://review.openstack.org/522346 | 20:43 |
| cloudnull | hamza: I need to continue working on that PR | 20:44 |
| cloudnull | if you have thoughts on it I'd really appreciate the feedback | 20:44 |
| cloudnull | it needs a rebase, and I think I can cut out quite a bit of the PR with recent changes being added in. | 20:45 |
| * cloudnull just needs more hours in the day | 20:45 | |
| bndzor | hmm | 20:46 |
| bndzor | how do i detach a volume ? | 20:46 |
| cloudnull | hamza: quite a few of the parts of that patch have landed in our existing lxc setup | 20:46 |
| cloudnull | so we can do things like have contianer images managed by machinectl | 20:46 |
| bndzor | ah, nevermind, had to power down | 20:48 |
| cloudnull | I just wrote a blog post on using machinectl to preseed images on a deployment https://cloudnull.io/2017/11/openstack-ansible-machinectl-image-management/ | 20:48 |
| cloudnull | ^ shameless plug | 20:48 |
| cloudnull | also covering the backend storage options https://cloudnull.io/2017/11/storage-setup-for-machinectl/ | 20:49 |
| cloudnull | so we can get really close to nspawn now. and hopfully in the very near future we can actually just use nspawn | 20:50 |
| cloudnull | at least as an option. | 20:50 |
| cloudnull | I hope that helps hamza? | 20:50 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-tests stable/ocata: Depth argument is prome to failure under ansible 2.1 https://review.openstack.org/522386 | 20:53 |
| *** TxGirlGeek has quit IRC | 20:56 | |
| *** viktor_ has quit IRC | 20:59 | |
| *** gouthamr has quit IRC | 21:03 | |
| hamza | Thanks cloudnull it helps, i was not aware of those blog posts. Actually, i have been using LXD with btrfs filesystem and macvlan for networking that can be also used with systemd-nspawn, | 21:10 |
| cloudnull | ++ that's where I want to take the nspawn work | 21:11 |
| hamza | which adds good features like snapshotting, migration... also macvlan is better in terms of performance | 21:11 |
| cloudnull | the pr as it stands is using macvlan and machinectl (btrfs). | 21:12 |
| cloudnull | I have that setup running my home cloud on one of the controllers. so far so good. | 21:12 |
| hamza | i did not get any problem using them with LXD, We have Ocata LXC deployment works fine, but i think with nspawn OSA could be better | 21:14 |
| cloudnull | I tend to agree. :D | 21:15 |
| *** gouthamr has joined #openstack-ansible | 21:15 | |
| cloudnull | hamza: https://cloudnull.io/2017/06/nspawning-openstack-ansible/ - also might be of some use from a perspective point of view. | 21:19 |
| cloudnull | its at least what I've been working toward | 21:19 |
| bndzor | running setup-everything.yml takes 7 hard years :/ | 21:22 |
| *** xingchao has joined #openstack-ansible | 21:23 | |
| bndzor | hmm, i belive ceph reports incorrect numbers in horizon | 21:26 |
| bndzor | Local Disk Usage | 21:26 |
| bndzor | Used 280GB of 218.3TB | 21:26 |
| bndzor | now, the 218TB, i got half of that, but it counts each nodes as one because it shows up as local. | 21:26 |
| cloudnull | setup-eveything can take a while. | 21:26 |
| cloudnull | sounds like you have plenty of space :D | 21:26 |
| bndzor | adding 5 more compute nodes, so i guess its going to count 100*7tb | 21:26 |
| bndzor | well not so extremly much, 3 storage nodes | 21:27 |
| bndzor | 40tb each | 21:27 |
| cloudnull | 40TiB ~= 218TiB :P | 21:27 |
| bndzor | 40 would be the correct number per node, so 120 | 21:28 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-tests stable/ocata: Depth argument is prome to failure for ansible<2.3 https://review.openstack.org/522386 | 21:28 |
| bndzor | /dev/sdb2 208896 93767828622 93767619727 43.7T unknown | 21:28 |
| bndzor | btw, is anyone running hadoop here? Would you say its ok to run inside nova or should i go for ironic? | 21:31 |
| hamza | thank you cloudnull, you did a goood job. it is well explained :) | 21:34 |
| cloudnull | we have folks that run hadoop workloads in VMs. | 21:34 |
| cloudnull | it seems to work for them | 21:34 |
| cloudnull | hamza: thanks! if you have any feedback on the PR please let me know | 21:35 |
| cloudnull | bndzor: but I've never run it for any reason other than to play around. | 21:35 |
| cloudnull | so idk if and where the ugly parts are | 21:36 |
| bndzor | ah.. im just worried about if i loose lets say 10-15% using it on a compute node or not | 21:36 |
| bndzor | cause i got like 15 of machines for that | 21:36 |
| bndzor | so 10% would be a bit | 21:36 |
| bndzor | and the number, i got from reading some old benchmarks | 21:37 |
| *** dave-mcc_ has quit IRC | 21:39 | |
| *** threestrands has joined #openstack-ansible | 21:45 | |
| *** threestrands has quit IRC | 21:45 | |
| *** threestrands has joined #openstack-ansible | 21:45 | |
| *** hamza has quit IRC | 21:51 | |
| bndzor | logan-: i think the ceph storage is a bit buggy when displaying in nova -> http://upload.boras.org/upload/files/5a15f8365f61f5a15f8365f6bd.png | 21:56 |
| *** xingchao has quit IRC | 21:57 | |
| bndzor | err horizon. i need sleep | 21:59 |
| gun1x | bndzor: nice infrastructure. | 21:59 |
| gun1x | where's the bug? | 22:00 |
| bndzor | so, im using ceph as storage | 22:00 |
| bndzor | the total storage size is 109.2TB | 22:00 |
| bndzor | but it calculates every machine (because it shows up as local) | 22:00 |
| gun1x | oh | 22:01 |
| gun1x | is this pike or queens? | 22:01 |
| bndzor | pike | 22:01 |
| gun1x | this is bad news | 22:01 |
| gun1x | there must be some patch in upstream for horizon for this | 22:02 |
| gun1x | probably 1 line of vode | 22:02 |
| gun1x | *code | 22:02 |
| gun1x | does this look ok via openstack api? | 22:02 |
| bndzor | how would i check that ? | 22:02 |
| gun1x | should be hypervizor list and after that hypervizor show name | 22:03 |
| gun1x | if you don't know what i am talking about, find the utility container on your controller node and load the rc file with environment variables (should be in home folder of root user) and type "openstack" | 22:03 |
| bndzor | doesnt show any diskspace with hypervisor list --long | 22:04 |
| bndzor | +----+----------------------------+-----------------+---------------+-------+------------+-------+----------------+-----------+ | 22:04 |
| bndzor | | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | | 22:04 |
| bndzor | +----+----------------------------+-----------------+---------------+-------+------------+-------+----------------+-----------+ | 22:04 |
| bndzor | | 5 | appserver2.openstack.local | QEMU | 172.29.236.52 | up | 0 | 48 | 2048 | 515883 | | 22:04 |
| bndzor | etc | 22:04 |
| gun1x | bpaste :( | 22:04 |
| bndzor | looks fine here, get a wider screen! :D | 22:04 |
| gun1x | what about hypervisor show spindles | 22:04 |
| gun1x | sorry wrong paste | 22:05 |
| gun1x | what about hypervisor show appserver2.openstack.local | 22:05 |
| gun1x | damn thank god i didn't have something strange in my clipboard | 22:05 |
| bndzor | haha | 22:05 |
| bndzor | disk_available_least | 111425 | 22:06 |
| bndzor | free_disk_gb | 111779 | 22:06 |
| *** xingchao has joined #openstack-ansible | 22:06 | |
| bndzor | local_gb | 111779 | 22:06 |
| gun1x | so this is right | 22:06 |
| bndzor | yup | 22:06 |
| bndzor | checked another node and its correct there also | 22:07 |
| gun1x | if this doesn't get fixed till i get to prod with this, i'll have to dust off my django skills | 22:07 |
| bndzor | so its only cosmetic | 22:07 |
| gun1x | https://www.youtube.com/watch?v=aHaGipVcIJ4 | 22:08 |
| gun1x | watching this atm | 22:08 |
| bndzor | first thing i saw was redhat so i hit ctrl + w quickly | 22:09 |
| gun1x | omg man :D | 22:10 |
| *** Dinesh_Bhor has quit IRC | 22:11 | |
| bndzor | :) | 22:12 |
| gun1x | bndzor: is this the deploy you were trying to do a few days ago, bndzor ? | 22:12 |
| bndzor | yeah | 22:13 |
| bndzor | its working fine now | 22:13 |
| bndzor | tried breaking ceph, adding back ceph nodes, adding in more compute nodes etc, all seems working.. small small things but solved easy | 22:14 |
| gun1x | how did you deploy ceph? | 22:14 |
| gun1x | with OSA ? | 22:14 |
| bndzor | https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/app-config-prod-ceph.html | 22:14 |
| gun1x | bndzor: ok, so not with ceph-ansible. everything from top to bottom with openstack-ansible | 22:15 |
| bndzor | yep! :) | 22:15 |
| bndzor | the only thing i dont like about it is that its slower than a train in india, but atleast it works | 22:16 |
| gun1x | did you set osd_objectstore: bluestore | 22:16 |
| gun1x | in user variables? | 22:16 |
| bndzor | yep | 22:16 |
| gun1x | "atleast it works" :D | 22:16 |
| gun1x | how much time did it take to build everything? | 22:16 |
| bndzor | 4 hours | 22:17 |
| bndzor | well, maybe 3 | 22:17 |
| bndzor | hard to keep track of time hehe | 22:17 |
| bgmccollum | did you change the number of forks ansible uses? That can speed it up depending on the number of nodes you are deploying to... | 22:18 |
| bgmccollum | i think the default is like 5 | 22:18 |
| gun1x | bndzor: dude, i find 4 hours to be really ok. | 22:18 |
| bndzor | gun1x: my first openstack hack install was bashscripted with ssh and apt packages, ~10 minutes | 22:18 |
| bndzor | but, i didnt like to have my own hacks for that | 22:19 |
| gun1x | ... | 22:19 |
| gun1x | bndzor: i have a request, i hope you have the time :D | 22:19 |
| bndzor | sup | 22:19 |
| gun1x | can you please show me df -h on the OSD nodes? | 22:19 |
| bndzor | sure sec | 22:20 |
| gun1x | and also, are the OSD nodes separate from computes? | 22:20 |
| gun1x | or on top of them? | 22:20 |
| bndzor | they are running on the controllers | 22:20 |
| bndzor | or wait | 22:20 |
| bndzor | osd = storage machines | 22:20 |
| gun1x | yes. | 22:20 |
| bndzor | yeah they run on the storage nodes | 22:21 |
| gun1x | share your playbook if you can, i'll figure from there | 22:21 |
| bndzor | so i have right now 3 controllers, 7 compute nodes and 3 storage nodes | 22:21 |
| bndzor | Filesystem Size Used Avail Use% Mounted on | 22:21 |
| bndzor | udev 32G 0 32G 0% /dev | 22:21 |
| bndzor | tmpfs 6.3G 34M 6.3G 1% /run | 22:21 |
| bndzor | /dev/sda1 147G 11G 129G 8% / | 22:21 |
| bndzor | tmpfs 32G 0 32G 0% /dev/shm | 22:21 |
| bndzor | tmpfs 5.0M 0 5.0M 0% /run/lock | 22:21 |
| bndzor | tmpfs 32G 0 32G 0% /sys/fs/cgroup | 22:21 |
| bndzor | /dev/sdb1 94M 5.4M 89M 6% /var/lib/ceph/osd/ceph-3 | 22:21 |
| bndzor | tmpfs 6.3G 0 6.3G 0% /run/user/1000 | 22:21 |
| gun1x | i am curious because i will need to do the same thing if i really get the hardware i was promised today .. | 22:21 |
| bndzor | thats how it looks like | 22:21 |
| gun1x | dude BPASTE | 22:21 |
| bndzor | whats bpaste | 22:21 |
| gun1x | https://bpaste.net | 22:22 |
| gun1x | a tool that prevents you from spaming irc | 22:22 |
| bndzor | https://bpaste.net/show/be635d155b21 | 22:22 |
| bndzor | its not 1992 anymore, nobody is gonna get flooded from 10 lines | 22:22 |
| gun1x | :D | 22:22 |
| bndzor | well, thats how it looks like | 22:22 |
| gun1x | some channels have rulles about this | 22:22 |
| bndzor | ye, but they are stuck in 1992 and still live in the basement @ mommys | 22:23 |
| gun1x | so yea, i will have to do something similar and i always scare about doing anything (even a poc) wihtout testing at home before. i don't have hardware tot est this at hoem :D | 22:23 |
| bndzor | ye well its very straight forward | 22:23 |
| gun1x | can you share your playbook? and lsblk on OSD hosts? | 22:23 |
| gun1x | i mean, how come only one storage device for ceph? | 22:23 |
| bndzor | its litteraly the same one you see on the link i gave you | 22:23 |
| bndzor | i have 3 storage nodes | 22:24 |
| bndzor | so, that paste x 3 | 22:24 |
| gun1x | that's df -h, not lsblk | 22:24 |
| bndzor | well, i guess the ceph-X | 22:24 |
| gun1x | lsblk shows all storage devices | 22:24 |
| gun1x | df -h shows only filesystems | 22:24 |
| bndzor | ah my bad | 22:24 |
| bndzor | https://bpaste.net/show/b744428e1ea6 | 22:24 |
| *** xingchao has quit IRC | 22:24 | |
| bndzor | So, sdb is a raid 0 on each host | 22:24 |
| gun1x | raid 0? | 22:25 |
| bndzor | first install was with sdb beeing a raid 1, but since ceph has the redudancy, i dont need to waste space, so i took down the machine, destroyed the raid and did a raid0 | 22:25 |
| gun1x | like ... you have 20 TB disks? what? | 22:25 |
| bndzor | that machine has 40tb in total | 22:25 |
| gun1x | yea, raid 0 is 2 disks stripe | 22:25 |
| bndzor | i used to have 20 as it was raided, but removed that since it would waste | 22:25 |
| bndzor | now il just let ceph take care of it heh | 22:25 |
| gun1x | so disks have to be 20TB disks to get 40 TB | 22:26 |
| bndzor | no ? | 22:26 |
| gun1x | bndzor: btw raids are not really supported by ceph, i guess you know that. they would rather see the raid controllers in JBOD mode and let ceph juggle multiple disks | 22:26 |
| bndzor | i have like 8 drives x 6 something | 22:26 |
| bndzor | this is a hardware raid, so its all gucci | 22:26 |
| bndzor | all works fine | 22:27 |
| bndzor | :) | 22:27 |
| gun1x | just saying | 22:27 |
| bndzor | yup | 22:27 |
| gun1x | so yea, did OSA figure by its own that sda1 is OS and did it use sdb to create sdb1 without manual intervention? | 22:27 |
| bndzor | yeah | 22:28 |
| gun1x | nice | 22:28 |
| gun1x | i guess it uses all discs by default, excluding os disk | 22:28 |
| bndzor | only, when i took down the machine to remove the raid1 and make it a raid0 i ran into issues | 22:28 |
| bndzor | so i ran the playbook, it failed, ran it again, it worked. | 22:28 |
| bndzor | did the same thing on next storage, same thing | 22:28 |
| gun1x | so this means you have no journals | 22:28 |
| bndzor | then you need to remove the old osd:s | 22:28 |
| bndzor | since it creates new | 22:29 |
| bndzor | services: | 22:29 |
| bndzor | mon: 3 daemons, quorum controller1-ceph-mon-container-22820959,controller3-ceph-mon-container-98c90950,controller2-ceph-mon-container-cb90beb9 | 22:29 |
| bndzor | mgr: controller1-ceph-mon-container-22820959(active), standbys: controller2-ceph-mon-container-cb90beb9, controller3-ceph-mon-container-98c90950 | 22:29 |
| bndzor | osd: 3 osds: 3 up, 3 in | 22:29 |
| gun1x | i see there is journal_size: 10240 within the recommanded user variables... however it doesn't let you specify a device for journal | 22:30 |
| gun1x | my bad, it has osd_scenario: collocated too | 22:32 |
| gun1x | with ceph-ansible you need to specify dedicated_devices: for journal | 22:34 |
| gun1x | and devices: for OSDs | 22:34 |
| gun1x | i wonder if this works on OSA | 22:34 |
| bndzor | oh, no idea | 22:35 |
| bndzor | but im sure logan- knows | 22:35 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Modify LXC container create to JIT create https://review.openstack.org/490976 | 22:42 |
| bndzor | oh well, gonna head home, bbl | 22:43 |
| gun1x | bndzor: i found it within files on local system, searching on github now to find it there too | 22:43 |
| gun1x | bndzor: https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-osd/defaults/main.yml#L118-L170 | 22:45 |
| *** marst has quit IRC | 22:46 | |
| *** pmannidi has joined #openstack-ansible | 23:22 | |
| *** lbragstad has quit IRC | 23:23 | |
| *** gouthamr has quit IRC | 23:34 | |
| *** cloudtoko has joined #openstack-ansible | 23:34 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Modify LXC container create to JIT create https://review.openstack.org/490976 | 23:34 |
| *** cloudtoko has quit IRC | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
