| *** fatdragon has quit IRC | 00:19 | |
| *** gus has left #openstack-ansible | 00:24 | |
| *** sawblade6 has joined #openstack-ansible | 00:28 | |
| *** sawblade_ has quit IRC | 00:30 | |
| *** odyssey4me has quit IRC | 00:35 | |
| *** odyssey4me has joined #openstack-ansible | 00:35 | |
| *** masber has joined #openstack-ansible | 00:37 | |
| *** vnogin has joined #openstack-ansible | 00:49 | |
| *** vnogin has quit IRC | 00:53 | |
| *** fatdragon1 has joined #openstack-ansible | 01:05 | |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Add CI/docs/releasenotes scaffolding and basic test https://review.openstack.org/556074 | 01:17 |
|---|---|---|
| *** fatdragon1 has quit IRC | 01:21 | |
| *** NWOutlier__ has joined #openstack-ansible | 01:28 | |
| *** fatdragon1 has joined #openstack-ansible | 02:15 | |
| fatdragon1 | on image can't be uploaded,I am seeing in cinder api log Can't connect to MySQL server on '172.29.236.100, all of this should have been setup by OSA! | 02:16 |
| fatdragon1 | where is the credential stored at? | 02:17 |
| fatdragon1 | I see many tcp 3306 established to other openstack IPs | 02:18 |
| fatdragon1 | why cinder complaing? | 02:18 |
| *** masber has quit IRC | 02:21 | |
| fatdragon1 | I see the pass in user_secrets for cinder mysql pass!!! | 02:24 |
| *** savvas has joined #openstack-ansible | 02:52 | |
| *** savvas has quit IRC | 02:56 | |
| *** geb has quit IRC | 03:04 | |
| *** geb has joined #openstack-ansible | 03:05 | |
| *** masber has joined #openstack-ansible | 03:17 | |
| *** ianychoi has joined #openstack-ansible | 03:29 | |
| cloudnull | fatdragon1: what's going on ? | 03:29 |
| cloudnull | something we can help with ? | 03:29 |
| *** dave-mccowan has quit IRC | 03:34 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Convert role to use a common systemd role(s) https://review.openstack.org/552047 | 03:49 |
| *** vnogin has joined #openstack-ansible | 03:49 | |
| *** vnogin has quit IRC | 03:54 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Add playbook to ship journals from hosts https://review.openstack.org/555982 | 04:02 |
| prometheanfire | cloudnull: have a few min to review some python? | 04:02 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Add playbook to ship journals from hosts https://review.openstack.org/555982 | 04:04 |
| cloudnull | prometheanfire: sure | 04:07 |
| cloudnull | what's up ? | 04:07 |
| *** bhujay has joined #openstack-ansible | 04:09 | |
| prometheanfire | cloudnull: https://review.openstack.org/#/c/548604/5/glean/cmd.py | 04:14 |
| openstackgerrit | Merged openstack/openstack-ansible-os_searchlight master: ansible-role-requirements: Add common systemd_service role https://review.openstack.org/552607 | 04:18 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca master: Make unstable functional test non-voting https://review.openstack.org/556250 | 04:19 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca master: ansible-role-requirements: Add common systemd_service role https://review.openstack.org/552600 | 04:19 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca master: ansible-role-requirements: Add common python_venv_build role https://review.openstack.org/556106 | 04:19 |
| *** indistylo has joined #openstack-ansible | 04:19 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Convert rsyslog to an include_task with group_vars https://review.openstack.org/556002 | 04:20 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca master: Follow the new PTI for document build https://review.openstack.org/551120 | 04:21 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca-agent master: Make unstable functional test non-voting https://review.openstack.org/556278 | 04:21 |
| cloudnull | prometheanfire it all looks good. i have no idea if it works, but it looks ok. | 04:22 |
| prometheanfire | cloudnull: passes tests :P | 04:23 |
| prometheanfire | cloudnull: can you verify the tests look like good networkd stuff? | 04:24 |
| *** NWOutlier_ has quit IRC | 04:27 | |
| *** NWOutlier__ has quit IRC | 04:27 | |
| cloudnull | looks like good networkd stuff | 04:29 |
| cloudnull | does glean split on "###" ? | 04:29 |
| *** armaan has joined #openstack-ansible | 04:29 | |
| cloudnull | for those test files? | 04:29 |
| cloudnull | ie https://review.openstack.org/#/c/548604/5/glean/tests/fixtures/test/liberty.networkd.network.out | 04:29 |
| *** armaan has quit IRC | 04:30 | |
| prometheanfire | cloudnull: for the test files, those are used as file separators/descriptors | 04:34 |
| *** udesale has joined #openstack-ansible | 04:35 | |
| cloudnull | prometheanfire: i don't see any bonding other than 802.3ad | 04:38 |
| cloudnull | but otherwise the files look good to me | 04:38 |
| prometheanfire | cloudnull: that's the only bonding mode in the tests, thanks for the ack | 04:39 |
| cloudnull | ok going to cras hout. | 04:40 |
| cloudnull | **crash | 04:40 |
| cloudnull | see you tomorrow. | 04:40 |
| prometheanfire | cloudnull: nn | 04:41 |
| *** NWOutlier__ has joined #openstack-ansible | 04:44 | |
| *** masber has quit IRC | 04:47 | |
| *** chhagarw has joined #openstack-ansible | 04:48 | |
| *** masber has joined #openstack-ansible | 04:49 | |
| *** aruns has joined #openstack-ansible | 06:13 | |
| *** indistylo has quit IRC | 06:16 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Adds variables to ceilometer, because it fails when run playbook with telemetry enabled. https://review.openstack.org/555734 | 06:20 |
| *** ericzolf has joined #openstack-ansible | 06:23 | |
| *** ianychoi_ has joined #openstack-ansible | 06:26 | |
| *** ianychoi has quit IRC | 06:29 | |
| *** sar has joined #openstack-ansible | 06:32 | |
| evrardjp | good morning | 06:35 |
| prometheanfire | moin | 06:36 |
| jafeha | good morning | 06:39 |
| *** aruns__ has joined #openstack-ansible | 06:40 | |
| *** masber has quit IRC | 06:41 | |
| *** Sha000000 has quit IRC | 06:42 | |
| *** armaan has joined #openstack-ansible | 06:42 | |
| *** aruns has quit IRC | 06:43 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Simplify docs configuration https://review.openstack.org/555880 | 07:01 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/queens: [Docs] Simplify overview page https://review.openstack.org/555371 | 07:03 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/queens: [Docs] Adapt the wording for disk requirements https://review.openstack.org/556360 | 07:07 |
| *** pcaruana has joined #openstack-ansible | 07:08 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/queens: [Docs] Include OpenStack-Ansible Manifesto https://review.openstack.org/556362 | 07:08 |
| *** Nattanon has joined #openstack-ansible | 07:08 | |
| Nattanon | Sorry guys, Do someone ever face through the problem with RabbitMQ ? | 07:11 |
| evrardjp | the problem? | 07:17 |
| evrardjp | I am not sure we have a problem with rabbitmq, what your issue? | 07:17 |
| Nattanon | In my environment message outstanding on RabbitMQ always increase won't be going down. | 07:17 |
| Nattanon | Sorry forget to press enter button T_T. | 07:18 |
| evrardjp | Nattanon: you have a very large deploy? | 07:20 |
| evrardjp | like thousands of nodes? | 07:20 |
| evrardjp | did you configure different rabbitmq servers / vhosts per service maybe? | 07:21 |
| evrardjp | or is that a hardware/networking issue? | 07:21 |
| jafeha | i'd guess some consumers are not reaching the mq or if there were no changes but increasing load an hardware sizing issue (less likely imho) | 07:22 |
| jafeha | i'd search in the logs for disconnects from the rabbitmq and look for networking issues just like evrardjp said. | 07:23 |
| Nattanon | evrardjp: you have a very large deploy? -> For the first question it not large deploy yet. | 07:26 |
| evrardjp | mattt: Where are we in this patch? I seem to see tempest page there, so I like it good enough... Is there any reason you keep it as "WIP" ? https://review.openstack.org/#/c/555621/3 | 07:27 |
| evrardjp | Nattanon: then no reason to have a rabbitmq scaling issue | 07:27 |
| evrardjp | works by default | 07:27 |
| evrardjp | Nattanon: which branch? | 07:28 |
| Nattanon | evrardjp: did you configure different rabbitmq servers / vhosts per service maybe? -> It just only one cluster of rabbit for every vhost. | 07:28 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-tests stable/queens: Allow to run tests outside openstack infra https://review.openstack.org/556375 | 07:30 |
| evrardjp | Nattanon: so, default configuration I guess. Yeah, default configuration works by default | 07:30 |
| *** electrofelix has joined #openstack-ansible | 07:31 | |
| evrardjp | check for connectivity and resource issues | 07:31 |
| Nattanon | evrardjp: We are using tag 16.0.8 , Sorry for slow replied. | 07:34 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Make docs be container agnostic https://review.openstack.org/552004 | 07:35 |
| evrardjp | Nattanon: no worries, I understand being busy :p | 07:35 |
| evrardjp | 16 is a stable branch, you should have any issue. | 07:35 |
| evrardjp | shouldn't* | 07:36 |
| evrardjp | it's not like we had a target moving or something. | 07:36 |
| evrardjp | I guess the best way we can help you is if you file a bug with the details of your environment, tell us what is happening (and how you see that happening), and what you expected. | 07:37 |
| evrardjp | Nattanon: ^ | 07:37 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Restructure inventory documentation https://review.openstack.org/552081 | 07:37 |
| Nattanon | evrardjp: Okay got it. I will try to check all environment. Do you have any specific variable or something i should care first ? | 07:41 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/queens: [Docs] Restructure inventory documentation https://review.openstack.org/556378 | 07:41 |
| *** dhellmann has quit IRC | 07:41 | |
| *** aruns has joined #openstack-ansible | 07:42 | |
| Nattanon | evrardjp: I'm quite new, also there is many things i need to care about. | 07:43 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/queens: [Docs] Restructure inventory documentation https://review.openstack.org/556378 | 07:43 |
| *** aruns__ has quit IRC | 07:44 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/queens: [Docs] Include OpenStack-Ansible Manifesto https://review.openstack.org/556362 | 07:46 |
| Nattanon | evrardjp: For more information our message outstanding for right now is about 55k message. | 07:47 |
| Nattanon | evrardjp: It happen with OSA and mirantis(for lab i setup for testing). | 07:47 |
| evrardjp | then it's maybe not a software issue, but maybe a networking / machine issue | 07:49 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Fix Doc8 issue https://review.openstack.org/556383 | 07:51 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: [WIP] Ensure doc8 is run https://review.openstack.org/556384 | 07:52 |
| Nattanon | evrardjp: Thank you so much. If i can solve this problem i will report back what happening with us. | 07:53 |
| *** shardy has joined #openstack-ansible | 07:55 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Use Ansible 2.5 https://review.openstack.org/556388 | 07:59 |
| evrardjp | Nattanon: as I said, if you want the best help, it's probably easier to file a bug. More ppl can have a look at the bug and ask questions | 07:59 |
| evrardjp | instead of repeating yourself over and over :) | 07:59 |
| *** masber has joined #openstack-ansible | 08:09 | |
| *** Sha000000 has joined #openstack-ansible | 08:11 | |
| *** alefra has joined #openstack-ansible | 08:13 | |
| *** radeks has joined #openstack-ansible | 08:13 | |
| hwoarang | good morning | 08:17 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_monasca master: Updated from global requirements https://review.openstack.org/551568 | 08:17 |
| hwoarang | an ansible question for you. What's the best way to have something like foobar: {{ foo | ternary('bar', omit) }} | 08:18 |
| hwoarang | so what i want is to pass foobar:'bar' if foo is enabled, otherwise ignore foobar. The 'omit' there simply makes foobar:' ' which is not what i want | 08:19 |
| evrardjp | good morning hwoarang | 08:20 |
| evrardjp | hwoarang: where ? | 08:20 |
| evrardjp | is it in a task, inside the module? | 08:20 |
| evrardjp | in a conditional? | 08:20 |
| hwoarang | evrardjp: as sorry, 'foobar' is a module param | 08:21 |
| evrardjp | foo is a bool then? | 08:21 |
| evrardjp | should be fine | 08:21 |
| hwoarang | yeah it's bool. doesn't work though because if 'foo' is false it passes foobar: '' | 08:22 |
| evrardjp | the module argument should ofc be optional | 08:22 |
| hwoarang | it is | 08:22 |
| evrardjp | oh yes i understand | 08:23 |
| evrardjp | tricky | 08:23 |
| evrardjp | can't you have a conditional? | 08:23 |
| hwoarang | i am trying to be smart :p | 08:23 |
| evrardjp | or a list? | 08:23 |
| evrardjp | (I mean a lookup) | 08:24 |
| evrardjp | if you show the task somewhere I might help you | 08:24 |
| hwoarang | evrardjp: https://review.openstack.org/#/c/555818/1/playbooks/roles/bifrost-create-vm-nodes/tasks/prepare_libvirt.yml | 08:24 |
| hwoarang | this passes virtualenv: '' if enable_venv is false | 08:24 |
| evrardjp | let me try real quick | 08:25 |
| *** NWOutlier__ has quit IRC | 08:27 | |
| evrardjp | hwoarang: works for me | 08:30 |
| hwoarang | hmm | 08:31 |
| hwoarang | strange | 08:31 |
| evrardjp | http://paste.ubuntu.com/p/GPVxZfX8Zw/ | 08:32 |
| hwoarang | in if drop that line, what you see in the logs is "virtualenv: null," but with that line it's "virtualenv: ''", | 08:32 |
| evrardjp | http://paste.ubuntu.com/p/XkTsv8FbDz/ | 08:32 |
| evrardjp | http://paste.ubuntu.com/p/Cbbv8Gd5FD/ | 08:32 |
| evrardjp | http://paste.ubuntu.com/p/DYpvJqTF5Q/ | 08:33 |
| evrardjp | as you can see in the latest paste, the virtualenv is passed null | 08:33 |
| hwoarang | interesting... | 08:33 |
| hwoarang | why did you filter it with | bool since it's already set to true/false? | 08:34 |
| evrardjp | It's not | 08:34 |
| *** armaan has quit IRC | 08:34 | |
| evrardjp | true/false is a pseudo bool | 08:34 |
| evrardjp | True and False are bools | 08:34 |
| evrardjp | IIRC | 08:34 |
| evrardjp | let me try that without the filter | 08:34 |
| hwoarang | i will check again maybe i miss something :( | 08:35 |
| hwoarang | thank you very much for checking! | 08:35 |
| evrardjp | still works | 08:35 |
| evrardjp | but jinja bools have to be True/False IIRC | 08:35 |
| evrardjp | I have to double check | 08:35 |
| evrardjp | hwoarang: my bad | 08:37 |
| evrardjp | it's all lowercase | 08:37 |
| evrardjp | http://jinja.pocoo.org/docs/2.10/templates/#literals | 08:37 |
| evrardjp | (see note) | 08:37 |
| evrardjp | but next to that you see in the docs things like this: http://jinja.pocoo.org/docs/2.10/templates/#whitespace-control | 08:38 |
| evrardjp | (see first code example with if True ) | 08:39 |
| hwoarang | i see | 08:39 |
| evrardjp | anyway, there is maybe something wrong in your var? | 08:39 |
| evrardjp | let me check my ansible ver | 08:40 |
| evrardjp | 2.4.3 | 08:40 |
| evrardjp | trying on 2.5 | 08:41 |
| *** vnogin has joined #openstack-ansible | 08:42 | |
| evrardjp | works on ansible on 2.3 and 2.5 | 08:43 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: [WIP] Ensure doc8 is run https://review.openstack.org/556384 | 08:45 |
| hwoarang | see what i mean http://logs.openstack.org/18/555818/1/check/bifrost-integration-tinyipa/ee23765/job-output.txt.gz#_2018-03-26_08_04_39_577458 | 08:45 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Ensure doc8 is run https://review.openstack.org/556384 | 08:45 |
| hwoarang | anyway i will figure it out | 08:45 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Ensure doc8 is run https://review.openstack.org/556384 | 08:45 |
| hwoarang | i still thinks we are in a virtualenv | 08:46 |
| evrardjp | yeah that's different than my output indeed | 08:47 |
| *** armaan has joined #openstack-ansible | 08:47 | |
| evrardjp | hwoarang: could you vote on https://review.openstack.org/#/c/556383/1 ? | 08:47 |
| *** bjolo has joined #openstack-ansible | 08:49 | |
| *** armaan has quit IRC | 08:52 | |
| evrardjp | hwoarang: can you add a -vvv ? | 08:52 |
| evrardjp | oh no, not needed | 08:52 |
| evrardjp | btw previous task can have a | to simplify readability | 08:54 |
| evrardjp | no need for the " anymore there | 08:54 |
| *** masber has quit IRC | 08:54 | |
| evrardjp | and should probably tell to use bash | 08:55 |
| evrardjp | :p | 08:55 |
| evrardjp | omg | 08:55 |
| evrardjp | I am already doing scope creep. | 08:55 |
| openstackgerrit | wangqi proposed openstack/openstack-ansible-os_octavia master: fix a typo https://review.openstack.org/556448 | 08:55 |
| evrardjp | or alternatively the task even before that could get the permissions set, in a recursive way | 08:56 |
| *** masber has joined #openstack-ansible | 08:57 | |
| evrardjp | let me patch that quick | 09:00 |
| *** masber has quit IRC | 09:02 | |
| *** chhagarw has quit IRC | 09:02 | |
| *** chhavi__ has joined #openstack-ansible | 09:02 | |
| evrardjp | https://review.openstack.org/#/c/556449/1/playbooks/roles/bifrost-create-vm-nodes/tasks/prepare_libvirt.yml | 09:03 |
| evrardjp | :) | 09:03 |
| evrardjp | I feel better | 09:03 |
| *** masber has joined #openstack-ansible | 09:03 | |
| hwoarang | lol | 09:05 |
| *** armaan has joined #openstack-ansible | 09:05 | |
| openstackgerrit | wangqi proposed openstack/ansible-config_template master: update url in readme file https://review.openstack.org/556452 | 09:07 |
| *** aruns__ has joined #openstack-ansible | 09:07 | |
| hwoarang | hmm it worked now | 09:07 |
| hwoarang | maybe the |bool did the trick | 09:07 |
| hwoarang | evrardjp: thanks again for looking at it | 09:08 |
| * hwoarang goes back to some systemd problems | 09:08 | |
| evrardjp | haha | 09:08 |
| evrardjp | we'll never know. Maybe there was some bad text in some of your variables. | 09:08 |
| evrardjp | or depending on the python version | 09:09 |
| evrardjp | you might have a unicode string that wasn't properly parsed | 09:09 |
| evrardjp | who knows | 09:09 |
| evrardjp | is there any way I can help you there? | 09:09 |
| *** aruns has quit IRC | 09:09 | |
| *** djdule has joined #openstack-ansible | 09:10 | |
| djdule | Hi | 09:10 |
| djdule | I do have networking problem that I cannot solve for few days and I really would appreciate any help | 09:10 |
| djdule | I installed Pike using OpenStack ansible | 09:10 |
| djdule | everything is up and running | 09:10 |
| *** udesale_ has joined #openstack-ansible | 09:10 | |
| djdule | and looks OK | 09:10 |
| *** udesale has quit IRC | 09:10 | |
| djdule | it is not AIO, I do have separate single controller and compute nodes | 09:11 |
| djdule | so networking works, except connecting to rest of the world | 09:11 |
| djdule | I tried creating external networks, it is being created | 09:11 |
| djdule | but from qrouter name space I cannot pint anything | 09:11 |
| djdule | and I am stuck, i do not know where to look | 09:12 |
| djdule | any ideas please ? | 09:12 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Ensure doc8 is run https://review.openstack.org/556384 | 09:12 |
| evrardjp | hwoarang: no space left on device issues appearing again | 09:15 |
| evrardjp | :/ | 09:15 |
| hwoarang | :( | 09:15 |
| evrardjp | queens now | 09:15 |
| *** udesale_ has quit IRC | 09:15 | |
| evrardjp | do you know what has been backported or not? | 09:15 |
| *** udesale has joined #openstack-ansible | 09:16 | |
| evrardjp | else I can check with odyssey4me 's later | 09:16 |
| *** fatdragon1 has quit IRC | 09:19 | |
| hwoarang | i can't remember righ tnow | 09:20 |
| hwoarang | i think everything went to queens too | 09:20 |
| evrardjp | ok | 09:21 |
| evrardjp | testing on my machines with less space than nodepool nodes | 09:21 |
| evrardjp | see how it goes | 09:21 |
| evrardjp | then I will try to find the disk space leak :p | 09:21 |
| evrardjp | djdule: mmm | 09:24 |
| evrardjp | did you follow the troubleshooting page? | 09:25 |
| evrardjp | djdule: it's hard to know what's going on if you didn't follow troubleshooting page | 09:25 |
| *** admin0 has joined #openstack-ansible | 09:29 | |
| djdule | troubleshooting page helps only in instance to instance connectivity | 09:29 |
| djdule | and that one works | 09:29 |
| djdule | also host to host connectivity works | 09:29 |
| djdule | only thing that is not working is connecting instances to external network | 09:30 |
| *** dhellmann has joined #openstack-ansible | 09:31 | |
| admin0 | morning | 09:34 |
| admin0 | djdule, for that, are you able to ping the router ? | 09:34 |
| djdule | to ping the router from where ? | 09:35 |
| djdule | when I create provider net and add router to it, I cannot ping default gateway, and that is my problem | 09:35 |
| djdule | I use containerized version, and this is linuxbridge_agent.ini relevant line: | 09:36 |
| djdule | physical_interface_mappings = flat:eth12,vlan:eth11 | 09:36 |
| djdule | so when i: | 09:36 |
| djdule | openstack network create --external --share --provider-physical-network flat --provider-network-type flat provider1 | 09:36 |
| djdule | and all other things (create subnet, gateway, etc) | 09:37 |
| djdule | i can see them in namespaces on controller | 09:37 |
| djdule | but I cannot ping default gatewa from router namespace | 09:37 |
| djdule | and I suppose that is the reason why connectivity is not working | 09:37 |
| djdule | but I do not know why I cannot ping default gateway and I am stuck here for day | 09:37 |
| djdule | s | 09:37 |
| *** masber has quit IRC | 09:38 | |
| djdule | any ideas where to look would help | 09:39 |
| evrardjp | it's been so long I haven't tried this -- it looks okay though. | 09:43 |
| evrardjp | djdule: how is wired your eth12? | 09:44 |
| evrardjp | that's what you should I guess follow first | 09:44 |
| djdule | I do not know, openstack-ansible did set it up I guess | 09:44 |
| evrardjp | we set up the eth12, and we connect to a bridge of yours, but you need to setup the bridge yourself | 09:44 |
| evrardjp | what's in your openstack_user_config ? | 09:45 |
| *** armaan_ has joined #openstack-ansible | 09:45 | |
| djdule | - network: container_bridge: "br-vlan" container_type: "veth" container_interface: "eth12" type: "flat" net_name: "flat" group_binds: - neutron_linuxbridge_agent | 09:46 |
| djdule | that is relevant part I suppose | 09:46 |
| evrardjp | and br-vlan is on your neutron nodes? | 09:48 |
| evrardjp | (infra by default I guess) | 09:48 |
| djdule | there is vr-vlan on controller node, where container with agent is running | 09:48 |
| *** armaan has quit IRC | 09:48 | |
| evrardjp | try injecting traffic and see how it goes with a tcpdump | 09:49 |
| evrardjp | that's what I'd do | 09:49 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_monasca stable/pike: Make unstable functional test non-voting https://review.openstack.org/556467 | 09:54 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_monasca stable/pike: Make unstable functional test non-voting https://review.openstack.org/556467 | 09:55 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_monasca stable/queens: Make unstable functional test non-voting https://review.openstack.org/556469 | 09:55 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_monasca stable/queens: Make unstable functional test non-voting https://review.openstack.org/556469 | 09:55 |
| djdule | ok, just to confirm one thing | 09:56 |
| djdule | external connectivity should go through br-vlan interface | 09:56 |
| djdule | and I should have externally flat network | 09:57 |
| djdule | for that to work ? | 09:57 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_monasca stable/queens: Updated from OpenStack Ansible Tests https://review.openstack.org/544927 | 09:58 |
| djdule | or in other words, what interface should I use to connect my openstack, that is installed using openstack-ansible, with rest of the world | 09:59 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_monasca stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516299 | 09:59 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_monasca stable/pike: Zuul: Remove project name https://review.openstack.org/541854 | 09:59 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_monasca stable/ocata: Initial OSA zuul v3 role jobs https://review.openstack.org/516646 | 10:01 |
| evrardjp | djdule: it all depends on your configuration | 10:03 |
| evrardjp | check at the multi node aio for example | 10:03 |
| evrardjp | djdule: https://docs.openstack.org/openstack-ansible/latest/admin/openstack-firstrun.html#configure-your-first-networks | 10:05 |
| evrardjp | djdule: which should point you to many tasks, which take these vars: https://github.com/openstack/openstack-ansible-ops/blob/master/multi-node-aio/playbooks/vars/openstack-service-config.yml | 10:07 |
| *** ibmko has quit IRC | 10:07 | |
| evrardjp | FYI that's the o_u_c that goes with it | 10:07 |
| evrardjp | https://github.com/openstack/openstack-ansible-ops/blob/master/multi-node-aio/playbooks/osa/openstack_user_config.yml | 10:07 |
| evrardjp | if you see something missing... | 10:07 |
| admin0 | djdule, first nova show UUID .. check in which compute node it is .. then ssh to that compute node, then virsh dumpxml UUID ... and note down the network interface names . then brctl show | less .. or ovs-vsctl show | less and note down in which bridges they are .. then you will know how the VM is wired logically and via what interface its supposed to leave the compute node .. then tcpdump is your friend | 10:08 |
| odyssey4me | evrardjp hwoarang well, https://review.openstack.org/#/q/I8ddc9931cbca5db8c1a411fd2581b34763179d01 has been ported, but I'm not sure if the role SHA's are using it yet | 10:08 |
| evrardjp | odyssey4me: thanks for the info | 10:09 |
| evrardjp | odyssey4me: interesting story, I have LESS disk than infra, and it passed my testing | 10:10 |
| evrardjp | well it's still busy but it's further than repo build | 10:10 |
| evrardjp | so we have something in infra or that's at hidden | 10:10 |
| evrardjp | but I will check the sha | 10:10 |
| odyssey4me | well, what's your disk layout - one single disk, or two disks? | 10:10 |
| evrardjp | one disk | 10:11 |
| odyssey4me | yeah, that seems to work fine - it's the two disk layout that seems to be when it flakes out | 10:11 |
| evrardjp | oh yeah I remember you fixed that | 10:11 |
| odyssey4me | you can try using a performance1-8 instance in pub cloud | 10:11 |
| evrardjp | oh yeah I should try this | 10:11 |
| odyssey4me | now that ansible-role-python_venv_build has been imported, don't forget to clone it down and add it to your review dashboard | 10:12 |
| odyssey4me | related: https://review.openstack.org/556121 | 10:13 |
| evrardjp | yes I have to change my review dashboards, it's far from ready | 10:13 |
| evrardjp | odyssey4me: I don't know if you've seen that I added as reviewer on a patch that goes into changes of the venv build process | 10:14 |
| evrardjp | I can't remember the link | 10:14 |
| evrardjp | oh yeah you've reviewed | 10:15 |
| evrardjp | thanks | 10:15 |
| *** Sha000000 has quit IRC | 10:17 | |
| odyssey4me | yup | 10:18 |
| odyssey4me | and the new role doesn't have the same issue | 10:19 |
| odyssey4me | this is a very strange ansible-lint failure: http://logs.openstack.org/66/542466/1/check/openstack-ansible-linters/03376ea/job-output.txt.gz#_2018-03-25_19_09_42_223511 from https://review.openstack.org/542466 | 10:19 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/queens: Urgent bump of lxc_hosts https://review.openstack.org/556474 | 10:19 |
| evrardjp | odyssey4me: oh fun | 10:21 |
| evrardjp | installing ansible 2.4 | 10:21 |
| evrardjp | it's because ara's not installing with constraints or something | 10:21 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca-agent master: Updated from global requirements https://review.openstack.org/552218 | 10:22 |
| evrardjp | odyssey4me: I guess a few patches I did in the past didn't land there? | 10:22 |
| odyssey4me | ah, ok lemme hunt that down and fix it | 10:25 |
| evrardjp | let me track that | 10:25 |
| evrardjp | ahah | 10:25 |
| evrardjp | I was busy tracking it | 10:25 |
| evrardjp | but I have the impression it should be from tests repo and therefore should not have a problem | 10:26 |
| odyssey4me | yup | 10:27 |
| *** armaan_ has quit IRC | 10:30 | |
| *** armaan has joined #openstack-ansible | 10:30 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/pike: ARA: Prepare the ARA plugin for all the tests https://review.openstack.org/556475 | 10:32 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/pike: ARA: Prepare the ARA plugin for all the tests https://review.openstack.org/556475 | 10:33 |
| evrardjp | odyssey4me: found it | 10:33 |
| evrardjp | we should dig too deep | 10:33 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca master: Updated from OpenStack Ansible Tests https://review.openstack.org/544653 | 10:34 |
| evrardjp | (I was llike following all the scripts) | 10:34 |
| evrardjp | where at the end | 10:34 |
| evrardjp | https://github.com/openstack/openstack-ansible-os_zaqar/blob/stable/pike/tox.ini#L56-L60 | 10:34 |
| evrardjp | see the tox action here: https://github.com/openstack/openstack-ansible-os_zaqar/blob/stable/pike/tox.ini#L101 | 10:34 |
| odyssey4me | ah, of course! | 10:34 |
| odyssey4me | ok cool - I'll patch that up | 10:34 |
| evrardjp | I can patch this I found it :) | 10:35 |
| evrardjp | but I am trying to find out why we are out of sync first | 10:35 |
| odyssey4me | sure thing - go for it! | 10:35 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca master: Zuul: Remove project name https://review.openstack.org/540460 | 10:35 |
| evrardjp | (I know we do not receive updates for tox.ini, but I am curious what we missed else) | 10:36 |
| *** ibmko has joined #openstack-ansible | 10:38 | |
| odyssey4me | evrardjp could you vote these through please? https://review.openstack.org/#/q/I95f084c26bc3c8c00a326a45d3aa94a0ee539df7 | 10:38 |
| adrianreza | Hi evrardjp, I'following example test configuration from docs, and get galera container failed with connection refused | 10:39 |
| adrianreza | I tried on infra host root@infra01:/opt/openstack-ansible/playbooks# wget http://192.168.37.11:8181/os-releases/16.0.10/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt | 10:39 |
| adrianreza | and got Connecting to 192.168.37.11:8181... failed: Connection refused. | 10:39 |
| *** aruns__ has quit IRC | 10:40 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_zaqar master: Fix tests repo cloning https://review.openstack.org/556477 | 10:40 |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Ensure python/paths are updated if folder was not present https://review.openstack.org/556121 | 10:40 |
| evrardjp | odyssey4me: ^ | 10:40 |
| evrardjp | odyssey4me: sure | 10:40 |
| admin0 | adrianreza, check haproxy | 10:40 |
| adrianreza | admon0, is it internal_lb_vip_address and external_lb_vip_address related? | 10:40 |
| admin0 | yes | 10:40 |
| evrardjp | odyssey4me: convenient little thing | 10:40 |
| evrardjp | odyssey4me: +w if you're okay with the zaqar fix | 10:41 |
| adrianreza | admin0, here is my config https://paste.ubuntu.com/p/T6dzXrpfwn/ I put same address for internal_lb_vip_address and external_lb_vip_address since it only single infra node | 10:41 |
| evrardjp | this way we can backport fast and iterate | 10:41 |
| odyssey4me | thanks - will see if it passes the test before doing that | 10:41 |
| admin0 | adrianreza, is that IP up, is haproxy listening to that, and when you view the haproxy settings, it will show where 8181 is pointing to internally .. curl there as well directly | 10:42 |
| *** aruns__ has joined #openstack-ansible | 10:43 | |
| adrianreza | admin0, yes IP is up and curl get refused also | 10:43 |
| admin0 | :D | 10:43 |
| admin0 | so go to haproxy and see where it lands | 10:43 |
| admin0 | and then fix it in the source | 10:43 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove stable/pike: Make unstable functional test non-voting https://review.openstack.org/556478 | 10:43 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove stable/ocata: Make unstable functional test non-voting https://review.openstack.org/556479 | 10:44 |
| Nattanon | evrardjp: Does notifications.info message is needed or i can somehow close this notifications in nova.conf | 10:44 |
| *** aruns has joined #openstack-ansible | 10:44 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove stable/ocata: Make unstable functional test non-voting https://review.openstack.org/556479 | 10:44 |
| evrardjp | Nattanon: I have no idea. sorry | 10:45 |
| evrardjp | adrianreza: networking issue. | 10:45 |
| *** aruns has quit IRC | 10:45 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove stable/pike: Zuul: Remove project name https://review.openstack.org/542811 | 10:45 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: test-ansible-deps: Bump to latest ansible and ansible-lint https://review.openstack.org/556480 | 10:45 |
| evrardjp | adrianreza: I see you're in good hands with admin0 :) | 10:45 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove stable/pike: Replace virtualenv-tools by a script https://review.openstack.org/545344 | 10:46 |
| *** aruns has joined #openstack-ansible | 10:46 | |
| admin0 | evrardjp, glad to help | 10:46 |
| Nattanon | evrardjp: No problems. :D | 10:46 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove stable/pike: Remove Trove's support for Heat https://review.openstack.org/550905 | 10:46 |
| adrianreza | evrardjp, thanks | 10:46 |
| adrianreza | admin0, http://paste.openstack.org/show/713906/ I checked haproxy.cfg | 10:46 |
| evrardjp | hwoarang: interesting we were out of sync. That's not great. | 10:46 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove stable/ocata: Zuul: Remove project name https://review.openstack.org/541301 | 10:47 |
| evrardjp | hwoarang: and on top of that I just posted a commit to bump ansible to 2.5 | 10:47 |
| admin0 | different ntp servers :D ? | 10:47 |
| hwoarang | yeah it's easy to miss that | 10:47 |
| *** aruns__ has quit IRC | 10:47 | |
| evrardjp | bumping the two together is a good way to avoid these. | 10:47 |
| evrardjp | But refactoring the tests is even better. We discussed with odyssey4me about that in the past, and I'd like to give it a shot when I can | 10:48 |
| odyssey4me | evrardjp hwoarang I did some work on the w/end to simplify how the role tests are executed | 10:49 |
| adrianreza | admin0, no I use same ntp server on all nodes | 10:49 |
| odyssey4me | My aim is to remove tox from all lint/functional testing that we do and to try to use ansible more - and bash less. | 10:50 |
| admin0 | adrianreza, sorry .. that was a joke for evrardjp and hwoarang when they said they were out of sync | 10:50 |
| odyssey4me | I'll try and work up a patch to show the work so far. | 10:50 |
| admin0 | adrianreza, 192.168.238.148:8181 -- this works ? | 10:51 |
| admin0 | curl 192.168.238.148:8181 ? | 10:51 |
| admin0 | and of course it will not work right :) | 10:52 |
| admin0 | can you see your mistake ? | 10:52 |
| adrianreza | admin0, yes it works https://paste.ubuntu.com/p/Qg8ZCRpfMs/ | 10:52 |
| admin0 | from wher ? | 10:53 |
| admin0 | from all containers ? | 10:53 |
| adrianreza | admin0, from infra node | 10:53 |
| admin0 | adrianreza, the issue is that your internal lb vip address is not in the mgmt range | 10:53 |
| admin0 | one of the internal mgmt range | 10:53 |
| admin0 | so it depends on the routing of the container to reach it | 10:53 |
| admin0 | and that is where the issue lies | 10:54 |
| admin0 | why not just specify an internal IP range of the mgmt ip there | 10:54 |
| admin0 | and just a comment .. there is no need to change the IPs specified .. 172.29 to 192.168 .. as they are all internal .. and free .. by changing, to help you, i have to look into your ip config all the time to see why/how its mapped | 10:55 |
| admin0 | and by changing this also, you are creating an extra burden for the next person who works in your office to take over becaue now he cannto also follow the documentaiton directly and have to lookup yoru own internal mapping | 10:55 |
| admin0 | doh .. my typing ! | 10:57 |
| adrianreza | admin0, here is my situation, those range 192.168 I put it on bond0 directly | 10:58 |
| adrianreza | https://paste.ubuntu.com/p/T6dzXrpfwn/ here is my openstack_user_config | 10:59 |
| admin0 | adrianreza, if you use both LB vip address as 192.168.238.11 .. why would this not work for you ? | 10:59 |
| admin0 | or use .11 for internal and keep the 37 for external | 10:59 |
| *** djdule has quit IRC | 11:00 | |
| adrianreza | admin0. here my interface config http://paste.ubuntu.com/p/DGcKdXc5JG/ | 11:00 |
| openstackgerrit | Merged openstack/openstack-ansible-os_zaqar master: Fix tests repo cloning https://review.openstack.org/556477 | 11:01 |
| adrianreza | admin0, does the mgmt cidr need access to internet? | 11:01 |
| admin0 | adrianreza, your internal vip should be: 192.168.238.11 | 11:01 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: [WIP] Remove tox from lint/functional testing https://review.openstack.org/556484 | 11:01 |
| admin0 | what is the variables file ? | 11:01 |
| admin0 | paste your variables conf as well | 11:01 |
| adrianreza | as in my case, only bond0 has external access to internet | 11:02 |
| admin0 | external 192.168.37.11 is OK | 11:02 |
| adrianreza | admin0 oke wait a sec | 11:02 |
| admin0 | internal must be 192.168.238.11 | 11:02 |
| adrianreza | admin0 here is user variables | 11:03 |
| adrianreza | http://paste.ubuntu.com/p/76hSNcBJhR/ | 11:03 |
| admin0 | remove the # from the haproxy lines | 11:03 |
| admin0 | ## Load Balancer Configuration (haproxy/keepalived) | 11:03 |
| admin0 | and then rerun the haproxy playbook | 11:04 |
| admin0 | and then curl again .. should work and let you continue | 11:04 |
| adrianreza | https://www.irccloud.com/pastebin/YNDqysLJ/ | 11:04 |
| adrianreza | admin0 so it should like ˆ | 11:05 |
| admin0 | yes | 11:05 |
| adrianreza | admin0 okay thanks I'll try it | 11:05 |
| evrardjp | odyssey4me: I guess except the docs jobs, just for the rest? | 11:07 |
| *** Sha0000 has joined #openstack-ansible | 11:10 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: [WIP] Remove tox from lint/functional testing https://review.openstack.org/556484 | 11:11 |
| admin0 | adrianreza, you are to run setup haproxy playbook | 11:11 |
| admin0 | and then curl | 11:11 |
| admin0 | it should fix the mappings | 11:11 |
| admin0 | and if the curl works, rerun setup infra again | 11:11 |
| admin0 | becuase of the ip change, the apt proxy needs to be updated in every container | 11:12 |
| odyssey4me | evrardjp well, I don't know if the docs jobs actually require any tox - I have the impression that they actually don't - but given you'd want to test locally, tox could stay for that, yes | 11:12 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_zaqar stable/queens: Fix tests repo cloning https://review.openstack.org/556488 | 11:12 |
| *** haad1 has joined #openstack-ansible | 11:12 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_zaqar stable/pike: Fix tests repo cloning https://review.openstack.org/556489 | 11:13 |
| evrardjp | I meant our docs linters, but I guess you counted that into linters :) | 11:15 |
| *** bhujay has quit IRC | 11:15 | |
| evrardjp | odyssey4me: I've +w the zaqar tox changes, so you can continue your spree! | 11:16 |
| odyssey4me | ta | 11:16 |
| *** bhujay has joined #openstack-ansible | 11:16 | |
| evrardjp | odyssey4me: mmm, do you know why the aio_metal job didn't run there: https://review.openstack.org/#/c/556388/ ? | 11:18 |
| odyssey4me | nope, that's odd | 11:19 |
| evrardjp | yup. | 11:20 |
| evrardjp | and it's for all the jobs we have that. | 11:21 |
| odyssey4me | what do you mean? | 11:21 |
| evrardjp | I checked all the running builds in zuul for master branch in the integrated repo | 11:22 |
| evrardjp | they all not run metal | 11:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_zaqar stable/pike: Zuul: Remove project name https://review.openstack.org/542466 | 11:22 |
| evrardjp | 13 march is the last time they ran | 11:24 |
| odyssey4me | did we break the job config somehow? | 11:24 |
| evrardjp | yes | 11:25 |
| evrardjp | I think | 11:25 |
| evrardjp | and I think I know | 11:25 |
| evrardjp | last to merge before this stopped is: Merge "Remove meta job map for aio->aio_lxc" | 11:26 |
| evrardjp | so I guess I will check if I can replace with aio_lxc or something like that | 11:26 |
| evrardjp | well | 11:26 |
| evrardjp | I will check all of that | 11:26 |
| evrardjp | it can also come from zuul side | 11:26 |
| *** ansmith has quit IRC | 11:27 | |
| *** shardy has quit IRC | 11:30 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_zaqar stable/queens: Fix tests repo cloning https://review.openstack.org/556488 | 11:31 |
| openstackgerrit | Merged openstack/openstack-ansible-os_zaqar stable/pike: Fix tests repo cloning https://review.openstack.org/556489 | 11:31 |
| evrardjp | odyssey4me: FYI, the docs testing of doc8 is behaving inappropriately | 11:32 |
| evrardjp | https://review.openstack.org/#/c/556384/1 | 11:33 |
| evrardjp | the patchset 1 is showing a failure as green. | 11:34 |
| evrardjp | I think our tox.ini is starting to get ignored. | 11:34 |
| *** shardy has joined #openstack-ansible | 11:36 | |
| evrardjp | so we need to put that into linters | 11:36 |
| evrardjp | and we can indeed probably get rid of the whole tox in the future | 11:37 |
| openstackgerrit | Merged openstack/openstack-ansible stable/ocata: Fix typo and misleading task name for nova-consoleauth https://review.openstack.org/553823 | 11:38 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/ocata: Add non-voting ubuntu-xenial functional test job https://review.openstack.org/556281 | 11:42 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/queens: Add non-voting ubuntu-xenial functional test job https://review.openstack.org/556279 | 11:42 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/pike: Add non-voting ubuntu-xenial functional test job https://review.openstack.org/556280 | 11:42 |
| *** savvas has joined #openstack-ansible | 11:53 | |
| hwoarang | mnaser: blueprint is ready https://blueprints.launchpad.net/openstack-ansible/+spec/openstack-distribution-packages i will draft the spec now | 11:54 |
| *** armaan has quit IRC | 11:57 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: [WIP] Remove tox from lint/functional testing https://review.openstack.org/556484 | 11:57 |
| *** armaan has joined #openstack-ansible | 11:57 | |
| dcdamien | evrardjp: I've done steps you recommended. I've checked dynamic inventory generation on every single file in env.d. It follows that this -> http://paste.openstack.org/show/708047/ (env.d/haproxy.yml) breaks things. | 12:04 |
| *** dave-mccowan has joined #openstack-ansible | 12:04 | |
| *** ibmko has quit IRC | 12:06 | |
| evrardjp | dcdamien: oh fun stuff, the same issue as logan! | 12:07 |
| evrardjp | dcdamien: for your use case do you need that? | 12:08 |
| dcdamien | yep :D | 12:08 |
| evrardjp | it's not recommended. | 12:08 |
| evrardjp | unless you know what you're doing :p | 12:08 |
| evrardjp | but ok | 12:09 |
| evrardjp | any other env.d override is good dcdamien ? | 12:09 |
| evrardjp | or it's the only override you have? | 12:09 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: [WIP] Remove tox from lint/functional testing https://review.openstack.org/556484 | 12:10 |
| dcdamien | I have 3 another overrides but they don't cause problems | 12:11 |
| evrardjp | odyssey4me: thanks for pushing that up so I can put my eyes on it | 12:11 |
| evrardjp | :) | 12:11 |
| evrardjp | good you used the git review! | 12:12 |
| odyssey4me | yup, it makes sense to use it as a key | 12:15 |
| *** chhavi__ has quit IRC | 12:16 | |
| evrardjp | I also like the fact you're using run_tests.sh as minimum thing, and we can call things from it. So having it a router is very nice | 12:16 |
| evrardjp | (I guess it's not 100% like that, but we can evolve towards that) | 12:16 |
| dcdamien | evrardjp: The most interesting part is if I get whole env file from `inventory/end.d` copy it to `/etc/openstack_deploy/env.d` and change is_metal to false it fails on dynamic inventory too. It looks like it isn't caused by partial override. | 12:17 |
| odyssey4me | yep, run_tests should just clone the tests repo - then everything else should be in the tests repo | 12:17 |
| odyssey4me | I think I'm going to break this up into smaller patches. | 12:17 |
| *** armaan has quit IRC | 12:18 | |
| *** armaan has joined #openstack-ansible | 12:18 | |
| evrardjp | odyssey4me: we can discuss it up if you want | 12:19 |
| *** sawblade6 has quit IRC | 12:19 | |
| evrardjp | dcdamien: yeah I have the feeling there is something wrong with the default haproxy file | 12:19 |
| *** sawblade6 has joined #openstack-ansible | 12:20 | |
| evrardjp | but I have so many other things I'd like to fix first ;( | 12:20 |
| dcdamien | Ok. I'll check this file out. | 12:20 |
| openstackgerrit | Merged openstack/openstack-ansible-os_zaqar stable/pike: Zuul: Remove project name https://review.openstack.org/542466 | 12:22 |
| *** ibmko has joined #openstack-ansible | 12:23 | |
| *** Nattanon has quit IRC | 12:23 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: ansible-role-requirements: Use systemd_service role from openstack namespace https://review.openstack.org/556237 | 12:30 |
| admin0 | is queens stable now ? | 12:30 |
| admin0 | when will it be out of rc2 ? | 12:31 |
| *** aruns__ has joined #openstack-ansible | 12:33 | |
| *** savvas_ has joined #openstack-ansible | 12:33 | |
| *** aruns has quit IRC | 12:35 | |
| *** kstev has joined #openstack-ansible | 12:35 | |
| evrardjp | admin0: first release has been cut | 12:35 |
| evrardjp | tagged* | 12:36 |
| evrardjp | I will bump the roles and all end of the week | 12:36 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca master: Updated from global requirements https://review.openstack.org/551568 | 12:36 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca stable/pike: Make unstable functional test non-voting https://review.openstack.org/556467 | 12:36 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca stable/pike: Zuul: Remove project name https://review.openstack.org/541854 | 12:36 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516299 | 12:36 |
| evrardjp | so you'll have a .1 | 12:36 |
| admin0 | is there a blocker to not have it sooner :) | 12:36 |
| evrardjp | we are on the release train now | 12:37 |
| evrardjp | every two weeks a new release | 12:37 |
| *** savvas has quit IRC | 12:37 | |
| admin0 | if i go with Rc2 now , i guess i can easily upgrade .. unless you say otherwise | 12:37 |
| evrardjp | you can go with .0 ? | 12:37 |
| admin0 | ok | 12:38 |
| admin0 | i can | 12:38 |
| evrardjp | it's the same :) | 12:38 |
| admin0 | is stable/queens also the same as 17.0.0 ? | 12:38 |
| evrardjp | but there is a bug for multi node that's on its way back to queens | 12:38 |
| evrardjp | stable/queens is the 17.* branch | 12:39 |
| evrardjp | 17.0.0.0rc2 is the same as the first 17.0.0 tag | 12:39 |
| evrardjp | as the first tag of this branch, 17.0.0* | 12:39 |
| evrardjp | better phrased that way | 12:39 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca stable/queens: Make unstable functional test non-voting https://review.openstack.org/556469 | 12:39 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca stable/queens: Updated from OpenStack Ansible Tests https://review.openstack.org/544927 | 12:39 |
| evrardjp | stable/queens now has extra patches, one of which should land soon that is solving something critical | 12:40 |
| *** kstev has quit IRC | 12:40 | |
| admin0 | can't that patch be pushed today so that that one is out :) | 12:40 |
| admin0 | if that patch is pushed and it lands on stable/queen i can use that to deploy and then when .1 is out, follow the upgrade docs | 12:41 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/queens: Urgent bump of lxc_hosts https://review.openstack.org/556474 | 12:42 |
| admin0 | i have 3 seperate multi-node clusters to deliver for the internal demos | 12:43 |
| evrardjp | admin0: I'd wait for the next tag. | 12:44 |
| evrardjp | myself | 12:44 |
| admin0 | i will miss my KPI :D | 12:44 |
| evrardjp | admin0: you can use stable/queens branch when this will land: https://review.openstack.org/#/c/556275/ | 12:45 |
| *** odyssey4me has quit IRC | 12:47 | |
| *** odyssey4me has joined #openstack-ansible | 12:47 | |
| *** kstev has joined #openstack-ansible | 12:49 | |
| evrardjp | andymccr: do you have any cycles to check on something ceph related? http://logs.openstack.org/88/556388/1/check/openstack-ansible-deploy-ceph-ubuntu-xenial/ef54dcc/job-output.txt.gz#_2018-03-26_09_05_03_103815 | 12:50 |
| evrardjp | andymccr: I'll handle the 555013 | 12:52 |
| *** ansmith has joined #openstack-ansible | 12:52 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/ansible-config_template master: Fix pep8 for config_template.py https://review.openstack.org/555013 | 12:54 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Have run_tests clone the test repo https://review.openstack.org/556518 | 13:00 |
| openstackgerrit | Merged openstack/openstack-ansible stable/queens: Disable ceph-ansible NTP installation https://review.openstack.org/556271 | 13:01 |
| *** aruns has joined #openstack-ansible | 13:01 | |
| andymccr | evrardjp: i'll try build an aio and repeat that | 13:01 |
| andymccr | that error seems weird though | 13:01 |
| andymccr | since all the mon plays succeeded etc/ceph/ceph.conf should just exist | 13:01 |
| *** niraj_singh has joined #openstack-ansible | 13:02 | |
| evrardjp | it's been a certain time that its broken but I didn't get the chance to look at it (in case it helps finding the root cause) | 13:03 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/queens: Allow to run tests outside openstack infra https://review.openstack.org/556375 | 13:03 |
| *** aruns__ has quit IRC | 13:03 | |
| *** Sha0000 has quit IRC | 13:04 | |
| *** geb has quit IRC | 13:04 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Have run_tests clone the test repo https://review.openstack.org/556518 | 13:05 |
| *** geb has joined #openstack-ansible | 13:07 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Run the on metal job https://review.openstack.org/556521 | 13:08 |
| odyssey4me | evrardjp or hwoarang - a simple lint fix which needs a review: https://review.openstack.org/#/c/556293/1 | 13:12 |
| *** savvas_ has quit IRC | 13:12 | |
| *** savvas has joined #openstack-ansible | 13:13 | |
| *** chhagarw has joined #openstack-ansible | 13:13 | |
| *** ibmko has quit IRC | 13:17 | |
| *** savvas has quit IRC | 13:18 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Run the on metal job https://review.openstack.org/556521 | 13:19 |
| *** savvas has joined #openstack-ansible | 13:21 | |
| evrardjp | odyssey4me: reinstating some jobs ^ | 13:23 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat stable/pike: Fix doc config pep8 failures https://review.openstack.org/556293 | 13:23 |
| alefra | hi all, i'm unable to install vpnaas on queens, the service not start but the packages seems installed correctly. This is the output of daemon.log: neutron-vpn-agent.service: Failed at step EXEC spawning /openstack/venvs/neutron-17.0.0/bin/neutron-vpn-agent: No such file or directory | 13:23 |
| odyssey4me | evrardjp ah, I guess those somehow got missed when I renamed the other ones | 13:23 |
| *** RandomTech has joined #openstack-ansible | 13:24 | |
| odyssey4me | sorry about that | 13:24 |
| *** Sha0000 has joined #openstack-ansible | 13:24 | |
| evrardjp | no worries | 13:24 |
| *** electrofelix has quit IRC | 13:31 | |
| *** savvas has quit IRC | 13:32 | |
| *** savvas has joined #openstack-ansible | 13:35 | |
| *** hamza21 has joined #openstack-ansible | 13:37 | |
| *** haad1 has quit IRC | 13:38 | |
| Miouge | nmap shows “25/open/tcp//smtp//Postfix smtpd/” on my openstack-ansible based clusters. Is that from openstack-ansible or from my bare metal provisioniner? | 13:39 |
| mattt | evrardjp: howdy | 13:42 |
| evrardjp | hey odyssey4me mattt | 13:42 |
| evrardjp | mattt: I don't think the subunit results are used for the health dashboard anymore | 13:42 |
| evrardjp | sorry | 13:42 |
| evrardjp | I meant | 13:42 |
| evrardjp | I don't think the TEMPEST subunit results are used in our health dashboard anymore | 13:42 |
| evrardjp | http://status.openstack.org/openstack-health/#/g/project/openstack~2Fopenstack-ansible?searchJob=openstack%2Fopenstack-ansible | 13:43 |
| mattt | yeah i couldn't see anything indicating that they were, but wanted to be sure https://review.openstack.org/#/c/555621/3/tasks/tempest_run.yml wasn't going to break anything | 13:43 |
| evrardjp | I think the only ones we are using are the ARA -> subunit output. | 13:43 |
| mattt | i mean, i can look at fixing those in a subsequent review, if there's value there | 13:43 |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: Disable ceph-ansible NTP installation https://review.openstack.org/556272 | 13:44 |
| openstackgerrit | Merged openstack/openstack-ansible stable/queens: Update cinder role SHA for cinder_backends fix https://review.openstack.org/556275 | 13:44 |
| evrardjp | I have never used elastic-recheck, and logstash, so I can't say if that's useful | 13:44 |
| evrardjp | I know odyssey4me has done it in the past | 13:44 |
| evrardjp | so he probably needs it more than me | 13:44 |
| odyssey4me | needs what? | 13:44 |
| evrardjp | for me, as long as the tempest subunit file is properly generated and gathered in our logs it's good enough for me | 13:45 |
| odyssey4me | logstash/elastic-recheck have nothing to do with subunit output :p | 13:45 |
| evrardjp | odyssey4me: the subunit output of tempest -- maybe stored to the standard location | 13:45 |
| *** masuberu has joined #openstack-ansible | 13:45 | |
| evrardjp | odyssey4me: it's just for health? | 13:45 |
| odyssey4me | yes | 13:45 |
| evrardjp | okay | 13:45 |
| mattt | odyssey4me: my question is if changing filenames in https://review.openstack.org/#/c/555621/3/tasks/tempest_run.yml breaks anything, or if it's all broken to begin with | 13:46 |
| evrardjp | mattt: it's not broken, it's just a "we don't care" | 13:46 |
| *** haad1 has joined #openstack-ansible | 13:46 | |
| evrardjp | or a "we don't care anymore" | 13:46 |
| mattt | fair enough :) | 13:47 |
| odyssey4me | mattt ah ok, lemme dig up the place where the file names are mentioned | 13:47 |
| mattt | odyssey4me: https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/files/logstash/jenkins-log-client.yaml#L14 | 13:47 |
| odyssey4me | mattt yep, that's the place | 13:48 |
| *** hamza21 has quit IRC | 13:48 | |
| odyssey4me | the subunit file just needs to be named 'testrepository.subunit' - the rest doesn't matter | 13:49 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Use Ansible 2.5 https://review.openstack.org/556388 | 13:49 |
| mattt | odyssey4me: so you'd suggest leaving hte filenames as they were then? | 13:50 |
| odyssey4me | just the subunit file - the others can change | 13:50 |
| evrardjp | odyssey4me: but that's for appearing on health only right? | 13:50 |
| mattt | i'll leave it and make a note why it's not changing | 13:51 |
| evrardjp | odyssey4me: and that's only for ROOT_DIR/testrepository.subunit right? | 13:51 |
| odyssey4me | evrardjp yes | 13:51 |
| evrardjp | ok | 13:51 |
| odyssey4me | um, no - I think the file can be named that anywhere, but you might be right | 13:51 |
| evrardjp | mattt: if you keep the filename that's a good bonus, but I don't think it's the most important thing of that patch | 13:51 |
| evrardjp | for me it matters to see the results, and right now, I have none | 13:52 |
| odyssey4me | evrardjp I think for it to be ingested into openstack health, the file path needs to be one of these: https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/files/logstash/jenkins-log-client.yaml#L6-L27 | 13:52 |
| odyssey4me | so we need to ensure that we have it in the root of our logs folder | 13:53 |
| *** udesale has quit IRC | 13:53 | |
| evrardjp | odyssey4me: yes. So I don't think it matters at all | 13:53 |
| odyssey4me | evrardjp you're being very confusing - does it matter to you or not? | 13:54 |
| evrardjp | but if it's always named the same way in our logs, it's easy to create a symlink to that file | 13:54 |
| evrardjp | haha | 13:54 |
| evrardjp | it doesn't matter to me. I just think we can do it afterwards. | 13:54 |
| evrardjp | the one we need to keep is the subunit output from ARA, that's cool to have. | 13:55 |
| *** udesale has joined #openstack-ansible | 13:55 | |
| evrardjp | but recently I spent more on the zuul dashboard than on OS Health | 13:55 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_tempest master: Switch from testrepository to stestr https://review.openstack.org/555621 | 13:56 |
| odyssey4me | well, if the ARA subunits are showing up in openstack health, then my suspicion was correct - we can have many files of that name anywhere in the logs folder (including subdirectories) | 13:56 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat stable/pike: Zuul: Remove project name https://review.openstack.org/543672 | 13:57 |
| evrardjp | odyssey4me: that's indeed location of the ara generated subunit folder: http://logs.openstack.org/02/544002/4/gate/openstack-ansible-varstest-aio_lxc-ubuntu-xenial/218e289/logs/ara/ | 13:58 |
| RandomTech | Hello, Does anyone how a moment to look over my configuration files. I have a feeling I misconfigured something due to how the deployment is behaving, but im not sure how. | 13:58 |
| evrardjp | odyssey4me: ok so we are good | 13:59 |
| evrardjp | odyssey4me: you can vote on mattt 's patch then :) | 13:59 |
| odyssey4me | evrardjp yes, that's a subdirectory not in this list https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/files/logstash/jenkins-log-client.yaml#L6-L27 so clearly the ingestion must be driven in some other way | 13:59 |
| evrardjp | yup, maybe it's doing a recursive glob search | 14:00 |
| evrardjp | we'll see | 14:00 |
| evrardjp | double health! | 14:00 |
| evrardjp | "QUAD DAMAGE" when break failure. | 14:01 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Have run_tests clone the test repo https://review.openstack.org/556518 | 14:08 |
| odyssey4me | evrardjp ^ thoughts on that? | 14:10 |
| *** kstev has quit IRC | 14:10 | |
| *** Sha0000 has quit IRC | 14:12 | |
| *** masuberu has quit IRC | 14:12 | |
| *** masuberu has joined #openstack-ansible | 14:12 | |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: test-ansible-deps: Bump to latest ansible and ansible-lint https://review.openstack.org/556480 | 14:13 |
| *** haad1 has quit IRC | 14:13 | |
| *** admin0 has quit IRC | 14:18 | |
| evrardjp | odyssey4me: I have two questions | 14:20 |
| evrardjp | for now you want to reduce the dependency on many scripts, so that we can move out of tox easier, right? | 14:20 |
| evrardjp | (simplify the user interface) | 14:20 |
| evrardjp | my second question is: why the required projects is removed? It was handy | 14:21 |
| evrardjp | (but indeed very zuul specific) | 14:21 |
| odyssey4me | evrardjp the required projects is not removed - the linters test now uses the functional test job definition as a base | 14:21 |
| *** kstev has joined #openstack-ansible | 14:22 | |
| odyssey4me | the functional test job definition still has all the required projects | 14:22 |
| odyssey4me | evrardjp for now I'm just trying to reduce the need to replicate changes to every repo unnecessarily | 14:24 |
| *** cjloader has joined #openstack-ansible | 14:24 | |
| odyssey4me | we try to keep the number of things every repo needs as slim as possible | 14:24 |
| evrardjp | good thanks. | 14:24 |
| *** aruns has quit IRC | 14:25 | |
| *** ibmko has joined #openstack-ansible | 14:25 | |
| odyssey4me | to remove tox, we'll have to move the settings for each job from tox.ini into environment vars on the job definition in the repo - which is fine, but then we also need something that can read the job definition and convert it into ansible vars | 14:26 |
| evrardjp | odyssey4me: I am thinking the same way we do gate-check-commit "aio_lxc" "deploy" | 14:27 |
| evrardjp | having something like ./run_tests linters || ./run_tests functional (...) | 14:27 |
| odyssey4me | that is already our interface | 14:28 |
| evrardjp | ok I am using tox all the time :p | 14:28 |
| evrardjp | I will adapt my fingers. | 14:28 |
| *** savvas_ has joined #openstack-ansible | 14:28 | |
| odyssey4me | if you do './run_tests.sh functional', then today it runs the tox environment by the name 'functional' | 14:29 |
| evrardjp | https://images.duckduckgo.com/iu/?u=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F10rsLtGrOcCR0s%2Fgiphy.gif&f=1 | 14:29 |
| evrardjp | that's definitely https://media.giphy.com/media/8fen5LSZcHQ5O/giphy.gif | 14:30 |
| *** savvas has quit IRC | 14:30 | |
| odyssey4me | what I'm thinking now is that I'll build a python script which reads the in-repo jobs.yml definition, and it should then by default list out the available jobs by name | 14:31 |
| odyssey4me | so, for example looking at https://github.com/openstack/openstack-ansible-os_glance/blob/master/zuul.d/jobs.yaml | 14:31 |
| odyssey4me | you would be able to do: ./run_tests.sh openstack-ansible-v1_api_enabled | 14:31 |
| evrardjp | maybe the run tests must evolve | 14:32 |
| evrardjp | to be more generic then | 14:32 |
| evrardjp | and more zuul | 14:32 |
| openstackgerrit | Merged openstack/openstack-ansible master: Add support for Horizon Octavia Dashboard https://review.openstack.org/544002 | 14:33 |
| odyssey4me | well, run_tests.sh is just a wrapper that gets things started | 14:33 |
| evrardjp | but you're right: the center is to have, inside the repo, a way to list all the jobs attached to it | 14:33 |
| *** markvoelker_ has joined #openstack-ansible | 14:34 | |
| *** bhujay has quit IRC | 14:34 | |
| *** markvoelker has quit IRC | 14:35 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_tempest master: Switch from testrepository to stestr https://review.openstack.org/555621 | 14:35 |
| *** armaan has quit IRC | 14:36 | |
| odyssey4me | evrardjp hmm, https://review.openstack.org/556474 failed with no space on device again | 14:36 |
| *** markvoelker has joined #openstack-ansible | 14:37 | |
| odyssey4me | http://logs.openstack.org/74/556474/2/check/openstack-ansible-deploy-aio_lxc-ubuntu-xenial/36f5235/logs/openstack/aio1_repo_container-ec3c1b97/repo/venv_build_tempest-17.0.0-x86_64.log.txt.gz | 14:37 |
| evrardjp | oh fun odyssey4me | 14:37 |
| evrardjp | what the hell is going on | 14:38 |
| odyssey4me | I have no idea, but clearly that issue is not resolved | 14:38 |
| evrardjp | is tmp a ramdisk? | 14:38 |
| evrardjp | cloudnull: could you have a look ? | 14:38 |
| *** markvoelker_ has quit IRC | 14:39 | |
| odyssey4me | not as far as I saw - when I looked /tmp was in the root | 14:39 |
| odyssey4me | root of the container fs I mean | 14:39 |
| evrardjp | this is a nightmare. It was working perfectly fine before using machinectl. | 14:39 |
| cjloader | hello all | 14:40 |
| evrardjp | hello. | 14:40 |
| *** indistylo has joined #openstack-ansible | 14:40 | |
| evrardjp | how are things cjloader ? | 14:40 |
| cjloader | good, you? | 14:41 |
| cloudnull | evrardjp: I can look in a few. | 14:41 |
| evrardjp | cloudnull: thanks. | 14:41 |
| cloudnull | IDK if this is a machinctl issue, we've been using that for a while. | 14:42 |
| cloudnull | though we could reset that to just dir | 14:42 |
| cloudnull | lxc container backend dir | 14:42 |
| cjloader | evrardjp: any pressing items need review? | 14:42 |
| evrardjp | cloudnull: no quotas? | 14:43 |
| evrardjp | when using dir? | 14:43 |
| evrardjp | is that just a btrfs thing? | 14:43 |
| evrardjp | wasn't it overlayfs by default at some point? | 14:44 |
| odyssey4me | evrardjp pike is overlayfs for AIO tests, yes | 14:44 |
| evrardjp | maybe worth moving back to overlayfs with no quotas, until we find out the root cause | 14:44 |
| odyssey4me | well, the thing is that there are no quotas already | 14:44 |
| cloudnull | that | 14:44 |
| cloudnull | I think its one of our bind mounts | 14:45 |
| cloudnull | I just cant nail it down | 14:45 |
| evrardjp | wait, you mean we are in a "no quota" mode, but it's painful anyway? | 14:45 |
| odyssey4me | yes | 14:45 |
| evrardjp | mmm I see. | 14:45 |
| odyssey4me | but it's only hurting us when we use rax nodes (which are the two-disk implementations) | 14:45 |
| cloudnull | yea. and when using the rax cloud to try and recraete it I'm not able to ... | 14:47 |
| odyssey4me | I wonder if it's *always* failing on rax nodes - or only intermittently | 14:49 |
| cloudnull | I wonder if we can nail down a particular region where its failing | 14:50 |
| odyssey4me | cloudnull busy looking at logstash.o.o - looks like it's happening in multiple regions | 14:52 |
| odyssey4me | dfw, ord, iad | 14:52 |
| *** Sha0000 has joined #openstack-ansible | 14:52 | |
| *** esberglu has joined #openstack-ansible | 14:54 | |
| *** Sha0000 has quit IRC | 14:54 | |
| *** Sha0000 has joined #openstack-ansible | 14:54 | |
| *** Sha0000 has quit IRC | 14:59 | |
| cloudnull | its is all on the same flavor ? | 14:59 |
| cloudnull | performance1-8 | 14:59 |
| odyssey4me | hmm, so it appears that we're getting frequent success in rax environments | 15:02 |
| *** Sha0000 has joined #openstack-ansible | 15:02 | |
| logan- | does anyone know of any ops running control plane on single proc E3 / "low memory" (ie 32gb) type of servers? | 15:04 |
| logan- | id like to hear how their experience over time is, and what worker count tunings they've made | 15:04 |
| *** dmsimard is now known as dmsimard|cave | 15:05 | |
| odyssey4me | cloudnull in the last day according to logstash.o.o we've successfully run 15 AIO builds on RAX, with only 3 failures | 15:07 |
| odyssey4me | the failures are across regions | 15:07 |
| logan- | do you have log urls for some of the failures? does it fail in tempest or randomly throughout the run? maybe we could check the dstat timing for disk util and see where in the playbook run that is happening? | 15:07 |
| RandomTech | out of curiosity what does AIO stand for? | 15:08 |
| odyssey4me | logan- it fails in the venv build with a disk full error | 15:08 |
| logan- | wow | 15:08 |
| evrardjp | logan-: I think it would be great to mention that into the Vancouver forum etherpad | 15:08 |
| evrardjp | RandomTech: all in one | 15:08 |
| RandomTech | ahh | 15:09 |
| odyssey4me | RandomTech AIO = All-In-One, with we use when referring to our test instances which build OpenStack on a single VM | 15:09 |
| logan- | thanks evrardjp will do | 15:09 |
| *** XveD has joined #openstack-ansible | 15:09 | |
| *** chyka has joined #openstack-ansible | 15:10 | |
| RandomTech | evrardjp, odyssey4me Is it possible for one of you to have a look over my configuration files sometime before 1pm EST? If not is it possible sometime tomorrow? It would be a great help to me and my coworkers. | 15:11 |
| RandomTech | It is fine if you cannot of course | 15:12 |
| odyssey4me | RandomTech if you sanitise it and post a link to the sanitised version in channel, then perhaps some can look at it | 15:12 |
| *** XveD is now known as DevX | 15:12 | |
| cloudnull | odyssey4me: well that's not too terrible of a failure rate | 15:13 |
| odyssey4me | cloudnull yeah, it's annoying - but seems to be inconsistent and I can't find any differences in the hardware profile of each of those builds | 15:16 |
| RandomTech | odyssey4me: What are you meaning by sanitise it? | 15:16 |
| evrardjp | logan-: https://etherpad.openstack.org/p/YVR-openstack-ansible-brainstorming | 15:16 |
| cloudnull | in https://review.openstack.org/#/c/552047 - Ive added https://review.openstack.org/#/c/552047/35/scripts/scripts-library.sh to help try and debug anything failing with machinectl | 15:16 |
| cloudnull | but not being able to recreate teh issue is frustrating | 15:17 |
| odyssey4me | RandomTech make sure there's no bit in it that you shouldn't be sharing - like your public IP's, passwords, etc | 15:17 |
| RandomTech | ahh of course | 15:17 |
| RandomTech | should i put them all in one paste or in seperate ones? | 15:17 |
| *** geb has quit IRC | 15:17 | |
| evrardjp | odyssey4me: could you check the nodepool info? | 15:18 |
| evrardjp | for those failing/succeeding? | 15:18 |
| odyssey4me | RandomTech up to you, you might want to use something like gist instead | 15:18 |
| evrardjp | I mean the node build details | 15:18 |
| *** jwitko has joined #openstack-ansible | 15:18 | |
| *** pcaruana has quit IRC | 15:18 | |
| odyssey4me | evrardjp what do you mean by 'node build details' ? | 15:18 |
| *** admin0 has joined #openstack-ansible | 15:18 | |
| evrardjp | (zuul-info/) | 15:18 |
| evrardjp | odyssey4me: ^ | 15:19 |
| odyssey4me | yes, there's a link to the log directory - so all logs from the jobs are available | 15:19 |
| odyssey4me | you can check it out yourself if you like | 15:19 |
| odyssey4me | go to logstash.openstack.org | 15:20 |
| odyssey4me | add this into the query panel | 15:20 |
| odyssey4me | project:"openstack/openstack-ansible" AND message:"LOOP [upload-logs : gzip console log and json output]" AND (node_provider:"rax-ord" OR node_provider:"rax-iad" OR node_provider:"rax-dfw") AND (build_branch:"master" OR build_branch:"stable/queens") AND build_node:"ubuntu-xenial" AND build_name:"openstack-ansible-deploy-aio_lxc-ubuntu-xenial" | 15:20 |
| odyssey4me | then change the time frame drop-down to 7 days | 15:21 |
| *** geb has joined #openstack-ansible | 15:21 | |
| odyssey4me | then you can select any event and it'll open up the rest of the data for the event, including a link to the logs for the build | 15:22 |
| evrardjp | odyssey4me: oh yes I can select the failed ones | 15:22 |
| evrardjp | AND build_status I guess | 15:22 |
| evrardjp | good | 15:22 |
| odyssey4me | you can also check the fields on the left to show those colums in the event list | 15:22 |
| logan- | thanks odyssey4me | 15:22 |
| odyssey4me | for example, select 'build_status' and you'll see build status for each event | 15:23 |
| cloudnull | logan-: i have some old L5520 servers with 64GB or ram that I can run things on if you need | 15:23 |
| evrardjp | wait I do not need to do that in query? | 15:23 |
| evrardjp | let me toy with this :) | 15:23 |
| logan- | cloudnull: id be curious to see: for i in `lxc-ls -1`; do echo $i; lxc-info -n $i | grep Memory; done | 15:23 |
| odyssey4me | the check boxes on the bottom left are the fields to expose on the panel on the bottom right | 15:24 |
| odyssey4me | by default it only shows the timestamp and message fields | 15:24 |
| logan- | cloudnull: assuming you have an OSA install sitting on them :P | 15:24 |
| *** indistylo has quit IRC | 15:25 | |
| odyssey4me | logan- did cloudnull sort out your concerns with https://review.openstack.org/552047 sufficiently? | 15:25 |
| odyssey4me | evrardjp does my response in https://review.openstack.org/556518 make sense? | 15:26 |
| * logan- reads thru | 15:26 | |
| cloudnull | logan-: http://paste.openstack.org/show/714225/ | 15:26 |
| evrardjp | odyssey4me: on it | 15:27 |
| logan- | thx cloudnull | 15:27 |
| *** exodusftw has quit IRC | 15:31 | |
| RandomTech | If anyone has a moment it would really help if someone was able to look at my configuration files. They should be found here: https://gist.github.com/AlanCFleming/1e883db0a9a600e7faa1333987f42d2e | 15:33 |
| odyssey4me | RandomTech compute_hosts is specified twice | 15:35 |
| odyssey4me | one with the anchor, then one using the anchor | 15:35 |
| odyssey4me | compute_hosts: &compute_hosts ... compute_hosts: *compute_hosts | 15:35 |
| RandomTech | im guessing that may cause some issues | 15:36 |
| RandomTech | didnt notice that before | 15:36 |
| RandomTech | could that possibly explain why nova is buggy odyssey4me ? | 15:37 |
| odyssey4me | otherwise looks fine to me, but I'm no expert at either networking or ceph | 15:37 |
| odyssey4me | nova being buggy - not sure what you mean there, so I can't really answer the question | 15:37 |
| RandomTech | Instances when created dont work properly and wont delete | 15:39 |
| *** Sha0000 has quit IRC | 15:39 | |
| *** bhujay has joined #openstack-ansible | 15:39 | |
| *** haad1 has joined #openstack-ansible | 15:40 | |
| *** exodusftw has joined #openstack-ansible | 15:41 | |
| *** DanyC has joined #openstack-ansible | 15:41 | |
| odyssey4me | RandomTech you'd have to trawl logs to figure out why they won't delete, but that's the kind of nonsense you'd find if your time was out of sync or your rabbit cluster was borked | 15:42 |
| RandomTech | when i look in the logs i find what seems to be near endless python errors odyssey4me | 15:43 |
| evrardjp | or slow to answer | 15:43 |
| evrardjp | wow | 15:43 |
| evrardjp | that's bad | 15:43 |
| *** savvas_ has quit IRC | 15:43 | |
| evrardjp | endless python errors is something you should paste somewhere. | 15:43 |
| odyssey4me | ok, and what do those errors tell you? | 15:44 |
| *** savvas has joined #openstack-ansible | 15:44 | |
| RandomTech | i attempted to show people the other day but couldnt get ahold of anyone | 15:44 |
| RandomTech | that seem to be dealing with sockets, so i believe pieces arnt communicating with each other right | 15:44 |
| odyssey4me | can you add a sample of some of those errors you're seeing to the gist? | 15:44 |
| RandomTech | i can attempt to get a snippet of it for you | 15:44 |
| *** haad1 has quit IRC | 15:45 | |
| RandomTech | sure one moment while i grab them | 15:45 |
| *** bhujay has quit IRC | 15:47 | |
| RandomTech | there you go evrardjp odyssey4me, sorry if it looks a bit jumbled. If you need more of them i have plent | 15:49 |
| RandomTech | plenty* | 15:49 |
| odyssey4me | RandomTech aha, you're missing the compute-infra_hosts group | 15:50 |
| *** exodusftw has quit IRC | 15:50 | |
| odyssey4me | so none of the nova control plane services are running | 15:51 |
| RandomTech | aha! | 15:51 |
| RandomTech | should that be the infrastructure/control hosts listed under that or my compute hosts | 15:51 |
| odyssey4me | any cores available to look through https://review.openstack.org/556518 ? | 15:51 |
| odyssey4me | RandomTech just add: compute-infra_hosts: &infrastructure_hosts | 15:52 |
| odyssey4me | sorry, not that | 15:52 |
| odyssey4me | compute-infra_hosts: *infrastructure_hosts | 15:52 |
| shananigans | RandomTech: on a side note, the type 'flat' provider networks should be set to the physical device behind the bridge and not the bridge itself. | 15:53 |
| RandomTech | should i be able to just rerun my playbooks to test it odyssey4me, or should i just do a reinstall before testing it? | 15:54 |
| RandomTech | shananigans: are you able to give an example or how it should look? | 15:54 |
| RandomTech | of* | 15:54 |
| odyssey4me | RandomTech once you've added that, just run openstack-ansible setup-hosts.yml setup-infrastructure.yml setup-openstack.yml | 15:55 |
| shananigans | for the br-vlan 'flat' network, the host_bind_override should be enp4s0 i think. | 15:55 |
| *** pabelanger has quit IRC | 15:56 | |
| shananigans | doesn't look like eth12 exists on the host which is what its currently set to. | 15:56 |
| RandomTech | i was honestly given that sugested configuration file. im definately willing to do changes for the next go shananigans | 15:56 |
| RandomTech | and i think i see what your talking about shananigans, i think i missed changing that line | 15:57 |
| *** ansmith has quit IRC | 15:57 | |
| *** exodusftw has joined #openstack-ansible | 15:58 | |
| RandomTech | wait shananigans, i was told to hang a virtual eth12 interface off my bridge. should is till do that? | 15:58 |
| *** pabelanger has joined #openstack-ansible | 16:00 | |
| RandomTech | or can anyone else confirm if i should be doing so? | 16:00 |
| shananigans | That may work as well. I must have missed the virt interface in the configs. | 16:00 |
| *** DanyC has quit IRC | 16:00 | |
| *** savvas has quit IRC | 16:00 | |
| shananigans | nm, I see all of the pre-up lines on br-vlan for eth12. | 16:01 |
| *** savvas has joined #openstack-ansible | 16:01 | |
| *** ansmith has joined #openstack-ansible | 16:02 | |
| RandomTech | its fine, i have it there as someone said i needed that and host_bind_override to make it work. If i could avoid doing that it would also be nice | 16:02 |
| shananigans | Either way should be good as long as its not the bridge. One is just more work. | 16:03 |
| *** vnogin has quit IRC | 16:03 | |
| RandomTech | oh okay | 16:04 |
| *** bhujay has joined #openstack-ansible | 16:04 | |
| *** throwsb1 has joined #openstack-ansible | 16:04 | |
| RandomTech | i was told if i did host_bind_override on the interface the bridge attached to it would not work | 16:04 |
| RandomTech | okay my new playbooks are running | 16:08 |
| jrosser | logan-: my control plane is 4C/8T/64G per node, but lots of them, a pair of microclouds | 16:09 |
| jrosser | time will tell if this was a good/bad idea | 16:09 |
| *** ansmith has quit IRC | 16:10 | |
| logan- | gotcha. you've got the containers sort of spread across them? | 16:10 |
| jrosser | 3 for shared (db etc), 3 for apis, 2 for haproxy, 3 ceph mons, log | 16:11 |
| logan- | i have a few small clouds with 4C/8T/32G on microcloud sleds but it is a 3 node control plane, and I have seen performance degrade steadily since mitaka to almost unusable now due to the default worker counts + additional services being added like nova-placement | 16:11 |
| jrosser | i dividied galeta/rabbit/memcache onto seperate nodes | 16:11 |
| logan- | cool | 16:11 |
| jrosser | seen there is a new xeon-d based microcloud? | 16:12 |
| jrosser | 18core per sled | 16:12 |
| odyssey4me | Our default worker/process values are probably too high, but we've never really had anyone provide some sort of data to help adjust the defaults appropriately. | 16:12 |
| *** ansmith has joined #openstack-ansible | 16:13 | |
| RandomTech | does anyone know how to increase the scrollable length on tmux? | 16:13 |
| odyssey4me | RandomTech before starting your session, do something like: echo 'set-option -g history-limit 9999' > ~/.tmux.conf | 16:14 |
| jrosser | odyssey4me: part my my reasoning for lots of small nodes was to be able to work out where when to need larger | 16:14 |
| jrosser | rather than just pile it all together and get confused | 16:14 |
| RandomTech | thanks odyssey4me, ill do that for setup infrastructure as i forgot before setup-hosts | 16:14 |
| *** udesale has quit IRC | 16:17 | |
| *** udesale has joined #openstack-ansible | 16:17 | |
| *** masuberu has quit IRC | 16:18 | |
| *** indistylo has joined #openstack-ansible | 16:24 | |
| *** savvas has quit IRC | 16:25 | |
| *** savvas has joined #openstack-ansible | 16:25 | |
| *** fatdragon has joined #openstack-ansible | 16:26 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Have run_tests clone the test repo https://review.openstack.org/556518 | 16:27 |
| RandomTech | odyssey4me: evrardjp: setup hosts ran successfully now onto setup-infrastructure | 16:30 |
| *** masuberu has joined #openstack-ansible | 16:31 | |
| RandomTech | oh ya is the fix posted here: https://review.openstack.org/#/c/551793/ in the main branch yet | 16:32 |
| odyssey4me | RandomTech it's in stable/pike, yes | 16:32 |
| odyssey4me | it's not release yet though, but it's in the branch HEAD | 16:32 |
| shananigans | RandomTech Forget what I said, I'm an idiot. Most of my envs have its own flat vlan or physical interface. It would have issues if you tried to use both the vlan and flat on the same bridge. | 16:33 |
| RandomTech | okay thank you shananigans, im not even sure if i need all the networks i listed in the first place if you see anything that can be trimmed down | 16:33 |
| RandomTech | there should be a link to a diagram in there showing how the physical network looks | 16:34 |
| *** bhujay has quit IRC | 16:34 | |
| RandomTech | okay good to know odyssey4me, ill make sure to redo the patch if i pull it down again. Do you know if i will need to reinstall my deployment host during a re-kick? | 16:35 |
| *** udesale has quit IRC | 16:35 | |
| odyssey4me | if your deployment host is dedicated to that purpose, I don't think so | 16:37 |
| RandomTech | okay thats good to know odyssey4me, it makes things alot easier :) | 16:40 |
| RandomTech | once we get things working properly we plan to do i nice clean install so that everything is fresh/not patched togeather | 16:40 |
| *** radeks has quit IRC | 16:41 | |
| evrardjp | logan-: okay, got a few minutes today | 16:41 |
| evrardjp | let's fix the env.d | 16:41 |
| *** pcaruana has joined #openstack-ansible | 16:44 | |
| RandomTech | hey odyssey4me, evrardjp, I keep getting outputs from ansible that look like this: http://paste.openstack.org/show/714324/ | 16:45 |
| RandomTech | i thought it might be usefull to know | 16:45 |
| evrardjp | always_run? | 16:46 |
| evrardjp | it's fine I think | 16:46 |
| evrardjp | but still wondering why you have always_run | 16:46 |
| logan- | evrardjp: my best guess is ansible 2.4 doesn't like something with how our dynamic inventory maps everything when host group == service group | 16:46 |
| logan- | haproxy is the only main service I know of where this scenario exists | 16:47 |
| evrardjp | my best guess is group var BS | 16:47 |
| evrardjp | :p | 16:47 |
| RandomTech | okay, just letting you know as i thought that some playbooks may need updated if there using near depricated commands | 16:47 |
| *** Leo_m has joined #openstack-ansible | 16:47 | |
| odyssey4me | RandomTech it'd be more useful to know which playbooks are giving out that error | 16:48 |
| odyssey4me | that warning I mean | 16:48 |
| *** ibmko has quit IRC | 16:48 | |
| RandomTech | currently im running setup-infrastructure and its in the middle of some ceph stuff. | 16:48 |
| RandomTech | ill check where else it is however | 16:48 |
| RandomTech | well it seems most if not all of it is from ceph playbooks | 16:49 |
| RandomTech | so its probably them. Sorry i cant find more, my tmux history didnt increase like i thought it would | 16:50 |
| odyssey4me | you would have had to exist your current tmux session and start a new one after setting it in your ~/.tmux.conf | 16:50 |
| evrardjp | logan-: I think I found it | 16:52 |
| logan- | o rly | 16:52 |
| openstackgerrit | Matthew Thode proposed openstack/openstack-ansible-specs master: create Designate back end provider spec https://review.openstack.org/556586 | 16:52 |
| RandomTech | i did so odyssey4me, the number may now have meant what i thought it did however. I believed it meant number of lines | 16:52 |
| RandomTech | or at least i thought i did | 16:53 |
| *** indistylo has quit IRC | 16:53 | |
| *** chhagarw has quit IRC | 16:53 | |
| odyssey4me | yeah, it is # of lines | 16:53 |
| evrardjp | logan-: could you try something for me? | 16:54 |
| logan- | yes | 16:54 |
| odyssey4me | RandomTech there is a log file by default too - check /openstack/log/ansible-logging/ansible.log and you should see it there | 16:54 |
| evrardjp | I'll put up a patch | 16:54 |
| logan- | ok | 16:54 |
| evrardjp | tell me how it goes for you | 16:54 |
| *** savvas has quit IRC | 16:54 | |
| prometheanfire | evrardjp: finially got around to creating that spec | 16:54 |
| RandomTech | odyssey4me: that is an extremely nice log to know about, i have to leave soon but ills ee if i can find it elsewhere real quick | 16:56 |
| RandomTech | okay is seems to only be ceph | 16:57 |
| *** indistylo has joined #openstack-ansible | 16:57 | |
| RandomTech | most likely an issue with there playbooks | 16:57 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: [WIP] Change the index group for haproxy https://review.openstack.org/556588 | 16:58 |
| evrardjp | logan-: ^ | 16:58 |
| logan- | ok i will take a look, but that's a bad change :/ | 16:59 |
| logan- | since service group == host group in this case, haproxy_all includes both the metal hosts and the containers | 16:59 |
| logan- | testing tho :P | 17:00 |
| RandomTech | im heading off to class for today. thanks again for the help odyssey4me evrardjp shananigans and logan- | 17:00 |
| evrardjp | we can rename the group | 17:00 |
| evrardjp | it's about the same issue. | 17:01 |
| odyssey4me | RandomTech sure thing, glad there's some progress | 17:01 |
| RandomTech | ill tell you guys how it went when im in tomorrow | 17:01 |
| *** RandomTech has quit IRC | 17:01 | |
| evrardjp | logan-: would you be better if I rename the group_vars/haproxy_all to group_vars/haproxy ? | 17:02 |
| logan- | yep | 17:02 |
| logan- | that sounds good | 17:02 |
| evrardjp | ok | 17:02 |
| *** savvas has joined #openstack-ansible | 17:02 | |
| evrardjp | could you try it then? | 17:02 |
| logan- | yup | 17:02 |
| evrardjp | thanks let me put a patch real quick then | 17:03 |
| *** savvas has quit IRC | 17:03 | |
| *** savvas has joined #openstack-ansible | 17:03 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: [WIP] Change the index group for haproxy https://review.openstack.org/556588 | 17:04 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Change the index group for haproxy https://review.openstack.org/556588 | 17:04 |
| evrardjp | voila! | 17:05 |
| evrardjp | oh wait bug number. | 17:05 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Re-use run_tests.sh variables in run_tests_common.sh https://review.openstack.org/556591 | 17:06 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Change the index group for haproxy https://review.openstack.org/556588 | 17:06 |
| *** masuberu has quit IRC | 17:07 | |
| logan- | evrardjp: waiting around for bootstrap-ansible, ill get back to ya in 15-20 min on it | 17:07 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Apply haproxy vars to haproxy group instead of haproxy_all https://review.openstack.org/556588 | 17:07 |
| evrardjp | logan-: thanks | 17:08 |
| *** hvvclint has joined #openstack-ansible | 17:10 | |
| *** alefra has quit IRC | 17:10 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Re-use env var for setting the test branch https://review.openstack.org/556594 | 17:11 |
| openstackgerrit | Merged openstack/ansible-config_template master: Fix pep8 for config_template.py https://review.openstack.org/555013 | 17:12 |
| logan- | evrardjp: works great | 17:12 |
| logan- | great catch | 17:12 |
| *** hvvclint has quit IRC | 17:15 | |
| evrardjp | I think we can backport it back to Q. | 17:16 |
| evrardjp | I will check with P | 17:17 |
| evrardjp | P doesn't use index | 17:17 |
| evrardjp | let's keep it to master/Q | 17:17 |
| fatdragon | cloudnull: thanks for asking, I should be back in few hours..I have finish OSA installed and seems like I have two issues..cinder not connecting to mysql perhaps the reason I can't create images and the other one is I can only ping out of my cirros vm, while I have all the rules/routes in place..strange! | 17:17 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Have run_tests clone the test repo https://review.openstack.org/556518 | 17:18 |
| *** savvas has quit IRC | 17:24 | |
| *** szaher has joined #openstack-ansible | 17:25 | |
| *** admin0 has quit IRC | 17:28 | |
| *** admin0 has joined #openstack-ansible | 17:29 | |
| *** savvas has joined #openstack-ansible | 17:32 | |
| *** savvas has quit IRC | 17:37 | |
| *** savvas has joined #openstack-ansible | 17:38 | |
| *** Sha0000 has joined #openstack-ansible | 17:47 | |
| *** savvas has quit IRC | 17:47 | |
| *** savvas has joined #openstack-ansible | 17:51 | |
| *** savvas has quit IRC | 17:55 | |
| *** hamza21 has joined #openstack-ansible | 17:57 | |
| *** poopcat has joined #openstack-ansible | 18:00 | |
| logan- | small bit of news, now that https://review.openstack.org/#/c/556554/ has merged we have an OSA deployed cloud active in the openstack ci test pool (http://grafana.openstack.org/dashboard/db/nodepool-limestone) with an open config published at https://github.com/continuouspw/openstack-ci-cloud/ | 18:06 |
| odyssey4me | woot! :) | 18:06 |
| *** savvas has joined #openstack-ansible | 18:09 | |
| pabelanger | exciting! | 18:10 |
| logan- | and thanks to evrardjp's haproxy fix for queens we can see how well CI driven upgrades work in the coming weeks :P | 18:11 |
| odyssey4me | hehe, that'll be a fun time | 18:12 |
| odyssey4me | it'd be great if we could get some sort of monitoring on the api endpoints visible/graphed | 18:13 |
| logan- | ++ | 18:13 |
| *** savvas has quit IRC | 18:13 | |
| pabelanger | logan-: looking at os_nova, I guess we'd need to add nova_force_raw_images option. Don't see a way to set it up | 18:16 |
| logan- | we can inject the config via config_template easily | 18:16 |
| odyssey4me | how often are you making jenkins execute whatever's currently merged in the repo? | 18:16 |
| pabelanger | logan-: ah, let me look at that | 18:16 |
| logan- | currently once per day or however often there is a commit to master | 18:16 |
| odyssey4me | ok, so it happens once per day regardless, but also has a merge trigger | 18:16 |
| logan- | yup | 18:17 |
| odyssey4me | awesome | 18:17 |
| logan- | https://jenkins.lstn.net/job/openstack-upstream/job/openstackci-osa/ averaging about 2 hrs | 18:17 |
| *** admin0 has quit IRC | 18:17 | |
| odyssey4me | that's pike, so that'll be a longer run time than queens I think | 18:18 |
| pabelanger | odyssey4me: logan-: I mean, if this cloud is going to live a long times, imagine if we drived it directly from zuul.o.o :) just saying :D | 18:18 |
| pabelanger | that would be a fun project to make hapen | 18:18 |
| *** savvas has joined #openstack-ansible | 18:19 | |
| logan- | pabelanger: would love to see that happen | 18:21 |
| *** savvas has quit IRC | 18:23 | |
| *** savvas has joined #openstack-ansible | 18:23 | |
| logan- | from my view the whole exercise has been to see how well OSA can be driven thru automation, because I think most ops are babysitting deploy runs at this point. as we do minor/major version upgrades I think we'll learn a lot | 18:23 |
| pabelanger | ++ | 18:24 |
| odyssey4me | yep, it'd be great to work towards having this driven from master rather than stable, but let's walk before we run :) | 18:24 |
| pabelanger | I always thought something like trystack.org could me managed by the community as an OPs project, same with infracloud. But we never really got there. | 18:25 |
| pabelanger | it would be awesome if limestone would get there | 18:26 |
| pabelanger | and I'd love the help with that | 18:26 |
| logan- | awesome. its an open project with dedicated hardware, lets see what we can make from it. :) im looking forward to throwing some more computes into the mix soon as the build crew has time too. | 18:31 |
| openstackgerrit | Antony Messerli proposed openstack/openstack-ansible-ops master: Adds iPXE UEFI support to Multi Node AIO https://review.openstack.org/552088 | 18:32 |
| *** admin0 has joined #openstack-ansible | 18:36 | |
| odyssey4me | pabelanger I wonder if given your experience working with zuul you could help with something else? | 18:36 |
| odyssey4me | I want to build a tool which can take the zuul config and spit out parts of the job config in a way that we can easily consume the data to run the same tests outside zuul. So, given the .zuul.yaml in https://gist.github.com/odyssey4me/32ad76c2a963aea3a234e26b8d7cf161 I'd like to see the resulting output.yaml in that gist | 18:37 |
| odyssey4me | so we'd be looking for the children to be merged into the parents, just like zuul does - but to skip the merge of 'base' because we won't have access to it | 18:39 |
| odyssey4me | we'd only want to show the children for which the .zuul.yaml file has configuration in the 'project' check set | 18:42 |
| pabelanger | odyssey4me: yah, this has come up a few times. I want to say mordred or corvus have a better idea how that might look. But merging parent / child jobs has been asked when running locally | 18:43 |
| pabelanger | or at least, know what it would look like | 18:43 |
| pabelanger | It might also be done via zuul-dashboard API too | 18:43 |
| odyssey4me | hmm, via the api could work quite nicely too | 18:44 |
| pabelanger | odyssey4me: yah, I think once 3.0 is out the door, that is one of the next items to get | 18:45 |
| *** pcaruana has quit IRC | 18:46 | |
| mattt | odyssey4me: did you test https://review.openstack.org/#/c/556046/ on a multi-node? not sure if i messed up my testing | 18:52 |
| odyssey4me | mattt nope, I didn't | 18:53 |
| logan- | mattt: someone on openstack-dev ML said it fixed their issue http://lists.openstack.org/pipermail/openstack/2018-March/046289.html, thats the only test I know of so far | 18:55 |
| mattt | logan-: well that seems positive, let me try again | 18:57 |
| *** savvas_ has joined #openstack-ansible | 19:04 | |
| *** savvas has quit IRC | 19:07 | |
| jrosser | odyssey4me: looking at my recently deployed fairly stock pike deploy the most ram by a huge margin is ceilometer-agent-notification, seem have around 220 threads on a 4C/8T box just for that | 19:18 |
| jrosser | ^ infra node | 19:19 |
| *** flemingo has joined #openstack-ansible | 19:20 | |
| *** flemingo_ has joined #openstack-ansible | 19:21 | |
| odyssey4me | jrosser hmm, I wonder if https://github.com/openstack/openstack-ansible-os_ceilometer/blob/master/defaults/main.yml#L92-L94 is perhaps not being applied properly | 19:21 |
| *** admin0 has quit IRC | 19:23 | |
| *** flemingo has quit IRC | 19:24 | |
| jrosser | odyssey4me: there are 4 worker processes which looks ok, but each seems to have 32 threads | 19:29 |
| jrosser | hmm maybe it's actually 36 | 19:31 |
| *** admin0 has joined #openstack-ansible | 19:34 | |
| *** indistylo has quit IRC | 19:41 | |
| odyssey4me | now, the question is really how many should there be | 19:49 |
| *** ibmko has joined #openstack-ansible | 19:49 | |
| jrosser | indeed - any i see some settings in the ceileometer docs but really getting quite far beyond what i understand | 19:54 |
| jrosser | *and i see... | 19:55 |
| odyssey4me | it might be best to find the appropriate IRC channel to ask them directly | 19:56 |
| odyssey4me | mnaser where do the telemetry crew hang out? | 19:56 |
| *** EmilienM has quit IRC | 19:57 | |
| jrosser | only reason i looked at this was logan- comment earlier about smaller systems for infra nodes, we conincidentally put some snmp collection on everything today and this jumped out head/shoulders abouve everything else | 19:57 |
| *** EmilienM has joined #openstack-ansible | 19:57 | |
| *** savvas_ has quit IRC | 19:58 | |
| *** savvas has joined #openstack-ansible | 19:58 | |
| *** savvas_ has joined #openstack-ansible | 19:59 | |
| *** savvas_ has quit IRC | 20:00 | |
| *** savvas_ has joined #openstack-ansible | 20:01 | |
| *** savvas has quit IRC | 20:03 | |
| openstackgerrit | German Eichberger proposed openstack/openstack-ansible master: Moves Octavia gate from experimental to required https://review.openstack.org/545020 | 20:26 |
| *** Leo_m_ has joined #openstack-ansible | 20:36 | |
| *** Leo_m has quit IRC | 20:39 | |
| *** Leo_m_ has quit IRC | 20:41 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/queens: Urgent bump of lxc_hosts https://review.openstack.org/556474 | 20:48 |
| *** savvas_ has quit IRC | 20:51 | |
| *** savvas has joined #openstack-ansible | 20:52 | |
| *** ansmith has quit IRC | 20:55 | |
| *** hamza21 has quit IRC | 21:05 | |
| *** jwitko_ has joined #openstack-ansible | 21:26 | |
| *** dave-mccowan has quit IRC | 21:27 | |
| *** jwitko has quit IRC | 21:30 | |
| *** jwitko_ has quit IRC | 21:31 | |
| *** Sha0000 has quit IRC | 21:34 | |
| *** savvas has quit IRC | 21:41 | |
| *** sawblade_ has joined #openstack-ansible | 21:47 | |
| *** sawblade_ has quit IRC | 21:50 | |
| *** sawblade6 has quit IRC | 21:50 | |
| *** rromans has quit IRC | 21:50 | |
| *** throwsb1 has quit IRC | 21:51 | |
| *** NWOutlier__ has joined #openstack-ansible | 21:56 | |
| *** ansmith has joined #openstack-ansible | 21:57 | |
| *** cjloader has quit IRC | 22:00 | |
| *** cjloader has joined #openstack-ansible | 22:00 | |
| *** jwitko has joined #openstack-ansible | 22:02 | |
| *** aludwar has quit IRC | 22:03 | |
| *** esberglu has quit IRC | 22:04 | |
| *** esberglu has joined #openstack-ansible | 22:04 | |
| *** sawblade6 has joined #openstack-ansible | 22:04 | |
| *** cjloader has quit IRC | 22:04 | |
| *** aludwar has joined #openstack-ansible | 22:06 | |
| *** esberglu has quit IRC | 22:09 | |
| *** kstev has quit IRC | 22:13 | |
| *** sawblade_ has joined #openstack-ansible | 22:15 | |
| *** sawblade6 has quit IRC | 22:17 | |
| *** savvas has joined #openstack-ansible | 22:24 | |
| *** dave-mccowan has joined #openstack-ansible | 22:28 | |
| *** savvas has quit IRC | 22:45 | |
| *** savvas has joined #openstack-ansible | 22:45 | |
| *** savvas has quit IRC | 22:50 | |
| *** savvas has joined #openstack-ansible | 22:54 | |
| *** pabelanger has quit IRC | 23:01 | |
| *** pabelanger has joined #openstack-ansible | 23:01 | |
| *** vnogin has joined #openstack-ansible | 23:05 | |
| *** vnogin has quit IRC | 23:09 | |
| *** admin0 has quit IRC | 23:12 | |
| *** flemingo has joined #openstack-ansible | 23:30 | |
| *** flemingo_ has quit IRC | 23:33 | |
| *** flemingo has quit IRC | 23:35 | |
| *** savvas has quit IRC | 23:42 | |
| *** sawblade6 has joined #openstack-ansible | 23:43 | |
| *** sawblade_ has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
