| *** thorst has joined #openstack-ansible | 00:08 | |
| *** thorst has quit IRC | 00:12 | |
| *** yifei has joined #openstack-ansible | 00:24 | |
| *** sdake_ has quit IRC | 00:32 | |
| *** sdake_ has joined #openstack-ansible | 00:32 | |
| *** hw_wutianwei has joined #openstack-ansible | 00:43 | |
| *** charcol has joined #openstack-ansible | 00:57 | |
| *** markvoelker has joined #openstack-ansible | 01:15 | |
| *** woodard has joined #openstack-ansible | 01:20 | |
| *** woodard has quit IRC | 01:25 | |
| *** charcol has quit IRC | 01:30 | |
| *** charcol has joined #openstack-ansible | 01:39 | |
| *** markvoelker has quit IRC | 02:02 | |
| *** markvoelker has joined #openstack-ansible | 02:02 | |
| *** thorst has joined #openstack-ansible | 02:13 | |
| openstackgerrit | zhongshengping proposed openstack/openstack-ansible-os_sahara master: Add ceph_client role https://review.openstack.org/485000 | 02:15 |
|---|---|---|
| *** charcol has quit IRC | 02:17 | |
| *** thorst has quit IRC | 02:18 | |
| *** woodard has joined #openstack-ansible | 02:21 | |
| *** woodard has quit IRC | 02:25 | |
| *** markvoelker has quit IRC | 02:34 | |
| *** thorst has joined #openstack-ansible | 03:14 | |
| *** thorst has quit IRC | 03:19 | |
| *** gkadam has joined #openstack-ansible | 03:32 | |
| *** hybridpollo has quit IRC | 03:46 | |
| *** charcol has joined #openstack-ansible | 03:49 | |
| *** udesale has joined #openstack-ansible | 04:06 | |
| *** udesale has quit IRC | 04:15 | |
| *** udesale has joined #openstack-ansible | 04:18 | |
| *** woodard has joined #openstack-ansible | 04:22 | |
| *** woodard has quit IRC | 04:27 | |
| *** udesale has quit IRC | 04:40 | |
| *** udesale has joined #openstack-ansible | 04:41 | |
| *** zerda3 has joined #openstack-ansible | 04:44 | |
| *** zerda3 has quit IRC | 04:57 | |
| *** thorst has joined #openstack-ansible | 05:15 | |
| *** thorst has quit IRC | 05:20 | |
| *** Oku_OS-away is now known as Oku_OS | 06:09 | |
| *** tobberydberg has joined #openstack-ansible | 06:10 | |
| *** drifterza has joined #openstack-ansible | 06:23 | |
| *** woodard has joined #openstack-ansible | 06:24 | |
| *** woodard has quit IRC | 06:28 | |
| *** perniciouscaffei has quit IRC | 06:34 | |
| *** jvidal has joined #openstack-ansible | 06:35 | |
| *** udesale__ has joined #openstack-ansible | 06:41 | |
| *** cshen has joined #openstack-ansible | 06:42 | |
| *** cshen has quit IRC | 06:42 | |
| *** udesale has quit IRC | 06:43 | |
| *** udesale__ has quit IRC | 06:48 | |
| *** udesale has joined #openstack-ansible | 06:49 | |
| *** flaviosr has quit IRC | 06:52 | |
| *** flaviosr has joined #openstack-ansible | 06:52 | |
| *** armaan has joined #openstack-ansible | 07:01 | |
| *** charcol has quit IRC | 07:02 | |
| *** armaan has quit IRC | 07:02 | |
| *** armaan has joined #openstack-ansible | 07:03 | |
| *** shardy has joined #openstack-ansible | 07:07 | |
| *** mbuil has joined #openstack-ansible | 07:15 | |
| *** thorst has joined #openstack-ansible | 07:16 | |
| *** markus_z has joined #openstack-ansible | 07:18 | |
| *** yolanda has joined #openstack-ansible | 07:20 | |
| *** thorst has quit IRC | 07:21 | |
| *** udesale__ has joined #openstack-ansible | 07:22 | |
| *** udesale has quit IRC | 07:23 | |
| *** mpranjic has quit IRC | 07:23 | |
| *** woodard has joined #openstack-ansible | 07:25 | |
| *** lvdombrkr has joined #openstack-ansible | 07:25 | |
| *** mpranjic has joined #openstack-ansible | 07:28 | |
| *** woodard has quit IRC | 07:29 | |
| mpranjic | morning | 07:30 |
| *** udesale__ has quit IRC | 07:42 | |
| *** udesale has joined #openstack-ansible | 07:42 | |
| *** xdfil has quit IRC | 07:45 | |
| lvdombrkr | morning, i have question about openstack-ansible re-deployment... if after deployment i want change some data in user_variables.yml what i need to do after data changed? depploy all infrostructure for begining or what? | 07:47 |
| andymccr | lvdombrkr: it depends on what you have changed - for example if you just want to change the nova_memory_allocation_ratio thats just a conf change and service restart within nova, so you could rerun the nova playbook only, or re-run it with the "nova-config" ansible tag to only adjust the config | 07:50 |
| andymccr | in general you shoudlnt need to redeploy unless you are changing some architectural that impacts everything (but then i'd question whether that change is required!) - saying that sometimes a redeploy is easier when you havent even got the deploy up and running, or if its just a test AIO and you are changing something really large. | 07:57 |
| *** jafeha has joined #openstack-ansible | 08:08 | |
| evrardjp | good morning | 08:12 |
| eoseq | morning, do you know how to disable setting "physical_interface_mappings" on a compute node and have it only on "neutron_l3_agent" group? | 08:13 |
| andymccr | eoseq: what are you trying to do by changing that? | 08:17 |
| *** markvoelker has joined #openstack-ansible | 08:17 | |
| evrardjp | yeah, what lead you to do that, eoseq? | 08:19 |
| *** markvoelker_ has joined #openstack-ansible | 08:19 | |
| eoseq | I have such settings: http://paste.openstack.org/show/617643/ | 08:21 |
| eoseq | so after ansibling I got "physical_interface_mappings = routers01:ethv22" in /etc/neutron/plugins/ml2/linuxbridge_agent.ini | 08:22 |
| eoseq | on compute nodes, but I want it to have it only on the l3 | 08:23 |
| *** markvoelker has quit IRC | 08:23 | |
| eoseq | on the compute node I get: ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface ethv22 for physical network routers01 does not exist. Agent terminated! | 08:24 |
| evrardjp | eoseq: our openstack_user_config already helps you by setting up the container interfaces, and their carrying interface on the host | 08:25 |
| evrardjp | you shouldn't need this user_variables. | 08:25 |
| *** woodard has joined #openstack-ansible | 08:25 | |
| eoseq | evrardjp: I can't find how to set network_mappings: "routers01:ethv22" in openstack_user_config.yml | 08:26 |
| evrardjp | have a look at the host bind overrides too | 08:26 |
| eoseq | evrardjp: ok, so you mean that host overrides it the best/only way? | 08:26 |
| eoseq | is | 08:26 |
| evrardjp | no i meant global_overrides with provider_networks are good enough | 08:27 |
| evrardjp | see for example: https://docs.openstack.org/project-deploy-guide/openstack-ansible/ocata/app-config-test.html#test-environment-config | 08:27 |
| *** woodard has quit IRC | 08:30 | |
| openstackgerrit | Manuel Buil proposed openstack/openstack-ansible-os_tacker master: Fill the tests directory https://review.openstack.org/490440 | 08:30 |
| eoseq | evrardjp: ok, so what I don't have now is the idea what variable should I put into that "global_overrides/providers_network/network" path to get it converted to "physical_interface_mappings" in the "neutron_l3_agent" group | 08:32 |
| evrardjp | let's take a step back again. | 08:33 |
| evrardjp | I know that's what you are trying to achieve, but why? | 08:33 |
| evrardjp | Isn't what ansible configures for you correct? | 08:34 |
| evrardjp | do you have one specific host that's different? Why is it different? Is that a good long term approach? | 08:34 |
| evrardjp | eoseq: because ansible will generate the physical_interface_mappings for you | 08:36 |
| evrardjp | let me show you | 08:36 |
| evrardjp | that is where we run neutron and gether provider networks data beforehand: https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-playbooks/neutron.yml#L69-L88 | 08:39 |
| evrardjp | and | 08:39 |
| evrardjp | https://github.com/openstack/openstack-ansible-os_neutron/blob/a5357ecac5d25e9623e2dbf6fc2130454160cb41/templates/plugins/ml2/linuxbridge_agent.ini.j2 | 08:39 |
| evrardjp | is where we generate the physical_interface_mappings for ml2 lxb for those neutron_provider_networks | 08:39 |
| evrardjp | eoseq: with that in mind, global_overrides probably give you what you need. | 08:41 |
| eoseq | so it's configured on every linuxbridge agent | 08:41 |
| evrardjp | yeah, ovs do the same IIRC | 08:42 |
| evrardjp | but you are LXB right? | 08:42 |
| eoseq | yes, LXB | 08:42 |
| evrardjp | ok | 08:42 |
| evrardjp | so? | 08:42 |
| evrardjp | There is something I still don't understand :D | 08:42 |
| eoseq | so I have to have that "external/public" bridge on every compute node | 08:43 |
| evrardjp | oh ok now I understand your problem | 08:44 |
| eoseq | but I think I need that only on l3 agents | 08:44 |
| eoseq | so i.e. infra hosts | 08:44 |
| evrardjp | so how do you intend to communicate between your compute node and your l3 agent on this network? | 08:45 |
| eoseq | vxlan? | 08:45 |
| evrardjp | ok | 08:45 |
| evrardjp | so | 08:45 |
| evrardjp | why don't you try to change the group_binds for the flat/br-vlan interfaces? | 08:46 |
| evrardjp | I have never tried myself, but that's what I'd try to do. | 08:47 |
| evrardjp | We should probably document this | 08:47 |
| evrardjp | for now we require br-vlan to be defined everywhere though: https://docs.openstack.org/project-deploy-guide/openstack-ansible/ocata/overview-network-arch.html | 08:48 |
| eoseq | evrardjp: thx, I'll try to play with a little bit more and will let you know :-) | 08:50 |
| eoseq | the idea is to have br-vxlan on compute nodes and route it to flat network on l3 agent nodes | 08:51 |
| evrardjp | you're heading down an undocumented path, but I understand why. We should technically be able to do it. If not, we should implement it. | 08:52 |
| evrardjp | yeah I guess you want to have a vtep in neutron agent container, and from then on speak to outside world | 08:52 |
| evrardjp | it makes sense. | 08:52 |
| evrardjp | double check your flat is well wired too on your neutron agents containers. | 08:53 |
| eoseq | so as for now that only way is to override infra host setting to keep it empty on compute nodes? :-) | 08:53 |
| eoseq | works like a charm when I delete that | 08:54 |
| eoseq | "physical_interface_mappings" on a compute node | 08:54 |
| eoseq | evrardjp: thanks a lot for trying to understand our case :-) | 08:55 |
| yifei | Hi all, I want to create the LXCs locally (using a local image). How can I achieve this in OSA? Thanks | 08:59 |
| yifei | or is there anyone can help with building a lxc image server? Thanks | 09:00 |
| evrardjp | lxc doesn't have a notion of "image server" itself yifei | 09:01 |
| evrardjp | lxc has a template that downloads from an external source though | 09:01 |
| evrardjp | yifei: what are you trying to achieve? | 09:01 |
| evrardjp | if you want to have an webserver that holds your images, and point to it, it's possible. | 09:02 |
| yifei | evrardjp: but i found that "lxc_container_download_template_options" indicates the --server {{ lxc_image_cache_server }} | 09:02 |
| evrardjp | yeah the download template will automatically fetch from ubuntu server, but you can use this variable to bypass it indeed, and use your own server. | 09:03 |
| evrardjp | advanced topic :p | 09:03 |
| yifei | evrardjp: yes I have. I have build all the deb and pip packages in the local docker. I would use it as a local source, so i can deploy osa offline | 09:03 |
| openstackgerrit | Manuel Buil proposed openstack/openstack-ansible-os_tacker master: Fill the tests directory https://review.openstack.org/490440 | 09:04 |
| yifei | evrardjp: how to build a lxc image server? | 09:04 |
| yifei | evrardjp: i have googled it, but no guide | 09:05 |
| yifei | evrardjp: sorry, got something now. Will try. Thanks for your help | 09:07 |
| evrardjp | yifei: sorry I was busy | 09:11 |
| yifei | evrardjp: got it now, thank you again, hah | 09:11 |
| evrardjp | ok | 09:11 |
| evrardjp | well, don't hesitate to share how you did it. We probably did it differently, so it's maybe better to come with one single approach in the future. | 09:12 |
| odyssey4me | yifei the lxc server has a very simple API to serve images, you can reverse engineer it by sniffing the traffic when creating instances using the default tooling | 09:12 |
| odyssey4me | we will very likely be doing some work in Queens to move towards an approach where everything is built, then deployed rather than the current approach | 09:13 |
| odyssey4me | yifei if you'd like to see how we've put together artifacts for newton/ocata so far, feel free to inspect https://github.com/rcbops/rpc-openstack/tree/master/scripts/artifacts-building | 09:14 |
| yifei | odyssey4me: I just found a command : lxc image copy | 09:14 |
| yifei | odyssey4me: use it to copy the image to my webserver and configure it in lxc-create | 09:15 |
| evrardjp | yifei: pay attention to lxc command! | 09:15 |
| evrardjp | lxc command is part of lxd tooling. | 09:16 |
| evrardjp | and lxd requires different metadata. | 09:16 |
| evrardjp | if you want pure lxc metadata, check at what odyssey4me pasted you. | 09:16 |
| yifei | odyssey4me: wow it's great is there any doc or readme about it ? | 09:16 |
| yifei | evrardjp: currently just use lxc | 09:17 |
| *** thorst has joined #openstack-ansible | 09:17 | |
| evrardjp | yifei: we don't have docs at the moment. | 09:18 |
| evrardjp | for that | 09:18 |
| odyssey4me | yifei the code is fairly well documented, but as that's a short term hack it is as it is | 09:19 |
| yifei | odyssey4me: found the readme, checking its detail now | 09:19 |
| yifei | odyssey4me: I took a lot time to build a webserver in docker for offline deployment of OSA..... | 09:20 |
| yifei | odyssey4me: i am nearly crazy now...... | 09:20 |
| odyssey4me | yes, an offline build is not simple at this time - it'd be ideal, though, to document everything needed so that we can share and improve it | 09:21 |
| yifei | odyssey4me: how it could be good it if I asked it here eralier | 09:21 |
| *** thorst has quit IRC | 09:22 | |
| evrardjp | yeah :) | 09:24 |
| yifei | odyssey4me: I wrote some scripts to scan the distro and pip packages in every role and used docker to download them and then used the packages to build a PPA and a pip repo..... | 09:25 |
| odyssey4me | yifei we already have a lookup that does that scan - you didb't need to write anything ;) | 09:25 |
| evrardjp | ... | 09:26 |
| odyssey4me | also, the pip repo is built as part of the normal deployment - so you could have just used that | 09:26 |
| odyssey4me | and the repo also contains a cache, so that could have perhaps been captured and used | 09:26 |
| yifei | odyssey4me: i did not know that.. I found it in OSA wiki but did not found it, so i wrote them myself | 09:26 |
| *** woodard has joined #openstack-ansible | 09:26 | |
| evrardjp | OSA wiki? | 09:27 |
| yifei | evrardjp: openstack docs about osa | 09:27 |
| yifei | https://docs.openstack.org/openstack-ansible/latest/ | 09:28 |
| evrardjp | oh the docs. | 09:28 |
| evrardjp | ok | 09:28 |
| evrardjp | yeah | 09:28 |
| evrardjp | offline install to be improved next cycle. | 09:28 |
| *** fxpester has joined #openstack-ansible | 09:29 | |
| *** stuartgr has joined #openstack-ansible | 09:30 | |
| *** woodard has quit IRC | 09:31 | |
| andymccr | make it so evrardjp. make it so! | 09:37 |
| evrardjp | yeah. I'm looking forward to it :D | 09:39 |
| *** jafeha has quit IRC | 09:46 | |
| *** charcol has joined #openstack-ansible | 09:49 | |
| *** chhavi has joined #openstack-ansible | 09:51 | |
| mbuil | guys, I need a bit of help. I am trying to fix the tests executed in the tacker role using Vagrant. Things are deployed successfully but the tests fails. How can I run the tests manually? I am inside my "infra1" container of the Vagrant VM but I cannot find any ansible command or my tacker role repo with the playbooks :( | 10:03 |
| *** thorst has joined #openstack-ansible | 10:05 | |
| openstackgerrit | Andy McCrae proposed openstack/openstack-ansible-os_sahara master: Implement uWSGI for sahara-api https://review.openstack.org/491421 | 10:06 |
| *** charcol has quit IRC | 10:08 | |
| andymccr | mbuil: the test is just doing a curl on http://localhost:8888 from inside the tacker containers | 10:08 |
| andymccr | mbuil: you could try that manually and see what response you get (should be a 300 based on the test you setup) | 10:08 |
| mbuil | andymccr: yes, but can I just run that test? | 10:09 |
| mbuil | andymccr: I mean, can I run the playbook "test-tacker-functional.yml"? | 10:09 |
| andymccr | mbuil: sure an easy way would be to edit the tests/test.yml and comment out all the includes except the test-tacker-functional.yml and then do a ./run_tests.sh | 10:10 |
| *** thorst has quit IRC | 10:11 | |
| mbuil | andymccr: ok. But there is no way to do "ansible-playbook test-tacker-functional.yml" inside the infra1 container or the tacker container? | 10:11 |
| *** deadnull has joined #openstack-ansible | 10:11 | |
| openstackgerrit | Manuel Buil proposed openstack/openstack-ansible-os_tacker master: Fill the tests directory https://review.openstack.org/490440 | 10:13 |
| *** deadnull has quit IRC | 10:13 | |
| andymccr | mbuil: wouldnt be inside the infra1 container, since thats where keystone/rabbit etc are deployed. | 10:15 |
| andymccr | the tests/files etc would exist outside of the infra1 container | 10:15 |
| mbuil | andymccr: ok | 10:15 |
| *** yifei has quit IRC | 10:17 | |
| *** lvdombrkr has quit IRC | 10:19 | |
| *** lvdombrkr has joined #openstack-ansible | 10:19 | |
| lvdombrkr | guys, last weeek i created and lounch first instance without any problems today when i tried tou lounch istance i get error: Message | 10:27 |
| lvdombrkr | No valid host was found. There are not enough hosts available. | 10:27 |
| *** woodard has joined #openstack-ansible | 10:27 | |
| lvdombrkr | there is no problems with resources on node | 10:27 |
| *** woodard has quit IRC | 10:31 | |
| andymccr | lvdombrkr: check your nova logs - i think those logs go in the scheduler or the api-os-compute log. | 10:34 |
| andymccr | lvdombrkr: it should tell you why its filtering out the host | 10:34 |
| *** cshen has joined #openstack-ansible | 10:35 | |
| lvdombrkr | andymccr: nova_scheduler node log http://paste.openstack.org/raw/617667/ | 10:41 |
| *** pbandark has joined #openstack-ansible | 10:42 | |
| andymccr | lvdombrkr: you'll need to learn how to debug the logs :) but that one looks like AggregateDiskFilter: (start: 1, end: 0)'] so something to do with the disk space on the compute host | 10:42 |
| lvdombrkr | andymccr: nova_conductor logs: http://paste.openstack.org/raw/617669/ | 10:45 |
| andymccr | lvdombrkr: the scheduler filters hosts out based on criteria - https://docs.openstack.org/nova/latest/user/filter-scheduler.html | 10:47 |
| mbuil | andymccr: finally it worked :). Thanks for your suggestion ==> https://review.openstack.org/#/c/490440/ | 10:47 |
| andymccr | mbuil: no problem! | 10:47 |
| andymccr | mbuil: makes sense that its a 200 instead of a 300 i guess :) | 10:47 |
| mbuil | andymccr: yes :) | 10:47 |
| *** markvoelker has joined #openstack-ansible | 10:49 | |
| *** markvoelker_ has quit IRC | 10:53 | |
| *** markvoelker_ has joined #openstack-ansible | 10:54 | |
| *** markvoelker has quit IRC | 10:54 | |
| *** markvoelker has joined #openstack-ansible | 10:57 | |
| *** schwicht has joined #openstack-ansible | 10:58 | |
| *** markvoelker has quit IRC | 10:59 | |
| *** markvoelker_ has quit IRC | 11:00 | |
| *** markvoelker has joined #openstack-ansible | 11:01 | |
| *** strigazi_OFF is now known as strigazi | 11:05 | |
| *** markvoelker_ has joined #openstack-ansible | 11:12 | |
| *** markvoelker has quit IRC | 11:15 | |
| lvdombrkr | andymccr:thank you, but looks there is no problem with disk space | 11:20 |
| andymccr | lvdombrkr: you'll need to check why that filter is rejecting the host then - perhaps you have some builds in error that are counting towards reserved space on the host | 11:21 |
| andymccr | or some other issue along those lines | 11:21 |
| *** markvoelker_ has quit IRC | 11:28 | |
| *** thorst has joined #openstack-ansible | 11:28 | |
| *** drifterza has left #openstack-ansible | 11:29 | |
| *** markvoelker has joined #openstack-ansible | 11:29 | |
| *** drifterza has joined #openstack-ansible | 11:29 | |
| *** drifterza has left #openstack-ansible | 11:30 | |
| *** drifterza has joined #openstack-ansible | 11:32 | |
| *** thorst has quit IRC | 11:33 | |
| *** markvoelker has quit IRC | 11:34 | |
| openstackgerrit | Andy McCrae proposed openstack/openstack-ansible-os_ironic master: Implement uWSGI for ironic-api https://review.openstack.org/491454 | 11:34 |
| eoseq | evrardjp: if there is no override for linuxbridge_agent.ini.j2 here: https://docs.openstack.org/project-deploy-guide/openstack-ansible/ocata/app-advanced-config-override.html#currently-available-overrides the only way is to modify the template? | 11:35 |
| *** markvoelker has joined #openstack-ansible | 11:40 | |
| *** markvoelker_ has joined #openstack-ansible | 11:41 | |
| *** schwicht has quit IRC | 11:42 | |
| *** schwicht has joined #openstack-ansible | 11:42 | |
| *** markvoelker_ has quit IRC | 11:42 | |
| *** markvoelker_ has joined #openstack-ansible | 11:44 | |
| *** markvoelker has quit IRC | 11:44 | |
| odyssey4me | eoseq that list is likely out of date, every ini file has an override | 11:44 |
| eoseq | odyssey4me: great, thx! :-) | 11:45 |
| odyssey4me | check the defaults for the role to find the current list | 11:45 |
| odyssey4me | we should probably remove that list from the docs and tell people to do that | 11:45 |
| eoseq | neutron_linuxbridge_agent_init_overrides: {} | 11:45 |
| odyssey4me | if you could push a patch for that, or register a bug for it, that's be great | 11:45 |
| *** markvoelker has joined #openstack-ansible | 11:47 | |
| eoseq | odyssey4me: ok, I'll submit a bug | 11:47 |
| *** pbandark1 has joined #openstack-ansible | 11:49 | |
| *** schwicht has quit IRC | 11:49 | |
| *** markvoelker_ has quit IRC | 11:50 | |
| lvdombrkr | andymccr: thank you, can i try just reboot nova containers, or its senseless? | 11:50 |
| *** markvoelker_ has joined #openstack-ansible | 11:51 | |
| *** holmsten has joined #openstack-ansible | 11:51 | |
| *** pbandark has quit IRC | 11:51 | |
| *** pbandark1 is now known as pbandark | 11:51 | |
| *** shardy is now known as shardy_lunch | 11:51 | |
| *** markvoelker has quit IRC | 11:52 | |
| *** markvoelker has joined #openstack-ansible | 11:53 | |
| *** markvoelker_ has quit IRC | 11:56 | |
| *** markvoelker_ has joined #openstack-ansible | 11:56 | |
| andymccr | lvdombrkr: well it'd be the compute service on the host - but yes probably, you need to figure out why nova thinks you dont have enough space | 11:57 |
| *** cshen has quit IRC | 11:59 | |
| *** markvoelker has quit IRC | 12:00 | |
| *** markvoelker has joined #openstack-ansible | 12:02 | |
| *** cshen has joined #openstack-ansible | 12:03 | |
| *** markvoelker_ has quit IRC | 12:03 | |
| *** thorst has joined #openstack-ansible | 12:05 | |
| jvidal | hello | 12:08 |
| jvidal | I am getting failures on vagrant tests, and I saw some gate checks with the same failure | 12:09 |
| jvidal | tempest does not run due to missing dependencies | 12:09 |
| jvidal | is anybody else having the same errors? | 12:09 |
| *** thorst has quit IRC | 12:09 | |
| jvidal | http://logs.openstack.org/43/490543/3/check/gate-openstack-ansible-os_neutron-ansible-func-ubuntu-xenial/b16350e/console.html.gz | 12:10 |
| jvidal | example of the failure ^ | 12:10 |
| *** mgariepy has quit IRC | 12:10 | |
| *** thorst has joined #openstack-ansible | 12:10 | |
| *** cshen has quit IRC | 12:12 | |
| *** gkadam has quit IRC | 12:22 | |
| lvdombrkr | andymccr: in api_placement container in logs i found : 2017/08/07 15:21:42 [info] 136#0: *82 recv() failed (104: Connection reset by peer) while sending response to client, client: 172.16.84.124, server: , request: "HEAD / HTTP/1.0" | 12:24 |
| lvdombrkr | can it be releated with my issue? | 12:24 |
| *** cshen has joined #openstack-ansible | 12:24 | |
| *** randy^^ has joined #openstack-ansible | 12:26 | |
| *** prometheanfire has quit IRC | 12:27 | |
| randy^^ | hello/good afternoon. | 12:27 |
| *** woodard has joined #openstack-ansible | 12:29 | |
| *** schwicht has joined #openstack-ansible | 12:32 | |
| *** woodard has quit IRC | 12:33 | |
| *** cshen has quit IRC | 12:35 | |
| *** schwicht has quit IRC | 12:39 | |
| *** mgariepy has joined #openstack-ansible | 12:42 | |
| *** hw_wutianwei has quit IRC | 12:44 | |
| mgariepy | good morning everyone | 12:44 |
| *** schwicht has joined #openstack-ansible | 12:47 | |
| *** shardy_lunch is now known as shardy | 12:52 | |
| evrardjp | hello mgariepy! | 12:54 |
| mgariepy | what's up ? | 12:55 |
| *** esberglu has joined #openstack-ansible | 13:03 | |
| *** schwicht has quit IRC | 13:04 | |
| cloudnull | mornings | 13:10 |
| *** smatzek has joined #openstack-ansible | 13:11 | |
| *** lucasxu has joined #openstack-ansible | 13:12 | |
| mgariepy | odyssey4me, are you around ? | 13:16 |
| odyssey4me | o/ mgariepy cloudnull | 13:16 |
| mgariepy | so did you had time to dig out repo server issues ? | 13:17 |
| odyssey4me | mgariepy I haven't been able to do any further testing. I think someone had a review up though. | 13:17 |
| *** woodard has joined #openstack-ansible | 13:18 | |
| odyssey4me | mgariepy here it is https://review.openstack.org/489188 | 13:19 |
| *** woodard has quit IRC | 13:19 | |
| *** woodard has joined #openstack-ansible | 13:19 | |
| *** schwicht has joined #openstack-ansible | 13:20 | |
| mgariepy | ok great, i'll try to dig that out then. | 13:21 |
| cloudnull | are we seeing timeouts in general within the integrated gate? | 13:21 |
| odyssey4me | cloudnull yep, and even for roles tests | 13:24 |
| odyssey4me | without OSIC, it seems we're quite stuck | 13:24 |
| cloudnull | :'( | 13:25 |
| openstackgerrit | Andy McCrae proposed openstack/openstack-ansible-os_sahara master: Implement uWSGI for sahara-api https://review.openstack.org/491421 | 13:25 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Add subunit output to ARA reporting https://review.openstack.org/491489 | 13:27 |
| *** drifterza has quit IRC | 13:29 | |
| openstackgerrit | Andy McCrae proposed openstack/openstack-ansible-os_ironic master: Implement uWSGI for ironic-api https://review.openstack.org/491454 | 13:30 |
| mhayden | dmsimard: yeah, i was noticing some slow gates w/o osic :/ | 13:34 |
| mhayden | happy monday | 13:35 |
| cloudnull | I wonder if that is something being generally felt todya | 13:35 |
| cloudnull | or if it's just us | 13:35 |
| cloudnull | ^ maybe we could/should reintroduce the hyperconverge things to try push runtimes back down ? | 13:35 |
| *** lbragstad has joined #openstack-ansible | 13:36 | |
| dmsimard | FYI I made a thing http://grafana.openstack.org/dashboard/db/openstack-ansible | 13:38 |
| dmsimard | But I'm not sure it really works the way I think it was supposed to | 13:38 |
| odyssey4me | cloudnull I hate to inform you that the revert still hasn't merged, so your changes are still in place: https://review.openstack.org/471822 | 13:38 |
| * mhayden was wrestling with swift until late friday :/ | 13:39 | |
| odyssey4me | we would have to consolidate even more, and considering that the role tests are all converged onto a minimal set of containers, I don't think it'll help all that much | 13:39 |
| logan- | http://grafana.openstack.org/dashboard/db/openstack-ansible?panelId=2&fullscreen&from=1499521222557&to=1502113222557 nice last 30 days view of gate timings | 13:40 |
| odyssey4me | the information is a little weird due to the lack of periodics | 13:42 |
| odyssey4me | if we implemented per-repo periodics then that might be a little more consistent and useful | 13:43 |
| odyssey4me | but, I suppose we could build a dashboard for the periodic tests - I wasn't aware that this was available | 13:43 |
| logan- | it seems like there should be a lot more data points on that | 13:43 |
| logan- | maybe it is averaged per-day | 13:44 |
| dmsimard | odyssey4me: yeah I submitted those graphs just to try and experiment with the available data | 13:51 |
| dmsimard | I'm not sure why it's not picking up the centos jobs | 13:51 |
| dmsimard | surely something silly in https://git.openstack.org/cgit/openstack-infra/project-config/tree/grafana/openstack-ansible.yaml | 13:52 |
| andymccr | +2 to the revert patch! | 13:55 |
| andymccr | mhayden: yeah - mines timing out :( even with the fix to ensure the erasure code stuff doesnt impact | 13:55 |
| cloudnull | odyssey4me: ah, so... maybe we need to do more of it ? | 13:56 |
| odyssey4me | cloudnull given the role results, I don't think it'll make that much difference | 13:57 |
| cloudnull | ok | 13:57 |
| odyssey4me | it may or may not - no idea... th epoint is that the roles are already hyperconverged to use only two containers in most cases... and they're also timing out | 13:57 |
| odyssey4me | timing it with a shorter (1hr timeout), but still timing out | 13:57 |
| andymccr | well centos is taking ridiculous amounts of time | 13:58 |
| cloudnull | maybe we need to break up the installation into multiple scenarios ? | 13:58 |
| andymccr | i had a swift job time out after an hour and it hadnt even installed rabbitmq yet on the infra container | 13:58 |
| andymccr | which suggests that could impact any role test - since we do that for almost all of them | 13:58 |
| odyssey4me | the rabbit and mariadb roles could be radically simplified to improve performance | 13:58 |
| andymccr | as long as it has a centos job enabled | 13:58 |
| dmsimard | Aye, I needed to turn the OSA jobs to non-voting in the ARA gate this weekend because I was clocking in on 8+ rechecks on a trivial patch | 13:58 |
| andymccr | dmsimard: :( sorry | 13:58 |
| mhayden | andymccr: i put in a patch to remove rabbitmq from the swift gate | 13:58 |
| andymccr | dmsimard: is that on the integrated test build? | 13:58 |
| mhayden | but it won't pass :| | 13:58 |
| dmsimard | so I'm not sure if there is any one problem or if the timeout just needs to be bumped | 13:58 |
| *** asettle_ is now known as asettle | 13:59 | |
| dmsimard | andymccr: no, osa-tests | 13:59 |
| andymccr | dmsimard: what is the timeout? | 13:59 |
| odyssey4me | the timeout is an hour for the roles | 13:59 |
| dmsimard | andymccr: different places, see here https://review.openstack.org/#/c/491029/ | 13:59 |
| odyssey4me | and osa-tests only does two services IIRC | 13:59 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Revert "Reduce container counts for cinder, ironic and trove" https://review.openstack.org/471822 | 13:59 |
| dmsimard | ok, I lie, maybe it's getting timeouts running tempest | 14:00 |
| dmsimard | http://logs.openstack.org/29/491029/1/gate/gate-ara-integration-openstack-ansible-func-centos-7/a27f9c5/console.html#_2017-08-06_06_11_50_848866 | 14:00 |
| dmsimard | http://logs.openstack.org/29/491029/1/gate/gate-ara-integration-openstack-ansible-func-centos-7/d302dba/console.html#_2017-08-06_02_03_01_872043 | 14:00 |
| dmsimard | but it starts running tempest with like 5 minutes left.. | 14:00 |
| dmsimard | sounds awfully slow | 14:00 |
| dmsimard | this one failed with timeout but just barely completed so it has logs http://logs.openstack.org/29/491029/1/check/gate-ara-integration-openstack-ansible-func-centos-7/748d366/console.html | 14:02 |
| dmsimard | sorting by duration in ara, we're seeing tempest tests at ~11 minutes but there is a lot of package installation tasks taking a long time | 14:03 |
| andymccr | dmsimard: for the purposes of ARA, do you need the CentOS gate? we are having issues with that - or have you seen the ubuntu gate fail too? | 14:04 |
| dmsimard | there's almost 30 minutes worth just of yum install and pip install in there | 14:04 |
| odyssey4me | dmsimard yeah, the centos gate specifically is known to take a very long time and often time out | 14:04 |
| andymccr | dmsimard: yea! | 14:04 |
| andymccr | dmsimard: yeah we're battling that rn, i know mhayden was doing some work on that already | 14:04 |
| andymccr | but right now its still a bit bad and really inconsistent | 14:04 |
| dmsimard | yeah | 14:05 |
| dmsimard | tbh I don't *need* the OSA centos coverage so much as I like having coverage on a deb and a rpm distro | 14:05 |
| andymccr | dmsimard: agreed, for now though i think disabling the centos tests would work better :( until we can figure out the deal with that. | 14:06 |
| dmsimard | it's non-voting right now so it's okay, I'm not going to merge things if there is a legit issue | 14:06 |
| Tahvok | Hey guys! Currently I don't have any instances running on my environment (so I don't care about downtime). I would like to change the 'tunnel' cidr addresses. From what I saw only the neutron agent container has ip assigned from this cidr. How should I proceed on changing it? I've alreayd modified the openstack_user_config.yml file with correct cidr | 14:07 |
| andymccr | dmsimard: yeah - also from our perspective a lot will fail if that integrated test repo fails on centos :P | 14:08 |
| dmsimard | hm, I thought passing a list to "package" was supposed to do everything in one transaction ? | 14:08 |
| dmsimard | http://logs.openstack.org/29/491029/1/check/gate-ara-integration-openstack-ansible-func-centos-7/748d366/logs/ara/file/4ba2983d-4380-4ccb-8311-38f743706faf/#line-77 | 14:08 |
| dmsimard | yields http://logs.openstack.org/29/491029/1/check/gate-ara-integration-openstack-ansible-func-centos-7/748d366/logs/ara/result/9d5c9db8-4b97-4302-9b34-9b744c44d3dd/ | 14:09 |
| *** marst has joined #openstack-ansible | 14:09 | |
| *** cshen has joined #openstack-ansible | 14:14 | |
| *** perniciouscaffei has joined #openstack-ansible | 14:18 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible master: [WIP] Testing dnf with host/containers https://review.openstack.org/491084 | 14:23 |
| *** schwicht has quit IRC | 14:24 | |
| *** drifterza has joined #openstack-ansible | 14:25 | |
| lvdombrkr | andymccr: quick question after physical node restart all services must be up and running automatically? | 14:26 |
| randy^^ | mhm, is there any documentation about "how to contribute to openstack-ansible"? | 14:31 |
| *** hw_wutianwei has joined #openstack-ansible | 14:32 | |
| cloudnull | lvdombrkr: yes the services should all automatically restart | 14:34 |
| cloudnull | randy^^: what are you looking for? | 14:35 |
| cloudnull | we've the developer docs | 14:35 |
| cloudnull | https://docs.openstack.org/openstack-ansible/newton/newton/developer-docs/contribute.html | 14:35 |
| spotz | +2 on the docs:) | 14:35 |
| cloudnull | if that's what you are looking for ? | 14:35 |
| evrardjp | randy^^: contributions welcomed :) | 14:36 |
| *** schwicht has joined #openstack-ansible | 14:37 | |
| spotz | And there's some good and improving docs for getting setup for the community as a hole but can help with that if you get stuck | 14:37 |
| *** schwicht has quit IRC | 14:37 | |
| openstackgerrit | Logan V proposed openstack/openstack-ansible master: Run resolvconf role on hosts only after unbound is deployed https://review.openstack.org/491508 | 14:37 |
| odyssey4me | cloudnull that would unfortunately be some sort of outdated thing, this is the link: https://docs.openstack.org/openstack-ansible/latest/contributor/index.html | 14:37 |
| evrardjp | I think everythink is linked from that page spotz :) | 14:37 |
| *** armaan has quit IRC | 14:37 | |
| evrardjp | if not, let's fix it. | 14:38 |
| *** armaan has joined #openstack-ansible | 14:38 | |
| spotz | evrardjp: thingee is working on something new and cool:) | 14:38 |
| evrardjp | yeah I remember that part. We could link that from our guides. | 14:38 |
| odyssey4me | it's two links deep from the index page | 14:39 |
| odyssey4me | that's hardly worth changing the index for | 14:40 |
| randy^^ | cloudnull: thanks. just want to report som ebugs - or better, patches :) | 14:42 |
| *** armaan has quit IRC | 14:42 | |
| randy^^ | just minor things, missing environment settings in various roles | 14:42 |
| *** woodard has quit IRC | 14:43 | |
| cloudnull | cool | 14:44 |
| cloudnull | well not cool that there are bugs , but nice to get them reported and even better to get them fixed :) | 14:44 |
| evrardjp | randy^^: yeah don't hesitate to report bugs and even patch them, according to practices! :D | 14:44 |
| *** schwicht has joined #openstack-ansible | 14:45 | |
| evrardjp | we can help you in any case. | 14:45 |
| *** tobberyd_ has joined #openstack-ansible | 14:45 | |
| evrardjp | there are variables that we don't define explicitly though. | 14:45 |
| randy^^ | btw - is anybody of you from rackspace germany, sitting in munich office? | 14:46 |
| * cloudnull US | 14:46 | |
| evrardjp | I don't think so | 14:47 |
| evrardjp | randy^^: are you from germany? | 14:47 |
| randy^^ | we swapped "office rooms" last month - RS DE got our 4 former offices and we (company) took theire 2 rooms :) | 14:47 |
| randy^^ | i should get some nerfguns as well, than we can fight against them "over the floor" :> | 14:48 |
| randy^^ | evrardjp: yes, munich. | 14:48 |
| *** tobberydberg has quit IRC | 14:48 | |
| *** tobberyd_ has quit IRC | 14:50 | |
| holmsten | Hello! I'm looking for best practices for using OSA to deploy Openstack with 3rd party neutron agent and ml2 driver (more specifically Cisco ACI OpenStack Plugin). I'm afraid this isn't possible without rewriting current os_neutron playbook? | 14:52 |
| dmsimard | andymccr: there's really something weird going on with those package resources being split into different transactions | 14:55 |
| dmsimard | seems like #ansible-devel is of my opinion that there should not be different transactions | 14:56 |
| evrardjp | holmsten: you could contribute to our os_neutron role | 14:57 |
| evrardjp | there are already different 3rd party neutron agents. | 14:57 |
| logan- | why does our specs page only show newton and older http://specs.openstack.org/openstack/openstack-ansible-specs/ | 14:57 |
| evrardjp | wow. | 14:58 |
| evrardjp | good catch logan- | 14:58 |
| odyssey4me | dmsimard I think you may be seeing what made mhayden decide that dnf was the answer. | 14:58 |
| logan- | yeah i was just going to link dmsimard to the dnf spec | 14:58 |
| logan- | hhe | 14:58 |
| logan- | heh * | 14:58 |
| evrardjp | holmsten: you could also have your thing on the side, and tap into the process, but I think it would be a great contribution to the community to adapt the os_neutron role. | 14:58 |
| mhayden | dmsimard / odyssey4me: we have a few straggling package tasks that aren't passing as a list :/ | 14:58 |
| odyssey4me | logan- 'cos we need to update the index I reckon | 14:59 |
| mhayden | which is why i added that linter | 14:59 |
| logan- | dmsimard: http://docs-draft.openstack.org/24/488424/1/gate/gate-openstack-ansible-specs-docs-ubuntu-xenial/c933a16//doc/build/html/specs/pike/centos-and-dnf.html | 14:59 |
| evrardjp | mhayden: you added a linter? | 14:59 |
| mhayden | https://review.openstack.org/490899 | 14:59 |
| mhayden | jmccrory is excited about it, but it needs more reviews ;) | 14:59 |
| logan- | mhayden: he put an example above that looked like it was using a list though and doing some weird stuff | 14:59 |
| dmsimard | mhayden: yeah but even though we're passing items as a list instead of with_items, it looks like the underlying module is still passing it as sort-of with_items | 14:59 |
| logan- | http://logs.openstack.org/29/491029/1/check/gate-ara-integration-openstack-ansible-func-centos-7/748d366/logs/ara/file/4ba2983d-4380-4ccb-8311-38f743706faf/#line-77 | 15:00 |
| evrardjp | marked | 15:00 |
| logan- | -> http://logs.openstack.org/29/491029/1/check/gate-ara-integration-openstack-ansible-func-centos-7/748d366/logs/ara/result/9d5c9db8-4b97-4302-9b34-9b744c44d3dd/ | 15:00 |
| mhayden | swift still has a with_items on a package task -- can't get it merged :| | 15:00 |
| dmsimard | mhayden: yeah see logan-'s links | 15:00 |
| odyssey4me | logan- oh weird - there's a pike index and spec, but not published | 15:00 |
| * mhayden is about ready to just call apt/yum/dnf/zypper directly with shell :P | 15:00 | |
| logan- | haha | 15:00 |
| dmsimard | mhayden: ara is picking up multiple results just as if the task used with_items | 15:00 |
| mhayden | weird | 15:00 |
| evrardjp | mhayden: I love it | 15:00 |
| mhayden | logan-: well, it all goes in one transaction,b ut ansible parses the output multiple times | 15:01 |
| mhayden | based on what i saw in the plugin code | 15:01 |
| logan- | oh right | 15:01 |
| mhayden | evrardjp: feel free to rub a little workflow on that one if you like it | 15:01 |
| mhayden | logan-: ansible's yum/dnf modules read over that output bunches of times | 15:01 |
| mhayden | oh, btw -- i updated https://github.com/major/osa-gate-parser so that it will output tasks that time out | 15:02 |
| mhayden | example -> https://paste.fedoraproject.org/paste/eBSjXRp223IMz4UlX7clng/raw | 15:02 |
| *** tobberydberg has joined #openstack-ansible | 15:02 | |
| logan- | very nice | 15:03 |
| *** markvoelker has quit IRC | 15:03 | |
| mhayden | evrardjp: i've gotta read up on how to write them linter tests | 15:04 |
| odyssey4me | mhayden until the erasure encoding disable is merged, centos for swift is broken | 15:05 |
| odyssey4me | and that patch is failing due to some weird bindep thing | 15:05 |
| mhayden | i might have to combine that with mine | 15:05 |
| mhayden | the rabbitmq removal one | 15:05 |
| *** markvoelker has joined #openstack-ansible | 15:06 | |
| evrardjp | mhayden: worst case scenario: call in our openstack-ansible-tests repo a linting script that runs a failing play on purpose, catch the output and return "OK" :p | 15:06 |
| mhayden | odyssey4me: let me see if i can combine a few of these together | 15:06 |
| dmsimard | mhayden: sweet | 15:07 |
| *** tobberydberg has quit IRC | 15:07 | |
| dmsimard | mhayden: I could go and say that ara provides that already but /me shrug | 15:07 |
| holmsten | evrardjp: Thanks. If we end up writing our own thing we will most likely push it upstream (assuming it's good enough) to contribute to the community. :) | 15:08 |
| evrardjp | please don't rewrite os_neutron if you write something external :p | 15:08 |
| dmsimard | mhayden: oh, wait, tasks that time out ? What do you mean ? | 15:09 |
| mhayden | dmsimard: so if a task is running with zuul kills it | 15:09 |
| dmsimard | oh, if the job timed out, not individual tasks | 15:09 |
| mhayden | the script catches that now | 15:09 |
| mhayden | si | 15:09 |
| *** drifterza has quit IRC | 15:09 | |
| evrardjp | dmsimard: also long running tasks can "timeout" | 15:10 |
| evrardjp | that's how async appeared with timeout: | 15:10 |
| evrardjp | IIRC | 15:10 |
| evrardjp | ssh timeout | 15:10 |
| evrardjp | due to( | 15:10 |
| evrardjp | grr | 15:10 |
| evrardjp | I can't type | 15:10 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_swift master: Optimize swift gate https://review.openstack.org/491089 | 15:10 |
| mhayden | andymccr: ^^ added your change | 15:10 |
| dmsimard | yeah and I guess ara report or database isn't available if job didn't collect logs | 15:11 |
| dmsimard | collecting logs should be part of the job publisher, though, not the actual job -- so that you can still retrieve logs if the job times out ? | 15:11 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_swift master: Optimize swift gate https://review.openstack.org/491089 | 15:11 |
| mhayden | okay, i jammed together three speedups for the swift gate -> https://review.openstack.org/491089 | 15:12 |
| odyssey4me | dmsimard yeah, we've always used an in-repo script so that we can modify it per-branch and not have to patch project-config all the time | 15:12 |
| * palendae grabs swift gate by the lapels and shakes it | 15:12 | |
| mhayden | if it fails, imma move to a van down by the river | 15:12 |
| *** Oku_OS is now known as Oku_OS-away | 15:12 | |
| dmsimard | odyssey4me: you can still use that in-repo script, but have a publisher call it instead of a job builder | 15:12 |
| odyssey4me | once zuul-v3 is ready for us we'll be able to achieve the same thing, but let it be in the post-test phase | 15:12 |
| mhayden | or i might move to canada | 15:12 |
| odyssey4me | or yeah, that's an option | 15:12 |
| dmsimard | mhayden: come say hi if you do o/ | 15:12 |
| odyssey4me | back then we didn't understand all this stuff | 15:13 |
| mhayden | will so | 15:13 |
| palendae | So you've rolled https://review.openstack.org/#/c/490462/ in to that? | 15:13 |
| mhayden | s/so/do/ | 15:13 |
| mhayden | si | 15:13 |
| mhayden | 491089 has the bindep update + packages as list + erasure code fix | 15:13 |
| palendae | A smorgasbord | 15:13 |
| mhayden | yeah | 15:13 |
| mhayden | hopefully it passes swiftly | 15:13 |
| palendae | No we use Python here | 15:14 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_swift master: Add dnf support https://review.openstack.org/490157 | 15:14 |
| palendae | Ok, what does 'dnf' stand for? | 15:14 |
| palendae | Cause I'm never gonna read it as anything but did not finish | 15:14 |
| *** nollide has joined #openstack-ansible | 15:14 | |
| evrardjp | mhayden: I don't know why I giggled on that joke. | 15:14 |
| mhayden | hah | 15:15 |
| mhayden | evrardjp: on the linter -- do we have tests for custom linters anywhere? | 15:15 |
| dmsimard | palendae: https://dnf.readthedocs.io/en/latest/user_faq.html#what-does-dnf-stand-for | 15:15 |
| odyssey4me | mhayden nope, so you can be the first to write one! | 15:15 |
| evrardjp | it's not a reason to not have them :D | 15:15 |
| mhayden | damn | 15:15 |
| mhayden | walked right into that one | 15:15 |
| evrardjp | odyssey4me: D | 15:15 |
| palendae | dmsimard: lol | 15:15 |
| evrardjp | good job there odyssey4me. | 15:15 |
| odyssey4me | what a great opportunity! | 15:15 |
| evrardjp | We didn't even sync before doing it. | 15:16 |
| openstackgerrit | Juan Vidal Allende proposed openstack/openstack-ansible-os_neutron master: Add option to deploy OpenDaylight SDN Controller https://review.openstack.org/480128 | 15:16 |
| odyssey4me | ok, now that I know what dnf stands for - my previous state remains | 15:16 |
| odyssey4me | but for a different reason | 15:16 |
| mbuil | odyssey4me: after fighting a bit, I managed to fix the tacker tests issues. Could you check once again please? Then I can continue with the other patch :) Thanks! ==> https://review.openstack.org/#/c/490440/ | 15:17 |
| *** jvidal has quit IRC | 15:22 | |
| odyssey4me | mbuil builting a functional test is hard, but once it works it keeps you honest :) | 15:36 |
| mbuil | odyssey4me: Yes, I feel proud :) | 15:37 |
| *** markus_z has quit IRC | 15:37 | |
| mbuil | odyssey4me: how long do we have until the "code freeze" for Pike? | 15:37 |
| odyssey4me | mbuil it's already too late to have tacker in the integrated release - a new repo has to be included for two milestones | 15:38 |
| *** holmsten has quit IRC | 15:38 | |
| mbuil | odyssey4me ok I was more thinking about the changes that jvidal did to integrate ODL in the neutron role. I have several patches that I wanted to add on top to provide SFC functionality. Right now, I am working on a private branch | 15:39 |
| odyssey4me | ah ok, as the patch is in review already and receiving some attention, it'll get in - no worries there | 15:40 |
| odyssey4me | you can base patches on top of a patch in review, although obviously rebasing gets tricky | 15:40 |
| mbuil | yep, that's why I was waiting for it to be merged before adding my patches | 15:41 |
| *** woodard has joined #openstack-ansible | 15:41 | |
| odyssey4me | mbuil looks fine, just a whitespace nit and a question in https://review.openstack.org/#/c/490440 which may need a comment added | 15:42 |
| mbuil | ok, good. I need to run but I'll probably create the patch then | 15:42 |
| *** mbuil has quit IRC | 15:42 | |
| *** cshen has quit IRC | 15:46 | |
| cloudnull | so looking over ara for our runtimes, it doesn't look like we're doing too bad in terms of tasks however the gate vms are swapping like crazy | 15:48 |
| cloudnull | http://paste.openstack.org/show/617694/ | 15:48 |
| cloudnull | odyssey4me: I wonder if we can reduce some of that. | 15:48 |
| cloudnull | ? | 15:48 |
| odyssey4me | yeah, OSIC was the only provider that wasn't overcommitted in terms of RAM and CPU | 15:48 |
| *** drifterza has joined #openstack-ansible | 15:50 | |
| cloudnull | even so, the vms themselves are swapping pretty heavily under test conditions. | 15:50 |
| odyssey4me | well, I expect that reducing the containers down to a smaller set will help - but that would mean continuing *and* finishing the hyperconvergence path | 15:51 |
| odyssey4me | which requires the spec | 15:51 |
| odyssey4me | unless anyone has better ideas | 15:51 |
| cloudnull | I've a local vm with 16GiB of ram and its still swapping which is quite impactful in terms of runtime | 15:51 |
| palendae | Yeah, I've got a few VMs that are maxing out at 16GiB too | 15:52 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-tests master: Add linter for passing pkgs as list https://review.openstack.org/490899 | 15:52 |
| mhayden | evrardjp: ^^ | 15:52 |
| *** markvoelker has quit IRC | 15:53 | |
| odyssey4me | perhaps the 'aio' scenario needs some converged env.d overrides | 15:54 |
| cloudnull | maybe so | 15:54 |
| cloudnull | i'm poking around | 15:54 |
| odyssey4me | we can then perhaps figure out what the right hyperconverged default is next cycle | 15:54 |
| cloudnull | maybe there are some tunables we can use | 15:54 |
| palendae | cloudnull: Weren't you looking at combining a bunch of the OpenStack API containers? | 15:54 |
| palendae | I seem to recall a mention of that | 15:54 |
| cloudnull | mhayden: and i were | 15:54 |
| odyssey4me | palendae yep, that's the 'hyperconverged' thing | 15:54 |
| palendae | Gotcha | 15:55 |
| * mhayden is running glance + nova hyperconverged on centos + ocata | 15:55 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara master: Add ceph_client role https://review.openstack.org/485000 | 15:55 |
| mhayden | the nova hyperconverged setup saves a bunch of hassle | 15:55 |
| odyssey4me | I'm totally in favor of moving to something far simpler - like one container per upstream project, or perhaps even simpler... but we need to ensure that we do so responsibly and provide an automated upgrade path. | 15:55 |
| palendae | ^ | 15:55 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible master: [WIP] Testing dnf with host/containers https://review.openstack.org/491084 | 15:56 |
| odyssey4me | if we move to doing JIT container creation, then we'd also be able to prep per-role rootfs bases and build the containers using cow | 15:57 |
| odyssey4me | that would be super-fast | 15:57 |
| odyssey4me | but that's work for queens really | 15:58 |
| *** hw_wutianwei has quit IRC | 16:00 | |
| cloudnull | another interesting thing going on in the gate | 16:01 |
| cloudnull | https://snag.gy/1o3pru.jpg | 16:01 |
| cloudnull | cinder seems to be freaking out | 16:01 |
| cloudnull | odyssey4me: I've got a patch up for JIT install of containers. | 16:01 |
| cloudnull | it recently passed 1 gate but timedout on the other. | 16:02 |
| odyssey4me | cloudnull yep, saw that - I was working on one too, but you got further than me... so once youre is in I'll add my adjustments on top which will optimise further | 16:02 |
| evrardjp | mhayden: oh nice well have a look | 16:02 |
| evrardjp | mhayden: maybe running the ansible-lint with -T could also help by showing the rule is effectively loaded | 16:03 |
| evrardjp | IIRC | 16:03 |
| evrardjp | let me see a linter result :D | 16:03 |
| mhayden | oh fudge, i missed half the commit | 16:04 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-tests master: Add linter for passing pkgs as list https://review.openstack.org/490899 | 16:04 |
| mhayden | now it should make more sense | 16:04 |
| cloudnull | odyssey4me: https://snag.gy/afoGuh.jpg | 16:07 |
| cloudnull | the first is with cinder having a panic attack | 16:07 |
| cloudnull | the second is after I killed it | 16:07 |
| cloudnull | I suspect this is a major cause of our gate situation | 16:08 |
| *** udesale has quit IRC | 16:08 | |
| *** udesale has joined #openstack-ansible | 16:08 | |
| cloudnull | the volume service and scheduler are being bad actors | 16:08 |
| *** woodard has quit IRC | 16:08 | |
| palendae | Hopefully there's a findable bug to report in there | 16:09 |
| *** woodard has joined #openstack-ansible | 16:09 | |
| cloudnull | i've not looked for a bug just yet, still investigating the issues | 16:09 |
| palendae | Yeah | 16:10 |
| logan- | quite a lot of swapping going on https://jenkins.lstn.net/job/openstack-upstream/job/openstack-ansible-check/82/artifact/logs/munin/localdomain/localhost.localdomain/memory.html | 16:10 |
| logan- | reducing memory usage would help some | 16:11 |
| cloudnull | after restarting the cinder services they seem to be behaving, i wonder if there's something we're doing that is causing it to run crazy? | 16:13 |
| palendae | cloudnull: Before or after tempest? | 16:14 |
| *** tobberydberg has joined #openstack-ansible | 16:14 | |
| *** drifterza has quit IRC | 16:18 | |
| *** tobberydberg has quit IRC | 16:18 | |
| andymccr | ahh mhayden nice work on the swift patch | 16:26 |
| mhayden | yw | 16:26 |
| andymccr | mhayden: thoughts on applying that with_items -> package change to all repos? now that we're hitting feature freeze im tempted to apply some standards like that | 16:26 |
| palendae | andymccr: +1 | 16:26 |
| mhayden | damn, so 490157 passed but 491089 is going to fail due to the lxc image download :| | 16:26 |
| *** cshen has joined #openstack-ansible | 16:26 | |
| mhayden | andymccr: well i did propose a linter ;) | 16:26 |
| mhayden | so we will know where it is | 16:27 |
| mhayden | or where those bad tasks are | 16:27 |
| mhayden | andymccr: https://review.openstack.org/490899 | 16:27 |
| andymccr | mhayden: why you stealing my ideas?! | 16:27 |
| andymccr | nice one :D | 16:27 |
| mhayden | gracias | 16:27 |
| andymccr | ok cool i'll take a look at that - but lets get some standards, maybe we can discuss, i was thinking the service_list thing would be a good one to apply too | 16:27 |
| palendae | mhayden: Is lxc image download broken at the mirror level? | 16:28 |
| palendae | I'm doing a kilo deploy for upgrade testing, and it won't get past downloading xenial amd64 images | 16:28 |
| mhayden | i think it flaked out during that one job :/ | 16:28 |
| palendae | (Kilo downloads everything in the repo, not scopred) | 16:28 |
| palendae | scoped* | 16:28 |
| * cloudnull was getting coffee | 16:31 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_swift master: Optimize swift gate https://review.openstack.org/491089 | 16:31 |
| cloudnull | palendae: that was before tempest | 16:31 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_swift master: Add dnf support https://review.openstack.org/490157 | 16:31 |
| *** udesale has quit IRC | 16:31 | |
| cloudnull | logan-: that munin change, has that merged in the integrated gate yet ? | 16:31 |
| logan- | no | 16:31 |
| cloudnull | maybe we can enable that by default? | 16:32 |
| cloudnull | https://review.openstack.org/#/c/491005/ | 16:32 |
| logan- | yeah I'd like to eventually. it needs a lot of smoothing out | 16:32 |
| logan- | i'm not sure if munin is the right tool yet but I haven't found something better for resource monitoring + static metrics site generation | 16:33 |
| logan- | munin kinda sucks at the time period scoping with our relatively short gate job lifetimes | 16:33 |
| *** nollide_ has joined #openstack-ansible | 16:34 | |
| logan- | its not really a drop in, gonna have to hack around and see how to get the charts actually useful | 16:34 |
| cloudnull | logan-: it looks like your job was having the same symptoms mine was https://jenkins.lstn.net/job/openstack-upstream/job/openstack-ansible-check/82/artifact/logs/munin/localdomain/localhost.localdomain/cpu.html | 16:34 |
| logan- | yeah | 16:34 |
| logan- | 148% avg usage, so cinder is sitting there burning up a core | 16:35 |
| *** nollide has quit IRC | 16:35 | |
| *** nollide_ is now known as nollide | 16:35 | |
| cloudnull | we have sar on the hosts. | 16:35 |
| cloudnull | I wonder if we could just use something like sargraph | 16:35 |
| cloudnull | or we used to , maybe we dont any more | 16:39 |
| cloudnull | oh no, its still there. | 16:40 |
| cloudnull | in the openstack_host role | 16:40 |
| logan- | i also put in https://review.openstack.org/#/c/491151/ to help us with the charting | 16:40 |
| cloudnull | cool | 16:42 |
| *** openstackstatus has quit IRC | 16:42 | |
| *** openstackstatus has joined #openstack-ansible | 16:42 | |
| *** ChanServ sets mode: +v openstackstatus | 16:42 | |
| logan- | sargraph looks like it might do the trick | 16:45 |
| logan- | maybe not though, doesnt really look like it has a static site export type of setup | 16:45 |
| logan- | this person is making rrds from sar data https://labs.eleks.com/2013/12/rrd-and-rrdtool-sar-graphs-using-pyrrd.html | 16:46 |
| *** cshen has quit IRC | 16:51 | |
| cloudnull | oh thats pretty cool | 16:53 |
| *** perniciouscaffei has quit IRC | 16:55 | |
| cloudnull | so i can recreate the cinder freak out, detroy the cinder container on an aio, then rerun the cinder playbook. | 16:57 |
| cloudnull | when the scheduler starts | 16:57 |
| cloudnull | it goes 100% cpu and never calms down | 16:58 |
| *** armaan has joined #openstack-ansible | 16:58 | |
| cloudnull | it looks as if the scheduler starts before the api | 16:58 |
| cloudnull | well it only looks like the first start is the issue | 17:01 |
| cloudnull | after that restarting the services in any order does not exhibit the same behavior | 17:01 |
| *** shardy has quit IRC | 17:04 | |
| *** woodard_ has joined #openstack-ansible | 17:10 | |
| mhayden | cloudnull: any more feedback on https://review.openstack.org/490192 ? | 17:10 |
| *** woodard has quit IRC | 17:13 | |
| *** woodard_ has quit IRC | 17:14 | |
| cloudnull | nope, my questions are answered | 17:15 |
| *** lostRhino has joined #openstack-ansible | 17:15 | |
| *** pcaruana has quit IRC | 17:15 | |
| *** nollide has quit IRC | 17:16 | |
| mhayden | cloudnull: gracias | 17:17 |
| cloudnull | mucho si | 17:18 |
| *** tobberydberg has joined #openstack-ansible | 17:18 | |
| spotz | cloudnull: you're letting mhayden off to easy... | 17:20 |
| lostRhino | all - we’re having issues with the novnc repo override (nova_novncproxy_git_repo:) - 2 weeks ago we updated our internal repos for stable/newton. After the update of the repos the build for some reason still wants to go to GitHub.com no matter what we do. I found a workaround that would pickup the internal repo if I specified the branch as stable/v0.6. However the repo is still causing issues when we are in a fully disconnected environme | 17:22 |
| lostRhino | Can someone help me understand why the repo server is not honoring the override in bothe the remote_package_parts and the remote_packages (where the repo points back to GitHub)?? thanks | 17:22 |
| *** tobberydberg has quit IRC | 17:22 | |
| odyssey4me | lostRhino when you run the playbook there's a bunch of json output before it executes the role - can you pastebin that? | 17:23 |
| lostRhino | odyssey4me: sure - do you want everything or just the corresponding information…. its A LOT ... | 17:24 |
| logan- | lostRhino: do you have both nova_novncproxy_git_repo AND nova_novncproxy_git_install_branch specified in your overrides file | 17:24 |
| odyssey4me | gimme a sec | 17:24 |
| lostRhino | yes, recently I had to use the branch to get the internal repo to pick up | 17:25 |
| logan- | i have seen the py pkgs filter ignore override definitions if you _only_ provide a repo url override, you also need to specify _install_branch in the same file, even if you're using the same sha as upstream OSA | 17:25 |
| lostRhino | very very odd - didnt have to before… but things do change in 3 months | 17:25 |
| odyssey4me | check whether this piece of data has the right git repo info: http://logs.openstack.org/88/490988/1/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial/fd610f9/console.html#_2017-08-04_17_23_40_501296 | 17:25 |
| mgariepy | hmm odyssey4me with 14.2.7 tag. http://paste.openstack.org/show/617699/ | 17:25 |
| odyssey4me | mgariepy did you ensure you updated ansible and all that? never seen that before | 17:27 |
| mgariepy | yep i ran the bootstap ansible script | 17:27 |
| lostRhino | our egg name is all lower- does that matter? | 17:27 |
| mgariepy | http://paste.openstack.org/show/617700/ | 17:27 |
| lostRhino | novnc vs noVNC | 17:28 |
| *** nollide has joined #openstack-ansible | 17:28 | |
| odyssey4me | mgariepy this is suposed to lay it down: https://github.com/openstack/openstack-ansible-repo_build/blob/master/tasks/repo_build_prepare.yml#L16 | 17:28 |
| lostRhino | the odd thing that is happening is that it sees the correct repo under remote_package_parts / but under remote_packages it does not | 17:28 |
| odyssey4me | lostRhino no, it does not | 17:29 |
| logan- | lostRhino: in the override url? no it does not matter | 17:29 |
| odyssey4me | oh really? well that's interesting | 17:29 |
| lostRhino | http://logs.openstack.org/88/490988/1/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial/fd610f9/console.html#_2017-08-04_17_23_40_504979 | 17:29 |
| lostRhino | is where its github | 17:29 |
| lostRhino | http://logs.openstack.org/88/490988/1/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial/fd610f9/console.html#_2017-08-04_17_23_40_501380 | 17:29 |
| lostRhino | is where its internal | 17:29 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Change the cinder playbook to run API commands first https://review.openstack.org/491550 | 17:30 |
| cloudnull | logan-: odyssey4me ^ | 17:30 |
| odyssey4me | cloudnull hmm, lemme check that | 17:30 |
| odyssey4me | I know with some services I had to guess the order | 17:31 |
| *** cshen has joined #openstack-ansible | 17:31 | |
| cloudnull | with that order changed I'm not seeing the 100% cpu anymore | 17:31 |
| odyssey4me | cloudnull well, something else is not right then - because https://docs.openstack.org/cinder/latest/upgrade.html#after-maintenance-window specifies that cinder-api should be last | 17:32 |
| mgariepy | odyssey4me, https://github.com/openstack/openstack-ansible-repo_build/blob/master/tasks/repo_build_prepare.yml#L26 | 17:32 |
| cloudnull | odyssey4me: maybe its the speed at which the services are restarted ? | 17:33 |
| cloudnull | however I tried this three times, with the current order it ended in cinder running at 100% | 17:34 |
| odyssey4me | ok, so basically it's in compatibility mode at that time - handling both RPC versions | 17:34 |
| odyssey4me | which kinda makes sense | 17:35 |
| logan- | even after the initial greenfield install it still does that if you restart in that order? | 17:35 |
| odyssey4me | this is something that should be troubleshot with the upstream guys I would think | 17:35 |
| odyssey4me | either the docs are bad, or the code is bad, or we're doing something wrong | 17:36 |
| odyssey4me | whichever way, I'd rather we made informed decisions instead of guess work | 17:36 |
| openstackgerrit | Marc Gariépy proposed openstack/openstack-ansible-repo_build stable/newton: Add protection against no local facts being defined https://review.openstack.org/491552 | 17:36 |
| logan- | do we need that in ocata also mgariepy? | 17:37 |
| cloudnull | logan-: to recreate the issue on an AIO: destroy / recreate the cinder api container. stop the cinder-volume,backup services on the host. run the os-cinder-install.yml play | 17:37 |
| mgariepy | logan-, probably, but i'll just run a quick test to see something | 17:37 |
| cloudnull | at the end of the play, cinder scheduler will be running at 100% | 17:38 |
| logan- | ouch | 17:38 |
| cloudnull | it happens as soon as this task fires | 17:39 |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-cinder-install.yml#L164 | 17:39 |
| cloudnull | odyssey4me: how do you want to proceed ? | 17:39 |
| odyssey4me | cloudnull ideally get hold of a cinder developer and dig deeper into why that's happening and whether it's expected or not | 17:40 |
| odyssey4me | we need to know if our start/restart order needs adjusting, and if so then their docs need adjusting | 17:40 |
| odyssey4me | if it's done the right way, then what's causing the lock-up | 17:41 |
| cloudnull | odyssey4me: maybe there's a way to signify a greenfield deployment and skip all of this https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-cinder-install.yml#L145-L237 | 17:41 |
| logan- | i wonder if a reboot could trigger it too depending on container restart order | 17:42 |
| odyssey4me | cloudnull there are possibly, but then we start creating two code paths - one of which is never tested | 17:42 |
| logan- | it seems like a pretty bad bug even if the playbook ordering is "wrong" and the docs are adjusted | 17:42 |
| odyssey4me | it is better to just do what we're doing here and work out where the bugs lie and fix them | 17:42 |
| odyssey4me | working around things all the time is not good | 17:42 |
| andymccr | if we can get a run down of the things im happy to talk to the cinder peeps if nobody else has bandwidth for it | 17:42 |
| cloudnull | andymccr: the tl;dr is when the cinder services are hup'd / restarted in the following order HUP: [backup, volume, scheduler] RESTART: [api] -- we're seeing cinder scheduler run at 100% cpu | 17:44 |
| openstackgerrit | Marc Gariépy proposed openstack/openstack-ansible-repo_build stable/ocata: Add protection against no local facts being defined https://review.openstack.org/491556 | 17:45 |
| andymccr | cloudnull: got it, but only on startup? | 17:45 |
| cloudnull | so a greenfield deployment with already running services then has that restart dance done and it begins to freakout. | 17:46 |
| cloudnull | the scheduler never recovers until its restarted manually. | 17:46 |
| andymccr | ok cool | 17:47 |
| cloudnull | odyssey4me: maybe we need to make the sheduler just restart, instead of hup? | 17:47 |
| jmccrory | cloudnull : just tried and seeing the same freakout. do you have this in your cinder-scheduler log too? 'Possible hang: wait is waiting for stop to complete' | 17:49 |
| cloudnull | jmccrory: let me go look | 17:49 |
| mgariepy | arf. was a cache issue.. needed to rm /etc/openstack_deploy/ansible_facts/*repo_container* | 17:49 |
| cloudnull | jmccrory: yes | 17:50 |
| cloudnull | http://paste.openstack.org/show/617701/ | 17:50 |
| cloudnull | so maybe the easier fix would be to restart the cinder-scheduler instead? | 17:52 |
| cloudnull | we can hup the others and then restart that one | 17:52 |
| cloudnull | I guess we also should wait until we've heard back from the cinder folks | 17:52 |
| cloudnull | this potential impacts ocata - https://github.com/openstack/openstack-ansible/blob/stable/ocata/playbooks/os-cinder-install.yml#L140-L162 and newton https://github.com/openstack/openstack-ansible/blob/stable/newton/playbooks/os-cinder-install.yml#L140-L162 | 17:53 |
| odyssey4me | well, the issue with that is that if any transactions have started, they will be lost | 17:53 |
| odyssey4me | the point of the hup is to not lose transactions | 17:53 |
| odyssey4me | and that matters to everything except the API | 17:54 |
| *** markvoelker has joined #openstack-ansible | 17:54 | |
| cloudnull | this is true | 17:54 |
| odyssey4me | anyway, I'm out for the eve - time to get home | 17:54 |
| cloudnull | take care odyssey4me | 17:54 |
| odyssey4me | looking forward to seeing whether you find any answers - if you don't, we can pick it up in the morning | 17:55 |
| *** cshen has quit IRC | 17:55 | |
| spotz | night odyssey4me | 17:55 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Add support for OpenDaylight deployment in neutron https://review.openstack.org/480131 | 17:55 |
| lostRhino | odyssey4me: can i follow up with you tomorrow? | 17:56 |
| evrardjp | lostRhino: most probably :p | 17:57 |
| lostRhino | thanks | 17:57 |
| *** cshen has joined #openstack-ansible | 17:57 | |
| *** gouthamr has joined #openstack-ansible | 17:58 | |
| *** lostRhino has left #openstack-ansible | 18:00 | |
| mhayden | andymccr: OMG MERGE IT https://review.openstack.org/491089 | 18:03 |
| mhayden | odyssey4me ^^ | 18:04 |
| *** poopcat has joined #openstack-ansible | 18:04 | |
| *** cshen has quit IRC | 18:04 | |
| *** albertcard1 has joined #openstack-ansible | 18:06 | |
| spotz | mhayden: that a hint? | 18:08 |
| mhayden | haha finally passed the darned gate | 18:08 |
| mhayden | but barely | 18:08 |
| *** poopcat has quit IRC | 18:08 | |
| *** cshen has joined #openstack-ansible | 18:09 | |
| *** albertcard1 is now known as poopcat | 18:12 | |
| spotz | otherone failed still:( | 18:12 |
| *** stuartgr has quit IRC | 18:13 | |
| openstackgerrit | Merged openstack/openstack-ansible-repo_build stable/newton: Add protection against no local facts being defined https://review.openstack.org/491552 | 18:13 |
| LiterateHawk | Hi all - Is the Centos version of OSA functional with linuxbridges? I've installed a small deployment with ubuntu, but we're a ~100% centos/RHEL shop, so my next step is to see if I can't move nodes over to centos | 18:14 |
| *** cshen has quit IRC | 18:14 | |
| LiterateHawk | Of course, if it's still heavily under development, that might not be the best choice | 18:14 |
| mgariepy | LiterateHawk, Centos is working but it still under dev. and might be a little bumpy for upgrades. | 18:16 |
| mgariepy | if you have time to tests stuff it would be great tho. | 18:17 |
| *** woodard has joined #openstack-ansible | 18:19 | |
| *** woodard has quit IRC | 18:19 | |
| *** armaan has quit IRC | 18:19 | |
| mgariepy | LiterateHawk, mhayden, have a running centos cluster. | 18:19 |
| *** woodard has joined #openstack-ansible | 18:19 | |
| *** armaan has joined #openstack-ansible | 18:19 | |
| *** sauloaislan has joined #openstack-ansible | 18:20 | |
| mgariepy | I had one but it's not dismantled as I needed the HW for other purpose. | 18:20 |
| sauloaislan | Morning! | 18:21 |
| *** cshen has joined #openstack-ansible | 18:23 | |
| sauloaislan | Is anyone having trouble launching an OSA? | 18:23 |
| *** pbandark has quit IRC | 18:24 | |
| *** tobberydberg has joined #openstack-ansible | 18:24 | |
| sauloaislan | I can not launch, I'm having some errors | 18:24 |
| *** prometheanfire has joined #openstack-ansible | 18:24 | |
| *** tobberydberg has quit IRC | 18:28 | |
| *** cshen has quit IRC | 18:28 | |
| *** markvoelker has quit IRC | 18:28 | |
| *** cshen has joined #openstack-ansible | 18:29 | |
| *** chhavi has quit IRC | 18:31 | |
| andymccr | mhayden: nice! :D | 18:36 |
| *** cshen has quit IRC | 18:37 | |
| LiterateHawk | mgariepy: I don't have a problem putting in bug reports. My ansible knowledge is ~minimal, though... | 18:38 |
| LiterateHawk | mgariepy: Presumably I could drop/wipe a single master node and reinstall w/centos? | 18:39 |
| LiterateHawk | Or would it be best to just blow the whole thing away | 18:39 |
| mhayden | that darned pip_install role keeps giving me the sads https://paste.fedoraproject.org/paste/Rb78hph917HUVJZSsNp2sw/raw | 18:40 |
| *** nollide has quit IRC | 18:40 | |
| *** cshen has joined #openstack-ansible | 18:41 | |
| sauloaislan | I have fail in "Gather variables for each operating system" task | 18:42 |
| odyssey4me | mhayden whatever happened to your experiment to install it in playbooks instead of roles? | 18:42 |
| mhayden | still scratching my head on that one | 18:42 |
| *** cshen has quit IRC | 18:50 | |
| mgariepy | LiterateHawk, I think it's easier to start over with centos instead of trying to migrate from ubuntu to centos | 18:53 |
| *** nollide has joined #openstack-ansible | 18:53 | |
| *** tobberydberg has joined #openstack-ansible | 18:57 | |
| LiterateHawk | mgariepy: Fair enough, I'm just running a shedload of cirros images anyway.. | 18:59 |
| *** hybridpollo has joined #openstack-ansible | 19:04 | |
| *** cshen has joined #openstack-ansible | 19:05 | |
| *** vishwanathj has quit IRC | 19:06 | |
| *** vishwanathj has joined #openstack-ansible | 19:06 | |
| *** tobberydberg has quit IRC | 19:07 | |
| *** nollide has quit IRC | 19:08 | |
| *** cshen has quit IRC | 19:08 | |
| *** cshen has joined #openstack-ansible | 19:09 | |
| *** nollide has joined #openstack-ansible | 19:10 | |
| *** hybridpollo has quit IRC | 19:13 | |
| *** hybridpollo has joined #openstack-ansible | 19:15 | |
| *** cshen has quit IRC | 19:15 | |
| *** nollide has quit IRC | 19:17 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Add dnf support https://review.openstack.org/490192 | 19:19 |
| *** tobberydberg has joined #openstack-ansible | 19:22 | |
| *** tobberyd_ has joined #openstack-ansible | 19:30 | |
| *** armaan has quit IRC | 19:30 | |
| *** tobberydberg has quit IRC | 19:31 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible master: Add dnf support to ceph playbook https://review.openstack.org/491584 | 19:34 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible master: [WIP] Testing dnf with host/containers https://review.openstack.org/491084 | 19:35 |
| *** tobberyd_ has quit IRC | 19:37 | |
| *** nollide has joined #openstack-ansible | 19:41 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-pip_install master: Rename package install task https://review.openstack.org/491585 | 19:51 |
| *** cshen has joined #openstack-ansible | 19:53 | |
| mhayden | i wonder if we could possibly have a directory bind mounted in the gate that would allow us to re-use wheels | 20:00 |
| mhayden | for example, we build cryptography a bunch of times on the neutron CI job | 20:00 |
| *** thorst has quit IRC | 20:00 | |
| mhayden | but if we could stash compiled wheels in a shared directory, we would only need to build it once | 20:00 |
| *** thorst has joined #openstack-ansible | 20:03 | |
| *** thorst has quit IRC | 20:07 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Change the cinder playbook to run API commands first https://review.openstack.org/491550 | 20:12 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_swift master: Add dnf support https://review.openstack.org/490157 | 20:16 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_neutron master: Add dnf support https://review.openstack.org/490144 | 20:16 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_swift master: Disable Erasure Code test for CentOS/RHEL https://review.openstack.org/490462 | 20:16 |
| *** gouthamr has quit IRC | 20:17 | |
| *** thorst has joined #openstack-ansible | 20:18 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_swift master: Add dnf support https://review.openstack.org/490157 | 20:19 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Revert "Reduce container counts for cinder, ironic and trove" https://review.openstack.org/471822 | 20:20 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible master: Apply pip_install after container creation https://review.openstack.org/485203 | 20:21 |
| cloudnull | mhayden: if we use the upstream wheel mirrors we'd not have to build it at all | 20:22 |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/master/scripts/fastest-infra-wheel-mirror.py#L32-L40 | 20:22 |
| cloudnull | we could improve our gate test runtimes by pointing at these mirrors, if not already doing so | 20:23 |
| *** gouthamr has joined #openstack-ansible | 20:24 | |
| *** asura has quit IRC | 20:25 | |
| *** tobberydberg has joined #openstack-ansible | 20:25 | |
| mhayden | hmm, are we using those? | 20:27 |
| mhayden | i thought we had a task for that | 20:27 |
| mhayden | perhaps we're using it for the integrated build | 20:27 |
| mhayden | but not roles? | 20:28 |
| * mhayden scurries off to check | 20:28 | |
| mhayden | it seems like we are | 20:29 |
| *** tobberydberg has quit IRC | 20:30 | |
| *** smatzek has quit IRC | 20:31 | |
| *** radafshar1983 has quit IRC | 20:35 | |
| cloudnull | then it should not be building any wheels unless they're not in the mirror | 20:36 |
| cloudnull | or we're somehow passing in the --isolated flag | 20:36 |
| *** gouthamr has quit IRC | 20:36 | |
| mhayden | well we don't take the binary for quite a few things, including cryptography | 20:39 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-openstack_hosts master: Rename package install task https://review.openstack.org/491607 | 21:01 |
| *** lucasxu has quit IRC | 21:15 | |
| *** tobberydberg has joined #openstack-ansible | 21:17 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible master: Update Ansible to 2.3.2.0 https://review.openstack.org/481745 | 21:25 |
| *** markvoelker has joined #openstack-ansible | 21:26 | |
| *** tobberydberg has quit IRC | 21:30 | |
| mhayden | weird, got a commit where the short sha was all numbers | 21:47 |
| mhayden | had to do a double take | 21:47 |
| mhayden | oh man, the bundle of swift patches just might merge | 21:48 |
| mhayden | just maybe | 21:48 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-os_sahara master: Updated from OpenStack Ansible Tests https://review.openstack.org/488476 | 21:56 |
| cloudnull | mhayden: fingers crossed | 21:56 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-os_sahara master: Update URLs in documents according to document migration https://review.openstack.org/483707 | 21:56 |
| mhayden | it started the functional tests at 47 minutes in | 21:56 |
| mhayden | which means it's likely to pass | 21:56 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-os_sahara master: Updated from global requirements https://review.openstack.org/488068 | 21:56 |
| mhayden | why do i know these things by heart | 21:56 |
| mhayden | this is sad | 21:56 |
| *** cshen has quit IRC | 21:57 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-os_sahara master: Implement uWSGI for sahara-api https://review.openstack.org/491421 | 21:57 |
| openstackgerrit | Merged openstack/openstack-ansible-plugins master: Updated from OpenStack Ansible Tests https://review.openstack.org/490934 | 21:57 |
| *** markvoelker has quit IRC | 22:00 | |
| mhayden | cloudnull: test failed :( | 22:02 |
| mhayden | no timeout, something blew up in functional tests | 22:02 |
| *** sliver has quit IRC | 22:04 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible master: Add dnf support to ceph playbook https://review.openstack.org/491584 | 22:07 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible master: [WIP] Testing dnf with host/containers https://review.openstack.org/491084 | 22:08 |
| *** esberglu has quit IRC | 22:09 | |
| *** esberglu has joined #openstack-ansible | 22:10 | |
| *** esberglu has quit IRC | 22:14 | |
| *** gtrxcb has joined #openstack-ansible | 22:15 | |
| *** tobberydberg has joined #openstack-ansible | 22:16 | |
| *** nollide has left #openstack-ansible | 22:16 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara master: Update URLs in documents according to document migration https://review.openstack.org/483707 | 22:19 |
| *** tobberydberg has quit IRC | 22:23 | |
| *** esberglu has joined #openstack-ansible | 22:25 | |
| *** gouthamr has joined #openstack-ansible | 22:29 | |
| *** thorst has quit IRC | 22:32 | |
| *** markvoelker has joined #openstack-ansible | 22:32 | |
| *** marst has quit IRC | 22:36 | |
| *** tobberydberg has joined #openstack-ansible | 22:40 | |
| *** tobberydberg has quit IRC | 22:51 | |
| *** markvoelker has quit IRC | 22:54 | |
| *** markvoelker has joined #openstack-ansible | 22:54 | |
| *** tobberydberg has joined #openstack-ansible | 22:55 | |
| *** markvoelker has quit IRC | 22:59 | |
| *** tobberydberg has quit IRC | 22:59 | |
| openstackgerrit | Logan V proposed openstack/openstack-ansible-galera_server master: Allow external restart handler orchestration https://review.openstack.org/491211 | 23:00 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible master: Implement galera restarts in playbook https://review.openstack.org/491212 | 23:00 |
| *** chhavi has joined #openstack-ansible | 23:08 | |
| openstackgerrit | Logan V proposed openstack/openstack-ansible master: Implement galera restarts in playbook https://review.openstack.org/491212 | 23:12 |
| *** tobberydberg has joined #openstack-ansible | 23:29 | |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org will be offline momentarily at 00:00 utc for a quick reconfiguration-related restart | 23:32 | |
| cloudnull | mhayden: :'( | 23:33 |
| cloudnull | logan-: jmccrory: when you get a chance https://review.openstack.org/#/c/491550/ | 23:33 |
| cloudnull | if you wouldn't mind testing that | 23:33 |
| cloudnull | it seems to fix the cinder cpu freakout. | 23:33 |
| *** tobberydberg has quit IRC | 23:33 | |
| cloudnull | in my env but it'd be good to get validation on that point. | 23:33 |
| *** schwicht has quit IRC | 23:36 | |
| jmccrory | i was seeing cinder-backup spiking at 100% earlier too, will give this a shot and see though | 23:37 |
| cloudnull | I had that one go crazy too initially | 23:37 |
| cloudnull | however i've not been able to reproduce it | 23:37 |
| cloudnull | and i dont have that halt message in my error log | 23:37 |
| cloudnull | we can make it restart instead of reload with that patch if we must | 23:37 |
| cloudnull | looking back through ara nothing stands out as taking too long any more. | 23:38 |
| jmccrory | yeah, didn't have a halt message for it either | 23:38 |
| cloudnull | we're still swapping like crazy in the gate | 23:40 |
| cloudnull | http://paste.openstack.org/show/617718/ | 23:40 |
| cloudnull | http://logs.openstack.org/50/491550/2/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial/2ffb40f/logs/openstack/instance-info/host_system_info_22-10-32.log.txt.gz | 23:40 |
| cloudnull | but its at least succeeding | 23:40 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
