| openstackgerrit | Merged openstack/openstack-ansible-os_tempest stable/pike: Allow for single tempest image upload https://review.openstack.org/560011 | 00:18 |
|---|---|---|
| *** cshen has joined #openstack-ansible | 00:40 | |
| *** cshen has quit IRC | 00:45 | |
| *** dave-mccowan has joined #openstack-ansible | 00:47 | |
| *** odyssey4me has quit IRC | 00:47 | |
| *** odyssey4me has joined #openstack-ansible | 00:47 | |
| cmart | ah it was an ACL issue in HAProxy. all good now | 00:49 |
| *** osnaya has joined #openstack-ansible | 01:06 | |
| osnaya | quick question - For Pike release OSA deployment on Ubuntu16.04 LTS what version of Python is used? | 01:10 |
| *** osnaya has quit IRC | 01:15 | |
| *** dkehn has joined #openstack-ansible | 01:32 | |
| *** cshen has joined #openstack-ansible | 01:35 | |
| *** cshen has quit IRC | 01:39 | |
| *** nicolasbock has quit IRC | 02:14 | |
| cloudnull | osnaya we use python 2.7 | 02:25 |
| *** cshen has joined #openstack-ansible | 02:29 | |
| *** cshen has quit IRC | 02:33 | |
| *** ianychoi has quit IRC | 02:35 | |
| *** dave-mccowan has quit IRC | 02:56 | |
| *** uiet has quit IRC | 03:07 | |
| *** cjloader has joined #openstack-ansible | 03:14 | |
| *** cjloader has quit IRC | 03:19 | |
| *** ianychoi has joined #openstack-ansible | 03:19 | |
| *** cshen has joined #openstack-ansible | 03:23 | |
| *** cshen has quit IRC | 03:27 | |
| *** udesale has joined #openstack-ansible | 03:34 | |
| *** poopcat has quit IRC | 04:07 | |
| *** cjloader has joined #openstack-ansible | 04:15 | |
| *** cshen has joined #openstack-ansible | 04:17 | |
| *** cjloader has quit IRC | 04:20 | |
| *** cshen has quit IRC | 04:21 | |
| *** udesale has quit IRC | 04:26 | |
| *** cmart has quit IRC | 04:30 | |
| adrianreza | Hi anyone, when I run setup-infrastructure.yml playbook I got error with Apply rabbitmq policies task http://paste.openstack.org/show/718874/ | 04:33 |
| *** chhagarw has joined #openstack-ansible | 04:34 | |
| adrianreza | do you know how to fix Hostname mismatch: node \"rabbit@infra1-rabbit-mq-container-6fda54a4\" believes its host is different. Please ensure that hostnames resolve the same way locally and on \"rabbit@infra1-rabbit-mq-co | 04:34 |
| adrianreza | ntainer-6fda54a4\"", | 04:34 |
| ivve | adrianreza: deploying master branch? | 04:38 |
| *** udesale has joined #openstack-ansible | 04:45 | |
| *** udesale has quit IRC | 04:46 | |
| *** udesale has joined #openstack-ansible | 04:47 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update elk 6.x playbooks https://review.openstack.org/560168 | 05:01 |
| *** cshen has joined #openstack-ansible | 05:11 | |
| *** cjloader has joined #openstack-ansible | 05:15 | |
| *** cshen has quit IRC | 05:15 | |
| *** cjloader has quit IRC | 05:20 | |
| *** markvoelker_ has quit IRC | 05:39 | |
| *** armaan has quit IRC | 05:42 | |
| *** armaan has joined #openstack-ansible | 05:44 | |
| evrardjp | cloudnull: Apparently we will meet :) | 05:52 |
| cloudnull | stella all around ! | 05:52 |
| ivve | im still getting those ssh errors during cell_v2 even after cleaning out ansible_facts | 05:55 |
| ivve | annoying | 05:55 |
| ivve | ill use a new deployment host now.. grrr ;( | 05:56 |
| adrianreza | ivve: nope I use 17.0.1 branch | 05:58 |
| ivve | adrianreza: ait, i got it in master yesterday but gave up and switched to stable.. i want a fresh install to troubleshoot another bug | 05:59 |
| ivve | namely nova host-evacuate-live | 05:59 |
| adrianreza | ivve: so did stable branch solve the same issue? | 06:00 |
| *** mardim has joined #openstack-ansible | 06:00 | |
| ivve | yes, but instead i got those random ssh errors in cell_v2 | 06:00 |
| ivve | but my deployment host is not fresh | 06:01 |
| ivve | also its a centos7 | 06:01 |
| ivve | so a few taint-potential factors :P | 06:01 |
| *** rodolof has joined #openstack-ansible | 06:01 | |
| *** qwebirc23189 has quit IRC | 06:08 | |
| *** rodolof has quit IRC | 06:08 | |
| *** rodolof has joined #openstack-ansible | 06:09 | |
| *** cjloader has joined #openstack-ansible | 06:14 | |
| *** evin has joined #openstack-ansible | 06:17 | |
| *** cjloader has quit IRC | 06:19 | |
| *** armaan has quit IRC | 06:19 | |
| *** armaan has joined #openstack-ansible | 06:20 | |
| *** armaan has quit IRC | 06:24 | |
| *** armaan has joined #openstack-ansible | 06:25 | |
| *** jhesketh_ is now known as jhesketh | 06:29 | |
| *** radeks has joined #openstack-ansible | 06:33 | |
| *** mma has joined #openstack-ansible | 06:36 | |
| *** haad1 has joined #openstack-ansible | 07:05 | |
| *** cjloader has joined #openstack-ansible | 07:14 | |
| *** shardy has joined #openstack-ansible | 07:14 | |
| *** mbuil has joined #openstack-ansible | 07:14 | |
| *** jwitko_ has joined #openstack-ansible | 07:17 | |
| *** epalper has joined #openstack-ansible | 07:18 | |
| *** cjloader has quit IRC | 07:19 | |
| *** cshen has joined #openstack-ansible | 07:20 | |
| *** jwitko has quit IRC | 07:20 | |
| *** gordon has joined #openstack-ansible | 07:24 | |
| *** gordon is now known as Guest47834 | 07:24 | |
| *** Guest47834 has quit IRC | 07:27 | |
| openstackgerrit | Manuel Buil proposed openstack/openstack-ansible-os_neutron master: [DNM] - Checking ODL gates https://review.openstack.org/560297 | 07:28 |
| *** flaviosr has quit IRC | 07:45 | |
| *** flaviosr has joined #openstack-ansible | 07:50 | |
| *** markvoelker has joined #openstack-ansible | 08:01 | |
| jrosser | how can i check which versions packages ended up in my nova venv? | 08:02 |
| *** gkadam has joined #openstack-ansible | 08:06 | |
| *** vnogin has joined #openstack-ansible | 08:07 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update elk 6.x playbooks https://review.openstack.org/560168 | 08:12 |
| jrosser | my nova api container deploy stalls at this - http://paste.openstack.org/show/718889/ | 08:12 |
| *** pcaruana has joined #openstack-ansible | 08:13 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_keystone master: templates: keystone-uwsgi: Explicitly load the required uwsgi plugins https://review.openstack.org/560307 | 08:25 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_keystone master: [RFC] Add support for using distribution packages for OpenStack services https://review.openstack.org/560308 | 08:25 |
| *** vnogin has quit IRC | 08:39 | |
| *** electrofelix has joined #openstack-ansible | 08:40 | |
| *** radeks has quit IRC | 08:41 | |
| *** ianychoi has quit IRC | 08:42 | |
| *** radeks has joined #openstack-ansible | 08:42 | |
| Taseer | evrardjp: Good Morning | 08:45 |
| Taseer | long time | 08:45 |
| evrardjp | good morning | 08:45 |
| Taseer | I am working on blazar integration with OSA | 08:46 |
| admin0 | hi all .. my setup-hosts fails on pip install because its looking for the repo RetryError: HTTPConnectionPool(host='int-dev.atl.domain.cloud', port=8181): Max retries exceeded with url: /os-releases/17.0.2/ubuntu-16.04-x86_64/ (Caused by ResponseError('too many 503 error responses . | 08:46 |
| Taseer | so I am trying to test as much as possible beforehand | 08:46 |
| Taseer | https://hastebin.com/vonatitiba.sql | 08:46 |
| admin0 | that URL is done on setup-infra, while this fails on setup-hosts | 08:46 |
| admin0 | how do I clear/fix this ? | 08:46 |
| Taseer | can you help me with this => https://hastebin.com/vonatitiba.sql | 08:46 |
| evrardjp | Nope I am sorry I can't work on this right now, sorry Taseer and admin0 | 08:47 |
| Taseer | evrardjp: no problem | 08:48 |
| admin0 | evrardjp, can't you take bribes :D ? | 08:49 |
| admin0 | will wait :) | 08:49 |
| hwoarang | Taseer: i think the error message is self-explanatory | 08:50 |
| hwoarang | you prob have an old version of run_tests.sh around | 08:50 |
| hwoarang | but it's bash so you easily debug it ;) | 08:50 |
| admin0 | hwoarang, give me some pointers as well | 08:51 |
| hwoarang | in a moment i'm trying to multitask | 08:51 |
| admin0 | check.. | 08:52 |
| *** haad1 has quit IRC | 08:52 | |
| Taseer | hwoarang: I worked around it | 08:53 |
| hwoarang | well you may have workarounded it but if you dont do it properly you will see some -1s :D | 08:53 |
| Taseer | -1s dont bother me much, I get too many of 'em | 08:55 |
| *** vnogin has joined #openstack-ansible | 08:56 | |
| hwoarang | lol | 08:57 |
| *** DanyC has joined #openstack-ansible | 09:00 | |
| *** DanyC has quit IRC | 09:01 | |
| *** DanyC has joined #openstack-ansible | 09:01 | |
| *** nattanon has joined #openstack-ansible | 09:09 | |
| nattanon | Hello guys !!! I found somethings that i need to prove does it should be or not ? | 09:10 |
| nattanon | After ran OSA, I have found many process running by haproxy user on controller node but the command it quite related to haproxy. | 09:12 |
| nattanon | haproxy 15532 20.2 0.1 6981568 224136 ? Sl Mar26 4651:39 /openstack/venvs/cinder-16.0.8/bin/python /openstack/venvs/cinder-16.0.8/bin/cinder-volume --log-file=/var/log/cinder/cinder-volume.log | 09:15 |
| nattanon | And so many more process could someone explained to me what process are running for ? | 09:16 |
| nattanon | *** it quite NOT related to haproxy | 09:17 |
| admin0 | nattanon, | 09:20 |
| admin0 | controller has multiple lxc-nodes | 09:20 |
| admin0 | and the processes run inside the lxc | 09:20 |
| admin0 | but he lxc user mapping and the host user mapping is incorrect | 09:20 |
| admin0 | so it sometimes appear like this | 09:20 |
| nattanon | Okay pid is not match right | 09:20 |
| admin0 | they try to map the uid with the user | 09:21 |
| admin0 | which shows this mismatch | 09:21 |
| admin0 | they = ps | 09:21 |
| admin0 | it tries* | 09:21 |
| nattanon | Okay, Thank you so much :D | 09:21 |
| admin0 | ps on host = show global procesess running ( even from inside containers ) .. map to user = takes /etc/passwd mapping on the host | 09:22 |
| Tahvok | odyssey4me: I'm not sure where you saw that octavia_service_publicuri is used? https://review.openstack.org/#/c/549267/ | 09:22 |
| *** cshen has quit IRC | 09:25 | |
| *** haad1 has joined #openstack-ansible | 09:28 | |
| *** cshen has joined #openstack-ansible | 09:28 | |
| nattanon | admin0: Love you !! | 09:29 |
| *** nattanon has quit IRC | 09:29 | |
| olivierb- | hello everyone, having followed the guidelines in https://docs.openstack.org/project-deploy-guide/openstack-ansible/ocata/app-limited-connectivity.html and having configured a squid proxy successfully in my infrastucture I was able to successfully run the 2 bootstrap scripts from within my AIO machine and /etc/environment contains the proper informations. However when I run the setup-hosts.yml playbook task TASK [lxc_hosts : | 09:30 |
| olivierb- | Ensure that the LXC cache has been prepared] is failing after the 60 retries. At this step I do not see any activity in my cache access log and this step is working fine on the same machine when I had direct internet connectivity. Any clue where to look into ? | 09:30 |
| odyssey4me | Tahvok it might be named something else, but what I'm saying is that in the role and in the playbooks there are only ever two url's/uri's and the url's are the same as the uri | 09:31 |
| odyssey4me | olivierb- look at the cache prep log on the lxc host and see where it failed | 09:32 |
| Tahvok | odyssey4me: that's true, so I should just specify the uri in the group vars, and use that in the neutron temlpate? | 09:32 |
| odyssey4me | Tahvok I suggest that the roles all be changed to use a only two variables - the internal or public uri.... and the groups vars the same | 09:33 |
| odyssey4me | jrosser you can activate the venv and pip freeze, or some binaries have --version | 09:34 |
| odyssey4me | that's an odd error - never seen that | 09:34 |
| Tahvok | Shouldn't this be a different patch then? As I'm cleaning currently the group_vars, and not the roles. Because such cleaning could be done in many other roles as well | 09:34 |
| olivierb- | odyssey4me it is still progressing installing packages but seems way slower than ususal | 09:34 |
| olivierb- | may be some side issues | 09:34 |
| jrosser | odyssey4me: i found it, cruft left over from fixes required for P making a bogus nova api venv | 09:35 |
| olivierb- | succeeded now, will retry from scratch for more root cause analysis, thx for the help anyways | 09:37 |
| jrosser | olivierb-: which version are you trying? | 09:38 |
| olivierb- | jrosser pike | 09:38 |
| jrosser | we experienced horrific slowness with pike+proxies | 09:38 |
| jrosser | which is due to an ansible bug | 09:38 |
| olivierb- | that explains it | 09:38 |
| jrosser | hold on | 09:38 |
| jrosser | olivierb-: we set this env var on our deploy host ANSIBLE_PACKAGE: "git+https://github.com/bbc/ansible@0b7a565ba2b56955072f024c1b604d3aa4b34ddb" | 09:41 |
| jrosser | which points to a fixed up version of ansible, thats a public repo so you can see what we had to do | 09:41 |
| *** cshen_ has joined #openstack-ansible | 09:42 | |
| jrosser | olivierb-: i think it's specifically this https://github.com/bbc/ansible/commit/0b7a565ba2b56955072f024c1b604d3aa4b34ddb | 09:42 |
| olivierb- | jrosser many thanks, will look at this right away | 09:44 |
| *** cshen has quit IRC | 09:45 | |
| *** ianychoi has joined #openstack-ansible | 09:46 | |
| *** cshen_ has quit IRC | 09:46 | |
| *** haad1 has quit IRC | 09:57 | |
| *** maybebuggy has joined #openstack-ansible | 09:58 | |
| *** mathlin has quit IRC | 10:01 | |
| *** maybebuggy has quit IRC | 10:01 | |
| *** maybebuggy has joined #openstack-ansible | 10:02 | |
| *** epalper has quit IRC | 10:02 | |
| *** haad1 has joined #openstack-ansible | 10:02 | |
| admin0 | anyone has time to ook into my issue :D | 10:02 |
| *** haad2 has joined #openstack-ansible | 10:02 | |
| admin0 | look* | 10:02 |
| admin0 | hear me out :) | 10:03 |
| odyssey4me | admin0 it looks to me like you're getting 500 errors from haproxy or the nginx/repo server | 10:03 |
| admin0 | odyssey4me, yes .. but it comes in setup-hosts .. and the repo is done on setup-infra | 10:03 |
| admin0 | so i am in a chicken-egg | 10:03 |
| odyssey4me | you know enough about things to trace the path and figure out where the issue is | 10:03 |
| odyssey4me | hmm, is this queens? | 10:04 |
| admin0 | yes | 10:04 |
| admin0 | stable/queens .. checked out yesterday | 10:04 |
| odyssey4me | do you have this patch in it? https://review.openstack.org/559725 | 10:04 |
| *** vnogin has quit IRC | 10:06 | |
| odyssey4me | also, which playbook specifically is it failing in? | 10:07 |
| odyssey4me | openstack-hosts, or containers-deploy? | 10:07 |
| odyssey4me | or maybe a specific task? | 10:07 |
| *** tomtom001 has quit IRC | 10:08 | |
| *** epalper has joined #openstack-ansible | 10:08 | |
| *** tomtom001 has joined #openstack-ansible | 10:08 | |
| mbuil | evrardjp, hwoarang: ODL problem in deb is solved using hwoarang's workaround: https://git.opendaylight.org/gerrit/#/c/70720/ :) | 10:13 |
| mbuil | odl gates pass that step | 10:14 |
| *** vnogin has joined #openstack-ansible | 10:18 | |
| olivierb- | jrosser sorry but I do not quite see the relationship between the issue and the fix you mentioned earlier, could you please elaborate a bit ? | 10:19 |
| jrosser | olivierb-: are you using deployment_environment_variables or global_environment_variables to set your proxy? | 10:21 |
| olivierb- | neither of these, I am relying on the content of /etc/environment only | 10:22 |
| jrosser | and for apt? | 10:22 |
| olivierb- | this contents is set prior to launch any OSA script to get ssytem wide proper config for all running apps | 10:23 |
| admin0 | odyssey4me, that patch is sthere | 10:23 |
| admin0 | is there* | 10:23 |
| admin0 | still face issue | 10:23 |
| olivierb- | and this is also noted in the OSA doc for reduced network connectivity | 10:23 |
| jrosser | olivierb-: this is what i meant by global/deployment env vars https://github.com/openstack/openstack-ansible/blob/35972daf6fed6dfbf55f17bbd7cef01f0e778c83/etc/openstack_deploy/user_variables.yml#L124-L138 | 10:28 |
| jrosser | and the ansible bug we encountered is relevant to the deployment_environment_variables case | 10:29 |
| olivierb- | ok I think I get the idea now | 10:30 |
| *** mathlin has joined #openstack-ansible | 10:31 | |
| *** exodusftw has quit IRC | 10:31 | |
| olivierb- | thx for clarifying all this | 10:31 |
| jrosser | no worries :) | 10:31 |
| *** armaan has quit IRC | 10:36 | |
| *** shardy is now known as shardy_afk | 10:36 | |
| *** armaan has joined #openstack-ansible | 10:36 | |
| *** nicolasbock has joined #openstack-ansible | 10:37 | |
| *** exodusftw has joined #openstack-ansible | 10:38 | |
| *** cshen has joined #openstack-ansible | 10:39 | |
| *** vincent has joined #openstack-ansible | 10:49 | |
| *** vincent is now known as Guest25937 | 10:50 | |
| *** Guest57868 has quit IRC | 10:51 | |
| *** jwitko_ has quit IRC | 10:57 | |
| *** shardy_afk is now known as shardy | 11:04 | |
| *** Administrator_ has joined #openstack-ansible | 11:10 | |
| *** ansmith has quit IRC | 11:11 | |
| *** Guest25937 has quit IRC | 11:13 | |
| *** vnogin has quit IRC | 11:16 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Update elk 6.x playbooks https://review.openstack.org/560168 | 11:18 |
| *** haad2 has quit IRC | 11:19 | |
| *** vnogin has joined #openstack-ansible | 11:22 | |
| evrardjp | odyssey4me: thanks for the follow up | 11:30 |
| evrardjp | on ^ | 11:30 |
| odyssey4me | :) | 11:30 |
| mma | Any plan on adding support for using client certificates for rabbitmq authentication in the future? | 11:32 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-galera_server master: Do not log passwords https://review.openstack.org/560375 | 11:34 |
| evrardjp | we are not against it mma | 11:34 |
| mma | evrardjp: Got it :). We will most likely need that in one of our installations as it is defined in the security guide. If so we could look into possibility of adding support for something similar in the osa playbook. | 11:36 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Do not log passwords https://review.openstack.org/560376 | 11:36 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-ops master: Do not log passwords https://review.openstack.org/560377 | 11:38 |
| Miouge | I’m on origin/stable/pike (ed0bd7), I get the following error message when running playbooks/os-nova-install.yml “fatal: [compute02]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname controller03_nova_conductor_container-da76d8dc: Name or service not known\r\n", "unreachable": true}”, that host is part of openstack_inventory.json so I would expect the IP to be resolve there rath | 11:38 |
| Miouge | than via DNS & hosts file? | 11:38 |
| *** armaan has quit IRC | 11:39 | |
| *** foutatoro has joined #openstack-ansible | 11:39 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-galera_server master: Do not log passwords https://review.openstack.org/560375 | 11:39 |
| *** armaan has joined #openstack-ansible | 11:39 | |
| *** haad1 has joined #openstack-ansible | 11:41 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_barbican master: Do not log passwords https://review.openstack.org/560378 | 11:41 |
| odyssey4me | Miouge that was discussed yesterday in the channel, and I was hoping that we'd see a patch up for it already: http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2018-04-10.log.html#t2018-04-10T15:16:16 | 11:42 |
| odyssey4me | coreywright will hopefully do one soon | 11:42 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_cloudkitty master: Do not log passwords https://review.openstack.org/560380 | 11:42 |
| odyssey4me | a workaround was mentioned here: http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2018-04-10.log.html#t2018-04-10T16:33:46 | 11:43 |
| Miouge | odyssey4me: cool! Thanks a lot | 11:43 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_keystone master: Do not log passwords https://review.openstack.org/560382 | 11:44 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_molteniron master: Do not log passwords https://review.openstack.org/560383 | 11:46 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_rally master: Do not log passwords https://review.openstack.org/560384 | 11:47 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_watcher master: Do not log passwords https://review.openstack.org/560385 | 11:48 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-tests master: Do not log passwords https://review.openstack.org/560386 | 11:49 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Do not log passwords https://review.openstack.org/560388 | 11:52 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_cloudkitty master: Do not log passwords https://review.openstack.org/560390 | 11:54 |
| Miouge | odyssey4me: from what I understand of the bug report #1762742 it is related to lazy eval and random filter ebing evaluted twice. How about changing “random” to “first” at least it would be the same result all the time. | 11:54 |
| *** gkadam has quit IRC | 11:54 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_monasca-agent master: Do not log passwords https://review.openstack.org/560391 | 11:55 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_tacker master: Do not log passwords https://review.openstack.org/560392 | 11:56 |
| odyssey4me | Miouge yeah, given that it's a run_once task now, using a random conductor isn't necessary any more | 11:58 |
| odyssey4me | we can just rather use the first conductor | 11:58 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-tests master: Do not log passwords https://review.openstack.org/560394 | 11:59 |
| *** gkadam has joined #openstack-ansible | 11:59 | |
| openstackgerrit | Maxime Guyot proposed openstack/openstack-ansible-os_nova master: Use first conductor for nova-manage cell_v2 discover_hosts https://review.openstack.org/560395 | 12:00 |
| *** haad1 has quit IRC | 12:01 | |
| Miouge | ^ something like that is what I had in mind, curious to see if it passes the tests | 12:02 |
| odyssey4me | Miouge unfortunately the issue is never seen in the tests we have, so we'll need to have the people affected by it verify whether it fixes it. | 12:04 |
| Miouge | I can give it a try, it takes ~60min for the setup-openstack.yml to reach that error | 12:04 |
| odyssey4me | Miouge with that fix, you should e able to just run os-nova-install.yml instead of the whole openstack playbook set | 12:05 |
| *** haad1 has joined #openstack-ansible | 12:10 | |
| *** odyssey4me has quit IRC | 12:14 | |
| *** odyssey4me has joined #openstack-ansible | 12:14 | |
| *** haad1 has quit IRC | 12:20 | |
| openstackgerrit | Maxime Guyot proposed openstack/openstack-ansible-os_nova master: Use first conductor for nova-manage cell_v2 discover_hosts https://review.openstack.org/560395 | 12:27 |
| *** dave-mccowan has joined #openstack-ansible | 12:35 | |
| *** vnogin has quit IRC | 12:44 | |
| *** MikeW has joined #openstack-ansible | 12:46 | |
| MikeW | Hey guys does anyone have a working network edge setup with f5 integration working on ocata or newer? I've got like 4 weeks to deploy a production cloud, and I'm trying to get the network architecture right so I can add that in later | 12:48 |
| *** ansmith has joined #openstack-ansible | 12:57 | |
| *** armaan has quit IRC | 12:58 | |
| *** armaan has joined #openstack-ansible | 12:58 | |
| *** gkadam_ has joined #openstack-ansible | 12:59 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-tests master: Add a rule to prevent passwords to be logged https://review.openstack.org/560405 | 13:01 |
| *** gkadam has quit IRC | 13:02 | |
| *** armaan has quit IRC | 13:03 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-tests master: Add a rule to prevent passwords to be logged https://review.openstack.org/560405 | 13:05 |
| *** vnogin has joined #openstack-ansible | 13:05 | |
| *** gkadam_ has quit IRC | 13:08 | |
| *** haad1 has joined #openstack-ansible | 13:15 | |
| *** radeks has quit IRC | 13:21 | |
| *** radeks has joined #openstack-ansible | 13:22 | |
| evrardjp | hwoarang: my vote is covered by odyssey4me 's https://review.openstack.org/#/c/559757/1 | 13:27 |
| hwoarang | ok | 13:28 |
| *** ansmith has quit IRC | 13:32 | |
| *** esberglu has joined #openstack-ansible | 13:35 | |
| *** jwitko has joined #openstack-ansible | 13:38 | |
| *** ansmith has joined #openstack-ansible | 13:38 | |
| *** tomtom001 has quit IRC | 13:38 | |
| *** tomtom001 has joined #openstack-ansible | 13:39 | |
| evrardjp | hwoarang: I've reviewed your spec. It's nice. Thank you! | 13:39 |
| *** throwsb1 has joined #openstack-ansible | 13:39 | |
| *** jwitko has quit IRC | 13:39 | |
| *** jwitko has joined #openstack-ansible | 13:40 | |
| *** coreywright has quit IRC | 13:49 | |
| *** AR2018 has joined #openstack-ansible | 13:49 | |
| *** udesale has quit IRC | 13:51 | |
| *** AR2018 has quit IRC | 13:51 | |
| *** armaan has joined #openstack-ansible | 13:51 | |
| *** vnogin has quit IRC | 13:52 | |
| *** vnogin has joined #openstack-ansible | 13:59 | |
| *** udesale has joined #openstack-ansible | 14:00 | |
| *** coreywright has joined #openstack-ansible | 14:02 | |
| *** Administrator_ has quit IRC | 14:04 | |
| MikeW | Hey does anyone have hardware specs on a prod environment they'd be willing to share details on? I'm getting the option of having 3 1u pizza boxes for my control plane, and don't know what to get | 14:07 |
| *** vnogin has quit IRC | 14:08 | |
| Tahvok | I'm trying to move a compute host from one environment to other. I've disabled nova-compute and neutron agent services | 14:09 |
| Tahvok | Removed the ssh keys from /var/lib/nova | 14:09 |
| Tahvok | And now running with osa on it | 14:09 |
| *** adcpdk has quit IRC | 14:09 | |
| Tahvok | the setup-hosts playbook completed fine, however, this failed: openstack-ansible setup-openstack.yml --skip-tags nova-key-distribute --limit NEW_HOST_NAME | 14:09 |
| Tahvok | With this error: https://gist.github.com/Tahvok/af097385590221ec6607f7ffc6c0d3de | 14:10 |
| Tahvok | What is this secret? | 14:10 |
| Tahvok | Is it the ceph secret? | 14:11 |
| odyssey4me | jamespage could we get some eyes on https://review.openstack.org/#/c/559190/ please? our lxd gate is failing without it | 14:11 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Re-order database migrations https://review.openstack.org/559354 | 14:12 |
| evrardjp | jmccrory: is so cool. | 14:13 |
| *** vnogin has joined #openstack-ansible | 14:14 | |
| * odyssey4me agrees | 14:15 | |
| odyssey4me | evrardjp FYI apparently ansible 2.1 cannot cope with multiple handlers in the same play with the same name | 14:16 |
| odyssey4me | so newton is a bit broken without this fix: https://github.com/rcbops/openstack-ansible-os_neutron/pull/1/files | 14:16 |
| odyssey4me | I'm just wondering how we write that up in a release note | 14:16 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-tests master: Add a rule to prevent passwords to be logged https://review.openstack.org/560405 | 14:17 |
| *** cristicalin has joined #openstack-ansible | 14:17 | |
| *** maybebuggy has quit IRC | 14:18 | |
| odyssey4me | coreywright Miouge please let us know in https://review.openstack.org/560395 whether it has resolved the issue you've been seeing with the delegation | 14:20 |
| odyssey4me | admin0 I'm not sure if you were seeing the issue too? | 14:20 |
| odyssey4me | if so, can you test it? | 14:20 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-tests master: Add a rule to prevent passwords to be logged https://review.openstack.org/560405 | 14:21 |
| evrardjp | odyssey4me: can we remove etcd from the play conditionally, so that this doesn't load? | 14:22 |
| evrardjp | or etcd is in the role meta? | 14:22 |
| odyssey4me | role meta IIRC | 14:22 |
| evrardjp | darn | 14:23 |
| odyssey4me | yup, confirmed | 14:23 |
| evrardjp | that shucks | 14:23 |
| coreywright | odyssey4me, Miouge, I fully expect moving from a random nova-conductor to the first nova-conductor should resolve the issue, but I'll test it and reply to the review shortly | 14:23 |
| admin0 | odyssey4me, logs: https://gist.github.com/a1git/dc82c6ff52e6f1fe456bdec1616663ba | 14:23 |
| odyssey4me | excellent thanks coreywright - really appreciate your deep diving into the issue | 14:24 |
| odyssey4me | admin0 oh, that's a different issue | 14:24 |
| admin0 | odyssey4me, what could be the fix | 14:25 |
| *** hw_wutianwei has quit IRC | 14:26 | |
| Miouge | odyssey4me: I ran into some other problems in my first test run, I worked in fixing that, now another test run in progress but it takes a while in my env | 14:26 |
| odyssey4me | admin0 please can you expose more of the log - I'm looking for the verbose output of these: https://github.com/openstack/openstack-ansible/blob/stable/queens/playbooks/common-tasks/set-upper-constraints.yml and https://github.com/openstack/openstack-ansible/blob/stable/queens/playbooks/common-tasks/set-pip-upstream-url.yml | 14:27 |
| odyssey4me | Miouge thanks, appreciate the time | 14:27 |
| ivve | is it possible to skip the log_hosts role all together? | 14:31 |
| odyssey4me | ivve if you have no hosts allocated to the host group, it will skip | 14:37 |
| ivve | thats what i thought but then i saw it was "required" i think in the example openstack_user_config.yml | 14:37 |
| odyssey4me | ivve oh? can you paste a link to where you saw that? | 14:38 |
| odyssey4me | the example test environment in the docs doesn't include it | 14:38 |
| ivve | i know :) | 14:38 |
| ivve | but | 14:39 |
| ivve | incoming link | 14:39 |
| ivve | https://github.com/openstack/openstack-ansible/search?utf8=%E2%9C%93&q=%23+Level%3A+log_hosts+%28required%29&type= | 14:39 |
| ivve | at the bottom | 14:39 |
| ivve | i've always used it | 14:40 |
| ivve | and if no elk stack, then its useful. but now i want to include filebeat instead | 14:40 |
| odyssey4me | aha, can you push up a patch or register a bug for that | 14:40 |
| *** ssmith has joined #openstack-ansible | 14:40 | |
| ivve | sure | 14:40 |
| *** cpusmith has joined #openstack-ansible | 14:42 | |
| evrardjp | odyssey4me: how can we ensure the code versions are well tested in case of an upgrade? | 14:44 |
| jamespage | tinwood: hey do you have cycles to look at https://review.openstack.org/#/c/559190 | 14:46 |
| *** ssmith has quit IRC | 14:46 | |
| *** evin has quit IRC | 14:47 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-tests master: Add a rule to prevent passwords to be logged https://review.openstack.org/560405 | 14:48 |
| *** nelsson has joined #openstack-ansible | 14:50 | |
| admin0 | odyssey4me, global rm -rf of /root/.pip folder and deletion of ansible_facts before a re-run fixed it for me | 14:52 |
| nelsson | hi guys! i have a question. i deployed openstack ansible stable/pike in production env. now i need to shutdown every node. if i shutdown every node .would any error happen to openstack? | 14:52 |
| *** markvoelker has quit IRC | 14:55 | |
| *** markvoelker has joined #openstack-ansible | 14:56 | |
| *** cjloader has joined #openstack-ansible | 14:56 | |
| *** kstev has joined #openstack-ansible | 14:57 | |
| admin0 | odyssey4me, and it failed | 14:57 |
| admin0 | will do a -vvvv loggging and paste full log | 14:58 |
| tinwood | jamespage, I'll take a look shortly. | 14:58 |
| evrardjp | nelsson: it won't be available? | 14:58 |
| evrardjp | please check our operations guide | 14:59 |
| *** markvoelker has quit IRC | 15:00 | |
| nelsson | evrardjp i can't understand what you mean by saying "it won't be available". maybe i've asked my question in wrong way.i'm not sure that turning off the nodes will cause a problem or not. | 15:02 |
| evrardjp | which nodes? what problem? | 15:02 |
| evrardjp | nelsson: and please check our ops guide :) | 15:03 |
| evrardjp | we have maybe answers there. | 15:03 |
| odyssey4me | evrardjp I think I might need more context for that question. | 15:06 |
| Miouge | odyssey4me and coreywright I get an “Playbook execution success” | 15:06 |
| Miouge | odyssey4me: no worries, I dind’t do much. It’s coreywright who did most of the work, so kudos to him | 15:07 |
| shananigans | nelsson: From what I have seen when doing this for mnaio snapshots, the galera cluster will be your biggest issue. Might need to shutdown infra01 last and bring up mysql manually with --wsrep-new-cluster before bringing up infra02 & 3. | 15:07 |
| *** armaan has quit IRC | 15:08 | |
| *** armaan has joined #openstack-ansible | 15:09 | |
| *** gkadam has joined #openstack-ansible | 15:09 | |
| nelsson | shananigans so, can i turn off the storage and compute nodes without any problem ? | 15:10 |
| *** haad1 has quit IRC | 15:11 | |
| coreywright | odyssey4me, Miouge i'm running it before and after and my results to far: before change 4 out of 6 attempts fail, after change 0 out of 3 attempts fail (ie no failures), but i plan on finish the 3 last "after" tests so there's an equal number of tests runs for both before & after (ie 6) | 15:12 |
| coreywright | (my testing is limited to "openstack-ansible setup-openstack.yml -l nova_compute") | 15:12 |
| coreywright | so test runs only take 3.5 minutes | 15:13 |
| shananigans | nelsson: Not sure on the storage. swift & cinder seemed to handle it well in the mnaio env. Computes shouldn't be an issue. | 15:13 |
| shananigans | nellson: I would suggest reading the docs and maybe doing it in a test environment to get the process down and familiar: https://github.com/openstack/openstack-ansible-ops/tree/master/multi-node-aio | 15:15 |
| ivve | mnaser: just deployed pike/stable all fresh and lots of gnocchi roles seems to be skipped for no good reason. i have a log of the installation but it seems to be a lot (the deploy is 3x controllers with hyperconv compute) i have the openstack_user_config & user_variables available as well | 15:15 |
| ivve | its very vanilla installation | 15:16 |
| mnaser | ivve: if you can toss those (sanitized) in a paste up somewhere I can have a look :) | 15:17 |
| admin0 | nelsson, you won't lose data .. when bringing it up, bring up only 1 controller first | 15:17 |
| ivve | aye | 15:17 |
| admin0 | 1 controller + everything else | 15:17 |
| admin0 | when all is up, then bring up the 2nd controller | 15:18 |
| admin0 | when you bring up mysql , you need to bring it up manually | 15:18 |
| admin0 | btw, why shutting it down ? | 15:19 |
| nelsson | shananigans thanks for your advise! | 15:19 |
| *** vnogin has quit IRC | 15:19 | |
| nelsson | admin0 how should i bring mysql up ? isn't there any issue with galera wsrep_ready ? | 15:19 |
| admin0 | it might fail to start and you need to do the wsrep-new-cluster as shananigans said | 15:20 |
| nelsson | admin0 i'm going to a trip and i'm not at lab for a while so maybe it would be harmfull for my pcs and servers to be turned on for a while | 15:20 |
| admin0 | aah lab :) .. then turn it off .. when u come back, document your journey to bring it back up | 15:21 |
| admin0 | :D | 15:21 |
| shananigans | nelsson: haproxy usually sets up infra01's galera as the default and only uses the other two if it fails. If you bring up the galera servers in the wrong order, the cluster won't come up. So infra01 last down and infra01 first up seems to be the easiest approach. | 15:22 |
| *** evin has joined #openstack-ansible | 15:23 | |
| nelsson | shananigans if i have just one controller as infra01. so i hopefully may not face any issue! am i right ? | 15:23 |
| *** vnogin has joined #openstack-ansible | 15:24 | |
| nelsson | admin0 i hope there would be no need for documentation :)))) | 15:24 |
| shananigans | nelsson: correct. just gets a little dodgy with multiple controllers. | 15:24 |
| admin0 | nelsson, we don't have a "when you are back from vacation" howtos | 15:25 |
| odyssey4me | coreywright sounds very positive - could you perhaps add a vote and comment on its effectively in the review itself? | 15:26 |
| coreywright | adding review reply as we speak/type | 15:26 |
| nelsson | shananigans thanks for your help ! | 15:28 |
| nelsson | admin0 i think openstack should add this documentation :D | 15:29 |
| nelsson | admin0 thank you too | 15:29 |
| *** cshen has quit IRC | 15:29 | |
| shananigans | nelsson: 1st step: wait for hangover to end, drink lots o water. | 15:30 |
| *** cshen has joined #openstack-ansible | 15:31 | |
| *** rodolof has quit IRC | 15:32 | |
| nelsson | shananigans i think we would have too much bug reports for this step :P | 15:33 |
| shananigans | lol. 2nd step. Shake all beach sand out of clothes on back patio :) | 15:34 |
| Miouge | odyssey4me: If/once #560395 is approved, it needs to be backported into stable/* and a change in ansible-requirements.yml is also needed or not? | 15:34 |
| admin0 | odyssey4me, https://gist.github.com/a1git/789d1d223b9af505ab79b22fbf805a19 -- maybe this helps ? | 15:38 |
| odyssey4me | Miouge yes | 15:38 |
| odyssey4me | we'll have to bring that role sha forward to ensure the fix is in the next release, I don't think this is something that can wait | 15:39 |
| *** gkadam_ has joined #openstack-ansible | 15:39 | |
| admin0 | odyssey4me, it truncated the file :( | 15:39 |
| *** gkadam has quit IRC | 15:40 | |
| *** vnogin has quit IRC | 15:40 | |
| *** idlemind has quit IRC | 15:41 | |
| admin0 | odyssey4me, sorry here: http://gw.am/setup-log.txt | 15:41 |
| *** idlemind has joined #openstack-ansible | 15:42 | |
| *** foutatoro has quit IRC | 15:47 | |
| *** DanyC has quit IRC | 15:48 | |
| *** DanyC has joined #openstack-ansible | 15:48 | |
| odyssey4me | admin0 is there a /root/.pip/pip.conf on those hosts? | 15:50 |
| odyssey4me | is this a clean setup, or some sort of previously implemented build that's being updated? | 15:50 |
| admin0 | new setup .. | 15:51 |
| evrardjp | odyssey4me: woot: http://logs.openstack.org/05/560405/5/check/openstack-ansible-linters/52b8a04/job-output.txt.gz#_2018-04-11_15_34_00_733092 | 15:51 |
| admin0 | odyssey4me, https://gist.github.com/a1git/fb14e05405c238729f8a6e7bc7f56b72 - this is in the pip.conf everywhere | 15:52 |
| admin0 | but because setup-infra never ran and it fails on setup-hosts, those url/endpoints do not exist | 15:52 |
| *** DanyC has quit IRC | 15:53 | |
| odyssey4me | well, I think it may be bad handling by pip - usually it ignores an unreachable URL, but in this case it's a 503 and it doesn't seem to be handling that properly | 15:54 |
| odyssey4me | this is quite weird because none of this has been a problem before | 15:54 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove default pip_links value https://review.openstack.org/549012 | 15:55 |
| *** hamzy has quit IRC | 15:55 | |
| *** kstev1 has joined #openstack-ansible | 15:56 | |
| odyssey4me | admin0 I'm not sure, but https://review.openstack.org/549012 may be helpful for this situation. Can you give it a go? | 15:56 |
| *** kstev has quit IRC | 15:57 | |
| odyssey4me | evrardjp awesome, can you make that more generalised - "Secrets should not be logged." | 15:57 |
| odyssey4me | It's in our role guidelines. | 15:57 |
| admin0 | odyssey4me, will give this a go | 15:58 |
| evrardjp | odyssey4me: right now it's only passwords | 15:58 |
| evrardjp | not sure how we can determine if what we call a "secret" is used | 15:58 |
| evrardjp | because that's just a var | 15:58 |
| *** cshen has quit IRC | 16:00 | |
| *** gkadam__ has joined #openstack-ansible | 16:01 | |
| *** gkadam_ has quit IRC | 16:04 | |
| *** gkadam has joined #openstack-ansible | 16:04 | |
| *** cristicalin has quit IRC | 16:05 | |
| *** vnogin has joined #openstack-ansible | 16:05 | |
| *** gkadam__ has quit IRC | 16:06 | |
| *** mma has quit IRC | 16:09 | |
| *** vnogin has quit IRC | 16:09 | |
| admin0 | odyssey4me, it passed that stage and failed at [lxc_hosts : Ensure image has been pre-staged] .. | 16:10 |
| admin0 | rerunning it again to check | 16:10 |
| *** mma has joined #openstack-ansible | 16:10 | |
| *** haad1 has joined #openstack-ansible | 16:10 | |
| *** armaan has quit IRC | 16:12 | |
| *** armaan has joined #openstack-ansible | 16:12 | |
| *** mma has quit IRC | 16:15 | |
| *** haad1 has quit IRC | 16:15 | |
| *** epalper has quit IRC | 16:16 | |
| *** kstev has joined #openstack-ansible | 16:26 | |
| *** kstev1 has quit IRC | 16:28 | |
| *** cristicalin has joined #openstack-ansible | 16:31 | |
| openstackgerrit | Shannon Mitchell proposed openstack/openstack-ansible-ops master: mnaio service setup skips shade install if using tags instead of skip-tags https://review.openstack.org/560560 | 16:31 |
| admin0 | odyssey4me, its moving forward | 16:32 |
| admin0 | that cherry pick helped this use case | 16:32 |
| *** udesale has quit IRC | 16:35 | |
| jrosser | i'm finding surprising results when using these constructs https://github.com/openstack/openstack-ansible-os_tempest/blob/master/tasks/tempest_resources.yml#L43 | 16:45 |
| jrosser | it seems that using target-host as the downloader tries to ssh directly to the container rather than use the ssh plugin | 16:45 |
| *** cmart has joined #openstack-ansible | 16:47 | |
| openstackgerrit | Shannon Mitchell proposed openstack/openstack-ansible-ops master: mnaio service setup skips shade install if using tags instead of skip-tags https://review.openstack.org/560560 | 16:48 |
| odyssey4me | jrosser oh yeah? hmm, that's no good | 16:49 |
| jrosser | i am very surprised tbh | 16:49 |
| jrosser | let me get a paste of it | 16:49 |
| tomtom001 | admin0 and odyssey4me, does that lxc issue you are discussing above have anything to do with the lxc image not preparing the cache correctly? | 16:50 |
| odyssey4me | tomtom001 nope, the issue is the nova-lxd driver | 16:50 |
| odyssey4me | nothing to do with the image prep process we do | 16:50 |
| *** madorn has quit IRC | 16:51 | |
| odyssey4me | tomtom001 I'm not aware of any image prep problems. Do you have a bug or something to look at? So far most of the time if there're issues they're due to proxies, slow mirrors or bad networking. | 16:51 |
| *** madorn has joined #openstack-ansible | 16:51 | |
| jrosser | odyssey4me: http://paste.openstack.org/show/718959/ | 16:52 |
| openstackgerrit | Shannon Mitchell proposed openstack/openstack-ansible-ops master: mnaio service setup skips shade install if using tags instead of skip-tags https://review.openstack.org/560560 | 16:53 |
| tomtom001 | my guess is that it's a slow mirror, because the build is taking forever, this is what is in my sources.list.d: deb http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/pike main is there any changing that? Is that where the lxc is coming from? | 16:53 |
| *** electrofelix has quit IRC | 16:53 | |
| tomtom001 | I also just upgraded the system to ocata last week, so I was not thinking it was bad networking on my end. | 16:55 |
| tomtom001 | I am also not behind a proxy | 16:55 |
| odyssey4me | tomtom001 the cache is updated with packages from /etc/apt/sources.list mostly | 16:56 |
| odyssey4me | is the issue in downloading the cahce or prepping it? | 16:56 |
| odyssey4me | if its taking long, you can also add a longer timeout | 16:57 |
| *** gkadam has quit IRC | 16:59 | |
| Miouge | Now that “cell_v2 discover” works, I ran into another error on my setup-openstack.yml: pip_install : Gather variables for each operating system fatal: [controller01_utility_container-07b32cd6]: FAILED! => {"failed": true, "msg": "No file was found when using with_first_found. Use the 'skip: true' option to allow this task to be skipped if no files are found”}. Any clues? | 17:07 |
| *** cristicalin has quit IRC | 17:13 | |
| odyssey4me | Miouge sounds like stale fact cache | 17:13 |
| tomtom001 | odyssey4me thank you | 17:14 |
| tomtom001 | issue seems to be prepping it, it dies in the middle of the apt-get install -y | 17:14 |
| odyssey4me | tomtom001 ok, try adding https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/defaults/main.yml#L170-L174 with a larger value in your user_variables.yml file | 17:16 |
| *** cmart has quit IRC | 17:17 | |
| *** cmart has joined #openstack-ansible | 17:18 | |
| *** hamzy has joined #openstack-ansible | 17:18 | |
| *** portante has quit IRC | 17:23 | |
| Miouge | odyssey4me: OK --flush-cache here I come then :) | 17:31 |
| *** portante has joined #openstack-ansible | 17:31 | |
| *** hamza21 has joined #openstack-ansible | 17:38 | |
| *** cshen has joined #openstack-ansible | 17:38 | |
| *** DanyC has joined #openstack-ansible | 17:42 | |
| *** DanyC_ has joined #openstack-ansible | 17:43 | |
| mattt | evrardjp odyssey4me : either of you still around? :) | 17:47 |
| *** DanyC has quit IRC | 17:47 | |
| odyssey4me | tomtom001 ok, try adding https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/defaults/main.yml#L170-L174 with a larger value in your user_variables.yml file | 17:48 |
| odyssey4me | oops | 17:48 |
| odyssey4me | yep, I'm around - lemme just get some water | 17:48 |
| mattt | evrardjp odyssey4me : so https://review.openstack.org/#/c/560025/ passes in pike, but fails in master ... any suggestions on what we should do here? | 17:49 |
| evrardjp | I am leaving soon | 17:49 |
| *** mbuil has quit IRC | 17:50 | |
| odyssey4me | mattt personally I'd say let's add an explanation of why this is being patched directly into the stable branch and not being a backport. To me it's simple - for stable environments we need to ensure that we deliver a consistent result. In master we need another solution due to disparate constraints. So I'd say let's merge this into the stable branches and figure out what to do in master afterwards. | 17:51 |
| mattt | odyssey4me: sounds good | 17:53 |
| *** armaan has quit IRC | 17:54 | |
| *** armaan has joined #openstack-ansible | 17:54 | |
| *** osnaya has joined #openstack-ansible | 18:01 | |
| osnaya | @cloudnull @admin0 @evrardjp @odyssey4me Ubuntu 16.04 LTS comes with Python version 3.5.... Does OSA work with both Python 3.5 and/or Python 2.7.12 for the deployment? | 18:03 |
| logan- | osnaya: osa will install python2 immediately when beginning to set up a host https://github.com/openstack/openstack-ansible/blob/9b6162c27a662f191cb1415860dc8da717258f55/playbooks/openstack-hosts-setup.yml#L22-L37 | 18:05 |
| osnaya | @logan you mean when running setup-hosts.yml.... ? | 18:06 |
| odyssey4me | yes, that playbook is included in setup-hosts | 18:06 |
| * jrosser curses python3, still having trouble... | 18:08 | |
| jrosser | i never really got a straight answer about the deploy host | 18:08 |
| jrosser | boostrap seems to install/detect/use pyton3 | 18:09 |
| odyssey4me | jrosser I wonder if we're not seeing that in gating (or in my own builds) because python2 is already there too | 18:10 |
| logan- | jrosser: yeah the ansible-runtime venv has used py3 since pike | 18:10 |
| odyssey4me | the deploy host needs both py2 and py3, but the venv should only need 3 | 18:11 |
| jrosser | well on my pike deploy host 2.7 was in the venv | 18:11 |
| jrosser | and queens its put 3 there | 18:11 |
| jrosser | i get a very obvious 2 vs. 3 blowup almost immediatly with queens | 18:12 |
| jrosser | which clearly doesnt happen in the gates | 18:12 |
| tomtom001 | odyssey4me thank you for the info, changing the delay as you proposed in the user_variable.yml file worked! | 18:13 |
| odyssey4me | tomtom001 excellent :) | 18:13 |
| *** poopcat has joined #openstack-ansible | 18:13 | |
| odyssey4me | sad that you have to wait so long, but happy that it's working | 18:14 |
| tomtom001 | Yeah I though it was the delay in the role where the note is about setting the delay. | 18:15 |
| tomtom001 | fwiw: I thought the issue was here: /etc/ansible/roles/lxc_hosts/tasks/lxc_cache_preparation.yml and that changing the delay there would resolve the issue, but after many attempts I ran into the same problem with the container not building. | 18:17 |
| tomtom001 | Line 130 of that file | 18:17 |
| logan- | jrosser: youre right about pike.. we ran with py3 during the whole dev cycle for pike and then reverted the venv back to py2 for release so that change could bake for another cycle | 18:18 |
| logan- | weird that youre seeing the issue on queens though | 18:18 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_cinder master: (WIP) Update to use oslo.messaging service for RPC and Notify https://review.openstack.org/560574 | 18:19 |
| logan- | so the gate has been bootstrapping py3 ansible venvs for pike, queens, and rocky so far. but i guess deployers are only beginning to see py3 ansible-runtime venvs in queens | 18:19 |
| *** radeks has quit IRC | 18:21 | |
| *** radeks has joined #openstack-ansible | 18:22 | |
| odyssey4me | jrosser have you registered a bug for it yet? | 18:22 |
| *** cshen has quit IRC | 18:24 | |
| jrosser | odyssey4me: not yet i have a bit too much in flught today, but can look tomorrow in more detail | 18:29 |
| jrosser | i have a hack around it (disable py3 detection) so ive had a queens deploy run through completely today like that | 18:30 |
| *** armaan has quit IRC | 18:31 | |
| *** armaan has joined #openstack-ansible | 18:31 | |
| *** armaan has quit IRC | 18:38 | |
| *** armaan has joined #openstack-ansible | 18:39 | |
| *** cshen has joined #openstack-ansible | 18:39 | |
| *** osnaya has quit IRC | 18:43 | |
| *** cshen has quit IRC | 18:44 | |
| *** cshen has joined #openstack-ansible | 18:45 | |
| jrosser | logan-: i was wondering if this was all about a mismatch between the deploy host and the rest of the nodes, ansible+py3 on the deploy, and py2 everywhere else | 18:47 |
| *** pcaruana has quit IRC | 18:55 | |
| logan- | your osa hosts (not the deployment host).. what python is on there? clearly python2 since we install that ourselves. do they have py3 also? | 18:56 |
| logan- | it'll be interesting to see where it fails once you have time to file a bug | 18:56 |
| *** chhagarw has quit IRC | 18:58 | |
| *** cshen has quit IRC | 19:09 | |
| *** mma has joined #openstack-ansible | 19:10 | |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-tests master: (WIP) Update to use oslo.messaging services for RPC and Notify https://review.openstack.org/559127 | 19:13 |
| *** mma has quit IRC | 19:14 | |
| *** adcpdk has joined #openstack-ansible | 19:14 | |
| adcpdk | odyssey4me are you available? | 19:16 |
| adcpdk | I have partial success adding xen-based hypervisor as a compute node | 19:17 |
| adcpdk | at first I found that the neutron service wasn't running, so I had to run the following command: | 19:23 |
| adcpdk | ccn@kvm01:/opt/openstack-ansible/playbooks$ sudo openstack-ansible os-neutron-install.yml --limit compute-04 | 19:23 |
| adcpdk | then rebooted the compute host | 19:23 |
| adcpdk | now I don't have the networking problem and the instance gets created, and destroyed by the hypervisor right after | 19:24 |
| adcpdk | https://paste.ubuntu.com/p/Q7YHB3dPP6/ | 19:24 |
| adcpdk | here are nova logs I have: https://paste.ubuntu.com/p/TWrCnjCWps/ | 19:25 |
| adcpdk | is there anyone to assist? | 19:25 |
| adcpdk | this line in logs disturbs me: | 19:26 |
| adcpdk | 2018-04-11 13:54:38.836 4826 ERROR nova.virt.libvirt.guest [req-23241830-812d-484e-a20d-3dd1425d48d7 162c881cfcb64f67b13848096c8b7148 f1089f0010bb4398bf1f0931477fc5d3 - default default] Error launching a defined domain with XML: <domain type='xen'> | 19:26 |
| *** cshen has joined #openstack-ansible | 19:36 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add test for an omitted delegation https://review.openstack.org/560595 | 19:38 |
| *** cshen has quit IRC | 19:40 | |
| *** cshen has joined #openstack-ansible | 19:43 | |
| *** vnogin has joined #openstack-ansible | 19:51 | |
| *** haad has joined #openstack-ansible | 19:53 | |
| *** coreywright has quit IRC | 19:53 | |
| admin0 | lxc-containers-create: fatal: [c3_nova_api_container-1a2c31e3]: FAILED! => {"changed": false, "module_stderr": "mesg: ttyname failed: Inappropriate ioctl for device\n/bin/sh: 1: | 19:55 |
| admin0 | /usr/bin/python: not found\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 127} | 19:55 |
| admin0 | sorry .. multiline | 19:55 |
| admin0 | i am surely facing an issue never seen before | 19:55 |
| *** adcpdk has quit IRC | 19:57 | |
| *** adcpdk has joined #openstack-ansible | 19:58 | |
| mattt | admin0: the "/usr/bin/python: not found" is something i'd probably start with :) | 20:00 |
| *** jafeha has joined #openstack-ansible | 20:04 | |
| *** jafeha__ has quit IRC | 20:05 | |
| *** jwitko has quit IRC | 20:05 | |
| jrosser | oh man i broke the connection plugin http://logs.openstack.org/95/560595/1/check/openstack-ansible-functional-ubuntu-xenial/4344e12/job-output.txt.gz#_2018-04-11_19_56_29_818433 | 20:07 |
| jrosser | odyssey4me: ^ i wrote a test for that delegate/omit pattern | 20:07 |
| *** coreywright has joined #openstack-ansible | 20:12 | |
| admin0 | mattt, right .. but i can't also do apt install python in all the lxc images/containers .. i start with one and then i have to do all the depndencies myself .. i am trying to figure out whats diff in the stable/queens and my env | 20:14 |
| *** vnogin has quit IRC | 20:18 | |
| *** markvoelker has joined #openstack-ansible | 20:20 | |
| *** liangqi_ has joined #openstack-ansible | 20:20 | |
| *** mubix_ has joined #openstack-ansible | 20:21 | |
| *** xdfil_ has joined #openstack-ansible | 20:21 | |
| *** berendt_ has joined #openstack-ansible | 20:21 | |
| *** geekinutah_ has joined #openstack-ansible | 20:21 | |
| *** v1k0d3n_ has joined #openstack-ansible | 20:21 | |
| *** ansmith has quit IRC | 20:23 | |
| *** tomcsi_ has joined #openstack-ansible | 20:24 | |
| *** toanster has joined #openstack-ansible | 20:24 | |
| *** osnaya has joined #openstack-ansible | 20:24 | |
| *** vnogin has joined #openstack-ansible | 20:25 | |
| *** leader716 has joined #openstack-ansible | 20:25 | |
| *** tobberydberg_ has joined #openstack-ansible | 20:27 | |
| *** liangqi has quit IRC | 20:28 | |
| *** galstrom_zzz has quit IRC | 20:28 | |
| *** tomcsi has quit IRC | 20:28 | |
| *** toan has quit IRC | 20:28 | |
| *** v1k0d3n has quit IRC | 20:28 | |
| *** xdfil has quit IRC | 20:28 | |
| *** mubix has quit IRC | 20:28 | |
| *** geekinutah has quit IRC | 20:28 | |
| *** johnsom has quit IRC | 20:28 | |
| *** berendt has quit IRC | 20:28 | |
| *** ArchiFleKs has quit IRC | 20:28 | |
| *** jpward has quit IRC | 20:28 | |
| *** tobberydberg has quit IRC | 20:28 | |
| *** astellwag has quit IRC | 20:28 | |
| *** liangqi_ is now known as liangqi | 20:28 | |
| *** xdfil_ is now known as xdfil | 20:28 | |
| *** geekinutah_ is now known as geekinutah | 20:28 | |
| *** v1k0d3n_ is now known as v1k0d3n | 20:28 | |
| *** berendt_ is now known as berendt | 20:28 | |
| *** mubix_ is now known as mubix | 20:28 | |
| *** galstrom_zzz has joined #openstack-ansible | 20:29 | |
| *** johnsom has joined #openstack-ansible | 20:29 | |
| *** astellwag has joined #openstack-ansible | 20:29 | |
| *** ArchiFleKs has joined #openstack-ansible | 20:29 | |
| *** laser__ has joined #openstack-ansible | 20:32 | |
| *** cshen has quit IRC | 20:35 | |
| laser__ | I'm having an issue deploying queens on ubuntu hosts using the "Test" deployment model with 3 hosts. On the compute host when running setup-infrastructure.yaml the Apply rabbitmq policies errors out with:.....Hostname mismatch: node \"rabbit@infra1-rabbit-mq-container-427bd52e\" believes its host is different...... Any clue where to start looking? I've done the lookup on the compute host and inside the container and both | 20:35 |
| laser__ | return fine. | 20:35 |
| *** evin has quit IRC | 20:40 | |
| *** hamza21 has quit IRC | 20:47 | |
| *** cpusmith has quit IRC | 20:51 | |
| *** cshen has joined #openstack-ansible | 20:51 | |
| *** cshen has quit IRC | 20:56 | |
| *** vnogin has quit IRC | 21:13 | |
| *** vnogin has joined #openstack-ansible | 21:18 | |
| *** vnogin has quit IRC | 21:24 | |
| osnaya | @evrardjp @cloudnull @odyssey4me @idlemind @logan setup-hosts.yml throwing the following 2 fatal errors on controller node on TASK [pip-install: Install pip packages] pls see http://paste.openstack.org/show/718992/ any pointers? | 21:25 |
| osnaya | @evrardjp @cloudnull @odyssey4me @idlemind @logan "ERROR: ERROR: You must give at least one requirement to install (see \\"pip help install\\")\\n", ... During installation is there any manual requirements that I need to be aware of? | 21:27 |
| *** mma has joined #openstack-ansible | 21:28 | |
| osnaya | <osnaya> @evrardjp @cloudnull @odyssey4me @idlemind @logan going thru log... I believe the same error is thrown twice (once on the first run after 6 retries and then same is thrown on (fall back mode) after its 6 retries. | 21:32 |
| *** DanyC_ has quit IRC | 21:33 | |
| *** mma has quit IRC | 21:33 | |
| *** mma has joined #openstack-ansible | 21:33 | |
| *** DanyC has joined #openstack-ansible | 21:33 | |
| *** kstev has quit IRC | 21:35 | |
| *** ansmith has joined #openstack-ansible | 21:37 | |
| *** DanyC has quit IRC | 21:38 | |
| *** cjloader_ has joined #openstack-ansible | 21:42 | |
| *** cjloader has quit IRC | 21:42 | |
| *** cjloader_ has quit IRC | 21:47 | |
| *** esberglu has quit IRC | 21:47 | |
| *** MikeW has quit IRC | 21:48 | |
| *** idlemind has quit IRC | 21:49 | |
| *** throwsb1 has quit IRC | 21:52 | |
| *** laser__ has quit IRC | 21:53 | |
| *** dariko has joined #openstack-ansible | 21:54 | |
| osnaya | <osnaya> @evrardjp @cloudnull @odyssey4me @idlemind @logan @admin0 checked /opt/global-requirement-pins.txt has the following entries ---> pip==9.0.1 setuptools==38.5.0 wheel==0.30.0. and https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?id=54d1a42531aadf9ed166961566c676d9ac3e1521 shows up a huge list packages with version numbers... Any pointers as to why ERROR says "you must give at least | 21:56 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Allow public network name to be changed https://review.openstack.org/560670 | 21:56 |
| admin0 | osnaya, i had that error which was fixed by a cherry pick .but i forgot which cherry pick | 21:57 |
| admin0 | it was something from cloudnull | 21:57 |
| admin0 | check irc logs here .. its midnight for me . so i am out :D | 21:58 |
| admin0 | odyssey4me pointed me to the patch . he might recall... | 21:58 |
| osnaya | @admin0 Thanks. Please get some sleep | 21:59 |
| osnaya | @odyssey4me ... can you please help.... for workaround or cherry pick that admin0 mentioned above? | 22:01 |
| *** jra has joined #openstack-ansible | 22:07 | |
| *** hamzy has quit IRC | 22:07 | |
| *** radeks has quit IRC | 22:10 | |
| jra | We're installing a small dev cluster on pike with OSA tag 16.0.10; at the end of the nova role, it tries to do a cell_v2 host discovery, and fails with an error suggesting that ssh-ing to a conductor node failed because the DNS name couldn't be resolved | 22:12 |
| jra | it seems like the deployment host needs a hosts file that includes the container names and IPs, such as is created on the compute nodes | 22:13 |
| jra | but no such file is created by the OSA plays | 22:13 |
| jra | is this a known thing? | 22:13 |
| *** dariko has quit IRC | 22:21 | |
| SamYaple | andymccr: ping -- you still working with the uwsgi stuff? | 22:22 |
| *** cshen has joined #openstack-ansible | 22:25 | |
| -openstackstatus- NOTICE: zuul was restarted to updated to the latest code; you may need to recheck changes uploaded or approvals added between 21:30 and 21:45 | 22:29 | |
| *** cshen has quit IRC | 22:30 | |
| *** cjloader has joined #openstack-ansible | 22:32 | |
| *** cmart has quit IRC | 22:33 | |
| *** cjloader has quit IRC | 22:37 | |
| jra | anybody? I can open a bug on this if folks think that'd be worthwhile | 22:40 |
| *** jafeha__ has joined #openstack-ansible | 22:41 | |
| *** jafeha has quit IRC | 22:42 | |
| shananigans | jra: sounds like https://bugs.launchpad.net/openstack-ansible/+bug/1762742 | 22:47 |
| openstack | Launchpad bug 1762742 in openstack-ansible "Some delegated tasks are failing when deployer is node part of a deployment (eg. controller node)" [Undecided,New] | 22:47 |
| jra | Actually, I just found this which is an exact match: https://bugs.launchpad.net/openstack-ansible/+bug/1752540 | 22:49 |
| openstack | Launchpad bug 1752540 in openstack-ansible "os_nova cell_v2 discover failure" [Undecided,In progress] - Assigned to git-harry (git-harry) | 22:49 |
| osnaya | @jra just curious since you are deploying dev cluster using OSA... Which linux flavor are you deploying on ? Ubuntu/CentOS? | 22:49 |
| jra | osnaya: CentOS | 22:49 |
| jra | shananigans: reading the bug you posted closely, I agree these are the same base issue | 22:50 |
| jra | it's down to how Ansible handles the random conductor selection for delegation | 22:50 |
| osnaya | @jra by chance did you hit the issue that I am hitting during setup-hosts.yml ... http://paste.openstack.org/show/718992/ | 22:51 |
| jra | osnaya: I was literally fighting with that error this morning | 22:52 |
| jra | We got past it by up-ing our OSA tag to 16.0.10 from 16.0.8 | 22:53 |
| jra | which got us to the error I'm complaining about now | 22:53 |
| osnaya | @jra BTW, I am using Pike | 22:53 |
| jra | @osnaya yes, us too | 22:54 |
| *** haad has quit IRC | 23:01 | |
| osnaya | @jra I downloaded Pike more than a week ago.... do you know how I can check what version the installed OSA is? (not sure minor version 16.0.xx?) | 23:01 |
| jra | @osnaya: git describe --tags | 23:03 |
| osnaya | @jra thanks. Yes mine is older too.... When I look at the https://docs.openstack.org/project-deploy-guide/openstack-ansible/pike/overview-osa.html now.... it shows up 10.0.11 | 23:05 |
| jra | yep, but the deployment guide still calls for 16.0.10 | 23:07 |
| jra | and the release notes don't describe .11 yet | 23:07 |
| jra | so I presume it's in progress but not released | 23:07 |
| osnaya | @jra so If I need to up the version, do I just do git clone again.... Do I need to recreate configurations under /etc/openstack-ansible.....? or just changing source version is all that I need to do? What is your experience? | 23:13 |
| jra | @osnaya: Move your current /opt/openstack-ansible out of the way, then "git clone -b 16.0.10 https://git.openstack.org/openstack/openstack-ansible /opt/openstack-ansible" | 23:14 |
| jra | Then run the bootstrap again | 23:15 |
| osnaya | @jra Cool... So changes in /etc/openstack-ansible configurations... will try the above. Thanks. | 23:17 |
| *** cshen has joined #openstack-ansible | 23:20 | |
| *** cshen has quit IRC | 23:24 | |
| osnaya | @jra since I had run setup-hosts.yml already on target host.... now that I am upgrading OSA version.... Can I just run setup-hosts.yml again or do I need to do cleanup on target before I run it again.... with ansible idempotent nature, running it again should work, right? | 23:30 |
| jra | @osnaya It'll probably work, but our procedure is to wipe the host back to the metal before every OSA run | 23:45 |
| osnaya | @jra ok.... is there a clean up script to do it or re-image and re-create all bridges etc? | 23:46 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!