| *** jamielennox|away is now known as jamielennox | 00:01 | |
| *** jamesdenton has quit IRC | 00:03 | |
| *** sdake has quit IRC | 00:10 | |
| *** daneyon has joined #openstack-ansible | 00:18 | |
| cloudnull | odyssey4me: this needs a bit more testing but looks like it works and is feature complete w/ pipelining and all https://gist.github.com/cloudnull/9c798af2920ec1593b028be928c15513 | 00:20 |
|---|---|---|
| *** DrifterZA has quit IRC | 00:22 | |
| *** wadeholl_ has quit IRC | 00:23 | |
| openstackgerrit | Michael Gugino proposed openstack/openstack-ansible-os_tempest: Add additional test coverage https://review.openstack.org/350806 | 00:28 |
| coolj | cloudnull: i think this is a typo? https://gist.github.com/cloudnull/9c798af2920ec1593b028be928c15513#file-lxc_ssh-py-L117 | 00:29 |
| cloudnull | coolj: nice catch | 00:30 |
| cloudnull | :) | 00:30 |
| cloudnull | updated. | 00:30 |
| *** johnmilton has joined #openstack-ansible | 00:34 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_ceilometer: RabbitMQ config rework https://review.openstack.org/350839 | 00:39 |
| *** thorst has joined #openstack-ansible | 00:45 | |
| openstackgerrit | Michael Gugino proposed openstack/openstack-ansible-os_tempest: Add additional test coverage https://review.openstack.org/350806 | 00:47 |
| *** johnmilton has quit IRC | 00:59 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible: Support multiple rabbitmq clusters https://review.openstack.org/344884 | 01:01 |
| *** thorst has quit IRC | 01:04 | |
| *** thorst has joined #openstack-ansible | 01:05 | |
| *** elo has quit IRC | 01:07 | |
| *** thorst has quit IRC | 01:13 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Implement an opportunistic strategy https://review.openstack.org/349485 | 01:15 |
| cloudnull | odyssey4me: ^ idk it'll gate but that its worth a look. | 01:16 |
| cloudnull | coolj: -cc ^ | 01:17 |
| coolj | nice! | 01:23 |
| *** asettle has joined #openstack-ansible | 01:24 | |
| *** asettle has quit IRC | 01:28 | |
| openstackgerrit | Michael Gugino proposed openstack/openstack-ansible-os_tempest: Enable broader image support https://review.openstack.org/350781 | 01:32 |
| openstackgerrit | David Wittman proposed openstack/openstack-ansible: Fix incorrect operations link in run-playbooks.sh https://review.openstack.org/350858 | 01:32 |
| cloudnull | DWitt in the house? | 01:34 |
| *** jamielennox is now known as jamielennox|away | 01:34 | |
| openstackgerrit | Michael Gugino proposed openstack/openstack-ansible-os_tempest: Add additional test coverage https://review.openstack.org/350806 | 01:39 |
| *** weshay has quit IRC | 01:40 | |
| coolj | looks like it | 01:44 |
| *** retreved has joined #openstack-ansible | 01:44 | |
| *** thorst has joined #openstack-ansible | 01:46 | |
| *** smatzek has joined #openstack-ansible | 01:46 | |
| *** thorst has quit IRC | 01:47 | |
| *** thorst has joined #openstack-ansible | 01:47 | |
| *** thorst has quit IRC | 01:56 | |
| *** dmellado has quit IRC | 01:57 | |
| *** ajo has quit IRC | 01:57 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_ceilometer: RabbitMQ config rework https://review.openstack.org/350839 | 01:59 |
| *** dmellado has joined #openstack-ansible | 02:02 | |
| *** ajo has joined #openstack-ansible | 02:03 | |
| *** retreved has quit IRC | 02:04 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Implement an opportunistic strategy https://review.openstack.org/349485 | 02:12 |
| *** thorst has joined #openstack-ansible | 02:18 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Update all SHAs for Newton 2016-08-01 https://review.openstack.org/349611 | 02:23 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-apt_package_pinning: Adding idempotency testing to the role func test https://review.openstack.org/350870 | 02:23 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-apt_package_pinning: Adding Vagrantfile for developer testing https://review.openstack.org/350872 | 02:25 |
| *** markvoelker has joined #openstack-ansible | 02:26 | |
| *** thorst has quit IRC | 02:27 | |
| *** thorst has joined #openstack-ansible | 02:28 | |
| openstackgerrit | Darren Chan proposed openstack/openstack-ansible: [docs] Revise deployment host and target hosts chapter https://review.openstack.org/350874 | 02:28 |
| *** furlongm_ has joined #openstack-ansible | 02:28 | |
| *** furlongm has quit IRC | 02:29 | |
| *** jamielennox|away is now known as jamielennox | 02:36 | |
| *** thorst has quit IRC | 02:37 | |
| *** zerda2 has joined #openstack-ansible | 02:44 | |
| *** chandanc_ has joined #openstack-ansible | 02:49 | |
| openstackgerrit | Darren Chan proposed openstack/openstack-ansible: [docs] Move ops content and fix build errors https://review.openstack.org/350936 | 02:50 |
| openstackgerrit | Darren Chan proposed openstack/openstack-ansible: [docs] Revise deployment host and target hosts chapter https://review.openstack.org/350874 | 02:53 |
| *** itlinux has quit IRC | 02:54 | |
| *** itlinux has joined #openstack-ansible | 02:56 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-openstack_hosts: Adding idempotency testing to the role func test https://review.openstack.org/350939 | 03:00 |
| *** sdake has joined #openstack-ansible | 03:04 | |
| *** fignew has quit IRC | 03:08 | |
| odyssey4me | cloudnull the common testing repo is on the way: https://review.openstack.org/350503 | 03:16 |
| odyssey4me | cloudnull nice work on the ssh_lxc connection plugin... it looks so much simpler than I thought it would be :) I'll have to give it a whirl asap | 03:19 |
| *** smatzek has quit IRC | 03:20 | |
| *** sdake has quit IRC | 03:23 | |
| cloudnull | i think there are bugs with it, but it gives you an idea of whats involved. | 03:24 |
| *** spedione|AWAY is now known as spedione | 03:24 | |
| odyssey4me | heh, things have grown since I last looked: http://docs.openstack.org/developer/performance-docs/ | 03:25 |
| *** spedione is now known as spedione|AWAY | 03:27 | |
| odyssey4me | cloudnull if you have a moment to review https://review.openstack.org/344912 I'd appreciate it | 03:28 |
| odyssey4me | not sure if anyone else is around to do the same? | 03:28 |
| cloudnull | ++ | 03:29 |
| odyssey4me | jmccrory you're sometimes around at this time :) | 03:29 |
| *** afred312_ has quit IRC | 03:29 | |
| *** persia has quit IRC | 03:30 | |
| *** persia has joined #openstack-ansible | 03:33 | |
| *** thorst has joined #openstack-ansible | 03:35 | |
| logan- | i think i found the issue with the horizon gate | 03:37 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_aodh: Add project group to role https://review.openstack.org/350948 | 03:37 |
| odyssey4me | logan- oh? | 03:37 |
| logan- | anytime i build an AIO with debug=false it fails to start due to http://cdn.pasteraw.com/80bqvxborw4f3a74yl7mb7get40lkit .. turns out django doesn't mind invalid hostnames if debug is enabled https://github.com/django/django/blob/master/django/http/request.py#L100-L112 | 03:38 |
| odyssey4me | cloudnull is https://review.openstack.org/350948 correct for the changes made to py_pkgs to optimise the wheel build? | 03:38 |
| logan- | i think in the horizon gate debug is disabled | 03:38 |
| logan- | don't see any vars enabling it | 03:38 |
| odyssey4me | ah | 03:38 |
| odyssey4me | good unintended test coverage :p | 03:38 |
| logan- | going to push a patch with debug=true just to test real quick, because i'm not 100% sure whether or not the horizon gate has a hostname with "_" in it | 03:40 |
| logan- | nvm, looks like the hostname is horizon1 so doesn't look like it | 03:41 |
| *** fignew has joined #openstack-ansible | 03:41 | |
| logan- | still an issue in the integrated build though since the apache ServerName is just getting set to the container name | 03:42 |
| *** thorst has quit IRC | 03:42 | |
| odyssey4me | logan- I thought we had addressed that though | 03:45 |
| odyssey4me | hmm, it appears that https://github.com/openstack/openstack-ansible-lxc_container_create/blob/master/tasks/container_create.yml#L281-L308 is not in mitaka | 03:46 |
| *** markvoelker has quit IRC | 03:46 | |
| odyssey4me | not sure if it should be, but that's what I though handled the issue of the hostname being correct | 03:46 |
| odyssey4me | logan- ah https://review.openstack.org/308440 | 03:49 |
| odyssey4me | for master that was moved into the role at a later stage: https://review.openstack.org/313821 | 03:49 |
| cloudnull | odyssey4me: yes. | 03:49 |
| cloudnull | however I think it'd be best to simple set the *_role_project_group variable to just "all" and we can limit them within the integrated build. | 03:50 |
| *** sdake has joined #openstack-ansible | 03:50 | |
| cloudnull | IE what we did here https://review.openstack.org/#/c/347936/ | 03:50 |
| odyssey4me | cloudnull hmm, but py_pkgs defaults to having it as 'all' if it doesn't find a var, doesn't it? | 03:50 |
| cloudnull | it does. | 03:51 |
| logan- | the hostname of the container is correct | 03:51 |
| logan- | but the apache conf in /etc/apache2/sites-available/openstack-dashboard.conf is wrong | 03:51 |
| logan- | ServerName aio1_horizon_container-a63be9bc | 03:51 |
| odyssey4me | cloudnull so if we, simply in group_vars - maybe just the repo_all group vars, set a list of the var: group mappings for all the roles... then it'll work? | 03:51 |
| cloudnull | so thats another thing, it really doesn't need to be set in the role. | 03:51 |
| cloudnull | odyssey4me: https://review.openstack.org/#/c/347936/ merged it should already be working for the integrated build. | 03:52 |
| logan- | /etc/ansible/roles/os_horizon/templates/openstack_dashboard.conf.j2: ServerName {{ horizon_server_name }} | 03:52 |
| logan- | openstack-ansible/playbooks/os-horizon-install.yml: horizon_server_name: "{{ container_name }}" | 03:52 |
| odyssey4me | https://github.com/openstack/openstack-ansible-os_horizon/blob/master/defaults/main.yml#L62 | 03:53 |
| logan- | need an rfc compliant hostname getting loaded into horizon_server_name | 03:53 |
| odyssey4me | ugh, that override in the playbook is silly | 03:53 |
| odyssey4me | not sure why that's there | 03:53 |
| logan- | ah yeah, the default in the horizon role is ansible_hostname, which i guess probably has the correct format | 03:54 |
| odyssey4me | cloudnull testing it on a built AIO quickly | 03:55 |
| cloudnull | ok | 03:56 |
| odyssey4me | cloudnull ok, I think you may have your wires crossed :) | 03:58 |
| odyssey4me | with https://review.openstack.org/347936 we only added the git repo -> host group association | 03:59 |
| odyssey4me | that is for the selective git cloning | 03:59 |
| cloudnull | yes | 03:59 |
| cloudnull | oh, we're talking about wheel building ? | 03:59 |
| odyssey4me | for the selective wheel building, we need to use the role_project_groups | 03:59 |
| odyssey4me | and that needs the var in the role in order to associate the role name with the group | 04:00 |
| cloudnull | ah yea. | 04:00 |
| cloudnull | so yes, that var needs to be in the role. | 04:00 |
| * cloudnull grabs another beer | 04:00 | |
| odyssey4me | :) | 04:00 |
| *** daneyon has quit IRC | 04:01 | |
| *** albertcard has left #openstack-ansible | 04:01 | |
| odyssey4me | ok, let me bang a few of those patches out in the hope that I can get some reviews of https://review.openstack.org/344912 today and get it merged, assuming it's well liked | 04:01 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_barbican: Add project group to role https://review.openstack.org/350952 | 04:04 |
| *** hybridpollo has quit IRC | 04:04 | |
| * odyssey4me makes coffee | 04:08 | |
| openstackgerrit | Logan V proposed openstack/openstack-ansible: Do not override Horizon ServerName in playbook https://review.openstack.org/350955 | 04:08 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer: Add project group to role https://review.openstack.org/350956 | 04:09 |
| *** sguduru has joined #openstack-ansible | 04:09 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder: Add project group to role https://review.openstack.org/350957 | 04:10 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cloudkitty: Add project group to role https://review.openstack.org/350958 | 04:10 |
| *** sguduru has quit IRC | 04:11 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_designate: Add project group to role https://review.openstack.org/350959 | 04:11 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance: Add project group to role https://review.openstack.org/350960 | 04:12 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance: Add project group to role https://review.openstack.org/350960 | 04:12 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_gnocchi: Add project group to role https://review.openstack.org/350962 | 04:13 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_heat: Add project group to role https://review.openstack.org/350963 | 04:13 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_horizon: Add project group to role https://review.openstack.org/350964 | 04:14 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ironic: Add project group to role https://review.openstack.org/350965 | 04:15 |
| *** xnodeza has joined #openstack-ansible | 04:15 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone: Add project group to role https://review.openstack.org/350966 | 04:16 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible: Do not override Horizon ServerName in playbook https://review.openstack.org/350967 | 04:16 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_magnum: Add project group to role https://review.openstack.org/350968 | 04:16 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron: Add project group to role https://review.openstack.org/350969 | 04:17 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_rally: Add project group to role https://review.openstack.org/350971 | 04:18 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_sahara: Add project group to role https://review.openstack.org/350972 | 04:18 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_swift: Add project group to role https://review.openstack.org/350973 | 04:19 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest: Add project group to role https://review.openstack.org/350974 | 04:20 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_zaqar: Add project group to role https://review.openstack.org/350975 | 04:20 |
| odyssey4me | that takes care of that | 04:21 |
| *** markvoelker has joined #openstack-ansible | 04:22 | |
| mcarden | All manual? | 04:23 |
| odyssey4me | mcarden yep :) | 04:24 |
| odyssey4me | could perhaps have scripted it, but meh | 04:24 |
| mcarden | Figured so because of the designate spill into glance. :) | 04:24 |
| odyssey4me | on to a different topic, not sure if you have the energy for it cloudnull but I'd like your thoughts ont he matter | 04:24 |
| odyssey4me | I've been looking into container start failure in the integrated build. | 04:25 |
| cloudnull | what's up? | 04:25 |
| odyssey4me | As it turns out the container starts just fine - it's just that *sometimes* the secondary interfaces don't. | 04:25 |
| odyssey4me | And due to the fact that we rely on the secondary interface (br-mgmt) for Ansible to SSH to, we fail. | 04:25 |
| odyssey4me | The trouble is that we don't collect any useful logs on the host to troubleshoot that. | 04:26 |
| odyssey4me | right now we do a bind-mount to store the 'service' for the container in /openstack/log | 04:26 |
| *** chandanc_ has quit IRC | 04:26 | |
| *** markvoelker has quit IRC | 04:26 | |
| odyssey4me | ie for the glance container, we bind mount to /openstack/log/<container name> on the host from /var/log/glance in the container | 04:27 |
| odyssey4me | I was wondering whether we should perhaps change things up and actually just bind mount /var/log from the container to the host. | 04:27 |
| odyssey4me | that would give us all the upstart, dmesg, syslog, etc log files | 04:28 |
| odyssey4me | we *could* do this just for the gate... but I'm wondering whether this isn't a useful idea for production | 04:28 |
| odyssey4me | logan- you run a lean environment - what're your thoughts? | 04:29 |
| odyssey4me | mcarden what do you think? | 04:30 |
| mcarden | I think that any mechanism that help gather useful logging is worth a try | 04:31 |
| odyssey4me | before you go down the rsyslog rabbit hole, the problem I'm trying to tackle happens *before* rsyslog is setup | 04:31 |
| mcarden | BTW, I'm observing these container issues (one random one each time) on Trusty AIOs but not on Xenail AIOs. | 04:32 |
| odyssey4me | cloudnull by the way, I see the POST_failure and will ask infra to look into it | 04:33 |
| *** sguduru has joined #openstack-ansible | 04:33 | |
| odyssey4me | it appears that http://logs.openstack.org/11/349611/2/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-trusty/8c18f82/_zuul_ansible/ansible_log.txt is showing that the copy is failing due to missing keys or something | 04:33 |
| odyssey4me | mcarden yeah, we aren't yet testing xenial extensively for the integrated build - although an experimental job is available to do so | 04:34 |
| odyssey4me | mcarden if you want to help work out the kinks for xenial/centos then please feel free to go ahead with it... xenial support is a key deliverable for this cycle | 04:35 |
| mcarden | Yep - this is just me running up test AIO in the rax public cloud. Xenial never seems to have these container problems but Trusty consistently does. | 04:35 |
| odyssey4me | yeah, it could be a myriad of things causing it - kernel versions are the most likely culprit... but the point is that to figure out the cause and then determine what to do about it will require more logs, which I don't have access to today | 04:36 |
| odyssey4me | when I build on a trusty image, it's very rare that I hit this | 04:36 |
| mcarden | Yep. | 04:36 |
| *** sdake has quit IRC | 04:37 | |
| odyssey4me | I have hit the bug referenced in https://review.openstack.org/350684 once or twice. As soon as I understood the cause I knew how best to fix it. | 04:37 |
| logan- | I think it will not be a big difference in my env. The lean nodes do udp syslog only and write nothing to disk. On the controllers it will change the root location of the logs of course but no net difference since it's the same partition for /openstack as lxc roots | 04:38 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts: Implement primary and secondary keyserver usage for cache prep https://review.openstack.org/350684 | 04:38 |
| odyssey4me | it appears that I got trigger happy with hyphens there :p | 04:38 |
| odyssey4me | logan- ok, so it won't negatively impact your environment at least | 04:39 |
| odyssey4me | do you think having the logs on the host would be a positive operational impact? | 04:39 |
| odyssey4me | one side-effect is that log storage is outside of the container disk, so it doesn't use container disk allocations | 04:40 |
| *** thorst has joined #openstack-ansible | 04:41 | |
| logan- | Yeah it is a big change for sure but I'm struggling to think of a lot of common scenarios where it breaks things for people. I guess if you rely on collecting logs from the current dir structure | 04:41 |
| logan- | In /openstack | 04:41 |
| mcarden | Hmm, So I bring up a fresh Trusty VM, update it, reboot it and try to bring up an AIO from HEAD. lxc_container_create is consistemtly failing on a different container each time for 'Wait for ssh to be availabie' aka can't reach the container. Happens every time for me on Trusty at the moment, but I have yet to see it on Xenial. | 04:41 |
| logan- | Ok time for sleep | 04:42 |
| odyssey4me | mcarden if it happens, check whether the container has actually started - then check whether all interfaces it should have are up - look specifically for the br-mgmt interface because that's the one ansible tries to reach | 04:42 |
| odyssey4me | if you're getting it relatively easily, then perhaps you can get the log entries needed to isolate the cause | 04:43 |
| mcarden | I'm doing one now and I'll check. IIRC the container is there but not reachable. | 04:43 |
| odyssey4me | it *may* just be that we need a larger delay or something, but I've already increased it, and make the task retry a few times | 04:43 |
| odyssey4me | thanks mcarden | 04:43 |
| mcarden | NP | 04:43 |
| odyssey4me | night logan- | 04:44 |
| *** thorst has quit IRC | 04:47 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Implement an opportunistic strategy https://review.openstack.org/349485 | 04:51 |
| cloudnull | odyssey4me: regarding the binding. | 04:53 |
| *** zerda2 has quit IRC | 04:53 | |
| cloudnull | we could bind mount /var/log from the container to /openstack/CONTAINER/log or something similar. | 04:53 |
| cloudnull | that would give us more logs | 04:54 |
| odyssey4me | yeah, that's kinda what I was thinking | 04:54 |
| odyssey4me | although instead to /openstack/log/CONTAINER | 04:54 |
| cloudnull | that would also solve an issue that happens when containers are lvm backed and log fill up the container storage. | 04:54 |
| odyssey4me | because /openstack/CONTAINER is for precious data | 04:54 |
| cloudnull | but upgrades would be... interesting. | 04:55 |
| odyssey4me | they always are ;) | 04:55 |
| odyssey4me | I suppose I can put in a patch and see how it goes. At the very least it'll help diagnose the issue. | 04:56 |
| odyssey4me | by the way, the POST_FAILURES are caused by the swift logs not being readable by jenkins | 04:56 |
| cloudnull | :( | 04:56 |
| odyssey4me | the perms are 640 | 04:57 |
| odyssey4me | looking into why that is to see if we can address it | 04:57 |
| odyssey4me | I thought we had addressed this ages ago though | 04:57 |
| cloudnull | i had thought the container specific loging bits were accounted for too. | 04:58 |
| cloudnull | we used to have a lot more logs available to us. | 04:58 |
| odyssey4me | heh, from a recent build it appears not | 04:58 |
| odyssey4me | but for as long as I've worked with AIO's we haven't had more logs | 04:58 |
| cloudnull | the other thing we could do to provide the gate more logs is to simply grab from the rsyslog_server | 04:59 |
| odyssey4me | the rsyslog server collects more logs than there are in /openstack/log/ | 04:59 |
| odyssey4me | but this is an issue to troubleshoot before rsyslog is setup | 04:59 |
| *** javeriak has joined #openstack-ansible | 04:59 | |
| *** afred312 has joined #openstack-ansible | 04:59 | |
| odyssey4me | we already grab from the rsyslog server, because it stores everything in /openstack/log/<rsyslog container> | 04:59 |
| cloudnull | like we used to have http://logs.openstack.org/75/336075/2/gate/gate-openstack-ansible-dsvm-commit/5491a40/logs/http://logs.openstack.org/75/336075/2/gate/gate-openstack-ansible-dsvm-commit/5491a40/logs/ | 05:00 |
| cloudnull | which was from kilo | 05:00 |
| odyssey4me | the issue here is that rsyslog isn't setup on the server or clients yet | 05:00 |
| cloudnull | ^ thats true | 05:00 |
| odyssey4me | yeah that's fine - but notice that we only have specific service logs: http://logs.openstack.org/75/336075/2/gate/gate-openstack-ansible-dsvm-commit/5491a40/logs/aio1_ceilometer_api_container-743d4f6c/ | 05:00 |
| odyssey4me | in this case I need dmesg and syslog at least | 05:01 |
| cloudnull | this is true too. but its more than we have currently. | 05:01 |
| odyssey4me | the rsyslog server has those, but that's not yet up at the time of the problem | 05:01 |
| odyssey4me | cloudnull I think you're thinking there's more because there are more containers in the kilo AIO | 05:01 |
| odyssey4me | we cut the container count down due to build timeouts | 05:02 |
| odyssey4me | we don't do the clusters any more | 05:02 |
| *** janki has joined #openstack-ansible | 05:02 | |
| odyssey4me | if we can get the build time down to pass frequently on OVH, then we can get that back in place again | 05:03 |
| cloudnull | no, in master there are no logs from the specific containers | 05:03 |
| cloudnull | http://logs.openstack.org/05/350305/2/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-trusty/e7efd28/logs/ | 05:03 |
| odyssey4me | that's a build fail though | 05:03 |
| odyssey4me | but yes, in master I have noticed some missing things | 05:03 |
| odyssey4me | working on rectifying that | 05:04 |
| *** afred312 has quit IRC | 05:04 | |
| *** javeriak has quit IRC | 05:04 | |
| cloudnull | thats a success http://logs.openstack.org/68/349368/2/check/gate-openstack-ansible-dsvm-commit/5d312e5/logs/ | 05:04 |
| cloudnull | still no container specific logs | 05:05 |
| odyssey4me | wow, yeah I see that | 05:05 |
| odyssey4me | likely due to empty folders | 05:05 |
| cloudnull | like compaired to http://logs.openstack.org/75/336075/2/gate/gate-openstack-ansible-dsvm-commit/5491a40/logs/ | 05:05 |
| *** zerda2 has joined #openstack-ansible | 05:06 | |
| odyssey4me | I'm still reworking when those folders get created and all that - when that's done we can revisit it. | 05:06 |
| odyssey4me | I think it'll help. | 05:06 |
| *** janki has quit IRC | 05:06 | |
| cloudnull | ok | 05:06 |
| cloudnull | as for ssh container issues and things https://review.openstack.org/#/c/349485/ | 05:06 |
| cloudnull | ^ that should solve that. | 05:07 |
| cloudnull | as in there'd never be an ssh command to a container ever again. | 05:07 |
| odyssey4me | haha :) looking forward to seeing the results | 05:07 |
| odyssey4me | although the issue with the ssh is actually that the br-mgmt interface doesn't come up | 05:07 |
| cloudnull | I think i've fixed the issue I saw. which was due to task delegation , | 05:08 |
| odyssey4me | fyi https://review.openstack.org/347400 could do with a review | 05:08 |
| odyssey4me | I see that the logs aren't present there either - so I'll dig into why. | 05:08 |
| odyssey4me | so with that module, the container is still accessible via SSH, but we just don't bother using it? | 05:09 |
| odyssey4me | sorry - the conneciton plugin and strategy change | 05:09 |
| cloudnull | my one worry with that patch is due to hosts being in multiple groups causing a precedence issue when setting a variable with the same name in it. | 05:09 |
| cloudnull | groups are loaded alphabetically and the last one will always win. | 05:10 |
| odyssey4me | cloudnull can you explain what you mean with a specific example? | 05:10 |
| cloudnull | setting a group_var that isn't namespaced will clobber another groupvar | 05:11 |
| odyssey4me | note that I'm only using leaf groups | 05:11 |
| cloudnull | if a node exists in more than one group | 05:11 |
| odyssey4me | hmm, ok - but I don't see that happening here at all? | 05:12 |
| cloudnull | its not, but we're providing an interface where it will happen | 05:12 |
| odyssey4me | it is a general design issue, but we are able to manage that design issue | 05:12 |
| odyssey4me | effectively it is important to know that the more specific group_var will apply when using this interface | 05:12 |
| odyssey4me | for example group_vars/all.yml has the default lxc-openstack aa profile... but the containers that need a different profile have it set in their specific groups | 05:13 |
| odyssey4me | it's intuitive in an ansible manner | 05:13 |
| cloudnull | but i can see this happening, | 05:14 |
| *** janki has joined #openstack-ansible | 05:14 | |
| cloudnull | cinder is using a custom set of lxc container configuration vars that are VERY similar to neutron as found here https://review.openstack.org/#/c/347400/12/playbooks/haproxy-install.yml | 05:15 |
| cloudnull | one day someone is going to see that those vars arn't in a group var and move it. | 05:15 |
| odyssey4me | I'm confused - you're pointing at the haproxy file? | 05:16 |
| cloudnull | no.. sorry was cycling, with keybindings. | 05:16 |
| odyssey4me | and by the way - this patch isn't the final... I'm working towards eliminating the common lxc container config task | 05:16 |
| cloudnull | https://review.openstack.org/#/c/347400/12/playbooks/os-cinder-install.yml | 05:16 |
| *** markvoelker has joined #openstack-ansible | 05:16 | |
| odyssey4me | I'll follow on with another one to take care of the conditionals | 05:16 |
| odyssey4me | I didn't want to change too much at once. | 05:16 |
| odyssey4me | ah yes - that's planned to take care of in the next cycle | 05:17 |
| odyssey4me | not cycle -next patch | 05:17 |
| odyssey4me | I want to figure out how best to address that in the group vars too. | 05:17 |
| cloudnull | more importantly things like nova and neutron exist in the same groups. | 05:18 |
| cloudnull | so having a group_var that has a shared key with different values is going to cause a problem | 05:18 |
| cloudnull | especially if you run nova from a container. | 05:18 |
| cloudnull | which would be perfectly possible with things like the nova-lxd driver. | 05:18 |
| cloudnull | so its not a deal breaker. | 05:19 |
| cloudnull | just something i know can and will cause an issue | 05:19 |
| odyssey4me | so yes, but that can be addressed when that bridge comes up | 05:19 |
| cloudnull | and something that was a massive headache in Juno | 05:19 |
| odyssey4me | right now the patch is trying to address a problem with what we have, rather than what *might* be | 05:19 |
| odyssey4me | lol, I think mcarden picked up a copy-paste fail in https://review.openstack.org/#/c/349485 | 05:21 |
| mcarden | ...but was carfeul not to assert as much . :) | 05:21 |
| mcarden | *careful | 05:21 |
| *** markvoelker has quit IRC | 05:22 | |
| cloudnull | odyssey4me: agreeded. however I think we need to look at where things are going and make sure whatever it is we're implementing can provide for that eventuallity. | 05:22 |
| cloudnull | If we can fense it properly i'm all good with the change. | 05:23 |
| odyssey4me | cloudnull of course, if you have suggestions as to how we can do then then I'm all ears | 05:23 |
| odyssey4me | otherwise I'd prefer that we go with what we have and iterate | 05:23 |
| cloudnull | i dont, ansible group_var precedence is aweful. | 05:23 |
| odyssey4me | then we're just going to have to be careful to watch out for patches which cross the lines | 05:24 |
| odyssey4me | and if anything sneaks through fix it quickly | 05:24 |
| odyssey4me | ideally we should test for things which would cause problems somehow | 05:25 |
| cloudnull | thats another issue, line crossing is easy, just change the env.d bits and all lines could be converged and crossed. | 05:25 |
| cloudnull | and we are seeing folks do that. | 05:25 |
| cloudnull | i personally perfer namespacing everything | 05:26 |
| cloudnull | but maybe there's a way to inherit vars from other groups and merge/union . | 05:26 |
| cloudnull | if set that is | 05:27 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_swift: Allow swift log file permissions to be set https://review.openstack.org/350988 | 05:37 |
| odyssey4me | yeah, there is that option to merge vars... but that sounds like hell on earth | 05:38 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Implement an opportunistic strategy https://review.openstack.org/349485 | 05:38 |
| cloudnull | mcarden: good catch. i'm not sure how i douple pasted. but... | 05:39 |
| cloudnull | odyssey4me: yea merge strategy would be a bad time. | 05:39 |
| mcarden | I blame key debounce! | 05:39 |
| cloudnull | i'm thinking more along the lines of trying to check if a var is already defined and merge/union it to the name spaced version of it. | 05:41 |
| odyssey4me | cloudnull so you're thinking that the group_var has a namespaced var, then there's some sort of mapping? | 05:41 |
| cloudnull | idk if it'd work but something like ``lxc_container_bind_mounts: "{{ lxc_container_bind_mounts | default([])) | union(_namespaced_lxc_container_bind_mounts) }}"`` | 05:42 |
| cloudnull | maybe a node within more than one group would pickup the var lxc_container_bind_mounts as already being defined and then we load the namespaced version on top | 05:43 |
| cloudnull | idk | 05:43 |
| * cloudnull spit balling | 05:44 | |
| cloudnull | ok, im off to sleep | 05:44 |
| *** thorst has joined #openstack-ansible | 05:44 | |
| cloudnull | cheers alll | 05:44 |
| odyssey4me | night cloudnull | 05:46 |
| *** thorst has quit IRC | 05:52 | |
| *** afred312 has joined #openstack-ansible | 06:00 | |
| *** javeriak has joined #openstack-ansible | 06:02 | |
| *** afred312 has quit IRC | 06:06 | |
| *** asettle has joined #openstack-ansible | 06:09 | |
| *** pcaruana has joined #openstack-ansible | 06:10 | |
| *** asettle has quit IRC | 06:14 | |
| *** mfisch has quit IRC | 06:29 | |
| *** mfisch has joined #openstack-ansible | 06:34 | |
| *** mfisch has quit IRC | 06:34 | |
| *** mfisch has joined #openstack-ansible | 06:34 | |
| odyssey4me | afk while relocating | 06:39 |
| *** javeriak has quit IRC | 06:39 | |
| *** thorst has joined #openstack-ansible | 06:51 | |
| *** bootsha has joined #openstack-ansible | 06:52 | |
| *** permalac has quit IRC | 06:54 | |
| *** thorst has quit IRC | 06:57 | |
| *** afred312 has joined #openstack-ansible | 07:02 | |
| *** javeriak has joined #openstack-ansible | 07:05 | |
| *** afred312 has quit IRC | 07:08 | |
| *** mikelk has joined #openstack-ansible | 07:10 | |
| *** mikelk has quit IRC | 07:14 | |
| *** aszeszo has joined #openstack-ansible | 07:18 | |
| *** berendt has quit IRC | 07:30 | |
| *** openstackgerrit has quit IRC | 07:33 | |
| *** openstackgerrit has joined #openstack-ansible | 07:34 | |
| *** javeriak has quit IRC | 07:38 | |
| *** admin0 has joined #openstack-ansible | 07:43 | |
| *** DrifterZA has joined #openstack-ansible | 07:49 | |
| *** thorst has joined #openstack-ansible | 07:54 | |
| *** furlongm_ has quit IRC | 07:54 | |
| *** admin0 has quit IRC | 07:57 | |
| *** markvoelker has joined #openstack-ansible | 07:58 | |
| *** thorst has quit IRC | 08:02 | |
| *** afred312 has joined #openstack-ansible | 08:03 | |
| *** markvoelker has quit IRC | 08:03 | |
| *** afred312 has quit IRC | 08:08 | |
| *** bootsha has quit IRC | 08:19 | |
| *** admin0 has joined #openstack-ansible | 08:30 | |
| *** javeriak has joined #openstack-ansible | 08:30 | |
| *** Andrew_jedi has joined #openstack-ansible | 08:33 | |
| Andrew_jedi | Hello | 08:34 |
| Andrew_jedi | haproxy is screwing up the mitaka deployment for me. | 08:35 |
| *** asettle has joined #openstack-ansible | 08:40 | |
| *** javeriak has quit IRC | 08:41 | |
| *** bootsha has joined #openstack-ansible | 08:46 | |
| *** markvoelker has joined #openstack-ansible | 08:53 | |
| *** markvoelker has quit IRC | 08:57 | |
| *** thorst has joined #openstack-ansible | 09:00 | |
| *** jiteka has joined #openstack-ansible | 09:00 | |
| *** afred312 has joined #openstack-ansible | 09:04 | |
| odyssey4me | o/ | 09:04 |
| odyssey4me | andymccr please review https://review.openstack.org/350988 to resolve POST_FAILURE results in the integrated build | 09:04 |
| *** thorst has quit IRC | 09:06 | |
| *** afred312 has quit IRC | 09:08 | |
| odyssey4me | if you have a bit of additional time, reviews of the following would also be appreciated | 09:09 |
| odyssey4me | https://review.openstack.org/350684 (lxc container create keyserver fallback) | 09:09 |
| odyssey4me | https://review.openstack.org/347400 (container bind mount options move to group_vars) | 09:09 |
| *** javeriak has joined #openstack-ansible | 09:09 | |
| odyssey4me | https://review.openstack.org/344912 (selective wheel building) | 09:09 |
| Andrew_jedi | Is there any change in Mitaka regarding haproxy | 09:16 |
| Andrew_jedi | Are we not supposed to deploy haproxy_hosts? | 09:16 |
| odyssey4me | Andrew_jedi probably many | 09:17 |
| Andrew_jedi | odyssey4me: hello :) | 09:17 |
| odyssey4me | Andrew_jedi it all depends what happened that changed things - did you upgrade? if so then from what version of OSA to what new version? | 09:17 |
| *** berendt has joined #openstack-ansible | 09:17 | |
| Andrew_jedi | odyssey4me: I am doing a fresh mitaka install. | 09:18 |
| odyssey4me | haproxy is still deployed to haproxy_all: https://github.com/openstack/openstack-ansible/blob/stable/mitaka/playbooks/haproxy-install.yml#L17 | 09:18 |
| odyssey4me | ok, then perhaps you should be clearer about what's broken - all you said was 'haproxy is screwing up the mitaka deployment for me' which is not a very actionable problem report | 09:19 |
| *** janki has quit IRC | 09:21 | |
| DrifterZA | Hi Jesse | 09:23 |
| *** xek_ has joined #openstack-ansible | 09:23 | |
| *** xek has quit IRC | 09:24 | |
| Andrew_jedi | odyssey4me: sorry, the "Create OpenStack-Ansible requirement wheels" task fails on the host where the haproxy address is active. | 09:26 |
| odyssey4me | Andrew_jedi do you perhaps have an IP address conflict or something? | 09:26 |
| odyssey4me | because that task only has ansible coming in, and goes out to the internet to fetch files it needs | 09:26 |
| Andrew_jedi | odyssey4me: This is my config, http://paste.openstack.org/show/547908/ | 09:26 |
| odyssey4me | Andrew_jedi please pastebin a verbose output from the task | 09:27 |
| odyssey4me | ie -vvv | 09:27 |
| Andrew_jedi | odyssey4me: Roger that. | 09:27 |
| odyssey4me | Andrew_jedi note that the affinity settings you have are unnecessary | 09:28 |
| odyssey4me | an affinity of 1 is the default | 09:28 |
| Andrew_jedi | odyssey4me: They are there so that i can increase galera and rabbitmq affinity in case of a controller node failure. | 09:28 |
| odyssey4me | ah ok | 09:29 |
| odyssey4me | also it seems that your hosts are not given addresses on the container/mgmt network? | 09:29 |
| odyssey4me | the CIDR for the mgmt network is 172.29.236.0/22 but your hosts are on 192.168.0.0 ? | 09:30 |
| *** janki has joined #openstack-ansible | 09:30 | |
| Andrew_jedi | odyssey4me: The systems are remote so i need 192.168.0.0 for routing. I have specified 172.29.236.0/22 in a seperate openstack.cfg file. | 09:31 |
| odyssey4me | ok, I'm just noting that it's unusual | 09:31 |
| Andrew_jedi | odyssey4me: Yes it is. I have only two nics and i wanted to use vxlan. This is my /etc/netwok/interfaces.d/openstack.cfg file, http://paste.openstack.org/show/547911/ | 09:32 |
| odyssey4me | Andrew_jedi Unfortunately I'm quite terrible at networking and in the middle of doing some other work, so I can't really advise you properly. | 09:34 |
| odyssey4me | The repo build should not fail though, and has nothing to do with haproxy other than haproxy doing the front end load balancing for access to the repo. | 09:34 |
| odyssey4me | we build mitaka often without trouble | 09:35 |
| *** electrofelix has joined #openstack-ansible | 09:35 | |
| odyssey4me | if you're using the absolute latest tag, or the head of stable/mitaka then it is plausible that there may be an issue which I'm busy trying to get resolved... so you may wish to try using a tag back | 09:36 |
| Andrew_jedi | odyssey4me: one more info, i was using a local mirror yesterday and any server having the haproxy address was failing to get packages from the local mirror. | 09:36 |
| Andrew_jedi | odyssey4me: I am using the stable/mitaka branch | 09:36 |
| odyssey4me | I don't understand what haproxy has to do with anything. Do you mean the repo container on that same host was not able to fetch packages? | 09:37 |
| DrifterZA | odyssey4me: Dude just wanted to double check, when doing a patchset do you use git diff -U ? | 09:37 |
| odyssey4me | DrifterZA uh, no - what're you trying to achieve? | 09:37 |
| Andrew_jedi | odyssey4me: Yes indeed. | 09:37 |
| DrifterZA | odyssey4me: just want to make sure my diffs are correctly formatted. | 09:38 |
| odyssey4me | Andrew_jedi then that translates more than likely to a network config issue - get into the container and do the basic network troubleshooting | 09:38 |
| odyssey4me | DrifterZA you use 'git review' to upload patches | 09:38 |
| Andrew_jedi | odyssey4me: hmmm, ok let me double check the network setup. | 09:38 |
| odyssey4me | clone the repo, change into it, execute 'git review -s' to setup the gerrit remote, checkout the branch to work in, make your changes, commit the patch, then 'git review' to upload it | 09:39 |
| odyssey4me | DrifterZA you may wish to take a look through http://docs.openstack.org/infra/manual/developers.html | 09:40 |
| DrifterZA | Ah okay, i've done the git review setup already. | 09:41 |
| DrifterZA | so git review does the patchset for you | 09:41 |
| *** aszeszo has quit IRC | 09:41 | |
| DrifterZA | thats cool | 09:41 |
| odyssey4me | yep | 09:41 |
| odyssey4me | each commit will be an individual review | 09:41 |
| odyssey4me | so typically you would squash commits that relate to solving a single problem or achieving a single goal | 09:42 |
| odyssey4me | (before you put them up for review) | 09:42 |
| odyssey4me | also note that all patches must go into master first, and merge, before patches will be considered for stable branches | 09:42 |
| DrifterZA | got it thanks | 09:42 |
| odyssey4me | that's very necessary to ensure that we don't regress | 09:42 |
| openstackgerrit | Donovan Francesco proposed openstack/openstack-ansible: Adding Magnum-UI Horizon support https://review.openstack.org/351089 | 09:45 |
| Andrew_jedi | odyssey4me: Which tag do you recommend? 13.3.0 ? btw this task -vvv output http://paste.openstack.org/show/547916/ | 09:47 |
| *** markvoelker has joined #openstack-ansible | 09:47 | |
| openstackgerrit | Donovan Francesco proposed openstack/openstack-ansible-os_horizon: Adding Magnum-UI support to horizon https://review.openstack.org/351091 | 09:48 |
| *** permalac has joined #openstack-ansible | 09:49 | |
| DrifterZA | odyssey4me: is that okay ? | 09:50 |
| *** markvoelker has quit IRC | 09:52 | |
| *** aszeszo has joined #openstack-ansible | 09:53 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift: Allow swift log file permissions to be set https://review.openstack.org/350988 | 09:59 |
| *** kamsz_ has joined #openstack-ansible | 10:03 | |
| kamsz_ | hi there, is there any way to diagnose why vip is not working? | 10:03 |
| *** kamsz_ is now known as kamsz | 10:03 | |
| *** kamsz has quit IRC | 10:03 | |
| *** kamsz has joined #openstack-ansible | 10:03 | |
| kamsz | keepalived is up and running, but vip is not pingable | 10:03 |
| *** afred312 has joined #openstack-ansible | 10:04 | |
| *** afred312 has quit IRC | 10:09 | |
| Andrew_jedi | kamsz: Can you ping any other ip from the same subnet as of your vip ? | 10:10 |
| *** furlongm_ has joined #openstack-ansible | 10:12 | |
| *** jimbaker has quit IRC | 10:30 | |
| *** thorst has joined #openstack-ansible | 10:31 | |
| *** jimbaker has joined #openstack-ansible | 10:34 | |
| *** jimbaker has quit IRC | 10:34 | |
| *** jimbaker has joined #openstack-ansible | 10:34 | |
| *** thorst has quit IRC | 10:36 | |
| *** markvoelker has joined #openstack-ansible | 10:41 | |
| *** markvoelker has quit IRC | 10:46 | |
| *** janki has quit IRC | 10:46 | |
| *** wadeholler has joined #openstack-ansible | 10:47 | |
| *** janki has joined #openstack-ansible | 10:50 | |
| odyssey4me | DrifterZA The build failure is unrelated to the patch. Unfortunately no-one has looked into the cause of it yet. If you can do a local build and debug the issue it would be wonderful. | 10:52 |
| *** smatzek has joined #openstack-ansible | 10:54 | |
| DrifterZA | k will do | 10:55 |
| *** janki has quit IRC | 10:56 | |
| *** bootsha has quit IRC | 11:01 | |
| odyssey4me | DrifterZA FYI when iterating on a review, make sure you 'git commit -a --amend' to merge your changes into the same commit | 11:02 |
| odyssey4me | when you push that up for review, gerrit will mark it as a second patch set | 11:02 |
| DrifterZA | cool will do. | 11:02 |
| DrifterZA | The comment you made, I thought it looked the same as the git commit you referenced. Did I miss something ? | 11:02 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_build: Only clone git repositories for services which are deployed https://review.openstack.org/345642 | 11:02 |
| odyssey4me | DrifterZA the commit msg is missing some stuff, and you're missing a release note | 11:03 |
| odyssey4me | http://docs.openstack.org/developer/reno/usage.html | 11:03 |
| *** thorst has joined #openstack-ansible | 11:03 | |
| odyssey4me | http://docs.openstack.org/developer/openstack-ansible/developer-docs/contribute.html#release-notes | 11:03 |
| odyssey4me | release notes are important to announce new features :) | 11:04 |
| DrifterZA | Aah, I didnt add the release note cos I didnt have a git commit sha | 11:04 |
| DrifterZA | LoL, I knew my first time would be issues. | 11:04 |
| odyssey4me | DrifterZA it's not the commit SHA, it's just a random thing that the reno tool adds to the clug | 11:04 |
| odyssey4me | *slug | 11:04 |
| odyssey4me | you can see here how release notes are published: http://docs.openstack.org/releasenotes/openstack-ansible/mitaka.html | 11:05 |
| DrifterZA | jeeze so much to learn | 11:05 |
| DrifterZA | lol | 11:05 |
| *** afred312 has joined #openstack-ansible | 11:05 | |
| DrifterZA | I'll eventually get the hang of this | 11:05 |
| odyssey4me | yeah, it takes a little orientation - but once you have it, it's done | 11:06 |
| *** afred312 has quit IRC | 11:10 | |
| *** sguduru_ has joined #openstack-ansible | 11:11 | |
| *** sguduru__ has joined #openstack-ansible | 11:13 | |
| *** sguduru has quit IRC | 11:13 | |
| *** janki has joined #openstack-ansible | 11:13 | |
| *** bootsha has joined #openstack-ansible | 11:14 | |
| *** sguduru_ has quit IRC | 11:16 | |
| *** zerda2 has quit IRC | 11:20 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [TESTING] Try to determine the cause of container secondary interface start failures https://review.openstack.org/351128 | 11:27 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [TESTING] Try to determine the cause of container secondary interface start failures https://review.openstack.org/351128 | 11:29 |
| *** janki has quit IRC | 11:35 | |
| *** markvoelker has joined #openstack-ansible | 11:35 | |
| *** markvoelker has quit IRC | 11:39 | |
| *** kamsz_ has joined #openstack-ansible | 11:40 | |
| *** xek__ has joined #openstack-ansible | 11:41 | |
| *** johnmilton has joined #openstack-ansible | 11:42 | |
| *** retreved has joined #openstack-ansible | 11:42 | |
| *** kamsz has quit IRC | 11:42 | |
| *** xek_ has quit IRC | 11:42 | |
| *** janki has joined #openstack-ansible | 11:47 | |
| *** johnmilton has quit IRC | 11:50 | |
| *** sguduru__ has quit IRC | 11:50 | |
| *** psilvad has joined #openstack-ansible | 11:54 | |
| *** weshay has joined #openstack-ansible | 11:56 | |
| *** asettle has quit IRC | 11:57 | |
| *** asettle has joined #openstack-ansible | 11:57 | |
| *** janki has quit IRC | 12:02 | |
| *** janki has joined #openstack-ansible | 12:02 | |
| *** asettle has quit IRC | 12:02 | |
| *** johnmilton has joined #openstack-ansible | 12:03 | |
| *** markvoelker has joined #openstack-ansible | 12:05 | |
| *** afred312 has joined #openstack-ansible | 12:06 | |
| *** janki has quit IRC | 12:09 | |
| *** afred312 has quit IRC | 12:11 | |
| *** janki has joined #openstack-ansible | 12:24 | |
| Andrew_jedi | stable/mitaka failing at this task, TASK: [repo_build | Run venv process script] . Is it just me or anybody else facing the same issue? | 12:25 |
| *** asettle has joined #openstack-ansible | 12:26 | |
| odyssey4me | not me, nor the gate | 12:27 |
| Andrew_jedi | damn :/ | 12:28 |
| Andrew_jedi | odyssey4me: Thank you for confirming ! | 12:28 |
| odyssey4me | the venv build is mostly isolated on the repo server, so if it's failing it's likely failing due to something beforehand | 12:29 |
| odyssey4me | there is a log file for it - attach to the container and check /var/log/repo/ | 12:29 |
| *** jiteka has left #openstack-ansible | 12:33 | |
| Andrew_jedi | checking logs | 12:37 |
| *** woodard has joined #openstack-ansible | 12:38 | |
| *** woodard has quit IRC | 12:38 | |
| *** woodard has joined #openstack-ansible | 12:39 | |
| Andrew_jedi | odyssey4me: /var/log/repo looks fine but i got this from syslog on the repo container, http://paste.openstack.org/show/549220/ | 12:43 |
| *** janki has quit IRC | 12:48 | |
| *** psilvad has quit IRC | 12:50 | |
| odyssey4me | evrardjp cloudnull This appears to be where the countainer log bind mounts disappeared | 12:53 |
| odyssey4me | https://review.openstack.org/#/c/311796/10/tasks/container_create.yml@95 | 12:53 |
| *** javeriak has quit IRC | 12:54 | |
| evrardjp | l95->l83 is indeed missing | 12:54 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible: Add haproxy_service_enabled boolean https://review.openstack.org/351158 | 12:54 |
| evrardjp | 75 not 83 | 12:54 |
| *** psilvad has joined #openstack-ansible | 12:55 | |
| evrardjp | good that it's easy to fix | 12:55 |
| mhayden | buenos dias | 12:56 |
| *** javeriak has joined #openstack-ansible | 12:57 | |
| *** xnodeza has quit IRC | 12:59 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: [TESTING] Performance testing upstream Ansible improvements https://review.openstack.org/351165 | 13:00 |
| *** psilvad has quit IRC | 13:00 | |
| *** abelur has quit IRC | 13:01 | |
| cloudnull | mornings | 13:02 |
| evrardjp | morning cloudnull | 13:02 |
| cloudnull | hows it evrardjp? | 13:03 |
| evrardjp | fine, what about you? | 13:03 |
| *** psilvad has joined #openstack-ansible | 13:04 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_container_create: Fix default container log bind mount https://review.openstack.org/351169 | 13:05 |
| *** afred312 has joined #openstack-ansible | 13:07 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts: Implement primary and secondary keyserver usage for cache prep https://review.openstack.org/350684 | 13:08 |
| *** messy has joined #openstack-ansible | 13:11 | |
| *** sdake has joined #openstack-ansible | 13:11 | |
| *** klamath has joined #openstack-ansible | 13:11 | |
| *** klamath has quit IRC | 13:11 | |
| *** afred312 has quit IRC | 13:11 | |
| *** klamath has joined #openstack-ansible | 13:12 | |
| cloudnull | odyssey4me: regarding these binds. I think we can do something like this https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-glance-install.yml#L25-L29 to effect all of the containers we want bind mount logs from in a far more controlled way. | 13:13 |
| *** markvoelker has quit IRC | 13:13 | |
| odyssey4me | cloudnull yeah, lemme revise that patch | 13:13 |
| odyssey4me | I meant it to be a band-aid for the moment. | 13:14 |
| cloudnull | hell, we could simply bind mount all of /var/log like we were talking about yesterday . | 13:14 |
| *** jamesdenton has joined #openstack-ansible | 13:14 | |
| odyssey4me | although I was thinking of doing something with the bind mounts similar to what I did with container_config | 13:14 |
| odyssey4me | ie have a default: https://github.com/openstack/openstack-ansible-lxc_container_create/blob/master/defaults/main.yml#L23 | 13:15 |
| odyssey4me | then merge it with a user-provided addition | 13:15 |
| *** janki has joined #openstack-ansible | 13:15 | |
| odyssey4me | like so https://github.com/openstack/openstack-ansible-lxc_container_create/blob/master/tasks/container_create.yml#L121 | 13:15 |
| odyssey4me | I'm also looking to pull back a bunch of stuff into the cache creation - we do a lot of the same things in every container, and if it can go into the cache prep it'll save that task execution. | 13:16 |
| cloudnull | that'd work i think . the tough part there is that dirs need to exist on both sides before the config or the container will fail to start. | 13:17 |
| odyssey4me | and yeah, maybe just bind-mounting /var/log to the container dir is better... that said I think the gate will puke due to perms for files that the jenkins user can't read | 13:17 |
| odyssey4me | cloudnull yep - that sequence is already in place: https://github.com/openstack/openstack-ansible-lxc_container_create/blob/master/tasks/container_create.yml#L128-L153 | 13:18 |
| *** janki has quit IRC | 13:18 | |
| odyssey4me | I just need to implement the union with a default set | 13:18 |
| odyssey4me | I'll work up a patch now. | 13:18 |
| odyssey4me | the backup bind mount can also join it | 13:18 |
| cloudnull | odyssey4me: the gate should be ok with the logs | 13:18 |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/master/scripts/scripts-library.sh#L135-L137 | 13:18 |
| cloudnull | we chown them 777 as the gate exits. | 13:19 |
| *** michaelgugino has joined #openstack-ansible | 13:19 | |
| odyssey4me | ha! I thought we had a workaround in place for that | 13:19 |
| odyssey4me | well the POST_FAILUREs were due to the inability to access the swift logs... which is why I worked https://review.openstack.org/350988 up this morning | 13:19 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-rsyslog_client: Prevent syslog leakage https://review.openstack.org/351182 | 13:22 |
| *** timl0101 has quit IRC | 13:22 | |
| *** sdake has quit IRC | 13:24 | |
| Andrew_jedi | I am seeing these notifications, "Could not find a version that satisfies the requirement oslo.middleware>=3.11.0 (from gnocchi[ceph,file,mysql,swift]) (from versions: 3.8.0)No matching distribution found for oslo.middleware>=3.11.0 (from gnocchi[ceph,file,mysql,swift])". Should i lower the version of oslo.middleware? | 13:25 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible: Prevent syslog leakage https://review.openstack.org/351184 | 13:25 |
| odyssey4me | Andrew_jedi which tag/branch are you running? | 13:25 |
| Andrew_jedi | odyssey4me: 13.0.0 | 13:26 |
| odyssey4me | That may be caused by the issue I mentioned earlier, which is why I suggested using the tag befoe the latest | 13:26 |
| odyssey4me | 13.0.0? heh, that is a terrible idea | 13:26 |
| odyssey4me | that was pre-release mitaka | 13:26 |
| Andrew_jedi | odyssey4me: sorry, 13.3.0 | 13:26 |
| odyssey4me | that's the latest tag - please try 13.2.0 instead | 13:27 |
| Andrew_jedi | odyssey4me: roger that | 13:27 |
| odyssey4me | we have a known issue which isn't quite ironed out yet | 13:27 |
| odyssey4me | Andrew_jedi related: https://review.openstack.org/#/q/topic:bug/1605846 | 13:27 |
| *** galstrom_zzz is now known as galstrom | 13:28 | |
| Andrew_jedi | Ahhh, ok !! | 13:29 |
| mhayden | messy: ah, sorry if i stepped on your toes with the backport :P | 13:34 |
| *** Jeffrey4l_ has quit IRC | 13:34 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Implemenet container bind mount for all logs https://review.openstack.org/351188 | 13:36 |
| messy | mhayden: its fine haha | 13:36 |
| odyssey4me | cloudnull I'm doing something similar, but better | 13:37 |
| cloudnull | odyssey4me: ok, that's using the current API. | 13:37 |
| odyssey4me | cloudnull ok, fair enough | 13:38 |
| *** timl0101 has joined #openstack-ansible | 13:38 | |
| cloudnull | its repetitive but but very controlled. | 13:38 |
| odyssey4me | yeah, I'm implementing a facility to do that at container creation instead - but it will take a patch to the lxc-container-create repo first, then the integrated repo | 13:39 |
| odyssey4me | so yeah - incoming | 13:39 |
| cloudnull | ok | 13:40 |
| openstackgerrit | Donovan Francesco proposed openstack/openstack-ansible-os_horizon: Adding Magnum-UI support to horizon https://review.openstack.org/351091 | 13:41 |
| openstackgerrit | Merged openstack/openstack-ansible-apt_package_pinning: Adding idempotency testing to the role func test https://review.openstack.org/350870 | 13:43 |
| openstackgerrit | Merged openstack/openstack-ansible-repo_build: Only build wheels for services which are deployed https://review.openstack.org/344912 | 13:44 |
| michaelgugino | I cut some groovy patches for os_tempest, I would appreciate some reviews, they are needed by the nova-lxd tests. | 13:44 |
| cloudnull | ++ | 13:45 |
| *** ametts has joined #openstack-ansible | 13:45 | |
| *** pjm6 has quit IRC | 13:47 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_container_create: Ensure that the default bind mounts are complete https://review.openstack.org/351169 | 13:47 |
| *** javeriak has quit IRC | 13:49 | |
| *** markvoelker has joined #openstack-ansible | 13:51 | |
| *** javeriak has joined #openstack-ansible | 13:52 | |
| openstackgerrit | Donovan Francesco proposed openstack/openstack-ansible: Adding Magnum-UI Horizon support https://review.openstack.org/351089 | 13:52 |
| *** v1k0d3n has joined #openstack-ansible | 13:52 | |
| DrifterZA | odyssey4me | 13:53 |
| *** sdake has joined #openstack-ansible | 13:54 | |
| DrifterZA | Just a small question. What do I do to link my changes to the bp ? | 13:54 |
| odyssey4me | DrifterZA see the comment I added to the commit message in the review | 13:55 |
| odyssey4me | it's a combination of a commit message entry, and the 'topic' (which is the branch name)... I changed the topic for you to the right thing | 13:55 |
| cloudnull | DrifterZA: have a look here http://docs.openstack.org/infra/manual/developers.html#committing-a-change | 13:55 |
| cloudnull | that covers more than just blueprints | 13:55 |
| cloudnull | but should give you an idea of what to do | 13:56 |
| logan- | fresh aio from master, anyone else seen this? http://cdn.pasteraw.com/4qqt8076hq9y1eunoo4b87n1dh6823j | 13:56 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest: Enable broader image support https://review.openstack.org/350781 | 13:56 |
| openstackgerrit | Donovan Francesco proposed openstack/openstack-ansible: Adding Magnum-UI Horizon support https://review.openstack.org/351089 | 13:56 |
| cloudnull | logan-: nope | 13:57 |
| openstackgerrit | Donovan Francesco proposed openstack/openstack-ansible-os_horizon: Adding Magnum-UI support to horizon https://review.openstack.org/351091 | 13:57 |
| logan- | ok thanks cloudnull. will dig into it some more | 13:58 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer: RabbitMQ config rework https://review.openstack.org/350839 | 13:58 |
| DrifterZA | odyssey4me: could you just check if I did it right now ? | 13:58 |
| cloudnull | logan-: you should have it at /lib/x86_64-linux-gnu/libdbus-1.so.3 | 13:59 |
| logan- | yeah it existed, looked fine, i could stat it from sudo -u neutron and stuff. going to rebuild aio and see if it reoccurs | 13:59 |
| cloudnull | also in the agents container you should have a bind mount for the modules "lxc.mount.entry = /lib/modules lib/modules none bind 0 0" | 14:00 |
| odyssey4me | DrifterZA looks good to me - one nit, but I'm happy with it as-is. | 14:01 |
| DrifterZA | Get your point, let me update it | 14:02 |
| *** aszeszo has quit IRC | 14:03 | |
| openstackgerrit | Donovan Francesco proposed openstack/openstack-ansible-os_horizon: Adding Magnum-UI support to horizon https://review.openstack.org/351091 | 14:04 |
| DrifterZA | I think its good now odyssey4me | 14:07 |
| DrifterZA | and I think i've got the process down now too. | 14:07 |
| *** afred312 has joined #openstack-ansible | 14:07 | |
| odyssey4me | DrifterZA thanks for being patient :) | 14:08 |
| odyssey4me | now all we need is to figure out why the role test is failing and fix that, so that your patch can merge :) | 14:08 |
| DrifterZA | Its all good man, want to get it right and keep getting it right | 14:08 |
| *** spotz_zzz is now known as spotz | 14:09 | |
| openstackgerrit | Logan V proposed openstack/openstack-ansible: Add haproxy_service_enabled boolean https://review.openstack.org/351158 | 14:09 |
| cloudnull | odyssey4me: regarding https://review.openstack.org/#/c/351188 | 14:11 |
| *** afred312 has quit IRC | 14:12 | |
| cloudnull | the path's are being set to "/openstack/{{ inventory_hostname }}/log" because "/openstack/{{ inventory_hostname }}" is already used all over the place. | 14:12 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible-os_neutron: Allow Neutron services to specify config path https://review.openstack.org/349210 | 14:12 |
| odyssey4me | cloudnull yeah, but the /openstack/log/{{ inventory_hostname }} is where logs are in mitaka and earlier | 14:13 |
| cloudnull | with a bind mount of /var/log to /openstack/{{ inventory_hostname }}/ we're going to see some oddness. | 14:13 |
| cloudnull | like the fact that /var/lib/mysql is mounted there. | 14:13 |
| odyssey4me | I do prefer actually moving towards something more like /openstack/{{ inventory_hostname }}/{log, lib/mysql, etc} | 14:13 |
| odyssey4me | cloudnull you're missing what I said | 14:13 |
| odyssey4me | there is a 'log' before the container dir | 14:14 |
| michaelgugino | I saw some permission errors related to logs running the test for os_horizon | 14:14 |
| michaelgugino | I think it might have something to do with the bind mounting, lots of funny stuff going on | 14:14 |
| odyssey4me | /openstack/{{ inventory_hostname }} is where data lives in the current binds, and /openstack/***log***/{{ inventory_hostname }} is where logs live | 14:14 |
| *** Mudpuppy has joined #openstack-ansible | 14:15 | |
| cloudnull | ah. I see what you're getting at, I can move it there. | 14:15 |
| odyssey4me | michaelgugino log entry please? none of the os_ roles are doing bind mounts to my knowledge | 14:15 |
| odyssey4me | cloud, instead of doing that though - you could just patch on top of https://review.openstack.org/347400 in an extra patch which adds the log bind mounts | 14:16 |
| odyssey4me | cloudnull ^ | 14:16 |
| michaelgugino | http://paste.openstack.org/show/549235/ | 14:16 |
| michaelgugino | that was during the run for os_horizon run_tests.sh | 14:17 |
| odyssey4me | michaelgugino hmm | 14:17 |
| michaelgugino | logan-: ^^ looks like I was seeing something similar to you | 14:17 |
| michaelgugino | that's on trusty | 14:17 |
| *** javeriak has quit IRC | 14:18 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Implemenet container bind mount for all logs https://review.openstack.org/351188 | 14:18 |
| logan- | interesting michaelgugino. i haven't run the horizon gate locally yet, was going to dig into that later today time permitting. the thing i found last night doesnt appear to affect the role gate, just the integrated build. | 14:19 |
| michaelgugino | yeah, I was running locally, but that role is also failing the gate | 14:19 |
| logan- | im adding renos to the horizon reviews i placed but they are sort of critical as anyone deploying horizon with debug=false will receive a broken deployment | 14:19 |
| logan- | mitaka and master are affected, didn't check liberty but i doubt it | 14:20 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible: Do not override Horizon ServerName in playbook https://review.openstack.org/350955 | 14:21 |
| cloudnull | odyssey4me: the issue with that is exactly what I was talking about last night. w/ regard to groups clobbering variables from other groups because they're not namespaced and the lists wont selectively merge. | 14:21 |
| odyssey4me | cloudnull what's the problem - just add a list entry for the bind mounts? | 14:21 |
| *** janki has joined #openstack-ansible | 14:22 | |
| cloudnull | we'd have to do it for all of the groups | 14:22 |
| odyssey4me | cloudnull yes, of course | 14:22 |
| odyssey4me | if they're all the same - then just one in all is fine | 14:22 |
| odyssey4me | but as I recall they are not all the same | 14:22 |
| odyssey4me | some groups have extras, in which case we have to ensure that the extras are added to the default | 14:23 |
| odyssey4me | but if they are all the same, then they should be in the container-create role... | 14:23 |
| odyssey4me | it looks to me like some of the roles had a special directory though | 14:23 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible: Do not override Horizon ServerName in playbook https://review.openstack.org/350967 | 14:23 |
| odyssey4me | cloudnull see https://github.com/openstack/openstack-ansible/blob/master/playbooks/inventory/env.d/galera.yml#L29 as an example | 14:24 |
| *** afred312 has joined #openstack-ansible | 14:24 | |
| cloudnull | odyssey4me: i dont think that used anywhere. | 14:26 |
| odyssey4me | only for galera :/ | 14:26 |
| odyssey4me | a special snowflake :p | 14:26 |
| odyssey4me | https://github.com/openstack/openstack-ansible/blob/liberty/playbooks/roles/lxc_container_create/tasks/container_create.yml#L104 | 14:26 |
| cloudnull | thats a legacy option. its not used anywhere. | 14:27 |
| cloudnull | i think since mitaka | 14:27 |
| odyssey4me | cloudnull I think you are mistaken. As far as I recall the galera logs are still in /var/log/mysql_logs | 14:27 |
| odyssey4me | it is perhaps time to change it though | 14:28 |
| cloudnull | it is that, but the properties aren't used | 14:28 |
| cloudnull | grep -rnil log_directory /etc/ansible/ /opt/openstack-ansible/playbooks/ | 14:28 |
| cloudnull | neither are "properties.service_name" | 14:29 |
| odyssey4me | cloudnull yes of course, this is the line that got left out when the multi-OS was done | 14:29 |
| cloudnull | that entire key can removed | 14:29 |
| cloudnull | I think the only one we still use is properties.is_metal | 14:30 |
| odyssey4me | yeah, and that's needed due to the inventory creation of lxc containers... for now at least | 14:30 |
| logan- | michaelgugino: looking at that log you're seeing some other issue than i was. the one i found is here: https://review.openstack.org/#/c/350955/. let me see if i can reproduce what you're seeing though.. are you doing a role test on horizon role itself or was that log from an integrated build? | 14:30 |
| cloudnull | odyssey4me: ++ | 14:31 |
| michaelgugino | horizon | 14:31 |
| odyssey4me | cloudnull if we could successfully use a remote lxc connection plugin then we could actually get rid of all that and let ansible fact gathering give us the IP's, etc | 14:31 |
| odyssey4me | ie LXC could just allocate IP's if we wanted to let it do so | 14:31 |
| odyssey4me | or some IPAM or whatever | 14:32 |
| odyssey4me | anyway I digress | 14:32 |
| odyssey4me | coffee time | 14:32 |
| *** catintheroof has joined #openstack-ansible | 14:32 | |
| cloudnull | https://review.openstack.org/#/c/349485/ the last check was success, but when return was done the gate timedout on ovh before Jenkins posted. | 14:34 |
| cloudnull | odyssey4me: ^ | 14:34 |
| *** javeriak has joined #openstack-ansible | 14:34 | |
| odyssey4me | cloudnull yeah, saw that | 14:35 |
| *** janki has quit IRC | 14:35 | |
| mhayden | odyssey4me: no love on bindep yet? | 14:36 |
| odyssey4me | mhayden they're still faffing: https://review.openstack.org/#/q/project:openstack-infra/bindep | 14:37 |
| mhayden | odyssey4me: ah, this was the one they wanted to get in prior to release, right? https://review.openstack.org/#/c/350592/ | 14:38 |
| *** asettle has quit IRC | 14:38 | |
| *** asettle has joined #openstack-ansible | 14:39 | |
| odyssey4me | mhayden I guess so - not sure | 14:41 |
| odyssey4me | mhayden perhaps ping fungi in #openstack-infra | 14:42 |
| *** Jeffrey4l has joined #openstack-ansible | 14:45 | |
| mhayden | can do | 14:46 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest: Add project group to role https://review.openstack.org/350974 | 14:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_gnocchi: Add project group to role https://review.openstack.org/350962 | 14:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron: Add project group to role https://review.openstack.org/350969 | 14:49 |
| *** berendt has quit IRC | 14:50 | |
| openstackgerrit | Andy McCrae proposed openstack/openstack-ansible-os_swift: Refactor and improve swift-rings.py https://review.openstack.org/351227 | 14:51 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally: Add project group to role https://review.openstack.org/350971 | 14:51 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cloudkitty: Add project group to role https://review.openstack.org/350958 | 14:51 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican: Add project group to role https://review.openstack.org/350952 | 14:51 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat: Add project group to role https://review.openstack.org/350963 | 14:52 |
| *** aszeszo has joined #openstack-ansible | 14:52 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance: Add project group to role https://review.openstack.org/350960 | 14:53 |
| mhayden | so bindep 2.0.1 is the one we want, and it's in pypi | 14:53 |
| openstackgerrit | Merged openstack/openstack-ansible-os_zaqar: Add project group to role https://review.openstack.org/350975 | 14:53 |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh: Add project group to role https://review.openstack.org/350948 | 14:53 |
| *** javeriak has quit IRC | 14:53 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer: Add project group to role https://review.openstack.org/350956 | 14:54 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic: Add project group to role https://review.openstack.org/350965 | 14:54 |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara: Add project group to role https://review.openstack.org/350972 | 14:55 |
| odyssey4me | mhayden so hopefully we should be able to recheck the review (after fixing it again) | 14:55 |
| *** Mudpuppy has quit IRC | 14:56 | |
| odyssey4me | although it's plausible that we may need to wait for the nodepool images to be refreshed | 14:56 |
| mhayden | odyssey4me: ah, you might be right on the image stuff :| | 14:56 |
| mhayden | i forgot that fungi mentioned that | 14:56 |
| mhayden | well at least the recheck will fail super quick if it isn't updated :P | 14:57 |
| mhayden | fungi said -> "if our image uploads succeed today, we should have 2.0.1 in the bindep virtualenv on them" | 14:57 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-pip_install: Add SNI support via OS packages https://review.openstack.org/348887 | 14:57 |
| odyssey4me | mhayden yeah, thought as much | 14:57 |
| odyssey4me | ok, so we wait some more | 14:57 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-pip_install: Add SNI support via OS packages https://review.openstack.org/348887 | 14:58 |
| odyssey4me | ^ restored to the former state | 14:58 |
| * mhayden tips his hat to odyssey4me | 15:00 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate: Add project group to role https://review.openstack.org/350959 | 15:00 |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift: Add project group to role https://review.openstack.org/350973 | 15:00 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [DOCS] - Cleanup Telemetry docs https://review.openstack.org/349693 | 15:03 |
| *** Mudpuppy has joined #openstack-ansible | 15:03 | |
| itlinux | hello all, once I reboot my mysql does not restart any suggestions on that.. thanks | 15:08 |
| spotz | itlinux what does it say in /var/log/syslog? | 15:08 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-pip_install: Add SNI support via OS packages https://review.openstack.org/348887 | 15:09 |
| itlinux | let me check spotz: I just rebooted | 15:09 |
| openstackgerrit | Merged openstack/openstack-ansible-apt_package_pinning: Adding Vagrantfile for developer testing https://review.openstack.org/350872 | 15:09 |
| spotz | itlinux: also make sure you're enabled to start on boot up | 15:10 |
| itlinux | yes they are coming up! | 15:10 |
| itlinux | but mysql is having some issues.. | 15:10 |
| odyssey4me | itlinux if you just rebooted all the cluster members - then you'll need to rebootstrap the cluster | 15:11 |
| *** daneyon_ has joined #openstack-ansible | 15:11 | |
| *** daneyon has joined #openstack-ansible | 15:12 | |
| *** Mudpuppy has quit IRC | 15:13 | |
| itlinux | WSREP: wsrep::connect(gcomm://172.29.238.141) failed: 7 | 15:15 |
| *** daneyon_ has quit IRC | 15:16 | |
| Andrew_jedi | odyssey4me: 13.2.0 is also failing to build the repo container :( | 15:19 |
| itlinux | I just checked back the website and there is a note on galera openstack-ansible -e galera_ignore_cluster_state=true galera-install.yml | 15:24 |
| itlinux | will try that. Thanks | 15:24 |
| *** catintheroof has quit IRC | 15:25 | |
| *** Mudpuppy has joined #openstack-ansible | 15:26 | |
| *** timl0101 has quit IRC | 15:29 | |
| *** timl0101 has joined #openstack-ansible | 15:29 | |
| *** janki has joined #openstack-ansible | 15:32 | |
| *** aszeszo has quit IRC | 15:33 | |
| *** aszeszo has joined #openstack-ansible | 15:33 | |
| odyssey4me | mhayden heh, it worked: http://logs.openstack.org/87/348887/10/check/gate-openstack-ansible-pip_install-ansible-func-centos-7/3c40e3e/ | 15:36 |
| odyssey4me | we just need it to work on xenial & trusty too now | 15:36 |
| * mhayden cries | 15:36 | |
| mhayden | :P | 15:37 |
| mhayden | meeting coming up in ~ 23 minutes, folks | 15:37 |
| palendae | Thanks Mr Alarm Clock | 15:40 |
| DrifterZA | oh the utter sigh | 15:41 |
| *** v1k0d3n has quit IRC | 15:41 | |
| DrifterZA | odyssey4me: I know why its breaking | 15:42 |
| odyssey4me | DrifterZA ? | 15:43 |
| openstackgerrit | Andy McCrae proposed openstack/openstack-ansible-os_swift: Properly default pretend_min_part_hours_passed https://review.openstack.org/351256 | 15:43 |
| DrifterZA | So I need to create a pypi package for it to build into horizon venv | 15:43 |
| DrifterZA | :( | 15:44 |
| odyssey4me | DrifterZA hmm, you don't :) | 15:44 |
| odyssey4me | but all the other extensions happen to be on pypi | 15:44 |
| odyssey4me | https://pypi.python.org/pypi/magnum-ui/0.0.1 ? | 15:45 |
| odyssey4me | it only needs to work for the role test | 15:45 |
| DrifterZA | its only a place holder | 15:45 |
| odyssey4me | but if that doesn't work, you'll need more machinery | 15:45 |
| DrifterZA | Theres no code in 0.0.1 | 15:45 |
| DrifterZA | :( | 15:45 |
| odyssey4me | ah ok - lemme point you in the direction | 15:45 |
| odyssey4me | the same bits you added to the integrated repo will need to be added to the role defaults, similar to https://github.com/openstack/openstack-ansible-os_horizon/blob/master/defaults/main.yml#L19-L25 | 15:46 |
| odyssey4me | but that should point at the magnum-ui repo | 15:46 |
| odyssey4me | you should only need the git_repo, git_install_branch, and developer_constraints bits | 15:47 |
| DrifterZA | will the venv build it then even without the pypi | 15:47 |
| DrifterZA | if I put the integrated repo stuff in there | 15:47 |
| odyssey4me | DrifterZA yeah, then you'll build directly from source - just like horizon does in the role | 15:47 |
| DrifterZA | ah so extra things in the default/main.yml get built in repo | 15:48 |
| odyssey4me | DrifterZA no, the role doesn't build a repo at all | 15:49 |
| odyssey4me | it just installs from source directly | 15:49 |
| odyssey4me | DrifterZA ok, so add a git repo, then add an item to the horizon_developer_constraints which includes the appropriate egg name | 15:52 |
| odyssey4me | then there's already a task in the install tasks which will build it from source... no more machinery needed | 15:52 |
| mhayden | Weekly meeting in ~ 4-5 minutes in #openstack-meeting-4! cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, mhayden, scarlisle, luckyinva, ntt, javeriak, automagically, spotz, vdo, jmccrory, alextricity25, | 15:55 |
| mhayden | jasondotstar, admin0, michaelgugino, ametts, v1k0d3n, severion, bgmccollum, darrenc, JRobinson__, asettle, colinmcnamara, thorst, adreznec, eil397 | 15:55 |
| *** admin0 has quit IRC | 15:55 | |
| DrifterZA | testing odyssey4me | 15:56 |
| *** ametts has quit IRC | 15:57 | |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_client: Prevent syslog leakage https://review.openstack.org/351182 | 15:57 |
| *** eil397 has joined #openstack-ansible | 15:58 | |
| cloader89 | hi all | 16:01 |
| *** palendae has left #openstack-ansible | 16:03 | |
| *** palendae has joined #openstack-ansible | 16:03 | |
| *** ametts has joined #openstack-ansible | 16:04 | |
| *** TxGirlGeek has joined #openstack-ansible | 16:18 | |
| *** TxGirlGeek has quit IRC | 16:19 | |
| *** weshay is now known as weshay_food | 16:21 | |
| evrardjp | hi cloader89 | 16:22 |
| *** janki has quit IRC | 16:24 | |
| *** DrifterZA has quit IRC | 16:31 | |
| *** kro_focused is now known as krotscheck | 16:32 | |
| *** janki has joined #openstack-ansible | 16:32 | |
| *** weshay_food is now known as weshay | 16:43 | |
| *** janki has quit IRC | 16:50 | |
| *** david-lyle has quit IRC | 16:53 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_build: Fix repo constraints construction and package installation https://review.openstack.org/350576 | 16:53 |
| *** david-lyle has joined #openstack-ansible | 16:53 | |
| *** Mudpuppy has quit IRC | 16:53 | |
| *** berendt has joined #openstack-ansible | 16:55 | |
| *** asettle has quit IRC | 16:55 | |
| odyssey4me | boo, mhayden it appears that some nodes have the new bindep and some don't | 17:00 |
| odyssey4me | I guess the SNI fix waits until tomorrow - or next week | 17:00 |
| odyssey4me | for now, we keep rechecking | 17:00 |
| mhayden | razzafrazza | 17:01 |
| mhayden | :| | 17:01 |
| mhayden | i'll be out tomorrow, so if someone can nudge it, i'd be much obliged | 17:01 |
| odyssey4me | yeah, I'll keep an eye on it | 17:02 |
| odyssey4me | once it goes, if you're not around then I'll replicate it | 17:02 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible-os_neutron: WIP: Project Calico integration https://review.openstack.org/340174 | 17:02 |
| jmccrory | hmm so is aodh's package list breaking the integrated gate now? http://logs.openstack.org/88/351188/2/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-trusty/8f7b495/console.html#_2016-08-04_15_25_37_531686 https://github.com/openstack/openstack-ansible-os_aodh/blob/master/defaults/main.yml#L105 | 17:03 |
| logan- | jmccrory: yes | 17:03 |
| logan- | i'm seeing it as of the last hour or two locally | 17:04 |
| mhayden | jmccrory / automagically / thorst / adreznec: y'all are all preregistered at the rackspace front desk | 17:04 |
| jmccrory | thanks mhayden | 17:04 |
| mhayden | they said we have to keep an extra eye on automagically, though | 17:04 |
| mhayden | that last time he switched all the breakroom TV's to NBC | 17:05 |
| adreznec | Awesome | 17:05 |
| thorst | mhayden: thx! | 17:05 |
| odyssey4me | jmccrory logan- oh dear | 17:06 |
| odyssey4me | it might be the way we're treating the data in the selective wheel build - the [] may be messing with it | 17:06 |
| odyssey4me | we may have to do something like explcitly treat the data as a string | 17:06 |
| odyssey4me | not sure | 17:06 |
| odyssey4me | jmccrory can you figure out a fix please? ideally modify the data we test with in the repo_build role so that this case is tested too | 17:08 |
| odyssey4me | a temporary fix we could merge quickly would be to disable selective wheel building? but doing so means our builds rarely pass anyway due to timeouts | 17:08 |
| odyssey4me | I need a break. I've been working for 12 hours straight already, and have visitors this evening. | 17:09 |
| jmccrory | odyssey4me : sure, i'll look into it | 17:10 |
| odyssey4me | thanks jmccrory | 17:10 |
| *** permalac has quit IRC | 17:11 | |
| odyssey4me | err: http://logs.openstack.org/88/351188/2/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-trusty/8f7b495/logs/aio1_repo_container-d5fe772f/ | 17:13 |
| *** Mudpuppy has joined #openstack-ansible | 17:13 | |
| odyssey4me | lol, looks like we have a bind mount overlap or something | 17:13 |
| logan- | tested with selective wheel build disabled and the repo-build completed | 17:18 |
| odyssey4me | logan- yeah, that'll definitely work - but the integrated gate just keeps hitting the timeout | 17:19 |
| logan- | ah, yeah | 17:19 |
| odyssey4me | I happen to still have my test env - so spending a few mins trying some things | 17:20 |
| odyssey4me | hmm, odd - it works | 17:20 |
| odyssey4me | something weird | 17:20 |
| odyssey4me | heh, ok - just as luck would have it the test doesn't test building aodh at all - lemme fix that | 17:21 |
| itlinux | what's the best way to replace the ips floating 172.29.248 to something like 192.168.xxx on aio thanks | 17:25 |
| odyssey4me | itlinux you do that using openstack commands | 17:26 |
| *** smatzek_ has joined #openstack-ansible | 17:26 | |
| itlinux | ok sounds good. thanks | 17:26 |
| *** smatzek has quit IRC | 17:30 | |
| *** eil397 has left #openstack-ansible | 17:30 | |
| *** javeriak has joined #openstack-ansible | 17:32 | |
| odyssey4me | jmccrory so the issue is that it's getting a requirements line that looks like this | 17:33 |
| odyssey4me | aodh[mysql]aodh[mysql] | 17:33 |
| jmccrory | hmm wonder why | 17:35 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_aodh: Disable stderr logging https://review.openstack.org/351299 | 17:36 |
| odyssey4me | heh, the issue is with how requirement_normalized is defined | 17:36 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_barbican: Disable stderr logging https://review.openstack.org/351300 | 17:36 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_ceilometer: Disable stderr logging https://review.openstack.org/351301 | 17:36 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_cinder: Disable stderr logging https://review.openstack.org/351302 | 17:36 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_cloudkitty: Disable stderr logging https://review.openstack.org/351303 | 17:36 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_designate: Disable stderr logging https://review.openstack.org/351304 | 17:36 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_glance: Disable stderr logging https://review.openstack.org/351305 | 17:36 |
| odyssey4me | name: aodh[mysql] | 17:36 |
| odyssey4me | data: aodh[mysql] | 17:36 |
| odyssey4me | name_normalised: aodh[mysql] | 17:36 |
| odyssey4me | requirement_normalized: aodh[mysql]aodh[mysql] | 17:36 |
| odyssey4me | key: aodh | 17:36 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_gnocchi: Disable stderr logging https://review.openstack.org/351306 | 17:36 |
| odyssey4me | aodh[mysql]aodh[mysql] | 17:36 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_heat: Disable stderr logging https://review.openstack.org/351307 | 17:36 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_ironic: Disable stderr logging https://review.openstack.org/351308 | 17:36 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_keystone: Disable stderr logging https://review.openstack.org/351309 | 17:36 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_magnum: Disable stderr logging https://review.openstack.org/351310 | 17:36 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_neutron: Disable stderr logging https://review.openstack.org/351311 | 17:36 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_nova: Disable stderr logging https://review.openstack.org/351312 | 17:37 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_rally: Disable stderr logging https://review.openstack.org/351313 | 17:37 |
| sigmavirus | look at all of those reviews | 17:37 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_sahara: Disable stderr logging https://review.openstack.org/351314 | 17:37 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_swift: Disable stderr logging https://review.openstack.org/351315 | 17:37 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_tempest: Disable stderr logging https://review.openstack.org/351316 | 17:37 |
| jmccrory | odyssey4me: oh, name+data | 17:37 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_zaqar: Disable stderr logging https://review.openstack.org/351317 | 17:37 |
| odyssey4me | jmccrory yeah, the split of name + data in the initial regex doesn't work right | 17:37 |
| logan- | cloudnull: use_stderr defined twice in all those patches :( | 17:39 |
| cloudnull | yup... | 17:40 |
| cloudnull | :( | 17:40 |
| cloudnull | bash foo failed me | 17:40 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_aodh: Disable stderr logging https://review.openstack.org/351299 | 17:41 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_barbican: Disable stderr logging https://review.openstack.org/351300 | 17:41 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_barbican: Disable stderr logging https://review.openstack.org/351300 | 17:41 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_ceilometer: Disable stderr logging https://review.openstack.org/351301 | 17:42 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_cinder: Disable stderr logging https://review.openstack.org/351302 | 17:42 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_cloudkitty: Disable stderr logging https://review.openstack.org/351303 | 17:42 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_designate: Disable stderr logging https://review.openstack.org/351304 | 17:42 |
| *** admin0 has joined #openstack-ansible | 17:43 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_glance: Disable stderr logging https://review.openstack.org/351305 | 17:43 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_gnocchi: Disable stderr logging https://review.openstack.org/351306 | 17:43 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_heat: Disable stderr logging https://review.openstack.org/351307 | 17:43 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible-os_neutron: Allow Neutron services to specify config path https://review.openstack.org/349210 | 17:43 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_ironic: Disable stderr logging https://review.openstack.org/351308 | 17:44 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_keystone: Disable stderr logging https://review.openstack.org/351309 | 17:44 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_magnum: Disable stderr logging https://review.openstack.org/351310 | 17:44 |
| *** admin0 has quit IRC | 17:44 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_neutron: Disable stderr logging https://review.openstack.org/351311 | 17:44 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_nova: Disable stderr logging https://review.openstack.org/351312 | 17:44 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_rally: Disable stderr logging https://review.openstack.org/351313 | 17:45 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_sahara: Disable stderr logging https://review.openstack.org/351314 | 17:45 |
| odyssey4me | jmccrory I thikn I found a suitable solution | 17:45 |
| odyssey4me | patch incoming | 17:45 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible-os_neutron: WIP: Project Calico integration https://review.openstack.org/340174 | 17:45 |
| mhayden | if anyone has input on the security stuff i sent to the ML, please reply! http://lists.openstack.org/pipermail/openstack-dev/2016-August/100883.html | 17:46 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_swift: Disable stderr logging https://review.openstack.org/351315 | 17:46 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_tempest: Disable stderr logging https://review.openstack.org/351316 | 17:46 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_zaqar: Disable stderr logging https://review.openstack.org/351317 | 17:46 |
| cloudnull | logan-: fixed :) | 17:47 |
| logan- | :D | 17:47 |
| cloudnull | sigmavirus: ^^ | 17:47 |
| sigmavirus | cloudnull: reviewing | 17:47 |
| cloudnull | d34dh0r53 sigmavirus those patches, once merged will need to be proposed for backport into mitaka. | 17:48 |
| sigmavirus | cloudnull: I can do that | 17:48 |
| cloudnull | ok thanks :) | 17:48 |
| *** javeriak has quit IRC | 17:49 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_build: Fix selective repo_build for aodh[mysql] requirement https://review.openstack.org/351322 | 17:52 |
| odyssey4me | jmccrory ^ that should do it | 17:52 |
| odyssey4me | ok, I'm out for the night - cheers all | 17:53 |
| palendae | Later | 17:53 |
| jmccrory | odyssey4me: yep, looks to work. see ya | 17:54 |
| *** maurosr has quit IRC | 17:54 | |
| *** thorst has quit IRC | 17:55 | |
| *** thorst has joined #openstack-ansible | 17:55 | |
| logan- | michaelgugino: i ran the horizon tests locally and they failed with http://cdn.pasteraw.com/cj935d6u5ka0wfqiok6fsxdjya0i5ic | 17:56 |
| logan- | that looks suspiciously like a working horizon to me :) | 17:56 |
| michaelgugino | logan-: that is what the gate check is failing on | 17:57 |
| michaelgugino | I couldn't even get it to install locally | 17:57 |
| logan- | yeah I didn't run into that keystone error you got | 17:57 |
| michaelgugino | that error is indeed in error. In the tempest test, the preceding step is to login, so that check serves to check if login was successful | 17:57 |
| michaelgugino | you're seeing the login page, so login was unsuccessful | 17:57 |
| logan- | ahh | 17:57 |
| michaelgugino | but, you've got it working locally, maybe you can check the keystone / horizon logs and see what the issue is | 17:58 |
| logan- | yeah i'm digging thru keystone logs right now. | 17:58 |
| *** eil397 has joined #openstack-ansible | 17:58 | |
| d34dh0r53 | Is there a glob on review.openstack.org to look at all open PRs for openstack-ansible-*? | 17:58 |
| *** eil397_ has joined #openstack-ansible | 17:59 | |
| michaelgugino | d34dh0r53: in the topic of this channel | 17:59 |
| eil397 | https://review.openstack.org/#/q/project:^openstack/openstack-ansible.* status:open,n,z ? | 17:59 |
| michaelgugino | https://goo.gl/tTmdgs | 17:59 |
| d34dh0r53 | ahha, was missing the ., thanks michaelgugino | 17:59 |
| *** berendt has quit IRC | 18:00 | |
| michaelgugino | yw | 18:00 |
| *** thorst_ has joined #openstack-ansible | 18:01 | |
| openstackgerrit | Michael Gugino proposed openstack/openstack-ansible-os_nova: WIP: Implement tests for nova-lxd hypervisor https://review.openstack.org/350226 | 18:02 |
| *** thorst has quit IRC | 18:03 | |
| *** weezS has joined #openstack-ansible | 18:05 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible: Docs: Minor security overview update https://review.openstack.org/351325 | 18:15 |
| michaelgugino | gate checks where 'port not found' errors are taking out tempest seems to have been an identified bug in neutron | 18:17 |
| michaelgugino | https://bugs.launchpad.net/tempest/+bug/1586931 | 18:17 |
| openstack | Launchpad bug 1586931 in tempest "TestServerBasicOps: Test fails when deleting server and floating ip almost at the same time" [Undecided,In progress] - Assigned to Monica Tian (ytianyt) | 18:17 |
| michaelgugino | so, if we can get the sha's updated on neutron, that would be cool | 18:18 |
| michaelgugino | of course, it shows that as merged in July, so not sure why we're seeing it. | 18:19 |
| *** javeriak has joined #openstack-ansible | 18:22 | |
| eil397 | michaelgugino: we are not using head of tempest | 18:24 |
| eil397 | michaelgugino: https://github.com/openstack/openstack-ansible/blob/master/playbooks/defaults/repo_packages/openstack_other.yml#L30 | 18:24 |
| michaelgugino | the fix is in neutron, not tempest | 18:25 |
| michaelgugino | hmm, not sure why we're still seeing the error then, it looks like that code should be in the neutron deployments. | 18:28 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-security: Adjust TCP syncookes variable to bool https://review.openstack.org/351329 | 18:29 |
| michaelgugino | http://logs.openstack.org/06/350806/4/check/gate-openstack-ansible-os_tempest-ansible-func-ubuntu-trusty/6a2c7cc/console.html#_2016-08-04_16_06_59_878324 | 18:29 |
| eil397 | michaelgugino: neutron version should HEAD as of 27.07 https://github.com/openstack/openstack-ansible/blob/master/playbooks/defaults/repo_packages/openstack_services.yml#L110 | 18:31 |
| michaelgugino | yes, should be covered, but in automated testing, it's pulling directly from master | 18:31 |
| michaelgugino | http://logs.openstack.org/06/350806/4/check/gate-openstack-ansible-os_tempest-ansible-func-ubuntu-trusty/6a2c7cc/console.html#_2016-08-04_15_54_20_044407 | 18:31 |
| *** spedione|AWAY has quit IRC | 18:32 | |
| michaelgugino | git+https://git.openstack.org/openstack/neutron@master#egg=neutron | 18:32 |
| *** spedione|AWAY has joined #openstack-ansible | 18:36 | |
| *** spedione|AWAY is now known as spedione | 18:36 | |
| *** maurosr has joined #openstack-ansible | 18:41 | |
| logan- | digging around in some containers that failed lxc_container_create because of the interface failing to start http://cdn.pasteraw.com/4aompfw0qj0nzjr9qw3gqmgvfjyhfw7 | 18:41 |
| *** TxGirlGeek has joined #openstack-ansible | 18:43 | |
| logan- | also https://github.com/openstack/openstack-ansible-os_ceilometer/commit/14329548f565625283d3a148e49b1e7a7aed7524#diff-233562701f658aaa8c40c5e5c3f07159R57 has broken the integrated gate now due to http://cdn.pasteraw.com/s80gchfesyrs1iegwestqpqvp2pexnl | 18:46 |
| logan- | so https://review.openstack.org/#/c/344884/ has to merge for that to work | 18:47 |
| *** bootsha has quit IRC | 18:51 | |
| *** Andrew_jedi has quit IRC | 18:55 | |
| *** TxGirlGeek has quit IRC | 19:01 | |
| *** TxGirlGeek has joined #openstack-ansible | 19:01 | |
| *** jthorne has joined #openstack-ansible | 19:03 | |
| mrhillsman | does openstack-ansible support rolling upgrade of individual service(s)? | 19:04 |
| openstackgerrit | Merged openstack/openstack-ansible: Prevent syslog leakage https://review.openstack.org/351184 | 19:04 |
| eil397_ | mrhillsman: openstack services ? http://docs.openstack.org/developer/openstack-ansible/liberty/upgrade-guide/manual-upgrade.html#upgrade-openstack | 19:10 |
| eil397_ | mrhillsman: or infrastructure services. for example . upgrade galera itslef ? | 19:11 |
| *** hybridpollo has joined #openstack-ansible | 19:13 | |
| eil397_ | mrhillsman: i've sent link on liberty doc. by accident. | 19:13 |
| *** sdake_ has joined #openstack-ansible | 19:13 | |
| eil397_ | mrhillsman: http://docs.openstack.org/developer/openstack-ansible/mitaka/upgrade-guide/manual-upgrade.html#upgrade-openstack | 19:13 |
| *** javeriak has quit IRC | 19:13 | |
| *** sdake has quit IRC | 19:16 | |
| *** thorst_ is now known as thorst | 19:16 | |
| *** javeriak has joined #openstack-ansible | 19:16 | |
| ametts | eil397_, Right now Nova asserts that it supports rolling upgrades, Neutron doesn't have this capability yet. We're wondering if OSA will take advantage of Nova's RU capability and experience less downtime during the upgrade. | 19:16 |
| *** TxGirlGeek has quit IRC | 19:18 | |
| *** TxGirlGeek has joined #openstack-ansible | 19:18 | |
| *** TxGirlGeek has quit IRC | 19:22 | |
| *** spedione has quit IRC | 19:22 | |
| *** TxGirlGeek has joined #openstack-ansible | 19:22 | |
| *** thorst has quit IRC | 19:25 | |
| *** thorst has joined #openstack-ansible | 19:28 | |
| *** thorst has quit IRC | 19:29 | |
| *** TxGirlGeek has quit IRC | 19:29 | |
| eil397_ | ametts: are you talking about upgrade with minimal downtime ? https://wiki.openstack.org/wiki/Upgrade-with-minimal-downtime#Rolling_upgrade | 19:29 |
| ametts | eil397_, yes | 19:30 |
| *** spedione|AWAY has joined #openstack-ansible | 19:30 | |
| *** spedione|AWAY is now known as spedione | 19:30 | |
| *** jthorne has quit IRC | 19:31 | |
| *** thorst has joined #openstack-ansible | 19:32 | |
| *** javeriak has quit IRC | 19:35 | |
| *** berendt has joined #openstack-ansible | 19:35 | |
| eil397_ | ametts: this spec/blueprint is little bit outdated. i think. | 19:36 |
| eil397_ | ametts: maybe this doc is good to describe upgrade process. http://docs.openstack.org/developer/nova/upgrade.html | 19:37 |
| eil397_ | ametts: and "nova-manage db online_data_migrations" is part of this process. https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_db_setup.yml#L35 | 19:38 |
| eil397_ | ametts: Perform Nova online data migrations | 19:38 |
| ametts | eil397_, Ah... so maybe OSA can indeed perform a "rolling" Nova upgrade? | 19:39 |
| messy | cloudnull: you submitted your multi-node aio project to be merged with osa right? Where can I go to look at it? | 19:41 |
| eil397_ | ametts: osa will do upgrade as it mentioned in doc I've sent above to mrhillsman | 19:41 |
| ametts | Ah. got it -- thanks! | 19:41 |
| eil397_ | messy: I've seen it in review https://review.openstack.org/#/c/349662/ | 19:42 |
| messy | thanks eil397_ | 19:42 |
| eil397_ | yw | 19:42 |
| eil397_ | ametts: and for neutron db migration works like "neutron-db-manage upgrade --expand" http://docs.openstack.org/developer/neutron/devref/upgrade.html | 19:45 |
| eil397_ | ametts: https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/neutron_db_setup.yml#L30 | 19:45 |
| logan- | michaelgugino: horizon logs. keystone: http://cdn.pasteraw.com/hnrx15rr89bmf14qsj0ybkenr5soo34 horizon: http://cdn.pasteraw.com/d1yn5m0yk7u3kdpzowe4sowcl33l9gc ngrep of keystone traffic: http://cdn.pasteraw.com/dna1j9fhr0q4f5lq31yaoxd3a94lon2 | 19:47 |
| logan- | i guess our issue there is the snippet from the keystone log: "Authorization failed. The request you have made requires authentication. from 10.100.100.166" | 19:47 |
| logan- | i can log into the horizon panel just fine from my browser heh | 19:47 |
| ametts | thanks, eil397_ We're trying to figure out what needs to be done to support the rolling/minimal downtime options. We'd like to test what can/cannot be done, and start iterating on improvements from there. | 19:48 |
| eil397_ | ametts: code upgrade and db migration steps , of course, are not 100% of success. state of network infra depends on your configuration. | 19:48 |
| ametts | eil397_, Agree. We're also working to define the test cases that indicate "it worked" when the process completes. | 19:49 |
| michaelgugino | logan-: that's interesting. During my local build, the fernet keys step failed to run during deploying keystone | 19:49 |
| michaelgugino | and that's the step right about the authorization failure | 19:49 |
| logan- | yea very interesting | 19:50 |
| *** aszeszo1 has joined #openstack-ansible | 19:50 | |
| *** aszeszo has quit IRC | 19:51 | |
| *** Andrew_jedi has joined #openstack-ansible | 19:52 | |
| logan- | it doesn't seem like keystone is broken though, since i can auth thru browser | 19:55 |
| *** asettle has joined #openstack-ansible | 20:00 | |
| *** TxGirlGeek has joined #openstack-ansible | 20:01 | |
| logan- | heh just got dnsmasq: error while loading shared libraries: libdbus-1.so.3: cannot stat shared object: Permission denied in another master aio | 20:03 |
| michaelgugino | if you can auth through the browser, perhaps the credentials for the temporary keystone users are not being created correctly | 20:03 |
| *** asettle has quit IRC | 20:04 | |
| michaelgugino | tempest has the ability to use 'dynamic_credentials' which create new hosts and whatnot during testing to avoid breaking things with an already established user/project | 20:04 |
| *** ametts has quit IRC | 20:05 | |
| logan- | right, in the network trace you can see it authing to one of the tempest random users it created. guess that's possible, that keystone is failing there | 20:06 |
| michaelgugino | I think we'll need debug logging enabled in keystone, need to see the post data | 20:10 |
| logan- | easy enough, one sec | 20:10 |
| *** ametts has joined #openstack-ansible | 20:10 | |
| logan- | i think after the repo_build and ceilometer gate blockers, we're going to hit this http://cdn.pasteraw.com/q3zsfblxkpkpz6iq4ydwu543poi6p53 | 20:11 |
| michaelgugino | I think cloudnull and odyssey4me were working on some bind mounting issues that might be causing that? | 20:12 |
| *** markvoelker has quit IRC | 20:13 | |
| cloudnull | sorry was afk in meetings. | 20:19 |
| cloudnull | logan- michaelgugino: what are the current gate blockers? | 20:19 |
| logan- | 1) https://review.openstack.org/351322 | 20:19 |
| logan- | 2) also https://github.com/openstack/openstack-ansible-os_ceilometer/commit/14329548f565625283d3a148e49b1e7a7aed7524#diff-233562701f658aaa8c40c5e5c3f07159R57 has broken the integrated gate now due to http://cdn.pasteraw.com/s80gchfesyrs1iegwestqpqvp2pexnl | 20:20 |
| logan- | so https://review.openstack.org/#/c/344884/ has to merge for that to work | 20:20 |
| michaelgugino | I added OSA to this bug, cloudnull https://bugs.launchpad.net/tempest/+bug/1586931 | 20:20 |
| openstack | Launchpad bug 1586931 in tempest "TestServerBasicOps: Test fails when deleting server and floating ip almost at the same time" [Undecided,In progress] - Assigned to Monica Tian (ytianyt) | 20:20 |
| cloudnull | michaelgugino: ++ | 20:20 |
| cloudnull | :| | 20:21 |
| openstackgerrit | Michael Gugino proposed openstack/openstack-ansible-os_nova: WIP: Implement tests for nova-lxd hypervisor https://review.openstack.org/350226 | 20:21 |
| logan- | michaelgugino: http://cdn.pasteraw.com/4yb07gsy0jzb6dgd7bc2n9s0su7yldq | 20:22 |
| logan- | keystone+debug | 20:22 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum: Disable stderr logging https://review.openstack.org/351310 | 20:23 |
| logan- | ah there we go "User 4359918be27c4768b2e83b7f985db74b has no access to domain default" | 20:23 |
| logan- | becaise the domain is "Default" probably | 20:26 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer: Disable stderr logging https://review.openstack.org/351301 | 20:28 |
| cloudnull | logan-: do we need to disable ceilometer for the time being ? | 20:29 |
| *** catintheroof has joined #openstack-ansible | 20:30 | |
| cloudnull | -cc odyssey4me? | 20:30 |
| logan- | yeah either that or pin the role imo | 20:30 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: disable ceilometer and its supporting services https://review.openstack.org/351385 | 20:33 |
| *** Jeffrey4l has quit IRC | 20:34 | |
| *** Jeffrey4l_ has joined #openstack-ansible | 20:35 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican: Disable stderr logging https://review.openstack.org/351300 | 20:35 |
| michaelgugino | logan-: I cannot see where 4359918be27c4768b2e83b7f985db74b is even being created. It's deleted, but I never see a call to keystone for it's creation | 20:37 |
| openstackgerrit | Ian Cordasco proposed openstack/openstack-ansible-os_ceilometer: Disable stderr logging https://review.openstack.org/351387 | 20:38 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic: Disable stderr logging https://review.openstack.org/351308 | 20:38 |
| logan- | 2016-08-04 20:19:12.794 3847 INFO keystone.common.wsgi [req-f0e73b1c-b708-4676-8ca1-d82e20eab46b 7bdbf3159a8942698b9e1c08a0f0bef3 2a14cbaae98a4ecc9e6458450bb09a55 - default default] PUT http://10.100.100.3:35357/v3/projects/3971cfd3a67c46768b9ca7a06d63cc56/users/4359918be27c4768b2e83b7f985db74b/roles/ed00717fd0c74837ba3c8f811d6fe8e0 | 20:38 |
| *** woodard has quit IRC | 20:38 | |
| openstackgerrit | Ian Cordasco proposed openstack/openstack-ansible-os_ceilometer: Disable stderr logging https://review.openstack.org/351387 | 20:38 |
| michaelgugino | possibly here: 04/Aug/2016:20:19:12 | 20:39 |
| michaelgugino | but I cannot see the response data | 20:39 |
| *** jthorne has joined #openstack-ansible | 20:40 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone: Disable stderr logging https://review.openstack.org/351309 | 20:40 |
| openstackgerrit | Ian Cordasco proposed openstack/openstack-ansible-os_ironic: Disable stderr logging https://review.openstack.org/351390 | 20:40 |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh: Disable stderr logging https://review.openstack.org/351299 | 20:41 |
| logan- | yeah I'm guessing it is the identity:create_user at 2016-08-04 20:19:12.562 | 20:41 |
| logan- | username tempest-TestDashboardBasicOps-618778480 | 20:42 |
| *** mikelk has joined #openstack-ansible | 20:42 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_cloudkitty: Disable stderr logging https://review.openstack.org/351303 | 20:42 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat: Disable stderr logging https://review.openstack.org/351307 | 20:42 |
| michaelgugino | hmm, default domain is a relatively new addition to tempest-horizon: https://github.com/openstack/tempest-horizon/commit/64302f192f9be68e4072c43262fb0928923e8cba | 20:44 |
| *** smatzek_ has quit IRC | 20:44 | |
| michaelgugino | well, looks like it's a brand new project :/ | 20:45 |
| michaelgugino | it looks like it should be creating a new domain called tempest_horizon | 20:47 |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance: Disable stderr logging https://review.openstack.org/351305 | 20:47 |
| michaelgugino | nvm, I don't know what it's supposed to be doing | 20:49 |
| logan- | i don't either but i think its the test that's broken, not the role :D | 20:50 |
| *** KLevenstein has joined #openstack-ansible | 20:51 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift: Disable stderr logging https://review.openstack.org/351315 | 20:52 |
| openstackgerrit | Ian Cordasco proposed openstack/openstack-ansible-os_aodh: Disable stderr logging https://review.openstack.org/351397 | 20:55 |
| openstackgerrit | Ian Cordasco proposed openstack/openstack-ansible-os_swift: Disable stderr logging https://review.openstack.org/351399 | 20:56 |
| sigmavirus | so a bunch of these don't cherry-pick cleanly | 20:58 |
| *** retreved has quit IRC | 20:59 | |
| *** psilvad has quit IRC | 21:01 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron: Disable stderr logging https://review.openstack.org/351311 | 21:01 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts: Implement primary and secondary keyserver usage for cache prep https://review.openstack.org/350684 | 21:03 |
| *** Mudpuppy has quit IRC | 21:06 | |
| *** abelur has joined #openstack-ansible | 21:08 | |
| *** TxGirlGeek has quit IRC | 21:08 | |
| mrhillsman | eil397_ apologies for delay in replying but looks like ametts picked up the slack | 21:09 |
| *** thorst has quit IRC | 21:10 | |
| eil397_ | mrhillsman: np, thank you for question. was good chance for me to check what I know | 21:12 |
| mrhillsman | :) | 21:13 |
| *** thorst has joined #openstack-ansible | 21:15 | |
| *** thorst has quit IRC | 21:19 | |
| *** woodard has joined #openstack-ansible | 21:22 | |
| *** woodard has quit IRC | 21:22 | |
| *** woodard has joined #openstack-ansible | 21:23 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_gnocchi: Disable stderr logging https://review.openstack.org/351306 | 21:23 |
| michaelgugino | I'm going to add a few items to the os_tempest role config file, remove some deprecated / unsupported options (again) | 21:23 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally: Disable stderr logging https://review.openstack.org/351313 | 21:23 |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara: Disable stderr logging https://review.openstack.org/351314 | 21:24 |
| mrda | Morning OSA | 21:25 |
| *** ametts has quit IRC | 21:26 | |
| *** mikelk has quit IRC | 21:26 | |
| *** wadeholler has quit IRC | 21:35 | |
| *** wadeholler has joined #openstack-ansible | 21:36 | |
| michaelgugino | morning mrda | 21:37 |
| *** wadeholler has quit IRC | 21:40 | |
| openstackgerrit | Michael Gugino proposed openstack/openstack-ansible-os_tempest: Fix deprecations in tempest.conf https://review.openstack.org/351410 | 21:41 |
| *** galstrom is now known as galstrom_zzz | 21:42 | |
| *** GMAzrael has quit IRC | 21:46 | |
| *** michaelgugino has quit IRC | 21:46 | |
| *** GMAzrael has joined #openstack-ansible | 21:47 | |
| spotz | hey mrda | 21:52 |
| *** KLevenstein has quit IRC | 21:53 | |
| *** thorst has joined #openstack-ansible | 21:56 | |
| mrda | o/ | 21:57 |
| *** kulinacs has quit IRC | 21:59 | |
| *** jamesdenton has quit IRC | 22:01 | |
| *** thorst has quit IRC | 22:03 | |
| *** thorst has joined #openstack-ansible | 22:05 | |
| *** thorst has quit IRC | 22:07 | |
| *** thorst has joined #openstack-ansible | 22:08 | |
| *** thorst has quit IRC | 22:16 | |
| *** messy has quit IRC | 22:18 | |
| *** klamath has quit IRC | 22:19 | |
| *** spotz is now known as spotz_zzz | 22:23 | |
| *** aszeszo1 has quit IRC | 22:24 | |
| *** daneyon__ has joined #openstack-ansible | 22:24 | |
| *** weezS has quit IRC | 22:26 | |
| *** daneyon has quit IRC | 22:28 | |
| *** daneyon__ has quit IRC | 22:28 | |
| *** furlongm_ has quit IRC | 22:30 | |
| *** jthorne has quit IRC | 22:30 | |
| *** furlongm_ has joined #openstack-ansible | 22:31 | |
| *** abelur is now known as abelur|brb | 22:58 | |
| *** eil397 has left #openstack-ansible | 23:01 | |
| *** eil397_ has left #openstack-ansible | 23:01 | |
| *** asettle has joined #openstack-ansible | 23:03 | |
| *** asettle has quit IRC | 23:07 | |
| *** abelur|brb is now known as abelur | 23:11 | |
| *** catintheroof has quit IRC | 23:13 | |
| *** thorst has joined #openstack-ansible | 23:15 | |
| openstackgerrit | Logan V proposed openstack/openstack-ansible: WIP: Project Calico integration https://review.openstack.org/340175 | 23:20 |
| *** thorst has quit IRC | 23:22 | |
| *** markvoelker has joined #openstack-ansible | 23:28 | |
| *** coolj has quit IRC | 23:30 | |
| *** jamielennox is now known as jamielennox|away | 23:30 | |
| *** coolj has joined #openstack-ansible | 23:32 | |
| *** markvoelker_ has joined #openstack-ansible | 23:37 | |
| *** berendt has quit IRC | 23:38 | |
| *** Andrew_jedi has quit IRC | 23:40 | |
| *** markvoelker has quit IRC | 23:41 | |
| *** jamielennox|away is now known as jamielennox | 23:41 | |
| *** Jeffrey4l_ has quit IRC | 23:42 | |
| *** sdake has joined #openstack-ansible | 23:44 | |
| *** Jeffrey4l has joined #openstack-ansible | 23:45 | |
| *** catintheroof has joined #openstack-ansible | 23:46 | |
| *** sdake_ has quit IRC | 23:46 | |
| *** thorst has joined #openstack-ansible | 23:56 | |
| *** weshay has quit IRC | 23:56 | |
| *** thorst has quit IRC | 23:57 | |
| *** thorst has joined #openstack-ansible | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!