| openstackgerrit | Merged openstack/openstack-ansible stable/newton: Correctly map nova git sources https://review.openstack.org/481910 | 00:00 |
|---|---|---|
| *** charcol has joined #openstack-ansible | 00:07 | |
| *** yifei has joined #openstack-ansible | 00:13 | |
| *** thorst has quit IRC | 00:17 | |
| *** jbadiapa_ has quit IRC | 00:17 | |
| *** vishwanathj has quit IRC | 00:30 | |
| *** vishwanathj has joined #openstack-ansible | 00:30 | |
| *** perniciouscaffei has joined #openstack-ansible | 00:31 | |
| *** jbadiapa_ has joined #openstack-ansible | 00:32 | |
| *** woodard_ has joined #openstack-ansible | 00:47 | |
| *** woodard has quit IRC | 00:47 | |
| *** vishwanathj has quit IRC | 00:53 | |
| *** vishwanathj has joined #openstack-ansible | 00:54 | |
| *** acormier has joined #openstack-ansible | 00:58 | |
| *** gouthamr has quit IRC | 01:06 | |
| *** dixiaoli has joined #openstack-ansible | 01:09 | |
| *** acormier has quit IRC | 01:12 | |
| *** acormier has joined #openstack-ansible | 01:12 | |
| *** cathrichardson has quit IRC | 01:12 | |
| *** cathrichardson has joined #openstack-ansible | 01:13 | |
| *** perniciouscaffei has quit IRC | 01:13 | |
| *** vishwanathj has quit IRC | 01:16 | |
| *** vishwanathj has joined #openstack-ansible | 01:17 | |
| *** vishwanathj has quit IRC | 01:18 | |
| *** vishwanathj has joined #openstack-ansible | 01:18 | |
| *** acormier has quit IRC | 01:26 | |
| *** acormier has joined #openstack-ansible | 01:26 | |
| *** dfflanders has quit IRC | 01:26 | |
| *** galstrom_zzz is now known as galstrom | 01:42 | |
| *** vnogin has joined #openstack-ansible | 02:00 | |
| *** cmart has quit IRC | 02:03 | |
| *** smatzek has joined #openstack-ansible | 02:03 | |
| *** vnogin has quit IRC | 02:04 | |
| *** acormier has quit IRC | 02:08 | |
| *** acormier has joined #openstack-ansible | 02:08 | |
| *** cmart has joined #openstack-ansible | 02:14 | |
| *** dxiri has joined #openstack-ansible | 02:14 | |
| *** dxiri has quit IRC | 02:20 | |
| *** gouthamr has joined #openstack-ansible | 02:25 | |
| *** perniciouscaffei has joined #openstack-ansible | 02:34 | |
| *** thorst has joined #openstack-ansible | 02:39 | |
| *** thorst has quit IRC | 02:44 | |
| *** smatzek has quit IRC | 03:00 | |
| *** perniciouscaffei has quit IRC | 03:04 | |
| *** acormier has quit IRC | 03:10 | |
| *** acormier has joined #openstack-ansible | 03:10 | |
| *** acormier has quit IRC | 03:15 | |
| *** brad[] has quit IRC | 03:16 | |
| *** ricardoas1 has joined #openstack-ansible | 03:26 | |
| *** ricardoas has quit IRC | 03:27 | |
| *** galstrom is now known as galstrom_zzz | 03:34 | |
| *** schwicht has quit IRC | 03:35 | |
| cloudnull | evenings | 03:36 |
| cloudnull | LiterateHawk: what are you needing to do ? | 03:36 |
| cloudnull | the rabbitmq containers should already be in the inventory | 03:36 |
| cloudnull | which is part of the infra hosts | 03:36 |
| *** udesale has joined #openstack-ansible | 03:44 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-os_barbican master: Perform an atomic policy file change https://review.openstack.org/461081 | 03:45 |
| *** gouthamr has quit IRC | 03:46 | |
| *** gkadam has joined #openstack-ansible | 03:47 | |
| *** vnogin has joined #openstack-ansible | 03:48 | |
| *** vnogin has quit IRC | 03:53 | |
| *** dxiri has joined #openstack-ansible | 03:56 | |
| LiterateHawk | cloudnull I ran the playbook and it failed at rabbitmq. I tried deleting the containers and re-running and it didn't help. I deleted everything and re-ran, and it worked this time. | 03:59 |
| cloudnull | ok | 04:00 |
| LiterateHawk | Somehow the node name that the CLI was using wasn't matching what erlang expected | 04:00 |
| cloudnull | the nuclear option :) | 04:00 |
| cloudnull | if you ever need to remove something from inventory | 04:01 |
| cloudnull | theres an inventory manage command | 04:01 |
| cloudnull | which can purge an entry | 04:01 |
| LiterateHawk | It's weird how people not versed in computers think "turn it off and turn it on again" is a copout, but it actually works | 04:01 |
| cloudnull | ++ :) | 04:01 |
| LiterateHawk | I'd be curious to hear about it | 04:01 |
| LiterateHawk | I assumed there was something cached somewhere that was holding things up, but I couldn't find it for the life of me | 04:02 |
| *** weezS has joined #openstack-ansible | 04:04 | |
| *** weezS has quit IRC | 04:05 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: Optimise role execution https://review.openstack.org/482116 | 04:06 |
| *** poopcat has quit IRC | 04:08 | |
| *** NostawRm has quit IRC | 04:08 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon master: Optimise the role execution https://review.openstack.org/482101 | 04:10 |
| *** weezS has joined #openstack-ansible | 04:10 | |
| *** NostawRm has joined #openstack-ansible | 04:13 | |
| *** weezS has quit IRC | 04:19 | |
| cloudnull | LiterateHawk: the inventory is stored in a json file within /etc/openstack_deploy/openstack_inventory.json | 04:26 |
| *** cpuga has joined #openstack-ansible | 04:26 | |
| cloudnull | the manage command https://github.com/openstack/openstack-ansible/blob/master/osa_toolkit/manage.py | 04:26 |
| cloudnull | can interface with the inventory and take care of some of the maint as needed | 04:27 |
| *** cNilesh has joined #openstack-ansible | 04:27 | |
| *** dxiri_ has joined #openstack-ansible | 04:32 | |
| LiterateHawk | cloudnull, Ugh, wish I had known that earlier! Thanks for the tip. This is just a 5 node PoC, so that'll be helpfulto remember if we decide to scale it up | 04:33 |
| cloudnull | ++ | 04:34 |
| cloudnull | It's saved me a few times :) | 04:34 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/ocata: Update neutron test playbook https://review.openstack.org/481685 | 04:35 |
| *** dxiri has quit IRC | 04:36 | |
| *** chhavi has joined #openstack-ansible | 04:40 | |
| *** thorst has joined #openstack-ansible | 04:40 | |
| *** pcaruana has joined #openstack-ansible | 04:41 | |
| *** thorst has quit IRC | 04:45 | |
| *** charcol has quit IRC | 04:57 | |
| *** pcaruana has quit IRC | 04:58 | |
| *** tobberydberg has joined #openstack-ansible | 04:59 | |
| *** cpuga has quit IRC | 05:01 | |
| *** NostawRm has quit IRC | 05:16 | |
| *** NostawRm has joined #openstack-ansible | 05:16 | |
| *** cpuga has joined #openstack-ansible | 05:21 | |
| *** charcol has joined #openstack-ansible | 05:28 | |
| *** cpuga has quit IRC | 05:28 | |
| *** vnogin has joined #openstack-ansible | 05:36 | |
| *** vnogin has quit IRC | 05:41 | |
| *** chhavi_ has joined #openstack-ansible | 05:44 | |
| *** chhavi has quit IRC | 05:44 | |
| *** dxiri_ has quit IRC | 05:46 | |
| *** perniciouscaffei has joined #openstack-ansible | 05:49 | |
| *** gtrxcb has joined #openstack-ansible | 06:13 | |
| *** gtrxcb has quit IRC | 06:13 | |
| *** perniciouscaffei has quit IRC | 06:14 | |
| *** Oku_OS-away is now known as Oku_OS | 06:19 | |
| *** ricardoas1 has quit IRC | 06:23 | |
| *** hybridpollo has quit IRC | 06:27 | |
| *** strattao has quit IRC | 06:33 | |
| *** cshen_ has joined #openstack-ansible | 06:40 | |
| *** dixiaoli has quit IRC | 06:41 | |
| *** thorst has joined #openstack-ansible | 06:41 | |
| *** perniciouscaffei has joined #openstack-ansible | 06:43 | |
| *** thorst has quit IRC | 06:46 | |
| *** openstackgerrit has quit IRC | 06:48 | |
| -openstackstatus- NOTICE: FYI, ask.openstack.org is down, review.o.o is slow - please have patience until this is fixed | 06:50 | |
| *** kman has joined #openstack-ansible | 06:53 | |
| *** perniciouscaffei has quit IRC | 06:58 | |
| *** mbuil has joined #openstack-ansible | 06:59 | |
| *** udesale has quit IRC | 07:03 | |
| *** LiterateHawk has quit IRC | 07:09 | |
| *** shardy has joined #openstack-ansible | 07:13 | |
| *** cshen_ has quit IRC | 07:16 | |
| *** kman has quit IRC | 07:17 | |
| mbuil | andymccr: ping | 07:23 |
| *** pcaruana has joined #openstack-ansible | 07:31 | |
| *** jvidal has joined #openstack-ansible | 07:32 | |
| *** pcaruana has quit IRC | 07:41 | |
| *** kristian__ has joined #openstack-ansible | 07:44 | |
| *** kristian__ has quit IRC | 07:45 | |
| *** pcaruana has joined #openstack-ansible | 07:45 | |
| -openstackstatus- NOTICE: Gerrit is going to be restarted, due to low performance | 07:48 | |
| *** tobberydberg has quit IRC | 07:51 | |
| *** tobberydberg has joined #openstack-ansible | 07:52 | |
| -openstackstatus- NOTICE: Gerrit has been successfully restarted | 07:54 | |
| *** tobberydberg has quit IRC | 07:56 | |
| *** openstackgerrit has joined #openstack-ansible | 07:58 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: test-setup-swifthosts: Add -K option to mkfs.xfs https://review.openstack.org/482835 | 07:58 |
| *** kristian__ has joined #openstack-ansible | 08:00 | |
| *** gouthamr has joined #openstack-ansible | 08:02 | |
| *** lkoranda has joined #openstack-ansible | 08:03 | |
| openstackgerrit | Adrien Cunin proposed openstack/openstack-ansible master: Vars passed to openstack-ansible take precedence https://review.openstack.org/482440 | 08:06 |
| *** cmart has quit IRC | 08:13 | |
| *** tobberydberg has joined #openstack-ansible | 08:15 | |
| *** gouthamr has quit IRC | 08:21 | |
| *** cmart has joined #openstack-ansible | 08:22 | |
| *** kristian__ has quit IRC | 08:26 | |
| *** pcaruana has quit IRC | 08:33 | |
| *** armaan has joined #openstack-ansible | 08:39 | |
| armaan | Hello, folks is there a way to tell OSA to store the contents of "/var/lib/nova/save" in Ceph storage. | 08:40 |
| *** thorst has joined #openstack-ansible | 08:42 | |
| andymccr | armaan: i dont think that'd be an OSA specific thing, if nova/ceph can interact that way then sure OSA could do it | 08:45 |
| andymccr | mbuil: pong! does that mean you're ready to go? | 08:45 |
| *** tobberydberg has quit IRC | 08:46 | |
| *** pcaruana has joined #openstack-ansible | 08:46 | |
| armaan | andymccr: I have no space left in /var on my compute node because /var/lib/nova/save consumed all of the space. I am not sure where to go from where, i thought the entire /var/lib/nova will go in ceph | 08:46 |
| *** tobberydberg has joined #openstack-ansible | 08:46 | |
| *** thorst has quit IRC | 08:47 | |
| *** dxiri has joined #openstack-ansible | 08:47 | |
| *** electrofelix has joined #openstack-ansible | 08:48 | |
| mbuil | andymccr: yes, the PR was merged | 08:48 |
| mbuil | andymccr: this is the repo: https://github.com/jrametta/openstack-ansible-os_tacker | 08:49 |
| andymccr | armaan: i dont think that happens by default, you need to make your ephemeral disks use the ceph backend. i cant remember how to do that, but if you search about nova/ceph docs im sure you'll find something | 08:49 |
| andymccr | mbuil: ok cool - i'll put the PR in now | 08:50 |
| *** jbadiapa_ has quit IRC | 08:50 | |
| *** dxiri has quit IRC | 08:52 | |
| *** jbadiapa_ has joined #openstack-ansible | 08:52 | |
| armaan | andymccr: Yes, i have that configures, "images_type = rbd" | 08:53 |
| *** admin0 has joined #openstack-ansible | 08:54 | |
| andymccr | so is /var/lib/nova/instances stored in ceph? I'm not too sure what goes in /var/lib/nova/save - but again, its more of a nova/ceph integration question than an OSA question at this point. | 08:56 |
| *** vnogin has joined #openstack-ansible | 09:00 | |
| *** pbandark has joined #openstack-ansible | 09:01 | |
| *** udesale has joined #openstack-ansible | 09:02 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Correct major upgrade detection logic https://review.openstack.org/482658 | 09:10 |
| armaan | andymccr: This patch changed the location, https://review.openstack.org/#/c/320624/ | 09:13 |
| *** cshen_ has joined #openstack-ansible | 09:15 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Correctly map nova git sources https://review.openstack.org/481909 | 09:15 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Set python interpreter for connection:local plays https://review.openstack.org/482090 | 09:16 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_heat stable/ocata: Optimise role execution https://review.openstack.org/482862 | 09:18 |
| andymccr | armaan: im not sure that would cause the issue since i dont think it would be using cephFS - so it wouldnt be trying to write to a filesystem if its setup properly. but you could test by setting the nova_libvirt_save_path var to be /var/lib/libvirt/qemu/save (or fixing the symlink manually), if that works then thats a bug that needs to get fixed asap. | 09:20 |
| odyssey4me | the /var/lib/nova/save directory is where the memory is stored for paused instances if I recall correctly | 09:21 |
| armaan | odyssey4me: yes, absolutely | 09:25 |
| armaan | odyssey4me: We are using a lot of suspend/resume for multiple heat stacks -- which filled up the directory | 09:26 |
| odyssey4me | well, there you have it | 09:26 |
| odyssey4me | I'm not sure if there's away for those to go into ceph | 09:26 |
| odyssey4me | it may be best to just ensure that you partition a lot of space in that location instead | 09:26 |
| armaan | odyssey4me: but vm pause is not same as vm suspend | 09:28 |
| armaan | IIRC | 09:28 |
| *** tots has joined #openstack-ansible | 09:31 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_horizon stable/ocata: Optimise the role execution https://review.openstack.org/482871 | 09:32 |
| *** pcaruana has quit IRC | 09:32 | |
| *** askb has quit IRC | 09:32 | |
| *** cmart has quit IRC | 09:32 | |
| *** pcaruana has joined #openstack-ansible | 09:32 | |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_server master: Consolidate package install tasks https://review.openstack.org/482715 | 09:33 |
| odyssey4me | armaan I wasn't quite sure if it was pause/suspend. The point is that's what's using the space. | 09:37 |
| armaan | odyssey4me: You are right, libvirt invokes a managedsave command which store the instance state in a file at "/var/lib/nova/save" | 09:41 |
| *** yifei has quit IRC | 09:43 | |
| *** thorst has joined #openstack-ansible | 09:43 | |
| admin0 | morning \o | 09:44 |
| admin0 | :D | 09:44 |
| admin0 | work late, get up late :) | 09:44 |
| *** thorst has quit IRC | 09:47 | |
| *** yifei has joined #openstack-ansible | 09:48 | |
| *** kristian__ has joined #openstack-ansible | 09:49 | |
| *** vnogin has quit IRC | 09:50 | |
| *** mpranjic has quit IRC | 09:53 | |
| *** mpranjic has joined #openstack-ansible | 09:53 | |
| *** brad[] has joined #openstack-ansible | 09:57 | |
| *** vnogin has joined #openstack-ansible | 09:59 | |
| *** yifei has quit IRC | 09:59 | |
| hwoarang | mhayden: https://review.openstack.org/#/c/481281/2 is ready to go :) | 10:06 |
| *** Martin___ has joined #openstack-ansible | 10:11 | |
| Martin___ | Hi all, I am trying to create a repository_container for arm architecture as a workaround for https://bugs.launchpad.net/openstack-ansible/+bug/1703618/. But the repo-install playbook fails with an error message I cannot resolve: | 10:14 |
| openstack | Launchpad bug 1703618 in openstack-ansible "repo_container on arm host contains packages for x86 hosts only" [Undecided,New] | 10:14 |
| Martin___ | fatal: [localhost]: FAILED! => {"failed": true, "msg": "The conditional check 'item in groups['repo_servers_' + ansible_distribution_version + '_' + ansible_architecture][0]' failed. The error was: error while evaluating conditional (item in groups['repo_servers_' + ansible_distribution_version + '_' + ansible_architecture][0]): 'dict object' has no attribute u'repo_servers_16.04_armv7l'\n\nThe error appears to have bee | 10:14 |
| Martin___ | I tried redefining the variable ansible_architecture which does not solve the problem. I guess I need to alter the dictionary. But where do I find that? | 10:15 |
| Martin___ | As a side note: I am currently trying to execute all playbooks on the arm node. Until the reported error, everything completed without errors. | 10:16 |
| andymccr | Martin___: which task/role caused the error? | 10:17 |
| Martin___ | andymccr TASK [Prepare group of master repo servers] | 10:18 |
| Martin___ | I think that is part of the role repo-install | 10:18 |
| Martin___ | andymccr: correction: its in the repo-build playbook | 10:19 |
| andymccr | Martin___: found it :) | 10:19 |
| odyssey4me | Martin___ that's a fact returned by ansible | 10:19 |
| odyssey4me | ansible_architecture needs to be returned by ansible as a fact - check by executing: ansible -m setup -a "filter=ansible_architecture" repo_all | 10:20 |
| Martin___ | that returns: "ansible_facts": { "ansible_architecture": "armv7l" } | 10:21 |
| andymccr | it looks like it did though odyssey4me: repo_servers_16.04_armv7l so im guessing ansible_architecture is armv7l | 10:21 |
| Martin___ | As we would expect, yes ;) | 10:22 |
| andymccr | but the attribute repo_servers_16.04_armv7l which im guessing we should set somewhere, doesnt exist | 10:22 |
| Martin___ | What I don't understand is where the problem with the dictionary comes from | 10:22 |
| *** cmart has joined #openstack-ansible | 10:22 | |
| *** udesale__ has joined #openstack-ansible | 10:24 | |
| jvidal | hello | 10:25 |
| Martin___ | I just saw: something shortened the error message, here is the rest: The error appears to have been in '/opt/openstack-ansible/playbooks/repo-build.yml': line 36, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n tasks:\n - name: Prepare group of master repo servers\n ^ here\n"} | 10:25 |
| jvidal | I am deploying with OSA for OPNFV, but I am getting some errors in lxc_containers_create lately | 10:25 |
| jvidal | TASK [lxc_container_create : Drop container network file (interfaces)] ******** | 10:26 |
| Martin___ | jvidal: did you run the deployment on a fresh system? That was very important for me ;) | 10:26 |
| jvidal | fatal: [controller00_nova_api_placement_container-36bb4105]: FAILED! => {"failed": true, "msg": "Failed to get information on remote file (/etc/network/interfaces.d/eth1.cfg): No passwd entry for user 'broot'\n"} | 10:26 |
| *** udesale has quit IRC | 10:27 | |
| jvidal | Martin___: yes, we reset the whole environment when deploying | 10:27 |
| Martin___ | jvidal: who is user 'broot'? | 10:27 |
| jvidal | I rebased OSA a couple of days ago, and it started | 10:27 |
| jvidal | Martin___: good question. I assume it should be 'root' | 10:27 |
| odyssey4me | andymccr Martin___ it's created here: https://github.com/openstack/openstack-ansible/blob/master/playbooks/repo-build.yml#L24 | 10:28 |
| jvidal | Martin___: but looked at the taskfile and looks ok: https://github.com/openstack/openstack-ansible-lxc_container_create/blob/master/tasks/container_create.yml#L314 | 10:28 |
| andymccr | odyssey4me: haha was just about to say that | 10:28 |
| odyssey4me | we group the servers by arch, then use the first server in each group to create the 'repo_masters' group, then target the build at that group | 10:28 |
| odyssey4me | jvidal did you rebootstrap ansible, because that issue has been solved in the latest tags | 10:29 |
| odyssey4me | but when you change your git checkout, you always have to update the roles too | 10:29 |
| andymccr | Martin___: so something is going wrong with that task odyssey4me linked - that means the group doesnt exist, so its failing. | 10:30 |
| admin0 | jvidal, i also recently setup OSA + OVS .. i noticed the same issue as well.. but found out that though it expects the bridge to be LB, it also maps to the actual ovs bridge and that error was just due to the LB command failing | 10:31 |
| jvidal | admin0: I am using ovs as well | 10:31 |
| admin0 | what I did was we left the controllers in LB and then setup OVS only in the compute nodes, and moved the agents to metal on compute node | 10:31 |
| admin0 | so all of my compute node are also network nodes (better for load distribution ) | 10:32 |
| andymccr | as a side note on the whole OVS thing, if an OVS expert wants to give a go at adding a proper OVS scenario test for nova or so - that would be really amazing *nudge nudge* | 10:32 |
| andymccr | (and some docs) | 10:32 |
| admin0 | ^^ | 10:32 |
| admin0 | on its way actually | 10:32 |
| admin0 | i have sections that i have prepared for ovs and also for cinder + qnap | 10:33 |
| jvidal | odyssey4me: yes, I rebootstraped OSA, roles are recent as well (27/06) | 10:34 |
| andymccr | admin0: thats awesome | 10:34 |
| jvidal | oddyssey4me: do I need to get a newer role from lxc_containers_create? | 10:35 |
| *** cmart has quit IRC | 10:35 | |
| admin0 | right now, struggling over cluster_metrics .. dashboard is up, grafana is up , influx is mapped , just that the data is not there :D | 10:36 |
| *** udesale has joined #openstack-ansible | 10:37 | |
| *** udesale__ has quit IRC | 10:38 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Revert "Only sync glance database on major versions" https://review.openstack.org/482898 | 10:41 |
| Martin___ | andymmcr odyssey4me I found the problem: TASK [Group repo servers by architecture and os version] returns repo_servers_16.10_armv7l | 10:42 |
| *** yifei has joined #openstack-ansible | 10:42 | |
| Martin___ | That is supposedly because I changed the ubuntu repos to the yakkety ones as a workaround for some python package conflict I encoutnered | 10:42 |
| Martin___ | So I have to think on how to combine this | 10:43 |
| Martin___ | andymccr odyssey4me Thanks for the hints :) | 10:43 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Revert "Only sync glance database on major versions" https://review.openstack.org/482898 | 10:43 |
| andymccr | Martin___: ahh | 10:43 |
| *** udesale has quit IRC | 10:45 | |
| *** cshen_ has quit IRC | 10:50 | |
| *** cmart has joined #openstack-ansible | 10:51 | |
| *** stuartgr has joined #openstack-ansible | 10:56 | |
| *** markvoelker has quit IRC | 11:09 | |
| *** markvoelker has joined #openstack-ansible | 11:10 | |
| odyssey4me | andymccr unfortunately in minor upgrade tests I'm seeing failures for glance, so I've pushed https://review.openstack.org/482898 as the patch in the role appears to have done nothing more but create complication | 11:11 |
| *** markvoelker has quit IRC | 11:14 | |
| andymccr | odyssey4me: ok | 11:14 |
| andymccr | you mean the upgrade jobs are failing on glance, or you mean there are api-test failures? (i noticed those) | 11:15 |
| odyssey4me | the api test fails are expected, because there's only one glance container | 11:15 |
| odyssey4me | but a minor upgrade implementation fails | 11:15 |
| andymccr | ok makes sense | 11:16 |
| odyssey4me | one part of the failure is due to us (in RPC) using a funny version tag which does not comply with semver, so the version_compare fails | 11:16 |
| odyssey4me | another part of the failure is that the test for the major version will never ever detect a major version | 11:16 |
| odyssey4me | because it's comparing the venv tag against the same venv tag | 11:16 |
| odyssey4me | see in https://github.com/openstack/openstack-ansible-os_glance/blob/master/tasks/glance_db_setup.yml#L29 'existing' is set to the value of glance_venv_tag | 11:17 |
| odyssey4me | then the values for existing are compared later to the value of glance_venv_tag | 11:18 |
| odyssey4me | it makes absolutely no sense | 11:18 |
| *** smatzek has joined #openstack-ansible | 11:18 | |
| odyssey4me | so basically what's happening in there is that 'upgrade_type' always = 'major', so the db sync is always run | 11:19 |
| odyssey4me | so we may as well remove all that stuff | 11:20 |
| odyssey4me | I put in a patch to fix the logic, but I realised that the logic is pointless. | 11:21 |
| andymccr | yeah. ok - i mean upgrades are complicated i guess, so its a fine line - but obviously if its not doing the right thing that sucks :P | 11:21 |
| *** vnogin has quit IRC | 11:22 | |
| odyssey4me | I'd prefer to revert, then revisit if our further testing picks up that there is some sort of outage which we need to work around. | 11:22 |
| odyssey4me | Right now, though, there's no evidence I'm aware of that indicates that there is an outage when db sync is always run. | 11:22 |
| andymccr | fair enough - gonna get some lunch real quick! i'll look at bit closer then. but happy to revert based on the above. | 11:24 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance stable/ocata: Revert "Only sync glance database on major versions" https://review.openstack.org/482917 | 11:28 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance stable/newton: Revert "Only sync glance database on major versions" https://review.openstack.org/482919 | 11:28 |
| *** acormier has joined #openstack-ansible | 11:29 | |
| *** vnogin has joined #openstack-ansible | 11:30 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance stable/ocata: Revert "Only sync glance database on major versions" https://review.openstack.org/482917 | 11:31 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance stable/ocata: Revert "Only sync glance database on major versions" https://review.openstack.org/482917 | 11:31 |
| *** chhavi_ has quit IRC | 11:32 | |
| *** shardy is now known as shardy_lunch | 11:32 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_horizon master: tasks: horizon_apache: Use the apache2_module Ansible module https://review.openstack.org/482923 | 11:33 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_horizon master: Add support for the openSUSE Leap distributions https://review.openstack.org/482924 | 11:33 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance stable/newton: Revert "Only sync glance database on major versions" https://review.openstack.org/482919 | 11:33 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance stable/newton: Revert "Only sync glance database on major versions" https://review.openstack.org/482919 | 11:34 |
| *** acormier has quit IRC | 11:37 | |
| *** acormier has joined #openstack-ansible | 11:38 | |
| *** thorst has joined #openstack-ansible | 11:38 | |
| *** schwicht_at_work has joined #openstack-ansible | 11:42 | |
| *** acormier has quit IRC | 11:42 | |
| *** kristia__ has joined #openstack-ansible | 11:45 | |
| *** kristian__ has quit IRC | 11:47 | |
| *** yifei has quit IRC | 11:48 | |
| *** smatzek has quit IRC | 11:50 | |
| *** smatzek has joined #openstack-ansible | 11:53 | |
| *** markvoelker has joined #openstack-ansible | 11:55 | |
| *** vnogin has quit IRC | 11:56 | |
| *** vnogin has joined #openstack-ansible | 12:01 | |
| armaan | Is glance broken in stable/newton? | 12:05 |
| *** yifei has joined #openstack-ansible | 12:08 | |
| *** schwicht_at_work has quit IRC | 12:08 | |
| *** eoseq has joined #openstack-ansible | 12:08 | |
| *** shardy_lunch is now known as shardy | 12:10 | |
| *** coolkil has joined #openstack-ansible | 12:14 | |
| *** udesale has joined #openstack-ansible | 12:16 | |
| armaan | folks, any idea how to fix this http://paste.openstack.org/show/615146/ | 12:18 |
| mhayden | buenos dias a todos | 12:20 |
| mhayden | armaan: is this is a new deployment? an upgrade? | 12:21 |
| *** acormier has joined #openstack-ansible | 12:21 | |
| *** jamesden_ has joined #openstack-ansible | 12:21 | |
| armaan | mhayden: hello, new deployment running stable/newton | 12:21 |
| *** acormier has quit IRC | 12:22 | |
| *** tots has quit IRC | 12:22 | |
| *** acormier has joined #openstack-ansible | 12:22 | |
| mhayden | armaan: taking a look -- i just woke up and i'm just starting my first cup of coffee | 12:23 |
| *** schwicht has joined #openstack-ansible | 12:24 | |
| *** jamesdenton has quit IRC | 12:24 | |
| armaan | mhayden: Ah, coffee is lifesaver, I prefer Tea though :) | 12:24 |
| mhayden | hmm, those variables are in there | 12:24 |
| *** acormier has quit IRC | 12:27 | |
| odyssey4me | armaan haha, new bug | 12:28 |
| odyssey4me | lemme patch that up real quick | 12:28 |
| *** schwicht has quit IRC | 12:28 | |
| armaan | odyssey4me: ooh :/ | 12:29 |
| odyssey4me | you're using the head of stable/newton, right? | 12:29 |
| andymccr | how did that pass gate? | 12:31 |
| armaan | odyssey4me: Yes. | 12:32 |
| armaan | odyssey4me: Can i move it back to any other commit? | 12:32 |
| kristia__ | Hi, Im getting "Error: container ubuntu-xenial-amd64 is not defined" while deploying openstack-ansible aio | 12:32 |
| kristia__ | What shall I do? | 12:32 |
| coolkil | odyssey4me thnx for review of the code wil fix the remarks tonight or tommorow (been away for a few days) https://review.openstack.org/#/c/479844/ | 12:33 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/newton: Add missing group_vars for glance https://review.openstack.org/482952 | 12:33 |
| mhayden | odyssey4me: i was scratching my head on that one too -- i know palendae did some work in here lately | 12:33 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Add missing group_vars for glance https://review.openstack.org/482953 | 12:33 |
| odyssey4me | armaan ^ those patches will fix it | 12:34 |
| *** schwicht has joined #openstack-ansible | 12:34 | |
| odyssey4me | depending on the branch you're using | 12:34 |
| armaan | odyssey4me: Ah, thanks a lot :) | 12:36 |
| *** cathrich_ has joined #openstack-ansible | 12:37 | |
| eoseq | odyssey4me: priv message | 12:38 |
| *** vnogin has quit IRC | 12:38 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible master: Vars passed to openstack-ansible take precedence https://review.openstack.org/482440 | 12:38 |
| odyssey4me | eoseq anything you wish to ask can be asked in the channel, because more than just me could provide an answer | 12:38 |
| mhayden | odyssey4me: if you were a pokemon, which pokemon would you be | 12:39 |
| *** vnogin has joined #openstack-ansible | 12:40 | |
| *** stuartgr has quit IRC | 12:40 | |
| *** persia__ has joined #openstack-ansible | 12:40 | |
| *** Martin___ has quit IRC | 12:40 | |
| *** v1k0d3n_ has joined #openstack-ansible | 12:40 | |
| *** szaher has quit IRC | 12:40 | |
| odyssey4me | eoseq for https://bugs.launchpad.net/openstack-ansible/+bug/1669691 and https://bugs.launchpad.net/openstack-ansible/+bug/1691130 the request has been for more information about how to replicate the problem | 12:40 |
| openstack | Launchpad bug 1669691 in openstack-ansible "repo-install.yml failed in tasks of repo_pre_build.yml " [Undecided,Incomplete] - Assigned to Jesse Pretorius (jesse-pretorius) | 12:40 |
| openstack | Launchpad bug 1691130 in openstack-ansible "fails at repo_build : Build package requirements file" [Undecided,Incomplete] | 12:40 |
| *** stuartgr has joined #openstack-ansible | 12:40 | |
| *** zerick has quit IRC | 12:41 | |
| odyssey4me | in order to identify the cause, we need to understand how to replicate the issue | 12:41 |
| *** cNilesh has quit IRC | 12:41 | |
| *** aetaric_ has joined #openstack-ansible | 12:41 | |
| *** v1k0d3n has quit IRC | 12:41 | |
| *** aetaric has quit IRC | 12:41 | |
| *** persia has quit IRC | 12:41 | |
| *** cathrichardson has quit IRC | 12:41 | |
| *** galstrom_zzz has quit IRC | 12:41 | |
| *** eglute_s has quit IRC | 12:41 | |
| odyssey4me | please provide feedback or ask questions in the bug in question | 12:41 |
| *** v1k0d3n_ is now known as v1k0d3n | 12:41 | |
| *** zerick has joined #openstack-ansible | 12:41 | |
| *** cNilesh has joined #openstack-ansible | 12:41 | |
| *** eglute has joined #openstack-ansible | 12:42 | |
| *** galstrom_zzz has joined #openstack-ansible | 12:42 | |
| eoseq | odyssey4me: ok, thx | 12:42 |
| odyssey4me | eoseq either discuss in the channel here, or discuss in the bug itself | 12:43 |
| odyssey4me | private messages will be ignored | 12:43 |
| *** gkadam has quit IRC | 12:43 | |
| odyssey4me | we discuss everything within the community to ensure that everyone is able to respond, and everyone is able to learn | 12:44 |
| odyssey4me | also, all conversation in the channel is recorded in logs, giving people a record to go back to for reference | 12:44 |
| *** szaher has joined #openstack-ansible | 12:44 | |
| odyssey4me | private conversations are private and help no-one else | 12:44 |
| *** dgonzalez has quit IRC | 12:44 | |
| kristia__ | odyssey4me could you help me? | 12:44 |
| odyssey4me | kristia__ with regards to the missing container? | 12:45 |
| odyssey4me | is the container there? what have you done to reach that point? you're not really giving much information to go on | 12:45 |
| kristia__ | lxc-ls shows me nothing | 12:45 |
| odyssey4me | have you executed the setup--hosts playbook? | 12:46 |
| *** dgonzalez has joined #openstack-ansible | 12:46 | |
| odyssey4me | is it an AIO, have you modified the config | 12:46 |
| kristia__ | I did bootstrap-ansible, bootstrap-aio, edit configs (just ips), run-playbooks | 12:47 |
| odyssey4me | kristia__ right, so clearly something didn't finish | 12:48 |
| odyssey4me | or you changed something which broke a config | 12:48 |
| kristia__ | those bootstrap scripts finished successfully | 12:48 |
| odyssey4me | so you need to compare against a working config to figure it out | 12:49 |
| kristia__ | let me remove /etc/openstack_deploy dir and start again | 12:49 |
| kristia__ | I will change nothing there | 12:49 |
| *** woodard_ has quit IRC | 12:49 | |
| odyssey4me | I'm afraid I'm engaged with other work and can't help you step through every possibility. If you're changing the config, then obviously you should start with examining your changes. | 12:49 |
| odyssey4me | also, don't bother using run-playbooks | 12:50 |
| *** woodard has joined #openstack-ansible | 12:50 | |
| kristia__ | ok, so what instead of it? | 12:50 |
| odyssey4me | rather use the setup-hosts, setup-infrastructure and setup-openstack playbooks, then verify after each step that everything is OK | 12:50 |
| odyssey4me | run-playbooks is a gating tool | 12:50 |
| kristia__ | ok, I will | 12:51 |
| kristia__ | we will see | 12:51 |
| *** woodard has quit IRC | 12:54 | |
| *** schwicht has quit IRC | 12:56 | |
| *** woodard has joined #openstack-ansible | 12:58 | |
| *** woodard has quit IRC | 12:58 | |
| *** ricardoas has joined #openstack-ansible | 12:58 | |
| *** woodard has joined #openstack-ansible | 12:58 | |
| *** cmart has quit IRC | 13:01 | |
| *** cmart has joined #openstack-ansible | 13:07 | |
| *** schwicht has joined #openstack-ansible | 13:08 | |
| *** cmart has quit IRC | 13:12 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance stable/ocata: Revert "Only sync glance database on major versions" https://review.openstack.org/482917 | 13:13 |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance stable/newton: Revert "Only sync glance database on major versions" https://review.openstack.org/482919 | 13:13 |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: Revert "Only sync glance database on major versions" https://review.openstack.org/482898 | 13:13 |
| *** cNilesh has quit IRC | 13:15 | |
| *** kylek3h has joined #openstack-ansible | 13:15 | |
| *** tobberyd_ has joined #openstack-ansible | 13:18 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Include glance db sync revert https://review.openstack.org/482965 | 13:19 |
| *** chhavi_ has joined #openstack-ansible | 13:20 | |
| *** tobberydberg has quit IRC | 13:21 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/newton: Include glance db sync revert https://review.openstack.org/482966 | 13:23 |
| *** cathrichardson has joined #openstack-ansible | 13:30 | |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible master: Install swift serially for rolling updates https://review.openstack.org/478580 | 13:31 |
| *** cathrich_ has quit IRC | 13:32 | |
| *** woodard has quit IRC | 13:36 | |
| *** weezS has joined #openstack-ansible | 13:42 | |
| *** cmart has joined #openstack-ansible | 13:44 | |
| *** dxiri has joined #openstack-ansible | 13:45 | |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible master: Cleanly catch keyboard interrupts during tests https://review.openstack.org/481199 | 13:46 |
| *** schwicht has quit IRC | 13:46 | |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible master: Use python import machinery for inventory code https://review.openstack.org/481228 | 13:46 |
| *** dxiri has quit IRC | 13:47 | |
| *** schwicht has joined #openstack-ansible | 13:48 | |
| *** cpuga has joined #openstack-ansible | 13:49 | |
| *** vnogin has quit IRC | 13:50 | |
| *** cpuga_ has joined #openstack-ansible | 13:50 | |
| *** kristia__ has quit IRC | 13:52 | |
| *** cmart has quit IRC | 13:52 | |
| armaan | odyssey4me: Perhaps, another bug. During the keystone installation, apache server fails to start and playbook run fails with a timeout. | 13:52 |
| armaan | odyssey4me: However, It is possible to interrupt the playbook execution with ^Z, then ssh into the keystone container, start apache2, then continue the playbook, and it succeeds | 13:53 |
| odyssey4me | armaan better to figure out why it failed in the first place and solve that | 13:54 |
| *** cpuga has quit IRC | 13:54 | |
| armaan | odyssey4me: okay! | 13:54 |
| odyssey4me | can you share a debug log of the playbook run when it fails? | 13:54 |
| odyssey4me | or some sort of apache/keystone log showing why it failed? | 13:54 |
| jamesden_ | odyssey4me I'm actually looking at that issue, too. Appears to have been introduced around 15.1.6 | 13:55 |
| jamesden_ | the service is simply not being restarted in a task | 13:55 |
| odyssey4me | jamesden_ what do you mean? | 13:56 |
| *** klamath has joined #openstack-ansible | 13:56 | |
| *** klamath has quit IRC | 13:57 | |
| jamesden_ | During os-keystone-install, there's a task that stops the service, performs some actions, then the play fails on 'waiting for services to be up'. But nothing is triggering the service to start | 13:57 |
| *** klamath has joined #openstack-ansible | 13:58 | |
| odyssey4me | ah ok, lemme take a peek at that part then | 13:58 |
| *** deadnull has joined #openstack-ansible | 13:58 | |
| odyssey4me | that'd be in here: https://github.com/openstack/openstack-ansible-os_keystone/blob/stable/ocata/handlers/main.yml | 13:58 |
| jamesden_ | aye. That handler is not executing | 13:59 |
| odyssey4me | possibly https://github.com/openstack/openstack-ansible-os_keystone/blob/stable/ocata/handlers/main.yml#L83-L84 is missing from https://github.com/openstack/openstack-ansible-os_keystone/blob/stable/ocata/handlers/main.yml#L87 | 13:59 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-ops master: Do not exit if the REDEPLOY_EXTRA_SCRIPT is null or not set https://review.openstack.org/482716 | 13:59 |
| *** galstrom_zzz is now known as galstrom | 14:00 | |
| odyssey4me | yep, that seems sensible to me | 14:00 |
| *** deadnull has quit IRC | 14:01 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-ops master: Do not exit if the REDEPLOY_EXTRA_SCRIPT is null or not set https://review.openstack.org/482716 | 14:01 |
| *** dxiri has joined #openstack-ansible | 14:02 | |
| odyssey4me | oh hang on, that condition is there | 14:02 |
| *** kristian__ has joined #openstack-ansible | 14:04 | |
| odyssey4me | armaan jamesden_ are you seeing this on a fresh build or an upgrade? also is it a multi-node build or AIO ? how do I replicate it? | 14:04 |
| jamesden_ | So, i'm pretty sure i deployed a clean AIO with 15.1.6 the other day with no issue, but a subsequent os-keystone-install run post-install would replicate the issue | 14:05 |
| armaan | odyssey4me: sorry, i did n't see your message and nuked that environment. It was a fresh stable/newton build. | 14:05 |
| *** dxiri has quit IRC | 14:06 | |
| odyssey4me | ok, so it's a re-run that does it | 14:06 |
| odyssey4me | ok, lemme give it a bash | 14:06 |
| *** vnogin has joined #openstack-ansible | 14:10 | |
| *** kristian__ has quit IRC | 14:11 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-ops master: Do not exit if the REDEPLOY_EXTRA_SCRIPT is null or not set https://review.openstack.org/482716 | 14:11 |
| *** kristian__ has joined #openstack-ansible | 14:12 | |
| *** tobberyd_ has quit IRC | 14:12 | |
| *** vnogin has quit IRC | 14:12 | |
| mgariepy | morning everyone | 14:12 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Do not exit if the REDEPLOY_EXTRA_SCRIPT is null or not set https://review.openstack.org/482716 | 14:13 |
| *** tobberydberg has joined #openstack-ansible | 14:14 | |
| *** cshen__ has joined #openstack-ansible | 14:16 | |
| *** vnogin has joined #openstack-ansible | 14:16 | |
| *** acormier has joined #openstack-ansible | 14:16 | |
| *** cshen__ has quit IRC | 14:18 | |
| cloudnull | mornings | 14:18 |
| jamesden_ | hey cloudnull | 14:19 |
| cloudnull | o/ | 14:19 |
| cshen | morning. | 14:20 |
| *** cmart has joined #openstack-ansible | 14:22 | |
| *** tobberydberg has quit IRC | 14:25 | |
| *** Bico_Fino has joined #openstack-ansible | 14:25 | |
| *** tobberydberg has joined #openstack-ansible | 14:25 | |
| *** yifei has quit IRC | 14:26 | |
| *** woodard has joined #openstack-ansible | 14:28 | |
| kristian__ | Please could someone help me? Im getting this error while running setup-infrastructure playbook. Error: container ubuntu-xenial-amd64 is not defined | 14:29 |
| kristian__ | Im not very good at lxc | 14:29 |
| *** tobberydberg has quit IRC | 14:29 | |
| *** coolkil has quit IRC | 14:30 | |
| *** yifei has joined #openstack-ansible | 14:30 | |
| *** armaan has quit IRC | 14:30 | |
| odyssey4me | kristian__ did you run the setup-hosts playbook? that would have created it | 14:31 |
| *** armaan has joined #openstack-ansible | 14:31 | |
| *** tobberydberg has joined #openstack-ansible | 14:31 | |
| kristian__ | I did run it | 14:31 |
| kristian__ | but I can rerun it | 14:31 |
| odyssey4me | and it will only be needed if you're using overlayfs or a different copy-on-write back-end | 14:31 |
| odyssey4me | if you've modified the config, then you've not completed the changes to not require that container | 14:32 |
| odyssey4me | are you using an AIO? | 14:32 |
| kristian__ | Im using AIO, and only modified tunnel net settings in the config | 14:32 |
| kristian__ | and ext lb | 14:33 |
| *** tobberydberg has quit IRC | 14:33 | |
| odyssey4me | that doesn't make sense then | 14:33 |
| odyssey4me | somehow it thinks you want an overlayfs backing store for the containers | 14:34 |
| *** tobberydberg has joined #openstack-ansible | 14:34 | |
| odyssey4me | you can remove that by removing lxc_container_backing_store from user_variables.yml | 14:34 |
| kristian__ | is there a way to do cleanup without reinstall? | 14:34 |
| kristian__ | ok I will check it | 14:34 |
| *** gouthamr has joined #openstack-ansible | 14:35 | |
| odyssey4me | the AIO is a development tool, please bear that in mind - you should not just use its config verbatim for your environment unless you understand what the settings are for | 14:35 |
| odyssey4me | it is deliberately build for a very restricted, very insecure build | 14:35 |
| odyssey4me | resitricted in terms of memory/cpu utilisation, insecure in terms of how the networking is configured | 14:36 |
| kristian__ | for now its a testing deployment | 14:36 |
| *** udesale has quit IRC | 14:38 | |
| openstackgerrit | Miguel Alex Cantu (alextricity25) proposed openstack/openstack-ansible-ops master: Improve openstack-release file discovery https://review.openstack.org/482608 | 14:38 |
| *** ricardoas has quit IRC | 14:43 | |
| *** firebat has joined #openstack-ansible | 14:43 | |
| firebat | Hey is there an easy way to find definitions for the variables we can customize in user_variables.yml | 14:43 |
| *** tots has joined #openstack-ansible | 14:43 | |
| tots | hello should linuxbridge agent be running on all network hosts ? | 14:43 |
| jamesden_ | tots yes, most likely, especially if l3-agent or dhcp-agent are installed there | 14:44 |
| tots | i hve 3 networks hosts and linux bridge is installed only in 2 | 14:45 |
| *** vnogin has quit IRC | 14:45 | |
| jamesden_ | are all three defined in openstack_user_config.yml? | 14:45 |
| tots | of course they are yes jamesden_ | 14:46 |
| *** woodard has quit IRC | 14:53 | |
| *** woodard has joined #openstack-ansible | 14:54 | |
| *** phalmos has joined #openstack-ansible | 14:55 | |
| *** ricardoas has joined #openstack-ansible | 14:58 | |
| *** vnogin has joined #openstack-ansible | 14:59 | |
| *** vnogin has quit IRC | 15:00 | |
| *** marst has joined #openstack-ansible | 15:00 | |
| *** armaan has quit IRC | 15:01 | |
| *** vnogin has joined #openstack-ansible | 15:01 | |
| *** armaan has joined #openstack-ansible | 15:02 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Only gather facts when necessary https://review.openstack.org/483003 | 15:09 |
| *** tobberyd_ has joined #openstack-ansible | 15:11 | |
| openstackgerrit | Miguel Alex Cantu (alextricity25) proposed openstack/openstack-ansible-ops master: Improve openstack-release file discovery https://review.openstack.org/482608 | 15:12 |
| *** tobberydberg has quit IRC | 15:14 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove run-playbooks https://review.openstack.org/482253 | 15:14 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove run-playbooks https://review.openstack.org/482253 | 15:14 |
| *** tobberyd_ has quit IRC | 15:16 | |
| *** chyka has joined #openstack-ansible | 15:22 | |
| *** yifei has quit IRC | 15:23 | |
| openstackgerrit | Miguel Alex Cantu (alextricity25) proposed openstack/openstack-ansible-ops master: Improve openstack-release file discovery https://review.openstack.org/482608 | 15:24 |
| asura | Anyone know what service is running on port 35337 inside the glance-container? | 15:26 |
| asura | sorry port 35357 | 15:26 |
| openstackgerrit | Merged openstack/openstack-ansible stable/newton: Add missing group_vars for glance https://review.openstack.org/482952 | 15:26 |
| openstackgerrit | Miguel Alex Cantu (alextricity25) proposed openstack/openstack-ansible-ops master: Improve openstack-release file discovery https://review.openstack.org/482608 | 15:26 |
| *** chyka_ has joined #openstack-ansible | 15:26 | |
| andymccr | asura: inside the glance container? that's a keystone port but that shouldnt be in the glance container. | 15:27 |
| asura | Does it use apache for its service? | 15:27 |
| andymccr | asura: yeah, it uses apache with mod_wsgi (depending on which version) otherwise nginx /w uwsgi. | 15:28 |
| *** Oku_OS is now known as Oku_OS-away | 15:28 | |
| *** dxiri has joined #openstack-ansible | 15:28 | |
| *** chyka has quit IRC | 15:29 | |
| kristian__ | odyssey4me: Now network is not working in the container. I havent changed any networking settings to be sure. | 15:29 |
| kristian__ | what shall I do? | 15:29 |
| *** tobberydberg has joined #openstack-ansible | 15:30 | |
| odyssey4me | troubleshoot as you would any other host | 15:30 |
| openstackgerrit | Miguel Alex Cantu (alextricity25) proposed openstack/openstack-ansible-ops master: Improve openstack-release file discovery https://review.openstack.org/482608 | 15:30 |
| odyssey4me | there are details for how the networking all works in the deploy guide architecture and the appendices | 15:30 |
| kristian__ | Im in the container. Pinging google dns (8.8.8.8) failed | 15:30 |
| asura | Thanks, appears apache2 needed a restart for the playbook to execute the following task https://pastebin.com/uqdgwHgB I'm doing a fresh first install of OSA following the online manual | 15:30 |
| kristian__ | and no traceroute there | 15:31 |
| *** Soufiane68 has joined #openstack-ansible | 15:33 | |
| Soufiane68 | Hi | 15:33 |
| *** LiterateHawk has joined #openstack-ansible | 15:33 | |
| Soufiane68 | who ever deployed os-trove in openstack-ansible? | 15:33 |
| *** tobberydberg has quit IRC | 15:34 | |
| openstackgerrit | Miguel Alex Cantu (alextricity25) proposed openstack/openstack-ansible-ops master: Improve openstack-release file discovery https://review.openstack.org/482608 | 15:35 |
| asura | Any idea why my glance container is having an authorization error ? https://pastebin.com/6FTwFpTc | 15:35 |
| *** perniciouscaffei has joined #openstack-ansible | 15:35 | |
| *** schwicht has quit IRC | 15:36 | |
| openstackgerrit | Andy McCrae proposed openstack/ansible-hardening master: Conditionally install EPEL if needed https://review.openstack.org/480734 | 15:40 |
| *** schwicht has joined #openstack-ansible | 15:41 | |
| Soufiane68 | i got an error /var/log/trove/trove-taskmanager.log Error --> Fernet token <token > is not recognized | 15:42 |
| Soufiane68 | any ideas guys? | 15:42 |
| asura | root@deploy-glance-container-b862d6e6:~# curl http://172.29.236.11:35357/v3/auth/tokens curl: (52) Empty reply from server | 15:45 |
| asura | I'm able to open the port from inside the container | 15:45 |
| asura | So there must be some authentication somewhere not happening | 15:46 |
| firebat | Hey guys when I get to the keystone_service_setup.yml part of the deploy when it trys to wait for the keystone service to be up I get this error: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:590)>. Which when I curl is this error: gnutls_handshake() failed: An unexpected TLS packet was received. | 15:46 |
| firebat | Can anyone help me understand what's going on here? | 15:46 |
| jamesden_ | hey odyssey4me - Did you have a chance to replicate that keystone playbook failure? Looks OK on 15.1.5 here | 15:47 |
| odyssey4me | jamesden_ so I just built an ocata from the head of stable/ocata, ran a standard deploy, then re-ran the os-keystone-install playbook - no errors | 15:48 |
| odyssey4me | I'll do a config change now | 15:48 |
| jamesden_ | k | 15:48 |
| odyssey4me | heh, oh wait - there is an error | 15:48 |
| odyssey4me | firebat this is possibly the same issue | 15:49 |
| odyssey4me | ok, so it's not the handler, it's this one: https://github.com/openstack/openstack-ansible-os_keystone/blob/stable/ocata/tasks/keystone_service_setup.yml#L16 | 15:50 |
| Soufiane68 | i got an error /var/log/trove/trove-taskmanager.log Error --> Fernet token <token > is not recognized | 15:51 |
| Soufiane68 | any ideas for this error? | 15:51 |
| odyssey4me | hmm, that's curious - it appears that the handlers did not fire at all | 15:51 |
| lbragstad | Soufiane68: keystone will emit that error when the token can't be decrypted (e.g. it was tampered with) | 15:52 |
| openstackgerrit | Miguel Alex Cantu (alextricity25) proposed openstack/openstack-ansible-ops master: Improve openstack-release file discovery https://review.openstack.org/482608 | 15:53 |
| lbragstad | Soufiane68: https://github.com/openstack/keystone/blob/9b3d99ea24da0fa1c6257ecd633d1349a2a5fbe7/keystone/token/providers/fernet/token_formatters.py#L81-L94 | 15:53 |
| Soufiane68 | lbragstad I will have a look thx | 15:54 |
| Soufiane68 | lbragstad the thing is that I did not edit the default config provided by openstack-ansible os_trove playboojs | 15:55 |
| Soufiane68 | playbook* | 15:55 |
| odyssey4me | jamesden_ I'm not seeing these - are you? http://logs.openstack.org/24/481924/2/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial/eb9b1e8/console.html#_2017-07-10_12_28_36_597064 | 15:58 |
| jamesden_ | looking | 15:59 |
| *** jamesden_ has quit IRC | 15:59 | |
| *** jamesdenton has joined #openstack-ansible | 16:01 | |
| *** schwicht has quit IRC | 16:02 | |
| jamesdenton | i'm certainly not seeing them on the rerun of the playbook, but i don't have the logs from the initial install | 16:02 |
| odyssey4me | jamesdenton ok, now I understand why I wasn't seeing them - it was a re-run with no config change, so it makes sense | 16:03 |
| odyssey4me | lemme dig a little deeper then | 16:04 |
| *** esberglu has quit IRC | 16:04 | |
| odyssey4me | interesting, it seems like on the re-run apache was stopped for some reason | 16:05 |
| jamesdenton | yes | 16:06 |
| odyssey4me | something fired the notifier | 16:07 |
| openstackgerrit | Merged openstack/ansible-hardening master: Skip shadow checks for users w/o shadow data https://review.openstack.org/481220 | 16:07 |
| openstackgerrit | Merged openstack/ansible-hardening master: Avoid skipped package-related tasks https://review.openstack.org/478952 | 16:07 |
| asura | from the glance container should i be able to do something like, curl -d '{"auth":{"passwordCredentials":{"username": "admin","password": "ca5903c1110197f7add33ee33597c3fc3f8125c632"},"tenantName": "customer-A"}}' -H "Content-Type: application/json" http://172.29.236.11:35357/v3.0/tokens | 16:11 |
| openstackgerrit | Merged openstack/ansible-hardening master: Split long running tasks https://review.openstack.org/479383 | 16:11 |
| openstackgerrit | Merged openstack/ansible-hardening master: Correct the list of supported OS versions https://review.openstack.org/478630 | 16:11 |
| openstackgerrit | Merged openstack/ansible-hardening stable/ocata: tasks: rhel7stig: aide: Use 'aide -i' if 'aideinit' is not available https://review.openstack.org/478258 | 16:11 |
| openstackgerrit | Merged openstack/ansible-hardening stable/ocata: tasks: rhel7stig: aide: Fix conditionals for Ubuntu exclusions https://review.openstack.org/478260 | 16:11 |
| asura | Where the password is pulled from keystone_service_password /etc/openstack_deploy/user_secrets.yml | 16:12 |
| *** mbuil has quit IRC | 16:13 | |
| *** dxiri has quit IRC | 16:13 | |
| odyssey4me | jamesdenton well, ok, one problem found - keystone-manage db_sync --check only exists in Pike | 16:14 |
| odyssey4me | not sure how that passed all the gate checks, but that must come out | 16:14 |
| *** Soufiane68 has quit IRC | 16:14 | |
| odyssey4me | jamesdenton well, that is actually the cause, because it makes this happen: https://github.com/openstack/openstack-ansible-os_keystone/blob/stable/ocata/tasks/keystone_db_setup.yml#L46 | 16:16 |
| jamesdenton | i was just looking at that | 16:17 |
| odyssey4me | and it appears that we're not notifying the service to restart | 16:17 |
| odyssey4me | that's a bug | 16:17 |
| odyssey4me | will have a patch up shortly | 16:17 |
| jamesdenton | kk | 16:17 |
| *** esberglu has joined #openstack-ansible | 16:18 | |
| odyssey4me | thanks for raising it! | 16:18 |
| *** NewJorg has quit IRC | 16:18 | |
| jamesdenton | sure! thanks for digging in | 16:19 |
| openstackgerrit | Merged openstack/ansible-hardening master: Add equalto Jinja2 test for EL7 https://review.openstack.org/479318 | 16:19 |
| *** NewJorg has joined #openstack-ansible | 16:20 | |
| *** esberglu has quit IRC | 16:22 | |
| *** shardy has quit IRC | 16:25 | |
| *** esberglu has joined #openstack-ansible | 16:27 | |
| *** kristian__ has quit IRC | 16:29 | |
| firebat | odyssey4me: Should I chill until you push a patch or is this issue not related to mine? | 16:31 |
| odyssey4me | firebat not sure, but once I have a patch you can try with it | 16:32 |
| firebat | You pushing to ocata/stable or will I need to pull from your personal fork? | 16:32 |
| asura | ttyname failed: Inappropriate ioctl for device\nTraceback (most recent call last); https://pastebin.com/6FTwFpTc Does this mean that my tty device failed rather then an authentication issue? | 16:33 |
| asura | Wow, this playbook is really challenging... just following the user guide to do a basic install ... so many errors. Guess, I'll have to learn ansible to just do a basic install | 16:37 |
| *** cpuga_ is now known as cpuga | 16:38 | |
| *** vnogin has quit IRC | 16:39 | |
| *** dxiri has joined #openstack-ansible | 16:40 | |
| andymccr | asura: what are you trying to do? an AIO or a full deploy? if you're following the guide it should work - its been run through a few times :) although obviously would be glad to improve it if we can | 16:41 |
| andymccr | the issue itself in your log looks like it cant connect to your keystone service to setup the glance user. that could be because the keystone service isnt up, the haproxy loadbalancer isnt setup properly, there is on network access to that location, or some other connectivity reason | 16:43 |
| andymccr | *no network access rather | 16:43 |
| asura | Well the container can telnet to the ip and port | 16:44 |
| andymccr | asura: sure, but im guessing that ip is your haproxy service | 16:44 |
| andymccr | so if all the backends are down it will still be able to telnet to the ip and port | 16:44 |
| *** dxiri has quit IRC | 16:45 | |
| andymccr | which could mean haproxy is misconfigured, or the service inside the keystone containers is not up/running/not allowing access or some other reason | 16:45 |
| *** jamesden_ has joined #openstack-ansible | 16:48 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone stable/ocata: Replace db sync check with alternative process https://review.openstack.org/483033 | 16:49 |
| odyssey4me | jamesdenton ^ give that a whirl if you can please | 16:50 |
| odyssey4me | firebat ^ | 16:50 |
| *** jamesdenton has quit IRC | 16:50 | |
| odyssey4me | you'll need to restart your apache2 service yourself before re-running the playbook with that | 16:51 |
| jamesden_ | k | 16:51 |
| odyssey4me | alternatively, make a change in config | 16:51 |
| firebat | odyssey4me Thanks will do | 16:51 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone stable/newton: Replace db sync check with alternative process https://review.openstack.org/483034 | 16:52 |
| firebat | I'm rebuilding my infra host right now since I'm not using VMs I'll ping with an update when done | 16:52 |
| asura | The service inside keystone container would just be apache2? | 16:52 |
| andymccr | asura: it depends which version of OSA you're deploying, on stable branches it would be apache2 - but you could check with netstat to see if anything is listening on that port | 16:53 |
| *** dxiri has joined #openstack-ansible | 16:54 | |
| asura | So my keystone identity service isnt running? https://pastebin.com/0Xp6cqRM | 16:54 |
| *** pcaruana has quit IRC | 16:56 | |
| jamesden_ | asura If i'm not mistaken, we were running into the keystone/apache bug yesterday afternoon? | 16:56 |
| asura | Yes, I had to restart apache2 | 16:56 |
| asura | But, now it errors on glance container | 16:56 |
| asura | attempting to talk to keystone again | 16:56 |
| asura | I assume | 16:56 |
| asura | Actually, I can see it is listening on 35357 so maybe it is this HAPROXY thing andymccr is talking about | 16:57 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone master: Ensure that keystone restarts after db sync https://review.openstack.org/483036 | 16:58 |
| odyssey4me | asura for ocata/newton yes | 16:58 |
| asura | git clone -b 15.1.6 https://git.openstack.org/openstack/openstack-ansible /opt/openstack-ansible | 16:58 |
| asura | that is ocata? | 16:58 |
| jamesden_ | yes | 16:58 |
| asura | So this is a bug then? | 16:58 |
| jamesden_ | you can try to curl the service within the keystone container to verify a response. .11 is the VIP, so haproxy may be failing. healthcheck failing or something | 16:59 |
| *** kristian__ has joined #openstack-ansible | 16:59 | |
| *** armaan has quit IRC | 17:00 | |
| asura | Do i have to pass the keystone_auth_admin_password when doing the curl command? | 17:01 |
| asura | curl seems to work inside keystone container https://pastebin.com/6Lbu04ET | 17:01 |
| *** tots has quit IRC | 17:02 | |
| jamesden_ | odyssey4me https://etherpad.openstack.org/p/483033 | 17:02 |
| admin0 | when a new router is created, are all supposed to be active at the same time ? | 17:02 |
| jamesden_ | asura now try to curl https://172.29.236.11:35357 | 17:02 |
| jamesden_ | or http,rather | 17:02 |
| jamesden_ | admin0 that depends on the type of router | 17:02 |
| asura | yeah that one fails | 17:02 |
| admin0 | just did a create router ( no DVR in this setup ) | 17:03 |
| jamesden_ | asura Does https work by chance? | 17:03 |
| jamesden_ | admin0 In most cases a standalone router is created by default, so it would only be scheduled to a single l3 agent at a time | 17:04 |
| asura | if i pass --insecure it does | 17:04 |
| jamesden_ | both your external and internal vips were the same, weren't they? | 17:05 |
| odyssey4me | jamesden_ is that ocata or newton? | 17:05 |
| jamesden_ | in openstack_user_config? | 17:05 |
| asura | yes | 17:05 |
| jamesden_ | Change one | 17:05 |
| jamesden_ | .11 and .12 | 17:05 |
| admin0 | i think i remmber seeing 1 as active and rest as standby/backup | 17:05 |
| jamesden_ | odyssey4me ocata | 17:05 |
| admin0 | but here, i see 4 , all active, and none return a ping | 17:06 |
| jamesden_ | odyssey4me that's after applying the patch | 17:06 |
| jamesden_ | asura You'll want to rerun the haproxy playbook | 17:07 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone stable/newton: Replace db sync check with alternative process https://review.openstack.org/483034 | 17:07 |
| mhayden | lhinds: the jinja2 tests fixes merged into ansible-hardening in master | 17:07 |
| odyssey4me | jamesden_ odd https://github.com/openstack/openstack-ansible-os_keystone/blob/stable/ocata/handlers/main.yml#L52 | 17:08 |
| odyssey4me | it's clearly there | 17:08 |
| asura | Sorry, I've learned chef but I'm new to puppet I was on openstack-ansible setup-openstack.yml how to run just the haproxy playbook? | 17:08 |
| asura | new to ansible* | 17:08 |
| openstackgerrit | Major Hayden proposed openstack/ansible-hardening stable/ocata: Add equalto Jinja2 test for EL7 https://review.openstack.org/483038 | 17:09 |
| odyssey4me | asura inside setup-*.yml you'll see many other included playbooks | 17:09 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest stable/newton: Add block/always to tempest result file tasks https://review.openstack.org/482186 | 17:09 |
| admin0 | asura, openstack-ansible haproxy-TAB :D | 17:09 |
| odyssey4me | you can run any one of them individually | 17:09 |
| openstackgerrit | Merged openstack/openstack-ansible stable/newton: Include glance db sync revert https://review.openstack.org/482966 | 17:09 |
| jamesden_ | odyssey4me one sec, i think i patch it incorrectly :/ | 17:10 |
| asura | ran, openstack-ansible haproxy-install.yml it worked but openstack-ansible -vvvv setup-openstack.yml fails on TASK [Create DB for service] unable to connect to mysql server 172.29.236.11. I need to run the setup-hosts.yml and setup-infrastructure.yml playbooks again I think? | 17:13 |
| mhayden | so getty got a 'size' column recently -> http://imgur.com/a/vdu5w | 17:18 |
| mhayden | fancy | 17:18 |
| jamesden_ | odyssey4me sorry about the confusion, i had been mucking around with main.yml earlier and hadn't reverted it. I've run os-keystone-install.yml twice with success w/ the patch | 17:22 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_horizon master: tasks: horizon_apache: Use the apache2_module Ansible module https://review.openstack.org/482923 | 17:23 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_horizon master: Add support for the openSUSE Leap distributions https://review.openstack.org/482924 | 17:23 |
| odyssey4me | jamesden_ ok, great - thanks for testing! | 17:23 |
| odyssey4me | please feel free to +1 the patches with a comment about your test | 17:23 |
| jamesden_ | will do | 17:23 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: test-setup-swifthosts: Add -K option to mkfs.xfs https://review.openstack.org/482835 | 17:25 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible master: Restart glance services only when necessary https://review.openstack.org/482727 | 17:25 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible master: Support cidr_networks in L3 network environments https://review.openstack.org/448850 | 17:25 |
| odyssey4me | hmm, the ceph jobs are failing a little too often for my liking | 17:31 |
| *** kristian__ has quit IRC | 17:32 | |
| logan- | timeout roulette? or actually failing | 17:33 |
| jmccrory | looks like missing rbd libraries http://logs.openstack.org/53/482953/1/check/gate-openstack-ansible-openstack-ansible-ceph-ubuntu-xenial/1c3fef5/logs/openstack/aio1-nova/nova-compute.log.txt.gz#_2017-07-12_16_45_48_831 | 17:35 |
| odyssey4me | logan- failing after tempest - the full run is often below an hour | 17:35 |
| odyssey4me | jmccrory ah, good catch | 17:36 |
| logan- | hmm interesting | 17:36 |
| odyssey4me | we've had that one pop up anecdotally, so I'm happy to see it come up in gating | 17:36 |
| odyssey4me | time to make it all work better :) | 17:36 |
| logan- | with how we deploy ceph the rbd library failure always occurs the first time nova-compute starts | 17:36 |
| logan- | on a greenfield | 17:36 |
| logan- | and then ceph_client runs and restarts nova-compute after dropping the libraries | 17:37 |
| odyssey4me | argh, that sounds terrible | 17:37 |
| odyssey4me | why don't we do the ceph client stuff in the respective roles (nova, glance, cinder) ? | 17:37 |
| evrardjp | we didn't have include roles before | 17:38 |
| evrardjp | but that sounds a good use case | 17:38 |
| evrardjp | maybe | 17:38 |
| logan- | this log doesn't look right though | 17:39 |
| logan- | nova-compute only starts once here | 17:39 |
| logan- | it never restarts a second time | 17:39 |
| *** schwicht has joined #openstack-ansible | 17:39 | |
| jmccrory | yeah, tempest error was 'no valid host' | 17:39 |
| logan- | http://logs.openstack.org/53/482953/1/check/gate-openstack-ansible-openstack-ansible-ceph-ubuntu-xenial/1c3fef5/console.html#_2017-07-12_16_48_04_077292 | 17:45 |
| logan- | nova log cuts off at this restart | 17:46 |
| logan- | based on the timestamps | 17:46 |
| logan- | http://logs.openstack.org/53/482953/1/check/gate-openstack-ansible-openstack-ansible-ceph-ubuntu-xenial/1c3fef5/logs/host/syslog.txt.gz#_Jul_12_16_48_08 | 17:48 |
| logan- | it does a reload instead of a restart apparently | 17:48 |
| odyssey4me | yup | 17:48 |
| odyssey4me | everything except the placement-api does a reload | 17:49 |
| logan- | that restart occurs in ceph_client, not os_nova btw | 17:49 |
| odyssey4me | oh, that's odd | 17:49 |
| odyssey4me | that doesn't make sense though, because the line before it is the fact gathering | 17:50 |
| logan- | https://github.com/openstack/openstack-ansible-ceph_client/blob/stable/ocata/handlers/main.yml | 17:50 |
| logan- | that's the task running at 2017-07-12 16:47:30.842213 | 17:50 |
| odyssey4me | which looks suspiciously like https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-nova-install.yml#L144 | 17:50 |
| logan- | oh you're right | 17:52 |
| logan- | http://logs.openstack.org/53/482953/1/check/gate-openstack-ansible-openstack-ansible-ceph-ubuntu-xenial/1c3fef5/logs/host/syslog.txt.gz#_Jul_12_16_47_34 | 17:52 |
| logan- | that is the restart | 17:52 |
| logan- | no useful output there i guess that would indicate why the service failed to start | 17:52 |
| odyssey4me | Started nova openstack service follows soon after | 17:53 |
| logan- | maybe reloading before nova-compute initializes is causing it to die | 17:54 |
| odyssey4me | could be - I guess the load time is longer | 17:55 |
| odyssey4me | we could perhaps do a nova-compute reload after all the other services are done | 17:56 |
| logan- | interesting, if you look at the first line of output in that log "Loaded VIF plugin class" | 17:56 |
| *** electrofelix has quit IRC | 17:57 | |
| logan- | you see it again at the bottom of the file when the ceph_client restart hits | 17:57 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat stable/ocata: Optimise role execution https://review.openstack.org/482862 | 17:59 |
| odyssey4me | why do we do ceph_client after the nova role? it seems like the wrong order | 17:59 |
| logan- | because the venv doesn't exist beforehand | 17:59 |
| logan- | so you can't link the libraries yet | 17:59 |
| odyssey4me | hmm | 18:00 |
| logan- | include_role is a good suggestion as evrardjp said | 18:00 |
| *** jamesden_ has quit IRC | 18:00 | |
| odyssey4me | it seems like it'd be smarter to let the ceph_client role just handle getting the ceph packages installed, then let the nova role do the symlink | 18:00 |
| logan- | maybe. but there's some gotchas on what the libraries are named depending on which ceph version to run | 18:01 |
| logan- | ceph version you run* | 18:01 |
| *** dxiri has quit IRC | 18:01 | |
| logan- | and the linking has to occur in glance and cinder also | 18:01 |
| *** jamesdenton has joined #openstack-ansible | 18:01 | |
| logan- | so you duplicate a lot of code if you do the linking in all 3 places | 18:01 |
| odyssey4me | a small price to play for detanggling all this | 18:01 |
| odyssey4me | or yeah, include role | 18:01 |
| *** dxiri has joined #openstack-ansible | 18:02 | |
| *** armaan has joined #openstack-ansible | 18:02 | |
| *** jvidal has quit IRC | 18:09 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon stable/ocata: Optimise the role execution https://review.openstack.org/482871 | 18:10 |
| *** chhavi_ has quit IRC | 18:14 | |
| openstackgerrit | Major Hayden proposed openstack/ansible-hardening master: Allow epel-release package name customization https://review.openstack.org/480737 | 18:14 |
| openstackgerrit | Major Hayden proposed openstack/ansible-hardening master: Re-organize defaults/main.yml https://review.openstack.org/480739 | 18:15 |
| mhayden | odyssey4me: would you mind giving this a looksee? https://review.openstack.org/480734 | 18:16 |
| odyssey4me | heh, andymccr schooled you! | 18:17 |
| mhayden | thanks, odyssey4me! :) | 18:18 |
| *** poopcat has joined #openstack-ansible | 18:19 | |
| openstackgerrit | Logan V proposed openstack/openstack-ansible-os_nova master: Include ceph_client role if needed https://review.openstack.org/483061 | 18:20 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ansible-hardening master: Updated from OpenStack Ansible Tests https://review.openstack.org/481281 | 18:21 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible master: Remove ceph_client role execution from Nova play https://review.openstack.org/483062 | 18:22 |
| *** albertcard1 has joined #openstack-ansible | 18:25 | |
| *** poopcat has quit IRC | 18:25 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone master: Ensure that keystone restarts after db sync https://review.openstack.org/483036 | 18:27 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove run-playbooks https://review.openstack.org/482253 | 18:30 |
| *** cshen_ has joined #openstack-ansible | 18:36 | |
| *** albertcard1 is now known as poopcat | 18:38 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove run-playbooks https://review.openstack.org/482253 | 18:41 |
| *** armaan has quit IRC | 18:41 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove run-playbooks https://review.openstack.org/482253 | 18:41 |
| openstackgerrit | Merged openstack/ansible-hardening master: Conditionally install EPEL if needed https://review.openstack.org/480734 | 18:41 |
| mhayden | woot | 18:42 |
| openstackgerrit | Merged openstack/ansible-hardening master: Allow epel-release package name customization https://review.openstack.org/480737 | 18:43 |
| openstackgerrit | Merged openstack/ansible-hardening master: Re-organize defaults/main.yml https://review.openstack.org/480739 | 18:44 |
| *** armaan has joined #openstack-ansible | 18:55 | |
| *** dxiri has quit IRC | 18:59 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-os_cinder master: Sort key-value pairs in backend dictionaries https://review.openstack.org/483070 | 18:59 |
| *** dxiri has joined #openstack-ansible | 18:59 | |
| odyssey4me | nice catch there jmccrory | 19:01 |
| odyssey4me | I've been thinking that perhaps we need to figure out a way to implement either a multi-node job, or a multi-container config for the AIO when doing the periodic job. | 19:02 |
| odyssey4me | That said, getting everything working with py3 is important too. | 19:03 |
| *** armaan has quit IRC | 19:04 | |
| odyssey4me | hmm, I'll have to figure out this keystone stuff tomorrow - clearly the mix of possible configs complicates things | 19:05 |
| odyssey4me | chat tomorrow all - have a great day! | 19:05 |
| *** perniciouscaffei has quit IRC | 19:06 | |
| spotz | night odyssey4me | 19:07 |
| LiterateHawk | Hi all! If I run lxc-container-destroy.yml then delete /etc/openstack_deploy/{ansible_facts,openstack_hostnames_ips.yml,openstack_inventory.json}, is that enough to have the next install run choose new IPs for all the containers? | 19:10 |
| logan- | yes LiterateHawk, that should do the trick | 19:11 |
| LiterateHawk | logan-, thanks! I've still not quite figured out the correct network configuration for this PoC, so I need to move subnets | 19:12 |
| openstackgerrit | Merged openstack/ansible-hardening master: Updated from OpenStack Ansible Tests https://review.openstack.org/481281 | 19:14 |
| *** armaan has joined #openstack-ansible | 19:17 | |
| *** kristian__ has joined #openstack-ansible | 19:19 | |
| *** hybridpollo has joined #openstack-ansible | 19:21 | |
| *** armaan has quit IRC | 19:27 | |
| firebat | odyssey4me Looks like your patch helped me get further in my build | 19:31 |
| firebat | Now I'm gettin an even weirder error though | 19:31 |
| firebat | glance_container_bind_mounts is 'undefined'. Can someone point me to a reference of this var and what should be there? | 19:31 |
| LiterateHawk | firebat, I hit that yesterday. I had to add a image_hosts section | 19:32 |
| firebat | LiterateHawk: To user_variables.yml? | 19:32 |
| LiterateHawk | firebat, to openstack_user_config.yml .. It's real similar to the cinder section. You can find an example in the prod config (IIRC) | 19:33 |
| DimGR | question , when you have 3 hosts under network_hosts , how do you find out which is the master neutron host ? if you want to make changes to neutron how do you know in which host to do it? | 19:33 |
| LiterateHawk | Pretty sure that since they share a database, it doesn't matter | 19:34 |
| firebat | Hmm so I have that section in my config already... I'm running glance in a container on top of a bare metal machine... should the IP in there be the IP of the lxc container? | 19:35 |
| LiterateHawk | That I don't know. I've just started working with OSA this week | 19:36 |
| firebat | Yeah I'm not far in myself and I'm not an ansible guy so it's been going a bit slow for me | 19:36 |
| LiterateHawk | Ditto | 19:37 |
| firebat | To answer your question I've been able to destroy the containers and run again without issue some of the time... but every now and then I have to rebuild my machine completely because some stuff doesn't get removed on the container host | 19:37 |
| LiterateHawk | Yeah, I'm trying to figure out what network configuration is correct, so periodically I have to kill the whole thing and start fresh | 19:39 |
| firebat | Ok so this is the error I get when I'm running setup-openstack.yml. I have a section for image_hosts in my openstack_user_config.yml so that's not my issue. Anyone have any idea what else this error could mean? https://pastebin.com/fxkujHVV | 19:40 |
| openstackgerrit | Major Hayden proposed openstack/ansible-hardening master: Install libpam-pwquality on Ubuntu https://review.openstack.org/483079 | 19:41 |
| *** cathrichardson has quit IRC | 19:43 | |
| *** cathrichardson has joined #openstack-ansible | 19:44 | |
| jamesdenton | DimGR There is no master hosts. All 3 services would be evenly load balanced. Simply interact using the client. All 3 consume the same database | 19:44 |
| DimGR | so if lets say you make a change in linux bridge agent in one host , the change will also happen in the others ? | 19:45 |
| jamesdenton | firebat which version are you using? | 19:45 |
| jamesdenton | DimGR If you are modifying config files by hand, they only apply to that particular node | 19:46 |
| DimGR | im asking because after a fresh install 2 of the 3 network hosts have a bunch of iptables rules but one host does not | 19:46 |
| firebat | jamesdenton: I'm on ocata... I think it's 15.1.2? | 19:46 |
| firebat | Well actually I'm on ocata/stable right now | 19:46 |
| jamesdenton | DimGR iptables may vary depending on what objects are scheduled to that particular node | 19:47 |
| DimGR | jamesdenton so if you want to enable flat in linux bridge agent how would you pass this in the user_variables ? | 19:47 |
| DimGR | up untill now i was editing the file on each host by hand :P | 19:48 |
| jamesdenton | typically network types like that are applied through openstack_user_config.yml. https://github.com/openstack/openstack-ansible/blob/stable/ocata/etc/openstack_deploy/openstack_user_config.yml.example#L281 | 19:48 |
| jamesdenton | So a working example might look like: https://gist.github.com/antonym/f43a967fac0f1c2bcf00f976295cba0f#file-gistfile1-txt-L45 | 19:50 |
| jamesdenton | Where he has specified the bond1 interface will be used for the flat network. The provider_mapping is called 'flat' | 19:51 |
| *** lucasxu has joined #openstack-ansible | 19:51 | |
| jamesdenton | in the neutron containers, a veth interface named 'eth12' will be used. computes will use bond1 | 19:51 |
| jamesdenton | there can only be one flat network per mapping. | 19:51 |
| DimGR | jamesdenton another question , one network host is also defined in compute hosts . that said host neutron installed in the system by the role and also in containers ( neutron agents and neutron server ) . if you want to see logs do you see logs from var/log/neutron or from the container's /var/log | 19:52 |
| *** cpuga has quit IRC | 19:52 | |
| *** cpuga has joined #openstack-ansible | 19:52 | |
| jamesdenton | Something like that would be allowed, but maybe not ideal in a prod setting | 19:53 |
| LiterateHawk | To allow OS to use the same network as the management interface for instances, is it as simple as adding another item in provider_networks with type 'flat' attached to the same br-mgmt network? Or do I need to add a 2nd bridge? | 20:05 |
| jamesdenton | firebat What you're experiencing may be a bug. The group_var glance_container_bind_mounts may not exist | 20:05 |
| firebat | Is there a way I can find out what that should be if it should exist? I can put it in myself | 20:06 |
| firebat | I was having issues with the tag I was on | 20:06 |
| jamesdenton | Do you have a directory named /opt/openstack-ansible/group_vars ? | 20:07 |
| *** lucasxu has quit IRC | 20:08 | |
| jamesdenton | and/or /opt/openstack-ansible/playbooks/common-playbooks? | 20:08 |
| DimGR | jamesdenton last question , if you create 5 instances at once and they build fine then the next day you create 5 instances at once and they fail with logs saying failed to bind port , where would you look at for a possible fix ? i think i have fixed that but i want to make sure its fixed | 20:10 |
| jamesdenton | LiterateHawk Things get real tricky when you try to collapse everything on the same interface. You can use a single interface, but because of all of the bridges involved you'd need to tag many of those networks, including the actual ssh/mgmt interface on the host if you want to have a 'flat' network for neutron to use | 20:11 |
| jamesdenton | port binding failures are often the result of an agent being able to perform the task you've asked it, so invalid network type or interface or whatever | 20:12 |
| *** dxiri has quit IRC | 20:12 | |
| DimGR | i think i had also seen logs like invalid block device mapping jamesdenton but i think i have fixed that | 20:15 |
| LiterateHawk | jamesdenton, Thanks for the pointer. Perhaps I'm doing the wrong thing. The machines have two nics, but the 2nd was brought up just for this PoC and is on a separate VLAN. I've got the 2nd nic configured for VXLAN tunneling, and now I'm trying to sort out how to attach the existing network on eth0 as well | 20:16 |
| *** cshen_ has quit IRC | 20:16 | |
| skape | Hi all ! while running "openstack-ansible setup-infrastructure.yml" I have an error in the task [Prepare group of master repo servers] but the thing that bothes me is that the error is in the localhost in the deploy machine | 20:16 |
| LiterateHawk | Unfortunately, this docs reshuffling on openstack.org is making a lot of promising links be dead :( | 20:16 |
| jamesdenton | LiterateHawk In our case, we have two NICs as well. eth0 (untagged) is management, eth0.100 is in br-mgmt bridge, and eth0.101 might be in br-storage. eth1.200 is in br-vxlan for tunnels, and eth1 (untagged) is in br-vlan, which allows Neutron to handle the actual vlan tagging | 20:17 |
| LiterateHawk | jamesdenton, Gotcha. I was trying to dodge adding additional VLANs the the existing network, because of the hassle of change-management (especially since I don't 100% know what changes I want them to do). | 20:19 |
| LiterateHawk | Is there a huge difference in OSA support between ovs and linuxbridge networking? | 20:20 |
| DimGR | i can verify LiterateHawk that networking is the most hard thing to understand in openstack let alone configuring it | 20:20 |
| jamesdenton | i hear ya. If you can work to have a dedicated set of vlans available for this PoC (say 10), and have them trunked to both eth0 and eth1 in this case, it gives you flexibility to move them around as needed and reserve some for neutron | 20:20 |
| LiterateHawk | Let me see if I can get at least that. The dumb thing is that this PoC is on a rack that's EOL'd with nothing production on it, but I still have to jump through the same hoops to get any changes :/ | 20:21 |
| jamesdenton | otherwisei feel your pain | 20:21 |
| LiterateHawk | DimGR, it absolutely is, and I'm not even looking to do anything real special | 20:22 |
| DimGR | it took me 4 months to undersand flat networking | 20:22 |
| jamesdenton | Does this help at all? | 20:22 |
| jamesdenton | https://docs.openstack.org/project-deploy-guide/openstack-ansible/newton/app-networking.html#network-appendix | 20:22 |
| LiterateHawk | I've read through that a couple of times. I think I get it conceptually, it's just the concepts that trip me up | 20:23 |
| LiterateHawk | s/concepts/details/ | 20:23 |
| jamesdenton | Do you have a subscription to O'Reilly or anything like that? | 20:24 |
| DimGR | what book you recomment jamesdenton | 20:24 |
| LiterateHawk | Yeah, I work for a university, so we have literally every subscription | 20:25 |
| jamesdenton | Well, at the sake of self-promotion, I recommend Learning OpenStack Networking (Second Edition), or at least Essentials | 20:25 |
| DimGR | ok LiterateHawk you can email me the book he will recommend :) | 20:25 |
| *** dxiri has joined #openstack-ansible | 20:26 | |
| LiterateHawk | jamesdenton, I'll grab them. Thanks for the tips! | 20:26 |
| jamesdenton | sure thing. i will also try and work on a post to break down the provider_network section of the openstack_user_config. | 20:27 |
| skape | thanks for the tips on the books I hope it end the guess work around flat network | 20:28 |
| *** armaan has joined #openstack-ansible | 20:28 | |
| jamesdenton | it's really geared towards vanilla Neutron. The OSA bits manipulate those config files accordingly. So yes, it helps understand the foundation | 20:29 |
| skape | but back to my question it seams that repo_servers is empty | 20:31 |
| skape | http://paste.openstack.org/show/615209/ | 20:31 |
| skape | why is the error on the localhost on the deploy machine | 20:31 |
| jamesdenton | i'm not sure 16.10 is a supported platform, yet | 20:34 |
| LiterateHawk | Yeah, you have to back up to 16.04 (learned that lesson the hard way) | 20:34 |
| skape | hum also on the deploy machine it must be 16.4 | 20:35 |
| skape | hum what exactly is been installed on the deploy machine ? | 20:36 |
| jamesdenton | yes, it looks like it does | 20:38 |
| jamesdenton | i'm not sure what would be installed on a pure deploy host, but it may be part of the inventory anyway and caught up in process | 20:39 |
| skape | ok in theory I can shutdown the deploy host after the installation ? | 20:41 |
| DimGR | is there a tutorial on how to shutdown your cluster? | 20:41 |
| skape | thank god the kickstart files work for 16.10 also work for 16.4 | 20:41 |
| jamesdenton | skape I suppose so, but you'll want it to make configuration changes along the way if needed | 20:42 |
| jamesdenton | ideally you would modify user_variables.yml and run a subset of playbooks and tags to manage the openstack infra | 20:42 |
| jamesdenton | DimGR May want to check the appendix in the online guide for bootstrapping a galera cluster on startup | 20:43 |
| skape | jamesdenton thx | 20:43 |
| jamesdenton | but shutdown should be pretty straightforward in my experience. save the DBs for last i guess | 20:43 |
| DimGR | so there is no particular order on shutting them down or rebooting a host but if you shutdown all hosts or reboot one of them, you need to check galera | 20:44 |
| DimGR | i see | 20:44 |
| jamesdenton | well the infra nodes are special in that case. rolling reboots are recommended | 20:44 |
| jamesdenton | and startup may have a sequence | 20:44 |
| DimGR | the sequence i am looking to find | 20:46 |
| DimGR | just to be safe when time comes | 20:46 |
| *** smatzek has quit IRC | 20:49 | |
| jamesdenton | Sorry DimGR - I don't have anything like that available. You may want to dig up some standard openstack best practices, as they would apply here as well | 20:52 |
| skape | I know that a ceph cluster has an shutdown order , so maybe swift has it also to avoid data inconstancy | 20:57 |
| cloudnull | anyone encounter this in master recently http://paste.openstack.org/show/615212/ ? | 20:59 |
| cloudnull | looks like py 3.x related | 20:59 |
| cloudnull | curious if there's a fix already in flight ? | 20:59 |
| cloudnull | jmccrory: | 20:59 |
| cloudnull | ^ | 20:59 |
| jmccrory | cloudnull: hmm i havent seen that before | 21:05 |
| cloudnull | it may just be my env | 21:06 |
| cloudnull | its a test running from tox | 21:06 |
| cloudnull | which is using py2 | 21:06 |
| asura | jamesden_ I've changed external and internal vips in openstack_user_config, but I'm not getting an error on Task pip_install : Install pip packages (fall back mode) https://pastebin.com/QwXg7t0M | 21:09 |
| *** galstrom is now known as galstrom_zzz | 21:09 | |
| *** pbandark has quit IRC | 21:16 | |
| *** kylek3h has quit IRC | 21:19 | |
| *** phalmos has quit IRC | 21:21 | |
| jamesdenton | asura make sure both vip addresses are configured on an interface, br-mgmt would prob be OK for both. You can curl the url in the error to ensure connectivity at that point. the repo containers must be up and services accessible within the container | 21:22 |
| jamesdenton | the environment may be in a wonky state after the various changes that have been made, unless you wiped and started over | 21:23 |
| *** thorst has quit IRC | 21:23 | |
| *** woodard_ has joined #openstack-ansible | 21:24 | |
| *** woodard has quit IRC | 21:27 | |
| *** woodard_ has quit IRC | 21:29 | |
| *** cpuga has quit IRC | 21:30 | |
| *** ivve has quit IRC | 21:37 | |
| cloudnull | jmccrory: so yea it looks like if tox w/ py2 calls the ansible-runtime and it's using py3 things get all confused. | 21:37 |
| cloudnull | I'll have to see if this is just something in my test env or if its something upstream I can resolve | 21:37 |
| jmccrory | cloudnull : maybe ansible_python_interpreter ? there's a few reviews out like https://review.openstack.org/#/c/482090/ | 21:44 |
| *** armaan has quit IRC | 21:53 | |
| *** schwicht has quit IRC | 21:54 | |
| cloudnull | maybe | 21:54 |
| *** klamath has quit IRC | 21:54 | |
| cloudnull | ill look into that too | 21:54 |
| *** weezS has quit IRC | 21:58 | |
| *** galstrom_zzz is now known as galstrom | 22:05 | |
| *** esberglu has quit IRC | 22:11 | |
| *** agrebennikov has joined #openstack-ansible | 22:11 | |
| *** jamesdenton has quit IRC | 22:14 | |
| *** jamesdenton has joined #openstack-ansible | 22:14 | |
| *** jamesdenton has quit IRC | 22:19 | |
| *** schwicht has joined #openstack-ansible | 22:25 | |
| *** kristian__ has quit IRC | 22:26 | |
| *** kristian__ has joined #openstack-ansible | 22:26 | |
| *** woodard has joined #openstack-ansible | 22:26 | |
| *** albertcard1 has joined #openstack-ansible | 22:31 | |
| *** LiterateHawk has quit IRC | 22:31 | |
| *** d34dh0r53 has quit IRC | 22:32 | |
| *** kristian__ has quit IRC | 22:32 | |
| *** d34dh0r53 has joined #openstack-ansible | 22:32 | |
| *** admin0 has quit IRC | 22:32 | |
| *** poopcat has quit IRC | 22:32 | |
| *** admin0 has joined #openstack-ansible | 22:32 | |
| *** raginbajin has quit IRC | 22:35 | |
| *** flart has quit IRC | 22:35 | |
| *** flart has joined #openstack-ansible | 22:35 | |
| *** szaher has quit IRC | 22:37 | |
| *** szaher has joined #openstack-ansible | 22:37 | |
| *** raginbajin has joined #openstack-ansible | 22:42 | |
| *** thorst has joined #openstack-ansible | 22:45 | |
| *** thorst has quit IRC | 22:45 | |
| *** schwicht has quit IRC | 22:46 | |
| *** schwicht has joined #openstack-ansible | 22:51 | |
| *** schwicht has quit IRC | 22:54 | |
| *** acormier has quit IRC | 22:58 | |
| *** acormier has joined #openstack-ansible | 22:59 | |
| *** admin0 has quit IRC | 22:59 | |
| *** admin0 has joined #openstack-ansible | 22:59 | |
| *** acormier has quit IRC | 23:03 | |
| *** galstrom is now known as galstrom_zzz | 23:06 | |
| *** admin0 has quit IRC | 23:19 | |
| *** schwicht has joined #openstack-ansible | 23:26 | |
| *** chyka_ has quit IRC | 23:40 | |
| openstackgerrit | Logan V proposed openstack/openstack-ansible-os_nova master: Include ceph_client role if needed https://review.openstack.org/483061 | 23:40 |
| *** chyka has joined #openstack-ansible | 23:41 | |
| *** marst has quit IRC | 23:41 | |
| *** chyka has quit IRC | 23:45 | |
| *** agrebennikov has quit IRC | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!