| *** poopcat has quit IRC | 00:06 | |
| *** tux_ has joined #openstack-ansible | 00:07 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 00:08 |
|---|---|---|
| *** markvoelker has quit IRC | 00:25 | |
| *** vnogin has joined #openstack-ansible | 00:35 | |
| *** cshen has joined #openstack-ansible | 00:38 | |
| *** cshen has quit IRC | 00:42 | |
| *** aludwar has quit IRC | 00:52 | |
| *** aludwar has joined #openstack-ansible | 00:52 | |
| *** gregoryo has joined #openstack-ansible | 01:10 | |
| logan- | your job needs a 'parent' cloudnull | 01:20 |
| cloudnull | oh good cal. | 01:20 |
| logan- | id suggest parent: openstack-ansible-base | 01:20 |
| cloudnull | done. | 01:22 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 01:24 |
| logan- | post -> post-run | 01:25 |
| cloudnull | done. | 01:25 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 01:26 |
| logan- | it queued | 01:26 |
| logan- | :) | 01:26 |
| cloudnull | winning! | 01:26 |
| logan- | oh nvm | 01:26 |
| cloudnull | hahaha. | 01:26 |
| logan- | still says 'post' int he job def | 01:27 |
| cloudnull | ah. | 01:28 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 01:28 |
| cloudnull | i renamed the play, not the job key | 01:28 |
| cloudnull | hahaha | 01:28 |
| logan- | haha | 01:29 |
| logan- | worked this time | 01:29 |
| cloudnull | now we see it all break in the gate :P | 01:30 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible-ops master: [DNM] Test commit should not trigger ELK job https://review.openstack.org/583807 | 01:31 |
| logan- | it didnt queue the elk job for ^ so i think you're gtg | 01:32 |
| cloudnull | woot! | 01:38 |
| cloudnull | this will be amazing if it works :) | 01:39 |
| logan- | immediately after you said that it failed lol | 01:39 |
| logan- | 1/2 way there.. the job works at least | 01:40 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 01:43 |
| cloudnull | its running at least and it looks like the wiring it working so all winning in my book! | 01:43 |
| *** evrardjp has quit IRC | 01:44 | |
| openstackgerrit | weizj proposed openstack/ansible-hardening master: Delete the unnecessary space https://review.openstack.org/583808 | 01:47 |
| *** vnogin has quit IRC | 01:49 | |
| *** evrardjp has joined #openstack-ansible | 01:49 | |
| *** tux_ has quit IRC | 01:54 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 01:54 |
| *** evrardjp has quit IRC | 02:02 | |
| *** evrardjp has joined #openstack-ansible | 02:07 | |
| *** tux_ has joined #openstack-ansible | 02:10 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 02:23 |
| *** markvoelker has joined #openstack-ansible | 02:26 | |
| *** cshen has joined #openstack-ansible | 02:30 | |
| *** cshen has quit IRC | 02:34 | |
| *** dave-mcc_ has quit IRC | 02:45 | |
| *** FrankZhang has quit IRC | 02:53 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 02:56 |
| *** markvoelker has quit IRC | 03:00 | |
| *** m|y|k has joined #openstack-ansible | 03:03 | |
| *** m|y|k has quit IRC | 03:03 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 03:04 |
| *** kstev has quit IRC | 03:05 | |
| *** parasitid has quit IRC | 03:38 | |
| *** tux_ has quit IRC | 03:45 | |
| *** parasitid has joined #openstack-ansible | 03:54 | |
| *** markvoelker has joined #openstack-ansible | 03:58 | |
| *** udesale has joined #openstack-ansible | 04:02 | |
| *** pcaruana has joined #openstack-ansible | 04:23 | |
| *** markvoelker has quit IRC | 04:31 | |
| *** pcaruana has quit IRC | 04:34 | |
| *** cshen has joined #openstack-ansible | 04:36 | |
| *** mattinator has quit IRC | 04:39 | |
| *** cshen has quit IRC | 04:41 | |
| *** prasen has joined #openstack-ansible | 04:51 | |
| *** cshen has joined #openstack-ansible | 04:56 | |
| *** cshen has quit IRC | 05:00 | |
| *** mpjetta has quit IRC | 05:03 | |
| *** vnogin has joined #openstack-ansible | 05:06 | |
| *** Bhujay has joined #openstack-ansible | 05:12 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 05:14 |
| *** mpjetta has joined #openstack-ansible | 05:21 | |
| *** cshen has joined #openstack-ansible | 05:24 | |
| *** markvoelker has joined #openstack-ansible | 05:27 | |
| Taseer | evrardjp: only you are between this patch getting through ! => https://review.openstack.org/#/c/571388/ | 05:28 |
| *** wlmbasson_ has joined #openstack-ansible | 05:28 | |
| *** fdegir has quit IRC | 05:29 | |
| *** jamespage has quit IRC | 05:29 | |
| *** wlmbasson has quit IRC | 05:29 | |
| *** aetaric has quit IRC | 05:29 | |
| *** wlmbasson_ is now known as wlmbasson | 05:29 | |
| *** xdfil_ has joined #openstack-ansible | 05:29 | |
| *** jamespage has joined #openstack-ansible | 05:29 | |
| *** cshen has quit IRC | 05:29 | |
| *** xdfil has quit IRC | 05:29 | |
| *** johnsom has quit IRC | 05:29 | |
| *** jamespage has quit IRC | 05:29 | |
| *** jamespage has joined #openstack-ansible | 05:29 | |
| *** fdegir has joined #openstack-ansible | 05:29 | |
| *** xdfil_ is now known as xdfil | 05:29 | |
| *** serverascode_ has joined #openstack-ansible | 05:30 | |
| *** johnsom has joined #openstack-ansible | 05:30 | |
| *** portdirect_ has joined #openstack-ansible | 05:30 | |
| *** serverascode has quit IRC | 05:30 | |
| *** portdirect has quit IRC | 05:30 | |
| *** portdirect_ is now known as portdirect | 05:30 | |
| *** gregoryo has quit IRC | 05:31 | |
| *** hamzy has quit IRC | 05:31 | |
| *** flaviosr has quit IRC | 05:31 | |
| *** aspiers has quit IRC | 05:31 | |
| *** jmccrory has quit IRC | 05:31 | |
| *** portdirect is now known as Guest72952 | 05:32 | |
| *** jmccrory has joined #openstack-ansible | 05:35 | |
| *** hamzy has joined #openstack-ansible | 05:36 | |
| *** aspiers has joined #openstack-ansible | 05:37 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 05:44 |
| *** gregoryo has joined #openstack-ansible | 05:45 | |
| *** flaviosr has joined #openstack-ansible | 05:45 | |
| *** threestrands has quit IRC | 05:49 | |
| *** markvoelker has quit IRC | 06:01 | |
| *** nyloc has quit IRC | 06:04 | |
| *** nyloc has joined #openstack-ansible | 06:04 | |
| *** aetaric has joined #openstack-ansible | 06:05 | |
| openstackgerrit | weizj proposed openstack/openstack-ansible master: Optimise the annotate for keepalived.yml https://review.openstack.org/583864 | 06:08 |
| *** threestrands has joined #openstack-ansible | 06:09 | |
| *** threestrands has quit IRC | 06:09 | |
| *** threestrands has joined #openstack-ansible | 06:09 | |
| evrardjp[m] | Taseer: i will review. I am not the only one to be able to review, no. :p | 06:11 |
| evrardjp | cloudnull: why did you re-merge the addition of pyldap ? | 06:15 |
| evrardjp | https://review.openstack.org/#/c/573508/5 | 06:15 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_keystone master: Pull in ldap dependencies via ldappool https://review.openstack.org/583865 | 06:17 |
| *** tomcsi_ has quit IRC | 06:19 | |
| *** jwitko has quit IRC | 06:22 | |
| *** udesale_ has joined #openstack-ansible | 06:23 | |
| *** udesale has quit IRC | 06:25 | |
| *** gkadam has joined #openstack-ansible | 06:26 | |
| *** tomcsi has joined #openstack-ansible | 06:27 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_keystone master: Pull in ldap dependencies via ldappool https://review.openstack.org/583865 | 06:35 |
| evrardjp | cloudnull: please vote on https://review.openstack.org/#/c/583865/ | 06:35 |
| *** threestrands has quit IRC | 06:36 | |
| *** chyka has joined #openstack-ansible | 06:37 | |
| *** pcaruana has joined #openstack-ansible | 06:37 | |
| *** gkadam has quit IRC | 06:38 | |
| evrardjp | Taseer: I am standing between your patch and merging anymore, the quality is good enough now. | 06:38 |
| evrardjp | :) | 06:38 |
| evrardjp | I can see further improvements, but it's a "good enough" patch for me. | 06:38 |
| Taseer | evrardjp: Thanks ! | 06:39 |
| Taseer | and suddenly there are other people as well ! | 06:39 |
| evrardjp | Dmitriy: No need for redefining the opensuse 15 and bionic -nv | 06:41 |
| evrardjp | I will remove it because it masks things | 06:42 |
| evrardjp | see the template here: https://github.com/openstack/openstack-ansible-tests/blob/master/zuul.d/project-templates.yaml | 06:42 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_masakari master: Added required defaults, env example, Manage LB hook https://review.openstack.org/581459 | 06:42 |
| *** chyka has quit IRC | 06:42 | |
| openstackgerrit | Taseer Ahmed proposed openstack/openstack-ansible-os_blazar master: This patch ensures that the functional tests for Blazar pass. https://review.openstack.org/571388 | 06:43 |
| evrardjp | Dmitriy: on top of it you can see patchset 8 got a success. So I am reverting to it. | 06:43 |
| evrardjp | Taseer: that's the power of the community :) | 06:45 |
| Taseer | haha ! | 06:46 |
| *** cshen has joined #openstack-ansible | 06:49 | |
| *** udesale__ has joined #openstack-ansible | 06:51 | |
| openstackgerrit | Christian Zunker proposed openstack/openstack-ansible master: Example setup for gnocchi with ceph and redis https://review.openstack.org/582946 | 06:52 |
| *** udesale_ has quit IRC | 06:54 | |
| *** markvoelker has joined #openstack-ansible | 06:58 | |
| *** openstackgerrit has quit IRC | 07:04 | |
| *** poul has joined #openstack-ansible | 07:04 | |
| *** shardy has joined #openstack-ansible | 07:13 | |
| *** shardy has quit IRC | 07:13 | |
| *** shardy has joined #openstack-ansible | 07:13 | |
| *** rmart04 has joined #openstack-ansible | 07:14 | |
| *** admin0 has joined #openstack-ansible | 07:15 | |
| odyssey4me | o/ | 07:18 |
| admin0 | \o | 07:19 |
| admin0 | odyssey4me, this were 2 ways on how to specify vars as told: http://paste.openstack.org/raw/726234/ .. the 1st method is not working .. | 07:20 |
| admin0 | do you know of a good method | 07:20 |
| admin0 | the issue is in our full docs on ceph https://docs.openstack.org/openstack-ansible/latest/user/ceph/full-deploy.html , nowhere is mentioned how to specify the disks on the osd hosts and i am trying to find a good method for this | 07:21 |
| odyssey4me | admin0 which series? | 07:21 |
| admin0 | 17.0.9 | 07:21 |
| admin0 | 17.0.6 | 07:21 |
| odyssey4me | OK, have you placed an inventory.ini file into /etc/openstack_deploy? | 07:22 |
| admin0 | this is the first time i am hearning the existence of this file | 07:22 |
| odyssey4me | I don't think it needs anything in it. | 07:22 |
| odyssey4me | But I think that's required for the host_vars/group_vars in user-space to work. | 07:23 |
| odyssey4me | evrardjp: ^ ? | 07:23 |
| admin0 | so touch inventory.ini and thats it :) ? | 07:23 |
| odyssey4me | yes I think so | 07:23 |
| admin0 | trying now | 07:23 |
| odyssey4me | there's reference to it in https://github.com/openstack/openstack-ansible/blob/stable/queens/scripts/openstack-ansible.rc#L17 | 07:23 |
| odyssey4me | but yeah, I don't think it's documented :/ | 07:23 |
| admin0 | hmm.. i see it created it .. inventory.ini with the line [all] | 07:23 |
| odyssey4me | yeah, that's fine | 07:24 |
| odyssey4me | with that there, it should hopefully pick up those host vars you set | 07:24 |
| odyssey4me | if not, try adding the host's inventory name under [all], so in this case just add 's1' on a line under '[all]' | 07:26 |
| odyssey4me | if that doesn't work, perhaps better to wait for evrardjp to get in because he'll know | 07:28 |
| odyssey4me | unless you're in a rush, in which case the old school methods will still work: https://evrard.me/group-and-host-variables-overriding-in-openstack-ansible | 07:29 |
| *** lvdombrkr has joined #openstack-ansible | 07:30 | |
| evrardjp | what's the issue? | 07:31 |
| evrardjp | in Queens you have two choices. Go for the static inventory route or go for the dynamic inventory route. | 07:32 |
| *** markvoelker has quit IRC | 07:32 | |
| evrardjp | Please have a look at https://docs.openstack.org/openstack-ansible/latest/reference/configuration/using-overrides.html | 07:33 |
| admin0 | evrardjp, bigger issue is this .. i am trying to do my first ceph+osa build and i find that in all our documenation, we have zero documenation to show how the disks are mapped/passed/specified in the osd hosts | 07:33 |
| evrardjp | If it's for the dynamic inventory, go check the dynamic inventory section of the docs. | 07:34 |
| evrardjp | https://docs.openstack.org/openstack-ansible/latest/reference/inventory/inventory.html | 07:34 |
| admin0 | yesterday cloudnull and logan gave me their 2 diff versions of how they are passing the disks, ending up with 3 different ways now | 07:34 |
| evrardjp | container_vars still apply | 07:35 |
| evrardjp | so? | 07:35 |
| odyssey4me | evrardjp: he's trying to use user-space host_vars instead, which is definitely cleaner | 07:35 |
| odyssey4me | but they aren't working | 07:35 |
| evrardjp | in Queens the host/group vars can only be used for something that's NOT already in group vars/host vars | 07:36 |
| evrardjp | due to ansible behavior | 07:36 |
| evrardjp | for now | 07:36 |
| evrardjp | if I am not mistaken :p | 07:37 |
| admin0 | oh | 07:37 |
| *** tosky has joined #openstack-ansible | 07:37 | |
| evrardjp | oh no that was Pike | 07:37 |
| evrardjp | yeah queens should load the appropriate vars | 07:37 |
| evrardjp | in host_vars | 07:38 |
| evrardjp | want me to try it ? | 07:38 |
| admin0 | evrardjp, https://gist.github.com/a1git/af6467b1ce523d6f95c9f705ba8f1318 | 07:40 |
| *** openstackgerrit has joined #openstack-ansible | 07:40 | |
| openstackgerrit | Merged openstack/ansible-hardening master: Fix instructions for opting in to V-72115 https://review.openstack.org/574844 | 07:40 |
| evrardjp | works for me https://paste.ubuntu.com/p/h8ThDprJxG/ | 07:44 |
| openstackgerrit | Taseer Ahmed proposed openstack/openstack-ansible-os_blazar master: This patch ensures that the functional tests for Blazar pass. https://review.openstack.org/571388 | 07:44 |
| admin0 | evrardjp, i will try to run it again with some extra debug | 07:47 |
| openstackgerrit | Tuan Do Anh proposed openstack/ansible-hardening master: Add release note link in README https://review.openstack.org/581227 | 07:47 |
| evrardjp | admin0: run a play that shows the var, like I did | 07:48 |
| evrardjp | http://paste.ubuntu.com/p/qNt98zdkmY/ | 07:48 |
| openstackgerrit | Tuan Do Anh proposed openstack/openstack-ansible master: Update pypi url to new url https://review.openstack.org/582096 | 07:50 |
| *** hachi has joined #openstack-ansible | 07:50 | |
| *** DanyC has joined #openstack-ansible | 07:52 | |
| admin0 | evrardjp, .. it shows the devices | 07:53 |
| evrardjp | ok. | 07:53 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Preparing group of host is not a "change" https://review.openstack.org/583879 | 07:53 |
| evrardjp | so it works | 07:53 |
| admin0 | i am going to re-run the playbook again from setup-hosts | 07:53 |
| admin0 | maybe it fixes itself | 07:54 |
| hwoarang | good morning | 07:54 |
| evrardjp | maybe you have another override somewhere. | 07:54 |
| evrardjp | Check everywhere. | 07:54 |
| evrardjp | good morning hwoarang | 07:54 |
| admin0 | i paste you the configs | 07:54 |
| admin0 | i have nothing more than that | 07:54 |
| evrardjp | in our code / ceph-ansible code | 07:54 |
| *** gregoryo has quit IRC | 07:56 | |
| admin0 | is -vvvv the best way to print expectations(from the script) vs results(what it got) from the ansible playbook run ? | 07:56 |
| evrardjp | nope I am not sure it will help you resolve this thing | 07:57 |
| evrardjp | -vv is enough to show vars, and -vvv for details. | 07:57 |
| evrardjp | grep is your best friend I think. | 07:57 |
| *** suggestable has joined #openstack-ansible | 07:57 | |
| admin0 | i see one extra quote in my dedicated_devices section .. | 07:58 |
| evrardjp | ok. | 07:58 |
| admin0 | checking/fixing | 07:58 |
| admin0 | yes .. i can see that .. even in the pastebin config i provided .. at the end of the devices, i have one open ' .. | 07:59 |
| *** mbuil has joined #openstack-ansible | 08:03 | |
| *** taseer1 has joined #openstack-ansible | 08:04 | |
| *** Taseer has quit IRC | 08:07 | |
| *** gkadam has joined #openstack-ansible | 08:11 | |
| *** gkadam is now known as gkadam-brb | 08:12 | |
| mnaser | ugh | 08:13 |
| mnaser | any ideas why dnsmasq would give SERVFAIL in trying to resolve anything (from containers)? | 08:13 |
| openstackgerrit | Taseer Ahmed proposed openstack/openstack-ansible-os_blazar master: This patch ensures that the functional tests for Blazar pass. https://review.openstack.org/571388 | 08:13 |
| evrardjp | mnaser: recursive issues, or in its autoritative part? | 08:16 |
| mnaser | recursive | 08:16 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: Added required defaults, env example, Manage LB hook https://review.openstack.org/581459 | 08:16 |
| mnaser | can't ping google or anything | 08:16 |
| evrardjp | mnaser: routing? | 08:16 |
| mnaser | lxc-dnsmasq is running though.. | 08:16 |
| evrardjp | or nat | 08:16 |
| evrardjp | check the iptables rules? | 08:16 |
| mnaser | nat. iptables rules are there, the dns server is reachable | 08:16 |
| mnaser | its giving servfail | 08:16 |
| mnaser | oh wait | 08:16 |
| mnaser | hmm | 08:16 |
| mnaser | the containers nat is failing so i guess thats why it would servfail | 08:17 |
| evrardjp | tell me the "oh wait" thing :) | 08:17 |
| evrardjp | yeah. | 08:17 |
| mnaser | the rules are there | 08:17 |
| mnaser | but for some reason | 08:17 |
| evrardjp | ping 8.8.8.8 doesn't work? | 08:17 |
| evrardjp | :p | 08:17 |
| mnaser | nope | 08:17 |
| mnaser | container traffic egress lxcbr0 | 08:17 |
| mnaser | and then | 08:17 |
| mnaser | nat, egress br-mgmt | 08:17 |
| mnaser | but when it comes back | 08:17 |
| mnaser | it gets 'de-natted' but sent over br-mgmt, not lxcbr0 | 08:18 |
| evrardjp | asymetric routing? | 08:18 |
| evrardjp | mmm | 08:18 |
| *** pmannidi has quit IRC | 08:18 | |
| evrardjp | I can't say more than check routing and fw-ing | 08:18 |
| mnaser | i dont know why the system thinks 10.x is reachable via br-mgmt | 08:18 |
| mnaser | arp -n | grep <container-ip> => lxcbr0 | 08:18 |
| evrardjp | it's hard without knowing your infra : ) | 08:19 |
| evrardjp | but you found out the first part of the issue, so you can dig deeper now : ) | 08:19 |
| evrardjp | it's the first time I hear it though. | 08:19 |
| evrardjp | Never got that issue myself. | 08:19 |
| evrardjp | and I had weird routing, I can tell you : p | 08:19 |
| evrardjp | so it's the traffic back? | 08:20 |
| mnaser | the only thing i changed is use systemd_networkd to config the bridges, but it wasnt anything wild | 08:20 |
| evrardjp | check your hosts on the path | 08:20 |
| evrardjp | I haven't used that. | 08:20 |
| mnaser | the problem is within that host | 08:20 |
| evrardjp | ok | 08:20 |
| mnaser | the traffic comes back fine | 08:20 |
| mnaser | but instead of the host sending it over lxcbr0 | 08:21 |
| mnaser | it sends it over br-mgmt | 08:21 |
| evrardjp | ok so it's directly the host that doesn't it back | 08:21 |
| evrardjp | correctly | 08:21 |
| evrardjp | ok | 08:21 |
| mnaser | and linux refuses to accept it because it came on the wrong interface | 08:21 |
| admin0 | how does ip route show look on the host ? | 08:21 |
| evrardjp | is that a new thing though? : p linux was very friendly into its INPUT before, but not FWD | 08:21 |
| evrardjp | contrary to BSD | 08:21 |
| evrardjp | anyway | 08:21 |
| evrardjp | is there any sysctl you changed? | 08:22 |
| evrardjp | or firewall rules? | 08:22 |
| mnaser | nope, firewall is empty other than the rule | 08:22 |
| mnaser | (which doesn't seem to increment much even tho i'm running a ping) | 08:23 |
| admin0 | mnaser, without the ip routes .. we are all just guesing | 08:23 |
| mnaser | ip route matches exactly another host. | 08:23 |
| mnaser | (that works) | 08:23 |
| mnaser | only difference is default route is 'proto static' in the non-working one, and doesnt have that in the working one | 08:23 |
| mnaser | but i dont really think thats the issue | 08:24 |
| admin0 | mnaser, i am a visual guy .. i need to see it written somewhere to get my head around lit .. liek container is a.b.c.d,gateay is a.b.c.1 , it lives on x.y.z host, the host has this route type | 08:24 |
| mnaser | admin0: i dunno if those details would help much. the environment works fine, it's just probably a setting that was borked somehow | 08:25 |
| admin0 | how are we able to help you with anything on "ip route matches exactly another host that works" line :D | 08:25 |
| mnaser | the other hosts work fine, so there's no misconfig. i'm just poking brains if someone can think of a possiblity of why traffic wouldn't come back on the same bridge | 08:25 |
| mnaser | that's my problem. | 08:26 |
| admin0 | on routing, you can only control yourself .. so you control how the packets goes out .. but you cannot control how it comes in .. for that, check the guy sending it to you | 08:26 |
| admin0 | so instead of receiver, look into the sender | 08:27 |
| admin0 | where the routing rules are affected that it is sending via br-mgmt and not via your common route | 08:27 |
| mnaser | the packet egresses br-mgmt with the source ip of br-mgmt because it's doing NAT | 08:28 |
| mnaser | it then gets a response on br-mgmt (logically) | 08:28 |
| admin0 | and then trace it back as well | 08:28 |
| mnaser | that response SHOULD then go to lxcbr0 and into the container | 08:28 |
| mnaser | instead the kernel is sending the packet via br-mgmt. | 08:28 |
| admin0 | and you got tcpdumps, ip routes to show this ? | 08:29 |
| mnaser | tcpdumps show this. | 08:29 |
| mnaser | arp shows the right path to go is lxcbr0 | 08:29 |
| mnaser | there is a /24 route to go via lxcbr0 | 08:29 |
| admin0 | this lxc container, is it openstack infra or openstack instance ? | 08:29 |
| evrardjp | mnaser: it's the only one with systemd_networkd? | 08:29 |
| *** markvoelker has joined #openstack-ansible | 08:29 | |
| mnaser | evrardjp: yeah | 08:30 |
| mnaser | and all i did was systemd_netdevs br-mgmt | 08:30 |
| evrardjp | and ansible hardening role? | 08:30 |
| mnaser | the hardening role ran way before | 08:30 |
| mnaser | i'm kinda 'redoing' the bridges to use networkd | 08:30 |
| evrardjp | yeah. I would start digging in networkd part. | 08:31 |
| mnaser | it's def the culprit | 08:31 |
| mnaser | but like | 08:31 |
| mnaser | all it does is just create a bridge? | 08:31 |
| mnaser | and attach a port? | 08:31 |
| evrardjp | I am expecting this to "do the right thing" and be less lenient in INPUT | 08:31 |
| evrardjp | mnaser: who knows if the internals of the chains haven't been altered | 08:31 |
| evrardjp | mnaser: I'd say go ask cloudnull he has touched this far more than us | 08:32 |
| admin0 | is it setting up vrfs that might be affecting the routing ? | 08:32 |
| mnaser | it doesn't setup vrfs | 08:33 |
| mnaser | but i just noticed | 08:33 |
| evrardjp | admin0: VRF can mean so many things :p | 08:33 |
| mnaser | the ordering of fw rules changed | 08:33 |
| evrardjp | mnaser: mmmm | 08:33 |
| evrardjp | that might be the cause too | 08:33 |
| evrardjp | you said fw was the same :p | 08:33 |
| *** taseer2 has joined #openstack-ansible | 08:33 | |
| evrardjp | or did I misread? | 08:33 |
| evrardjp | :) | 08:34 |
| mnaser | http://paste.openstack.org/show/726261/ | 08:34 |
| admin0 | i mean i am trying to think what affects routing ..its one normal routing, iptable sabels and with systemd-network, vrfs | 08:34 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible master: Add support for using distribution packages for OpenStack services https://review.openstack.org/579770 | 08:34 |
| mnaser | i was looking at the nat table evrardjp | 08:34 |
| hwoarang | could i get some reviews on ^^ please because it's a bit patch and it's giving me conflicts every now and then | 08:34 |
| hwoarang | *big | 08:34 |
| evrardjp | maybe hwoarang can help on this too mnaser , he is a networking person | 08:34 |
| evrardjp | hwoarang: sure! | 08:34 |
| evrardjp | I understand the pain hwoarang | 08:35 |
| hwoarang | prob is i will be on jollies soon so it will rot :) | 08:35 |
| evrardjp | hwoarang: why did you bring the 3.3 back? | 08:35 |
| mnaser | well the iptables rules are matching and this is still not going | 08:35 |
| * mnaser flips | 08:35 | |
| evrardjp | due to the stacktraces? | 08:35 |
| mnaser | tables | 08:36 |
| mnaser | :( | 08:36 |
| admin0 | hwoarang, who/what are jollies :) | 08:36 |
| evrardjp | admin0: jollies days | 08:36 |
| hwoarang | evrardjp: 3.3 is not mine. it came from I61a7234dc4fbde1b9319874d82fe33578f490abc | 08:36 |
| evrardjp | yeah | 08:36 |
| evrardjp | that's not good, we just pinned to 3.2 | 08:36 |
| *** taseer1 has quit IRC | 08:37 | |
| Tahvok | Hey guys from Rackspace! We're having our first conversation with Rackspace for openstack support, and currently speaking with Geoff. So you could come and say hello to us if you're near him :) | 08:37 |
| odyssey4me | evrardjp: no, we just pinned to 3.3 ;) | 08:37 |
| evrardjp | odyssey4me: we pinned to 3.2 in stable branches, right? | 08:37 |
| odyssey4me | lol @ Tahvok - those of us at Rackspace work from home :p | 08:37 |
| evrardjp | Tahvok: sorry I left those guys :p | 08:37 |
| evrardjp | Tell hello to Geoff from me though :p | 08:38 |
| Tahvok | odyssey4me: nice :) | 08:38 |
| hwoarang | evrardjp: if you revert the patch then i will resolve the conflict again | 08:38 |
| odyssey4me | evrardjp: it's a bit of a mixed bag - but yes stable is still 3.2 as far as I know | 08:38 |
| evrardjp | hwoarang: no we'll merge this one, and see how it goes. | 08:38 |
| odyssey4me | there's still one more patch to merge FYI if anyone's got a gap to review: https://review.openstack.org/#/q/topic:bug/1779534+(status:open+OR+status:merged) | 08:38 |
| admin0 | sorry hwoarang .. i still could not get the " i will be on jollies soon so it will rot" meaning :) | 08:38 |
| hwoarang | admin0: i was told that 'jollies' mean holidays in UK | 08:39 |
| hwoarang | so i try to blend with the local population | 08:39 |
| evrardjp | odyssey4me: yeah so we have a stacktracing stable and a too high version master, that's the deal? | 08:39 |
| hwoarang | :D | 08:39 |
| admin0 | :) | 08:39 |
| evrardjp | hwoarang: haha I completely understood you. | 08:39 |
| odyssey4me | 3.3 worked fine for me and solved the problem originally intended to solve - I only went back to 3.2 on the advice of the bug reporter | 08:39 |
| evrardjp | But that must be from Manchester! (even if you're not from there) | 08:39 |
| evrardjp | odyssey4me: ok. | 08:39 |
| hwoarang | no clue! but i've heard it here too | 08:40 |
| odyssey4me | looks like one of those 3.3 patches has been ported back, but not merged yet: https://review.openstack.org/#/q/Iaa824cc64bd0e8bdfdda4de243d80400f1529e8c | 08:40 |
| evrardjp | hwoarang: hahah I heard it too in London, but I am not sure if it's local | 08:40 |
| evrardjp | maybe andymccr can help us demystify this | 08:40 |
| evrardjp | or hughsaunders | 08:40 |
| odyssey4me | I've not personally seen a stacktrace for 3.2, so I can't really verify whether those patches hsould be ported back. | 08:40 |
| hwoarang | what's wrong with 3.3? | 08:40 |
| hwoarang | it is not bringing pip-10 is it? | 08:40 |
| evrardjp | hwoarang: maybe bothing | 08:41 |
| evrardjp | hwoarang: but that's what I thought it was. | 08:41 |
| evrardjp | (the pip 10) | 08:41 |
| hwoarang | well we will find out | 08:41 |
| evrardjp | so that's why I thought we took 3.2 -- the highest with no pip 10 | 08:41 |
| evrardjp | I can check real quick | 08:41 |
| evrardjp | just a sec | 08:41 |
| odyssey4me | we're telling it which version of pip to install, so it should do the right things - I don't really see the issue | 08:41 |
| evrardjp | hwoarang: we should not be using this anyway | 08:41 |
| odyssey4me | we needed a version of the script with the -d option, and 3.2/3.3 provide that | 08:42 |
| evrardjp | https://bootstrap.pypa.io/3.3/get-pip.py | 08:42 |
| openstackgerrit | Merged openstack/ansible-hardening master: Delete the unnecessary space https://review.openstack.org/583808 | 08:42 |
| evrardjp | line 7 | 08:42 |
| odyssey4me | and yes, we can be rid of using this hopefully soon, but we care about it for stable branches | 08:42 |
| evrardjp | this can break some things like tempest. | 08:42 |
| evrardjp | great | 08:42 |
| evrardjp | between plague and cholera | 08:43 |
| hwoarang | ok so it's pip10 | 08:43 |
| hwoarang | :( | 08:43 |
| evrardjp | yeah. | 08:43 |
| odyssey4me | could we at least get https://review.openstack.org/583331 in so that master is consistent | 08:43 |
| evrardjp | we should move to pip10 though | 08:43 |
| evrardjp | in master | 08:43 |
| hwoarang | ok | 08:44 |
| odyssey4me | then we need to work out if anything's broken due to this, and whether we should port it back | 08:44 |
| evrardjp | odyssey4me: wait for it to pass the integrated. | 08:44 |
| evrardjp | odyssey4me: let's not port back. | 08:44 |
| evrardjp | let's wait. | 08:44 |
| admin0 | evrardjp, it failed again with verify devices have been provided step | 08:44 |
| evrardjp | I don't like brining pip 10 in stable branches. | 08:44 |
| odyssey4me | it already did: https://review.openstack.org/583342 | 08:44 |
| admin0 | for some reasons, its not taking my config | 08:44 |
| admin0 | i will try to put it in the config directly and see if that helps | 08:45 |
| odyssey4me | none of the role values have any effect, because they're overridden by the integrated build group_var | 08:45 |
| evrardjp | odyssey4me: cloudnull has merged a series of things without 2 +2 recently. | 08:45 |
| admin0 | anyone who has ceph+osa , can they please share their configs | 08:45 |
| evrardjp | ok we are mixing too many conversations | 08:45 |
| evrardjp | let's first deal with hwoarang 's work | 08:46 |
| evrardjp | because it's a big work, and it's painful to deal when merge conflicts appear | 08:46 |
| evrardjp | could we merge this? | 08:46 |
| evrardjp | we can improve it later if need be | 08:46 |
| evrardjp | then, for the get-pip story in master branch, hwoarang do you think we can get rid of this with distro packages nowadays? | 08:47 |
| evrardjp | for the get-pip story in stable branches, we can leave it as 3.2, IMO | 08:47 |
| evrardjp | we'll see how it goes. | 08:47 |
| evrardjp | if it doesn't play well, we'll check what we can do. | 08:47 |
| evrardjp | but jumping to version 10 in stable branches will bring a lot of issues to deal with, which I don't want to deal with. | 08:48 |
| hwoarang | evrardjp: like how? fetch a good pip from distros? | 08:48 |
| evrardjp | hwoarang: just use the distro pip yeah | 08:48 |
| evrardjp | ubuntu is good -- opensuse leap 423 is good I think, right? | 08:49 |
| hwoarang | the pip_install role already skips get-pip for opensuse and ubuntu i think | 08:49 |
| evrardjp | ok | 08:49 |
| evrardjp | let's make sure it really does | 08:49 |
| hwoarang | we check if it's >=7.X.X and then we do nothing | 08:49 |
| hwoarang | i think... | 08:49 |
| evrardjp | because I am not sure Dave is using centos. | 08:49 |
| hwoarang | in any case thank you for the +2 on the distro stuff. once i fix horizon i can enable distro jobs for the full kit, not just basekit | 08:50 |
| *** viks_ has joined #openstack-ansible | 08:50 | |
| evrardjp | odyssey4me: the stacktrace point is moot IMO. | 08:50 |
| evrardjp | odyssey4me: the stacktrace happened when trying to install a lower version of pip on top of a more recent version of pip. Which is not what we are trying to do, we are trying to upgrade pip to a more recent version. | 08:51 |
| *** electrofelix has joined #openstack-ansible | 08:51 | |
| evrardjp | http://paste.openstack.org/show/726064/ clearly shows it doesn't try to install pip 9 with constraints. That's not what our usage of get-pip was meant for. | 08:52 |
| evrardjp | but I haven't digged deeper | 08:52 |
| evrardjp | hwoarang: let me check the assumption we skip things | 08:52 |
| *** udesale_ has joined #openstack-ansible | 08:54 | |
| evrardjp | hwoarang: ok we are using it | 08:56 |
| evrardjp | hwoarang: so we are skipping it in the first place to install pip/virtualenv | 08:56 |
| *** udesale__ has quit IRC | 08:57 | |
| evrardjp | but then when the virtualenv is created, we still use it to put pip and setuptools inside the venv | 08:57 |
| hwoarang | ah true | 08:57 |
| evrardjp | I am not sure why we do this | 08:57 |
| evrardjp | couldn't we just copy the dist-packages pip if it's an "okay" version, and just upgrade it inside the venv with the proper constraints? | 08:57 |
| evrardjp | odyssey4me: ^ | 08:58 |
| evrardjp | I am checking pip_install role now | 08:58 |
| odyssey4me | evrardjp: we already do that, centos is the only one that really needs to use get-pip | 09:00 |
| odyssey4me | in master I mean | 09:00 |
| evrardjp | odyssey4me: we should remove this maybe: https://github.com/openstack/openstack-ansible-pip_install/blob/master/tasks/install_source.yml#L30-L43 | 09:01 |
| odyssey4me | for stable branches we need to keep the status quo or risk quite a bit of destabilisation | 09:01 |
| evrardjp | for pip_install role | 09:01 |
| *** markvoelker has quit IRC | 09:01 | |
| evrardjp | odyssey4me: hwoarang so my analysis is that we need to change the bootstrap ansible and the pip_install role to get rid of get-pip in ubuntu/suse | 09:02 |
| odyssey4me | bootstrap-ansible only uses it if pip isn't there already at the right version | 09:02 |
| evrardjp | odyssey4me: check what I said above. | 09:02 |
| odyssey4me | in a meeting, brb | 09:03 |
| evrardjp | odyssey4me: we should refactor those checks in the get_pip function of the scripts library | 09:03 |
| evrardjp | I mean we should change that script library "get_pip" function to have the checks of the version included. So all the calls would automatically be fine. | 09:05 |
| *** vnogin has quit IRC | 09:05 | |
| evrardjp | okay, newsflash for M3 -- it's not gonna be easy to do this bump. Keystone has a problem, reverting to older needs older requirements, breaking neutron. Adapting requirenemts, neutron, then breaks nova. | 09:07 |
| evrardjp | so I will do a series of patches, solving each of the issues independently. | 09:07 |
| evrardjp | so we'll have to bump slowly. | 09:07 |
| evrardjp | amazing job. | 09:07 |
| *** FrankZhang has joined #openstack-ansible | 09:12 | |
| odyssey4me | evrardjp: I'd like a clearer understanding of the actual problem, because I've not seen a problem with what's there now. | 09:13 |
| evrardjp | about what? | 09:13 |
| evrardjp | https://github.com/openstack/openstack-ansible/blob/master/scripts/bootstrap-ansible.sh#L163 | 09:14 |
| evrardjp | https://github.com/openstack/openstack-ansible/blob/master/scripts/scripts-library.sh#L363-L389 | 09:14 |
| evrardjp | no conditionals for get-pip | 09:14 |
| evrardjp | on that thing | 09:14 |
| evrardjp | I will get rid of it. | 09:14 |
| odyssey4me | yes, but that mustn't be conditional | 09:16 |
| odyssey4me | we use that to get the exact version of pip we want | 09:16 |
| *** mbuil has quit IRC | 09:16 | |
| *** a0 has joined #openstack-ansible | 09:16 | |
| odyssey4me | the venv is built without pip | 09:16 |
| *** a0 has quit IRC | 09:17 | |
| *** udesale__ has joined #openstack-ansible | 09:17 | |
| *** a0 has joined #openstack-ansible | 09:19 | |
| *** udesale_ has quit IRC | 09:20 | |
| evrardjp | odyssey4me: the venv should get the pip version we want, that's the only thing we care. | 09:20 |
| odyssey4me | and it does exactly that today | 09:20 |
| evrardjp | it does it using get-pip | 09:21 |
| evrardjp | it doesn't do it using "pip" | 09:21 |
| *** admin0 has quit IRC | 09:21 | |
| *** a0 is now known as admin0 | 09:21 | |
| evrardjp | so we make sure pip is installed, but then we don't care. | 09:21 |
| evrardjp | because we are using get-pip anyway | 09:21 |
| evrardjp | that doesn't make sense | 09:21 |
| odyssey4me | evrardjp: I disagree. | 09:21 |
| evrardjp | we should use get-pip for getting pip if pip is not ready . | 09:21 |
| evrardjp | and then use pip. | 09:21 |
| evrardjp | odyssey4me: tell me why you disagree | 09:22 |
| evrardjp | what's the reason to still use get-pip if you have pip | 09:22 |
| odyssey4me | Changing it makes it mosre complicated, and we don't need more complexity. So again, what is the problem we're trying to solve here? I've not seen it. | 09:22 |
| evrardjp | we are removing our dependency over get-pip which is not to be used in production | 09:22 |
| evrardjp | that is silly we don't even check if virtualenv is installed with distro packages | 09:24 |
| openstackgerrit | Merged openstack/openstack-ansible-repo_build master: Update get-pip to version 3.3 https://review.openstack.org/583331 | 09:24 |
| evrardjp | ok we install it by default | 09:24 |
| evrardjp | that reassures me. | 09:24 |
| *** parasitid has quit IRC | 09:35 | |
| *** epalper has joined #openstack-ansible | 09:37 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Simplify bootstrap-ansible.sh https://review.openstack.org/583905 | 09:42 |
| *** taseer2 is now known as Taseer | 09:42 | |
| evrardjp | hwoarang: what do you think of ^ | 09:43 |
| evrardjp | odyssey4me: ^ | 09:43 |
| evrardjp | we'll see what gates think of this | 09:44 |
| odyssey4me | evrardjp: get-pip.py is what is recommended to be use by pypa, so 'it should not be used in production' is unsubstantiated | 09:45 |
| *** shardy has quit IRC | 09:46 | |
| odyssey4me | And again, I have not seen the definition of the problem. 'It should not be used in production' is not a problem definition. | 09:47 |
| evrardjp | we have packages -- why do we bother | 09:48 |
| evrardjp | look at the simplicity | 09:49 |
| evrardjp | you're opposing to something that's reducing tech debt | 09:49 |
| odyssey4me | evrardjp: perhaps you need to take a more careful look at how we got there | 09:49 |
| odyssey4me | doing what you're doing now is how we started | 09:49 |
| evrardjp | odyssey4me: long ago -- times change | 09:49 |
| odyssey4me | we grew to build the virtualenv without pip because of system package issues, we also isolated the venv build due to system package problems | 09:49 |
| evrardjp | code changes | 09:49 |
| odyssey4me | https://github.com/openstack/openstack-ansible/commit/b2be073da6d7899f860893984a5e96e30bcc3c72#diff-30c3701b5527d757251851f9253bbb3c for example | 09:50 |
| *** parasitid has joined #openstack-ansible | 09:51 | |
| odyssey4me | https://github.com/openstack/openstack-ansible/commit/86262ae2ff82feac37fd34227b4199d642a14247 | 09:51 |
| odyssey4me | https://github.com/openstack/openstack-ansible/commit/37ce304eb4ff546e0be283951b2e36f4e007abe7 | 09:52 |
| hwoarang | evrardjp: give me some time i am working on a dozen of things right now :/ | 09:53 |
| odyssey4me | so yes, times change and perhaps it's better now - but this code is the embodiment of many lessons learned | 09:53 |
| evrardjp | odyssey4me: I agree. I am checking all those commits, and you're just proving the use of a well defined venv | 09:53 |
| evrardjp | this is not opposite | 09:54 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Simplify bootstrap-ansible.sh https://review.openstack.org/583905 | 09:54 |
| odyssey4me | evrardjp: give me a problem definition and I'll understand what you're trying to achieve. right now, I don't see a problem. | 09:54 |
| odyssey4me | Also bear in mind that the infra images already have virtualenv and pip installed via get-pip.py and pip, so installing the package over the top of it may destabilise it. | 09:55 |
| evrardjp | problem is simple: I don't want to rely on get-pip and choose between plague (3.2, with stacktraces) and cholera (3.3, embedding a different major version of pip which I don't want to downgrade) | 09:55 |
| odyssey4me | Show me a 3.2 stacktrace, and show me where using 3.3 does not work? | 09:55 |
| evrardjp | I haven't finished the problem description | 09:56 |
| evrardjp | If there is an alternative we've never tried to use to reduce the external dependencies, we should use it. | 09:57 |
| hwoarang | that's true. the infra images could mess things up a bit. | 09:57 |
| evrardjp | I am not sure why I am fighting to reduce tech debt, improving offline installs. | 09:57 |
| odyssey4me | well, tell you what - I'll not stand in your way - if it works, great | 09:57 |
| odyssey4me | if you think this is back portable, go ahead | 09:58 |
| evrardjp | I don't want to change stable branches -- stable branches are working. | 09:58 |
| odyssey4me | I have too much else to get done, and I don't have enough head space to properly think this through. | 09:58 |
| evrardjp | Let's not change what's working. | 09:58 |
| odyssey4me | Are they working? I thought that was the entire premise of this discussion. | 09:58 |
| odyssey4me | If this is for master onwards, I support it wholesale. | 09:58 |
| evrardjp | yes it's only master. | 09:59 |
| odyssey4me | But the 3.2/3.3 problems you're describing count for all stable branches. | 09:59 |
| evrardjp | 3.2 works for me on stable branches. Let's not change things there. | 09:59 |
| *** markvoelker has joined #openstack-ansible | 09:59 | |
| evrardjp | I just don't want to put more patches on the story 3.2/3.3/whatever on master. | 09:59 |
| evrardjp | starting from now we now we can rely on distro packages -- let's make use of them. | 09:59 |
| hwoarang | may i suggest to calm down a bit... | 09:59 |
| odyssey4me | OK, then as I've said many times before - if we can ditch get-pip from master, let's do it. I thought that CentOS was the only reason we still had it TBH. | 10:00 |
| evrardjp | Aren't we calm? | 10:00 |
| hwoarang | if we are changing core stuff perhaps we need a spec to explain the situation and proposal? | 10:00 |
| evrardjp | odyssey4me: yes that's what I thought too :) | 10:00 |
| evrardjp | we are talking about the same thing I think :p | 10:00 |
| odyssey4me | I guess CentOS with 7.5 probably fixed that problem. | 10:00 |
| evrardjp | yeah | 10:00 |
| hwoarang | centos ships with a very old virtualenv i think | 10:00 |
| hwoarang | or very old pip | 10:00 |
| evrardjp | hwoarang: check the comment in the patch | 10:00 |
| evrardjp | showing the version from centos packages. | 10:01 |
| evrardjp | It's not like there is no evidence | 10:01 |
| hwoarang | ah ok | 10:01 |
| odyssey4me | As long as the pip version supports constraints, we're good with it... although perhaps our pip.conf may have to adjust too. | 10:01 |
| hwoarang | sorry | 10:01 |
| odyssey4me | I have no idea what pip does if it doesn't recognise a config option its given. | 10:01 |
| evrardjp | odyssey4me: I've listed the versions, and as you can see they match what was in the code. | 10:01 |
| Dmitriy | evrardjp: thanks for fixing, I was trying to find this template but I even didn't know where to start search, as I'm new to zuul | 10:01 |
| evrardjp | Dmitriy: no worries, our code is also spread out : ) | 10:02 |
| evrardjp | there is a good tool for that | 10:02 |
| evrardjp | http://codesearch.openstack.org | 10:02 |
| evrardjp | odyssey4me: I think all the config options will be supported, but I will see in the gates. | 10:02 |
| odyssey4me | it seems that the stack trace in https://review.openstack.org/#/c/583077/ came from somewhere - was that from someone using a stable branch or not? | 10:03 |
| evrardjp | I am strongly looking forward to work with infra if things are messed up there. | 10:03 |
| evrardjp | odyssey4me: the stacktrace is,probably wrong | 10:03 |
| evrardjp | look at the CLI used | 10:03 |
| evrardjp | no pip version and constrained ask | 10:03 |
| evrardjp | so I suppose the stacktrace comes from the downgrade version of pip | 10:03 |
| odyssey4me | :/ | 10:04 |
| evrardjp | downgrading pip is not a good idea | 10:04 |
| odyssey4me | yeah, as I thought - some sort of knee-jerk fix with no evidence | 10:04 |
| odyssey4me | so the backports of that should be blocked then I guess | 10:04 |
| evrardjp | YES | 10:04 |
| evrardjp | :) | 10:04 |
| odyssey4me | the original bug reporter suggested 3.2 worked for him, so if someone else thinks otherwise there should be a bug for it | 10:05 |
| evrardjp | bringing 3.3 is opening a can of worms | 10:05 |
| odyssey4me | https://bugs.launchpad.net/openstack-ansible/+bug/1782051 | 10:05 |
| openstack | Launchpad bug 1779534 in openstack-ansible "duplicate for #1782051 pip install offline fails with new version of get-pip.py" [High,Fix released] - Assigned to Jesse Pretorius (jesse-pretorius) | 10:05 |
| evrardjp | I don't want that on stable branches -- that was the first thing I said on the chan :) | 10:05 |
| odyssey4me | This is tux_ I think. We should ask him. | 10:06 |
| evrardjp | yeah | 10:06 |
| odyssey4me | 3.3 worked for him, and that's a stable branch issue. | 10:06 |
| odyssey4me | so this means we should port back 3.3 and be done with this | 10:07 |
| evrardjp | I think that's another part of the code I'd like to touch today | 10:07 |
| evrardjp | compltely different issue | 10:07 |
| evrardjp | https://github.com/openstack/openstack-ansible-pip_install/blob/master/tasks/install_source.yml#L30-L43 | 10:07 |
| evrardjp | if it fails, it fallbacks to get-pip | 10:07 |
| Dmitriy | evrardjp: I'll add to my bookmarks:) I'm actually prety new to openstack as well, I'm in since this February, so I'll be very grateful for understanding and pointing me to some things, like this) Thanks again. | 10:07 |
| evrardjp | not sure if we should do this | 10:07 |
| evrardjp | odyssey4me: do you know why we did this? | 10:07 |
| odyssey4me | evrardjp: if we're not using get-pip, then we should eliminate it everywhere | 10:07 |
| evrardjp | (keeping in mind we have https://github.com/openstack/openstack-ansible-pip_install/blob/master/tasks/main.yml#L39-L43 ) | 10:08 |
| odyssey4me | the isolated fall back mode is necessary due to the locked down pip.conf | 10:08 |
| evrardjp | yeah the isolated fallback makes sense | 10:08 |
| odyssey4me | we do that in a lot of places to be able to cater for upgrades | 10:08 |
| evrardjp | but not with a different pip | 10:08 |
| evrardjp | yeah that's fine | 10:08 |
| odyssey4me | it's the same pip? | 10:08 |
| odyssey4me | oh, heh, I see what you mean | 10:08 |
| evrardjp | yeah | 10:08 |
| odyssey4me | first task uses pip_install_script.stdout and the second does not | 10:09 |
| odyssey4me | that's a bug | 10:09 |
| evrardjp | replacing by {{ pip_install_script.stdout }} | 10:09 |
| evrardjp | yeah | 10:09 |
| evrardjp | that's backportable. | 10:09 |
| odyssey4me | no, the use of pip_install_script.stdout isn't in the stable branches | 10:09 |
| evrardjp | ahah no | 10:09 |
| evrardjp | that's new | 10:09 |
| evrardjp | yeah | 10:09 |
| evrardjp | that's the thing | 10:09 |
| evrardjp | okay | 10:09 |
| evrardjp | I will fix that | 10:09 |
| evrardjp | for master | 10:09 |
| odyssey4me | good spot | 10:09 |
| evrardjp | for stable... that's a different thing then | 10:09 |
| evrardjp | maybe we should backport those patches? | 10:10 |
| odyssey4me | well, maybe - maybe not | 10:10 |
| odyssey4me | If we do that it'll touch a lot of things, and I'm not sure if the transition will work well. | 10:10 |
| evrardjp | https://github.com/openstack/openstack-ansible-pip_install/commit/40373daf7e27ab8cef00611945acf68c3e86fe82#diff-2444ad0870f91f17ca6c2a5e96b26823 | 10:10 |
| odyssey4me | I think it may be better to ensure that our periodics are healthy before we risk more destabilisation. | 10:11 |
| evrardjp | yeah that's a risk | 10:11 |
| evrardjp | yeah agreed. | 10:11 |
| evrardjp | I can propose and -W | 10:11 |
| odyssey4me | I mean, given that all our stable branches are now using the same distribution version as master, we could perhaps do it - but then we need a focused two weeks of changes and thorough testing | 10:12 |
| odyssey4me | We could perhaps more safely just back port to Queens only, and leave the older branches alone. | 10:12 |
| evrardjp | odyssey4me: so I don't want to backport the getting rid of get-pip in stable branches | 10:12 |
| evrardjp | I just want to merely fix the downgrade issue that person sees | 10:12 |
| *** chyka has joined #openstack-ansible | 10:13 | |
| evrardjp | odyssey4me: or we can re-implement things in stable branches to ensure pip_install doesn't install things when version x is already available. | 10:13 |
| evrardjp | bad idea. | 10:13 |
| evrardjp | except if x is VERY WELL defined. | 10:14 |
| odyssey4me | bootstrap-ansible only implements get-pip if virtualenv isn't there or at the right version, and pip isn't there and at the right version | 10:14 |
| odyssey4me | everything else is done by pip_install | 10:14 |
| evrardjp | yeah it's two different conversations :p | 10:14 |
| evrardjp | let's not mix them | 10:15 |
| odyssey4me | so yeah, if we port back changes to pip_install which make things better it'll do us some good | 10:15 |
| evrardjp | I will explain clearly what I meant with that | 10:15 |
| *** v1a4 has joined #openstack-ansible | 10:17 | |
| *** chyka has quit IRC | 10:18 | |
| odyssey4me | hmm, one thing to consider is that by using the distro packages for virtualenv we now have inconsistent pip/setuptools/wheel versions in the venv when we build it | 10:18 |
| odyssey4me | we could go with it and see if problems arise, as they did in the past - or we could proactively do something now to ensure we have consistency in the venv | 10:18 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-pip_install master: Ensure the fallback uses the same install method https://review.openstack.org/583910 | 10:21 |
| odyssey4me | evrardjp looks like the chain to backport would be https://review.openstack.org/#/c/562243/7 & https://review.openstack.org/#/c/562243/ & https://review.openstack.org/#/c/570327/ | 10:21 |
| odyssey4me | and the one you just proposed | 10:22 |
| *** Bhujay has quit IRC | 10:22 | |
| evrardjp | odyssey4me: so for the bootstrap ansible, I ensure the version by forcing an install of pip setuptools wheel with the constraints. | 10:22 |
| evrardjp | I think this is reasonable to have the same thing in the venvs. | 10:22 |
| odyssey4me | evrardjp: that's an extra task in every single role, except in master where we could do it in the common role | 10:22 |
| admin0 | how do i print out all vars during a playbook run .. i think even if i have setup bluestore in the variables, its defaulting to filestore somewhere | 10:23 |
| odyssey4me | evrardjp I guess for the stable branches it'd be best to just let it be? | 10:24 |
| evrardjp | odyssey4me: yeah not change things for stable branches until a bug is there | 10:25 |
| evrardjp | I will wait | 10:25 |
| evrardjp | I will think about what we can backport | 10:25 |
| evrardjp | but be very conservative, as you know me :p | 10:25 |
| odyssey4me | by that bug, do you think 3.3 is the solution? or are you thinking of an alternative | 10:29 |
| evrardjp | 3.3 is a solution but bringing issues | 10:30 |
| evrardjp | because now you have to bump to 10 everywhere | 10:30 |
| evrardjp | you'd have to* | 10:30 |
| evrardjp | so stable branches bump to 10. is annoying because of many things | 10:30 |
| evrardjp | I'd rather we bump 10. in master and see how it goes. | 10:31 |
| evrardjp | I think we must find another solution with what we have on stable branches, and I think those backports "could" do, as they would skip, for centos, the installation using get-pip | 10:31 |
| evrardjp | I have to double check though | 10:31 |
| *** Bhujay has joined #openstack-ansible | 10:32 | |
| *** markvoelker has quit IRC | 10:32 | |
| evrardjp | yeah it would. | 10:32 |
| evrardjp | assuming the user has installed python-pip | 10:32 |
| evrardjp | that's a good enough workaround IMO | 10:33 |
| evrardjp | oh that's gonna be harder that that,great :p | 10:34 |
| odyssey4me | evrardjp: the bug is there | 10:34 |
| odyssey4me | but yeah, perhaps https://review.openstack.org/#/c/562243/7 with https://review.openstack.org/583910 alone will actually cover this quite safely | 10:36 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Simplify bootstrap-ansible.sh https://review.openstack.org/583905 | 10:36 |
| odyssey4me | referencing https://bugs.launchpad.net/openstack-ansible/+bug/1782051 as a related bug I think | 10:36 |
| openstack | Launchpad bug 1779534 in openstack-ansible "duplicate for #1782051 pip install offline fails with new version of get-pip.py" [High,Fix released] - Assigned to Jesse Pretorius (jesse-pretorius) | 10:36 |
| odyssey4me | anyway, I've spent too much brain power on this today - time to get on with other things | 10:37 |
| evrardjp | haha sorry | 10:37 |
| odyssey4me | I'll switch attention to figuring out the rest of those failures for my patches later today - magnum, sahara, etc. | 10:38 |
| odyssey4me | For now I need to get some other stuff done. | 10:39 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_neutron master: tasks: providers: Rename 'ovs' configuration file https://review.openstack.org/583919 | 10:42 |
| *** tux_ has joined #openstack-ansible | 10:47 | |
| hwoarang | evrardjp: do you understand this problem http://logs.openstack.org/70/579770/22/check/openstack-ansible-deploy-aio_basekit-opensuse-423/babc0bd/job-output.txt.gz#_2018-07-19_08_52_43_276239 ? | 10:48 |
| hwoarang | started poping up in the aio_distro_basekit job | 10:48 |
| odyssey4me | evrardjp: apologies for blowing a bit of a gasket earlier... not sure why that hit a nerve but it was unnecessary - happy we've come to a good conclusion after all that :p | 10:50 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-pip_install stable/queens: Ensure get-pip is not used when good version of pip is present https://review.openstack.org/583925 | 10:50 |
| evrardjp | odyssey4me: no offense taken, I know you. | 10:50 |
| hwoarang | 'blowing a bit of a gasket' nice phrase i am going to steal it | 10:50 |
| evrardjp | :p | 10:50 |
| odyssey4me | ;) | 10:51 |
| *** tux_ has quit IRC | 10:52 | |
| evrardjp | hwoarang: haha I will try to remember it too | 10:52 |
| evrardjp | hwoarang: checking | 10:52 |
| evrardjp | hwoarang: that's bad if that's true | 10:53 |
| hwoarang | tell me more | 10:53 |
| *** hachi has quit IRC | 10:53 | |
| evrardjp | it means 39.2 was not found on the mirror, and that version was asked by our pins | 10:53 |
| evrardjp | hwoarang: I am trying to bump those pins with my M3 patch | 10:54 |
| evrardjp | but as you can see I was busy with other things too :) | 10:54 |
| hwoarang | hmm | 10:54 |
| evrardjp | I am taking another approach for the M3 patch right now, which is doing small steps increments. | 10:54 |
| evrardjp | but I can do that bump separately too if you prefer. | 10:55 |
| hwoarang | so we are asking for a version that doesn't exist? when did that change? | 10:55 |
| evrardjp | hwoarang: I've noticed in the past that 39.2 was not found on certain mirroring. | 10:55 |
| evrardjp | it exists | 10:55 |
| hwoarang | ah | 10:55 |
| evrardjp | it just not well mirrored. | 10:55 |
| hwoarang | i see | 10:55 |
| hwoarang | ok so recheck it is ;p | 10:55 |
| hwoarang | should we tell infra ? | 10:56 |
| evrardjp | recheck -- it's often temporary | 10:56 |
| hwoarang | ok then | 10:56 |
| evrardjp | https://github.com/pypa/setuptools/tree/v39.2.0 exists :) | 10:56 |
| evrardjp | hwoarang: please note that this failure was the reason I didn't bump it in the stable branch | 10:57 |
| evrardjp | weirdly | 10:57 |
| *** cshen has quit IRC | 10:57 | |
| evrardjp | it was always temporary, so if you can reproduce it 100% of the time on a certain provider, we should bug infra. | 10:58 |
| hwoarang | ok | 10:58 |
| *** prasen has quit IRC | 10:59 | |
| *** cshen has joined #openstack-ansible | 11:05 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: [DNM] - Testing openSUSE nova role https://review.openstack.org/495767 | 11:07 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-pip_install stable/queens: Ensure get-pip is not used when good version of pip is present https://review.openstack.org/583925 | 11:15 |
| openstackgerrit | Merged openstack/openstack-ansible-pip_install master: Ensure the fallback uses the same install method https://review.openstack.org/583910 | 11:18 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts master: bionic: Remove python-software-properties https://review.openstack.org/583708 | 11:19 |
| *** hachi has joined #openstack-ansible | 11:23 | |
| *** udesale__ has quit IRC | 11:28 | |
| odyssey4me | evrardjp: did you have an old copy of pip_install perhaps? I'm curious why https://review.openstack.org/#/c/583925/1..2/tests/test.yml was needed when https://review.openstack.org/#/c/563521/1 mrged ages ago | 11:29 |
| *** markvoelker has joined #openstack-ansible | 11:30 | |
| evrardjp | I had | 11:31 |
| evrardjp | I will rebase | 11:31 |
| evrardjp | wait | 11:31 |
| evrardjp | it's probably got pulled in | 11:32 |
| jenglisch_ | i'd like to configure haproxy to use SNI + hostnames instead of using different ports. there seems no possiblity to easily implement this with openstack-ansible (the haproxy template has no 'options' for that). is there a reason ? is this setup a bad idea? or is it just uncommon? | 11:32 |
| evrardjp | in the cherry pick | 11:33 |
| evrardjp | jenglisch_: uncommon | 11:33 |
| odyssey4me | jenglisch_: I think that's something we will want to do, but we're not yet sure whether all services support that yet | 11:33 |
| evrardjp | you can have your own haproxy configuration if you want. You will need to override the endpoints and all. | 11:33 |
| odyssey4me | it'll require all clients to support it, and all services to support it | 11:34 |
| odyssey4me | if you're keen on having it then we're keen to see whether it works! | 11:34 |
| evrardjp | odyssey4me: hasn't logan- done that or partially done it? It is an effort so it depends how nice you want to be with your customers at the end jenglisch_ :) | 11:35 |
| odyssey4me | it will certainly help simplify a lot of things, and be complementary to the https://github.com/openstack/openstack-ansible-specs/blob/master/specs/rocky/centralized-nginx.rst work which hasn't really happened yet | 11:35 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible master: [WIP] Add qdrouterd role for messaging backend deployment https://review.openstack.org/570261 | 11:36 |
| jenglisch_ | >but we're not yet sure whether all services support that yet | 11:37 |
| jenglisch_ | i'm currently just thinking about the public api endpoints, i'll test a few clients that i'm interested in (python openstack client, terraform, ...) | 11:37 |
| jenglisch_ | however i think every client in 2018 should support SNI :) | 11:37 |
| jenglisch_ | a major problem for a long time was win xp (< SP3), which does not support SNI, but thats not my target audience anyway :) | 11:38 |
| admin0 | i think our osa-ceph does not see bluestore yet | 11:41 |
| odyssey4me | jenglisch_: one would think, but the openstack clients might not ;) they're dependent on the community making it happen | 11:42 |
| admin0 | even with the variable override, it forces filestore and its syntax check | 11:42 |
| odyssey4me | admin0: you'd need to validate against the pinned version of ceph-ansible what's supported | 11:42 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_swift master: Remove httplib https://review.openstack.org/583933 | 11:43 |
| admin0 | did that and found out :) | 11:46 |
| admin0 | is there a way to unpin/force it to use a higher version or master branch | 11:46 |
| jenglisch_ | evrardjp: is overriding the full haproxy config possible with openstack-ansible-haproxy_server ? i think i'm only allowed to reconfigure single services https://github.com/openstack/openstack-ansible-haproxy_server/blob/master/defaults/main.yml#L41 | 11:47 |
| jenglisch_ | but probably thats sufficient with haproxy_acls | 11:47 |
| *** shardy has joined #openstack-ansible | 11:50 | |
| *** admin0 has left #openstack-ansible | 11:52 | |
| *** admin0 has joined #openstack-ansible | 11:52 | |
| odyssey4me | admin0: sure, just update the ansible-role-requirements.yml in your own fork, or set the env var to use a different one when you bootstrap-ansible... but be aware that you're stepping into untested waters | 11:54 |
| odyssey4me | admin0: we had ceph-ansible in master pinned for around 3 months to work out stuff that got broken due to changes in ceph-ansible because they're now using their own modules/lookups/etc | 11:54 |
| jenglisch_ | [13:47] <jenglisch_> but probably thats sufficient with haproxy_acls << it's not, since openstack-ansible defines a service as a frontend/backend pair, while a SNI configuration would have just one frontend and multiple backends :/ | 11:54 |
| odyssey4me | so it won't *just work* | 11:54 |
| *** kstev has joined #openstack-ansible | 11:57 | |
| *** dave-mccowan has joined #openstack-ansible | 11:59 | |
| *** kstev has quit IRC | 12:01 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Pull in ldap dependencies via ldappool https://review.openstack.org/583865 | 12:02 |
| *** markvoelker has quit IRC | 12:02 | |
| *** markvoelker has joined #openstack-ansible | 12:02 | |
| *** kstev has joined #openstack-ansible | 12:02 | |
| odyssey4me | jenglisch_: I expect you could have both implemented - our standard ones, and a new front-end which uses all the other front ends? | 12:03 |
| openstackgerrit | Dmitriy R. proposed openstack/openstack-ansible-ceph_client master: Added possibility to specify ssh username and port for ceph mons. https://review.openstack.org/583943 | 12:08 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Update all SHAs for milestone 3 https://review.openstack.org/574006 | 12:11 |
| admin0 | odyssey4me, in that case, i can use the ceph role to setup ansible separately and then just specify it in osa i guess | 12:12 |
| admin0 | not sure how though :D | 12:12 |
| odyssey4me | admin0: what you can do is use ceph-ansible independently of OSA to setup the ceph-cluster as you wish, using its own ansible runtime (in a venv) and its own ansible.cfg pointing to where the roles should be found, etc | 12:26 |
| odyssey4me | then - once you have that, you can use OSA's ceph client integration to setup openstack to use the cluster | 12:26 |
| *** ssbarnea1 has quit IRC | 12:36 | |
| ansmith | evrardjp: hello, iiuc, I believe the pinning we observed comes from https://github.com/openstack/openstack-ansible-repo_build/blob/8cbd722ac4de268cbf6c283f9825958071b6a6bb/templates/requirements_constraints.txt.j2 | 12:36 |
| evrardjp | ansmith: yes it does, but that's generated based on live requirements :) | 12:37 |
| ansmith | evrardjp: so version at point in time from defaults/repo_packages/openstack_services.yml will become <= | 12:37 |
| evrardjp | so there is something going on there | 12:37 |
| evrardjp | ansmith: yeah those things come from py_pkgs | 12:37 |
| *** ssbarnea has joined #openstack-ansible | 12:37 | |
| evrardjp | which gets the info from defaults/repo... and userspace things | 12:37 |
| evrardjp | but it should have resolved properly, unless there is a requirement that requires a certain version conflicting with our upper constraints | 12:38 |
| evrardjp | that's the one we need to flag | 12:38 |
| ansmith | won't openstack reqs be tied to the sha1 or am i missing something | 12:38 |
| ansmith | e.g. tied to 31-03-2018 which corresponded to oslo.messaging of 6.0.0 at that point | 12:39 |
| *** v1a4 has quit IRC | 12:40 | |
| *** Guest72952 is now known as portdirect | 12:41 | |
| evrardjp | ansmith: in defaults/repo.... you see requirements there | 12:42 |
| evrardjp | they are bumped at the same time | 12:42 |
| evrardjp | here the messaging from requirements is 8.0.0 | 12:42 |
| evrardjp | (iirc) | 12:42 |
| evrardjp | so there is something going on | 12:42 |
| evrardjp | I haven't got the chance to work on this -- as I am trying to bump said file, but there are LOTS of issues that have appeared. | 12:43 |
| evrardjp | including checks disabled upstream. | 12:43 |
| ansmith | will try your bump to m3, just wanted to provide an update and make sure I am connecting the dots correctly | 12:45 |
| -openstackstatus- NOTICE: logs.openstack.org is offline, causing POST_FAILURE results from Zuul. Cause and resolution timeframe currently unknown. | 12:54 | |
| *** ChanServ changes topic to "logs.openstack.org is offline, causing POST_FAILURE results from Zuul. Cause and resolution timeframe currently unknown." | 12:54 | |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible master: [WIP] Add qdrouterd role for messaging backend deployment https://review.openstack.org/570261 | 12:57 |
| Dmitriy | Hm, guys, smth strange is happening with zuul for https://review.openstack.org/#/c/583943/ | 13:01 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_tempest master: Disable Volume Backup Tests https://review.openstack.org/583955 | 13:08 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_tempest master: Disable Volume Backup Tests https://review.openstack.org/583955 | 13:16 |
| *** udesale has joined #openstack-ansible | 13:18 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_tempest master: Disable Volume Backup Tests https://review.openstack.org/583955 | 13:18 |
| Bhujay | OSA haproxy self signed certificate allows us to enter a subjectAlternatename in the user var , however a magnum kube cluster dont recognize this , it gives error Post https://10.174.124.14:5000/v3/auth/tokens: x509: cannot validate certificate for 10.174.124.14 because | 13:22 |
| Bhujay | it doesn't contain any IP SANs | 13:22 |
| Bhujay | the problem was resolved once manually a self certificate was recreated with proper SAN attributes . | 13:26 |
| evrardjp | Bhujay: better to use real certs. | 13:26 |
| evrardjp | :p | 13:26 |
| odyssey4me | Bhujay: if you can suggest some config changes to improve the self-signed certs being generated then that'd be welcome, but yes - they're not really meant to be used in production | 13:27 |
| evrardjp | maybe we should remove certificate generation, and do it upfront. | 13:27 |
| evrardjp | Bhujay: any patch is welcomed there :p | 13:28 |
| evrardjp | odyssey4me: that's the opposite path of what we are currently doing, but hey, why not! | 13:28 |
| evrardjp | or maybe we should the way we do now | 13:29 |
| Bhujay | Sure , I had got the clue from a post by another user he generated it from a template file to solve the SAN issue . I will test it out and take your opinion | 13:29 |
| Bhujay | the current method suffices encrypted communication and have worked for me until i tried a kube cluster in magnum . So it is good but we can improve it .. | 13:30 |
| *** tux_ has joined #openstack-ansible | 13:31 | |
| *** tux__ has joined #openstack-ansible | 13:32 | |
| tux__ | folks, quick question i have install OSA with Ceph and all looking good my glance talking to shared storage etc but i found Cinder-api container doesn't have ceph.conf file and keyring | 13:32 |
| tux__ | what could be wrong there? | 13:32 |
| odyssey4me | tux__: why would it? only the cinder-volume service speaks to ceph IIRC | 13:33 |
| tux__ | hmm! i thought api need to talk to ceph | 13:33 |
| odyssey4me | nope - client -> api -> scheduler -> volume | 13:34 |
| odyssey4me | -> ceph | 13:34 |
| tux__ | I am trying to create volume and its failing not showing any error | 13:34 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Update all SHAs for milestone 3 https://review.openstack.org/574006 | 13:34 |
| odyssey4me | tux__: did the volume service get the request? if not, check scheduler, then check api | 13:35 |
| Bhujay | openstack volume service list , is the output showing all @rbd up ? | 13:36 |
| *** tux_ has quit IRC | 13:36 | |
| *** taseer1 has joined #openstack-ansible | 13:37 | |
| *** ChanServ changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: https://goo.gl/tTmdgs" | 13:37 | |
| -openstackstatus- NOTICE: logs.openstack.org is back on-line. Changes with "POST_FAILURE" job results should be rechecked. | 13:37 | |
| *** Taseer has quit IRC | 13:38 | |
| *** taseer2 has joined #openstack-ansible | 13:38 | |
| *** taseer1 has quit IRC | 13:41 | |
| evrardjp | woot recheck time ! | 13:42 |
| *** tux__ has quit IRC | 13:50 | |
| *** joabdearaujo_ has joined #openstack-ansible | 13:50 | |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_cinder master: [New files - needs update] Update paste, policy and rootwrap configurations 2018-07-19 https://review.openstack.org/583963 | 13:55 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_heat master: [New files - needs update] Update paste, policy and rootwrap configurations 2018-07-19 https://review.openstack.org/583964 | 13:55 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_neutron master: Update paste, policy and rootwrap configurations 2018-07-19 https://review.openstack.org/583966 | 13:56 |
| odyssey4me | heh, looks like mnaser didn't adjust the source branch updater to skip rootwraps | 14:04 |
| ansmith | my mistake ^^^, was not intended | 14:06 |
| *** joabdearaujo_ has left #openstack-ansible | 14:06 | |
| odyssey4me | ansmith: no worries, I think that's like a right of passage for this project ;) | 14:06 |
| odyssey4me | we should probably have a release-tools folder and put it there | 14:07 |
| odyssey4me | and evrardjp should share his new magic tool | 14:07 |
| evrardjp | odyssey4me: release is still in the old non-magic | 14:09 |
| evrardjp | I haven't got the chance to work on it | 14:09 |
| evrardjp | but I will need to | 14:09 |
| evrardjp | odyssey4me: the first step I will do is remove that git review step. | 14:10 |
| evrardjp | and the removal of the folder | 14:11 |
| evrardjp | taht's too scary | 14:11 |
| evrardjp | : p | 14:11 |
| evrardjp | but then I will move my stuff there. | 14:11 |
| evrardjp | I will remove the click dependency too | 14:11 |
| evrardjp | as this is convenient for me, but not for other ppl. | 14:11 |
| evrardjp | I will just lazy import them. | 14:12 |
| evrardjp | :p | 14:12 |
| *** FrankZhang has quit IRC | 14:13 | |
| odyssey4me | evrardjp: as long as click is not in our requirements.txt, it's fine to have it in the tools I think | 14:14 |
| odyssey4me | you'll just have to handle the error and tell them to install it | 14:14 |
| odyssey4me | it's only because click is not in the g-r process that we can't include it | 14:15 |
| evrardjp | I think those functions are self-contained, and some are refering to a lib, but the contained scripts can be called directly | 14:15 |
| odyssey4me | perhaps some day someone will bother to convert it to use cliff instead, but meh | 14:15 |
| evrardjp | so after a quick rewrite we should be freed of the dependency | 14:15 |
| evrardjp | but it could still get included in a package if we need to | 14:16 |
| odyssey4me | ja, so instead of you wasting time on it - push up the script as-is and perhaps someone else will | 14:16 |
| evrardjp | well that is already pushed into my graveyard, but since the release was changed, I'd like to make sure it works fine again | 14:16 |
| *** Guest1593 has joined #openstack-ansible | 14:16 | |
| evrardjp | (we don't need that many functions) | 14:17 |
| evrardjp | now a few of those are included in releases | 14:17 |
| odyssey4me | ok, I have no context so I can't comment ;) | 14:17 |
| evrardjp | I sent it to the ML though : p | 14:17 |
| odyssey4me | we do need tooling to do that stuff for all our branches though | 14:17 |
| evrardjp | we are not tagging all the roles anymore | 14:17 |
| evrardjp | yeah | 14:17 |
| evrardjp | that's still valid | 14:17 |
| odyssey4me | oh ok that - yes that's great | 14:18 |
| evrardjp | yeah I just want to do a quick test of what we can do in governance to see if releases break | 14:18 |
| *** Guest1593 has quit IRC | 14:18 | |
| evrardjp | and then we can be freed of the last functions | 14:18 |
| evrardjp | free* | 14:18 |
| evrardjp | so it's a process... | 14:19 |
| evrardjp | :D | 14:19 |
| evrardjp | slowly evolving as it doesn't touch many ppl | 14:19 |
| *** cshen has quit IRC | 14:24 | |
| *** joabdearaujo has joined #openstack-ansible | 14:24 | |
| Dmitriy | what do you think about https://review.openstack.org/#/c/583943/ ? It's infra role, but still... | 14:25 |
| Dmitriy | it doesn't resolve the question of empty ceph_mons (if we don't want to provide access to ceph monitors), as we still need auth keys, but it's step in this direction | 14:29 |
| *** tux_ has joined #openstack-ansible | 14:32 | |
| tux_ | Bhujay: | 14:33 |
| tux_ | Bhujay: "openstack volume service list" where should i run this command | 14:33 |
| *** tux__ has joined #openstack-ansible | 14:34 | |
| tux__ | odyssey4me: sorry i was in meeting | 14:34 |
| tux__ | i am back looking cinder-volume | 14:35 |
| tux__ | If i am using Ceph storage then my volume IO will directly go to ceph why we need volume node? | 14:35 |
| tux__ | I am confused here | 14:35 |
| Bhujay | tux_: from any client with admin user or login from horizon as admin and see the system-system info-volume services | 14:36 |
| Bhujay | you need volume node which has the rbd drivers and configuration to issue command to ceph mon for managing storage volumes life cycle | 14:37 |
| *** tux_ has quit IRC | 14:38 | |
| tux__ | Bhujay: if i attach storage volume to instance then that IO will hit cinder volume or go to directly ceph for data read/write? | 14:40 |
| *** hachi has quit IRC | 14:40 | |
| *** hachi has joined #openstack-ansible | 14:41 | |
| Bhujay | cinder will create and attach the vol to instance , then from vm the intercation with ceph shd be direct as per my understanding | 14:41 |
| tux__ | Bhujay: Ah! that make sense otherwise i have to drop my cinder plan :) | 14:42 |
| tux__ | Bhujay: hold on let me check my cinder-volume as you said | 14:43 |
| evrardjp | cores, can I have some votes on https://review.openstack.org/#/c/583568/ ? It would make another patch very light. | 14:43 |
| tux__ | I have ostack-controller-03_cinder_api_container-a41fe37c & ostack-controller-03_cinder_scheduler_container-9e0724ca | 14:43 |
| tux__ | i can't see any cinder-volume container | 14:44 |
| tux__ | am i missing something here? | 14:44 |
| Bhujay | there is no containers , cinder-vol is on metal | 14:44 |
| tux__ | why do we need cinder-metal if IO directly going to Ceph ? | 14:45 |
| tux__ | Something is wrong here... | 14:45 |
| evrardjp | tux__: you don't and can containerize it | 14:46 |
| tux__ | evrardjp: can you please explain me i am totally confused, This is what i want, I have openstack and ceph cluster and i want to use cinder for boot volume so my instance boot from cinder volume | 14:46 |
| tux__ | evrardjp: I can see only ceinder-api & cinder-scheduler container in OSA | 14:47 |
| evrardjp | ceph production example: https://docs.openstack.org/openstack-ansible/latest/user/ceph/full-deploy.html#environment-customizations | 14:47 |
| evrardjp | it's all in the docs | 14:47 |
| tux__ | Reading it | 14:48 |
| tux__ | evrardjp: Can you confirm cinder-volume is just a middle man when using Ceph ? | 14:48 |
| tux__ | I want my VM directly go to ceph not via Cinder-volume or any intermediate path | 14:49 |
| evrardjp | you should read cinder docs for that | 14:49 |
| evrardjp | but yeah. | 14:50 |
| Dmitriy | + | 14:50 |
| devx | d34dh0r53 or cjloader: have you had any issues with a multi node aio? I keep getting an error ERROR! Unexpected Exception, this is probably a bug: (cryptography 1.2.3 (/usr/lib/python2.7/dist-packages), Requirement.parse('cryptography>=1.5'), set(['paramiko'])) | 14:50 |
| evrardjp | you will go through cinder-volume | 14:50 |
| evrardjp | it depends on what you're trying to achieve | 14:50 |
| evrardjp | you're not forced to use cinder | 14:50 |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_zaqar master: Completed required variables for README.rst https://review.openstack.org/583976 | 14:50 |
| evrardjp | do your research | 14:50 |
| cjloader | devx: I have not | 14:50 |
| tux__ | evrardjp: all i want boot instance from cinder volume living on shared storage for HA and live migration | 14:50 |
| devx | cjloader I just started getting this errors yesterday afternoon | 14:51 |
| tux__ | Let me explain what i am trying to do so you can point me to right direction.. | 14:51 |
| devx | ok i'll dig in to thanks | 14:51 |
| evrardjp | tux__: that or you want to boot from an image that gets persisted on node, which mounts ceph? | 14:51 |
| evrardjp | that's completely different | 14:51 |
| evrardjp | cinder you will need cinder things | 14:51 |
| evrardjp | including cinder-volume | 14:52 |
| *** biffi has quit IRC | 14:52 | |
| cjloader | devx: last mnaio I set up was like a month ago | 14:52 |
| evrardjp | hey cjloader | 14:52 |
| evrardjp | how are things? | 14:52 |
| evrardjp | been a while :) | 14:53 |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_trove master: Completed required variables for README.rst https://review.openstack.org/583978 | 14:53 |
| devx | Thanks cjloader | 14:53 |
| cjloader | evrardjp: been good, busy. | 14:53 |
| cjloader | you? | 14:54 |
| evrardjp | busy busy indeed :) | 14:54 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_monasca-agent master: Change functional test to voting https://review.openstack.org/583979 | 14:54 |
| tux__ | evrardjp: We have 20 compute node and 5 ceph cluster node, I want to use shared storage for all instance so i have live-migration option and all shared storage advantage, but i have few application they are super IO intensive which i want to run on LOCAL disk of compute node so they won't use shared storage, someone told me use cinder boot-volume so you can boot VM from cinder volume and they will be on Ce | 14:54 |
| tux__ | ph storage and other instance which you don't want to be on ceph boot them from image so they will on LOCAL disk. | 14:54 |
| cjloader | devx: d34dh0r53 deals with mnaios alot, i'd ask him | 14:55 |
| tux__ | evrardjp: do you think i don't need cinder for that to isolate some VM on shared storage and some of them on LOCAL compute disk? | 14:56 |
| *** FrankZhang has joined #openstack-ansible | 14:57 | |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_trove master: Completed required variables for README.rst https://review.openstack.org/583978 | 14:57 |
| tux__ | evrardjp: what solution will be better for my requirement? I am sure there must be a way | 14:58 |
| *** gillesMo has joined #openstack-ansible | 14:58 | |
| evrardjp | I am sorry I don't have time to answer those questions tux__ -- that's architecting your solution -- I don't have enough time or info to do so | 14:59 |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_zaqar master: Add the tags into example playbook https://review.openstack.org/583980 | 14:59 |
| tux__ | :( | 14:59 |
| tux__ | all i want to understand does OSA can isolate config to make compute-1 to local and compute-2 to ceph ? | 15:00 |
| guilhermesp | tux__: this is openstack level at all. You can use cells or host aggregates | 15:01 |
| evrardjp | guilhermesp: that's if using cinder volumes | 15:01 |
| evrardjp | but yeah | 15:01 |
| evrardjp | else he could probably mount ceph for some nodes on some location | 15:01 |
| evrardjp | and no cinder | 15:01 |
| evrardjp | else... | 15:02 |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_cloudkitty master: Completed required variables for README.rst https://review.openstack.org/583981 | 15:02 |
| evrardjp | probably ignore the ceph for live migration and do standard block migrate | 15:02 |
| evrardjp | slower but works | 15:02 |
| evrardjp | I don't know the requirements :p | 15:02 |
| tux__ | guilhermesp: if my all compute node configure to use Ceph then how do i tell compute-X to use LOCAL disk ? | 15:02 |
| d34dh0r53 | devx: at the Drs office currently but IIRC I haven't seen that particular error. Does it happen during the deployment? | 15:02 |
| tux__ | host aggregates doesn't support that | 15:02 |
| guilhermesp | tux__: you define in your nova.conf if your compute node is going to use rbd driver, for exemple | 15:03 |
| *** lbragstad has joined #openstack-ansible | 15:03 | |
| *** rmart04 has quit IRC | 15:03 | |
| *** vakuznet has joined #openstack-ansible | 15:04 | |
| tux__ | guilhermesp: Can i put that definition in Openstack_user_config? | 15:04 |
| tux__ | guilhermesp: that would be my solution then | 15:04 |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_searchlight master: Completed required variables for README.rst https://review.openstack.org/583983 | 15:05 |
| *** Bhujay has quit IRC | 15:05 | |
| guilhermesp | yes tux__ this is one exemple of 3 computes of my env http://paste.openstack.org/show/726277/ | 15:06 |
| *** kstev has quit IRC | 15:07 | |
| guilhermesp | compute9 and 11 launches instances with ceph backend.. compute10 uses local storage to launch instances | 15:07 |
| tux__ | guilhermesp: oh!! so 9 using ceph and 10 using local disk :) | 15:07 |
| tux__ | guilhermesp: you are awesome!!!! | 15:08 |
| tux__ | guilhermesp: i don't need cinder anymore | 15:08 |
| tux__ | guilhermesp: love that piece of codes thanks man!! | 15:08 |
| tux__ | we should document this kind of small solution :) i didn't find any good example like that | 15:09 |
| openstackgerrit | zhulingjie proposed openstack/openstack-ansible-os_searchlight master: Completed required variables for README.rst https://review.openstack.org/583983 | 15:09 |
| *** pcaruana has quit IRC | 15:12 | |
| *** taseer2 is now known as Taseer | 15:13 | |
| guilhermesp | tux__: then, after define this, I have a script that adds the compute node http://paste.openstack.org/show/726278/ | 15:14 |
| tux__ | so this script will add new compute node into cluster right? | 15:16 |
| tux__ | guilhermesp: really cool! what else you have please shared with me... This is what i am looking for | 15:17 |
| tux__ | guilhermesp: could you also shared some info about did you create Host aggregation and map with flavors? | 15:17 |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_masakari master: Add the missing "#" https://review.openstack.org/583991 | 15:20 |
| *** cshen has joined #openstack-ansible | 15:21 | |
| *** gkadam-brb is now known as gkadam | 15:21 | |
| *** kstev has joined #openstack-ansible | 15:22 | |
| vakuznet | cloudnull have you seen this error http://paste.openstack.org/show/726279/ ? | 15:24 |
| bgmccollum | in neutron, have you always been able to attach a port to an instance, created from a network with port security disabled and doesn't have a subnet? cause i just discovered this... | 15:25 |
| bgmccollum | opens up fun things like VyOS instances that manage everything about the LAN side of the router... | 15:27 |
| *** chyka has joined #openstack-ansible | 15:27 | |
| gillesMo | Hello ! I want to change the default domain name `openstack.local` of my hosts, and containers. My cluster is already deployed and in use. Will I break everything if I change lxc_container_domain and openstack_domain and relaunch setup-hosts.yml playbook ? | 15:28 |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_panko master: test https://review.openstack.org/583993 | 15:28 |
| *** DanyC has quit IRC | 15:30 | |
| *** DanyC has joined #openstack-ansible | 15:30 | |
| openstackgerrit | zhulingjie proposed openstack/openstack-ansible-os_panko master: Remove the unnecessary "=" https://review.openstack.org/583993 | 15:31 |
| *** poul has quit IRC | 15:32 | |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_aodh master: Remove the unnecessary = https://review.openstack.org/583995 | 15:35 |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_tacker master: Remove the unnecessary = https://review.openstack.org/583996 | 15:35 |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_searchlight master: Remove the unnecessary = https://review.openstack.org/583997 | 15:35 |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_watcher master: Remove the unnecessary = https://review.openstack.org/583998 | 15:36 |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_masakari master: Remove the unnecessary = https://review.openstack.org/583999 | 15:36 |
| *** lvdombrkr has quit IRC | 15:36 | |
| *** vnogin has joined #openstack-ansible | 15:36 | |
| vakuznet | odyssey4me have you seen this error http://paste.openstack.org/show/726279/ | 15:37 |
| tux__ | vakuznet: look like its relayed to get-pip.py version.. i had same issue last week | 15:37 |
| vakuznet | osa 17.0.6? | 15:38 |
| tux__ | let me find ticket | 15:38 |
| tux__ | or bug | 15:39 |
| *** DanyC has quit IRC | 15:39 | |
| tux__ | get-pip.py version 3.3 you need | 15:39 |
| tux__ | vakuznet: https://bugs.launchpad.net/bugs/1779534 | 15:39 |
| openstack | Launchpad bug 1779534 in openstack-ansible "pip install offline fails with new version of get-pip.py" [High,Fix released] - Assigned to Jesse Pretorius (jesse-pretorius) | 15:39 |
| *** DanyC has joined #openstack-ansible | 15:39 | |
| openstackgerrit | weizj proposed openstack/openstack-ansible-os_trove master: Remove the unnecessary "=" https://review.openstack.org/584001 | 15:39 |
| *** vnogin has quit IRC | 15:41 | |
| tux__ | put this in your user_variables and try http://paste.openstack.org/show/726280/ | 15:41 |
| *** DanyC has quit IRC | 15:44 | |
| *** rpittau_ has joined #openstack-ansible | 15:45 | |
| *** udesale has quit IRC | 15:46 | |
| *** rpittau has quit IRC | 15:48 | |
| *** mbuil_ has quit IRC | 15:48 | |
| vakuznet | tux__ thanks | 15:49 |
| *** mbuil has joined #openstack-ansible | 15:50 | |
| *** hachi has quit IRC | 15:50 | |
| *** chyka_ has joined #openstack-ansible | 15:53 | |
| mbuil | I need jinja2 support. Does anybody see anything wrong here: https://hastebin.com/sebigatamu.cs | 15:56 |
| *** chyka has quit IRC | 15:57 | |
| *** Bhujay has joined #openstack-ansible | 15:59 | |
| *** vakuznet has quit IRC | 16:05 | |
| *** cshen has quit IRC | 16:06 | |
| *** shardy has quit IRC | 16:11 | |
| *** tux__ has quit IRC | 16:13 | |
| cloudnull | afternoons ! | 16:18 |
| admin0 | hello cloudnull | 16:19 |
| cloudnull | qq - https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/vars/ubuntu-16.04.yml#L78 - does anyone know why we have that package being installed by default? | 16:19 |
| cloudnull | o/ admin0 hows it ? | 16:19 |
| admin0 | still struggling with ceph :D | 16:20 |
| evrardjp | cloudnull: that reminds me something | 16:20 |
| cloudnull | I have a cloud where its running "linux-image-4.15.0-24-generic" and there is no "extra" package for it ? | 16:20 |
| evrardjp | did you reboot? | 16:20 |
| cloudnull | so the deployment is essentially blocked unless I downgrade to "linux-image-extra-4.15.0-15-generic" | 16:20 |
| evrardjp | I mean is your system in sync with what's running ? | 16:20 |
| evrardjp | (the facts) | 16:21 |
| cloudnull | yes. | 16:21 |
| cloudnull | the host is infact running 4.15.0-24 | 16:21 |
| cloudnull | I wonder if that extra package things is needed? | 16:21 |
| evrardjp | 4.15? | 16:22 |
| evrardjp | on xenial? | 16:22 |
| cloudnull | yes | 16:22 |
| cloudnull | running pike | 16:22 |
| cloudnull | https://github.com/openstack/openstack-ansible-openstack_hosts/blob/stable/pike/vars/ubuntu-16.04.yml#L78 | 16:23 |
| *** kberger has quit IRC | 16:23 | |
| cloudnull | the hosts were pre-kicked and handed over as-is | 16:23 |
| cloudnull | I could downgrade the kernel(s) however I'd rather not. | 16:24 |
| evrardjp | cloudnull: https://github.com/openstack/openstack-ansible-openstack_hosts/commit/48eeb134779dc51bd0d3c1374ff4090f20b1a6f6 | 16:24 |
| *** KeithMnemonic has joined #openstack-ansible | 16:24 | |
| evrardjp | cloudnull: could your issue be stale facts vs a system you just restarted? | 16:25 |
| evrardjp | else all the packages should be in your mirror and that's a mirroring issue | 16:25 |
| cloudnull | nope. | 16:26 |
| evrardjp | except if you built your own kernel | 16:26 |
| cloudnull | there's no extra package for that kernel | 16:26 |
| cloudnull | http://paste.openstack.org/show/726285/ | 16:26 |
| cloudnull | maybe that was an old issue w/ vhost_net | 16:26 |
| cloudnull | and not something we really need anymore? | 16:26 |
| cloudnull | http://paste.openstack.org/show/726287/ | 16:27 |
| cloudnull | those are all the 4.15 kernels | 16:27 |
| *** coreywright has quit IRC | 16:27 | |
| cloudnull | admin0 did you get ceph running w/ the OSA integrated ceph? | 16:28 |
| * cloudnull hasn't read much scroll back | 16:28 | |
| evrardjp | not sure if it's a packaging problem or not | 16:28 |
| evrardjp | cloudnull: yeah I was apt searching through things | 16:28 |
| evrardjp | cloudnull: can you modprobe vhost_net onto these kernels? | 16:29 |
| cloudnull | the other 4.15 kernels? | 16:29 |
| evrardjp | your 4.15.0-24 | 16:29 |
| cloudnull | yes thats what i have here http://paste.openstack.org/show/726285/ | 16:30 |
| evrardjp | 4.15.0-15 is fine for getting the extra | 16:30 |
| evrardjp | cloudnull: I think it's worth having an extra task | 16:30 |
| evrardjp | if linux-image-extra-{{ ansible_kernel }} exists, then add it to the list | 16:31 |
| evrardjp | else ignore | 16:31 |
| cloudnull | ++ that might be good | 16:31 |
| evrardjp | does that make sense ? | 16:31 |
| evrardjp | because some still ship | 16:31 |
| evrardjp | and I am not savvy into the reason they ship separated | 16:31 |
| evrardjp | I remember I had to add it on some of my nodes, and so it's easy to upgrade because it gets resolved | 16:31 |
| evrardjp | but for someone that has latest and only deploys after the fact, that's a different story | 16:32 |
| *** suggestable has quit IRC | 16:33 | |
| evrardjp | maybe ppl are just using virtio | 16:33 |
| evrardjp | : p | 16:33 |
| evrardjp | anyway | 16:34 |
| evrardjp | yeah probably worth making this optional | 16:34 |
| evrardjp | ok I have to run | 16:34 |
| evrardjp | ttyl | 16:34 |
| cloudnull | chao | 16:35 |
| cloudnull | have a good one | 16:35 |
| evrardjp | I am getting those issues a lot nowadays | 16:35 |
| evrardjp | http://logs.openstack.org/55/583955/3/check/openstack-ansible-functional-ubuntu-xenial/af69934/job-output.txt.gz#_2018-07-19_15_52_31_732761 | 16:35 |
| *** jwitko has joined #openstack-ansible | 16:35 | |
| evrardjp | that's painful. | 16:35 |
| evrardjp | I will be back later | 16:35 |
| evrardjp | I will merge the two patches for getting M3 running | 16:36 |
| evrardjp | I just have to go rn | 16:36 |
| cloudnull | cool | 16:36 |
| cloudnull | take care ttys | 16:36 |
| *** epalper has quit IRC | 16:38 | |
| cloudnull | evrardjp i suspect the http://logs.openstack.org/55/583955/3/check/openstack-ansible-functional-ubuntu-xenial/af69934/job-output.txt.gz#_2018-07-19_15_52_31_732761 issue is related to the testing changes related to the use of become, setting the user, etc | 16:39 |
| cloudnull | looks like it's failing to connect to localhost, which i suspect is not using connection local, etc. | 16:39 |
| *** coreywright has joined #openstack-ansible | 16:40 | |
| *** tux_ has joined #openstack-ansible | 16:43 | |
| odyssey4me | evrardjp: was that from the nova role test, or from another repo? | 16:43 |
| odyssey4me | FYI I've noticed that we have varying settings for lxc in tests, so https://review.openstack.org/583241 aims to resolve that | 16:45 |
| *** vnogin has joined #openstack-ansible | 16:46 | |
| evrardjp | odyssey4me: tempest | 16:46 |
| *** suggestable has joined #openstack-ansible | 16:46 | |
| odyssey4me | I wonder if removing the python interpreter for localhost's host_vars would help | 16:50 |
| *** vnogin has quit IRC | 16:50 | |
| odyssey4me | otherwise, perhaps that connection override isn't working and we need to change the play to separate out nova_compute from the rest | 16:51 |
| odyssey4me | oh, it already is - what's there worked for ironic (compute in a container), but hasn't worked when compute is on localhost | 16:52 |
| *** suggestable has quit IRC | 16:53 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Move MQ vhost/user creation into role https://review.openstack.org/583378 | 16:54 |
| odyssey4me | evrardjp cloudnull thoughts on how to tackle this? I'm thinking that perhaps we just make all tests doing nova_compute must do it on localhost and eliminate this tragedy: https://github.com/openstack/openstack-ansible-tests/blob/master/test-install-nova.yml#L105-L106 ? | 16:58 |
| odyssey4me | unless someone has a better dea? | 16:58 |
| odyssey4me | *idea | 16:58 |
| odyssey4me | oh wait, that's a different play | 16:59 |
| odyssey4me | this is the part crapping out: https://github.com/openstack/openstack-ansible-tests/blob/master/test-install-nova.yml#L21-L28 | 17:00 |
| odyssey4me | I have an idea. | 17:01 |
| cloudnull | i support your idea :) | 17:02 |
| odyssey4me | yeah, I think I'm just going to go with changing the ironic role to put compute on localhost | 17:07 |
| odyssey4me | and the cinder fails will need to be forced to put some services on localhost too | 17:07 |
| logan- | odyssey4me: idk the story behind the shift to connection: local stuff recently but in my tests i usually try to never override the connection, and i also do not override the implicit localhost, that way you can always assume your default connection is going to be ssh (same in test as prod), and your implicit localhost will be intact with connection=local. then target your playbook nova_all at something like http://paste.openstack.org/raw/72629 | 17:07 |
| logan- | 2/ | 17:07 |
| logan- | http://paste.openstack.org/raw/726292/ * | 17:07 |
| logan- | so your "remote" host is testhost1, and your delegate_to: localhost still works exactly as it would in non-test envs | 17:08 |
| odyssey4me | logan-: yeah, I was trying to work through this without touching every role inventory :/ | 17:08 |
| *** gkadam has quit IRC | 17:09 | |
| odyssey4me | the issue at hand here is that if we use 'remote_user: root' and 'become: yes' together, and it targets localhost - we lose all the env vars for the user | 17:09 |
| odyssey4me | that breaks all kinds of things | 17:09 |
| odyssey4me | especially given that we're now delegating a bunch of stuff back to localhost in the roles (service setup, etc) | 17:09 |
| logan- | gotcha | 17:11 |
| logan- | why not create a host to target the play at with ansible_become=yes and leave the implicit localhost intact without become | 17:11 |
| odyssey4me | initially I pulled 'become: yes' out, given that we're using 'remote_user: root', but then all sorts of things crapped out when the play targeted localhost | 17:11 |
| odyssey4me | I guess that's because of the implicit switch to 'connection: local' | 17:12 |
| odyssey4me | So I guess we either assume localhost for some services, and adjust inventories accordingly; or we adjust inventories for those services to use an inventory_hostname other than localhost and has the address 127.0.0.1 | 17:13 |
| *** vakuznet has joined #openstack-ansible | 17:13 | |
| odyssey4me | I suppose the second option, as you suggest, is perhaps cleaner. | 17:14 |
| *** Bhujay has quit IRC | 17:14 | |
| odyssey4me | ok, I like it - lemme sort all that out | 17:15 |
| odyssey4me | I'd like to get this nonsense behind me | 17:15 |
| logan- | :) | 17:15 |
| odyssey4me | oh, for the sake of interest - in retrospect this would probably have been easier to achieve in python, but I gave it a go: https://gist.github.com/odyssey4me/5b0eacf86b544dec0a8a4ef4229c277c | 17:17 |
| guilhermesp | guys, see you can figure out with me a strange behaviour with aio using 17.0.6 that I've been facing since yesterday ( I was discussing this with logan- anyway ) | 17:17 |
| odyssey4me | I used that playbook for the 'Move MQ tasks into the roles' patches | 17:17 |
| guilhermesp | a fresh clean install with xenial, running setup-everythings breaks here http://paste.openstack.org/show/726294/ | 17:18 |
| logan- | hah nice odyssey4me | 17:18 |
| guilhermesp | this is kinda of related I guess https://bugs.launchpad.net/openstack-ansible/+bug/1777655 | 17:18 |
| openstack | Launchpad bug 1777655 in openstack-ansible "neutron-db-manage fails during OSA installation" [Undecided,Fix released] - Assigned to Maksim Malchuk (mmalchuk) | 17:18 |
| logan- | looking guilhermesp | 17:18 |
| guilhermesp | the solution that I found was: destroy neutron server container, recreate and then run os-neutron-install.yml | 17:18 |
| guilhermesp | so the installation flows as expected | 17:18 |
| guilhermesp | not http proxy behind me anyway | 17:19 |
| guilhermesp | no* | 17:19 |
| guilhermesp | is not that urgent. I'm just putting myself in a position of a new osa-user | 17:19 |
| logan- | guilhermesp https://github.com/openstack/openstack-ansible/blob/17.0.6/playbooks/defaults/repo_packages/openstack_services.yml#L33-L34 | 17:22 |
| logan- | the date on that tag does not contain the fix from the bug | 17:22 |
| logan- | because it wasn't merged until 6/19 | 17:23 |
| guilhermesp | ah k, makes sense now | 17:23 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 17:23 |
| logan- | the latest queens sha in requirements is 96b284d498c9302a07554023bc865700e8254a95 | 17:23 |
| guilhermesp | what doens't makes sense is why destroying and recreating the containers makes it work | 17:24 |
| guilhermesp | neutron container* | 17:24 |
| logan- | yeah without seeing the logs who knows, but you could try bumping your requirements sha forward to apply the bugfix | 17:24 |
| odyssey4me | guilhermesp: probably because the first attempt does some database things, then fails | 17:25 |
| odyssey4me | then the second may be able to recover | 17:25 |
| guilhermesp | odyssey4me the second with the recreated neutron container? | 17:25 |
| odyssey4me | guilhermesp: full disclosure, I haven't looked at the bug or paste :p | 17:26 |
| guilhermesp | odyssey4me haha | 17:26 |
| guilhermesp | anyway, I was just curious but logan- made it clearer | 17:27 |
| guilhermesp | my motivation was to reproduce this bug https://bugs.launchpad.net/openstack-ansible/+bug/1782372 | 17:27 |
| openstack | Launchpad bug 1782372 in openstack-ansible "Rally install fails on existing deployment creation" [Undecided,New] | 17:27 |
| guilhermesp | I think I may know the fix, so do we wait until next bug triage session or does anyone who want to apply a fix can do that? | 17:27 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Connect to localhost as remote host for cinder/nova https://review.openstack.org/584033 | 17:30 |
| admin0 | logan-, i think bluestore is not picked up .. so no matter how i specify it, it did not worked .. when you have time, can u confirm on this ? | 17:31 |
| logan- | admin0: osd_objectstore: bluestore in user_variables didn't work? | 17:31 |
| logan- | these are new osd builds right? | 17:32 |
| logan- | it won't change over any existing osds | 17:32 |
| admin0 | brand new | 17:32 |
| admin0 | it did not worked | 17:32 |
| logan- | any old metadata on the drives you can see with ceph-disk list? | 17:32 |
| guilhermesp | logan- going to have a look carefully next time https://imgur.com/a/trWup0X :) +1 skill acquired, thanks | 17:32 |
| admin0 | logan-, no matter what I did , even /dev/sda /dev/sdb, it always ends up with: fatal: [s1]: FAILED! => {"changed": false, "failed": true, "msg": "please provide devices and dedicated_devices to your osd scenario"} | 17:33 |
| *** epalper has joined #openstack-ansible | 17:34 | |
| admin0 | if i give dedicated_devices, it says give devices | 17:34 |
| logan- | admin0: that doesn't have anything to do with bluestore then | 17:34 |
| logan- | admin0: http://docs.ceph.com/ceph-ansible/master/osds/scenarios.html | 17:34 |
| admin0 | there is a if condition that checkes if filestore/bluestore .. .. so when i follow that link and give disks , it says no devices | 17:34 |
| admin0 | data -- tried this as well | 17:35 |
| admin0 | data - /dev/sda , /dev/sdb | 17:35 |
| logan- | do you have the 'osd_scenario' and 'devices' vars defined | 17:35 |
| admin0 | osd_scenario yes | 17:35 |
| admin0 | devices are in the host_vars as yo have showed | 17:35 |
| *** rmart04 has joined #openstack-ansible | 17:36 | |
| admin0 | and i did verify with debug that it sees the devices | 17:36 |
| logan- | hmm | 17:36 |
| logan- | are you using a non-collocated scenario? | 17:37 |
| logan- | or collocated? | 17:37 |
| admin0 | tried all :) | 17:37 |
| admin0 | the only thing not tried is to not use bluestore because my boss wants bluestore and nothing else | 17:37 |
| admin0 | i am setting it up again .. | 17:38 |
| logan- | so collocated scenario with devices, and 'ansible s1 -m debug -a msg="{{ osd_scenario }} {{ devices }}" should show you stuff if the host_vars setting is working | 17:38 |
| *** epalper has quit IRC | 17:39 | |
| admin0 | evrardjp showed me a different way to validate those | 17:39 |
| admin0 | let me quick set it up again and then give results here | 17:39 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest master: Switch compute to use different inventory_hostname https://review.openstack.org/584037 | 17:39 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ironic master: Remove localhost from neutron agents group https://review.openstack.org/584038 | 17:43 |
| *** rmart04 has quit IRC | 17:43 | |
| *** rmart04 has joined #openstack-ansible | 17:44 | |
| admin0 | if i have to redo an env again, i delete the lxc-containers and the 00apt proxy .. how is the pip proxy being added ? | 17:44 |
| admin0 | because pip fails .. i think its pointing to some proxy as well | 17:44 |
| admin0 | nvm | 17:46 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest master: Switch compute to use different inventory_hostname https://review.openstack.org/584037 | 17:48 |
| *** rmart04 has quit IRC | 17:50 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_magnum master: Switch compute/storage to use different inventory_hostname https://review.openstack.org/584042 | 17:53 |
| *** electrofelix has quit IRC | 17:54 | |
| *** mattinator has joined #openstack-ansible | 17:58 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Switch compute to use different inventory_hostname https://review.openstack.org/584048 | 17:58 |
| *** gillesMo has quit IRC | 17:58 | |
| *** hachi has joined #openstack-ansible | 18:00 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Switch compute to use different inventory_hostname https://review.openstack.org/584048 | 18:00 |
| *** poopcat has joined #openstack-ansible | 18:01 | |
| *** cshen has joined #openstack-ansible | 18:03 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Switch compute to use different inventory_hostname https://review.openstack.org/584049 | 18:03 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest master: Switch compute to use different inventory_hostname https://review.openstack.org/584037 | 18:04 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_magnum master: Switch compute/storage to use different inventory_hostname https://review.openstack.org/584042 | 18:05 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Switch compute to use different inventory_hostname https://review.openstack.org/584048 | 18:05 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Switch compute to use different inventory_hostname https://review.openstack.org/584048 | 18:06 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Switch compute to use different inventory_hostname https://review.openstack.org/584049 | 18:06 |
| *** cshen has quit IRC | 18:07 | |
| *** armaan has joined #openstack-ansible | 18:07 | |
| *** DanyC has joined #openstack-ansible | 18:08 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Move MQ vhost/user creation into role https://review.openstack.org/583387 | 18:09 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Move MQ vhost/user creation into role https://review.openstack.org/583388 | 18:09 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder master: Switch storage to use different inventory_hostname https://review.openstack.org/584052 | 18:14 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Switch compute to use different inventory_hostname https://review.openstack.org/584049 | 18:14 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Move MQ vhost/user creation into role https://review.openstack.org/583387 | 18:14 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder master: Move MQ vhost/user creation into role https://review.openstack.org/583366 | 18:15 |
| jamesdenton | Quick question. I'm updating a role to install a pip package, but the version in the repo server is older than what I need. How can I update that? | 18:17 |
| *** hachi has quit IRC | 18:19 | |
| odyssey4me | jamesdenton: you can override the upper-constraints, then rebuild the repo... but if that package is used by other services then it will get implemented for them all and may have side-effects | 18:20 |
| odyssey4me | which package is this? | 18:20 |
| odyssey4me | (if I may ask) | 18:20 |
| jamesdenton | neutron_lib | 18:20 |
| jamesdenton | ya know, an important one | 18:21 |
| jamesdenton | lol | 18:21 |
| odyssey4me | is this a production env, or master? | 18:21 |
| jamesdenton | master | 18:21 |
| odyssey4me | (a test env) | 18:21 |
| odyssey4me | oh ok, meh then should be fine I guess | 18:21 |
| *** DanyC has quit IRC | 18:21 | |
| odyssey4me | although we've found nova/keystone/neutron have competing requirements at the moment and they each break each other apparently | 18:21 |
| odyssey4me | https://review.openstack.org/574006 has been ongoing for some time | 18:22 |
| *** DanyC has joined #openstack-ansible | 18:22 | |
| jamesdenton | ok good to know | 18:22 |
| odyssey4me | anyway, the var to se in your user_variables to override constraints is https://github.com/openstack/openstack-ansible-repo_build/blob/master/defaults/main.yml#L29-L32 | 18:22 |
| odyssey4me | so you just add 'neutron_lib==x.y.z' as a list item to it | 18:23 |
| odyssey4me | jrosser & logan- this may interest you: https://review.openstack.org/583943 | 18:23 |
| *** hachi has joined #openstack-ansible | 18:24 | |
| odyssey4me | (non-root ceph cluster key fetching) | 18:24 |
| jamesdenton | ok perfect, thanks! | 18:24 |
| logan- | ah slick | 18:24 |
| *** DanyC has quit IRC | 18:27 | |
| *** tux_ has quit IRC | 18:27 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 18:32 |
| *** vnogin has joined #openstack-ansible | 18:34 | |
| *** vnogin has quit IRC | 18:38 | |
| cloudnull | d34dh0r53 tux_ looks like https://review.openstack.org/#/c/583234/ is dead in the water... | 18:40 |
| cloudnull | its in for master, and an override can be set, but the upstream fix is blocked. | 18:42 |
| *** cshen has joined #openstack-ansible | 18:42 | |
| d34dh0r53 | hmm, I can see the rationale but I think we've seen the 3.2 issues in our internal gates | 18:42 |
| odyssey4me | an alternative has been proposed which forgoes the need to go through a pip downgrade https://review.openstack.org/583925 | 18:44 |
| odyssey4me | evrvjust needs to unblock it | 18:46 |
| odyssey4me | evrardjp: | 18:46 |
| cloudnull | seems like the latter has more moving parts than just setting the script version to 3.3 | 18:46 |
| odyssey4me | hey, I'm just the messenger | 18:46 |
| * cloudnull not shotting the messanger | 18:47 | |
| admin0 | someone adds pip.conf with the proxy in setup-hosts .. i delete and it adds it back .. how to prevent that ? | 18:47 |
| odyssey4me | moving to 3.3 puts us into a place where we have to downgrade pip, which has its own problems - that patch will upgrade from an earlier version in place - and 3.2 will do fine on a clean install, which is the only time get-pip will get used | 18:47 |
| admin0 | something* | 18:47 |
| odyssey4me | so yeah, I think it will be better in the long term | 18:48 |
| odyssey4me | admin0: you can't, it does that and should do that - the config in pip.conf depends on whether the repo server is live or not though | 18:48 |
| admin0 | but i deleted all the containers and deleted the mappings and inventory and this is the 2nd go around | 18:49 |
| admin0 | no containers are created in this 2nd round, because it does not go pass this step | 18:49 |
| odyssey4me | admin0 however, https://review.openstack.org/564422 + https://review.openstack.org/568502 + https://review.openstack.org/568637 make it better - I just haven't figured out why it fails the gate | 18:49 |
| odyssey4me | not sure if anyone has cycles to try and figure that out | 18:50 |
| odyssey4me | lemme rebase those and fire up a test node to see | 18:50 |
| admin0 | chattr +i will fail it again, so i was thinking to do while true do echo > /root/.pip/pip.conf, so that the setup-hosts goes through | 18:51 |
| odyssey4me | admin0: then you'll get other breakage | 18:52 |
| admin0 | :D | 18:52 |
| odyssey4me | try clearing the fact cache? | 18:52 |
| admin0 | :( .. i always miss tis | 18:52 |
| admin0 | this | 18:52 |
| admin0 | removed facts cache, removed the pip.conf .. no containers created yet .. running setup-hosts again | 18:53 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: Adds networking-ovn support https://review.openstack.org/584067 | 18:54 |
| admin0 | and it came back again and fails pip | 18:55 |
| admin0 | :( | 18:55 |
| odyssey4me | what's the failure admin0 ? | 18:56 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Ensure that repo checks use the right values https://review.openstack.org/564422 | 18:56 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Ensure that the repo servers also use the pypi cache https://review.openstack.org/568502 | 18:56 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Remove default pip_links value https://review.openstack.org/568637 | 18:56 |
| admin0 | RetryError: HTTPConnectionPool(host='cloud-int.domain.com', port=8181): Max retries exceeded with url: /os-releases/17.0.6/ubuntu-16.04-x86_64/ (Caused by ResponseError('too many 503 error responses',))"], "stdout": "", "stdout_lines": []} | 18:56 |
| admin0 | because that container is already deleted | 18:57 |
| admin0 | so there is no 8181 | 18:57 |
| odyssey4me | yeah, that's the damned links thing | 18:57 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible master: Adds haproxy endpoints for networking-ovn support https://review.openstack.org/584069 | 18:57 |
| odyssey4me | ok, gimme some time to fire something up to try it | 18:57 |
| cloudnull | odyssey4me http://paste.openstack.org/show/726303/ - it doesn't look like it downgrades at all ? | 18:59 |
| odyssey4me | cloudnull: well, I don't know what to say - I agree, but don't have the energy for it | 19:00 |
| odyssey4me | both ways will work, and I think evrardjp's proposal is perhaps better, but they'll both work... | 19:01 |
| odyssey4me | perhaps better to discuss with him when he's online again | 19:01 |
| logan- | we still have to rely on 3.2 working for the initial install with evrardjp's method. does the 3.2 breakage only occur during upgrades? | 19:01 |
| odyssey4me | logan-: good question | 19:02 |
| odyssey4me | the original bug reported said that 3.2 worked well for him | 19:02 |
| odyssey4me | 3.3 was a follow up by someone else | 19:02 |
| admin0 | odyssey4me, for now, is there anything i can delete to do this again | 19:03 |
| cloudnull | the issue was 3.2 broke on cent w/ py 2.7.5 | 19:03 |
| odyssey4me | luckily for master we can ditch get-pip entirely and never deal with this nonsense again | 19:03 |
| admin0 | repos are on setup infra, i am stuck on setup hosts | 19:03 |
| cloudnull | which tux and I spent the better part of an evening debugging | 19:03 |
| logan- | so cloudnull even with evrardjp's patch it should still fail the initial pip install right? | 19:04 |
| odyssey4me | admin0 if you checkout https://review.openstack.org/#/c/568637/ with all its deps then that should work: git fetch https://git.openstack.org/openstack/openstack-ansible refs/changes/37/568637/6 && git checkout FETCH_HEAD | 19:04 |
| cloudnull | and there was no upgrade/downgrade shinanigans, I had tux uninstall pip and try again | 19:04 |
| cloudnull | logan- i'd say yes. | 19:04 |
| logan- | ok yea thats what im thinking | 19:04 |
| *** tux_ has joined #openstack-ansible | 19:04 | |
| odyssey4me | best to review the patch then | 19:04 |
| odyssey4me | perhaps combining both fixes would be good, because then if pip is on the host, it'll use that pip - otherwise it'll download 3.3 and use that | 19:05 |
| odyssey4me | *get-pip 3.3 | 19:05 |
| admin0 | done .. running it again | 19:06 |
| cloudnull | when tux_ is around again maybe we can get more on that issue. | 19:07 |
| odyssey4me | cloudnull: all the patches for 3.3 from master would need porting back for 3.3 to be used: https://review.openstack.org/#/q/topic:bug/1779534 | 19:07 |
| odyssey4me | the pip_install one is useless on its own, because its values are overridden by the integrated repo | 19:08 |
| admin0 | odyssey4me, thanks .it added the pip.conf but not that line :) | 19:08 |
| odyssey4me | then the repo_build patch is there for completeness and consistency mostly | 19:08 |
| * cloudnull lives in a world where everything is master ATM and was trying to fix this for a user stable. | 19:11 | |
| tux_ | cloudnull: here | 19:11 |
| cloudnull | tux_ do you have the ability to validate https://review.openstack.org/#/c/583925/ | 19:12 |
| cloudnull | re: that get-pip issue you were seeing the other day | 19:12 |
| evrardjp[m] | hey... my phone vibrated too much. it's normal 3.2 would fail on centos 7.5 as centos ships with a more recent version. | 19:12 |
| cloudnull | if that works in your env | 19:12 |
| evrardjp[m] | we just need to make sure 3.2 is not used | 19:13 |
| cloudnull | evrardjp[m] it works w/ 3.3 | 19:13 |
| tux_ | I have all centos7.5 host and it was failing | 19:13 |
| evrardjp[m] | i am sure of it | 19:13 |
| evrardjp[m] | it would upgrade pip to 10 | 19:13 |
| cloudnull | also tux was running an upgrade from stable tag to stable tag | 19:13 |
| evrardjp[m] | that's a bad idea. | 19:13 |
| evrardjp[m] | yeah | 19:13 |
| odyssey4me | evrardjp[m]: what if pip is not on the host already | 19:13 |
| cloudnull | so i dont think that right | 19:13 |
| evrardjp[m] | not surprised | 19:13 |
| odyssey4me | evrardjp[m]: we execute get-pip with constraints, so it doesn't install pip10 | 19:14 |
| cloudnull | evrardjp upgrading from stable/tag to stable/tag is not a bad idea... | 19:14 |
| evrardjp[m] | mmm that's fair odyssey4me | 19:15 |
| logan- | yea the constraints thing got fixed when pip10 was released | 19:15 |
| evrardjp[m] | yeah cloudnull i don't think it would be a problem from tag to tag | 19:15 |
| odyssey4me | I think it might be good to implement both patches, because evrardjp's patch will make it use an existing pip if it's there | 19:16 |
| logan- | thats what im thinking odyssey4me | 19:16 |
| logan- | i agree | 19:16 |
| odyssey4me | but the 3.3 patch will make it use get-pip.py 3.3 with constraints if there is no pip | 19:16 |
| evrardjp[m] | agreed | 19:16 |
| *** tosky has quit IRC | 19:16 | |
| odyssey4me | evrardjp[m]: if you could unblock all those patches it'd be nice :) | 19:16 |
| tux_ | vakuznet: was also getting same issue to today he mentioned that in earlier post http://paste.openstack.org/show/726279/ | 19:16 |
| odyssey4me | (including your own) | 19:16 |
| cloudnull | evrardjp[m] it was a problem for tux_ ? | 19:17 |
| evrardjp[m] | odyssey4me: my concern is not for pip install role | 19:18 |
| evrardjp[m] | assuming my patch gets in and 3.3 on pip install role happens. | 19:18 |
| odyssey4me | evrardjp[m]: yeah, bootstrap-ansible is where your concern lies | 19:18 |
| evrardjp[m] | yeah. or bumping the constraints to 10. | 19:18 |
| *** armaan has quit IRC | 19:19 | |
| evrardjp[m] | I think we should do it, eventually | 19:19 |
| odyssey4me | evrardjp[m]: I can fix bootstrap-ansible to sort that out, because I think your other patch to remove its use for master is good, but we need a solution for stable branches. That solution is to constrain. | 19:19 |
| evrardjp[m] | i just rather wait for results of the change in master. | 19:19 |
| evrardjp[m] | yeah | 19:19 |
| evrardjp[m] | lgtm | 19:19 |
| evrardjp[m] | will unblock all of those now that we have discussed it. | 19:20 |
| odyssey4me | well, I think that patch failed - but haven't checked why... I guess I could patch master for now, then your patch can sort things out later? | 19:20 |
| evrardjp[m] | when I'll be back home :) | 19:20 |
| odyssey4me | oh, we already do constrain installing pip for bootstrap-ansible | 19:21 |
| evrardjp[m] | do we ? | 19:22 |
| odyssey4me | https://github.com/openstack/openstack-ansible/blob/master/scripts/scripts-library.sh#L385-L388 | 19:22 |
| tux_ | cloudnull: as soon as i get time i will try to fire up on my LAB 16.0.15 version | 19:22 |
| evrardjp[m] | yeah that part yes just a sec | 19:23 |
| odyssey4me | it's part of the function, so that happens for the host and venv | 19:23 |
| evrardjp[m] | yeah | 19:25 |
| evrardjp[m] | sounds good. | 19:25 |
| evrardjp[m] | missed that. it would be good enough for stable branches, assuming we do this 3.3 thing everywhere. | 19:26 |
| evrardjp[m] | cloudnull: can you bring the other patches in? I will review them and remove my votes. | 19:26 |
| odyssey4me | tyvm evrardjp[m] for taking time out of your evening | 19:27 |
| cloudnull | sure. | 19:27 |
| evrardjp[m] | upgrade my votes at least. Hoping no issue and no pip10 will reach the hosts | 19:27 |
| evrardjp[m] | thanks | 19:27 |
| evrardjp[m] | i said cloudnull but it could as well be tux__ or d34dh0r53 | 19:28 |
| evrardjp[m] | i don't want to add more work to cloudnull particularily ;) | 19:28 |
| * cloudnull appreciates that | 19:29 | |
| odyssey4me | :) | 19:29 |
| evrardjp[m] | odyssey4me: lady is giving me the looks already, I should probably do something :p | 19:29 |
| odyssey4me | haha, be with your partner... sorry for the interruption | 19:29 |
| odyssey4me | cloudnull: are you picking the backports of https://review.openstack.org/#/q/topic:bug/1779534+is:starred up, or shall we let tux_ do it if he wishes? | 19:30 |
| tux_ | evrardjp[m]: what i need to do, just to make sure i am on same page.. I will build OSA with 16.0.15 soon once i finish my other duty.. | 19:30 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 19:31 |
| odyssey4me | lol, I should probably have starred by or whatever | 19:31 |
| cloudnull | odyssey4me im good either way? | 19:32 |
| cloudnull | d34dh0r53 tux_? | 19:32 |
| odyssey4me | it'd be https://review.openstack.org/583331 and https://review.openstack.org/583342 and https://review.openstack.org/583298 | 19:32 |
| odyssey4me | 2 I think can just be done in gerrit, one will need fixing for queens, then can be ported from gerrit for the rest | 19:33 |
| tux_ | cloudnull: i have setup my lab currently i am running 16.0.15 so i will try to downgrade and then upgrade to test pip, otherwise i can try 17.0.X to give it a try | 19:36 |
| jrosser | odyssey4me: we ran an aio today with those 3 stubborn patches starting at https://review.openstack.org/564422. it looks like it fails here the same way in the gate | 19:36 |
| odyssey4me | jrosser: orly? that's good news - I'm busy doing exactly that - did you figure anything out about what the cause might be? | 19:37 |
| jrosser | not yet - only that this is the point it bails out http://paste.openstack.org/show/726305/ | 19:38 |
| jrosser | thats all done in jenkins/vagrant so theres nothing to go poking at after it finishes | 19:40 |
| odyssey4me | yep, my guess was that somehow the reverse proxy is failing | 19:40 |
| odyssey4me | I thought it was DNS resolution, but no - some test patches ruled that out. | 19:41 |
| jrosser | but at least theres now something running regularly in a proxied environment so we'll see problems earlier | 19:41 |
| odyssey4me | I am very seriously toying with the idea of reverting the nginx reverse proxy to get queens back to the state of pike with regards to the repo... but I have no idea what dragons lurk due to the volume of intermingled changes that were made. | 19:42 |
| jrosser | the only thing thing that stands out in that error is that all the pythion version references are to 3.x | 19:42 |
| odyssey4me | oh wait, that died when trying to do galera_client on the repo container itself | 19:42 |
| jrosser | which may be bogus, but looks wierd | 19:43 |
| odyssey4me | here I have the repo happily building :/ | 19:43 |
| jrosser | let me look again | 19:43 |
| odyssey4me | oh no, those are standard warnings if your constraints file has py3 references... they're totally cosmetic | 19:43 |
| odyssey4me | the real problem there is 'No matching distribution found for MySQL-python' | 19:44 |
| odyssey4me | hmm, and my build is running smoothly :/ | 19:45 |
| odyssey4me | it may have to do with what's on the host at the start... my host isn't totally clean | 19:45 |
| jrosser | so that looks a bit like this? http://logs.openstack.org/22/564422/8/check/openstack-ansible-deploy-aio_lxc-ubuntu-xenial/b1fb7d8/job-output.txt.gz#_2018-07-17_20_35_52_307725http://logs.openstack.org/22/564422/8/check/openstack-ansible-deploy-aio_lxc-ubuntu-xenial/b1fb7d8/job-output.txt.gz#_2018-07-17_20_35_52_307725 | 19:45 |
| jrosser | argh | 19:45 |
| jrosser | http://logs.openstack.org/22/564422/8/check/openstack-ansible-deploy-aio_lxc-ubuntu-xenial/b1fb7d8/job-output.txt.gz#_2018-07-17_20_35_52_307725 | 19:45 |
| odyssey4me | so check this: http://logs.openstack.org/22/564422/8/check/openstack-ansible-deploy-aio_lxc-ubuntu-xenial/b1fb7d8/logs/openstack/aio1_repo_container-c0c7b5e0/pypiserver/pypiserver.log | 19:46 |
| jrosser | as far as i can see my build has failed just like that one | 19:46 |
| odyssey4me | clearly the package query came in | 19:46 |
| odyssey4me | and see this: http://logs.openstack.org/22/564422/8/check/openstack-ansible-deploy-aio_lxc-ubuntu-xenial/b1fb7d8/logs/openstack/aio1_repo_container-c0c7b5e0/nginx/openstack-slushee.error.log | 19:47 |
| odyssey4me | 'upstream server temporarily disabled while connecting to upstream' is what I've suspected as the culprit | 19:48 |
| jrosser | err ipv6 going on there? | 19:48 |
| odyssey4me | yep, that's why I tried https://review.openstack.org/#/c/571436/1 but same results | 19:48 |
| odyssey4me | so, given that this is all working nicely for pike - and nicely for master... perhaps the best strategy is to back out the reverse proxy thing for queens | 19:50 |
| jrosser | it would be nice to have an actual box to poke at this on | 19:50 |
| odyssey4me | yeah, the trouble is that I've tried several times and can't replicate it | 19:51 |
| jrosser | becasue it might be a minor config change somewhere to clobber the v6 resolution | 19:51 |
| odyssey4me | hmm, maybe I should try with virtualbox | 19:51 |
| odyssey4me | or perhaps the MNAIO tooling | 19:51 |
| jrosser | i was talking with rick today and we were going to to vagrant/virtualbox next | 19:52 |
| odyssey4me | well, another approach is to try using pypiserver as an extra-index (as the docs suggest) and have nginx be purely a reverse proxy for those paths | 19:54 |
| odyssey4me | ie nginx won't have to try pypiserver first, then upstream | 19:55 |
| odyssey4me | perhaps the results will be more consistent | 19:55 |
| odyssey4me | in master none of this matters because we no longer pip install things on the host, mostly - and those we do are after the repo build | 19:56 |
| jrosser | if your tests are somewhere with working v6 perhaps thats why it was ok | 19:56 |
| jrosser | i dont have any v6 which makes it out of the internal network | 19:56 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: [TEST] do not use ipv6 for nginx reverse proxy https://review.openstack.org/571436 | 19:57 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible master: [WIP] Add qdrouterd role for messaging backend deployment https://review.openstack.org/570261 | 19:57 |
| odyssey4me | restored and rebased those test patches, given we don't have those logs any more - it'd be good to see them again to see if the disabling of ipv6 resolution worked | 19:58 |
| jrosser | i need to add some artefact gathering to my jenkins job | 19:59 |
| jrosser | i bet it's the same sort of trouble | 19:59 |
| odyssey4me | hang on a minute - jrosser aren't you overriding the pypi address? | 19:59 |
| odyssey4me | ie https://github.com/openstack/openstack-ansible-repo_server/blob/master/templates/nginx-pypi.conf.j2#L10 | 20:00 |
| odyssey4me | so in your case, ipv6 is not a facter, because a fail if you have that override set to your local pypi repo means that something else broke | 20:00 |
| odyssey4me | *factor | 20:01 |
| jrosser | gimme a moment.... | 20:01 |
| odyssey4me | heh, in fact if I make sure that it overridden in our tests to the infra mirror, that might just work | 20:02 |
| jrosser | this is an aio, just on a box with the proxy env vars set | 20:02 |
| jrosser | i don't think it is trying to also use the pip mirror, but i need to double check that | 20:03 |
| odyssey4me | well, that may actually be the deciding factor here | 20:03 |
| odyssey4me | it may actually be an ipv6 routing/resolution issue | 20:03 |
| *** FrankZhang has quit IRC | 20:04 | |
| odyssey4me | :( we set the infra mirror already: https://github.com/openstack/openstack-ansible/blob/stable/queens/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#L190 | 20:06 |
| *** cshen has quit IRC | 20:06 | |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_rally stable/queens: Change rally pip package https://review.openstack.org/584089 | 20:09 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/queens: Update get-pip to version 3.3 https://review.openstack.org/584090 | 20:11 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/pike: Update get-pip to version 3.3 https://review.openstack.org/584091 | 20:11 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/ocata: Update get-pip to version 3.3 https://review.openstack.org/584092 | 20:12 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 20:12 |
| evrardjp | odyssey4me: do we really want a comment patch on master for https://review.openstack.org/#/c/583925/ ? | 20:13 |
| odyssey4me | sure, until that all changes it's important to address the status | 20:13 |
| odyssey4me | I can PR that comment if you like | 20:14 |
| evrardjp | I am fine with no comment as this is clear in the code, but ok if you like | 20:14 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_rally stable/queens: Change rally pip package https://review.openstack.org/584089 | 20:14 |
| evrardjp | we can backport that separately if we are in a hurry | 20:14 |
| evrardjp | removed the -w | 20:14 |
| odyssey4me | yeah, I'll do a patch to master | 20:14 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 20:15 |
| guilhermesp | odyssey4me I saw your comment on https://review.openstack.org/#/c/584089/ | 20:15 |
| guilhermesp | I will propose there now | 20:15 |
| evrardjp | I forgot to ask a question earlier for tux_ and cloudnull | 20:16 |
| *** cshen has joined #openstack-ansible | 20:16 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Pin get-pip.py to 3.2 https://review.openstack.org/584093 | 20:16 |
| evrardjp | the stacktracing we see | 20:16 |
| evrardjp | it makes sense if pip is downgraded | 20:17 |
| evrardjp | but you said it happened on an tag to tag upgrade | 20:17 |
| evrardjp | so I am not surprised we had a gap there that should be fixed by the pip install patch | 20:17 |
| evrardjp | but is there a need to bump to 3.3 ? | 20:17 |
| evrardjp | because new installs would not stacktrace with 3.2 | 20:18 |
| evrardjp | upgrades would take existing pip on the host -> everything is fine | 20:18 |
| evrardjp | There is a gap in my understanding | 20:18 |
| cloudnull | tux_ ^ I don't remember the version you had, and the version you went to ? | 20:18 |
| evrardjp | when I meant upgrades I meant "OSA branch upgrades, whether tag per tag or major" | 20:18 |
| cloudnull | I believe it was pike to pike | 20:19 |
| cloudnull | but i dont remember the tag numbers. | 20:19 |
| evrardjp | yeah so this patch in not in pike so that would explain | 20:19 |
| tux_ | cloudnull: i was on 16.0.14 and trying to upgrade 16.0.15 | 20:19 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Update get-pip to version 3.3 https://review.openstack.org/584095 | 20:19 |
| vakuznet | upgrade from 17.0.5 to 17.0.6 failed for me with get-pip 3.2 | 20:19 |
| evrardjp | tux_: and you had pip installed on the host, right? | 20:19 |
| evrardjp | vakuznet: stacktrace too? | 20:19 |
| vakuznet | yes | 20:20 |
| evrardjp | vakuznet: centos too? | 20:20 |
| tux_ | yes i had pip 9.0.3 already | 20:20 |
| vakuznet | hoyes | 20:20 |
| evrardjp | ok makes all sense | 20:20 |
| evrardjp | so if we were to use pip on the host it would have been just fine | 20:20 |
| evrardjp | so there is no need to bump to 3.3 | 20:21 |
| evrardjp | 3.3 could fix it | 20:21 |
| vakuznet | 3.3 bump fixed this for me | 20:21 |
| evrardjp | and IS fixing it (by running newer code) | 20:21 |
| evrardjp | but is not needed | 20:21 |
| evrardjp | per se | 20:21 |
| evrardjp | yeah | 20:21 |
| cloudnull | it didnt seem like 3.2 worked at all. | 20:21 |
| evrardjp | well it worked for us for years :p | 20:21 |
| evrardjp | or months at least | 20:22 |
| cloudnull | I believe it was unpinned for the most part | 20:22 |
| *** vnogin has joined #openstack-ansible | 20:22 | |
| evrardjp | it was | 20:22 |
| evrardjp | but it was effectively the 3.2 | 20:22 |
| evrardjp | the problem is the new package that went out 18 days ago | 20:22 |
| evrardjp | python-virtualenv and others | 20:22 |
| evrardjp | the problem is that we have pip installed on the host and try to run a very old pip from 3.2 | 20:23 |
| vakuznet | yesterday i upgraded other environment with older repo server and it worked with get-pip 3.2 | 20:23 |
| evrardjp | that's bad | 20:23 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Update get-pip to version 3.3 https://review.openstack.org/584095 | 20:23 |
| evrardjp | vakuznet: because it had no pip installed on the node | 20:23 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_rally master: Change rally pip package https://review.openstack.org/584096 | 20:23 |
| tux_ | here is the details https://bugs.launchpad.net/openstack-ansible/+bug/1782051 | 20:23 |
| openstack | Launchpad bug 1779534 in openstack-ansible "duplicate for #1782051 pip install offline fails with new version of get-pip.py" [High,Fix released] - Assigned to Jesse Pretorius (jesse-pretorius) | 20:23 |
| evrardjp | so I confirm my position on the absence of need of get-pip 3.3 if the other patch to pip install go in | 20:23 |
| evrardjp | tux_: I read the bug, and for me, I fixed it | 20:24 |
| tux_ | evrardjp: it was failing here | 20:24 |
| vakuznet | how repo server can not have pip?? | 20:24 |
| evrardjp | tux_: but you didn't try my patch, did you? | 20:24 |
| tux_ | no i didn't :( | 20:24 |
| tux_ | dealing with production issue :( | 20:24 |
| evrardjp | anyway I am fine with backporting 3.3 | 20:24 |
| tux_ | I will try soon but and report back | 20:24 |
| evrardjp | I just way beyond the point of caring | 20:24 |
| evrardjp | if ppl want to backport, fine for me :) | 20:25 |
| evrardjp | I just think it's not a patch targetting the root cause, but the effects. | 20:25 |
| odyssey4me | evrardjp: both is good | 20:26 |
| evrardjp | odyssey4me: agreed. | 20:26 |
| *** vnogin has quit IRC | 20:26 | |
| evrardjp | that's what we said earlier IIRC :p | 20:26 |
| jrosser | btw evrardjp i had a very good day with json_query today after seeing you'd used that | 20:26 |
| evrardjp | jrosser: :) | 20:26 |
| evrardjp | jrosser: I love it | 20:27 |
| jrosser | it basically saved my life with all the changes to the nxos_* modules in 2.6 | 20:27 |
| evrardjp | it's been a while I've been using it, and I have a bug in OSA to make use more of it | 20:27 |
| evrardjp | let me find that | 20:27 |
| evrardjp | jrosser: if you want to tackle it: https://bugs.launchpad.net/openstack-ansible/+bug/1769100 | 20:28 |
| openstack | Launchpad bug 1769100 in openstack-ansible "Use json_query whenever possible" [Wishlist,Confirmed] | 20:28 |
| evrardjp | all of these conditions are one liners | 20:28 |
| evrardjp | odyssey4me: you worked on the tempest issue right? | 20:29 |
| evrardjp | cloudnull: talking about pining of 3.2 or whatever,I prefer the situation now, I prefer it's pinned. It's safer for the future. I think we are in a place that's VERY stable, compared to what I remember from Kilo : ) | 20:29 |
| evrardjp | good old times :) | 20:29 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: Update get-pip to version 3.3 https://review.openstack.org/584098 | 20:30 |
| cloudnull | ah the good old days | 20:30 |
| odyssey4me | heh, any new pypi release and all hell broke loose | 20:30 |
| jrosser | odyssey4me: i have an aio started off to look see if it is v6 or something else | 20:30 |
| odyssey4me | pip/setuptools would break stable deployments every week | 20:30 |
| evrardjp | hahah | 20:31 |
| jrosser | that will be sufficiently cooked by the morning to see whats happened | 20:31 |
| cloudnull | pip 2.6 for life ! | 20:31 |
| evrardjp | cloudnull: lol | 20:31 |
| evrardjp | easy_install fowever | 20:31 |
| odyssey4me | distro packages would overwrite pip packages (we have no venvs then) | 20:31 |
| evrardjp | yeah! | 20:31 |
| evrardjp | we've gone a long way | 20:31 |
| cloudnull | ah memories | 20:31 |
| odyssey4me | oh yeah, and ansible 1.5 or so IIRC | 20:31 |
| odyssey4me | might even have been 1.3 | 20:32 |
| evrardjp | nah | 20:32 |
| odyssey4me | you know, its own var precedence was a thing of beauty | 20:32 |
| evrardjp | It was already 1.5 | 20:32 |
| evrardjp | I am pretty sure it was 1.5 or above | 20:32 |
| odyssey4me | depending which way the wind blew, it decided what should be higher | 20:32 |
| evrardjp | but I can be wrong ofc | 20:32 |
| evrardjp | hahaha | 20:32 |
| evrardjp | true | 20:32 |
| evrardjp | well | 20:32 |
| cloudnull | 1.3 was icehouse / havana trials | 20:32 |
| tux_ | Good policy is don't upgrade pip until we need it.. minor OSA upgrade shouldn't touch existing pip | 20:32 |
| evrardjp | it was mostly 1.5/1.6/no change in precedence/1.9.2/1.9.3/1.9.5/(can't remember if there was more)/2.0 | 20:33 |
| cloudnull | hell the first crack at a lot of this was chef :P | 20:33 |
| evrardjp | tux_: that's what I've trying to keep -- that's why get-pip 3.3 is scary to me: it's bringing (if a problem of any kind) a runtime of pip10 ... ! scary! | 20:34 |
| evrardjp | hahaha | 20:34 |
| evrardjp | cloudnull: that's WAY before my time. | 20:34 |
| evrardjp | I didn't know what openstack was at that time. | 20:34 |
| evrardjp | jrosser: odyssey4me could you vote on https://review.openstack.org/#/c/583568/ ? | 20:35 |
| evrardjp | same for https://review.openstack.org/#/c/575715/ | 20:35 |
| cloudnull | https://github.com/cloudnull/chef-lxc-openstack | 20:35 |
| cloudnull | there's a walk down memory lane :) | 20:35 |
| evrardjp | cloudnull: so for M3, I am bringing all my patches into the sha bump now. | 20:36 |
| cloudnull | woot! | 20:36 |
| tux_ | I have 16.0.7 running on my OLD lab i am going to upgrade to latest 16.0.x will report you then | 20:38 |
| cloudnull | thanks tux_ | 20:38 |
| cloudnull | really appreciate you running through all that again . | 20:38 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Update get-pip to version 3.3 https://review.openstack.org/584101 | 20:39 |
| odyssey4me | I already voted for https://review.openstack.org/#/c/575715/ - someone else needs to | 20:40 |
| jrosser | evrardjp: https://review.openstack.org/#/c/575715/ <- is there a same thing for unsupported architecture, arm for example | 20:41 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Update get-pip to version 3.3 https://review.openstack.org/584095 | 20:41 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Update all SHAs for milestone 3 https://review.openstack.org/574006 | 20:42 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Update get-pip to version 3.3 https://review.openstack.org/584095 | 20:42 |
| evrardjp | jrosser: isn't arm supported? | 20:43 |
| evrardjp | I mean not officially, but you can try, right? | 20:43 |
| jrosser | i have a vauge memory of there being a check for that somewhere | 20:44 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/queens: Update get-pip to version 3.3 https://review.openstack.org/584090 | 20:44 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-pip_install stable/ocata: Update get-pip to version 3.3 https://review.openstack.org/584109 | 20:45 |
| evrardjp | jrosser: maybe in repo build? | 20:46 |
| logan- | fyi all we'll have to watch out for this a bit while adding bionic support https://review.openstack.org/#/c/583708/ | 20:46 |
| logan- | i had a chat in infra about it yesterday http://paste.openstack.org/raw/726308/ | 20:46 |
| jrosser | perhaps, i was only thinking that if there was a switch for unsupported distros then it might want similar unusupported arch one | 20:46 |
| odyssey4me | arm apparently does work, along with s380 and the other big iron I forget | 20:46 |
| odyssey4me | s390x I mean | 20:46 |
| logan- | it probably won't affect many other packages but its worth keeping an eye out for | 20:46 |
| *** FrankZhang has joined #openstack-ansible | 20:46 | |
| odyssey4me | oh of course, POWER :p | 20:47 |
| evrardjp | yeah :) | 20:47 |
| logan- | we had openstack_hosts fake passing for bionic with packages that dont exist upstream | 20:47 |
| evrardjp | logan-: oh shit | 20:47 |
| evrardjp | I will have a look | 20:47 |
| odyssey4me | oh wow, that's no good | 20:47 |
| evrardjp | logan-: on top of it, that's pretty much a big deal, isn't that the python2 thing to have the add-apt-repository cli? | 20:49 |
| logan- | yep | 20:50 |
| *** tux_ has quit IRC | 20:50 | |
| logan- | i think that package also makes lsb_release work too | 20:50 |
| logan- | iirc | 20:50 |
| odyssey4me | ok, time for me to !computer - night folks | 20:56 |
| *** DanyC has joined #openstack-ansible | 20:57 | |
| odyssey4me | evrardjp: fyi, tempest passed with https://review.openstack.org/584037 | 20:57 |
| admin0 | can ceph-install playbook be run after setup hosts ? | 20:57 |
| odyssey4me | which depends on https://review.openstack.org/584033 | 20:57 |
| admin0 | i guess it does not share infra ? | 20:57 |
| odyssey4me | cores - please review the above-mentioned, and https://review.openstack.org/583672 | 20:58 |
| jrosser | odyssey4me: i got the same repo fail | 20:59 |
| odyssey4me | looks like https://review.openstack.org/584052 got https://review.openstack.org/583366 passing too | 20:59 |
| odyssey4me | ah, and magnum's passing again: https://review.openstack.org/584042 | 20:59 |
| *** parasitid has quit IRC | 20:59 | |
| *** dave-mccowan has quit IRC | 21:00 | |
| odyssey4me | jrosser: ipv6 or not? | 21:00 |
| jrosser | http://paste.openstack.org/show/726309/ | 21:00 |
| jrosser | oh thats only part of it, let me try that again | 21:01 |
| odyssey4me | hmm , looks like it tried both ipv4 and 6 | 21:01 |
| jrosser | right, yes | 21:01 |
| jrosser | so that says to me that the http proxy setting hasnt made it to there | 21:01 |
| odyssey4me | yeah, that makes sense | 21:01 |
| jrosser | as theres no v4 route anyway | 21:01 |
| *** dave-mccowan has joined #openstack-ansible | 21:01 | |
| jrosser | that will be a slightly different issue to the one in the gate though | 21:02 |
| *** armaan has joined #openstack-ansible | 21:02 | |
| odyssey4me | ok, will have to pick that up tomorrow - but it seems to me that trying to rev proxy pypi is a bad idea... better for us to do what we did in pike... access directly before the repo build is there, then use the repo build | 21:02 |
| odyssey4me | we can still use pypiserver, but ditch the rev proxy of pypi | 21:03 |
| jrosser | yeah, the reverse proxy has to know about using a fwd proxy in my case | 21:03 |
| odyssey4me | this issue has stood too long, and master changes to sort this all out anyway | 21:03 |
| odyssey4me | ok, I'll work up some patches tomorrow | 21:03 |
| jrosser | cool, just shout if you want anything testing | 21:04 |
| odyssey4me | night all | 21:04 |
| evrardjp | odyssey4me: cool | 21:09 |
| evrardjp | odyssey4me: gn | 21:09 |
| *** armaan has quit IRC | 21:14 | |
| *** parasitid has joined #openstack-ansible | 21:18 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_tempest master: Disable Volume Backup Tests https://review.openstack.org/583955 | 21:18 |
| *** jwitko has quit IRC | 21:26 | |
| *** DanyC has quit IRC | 21:26 | |
| openstackgerrit | Merged openstack/openstack-ansible-pip_install stable/queens: Ensure get-pip is not used when good version of pip is present https://review.openstack.org/583925 | 21:28 |
| evrardjp | jrosser: what do you think of https://review.openstack.org/#/c/583879/ ? | 21:28 |
| evrardjp | The idea is to reduce the amount of tasks that return "changed" when they do nothing. | 21:29 |
| evrardjp | the less "changed" tasks on a re-run, the happier I am. | 21:29 |
| evrardjp | I like green more than yellow on an ansible run | 21:29 |
| *** Guest34 has joined #openstack-ansible | 21:32 | |
| *** Guest34 is now known as mike___brennan | 21:32 | |
| *** kstev has quit IRC | 21:35 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_container_create master: Update qgroup setup to be more explicit https://review.openstack.org/584122 | 21:41 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: [WIP] Add functional tests to elk_metrics_6x https://review.openstack.org/583794 | 21:43 |
| jrosser | evrardjp: that looks good | 21:44 |
| mnaser | gah | 21:50 |
| mnaser | nginx repo keeps being flakey. | 21:50 |
| mnaser | so right now repo_server and os_keystone both use nginx | 21:52 |
| mnaser | i think it's time we write a basic nginx role that they can both depend on, or maybe use one that's available already | 21:52 |
| mnaser | it'll clean up the bare metal story | 21:53 |
| *** hamzy has quit IRC | 21:53 | |
| mnaser | and it will also make it all unifed in one place | 21:53 |
| *** hamzy has joined #openstack-ansible | 21:53 | |
| mnaser | https://github.com/nginxinc/ansible-role-nginx oh this is cool | 21:54 |
| cloudnull | I wonder if they'll take a PR for suse 42 ? | 21:56 |
| *** cshen has quit IRC | 21:57 | |
| evrardjp | mnaser: good find. | 22:01 |
| evrardjp | else IIRC jdauphant's nginx role was not too bad | 22:01 |
| evrardjp | but yeah using an existing role is :thumbs_up: | 22:02 |
| mnaser | it already has suse support looks like cloudnull | 22:03 |
| mnaser | oh | 22:03 |
| mnaser | 42 | 22:03 |
| mnaser | thing is i think the problem with all these roles is they assume a single run will contain all the vhosts which we cant do | 22:04 |
| mnaser | we'd probably want to have 2 roles | 22:05 |
| mnaser | one that does the install/config, and one for every 'vhost' | 22:05 |
| mnaser | that way one system doesnt wipe out the other | 22:06 |
| mnaser | do we have an OSA boilerplate? | 22:06 |
| mnaser | (for roles) | 22:06 |
| logan- | doesn't nginx do conf.d? maybe you could use their role without the vhosts and then template your vhosts into the conf.d and reload/restart | 22:08 |
| mnaser | logan-: it does, i guess i was just trying to over-clean-up and get all the nginx stuff in teh same 'codepath' | 22:08 |
| logan- | https://github.com/evrardjp/osa-role-boilerplate | 22:09 |
| mnaser | https://github.com/nginxinc/ansible-role-nginx/blob/master/tasks/conf/push-config.yml | 22:10 |
| mnaser | it looks like this is fairly usable though | 22:10 |
| mnaser | non-namespaced vars make me sad. | 22:11 |
| logan- | yeah :/ | 22:11 |
| mnaser | gonna see if https://github.com/jdauphant/ansible-role-nginx can do the (as in doesnt override stuff) | 22:11 |
| evrardjp | logan-: that's 2017! | 22:12 |
| mnaser | ok | 22:12 |
| logan- | evrardjp: yeah :( i couldn't find a more recent example lol | 22:12 |
| mnaser | https://github.com/jdauphant/ansible-role-nginx | 22:12 |
| mnaser | it looks like this might do the job because | 22:12 |
| evrardjp | of what? | 22:12 |
| logan- | mnaser was looking for a role template | 22:13 |
| mnaser | https://github.com/jdauphant/ansible-role-nginx/blob/0d9019cf89283b878f0e077f5da4d2748b7c352a/tasks/configuration.yml#L18-L25 | 22:13 |
| mnaser | this just gets templated | 22:13 |
| mnaser | https://github.com/jdauphant/ansible-role-nginx/blob/0d9019cf89283b878f0e077f5da4d2748b7c352a/tasks/main.yml#L21-L23 | 22:13 |
| evrardjp | oh ok | 22:14 |
| mnaser | err thats an older version but | 22:14 |
| mnaser | https://github.com/jdauphant/ansible-role-nginx/blob/master/tasks/main.yml#L29-L31 | 22:14 |
| mnaser | we can use that to not let it remove stuff | 22:14 |
| evrardjp | it was to use with https://review.openstack.org/#/c/458562/ | 22:14 |
| evrardjp | iirc | 22:15 |
| evrardjp | that's long ago | 22:15 |
| evrardjp | I think there was one with cookie cutter but I wasn't really happy with it | 22:15 |
| evrardjp | and it appeared after I started with this | 22:16 |
| evrardjp | but before I spent time on a refresh | 22:16 |
| mnaser | ok i think i might hack on os_keystone to make it use that role | 22:16 |
| mnaser | that way we don't have to manage stuff anymore | 22:16 |
| *** hachi has quit IRC | 22:17 | |
| mnaser | because centos 7 is failing unfortunately because of this | 22:18 |
| mnaser | i'd love to have the distro support merge with passing centos | 22:18 |
| *** cshen has joined #openstack-ansible | 22:23 | |
| evrardjp | mnaser: you think the next move is to only have package installs for centos? | 22:24 |
| evrardjp | I'd not be against | 22:24 |
| mnaser | evrardjp: id like to have both working but honestly i'd be more motivated to maintain packages. | 22:24 |
| evrardjp | :D | 22:24 |
| mnaser | it might be a lot more work overall | 22:24 |
| evrardjp | yeah, the load is spread | 22:24 |
| evrardjp | you think? | 22:24 |
| evrardjp | oh you mean to maintain both | 22:25 |
| mnaser | it might not be, we'll see how it goes | 22:25 |
| evrardjp | yeah | 22:25 |
| mnaser | centos integrated hasnt passed in ages but it looks like there is something small left | 22:25 |
| mnaser | its trying to use tgtadm even if its configured to iscsi_helper=lioadm | 22:25 |
| mnaser | so im wondering if something different is happening in integrated repo... | 22:25 |
| mnaser | lol | 22:26 |
| mnaser | http://logs.openstack.org/52/584052/1/check/openstack-ansible-functional-distro_install-centos-7/ebcccb9/logs/ara-report/result/969351a2-318a-4e3e-a699-32763f2835e8/ | 22:27 |
| mnaser | we're not testing much. | 22:27 |
| *** cshen has quit IRC | 22:27 | |
| evrardjp | we just disabled cinder_backup | 22:28 |
| evrardjp | like devstack! | 22:28 |
| evrardjp | for the rest I don't know :p | 22:28 |
| mnaser | ok i know whats going on here | 22:28 |
| evrardjp | if the testing is not well done, we should rethink it | 22:29 |
| evrardjp | mnaser: maybe the scenario doesn't have things in the whitelist | 22:29 |
| evrardjp | https://github.com/openstack/openstack-ansible-os_cinder/blob/master/tests/os_cinder-overrides.yml#L28 | 22:29 |
| evrardjp | I've discovered recently we don't cover things properly | 22:30 |
| evrardjp | We should have BOTH tempest plugins and tempest core testing whitelisted | 22:30 |
| evrardjp | not sure it applies in your case | 22:30 |
| openstackgerrit | Merged openstack/openstack-ansible master: Remove useless group var https://review.openstack.org/583568 | 22:31 |
| evrardjp | I'm off to bed, if you have things I need to continue, ping that to me mnaser | 22:32 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Change deprecated option iscsi_helper to target_helper https://review.openstack.org/584151 | 22:35 |
| *** tux_ has joined #openstack-ansible | 22:36 | |
| mnaser | evrardjp cloudnull logan- ^ | 22:36 |
| mnaser | i think that migth be the last missing step to get centos to pass, i think the deprecated paramters are not being copied to default_backend properly | 22:36 |
| *** tux_ has quit IRC | 22:40 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Allow a bypass of operating system https://review.openstack.org/575715 | 22:55 |
| *** vnogin has joined #openstack-ansible | 23:08 | |
| *** chyka_ has quit IRC | 23:12 | |
| *** tux_ has joined #openstack-ansible | 23:20 | |
| *** vnogin has quit IRC | 23:37 | |
| *** pmannidi has joined #openstack-ansible | 23:49 | |
| *** poopcat has quit IRC | 23:53 | |
| *** poopcat has joined #openstack-ansible | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!