| *** cjloader has quit IRC | 00:01 | |
| *** lbragstad has quit IRC | 00:17 | |
| *** ashak has joined #openstack-ansible | 00:23 | |
| mmercer | ouch, thats what i get for not actually reading the script first.... forcefully wipes out existing partitions on a multi disk system regardless of how the system is partitions | 00:37 |
|---|---|---|
| *** cjloader has joined #openstack-ansible | 00:46 | |
| *** chigang__ has joined #openstack-ansible | 00:55 | |
| *** sep__ has quit IRC | 01:03 | |
| *** spine55 has quit IRC | 01:03 | |
| *** sep__ has joined #openstack-ansible | 01:03 | |
| *** spine55 has joined #openstack-ansible | 01:03 | |
| *** threestrands has joined #openstack-ansible | 01:05 | |
| *** pmannidi has quit IRC | 01:11 | |
| *** vnogin has joined #openstack-ansible | 01:19 | |
| *** pmannidi has joined #openstack-ansible | 01:20 | |
| *** thedini1 has joined #openstack-ansible | 01:22 | |
| thedini1 | has anyone been playing with ansible-hardening lately | 01:23 |
| *** vnogin has quit IRC | 01:23 | |
| cloudnull | thedini1: I use it all the time. | 01:33 |
| *** spine55 has quit IRC | 01:37 | |
| thedini1 | cloudnull: i have been playing around with it and now I am getting a little more serious and was wondering how to start contributing back | 01:39 |
| cloudnull | ah, have you ever contributed to openstack before ? | 01:40 |
| cloudnull | heres the complete getting started guide if you've not | 01:41 |
| cloudnull | https://docs.openstack.org/contributors/code-and-documentation/index.html | 01:41 |
| thedini1 | nope... havn't contributed anywhere. THANKS | 01:41 |
| cloudnull | relevant parts are https://docs.openstack.org/contributors/common/accounts.html | 01:41 |
| cloudnull | and https://docs.openstack.org/contributors/common/setup-gerrit.html | 01:41 |
| cloudnull | once you have an account and have setup gerrit you're off to the races. | 01:42 |
| cloudnull | the biggest change in your git workflow will be from `git push` to `git review` | 01:42 |
| cloudnull | once you are off and running, and contributing, you'll see your changes here https://review.openstack.org/#/q/project:openstack/ansible-hardening | 01:43 |
| cloudnull | if you're ever just wanting to review patches, we'd greatly appreciate it! | 01:43 |
| thedini1 | kk, I will start getting more involved | 01:44 |
| hw_wutianwei_ | cloudnull: odyssey4me logan- hi, I met a issue in TASK [lxc_host: Ensure that the LXC cache has been prepared], when I deploy pike and queens. Here is the log http://paste.openstack.org/show/720253/, could you tell me how to solve? | 01:44 |
| cloudnull | hw_wutianwei_: I think that's related to https://bugs.launchpad.net/openstack-ansible/+bug/1768592 | 01:45 |
| openstack | Launchpad bug 1768592 in openstack-ansible "/usr/local/bin/cache-prep-commands.sh failing to write to /etc/resolv.conf when resolvconf used on Xenial" [Undecided,New] | 01:45 |
| hw_wutianwei_ | cloudnull: have you fixed this bug? | 01:46 |
| cloudnull | I've not. | 01:47 |
| cloudnull | I just searched for it and found it :) | 01:47 |
| hw_wutianwei_ | :) | 01:48 |
| cloudnull | to fix that I'd try changing https://github.com/openstack/openstack-ansible-lxc_hosts/blob/stable/pike/vars/ubuntu-16.04.yml#L65-L67 to just "rm /etc/resolv.conf" | 01:48 |
| cloudnull | which would just delete the file allowing the script to create the new one. | 01:49 |
| cloudnull | hw_wutianwei_: mind sharing the contents of "/var/log/lxc-cache-prep-commands.log" on the host ? | 01:49 |
| hw_wutianwei_ | cloudnull: ok, wait a moment | 01:50 |
| cloudnull | this is what we do in master | 01:52 |
| cloudnull | https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/templates/prep-scripts/_container_sys_setup.sh.j2#L17-L21 | 01:52 |
| cloudnull | which is basically delete the file and create it :) | 01:52 |
| * cloudnull goes to make a patch | 01:52 | |
| hw_wutianwei_ | cloudnull: + mkdir -p /etc/ansible/facts.d/ | 01:53 |
| hw_wutianwei_ | + '[' -a /etc/resolv.conf ']' | 01:53 |
| hw_wutianwei_ | + echo 'nameserver 10.0.3.1' | 01:53 |
| hw_wutianwei_ | /usr/local/bin/cache-prep-commands.sh: line 8: /etc/resolv.conf: No such file or directory | 01:53 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_hosts stable/queens: Fix lxc cache prep resolvers https://review.openstack.org/565933 | 01:57 |
| cloudnull | hw_wutianwei_: ^ | 01:58 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_hosts stable/queens: Fix lxc cache prep resolvers https://review.openstack.org/565933 | 01:58 |
| cloudnull | so with that change it should do the right things | 02:00 |
| cloudnull | looks like recent releases of the lxc image from upstream are breaking things | 02:00 |
| *** cjloader has quit IRC | 02:07 | |
| *** cjloader has joined #openstack-ansible | 02:08 | |
| *** cjloader has quit IRC | 02:12 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Convert rsyslog to an include_task with group_vars https://review.openstack.org/556002 | 02:23 |
| cloudnull | hw_wutianwei_: did that work for you ? | 02:23 |
| *** lbragstad has joined #openstack-ansible | 02:24 | |
| hw_wutianwei_ | cloudnull: I am testing. when rm resolv.conf manual, it works. | 02:25 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Convert rsyslog to an include_task with group_vars https://review.openstack.org/556002 | 02:25 |
| cloudnull | hw_wutianwei_: cool! | 02:26 |
| *** mwarad has joined #openstack-ansible | 02:28 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-rabbitmq_server master: Tune-up the rabbitmq role for efficiency https://review.openstack.org/524028 | 02:43 |
| *** thedini2 has joined #openstack-ansible | 02:44 | |
| *** thedini1 has quit IRC | 02:46 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-openstack_hosts master: Add IPv6 kernel module https://review.openstack.org/565940 | 02:49 |
| *** thedini2 has quit IRC | 02:53 | |
| *** spsurya has joined #openstack-ansible | 02:54 | |
| *** dxiri has quit IRC | 03:09 | |
| *** mma has joined #openstack-ansible | 03:09 | |
| *** thedini2 has joined #openstack-ansible | 03:10 | |
| *** mma has quit IRC | 03:13 | |
| *** cjloader has joined #openstack-ansible | 03:14 | |
| *** cjloader has quit IRC | 03:18 | |
| *** thedini3 has joined #openstack-ansible | 03:21 | |
| *** nicolasbock has quit IRC | 03:21 | |
| *** thedini2 has quit IRC | 03:23 | |
| *** hamzy has joined #openstack-ansible | 03:34 | |
| *** udesale has joined #openstack-ansible | 03:36 | |
| *** cjloader has joined #openstack-ansible | 03:55 | |
| *** cjloader has quit IRC | 03:59 | |
| *** poopcat has quit IRC | 04:03 | |
| *** cjloader has joined #openstack-ansible | 04:13 | |
| *** cjloader has quit IRC | 04:17 | |
| *** vnogin has joined #openstack-ansible | 04:19 | |
| *** vnogin has quit IRC | 04:23 | |
| *** thedini3 has quit IRC | 04:25 | |
| *** lhinds has quit IRC | 04:29 | |
| *** portante has quit IRC | 04:29 | |
| *** lhinds has joined #openstack-ansible | 04:31 | |
| *** pabelanger has quit IRC | 04:31 | |
| *** portante has joined #openstack-ansible | 04:31 | |
| *** pabelanger has joined #openstack-ansible | 04:32 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Automatically prune the inventory backup https://review.openstack.org/565950 | 04:35 |
| *** Taseer has joined #openstack-ansible | 04:37 | |
| *** ianychoi_ has joined #openstack-ansible | 04:38 | |
| *** ianychoi has quit IRC | 04:41 | |
| *** gyee has quit IRC | 04:41 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Properly configure cinder-volume containers https://review.openstack.org/565951 | 04:44 |
| *** radeks_ has joined #openstack-ansible | 04:49 | |
| *** hachi has quit IRC | 04:59 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible stable/ocata: Backport nova placement api healthcheck from pike https://review.openstack.org/565955 | 05:01 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Install os-testr https://review.openstack.org/564776 | 05:02 |
| evrardjp | good morning | 05:04 |
| olivierb- | hello everyone, any thing changed in pike recently that could explain that I now have my AIO deployment not working anymore due to ansible-harderning ? The 1 serror I got was due to the fact that I only have 1 DNS server configured on my system (which I tricked very easily) and now I have a second stuff due to the NOPASSWD in my sudoers :-(. Looking at the latest changes I can not see something which could explain it was working | 05:04 |
| olivierb- | last week, any clue ? | 05:04 |
| evrardjp | mmm | 05:05 |
| olivierb- | morning evrardjp | 05:05 |
| evrardjp | could you repeat the issue for me? It's early in the morning I am not fully awake. | 05:06 |
| evrardjp | everything in hardening could be skipped on a case by case basis | 05:06 |
| evrardjp | NOPASSWD seems indeed something hardening would trip with | 05:06 |
| olivierb- | same for me, I think I'll go grab cup of coffee soon ;-) | 05:06 |
| evrardjp | right now bed is more tempting. | 05:07 |
| olivierb- | the problem is that with the same config (1 DNS + NOPASSWD) everything was deploying perfectly last week | 05:07 |
| olivierb- | today these 2 errors occured without me being able to tell why | 05:07 |
| evrardjp | could you paste the issue? | 05:07 |
| olivierb- | sure | 05:08 |
| evrardjp | did you update your roles? | 05:08 |
| olivierb- | not that I know of | 05:08 |
| evrardjp | ok | 05:08 |
| *** markvoelker has quit IRC | 05:08 | |
| evrardjp | so you think of an idempotency failure | 05:08 |
| olivierb- | no, because I even restarted my deployment from scratched machines :-( | 05:08 |
| olivierb- | brand new installation | 05:09 |
| olivierb- | and applying all steps which were working previously untouched | 05:09 |
| olivierb- | I think I somehow inherited some change in conf/env/... from I do not know where | 05:10 |
| olivierb- | DNS error seems related to ansible-hardening/tasks/rhel7stig/misc.yml: | 05:11 |
| olivierb- | DEBUG: [V-72281 - For systems using DNS resolution, at least two name servers must be configured.] | 05:11 |
| olivierb- | Two or more nameservers must be configured in /etc/resolv.conf. | 05:11 |
| olivierb- | Nameservers found: 1 | 05:11 |
| evrardjp | olivierb-: check the user variables. | 05:11 |
| olivierb- | can be easily tricked using a commented dummy line ;-) | 05:11 |
| evrardjp | in /etc/openstack_deploy/user_*.yml | 05:11 |
| evrardjp | that's bad | 05:11 |
| evrardjp | you can skip V-72281 for your case | 05:12 |
| olivierb- | nothing weird/different in my user variables as far as I can tell. Did not see any difference from last week | 05:13 |
| olivierb- | anything I should grep for in particular | 05:13 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_nova master: Add option to define the qemu security driver https://review.openstack.org/565958 | 05:13 |
| olivierb- | going for coffee right now, hoepfully mind should clear up a bit after this | 05:15 |
| evrardjp | olivierb-: then it's probably the re-roll of the machine | 05:18 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/ansible-hardening master: Add option to skip sudoers NOPASSWD check https://review.openstack.org/565959 | 05:22 |
| evrardjp | cloudnull: thanks for that patch | 05:27 |
| cloudnull | ? | 05:27 |
| evrardjp | I will check if the docs gets changed | 05:27 |
| evrardjp | for the skip sudoers with a var | 05:27 |
| cloudnull | oh, im just churning through the open bugs list and saw that . | 05:27 |
| cloudnull | is that an issue we're seeing elsewhere? | 05:27 |
| *** armaan has joined #openstack-ansible | 05:28 | |
| evrardjp | someone this morning wanted to by pass it, just a few lines above. | 05:29 |
| evrardjp | :) | 05:29 |
| evrardjp | I thought this was reactive to the conversation :D | 05:29 |
| olivierb- | evrardjp I do not think so because I am using same snapshots than last week to re-roll "machines" so I suspect something else | 05:29 |
| evrardjp | are you using the same openstack-ansible version? | 05:30 |
| olivierb- | and yes many thanks cloudnull for the patch, will try it right in a few moments | 05:30 |
| evrardjp | which sha is that? | 05:30 |
| evrardjp | well no cloudnull 's patch just makes it convenient for you to override | 05:31 |
| evrardjp | there is a regression somewhere we need to find out | 05:31 |
| evrardjp | (instead of using the skip tags) | 05:31 |
| evrardjp | olivierb-: what SHA are you using? | 05:31 |
| olivierb- | the openstack-ansible version is patched for pike as 2.3.3.0 has issue when running in reduced-connectivity mode | 05:31 |
| olivierb- | https://github.com/obourdon/ansible/commits/2.3-opennext-osa-pike | 05:32 |
| evrardjp | I am not sure what you mean | 05:33 |
| openstackgerrit | Christian Zunker proposed openstack/ansible-hardening master: Use absolute path for aide binary in cronjob https://review.openstack.org/565960 | 05:33 |
| evrardjp | olivierb-: which version of openstack-ansible? | 05:33 |
| olivierb- | ansible-playbook 2.3.3.0 | 05:33 |
| evrardjp | not answeringmy question | 05:34 |
| olivierb- | again as this is fixed it has not changed since last week | 05:34 |
| evrardjp | you're using ansible-hardening right? | 05:34 |
| evrardjp | which code version of ansible-hardening is that? | 05:34 |
| evrardjp | there is no magic, something must have drifted | 05:35 |
| olivierb- | frankly if I do use it it is not from my will, I inherit it from the instructions I use to deploy AIO | 05:35 |
| evrardjp | either it's in our code, or it's in your environment :) | 05:35 |
| evrardjp | olivierb-: hehe :) | 05:35 |
| evrardjp | it's an AIO with pike? | 05:36 |
| olivierb- | yes | 05:36 |
| evrardjp | is Pike restricted to a specific version? | 05:36 |
| olivierb- | of ansible yes 2.3.3 | 05:36 |
| evrardjp | or is that stable/pike of openstack-ansible ? | 05:36 |
| evrardjp | olivierb-: I don't care about ansible --version for now :) | 05:36 |
| evrardjp | let's do this: | 05:36 |
| evrardjp | could you do a git show inside /opt/openstack-ansible ? | 05:37 |
| olivierb- | sure | 05:37 |
| evrardjp | or alternatively show me your instructions :) | 05:37 |
| olivierb- | https://gist.github.com/obourdon/aabd34a08acfed6c51cb7026191b0869 | 05:38 |
| *** evin has quit IRC | 05:39 | |
| olivierb- | https://gist.github.com/obourdon/a75cbd2a2bbbd98de30005557ae7d886 | 05:39 |
| olivierb- | AFAIK nothing changed from this side since last week when everything was working fine | 05:40 |
| olivierb- | grabbing another cup of coffee, seems like I need it this morning sorry | 05:40 |
| evrardjp | olivierb-: haha | 05:42 |
| evrardjp | olivierb-: where do you fetch your /opt/openstack-ansible/ ? | 05:43 |
| evrardjp | in your instructions? | 05:43 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Configure cors for glance for additional usability https://review.openstack.org/565961 | 05:47 |
| olivierb- | /usr/local/bin/openstack-ansible and it should come from the scripts/bootstrap-ansible.sh | 05:50 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/ansible-hardening master: Add option to skip sudoers NOPASSWD check https://review.openstack.org/565959 | 05:51 |
| cloudnull | ok, im off. take care all | 05:52 |
| evrardjp | cloudnull: have a good night :) | 05:52 |
| evrardjp | olivierb-: that's true but that's not answering my question | 05:52 |
| olivierb- | and I just ran find on my system, this is the only one | 05:52 |
| evrardjp | I am not searching for the binary named openstack-ansible | 05:52 |
| evrardjp | (script I mean) | 05:52 |
| evrardjp | I am looking for where the scripts/bootstrap-ansible.sh is | 05:53 |
| evrardjp | it's in a folder, that folder has some code in it | 05:53 |
| evrardjp | that code is versioned. | 05:53 |
| evrardjp | that's the openstack-ansible version we are discussion | 05:53 |
| evrardjp | discussing* | 05:53 |
| olivierb- | /opt/openstack-ansible/scripts/bootstrap-ansible.sh from stable/pike + the slight modification to grab patched ansible 2.3.3 from the repo I pasted above: aka | 05:56 |
| olivierb- | https://raw.githubusercontent.com/obourdon/openstack-ansible/stable/pike/scripts/bootstrap-ansible.sh | 05:56 |
| olivierb- | should be synched with latest pike from official openstack-ansible repo | 05:56 |
| *** mma has joined #openstack-ansible | 05:57 | |
| evrardjp | olivierb-: ok so you're using a moving target stable/pike. | 05:58 |
| evrardjp | not a fixed sha | 05:58 |
| evrardjp | that could be the reason | 05:58 |
| evrardjp | please check what you had before, because the last sha bump didn't change anything for sudoers. | 05:59 |
| evrardjp | latest sha bump of ansible-hardening inside openstack-ansible | 05:59 |
| evrardjp | I have to go | 05:59 |
| *** DanyC has joined #openstack-ansible | 06:08 | |
| *** markvoelker has joined #openstack-ansible | 06:09 | |
| *** ppetit has joined #openstack-ansible | 06:11 | |
| *** cjloader has joined #openstack-ansible | 06:12 | |
| *** evin has joined #openstack-ansible | 06:12 | |
| *** cjloader has quit IRC | 06:17 | |
| *** DanyC has quit IRC | 06:24 | |
| openstackgerrit | lu.li proposed openstack/ansible-hardening master: Update the homepage url https://review.openstack.org/565975 | 06:26 |
| *** markvoelker has quit IRC | 06:44 | |
| *** chigang__ has quit IRC | 06:44 | |
| *** eumel8 has joined #openstack-ansible | 06:52 | |
| *** pcaruana has joined #openstack-ansible | 06:53 | |
| *** threestrands has quit IRC | 06:58 | |
| *** ianychoi_ is now known as ianychoi | 07:00 | |
| *** radeks_ has quit IRC | 07:03 | |
| *** chigang__ has joined #openstack-ansible | 07:03 | |
| olivierb- | can someone please explain this CI error to me ? http://logs.openstack.org/62/565762/1/check/openstack-ansible-deploy-aio_lxc-ubuntu-xenial/cee8e6d/logs/host/lxc-cache-prep-commands.log | 07:04 |
| *** pmannidi has quit IRC | 07:04 | |
| olivierb- | do I need to do something on my side ? | 07:04 |
| olivierb- | from what I was told yesterday, should have been transient issue but it has been transient overnight | 07:05 |
| olivierb- | thx | 07:05 |
| openstackgerrit | Merged openstack/openstack-ansible-os_trove master: Deprecate auth_uri option https://review.openstack.org/558374 | 07:10 |
| *** jbadiapa has joined #openstack-ansible | 07:16 | |
| *** epalper has joined #openstack-ansible | 07:17 | |
| *** mbuil has joined #openstack-ansible | 07:17 | |
| *** tosky has joined #openstack-ansible | 07:29 | |
| evrardjp | looks like there is no resolv.conf file? | 07:34 |
| *** gkadam has joined #openstack-ansible | 07:39 | |
| *** markvoelker has joined #openstack-ansible | 07:41 | |
| *** radeks has joined #openstack-ansible | 07:42 | |
| *** electrofelix has joined #openstack-ansible | 07:57 | |
| *** cjloader has joined #openstack-ansible | 07:58 | |
| *** DanyC has joined #openstack-ansible | 08:01 | |
| *** cjloader has quit IRC | 08:03 | |
| *** shardy has joined #openstack-ansible | 08:08 | |
| *** vnogin has joined #openstack-ansible | 08:13 | |
| *** markvoelker has quit IRC | 08:13 | |
| hwoarang | morning | 08:17 |
| hwoarang | who is up for some reviews? :) | 08:22 |
| *** radeks has quit IRC | 08:29 | |
| *** jwitko has quit IRC | 08:30 | |
| andymccr | hwoarang: what you got | 08:34 |
| hwoarang | maaany things | 08:35 |
| hwoarang | this one for example https://review.openstack.org/#/c/565347/ | 08:35 |
| hwoarang | and this https://review.openstack.org/#/c/565754/ | 08:36 |
| hwoarang | and a more interesting one https://review.openstack.org/#/c/562606/ | 08:37 |
| andymccr | i'll try get through those this morning | 08:38 |
| hwoarang | gracias | 08:39 |
| odyssey4me | mmercer yeah, gate-check-commit is brutal because it assumes you don't care about preserving anything on the host - if you do, better to walk yourself through the steps in the AIO guide | 08:41 |
| evrardjp | or the deploy guide even! | 08:44 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest stable/queens: Install os-testr https://review.openstack.org/565999 | 08:46 |
| evrardjp | hwoarang: can you clarify https://review.openstack.org/#/c/562606/16/vars/ubuntu-16.04.yml comment on setuptools? | 08:46 |
| olivierb- | evrardjp yep very strange this "disappearing" of resolv.conf. Furthermore as some other patchset tests have passed successfully | 08:47 |
| hwoarang | evrardjp: on ubuntu, setuptools and pkg-resources are two different packages | 08:48 |
| evrardjp | and you're doing that temporarily until everything is onto the host as proper distro packages | 08:49 |
| *** mwarad has quit IRC | 08:49 | |
| hwoarang | in other distros it's one. so when you use setuptools from ubuntu, and you use pip, then pip tries to install pkg-resources from pip and there is no such wheel in our index. it's what you see here http://logs.openstack.org/06/562606/12/check/openstack-ansible-functional-ubuntu-xenial/8f2c9d7/job-output.txt.gz#_2018-05-02_12_30_40_507990 | 08:49 |
| hwoarang | evrardjp: tempest is a bit special. not all distros have tempest packaged and the plugins make it harder to do it properly. i only want to minimize the stuff we install with pip | 08:50 |
| evrardjp | _tempest_requires_pip_packages that's for pip packages not for distro, so it should be the same? | 08:50 |
| hwoarang | pip packages on the host. these are only needed to prepare the host for tempest like creating networks etc. you dont need to use the pip packages for that | 08:51 |
| evrardjp | ok maybe my brain is deficient here. Didn't we say do everything in venv for tempest, and all the required resources to setup with ansible should be using ansible_python_interpreter to that venv? | 08:51 |
| *** vnogin has quit IRC | 08:51 | |
| evrardjp | is that patch too big maybe? | 08:52 |
| hwoarang | it's too invasive to do it at once. plus venv creation happens in the middle of the role. so somehow you need to switch from host ansible to venv ansible | 08:53 |
| odyssey4me | the reason we're installing this stuff is to do the resource creation tasks... we could also just build a venv and have those tasks use the venv.... or another approach would be to use the ansible-runtime venv and delegate the tasks to localhost | 08:53 |
| hwoarang | in reality, all i want ot happen in that patch is to use python-shade instead of pip shade for opensuse | 08:53 |
| evrardjp | ok. | 08:53 |
| evrardjp | call a cat a cat :p | 08:53 |
| hwoarang | well not only shade. anything that depends on 'cryptography' | 08:54 |
| evrardjp | oh yeah | 08:54 |
| evrardjp | I see | 08:54 |
| evrardjp | that's a good first step | 08:54 |
| hwoarang | because distro cryptography and pip cryptography can't co-exist. so anything that brings pip cryptography in has to either be in venv or switch to distro package | 08:55 |
| evrardjp | odyssey4me: that's what I meant by "all the required resources" :) | 08:55 |
| *** cjloader has joined #openstack-ansible | 08:55 | |
| evrardjp | but I think what you're proposing, including the delegation, would basically require setting up the ansible_python_interpreter | 08:55 |
| evrardjp | I am surprised they can't co-exist | 08:56 |
| evrardjp | but let's talk about that later, cryptography has been a pain all along | 08:56 |
| *** mbuil has quit IRC | 08:56 | |
| evrardjp | I like that. | 08:56 |
| hwoarang | it's only for suse. they are built using different openssl versions and some symbols dont like each other | 08:56 |
| odyssey4me | jamespage Is there some idea of when we can expect to see UCA for Rocky show up? It looks like we're needing python-shade >= 1.9.0 and UCA for Queens is only giving us 1.7.0 | 08:56 |
| hwoarang | we had a bug about that a few months ago | 08:56 |
| evrardjp | It's just that it seems a distraction towards simplification | 08:56 |
| evrardjp | odyssey4me: good call :) | 08:57 |
| evrardjp | I guess everything is in 18.04 :p | 08:57 |
| evrardjp | omg so many things to do :/ | 08:57 |
| hwoarang | great i was about to ask that... | 08:57 |
| evrardjp | we are stretched so thing. | 08:57 |
| hwoarang | about shade on ubuntu i mean... | 08:57 |
| odyssey4me | 18.04 is Queens too, so it's unlikely to make a difference | 08:57 |
| evrardjp | that's true. | 08:58 |
| jrosser | i did a patch to put shade on the deploy host if that helps any? might be another axis to decouple that from whats going on with the target nodes | 08:58 |
| evrardjp | distro packages everywhere and our life will be easier for those things :) | 08:59 |
| odyssey4me | hwoarang I'm curious - where does the shade requirement come from... and more especially, v1.9.0 - upper constraints only has 1.27.1: https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L256 | 08:59 |
| *** cjloader has quit IRC | 08:59 | |
| odyssey4me | also, 1.28.0 is the latest version... where does 1.8.0 come from? | 09:00 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts master: vars: ubuntu: Explicitly add virtualenv package https://review.openstack.org/565347 | 09:00 |
| odyssey4me | ah, 28>8.... wow, u-c has a really, really old version then | 09:00 |
| odyssey4me | agh, not u-c... UCA.... | 09:01 |
| * odyssey4me gets more coffee | 09:01 | |
| * odyssey4me wonders how the distributions decide which version they package... given that u-c for shade is 1.27.1 | 09:02 | |
| odyssey4me | (in queens) | 09:02 |
| *** radeks has joined #openstack-ansible | 09:02 | |
| *** markvoelker has joined #openstack-ansible | 09:10 | |
| hwoarang | odyssey4me: exactly | 09:11 |
| hwoarang | so on xenial you can't use the os_* modules :( | 09:11 |
| hwoarang | i wonder how they workaround that | 09:12 |
| andymccr | could we not install python-shade from the uca repo? | 09:16 |
| andymccr | and/or get the uca team to update the version in the repo? | 09:16 |
| andymccr | ok ignore me :P | 09:17 |
| * andymccr gets another coffee | 09:19 | |
| odyssey4me | hwoarang yeah, so my personal preference is actually to try to get rid of our own module usage and to try to use shade more... and also to use delegation to localhost and run_once for that resource stuff.... we shouldn't need to install so much on the host | 09:20 |
| odyssey4me | but I get you're trying to just switch to packages here, rather than change how it works | 09:21 |
| hwoarang | odyssey4me: what do you mean 'our own module' ? | 09:21 |
| odyssey4me | hwoarang I mean that we're using these modules: https://github.com/openstack/openstack-ansible-plugins/tree/master/library | 09:21 |
| odyssey4me | and those are bringing in the extra requirements (glance client, keystone client, etc) | 09:21 |
| *** vnogin has joined #openstack-ansible | 09:22 | |
| odyssey4me | if we can switch to using the ansible shade-based modules, then we potentially don't need glance client, etc | 09:22 |
| *** cedlerouge has joined #openstack-ansible | 09:22 | |
| odyssey4me | also, if we delegate the resource creation tasks to the deploy host then we won't need any of these packages installed | 09:25 |
| odyssey4me | some might need to be added to the ansible runtime though | 09:25 |
| hw_wutianwei_ | odyssey4me: I met a issue that some hosts lxc containers didn't install python. so It would fail at "TASK [lxc_container_create : Drop container network file (interfaces)]" | 09:29 |
| hwoarang | i didn't realize we had our own modules | 09:30 |
| odyssey4me | hw_wutianwei_ that is very odd, given that python is installed in the lxc host preparation I'm not sure how you would have gotten into that situation | 09:31 |
| nsingh | evrardjp: can you help me with the configuration file for container and host setup. | 09:33 |
| nsingh | How should i configure /opt/openstack-ansible/inventory/env.d/masakary.yml file so that masakarimonitor services will install on compute nodes. | 09:33 |
| nsingh | http://paste.openstack.org/show/720274/ | 09:33 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: common-tasks: Do not sync preconfigured /etc/pip.conf file https://review.openstack.org/565754 | 09:33 |
| *** chigang__ has quit IRC | 09:34 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_masakari master: Updated from OpenStack Ansible Tests https://review.openstack.org/566007 | 09:38 |
| evin | Why are there no gnocchi_git_* entries in /playbooks/defaults/repo_packages/openstack_services.yml pinning gnocchi to a particular commit? | 09:42 |
| Tahvok | cloudnull: I'm not sure how your review is fixing this problem: https://bugs.launchpad.net/openstack-ansible/+bug/1766636 | 09:42 |
| openstack | Launchpad bug 1766636 in openstack-ansible "No need to restart rabbitmq if there is no version upgrade" [High,In progress] - Assigned to Kevin Carter (kevin-carter) | 09:42 |
| Tahvok | Perhaps you put wrong bug number? | 09:42 |
| odyssey4me | evin we pin the versions of packages used so that when we release that tagged release delivers a consistent result | 09:42 |
| odyssey4me | those pins get updated for each release, generally around twice per month | 09:43 |
| odyssey4me | you can override that pin to something else if you want to - see https://gist.github.com/odyssey4me/fc69b1eb68f250e37815246e37fd13f2 for an example of how you do it | 09:43 |
| *** markvoelker has quit IRC | 09:44 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_masakari master: Remove tests-repo-clone.sh https://review.openstack.org/566012 | 09:47 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_masakari master: Remove tests-repo-clone.sh https://review.openstack.org/566012 | 09:48 |
| *** yolanda_ has joined #openstack-ansible | 09:49 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_masakari master: Only replace python when re-initializing the venv https://review.openstack.org/566015 | 09:50 |
| *** yolanda has quit IRC | 09:51 | |
| *** cjloader has joined #openstack-ansible | 09:51 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Update the output for "openstack floating" command https://review.openstack.org/565727 | 09:51 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts master: Remove jinja templating delimiters https://review.openstack.org/564856 | 09:54 |
| *** yolanda has joined #openstack-ansible | 09:56 | |
| *** yolanda_ has quit IRC | 09:59 | |
| evrardjp | nsingh: I can help you tomorrow, today is quite busy... I am stuck with bad hardware, taking 100% of my hands. | 10:00 |
| nsingh | evrardjp: Ohh ok no problem. thank you. Will ping you tomorrow. :) | 10:01 |
| evrardjp | thanks :) | 10:01 |
| evrardjp | so the role is working now? | 10:01 |
| evrardjp | and you're now integrating it? | 10:01 |
| evrardjp | I haven't seen patches though | 10:01 |
| odyssey4me | evrardjp are the base jobs up for review for the masakari role? | 10:02 |
| *** yolanda has quit IRC | 10:02 | |
| evrardjp | mmm good question | 10:02 |
| evrardjp | I guess it's waiting for me? | 10:02 |
| evrardjp | :p | 10:02 |
| odyssey4me | I specifically mean the jobs set out in project-config? | 10:02 |
| odyssey4me | I ask because I'm not seeing any jobs fire. | 10:02 |
| evrardjp | oh wait let me check taht | 10:02 |
| odyssey4me | I can do the zuul jobs for it, but I would have thought you'd have done the base jobs in project-config with the role creation? | 10:03 |
| nsingh | evrardjp: yes role is working now. I am working on few things. | 10:05 |
| nsingh | i will update the repo when all set from my side. | 10:07 |
| openstackgerrit | Albert Mikaelyan proposed openstack/openstack-ansible-rabbitmq_server master: Do not restart rabbitmq when no version is changed https://review.openstack.org/566017 | 10:08 |
| *** yolanda has joined #openstack-ansible | 10:15 | |
| nsingh | evrardjp: oh i can see the https://github.com/openstack/openstack-ansible-os_masakari now. Thank you :) i didn't got any mail regariding this. BDW thank you so much | 10:15 |
| odyssey4me | nsingh no, thank YOU :) | 10:22 |
| *** yolanda has quit IRC | 10:25 | |
| *** nicolasbock has joined #openstack-ansible | 10:31 | |
| *** yolanda has joined #openstack-ansible | 10:39 | |
| *** markvoelker has joined #openstack-ansible | 10:41 | |
| evrardjp | odyssey4me: https://review.openstack.org/#/c/562619/ | 10:41 |
| evrardjp | nsingh: it's because it's not done yet | 10:41 |
| evrardjp | I will talk with infra to make this happen today | 10:41 |
| *** cjloader has quit IRC | 10:51 | |
| *** cjloader has joined #openstack-ansible | 10:52 | |
| hw_wutianwei_ | https://www.irccloud.com/pastebin/ooVBJwPz/ | 10:53 |
| hw_wutianwei_ | odyssey4me: I am not sure | 10:54 |
| *** cjloader has quit IRC | 10:56 | |
| *** ppetit has quit IRC | 11:01 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Move radosgw keystone config tasks to their own playbook https://review.openstack.org/565701 | 11:03 |
| *** persia has joined #openstack-ansible | 11:06 | |
| *** vnogin has quit IRC | 11:08 | |
| *** mbuil has joined #openstack-ansible | 11:12 | |
| *** radeks has quit IRC | 11:14 | |
| *** markvoelker has quit IRC | 11:14 | |
| *** blinkiz has quit IRC | 11:19 | |
| *** geb has quit IRC | 11:22 | |
| olivierb- | BTW evrardjp you also have the same CI issue I am having (see https://review.openstack.org/565029) therefore I conclude that I'll wait for CI to be fixed ;-) | 11:22 |
| olivierb- | BTW evrardjp you also have the same CI issue I am having (see https://review.openstack.org/565029) therefore I conclude that I'll wait for CI to be fixed ;-) | 11:22 |
| *** vnogin has joined #openstack-ansible | 11:24 | |
| evrardjp | thanks for noticing olivierb- ! | 11:26 |
| *** geb has joined #openstack-ansible | 11:26 | |
| olivierb- | no pb, was worrying about my fixes to have broken something but could not relate them to the errors | 11:27 |
| *** srihas has quit IRC | 11:28 | |
| *** srihas has joined #openstack-ansible | 11:40 | |
| evrardjp | did you find the root cause? | 11:41 |
| evrardjp | I haven't got the chance to look at it yet | 11:41 |
| olivierb- | nope, I remember cloudnull telling me it was transient errors yesterday but this transient seems quite recurent now | 11:44 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_tempest stable/queens: Install os-testr https://review.openstack.org/565999 | 11:45 |
| *** blinkiz has joined #openstack-ansible | 11:45 | |
| olivierb- | I started creating a full test env on my dev system but currently have no time to go on with this unfortunately | 11:46 |
| blinkiz | On page https://docs.openstack.org/project-deploy-guide/openstack-ansible/queens/targethosts.html it says I should install ntp package when preparing my hosts. But if I recall correctly, chronyd is later installed. Is ntp package really needed? | 11:47 |
| evrardjp | blinkiz: you caught a documentation bug :) | 11:48 |
| evrardjp | I think this is fixed in master and may require backport | 11:48 |
| evrardjp | if not,let's make a patch to fix that :) | 11:49 |
| *** ansmith has quit IRC | 11:52 | |
| blinkiz | evrardjp, where should I create/submit this patch? Point me in the right direction and I can probably find the solution :) | 11:52 |
| hwoarang | evrardjp: olivierb- maybe this https://review.openstack.org/#/c/565933/ fixes it? | 11:58 |
| *** ppetit has joined #openstack-ansible | 11:59 | |
| olivierb- | hwoarang seems like a good candidate indeed | 12:00 |
| *** armaan has quit IRC | 12:00 | |
| olivierb- | thx for pointing it out | 12:01 |
| jrosser | blinkiz: you probably want to patch this file https://github.com/openstack/openstack-ansible/blob/master/deploy-guide/source/targethosts-prepare.rst | 12:01 |
| blinkiz | ok! | 12:01 |
| *** armaan has joined #openstack-ansible | 12:01 | |
| jrosser | blinkiz: are you all set up for pushing patches to review.openstack.org? | 12:02 |
| blinkiz | No :P | 12:02 |
| jrosser | ok :) start with this https://docs.openstack.org/infra/manual/developers.html | 12:03 |
| olivierb- | odyssey4me hwoarang seems like it also requires a backport from queens to pike https://review.openstack.org/565725 | 12:04 |
| hwoarang | requires? | 12:04 |
| hwoarang | what requires that? | 12:05 |
| *** jillr has quit IRC | 12:05 | |
| *** thedini3 has joined #openstack-ansible | 12:09 | |
| *** markvoelker has joined #openstack-ansible | 12:11 | |
| *** cjloader has joined #openstack-ansible | 12:12 | |
| olivierb- | the fact that the same CI error occurs on the stable/pike branch as I suppose that openstack-ansible and openstack-ansible-lxc_hosts branches should be in sync | 12:13 |
| olivierb- | just a wild guess | 12:13 |
| hwoarang | olivierb-: because the url you posted above is for a different thing | 12:15 |
| olivierb- | yes I have submitted 2 backports of a master fix for queens and pike and both are failing in CI for the same reason | 12:16 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts stable/queens: Fix lxc cache prep resolvers https://review.openstack.org/565933 | 12:16 |
| olivierb- | the url above and https://review.openstack.org/#/c/565762/ are the 2 backports | 12:17 |
| *** cjloader has quit IRC | 12:17 | |
| *** markvoelker has quit IRC | 12:20 | |
| *** markvoelker has joined #openstack-ansible | 12:20 | |
| olivierb- | therefore my question about the backport of the fix for resolv.conf into pike | 12:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts stable/pike: Fix lxc cache prep resolvers https://review.openstack.org/566046 | 12:23 |
| odyssey4me | hwoarang olivierb- ^ that's a backport to pike for the same issue, I guess there may also need to be one for Ocata, but I'm a little tied up at the moment | 12:24 |
| olivierb- | thx odyssey4me will try to submit for ocata during my forthcoming meeting if time permits | 12:25 |
| odyssey4me | great, thanks olivierb- | 12:25 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_keystone master: Add support for using distribution packages for OpenStack services https://review.openstack.org/560308 | 12:31 |
| openstackgerrit | Olivier Bourdon proposed openstack/openstack-ansible-lxc_hosts stable/ocata: Fix lxc cache prep resolvers https://review.openstack.org/566054 | 12:36 |
| olivierb- | odyssey4me hwoarang dones https://review.openstack.org/566054 | 12:37 |
| odyssey4me | awesome, thanks olivierb- :) | 12:38 |
| olivierb- | yavw | 12:39 |
| olivierb- | thx to you for the original patch | 12:39 |
| olivierb- | reviews and merge | 12:39 |
| odyssey4me | well, to cloudnull actually :) | 12:39 |
| *** ansmith has joined #openstack-ansible | 12:49 | |
| olivierb- | on another subject and besides the fact that AIO is a demo/poc/... environment, any good reason why bootstrap-aio.yml and more specifically tests/roles/bootstrap-host/tasks/prepare_data_disk.yml formats only in ext4 (no xfs nor lvm) and furthermore if /openstack and/or /var/lib/liblxc are/is already mounted it does not take it into account ? | 12:52 |
| olivierb- | Just wondering if it's worth submitting a patch for this | 12:53 |
| olivierb- | (or several patches more likely) | 12:53 |
| *** armaan has quit IRC | 12:54 | |
| olivierb- | odyssey4me yep, cloudnull will find out later when back from his well deserved night ;-) | 12:54 |
| *** nicolasbock has quit IRC | 12:57 | |
| *** yolanda has quit IRC | 12:57 | |
| mnaser | hmm | 12:58 |
| mnaser | it looks like ocata pip_install doesnt have the pinnings | 12:58 |
| mnaser | so it deploys pip 10.0.1 | 12:58 |
| mnaser | which fails the deployment/test of pip_install | 12:59 |
| odyssey4me | mnaser oh bother - any chance you can figure out a patch based on the stuff done to pike onwards? I would guess that pip_install and repo_build might need some work. | 12:59 |
| odyssey4me | Perhaps also the ansible bootstrap. | 12:59 |
| mnaser | odyssey4me: do you know where the patches are that pinned us to 9.x ? | 12:59 |
| mnaser | http://logs.openstack.org/54/566054/1/check/openstack-ansible-functional-centos-7/ac19a2e/job-output.txt.gz#_2018-05-03_12_45_44_508118 | 13:00 |
| mnaser | this is the task that seems to upgrade it | 13:00 |
| mnaser | or actually its already at 10 at that point | 13:00 |
| odyssey4me | is that a role test? | 13:00 |
| mnaser | yes | 13:00 |
| mnaser | for pip_install | 13:00 |
| mnaser | (for stable/ocata) | 13:01 |
| mnaser | https://review.openstack.org/#/c/566054/ | 13:01 |
| odyssey4me | role tests use the constraints set here https://github.com/openstack/openstack-ansible-tests/blob/stable/ocata/test-vars.yml#L453 - and the global pins are used there | 13:01 |
| mnaser | interesting, http://git.openstack.org/cgit/openstack/openstack-ansible/plain/global-requirement-pins.txt?h=stable/ocata shows it pinned at 9.0.1 | 13:02 |
| mnaser | the thing is | 13:03 |
| mnaser | "Found existing installation: pip 10.0.1" | 13:03 |
| odyssey4me | hmm, and https://review.openstack.org/#/q/I32603fd34b60183607c6bd9653c36432cbe6b07a was merged too | 13:03 |
| mnaser | so when it reaches the install pip task | 13:03 |
| mnaser | it's already at pip 10 | 13:03 |
| odyssey4me | yes, it will start at the later version because it's already on the host | 13:03 |
| odyssey4me | we downgrade it | 13:03 |
| mnaser | oh i see | 13:03 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Do not clone role being tested as a dep outside openstack-ci https://review.openstack.org/565256 | 13:04 |
| odyssey4me | looks like https://review.openstack.org/#/q/If1b68fb21e0eb8f2f8c33a6bec952c2972e3e5e3 didn't go to ocata though | 13:04 |
| mnaser | odyssey4me: the parent commit of this is not the one that includes that fix | 13:04 |
| mnaser | do we maybe we need to rebase? | 13:04 |
| mgariepy | good morning everyone | 13:05 |
| mnaser | o/ mgariepy | 13:05 |
| odyssey4me | mnaser eh? I'm not sure I understand what you mean | 13:06 |
| mnaser | https://review.openstack.org/#/c/566054/ -- if you click on (gitweb) next to parent, you notice that the parent is the zuul remove project patch | 13:06 |
| mnaser | which is a commit from 4 months ago | 13:06 |
| mnaser | which feels like the patch was pushed without a local rebase | 13:06 |
| mnaser | so all the stuff we did about pinning isn't being applied, because the patch is being tested on top of that commit only rather than tip of stable/ocata | 13:07 |
| mnaser | let me try rebasing the patch to tip of stable/ocata and see what happens | 13:07 |
| odyssey4me | mnaser but that's in a different role - that role test will use the tip of stable/ocata for pip_install | 13:07 |
| mnaser | odyssey4me: this is a patch for pip_install being tested and i assume what is being tested is the checked out version by zuul, correct? | 13:08 |
| odyssey4me | mnaser but https://review.openstack.org/#/c/566054/ is a lxc_hosts role change, it's not a pip_install patch | 13:08 |
| *** vakuznet has joined #openstack-ansible | 13:08 | |
| mnaser | ok | 13:09 |
| mnaser | i need to get to my coffee | 13:09 |
| odyssey4me | it will consume pip_install from the tip of stable/ocata which seems to have the required fix | 13:09 |
| mnaser | sorry for that | 13:09 |
| mnaser | i thought both were pip_install fixes | 13:09 |
| odyssey4me | ah, https://review.openstack.org/#/q/If1b68fb21e0eb8f2f8c33a6bec952c2972e3e5e3 isn't needed in stable/ocata because it's an identical tree - so I guess the ocata branch already has that patched merged into another review | 13:10 |
| mnaser | https://review.openstack.org/#/c/561577/ - patch from 2 weeks ago to the same branch with the same issue so this might have been lingering around for a while | 13:10 |
| odyssey4me | yeah, this may be a new issue - not sure | 13:10 |
| *** thedini3 has quit IRC | 13:10 | |
| *** yolanda has joined #openstack-ansible | 13:11 | |
| *** cjloader has joined #openstack-ansible | 13:13 | |
| *** nicolasbock has joined #openstack-ansible | 13:14 | |
| *** cjloader has quit IRC | 13:17 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Move radosgw keystone config tasks to their own playbook https://review.openstack.org/565701 | 13:20 |
| *** vakuznet has quit IRC | 13:21 | |
| *** ansmith has quit IRC | 13:26 | |
| *** vakuznet has joined #openstack-ansible | 13:27 | |
| *** eumel8 has quit IRC | 13:28 | |
| *** jwitko has joined #openstack-ansible | 13:34 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Tidy registered variable names in rgw install https://review.openstack.org/565452 | 13:37 |
| olivierb- | odyssey4me mnaser I am a bit confused by what you wrote above furthermore as the backport of resolv.conf fix to ocata failed to pass CI | 13:37 |
| vakuznet | is this known issue: "repo-container-2aa1521e nginx[65763]: nginx: configuration file /etc/nginx/nginx.conf test failed" ? | 13:39 |
| *** cmart has joined #openstack-ansible | 13:46 | |
| *** jwitko has quit IRC | 13:48 | |
| *** jwitko has joined #openstack-ansible | 13:48 | |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts stable/pike: Fix lxc cache prep resolvers https://review.openstack.org/566046 | 13:50 |
| *** cjloader has joined #openstack-ansible | 13:58 | |
| *** cjloader has quit IRC | 13:59 | |
| *** cjloader has joined #openstack-ansible | 13:59 | |
| mnaser | vakuznet: i think i noticed that too when running repo-build | 14:02 |
| mnaser | a re-run somehow cleaned it up.. | 14:02 |
| *** kstev has joined #openstack-ansible | 14:03 | |
| *** evin has quit IRC | 14:03 | |
| openstackgerrit | Nicolas Bock proposed openstack/openstack-ansible-os_nova master: Define lxd.pool in nova.conf based on lxd_storage_pool https://review.openstack.org/565891 | 14:04 |
| *** throwsb1 has joined #openstack-ansible | 14:04 | |
| vakuznet | mnaser, it was self inflicted issue. resolved. | 14:06 |
| *** esberglu has joined #openstack-ansible | 14:06 | |
| idlemind | grr i broke my cloud and can't seem to get os-nova to complete ... keeps blowing up on conductor containers, i'll delete the container and re-run then it fails on something. i had to pin the galera_minor_distribution to 32 now the newest container just failed the pip_install (install distro packages) step | 14:08 |
| idlemind | (stable/pike) | 14:08 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/queens: Update all SHAs for 17.0.4 https://review.openstack.org/565029 | 14:09 |
| idlemind | horizon kinda works but i can't browse to most of the pages. all the problems started when i moved from a single hosted vip to a multi-node cluster w/haproxy and keepalived, horizon is still trying to hit the old vip ip probably because something in os_nova or another playbook down the line in setup-everything has to complete to update the URLs it tries to hit so i'm basically dead in the water | 14:09 |
| idlemind | but that seems ancillary to not being able to rebuild the necessary containers to reliably get past os-nova | 14:10 |
| idlemind | is there a "base" container that i'd need to update or anything? | 14:15 |
| idlemind | that get's cloned or something? it seems a new container is getting the wrong proxy IP after creation in YUM | 14:16 |
| idlemind | it's the ip of the old internal_lb_vip_addr | 14:17 |
| *** yolanda has quit IRC | 14:19 | |
| *** evin has joined #openstack-ansible | 14:21 | |
| *** esberglu has quit IRC | 14:23 | |
| *** kstev1 has joined #openstack-ansible | 14:25 | |
| *** kstev has quit IRC | 14:27 | |
| idlemind | i don't have that configured anywhere but it keeps popping up to bite at my heels | 14:28 |
| *** dxiri has joined #openstack-ansible | 14:34 | |
| *** esberglu has joined #openstack-ansible | 14:34 | |
| *** vnogin has quit IRC | 14:34 | |
| *** pabelanger has quit IRC | 14:35 | |
| *** pabelanger has joined #openstack-ansible | 14:35 | |
| *** vnogin has joined #openstack-ansible | 14:35 | |
| *** esberglu_ has joined #openstack-ansible | 14:36 | |
| *** epalper has quit IRC | 14:38 | |
| *** vnogin has quit IRC | 14:40 | |
| *** esberglu has quit IRC | 14:40 | |
| *** esberglu_ is now known as esberglu | 14:40 | |
| *** _d34dh0r53_ is now known as d34dh0r53 | 14:46 | |
| idlemind | k, repo_cache is setup correctly on 99% of my stuff, if i delete a container and recreate it, it comes back to life w/the old internal_lb_vip_addr not the new "right" one ... running setup-everything.yml w/a --limit of the new container doesn't do anything just fails at pip_install install distro packages because it can't fetch packages (yum update at the command line fails too inside the container) | 14:47 |
| *** spine55 has joined #openstack-ansible | 14:48 | |
| *** yolanda has joined #openstack-ansible | 14:53 | |
| *** klamath has joined #openstack-ansible | 14:57 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_glance master: defaults: Allow uwsgi to autoload required plugins. https://review.openstack.org/566092 | 14:58 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_glance master: Add support for using distribution packages for OpenStack services https://review.openstack.org/566093 | 14:58 |
| *** vnogin has joined #openstack-ansible | 14:59 | |
| *** vnogin has quit IRC | 15:00 | |
| *** ppetit has quit IRC | 15:03 | |
| *** DanyC has quit IRC | 15:03 | |
| cloudnull | Tahvok: opps, yup added the wrong bug number there. | 15:05 |
| evrardjp | idlemind: hey, do you have an override? | 15:05 |
| cloudnull | good catch :) | 15:05 |
| evrardjp | I mean a wrong override? | 15:05 |
| idlemind | evrardjp in user_variables you mean for repo_cache_proxy_url (or something close to that) | 15:05 |
| idlemind | evrardjp no not before i started, i've since added one but i can't seem to find which play applies that to an existing container | 15:05 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-openstack_hosts master: Add IPv6 kernel module https://review.openstack.org/565940 | 15:05 |
| idlemind | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ IPv6!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! cloudnull i knew i loved you | 15:06 |
| evrardjp | we do have a base image that we use for creating containers | 15:06 |
| cloudnull | idlemind: :) | 15:06 |
| idlemind | evrardjp is that the /var/lib/machines/<$$> | 15:06 |
| cloudnull | I pay that one day we'll have an IPv6 only cloud ! | 15:07 |
| cloudnull | :D | 15:07 |
| cloudnull | **I pray that... | 15:07 |
| *** ppetit has joined #openstack-ansible | 15:07 | |
| idlemind | mine (when it's not broken) gives public ipv6 networks to ppl and routes via bgp | 15:08 |
| evrardjp | you pay? :D | 15:08 |
| evrardjp | I think if that was really requested we could do it | 15:08 |
| idlemind | evrardjp it'd be something i'd be keen to work on ... i'm an ipv6 crazy | 15:08 |
| evrardjp | another good way to get rid of the mac/ip generation in the inventory :D | 15:09 |
| idlemind | so lxc_container_create works and completes w/o issue on the trouble container ... | 15:09 |
| evrardjp | idlemind: so what's the issue? | 15:09 |
| idlemind | the container's yum proxy is set to an old internal_lb_vip_addr | 15:09 |
| idlemind | even after a delete / recreate | 15:09 |
| evrardjp | you said basically changed the endpoint url (changing internal_lb_vip_address) | 15:09 |
| idlemind | and i can't see how or why that happens | 15:10 |
| evrardjp | yes so | 15:10 |
| idlemind | i can't run common-tasks/package-cache-proxy.yml it looks like it needs to be imported by another play? | 15:10 |
| evrardjp | if you changed the internal_lb_vip_address, you'd have to rebuild the repo server | 15:10 |
| idlemind | ahhhhhhhhhhhhhhhhhhhhhhh | 15:10 |
| evrardjp | and then run all the playbooks | 15:10 |
| idlemind | so that's probably what i missed | 15:10 |
| evrardjp | don't forget haproxy | 15:10 |
| idlemind | is that a delete / rebuild of the repo server? | 15:10 |
| idlemind | ya haproxy completed | 15:11 |
| idlemind | i didn't think i'd have to redo the repo server | 15:11 |
| evrardjp | wait a sec | 15:11 |
| evrardjp | maybe I am tired there | 15:11 |
| evrardjp | the IP of the container shouldn't change in itself | 15:11 |
| evrardjp | I'd re-run the repo-server play | 15:11 |
| idlemind | ya just internal_lb_vip_addr but maybe that depends on something | 15:11 |
| *** weezS has joined #openstack-ansible | 15:11 | |
| evrardjp | not deleting the container | 15:12 |
| evrardjp | well, I'd rerun all the plays | 15:12 |
| evrardjp | check if your inventory is alright before starting | 15:12 |
| evrardjp | (open your /etc/openstack_deploy/inventory.json , search for your old internal_lb_vip_address) | 15:12 |
| idlemind | ahh k | 15:12 |
| evrardjp | just in case | 15:13 |
| evrardjp | I am cautious :) | 15:13 |
| evrardjp | don't edit it | 15:13 |
| evrardjp | but if you see something there, then there is an issue :p | 15:13 |
| cloudnull | evrardjp: mind giving https://review.openstack.org/#/c/565950/ a review when you can ? | 15:13 |
| idlemind | k i don't have inventory.json in that path on my deployment host | 15:14 |
| nicolasbock | How does this work with running a gerrit change set through CI? | 15:15 |
| evrardjp | cloudnull: when we tar we put a whole inventory into the file? | 15:15 |
| nicolasbock | Do I need to manually trigger a CI run? | 15:15 |
| cloudnull | evrardjp: yup | 15:15 |
| cloudnull | the entire inventory file is added to the file | 15:15 |
| evrardjp | yeah I see | 15:15 |
| evrardjp | mmm | 15:15 |
| evrardjp | let me review real quick | 15:16 |
| cloudnull | so instead of saving all inventories forever, we can save the last 15 | 15:16 |
| odyssey4me | nicolasbock nope - what's the issue? you can look at http://zuul.openstack.org/status.html to see the status of any patches busy being tested | 15:16 |
| openstackgerrit | German Eichberger proposed openstack/openstack-ansible-os_octavia stable/queens: Adds certificate generation https://review.openstack.org/565845 | 15:16 |
| *** RandomTech has joined #openstack-ansible | 15:17 | |
| nicolasbock | Ah odyssey4me thanks! | 15:17 |
| nicolasbock | I see the change set in there | 15:17 |
| nicolasbock | I was only looking at gerrit | 15:17 |
| RandomTech | Hello, when i ran the playbooks it created CEPH partitioned drives but didnt add them to the CEPH cluster. Any ideas on why this could happen or how to fix it | 15:17 |
| odyssey4me | cloudnull evrardjp perhaps we should rather created a timestamped tarfile and leave the pruning to the operator? | 15:18 |
| cloudnull | odyssey4me: all of the inventories within the tar file are timestamped. | 15:19 |
| cloudnull | so maybe we should just outline how to prune the inventory ? | 15:19 |
| idlemind | k, just for good measure i dropped and am now recreating the container using lxc_container_delete / create from the doc's i'll triple check the lb addr -> yum proxy url is still incorrectly set and that's not happening further down the line | 15:21 |
| odyssey4me | cloudnull tbh I didn't know the tarbal contained history... we should probably tell people about this somewhere ;) | 15:22 |
| odyssey4me | *tarball | 15:22 |
| idlemind | where is this amazing tarball we're speaking of? | 15:22 |
| cloudnull | # /etc/openstack_deploy/backup_openstack_inventory.tar | 15:22 |
| cloudnull | its a running backup of inventory | 15:23 |
| idlemind | thx | 15:23 |
| cloudnull | odyssey4me: I thought it was in the docs somewhere, maybe not... | 15:23 |
| cloudnull | confirmed, its not... | 15:24 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Automatically prune the inventory backup https://review.openstack.org/565950 | 15:24 |
| cloudnull | we should add a blerb about this is deployers guide. | 15:25 |
| evrardjp | cloudnull: it's crazy the tarfile module doesn't have a delete thing | 15:26 |
| cloudnull | yea, i was frustrated by that... | 15:26 |
| *** vnogin has joined #openstack-ansible | 15:26 | |
| evrardjp | you basically have to unarchive and from tarinfo rebuild your archive... how insane is that? :p | 15:26 |
| odyssey4me | cloudnull perhaps better in the operator's guide? | 15:26 |
| evrardjp | cloudnull: however I am afraid of what this does, as you have a race condition in code | 15:27 |
| cloudnull | ? | 15:28 |
| cloudnull | I'm not seeing the race condition. | 15:31 |
| cloudnull | its a basic for loop, and the backup is called serially. | 15:31 |
| evrardjp | yes but you close later, and add to something already open | 15:32 |
| cloudnull | you mean someone else could be writing to the inventory file at the same time we're pruning it ? | 15:33 |
| cloudnull | **inventory backup file | 15:33 |
| cloudnull | **someone else with a different shell | 15:34 |
| cloudnull | the context manager only keeps the archive open for as long as it takes to prune and save the backup, so that shouldn't be possible within the inventory application | 15:35 |
| cloudnull | though it could be possible if different users invoke the inventory at the exact same time. | 15:36 |
| odyssey4me | cloudnull perhaps there should be a lock or something to protect against that - I vaguely remember that there was a bug about multiple executions breaking the archive | 15:36 |
| odyssey4me | not sure how to properly handle it though - what do we do? if there's a lock, skip the archive, or wait? | 15:37 |
| odyssey4me | this is kinda why I think that perhaps we should just do a timestamped archive and never modify the archive again | 15:37 |
| odyssey4me | that way we leave the clean up to the deployer, and don't have to deal with any races or locks or anything | 15:38 |
| cloudnull | odyssey4me: where should we put those archives ? | 15:42 |
| olivierb- | sorry to bother with same matter again but this morning with evrardjp I tried to understand what had changed since last week for deploying AIO and digged a bit more during the day without any more success. | 15:42 |
| olivierb- | I have been using https://gist.github.com/obourdon/a75cbd2a2bbbd98de30005557ae7d886 for several weeks whithout any problem | 15:42 |
| olivierb- | but since I retried this week I get https://gist.github.com/obourdon/3242923baddc94313b881ebe457f447a | 15:44 |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server master: Do not restart rabbitmq when no version is changed https://review.openstack.org/566017 | 15:44 |
| olivierb- | which seems like stuff coming from openstack-ansible-hardening | 15:44 |
| olivierb- | right now I do not have a single clue on why this is now active whereas it was not last week and looking at the various commits in various repo did not help but I must have missed something | 15:45 |
| *** spine55 has quit IRC | 15:45 | |
| idlemind | ok, so dropped my trouble container and recreated it with lxc-container-create and it doesn't have any proxy url's for yum configured at all | 15:46 |
| olivierb- | thanks for any idea/pointer/help | 15:46 |
| idlemind | that must come from a later playbook? | 15:46 |
| *** mma has quit IRC | 15:46 | |
| olivierb- | and no I checked my user_*.yml stuff and everything else did not change either | 15:46 |
| *** RandomTech has quit IRC | 15:47 | |
| olivierb- | note that I am also using stable/pike release which did not changed much since last week either therefore my very puzzled state | 15:47 |
| idlemind | olivierb is there an actual failure above that? | 15:47 |
| *** gyee has joined #openstack-ansible | 15:47 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Add information about restoring inventory from backup https://review.openstack.org/566099 | 15:48 |
| olivierb- | idlemind stupid me, I thought these where the actual errors but it's related to the resolv.conf issue fixed by cloudnull earlier today, I should start reading ogs more carefully or stop using -vvv in the 1st place many thanks | 15:50 |
| *** pcaruana has quit IRC | 15:50 | |
| *** hamzy has quit IRC | 15:56 | |
| evrardjp | cloudnull: I meant the context manager is supposed to be doing things for the context, so opening and closing files, but here you have subshell to manipulate the tarfile at the same time | 16:03 |
| evrardjp | which is kinda weird | 16:03 |
| *** vnogin has quit IRC | 16:04 | |
| cloudnull | right but its deleting not writing, so there should be no race there. | 16:04 |
| evrardjp | but you are writing in the same context :) | 16:04 |
| evrardjp | that's the thing | 16:04 |
| cloudnull | after the delete has finished | 16:05 |
| evrardjp | you want me to just update it real quick? | 16:05 |
| cloudnull | sure. | 16:05 |
| *** yolanda has quit IRC | 16:06 | |
| odyssey4me | cloudnull well, yeah - where indeed | 16:10 |
| odyssey4me | it strikes me that perhaps we're trying too hard to do too many things, and perhaps we should let operators do their own inventory backups | 16:11 |
| rschulman | Given OSA's use of tagged VLANs, do people generally use general mode for port switches? | 16:12 |
| rschulman | Er... s/port switches/switch ports/ | 16:13 |
| jrosser | odyssey4me: + on that, i have /etc/openstack-deploy symlinked out to its own git repo, and then manage "disaster" of that myself | 16:13 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Add information about restoring inventory from backup https://review.openstack.org/566099 | 16:17 |
| evrardjp | odyssey4me: they will, when using static inventory :p | 16:19 |
| evrardjp | no choice! | 16:19 |
| cloudnull | rschulman: its totally up to you. top of rack is a choose your own adventure :) | 16:19 |
| *** spine55 has joined #openstack-ansible | 16:19 | |
| odyssey4me | jrosser yep, I wouldn't do it any other way | 16:19 |
| rschulman | cloudnull: Yeah, of course, but I'm seeking wizened advice. :) | 16:20 |
| cloudnull | hahaha | 16:20 |
| evrardjp | olivierb-: I am sure it's the image that changed :) I am pretty sure we haven't changed anything in ansible-hardening. | 16:20 |
| jrosser | odyssey4me: would be nice if there were an env var to point at the config dir rather than needing a symlink | 16:21 |
| cloudnull | rschulman: one sec . | 16:21 |
| evrardjp | rschulman: osa is flexible, you just have to configure it for what you want :) | 16:21 |
| evrardjp | vlans work :p | 16:21 |
| idlemind | ugh stranger still. in stable/pike i run os-nova-install on the newly created conductor container ... it sets the general proxy in yum.conf to the right ip (172.29.236.8) which is the new internal_lb_vip_address ... in /etc/yum.repos.d/CentOS-Base.repo it has the old IP (172.29.236.11) ... no clue were it's getting that | 16:21 |
| olivierb- | evrardjp neverming I must read logs more carefully, these are debug statements and as written in the very end I thought the error came from there but it is the resolv.conf issue which stroke again | 16:22 |
| olivierb- | and I also must stop using -vvv as my default ;-) | 16:22 |
| olivierb- | to many info kills the info | 16:22 |
| odyssey4me | idlemind the centos repo was copied directly from the host | 16:22 |
| idlemind | ahh look at you | 16:23 |
| idlemind | it's incorrectly set there | 16:23 |
| idlemind | manually editing that is probably not suggested ... what task would normally update that in stable/pike? openstack-hosts? | 16:23 |
| olivierb- | anyways it does not impact the fact that some checks (like nameserver names number in /etc/resolv.conf) could be a litlle bit more "strict" in the sense that you can not fool them using a comment dummy entry #nameserver XXXXX for instance ;-) | 16:23 |
| idlemind | or lxc-hosts-setup? | 16:24 |
| rschulman | evrardjp: Yeah, I know. Almost TOO flexible for a first timer. :) | 16:25 |
| odyssey4me | idlemind it's only copied in when the base cache is created - I don't think there's anything touching it from then on - so you can fix it using ansible with something like: ansible -m lineinfile -a "...insert args here..." all | 16:25 |
| *** mma has joined #openstack-ansible | 16:25 | |
| odyssey4me | or write a little playbook or something | 16:25 |
| idlemind | odyssey4me so i "could" drop the host completely, reimage it w/centos and start building it back up from there ... or just edit the repo files manually (with ansible or vi) | 16:27 |
| *** ppetit has quit IRC | 16:30 | |
| odyssey4me | idlemind yeah, or you could wipe the cache to force it being rebuilt - but then you'd need to delete the containers and rebuild them | 16:31 |
| idlemind | wipe the cache? | 16:33 |
| idlemind | you mean drop the repo server container and rebuil it? | 16:34 |
| idlemind | and then drop and rebuild all of my containers? | 16:34 |
| odyssey4me | idlemind no, the lxc base cache - where/what it is depends on the lxc back-end you're using | 16:34 |
| idlemind | ahh | 16:34 |
| idlemind | cat /var/lib/machines/centos-7-amd64/etc/yum.repos.d/CentOS-Base.repo ... for me confirms the bad proxy URL | 16:36 |
| idlemind | [root@hc2 ~]# cat /etc/yum.repos.d/CentOS-Base.repo ... also has the bad proxy url ... but as we said the only way to fix that is manually (automated via custom ansible) or format the host and start fresh right? | 16:37 |
| idlemind | no play runs and updates repo files on a base host after the initial pass? | 16:38 |
| *** DanyC has joined #openstack-ansible | 16:38 | |
| odyssey4me | yep, OSA can't fix what you broke ;) | 16:38 |
| odyssey4me | the centos base repo on the host doesn't get touched by OSA as far as I know | 16:39 |
| * SamYaple says nothing | 16:39 | |
| odyssey4me | I haven't really worked with CentOS much. | 16:39 |
| SamYaple | im gonna use that line odyssey4me | 16:39 |
| *** DanyC_ has joined #openstack-ansible | 16:39 | |
| SamYaple | i lied i said something | 16:39 |
| odyssey4me | lol | 16:39 |
| Tahvok | odyssey4me, cloudnull: that was fast. Thanks for reviewing this | 16:40 |
| Tahvok | Is it possible to submit reviews for ocata/pike/queens branches for this bug? https://bugs.launchpad.net/openstack-ansible/+bug/1766636 | 16:41 |
| openstack | Launchpad bug 1766636 in openstack-ansible "No need to restart rabbitmq if there is no version upgrade" [High,Fix released] - Assigned to Albert Mikaelyan (tahvok) | 16:41 |
| evrardjp | SamYaple: :) | 16:41 |
| evrardjp | Tahvok: fine for me to bp | 16:41 |
| idlemind | odyssey4me it must at some point ... i didn't go into it and set it to point to the proxy :( :( if anything it seems to set it and never revisit it ... so if you ever change your internal_lb_vip addr this happens | 16:42 |
| *** DanyC has quit IRC | 16:42 | |
| Tahvok | Please remind me how do I submit reviews for other branches? | 16:43 |
| idlemind | it seems to update yum.conf just not the yum.repos.d stuff | 16:43 |
| idlemind | i'm trying to see where that would be added now but i'm probably not as fast as you vets | 16:44 |
| odyssey4me | Tahvok the same way as any other patches - checkout the branch, prepare the patch, then git review... if it's a backport, then use 'git review -X <review #>' to cherry-pick it | 16:45 |
| rschulman | cloudnull: Did you ever follow up from that "one sec"? I don't see it. | 16:48 |
| Tahvok | So I first do my changes, then use git review -X I523be647b5e82e6f088428bf2db24dc4cd2cfb53 ? | 16:48 |
| odyssey4me | Tahvok nope, git review -X <review number, not change-id> will cherry-pick the change... all changes should go into master first, so they should usually be cherry-picked down | 16:50 |
| odyssey4me | if it needs changes, then do them after the cherry-pick | 16:50 |
| *** Taseer has quit IRC | 16:51 | |
| odyssey4me | for that change id, the review number is 566017 | 16:51 |
| *** Taseer has joined #openstack-ansible | 16:51 | |
| odyssey4me | you can also just use the gerrit interface and click the cherry-pick button, then select the branch | 16:51 |
| odyssey4me | that assumes a clean pick is possible though | 16:52 |
| odyssey4me | which is very likely in this case | 16:52 |
| Tahvok | Learning new stuff | 16:52 |
| Tahvok | Will try both actually :) | 16:52 |
| *** yolanda has joined #openstack-ansible | 16:53 | |
| openstackgerrit | Albert Mikaelyan proposed openstack/openstack-ansible-rabbitmq_server stable/queens: Do not restart rabbitmq when no version is changed https://review.openstack.org/566116 | 16:55 |
| Tahvok | odyssey4me: should I just checkout the remotes/origin/stable/ocata branch, or actually create my own out of that branch? | 16:59 |
| *** yolanda_ has joined #openstack-ansible | 17:01 | |
| idlemind | is it possible that the yum.repos.d gets edited here? https://github.com/openstack/openstack-ansible-openstack_hosts/blob/stable/pike/tasks/openstack_host_install_yum.yml ... i don't see anything about setting the proxy specifically but maybe it's a knock on effect? | 17:02 |
| *** mbuil has quit IRC | 17:02 | |
| *** gkadam has quit IRC | 17:02 | |
| *** shardy has quit IRC | 17:03 | |
| odyssey4me | Tahvok I typically do something like: git checkout origin/stable/ocata | 17:03 |
| xgerman_ | wonder if we have seen that already: http://logs.openstack.org/45/565845/2/check/openstack-ansible-functional-ubuntu-xenial/1fb54e5/job-output.txt.gz#_2018-05-03_16_09_15_955190 | 17:03 |
| *** hamzy has joined #openstack-ansible | 17:03 | |
| odyssey4me | well, typically I do this: git fetch --all; git checkout origin/stable/ocata | 17:03 |
| *** yolanda has quit IRC | 17:04 | |
| SamYaple | `git pull` on the stable/branch and fork. easy | 17:05 |
| *** yolanda has joined #openstack-ansible | 17:05 | |
| Tahvok | So after git review -X - I still need to upload my review with git review? | 17:05 |
| odyssey4me | idlemind it looks like it's designed to only add repositories that are missing, but I might be wrong - you'd have to check the playbook execution output or look at it more carefully... I'm a little tied up right now, so can't really think it through properly | 17:05 |
| odyssey4me | Tahvok yep, git review -X just cherry-picks to your local repo - git review uploads it | 17:06 |
| idlemind | ya as a test i dropped the proxy line from the centos-base.repo file and i'm rerunning setup-everything limited to that host to see it shows back up ... maybe it's a knock on effect of one of the yum commands | 17:06 |
| Tahvok | Ok, so that's what I was missing, and it was not a wrong branch :) | 17:06 |
| idlemind | i also might just drop this hypervisor and start fresh to see if it was a fluke from an older code set that brought those proxy lines into the yum.repos.d/* stuff | 17:06 |
| openstackgerrit | Albert Mikaelyan proposed openstack/openstack-ansible-rabbitmq_server stable/pike: Do not restart rabbitmq when no version is changed https://review.openstack.org/566119 | 17:07 |
| Tahvok | Nice, worked :) | 17:07 |
| idlemind | ok that was fast, my first test was openstack-ansible openstack-hosts-setup.yml --limit hc2 | 17:07 |
| idlemind | the repo's i deleted the proxy from | 17:07 |
| idlemind | came back with the correct line | 17:07 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Provide an example for switch port configurations https://review.openstack.org/566120 | 17:07 |
| idlemind | (updated to the new proxy ip) | 17:07 |
| idlemind | the ones i left ... stayed as the old (wrong) ip | 17:07 |
| *** yolanda_ has quit IRC | 17:08 | |
| idlemind | so something we're doing in that play is causing the proxy ip to be set in the repo only if it doesn't exist | 17:08 |
| cloudnull | rschulman: rschulman ^ | 17:08 |
| openstackgerrit | Albert Mikaelyan proposed openstack/openstack-ansible-rabbitmq_server stable/ocata: Do not restart rabbitmq when no version is changed https://review.openstack.org/566121 | 17:08 |
| idlemind | and not updating it if it does exist | 17:08 |
| Tahvok | odyssey4me: thanks a lot! all reviews uploaded | 17:08 |
| idlemind | cloudnull how can i view that i'd be curious to review on it | 17:09 |
| idlemind | (do i have to cherry-pick the patch or cna i see it right in gerrit) | 17:09 |
| odyssey4me | Tahvok great, thank you :) | 17:10 |
| *** DanyC_ has quit IRC | 17:11 | |
| odyssey4me | idlemind what branch are you using? | 17:11 |
| odyssey4me | xgerman_ the failure is further up: http://logs.openstack.org/45/565845/2/check/openstack-ansible-functional-ubuntu-xenial/1fb54e5/job-output.txt.gz#_2018-05-03_16_09_15_483822 | 17:12 |
| xgerman_ | queens | 17:12 |
| odyssey4me | xgerman_ also, you might find looking at the ara-report easier to diagnose issues with: http://logs.openstack.org/45/565845/2/check/openstack-ansible-functional-ubuntu-xenial/1fb54e5/logs/ara-report/ | 17:13 |
| xgerman_ | thx | 17:15 |
| rschulman | cloudnull: I will study with great interest. Thank you! | 17:18 |
| *** mma has quit IRC | 17:24 | |
| *** mma has joined #openstack-ansible | 17:24 | |
| *** mma has quit IRC | 17:29 | |
| *** DanyC has joined #openstack-ansible | 17:30 | |
| *** mma has joined #openstack-ansible | 17:33 | |
| mnaser | is there a way for repo build to build one specific venv? | 17:39 |
| mnaser | ex: changed the cinder hash commit and dont want to wait ages | 17:39 |
| mnaser | :P | 17:39 |
| *** mma has quit IRC | 17:41 | |
| *** mma has joined #openstack-ansible | 17:42 | |
| *** spsurya has quit IRC | 17:42 | |
| *** mma has quit IRC | 17:46 | |
| idlemind | odyssey4me pike | 17:47 |
| *** udesale has quit IRC | 17:50 | |
| idlemind | so cloudnull your switches ... is that to cisco switches that support cross chassis port channels (or a switch stack)? | 17:55 |
| *** poopcat has joined #openstack-ansible | 17:59 | |
| idlemind | cloudnull https://imgur.com/a/ovC9oSa here is basically what i did. i made my hosts redundant with spanning-tree (#ghettobutworksreliably) this kept me away from a vpc or stackable reliant position without having to go routed completely. i am thinking of routed racks which was presented on at one of the conferences by the scaling group to create multiple spanning-tree domains so a spanning-tree event won't take an | 18:04 |
| idlemind | entire dc down. a start of that was bgp based automatic provider networks for me | 18:04 |
| jrosser | hmm i would be carefu | 18:08 |
| jrosser | l with using stp like that for pseudo HA | 18:09 |
| idlemind | jrosser why? you no trust stp? | 18:09 |
| jrosser | i have no issues with nxos/vpc in my control plane | 18:09 |
| jrosser | hand run l3 routed for my racks of compute | 18:09 |
| *** vakuznet has quit IRC | 18:10 | |
| jrosser | if you want l3 with vlan tenant networks look at neutron segmented networks | 18:10 |
| idlemind | jrosser vpc related issues and single point of failure during upgrades in stackable models is why i'd avoid those technologies ... i've seen vpc fail and drop connections by shutting the newly crowned master's port-channels too often because it's state is inconsistent for some oddball reason ... stp is trusty rusty for me | 18:10 |
| *** vakuznet has joined #openstack-ansible | 18:10 | |
| idlemind | she might be old and seem kinda ugly but it's simple, supported everywhere and just works | 18:11 |
| jrosser | otherwise do vxlan and l3 underlay net | 18:11 |
| Tahvok | How can I check what nova version I actually use when using osa? | 18:12 |
| idlemind | no specifically for routed lans, the customer networks would vxlan'd but you bring up a good point, if i want those provider side networks, vxlan on an underlay is probably the better long term | 18:12 |
| idlemind | but then you get into the undercloud to run your overcloud discussion | 18:12 |
| jrosser | ? | 18:12 |
| jrosser | ovverlay network | 18:12 |
| jrosser | not cloud | 18:12 |
| idlemind | jrosser right i meant the undercloud to provide the vxlan to the overcloud unless you wanted to implement and manage vxlan manually (or another way, aci, etc) | 18:13 |
| jrosser | im lost now :) anyway can talk more later if you like | 18:14 |
| idlemind | just spit ballin' | 18:14 |
| idlemind | good stuff to talk about but probably way better to do over a glass of beer and a napkin if i can ever get out to one of the conferences | 18:14 |
| idlemind | onto other things ... odyssey4me the item of interest seems to be "yum-config-manager" it lists the "proxy" value for repo's when they don't have it in the configuration file (because it's learned from yum.conf) so when the yum-config-manager steps through all repo's it updates it from yum.conf but only if the repo already does not have a setting | 18:15 |
| idlemind | in places with an incorrect "proxy" value we'd have to check for and update it | 18:16 |
| idlemind | so maybe something like: when: proxyfromyum-config-manager-gather-repos != internal_lb_vip_addr ... set new proxy w/yum-config-manager ... | 18:17 |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server stable/queens: Do not restart rabbitmq when no version is changed https://review.openstack.org/566116 | 18:19 |
| idlemind | odyssey4me https://github.com/openstack/openstack-ansible-openstack_hosts/blob/stable/pike/tasks/openstack_host_install_yum.yml#L84-L85 | 18:19 |
| *** hachi has joined #openstack-ansible | 18:20 | |
| idlemind | that's how it seems to be getting "all" repo's | 18:20 |
| *** yolanda has quit IRC | 18:20 | |
| odyssey4me | mnaser if you only change one requirement it'll only build the new things - it will take time though, but not as much time as the first... unfortunately pip is not very fast at processing through files even if it's already got them | 18:22 |
| idlemind | http://paste.openstack.org/show/720314/ | 18:22 |
| idlemind | not sure how we'd parse the proxy out of that most correctly for comparison / updating | 18:22 |
| odyssey4me | Tahvok you can activate the venv and do 'pip freeze' - or just look at which wheels are in the os-releases/<your tag> folder | 18:23 |
| idlemind | also, probably not the best to do it in the "openstack-hosts" play if it effectively causes all other containers to need to be rebuilt to be fixed | 18:23 |
| openstackgerrit | German Eichberger proposed openstack/openstack-ansible-os_octavia stable/queens: Adds certificate generation https://review.openstack.org/565845 | 18:24 |
| odyssey4me | idlemind tbh how centos works and best practises there are a mystery to me... mhayden used to be a key maintainer there, but perhaps mnaser can pick up where he left off to have an intelligent discussion with you about it... :) | 18:24 |
| Tahvok | odyssey4me: it says 15.1.1.dev43 | 18:24 |
| odyssey4me | Tahvok and there you have it | 18:24 |
| idlemind | odyssey4me thx major left osa? | 18:24 |
| idlemind | #cry | 18:24 |
| * mnaser will read buffer in a second | 18:24 | |
| Tahvok | odyssey4me: but that does not mean 15.1.1 as we found at #openstack-nova channel... | 18:25 |
| Tahvok | First, 15.1.1 was released only yesterday | 18:25 |
| idlemind | oh hey he works for redhat now lol | 18:25 |
| Tahvok | And secondly, I do not have the patches that were released with it... | 18:25 |
| odyssey4me | Tahvok no, because we build from git source and pin at a SHA, the version will come from pbr and be determined as 15.1.1 + 43 commits on top of that | 18:26 |
| odyssey4me | Tahvok just because it was released yesterday, doesn't mean the sha they're using is not quite old ;) | 18:26 |
| Tahvok | I'm using :) | 18:26 |
| Tahvok | Anyway, can I request a sha bump? | 18:26 |
| idlemind | k chow time bb in a bit odyssey4me and mnaser | 18:26 |
| Tahvok | It fixes at least 2 important bugs with 'no bootable device' issue and 'live migration not working with scsi devices' | 18:27 |
| *** mma has joined #openstack-ansible | 18:27 | |
| *** hachi has quit IRC | 18:27 | |
| odyssey4me | Tahvok if you need a later SHA you can override it yourself temporarily - unless the fix you need/want would be useful to most others - then perhaps we should do a general bump... bumping a single SHA doesn't always work so nicely because of changing stuff in upper constraints | 18:27 |
| Tahvok | odyssey4me: the bugs are probably known to other ceph users... | 18:28 |
| Tahvok | So I would says it would be useful to others.. | 18:28 |
| mnaser | idlemind: let me know what you're looking to accomplish in centos world and id be happy to help | 18:29 |
| Tahvok | I can bump the sha for me, but I think others might benefit from this release | 18:29 |
| odyssey4me | ah ok... then see if just bumping the one sha in your test env works - if it does, push a patch up | 18:29 |
| Tahvok | Did we have any other plans for bumping shas in the coming weeks? | 18:29 |
| odyssey4me | given that we just bumped them, nope | 18:29 |
| odyssey4me | it's always around mid month and end of month | 18:30 |
| odyssey4me | Tahvok see https://docs.openstack.org/openstack-ansible/latest/contributor/periodic-work.html#releasing | 18:30 |
| odyssey4me | don't use that dependency update script though, it'll bump everything | 18:31 |
| Tahvok | So there should be a release in around 2 weeks? | 18:31 |
| Tahvok | Or I don't understand something... | 18:31 |
| *** mma has quit IRC | 18:32 | |
| odyssey4me | Tahvok yes, the current head of the branch will be released every two weeks... | 18:32 |
| odyssey4me | between each general bump we can do small bumps if we find crucial bugs, like the ones you're mentioning | 18:33 |
| *** yolanda has joined #openstack-ansible | 18:35 | |
| Tahvok | So just to make sure. I update the file playbooks/defaults/repo_packages/openstack_services.yml with the new sha I need | 18:36 |
| Tahvok | Test it, and then submit a review, or request here at the channel? | 18:36 |
| jrosser | if theres any ceph folks around this i'd appreciate some feedback on this https://review.openstack.org/#/c/565701/ | 18:38 |
| *** mma has joined #openstack-ansible | 18:39 | |
| Tahvok | jrosser: is it the ceph installation role? I'm not familiar with this... | 18:40 |
| Tahvok | We use only the ceph client in our case, and install ceph with ceph-ansible | 18:41 |
| jrosser | ok so i'm doing the same thing | 18:41 |
| jrosser | i've been working towards getting the osa playbooks to consume a list of externally provisioned rgw | 18:42 |
| jrosser | and only do the keystone and haproxy parts, rather than actually deploy the rgw itself | 18:42 |
| odyssey4me | Tahvok yes, you can submit the review - describe in the commit msg why it's important, preferably with the appropriate bugs/reviews referenced | 18:43 |
| Tahvok | odyssey4me: I'll have a production deploy on sunday with this sha bump, so I'll test it then and submit a review | 18:44 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_keystone master: Add support for using distribution packages for OpenStack services https://review.openstack.org/560308 | 18:45 |
| odyssey4me | cores - can I get some reviews for https://review.openstack.org/565999 please to help finalise the fixes to tempest | 18:53 |
| odyssey4me | it's a backport of the master patch, so easy peasey | 18:53 |
| cloudnull | idlemind: sorry was off eating :) | 18:57 |
| cloudnull | that config was from what we used to use in the OSIC | 18:58 |
| cloudnull | those were all cisco 9k switchs. | 18:58 |
| mgariepy | odyssey4me, done. | 19:00 |
| *** mma has quit IRC | 19:01 | |
| *** radeks has joined #openstack-ansible | 19:03 | |
| *** openstackgerrit has quit IRC | 19:05 | |
| *** mma has joined #openstack-ansible | 19:05 | |
| idlemind | cloudnull so likely vpc then | 19:09 |
| *** hamza21 has joined #openstack-ansible | 19:18 | |
| *** mma has quit IRC | 19:23 | |
| idlemind | cloudnull no food for you! jk i saw you posting changes at like 1am, maybe a nap too? | 19:26 |
| *** aludwar1 has joined #openstack-ansible | 19:50 | |
| *** mma has joined #openstack-ansible | 19:53 | |
| cloudnull | idlemind: yup, a nap would be nice :) | 19:54 |
| *** aludwar1 has quit IRC | 19:57 | |
| *** mma has quit IRC | 19:58 | |
| *** hamza21 has quit IRC | 20:14 | |
| *** radeks has quit IRC | 20:14 | |
| throwsb1 | I am running a fresh install of OSA with 3 infra, 2 storage with LVM, and 2 computes. I am running into issues of vm creation failing due to storage creation timing out. It looks like it round robins between storage nodes. This seems to be causing an error running up against timeouts for cinder. | 20:15 |
| throwsb1 | Has anyone run an environment with 2 storage nodes and had the same issues? Is it better to not have 2 storage nodes? If so, is there an easy or recommended way to remove one? | 20:16 |
| throwsb1 | Seems like on creation, it times out due to cluster communication. | 20:17 |
| *** aludwar1 has joined #openstack-ansible | 20:22 | |
| *** aludwar1 has quit IRC | 20:23 | |
| *** mma has joined #openstack-ansible | 20:39 | |
| *** evin has quit IRC | 20:42 | |
| idlemind | throwsb1 yes w/lvm you'll want to adjust timers | 20:44 |
| idlemind | i have a bug on it i think w/the correct adjustments | 20:44 |
| idlemind | throwsb1 http://paste.openstack.org/show/720323/ | 20:47 |
| idlemind | ^^ in user_variables.yml | 20:47 |
| idlemind | you'll see all deploy's for disks take at least 30 seconds in horizon | 20:48 |
| idlemind | w/that config | 20:48 |
| idlemind | and it will try 120 + 1 times | 20:48 |
| idlemind | i did such a long window because i had the 11gb or so windows image ... that always took forever to clone 3 or 4 of 'em | 20:49 |
| idlemind | w/lvm | 20:49 |
| *** vakuznet has quit IRC | 20:49 | |
| *** hamzy has quit IRC | 20:53 | |
| *** openstackgerrit has joined #openstack-ansible | 21:00 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_glance master: Add support for using distribution packages for OpenStack services https://review.openstack.org/566093 | 21:00 |
| *** kstev1 has quit IRC | 21:10 | |
| *** kstev has joined #openstack-ansible | 21:16 | |
| *** esberglu has quit IRC | 21:26 | |
| throwsb1 | idlemind: Thanks!, I will give i a try. | 21:26 |
| *** jwitko_ has joined #openstack-ansible | 21:32 | |
| *** jwitko has quit IRC | 21:36 | |
| *** jwitko_ has quit IRC | 21:37 | |
| *** esberglu has joined #openstack-ansible | 21:39 | |
| *** kstev1 has joined #openstack-ansible | 21:51 | |
| *** kstev has quit IRC | 21:52 | |
| throwsb1 | idlemind: I put it in place and still failing. I only pushed to the compute nodes, does it need to go to others as well? | 21:57 |
| throwsb1 | I did restart the nova services on compute nodes | 21:58 |
| *** jwitko has joined #openstack-ansible | 22:01 | |
| idlemind | throwsb1 you'll have to re-run the os-nova and possibly os-cinder plays at least | 22:07 |
| idlemind | (w/osa you shouldn't be editing config files direct) | 22:08 |
| throwsb1 | I just deployed the user vars and deployed os-nova-install.yml | 22:08 |
| idlemind | ya run os-cinder and possibly os-horizon as well just to be safe | 22:09 |
| throwsb1 | ok | 22:09 |
| idlemind | it's been a minute since i've hit it | 22:09 |
| throwsb1 | idlemind: is it bad to just rerun the 3 playbooks, setup-hosts, setup-infrastructure, and setup-openstack? I am still a nub and still learning. | 22:12 |
| idlemind | throwsb1 no it "shouldn't" hurt anything at all | 22:13 |
| idlemind | i think recommendation from the wizards is just to cyclically run them for the hell of it | 22:14 |
| throwsb1 | ok. thanks. | 22:14 |
| idlemind | * is just to NOT | 22:14 |
| idlemind | if you got a reason go for it tho | 22:14 |
| idlemind | ok, so yum aside i cannot get past os-nova w/this error. i have 2 compute nodes and each one fails when doing a cell_v2 discover. it seems to be trying to ssh into the the conductor not local to itself and dies. if i go to the console i'm able to ssh. | 22:34 |
| idlemind | http://paste.openstack.org/show/720325/ | 22:34 |
| idlemind | deleting and recreating the conductor containers has not helped | 22:36 |
| dmsimard | cloudnull, evrardjp, odyssey4me, mnaser: oi, ara 0.15.0 has been released -- keep an eye out and let me know if there's anything weird ? | 22:39 |
| *** kstev1 has quit IRC | 22:43 | |
| *** throwsb1 has quit IRC | 22:48 | |
| idlemind | ok are my nova compute nodes (metal) supposed to have database sections in the nova.conf? | 22:48 |
| idlemind | cuz mine don't | 22:48 |
| idlemind | i add them in and the command succeeds instead of fails. if the playbook doesn't reset my nova.conf i suspect it will clear this stage /openstack/venvs/nova-16.0.10/bin/nova-manage cell_v2 discover_hosts --verbose | 22:49 |
| *** threestrands has joined #openstack-ansible | 22:59 | |
| *** klamath has quit IRC | 23:17 | |
| evrardjp | dmsimard: yup will bump it -- adding this to my super long todo list. | 23:21 |
| *** masber has joined #openstack-ansible | 23:23 | |
| idlemind | http://paste.openstack.org/show/720326/ | 23:33 |
| idlemind | it would seem my cell's are mapped to statically that way | 23:33 |
| *** weezS has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!