| *** tosky has quit IRC | 00:11 | |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts master: Remove single task from lxc_apparmor file https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/776240 | 00:44 |
|---|---|---|
| *** LowKey has quit IRC | 01:28 | |
| *** LowKey has joined #openstack-ansible | 01:28 | |
| *** tinwood has quit IRC | 03:10 | |
| *** tinwood has joined #openstack-ansible | 03:13 | |
| *** cloudnull has joined #openstack-ansible | 03:15 | |
| *** LowKey has quit IRC | 04:22 | |
| *** cloudnull has quit IRC | 04:58 | |
| *** cloudnull has joined #openstack-ansible | 05:00 | |
| *** cloudnull has quit IRC | 05:31 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** cloudnull has joined #openstack-ansible | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:33 | |
| *** cloudnull has quit IRC | 06:13 | |
| *** cloudnull has joined #openstack-ansible | 06:14 | |
| *** miloa has joined #openstack-ansible | 06:55 | |
| *** SiavashSardari has joined #openstack-ansible | 06:58 | |
| *** cloudnull has quit IRC | 07:10 | |
| *** cloudnull has joined #openstack-ansible | 07:15 | |
| *** noonedeadpunk has joined #openstack-ansible | 07:38 | |
| *** maharg101 has joined #openstack-ansible | 07:47 | |
| *** pcaruana has joined #openstack-ansible | 08:01 | |
| *** rpittau|afk is now known as rpittau | 08:04 | |
| noonedeadpunk | mornings | 08:04 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: update to osquery https://review.opendev.org/c/openstack/openstack-ansible-ops/+/613136 | 08:07 |
| *** cloudnull has quit IRC | 08:10 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Get elastic plugins via proxy https://review.opendev.org/c/openstack/openstack-ansible-ops/+/769701 | 08:13 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Allow longer timeouts for elasticsearch.service https://review.opendev.org/c/openstack/openstack-ansible-ops/+/769695 | 08:14 |
| *** cloudnull has joined #openstack-ansible | 08:14 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Fix elk 7 {conf,env}.d sample https://review.opendev.org/c/openstack/openstack-ansible-ops/+/762331 | 08:15 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Fix regex_replace mapping https://review.opendev.org/c/openstack/openstack-ansible-ops/+/762519 | 08:15 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Make ELK Journalbeat seek configuration a variable https://review.opendev.org/c/openstack/openstack-ansible-ops/+/744180 | 08:16 |
| *** miloa has quit IRC | 08:16 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Remove `elastic_lxc_template_config` mapping https://review.opendev.org/c/openstack/openstack-ansible-ops/+/762501 | 08:16 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Correct spelling mistakes https://review.opendev.org/c/openstack/openstack-ansible-ops/+/619530 | 08:16 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Switch deprecated options to "modern" equivalents https://review.opendev.org/c/openstack/openstack-ansible-ops/+/770064 | 08:17 |
| *** luksky has joined #openstack-ansible | 08:17 | |
| *** tosky has joined #openstack-ansible | 08:20 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Fixes for tests/gates https://review.opendev.org/c/openstack/openstack-ansible-ops/+/741997 | 08:33 |
| noonedeadpunk | are we ok with https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/777559 ? | 08:42 |
| noonedeadpunk | As I'd love to backport to V and finally get 22.1.0 | 08:42 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia stable/victoria: Add variable to override keypair setup python interpreter https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/773962 | 08:44 |
| noonedeadpunk | probably worth including octavia patch as well though... | 08:44 |
| jrosser | morning | 08:50 |
| jrosser | noonedeadpunk: also this needed for V release https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/777798 | 08:51 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Make focal jobs voting https://review.opendev.org/c/openstack/openstack-ansible-ops/+/777940 | 08:52 |
| noonedeadpunk | um... | 08:53 |
| noonedeadpunk | it's not valid at this shape.... | 08:54 |
| noonedeadpunk | it hsould be revert of https://opendev.org/openstack/ansible-role-python_venv_build/commit/ab68e944256648cb6d714e02dbe88d228a2c04a5 then | 08:54 |
| noonedeadpunk | at least it's missing `venv_wheel_build_enable | bool` condition? | 08:56 |
| jrosser | oh whoops yes | 08:56 |
| noonedeadpunk | and doesn;t it solve the same issue as https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/777559 ? | 08:57 |
| jrosser | no, becasue it seems run_once behaviour goes completely wrong | 08:57 |
| jrosser | see looong debug in irc scrollback with mgariepy | 08:58 |
| noonedeadpunk | uh, I think I don't have it. ned to go to eavesdrop then... | 08:58 |
| noonedeadpunk | the nasty thing it that we've dropped 200 skipped tasks with include? | 08:59 |
| noonedeadpunk | maybe we can workaround in some better way.... | 09:00 |
| jrosser | yes we should revisit | 09:02 |
| noonedeadpunk | btw we don't set fact anywhere in `python_venv_wheel_build` | 09:02 |
| * noonedeadpunk goes to eavesdrop | 09:02 | |
| jrosser | well register/set_fact are similar in this respect i think | 09:02 |
| jrosser | i got the wrong term there but i think the outcome is similar | 09:03 |
| noonedeadpunk | it's about `_constraints_file_slurp` I guess | 09:03 |
| jrosser | yes this https://github.com/openstack/ansible-role-python_venv_build/blob/master/tasks/python_venv_wheel_build.yml#L168-L173 | 09:03 |
| jrosser | needs to be set for all hosts even though the wheel build is delegated/run_once | 09:04 |
| noonedeadpunk | why not to move it just out of there? | 09:04 |
| noonedeadpunk | in case lsync is working, it don't matter where we get it afterwards | 09:05 |
| jrosser | not really, it's for here https://github.com/openstack/ansible-role-python_venv_build/blob/ab68e944256648cb6d714e02dbe88d228a2c04a5/tasks/python_venv_install.yml#L66-L79 | 09:05 |
| noonedeadpunk | ok, let me test this. I think currently things will be jsut broken, right? | 09:06 |
| jrosser | and the bug was we end up with {{ _venv_pip_packages }} in the constraints on the 2nd/3rd nodes | 09:06 |
| noonedeadpunk | when wheel built is enabled | 09:06 |
| jrosser | whilst the first node gets _constraints_file_slurp.content | 09:07 |
| jrosser | so this only shows up for multinode, which is why we didnt see a problem in CI | 09:07 |
| noonedeadpunk | ok, have multinode as well atm | 09:07 |
| jrosser | yeah, so the git://<url> was being written to the target node constraints, rather than the list of constraints from the wheel build process | 09:08 |
| noonedeadpunk | ok, gotcha. Just really loved to leave include here tbh | 09:09 |
| jrosser | we could move the register task up to the main.yml, and add the condition/run_once to that | 09:13 |
| noonedeadpunk | yeah, that eventually what I meant under move it out I guess | 09:14 |
| *** fresta_ has quit IRC | 09:14 | |
| noonedeadpunk | or even to python_venv_install.yml | 09:15 |
| noonedeadpunk | because it's the only place where it's used | 09:15 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Make ELK Journalbeat seek configuration a variable https://review.opendev.org/c/openstack/openstack-ansible-ops/+/744180 | 09:18 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Get elastic plugins via proxy https://review.opendev.org/c/openstack/openstack-ansible-ops/+/769701 | 09:21 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Remove `elastic_lxc_template_config` mapping https://review.opendev.org/c/openstack/openstack-ansible-ops/+/762501 | 09:21 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Fix regex_replace mapping https://review.opendev.org/c/openstack/openstack-ansible-ops/+/762519 | 09:21 |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Use ansible_facts[] instead of fact variables https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/777068 | 09:43 |
| *** MickyMan77 has joined #openstack-ansible | 09:59 | |
| openstackgerrit | Merged openstack/ansible-role-systemd_service master: Make systemd.service more consistent https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/771869 | 10:08 |
| jrosser | noonedeadpunk: did you see this on the ML? https://bugs.launchpad.net/kolla-ansible/+bug/1917068 | 10:10 |
| openstack | Launchpad bug 1917068 in kolla-ansible wallaby "Connections to DB are refusing to die after VIP is switched" [Medium,In progress] - Assigned to Michal Arbet (michalarbet) | 10:10 |
| noonedeadpunk | nope | 10:11 |
| jrosser | feels like we should be setting that too i think | 10:11 |
| *** fresta has joined #openstack-ansible | 10:13 | |
| noonedeadpunk | yeah, agreed | 10:13 |
| noonedeadpunk | we can set that in openstack_hosts? | 10:14 |
| jrosser | we can | 10:15 |
| noonedeadpunk | oh, hm | 10:15 |
| noonedeadpunk | we can patch keepalived role instead I guess | 10:16 |
| noonedeadpunk | https://github.com/evrardjp/ansible-keepalived/blob/master/tasks/main.yml#L54-L65 | 10:16 |
| *** jawad_axd has joined #openstack-ansible | 10:21 | |
| noonedeadpunk | let's see how it goes https://github.com/evrardjp/ansible-keepalived/pull/168 | 10:23 |
| *** miloa has joined #openstack-ansible | 10:28 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Use ansible_facts[] instead of fact variables https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/777975 | 10:30 |
| *** miloa has quit IRC | 10:37 | |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Gather facts for repo containers https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/777559 | 10:37 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build stable/victoria: Gather facts for repo containers https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/777908 | 10:42 |
| openstackgerrit | Merged openstack/openstack-ansible-os_zun master: Improve image and network cleanup procedure https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/771218 | 10:45 |
| *** SiavashSardari has quit IRC | 10:51 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Fix elk 7 {conf,env}.d sample https://review.opendev.org/c/openstack/openstack-ansible-ops/+/762331 | 11:02 |
| jawad_axd | Hi , a quick question. Are encrypted cinder volumes supported with ceph rbd backend ? Couldt find any update on this one. I am using OSA stein 19.0.3/centos7.9. @noonedeadpunk if you can drop a comment on this one. Thanks | 11:08 |
| noonedeadpunk | jawad_axd: yep, they are. Barbican in stein might not be configured super properly - you need to ensure you're overriding master key in config in case of simple crypto backend | 11:28 |
| *** gshippey has joined #openstack-ansible | 11:57 | |
| evrardjp | good morning folks | 12:03 |
| noonedeadpunk | o/ | 12:03 |
| evrardjp | noonedeadpunk: I am checking why the test jobs are "scheduled" and not running | 12:03 |
| evrardjp | I find that super weird the state of the job is not shown in the testing itself. | 12:04 |
| evrardjp | (of the PR) | 12:04 |
| noonedeadpunk | yeah, eventually that's wrong at the first place | 12:04 |
| noonedeadpunk | regarding setting value - let's make it variable indeed? I was also thinking if we should move this whole list to the variable? | 12:05 |
| noonedeadpunk | just not sure about adding another task there, and `ip_nonlocal_bind` we want to be always set to 1 anyway | 12:06 |
| evrardjp | yeah, I would keep it the same task, just add an extra variable for the brave ppl, and put it in 15 by default | 12:08 |
| evrardjp | I would also add a comment on the var to explain a more aggressive solution would be 3 (with a link to RH KB), 6 gives around 10 seconds and 8 is the recommended minimum for RFC. | 12:09 |
| noonedeadpunk | well, 15 results in 15 minutes of fallback? | 12:09 |
| evrardjp | 15 is the current default | 12:09 |
| noonedeadpunk | maybe we set default to 8? | 12:09 |
| evrardjp | and yeah 13 minutes | 12:10 |
| evrardjp | 8 makes it a minute. It's fine indeed. | 12:10 |
| evrardjp | as long as we document why we do it, and suggestions on how to tweak, I think we are good | 12:10 |
| noonedeadpunk | because default is way too much imo... But agree that 3 is too agressive | 12:10 |
| evrardjp | yeah I can totally see why in some environment 3 would be a disaster | 12:11 |
| evrardjp | I am not sure why the lint is failing on galaxy_info.author | 12:11 |
| evrardjp | Apparently it has changed ? | 12:11 |
| evrardjp | And now it needs to be in the form namespace.role ? | 12:11 |
| evrardjp | Did you change anything in OSA? | 12:12 |
| noonedeadpunk | nope | 12:12 |
| noonedeadpunk | we ignore meta checks completely from other hand | 12:12 |
| evrardjp | even the docs would be not pass the ansible-lint >< https://galaxy.ansible.com/docs/contributing/creating_role.html#role-metadata | 12:14 |
| noonedeadpunk | oh, and we have ansivble-lint bumped :p | 12:15 |
| evrardjp | what do you mean? | 12:15 |
| evrardjp | I am using latest, IIRC | 12:15 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/test-requirements.txt#L14 | 12:17 |
| noonedeadpunk | so we are out of chances to catch it anyway | 12:17 |
| evrardjp | oh you meant that it's frozen :) Ok got it :) | 12:20 |
| evrardjp | ofc if it's regularly bumped it means it's frozen . | 12:20 |
| evrardjp | <- dumbass | 12:20 |
| noonedeadpunk | Nah, I used wierd wording actually... | 12:21 |
| evrardjp | Interstingly, I can't even cancel the workflow that's stuck in scheduling. | 12:21 |
| noonedeadpunk | hu, and now it's launched | 12:22 |
| evrardjp | oh really? | 12:22 |
| noonedeadpunk | feels like smth wrong with github itself I guess | 12:22 |
| noonedeadpunk | well, I force pushed | 12:22 |
| evrardjp | well, you have two runs! :p | 12:22 |
| evrardjp | https://github.com/evrardjp/ansible-keepalived/actions/workflows/molecule.yml | 12:22 |
| evrardjp | previous is still scheduled and stuck | 12:22 |
| evrardjp | sadly, they are gonna fail, due to the meta problem I just mentioned. I am testing a fix. | 12:23 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Switch deprecated options to "modern" equivalents https://review.opendev.org/c/openstack/openstack-ansible-ops/+/770064 | 12:23 |
| noonedeadpunk | what I hate the most is a rebase process in the github | 12:24 |
| evrardjp | in another repo, I made a nice rebase tooling. it's super neat for the contributors. I didn't do it here, I think the most regular experience is better :) | 12:28 |
| noonedeadpunk | is it called gerrit ?:) | 12:28 |
| evrardjp | Anyway, trying to understand why cryptography is complaining a bit, and I will contact you on github when I have fixed CI. | 12:28 |
| evrardjp | haha | 12:28 |
| evrardjp | ofc gerrit still beats github :D | 12:29 |
| noonedeadpunk | oh, modern cryptography requires rust | 12:29 |
| noonedeadpunk | which is so frustrating... | 12:29 |
| noonedeadpunk | jrosser: I'm not able to see issues with venv build with current master of python_venv_build... http://paste.openstack.org/show/803092/ | 12:29 |
| noonedeadpunk | am I missing smth? | 12:29 |
| jrosser | i think we should check in with mgariepy | 12:31 |
| jrosser | is yours on focal? | 12:31 |
| noonedeadpunk | I think bionic | 12:31 |
| noonedeadpunk | should it matter? | 12:31 |
| jrosser | did you see in the scrollback we even added some debug tasks which showed that the slurp var was undefined? | 12:31 |
| jrosser | defined on host1, not defined on 2 & 3 | 12:32 |
| noonedeadpunk | re-reading it | 12:32 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Gather minimal facts for dynamic_address_fact https://review.opendev.org/c/openstack/openstack-ansible/+/777989 | 12:36 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Correct spelling mistakes https://review.opendev.org/c/openstack/openstack-ansible-ops/+/619530 | 12:38 |
| evrardjp | noonedeadpunk: correct, that's indeed the problem I see. I am just trying to check what I can do. | 12:43 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Gather minimal facts in common playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/777990 | 12:43 |
| *** tbarron|out is now known as tbarron | 12:51 | |
| noonedeadpunk | jrosser: ok, I dunno how, but I don't see any issue happening http://paste.openstack.org/show/803094/ | 13:07 |
| noonedeadpunk | (with same debug actually) | 13:07 |
| *** zul has joined #openstack-ansible | 13:08 | |
| *** miloa has joined #openstack-ansible | 13:10 | |
| *** miloa has quit IRC | 13:10 | |
| *** rh-jlabarre has joined #openstack-ansible | 13:12 | |
| noonedeadpunk | oh, ok, I see | 13:14 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: test-ansible-env-prep: simplify ara setup https://review.opendev.org/c/openstack/openstack-ansible-tests/+/777697 | 13:19 |
| noonedeadpunk | hm, why do we even slurp, when we can use get_url safely here? | 13:20 |
| jrosser | maybe to decouple python_venv_build from the repo server? | 13:22 |
| jrosser | perhaps it's coupled anyway..... | 13:22 |
| jrosser | ideally the role is generic and can stand alone | 13:22 |
| noonedeadpunk | yeah, well, you're right... | 13:22 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Slurp constraints for all hosts https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/777994 | 13:23 |
| noonedeadpunk | I'm wondering how log we had this issue? | 13:24 |
| noonedeadpunk | because https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/stable/train/tasks/main.yml#L21 | 13:25 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-lxc_hosts master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/777995 | 13:25 |
| jrosser | right, but run_once has a side effect of applying things like facts and registered vars to all the play hosts | 13:27 |
| jrosser | even though the task only runs on one of them | 13:27 |
| noonedeadpunk | ah, unless it's include? | 13:28 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/777996 | 13:28 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in group_vars https://review.opendev.org/c/openstack/openstack-ansible/+/777997 | 13:28 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in tests https://review.opendev.org/c/openstack/openstack-ansible/+/777998 | 13:28 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in AIO config https://review.opendev.org/c/openstack/openstack-ansible/+/777999 | 13:28 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in docs https://review.opendev.org/c/openstack/openstack-ansible/+/778000 | 13:28 |
| jrosser | feels like swapping include/import broke that behaviour somehow | 13:28 |
| jrosser | set_fact+run_once is super useful | 13:29 |
| jrosser | actually in this case it's delegate_to as well, so it runs the task against something thats not one of the play hosts, but applies the registered var to them anyway | 13:30 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Slurp constraints for all hosts https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/777994 | 13:32 |
| *** mgariepy has joined #openstack-ansible | 13:44 | |
| *** luksky has quit IRC | 13:48 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Use ansible_facts[] instead of fact variables https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/776959 | 13:52 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Gather minimal facts in common playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/777990 | 13:56 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/777996 | 13:56 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in group_vars https://review.opendev.org/c/openstack/openstack-ansible/+/777997 | 13:56 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in tests https://review.opendev.org/c/openstack/openstack-ansible/+/777998 | 13:56 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in AIO config https://review.opendev.org/c/openstack/openstack-ansible/+/777999 | 13:56 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in docs https://review.opendev.org/c/openstack/openstack-ansible/+/778000 | 13:56 |
| *** luksky has joined #openstack-ansible | 14:01 | |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts master: Remove workarounds for old systemd bugs https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/776241 | 14:05 |
| *** luksky has quit IRC | 14:17 | |
| noonedeadpunk | evrardjp: while thinking through tcp_retries2 one more time - isn't it smth that should be applied to all hosts that reach keepalived but not keepalived itself? | 14:22 |
| evrardjp | I am no expert in this, but it makes sense to me that keepalived retries more aggresively on lost backend | 14:24 |
| evrardjp | it might not be enough coverage for your case if it's a flap between keepalived masters | 14:24 |
| evrardjp | what's the root cause that brought you to this problem? | 14:25 |
| noonedeadpunk | It was eventually https://bugs.launchpad.net/kolla-ansible/+bug/1917068 | 14:25 |
| openstack | Launchpad bug 1917068 in kolla-ansible wallaby "Connections to DB are refusing to die after VIP is switched" [Medium,In progress] - Assigned to Michal Arbet (michalarbet) | 14:25 |
| evrardjp | noonedeadpunk: yeah that will not be enough for OSA cases. | 14:30 |
| evrardjp | noonedeadpunk: what about openstack-hosts role? Would it make sense to have it there too, if possible wired with the same variable name, so that it's easier to set for all our users? | 14:31 |
| *** luksky has joined #openstack-ansible | 14:32 | |
| noonedeadpunk | yeah, it was my first thought... | 14:33 |
| evrardjp | https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/defaults/main.yml#L125 maybe there? | 14:34 |
| noonedeadpunk | I think that applying kernel options on host will apply it on containers as well | 14:34 |
| evrardjp | not all apply, that's why I think it's best to test :) Apply it to host first, see how that goes when you spin up your container | 14:35 |
| noonedeadpunk | yeah, this needs to be applied on all containers.... uh.. | 14:38 |
| evrardjp | you can write a release note that ppl should restart the sysctl-container service | 14:39 |
| evrardjp | if that works | 14:39 |
| evrardjp | else, it's probably worth adding into the lxc_container_create role, wdyt? | 14:40 |
| noonedeadpunk | is sysctl-container smth from k8s world? | 14:40 |
| evrardjp | nope it's something we wrote | 14:41 |
| evrardjp | IIRC | 14:41 |
| evrardjp | Could you rebase https://github.com/evrardjp/ansible-keepalived/pull/168 ? | 14:41 |
| noonedeadpunk | yeah, sure | 14:41 |
| jamesdenton | evrardjp! | 14:41 |
| evrardjp | noonedeadpunk: https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/master/tasks/lxc_container_config.yml#L318-L327 + https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/master/templates/sysctl-container.init.j2 + https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/master/handlers/main.yml#L116-L126 | 14:43 |
| evrardjp | jamesdenton: hey buddy! How are things? | 14:43 |
| evrardjp | It's nice to you! | 14:43 |
| evrardjp | It's nice to see all of you in fact! :) | 14:43 |
| jamesdenton | things are good! great to see you, too | 14:44 |
| evrardjp | It would be better in real life though! :) | 14:44 |
| jamesdenton | i've been MIA but trying to work my way back in :D | 14:44 |
| noonedeadpunk | til | 14:44 |
| jamesdenton | yes, agreed | 14:44 |
| noonedeadpunk | and found that we need to use systemd_service role for this as well :p | 14:44 |
| evrardjp | noonedeadpunk: why? | 14:44 |
| evrardjp | it's just done on the creation of the image, after that it should simply roll, isn't it? | 14:45 |
| evrardjp | noonedeadpunk: or do you mean that this variable isn't applied, even with that hack? (It's possible !) | 14:45 |
| evrardjp | jamesdenton: hehe I can relate :) | 14:46 |
| noonedeadpunk | I mean we should not store template? as we can generate it with the role? | 14:46 |
| noonedeadpunk | nah, It's applied in case it's in containers sysctl | 14:47 |
| evrardjp | noonedeadpunk: sure, I guess we could use the systemd_service role. Not sure if we need the full blown thing, as it's not meant to be customized by our users | 14:48 |
| evrardjp | I think that was the decision we took long ago... Not everything need to be a systemd_service. | 14:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon master: Use ansible_facts[] instead of fact variables https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/777651 | 14:48 |
| evrardjp | If you folks think that some services fall into the cracks because they don't follow a standard, you should indeed update things :) | 14:49 |
| evrardjp | noonedeadpunk: so you mean that if you change it in the hosts, and restart the service, that's not enough? | 14:50 |
| noonedeadpunk | yeah maybe in this case it's fair indeed | 14:50 |
| evrardjp | restart the service on containers* | 14:50 |
| noonedeadpunk | http://paste.openstack.org/show/803097/ | 14:52 |
| noonedeadpunk | well, it runs `/sbin/sysctl -p` and `/etc/sysctl.conf` is empty inside container | 14:53 |
| noonedeadpunk | so it shouldn't apply it I guess? | 14:53 |
| evrardjp | mmm yes, so that's what I was afraid of | 14:54 |
| noonedeadpunk | we probably should somehow bind mount sysctl file inside container but not sure | 14:55 |
| noonedeadpunk | (for this to work) | 14:55 |
| evrardjp | I guess if you sysctl -w net.ipv4.tcp_retries2 you can edit the variable | 14:55 |
| evrardjp | in the container, right? | 14:55 |
| noonedeadpunk | yep | 14:55 |
| noonedeadpunk | http://paste.openstack.org/show/803098/ | 14:55 |
| evrardjp | I have to dig into OSA , I didn't follow what we last did. IIRC I wanted to have openstack_hosts running also in containers | 14:56 |
| evrardjp | that could be a solution | 14:56 |
| noonedeadpunk | we have it running | 14:56 |
| noonedeadpunk | just not part with `openstack_kernel_options` | 14:57 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/tasks/main.yml#L70-L75 | 14:58 |
| evrardjp | yeah, I think this might be worth doing on a separate task, outside the is_metal flag | 14:58 |
| noonedeadpunk | yeah, agree | 14:59 |
| evrardjp | I am curious about _some_ of the variables we have there, like https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/defaults/main.yml#L111 | 15:00 |
| evrardjp | is that well applied into the containers right now? | 15:00 |
| noonedeadpunk | it's not. all these `openstack_kernel_options` are bare metal thing only | 15:01 |
| evrardjp | yeah, but some are _meant_ to be also for containers, IIRC? | 15:01 |
| evrardjp | so we might hit a bug for those? | 15:01 |
| openstackgerrit | Merged openstack/openstack-ansible master: scripts-library: simplify ara setup https://review.opendev.org/c/openstack/openstack-ansible/+/777696 | 15:02 |
| noonedeadpunk | well this specific thing was implemented with https://opendev.org/openstack/openstack-ansible-openstack_hosts/commit/7cbf51170c9ad25ac6a36ebc94561018a3b310b5 | 15:02 |
| noonedeadpunk | neutron agents are on bare metal by default... | 15:02 |
| evrardjp | Yeah I am checking, but I remember that they are on bare metal by default now | 15:03 |
| evrardjp | yeah exactly | 15:03 |
| noonedeadpunk | but theoretically we might want to have this in containers | 15:03 |
| evrardjp | I think we moved those to bare metal because it was a PITA in upgrades | 15:03 |
| evrardjp | I suppose we won't rollback on that idea soon, so I guess the only thing remaining is to have an extra task in the role for sysctl, and therefore no need for a release note saying that ppl need to run a specific command to get that taken into consideration :D | 15:04 |
| evrardjp | I am sorry if I don't have much time to do it myself :s | 15:05 |
| noonedeadpunk | I'm just can't stop thinking that there's a place where we configure sysctl for containers... | 15:06 |
| evrardjp | most likely | 15:07 |
| evrardjp | but in this case you want it to apply _everywhere_ so it makes sense to apply it there, doesn't it? | 15:07 |
| noonedeadpunk | well... I guess you are right as ususal | 15:09 |
| evrardjp | lol | 15:10 |
| evrardjp | tell that to my wife. | 15:11 |
| evrardjp | :D | 15:11 |
| noonedeadpunk | lol | 15:11 |
| noonedeadpunk | I can bet no wife will agree with that statement anyway | 15:11 |
| evrardjp | I suppose that also applies to husband and significant others :) | 15:12 |
| evrardjp | haha | 15:12 |
| noonedeadpunk | totally | 15:13 |
| evrardjp | so I checked, all the roles are indeed surgically changing the sysctl, which makes sense | 15:13 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible master: Add integrated test for Ubuntu using ML2/OVS driver https://review.opendev.org/c/openstack/openstack-ansible/+/778008 | 15:13 |
| noonedeadpunk | I't kind of basisc of all relationships I guess, that another half can't be right | 15:13 |
| evrardjp | so if you think this var should be surgically edited, it might be worth done in a more specific role | 15:13 |
| evrardjp | noonedeadpunk: maybe it's just that everyone wants to be right :) | 15:13 |
| noonedeadpunk | or that) | 15:13 |
| jamesdenton | noonedeadpunk yes, i think the intention was to change that on the os_neutron side, i'm just rusty :D | 15:18 |
| jrosser | something is definatly wrong with the log collection | 15:22 |
| jrosser | heres a glance container from a job that passed/merged https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_333/777696/1/gate/openstack-ansible-deploy-aio_lxc-ubuntu-focal/3333292/logs/openstack/aio1_glance_container-4537b1d9/ | 15:22 |
| jrosser | but no journal | 15:22 |
| jrosser | hmmm https://zuul.opendev.org/t/openstack/build/33332926b03c473089332060af29e8d4/log/job-output.txt#27774-27787 | 15:24 |
| jawad_axd | @noonedeadpunk Thanks | 15:34 |
| noonedeadpunk | evrardjp: btw, where you have found sysctl being applied for containers? | 15:42 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible master: Add ML2/OVS integrated test vars https://review.opendev.org/c/openstack/openstack-ansible/+/778008 | 15:44 |
| noonedeadpunk | jamesdenton: well, we kind of can define jobs.yaml in integrated repo, buit shouldn't in project-templates.yaml :) | 15:46 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: Add integrated tests for ML2/OVS on Ubuntu and CentOS https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/778015 | 15:46 |
| jamesdenton | can the jobs not be defined within os_neutron? | 15:47 |
| noonedeadpunk | this should work as well though | 15:47 |
| jamesdenton | kk | 15:47 |
| noonedeadpunk | yes, they can for sure | 15:47 |
| noonedeadpunk | as jobs are shared across all openstack projects | 15:47 |
| *** spatel has joined #openstack-ansible | 15:48 | |
| jamesdenton | if this works i'll move the ovn job out and into os_neutron | 15:48 |
| noonedeadpunk | at least out of project-templates :) | 15:49 |
| jamesdenton | sure ok :) | 15:49 |
| jamesdenton | i'll try to make it consistent | 15:49 |
| noonedeadpunk | cool, thanks | 15:49 |
| noonedeadpunk | really appreciated as I should have done that a month ago :( | 15:50 |
| jamesdenton | no problem | 15:51 |
| evrardjp | noonedeadpunk: in some roles, like neutron, nova IIRC | 15:56 |
| evrardjp | or lxc_hosts | 15:56 |
| jrosser | noonedeadpunk: something is broken with the journal collection - i don't really see what though | 16:13 |
| * jrosser looks again at https://review.opendev.org/c/openstack/openstack-ansible/+/734883/4/scripts/journal_dump.py | 16:13 | |
| *** macz_ has joined #openstack-ansible | 16:13 | |
| *** macz_ has quit IRC | 16:13 | |
| *** macz_ has joined #openstack-ansible | 16:14 | |
| noonedeadpunk | oh, btw we don't need lxc now | 16:19 |
| noonedeadpunk | hm, wondering if that might be related to bind mount somehow | 16:19 |
| noonedeadpunk | *to fixed bind mount | 16:19 |
| jrosser | i was just looking to take that out | 16:20 |
| jrosser | though we still would need it to get the pid and retrieve the files in /etc/ | 16:20 |
| jrosser | perhaps that can be worked around by calling out to lxc-info though | 16:20 |
| jrosser | yes actually lxc-info --pid <container-name> | 16:24 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Decrease TCP retries in case of VIP failover https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/778028 | 16:25 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/772042 | 16:26 |
| *** macz_ has quit IRC | 16:27 | |
| *** macz_ has joined #openstack-ansible | 16:28 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/772018 | 16:28 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Decrease TCP retries in case of VIP failover https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/778028 | 16:29 |
| *** gyee has joined #openstack-ansible | 16:53 | |
| *** rpittau is now known as rpittau|afk | 17:00 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Convert host and container journal collection to python https://review.opendev.org/c/openstack/openstack-ansible/+/734883 | 17:01 |
| *** maharg101 has quit IRC | 17:08 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Gather minimal facts in common playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/777990 | 17:12 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/777996 | 17:12 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in group_vars https://review.opendev.org/c/openstack/openstack-ansible/+/777997 | 17:12 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in tests https://review.opendev.org/c/openstack/openstack-ansible/+/777998 | 17:12 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in AIO config https://review.opendev.org/c/openstack/openstack-ansible/+/777999 | 17:12 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Use ansible_facts[] instead of fact variables in docs https://review.opendev.org/c/openstack/openstack-ansible/+/778000 | 17:12 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Convert host and container journal collection to python https://review.opendev.org/c/openstack/openstack-ansible/+/734883 | 17:22 |
| noonedeadpunk | I really feel troublesome to maintain common_etc_log_names in both places. I can bet we will forget at some point... | 17:23 |
| jrosser | yeah, like i did already :) | 17:25 |
| noonedeadpunk | I can bet I would be next one | 17:27 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Convert host and container journal collection to python https://review.opendev.org/c/openstack/openstack-ansible/+/734883 | 17:44 |
| jrosser | ^ that takes the first arg as a-r-r.yml then all the following ones are service names | 17:45 |
| noonedeadpunk | oh, neat, indeed it's not new line, shame on me | 17:48 |
| noonedeadpunk | mgariepy: can you check if https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/777994 also covers your issue which you figured out back on friday? | 17:49 |
| noonedeadpunk | btw we need to push cycle-highlights. but let's discuss this tomorrow during meeting | 17:49 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Use barbican for certificates storage https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/768514 | 17:51 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/769952 | 17:52 |
| * jrosser thinks we should start the pki stuff :/ | 17:52 | |
| jrosser | so much to fix... | 17:52 |
| *** MrClayPole_ has quit IRC | 17:53 | |
| noonedeadpunk | yeah and fwiw I'll have vacation with super limited internet connectivity till 15th March | 17:53 |
| *** MrClayPole has joined #openstack-ansible | 17:53 | |
| noonedeadpunk | I hope to have time for pli right after that | 17:56 |
| noonedeadpunk | *pki | 17:56 |
| noonedeadpunk | though we should put it to highlights IMO. | 17:56 |
| noonedeadpunk | despite it's big goal and nothing done atm, we can't release without rabbit being secured anyway I guess? And we would need root CA for it anyway | 17:57 |
| noonedeadpunk | So we must have at least smth anyway | 17:57 |
| noonedeadpunk | And CentOS Stream and Bullseye support I guess? | 17:58 |
| jrosser | hopefully yes | 18:02 |
| noonedeadpunk | Not so sure Bullseye since it feels it won't be released until openstack release at least... | 18:03 |
| jrosser | it means getting images and stuff made | 18:06 |
| jrosser | I did the first patch today to add it to the infra mirrors | 18:07 |
| noonedeadpunk | yep, saw it | 18:07 |
| mgariepy | noonedeadpunk, i'm looking yep should do the job\ | 18:28 |
| mgariepy | i can test it in an hour or so. | 18:29 |
| noonedeadpunk | sure, no rush I guess?:) | 18:29 |
| noonedeadpunk | your vote in your hands:) | 18:29 |
| mgariepy | lol | 18:29 |
| mgariepy | well i need to fixup some neutron stuff then i'll deploy neutron for the upgrade. | 18:30 |
| mgariepy | i can test it then. | 18:30 |
| noonedeadpunk | I just don't want to replace import back, since then we have +200 skipped tasks in aio | 18:30 |
| mgariepy | no worries. | 18:31 |
| mgariepy | after reading it i'm pretty sure it will work. | 18:32 |
| mgariepy | i just want to test it on my deployment. | 18:32 |
| mgariepy | out of curiosity do you guys sometimes experiences task failure somewhat randomly when you run againts 100-ish hosts ? | 18:49 |
| jamesdenton | noonedeadpunk any preference for bionic vs focal tests? | 18:54 |
| noonedeadpunk | focal | 19:17 |
| noonedeadpunk | no need in bionic nowadays | 19:17 |
| jamesdenton | cool thanks! | 19:20 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible master: Add Neutron Role Testing Overrides https://review.opendev.org/c/openstack/openstack-ansible/+/778008 | 19:20 |
| mgariepy | noonedeadpunk, it works !! | 19:21 |
| mgariepy | better than my config hahaha | 19:21 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: Add integrated tests various Neutron deployment scenarios https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/778015 | 19:25 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: Add integrated tests various Neutron deployment scenarios https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/778015 | 19:39 |
| noonedeadpunk | mgariepy: great news, thanks! | 19:41 |
| mgariepy | not for my config lol | 19:42 |
| *** jawad_axd has quit IRC | 20:01 | |
| noonedeadpunk | lol | 20:06 |
| *** jawad_axd has joined #openstack-ansible | 20:07 | |
| *** jawad_axd has quit IRC | 20:10 | |
| admin0 | is it noticable that TASK [python_venv_build : Install python packages into the venv ] step in the utility container takes a very long time .. (setup-infra) .. or is it just me ? | 20:20 |
| admin0 | i see that this tasks installs every known client .. even that is not in use or specified in openstack | 20:24 |
| admin0 | i mean i never use sahara, zun, virtage, murano etc .. but this steps downloads every known client .. since osa alrady detects what componets are being used by *-infra-hosts or *-hosts, maybe its a good idea to only install those detected clients ? | 20:27 |
| mgariepy | http://paste.openstack.org/show/803121/ | 20:27 |
| mgariepy | isn't ovs deploy supposed to create the bridge automatically | 20:29 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/772027 | 20:37 |
| openstackgerrit | Merged openstack/openstack-ansible-os_octavia master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/772042 | 20:37 |
| admin0 | mgariepy, having trouble with ovs ? | 20:37 |
| mgariepy | yep https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/ovs_config.yml | 20:38 |
| mgariepy | the bridge_mapping is not defined.. | 20:38 |
| admin0 | i have netplan and non netplan ovs + osa .. few clusters | 20:39 |
| admin0 | not sure what this file is supposed to do :) | 20:39 |
| mgariepy | do you create your ovs brigdge via netplan ? | 20:39 |
| admin0 | i creates the bridges beforehand like you are supposed to do | 20:40 |
| admin0 | then osa takes care of the rest to plug it in | 20:40 |
| mgariepy | https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-openvswitch.html#configuring-bridges-open-vswitch | 20:40 |
| mgariepy | Bridges specified here will be created automatically. If network_interface is defined, the interface will be placed into the bridge automatically. | 20:40 |
| *** mgagne has joined #openstack-ansible | 20:40 | |
| jamesdenton | bridge mapping comes from neutron_provider_networks.network_mappings | 20:41 |
| jamesdenton | it's just a temporary var | 20:41 |
| admin0 | "Bridges specified here will be created automatically" -- this i never tested coz i always pre-create and ping the bridges befor i start osa stuff | 20:41 |
| admin0 | to ensure its not ip or vlan issues later on | 20:41 |
| admin0 | for example if your br-vlan or br-vlan is on top of a vlan or a bond , you want to make sure those settings are good before you start osa | 20:42 |
| mgariepy | jamesdenton, yep that's with i tought also. and it was working before V in my case. | 20:42 |
| jamesdenton | well, that's concerning. | 20:42 |
| jamesdenton | :D | 20:42 |
| admin0 | hey jamesdenton .. any new books coming up :D ? | 20:42 |
| jamesdenton | i have not done a deployment recently | 20:42 |
| jamesdenton | admin0 sure! you can be the ghostwriter | 20:42 |
| admin0 | :) | 20:43 |
| mgariepy | hmm.. it also work on my network node.. lol | 20:43 |
| jamesdenton | mgariepy are you still using LXC? | 20:44 |
| mgariepy | i moved my network nodes on metal. | 20:44 |
| jamesdenton | and are you expecting an linux bridge to be created? That task should create an OVS bridge. But you don't need one for tunnel traffic like that. the OVS agent creates br-tun for you | 20:44 |
| mgariepy | no lxb needed. | 20:45 |
| mgariepy | ovs all the way ;P | 20:45 |
| mgariepy | ha. so there is a missing condition then. | 20:46 |
| *** luksky has quit IRC | 20:46 | |
| jamesdenton | probably neutron_provider_networks.network_mappings is defined | 20:47 |
| mgariepy | it works for my network node because it needs to create br-client ( for public) but not the one for tunnel. | 20:47 |
| jamesdenton | you don't really need a bridge for tunnel, the IP just needs to exist somewhere (vlan800 in your example) | 20:48 |
| mgariepy | yeah. | 20:48 |
| admin0 | i am stuck at TASK [python_venv_build : Install python packages into the venv] .. its sloooow .. i can see tail -f /var/log/python_venv_build.log in the utility and i see it working (slowly) .. . i think i am hitting the task timeout before this completes | 20:51 |
| mgariepy | jamesdenton, http://paste.openstack.org/show/803123/ | 20:55 |
| admin0 | i think this was seen before ? Failed building wheel for netifaces | 20:55 |
| admin0 | i failed to document the fix for this | 20:55 |
| mgariepy | i guess we should test like this one: https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/ovs_config.yml#L41 | 20:56 |
| admin0 | what could cause this error: https://pastebin.com/raw/qhf8e4p1 -- my build is stuck here | 20:57 |
| admin0 | the time i recall this error was a mtu issue ... i can validate that i don;t have a mtu issue in the new one | 20:59 |
| admin0 | i don't recall other | 20:59 |
| jamesdenton | mgariepy that sounds reasonable | 21:00 |
| jamesdenton | so the task actually fails for you? | 21:00 |
| jamesdenton | wonder if behavior changed for 'is defined != empty' or something | 21:02 |
| mgariepy | yes it does because the .split(",") returns nothing then bridge_mapping is not defined ;) | 21:02 |
| mgariepy | wekll it returns "" | 21:02 |
| mgariepy | maybe it changed.. | 21:02 |
| *** luksky has joined #openstack-ansible | 21:02 | |
| mgariepy | i dont have the trace anymore for these runs.. | 21:03 |
| mgariepy | exotic configuration always means troubles haha | 21:05 |
| jamesdenton | sorry to have failed you | 21:06 |
| jamesdenton | :D | 21:06 |
| mgariepy | well, i fialed myself. | 21:06 |
| mgariepy | i usualy do test on a testbed first.. | 21:06 |
| mgariepy | but.. i was lazy this time around haha | 21:06 |
| mgariepy | it was such a minor change.. U > V .. | 21:07 |
| mgariepy | i fixed the ovs flow issue i encounter the other day. | 21:07 |
| jamesdenton | oh? i vaguely remember | 21:09 |
| mgariepy | https://github.com/openstack/neutron/commit/f9827eba86690f16818fad1c5684da3fc93c433b | 21:10 |
| jamesdenton | oh right, very nice! | 21:11 |
| mgariepy | when ovs restart (like when there was a pkg update) neutron was forgetting some stuff. haha | 21:11 |
| mgariepy | no filter named 'ength'. >< typos!! | 21:13 |
| openstackgerrit | Marc Gariépy proposed openstack/openstack-ansible-os_neutron master: Fix conditionali for ovs https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/778075 | 21:23 |
| openstackgerrit | Marc Gariépy proposed openstack/openstack-ansible-os_neutron master: Fix conditional for ovs https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/778075 | 21:24 |
| *** LowKey has joined #openstack-ansible | 21:28 | |
| *** spatel has quit IRC | 21:33 | |
| *** spatel has joined #openstack-ansible | 21:34 | |
| mgariepy | EXIT NOTICE [Playbook execution success] ************************************** | 21:49 |
| mgariepy | \o/ | 21:49 |
| jamesdenton | congrats :) | 21:50 |
| mgariepy | jamesdenton, thanks | 21:50 |
| openstackgerrit | Ebbex proposed openstack/openstack-ansible-ops stable/victoria: Fix regex_replace mapping https://review.opendev.org/c/openstack/openstack-ansible-ops/+/778096 | 21:59 |
| openstackgerrit | Ebbex proposed openstack/openstack-ansible-ops stable/victoria: Fixes for tests and gates, stable/victoria https://review.opendev.org/c/openstack/openstack-ansible-ops/+/778084 | 22:25 |
| *** gshippey has quit IRC | 22:29 | |
| *** luksky has quit IRC | 22:41 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Make focal jobs voting https://review.opendev.org/c/openstack/openstack-ansible-ops/+/777940 | 22:46 |
| *** luksky has joined #openstack-ansible | 22:54 | |
| openstackgerrit | Ebbex proposed openstack/openstack-ansible-ops stable/victoria: Fixes for tests and gates, stable/victoria https://review.opendev.org/c/openstack/openstack-ansible-ops/+/778084 | 22:57 |
| *** spatel has quit IRC | 23:00 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Convert host and container journal collection to python https://review.opendev.org/c/openstack/openstack-ansible/+/734883 | 23:04 |
| admin0 | msg": "unable to connect to database, check login_user and login_password are correct or /root/.my.cnf has the credentials. -- os-keystone .. from where is this attempted to create the database ? | 23:20 |
| admin0 | is it from the keystone container itself, or util or repo ? | 23:20 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Convert host and container journal collection to python https://review.opendev.org/c/openstack/openstack-ansible/+/734883 | 23:21 |
| admin0 | telnet 172.29.239.79 3306 from keystone container to galera works fine | 23:21 |
| admin0 | hmm.. ERROR 2013 (HY000): Lost connection to MySQL server at 'handshake: reading initial communication packet', system error: 11 | 23:22 |
| admin0 | that happens when i do mysql from the util container | 23:22 |
| admin0 | checking.. | 23:22 |
| admin0 | mysql itself is fine | 23:23 |
| admin0 | maybe haproxy | 23:23 |
| jrosser | admin0> is it from the keystone container itself, or util or repo ? | 23:24 |
| jrosser | ^ all of this is in the ansible log which you don't post :/ | 23:24 |
| jrosser | i couldnt really help earlier with the venv build stuff becasue of no context on where you error was | 23:25 |
| admin0 | jrosser, that venv error was in 22.0.1 , but i did a stable/victoria checkout today and it was gone | 23:27 |
| admin0 | this is what i am seeing: https://gist.github.com/a1git/5775af6ee102b5fdce91a769f961c903 | 23:27 |
| admin0 | related to mysql error | 23:28 |
| admin0 | for some reason, accessing galera via haproxy ip is not working | 23:28 |
| jrosser | you're using the localhost unix socket arent you when logging into the galera container? | 23:29 |
| jrosser | not tcp, so i don't think its comparing like with like | 23:29 |
| admin0 | mysql -h 172.29.236.11 -u root -p -- works | 23:30 |
| admin0 | where the -h is the hostname | 23:30 |
| admin0 | sorry .. .11 is the lb does not work | 23:30 |
| admin0 | mysql -h 172.29.239.79 -u root -p = direct ip = works | 23:30 |
| admin0 | in other clusters, -h lb ip also works | 23:31 |
| jrosser | well your troubles usually turn out to be network related :) | 23:31 |
| jrosser | this doesnt feel like an OSA issue really | 23:31 |
| admin0 | .11 is the ip of the single controller c1 ,, where the galera lxc container is | 23:32 |
| admin0 | this config was created like 5 mins ago by the playbook | 23:32 |
| admin0 | i don't see where my network has this into play | 23:32 |
| admin0 | telnet 172.29.239.79 3306 from controller itself ( where haproxy runs_) also works fine | 23:33 |
| admin0 | i will try to check what the issue is | 23:34 |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Slurp constraints for all hosts https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/777994 | 23:40 |
| admin0 | so i # the galera setting in haproxy, installed rinetd .. did 172.29.236.11 3306 172.29.239.79 3306 .. which is basically the same as haproxy and it works fine | 23:42 |
| admin0 | so its not the network, but something in haproxy | 23:42 |
| watersj | Hi all, can ironic and nova (vm instances) be hosted within same cloud? meaning having bare metal instances mixed with vm instances? | 23:46 |
| openstackgerrit | Ebbex proposed openstack/openstack-ansible-ops stable/victoria: Fixes for tests and gates, stable/victoria https://review.opendev.org/c/openstack/openstack-ansible-ops/+/778084 | 23:46 |
| admin0 | watersj, they can be | 23:49 |
| admin0 | but a node dedicated to ironic will not host vms | 23:49 |
| admin0 | its just there sitting idle waiting for someone to book it and make use of it | 23:50 |
| jrosser | admin0: http://paste.openstack.org/show/803129/ | 23:56 |
| admin0 | is 236.101 your lb ip ? | 23:57 |
| jrosser | it's an AIO, standard layout from that | 23:57 |
| jrosser | but yes, internal lb ip | 23:57 |
| jrosser | you should check in the galera journal that the service has actually started correctly | 23:58 |
| admin0 | galera is fine .. i was able to get os-keystone working by using rinetd for galera instead of haproxy .. so i do not see any issues in the galera side | 23:59 |
| admin0 | right now, i am trying to figure out why haproxy (process) -> galera is broken | 23:59 |
| jrosser | and the healthcheck? | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!